[
  {
    "path": ".gitignore",
    "content": "models/*.tar.gz\nmodels/caffemodels/\nmodels/torchmodels/"
  },
  {
    "path": "LICENSE",
    "content": "MIT License\n\nCopyright (c) 2017 Deepak Pathak\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\nSOFTWARE."
  },
  {
    "path": "README.md",
    "content": "## Learning Features by Watching Objects Move ##\nIn CVPR 2017. [[Project Website]](http://cs.berkeley.edu/~pathak/unsupervised_video/).\n\n[Deepak Pathak](https://people.eecs.berkeley.edu/~pathak/), [Ross Girshick](http://www.rossgirshick.info/), [Piotr Doll&aacute;r](https://pdollar.github.io/), [Trevor Darrell](https://people.eecs.berkeley.edu/~trevor/), [Bharath Hariharan](http://home.bharathh.info/)<br/>\nUniversity of California, Berkeley<br/>\nFacebook AI Research (FAIR)<br/>\n\n<img src=\"images/overview.jpg\" width=\"550\">\n\nThis is the code for our [CVPR 2017 paper on Unsupervised Learning using unlabeled videos](http://cs.berkeley.edu/~pathak/unsupervised_video/). This repository contains models trained by the unsupervised motion grouping algorithm both in Caffe and Torch. If you find this work useful in your research, please cite:\n\n    @inproceedings{pathakCVPR17learning,\n        Author = {Pathak, Deepak and Girshick, Ross and Doll\\'{a}r,\n                  Piotr and Darrell, Trevor and Hariharan, Bharath},\n        Title = {Learning Features by Watching Objects Move},\n        Booktitle = {Computer Vision and Pattern Recognition ({CVPR})},\n        Year = {2017}\n    }\n\n### 1) Fetching Models for Unsupervised Transfer\nThe models below only contains the layer that are used for unsupervised transfer learning. For the full model that contains motion segmentation, see next section.\n\n1. Clone the repository\n  ```Shell\n  git clone https://github.com/pathak22/unsupervised-video.git\n  ```\n\n2. Fetch caffe models\n  ```Shell\n  cd unsupervised-video/\n  bash ./models/download_caffe_models.sh\n  # This will populate the `./models/` folder with trained models.\n  ```\n  The models were initially trained in Torch and then converted to caffe. Hence, please include pycaffe based `image_transform_layer.py` in your folder. It converts the scale and mean of the input image as needed.\n\n3. Fetch torch models\n  ```Shell\n  cd unsupervised-video/\n  bash ./models/download_torch_models.sh\n  # This will populate the `./models/` folder with trained models.\n  ```\n\n### 2) Fetching Motion Segmentation models\nFollow the instructions below to download full motion segmentation model trained on the automatically selected 205K videos from YFCC100m. I trained it in Torch, but you can train your own model from the full data [available here](https://people.eecs.berkeley.edu/~pathak/unsupervised_video/index.html#data) in any deep learning package using the training details from paper.\n```Shell\ncd unsupervised-video/\nbash ./models/download_torch_motion_model.sh\n# This will populate the `./models/` folder with trained model.\n\ncd motionseg/\nth load_motionmodel.lua -input ../models/motionSegmenter_fullModel.t7\n```\n\n### 3) Additional Software Packages\n\nWe are releasing software packages which were developed in the project, but could be generally useful for computer vision research. If you find them useful, please consider citing our work. These include:\n\n(a) <a href='https://github.com/pathak22/videoseg'><b>uNLC [github]</b></a>: Implementation of unsupervised bottom-up video segmentation algorithm which is unsupervised adaptation of NLC algorithm by Faktor and Irani, BMVC 2014. For additional details, see section 5.1 in the <a href=\"http://cs.berkeley.edu/~pathak/unsupervised_video/\">paper</a>.<br/><br/>\n(b) <a href='https://github.com/pathak22/pyflow'><b>PyFlow [github]</b></a>: This is python wrapper around Ce Liu's <a href=\"http://people.csail.mit.edu/celiu/OpticalFlow/\" target=\"_blank\">C++ implementation</a> of Coarse2Fine Optical Flow. This is used inside uNLC implementation, and also generally useful as an independent package.\n"
  },
  {
    "path": "image_transform_layer.py",
    "content": "\"\"\"\nTransform images for compatibility with models trained with\nhttps://github.com/facebook/fb.resnet.torch.\n\nUsage in model prototxt:\nlayer {\n  name: 'data_xform'\n  type: 'Python'\n  bottom: 'data_caffe'\n  top: 'data'\n  python_param {\n    module: 'image_transform_layer'\n    layer: 'TorchImageTransformLayer'\n  }\n}\n\"\"\"\n\nimport caffe\nimport numpy as np\n\n\nclass TorchImageTransformLayer(caffe.Layer):\n    def setup(self, bottom, top):\n        # (1, 3, 1, 1) shaped arrays\n        self.PIXEL_MEANS = \\\n            np.array([[[[0.485]],\n                       [[0.456]],\n                       [[0.406]]]])\n        self.PIXEL_STDS = \\\n            np.array([[[[0.229]],\n                       [[0.224]],\n                       [[0.225]]]])\n        top[0].reshape(*(bottom[0].shape))\n\n    def forward(self, bottom, top):\n        ims = bottom[0].data\n        # 1. Permute BGR to RGB and normalize to [0, 1]\n        ims = ims[:, [2, 1, 0], :, :] / 255.0\n        # 2. Remove channel means\n        ims -= self.PIXEL_MEANS\n        # 3. Standardize channels\n        ims /= self.PIXEL_STDS\n        top[0].reshape(*(ims.shape))\n        top[0].data[...] = ims\n\n    def backward(self, top, propagate_down, bottom):\n        \"\"\"This layer does not propagate gradients.\"\"\"\n        pass\n\n    def reshape(self, bottom, top):\n        \"\"\"Reshaping happens during the call to forward.\"\"\"\n        pass\n"
  },
  {
    "path": "models/download_caffe_models.sh",
    "content": "#!/bin/bash\n\nDIR=\"$( cd \"$( dirname \"${BASH_SOURCE[0]}\" )/\" && pwd )\"\ncd $DIR\n\nFILE=caffemodels.tar.gz\nURL=https://dl.fbaipublicfiles.com/unsupervised-video/$FILE\nCHECKSUM=29e4a50f4fc77b0563a201f28577a895\n\nif [ ! -f $FILE ]; then\n  echo \"Downloading the unsupervised video caffemodels (829MB)...\"\n  wget $URL -O $FILE\n  echo \"Unzipping...\"\n  tar zxvf $FILE\n  echo \"Downloading Done.\"\nelse\n  echo \"File already exists. Checking md5...\"\nfi\n\nos=`uname -s`\nif [ \"$os\" = \"Linux\" ]; then\n  checksum=`md5sum $FILE | awk '{ print $1 }'`\nelif [ \"$os\" = \"Darwin\" ]; then\n  checksum=`cat $FILE | md5`\nelif [ \"$os\" = \"SunOS\" ]; then\n  checksum=`digest -a md5 -v $FILE | awk '{ print $4 }'`\nfi\nif [ \"$checksum\" = \"$CHECKSUM\" ]; then\n  echo \"Checksum is correct. File was correctly downloaded.\"\n  exit 0\nelse\n  echo \"Checksum is incorrect. DELETE and download again.\"\nfi\n"
  },
  {
    "path": "models/download_torch_models.sh",
    "content": "#!/bin/bash\n\nDIR=\"$( cd \"$( dirname \"${BASH_SOURCE[0]}\" )/\" && pwd )\"\ncd $DIR\n\nFILE=torchmodels.tar.gz\nURL=https://dl.fbaipublicfiles.com/unsupervised-video/$FILE\nCHECKSUM=6ead77d7b387b51426ccc5d3c95f78bb\n\nif [ ! -f $FILE ]; then\n  echo \"Downloading the unsupervised video torchmodels (803MB)...\"\n  wget $URL -O $FILE\n  echo \"Unzipping...\"\n  tar zxvf $FILE\n  echo \"Downloading Done.\"\nelse\n  echo \"File already exists. Checking md5...\"\nfi\n\nos=`uname -s`\nif [ \"$os\" = \"Linux\" ]; then\n  checksum=`md5sum $FILE | awk '{ print $1 }'`\nelif [ \"$os\" = \"Darwin\" ]; then\n  checksum=`cat $FILE | md5`\nelif [ \"$os\" = \"SunOS\" ]; then\n  checksum=`digest -a md5 -v $FILE | awk '{ print $4 }'`\nfi\nif [ \"$checksum\" = \"$CHECKSUM\" ]; then\n  echo \"Checksum is correct. File was correctly downloaded.\"\n  exit 0\nelse\n  echo \"Checksum is incorrect. DELETE and download again.\"\nfi\n"
  },
  {
    "path": "models/download_torch_motion_model.sh",
    "content": "#!/bin/bash\n\nDIR=\"$( cd \"$( dirname \"${BASH_SOURCE[0]}\" )/\" && pwd )\"\ncd $DIR\n\nFILE=torchmodels_motion.tar.gz\nURL=https://dl.fbaipublicfiles.com/unsupervised-video/$FILE\nCHECKSUM=497efcdf10630cf6fd83d9b367765934\n\nif [ ! -f $FILE ]; then\n  echo \"Downloading the unsupervised video motion segmentation torchmodel (238MB)...\"\n  wget $URL -O $FILE\n  echo \"Unzipping...\"\n  tar zxvf $FILE\n  echo \"Downloading Done.\"\nelse\n  echo \"File already exists. Checking md5...\"\nfi\n\nos=`uname -s`\nif [ \"$os\" = \"Linux\" ]; then\n  checksum=`md5sum $FILE | awk '{ print $1 }'`\nelif [ \"$os\" = \"Darwin\" ]; then\n  checksum=`cat $FILE | md5`\nelif [ \"$os\" = \"SunOS\" ]; then\n  checksum=`digest -a md5 -v $FILE | awk '{ print $4 }'`\nfi\nif [ \"$checksum\" = \"$CHECKSUM\" ]; then\n  echo \"Checksum is correct. File was correctly downloaded.\"\n  exit 0\nelse\n  echo \"Checksum is incorrect. DELETE and download again.\"\nfi\n"
  },
  {
    "path": "motionseg/DeepMaskAlexNet.lua",
    "content": "--[[ DeepMask model:\nWhen initialized, it creates/load the common trunk, the maskBranch and the\nscoreBranch or the colorBranch or the flowBranch.\n---- deepmask class members:\n-- self.trunk: the common trunk (pre-trained resnet50)\n-- self.maskBranch: the mask head architecture\n-- self.scoreBranch: the score head architecture\n-- self.colorBranch: the colorization head architecture\n-- self.flowBranch: the flow head architecture\n]]\n\nrequire 'nn'\nrequire 'nnx'\nrequire 'cunn'\nrequire 'cudnn'\nlocal utils = paths.dofile('utilsModel.lua')\npaths.dofile('SpatialSymmetricPadding.lua')\n\nlocal DeepMask,_ = torch.class('nn.DeepMask','nn.Container')\n\n-- function: conv2linear\nlocal function linear2conv(x)\n  if torch.typename(x):find('Linear') then\n    -- hard-coding for fc6 and fc7: kSz=kernelSize=inputFeatureMapSize\n    local kSz = x.weight:size(2) > 5000 and 6 or 1\n\n    local nInp = x.weight:size(2)/(kSz*kSz)\n    local nOut = x.weight:size(1)\n    local w = torch.reshape(x.weight,nOut,nInp,kSz,kSz)\n    local y = cudnn.SpatialConvolution(nInp,nOut,kSz,kSz,1,1,0,0)\n    y.weight:copy(w)\n    y.gradWeight:copy(w)\n    if x.bias~=nil then\n      y.bias:copy(x.bias)\n      y.gradBias:copy(x.gradBias)\n    end\n    return y\n  elseif torch.typename(x):find('cudnn.BatchNormalization') then\n     x.nDim = 4\n     return x\n  else\n    return x\n  end\nend\n\n--------------------------------------------------------------------------------\n-- function: constructor\nfunction DeepMask:__init(config)\n   self.color = config.color\n   self.flow = config.flow\n   if config.noFC then\n      print('| create AlexNet (w/o FCs) Trunk')\n   else\n      print('| create AlexNet (including FCs) Trunk')\n   end\n   if config.symmPad then\n      print('| using symmetric padding')\n   else\n      print('| no symmetric padding')\n   end\n   if config.centralCrop then\n      print('| using central cropping')\n   else\n      print('| no central cropping')\n   end\n   if config.bottleneck then\n      print('| using bottleneck')\n   else\n      print('| no bottleneck')\n   end\n\n   -- create common trunk\n   self:createTrunk(config)\n   local npt = 0\n   local p1  = self.trunk:parameters()\n   for k,v in pairs(p1) do npt = npt+v:nElement() end\n   print(string.format('| number of paramaters trunk: %d', npt))\n\n   if self.flow then\n         -- create flow head\n         self:createFlowBranch(config)\n\n         local p5, npf = self.flowBranch:parameters(), 0\n         for k,v in pairs(p5) do npf = npf+v:nElement() end\n         print(string.format('| number of paramaters flow branch: %d', npf))\n         print(string.format('| number of paramaters total: %d', npt+npf))\n         return\n   end\n\n   -- create mask head\n   self:createMaskBranch(config)\n   local npm = 0\n   local p2  = self.maskBranch:parameters()\n   for k,v in pairs(p2) do npm = npm+v:nElement() end\n   print(string.format('| number of paramaters mask branch: %d', npm))\n\n   if self.color then\n      -- create colorization head\n      self:createColorBranch(config)\n\n      local p4, npc = self.colorBranch:parameters(), 0\n      for k,v in pairs(p4) do npc = npc+v:nElement() end\n      print(string.format('| number of paramaters color branch: %d', npc))\n      print(string.format('| number of paramaters total: %d', npt+npm+npc))\n   else\n      -- create score head\n      self:createScoreBranch(config)\n\n      local p3, nps = self.scoreBranch:parameters(), 0\n      for k,v in pairs(p3) do nps = nps+v:nElement() end\n      print(string.format('| number of paramaters score branch: %d', nps))\n      print(string.format('| number of paramaters total: %d', npt+nps+npm))\n   end\nend\n\n--------------------------------------------------------------------------------\n-- function: create common trunk\nfunction DeepMask:createTrunk(config)\n   -- size of feature maps at end of trunk\n   if config.padAlexNet then\n      if config.iSz==180 then\n         -- self.fSz = config.noFC and 12 or 5  -- alexnet_padded w/o dilation\n         self.fSz = 12  -- alexnet_padded w/ dilation\n      else\n         print('Unknown size setting !! Cant create AlexNet trunk')\n         os.exit()\n      end\n   else\n      -- iSz=227 ; for w/ FC\n      -- iSz=179 ; for w/o FC\n      if config.iSz==160 then\n         self.fSz = config.noFC and 8 or -1\n      elseif config.iSz==179 then\n         self.fSz = config.noFC and 10 or -1\n      elseif config.iSz==227 then\n         self.fSz = config.noFC and 13 or 1\n      else\n         print('Unknown size setting !! Cant create AlexNet trunk')\n         os.exit()\n      end\n   end\n   self.channels = config.noFC and 128 or 4096\n   self.bottleneck = self.channels*self.fSz*self.fSz\n\n   -- load trunk\n   local trunk\n   print('    | creating trunk:')\n   if #config.useImagenet > 0 then\n     print(string.format('    | using Imagenet pre-trained AlexNet: %s',\n        config.useImagenet))\n     trunk = torch.load(config.useImagenet)\n     -- Format of sgross's old fb.resnet training code\n     if trunk.state ~= nil then\n       trunk = trunk.state.network\n     end\n     -- remove DataParallelTable\n     if torch.type(trunk) == 'nn.DataParallelTable' then\n       trunk = trunk:get(1)\n     end\n     if config.useBN then\n       print('    | keeping BatchNorm in pre-trained model (if present)')\n     else\n       print('    | fixing BatchNorm in pre-trained model (if present)')\n       utils.BNtoFixed(trunk, true)\n     end\n   elseif config.useBN then\n     print('    | using AlexNet with BatchNorm from scratch !')\n     local alexnet = paths.dofile('./models/alexnetbn.lua')\n     trunk = alexnet()\n   else\n     print('    | using AlexNet without BatchNorm from scratch !')\n     local alexnet = config.padAlexNet and paths.dofile(\n         './models/alexnet_padded.lua') or paths.dofile('./models/alexnet.lua')\n     trunk = alexnet()\n   end\n  --  print('    | loaded trunk model:')\n  --  print(trunk)\n\n   -- remove fc8\n   trunk:remove();\n\n   if config.noFC then\n      -- remove fc7\n      trunk:remove();trunk:remove();\n      if torch.typename(trunk.modules[#trunk.modules]):find('BatchNorm') then\n        trunk:remove();\n      end\n      trunk:remove();\n      -- remove fc6\n      trunk:remove();trunk:remove();\n      if torch.typename(trunk.modules[#trunk.modules]):find('BatchNorm') then\n        trunk:remove();\n      end\n      trunk:remove();\n      if torch.typename(trunk.modules[#trunk.modules]):find('View') then\n        trunk:remove();\n      end\n      -- remove pool5\n      trunk:remove();\n\n      -- crop central pad : see DataSamplerCoco.wSz\n      if config.centralCrop then\n         trunk:add(nn.SpatialZeroPadding(-1,-1,-1,-1))\n      end\n\n      -- add common extra layers\n      trunk:add(cudnn.SpatialConvolution(256,128,1,1,1,1))\n      if config.useBN then\n        trunk:add(cudnn.SpatialBatchNormalization(128))\n      end\n      trunk:add(nn.ReLU(true))\n   else\n      if #config.useImagenet > 0 then\n        print('    | FC to Conv conversion in pre-trained model')\n        local startFCLayer = 16\n        if config.useBN then\n          startFCLayer = 19\n        end\n        local j=startFCLayer\n        for i=startFCLayer,#trunk.modules do\n          if not torch.typename(trunk.modules[i]):find('View') then\n           trunk.modules[j] = linear2conv(trunk.modules[i])\n           j=j+1\n          end\n        end\n        for j=j,#trunk.modules do\n          trunk:remove()\n        end\n      end\n\n      -- crop central pad : see DataSamplerCoco.wSz\n      if config.centralCrop then\n         trunk:add(nn.SpatialZeroPadding(-1,-1,-1,-1))\n      end\n   end\n   -- trunk:add(nn.View(config.batch,self.bottleneck))\n\n   -- low-rank bottleneck\n   if config.bottleneck then\n      trunk:add(nn.Linear(self.bottleneck,512))\n      if config.useBN then\n         trunk:add(cudnn.BatchNormalization(512))\n      end\n      self.bottleneck = 512\n    end\n\n   -- mirrorPadding\n   if config.symmPad then\n      utils.updatePadding(trunk, nn.SpatialSymmetricPadding)\n   end\n\n   self.trunk = trunk:cuda()\n\n   print('    | finalized trunk model:')\n   print(trunk)\n   return trunk\nend\n\n--------------------------------------------------------------------------------\n-- function: create mask branch\nfunction DeepMask:createMaskBranch(config)\n   local maskBranch = nn.Sequential()\n\n   -- maskBranch\n   if not config.bottleneck then\n      maskBranch:add(nn.View(config.batch,self.bottleneck))\n   end\n   maskBranch:add(nn.Linear(self.bottleneck,config.oSz*config.oSz))\n   self.maskBranch = nn.Sequential():add(maskBranch:cuda())\n\n   -- upsampling layer\n   if config.gSz > config.oSz then\n      local upSample = nn.Sequential()\n      upSample:add(nn.Copy('torch.CudaTensor','torch.FloatTensor'))\n      upSample:add(nn.View(config.batch,config.oSz,config.oSz))\n      upSample:add(nn.SpatialReSamplingEx{owidth=config.gSz,oheight=config.gSz,\n         mode='bilinear'})\n      upSample:add(nn.View(config.batch,config.gSz*config.gSz))\n      upSample:add(nn.Copy('torch.FloatTensor','torch.CudaTensor'))\n      self.maskBranch:add(upSample)\n   end\n\n   print('    | finalized mask model:')\n   print(self.maskBranch)\n   return self.maskBranch\nend\n\n--------------------------------------------------------------------------------\n-- function: create score branch\nfunction DeepMask:createScoreBranch(config)\n   local scoreBranch = nn.Sequential()\n   if not config.bottleneck then\n      scoreBranch:add(nn.View(config.batch,self.bottleneck))\n   end\n   scoreBranch:add(nn.Dropout(.5))\n   scoreBranch:add(nn.Linear(self.bottleneck,1024))\n   if config.useBN then\n     scoreBranch:add(cudnn.BatchNormalization(1024))\n   end\n   scoreBranch:add(nn.Threshold(0, 1e-6))\n\n   scoreBranch:add(nn.Dropout(.5))\n   scoreBranch:add(nn.Linear(1024,1))\n\n   self.scoreBranch = scoreBranch:cuda()\n   print('    | finalized score model:')\n   print(self.scoreBranch)\n   return self.scoreBranch\nend\n\n--------------------------------------------------------------------------------\n-- function: create colorization branch\nfunction DeepMask:createColorBranch(config)\n   if config.bottleneck then\n      print('config.bottleneck in trunk is not supported with Color Task !!')\n      os.exit()\n   end\n   local colorBranch = nn.Sequential()\n   colorBranch:add(nn.SpatialFullConvolution(self.channels,256,4,4,2,2,1,1))\n   colorBranch:add(nn.ReLU(true))\n   colorBranch:add(cudnn.SpatialConvolution(256,313,3,3,1,1,1,1))\n   colorBranch:add(nn.SpatialUpSamplingBilinear({oheight=config.cgSz,\n                                                   owidth=config.cgSz}))\n   self.colorBranch = colorBranch:cuda()\n   print('    | finalized color model:')\n   print(self.colorBranch)\n   return self.colorBranch\nend\n\n--------------------------------------------------------------------------------\n-- function: create flow branch\nfunction DeepMask:createFlowBranch(config)\n   if config.bottleneck then\n      print('config.bottleneck in trunk is not supported with Flow Task !!')\n      os.exit()\n   end\n   local flowBranch = nn.Sequential()\n   flowBranch:add(cudnn.SpatialConvolution(self.channels,\n                                             config.numCl,3,3,1,1,1,1))\n   -- upsample if fgSz > 12 (e.g. 100)\n   -- flowBranch:add(nn.SpatialUpSamplingBilinear({oheight=config.fgSz,\n   --                                                 owidth=config.fgSz}))\n   self.flowBranch = flowBranch:cuda()\n   print('    | finalized flow model:')\n   print(self.flowBranch)\n   return self.flowBranch\nend\n\n--------------------------------------------------------------------------------\n-- function: training\nfunction DeepMask:training()\n   self.trunk:training()\n   if self.flow then\n      self.flowBranch:training()\n      return\n   end\n   self.maskBranch:training()\n   if self.color then\n      self.colorBranch:training()\n   else\n      self.scoreBranch:training()\n   end\nend\n\n--------------------------------------------------------------------------------\n-- function: evaluate\nfunction DeepMask:evaluate()\n   self.trunk:evaluate()\n   if self.flow then\n      self.flowBranch:evaluate()\n      return\n   end\n   self.maskBranch:evaluate()\n   if self.color then\n      self.colorBranch:evaluate()\n   else\n      self.scoreBranch:evaluate()\n   end\nend\n\n--------------------------------------------------------------------------------\n-- function: to cuda\nfunction DeepMask:cuda()\n   self.trunk:cuda()\n   if self.flow then\n      self.flowBranch:cuda()\n      return\n   end\n   self.maskBranch:cuda()\n   if self.color then\n      self.colorBranch:cuda()\n   else\n      self.scoreBranch:cuda()\n   end\nend\n\n--------------------------------------------------------------------------------\n-- function: to float\nfunction DeepMask:float()\n   self.trunk:float()\n   if self.flow then\n      self.flowBranch:float()\n      return\n   end\n   self.maskBranch:float()\n   if self.color then\n      self.colorBranch:float()\n   else\n      self.scoreBranch:float()\n   end\nend\n\n--------------------------------------------------------------------------------\n-- function: inference (used for full scene inference)\nfunction DeepMask:inference()\n   self:cuda()\n   utils.linear2convTrunk(self.trunk,self.fSz)\n   self.trunk:evaluate()\n   self.trunk:forward(torch.CudaTensor(1,3,800,800))\n   if self.flow then\n      utils.linear2convHead(self.flowBranch)\n      self.flowBranch:evaluate()\n      self.flowBranch:forward(torch.CudaTensor(1,512,300,300))\n      return\n   end\n\n   utils.linear2convHead(self.maskBranch.modules[1])\n   self.maskBranch = self.maskBranch.modules[1]\n   self.maskBranch:evaluate()\n   self.maskBranch:forward(torch.CudaTensor(1,512,300,300))\n\n   if self.color then\n      utils.linear2convHead(self.colorBranch)\n      self.colorBranch:evaluate()\n      self.colorBranch:forward(torch.CudaTensor(1,512,300,300))\n   else\n      utils.linear2convHead(self.scoreBranch)\n      self.scoreBranch:evaluate()\n      self.scoreBranch:forward(torch.CudaTensor(1,512,300,300))\n   end\nend\n\n--------------------------------------------------------------------------------\n-- function: clone\nfunction DeepMask:clone(...)\n   local f = torch.MemoryFile(\"rw\"):binary()\n   f:writeObject(self)\n   f:seek(1)\n   local clone = f:readObject()\n   f:close()\n\n   if select('#',...) > 0 then\n      clone.trunk:share(self.trunk,...)\n      if self.flow then\n         clone.flowBranch:share(self.flowBranch,...)\n         return clone\n      end\n      clone.maskBranch:share(self.maskBranch,...)\n      if self.color then\n         clone.colorBranch:share(self.colorBranch,...)\n      else\n         clone.scoreBranch:share(self.scoreBranch,...)\n      end\n   end\n\n   return clone\nend\n\nreturn DeepMask\n"
  },
  {
    "path": "motionseg/SpatialSymmetricPadding.lua",
    "content": "--[[----------------------------------------------------------------------------\nCopyright (c) 2016-present, Facebook, Inc. All rights reserved.\nThis source code is licensed under the BSD-style license found in the\nLICENSE file in the root directory of this source tree. An additional grant\nof patent rights can be found in the PATENTS file in the same directory.\n\nSpatialSymmetricPadding module\n\nThe forward(A) pads input array A with mirror reflections of itself\nIt is the same function as Matlab padarray(A, padsize, 'symmetric' )\nThe updateGradInput(input, gradOutput) is inherited from nn.SpatialZeroPadding\nwhere the padded region is treated as constant and\nthe gradients would not be accumulated in the backward pass\n------------------------------------------------------------------------------]]\n\nlocal SpatialSymmetricPadding, parent =\n  torch.class('nn.SpatialSymmetricPadding', 'nn.SpatialZeroPadding')\n\nfunction SpatialSymmetricPadding:__init(pad_l, pad_r, pad_t, pad_b)\n   parent.__init(self, pad_l, pad_r, pad_t, pad_b)\nend\n\nfunction SpatialSymmetricPadding:updateOutput(input)\n  assert(input:dim()==4, \"only Dimension=4 implemented\")\n  -- sizes\n  local h = input:size(3) + self.pad_t + self.pad_b\n  local w = input:size(4) + self.pad_l + self.pad_r\n  if w < 1 or h < 1 then error('input is too small') end\n  self.output:resize(input:size(1), input:size(2), h, w)\n  self.output:zero()\n  -- crop input if necessary\n  local c_input = input\n  if self.pad_t < 0 then\n    c_input = c_input:narrow(3, 1 - self.pad_t, c_input:size(3) + self.pad_t)\n  end\n  if self.pad_b < 0 then\n    c_input = c_input:narrow(3, 1, c_input:size(3) + self.pad_b)\n  end\n  if self.pad_l < 0 then\n    c_input = c_input:narrow(4, 1 - self.pad_l, c_input:size(4) + self.pad_l)\n  end\n  if self.pad_r < 0 then\n    c_input = c_input:narrow(4, 1, c_input:size(4) + self.pad_r)\n  end\n  -- crop outout if necessary\n  local c_output = self.output\n  if self.pad_t > 0 then\n    c_output = c_output:narrow(3, 1 + self.pad_t, c_output:size(3) - self.pad_t)\n  end\n  if self.pad_b > 0 then\n    c_output = c_output:narrow(3, 1, c_output:size(3) - self.pad_b)\n  end\n  if self.pad_l > 0 then\n    c_output = c_output:narrow(4, 1 + self.pad_l, c_output:size(4) - self.pad_l)\n  end\n  if self.pad_r > 0 then\n    c_output = c_output:narrow(4, 1, c_output:size(4) - self.pad_r)\n  end\n  -- copy input to output\n  c_output:copy(c_input)\n  -- symmetric padding that fills in values on the padded region\n  if w<2*self.pad_l or w<2*self.pad_r or h<2*self.pad_t or h<2*self.pad_b then\n    error('input is too small')\n  end\n  for i=1,self.pad_t do\n    self.output:narrow(3,self.pad_t-i+1,1):copy(\n    self.output:narrow(3,i+self.pad_t,1))\n  end\n  for i=1,self.pad_b do\n    self.output:narrow(3,self.output:size(3)-self.pad_b+i,1):copy(\n    self.output:narrow(3,self.output:size(3)-self.pad_b-i+1,1))\n  end\n  for i=1,self.pad_l do\n    self.output:narrow(4,self.pad_l-i+1,1):copy(\n    self.output:narrow(4,i+self.pad_l,1))\n  end\n  for i=1,self.pad_r do\n    self.output:narrow(4,self.output:size(4)-self.pad_r+i,1):copy(\n    self.output:narrow(4,self.output:size(4)-self.pad_r-i+1,1))\n  end\n  return self.output\nend\n"
  },
  {
    "path": "motionseg/load_motionmodel.lua",
    "content": "require 'nn';\nrequire 'cunn';\nrequire 'cudnn';\n\npaths.dofile('DeepMaskAlexNet.lua');\nlocal cmd = torch.CmdLine()\ncmd:text()\ncmd:text('Helper script for loading model')\ncmd:text()\ncmd:option('-input', '', 'Path to input Torch model to be converted')\nlocal config = cmd:parse(arg)\n\nlocal model = torch.load(config.input);\nprint(model)\nmodel = model:float()\nmodel:evaluate()\n"
  },
  {
    "path": "motionseg/utilsModel.lua",
    "content": "-- utility functions for models\n\nlocal utils = {}\n\n--------------------------------------------------------------------------------\n-- SpatialConstDiagonal module\n-- all BN modules in resnet to be transformed into SpatialConstDiagonal\nif not nn.SpatialConstDiagonal then\n   local module, parent = torch.class('nn.SpatialConstDiagonal', 'nn.Module')\n\n   function module:__init(nOutputPlane, inplace)\n      parent.__init(self)\n      self.a = torch.Tensor(1,nOutputPlane,1,1)\n      self.b = torch.Tensor(1,nOutputPlane,1,1)\n      self.inplace = inplace\n      self:reset()\n   end\n\n   function module:reset()\n      self.a:fill(1)\n      self.b:zero()\n   end\n\n   function module:updateOutput(input)\n      if self.inplace then\n         self.output:set(input)\n      else\n         self.output:resizeAs(input):copy(input)\n      end\n      self.output:cmul(self.a:expandAs(input))\n      self.output:add(self.b:expandAs(input))\n      return self.output\n   end\n\n   function module:updateGradInput(input, gradOutput)\n      if self.inplace then\n         self.gradInput:set(gradOutput)\n      else\n         self.gradInput:resizeAs(gradOutput):copy(gradOutput)\n      end\n      self.gradInput:cmul(self.a:expandAs(gradOutput))\n      return self.gradInput\n   end\nend\n\n--------------------------------------------------------------------------------\n-- function: goes over a net and recursively replaces modules\n-- using callback function\nlocal function replace(self, callback)\n   local out = callback(self)\n   if self.modules then\n      for i=#self.modules,1,-1 do\n         local m = self.modules[i]\n         local mm = replace(m, callback)\n         if mm then self.modules[i] = mm else self:remove(i) end\n      end\n   end\n   return out\nend\n\n--------------------------------------------------------------------------------\n-- function: replace BN layer to SpatialConstDiagonal\nfunction utils.BNtoFixed(net, ip)\n   return replace(\n      net,\n      function(x)\n      if torch.typename(x):find'SpatialBatchNormalization' then\n         local no = x.running_mean:numel()\n         local y = nn.SpatialConstDiagonal(no, ip):type(x._type)\n         if x.running_var then\n            x.running_std = x.running_var:pow(-0.5)\n         end\n         y.a:copy(x.running_std)\n         y.b:add(-1,x.running_mean):cmul(x.running_std)\n         if x.affine then\n            y.a:cmul(x.weight)\n            y.b:cmul(x.weight):add(x.bias)\n         end\n         return y\n      else\n         return x\n      end\n   end\n   )\nend\n\n--------------------------------------------------------------------------------\n-- function: replace 0-padding of 3x3 conv into mirror-padding\nfunction utils.updatePadding(net, nn_padding)\n   if torch.typename(net) == \"nn.Sequential\" or\n      torch.typename(net) == \"nn.ConcatTable\" then\n      for i = #net.modules,1,-1 do\n         local out = utils.updatePadding(net:get(i), nn_padding)\n         if out ~= -1 then\n            local pw, ph = out[1], out[2]\n            net.modules[i] = nn.Sequential():add(nn_padding(pw,pw,ph,ph))\n               :add(net.modules[i]):cuda()\n         end\n      end\n   else\n      if torch.typename(net) == \"nn.SpatialConvolution\" or\n         torch.typename(net) == \"cudnn.SpatialConvolution\" then\n         if (net.kW == 3 and net.kH == 3) or (net.kW==7 and net.kH==7) or\n            (net.kW == 5 and net.kH == 5) then\n            local pw, ph = net.padW, net.padH\n            net.padW, net.padH = 0, 0\n            return {pw,ph}\n         end\n      end\n    end\n   return -1\nend\n\n--------------------------------------------------------------------------------\n-- function: linear2convTrunk\nfunction utils.linear2convTrunk(net,fSz)\n   return replace(\n   net,\n   function(x)\n      if torch.typename(x):find('Linear') then\n         local nInp,nOut = x.weight:size(2)/(fSz*fSz),x.weight:size(1)\n         local w = torch.reshape(x.weight,nOut,nInp,fSz,fSz)\n         local y = cudnn.SpatialConvolution(nInp,nOut,fSz,fSz,1,1)\n         y.weight:copy(w)\n         y.gradWeight:copy(w)\n         y.bias:copy(x.bias)\n         return y\n      elseif torch.typename(x):find('cudnn.BatchNormalization') or\n        torch.typename(x):find('nn.BatchNormalization') then\n        --  x.nDim = 4\n        --  return x\n         local nOut = x.running_mean:size(1)\n         local y = cudnn.SpatialBatchNormalization(nOut)\n         y.weight:copy(x.weight)\n         y.bias:copy(x.bias)\n         y.gradWeight:copy(x.gradWeight)\n         y.gradBias:copy(x.gradBias)\n         y.running_mean:copy(x.running_mean)\n        --  y.running_var:copy(x.running_var)\n        --  y.save_mean:copy(x.save_mean)\n        --  y.save_std:copy(x.save_std)\n         return y\n      elseif torch.typename(x):find('Threshold') then\n         return cudnn.ReLU()\n      elseif not torch.typename(x):find('View') and\n         not torch.typename(x):find('SpatialZeroPadding') then\n         return x\n      end\n   end\n   )\nend\n\n--------------------------------------------------------------------------------\n-- function: linear2convHeads\nfunction utils.linear2convHead(net)\n   return replace(\n   net,\n   function(x)\n      if torch.typename(x):find('Linear') then\n         local nInp,nOut = x.weight:size(2),x.weight:size(1)\n         local w = torch.reshape(x.weight,nOut,nInp,1,1)\n         local y = cudnn.SpatialConvolution(nInp,nOut,1,1,1,1)\n         y.weight:copy(w)\n         y.gradWeight:copy(w)\n         y.bias:copy(x.bias)\n         return y\n      elseif torch.typename(x):find('cudnn.BatchNormalization') or\n          torch.typename(x):find('nn.BatchNormalization') then\n          -- x.nDim = 4\n          -- return x\n          local nOut = x.running_mean:size(1)\n          local y = cudnn.SpatialBatchNormalization(nOut)\n          y.weight:copy(x.weight)\n          y.bias:copy(x.bias)\n          y.gradWeight:copy(x.gradWeight)\n          y.gradBias:copy(x.gradBias)\n          y.running_mean:copy(x.running_mean)\n          -- y.running_var:copy(x.running_var)\n          -- y.save_mean:copy(x.save_mean)\n          -- y.save_std:copy(x.save_std)\n          return y\n      elseif torch.typename(x):find('Threshold') then\n         return cudnn.ReLU()\n      elseif not torch.typename(x):find('View') and\n         not torch.typename(x):find('Copy') then\n         return x\n      end\n   end\n   )\nend\n\nreturn utils\n"
  }
]