[ { "path": ".gitignore", "content": "# Folders\nmodels/\ndata/\n\n# Temporal files\n*.t7\n*.mat\n*.jpg\n*.png\n\n" }, { "path": "LICENSE", "content": "The MIT License (MIT)\n\nCopyright (c) 2016 Xinchen Yan, Jimei Yang, Ersin Yumer, Yijie Guo and Honglak Lee\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\nSOFTWARE.\n" }, { "path": "README.md", "content": "# Perspective Transformer Nets (PTN)\n\nThis is the code for NIPS 2016 paper [Perspective Transformer Nets: Learning Single-View 3D Object Reconstruction without 3D Supervision](https://papers.nips.cc/paper/6206-perspective-transformer-nets-learning-single-view-3d-object-reconstruction-without-3d-supervision.pdf) by Xinchen Yan, Jimei Yang, Ersin Yumer, Yijie Guo and Honglak Lee\n\n\n\nPlease follow the instructions to run the code.\n\n## Requirements\nPTN requires or works with \n* Mac OS X or Linux\n* NVIDIA GPU\n\n## Installing Dependency\n* Install [Torch](http://torch.ch)\n* Install [Mattorch](https://github.com/clementfarabet/lua---mattorch)\n* Install [Perspective Transformer Layer](https://github.com/xcyan/ptnbhwd.git)\n\nThe following command installs the Perspective Transformer Layer:\n```\n./install_ptnbhwd.sh\n```\n\n## Dataset Downloading\n* Please run the command to download the pre-processed dataset (including rendered 2D views and 3D volumes):\n```\n./prepare_data.sh\n```\n* Disclaimer: Please cite the [ShapeNet paper](https://arxiv.org/pdf/1512.03012.pdf) as well.\n\n## Pre-trained Models Downloading (single-class experiment)\n\nPTN-Proj: ptn_proj.t7\n\nPTN-Comb: ptn_comb.t7\n\nCNN-Vol: cnn_vol.t7\n\n* The following command downloads the pre-trained models:\n```\n./download_models.sh\n```\n\n## Testing using Pre-trained Models (single-class experiment)\n\n* The following command evaluates the pre-trained models:\n```\n./eval_models.sh\n```\n\n## Training (single-class experiment)\n* If you want to pre-train the view-point indepedent image encoder on single-class, please run the following command.\nNote that the pre-training could take a few days on a single TITAN X GPU.\n```\n./demo_pretrain_singleclass.sh\n```\n* If you want to train PTN-Proj (unsupervised) on single-class based on pre-trained encoder, please run the command.\n```\n./demo_train_ptn_proj_singleclass.sh\n```\n* If you want to train PTN-Comb (3D supervision) on single-class based on pre-trained encoder, please run the command.\n```\n./demo_train_ptn_comb_singleclass.sh\n```\n* If you want to train CNN-Vol (3D supervision) on single-class based on pre-trained encoder, please run the command.\n```\n./demo_train_cnn_vol_singleclass.sh\n```\n\n## Using your own camera\n* In many cases, you want to implement your own camera matrix (e.g., intrinsic or extrinsic). \nPlease feel free to modify [this function](https://github.com/xcyan/nips16_PTN/blob/master/scripts/train_PTN.lua#L207).\n\n* Before start your own implementation, we recommand to go through some basic camera geometry in [this computer vision textbook](http://szeliski.org/Book/drafts/SzeliskiBook_20100903_draft.pdf) written by Richard Szeliski (see Eq 2.59 at Page 53).\n\n* Note that in our voxel ray-tracing implementation, we used the inverse camera matrix. \n\n## Third-party Implementation\n\nBesides our torch implementation, we recommend to see also the following third-party re-implementation:\n* [TensorFlow Implementation](https://github.com/tensorflow/models/tree/archive/research/ptn): This re-implementation was developed during Xinchen's Google internship; If you find a bug, please file a bug including @xcyan. \n\n## Citation\n\nIf you find this useful, please cite our work as follows:\n```\n@incollection{NIPS2016_6206,\ntitle = {Perspective Transformer Nets: Learning Single-View 3D Object Reconstruction without 3D Supervision},\nauthor = {Yan, Xinchen and Yang, Jimei and Yumer, Ersin and Guo, Yijie and Lee, Honglak},\nbooktitle = {Advances in Neural Information Processing Systems 29},\neditor = {D. D. Lee and M. Sugiyama and U. V. Luxburg and I. Guyon and R. Garnett},\npages = {1696--1704},\nyear = {2016},\npublisher = {Curran Associates, Inc.},\nurl = {http://papers.nips.cc/paper/6206-perspective-transformer-nets-learning-single-view-3d-object-reconstruction-without-3d-supervision.pdf}\n}\n```\n" }, { "path": "demo_pretrain_singleclass.sh", "content": "mkdir ./models\nth scripts/train_rotatorRNN_base.lua --gpu 1 --niter 160 --save_every 40\nth scripts/train_rotatorRNN_curriculum.lua --gpu 1 --kstep 2 --batch_size 32 --adam 1 --niter 40 --save_every 20\nth scripts/train_rotatorRNN_curriculum.lua --gpu 1 --kstep 4 --batch_size 32 --adam 2 --niter 40 --save_every 20\nth scripts/train_rotatorRNN_curriculum.lua --gpu 1 --kstep 8 --batch_size 32 --adam 2 --niter 40 --save_every 20\nth scripts/train_rotatorRNN_curriculum.lua --gpu 1 --kstep 12 --batch_size 16 --adam 2 --niter 40 --save_every 20\nth scripts/train_rotatorRNN_curriculum.lua --gpu 1 --kstep 16 --batch_size 8 --adam 2 --niter 40 --save_every 20\n\n" }, { "path": "demo_train_cnn_vol_singleclass.sh", "content": "mkdir ./models\nth scripts/train_PTN.lua --gpu 3 --niter 100 --lambda_vox 1 --lambda_msk 0\n\n" }, { "path": "demo_train_ptn_comb_singleclass.sh", "content": "mkdir ./models\nth scripts/train_PTN.lua --gpu 2 --niter 100 --lambda_vox 1 --lambda_msk 1\n\n" }, { "path": "demo_train_ptn_proj_singleclass.sh", "content": "mkdir ./models\nth scripts/train_PTN.lua --gpu 1 --niter 100 --lambda_vox 0 --lambda_msk 1\n\n" }, { "path": "download_models.sh", "content": "mkdir ./models\nwget -O models/ptn_proj.t7 https://umich.box.com/shared/static/5z5oci8lwdhzszf88s5azbeq1nsa15ad.t7\nwget -O models/ptn_comb.t7 https://umich.box.com/shared/static/pq7axx8ypva8c0e7jm7ajbfsm972d1de.t7 \nwget -O models/cnn_vol.t7 https://umich.box.com/shared/static/2ur48aj419slj8cwkeobubhwnvripc0s.t7\n\n" }, { "path": "eval_models.sh", "content": "th scripts/eval_quant_test.lua --gpu 1\n\n" }, { "path": "exp_multiclass.txt", "content": "02691156\n02828884\n02933112\n02958343\n03001627\n03211117\n03636649\n03691459\n04090263\n04256520\n04379243\n04401088\n04530566\n" }, { "path": "exp_singleclass.txt", "content": "03001627\n" }, { "path": "install_ptnbhwd.sh", "content": "mkdir pt_layer\ngit clone https://github.com/xcyan/ptnbhwd.git pt_layer\ncd pt_layer\nluarocks make ptnbhwd-scm-1.rockspec\ncd ..\n" }, { "path": "prepare_data.sh", "content": "mkdir ./data\nwget -O data/all_ids.tar.gz https://umich.box.com/shared/static/4m1mr6aud793gwi7jn266t32w83ml25l.gz\nwget -O data/all_viewdata.tar.gz https://umich.box.com/shared/static/ckvihxh4berjzcgd3s8aiu87aurv3gms.gz\nwget -O data/all_voxdata.tar.gz https://umich.box.com/shared/static/bwyx8qsby2f38ju1ybcrp50a1q9uzenu.gz\ntar xf data/all_ids.tar.gz -C data/\necho \"It may take a while, please be patient.\"\ntar xf data/all_viewdata.tar.gz -C data/\ntar xf data/all_voxdata.tar.gz -C data/\n" }, { "path": "scripts/arch_PTN.lua", "content": "local PTN = {}\n\nfunction PTN.create(opt)\n local encoder = PTN.create_encoder(opt)\n local voxel_dec = PTN.create_voxel_dec(opt)\n local projector = PTN.create_projector(opt)\n return encoder, voxel_dec, projector\nend\n\nfunction PTN.create_encoder(opt)\n local encoder = nn.Sequential()\n -- 64 x 64 x 3 --> 32 x 32 x 64\n encoder:add(nn.SpatialConvolution(3, 64, 5, 5, 2, 2, 2, 2))\n encoder:add(nn.ReLU()) \n -- 32 x 32 x 64 --> 16 x 16 x 128\n encoder:add(nn.SpatialConvolution(64, 128, 5, 5, 2, 2, 2, 2))\n encoder:add(nn.ReLU())\n -- 16 x 16 x 128 --> 8 x 8 x 256\n encoder:add(nn.SpatialConvolution(128, 256, 5, 5, 2, 2, 2, 2))\n encoder:add(nn.ReLU())\n -- 8 x 8 x 256 --> 1024\n encoder:add(nn.Reshape(8*8*256))\n encoder:add(nn.Linear(8*8*256, 1024))\n encoder:add(nn.ReLU())\n -- 1024 --> 1024\n encoder:add(nn.Linear(1024, 1024))\n encoder:add(nn.ReLU())\n\n -- identity unit\n local eid = nn.Sequential()\n eid:add(nn.Linear(1024, opt.nz))\n eid:add(nn.ReLU())\n \n -- viewpoint unit\n local erot = nn.Sequential()\n erot:add(nn.Linear(1024, opt.nz))\n erot:add(nn.ReLU())\n\n encoder:add(nn.ConcatTable():add(eid):add(erot))\n return encoder\nend\n\nfunction PTN.create_voxel_dec(opt)\n local voxel_dec = nn.Sequential()\n \n voxel_dec:add(nn.Linear(opt.nz, 3*3*3*512))\n voxel_dec:add(nn.ReLU())\n voxel_dec:add(nn.Reshape(512, 3, 3, 3))\n\n -- 512 x 3 x 3 x 3 --> 256 x 6 x 6 x 6\n voxel_dec:add(nn.VolumetricFullConvolution(512, 256, 4, 4, 4, 1, 1, 1, 0, 0, 0))\n voxel_dec:add(nn.ReLU())\n -- 256 x 6 x 6 x 6 --> 96 x 15 x 15 x 15\n voxel_dec:add(nn.VolumetricFullConvolution(256, 96, 5, 5, 5, 2, 2, 2, 0, 0, 0))\n voxel_dec:add(nn.ReLU())\n -- 96 x 15 x 15 x 15 --> 1 x 32 x 32 x 32\n voxel_dec:add(nn.VolumetricFullConvolution(96, 1, 6, 6, 6, 2, 2, 2, 1, 1, 1))\n voxel_dec:add(nn.Sigmoid())\n\n return voxel_dec\nend\n\nfunction PTN.create_projector(opt)\n local grid_stream = nn.PerspectiveGridGenerator(opt.vox_size, opt.vox_size, opt.vox_size, opt.focal_length)\n local input_stream = nn.Transpose({2,4},{4,5})\n local projector = nn.Sequential()\n projector:add(nn.ParallelTable():add(input_stream):add(grid_stream))\n projector:add(nn.BilinearSamplerPerspective(opt.focal_length))\n projector:add(nn.Transpose({4,5}, {2,4}))\n -- B x c x Dim1 x Dim2 x Dim3\n projector:add(nn.Max(4)) \n return projector\nend\n\nreturn PTN\n" }, { "path": "scripts/arch_rotatorRNN.lua", "content": "local rotatorRNN = {}\n\nfunction rotatorRNN.create(opt)\n local encoder = rotatorRNN.create_encoder(opt)\n local actor = rotatorRNN.create_actor(opt)\n local mixer = rotatorRNN.create_mixer(opt)\n local decoder_msk = rotatorRNN.create_decoder_msk(opt)\n local decoder_im = rotatorRNN.create_decoder_im(opt)\n return encoder, actor, mixer, decoder_msk, decoder_im\nend\n\nfunction rotatorRNN.create_encoder(opt)\n local encoder = nn.Sequential()\n -- 64 x 64 x 3 --> 32 x 32 x 64\n encoder:add(nn.SpatialConvolution(3, 64, 5, 5, 2, 2, 2, 2))\n encoder:add(nn.ReLU())\n\n -- 32 x 32 x 64 --> 16 x 16 x 128\n encoder:add(nn.SpatialConvolution(64, 128, 5, 5, 2, 2, 2, 2))\n encoder:add(nn.ReLU())\n \n -- 16 x 16 x 128 --> 8 x 8 x 256\n encoder:add(nn.SpatialConvolution(128, 256, 5, 5, 2, 2, 2, 2))\n encoder:add(nn.ReLU())\n \n -- 8 x 8 x 256 --> 1024\n encoder:add(nn.Reshape(8*8*256))\n encoder:add(nn.Linear(8*8*256, 1024))\n encoder:add(nn.ReLU())\n\n -- 1024 --> 1024\n encoder:add(nn.Linear(1024, 1024))\n encoder:add(nn.ReLU())\n\n -- identity unit\n local eid = nn.Sequential()\n eid:add(nn.Linear(1024, opt.nz))\n eid:add(nn.ReLU())\n\n -- viewpoint unit\n local erot = nn.Sequential()\n erot:add(nn.Linear(1024, opt.nz))\n erot:add(nn.ReLU())\n\n encoder:add(nn.ConcatTable():add(eid):add(erot))\n return encoder\nend\n\nfunction rotatorRNN.create_actor(opt)\n -- h1, a --> h2\n local actor = nn.Sequential()\n actor:add(nn.Bilinear(opt.nz, opt.na, opt.nz))\n actor:add(nn.ReLU())\n return actor\nend\n\nfunction rotatorRNN.create_mixer(opt)\n \n local mixer = nn.Sequential()\n mixer:add(nn.JoinTable(2))\n \n mixer:add(nn.Linear(opt.nz*2, 1024))\n mixer:add(nn.ReLU())\n \n mixer:add(nn.Linear(1024, 1024))\n mixer:add(nn.ReLU())\n return mixer\nend\n\nfunction rotatorRNN.create_decoder_msk(opt)\n local decoderM = nn.Sequential()\n -- 1024 --> 8 x 8 x 128\n decoderM:add(nn.Linear(1024, 8*8*128))\n decoderM:add(nn.ReLU())\n\n decoderM:add(nn.Reshape(128, 8, 8))\n -- 8 x 8 x 128 --> 16 x 16 x 64\n decoderM:add(nn.SpatialUpSamplingNearest(2))\n decoderM:add(nn.SpatialConvolution(128, 64, 5, 5, 1, 1, 2, 2))\n decoderM:add(nn.ReLU())\n \n -- 16 x 16 x 64 --> 32 x 32 x 32\n decoderM:add(nn.SpatialUpSamplingNearest(2))\n decoderM:add(nn.SpatialConvolution(64, 32, 5, 5, 1, 1, 2, 2))\n decoderM:add(nn.ReLU())\n \n -- 32 x 32 x 32 --> 64 x 64 x 1\n decoderM:add(nn.SpatialUpSamplingNearest(2))\n decoderM:add(nn.SpatialConvolution(32, 1, 5, 5, 1, 1, 2, 2))\n decoderM:add(nn.Sigmoid())\n\n return decoderM\nend\n\nfunction rotatorRNN.create_decoder_im(opt)\n local decoderI = nn.Sequential()\n -- 1024 --> 8 x 8 x 256\n decoderI:add(nn.Linear(1024, 8*8*256))\n decoderI:add(nn.ReLU())\n decoderI:add(nn.Reshape(256, 8, 8)) \n\n -- 8 x 8 x 256 --> 16 x 16 x 128\n decoderI:add(nn.SpatialUpSamplingNearest(2))\n decoderI:add(nn.SpatialConvolution(256, 128, 5, 5, 1, 1, 2, 2))\n decoderI:add(nn.ReLU())\n\n -- 16 x 16 x 128 --> 32 x 32 x 64\n decoderI:add(nn.SpatialUpSamplingNearest(2))\n decoderI:add(nn.SpatialConvolution(128, 64, 5, 5, 1, 1, 2, 2))\n decoderI:add(nn.ReLU())\n\n -- 32 x 32 x 64 --> 64 x 64 x 3\n decoderI:add(nn.SpatialUpSamplingNearest(2))\n decoderI:add(nn.SpatialConvolution(64, 3, 5, 5, 1, 1, 2, 2))\n decoderI:add(nn.Tanh())\n\n return decoderI\nend\n\nreturn rotatorRNN\n" }, { "path": "scripts/eval_quant_test.lua", "content": "require 'torch'\nrequire 'nn'\nrequire 'cunn'\nrequire 'cudnn'\nrequire 'ptn'\nrequire 'nngraph'\nrequire 'optim'\nrequire 'image'\nrequire 'mattorch'\n\nmodel_utils = require 'utils.model_utils'\noptim_utils = require 'utils.adam_v2'\n\nopt = lapp[[\n --save_every (default 20)\n --print_every (default 1)\n --data_root (default 'data')\n --data_id_path (default 'data/shapenetcore_ids')\n --data_view_path (default 'data/shapenetcore_viewdata')\n --data_vox_path (default 'data/shapenetcore_voxdata')\n --dataset (default 'dataset_ptn')\n --gpu (default 0)\n --use_cudnn (default 1)\n --nz (default 512)\n --na (default 3)\n --nview (default 24)\n --nThreads (default 1)\n --niter (default 100)\n --display (default 1)\n --checkpoint_dir (default 'models/')\n --kstep (default 24)\n --batch_size (default 6)\n --adam (default 1)\n --arch_name (default 'arch_PTN')\n --weight_decay (default 0.001)\n --exp_list (default 'singleclass')\n --load_size (default 64)\n --vox_size (default 32)\n --thresh (default 0.5)\n]]\n\nopt.focal_length = math.sqrt(3)/2\nopt.ntrain = math.huge\nfor k,v in pairs(opt) do opt[k] = tonumber(os.getenv(k)) or os.getenv(k) or opt[k] end\nprint(opt)\nif opt.display == 0 then opt.display = false end\n\nif opt.gpu > 0 then\n ok, cunn = pcall(require, 'cunn')\n ok2, cutorch = pcall(require, 'cutorch')\n cutorch.setDevice(opt.gpu)\nend\n\nopt.manualSeed = torch.random(1, 10000) -- fix seed\nprint(\"Random Seed: \" .. opt.manualSeed)\ntorch.manualSeed(opt.manualSeed)\ntorch.setnumthreads(1)\ntorch.setdefaulttensortype('torch.FloatTensor')\n\n-- create data loader\nlocal TestLoader = require('utils/data_test.lua')\n\nbase_loader = torch.load(opt.checkpoint_dir .. 'cnn_vol.t7')\nencoder = base_loader.encoder\nbase_voxel_dec = base_loader.voxel_dec\n\nunsup_loader = torch.load(opt.checkpoint_dir .. 'ptn_proj.t7')\nunsup_voxel_dec = unsup_loader.voxel_dec\n\nsup_loader = torch.load(opt.checkpoint_dir .. 'ptn_comb.t7')\nsup_voxel_dec = sup_loader.voxel_dec\n\ncollectgarbage()\n\nlocal criterion_vox = nn.MSECriterion()\ncriterion_vox.sizeAverage = false\n\n----------------------------------------------\nlocal batch_im_in = torch.Tensor(opt.batch_size * opt.kstep, 3, opt.load_size, opt.load_size)\nlocal batch_vox = torch.Tensor(opt.batch_size * opt.kstep, 1, opt.vox_size, opt.vox_size, opt.vox_size)\n\nlocal epoch_tm = torch.Timer()\nlocal tm = torch.Timer()\nlocal data_tm = torch.Timer()\n\nif opt.gpu > 0 then\n batch_im_in = batch_im_in:cuda()\n batch_vox = batch_vox:cuda()\n encoder:cuda()\n base_voxel_dec:cuda()\n unsup_voxel_dec:cuda()\n sup_voxel_dec:cuda()\n criterion_vox:cuda()\nend\n\nparamEnc, gradEnc = encoder:getParameters()\nbase_params, base_grads = base_voxel_dec:getParameters()\nunsup_params, unsup_grads = unsup_voxel_dec:getParameters()\nsup_params, sup_grads = sup_voxel_dec:getParameters()\n\nencoder:evaluate()\nbase_voxel_dec:evaluate()\nsup_voxel_dec:evaluate()\nunsup_voxel_dec:evaluate()\n\n--LIST = {'airplane', 'bench', 'dresser', 'car', 'chair', 'display', 'lamp', 'loudspeaker', 'rifle', 'sofa', 'table', 'telephone', 'vessel'}\nLIST = {'chair'}\n\nfor category_idx = 1, #LIST do\n -- load data\n opt.eval_list = LIST[category_idx]\n local data = TestLoader.new(opt.nThreads, opt.dataset, opt)\n\n local base_iouVOX = 0\n local sup_iouVOX = 0\n local unsup_iouVOX = 0\n \n for i = 1, data:size() / opt.batch_size do\n xlua.progress(i, math.floor(data:size() / opt.batch_size))\n\n tm:reset()\n \n base_grads:zero()\n unsup_grads:zero()\n sup_grads:zero()\n gradEnc:zero()\n --\n data_tm:reset(); data_tm:resume()\n cur_ims, cur_vox, _ = data:getBatch()\n data_tm:stop()\n\n for m = 1, opt.batch_size do\n for k = 1, opt.kstep do\n local rng_rot = math.random(2)\n local delta \n if rng_rot == 1 then\n delta = -1 \n elseif rng_rot == 2 then\n delta = 1\n end\n batch_im_in[(m-1)*opt.kstep+k]:copy(cur_ims[m][k]:mul(2):add(-1)) \n batch_vox[(m-1)*opt.kstep+k]:copy(cur_vox[m])\n end\n end\n \n local f_id = encoder:forward(batch_im_in)[1]:clone()\n local f_base_vox = base_voxel_dec:forward(f_id)\n local f_unsup_vox = unsup_voxel_dec:forward(f_id)\n local f_sup_vox = sup_voxel_dec:forward(f_id)\n\n local base_fg_thresh = torch.gt(f_base_vox, opt.thresh):double()\n local base_area_intersc = torch.cmul(base_fg_thresh, batch_vox:double())\n local base_area_union = (base_fg_thresh+batch_vox:double()):gt(0.9)\n \n local sup_fg_thresh = torch.gt(f_sup_vox, opt.thresh):double()\n local sup_area_intersc = torch.cmul(sup_fg_thresh, batch_vox:double())\n local sup_area_union = (sup_fg_thresh+batch_vox:double()):gt(0.9)\n\n local unsup_fg_thresh = torch.gt(f_unsup_vox, opt.thresh):double()\n local unsup_area_intersc = torch.cmul(unsup_fg_thresh, batch_vox:double())\n local unsup_area_union = (unsup_fg_thresh+batch_vox:double()):gt(0.9)\n\n for m = 1, opt.batch_size do\n for k = 1, opt.kstep do\n local base_curIOU = base_area_intersc[(m-1)*opt.kstep+k]:sum() / base_area_union[(m-1)*opt.kstep+k]:sum()\n local sup_curIOU = sup_area_intersc[(m-1)*opt.kstep+k]:sum() / sup_area_union[(m-1)*opt.kstep+k]:sum()\n local unsup_curIOU = unsup_area_intersc[(m-1)*opt.kstep+k]:sum() / unsup_area_union[(m-1)*opt.kstep+k]:sum()\n\n base_iouVOX = base_iouVOX + base_curIOU\n sup_iouVOX = sup_iouVOX + sup_curIOU\n unsup_iouVOX = unsup_iouVOX + unsup_curIOU\n\n --print(string.format('[%d, %d]: %.4f', m, k, unsup_curIOU))\n end\n end\n end\n\n local dataSize = math.floor(data:size() / opt.batch_size) * opt.batch_size\n base_iouVOX = base_iouVOX / (dataSize * opt.kstep)\n sup_iouVOX = sup_iouVOX / (dataSize * opt.kstep)\n unsup_iouVOX = unsup_iouVOX / (dataSize * opt.kstep)\n print(string.format('cat [%s]:\\tCNN-VOL IOU = %g\\tPTN-COMB IOU = %g\\tPTN-PROJ IOU = %g', LIST[category_idx], base_iouVOX, sup_iouVOX, unsup_iouVOX))\nend\n\n--------------------------------------------------\n\n\n" }, { "path": "scripts/train_PTN.lua", "content": "require 'torch'\nrequire 'nn'\nrequire 'cunn'\n--require 'cudnn'\nrequire 'ptn'\nrequire 'nngraph'\nrequire 'optim'\nrequire 'image'\nrequire 'mattorch'\n\nmodel_utils = require 'utils.model_utils'\noptim_utils = require 'utils.adam_v2'\n\nopt = lapp[[\n --save_every (default 20)\n --print_every (default 1)\n --data_root (default 'data')\n --data_id_path (default 'data/shapenetcore_ids')\n --data_view_path (default 'data/shapenetcore_viewdata')\n --data_vox_path (default 'data/shapenetcore_voxdata')\n --dataset (default 'dataset_ptn')\n --gpu (default 0)\n --use_cudnn (default 1)\n --nz (default 512)\n --na (default 3)\n --nview (default 24)\n --nThreads (default 4)\n --niter (default 100)\n --display (default 1)\n --checkpoint_dir (default 'models/')\n --lambda_msk (default 1)\n --lambda_vox (default 0)\n --kstep (default 24)\n --batch_size (default 6)\n --adam (default 1)\n --arch_name (default 'arch_PTN')\n --weight_decay (default 0.001)\n --exp_list (default 'singleclass')\n --load_size (default 64)\n --vox_size (default 32)\n]]\n\nopt.focal_length = math.sqrt(3)/2\nopt.ntrain = math.huge\nfor k,v in pairs(opt) do opt[k] = tonumber(os.getenv(k)) or os.getenv(k) or opt[k] end\nprint(opt)\nif opt.display == 0 then opt.display = false end\n\nif opt.gpu > 0 then\n ok, cunn = pcall(require, 'cunn')\n ok2, cutorch = pcall(require, 'cutorch')\n cutorch.setDevice(opt.gpu)\nend\n\nopt.manualSeed = torch.random(1, 10000) -- fix seed\nprint(\"Random Seed: \" .. opt.manualSeed)\ntorch.manualSeed(opt.manualSeed)\ntorch.setnumthreads(1)\ntorch.setdefaulttensortype('torch.FloatTensor')\n\n-- create data loader\nlocal TrainLoader = require('utils/data.lua')\nlocal ValLoader = require('utils/data_val.lua')\n\nlocal data = TrainLoader.new(opt.nThreads, opt.dataset, opt)\nlocal data_val = ValLoader.new(opt.nThreads, opt.dataset, opt)\n\nprint(\"Dataset: \" .. opt.dataset, \"train_size: \", data:size(), \"val_size: \", data_val:size())\n\nlocal function weights_init(m)\n local name = torch.type(m)\n if name:find('Convolution') and name:find('Spatial') then\n local nin = m.nInputPlane*m.kH*m.kW\n m.weight:uniform(-0.08, 0.08):mul(math.sqrt(1/nin))\n m.bias:fill(0)\n elseif name:find('Convolution') and name:find('Volumeric') then\n local nin = m.nInputPlane*m.kT*m.kH*m.kW\n m.weight:uniform(-0.08, 0.08):mul(math.sqrt(1/nin))\n m.bias:fill(0)\n elseif name:find('Linear') then\n local nin = m.weight:size(2)\n m.weight:uniform(-0.08, 0.08):mul(math.sqrt(1/nin))\n m.bias:fill(0)\n elseif name:find('BatchNormalization') then\n if m.weight then m.weight:normal(1.0, 0.02) end\n if m.bias then m.bias:fill(0) end\n end\nend\n\nopt.model_name = string.format('%s_%s_nv%d_adam%d_bs%d_nz%d_wd%g_lbg(%g,%g)_ks%d_vs%d', \n opt.arch_name, opt.exp_list, opt.nview, opt.adam, opt.batch_size, opt.nz,\n opt.weight_decay, opt.lambda_msk, opt.lambda_vox, opt.kstep, opt.vox_size)\n\n-- initialize parameters\ninit_models = dofile('scripts/' .. opt.arch_name .. '.lua')\nencoder, voxel_dec, projector = init_models.create(opt)\nencoder:apply(weights_init)\nvoxel_dec:apply(weights_init)\nprojector:apply(weights_init)\n\nopt.model_path = opt.checkpoint_dir .. opt.model_name\nif not paths.dirp(opt.model_path) then\n paths.mkdir(opt.model_path)\nend\n\n-- load encoder from RNN-16\nif opt.exp_list == 'singleclass' then\n opt.basemodel_name = string.format('%s_%s_nv%d_adam%d_bs%d_nz%d_wd%g_lbg%g_ks%d',\n 'arch_rotatorRNN', opt.exp_list, opt.nview, 2, 8, opt.nz, \n opt.weight_decay, 10, 16)\n opt.basemodel_epoch = 20 \n--[[elseif opt.exp_list == 'multiclass' then\n opt.basemodel_name = string.format('%s_%s_nv%d_adam%d_bs%d_nz%d_wd%g_lbg%g_ks%d',\n 'rotatorRNN1_64', opt.exp_list, opt.nview, 2, 8, opt.nz, \n opt.weight_decay, 10, 16) \n opt.basemodel_epoch = 20]]\n loader = torch.load(opt.checkpoint_dir .. opt.basemodel_name .. string.format('/net-epoch-%d.t7', opt.basemodel_epoch))\n encoder = loader.encoder\nend\n\ncollectgarbage()\n\n-- load model from previos iterations\nprev_iter = 0\nfor i = opt.niter, 1, -opt.save_every do\n print(opt.model_path .. string.format('/net-epoch-%d.t7', i))\n if paths.filep(opt.model_path .. string.format('/net-epoch-%d.t7', i)) then\n prev_iter = i\n loader = torch.load(opt.model_path .. string.format('/net-epoch-%d.t7', i))\n state = torch.load(opt.model_path .. '/state.t7')\n print(string.format('resuming from epoch %d', i))\n break\n end\nend\n\n-- build nngraph\nif prev_iter > 0 then\n encoder = loader.encoder\n voxel_dec = loader.voxel_dec\n projector = loader.projector\nend\n\n-- criterion\nlocal criterion_vox = nn.MSECriterion()\ncriterion_vox.sizeAverage = false\nlocal criterion_msk = nn.MSECriterion()\ncriterion_msk.sizeAverage = false\n\n-- hyperparams\nfunction getAdamParams(opt)\n config = {}\n if opt.adam == 1 then\n config.learningRate = 0.0001\n config.epsilon = 1e-8\n config.beta1 = 0.9\n config.beta2 = 0.999\n config.weightDecay = opt.weight_decay\n elseif opt.adam == 2 then\n config.learningRate = 0.1\n config.epsilon = 1e-8\n config.beta1 = 0.5\n config.beta2 = 0.999\n config.weightDecay = opt.weight_decay\n end\n return config\nend\n\nconfig = getAdamParams(opt)\nprint(config)\n-------------------------------------------\n\nlocal batch_im_in = torch.Tensor(opt.batch_size, 3, opt.load_size, opt.load_size)\nlocal batch_feat = torch.Tensor(opt.batch_size * opt.kstep, opt.nz)\nlocal batch_vox = torch.Tensor(opt.batch_size * opt.kstep, 1, opt.vox_size, opt.vox_size, opt.vox_size)\nlocal batch_proj = torch.Tensor(opt.batch_size * opt.kstep, 1, opt.vox_size, opt.vox_size)\nlocal batch_trans = torch.Tensor(opt.batch_size * opt.kstep, 4, 4)\n\nlocal tmp_gt_im = torch.Tensor(opt.batch_size, 3, opt.load_size, opt.load_size)\nlocal tmp_pred_proj = torch.Tensor(opt.batch_size, 1, opt.vox_size, opt.vox_size)\nlocal tmp_gt_proj = torch.Tensor(opt.batch_size, 1, opt.vox_size, opt.vox_size)\n\nlocal errVOX, errMSK\nlocal epoch_tm = torch.Timer()\nlocal tm = torch.Timer()\nlocal data_tm = torch.Timer()\n--------------------------------------------\n\nif opt.gpu > 0 then\n batch_im_in = batch_im_in:cuda()\n batch_feat = batch_feat:cuda()\n batch_vox = batch_vox:cuda()\n batch_proj = batch_proj:cuda()\n batch_trans = batch_trans:cuda()\n encoder:cuda()\n voxel_dec:cuda()\n projector:cuda()\n criterion_vox:cuda()\n criterion_msk:cuda()\nend\n\nparams, grads = voxel_dec:getParameters()\nparamEnc, gradEnc = encoder:getParameters()\nparamProj, gradProj = projector:getParameters()\n\n-- perspective projection\n--------------------------------------------------\nlocal specify_pers_transformation = function(phi, theta, focal_length)\n local T = torch.Tensor(4, 4):zero()\n local K = torch.Tensor(4, 4):eye(4)\n local E = torch.Tensor(4, 4):eye(4)\n\n local sin_phi = math.sin(phi*math.pi/180.0)\n local cos_phi = math.cos(phi*math.pi/180.0)\n local sin_theta = math.sin((-theta)*math.pi/180.0)\n local cos_theta = math.cos((-theta)*math.pi/180.0)\n -- rotation axis -- z\n R_azi = torch.Tensor(3, 3):zero()\n R_azi[1][1] = cos_theta\n R_azi[3][3] = cos_theta\n R_azi[1][3] = -sin_theta\n R_azi[3][1] = sin_theta\n R_azi[2][2] = 1\n\n -- rotation axis -- x\n R_ele = torch.Tensor(3, 3):zero()\n R_ele[1][1] = cos_phi\n R_ele[1][2] = sin_phi\n R_ele[2][1] = -sin_phi\n R_ele[2][2] = cos_phi\n R_ele[3][3] = 1\n R_comb = R_azi * R_ele\n\n local colR = torch.Tensor(3,1):zero()\n --local focal_length = math.sqrt(3)/2\n colR[1][1] = opt.focal_length + math.sqrt(1)/2\n colR = R_comb * colR\n E[{{1,3}, {1,3}}] = R_comb:clone()\n E[{{1,3}, {4}}] = -colR:clone()\n \n K[3][3] = 1/opt.focal_length\n K[2][2] = 1/opt.focal_length\n T = E * K\n\n return T\nend\n\nlocal getTransMatrix = function(vid)\n local T = specify_pers_transformation(30, vid*15, opt.focal_length)\n return T\nend\n--------------------------------------------------\nlocal opfunc = function(x)\n collectgarbage()\n if x ~= params then\n params:copy(x)\n end\n grads:zero()\n\n -- train\n data_tm:reset(); data_tm:resume()\n cur_train_ims, cur_train_vox, _ = data:getBatch()\n data_tm:stop()\n\n for m = 1, opt.batch_size do\n \n local view_in = torch.random(opt.nview)\n local rng_rot = math.random(2)\n local delta\n if rng_rot == 1 then\n delta = -1\n elseif rng_rot == 2 then\n delta = 1\n end\n \n batch_im_in[m]:copy(cur_train_ims[m][view_in]:mul(2):add(-1))\n\n local view_out = view_in\n for k = 1, opt.kstep do\n view_out = view_out + delta\n if view_out > opt.nview then view_out = 1 end\n if view_out < 1 then view_out = opt.nview end\n batch_vox[(m-1)*opt.kstep+k]:copy(cur_train_vox[m])\n batch_trans[(m-1)*opt.kstep+k]:copy(getTransMatrix(view_out))\n end\n end\n\n gradEnc:zero()\n local f_id = encoder:forward(batch_im_in)[1]:clone()\n for m = 1, opt.batch_size do\n for k = 1, opt.kstep do\n batch_feat[(m-1)*opt.kstep+k]:copy(f_id[m])\n end\n end\n\n gradProj:zero() \n batch_proj = projector:forward({batch_vox, batch_trans}):clone()\n\n local f_vox = voxel_dec:forward(batch_feat)\n local f_proj = projector:forward({f_vox, batch_trans})\n \n errVOX = criterion_vox:forward(f_vox, batch_vox) / (2 * opt.batch_size * opt.kstep)\n local df_dVOX = criterion_vox:backward(f_vox, batch_vox):mul(opt.lambda_vox):div(2 * opt.batch_size * opt.kstep)\n \n errMSK = criterion_msk:forward(f_proj, batch_proj) / (2 * opt.batch_size * opt.kstep)\n local df_dMSK = criterion_msk:backward(f_proj, batch_proj):mul(opt.lambda_msk):div(2 * opt.batch_size * opt.kstep)\n\n local df_dproj = projector:backward({f_vox, batch_trans}, df_dMSK)\n local df_dvox = voxel_dec:backward(batch_feat, df_dproj[1]:clone() + df_dVOX:clone())\n\n local err = errVOX * opt.lambda_vox + errMSK * opt.lambda_msk\n\n return err, grads\nend\n--------------------------------------------------------\n\nlocal feedforward = function(x)\n collectgarbage()\n if x ~= params then\n params:copy(x)\n end\n grads:zero()\n\n -- val\n data_tm:reset(); data_tm:resume()\n cur_ims, cur_vox, _ = data_val:getBatch()\n data_tm:stop()\n\n for m = 1, opt.batch_size do\n local view_in = torch.random(opt.nview) \n local rng_rot = math.random(2)\n local delta \n if rng_rot == 1 then\n delta = -1\n elseif rng_rot == 2 then\n delta = 1\n end\n\n batch_im_in[m]:copy(cur_ims[m][view_in]:mul(2):add(-1))\n\n local view_out = view_in\n for k = 1, opt.kstep do\n view_out = view_out + delta\n if view_out > opt.nview then view_out = 1 end\n if view_out < 1 then view_out = opt.nview end\n batch_trans[(m-1)*opt.kstep+k]:copy(getTransMatrix(view_out))\n batch_vox[(m-1)*opt.kstep+k]:copy(cur_vox[m])\n end\n end\n\n gradEnc:zero()\n local f_id = encoder:forward(batch_im_in)[1]:clone()\n for m = 1, opt.batch_size do\n for k = 1, opt.kstep do\n batch_feat[(m-1)*opt.kstep+k]:copy(f_id[m])\n end\n end\n\n gradProj:zero()\n batch_proj = projector:forward({batch_vox, batch_trans}):clone()\n\n local f_vox = voxel_dec:forward(batch_feat)\n local f_proj = projector:forward({f_vox, batch_trans}):clone()\n\n errVOX = criterion_vox:forward(f_vox, batch_vox) / (2 * opt.batch_size * opt.kstep)\n errMSK = criterion_msk:forward(f_proj, batch_proj) / (2 * opt.batch_size * opt.kstep)\n \n for m = 1, opt.batch_size do\n k = torch.random(opt.kstep)\n tmp_gt_im[m] = batch_im_in[m]:float():clone()\n tmp_pred_proj[m] = f_proj[(m-1)*opt.kstep+k]:float():clone()\n tmp_gt_proj[m] = batch_proj[(m-1)*opt.kstep+k]:float():clone()\n end\n\n local err = errVOX * opt.lambda_vox + errMSK * opt.lambda_msk\nend\n------------------------------------------------------------\n\nencoder:evaluate()\n-- train & val\nfor epoch = prev_iter + 1, opt.niter do\n epoch_tm:reset()\n local counter = 0\n -- train\n voxel_dec:training()\n projector:training()\n\n for i = 1, math.min(data:size() / (opt.batch_size), opt.ntrain) do\n tm:reset()\n optim_utils.adam_v2(opfunc, params, config, state)\n counter = counter + 1\n print(string.format('Epoch: [%d][%8d / %8d]\\t Time: %.3f DataTime: %.3f '\n .. ' Err_Vox: %.4f, Err_Msk: %.4f', epoch, i-1,\n math.min(data:size() / (opt.batch_size), opt.ntrain),\n tm:time().real, data_tm:time().real,\n errVOX and errVOX or -1, errMSK and errMSK or -1))\n end\n\n -- val\n voxel_dec:evaluate()\n projector:evaluate()\n\n --for i = 1, 1 do\n tm:reset()\n local to_plot = {}\n\n for i = 1, 24 / opt.batch_size do\n local err = feedforward(params)\n\n for j = 1, opt.batch_size do\n local res = tmp_gt_im[j]:float():clone()\n res = torch.squeeze(res)\n res:add(1):mul(0.5)\n to_plot[#to_plot+1] = res:clone()\n\n local res = tmp_pred_proj[j]:float():clone()\n res = torch.squeeze(res)\n res = res:repeatTensor(3, 1, 1)\n res = image.vflip(res)\n res = image.scale(res, opt.load_size, opt.load_size)\n res:mul(-1):add(1)\n to_plot[#to_plot+1] = res:clone()\n\n local res = tmp_gt_proj[j]:float():clone()\n res = torch.squeeze(res)\n res = res:repeatTensor(3, 1, 1)\n res = image.vflip(res)\n res = image.scale(res, opt.load_size, opt.load_size)\n res:mul(-1):add(1)\n to_plot[#to_plot+1] = res:clone()\n end\n end\n\n local formatted = image.toDisplayTensor({input=to_plot, nrow = 12})\n formatted = formatted:double()\n formatted:mul(255)\n\n formatted = formatted:byte()\n image.save(opt.model_path .. string.format('/sample-%03d.jpg', epoch), formatted)\n\n if epoch % opt.save_every == 0 then\n torch.save((opt.model_path .. string.format('/net-epoch-%d.t7', epoch)), \n {encoder = encoder, voxel_dec = voxel_dec, projector = projector})\n torch.save((opt.model_path .. '/state.t7'), state)\n end\nend\n" }, { "path": "scripts/train_rotatorRNN_base.lua", "content": "-- torch reimplementation of deepRotator: https://github.com/jimeiyang/deepRotator.git\nrequire 'torch'\nrequire 'nn'\nrequire 'cunn'\n--require 'cudnn'\nrequire 'nngraph'\nrequire 'optim'\nrequire 'image'\n\nmodel_utils = require 'utils.model_utils'\noptim_utils = require 'utils.adam_v2'\n\nopt = lapp[[\n --save_every (default 40)\n --print_every (default 1)\n --data_root (default 'data')\n --data_id_path (default 'data/shapenetcore_ids')\n --data_view_path (default 'data/shapenetcore_viewdata')\n --dataset (default 'dataset_rotatorRNN_base')\n --gpu (default 0)\n --nz (default 512)\n --na (default 3)\n --nview (default 24)\n --nThreads (default 4)\n --niter (default 160)\n --display (default 1)\n --checkpoint_dir (default 'models/')\n --lambda (default 10)\n --kstep (default 1)\n --batch_size (default 32)\n --adam (default 1)\n --arch_name (default 'arch_rotatorRNN')\n --weight_decay (default 0.001)\n --exp_list (default 'singleclass')\n --load_size (default 64)\n]]\n\nopt.ntrain = math.huge\nfor k,v in pairs(opt) do opt[k] = tonumber(os.getenv(k)) or os.getenv(k) or opt[k] end\nprint(opt)\nif opt.display == 0 then opt.display = false end\n\nif opt.gpu > 0 then \n ok, cunn = pcall(require, 'cunn')\n ok2, cutorch = pcall(require, 'cutorch')\n cutorch.setDevice(opt.gpu)\nend\n\nopt.manualSeed = torch.random(1, 10000) -- fix seed\nprint(\"Random Seed: \" .. opt.manualSeed)\ntorch.manualSeed(opt.manualSeed)\ntorch.setnumthreads(1)\ntorch.setdefaulttensortype('torch.FloatTensor')\n\n-- create data loader\nlocal TrainLoader = require 'utils/data.lua'\nlocal ValLoader = require 'utils/data_val.lua'\nlocal data = TrainLoader.new(opt.nThreads, opt.dataset, opt)\nlocal data_val = ValLoader.new(opt.nThreads, opt.dataset, opt)\n\nprint(\"dataset: \" .. opt.dataset, \"train size: \", data:size(), \"val size: \", data_val:size())\n----------------------------------------------------------------\nlocal function weights_init(m)\n local name = torch.type(m)\n if name:find('Convolution') and name:find('Spatial') then\n local nin = m.nInputPlane*m.kH*m.kW\n m.weight:uniform(-0.08, 0.08):mul(math.sqrt(1/nin))\n m.bias:fill(0)\n elseif name:find('Convolution') and name:find('Volumetric') then\n local nin = m.nInputPlane*m.kT*m.kH*m.kW\n m.weight:uniform(-0.08, 0.08):mul(math.sqrt(1/nin))\n m.bias:fill(0)\n elseif name:find('Linear') then\n local nin = m.weight:size(2)\n m.weight:uniform(-0.08, 0.08):mul(math.sqrt(1/nin))\n m.bias:fill(0)\n elseif name:find('BatchNormalization') then\n if m.weight then m.weight:normal(1.0, 0.02) end\n if m.bias then m.bias:fill(0) end\n end\nend\n\nopt.model_name = string.format('%s_%s_nv%d_adam%d_bs%d_nz%d_wd%g_lbg%g_ks%d',\n opt.arch_name, opt.exp_list, opt.nview, opt.adam, opt.batch_size, opt.nz,\n opt.weight_decay, opt.lambda, opt.kstep)\n\n-- initialize parameters\ninit_models = dofile('scripts/' .. opt.arch_name .. '.lua')\nencoder, actor, mixer, decoder_msk, decoder_im = init_models.create(opt)\nencoder:apply(weights_init)\nactor:apply(weights_init)\nmixer:apply(weights_init)\ndecoder_msk:apply(weights_init)\ndecoder_im:apply(weights_init)\n\nopt.model_path = opt.checkpoint_dir .. opt.model_name\nif not paths.dirp(opt.model_path) then\n paths.mkdir(opt.model_path)\nend\n\nprev_iter = 0\n-- load model from previous iterations\nfor i = opt.niter, 1, -opt.save_every do\n print(opt.model_path .. string.format('/net-epoch-%d.t7', i))\n if paths.filep(opt.model_path .. string.format('/net-epoch-%d.t7', i)) then\n prev_iter = i\n loader = torch.load(opt.model_path .. string.format('/net-epoch-%d.t7', i))\n state = torch.load(opt.model_path .. '/state.t7')\n print(string.format('resuming from epoch %d', i))\n break\n end\nend\n\n-- build nngraph\nif prev_iter > 0 then\n encoder = loader.encoder\n actor = loader.actor\n mixer = loader.mixer\n decoder_msk = loader.decoder_msk\n decoder_im = loader.decoder_im\nend\n\n-- criterion\nlocal criterion_im = nn.MSECriterion()\ncriterion_im.sizeAverage = false\nlocal criterion_msk = nn.MSECriterion()\ncriterion_msk.sizeAverage = false\n\n-- hyperparams\nfunction getAdamParams(opt)\n config = {}\n if opt.adam == 1 then\n config.learningRate = 0.0001\n config.epsilon = 1e-8\n config.beta1 = 0.9\n config.beta2 = 0.999\n config.weightDecay = opt.weight_decay\n end\n return config\nend\n\nconfig = getAdamParams(opt)\nprint(config)\n-------------------------------------------------\nlocal batch_im_in = torch.Tensor(opt.batch_size, 3, opt.load_size, opt.load_size)\nlocal batch_rot = torch.Tensor(opt.batch_size, opt.na):zero()\nlocal batch_outputs = {}\nfor k = 1, opt.kstep do\n batch_outputs[2*k-1] = torch.Tensor(opt.batch_size, 3, opt.load_size, opt.load_size)\n batch_outputs[2*k] = torch.Tensor(opt.batch_size, 1, opt.load_size, opt.load_size)\nend\nlocal preds = {}\nfor k = 1, opt.kstep do\n preds[2*k-1] = torch.Tensor(opt.batch_size, 3, opt.load_size, opt.load_size)\n preds[2*k] = torch.Tensor(opt.batch_size, 1, opt.load_size, opt.load_size)\nend\n\nlocal errIM, errMSK\nlocal epoch_tm = torch.Timer()\nlocal tm = torch.Timer()\nlocal data_tm = torch.Timer()\n------------------------------------------------\nif opt.gpu > 0 then\n batch_im_in = batch_im_in:cuda()\n batch_rot = batch_rot:cuda()\n for k = 1, opt.kstep do\n batch_outputs[2*k-1] = batch_outputs[2*k-1]:cuda()\n batch_outputs[2*k] = batch_outputs[2*k]:cuda()\n end\n encoder:cuda()\n actor:cuda()\n mixer:cuda()\n decoder_msk:cuda()\n decoder_im:cuda()\n criterion_im:cuda()\n criterion_msk:cuda()\nend\n\nlocal inputs = {nn.Identity()(), nn.Identity()()}\nlocal h_enc_id, h_enc_rot = encoder(inputs[1]):split(2)\nlocal outputs = {}\nlocal h_dec_rot = actor({h_enc_rot, inputs[2]})\nlocal h_mix = mixer({h_enc_id, h_dec_rot})\nlocal h_dec_msk = decoder_msk(h_mix)\nlocal h_dec_im = decoder_im(h_mix)\ntable.insert(outputs, h_dec_im)\ntable.insert(outputs, h_dec_msk)\n\nrotatorRNN = nn.gModule(inputs, outputs)\nparams, grads = rotatorRNN:getParameters()\n\nlocal opfunc = function(x)\n collectgarbage()\n if x ~= params then\n params:copy(x)\n end\n\n grads:zero()\n\n -- train\n data_tm:reset(); data_tm:resume()\n cur_im_in, cur_outputs, cur_rot, _ = data:getBatch() \n data_tm:stop()\n\n batch_im_in:copy(cur_im_in:mul(2):add(-1))\n for k = 1, opt.kstep do\n batch_outputs[k*2-1]:copy(cur_outputs[k*2-1]:mul(2):add(-1))\n batch_outputs[k*2]:copy(cur_outputs[k*2])\n end\n batch_rot:copy(cur_rot)\n\n local f = rotatorRNN:forward({batch_im_in, batch_rot})\n errIM = 0\n errMSK = 0\n local df_dw = {}\n\n for k = 1, opt.kstep do \n -- fast forward (actor, mixer, decoder)\n errIM = errIM + criterion_im:forward(f[2*k-1], batch_outputs[2*k-1]) / (8 * opt.batch_size)\n errMSK = errMSK + criterion_msk:forward(f[2*k], batch_outputs[2*k]) / (2 * opt.batch_size)\n local df_dIM = criterion_im:backward(f[2*k-1], batch_outputs[2*k-1]):mul(opt.lambda):div(8 * opt.batch_size)\n local df_dMSK = criterion_msk:backward(f[2*k], batch_outputs[2*k]):div(2 * opt.batch_size)\n df_dw[2*k-1] = df_dIM:clone()\n df_dw[2*k] = df_dMSK:clone()\n end\n rotatorRNN:backward({batch_im_in, batch_rot}, df_dw)\n\n local err = errIM * opt.lambda + errMSK\n return err, grads\nend\n-------------------------------------------------\nlocal feedforward = function(x)\n collectgarbage()\n if x ~= params then\n params:copy(x)\n end\n\n grads:zero()\n \n -- val\n data_tm:reset(); data_tm:resume()\n cur_im_in, cur_outputs, cur_rot, _ = data_val:getBatch() \n data_tm:stop()\n\n batch_im_in:copy(cur_im_in:mul(2):add(-1))\n for k = 1, opt.kstep do\n batch_outputs[k*2-1]:copy(cur_outputs[k*2-1]:mul(2):add(-1))\n batch_outputs[k*2]:copy(cur_outputs[k*2])\n end\n batch_rot:copy(cur_rot)\n\n local f = rotatorRNN:forward({batch_im_in, batch_rot})\n errIM = 0\n errMSK = 0\n \n for k = 1, opt.kstep do\n errIM = errIM + criterion_im:forward(f[2*k-1], batch_outputs[2*k-1]) / (8 * opt.batch_size)\n errMSK = errMSK + criterion_msk:forward(f[2*k], batch_outputs[2*k]) / (2 * opt.batch_size)\n preds[2*k-1] = f[2*k-1]:float():clone()\n preds[2*k] = f[2*k]:float():clone()\n end\n\n local err = errIM * opt.lambda + errMSK\n return err\nend\n--------------------------------------------------\n\n-- train & val\nfor epoch = prev_iter + 1, opt.niter do\n epoch_tm:reset()\n local counter = 0\n -- train\n rotatorRNN:training()\n for i = 1, math.min(data:size() * opt.nview / 2 , opt.ntrain), opt.batch_size do\n tm:reset()\n optim_utils.adam_v2(opfunc, params, config, state)\n counter = counter + 1\n \n print(string.format('Epoch: [%d][%8d / %8d]\\t Time: %.3f DataTime: %.3f '\n .. ' Err_Im: %.4f , Err_Msk: %.4f', epoch, ((i-1) / opt.batch_size),\n math.floor(math.min(data:size() * opt.nview / 2, opt.ntrain) / opt.batch_size),\n tm:time().real, data_tm:time().real,\n errIM and errIM or -1, errMSK and errMSK or -1))\n end\n\n -- val\n rotatorRNN:evaluate()\n for i = 1, opt.batch_size do\n tm:reset()\n local err = feedforward(params)\n end\n \n -- plot\n local to_plot = {}\n for i = 1, 32 do\n for k = 1, opt.kstep do\n local res = batch_im_in[i]:float():clone()\n res = torch.squeeze(res)\n res:add(1):mul(0.5)\n to_plot[#to_plot+1] = res:clone()\n\n local res = preds[2*k][i]:float()\n res = torch.squeeze(res)\n res = res:repeatTensor(3, 1, 1)\n res:mul(-1):add(1)\n to_plot[#to_plot+1] = res:clone()\n\n local res = preds[2*k-1][i]:float()\n res = torch.squeeze(res)\n res:add(1):mul(0.5)\n to_plot[#to_plot+1] = res:clone()\n\n local res = batch_outputs[2*k-1][i]:float():clone()\n res = torch.squeeze(res)\n res:add(1):mul(0.5)\n to_plot[#to_plot+1] = res:clone()\n end\n end\n\n local formatted = image.toDisplayTensor({input=to_plot, nrow = 16})\n formatted = formatted:double()\n formatted:mul(255)\n\n formatted = formatted:byte()\n image.save(opt.model_path .. string.format('/sample-%03d.jpg', epoch), formatted)\n\n if epoch % opt.save_every == 0 then\n torch.save((opt.model_path .. string.format('/net-epoch-%d.t7', epoch)),\n {encoder = encoder, actor = actor, mixer = mixer,\n decoder_msk = decoder_msk, decoder_im = decoder_im})\n torch.save((opt.model_path .. '/state.t7'), state)\n end\nend\n" }, { "path": "scripts/train_rotatorRNN_curriculum.lua", "content": "-- torch reimplementation of deepRotator: https://github.com/jimeiyang/deepRotator.git\nrequire 'torch'\nrequire 'nn'\nrequire 'cunn'\n-- require 'cudnn'\nrequire 'nngraph'\nrequire 'optim'\nrequire 'image'\n\nmodel_utils = require 'utils.model_utils'\noptim_utils = require 'utils.adam_v2'\n\nopt = lapp[[\n --save_every (default 20)\n --print_every (default 1)\n --data_root (default 'data')\n --data_id_path (default 'data/shapenetcore_ids')\n --data_view_path (default 'data/shapenetcore_viewdata')\n --dataset (default 'dataset_rotatorRNN_curriculum')\n --gpu (default 0)\n --use_cudnn (default 1)\n --nz (default 512)\n --na (default 3)\n --nview (default 24)\n --nThreads (default 4)\n --niter (default 40)\n --display (default 1)\n --checkpoint_dir (default 'models/')\n --lambda (default 10)\n --kstep (default 2)\n --batch_size (default 32)\n --adam (default 1)\n --arch_name (default 'arch_rotatorRNN')\n --weight_decay (default 0.001)\n --exp_list (default 'singleclass')\n --load_size (default 64)\n]]\n\nopt.ntrain = math.huge\nfor k,v in pairs(opt) do opt[k] = tonumber(os.getenv(k)) or os.getenv(k) or opt[k] end\nprint(opt)\nif opt.display == 0 then opt.display = false end\n\nif opt.gpu > 0 then\n ok, cunn = pcall(require, 'cunn')\n ok2, cutorch = pcall(require, 'cutorch')\n cutorch.setDevice(opt.gpu)\nend\n\nopt.manualSeed = torch.random(1, 10000) -- fix seed\nprint(\"Random Seed: \" .. opt.manualSeed)\ntorch.manualSeed(opt.manualSeed)\ntorch.setnumthreads(1)\ntorch.setdefaulttensortype('torch.FloatTensor')\n\n-- create data loader\nlocal TrainLoader = require('utils/data.lua')\nlocal ValLoader = require('utils/data_val.lua')\nlocal data = TrainLoader.new(opt.nThreads, opt.dataset, opt)\nlocal data_val = ValLoader.new(opt.nThreads, opt.dataset, opt)\nprint(\"Dataset: \" .. opt.dataset, \"train_size: \", data:size(), \"val_size: \", data_val:size())\n\n------------------------------------------------\n\nif opt.exp_list == 'singleclass' then\n if opt.kstep == 2 then\n opt.basemodel_name = string.format('%s_%s_nv%d_adam%d_bs%d_nz%d_wd%g_lbg%g_ks%d',\n opt.arch_name, opt.exp_list, opt.nview, 1, 32, opt.nz, \n opt.weight_decay, opt.lambda, 1)\n opt.basemodel_epoch = 160\n loader = torch.load(opt.checkpoint_dir .. opt.basemodel_name .. string.format('/net-epoch-%d.t7', opt.basemodel_epoch))\n elseif opt.kstep == 4 then\n opt.basemodel_name = string.format('%s_%s_nv%d_adam%d_bs%d_nz%d_wd%g_lbg%g_ks%d',\n opt.arch_name, opt.exp_list, opt.nview, 1, 32, opt.nz, \n opt.weight_decay, opt.lambda, 2)\n opt.basemodel_epoch = 40\n loader = torch.load(opt.checkpoint_dir .. opt.basemodel_name .. string.format('/net-epoch-%d.t7', opt.basemodel_epoch))\n elseif opt.kstep == 8 then\n opt.basemodel_name = string.format('%s_%s_nv%d_adam%d_bs%d_nz%d_wd%g_lbg%g_ks%d',\n opt.arch_name, opt.exp_list, opt.nview, 2, 32, opt.nz, \n opt.weight_decay, opt.lambda, 4)\n opt.basemodel_epoch = 40\n loader = torch.load(opt.checkpoint_dir .. opt.basemodel_name .. string.format('/net-epoch-%d.t7', opt.basemodel_epoch)) \n elseif opt.kstep == 12 then\n opt.basemodel_name = string.format('%s_%s_nv%d_adam%d_bs%d_nz%d_wd%g_lbg%g_ks%d',\n opt.arch_name, opt.exp_list, opt.nview, 2, 32, opt.nz, \n opt.weight_decay, opt.lambda, 8)\n opt.basemodel_epoch = 40\n loader = torch.load(opt.checkpoint_dir .. opt.basemodel_name .. string.format('/net-epoch-%d.t7', opt.basemodel_epoch)) \n elseif opt.kstep == 16 then\n opt.basemodel_name = string.format('%s_%s_nv%d_adam%d_bs%d_nz%d_wd%g_lbg%g_ks%d',\n opt.arch_name, opt.exp_list, opt.nview, 2, 16, opt.nz, \n opt.weight_decay, opt.lambda, 12)\n opt.basemodel_epoch = 40\n loader = torch.load(opt.checkpoint_dir .. opt.basemodel_name .. string.format('/net-epoch-%d.t7', opt.basemodel_epoch)) \n end\n--[[elseif opt.exp_list == 'multiclass' then\n if opt.kstep == 2 then\n opt.basemodel_name = string.format('%s_%s_nv%d_adam%d_bs%d_nz%d_wd%g_lbg%g_ks%d',\n opt.arch_name, opt.exp_list, opt.nview, 1, 32, opt.nz, \n opt.weight_decay, opt.lambda, 1)\n opt.basemodel_epoch = 160\n loader = torch.load(opt.checkpoint_dir .. opt.basemodel_name .. string.format('/net-epoch-%d.t7', opt.basemodel_epoch))\n elseif opt.kstep == 4 then\n opt.basemodel_name = string.format('%s_%s_nv%d_adam%d_bs%d_nz%d_wd%g_lbg%g_ks%d',\n opt.arch_name, opt.exp_list, opt.nview, 1, 32, opt.nz, \n opt.weight_decay, opt.lambda, 2)\n opt.basemodel_epoch = 40\n loader = torch.load(opt.checkpoint_dir .. opt.basemodel_name .. string.format('/net-epoch-%d.t7', opt.basemodel_epoch))\n elseif opt.kstep == 8 then\n opt.basemodel_name = string.format('%s_%s_nv%d_adam%d_bs%d_nz%d_wd%g_lbg%g_ks%d',\n opt.arch_name, opt.exp_list, opt.nview, 2, 8, opt.nz, \n opt.weight_decay, opt.lambda, 4)\n opt.basemodel_epoch = 40\n loader = torch.load(opt.checkpoint_dir .. opt.basemodel_name .. string.format('/net-epoch-%d.t7', opt.basemodel_epoch)) \n elseif opt.kstep == 12 then\n opt.basemodel_name = string.format('%s_%s_nv%d_adam%d_bs%d_nz%d_wd%g_lbg%g_ks%d',\n opt.arch_name, opt.exp_list, opt.nview, 3, 4, opt.nz, \n opt.weight_decay, opt.lambda, 8)\n opt.basemodel_epoch = 40\n loader = torch.load(opt.checkpoint_dir .. opt.basemodel_name .. string.format('/net-epoch-%d.t7', opt.basemodel_epoch)) \n elseif opt.kstep == 16 then\n opt.basemodel_name = string.format('%s_%s_nv%d_adam%d_bs%d_nz%d_wd%g_lbg%g_ks%d',\n opt.arch_name, opt.exp_list, opt.nview, 3, 3, opt.nz,\n opt.weight_decay, opt.lambda, 12)\n opt.basemodel_epoch = 40\n loader = torch.load(opt.checkpoint_dir .. opt.basemodel_name .. string.format('/net-epoch-%d.t7', opt.basemodel_epoch))\n end]]\nend\n\nopt.model_name = string.format('%s_%s_nv%d_adam%d_bs%d_nz%d_wd%g_lbg%g_ks%d',\n opt.arch_name, opt.exp_list, opt.nview, opt.adam, opt.batch_size, opt.nz,\n opt.weight_decay, opt.lambda, opt.kstep)\n\nopt.model_path = opt.checkpoint_dir .. opt.model_name\nif not paths.dirp(opt.model_path) then\n paths.mkdir(opt.model_path)\nend\n\nprev_iter = 0\n-- load model from previous iterations\nfor i = opt.niter, 1, -opt.save_every do\n print(opt.model_path .. string.format('/net-epoch-%d.t7', i))\n if paths.filep(opt.model_path .. string.format('/net-epoch-%d.t7', i)) then\n prev_iter = i\n loader = torch.load(opt.model_path .. string.format('/net-epoch-%d.t7', i))\n state = torch.load(opt.model_path .. '/state.t7')\n print(string.format('resuming from epoch %d', i))\n break\n end\nend\n\n-- build nngraph\nencoder = loader.encoder\nactor = loader.actor\nmixer = loader.mixer\ndecoder_msk = loader.decoder_msk\ndecoder_im = loader.decoder_im\n\n-- criterion\nlocal criterion_im = nn.MSECriterion()\ncriterion_im.sizeAverage = false\nlocal criterion_msk = nn.MSECriterion()\ncriterion_msk.sizeAverage = false\n\n-- hyperparams\nfunction getAdamParams(opt)\n config = {}\n if opt.adam == 1 then\n config.learningRate = 0.0001\n config.epsilon = 1e-8\n config.beta1 = 0.9\n config.beta2 = 0.999\n config.weightDecay = opt.weight_decay\n elseif opt.adam == 2 then\n config.learningRate = 0.00001\n config.epsilon = 1e-8\n config.beta1 = 0.9\n config.beta2 = 0.999\n config.weightDecay = opt.weight_decay\n elseif opt.adam == 3 then\n config.learningRate = 0.000003\n config.epsilon = 1e-8\n config.beta1 = 0.9\n config.beta2 = 0.999\n config.weightDecay = opt.weight_decay\n end\n return config\nend\n\nconfig = getAdamParams(opt)\nprint(config)\n--------------------------------------------------------\n\nlocal batch_im_in = torch.Tensor(opt.batch_size, 3, opt.load_size, opt.load_size)\nlocal batch_rot = torch.Tensor(opt.batch_size, opt.na):zero()\nlocal batch_outputs = {}\nfor k = 1, opt.kstep do \n batch_outputs[2*k-1] = torch.Tensor(opt.batch_size, 3, opt.load_size, opt.load_size)\n batch_outputs[2*k] = torch.Tensor(opt.batch_size, 1, opt.load_size, opt.load_size)\nend\nlocal preds = {}\nfor k = 1, opt.kstep do\n preds[2*k-1] = torch.Tensor(opt.batch_size, 3, opt.load_size, opt.load_size)\n preds[2*k] = torch.Tensor(opt.batch_size, 1, opt.load_size, opt.load_size)\nend\n\nlocal errIM, errMSK\nlocal epoch_tm = torch.Timer()\nlocal tm = torch.Timer()\nlocal data_tm = torch.Timer()\n--------------------------------------------------------\nif opt.gpu > 0 then\n batch_im_in = batch_im_in:cuda()\n batch_rot = batch_rot:cuda()\n for k = 1, opt.kstep do \n batch_outputs[k*2-1] = batch_outputs[k*2-1]:cuda()\n batch_outputs[k*2] = batch_outputs[k*2]:cuda()\n end\n encoder:cuda()\n actor:cuda()\n mixer:cuda()\n decoder_msk:cuda()\n decoder_im:cuda()\n criterion_im:cuda()\n criterion_msk:cuda()\nend\n\nparams, grads = model_utils.combine_all_parameters(encoder, \n actor, mixer, decoder_msk, decoder_im)\n\nclone_actor = model_utils.clone_many_times(actor, opt.kstep)\n\nnelem = opt.batch_size * opt.kstep\n-------------------------------------------\nlocal opfunc = function(x)\n collectgarbage()\n if x ~= params then\n params:copy(x)\n end\n\n grads:zero()\n\n -- train\n data_tm:reset(); data_tm:resume()\n cur_im_in, cur_outputs, cur_rot, _ = data:getBatch() \n data_tm:stop()\n\n batch_im_in:copy(cur_im_in:mul(2):add(-1))\n for k = 1, opt.kstep do\n batch_outputs[k*2-1]:copy(cur_outputs[k*2-1]:mul(2):add(-1))\n batch_outputs[k*2]:copy(cur_outputs[k*2])\n end\n batch_rot:copy(cur_rot)\n \n local f_enc = encoder:forward(batch_im_in)\n errIM = 0\n errMSK = 0\n local df_enc_id = f_enc[1]:clone():zero()\n local df_enc_view = f_enc[2]:clone():zero()\n\n rnn_state = {f_enc[2]:clone()}\n drnn_state = {}\n for k = 1, opt.kstep do\n -- fast forward (actor, mixer, decoder)\n local f_act = clone_actor[k]:forward({rnn_state[k], batch_rot})\n table.insert(rnn_state, f_act:clone())\n local f_mix = mixer:forward({f_enc[1]:clone(), f_act})\n local f_dec_im = decoder_im:forward(f_mix)\n local f_dec_msk = decoder_msk:forward(f_mix)\n errIM = errIM + criterion_im:forward(f_dec_im, batch_outputs[k*2-1]) / (8 * nelem)\n errMSK = errMSK + criterion_msk:forward(f_dec_msk, batch_outputs[k*2]) / (2 * nelem)\n local df_dIM = criterion_im:backward(f_dec_im, batch_outputs[k*2-1]):mul(opt.lambda):div(2 * nelem)\n local df_dMSK = criterion_msk:backward(f_dec_msk, batch_outputs[k*2]):div(2 * nelem)\n -- backward (decoder_mixer)\n local df_dec_im = decoder_im:backward(f_mix, df_dIM)\n local df_dec_msk = decoder_msk:backward(f_mix, df_dMSK)\n local df_dec = df_dec_im + df_dec_msk\n local df_mix = mixer:backward({f_enc[1]:clone(), f_act}, df_dec)\n df_enc_id = df_enc_id + df_mix[1]:clone()\n table.insert(drnn_state, df_mix[2]:clone())\n end\n -- backward (actor)\n local sum_df_actor = drnn_state[opt.kstep]:clone():zero()\n for k = opt.kstep, 1, -1 do\n local tmp = clone_actor[k]:backward({rnn_state[k], batch_rot}, sum_df_actor+drnn_state[k])\n sum_df_actor = tmp[1]:clone()\n end\n df_enc_view = df_enc_view + sum_df_actor\n\n local df_enc = encoder:backward(batch_im_in, {df_enc_id, df_enc_view})\n \n local err = errIM * opt.lambda + errMSK\n return err, grads\nend\n-------------------------------------------\nlocal feedforward = function(x)\n collectgarbage()\n if x ~= params then\n params:copy(x)\n end\n \n grads:zero()\n \n -- val\n data_tm:reset(); data_tm:resume()\n cur_im_in, cur_outputs, cur_rot, _ = data_val:getBatch()\n data_tm:stop()\n\n batch_im_in:copy(cur_im_in:mul(2):add(-1))\n for k = 1, opt.kstep do\n batch_outputs[k*2-1]:copy(cur_outputs[k*2-1]:mul(2):add(-1))\n batch_outputs[k*2]:copy(cur_outputs[k*2])\n end\n batch_rot:copy(cur_rot)\n\n local f_enc = encoder:forward(batch_im_in)\n errIM = 0\n errMSK = 0\n \n rnn_state = {f_enc[2]:clone()}\n for k = 1, opt.kstep do\n -- fast forward (actor, mixer, decoder)\n local f_act = clone_actor[k]:forward({rnn_state[k], batch_rot})\n table.insert(rnn_state, f_act:clone())\n local f_mix = mixer:forward({f_enc[1]:clone(), f_act})\n local f_dec_im = decoder_im:forward(f_mix)\n local f_dec_msk = decoder_msk:forward(f_mix)\n errIM = errIM + criterion_im:forward(f_dec_im, batch_outputs[k*2-1]) / (8 * nelem)\n errMSK = errMSK + criterion_msk:forward(f_dec_msk, batch_outputs[k*2]) / (2 * nelem)\n preds[k*2-1] = f_dec_im:float():clone()\n preds[k*2] = f_dec_msk:float():clone()\n end\n\n local err = errIM * opt.lambda + errMSK\n return err\nend\n-------------------------------------------\n\n-- train & val\nfor epoch = prev_iter + 1, opt.niter do\n epoch_tm:reset()\n local counter = 0\n -- train\n encoder:training()\n mixer:training()\n decoder_msk:training()\n decoder_im:training()\n for k = 1, opt.kstep do\n clone_actor[k]:training()\n end\n for i = 1, math.min(data:size() / 5, opt.ntrain) do\n tm:reset()\n optim_utils.adam_v2(opfunc, params, config, state)\n counter = counter + 1\n print(string.format('Epoch: [%d][%8d / %8d]\\t Time: %.3f DataTime: %.3f '\n .. ' Err_Im: %.4f, Err_Msk: %.4f', epoch, i-1,\n math.min(data:size() / 5, opt.ntrain),\n tm:time().real, data_tm:time().real, \n errIM and errIM or -1, errMSK and errMSK or -1))\n end\n \n -- val\n encoder:evaluate()\n mixer:evaluate()\n decoder_msk:evaluate()\n decoder_im:evaluate()\n for k = 1, opt.kstep do\n clone_actor[k]:evaluate()\n end\n for i = 1, math.ceil(32/opt.kstep) do\n tm:reset()\n local err = feedforward(params)\n end\n\n -- plot\n local to_plot = {}\n for i = 1, math.ceil(32/opt.kstep) do\n for k = 1, opt.kstep do\n local res = batch_im_in[i]:float():clone()\n res = torch.squeeze(res)\n res:add(1):mul(0.5)\n to_plot[#to_plot+1] = res:clone()\n \n local res = preds[k*2][i]:float():clone()\n res = torch.squeeze(res)\n res = res:repeatTensor(3, 1, 1)\n res:mul(-1):add(1)\n to_plot[#to_plot+1] = res:clone()\n\n local res = preds[k*2-1][i]:float():clone()\n res = torch.squeeze(res)\n res:add(1):mul(0.5)\n to_plot[#to_plot+1] = res:clone()\n\n local res = batch_outputs[k*2-1][i]:float():clone()\n res = torch.squeeze(res)\n res:add(1):mul(0.5)\n to_plot[#to_plot+1] = res:clone()\n end\n end\n\n local formatted = image.toDisplayTensor({input=to_plot, nrow = 16})\n formatted = formatted:double()\n formatted:mul(255)\n\n formatted = formatted:byte()\n image.save(opt.model_path .. string.format('/sample-%03d.jpg', epoch), formatted)\n\n if epoch % opt.save_every == 0 then\n torch.save((opt.model_path .. string.format('/net-epoch-%d.t7', epoch)),\n {encoder = encoder, actor = actor, mixer = mixer, \n decoder_msk = decoder_msk, decoder_im = decoder_im})\n torch.save((opt.model_path .. '/state.t7'), state)\n end\nend\n\n\n" }, { "path": "utils/adam_v2.lua", "content": "local optim2 = {}\n\nfunction optim2.adam_v2(opfunc, x, config, state)\n local config = config or {}\n local state = state or config\n local lr = config.learningRate or 0.001\n local wd = config.weightDecay or 0.004\n\n local beta1 = config.beta1 or 0.1\n local beta2 = config.beta2 or 0.001\n local epsilon = config.epsilon or 1e-8\n\n local fx, dfdx = opfunc(x)\n\n if wd ~= 0 then\n dfdx:add(wd, x)\n end\n\n state.t = state.t or 0\n state.m = state.m or x.new(dfdx:size()):zero()\n state.v = state.v or x.new(dfdx:size()):zero()\n\n state.denom = state.denom or x.new(dfdx:size()):zero()\n\n state.t = state.t + 1\n\n --print(dfdx:size())\n --print(state.m:size())\n state.m:mul(beta1):add(1-beta1, dfdx)\n state.v:mul(beta2):addcmul(1-beta2, dfdx, dfdx)\n\n state.denom:copy(state.v):sqrt():add(epsilon)\n\n if state.t < 10000 then\n local biasCorrection1 = 1 - beta1^state.t\n local biasCorrection2 = 1 - beta2^state.t\n lr = lr * math.sqrt(biasCorrection2)/biasCorrection1\n end\n\n --print('lr = %g', lr)\n x:addcdiv(-lr, state.m, state.denom)\n\n return x, {fx}\nend\n\nreturn optim2\n" }, { "path": "utils/data.lua", "content": "-- code adapted from https://github.com/soumith/dcgan.torch.git\nlocal Threads = require 'threads'\nThreads.serialization('threads.sharedserialize')\n\nlocal data = {}\n\nlocal result = {}\nlocal unpack = unpack and unpack or table.unpack\n\nfunction data.new(n, dataset_name, opt_)\n opt_ = opt_ or {}\n local self = {}\n for k,v in pairs(data) do\n self[k] = v\n end\n\n local donkey_file \n if dataset_name == 'dataset_rotatorRNN_base' then\n donkey_file = 'trainset_rotatorRNN_base.lua'\n elseif dataset_name == 'dataset_rotatorRNN_curriculum' then\n donkey_file = 'trainset_rotatorRNN_curriculum.lua'\n elseif dataset_name == 'dataset_ptn' then\n donkey_file = 'trainset_ptn.lua'\n end\n\n if n > 0 then\n local options = opt_\n self.threads = Threads(n,\n function() require 'torch' end,\n function(idx)\n opt = options\n tid = idx\n local seed = (opt.manualSeed and opt.manualSeed or 0) + idx\n torch.manualSeed(seed)\n torch.setnumthreads(1)\n print(string.format('Starting donkey with id: %d seed %d', tid, seed))\n assert(options, 'options not found')\n assert(opt, 'opts not given')\n paths.dofile(donkey_file)\n end\n )\n else\n if donkey_file then paths.dofile(donkey_file) end\n self.threads = {}\n function self.threads:addjob(f1, f2) f2(f1()) end\n function self.threads:dojob() end\n function self.threads:synchronize() end\n end\n\n local nSamples = 0\n self.threads:addjob(function() return dataLoader:size() end,\n function(c) nSamples = c end)\n self.threads:synchronize()\n self._size = nSamples\n\n for i = 1, n do\n self.threads:addjob(self._getFromThreads,\n self._pushResult)\n end\n\n return self\nend\n\nfunction data._getFromThreads()\n assert(opt.batch_size, 'opt.batch_size not found')\n return dataLoader:sample(opt.batch_size)\nend\n\nfunction data._pushResult(...)\n local res = {...}\n if res == nil then\n self.threads:synchronize()\n end\n result[1] = res\nend\n\nfunction data:getBatch()\n self.threads:addjob(self._getFromThreads, self._pushResult)\n self.threads:dojob()\n local res = result[1]\n result[1] = nil\n if torch.type(res) == 'table' then\n return unpack(res)\n end\n return res\nend\n\nfunction data:size()\n return self._size\nend\n\nreturn data\n\n" }, { "path": "utils/data_test.lua", "content": "-- code adapted from https://github.com/soumith/dcgan.torch.git\nlocal Threads = require 'threads'\nThreads.serialization('threads.sharedserialize')\n\nlocal data = {}\n\nlocal result = {}\nlocal unpack = unpack and unpack or table.unpack\n\nfunction data.new(n, dataset_name, opt_)\n opt_ = opt_ or {}\n local self = {}\n for k,v in pairs(data) do\n self[k] = v\n end\n\n local donkey_file \n if dataset_name == 'dataset_rotatorRNN_base' then\n donkey_file = 'testset_rotatorRNN_base.lua'\n elseif dataset_name == 'dataset_rotatorRNN_curriculum' then\n donkey_file = 'testset_rotatorRNN_curriculum.lua'\n elseif dataset_name == 'dataset_ptn' then\n donkey_file = 'testset_ptn.lua'\n end\n\n if n > 0 then\n local options = opt_\n self.threads = Threads(n,\n function() require 'torch' end,\n function(idx)\n opt = options\n tid = idx\n local seed = (opt.manualSeed and opt.manualSeed or 0) + idx\n torch.manualSeed(seed)\n torch.setnumthreads(1)\n print(string.format('Starting donkey with id: %d seed %d', tid, seed))\n assert(options, 'options not found')\n assert(opt, 'opts not given')\n paths.dofile(donkey_file)\n end\n )\n else\n if donkey_file then paths.dofile(donkey_file) end\n self.threads = {}\n function self.threads:addjob(f1, f2) f2(f1()) end\n function self.threads:dojob() end\n function self.threads:synchronize() end\n end\n\n local nSamples = 0\n self.threads:addjob(function() return dataLoader:size() end,\n function(c) nSamples = c end)\n self.threads:synchronize()\n self._size = nSamples\n\n for i = 1, n do\n self.threads:addjob(self._getFromThreads,\n self._pushResult)\n end\n\n return self\nend\n\nfunction data._getFromThreads()\n assert(opt.batch_size, 'opt.batch_size not found')\n return dataLoader:sample(opt.batch_size)\nend\n\nfunction data._pushResult(...)\n local res = {...}\n if res == nil then\n self.threads:synchronize()\n end\n result[1] = res\nend\n\nfunction data:getBatch()\n self.threads:addjob(self._getFromThreads, self._pushResult)\n self.threads:dojob()\n local res = result[1]\n result[1] = nil\n if torch.type(res) == 'table' then\n return unpack(res)\n end\n return res\nend\n\nfunction data:size()\n return self._size\nend\n\nreturn data\n\n" }, { "path": "utils/data_val.lua", "content": "-- code adapted from https://github.com/soumith/dcgan.torch.git\nlocal Threads = require 'threads'\nThreads.serialization('threads.sharedserialize')\n\nlocal data = {}\n\nlocal result = {}\nlocal unpack = unpack and unpack or table.unpack\n\nfunction data.new(n, dataset_name, opt_)\n opt_ = opt_ or {}\n local self = {}\n for k,v in pairs(data) do\n self[k] = v\n end\n\n local donkey_file \n if dataset_name == 'dataset_rotatorRNN_base' then\n donkey_file = 'valset_rotatorRNN_base.lua'\n elseif dataset_name == 'dataset_rotatorRNN_curriculum' then\n donkey_file = 'valset_rotatorRNN_curriculum.lua'\n elseif dataset_name == 'dataset_ptn' then\n donkey_file = 'valset_ptn.lua'\n end\n\n if n > 0 then\n local options = opt_\n self.threads = Threads(n,\n function() require 'torch' end,\n function(idx)\n opt = options\n tid = idx\n local seed = (opt.manualSeed and opt.manualSeed or 0) + idx\n torch.manualSeed(seed)\n torch.setnumthreads(1)\n print(string.format('Starting donkey with id: %d seed %d', tid, seed))\n assert(options, 'options not found')\n assert(opt, 'opts not given')\n paths.dofile(donkey_file)\n end\n )\n else\n if donkey_file then paths.dofile(donkey_file) end\n self.threads = {}\n function self.threads:addjob(f1, f2) f2(f1()) end\n function self.threads:dojob() end\n function self.threads:synchronize() end\n end\n\n local nSamples = 0\n self.threads:addjob(function() return dataLoader:size() end,\n function(c) nSamples = c end)\n self.threads:synchronize()\n self._size = nSamples\n\n for i = 1, n do\n self.threads:addjob(self._getFromThreads,\n self._pushResult)\n end\n\n return self\nend\n\nfunction data._getFromThreads()\n assert(opt.batch_size, 'opt.batch_size not found')\n return dataLoader:sample(opt.batch_size)\nend\n\nfunction data._pushResult(...)\n local res = {...}\n if res == nil then\n self.threads:synchronize()\n end\n result[1] = res\nend\n\nfunction data:getBatch()\n self.threads:addjob(self._getFromThreads, self._pushResult)\n self.threads:dojob()\n local res = result[1]\n result[1] = nil\n if torch.type(res) == 'table' then\n return unpack(res)\n end\n return res\nend\n\nfunction data:size()\n return self._size\nend\n\nreturn data\n\n" }, { "path": "utils/dataset.lua", "content": "[[\n Copyright (c) 2015-present, Facebook, Inc.\n All rights reserved.\n\n This source code is licensed under the BSD-style license found in the\n LICENSE file in the root directory of this source tree. An additional grant\n of patent rights can be found in the PATENTS file in the same directory.\n]]\n\nrequire 'torch'\ntorch.setdefaulttensortype('torch.FloatTensor')\nlocal ffi = require 'ffi'\nlocal class = require('pl.class')\nlocal dir = require 'pl.dir'\nlocal tablex = require 'pl.tablex'\nlocal argcheck = require 'argcheck'\nrequire 'sys'\nrequire 'xlua'\nrequire 'image'\n\nlocal dataset = torch.class('dataLoader')\n\nlocal initcheck = argcheck{\n pack=true,\n help=[[\n A dataset class for images in a flat folder structure (folder-name is class-name).\n Optimized for extremely large datasets (upwards of 14 million images).\n Tested only on Linux (as it uses command-line linux utilities to scale up)\n]],\n {check=function(paths)\n local out = true;\n for k,v in ipairs(paths) do\n if type(v) ~= 'string' then\n print('paths can only be of string input');\n out = false\n end\n end\n return out\n end,\n name=\"paths\",\n type=\"table\",\n help=\"Multiple paths of directories with images\"},\n\n {name=\"sampleSize\",\n type=\"table\",\n help=\"a consistent sample size to resize the images\"},\n\n {name=\"split\",\n type=\"number\",\n help=\"Percentage of split to go to Training\"\n },\n\n {name=\"samplingMode\",\n type=\"string\",\n help=\"Sampling mode: random | balanced \",\n default = \"balanced\"},\n\n {name=\"verbose\",\n type=\"boolean\",\n help=\"Verbose mode during initialization\",\n default = false},\n\n {name=\"loadSize\",\n type=\"table\",\n help=\"a size to load the images to, initially\",\n opt = true},\n\n {name=\"forceClasses\",\n type=\"table\",\n help=\"If you want this loader to map certain classes to certain indices, \"\n .. \"pass a classes table that has {classname : classindex} pairs.\"\n .. \" For example: {3 : 'dog', 5 : 'cat'}\"\n .. \"This function is very useful when you want two loaders to have the same \"\n .. \"class indices (trainLoader/testLoader for example)\",\n opt = true},\n\n {name=\"sampleHookTrain\",\n type=\"function\",\n help=\"applied to sample during training(ex: for lighting jitter). \"\n .. \"It takes the image path as input\",\n opt = true},\n\n {name=\"sampleHookTest\",\n type=\"function\",\n help=\"applied to sample during testing\",\n opt = true},\n}\n\nfunction dataset:__init(...)\n\n -- argcheck\n local args = initcheck(...)\n print(args)\n for k,v in pairs(args) do self[k] = v end\n\n if not self.loadSize then self.loadSize = self.sampleSize; end\n\n if not self.sampleHookTrain then self.sampleHookTrain = self.defaultSampleHook end\n if not self.sampleHookTest then self.sampleHookTest = self.defaultSampleHook end\n\n -- find class names\n self.classes = {}\n local classPaths = {}\n if self.forceClasses then\n for k,v in pairs(self.forceClasses) do\n self.classes[k] = v\n classPaths[k] = {}\n end\n end\n local function tableFind(t, o) for k,v in pairs(t) do if v == o then return k end end end\n -- loop over each paths folder, get list of unique class names,\n -- also store the directory paths per class\n -- for each class,\n for k,path in ipairs(self.paths) do\n local dirs = dir.getdirectories(path);\n for k,dirpath in ipairs(dirs) do\n local class = paths.basename(dirpath)\n local idx = tableFind(self.classes, class)\n if not idx then\n table.insert(self.classes, class)\n idx = #self.classes\n classPaths[idx] = {}\n end\n if not tableFind(classPaths[idx], dirpath) then\n table.insert(classPaths[idx], dirpath);\n end\n end\n end\n\n self.classIndices = {}\n for k,v in ipairs(self.classes) do\n self.classIndices[v] = k\n end\n\n -- define command-line tools, try your best to maintain OSX compatibility\n local wc = 'wc'\n local cut = 'cut'\n local find = 'find'\n if jit.os == 'OSX' then\n wc = 'gwc'\n cut = 'gcut'\n find = 'gfind'\n end\n ----------------------------------------------------------------------\n -- Options for the GNU find command\n local extensionList = {'jpg', 'png','JPG','PNG','JPEG', 'ppm', 'PPM', 'bmp', 'BMP'}\n local findOptions = ' -iname \"*.' .. extensionList[1] .. '\"'\n for i=2,#extensionList do\n findOptions = findOptions .. ' -o -iname \"*.' .. extensionList[i] .. '\"'\n end\n\n -- find the image path names\n self.imagePath = torch.CharTensor() -- path to each image in dataset\n self.imageClass = torch.LongTensor() -- class index of each image (class index in self.classes)\n self.classList = {} -- index of imageList to each image of a particular class\n self.classListSample = self.classList -- the main list used when sampling data\n\n print('running \"find\" on each class directory, and concatenate all'\n .. ' those filenames into a single file containing all image paths for a given class')\n -- so, generates one file per class\n local classFindFiles = {}\n for i=1,#self.classes do\n classFindFiles[i] = os.tmpname()\n end\n local combinedFindList = os.tmpname();\n\n local tmpfile = os.tmpname()\n local tmphandle = assert(io.open(tmpfile, 'w'))\n -- iterate over classes\n for i, class in ipairs(self.classes) do\n -- iterate over classPaths\n for j,path in ipairs(classPaths[i]) do\n local command = find .. ' \"' .. path .. '\" ' .. findOptions\n .. ' >>\"' .. classFindFiles[i] .. '\" \\n'\n tmphandle:write(command)\n end\n end\n io.close(tmphandle)\n os.execute('bash ' .. tmpfile)\n os.execute('rm -f ' .. tmpfile)\n\n print('now combine all the files to a single large file')\n local tmpfile = os.tmpname()\n local tmphandle = assert(io.open(tmpfile, 'w'))\n -- concat all finds to a single large file in the order of self.classes\n for i=1,#self.classes do\n local command = 'cat \"' .. classFindFiles[i] .. '\" >>' .. combinedFindList .. ' \\n'\n tmphandle:write(command)\n end\n io.close(tmphandle)\n os.execute('bash ' .. tmpfile)\n os.execute('rm -f ' .. tmpfile)\n\n --==========================================================================\n print('load the large concatenated list of sample paths to self.imagePath')\n local maxPathLength = tonumber(sys.fexecute(wc .. \" -L '\"\n .. combinedFindList .. \"' |\"\n .. cut .. \" -f1 -d' '\")) + 1\n local length = tonumber(sys.fexecute(wc .. \" -l '\"\n .. combinedFindList .. \"' |\"\n .. cut .. \" -f1 -d' '\"))\n assert(length > 0, \"Could not find any image file in the given input paths\")\n assert(maxPathLength > 0, \"paths of files are length 0?\")\n self.imagePath:resize(length, maxPathLength):fill(0)\n local s_data = self.imagePath:data()\n local count = 0\n for line in io.lines(combinedFindList) do\n ffi.copy(s_data, line)\n s_data = s_data + maxPathLength\n if self.verbose and count % 10000 == 0 then\n xlua.progress(count, length)\n end;\n count = count + 1\n end\n\n self.numSamples = self.imagePath:size(1)\n if self.verbose then print(self.numSamples .. ' samples found.') end\n --==========================================================================\n print('Updating classList and imageClass appropriately')\n self.imageClass:resize(self.numSamples)\n local runningIndex = 0\n for i=1,#self.classes do\n if self.verbose then xlua.progress(i, #(self.classes)) end\n local length = tonumber(sys.fexecute(wc .. \" -l '\"\n .. classFindFiles[i] .. \"' |\"\n .. cut .. \" -f1 -d' '\"))\n if length == 0 then\n error('Class has zero samples')\n else\n self.classList[i] = torch.linspace(runningIndex + 1, runningIndex + length, length):long()\n self.imageClass[{{runningIndex + 1, runningIndex + length}}]:fill(i)\n end\n runningIndex = runningIndex + length\n end\n\n --==========================================================================\n -- clean up temporary files\n print('Cleaning up temporary files')\n local tmpfilelistall = ''\n for i=1,#(classFindFiles) do\n tmpfilelistall = tmpfilelistall .. ' \"' .. classFindFiles[i] .. '\"'\n if i % 1000 == 0 then\n os.execute('rm -f ' .. tmpfilelistall)\n tmpfilelistall = ''\n end\n end\n os.execute('rm -f ' .. tmpfilelistall)\n os.execute('rm -f \"' .. combinedFindList .. '\"')\n --==========================================================================\n\n if self.split == 100 then\n self.testIndicesSize = 0\n else\n print('Splitting training and test sets to a ratio of '\n .. self.split .. '/' .. (100-self.split))\n self.classListTrain = {}\n self.classListTest = {}\n self.classListSample = self.classListTrain\n local totalTestSamples = 0\n -- split the classList into classListTrain and classListTest\n for i=1,#self.classes do\n local list = self.classList[i]\n local count = self.classList[i]:size(1)\n local splitidx = math.floor((count * self.split / 100) + 0.5) -- +round\n local perm = torch.randperm(count)\n self.classListTrain[i] = torch.LongTensor(splitidx)\n for j=1,splitidx do\n self.classListTrain[i][j] = list[perm[j]]\n end\n if splitidx == count then -- all samples were allocated to train set\n self.classListTest[i] = torch.LongTensor()\n else\n self.classListTest[i] = torch.LongTensor(count-splitidx)\n totalTestSamples = totalTestSamples + self.classListTest[i]:size(1)\n local idx = 1\n for j=splitidx+1,count do\n self.classListTest[i][idx] = list[perm[j]]\n idx = idx + 1\n end\n end\n end\n -- Now combine classListTest into a single tensor\n self.testIndices = torch.LongTensor(totalTestSamples)\n self.testIndicesSize = totalTestSamples\n local tdata = self.testIndices:data()\n local tidx = 0\n for i=1,#self.classes do\n local list = self.classListTest[i]\n if list:dim() ~= 0 then\n local ldata = list:data()\n for j=0,list:size(1)-1 do\n tdata[tidx] = ldata[j]\n tidx = tidx + 1\n end\n end\n end\n end\nend\n\n-- size(), size(class)\nfunction dataset:size(class, list)\n list = list or self.classList\n if not class then\n return self.numSamples\n elseif type(class) == 'string' then\n return list[self.classIndices[class]]:size(1)\n elseif type(class) == 'number' then\n return list[class]:size(1)\n end\nend\n\n-- getByClass\nfunction dataset:getByClass(class)\n local index = math.ceil(torch.uniform() * self.classListSample[class]:nElement())\n local imgpath = ffi.string(torch.data(self.imagePath[self.classListSample[class][index]]))\n return self:sampleHookTrain(imgpath)\nend\n\n-- converts a table of samples (and corresponding labels) to a clean tensor\nlocal function tableToOutput(self, dataTable, scalarTable)\n local data, scalarLabels, labels\n local quantity = #scalarTable\n assert(dataTable[1]:dim() == 3)\n data = torch.Tensor(quantity,\n\t\t self.sampleSize[1], self.sampleSize[2], self.sampleSize[3])\n scalarLabels = torch.LongTensor(quantity):fill(-1111)\n for i=1,#dataTable do\n data[i]:copy(dataTable[i])\n scalarLabels[i] = scalarTable[i]\n end\n return data, scalarLabels\nend\n\n-- sampler, samples from the training set.\nfunction dataset:sample(quantity)\n assert(quantity)\n local dataTable = {}\n local scalarTable = {}\n for i=1,quantity do\n local class = torch.random(1, #self.classes)\n local out = self:getByClass(class)\n table.insert(dataTable, out)\n table.insert(scalarTable, class)\n end\n local data, scalarLabels = tableToOutput(self, dataTable, scalarTable)\n return data, scalarLabels\nend\n\nfunction dataset:get(i1, i2)\n local indices = torch.range(i1, i2);\n local quantity = i2 - i1 + 1;\n assert(quantity > 0)\n -- now that indices has been initialized, get the samples\n local dataTable = {}\n local scalarTable = {}\n for i=1,quantity do\n -- load the sample\n local imgpath = ffi.string(torch.data(self.imagePath[indices[i]]))\n local out = self:sampleHookTest(imgpath)\n table.insert(dataTable, out)\n table.insert(scalarTable, self.imageClass[indices[i]])\n end\n local data, scalarLabels = tableToOutput(self, dataTable, scalarTable)\n return data, scalarLabels\nend\n\nreturn dataset\n" }, { "path": "utils/model_utils.lua", "content": "\n-- adapted from https://github.com/wojciechz/learning_to_execute\n-- utilities for combining/flattening parameters in a model\n-- the code in this script is more general than it needs to be, which is \n-- why it is kind of a large\n\nrequire 'torch'\nlocal model_utils = {}\nfunction model_utils.combine_all_parameters(...)\n --[[ like module:getParameters, but operates on many modules ]]--\n\n -- get parameters\n local networks = {...}\n local parameters = {}\n local gradParameters = {}\n for i = 1, #networks do\n local net_params, net_grads = networks[i]:parameters()\n\n if net_params then\n for _, p in pairs(net_params) do\n parameters[#parameters + 1] = p\n end\n for _, g in pairs(net_grads) do\n gradParameters[#gradParameters + 1] = g\n end\n end\n end\n\n local function storageInSet(set, storage)\n local storageAndOffset = set[torch.pointer(storage)]\n if storageAndOffset == nil then\n return nil\n end\n local _, offset = unpack(storageAndOffset)\n return offset\n end\n\n -- this function flattens arbitrary lists of parameters,\n -- even complex shared ones\n local function flatten(parameters)\n if not parameters or #parameters == 0 then\n return torch.Tensor()\n end\n local Tensor = parameters[1].new\n\n local storages = {}\n local nParameters = 0\n for k = 1,#parameters do\n local storage = parameters[k]:storage()\n if not storageInSet(storages, storage) then\n storages[torch.pointer(storage)] = {storage, nParameters}\n nParameters = nParameters + storage:size()\n end\n end\n\n local flatParameters = Tensor(nParameters):fill(1)\n local flatStorage = flatParameters:storage()\n\n for k = 1,#parameters do\n local storageOffset = storageInSet(storages, parameters[k]:storage())\n parameters[k]:set(flatStorage,\n storageOffset + parameters[k]:storageOffset(),\n parameters[k]:size(),\n parameters[k]:stride())\n parameters[k]:zero()\n end\n\n local maskParameters= flatParameters:float():clone()\n local cumSumOfHoles = flatParameters:float():cumsum(1)\n local nUsedParameters = nParameters - cumSumOfHoles[#cumSumOfHoles]\n local flatUsedParameters = Tensor(nUsedParameters)\n local flatUsedStorage = flatUsedParameters:storage()\n\n for k = 1,#parameters do\n local offset = cumSumOfHoles[parameters[k]:storageOffset()]\n parameters[k]:set(flatUsedStorage,\n parameters[k]:storageOffset() - offset,\n parameters[k]:size(),\n parameters[k]:stride())\n end\n\n for _, storageAndOffset in pairs(storages) do\n local k, v = unpack(storageAndOffset)\n flatParameters[{{v+1,v+k:size()}}]:copy(Tensor():set(k))\n end\n\n if cumSumOfHoles:sum() == 0 then\n flatUsedParameters:copy(flatParameters)\n else\n local counter = 0\n for k = 1,flatParameters:nElement() do\n if maskParameters[k] == 0 then\n counter = counter + 1\n flatUsedParameters[counter] = flatParameters[counter+cumSumOfHoles[k]]\n end\n end\n assert (counter == nUsedParameters)\n end\n return flatUsedParameters\n end\n\n -- flatten parameters and gradients\n local flatParameters = flatten(parameters)\n local flatGradParameters = flatten(gradParameters)\n\n -- return new flat vector that contains all discrete parameters\n return flatParameters, flatGradParameters\nend\n\n\n\n\nfunction model_utils.clone_many_times(net, T)\n local clones = {}\n\n local params, gradParams\n if net.parameters then\n params, gradParams = net:parameters()\n if params == nil then\n params = {}\n end\n end\n\n local paramsNoGrad\n if net.parametersNoGrad then\n paramsNoGrad = net:parametersNoGrad()\n end\n\n local mem = torch.MemoryFile(\"w\"):binary()\n mem:writeObject(net)\n\n for t = 1, T do\n -- We need to use a new reader for each clone.\n -- We don't want to use the pointers to already read objects.\n local reader = torch.MemoryFile(mem:storage(), \"r\"):binary()\n local clone = reader:readObject()\n reader:close()\n\n if net.parameters then\n local cloneParams, cloneGradParams = clone:parameters()\n local cloneParamsNoGrad\n for i = 1, #params do\n cloneParams[i]:set(params[i])\n cloneGradParams[i]:set(gradParams[i])\n end\n if paramsNoGrad then\n cloneParamsNoGrad = clone:parametersNoGrad()\n for i =1,#paramsNoGrad do\n cloneParamsNoGrad[i]:set(paramsNoGrad[i])\n end\n end\n end\n\n clones[t] = clone\n collectgarbage()\n end\n\n mem:close()\n return clones\nend\n\nreturn model_utils\n" }, { "path": "utils/testset_ptn.lua", "content": "-- code adapted from https://github.com/soumith/dcgan.torch.git\nrequire 'image'\nrequire 'mattorch'\ndir = require 'pl.dir'\n\ndataLoader = {}\ndataLoader.counter = 0\n\nlocal categories = {}\nlocal files = {}\nlocal size = 0\n\nfor cat in io.lines('exp_' .. opt.exp_list .. '.txt') do\n print(cat)\n categories[#categories + 1] = cat\n local dirpath = opt.data_root .. '/' .. cat\n\n local list = opt.data_id_path .. '/' .. cat .. '_testids.txt'\n cls_files = {}\n for line in io.lines(list) do\n cls_files[#cls_files + 1] = line\n size = size + 1\n end\n files[#files + 1] = cls_files\nend\n\n------------------------------------\n\nlocal load_size = {3, opt.load_size}\n\nlocal function loadImage(path)\n local input = image.load(path, 3, 'float')\n input = image.scale(input, load_size[2], load_size[2])\n return input\nend\n\n----------------------------------------------------\nfunction dataLoader:sample(quantity)\n local class_idx_batch = torch.Tensor(quantity)\n for n = 1, quantity do\n class_idx_batch[n] = torch.randperm(#categories)[1]\n end\n\n local batch_ims = {}\n for n = 1, quantity do\n batch_ims[n] = torch.Tensor(opt.nview, 3, load_size[2], load_size[2])\n end\n local batch_vox = torch.Tensor(quantity, 1, opt.vox_size, opt.vox_size, opt.vox_size)\n\n for n = 1, quantity do\n local cls_files \n\n cls_files = files[class_idx_batch[n]]\n\n local file_idx = self.counter + n\n\n local obj_list = opt.data_view_path .. '/' .. cls_files[file_idx]\n for k = 1, opt.nview do\n local img_in = loadImage(string.format('%s/imgs/a%03d_e030.jpg', obj_list, k*(360/opt.nview)))\n batch_ims[n][k]:copy(img_in)\n end\n \n local vox_path = opt.data_vox_path .. '/' .. cls_files[file_idx]\n local vox_loader = mattorch.load(string.format('%s/model_%d.mat', vox_path, opt.vox_size))\n local vox_instance = vox_loader.voxel\n batch_vox[n]:copy(vox_instance)\n end\n\n self.counter = self.counter + quantity\n\n collectgarbage()\n\n return batch_ims, batch_vox, class_idx_batch \nend\n\nfunction dataLoader:size()\n return size\nend\n\n" }, { "path": "utils/trainset_ptn.lua", "content": "-- code adapted from https://github.com/soumith/dcgan.torch.git\nrequire 'image'\nrequire 'mattorch'\ndir = require 'pl.dir'\n\ndataLoader = {}\n\nlocal categories = {}\nlocal files = {}\nlocal size = 0\n\nfor cat in io.lines('exp_' .. opt.exp_list .. '.txt') do\n print(cat)\n categories[#categories + 1] = cat\n local dirpath = opt.data_root .. '/' .. cat\n\n local list = opt.data_id_path .. '/' .. cat .. '_trainids.txt'\n cls_files = {}\n for line in io.lines(list) do\n cls_files[#cls_files + 1] = line\n size = size + 1\n end\n files[#files + 1] = cls_files\nend\n\n------------------------------------\n\nlocal load_size = {3, opt.load_size}\n\nlocal function loadImage(path)\n local input = image.load(path, 3, 'float')\n input = image.scale(input, load_size[2], load_size[2])\n return input\nend\n\n----------------------------------------------------\nfunction dataLoader:sample(quantity)\n local class_idx_batch = torch.Tensor(quantity)\n for n = 1, quantity do\n class_idx_batch[n] = torch.randperm(#categories)[1]\n end\n\n local batch_ims = {}\n for n = 1, quantity do\n batch_ims[n] = torch.Tensor(opt.nview, 3, load_size[2], load_size[2])\n end\n local batch_vox = torch.Tensor(quantity, 1, opt.vox_size, opt.vox_size, opt.vox_size)\n\n for n = 1, quantity do\n local cls_files \n\n cls_files = files[class_idx_batch[n]]\n\n local file_idx = torch.randperm(#cls_files)[1]\n\n local obj_list = opt.data_view_path .. '/' .. cls_files[file_idx]\n for k = 1, opt.nview do\n local img_in = loadImage(string.format('%s/imgs/a%03d_e030.jpg', obj_list, k*(360/opt.nview)))\n batch_ims[n][k]:copy(img_in)\n end\n \n local vox_path = opt.data_vox_path .. '/' .. cls_files[file_idx]\n local vox_loader = mattorch.load(string.format('%s/model_%d.mat', vox_path, opt.vox_size))\n local vox_instance = vox_loader.voxel\n batch_vox[n]:copy(vox_instance)\n end\n\n collectgarbage()\n\n return batch_ims, batch_vox, class_idx_batch \nend\n\nfunction dataLoader:size()\n return size\nend\n\n" }, { "path": "utils/trainset_rotatorRNN_base.lua", "content": "-- code adapted from https://github.com/soumith/dcgan.torch.git\nrequire 'image'\ndir = require 'pl.dir'\n\ndataLoader = {}\n\nlocal categories = {}\nlocal files = {}\nlocal size = 0\n\nfor cat in io.lines('exp_' .. opt.exp_list .. '.txt') do\n print(cat)\n categories[#categories + 1] = cat\n local dirpath = opt.data_root .. '/' .. cat\n \n local list = opt.data_id_path .. '/' .. cat .. '_trainids.txt'\n cls_files = {}\n for line in io.lines(list) do\n cls_files[#cls_files + 1] = line\n size = size + 1\n end\n files[#files + 1] = cls_files\nend\n\n--------------------------\nlocal load_size = {3, opt.load_size}\n\nlocal function loadImage(path)\n local input = image.load(path, 3, 'float')\n input = image.scale(input, load_size[2], load_size[2])\n return input\nend\n\nfunction dataLoader:sample(quantity)\n local class_idx_batch = torch.Tensor(quantity)\n for n = 1, quantity do\n class_idx_batch[n] = torch.randperm(#categories)[1]\n end\n\n local batch_im_in = torch.Tensor(quantity, 3, load_size[2], load_size[2])\n local batch_rot = torch.Tensor(quantity, opt.na):zero()\n local batch_outputs = {}\n batch_outputs[1] = torch.Tensor(quantity, 3, load_size[2], load_size[2])\n batch_outputs[2] = torch.Tensor(quantity, 1, load_size[2], load_size[2])\n\n for n = 1, quantity do\n local cls_files = files[class_idx_batch[n]]\n local file_idx = torch.randperm(#cls_files)[1]\n\n local obj_list = opt.data_view_path .. '/' .. cls_files[file_idx]\n local view_in = torch.random(opt.nview)\n local rng_rot = math.random(3)\n local delta \n if rng_rot == 1 then\n delta = -1\n batch_rot[n][3] = 1\n elseif rng_rot == 2 then\n delta = 1\n batch_rot[n][1] = 1\n elseif rng_rot == 3 then\n delta = 0\n batch_rot[n][2] = 1\n end\n local view_out = view_in + delta\n if view_out > opt.nview then view_out = 1 end\n if view_out < 1 then view_out = opt.nview end\n\n local img_in = loadImage(string.format('%s/imgs/a%03d_e030.jpg', obj_list, view_in*(360/opt.nview)))\n local img_out = loadImage(string.format('%s/imgs/a%03d_e030.jpg', obj_list, view_out*(360/opt.nview)))\n local msk_out = loadImage(string.format('%s/masks/a%03d_e030.jpg', obj_list, view_out*(360/opt.nview)))\n\n batch_im_in[n]:copy(img_in)\n batch_outputs[1][n]:copy(img_out)\n batch_outputs[2][n]:copy(msk_out[1])\n end\n\n collectgarbage()\n return batch_im_in, batch_outputs, batch_rot, class_idx_batch\nend\n\nfunction dataLoader:size()\n return size\nend\n\n" }, { "path": "utils/trainset_rotatorRNN_curriculum.lua", "content": "-- code adapted from https://github.com/soumith/dcgan.torch.git\nrequire 'image'\ndir = require 'pl.dir'\n\ndataLoader = {}\n\nlocal categories = {}\nlocal files = {}\nlocal size = 0\n\nfor cat in io.lines('exp_' .. opt.exp_list .. '.txt') do\n print(cat)\n categories[#categories + 1] = cat\n local dirpath = opt.data_root .. '/' .. cat\n\n local list = opt.data_id_path .. '/' .. cat .. '_trainids.txt'\n cls_files = {}\n for line in io.lines(list) do\n cls_files[#cls_files + 1] = line\n size = size + 1\n end\n files[#files + 1] = cls_files\n\nend\n\n-----------------------------------\nlocal load_size = {3, opt.load_size}\n\nlocal function loadImage(path)\n local input = image.load(path, 3, 'float')\n input = image.scale(input, load_size[2], load_size[2])\n return input\nend\n\nfunction dataLoader:sample(quantity)\n local class_idx_batch = torch.Tensor(quantity)\n for n = 1, quantity do\n class_idx_batch[n] = torch.randperm(#categories)[1]\n end\n\n local batch_im_in = torch.Tensor(quantity, 3, load_size[2], load_size[2])\n local batch_rot = torch.Tensor(quantity, opt.na):zero()\n local batch_outputs = {}\n for k = 1, opt.kstep do\n batch_outputs[k*2-1] = torch.Tensor(quantity, 3, load_size[2], load_size[2])\n batch_outputs[k*2] = torch.Tensor(quantity, 1, load_size[2], load_size[2])\n end\n\n for n = 1, quantity do\n local cls_files = files[class_idx_batch[n]]\n local file_idx = torch.randperm(#cls_files)[1]\n\n local obj_list = opt.data_view_path .. '/' .. cls_files[file_idx]\n local view_in = torch.random(opt.nview)\n local rng_rot = math.random(2)\n local delta\n if rng_rot == 1 then\n delta = -1\n batch_rot[n][3] = 1\n elseif rng_rot == 2 then \n delta = 1 \n batch_rot[n][1] = 1\n end\n \n local img_in = loadImage(string.format('%s/imgs/a%03d_e030.jpg', obj_list, view_in*(360/opt.nview)))\n batch_im_in[n]:copy(img_in)\n\n local view_out = view_in\n for k = 1, opt.kstep do\n view_out = view_out + delta\n if view_out > opt.nview then view_out = 1 end\n if view_out < 1 then view_out = opt.nview end\n \n local img_out = loadImage(string.format('%s/imgs/a%03d_e030.jpg', obj_list, view_out*(360/opt.nview)))\n local msk_out = loadImage(string.format('%s/masks/a%03d_e030.jpg', obj_list, view_out*(360/opt.nview)))\n \n batch_outputs[k*2-1][n]:copy(img_out)\n batch_outputs[k*2][n]:copy(msk_out[1])\n end \n end\n\n collectgarbage()\n return batch_im_in, batch_outputs, batch_rot, class_idx_batch\nend\n\nfunction dataLoader:size()\n return size\nend\n\n" }, { "path": "utils/valset_ptn.lua", "content": "-- code adapted from https://github.com/soumith/dcgan.torch.git\nrequire 'image'\nrequire 'mattorch'\ndir = require 'pl.dir'\n\ndataLoader = {}\n\nlocal categories = {}\nlocal files = {}\nlocal size = 0\n\nfor cat in io.lines('exp_' .. opt.exp_list .. '.txt') do\n print(cat)\n categories[#categories + 1] = cat\n local dirpath = opt.data_root .. '/' .. cat\n\n local list = opt.data_id_path .. '/' .. cat .. '_valids.txt'\n cls_files = {}\n for line in io.lines(list) do\n cls_files[#cls_files + 1] = line\n size = size + 1\n end\n files[#files + 1] = cls_files\nend\n\n------------------------------------\n\nlocal load_size = {3, opt.load_size}\n\nlocal function loadImage(path)\n local input = image.load(path, 3, 'float')\n input = image.scale(input, load_size[2], load_size[2])\n return input\nend\n\n----------------------------------------------------\nfunction dataLoader:sample(quantity)\n local class_idx_batch = torch.Tensor(quantity)\n for n = 1, quantity do\n class_idx_batch[n] = torch.randperm(#categories)[1]\n end\n\n local batch_ims = {}\n for n = 1, quantity do\n batch_ims[n] = torch.Tensor(opt.nview, 3, load_size[2], load_size[2])\n end\n local batch_vox = torch.Tensor(quantity, 1, opt.vox_size, opt.vox_size, opt.vox_size)\n\n for n = 1, quantity do\n local cls_files \n\n cls_files = files[class_idx_batch[n]]\n\n local file_idx = torch.randperm(#cls_files)[1]\n\n local obj_list = opt.data_view_path .. '/' .. cls_files[file_idx]\n for k = 1, opt.nview do\n local img_in = loadImage(string.format('%s/imgs/a%03d_e030.jpg', obj_list, k*(360/opt.nview)))\n batch_ims[n][k]:copy(img_in)\n end\n \n local vox_path = opt.data_vox_path .. '/' .. cls_files[file_idx]\n local vox_loader = mattorch.load(string.format('%s/model_%d.mat', vox_path, opt.vox_size))\n local vox_instance = vox_loader.voxel\n batch_vox[n]:copy(vox_instance)\n end\n\n collectgarbage()\n\n return batch_ims, batch_vox, class_idx_batch \nend\n\nfunction dataLoader:size()\n return size\nend\n\n" }, { "path": "utils/valset_rotatorRNN_base.lua", "content": "-- code adapted from https://github.com/soumith/dcgan.torch.git\nrequire 'image'\ndir = require 'pl.dir'\n\ndataLoader = {}\n\nlocal categories = {}\nlocal files = {}\nlocal size = 0\n\nfor cat in io.lines('exp_' .. opt.exp_list .. '.txt') do\n print(cat)\n categories[#categories + 1] = cat\n local dirpath = opt.data_root .. '/' .. cat\n \n local list = opt.data_id_path .. '/' .. cat .. '_valids.txt'\n cls_files = {}\n for line in io.lines(list) do\n cls_files[#cls_files + 1] = line\n size = size + 1\n end\n files[#files + 1] = cls_files\nend\n\n--------------------------\nlocal load_size = {3, opt.load_size}\n\nlocal function loadImage(path)\n local input = image.load(path, 3, 'float')\n input = image.scale(input, load_size[2], load_size[2])\n return input\nend\n\nfunction dataLoader:sample(quantity)\n local class_idx_batch = torch.Tensor(quantity)\n for n = 1, quantity do\n class_idx_batch[n] = torch.randperm(#categories)[1]\n end\n\n local batch_im_in = torch.Tensor(quantity, 3, load_size[2], load_size[2])\n local batch_rot = torch.Tensor(quantity, opt.na):zero()\n local batch_outputs = {}\n batch_outputs[1] = torch.Tensor(quantity, 3, load_size[2], load_size[2])\n batch_outputs[2] = torch.Tensor(quantity, 1, load_size[2], load_size[2])\n\n for n = 1, quantity do\n local cls_files = files[class_idx_batch[n]]\n local file_idx = torch.randperm(#cls_files)[1]\n\n local obj_list = opt.data_view_path .. '/' .. cls_files[file_idx]\n local view_in = torch.random(opt.nview)\n local rng_rot = math.random(3)\n local delta \n if rng_rot == 1 then\n delta = -1\n batch_rot[n][3] = 1\n elseif rng_rot == 2 then\n delta = 1\n batch_rot[n][1] = 1\n elseif rng_rot == 3 then\n delta = 0\n batch_rot[n][2] = 1\n end\n local view_out = view_in + delta\n if view_out > opt.nview then view_out = 1 end\n if view_out < 1 then view_out = opt.nview end\n\n local img_in = loadImage(string.format('%s/imgs/a%03d_e030.jpg', obj_list, view_in*(360/opt.nview)))\n local img_out = loadImage(string.format('%s/imgs/a%03d_e030.jpg', obj_list, view_out*(360/opt.nview)))\n local msk_out = loadImage(string.format('%s/masks/a%03d_e030.jpg', obj_list, view_out*(360/opt.nview)))\n\n batch_im_in[n]:copy(img_in)\n batch_outputs[1][n]:copy(img_out)\n batch_outputs[2][n]:copy(msk_out[1])\n end\n\n collectgarbage()\n return batch_im_in, batch_outputs, batch_rot, class_idx_batch\nend\n\nfunction dataLoader:size()\n return size\nend\n\n" }, { "path": "utils/valset_rotatorRNN_curriculum.lua", "content": "-- code adapted from https://github.com/soumith/dcgan.torch.git\nrequire 'image'\ndir = require 'pl.dir'\n\ndataLoader = {}\n\nlocal categories = {}\nlocal files = {}\nlocal size = 0\n\nfor cat in io.lines('exp_' .. opt.exp_list .. '.txt') do\n print(cat)\n categories[#categories + 1] = cat\n local dirpath = opt.data_root .. '/' .. cat\n\n local list = opt.data_id_path .. '/' .. cat .. '_valids.txt'\n cls_files = {}\n for line in io.lines(list) do\n cls_files[#cls_files + 1] = line\n size = size + 1\n end\n files[#files + 1] = cls_files\n\nend\n\n-----------------------------------\nlocal load_size = {3, opt.load_size}\n\nlocal function loadImage(path)\n local input = image.load(path, 3, 'float')\n input = image.scale(input, load_size[2], load_size[2])\n return input\nend\n\nfunction dataLoader:sample(quantity)\n local class_idx_batch = torch.Tensor(quantity)\n for n = 1, quantity do\n class_idx_batch[n] = torch.randperm(#categories)[1]\n end\n\n local batch_im_in = torch.Tensor(quantity, 3, load_size[2], load_size[2])\n local batch_rot = torch.Tensor(quantity, opt.na):zero()\n local batch_outputs = {}\n for k = 1, opt.kstep do\n batch_outputs[k*2-1] = torch.Tensor(quantity, 3, load_size[2], load_size[2])\n batch_outputs[k*2] = torch.Tensor(quantity, 1, load_size[2], load_size[2])\n end\n\n for n = 1, quantity do\n local cls_files = files[class_idx_batch[n]]\n local file_idx = torch.randperm(#cls_files)[1]\n\n local obj_list = opt.data_view_path .. '/' .. cls_files[file_idx]\n local view_in = torch.random(opt.nview)\n local rng_rot = math.random(2)\n local delta\n if rng_rot == 1 then\n delta = -1\n batch_rot[n][3] = 1\n elseif rng_rot == 2 then \n delta = 1 \n batch_rot[n][1] = 1\n end\n \n local img_in = loadImage(string.format('%s/imgs/a%03d_e030.jpg', obj_list, view_in*(360/opt.nview)))\n batch_im_in[n]:copy(img_in)\n\n local view_out = view_in\n for k = 1, opt.kstep do\n view_out = view_out + delta\n if view_out > opt.nview then view_out = 1 end\n if view_out < 1 then view_out = opt.nview end\n \n local img_out = loadImage(string.format('%s/imgs/a%03d_e030.jpg', obj_list, view_out*(360/opt.nview)))\n local msk_out = loadImage(string.format('%s/masks/a%03d_e030.jpg', obj_list, view_out*(360/opt.nview)))\n \n batch_outputs[k*2-1][n]:copy(img_out)\n batch_outputs[k*2][n]:copy(msk_out[1])\n end \n end\n\n collectgarbage()\n return batch_im_in, batch_outputs, batch_rot, class_idx_batch\nend\n\nfunction dataLoader:size()\n return size\nend\n\n" } ]