[ { "path": ".gitignore", "content": "# Created by .ignore support plugin (hsz.mobi)\n### Python template\n# Byte-compiled / optimized / DLL files\n__pycache__/\n*.py[cod]\n*$py.class\n\n# C extensions\n*.so\n\n*.npy\n#*.png\n*.jpg\n*.out\noutput\ndata/*/data\n\n\n# Distribution / packaging\n.Python\nbuild/\ndevelop-eggs/\ndist/\ndownloads/\neggs/\n.eggs/\n\nlib64/\nparts/\nsdist/\nvar/\nwheels/\n*.egg-info/\n.installed.cfg\n*.egg\nMANIFEST\n\n# PyInstaller\n# Usually these files are written by a python script from a template\n# before PyInstaller builds the exe, so as to inject date/other infos into it.\n*.manifest\n*.spec\n\n# Installer logs\npip-log.txt\npip-delete-this-directory.txt\n\n# Unit test / coverage reports\nhtmlcov/\n.tox/\n.coverage\n.coverage.*\n.cache\nnosetests.xml\ncoverage.xml\n*.cover\n.hypothesis/\n.pytest_cache/\n\n# Translations\n*.mo\n*.pot\n\n# Django stuff:\n*.log\nlocal_settings.py\ndb.sqlite3\n\n# Flask stuff:\ninstance/\n.webassets-cache\n\n# Scrapy stuff:\n.scrapy\n\n# Sphinx documentation\ndocs/_build/\n\n# PyBuilder\ntarget/\n\n# Jupyter Notebook\n.ipynb_checkpoints\n\n# pyenv\n.python-version\n\n# celery beat schedule file\ncelerybeat-schedule\n\n# SageMath parsed files\n*.sage.py\n\n# Environments\n.env\n.venv\nenv/\nvenv/\nENV/\nenv.bak/\nvenv.bak/\n\n# Spyder project settings\n.spyderproject\n.spyproject\n\n# Rope project settings\n.ropeproject\n\n# mkdocs documentation\n/site\n\n# mypy\n.mypy_cache/\n### macOS template\n# General\n.DS_Store\n.AppleDouble\n.LSOverride\n\n# Icon must end with two \\r\nIcon\n\n# Thumbnails\n._*\n\n# Files that might appear in the root of a volume\n.DocumentRevisions-V100\n.fseventsd\n.Spotlight-V100\n.TemporaryItems\n.Trashes\n.VolumeIcon.icns\n.com.apple.timemachine.donotpresent\n\n# Directories potentially created on remote AFP share\n.AppleDB\n.AppleDesktop\nNetwork Trash Folder\nTemporary Items\n.apdisk\n### JetBrains template\n# Covers JetBrains IDEs: IntelliJ, RubyMine, PhpStorm, AppCode, PyCharm, CLion, Android Studio and WebStorm\n# Reference: https://intellij-support.jetbrains.com/hc/en-us/articles/206544839\n\n# User-specific stuff\n.idea/**/workspace.xml\n.idea/**/tasks.xml\n.idea/**/dictionaries\n.idea/**/shelf\n\n# Sensitive or high-churn files\n.idea/**/dataSources/\n.idea/**/dataSources.ids\n.idea/**/dataSources.local.xml\n.idea/**/sqlDataSources.xml\n.idea/**/dynamic.xml\n.idea/**/uiDesigner.xml\n.idea/**/dbnavigator.xml\n\n# Gradle\n.idea/**/gradle.xml\n.idea/**/libraries\n\n# CMake\ncmake-build-debug/\ncmake-build-release/\n\n# Mongo Explorer plugin\n.idea/**/mongoSettings.xml\n\n# File-based project format\n*.iws\n\n# IntelliJ\nout/\n\n# mpeltonen/sbt-idea plugin\n.idea_modules/\n\n# JIRA plugin\natlassian-ide-plugin.xml\n\n# Cursive Clojure plugin\n.idea/replstate.xml\n\n# Crashlytics plugin (for Android Studio and IntelliJ)\ncom_crashlytics_export_strings.xml\ncrashlytics.properties\ncrashlytics-build.properties\nfabric.properties\n\n# Editor-based Rest Client\n.idea/httpRequests\n\n" }, { "path": "README.md", "content": "# HandOccNet: Occlusion-Robust 3D Hand Mesh Estimation Network\n\n## Introduction\nThis repository is the offical [Pytorch](https://pytorch.org/) implementation of **[HandOccNet: Occlusion-Robust 3D Hand Mesh Estimation Network (CVPR 2022)](https://arxiv.org/abs/2203.14564)**. Below is the overall pipeline of HandOccNet.\n![overall pipeline](./asset/model.png)\n\n## Quick demo\n* Install **[PyTorch](https://pytorch.org)** and Python >= 3.7.4 and run `sh requirements.sh`.\n* Download `snapshot_demo.pth.tar` from [here](https://drive.google.com/drive/folders/1OlyV-qbzOmtQYdzV6dbQX4OtAU5ajBOa?usp=sharing) and place at `demo` folder.\n* Prepare `input.jpg` at `demo` folder.\n* Download `MANO_RIGHT.pkl` from [here](https://mano.is.tue.mpg.de/) and place at `common/utils/manopth/mano/models`.\n* Go to `demo` folder and edit `bbox` in [here](https://github.com/namepllet/HandOccNet/blob/185492e0e5b08c47e37039c5d67e3f2b099a6f9e/demo/demo.py#L61).\n* Run `python demo.py --gpu 0` if you want to run on gpu 0.\n* You can see `hand_bbox.png`, `hand_image.png`, and `output.obj`.\n* Run `python demo_fitting.py --gpu 0 --depth 0.5` if you want to get the hand mesh's translation from the camera. The depth argument is initialization for the optimization.\n* You can see `fitting_input_3d_mesh.json` that contains the translation and MANO parameters, `fitting_input_3dmesh.obj`, `fitting_input_2d_prediction.png`, and `fitting_input_projection.png`.\n \n## Directory\n### Root \nThe `${ROOT}` is described as below. \n``` \n${ROOT} \n|-- data \n|-- demo\n|-- common \n|-- main \n|-- output \n``` \n* `data` contains data loading codes and soft links to images and annotations directories. \n* `demo` contains demo codes.\n* `common` contains kernel codes for HandOccNet. \n* `main` contains high-level codes for training or testing the network. \n* `output` contains log, trained models, visualized outputs, and test result. \n\n### Data \nYou need to follow directory structure of the `data` as below. \n``` \n${ROOT} \n|-- data \n| |-- HO3D\n| | |-- data\n| | | |-- train\n| | | | |-- ABF10\n| | | | |-- ......\n| | | |-- evaluation\n| | | |-- annotations\n| | | | |-- HO3D_train_data.json\n| | | | |-- HO3D_evaluation_data.json\n| |-- DEX_YCB\n| | |-- data\n| | | |-- 20200709-subject-01\n| | | |-- ......\n| | | |-- annotations\n| | | | |--DEX_YCB_s0_train_data.json\n| | | | |--DEX_YCB_s0_test_data.json\n``` \n* Download HO3D(version 2) data and annotation files [[data](https://www.tugraz.at/institute/icg/research/team-lepetit/research-projects/hand-object-3d-pose-annotation/)][[annotation files](https://drive.google.com/drive/folders/1pmRpgv38PXvlLOODtoxpTYnIpYTkNV6b?usp=sharing)]\n* Download DexYCB data and annotation files [[data](https://dex-ycb.github.io/)][[annotation files](https://drive.google.com/drive/folders/1pmRpgv38PXvlLOODtoxpTYnIpYTkNV6b?usp=sharing)] \n\n### Pytorch MANO layer\n* For the MANO layer, I used [manopth](https://github.com/hassony2/manopth). The repo is already included in `common/utils/manopth`.\n* Download `MANO_RIGHT.pkl` from [here](https://mano.is.tue.mpg.de/) and place at `common/utils/manopth/mano/models`.\n\n### Output \nYou need to follow the directory structure of the `output` folder as below. \n``` \n${ROOT} \n|-- output \n| |-- log \n| |-- model_dump \n| |-- result \n| |-- vis \n``` \n* Creating `output` folder as soft link form is recommended instead of folder form because it would take large storage capacity. \n* `log` folder contains training log file. \n* `model_dump` folder contains saved checkpoints for each epoch. \n* `result` folder contains final estimation files generated in the testing stage. \n* `vis` folder contains visualized results. \n\n## Running HandOccNet\n### Start \n* Install **[PyTorch](https://pytorch.org)** and Python >= 3.7.4 and run `sh requirements.sh`.\n* In the `main/config.py`, you can change settings of the model including dataset to use and input size and so on. \n\n### Train \nIn the `main` folder, set trainset in `config.py` (as 'HO3D' or 'DEX_YCB') and run \n```bash \npython train.py --gpu 0-3\n``` \nto train HandOccNet on the GPU 0,1,2,3. `--gpu 0,1,2,3` can be used instead of `--gpu 0-3`.\n\n### Test \nPlace trained model at the `output/model_dump/`.\n \nIn the `main` folder, set testset in `config.py` (as 'HO3D' or 'DEX_YCB') and run \n```bash \npython test.py --gpu 0-3 --test_epoch {test epoch} \n``` \nto test HandOccNet on the GPU 0,1,2,3 with {test epoch}th epoch trained model. `--gpu 0,1,2,3` can be used instead of `--gpu 0-3`.\n\n* For the HO3D dataset, pred{test epoch}.zip will be generated in `output/result` folder. You can upload it to the [codalab challenge](https://competitions.codalab.org/competitions/22485) and see the results.\n* Our trained model can be downloaded from [here](https://drive.google.com/drive/folders/1OlyV-qbzOmtQYdzV6dbQX4OtAU5ajBOa?usp=sharing)\n\n## Results \nHere I report the performance of the HandOccNet.\n

\n\n

\n\n

\n\n

\n\n## Reference\n``` \n@InProceedings{Park_2022_CVPR_HandOccNet, \nauthor = {Park, JoonKyu and Oh, Yeonguk and Moon, Gyeongsik and Choi, Hongsuk and Lee, Kyoung Mu}, \ntitle = {HandOccNet: Occlusion-Robust 3D Hand Mesh Estimation Network}, \nbooktitle = {Conference on Computer Vision and Pattern Recognition (CVPR)}, \nyear = {2022} \n} \n```\n## Acknowledgements\nFor this project, we relied on research codes from:\n* [I2L-MeshNet_RELEASE](https://github.com/mks0601/I2L-MeshNet_RELEASE)\n* [Semi-Hand-Object](https://github.com/stevenlsw/Semi-Hand-Object)\n* [attention-module](https://github.com/Jongchan/attention-module)\n" }, { "path": "common/base.py", "content": "import os\nimport os.path as osp\nimport math\nimport time\nimport glob\nimport abc\nfrom torch.utils.data import DataLoader\nimport torch.optim\nimport torchvision.transforms as transforms\nfrom timer import Timer\nfrom logger import colorlogger\nfrom torch.nn.parallel.data_parallel import DataParallel\nfrom config import cfg\nfrom model import get_model\n\n# dynamic dataset import\nexec('from ' + cfg.trainset + ' import ' + cfg.trainset)\nexec('from ' + cfg.testset + ' import ' + cfg.testset)\n\nclass Base(object):\n __metaclass__ = abc.ABCMeta\n\n def __init__(self, log_name='logs.txt'):\n \n self.cur_epoch = 0\n\n # timer\n self.tot_timer = Timer()\n self.gpu_timer = Timer()\n self.read_timer = Timer()\n\n # logger\n self.logger = colorlogger(cfg.log_dir, log_name=log_name)\n\n @abc.abstractmethod\n def _make_batch_generator(self):\n return\n\n @abc.abstractmethod\n def _make_model(self):\n return\n\nclass Trainer(Base):\n def __init__(self):\n super(Trainer, self).__init__(log_name = 'train_logs.txt')\n\n def get_optimizer(self, model):\n model_params = filter(lambda p: p.requires_grad, model.parameters())\n optimizer = torch.optim.Adam(model_params, lr=cfg.lr)\n return optimizer\n\n def save_model(self, state, epoch):\n file_path = osp.join(cfg.model_dir,'snapshot_{}.pth.tar'.format(str(epoch)))\n torch.save(state, file_path)\n self.logger.info(\"Write snapshot into {}\".format(file_path))\n\n def load_model(self, model, optimizer):\n model_file_list = glob.glob(osp.join(cfg.model_dir,'*.pth.tar'))\n cur_epoch = max([int(file_name[file_name.find('snapshot_') + 9 : file_name.find('.pth.tar')]) for file_name in model_file_list])\n ckpt_path = osp.join(cfg.model_dir, 'snapshot_' + str(cur_epoch) + '.pth.tar')\n ckpt = torch.load(ckpt_path) \n start_epoch = ckpt['epoch'] + 1\n model.load_state_dict(ckpt['network'], strict=False)\n #optimizer.load_state_dict(ckpt['optimizer'])\n\n self.logger.info('Load checkpoint from {}'.format(ckpt_path))\n return start_epoch, model, optimizer\n\n def set_lr(self, epoch):\n for e in cfg.lr_dec_epoch:\n if epoch < e:\n break\n if epoch < cfg.lr_dec_epoch[-1]:\n idx = cfg.lr_dec_epoch.index(e)\n for g in self.optimizer.param_groups:\n g['lr'] = cfg.lr * (cfg.lr_dec_factor ** idx)\n else:\n for g in self.optimizer.param_groups:\n g['lr'] = cfg.lr * (cfg.lr_dec_factor ** len(cfg.lr_dec_epoch))\n\n def get_lr(self):\n for g in self.optimizer.param_groups:\n cur_lr = g['lr']\n return cur_lr\n \n def _make_batch_generator(self):\n # data load and construct batch generator\n self.logger.info(\"Creating dataset...\")\n train_dataset = eval(cfg.trainset)(transforms.ToTensor(), \"train\")\n \n self.itr_per_epoch = math.ceil(len(train_dataset) / cfg.num_gpus / cfg.train_batch_size)\n self.batch_generator = DataLoader(dataset=train_dataset, batch_size=cfg.num_gpus*cfg.train_batch_size, shuffle=True, num_workers=cfg.num_thread, pin_memory=True)\n\n def _make_model(self):\n # prepare network\n self.logger.info(\"Creating graph and optimizer...\")\n model = get_model('train')\n\n model = DataParallel(model).cuda()\n optimizer = self.get_optimizer(model)\n if cfg.continue_train:\n start_epoch, model, optimizer = self.load_model(model, optimizer)\n else:\n start_epoch = 0\n model.train()\n\n self.start_epoch = start_epoch\n self.model = model\n self.optimizer = optimizer\n\nclass Tester(Base):\n def __init__(self, test_epoch):\n self.test_epoch = int(test_epoch)\n super(Tester, self).__init__(log_name = 'test_logs.txt')\n\n def _make_batch_generator(self):\n # data load and construct batch generator\n self.logger.info(\"Creating dataset...\")\n self.test_dataset = eval(cfg.testset)(transforms.ToTensor(), \"test\")\n self.batch_generator = DataLoader(dataset=self.test_dataset, batch_size=cfg.num_gpus*cfg.test_batch_size, shuffle=False, num_workers=cfg.num_thread, pin_memory=True)\n \n def _make_model(self):\n model_path = os.path.join(cfg.model_dir, 'snapshot_%d.pth.tar' % self.test_epoch)\n assert os.path.exists(model_path), 'Cannot find model at ' + model_path\n self.logger.info('Load checkpoint from {}'.format(model_path))\n \n # prepare network\n self.logger.info(\"Creating graph...\")\n model = get_model('test')\n model = DataParallel(model).cuda()\n ckpt = torch.load(model_path)\n model.load_state_dict(ckpt['network'], strict=False)\n model.eval()\n\n self.model = model\n\n def _evaluate(self, outs, cur_sample_idx):\n eval_result = self.test_dataset.evaluate(outs, cur_sample_idx)\n return eval_result\n\n def _print_eval_result(self, test_epoch):\n self.test_dataset.print_eval_result(test_epoch)" }, { "path": "common/logger.py", "content": "import logging\nimport os\n\nOK = '\\033[92m'\nWARNING = '\\033[93m'\nFAIL = '\\033[91m'\nEND = '\\033[0m'\n\nPINK = '\\033[95m'\nBLUE = '\\033[94m'\nGREEN = OK\nRED = FAIL\nWHITE = END\nYELLOW = WARNING\n\nclass colorlogger():\n def __init__(self, log_dir, log_name='train_logs.txt'):\n # set log\n self._logger = logging.getLogger(log_name)\n self._logger.setLevel(logging.INFO)\n log_file = os.path.join(log_dir, log_name)\n if not os.path.exists(log_dir):\n os.makedirs(log_dir)\n file_log = logging.FileHandler(log_file, mode='a')\n file_log.setLevel(logging.INFO)\n console_log = logging.StreamHandler()\n console_log.setLevel(logging.INFO)\n formatter = logging.Formatter(\n \"{}%(asctime)s{} %(message)s\".format(GREEN, END),\n \"%m-%d %H:%M:%S\")\n file_log.setFormatter(formatter)\n console_log.setFormatter(formatter)\n self._logger.addHandler(file_log)\n self._logger.addHandler(console_log)\n\n def debug(self, msg):\n self._logger.debug(str(msg))\n\n def info(self, msg):\n self._logger.info(str(msg))\n\n def warning(self, msg):\n self._logger.warning(WARNING + 'WRN: ' + str(msg) + END)\n\n def critical(self, msg):\n self._logger.critical(RED + 'CRI: ' + str(msg) + END)\n\n def error(self, msg):\n self._logger.error(RED + 'ERR: ' + str(msg) + END)" }, { "path": "common/nets/backbone.py", "content": "import torch.nn as nn\nimport torch.nn.functional as F\nimport torch.utils.model_zoo as model_zoo\n\nfrom torchvision import ops\nimport torch\n\nfrom nets.cbam import SpatialGate\n\nclass FPN(nn.Module):\n def __init__(self, pretrained=True):\n super(FPN, self).__init__()\n self.in_planes = 64\n\n resnet = resnet50(pretrained=pretrained)\n\n self.toplayer = nn.Conv2d(2048, 256, kernel_size=1, stride=1, padding=0) # Reduce channels\n\n self.layer0 = nn.Sequential(resnet.conv1, resnet.bn1, resnet.leakyrelu, resnet.maxpool)\n self.layer1 = nn.Sequential(resnet.layer1)\n self.layer2 = nn.Sequential(resnet.layer2)\n self.layer3 = nn.Sequential(resnet.layer3)\n self.layer4 = nn.Sequential(resnet.layer4)\n\n # Smooth layers\n #self.smooth1 = nn.Conv2d(256, 256, kernel_size=3, stride=1, padding=1)\n self.smooth2 = nn.Conv2d(256, 256, kernel_size=3, stride=1, padding=1)\n self.smooth3 = nn.Conv2d(256, 256, kernel_size=3, stride=1, padding=1)\n\n # Lateral layers\n self.latlayer1 = nn.Conv2d(1024, 256, kernel_size=1, stride=1, padding=0)\n self.latlayer2 = nn.Conv2d( 512, 256, kernel_size=1, stride=1, padding=0)\n self.latlayer3 = nn.Conv2d( 256, 256, kernel_size=1, stride=1, padding=0)\n\n # Attention Module\n self.attention_module = SpatialGate()\n\n self.pool = nn.AvgPool2d(2, stride=2)\n\n def _upsample_add(self, x, y):\n _, _, H, W = y.size()\n return F.interpolate(x, size=(H,W), mode='bilinear', align_corners=False) + y\n\n def forward(self, x):\n # Bottom-up\n c1 = self.layer0(x)\n c2 = self.layer1(c1)\n c3 = self.layer2(c2)\n c4 = self.layer3(c3)\n c5 = self.layer4(c4)\n # Top-down\n p5 = self.toplayer(c5)\n p4 = self._upsample_add(p5, self.latlayer1(c4))\n p3 = self._upsample_add(p4, self.latlayer2(c3))\n p2 = self._upsample_add(p3, self.latlayer3(c2))\n # Smooth\n #p4 = self.smooth1(p4)\n p3 = self.smooth2(p3)\n p2 = self.smooth3(p2)\n \n # Attention\n p2 = self.pool(p2)\n primary_feats, secondary_feats = self.attention_module(p2)\n \n return primary_feats, secondary_feats\n\n\nclass ResNet(nn.Module):\n def __init__(self, block, layers, num_classes=1000):\n self.inplanes = 64\n super(ResNet, self).__init__()\n self.conv1 = nn.Conv2d(3, 64, kernel_size=7, stride=2, padding=3, bias=False)\n self.bn1 = nn.BatchNorm2d(64)\n self.leakyrelu = nn.LeakyReLU(inplace=True)\n self.maxpool = nn.MaxPool2d(kernel_size=3, stride=2, padding=1)\n self.layer1 = self._make_layer(block, 64, layers[0])\n self.layer2 = self._make_layer(block, 128, layers[1], stride=2)\n self.layer3 = self._make_layer(block, 256, layers[2], stride=2)\n self.layer4 = self._make_layer(block, 512, layers[3], stride=2)\n self.avgpool = nn.AvgPool2d(7, stride=1)\n self.fc = nn.Linear(512 * block.expansion, num_classes)\n\n for m in self.modules():\n if isinstance(m, nn.Conv2d):\n nn.init.kaiming_normal_(m.weight, mode=\"fan_out\", nonlinearity=\"leaky_relu\")\n elif isinstance(m, nn.BatchNorm2d):\n nn.init.constant_(m.weight, 1)\n nn.init.constant_(m.bias, 0)\n\n def _make_layer(self, block, planes, blocks, stride=1):\n downsample = None\n if stride != 1 or self.inplanes != planes * block.expansion:\n downsample = nn.Sequential(\n nn.Conv2d(self.inplanes, planes * block.expansion,\n kernel_size=1, stride=stride, bias=False),\n nn.BatchNorm2d(planes * block.expansion))\n layers = []\n layers.append(block(self.inplanes, planes, stride, downsample))\n self.inplanes = planes * block.expansion\n for i in range(1, blocks):\n layers.append(block(self.inplanes, planes))\n\n return nn.Sequential(*layers)\n\n def forward(self, x):\n x = self.conv1(x)\n x = self.bn1(x)\n x = self.leakyrelu(x)\n x = self.maxpool(x)\n\n x = self.layer1(x)\n x = self.layer2(x)\n x = self.layer3(x)\n x = self.layer4(x)\n\n x = x.mean(3).mean(2)\n x = x.view(x.size(0), -1)\n x = self.fc(x)\n return x\n\n\ndef resnet50(pretrained=False, **kwargs):\n \"\"\"Constructs a ResNet-50 model Encoder\"\"\"\n model = ResNet(Bottleneck, [3, 4, 6, 3], **kwargs)\n if pretrained:\n model.load_state_dict(model_zoo.load_url(\"https://download.pytorch.org/models/resnet50-19c8e357.pth\"))\n return model\n\n\ndef conv3x3(in_planes, out_planes, stride=1):\n return nn.Conv2d(in_planes, out_planes, kernel_size=3, stride=stride, padding=1, bias=False)\n\n\nclass BasicBlock(nn.Module):\n expansion = 1\n\n def __init__(self, inplanes, planes, stride=1, downsample=None):\n super(BasicBlock, self).__init__()\n self.conv1 = conv3x3(inplanes, planes, stride)\n self.bn1 = nn.BatchNorm2d(planes)\n self.leakyrelu = nn.LeakyReLU(inplace=True)\n self.conv2 = conv3x3(planes, planes)\n self.bn2 = nn.BatchNorm2d(planes)\n self.downsample = downsample\n self.stride = stride\n\n def forward(self, x):\n residual = x\n\n out = self.conv1(x)\n out = self.bn1(out)\n out = self.leakyrelu(out)\n\n out = self.conv2(out)\n out = self.bn2(out)\n\n if self.downsample is not None:\n residual = self.downsample(x)\n\n out += residual\n out = self.leakyrelu(out)\n\n return out\n\n\nclass Bottleneck(nn.Module):\n expansion = 4\n\n def __init__(self, inplanes, planes, stride=1, downsample=None):\n super(Bottleneck, self).__init__()\n self.conv1 = nn.Conv2d(inplanes, planes, kernel_size=1, bias=False)\n self.bn1 = nn.BatchNorm2d(planes)\n self.conv2 = nn.Conv2d(\n planes, planes, kernel_size=3, stride=stride, padding=1, bias=False\n )\n self.bn2 = nn.BatchNorm2d(planes)\n self.conv3 = nn.Conv2d(\n planes, planes * self.expansion, kernel_size=1, bias=False\n )\n self.bn3 = nn.BatchNorm2d(planes * self.expansion)\n self.leakyrelu = nn.LeakyReLU(inplace=True)\n self.downsample = downsample\n self.stride = stride\n\n def forward(self, x):\n residual = x\n\n out = self.conv1(x)\n out = self.bn1(out)\n out = self.leakyrelu(out)\n\n out = self.conv2(out)\n out = self.bn2(out)\n out = self.leakyrelu(out)\n\n out = self.conv3(out)\n out = self.bn3(out)\n\n if self.downsample is not None:\n residual = self.downsample(x)\n\n out += residual\n out = self.leakyrelu(out)\n\n return out" }, { "path": "common/nets/cbam.py", "content": "import torch\nimport math\nimport torch.nn as nn\nimport torch.nn.functional as F\n\nclass BasicConv(nn.Module):\n def __init__(self, in_planes, out_planes, kernel_size, stride=1, padding=0, dilation=1, groups=1, relu=True, bn=True, bias=False):\n super(BasicConv, self).__init__()\n self.out_channels = out_planes\n self.conv = nn.Conv2d(in_planes, out_planes, kernel_size=kernel_size, stride=stride, padding=padding, dilation=dilation, groups=groups, bias=bias)\n self.bn = nn.BatchNorm2d(out_planes,eps=1e-5, momentum=0.01, affine=True) if bn else None\n self.relu = nn.ReLU() if relu else None\n\n def forward(self, x):\n x = self.conv(x)\n if self.bn is not None:\n x = self.bn(x)\n if self.relu is not None:\n x = self.relu(x)\n return x\n\nclass Flatten(nn.Module):\n def forward(self, x):\n return x.view(x.size(0), -1)\n\nclass ChannelGate(nn.Module):\n def __init__(self, gate_channels, reduction_ratio=16, pool_types=['avg', 'max']):\n super(ChannelGate, self).__init__()\n self.gate_channels = gate_channels\n self.mlp = nn.Sequential(\n Flatten(),\n nn.Linear(gate_channels, gate_channels // reduction_ratio),\n nn.ReLU(),\n nn.Linear(gate_channels // reduction_ratio, gate_channels)\n )\n self.pool_types = pool_types\n def forward(self, x):\n channel_att_sum = None\n for pool_type in self.pool_types:\n if pool_type=='avg':\n avg_pool = F.avg_pool2d( x, (x.size(2), x.size(3)), stride=(x.size(2), x.size(3)))\n channel_att_raw = self.mlp( avg_pool )\n elif pool_type=='max':\n max_pool = F.max_pool2d( x, (x.size(2), x.size(3)), stride=(x.size(2), x.size(3)))\n channel_att_raw = self.mlp( max_pool )\n elif pool_type=='lp':\n lp_pool = F.lp_pool2d( x, 2, (x.size(2), x.size(3)), stride=(x.size(2), x.size(3)))\n channel_att_raw = self.mlp( lp_pool )\n elif pool_type=='lse':\n # LSE pool only\n lse_pool = logsumexp_2d(x)\n channel_att_raw = self.mlp( lse_pool )\n\n if channel_att_sum is None:\n channel_att_sum = channel_att_raw\n else:\n channel_att_sum = channel_att_sum + channel_att_raw\n\n scale = F.sigmoid( channel_att_sum ).unsqueeze(2).unsqueeze(3).expand_as(x)\n return x * scale\n\ndef logsumexp_2d(tensor):\n tensor_flatten = tensor.view(tensor.size(0), tensor.size(1), -1)\n s, _ = torch.max(tensor_flatten, dim=2, keepdim=True)\n outputs = s + (tensor_flatten - s).exp().sum(dim=2, keepdim=True).log()\n return outputs\n\nclass ChannelPool(nn.Module):\n def forward(self, x):\n return torch.cat( (torch.max(x,1)[0].unsqueeze(1), torch.mean(x,1).unsqueeze(1)), dim=1 )\n\nclass SpatialGate(nn.Module):\n def __init__(self):\n super(SpatialGate, self).__init__()\n kernel_size = 7\n self.compress = ChannelPool()\n self.spatial = BasicConv(2, 1, kernel_size, stride=1, padding=(kernel_size-1) // 2, relu=False)\n def forward(self, x):\n x_compress = self.compress(x)\n x_out = self.spatial(x_compress)\n scale = F.sigmoid(x_out) # broadcasting\n return x*scale, x*(1-scale)\n\nclass CBAM(nn.Module):\n def __init__(self, gate_channels, reduction_ratio=16, pool_types=['avg', 'max'], no_spatial=False):\n super(CBAM, self).__init__()\n self.ChannelGate = ChannelGate(gate_channels, reduction_ratio, pool_types)\n self.no_spatial=no_spatial\n if not no_spatial:\n self.SpatialGate = SpatialGate()\n def forward(self, x):\n x_out = self.ChannelGate(x)\n if not self.no_spatial:\n x_out = self.SpatialGate(x_out)\n return x_out" }, { "path": "common/nets/hand_head.py", "content": "import torch\nfrom torch import nn\nimport torch.nn.functional as F\n\nclass hand_regHead(nn.Module):\n def __init__(self, roi_res=32, joint_nb=21, stacks=1, channels=256, blocks=1):\n \"\"\"\n Args:\n inr_res: input image size\n joint_nb: hand joint num\n \"\"\"\n super(hand_regHead, self).__init__()\n\n # hand head\n self.out_res = roi_res\n self.joint_nb = joint_nb\n\n self.channels = channels\n self.blocks = blocks\n self.stacks = stacks\n\n self.betas = nn.Parameter(torch.ones((self.joint_nb, 1), dtype=torch.float32))\n\n center_offset = 0.5\n vv, uu = torch.meshgrid(torch.arange(self.out_res).float(), torch.arange(self.out_res).float())\n uu, vv = uu + center_offset, vv + center_offset\n self.register_buffer(\"uu\", uu / self.out_res)\n self.register_buffer(\"vv\", vv / self.out_res)\n\n self.softmax = nn.Softmax(dim=2)\n block = Bottleneck\n self.features = self.channels // block.expansion\n\n hg, res, fc, score, fc_, score_ = [], [], [], [], [], []\n for i in range(self.stacks):\n hg.append(Hourglass(block, self.blocks, self.features, 4))\n res.append(self.make_residual(block, self.channels, self.features, self.blocks))\n fc.append(BasicBlock(self.channels, self.channels, kernel_size=1))\n score.append(nn.Conv2d(self.channels, self.joint_nb, kernel_size=1, bias=True))\n if i < self.stacks - 1:\n fc_.append(nn.Conv2d(self.channels, self.channels, kernel_size=1, bias=True))\n score_.append(nn.Conv2d(self.joint_nb, self.channels, kernel_size=1, bias=True))\n\n self.hg = nn.ModuleList(hg)\n self.res = nn.ModuleList(res)\n self.fc = nn.ModuleList(fc)\n self.score = nn.ModuleList(score)\n self.fc_ = nn.ModuleList(fc_)\n self.score_ = nn.ModuleList(score_)\n\n def make_residual(self, block, inplanes, planes, blocks, stride=1):\n skip = None\n if stride != 1 or inplanes != planes * block.expansion:\n skip = nn.Sequential(\n nn.Conv2d(inplanes, planes * block.expansion, kernel_size=1, stride=stride, bias=True))\n layers = []\n layers.append(block(inplanes, planes, stride, skip))\n for i in range(1, blocks):\n layers.append(block(inplanes, planes))\n return nn.Sequential(*layers)\n\n def spatial_softmax(self, latents):\n latents = latents.view((-1, self.joint_nb, self.out_res ** 2))\n latents = latents * self.betas\n heatmaps = self.softmax(latents)\n heatmaps = heatmaps.view(-1, self.joint_nb, self.out_res, self.out_res)\n return heatmaps\n\n def generate_output(self, heatmaps):\n predictions = torch.stack((\n torch.sum(torch.sum(heatmaps * self.uu, dim=2), dim=2),\n torch.sum(torch.sum(heatmaps * self.vv, dim=2), dim=2)), dim=2)\n return predictions\n\n def forward(self, x):\n out, encoding, preds = [], [], []\n for i in range(self.stacks):\n y = self.hg[i](x)\n y = self.res[i](y)\n y = self.fc[i](y)\n latents = self.score[i](y)\n heatmaps= self.spatial_softmax(latents)\n out.append(heatmaps)\n predictions = self.generate_output(heatmaps)\n preds.append(predictions)\n if i < self.stacks - 1:\n fc_ = self.fc_[i](y)\n score_ = self.score_[i](heatmaps)\n x = x + fc_ + score_\n encoding.append(x)\n else:\n encoding.append(y)\n return out, encoding, preds\n\n\nclass BasicBlock(nn.Module):\n def __init__(self, in_planes, out_planes, kernel_size,groups=1):\n super(BasicBlock, self).__init__()\n self.block = nn.Sequential(\n nn.Conv2d(in_planes, out_planes, kernel_size=kernel_size,\n stride=1, padding=((kernel_size - 1) // 2),\n groups=groups,bias=True),\n nn.BatchNorm2d(out_planes),\n nn.LeakyReLU(inplace=True)\n )\n\n def forward(self, x):\n return self.block(x)\n\n\nclass Residual(nn.Module):\n def __init__(self, numIn, numOut):\n super(Residual, self).__init__()\n self.numIn = numIn\n self.numOut = numOut\n self.bn = nn.BatchNorm2d(self.numIn)\n self.leakyrelu = nn.LeakyReLU(inplace=True)\n self.conv1 = nn.Conv2d(self.numIn, self.numOut // 2, bias=True, kernel_size=1)\n self.bn1 = nn.BatchNorm2d(self.numOut // 2)\n self.conv2 = nn.Conv2d(self.numOut // 2, self.numOut // 2, bias=True, kernel_size=3, stride=1, padding=1)\n self.bn2 = nn.BatchNorm2d(self.numOut // 2)\n self.conv3 = nn.Conv2d(self.numOut // 2, self.numOut, bias=True, kernel_size=1)\n\n if self.numIn != self.numOut:\n self.conv4 = nn.Conv2d(self.numIn, self.numOut, bias=True, kernel_size=1)\n\n def forward(self, x):\n residual = x\n out = self.bn(x)\n out = self.leakyrelu(out)\n out = self.conv1(out)\n out = self.bn1(out)\n out = self.leakyrelu(out)\n out = self.conv2(out)\n out = self.bn2(out)\n out = self.leakyrelu(out)\n out = self.conv3(out)\n\n if self.numIn != self.numOut:\n residual = self.conv4(x)\n\n return out + residual\n\n\nclass Bottleneck(nn.Module):\n expansion = 2\n\n def __init__(self, inplanes, planes, stride=1, skip=None, groups=1):\n super(Bottleneck, self).__init__()\n\n self.bn1 = nn.BatchNorm2d(inplanes)\n self.conv1 = nn.Conv2d(inplanes, planes, kernel_size=1, bias=True, groups=groups)\n self.bn2 = nn.BatchNorm2d(planes)\n self.conv2 = nn.Conv2d(planes, planes, kernel_size=3, stride=stride,\n padding=1, bias=True, groups=groups)\n self.bn3 = nn.BatchNorm2d(planes)\n self.conv3 = nn.Conv2d(planes, planes * 2, kernel_size=1, bias=True, groups=groups)\n self.leakyrelu = nn.LeakyReLU(inplace=True) # negative_slope=0.01\n self.skip = skip\n self.stride = stride\n\n def forward(self, x):\n residual = x\n\n out = self.bn1(x)\n out = self.leakyrelu(out)\n out = self.conv1(out)\n\n out = self.bn2(out)\n out = self.leakyrelu(out)\n out = self.conv2(out)\n\n out = self.bn3(out)\n out = self.leakyrelu(out)\n out = self.conv3(out)\n\n if self.skip is not None:\n residual = self.skip(x)\n\n out += residual\n\n return out\n\n\nclass Hourglass(nn.Module):\n def __init__(self, block, num_blocks, planes, depth):\n\n super(Hourglass, self).__init__()\n self.depth = depth\n self.block = block\n self.hg = self._make_hour_glass(block, num_blocks, planes, depth)\n\n def _make_residual(self, block, num_blocks, planes):\n\n layers = []\n for i in range(0, num_blocks):\n # channel changes: planes*block.expansion->planes->2*planes\n layers.append(block(planes * block.expansion, planes))\n return nn.Sequential(*layers)\n\n def _make_hour_glass(self, block, num_blocks, planes, depth):\n hg = []\n for i in range(depth):\n res = []\n for j in range(3):\n # 3 residual modules composed of a residual unit\n # <2*planes><2*planes>\n res.append(self._make_residual(block, num_blocks, planes))\n if i == 0:\n # i=0 in a recursive construction build the basic network path\n # see: low2 = self.hg[n-1][3](low1)\n # <2*planes><2*planes>\n res.append(self._make_residual(block, num_blocks, planes))\n hg.append(nn.ModuleList(res))\n return nn.ModuleList(hg)\n\n def _hour_glass_forward(self, n, x):\n up1 = self.hg[n - 1][0](x) # skip branches\n low1 = F.max_pool2d(x, 2, stride=2)\n low1 = self.hg[n - 1][1](low1)\n\n if n > 1:\n low2 = self._hour_glass_forward(n - 1, low1)\n else:\n low2 = self.hg[n - 1][3](low1) # only for depth=1 basic path of the hourglass network\n low3 = self.hg[n - 1][2](low2)\n up2 = F.interpolate(low3, scale_factor=2) # scale_factor=2 should be consistent with F.max_pool2d(2,stride=2)\n out = up1 + up2\n return out\n\n def forward(self, x):\n # depth: order of the hourglass network\n # do network forward recursively\n return self._hour_glass_forward(self.depth, x)\n\n\nclass hand_Encoder(nn.Module):\n def __init__(self, num_heatmap_chan=21, num_feat_chan=256, size_input_feature=(32, 32),\n nRegBlock=4, nRegModules=2):\n super(hand_Encoder, self).__init__()\n\n self.num_heatmap_chan = num_heatmap_chan\n self.num_feat_chan = num_feat_chan\n self.size_input_feature = size_input_feature\n\n self.nRegBlock = nRegBlock\n self.nRegModules = nRegModules\n\n self.heatmap_conv = nn.Conv2d(self.num_heatmap_chan, self.num_feat_chan,\n bias=True, kernel_size=1, stride=1)\n self.encoding_conv = nn.Conv2d(self.num_feat_chan, self.num_feat_chan,\n bias=True, kernel_size=1, stride=1)\n\n reg = []\n for i in range(self.nRegBlock):\n for j in range(self.nRegModules):\n reg.append(Residual(self.num_feat_chan, self.num_feat_chan))\n\n self.reg = nn.ModuleList(reg)\n self.maxpool = nn.MaxPool2d(kernel_size=2, stride=2)\n self.downsample_scale = 2 ** self.nRegBlock\n\n # fc layers\n self.num_feat_out = self.num_feat_chan * (size_input_feature[0] * size_input_feature[1] // (self.downsample_scale ** 2))\n\n def forward(self, hm_list, encoding_list):\n x = self.heatmap_conv(hm_list[-1]) + self.encoding_conv(encoding_list[-1])\n if len(encoding_list) > 1:\n x = x + encoding_list[-2]\n\n # x: B x num_feat_chan x 32 x 32\n for i in range(self.nRegBlock):\n for j in range(self.nRegModules):\n x = self.reg[i * self.nRegModules + j](x)\n x = self.maxpool(x)\n\n # x: B x num_feat_chan x 2 x 2\n out = x.view(x.size(0), -1)\n\n return out\n" }, { "path": "common/nets/mano_head.py", "content": "import torch\nfrom torch import nn\nfrom torch.nn import functional as F\nfrom utils.mano import MANO\nmano = MANO()\n\ndef batch_rodrigues(theta):\n # theta N x 3\n l1norm = torch.norm(theta + 1e-8, p=2, dim=1)\n angle = torch.unsqueeze(l1norm, -1)\n normalized = torch.div(theta, angle)\n angle = angle * 0.5\n v_cos = torch.cos(angle)\n v_sin = torch.sin(angle)\n quat = torch.cat([v_cos, v_sin * normalized], dim=1)\n\n return quat2mat(quat)\n\n\ndef quat2mat(quat):\n \"\"\"Convert quaternion coefficients to rotation matrix.\n \"\"\"\n norm_quat = quat\n norm_quat = norm_quat / norm_quat.norm(p=2, dim=1, keepdim=True)\n w, x, y, z = norm_quat[:, 0], norm_quat[:, 1], norm_quat[:, 2], norm_quat[:, 3]\n\n B = quat.size(0)\n\n w2, x2, y2, z2 = w.pow(2), x.pow(2), y.pow(2), z.pow(2)\n wx, wy, wz = w * x, w * y, w * z\n xy, xz, yz = x * y, x * z, y * z\n\n rotMat = torch.stack([w2 + x2 - y2 - z2, 2 * xy - 2 * wz, 2 * wy + 2 * xz,\n 2 * wz + 2 * xy, w2 - x2 + y2 - z2, 2 * yz - 2 * wx,\n 2 * xz - 2 * wy, 2 * wx + 2 * yz, w2 - x2 - y2 + z2], dim=1).view(B, 3, 3)\n return rotMat\n\n\ndef quat2aa(quaternion):\n \"\"\"Convert quaternion vector to angle axis of rotation.\"\"\"\n if not torch.is_tensor(quaternion):\n raise TypeError(\"Input type is not a torch.Tensor. Got {}\".format(\n type(quaternion)))\n\n if not quaternion.shape[-1] == 4:\n raise ValueError(\"Input must be a tensor of shape Nx4 or 4. Got {}\"\n .format(quaternion.shape))\n # unpack input and compute conversion\n q1 = quaternion[..., 1]\n q2 = quaternion[..., 2]\n q3 = quaternion[..., 3]\n sin_squared_theta = q1 * q1 + q2 * q2 + q3 * q3\n\n sin_theta = torch.sqrt(sin_squared_theta)\n cos_theta = quaternion[..., 0]\n two_theta = 2.0 * torch.where(\n cos_theta < 0.0,\n torch.atan2(-sin_theta, -cos_theta),\n torch.atan2(sin_theta, cos_theta))\n\n k_pos = two_theta / sin_theta\n k_neg = 2.0 * torch.ones_like(sin_theta)\n k = torch.where(sin_squared_theta > 0.0, k_pos, k_neg)\n\n angle_axis = torch.zeros_like(quaternion)[..., :3]\n angle_axis[..., 0] += q1 * k\n angle_axis[..., 1] += q2 * k\n angle_axis[..., 2] += q3 * k\n return angle_axis\n\n\ndef mat2quat(rotation_matrix, eps=1e-6):\n \"\"\"Convert 3x4 rotation matrix to 4d quaternion vector\"\"\"\n if not torch.is_tensor(rotation_matrix):\n raise TypeError(\"Input type is not a torch.Tensor. Got {}\".format(\n type(rotation_matrix)))\n\n if len(rotation_matrix.shape) > 3:\n raise ValueError(\n \"Input size must be a three dimensional tensor. Got {}\".format(\n rotation_matrix.shape))\n if not rotation_matrix.shape[-2:] == (3, 4):\n raise ValueError(\n \"Input size must be a N x 3 x 4 tensor. Got {}\".format(\n rotation_matrix.shape))\n\n rmat_t = torch.transpose(rotation_matrix, 1, 2)\n\n mask_d2 = rmat_t[:, 2, 2] < eps\n\n mask_d0_d1 = rmat_t[:, 0, 0] > rmat_t[:, 1, 1]\n mask_d0_nd1 = rmat_t[:, 0, 0] < -rmat_t[:, 1, 1]\n\n t0 = 1 + rmat_t[:, 0, 0] - rmat_t[:, 1, 1] - rmat_t[:, 2, 2]\n q0 = torch.stack([rmat_t[:, 1, 2] - rmat_t[:, 2, 1],\n t0, rmat_t[:, 0, 1] + rmat_t[:, 1, 0],\n rmat_t[:, 2, 0] + rmat_t[:, 0, 2]], -1)\n t0_rep = t0.repeat(4, 1).t()\n\n t1 = 1 - rmat_t[:, 0, 0] + rmat_t[:, 1, 1] - rmat_t[:, 2, 2]\n q1 = torch.stack([rmat_t[:, 2, 0] - rmat_t[:, 0, 2],\n rmat_t[:, 0, 1] + rmat_t[:, 1, 0],\n t1, rmat_t[:, 1, 2] + rmat_t[:, 2, 1]], -1)\n t1_rep = t1.repeat(4, 1).t()\n\n t2 = 1 - rmat_t[:, 0, 0] - rmat_t[:, 1, 1] + rmat_t[:, 2, 2]\n q2 = torch.stack([rmat_t[:, 0, 1] - rmat_t[:, 1, 0],\n rmat_t[:, 2, 0] + rmat_t[:, 0, 2],\n rmat_t[:, 1, 2] + rmat_t[:, 2, 1], t2], -1)\n t2_rep = t2.repeat(4, 1).t()\n\n t3 = 1 + rmat_t[:, 0, 0] + rmat_t[:, 1, 1] + rmat_t[:, 2, 2]\n q3 = torch.stack([t3, rmat_t[:, 1, 2] - rmat_t[:, 2, 1],\n rmat_t[:, 2, 0] - rmat_t[:, 0, 2],\n rmat_t[:, 0, 1] - rmat_t[:, 1, 0]], -1)\n t3_rep = t3.repeat(4, 1).t()\n\n mask_c0 = mask_d2 * mask_d0_d1\n mask_c1 = mask_d2 * ~mask_d0_d1\n mask_c2 = ~mask_d2 * mask_d0_nd1\n mask_c3 = ~mask_d2 * ~mask_d0_nd1\n mask_c0 = mask_c0.view(-1, 1).type_as(q0)\n mask_c1 = mask_c1.view(-1, 1).type_as(q1)\n mask_c2 = mask_c2.view(-1, 1).type_as(q2)\n mask_c3 = mask_c3.view(-1, 1).type_as(q3)\n\n q = q0 * mask_c0 + q1 * mask_c1 + q2 * mask_c2 + q3 * mask_c3\n q /= torch.sqrt(t0_rep * mask_c0 + t1_rep * mask_c1 + # noqa\n t2_rep * mask_c2 + t3_rep * mask_c3) # noqa\n q *= 0.5\n return q\n\n\ndef rot6d2mat(x):\n \"\"\"Convert 6D rotation representation to 3x3 rotation matrix.\n Based on Zhou et al., \"On the Continuity of Rotation Representations in Neural Networks\", CVPR 2019\n \"\"\"\n a1 = x[:, 0:3]\n a2 = x[:, 3:6]\n b1 = F.normalize(a1)\n b2 = F.normalize(a2 - torch.einsum('bi,bi->b', b1, a2).unsqueeze(-1) * b1)\n b3 = torch.cross(b1, b2, dim=1)\n return torch.stack((b1, b2, b3), dim=-1)\n\n\ndef mat2aa(rotation_matrix):\n \"\"\"Convert 3x4 rotation matrix to Rodrigues vector\"\"\"\n\n def convert_points_to_homogeneous(points):\n if not torch.is_tensor(points):\n raise TypeError(\"Input type is not a torch.Tensor. Got {}\".format(\n type(points)))\n if len(points.shape) < 2:\n raise ValueError(\"Input must be at least a 2D tensor. Got {}\".format(\n points.shape))\n\n return F.pad(points, (0, 1), \"constant\", 1.0)\n\n if rotation_matrix.shape[1:] == (3, 3):\n rotation_matrix = convert_points_to_homogeneous(rotation_matrix)\n quaternion = mat2quat(rotation_matrix)\n aa = quat2aa(quaternion)\n aa[torch.isnan(aa)] = 0.0\n return aa\n\n\nclass mano_regHead(nn.Module):\n def __init__(self, mano_layer=mano.layer, feature_size=1024, mano_neurons=[1024, 512]):\n super(mano_regHead, self).__init__()\n\n # 6D representation of rotation matrix\n self.pose6d_size = 16 * 6\n self.mano_pose_size = 16 * 3\n\n # Base Regression Layers\n mano_base_neurons = [feature_size] + mano_neurons\n base_layers = []\n for layer_idx, (inp_neurons, out_neurons) in enumerate(\n zip(mano_base_neurons[:-1], mano_base_neurons[1:])):\n base_layers.append(nn.Linear(inp_neurons, out_neurons))\n base_layers.append(nn.LeakyReLU(inplace=True))\n self.mano_base_layer = nn.Sequential(*base_layers)\n # Pose layers\n self.pose_reg = nn.Linear(mano_base_neurons[-1], self.pose6d_size)\n # Shape layers\n self.shape_reg = nn.Linear(mano_base_neurons[-1], 10)\n\n self.mano_layer = mano_layer\n\n def forward(self, features, gt_mano_params=None):\n mano_features = self.mano_base_layer(features)\n pred_mano_pose_6d = self.pose_reg(mano_features)\n \n pred_mano_pose_rotmat = rot6d2mat(pred_mano_pose_6d.view(-1, 6)).view(-1, 16, 3, 3).contiguous()\n pred_mano_shape = self.shape_reg(mano_features)\n pred_mano_pose = mat2aa(pred_mano_pose_rotmat.view(-1, 3, 3)).contiguous().view(-1, self.mano_pose_size)\n pred_verts, pred_joints, pred_manojoints2cam = self.mano_layer(th_pose_coeffs=pred_mano_pose, th_betas=pred_mano_shape)\n\n pred_verts /= 1000\n pred_joints /= 1000\n\n pred_mano_results = {\n \"verts3d\": pred_verts,\n \"joints3d\": pred_joints,\n \"mano_shape\": pred_mano_shape,\n \"mano_pose\": pred_mano_pose_rotmat,\n \"mano_pose_aa\": pred_mano_pose,\n \"manojoints2cam\": pred_manojoints2cam \n }\n\n if gt_mano_params is not None:\n gt_mano_shape = gt_mano_params[:, self.mano_pose_size:]\n gt_mano_pose = gt_mano_params[:, :self.mano_pose_size].contiguous()\n gt_mano_pose_rotmat = batch_rodrigues(gt_mano_pose.view(-1, 3)).view(-1, 16, 3, 3)\n gt_verts, gt_joints = self.mano_layer(th_pose_coeffs=gt_mano_pose, th_betas=gt_mano_shape)\n\n gt_verts /= 1000\n gt_joints /= 1000\n\n gt_mano_results = {\n \"verts3d\": gt_verts,\n \"joints3d\": gt_joints,\n \"mano_shape\": gt_mano_shape,\n \"mano_pose\": gt_mano_pose_rotmat}\n else:\n gt_mano_results = None\n\n return pred_mano_results, gt_mano_results\n" }, { "path": "common/nets/regressor.py", "content": "import torch\nimport torch.nn as nn\nfrom torch.nn import functional as F\nfrom utils.mano import MANO\nfrom nets.hand_head import hand_regHead, hand_Encoder\nfrom nets.mano_head import mano_regHead\n\nclass Regressor(nn.Module):\n def __init__(self):\n super(Regressor, self).__init__()\n self.hand_regHead = hand_regHead()\n self.hand_Encoder = hand_Encoder()\n self.mano_regHead = mano_regHead()\n \n def forward(self, feats, gt_mano_params=None):\n out_hm, encoding, preds_joints_img = self.hand_regHead(feats)\n mano_encoding = self.hand_Encoder(out_hm, encoding)\n pred_mano_results, gt_mano_results = self.mano_regHead(mano_encoding, gt_mano_params)\n\n return pred_mano_results, gt_mano_results, preds_joints_img\n" }, { "path": "common/nets/transformer.py", "content": "import torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nfrom einops import repeat\n\nclass Transformer(nn.Module):\n def __init__(self, inp_res=32, dim=256, depth=2, num_heads=4, mlp_ratio=4., injection=True):\n super().__init__()\n\n self.injection=injection\n \n self.layers = nn.ModuleList([])\n for _ in range(depth):\n self.layers.append(Block(dim=dim, num_heads=num_heads, mlp_ratio=mlp_ratio, injection=injection))\n\n if self.injection:\n self.conv1 = nn.Sequential(\n nn.Conv2d(dim*2, dim, 3, padding=1),\n nn.ReLU(),\n nn.Conv2d(dim, dim, 3, padding=1),\n )\n self.conv2 = nn.Sequential(\n nn.Conv2d(dim*2, dim, 1, padding=0),\n )\n\n def forward(self, query, key):\n output = query\n for i, layer in enumerate(self.layers):\n output = layer(query=output, key=key)\n \n if self.injection:\n output = torch.cat([key, output], dim=1)\n output = self.conv1(output) + self.conv2(output)\n\n return output\n\nclass Mlp(nn.Module):\n def __init__(self, in_features, hidden_features=None, out_features=None, act_layer=nn.GELU, drop=0.):\n super().__init__()\n out_features = out_features or in_features\n hidden_features = hidden_features or in_features\n self.fc1 = nn.Linear(in_features, hidden_features)\n self.act = act_layer()\n self.fc2 = nn.Linear(hidden_features, out_features)\n self.drop = nn.Dropout(drop)\n self._init_weights()\n\n def forward(self, x):\n x = self.fc1(x)\n x = self.act(x)\n x = self.drop(x)\n x = self.fc2(x)\n x = self.drop(x)\n return x\n\n def _init_weights(self):\n nn.init.xavier_uniform_(self.fc1.weight)\n nn.init.xavier_uniform_(self.fc2.weight)\n nn.init.normal_(self.fc1.bias, std=1e-6)\n nn.init.normal_(self.fc2.bias, std=1e-6)\n\n\nclass Attention(nn.Module):\n def __init__(self, dim, num_heads=1):\n super().__init__()\n self.num_heads = num_heads\n head_dim = dim // num_heads\n self.scale = head_dim ** -0.5\n self.sigmoid = nn.Sigmoid()\n \n def forward(self, query, key, value, query2, key2, use_sigmoid):\n B, N, C = query.shape\n query = query.reshape(B, N, self.num_heads, C // self.num_heads).permute(0, 2, 1, 3)\n key = key.reshape(B, N, self.num_heads, C // self.num_heads).permute(0, 2, 1, 3)\n value = value.reshape(B, N, self.num_heads, C // self.num_heads).permute(0, 2, 1, 3)\n attn = torch.matmul(query, key.transpose(-2, -1)) * self.scale\n attn = attn.softmax(dim=-1)\n \n if use_sigmoid:\n query2 = query2.reshape(B, N, self.num_heads, C // self.num_heads).permute(0, 2, 1, 3)\n key2 = key2.reshape(B, N, self.num_heads, C // self.num_heads).permute(0, 2, 1, 3)\n attn2 = torch.matmul(query2, key2.transpose(-2, -1)) * self.scale\n attn2 = torch.sum(attn2, dim=-1)\n attn2 = self.sigmoid(attn2)\n attn = attn * attn2.unsqueeze(3) \n \n x = torch.matmul(attn, value).transpose(1, 2).reshape(B, N, C)\n return x\n\nclass Block(nn.Module):\n\n def __init__(self, dim, num_heads, mlp_ratio=4., act_layer=nn.GELU, norm_layer=nn.LayerNorm, injection=True):\n super().__init__()\n\n self.injection = injection\n\n self.channels = dim\n\n self.encode_value = nn.Conv2d(in_channels=dim, out_channels=dim, kernel_size=1, stride=1, padding=0)\n self.encode_query = nn.Conv2d(in_channels=dim, out_channels=dim, kernel_size=1, stride=1, padding=0)\n self.encode_key = nn.Conv2d(in_channels=dim, out_channels=dim, kernel_size=1, stride=1, padding=0)\n\n if self.injection:\n self.encode_query2 = nn.Conv2d(in_channels=dim, out_channels=dim, kernel_size=1, stride=1, padding=0)\n self.encode_key2 = nn.Conv2d(in_channels=dim, out_channels=dim, kernel_size=1, stride=1, padding=0)\n\n self.attn = Attention(dim, num_heads=num_heads)\n self.norm2 = norm_layer(dim)\n mlp_hidden_dim = int(dim * mlp_ratio)\n self.mlp = Mlp(in_features=dim, hidden_features=mlp_hidden_dim, act_layer=act_layer)\n self.q_embedding = nn.Parameter(torch.randn(1, 256, 32, 32))\n self.k_embedding = nn.Parameter(torch.randn(1, 256, 32, 32))\n\n def with_pos_embed(self, tensor, pos):\n return tensor if pos is None else tensor + pos\n\n def forward(self, query, key, query_embed=None, key_embed=None):\n b, c, h, w = query.shape\n query_embed = repeat(self.q_embedding, '() n c d -> b n c d', b = b)\n key_embed = repeat(self.k_embedding, '() n c d -> b n c d', b = b)\n\n q_embed = self.with_pos_embed(query, query_embed)\n k_embed = self.with_pos_embed(key, key_embed)\n\n v = self.encode_value(key).view(b, self.channels, -1)\n v = v.permute(0, 2, 1)\n\n q = self.encode_query(q_embed).view(b, self.channels, -1)\n q = q.permute(0, 2, 1)\n\n k = self.encode_key(k_embed).view(b, self.channels, -1)\n k = k.permute(0, 2, 1)\n \n query = query.view(b, self.channels, -1).permute(0, 2, 1)\n\n if self.injection:\n q2 = self.encode_query2(q_embed).view(b, self.channels, -1)\n q2 = q2.permute(0, 2, 1)\n\n k2 = self.encode_key2(k_embed).view(b, self.channels, -1)\n k2 = k2.permute(0, 2, 1)\n\n query = self.attn(query=q, key=k, value=v,query2 = q2, key2 = k2, use_sigmoid=True)\n else:\n q2 = None\n k2 = None\n\n query = query + self.attn(query=q, key=k, value=v, query2 = q2, key2 = k2, use_sigmoid=False)\n \n query = query + self.mlp(self.norm2(query))\n query = query.permute(0, 2, 1).contiguous().view(b, self.channels, h, w)\n\n return query\n" }, { "path": "common/timer.py", "content": "# --------------------------------------------------------\n# Fast R-CNN\n# Copyright (c) 2015 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Written by Ross Girshick\n# --------------------------------------------------------\n\nimport time\n\nclass Timer(object):\n \"\"\"A simple timer.\"\"\"\n def __init__(self):\n self.total_time = 0.\n self.calls = 0\n self.start_time = 0.\n self.diff = 0.\n self.average_time = 0.\n self.warm_up = 0\n\n def tic(self):\n # using time.time instead of time.clock because time time.clock\n # does not normalize for multithreading\n self.start_time = time.time()\n\n def toc(self, average=True):\n self.diff = time.time() - self.start_time\n if self.warm_up < 10:\n self.warm_up += 1\n return self.diff\n else:\n self.total_time += self.diff\n self.calls += 1\n self.average_time = self.total_time / self.calls\n\n if average:\n return self.average_time\n else:\n return self.diff" }, { "path": "common/utils/__init__.py", "content": "" }, { "path": "common/utils/camera.py", "content": "# -*- coding: utf-8 -*-\n\n# Max-Planck-Gesellschaft zur Förderung der Wissenschaften e.V. (MPG) is\n# holder of all proprietary rights on this computer program.\n# You can only use this computer program if you have closed\n# a license agreement with MPG or you get the right to use the computer\n# program from someone who is authorized to grant you that right.\n# Any use of the computer program without a valid license is prohibited and\n# liable to prosecution.\n#\n# Copyright©2019 Max-Planck-Gesellschaft zur Förderung\n# der Wissenschaften e.V. (MPG). acting on behalf of its Max Planck Institute\n# for Intelligent Systems and the Max Planck Institute for Biological\n# Cybernetics. All rights reserved.\n#\n# Contact: ps-license@tuebingen.mpg.de\n\nfrom __future__ import absolute_import\nfrom __future__ import print_function\nfrom __future__ import division\n\nfrom collections import namedtuple\n\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\n\n\nPerspParams = namedtuple('ModelOutput',\n ['rotation', 'translation', 'center',\n 'focal_length'])\n\ndef transform_mat(R: torch.tensor, t: torch.tensor) -> torch.Tensor:\n ''' Creates a batch of transformation matrices\n Args:\n - R: Bx3x3 array of a batch of rotation matrices\n - t: Bx3x1 array of a batch of translation vectors\n Returns:\n - T: Bx4x4 Transformation matrix\n '''\n # No padding left or right, only add an extra row\n return torch.cat([F.pad(R, [0, 0, 0, 1]),\n F.pad(t, [0, 0, 0, 1], value=1)], dim=2)\n\n\n\ndef create_camera(camera_type='persp', **kwargs):\n if camera_type.lower() == 'persp':\n return PerspectiveCamera(**kwargs)\n else:\n raise ValueError('Uknown camera type: {}'.format(camera_type))\n\n\nclass PerspectiveCamera(nn.Module):\n\n FOCAL_LENGTH = 500\n\n def __init__(self, rotation=None, translation=None,\n focal_length_x=None, focal_length_y=None,\n batch_size=1,\n center=None, dtype=torch.float32, **kwargs):\n super(PerspectiveCamera, self).__init__()\n self.name = ''\n self.batch_size = batch_size\n self.dtype = dtype\n # Make a buffer so that PyTorch does not complain when creating\n # the camera matrix\n self.register_buffer('zero',\n torch.zeros([batch_size], dtype=dtype))\n\n if focal_length_x is None or type(focal_length_x) == float:\n focal_length_x = torch.full(\n [batch_size],\n self.FOCAL_LENGTH if focal_length_x is None else\n focal_length_x,\n dtype=dtype)\n\n if focal_length_y is None or type(focal_length_y) == float:\n focal_length_y = torch.full(\n [batch_size],\n self.FOCAL_LENGTH if focal_length_y is None else\n focal_length_y,\n dtype=dtype)\n\n self.register_buffer('focal_length_x', focal_length_x)\n self.register_buffer('focal_length_y', focal_length_y)\n\n if center is None:\n center = torch.zeros([batch_size, 2], dtype=dtype)\n self.register_buffer('center', center)\n\n if rotation is None:\n rotation = torch.eye(\n 3, dtype=dtype).unsqueeze(dim=0).repeat(batch_size, 1, 1)\n\n rotation = nn.Parameter(rotation, requires_grad=True)\n self.register_parameter('rotation', rotation)\n\n if translation is None:\n translation = torch.zeros([batch_size, 3], dtype=dtype)\n\n translation = nn.Parameter(translation,\n requires_grad=True)\n self.register_parameter('translation', translation)\n\n def forward(self, points):\n device = points.device\n\n with torch.no_grad():\n camera_mat = torch.zeros([self.batch_size, 2, 2],\n dtype=self.dtype, device=points.device)\n camera_mat[:, 0, 0] = self.focal_length_x\n camera_mat[:, 1, 1] = self.focal_length_y\n\n camera_transform = transform_mat(self.rotation,\n self.translation.unsqueeze(dim=-1))\n homog_coord = torch.ones(list(points.shape)[:-1] + [1],\n dtype=points.dtype,\n device=device)\n # Convert the points to homogeneous coordinates\n points_h = torch.cat([points, homog_coord], dim=-1)\n\n projected_points = torch.einsum('bki,bji->bjk',\n [camera_transform, points_h])\n\n img_points = torch.div(projected_points[:, :, :2],\n projected_points[:, :, 2].unsqueeze(dim=-1))\n img_points = torch.einsum('bki,bji->bjk', [camera_mat, img_points]) \\\n + self.center.unsqueeze(dim=1)\n return img_points" }, { "path": "common/utils/dir.py", "content": "import os\nimport sys\n\ndef make_folder(folder_name):\n if not os.path.exists(folder_name):\n os.makedirs(folder_name)\n\ndef add_pypath(path):\n if path not in sys.path:\n sys.path.insert(0, path)" }, { "path": "common/utils/fitting.py", "content": "import numpy as np\nimport torch\nimport torch.nn as nn\n\n\n\ndef to_tensor(tensor, dtype=torch.float32):\n if torch.Tensor == type(tensor):\n return tensor.clone().detach()\n else:\n return torch.tensor(tensor, dtype=dtype)\n\n\ndef rel_change(prev_val, curr_val):\n return (prev_val - curr_val) / max([np.abs(prev_val), np.abs(curr_val), 1])\n\nclass FittingMonitor(object):\n def __init__(self, summary_steps=1, visualize=False,\n maxiters=300, ftol=1e-10, gtol=1e-09,\n body_color=(1.0, 1.0, 0.9, 1.0),\n model_type='mano',\n **kwargs):\n super(FittingMonitor, self).__init__()\n\n self.maxiters = maxiters\n self.ftol = ftol\n self.gtol = gtol\n\n self.visualize = visualize\n self.summary_steps = summary_steps\n self.body_color = body_color\n self.model_type = model_type\n\n def __enter__(self):\n self.steps = 0\n\n return self\n \n def __exit__(self, exception_type, exception_value, traceback):\n pass\n \n def set_colors(self, vertex_color):\n batch_size = self.colors.shape[0]\n\n self.colors = np.tile(\n np.array(vertex_color).reshape(1, 3),\n [batch_size, 1])\n\n def run_fitting(self, optimizer, closure, params,\n **kwargs):\n ''' Helper function for running an optimization process\n Parameters\n ----------\n optimizer: torch.optim.Optimizer\n The PyTorch optimizer object\n closure: function\n The function used to calculate the gradients\n params: list\n List containing the parameters that will be optimized\n\n Returns\n -------\n loss: float\n The final loss value\n '''\n prev_loss = None\n for n in range(self.maxiters):\n loss = optimizer.step(closure)\n\n if torch.isnan(loss).sum() > 0:\n print('NaN loss value, stopping!')\n break\n\n if torch.isinf(loss).sum() > 0:\n print('Infinite loss value, stopping!')\n break\n\n if n > 0 and prev_loss is not None and self.ftol > 0:\n loss_rel_change = rel_change(prev_loss, loss.item())\n\n if loss_rel_change <= self.ftol:\n break\n\n if all([torch.abs(var.grad.view(-1).max()).item() < self.gtol\n for var in params if var.grad is not None]):\n break\n \n prev_loss = loss.item()\n \n return prev_loss\n\n def create_fitting_closure(self,\n optimizer, \n camera=None,\n joint_cam=None, \n joint_img=None,\n hand_translation=None,\n hand_scale=None,\n loss=None,\n joints_conf=None,\n joint_weights=None,\n create_graph=False,\n **kwargs):\n\n def fitting_func(backward=True):\n if backward:\n optimizer.zero_grad()\n\n total_loss = loss(camera=camera,\n joint_cam=joint_cam,\n joint_img=joint_img,\n hand_translation=hand_translation,\n hand_scale =hand_scale,\n **kwargs)\n\n if backward:\n total_loss.backward(create_graph=create_graph)\n\n self.steps += 1\n\n return total_loss\n\n return fitting_func\n\n\n\n\nclass ScaleTranslationLoss(nn.Module):\n\n def __init__(self, init_joints_idxs, trans_estimation=None,\n reduction='sum',\n data_weight=1.0,\n depth_loss_weight=1e3, dtype=torch.float32,\n **kwargs):\n super(ScaleTranslationLoss, self).__init__()\n self.dtype = dtype\n\n if trans_estimation is not None:\n self.register_buffer(\n 'trans_estimation',\n to_tensor(trans_estimation, dtype=dtype))\n else:\n self.trans_estimation = trans_estimation\n\n self.register_buffer('data_weight',\n torch.tensor(data_weight, dtype=dtype))\n self.register_buffer(\n 'init_joints_idxs',\n to_tensor(init_joints_idxs, dtype=torch.long))\n self.register_buffer('depth_loss_weight',\n torch.tensor(depth_loss_weight, dtype=dtype))\n\n def reset_loss_weights(self, loss_weight_dict):\n for key in loss_weight_dict:\n if hasattr(self, key):\n weight_tensor = getattr(self, key)\n weight_tensor = torch.tensor(loss_weight_dict[key],\n dtype=weight_tensor.dtype,\n device=weight_tensor.device)\n setattr(self, key, weight_tensor)\n\n def forward(self, camera, joint_cam, joint_img, hand_translation, hand_scale, **kwargs):\n\n projected_joints = camera(\n hand_scale * joint_cam + hand_translation)\n \n joint_error = \\\n torch.index_select(joint_img, 1, self.init_joints_idxs) - \\\n torch.index_select(projected_joints, 1, self.init_joints_idxs)\n joint_loss = torch.sum(joint_error.abs()) * self.data_weight ** 2\n\n depth_loss = 0.0\n if (self.depth_loss_weight.item() > 0 and self.trans_estimation is not None):\n depth_loss = self.depth_loss_weight * torch.sum((\n hand_translation[2] - self.trans_estimation[2]).abs() ** 2)\n\n return joint_loss + depth_loss" }, { "path": "common/utils/mano.py", "content": "import numpy as np\nimport torch\nimport os.path as osp\nimport json\nfrom config import cfg\n\nimport sys\nsys.path.insert(0, cfg.mano_path)\nimport manopth\nfrom manopth.manolayer import ManoLayer\n\nclass MANO(object):\n def __init__(self):\n # TEMP\n self.left_layer = ManoLayer(mano_root=osp.join(cfg.mano_path, 'mano', 'models'), flat_hand_mean=False, use_pca=False, side='left') # load right hand MANO model\n self.layer = self.get_layer()\n self.vertex_num = 778\n self.face = self.layer.th_faces.numpy()\n self.joint_regressor = self.layer.th_J_regressor.numpy()\n\n self.joint_num = 21\n self.joints_name = ('Wrist', 'Thumb_1', 'Thumb_2', 'Thumb_3', 'Thumb_4', 'Index_1', 'Index_2', 'Index_3', 'Index_4', 'Middle_1', 'Middle_2', 'Middle_3', 'Middle_4', 'Ring_1', 'Ring_2', 'Ring_3', 'Ring_4', 'Pinky_1', 'Pinky_2', 'Pinky_3', 'Pinly_4')\n self.skeleton = ( (0,1), (0,5), (0,9), (0,13), (0,17), (1,2), (2,3), (3,4), (5,6), (6,7), (7,8), (9,10), (10,11), (11,12), (13,14), (14,15), (15,16), (17,18), (18,19), (19,20) )\n self.root_joint_idx = self.joints_name.index('Wrist')\n\n # add fingertips to joint_regressor\n self.fingertip_vertex_idx = [728, 353, 442, 576, 694] # mesh vertex idx\n\n thumbtip_onehot = np.array([1 if i == 728 else 0 for i in range(self.joint_regressor.shape[1])], dtype=np.float32).reshape(1,-1)\n indextip_onehot = np.array([1 if i == 353 else 0 for i in range(self.joint_regressor.shape[1])], dtype=np.float32).reshape(1,-1)\n middletip_onehot = np.array([1 if i == 442 else 0 for i in range(self.joint_regressor.shape[1])], dtype=np.float32).reshape(1,-1)\n ringtip_onehot = np.array([1 if i == 576 else 0 for i in range(self.joint_regressor.shape[1])], dtype=np.float32).reshape(1,-1)\n pinkytip_onehot = np.array([1 if i == 694 else 0 for i in range(self.joint_regressor.shape[1])], dtype=np.float32).reshape(1,-1)\n\n self.joint_regressor = np.concatenate((self.joint_regressor, thumbtip_onehot, indextip_onehot, middletip_onehot, ringtip_onehot, pinkytip_onehot))\n self.joint_regressor = self.joint_regressor[[0, 13, 14, 15, 16, 1, 2, 3, 17, 4, 5, 6, 18, 10, 11, 12, 19, 7, 8, 9, 20],:]\n\n def get_layer(self):\n return ManoLayer(mano_root=osp.join(cfg.mano_path, 'mano', 'models'), flat_hand_mean=False, use_pca=False, side='right') # load right hand MANO model" }, { "path": "common/utils/manopth/.gitignore", "content": "*.sw*\n*.bak\n*_bak.py\n\n.cache/\n__pycache__/\nbuild/\ndist/\nmanopth_hassony2.egg-info/\n\nmano/models\nassets/mano_layer.svg\n" }, { "path": "common/utils/manopth/LICENSE", "content": " GNU GENERAL PUBLIC LICENSE\n Version 3, 29 June 2007\n\n Copyright (C) 2007 Free Software Foundation, Inc. \n Everyone is permitted to copy and distribute verbatim copies\n of this license document, but changing it is not allowed.\n\n Preamble\n\n The GNU General Public License is a free, copyleft license for\nsoftware and other kinds of works.\n\n The licenses for most software and other practical works are designed\nto take away your freedom to share and change the works. By contrast,\nthe GNU General Public License is intended to guarantee your freedom to\nshare and change all versions of a program--to make sure it remains free\nsoftware for all its users. We, the Free Software Foundation, use the\nGNU General Public License for most of our software; it applies also to\nany other work released this way by its authors. You can apply it to\nyour programs, too.\n\n When we speak of free software, we are referring to freedom, not\nprice. Our General Public Licenses are designed to make sure that you\nhave the freedom to distribute copies of free software (and charge for\nthem if you wish), that you receive source code or can get it if you\nwant it, that you can change the software or use pieces of it in new\nfree programs, and that you know you can do these things.\n\n To protect your rights, we need to prevent others from denying you\nthese rights or asking you to surrender the rights. Therefore, you have\ncertain responsibilities if you distribute copies of the software, or if\nyou modify it: responsibilities to respect the freedom of others.\n\n For example, if you distribute copies of such a program, whether\ngratis or for a fee, you must pass on to the recipients the same\nfreedoms that you received. You must make sure that they, too, receive\nor can get the source code. And you must show them these terms so they\nknow their rights.\n\n Developers that use the GNU GPL protect your rights with two steps:\n(1) assert copyright on the software, and (2) offer you this License\ngiving you legal permission to copy, distribute and/or modify it.\n\n For the developers' and authors' protection, the GPL clearly explains\nthat there is no warranty for this free software. For both users' and\nauthors' sake, the GPL requires that modified versions be marked as\nchanged, so that their problems will not be attributed erroneously to\nauthors of previous versions.\n\n Some devices are designed to deny users access to install or run\nmodified versions of the software inside them, although the manufacturer\ncan do so. This is fundamentally incompatible with the aim of\nprotecting users' freedom to change the software. The systematic\npattern of such abuse occurs in the area of products for individuals to\nuse, which is precisely where it is most unacceptable. Therefore, we\nhave designed this version of the GPL to prohibit the practice for those\nproducts. If such problems arise substantially in other domains, we\nstand ready to extend this provision to those domains in future versions\nof the GPL, as needed to protect the freedom of users.\n\n Finally, every program is threatened constantly by software patents.\nStates should not allow patents to restrict development and use of\nsoftware on general-purpose computers, but in those that do, we wish to\navoid the special danger that patents applied to a free program could\nmake it effectively proprietary. To prevent this, the GPL assures that\npatents cannot be used to render the program non-free.\n\n The precise terms and conditions for copying, distribution and\nmodification follow.\n\n TERMS AND CONDITIONS\n\n 0. Definitions.\n\n \"This License\" refers to version 3 of the GNU General Public License.\n\n \"Copyright\" also means copyright-like laws that apply to other kinds of\nworks, such as semiconductor masks.\n\n \"The Program\" refers to any copyrightable work licensed under this\nLicense. Each licensee is addressed as \"you\". \"Licensees\" and\n\"recipients\" may be individuals or organizations.\n\n To \"modify\" a work means to copy from or adapt all or part of the work\nin a fashion requiring copyright permission, other than the making of an\nexact copy. The resulting work is called a \"modified version\" of the\nearlier work or a work \"based on\" the earlier work.\n\n A \"covered work\" means either the unmodified Program or a work based\non the Program.\n\n To \"propagate\" a work means to do anything with it that, without\npermission, would make you directly or secondarily liable for\ninfringement under applicable copyright law, except executing it on a\ncomputer or modifying a private copy. Propagation includes copying,\ndistribution (with or without modification), making available to the\npublic, and in some countries other activities as well.\n\n To \"convey\" a work means any kind of propagation that enables other\nparties to make or receive copies. Mere interaction with a user through\na computer network, with no transfer of a copy, is not conveying.\n\n An interactive user interface displays \"Appropriate Legal Notices\"\nto the extent that it includes a convenient and prominently visible\nfeature that (1) displays an appropriate copyright notice, and (2)\ntells the user that there is no warranty for the work (except to the\nextent that warranties are provided), that licensees may convey the\nwork under this License, and how to view a copy of this License. If\nthe interface presents a list of user commands or options, such as a\nmenu, a prominent item in the list meets this criterion.\n\n 1. Source Code.\n\n The \"source code\" for a work means the preferred form of the work\nfor making modifications to it. \"Object code\" means any non-source\nform of a work.\n\n A \"Standard Interface\" means an interface that either is an official\nstandard defined by a recognized standards body, or, in the case of\ninterfaces specified for a particular programming language, one that\nis widely used among developers working in that language.\n\n The \"System Libraries\" of an executable work include anything, other\nthan the work as a whole, that (a) is included in the normal form of\npackaging a Major Component, but which is not part of that Major\nComponent, and (b) serves only to enable use of the work with that\nMajor Component, or to implement a Standard Interface for which an\nimplementation is available to the public in source code form. A\n\"Major Component\", in this context, means a major essential component\n(kernel, window system, and so on) of the specific operating system\n(if any) on which the executable work runs, or a compiler used to\nproduce the work, or an object code interpreter used to run it.\n\n The \"Corresponding Source\" for a work in object code form means all\nthe source code needed to generate, install, and (for an executable\nwork) run the object code and to modify the work, including scripts to\ncontrol those activities. However, it does not include the work's\nSystem Libraries, or general-purpose tools or generally available free\nprograms which are used unmodified in performing those activities but\nwhich are not part of the work. For example, Corresponding Source\nincludes interface definition files associated with source files for\nthe work, and the source code for shared libraries and dynamically\nlinked subprograms that the work is specifically designed to require,\nsuch as by intimate data communication or control flow between those\nsubprograms and other parts of the work.\n\n The Corresponding Source need not include anything that users\ncan regenerate automatically from other parts of the Corresponding\nSource.\n\n The Corresponding Source for a work in source code form is that\nsame work.\n\n 2. Basic Permissions.\n\n All rights granted under this License are granted for the term of\ncopyright on the Program, and are irrevocable provided the stated\nconditions are met. This License explicitly affirms your unlimited\npermission to run the unmodified Program. The output from running a\ncovered work is covered by this License only if the output, given its\ncontent, constitutes a covered work. This License acknowledges your\nrights of fair use or other equivalent, as provided by copyright law.\n\n You may make, run and propagate covered works that you do not\nconvey, without conditions so long as your license otherwise remains\nin force. You may convey covered works to others for the sole purpose\nof having them make modifications exclusively for you, or provide you\nwith facilities for running those works, provided that you comply with\nthe terms of this License in conveying all material for which you do\nnot control copyright. Those thus making or running the covered works\nfor you must do so exclusively on your behalf, under your direction\nand control, on terms that prohibit them from making any copies of\nyour copyrighted material outside their relationship with you.\n\n Conveying under any other circumstances is permitted solely under\nthe conditions stated below. Sublicensing is not allowed; section 10\nmakes it unnecessary.\n\n 3. Protecting Users' Legal Rights From Anti-Circumvention Law.\n\n No covered work shall be deemed part of an effective technological\nmeasure under any applicable law fulfilling obligations under article\n11 of the WIPO copyright treaty adopted on 20 December 1996, or\nsimilar laws prohibiting or restricting circumvention of such\nmeasures.\n\n When you convey a covered work, you waive any legal power to forbid\ncircumvention of technological measures to the extent such circumvention\nis effected by exercising rights under this License with respect to\nthe covered work, and you disclaim any intention to limit operation or\nmodification of the work as a means of enforcing, against the work's\nusers, your or third parties' legal rights to forbid circumvention of\ntechnological measures.\n\n 4. Conveying Verbatim Copies.\n\n You may convey verbatim copies of the Program's source code as you\nreceive it, in any medium, provided that you conspicuously and\nappropriately publish on each copy an appropriate copyright notice;\nkeep intact all notices stating that this License and any\nnon-permissive terms added in accord with section 7 apply to the code;\nkeep intact all notices of the absence of any warranty; and give all\nrecipients a copy of this License along with the Program.\n\n You may charge any price or no price for each copy that you convey,\nand you may offer support or warranty protection for a fee.\n\n 5. Conveying Modified Source Versions.\n\n You may convey a work based on the Program, or the modifications to\nproduce it from the Program, in the form of source code under the\nterms of section 4, provided that you also meet all of these conditions:\n\n a) The work must carry prominent notices stating that you modified\n it, and giving a relevant date.\n\n b) The work must carry prominent notices stating that it is\n released under this License and any conditions added under section\n 7. This requirement modifies the requirement in section 4 to\n \"keep intact all notices\".\n\n c) You must license the entire work, as a whole, under this\n License to anyone who comes into possession of a copy. This\n License will therefore apply, along with any applicable section 7\n additional terms, to the whole of the work, and all its parts,\n regardless of how they are packaged. This License gives no\n permission to license the work in any other way, but it does not\n invalidate such permission if you have separately received it.\n\n d) If the work has interactive user interfaces, each must display\n Appropriate Legal Notices; however, if the Program has interactive\n interfaces that do not display Appropriate Legal Notices, your\n work need not make them do so.\n\n A compilation of a covered work with other separate and independent\nworks, which are not by their nature extensions of the covered work,\nand which are not combined with it such as to form a larger program,\nin or on a volume of a storage or distribution medium, is called an\n\"aggregate\" if the compilation and its resulting copyright are not\nused to limit the access or legal rights of the compilation's users\nbeyond what the individual works permit. Inclusion of a covered work\nin an aggregate does not cause this License to apply to the other\nparts of the aggregate.\n\n 6. Conveying Non-Source Forms.\n\n You may convey a covered work in object code form under the terms\nof sections 4 and 5, provided that you also convey the\nmachine-readable Corresponding Source under the terms of this License,\nin one of these ways:\n\n a) Convey the object code in, or embodied in, a physical product\n (including a physical distribution medium), accompanied by the\n Corresponding Source fixed on a durable physical medium\n customarily used for software interchange.\n\n b) Convey the object code in, or embodied in, a physical product\n (including a physical distribution medium), accompanied by a\n written offer, valid for at least three years and valid for as\n long as you offer spare parts or customer support for that product\n model, to give anyone who possesses the object code either (1) a\n copy of the Corresponding Source for all the software in the\n product that is covered by this License, on a durable physical\n medium customarily used for software interchange, for a price no\n more than your reasonable cost of physically performing this\n conveying of source, or (2) access to copy the\n Corresponding Source from a network server at no charge.\n\n c) Convey individual copies of the object code with a copy of the\n written offer to provide the Corresponding Source. This\n alternative is allowed only occasionally and noncommercially, and\n only if you received the object code with such an offer, in accord\n with subsection 6b.\n\n d) Convey the object code by offering access from a designated\n place (gratis or for a charge), and offer equivalent access to the\n Corresponding Source in the same way through the same place at no\n further charge. You need not require recipients to copy the\n Corresponding Source along with the object code. If the place to\n copy the object code is a network server, the Corresponding Source\n may be on a different server (operated by you or a third party)\n that supports equivalent copying facilities, provided you maintain\n clear directions next to the object code saying where to find the\n Corresponding Source. Regardless of what server hosts the\n Corresponding Source, you remain obligated to ensure that it is\n available for as long as needed to satisfy these requirements.\n\n e) Convey the object code using peer-to-peer transmission, provided\n you inform other peers where the object code and Corresponding\n Source of the work are being offered to the general public at no\n charge under subsection 6d.\n\n A separable portion of the object code, whose source code is excluded\nfrom the Corresponding Source as a System Library, need not be\nincluded in conveying the object code work.\n\n A \"User Product\" is either (1) a \"consumer product\", which means any\ntangible personal property which is normally used for personal, family,\nor household purposes, or (2) anything designed or sold for incorporation\ninto a dwelling. In determining whether a product is a consumer product,\ndoubtful cases shall be resolved in favor of coverage. For a particular\nproduct received by a particular user, \"normally used\" refers to a\ntypical or common use of that class of product, regardless of the status\nof the particular user or of the way in which the particular user\nactually uses, or expects or is expected to use, the product. A product\nis a consumer product regardless of whether the product has substantial\ncommercial, industrial or non-consumer uses, unless such uses represent\nthe only significant mode of use of the product.\n\n \"Installation Information\" for a User Product means any methods,\nprocedures, authorization keys, or other information required to install\nand execute modified versions of a covered work in that User Product from\na modified version of its Corresponding Source. The information must\nsuffice to ensure that the continued functioning of the modified object\ncode is in no case prevented or interfered with solely because\nmodification has been made.\n\n If you convey an object code work under this section in, or with, or\nspecifically for use in, a User Product, and the conveying occurs as\npart of a transaction in which the right of possession and use of the\nUser Product is transferred to the recipient in perpetuity or for a\nfixed term (regardless of how the transaction is characterized), the\nCorresponding Source conveyed under this section must be accompanied\nby the Installation Information. But this requirement does not apply\nif neither you nor any third party retains the ability to install\nmodified object code on the User Product (for example, the work has\nbeen installed in ROM).\n\n The requirement to provide Installation Information does not include a\nrequirement to continue to provide support service, warranty, or updates\nfor a work that has been modified or installed by the recipient, or for\nthe User Product in which it has been modified or installed. Access to a\nnetwork may be denied when the modification itself materially and\nadversely affects the operation of the network or violates the rules and\nprotocols for communication across the network.\n\n Corresponding Source conveyed, and Installation Information provided,\nin accord with this section must be in a format that is publicly\ndocumented (and with an implementation available to the public in\nsource code form), and must require no special password or key for\nunpacking, reading or copying.\n\n 7. Additional Terms.\n\n \"Additional permissions\" are terms that supplement the terms of this\nLicense by making exceptions from one or more of its conditions.\nAdditional permissions that are applicable to the entire Program shall\nbe treated as though they were included in this License, to the extent\nthat they are valid under applicable law. If additional permissions\napply only to part of the Program, that part may be used separately\nunder those permissions, but the entire Program remains governed by\nthis License without regard to the additional permissions.\n\n When you convey a copy of a covered work, you may at your option\nremove any additional permissions from that copy, or from any part of\nit. (Additional permissions may be written to require their own\nremoval in certain cases when you modify the work.) You may place\nadditional permissions on material, added by you to a covered work,\nfor which you have or can give appropriate copyright permission.\n\n Notwithstanding any other provision of this License, for material you\nadd to a covered work, you may (if authorized by the copyright holders of\nthat material) supplement the terms of this License with terms:\n\n a) Disclaiming warranty or limiting liability differently from the\n terms of sections 15 and 16 of this License; or\n\n b) Requiring preservation of specified reasonable legal notices or\n author attributions in that material or in the Appropriate Legal\n Notices displayed by works containing it; or\n\n c) Prohibiting misrepresentation of the origin of that material, or\n requiring that modified versions of such material be marked in\n reasonable ways as different from the original version; or\n\n d) Limiting the use for publicity purposes of names of licensors or\n authors of the material; or\n\n e) Declining to grant rights under trademark law for use of some\n trade names, trademarks, or service marks; or\n\n f) Requiring indemnification of licensors and authors of that\n material by anyone who conveys the material (or modified versions of\n it) with contractual assumptions of liability to the recipient, for\n any liability that these contractual assumptions directly impose on\n those licensors and authors.\n\n All other non-permissive additional terms are considered \"further\nrestrictions\" within the meaning of section 10. If the Program as you\nreceived it, or any part of it, contains a notice stating that it is\ngoverned by this License along with a term that is a further\nrestriction, you may remove that term. If a license document contains\na further restriction but permits relicensing or conveying under this\nLicense, you may add to a covered work material governed by the terms\nof that license document, provided that the further restriction does\nnot survive such relicensing or conveying.\n\n If you add terms to a covered work in accord with this section, you\nmust place, in the relevant source files, a statement of the\nadditional terms that apply to those files, or a notice indicating\nwhere to find the applicable terms.\n\n Additional terms, permissive or non-permissive, may be stated in the\nform of a separately written license, or stated as exceptions;\nthe above requirements apply either way.\n\n 8. Termination.\n\n You may not propagate or modify a covered work except as expressly\nprovided under this License. Any attempt otherwise to propagate or\nmodify it is void, and will automatically terminate your rights under\nthis License (including any patent licenses granted under the third\nparagraph of section 11).\n\n However, if you cease all violation of this License, then your\nlicense from a particular copyright holder is reinstated (a)\nprovisionally, unless and until the copyright holder explicitly and\nfinally terminates your license, and (b) permanently, if the copyright\nholder fails to notify you of the violation by some reasonable means\nprior to 60 days after the cessation.\n\n Moreover, your license from a particular copyright holder is\nreinstated permanently if the copyright holder notifies you of the\nviolation by some reasonable means, this is the first time you have\nreceived notice of violation of this License (for any work) from that\ncopyright holder, and you cure the violation prior to 30 days after\nyour receipt of the notice.\n\n Termination of your rights under this section does not terminate the\nlicenses of parties who have received copies or rights from you under\nthis License. If your rights have been terminated and not permanently\nreinstated, you do not qualify to receive new licenses for the same\nmaterial under section 10.\n\n 9. Acceptance Not Required for Having Copies.\n\n You are not required to accept this License in order to receive or\nrun a copy of the Program. Ancillary propagation of a covered work\noccurring solely as a consequence of using peer-to-peer transmission\nto receive a copy likewise does not require acceptance. However,\nnothing other than this License grants you permission to propagate or\nmodify any covered work. These actions infringe copyright if you do\nnot accept this License. Therefore, by modifying or propagating a\ncovered work, you indicate your acceptance of this License to do so.\n\n 10. Automatic Licensing of Downstream Recipients.\n\n Each time you convey a covered work, the recipient automatically\nreceives a license from the original licensors, to run, modify and\npropagate that work, subject to this License. You are not responsible\nfor enforcing compliance by third parties with this License.\n\n An \"entity transaction\" is a transaction transferring control of an\norganization, or substantially all assets of one, or subdividing an\norganization, or merging organizations. If propagation of a covered\nwork results from an entity transaction, each party to that\ntransaction who receives a copy of the work also receives whatever\nlicenses to the work the party's predecessor in interest had or could\ngive under the previous paragraph, plus a right to possession of the\nCorresponding Source of the work from the predecessor in interest, if\nthe predecessor has it or can get it with reasonable efforts.\n\n You may not impose any further restrictions on the exercise of the\nrights granted or affirmed under this License. For example, you may\nnot impose a license fee, royalty, or other charge for exercise of\nrights granted under this License, and you may not initiate litigation\n(including a cross-claim or counterclaim in a lawsuit) alleging that\nany patent claim is infringed by making, using, selling, offering for\nsale, or importing the Program or any portion of it.\n\n 11. Patents.\n\n A \"contributor\" is a copyright holder who authorizes use under this\nLicense of the Program or a work on which the Program is based. The\nwork thus licensed is called the contributor's \"contributor version\".\n\n A contributor's \"essential patent claims\" are all patent claims\nowned or controlled by the contributor, whether already acquired or\nhereafter acquired, that would be infringed by some manner, permitted\nby this License, of making, using, or selling its contributor version,\nbut do not include claims that would be infringed only as a\nconsequence of further modification of the contributor version. For\npurposes of this definition, \"control\" includes the right to grant\npatent sublicenses in a manner consistent with the requirements of\nthis License.\n\n Each contributor grants you a non-exclusive, worldwide, royalty-free\npatent license under the contributor's essential patent claims, to\nmake, use, sell, offer for sale, import and otherwise run, modify and\npropagate the contents of its contributor version.\n\n In the following three paragraphs, a \"patent license\" is any express\nagreement or commitment, however denominated, not to enforce a patent\n(such as an express permission to practice a patent or covenant not to\nsue for patent infringement). To \"grant\" such a patent license to a\nparty means to make such an agreement or commitment not to enforce a\npatent against the party.\n\n If you convey a covered work, knowingly relying on a patent license,\nand the Corresponding Source of the work is not available for anyone\nto copy, free of charge and under the terms of this License, through a\npublicly available network server or other readily accessible means,\nthen you must either (1) cause the Corresponding Source to be so\navailable, or (2) arrange to deprive yourself of the benefit of the\npatent license for this particular work, or (3) arrange, in a manner\nconsistent with the requirements of this License, to extend the patent\nlicense to downstream recipients. \"Knowingly relying\" means you have\nactual knowledge that, but for the patent license, your conveying the\ncovered work in a country, or your recipient's use of the covered work\nin a country, would infringe one or more identifiable patents in that\ncountry that you have reason to believe are valid.\n\n If, pursuant to or in connection with a single transaction or\narrangement, you convey, or propagate by procuring conveyance of, a\ncovered work, and grant a patent license to some of the parties\nreceiving the covered work authorizing them to use, propagate, modify\nor convey a specific copy of the covered work, then the patent license\nyou grant is automatically extended to all recipients of the covered\nwork and works based on it.\n\n A patent license is \"discriminatory\" if it does not include within\nthe scope of its coverage, prohibits the exercise of, or is\nconditioned on the non-exercise of one or more of the rights that are\nspecifically granted under this License. You may not convey a covered\nwork if you are a party to an arrangement with a third party that is\nin the business of distributing software, under which you make payment\nto the third party based on the extent of your activity of conveying\nthe work, and under which the third party grants, to any of the\nparties who would receive the covered work from you, a discriminatory\npatent license (a) in connection with copies of the covered work\nconveyed by you (or copies made from those copies), or (b) primarily\nfor and in connection with specific products or compilations that\ncontain the covered work, unless you entered into that arrangement,\nor that patent license was granted, prior to 28 March 2007.\n\n Nothing in this License shall be construed as excluding or limiting\nany implied license or other defenses to infringement that may\notherwise be available to you under applicable patent law.\n\n 12. No Surrender of Others' Freedom.\n\n If conditions are imposed on you (whether by court order, agreement or\notherwise) that contradict the conditions of this License, they do not\nexcuse you from the conditions of this License. If you cannot convey a\ncovered work so as to satisfy simultaneously your obligations under this\nLicense and any other pertinent obligations, then as a consequence you may\nnot convey it at all. For example, if you agree to terms that obligate you\nto collect a royalty for further conveying from those to whom you convey\nthe Program, the only way you could satisfy both those terms and this\nLicense would be to refrain entirely from conveying the Program.\n\n 13. Use with the GNU Affero General Public License.\n\n Notwithstanding any other provision of this License, you have\npermission to link or combine any covered work with a work licensed\nunder version 3 of the GNU Affero General Public License into a single\ncombined work, and to convey the resulting work. The terms of this\nLicense will continue to apply to the part which is the covered work,\nbut the special requirements of the GNU Affero General Public License,\nsection 13, concerning interaction through a network will apply to the\ncombination as such.\n\n 14. Revised Versions of this License.\n\n The Free Software Foundation may publish revised and/or new versions of\nthe GNU General Public License from time to time. Such new versions will\nbe similar in spirit to the present version, but may differ in detail to\naddress new problems or concerns.\n\n Each version is given a distinguishing version number. If the\nProgram specifies that a certain numbered version of the GNU General\nPublic License \"or any later version\" applies to it, you have the\noption of following the terms and conditions either of that numbered\nversion or of any later version published by the Free Software\nFoundation. If the Program does not specify a version number of the\nGNU General Public License, you may choose any version ever published\nby the Free Software Foundation.\n\n If the Program specifies that a proxy can decide which future\nversions of the GNU General Public License can be used, that proxy's\npublic statement of acceptance of a version permanently authorizes you\nto choose that version for the Program.\n\n Later license versions may give you additional or different\npermissions. However, no additional obligations are imposed on any\nauthor or copyright holder as a result of your choosing to follow a\nlater version.\n\n 15. Disclaimer of Warranty.\n\n THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY\nAPPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT\nHOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM \"AS IS\" WITHOUT WARRANTY\nOF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,\nTHE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR\nPURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM\nIS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF\nALL NECESSARY SERVICING, REPAIR OR CORRECTION.\n\n 16. Limitation of Liability.\n\n IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING\nWILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS\nTHE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY\nGENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE\nUSE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF\nDATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD\nPARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS),\nEVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF\nSUCH DAMAGES.\n\n 17. Interpretation of Sections 15 and 16.\n\n If the disclaimer of warranty and limitation of liability provided\nabove cannot be given local legal effect according to their terms,\nreviewing courts shall apply local law that most closely approximates\nan absolute waiver of all civil liability in connection with the\nProgram, unless a warranty or assumption of liability accompanies a\ncopy of the Program in return for a fee.\n\n END OF TERMS AND CONDITIONS\n\n How to Apply These Terms to Your New Programs\n\n If you develop a new program, and you want it to be of the greatest\npossible use to the public, the best way to achieve this is to make it\nfree software which everyone can redistribute and change under these terms.\n\n To do so, attach the following notices to the program. It is safest\nto attach them to the start of each source file to most effectively\nstate the exclusion of warranty; and each file should have at least\nthe \"copyright\" line and a pointer to where the full notice is found.\n\n \n Copyright (C) \n\n This program is free software: you can redistribute it and/or modify\n it under the terms of the GNU General Public License as published by\n the Free Software Foundation, either version 3 of the License, or\n (at your option) any later version.\n\n This program is distributed in the hope that it will be useful,\n but WITHOUT ANY WARRANTY; without even the implied warranty of\n MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n GNU General Public License for more details.\n\n You should have received a copy of the GNU General Public License\n along with this program. If not, see .\n\nAlso add information on how to contact you by electronic and paper mail.\n\n If the program does terminal interaction, make it output a short\nnotice like this when it starts in an interactive mode:\n\n Copyright (C) \n This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'.\n This is free software, and you are welcome to redistribute it\n under certain conditions; type `show c' for details.\n\nThe hypothetical commands `show w' and `show c' should show the appropriate\nparts of the General Public License. Of course, your program's commands\nmight be different; for a GUI interface, you would use an \"about box\".\n\n You should also get your employer (if you work as a programmer) or school,\nif any, to sign a \"copyright disclaimer\" for the program, if necessary.\nFor more information on this, and how to apply and follow the GNU GPL, see\n.\n\n The GNU General Public License does not permit incorporating your program\ninto proprietary programs. If your program is a subroutine library, you\nmay consider it more useful to permit linking proprietary applications with\nthe library. If this is what you want to do, use the GNU Lesser General\nPublic License instead of this License. But first, please read\n.\n" }, { "path": "common/utils/manopth/README.md", "content": "Manopth\n=======\n\n[MANO](http://mano.is.tue.mpg.de) layer for [PyTorch](https://pytorch.org/) (tested with v0.4 and v1.x)\n\nManoLayer is a differentiable PyTorch layer that deterministically maps from pose and shape parameters to hand joints and vertices.\nIt can be integrated into any architecture as a differentiable layer to predict hand meshes.\n\n![image](assets/mano_layer.png)\n\nManoLayer takes **batched** hand pose and shape vectors and outputs corresponding hand joints and vertices.\n\nThe code is mostly a PyTorch port of the original [MANO](http://mano.is.tue.mpg.de) model from [chumpy](https://github.com/mattloper/chumpy) to [PyTorch](https://pytorch.org/).\nIt therefore builds directly upon the work of Javier Romero, Dimitrios Tzionas and Michael J. Black.\n\nThis layer was developped and used for the paper *Learning joint reconstruction of hands and manipulated objects* for CVPR19.\nSee [project page](https://github.com/hassony2/obman) and [demo+training code](https://github.com/hassony2/obman_train).\n\n\nIt [reuses](https://github.com/hassony2/manopth/blob/master/manopth/rodrigues_layer.py) [part of the great code](https://github.com/MandyMo/pytorch_HMR/blob/master/src/util.py) from the [Pytorch layer for the SMPL body model](https://github.com/MandyMo/pytorch_HMR/blob/master/README.md) by Zhang Xiong ([MandyMo](https://github.com/MandyMo)) to compute the rotation utilities !\n\nIt also includes in `mano/webuser` partial content of files from the original [MANO](http://mano.is.tue.mpg.de) code ([posemapper.py](mano/webuser/posemapper.py), [serialization.py](mano/webuser/serialization.py), [lbs.py](mano/webuser/lbs.py), [verts.py](mano/webuser/verts.py), [smpl_handpca_wrapper_HAND_only.py](mano/webuser/smpl_handpca_wrapper_HAND_only.py)).\n\nIf you find this code useful for your research, consider citing:\n\n- the original [MANO](http://mano.is.tue.mpg.de) publication:\n\n```\n@article{MANO:SIGGRAPHASIA:2017,\n title = {Embodied Hands: Modeling and Capturing Hands and Bodies Together},\n author = {Romero, Javier and Tzionas, Dimitrios and Black, Michael J.},\n journal = {ACM Transactions on Graphics, (Proc. SIGGRAPH Asia)},\n publisher = {ACM},\n month = nov,\n year = {2017},\n url = {http://doi.acm.org/10.1145/3130800.3130883},\n month_numeric = {11}\n}\n```\n\n- the publication this PyTorch port was developped for:\n\n```\n@INPROCEEDINGS{hasson19_obman,\n title = {Learning joint reconstruction of hands and manipulated objects},\n author = {Hasson, Yana and Varol, G{\\\"u}l and Tzionas, Dimitris and Kalevatykh, Igor and Black, Michael J. and Laptev, Ivan and Schmid, Cordelia},\n booktitle = {CVPR},\n year = {2019}\n}\n```\n\nThe training code associated with this paper, compatible with manopth can be found [here](https://github.com/hassony2/obman_train). The release includes a model trained on a variety of hand datasets.\n\n# Installation\n\n## Get code and dependencies\n\n- `git clone https://github.com/hassony2/manopth`\n- `cd manopth`\n- Install the dependencies listed in [environment.yml](environment.yml)\n - In an existing conda environment, `conda env update -f environment.yml`\n - In a new environment, `conda env create -f environment.yml`, will create a conda environment named `manopth`\n\n## Download MANO pickle data-structures\n\n- Go to [MANO website](http://mano.is.tue.mpg.de/)\n- Create an account by clicking *Sign Up* and provide your information\n- Download Models and Code (the downloaded file should have the format `mano_v*_*.zip`). Note that all code and data from this download falls under the [MANO license](http://mano.is.tue.mpg.de/license).\n- unzip and copy the `models` folder into the `manopth/mano` folder\n- Your folder structure should look like this:\n```\nmanopth/\n mano/\n models/\n MANO_LEFT.pkl\n MANO_RIGHT.pkl\n ...\n manopth/\n __init__.py\n ...\n```\n\nTo check that everything is going well, run `python examples/manopth_mindemo.py`, which should generate from a random hand using the MANO layer !\n\n## Install `manopth` package\n\nTo be able to import and use `ManoLayer` in another project, go to your `manopth` folder and run `pip install .`\n\n\n`cd /path/to/other/project`\n\nYou can now use `from manopth import ManoLayer` in this other project!\n\n# Usage \n\n## Minimal usage script\n\nSee [examples/manopth_mindemo.py](examples/manopth_mindemo.py)\n\nSimple forward pass with random pose and shape parameters through MANO layer\n\n```python\nimport torch\nfrom manopth.manolayer import ManoLayer\nfrom manopth import demo\n\nbatch_size = 10\n# Select number of principal components for pose space\nncomps = 6\n\n# Initialize MANO layer\nmano_layer = ManoLayer(mano_root='mano/models', use_pca=True, ncomps=ncomps)\n\n# Generate random shape parameters\nrandom_shape = torch.rand(batch_size, 10)\n# Generate random pose parameters, including 3 values for global axis-angle rotation\nrandom_pose = torch.rand(batch_size, ncomps + 3)\n\n# Forward pass through MANO layer\nhand_verts, hand_joints = mano_layer(random_pose, random_shape)\ndemo.display_hand({'verts': hand_verts, 'joints': hand_joints}, mano_faces=mano_layer.th_faces)\n```\n\nResult :\n\n![random hand](assets/random_hand.png)\n\n## Demo \n\nWith more options, forward and backward pass, and a loop for quick profiling, look at [examples/manopth_demo.py](examples/manopth_demo.py).\n\nYou can run it locally with:\n\n`python examples/manopth_demo.py`\n\n" }, { "path": "common/utils/manopth/environment.yml", "content": "name: manopth\n\ndependencies:\n - opencv\n - python=3.7\n - matplotlib\n - numpy\n - pytorch\n - tqdm\n - git\n - pip:\n - git+https://github.com/hassony2/chumpy.git\n" }, { "path": "common/utils/manopth/examples/manopth_demo.py", "content": "import argparse\n\nfrom matplotlib import pyplot as plt\nfrom mpl_toolkits.mplot3d import Axes3D\nimport torch\nfrom tqdm import tqdm\n\nfrom manopth import argutils\nfrom manopth.manolayer import ManoLayer\nfrom manopth.demo import display_hand\n\nif __name__ == '__main__':\n parser = argparse.ArgumentParser()\n parser.add_argument('--batch_size', default=1, type=int)\n parser.add_argument('--cuda', action='store_true')\n parser.add_argument(\n '--no_display',\n action='store_true',\n help=\"Disable display output of ManoLayer given random inputs\")\n parser.add_argument('--side', default='left', choices=['left', 'right'])\n parser.add_argument('--random_shape', action='store_true', help=\"Random hand shape\")\n parser.add_argument('--rand_mag', type=float, default=1, help=\"Controls pose variability\")\n parser.add_argument(\n '--flat_hand_mean',\n action='store_true',\n help=\"Use flat hand as mean instead of average hand pose\")\n parser.add_argument(\n '--iters',\n type=int,\n default=1,\n help=\n \"Use for quick profiling of forward and backward pass accross ManoLayer\"\n )\n parser.add_argument('--mano_root', default='mano/models')\n parser.add_argument('--root_rot_mode', default='axisang', choices=['rot6d', 'axisang'])\n parser.add_argument('--no_pca', action='store_true', help=\"Give axis-angle or rotation matrix as inputs instead of PCA coefficients\")\n parser.add_argument('--joint_rot_mode', default='axisang', choices=['rotmat', 'axisang'], help=\"Joint rotation inputs\")\n parser.add_argument(\n '--mano_ncomps', default=6, type=int, help=\"Number of PCA components\")\n args = parser.parse_args()\n\n argutils.print_args(args)\n\n layer = ManoLayer(\n flat_hand_mean=args.flat_hand_mean,\n side=args.side,\n mano_root=args.mano_root,\n ncomps=args.mano_ncomps,\n use_pca=not args.no_pca,\n root_rot_mode=args.root_rot_mode,\n joint_rot_mode=args.joint_rot_mode)\n if args.root_rot_mode == 'axisang':\n rot = 3\n else:\n rot = 6\n print(rot)\n if args.no_pca:\n args.mano_ncomps = 45\n\n # Generate random pose coefficients\n pose_params = args.rand_mag * torch.rand(args.batch_size, args.mano_ncomps + rot)\n pose_params.requires_grad = True\n if args.random_shape:\n shape = torch.rand(args.batch_size, 10)\n else:\n shape = torch.zeros(1) # Hack to act like None for PyTorch JIT\n if args.cuda:\n pose_params = pose_params.cuda()\n shape = shape.cuda()\n layer.cuda()\n\n # Loop for forward/backward quick profiling\n for idx in tqdm(range(args.iters)):\n # Forward pass\n verts, Jtr = layer(pose_params, th_betas=shape)\n\n # Backward pass\n loss = torch.norm(verts)\n loss.backward()\n\n if not args.no_display:\n verts, Jtr = layer(pose_params, th_betas=shape)\n joints = Jtr.cpu().detach()\n verts = verts.cpu().detach()\n\n # Draw obtained vertices and joints\n display_hand({\n 'verts': verts,\n 'joints': joints\n },\n mano_faces=layer.th_faces)\n" }, { "path": "common/utils/manopth/examples/manopth_mindemo.py", "content": "import torch\nfrom manopth.manolayer import ManoLayer\nfrom manopth import demo\n\nbatch_size = 10\n# Select number of principal components for pose space\nncomps = 6\n\n# Initialize MANO layer\nmano_layer = ManoLayer(\n mano_root='mano/models', use_pca=True, ncomps=ncomps, flat_hand_mean=False)\n\n# Generate random shape parameters\nrandom_shape = torch.rand(batch_size, 10)\n# Generate random pose parameters, including 3 values for global axis-angle rotation\nrandom_pose = torch.rand(batch_size, ncomps + 3)\n\n# Forward pass through MANO layer\nhand_verts, hand_joints = mano_layer(random_pose, random_shape)\ndemo.display_hand({\n 'verts': hand_verts,\n 'joints': hand_joints\n},\n mano_faces=mano_layer.th_faces)\n" }, { "path": "common/utils/manopth/mano/__init__.py", "content": "" }, { "path": "common/utils/manopth/mano/webuser/__init__.py", "content": "" }, { "path": "common/utils/manopth/mano/webuser/lbs.py", "content": "'''\nCopyright 2017 Javier Romero, Dimitrios Tzionas, Michael J Black and the Max Planck Gesellschaft. All rights reserved.\nThis software is provided for research purposes only.\nBy using this software you agree to the terms of the MANO/SMPL+H Model license here http://mano.is.tue.mpg.de/license\n\nMore information about MANO/SMPL+H is available at http://mano.is.tue.mpg.de.\nFor comments or questions, please email us at: mano@tue.mpg.de\n\n\nAbout this file:\n================\nThis file defines a wrapper for the loading functions of the MANO model.\n\nModules included:\n- load_model:\n loads the MANO model from a given file location (i.e. a .pkl file location),\n or a dictionary object.\n\n'''\n\n\nfrom mano.webuser.posemapper import posemap\nimport chumpy\nimport numpy as np\n\n\ndef global_rigid_transformation(pose, J, kintree_table, xp):\n results = {}\n pose = pose.reshape((-1, 3))\n id_to_col = {kintree_table[1, i]: i for i in range(kintree_table.shape[1])}\n parent = {\n i: id_to_col[kintree_table[0, i]]\n for i in range(1, kintree_table.shape[1])\n }\n\n if xp == chumpy:\n from mano.webuser.posemapper import Rodrigues\n rodrigues = lambda x: Rodrigues(x)\n else:\n import cv2\n rodrigues = lambda x: cv2.Rodrigues(x)[0]\n\n with_zeros = lambda x: xp.vstack((x, xp.array([[0.0, 0.0, 0.0, 1.0]])))\n results[0] = with_zeros(\n xp.hstack((rodrigues(pose[0, :]), J[0, :].reshape((3, 1)))))\n\n for i in range(1, kintree_table.shape[1]):\n results[i] = results[parent[i]].dot(\n with_zeros(\n xp.hstack((rodrigues(pose[i, :]), ((J[i, :] - J[parent[i], :]\n ).reshape((3, 1)))))))\n\n pack = lambda x: xp.hstack([np.zeros((4, 3)), x.reshape((4, 1))])\n\n results = [results[i] for i in sorted(results.keys())]\n results_global = results\n\n if True:\n results2 = [\n results[i] - (pack(results[i].dot(xp.concatenate(((J[i, :]), 0)))))\n for i in range(len(results))\n ]\n results = results2\n result = xp.dstack(results)\n return result, results_global\n\n\ndef verts_core(pose, v, J, weights, kintree_table, want_Jtr=False, xp=chumpy):\n A, A_global = global_rigid_transformation(pose, J, kintree_table, xp)\n T = A.dot(weights.T)\n\n rest_shape_h = xp.vstack((v.T, np.ones((1, v.shape[0]))))\n\n v = (T[:, 0, :] * rest_shape_h[0, :].reshape(\n (1, -1)) + T[:, 1, :] * rest_shape_h[1, :].reshape(\n (1, -1)) + T[:, 2, :] * rest_shape_h[2, :].reshape(\n (1, -1)) + T[:, 3, :] * rest_shape_h[3, :].reshape((1, -1))).T\n\n v = v[:, :3]\n\n if not want_Jtr:\n return v\n Jtr = xp.vstack([g[:3, 3] for g in A_global])\n return (v, Jtr)\n" }, { "path": "common/utils/manopth/mano/webuser/posemapper.py", "content": "'''\nCopyright 2017 Javier Romero, Dimitrios Tzionas, Michael J Black and the Max Planck Gesellschaft. All rights reserved.\nThis software is provided for research purposes only.\nBy using this software you agree to the terms of the MANO/SMPL+H Model license here http://mano.is.tue.mpg.de/license\n\nMore information about MANO/SMPL+H is available at http://mano.is.tue.mpg.de.\nFor comments or questions, please email us at: mano@tue.mpg.de\n\n\nAbout this file:\n================\nThis file defines a wrapper for the loading functions of the MANO model.\n\nModules included:\n- load_model:\n loads the MANO model from a given file location (i.e. a .pkl file location),\n or a dictionary object.\n\n'''\n\n\nimport chumpy as ch\nimport numpy as np\nimport cv2\n\n\nclass Rodrigues(ch.Ch):\n dterms = 'rt'\n\n def compute_r(self):\n return cv2.Rodrigues(self.rt.r)[0]\n\n def compute_dr_wrt(self, wrt):\n if wrt is self.rt:\n return cv2.Rodrigues(self.rt.r)[1].T\n\n\ndef lrotmin(p):\n if isinstance(p, np.ndarray):\n p = p.ravel()[3:]\n return np.concatenate(\n [(cv2.Rodrigues(np.array(pp))[0] - np.eye(3)).ravel()\n for pp in p.reshape((-1, 3))]).ravel()\n if p.ndim != 2 or p.shape[1] != 3:\n p = p.reshape((-1, 3))\n p = p[1:]\n return ch.concatenate([(Rodrigues(pp) - ch.eye(3)).ravel()\n for pp in p]).ravel()\n\n\ndef posemap(s):\n if s == 'lrotmin':\n return lrotmin\n else:\n raise Exception('Unknown posemapping: %s' % (str(s), ))\n" }, { "path": "common/utils/manopth/mano/webuser/serialization.py", "content": "'''\nCopyright 2017 Javier Romero, Dimitrios Tzionas, Michael J Black and the Max Planck Gesellschaft. All rights reserved.\nThis software is provided for research purposes only.\nBy using this software you agree to the terms of the MANO/SMPL+H Model license here http://mano.is.tue.mpg.de/license\n\nMore information about MANO/SMPL+H is available at http://mano.is.tue.mpg.de.\nFor comments or questions, please email us at: mano@tue.mpg.de\n\n\nAbout this file:\n================\nThis file defines a wrapper for the loading functions of the MANO model.\n\nModules included:\n- load_model:\n loads the MANO model from a given file location (i.e. a .pkl file location),\n or a dictionary object.\n\n'''\n\n\n__all__ = ['load_model', 'save_model']\n\nimport numpy as np\nimport pickle\nimport chumpy as ch\nfrom chumpy.ch import MatVecMult\nfrom mano.webuser.posemapper import posemap\nfrom mano.webuser.verts import verts_core\n\ndef ready_arguments(fname_or_dict):\n\n if not isinstance(fname_or_dict, dict):\n dd = pickle.load(open(fname_or_dict, 'rb'), encoding='latin1')\n else:\n dd = fname_or_dict\n\n backwards_compatibility_replacements(dd)\n\n want_shapemodel = 'shapedirs' in dd\n nposeparms = dd['kintree_table'].shape[1] * 3\n\n if 'trans' not in dd:\n dd['trans'] = np.zeros(3)\n if 'pose' not in dd:\n dd['pose'] = np.zeros(nposeparms)\n if 'shapedirs' in dd and 'betas' not in dd:\n dd['betas'] = np.zeros(dd['shapedirs'].shape[-1])\n\n for s in [\n 'v_template', 'weights', 'posedirs', 'pose', 'trans', 'shapedirs',\n 'betas', 'J'\n ]:\n if (s in dd) and not hasattr(dd[s], 'dterms'):\n dd[s] = ch.array(dd[s])\n\n if want_shapemodel:\n dd['v_shaped'] = dd['shapedirs'].dot(dd['betas']) + dd['v_template']\n v_shaped = dd['v_shaped']\n J_tmpx = MatVecMult(dd['J_regressor'], v_shaped[:, 0])\n J_tmpy = MatVecMult(dd['J_regressor'], v_shaped[:, 1])\n J_tmpz = MatVecMult(dd['J_regressor'], v_shaped[:, 2])\n dd['J'] = ch.vstack((J_tmpx, J_tmpy, J_tmpz)).T\n dd['v_posed'] = v_shaped + dd['posedirs'].dot(\n posemap(dd['bs_type'])(dd['pose']))\n else:\n dd['v_posed'] = dd['v_template'] + dd['posedirs'].dot(\n posemap(dd['bs_type'])(dd['pose']))\n\n return dd\n\n\ndef load_model(fname_or_dict):\n dd = ready_arguments(fname_or_dict)\n\n args = {\n 'pose': dd['pose'],\n 'v': dd['v_posed'],\n 'J': dd['J'],\n 'weights': dd['weights'],\n 'kintree_table': dd['kintree_table'],\n 'xp': ch,\n 'want_Jtr': True,\n 'bs_style': dd['bs_style']\n }\n\n result, Jtr = verts_core(**args)\n result = result + dd['trans'].reshape((1, 3))\n result.J_transformed = Jtr + dd['trans'].reshape((1, 3))\n\n for k, v in dd.items():\n setattr(result, k, v)\n\n return result\n" }, { "path": "common/utils/manopth/mano/webuser/smpl_handpca_wrapper_HAND_only.py", "content": "'''\nCopyright 2017 Javier Romero, Dimitrios Tzionas, Michael J Black and the Max Planck Gesellschaft. All rights reserved.\nThis software is provided for research purposes only.\nBy using this software you agree to the terms of the MANO/SMPL+H Model license here http://mano.is.tue.mpg.de/license\n\nMore information about MANO/SMPL+H is available at http://mano.is.tue.mpg.de.\nFor comments or questions, please email us at: mano@tue.mpg.de\n\n\nAbout this file:\n================\nThis file defines a wrapper for the loading functions of the MANO model.\n\nModules included:\n- load_model:\n loads the MANO model from a given file location (i.e. a .pkl file location),\n or a dictionary object.\n\n'''\n\n\ndef ready_arguments(fname_or_dict, posekey4vposed='pose'):\n import numpy as np\n import pickle\n import chumpy as ch\n from chumpy.ch import MatVecMult\n from mano.webuser.posemapper import posemap\n\n if not isinstance(fname_or_dict, dict):\n dd = pickle.load(open(fname_or_dict, 'rb'), encoding='latin1')\n # dd = pickle.load(open(fname_or_dict, 'rb'))\n else:\n dd = fname_or_dict\n\n want_shapemodel = 'shapedirs' in dd\n nposeparms = dd['kintree_table'].shape[1] * 3\n\n if 'trans' not in dd:\n dd['trans'] = np.zeros(3)\n if 'pose' not in dd:\n dd['pose'] = np.zeros(nposeparms)\n if 'shapedirs' in dd and 'betas' not in dd:\n dd['betas'] = np.zeros(dd['shapedirs'].shape[-1])\n\n for s in [\n 'v_template', 'weights', 'posedirs', 'pose', 'trans', 'shapedirs',\n 'betas', 'J'\n ]:\n if (s in dd) and not hasattr(dd[s], 'dterms'):\n dd[s] = ch.array(dd[s])\n\n assert (posekey4vposed in dd)\n if want_shapemodel:\n dd['v_shaped'] = dd['shapedirs'].dot(dd['betas']) + dd['v_template']\n v_shaped = dd['v_shaped']\n J_tmpx = MatVecMult(dd['J_regressor'], v_shaped[:, 0])\n J_tmpy = MatVecMult(dd['J_regressor'], v_shaped[:, 1])\n J_tmpz = MatVecMult(dd['J_regressor'], v_shaped[:, 2])\n dd['J'] = ch.vstack((J_tmpx, J_tmpy, J_tmpz)).T\n pose_map_res = posemap(dd['bs_type'])(dd[posekey4vposed])\n dd['v_posed'] = v_shaped + dd['posedirs'].dot(pose_map_res)\n else:\n pose_map_res = posemap(dd['bs_type'])(dd[posekey4vposed])\n dd_add = dd['posedirs'].dot(pose_map_res)\n dd['v_posed'] = dd['v_template'] + dd_add\n\n return dd\n\n\ndef load_model(fname_or_dict, ncomps=6, flat_hand_mean=False, v_template=None):\n ''' This model loads the fully articulable HAND SMPL model,\n and replaces the pose DOFS by ncomps from PCA'''\n\n from mano.webuser.verts import verts_core\n import numpy as np\n import chumpy as ch\n import pickle\n import scipy.sparse as sp\n np.random.seed(1)\n\n if not isinstance(fname_or_dict, dict):\n smpl_data = pickle.load(open(fname_or_dict, 'rb'), encoding='latin1')\n # smpl_data = pickle.load(open(fname_or_dict, 'rb'))\n else:\n smpl_data = fname_or_dict\n\n rot = 3 # for global orientation!!!\n\n hands_components = smpl_data['hands_components']\n hands_mean = np.zeros(hands_components.shape[\n 1]) if flat_hand_mean else smpl_data['hands_mean']\n hands_coeffs = smpl_data['hands_coeffs'][:, :ncomps]\n\n selected_components = np.vstack((hands_components[:ncomps]))\n hands_mean = hands_mean.copy()\n\n pose_coeffs = ch.zeros(rot + selected_components.shape[0])\n full_hand_pose = pose_coeffs[rot:(rot + ncomps)].dot(selected_components)\n\n smpl_data['fullpose'] = ch.concatenate((pose_coeffs[:rot],\n hands_mean + full_hand_pose))\n smpl_data['pose'] = pose_coeffs\n\n Jreg = smpl_data['J_regressor']\n if not sp.issparse(Jreg):\n smpl_data['J_regressor'] = (sp.csc_matrix(\n (Jreg.data, (Jreg.row, Jreg.col)), shape=Jreg.shape))\n\n # slightly modify ready_arguments to make sure that it uses the fullpose\n # (which will NOT be pose) for the computation of posedirs\n dd = ready_arguments(smpl_data, posekey4vposed='fullpose')\n\n # create the smpl formula with the fullpose,\n # but expose the PCA coefficients as smpl.pose for compatibility\n args = {\n 'pose': dd['fullpose'],\n 'v': dd['v_posed'],\n 'J': dd['J'],\n 'weights': dd['weights'],\n 'kintree_table': dd['kintree_table'],\n 'xp': ch,\n 'want_Jtr': True,\n 'bs_style': dd['bs_style'],\n }\n\n result_previous, meta = verts_core(**args)\n\n result = result_previous + dd['trans'].reshape((1, 3))\n result.no_translation = result_previous\n\n if meta is not None:\n for field in ['Jtr', 'A', 'A_global', 'A_weighted']:\n if (hasattr(meta, field)):\n setattr(result, field, getattr(meta, field))\n\n setattr(result, 'Jtr', meta)\n if hasattr(result, 'Jtr'):\n result.J_transformed = result.Jtr + dd['trans'].reshape((1, 3))\n\n for k, v in dd.items():\n setattr(result, k, v)\n\n if v_template is not None:\n result.v_template[:] = v_template\n\n return result\n\n\nif __name__ == '__main__':\n load_model()\n" }, { "path": "common/utils/manopth/mano/webuser/verts.py", "content": "'''\nCopyright 2017 Javier Romero, Dimitrios Tzionas, Michael J Black and the Max Planck Gesellschaft. All rights reserved.\nThis software is provided for research purposes only.\nBy using this software you agree to the terms of the MANO/SMPL+H Model license here http://mano.is.tue.mpg.de/license\n\nMore information about MANO/SMPL+H is available at http://mano.is.tue.mpg.de.\nFor comments or questions, please email us at: mano@tue.mpg.de\n\n\nAbout this file:\n================\nThis file defines a wrapper for the loading functions of the MANO model.\n\nModules included:\n- load_model:\n loads the MANO model from a given file location (i.e. a .pkl file location),\n or a dictionary object.\n\n'''\n\n\nimport chumpy\nimport mano.webuser.lbs as lbs\nfrom mano.webuser.posemapper import posemap\nimport scipy.sparse as sp\nfrom chumpy.ch import MatVecMult\n\n\ndef ischumpy(x):\n return hasattr(x, 'dterms')\n\n\ndef verts_decorated(trans,\n pose,\n v_template,\n J_regressor,\n weights,\n kintree_table,\n bs_style,\n f,\n bs_type=None,\n posedirs=None,\n betas=None,\n shapedirs=None,\n want_Jtr=False):\n\n for which in [\n trans, pose, v_template, weights, posedirs, betas, shapedirs\n ]:\n if which is not None:\n assert ischumpy(which)\n\n v = v_template\n\n if shapedirs is not None:\n if betas is None:\n betas = chumpy.zeros(shapedirs.shape[-1])\n v_shaped = v + shapedirs.dot(betas)\n else:\n v_shaped = v\n\n if posedirs is not None:\n v_posed = v_shaped + posedirs.dot(posemap(bs_type)(pose))\n else:\n v_posed = v_shaped\n\n v = v_posed\n\n if sp.issparse(J_regressor):\n J_tmpx = MatVecMult(J_regressor, v_shaped[:, 0])\n J_tmpy = MatVecMult(J_regressor, v_shaped[:, 1])\n J_tmpz = MatVecMult(J_regressor, v_shaped[:, 2])\n J = chumpy.vstack((J_tmpx, J_tmpy, J_tmpz)).T\n else:\n assert (ischumpy(J))\n\n assert (bs_style == 'lbs')\n result, Jtr = lbs.verts_core(\n pose, v, J, weights, kintree_table, want_Jtr=True, xp=chumpy)\n\n tr = trans.reshape((1, 3))\n result = result + tr\n Jtr = Jtr + tr\n\n result.trans = trans\n result.f = f\n result.pose = pose\n result.v_template = v_template\n result.J = J\n result.J_regressor = J_regressor\n result.weights = weights\n result.kintree_table = kintree_table\n result.bs_style = bs_style\n result.bs_type = bs_type\n if posedirs is not None:\n result.posedirs = posedirs\n result.v_posed = v_posed\n if shapedirs is not None:\n result.shapedirs = shapedirs\n result.betas = betas\n result.v_shaped = v_shaped\n if want_Jtr:\n result.J_transformed = Jtr\n return result\n\n\ndef verts_core(pose,\n v,\n J,\n weights,\n kintree_table,\n bs_style,\n want_Jtr=False,\n xp=chumpy):\n\n if xp == chumpy:\n assert (hasattr(pose, 'dterms'))\n assert (hasattr(v, 'dterms'))\n assert (hasattr(J, 'dterms'))\n assert (hasattr(weights, 'dterms'))\n\n assert (bs_style == 'lbs')\n result = lbs.verts_core(pose, v, J, weights, kintree_table, want_Jtr, xp)\n return result\n" }, { "path": "common/utils/manopth/manopth/__init__.py", "content": "name = 'manopth'\n" }, { "path": "common/utils/manopth/manopth/argutils.py", "content": "import datetime\nimport os\nimport pickle\nimport subprocess\nimport sys\n\n\ndef print_args(args):\n opts = vars(args)\n print('======= Options ========')\n for k, v in sorted(opts.items()):\n print('{}: {}'.format(k, v))\n print('========================')\n\n\ndef save_args(args, save_folder, opt_prefix='opt', verbose=True):\n opts = vars(args)\n # Create checkpoint folder\n if not os.path.exists(save_folder):\n os.makedirs(save_folder, exist_ok=True)\n\n # Save options\n opt_filename = '{}.txt'.format(opt_prefix)\n opt_path = os.path.join(save_folder, opt_filename)\n with open(opt_path, 'a') as opt_file:\n opt_file.write('====== Options ======\\n')\n for k, v in sorted(opts.items()):\n opt_file.write(\n '{option}: {value}\\n'.format(option=str(k), value=str(v)))\n opt_file.write('=====================\\n')\n opt_file.write('launched {} at {}\\n'.format(\n str(sys.argv[0]), str(datetime.datetime.now())))\n\n # Add git info\n label = subprocess.check_output([\"git\", \"describe\",\n \"--always\"]).strip()\n if subprocess.call(\n [\"git\", \"branch\"],\n stderr=subprocess.STDOUT,\n stdout=open(os.devnull, 'w')) == 0:\n opt_file.write('=== Git info ====\\n')\n opt_file.write('{}\\n'.format(label))\n commit = subprocess.check_output(['git', 'rev-parse', 'HEAD'])\n opt_file.write('commit : {}\\n'.format(commit.strip()))\n\n opt_picklename = '{}.pkl'.format(opt_prefix)\n opt_picklepath = os.path.join(save_folder, opt_picklename)\n with open(opt_picklepath, 'wb') as opt_file:\n pickle.dump(opts, opt_file)\n if verbose:\n print('Saved options to {}'.format(opt_path))\n" }, { "path": "common/utils/manopth/manopth/demo.py", "content": "from matplotlib import pyplot as plt\nfrom mpl_toolkits.mplot3d import Axes3D\nfrom mpl_toolkits.mplot3d.art3d import Poly3DCollection\nimport numpy as np\nimport torch\n\nfrom manopth.manolayer import ManoLayer\n\n\ndef generate_random_hand(batch_size=1, ncomps=6, mano_root='mano/models'):\n nfull_comps = ncomps + 3 # Add global orientation dims to PCA\n random_pcapose = torch.rand(batch_size, nfull_comps)\n mano_layer = ManoLayer(mano_root=mano_root)\n verts, joints = mano_layer(random_pcapose)\n return {'verts': verts, 'joints': joints, 'faces': mano_layer.th_faces}\n\n\ndef display_hand(hand_info, mano_faces=None, ax=None, alpha=0.2, batch_idx=0, show=True):\n \"\"\"\n Displays hand batch_idx in batch of hand_info, hand_info as returned by\n generate_random_hand\n \"\"\"\n if ax is None:\n fig = plt.figure()\n ax = fig.add_subplot(111, projection='3d')\n verts, joints = hand_info['verts'][batch_idx], hand_info['joints'][\n batch_idx]\n if mano_faces is None:\n ax.scatter(verts[:, 0], verts[:, 1], verts[:, 2], alpha=0.1)\n else:\n mesh = Poly3DCollection(verts[mano_faces], alpha=alpha)\n face_color = (141 / 255, 184 / 255, 226 / 255)\n edge_color = (50 / 255, 50 / 255, 50 / 255)\n mesh.set_edgecolor(edge_color)\n mesh.set_facecolor(face_color)\n ax.add_collection3d(mesh)\n ax.scatter(joints[:, 0], joints[:, 1], joints[:, 2], color='r')\n cam_equal_aspect_3d(ax, verts.numpy())\n if show:\n plt.show()\n\n\ndef cam_equal_aspect_3d(ax, verts, flip_x=False):\n \"\"\"\n Centers view on cuboid containing hand and flips y and z axis\n and fixes azimuth\n \"\"\"\n extents = np.stack([verts.min(0), verts.max(0)], axis=1)\n sz = extents[:, 1] - extents[:, 0]\n centers = np.mean(extents, axis=1)\n maxsize = max(abs(sz))\n r = maxsize / 2\n if flip_x:\n ax.set_xlim(centers[0] + r, centers[0] - r)\n else:\n ax.set_xlim(centers[0] - r, centers[0] + r)\n # Invert y and z axis\n ax.set_ylim(centers[1] + r, centers[1] - r)\n ax.set_zlim(centers[2] + r, centers[2] - r)\n" }, { "path": "common/utils/manopth/manopth/manolayer.py", "content": "import os\n\nimport numpy as np\nimport torch\nfrom torch.nn import Module\n\nfrom mano.webuser.smpl_handpca_wrapper_HAND_only import ready_arguments\nfrom manopth import rodrigues_layer, rotproj, rot6d\nfrom manopth.tensutils import (th_posemap_axisang, th_with_zeros, th_pack,\n subtract_flat_id, make_list)\n\n\nclass ManoLayer(Module):\n __constants__ = [\n 'use_pca', 'rot', 'ncomps', 'ncomps', 'kintree_parents', 'check',\n 'side', 'center_idx', 'joint_rot_mode'\n ]\n\n def __init__(self,\n center_idx=None,\n flat_hand_mean=True,\n ncomps=6,\n side='right',\n mano_root='mano/models',\n use_pca=True,\n root_rot_mode='axisang',\n joint_rot_mode='axisang',\n robust_rot=False):\n \"\"\"\n Args:\n center_idx: index of center joint in our computations,\n if -1 centers on estimate of palm as middle of base\n of middle finger and wrist\n flat_hand_mean: if True, (0, 0, 0, ...) pose coefficients match\n flat hand, else match average hand pose\n mano_root: path to MANO pkl files for left and right hand\n ncomps: number of PCA components form pose space (<45)\n side: 'right' or 'left'\n use_pca: Use PCA decomposition for pose space.\n joint_rot_mode: 'axisang' or 'rotmat', ignored if use_pca\n \"\"\"\n super().__init__()\n\n self.center_idx = center_idx\n self.robust_rot = robust_rot\n if root_rot_mode == 'axisang':\n self.rot = 3\n else:\n self.rot = 6\n self.flat_hand_mean = flat_hand_mean\n self.side = side\n self.use_pca = use_pca\n self.joint_rot_mode = joint_rot_mode\n self.root_rot_mode = root_rot_mode\n if use_pca:\n self.ncomps = ncomps\n else:\n self.ncomps = 45\n\n if side == 'right':\n self.mano_path = os.path.join(mano_root, 'MANO_RIGHT.pkl')\n elif side == 'left':\n self.mano_path = os.path.join(mano_root, 'MANO_LEFT.pkl')\n\n smpl_data = ready_arguments(self.mano_path)\n\n hands_components = smpl_data['hands_components']\n\n self.smpl_data = smpl_data\n\n self.register_buffer('th_betas',\n torch.Tensor(smpl_data['betas'].r).unsqueeze(0))\n self.register_buffer('th_shapedirs',\n torch.Tensor(smpl_data['shapedirs'].r))\n self.register_buffer('th_posedirs',\n torch.Tensor(smpl_data['posedirs'].r))\n self.register_buffer(\n 'th_v_template',\n torch.Tensor(smpl_data['v_template'].r).unsqueeze(0))\n self.register_buffer(\n 'th_J_regressor',\n torch.Tensor(np.array(smpl_data['J_regressor'].toarray())))\n self.register_buffer('th_weights',\n torch.Tensor(smpl_data['weights'].r))\n self.register_buffer('th_faces',\n torch.Tensor(smpl_data['f'].astype(np.int32)).long())\n\n # Get hand mean\n hands_mean = np.zeros(hands_components.shape[1]\n ) if flat_hand_mean else smpl_data['hands_mean']\n hands_mean = hands_mean.copy()\n th_hands_mean = torch.Tensor(hands_mean).unsqueeze(0)\n\n if self.use_pca or self.joint_rot_mode == 'axisang':\n # Save as axis-angle\n self.register_buffer('th_hands_mean', th_hands_mean)\n selected_components = hands_components[:ncomps]\n self.register_buffer('th_selected_comps',\n torch.Tensor(selected_components))\n else:\n th_hands_mean_rotmat = rodrigues_layer.batch_rodrigues(\n th_hands_mean.view(15, 3)).reshape(15, 3, 3)\n self.register_buffer('th_hands_mean_rotmat', th_hands_mean_rotmat)\n\n # Kinematic chain params\n self.kintree_table = smpl_data['kintree_table']\n parents = list(self.kintree_table[0].tolist())\n self.kintree_parents = parents\n\n def forward(self,\n th_pose_coeffs,\n th_betas=torch.zeros(1),\n th_trans=torch.zeros(1),\n root_palm=torch.Tensor([0]),\n share_betas=torch.Tensor([0]),\n ):\n \"\"\"\n Args:\n th_trans (Tensor (batch_size x ncomps)): if provided, applies trans to joints and vertices\n th_betas (Tensor (batch_size x 10)): if provided, uses given shape parameters for hand shape\n else centers on root joint (9th joint)\n root_palm: return palm as hand root instead of wrist\n \"\"\"\n # if len(th_pose_coeffs) == 0:\n # return th_pose_coeffs.new_empty(0), th_pose_coeffs.new_empty(0)\n\n batch_size = th_pose_coeffs.shape[0]\n # Get axis angle from PCA components and coefficients\n if self.use_pca or self.joint_rot_mode == 'axisang':\n # Remove global rot coeffs\n th_hand_pose_coeffs = th_pose_coeffs[:, self.rot:self.rot +\n self.ncomps]\n if self.use_pca:\n # PCA components --> axis angles\n th_full_hand_pose = th_hand_pose_coeffs.mm(self.th_selected_comps)\n else:\n th_full_hand_pose = th_hand_pose_coeffs\n \n # Concatenate back global rot\n th_full_pose = torch.cat([\n th_pose_coeffs[:, :self.rot],\n self.th_hands_mean + th_full_hand_pose\n ], 1)\n \n if self.root_rot_mode == 'axisang':\n # compute rotation matrixes from axis-angle while skipping global rotation\n th_pose_map, th_rot_map = th_posemap_axisang(th_full_pose)\n root_rot = th_rot_map[:, :9].view(batch_size, 3, 3)\n th_rot_map = th_rot_map[:, 9:]\n th_pose_map = th_pose_map[:, 9:]\n else:\n # th_posemap offsets by 3, so add offset or 3 to get to self.rot=6\n th_pose_map, th_rot_map = th_posemap_axisang(th_full_pose[:, 6:])\n if self.robust_rot:\n root_rot = rot6d.robust_compute_rotation_matrix_from_ortho6d(th_full_pose[:, :6])\n else:\n root_rot = rot6d.compute_rotation_matrix_from_ortho6d(th_full_pose[:, :6])\n else:\n assert th_pose_coeffs.dim() == 4, (\n 'When not self.use_pca, '\n 'th_pose_coeffs should have 4 dims, got {}'.format(\n th_pose_coeffs.dim()))\n assert th_pose_coeffs.shape[2:4] == (3, 3), (\n 'When not self.use_pca, th_pose_coeffs have 3x3 matrix for two'\n 'last dims, got {}'.format(th_pose_coeffs.shape[2:4]))\n th_pose_rots = rotproj.batch_rotprojs(th_pose_coeffs)\n th_rot_map = th_pose_rots[:, 1:].view(batch_size, -1)\n th_pose_map = subtract_flat_id(th_rot_map)\n root_rot = th_pose_rots[:, 0]\n\n # Full axis angle representation with root joint\n if th_betas is None or th_betas.numel() == 1:\n th_v_shaped = torch.matmul(self.th_shapedirs,\n self.th_betas.transpose(1, 0)).permute(\n 2, 0, 1) + self.th_v_template\n th_j = torch.matmul(self.th_J_regressor, th_v_shaped).repeat(\n batch_size, 1, 1)\n\n else:\n if share_betas:\n th_betas = th_betas.mean(0, keepdim=True).expand(th_betas.shape[0], 10)\n th_v_shaped = torch.matmul(self.th_shapedirs,\n th_betas.transpose(1, 0)).permute(\n 2, 0, 1) + self.th_v_template\n th_j = torch.matmul(self.th_J_regressor, th_v_shaped)\n # th_pose_map should have shape 20x135\n\n th_v_posed = th_v_shaped + torch.matmul(\n self.th_posedirs, th_pose_map.transpose(0, 1)).permute(2, 0, 1)\n # Final T pose with transformation done !\n\n # Global rigid transformation\n\n root_j = th_j[:, 0, :].contiguous().view(batch_size, 3, 1)\n root_trans = th_with_zeros(torch.cat([root_rot, root_j], 2))\n\n all_rots = th_rot_map.view(th_rot_map.shape[0], 15, 3, 3)\n lev1_idxs = [1, 4, 7, 10, 13]\n lev2_idxs = [2, 5, 8, 11, 14]\n lev3_idxs = [3, 6, 9, 12, 15]\n lev1_rots = all_rots[:, [idx - 1 for idx in lev1_idxs]]\n lev2_rots = all_rots[:, [idx - 1 for idx in lev2_idxs]]\n lev3_rots = all_rots[:, [idx - 1 for idx in lev3_idxs]]\n lev1_j = th_j[:, lev1_idxs]\n lev2_j = th_j[:, lev2_idxs]\n lev3_j = th_j[:, lev3_idxs]\n\n # From base to tips\n # Get lev1 results\n all_transforms = [root_trans.unsqueeze(1)]\n lev1_j_rel = lev1_j - root_j.transpose(1, 2)\n lev1_rel_transform_flt = th_with_zeros(torch.cat([lev1_rots, lev1_j_rel.unsqueeze(3)], 3).view(-1, 3, 4))\n root_trans_flt = root_trans.unsqueeze(1).repeat(1, 5, 1, 1).view(root_trans.shape[0] * 5, 4, 4)\n lev1_flt = torch.matmul(root_trans_flt, lev1_rel_transform_flt)\n all_transforms.append(lev1_flt.view(all_rots.shape[0], 5, 4, 4))\n\n # Get lev2 results\n lev2_j_rel = lev2_j - lev1_j\n lev2_rel_transform_flt = th_with_zeros(torch.cat([lev2_rots, lev2_j_rel.unsqueeze(3)], 3).view(-1, 3, 4))\n lev2_flt = torch.matmul(lev1_flt, lev2_rel_transform_flt)\n all_transforms.append(lev2_flt.view(all_rots.shape[0], 5, 4, 4))\n\n # Get lev3 results\n lev3_j_rel = lev3_j - lev2_j\n lev3_rel_transform_flt = th_with_zeros(torch.cat([lev3_rots, lev3_j_rel.unsqueeze(3)], 3).view(-1, 3, 4))\n lev3_flt = torch.matmul(lev2_flt, lev3_rel_transform_flt)\n all_transforms.append(lev3_flt.view(all_rots.shape[0], 5, 4, 4))\n\n reorder_idxs = [0, 1, 6, 11, 2, 7, 12, 3, 8, 13, 4, 9, 14, 5, 10, 15]\n th_results = torch.cat(all_transforms, 1)[:, reorder_idxs]\n th_results_global = th_results\n\n joint_js = torch.cat([th_j, th_j.new_zeros(th_j.shape[0], 16, 1)], 2)\n tmp2 = torch.matmul(th_results, joint_js.unsqueeze(3))\n th_results2 = (th_results - torch.cat([tmp2.new_zeros(*tmp2.shape[:2], 4, 3), tmp2], 3)).permute(0, 2, 3, 1)\n\n th_T = torch.matmul(th_results2, self.th_weights.transpose(0, 1))\n\n th_rest_shape_h = torch.cat([\n th_v_posed.transpose(2, 1),\n torch.ones((batch_size, 1, th_v_posed.shape[1]),\n dtype=th_T.dtype,\n device=th_T.device),\n ], 1)\n\n th_verts = (th_T * th_rest_shape_h.unsqueeze(1)).sum(2).transpose(2, 1)\n th_verts = th_verts[:, :, :3]\n th_jtr = th_results_global[:, :, :3, 3]\n # In addition to MANO reference joints we sample vertices on each finger\n # to serve as finger tips\n if self.side == 'right':\n tips = th_verts[:, [745, 317, 444, 556, 673]]\n else:\n tips = th_verts[:, [745, 317, 445, 556, 673]]\n if bool(root_palm):\n palm = (th_verts[:, 95] + th_verts[:, 22]).unsqueeze(1) / 2\n th_jtr = torch.cat([palm, th_jtr[:, 1:]], 1)\n th_jtr = torch.cat([th_jtr, tips], 1)\n\n # Reorder joints to match visualization utilities\n th_jtr = th_jtr[:, [0, 13, 14, 15, 16, 1, 2, 3, 17, 4, 5, 6, 18, 10, 11, 12, 19, 7, 8, 9, 20]]\n\n if th_trans is None or bool(torch.norm(th_trans) == 0):\n if self.center_idx is not None:\n center_joint = th_jtr[:, self.center_idx].unsqueeze(1)\n th_jtr = th_jtr - center_joint\n th_verts = th_verts - center_joint\n else:\n th_jtr = th_jtr + th_trans.unsqueeze(1)\n th_verts = th_verts + th_trans.unsqueeze(1)\n\n # Scale to milimeters\n th_verts = th_verts * 1000\n th_jtr = th_jtr * 1000\n return th_verts, th_jtr, th_results_global\n" }, { "path": "common/utils/manopth/manopth/rodrigues_layer.py", "content": "\"\"\"\nThis part reuses code from https://github.com/MandyMo/pytorch_HMR/blob/master/src/util.py\nwhich is part of a PyTorch port of SMPL.\nThanks to Zhang Xiong (MandyMo) for making this great code available on github !\n\"\"\"\n\nimport argparse\nfrom torch.autograd import gradcheck\nimport torch\nfrom torch.autograd import Variable\n\nfrom manopth import argutils\n\n\ndef quat2mat(quat):\n \"\"\"Convert quaternion coefficients to rotation matrix.\n Args:\n quat: size = [batch_size, 4] 4 <===>(w, x, y, z)\n Returns:\n Rotation matrix corresponding to the quaternion -- size = [batch_size, 3, 3]\n \"\"\"\n norm_quat = quat\n norm_quat = norm_quat / norm_quat.norm(p=2, dim=1, keepdim=True)\n w, x, y, z = norm_quat[:, 0], norm_quat[:, 1], norm_quat[:,\n 2], norm_quat[:,\n 3]\n\n batch_size = quat.size(0)\n\n w2, x2, y2, z2 = w.pow(2), x.pow(2), y.pow(2), z.pow(2)\n wx, wy, wz = w * x, w * y, w * z\n xy, xz, yz = x * y, x * z, y * z\n\n rotMat = torch.stack([\n w2 + x2 - y2 - z2, 2 * xy - 2 * wz, 2 * wy + 2 * xz, 2 * wz + 2 * xy,\n w2 - x2 + y2 - z2, 2 * yz - 2 * wx, 2 * xz - 2 * wy, 2 * wx + 2 * yz,\n w2 - x2 - y2 + z2\n ],\n dim=1).view(batch_size, 3, 3)\n return rotMat\n\n\ndef batch_rodrigues(axisang):\n #axisang N x 3\n axisang_norm = torch.norm(axisang + 1e-8, p=2, dim=1)\n angle = torch.unsqueeze(axisang_norm, -1)\n axisang_normalized = torch.div(axisang, angle)\n angle = angle * 0.5\n v_cos = torch.cos(angle)\n v_sin = torch.sin(angle)\n quat = torch.cat([v_cos, v_sin * axisang_normalized], dim=1)\n rot_mat = quat2mat(quat)\n rot_mat = rot_mat.view(rot_mat.shape[0], 9)\n return rot_mat\n\n\ndef th_get_axis_angle(vector):\n angle = torch.norm(vector, 2, 1)\n axes = vector / angle.unsqueeze(1)\n return axes, angle\n\n\nif __name__ == '__main__':\n parser = argparse.ArgumentParser()\n parser.add_argument('--batch_size', default=1, type=int)\n parser.add_argument('--cuda', action='store_true')\n args = parser.parse_args()\n\n argutils.print_args(args)\n\n n_components = 6\n rot = 3\n inputs = torch.rand(args.batch_size, rot)\n inputs_var = Variable(inputs.double(), requires_grad=True)\n if args.cuda:\n inputs = inputs.cuda()\n # outputs = batch_rodrigues(inputs)\n test_function = gradcheck(batch_rodrigues, (inputs_var, ))\n print('batch test passed !')\n\n inputs = torch.rand(rot)\n inputs_var = Variable(inputs.double(), requires_grad=True)\n test_function = gradcheck(th_cv2_rod_sub_id.apply, (inputs_var, ))\n print('th_cv2_rod test passed')\n\n inputs = torch.rand(rot)\n inputs_var = Variable(inputs.double(), requires_grad=True)\n test_th = gradcheck(th_cv2_rod.apply, (inputs_var, ))\n print('th_cv2_rod_id test passed !')\n" }, { "path": "common/utils/manopth/manopth/rot6d.py", "content": "import torch\n\n\ndef compute_rotation_matrix_from_ortho6d(poses):\n \"\"\"\n Code from\n https://github.com/papagina/RotationContinuity\n On the Continuity of Rotation Representations in Neural Networks\n Zhou et al. CVPR19\n https://zhouyisjtu.github.io/project_rotation/rotation.html\n \"\"\"\n x_raw = poses[:, 0:3] # batch*3\n y_raw = poses[:, 3:6] # batch*3\n \n x = normalize_vector(x_raw) # batch*3\n z = cross_product(x, y_raw) # batch*3\n z = normalize_vector(z) # batch*3\n y = cross_product(z, x) # batch*3\n \n x = x.view(-1, 3, 1)\n y = y.view(-1, 3, 1)\n z = z.view(-1, 3, 1)\n matrix = torch.cat((x, y, z), 2) # batch*3*3\n return matrix\n\ndef robust_compute_rotation_matrix_from_ortho6d(poses):\n \"\"\"\n Instead of making 2nd vector orthogonal to first\n create a base that takes into account the two predicted\n directions equally\n \"\"\"\n x_raw = poses[:, 0:3] # batch*3\n y_raw = poses[:, 3:6] # batch*3\n\n x = normalize_vector(x_raw) # batch*3\n y = normalize_vector(y_raw) # batch*3\n middle = normalize_vector(x + y)\n orthmid = normalize_vector(x - y)\n x = normalize_vector(middle + orthmid)\n y = normalize_vector(middle - orthmid)\n # Their scalar product should be small !\n # assert torch.einsum(\"ij,ij->i\", [x, y]).abs().max() < 0.00001\n z = normalize_vector(cross_product(x, y))\n\n x = x.view(-1, 3, 1)\n y = y.view(-1, 3, 1)\n z = z.view(-1, 3, 1)\n matrix = torch.cat((x, y, z), 2) # batch*3*3\n # Check for reflection in matrix ! If found, flip last vector TODO\n assert (torch.stack([torch.det(mat) for mat in matrix ])< 0).sum() == 0\n return matrix\n\n\ndef normalize_vector(v):\n batch = v.shape[0]\n v_mag = torch.sqrt(v.pow(2).sum(1)) # batch\n v_mag = torch.max(v_mag, v.new([1e-8]))\n v_mag = v_mag.view(batch, 1).expand(batch, v.shape[1])\n v = v/v_mag\n return v\n\n\ndef cross_product(u, v):\n batch = u.shape[0]\n i = u[:, 1] * v[:, 2] - u[:, 2] * v[:, 1]\n j = u[:, 2] * v[:, 0] - u[:, 0] * v[:, 2]\n k = u[:, 0] * v[:, 1] - u[:, 1] * v[:, 0]\n \n out = torch.cat((i.view(batch, 1), j.view(batch, 1), k.view(batch, 1)), 1)\n \n return out\n" }, { "path": "common/utils/manopth/manopth/rotproj.py", "content": "import torch\n\n\ndef batch_rotprojs(batches_rotmats):\n proj_rotmats = []\n for batch_idx, batch_rotmats in enumerate(batches_rotmats):\n proj_batch_rotmats = []\n for rot_idx, rotmat in enumerate(batch_rotmats):\n # GPU implementation of svd is VERY slow\n # ~ 2 10^-3 per hit vs 5 10^-5 on cpu\n U, S, V = rotmat.cpu().svd()\n rotmat = torch.matmul(U, V.transpose(0, 1))\n orth_det = rotmat.det()\n # Remove reflection\n if orth_det < 0:\n rotmat[:, 2] = -1 * rotmat[:, 2]\n\n rotmat = rotmat.cuda()\n proj_batch_rotmats.append(rotmat)\n proj_rotmats.append(torch.stack(proj_batch_rotmats))\n return torch.stack(proj_rotmats)\n" }, { "path": "common/utils/manopth/manopth/tensutils.py", "content": "import torch\n\nfrom manopth import rodrigues_layer\n\n\ndef th_posemap_axisang(pose_vectors):\n rot_nb = int(pose_vectors.shape[1] / 3)\n pose_vec_reshaped = pose_vectors.contiguous().view(-1, 3)\n rot_mats = rodrigues_layer.batch_rodrigues(pose_vec_reshaped)\n rot_mats = rot_mats.view(pose_vectors.shape[0], rot_nb * 9)\n pose_maps = subtract_flat_id(rot_mats)\n return pose_maps, rot_mats\n\n\ndef th_with_zeros(tensor):\n batch_size = tensor.shape[0]\n padding = tensor.new([0.0, 0.0, 0.0, 1.0])\n padding.requires_grad = False\n\n concat_list = [tensor, padding.view(1, 1, 4).repeat(batch_size, 1, 1)]\n cat_res = torch.cat(concat_list, 1)\n return cat_res\n\n\ndef th_pack(tensor):\n batch_size = tensor.shape[0]\n padding = tensor.new_zeros((batch_size, 4, 3))\n padding.requires_grad = False\n pack_list = [padding, tensor]\n pack_res = torch.cat(pack_list, 2)\n return pack_res\n\n\ndef subtract_flat_id(rot_mats):\n # Subtracts identity as a flattened tensor\n rot_nb = int(rot_mats.shape[1] / 9)\n id_flat = torch.eye(\n 3, dtype=rot_mats.dtype, device=rot_mats.device).view(1, 9).repeat(\n rot_mats.shape[0], rot_nb)\n # id_flat.requires_grad = False\n results = rot_mats - id_flat\n return results\n\n\ndef make_list(tensor):\n # type: (List[int]) -> List[int]\n return tensor\n" }, { "path": "common/utils/manopth/setup.py", "content": "from setuptools import find_packages, setup\nimport warnings\n\nDEPENDENCY_PACKAGE_NAMES = [\"matplotlib\", \"torch\", \"tqdm\", \"numpy\", \"cv2\",\n \"chumpy\"]\n\n\ndef check_dependencies():\n missing_dependencies = []\n for package_name in DEPENDENCY_PACKAGE_NAMES:\n try:\n __import__(package_name)\n except ImportError:\n missing_dependencies.append(package_name)\n\n if missing_dependencies:\n warnings.warn(\n 'Missing dependencies: {}. We recommend you follow '\n 'the installation instructions at '\n 'https://github.com/hassony2/manopth#installation'.format(\n missing_dependencies))\n\n\nwith open(\"README.md\", \"r\") as fh:\n long_description = fh.read()\n\ncheck_dependencies()\n\nsetup(\n name=\"manopth\",\n version=\"0.0.1\",\n author=\"Yana Hasson\",\n author_email=\"yana.hasson.inria@gmail.com\",\n packages=find_packages(exclude=('tests',)),\n python_requires=\">=3.5.0\",\n description=\"PyTorch mano layer\",\n long_description=long_description,\n long_description_content_type=\"text/markdown\",\n url=\"https://github.com/hassony2/manopth\",\n classifiers=[\n \"Programming Language :: Python :: 3\",\n \"License :: OSI Approved :: GNU GENERAL PUBLIC LICENSE\",\n \"Operating System :: OS Independent\",\n ],\n)\n" }, { "path": "common/utils/manopth/test/test_demo.py", "content": "import torch\n\nfrom manopth.demo import generate_random_hand\n\n\ndef test_generate_random_hand():\n batch_size = 3\n hand_info = generate_random_hand(batch_size=batch_size, ncomps=6)\n verts = hand_info['verts']\n joints = hand_info['joints']\n assert verts.shape == (batch_size, 778, 3)\n assert joints.shape == (batch_size, 21, 3)\n" }, { "path": "common/utils/optimizers/__init__.py", "content": "# -*- coding: utf-8 -*-\n\n# Max-Planck-Gesellschaft zur Förderung der Wissenschaften e.V. (MPG) is\n# holder of all proprietary rights on this computer program.\n# You can only use this computer program if you have closed\n# a license agreement with MPG or you get the right to use the computer\n# program from someone who is authorized to grant you that right.\n# Any use of the computer program without a valid license is prohibited and\n# liable to prosecution.\n#\n# Copyright©2019 Max-Planck-Gesellschaft zur Förderung\n# der Wissenschaften e.V. (MPG). acting on behalf of its Max Planck Institute\n# for Intelligent Systems and the Max Planck Institute for Biological\n# Cybernetics. All rights reserved.\n#\n# Contact: ps-license@tuebingen.mpg.de" }, { "path": "common/utils/optimizers/lbfgs_ls.py", "content": "# PyTorch implementation of L-BFGS with Strong Wolfe line search\n# Will be removed once https://github.com/pytorch/pytorch/pull/8824\n# is merged\n\nimport torch\nfrom functools import reduce\n\nfrom torch.optim import Optimizer\n\n\ndef _cubic_interpolate(x1, f1, g1, x2, f2, g2, bounds=None):\n # ported from https://github.com/torch/optim/blob/master/polyinterp.lua\n # Compute bounds of interpolation area\n if bounds is not None:\n xmin_bound, xmax_bound = bounds\n else:\n xmin_bound, xmax_bound = (x1, x2) if x1 <= x2 else (x2, x1)\n\n # Code for most common case: cubic interpolation of 2 points\n # w/ function and derivative values for both\n # Solution in this case (where x2 is the farthest point):\n # d1 = g1 + g2 - 3*(f1-f2)/(x1-x2);\n # d2 = sqrt(d1^2 - g1*g2);\n # min_pos = x2 - (x2 - x1)*((g2 + d2 - d1)/(g2 - g1 + 2*d2));\n # t_new = min(max(min_pos,xmin_bound),xmax_bound);\n d1 = g1 + g2 - 3 * (f1 - f2) / (x1 - x2)\n d2_square = d1 ** 2 - g1 * g2\n if d2_square >= 0:\n d2 = d2_square.sqrt()\n if x1 <= x2:\n min_pos = x2 - (x2 - x1) * ((g2 + d2 - d1) / (g2 - g1 + 2 * d2))\n else:\n min_pos = x1 - (x1 - x2) * ((g1 + d2 - d1) / (g1 - g2 + 2 * d2))\n return min(max(min_pos, xmin_bound), xmax_bound)\n else:\n return (xmin_bound + xmax_bound) / 2.\n\n\ndef _strong_Wolfe(obj_func, x, t, d, f, g, gtd, c1=1e-4, c2=0.9, tolerance_change=1e-9,\n max_iter=20,\n max_ls=25):\n # ported from https://github.com/torch/optim/blob/master/lswolfe.lua\n d_norm = d.abs().max()\n g = g.clone()\n # evaluate objective and gradient using initial step\n f_new, g_new = obj_func(x, t, d)\n ls_func_evals = 1\n gtd_new = g_new.dot(d)\n\n # bracket an interval containing a point satisfying the Wolfe criteria\n t_prev, f_prev, g_prev, gtd_prev = 0, f, g, gtd\n done = False\n ls_iter = 0\n while ls_iter < max_ls:\n # check conditions\n if f_new > (f + c1 * t * gtd) or (ls_iter > 1 and f_new >= f_prev):\n bracket = [t_prev, t]\n bracket_f = [f_prev, f_new]\n bracket_g = [g_prev, g_new.clone()]\n bracket_gtd = [gtd_prev, gtd_new]\n break\n\n if abs(gtd_new) <= -c2 * gtd:\n bracket = [t]\n bracket_f = [f_new]\n bracket_g = [g_new]\n bracket_gtd = [gtd_new]\n done = True\n break\n\n if gtd_new >= 0:\n bracket = [t_prev, t]\n bracket_f = [f_prev, f_new]\n bracket_g = [g_prev, g_new.clone()]\n bracket_gtd = [gtd_prev, gtd_new]\n break\n\n # interpolate\n min_step = t + 0.01 * (t - t_prev)\n max_step = t * 10\n tmp = t\n t = _cubic_interpolate(t_prev, f_prev, gtd_prev, t, f_new, gtd_new,\n bounds=(min_step, max_step))\n\n # next step\n t_prev = tmp\n f_prev = f_new\n g_prev = g_new.clone()\n gtd_prev = gtd_new\n f_new, g_new = obj_func(x, t, d)\n ls_func_evals += 1\n gtd_new = g_new.dot(d)\n ls_iter += 1\n\n # reached max number of iterations?\n if ls_iter == max_ls:\n bracket = [0, t]\n bracket_f = [f, f_new]\n bracket_g = [g, g_new]\n bracket_gtd = [gtd, gtd_new]\n\n # zoom phase: we now have a point satisfying the criteria, or\n # a bracket around it. We refine the bracket until we find the\n # exact point satisfying the criteria\n insuf_progress = False\n # find high and low points in bracket\n low_pos, high_pos = (0, 1) if bracket_f[0] <= bracket_f[-1] else (1, 0)\n while not done and ls_iter < max_iter:\n # compute new trial value\n t = _cubic_interpolate(bracket[0], bracket_f[0], bracket_gtd[0],\n bracket[1], bracket_f[1], bracket_gtd[1])\n\n # test what we are making sufficient progress\n eps = 0.1 * (max(bracket) - min(bracket))\n if min(max(bracket) - t, t - min(bracket)) < eps:\n # interpolation close to boundary\n if insuf_progress or t >= max(bracket) or t <= min(bracket):\n # evaluate at 0.1 away from boundary\n if abs(t - max(bracket)) < abs(t - min(bracket)):\n t = max(bracket) - eps\n else:\n t = min(bracket) + eps\n insuf_progress = False\n else:\n insuf_progress = True\n else:\n insuf_progress = False\n\n # Evaluate new point\n f_new, g_new = obj_func(x, t, d)\n ls_func_evals += 1\n gtd_new = g_new.dot(d)\n ls_iter += 1\n\n if f_new > (f + c1 * t * gtd) or f_new >= bracket_f[low_pos]:\n # Armijo condition not satisfied or not lower than lowest point\n bracket[high_pos] = t\n bracket_f[high_pos] = f_new\n bracket_g[high_pos] = g_new.clone()\n bracket_gtd[high_pos] = gtd_new\n low_pos, high_pos = (0, 1) if bracket_f[0] <= bracket_f[1] else (1, 0)\n else:\n if abs(gtd_new) <= -c2 * gtd:\n # Wolfe conditions satisfied\n done = True\n elif gtd_new * (bracket[high_pos] - bracket[low_pos]) >= 0:\n # old high becomes new low\n bracket[high_pos] = bracket[low_pos]\n bracket_f[high_pos] = bracket_f[low_pos]\n bracket_g[high_pos] = bracket_g[low_pos]\n bracket_gtd[high_pos] = bracket_gtd[low_pos]\n\n # new point becomes new low\n bracket[low_pos] = t\n bracket_f[low_pos] = f_new\n bracket_g[low_pos] = g_new.clone()\n bracket_gtd[low_pos] = gtd_new\n\n # line-search bracket is so small\n if abs(bracket[1] - bracket[0]) * d_norm < tolerance_change:\n break\n\n # return stuff\n t = bracket[low_pos]\n f_new = bracket_f[low_pos]\n g_new = bracket_g[low_pos]\n return f_new, g_new, t, ls_func_evals\n\n\n# LBFGS with strong Wolfe line search introduces in PR #8824\n# Will be removed once merged with master\nclass LBFGS(Optimizer):\n \"\"\"Implements L-BFGS algorithm, heavily inspired by `minFunc\n `.\n .. warning::\n This optimizer doesn't support per-parameter options and parameter\n groups (there can be only one).\n .. warning::\n Right now all parameters have to be on a single device. This will be\n improved in the future.\n .. note::\n This is a very memory intensive optimizer (it requires additional\n ``param_bytes * (history_size + 1)`` bytes). If it doesn't fit in memory\n try reducing the history size, or use a different algorithm.\n Arguments:\n lr (float): learning rate (default: 1)\n max_iter (int): maximal number of iterations per optimization step\n (default: 20)\n max_eval (int): maximal number of function evaluations per optimization\n step (default: max_iter * 1.25).\n tolerance_grad (float): termination tolerance on first order optimality\n (default: 1e-5).\n tolerance_change (float): termination tolerance on function\n value/parameter changes (default: 1e-9).\n history_size (int): update history size (default: 100).\n line_search_fn (str): either 'strong_Wolfe' or None (default: None).\n \"\"\"\n\n def __init__(self, params, lr=1, max_iter=20, max_eval=None,\n tolerance_grad=1e-5, tolerance_change=1e-9, history_size=100,\n line_search_fn=None):\n if max_eval is None:\n max_eval = max_iter * 5 // 4\n defaults = dict(lr=lr, max_iter=max_iter, max_eval=max_eval,\n tolerance_grad=tolerance_grad, tolerance_change=tolerance_change,\n history_size=history_size, line_search_fn=line_search_fn)\n super(LBFGS, self).__init__(params, defaults)\n\n if len(self.param_groups) != 1:\n raise ValueError(\"LBFGS doesn't support per-parameter options \"\n \"(parameter groups)\")\n\n self._params = self.param_groups[0]['params']\n self._numel_cache = None\n\n def _numel(self):\n if self._numel_cache is None:\n self._numel_cache = reduce(lambda total, p: total + p.numel(), self._params, 0)\n return self._numel_cache\n\n def _gather_flat_grad(self):\n views = []\n for p in self._params:\n if p.grad is None:\n view = p.new(p.numel()).zero_()\n elif p.grad.is_sparse:\n view = p.grad.to_dense().view(-1)\n else:\n view = p.grad.view(-1)\n views.append(view)\n return torch.cat(views, 0)\n\n def _add_grad(self, step_size, update):\n offset = 0\n for p in self._params:\n numel = p.numel()\n # view as to avoid deprecated pointwise semantics\n p.data.add_(step_size, update[offset:offset + numel].view_as(p.data))\n offset += numel\n assert offset == self._numel()\n\n def _clone_param(self):\n return [p.clone() for p in self._params]\n\n def _set_param(self, params_data):\n for p, pdata in zip(self._params, params_data):\n p.data.copy_(pdata)\n\n def _directional_evaluate(self, closure, x, t, d):\n self._add_grad(t, d)\n loss = float(closure())\n flat_grad = self._gather_flat_grad()\n self._set_param(x)\n return loss, flat_grad\n\n def step(self, closure):\n \"\"\"Performs a single optimization step.\n Arguments:\n closure (callable): A closure that reevaluates the model\n and returns the loss.\n \"\"\"\n assert len(self.param_groups) == 1\n\n group = self.param_groups[0]\n lr = group['lr']\n max_iter = group['max_iter']\n max_eval = group['max_eval']\n tolerance_grad = group['tolerance_grad']\n tolerance_change = group['tolerance_change']\n line_search_fn = group['line_search_fn']\n history_size = group['history_size']\n\n # NOTE: LBFGS has only global state, but we register it as state for\n # the first param, because this helps with casting in load_state_dict\n state = self.state[self._params[0]]\n state.setdefault('func_evals', 0)\n state.setdefault('n_iter', 0)\n\n # evaluate initial f(x) and df/dx\n orig_loss = closure()\n loss = float(orig_loss)\n current_evals = 1\n state['func_evals'] += 1\n\n flat_grad = self._gather_flat_grad()\n opt_cond = flat_grad.abs().max() <= tolerance_grad\n\n # optimal condition\n if opt_cond:\n return orig_loss\n\n # tensors cached in state (for tracing)\n d = state.get('d')\n t = state.get('t')\n old_dirs = state.get('old_dirs')\n old_stps = state.get('old_stps')\n ro = state.get('ro')\n H_diag = state.get('H_diag')\n prev_flat_grad = state.get('prev_flat_grad')\n prev_loss = state.get('prev_loss')\n\n n_iter = 0\n # optimize for a max of max_iter iterations\n while n_iter < max_iter:\n # keep track of nb of iterations\n n_iter += 1\n state['n_iter'] += 1\n\n ############################################################\n # compute gradient descent direction\n ############################################################\n if state['n_iter'] == 1:\n d = flat_grad.neg()\n old_dirs = []\n old_stps = []\n ro = []\n H_diag = 1\n else:\n # do lbfgs update (update memory)\n y = flat_grad.sub(prev_flat_grad)\n s = d.mul(t)\n ys = y.dot(s) # y*s\n if ys > 1e-10:\n # updating memory\n if len(old_dirs) == history_size:\n # shift history by one (limited-memory)\n old_dirs.pop(0)\n old_stps.pop(0)\n ro.pop(0)\n\n # store new direction/step\n old_dirs.append(y)\n old_stps.append(s)\n ro.append(1. / ys)\n\n # update scale of initial Hessian approximation\n H_diag = ys / y.dot(y) # (y*y)\n\n # compute the approximate (L-BFGS) inverse Hessian\n # multiplied by the gradient\n num_old = len(old_dirs)\n\n if 'al' not in state:\n state['al'] = [None] * history_size\n al = state['al']\n\n # iteration in L-BFGS loop collapsed to use just one buffer\n q = flat_grad.neg()\n for i in range(num_old - 1, -1, -1):\n al[i] = old_stps[i].dot(q) * ro[i]\n q.add_(-al[i], old_dirs[i])\n\n # multiply by initial Hessian\n # r/d is the final direction\n d = r = torch.mul(q, H_diag)\n for i in range(num_old):\n be_i = old_dirs[i].dot(r) * ro[i]\n r.add_(al[i] - be_i, old_stps[i])\n\n if prev_flat_grad is None:\n prev_flat_grad = flat_grad.clone()\n else:\n prev_flat_grad.copy_(flat_grad)\n prev_loss = loss\n\n ############################################################\n # compute step length\n ############################################################\n # reset initial guess for step size\n if state['n_iter'] == 1:\n t = min(1., 1. / flat_grad.abs().sum()) * lr\n else:\n t = lr\n\n # directional derivative\n gtd = flat_grad.dot(d) # g * d\n\n # directional derivative is below tolerance\n if gtd > -tolerance_change:\n break\n\n # optional line search: user function\n ls_func_evals = 0\n if line_search_fn is not None:\n # perform line search, using user function\n if line_search_fn != \"strong_Wolfe\":\n raise RuntimeError(\"only 'strong_Wolfe' is supported\")\n else:\n x_init = self._clone_param()\n\n def obj_func(x, t, d):\n return self._directional_evaluate(closure, x, t, d)\n loss, flat_grad, t, ls_func_evals = _strong_Wolfe(obj_func, x_init, t, d,\n loss,\n flat_grad,\n gtd,\n max_iter=max_iter)\n self._add_grad(t, d)\n opt_cond = flat_grad.abs().max() <= tolerance_grad\n else:\n # no line search, simply move with fixed-step\n self._add_grad(t, d)\n if n_iter != max_iter:\n # re-evaluate function only if not in last iteration\n # the reason we do this: in a stochastic setting,\n # no use to re-evaluate that function here\n loss = float(closure())\n flat_grad = self._gather_flat_grad()\n opt_cond = flat_grad.abs().max() <= tolerance_grad\n ls_func_evals = 1\n\n # update func eval\n current_evals += ls_func_evals\n state['func_evals'] += ls_func_evals\n\n ############################################################\n # check conditions\n ############################################################\n if n_iter == max_iter:\n break\n\n if current_evals >= max_eval:\n break\n\n # optimal condition\n if opt_cond:\n break\n\n # lack of progress\n if d.mul(t).abs().max() <= tolerance_change:\n break\n\n if abs(loss - prev_loss) < tolerance_change:\n break\n\n state['d'] = d\n state['t'] = t\n state['old_dirs'] = old_dirs\n state['old_stps'] = old_stps\n state['ro'] = ro\n state['H_diag'] = H_diag\n state['prev_flat_grad'] = prev_flat_grad\n state['prev_loss'] = prev_loss\n\n return orig_loss" }, { "path": "common/utils/optimizers/optim_factory.py", "content": "# -*- coding: utf-8 -*-\n\n# Max-Planck-Gesellschaft zur Förderung der Wissenschaften e.V. (MPG) is\n# holder of all proprietary rights on this computer program.\n# You can only use this computer program if you have closed\n# a license agreement with MPG or you get the right to use the computer\n# program from someone who is authorized to grant you that right.\n# Any use of the computer program without a valid license is prohibited and\n# liable to prosecution.\n#\n# Copyright©2019 Max-Planck-Gesellschaft zur Förderung\n# der Wissenschaften e.V. (MPG). acting on behalf of its Max Planck Institute\n# for Intelligent Systems and the Max Planck Institute for Biological\n# Cybernetics. All rights reserved.\n#\n# Contact: ps-license@tuebingen.mpg.de\n\n\nfrom __future__ import absolute_import\nfrom __future__ import print_function\nfrom __future__ import division\n\nimport torch.optim as optim\nfrom .lbfgs_ls import LBFGS as LBFGSLs\n\n\ndef create_optimizer(parameters, optim_type='lbfgs',\n lr=1e-3,\n momentum=0.9,\n use_nesterov=True,\n beta1=0.9,\n beta2=0.999,\n epsilon=1e-8,\n use_locking=False,\n weight_decay=0.0,\n centered=False,\n rmsprop_alpha=0.99,\n maxiters=20,\n gtol=1e-6,\n ftol=1e-9,\n **kwargs):\n ''' Creates the optimizer\n '''\n if optim_type == 'adam':\n return (optim.Adam(parameters, lr=lr, betas=(beta1, beta2),\n weight_decay=weight_decay),\n False)\n elif optim_type == 'lbfgs':\n return (optim.LBFGS(parameters, lr=lr, max_iter=maxiters), False)\n elif optim_type == 'lbfgsls':\n return LBFGSLs(parameters, lr=lr, max_iter=maxiters,\n line_search_fn='strong_Wolfe'), False\n elif optim_type == 'rmsprop':\n return (optim.RMSprop(parameters, lr=lr, epsilon=epsilon,\n alpha=rmsprop_alpha,\n weight_decay=weight_decay,\n momentum=momentum, centered=centered),\n False)\n elif optim_type == 'sgd':\n return (optim.SGD(parameters, lr=lr, momentum=momentum,\n weight_decay=weight_decay,\n nesterov=use_nesterov),\n False)\n else:\n raise ValueError('Optimizer {} not supported!'.format(optim_type))" }, { "path": "common/utils/preprocessing.py", "content": "import numpy as np\nimport cv2\nimport random\nfrom config import cfg\nimport math\nimport torchvision\n\ndef load_img(path, order='RGB'):\n img = cv2.imread(path, cv2.IMREAD_COLOR | cv2.IMREAD_IGNORE_ORIENTATION)\n if not isinstance(img, np.ndarray):\n raise IOError(\"Fail to read %s\" % path)\n\n if order=='RGB':\n img = img[:,:,::-1].copy()\n \n img = img.astype(np.float32)\n return img\n\ndef get_bbox(joint_img, joint_valid, expansion_factor=1.0):\n\n x_img, y_img = joint_img[:,0], joint_img[:,1]\n x_img = x_img[joint_valid==1]; y_img = y_img[joint_valid==1];\n xmin = min(x_img); ymin = min(y_img); xmax = max(x_img); ymax = max(y_img);\n\n x_center = (xmin+xmax)/2.; width = (xmax-xmin)*expansion_factor;\n xmin = x_center - 0.5*width\n xmax = x_center + 0.5*width\n \n y_center = (ymin+ymax)/2.; height = (ymax-ymin)*expansion_factor;\n ymin = y_center - 0.5*height\n ymax = y_center + 0.5*height\n\n bbox = np.array([xmin, ymin, xmax - xmin, ymax - ymin]).astype(np.float32)\n return bbox\n\ndef process_bbox(bbox, img_width, img_height, expansion_factor=1.25):\n # sanitize bboxes\n x, y, w, h = bbox\n x1 = np.max((0, x))\n y1 = np.max((0, y))\n x2 = np.min((img_width - 1, x1 + np.max((0, w - 1))))\n y2 = np.min((img_height - 1, y1 + np.max((0, h - 1))))\n if w*h > 0 and x2 >= x1 and y2 >= y1:\n bbox = np.array([x1, y1, x2-x1, y2-y1])\n else:\n return None\n\n # aspect ratio preserving bbox\n w = bbox[2]\n h = bbox[3]\n c_x = bbox[0] + w/2.\n c_y = bbox[1] + h/2.\n aspect_ratio = cfg.input_img_shape[1]/cfg.input_img_shape[0]\n if w > aspect_ratio * h:\n h = w / aspect_ratio\n elif w < aspect_ratio * h:\n w = h * aspect_ratio\n bbox[2] = w*expansion_factor\n bbox[3] = h*expansion_factor\n bbox[0] = c_x - bbox[2]/2.\n bbox[1] = c_y - bbox[3]/2.\n\n return bbox\n\ndef get_aug_config():\n scale_factor = 0.25\n rot_factor = 30\n color_factor = 0.2\n \n scale = np.clip(np.random.randn(), -1.0, 1.0) * scale_factor + 1.0\n rot = np.clip(np.random.randn(), -2.0,\n 2.0) * rot_factor if random.random() <= 0.6 else 0\n c_up = 1.0 + color_factor\n c_low = 1.0 - color_factor\n color_scale = np.array([random.uniform(c_low, c_up), random.uniform(c_low, c_up), random.uniform(c_low, c_up)])\n\n return scale, rot, color_scale\n\ndef augmentation(img, bbox, data_split, do_flip=False):\n if data_split == 'train':\n scale, rot, color_scale = get_aug_config()\n else:\n scale, rot, color_scale = 1.0, 0.0, np.array([1,1,1])\n img, trans, inv_trans = generate_patch_image(img, bbox, scale, rot, do_flip, cfg.input_img_shape)\n\n img = np.clip(img * color_scale[None,None,:], 0, 255)\n return img, trans, inv_trans, rot, scale\n\ndef generate_patch_image(cvimg, bbox, scale, rot, do_flip, out_shape):\n img = cvimg.copy()\n img_height, img_width, img_channels = img.shape\n \n bb_c_x = float(bbox[0] + 0.5*bbox[2])\n bb_c_y = float(bbox[1] + 0.5*bbox[3])\n bb_width = float(bbox[2])\n bb_height = float(bbox[3])\n\n if do_flip:\n img = img[:, ::-1, :]\n bb_c_x = img_width - bb_c_x - 1\n\n trans = gen_trans_from_patch_cv(bb_c_x, bb_c_y, bb_width, bb_height, out_shape[1], out_shape[0], scale, rot)\n img_patch = cv2.warpAffine(img, trans, (int(out_shape[1]), int(out_shape[0])), flags=cv2.INTER_LINEAR)\n img_patch = img_patch.astype(np.float32)\n inv_trans = gen_trans_from_patch_cv(bb_c_x, bb_c_y, bb_width, bb_height, out_shape[1], out_shape[0], scale, rot, inv=True)\n\n return img_patch, trans, inv_trans\n\ndef rotate_2d(pt_2d, rot_rad):\n x = pt_2d[0]\n y = pt_2d[1]\n sn, cs = np.sin(rot_rad), np.cos(rot_rad)\n xx = x * cs - y * sn\n yy = x * sn + y * cs\n return np.array([xx, yy], dtype=np.float32)\n\ndef gen_trans_from_patch_cv(c_x, c_y, src_width, src_height, dst_width, dst_height, scale, rot, inv=False):\n # augment size with scale\n src_w = src_width * scale\n src_h = src_height * scale\n src_center = np.array([c_x, c_y], dtype=np.float32)\n\n # augment rotation\n rot_rad = np.pi * rot / 180\n src_downdir = rotate_2d(np.array([0, src_h * 0.5], dtype=np.float32), rot_rad)\n src_rightdir = rotate_2d(np.array([src_w * 0.5, 0], dtype=np.float32), rot_rad)\n\n dst_w = dst_width\n dst_h = dst_height\n dst_center = np.array([dst_w * 0.5, dst_h * 0.5], dtype=np.float32)\n dst_downdir = np.array([0, dst_h * 0.5], dtype=np.float32)\n dst_rightdir = np.array([dst_w * 0.5, 0], dtype=np.float32)\n\n src = np.zeros((3, 2), dtype=np.float32)\n src[0, :] = src_center\n src[1, :] = src_center + src_downdir\n src[2, :] = src_center + src_rightdir\n\n dst = np.zeros((3, 2), dtype=np.float32)\n dst[0, :] = dst_center\n dst[1, :] = dst_center + dst_downdir\n dst[2, :] = dst_center + dst_rightdir\n \n if inv:\n trans = cv2.getAffineTransform(np.float32(dst), np.float32(src))\n else:\n trans = cv2.getAffineTransform(np.float32(src), np.float32(dst))\n\n trans = trans.astype(np.float32)\n return trans\n" }, { "path": "common/utils/transforms.py", "content": "import torch\nimport numpy as np\nfrom config import cfg\n\ndef cam2pixel(cam_coord, f, c):\n x = cam_coord[:,0] / cam_coord[:,2] * f[0] + c[0]\n y = cam_coord[:,1] / cam_coord[:,2] * f[1] + c[1]\n z = cam_coord[:,2]\n return np.stack((x,y,z),1)\n\ndef pixel2cam(pixel_coord, f, c):\n x = (pixel_coord[:,0] - c[0]) / f[0] * pixel_coord[:,2]\n y = (pixel_coord[:,1] - c[1]) / f[1] * pixel_coord[:,2]\n z = pixel_coord[:,2]\n return np.stack((x,y,z),1)\n\ndef world2cam(world_coord, R, t):\n cam_coord = np.dot(R, world_coord.transpose(1,0)).transpose(1,0) + t.reshape(1,3)\n return cam_coord\n\ndef cam2world(cam_coord, R, t):\n world_coord = np.dot(np.linalg.inv(R), (cam_coord - t.reshape(1,3)).transpose(1,0)).transpose(1,0)\n return world_coord\n\ndef rigid_transform_3D(A, B):\n n, dim = A.shape\n centroid_A = np.mean(A, axis = 0)\n centroid_B = np.mean(B, axis = 0)\n H = np.dot(np.transpose(A - centroid_A), B - centroid_B) / n\n U, s, V = np.linalg.svd(H)\n R = np.dot(np.transpose(V), np.transpose(U))\n if np.linalg.det(R) < 0:\n s[-1] = -s[-1]\n V[2] = -V[2]\n R = np.dot(np.transpose(V), np.transpose(U))\n\n varP = np.var(A, axis=0).sum()\n c = 1/varP * np.sum(s) \n\n t = -np.dot(c*R, np.transpose(centroid_A)) + np.transpose(centroid_B)\n return c, R, t\n\ndef rigid_align(A, B):\n c, R, t = rigid_transform_3D(A, B)\n A2 = np.transpose(np.dot(c*R, np.transpose(A))) + t\n return A2\n\ndef transform_joint_to_other_db(src_joint, src_name, dst_name):\n src_joint_num = len(src_name)\n dst_joint_num = len(dst_name)\n\n new_joint = np.zeros(((dst_joint_num,) + src_joint.shape[1:]), dtype=np.float32)\n for src_idx in range(len(src_name)):\n name = src_name[src_idx]\n if name in dst_name:\n dst_idx = dst_name.index(name)\n new_joint[dst_idx] = src_joint[src_idx]\n\n return new_joint" }, { "path": "common/utils/vis.py", "content": "import os\nimport cv2\nimport numpy as np\nfrom mpl_toolkits.mplot3d import Axes3D\nimport matplotlib.pyplot as plt\nimport matplotlib as mpl\nos.environ[\"PYOPENGL_PLATFORM\"] = \"egl\"\nimport pyrender\nimport trimesh\n\ndef vis_keypoints_with_skeleton(img, kps, kps_lines, kp_thresh=0.4, alpha=1):\n # Convert from plt 0-1 RGBA colors to 0-255 BGR colors for opencv.\n cmap = plt.get_cmap('rainbow')\n colors = [cmap(i) for i in np.linspace(0, 1, len(kps_lines) + 2)]\n colors = [(c[2] * 255, c[1] * 255, c[0] * 255) for c in colors]\n\n # Perform the drawing on a copy of the image, to allow for blending.\n kp_mask = np.copy(img)\n\n # Draw the keypoints.\n for l in range(len(kps_lines)):\n i1 = kps_lines[l][0]\n i2 = kps_lines[l][1]\n p1 = kps[0, i1].astype(np.int32), kps[1, i1].astype(np.int32)\n p2 = kps[0, i2].astype(np.int32), kps[1, i2].astype(np.int32)\n if kps[2, i1] > kp_thresh and kps[2, i2] > kp_thresh:\n cv2.line(\n kp_mask, p1, p2,\n color=colors[l], thickness=2, lineType=cv2.LINE_AA)\n if kps[2, i1] > kp_thresh:\n cv2.circle(\n kp_mask, p1,\n radius=3, color=colors[l], thickness=-1, lineType=cv2.LINE_AA)\n if kps[2, i2] > kp_thresh:\n cv2.circle(\n kp_mask, p2,\n radius=3, color=colors[l], thickness=-1, lineType=cv2.LINE_AA)\n\n # Blend the keypoints.\n return cv2.addWeighted(img, 1.0 - alpha, kp_mask, alpha, 0)\n\ndef vis_keypoints(img, kps, alpha=1):\n # Convert from plt 0-1 RGBA colors to 0-255 BGR colors for opencv.\n cmap = plt.get_cmap('rainbow')\n colors = [cmap(i) for i in np.linspace(0, 1, len(kps) + 2)]\n colors = [(c[2] * 255, c[1] * 255, c[0] * 255) for c in colors]\n\n # Perform the drawing on a copy of the image, to allow for blending.\n kp_mask = np.copy(img)\n\n # Draw the keypoints.\n for i in range(len(kps)):\n p = kps[i][0].astype(np.int32), kps[i][1].astype(np.int32)\n cv2.circle(kp_mask, p, radius=3, color=colors[i], thickness=-1, lineType=cv2.LINE_AA)\n\n # Blend the keypoints.\n return cv2.addWeighted(img, 1.0 - alpha, kp_mask, alpha, 0)\n\ndef vis_mesh(img, mesh_vertex, alpha=0.5):\n # Convert from plt 0-1 RGBA colors to 0-255 BGR colors for opencv.\n cmap = plt.get_cmap('rainbow')\n colors = [cmap(i) for i in np.linspace(0, 1, len(mesh_vertex))]\n colors = [(c[2] * 255, c[1] * 255, c[0] * 255) for c in colors]\n\n # Perform the drawing on a copy of the image, to allow for blending.\n mask = np.copy(img)\n\n # Draw the mesh\n for i in range(len(mesh_vertex)):\n p = mesh_vertex[i][0].astype(np.int32), mesh_vertex[i][1].astype(np.int32)\n cv2.circle(mask, p, radius=1, color=colors[i], thickness=-1, lineType=cv2.LINE_AA)\n\n # Blend the keypoints.\n return cv2.addWeighted(img, 1.0 - alpha, mask, alpha, 0)\n\ndef vis_3d_skeleton(kpt_3d, kpt_3d_vis, kps_lines, filename=None):\n\n fig = plt.figure()\n ax = fig.add_subplot(111, projection='3d')\n\n # Convert from plt 0-1 RGBA colors to 0-255 BGR colors for opencv.\n cmap = plt.get_cmap('rainbow')\n colors = [cmap(i) for i in np.linspace(0, 1, len(kps_lines) + 2)]\n colors = [np.array((c[2], c[1], c[0])) for c in colors]\n\n for l in range(len(kps_lines)):\n i1 = kps_lines[l][0]\n i2 = kps_lines[l][1]\n x = np.array([kpt_3d[i1,0], kpt_3d[i2,0]])\n y = np.array([kpt_3d[i1,1], kpt_3d[i2,1]])\n z = np.array([kpt_3d[i1,2], kpt_3d[i2,2]])\n\n if kpt_3d_vis[i1,0] > 0 and kpt_3d_vis[i2,0] > 0:\n ax.plot(x, z, -y, c=colors[l], linewidth=2)\n if kpt_3d_vis[i1,0] > 0:\n ax.scatter(kpt_3d[i1,0], kpt_3d[i1,2], -kpt_3d[i1,1], c=colors[l], marker='o')\n if kpt_3d_vis[i2,0] > 0:\n ax.scatter(kpt_3d[i2,0], kpt_3d[i2,2], -kpt_3d[i2,1], c=colors[l], marker='o')\n\n if filename is None:\n ax.set_title('3D vis')\n else:\n ax.set_title(filename)\n\n ax.set_xlabel('X Label')\n ax.set_ylabel('Z Label')\n ax.set_zlabel('Y Label')\n ax.legend()\n\n #plt.show()\n #cv2.waitKey(0)\n\n plt.savefig(filename)\n\ndef save_obj(v, f, file_name='output.obj'):\n obj_file = open(file_name, 'w')\n for i in range(len(v)):\n obj_file.write('v ' + str(v[i][0]) + ' ' + str(v[i][1]) + ' ' + str(v[i][2]) + '\\n')\n for i in range(len(f)):\n obj_file.write('f ' + str(f[i][0]+1) + '/' + str(f[i][0]+1) + ' ' + str(f[i][1]+1) + '/' + str(f[i][1]+1) + ' ' + str(f[i][2]+1) + '/' + str(f[i][2]+1) + '\\n')\n obj_file.close()\n\ndef render_mesh(img, mesh, face, cam_param):\n # mesh\n mesh = trimesh.Trimesh(mesh, face)\n rot = trimesh.transformations.rotation_matrix(\n\tnp.radians(180), [1, 0, 0])\n mesh.apply_transform(rot)\n material = pyrender.MetallicRoughnessMaterial(metallicFactor=0.0, alphaMode='OPAQUE', baseColorFactor=(1.0, 1.0, 0.9, 1.0))\n mesh = pyrender.Mesh.from_trimesh(mesh, material=material, smooth=False)\n scene = pyrender.Scene(ambient_light=(0.3, 0.3, 0.3))\n scene.add(mesh, 'mesh')\n \n focal, princpt = cam_param['focal'], cam_param['princpt']\n camera = pyrender.IntrinsicsCamera(fx=focal[0], fy=focal[1], cx=princpt[0], cy=princpt[1])\n scene.add(camera)\n \n # renderer\n renderer = pyrender.OffscreenRenderer(viewport_width=img.shape[1], viewport_height=img.shape[0], point_size=1.0)\n \n # light\n light = pyrender.DirectionalLight(color=[1.0, 1.0, 1.0], intensity=0.8)\n light_pose = np.eye(4)\n light_pose[:3, 3] = np.array([0, -1, 1])\n scene.add(light, pose=light_pose)\n light_pose[:3, 3] = np.array([0, 1, 1])\n scene.add(light, pose=light_pose)\n light_pose[:3, 3] = np.array([1, 1, 2])\n scene.add(light, pose=light_pose)\n\n # render\n rgb, depth = renderer.render(scene, flags=pyrender.RenderFlags.RGBA)\n rgb = rgb[:,:,:3].astype(np.float32)\n valid_mask = (depth > 0)[:,:,None]\n\n # save to image\n img = rgb * valid_mask*0.5 + img #* (1-valid_mask)\n return img" }, { "path": "data/DEX_YCB/DEX_YCB.py", "content": "import os\nimport os.path as osp\nimport numpy as np\nimport torch\nimport cv2\nimport random\nimport json\nimport math\nimport copy\nfrom pycocotools.coco import COCO\nfrom config import cfg\nfrom utils.preprocessing import load_img, get_bbox, process_bbox, generate_patch_image, augmentation\nfrom utils.transforms import world2cam, cam2pixel, pixel2cam, rigid_align, transform_joint_to_other_db\nfrom utils.vis import vis_keypoints, vis_mesh, save_obj, vis_keypoints_with_skeleton, render_mesh, vis_3d_skeleton\nfrom utils.mano import MANO\nmano = MANO()\n\n# # TEMP; test set\n# target_img_list = {\n# 1: ['20200820-subject-03/20200820_135508/836212060125/color_000030.jpg', '20200820-subject-03/20200820_135508/836212060125/color_000060.jpg', '20200903-subject-04/20200903_103828/836212060125/color_000030.jpg', '20200903-subject-04/20200903_103828/836212060125/color_000060.jpg', '20200908-subject-05/20200908_143535/836212060125/color_000060.jpg', '20200908-subject-05/20200908_143535/932122060857/color_000030.jpg', '20200918-subject-06/20200918_113137/836212060125/color_000030.jpg', '20200918-subject-06/20200918_113137/836212060125/color_000060.jpg',\n# '20200928-subject-07/20200928_143500/836212060125/color_000060.jpg', '20200928-subject-07/20200928_143500/932122060857/color_000030.jpg', '20201002-subject-08/20201002_104827/836212060125/color_000060.jpg', '20201002-subject-08/20201002_104827/932122060861/color_000030.jpg', '20201015-subject-09/20201015_142844/836212060125/color_000030.jpg', '20201015-subject-09/20201015_142844/836212060125/color_000060.jpg', '20201015-subject-09/20201015_142844/841412060263/color_000000.jpg', '20201022-subject-10/20201022_110947/840412060917/color_000060.jpg', '20201022-subject-10/20201022_110947/932122060857/color_000030.jpg'],\n# 2: ['20200820-subject-03/20200820_135810/836212060125/color_000060.jpg', '20200820-subject-03/20200820_135810/839512060362/color_000030.jpg', '20200903-subject-04/20200903_104115/836212060125/color_000030.jpg', '20200903-subject-04/20200903_104115/839512060362/color_000060.jpg', '20200908-subject-05/20200908_143832/836212060125/color_000030.jpg', '20200908-subject-05/20200908_143832/836212060125/color_000060.jpg', '20200918-subject-06/20200918_113405/839512060362/color_000060.jpg', '20200918-subject-06/20200918_113405/840412060917/color_000030.jpg', '20200928-subject-07/20200928_143727/839512060362/color_000060.jpg', '20200928-subject-07/20200928_143727/840412060917/color_000030.jpg', '20201002-subject-08/20201002_105058/836212060125/color_000060.jpg', '20201002-subject-08/20201002_105058/840412060917/color_000030.jpg', '20201015-subject-09/20201015_143113/836212060125/color_000030.jpg', '20201015-subject-09/20201015_143113/836212060125/color_000060.jpg', '20201015-subject-09/20201015_143113/840412060917/color_000000.jpg', '20201022-subject-10/20201022_111144/840412060917/color_000030.jpg', '20201022-subject-10/20201022_111144/840412060917/color_000060.jpg'],\n# 10: ['20200820-subject-03/20200820_142158/932122060861/color_000030.jpg', '20200820-subject-03/20200820_142158/932122060861/color_000060.jpg', '20200903-subject-04/20200903_110342/836212060125/color_000060.jpg', '20200908-subject-05/20200908_145938/836212060125/color_000060.jpg', '20200908-subject-05/20200908_145938/839512060362/color_000030.jpg', '20200918-subject-06/20200918_115139/839512060362/color_000060.jpg', '20200918-subject-06/20200918_115139/840412060917/color_000030.jpg', '20200928-subject-07/20200928_153732/836212060125/color_000030.jpg', '20200928-subject-07/20200928_153732/932122060857/color_000060.jpg', '20201002-subject-08/20201002_110854/836212060125/color_000060.jpg', '20201015-subject-09/20201015_145212/836212060125/color_000030.jpg', '20201015-subject-09/20201015_145212/839512060362/color_000060.jpg'],\n# 15: ['20200820-subject-03/20200820_143802/836212060125/color_000060.jpg', '20200820-subject-03/20200820_143802/840412060917/color_000030.jpg', '20200903-subject-04/20200903_112724/836212060125/color_000060.jpg', '20200903-subject-04/20200903_112724/841412060263/color_000030.jpg', '20200908-subject-05/20200908_151328/836212060125/color_000060.jpg', '20200908-subject-05/20200908_151328/840412060917/color_000030.jpg', '20200918-subject-06/20200918_120310/836212060125/color_000030.jpg', '20200918-subject-06/20200918_120310/836212060125/color_000060.jpg', '20200928-subject-07/20200928_154943/836212060125/color_000030.jpg', '20200928-subject-07/20200928_154943/836212060125/color_000060.jpg', '20201002-subject-08/20201002_112045/836212060125/color_000030.jpg', '20201002-subject-08/20201002_112045/836212060125/color_000060.jpg', '20201015-subject-09/20201015_150413/836212060125/color_000030.jpg', '20201015-subject-09/20201015_150413/836212060125/color_000060.jpg', '20201022-subject-10/20201022_113909/836212060125/color_000060.jpg']\n# }\n# # TEMP; val set\n# # target_img_list = {\n# # 1: ['20200709-subject-01/20200709_142123/836212060125/color_000030.jpg', '20200709-subject-01/20200709_142123/836212060125/color_000060.jpg', '20200813-subject-02/20200813_145612/836212060125/color_000030.jpg', '20200813-subject-02/20200813_145612/836212060125/color_000060.jpg'],\n# # 2: ['20200709-subject-01/20200709_142446/840412060917/color_000030.jpg', '20200709-subject-01/20200709_142446/840412060917/color_000060.jpg', '20200813-subject-02/20200813_145920/836212060125/color_000030.jpg', '20200813-subject-02/20200813_145920/836212060125/color_000060.jpg'],\n# # 10: ['20200709-subject-01/20200709_145743/839512060362/color_000060.jpg', '20200709-subject-01/20200709_145743/932122061900/color_000030.jpg', '20200813-subject-02/20200813_152842/836212060125/color_000060.jpg', '20200813-subject-02/20200813_152842/841412060263/color_000030.jpg'],\n# # 15: ['20200709-subject-01/20200709_151632/836212060125/color_000060.jpg', '20200709-subject-01/20200709_151632/840412060917/color_000030.jpg', '20200813-subject-02/20200813_154408/836212060125/color_000030.jpg', '20200813-subject-02/20200813_154408/836212060125/color_000060.jpg'],\n\n# # }\n\n# target_img_list_sum = []\n# for key, val in target_img_list.items():\n# target_img_list_sum.extend(val) \n\n#with open('/home/hongsuk.c/Projects/HandOccNet/main/novel_object_test_list.json', 'r') as f:\n# target_img_list_sum = json.load(f)\n#print(\"TARGET LENGTH: \", len(target_img_list_sum)) \n\nclass DEX_YCB(torch.utils.data.Dataset):\n def __init__(self, transform, data_split):\n self.transform = transform\n self.data_split = data_split if data_split == 'train' else 'val'\n self.root_dir = osp.join('..', 'data', 'DEX_YCB', 'data')\n self.annot_path = osp.join(self.root_dir, 'annotations')\n self.root_joint_idx = 0\n\n self.datalist = self.load_data()\n if self.data_split != 'train':\n self.eval_result = [[],[]] #[mpjpe_list, pa-mpjpe_list]\n print(\"TEST DATA LEN: \", len(self.datalist))\n def load_data(self):\n db = COCO(osp.join(self.annot_path, \"DEX_YCB_s0_{}_data.json\".format(self.data_split)))\n \n datalist = []\n for aid in db.anns.keys():\n ann = db.anns[aid]\n image_id = ann['image_id']\n img = db.loadImgs(image_id)[0]\n img_path = osp.join(self.root_dir, img['file_name'])\n img_shape = (img['height'], img['width'])\n if self.data_split == 'train':\n joints_coord_cam = np.array(ann['joints_coord_cam'], dtype=np.float32) # meter\n cam_param = {k:np.array(v, dtype=np.float32) for k,v in ann['cam_param'].items()}\n joints_coord_img = np.array(ann['joints_img'], dtype=np.float32)\n hand_type = ann['hand_type']\n\n bbox = get_bbox(joints_coord_img[:,:2], np.ones_like(joints_coord_img[:,0]), expansion_factor=1.5)\n bbox = process_bbox(bbox, img['width'], img['height'], expansion_factor=1.0)\n\n if bbox is None:\n continue\n\n mano_pose = np.array(ann['mano_param']['pose'], dtype=np.float32)\n mano_shape = np.array(ann['mano_param']['shape'], dtype=np.float32)\n\n data = {\"img_path\": img_path, \"img_shape\": img_shape, \"joints_coord_cam\": joints_coord_cam, \"joints_coord_img\": joints_coord_img,\n \"bbox\": bbox, \"cam_param\": cam_param, \"mano_pose\": mano_pose, \"mano_shape\": mano_shape, \"hand_type\": hand_type}\n else:\n# if '/'.join(img_path.split('/')[-4:]) not in target_img_list_sum:\n# continue\n\n\n\n joints_coord_cam = np.array(ann['joints_coord_cam'], dtype=np.float32)\n root_joint_cam = copy.deepcopy(joints_coord_cam[0])\n joints_coord_img = np.array(ann['joints_img'], dtype=np.float32)\n hand_type = ann['hand_type']\n\n if False and hand_type == 'left':\n\n # mano_pose = np.array(ann['mano_param']['pose'], dtype=np.float32)\n # mano_shape = np.array(ann['mano_param']['shape'], dtype=np.float32)\n\n # vertices, joints, manojoints2cam = mano.left_layer(torch.from_numpy(mano_pose)[None, :], torch.from_numpy(mano_shape)[None, :])\n # vertices = vertices[0].numpy()\n # # save_obj(vertices, mano.left_layer.th_faces.numpy(), 'org_left.obj')\n # joints = joints[0].numpy()\n # joints /= 1000\n # joints = joints - joints[0:1] + root_joint_cam[None, :]\n # focal, princpt = ann['cam_param']['focal'], ann['cam_param']['princpt']\n # proj_joints = cam2pixel(joints, focal, princpt)\n # img = cv2.imread(img_path)\n # vis_img = vis_keypoints(img, proj_joints)\n # cv2.imshow('check cam', vis_img)\n # cv2.waitKey(0)\n # import pdb; pdb.set_trace()\n\n # mano_pose = mano_pose.reshape(-1,3)\n # mano_pose[:,1:] *= -1\n # mano_pose = mano_pose.reshape(-1)\n # vertices, joints, _ = mano.layer(torch.from_numpy(mano_pose)[None, :], torch.from_numpy(mano_shape)[None, :])\n # joints = joints[0].numpy()\n # joints /= 1000\n # joints = joints - joints[0:1] \n # joints[:, 0] *= -1\n # joints = joints + root_joint_cam[None, :]\n \n # focal, princpt = ann['cam_param']['focal'], ann['cam_param']['princpt']\n # proj_joints = cam2pixel(joints, focal, princpt)\n # img = cv2.imread(img_path)\n # vis_img = vis_keypoints(img, proj_joints)\n # cv2.imshow('check flip', vis_img)\n # cv2.waitKey(0)\n # import pdb; pdb.set_trace()\n\n import pdb; pdb.set_trace()\n\n\n\n bbox = get_bbox(joints_coord_img[:,:2], np.ones_like(joints_coord_img[:,0]), expansion_factor=1.5)\n bbox = process_bbox(bbox, img['width'], img['height'], expansion_factor=1.0)\n if bbox is None:\n bbox = np.array([0,0,img['width']-1, img['height']-1], dtype=np.float32)\n\n cam_param = {k:np.array(v, dtype=np.float32) for k,v in ann['cam_param'].items()}\n\n \n data = {\"img_path\": img_path, \"img_shape\": img_shape, \"joints_coord_cam\": joints_coord_cam, \"root_joint_cam\": root_joint_cam,\n \"bbox\": bbox, \"cam_param\": cam_param, \"image_id\": image_id, 'hand_type': hand_type}\n \n datalist.append(data)\n return datalist\n\n def __len__(self):\n return len(self.datalist)\n\n def __getitem__(self, idx):\n data = copy.deepcopy(self.datalist[idx])\n img_path, img_shape, bbox = data['img_path'], data['img_shape'], data['bbox']\n hand_type = data['hand_type']\n do_flip = (hand_type == 'left')\n\n # img\n img = load_img(img_path)\n orig_img = copy.deepcopy(img)[:,:,::-1]\n img, img2bb_trans, bb2img_trans, rot, scale = augmentation(img, bbox, self.data_split, do_flip=do_flip)\n img = self.transform(img.astype(np.float32))/255.\n\n if self.data_split == 'train':\n ## 2D joint coordinate\n joints_img = data['joints_coord_img']\n if do_flip:\n joints_img[:,0] = img_shape[1] - joints_img[:,0] - 1\n joints_img_xy1 = np.concatenate((joints_img[:,:2], np.ones_like(joints_img[:,:1])),1)\n joints_img = np.dot(img2bb_trans, joints_img_xy1.transpose(1,0)).transpose(1,0)[:,:2]\n # normalize to [0,1]\n joints_img[:,0] /= cfg.input_img_shape[1]\n joints_img[:,1] /= cfg.input_img_shape[0]\n\n ## 3D joint camera coordinate\n joints_coord_cam = data['joints_coord_cam']\n root_joint_cam = copy.deepcopy(joints_coord_cam[self.root_joint_idx])\n joints_coord_cam -= joints_coord_cam[self.root_joint_idx,None,:] # root-relative\n if do_flip:\n joints_coord_cam[:,0] *= -1\n\n # 3D data rotation augmentation\n rot_aug_mat = np.array([[np.cos(np.deg2rad(-rot)), -np.sin(np.deg2rad(-rot)), 0], \n [np.sin(np.deg2rad(-rot)), np.cos(np.deg2rad(-rot)), 0],\n [0, 0, 1]], dtype=np.float32)\n joints_coord_cam = np.dot(rot_aug_mat, joints_coord_cam.transpose(1,0)).transpose(1,0)\n \n ## mano parameter\n mano_pose, mano_shape = data['mano_pose'], data['mano_shape']\n \n # 3D data rotation augmentation\n mano_pose = mano_pose.reshape(-1,3)\n if do_flip:\n mano_pose[:,1:] *= -1\n root_pose = mano_pose[self.root_joint_idx,:]\n root_pose, _ = cv2.Rodrigues(root_pose)\n root_pose, _ = cv2.Rodrigues(np.dot(rot_aug_mat,root_pose))\n mano_pose[self.root_joint_idx] = root_pose.reshape(3)\n mano_pose = mano_pose.reshape(-1)\n\n inputs = {'img': img}\n targets = {'joints_img': joints_img, 'joints_coord_cam': joints_coord_cam, 'mano_pose': mano_pose, 'mano_shape': mano_shape}\n meta_info = {'root_joint_cam': root_joint_cam}\n \n else:\n root_joint_cam = data['root_joint_cam']\n inputs = {'img': img}\n targets = {}\n meta_info = {'root_joint_cam': root_joint_cam, 'img_path': img_path}\n\n return inputs, targets, meta_info \n\n def evaluate(self, outs, cur_sample_idx):\n annots = self.datalist\n sample_num = len(outs)\n for n in range(sample_num):\n annot = annots[cur_sample_idx + n]\n \n out = outs[n]\n \n joints_out = out['joints_coord_cam']\n \n # root centered\n joints_out -= joints_out[self.root_joint_idx]\n\n # flip back to left hand\n if annot['hand_type'] == 'left':\n joints_out[:,0] *= -1\n \n # root align\n gt_root_joint_cam = annot['root_joint_cam']\n joints_out += gt_root_joint_cam\n \n # GT and rigid align\n joints_gt = annot['joints_coord_cam']\n joints_out_aligned = rigid_align(joints_out, joints_gt)\n\n # m to mm\n joints_out *= 1000\n joints_out_aligned *= 1000\n joints_gt *= 1000\n\n self.eval_result[0].append(np.sqrt(np.sum((joints_out - joints_gt)**2,1)).mean())\n self.eval_result[1].append(np.sqrt(np.sum((joints_out_aligned - joints_gt)**2,1)).mean())\n \n def print_eval_result(self, test_epoch):\n print('MPJPE : %.2f mm' % np.mean(self.eval_result[0]))\n print('PA MPJPE : %.2f mm' % np.mean(self.eval_result[1]))\n \n \"\"\"\n def evaluate(self, outs, cur_sample_idx):\n annots = self.datalist\n sample_num = len(outs)\n for n in range(sample_num):\n annot = annots[cur_sample_idx + n]\n \n out = outs[n]\n \n verts_out = out['mesh_coord_cam']\n joints_out = out['joints_coord_cam']\n \n # root centered\n verts_out -= joints_out[self.root_joint_idx]\n joints_out -= joints_out[self.root_joint_idx]\n\n # flip back to left hand\n if annot['hand_type'] == 'left':\n verts_out[:,0] *= -1\n joints_out[:,0] *= -1\n \n # root align\n gt_root_joint_cam = annot['root_joint_cam']\n verts_out += gt_root_joint_cam\n joints_out += gt_root_joint_cam\n\n # m to mm\n verts_out *= 1000\n joints_out *= 1000\n\n self.eval_result[0].append(joints_out)\n self.eval_result[1].append(verts_out)\n\n def print_eval_result(self, test_epoch):\n output_file_path = osp.join(cfg.result_dir, \"DEX_RESULTS_EPOCH{}.txt\".format(test_epoch))\n\n with open(output_file_path, 'w') as output_file:\n for i, pred_joints in enumerate(self.eval_result[0]):\n image_id = self.datalist[i]['image_id']\n output_file.write(str(image_id) + ',' + ','.join(pred_joints.ravel().astype(str).tolist()) + '\\n')\n \"\"\"\n" }, { "path": "data/HO3D/HO3D.py", "content": "import os\nimport os.path as osp\nimport numpy as np\nimport torch\nimport cv2\nimport random\nimport json\nimport math\nimport copy\nfrom pycocotools.coco import COCO\nfrom config import cfg\nfrom utils.preprocessing import load_img, get_bbox, process_bbox, generate_patch_image, augmentation\nfrom utils.transforms import world2cam, cam2pixel, pixel2cam, rigid_align, transform_joint_to_other_db\nfrom utils.vis import vis_keypoints, vis_mesh, save_obj, vis_keypoints_with_skeleton\nfrom utils.mano import MANO\nmano = MANO()\n\nclass HO3D(torch.utils.data.Dataset):\n def __init__(self, transform, data_split):\n self.transform = transform\n self.data_split = data_split if data_split == 'train' else 'evaluation'\n self.root_dir = osp.join('..', 'data', 'HO3D', 'data')\n self.annot_path = osp.join(self.root_dir, 'annotations')\n self.root_joint_idx = 0\n\n self.datalist = self.load_data()\n if self.data_split != 'train':\n self.eval_result = [[],[]] #[pred_joints_list, pred_verts_list]\n self.joints_name = ('Wrist', 'Index_1', 'Index_2', 'Index_3', 'Middle_1', 'Middle_2', 'Middle_3', 'Pinky_1', 'Pinky_2', 'Pinky_3', 'Ring_1', 'Ring_2', 'Ring_3', 'Thumb_1', 'Thumb_2', 'Thumb_3', 'Thumb_4', 'Index_4', 'Middle_4', 'Ring_4', 'Pinly_4')\n \n def load_data(self):\n db = COCO(osp.join(self.annot_path, \"HO3D_{}_data.json\".format(self.data_split)))\n # db = COCO(osp.join(self.annot_path, 'HO3Dv3_partial_test_multiseq_coco.json'))\n\n datalist = []\n for aid in db.anns.keys():\n ann = db.anns[aid]\n image_id = ann['image_id']\n img = db.loadImgs(image_id)[0]\n img_path = osp.join(self.root_dir, self.data_split, img['file_name'])\n # TEMP\n # img_path = osp.join(self.root_dir, 'train', img['sequence_name'], 'rgb', img['file_name'])\n\n img_shape = (img['height'], img['width'])\n if self.data_split == 'train':\n joints_coord_cam = np.array(ann['joints_coord_cam'], dtype=np.float32) # meter\n cam_param = {k:np.array(v, dtype=np.float32) for k,v in ann['cam_param'].items()}\n joints_coord_img = cam2pixel(joints_coord_cam, cam_param['focal'], cam_param['princpt'])\n bbox = get_bbox(joints_coord_img[:,:2], np.ones_like(joints_coord_img[:,0]), expansion_factor=1.5)\n bbox = process_bbox(bbox, img['width'], img['height'], expansion_factor=1.0)\n if bbox is None:\n continue\n\n mano_pose = np.array(ann['mano_param']['pose'], dtype=np.float32)\n mano_shape = np.array(ann['mano_param']['shape'], dtype=np.float32)\n\n data = {\"img_path\": img_path, \"img_shape\": img_shape, \"joints_coord_cam\": joints_coord_cam, \"joints_coord_img\": joints_coord_img,\n \"bbox\": bbox, \"cam_param\": cam_param, \"mano_pose\": mano_pose, \"mano_shape\": mano_shape}\n else:\n root_joint_cam = np.array(ann['root_joint_cam'], dtype=np.float32)\n cam_param = {k:np.array(v, dtype=np.float32) for k,v in ann['cam_param'].items()}\n # TEMP\n # root_joint_cam = np.zeros(0)\n # cam_param = np.zeros(0)\n bbox = np.array(ann['bbox'], dtype=np.float32)\n bbox = process_bbox(bbox, img['width'], img['height'], expansion_factor=1.5)\n \n data = {\"img_path\": img_path, \"img_shape\": img_shape, \"root_joint_cam\": root_joint_cam,\n \"bbox\": bbox, \"cam_param\": cam_param}\n\n datalist.append(data)\n\n return datalist\n\n def __len__(self):\n return len(self.datalist)\n\n def __getitem__(self, idx):\n data = copy.deepcopy(self.datalist[idx])\n img_path, img_shape, bbox = data['img_path'], data['img_shape'], data['bbox']\n\n # img\n img = load_img(img_path)\n img, img2bb_trans, bb2img_trans, rot, scale = augmentation(img, bbox, self.data_split, do_flip=False)\n img = self.transform(img.astype(np.float32))/255.\n\n if self.data_split == 'train':\n ## 2D joint coordinate\n joints_img = data['joints_coord_img']\n joints_img_xy1 = np.concatenate((joints_img[:,:2], np.ones_like(joints_img[:,:1])),1)\n joints_img = np.dot(img2bb_trans, joints_img_xy1.transpose(1,0)).transpose(1,0)[:,:2]\n # normalize to [0,1]\n joints_img[:,0] /= cfg.input_img_shape[1]\n joints_img[:,1] /= cfg.input_img_shape[0]\n\n ## 3D joint camera coordinate\n joints_coord_cam = data['joints_coord_cam']\n root_joint_cam = copy.deepcopy(joints_coord_cam[self.root_joint_idx])\n joints_coord_cam -= joints_coord_cam[self.root_joint_idx,None,:] # root-relative\n # 3D data rotation augmentation\n rot_aug_mat = np.array([[np.cos(np.deg2rad(-rot)), -np.sin(np.deg2rad(-rot)), 0], \n [np.sin(np.deg2rad(-rot)), np.cos(np.deg2rad(-rot)), 0],\n [0, 0, 1]], dtype=np.float32)\n joints_coord_cam = np.dot(rot_aug_mat, joints_coord_cam.transpose(1,0)).transpose(1,0)\n \n ## mano parameter\n mano_pose, mano_shape = data['mano_pose'], data['mano_shape']\n # 3D data rotation augmentation\n mano_pose = mano_pose.reshape(-1,3)\n root_pose = mano_pose[self.root_joint_idx,:]\n root_pose, _ = cv2.Rodrigues(root_pose)\n root_pose, _ = cv2.Rodrigues(np.dot(rot_aug_mat,root_pose))\n mano_pose[self.root_joint_idx] = root_pose.reshape(3)\n mano_pose = mano_pose.reshape(-1)\n\n inputs = {'img': img}\n targets = {'joints_img': joints_img, 'joints_coord_cam': joints_coord_cam, 'mano_pose': mano_pose, 'mano_shape': mano_shape}\n meta_info = {'root_joint_cam': root_joint_cam}\n\n else:\n root_joint_cam = data['root_joint_cam']\n inputs = {'img': img}\n targets = {}\n meta_info = {'root_joint_cam': root_joint_cam, 'img_path': img_path}\n\n return inputs, targets, meta_info\n \n\n def evaluate(self, outs, cur_sample_idx):\n annots = self.datalist\n sample_num = len(outs)\n for n in range(sample_num):\n annot = annots[cur_sample_idx + n]\n \n out = outs[n]\n \n verts_out = out['mesh_coord_cam']\n joints_out = out['joints_coord_cam']\n \n # root align\n gt_root_joint_cam = annot['root_joint_cam']\n verts_out = verts_out - joints_out[self.root_joint_idx] + gt_root_joint_cam\n joints_out = joints_out - joints_out[self.root_joint_idx] + gt_root_joint_cam\n \n # convert to openGL coordinate system.\n verts_out *= np.array([1, -1, -1])\n joints_out *= np.array([1, -1, -1])\n\n # convert joint ordering from MANO to HO3D.\n joints_out = transform_joint_to_other_db(joints_out, mano.joints_name, self.joints_name)\n\n self.eval_result[0].append(joints_out.tolist())\n self.eval_result[1].append(verts_out.tolist())\n\n def print_eval_result(self, test_epoch):\n output_json_file = osp.join(cfg.result_dir, 'pred{}.json'.format(test_epoch)) \n output_zip_file = osp.join(cfg.result_dir, 'pred{}.zip'.format(test_epoch))\n \n with open(output_json_file, 'w') as f:\n json.dump(self.eval_result, f)\n print('Dumped %d joints and %d verts predictions to %s' % (len(self.eval_result[0]), len(self.eval_result[1]), output_json_file))\n\n cmd = 'zip -j ' + output_zip_file + ' ' + output_json_file\n print(cmd)\n os.system(cmd)\n\n " }, { "path": "demo/demo.py", "content": "import sys\nimport os\nimport os.path as osp\nimport argparse\nimport numpy as np\nimport cv2\nimport torch\nimport torchvision.transforms as transforms\nfrom torch.nn.parallel.data_parallel import DataParallel\nimport torch.backends.cudnn as cudnn\n\nsys.path.insert(0, osp.join('..', 'main'))\nsys.path.insert(0, osp.join('..', 'common'))\nfrom config import cfg\nfrom model import get_model\nfrom utils.preprocessing import load_img, process_bbox, generate_patch_image\nfrom utils.vis import save_obj\nfrom utils.mano import MANO\nmano = MANO()\n\ndef parse_args():\n parser = argparse.ArgumentParser()\n parser.add_argument('--gpu', type=str, dest='gpu_ids')\n args = parser.parse_args()\n\n # test gpus\n if not args.gpu_ids:\n assert 0, print(\"Please set proper gpu ids\")\n\n if '-' in args.gpu_ids:\n gpus = args.gpu_ids.split('-')\n gpus[0] = int(gpus[0])\n gpus[1] = int(gpus[1]) + 1\n args.gpu_ids = ','.join(map(lambda x: str(x), list(range(*gpus))))\n\n return args\n\n# argument parsing\nargs = parse_args()\ncfg.set_args(args.gpu_ids)\ncudnn.benchmark = True\n\n# snapshot load\nmodel_path = './snapshot_demo.pth.tar'\nassert osp.exists(model_path), 'Cannot find model at ' + model_path\nprint('Load checkpoint from {}'.format(model_path))\nmodel = get_model('test')\n\nmodel = DataParallel(model).cuda()\nckpt = torch.load(model_path)\nmodel.load_state_dict(ckpt['network'], strict=False)\nmodel.eval()\n\n# prepare input image\ntransform = transforms.ToTensor()\nimg_path = 'input.png'\noriginal_img = load_img(img_path)\noriginal_img_height, original_img_width = original_img.shape[:2]\n\n# prepare bbox\nbbox = [340.8, 232.0, 20.7, 20.7] # xmin, ymin, width, height \n\nbbox = process_bbox(bbox, original_img_width, original_img_height)\nimg, img2bb_trans, bb2img_trans = generate_patch_image(original_img, bbox, 1.0, 0.0, False, cfg.input_img_shape) \nimg = transform(img.astype(np.float32))/255\nimg = img.cuda()[None,:,:,:]\n\n# forward\ninputs = {'img': img}\ntargets = {}\nmeta_info = {}\nwith torch.no_grad():\n out = model(inputs, targets, meta_info, 'test')\nimg = (img[0].cpu().numpy().transpose(1,2,0)*255).astype(np.uint8) # cfg.input_img_shape[1], cfg.input_img_shape[0], 3\nverts_out = out['mesh_coord_cam'][0].cpu().numpy()\n\n# bbox for input hand image\nbbox_vis = np.array(bbox, int)\nbbox_vis[2:] += bbox_vis[:2]\ncvimg = cv2.rectangle(original_img.copy(), bbox_vis[:2], bbox_vis[2:], (255,0,0), 3)\ncv2.imwrite('hand_bbox.png', cvimg[:,:,::-1])\n\n## input hand image\ncv2.imwrite('hand_image.png', img[:,:,::-1])\n\n# save mesh (obj)\nsave_obj(verts_out*np.array([1,-1,-1]), mano.face, 'output.obj')\n" }, { "path": "demo/demo_fitting.py", "content": "import sys\nimport glob\nimport os\nimport os.path as osp\nimport argparse\nimport json\nimport numpy as np\nimport cv2\nimport torch\nfrom PIL import Image\nimport torchvision.transforms as transforms\nfrom torch.nn.parallel.data_parallel import DataParallel\nimport torch.backends.cudnn as cudnn\nfrom tqdm import tqdm\n\nsys.path.insert(0, osp.join('..', 'main'))\nsys.path.insert(0, osp.join('..', 'common'))\nfrom config import cfg\nfrom model import get_model\nfrom utils.preprocessing import load_img, process_bbox, generate_patch_image\nfrom utils.vis import save_obj, vis_keypoints_with_skeleton\nfrom utils.mano import MANO\nfrom utils.camera import PerspectiveCamera\nmano = MANO()\n\ndef parse_args():\n parser = argparse.ArgumentParser()\n parser.add_argument('--gpu', type=str, dest='gpu_ids')\n parser.add_argument('--depth', type=float, default='0.5')\n\n args = parser.parse_args()\n\n # test gpus\n if not args.gpu_ids:\n assert 0, print(\"Please set proper gpu ids\")\n\n if '-' in args.gpu_ids:\n gpus = args.gpu_ids.split('-')\n gpus[0] = int(gpus[0])\n gpus[1] = int(gpus[1]) + 1\n args.gpu_ids = ','.join(map(lambda x: str(x), list(range(*gpus))))\n\n return args\n\ndef load_camera(cam_path, cam_idx='0'):\n with open(cam_path, 'r') as f:\n cam_data = json.load(f)\n\n camera = PerspectiveCamera()\n\n camera.focal_length_x = torch.full([1], cam_data[cam_idx]['fx'])\n camera.focal_length_y = torch.full([1], cam_data[cam_idx]['fy'])\n camera.center = torch.tensor(\n [cam_data[cam_idx]['cx'], cam_data[cam_idx]['cy']]).unsqueeze(0)\n # only intrinsics\n # rotation, _ = cv2.Rodrigues(\n # np.array(cam_data[cam_idx]['rvec'], dtype=np.float32))\n # camera.rotation.data = torch.from_numpy(rotation).unsqueeze(0)\n # camera.translation.data = torch.tensor(\n # cam_data[cam_idx]['tvec']).unsqueeze(0) / 1000.\n camera.rotation.requires_grad = False\n camera.translation.requires_grad = False\n camera.name = str(cam_idx)\n\n return camera\n\nif __name__ == '__main__':\n # argument parsing\n args = parse_args()\n cfg.set_args(args.gpu_ids)\n cudnn.benchmark = True\n transform = transforms.ToTensor()\n \n # hard coding\n save_dir = './'\n init_depth = args.depth\n img_path = 'fitting_input.jpg'\n bbox = [300, 330, 90, 50]#[340.8, 232.0, 20.7, 20.7] # xmin, ymin, width, height \n\n # model snapshot load\n model_path = './snapshot_demo.pth.tar'\n assert osp.exists(model_path), 'Cannot find model at ' + model_path\n print('Load checkpoint from {}'.format(model_path))\n model = get_model('test')\n\n model = DataParallel(model).cuda()\n ckpt = torch.load(model_path)\n model.load_state_dict(ckpt['network'], strict=False)\n model.eval()\n\n # prepare input image\n transform = transforms.ToTensor()\n original_img = load_img(img_path)\n original_img_height, original_img_width = original_img.shape[:2]\n\n # prepare bbox\n bbox = process_bbox(bbox, original_img_width, original_img_height)\n img, img2bb_trans, bb2img_trans = generate_patch_image(original_img, bbox, 1.0, 0.0, False, cfg.input_img_shape) \n img = transform(img.astype(np.float32))/255\n img = img.cuda()[None,:,:,:]\n\n # get camera for projection\n camera = PerspectiveCamera()\n camera.rotation.requires_grad = False\n camera.translation.requires_grad = False\n camera.center[0, 0] = original_img.shape[1] / 2 \n camera.center[0, 1] = original_img.shape[0] / 2 \n camera.cuda()\n\n # forward pass to the model\n inputs = {'img': img} # cfg.input_img_shape[1], cfg.input_img_shape[0], 3\n targets = {}\n meta_info = {}\n with torch.no_grad():\n out = model(inputs, targets, meta_info, 'test')\n img = (img[0].cpu().numpy().transpose(1, 2, 0)*255).astype(np.uint8) # \n verts_out = out['mesh_coord_cam'][0].cpu().numpy()\n \n # get hand mesh's scale and translation by fitting joint cam to joint img\n joint_img, joint_cam = out['joints_coord_img'], out['joints_coord_cam']\n\n # denormalize joint_img from 0 ~ 1 to actual 0 ~ original height and width\n H, W = img.shape[:2]\n joint_img[:, :, 0] *= W\n joint_img[:, :, 1] *= H\n torch_bb2img_trans = torch.tensor(bb2img_trans).to(joint_img)\n homo_joint_img = torch.cat([joint_img, torch.ones_like(joint_img[:, :, :1])], dim=2)\n org_res_joint_img = homo_joint_img @ torch_bb2img_trans.transpose(0, 1)\n\n # depth initialization\n depth_map = None #np.asarray(Image.open(depth_path))\n hand_scale, hand_translation = model.module.get_mesh_scale_trans(\n org_res_joint_img, joint_cam, init_scale=1., init_depth=init_depth, camera=camera, depth_map=depth_map)\n\n np_joint_img = org_res_joint_img[0].cpu().numpy()\n np_joint_img = np.concatenate([np_joint_img, np.ones_like(np_joint_img[:, :1])], axis=1)\n vis_img = original_img.astype(np.uint8)[:, :, ::-1]\n pred_joint_img_overlay = vis_keypoints_with_skeleton(vis_img, np_joint_img.T, mano.skeleton)\n # cv2.imshow('2d prediction', pred_joint_img_overlay)\n save_path = osp.join(\n save_dir, f'{osp.basename(img_path)[:-4]}_2d_prediction.png')\n\n cv2.imwrite(save_path, pred_joint_img_overlay)\n projected_joints = camera(\n hand_scale * joint_cam + hand_translation)\n np_joint_img = projected_joints[0].detach().cpu().numpy()\n np_joint_img = np.concatenate([np_joint_img, np.ones_like(np_joint_img[:, :1])], axis=1)\n\n vis_img = original_img.astype(np.uint8)[:, :, ::-1]\n pred_joint_img_overlay = vis_keypoints_with_skeleton(vis_img, np_joint_img.T, mano.skeleton)\n # cv2.imshow('projection', pred_joint_img_overlay)\n # cv2.waitKey(0)\n save_path = osp.join(save_dir, f'{osp.basename(img_path)[:-4]}_projection.png')\n cv2.imwrite(save_path, pred_joint_img_overlay)\n \n # data to save\n data_to_save = {\n 'hand_scale': hand_scale.detach().cpu().numpy().tolist(), # 1\n 'hand_translation': hand_translation.detach().cpu().numpy().tolist(), # 3\n 'mano_pose': out['mano_pose'][0].detach().cpu().numpy().tolist(), # 48\n 'mano_shape': out['mano_shape'][0].detach().cpu().numpy().tolist(), # 10\n }\n save_path = osp.join(\n save_dir, f'{osp.basename(img_path)[:-4]}_3dmesh.json')\n with open(save_path, 'w') as f:\n json.dump(data_to_save, f)\n\n # # bbox for input hand image\n # bbox_vis = np.array(bbox, int)\n # bbox_vis[2:] += bbox_vis[:2]\n # cvimg = cv2.rectangle(original_img.copy(),\n # bbox_vis[:2], bbox_vis[2:], (255, 0, 0), 3)\n # cv2.imwrite(f'{osp.basename(img_path)[:-4]}_hand_bbox.png', cvimg[:, :, ::-1])\n # ## input hand image\n # cv2.imwrite(f'{osp.basename(img_path)[:-4]}_hand_image.png', img[:, :, ::-1])\n\n # save mesh (obj)\n save_path = osp.join(\n save_dir, f'{osp.basename(img_path)[:-4]}_3dmesh.obj')\n save_obj(verts_out*np.array([1, -1, -1]),\n mano.face, save_path)" }, { "path": "demo/output.obj", "content": "v 0.09704167395830154 0.0123032471165061 0.039412785321474075\nv 0.09976062923669815 0.010310064069926739 0.029586462303996086\nv 0.10385143011808395 0.014365013688802719 0.03625381737947464\nv 0.09955313056707382 0.017543060705065727 0.04449660703539848\nv 0.11498267948627472 0.016574472188949585 0.020893722772598267\nv 0.11550682783126831 0.02474113740026951 0.014739320613443851\nv 0.11841963976621628 0.028651203960180283 0.029420627281069756\nv 0.1187518835067749 0.020955543965101242 0.03313694894313812\nv 0.10000406950712204 0.02170594595372677 0.04935751110315323\nv 0.10712949186563492 0.017804410308599472 0.043026067316532135\nv 0.09772946685552597 0.023356322199106216 0.050800975412130356\nv 0.08702512085437775 0.027183635160326958 0.05566640943288803\nv 0.09220578521490097 0.024990126490592957 0.05519742891192436\nv 0.0820174366235733 0.02769414521753788 0.06143525615334511\nv 0.08001333475112915 0.02764720283448696 0.04892248660326004\nv 0.07448913156986237 0.023760424926877022 0.060431305319070816\nv 0.06879005581140518 0.006574998609721661 0.0407930389046669\nv 0.06904236972332001 -0.005039920099079609 0.038309454917907715\nv 0.06741352379322052 -0.006001491565257311 0.05440099164843559\nv 0.06813132017850876 0.00622991006821394 0.0552951879799366\nv 0.07286028563976288 -0.021629616618156433 0.046827297657728195\nv 0.06995420902967453 -0.012919521890580654 0.05173160508275032\nv 0.07021812349557877 -0.012072782032191753 0.035572897642850876\nv 0.07322045415639877 -0.019798777997493744 0.031946342438459396\nv 0.09612422436475754 0.000507109856698662 0.0327720120549202\nv 0.0982327088713646 -0.0007277392433024943 0.023071376606822014\nv 0.09740523248910904 0.003534032730385661 0.024903787299990654\nv 0.09577703475952148 0.004897149745374918 0.034299276769161224\nv 0.099307581782341 0.030571212992072105 0.04903818666934967\nv 0.09175309538841248 0.03211066126823425 0.04123810678720474\nv 0.09266895055770874 0.027560750022530556 0.04706060513854027\nv 0.10247763991355896 0.027115963399410248 0.051462795585393906\nv 0.10852036625146866 -0.00967042800039053 0.0010749474167823792\nv 0.1044674888253212 -0.009041075594723225 -0.010376684367656708\nv 0.10359753668308258 -0.0038258449640125036 -0.01053894404321909\nv 0.10706440359354019 -0.005605116952210665 -0.00010789186490001157\nv 0.10655821114778519 0.015718799084424973 -0.0038651900831609964\nv 0.10114449262619019 0.012555424124002457 -0.011676214635372162\nv 0.09330809116363525 0.019990479573607445 -0.01726539433002472\nv 0.09610037505626678 0.02422197535634041 -0.008547515608370304\nv 0.11166828125715256 0.020764684304594994 0.005238473881036043\nv 0.10379082709550858 0.028549164533615112 0.00103653222322464\nv 0.10910049080848694 0.03203720599412918 0.011868038214743137\nv 0.11276104301214218 0.03460574895143509 0.026923196390271187\nv 0.1037481427192688 -0.020462632179260254 -0.010825783014297485\nv 0.10694169253110886 -0.022308088839054108 -0.002532459795475006\nv 0.09702412784099579 0.021299252286553383 0.09731592983007431\nv 0.09826341271400452 0.016166748479008675 0.09627735614776611\nv 0.09706933796405792 0.01635078713297844 0.09417114406824112\nv 0.09600481390953064 0.02130129560828209 0.09562767297029495\nv 0.09107829630374908 -0.03202741593122482 -0.005628600716590881\nv 0.08212634921073914 -0.03188098967075348 -0.00568735646083951\nv 0.08398226648569107 -0.03263990953564644 -0.012086243368685246\nv 0.09229756146669388 -0.031173406168818474 -0.012397877871990204\nv 0.08960714936256409 -0.032796479761600494 0.003477819263935089\nv 0.08015740662813187 -0.030353110283613205 0.0033182355109602213\nv 0.09060245007276535 0.02204286865890026 0.10700210183858871\nv 0.09335938096046448 0.024315452203154564 0.1039421334862709\nv 0.08984217792749405 0.02442437969148159 0.10220398753881454\nv 0.0864965170621872 0.021755777299404144 0.10453414171934128\nv 0.0952310562133789 0.020760783925652504 0.05235325172543526\nv 0.09612379223108292 0.013916807249188423 0.04842321202158928\nv 0.09275881201028824 0.019942499697208405 0.06846490502357483\nv 0.09354271739721298 0.016712404787540436 0.06071989983320236\nv 0.09189996123313904 0.02278514765202999 0.05997182056307793\nv 0.09013500064611435 0.024846429005265236 0.06666994094848633\nv 0.0940636396408081 -0.0031293872743844986 0.05349469929933548\nv 0.09469340741634369 -0.0026052643079310656 0.04270586743950844\nv 0.09410218149423599 0.002816879888996482 0.04490778222680092\nv 0.09403892606496811 0.0038006349932402372 0.056964416056871414\nv 0.09825103729963303 -0.014537331648170948 0.04019445925951004\nv 0.09684037417173386 -0.0140069629997015 0.050926629453897476\nv 0.09873218834400177 -0.02082197554409504 0.04806012660264969\nv 0.10043563693761826 -0.019858956336975098 0.04097581282258034\nv 0.09248537570238113 -0.010444425977766514 0.06883682310581207\nv 0.09154201298952103 -0.009355786256492138 0.07404470443725586\nv 0.09260138124227524 -0.012381582520902157 0.07426491379737854\nv 0.09295526146888733 -0.013393072411417961 0.06776127964258194\nv 0.0847911611199379 -0.0330488346517086 -0.018598178401589394\nv 0.09283468127250671 -0.029433878138661385 -0.018831679597496986\nv 0.08127716183662415 -0.037603959441185 0.040029171854257584\nv 0.08886391669511795 -0.04075009375810623 0.03510737419128418\nv 0.09025923162698746 -0.0435173474252224 0.04641684144735336\nv 0.08072876930236816 -0.039703723043203354 0.04932333528995514\nv 0.0749884694814682 -0.026462497189641 -0.004455134738236666\nv 0.07592447847127914 -0.029161827638745308 -0.01113533228635788\nv 0.08868138492107391 0.01678049936890602 0.10864982008934021\nv 0.08478643000125885 0.016503214836120605 0.10636698454618454\nv 0.09663982689380646 0.035291120409965515 0.010118492878973484\nv 0.10477548092603683 0.0380316898226738 0.026483744382858276\nv 0.08999285846948624 0.030108794569969177 -0.002212435007095337\nv 0.0851031243801117 0.024710390716791153 -0.010170720517635345\nv 0.08258376270532608 0.021762587130069733 -0.018127508461475372\nv 0.09366891533136368 0.01160071324557066 0.06399420648813248\nv 0.09508023411035538 0.0076362332329154015 0.04296451807022095\nv 0.09705101698637009 -0.00388349243439734 0.03176907077431679\nv 0.10574627667665482 -0.029874518513679504 0.030231954529881477\nv 0.1072174534201622 -0.02885815128684044 0.020927833393216133\nv 0.10617993772029877 -0.02044968120753765 0.02524772845208645\nv 0.10448476672172546 -0.023264355957508087 0.03252417966723442\nv 0.09787759184837341 -0.04072023183107376 0.035247232764959335\nv 0.10269629210233688 -0.03633294627070427 0.037638574838638306\nv 0.10200449079275131 -0.03697844594717026 0.04497504234313965\nv 0.09784367680549622 -0.041939008980989456 0.0442439466714859\nv 0.0993509367108345 0.03691822290420532 0.04624485969543457\nv 0.09221241623163223 0.03617630898952484 0.03594726324081421\nv 0.09991807490587234 -0.028782188892364502 -0.005242601037025452\nv 0.09977457672357559 -0.026117412373423576 -0.011667192913591862\nv 0.0983252227306366 -0.025250444188714027 -0.019326167181134224\nv 0.10373659431934357 -0.004046760965138674 0.01158856600522995\nv 0.10632442682981491 -0.002683989703655243 -0.0010402277112007141\nv 0.10617887228727341 0.0013205824652686715 -0.0002762451767921448\nv 0.10243091732263565 -0.0014278828166425228 0.012040451169013977\nv 0.10552237927913666 0.008538869209587574 0.01999625377357006\nv 0.11025764048099518 0.011296899057924747 0.009791173040866852\nv 0.11036954075098038 0.012080628424882889 0.029019424691796303\nv 0.10254248231649399 0.00337228924036026 -0.01225954294204712\nv 0.10285349190235138 -0.0029982945416122675 -0.019830450415611267\nv 0.10075358301401138 0.0047887228429317474 -0.020481610670685768\nv 0.10294044762849808 -0.011905921623110771 -0.019881347194314003\nv 0.10184606164693832 -0.019946394488215446 -0.01970786787569523\nv 0.07410712540149689 -0.03153001144528389 -0.018500978127121925\nv 0.09734782576560974 0.011572230607271194 -0.020160282030701637\nv 0.1192983090877533 0.024943308904767036 0.041293080896139145\nv 0.11934367567300797 0.029254555702209473 0.04782867431640625\nv 0.11521545797586441 0.024670789018273354 0.05094749107956886\nv 0.11494014412164688 0.020241715013980865 0.045187536627054214\nv 0.07898958027362823 0.027264561504125595 0.06948734074831009\nv 0.085870660841465 0.0275066290050745 0.06830953061580658\nv 0.08816779404878616 0.027377676218748093 0.05996124446392059\nv 0.10879211127758026 -0.01456399355083704 0.010258094407618046\nv 0.10759785026311874 -0.02512398548424244 0.006774865556508303\nv 0.09227554500102997 0.014700433239340782 0.07183413952589035\nv 0.08323711156845093 0.02198031358420849 0.10114031285047531\nv 0.08718504011631012 0.025041446089744568 0.09874201565980911\nv 0.08179915696382523 0.026613231748342514 0.08361773192882538\nv 0.07698608934879303 0.02342488244175911 0.08595925569534302\nv 0.08710064738988876 0.010135228745639324 0.0794704407453537\nv 0.09131284803152084 0.013762109912931919 0.07629778236150742\nv 0.09036200493574142 0.012931560166180134 0.07877384126186371\nv 0.08829855918884277 0.008487006649374962 0.08210074156522751\nv 0.09455457329750061 -0.018902059644460678 0.0692327693104744\nv 0.09495323151350021 -0.019648753106594086 0.06373544782400131\nv 0.0700426921248436 0.005297053139656782 0.06998594105243683\nv 0.06758403778076172 0.016597703099250793 0.06780336052179337\nv 0.06917118281126022 0.015945417806506157 0.05677381530404091\nv 0.09529761224985123 -0.009283529594540596 0.05125439539551735\nv 0.09456868469715118 -0.010417206212878227 0.06084778532385826\nv 0.09512004256248474 -0.013825497590005398 0.059904929250478745\nv 0.09272909164428711 0.008574625477194786 0.06723345071077347\nv 0.09071554243564606 0.009755033999681473 0.07368878275156021\nv 0.09069853276014328 0.0059261927381157875 0.07144072651863098\nv 0.09285608679056168 0.004113199654966593 0.06521569937467575\nv 0.08840681612491608 -0.03841841593384743 0.023314980790019035\nv 0.09859631955623627 -0.038903333246707916 0.023828836157917976\nv 0.09550290554761887 0.024148257449269295 0.10024920850992203\nv 0.09340698271989822 0.02425946667790413 0.09864538908004761\nv 0.09598366916179657 -0.008685051463544369 0.04080434516072273\nv 0.07096162438392639 0.01585032045841217 0.043433088809251785\nv 0.09816217422485352 -0.008554179221391678 0.03062260150909424\nv 0.0815075933933258 -0.029755791649222374 0.06396094709634781\nv 0.0892564132809639 -0.03116283193230629 0.0650174468755722\nv 0.09104897081851959 -0.029866019263863564 0.06883867084980011\nv 0.08587896823883057 -0.029217805713415146 0.07037565857172012\nv 0.09131127595901489 0.007197975646704435 0.0943320244550705\nv 0.09518592059612274 0.010463752783834934 0.09308510273694992\nv 0.09583403915166855 0.010912451893091202 0.09478212147951126\nv 0.0926322415471077 0.007368505001068115 0.09736977517604828\nv 0.09202869981527328 0.01878918707370758 0.07468339800834656\nv 0.09020364284515381 0.024195468053221703 0.0735035389661789\nv 0.09103194624185562 0.022528868168592453 0.07921769469976425\nv 0.091405488550663 0.017365960404276848 0.07912317663431168\nv 0.08558965474367142 0.02679891139268875 0.07541236281394958\nv 0.08682217448949814 0.02605549991130829 0.08087122440338135\nv 0.08633208274841309 0.0067527759820222855 0.09735039621591568\nv 0.08223645389080048 0.010843859985470772 0.10099346190690994\nv 0.07649202644824982 0.010881617665290833 0.08659590035676956\nv 0.08258166164159775 0.007566985208541155 0.08595532923936844\nv 0.0719359740614891 -0.018325546756386757 0.018172213807702065\nv 0.07609406858682632 -0.027056368067860603 0.016072019934654236\nv 0.07732595503330231 -0.028749490156769753 0.029808150604367256\nv 0.07750879973173141 -0.03113008663058281 0.04351899027824402\nv 0.07843733578920364 -0.03255752474069595 0.05685797333717346\nv 0.07198290526866913 -0.024020137265324593 0.05965801700949669\nv 0.0805065929889679 -0.034317996352910995 0.026994476094841957\nv 0.08016543835401535 0.006380251608788967 0.07753480970859528\nv 0.07334155589342117 0.010266893543303013 0.07657967507839203\nv 0.07463667541742325 0.005383501295000315 0.0757342129945755\nv 0.10097857564687729 -0.014524447731673717 0.029919208958745003\nv 0.09139500558376312 0.024932367727160454 0.09599956125020981\nv 0.0688001736998558 0.01105081383138895 -0.009003818966448307\nv 0.0662405788898468 -0.0007938496419228613 -0.008673690259456635\nv 0.06709571182727814 0.0016311444342136383 -0.0007653982029296458\nv 0.07024703919887543 0.014973660930991173 -0.0008233339176513255\nv 0.09695196896791458 0.016333624720573425 0.0928298681974411\nv 0.09556055814027786 0.021853497251868248 0.09346045553684235\nv 0.096980519592762 -0.025056641548871994 0.06320597231388092\nv 0.09603075683116913 -0.023865845054388046 0.0682482123374939\nv 0.0957508385181427 -0.027027003467082977 0.06858387589454651\nv 0.09904640913009644 -0.029259080067276955 0.06490246206521988\nv 0.07856962829828262 -0.03079616092145443 0.013906420208513737\nv 0.09489047527313232 -0.044211000204086304 0.05243838578462601\nv 0.08535346388816833 -0.0411541610956192 0.05556129664182663\nv 0.0748315155506134 -0.02534787543118 0.004214860498905182\nv 0.07039519399404526 -0.018079280853271484 0.005920753348618746\nv 0.06964287161827087 -0.018454141914844513 -0.0030979143921285868\nv 0.07183246314525604 -0.013284113258123398 0.0679916962981224\nv 0.06955385208129883 -0.011937357485294342 0.021499576047062874\nv 0.06870152056217194 -0.012083805166184902 0.00831263605505228\nv 0.06776849180459976 -0.012296067550778389 -0.002287887269631028\nv 0.06653960794210434 -0.01300827320665121 -0.009996488690376282\nv 0.06929195672273636 -0.019739557057619095 -0.010172829031944275\nv 0.0818871408700943 0.016388699412345886 0.10195586085319519\nv 0.08530055731534958 0.010105051100254059 0.10577193647623062\nv 0.06699884682893753 -0.022000275552272797 -0.018073970451951027\nv 0.06492932885885239 -0.013940362259745598 -0.01775551773607731\nv 0.06795584410429001 -0.004060036037117243 0.010362938046455383\nv 0.0673689991235733 0.005898629315197468 0.012802742421627045\nv 0.06862117350101471 -0.004629070404917002 0.0230640210211277\nv 0.06832101196050644 0.00508287874981761 0.026582997292280197\nv 0.06534255295991898 -0.005554682109504938 0.06711199134588242\nv 0.10090867429971695 0.0079244589433074 0.11017333716154099\nv 0.09242511540651321 0.007050279527902603 0.10493535548448563\nv 0.09516659379005432 0.0074552311562001705 0.10032538324594498\nv 0.10270119458436966 0.006829131860285997 0.10684879869222641\nv 0.08892764151096344 0.006780424155294895 0.1011762022972107\nv 0.08952357620000839 0.010397772304713726 0.10790194571018219\nv 0.07174763083457947 0.016963528469204903 0.029318014159798622\nv 0.0881638377904892 0.00679026311263442 0.07698696106672287\nv 0.07325604557991028 0.020678525790572166 0.014157518744468689\nv 0.07877837121486664 0.028680259361863136 0.03671887516975403\nv 0.0822744369506836 0.03156600520014763 0.02582007274031639\nv 0.07717494666576385 0.025038650259375572 -0.0009599552140571177\nv 0.08428376168012619 0.03328302875161171 0.01146619115024805\nv 0.07378752529621124 0.017521288245916367 -0.016901493072509766\nv 0.07480984926223755 0.02029043808579445 -0.008761965669691563\nv 0.09288359433412552 0.0375823974609375 0.023677224293351173\nv 0.10484781861305237 0.01848609559237957 0.10087480396032333\nv 0.10492610186338425 0.013303219340741634 0.10009642690420151\nv 0.06771355122327805 0.009220598265528679 -0.01663634181022644\nv 0.11331065744161606 0.01634730026125908 0.03815999627113342\nv 0.10339796543121338 -0.011985878460109234 0.020528066903352737\nv 0.10093193501234055 -0.007741913199424744 0.021396173164248466\nv 0.10758031159639359 -0.0275997556746006 0.015313037671148777\nv 0.10722756385803223 -0.017614303156733513 0.018705317750573158\nv 0.10340719670057297 0.02215547300875187 0.10381390154361725\nv 0.08976732194423676 -0.010763966478407383 0.07729636877775192\nv 0.08927898854017258 -0.011879827827215195 0.07617679238319397\nv 0.11881337314844131 0.031483180820941925 0.03776717558503151\nv 0.11903183907270432 0.034737519919872284 0.04357830807566643\nv 0.10924222320318222 0.0417158268392086 0.0434027723968029\nv 0.11543767154216766 0.039488784968853 0.042573023587465286\nv 0.11382784694433212 0.037179965525865555 0.03536113351583481\nv 0.10756535083055496 0.03964782506227493 0.035578787326812744\nv 0.10484609007835388 -0.006275427062064409 0.011816337704658508\nv 0.10618429630994797 -0.009479009546339512 0.011916161514818668\nv 0.08927493542432785 -0.03559454157948494 0.012706540524959564\nv 0.09936922043561935 -0.03572707250714302 0.013979226350784302\nv 0.07982035726308823 0.026377057656645775 0.07672037184238434\nv 0.07280942052602768 0.023518485948443413 0.07021381705999374\nv 0.07480321824550629 0.022992417216300964 0.07794234901666641\nv 0.07436933368444443 0.017655448988080025 0.08536778390407562\nv 0.08488689363002777 0.006599490065127611 0.07948897033929825\nv 0.08173840492963791 0.008599556982517242 0.07996764034032822\nv 0.10777415335178375 0.006025450769811869 0.004545524716377258\nv 0.10240230709314346 0.003340059192851186 0.014720656909048557\nv 0.10854387283325195 0.020398039370775223 0.04769584909081459\nv 0.10885907709598541 0.023624904453754425 0.05239100381731987\nv 0.09343933314085007 -0.0023422404192388058 0.06189468130469322\nv 0.07709788531064987 -0.01116418931633234 0.0767970159649849\nv 0.07133799046278 -0.004780727904289961 0.07600247859954834\nv 0.09141908586025238 -0.0018139667809009552 0.06982462853193283\nv 0.10060973465442657 0.021158616989850998 0.11087476462125778\nv 0.10235165804624557 0.022806329652667046 0.10704373568296432\nv 0.07162736356258392 0.017459925264120102 0.07732491940259933\nv 0.0975027084350586 -0.02044527977705002 0.05729398876428604\nv 0.07793529331684113 -0.015420072712004185 0.07362351566553116\nv 0.08385299891233444 -0.013055804185569286 0.07817527651786804\nv 0.09956632554531097 -0.04146609827876091 0.04872610419988632\nv 0.06542887538671494 -0.0018546223873272538 -0.01682181842625141\nv 0.09792914986610413 0.01037816796451807 0.09713037312030792\nv 0.10412082076072693 0.009001982398331165 0.10142438858747482\nv 0.09897857159376144 0.010719705373048782 0.11212009936571121\nv 0.09813834726810455 0.01632429100573063 0.1126479059457779\nv 0.10029064863920212 -0.031451739370822906 0.005138911306858063\nv 0.09953232854604721 -0.0048093064688146114 0.022288620471954346\nv 0.0991264283657074 0.0383237861096859 0.03393479064106941\nv 0.10309038311243057 0.040303416550159454 0.04418168589472771\nv 0.08988425880670547 -0.0010531917214393616 0.07410041987895966\nv 0.0855180099606514 -0.035052668303251266 0.060899294912815094\nv 0.07667776942253113 -0.02446478046476841 0.06797457486391068\nv 0.08161953836679459 -0.02498033456504345 0.07547588646411896\nv 0.08246126025915146 -0.019260095432400703 0.07920373231172562\nv 0.08687400072813034 -0.014529784210026264 0.08011724054813385\nv 0.10432850569486618 0.010655212216079235 0.11375997960567474\nv 0.10344551503658295 0.015914831310510635 0.11433034390211105\nv 0.10537561029195786 0.02071419358253479 0.11231676489114761\nv 0.10645269602537155 0.022376686334609985 0.10867892950773239\nv 0.10674712806940079 0.021315833553671837 0.10477187484502792\nv 0.10635119676589966 0.0077679031528532505 0.11180484294891357\nv 0.10774705559015274 0.006964271422475576 0.10867329686880112\nv 0.10718252509832382 0.008627559058368206 0.10319996625185013\nv 0.11115296185016632 0.0075631109066307545 0.11188062280416489\nv 0.11258053034543991 0.010803195647895336 0.11535993218421936\nv 0.11866654455661774 0.02130640298128128 0.11167270690202713\nv 0.1134837344288826 0.020623892545700073 0.11338527500629425\nv 0.11559610068798065 0.019361982122063637 0.11443730443716049\nv 0.12016274780035019 0.019388625398278236 0.11349023878574371\nv 0.12495396286249161 0.01839272864162922 0.11356497555971146\nv 0.12400609999895096 0.020803557708859444 0.11144670099020004\nv 0.11324738711118698 0.01596684753894806 0.11562197655439377\nv 0.11953478306531906 0.015452609397470951 0.11489148437976837\nv 0.11936898529529572 0.011614941991865635 0.1146402657032013\nv 0.12463051825761795 0.011649989522993565 0.11455485224723816\nv 0.1249130591750145 0.014984075911343098 0.11498311907052994\nv 0.11513824760913849 0.012224603444337845 0.11546681076288223\nv 0.11089231818914413 0.01600341498851776 0.1158091127872467\nv 0.12782344222068787 0.018659541383385658 0.11182832717895508\nv 0.12759655714035034 0.01750166341662407 0.11384749412536621\nv 0.1277681440114975 0.014633995480835438 0.11440641433000565\nv 0.12999053299427032 0.01464011613279581 0.11272108554840088\nv 0.11078564822673798 0.022292813286185265 0.1089676097035408\nv 0.11853587627410889 0.02203471213579178 0.10931064933538437\nv 0.12389787286520004 0.020167050883173943 0.10859028249979019\nv 0.12770676612854004 0.018079636618494987 0.10924037545919418\nv 0.11934894323348999 0.020027881488204002 0.10615507513284683\nv 0.12366504967212677 0.018457714468240738 0.10648705065250397\nv 0.12675690650939941 0.01627535931766033 0.10692723840475082\nv 0.1199372187256813 0.016819141805171967 0.10400138050317764\nv 0.12363829463720322 0.01589057967066765 0.10510793328285217\nv 0.10959358513355255 0.017923621460795403 0.10293251276016235\nv 0.10899408906698227 0.02118832804262638 0.1050114631652832\nv 0.12605345249176025 0.015171049162745476 0.10536177456378937\nv 0.12999188899993896 0.013907549902796745 0.11072052270174026\nv 0.11776057630777359 0.008905084803700447 0.113240547478199\nv 0.1243242397904396 0.009664873592555523 0.11240724474191666\nv 0.11762671172618866 0.007713876198977232 0.11131990700960159\nv 0.124140664935112 0.008152167312800884 0.11079742759466171\nv 0.1285427212715149 0.010454194620251656 0.1128614991903305\nv 0.12737193703651428 0.012098709121346474 0.11405709385871887\nv 0.10669850558042526 0.017645055428147316 0.10186115652322769\nv 0.10692640393972397 0.012223990634083748 0.10131432861089706\nv 0.10949618369340897 0.012320034205913544 0.10224386304616928\nv 0.11984838545322418 0.012494608759880066 0.10334113985300064\nv 0.11022981256246567 0.008755449205636978 0.1039520874619484\nv 0.11218420416116714 0.006941710598766804 0.10855355113744736\nv 0.11868289858102798 0.007404694799333811 0.10873425006866455\nv 0.11922138929367065 0.008540659211575985 0.10582385957241058\nv 0.12369230389595032 0.008247007615864277 0.10841856896877289\nv 0.12342163175344467 0.009588262997567654 0.10652203857898712\nv 0.12383442372083664 0.012578068301081657 0.10468832403421402\nv 0.12716080248355865 0.010020898655056953 0.1082996353507042\nv 0.12811720371246338 0.00990602932870388 0.11090055853128433\nv 0.12843549251556396 0.013235095888376236 0.10798825323581696\nv 0.12606920301914215 0.010310059413313866 0.10682119429111481\nv 0.12647303938865662 0.013021416030824184 0.10641910880804062\nv 0.1032899022102356 -2.0626934201573022e-05 0.09740043431520462\nv 0.1050267368555069 -0.005203413777053356 0.09550277143716812\nv 0.10126051306724548 -0.004923070315271616 0.09563297778367996\nv 0.09977009892463684 -0.0006417878903448582 0.09715542942285538\nv 0.09856535494327545 -0.0016807205975055695 0.10945349931716919\nv 0.10053173452615738 0.0014948835596442223 0.1049874946475029\nv 0.09503168612718582 0.0008049957104958594 0.10512449592351913\nv 0.0928054079413414 -0.0036064647138118744 0.10891411453485489\nv 0.09803896397352219 -0.008300520479679108 0.11053139716386795\nv 0.09244404733181 -0.009959197603166103 0.10905933380126953\nv 0.08820600062608719 -0.0032606583554297686 0.10528319329023361\nv 0.09127867221832275 0.001195215154439211 0.10175878554582596\nv 0.08201218396425247 0.004234382417052984 0.08451277762651443\nv 0.07790355384349823 0.00033672296558506787 0.0865769311785698\nv 0.09278634190559387 -0.0064664846286177635 0.07855230569839478\nv 0.09150741249322891 -0.009379466064274311 0.07995625585317612\nv 0.10212410986423492 0.0015946170315146446 0.10049751400947571\nv 0.09768985956907272 0.0011344728991389275 0.1005544513463974\nv 0.099787138402462 -0.012475690804421902 0.09386718273162842\nv 0.10124480724334717 -0.00795657280832529 0.0927131250500679\nv 0.10239710658788681 -0.009099931456148624 0.09463819116353989\nv 0.10157150775194168 -0.013411249034106731 0.09706329554319382\nv 0.08842288702726364 0.0037187563721090555 0.08054692298173904\nv 0.09056572616100311 -0.00125436345115304 0.07900863885879517\nv 0.08564342558383942 0.004632189869880676 0.08239571750164032\nv 0.0961705893278122 -0.01589551381766796 0.09749633818864822\nv 0.0916682779788971 -0.015716375783085823 0.1016964539885521\nv 0.08085650950670242 -0.012673716060817242 0.08571226894855499\nv 0.0862443819642067 -0.013846339657902718 0.08371853828430176\nv 0.09471669048070908 0.0021382328122854233 0.09848080575466156\nv 0.10071486234664917 -0.0028719783294945955 0.09389521926641464\nv 0.09847711771726608 0.0008932040655054152 0.09527421742677689\nv 0.08791560679674149 -0.00986737571656704 0.1040862500667572\nv 0.09652187675237656 -0.015497344546020031 0.10606557875871658\nv 0.11529088020324707 -0.012561824172735214 0.10467150807380676\nv 0.10487128049135208 -0.015099695883691311 0.10269133001565933\nv 0.10607081651687622 -0.013364691287279129 0.09845399111509323\nv 0.11656473577022552 -0.011935455724596977 0.1004088819026947\nv 0.09964955598115921 -0.01593203842639923 0.10117506980895996\nv 0.1020800843834877 -0.013614996336400509 0.10707492381334305\nv 0.11608894914388657 0.00036618951708078384 0.09773847460746765\nv 0.11727289855480194 -0.004998189862817526 0.09578926861286163\nv 0.11524305492639542 0.0027495066169649363 0.10094457864761353\nv 0.07697568833827972 -0.006270498037338257 0.08658915758132935\nv 0.11171828210353851 0.0001718895509839058 0.10938061773777008\nv 0.11371675878763199 0.003038860857486725 0.10445452481508255\nv 0.10587313026189804 -0.00992021057754755 0.09524083137512207\nv 0.11760731041431427 -0.009385282173752785 0.09695809334516525\nv 0.11375200003385544 -0.010911057703197002 0.10806544125080109\nv 0.11064231395721436 -0.005629090126603842 0.11073490232229233\nv 0.11921840906143188 -0.009072351269423962 0.10858304798603058\nv 0.11730834096670151 -0.004213749431073666 0.11071636527776718\nv 0.1176086887717247 0.001306089572608471 0.10912740975618362\nv 0.1179000586271286 0.0034102164208889008 0.1044076606631279\nv 0.11809619516134262 0.002763833850622177 0.10091835260391235\nv 0.1205950602889061 -0.010853230953216553 0.10541022568941116\nv 0.12123463302850723 -0.01086338795721531 0.101219043135643\nv 0.12005504220724106 -0.008505551144480705 0.09752489626407623\nv 0.12680362164974213 -0.009110518731176853 0.10272692888975143\nv 0.12739567458629608 -0.0068908012472093105 0.1062706783413887\nv 0.13393165171146393 0.0074096922762691975 0.10004936903715134\nv 0.12925687432289124 0.0059366715140640736 0.10286907851696014\nv 0.1307617574930191 0.004409794230014086 0.10404996573925018\nv 0.13507777452468872 0.005574576556682587 0.10147656500339508\nv 0.12567788362503052 0.0036913270596414804 0.10684794187545776\nv 0.1277824342250824 0.002648480236530304 0.10663309693336487\nv 0.1278526932001114 -0.0015461203875020146 0.10747625678777695\nv 0.132225900888443 0.0003041401505470276 0.10406940430402756\nv 0.1324746310710907 -0.004004674032330513 0.10317099094390869\nv 0.13722461462020874 -0.0024953915271908045 0.10106129944324493\nv 0.13726931810379028 0.0016979874344542623 0.1021835058927536\nv 0.12943513691425323 -0.004318654537200928 0.10597266256809235\nv 0.13814668357372284 0.00557507062330842 0.09996220469474792\nv 0.13872301578521729 0.0065662139095366 0.09791400283575058\nv 0.12555423378944397 -0.00215710885822773 0.10932739824056625\nv 0.12123175710439682 0.004577899817377329 0.10402701050043106\nv 0.12824289500713348 0.006793555337935686 0.1003727912902832\nv 0.13350935280323029 0.007682312745600939 0.09715374559164047\nv 0.1386195719242096 0.00568285072222352 0.09526537358760834\nv 0.12808401882648468 0.005677978042513132 0.09672407805919647\nv 0.133301243185997 0.00625336030498147 0.09455254673957825\nv 0.13744251430034637 0.004262596368789673 0.09357324987649918\nv 0.12819388508796692 0.002913061762228608 0.09343418478965759\nv 0.13341882824897766 0.0032896308694034815 0.09212072938680649\nv 0.11956632882356644 0.0005384162068367004 0.09720668196678162\nv 0.11978227645158768 0.0037454182747751474 0.10009470582008362\nv 0.13591329753398895 0.003115421161055565 0.09243221580982208\nv 0.1414376050233841 0.003092440078034997 0.0983390212059021\nv 0.14092478156089783 0.002636101096868515 0.09526831656694412\nv 0.14035162329673767 0.002858113730326295 0.09966044872999191\nv 0.13230286538600922 -0.005614439491182566 0.10208465158939362\nv 0.13728801906108856 -0.003503291169181466 0.0992383360862732\nv 0.13217933475971222 -0.0068447524681687355 0.09972946345806122\nv 0.13746462762355804 -0.0040194489993155 0.0970582515001297\nv 0.14081242680549622 -0.0011098869144916534 0.0978732705116272\nv 0.14050474762916565 -0.00029730238020420074 0.09916521608829498\nv 0.11824000626802444 -0.00016960875655058771 0.09832077473402023\nv 0.11909108608961105 -0.00497772078961134 0.0967065840959549\nv 0.12023139744997025 -0.004343898966908455 0.09581071138381958\nv 0.12905673682689667 -0.0017059893580153584 0.09265042096376419\nv 0.12187985330820084 -0.008190642111003399 0.0969175174832344\nv 0.1253722608089447 -0.009524653665721416 0.09966295957565308\nv 0.1312437653541565 -0.007211018819361925 0.09732699394226074\nv 0.1304071992635727 -0.005472768563777208 0.09428226202726364\nv 0.1370144933462143 -0.004915190860629082 0.09529690444469452\nv 0.13478955626487732 -0.004163118079304695 0.09354081004858017\nv 0.13370241224765778 -0.0003833081864286214 0.09156336635351181\nv 0.14017125964164734 -0.002011124277487397 0.09424833208322525\nv 0.1408814638853073 -0.0014756117016077042 0.09626971185207367\nv 0.13964951038360596 0.0019788090139627457 0.09377981722354889\nv 0.13780616223812103 -0.0022428466472774744 0.09286052733659744\nv 0.13729874789714813 0.0010179132223129272 0.09224727749824524\nv 0.1095760315656662 -0.016615884378552437 0.08758565783500671\nv 0.11052798479795456 -0.020191552117466927 0.08374439179897308\nv 0.10841742157936096 -0.020467484369874 0.08345778286457062\nv 0.10719405859708786 -0.016919130459427834 0.08691997826099396\nv 0.10714749991893768 -0.02206660993397236 0.09729550778865814\nv 0.10840951651334763 -0.01814921945333481 0.09541823714971542\nv 0.10426029562950134 -0.018168959766626358 0.09496451169252396\nv 0.10168274492025375 -0.02199169248342514 0.09680542349815369\nv 0.1085871085524559 -0.026991549879312515 0.09650623798370361\nv 0.1013767272233963 -0.02784261852502823 0.09610500186681747\nv 0.09658120572566986 -0.021498102694749832 0.09444044530391693\nv 0.10032101720571518 -0.017769528552889824 0.09298742562532425\nv 0.08988618105649948 -0.015648139640688896 0.0828501284122467\nv 0.08648160845041275 -0.020051371306180954 0.08460512012243271\nv 0.10898952931165695 -0.01650252379477024 0.09169723093509674\nv 0.10607379674911499 -0.016543975099921227 0.09093539416790009\nv 0.10670298337936401 -0.028388774022459984 0.08115784078836441\nv 0.10762277245521545 -0.02452797256410122 0.08082887530326843\nv 0.10924586653709412 -0.02500881813466549 0.08211183547973633\nv 0.10859870910644531 -0.028557520359754562 0.08299507200717926\nv 0.0949457660317421 -0.014483774080872536 0.07502264529466629\nv 0.09551487118005753 -0.019120600074529648 0.07209043204784393\nv 0.09338255226612091 -0.013986478559672832 0.0792342945933342\nv 0.10377586632966995 -0.03176109120249748 0.08473232388496399\nv 0.09916339814662933 -0.03209473937749863 0.08951728790998459\nv 0.09009502828121185 -0.030594682320952415 0.07833980768918991\nv 0.09386198222637177 -0.030068349093198776 0.0744277834892273\nv 0.10315927118062973 -0.016214637085795403 0.08959729969501495\nv 0.10694342106580734 -0.019693566486239433 0.0818173736333847\nv 0.10554244369268417 -0.016409238800406456 0.08531935513019562\nv 0.09616407006978989 -0.026851776987314224 0.09265515208244324\nv 0.10445668548345566 -0.032046739012002945 0.09190448373556137\nv 0.12440983951091766 -0.029536955058574677 0.08964323252439499\nv 0.110344298183918 -0.031860072165727615 0.0874260887503624\nv 0.11092961579561234 -0.028540683910250664 0.0833253562450409\nv 0.12474595755338669 -0.027993855997920036 0.08590269833803177\nv 0.10729695856571198 -0.03202817216515541 0.08662267029285431\nv 0.10907372832298279 -0.03140468895435333 0.09252336621284485\nv 0.12305568158626556 -0.015860363841056824 0.08769822865724564\nv 0.12329458445310593 -0.019741596654057503 0.08452440053224564\nv 0.12173467129468918 -0.015603824518620968 0.09147083759307861\nv 0.08590803295373917 -0.026155725121498108 0.08293206989765167\nv 0.09640762954950333 -0.026218993589282036 0.07139524817466736\nv 0.12181458622217178 -0.019476499408483505 0.09702539443969727\nv 0.12163597345352173 -0.01650325581431389 0.09444383531808853\nv 0.11088816821575165 -0.02478843927383423 0.08239825814962387\nv 0.12425405532121658 -0.024658771231770515 0.08374037593603134\nv 0.12295478582382202 -0.028887292370200157 0.0936700701713562\nv 0.12127089500427246 -0.02483643777668476 0.09705769270658493\nv 0.12844830751419067 -0.027866505086421967 0.09307372570037842\nv 0.12707029283046722 -0.023851947858929634 0.09646270424127579\nv 0.12724201381206512 -0.018759028986096382 0.09688860177993774\nv 0.12630492448806763 -0.01590782403945923 0.09430805593729019\nv 0.1258469820022583 -0.01487682107836008 0.09135432541370392\nv 0.12922212481498718 -0.028047451749444008 0.08869515359401703\nv 0.1287967562675476 -0.026378409937024117 0.08551767468452454\nv 0.12595561146736145 -0.0235277209430933 0.08430075645446777\nv 0.13270562887191772 -0.02673397772014141 0.08673324435949326\nv 0.13472497463226318 -0.02587597444653511 0.09133416414260864\nv 0.14288993179798126 -0.013181854970753193 0.09015388786792755\nv 0.1382373720407486 -0.01460975967347622 0.09212429076433182\nv 0.1387365758419037 -0.015668341889977455 0.0922546312212944\nv 0.1436527669429779 -0.01438334584236145 0.09067200869321823\nv 0.13390889763832092 -0.017624683678150177 0.09518097341060638\nv 0.1355418860912323 -0.018122971057891846 0.09452484548091888\nv 0.13551238179206848 -0.021629348397254944 0.0940568670630455\nv 0.14058460295200348 -0.019915694370865822 0.09154394268989563\nv 0.13957111537456512 -0.02349298819899559 0.08770479261875153\nv 0.14361223578453064 -0.021253276616334915 0.08641652762889862\nv 0.14506292343139648 -0.018299361690878868 0.0899755209684372\nv 0.1360955834388733 -0.024486901238560677 0.09120263159275055\nv 0.14630025625228882 -0.01440724078565836 0.08931579440832138\nv 0.1468389481306076 -0.012528783641755581 0.08947775512933731\nv 0.1336020976305008 -0.022365108132362366 0.09514329582452774\nv 0.13117633759975433 -0.014368116855621338 0.09391950070858002\nv 0.13711103796958923 -0.012423975393176079 0.09102172404527664\nv 0.14179855585098267 -0.010468833148479462 0.08897820860147476\nv 0.14528124034404755 -0.011010020039975643 0.08682664483785629\nv 0.13575328886508942 -0.011703581549227238 0.08797794580459595\nv 0.14033447206020355 -0.010456173680722713 0.086470827460289\nv 0.1435161978006363 -0.010838608257472515 0.08461030572652817\nv 0.13496731221675873 -0.012413134798407555 0.08519537001848221\nv 0.13919849693775177 -0.011665272526443005 0.08403735607862473\nv 0.12714189291000366 -0.015014897100627422 0.08786505460739136\nv 0.12882135808467865 -0.01385860051959753 0.09104901552200317\nv 0.14203979074954987 -0.011701948009431362 0.08333799242973328\nv 0.14851023256778717 -0.015952996909618378 0.087407186627388\nv 0.14726553857326508 -0.014612779021263123 0.08392780274152756\nv 0.1470978856086731 -0.016681430861353874 0.08811084926128387\nv 0.1386866271495819 -0.024363934993743896 0.08652867376804352\nv 0.14392071962356567 -0.02192476950585842 0.08441463112831116\nv 0.13753776252269745 -0.024160267785191536 0.08415449410676956\nv 0.14276373386383057 -0.021789811551570892 0.08221741020679474\nv 0.14711010456085205 -0.019594179466366768 0.08487527072429657\nv 0.1466628611087799 -0.01945173367857933 0.08584865927696228\nv 0.12507112324237823 -0.016138751059770584 0.08801154047250748\nv 0.1250620186328888 -0.01945568434894085 0.08499415963888168\nv 0.12672342360019684 -0.018030479550361633 0.08485066145658493\nv 0.13499262928962708 -0.015015832148492336 0.08198078721761703\nv 0.12755253911018372 -0.02201845496892929 0.08362459391355515\nv 0.13193221390247345 -0.02468089759349823 0.08402301371097565\nv 0.13647852838039398 -0.02216380275785923 0.08217626810073853\nv 0.1358475387096405 -0.01896696723997593 0.08109667152166367\nv 0.141312375664711 -0.02028404362499714 0.08067960292100906\nv 0.13966237008571625 -0.018258867785334587 0.08075018972158432\nv 0.13905619084835052 -0.013918456621468067 0.08117491751909256\nv 0.14415103197097778 -0.017706168815493584 0.08139318972826004\nv 0.14611321687698364 -0.01896422915160656 0.08261416852474213\nv 0.14501075446605682 -0.013765060342848301 0.08226287364959717\nv 0.1423063427209854 -0.016569796949625015 0.08075698465108871\nv 0.14283911883831024 -0.013547507114708424 0.08142600953578949\nv 0.09640948474407196 -0.02291903831064701 0.0707865059375763\nv 0.11555365473031998 -0.03463287651538849 0.07482092827558517\nv 0.11637800931930542 -0.036234669387340546 0.07152833044528961\nv 0.1152157261967659 -0.03715549781918526 0.07131767272949219\nv 0.11340268701314926 -0.034956205636262894 0.0740366280078888\nv 0.11262009292840958 -0.04165138304233551 0.08157122880220413\nv 0.11372382193803787 -0.03766936808824539 0.08073669672012329\nv 0.10918040573596954 -0.038049448281526566 0.08034078776836395\nv 0.1078503280878067 -0.04178611561655998 0.08103957027196884\nv 0.1129443496465683 -0.045806992799043655 0.07955830544233322\nv 0.10869317501783371 -0.04607894644141197 0.07875346392393112\nv 0.10454851388931274 -0.04105950519442558 0.07884945720434189\nv 0.10591476410627365 -0.03722427412867546 0.07834935933351517\nv 0.09471819549798965 -0.033276867121458054 0.06806597858667374\nv 0.09210982173681259 -0.03744085505604744 0.06635653227567673\nv 0.10287626832723618 -0.04120049625635147 0.0532560832798481\nv 0.1038723811507225 -0.03621189668774605 0.05280766263604164\nv 0.10788387805223465 -0.03837212547659874 0.059563782066106796\nv 0.10601241141557693 -0.04260322451591492 0.05876469612121582\nv 0.11471642553806305 -0.03540494665503502 0.07811713963747025\nv 0.11106940358877182 -0.035143524408340454 0.07745856046676636\nv 0.11442932486534119 -0.04521026834845543 0.0677284300327301\nv 0.11513736099004745 -0.04079146683216095 0.06801683455705643\nv 0.11594807356595993 -0.039926741272211075 0.06897318363189697\nv 0.11634797602891922 -0.044014230370521545 0.06829552352428436\nv 0.1034734845161438 -0.03206121176481247 0.05595438554883003\nv 0.10098595917224884 -0.03020869567990303 0.05955047532916069\nv 0.10398795455694199 -0.030506044626235962 0.0626780092716217\nv 0.1077612042427063 -0.03453320637345314 0.06220429763197899\nv 0.10053461045026779 -0.030150016769766808 0.06676967442035675\nv 0.11220576614141464 -0.04800565913319588 0.06949389725923538\nv 0.10865067690610886 -0.04905819520354271 0.07329204678535461\nv 0.09821804612874985 -0.047106582671403885 0.0600256584584713\nv 0.10195900499820709 -0.04547635093331337 0.05843530222773552\nv 0.10922293365001678 -0.03513436019420624 0.07689525932073593\nv 0.11404668539762497 -0.037273213267326355 0.07005821913480759\nv 0.11283030360937119 -0.03526827692985535 0.07361394912004471\nv 0.10511423647403717 -0.045974042266607285 0.07685314863920212\nv 0.11187732219696045 -0.04838947206735611 0.07453969120979309\nv 0.12492034584283829 -0.04505646228790283 0.0712180882692337\nv 0.11709708720445633 -0.04675217345356941 0.07012289017438889\nv 0.11787078529596329 -0.04292157664895058 0.06794553250074387\nv 0.12490996718406677 -0.04279274493455887 0.06979856640100479\nv 0.11501188576221466 -0.04716724902391434 0.07049418240785599\nv 0.11571966111660004 -0.047947902232408524 0.07481832802295685\nv 0.12322351336479187 -0.03373130410909653 0.07569180428981781\nv 0.12351220846176147 -0.0362832210958004 0.07172036170959473\nv 0.12283299118280411 -0.03419327735900879 0.07845614850521088\nv 0.0928882583975792 -0.04316134378314018 0.06348787248134613\nv 0.1226518452167511 -0.03992471843957901 0.08181563019752502\nv 0.12309779971837997 -0.03609746694564819 0.08093160390853882\nv 0.11736834049224854 -0.038977235555648804 0.06913308054208755\nv 0.12421475350856781 -0.03959062322974205 0.06982284784317017\nv 0.12429871410131454 -0.046326544135808945 0.07478681206703186\nv 0.12267161160707474 -0.04461805522441864 0.07975012809038162\nv 0.12883834540843964 -0.04487496614456177 0.07487255334854126\nv 0.12736301124095917 -0.04319288954138756 0.07996422052383423\nv 0.127131387591362 -0.03816736862063408 0.08170734345912933\nv 0.12603113055229187 -0.035268835723400116 0.08053924888372421\nv 0.12550592422485352 -0.033567871898412704 0.0784558579325676\nv 0.12866318225860596 -0.04372087121009827 0.07105052471160889\nv 0.1279785931110382 -0.04150904342532158 0.06959593296051025\nv 0.1264481395483017 -0.03854823485016823 0.06956078857183456\nv 0.13232965767383575 -0.04210575297474861 0.07086850702762604\nv 0.1340535581111908 -0.04238269105553627 0.07472100853919983\nv 0.13880136609077454 -0.03165511414408684 0.07741370052099228\nv 0.13542649149894714 -0.033040665090084076 0.07862737029790878\nv 0.13594995439052582 -0.03406298905611038 0.0786876231431961\nv 0.1393207609653473 -0.0323011688888073 0.07808571308851242\nv 0.1319623589515686 -0.036194488406181335 0.08028090000152588\nv 0.13332200050354004 -0.03672036528587341 0.07967299222946167\nv 0.13308881223201752 -0.03970407694578171 0.07857070863246918\nv 0.13734233379364014 -0.03740420192480087 0.0775524377822876\nv 0.13829205930233002 -0.03963065892457962 0.07344000786542892\nv 0.1414940059185028 -0.037472229450941086 0.07314655929803848\nv 0.14053772389888763 -0.035912539809942245 0.07689360529184341\nv 0.13516783714294434 -0.04138747975230217 0.07574692368507385\nv 0.1418166607618332 -0.032181546092033386 0.07738256454467773\nv 0.14238114655017853 -0.031152108684182167 0.07717783749103546\nv 0.13185366988182068 -0.04066065698862076 0.07921022176742554\nv 0.12944656610488892 -0.033516060560941696 0.07966123521327972\nv 0.1350087672472 -0.031009618192911148 0.07789479941129684\nv 0.13936485350131989 -0.029799913987517357 0.07662581652402878\nv 0.1425800770521164 -0.030233165249228477 0.07537197321653366\nv 0.1335320919752121 -0.030230866745114326 0.07623272389173508\nv 0.13862499594688416 -0.02862175554037094 0.07496988028287888\nv 0.14167673885822296 -0.029169240966439247 0.07361249625682831\nv 0.13245542347431183 -0.030128996819257736 0.07342970371246338\nv 0.13753098249435425 -0.029064161702990532 0.07282303273677826\nv 0.12692132592201233 -0.03242115676403046 0.07430781424045563\nv 0.12801849842071533 -0.0323813334107399 0.07733704894781113\nv 0.14034420251846313 -0.02949734777212143 0.07257416844367981\nv 0.14422136545181274 -0.033513836562633514 0.07553848624229431\nv 0.14390791952610016 -0.03197803348302841 0.07348647713661194\nv 0.1434841752052307 -0.03441876545548439 0.0759851261973381\nv 0.13802334666252136 -0.03986458480358124 0.07232829183340073\nv 0.1417936533689499 -0.03731865435838699 0.07187829911708832\nv 0.13720126450061798 -0.03933681547641754 0.07021438330411911\nv 0.14137981832027435 -0.03622209280729294 0.07032937556505203\nv 0.14418812096118927 -0.03538662567734718 0.07289806008338928\nv 0.14323025941848755 -0.03588739037513733 0.07411963492631912\nv 0.1253904104232788 -0.033083509653806686 0.07484007626771927\nv 0.1253831535577774 -0.03543032705783844 0.07170548290014267\nv 0.12719188630580902 -0.03469128906726837 0.07112689316272736\nv 0.13246840238571167 -0.031955499202013016 0.07035962492227554\nv 0.128432959318161 -0.037773650139570236 0.06905286014080048\nv 0.1305241584777832 -0.04047928377985954 0.06930394470691681\nv 0.13590747117996216 -0.037656571716070175 0.06891538202762604\nv 0.13427859544754028 -0.03507319092750549 0.06853337585926056\nv 0.140676349401474 -0.03467491269111633 0.06937180459499359\nv 0.13886302709579468 -0.032644983381032944 0.06904475390911102\nv 0.1375000774860382 -0.030431805178523064 0.07029496133327484\nv 0.1427265852689743 -0.03256295621395111 0.07090723514556885\nv 0.14379926025867462 -0.03371872007846832 0.07161907106637955\nv 0.14277727901935577 -0.03106234036386013 0.07217742502689362\nv 0.14133180677890778 -0.03154513239860535 0.07038559764623642\nv 0.14087049663066864 -0.030500903725624084 0.0712389424443245\nv 0.10087884217500687 -0.043712932616472244 0.055240076035261154\nv 0.1051640510559082 0.03852478787302971 0.05778120085597038\nv 0.10518199950456619 0.03387000784277916 0.05965564027428627\nv 0.11939319968223572 0.03038926236331463 0.059217486530542374\nv 0.11351389437913895 0.028928274288773537 0.06133124232292175\nv 0.12266989797353745 0.03502143546938896 0.05674109607934952\nv 0.11874741315841675 0.04317871853709221 0.052810944616794586\nv 0.12178151309490204 0.039296869188547134 0.053744908422231674\nv 0.10870464891195297 0.030453111976385117 0.06126449629664421\nv 0.10724687576293945 0.04252826049923897 0.055330872535705566\nv 0.1129489541053772 0.04502785950899124 0.05280112847685814\nv 0.11090394109487534 0.045472461730241776 0.06210560351610184\nv 0.1167549192905426 0.04696273431181908 0.05919528380036354\nv 0.12272004038095474 0.04462628439068794 0.058877892792224884\nv 0.12496976554393768 0.03970523923635483 0.05928201600909233\nv 0.12550903856754303 0.03566904738545418 0.060544684529304504\nv 0.10817193984985352 0.04020961746573448 0.06370847672224045\nv 0.10827053338289261 0.0355290062725544 0.0653655156493187\nv 0.11149489134550095 0.030997347086668015 0.06540606170892715\nv 0.11117707937955856 0.041315946727991104 0.07133372128009796\nv 0.1178535595536232 0.04543782025575638 0.07687834650278091\nv 0.13446548581123352 0.04308575391769409 0.0816866010427475\nv 0.1325179636478424 0.042190633714199066 0.07598987221717834\nv 0.13096405565738678 0.04427921772003174 0.07706297188997269\nv 0.13315165042877197 0.04532121121883392 0.08194775879383087\nv 0.12979654967784882 0.04374125227332115 0.07222718745470047\nv 0.12871159613132477 0.044718723744153976 0.07401075214147568\nv 0.1237979307770729 0.0461345873773098 0.07464640587568283\nv 0.1260792464017868 0.047007832676172256 0.07924411445856094\nv 0.1202850267291069 0.045597318559885025 0.07757969200611115\nv 0.12099585682153702 0.04521786794066429 0.081334687769413\nv 0.1281311810016632 0.0475325733423233 0.08388058841228485\nv 0.1331184357404709 0.04686668887734413 0.08528085052967072\nv 0.13000676035881042 0.04762162268161774 0.08771510422229767\nv 0.1342526525259018 0.04440072923898697 0.08630099147558212\nv 0.12354106456041336 0.048041753470897675 0.07390006631612778\nv 0.1297176480293274 0.03986872360110283 0.06704854965209961\nv 0.13280314207077026 0.038916293531656265 0.0743185505270958\nv 0.1343654841184616 0.039632026106119156 0.08135875314474106\nv 0.133956640958786 0.04148607328534126 0.08673477172851562\nv 0.13148120045661926 0.0350736603140831 0.07419858872890472\nv 0.13217754662036896 0.0360928475856781 0.08089737594127655\nv 0.13109169900417328 0.03783329203724861 0.08589178323745728\nv 0.128106951713562 0.032354965806007385 0.07522817701101303\nv 0.12901556491851807 0.034297164529561996 0.08123848587274551\nv 0.12334364652633667 0.03128392994403839 0.06477745622396469\nv 0.1278221309185028 0.035046420991420746 0.06458807736635208\nv 0.12911275029182434 0.03640327230095863 0.08474811166524887\nv 0.13060098886489868 0.04663725942373276 0.0897684320807457\nv 0.13004396855831146 0.04319164156913757 0.0903364047408104\nv 0.11826733499765396 0.04372299835085869 0.08154419809579849\nv 0.12281353026628494 0.046459998935461044 0.08575589954853058\nv 0.12122657150030136 0.044250812381505966 0.08639732748270035\nv 0.11589593440294266 0.040840473026037216 0.08083096891641617\nv 0.12005145847797394 0.04173114895820618 0.08674366772174835\nv 0.1251567155122757 0.04564044624567032 0.08962538838386536\nv 0.12593677639961243 0.0472341850399971 0.08793806284666061\nv 0.12220243364572525 0.031029677018523216 0.061788566410541534\nv 0.1163644939661026 0.029354281723499298 0.06348821520805359\nv 0.11749694496393204 0.029980121180415154 0.06666118651628494\nv 0.12229026108980179 0.03153235837817192 0.07639746367931366\nv 0.1129160150885582 0.03190784528851509 0.06962337344884872\nv 0.11068496108055115 0.03690909966826439 0.07119645178318024\nv 0.11556120961904526 0.0372353233397007 0.08028476685285568\nv 0.11757533997297287 0.033564526587724686 0.07849548012018204\nv 0.1203417107462883 0.038381341844797134 0.0860460102558136\nv 0.12163040786981583 0.03626030683517456 0.08470739424228668\nv 0.1253446489572525 0.03419932350516319 0.08295675367116928\nv 0.12429992854595184 0.03960695117712021 0.08875499665737152\nv 0.12424265593290329 0.04269115626811981 0.08990246802568436\nv 0.12830759584903717 0.039398033171892166 0.08876646310091019\nv 0.12416199594736099 0.037653349339962006 0.08682838082313538\nv 0.12687736749649048 0.037137895822525024 0.08641650527715683\nv 0.10233891755342484 -0.03636704757809639 0.0490579791367054\nv 0.10220561921596527 -0.03123542293906212 0.0518113374710083\nv 0.10244607925415039 -0.03204593062400818 0.046612225472927094\nv 0.10412950068712234 -0.03131403028964996 0.03901771828532219\nv 0.10383128374814987 -0.03480549156665802 0.030231984332203865\nv 0.10297253727912903 -0.025509515777230263 0.04043158143758774\nv 0.1010696142911911 -0.02697722800076008 0.04863359406590462\nv 0.09973923861980438 -0.026383088901638985 0.05648477375507355\nv 0.10157529264688492 -0.018894892185926437 0.03418475762009621\nf 2/2 3/3 1/1\nf 1/1 3/3 4/4\nf 5/5 6/6 8/8\nf 8/8 6/6 7/7\nf 9/9 4/4 10/10\nf 10/10 4/4 3/3\nf 12/12 13/13 11/11\nf 11/11 13/13 9/9\nf 14/14 12/12 16/16\nf 16/16 12/12 15/15\nf 18/18 19/19 17/17\nf 17/17 19/19 20/20\nf 22/22 23/23 21/21\nf 21/21 23/23 24/24\nf 26/26 27/27 25/25\nf 25/25 27/27 28/28\nf 30/30 31/31 29/29\nf 29/29 31/31 32/32\nf 33/33 34/34 36/36\nf 36/36 34/34 35/35\nf 37/37 38/38 40/40\nf 40/40 38/38 39/39\nf 41/41 42/42 6/6\nf 6/6 42/42 43/43\nf 6/6 43/43 7/7\nf 7/7 43/43 44/44\nf 45/45 34/34 46/46\nf 46/46 34/34 33/33\nf 48/48 49/49 47/47\nf 47/47 49/49 50/50\nf 52/52 53/53 51/51\nf 51/51 53/53 54/54\nf 56/56 52/52 55/55\nf 55/55 52/52 51/51\nf 58/58 59/59 57/57\nf 57/57 59/59 60/60\nf 61/61 62/62 4/4\nf 4/4 62/62 1/1\nf 64/64 65/65 63/63\nf 63/63 65/65 66/66\nf 68/68 69/69 67/67\nf 67/67 69/69 70/70\nf 72/72 73/73 71/71\nf 71/71 73/73 74/74\nf 76/76 77/77 75/75\nf 75/75 77/77 78/78\nf 53/53 79/79 54/54\nf 54/54 79/79 80/80\nf 82/82 83/83 81/81\nf 81/81 83/83 84/84\nf 85/85 86/86 52/52\nf 52/52 86/86 53/53\nf 57/57 60/60 87/87\nf 87/87 60/60 88/88\nf 89/89 90/90 43/43\nf 43/43 90/90 44/44\nf 43/43 42/42 89/89\nf 89/89 42/42 91/91\nf 40/40 39/39 92/92\nf 92/92 39/39 93/93\nf 94/94 70/70 64/64\nf 64/64 70/70 69/69\nf 13/13 61/61 9/9\nf 9/9 61/61 4/4\nf 25/25 28/28 69/69\nf 69/69 28/28 95/95\nf 96/96 25/25 68/68\nf 68/68 25/25 69/69\nf 98/98 99/99 97/97\nf 97/97 99/99 100/100\nf 101/101 102/102 104/104\nf 104/104 102/102 103/103\nf 106/106 30/30 105/105\nf 105/105 30/30 29/29\nf 51/51 54/54 107/107\nf 107/107 54/54 108/108\nf 54/54 80/80 108/108\nf 108/108 80/80 109/109\nf 111/111 112/112 110/110\nf 110/110 112/112 113/113\nf 115/115 5/5 114/114\nf 114/114 5/5 116/116\nf 115/115 41/41 5/5\nf 5/5 41/41 6/6\nf 36/36 35/35 111/111\nf 111/111 35/35 117/117\nf 35/35 118/118 117/117\nf 117/117 118/118 119/119\nf 34/34 120/120 35/35\nf 35/35 120/120 118/118\nf 34/34 45/45 120/120\nf 120/120 45/45 121/121\nf 125/125 126/126 124/124\nf 124/124 126/126 127/127\nf 128/128 129/129 14/14\nf 14/14 129/129 130/130\nf 33/33 131/131 46/46\nf 46/46 131/131 132/132\nf 133/133 94/94 63/63\nf 63/63 94/94 64/64\nf 134/134 135/135 137/137\nf 137/137 135/135 136/136\nf 139/139 140/140 138/138\nf 138/138 140/140 141/141\nf 77/77 142/142 78/78\nf 78/78 142/142 143/143\nf 144/144 145/145 20/20\nf 20/20 145/145 146/146\nf 148/148 149/149 147/147\nf 147/147 149/149 72/72\nf 151/151 152/152 150/150\nf 150/150 152/152 153/153\nf 82/82 101/101 83/83\nf 83/83 101/101 104/104\nf 101/101 82/82 155/155\nf 155/155 82/82 154/154\nf 156/156 157/157 58/58\nf 58/58 157/157 59/59\nf 158/158 68/68 147/147\nf 147/147 68/68 67/67\nf 159/159 146/146 15/15\nf 15/15 146/146 16/16\nf 68/68 158/158 96/96\nf 96/96 158/158 160/160\nf 129/129 66/66 130/130\nf 130/130 66/66 65/65\nf 162/162 163/163 161/161\nf 161/161 163/163 164/164\nf 166/166 167/167 165/165\nf 165/165 167/167 168/168\nf 170/170 171/171 169/169\nf 169/169 171/171 172/172\nf 174/174 171/171 173/173\nf 173/173 171/171 170/170\nf 176/176 177/177 175/175\nf 175/175 177/177 178/178\nf 180/180 181/181 179/179\nf 179/179 181/181 24/24\nf 182/182 183/183 21/21\nf 21/21 183/183 184/184\nf 81/81 185/185 82/82\nf 82/82 185/185 154/154\nf 186/186 187/187 188/188\nf 188/188 187/187 144/144\nf 158/158 71/71 160/160\nf 160/160 71/71 189/189\nf 190/190 135/135 157/157\nf 157/157 135/135 59/59\nf 169/169 172/172 139/139\nf 139/139 172/172 140/140\nf 147/147 72/72 158/158\nf 158/158 72/72 71/71\nf 192/192 193/193 191/191\nf 191/191 193/193 194/194\nf 167/167 166/166 49/49\nf 49/49 166/166 195/195\nf 196/196 190/190 50/50\nf 50/50 190/190 157/157\nf 197/197 198/198 200/200\nf 200/200 198/198 199/199\nf 21/21 24/24 182/182\nf 182/182 24/24 181/181\nf 146/146 159/159 20/20\nf 20/20 159/159 17/17\nf 182/182 181/181 81/81\nf 81/81 181/181 185/185\nf 201/201 185/185 180/180\nf 180/180 185/185 181/181\nf 202/202 203/203 83/83\nf 83/83 203/203 84/84\nf 205/205 206/206 204/204\nf 204/204 206/206 85/85\nf 207/207 22/22 184/184\nf 184/184 22/22 21/21\nf 179/179 24/24 208/208\nf 208/208 24/24 23/23\nf 208/208 209/209 179/179\nf 179/179 209/209 205/205\nf 209/209 210/210 205/205\nf 205/205 210/210 206/206\nf 212/212 206/206 211/211\nf 211/211 206/206 210/210\nf 179/179 205/205 180/180\nf 180/180 205/205 204/204\nf 213/213 176/176 88/88\nf 88/88 176/176 214/214\nf 86/86 85/85 212/212\nf 212/212 85/85 206/206\nf 193/193 217/217 218/218\nf 217/217 219/219 218/218\nf 218/218 219/219 220/220\nf 219/219 18/18 220/220\nf 220/220 18/18 17/17\nf 19/19 221/221 20/20\nf 20/20 221/221 144/144\nf 222/222 223/223 225/225\nf 225/225 223/223 224/224\nf 175/175 226/226 176/176\nf 176/176 226/226 214/214\nf 227/227 87/87 214/214\nf 214/214 87/87 88/88\nf 220/220 17/17 228/228\nf 228/228 17/17 159/159\nf 134/134 213/213 60/60\nf 60/60 213/213 88/88\nf 138/138 229/229 151/151\nf 151/151 229/229 152/152\nf 193/193 192/192 210/210\nf 210/210 192/192 211/211\nf 230/230 194/194 218/218\nf 218/218 194/194 193/193\nf 231/231 232/232 228/228\nf 228/228 232/232 230/230\nf 233/233 234/234 91/91\nf 91/91 234/234 89/89\nf 235/235 236/236 93/93\nf 93/93 236/236 92/92\nf 230/230 232/232 234/234\nf 234/234 232/232 237/237\nf 47/47 238/238 48/48\nf 48/48 238/238 239/239\nf 240/240 191/191 235/235\nf 235/235 191/191 236/236\nf 5/5 8/8 116/116\nf 116/116 8/8 241/241\nf 242/242 243/243 189/189\nf 189/189 243/243 160/160\nf 244/244 245/245 98/98\nf 98/98 245/245 99/99\nf 156/156 246/246 47/47\nf 47/47 246/246 238/238\nf 76/76 247/247 77/77\nf 77/77 247/247 248/248\nf 250/250 125/125 249/249\nf 249/249 125/125 124/124\nf 252/252 253/253 251/251\nf 251/251 253/253 254/254\nf 111/111 110/110 36/36\nf 36/36 110/110 255/255\nf 256/256 33/33 255/255\nf 255/255 33/33 36/36\nf 257/257 258/258 154/154\nf 154/154 258/258 155/155\nf 63/63 169/169 133/133\nf 133/133 169/169 139/139\nf 66/66 170/170 63/63\nf 63/63 170/170 169/169\nf 173/173 170/170 129/129\nf 129/129 170/170 66/66\nf 129/129 128/128 173/173\nf 173/173 128/128 259/259\nf 128/128 260/260 259/259\nf 259/259 260/260 261/261\nf 213/213 134/134 262/262\nf 262/262 134/134 137/137\nf 263/263 264/264 186/186\nf 186/186 264/264 187/187\nf 112/112 265/265 113/113\nf 113/113 265/265 266/266\nf 11/11 267/267 268/268\nf 234/234 233/233 230/230\nf 230/230 233/233 194/194\nf 153/153 269/269 70/70\nf 70/70 269/269 67/67\nf 175/175 178/178 165/165\nf 165/165 178/178 141/141\nf 150/150 153/153 94/94\nf 94/94 153/153 70/70\nf 271/271 221/221 270/270\nf 270/270 221/221 207/207\nf 269/269 153/153 272/272\nf 272/272 153/153 152/152\nf 121/121 45/45 109/109\nf 109/109 45/45 108/108\nf 47/47 50/50 156/156\nf 156/156 50/50 157/157\nf 57/57 273/273 58/58\nf 58/58 273/273 274/274\nf 145/145 275/275 260/260\nf 260/260 275/275 261/261\nf 276/276 149/149 143/143\nf 143/143 149/149 78/78\nf 148/148 269/269 75/75\nf 75/75 269/269 272/272\nf 207/207 277/277 270/270\nf 270/270 277/277 278/278\nf 195/195 196/196 49/49\nf 49/49 196/196 50/50\nf 59/59 135/135 60/60\nf 60/60 135/135 134/134\nf 83/83 104/104 202/202\nf 202/202 104/104 279/279\nf 149/149 148/148 78/78\nf 78/78 148/148 75/75\nf 247/247 278/278 248/248\nf 248/248 278/278 277/277\nf 183/183 182/182 84/84\nf 84/84 182/182 81/81\nf 48/48 239/239 281/281\nf 281/281 239/239 282/282\nf 227/227 223/223 283/283\nf 283/283 223/223 222/222\nf 223/223 226/226 224/224\nf 224/224 226/226 168/168\nf 281/281 282/282 224/224\nf 224/224 282/282 225/225\nf 165/165 168/168 175/175\nf 175/175 168/168 226/226\nf 264/264 263/263 138/138\nf 138/138 263/263 229/229\nf 14/14 16/16 128/128\nf 128/128 16/16 260/260\nf 220/220 228/228 218/218\nf 218/218 228/228 230/230\nf 87/87 227/227 284/284\nf 284/284 227/227 283/283\nf 260/260 16/16 145/145\nf 145/145 16/16 146/146\nf 244/244 98/98 258/258\nf 258/258 98/98 155/155\nf 58/58 274/274 156/156\nf 156/156 274/274 246/246\nf 107/107 285/285 51/51\nf 51/51 285/285 55/55\nf 243/243 286/286 160/160\nf 160/160 286/286 96/96\nf 234/234 237/237 89/89\nf 89/89 237/237 90/90\nf 46/46 132/132 107/107\nf 107/107 132/132 285/285\nf 253/253 252/252 249/249\nf 249/249 252/252 250/250\nf 286/286 26/26 96/96\nf 96/96 26/26 25/25\nf 106/106 105/105 287/287\nf 287/287 105/105 288/288\nf 272/272 152/152 289/289\nf 289/289 152/152 229/229\nf 138/138 151/151 139/139\nf 139/139 151/151 133/133\nf 183/183 290/290 161/161\nf 161/161 290/290 162/162\nf 291/291 292/292 277/277\nf 277/277 292/292 293/293\nf 151/151 150/150 133/133\nf 133/133 150/150 94/94\nf 262/262 275/275 177/177\nf 177/177 275/275 187/187\nf 144/144 187/187 145/145\nf 145/145 187/187 275/275\nf 273/273 57/57 284/284\nf 284/284 57/57 87/87\nf 174/174 136/136 190/190\nf 190/190 136/136 135/135\nf 111/111 117/117 112/112\nf 112/112 117/117 38/38\nf 117/117 119/119 38/38\nf 38/38 119/119 123/123\nf 112/112 38/38 265/265\nf 265/265 38/38 37/37\nf 287/287 288/288 254/254\nf 254/254 288/288 251/251\nf 228/228 159/159 231/231\nf 231/231 159/159 15/15\nf 223/223 227/227 226/226\nf 226/226 227/227 214/214\nf 147/147 67/67 148/148\nf 148/148 67/67 269/269\nf 224/224 168/168 281/281\nf 281/281 168/168 167/167\nf 199/199 163/163 200/200\nf 200/200 163/163 162/162\nf 45/45 46/46 108/108\nf 108/108 46/46 107/107\nf 281/281 167/167 48/48\nf 48/48 167/167 49/49\nf 161/161 164/164 291/291\nf 291/291 164/164 292/292\nf 38/38 123/123 39/39\nf 293/293 294/294 277/277\nf 277/277 294/294 248/248\nf 284/284 283/283 296/296\nf 296/296 283/283 295/295\nf 284/284 296/296 273/273\nf 273/273 296/296 297/297\nf 274/274 273/273 298/298\nf 298/298 273/273 297/297\nf 246/246 274/274 299/299\nf 299/299 274/274 298/298\nf 283/283 222/222 295/295\nf 295/295 222/222 300/300\nf 222/222 225/225 300/300\nf 300/300 225/225 301/301\nf 301/301 225/225 302/302\nf 302/302 225/225 282/282\nf 303/303 304/304 300/300\nf 300/300 304/304 295/295\nf 306/306 307/307 305/305\nf 305/305 307/307 308/308\nf 309/309 310/310 308/308\nf 308/308 310/310 305/305\nf 308/308 307/307 312/312\nf 312/312 307/307 311/311\nf 313/313 314/314 312/312\nf 312/312 314/314 315/315\nf 311/311 316/316 312/312\nf 312/312 316/316 313/313\nf 317/317 304/304 311/311\nf 311/311 304/304 316/316\nf 315/315 309/309 312/312\nf 312/312 309/309 308/308\nf 318/318 319/319 321/321\nf 321/321 319/319 320/320\nf 319/319 318/318 309/309\nf 309/309 318/318 310/310\nf 311/311 307/307 317/317\nf 317/317 307/307 306/306\nf 306/306 297/297 317/317\nf 317/317 297/297 296/296\nf 306/306 322/322 297/297\nf 297/297 322/322 298/298\nf 305/305 323/323 306/306\nf 306/306 323/323 322/322\nf 323/323 305/305 324/324\nf 324/324 305/305 310/310\nf 304/304 317/317 295/295\nf 295/295 317/317 296/296\nf 318/318 325/325 310/310\nf 310/310 325/325 324/324\nf 323/323 324/324 326/326\nf 326/326 324/324 327/327\nf 324/324 325/325 327/327\nf 327/327 325/325 328/328\nf 326/326 327/327 329/329\nf 329/329 327/327 330/330\nf 329/329 331/331 326/326\nf 326/326 331/331 332/332\nf 333/333 330/330 328/328\nf 328/328 330/330 327/327\nf 323/323 326/326 322/322\nf 322/322 326/326 332/332\nf 322/322 332/332 298/298\nf 298/298 332/332 299/299\nf 325/325 318/318 334/334\nf 334/334 318/318 321/321\nf 313/313 316/316 335/335\nf 335/335 316/316 304/304\nf 335/335 336/336 313/313\nf 313/313 336/336 314/314\nf 304/304 303/303 335/335\nf 335/335 303/303 337/337\nf 336/336 335/335 338/338\nf 338/338 335/335 337/337\nf 336/336 339/339 314/314\nf 314/314 339/339 340/340\nf 246/246 299/299 238/238\nf 238/238 299/299 341/341\nf 239/239 342/342 282/282\nf 282/282 342/342 302/302\nf 342/342 239/239 341/341\nf 341/341 239/239 238/238\nf 341/341 331/331 342/342\nf 342/342 331/331 343/343\nf 331/331 329/329 343/343\nf 343/343 329/329 344/344\nf 342/342 343/343 302/302\nf 302/302 343/343 345/345\nf 341/341 299/299 331/331\nf 331/331 299/299 332/332\nf 346/346 301/301 345/345\nf 345/345 301/301 302/302\nf 346/346 303/303 301/301\nf 301/301 303/303 300/300\nf 347/347 337/337 346/346\nf 346/346 337/337 303/303\nf 347/347 346/346 348/348\nf 348/348 346/346 345/345\nf 349/349 347/347 350/350\nf 350/350 347/347 348/348\nf 343/343 344/344 345/345\nf 345/345 344/344 348/348\nf 344/344 351/351 348/348\nf 348/348 351/351 350/350\nf 337/337 347/347 338/338\nf 338/338 347/347 349/349\nf 353/353 338/338 352/352\nf 352/352 338/338 349/349\nf 352/352 355/355 354/354\nf 354/354 355/355 356/356\nf 352/352 349/349 355/355\nf 355/355 349/349 350/350\nf 352/352 354/354 353/353\nf 353/353 354/354 334/334\nf 353/353 334/334 339/339\nf 339/339 334/334 321/321\nf 339/339 336/336 353/353\nf 353/353 336/336 338/338\nf 329/329 330/330 344/344\nf 344/344 330/330 351/351\nf 356/356 351/351 333/333\nf 333/333 351/351 330/330\nf 356/356 355/355 351/351\nf 351/351 355/355 350/350\nf 354/354 356/356 328/328\nf 328/328 356/356 333/333\nf 328/328 325/325 354/354\nf 354/354 325/325 334/334\nf 309/309 315/315 319/319\nf 319/319 315/315 320/320\nf 320/320 340/340 321/321\nf 321/321 340/340 339/339\nf 320/320 315/315 340/340\nf 340/340 315/315 314/314\nf 137/137 136/136 261/261\nf 261/261 136/136 259/259\nf 140/140 166/166 141/141\nf 141/141 166/166 165/165\nf 172/172 171/171 195/195\nf 195/195 171/171 196/196\nf 171/171 174/174 196/196\nf 196/196 174/174 190/190\nf 178/178 177/177 264/264\nf 264/264 177/177 187/187\nf 172/172 195/195 140/140\nf 140/140 195/195 166/166\nf 137/137 261/261 262/262\nf 262/262 261/261 275/275\nf 178/178 264/264 141/141\nf 141/141 264/264 138/138\nf 262/262 177/177 213/213\nf 213/213 177/177 176/176\nf 136/136 174/174 259/259\nf 259/259 174/174 173/173\nf 358/358 359/359 357/357\nf 357/357 359/359 360/360\nf 362/362 363/363 361/361\nf 361/361 363/363 364/364\nf 361/361 364/364 365/365\nf 365/365 364/364 366/366\nf 367/367 368/368 370/370\nf 370/370 368/368 369/369\nf 76/76 371/371 247/247\nf 247/247 371/371 372/372\nf 373/373 374/374 362/362\nf 362/362 374/374 363/363\nf 376/376 377/377 375/375\nf 375/375 377/377 378/378\nf 289/289 229/229 380/380\nf 380/380 229/229 379/379\nf 381/381 379/379 263/263\nf 263/263 379/379 229/229\nf 383/383 384/384 382/382\nf 382/382 384/384 385/385\nf 386/386 368/368 374/374\nf 374/374 368/368 363/363\nf 76/76 289/289 371/371\nf 371/371 289/289 380/380\nf 377/377 376/376 359/359\nf 359/359 376/376 387/387\nf 388/388 386/386 360/360\nf 360/360 386/386 374/374\nf 389/389 383/383 366/366\nf 366/366 383/383 390/390\nf 392/392 393/393 391/391\nf 391/391 393/393 394/394\nf 383/383 382/382 390/390\nf 390/390 382/382 395/395\nf 396/396 365/365 390/390\nf 390/390 365/365 366/366\nf 389/389 366/366 367/367\nf 367/367 366/366 364/364\nf 357/357 397/397 358/358\nf 358/358 397/397 398/398\nf 373/373 399/399 357/357\nf 357/357 399/399 397/397\nf 389/389 367/367 400/400\nf 400/400 367/367 370/370\nf 382/382 385/385 375/375\nf 375/375 385/385 372/372\nf 357/357 360/360 373/373\nf 373/373 360/360 374/374\nf 361/361 401/401 362/362\nf 362/362 401/401 402/402\nf 387/387 388/388 359/359\nf 359/359 388/388 360/360\nf 363/363 368/368 364/364\nf 364/364 368/368 367/367\nf 358/358 398/398 403/403\nf 403/403 398/398 404/404\nf 396/396 392/392 405/405\nf 405/405 392/392 391/391\nf 392/392 395/395 393/393\nf 393/393 395/395 378/378\nf 393/393 403/403 394/394\nf 394/394 403/403 404/404\nf 375/375 378/378 382/382\nf 382/382 378/378 395/395\nf 365/365 396/396 406/406\nf 406/406 396/396 405/405\nf 362/362 402/402 373/373\nf 373/373 402/402 399/399\nf 271/271 270/270 400/400\nf 400/400 270/270 384/384\nf 401/401 361/361 406/406\nf 406/406 361/361 365/365\nf 381/381 369/369 386/386\nf 386/386 369/369 368/368\nf 392/392 396/396 395/395\nf 395/395 396/396 390/390\nf 393/393 378/378 403/403\nf 403/403 378/378 377/377\nf 403/403 377/377 358/358\nf 358/358 377/377 359/359\nf 406/406 405/405 408/408\nf 408/408 405/405 407/407\nf 406/406 408/408 401/401\nf 401/401 408/408 409/409\nf 401/401 409/409 402/402\nf 402/402 409/409 410/410\nf 402/402 410/410 399/399\nf 399/399 410/410 411/411\nf 405/405 391/391 407/407\nf 407/407 391/391 412/412\nf 391/391 394/394 412/412\nf 412/412 394/394 413/413\nf 394/394 404/404 413/413\nf 413/413 404/404 414/414\nf 415/415 416/416 412/412\nf 412/412 416/416 407/407\nf 418/418 419/419 417/417\nf 417/417 419/419 420/420\nf 421/421 422/422 418/418\nf 418/418 422/422 419/419\nf 419/419 422/422 424/424\nf 424/424 422/422 423/423\nf 425/425 426/426 424/424\nf 424/424 426/426 427/427\nf 423/423 428/428 424/424\nf 424/424 428/428 425/425\nf 427/427 420/420 424/424\nf 424/424 420/420 419/419\nf 420/420 429/429 417/417\nf 417/417 429/429 430/430\nf 423/423 422/422 431/431\nf 431/431 422/422 421/421\nf 421/421 409/409 431/431\nf 431/431 409/409 408/408\nf 421/421 432/432 409/409\nf 409/409 432/432 410/410\nf 418/418 433/433 421/421\nf 421/421 433/433 432/432\nf 434/434 433/433 417/417\nf 417/417 433/433 418/418\nf 431/431 416/416 423/423\nf 423/423 416/416 428/428\nf 416/416 431/431 407/407\nf 407/407 431/431 408/408\nf 435/435 434/434 430/430\nf 430/430 434/434 417/417\nf 433/433 434/434 436/436\nf 436/436 434/434 437/437\nf 434/434 435/435 437/437\nf 437/437 435/435 438/438\nf 436/436 437/437 439/439\nf 439/439 437/437 440/440\nf 439/439 441/441 436/436\nf 436/436 441/441 442/442\nf 438/438 443/443 437/437\nf 437/437 443/443 440/440\nf 436/436 442/442 433/433\nf 433/433 442/442 432/432\nf 432/432 442/442 410/410\nf 410/410 442/442 411/411\nf 435/435 430/430 445/445\nf 445/445 430/430 444/444\nf 430/430 429/429 444/444\nf 444/444 429/429 446/446\nf 425/425 428/428 447/447\nf 447/447 428/428 416/416\nf 426/426 425/425 448/448\nf 448/448 425/425 447/447\nf 416/416 415/415 447/447\nf 447/447 415/415 449/449\nf 447/447 449/449 448/448\nf 448/448 449/449 450/450\nf 451/451 452/452 448/448\nf 448/448 452/452 426/426\nf 446/446 452/452 444/444\nf 444/444 452/452 451/451\nf 399/399 411/411 397/397\nf 397/397 411/411 453/453\nf 398/398 454/454 404/404\nf 404/404 454/454 414/414\nf 454/454 398/398 453/453\nf 453/453 398/398 397/397\nf 453/453 441/441 454/454\nf 454/454 441/441 455/455\nf 441/441 439/439 455/455\nf 455/455 439/439 456/456\nf 454/454 455/455 414/414\nf 414/414 455/455 457/457\nf 441/441 453/453 442/442\nf 442/442 453/453 411/411\nf 458/458 413/413 457/457\nf 457/457 413/413 414/414\nf 458/458 415/415 413/413\nf 413/413 415/415 412/412\nf 458/458 459/459 415/415\nf 415/415 459/459 449/449\nf 459/459 458/458 460/460\nf 460/460 458/458 457/457\nf 461/461 459/459 462/462\nf 462/462 459/459 460/460\nf 455/455 456/456 457/457\nf 457/457 456/456 460/460\nf 456/456 463/463 460/460\nf 460/460 463/463 462/462\nf 459/459 461/461 449/449\nf 449/449 461/461 450/450\nf 465/465 450/450 464/464\nf 464/464 450/450 461/461\nf 464/464 467/467 466/466\nf 466/466 467/467 468/468\nf 464/464 461/461 467/467\nf 467/467 461/461 462/462\nf 464/464 466/466 465/465\nf 465/465 466/466 445/445\nf 465/465 445/445 451/451\nf 451/451 445/445 444/444\nf 451/451 448/448 465/465\nf 465/465 448/448 450/450\nf 439/439 440/440 456/456\nf 456/456 440/440 463/463\nf 468/468 463/463 443/443\nf 443/443 463/463 440/440\nf 468/468 467/467 463/463\nf 463/463 467/467 462/462\nf 466/466 468/468 438/438\nf 438/438 468/468 443/443\nf 438/438 435/435 466/466\nf 466/466 435/435 445/445\nf 420/420 427/427 429/429\nf 429/429 427/427 446/446\nf 446/446 427/427 452/452\nf 452/452 427/427 426/426\nf 370/370 369/369 188/188\nf 188/188 369/369 186/186\nf 371/371 376/376 372/372\nf 372/372 376/376 375/375\nf 380/380 379/379 387/387\nf 387/387 379/379 388/388\nf 379/379 381/381 388/388\nf 388/388 381/381 386/386\nf 385/385 384/384 278/278\nf 278/278 384/384 270/270\nf 380/380 387/387 371/371\nf 371/371 387/387 376/376\nf 400/400 370/370 271/271\nf 271/271 370/370 188/188\nf 385/385 278/278 372/372\nf 372/372 278/278 247/247\nf 400/400 384/384 389/389\nf 389/389 384/384 383/383\nf 369/369 381/381 186/186\nf 186/186 381/381 263/263\nf 221/221 271/271 144/144\nf 144/144 271/271 188/188\nf 221/221 19/19 207/207\nf 207/207 19/19 22/22\nf 19/19 18/18 22/22\nf 22/22 18/18 23/23\nf 18/18 219/219 23/23\nf 23/23 219/219 208/208\nf 469/469 470/470 472/472\nf 472/472 470/470 471/471\nf 474/474 475/475 473/473\nf 473/473 475/475 476/476\nf 473/473 476/476 477/477\nf 477/477 476/476 478/478\nf 480/480 481/481 479/479\nf 479/479 481/481 482/482\nf 483/483 484/484 474/474\nf 474/474 484/484 475/475\nf 486/486 487/487 485/485\nf 485/485 487/487 488/488\nf 489/489 490/490 77/77\nf 77/77 490/490 142/142\nf 491/491 489/489 248/248\nf 248/248 489/489 77/77\nf 493/493 494/494 492/492\nf 492/492 494/494 495/495\nf 496/496 480/480 484/484\nf 484/484 480/480 475/475\nf 486/486 497/497 487/487\nf 487/487 497/497 471/471\nf 498/498 496/496 472/472\nf 472/472 496/496 484/484\nf 499/499 493/493 478/478\nf 478/478 493/493 500/500\nf 501/501 502/502 504/504\nf 504/504 502/502 503/503\nf 493/493 492/492 500/500\nf 500/500 492/492 505/505\nf 506/506 477/477 500/500\nf 500/500 477/477 478/478\nf 499/499 478/478 479/479\nf 479/479 478/478 476/476\nf 469/469 507/507 470/470\nf 470/470 507/507 508/508\nf 483/483 509/509 469/469\nf 469/469 509/509 507/507\nf 479/479 482/482 499/499\nf 499/499 482/482 510/510\nf 492/492 495/495 485/485\nf 485/485 495/495 511/511\nf 483/483 469/469 484/484\nf 484/484 469/469 472/472\nf 473/473 512/512 474/474\nf 474/474 512/512 513/513\nf 497/497 498/498 471/471\nf 471/471 498/498 472/472\nf 475/475 480/480 476/476\nf 476/476 480/480 479/479\nf 470/470 508/508 514/514\nf 514/514 508/508 515/515\nf 506/506 502/502 516/516\nf 516/516 502/502 501/501\nf 503/503 502/502 488/488\nf 488/488 502/502 505/505\nf 514/514 515/515 503/503\nf 503/503 515/515 504/504\nf 485/485 488/488 492/492\nf 492/492 488/488 505/505\nf 477/477 506/506 517/517\nf 517/517 506/506 516/516\nf 474/474 513/513 483/483\nf 483/483 513/513 509/509\nf 510/510 292/292 494/494\nf 494/494 292/292 164/164\nf 512/512 473/473 517/517\nf 517/517 473/473 477/477\nf 491/491 481/481 496/496\nf 496/496 481/481 480/480\nf 502/502 506/506 505/505\nf 505/505 506/506 500/500\nf 503/503 488/488 514/514\nf 514/514 488/488 487/487\nf 514/514 487/487 470/470\nf 470/470 487/487 471/471\nf 517/517 516/516 519/519\nf 519/519 516/516 518/518\nf 517/517 519/519 512/512\nf 512/512 519/519 520/520\nf 512/512 520/520 513/513\nf 513/513 520/520 521/521\nf 513/513 521/521 509/509\nf 509/509 521/521 522/522\nf 516/516 501/501 518/518\nf 518/518 501/501 523/523\nf 501/501 504/504 523/523\nf 523/523 504/504 524/524\nf 524/524 504/504 525/525\nf 525/525 504/504 515/515\nf 526/526 527/527 523/523\nf 523/523 527/527 518/518\nf 529/529 530/530 528/528\nf 528/528 530/530 531/531\nf 532/532 533/533 529/529\nf 529/529 533/533 530/530\nf 530/530 533/533 535/535\nf 535/535 533/533 534/534\nf 536/536 537/537 535/535\nf 535/535 537/537 538/538\nf 534/534 539/539 535/535\nf 535/535 539/539 536/536\nf 538/538 531/531 535/535\nf 535/535 531/531 530/530\nf 540/540 541/541 531/531\nf 531/531 541/541 528/528\nf 533/533 532/532 534/534\nf 534/534 532/532 542/542\nf 532/532 520/520 542/542\nf 542/542 520/520 519/519\nf 532/532 543/543 520/520\nf 520/520 543/543 521/521\nf 529/529 544/544 532/532\nf 532/532 544/544 543/543\nf 544/544 529/529 545/545\nf 545/545 529/529 528/528\nf 534/534 542/542 539/539\nf 539/539 542/542 527/527\nf 527/527 542/542 518/518\nf 518/518 542/542 519/519\nf 541/541 546/546 528/528\nf 528/528 546/546 545/545\nf 547/547 544/544 548/548\nf 548/548 544/544 545/545\nf 546/546 549/549 545/545\nf 545/545 549/549 548/548\nf 550/550 547/547 551/551\nf 551/551 547/547 548/548\nf 547/547 550/550 553/553\nf 553/553 550/550 552/552\nf 549/549 554/554 548/548\nf 548/548 554/554 551/551\nf 544/544 547/547 543/543\nf 543/543 547/547 553/553\nf 543/543 553/553 521/521\nf 521/521 553/553 522/522\nf 546/546 541/541 556/556\nf 556/556 541/541 555/555\nf 541/541 540/540 555/555\nf 555/555 540/540 557/557\nf 536/536 539/539 558/558\nf 558/558 539/539 527/527\nf 537/537 536/536 559/559\nf 559/559 536/536 558/558\nf 527/527 526/526 558/558\nf 558/558 526/526 560/560\nf 559/559 558/558 561/561\nf 561/561 558/558 560/560\nf 559/559 562/562 537/537\nf 537/537 562/562 563/563\nf 557/557 563/563 555/555\nf 555/555 563/563 562/562\nf 509/509 522/522 507/507\nf 507/507 522/522 564/564\nf 508/508 565/565 515/515\nf 515/515 565/565 525/525\nf 565/565 508/508 564/564\nf 564/564 508/508 507/507\nf 564/564 552/552 565/565\nf 565/565 552/552 566/566\nf 552/552 550/550 566/566\nf 566/566 550/550 567/567\nf 565/565 566/566 525/525\nf 525/525 566/566 568/568\nf 564/564 522/522 552/552\nf 552/552 522/522 553/553\nf 569/569 524/524 568/568\nf 568/568 524/524 525/525\nf 569/569 526/526 524/524\nf 524/524 526/526 523/523\nf 570/570 560/560 569/569\nf 569/569 560/560 526/526\nf 570/570 569/569 571/571\nf 571/571 569/569 568/568\nf 572/572 570/570 573/573\nf 573/573 570/570 571/571\nf 566/566 567/567 568/568\nf 568/568 567/567 571/571\nf 567/567 574/574 571/571\nf 571/571 574/574 573/573\nf 570/570 572/572 560/560\nf 560/560 572/572 561/561\nf 576/576 561/561 575/575\nf 575/575 561/561 572/572\nf 575/575 578/578 577/577\nf 577/577 578/578 579/579\nf 575/575 572/572 578/578\nf 578/578 572/572 573/573\nf 575/575 577/577 576/576\nf 576/576 577/577 556/556\nf 576/576 556/556 562/562\nf 562/562 556/556 555/555\nf 562/562 559/559 576/576\nf 576/576 559/559 561/561\nf 550/550 551/551 567/567\nf 567/567 551/551 574/574\nf 579/579 574/574 554/554\nf 554/554 574/574 551/551\nf 579/579 578/578 574/574\nf 574/574 578/578 573/573\nf 579/579 554/554 577/577\nf 577/577 554/554 549/549\nf 549/549 546/546 577/577\nf 577/577 546/546 556/556\nf 531/531 538/538 540/540\nf 540/540 538/538 557/557\nf 557/557 538/538 563/563\nf 563/563 538/538 537/537\nf 481/481 294/294 482/482\nf 482/482 294/294 293/293\nf 580/580 486/486 511/511\nf 511/511 486/486 485/485\nf 490/490 489/489 497/497\nf 497/497 489/489 498/498\nf 489/489 491/491 498/498\nf 498/498 491/491 496/496\nf 163/163 495/495 164/164\nf 164/164 495/495 494/494\nf 580/580 490/490 486/486\nf 486/486 490/490 497/497\nf 482/482 293/293 510/510\nf 510/510 293/293 292/292\nf 495/495 163/163 511/511\nf 511/511 163/163 199/199\nf 494/494 493/493 510/510\nf 510/510 493/493 499/499\nf 481/481 491/491 294/294\nf 294/294 491/491 248/248\nf 582/582 583/583 581/581\nf 581/581 583/583 584/584\nf 586/586 587/587 585/585\nf 585/585 587/587 588/588\nf 585/585 588/588 589/589\nf 589/589 588/588 590/590\nf 592/592 593/593 591/591\nf 591/591 593/593 594/594\nf 595/595 596/596 598/598\nf 598/598 596/596 597/597\nf 599/599 600/600 586/586\nf 586/586 600/600 587/587\nf 601/601 602/602 604/604\nf 604/604 602/602 603/603\nf 605/605 606/606 608/608\nf 608/608 606/606 607/607\nf 609/609 607/607 200/200\nf 200/200 607/607 606/606\nf 610/610 611/611 613/613\nf 613/613 611/611 612/612\nf 600/600 614/614 587/587\nf 587/587 614/614 592/592\nf 596/596 605/605 597/597\nf 597/597 605/605 608/608\nf 602/602 615/615 603/603\nf 603/603 615/615 583/583\nf 616/616 614/614 584/584\nf 584/584 614/614 600/600\nf 611/611 618/618 617/617\nf 617/617 618/618 590/590\nf 620/620 621/621 619/619\nf 619/619 621/621 622/622\nf 610/610 623/623 611/611\nf 611/611 623/623 618/618\nf 624/624 589/589 618/618\nf 618/618 589/589 590/590\nf 617/617 590/590 591/591\nf 591/591 590/590 588/588\nf 581/581 625/625 582/582\nf 582/582 625/625 626/626\nf 599/599 627/627 581/581\nf 581/581 627/627 625/625\nf 591/591 594/594 617/617\nf 617/617 594/594 628/628\nf 601/601 610/610 598/598\nf 598/598 610/610 613/613\nf 581/581 584/584 599/599\nf 599/599 584/584 600/600\nf 585/585 629/629 586/586\nf 586/586 629/629 630/630\nf 615/615 616/616 583/583\nf 583/583 616/616 584/584\nf 587/587 592/592 588/588\nf 588/588 592/592 591/591\nf 582/582 626/626 631/631\nf 631/631 626/626 632/632\nf 624/624 620/620 633/633\nf 633/633 620/620 619/619\nf 621/621 620/620 604/604\nf 604/604 620/620 623/623\nf 631/631 632/632 621/621\nf 621/621 632/632 622/622\nf 610/610 601/601 623/623\nf 623/623 601/601 604/604\nf 589/589 624/624 634/634\nf 634/634 624/624 633/633\nf 586/586 630/630 599/599\nf 599/599 630/630 627/627\nf 203/203 202/202 628/628\nf 628/628 202/202 612/612\nf 629/629 585/585 634/634\nf 634/634 585/585 589/589\nf 609/609 593/593 614/614\nf 614/614 593/593 592/592\nf 620/620 624/624 623/623\nf 623/623 624/624 618/618\nf 631/631 621/621 603/603\nf 603/603 621/621 604/604\nf 582/582 631/631 583/583\nf 583/583 631/631 603/603\nf 634/634 633/633 636/636\nf 636/636 633/633 635/635\nf 634/634 636/636 629/629\nf 629/629 636/636 637/637\nf 629/629 637/637 630/630\nf 630/630 637/637 638/638\nf 630/630 638/638 627/627\nf 627/627 638/638 639/639\nf 633/633 619/619 635/635\nf 635/635 619/619 640/640\nf 619/619 622/622 640/640\nf 640/640 622/622 641/641\nf 641/641 622/622 642/642\nf 642/642 622/622 632/632\nf 644/644 635/635 643/643\nf 643/643 635/635 640/640\nf 645/645 646/646 648/648\nf 648/648 646/646 647/647\nf 649/649 650/650 646/646\nf 646/646 650/650 647/647\nf 647/647 650/650 652/652\nf 652/652 650/650 651/651\nf 652/652 653/653 655/655\nf 655/655 653/653 654/654\nf 651/651 656/656 652/652\nf 652/652 656/656 653/653\nf 655/655 648/648 652/652\nf 652/652 648/648 647/647\nf 648/648 657/657 645/645\nf 645/645 657/657 658/658\nf 650/650 649/649 651/651\nf 651/651 649/649 659/659\nf 649/649 637/637 659/659\nf 659/659 637/637 636/636\nf 649/649 660/660 637/637\nf 637/637 660/660 638/638\nf 661/661 660/660 646/646\nf 646/646 660/660 649/649\nf 662/662 661/661 645/645\nf 645/645 661/661 646/646\nf 651/651 659/659 656/656\nf 656/656 659/659 644/644\nf 644/644 659/659 635/635\nf 635/635 659/659 636/636\nf 663/663 662/662 658/658\nf 658/658 662/662 645/645\nf 664/664 661/661 665/665\nf 665/665 661/661 662/662\nf 663/663 666/666 662/662\nf 662/662 666/666 665/665\nf 667/667 664/664 668/668\nf 668/668 664/664 665/665\nf 664/664 667/667 670/670\nf 670/670 667/667 669/669\nf 666/666 671/671 665/665\nf 665/665 671/671 668/668\nf 661/661 664/664 660/660\nf 660/660 664/664 670/670\nf 660/660 670/670 638/638\nf 638/638 670/670 639/639\nf 658/658 672/672 663/663\nf 663/663 672/672 673/673\nf 658/658 657/657 672/672\nf 672/672 657/657 674/674\nf 653/653 656/656 675/675\nf 675/675 656/656 644/644\nf 653/653 675/675 654/654\nf 654/654 675/675 676/676\nf 675/675 644/644 677/677\nf 677/677 644/644 643/643\nf 676/676 675/675 678/678\nf 678/678 675/675 677/677\nf 679/679 680/680 676/676\nf 676/676 680/680 654/654\nf 674/674 680/680 672/672\nf 672/672 680/680 679/679\nf 627/627 639/639 625/625\nf 625/625 639/639 681/681\nf 626/626 682/682 632/632\nf 632/632 682/682 642/642\nf 681/681 682/682 625/625\nf 625/625 682/682 626/626\nf 669/669 683/683 681/681\nf 681/681 683/683 682/682\nf 667/667 684/684 669/669\nf 669/669 684/684 683/683\nf 682/682 683/683 642/642\nf 642/642 683/683 685/685\nf 681/681 639/639 669/669\nf 669/669 639/639 670/670\nf 686/686 641/641 685/685\nf 685/685 641/641 642/642\nf 641/641 686/686 640/640\nf 640/640 686/686 643/643\nf 686/686 687/687 643/643\nf 643/643 687/687 677/677\nf 687/687 686/686 688/688\nf 688/688 686/686 685/685\nf 689/689 687/687 690/690\nf 690/690 687/687 688/688\nf 683/683 684/684 685/685\nf 685/685 684/684 688/688\nf 684/684 691/691 688/688\nf 688/688 691/691 690/690\nf 687/687 689/689 677/677\nf 677/677 689/689 678/678\nf 693/693 678/678 692/692\nf 692/692 678/678 689/689\nf 692/692 695/695 694/694\nf 694/694 695/695 696/696\nf 692/692 689/689 695/695\nf 695/695 689/689 690/690\nf 692/692 694/694 693/693\nf 693/693 694/694 673/673\nf 693/693 673/673 679/679\nf 679/679 673/673 672/672\nf 679/679 676/676 693/693\nf 693/693 676/676 678/678\nf 684/684 667/667 691/691\nf 691/691 667/667 668/668\nf 696/696 691/691 671/671\nf 671/671 691/691 668/668\nf 696/696 695/695 691/691\nf 691/691 695/695 690/690\nf 694/694 696/696 666/666\nf 666/666 696/696 671/671\nf 666/666 663/663 694/694\nf 694/694 663/663 673/673\nf 648/648 655/655 657/657\nf 657/657 655/655 674/674\nf 674/674 655/655 680/680\nf 680/680 655/655 654/654\nf 593/593 162/162 594/594\nf 594/594 162/162 290/290\nf 598/598 597/597 601/601\nf 601/601 597/597 602/602\nf 615/615 608/608 616/616\nf 616/616 608/608 607/607\nf 607/607 609/609 616/616\nf 616/616 609/609 614/614\nf 697/697 595/595 613/613\nf 613/613 595/595 598/598\nf 597/597 608/608 602/602\nf 602/602 608/608 615/615\nf 594/594 290/290 628/628\nf 628/628 290/290 203/203\nf 628/628 612/612 617/617\nf 617/617 612/612 611/611\nf 593/593 609/609 162/162\nf 162/162 609/609 200/200\nf 105/105 29/29 698/698\nf 698/698 29/29 699/699\nf 268/268 126/126 701/701\nf 701/701 126/126 700/700\nf 126/126 125/125 700/700\nf 700/700 125/125 702/702\nf 252/252 703/703 250/250\nf 250/250 703/703 704/704\nf 32/32 268/268 705/705\nf 705/705 268/268 701/701\nf 288/288 105/105 706/706\nf 706/706 105/105 698/698\nf 29/29 32/32 699/699\nf 699/699 32/32 705/705\nf 707/707 251/251 706/706\nf 706/706 251/251 288/288\nf 250/250 704/704 125/125\nf 125/125 704/704 702/702\nf 703/703 252/252 707/707\nf 707/707 252/252 251/251\nf 706/706 708/708 707/707\nf 707/707 708/708 709/709\nf 707/707 709/709 703/703\nf 703/703 709/709 710/710\nf 703/703 710/710 704/704\nf 704/704 710/710 711/711\nf 704/704 711/711 702/702\nf 702/702 711/711 712/712\nf 706/706 698/698 708/708\nf 708/708 698/698 713/713\nf 698/698 699/699 713/713\nf 713/713 699/699 714/714\nf 699/699 705/705 714/714\nf 714/714 705/705 715/715\nf 717/717 708/708 716/716\nf 716/716 708/708 713/713\nf 719/719 720/720 718/718\nf 718/718 720/720 721/721\nf 722/722 723/723 719/719\nf 719/719 723/723 720/720\nf 724/724 725/725 723/723\nf 723/723 725/725 720/720\nf 724/724 726/726 725/725\nf 725/725 726/726 727/727\nf 728/728 721/721 725/725\nf 725/725 721/721 720/720\nf 721/721 728/728 729/729\nf 729/729 728/728 730/730\nf 729/729 731/731 721/721\nf 721/721 731/731 718/718\nf 724/724 723/723 732/732\nf 732/732 723/723 722/722\nf 722/722 710/710 732/732\nf 732/732 710/710 709/709\nf 722/722 733/733 710/710\nf 710/710 733/733 711/711\nf 719/719 734/734 722/722\nf 722/722 734/734 733/733\nf 734/734 719/719 735/735\nf 735/735 719/719 718/718\nf 724/724 732/732 726/726\nf 726/726 732/732 717/717\nf 717/717 732/732 708/708\nf 708/708 732/732 709/709\nf 731/731 736/736 718/718\nf 718/718 736/736 735/735\nf 737/737 734/734 738/738\nf 738/738 734/734 735/735\nf 736/736 739/739 735/735\nf 735/735 739/739 738/738\nf 740/740 737/737 741/741\nf 741/741 737/737 738/738\nf 737/737 740/740 743/743\nf 743/743 740/740 742/742\nf 739/739 744/744 738/738\nf 738/738 744/744 741/741\nf 734/734 737/737 733/733\nf 733/733 737/737 743/743\nf 733/733 743/743 711/711\nf 711/711 743/743 712/712\nf 736/736 731/731 746/746\nf 746/746 731/731 745/745\nf 731/731 729/729 745/745\nf 745/745 729/729 730/730\nf 727/727 726/726 747/747\nf 747/747 726/726 717/717\nf 748/748 727/727 749/749\nf 749/749 727/727 747/747\nf 747/747 717/717 750/750\nf 750/750 717/717 716/716\nf 749/749 747/747 751/751\nf 751/751 747/747 750/750\nf 749/749 752/752 748/748\nf 748/748 752/752 753/753\nf 730/730 753/753 745/745\nf 745/745 753/753 752/752\nf 728/728 725/725 748/748\nf 748/748 725/725 727/727\nf 730/730 728/728 753/753\nf 753/753 728/728 748/748\nf 700/700 702/702 754/754\nf 754/754 702/702 712/712\nf 705/705 701/701 715/715\nf 715/715 701/701 755/755\nf 754/754 755/755 700/700\nf 700/700 755/755 701/701\nf 742/742 756/756 754/754\nf 754/754 756/756 755/755\nf 740/740 757/757 742/742\nf 742/742 757/757 756/756\nf 755/755 756/756 715/715\nf 715/715 756/756 758/758\nf 754/754 712/712 742/742\nf 742/742 712/712 743/743\nf 759/759 714/714 758/758\nf 758/758 714/714 715/715\nf 714/714 759/759 713/713\nf 713/713 759/759 716/716\nf 759/759 760/760 716/716\nf 716/716 760/760 750/750\nf 760/760 759/759 761/761\nf 761/761 759/759 758/758\nf 762/762 760/760 763/763\nf 763/763 760/760 761/761\nf 756/756 757/757 758/758\nf 758/758 757/757 761/761\nf 757/757 764/764 761/761\nf 761/761 764/764 763/763\nf 760/760 762/762 750/750\nf 750/750 762/762 751/751\nf 766/766 751/751 765/765\nf 765/765 751/751 762/762\nf 767/767 765/765 769/769\nf 769/769 765/765 768/768\nf 765/765 762/762 768/768\nf 768/768 762/762 763/763\nf 765/765 767/767 766/766\nf 766/766 767/767 746/746\nf 766/766 746/746 752/752\nf 752/752 746/746 745/745\nf 752/752 749/749 766/766\nf 766/766 749/749 751/751\nf 757/757 740/740 764/764\nf 764/764 740/740 741/741\nf 769/769 764/764 744/744\nf 744/744 764/764 741/741\nf 769/769 768/768 764/764\nf 764/764 768/768 763/763\nf 767/767 769/769 739/739\nf 739/739 769/769 744/744\nf 739/739 736/736 767/767\nf 767/767 736/736 746/746\nf 219/219 217/217 208/208\nf 208/208 217/217 209/209\nf 209/209 217/217 210/210\nf 210/210 217/217 193/193\nf 198/198 580/580 199/199\nf 199/199 580/580 511/511\nf 76/76 75/75 289/289\nf 289/289 75/75 272/272\nf 277/277 207/207 291/291\nf 291/291 207/207 184/184\nf 290/290 183/183 203/203\nf 203/203 183/183 84/84\nf 490/490 580/580 142/142\nf 142/142 580/580 198/198\nf 142/142 198/198 143/143\nf 143/143 198/198 197/197\nf 149/149 276/276 72/72\nf 72/72 276/276 73/73\nf 104/104 103/103 279/279\nf 279/279 103/103 770/770\nf 770/770 596/596 279/279\nf 279/279 596/596 595/595\nf 605/605 596/596 771/771\nf 771/771 596/596 770/770\nf 771/771 770/770 772/772\nf 772/772 770/770 103/103\nf 772/772 103/103 773/773\nf 773/773 103/103 102/102\nf 155/155 774/774 101/101\nf 101/101 774/774 102/102\nf 773/773 102/102 97/97\nf 97/97 102/102 774/774\nf 775/775 773/773 100/100\nf 100/100 773/773 97/97\nf 773/773 775/775 772/772\nf 772/772 775/775 776/776\nf 772/772 776/776 771/771\nf 771/771 776/776 777/777\nf 200/200 606/606 197/197\nf 197/197 606/606 777/777\nf 189/189 71/71 778/778\nf 778/778 71/71 74/74\nf 97/97 774/774 98/98\nf 98/98 774/774 155/155\nf 183/183 161/161 184/184\nf 184/184 161/161 291/291\nf 777/777 276/276 197/197\nf 197/197 276/276 143/143\nf 777/777 606/606 771/771\nf 771/771 606/606 605/605\nf 613/613 612/612 697/697\nf 697/697 612/612 202/202\nf 595/595 697/697 279/279\nf 279/279 697/697 202/202\nf 73/73 276/276 776/776\nf 776/776 276/276 777/777\nf 776/776 775/775 73/73\nf 73/73 775/775 74/74\nf 100/100 778/778 775/775\nf 775/775 778/778 74/74\nf 778/778 100/100 189/189\nf 189/189 100/100 99/99\nf 245/245 242/242 99/99\nf 99/99 242/242 189/189\nf 131/131 33/33 256/256\nf 28/28 1/1 95/95\nf 95/95 1/1 62/62\nf 27/27 2/2 28/28\nf 28/28 2/2 1/1\nf 265/265 115/115 266/266\nf 266/266 115/115 114/114\nf 95/95 62/62 69/69\nf 69/69 62/62 64/64\nf 62/62 61/61 64/64\nf 64/64 61/61 65/65\nf 61/61 13/13 65/65\nf 65/65 13/13 130/130\nf 13/13 12/12 130/130\nf 130/130 12/12 14/14\nf 31/31 11/11 32/32\nf 32/32 11/11 268/268\nf 15/15 12/12 31/31\nf 31/31 12/12 11/11\nf 31/31 30/30 15/15\nf 15/15 30/30 231/231\nf 30/30 106/106 231/231\nf 231/231 106/106 232/232\nf 287/287 237/237 106/106\nf 106/106 237/237 232/232\nf 254/254 90/90 287/287\nf 287/287 90/90 237/237\nf 253/253 44/44 254/254\nf 254/254 44/44 90/90\nf 253/253 249/249 44/44\nf 44/44 249/249 7/7\nf 267/267 127/127 268/268\nf 268/268 127/127 126/126\nf 10/10 3/3 241/241\nf 241/241 3/3 116/116\nf 3/3 2/2 116/116\nf 116/116 2/2 114/114\nf 2/2 27/27 114/114\nf 114/114 27/27 266/266\nf 27/27 26/26 266/266\nf 266/266 26/26 113/113\nf 26/26 286/286 113/113\nf 113/113 286/286 110/110\nf 286/286 243/243 110/110\nf 110/110 243/243 255/255\nf 243/243 242/242 255/255\nf 255/255 242/242 256/256\nf 242/242 245/245 256/256\nf 256/256 245/245 131/131\nf 245/245 244/244 131/131\nf 131/131 244/244 132/132\nf 265/265 37/37 115/115\nf 115/115 37/37 41/41\nf 41/41 37/37 42/42\nf 42/42 37/37 40/40\nf 92/92 91/91 40/40\nf 40/40 91/91 42/42\nf 236/236 233/233 92/92\nf 92/92 233/233 91/91\nf 191/191 194/194 236/236\nf 236/236 194/194 233/233\nf 258/258 285/285 244/244\nf 244/244 285/285 132/132\nf 257/257 55/55 258/258\nf 258/258 55/55 285/285\nf 257/257 154/154 201/201\nf 201/201 154/154 185/185\nf 56/56 201/201 204/204\nf 204/204 201/201 180/180\nf 55/55 257/257 56/56\nf 56/56 257/257 201/201\nf 204/204 85/85 56/56\nf 56/56 85/85 52/52\nf 86/86 122/122 53/53\nf 53/53 122/122 79/79\nf 212/212 215/215 86/86\nf 86/86 215/215 122/122\nf 211/211 216/216 212/212\nf 212/212 216/216 215/215\nf 192/192 280/280 211/211\nf 211/211 280/280 216/216\nf 191/191 240/240 192/192\nf 192/192 240/240 280/280\nf 249/249 124/124 7/7\nf 7/7 124/124 8/8\nf 124/124 127/127 8/8\nf 8/8 127/127 241/241\nf 127/127 267/267 241/241\nf 241/241 267/267 10/10\nf 10/10 267/267 9/9\nf 9/9 267/267 11/11\n" }, { "path": "main/config.py", "content": "import os\nimport os.path as osp\nimport sys\nimport numpy as np\n\nclass Config:\n \n ## dataset\n # HO3D, DEX_YCB\n trainset = 'HO3D'\n testset = 'DEX_YCB'\n \n ## input, output\n input_img_shape = (256,256) \n \n ## training config\n if trainset == 'HO3D':\n lr_dec_epoch = [10*i for i in range(1,7)]\n end_epoch = 70\n lr = 1e-4\n lr_dec_factor = 0.7\n elif trainset == 'DEX_YCB':\n lr_dec_epoch = [i for i in range(1,25)]\n end_epoch = 25\n lr = 1e-4\n lr_dec_factor = 0.9\n train_batch_size = 16 # per GPU\n lambda_mano_verts = 1e4\n lambda_mano_joints = 1e4\n lambda_mano_pose = 10\n lambda_mano_shape = 0.1\n lambda_joints_img = 100\n ckpt_freq = 10\n\n ## testing config\n test_batch_size = 64\n\n ## others\n num_thread = 20\n gpu_ids = '0'\n num_gpus = 1\n continue_train = False\n \n ## directory\n cur_dir = osp.dirname(os.path.abspath(__file__))\n root_dir = osp.join(cur_dir, '..')\n data_dir = osp.join(root_dir, 'data')\n output_dir = osp.join(root_dir, 'output')\n model_dir = osp.join(output_dir, 'model_dump')\n vis_dir = osp.join(output_dir, 'vis')\n log_dir = osp.join(output_dir, 'log')\n result_dir = osp.join(output_dir, 'result')\n mano_path = osp.join(root_dir, 'common', 'utils', 'manopth')\n \n def set_args(self, gpu_ids, continue_train=False):\n self.gpu_ids = gpu_ids\n self.num_gpus = len(self.gpu_ids.split(','))\n self.continue_train = continue_train\n os.environ[\"CUDA_VISIBLE_DEVICES\"] = self.gpu_ids\n print('>>> Using GPU: {}'.format(self.gpu_ids))\n\ncfg = Config()\n\nsys.path.insert(0, osp.join(cfg.root_dir, 'common'))\nfrom utils.dir import add_pypath, make_folder\nadd_pypath(osp.join(cfg.data_dir))\nadd_pypath(osp.join(cfg.data_dir, cfg.trainset))\nadd_pypath(osp.join(cfg.data_dir, cfg.testset))\nmake_folder(cfg.model_dir)\nmake_folder(cfg.vis_dir)\nmake_folder(cfg.log_dir)\nmake_folder(cfg.result_dir)\n" }, { "path": "main/model.py", "content": "import torch\nimport torch.nn as nn\nfrom torch.nn import functional as F\nfrom nets.backbone import FPN\nfrom nets.transformer import Transformer\nfrom nets.regressor import Regressor\nfrom utils.mano import MANO\nfrom utils.fitting import ScaleTranslationLoss, FittingMonitor\nfrom utils.optimizers import optim_factory\nfrom utils.camera import PerspectiveCamera\nfrom config import cfg\nimport math\n\nclass Model(nn.Module):\n def __init__(self, backbone, FIT, SET, regressor):\n super(Model, self).__init__()\n self.backbone = backbone\n self.FIT = FIT\n self.SET = SET\n self.regressor = regressor\n \n self.fitting_loss = ScaleTranslationLoss(list(range(0, 21))) # fitting joint indices\n\n \n def forward(self, inputs, targets, meta_info, mode):\n p_feats, s_feats = self.backbone(inputs['img']) # primary, secondary feats\n feats = self.FIT(s_feats, p_feats)\n feats = self.SET(feats, feats)\n\n if mode == 'train':\n gt_mano_params = torch.cat([targets['mano_pose'], targets['mano_shape']], dim=1)\n else:\n gt_mano_params = None\n pred_mano_results, gt_mano_results, preds_joints_img = self.regressor(feats, gt_mano_params)\n \n if mode == 'train':\n # loss functions\n loss = {}\n loss['mano_verts'] = cfg.lambda_mano_verts * F.mse_loss(pred_mano_results['verts3d'], gt_mano_results['verts3d'])\n loss['mano_joints'] = cfg.lambda_mano_joints * F.mse_loss(pred_mano_results['joints3d'], gt_mano_results['joints3d'])\n loss['mano_pose'] = cfg.lambda_mano_pose * F.mse_loss(pred_mano_results['mano_pose'], gt_mano_results['mano_pose'])\n loss['mano_shape'] = cfg.lambda_mano_shape * F.mse_loss(pred_mano_results['mano_shape'], gt_mano_results['mano_shape'])\n loss['joints_img'] = cfg.lambda_joints_img * F.mse_loss(preds_joints_img[0], targets['joints_img'])\n return loss\n\n else:\n # test output\n out = {}\n out['joints_coord_img'] = preds_joints_img[0]\n out['mano_pose'] = pred_mano_results['mano_pose_aa']\n out['mano_shape'] = pred_mano_results['mano_shape']\n out['joints_coord_cam'] = pred_mano_results['joints3d']\n out['mesh_coord_cam'] = pred_mano_results['verts3d']\n out['manojoints2cam'] = pred_mano_results['manojoints2cam'] \n out['mano_pose_aa'] = pred_mano_results['mano_pose_aa']\n\n return out\n\n def get_mesh_scale_trans(self, pred_joint_img, pred_joint_cam, init_scale=1., init_depth=1., camera=None, depth_map=None):\n \"\"\"\n pred_joint_img: (batch_size, 21, 2)\n pred_joint_cam: (batch_size, 21, 3)\n \"\"\"\n if camera is None:\n camera = PerspectiveCamera()\n\n dtype, device = pred_joint_cam.dtype, pred_joint_cam.device\n hand_scale = torch.tensor([init_scale / 1.0], dtype=dtype, device=device, requires_grad=False)\n hand_translation = torch.tensor([0, 0, init_depth], dtype=dtype, device=device, requires_grad=True)\n if depth_map is not None:\n tensor_depth = torch.tensor(depth_map, device=device, dtype=dtype)[\n None, None, :, :]\n grid = pred_joint_img.clone()\n grid[:, :, 0] /= tensor_depth.shape[-1]\n grid[:, :, 1] /= tensor_depth.shape[-2]\n grid = 2 * grid - 1\n joints_depth = torch.nn.functional.grid_sample(\n tensor_depth, grid[:, None, :, :]) # (1, 1, 1, 21)\n joints_depth = joints_depth.reshape(1, 21, 1)\n hand_translation = torch.tensor(\n [0, 0, joints_depth[0, cfg.fitting_joint_idxs, 0].mean() / 1000.], device=device, requires_grad=True)\n\n # intended only for demo mesh rendering\n batch_size = 1\n self.fitting_loss.trans_estimation = hand_translation.clone()\n\n params = []\n params.append(hand_translation)\n params.append(hand_scale)\n optimizer, create_graph = optim_factory.create_optimizer(\n params, optim_type='lbfgsls', lr=1.0e-1)\n\n # optimization\n print(\"[Fitting]: fitting the hand scale and translation...\")\n with FittingMonitor(batch_size=batch_size) as monitor:\n fit_camera = monitor.create_fitting_closure(\n optimizer, camera, pred_joint_cam, pred_joint_img, hand_translation, hand_scale, self.fitting_loss, create_graph=create_graph)\n\n loss_val = monitor.run_fitting(\n optimizer, fit_camera, params)\n\n\n print(f\"[Fitting]: fitting finished with loss of {loss_val}\")\n print(f\"Scale: {hand_scale.detach().cpu().numpy()}, Translation: {hand_translation.detach().cpu().numpy()}\")\n return hand_scale, hand_translation\n \ndef init_weights(m):\n if type(m) == nn.ConvTranspose2d:\n nn.init.normal_(m.weight,std=0.001)\n elif type(m) == nn.Conv2d:\n nn.init.normal_(m.weight,std=0.001)\n nn.init.constant_(m.bias, 0)\n elif type(m) == nn.BatchNorm2d:\n nn.init.constant_(m.weight,1)\n nn.init.constant_(m.bias,0)\n elif type(m) == nn.Linear:\n nn.init.normal_(m.weight,std=0.01)\n nn.init.constant_(m.bias,0)\n\ndef get_model(mode):\n backbone = FPN(pretrained=True)\n FIT = Transformer(injection=True) # feature injecting transformer\n SET = Transformer(injection=False) # self enhancing transformer\n regressor = Regressor()\n \n if mode == 'train':\n FIT.apply(init_weights)\n SET.apply(init_weights)\n regressor.apply(init_weights)\n \n model = Model(backbone, FIT, SET, regressor)\n \n return model" }, { "path": "main/test.py", "content": "import torch\nimport argparse\nfrom tqdm import tqdm\nimport numpy as np\nimport torch.backends.cudnn as cudnn\nfrom config import cfg\nfrom base import Tester\n\ndef parse_args():\n parser = argparse.ArgumentParser()\n parser.add_argument('--gpu', type=str, dest='gpu_ids')\n parser.add_argument('--test_epoch', type=str, dest='test_epoch')\n args = parser.parse_args()\n\n if not args.gpu_ids:\n assert 0, \"Please set propoer gpu ids\"\n\n if '-' in args.gpu_ids:\n gpus = args.gpu_ids.split('-')\n gpus[0] = int(gpus[0])\n gpus[1] = int(gpus[1]) + 1\n args.gpu_ids = ','.join(map(lambda x: str(x), list(range(*gpus))))\n\n assert args.test_epoch, 'Test epoch is required.'\n return args\n\ndef main():\n\n args = parse_args()\n cfg.set_args(args.gpu_ids)\n cudnn.benchmark = True\n\n tester = Tester(args.test_epoch)\n tester._make_batch_generator()\n tester._make_model()\n \n eval_result = {}\n cur_sample_idx = 0\n for itr, (inputs, targets, meta_info) in enumerate(tqdm(tester.batch_generator)):\n \n # forward\n with torch.no_grad():\n out = tester.model(inputs, targets, meta_info, 'test')\n \n # save output\n out = {k: v.cpu().numpy() for k,v in out.items()}\n for k,v in out.items(): batch_size = out[k].shape[0]\n out = [{k: v[bid] for k,v in out.items()} for bid in range(batch_size)]\n\n # evaluate\n tester._evaluate(out, cur_sample_idx)\n cur_sample_idx += len(out)\n \n tester._print_eval_result(args.test_epoch)\n\nif __name__ == \"__main__\":\n main()" }, { "path": "main/train.py", "content": "import argparse\nfrom config import cfg\nimport torch\nfrom base import Trainer\nimport torch.backends.cudnn as cudnn\n\ndef parse_args():\n parser = argparse.ArgumentParser()\n parser.add_argument('--gpu', type=str, dest='gpu_ids')\n parser.add_argument('--continue', dest='continue_train', action='store_true')\n args = parser.parse_args()\n\n if not args.gpu_ids:\n assert 0, \"Please set propoer gpu ids\"\n \n if '-' in args.gpu_ids:\n gpus = args.gpu_ids.split('-')\n gpus[0] = int(gpus[0])\n gpus[1] = int(gpus[1]) + 1\n args.gpu_ids = ','.join(map(lambda x: str(x), list(range(*gpus))))\n \n return args\n\ndef main():\n \n # argument parse and create log\n args = parse_args()\n cfg.set_args(args.gpu_ids, args.continue_train)\n cudnn.benchmark = True\n\n trainer = Trainer()\n trainer._make_batch_generator()\n trainer._make_model()\n \n # train\n for epoch in range(trainer.start_epoch, cfg.end_epoch):\n \n trainer.set_lr(epoch)\n trainer.tot_timer.tic()\n trainer.read_timer.tic()\n for itr, (inputs, targets, meta_info) in enumerate(trainer.batch_generator):\n trainer.read_timer.toc()\n trainer.gpu_timer.tic()\n\n # forward\n trainer.optimizer.zero_grad()\n loss = trainer.model(inputs, targets, meta_info, 'train')\n loss = {k:loss[k].mean() for k in loss}\n\n # backward\n sum(loss[k] for k in loss).backward()\n trainer.optimizer.step()\n trainer.gpu_timer.toc()\n screen = [\n 'Epoch %d/%d itr %d/%d:' % (epoch, cfg.end_epoch, itr, trainer.itr_per_epoch),\n 'lr: %g' % (trainer.get_lr()),\n 'speed: %.2f(%.2fs r%.2f)s/itr' % (\n trainer.tot_timer.average_time, trainer.gpu_timer.average_time, trainer.read_timer.average_time),\n '%.2fh/epoch' % (trainer.tot_timer.average_time / 3600. * trainer.itr_per_epoch),\n ]\n screen += ['%s: %.4f' % ('loss_' + k, v.detach()) for k,v in loss.items()]\n trainer.logger.info(' '.join(screen))\n\n trainer.tot_timer.toc()\n trainer.tot_timer.tic()\n trainer.read_timer.tic()\n \n if (epoch+1)%cfg.ckpt_freq== 0 or epoch+1 == cfg.end_epoch:\n trainer.save_model({\n 'epoch': epoch,\n 'network': trainer.model.state_dict(),\n 'optimizer': trainer.optimizer.state_dict(),\n }, epoch+1)\n \n\nif __name__ == \"__main__\":\n main()\n" }, { "path": "requiremets.sh", "content": "pip install numpy==1.17.4 torch==1.9.1 torchvision==0.10.1 einops chumpy opencv-python pycocotools pyrender tqdm\n\n\n" } ]