[ { "path": "LICENSE", "content": "The MIT License (MIT)\n\n\nCopyright (c) 2017 Microsoft Corporation\n\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\nSOFTWARE.\n" }, { "path": "README.md", "content": "# Deep Feature Flow for Video Recognition\n\n\n## Introduction\n\n\n**Deep Feature Flow** is initially described in a [CVPR 2017 paper]\n\nIt provides a simple, fast, accurate, and end-to-end framework for video recognition (e.g., object detection and semantic segmentation in videos). It is worth noting that:\n\n* Deep Feature Flow significantly speeds up video recognition by applying the heavy-weight image recognition network (e.g., ResNet-101) on sparse key frames, and propagating the recognition outputs (feature maps) to the other frames by the light-weight flow network (e.g., [FlowNet].\n* The entire system is end-to-end trained for the task of video recognition, which is vital for improving the recognition accuracy. Directly adopting state-of-the-art flow estimation methods without end-to-end training would deliver noticable worse results.\n* Deep Feature Flow can easily make use of sparsely annotated video recognition datasets, where only a small portion of the frames are annotated with ground-truth labels.\n\n***Click image to watch our demo video***\n\n[![Demo Video on YouTube](https://media.giphy.com/media/14erFWP6f5tDVe/giphy.gif)]\n\n[![Demo Video on YouTube](https://media.giphy.com/media/xwB5LVfIjLtS/giphy.gif)] \n\n## Disclaimer\n\nThis is an official implementation for [Deep Feature Flow for Video Recognition](https://arxiv.org/abs/1611.07715) (DFF) based on MXNet. It is worth noticing that:\n\n * The original implementation is based on our internal Caffe version on Windows. There are slight differences in the final accuracy and running time due to the plenty details in platform switch.\n \n\n\n\n## License\n\n© Microsoft, 2018. Licensed under the [MIT](LICENSE) License.\n\n## Citing Deep Feature Flow\n\nIf you find Deep Feature Flow useful in your research, please consider citing:\n```\n@inproceedings{zhu17dff,\n Author = {Xizhou Zhu, Yuwen Xiong, Jifeng Dai, Lu Yuan, Yichen Wei},\n Title = {Deep Feature Flow for Video Recognition},\n Conference = {CVPR},\n Year = {2017}\n}\n\n@inproceedings{dai16rfcn,\n Author = {Jifeng Dai, Yi Li, Kaiming He, Jian Sun},\n Title = {{R-FCN}: Object Detection via Region-based Fully Convolutional Networks},\n Conference = {NIPS},\n Year = {2016}\n}\n```\n\n## Main Results\n\n\n| | training data | testing data | mAP@0.5 | time/image
(Tesla K40) | time/image
(Maxwell Titan X) |\n|---------------------------------|-------------------|--------------|---------|---------|--------|\n| Frame baseline
(R-FCN, ResNet-v1-101) | ImageNet DET train + VID train | ImageNet VID validation | 74.1 | 0.271s | 0.133s |\n| Deep Feature Flow
(R-FCN, ResNet-v1-101, FlowNet) | ImageNet DET train + VID train | ImageNet VID validation | 73.0 | 0.073s | 0.034s |\n\n*Running time is counted on a single GPU (mini-batch size is 1 in inference, key-frame duration length for Deep Feature Flow is 10).*\n\n*The runtime of the light-weight FlowNet seems to be a bit slower on MXNet than that on Caffe.*\n\n## Requirements: Software\n\n1. MXNet from \n Due to the rapid development of MXNet, it is recommended to checkout this version if you encounter any issues. We may maintain this repository periodically if MXNet adds important feature in future release.\n\n2. Python 2.7. We recommend using Anaconda2 as it already includes many common packages. We do not suppoort Python 3 yet, if you want to use Python 3 you need to modify the code to make it work.\n\n3. Python packages might missing: cython, opencv-python >= 3.2.0, easydict. If `pip` is set up on your system, those packages should be able to be fetched and installed by running\n\t```\n\tpip install Cython\n\tpip install opencv-python==3.2.0.6\n\tpip install easydict==1.6\n\t```\n4. For Windows users, Visual Studio 2015 is needed to compile cython module.\n\n\n## Requirements: Hardware\n\nAny NVIDIA GPUs with at least 6GB memory should be OK\n\n## Installation\n\n1. Clone the Deep Feature Flow repository, and we'll call the directory that you cloned Deep-Feature-Flow as ${DFF_ROOT}. \n\n~~~\ngit clone https://github.com/msracver/Deep-Feature-Flow.git\n~~~\n2. For Windows users, run ``cmd .\\init.bat``. For Linux user, run `sh ./init.sh`. The scripts will build cython module automatically and create some folders.\n\n3. Install MXNet:\n\n\t3.1 Clone MXNet and checkout to [MXNet@(commit 62ecb60)] by\n\t```\n\tgit clone --recursive https://github.com/dmlc/mxnet.git\n\tgit checkout 62ecb60\n\tgit submodule update\n\t```\n\t3.2 Copy operators in `$(DFF_ROOT)/dff_rfcn/operator_cxx` or `$(DFF_ROOT)/rfcn/operator_cxx` to `$(YOUR_MXNET_FOLDER)/src/operator/contrib` by\n\t```\n\tcp -r $(DFF_ROOT)/dff_rfcn/operator_cxx/* $(MXNET_ROOT)/src/operator/contrib/\n\t```\n\t3.3 Compile MXNet\n\t```\n\tcd ${MXNET_ROOT}\n\tmake -j4\n\t```\n\t3.4 Install the MXNet Python binding by\n\t\n\t***Note: If you will actively switch between different versions of MXNet, please follow 3.5 instead of 3.4***\n\t```\n\tcd python\n\tsudo python setup.py install\n\t```\n\t3.5 For advanced users, you may put your Python packge into `./external/mxnet/$(YOUR_MXNET_PACKAGE)`, and modify `MXNET_VERSION` in `./experiments/dff_rfcn/cfgs/*.yaml` to `$(YOUR_MXNET_PACKAGE)`. Thus you can switch among different versions of MXNet quickly.\n\n\n## Demo\n\n\n1. To run the demo with our trained model (on ImageNet DET + VID train), please download the model manually from [OneDrive](for users from Mainland China, please try [Baidu Yun]), and put it under folder `model/`.\n\n\tMake sure it looks like this:\n\t```\n\t./model/rfcn_vid-0000.params\n\t./model/rfcn_dff_flownet_vid-0000.params\n\t```\n2. Run (inference batch size = 1)\n\t```\n\tpython ./rfcn/demo.py\n\tpython ./dff_rfcn/demo.py\n\t```\n\tor run (inference batch size = 10)\n\t```\n\tpython ./rfcn/demo_batch.py\n\tpython ./dff_rfcn/demo_batch.py\n\t```\n\n## Preparation for Training & Testing\n\n1. Please download ILSVRC2015 DET and ILSVRC2015 VID dataset, and make sure it looks like this:\n\n\t```\n\t./data/ILSVRC2015/\n\t./data/ILSVRC2015/Annotations/DET\n\t./data/ILSVRC2015/Annotations/VID\n\t./data/ILSVRC2015/Data/DET\n\t./data/ILSVRC2015/Data/VID\n\t./data/ILSVRC2015/ImageSets\n\t```\n\n## FAQ\n\nQ: It says `AttributeError: 'module' object has no attribute 'MultiProposal'`.\n\nA: This is because either\n - you forget to copy the operators to your MXNet folder\n - or you copy to the wrong path\n - or you forget to re-compile and install\n - or you install the wrong MXNet\n\n Please print `mxnet.__path__` to make sure you use correct MXNet\n\n

\nQ: I encounter `segment fault` at the beginning.\n\nA: A compatibility issue has been identified between MXNet and opencv-python 3.0+. We suggest that you always `import cv2` first before `import mxnet` in the entry script. \n\n

\nQ: I find the training speed becomes slower when training for a long time.\n\nA: It has been identified that MXNet on Windows has this problem. So we recommend to run this program on Linux. You could also stop it and resume the training process to regain the training speed if you encounter this problem.\n\n

\nQ: Can you share your caffe implementation?\n\nA: Due to several reasons (code is based on a old, internal Caffe, port to public Caffe needs extra work, time limit, etc.). We do not plan to release our Caffe code. Since a warping layer is easy to implement, anyone who wish to do it is welcome to make a pull request.\n" }, { "path": "ThirdPartyNotices.txt", "content": "Deep Feature Flow\n\nTHIRD-PARTY SOFTWARE NOTICES AND INFORMATION\n\nThis project incorporates components from the projects listed below. The original copyright notices and the licenses under which Microsoft received such components are set forth below. Microsoft reserves all rights not expressly granted herein, whether by implication, estoppel or otherwise.\n\n1. MXNet (https://github.com/apache/incubator-mxnet)\n2. Fast R-CNN (https://github.com/rbgirshick/fast-rcnn)\n3. Faster R-CNN (https://github.com/rbgirshick/py-faster-rcnn)\n4. MS COCO API (https://github.com/cocodataset/cocoapi)\n\n\nMXNet\n\nCopyright (c) 2015-2016 by Contributors\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this file except in compliance with the License.\nYou may obtain a copy of the License at\n\n http://www.apache.org/licenses/LICENSE-2.0\n\nUnless required by applicable law or agreed to in writing, software\ndistributed under the License is distributed on an \"AS IS\" BASIS,\nWITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nSee the License for the specific language governing permissions and\nlimitations under the License.\n\n\nFast R-CNN\n\nCopyright (c) Microsoft Corporation\n\nAll rights reserved.\n\nMIT License\n\nPermission is hereby granted, free of charge, to any person obtaining a\ncopy of this software and associated documentation files (the \"Software\"),\nto deal in the Software without restriction, including without limitation\nthe rights to use, copy, modify, merge, publish, distribute, sublicense,\nand/or sell copies of the Software, and to permit persons to whom the\nSoftware is furnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included\nin all copies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL\nTHE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR\nOTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,\nARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR\nOTHER DEALINGS IN THE SOFTWARE.\n\n\nFaster R-CNN\n\nThe MIT License (MIT)\n\nCopyright (c) 2015 Microsoft Corporation\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in\nall copies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN\nTHE SOFTWARE.\n\n\nMS COCO API\n\nCopyright (c) 2014, Piotr Dollar and Tsung-Yi Lin\nAll rights reserved.\n\nRedistribution and use in source and binary forms, with or without\nmodification, are permitted provided that the following conditions are met:\n\n1. Redistributions of source code must retain the above copyright notice, this\n list of conditions and the following disclaimer.\n2. Redistributions in binary form must reproduce the above copyright notice,\n this list of conditions and the following disclaimer in the documentation\n and/or other materials provided with the distribution.\n\nTHIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\" AND\nANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED\nWARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE\nDISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR\nANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES\n(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;\nLOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND\nON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT\n(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS\nSOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\n\nThe views and conclusions contained in the software and documentation are those\nof the authors and should not be interpreted as representing official policies,\neither expressed or implied, of the FreeBSD Project.\n" }, { "path": "dff_rfcn/__init__.py", "content": "" }, { "path": "dff_rfcn/_init_paths.py", "content": "# --------------------------------------------------------\n# Deep Feature Flow\n# Copyright (c) 2017 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Written by Yuwen Xiong\n# --------------------------------------------------------\n\nimport os.path as osp\nimport sys\n\ndef add_path(path):\n if path not in sys.path:\n sys.path.insert(0, path)\n\nthis_dir = osp.dirname(__file__)\n\nlib_path = osp.join(this_dir, '..', 'lib')\nadd_path(lib_path)\n" }, { "path": "dff_rfcn/config/__init__.py", "content": "" }, { "path": "dff_rfcn/config/config.py", "content": "# --------------------------------------------------------\n# Deep Feature Flow\n# Copyright (c) 2017 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Modified by Xizhou Zhu, Yuwen Xiong, Bin Xiao\n# --------------------------------------------------------\n# Based on:\n# MX-RCNN\n# Copyright (c) 2016 by Contributors\n# Licence under The Apache 2.0 License\n# https://github.com/ijkguo/mx-rcnn/\n# --------------------------------------------------------\n\nimport yaml\nimport numpy as np\nfrom easydict import EasyDict as edict\n\nconfig = edict()\n\nconfig.MXNET_VERSION = ''\nconfig.output_path = ''\nconfig.symbol = ''\nconfig.gpus = ''\nconfig.CLASS_AGNOSTIC = True\nconfig.SCALES = [(600, 1000)] # first is scale (the shorter side); second is max size\n\n# default training\nconfig.default = edict()\nconfig.default.frequent = 20\nconfig.default.kvstore = 'device'\n\n# network related params\nconfig.network = edict()\nconfig.network.pretrained = ''\nconfig.network.pretrained_flow = ''\nconfig.network.pretrained_epoch = 0\nconfig.network.PIXEL_MEANS = np.array([0, 0, 0])\nconfig.network.IMAGE_STRIDE = 0\nconfig.network.RPN_FEAT_STRIDE = 16\nconfig.network.RCNN_FEAT_STRIDE = 16\nconfig.network.FIXED_PARAMS = ['gamma', 'beta']\nconfig.network.ANCHOR_SCALES = (8, 16, 32)\nconfig.network.ANCHOR_RATIOS = (0.5, 1, 2)\nconfig.network.NORMALIZE_RPN = True\nconfig.network.ANCHOR_MEANS = (0.0, 0.0, 0.0, 0.0)\nconfig.network.ANCHOR_STDS = (0.1, 0.1, 0.4, 0.4)\nconfig.network.NUM_ANCHORS = len(config.network.ANCHOR_SCALES) * len(config.network.ANCHOR_RATIOS)\nconfig.network.DFF_FEAT_DIM = 1024\n\n# dataset related params\nconfig.dataset = edict()\nconfig.dataset.dataset = 'ImageNetVID'\nconfig.dataset.image_set = 'DET_train_30classes+VID_train_15frames'\nconfig.dataset.test_image_set = 'VID_val_videos'\nconfig.dataset.root_path = './data'\nconfig.dataset.dataset_path = './data/ILSVRC2015'\nconfig.dataset.NUM_CLASSES = 31\n\n\nconfig.TRAIN = edict()\n\nconfig.TRAIN.lr = 0\nconfig.TRAIN.lr_step = ''\nconfig.TRAIN.lr_factor = 0.1\nconfig.TRAIN.warmup = False\nconfig.TRAIN.warmup_lr = 0\nconfig.TRAIN.warmup_step = 0\nconfig.TRAIN.momentum = 0.9\nconfig.TRAIN.wd = 0.0005\nconfig.TRAIN.begin_epoch = 0\nconfig.TRAIN.end_epoch = 0\nconfig.TRAIN.model_prefix = ''\n\n# whether resume training\nconfig.TRAIN.RESUME = False\n# whether flip image\nconfig.TRAIN.FLIP = True\n# whether shuffle image\nconfig.TRAIN.SHUFFLE = True\n# whether use OHEM\nconfig.TRAIN.ENABLE_OHEM = False\n# size of images for each device, 2 for rcnn, 1 for rpn and e2e\nconfig.TRAIN.BATCH_IMAGES = 2\n# e2e changes behavior of anchor loader and metric\nconfig.TRAIN.END2END = False\n# group images with similar aspect ratio\nconfig.TRAIN.ASPECT_GROUPING = True\n\n# R-CNN\n# rcnn rois batch size\nconfig.TRAIN.BATCH_ROIS = 128\nconfig.TRAIN.BATCH_ROIS_OHEM = 128\n# rcnn rois sampling params\nconfig.TRAIN.FG_FRACTION = 0.25\nconfig.TRAIN.FG_THRESH = 0.5\nconfig.TRAIN.BG_THRESH_HI = 0.5\nconfig.TRAIN.BG_THRESH_LO = 0.0\n# rcnn bounding box regression params\nconfig.TRAIN.BBOX_REGRESSION_THRESH = 0.5\nconfig.TRAIN.BBOX_WEIGHTS = np.array([1.0, 1.0, 1.0, 1.0])\n\n# RPN anchor loader\n# rpn anchors batch size\nconfig.TRAIN.RPN_BATCH_SIZE = 256\n# rpn anchors sampling params\nconfig.TRAIN.RPN_FG_FRACTION = 0.5\nconfig.TRAIN.RPN_POSITIVE_OVERLAP = 0.7\nconfig.TRAIN.RPN_NEGATIVE_OVERLAP = 0.3\nconfig.TRAIN.RPN_CLOBBER_POSITIVES = False\n# rpn bounding box regression params\nconfig.TRAIN.RPN_BBOX_WEIGHTS = (1.0, 1.0, 1.0, 1.0)\nconfig.TRAIN.RPN_POSITIVE_WEIGHT = -1.0\n\n# used for end2end training\n# RPN proposal\nconfig.TRAIN.CXX_PROPOSAL = True\nconfig.TRAIN.RPN_NMS_THRESH = 0.7\nconfig.TRAIN.RPN_PRE_NMS_TOP_N = 12000\nconfig.TRAIN.RPN_POST_NMS_TOP_N = 2000\nconfig.TRAIN.RPN_MIN_SIZE = config.network.RPN_FEAT_STRIDE\n# approximate bounding box regression\nconfig.TRAIN.BBOX_NORMALIZATION_PRECOMPUTED = True\nconfig.TRAIN.BBOX_MEANS = (0.0, 0.0, 0.0, 0.0)\nconfig.TRAIN.BBOX_STDS = (0.1, 0.1, 0.2, 0.2)\n\n# DFF, trained image sampled from [min_offset, max_offset]\nconfig.TRAIN.MIN_OFFSET = -9\nconfig.TRAIN.MAX_OFFSET = 0\n\nconfig.TEST = edict()\n\n# R-CNN testing\n# use rpn to generate proposal\nconfig.TEST.HAS_RPN = False\n# size of images for each device\nconfig.TEST.BATCH_IMAGES = 1\n\n# RPN proposal\nconfig.TEST.CXX_PROPOSAL = True\nconfig.TEST.RPN_NMS_THRESH = 0.7\nconfig.TEST.RPN_PRE_NMS_TOP_N = 6000\nconfig.TEST.RPN_POST_NMS_TOP_N = 300\nconfig.TEST.RPN_MIN_SIZE = config.network.RPN_FEAT_STRIDE\n\n# RCNN nms\nconfig.TEST.NMS = 0.3\n\n# DFF\nconfig.TEST.KEY_FRAME_INTERVAL = 10\n\nconfig.TEST.max_per_image = 300\n\n# Test Model Epoch\nconfig.TEST.test_epoch = 0\n\n\ndef update_config(config_file):\n exp_config = None\n with open(config_file) as f:\n exp_config = edict(yaml.load(f))\n for k, v in exp_config.items():\n if k in config:\n if isinstance(v, dict):\n if k == 'TRAIN':\n if 'BBOX_WEIGHTS' in v:\n v['BBOX_WEIGHTS'] = np.array(v['BBOX_WEIGHTS'])\n elif k == 'network':\n if 'PIXEL_MEANS' in v:\n v['PIXEL_MEANS'] = np.array(v['PIXEL_MEANS'])\n for vk, vv in v.items():\n config[k][vk] = vv\n else:\n if k == 'SCALES':\n config[k][0] = (tuple(v))\n else:\n config[k] = v\n else:\n raise ValueError(\"key must exist in config.py\")\n" }, { "path": "dff_rfcn/core/DataParallelExecutorGroup.py", "content": "# --------------------------------------------------------\n# Deep Feature Flow\n# Copyright (c) 2017 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Modified by Yuwen Xiong\n# --------------------------------------------------------\n# Based on:\n# MX-RCNN\n# Copyright (c) 2016 by Contributors\n# Licence under The Apache 2.0 License\n# https://github.com/ijkguo/mx-rcnn/\n# --------------------------------------------------------\n\nimport logging\nimport numpy as np\n\nfrom mxnet import context as ctx\nfrom mxnet import ndarray as nd\nfrom mxnet.io import DataDesc\nfrom mxnet.executor_manager import _split_input_slice\n\n\n\ndef _load_general(data, targets, major_axis):\n \"\"\"Load a list of arrays into a list of arrays specified by slices\"\"\"\n for d_src, d_targets in zip(data, targets):\n if isinstance(d_targets, nd.NDArray):\n d_src.copyto(d_targets)\n elif isinstance(d_src, (list, tuple)):\n for src, dst in zip(d_src, d_targets):\n src.copyto(dst)\n else:\n raise NotImplementedError\n\n\n\ndef _load_data(batch, targets, major_axis):\n \"\"\"Load data into sliced arrays\"\"\"\n _load_general(batch.data, targets, major_axis)\n\n\ndef _load_label(batch, targets, major_axis):\n \"\"\"Load label into sliced arrays\"\"\"\n _load_general(batch.label, targets, major_axis)\n\n\ndef _merge_multi_context(outputs, major_axis):\n \"\"\"Merge outputs that lives on multiple context into one, so that they look\n like living on one context.\n \"\"\"\n rets = []\n for tensors, axis in zip(outputs, major_axis):\n if axis >= 0:\n rets.append(nd.concatenate(tensors, axis=axis, always_copy=False))\n else:\n # negative axis means the there is no batch_size axis, and all the\n # results should be the same on each device. We simply take the\n # first one, without checking they are actually the same\n rets.append(tensors[0])\n return rets\n\n\n\nclass DataParallelExecutorGroup(object):\n \"\"\"DataParallelExecutorGroup is a group of executors that lives on a group of devices.\n This is a helper class used to implement data parallelization. Each mini-batch will\n be split and run on the devices.\n\n Parameters\n ----------\n symbol : Symbol\n The common symbolic computation graph for all executors.\n contexts : list\n A list of contexts.\n workload : list\n If not `None`, could be a list of numbers that specify the workload to be assigned\n to different context. Larger number indicate heavier workload.\n data_shapes : list\n Should be a list of (name, shape) tuples, for the shapes of data. Note the order is\n important and should be the same as the order that the `DataIter` provide the data.\n label_shapes : list\n Should be a list of (name, shape) tuples, for the shapes of label. Note the order is\n important and should be the same as the order that the `DataIter` provide the label.\n param_names : list\n A list of strings, indicating the names of parameters (e.g. weights, filters, etc.)\n in the computation graph.\n for_training : bool\n Indicate whether the executors should be bind for training. When not doing training,\n the memory for gradients will not be allocated.\n inputs_need_grad : bool\n Indicate whether the gradients for the input data should be computed. This is currently\n not used. It will be useful for implementing composition of modules.\n shared_group : DataParallelExecutorGroup\n Default is `None`. This is used in bucketing. When not `None`, it should be a executor\n group corresponding to a different bucket. In other words, it will correspond to a different\n symbol but with the same set of parameters (e.g. unrolled RNNs with different lengths).\n In this case, many memory will be shared.\n logger : Logger\n Default is `logging`.\n fixed_param_names: list of str\n Indicate parameters to be fixed during training. Parameters in this list will not allocate\n space for gradient, nor do gradient calculation.\n grad_req : str, list of str, dict of str to str\n Requirement for gradient accumulation. Can be 'write', 'add', or 'null'\n (default to 'write').\n Can be specified globally (str) or for each argument (list, dict).\n \"\"\"\n def __init__(self, symbol, contexts, workload, data_shapes, label_shapes, param_names,\n for_training, inputs_need_grad, shared_group=None, logger=logging,\n fixed_param_names=None, grad_req='write', state_names=None):\n self.param_names = param_names\n self.arg_names = symbol.list_arguments()\n self.aux_names = symbol.list_auxiliary_states()\n\n self.symbol = symbol\n self.contexts = contexts\n self.workload = workload\n\n self.for_training = for_training\n self.inputs_need_grad = inputs_need_grad\n\n self.logger = logger\n #In the future we should have a better way to profile memory per device (haibin)\n # self._total_exec_bytes = 0\n self.fixed_param_names = fixed_param_names\n if self.fixed_param_names is None:\n self.fixed_param_names = []\n\n self.state_names = state_names\n if self.state_names is None:\n self.state_names = []\n\n if not for_training:\n grad_req = 'null'\n\n # data_shapes = [x if isinstance(x, DataDesc) else DataDesc(*x) for x in data_shapes]\n # if label_shapes is not None:\n # label_shapes = [x if isinstance(x, DataDesc) else DataDesc(*x) for x in label_shapes]\n\n data_names = [x.name for x in data_shapes[0]]\n\n if isinstance(grad_req, str):\n self.grad_req = {}\n for k in self.arg_names:\n if k in self.param_names:\n self.grad_req[k] = 'null' if k in self.fixed_param_names else grad_req\n elif k in data_names:\n self.grad_req[k] = grad_req if self.inputs_need_grad else 'null'\n else:\n self.grad_req[k] = 'null'\n elif isinstance(grad_req, (list, tuple)):\n assert len(grad_req) == len(self.arg_names)\n self.grad_req = dict(zip(self.arg_names, grad_req))\n elif isinstance(grad_req, dict):\n self.grad_req = {}\n for k in self.arg_names:\n if k in self.param_names:\n self.grad_req[k] = 'null' if k in self.fixed_param_names else 'write'\n elif k in data_names:\n self.grad_req[k] = 'write' if self.inputs_need_grad else 'null'\n else:\n self.grad_req[k] = 'null'\n self.grad_req.update(grad_req)\n else:\n raise ValueError(\"grad_req must be one of str, list, tuple, or dict.\")\n\n if shared_group is not None:\n self.shared_data_arrays = shared_group.shared_data_arrays\n else:\n self.shared_data_arrays = [{} for _ in contexts]\n\n # initialize some instance variables\n self.batch_size = len(data_shapes)\n self.slices = None\n self.execs = []\n self._default_execs = None\n self.data_arrays = None\n self.label_arrays = None\n self.param_arrays = None\n self.state_arrays = None\n self.grad_arrays = None\n self.aux_arrays = None\n self.input_grad_arrays = None\n\n self.data_shapes = None\n self.label_shapes = None\n self.data_layouts = None\n self.label_layouts = None\n self.output_layouts = [DataDesc.get_batch_axis(self.symbol[name].attr('__layout__'))\n for name in self.symbol.list_outputs()]\n self.bind_exec(data_shapes, label_shapes, shared_group)\n\n def decide_slices(self, data_shapes):\n \"\"\"Decide the slices for each context according to the workload.\n\n Parameters\n ----------\n data_shapes : list\n list of (name, shape) specifying the shapes for the input data or label.\n \"\"\"\n assert len(data_shapes) > 0\n major_axis = [DataDesc.get_batch_axis(x.layout) for x in data_shapes]\n\n for (name, shape), axis in zip(data_shapes, major_axis):\n if axis == -1:\n continue\n\n batch_size = shape[axis]\n if self.batch_size is not None:\n assert batch_size == self.batch_size, (\"all data must have the same batch size: \"\n + (\"batch_size = %d, but \" % self.batch_size)\n + (\"%s has shape %s\" % (name, shape)))\n else:\n self.batch_size = batch_size\n self.slices = _split_input_slice(self.batch_size, self.workload)\n\n return major_axis\n\n def _collect_arrays(self):\n \"\"\"Collect internal arrays from executors.\"\"\"\n # convenient data structures\n self.data_arrays = [[e.arg_dict[name] for name, _ in self.data_shapes[0]] for e in self.execs]\n\n self.state_arrays = [[e.arg_dict[name] for e in self.execs]\n for name in self.state_names]\n\n if self.label_shapes is not None:\n self.label_arrays = [[e.arg_dict[name] for name, _ in self.label_shapes[0]] for e in self.execs]\n else:\n self.label_arrays = None\n\n self.param_arrays = [[exec_.arg_arrays[i] for exec_ in self.execs]\n for i, name in enumerate(self.arg_names)\n if name in self.param_names]\n if self.for_training:\n self.grad_arrays = [[exec_.grad_arrays[i] for exec_ in self.execs]\n for i, name in enumerate(self.arg_names)\n if name in self.param_names]\n else:\n self.grad_arrays = None\n\n data_names = [x[0] for x in self.data_shapes]\n if self.inputs_need_grad:\n self.input_grad_arrays = [[exec_.grad_arrays[i] for exec_ in self.execs]\n for i, name in enumerate(self.arg_names)\n if name in data_names]\n else:\n self.input_grad_arrays = None\n\n self.aux_arrays = [[exec_.aux_arrays[i] for exec_ in self.execs]\n for i in range(len(self.aux_names))]\n\n def bind_exec(self, data_shapes, label_shapes, shared_group=None, reshape=False):\n \"\"\"Bind executors on their respective devices.\n\n Parameters\n ----------\n data_shapes : list\n label_shapes : list\n shared_group : DataParallelExecutorGroup\n reshape : bool\n \"\"\"\n assert reshape or not self.execs\n\n for i in range(len(self.contexts)):\n data_shapes_i = data_shapes[i]\n if label_shapes is not None:\n label_shapes_i = label_shapes[i]\n else:\n label_shapes_i = []\n\n if reshape:\n self.execs[i] = self._default_execs[i].reshape(\n allow_up_sizing=True, **dict(data_shapes_i + label_shapes_i))\n else:\n self.execs.append(self._bind_ith_exec(i, data_shapes_i, label_shapes_i,\n shared_group))\n\n self.data_shapes = data_shapes\n self.label_shapes = label_shapes\n self._collect_arrays()\n\n def reshape(self, data_shapes, label_shapes):\n \"\"\"Reshape executors.\n\n Parameters\n ----------\n data_shapes : list\n label_shapes : list\n \"\"\"\n if self._default_execs is None:\n self._default_execs = [i for i in self.execs]\n for i in range(len(self.contexts)):\n self.execs[i] = self._default_execs[i].reshape(\n allow_up_sizing=True, **dict(data_shapes[i] + (label_shapes[i] if label_shapes is not None else []))\n )\n self.data_shapes = data_shapes\n self.label_shapes = label_shapes\n self._collect_arrays()\n\n\n def set_params(self, arg_params, aux_params):\n \"\"\"Assign, i.e. copy parameters to all the executors.\n\n Parameters\n ----------\n arg_params : dict\n A dictionary of name to `NDArray` parameter mapping.\n aux_params : dict\n A dictionary of name to `NDArray` auxiliary variable mapping.\n \"\"\"\n for exec_ in self.execs:\n exec_.copy_params_from(arg_params, aux_params)\n\n def get_params(self, arg_params, aux_params):\n \"\"\" Copy data from each executor to `arg_params` and `aux_params`.\n\n Parameters\n ----------\n arg_params : list of NDArray\n target parameter arrays\n aux_params : list of NDArray\n target aux arrays\n\n Notes\n -----\n - This function will inplace update the NDArrays in arg_params and aux_params.\n \"\"\"\n for name, block in zip(self.param_names, self.param_arrays):\n weight = sum(w.copyto(ctx.cpu()) for w in block) / len(block)\n weight.astype(arg_params[name].dtype).copyto(arg_params[name])\n for name, block in zip(self.aux_names, self.aux_arrays):\n weight = sum(w.copyto(ctx.cpu()) for w in block) / len(block)\n weight.astype(aux_params[name].dtype).copyto(aux_params[name])\n\n def forward(self, data_batch, is_train=None):\n \"\"\"Split `data_batch` according to workload and run forward on each devices.\n\n Parameters\n ----------\n data_batch : DataBatch\n Or could be any object implementing similar interface.\n is_train : bool\n The hint for the backend, indicating whether we are during training phase.\n Default is `None`, then the value `self.for_training` will be used.\n Returns\n -------\n\n \"\"\"\n _load_data(data_batch, self.data_arrays, self.data_layouts)\n if is_train is None:\n is_train = self.for_training\n\n if self.label_arrays is not None:\n assert not is_train or data_batch.label\n if data_batch.label:\n _load_label(data_batch, self.label_arrays, self.label_layouts)\n\n for exec_ in self.execs:\n exec_.forward(is_train=is_train)\n\n\n def get_outputs(self, merge_multi_context=True):\n \"\"\"Get outputs of the previous forward computation.\n\n Parameters\n ----------\n merge_multi_context : bool\n Default is `True`. In the case when data-parallelism is used, the outputs\n will be collected from multiple devices. A `True` value indicate that we\n should merge the collected results so that they look like from a single\n executor.\n\n Returns\n -------\n If `merge_multi_context` is `True`, it is like `[out1, out2]`. Otherwise, it\n is like `[[out1_dev1, out1_dev2], [out2_dev1, out2_dev2]]`. All the output\n elements are `NDArray`.\n \"\"\"\n outputs = [[exec_.outputs[i] for exec_ in self.execs]\n for i in range(len(self.execs[0].outputs))]\n if merge_multi_context:\n outputs = _merge_multi_context(outputs, self.output_layouts)\n return outputs\n\n def get_states(self, merge_multi_context=True):\n \"\"\"Get states from all devices\n\n Parameters\n ----------\n merge_multi_context : bool\n Default is `True`. In the case when data-parallelism is used, the states\n will be collected from multiple devices. A `True` value indicate that we\n should merge the collected results so that they look like from a single\n executor.\n\n Returns\n -------\n If `merge_multi_context` is `True`, it is like `[out1, out2]`. Otherwise, it\n is like `[[out1_dev1, out1_dev2], [out2_dev1, out2_dev2]]`. All the output\n elements are `NDArray`.\n \"\"\"\n assert not merge_multi_context, \\\n \"merge_multi_context=True is not supported for get_states yet.\"\n return self.state_arrays\n\n def set_states(self, states=None, value=None):\n \"\"\"Set value for states. Only one of states & value can be specified.\n\n Parameters\n ----------\n states : list of list of NDArrays\n source states arrays formatted like [[state1_dev1, state1_dev2],\n [state2_dev1, state2_dev2]].\n value : number\n a single scalar value for all state arrays.\n \"\"\"\n if states is not None:\n assert value is None, \"Only one of states & value can be specified.\"\n _load_general(states, self.state_arrays, (0,)*len(states))\n else:\n assert value is not None, \"At least one of states & value must be specified.\"\n assert states is None, \"Only one of states & value can be specified.\"\n for d_dst in self.state_arrays:\n for dst in d_dst:\n dst[:] = value\n\n def get_input_grads(self, merge_multi_context=True):\n \"\"\"Get the gradients with respect to the inputs of the module.\n\n Parameters\n ----------\n merge_multi_context : bool\n Default is `True`. In the case when data-parallelism is used, the outputs\n will be collected from multiple devices. A `True` value indicate that we\n should merge the collected results so that they look like from a single\n executor.\n\n Returns\n -------\n If `merge_multi_context` is `True`, it is like `[grad1, grad2]`. Otherwise, it\n is like `[[grad1_dev1, grad1_dev2], [grad2_dev1, grad2_dev2]]`. All the output\n elements are `NDArray`.\n \"\"\"\n assert self.inputs_need_grad\n if merge_multi_context:\n return _merge_multi_context(self.input_grad_arrays, self.data_layouts)\n return self.input_grad_arrays\n\n def backward(self, out_grads=None):\n \"\"\"Run backward on all devices. A backward should be called after\n a call to the forward function. Backward cannot be called unless\n `self.for_training` is `True`.\n\n Parameters\n ----------\n out_grads : NDArray or list of NDArray, optional\n Gradient on the outputs to be propagated back.\n This parameter is only needed when bind is called\n on outputs that are not a loss function.\n \"\"\"\n assert self.for_training, 're-bind with for_training=True to run backward'\n if out_grads is None:\n out_grads = []\n\n for i, exec_ in enumerate(self.execs):\n out_grads_slice = []\n exec_.backward(out_grads=out_grads_slice)\n\n def update_metric(self, eval_metric, labels):\n \"\"\"Accumulate the performance according to `eval_metric` on all devices.\n\n Parameters\n ----------\n eval_metric : EvalMetric\n The metric used for evaluation.\n labels : list of NDArray\n Typically comes from `label` of a `DataBatch`.\n \"\"\"\n for texec, labels in zip(self.execs, labels):\n eval_metric.update(labels, texec.outputs)\n\n def _bind_ith_exec(self, i, data_shapes, label_shapes, shared_group):\n \"\"\"Internal utility function to bind the i-th executor.\n \"\"\"\n shared_exec = None if shared_group is None else shared_group.execs[i]\n context = self.contexts[i]\n shared_data_arrays = self.shared_data_arrays[i]\n\n input_shapes = dict(data_shapes)\n if label_shapes is not None:\n input_shapes.update(dict(label_shapes))\n\n arg_shapes, _, aux_shapes = self.symbol.infer_shape(**input_shapes)\n assert arg_shapes is not None, \"shape inference failed\"\n\n input_types = {x.name: x.dtype for x in data_shapes}\n if label_shapes is not None:\n input_types.update({x.name: x.dtype for x in label_shapes})\n arg_types, _, aux_types = self.symbol.infer_type(**input_types)\n assert arg_types is not None, \"type inference failed\"\n\n arg_arrays = []\n grad_arrays = {} if self.for_training else None\n\n def _get_or_reshape(name, shared_data_arrays, arg_shape, arg_type, context, logger):\n \"\"\"Internal helper to get a memory block or re-use by re-shaping\"\"\"\n if name in shared_data_arrays:\n arg_arr = shared_data_arrays[name]\n\n if np.prod(arg_arr.shape) >= np.prod(arg_shape):\n # nice, we can directly re-use this data blob\n assert arg_arr.dtype == arg_type\n arg_arr = arg_arr.reshape(arg_shape)\n else:\n logger.warning(('bucketing: data \"%s\" has a shape %s' % (name, arg_shape)) +\n (', which is larger than already allocated ') +\n ('shape %s' % (arg_arr.shape,)) +\n ('. Need to re-allocate. Consider putting ') +\n ('default_bucket_key to') +\n (' be the bucket taking the largest input for better ') +\n ('memory sharing.'))\n arg_arr = nd.zeros(arg_shape, context, dtype=arg_type)\n\n # replace existing shared array because the new one is bigger\n shared_data_arrays[name] = arg_arr\n else:\n arg_arr = nd.zeros(arg_shape, context, dtype=arg_type)\n shared_data_arrays[name] = arg_arr\n\n return arg_arr\n\n # create or borrow arguments and gradients\n for j in range(len(self.arg_names)):\n name = self.arg_names[j]\n if name in self.param_names: # model parameters\n if shared_exec is None:\n arg_arr = nd.zeros(arg_shapes[j], context, dtype=arg_types[j])\n if self.grad_req[name] != 'null':\n grad_arr = nd.zeros(arg_shapes[j], context, dtype=arg_types[j])\n grad_arrays[name] = grad_arr\n else:\n arg_arr = shared_exec.arg_dict[name]\n assert arg_arr.shape == arg_shapes[j]\n assert arg_arr.dtype == arg_types[j]\n if self.grad_req[name] != 'null':\n grad_arrays[name] = shared_exec.grad_dict[name]\n else: # data, label, or states\n arg_arr = _get_or_reshape(name, shared_data_arrays, arg_shapes[j], arg_types[j],\n context, self.logger)\n\n # data might also need grad if inputs_need_grad is True\n if self.grad_req[name] != 'null':\n grad_arrays[name] = _get_or_reshape('grad of ' + name, shared_data_arrays,\n arg_shapes[j], arg_types[j], context,\n self.logger)\n\n arg_arrays.append(arg_arr)\n\n # create or borrow aux variables\n if shared_exec is None:\n aux_arrays = [nd.zeros(s, context, dtype=t) for s, t in zip(aux_shapes, aux_types)]\n else:\n for j, arr in enumerate(shared_exec.aux_arrays):\n assert aux_shapes[j] == arr.shape\n assert aux_types[j] == arr.dtype\n aux_arrays = shared_exec.aux_arrays[:]\n\n executor = self.symbol.bind(ctx=context, args=arg_arrays,\n args_grad=grad_arrays, aux_states=aux_arrays,\n grad_req=self.grad_req, shared_exec=shared_exec)\n # Get the total bytes allocated for this executor\n return executor\n\n def _sliced_shape(self, shapes, i, major_axis):\n \"\"\"Get the sliced shapes for the i-th executor.\n\n Parameters\n ----------\n shapes : list of (str, tuple)\n The original (name, shape) pairs.\n i : int\n Which executor we are dealing with.\n \"\"\"\n sliced_shapes = []\n for desc, axis in zip(shapes, major_axis):\n shape = list(desc.shape)\n if axis >= 0:\n shape[axis] = self.slices[i].stop - self.slices[i].start\n sliced_shapes.append(DataDesc(desc.name, tuple(shape), desc.dtype, desc.layout))\n return sliced_shapes\n\n def install_monitor(self, mon):\n \"\"\"Install monitor on all executors\"\"\"\n for exe in self.execs:\n mon.install(exe)\n" }, { "path": "dff_rfcn/core/__init__.py", "content": "" }, { "path": "dff_rfcn/core/callback.py", "content": "# --------------------------------------------------------\n# Deep Feature Flow\n# Copyright (c) 2017 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Modified by Yuwen Xiong\n# --------------------------------------------------------\n# Based on:\n# MX-RCNN\n# Copyright (c) 2016 by Contributors\n# Licence under The Apache 2.0 License\n# https://github.com/ijkguo/mx-rcnn/\n# --------------------------------------------------------\n\nimport time\nimport logging\nimport mxnet as mx\n\n\nclass Speedometer(object):\n def __init__(self, batch_size, frequent=50):\n self.batch_size = batch_size\n self.frequent = frequent\n self.init = False\n self.tic = 0\n self.last_count = 0\n\n def __call__(self, param):\n \"\"\"Callback to Show speed.\"\"\"\n count = param.nbatch\n if self.last_count > count:\n self.init = False\n self.last_count = count\n\n if self.init:\n if count % self.frequent == 0:\n speed = self.frequent * self.batch_size / (time.time() - self.tic)\n s = ''\n if param.eval_metric is not None:\n name, value = param.eval_metric.get()\n s = \"Epoch[%d] Batch [%d]\\tSpeed: %.2f samples/sec\\tTrain-\" % (param.epoch, count, speed)\n for n, v in zip(name, value):\n s += \"%s=%f,\\t\" % (n, v)\n else:\n s = \"Iter[%d] Batch [%d]\\tSpeed: %.2f samples/sec\" % (param.epoch, count, speed)\n\n logging.info(s)\n print(s)\n self.tic = time.time()\n else:\n self.init = True\n self.tic = time.time()\n\n\ndef do_checkpoint(prefix, means, stds):\n def _callback(iter_no, sym, arg, aux):\n weight = arg['rfcn_bbox_weight']\n bias = arg['rfcn_bbox_bias']\n repeat = bias.shape[0] / means.shape[0]\n\n arg['rfcn_bbox_weight_test'] = weight * mx.nd.repeat(mx.nd.array(stds), repeats=repeat).reshape((bias.shape[0], 1, 1, 1))\n arg['rfcn_bbox_bias_test'] = arg['rfcn_bbox_bias'] * mx.nd.repeat(mx.nd.array(stds), repeats=repeat) + mx.nd.repeat(mx.nd.array(means), repeats=repeat)\n mx.model.save_checkpoint(prefix, iter_no + 1, sym, arg, aux)\n arg.pop('rfcn_bbox_weight_test')\n arg.pop('rfcn_bbox_bias_test')\n return _callback" }, { "path": "dff_rfcn/core/loader.py", "content": "# --------------------------------------------------------\n# Deep Feature Flow\n# Copyright (c) 2017 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Modified by Xizhou Zhu, Yuwen Xiong\n# --------------------------------------------------------\n# Based on:\n# MX-RCNN\n# Copyright (c) 2016 by Contributors\n# Licence under The Apache 2.0 License\n# https://github.com/ijkguo/mx-rcnn/\n# --------------------------------------------------------\n\nimport numpy as np\nimport mxnet as mx\nfrom mxnet.executor_manager import _split_input_slice\n\nfrom config.config import config\nfrom utils.image import tensor_vstack\nfrom rpn.rpn import get_rpn_testbatch, get_rpn_pair_batch, assign_anchor\nfrom rcnn import get_rcnn_testbatch, get_rcnn_batch\n\nclass TestLoader(mx.io.DataIter):\n def __init__(self, roidb, config, batch_size=1, shuffle=False,\n has_rpn=False):\n super(TestLoader, self).__init__()\n\n # save parameters as properties\n self.cfg = config\n self.roidb = roidb\n self.batch_size = batch_size\n self.shuffle = shuffle\n self.has_rpn = has_rpn\n\n # infer properties from roidb\n self.size = np.sum([x['frame_seg_len'] for x in self.roidb])\n self.index = np.arange(self.size)\n\n # decide data and label names (only for training)\n self.data_name = ['data', 'im_info', 'data_key', 'feat_key']\n self.label_name = None\n\n #\n self.cur_roidb_index = 0\n self.cur_frameid = 0\n self.data_key = None\n self.key_frameid = 0\n self.cur_seg_len = 0\n self.key_frame_flag = -1\n\n # status variable for synchronization between get_data and get_label\n self.cur = 0\n self.data = None\n self.label = []\n self.im_info = None\n\n # get first batch to fill in provide_data and provide_label\n self.reset()\n self.get_batch()\n\n @property\n def provide_data(self):\n return [[(k, v.shape) for k, v in zip(self.data_name, idata)] for idata in self.data]\n\n @property\n def provide_label(self):\n return [None for _ in range(len(self.data))]\n\n @property\n def provide_data_single(self):\n return [(k, v.shape) for k, v in zip(self.data_name, self.data[0])]\n\n @property\n def provide_label_single(self):\n return None\n\n def reset(self):\n self.cur = 0\n if self.shuffle:\n np.random.shuffle(self.index)\n\n def iter_next(self):\n return self.cur < self.size\n\n def next(self):\n if self.iter_next():\n self.get_batch()\n self.cur += self.batch_size\n self.cur_frameid += 1\n if self.cur_frameid == self.cur_seg_len:\n self.cur_roidb_index += 1\n self.cur_frameid = 0\n self.key_frameid = 0\n elif self.cur_frameid - self.key_frameid == self.cfg.TEST.KEY_FRAME_INTERVAL:\n self.key_frameid = self.cur_frameid\n return self.im_info, self.key_frame_flag, mx.io.DataBatch(data=self.data, label=self.label,\n pad=self.getpad(), index=self.getindex(),\n provide_data=self.provide_data, provide_label=self.provide_label)\n else:\n raise StopIteration\n\n def getindex(self):\n return self.cur / self.batch_size\n\n def getpad(self):\n if self.cur + self.batch_size > self.size:\n return self.cur + self.batch_size - self.size\n else:\n return 0\n\n def get_batch(self):\n cur_roidb = self.roidb[self.cur_roidb_index].copy()\n cur_roidb['image'] = cur_roidb['pattern'] % self.cur_frameid\n self.cur_seg_len = cur_roidb['frame_seg_len']\n data, label, im_info = get_rpn_testbatch([cur_roidb], self.cfg)\n if self.key_frameid == self.cur_frameid: # key frame\n self.data_key = data[0]['data'].copy()\n if self.key_frameid == 0:\n self.key_frame_flag = 0\n else:\n self.key_frame_flag = 1\n else:\n self.key_frame_flag = 2\n extend_data = [{'data': data[0]['data'],\n 'im_info': data[0]['im_info'],\n 'data_key': self.data_key,\n 'feat_key': np.zeros((1,self.cfg.network.DFF_FEAT_DIM,1,1))}]\n self.data = [[mx.nd.array(extend_data[i][name]) for name in self.data_name] for i in xrange(len(data))]\n self.im_info = im_info\n\nclass AnchorLoader(mx.io.DataIter):\n\n def __init__(self, feat_sym, roidb, cfg, batch_size=1, shuffle=False, ctx=None, work_load_list=None,\n feat_stride=16, anchor_scales=(8, 16, 32), anchor_ratios=(0.5, 1, 2), allowed_border=0,\n aspect_grouping=False, normalize_target=False, bbox_mean=(0.0, 0.0, 0.0, 0.0),\n bbox_std=(0.1, 0.1, 0.4, 0.4)):\n \"\"\"\n This Iter will provide roi data to Fast R-CNN network\n :param feat_sym: to infer shape of assign_output\n :param roidb: must be preprocessed\n :param batch_size: must divide BATCH_SIZE(128)\n :param shuffle: bool\n :param ctx: list of contexts\n :param work_load_list: list of work load\n :param aspect_grouping: group images with similar aspects\n :param normalize_target: normalize rpn target\n :param bbox_mean: anchor target mean\n :param bbox_std: anchor target std\n :return: AnchorLoader\n \"\"\"\n super(AnchorLoader, self).__init__()\n\n # save parameters as properties\n self.feat_sym = feat_sym\n self.roidb = roidb\n self.cfg = cfg\n self.batch_size = batch_size\n self.shuffle = shuffle\n self.ctx = ctx\n if self.ctx is None:\n self.ctx = [mx.cpu()]\n self.work_load_list = work_load_list\n self.feat_stride = feat_stride\n self.anchor_scales = anchor_scales\n self.anchor_ratios = anchor_ratios\n self.allowed_border = allowed_border\n self.aspect_grouping = aspect_grouping\n self.normalize_target = normalize_target\n self.bbox_mean = bbox_mean\n self.bbox_std = bbox_std\n\n # infer properties from roidb\n self.size = len(roidb)\n self.index = np.arange(self.size)\n\n # decide data and label names\n if config.TRAIN.END2END:\n self.data_name = ['data', 'data_ref', 'eq_flag', 'im_info', 'gt_boxes']\n else:\n self.data_name = ['data']\n self.label_name = ['label', 'bbox_target', 'bbox_weight']\n\n # status variable for synchronization between get_data and get_label\n self.cur = 0\n self.batch = None\n self.data = None\n self.label = None\n\n # get first batch to fill in provide_data and provide_label\n self.reset()\n self.get_batch_individual()\n\n @property\n def provide_data(self):\n return [[(k, v.shape) for k, v in zip(self.data_name, self.data[i])] for i in xrange(len(self.data))]\n\n @property\n def provide_label(self):\n return [[(k, v.shape) for k, v in zip(self.label_name, self.label[i])] for i in xrange(len(self.data))]\n\n @property\n def provide_data_single(self):\n return [(k, v.shape) for k, v in zip(self.data_name, self.data[0])]\n\n @property\n def provide_label_single(self):\n return [(k, v.shape) for k, v in zip(self.label_name, self.label[0])]\n\n def reset(self):\n self.cur = 0\n if self.shuffle:\n if self.aspect_grouping:\n widths = np.array([r['width'] for r in self.roidb])\n heights = np.array([r['height'] for r in self.roidb])\n horz = (widths >= heights)\n vert = np.logical_not(horz)\n horz_inds = np.where(horz)[0]\n vert_inds = np.where(vert)[0]\n inds = np.hstack((np.random.permutation(horz_inds), np.random.permutation(vert_inds)))\n extra = inds.shape[0] % self.batch_size\n inds_ = np.reshape(inds[:-extra], (-1, self.batch_size))\n row_perm = np.random.permutation(np.arange(inds_.shape[0]))\n inds[:-extra] = np.reshape(inds_[row_perm, :], (-1,))\n self.index = inds\n else:\n np.random.shuffle(self.index)\n\n def iter_next(self):\n return self.cur + self.batch_size <= self.size\n\n def next(self):\n if self.iter_next():\n self.get_batch_individual()\n self.cur += self.batch_size\n return mx.io.DataBatch(data=self.data, label=self.label,\n pad=self.getpad(), index=self.getindex(),\n provide_data=self.provide_data, provide_label=self.provide_label)\n else:\n raise StopIteration\n\n def getindex(self):\n return self.cur / self.batch_size\n\n def getpad(self):\n if self.cur + self.batch_size > self.size:\n return self.cur + self.batch_size - self.size\n else:\n return 0\n\n def infer_shape(self, max_data_shape=None, max_label_shape=None):\n \"\"\" Return maximum data and label shape for single gpu \"\"\"\n if max_data_shape is None:\n max_data_shape = []\n if max_label_shape is None:\n max_label_shape = []\n max_shapes = dict(max_data_shape + max_label_shape)\n input_batch_size = max_shapes['data'][0]\n im_info = [[max_shapes['data'][2], max_shapes['data'][3], 1.0]]\n _, feat_shape, _ = self.feat_sym.infer_shape(**max_shapes)\n label = assign_anchor(feat_shape[0], np.zeros((0, 5)), im_info, self.cfg,\n self.feat_stride, self.anchor_scales, self.anchor_ratios, self.allowed_border,\n self.normalize_target, self.bbox_mean, self.bbox_std)\n label = [label[k] for k in self.label_name]\n label_shape = [(k, tuple([input_batch_size] + list(v.shape[1:]))) for k, v in zip(self.label_name, label)]\n return max_data_shape, label_shape\n\n def get_batch(self):\n # slice roidb\n cur_from = self.cur\n cur_to = min(cur_from + self.batch_size, self.size)\n roidb = [self.roidb[self.index[i]] for i in range(cur_from, cur_to)]\n\n # decide multi device slice\n work_load_list = self.work_load_list\n ctx = self.ctx\n if work_load_list is None:\n work_load_list = [1] * len(ctx)\n assert isinstance(work_load_list, list) and len(work_load_list) == len(ctx), \\\n \"Invalid settings for work load. \"\n slices = _split_input_slice(self.batch_size, work_load_list)\n\n # get testing data for multigpu\n data_list = []\n label_list = []\n for islice in slices:\n iroidb = [roidb[i] for i in range(islice.start, islice.stop)]\n data, label = get_rpn_pair_batch(iroidb, self.cfg)\n data_list.append(data)\n label_list.append(label)\n\n # pad data first and then assign anchor (read label)\n data_tensor = tensor_vstack([batch['data'] for batch in data_list])\n for data, data_pad in zip(data_list, data_tensor):\n data['data'] = data_pad[np.newaxis, :]\n\n new_label_list = []\n for data, label in zip(data_list, label_list):\n # infer label shape\n data_shape = {k: v.shape for k, v in data.items()}\n del data_shape['im_info']\n _, feat_shape, _ = self.feat_sym.infer_shape(**data_shape)\n feat_shape = [int(i) for i in feat_shape[0]]\n\n # add gt_boxes to data for e2e\n data['gt_boxes'] = label['gt_boxes'][np.newaxis, :, :]\n\n # assign anchor for label\n label = assign_anchor(feat_shape, label['gt_boxes'], data['im_info'], self.cfg,\n self.feat_stride, self.anchor_scales,\n self.anchor_ratios, self.allowed_border,\n self.normalize_target, self.bbox_mean, self.bbox_std)\n new_label_list.append(label)\n\n all_data = dict()\n for key in self.data_name:\n all_data[key] = tensor_vstack([batch[key] for batch in data_list])\n\n all_label = dict()\n for key in self.label_name:\n pad = -1 if key == 'label' else 0\n all_label[key] = tensor_vstack([batch[key] for batch in new_label_list], pad=pad)\n\n self.data = [mx.nd.array(all_data[key]) for key in self.data_name]\n self.label = [mx.nd.array(all_label[key]) for key in self.label_name]\n\n def get_batch_individual(self):\n cur_from = self.cur\n cur_to = min(cur_from + self.batch_size, self.size)\n roidb = [self.roidb[self.index[i]] for i in range(cur_from, cur_to)]\n # decide multi device slice\n work_load_list = self.work_load_list\n ctx = self.ctx\n if work_load_list is None:\n work_load_list = [1] * len(ctx)\n assert isinstance(work_load_list, list) and len(work_load_list) == len(ctx), \\\n \"Invalid settings for work load. \"\n slices = _split_input_slice(self.batch_size, work_load_list)\n rst = []\n for idx, islice in enumerate(slices):\n iroidb = [roidb[i] for i in range(islice.start, islice.stop)]\n rst.append(self.parfetch(iroidb))\n all_data = [_['data'] for _ in rst]\n all_label = [_['label'] for _ in rst]\n self.data = [[mx.nd.array(data[key]) for key in self.data_name] for data in all_data]\n self.label = [[mx.nd.array(label[key]) for key in self.label_name] for label in all_label]\n\n def parfetch(self, iroidb):\n # get testing data for multigpu\n data, label = get_rpn_pair_batch(iroidb, self.cfg)\n data_shape = {k: v.shape for k, v in data.items()}\n del data_shape['im_info']\n _, feat_shape, _ = self.feat_sym.infer_shape(**data_shape)\n feat_shape = [int(i) for i in feat_shape[0]]\n\n # add gt_boxes to data for e2e\n data['gt_boxes'] = label['gt_boxes'][np.newaxis, :, :]\n\n # assign anchor for label\n label = assign_anchor(feat_shape, label['gt_boxes'], data['im_info'], self.cfg,\n self.feat_stride, self.anchor_scales,\n self.anchor_ratios, self.allowed_border,\n self.normalize_target, self.bbox_mean, self.bbox_std)\n return {'data': data, 'label': label}\n\n" }, { "path": "dff_rfcn/core/metric.py", "content": "# --------------------------------------------------------\n# Deep Feature Flow\n# Copyright (c) 2017 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Modified by Yuwen Xiong\n# --------------------------------------------------------\n# Based on:\n# MX-RCNN\n# Copyright (c) 2016 by Contributors\n# Licence under The Apache 2.0 License\n# https://github.com/ijkguo/mx-rcnn/\n# --------------------------------------------------------\n\nimport mxnet as mx\nimport numpy as np\n\n\ndef get_rpn_names():\n pred = ['rpn_cls_prob', 'rpn_bbox_loss']\n label = ['rpn_label', 'rpn_bbox_target', 'rpn_bbox_weight']\n return pred, label\n\n\ndef get_rcnn_names(cfg):\n pred = ['rcnn_cls_prob', 'rcnn_bbox_loss']\n label = ['rcnn_label', 'rcnn_bbox_target', 'rcnn_bbox_weight']\n if cfg.TRAIN.ENABLE_OHEM or cfg.TRAIN.END2END:\n pred.append('rcnn_label')\n if cfg.TRAIN.END2END:\n rpn_pred, rpn_label = get_rpn_names()\n pred = rpn_pred + pred\n label = rpn_label\n return pred, label\n\n\nclass RPNAccMetric(mx.metric.EvalMetric):\n def __init__(self):\n super(RPNAccMetric, self).__init__('RPNAcc')\n self.pred, self.label = get_rpn_names()\n\n def update(self, labels, preds):\n pred = preds[self.pred.index('rpn_cls_prob')]\n label = labels[self.label.index('rpn_label')]\n\n # pred (b, c, p) or (b, c, h, w)\n pred_label = mx.ndarray.argmax_channel(pred).asnumpy().astype('int32')\n pred_label = pred_label.reshape((pred_label.shape[0], -1))\n # label (b, p)\n label = label.asnumpy().astype('int32')\n\n # filter with keep_inds\n keep_inds = np.where(label != -1)\n pred_label = pred_label[keep_inds]\n label = label[keep_inds]\n\n self.sum_metric += np.sum(pred_label.flat == label.flat)\n self.num_inst += len(pred_label.flat)\n\n\nclass RCNNAccMetric(mx.metric.EvalMetric):\n def __init__(self, cfg):\n super(RCNNAccMetric, self).__init__('RCNNAcc')\n self.e2e = cfg.TRAIN.END2END\n self.ohem = cfg.TRAIN.ENABLE_OHEM\n self.pred, self.label = get_rcnn_names(cfg)\n\n def update(self, labels, preds):\n pred = preds[self.pred.index('rcnn_cls_prob')]\n if self.ohem or self.e2e:\n label = preds[self.pred.index('rcnn_label')]\n else:\n label = labels[self.label.index('rcnn_label')]\n\n last_dim = pred.shape[-1]\n pred_label = pred.asnumpy().reshape(-1, last_dim).argmax(axis=1).astype('int32')\n label = label.asnumpy().reshape(-1,).astype('int32')\n\n # filter with keep_inds\n keep_inds = np.where(label != -1)\n pred_label = pred_label[keep_inds]\n label = label[keep_inds]\n\n self.sum_metric += np.sum(pred_label.flat == label.flat)\n self.num_inst += len(pred_label.flat)\n\n\nclass RPNLogLossMetric(mx.metric.EvalMetric):\n def __init__(self):\n super(RPNLogLossMetric, self).__init__('RPNLogLoss')\n self.pred, self.label = get_rpn_names()\n\n def update(self, labels, preds):\n pred = preds[self.pred.index('rpn_cls_prob')]\n label = labels[self.label.index('rpn_label')]\n\n # label (b, p)\n label = label.asnumpy().astype('int32').reshape((-1))\n # pred (b, c, p) or (b, c, h, w) --> (b, p, c) --> (b*p, c)\n pred = pred.asnumpy().reshape((pred.shape[0], pred.shape[1], -1)).transpose((0, 2, 1))\n pred = pred.reshape((label.shape[0], -1))\n\n # filter with keep_inds\n keep_inds = np.where(label != -1)[0]\n label = label[keep_inds]\n cls = pred[keep_inds, label]\n\n cls += 1e-14\n cls_loss = -1 * np.log(cls)\n cls_loss = np.sum(cls_loss)\n self.sum_metric += cls_loss\n self.num_inst += label.shape[0]\n\n\nclass RCNNLogLossMetric(mx.metric.EvalMetric):\n def __init__(self, cfg):\n super(RCNNLogLossMetric, self).__init__('RCNNLogLoss')\n self.e2e = cfg.TRAIN.END2END\n self.ohem = cfg.TRAIN.ENABLE_OHEM\n self.pred, self.label = get_rcnn_names(cfg)\n\n def update(self, labels, preds):\n pred = preds[self.pred.index('rcnn_cls_prob')]\n if self.ohem or self.e2e:\n label = preds[self.pred.index('rcnn_label')]\n else:\n label = labels[self.label.index('rcnn_label')]\n\n last_dim = pred.shape[-1]\n pred = pred.asnumpy().reshape(-1, last_dim)\n label = label.asnumpy().reshape(-1,).astype('int32')\n\n # filter with keep_inds\n keep_inds = np.where(label != -1)[0]\n label = label[keep_inds]\n cls = pred[keep_inds, label]\n\n cls += 1e-14\n cls_loss = -1 * np.log(cls)\n cls_loss = np.sum(cls_loss)\n self.sum_metric += cls_loss\n self.num_inst += label.shape[0]\n\n\nclass RPNL1LossMetric(mx.metric.EvalMetric):\n def __init__(self):\n super(RPNL1LossMetric, self).__init__('RPNL1Loss')\n self.pred, self.label = get_rpn_names()\n\n def update(self, labels, preds):\n bbox_loss = preds[self.pred.index('rpn_bbox_loss')].asnumpy()\n\n # calculate num_inst (average on those kept anchors)\n label = labels[self.label.index('rpn_label')].asnumpy()\n num_inst = np.sum(label != -1)\n\n self.sum_metric += np.sum(bbox_loss)\n self.num_inst += num_inst\n\n\nclass RCNNL1LossMetric(mx.metric.EvalMetric):\n def __init__(self, cfg):\n super(RCNNL1LossMetric, self).__init__('RCNNL1Loss')\n self.e2e = cfg.TRAIN.END2END\n self.ohem = cfg.TRAIN.ENABLE_OHEM\n self.pred, self.label = get_rcnn_names(cfg)\n\n def update(self, labels, preds):\n bbox_loss = preds[self.pred.index('rcnn_bbox_loss')].asnumpy()\n if self.ohem:\n label = preds[self.pred.index('rcnn_label')].asnumpy()\n else:\n if self.e2e:\n label = preds[self.pred.index('rcnn_label')].asnumpy()\n else:\n label = labels[self.label.index('rcnn_label')].asnumpy()\n\n # calculate num_inst (average on those kept anchors)\n num_inst = np.sum(label != -1)\n\n self.sum_metric += np.sum(bbox_loss)\n self.num_inst += num_inst\n" }, { "path": "dff_rfcn/core/module.py", "content": "# --------------------------------------------------------\n# Deep Feature Flow\n# Copyright (c) 2017 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Modified by Yuwen Xiong\n# --------------------------------------------------------\n# Based on:\n# MX-RCNN\n# Copyright (c) 2016 by Contributors\n# Licence under The Apache 2.0 License\n# https://github.com/ijkguo/mx-rcnn/\n# --------------------------------------------------------\n\n\n\"\"\"A `MutableModule` implement the `BaseModule` API, and allows input shape\nvarying with training iterations. If shapes vary, executors will rebind,\nusing shared arrays from the initial module binded with maximum shape.\n\"\"\"\n\nimport time\nimport logging\nimport warnings\n\nfrom mxnet import context as ctx\nfrom mxnet.initializer import Uniform, InitDesc\nfrom mxnet.module.base_module import BaseModule, _check_input_names, _parse_data_desc, _as_list\nfrom mxnet.model import _create_kvstore, _initialize_kvstore, _update_params, _update_params_on_kvstore, load_checkpoint, BatchEndParam\nfrom mxnet import metric\n\nfrom .DataParallelExecutorGroup import DataParallelExecutorGroup\nfrom mxnet import ndarray as nd\nfrom mxnet import optimizer as opt\n\n\nclass Module(BaseModule):\n \"\"\"Module is a basic module that wrap a `Symbol`. It is functionally the same\n as the `FeedForward` model, except under the module API.\n\n Parameters\n ----------\n symbol : Symbol\n data_names : list of str\n Default is `('data')` for a typical model used in image classification.\n label_names : list of str\n Default is `('softmax_label')` for a typical model used in image\n classification.\n logger : Logger\n Default is `logging`.\n context : Context or list of Context\n Default is `cpu()`.\n work_load_list : list of number\n Default `None`, indicating uniform workload.\n fixed_param_names: list of str\n Default `None`, indicating no network parameters are fixed.\n state_names : list of str\n states are similar to data and label, but not provided by data iterator.\n Instead they are initialized to 0 and can be set by set_states()\n \"\"\"\n def __init__(self, symbol, data_names=('data',), label_names=('softmax_label',),\n logger=logging, context=ctx.cpu(), work_load_list=None,\n fixed_param_names=None, state_names=None):\n super(Module, self).__init__(logger=logger)\n\n if isinstance(context, ctx.Context):\n context = [context]\n self._context = context\n if work_load_list is None:\n work_load_list = [1] * len(self._context)\n assert len(work_load_list) == len(self._context)\n self._work_load_list = work_load_list\n\n self._symbol = symbol\n\n data_names = list(data_names) if data_names is not None else []\n label_names = list(label_names) if label_names is not None else []\n state_names = list(state_names) if state_names is not None else []\n fixed_param_names = list(fixed_param_names) if fixed_param_names is not None else []\n\n _check_input_names(symbol, data_names, \"data\", True)\n _check_input_names(symbol, label_names, \"label\", False)\n _check_input_names(symbol, state_names, \"state\", True)\n _check_input_names(symbol, fixed_param_names, \"fixed_param\", True)\n\n arg_names = symbol.list_arguments()\n input_names = data_names + label_names + state_names\n self._param_names = [x for x in arg_names if x not in input_names]\n self._fixed_param_names = fixed_param_names\n self._aux_names = symbol.list_auxiliary_states()\n self._data_names = data_names\n self._label_names = label_names\n self._state_names = state_names\n self._output_names = symbol.list_outputs()\n\n self._arg_params = None\n self._aux_params = None\n self._params_dirty = False\n\n self._optimizer = None\n self._kvstore = None\n self._update_on_kvstore = None\n self._updater = None\n self._preload_opt_states = None\n self._grad_req = None\n\n self._exec_group = None\n self._data_shapes = None\n self._label_shapes = None\n\n @staticmethod\n def load(prefix, epoch, load_optimizer_states=False, **kwargs):\n \"\"\"Create a model from previously saved checkpoint.\n\n Parameters\n ----------\n prefix : str\n path prefix of saved model files. You should have\n \"prefix-symbol.json\", \"prefix-xxxx.params\", and\n optionally \"prefix-xxxx.states\", where xxxx is the\n epoch number.\n epoch : int\n epoch to load.\n load_optimizer_states : bool\n whether to load optimizer states. Checkpoint needs\n to have been made with save_optimizer_states=True.\n data_names : list of str\n Default is `('data')` for a typical model used in image classification.\n label_names : list of str\n Default is `('softmax_label')` for a typical model used in image\n classification.\n logger : Logger\n Default is `logging`.\n context : Context or list of Context\n Default is `cpu()`.\n work_load_list : list of number\n Default `None`, indicating uniform workload.\n fixed_param_names: list of str\n Default `None`, indicating no network parameters are fixed.\n \"\"\"\n sym, args, auxs = load_checkpoint(prefix, epoch)\n mod = Module(symbol=sym, **kwargs)\n mod._arg_params = args\n mod._aux_params = auxs\n mod.params_initialized = True\n if load_optimizer_states:\n mod._preload_opt_states = '%s-%04d.states'%(prefix, epoch)\n return mod\n\n def save_checkpoint(self, prefix, epoch, save_optimizer_states=False):\n \"\"\"Save current progress to checkpoint.\n Use mx.callback.module_checkpoint as epoch_end_callback to save during training.\n\n Parameters\n ----------\n prefix : str\n The file prefix to checkpoint to\n epoch : int\n The current epoch number\n save_optimizer_states : bool\n Whether to save optimizer states for continue training\n \"\"\"\n self._symbol.save('%s-symbol.json'%prefix)\n param_name = '%s-%04d.params' % (prefix, epoch)\n self.save_params(param_name)\n logging.info('Saved checkpoint to \\\"%s\\\"', param_name)\n if save_optimizer_states:\n state_name = '%s-%04d.states' % (prefix, epoch)\n self.save_optimizer_states(state_name)\n logging.info('Saved optimizer state to \\\"%s\\\"', state_name)\n\n def _reset_bind(self):\n \"\"\"Internal function to reset binded state.\"\"\"\n self.binded = False\n self._exec_group = None\n self._data_shapes = None\n self._label_shapes = None\n\n @property\n def data_names(self):\n \"\"\"A list of names for data required by this module.\"\"\"\n return self._data_names\n\n @property\n def label_names(self):\n \"\"\"A list of names for labels required by this module.\"\"\"\n return self._label_names\n\n @property\n def output_names(self):\n \"\"\"A list of names for the outputs of this module.\"\"\"\n return self._output_names\n\n @property\n def data_shapes(self):\n \"\"\"Get data shapes.\n Returns\n -------\n A list of `(name, shape)` pairs.\n \"\"\"\n assert self.binded\n return self._data_shapes\n\n @property\n def label_shapes(self):\n \"\"\"Get label shapes.\n Returns\n -------\n A list of `(name, shape)` pairs. The return value could be `None` if\n the module does not need labels, or if the module is not binded for\n training (in this case, label information is not available).\n \"\"\"\n assert self.binded\n return self._label_shapes\n\n @property\n def output_shapes(self):\n \"\"\"Get output shapes.\n Returns\n -------\n A list of `(name, shape)` pairs.\n \"\"\"\n assert self.binded\n return self._exec_group.get_output_shapes()\n\n def get_params(self):\n \"\"\"Get current parameters.\n Returns\n -------\n `(arg_params, aux_params)`, each a dictionary of name to parameters (in\n `NDArray`) mapping.\n \"\"\"\n assert self.binded and self.params_initialized\n\n if self._params_dirty:\n self._sync_params_from_devices()\n return (self._arg_params, self._aux_params)\n\n def init_params(self, initializer=Uniform(0.01), arg_params=None, aux_params=None,\n allow_missing=False, force_init=False):\n \"\"\"Initialize the parameters and auxiliary states.\n\n Parameters\n ----------\n initializer : Initializer\n Called to initialize parameters if needed.\n arg_params : dict\n If not None, should be a dictionary of existing arg_params. Initialization\n will be copied from that.\n aux_params : dict\n If not None, should be a dictionary of existing aux_params. Initialization\n will be copied from that.\n allow_missing : bool\n If true, params could contain missing values, and the initializer will be\n called to fill those missing params.\n force_init : bool\n If true, will force re-initialize even if already initialized.\n \"\"\"\n if self.params_initialized and not force_init:\n warnings.warn(\"Parameters already initialized and force_init=False. \"\n \"init_params call ignored.\", stacklevel=2)\n return\n assert self.binded, 'call bind before initializing the parameters'\n\n def _impl(name, arr, cache):\n \"\"\"Internal helper for parameter initialization\"\"\"\n if cache is not None:\n if name in cache:\n cache_arr = cache[name]\n\n # just in case the cached array is just the target itself\n if cache_arr is not arr:\n cache_arr.copyto(arr)\n else:\n if not allow_missing:\n raise RuntimeError(\"%s is not presented\" % name)\n if initializer != None:\n initializer(name, arr)\n else:\n initializer(name, arr)\n\n attrs = self._symbol.attr_dict()\n for name, arr in self._arg_params.items():\n desc = InitDesc(name, attrs.get(name, None))\n _impl(desc, arr, arg_params)\n\n for name, arr in self._aux_params.items():\n desc = InitDesc(name, attrs.get(name, None))\n _impl(desc, arr, aux_params)\n\n self.params_initialized = True\n self._params_dirty = False\n\n # copy the initialized parameters to devices\n self._exec_group.set_params(self._arg_params, self._aux_params)\n\n def set_params(self, arg_params, aux_params, allow_missing=False, force_init=True):\n \"\"\"Assign parameter and aux state values.\n\n Parameters\n ----------\n arg_params : dict\n Dictionary of name to value (`NDArray`) mapping.\n aux_params : dict\n Dictionary of name to value (`NDArray`) mapping.\n allow_missing : bool\n If true, params could contain missing values, and the initializer will be\n called to fill those missing params.\n force_init : bool\n If true, will force re-initialize even if already initialized.\n\n Examples\n --------\n An example of setting module parameters::\n >>> sym, arg_params, aux_params = \\\n >>> mx.model.load_checkpoint(model_prefix, n_epoch_load)\n >>> mod.set_params(arg_params=arg_params, aux_params=aux_params)\n \"\"\"\n if not allow_missing:\n self.init_params(initializer=None, arg_params=arg_params, aux_params=aux_params,\n allow_missing=allow_missing, force_init=force_init)\n return\n\n if self.params_initialized and not force_init:\n warnings.warn(\"Parameters already initialized and force_init=False. \"\n \"set_params call ignored.\", stacklevel=2)\n return\n\n self._exec_group.set_params(arg_params, aux_params)\n\n # because we didn't update self._arg_params, they are dirty now.\n self._params_dirty = True\n self.params_initialized = True\n\n def bind(self, data_shapes, label_shapes=None, for_training=True,\n inputs_need_grad=False, force_rebind=False, shared_module=None,\n grad_req='write'):\n \"\"\"Bind the symbols to construct executors. This is necessary before one\n can perform computation with the module.\n\n Parameters\n ----------\n data_shapes : list of (str, tuple)\n Typically is `data_iter.provide_data`.\n label_shapes : list of (str, tuple)\n Typically is `data_iter.provide_label`.\n for_training : bool\n Default is `True`. Whether the executors should be bind for training.\n inputs_need_grad : bool\n Default is `False`. Whether the gradients to the input data need to be computed.\n Typically this is not needed. But this might be needed when implementing composition\n of modules.\n force_rebind : bool\n Default is `False`. This function does nothing if the executors are already\n binded. But with this `True`, the executors will be forced to rebind.\n shared_module : Module\n Default is `None`. This is used in bucketing. When not `None`, the shared module\n essentially corresponds to a different bucket -- a module with different symbol\n but with the same sets of parameters (e.g. unrolled RNNs with different lengths).\n \"\"\"\n # force rebinding is typically used when one want to switch from\n # training to prediction phase.\n if force_rebind:\n self._reset_bind()\n\n if self.binded:\n self.logger.warning('Already binded, ignoring bind()')\n return\n\n self.for_training = for_training\n self.inputs_need_grad = inputs_need_grad\n self.binded = True\n self._grad_req = grad_req\n\n if not for_training:\n assert not inputs_need_grad\n else:\n pass\n # this is not True, as some module might not contains a loss function\n # that consumes the labels\n # assert label_shapes is not None\n\n # self._data_shapes, self._label_shapes = _parse_data_desc(\n # self.data_names, self.label_names, data_shapes, label_shapes)\n self._data_shapes, self._label_shapes = zip(*[_parse_data_desc(self.data_names, self.label_names, data_shape, label_shape)\n for data_shape, label_shape in zip(data_shapes, label_shapes)])\n if self._label_shapes.count(None) == len(self._label_shapes):\n self._label_shapes = None\n\n if shared_module is not None:\n assert isinstance(shared_module, Module) and \\\n shared_module.binded and shared_module.params_initialized\n shared_group = shared_module._exec_group\n else:\n shared_group = None\n\n self._exec_group = DataParallelExecutorGroup(self._symbol, self._context,\n self._work_load_list, self._data_shapes,\n self._label_shapes, self._param_names,\n for_training, inputs_need_grad,\n shared_group, logger=self.logger,\n fixed_param_names=self._fixed_param_names,\n grad_req=grad_req,\n state_names=self._state_names)\n # self._total_exec_bytes = self._exec_group._total_exec_bytes\n if shared_module is not None:\n self.params_initialized = True\n self._arg_params = shared_module._arg_params\n self._aux_params = shared_module._aux_params\n elif self.params_initialized:\n # if the parameters are already initialized, we are re-binding\n # so automatically copy the already initialized params\n self._exec_group.set_params(self._arg_params, self._aux_params)\n else:\n assert self._arg_params is None and self._aux_params is None\n param_arrays = [\n nd.zeros(x[0].shape, dtype=x[0].dtype)\n for x in self._exec_group.param_arrays\n ]\n self._arg_params = {name:arr for name, arr in zip(self._param_names, param_arrays)}\n\n aux_arrays = [\n nd.zeros(x[0].shape, dtype=x[0].dtype)\n for x in self._exec_group.aux_arrays\n ]\n self._aux_params = {name:arr for name, arr in zip(self._aux_names, aux_arrays)}\n\n if shared_module is not None and shared_module.optimizer_initialized:\n self.borrow_optimizer(shared_module)\n\n\n def reshape(self, data_shapes, label_shapes=None):\n \"\"\"Reshape the module for new input shapes.\n\n Parameters\n ----------\n data_shapes : list of (str, tuple)\n Typically is `data_iter.provide_data`.\n label_shapes : list of (str, tuple)\n Typically is `data_iter.provide_label`.\n \"\"\"\n assert self.binded\n # self._data_shapes, self._label_shapes = _parse_data_desc(\n # self.data_names, self.label_names, data_shapes, label_shapes)\n self._data_shapes, self._label_shapes = zip(*[_parse_data_desc(self.data_names, self.label_names, data_shape, label_shape)\n for data_shape, label_shape in zip(data_shapes, label_shapes)])\n\n self._exec_group.reshape(self._data_shapes, self._label_shapes)\n\n\n def init_optimizer(self, kvstore='local', optimizer='sgd',\n optimizer_params=(('learning_rate', 0.01),), force_init=False):\n \"\"\"Install and initialize optimizers.\n\n Parameters\n ----------\n kvstore : str or KVStore\n Default `'local'`.\n optimizer : str or Optimizer\n Default `'sgd'`\n optimizer_params : dict\n Default `(('learning_rate', 0.01),)`. The default value is not a dictionary,\n just to avoid pylint warning of dangerous default values.\n force_init : bool\n Default `False`, indicating whether we should force re-initializing the\n optimizer in the case an optimizer is already installed.\n \"\"\"\n assert self.binded and self.params_initialized\n\n if self.optimizer_initialized and not force_init:\n self.logger.warning('optimizer already initialized, ignoring...')\n return\n\n (kvstore, update_on_kvstore) = \\\n _create_kvstore(kvstore, len(self._context), self._arg_params)\n\n batch_size = self._exec_group.batch_size\n if kvstore and 'dist' in kvstore.type and '_sync' in kvstore.type:\n batch_size *= kvstore.num_workers\n rescale_grad = 1.0/batch_size\n\n if isinstance(optimizer, str):\n idx2name = {}\n if update_on_kvstore:\n idx2name.update(enumerate(self._exec_group.param_names))\n else:\n for k in range(len(self._context)):\n idx2name.update({i*len(self._context)+k: n\n for i, n in enumerate(self._exec_group.param_names)})\n optimizer_params = dict(optimizer_params)\n if 'rescale_grad' not in optimizer_params:\n optimizer_params['rescale_grad'] = rescale_grad\n optimizer = opt.create(optimizer,\n sym=self.symbol, param_idx2name=idx2name,\n **optimizer_params)\n else:\n assert isinstance(optimizer, opt.Optimizer)\n if optimizer.rescale_grad != rescale_grad:\n #pylint: disable=no-member\n warnings.warn(\n \"Optimizer created manually outside Module but rescale_grad \" +\n \"is not normalized to 1.0/batch_size/num_workers (%s vs. %s). \"%(\n optimizer.rescale_grad, rescale_grad) +\n \"Is this intended?\", stacklevel=2)\n\n self._optimizer = optimizer\n self._kvstore = kvstore\n self._update_on_kvstore = update_on_kvstore\n self._updater = None\n\n if kvstore:\n # copy initialized local parameters to kvstore\n _initialize_kvstore(kvstore=kvstore,\n param_arrays=self._exec_group.param_arrays,\n arg_params=self._arg_params,\n param_names=self._param_names,\n update_on_kvstore=update_on_kvstore)\n if update_on_kvstore:\n kvstore.set_optimizer(self._optimizer)\n else:\n self._updater = opt.get_updater(optimizer)\n\n self.optimizer_initialized = True\n\n if self._preload_opt_states is not None:\n self.load_optimizer_states(self._preload_opt_states)\n self._preload_opt_states = None\n\n def borrow_optimizer(self, shared_module):\n \"\"\"Borrow optimizer from a shared module. Used in bucketing, where exactly the same\n optimizer (esp. kvstore) is used.\n\n Parameters\n ----------\n shared_module : Module\n \"\"\"\n assert shared_module.optimizer_initialized\n self._optimizer = shared_module._optimizer\n self._kvstore = shared_module._kvstore\n self._update_on_kvstore = shared_module._update_on_kvstore\n self._updater = shared_module._updater\n self.optimizer_initialized = True\n\n def forward(self, data_batch, is_train=None):\n \"\"\"Forward computation.\n\n Parameters\n ----------\n data_batch : DataBatch\n Could be anything with similar API implemented.\n is_train : bool\n Default is `None`, which means `is_train` takes the value of `self.for_training`.\n \"\"\"\n assert self.binded and self.params_initialized\n self._exec_group.forward(data_batch, is_train)\n\n def backward(self, out_grads=None):\n \"\"\"Backward computation.\n\n Parameters\n ----------\n out_grads : NDArray or list of NDArray, optional\n Gradient on the outputs to be propagated back.\n This parameter is only needed when bind is called\n on outputs that are not a loss function.\n \"\"\"\n assert self.binded and self.params_initialized\n self._exec_group.backward(out_grads=out_grads)\n\n def update(self):\n \"\"\"Update parameters according to the installed optimizer and the gradients computed\n in the previous forward-backward batch.\n \"\"\"\n assert self.binded and self.params_initialized and self.optimizer_initialized\n\n self._params_dirty = True\n if self._update_on_kvstore:\n _update_params_on_kvstore(self._exec_group.param_arrays,\n self._exec_group.grad_arrays,\n self._kvstore)\n else:\n _update_params(self._exec_group.param_arrays,\n self._exec_group.grad_arrays,\n updater=self._updater,\n num_device=len(self._context),\n kvstore=self._kvstore)\n\n def get_outputs(self, merge_multi_context=True):\n \"\"\"Get outputs of the previous forward computation.\n\n Parameters\n ----------\n merge_multi_context : bool\n Default is `True`. In the case when data-parallelism is used, the outputs\n will be collected from multiple devices. A `True` value indicate that we\n should merge the collected results so that they look like from a single\n executor.\n\n Returns\n -------\n If `merge_multi_context` is `True`, it is like `[out1, out2]`. Otherwise, it\n is like `[[out1_dev1, out1_dev2], [out2_dev1, out2_dev2]]`. All the output\n elements are `NDArray`.\n \"\"\"\n assert self.binded and self.params_initialized\n return self._exec_group.get_outputs(merge_multi_context=merge_multi_context)\n\n def get_input_grads(self, merge_multi_context=True):\n \"\"\"Get the gradients with respect to the inputs of the module.\n\n Parameters\n ----------\n merge_multi_context : bool\n Default is `True`. In the case when data-parallelism is used, the outputs\n will be collected from multiple devices. A `True` value indicate that we\n should merge the collected results so that they look like from a single\n executor.\n\n Returns\n -------\n If `merge_multi_context` is `True`, it is like `[grad1, grad2]`. Otherwise, it\n is like `[[grad1_dev1, grad1_dev2], [grad2_dev1, grad2_dev2]]`. All the output\n elements are `NDArray`.\n \"\"\"\n assert self.binded and self.params_initialized and self.inputs_need_grad\n return self._exec_group.get_input_grads(merge_multi_context=merge_multi_context)\n\n def get_states(self, merge_multi_context=True):\n \"\"\"Get states from all devices\n\n Parameters\n ----------\n merge_multi_context : bool\n Default is `True`. In the case when data-parallelism is used, the states\n will be collected from multiple devices. A `True` value indicate that we\n should merge the collected results so that they look like from a single\n executor.\n\n Returns\n -------\n If `merge_multi_context` is `True`, it is like `[out1, out2]`. Otherwise, it\n is like `[[out1_dev1, out1_dev2], [out2_dev1, out2_dev2]]`. All the output\n elements are `NDArray`.\n \"\"\"\n assert self.binded and self.params_initialized\n return self._exec_group.get_states(merge_multi_context=merge_multi_context)\n\n def set_states(self, states=None, value=None):\n \"\"\"Set value for states. Only one of states & value can be specified.\n\n Parameters\n ----------\n states : list of list of NDArrays\n source states arrays formatted like [[state1_dev1, state1_dev2],\n [state2_dev1, state2_dev2]].\n value : number\n a single scalar value for all state arrays.\n \"\"\"\n assert self.binded and self.params_initialized\n self._exec_group.set_states(states, value)\n\n def update_metric(self, eval_metric, labels):\n \"\"\"Evaluate and accumulate evaluation metric on outputs of the last forward computation.\n\n Parameters\n ----------\n eval_metric : EvalMetric\n labels : list of NDArray\n Typically `data_batch.label`.\n \"\"\"\n self._exec_group.update_metric(eval_metric, labels)\n\n def _sync_params_from_devices(self):\n \"\"\"Synchronize parameters from devices to CPU. This function should be called after\n calling `update` that updates the parameters on the devices, before one can read the\n latest parameters from `self._arg_params` and `self._aux_params`.\n \"\"\"\n self._exec_group.get_params(self._arg_params, self._aux_params)\n self._params_dirty = False\n\n def save_optimizer_states(self, fname):\n \"\"\"Save optimizer (updater) state to file\n\n Parameters\n ----------\n fname : str\n Path to output states file.\n \"\"\"\n assert self.optimizer_initialized\n\n if self._update_on_kvstore:\n self._kvstore.save_optimizer_states(fname)\n else:\n with open(fname, 'wb') as fout:\n fout.write(self._updater.get_states())\n\n def load_optimizer_states(self, fname):\n \"\"\"Load optimizer (updater) state from file\n\n Parameters\n ----------\n fname : str\n Path to input states file.\n \"\"\"\n assert self.optimizer_initialized\n\n if self._update_on_kvstore:\n self._kvstore.load_optimizer_states(fname)\n else:\n self._updater.set_states(open(fname, 'rb').read())\n\n def install_monitor(self, mon):\n \"\"\" Install monitor on all executors \"\"\"\n assert self.binded\n self._exec_group.install_monitor(mon)\n\n\nclass MutableModule(BaseModule):\n \"\"\"A mutable module is a module that supports variable input data.\n\n Parameters\n ----------\n symbol : Symbol\n data_names : list of str\n label_names : list of str\n logger : Logger\n context : Context or list of Context\n work_load_list : list of number\n max_data_shapes : list of (name, shape) tuple, designating inputs whose shape vary\n max_label_shapes : list of (name, shape) tuple, designating inputs whose shape vary\n fixed_param_prefix : list of str, indicating fixed parameters\n \"\"\"\n def __init__(self, symbol, data_names, label_names,\n logger=logging, context=ctx.cpu(), work_load_list=None,\n max_data_shapes=None, max_label_shapes=None, fixed_param_prefix=None):\n super(MutableModule, self).__init__(logger=logger)\n self._symbol = symbol\n self._data_names = data_names\n self._label_names = label_names\n self._context = context\n self._work_load_list = work_load_list\n\n self._curr_module = None\n self._max_data_shapes = max_data_shapes\n self._max_label_shapes = max_label_shapes\n self._fixed_param_prefix = fixed_param_prefix\n\n fixed_param_names = list()\n if fixed_param_prefix is not None:\n for name in self._symbol.list_arguments():\n for prefix in self._fixed_param_prefix:\n if name.startswith(prefix):\n fixed_param_names.append(name)\n self._fixed_param_names = fixed_param_names\n self._preload_opt_states = None\n\n def _reset_bind(self):\n self.binded = False\n self._curr_module = None\n\n @property\n def data_names(self):\n return self._data_names\n\n @property\n def output_names(self):\n return self._symbol.list_outputs()\n\n @property\n def data_shapes(self):\n assert self.binded\n return self._curr_module.data_shapes\n\n @property\n def label_shapes(self):\n assert self.binded\n return self._curr_module.label_shapes\n\n @property\n def output_shapes(self):\n assert self.binded\n return self._curr_module.output_shapes\n\n def get_params(self):\n assert self.binded and self.params_initialized\n return self._curr_module.get_params()\n\n def init_params(self, initializer=Uniform(0.01), arg_params=None, aux_params=None,\n allow_missing=False, force_init=False):\n if self.params_initialized and not force_init:\n return\n assert self.binded, 'call bind before initializing the parameters'\n self._curr_module.init_params(initializer=initializer, arg_params=arg_params,\n aux_params=aux_params, allow_missing=allow_missing,\n force_init=force_init)\n self.params_initialized = True\n\n def bind(self, data_shapes, label_shapes=None, for_training=True,\n inputs_need_grad=False, force_rebind=False, shared_module=None, grad_req='write'):\n # in case we already initialized params, keep it\n if self.params_initialized:\n arg_params, aux_params = self.get_params()\n\n # force rebinding is typically used when one want to switch from\n # training to prediction phase.\n if force_rebind:\n self._reset_bind()\n\n if self.binded:\n self.logger.warning('Already binded, ignoring bind()')\n return\n\n assert shared_module is None, 'shared_module for MutableModule is not supported'\n\n self.for_training = for_training\n self.inputs_need_grad = inputs_need_grad\n self.binded = True\n\n max_shapes_dict = dict()\n if self._max_data_shapes is not None:\n max_shapes_dict.update(dict(self._max_data_shapes[0]))\n if self._max_label_shapes is not None:\n max_shapes_dict.update(dict(self._max_label_shapes[0]))\n\n max_data_shapes = list()\n for name, shape in data_shapes[0]:\n if name in max_shapes_dict:\n max_data_shapes.append((name, max_shapes_dict[name]))\n else:\n max_data_shapes.append((name, shape))\n\n max_label_shapes = list()\n if not label_shapes.count(None) == len(label_shapes):\n for name, shape in label_shapes[0]:\n if name in max_shapes_dict:\n max_label_shapes.append((name, max_shapes_dict[name]))\n else:\n max_label_shapes.append((name, shape))\n\n if len(max_label_shapes) == 0:\n max_label_shapes = None\n\n module = Module(self._symbol, self._data_names, self._label_names, logger=self.logger,\n context=self._context, work_load_list=self._work_load_list,\n fixed_param_names=self._fixed_param_names)\n module.bind([max_data_shapes for _ in xrange(len(self._context))], [max_label_shapes for _ in xrange(len(self._context))],\n for_training, inputs_need_grad, force_rebind=False, shared_module=None)\n self._curr_module = module\n\n # copy back saved params, if already initialized\n if self.params_initialized:\n self.set_params(arg_params, aux_params)\n\n def save_checkpoint(self, prefix, epoch, save_optimizer_states=False):\n \"\"\"Save current progress to checkpoint.\n Use mx.callback.module_checkpoint as epoch_end_callback to save during training.\n\n Parameters\n ----------\n prefix : str\n The file prefix to checkpoint to\n epoch : int\n The current epoch number\n save_optimizer_states : bool\n Whether to save optimizer states for continue training\n \"\"\"\n self._curr_module.save_checkpoint(prefix, epoch, save_optimizer_states)\n\n def init_optimizer(self, kvstore='local', optimizer='sgd',\n optimizer_params=(('learning_rate', 0.01),), force_init=False):\n assert self.binded and self.params_initialized\n if self.optimizer_initialized and not force_init:\n self.logger.warning('optimizer already initialized, ignoring.')\n return\n\n self._curr_module._preload_opt_states = self._preload_opt_states\n self._curr_module.init_optimizer(kvstore, optimizer, optimizer_params,\n force_init=force_init)\n self.optimizer_initialized = True\n\n def fit(self, train_data, eval_data=None, eval_metric='acc',\n epoch_end_callback=None, batch_end_callback=None, kvstore='local',\n optimizer='sgd', optimizer_params=(('learning_rate', 0.01),),\n eval_end_callback=None,\n eval_batch_end_callback=None, initializer=Uniform(0.01),\n arg_params=None, aux_params=None, allow_missing=False,\n force_rebind=False, force_init=False, begin_epoch=0, num_epoch=None,\n validation_metric=None, monitor=None, prefix=None):\n \"\"\"Train the module parameters.\n\n Parameters\n ----------\n train_data : DataIter\n eval_data : DataIter\n If not `None`, will be used as validation set and evaluate the performance\n after each epoch.\n eval_metric : str or EvalMetric\n Default `'acc'`. The performance measure used to display during training.\n epoch_end_callback : function or list of function\n Each callback will be called with the current `epoch`, `symbol`, `arg_params`\n and `aux_params`.\n batch_end_callback : function or list of function\n Each callback will be called with a `BatchEndParam`.\n kvstore : str or KVStore\n Default `'local'`.\n optimizer : str or Optimizer\n Default `'sgd'`\n optimizer_params : dict\n Default `(('learning_rate', 0.01),)`. The parameters for the optimizer constructor.\n The default value is not a `dict`, just to avoid pylint warning on dangerous\n default values.\n eval_end_callback : function or list of function\n These will be called at the end of each full evaluation, with the metrics over\n the entire evaluation set.\n eval_batch_end_callback : function or list of function\n These will be called at the end of each minibatch during evaluation\n initializer : Initializer\n Will be called to initialize the module parameters if not already initialized.\n arg_params : dict\n Default `None`, if not `None`, should be existing parameters from a trained\n model or loaded from a checkpoint (previously saved model). In this case,\n the value here will be used to initialize the module parameters, unless they\n are already initialized by the user via a call to `init_params` or `fit`.\n `arg_params` has higher priority to `initializer`.\n aux_params : dict\n Default `None`. Similar to `arg_params`, except for auxiliary states.\n allow_missing : bool\n Default `False`. Indicate whether we allow missing parameters when `arg_params`\n and `aux_params` are not `None`. If this is `True`, then the missing parameters\n will be initialized via the `initializer`.\n force_rebind : bool\n Default `False`. Whether to force rebinding the executors if already binded.\n force_init : bool\n Default `False`. Indicate whether we should force initialization even if the\n parameters are already initialized.\n begin_epoch : int\n Default `0`. Indicate the starting epoch. Usually, if we are resuming from a\n checkpoint saved at a previous training phase at epoch N, then we should specify\n this value as N+1.\n num_epoch : int\n Number of epochs to run training.\n\n Examples\n --------\n An example of using fit for training::\n >>> #Assume training dataIter and validation dataIter are ready\n >>> mod.fit(train_data=train_dataiter, eval_data=val_dataiter,\n optimizer_params={'learning_rate':0.01, 'momentum': 0.9},\n num_epoch=10)\n \"\"\"\n assert num_epoch is not None, 'please specify number of epochs'\n\n self.bind(data_shapes=train_data.provide_data, label_shapes=train_data.provide_label,\n for_training=True, force_rebind=force_rebind)\n if monitor is not None:\n self.install_monitor(monitor)\n self.init_params(initializer=initializer, arg_params=arg_params, aux_params=aux_params,\n allow_missing=allow_missing, force_init=force_init)\n self.init_optimizer(kvstore=kvstore, optimizer=optimizer,\n optimizer_params=optimizer_params)\n\n if validation_metric is None:\n validation_metric = eval_metric\n if not isinstance(eval_metric, metric.EvalMetric):\n eval_metric = metric.create(eval_metric)\n\n ################################################################################\n # training loop\n ################################################################################\n for epoch in range(begin_epoch, num_epoch):\n tic = time.time()\n eval_metric.reset()\n for nbatch, data_batch in enumerate(train_data):\n if monitor is not None:\n monitor.tic()\n self.forward_backward(data_batch)\n self.update()\n self.update_metric(eval_metric, data_batch.label)\n\n if monitor is not None:\n monitor.toc_print()\n\n if batch_end_callback is not None:\n batch_end_params = BatchEndParam(epoch=epoch, nbatch=nbatch,\n eval_metric=eval_metric,\n locals=locals())\n for callback in _as_list(batch_end_callback):\n callback(batch_end_params)\n\n # one epoch of training is finished\n for name, val in eval_metric.get_name_value():\n self.logger.info('Epoch[%d] Train-%s=%f', epoch, name, val)\n toc = time.time()\n self.logger.info('Epoch[%d] Time cost=%.3f', epoch, (toc-tic))\n\n # sync aux params across devices\n arg_params, aux_params = self.get_params()\n self.set_params(arg_params, aux_params)\n\n if epoch_end_callback is not None:\n for callback in _as_list(epoch_end_callback):\n callback(epoch, self.symbol, arg_params, aux_params)\n\n #----------------------------------------\n # evaluation on validation set\n if eval_data:\n res = self.score(eval_data, validation_metric,\n score_end_callback=eval_end_callback,\n batch_end_callback=eval_batch_end_callback, epoch=epoch)\n #TODO: pull this into default\n for name, val in res:\n self.logger.info('Epoch[%d] Validation-%s=%f', epoch, name, val)\n\n # end of 1 epoch, reset the data-iter for another epoch\n train_data.reset()\n\n\n def forward(self, data_batch, is_train=None):\n assert self.binded and self.params_initialized\n\n # get current_shapes\n if self._curr_module.label_shapes is not None:\n current_shapes = [dict(self._curr_module.data_shapes[i] + self._curr_module.label_shapes[i]) for i in xrange(len(self._context))]\n else:\n current_shapes = [dict(self._curr_module.data_shapes[i]) for i in xrange(len(self._context))]\n\n # get input_shapes\n if is_train:\n input_shapes = [dict(data_batch.provide_data[i] + data_batch.provide_label[i]) for i in xrange(len(self._context))]\n else:\n input_shapes = [dict(data_batch.provide_data[i]) for i in xrange(len(data_batch.provide_data))]\n\n # decide if shape changed\n shape_changed = len(current_shapes) != len(input_shapes)\n for pre, cur in zip(current_shapes, input_shapes):\n for k, v in pre.items():\n if v != cur[k]:\n shape_changed = True\n\n if shape_changed:\n # self._curr_module.reshape(data_batch.provide_data, data_batch.provide_label)\n module = Module(self._symbol, self._data_names, self._label_names,\n logger=self.logger, context=[self._context[i] for i in xrange(len(data_batch.provide_data))],\n work_load_list=self._work_load_list,\n fixed_param_names=self._fixed_param_names)\n module.bind(data_batch.provide_data, data_batch.provide_label, self._curr_module.for_training,\n self._curr_module.inputs_need_grad, force_rebind=False,\n shared_module=self._curr_module)\n self._curr_module = module\n\n self._curr_module.forward(data_batch, is_train=is_train)\n\n def backward(self, out_grads=None):\n assert self.binded and self.params_initialized\n self._curr_module.backward(out_grads=out_grads)\n\n def update(self):\n assert self.binded and self.params_initialized and self.optimizer_initialized\n self._curr_module.update()\n\n def get_outputs(self, merge_multi_context=True):\n assert self.binded and self.params_initialized\n return self._curr_module.get_outputs(merge_multi_context=merge_multi_context)\n def get_input_grads(self, merge_multi_context=True):\n assert self.binded and self.params_initialized and self.inputs_need_grad\n return self._curr_module.get_input_grads(merge_multi_context=merge_multi_context)\n\n def update_metric(self, eval_metric, labels):\n assert self.binded and self.params_initialized\n self._curr_module.update_metric(eval_metric, labels)\n\n def install_monitor(self, mon):\n \"\"\" Install monitor on all executors \"\"\"\n assert self.binded\n self._curr_module.install_monitor(mon)\n" }, { "path": "dff_rfcn/core/rcnn.py", "content": "# --------------------------------------------------------\n# Deep Feature Flow\n# Copyright (c) 2017 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Modified by Yuwen Xiong\n# --------------------------------------------------------\n# Based on:\n# MX-RCNN\n# Copyright (c) 2016 by Contributors\n# Licence under The Apache 2.0 License\n# https://github.com/ijkguo/mx-rcnn/\n# --------------------------------------------------------\n\n\"\"\"\nFast R-CNN:\ndata =\n {'data': [num_images, c, h, w],\n 'rois': [num_rois, 5]}\nlabel =\n {'label': [num_rois],\n 'bbox_target': [num_rois, 4 * num_classes],\n 'bbox_weight': [num_rois, 4 * num_classes]}\nroidb extended format [image_index]\n ['image', 'height', 'width', 'flipped',\n 'boxes', 'gt_classes', 'gt_overlaps', 'max_classes', 'max_overlaps', 'bbox_targets']\n\"\"\"\n\nimport numpy as np\nimport numpy.random as npr\n\nfrom utils.image import get_image, tensor_vstack\nfrom bbox.bbox_transform import bbox_overlaps, bbox_transform\nfrom bbox.bbox_regression import expand_bbox_regression_targets\n\n\ndef get_rcnn_testbatch(roidb, cfg):\n \"\"\"\n return a dict of testbatch\n :param roidb: ['image', 'flipped'] + ['boxes']\n :return: data, label, im_info\n \"\"\"\n # assert len(roidb) == 1, 'Single batch only'\n imgs, roidb = get_image(roidb, cfg)\n im_array = imgs\n im_info = [np.array([roidb[i]['im_info']], dtype=np.float32) for i in range(len(roidb))]\n\n im_rois = [roidb[i]['boxes'] for i in range(len(roidb))]\n rois = im_rois\n rois_array = [np.hstack((0 * np.ones((rois[i].shape[0], 1)), rois[i])) for i in range(len(rois))]\n\n data = [{'data': im_array[i],\n 'rois': rois_array[i]} for i in range(len(roidb))]\n label = {}\n\n return data, label, im_info\n\n\ndef get_rcnn_batch(roidb, cfg):\n \"\"\"\n return a dict of multiple images\n :param roidb: a list of dict, whose length controls batch size\n ['images', 'flipped'] + ['gt_boxes', 'boxes', 'gt_overlap'] => ['bbox_targets']\n :return: data, label\n \"\"\"\n num_images = len(roidb)\n imgs, roidb = get_image(roidb, cfg)\n im_array = tensor_vstack(imgs)\n\n assert cfg.TRAIN.BATCH_ROIS == -1 or cfg.TRAIN.BATCH_ROIS % cfg.TRAIN.BATCH_IMAGES == 0, \\\n 'BATCHIMAGES {} must divide BATCH_ROIS {}'.format(cfg.TRAIN.BATCH_IMAGES, cfg.TRAIN.BATCH_ROIS)\n\n if cfg.TRAIN.BATCH_ROIS == -1:\n rois_per_image = np.sum([iroidb['boxes'].shape[0] for iroidb in roidb])\n fg_rois_per_image = rois_per_image\n else:\n rois_per_image = cfg.TRAIN.BATCH_ROIS / cfg.TRAIN.BATCH_IMAGES\n fg_rois_per_image = np.round(cfg.TRAIN.FG_FRACTION * rois_per_image).astype(int)\n\n rois_array = list()\n labels_array = list()\n bbox_targets_array = list()\n bbox_weights_array = list()\n\n for im_i in range(num_images):\n roi_rec = roidb[im_i]\n\n # infer num_classes from gt_overlaps\n num_classes = roi_rec['gt_overlaps'].shape[1]\n\n # label = class RoI has max overlap with\n rois = roi_rec['boxes']\n labels = roi_rec['max_classes']\n overlaps = roi_rec['max_overlaps']\n bbox_targets = roi_rec['bbox_targets']\n\n im_rois, labels, bbox_targets, bbox_weights = \\\n sample_rois(rois, fg_rois_per_image, rois_per_image, num_classes, cfg,\n labels, overlaps, bbox_targets)\n\n # project im_rois\n # do not round roi\n rois = im_rois\n batch_index = im_i * np.ones((rois.shape[0], 1))\n rois_array_this_image = np.hstack((batch_index, rois))\n rois_array.append(rois_array_this_image)\n\n # add labels\n labels_array.append(labels)\n bbox_targets_array.append(bbox_targets)\n bbox_weights_array.append(bbox_weights)\n\n rois_array = np.array(rois_array)\n labels_array = np.array(labels_array)\n bbox_targets_array = np.array(bbox_targets_array)\n bbox_weights_array = np.array(bbox_weights_array)\n\n data = {'data': im_array,\n 'rois': rois_array}\n label = {'label': labels_array,\n 'bbox_target': bbox_targets_array,\n 'bbox_weight': bbox_weights_array}\n\n return data, label\n\n\ndef sample_rois(rois, fg_rois_per_image, rois_per_image, num_classes, cfg,\n labels=None, overlaps=None, bbox_targets=None, gt_boxes=None):\n \"\"\"\n generate random sample of ROIs comprising foreground and background examples\n :param rois: all_rois [n, 4]; e2e: [n, 5] with batch_index\n :param fg_rois_per_image: foreground roi number\n :param rois_per_image: total roi number\n :param num_classes: number of classes\n :param labels: maybe precomputed\n :param overlaps: maybe precomputed (max_overlaps)\n :param bbox_targets: maybe precomputed\n :param gt_boxes: optional for e2e [n, 5] (x1, y1, x2, y2, cls)\n :return: (labels, rois, bbox_targets, bbox_weights)\n \"\"\"\n if labels is None:\n overlaps = bbox_overlaps(rois[:, 1:].astype(np.float), gt_boxes[:, :4].astype(np.float))\n gt_assignment = overlaps.argmax(axis=1)\n overlaps = overlaps.max(axis=1)\n labels = gt_boxes[gt_assignment, 4]\n\n # foreground RoI with FG_THRESH overlap\n fg_indexes = np.where(overlaps >= cfg.TRAIN.FG_THRESH)[0]\n # guard against the case when an image has fewer than fg_rois_per_image foreground RoIs\n fg_rois_per_this_image = np.minimum(fg_rois_per_image, fg_indexes.size)\n # Sample foreground regions without replacement\n if len(fg_indexes) > fg_rois_per_this_image:\n fg_indexes = npr.choice(fg_indexes, size=fg_rois_per_this_image, replace=False)\n\n # Select background RoIs as those within [BG_THRESH_LO, BG_THRESH_HI)\n bg_indexes = np.where((overlaps < cfg.TRAIN.BG_THRESH_HI) & (overlaps >= cfg.TRAIN.BG_THRESH_LO))[0]\n # Compute number of background RoIs to take from this image (guarding against there being fewer than desired)\n bg_rois_per_this_image = rois_per_image - fg_rois_per_this_image\n bg_rois_per_this_image = np.minimum(bg_rois_per_this_image, bg_indexes.size)\n # Sample foreground regions without replacement\n if len(bg_indexes) > bg_rois_per_this_image:\n bg_indexes = npr.choice(bg_indexes, size=bg_rois_per_this_image, replace=False)\n\n # indexes selected\n keep_indexes = np.append(fg_indexes, bg_indexes)\n\n # pad more to ensure a fixed minibatch size\n while keep_indexes.shape[0] < rois_per_image:\n gap = np.minimum(len(rois), rois_per_image - keep_indexes.shape[0])\n gap_indexes = npr.choice(range(len(rois)), size=gap, replace=False)\n keep_indexes = np.append(keep_indexes, gap_indexes)\n\n # select labels\n labels = labels[keep_indexes]\n # set labels of bg_rois to be 0\n labels[fg_rois_per_this_image:] = 0\n rois = rois[keep_indexes]\n\n # load or compute bbox_target\n if bbox_targets is not None:\n bbox_target_data = bbox_targets[keep_indexes, :]\n else:\n targets = bbox_transform(rois[:, 1:], gt_boxes[gt_assignment[keep_indexes], :4])\n if cfg.TRAIN.BBOX_NORMALIZATION_PRECOMPUTED:\n targets = ((targets - np.array(cfg.TRAIN.BBOX_MEANS))\n / np.array(cfg.TRAIN.BBOX_STDS))\n bbox_target_data = np.hstack((labels[:, np.newaxis], targets))\n\n bbox_targets, bbox_weights = \\\n expand_bbox_regression_targets(bbox_target_data, num_classes, cfg)\n\n return rois, labels, bbox_targets, bbox_weights\n\n" }, { "path": "dff_rfcn/core/tester.py", "content": "# --------------------------------------------------------\n# Deep Feature Flow\n# Copyright (c) 2017 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Modified by Xizhou Zhu, Yuwen Xiong\n# --------------------------------------------------------\n# Based on:\n# MX-RCNN\n# Copyright (c) 2016 by Contributors\n# Licence under The Apache 2.0 License\n# https://github.com/ijkguo/mx-rcnn/\n# --------------------------------------------------------\n\nfrom multiprocessing.pool import ThreadPool as Pool\nimport cPickle\nimport os\nimport time\nimport mxnet as mx\nimport numpy as np\n\nfrom module import MutableModule\nfrom utils import image\nfrom bbox.bbox_transform import bbox_pred, clip_boxes\nfrom nms.nms import py_nms_wrapper, cpu_nms_wrapper, gpu_nms_wrapper\nfrom utils.PrefetchingIter import PrefetchingIter\n\n\nclass Predictor(object):\n def __init__(self, symbol, data_names, label_names,\n context=mx.cpu(), max_data_shapes=None,\n provide_data=None, provide_label=None,\n arg_params=None, aux_params=None):\n self._mod = MutableModule(symbol, data_names, label_names,\n context=context, max_data_shapes=max_data_shapes)\n self._mod.bind(provide_data, provide_label, for_training=False)\n self._mod.init_params(arg_params=arg_params, aux_params=aux_params)\n\n def predict(self, data_batch):\n self._mod.forward(data_batch)\n # [dict(zip(self._mod.output_names, _)) for _ in zip(*self._mod.get_outputs(merge_multi_context=False))]\n return [dict(zip(self._mod.output_names, _)) for _ in zip(*self._mod.get_outputs(merge_multi_context=False))]\n\n\ndef im_proposal(predictor, data_batch, data_names, scales):\n output_all = predictor.predict(data_batch)\n\n data_dict_all = [dict(zip(data_names, data_batch.data[i])) for i in xrange(len(data_batch.data))]\n scores_all = []\n boxes_all = []\n\n for output, data_dict, scale in zip(output_all, data_dict_all, scales):\n # drop the batch index\n boxes = output['rois_output'].asnumpy()[:, 1:]\n scores = output['rois_score'].asnumpy()\n\n # transform to original scale\n boxes = boxes / scale\n scores_all.append(scores)\n boxes_all.append(boxes)\n\n return scores_all, boxes_all, data_dict_all\n\n\ndef generate_proposals(predictor, test_data, imdb, cfg, vis=False, thresh=0.):\n \"\"\"\n Generate detections results using RPN.\n :param predictor: Predictor\n :param test_data: data iterator, must be non-shuffled\n :param imdb: image database\n :param vis: controls visualization\n :param thresh: thresh for valid detections\n :return: list of detected boxes\n \"\"\"\n assert vis or not test_data.shuffle\n data_names = [k[0] for k in test_data.provide_data[0]]\n\n if not isinstance(test_data, PrefetchingIter):\n test_data = PrefetchingIter(test_data)\n\n idx = 0\n t = time.time()\n imdb_boxes = list()\n original_boxes = list()\n for im_info, data_batch in test_data:\n t1 = time.time() - t\n t = time.time()\n\n scales = [iim_info[0, 2] for iim_info in im_info]\n scores_all, boxes_all, data_dict_all = im_proposal(predictor, data_batch, data_names, scales)\n t2 = time.time() - t\n t = time.time()\n for delta, (scores, boxes, data_dict, scale) in enumerate(zip(scores_all, boxes_all, data_dict_all, scales)):\n # assemble proposals\n dets = np.hstack((boxes, scores))\n original_boxes.append(dets)\n\n # filter proposals\n keep = np.where(dets[:, 4:] > thresh)[0]\n dets = dets[keep, :]\n imdb_boxes.append(dets)\n\n if vis:\n vis_all_detection(data_dict['data'].asnumpy(), [dets], ['obj'], scale, cfg)\n\n print 'generating %d/%d' % (idx + 1, imdb.num_images), 'proposal %d' % (dets.shape[0]), \\\n 'data %.4fs net %.4fs' % (t1, t2 / test_data.batch_size)\n idx += 1\n\n\n assert len(imdb_boxes) == imdb.num_images, 'calculations not complete'\n\n # save results\n rpn_folder = os.path.join(imdb.result_path, 'rpn_data')\n if not os.path.exists(rpn_folder):\n os.mkdir(rpn_folder)\n\n rpn_file = os.path.join(rpn_folder, imdb.name + '_rpn.pkl')\n with open(rpn_file, 'wb') as f:\n cPickle.dump(imdb_boxes, f, cPickle.HIGHEST_PROTOCOL)\n\n if thresh > 0:\n full_rpn_file = os.path.join(rpn_folder, imdb.name + '_full_rpn.pkl')\n with open(full_rpn_file, 'wb') as f:\n cPickle.dump(original_boxes, f, cPickle.HIGHEST_PROTOCOL)\n\n print 'wrote rpn proposals to {}'.format(rpn_file)\n return imdb_boxes\n\n\ndef im_detect(predictor, data_batch, data_names, scales, cfg):\n output_all = predictor.predict(data_batch)\n\n data_dict_all = [dict(zip(data_names, data_batch.data[i])) for i in xrange(len(data_batch.data))]\n scores_all = []\n pred_boxes_all = []\n for output, data_dict, scale in zip(output_all, data_dict_all, scales):\n if cfg.TEST.HAS_RPN:\n rois = output['rois_output'].asnumpy()[:, 1:]\n else:\n rois = data_dict['rois'].asnumpy().reshape((-1, 5))[:, 1:]\n im_shape = data_dict['data'].shape\n\n # save output\n scores = output['cls_prob_reshape_output'].asnumpy()[0]\n bbox_deltas = output['bbox_pred_reshape_output'].asnumpy()[0]\n\n # post processing\n pred_boxes = bbox_pred(rois, bbox_deltas)\n pred_boxes = clip_boxes(pred_boxes, im_shape[-2:])\n\n # we used scaled image & roi to train, so it is necessary to transform them back\n pred_boxes = pred_boxes / scale\n\n scores_all.append(scores)\n pred_boxes_all.append(pred_boxes)\n\n if output_all[0].has_key('feat_conv_3x3_relu_output'):\n feat = output_all[0]['feat_conv_3x3_relu_output']\n else:\n feat = None\n return scores_all, pred_boxes_all, data_dict_all, feat\n\n\ndef im_batch_detect(predictor, data_batch, data_names, scales, cfg):\n output_all = predictor.predict(data_batch)\n\n data_dict_all = [dict(zip(data_names, data_batch.data[i])) for i in xrange(len(data_batch.data))]\n scores_all = []\n pred_boxes_all = []\n for output, data_dict, scale in zip(output_all, data_dict_all, scales):\n im_infos = data_dict['im_info'].asnumpy()\n # save output\n scores = output['cls_prob_reshape_output'].asnumpy()[0]\n bbox_deltas = output['bbox_pred_reshape_output'].asnumpy()[0]\n rois = output['rois_output'].asnumpy()\n for im_idx in xrange(im_infos.shape[0]):\n bb_idxs = np.where(rois[:,0] == im_idx)[0]\n im_shape = im_infos[im_idx, :2].astype(np.int)\n\n # post processing\n pred_boxes = bbox_pred(rois[bb_idxs, 1:], bbox_deltas[bb_idxs, :])\n pred_boxes = clip_boxes(pred_boxes, im_shape)\n\n # we used scaled image & roi to train, so it is necessary to transform them back\n pred_boxes = pred_boxes / scale[im_idx]\n\n scores_all.append(scores[bb_idxs, :])\n pred_boxes_all.append(pred_boxes)\n\n return scores_all, pred_boxes_all, data_dict_all\n\n\ndef pred_eval(gpu_id, key_predictor, cur_predictor, test_data, imdb, cfg, vis=False, thresh=1e-4, logger=None, ignore_cache=True):\n \"\"\"\n wrapper for calculating offline validation for faster data analysis\n in this example, all threshold are set by hand\n :param predictor: Predictor\n :param test_data: data iterator, must be non-shuffle\n :param imdb: image database\n :param vis: controls visualization\n :param thresh: valid detection threshold\n :return:\n \"\"\"\n\n det_file = os.path.join(imdb.result_path, imdb.name + '_'+ str(gpu_id) + '_detections.pkl')\n if os.path.exists(det_file) and not ignore_cache:\n with open(det_file, 'rb') as fid:\n all_boxes, frame_ids = cPickle.load(fid)\n return all_boxes, frame_ids\n\n assert vis or not test_data.shuffle\n data_names = [k[0] for k in test_data.provide_data[0]]\n num_images = test_data.size\n roidb_frame_ids = [x['frame_id'] for x in test_data.roidb]\n\n if not isinstance(test_data, PrefetchingIter):\n test_data = PrefetchingIter(test_data)\n\n nms = py_nms_wrapper(cfg.TEST.NMS)\n\n # limit detections to max_per_image over all classes\n max_per_image = cfg.TEST.max_per_image\n\n # all detections are collected into:\n # all_boxes[cls][image] = N x 5 array of detections in\n # (x1, y1, x2, y2, score)\n all_boxes = [[[] for _ in range(num_images)]\n for _ in range(imdb.num_classes)]\n frame_ids = np.zeros(num_images, dtype=np.int)\n\n roidb_idx = -1\n roidb_offset = -1\n idx = 0\n data_time, net_time, post_time = 0.0, 0.0, 0.0\n t = time.time()\n for im_info, key_frame_flag, data_batch in test_data:\n t1 = time.time() - t\n t = time.time()\n\n scales = [iim_info[0, 2] for iim_info in im_info]\n if key_frame_flag != 2:\n scores_all, boxes_all, data_dict_all, feat = im_detect(key_predictor, data_batch, data_names, scales, cfg)\n else:\n data_batch.data[0][-1] = feat\n data_batch.provide_data[0][-1] = ('feat_key', feat.shape)\n scores_all, boxes_all, data_dict_all, _ = im_detect(cur_predictor, data_batch, data_names, scales, cfg)\n\n if key_frame_flag == 0:\n roidb_idx += 1\n roidb_offset = 0\n else:\n roidb_offset += 1\n\n frame_ids[idx] = roidb_frame_ids[roidb_idx] + roidb_offset\n\n t2 = time.time() - t\n t = time.time()\n for delta, (scores, boxes, data_dict) in enumerate(zip(scores_all, boxes_all, data_dict_all)):\n for j in range(1, imdb.num_classes):\n indexes = np.where(scores[:, j] > thresh)[0]\n cls_scores = scores[indexes, j, np.newaxis]\n cls_boxes = boxes[indexes, 4:8] if cfg.CLASS_AGNOSTIC else boxes[indexes, j * 4:(j + 1) * 4]\n cls_dets = np.hstack((cls_boxes, cls_scores))\n keep = nms(cls_dets)\n all_boxes[j][idx+delta] = cls_dets[keep, :]\n\n if max_per_image > 0:\n image_scores = np.hstack([all_boxes[j][idx+delta][:, -1]\n for j in range(1, imdb.num_classes)])\n if len(image_scores) > max_per_image:\n image_thresh = np.sort(image_scores)[-max_per_image]\n for j in range(1, imdb.num_classes):\n keep = np.where(all_boxes[j][idx+delta][:, -1] >= image_thresh)[0]\n all_boxes[j][idx+delta] = all_boxes[j][idx+delta][keep, :]\n\n if vis:\n boxes_this_image = [[]] + [all_boxes[j][idx+delta] for j in range(1, imdb.num_classes)]\n vis_all_detection(data_dict['data'].asnumpy(), boxes_this_image, imdb.classes, scales[delta], cfg)\n\n idx += test_data.batch_size\n t3 = time.time() - t\n t = time.time()\n data_time += t1\n net_time += t2\n post_time += t3\n print 'testing {}/{} data {:.4f}s net {:.4f}s post {:.4f}s'.format(idx, num_images, data_time / idx * test_data.batch_size, net_time / idx * test_data.batch_size, post_time / idx * test_data.batch_size)\n if logger:\n logger.info('testing {}/{} data {:.4f}s net {:.4f}s post {:.4f}s'.format(idx, num_images, data_time / idx * test_data.batch_size, net_time / idx * test_data.batch_size, post_time / idx * test_data.batch_size))\n\n with open(det_file, 'wb') as f:\n cPickle.dump((all_boxes, frame_ids), f, protocol=cPickle.HIGHEST_PROTOCOL)\n\n return all_boxes, frame_ids\n\ndef pred_eval_multiprocess(gpu_num, key_predictors, cur_predictors, test_datas, imdb, cfg, vis=False, thresh=1e-4, logger=None, ignore_cache=True):\n if gpu_num == 1:\n res = [pred_eval(0, key_predictors[0], cur_predictors[0], test_datas[0], imdb, cfg, vis, thresh, logger, ignore_cache),]\n else:\n pool = Pool(processes=gpu_num)\n multiple_results = [pool.apply_async(pred_eval,args=(i, key_predictors[i], cur_predictors[i], test_datas[i], imdb, cfg, vis, thresh, logger, ignore_cache)) for i in range(gpu_num)]\n pool.close()\n pool.join()\n res = [res.get() for res in multiple_results]\n info_str = imdb.evaluate_detections_multiprocess(res)\n if logger:\n logger.info('evaluate detections: \\n{}'.format(info_str))\n\ndef vis_all_detection(im_array, detections, class_names, scale, cfg, threshold=1e-4):\n \"\"\"\n visualize all detections in one image\n :param im_array: [b=1 c h w] in rgb\n :param detections: [ numpy.ndarray([[x1 y1 x2 y2 score]]) for j in classes ]\n :param class_names: list of names in imdb\n :param scale: visualize the scaled image\n :return:\n \"\"\"\n import matplotlib.pyplot as plt\n import random\n im = image.transform_inverse(im_array, cfg.network.PIXEL_MEANS)\n plt.imshow(im)\n for j, name in enumerate(class_names):\n if name == '__background__':\n continue\n color = (random.random(), random.random(), random.random()) # generate a random color\n dets = detections[j]\n for det in dets:\n bbox = det[:4] * scale\n score = det[-1]\n if score < threshold:\n continue\n rect = plt.Rectangle((bbox[0], bbox[1]),\n bbox[2] - bbox[0],\n bbox[3] - bbox[1], fill=False,\n edgecolor=color, linewidth=3.5)\n plt.gca().add_patch(rect)\n plt.gca().text(bbox[0], bbox[1] - 2,\n '{:s} {:.3f}'.format(name, score),\n bbox=dict(facecolor=color, alpha=0.5), fontsize=12, color='white')\n plt.show()\n\n\ndef draw_all_detection(im_array, detections, class_names, scale, cfg, threshold=1e-1):\n \"\"\"\n visualize all detections in one image\n :param im_array: [b=1 c h w] in rgb\n :param detections: [ numpy.ndarray([[x1 y1 x2 y2 score]]) for j in classes ]\n :param class_names: list of names in imdb\n :param scale: visualize the scaled image\n :return:\n \"\"\"\n import cv2\n import random\n color_white = (255, 255, 255)\n im = image.transform_inverse(im_array, cfg.network.PIXEL_MEANS)\n # change to bgr\n im = cv2.cvtColor(im, cv2.COLOR_RGB2BGR)\n for j, name in enumerate(class_names):\n if name == '__background__':\n continue\n color = (random.randint(0, 256), random.randint(0, 256), random.randint(0, 256)) # generate a random color\n dets = detections[j]\n for det in dets:\n bbox = det[:4] * scale\n score = det[-1]\n if score < threshold:\n continue\n bbox = map(int, bbox)\n cv2.rectangle(im, (bbox[0], bbox[1]), (bbox[2], bbox[3]), color=color, thickness=2)\n cv2.putText(im, '%s %.3f' % (class_names[j], score), (bbox[0], bbox[1] + 10),\n color=color_white, fontFace=cv2.FONT_HERSHEY_COMPLEX, fontScale=0.5)\n return im\n" }, { "path": "dff_rfcn/demo.py", "content": "# --------------------------------------------------------\n# Deep Feature Flow\n# Copyright (c) 2017 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Written by Xizhou Zhu, Yi Li, Haochen Zhang\n# --------------------------------------------------------\n\nimport _init_paths\n\nimport argparse\nimport os\nimport glob\nimport sys\nimport logging\nimport pprint\nimport cv2\nfrom config.config import config, update_config\nfrom utils.image import resize, transform\nimport numpy as np\n# get config\nos.environ['PYTHONUNBUFFERED'] = '1'\nos.environ['MXNET_CUDNN_AUTOTUNE_DEFAULT'] = '0'\nos.environ['MXNET_ENABLE_GPU_P2P'] = '0'\ncur_path = os.path.abspath(os.path.dirname(__file__))\nupdate_config(cur_path + '/../experiments/dff_rfcn/cfgs/dff_rfcn_vid_demo.yaml')\n\nsys.path.insert(0, os.path.join(cur_path, '../external/mxnet', config.MXNET_VERSION))\nimport mxnet as mx\nfrom core.tester import im_detect, Predictor\nfrom symbols import *\nfrom utils.load_model import load_param\nfrom utils.show_boxes import show_boxes, draw_boxes\nfrom utils.tictoc import tic, toc\nfrom nms.nms import py_nms_wrapper, cpu_nms_wrapper, gpu_nms_wrapper\n\ndef parse_args():\n parser = argparse.ArgumentParser(description='Show Deep Feature Flow demo')\n args = parser.parse_args()\n return args\n\nargs = parse_args()\n\ndef main():\n # get symbol\n pprint.pprint(config)\n config.symbol = 'resnet_v1_101_flownet_rfcn'\n model = '/../model/rfcn_dff_flownet_vid'\n sym_instance = eval(config.symbol + '.' + config.symbol)()\n key_sym = sym_instance.get_key_test_symbol(config)\n cur_sym = sym_instance.get_cur_test_symbol(config)\n\n # set up class names\n num_classes = 31\n classes = ['airplane', 'antelope', 'bear', 'bicycle',\n 'bird', 'bus', 'car', 'cattle',\n 'dog', 'domestic_cat', 'elephant', 'fox',\n 'giant_panda', 'hamster', 'horse', 'lion',\n 'lizard', 'monkey', 'motorcycle', 'rabbit',\n 'red_panda', 'sheep', 'snake', 'squirrel',\n 'tiger', 'train', 'turtle', 'watercraft',\n 'whale', 'zebra']\n\n # load demo data\n image_names = glob.glob(cur_path + '/../demo/ILSVRC2015_val_00007010/*.JPEG')\n output_dir = cur_path + '/../demo/rfcn_dff/'\n if not os.path.exists(output_dir):\n os.makedirs(output_dir)\n key_frame_interval = 10\n\n #\n\n data = []\n key_im_tensor = None\n for idx, im_name in enumerate(image_names):\n assert os.path.exists(im_name), ('%s does not exist'.format(im_name))\n im = cv2.imread(im_name, cv2.IMREAD_COLOR | cv2.IMREAD_IGNORE_ORIENTATION)\n target_size = config.SCALES[0][0]\n max_size = config.SCALES[0][1]\n im, im_scale = resize(im, target_size, max_size, stride=config.network.IMAGE_STRIDE)\n im_tensor = transform(im, config.network.PIXEL_MEANS)\n im_info = np.array([[im_tensor.shape[2], im_tensor.shape[3], im_scale]], dtype=np.float32)\n if idx % key_frame_interval == 0:\n key_im_tensor = im_tensor\n data.append({'data': im_tensor, 'im_info': im_info, 'data_key': key_im_tensor, 'feat_key': np.zeros((1,config.network.DFF_FEAT_DIM,1,1))})\n\n\n # get predictor\n data_names = ['data', 'im_info', 'data_key', 'feat_key']\n label_names = []\n data = [[mx.nd.array(data[i][name]) for name in data_names] for i in xrange(len(data))]\n max_data_shape = [[('data', (1, 3, max([v[0] for v in config.SCALES]), max([v[1] for v in config.SCALES]))),\n ('data_key', (1, 3, max([v[0] for v in config.SCALES]), max([v[1] for v in config.SCALES]))),]]\n provide_data = [[(k, v.shape) for k, v in zip(data_names, data[i])] for i in xrange(len(data))]\n provide_label = [None for i in xrange(len(data))]\n arg_params, aux_params = load_param(cur_path + model, 0, process=True)\n key_predictor = Predictor(key_sym, data_names, label_names,\n context=[mx.gpu(0)], max_data_shapes=max_data_shape,\n provide_data=provide_data, provide_label=provide_label,\n arg_params=arg_params, aux_params=aux_params)\n cur_predictor = Predictor(cur_sym, data_names, label_names,\n context=[mx.gpu(0)], max_data_shapes=max_data_shape,\n provide_data=provide_data, provide_label=provide_label,\n arg_params=arg_params, aux_params=aux_params)\n nms = gpu_nms_wrapper(config.TEST.NMS, 0)\n\n # warm up\n for j in xrange(2):\n data_batch = mx.io.DataBatch(data=[data[j]], label=[], pad=0, index=0,\n provide_data=[[(k, v.shape) for k, v in zip(data_names, data[j])]],\n provide_label=[None])\n scales = [data_batch.data[i][1].asnumpy()[0, 2] for i in xrange(len(data_batch.data))]\n if j % key_frame_interval == 0:\n scores, boxes, data_dict, feat = im_detect(key_predictor, data_batch, data_names, scales, config)\n else:\n data_batch.data[0][-1] = feat\n data_batch.provide_data[0][-1] = ('feat_key', feat.shape)\n scores, boxes, data_dict, _ = im_detect(cur_predictor, data_batch, data_names, scales, config)\n\n print \"warmup done\"\n # test\n time = 0\n count = 0\n for idx, im_name in enumerate(image_names):\n data_batch = mx.io.DataBatch(data=[data[idx]], label=[], pad=0, index=idx,\n provide_data=[[(k, v.shape) for k, v in zip(data_names, data[idx])]],\n provide_label=[None])\n scales = [data_batch.data[i][1].asnumpy()[0, 2] for i in xrange(len(data_batch.data))]\n\n tic()\n if idx % key_frame_interval == 0:\n scores, boxes, data_dict, feat = im_detect(key_predictor, data_batch, data_names, scales, config)\n else:\n data_batch.data[0][-1] = feat\n data_batch.provide_data[0][-1] = ('feat_key', feat.shape)\n scores, boxes, data_dict, _ = im_detect(cur_predictor, data_batch, data_names, scales, config)\n time += toc()\n count += 1\n print 'testing {} {:.4f}s'.format(im_name, time/count)\n\n boxes = boxes[0].astype('f')\n scores = scores[0].astype('f')\n dets_nms = []\n for j in range(1, scores.shape[1]):\n cls_scores = scores[:, j, np.newaxis]\n cls_boxes = boxes[:, 4:8] if config.CLASS_AGNOSTIC else boxes[:, j * 4:(j + 1) * 4]\n cls_dets = np.hstack((cls_boxes, cls_scores))\n keep = nms(cls_dets)\n cls_dets = cls_dets[keep, :]\n cls_dets = cls_dets[cls_dets[:, -1] > 0.7, :]\n dets_nms.append(cls_dets)\n # visualize\n im = cv2.imread(im_name)\n im = cv2.cvtColor(im, cv2.COLOR_BGR2RGB)\n # show_boxes(im, dets_nms, classes, 1)\n out_im = draw_boxes(im, dets_nms, classes, 1)\n _, filename = os.path.split(im_name)\n cv2.imwrite(output_dir + filename,out_im)\n\n print 'done'\n\nif __name__ == '__main__':\n main()\n" }, { "path": "dff_rfcn/demo_batch.py", "content": "# --------------------------------------------------------\n# Deep Feature Flow\n# Copyright (c) 2017 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Written by Xizhou Zhu, Yi Li, Haochen Zhang\n# --------------------------------------------------------\n\nimport _init_paths\n\nimport argparse\nimport os\nimport glob\nimport sys\nimport logging\nimport pprint\nimport cv2\nfrom config.config import config, update_config\nfrom utils.image import resize, transform\nimport numpy as np\n# get config\nos.environ['PYTHONUNBUFFERED'] = '1'\nos.environ['MXNET_CUDNN_AUTOTUNE_DEFAULT'] = '0'\nos.environ['MXNET_ENABLE_GPU_P2P'] = '0'\ncur_path = os.path.abspath(os.path.dirname(__file__))\nupdate_config(cur_path + '/../experiments/dff_rfcn/cfgs/dff_rfcn_vid_demo.yaml')\n\nsys.path.insert(0, os.path.join(cur_path, '../external/mxnet', config.MXNET_VERSION))\nimport mxnet as mx\nfrom core.tester import im_batch_detect, Predictor\nfrom symbols import *\nfrom utils.load_model import load_param\nfrom utils.show_boxes import show_boxes, draw_boxes\nfrom utils.tictoc import tic, toc\nfrom nms.nms import py_nms_wrapper, cpu_nms_wrapper, gpu_nms_wrapper\n\ndef parse_args():\n parser = argparse.ArgumentParser(description='Show Deep Feature Flow demo')\n args = parser.parse_args()\n return args\n\nargs = parse_args()\n\ndef main():\n # get symbol\n pprint.pprint(config)\n config.symbol = 'resnet_v1_101_flownet_rfcn'\n model = '/../model/rfcn_dff_flownet_vid'\n sym_instance = eval(config.symbol + '.' + config.symbol)()\n sym = sym_instance.get_batch_test_symbol(config)\n\n # set up class names\n num_classes = 31\n classes = ['airplane', 'antelope', 'bear', 'bicycle',\n 'bird', 'bus', 'car', 'cattle',\n 'dog', 'domestic_cat', 'elephant', 'fox',\n 'giant_panda', 'hamster', 'horse', 'lion',\n 'lizard', 'monkey', 'motorcycle', 'rabbit',\n 'red_panda', 'sheep', 'snake', 'squirrel',\n 'tiger', 'train', 'turtle', 'watercraft',\n 'whale', 'zebra']\n\n # load demo data\n image_names = glob.glob(cur_path + '/../demo/ILSVRC2015_val_00007010/*.JPEG')\n output_dir = cur_path + '/../demo/rfcn_dff_batch/'\n if not os.path.exists(output_dir):\n os.makedirs(output_dir)\n key_frame_interval = 10\n\n #\n\n data = []\n key_im_tensor = None\n cur_im_tensor = []\n im_info_tensor = []\n image_names_list = []\n image_names_batch = []\n for idx, im_name in enumerate(image_names):\n assert os.path.exists(im_name), ('%s does not exist'.format(im_name))\n im = cv2.imread(im_name, cv2.IMREAD_COLOR | cv2.IMREAD_IGNORE_ORIENTATION)\n target_size = config.SCALES[0][0]\n max_size = config.SCALES[0][1]\n im, im_scale = resize(im, target_size, max_size, stride=config.network.IMAGE_STRIDE)\n im_tensor = transform(im, config.network.PIXEL_MEANS)\n im_info = np.array([[im_tensor.shape[2], im_tensor.shape[3], im_scale]], dtype=np.float32)\n if idx % key_frame_interval == 0:\n key_im_tensor = im_tensor\n else:\n cur_im_tensor.append(im_tensor)\n im_info_tensor.append(im_info)\n image_names_batch.append(im_name)\n if (idx+1) % key_frame_interval == 0 or idx == len(image_names) - 1:\n data.append({'data_other': np.concatenate(cur_im_tensor), 'im_info': np.concatenate(im_info_tensor), 'data_key': key_im_tensor})\n key_im_tensor = None\n cur_im_tensor = []\n im_info_tensor = []\n image_names_list.append(image_names_batch)\n image_names_batch = []\n\n # get predictor\n data_names = ['data_other', 'im_info', 'data_key']\n label_names = []\n data = [[mx.nd.array(data[i][name]) for name in data_names] for i in xrange(len(data))]\n max_data_shape = [[('data_other', (key_frame_interval-1, 3, max([v[0] for v in config.SCALES]), max([v[1] for v in config.SCALES]))),\n ('data_key', (1, 3, max([v[0] for v in config.SCALES]), max([v[1] for v in config.SCALES]))),]]\n provide_data = [[(k, v.shape) for k, v in zip(data_names, data[i])] for i in xrange(len(data))]\n provide_label = [None for i in xrange(len(data))]\n arg_params, aux_params = load_param(cur_path + model, 0, process=True)\n predictor = Predictor(sym, data_names, label_names,\n context=[mx.gpu(0)], max_data_shapes=max_data_shape,\n provide_data=provide_data, provide_label=provide_label,\n arg_params=arg_params, aux_params=aux_params)\n nms = gpu_nms_wrapper(config.TEST.NMS, 0)\n\n # warm up\n for j in xrange(1):\n data_batch = mx.io.DataBatch(data=[data[j]], label=[], pad=0, index=0,\n provide_data=[[(k, v.shape) for k, v in zip(data_names, data[j])]],\n provide_label=[None])\n scales = [data_batch.data[i][1].asnumpy()[:, 2] for i in xrange(len(data_batch.data))]\n scores_all, boxes_all, data_dict = im_batch_detect(predictor, data_batch, data_names, scales, config)\n\n print \"warmup done\"\n # test\n time = 0\n count = 0\n for idx, im_names in enumerate(image_names_list):\n data_batch = mx.io.DataBatch(data=[data[idx]], label=[], pad=0, index=idx,\n provide_data=[[(k, v.shape) for k, v in zip(data_names, data[idx])]],\n provide_label=[None])\n scales = [data_batch.data[i][1].asnumpy()[:, 2] for i in xrange(len(data_batch.data))]\n\n tic()\n scores_all, boxes_all, data_dict = im_batch_detect(predictor, data_batch, data_names, scales, config)\n time += toc()\n count += len(scores_all)\n print 'testing {} {:.4f}s x {:d}'.format(im_names[0], time/count, len(scores_all))\n\n for batch_idx in xrange(len(scores_all)):\n boxes = boxes_all[batch_idx].astype('f')\n scores = scores_all[batch_idx].astype('f')\n dets_nms = []\n for j in range(1, scores.shape[1]):\n cls_scores = scores[:, j, np.newaxis]\n cls_boxes = boxes[:, 4:8] if config.CLASS_AGNOSTIC else boxes[:, j * 4:(j + 1) * 4]\n cls_dets = np.hstack((cls_boxes, cls_scores))\n keep = nms(cls_dets)\n cls_dets = cls_dets[keep, :]\n cls_dets = cls_dets[cls_dets[:, -1] > 0.7, :]\n dets_nms.append(cls_dets)\n # visualize\n im = cv2.imread(im_names[batch_idx])\n im = cv2.cvtColor(im, cv2.COLOR_BGR2RGB)\n # show_boxes(im, dets_nms, classes, 1)\n out_im = draw_boxes(im, dets_nms, classes, 1)\n _, filename = os.path.split(im_names[batch_idx])\n cv2.imwrite(output_dir + filename,out_im)\n\n\n print 'done'\n\nif __name__ == '__main__':\n main()\n" }, { "path": "dff_rfcn/function/__init__.py", "content": "" }, { "path": "dff_rfcn/function/test_rcnn.py", "content": "# --------------------------------------------------------\n# Deep Feature Flow\n# Copyright (c) 2017 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Modified by Yuwen Xiong\n# --------------------------------------------------------\n# Based on:\n# MX-RCNN\n# Copyright (c) 2016 by Contributors\n# Licence under The Apache 2.0 License\n# https://github.com/ijkguo/mx-rcnn/\n# --------------------------------------------------------\n\nimport argparse\nimport pprint\nimport logging\nimport time\nimport os\nimport numpy as np\nimport mxnet as mx\n\nfrom symbols import *\nfrom dataset import *\nfrom core.loader import TestLoader\nfrom core.tester import Predictor, pred_eval, pred_eval_multiprocess\nfrom utils.load_model import load_param\n\ndef get_predictor(sym, sym_instance, cfg, arg_params, aux_params, test_data, ctx):\n # infer shape\n data_shape_dict = dict(test_data.provide_data_single)\n sym_instance.infer_shape(data_shape_dict)\n sym_instance.check_parameter_shapes(arg_params, aux_params, data_shape_dict, is_train=False)\n\n # decide maximum shape\n data_names = [k[0] for k in test_data.provide_data_single]\n label_names = None\n max_data_shape = [[('data', (1, 3, max([v[0] for v in cfg.SCALES]), max([v[1] for v in cfg.SCALES]))),\n ('data_key', (1, 3, max([v[0] for v in cfg.SCALES]), max([v[1] for v in cfg.SCALES]))),]]\n\n # create predictor\n predictor = Predictor(sym, data_names, label_names,\n context=ctx, max_data_shapes=max_data_shape,\n provide_data=test_data.provide_data, provide_label=test_data.provide_label,\n arg_params=arg_params, aux_params=aux_params)\n return predictor\n\ndef test_rcnn(cfg, dataset, image_set, root_path, dataset_path,\n ctx, prefix, epoch,\n vis, ignore_cache, shuffle, has_rpn, proposal, thresh, logger=None, output_path=None):\n if not logger:\n assert False, 'require a logger'\n\n # print cfg\n pprint.pprint(cfg)\n logger.info('testing cfg:{}\\n'.format(pprint.pformat(cfg)))\n\n # load symbol and testing data\n key_sym_instance = eval(cfg.symbol + '.' + cfg.symbol)()\n cur_sym_instance = eval(cfg.symbol + '.' + cfg.symbol)()\n key_sym = key_sym_instance.get_key_test_symbol(cfg)\n cur_sym = cur_sym_instance.get_cur_test_symbol(cfg)\n imdb = eval(dataset)(image_set, root_path, dataset_path, result_path=output_path)\n roidb = imdb.gt_roidb()\n\n # get test data iter\n # split roidbs\n gpu_num = len(ctx)\n roidbs = [[] for x in range(gpu_num)]\n roidbs_seg_lens = np.zeros(gpu_num, dtype=np.int)\n for x in roidb:\n gpu_id = np.argmin(roidbs_seg_lens)\n roidbs[gpu_id].append(x)\n roidbs_seg_lens[gpu_id] += x['frame_seg_len']\n\n # get test data iter\n test_datas = [TestLoader(x, cfg, batch_size=1, shuffle=shuffle, has_rpn=has_rpn) for x in roidbs]\n\n # load model\n arg_params, aux_params = load_param(prefix, epoch, process=True)\n\n # create predictor\n key_predictors = [get_predictor(key_sym, key_sym_instance, cfg, arg_params, aux_params, test_datas[i], [ctx[i]]) for i in range(gpu_num)]\n cur_predictors = [get_predictor(cur_sym, cur_sym_instance, cfg, arg_params, aux_params, test_datas[i], [ctx[i]]) for i in range(gpu_num)]\n\n # start detection\n #pred_eval(0, key_predictors[0], cur_predictors[0], test_datas[0], imdb, cfg, vis=vis, ignore_cache=ignore_cache, thresh=thresh, logger=logger)\n pred_eval_multiprocess(gpu_num, key_predictors, cur_predictors, test_datas, imdb, cfg, vis=vis, ignore_cache=ignore_cache, thresh=thresh, logger=logger)\n" }, { "path": "dff_rfcn/function/test_rpn.py", "content": "# --------------------------------------------------------\n# Deep Feature Flow\n# Copyright (c) 2017 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Modified by Yuwen Xiong\n# --------------------------------------------------------\n# Based on:\n# MX-RCNN\n# Copyright (c) 2016 by Contributors\n# Licence under The Apache 2.0 License\n# https://github.com/ijkguo/mx-rcnn/\n# --------------------------------------------------------\n\nimport argparse\nimport pprint\nimport logging\nimport mxnet as mx\n\nfrom symbols import *\nfrom dataset import *\nfrom core.loader import TestLoader\nfrom core.tester import Predictor, generate_proposals\nfrom utils.load_model import load_param\n\n\ndef test_rpn(cfg, dataset, image_set, root_path, dataset_path,\n ctx, prefix, epoch,\n vis, shuffle, thresh, logger=None, output_path=None):\n # set up logger\n if not logger:\n logging.basicConfig()\n logger = logging.getLogger()\n logger.setLevel(logging.INFO)\n\n # rpn generate proposal cfg\n cfg.TEST.HAS_RPN = True\n\n # print cfg\n pprint.pprint(cfg)\n logger.info('testing rpn cfg:{}\\n'.format(pprint.pformat(cfg)))\n\n # load symbol\n sym_instance = eval(cfg.symbol + '.' + cfg.symbol)()\n sym = sym_instance.get_symbol_rpn(cfg, is_train=False)\n\n # load dataset and prepare imdb for training\n imdb = eval(dataset)(image_set, root_path, dataset_path, result_path=output_path)\n roidb = imdb.gt_roidb()\n test_data = TestLoader(roidb, cfg, batch_size=len(ctx), shuffle=shuffle, has_rpn=True)\n\n # load model\n arg_params, aux_params = load_param(prefix, epoch)\n\n # infer shape\n data_shape_dict = dict(test_data.provide_data_single)\n sym_instance.infer_shape(data_shape_dict)\n\n # check parameters\n sym_instance.check_parameter_shapes(arg_params, aux_params, data_shape_dict, is_train=False)\n\n # decide maximum shape\n data_names = [k[0] for k in test_data.provide_data[0]]\n label_names = None if test_data.provide_label[0] is None else [k[0] for k in test_data.provide_label[0]]\n max_data_shape = [[('data', (1, 3, max([v[0] for v in cfg.SCALES]), max([v[1] for v in cfg.SCALES])))]]\n\n # create predictor\n predictor = Predictor(sym, data_names, label_names,\n context=ctx, max_data_shapes=max_data_shape,\n provide_data=test_data.provide_data, provide_label=test_data.provide_label,\n arg_params=arg_params, aux_params=aux_params)\n\n # start testing\n imdb_boxes = generate_proposals(predictor, test_data, imdb, cfg, vis=vis, thresh=thresh)\n\n all_log_info = imdb.evaluate_recall(roidb, candidate_boxes=imdb_boxes)\n logger.info(all_log_info)\n" }, { "path": "dff_rfcn/function/train_rcnn.py", "content": "# --------------------------------------------------------\n# Deep Feature Flow\n# Copyright (c) 2017 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Modified by Yuwen Xiong\n# --------------------------------------------------------\n# Based on:\n# MX-RCNN\n# Copyright (c) 2016 by Contributors\n# Licence under The Apache 2.0 License\n# https://github.com/ijkguo/mx-rcnn/\n# --------------------------------------------------------\n\nimport argparse\nimport logging\nimport pprint\nimport os\nimport mxnet as mx\n\nfrom symbols import *\nfrom core import callback, metric\nfrom core.loader import ROIIter\nfrom core.module import MutableModule\nfrom bbox.bbox_regression import add_bbox_regression_targets\nfrom utils.load_data import load_proposal_roidb, merge_roidb, filter_roidb\nfrom utils.load_model import load_param\nfrom utils.PrefetchingIter import PrefetchingIter\nfrom utils.lr_scheduler import WarmupMultiFactorScheduler\n\n\ndef train_rcnn(cfg, dataset, image_set, root_path, dataset_path,\n frequent, kvstore, flip, shuffle, resume,\n ctx, pretrained, epoch, prefix, begin_epoch, end_epoch,\n train_shared, lr, lr_step, proposal, logger=None, output_path=None):\n # set up logger\n if not logger:\n logging.basicConfig()\n logger = logging.getLogger()\n logger.setLevel(logging.INFO)\n\n # load symbol\n sym_instance = eval(cfg.symbol + '.' + cfg.symbol)()\n sym = sym_instance.get_symbol_rfcn(cfg, is_train=True)\n\n # setup multi-gpu\n batch_size = len(ctx)\n input_batch_size = cfg.TRAIN.BATCH_IMAGES * batch_size\n\n # print cfg\n pprint.pprint(cfg)\n logger.info('training rcnn cfg:{}\\n'.format(pprint.pformat(cfg)))\n\n # load dataset and prepare imdb for training\n image_sets = [iset for iset in image_set.split('+')]\n roidbs = [load_proposal_roidb(dataset, image_set, root_path, dataset_path,\n proposal=proposal, append_gt=True, flip=flip, result_path=output_path)\n for image_set in image_sets]\n roidb = merge_roidb(roidbs)\n roidb = filter_roidb(roidb, cfg)\n means, stds = add_bbox_regression_targets(roidb, cfg)\n\n # load training data\n train_data = ROIIter(roidb, cfg, batch_size=input_batch_size, shuffle=shuffle,\n ctx=ctx, aspect_grouping=cfg.TRAIN.ASPECT_GROUPING)\n\n # infer max shape\n max_data_shape = [('data', (cfg.TRAIN.BATCH_IMAGES, 3, max([v[0] for v in cfg.SCALES]), max([v[1] for v in cfg.SCALES])))]\n\n # infer shape\n data_shape_dict = dict(train_data.provide_data_single + train_data.provide_label_single)\n sym_instance.infer_shape(data_shape_dict)\n\n # load and initialize params\n if resume:\n print('continue training from ', begin_epoch)\n arg_params, aux_params = load_param(prefix, begin_epoch, convert=True)\n else:\n arg_params, aux_params = load_param(pretrained, epoch, convert=True)\n sym_instance.init_weight_rfcn(cfg, arg_params, aux_params)\n\n # check parameter shapes\n sym_instance.check_parameter_shapes(arg_params, aux_params, data_shape_dict)\n\n # prepare training\n # create solver\n data_names = [k[0] for k in train_data.provide_data_single]\n label_names = [k[0] for k in train_data.provide_label_single]\n if train_shared:\n fixed_param_prefix = cfg.network.FIXED_PARAMS_SHARED\n else:\n fixed_param_prefix = cfg.network.FIXED_PARAMS\n mod = MutableModule(sym, data_names=data_names, label_names=label_names,\n logger=logger, context=ctx,\n max_data_shapes=[max_data_shape for _ in range(batch_size)], fixed_param_prefix=fixed_param_prefix)\n\n if cfg.TRAIN.RESUME:\n mod._preload_opt_states = '%s-%04d.states'%(prefix, begin_epoch)\n\n\n # decide training params\n # metric\n eval_metric = metric.RCNNAccMetric(cfg)\n cls_metric = metric.RCNNLogLossMetric(cfg)\n bbox_metric = metric.RCNNL1LossMetric(cfg)\n eval_metrics = mx.metric.CompositeEvalMetric()\n for child_metric in [eval_metric, cls_metric, bbox_metric]:\n eval_metrics.add(child_metric)\n # callback\n batch_end_callback = callback.Speedometer(train_data.batch_size, frequent=frequent)\n epoch_end_callback = [mx.callback.module_checkpoint(mod, prefix, period=1, save_optimizer_states=True),\n callback.do_checkpoint(prefix, means, stds)]\n # decide learning rate\n base_lr = lr\n lr_factor = cfg.TRAIN.lr_factor\n lr_epoch = [float(epoch) for epoch in lr_step.split(',')]\n lr_epoch_diff = [epoch - begin_epoch for epoch in lr_epoch if epoch > begin_epoch]\n lr = base_lr * (lr_factor ** (len(lr_epoch) - len(lr_epoch_diff)))\n lr_iters = [int(epoch * len(roidb) / batch_size) for epoch in lr_epoch_diff]\n print('lr', lr, 'lr_epoch_diff', lr_epoch_diff, 'lr_iters', lr_iters)\n lr_scheduler = WarmupMultiFactorScheduler(lr_iters, lr_factor, cfg.TRAIN.warmup, cfg.TRAIN.warmup_lr, cfg.TRAIN.warmup_step)\n # optimizer\n optimizer_params = {'momentum': cfg.TRAIN.momentum,\n 'wd': cfg.TRAIN.wd,\n 'learning_rate': lr,\n 'lr_scheduler': lr_scheduler,\n 'rescale_grad': 1.0,\n 'clip_gradient': None}\n\n # train\n\n if not isinstance(train_data, PrefetchingIter):\n train_data = PrefetchingIter(train_data)\n\n mod.fit(train_data, eval_metric=eval_metrics, epoch_end_callback=epoch_end_callback,\n batch_end_callback=batch_end_callback, kvstore=kvstore,\n optimizer='sgd', optimizer_params=optimizer_params,\n arg_params=arg_params, aux_params=aux_params, begin_epoch=begin_epoch, num_epoch=end_epoch)\n\n" }, { "path": "dff_rfcn/function/train_rpn.py", "content": "# --------------------------------------------------------\n# Deep Feature Flow\n# Copyright (c) 2017 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Modified by Yuwen Xiong\n# --------------------------------------------------------\n# Based on:\n# MX-RCNN\n# Copyright (c) 2016 by Contributors\n# Licence under The Apache 2.0 License\n# https://github.com/ijkguo/mx-rcnn/\n# --------------------------------------------------------\n\nimport argparse\nimport logging\nimport pprint\nimport mxnet as mx\n\nfrom symbols import *\nfrom core import callback, metric\nfrom core.loader import AnchorLoader\nfrom core.module import MutableModule\nfrom utils.load_data import load_gt_roidb, merge_roidb, filter_roidb\nfrom utils.load_model import load_param\nfrom utils.PrefetchingIter import PrefetchingIter\nfrom utils.lr_scheduler import WarmupMultiFactorScheduler\n\n\ndef train_rpn(cfg, dataset, image_set, root_path, dataset_path,\n frequent, kvstore, flip, shuffle, resume,\n ctx, pretrained, epoch, prefix, begin_epoch, end_epoch,\n train_shared, lr, lr_step, logger=None, output_path=None):\n # set up logger\n if not logger:\n logging.basicConfig()\n logger = logging.getLogger()\n logger.setLevel(logging.INFO)\n\n # set up config\n cfg.TRAIN.BATCH_IMAGES = cfg.TRAIN.ALTERNATE.RPN_BATCH_IMAGES\n\n # load symbol\n sym_instance = eval(cfg.symbol + '.' + cfg.symbol)()\n sym = sym_instance.get_symbol_rpn(cfg, is_train=True)\n feat_sym = sym.get_internals()['rpn_cls_score_output']\n\n # setup multi-gpu\n batch_size = len(ctx)\n input_batch_size = cfg.TRAIN.BATCH_IMAGES * batch_size\n\n # print cfg\n pprint.pprint(cfg)\n logger.info('training rpn cfg:{}\\n'.format(pprint.pformat(cfg)))\n\n # load dataset and prepare imdb for training\n image_sets = [iset for iset in image_set.split('+')]\n roidbs = [load_gt_roidb(dataset, image_set, root_path, dataset_path, result_path=output_path,\n flip=flip)\n for image_set in image_sets]\n roidb = merge_roidb(roidbs)\n roidb = filter_roidb(roidb, cfg)\n\n # load training data\n train_data = AnchorLoader(feat_sym, roidb, cfg, batch_size=input_batch_size, shuffle=shuffle,\n ctx=ctx, feat_stride=cfg.network.RPN_FEAT_STRIDE, anchor_scales=cfg.network.ANCHOR_SCALES,\n anchor_ratios=cfg.network.ANCHOR_RATIOS, aspect_grouping=cfg.TRAIN.ASPECT_GROUPING)\n\n # infer max shape\n max_data_shape = [('data', (cfg.TRAIN.BATCH_IMAGES, 3, max([v[0] for v in cfg.SCALES]), max([v[1] for v in cfg.SCALES])))]\n max_data_shape, max_label_shape = train_data.infer_shape(max_data_shape)\n print('providing maximum shape', max_data_shape, max_label_shape)\n\n # infer shape\n data_shape_dict = dict(train_data.provide_data_single + train_data.provide_label_single)\n sym_instance.infer_shape(data_shape_dict)\n\n # load and initialize params\n if resume:\n print('continue training from ', begin_epoch)\n arg_params, aux_params = load_param(prefix, begin_epoch, convert=True)\n else:\n arg_params, aux_params = load_param(pretrained, epoch, convert=True)\n sym_instance.init_weight_rpn(cfg, arg_params, aux_params)\n\n # check parameter shapes\n sym_instance.check_parameter_shapes(arg_params, aux_params, data_shape_dict)\n\n # create solver\n data_names = [k[0] for k in train_data.provide_data_single]\n label_names = [k[0] for k in train_data.provide_label_single]\n if train_shared:\n fixed_param_prefix = cfg.network.FIXED_PARAMS_SHARED\n else:\n fixed_param_prefix = cfg.network.FIXED_PARAMS\n mod = MutableModule(sym, data_names=data_names, label_names=label_names,\n logger=logger, context=ctx, max_data_shapes=[max_data_shape for _ in xrange(batch_size)],\n max_label_shapes=[max_label_shape for _ in xrange(batch_size)], fixed_param_prefix=fixed_param_prefix)\n\n # decide training params\n # metric\n eval_metric = metric.RPNAccMetric()\n cls_metric = metric.RPNLogLossMetric()\n bbox_metric = metric.RPNL1LossMetric()\n eval_metrics = mx.metric.CompositeEvalMetric()\n for child_metric in [eval_metric, cls_metric, bbox_metric]:\n eval_metrics.add(child_metric)\n # callback\n batch_end_callback = callback.Speedometer(train_data.batch_size, frequent=frequent)\n # epoch_end_callback = mx.callback.do_checkpoint(prefix)\n epoch_end_callback = mx.callback.module_checkpoint(mod, prefix, period=1, save_optimizer_states=True)\n # decide learning rate\n base_lr = lr\n lr_factor = cfg.TRAIN.lr_factor\n lr_epoch = [int(epoch) for epoch in lr_step.split(',')]\n lr_epoch_diff = [epoch - begin_epoch for epoch in lr_epoch if epoch > begin_epoch]\n lr = base_lr * (lr_factor ** (len(lr_epoch) - len(lr_epoch_diff)))\n lr_iters = [int(epoch * len(roidb) / batch_size) for epoch in lr_epoch_diff]\n print('lr', lr, 'lr_epoch_diff', lr_epoch_diff, 'lr_iters', lr_iters)\n lr_scheduler = WarmupMultiFactorScheduler(lr_iters, lr_factor, cfg.TRAIN.warmup, cfg.TRAIN.warmup_lr, cfg.TRAIN.warmup_step)\n # optimizer\n optimizer_params = {'momentum': cfg.TRAIN.momentum,\n 'wd': cfg.TRAIN.wd,\n 'learning_rate': lr,\n 'lr_scheduler': lr_scheduler,\n 'rescale_grad': 1.0,\n 'clip_gradient': None}\n\n if not isinstance(train_data, PrefetchingIter):\n train_data = PrefetchingIter(train_data)\n\n # train\n mod.fit(train_data, eval_metric=eval_metrics, epoch_end_callback=epoch_end_callback,\n batch_end_callback=batch_end_callback, kvstore=kvstore,\n optimizer='sgd', optimizer_params=optimizer_params,\n arg_params=arg_params, aux_params=aux_params, begin_epoch=begin_epoch, num_epoch=end_epoch)\n\n" }, { "path": "dff_rfcn/operator_cxx/multi_proposal-inl.h", "content": "/*!\n * Copyright (c) 2015 by Contributors\n * Copyright (c) 2017 Microsoft\n * Licensed under The MIT License [see LICENSE for details]\n * \\file multi_proposal-inl.h\n * \\brief MultiProposal Operator\n * \\author Piotr Teterwak, Bing Xu, Jian Guo, Xizhou Zhu\n*/\n#ifndef MXNET_OPERATOR_CONTRIB_PROPOSAL_INL_H_\n#define MXNET_OPERATOR_CONTRIB_PROPOSAL_INL_H_\n\n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \"../operator_common.h\"\n#include \"../mshadow_op.h\"\n\n// extend NumericalParam\nnamespace mxnet {\nnamespace op {\n\n/*!\n* \\brief structure for numerical tuple input\n* \\tparam VType data type of param\n*/\ntemplate\nstruct NumericalParam {\n NumericalParam() {}\n explicit NumericalParam(VType *begin, VType *end) {\n int32_t size = static_cast(end - begin);\n info.resize(size);\n for (int i = 0; i < size; ++i) {\n info[i] = *(begin + i);\n }\n }\n inline size_t ndim() const {\n return info.size();\n }\n std::vector info;\n};\n\ntemplate\ninline std::istream &operator>>(std::istream &is, NumericalParam ¶m) {\n while (true) {\n char ch = is.get();\n if (ch == '(') break;\n if (!isspace(ch)) {\n is.setstate(std::ios::failbit);\n return is;\n }\n }\n VType idx;\n std::vector tmp;\n // deal with empty case\n size_t pos = is.tellg();\n char ch = is.get();\n if (ch == ')') {\n param.info = tmp;\n return is;\n }\n is.seekg(pos);\n // finish deal\n while (is >> idx) {\n tmp.push_back(idx);\n char ch;\n do {\n ch = is.get();\n } while (isspace(ch));\n if (ch == ',') {\n while (true) {\n ch = is.peek();\n if (isspace(ch)) {\n is.get(); continue;\n }\n if (ch == ')') {\n is.get(); break;\n }\n break;\n }\n if (ch == ')') break;\n } else if (ch == ')') {\n break;\n } else {\n is.setstate(std::ios::failbit);\n return is;\n }\n }\n param.info = tmp;\n return is;\n}\n\ntemplate\ninline std::ostream &operator<<(std::ostream &os, const NumericalParam ¶m) {\n os << '(';\n for (index_t i = 0; i < param.info.size(); ++i) {\n if (i != 0) os << ',';\n os << param.info[i];\n }\n // python style tuple\n if (param.info.size() == 1) os << ',';\n os << ')';\n return os;\n}\n\n} // namespace op\n} // namespace mxnet\n\nnamespace mxnet {\nnamespace op {\n\nnamespace proposal {\nenum MultiProposalOpInputs {kClsProb, kBBoxPred, kImInfo};\nenum MultiProposalOpOutputs {kOut, kScore};\nenum MultiProposalForwardResource {kTempResource};\n} // proposal\n\nstruct MultiProposalParam : public dmlc::Parameter {\n int rpn_pre_nms_top_n;\n int rpn_post_nms_top_n;\n float threshold;\n int rpn_min_size;\n NumericalParam scales;\n NumericalParam ratios;\n int feature_stride;\n bool output_score;\n bool iou_loss;\n DMLC_DECLARE_PARAMETER(MultiProposalParam) {\n float tmp[] = {0, 0, 0, 0};\n DMLC_DECLARE_FIELD(rpn_pre_nms_top_n).set_default(6000)\n .describe(\"Number of top scoring boxes to keep after applying NMS to RPN proposals\");\n DMLC_DECLARE_FIELD(rpn_post_nms_top_n).set_default(300)\n .describe(\"Overlap threshold used for non-maximum\"\n \"suppresion(suppress boxes with IoU >= this threshold\");\n DMLC_DECLARE_FIELD(threshold).set_default(0.7)\n .describe(\"NMS value, below which to suppress.\");\n DMLC_DECLARE_FIELD(rpn_min_size).set_default(16)\n .describe(\"Minimum height or width in proposal\");\n tmp[0] = 4.0f; tmp[1] = 8.0f; tmp[2] = 16.0f; tmp[3] = 32.0f;\n DMLC_DECLARE_FIELD(scales).set_default(NumericalParam(tmp, tmp + 4))\n .describe(\"Used to generate anchor windows by enumerating scales\");\n tmp[0] = 0.5f; tmp[1] = 1.0f; tmp[2] = 2.0f;\n DMLC_DECLARE_FIELD(ratios).set_default(NumericalParam(tmp, tmp + 3))\n .describe(\"Used to generate anchor windows by enumerating ratios\");\n DMLC_DECLARE_FIELD(feature_stride).set_default(16)\n .describe(\"The size of the receptive field each unit in the convolution layer of the rpn,\"\n \"for example the product of all stride's prior to this layer.\");\n DMLC_DECLARE_FIELD(output_score).set_default(false)\n .describe(\"Add score to outputs\");\n DMLC_DECLARE_FIELD(iou_loss).set_default(false)\n .describe(\"Usage of IoU Loss\");\n }\n};\n\ntemplate\nOperator *CreateOp(MultiProposalParam param);\n\n#if DMLC_USE_CXX11\nclass MultiProposalProp : public OperatorProperty {\n public:\n void Init(const std::vector >& kwargs) override {\n param_.Init(kwargs);\n }\n\n std::map GetParams() const override {\n return param_.__DICT__();\n }\n\n bool InferShape(std::vector *in_shape,\n std::vector *out_shape,\n std::vector *aux_shape) const override {\n using namespace mshadow;\n CHECK_EQ(in_shape->size(), 3) << \"Input:[cls_prob, bbox_pred, im_info]\";\n const TShape &dshape = in_shape->at(proposal::kClsProb);\n if (dshape.ndim() == 0) return false;\n Shape<4> bbox_pred_shape;\n bbox_pred_shape = Shape4(dshape[0], dshape[1] * 2, dshape[2], dshape[3]);\n SHAPE_ASSIGN_CHECK(*in_shape, proposal::kBBoxPred,\n bbox_pred_shape);\n Shape<2> im_info_shape;\n im_info_shape = Shape2(dshape[0], 3);\n SHAPE_ASSIGN_CHECK(*in_shape, proposal::kImInfo, im_info_shape);\n out_shape->clear();\n // output\n out_shape->push_back(Shape2(dshape[0] * param_.rpn_post_nms_top_n, 5));\n // score\n out_shape->push_back(Shape2(dshape[0] * param_.rpn_post_nms_top_n, 1));\n return true;\n }\n\n OperatorProperty* Copy() const override {\n auto ptr = new MultiProposalProp();\n ptr->param_ = param_;\n return ptr;\n }\n\n std::string TypeString() const override {\n return \"_contrib_MultiProposal\";\n }\n\n std::vector ForwardResource(\n const std::vector &in_shape) const override {\n return {ResourceRequest::kTempSpace};\n }\n\n std::vector DeclareBackwardDependency(\n const std::vector &out_grad,\n const std::vector &in_data,\n const std::vector &out_data) const override {\n return {};\n }\n\n int NumVisibleOutputs() const override {\n if (param_.output_score) {\n return 2;\n } else {\n return 1;\n }\n }\n\n int NumOutputs() const override {\n return 2;\n }\n\n std::vector ListArguments() const override {\n return {\"cls_prob\", \"bbox_pred\", \"im_info\"};\n }\n\n std::vector ListOutputs() const override {\n return {\"output\", \"score\"};\n }\n\n Operator* CreateOperator(Context ctx) const override;\n\n private:\n MultiProposalParam param_;\n}; // class MultiProposalProp\n\n#endif // DMLC_USE_CXX11\n} // namespace op\n} // namespace mxnet\n\n//========================\n// Anchor Generation Utils\n//========================\nnamespace mxnet {\nnamespace op {\nnamespace utils {\n\ninline void _MakeAnchor(float w,\n float h,\n float x_ctr,\n float y_ctr,\n std::vector *out_anchors) {\n out_anchors->push_back(x_ctr - 0.5f * (w - 1.0f));\n out_anchors->push_back(y_ctr - 0.5f * (h - 1.0f));\n out_anchors->push_back(x_ctr + 0.5f * (w - 1.0f));\n out_anchors->push_back(y_ctr + 0.5f * (h - 1.0f));\n out_anchors->push_back(0.0f);\n}\n\ninline void _Transform(float scale,\n float ratio,\n const std::vector& base_anchor,\n std::vector *out_anchors) {\n float w = base_anchor[2] - base_anchor[1] + 1.0f;\n float h = base_anchor[3] - base_anchor[1] + 1.0f;\n float x_ctr = base_anchor[0] + 0.5 * (w - 1.0f);\n float y_ctr = base_anchor[1] + 0.5 * (h - 1.0f);\n float size = w * h;\n float size_ratios = std::floor(size / ratio);\n float new_w = std::floor(std::sqrt(size_ratios) + 0.5f) * scale;\n float new_h = std::floor((new_w / scale * ratio) + 0.5f) * scale;\n\n _MakeAnchor(new_w, new_h, x_ctr,\n y_ctr, out_anchors);\n}\n\n// out_anchors must have shape (n, 5), where n is ratios.size() * scales.size()\ninline void GenerateAnchors(const std::vector& base_anchor,\n const std::vector& ratios,\n const std::vector& scales,\n std::vector *out_anchors) {\n for (size_t j = 0; j < ratios.size(); ++j) {\n for (size_t k = 0; k < scales.size(); ++k) {\n _Transform(scales[k], ratios[j], base_anchor, out_anchors);\n }\n }\n}\n\n} // namespace utils\n} // namespace op\n} // namespace mxnet\n\n#endif // MXNET_OPERATOR_CONTRIB_PROPOSAL_INL_H_\n" }, { "path": "dff_rfcn/operator_cxx/multi_proposal.cc", "content": "/*!\n * Copyright (c) 2017 Microsoft\n * Licensed under The MIT License [see LICENSE for details]\n * \\file multi_proposal.cc\n * \\brief\n * \\author Xizhou Zhu\n*/\n\n#include \"./multi_proposal-inl.h\"\n\n\nnamespace mxnet {\nnamespace op {\n\ntemplate\nclass MultiProposalOp : public Operator{\n public:\n explicit MultiProposalOp(MultiProposalParam param) {\n this->param_ = param;\n }\n\n virtual void Forward(const OpContext &ctx,\n const std::vector &in_data,\n const std::vector &req,\n const std::vector &out_data,\n const std::vector &aux_states) {\n LOG(FATAL) << \"not implemented\";\n }\n\n virtual void Backward(const OpContext &ctx,\n const std::vector &out_grad,\n const std::vector &in_data,\n const std::vector &out_data,\n const std::vector &req,\n const std::vector &in_grad,\n const std::vector &aux_states) {\n LOG(FATAL) << \"not implemented\";\n }\n\n private:\n MultiProposalParam param_;\n}; // class MultiProposalOp\n\ntemplate<>\nOperator *CreateOp(MultiProposalParam param) {\n return new MultiProposalOp(param);\n}\n\nOperator* MultiProposalProp::CreateOperator(Context ctx) const {\n DO_BIND_DISPATCH(CreateOp, param_);\n}\n\nDMLC_REGISTER_PARAMETER(MultiProposalParam);\n\nMXNET_REGISTER_OP_PROPERTY(_contrib_MultiProposal, MultiProposalProp)\n.describe(\"Generate region proposals via RPN\")\n.add_argument(\"cls_score\", \"NDArray-or-Symbol\", \"Score of how likely proposal is object.\")\n.add_argument(\"bbox_pred\", \"NDArray-or-Symbol\", \"BBox Predicted deltas from anchors for proposals\")\n.add_argument(\"im_info\", \"NDArray-or-Symbol\", \"Image size and scale.\")\n.add_arguments(MultiProposalParam::__FIELDS__());\n\n} // namespace op\n} // namespace mxnet\n" }, { "path": "dff_rfcn/operator_cxx/multi_proposal.cu", "content": "/*!\n * Copyright (c) 2015 by Contributors\n * Copyright (c) 2017 Microsoft\n * Licensed under The MIT License [see LICENSE for details]\n * \\file multi_proposal.cu\n * \\brief MultiProposal Operator\n * \\author Shaoqing Ren, Xizhou Zhu, Jian Guo\n*/\n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n\n#include \n#include \n#include \n#include \n#include \n#include \n\n#include \"../operator_common.h\"\n#include \"../mshadow_op.h\"\n#include \"./multi_proposal-inl.h\"\n\n#define DIVUP(m, n) ((m) / (n) + ((m) % (n) > 0))\n\n#define FRCNN_CUDA_CHECK(condition) \\\n /* Code block avoids redefinition of cudaError_t error */ \\\n do { \\\n cudaError_t error = condition; \\\n CHECK_EQ(error, cudaSuccess) << \" \" << cudaGetErrorString(error); \\\n} while (0)\n\nnamespace mshadow {\nnamespace cuda {\nnamespace multi_proposal {\n\n// scores are (b, 2 * anchor, h, w)\n// workspace_proposals are (b, h * w * anchor, 5)\n// w defines \"x\" and h defines \"y\"\n// count should be total anchors numbers, h * w * anchors\ntemplate\n__global__ void ProposalGridKernel(const int count,\n const int num_anchors,\n const int height,\n const int width,\n const int feature_stride,\n const Dtype* scores,\n Dtype* workspace_proposals) {\n for (int index = blockIdx.x * blockDim.x + threadIdx.x;\n index < count;\n index += blockDim.x * gridDim.x) {\n int a = index % num_anchors;\n int w = (index / num_anchors) % width;\n int h = (index / num_anchors / width) % height;\n\tint b = index / num_anchors / width / height;\n\n workspace_proposals[index * 5 + 0] = workspace_proposals[a * 5 + 0] + w * feature_stride;\n workspace_proposals[index * 5 + 1] = workspace_proposals[a * 5 + 1] + h * feature_stride;\n workspace_proposals[index * 5 + 2] = workspace_proposals[a * 5 + 2] + w * feature_stride;\n workspace_proposals[index * 5 + 3] = workspace_proposals[a * 5 + 3] + h * feature_stride;\n\tworkspace_proposals[index * 5 + 4] = \n\t\tscores[((b * (2 * num_anchors) + a + num_anchors) * height + h) * width + w];\n\t//workspace_proposals[index * 5 + 4] = scores[(a * height + h) * width + w];\n }\n}\n\n// boxes are (b, h * w * anchor, 5)\n// deltas are (b, 4 * anchor, h, w)\n// out_pred_boxes are (b, h * w * anchor, 5)\n// count should be total anchors numbers, b * h * w * anchors\n// in-place write: boxes and out_pred_boxes are the same location\ntemplate\n__global__ void BBoxPredKernel(const int count,\n const int num_anchors,\n const int feat_height,\n const int feat_width,\n\t\t\t\t\t\t\t const int feature_stride,\n\t\t\t\t\t\t\t const Dtype* im_infos,\n const Dtype* boxes,\n const Dtype* deltas,\n Dtype* out_pred_boxes) {\n for (int index = blockIdx.x * blockDim.x + threadIdx.x;\n index < count;\n index += blockDim.x * gridDim.x) {\n int a = index % num_anchors;\n int w = (index / num_anchors) % feat_width;\n\tint h = (index / num_anchors / feat_width) % feat_height;\n\tint b = index / num_anchors / feat_width / feat_height;\n\n\tfloat im_height = im_infos[b * 3];\n\tfloat im_width = im_infos[b * 3 + 1];\n\tint real_height = static_cast(im_height / feature_stride);\n\tint real_width = static_cast(im_width / feature_stride);\n\n float width = boxes[index * 5 + 2] - boxes[index * 5 + 0] + 1.0f;\n float height = boxes[index * 5 + 3] - boxes[index * 5 + 1] + 1.0f;\n float ctr_x = boxes[index * 5 + 0] + 0.5f * (width - 1.0f);\n float ctr_y = boxes[index * 5 + 1] + 0.5f * (height - 1.0f);\n\n\tint ba = (b * num_anchors + a);\n float dx = deltas[((ba * 4) * feat_height + h) * feat_width + w];\n float dy = deltas[((ba * 4 + 1) * feat_height + h) * feat_width + w];\n float dw = deltas[((ba * 4 + 2) * feat_height + h) * feat_width + w];\n float dh = deltas[((ba * 4 + 3) * feat_height + h) * feat_width + w];\n\n float pred_ctr_x = dx * width + ctr_x;\n float pred_ctr_y = dy * height + ctr_y;\n float pred_w = exp(dw) * width;\n float pred_h = exp(dh) * height;\n\n float pred_x1 = pred_ctr_x - 0.5f * (pred_w - 1.0f);\n float pred_y1 = pred_ctr_y - 0.5f * (pred_h - 1.0f);\n float pred_x2 = pred_ctr_x + 0.5f * (pred_w - 1.0f);\n float pred_y2 = pred_ctr_y + 0.5f * (pred_h - 1.0f);\n\n pred_x1 = max(min(pred_x1, im_width - 1.0f), 0.0f);\n pred_y1 = max(min(pred_y1, im_height - 1.0f), 0.0f);\n pred_x2 = max(min(pred_x2, im_width - 1.0f), 0.0f);\n pred_y2 = max(min(pred_y2, im_height - 1.0f), 0.0f);\n\n out_pred_boxes[index * 5 + 0] = pred_x1;\n out_pred_boxes[index * 5 + 1] = pred_y1;\n out_pred_boxes[index * 5 + 2] = pred_x2;\n out_pred_boxes[index * 5 + 3] = pred_y2;\n\n if (h >= real_height || w >= real_width) {\n out_pred_boxes[index * 5 + 4] = -1.0f;\n }\n }\n}\n\n// boxes are (b, h * w * anchor, 5)\n// deltas are (b, 4 * anchor, h, w)\n// out_pred_boxes are (b, h * w * anchor, 5)\n// count should be total anchors numbers, b * h * w * anchors\n// in-place write: boxes and out_pred_boxes are the same location\ntemplate\n__global__ void IoUPredKernel(const int count,\n const int num_anchors,\n const int feat_height,\n const int feat_width,\n const int feature_stride,\n\t\t\t\t\t\t\t const Dtype* im_infos,\n const Dtype* boxes,\n const Dtype* deltas,\n Dtype* out_pred_boxes) {\n for (int index = blockIdx.x * blockDim.x + threadIdx.x;\n index < count;\n index += blockDim.x * gridDim.x) {\n int a = index % num_anchors;\n int w = (index / num_anchors) % feat_width;\n int h = (index / num_anchors / feat_width) % feat_height;\n\tint b = index / num_anchors / feat_width / feat_height;\n\n\tfloat im_height = im_infos[b * 3];\n\tfloat im_width = im_infos[b * 3 + 1];\n\tint real_height = static_cast(im_height / feature_stride);\n\tint real_width = static_cast(im_width / feature_stride);\n\n float x1 = boxes[index * 5 + 0];\n float y1 = boxes[index * 5 + 1];\n float x2 = boxes[index * 5 + 2];\n float y2 = boxes[index * 5 + 3];\n\n\tint ba = (b * num_anchors + a);\n float dx1 = deltas[((ba * 4) * feat_height + h) * feat_width + w];\n float dy1 = deltas[((ba * 4 + 1) * feat_height + h) * feat_width + w];\n float dx2 = deltas[((ba * 4 + 2) * feat_height + h) * feat_width + w];\n float dy2 = deltas[((ba * 4 + 3) * feat_height + h) * feat_width + w];\n\n float pred_x1 = max(min(x1 + dx1, im_width - 1.0f), 0.0f);\n float pred_y1 = max(min(y1 + dy1, im_height - 1.0f), 0.0f);\n float pred_x2 = max(min(x2 + dx2, im_width - 1.0f), 0.0f);\n float pred_y2 = max(min(y2 + dy2, im_height - 1.0f), 0.0f);\n\n out_pred_boxes[index * 5 + 0] = pred_x1;\n out_pred_boxes[index * 5 + 1] = pred_y1;\n out_pred_boxes[index * 5 + 2] = pred_x2;\n out_pred_boxes[index * 5 + 3] = pred_y2;\n\n\tif (h >= real_height || w >= real_width) {\n out_pred_boxes[index * 5 + 4] = -1.0f;\n }\n }\n}\n\n// filter box with stride less than rpn_min_size\n// filter: set score to zero\n// dets (b, n, 5)\ntemplate\n__global__ void FilterBoxKernel(const int count,\n\t\t\t\t\t\t\t\tconst int count_anchors,\n const float original_min_size,\n\t\t\t\t\t\t\t\tconst Dtype* im_infos,\n Dtype* dets) {\n for (int index = blockIdx.x * blockDim.x + threadIdx.x;\n index < count;\n index += blockDim.x * gridDim.x) {\n\tint b = index / count_anchors;\n float iw = dets[index * 5 + 2] - dets[index * 5 + 0] + 1.0f;\n float ih = dets[index * 5 + 3] - dets[index * 5 + 1] + 1.0f;\n\tfloat min_size = original_min_size * im_infos[b * 3 + 2];\n if (iw < min_size || ih < min_size) {\n dets[index * 5 + 0] -= min_size / 2;\n dets[index * 5 + 1] -= min_size / 2;\n dets[index * 5 + 2] += min_size / 2;\n dets[index * 5 + 3] += min_size / 2;\n dets[index * 5 + 4] = -1.0f;\n }\n }\n}\n\n// copy score and init order\n// dets (n, 5); score (n, ); order (n, )\n// count should be n (total anchors or proposals)\ntemplate\n__global__ void CopyScoreKernel(const int count,\n const Dtype* dets,\n Dtype* score,\n int* order) {\n for (int index = blockIdx.x * blockDim.x + threadIdx.x;\n index < count;\n index += blockDim.x * gridDim.x) {\n score[index] = dets[index * 5 + 4];\n order[index] = index;\n }\n}\n\n// reorder proposals according to order and keep the top_n proposals\n// prev_dets (n, 5); order (n, ); dets (n, 5)\n// count should be output anchor numbers (top_n)\ntemplate\n__global__ void ReorderProposalsKernel(const int count,\n const Dtype* prev_dets,\n const int* order,\n Dtype* dets) {\n for (int index = blockIdx.x * blockDim.x + threadIdx.x;\n index < count;\n index += blockDim.x * gridDim.x) {\n const int order_i = order[index];\n for (int j = 0; j < 5; j ++) {\n dets[index * 5 + j] = prev_dets[order_i * 5 + j];\n }\n }\n}\n\n__device__ inline float devIoU(float const * const a, float const * const b) {\n float left = max(a[0], b[0]), right = min(a[2], b[2]);\n float top = max(a[1], b[1]), bottom = min(a[3], b[3]);\n float width = max(right - left + 1, 0.f), height = max(bottom - top + 1, 0.f);\n float interS = width * height;\n float Sa = (a[2] - a[0] + 1) * (a[3] - a[1] + 1);\n float Sb = (b[2] - b[0] + 1) * (b[3] - b[1] + 1);\n return interS / (Sa + Sb - interS);\n}\n\n__global__ void nms_kernel(const int n_boxes, const float nms_overlap_thresh,\n const float *dev_boxes, uint64_t *dev_mask) {\n const int threadsPerBlock = sizeof(uint64_t) * 8;\n const int row_start = blockIdx.y;\n const int col_start = blockIdx.x;\n\n // if (row_start > col_start) return;\n\n const int row_size =\n min(n_boxes - row_start * threadsPerBlock, threadsPerBlock);\n const int col_size =\n min(n_boxes - col_start * threadsPerBlock, threadsPerBlock);\n\n __shared__ float block_boxes[threadsPerBlock * 5];\n if (threadIdx.x < col_size) {\n block_boxes[threadIdx.x * 5 + 0] =\n dev_boxes[(threadsPerBlock * col_start + threadIdx.x) * 5 + 0];\n block_boxes[threadIdx.x * 5 + 1] =\n dev_boxes[(threadsPerBlock * col_start + threadIdx.x) * 5 + 1];\n block_boxes[threadIdx.x * 5 + 2] =\n dev_boxes[(threadsPerBlock * col_start + threadIdx.x) * 5 + 2];\n block_boxes[threadIdx.x * 5 + 3] =\n dev_boxes[(threadsPerBlock * col_start + threadIdx.x) * 5 + 3];\n block_boxes[threadIdx.x * 5 + 4] =\n dev_boxes[(threadsPerBlock * col_start + threadIdx.x) * 5 + 4];\n }\n __syncthreads();\n\n if (threadIdx.x < row_size) {\n const int cur_box_idx = threadsPerBlock * row_start + threadIdx.x;\n const float *cur_box = dev_boxes + cur_box_idx * 5;\n int i = 0;\n uint64_t t = 0;\n int start = 0;\n if (row_start == col_start) {\n start = threadIdx.x + 1;\n }\n for (i = start; i < col_size; i++) {\n if (devIoU(cur_box, block_boxes + i * 5) > nms_overlap_thresh) {\n t |= 1ULL << i;\n }\n }\n const int col_blocks = DIVUP(n_boxes, threadsPerBlock);\n dev_mask[cur_box_idx * col_blocks + col_start] = t;\n }\n}\n\nvoid _nms(const mshadow::Tensor& boxes,\n const float nms_overlap_thresh,\n int *keep,\n int *num_out) {\n const int threadsPerBlock = sizeof(uint64_t) * 8;\n const int boxes_num = boxes.size(0);\n const int boxes_dim = boxes.size(1);\n\n float* boxes_dev = boxes.dptr_;\n uint64_t* mask_dev = NULL;\n\n const int col_blocks = DIVUP(boxes_num, threadsPerBlock);\n FRCNN_CUDA_CHECK(cudaMalloc(&mask_dev,\n boxes_num * col_blocks * sizeof(uint64_t)));\n\n dim3 blocks(DIVUP(boxes_num, threadsPerBlock),\n DIVUP(boxes_num, threadsPerBlock));\n dim3 threads(threadsPerBlock);\n nms_kernel<<>>(boxes_num,\n nms_overlap_thresh,\n boxes_dev,\n mask_dev);\n FRCNN_CUDA_CHECK(cudaPeekAtLastError());\n std::vector mask_host(boxes_num * col_blocks);\n FRCNN_CUDA_CHECK(cudaMemcpy(&mask_host[0],\n mask_dev,\n sizeof(uint64_t) * boxes_num * col_blocks,\n cudaMemcpyDeviceToHost));\n\n std::vector remv(col_blocks);\n memset(&remv[0], 0, sizeof(uint64_t) * col_blocks);\n\n int num_to_keep = 0;\n for (int i = 0; i < boxes_num; i++) {\n int nblock = i / threadsPerBlock;\n int inblock = i % threadsPerBlock;\n\n if (!(remv[nblock] & (1ULL << inblock))) {\n keep[num_to_keep++] = i;\n uint64_t *p = &mask_host[0] + i * col_blocks;\n for (int j = nblock; j < col_blocks; j++) {\n remv[j] |= p[j];\n }\n }\n }\n *num_out = num_to_keep;\n\n FRCNN_CUDA_CHECK(cudaFree(mask_dev));\n}\n\n// copy proposals to output\n// dets (top_n, 5); keep (top_n, ); out (top_n, )\n// count should be top_n (total anchors or proposals)\ntemplate\n__global__ void PrepareOutput(const int count,\n const Dtype* dets,\n const int* keep,\n const int out_size,\n\t\t\t\t\t\t\t const int image_index,\n Dtype* out,\n Dtype* score) {\n for (int index = blockIdx.x * blockDim.x + threadIdx.x;\n index < count;\n index += blockDim.x * gridDim.x) {\n out[index * 5] = image_index;\n if (index < out_size) {\n int keep_i = keep[index];\n for (int j = 0; j < 4; ++j) {\n out[index * 5 + j + 1] = dets[keep_i * 5 + j];\n }\n score[index] = dets[keep_i * 5 + 4];\n } else {\n int keep_i = keep[index % out_size];\n for (int j = 0; j < 4; ++j) {\n out[index * 5 + j + 1] = dets[keep_i * 5 + j];\n }\n score[index] = dets[keep_i * 5 + 4];\n }\n }\n}\n} // namespace multi_proposal\n} // namespace cuda\n} // namespace mshadow\n\nnamespace mxnet {\nnamespace op {\n\ntemplate\nclass MultiProposalGPUOp : public Operator{\n public:\n explicit MultiProposalGPUOp(MultiProposalParam param) {\n this->param_ = param;\n }\n\n virtual void Forward(const OpContext &ctx,\n const std::vector &in_data,\n const std::vector &req,\n const std::vector &out_data,\n const std::vector &aux_states) {\n using namespace mshadow;\n using namespace mshadow::expr;\n\tusing namespace mshadow::cuda;\n using namespace mshadow::cuda::multi_proposal;\n CHECK_EQ(in_data.size(), 3);\n CHECK_EQ(out_data.size(), 2);\n CHECK_GT(req.size(), 1);\n CHECK_EQ(req[proposal::kOut], kWriteTo);\n /*CHECK_EQ(in_data[proposal::kClsProb].shape_[0], 1)\n << \"Sorry, multiple images each device is not implemented.\";*/\n\n Stream *s = ctx.get_stream();\n\n Tensor scores = in_data[proposal::kClsProb].get(s);\n Tensor bbox_deltas = in_data[proposal::kBBoxPred].get(s);\n Tensor im_info = in_data[proposal::kImInfo].get(s);\n\n Tensor out = out_data[proposal::kOut].get(s);\n Tensor out_score = out_data[proposal::kScore].get(s);\n \n\tint num_images = scores.size(0);\n\tint num_anchors = scores.size(1) / 2;\n int height = scores.size(2);\n int width = scores.size(3);\n int count_anchors = num_anchors * height * width; // count of total anchors\n\tint count = num_images * count_anchors;\n // set to -1 for max\n int rpn_pre_nms_top_n = (param_.rpn_pre_nms_top_n > 0) ? param_.rpn_pre_nms_top_n : count_anchors;\n rpn_pre_nms_top_n = std::min(rpn_pre_nms_top_n, count_anchors);\n int rpn_post_nms_top_n = std::min(param_.rpn_post_nms_top_n, rpn_pre_nms_top_n);\n\n // Generate first anchors based on base anchor\n std::vector base_anchor(4);\n base_anchor[0] = 0.0;\n base_anchor[1] = 0.0;\n base_anchor[2] = param_.feature_stride - 1.0;\n base_anchor[3] = param_.feature_stride - 1.0;\n CHECK_EQ(num_anchors, param_.ratios.info.size() * param_.scales.info.size());\n std::vector anchors;\n utils::GenerateAnchors(base_anchor,\n param_.ratios.info,\n param_.scales.info,\n &anchors);\n\n // Copy generated anchors to GPU\n float* workspace_proposals_ptr = NULL;\n FRCNN_CUDA_CHECK(cudaMalloc(&workspace_proposals_ptr, sizeof(float) * num_images * count_anchors * 5));\n Tensor workspace_proposals(workspace_proposals_ptr, Shape3(num_images, count_anchors, 5));\n FRCNN_CUDA_CHECK(cudaMemcpy(workspace_proposals.dptr_,\n &anchors[0], sizeof(float) * anchors.size(),\n cudaMemcpyHostToDevice));\n\n // Copy proposals to a mesh grid\n dim3 dimGrid((count + kMaxThreadsPerBlock - 1) / kMaxThreadsPerBlock);\n dim3 dimBlock(kMaxThreadsPerBlock);\n CheckLaunchParam(dimGrid, dimBlock, \"ProposalGrid\");\n ProposalGridKernel<<>>(\n\t count, num_anchors, height, width, param_.feature_stride,\n scores.dptr_, workspace_proposals.dptr_);\n FRCNN_CUDA_CHECK(cudaPeekAtLastError());\n\n // Transform anchors and bbox_deltas into bboxes\n CheckLaunchParam(dimGrid, dimBlock, \"BBoxPred\");\n if (param_.iou_loss) {\n IoUPredKernel<<>>(\n\t\tcount, num_anchors, height, width, param_.feature_stride, im_info.dptr_,\n workspace_proposals.dptr_, bbox_deltas.dptr_, workspace_proposals.dptr_);\n } else {\n BBoxPredKernel<<>>(\n\t\tcount, num_anchors, height, width, param_.feature_stride, im_info.dptr_,\n workspace_proposals.dptr_, bbox_deltas.dptr_, workspace_proposals.dptr_);\n }\n FRCNN_CUDA_CHECK(cudaPeekAtLastError());\n\n // filter boxes with less than rpn_min_size\n CheckLaunchParam(dimGrid, dimBlock, \"FilterBox\");\n FilterBoxKernel<<>>(\n\t count, count_anchors, param_.rpn_min_size, im_info.dptr_, workspace_proposals.dptr_);\n FRCNN_CUDA_CHECK(cudaPeekAtLastError());\n\n\n\n\tdimGrid = dim3((count_anchors + kMaxThreadsPerBlock - 1) / kMaxThreadsPerBlock);\n\tdimBlock = dim3(kMaxThreadsPerBlock);\n\t// Copy score to a continuous memory\n\tfloat* score_ptr = NULL;\n\tFRCNN_CUDA_CHECK(cudaMalloc(&score_ptr, sizeof(float) * count_anchors));\n\tTensor score(score_ptr, Shape1(count_anchors));\n\tint* order_ptr = NULL;\n\tFRCNN_CUDA_CHECK(cudaMalloc(&order_ptr, sizeof(int) * count_anchors));\n\tTensor order(order_ptr, Shape1(count_anchors));\n\n\tfloat* workspace_ordered_proposals_ptr = NULL;\n\tFRCNN_CUDA_CHECK(cudaMalloc(&workspace_ordered_proposals_ptr,\n\t\tsizeof(float) * rpn_pre_nms_top_n * 5));\n\tTensor workspace_ordered_proposals(workspace_ordered_proposals_ptr,\n\t\tShape2(rpn_pre_nms_top_n, 5));\n\n\tint* keep;\n\tFRCNN_CUDA_CHECK(cudaMalloc(&keep, sizeof(int) * rpn_pre_nms_top_n));\n\n\tfor (int b = 0; b < num_images; b++) {\n\n\t\tCheckLaunchParam(dimGrid, dimBlock, \"CopyScore\");\n\t\tCopyScoreKernel << > >(\n\t\t\tcount_anchors, workspace_proposals.dptr_ + b * count_anchors * 5, score.dptr_, order.dptr_);\n\t\tFRCNN_CUDA_CHECK(cudaPeekAtLastError());\n\n\t\t// argsort score, save order\n\t\tthrust::stable_sort_by_key(thrust::device,\n\t\t\tscore.dptr_,\n\t\t\tscore.dptr_ + score.size(0),\n\t\t\torder.dptr_,\n\t\t\tthrust::greater());\n\t\tFRCNN_CUDA_CHECK(cudaPeekAtLastError());\n\n\t\t// Reorder proposals according to order\n\n\t\tdimGrid.x = (rpn_pre_nms_top_n + kMaxThreadsPerBlock - 1) / kMaxThreadsPerBlock;\n\t\tCheckLaunchParam(dimGrid, dimBlock, \"ReorderProposals\");\n\t\tReorderProposalsKernel << > >(\n\t\t\trpn_pre_nms_top_n, workspace_proposals.dptr_ + b * count_anchors * 5, order.dptr_, workspace_ordered_proposals.dptr_);\n\t\tFRCNN_CUDA_CHECK(cudaPeekAtLastError());\n\n\t\t// perform nms\n\t\tstd::vector _keep(workspace_ordered_proposals.size(0));\n\t\tint out_size = 0;\n\t\t_nms(workspace_ordered_proposals,\n\t\t\tparam_.threshold,\n\t\t\t&_keep[0],\n\t\t\t&out_size);\n\n\t\t// copy nms result to gpu\n\t\tFRCNN_CUDA_CHECK(cudaMemcpy(keep, &_keep[0], sizeof(int) * _keep.size(),\n\t\t\tcudaMemcpyHostToDevice));\n\n\t\t// copy results after nms\n\t\tdimGrid.x = (rpn_post_nms_top_n + kMaxThreadsPerBlock - 1) / kMaxThreadsPerBlock;\n\t\tCheckLaunchParam(dimGrid, dimBlock, \"PrepareOutput\");\n\t\tPrepareOutput << > >(\n\t\t\trpn_post_nms_top_n, workspace_ordered_proposals.dptr_, keep, out_size, b,\n\t\t\tout.dptr_ + b * rpn_post_nms_top_n * 5, out_score.dptr_ + b * rpn_post_nms_top_n);\n\t\tFRCNN_CUDA_CHECK(cudaPeekAtLastError());\n\t}\n\t// free temporary memory\n\tFRCNN_CUDA_CHECK(cudaFree(keep));\n\tFRCNN_CUDA_CHECK(cudaFree(workspace_ordered_proposals_ptr));\n\tFRCNN_CUDA_CHECK(cudaFree(workspace_proposals_ptr));\n\tFRCNN_CUDA_CHECK(cudaFree(score_ptr));\n\tFRCNN_CUDA_CHECK(cudaFree(order_ptr));\n }\n\n virtual void Backward(const OpContext &ctx,\n const std::vector &out_grad,\n const std::vector &in_data,\n const std::vector &out_data,\n const std::vector &req,\n const std::vector &in_grad,\n const std::vector &aux_states) {\n using namespace mshadow;\n using namespace mshadow::expr;\n CHECK_EQ(in_grad.size(), 3);\n\n Stream *s = ctx.get_stream();\n Tensor gscores = in_grad[proposal::kClsProb].get(s);\n Tensor gbbox = in_grad[proposal::kBBoxPred].get(s);\n Tensor ginfo = in_grad[proposal::kImInfo].get(s);\n\n // can not assume the grad would be zero\n Assign(gscores, req[proposal::kClsProb], 0);\n Assign(gbbox, req[proposal::kBBoxPred], 0);\n Assign(ginfo, req[proposal::kImInfo], 0);\n }\n\n private:\n MultiProposalParam param_;\n}; // class MultiProposalGPUOp\n\ntemplate<>\nOperator* CreateOp(MultiProposalParam param) {\n return new MultiProposalGPUOp(param);\n}\n} // namespace op\n} // namespace mxnet\n" }, { "path": "dff_rfcn/operator_cxx/psroi_pooling-inl.h", "content": "/*!\n * Copyright (c) 2017 by Contributors\n * Copyright (c) 2017 Microsoft\n * Licensed under The MIT License [see LICENSE for details]\n * \\file psroi_pooling-inl.h\n * \\brief psroi pooling operator and symbol\n * \\author Yi Li, Tairui Chen, Guodong Zhang, Jifeng Dai\n*/\n#ifndef MXNET_OPERATOR_PSROI_POOLING_INL_H_\n#define MXNET_OPERATOR_PSROI_POOLING_INL_H_\n\n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \"../mshadow_op.h\"\n#include \"../operator_common.h\"\n\n\nnamespace mxnet {\nnamespace op {\n\n// Declare enumeration of input order to make code more intuitive.\n// These enums are only visible within this header\nnamespace psroipool {\nenum PSROIPoolingOpInputs {kData, kBox};\nenum PSROIPoolingOpOutputs {kOut, kMappingChannel};\n} // psroipool\n\nstruct PSROIPoolingParam : public dmlc::Parameter {\n // TShape pooled_size;\n float spatial_scale;\n int output_dim;\n int pooled_size;\n int group_size;\n DMLC_DECLARE_PARAMETER(PSROIPoolingParam) {\n DMLC_DECLARE_FIELD(spatial_scale).set_range(0.0, 1.0)\n .describe(\"Ratio of input feature map height (or w) to raw image height (or w). \"\n \"Equals the reciprocal of total stride in convolutional layers\");\n DMLC_DECLARE_FIELD(output_dim).describe(\"fix output dim\");\n DMLC_DECLARE_FIELD(pooled_size).describe(\"fix pooled size\");\n DMLC_DECLARE_FIELD(group_size).set_default(0).describe(\"fix group size\");\n }\n};\n\ntemplate\nclass PSROIPoolingOp : public Operator {\n public:\n explicit PSROIPoolingOp(PSROIPoolingParam p) {\n this->param_ = p;\n }\n\n virtual void Forward(const OpContext &ctx,\n const std::vector &in_data,\n const std::vector &req,\n const std::vector &out_data,\n const std::vector &aux_args) {\n using namespace mshadow;\n size_t expected = 2;\n CHECK_EQ(in_data.size(), expected);\n CHECK_EQ(out_data.size(), expected);\n CHECK_EQ(out_data[psroipool::kOut].shape_[0], in_data[psroipool::kBox].shape_[0]);\n CHECK_EQ(out_data[psroipool::kMappingChannel].shape_[0], in_data[psroipool::kBox].shape_[0]);\n Stream *s = ctx.get_stream();\n\n Tensor data = in_data[psroipool::kData].get(s);\n Tensor bbox = in_data[psroipool::kBox].get(s);\n Tensor out = out_data[psroipool::kOut].get(s);\n Tensor mapping_channel = out_data[psroipool::kMappingChannel].get(s);\n CHECK_EQ(data.CheckContiguous(), true);\n CHECK_EQ(bbox.CheckContiguous(), true);\n CHECK_EQ(out.CheckContiguous(), true);\n CHECK_EQ(mapping_channel.CheckContiguous(), true);\n out = -FLT_MAX;\n mapping_channel = -1.0f;\n PSROIPoolForward(out, data, bbox, mapping_channel, param_.spatial_scale, param_.output_dim, param_.group_size);\n }\n\n virtual void Backward(const OpContext &ctx,\n const std::vector &out_grad,\n const std::vector &in_data,\n const std::vector &out_data,\n const std::vector &req,\n const std::vector &in_grad,\n const std::vector &aux_args) {\n using namespace mshadow;\n size_t expected = 2;\n CHECK_EQ(in_data.size(), expected);\n CHECK_EQ(out_data.size(), expected);\n CHECK_EQ(out_grad[psroipool::kOut].shape_[0], in_data[psroipool::kBox].shape_[0]);\n CHECK_EQ(out_data[psroipool::kMappingChannel].shape_[0], in_data[psroipool::kBox].shape_[0]);\n CHECK_NE(req[psroipool::kData], kWriteInplace) <<\n \"ROIPooling: Backward doesn't support kWriteInplace.\";\n CHECK_NE(req[psroipool::kBox], kWriteInplace) <<\n \"ROIPooling: Backward doesn't support kWriteInplace.\";\n Stream *s = ctx.get_stream();\n\n Tensor grad_out = out_grad[psroipool::kOut].get(s);\n Tensor bbox = in_data[psroipool::kBox].get(s);\n Tensor mapping_channel = out_data[psroipool::kMappingChannel].get(s);\n Tensor grad_in = in_grad[psroipool::kData].get(s);\n Tensor grad_roi = in_grad[psroipool::kBox].get(s);\n\n CHECK_EQ(grad_out.CheckContiguous(), true);\n CHECK_EQ(bbox.CheckContiguous(), true);\n CHECK_EQ(mapping_channel.CheckContiguous(), true);\n CHECK_EQ(grad_in.CheckContiguous(), true);\n\n if (kAddTo == req[psroipool::kData] || kWriteTo == req[psroipool::kData]) {\n if (kWriteTo == req[psroipool::kData]) {\n grad_in = 0.0f;\n }\n PSROIPoolBackwardAcc(grad_in, grad_out, bbox, mapping_channel, param_.spatial_scale, param_.output_dim);\n }\n if (kWriteTo == req[psroipool::kBox]) {\n grad_roi = 0.0f;\n }\n\n }\n\n private:\n PSROIPoolingParam param_;\n}; // class PSROIPoolingOp\n\n// Decalre Factory function, used for dispatch specialization\ntemplate\nOperator* CreateOp(PSROIPoolingParam param, int dtype);\n\n#if DMLC_USE_CXX11\nclass PSROIPoolingProp : public OperatorProperty {\n public:\n std::vector ListArguments() const override {\n return {\"data\", \"rois\"};\n }\n\n std::vector ListOutputs() const override {\n return {\"output\", \"maxidx\"};\n }\n\n int NumOutputs() const override {\n return 2;\n }\n\n int NumVisibleOutputs() const override {\n return 1;\n }\n\n void Init(const std::vector >& kwargs) override {\n param_.Init(kwargs);\n if (param_.group_size == 0) {\n param_.group_size = param_.pooled_size;\n }\n }\n\n std::map GetParams() const override {\n return param_.__DICT__();\n }\n\n bool InferShape(std::vector *in_shape,\n std::vector *out_shape,\n std::vector *aux_shape) const override {\n using namespace mshadow;\n CHECK_EQ(in_shape->size(), 2) << \"Input:[data, rois]\";\n\n // data: [batch_size, c, h, w]\n TShape dshape = in_shape->at(psroipool::kData);\n CHECK_EQ(dshape.ndim(), 4) << \"data should be a 4D tensor\";\n\n // bbox: [num_rois, 5]\n TShape bshape = in_shape->at(psroipool::kBox);\n CHECK_EQ(bshape.ndim(), 2) << \"bbox should be a 2D tensor of shape [batch, 5]\";\n CHECK_EQ(bshape[1], 5) << \"bbox should be a 2D tensor of shape [batch, 5]\";\n\n // out: [num_rois, c, pooled_h, pooled_w]\n // mapping_channel: [num_rois, c, pooled_h, pooled_w]\n out_shape->clear();\n out_shape->push_back(\n Shape4(bshape[0], param_.output_dim, param_.pooled_size, param_.pooled_size));\n out_shape->push_back(\n Shape4(bshape[0], param_.output_dim, param_.pooled_size, param_.pooled_size));\n return true;\n }\n\n bool InferType(std::vector *in_type,\n std::vector *out_type,\n std::vector *aux_type) const override {\n CHECK_EQ(in_type->size(), 2);\n int dtype = (*in_type)[0];\n CHECK_EQ(dtype, (*in_type)[1]);\n CHECK_NE(dtype, -1) << \"Input must have specified type\";\n\n out_type->clear();\n out_type->push_back(dtype);\n out_type->push_back(dtype);\n return true;\n }\n\n OperatorProperty* Copy() const override {\n PSROIPoolingProp* psroi_pooling_sym = new PSROIPoolingProp();\n psroi_pooling_sym->param_ = this->param_;\n return psroi_pooling_sym;\n }\n\n std::string TypeString() const override {\n return \"_contrib_PSROIPooling\";\n }\n\n // decalre dependency and inplace optimization options\n std::vector DeclareBackwardDependency(\n const std::vector &out_grad,\n const std::vector &in_data,\n const std::vector &out_data) const override {\n return {out_grad[psroipool::kOut], in_data[psroipool::kBox], out_data[psroipool::kMappingChannel]};\n }\n\n\n Operator* CreateOperator(Context ctx) const override {\n LOG(FATAL) << \"Not Implemented.\";\n return NULL;\n }\n\n Operator* CreateOperatorEx(Context ctx, std::vector *in_shape,\n std::vector *in_type) const override;\n\n\n private:\n PSROIPoolingParam param_;\n}; // class PSROIPoolingProp\n#endif\n} // namespace op\n} // namespace mxnet\n#endif // MXNET_OPERATOR_PSROI_POOLING_INL_H_" }, { "path": "dff_rfcn/operator_cxx/psroi_pooling.cc", "content": "/*!\n * Copyright (c) 2017 by Contributors\n * Copyright (c) 2017 Microsoft\n * Licensed under The MIT License [see LICENSE for details]\n * \\file psroi_pooling.cc\n * \\brief psroi pooling operator\n * \\author Yi Li, Tairui Chen, Guodong Zhang, Jifeng Dai\n*/\n#include \"./psroi_pooling-inl.h\"\n#include \n#include \n#include \n#include \n#include \n\nusing std::max;\nusing std::min;\nusing std::floor;\nusing std::ceil;\n\nnamespace mshadow {\ntemplate\ninline void PSROIPoolForward(const Tensor &out,\n const Tensor &data,\n const Tensor &bbox,\n const Tensor &mapping_channel,\n const float spatial_scale_,\n const int output_dim_, \n const int group_size_) {\n // NOT_IMPLEMENTED;\n return;\n}\n\ntemplate\ninline void PSROIPoolBackwardAcc(const Tensor &in_grad,\n const Tensor &out_grad,\n const Tensor &bbox,\n const Tensor &mapping_channel,\n const float spatial_scale_,\n const int output_dim_) {\n // NOT_IMPLEMENTED;\n return;\n}\n} // namespace mshadow\n\nnamespace mxnet {\nnamespace op {\n\ntemplate<>\nOperator *CreateOp(PSROIPoolingParam param, int dtype) {\n Operator* op = NULL;\n MSHADOW_REAL_TYPE_SWITCH(dtype, DType, {\n op = new PSROIPoolingOp(param);\n });\n return op;\n}\n\nOperator *PSROIPoolingProp::CreateOperatorEx(Context ctx, std::vector *in_shape,\n std::vector *in_type) const {\n std::vector out_shape, aux_shape;\n std::vector out_type, aux_type;\n CHECK(InferType(in_type, &out_type, &aux_type));\n CHECK(InferShape(in_shape, &out_shape, &aux_shape));\n DO_BIND_DISPATCH(CreateOp, param_, in_type->at(0));\n}\n\nDMLC_REGISTER_PARAMETER(PSROIPoolingParam);\n\nMXNET_REGISTER_OP_PROPERTY(_contrib_PSROIPooling, PSROIPoolingProp)\n.describe(\"Performs region-of-interest pooling on inputs. Resize bounding box coordinates by \"\n\"spatial_scale and crop input feature maps accordingly. The cropped feature maps are pooled \"\n\"by max pooling to a fixed size output indicated by pooled_size. batch_size will change to \"\n\"the number of region bounding boxes after PSROIPooling\")\n.add_argument(\"data\", \"Symbol\", \"Input data to the pooling operator, a 4D Feature maps\")\n.add_argument(\"rois\", \"Symbol\", \"Bounding box coordinates, a 2D array of \"\n\"[[batch_index, x1, y1, x2, y2]]. (x1, y1) and (x2, y2) are top left and down right corners \"\n\"of designated region of interest. batch_index indicates the index of corresponding image \"\n\"in the input data\")\n.add_arguments(PSROIPoolingParam::__FIELDS__());\n} // namespace op\n} // namespace mxnet" }, { "path": "dff_rfcn/operator_cxx/psroi_pooling.cu", "content": "/*!\n * Copyright (c) 2017 by Contributors\n * Copyright (c) 2017 Microsoft\n * Licensed under The MIT License [see LICENSE for details]\n * \\file psroi_pooling.cu\n * \\brief psroi pooling operator\n * \\author Yi Li, Tairui Chen, Guodong Zhang, Jifeng Dai\n*/\n#include \"./psroi_pooling-inl.h\"\n#include \n#include \n#include \n#include \n#include \"../../common/cuda_utils.h\"\n#include \"../mxnet_op.h\"\n\n#define PSROIPOOLING_CUDA_CHECK(condition) \\\n /* Code block avoids redefinition of cudaError_t error */ \\\n do { \\\n cudaError_t error = condition; \\\n CHECK_EQ(error, cudaSuccess) << \" \" << cudaGetErrorString(error); \\\n } while (0)\n#define CUDA_KERNEL_LOOP(i, n) \\\nfor (int i = blockIdx.x * blockDim.x + threadIdx.x; \\\n i < (n); \\\n i += blockDim.x * gridDim.x)\n\nnamespace mshadow {\nnamespace cuda {\n\ntemplate \n__global__ void PSROIPoolForwardKernel(\n const int count,\n const DType* bottom_data,\n const DType spatial_scale,\n const int channels,\n const int height, const int width,\n const int pooled_height, const int pooled_width,\n const DType* bottom_rois,\n const int output_dim,\n const int group_size,\n DType* top_data,\n DType* mapping_channel) {\n CUDA_KERNEL_LOOP(index, count) {\n // The output is in order (n, ctop, ph, pw)\n int pw = index % pooled_width;\n int ph = (index / pooled_width) % pooled_height;\n int ctop = (index / pooled_width / pooled_height) % output_dim;\n int n = index / pooled_width / pooled_height / output_dim;\n\n // [start, end) interval for spatial sampling\n bottom_rois += n * 5;\n int roi_batch_ind = bottom_rois[0];\n DType roi_start_w = static_cast(round(bottom_rois[1])) * spatial_scale;\n DType roi_start_h = static_cast(round(bottom_rois[2])) * spatial_scale;\n DType roi_end_w = static_cast(round(bottom_rois[3]) + 1.) * spatial_scale;\n DType roi_end_h = static_cast(round(bottom_rois[4]) + 1.) * spatial_scale;\n\n // Force too small ROIs to be 1x1\n DType roi_width = max(roi_end_w - roi_start_w, 0.1); //avoid 0\n DType roi_height = max(roi_end_h - roi_start_h, 0.1);\n\n // Compute w and h at bottom\n DType bin_size_h = roi_height / static_cast(pooled_height);\n DType bin_size_w = roi_width / static_cast(pooled_width);\n\n int hstart = floor(static_cast(ph) * bin_size_h\n + roi_start_h);\n int wstart = floor(static_cast(pw)* bin_size_w\n + roi_start_w);\n int hend = ceil(static_cast(ph + 1) * bin_size_h\n + roi_start_h);\n int wend = ceil(static_cast(pw + 1) * bin_size_w\n + roi_start_w);\n // Add roi offsets and clip to input boundaries\n hstart = min(max(hstart, 0), height);\n hend = min(max(hend, 0), height);\n wstart = min(max(wstart, 0),width);\n wend = min(max(wend, 0), width);\n bool is_empty = (hend <= hstart) || (wend <= wstart);\n\n int gw = floor(static_cast(pw)* group_size / pooled_width);\n int gh = floor(static_cast(ph)* group_size / pooled_height);\n gw = min(max(gw, 0), group_size - 1);\n gh = min(max(gh, 0), group_size - 1);\n int c = (ctop*group_size + gh)*group_size + gw;\n\n bottom_data += (roi_batch_ind * channels + c) * height * width;\n DType out_sum = 0;\n for (int h = hstart; h < hend; ++h){\n for (int w = wstart; w < wend; ++w){\n int bottom_index = h*width + w;\n out_sum += bottom_data[bottom_index];\n }\n }\n\n DType bin_area = (hend - hstart)*(wend - wstart);\n top_data[index] = is_empty? (DType)0. : out_sum/bin_area;\n mapping_channel[index] = c;\n }\n}\n\ntemplate\ninline void PSROIPoolForward(const Tensor &out,\n const Tensor &data,\n const Tensor &bbox,\n const Tensor &mapping_channel,\n const float spatial_scale,\n const int output_dim_,\n const int group_size_) {\n // LOG(INFO) << \"PSROIPoolForward\";\n const DType *bottom_data = data.dptr_;\n const DType *bottom_rois = bbox.dptr_;\n DType *top_data = out.dptr_;\n DType *mapping_channel_ptr = mapping_channel.dptr_;\n const int count = out.shape_.Size();\n const int channels = data.size(1);\n const int height = data.size(2);\n const int width = data.size(3);\n const int pooled_height = out.size(2);\n const int pooled_width = out.size(3);\n cudaStream_t stream = Stream::GetStream(out.stream_);\n PSROIPoolForwardKernel << > >(\n count, bottom_data, spatial_scale, channels, height, width,\n pooled_height, pooled_width, bottom_rois, output_dim_, group_size_, top_data, mapping_channel_ptr);\n PSROIPOOLING_CUDA_CHECK(cudaPeekAtLastError());\n}\n\n\ntemplate \n__global__ void PSROIPoolBackwardAccKernel(\n const int count,\n const DType* top_diff,\n const DType* mapping_channel,\n const int num_rois,\n const DType spatial_scale,\n const int channels,\n const int height, const int width,\n const int pooled_height, const int pooled_width,\n const int output_dim,\n DType* bottom_diff,\n const DType* bottom_rois) {\n CUDA_KERNEL_LOOP(index, count) {\n // The output is in order (n, ctop, ph, pw)\n int pw = index % pooled_width;\n int ph = (index / pooled_width) % pooled_height;\n int n = index / pooled_width / pooled_height / output_dim;\n\n // [start, end) interval for spatial sampling\n bottom_rois += n * 5;\n int roi_batch_ind = bottom_rois[0];\n DType roi_start_w = static_cast(round(bottom_rois[1])) * spatial_scale;\n DType roi_start_h = static_cast(round(bottom_rois[2])) * spatial_scale;\n DType roi_end_w = static_cast(round(bottom_rois[3]) + 1.) * spatial_scale;\n DType roi_end_h = static_cast(round(bottom_rois[4]) + 1.) * spatial_scale;\n\n // Force too small ROIs to be 1x1\n DType roi_width = max(roi_end_w - roi_start_w, 0.1); //avoid 0\n DType roi_height = max(roi_end_h - roi_start_h, 0.1);\n\n // Compute w and h at bottom\n DType bin_size_h = roi_height / static_cast(pooled_height);\n DType bin_size_w = roi_width / static_cast(pooled_width);\n\n int hstart = floor(static_cast(ph)* bin_size_h\n + roi_start_h);\n int wstart = floor(static_cast(pw)* bin_size_w\n + roi_start_w);\n int hend = ceil(static_cast(ph + 1) * bin_size_h\n + roi_start_h);\n int wend = ceil(static_cast(pw + 1) * bin_size_w\n + roi_start_w);\n // Add roi offsets and clip to input boundaries\n hstart = min(max(hstart, 0), height);\n hend = min(max(hend, 0), height);\n wstart = min(max(wstart, 0), width);\n wend = min(max(wend, 0), width);\n bool is_empty = (hend <= hstart) || (wend <= wstart);\n\n // Compute c at bottom\n int c = mapping_channel[index];\n DType* offset_bottom_diff = bottom_diff + (roi_batch_ind * channels + c) * height * width;\n DType bin_area = (hend - hstart)*(wend - wstart);\n DType diff_val = is_empty ? (DType)0. : top_diff[index] / bin_area;\n for (int h = hstart; h < hend; ++h){\n for (int w = wstart; w < wend; ++w){\n int bottom_index = h*width + w;\n // mxnet_gpu_atomic_add(diff_val, offset_bottom_diff + bottom_index);\n atomicAdd(offset_bottom_diff + bottom_index, diff_val);\n }\n }\n }\n}\n\n\ntemplate\ninline void PSROIPoolBackwardAcc(const Tensor &in_grad,\n const Tensor &out_grad,\n const Tensor &bbox,\n const Tensor &mapping_channel,\n const float spatial_scale,\n const int output_dim_) {\n // LOG(INFO) << \"PSROIPoolBackward\";\n const DType *top_diff = out_grad.dptr_;\n const DType *bottom_rois = bbox.dptr_;\n DType *bottom_diff = in_grad.dptr_;\n DType *mapping_channel_ptr = mapping_channel.dptr_;\n const int count = out_grad.shape_.Size();\n const int num_rois = bbox.size(0);\n const int channels = in_grad.size(1);\n const int height = in_grad.size(2);\n const int width = in_grad.size(3);\n const int pooled_height = out_grad.size(2);\n const int pooled_width = out_grad.size(3);\n cudaStream_t stream = Stream::GetStream(in_grad.stream_);\n PSROIPoolBackwardAccKernel << > >(\n count, top_diff, mapping_channel_ptr, num_rois, spatial_scale, channels, height, width,\n pooled_height, pooled_width, output_dim_, bottom_diff, bottom_rois);\n PSROIPOOLING_CUDA_CHECK(cudaPeekAtLastError());\n}\n\n} // namespace cuda\n\ntemplate\ninline void PSROIPoolForward(const Tensor &out,\n const Tensor &data,\n const Tensor &bbox,\n const Tensor &mapping_channel,\n const float spatial_scale,\n const int output_dim_,\n const int group_size_) {\n cuda::PSROIPoolForward(out, data, bbox, mapping_channel, spatial_scale, output_dim_, group_size_);\n}\n\ntemplate\ninline void PSROIPoolBackwardAcc(const Tensor &in_grad,\n const Tensor &out_grad,\n const Tensor &bbox,\n const Tensor &mapping_channel,\n const float spatial_scale,\n const int output_dim_) {\n cuda::PSROIPoolBackwardAcc(in_grad, out_grad, bbox, mapping_channel, spatial_scale, output_dim_);\n}\n\n} // namespace mshadow\n\n\nnamespace mxnet {\nnamespace op {\n\ntemplate<>\nOperator* CreateOp(PSROIPoolingParam param, int dtype) {\n Operator* op = NULL;\n MSHADOW_REAL_TYPE_SWITCH(dtype, DType, {\n op = new PSROIPoolingOp(param);\n });\n return op;\n}\n\n} // namespace op\n} // namespace mxnet" }, { "path": "dff_rfcn/operator_py/__init__.py", "content": "" }, { "path": "dff_rfcn/operator_py/box_annotator_ohem.py", "content": "# --------------------------------------------------------\n# Deep Feature Flow\n# Copyright (c) 2017 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Written by Yuwen Xiong\n# --------------------------------------------------------\n\n\"\"\"\nProposal Target Operator selects foreground and background roi and assigns label, bbox_transform to them.\n\"\"\"\n\nimport mxnet as mx\nimport numpy as np\nfrom distutils.util import strtobool\n\n\n\n\nclass BoxAnnotatorOHEMOperator(mx.operator.CustomOp):\n def __init__(self, num_classes, num_reg_classes, roi_per_img):\n super(BoxAnnotatorOHEMOperator, self).__init__()\n self._num_classes = num_classes\n self._num_reg_classes = num_reg_classes\n self._roi_per_img = roi_per_img\n\n def forward(self, is_train, req, in_data, out_data, aux):\n\n cls_score = in_data[0]\n bbox_pred = in_data[1]\n labels = in_data[2].asnumpy()\n bbox_targets = in_data[3]\n bbox_weights = in_data[4]\n\n per_roi_loss_cls = mx.nd.SoftmaxActivation(cls_score) + 1e-14\n per_roi_loss_cls = per_roi_loss_cls.asnumpy()\n per_roi_loss_cls = per_roi_loss_cls[np.arange(per_roi_loss_cls.shape[0], dtype='int'), labels.astype('int')]\n per_roi_loss_cls = -1 * np.log(per_roi_loss_cls)\n per_roi_loss_cls = np.reshape(per_roi_loss_cls, newshape=(-1,))\n\n per_roi_loss_bbox = bbox_weights * mx.nd.smooth_l1((bbox_pred - bbox_targets), scalar=1.0)\n per_roi_loss_bbox = mx.nd.sum(per_roi_loss_bbox, axis=1).asnumpy()\n\n top_k_per_roi_loss = np.argsort(per_roi_loss_cls + per_roi_loss_bbox)\n labels_ohem = labels\n labels_ohem[top_k_per_roi_loss[::-1][self._roi_per_img:]] = -1\n bbox_weights_ohem = bbox_weights.asnumpy()\n bbox_weights_ohem[top_k_per_roi_loss[::-1][self._roi_per_img:]] = 0\n\n labels_ohem = mx.nd.array(labels_ohem)\n bbox_weights_ohem = mx.nd.array(bbox_weights_ohem)\n\n for ind, val in enumerate([labels_ohem, bbox_weights_ohem]):\n self.assign(out_data[ind], req[ind], val)\n\n\n def backward(self, req, out_grad, in_data, out_data, in_grad, aux):\n for i in range(len(in_grad)):\n self.assign(in_grad[i], req[i], 0)\n\n\n@mx.operator.register('BoxAnnotatorOHEM')\nclass BoxAnnotatorOHEMProp(mx.operator.CustomOpProp):\n def __init__(self, num_classes, num_reg_classes, roi_per_img):\n super(BoxAnnotatorOHEMProp, self).__init__(need_top_grad=False)\n self._num_classes = int(num_classes)\n self._num_reg_classes = int(num_reg_classes)\n self._roi_per_img = int(roi_per_img)\n\n def list_arguments(self):\n return ['cls_score', 'bbox_pred', 'labels', 'bbox_targets', 'bbox_weights']\n\n def list_outputs(self):\n return ['labels_ohem', 'bbox_weights_ohem']\n\n def infer_shape(self, in_shape):\n labels_shape = in_shape[2]\n bbox_weights_shape = in_shape[4]\n\n return in_shape, \\\n [labels_shape, bbox_weights_shape]\n\n def create_operator(self, ctx, shapes, dtypes):\n return BoxAnnotatorOHEMOperator(self._num_classes, self._num_reg_classes, self._roi_per_img)\n\n def declare_backward_dependency(self, out_grad, in_data, out_data):\n return []\n" }, { "path": "dff_rfcn/operator_py/proposal.py", "content": "# --------------------------------------------------------\n# Deep Feature Flow\n# Copyright (c) 2017 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Modified by Yuwen Xiong\n# --------------------------------------------------------\n# Based on:\n# MX-RCNN\n# Copyright (c) 2016 by Contributors\n# Licence under The Apache 2.0 License\n# https://github.com/ijkguo/mx-rcnn/\n# --------------------------------------------------------\n\n\"\"\"\nProposal Operator transform anchor coordinates into ROI coordinates with prediction results on\nclassification probability and bounding box prediction results, and image size and scale information.\n\"\"\"\n\nimport mxnet as mx\nimport numpy as np\nimport numpy.random as npr\nfrom distutils.util import strtobool\n\nfrom bbox.bbox_transform import bbox_pred, clip_boxes\nfrom rpn.generate_anchor import generate_anchors\nfrom nms.nms import py_nms_wrapper, cpu_nms_wrapper, gpu_nms_wrapper\n\nDEBUG = False\n\n\nclass ProposalOperator(mx.operator.CustomOp):\n def __init__(self, feat_stride, scales, ratios, output_score,\n rpn_pre_nms_top_n, rpn_post_nms_top_n, threshold, rpn_min_size):\n super(ProposalOperator, self).__init__()\n self._feat_stride = feat_stride\n self._scales = np.fromstring(scales[1:-1], dtype=float, sep=',')\n self._ratios = np.fromstring(ratios[1:-1], dtype=float, sep=',')\n self._anchors = generate_anchors(base_size=self._feat_stride, scales=self._scales, ratios=self._ratios)\n self._num_anchors = self._anchors.shape[0]\n self._output_score = output_score\n self._rpn_pre_nms_top_n = rpn_pre_nms_top_n\n self._rpn_post_nms_top_n = rpn_post_nms_top_n\n self._threshold = threshold\n self._rpn_min_size = rpn_min_size\n\n if DEBUG:\n print 'feat_stride: {}'.format(self._feat_stride)\n print 'anchors:'\n print self._anchors\n\n def forward(self, is_train, req, in_data, out_data, aux):\n nms = gpu_nms_wrapper(self._threshold, in_data[0].context.device_id)\n\n batch_size = in_data[0].shape[0]\n if batch_size > 1:\n raise ValueError(\"Sorry, multiple images each device is not implemented\")\n\n # for each (H, W) location i\n # generate A anchor boxes centered on cell i\n # apply predicted bbox deltas at cell i to each of the A anchors\n # clip predicted boxes to image\n # remove predicted boxes with either height or width < threshold\n # sort all (proposal, score) pairs by score from highest to lowest\n # take top pre_nms_topN proposals before NMS\n # apply NMS with threshold 0.7 to remaining proposals\n # take after_nms_topN proposals after NMS\n # return the top proposals (-> RoIs top, scores top)\n\n pre_nms_topN = self._rpn_pre_nms_top_n\n post_nms_topN = self._rpn_post_nms_top_n\n min_size = self._rpn_min_size\n\n # the first set of anchors are background probabilities\n # keep the second part\n scores = in_data[0].asnumpy()[:, self._num_anchors:, :, :]\n bbox_deltas = in_data[1].asnumpy()\n im_info = in_data[2].asnumpy()[0, :]\n\n if DEBUG:\n print 'im_size: ({}, {})'.format(im_info[0], im_info[1])\n print 'scale: {}'.format(im_info[2])\n\n # 1. Generate proposals from bbox_deltas and shifted anchors\n # use real image size instead of padded feature map sizes\n height, width = int(im_info[0] / self._feat_stride), int(im_info[1] / self._feat_stride)\n\n if DEBUG:\n print 'score map size: {}'.format(scores.shape)\n print \"resudial: {}\".format((scores.shape[2] - height, scores.shape[3] - width))\n\n # Enumerate all shifts\n shift_x = np.arange(0, width) * self._feat_stride\n shift_y = np.arange(0, height) * self._feat_stride\n shift_x, shift_y = np.meshgrid(shift_x, shift_y)\n shifts = np.vstack((shift_x.ravel(), shift_y.ravel(), shift_x.ravel(), shift_y.ravel())).transpose()\n\n # Enumerate all shifted anchors:\n #\n # add A anchors (1, A, 4) to\n # cell K shifts (K, 1, 4) to get\n # shift anchors (K, A, 4)\n # reshape to (K*A, 4) shifted anchors\n A = self._num_anchors\n K = shifts.shape[0]\n anchors = self._anchors.reshape((1, A, 4)) + shifts.reshape((1, K, 4)).transpose((1, 0, 2))\n anchors = anchors.reshape((K * A, 4))\n\n # Transpose and reshape predicted bbox transformations to get them\n # into the same order as the anchors:\n #\n # bbox deltas will be (1, 4 * A, H, W) format\n # transpose to (1, H, W, 4 * A)\n # reshape to (1 * H * W * A, 4) where rows are ordered by (h, w, a)\n # in slowest to fastest order\n bbox_deltas = self._clip_pad(bbox_deltas, (height, width))\n bbox_deltas = bbox_deltas.transpose((0, 2, 3, 1)).reshape((-1, 4))\n\n # Same story for the scores:\n #\n # scores are (1, A, H, W) format\n # transpose to (1, H, W, A)\n # reshape to (1 * H * W * A, 1) where rows are ordered by (h, w, a)\n scores = self._clip_pad(scores, (height, width))\n scores = scores.transpose((0, 2, 3, 1)).reshape((-1, 1))\n\n # Convert anchors into proposals via bbox transformations\n proposals = bbox_pred(anchors, bbox_deltas)\n\n # 2. clip predicted boxes to image\n proposals = clip_boxes(proposals, im_info[:2])\n\n # 3. remove predicted boxes with either height or width < threshold\n # (NOTE: convert min_size to input image scale stored in im_info[2])\n keep = self._filter_boxes(proposals, min_size * im_info[2])\n proposals = proposals[keep, :]\n scores = scores[keep]\n\n # 4. sort all (proposal, score) pairs by score from highest to lowest\n # 5. take top pre_nms_topN (e.g. 6000)\n order = scores.ravel().argsort()[::-1]\n if pre_nms_topN > 0:\n order = order[:pre_nms_topN]\n proposals = proposals[order, :]\n scores = scores[order]\n\n # 6. apply nms (e.g. threshold = 0.7)\n # 7. take after_nms_topN (e.g. 300)\n # 8. return the top proposals (-> RoIs top)\n det = np.hstack((proposals, scores)).astype(np.float32)\n keep = nms(det)\n if post_nms_topN > 0:\n keep = keep[:post_nms_topN]\n # pad to ensure output size remains unchanged\n if len(keep) < post_nms_topN:\n pad = npr.choice(keep, size=post_nms_topN - len(keep))\n keep = np.hstack((keep, pad))\n proposals = proposals[keep, :]\n scores = scores[keep]\n\n # Output rois array\n # Our RPN implementation only supports a single input image, so all\n # batch inds are 0\n batch_inds = np.zeros((proposals.shape[0], 1), dtype=np.float32)\n blob = np.hstack((batch_inds, proposals.astype(np.float32, copy=False)))\n self.assign(out_data[0], req[0], blob)\n\n if self._output_score:\n self.assign(out_data[1], req[1], scores.astype(np.float32, copy=False))\n\n def backward(self, req, out_grad, in_data, out_data, in_grad, aux):\n self.assign(in_grad[0], req[0], 0)\n self.assign(in_grad[1], req[1], 0)\n self.assign(in_grad[2], req[2], 0)\n\n @staticmethod\n def _filter_boxes(boxes, min_size):\n \"\"\" Remove all boxes with any side smaller than min_size \"\"\"\n ws = boxes[:, 2] - boxes[:, 0] + 1\n hs = boxes[:, 3] - boxes[:, 1] + 1\n keep = np.where((ws >= min_size) & (hs >= min_size))[0]\n return keep\n\n @staticmethod\n def _clip_pad(tensor, pad_shape):\n \"\"\"\n Clip boxes of the pad area.\n :param tensor: [n, c, H, W]\n :param pad_shape: [h, w]\n :return: [n, c, h, w]\n \"\"\"\n H, W = tensor.shape[2:]\n h, w = pad_shape\n\n if h < H or w < W:\n tensor = tensor[:, :, :h, :w].copy()\n\n return tensor\n\n\n@mx.operator.register(\"proposal\")\nclass ProposalProp(mx.operator.CustomOpProp):\n def __init__(self, feat_stride='16', scales='(8, 16, 32)', ratios='(0.5, 1, 2)', output_score='False',\n rpn_pre_nms_top_n='6000', rpn_post_nms_top_n='300', threshold='0.3', rpn_min_size='16'):\n super(ProposalProp, self).__init__(need_top_grad=False)\n self._feat_stride = int(feat_stride)\n self._scales = scales\n self._ratios = ratios\n self._output_score = strtobool(output_score)\n self._rpn_pre_nms_top_n = int(rpn_pre_nms_top_n)\n self._rpn_post_nms_top_n = int(rpn_post_nms_top_n)\n self._threshold = float(threshold)\n self._rpn_min_size = int(rpn_min_size)\n\n def list_arguments(self):\n return ['cls_prob', 'bbox_pred', 'im_info']\n\n def list_outputs(self):\n if self._output_score:\n return ['output', 'score']\n else:\n return ['output']\n\n def infer_shape(self, in_shape):\n cls_prob_shape = in_shape[0]\n bbox_pred_shape = in_shape[1]\n assert cls_prob_shape[0] == bbox_pred_shape[0], 'ROI number does not equal in cls and reg'\n\n batch_size = cls_prob_shape[0]\n im_info_shape = (batch_size, 3)\n output_shape = (self._rpn_post_nms_top_n, 5)\n score_shape = (self._rpn_post_nms_top_n, 1)\n\n if self._output_score:\n return [cls_prob_shape, bbox_pred_shape, im_info_shape], [output_shape, score_shape]\n else:\n return [cls_prob_shape, bbox_pred_shape, im_info_shape], [output_shape]\n\n def create_operator(self, ctx, shapes, dtypes):\n return ProposalOperator(self._feat_stride, self._scales, self._ratios, self._output_score,\n self._rpn_pre_nms_top_n, self._rpn_post_nms_top_n, self._threshold, self._rpn_min_size)\n\n def declare_backward_dependency(self, out_grad, in_data, out_data):\n return []\n" }, { "path": "dff_rfcn/operator_py/proposal_target.py", "content": "# --------------------------------------------------------\n# Deep Feature Flow\n# Copyright (c) 2017 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Modified by Yuwen Xiong\n# --------------------------------------------------------\n# Based on:\n# MX-RCNN\n# Copyright (c) 2016 by Contributors\n# Licence under The Apache 2.0 License\n# https://github.com/ijkguo/mx-rcnn/\n# --------------------------------------------------------\n\n\"\"\"\nProposal Target Operator selects foreground and background roi and assigns label, bbox_transform to them.\n\"\"\"\n\nimport mxnet as mx\nimport numpy as np\nfrom distutils.util import strtobool\nfrom easydict import EasyDict as edict\nimport cPickle\n\n\nfrom core.rcnn import sample_rois\n\nDEBUG = False\n\n\nclass ProposalTargetOperator(mx.operator.CustomOp):\n def __init__(self, num_classes, batch_images, batch_rois, cfg, fg_fraction):\n super(ProposalTargetOperator, self).__init__()\n self._num_classes = num_classes\n self._batch_images = batch_images\n self._batch_rois = batch_rois\n self._cfg = cfg\n self._fg_fraction = fg_fraction\n\n if DEBUG:\n self._count = 0\n self._fg_num = 0\n self._bg_num = 0\n\n def forward(self, is_train, req, in_data, out_data, aux):\n assert self._batch_rois == -1 or self._batch_rois % self._batch_images == 0, \\\n 'batchimages {} must devide batch_rois {}'.format(self._batch_images, self._batch_rois)\n all_rois = in_data[0].asnumpy()\n gt_boxes = in_data[1].asnumpy()\n\n if self._batch_rois == -1:\n rois_per_image = all_rois.shape[0] + gt_boxes.shape[0]\n fg_rois_per_image = rois_per_image\n else:\n rois_per_image = self._batch_rois / self._batch_images\n fg_rois_per_image = np.round(self._fg_fraction * rois_per_image).astype(int)\n\n\n # Include ground-truth boxes in the set of candidate rois\n zeros = np.zeros((gt_boxes.shape[0], 1), dtype=gt_boxes.dtype)\n all_rois = np.vstack((all_rois, np.hstack((zeros, gt_boxes[:, :-1]))))\n # Sanity check: single batch only\n assert np.all(all_rois[:, 0] == 0), 'Only single item batches are supported'\n\n rois, labels, bbox_targets, bbox_weights = \\\n sample_rois(all_rois, fg_rois_per_image, rois_per_image, self._num_classes, self._cfg, gt_boxes=gt_boxes)\n\n if DEBUG:\n print \"labels=\", labels\n print 'num fg: {}'.format((labels > 0).sum())\n print 'num bg: {}'.format((labels == 0).sum())\n self._count += 1\n self._fg_num += (labels > 0).sum()\n self._bg_num += (labels == 0).sum()\n print \"self._count=\", self._count\n print 'num fg avg: {}'.format(self._fg_num / self._count)\n print 'num bg avg: {}'.format(self._bg_num / self._count)\n print 'ratio: {:.3f}'.format(float(self._fg_num) / float(self._bg_num))\n\n for ind, val in enumerate([rois, labels, bbox_targets, bbox_weights]):\n self.assign(out_data[ind], req[ind], val)\n\n def backward(self, req, out_grad, in_data, out_data, in_grad, aux):\n self.assign(in_grad[0], req[0], 0)\n self.assign(in_grad[1], req[1], 0)\n\n\n@mx.operator.register('proposal_target')\nclass ProposalTargetProp(mx.operator.CustomOpProp):\n def __init__(self, num_classes, batch_images, batch_rois, cfg, fg_fraction='0.25'):\n super(ProposalTargetProp, self).__init__(need_top_grad=False)\n self._num_classes = int(num_classes)\n self._batch_images = int(batch_images)\n self._batch_rois = int(batch_rois)\n self._cfg = cPickle.loads(cfg)\n self._fg_fraction = float(fg_fraction)\n\n def list_arguments(self):\n return ['rois', 'gt_boxes']\n\n def list_outputs(self):\n return ['rois_output', 'label', 'bbox_target', 'bbox_weight']\n\n def infer_shape(self, in_shape):\n rpn_rois_shape = in_shape[0]\n gt_boxes_shape = in_shape[1]\n\n rois = rpn_rois_shape[0] + gt_boxes_shape[0] if self._batch_rois == -1 else self._batch_rois\n\n output_rois_shape = (rois, 5)\n label_shape = (rois, )\n bbox_target_shape = (rois, self._num_classes * 4)\n bbox_weight_shape = (rois, self._num_classes * 4)\n\n return [rpn_rois_shape, gt_boxes_shape], \\\n [output_rois_shape, label_shape, bbox_target_shape, bbox_weight_shape]\n\n def create_operator(self, ctx, shapes, dtypes):\n return ProposalTargetOperator(self._num_classes, self._batch_images, self._batch_rois, self._cfg, self._fg_fraction)\n\n def declare_backward_dependency(self, out_grad, in_data, out_data):\n return []\n" }, { "path": "dff_rfcn/operator_py/rpn_inv_normalize.py", "content": "# --------------------------------------------------------\n# Deep Feature Flow\n# Copyright (c) 2017 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Written by Xizhou Zhu\n# --------------------------------------------------------\n\nimport mxnet as mx\nimport numpy as np\nfrom distutils.util import strtobool\n\nclass RPNInvNormalizeOperator(mx.operator.CustomOp):\n def __init__(self, num_anchors, bbox_mean, bbox_std):\n super(RPNInvNormalizeOperator, self).__init__()\n self._num_anchors = num_anchors\n self._bbox_mean = mx.ndarray.Reshape(mx.nd.array(bbox_mean), shape=(1,4,1,1))\n self._bbox_std = mx.ndarray.Reshape(mx.nd.array(bbox_std), shape=(1,4,1,1))\n\n def forward(self, is_train, req, in_data, out_data, aux):\n bbox_pred = in_data[0]\n tile_shape = (bbox_pred.shape[0], self._num_anchors, bbox_pred.shape[2], bbox_pred.shape[3])\n bbox_mean = mx.ndarray.tile(self._bbox_mean.as_in_context(bbox_pred.context), reps=tile_shape)\n bbox_std = mx.ndarray.tile(self._bbox_std.as_in_context(bbox_pred.context), reps=tile_shape)\n bbox_pred = bbox_pred * bbox_std + bbox_mean\n\n self.assign(out_data[0], req[0], bbox_pred)\n\n def backward(self, req, out_grad, in_data, out_data, in_grad, aux):\n self.assign(in_grad[0], req[0], 0)\n\n@mx.operator.register('rpn_inv_normalize')\nclass RPNInvNormalizeProp(mx.operator.CustomOpProp):\n def __init__(self, num_anchors, bbox_mean='(0.0, 0.0, 0.0, 0.0)', bbox_std='0.1, 0.1, 0.2, 0.2'):\n super(RPNInvNormalizeProp, self).__init__(need_top_grad=False)\n self._num_anchors = int(num_anchors)\n self._bbox_mean = np.fromstring(bbox_mean[1:-1], dtype=float, sep=',')\n self._bbox_std = np.fromstring(bbox_std[1:-1], dtype=float, sep=',')\n\n def list_arguments(self):\n return ['bbox_pred']\n\n def list_outputs(self):\n return ['out_bbox_pred']\n\n def infer_shape(self, in_shape):\n\n return [in_shape[0]], \\\n [in_shape[0]]\n\n def create_operator(self, ctx, shapes, dtypes):\n return RPNInvNormalizeOperator(self._num_anchors, self._bbox_mean, self._bbox_std)\n\n def declare_backward_dependency(self, out_grad, in_data, out_data):\n return []\n" }, { "path": "dff_rfcn/operator_py/tile_as.py", "content": "# --------------------------------------------------------\n# Deep Feature Flow\n# Copyright (c) 2017 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Written by Xizhou Zhu\n# --------------------------------------------------------\n\nimport mxnet as mx\nimport numpy as np\nfrom distutils.util import strtobool\n\nclass TileAsOperator(mx.operator.CustomOp):\n def __init__(self):\n super(TileAsOperator, self).__init__()\n\n def forward(self, is_train, req, in_data, out_data, aux):\n data_content = in_data[0]\n data_tiled = mx.ndarray.tile(data_content, reps=(in_data[1].shape[0], 1, 1, 1))\n self.assign(out_data[0], req[0], data_tiled)\n\n def backward(self, req, out_grad, in_data, out_data, in_grad, aux):\n self.assign(in_grad[0], req[0], 0)\n self.assign(in_grad[1], req[1], 0)\n\n\n@mx.operator.register('tile_as')\nclass TileAsProp(mx.operator.CustomOpProp):\n def __init__(self):\n super(TileAsProp, self).__init__(need_top_grad=False)\n\n def list_arguments(self):\n return ['data_content', 'data_shape']\n\n def list_outputs(self):\n return ['data_tiled']\n\n def infer_shape(self, in_shape):\n data_content_shape = in_shape[0]\n data_shape_shape = in_shape[1]\n\n tiled_data_shape = (data_shape_shape[0], data_content_shape[1], data_content_shape[2], data_content_shape[3])\n\n return [data_content_shape, data_shape_shape], \\\n [tiled_data_shape]\n\n def create_operator(self, ctx, shapes, dtypes):\n return TileAsOperator()\n\n def declare_backward_dependency(self, out_grad, in_data, out_data):\n return out_grad\n" }, { "path": "dff_rfcn/symbols/__init__.py", "content": "import resnet_v1_101_flownet_rfcn" }, { "path": "dff_rfcn/symbols/resnet_v1_101_flownet_rfcn.py", "content": "# --------------------------------------------------------\n# Deep Feature Flow\n# Copyright (c) 2017 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Written by Yuwen Xiong, Xizhou Zhu\n# --------------------------------------------------------\n\nimport cPickle\nimport mxnet as mx\nfrom utils.symbol import Symbol\nfrom operator_py.proposal import *\nfrom operator_py.proposal_target import *\nfrom operator_py.box_annotator_ohem import *\nfrom operator_py.rpn_inv_normalize import *\nfrom operator_py.tile_as import *\n\nclass resnet_v1_101_flownet_rfcn(Symbol):\n\n def __init__(self):\n \"\"\"\n Use __init__ to define parameter network needs\n \"\"\"\n self.eps = 1e-5\n self.use_global_stats = True\n self.workspace = 512\n self.units = (3, 4, 23, 3) # use for 101\n self.filter_list = [256, 512, 1024, 2048]\n\n def get_resnet_v1(self, data):\n conv1 = mx.symbol.Convolution(name='conv1', data=data , num_filter=64, pad=(3,3), kernel=(7,7), stride=(2,2), no_bias=True)\n bn_conv1 = mx.symbol.BatchNorm(name='bn_conv1', data=conv1 , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale_conv1 = bn_conv1\n conv1_relu = mx.symbol.Activation(name='conv1_relu', data=scale_conv1 , act_type='relu')\n pool1 = mx.symbol.Pooling(name='pool1', data=conv1_relu , pad=(1,1), kernel=(3,3), stride=(2,2), pool_type='max')\n res2a_branch1 = mx.symbol.Convolution(name='res2a_branch1', data=pool1 , num_filter=256, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn2a_branch1 = mx.symbol.BatchNorm(name='bn2a_branch1', data=res2a_branch1 , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale2a_branch1 = bn2a_branch1\n res2a_branch2a = mx.symbol.Convolution(name='res2a_branch2a', data=pool1 , num_filter=64, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn2a_branch2a = mx.symbol.BatchNorm(name='bn2a_branch2a', data=res2a_branch2a , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale2a_branch2a = bn2a_branch2a\n res2a_branch2a_relu = mx.symbol.Activation(name='res2a_branch2a_relu', data=scale2a_branch2a , act_type='relu')\n res2a_branch2b = mx.symbol.Convolution(name='res2a_branch2b', data=res2a_branch2a_relu , num_filter=64, pad=(1,1), kernel=(3,3), stride=(1,1), no_bias=True)\n bn2a_branch2b = mx.symbol.BatchNorm(name='bn2a_branch2b', data=res2a_branch2b , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale2a_branch2b = bn2a_branch2b\n res2a_branch2b_relu = mx.symbol.Activation(name='res2a_branch2b_relu', data=scale2a_branch2b , act_type='relu')\n res2a_branch2c = mx.symbol.Convolution(name='res2a_branch2c', data=res2a_branch2b_relu , num_filter=256, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn2a_branch2c = mx.symbol.BatchNorm(name='bn2a_branch2c', data=res2a_branch2c , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale2a_branch2c = bn2a_branch2c\n res2a = mx.symbol.broadcast_add(name='res2a', *[scale2a_branch1,scale2a_branch2c] )\n res2a_relu = mx.symbol.Activation(name='res2a_relu', data=res2a , act_type='relu')\n res2b_branch2a = mx.symbol.Convolution(name='res2b_branch2a', data=res2a_relu , num_filter=64, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn2b_branch2a = mx.symbol.BatchNorm(name='bn2b_branch2a', data=res2b_branch2a , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale2b_branch2a = bn2b_branch2a\n res2b_branch2a_relu = mx.symbol.Activation(name='res2b_branch2a_relu', data=scale2b_branch2a , act_type='relu')\n res2b_branch2b = mx.symbol.Convolution(name='res2b_branch2b', data=res2b_branch2a_relu , num_filter=64, pad=(1,1), kernel=(3,3), stride=(1,1), no_bias=True)\n bn2b_branch2b = mx.symbol.BatchNorm(name='bn2b_branch2b', data=res2b_branch2b , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale2b_branch2b = bn2b_branch2b\n res2b_branch2b_relu = mx.symbol.Activation(name='res2b_branch2b_relu', data=scale2b_branch2b , act_type='relu')\n res2b_branch2c = mx.symbol.Convolution(name='res2b_branch2c', data=res2b_branch2b_relu , num_filter=256, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn2b_branch2c = mx.symbol.BatchNorm(name='bn2b_branch2c', data=res2b_branch2c , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale2b_branch2c = bn2b_branch2c\n res2b = mx.symbol.broadcast_add(name='res2b', *[res2a_relu,scale2b_branch2c] )\n res2b_relu = mx.symbol.Activation(name='res2b_relu', data=res2b , act_type='relu')\n res2c_branch2a = mx.symbol.Convolution(name='res2c_branch2a', data=res2b_relu , num_filter=64, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn2c_branch2a = mx.symbol.BatchNorm(name='bn2c_branch2a', data=res2c_branch2a , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale2c_branch2a = bn2c_branch2a\n res2c_branch2a_relu = mx.symbol.Activation(name='res2c_branch2a_relu', data=scale2c_branch2a , act_type='relu')\n res2c_branch2b = mx.symbol.Convolution(name='res2c_branch2b', data=res2c_branch2a_relu , num_filter=64, pad=(1,1), kernel=(3,3), stride=(1,1), no_bias=True)\n bn2c_branch2b = mx.symbol.BatchNorm(name='bn2c_branch2b', data=res2c_branch2b , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale2c_branch2b = bn2c_branch2b\n res2c_branch2b_relu = mx.symbol.Activation(name='res2c_branch2b_relu', data=scale2c_branch2b , act_type='relu')\n res2c_branch2c = mx.symbol.Convolution(name='res2c_branch2c', data=res2c_branch2b_relu , num_filter=256, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn2c_branch2c = mx.symbol.BatchNorm(name='bn2c_branch2c', data=res2c_branch2c , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale2c_branch2c = bn2c_branch2c\n res2c = mx.symbol.broadcast_add(name='res2c', *[res2b_relu,scale2c_branch2c] )\n res2c_relu = mx.symbol.Activation(name='res2c_relu', data=res2c , act_type='relu')\n res3a_branch1 = mx.symbol.Convolution(name='res3a_branch1', data=res2c_relu , num_filter=512, pad=(0,0), kernel=(1,1), stride=(2,2), no_bias=True)\n bn3a_branch1 = mx.symbol.BatchNorm(name='bn3a_branch1', data=res3a_branch1 , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale3a_branch1 = bn3a_branch1\n res3a_branch2a = mx.symbol.Convolution(name='res3a_branch2a', data=res2c_relu , num_filter=128, pad=(0,0), kernel=(1,1), stride=(2,2), no_bias=True)\n bn3a_branch2a = mx.symbol.BatchNorm(name='bn3a_branch2a', data=res3a_branch2a , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale3a_branch2a = bn3a_branch2a\n res3a_branch2a_relu = mx.symbol.Activation(name='res3a_branch2a_relu', data=scale3a_branch2a , act_type='relu')\n res3a_branch2b = mx.symbol.Convolution(name='res3a_branch2b', data=res3a_branch2a_relu , num_filter=128, pad=(1,1), kernel=(3,3), stride=(1,1), no_bias=True)\n bn3a_branch2b = mx.symbol.BatchNorm(name='bn3a_branch2b', data=res3a_branch2b , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale3a_branch2b = bn3a_branch2b\n res3a_branch2b_relu = mx.symbol.Activation(name='res3a_branch2b_relu', data=scale3a_branch2b , act_type='relu')\n res3a_branch2c = mx.symbol.Convolution(name='res3a_branch2c', data=res3a_branch2b_relu , num_filter=512, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn3a_branch2c = mx.symbol.BatchNorm(name='bn3a_branch2c', data=res3a_branch2c , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale3a_branch2c = bn3a_branch2c\n res3a = mx.symbol.broadcast_add(name='res3a', *[scale3a_branch1,scale3a_branch2c] )\n res3a_relu = mx.symbol.Activation(name='res3a_relu', data=res3a , act_type='relu')\n res3b1_branch2a = mx.symbol.Convolution(name='res3b1_branch2a', data=res3a_relu , num_filter=128, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn3b1_branch2a = mx.symbol.BatchNorm(name='bn3b1_branch2a', data=res3b1_branch2a , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale3b1_branch2a = bn3b1_branch2a\n res3b1_branch2a_relu = mx.symbol.Activation(name='res3b1_branch2a_relu', data=scale3b1_branch2a , act_type='relu')\n res3b1_branch2b = mx.symbol.Convolution(name='res3b1_branch2b', data=res3b1_branch2a_relu , num_filter=128, pad=(1,1), kernel=(3,3), stride=(1,1), no_bias=True)\n bn3b1_branch2b = mx.symbol.BatchNorm(name='bn3b1_branch2b', data=res3b1_branch2b , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale3b1_branch2b = bn3b1_branch2b\n res3b1_branch2b_relu = mx.symbol.Activation(name='res3b1_branch2b_relu', data=scale3b1_branch2b , act_type='relu')\n res3b1_branch2c = mx.symbol.Convolution(name='res3b1_branch2c', data=res3b1_branch2b_relu , num_filter=512, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn3b1_branch2c = mx.symbol.BatchNorm(name='bn3b1_branch2c', data=res3b1_branch2c , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale3b1_branch2c = bn3b1_branch2c\n res3b1 = mx.symbol.broadcast_add(name='res3b1', *[res3a_relu,scale3b1_branch2c] )\n res3b1_relu = mx.symbol.Activation(name='res3b1_relu', data=res3b1 , act_type='relu')\n res3b2_branch2a = mx.symbol.Convolution(name='res3b2_branch2a', data=res3b1_relu , num_filter=128, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn3b2_branch2a = mx.symbol.BatchNorm(name='bn3b2_branch2a', data=res3b2_branch2a , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale3b2_branch2a = bn3b2_branch2a\n res3b2_branch2a_relu = mx.symbol.Activation(name='res3b2_branch2a_relu', data=scale3b2_branch2a , act_type='relu')\n res3b2_branch2b = mx.symbol.Convolution(name='res3b2_branch2b', data=res3b2_branch2a_relu , num_filter=128, pad=(1,1), kernel=(3,3), stride=(1,1), no_bias=True)\n bn3b2_branch2b = mx.symbol.BatchNorm(name='bn3b2_branch2b', data=res3b2_branch2b , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale3b2_branch2b = bn3b2_branch2b\n res3b2_branch2b_relu = mx.symbol.Activation(name='res3b2_branch2b_relu', data=scale3b2_branch2b , act_type='relu')\n res3b2_branch2c = mx.symbol.Convolution(name='res3b2_branch2c', data=res3b2_branch2b_relu , num_filter=512, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn3b2_branch2c = mx.symbol.BatchNorm(name='bn3b2_branch2c', data=res3b2_branch2c , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale3b2_branch2c = bn3b2_branch2c\n res3b2 = mx.symbol.broadcast_add(name='res3b2', *[res3b1_relu,scale3b2_branch2c] )\n res3b2_relu = mx.symbol.Activation(name='res3b2_relu', data=res3b2 , act_type='relu')\n res3b3_branch2a = mx.symbol.Convolution(name='res3b3_branch2a', data=res3b2_relu , num_filter=128, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn3b3_branch2a = mx.symbol.BatchNorm(name='bn3b3_branch2a', data=res3b3_branch2a , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale3b3_branch2a = bn3b3_branch2a\n res3b3_branch2a_relu = mx.symbol.Activation(name='res3b3_branch2a_relu', data=scale3b3_branch2a , act_type='relu')\n res3b3_branch2b = mx.symbol.Convolution(name='res3b3_branch2b', data=res3b3_branch2a_relu , num_filter=128, pad=(1,1), kernel=(3,3), stride=(1,1), no_bias=True)\n bn3b3_branch2b = mx.symbol.BatchNorm(name='bn3b3_branch2b', data=res3b3_branch2b , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale3b3_branch2b = bn3b3_branch2b\n res3b3_branch2b_relu = mx.symbol.Activation(name='res3b3_branch2b_relu', data=scale3b3_branch2b , act_type='relu')\n res3b3_branch2c = mx.symbol.Convolution(name='res3b3_branch2c', data=res3b3_branch2b_relu , num_filter=512, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn3b3_branch2c = mx.symbol.BatchNorm(name='bn3b3_branch2c', data=res3b3_branch2c , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale3b3_branch2c = bn3b3_branch2c\n res3b3 = mx.symbol.broadcast_add(name='res3b3', *[res3b2_relu,scale3b3_branch2c] )\n res3b3_relu = mx.symbol.Activation(name='res3b3_relu', data=res3b3 , act_type='relu')\n res4a_branch1 = mx.symbol.Convolution(name='res4a_branch1', data=res3b3_relu , num_filter=1024, pad=(0,0), kernel=(1,1), stride=(2,2), no_bias=True)\n bn4a_branch1 = mx.symbol.BatchNorm(name='bn4a_branch1', data=res4a_branch1 , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4a_branch1 = bn4a_branch1\n res4a_branch2a = mx.symbol.Convolution(name='res4a_branch2a', data=res3b3_relu , num_filter=256, pad=(0,0), kernel=(1,1), stride=(2,2), no_bias=True)\n bn4a_branch2a = mx.symbol.BatchNorm(name='bn4a_branch2a', data=res4a_branch2a , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4a_branch2a = bn4a_branch2a\n res4a_branch2a_relu = mx.symbol.Activation(name='res4a_branch2a_relu', data=scale4a_branch2a , act_type='relu')\n res4a_branch2b = mx.symbol.Convolution(name='res4a_branch2b', data=res4a_branch2a_relu , num_filter=256, pad=(1,1), kernel=(3,3), stride=(1,1), no_bias=True)\n bn4a_branch2b = mx.symbol.BatchNorm(name='bn4a_branch2b', data=res4a_branch2b , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4a_branch2b = bn4a_branch2b\n res4a_branch2b_relu = mx.symbol.Activation(name='res4a_branch2b_relu', data=scale4a_branch2b , act_type='relu')\n res4a_branch2c = mx.symbol.Convolution(name='res4a_branch2c', data=res4a_branch2b_relu , num_filter=1024, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn4a_branch2c = mx.symbol.BatchNorm(name='bn4a_branch2c', data=res4a_branch2c , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4a_branch2c = bn4a_branch2c\n res4a = mx.symbol.broadcast_add(name='res4a', *[scale4a_branch1,scale4a_branch2c] )\n res4a_relu = mx.symbol.Activation(name='res4a_relu', data=res4a , act_type='relu')\n res4b1_branch2a = mx.symbol.Convolution(name='res4b1_branch2a', data=res4a_relu , num_filter=256, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn4b1_branch2a = mx.symbol.BatchNorm(name='bn4b1_branch2a', data=res4b1_branch2a , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b1_branch2a = bn4b1_branch2a\n res4b1_branch2a_relu = mx.symbol.Activation(name='res4b1_branch2a_relu', data=scale4b1_branch2a , act_type='relu')\n res4b1_branch2b = mx.symbol.Convolution(name='res4b1_branch2b', data=res4b1_branch2a_relu , num_filter=256, pad=(1,1), kernel=(3,3), stride=(1,1), no_bias=True)\n bn4b1_branch2b = mx.symbol.BatchNorm(name='bn4b1_branch2b', data=res4b1_branch2b , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b1_branch2b = bn4b1_branch2b\n res4b1_branch2b_relu = mx.symbol.Activation(name='res4b1_branch2b_relu', data=scale4b1_branch2b , act_type='relu')\n res4b1_branch2c = mx.symbol.Convolution(name='res4b1_branch2c', data=res4b1_branch2b_relu , num_filter=1024, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn4b1_branch2c = mx.symbol.BatchNorm(name='bn4b1_branch2c', data=res4b1_branch2c , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b1_branch2c = bn4b1_branch2c\n res4b1 = mx.symbol.broadcast_add(name='res4b1', *[res4a_relu,scale4b1_branch2c] )\n res4b1_relu = mx.symbol.Activation(name='res4b1_relu', data=res4b1 , act_type='relu')\n res4b2_branch2a = mx.symbol.Convolution(name='res4b2_branch2a', data=res4b1_relu , num_filter=256, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn4b2_branch2a = mx.symbol.BatchNorm(name='bn4b2_branch2a', data=res4b2_branch2a , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b2_branch2a = bn4b2_branch2a\n res4b2_branch2a_relu = mx.symbol.Activation(name='res4b2_branch2a_relu', data=scale4b2_branch2a , act_type='relu')\n res4b2_branch2b = mx.symbol.Convolution(name='res4b2_branch2b', data=res4b2_branch2a_relu , num_filter=256, pad=(1,1), kernel=(3,3), stride=(1,1), no_bias=True)\n bn4b2_branch2b = mx.symbol.BatchNorm(name='bn4b2_branch2b', data=res4b2_branch2b , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b2_branch2b = bn4b2_branch2b\n res4b2_branch2b_relu = mx.symbol.Activation(name='res4b2_branch2b_relu', data=scale4b2_branch2b , act_type='relu')\n res4b2_branch2c = mx.symbol.Convolution(name='res4b2_branch2c', data=res4b2_branch2b_relu , num_filter=1024, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn4b2_branch2c = mx.symbol.BatchNorm(name='bn4b2_branch2c', data=res4b2_branch2c , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b2_branch2c = bn4b2_branch2c\n res4b2 = mx.symbol.broadcast_add(name='res4b2', *[res4b1_relu,scale4b2_branch2c] )\n res4b2_relu = mx.symbol.Activation(name='res4b2_relu', data=res4b2 , act_type='relu')\n res4b3_branch2a = mx.symbol.Convolution(name='res4b3_branch2a', data=res4b2_relu , num_filter=256, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn4b3_branch2a = mx.symbol.BatchNorm(name='bn4b3_branch2a', data=res4b3_branch2a , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b3_branch2a = bn4b3_branch2a\n res4b3_branch2a_relu = mx.symbol.Activation(name='res4b3_branch2a_relu', data=scale4b3_branch2a , act_type='relu')\n res4b3_branch2b = mx.symbol.Convolution(name='res4b3_branch2b', data=res4b3_branch2a_relu , num_filter=256, pad=(1,1), kernel=(3,3), stride=(1,1), no_bias=True)\n bn4b3_branch2b = mx.symbol.BatchNorm(name='bn4b3_branch2b', data=res4b3_branch2b , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b3_branch2b = bn4b3_branch2b\n res4b3_branch2b_relu = mx.symbol.Activation(name='res4b3_branch2b_relu', data=scale4b3_branch2b , act_type='relu')\n res4b3_branch2c = mx.symbol.Convolution(name='res4b3_branch2c', data=res4b3_branch2b_relu , num_filter=1024, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn4b3_branch2c = mx.symbol.BatchNorm(name='bn4b3_branch2c', data=res4b3_branch2c , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b3_branch2c = bn4b3_branch2c\n res4b3 = mx.symbol.broadcast_add(name='res4b3', *[res4b2_relu,scale4b3_branch2c] )\n res4b3_relu = mx.symbol.Activation(name='res4b3_relu', data=res4b3 , act_type='relu')\n res4b4_branch2a = mx.symbol.Convolution(name='res4b4_branch2a', data=res4b3_relu , num_filter=256, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn4b4_branch2a = mx.symbol.BatchNorm(name='bn4b4_branch2a', data=res4b4_branch2a , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b4_branch2a = bn4b4_branch2a\n res4b4_branch2a_relu = mx.symbol.Activation(name='res4b4_branch2a_relu', data=scale4b4_branch2a , act_type='relu')\n res4b4_branch2b = mx.symbol.Convolution(name='res4b4_branch2b', data=res4b4_branch2a_relu , num_filter=256, pad=(1,1), kernel=(3,3), stride=(1,1), no_bias=True)\n bn4b4_branch2b = mx.symbol.BatchNorm(name='bn4b4_branch2b', data=res4b4_branch2b , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b4_branch2b = bn4b4_branch2b\n res4b4_branch2b_relu = mx.symbol.Activation(name='res4b4_branch2b_relu', data=scale4b4_branch2b , act_type='relu')\n res4b4_branch2c = mx.symbol.Convolution(name='res4b4_branch2c', data=res4b4_branch2b_relu , num_filter=1024, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn4b4_branch2c = mx.symbol.BatchNorm(name='bn4b4_branch2c', data=res4b4_branch2c , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b4_branch2c = bn4b4_branch2c\n res4b4 = mx.symbol.broadcast_add(name='res4b4', *[res4b3_relu,scale4b4_branch2c] )\n res4b4_relu = mx.symbol.Activation(name='res4b4_relu', data=res4b4 , act_type='relu')\n res4b5_branch2a = mx.symbol.Convolution(name='res4b5_branch2a', data=res4b4_relu , num_filter=256, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn4b5_branch2a = mx.symbol.BatchNorm(name='bn4b5_branch2a', data=res4b5_branch2a , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b5_branch2a = bn4b5_branch2a\n res4b5_branch2a_relu = mx.symbol.Activation(name='res4b5_branch2a_relu', data=scale4b5_branch2a , act_type='relu')\n res4b5_branch2b = mx.symbol.Convolution(name='res4b5_branch2b', data=res4b5_branch2a_relu , num_filter=256, pad=(1,1), kernel=(3,3), stride=(1,1), no_bias=True)\n bn4b5_branch2b = mx.symbol.BatchNorm(name='bn4b5_branch2b', data=res4b5_branch2b , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b5_branch2b = bn4b5_branch2b\n res4b5_branch2b_relu = mx.symbol.Activation(name='res4b5_branch2b_relu', data=scale4b5_branch2b , act_type='relu')\n res4b5_branch2c = mx.symbol.Convolution(name='res4b5_branch2c', data=res4b5_branch2b_relu , num_filter=1024, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn4b5_branch2c = mx.symbol.BatchNorm(name='bn4b5_branch2c', data=res4b5_branch2c , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b5_branch2c = bn4b5_branch2c\n res4b5 = mx.symbol.broadcast_add(name='res4b5', *[res4b4_relu,scale4b5_branch2c] )\n res4b5_relu = mx.symbol.Activation(name='res4b5_relu', data=res4b5 , act_type='relu')\n res4b6_branch2a = mx.symbol.Convolution(name='res4b6_branch2a', data=res4b5_relu , num_filter=256, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn4b6_branch2a = mx.symbol.BatchNorm(name='bn4b6_branch2a', data=res4b6_branch2a , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b6_branch2a = bn4b6_branch2a\n res4b6_branch2a_relu = mx.symbol.Activation(name='res4b6_branch2a_relu', data=scale4b6_branch2a , act_type='relu')\n res4b6_branch2b = mx.symbol.Convolution(name='res4b6_branch2b', data=res4b6_branch2a_relu , num_filter=256, pad=(1,1), kernel=(3,3), stride=(1,1), no_bias=True)\n bn4b6_branch2b = mx.symbol.BatchNorm(name='bn4b6_branch2b', data=res4b6_branch2b , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b6_branch2b = bn4b6_branch2b\n res4b6_branch2b_relu = mx.symbol.Activation(name='res4b6_branch2b_relu', data=scale4b6_branch2b , act_type='relu')\n res4b6_branch2c = mx.symbol.Convolution(name='res4b6_branch2c', data=res4b6_branch2b_relu , num_filter=1024, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn4b6_branch2c = mx.symbol.BatchNorm(name='bn4b6_branch2c', data=res4b6_branch2c , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b6_branch2c = bn4b6_branch2c\n res4b6 = mx.symbol.broadcast_add(name='res4b6', *[res4b5_relu,scale4b6_branch2c] )\n res4b6_relu = mx.symbol.Activation(name='res4b6_relu', data=res4b6 , act_type='relu')\n res4b7_branch2a = mx.symbol.Convolution(name='res4b7_branch2a', data=res4b6_relu , num_filter=256, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn4b7_branch2a = mx.symbol.BatchNorm(name='bn4b7_branch2a', data=res4b7_branch2a , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b7_branch2a = bn4b7_branch2a\n res4b7_branch2a_relu = mx.symbol.Activation(name='res4b7_branch2a_relu', data=scale4b7_branch2a , act_type='relu')\n res4b7_branch2b = mx.symbol.Convolution(name='res4b7_branch2b', data=res4b7_branch2a_relu , num_filter=256, pad=(1,1), kernel=(3,3), stride=(1,1), no_bias=True)\n bn4b7_branch2b = mx.symbol.BatchNorm(name='bn4b7_branch2b', data=res4b7_branch2b , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b7_branch2b = bn4b7_branch2b\n res4b7_branch2b_relu = mx.symbol.Activation(name='res4b7_branch2b_relu', data=scale4b7_branch2b , act_type='relu')\n res4b7_branch2c = mx.symbol.Convolution(name='res4b7_branch2c', data=res4b7_branch2b_relu , num_filter=1024, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn4b7_branch2c = mx.symbol.BatchNorm(name='bn4b7_branch2c', data=res4b7_branch2c , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b7_branch2c = bn4b7_branch2c\n res4b7 = mx.symbol.broadcast_add(name='res4b7', *[res4b6_relu,scale4b7_branch2c] )\n res4b7_relu = mx.symbol.Activation(name='res4b7_relu', data=res4b7 , act_type='relu')\n res4b8_branch2a = mx.symbol.Convolution(name='res4b8_branch2a', data=res4b7_relu , num_filter=256, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn4b8_branch2a = mx.symbol.BatchNorm(name='bn4b8_branch2a', data=res4b8_branch2a , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b8_branch2a = bn4b8_branch2a\n res4b8_branch2a_relu = mx.symbol.Activation(name='res4b8_branch2a_relu', data=scale4b8_branch2a , act_type='relu')\n res4b8_branch2b = mx.symbol.Convolution(name='res4b8_branch2b', data=res4b8_branch2a_relu , num_filter=256, pad=(1,1), kernel=(3,3), stride=(1,1), no_bias=True)\n bn4b8_branch2b = mx.symbol.BatchNorm(name='bn4b8_branch2b', data=res4b8_branch2b , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b8_branch2b = bn4b8_branch2b\n res4b8_branch2b_relu = mx.symbol.Activation(name='res4b8_branch2b_relu', data=scale4b8_branch2b , act_type='relu')\n res4b8_branch2c = mx.symbol.Convolution(name='res4b8_branch2c', data=res4b8_branch2b_relu , num_filter=1024, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn4b8_branch2c = mx.symbol.BatchNorm(name='bn4b8_branch2c', data=res4b8_branch2c , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b8_branch2c = bn4b8_branch2c\n res4b8 = mx.symbol.broadcast_add(name='res4b8', *[res4b7_relu,scale4b8_branch2c] )\n res4b8_relu = mx.symbol.Activation(name='res4b8_relu', data=res4b8 , act_type='relu')\n res4b9_branch2a = mx.symbol.Convolution(name='res4b9_branch2a', data=res4b8_relu , num_filter=256, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn4b9_branch2a = mx.symbol.BatchNorm(name='bn4b9_branch2a', data=res4b9_branch2a , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b9_branch2a = bn4b9_branch2a\n res4b9_branch2a_relu = mx.symbol.Activation(name='res4b9_branch2a_relu', data=scale4b9_branch2a , act_type='relu')\n res4b9_branch2b = mx.symbol.Convolution(name='res4b9_branch2b', data=res4b9_branch2a_relu , num_filter=256, pad=(1,1), kernel=(3,3), stride=(1,1), no_bias=True)\n bn4b9_branch2b = mx.symbol.BatchNorm(name='bn4b9_branch2b', data=res4b9_branch2b , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b9_branch2b = bn4b9_branch2b\n res4b9_branch2b_relu = mx.symbol.Activation(name='res4b9_branch2b_relu', data=scale4b9_branch2b , act_type='relu')\n res4b9_branch2c = mx.symbol.Convolution(name='res4b9_branch2c', data=res4b9_branch2b_relu , num_filter=1024, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn4b9_branch2c = mx.symbol.BatchNorm(name='bn4b9_branch2c', data=res4b9_branch2c , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b9_branch2c = bn4b9_branch2c\n res4b9 = mx.symbol.broadcast_add(name='res4b9', *[res4b8_relu,scale4b9_branch2c] )\n res4b9_relu = mx.symbol.Activation(name='res4b9_relu', data=res4b9 , act_type='relu')\n res4b10_branch2a = mx.symbol.Convolution(name='res4b10_branch2a', data=res4b9_relu , num_filter=256, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn4b10_branch2a = mx.symbol.BatchNorm(name='bn4b10_branch2a', data=res4b10_branch2a , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b10_branch2a = bn4b10_branch2a\n res4b10_branch2a_relu = mx.symbol.Activation(name='res4b10_branch2a_relu', data=scale4b10_branch2a , act_type='relu')\n res4b10_branch2b = mx.symbol.Convolution(name='res4b10_branch2b', data=res4b10_branch2a_relu , num_filter=256, pad=(1,1), kernel=(3,3), stride=(1,1), no_bias=True)\n bn4b10_branch2b = mx.symbol.BatchNorm(name='bn4b10_branch2b', data=res4b10_branch2b , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b10_branch2b = bn4b10_branch2b\n res4b10_branch2b_relu = mx.symbol.Activation(name='res4b10_branch2b_relu', data=scale4b10_branch2b , act_type='relu')\n res4b10_branch2c = mx.symbol.Convolution(name='res4b10_branch2c', data=res4b10_branch2b_relu , num_filter=1024, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn4b10_branch2c = mx.symbol.BatchNorm(name='bn4b10_branch2c', data=res4b10_branch2c , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b10_branch2c = bn4b10_branch2c\n res4b10 = mx.symbol.broadcast_add(name='res4b10', *[res4b9_relu,scale4b10_branch2c] )\n res4b10_relu = mx.symbol.Activation(name='res4b10_relu', data=res4b10 , act_type='relu')\n res4b11_branch2a = mx.symbol.Convolution(name='res4b11_branch2a', data=res4b10_relu , num_filter=256, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn4b11_branch2a = mx.symbol.BatchNorm(name='bn4b11_branch2a', data=res4b11_branch2a , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b11_branch2a = bn4b11_branch2a\n res4b11_branch2a_relu = mx.symbol.Activation(name='res4b11_branch2a_relu', data=scale4b11_branch2a , act_type='relu')\n res4b11_branch2b = mx.symbol.Convolution(name='res4b11_branch2b', data=res4b11_branch2a_relu , num_filter=256, pad=(1,1), kernel=(3,3), stride=(1,1), no_bias=True)\n bn4b11_branch2b = mx.symbol.BatchNorm(name='bn4b11_branch2b', data=res4b11_branch2b , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b11_branch2b = bn4b11_branch2b\n res4b11_branch2b_relu = mx.symbol.Activation(name='res4b11_branch2b_relu', data=scale4b11_branch2b , act_type='relu')\n res4b11_branch2c = mx.symbol.Convolution(name='res4b11_branch2c', data=res4b11_branch2b_relu , num_filter=1024, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn4b11_branch2c = mx.symbol.BatchNorm(name='bn4b11_branch2c', data=res4b11_branch2c , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b11_branch2c = bn4b11_branch2c\n res4b11 = mx.symbol.broadcast_add(name='res4b11', *[res4b10_relu,scale4b11_branch2c] )\n res4b11_relu = mx.symbol.Activation(name='res4b11_relu', data=res4b11 , act_type='relu')\n res4b12_branch2a = mx.symbol.Convolution(name='res4b12_branch2a', data=res4b11_relu , num_filter=256, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn4b12_branch2a = mx.symbol.BatchNorm(name='bn4b12_branch2a', data=res4b12_branch2a , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b12_branch2a = bn4b12_branch2a\n res4b12_branch2a_relu = mx.symbol.Activation(name='res4b12_branch2a_relu', data=scale4b12_branch2a , act_type='relu')\n res4b12_branch2b = mx.symbol.Convolution(name='res4b12_branch2b', data=res4b12_branch2a_relu , num_filter=256, pad=(1,1), kernel=(3,3), stride=(1,1), no_bias=True)\n bn4b12_branch2b = mx.symbol.BatchNorm(name='bn4b12_branch2b', data=res4b12_branch2b , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b12_branch2b = bn4b12_branch2b\n res4b12_branch2b_relu = mx.symbol.Activation(name='res4b12_branch2b_relu', data=scale4b12_branch2b , act_type='relu')\n res4b12_branch2c = mx.symbol.Convolution(name='res4b12_branch2c', data=res4b12_branch2b_relu , num_filter=1024, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn4b12_branch2c = mx.symbol.BatchNorm(name='bn4b12_branch2c', data=res4b12_branch2c , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b12_branch2c = bn4b12_branch2c\n res4b12 = mx.symbol.broadcast_add(name='res4b12', *[res4b11_relu,scale4b12_branch2c] )\n res4b12_relu = mx.symbol.Activation(name='res4b12_relu', data=res4b12 , act_type='relu')\n res4b13_branch2a = mx.symbol.Convolution(name='res4b13_branch2a', data=res4b12_relu , num_filter=256, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn4b13_branch2a = mx.symbol.BatchNorm(name='bn4b13_branch2a', data=res4b13_branch2a , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b13_branch2a = bn4b13_branch2a\n res4b13_branch2a_relu = mx.symbol.Activation(name='res4b13_branch2a_relu', data=scale4b13_branch2a , act_type='relu')\n res4b13_branch2b = mx.symbol.Convolution(name='res4b13_branch2b', data=res4b13_branch2a_relu , num_filter=256, pad=(1,1), kernel=(3,3), stride=(1,1), no_bias=True)\n bn4b13_branch2b = mx.symbol.BatchNorm(name='bn4b13_branch2b', data=res4b13_branch2b , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b13_branch2b = bn4b13_branch2b\n res4b13_branch2b_relu = mx.symbol.Activation(name='res4b13_branch2b_relu', data=scale4b13_branch2b , act_type='relu')\n res4b13_branch2c = mx.symbol.Convolution(name='res4b13_branch2c', data=res4b13_branch2b_relu , num_filter=1024, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn4b13_branch2c = mx.symbol.BatchNorm(name='bn4b13_branch2c', data=res4b13_branch2c , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b13_branch2c = bn4b13_branch2c\n res4b13 = mx.symbol.broadcast_add(name='res4b13', *[res4b12_relu,scale4b13_branch2c] )\n res4b13_relu = mx.symbol.Activation(name='res4b13_relu', data=res4b13 , act_type='relu')\n res4b14_branch2a = mx.symbol.Convolution(name='res4b14_branch2a', data=res4b13_relu , num_filter=256, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn4b14_branch2a = mx.symbol.BatchNorm(name='bn4b14_branch2a', data=res4b14_branch2a , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b14_branch2a = bn4b14_branch2a\n res4b14_branch2a_relu = mx.symbol.Activation(name='res4b14_branch2a_relu', data=scale4b14_branch2a , act_type='relu')\n res4b14_branch2b = mx.symbol.Convolution(name='res4b14_branch2b', data=res4b14_branch2a_relu , num_filter=256, pad=(1,1), kernel=(3,3), stride=(1,1), no_bias=True)\n bn4b14_branch2b = mx.symbol.BatchNorm(name='bn4b14_branch2b', data=res4b14_branch2b , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b14_branch2b = bn4b14_branch2b\n res4b14_branch2b_relu = mx.symbol.Activation(name='res4b14_branch2b_relu', data=scale4b14_branch2b , act_type='relu')\n res4b14_branch2c = mx.symbol.Convolution(name='res4b14_branch2c', data=res4b14_branch2b_relu , num_filter=1024, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn4b14_branch2c = mx.symbol.BatchNorm(name='bn4b14_branch2c', data=res4b14_branch2c , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b14_branch2c = bn4b14_branch2c\n res4b14 = mx.symbol.broadcast_add(name='res4b14', *[res4b13_relu,scale4b14_branch2c] )\n res4b14_relu = mx.symbol.Activation(name='res4b14_relu', data=res4b14 , act_type='relu')\n res4b15_branch2a = mx.symbol.Convolution(name='res4b15_branch2a', data=res4b14_relu , num_filter=256, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn4b15_branch2a = mx.symbol.BatchNorm(name='bn4b15_branch2a', data=res4b15_branch2a , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b15_branch2a = bn4b15_branch2a\n res4b15_branch2a_relu = mx.symbol.Activation(name='res4b15_branch2a_relu', data=scale4b15_branch2a , act_type='relu')\n res4b15_branch2b = mx.symbol.Convolution(name='res4b15_branch2b', data=res4b15_branch2a_relu , num_filter=256, pad=(1,1), kernel=(3,3), stride=(1,1), no_bias=True)\n bn4b15_branch2b = mx.symbol.BatchNorm(name='bn4b15_branch2b', data=res4b15_branch2b , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b15_branch2b = bn4b15_branch2b\n res4b15_branch2b_relu = mx.symbol.Activation(name='res4b15_branch2b_relu', data=scale4b15_branch2b , act_type='relu')\n res4b15_branch2c = mx.symbol.Convolution(name='res4b15_branch2c', data=res4b15_branch2b_relu , num_filter=1024, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn4b15_branch2c = mx.symbol.BatchNorm(name='bn4b15_branch2c', data=res4b15_branch2c , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b15_branch2c = bn4b15_branch2c\n res4b15 = mx.symbol.broadcast_add(name='res4b15', *[res4b14_relu,scale4b15_branch2c] )\n res4b15_relu = mx.symbol.Activation(name='res4b15_relu', data=res4b15 , act_type='relu')\n res4b16_branch2a = mx.symbol.Convolution(name='res4b16_branch2a', data=res4b15_relu , num_filter=256, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn4b16_branch2a = mx.symbol.BatchNorm(name='bn4b16_branch2a', data=res4b16_branch2a , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b16_branch2a = bn4b16_branch2a\n res4b16_branch2a_relu = mx.symbol.Activation(name='res4b16_branch2a_relu', data=scale4b16_branch2a , act_type='relu')\n res4b16_branch2b = mx.symbol.Convolution(name='res4b16_branch2b', data=res4b16_branch2a_relu , num_filter=256, pad=(1,1), kernel=(3,3), stride=(1,1), no_bias=True)\n bn4b16_branch2b = mx.symbol.BatchNorm(name='bn4b16_branch2b', data=res4b16_branch2b , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b16_branch2b = bn4b16_branch2b\n res4b16_branch2b_relu = mx.symbol.Activation(name='res4b16_branch2b_relu', data=scale4b16_branch2b , act_type='relu')\n res4b16_branch2c = mx.symbol.Convolution(name='res4b16_branch2c', data=res4b16_branch2b_relu , num_filter=1024, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn4b16_branch2c = mx.symbol.BatchNorm(name='bn4b16_branch2c', data=res4b16_branch2c , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b16_branch2c = bn4b16_branch2c\n res4b16 = mx.symbol.broadcast_add(name='res4b16', *[res4b15_relu,scale4b16_branch2c] )\n res4b16_relu = mx.symbol.Activation(name='res4b16_relu', data=res4b16 , act_type='relu')\n res4b17_branch2a = mx.symbol.Convolution(name='res4b17_branch2a', data=res4b16_relu , num_filter=256, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn4b17_branch2a = mx.symbol.BatchNorm(name='bn4b17_branch2a', data=res4b17_branch2a , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b17_branch2a = bn4b17_branch2a\n res4b17_branch2a_relu = mx.symbol.Activation(name='res4b17_branch2a_relu', data=scale4b17_branch2a , act_type='relu')\n res4b17_branch2b = mx.symbol.Convolution(name='res4b17_branch2b', data=res4b17_branch2a_relu , num_filter=256, pad=(1,1), kernel=(3,3), stride=(1,1), no_bias=True)\n bn4b17_branch2b = mx.symbol.BatchNorm(name='bn4b17_branch2b', data=res4b17_branch2b , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b17_branch2b = bn4b17_branch2b\n res4b17_branch2b_relu = mx.symbol.Activation(name='res4b17_branch2b_relu', data=scale4b17_branch2b , act_type='relu')\n res4b17_branch2c = mx.symbol.Convolution(name='res4b17_branch2c', data=res4b17_branch2b_relu , num_filter=1024, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn4b17_branch2c = mx.symbol.BatchNorm(name='bn4b17_branch2c', data=res4b17_branch2c , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b17_branch2c = bn4b17_branch2c\n res4b17 = mx.symbol.broadcast_add(name='res4b17', *[res4b16_relu,scale4b17_branch2c] )\n res4b17_relu = mx.symbol.Activation(name='res4b17_relu', data=res4b17 , act_type='relu')\n res4b18_branch2a = mx.symbol.Convolution(name='res4b18_branch2a', data=res4b17_relu , num_filter=256, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn4b18_branch2a = mx.symbol.BatchNorm(name='bn4b18_branch2a', data=res4b18_branch2a , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b18_branch2a = bn4b18_branch2a\n res4b18_branch2a_relu = mx.symbol.Activation(name='res4b18_branch2a_relu', data=scale4b18_branch2a , act_type='relu')\n res4b18_branch2b = mx.symbol.Convolution(name='res4b18_branch2b', data=res4b18_branch2a_relu , num_filter=256, pad=(1,1), kernel=(3,3), stride=(1,1), no_bias=True)\n bn4b18_branch2b = mx.symbol.BatchNorm(name='bn4b18_branch2b', data=res4b18_branch2b , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b18_branch2b = bn4b18_branch2b\n res4b18_branch2b_relu = mx.symbol.Activation(name='res4b18_branch2b_relu', data=scale4b18_branch2b , act_type='relu')\n res4b18_branch2c = mx.symbol.Convolution(name='res4b18_branch2c', data=res4b18_branch2b_relu , num_filter=1024, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn4b18_branch2c = mx.symbol.BatchNorm(name='bn4b18_branch2c', data=res4b18_branch2c , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b18_branch2c = bn4b18_branch2c\n res4b18 = mx.symbol.broadcast_add(name='res4b18', *[res4b17_relu,scale4b18_branch2c] )\n res4b18_relu = mx.symbol.Activation(name='res4b18_relu', data=res4b18 , act_type='relu')\n res4b19_branch2a = mx.symbol.Convolution(name='res4b19_branch2a', data=res4b18_relu , num_filter=256, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn4b19_branch2a = mx.symbol.BatchNorm(name='bn4b19_branch2a', data=res4b19_branch2a , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b19_branch2a = bn4b19_branch2a\n res4b19_branch2a_relu = mx.symbol.Activation(name='res4b19_branch2a_relu', data=scale4b19_branch2a , act_type='relu')\n res4b19_branch2b = mx.symbol.Convolution(name='res4b19_branch2b', data=res4b19_branch2a_relu , num_filter=256, pad=(1,1), kernel=(3,3), stride=(1,1), no_bias=True)\n bn4b19_branch2b = mx.symbol.BatchNorm(name='bn4b19_branch2b', data=res4b19_branch2b , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b19_branch2b = bn4b19_branch2b\n res4b19_branch2b_relu = mx.symbol.Activation(name='res4b19_branch2b_relu', data=scale4b19_branch2b , act_type='relu')\n res4b19_branch2c = mx.symbol.Convolution(name='res4b19_branch2c', data=res4b19_branch2b_relu , num_filter=1024, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn4b19_branch2c = mx.symbol.BatchNorm(name='bn4b19_branch2c', data=res4b19_branch2c , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b19_branch2c = bn4b19_branch2c\n res4b19 = mx.symbol.broadcast_add(name='res4b19', *[res4b18_relu,scale4b19_branch2c] )\n res4b19_relu = mx.symbol.Activation(name='res4b19_relu', data=res4b19 , act_type='relu')\n res4b20_branch2a = mx.symbol.Convolution(name='res4b20_branch2a', data=res4b19_relu , num_filter=256, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn4b20_branch2a = mx.symbol.BatchNorm(name='bn4b20_branch2a', data=res4b20_branch2a , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b20_branch2a = bn4b20_branch2a\n res4b20_branch2a_relu = mx.symbol.Activation(name='res4b20_branch2a_relu', data=scale4b20_branch2a , act_type='relu')\n res4b20_branch2b = mx.symbol.Convolution(name='res4b20_branch2b', data=res4b20_branch2a_relu , num_filter=256, pad=(1,1), kernel=(3,3), stride=(1,1), no_bias=True)\n bn4b20_branch2b = mx.symbol.BatchNorm(name='bn4b20_branch2b', data=res4b20_branch2b , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b20_branch2b = bn4b20_branch2b\n res4b20_branch2b_relu = mx.symbol.Activation(name='res4b20_branch2b_relu', data=scale4b20_branch2b , act_type='relu')\n res4b20_branch2c = mx.symbol.Convolution(name='res4b20_branch2c', data=res4b20_branch2b_relu , num_filter=1024, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn4b20_branch2c = mx.symbol.BatchNorm(name='bn4b20_branch2c', data=res4b20_branch2c , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b20_branch2c = bn4b20_branch2c\n res4b20 = mx.symbol.broadcast_add(name='res4b20', *[res4b19_relu,scale4b20_branch2c] )\n res4b20_relu = mx.symbol.Activation(name='res4b20_relu', data=res4b20 , act_type='relu')\n res4b21_branch2a = mx.symbol.Convolution(name='res4b21_branch2a', data=res4b20_relu , num_filter=256, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn4b21_branch2a = mx.symbol.BatchNorm(name='bn4b21_branch2a', data=res4b21_branch2a , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b21_branch2a = bn4b21_branch2a\n res4b21_branch2a_relu = mx.symbol.Activation(name='res4b21_branch2a_relu', data=scale4b21_branch2a , act_type='relu')\n res4b21_branch2b = mx.symbol.Convolution(name='res4b21_branch2b', data=res4b21_branch2a_relu , num_filter=256, pad=(1,1), kernel=(3,3), stride=(1,1), no_bias=True)\n bn4b21_branch2b = mx.symbol.BatchNorm(name='bn4b21_branch2b', data=res4b21_branch2b , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b21_branch2b = bn4b21_branch2b\n res4b21_branch2b_relu = mx.symbol.Activation(name='res4b21_branch2b_relu', data=scale4b21_branch2b , act_type='relu')\n res4b21_branch2c = mx.symbol.Convolution(name='res4b21_branch2c', data=res4b21_branch2b_relu , num_filter=1024, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn4b21_branch2c = mx.symbol.BatchNorm(name='bn4b21_branch2c', data=res4b21_branch2c , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b21_branch2c = bn4b21_branch2c\n res4b21 = mx.symbol.broadcast_add(name='res4b21', *[res4b20_relu,scale4b21_branch2c] )\n res4b21_relu = mx.symbol.Activation(name='res4b21_relu', data=res4b21 , act_type='relu')\n res4b22_branch2a = mx.symbol.Convolution(name='res4b22_branch2a', data=res4b21_relu , num_filter=256, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn4b22_branch2a = mx.symbol.BatchNorm(name='bn4b22_branch2a', data=res4b22_branch2a , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b22_branch2a = bn4b22_branch2a\n res4b22_branch2a_relu = mx.symbol.Activation(name='res4b22_branch2a_relu', data=scale4b22_branch2a , act_type='relu')\n res4b22_branch2b = mx.symbol.Convolution(name='res4b22_branch2b', data=res4b22_branch2a_relu , num_filter=256, pad=(1,1), kernel=(3,3), stride=(1,1), no_bias=True)\n bn4b22_branch2b = mx.symbol.BatchNorm(name='bn4b22_branch2b', data=res4b22_branch2b , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b22_branch2b = bn4b22_branch2b\n res4b22_branch2b_relu = mx.symbol.Activation(name='res4b22_branch2b_relu', data=scale4b22_branch2b , act_type='relu')\n res4b22_branch2c = mx.symbol.Convolution(name='res4b22_branch2c', data=res4b22_branch2b_relu , num_filter=1024, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn4b22_branch2c = mx.symbol.BatchNorm(name='bn4b22_branch2c', data=res4b22_branch2c , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b22_branch2c = bn4b22_branch2c\n res4b22 = mx.symbol.broadcast_add(name='res4b22', *[res4b21_relu,scale4b22_branch2c] )\n res4b22_relu = mx.symbol.Activation(name='res4b22_relu', data=res4b22 , act_type='relu')\n res5a_branch1 = mx.symbol.Convolution(name='res5a_branch1', data=res4b22_relu , num_filter=2048, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn5a_branch1 = mx.symbol.BatchNorm(name='bn5a_branch1', data=res5a_branch1 , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale5a_branch1 = bn5a_branch1\n res5a_branch2a = mx.symbol.Convolution(name='res5a_branch2a', data=res4b22_relu , num_filter=512, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn5a_branch2a = mx.symbol.BatchNorm(name='bn5a_branch2a', data=res5a_branch2a , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale5a_branch2a = bn5a_branch2a\n res5a_branch2a_relu = mx.symbol.Activation(name='res5a_branch2a_relu', data=scale5a_branch2a , act_type='relu')\n res5a_branch2b = mx.symbol.Convolution(name='res5a_branch2b', data=res5a_branch2a_relu , num_filter=512, pad=(2,2), dilate=(2,2), kernel=(3,3), stride=(1,1), no_bias=True)\n bn5a_branch2b = mx.symbol.BatchNorm(name='bn5a_branch2b', data=res5a_branch2b , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale5a_branch2b = bn5a_branch2b\n res5a_branch2b_relu = mx.symbol.Activation(name='res5a_branch2b_relu', data=scale5a_branch2b , act_type='relu')\n res5a_branch2c = mx.symbol.Convolution(name='res5a_branch2c', data=res5a_branch2b_relu , num_filter=2048, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn5a_branch2c = mx.symbol.BatchNorm(name='bn5a_branch2c', data=res5a_branch2c , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale5a_branch2c = bn5a_branch2c\n res5a = mx.symbol.broadcast_add(name='res5a', *[scale5a_branch1,scale5a_branch2c] )\n res5a_relu = mx.symbol.Activation(name='res5a_relu', data=res5a , act_type='relu')\n res5b_branch2a = mx.symbol.Convolution(name='res5b_branch2a', data=res5a_relu , num_filter=512, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn5b_branch2a = mx.symbol.BatchNorm(name='bn5b_branch2a', data=res5b_branch2a , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale5b_branch2a = bn5b_branch2a\n res5b_branch2a_relu = mx.symbol.Activation(name='res5b_branch2a_relu', data=scale5b_branch2a , act_type='relu')\n res5b_branch2b = mx.symbol.Convolution(name='res5b_branch2b', data=res5b_branch2a_relu , num_filter=512, pad=(2,2), dilate=(2,2), kernel=(3,3), stride=(1,1), no_bias=True)\n bn5b_branch2b = mx.symbol.BatchNorm(name='bn5b_branch2b', data=res5b_branch2b , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale5b_branch2b = bn5b_branch2b\n res5b_branch2b_relu = mx.symbol.Activation(name='res5b_branch2b_relu', data=scale5b_branch2b , act_type='relu')\n res5b_branch2c = mx.symbol.Convolution(name='res5b_branch2c', data=res5b_branch2b_relu , num_filter=2048, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn5b_branch2c = mx.symbol.BatchNorm(name='bn5b_branch2c', data=res5b_branch2c , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale5b_branch2c = bn5b_branch2c\n res5b = mx.symbol.broadcast_add(name='res5b', *[res5a_relu,scale5b_branch2c] )\n res5b_relu = mx.symbol.Activation(name='res5b_relu', data=res5b , act_type='relu')\n res5c_branch2a = mx.symbol.Convolution(name='res5c_branch2a', data=res5b_relu , num_filter=512, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn5c_branch2a = mx.symbol.BatchNorm(name='bn5c_branch2a', data=res5c_branch2a , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale5c_branch2a = bn5c_branch2a\n res5c_branch2a_relu = mx.symbol.Activation(name='res5c_branch2a_relu', data=scale5c_branch2a , act_type='relu')\n res5c_branch2b = mx.symbol.Convolution(name='res5c_branch2b', data=res5c_branch2a_relu , num_filter=512, pad=(2,2), dilate=(2,2), kernel=(3,3), stride=(1,1), no_bias=True)\n bn5c_branch2b = mx.symbol.BatchNorm(name='bn5c_branch2b', data=res5c_branch2b , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale5c_branch2b = bn5c_branch2b\n res5c_branch2b_relu = mx.symbol.Activation(name='res5c_branch2b_relu', data=scale5c_branch2b , act_type='relu')\n res5c_branch2c = mx.symbol.Convolution(name='res5c_branch2c', data=res5c_branch2b_relu , num_filter=2048, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn5c_branch2c = mx.symbol.BatchNorm(name='bn5c_branch2c', data=res5c_branch2c , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale5c_branch2c = bn5c_branch2c\n res5c = mx.symbol.broadcast_add(name='res5c', *[res5b_relu,scale5c_branch2c] )\n res5c_relu = mx.symbol.Activation(name='res5c_relu', data=res5c , act_type='relu')\n\n feat_conv_3x3 = mx.sym.Convolution(\n data=res5c_relu, kernel=(3, 3), pad=(6, 6), dilate=(6, 6), num_filter=1024, name=\"feat_conv_3x3\")\n feat_conv_3x3_relu = mx.sym.Activation(data=feat_conv_3x3, act_type=\"relu\", name=\"feat_conv_3x3_relu\")\n return feat_conv_3x3_relu\n\n def get_flownet(self, img_cur, img_ref):\n data = mx.symbol.Concat(img_cur / 255.0, img_ref / 255.0, dim=1)\n resize_data = mx.symbol.Pooling(name='resize_data', data=data , pooling_convention='full', pad=(0,0), kernel=(2,2), stride=(2,2), pool_type='avg')\n flow_conv1 = mx.symbol.Convolution(name='flow_conv1', data=resize_data , num_filter=64, pad=(3,3), kernel=(7,7), stride=(2,2), no_bias=False)\n ReLU1 = mx.symbol.LeakyReLU(name='ReLU1', data=flow_conv1 , act_type='leaky', slope=0.1)\n conv2 = mx.symbol.Convolution(name='conv2', data=ReLU1 , num_filter=128, pad=(2,2), kernel=(5,5), stride=(2,2), no_bias=False)\n ReLU2 = mx.symbol.LeakyReLU(name='ReLU2', data=conv2 , act_type='leaky', slope=0.1)\n conv3 = mx.symbol.Convolution(name='conv3', data=ReLU2 , num_filter=256, pad=(2,2), kernel=(5,5), stride=(2,2), no_bias=False)\n ReLU3 = mx.symbol.LeakyReLU(name='ReLU3', data=conv3 , act_type='leaky', slope=0.1)\n conv3_1 = mx.symbol.Convolution(name='conv3_1', data=ReLU3 , num_filter=256, pad=(1,1), kernel=(3,3), stride=(1,1), no_bias=False)\n ReLU4 = mx.symbol.LeakyReLU(name='ReLU4', data=conv3_1 , act_type='leaky', slope=0.1)\n conv4 = mx.symbol.Convolution(name='conv4', data=ReLU4 , num_filter=512, pad=(1,1), kernel=(3,3), stride=(2,2), no_bias=False)\n ReLU5 = mx.symbol.LeakyReLU(name='ReLU5', data=conv4 , act_type='leaky', slope=0.1)\n conv4_1 = mx.symbol.Convolution(name='conv4_1', data=ReLU5 , num_filter=512, pad=(1,1), kernel=(3,3), stride=(1,1), no_bias=False)\n ReLU6 = mx.symbol.LeakyReLU(name='ReLU6', data=conv4_1 , act_type='leaky', slope=0.1)\n conv5 = mx.symbol.Convolution(name='conv5', data=ReLU6 , num_filter=512, pad=(1,1), kernel=(3,3), stride=(2,2), no_bias=False)\n ReLU7 = mx.symbol.LeakyReLU(name='ReLU7', data=conv5 , act_type='leaky', slope=0.1)\n conv5_1 = mx.symbol.Convolution(name='conv5_1', data=ReLU7 , num_filter=512, pad=(1,1), kernel=(3,3), stride=(1,1), no_bias=False)\n ReLU8 = mx.symbol.LeakyReLU(name='ReLU8', data=conv5_1 , act_type='leaky', slope=0.1)\n conv6 = mx.symbol.Convolution(name='conv6', data=ReLU8 , num_filter=1024, pad=(1,1), kernel=(3,3), stride=(2,2), no_bias=False)\n ReLU9 = mx.symbol.LeakyReLU(name='ReLU9', data=conv6 , act_type='leaky', slope=0.1)\n conv6_1 = mx.symbol.Convolution(name='conv6_1', data=ReLU9 , num_filter=1024, pad=(1,1), kernel=(3,3), stride=(1,1), no_bias=False)\n ReLU10 = mx.symbol.LeakyReLU(name='ReLU10', data=conv6_1 , act_type='leaky', slope=0.1)\n Convolution1 = mx.symbol.Convolution(name='Convolution1', data=ReLU10 , num_filter=2, pad=(1,1), kernel=(3,3), stride=(1,1), no_bias=False)\n deconv5 = mx.symbol.Deconvolution(name='deconv5', data=ReLU10 , num_filter=512, pad=(0,0), kernel=(4,4), stride=(2,2), no_bias=False)\n crop_deconv5 = mx.symbol.Crop(name='crop_deconv5', *[deconv5,ReLU8] , offset=(1,1))\n ReLU11 = mx.symbol.LeakyReLU(name='ReLU11', data=crop_deconv5 , act_type='leaky', slope=0.1)\n upsample_flow6to5 = mx.symbol.Deconvolution(name='upsample_flow6to5', data=Convolution1 , num_filter=2, pad=(0,0), kernel=(4,4), stride=(2,2), no_bias=False)\n crop_upsampled_flow6_to_5 = mx.symbol.Crop(name='crop_upsampled_flow6_to_5', *[upsample_flow6to5,ReLU8] , offset=(1,1))\n Concat2 = mx.symbol.Concat(name='Concat2', *[ReLU8,ReLU11,crop_upsampled_flow6_to_5] )\n Convolution2 = mx.symbol.Convolution(name='Convolution2', data=Concat2 , num_filter=2, pad=(1,1), kernel=(3,3), stride=(1,1), no_bias=False)\n deconv4 = mx.symbol.Deconvolution(name='deconv4', data=Concat2 , num_filter=256, pad=(0,0), kernel=(4,4), stride=(2,2), no_bias=False)\n crop_deconv4 = mx.symbol.Crop(name='crop_deconv4', *[deconv4,ReLU6] , offset=(1,1))\n ReLU12 = mx.symbol.LeakyReLU(name='ReLU12', data=crop_deconv4 , act_type='leaky', slope=0.1)\n upsample_flow5to4 = mx.symbol.Deconvolution(name='upsample_flow5to4', data=Convolution2 , num_filter=2, pad=(0,0), kernel=(4,4), stride=(2,2), no_bias=False)\n crop_upsampled_flow5_to_4 = mx.symbol.Crop(name='crop_upsampled_flow5_to_4', *[upsample_flow5to4,ReLU6] , offset=(1,1))\n Concat3 = mx.symbol.Concat(name='Concat3', *[ReLU6,ReLU12,crop_upsampled_flow5_to_4] )\n Convolution3 = mx.symbol.Convolution(name='Convolution3', data=Concat3 , num_filter=2, pad=(1,1), kernel=(3,3), stride=(1,1), no_bias=False)\n deconv3 = mx.symbol.Deconvolution(name='deconv3', data=Concat3 , num_filter=128, pad=(0,0), kernel=(4,4), stride=(2,2), no_bias=False)\n crop_deconv3 = mx.symbol.Crop(name='crop_deconv3', *[deconv3,ReLU4] , offset=(1,1))\n ReLU13 = mx.symbol.LeakyReLU(name='ReLU13', data=crop_deconv3 , act_type='leaky', slope=0.1)\n upsample_flow4to3 = mx.symbol.Deconvolution(name='upsample_flow4to3', data=Convolution3 , num_filter=2, pad=(0,0), kernel=(4,4), stride=(2,2), no_bias=False)\n crop_upsampled_flow4_to_3 = mx.symbol.Crop(name='crop_upsampled_flow4_to_3', *[upsample_flow4to3,ReLU4] , offset=(1,1))\n Concat4 = mx.symbol.Concat(name='Concat4', *[ReLU4,ReLU13,crop_upsampled_flow4_to_3] )\n Convolution4 = mx.symbol.Convolution(name='Convolution4', data=Concat4 , num_filter=2, pad=(1,1), kernel=(3,3), stride=(1,1), no_bias=False)\n deconv2 = mx.symbol.Deconvolution(name='deconv2', data=Concat4 , num_filter=64, pad=(0,0), kernel=(4,4), stride=(2,2), no_bias=False)\n crop_deconv2 = mx.symbol.Crop(name='crop_deconv2', *[deconv2,ReLU2] , offset=(1,1))\n ReLU14 = mx.symbol.LeakyReLU(name='ReLU14', data=crop_deconv2 , act_type='leaky', slope=0.1)\n upsample_flow3to2 = mx.symbol.Deconvolution(name='upsample_flow3to2', data=Convolution4 , num_filter=2, pad=(0,0), kernel=(4,4), stride=(2,2), no_bias=False)\n crop_upsampled_flow3_to_2 = mx.symbol.Crop(name='crop_upsampled_flow3_to_2', *[upsample_flow3to2,ReLU2] , offset=(1,1))\n Concat5 = mx.symbol.Concat(name='Concat5', *[ReLU2,ReLU14,crop_upsampled_flow3_to_2] )\n Concat5 = mx.symbol.Pooling(name='resize_concat5', data=Concat5 , pooling_convention='full', pad=(0,0), kernel=(2,2), stride=(2,2), pool_type='avg')\n Convolution5 = mx.symbol.Convolution(name='Convolution5', data=Concat5 , num_filter=2, pad=(1,1), kernel=(3,3), stride=(1,1), no_bias=False)\n\n Convolution5_scale_bias = mx.sym.Variable(name='Convolution5_scale_bias', lr_mult=0.0)\n Convolution5_scale = mx.symbol.Convolution(name='Convolution5_scale', data=Concat5 , num_filter=1024, pad=(0,0), kernel=(1,1), stride=(1,1),\n bias=Convolution5_scale_bias, no_bias=False)\n return Convolution5 * 2.5, Convolution5_scale\n\n def get_train_symbol(self, cfg):\n\n # config alias for convenient\n num_classes = cfg.dataset.NUM_CLASSES\n num_reg_classes = (2 if cfg.CLASS_AGNOSTIC else num_classes)\n num_anchors = cfg.network.NUM_ANCHORS\n\n data = mx.sym.Variable(name=\"data\")\n data_ref = mx.sym.Variable(name=\"data_ref\")\n eq_flag = mx.sym.Variable(name=\"eq_flag\")\n im_info = mx.sym.Variable(name=\"im_info\")\n gt_boxes = mx.sym.Variable(name=\"gt_boxes\")\n rpn_label = mx.sym.Variable(name='label')\n rpn_bbox_target = mx.sym.Variable(name='bbox_target')\n rpn_bbox_weight = mx.sym.Variable(name='bbox_weight')\n\n # shared convolutional layers\n conv_feat = self.get_resnet_v1(data_ref)\n flow, scale_map = self.get_flownet(data, data_ref)\n flow_grid = mx.sym.GridGenerator(data=flow, transform_type='warp', name='flow_grid')\n warp_conv_feat = mx.sym.BilinearSampler(data=conv_feat, grid=flow_grid, name='warping_feat')\n warp_conv_feat = warp_conv_feat * scale_map\n select_conv_feat = mx.sym.take(mx.sym.Concat(*[warp_conv_feat, conv_feat], dim=0), eq_flag)\n\n conv_feats = mx.sym.SliceChannel(select_conv_feat, axis=1, num_outputs=2)\n\n # RPN layers\n rpn_feat = conv_feats[0]\n rpn_cls_score = mx.sym.Convolution(\n data=rpn_feat, kernel=(1, 1), pad=(0, 0), num_filter=2 * num_anchors, name=\"rpn_cls_score\")\n rpn_bbox_pred = mx.sym.Convolution(\n data=rpn_feat, kernel=(1, 1), pad=(0, 0), num_filter=4 * num_anchors, name=\"rpn_bbox_pred\")\n\n # prepare rpn data\n rpn_cls_score_reshape = mx.sym.Reshape(\n data=rpn_cls_score, shape=(0, 2, -1, 0), name=\"rpn_cls_score_reshape\")\n\n # classification\n rpn_cls_prob = mx.sym.SoftmaxOutput(data=rpn_cls_score_reshape, label=rpn_label, multi_output=True,\n normalization='valid', use_ignore=True, ignore_label=-1, name=\"rpn_cls_prob\")\n # bounding box regression\n if cfg.network.NORMALIZE_RPN:\n rpn_bbox_loss_ = rpn_bbox_weight * mx.sym.smooth_l1(name='rpn_bbox_loss_', scalar=1.0, data=(rpn_bbox_pred - rpn_bbox_target))\n rpn_bbox_pred = mx.sym.Custom(\n bbox_pred=rpn_bbox_pred, op_type='rpn_inv_normalize', num_anchors=num_anchors,\n bbox_mean=cfg.network.ANCHOR_MEANS, bbox_std=cfg.network.ANCHOR_STDS)\n else:\n rpn_bbox_loss_ = rpn_bbox_weight * mx.sym.smooth_l1(name='rpn_bbox_loss_', scalar=3.0, data=(rpn_bbox_pred - rpn_bbox_target))\n rpn_bbox_loss = mx.sym.MakeLoss(name='rpn_bbox_loss', data=rpn_bbox_loss_, grad_scale=1.0 / cfg.TRAIN.RPN_BATCH_SIZE)\n\n # ROI proposal\n rpn_cls_act = mx.sym.SoftmaxActivation(\n data=rpn_cls_score_reshape, mode=\"channel\", name=\"rpn_cls_act\")\n rpn_cls_act_reshape = mx.sym.Reshape(\n data=rpn_cls_act, shape=(0, 2 * num_anchors, -1, 0), name='rpn_cls_act_reshape')\n\n if cfg.TRAIN.CXX_PROPOSAL:\n rois = mx.contrib.sym.Proposal(\n cls_prob=rpn_cls_act_reshape, bbox_pred=rpn_bbox_pred, im_info=im_info, name='rois',\n feature_stride=cfg.network.RPN_FEAT_STRIDE, scales=tuple(cfg.network.ANCHOR_SCALES), ratios=tuple(cfg.network.ANCHOR_RATIOS),\n rpn_pre_nms_top_n=cfg.TRAIN.RPN_PRE_NMS_TOP_N, rpn_post_nms_top_n=cfg.TRAIN.RPN_POST_NMS_TOP_N,\n threshold=cfg.TRAIN.RPN_NMS_THRESH, rpn_min_size=cfg.TRAIN.RPN_MIN_SIZE)\n else:\n rois = mx.sym.Custom(\n cls_prob=rpn_cls_act_reshape, bbox_pred=rpn_bbox_pred, im_info=im_info, name='rois',\n op_type='proposal', feat_stride=cfg.network.RPN_FEAT_STRIDE,\n scales=tuple(cfg.network.ANCHOR_SCALES), ratios=tuple(cfg.network.ANCHOR_RATIOS),\n rpn_pre_nms_top_n=cfg.TRAIN.RPN_PRE_NMS_TOP_N, rpn_post_nms_top_n=cfg.TRAIN.RPN_POST_NMS_TOP_N,\n threshold=cfg.TRAIN.RPN_NMS_THRESH, rpn_min_size=cfg.TRAIN.RPN_MIN_SIZE)\n\n # ROI proposal target\n gt_boxes_reshape = mx.sym.Reshape(data=gt_boxes, shape=(-1, 5), name='gt_boxes_reshape')\n rois, label, bbox_target, bbox_weight = mx.sym.Custom(rois=rois, gt_boxes=gt_boxes_reshape,\n op_type='proposal_target',\n num_classes=num_reg_classes,\n batch_images=cfg.TRAIN.BATCH_IMAGES,\n batch_rois=cfg.TRAIN.BATCH_ROIS,\n cfg=cPickle.dumps(cfg),\n fg_fraction=cfg.TRAIN.FG_FRACTION)\n\n # res5\n rfcn_feat = conv_feats[1]\n rfcn_cls = mx.sym.Convolution(data=rfcn_feat, kernel=(1, 1), num_filter=7*7*num_classes, name=\"rfcn_cls\")\n rfcn_bbox = mx.sym.Convolution(data=rfcn_feat, kernel=(1, 1), num_filter=7*7*4*num_reg_classes, name=\"rfcn_bbox\")\n psroipooled_cls_rois = mx.contrib.sym.PSROIPooling(name='psroipooled_cls_rois', data=rfcn_cls, rois=rois, group_size=7,pooled_size=7,\n output_dim=num_classes, spatial_scale=0.0625)\n psroipooled_loc_rois = mx.contrib.sym.PSROIPooling(name='psroipooled_loc_rois', data=rfcn_bbox, rois=rois, group_size=7,pooled_size=7,\n output_dim=8, spatial_scale=0.0625)\n cls_score = mx.sym.Pooling(name='ave_cls_scors_rois', data=psroipooled_cls_rois, pool_type='avg', global_pool=True, kernel=(7, 7))\n bbox_pred = mx.sym.Pooling(name='ave_bbox_pred_rois', data=psroipooled_loc_rois, pool_type='avg', global_pool=True, kernel=(7, 7))\n cls_score = mx.sym.Reshape(name='cls_score_reshape', data=cls_score, shape=(-1, num_classes))\n bbox_pred = mx.sym.Reshape(name='bbox_pred_reshape', data=bbox_pred, shape=(-1, 4 * num_reg_classes))\n\n\n # classification\n if cfg.TRAIN.ENABLE_OHEM:\n print 'use ohem!'\n labels_ohem, bbox_weights_ohem = mx.sym.Custom(op_type='BoxAnnotatorOHEM', num_classes=num_classes,\n num_reg_classes=num_reg_classes, roi_per_img=cfg.TRAIN.BATCH_ROIS_OHEM,\n cls_score=cls_score, bbox_pred=bbox_pred, labels=label,\n bbox_targets=bbox_target, bbox_weights=bbox_weight)\n cls_prob = mx.sym.SoftmaxOutput(name='cls_prob', data=cls_score, label=labels_ohem, normalization='valid', use_ignore=True, ignore_label=-1)\n bbox_loss_ = bbox_weights_ohem * mx.sym.smooth_l1(name='bbox_loss_', scalar=1.0, data=(bbox_pred - bbox_target))\n bbox_loss = mx.sym.MakeLoss(name='bbox_loss', data=bbox_loss_, grad_scale=1.0 / cfg.TRAIN.BATCH_ROIS_OHEM)\n rcnn_label = labels_ohem\n else:\n cls_prob = mx.sym.SoftmaxOutput(name='cls_prob', data=cls_score, label=label, normalization='valid')\n bbox_loss_ = bbox_weight * mx.sym.smooth_l1(name='bbox_loss_', scalar=1.0, data=(bbox_pred - bbox_target))\n bbox_loss = mx.sym.MakeLoss(name='bbox_loss', data=bbox_loss_, grad_scale=1.0 / cfg.TRAIN.BATCH_ROIS)\n rcnn_label = label\n\n # reshape output\n rcnn_label = mx.sym.Reshape(data=rcnn_label, shape=(cfg.TRAIN.BATCH_IMAGES, -1), name='label_reshape')\n cls_prob = mx.sym.Reshape(data=cls_prob, shape=(cfg.TRAIN.BATCH_IMAGES, -1, num_classes), name='cls_prob_reshape')\n bbox_loss = mx.sym.Reshape(data=bbox_loss, shape=(cfg.TRAIN.BATCH_IMAGES, -1, 4 * num_reg_classes), name='bbox_loss_reshape')\n\n group = mx.sym.Group([rpn_cls_prob, rpn_bbox_loss, cls_prob, bbox_loss, mx.sym.BlockGrad(rcnn_label)])\n self.sym = group\n return group\n\n def get_key_test_symbol(self, cfg):\n\n # config alias for convenient\n num_classes = cfg.dataset.NUM_CLASSES\n num_reg_classes = (2 if cfg.CLASS_AGNOSTIC else num_classes)\n num_anchors = cfg.network.NUM_ANCHORS\n\n data = mx.sym.Variable(name=\"data\")\n im_info = mx.sym.Variable(name=\"im_info\")\n data_key = mx.sym.Variable(name=\"data_key\")\n feat_key = mx.sym.Variable(name=\"feat_key\")\n\n # shared convolutional layers\n conv_feat = self.get_resnet_v1(data)\n conv_feats = mx.sym.SliceChannel(conv_feat, axis=1, num_outputs=2)\n\n # RPN\n rpn_feat = conv_feats[0]\n rpn_cls_score = mx.sym.Convolution(\n data=rpn_feat, kernel=(1, 1), pad=(0, 0), num_filter=2 * num_anchors, name=\"rpn_cls_score\")\n rpn_bbox_pred = mx.sym.Convolution(\n data=rpn_feat, kernel=(1, 1), pad=(0, 0), num_filter=4 * num_anchors, name=\"rpn_bbox_pred\")\n\n if cfg.network.NORMALIZE_RPN:\n rpn_bbox_pred = mx.sym.Custom(\n bbox_pred=rpn_bbox_pred, op_type='rpn_inv_normalize', num_anchors=num_anchors,\n bbox_mean=cfg.network.ANCHOR_MEANS, bbox_std=cfg.network.ANCHOR_STDS)\n\n # ROI Proposal\n rpn_cls_score_reshape = mx.sym.Reshape(\n data=rpn_cls_score, shape=(0, 2, -1, 0), name=\"rpn_cls_score_reshape\")\n rpn_cls_prob = mx.sym.SoftmaxActivation(\n data=rpn_cls_score_reshape, mode=\"channel\", name=\"rpn_cls_prob\")\n rpn_cls_prob_reshape = mx.sym.Reshape(\n data=rpn_cls_prob, shape=(0, 2 * num_anchors, -1, 0), name='rpn_cls_prob_reshape')\n if cfg.TEST.CXX_PROPOSAL:\n rois = mx.contrib.sym.Proposal(\n cls_prob=rpn_cls_prob_reshape, bbox_pred=rpn_bbox_pred, im_info=im_info, name='rois',\n feature_stride=cfg.network.RPN_FEAT_STRIDE, scales=tuple(cfg.network.ANCHOR_SCALES),\n ratios=tuple(cfg.network.ANCHOR_RATIOS),\n rpn_pre_nms_top_n=cfg.TEST.RPN_PRE_NMS_TOP_N, rpn_post_nms_top_n=cfg.TEST.RPN_POST_NMS_TOP_N,\n threshold=cfg.TEST.RPN_NMS_THRESH, rpn_min_size=cfg.TEST.RPN_MIN_SIZE)\n else:\n rois = mx.sym.Custom(\n cls_prob=rpn_cls_prob_reshape, bbox_pred=rpn_bbox_pred, im_info=im_info, name='rois',\n op_type='proposal', feat_stride=cfg.network.RPN_FEAT_STRIDE,\n scales=tuple(cfg.network.ANCHOR_SCALES), ratios=tuple(cfg.network.ANCHOR_RATIOS),\n rpn_pre_nms_top_n=cfg.TEST.RPN_PRE_NMS_TOP_N, rpn_post_nms_top_n=cfg.TEST.RPN_POST_NMS_TOP_N,\n threshold=cfg.TEST.RPN_NMS_THRESH, rpn_min_size=cfg.TEST.RPN_MIN_SIZE)\n\n # res5\n rfcn_feat = conv_feats[1]\n rfcn_cls = mx.sym.Convolution(data=rfcn_feat, kernel=(1, 1), num_filter=7*7*num_classes, name=\"rfcn_cls\")\n rfcn_bbox = mx.sym.Convolution(data=rfcn_feat, kernel=(1, 1), num_filter=7*7*4*num_reg_classes, name=\"rfcn_bbox\")\n psroipooled_cls_rois = mx.contrib.sym.PSROIPooling(name='psroipooled_cls_rois', data=rfcn_cls, rois=rois, group_size=7, pooled_size=7,\n output_dim=num_classes, spatial_scale=0.0625)\n psroipooled_loc_rois = mx.contrib.sym.PSROIPooling(name='psroipooled_loc_rois', data=rfcn_bbox, rois=rois, group_size=7, pooled_size=7,\n output_dim=8, spatial_scale=0.0625)\n cls_score = mx.sym.Pooling(name='ave_cls_scors_rois', data=psroipooled_cls_rois, pool_type='avg', global_pool=True, kernel=(7, 7))\n bbox_pred = mx.sym.Pooling(name='ave_bbox_pred_rois', data=psroipooled_loc_rois, pool_type='avg', global_pool=True, kernel=(7, 7))\n\n # classification\n cls_score = mx.sym.Reshape(name='cls_score_reshape', data=cls_score, shape=(-1, num_classes))\n cls_prob = mx.sym.SoftmaxActivation(name='cls_prob', data=cls_score)\n # bounding box regression\n bbox_pred = mx.sym.Reshape(name='bbox_pred_reshape', data=bbox_pred, shape=(-1, 4 * num_reg_classes))\n\n # reshape output\n cls_prob = mx.sym.Reshape(data=cls_prob, shape=(cfg.TEST.BATCH_IMAGES, -1, num_classes), name='cls_prob_reshape')\n bbox_pred = mx.sym.Reshape(data=bbox_pred, shape=(cfg.TEST.BATCH_IMAGES, -1, 4 * num_reg_classes), name='bbox_pred_reshape')\n\n # group output\n group = mx.sym.Group([data_key, feat_key, conv_feat, rois, cls_prob, bbox_pred])\n self.sym = group\n return group\n\n def get_cur_test_symbol(self, cfg):\n\n # config alias for convenient\n num_classes = cfg.dataset.NUM_CLASSES\n num_reg_classes = (2 if cfg.CLASS_AGNOSTIC else num_classes)\n num_anchors = cfg.network.NUM_ANCHORS\n\n data_cur = mx.sym.Variable(name=\"data\")\n im_info = mx.sym.Variable(name=\"im_info\")\n data_key = mx.sym.Variable(name=\"data_key\")\n conv_feat = mx.sym.Variable(name=\"feat_key\")\n\n # shared convolutional layers\n flow, scale_map = self.get_flownet(data_cur, data_key)\n flow_grid = mx.sym.GridGenerator(data=flow, transform_type='warp', name='flow_grid')\n conv_feat = mx.sym.BilinearSampler(data=conv_feat, grid=flow_grid, name='warping_feat')\n conv_feat = conv_feat * scale_map\n conv_feats = mx.sym.SliceChannel(conv_feat, axis=1, num_outputs=2)\n\n # RPN\n rpn_feat = conv_feats[0]\n rpn_cls_score = mx.sym.Convolution(\n data=rpn_feat, kernel=(1, 1), pad=(0, 0), num_filter=2 * num_anchors, name=\"rpn_cls_score\")\n rpn_bbox_pred = mx.sym.Convolution(\n data=rpn_feat, kernel=(1, 1), pad=(0, 0), num_filter=4 * num_anchors, name=\"rpn_bbox_pred\")\n\n if cfg.network.NORMALIZE_RPN:\n rpn_bbox_pred = mx.sym.Custom(\n bbox_pred=rpn_bbox_pred, op_type='rpn_inv_normalize', num_anchors=num_anchors,\n bbox_mean=cfg.network.ANCHOR_MEANS, bbox_std=cfg.network.ANCHOR_STDS)\n\n # ROI Proposal\n rpn_cls_score_reshape = mx.sym.Reshape(\n data=rpn_cls_score, shape=(0, 2, -1, 0), name=\"rpn_cls_score_reshape\")\n rpn_cls_prob = mx.sym.SoftmaxActivation(\n data=rpn_cls_score_reshape, mode=\"channel\", name=\"rpn_cls_prob\")\n rpn_cls_prob_reshape = mx.sym.Reshape(\n data=rpn_cls_prob, shape=(0, 2 * num_anchors, -1, 0), name='rpn_cls_prob_reshape')\n if cfg.TEST.CXX_PROPOSAL:\n rois = mx.contrib.sym.Proposal(\n cls_prob=rpn_cls_prob_reshape, bbox_pred=rpn_bbox_pred, im_info=im_info, name='rois',\n feature_stride=cfg.network.RPN_FEAT_STRIDE, scales=tuple(cfg.network.ANCHOR_SCALES),\n ratios=tuple(cfg.network.ANCHOR_RATIOS),\n rpn_pre_nms_top_n=cfg.TEST.RPN_PRE_NMS_TOP_N, rpn_post_nms_top_n=cfg.TEST.RPN_POST_NMS_TOP_N,\n threshold=cfg.TEST.RPN_NMS_THRESH, rpn_min_size=cfg.TEST.RPN_MIN_SIZE)\n else:\n rois = mx.sym.Custom(\n cls_prob=rpn_cls_prob_reshape, bbox_pred=rpn_bbox_pred, im_info=im_info, name='rois',\n op_type='proposal', feat_stride=cfg.network.RPN_FEAT_STRIDE,\n scales=tuple(cfg.network.ANCHOR_SCALES), ratios=tuple(cfg.network.ANCHOR_RATIOS),\n rpn_pre_nms_top_n=cfg.TEST.RPN_PRE_NMS_TOP_N, rpn_post_nms_top_n=cfg.TEST.RPN_POST_NMS_TOP_N,\n threshold=cfg.TEST.RPN_NMS_THRESH, rpn_min_size=cfg.TEST.RPN_MIN_SIZE)\n\n # res5\n rfcn_feat = conv_feats[1]\n rfcn_cls = mx.sym.Convolution(data=rfcn_feat, kernel=(1, 1), num_filter=7*7*num_classes, name=\"rfcn_cls\")\n rfcn_bbox = mx.sym.Convolution(data=rfcn_feat, kernel=(1, 1), num_filter=7*7*4*num_reg_classes, name=\"rfcn_bbox\")\n psroipooled_cls_rois = mx.contrib.sym.PSROIPooling(name='psroipooled_cls_rois', data=rfcn_cls, rois=rois, group_size=7, pooled_size=7,\n output_dim=num_classes, spatial_scale=0.0625)\n psroipooled_loc_rois = mx.contrib.sym.PSROIPooling(name='psroipooled_loc_rois', data=rfcn_bbox, rois=rois, group_size=7, pooled_size=7,\n output_dim=8, spatial_scale=0.0625)\n cls_score = mx.sym.Pooling(name='ave_cls_scors_rois', data=psroipooled_cls_rois, pool_type='avg', global_pool=True, kernel=(7, 7))\n bbox_pred = mx.sym.Pooling(name='ave_bbox_pred_rois', data=psroipooled_loc_rois, pool_type='avg', global_pool=True, kernel=(7, 7))\n\n # classification\n cls_score = mx.sym.Reshape(name='cls_score_reshape', data=cls_score, shape=(-1, num_classes))\n cls_prob = mx.sym.SoftmaxActivation(name='cls_prob', data=cls_score)\n # bounding box regression\n bbox_pred = mx.sym.Reshape(name='bbox_pred_reshape', data=bbox_pred, shape=(-1, 4 * num_reg_classes))\n\n # reshape output\n cls_prob = mx.sym.Reshape(data=cls_prob, shape=(cfg.TEST.BATCH_IMAGES, -1, num_classes), name='cls_prob_reshape')\n bbox_pred = mx.sym.Reshape(data=bbox_pred, shape=(cfg.TEST.BATCH_IMAGES, -1, 4 * num_reg_classes), name='bbox_pred_reshape')\n\n # group output\n group = mx.sym.Group([rois, cls_prob, bbox_pred])\n self.sym = group\n return group\n\n def get_batch_test_symbol(self, cfg):\n\n # config alias for convenient\n num_classes = cfg.dataset.NUM_CLASSES\n num_reg_classes = (2 if cfg.CLASS_AGNOSTIC else num_classes)\n num_anchors = cfg.network.NUM_ANCHORS\n\n data_key = mx.sym.Variable(name=\"data_key\")\n data_other = mx.sym.Variable(name=\"data_other\")\n im_info = mx.sym.Variable(name=\"im_info\")\n\n # shared convolutional layers\n conv_feat_key = self.get_resnet_v1(data_key)\n\n data_key_tiled = mx.sym.Custom(data_content=data_key, data_shape=data_other, op_type='tile_as')\n conv_feat_key_tiled = mx.sym.Custom(data_content=conv_feat_key, data_shape=data_other, op_type='tile_as')\n flow, scale_map = self.get_flownet(data_other, data_key_tiled)\n flow_grid = mx.sym.GridGenerator(data=flow, transform_type='warp', name='flow_grid')\n conv_feat_other = mx.sym.BilinearSampler(data=conv_feat_key_tiled, grid=flow_grid, name='warping_feat')\n conv_feat_other = conv_feat_other * scale_map\n\n conv_feat = mx.symbol.Concat(conv_feat_key, conv_feat_other, dim=0)\n\n conv_feats = mx.sym.SliceChannel(conv_feat, axis=1, num_outputs=2)\n\n # RPN\n rpn_feat = conv_feats[0]\n rpn_cls_score = mx.sym.Convolution(\n data=rpn_feat, kernel=(1, 1), pad=(0, 0), num_filter=2 * num_anchors, name=\"rpn_cls_score\")\n rpn_bbox_pred = mx.sym.Convolution(\n data=rpn_feat, kernel=(1, 1), pad=(0, 0), num_filter=4 * num_anchors, name=\"rpn_bbox_pred\")\n\n if cfg.network.NORMALIZE_RPN:\n rpn_bbox_pred = mx.sym.Custom(\n bbox_pred=rpn_bbox_pred, op_type='rpn_inv_normalize', num_anchors=num_anchors,\n bbox_mean=cfg.network.ANCHOR_MEANS, bbox_std=cfg.network.ANCHOR_STDS)\n\n # ROI Proposal\n rpn_cls_score_reshape = mx.sym.Reshape(\n data=rpn_cls_score, shape=(0, 2, -1, 0), name=\"rpn_cls_score_reshape\")\n rpn_cls_prob = mx.sym.SoftmaxActivation(\n data=rpn_cls_score_reshape, mode=\"channel\", name=\"rpn_cls_prob\")\n rpn_cls_prob_reshape = mx.sym.Reshape(\n data=rpn_cls_prob, shape=(0, 2 * num_anchors, -1, 0), name='rpn_cls_prob_reshape')\n if cfg.TEST.CXX_PROPOSAL:\n rois = mx.contrib.sym.MultiProposal(\n cls_prob=rpn_cls_prob_reshape, bbox_pred=rpn_bbox_pred, im_info=im_info, name='rois',\n feature_stride=cfg.network.RPN_FEAT_STRIDE, scales=tuple(cfg.network.ANCHOR_SCALES),\n ratios=tuple(cfg.network.ANCHOR_RATIOS),\n rpn_pre_nms_top_n=cfg.TEST.RPN_PRE_NMS_TOP_N, rpn_post_nms_top_n=cfg.TEST.RPN_POST_NMS_TOP_N,\n threshold=cfg.TEST.RPN_NMS_THRESH, rpn_min_size=cfg.TEST.RPN_MIN_SIZE)\n else:\n NotImplemented\n\n # res5\n rfcn_feat = conv_feats[1]\n rfcn_cls = mx.sym.Convolution(data=rfcn_feat, kernel=(1, 1), num_filter=7*7*num_classes, name=\"rfcn_cls\")\n rfcn_bbox = mx.sym.Convolution(data=rfcn_feat, kernel=(1, 1), num_filter=7*7*4*num_reg_classes, name=\"rfcn_bbox\")\n psroipooled_cls_rois = mx.contrib.sym.PSROIPooling(name='psroipooled_cls_rois', data=rfcn_cls, rois=rois, group_size=7, pooled_size=7,\n output_dim=num_classes, spatial_scale=0.0625)\n psroipooled_loc_rois = mx.contrib.sym.PSROIPooling(name='psroipooled_loc_rois', data=rfcn_bbox, rois=rois, group_size=7, pooled_size=7,\n output_dim=8, spatial_scale=0.0625)\n cls_score = mx.sym.Pooling(name='ave_cls_scors_rois', data=psroipooled_cls_rois, pool_type='avg', global_pool=True, kernel=(7, 7))\n bbox_pred = mx.sym.Pooling(name='ave_bbox_pred_rois', data=psroipooled_loc_rois, pool_type='avg', global_pool=True, kernel=(7, 7))\n\n # classification\n cls_score = mx.sym.Reshape(name='cls_score_reshape', data=cls_score, shape=(-1, num_classes))\n cls_prob = mx.sym.SoftmaxActivation(name='cls_prob', data=cls_score)\n # bounding box regression\n bbox_pred = mx.sym.Reshape(name='bbox_pred_reshape', data=bbox_pred, shape=(-1, 4 * num_reg_classes))\n\n # reshape output\n cls_prob = mx.sym.Reshape(data=cls_prob, shape=(cfg.TEST.BATCH_IMAGES, -1, num_classes), name='cls_prob_reshape')\n bbox_pred = mx.sym.Reshape(data=bbox_pred, shape=(cfg.TEST.BATCH_IMAGES, -1, 4 * num_reg_classes), name='bbox_pred_reshape')\n\n # group output\n group = mx.sym.Group([rois, cls_prob, bbox_pred])\n self.sym = group\n return group\n\n def init_weight(self, cfg, arg_params, aux_params):\n arg_params['Convolution5_scale_weight'] = mx.nd.zeros(shape=self.arg_shape_dict['Convolution5_scale_weight'])\n arg_params['Convolution5_scale_bias'] = mx.nd.ones(shape=self.arg_shape_dict['Convolution5_scale_bias'])\n\n arg_params['feat_conv_3x3_weight'] = mx.random.normal(0, 0.01, shape=self.arg_shape_dict['feat_conv_3x3_weight'])\n arg_params['feat_conv_3x3_bias'] = mx.nd.zeros(shape=self.arg_shape_dict['feat_conv_3x3_bias'])\n\n arg_params['rpn_cls_score_weight'] = mx.random.normal(0, 0.01, shape=self.arg_shape_dict['rpn_cls_score_weight'])\n arg_params['rpn_cls_score_bias'] = mx.nd.zeros(shape=self.arg_shape_dict['rpn_cls_score_bias'])\n arg_params['rpn_bbox_pred_weight'] = mx.random.normal(0, 0.01, shape=self.arg_shape_dict['rpn_bbox_pred_weight'])\n arg_params['rpn_bbox_pred_bias'] = mx.nd.zeros(shape=self.arg_shape_dict['rpn_bbox_pred_bias'])\n\n arg_params['rfcn_cls_weight'] = mx.random.normal(0, 0.01, shape=self.arg_shape_dict['rfcn_cls_weight'])\n arg_params['rfcn_cls_bias'] = mx.nd.zeros(shape=self.arg_shape_dict['rfcn_cls_bias'])\n arg_params['rfcn_bbox_weight'] = mx.random.normal(0, 0.01, shape=self.arg_shape_dict['rfcn_bbox_weight'])\n arg_params['rfcn_bbox_bias'] = mx.nd.zeros(shape=self.arg_shape_dict['rfcn_bbox_bias'])\n" }, { "path": "dff_rfcn/test.py", "content": "# --------------------------------------------------------\n# Deep Feature Flow\n# Copyright (c) 2017 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Modified by Yuwen Xiong\n# --------------------------------------------------------\n# Based on:\n# MX-RCNN\n# Copyright (c) 2016 by Contributors\n# Licence under The Apache 2.0 License\n# https://github.com/ijkguo/mx-rcnn/\n# --------------------------------------------------------\n\nimport _init_paths\n\nimport cv2\nimport argparse\nimport os\nimport sys\nimport time\nimport logging\nfrom config.config import config, update_config\n\ndef parse_args():\n parser = argparse.ArgumentParser(description='Test a R-FCN network')\n # general\n parser.add_argument('--cfg', help='experiment configure file name', required=True, type=str)\n\n args, rest = parser.parse_known_args()\n update_config(args.cfg)\n\n # rcnn\n parser.add_argument('--vis', help='turn on visualization', action='store_true')\n parser.add_argument('--ignore_cache', help='ignore cached results boxes', action='store_true')\n parser.add_argument('--thresh', help='valid detection threshold', default=1e-4, type=float)\n parser.add_argument('--shuffle', help='shuffle data on visualization', action='store_true')\n args = parser.parse_args()\n return args\n\nargs = parse_args()\ncurr_path = os.path.abspath(os.path.dirname(__file__))\nsys.path.insert(0, os.path.join(curr_path, '../external/mxnet', config.MXNET_VERSION))\n\nimport mxnet as mx\nfrom function.test_rcnn import test_rcnn\nfrom utils.create_logger import create_logger\n\n\ndef main():\n ctx = [mx.gpu(int(i)) for i in config.gpus.split(',')]\n print args\n\n logger, final_output_path = create_logger(config.output_path, args.cfg, config.dataset.test_image_set)\n\n test_rcnn(config, config.dataset.dataset, config.dataset.test_image_set, config.dataset.root_path, config.dataset.dataset_path,\n ctx, os.path.join(final_output_path, '..', '_'.join([iset for iset in config.dataset.image_set.split('+')]), config.TRAIN.model_prefix), config.TEST.test_epoch,\n args.vis, args.ignore_cache, args.shuffle, config.TEST.HAS_RPN, config.dataset.proposal, args.thresh, logger=logger, output_path=final_output_path)\n\nif __name__ == '__main__':\n main()\n" }, { "path": "dff_rfcn/train_end2end.py", "content": "# --------------------------------------------------------\n# Deep Feature Flow\n# Copyright (c) 2017 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Modified by Yuwen Xiong\n# --------------------------------------------------------\n# Based on:\n# MX-RCNN\n# Copyright (c) 2016 by Contributors\n# Licence under The Apache 2.0 License\n# https://github.com/ijkguo/mx-rcnn/\n# --------------------------------------------------------\n\nimport _init_paths\n\nimport cv2\nimport time\nimport argparse\nimport logging\nimport pprint\nimport os\nimport sys\nfrom config.config import config, update_config\n\ndef parse_args():\n parser = argparse.ArgumentParser(description='Train R-FCN network')\n # general\n parser.add_argument('--cfg', help='experiment configure file name', required=True, type=str)\n\n args, rest = parser.parse_known_args()\n # update config\n update_config(args.cfg)\n\n # training\n parser.add_argument('--frequent', help='frequency of logging', default=config.default.frequent, type=int)\n args = parser.parse_args()\n return args\n\nargs = parse_args()\ncurr_path = os.path.abspath(os.path.dirname(__file__))\nsys.path.insert(0, os.path.join(curr_path, '../external/mxnet', config.MXNET_VERSION))\n\nimport shutil\nimport numpy as np\nimport mxnet as mx\n\nfrom symbols import *\nfrom core import callback, metric\nfrom core.loader import AnchorLoader\nfrom core.module import MutableModule\nfrom utils.create_logger import create_logger\nfrom utils.load_data import load_gt_roidb, merge_roidb, filter_roidb\nfrom utils.load_model import load_param\nfrom utils.PrefetchingIter import PrefetchingIter\nfrom utils.lr_scheduler import WarmupMultiFactorScheduler\n\n\ndef train_net(args, ctx, pretrained, pretrained_flow, epoch, prefix, begin_epoch, end_epoch, lr, lr_step):\n logger, final_output_path = create_logger(config.output_path, args.cfg, config.dataset.image_set)\n prefix = os.path.join(final_output_path, prefix)\n\n # load symbol\n shutil.copy2(os.path.join(curr_path, 'symbols', config.symbol + '.py'), final_output_path)\n sym_instance = eval(config.symbol + '.' + config.symbol)()\n sym = sym_instance.get_train_symbol(config)\n feat_sym = sym.get_internals()['rpn_cls_score_output']\n\n # setup multi-gpu\n batch_size = len(ctx)\n input_batch_size = config.TRAIN.BATCH_IMAGES * batch_size\n\n # print config\n pprint.pprint(config)\n logger.info('training config:{}\\n'.format(pprint.pformat(config)))\n\n # load dataset and prepare imdb for training\n image_sets = [iset for iset in config.dataset.image_set.split('+')]\n roidbs = [load_gt_roidb(config.dataset.dataset, image_set, config.dataset.root_path, config.dataset.dataset_path,\n flip=config.TRAIN.FLIP)\n for image_set in image_sets]\n roidb = merge_roidb(roidbs)\n roidb = filter_roidb(roidb, config)\n # load training data\n train_data = AnchorLoader(feat_sym, roidb, config, batch_size=input_batch_size, shuffle=config.TRAIN.SHUFFLE, ctx=ctx,\n feat_stride=config.network.RPN_FEAT_STRIDE, anchor_scales=config.network.ANCHOR_SCALES,\n anchor_ratios=config.network.ANCHOR_RATIOS, aspect_grouping=config.TRAIN.ASPECT_GROUPING,\n normalize_target=config.network.NORMALIZE_RPN, bbox_mean=config.network.ANCHOR_MEANS,\n bbox_std=config.network.ANCHOR_STDS)\n\n # infer max shape\n max_data_shape = [('data', (config.TRAIN.BATCH_IMAGES, 3, max([v[0] for v in config.SCALES]), max([v[1] for v in config.SCALES]))),\n ('data_ref', (config.TRAIN.BATCH_IMAGES, 3, max([v[0] for v in config.SCALES]), max([v[1] for v in config.SCALES]))),\n ('eq_flag', (1,))]\n max_data_shape, max_label_shape = train_data.infer_shape(max_data_shape)\n max_data_shape.append(('gt_boxes', (config.TRAIN.BATCH_IMAGES, 100, 5)))\n print 'providing maximum shape', max_data_shape, max_label_shape\n\n data_shape_dict = dict(train_data.provide_data_single + train_data.provide_label_single)\n pprint.pprint(data_shape_dict)\n sym_instance.infer_shape(data_shape_dict)\n\n # load and initialize params\n if config.TRAIN.RESUME:\n print('continue training from ', begin_epoch)\n arg_params, aux_params = load_param(prefix, begin_epoch, convert=True)\n else:\n arg_params, aux_params = load_param(pretrained, epoch, convert=True)\n arg_params_flow, aux_params_flow = load_param(pretrained_flow, epoch, convert=True)\n arg_params.update(arg_params_flow)\n aux_params.update(aux_params_flow)\n sym_instance.init_weight(config, arg_params, aux_params)\n\n # check parameter shapes\n sym_instance.check_parameter_shapes(arg_params, aux_params, data_shape_dict)\n\n # create solver\n fixed_param_prefix = config.network.FIXED_PARAMS\n data_names = [k[0] for k in train_data.provide_data_single]\n label_names = [k[0] for k in train_data.provide_label_single]\n\n mod = MutableModule(sym, data_names=data_names, label_names=label_names,\n logger=logger, context=ctx, max_data_shapes=[max_data_shape for _ in range(batch_size)],\n max_label_shapes=[max_label_shape for _ in range(batch_size)], fixed_param_prefix=fixed_param_prefix)\n\n if config.TRAIN.RESUME:\n mod._preload_opt_states = '%s-%04d.states'%(prefix, begin_epoch)\n\n # decide training params\n # metric\n rpn_eval_metric = metric.RPNAccMetric()\n rpn_cls_metric = metric.RPNLogLossMetric()\n rpn_bbox_metric = metric.RPNL1LossMetric()\n eval_metric = metric.RCNNAccMetric(config)\n cls_metric = metric.RCNNLogLossMetric(config)\n bbox_metric = metric.RCNNL1LossMetric(config)\n eval_metrics = mx.metric.CompositeEvalMetric()\n # rpn_eval_metric, rpn_cls_metric, rpn_bbox_metric, eval_metric, cls_metric, bbox_metric\n for child_metric in [rpn_eval_metric, rpn_cls_metric, rpn_bbox_metric, eval_metric, cls_metric, bbox_metric]:\n eval_metrics.add(child_metric)\n # callback\n batch_end_callback = callback.Speedometer(train_data.batch_size, frequent=args.frequent)\n means = np.tile(np.array(config.TRAIN.BBOX_MEANS), 2 if config.CLASS_AGNOSTIC else config.dataset.NUM_CLASSES)\n stds = np.tile(np.array(config.TRAIN.BBOX_STDS), 2 if config.CLASS_AGNOSTIC else config.dataset.NUM_CLASSES)\n epoch_end_callback = [mx.callback.module_checkpoint(mod, prefix, period=1, save_optimizer_states=True), callback.do_checkpoint(prefix, means, stds)]\n # decide learning rate\n base_lr = lr\n lr_factor = config.TRAIN.lr_factor\n lr_epoch = [float(epoch) for epoch in lr_step.split(',')]\n lr_epoch_diff = [epoch - begin_epoch for epoch in lr_epoch if epoch > begin_epoch]\n lr = base_lr * (lr_factor ** (len(lr_epoch) - len(lr_epoch_diff)))\n lr_iters = [int(epoch * len(roidb) / batch_size) for epoch in lr_epoch_diff]\n print('lr', lr, 'lr_epoch_diff', lr_epoch_diff, 'lr_iters', lr_iters)\n lr_scheduler = WarmupMultiFactorScheduler(lr_iters, lr_factor, config.TRAIN.warmup, config.TRAIN.warmup_lr, config.TRAIN.warmup_step)\n # optimizer\n optimizer_params = {'momentum': config.TRAIN.momentum,\n 'wd': config.TRAIN.wd,\n 'learning_rate': lr,\n 'lr_scheduler': lr_scheduler,\n 'rescale_grad': 1.0,\n 'clip_gradient': None}\n\n if not isinstance(train_data, PrefetchingIter):\n train_data = PrefetchingIter(train_data)\n\n # train\n mod.fit(train_data, eval_metric=eval_metrics, epoch_end_callback=epoch_end_callback,\n batch_end_callback=batch_end_callback, kvstore=config.default.kvstore,\n optimizer='sgd', optimizer_params=optimizer_params,\n arg_params=arg_params, aux_params=aux_params, begin_epoch=begin_epoch, num_epoch=end_epoch)\n\n\ndef main():\n print('Called with argument:', args)\n ctx = [mx.gpu(int(i)) for i in config.gpus.split(',')]\n train_net(args, ctx, config.network.pretrained, config.network.pretrained_flow, config.network.pretrained_epoch, config.TRAIN.model_prefix,\n config.TRAIN.begin_epoch, config.TRAIN.end_epoch, config.TRAIN.lr, config.TRAIN.lr_step)\n\nif __name__ == '__main__':\n main()\n" }, { "path": "experiments/dff_rfcn/cfgs/dff_rfcn_vid_demo.yaml", "content": "---\nMXNET_VERSION: \"mxnet\"\noutput_path: \"./output/dff_rfcn/imagenet_vid\"\ngpus: '0'\nCLASS_AGNOSTIC: true\nSCALES:\n- 600\n- 1000\ndefault:\n frequent: 100\n kvstore: device\nnetwork:\n PIXEL_MEANS:\n - 103.06\n - 115.90\n - 123.15\n IMAGE_STRIDE: 0\n RCNN_FEAT_STRIDE: 16\n RPN_FEAT_STRIDE: 16\n FIXED_PARAMS:\n - conv1\n - bn_conv1\n - res2\n - bn2\n - gamma\n - beta\n ANCHOR_RATIOS:\n - 0.5\n - 1\n - 2\n ANCHOR_SCALES:\n - 8\n - 16\n - 32\n ANCHOR_MEANS:\n - 0.0\n - 0.0\n - 0.0\n - 0.0\n ANCHOR_STDS:\n - 0.1\n - 0.1\n - 0.4\n - 0.4\n NORMALIZE_RPN: TRUE\n NUM_ANCHORS: 9\ndataset:\n NUM_CLASSES: 31\n dataset: ImageNetVID\n dataset_path: \"./data/ILSVRC2015\"\n image_set: DET_train_30classes+VID_train_15frames\n root_path: \"./data\"\n test_image_set: VID_val_videos\n proposal: rpn\nTRAIN:\n lr: 0.00025\n lr_step: '1.333'\n warmup: false\n begin_epoch: 0\n end_epoch: 2\n model_prefix: 'dff_rfcn_vid'\n # whether resume training\n RESUME: false\n # whether flip image\n FLIP: true\n # whether shuffle image\n SHUFFLE: true\n # whether use OHEM\n ENABLE_OHEM: true\n # size of images for each device, 1 for e2e\n BATCH_IMAGES: 1\n # e2e changes behavior of anchor loader and metric\n END2END: true\n # group images with similar aspect ratio\n ASPECT_GROUPING: true\n # R-CNN\n # rcnn rois batch size\n BATCH_ROIS: -1\n BATCH_ROIS_OHEM: 128\n # rcnn rois sampling params\n FG_FRACTION: 0.25\n FG_THRESH: 0.5\n BG_THRESH_HI: 0.5\n BG_THRESH_LO: 0.0\n # rcnn bounding box regression params\n BBOX_REGRESSION_THRESH: 0.5\n BBOX_WEIGHTS:\n - 1.0\n - 1.0\n - 1.0\n - 1.0\n\n # RPN anchor loader\n # rpn anchors batch size\n RPN_BATCH_SIZE: 256\n # rpn anchors sampling params\n RPN_FG_FRACTION: 0.5\n RPN_POSITIVE_OVERLAP: 0.7\n RPN_NEGATIVE_OVERLAP: 0.3\n RPN_CLOBBER_POSITIVES: false\n # rpn bounding box regression params\n RPN_BBOX_WEIGHTS:\n - 1.0\n - 1.0\n - 1.0\n - 1.0\n RPN_POSITIVE_WEIGHT: -1.0\n # used for end2end training\n # RPN proposal\n CXX_PROPOSAL: true\n RPN_NMS_THRESH: 0.7\n RPN_PRE_NMS_TOP_N: 6000\n RPN_POST_NMS_TOP_N: 300\n RPN_MIN_SIZE: 0\n # approximate bounding box regression\n BBOX_NORMALIZATION_PRECOMPUTED: true\n BBOX_MEANS:\n - 0.0\n - 0.0\n - 0.0\n - 0.0\n BBOX_STDS:\n - 0.1\n - 0.1\n - 0.2\n - 0.2\nTEST:\n # use rpn to generate proposal\n HAS_RPN: true\n # size of images for each device\n BATCH_IMAGES: 1\n # RPN proposal\n CXX_PROPOSAL: true\n RPN_NMS_THRESH: 0.7\n RPN_PRE_NMS_TOP_N: 6000\n RPN_POST_NMS_TOP_N: 300\n RPN_MIN_SIZE: 0\n # RCNN nms\n NMS: 0.3\n test_epoch: 2\n" }, { "path": "experiments/dff_rfcn/cfgs/resnet_v1_101_flownet_imagenet_vid_rfcn_end2end_ohem.yaml", "content": "---\nMXNET_VERSION: \"mxnet\"\noutput_path: \"./output/dff_rfcn/imagenet_vid\"\nsymbol: resnet_v1_101_flownet_rfcn\ngpus: '0,1,2,3'\nCLASS_AGNOSTIC: true\nSCALES:\n- 600\n- 1000\ndefault:\n frequent: 100\n kvstore: device\nnetwork:\n pretrained: \"./model/pretrained_model/resnet_v1_101\"\n pretrained_flow: \"./model/pretrained_model/flownet\"\n pretrained_epoch: 0\n PIXEL_MEANS:\n - 103.06\n - 115.90\n - 123.15\n IMAGE_STRIDE: 0\n RCNN_FEAT_STRIDE: 16\n RPN_FEAT_STRIDE: 16\n FIXED_PARAMS:\n - conv1\n - bn_conv1\n - res2\n - bn2\n - gamma\n - beta\n ANCHOR_RATIOS:\n - 0.5\n - 1\n - 2\n ANCHOR_SCALES:\n - 8\n - 16\n - 32\n ANCHOR_MEANS:\n - 0.0\n - 0.0\n - 0.0\n - 0.0\n ANCHOR_STDS:\n - 0.1\n - 0.1\n - 0.4\n - 0.4\n NORMALIZE_RPN: TRUE\n NUM_ANCHORS: 9\ndataset:\n NUM_CLASSES: 31\n dataset: ImageNetVID\n dataset_path: \"./data/ILSVRC2015\"\n image_set: DET_train_30classes+VID_train_15frames\n root_path: \"./data\"\n test_image_set: VID_val_videos\n proposal: rpn\nTRAIN:\n lr: 0.00025\n lr_step: '1.333'\n warmup: false\n begin_epoch: 0\n end_epoch: 2\n model_prefix: 'dff_rfcn_vid'\n # whether resume training\n RESUME: false\n # whether flip image\n FLIP: true\n # whether shuffle image\n SHUFFLE: true\n # whether use OHEM\n ENABLE_OHEM: true\n # size of images for each device, 1 for e2e\n BATCH_IMAGES: 1\n # e2e changes behavior of anchor loader and metric\n END2END: true\n # group images with similar aspect ratio\n ASPECT_GROUPING: true\n # R-CNN\n # rcnn rois batch size\n BATCH_ROIS: -1\n BATCH_ROIS_OHEM: 128\n # rcnn rois sampling params\n FG_FRACTION: 0.25\n FG_THRESH: 0.5\n BG_THRESH_HI: 0.5\n BG_THRESH_LO: 0.0\n # rcnn bounding box regression params\n BBOX_REGRESSION_THRESH: 0.5\n BBOX_WEIGHTS:\n - 1.0\n - 1.0\n - 1.0\n - 1.0\n\n # RPN anchor loader\n # rpn anchors batch size\n RPN_BATCH_SIZE: 256\n # rpn anchors sampling params\n RPN_FG_FRACTION: 0.5\n RPN_POSITIVE_OVERLAP: 0.7\n RPN_NEGATIVE_OVERLAP: 0.3\n RPN_CLOBBER_POSITIVES: false\n # rpn bounding box regression params\n RPN_BBOX_WEIGHTS:\n - 1.0\n - 1.0\n - 1.0\n - 1.0\n RPN_POSITIVE_WEIGHT: -1.0\n # used for end2end training\n # RPN proposal\n CXX_PROPOSAL: true\n RPN_NMS_THRESH: 0.7\n RPN_PRE_NMS_TOP_N: 6000\n RPN_POST_NMS_TOP_N: 300\n RPN_MIN_SIZE: 0\n # approximate bounding box regression\n BBOX_NORMALIZATION_PRECOMPUTED: true\n BBOX_MEANS:\n - 0.0\n - 0.0\n - 0.0\n - 0.0\n BBOX_STDS:\n - 0.1\n - 0.1\n - 0.2\n - 0.2\nTEST:\n # use rpn to generate proposal\n HAS_RPN: true\n # size of images for each device\n BATCH_IMAGES: 1\n # RPN proposal\n CXX_PROPOSAL: true\n RPN_NMS_THRESH: 0.7\n RPN_PRE_NMS_TOP_N: 6000\n RPN_POST_NMS_TOP_N: 300\n RPN_MIN_SIZE: 0\n # RCNN nms\n NMS: 0.3\n test_epoch: 2\n" }, { "path": "experiments/dff_rfcn/dff_rfcn_end2end_train_test.py", "content": "# --------------------------------------------------------\n# Deep Feature Flow\n# Copyright (c) 2017 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Written by Yuwen Xiong\n# --------------------------------------------------------\n\nimport os\nimport sys\nos.environ['PYTHONUNBUFFERED'] = '1'\nos.environ['MXNET_CUDNN_AUTOTUNE_DEFAULT'] = '0'\nos.environ['MXNET_ENABLE_GPU_P2P'] = '0'\nthis_dir = os.path.dirname(__file__)\nsys.path.insert(0, os.path.join(this_dir, '..', '..', 'dff_rfcn'))\n\nimport train_end2end\nimport test\n\nif __name__ == \"__main__\":\n train_end2end.main()\n test.main()\n\n\n\n\n" }, { "path": "experiments/dff_rfcn/dff_rfcn_test.py", "content": "# --------------------------------------------------------\n# Deep Feature Flow\n# Copyright (c) 2017 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Written by Yuwen Xiong\n# --------------------------------------------------------\n\nimport os\nimport sys\nos.environ['PYTHONUNBUFFERED'] = '1'\nos.environ['MXNET_CUDNN_AUTOTUNE_DEFAULT'] = '0'\nos.environ['MXNET_ENABLE_GPU_P2P'] = '0'\nthis_dir = os.path.dirname(__file__)\nsys.path.insert(0, os.path.join(this_dir, '..', '..', 'dff_rfcn'))\n\nimport test\n\nif __name__ == \"__main__\":\n test.main()\n" }, { "path": "experiments/rfcn/cfgs/resnet_v1_101_imagenet_vid_rfcn_end2end_ohem.yaml", "content": "---\nMXNET_VERSION: \"mxnet\"\noutput_path: \"./output/rfcn/imagenet_vid\"\nsymbol: resnet_v1_101_rfcn\ngpus: '0,1,2,3'\nCLASS_AGNOSTIC: true\nSCALES:\n- 600\n- 1000\ndefault:\n frequent: 100\n kvstore: device\nnetwork:\n pretrained: \"./model/pretrained_model/resnet_v1_101\"\n pretrained_epoch: 0\n PIXEL_MEANS:\n - 103.06\n - 115.90\n - 123.15\n IMAGE_STRIDE: 0\n RCNN_FEAT_STRIDE: 16\n RPN_FEAT_STRIDE: 16\n FIXED_PARAMS:\n - conv1\n - bn_conv1\n - res2\n - bn2\n - gamma\n - beta\n ANCHOR_RATIOS:\n - 0.5\n - 1\n - 2\n ANCHOR_SCALES:\n - 8\n - 16\n - 32\n ANCHOR_MEANS:\n - 0.0\n - 0.0\n - 0.0\n - 0.0\n ANCHOR_STDS:\n - 0.1\n - 0.1\n - 0.4\n - 0.4\n NORMALIZE_RPN: TRUE\n NUM_ANCHORS: 9\ndataset:\n NUM_CLASSES: 31\n dataset: ImageNetVID\n dataset_path: \"./data/ILSVRC2015\"\n image_set: DET_train_30classes+VID_train_15frames\n root_path: \"./data\"\n test_image_set: VID_val_frames\n proposal: rpn\nTRAIN:\n lr: 0.00025\n lr_step: '1.333'\n warmup: false\n begin_epoch: 0\n end_epoch: 2\n model_prefix: 'rfcn_vid'\n # whether resume training\n RESUME: false\n # whether flip image\n FLIP: true\n # whether shuffle image\n SHUFFLE: true\n # whether use OHEM\n ENABLE_OHEM: true\n # size of images for each device, 1 for e2e\n BATCH_IMAGES: 1\n # e2e changes behavior of anchor loader and metric\n END2END: true\n # group images with similar aspect ratio\n ASPECT_GROUPING: true\n # R-CNN\n # rcnn rois batch size\n BATCH_ROIS: -1\n BATCH_ROIS_OHEM: 128\n # rcnn rois sampling params\n FG_FRACTION: 0.25\n FG_THRESH: 0.5\n BG_THRESH_HI: 0.5\n BG_THRESH_LO: 0.0\n # rcnn bounding box regression params\n BBOX_REGRESSION_THRESH: 0.5\n BBOX_WEIGHTS:\n - 1.0\n - 1.0\n - 1.0\n - 1.0\n\n # RPN anchor loader\n # rpn anchors batch size\n RPN_BATCH_SIZE: 256\n # rpn anchors sampling params\n RPN_FG_FRACTION: 0.5\n RPN_POSITIVE_OVERLAP: 0.7\n RPN_NEGATIVE_OVERLAP: 0.3\n RPN_CLOBBER_POSITIVES: false\n # rpn bounding box regression params\n RPN_BBOX_WEIGHTS:\n - 1.0\n - 1.0\n - 1.0\n - 1.0\n RPN_POSITIVE_WEIGHT: -1.0\n # used for end2end training\n # RPN proposal\n CXX_PROPOSAL: true\n RPN_NMS_THRESH: 0.7\n RPN_PRE_NMS_TOP_N: 6000\n RPN_POST_NMS_TOP_N: 300\n RPN_MIN_SIZE: 0\n # approximate bounding box regression\n BBOX_NORMALIZATION_PRECOMPUTED: true\n BBOX_MEANS:\n - 0.0\n - 0.0\n - 0.0\n - 0.0\n BBOX_STDS:\n - 0.1\n - 0.1\n - 0.2\n - 0.2\nTEST:\n # use rpn to generate proposal\n HAS_RPN: true\n # size of images for each device\n BATCH_IMAGES: 1\n # RPN proposal\n CXX_PROPOSAL: true\n RPN_NMS_THRESH: 0.7\n RPN_PRE_NMS_TOP_N: 6000\n RPN_POST_NMS_TOP_N: 300\n RPN_MIN_SIZE: 0\n # RCNN nms\n NMS: 0.3\n test_epoch: 2\n" }, { "path": "experiments/rfcn/cfgs/rfcn_vid_demo.yaml", "content": "---\nMXNET_VERSION: \"mxnet\"\noutput_path: \"./output/rfcn/imagenet_vid\"\ngpus: '0'\nCLASS_AGNOSTIC: true\nSCALES:\n- 600\n- 1000\ndefault:\n frequent: 100\n kvstore: device\nnetwork:\n PIXEL_MEANS:\n - 103.06\n - 115.90\n - 123.15\n IMAGE_STRIDE: 0\n RCNN_FEAT_STRIDE: 16\n RPN_FEAT_STRIDE: 16\n FIXED_PARAMS:\n - conv1\n - bn_conv1\n - res2\n - bn2\n - gamma\n - beta\n ANCHOR_RATIOS:\n - 0.5\n - 1\n - 2\n ANCHOR_SCALES:\n - 8\n - 16\n - 32\n ANCHOR_MEANS:\n - 0.0\n - 0.0\n - 0.0\n - 0.0\n ANCHOR_STDS:\n - 0.1\n - 0.1\n - 0.4\n - 0.4\n NORMALIZE_RPN: TRUE\n NUM_ANCHORS: 9\ndataset:\n NUM_CLASSES: 31\n dataset: ImageNetVID\n dataset_path: \"./data/ILSVRC2015\"\n image_set: DET_train_30classes+VID_train_15frames\n root_path: \"./data\"\n test_image_set: VID_val_frames\n proposal: rpn\nTRAIN:\n lr: 0.00025\n lr_step: '1.333'\n warmup: false\n begin_epoch: 0\n end_epoch: 2\n model_prefix: 'rfcn_vid'\n # whether resume training\n RESUME: false\n # whether flip image\n FLIP: true\n # whether shuffle image\n SHUFFLE: true\n # whether use OHEM\n ENABLE_OHEM: true\n # size of images for each device, 1 for e2e\n BATCH_IMAGES: 1\n # e2e changes behavior of anchor loader and metric\n END2END: true\n # group images with similar aspect ratio\n ASPECT_GROUPING: true\n # R-CNN\n # rcnn rois batch size\n BATCH_ROIS: -1\n BATCH_ROIS_OHEM: 128\n # rcnn rois sampling params\n FG_FRACTION: 0.25\n FG_THRESH: 0.5\n BG_THRESH_HI: 0.5\n BG_THRESH_LO: 0.0\n # rcnn bounding box regression params\n BBOX_REGRESSION_THRESH: 0.5\n BBOX_WEIGHTS:\n - 1.0\n - 1.0\n - 1.0\n - 1.0\n\n # RPN anchor loader\n # rpn anchors batch size\n RPN_BATCH_SIZE: 256\n # rpn anchors sampling params\n RPN_FG_FRACTION: 0.5\n RPN_POSITIVE_OVERLAP: 0.7\n RPN_NEGATIVE_OVERLAP: 0.3\n RPN_CLOBBER_POSITIVES: false\n # rpn bounding box regression params\n RPN_BBOX_WEIGHTS:\n - 1.0\n - 1.0\n - 1.0\n - 1.0\n RPN_POSITIVE_WEIGHT: -1.0\n # used for end2end training\n # RPN proposal\n CXX_PROPOSAL: true\n RPN_NMS_THRESH: 0.7\n RPN_PRE_NMS_TOP_N: 6000\n RPN_POST_NMS_TOP_N: 300\n RPN_MIN_SIZE: 0\n # approximate bounding box regression\n BBOX_NORMALIZATION_PRECOMPUTED: true\n BBOX_MEANS:\n - 0.0\n - 0.0\n - 0.0\n - 0.0\n BBOX_STDS:\n - 0.1\n - 0.1\n - 0.2\n - 0.2\nTEST:\n # use rpn to generate proposal\n HAS_RPN: true\n # size of images for each device\n BATCH_IMAGES: 1\n # RPN proposal\n CXX_PROPOSAL: true\n RPN_NMS_THRESH: 0.7\n RPN_PRE_NMS_TOP_N: 6000\n RPN_POST_NMS_TOP_N: 300\n RPN_MIN_SIZE: 0\n # RCNN nms\n NMS: 0.3\n test_epoch: 2\n" }, { "path": "experiments/rfcn/rfcn_end2end_train_test.py", "content": "# --------------------------------------------------------\n# Deep Feature Flow\n# Copyright (c) 2017 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Written by Yuwen Xiong\n# --------------------------------------------------------\n\nimport os\nimport sys\nos.environ['PYTHONUNBUFFERED'] = '1'\nos.environ['MXNET_CUDNN_AUTOTUNE_DEFAULT'] = '0'\nos.environ['MXNET_ENABLE_GPU_P2P'] = '0'\nthis_dir = os.path.dirname(__file__)\nsys.path.insert(0, os.path.join(this_dir, '..', '..', 'rfcn'))\n\nimport train_end2end\nimport test\n\nif __name__ == \"__main__\":\n train_end2end.main()\n test.main()\n\n\n\n\n" }, { "path": "experiments/rfcn/rfcn_test.py", "content": "# --------------------------------------------------------\n# Deep Feature Flow\n# Copyright (c) 2017 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Written by Yuwen Xiong\n# --------------------------------------------------------\n\nimport os\nimport sys\nos.environ['PYTHONUNBUFFERED'] = '1'\nos.environ['MXNET_CUDNN_AUTOTUNE_DEFAULT'] = '0'\nos.environ['MXNET_ENABLE_GPU_P2P'] = '0'\nthis_dir = os.path.dirname(__file__)\nsys.path.insert(0, os.path.join(this_dir, '..', '..', 'rfcn'))\n\nimport test\n\nif __name__ == \"__main__\":\n test.main()\n" }, { "path": "green2.py", "content": "# _*_ coding: utf-8 _*_\n\nimport datetime\nimport os\nimport random\nfrom random import choice\n\ncomments = [\n 'Update ReactFiberScheduler',\n 'Update CHANGELOG for 16.7',\n 'v16.7.0',\n 'Move SchedulerFeatureFlags fork to src directory to fix lint',\n 'Make scheduler debugging feature flag static',\n 'Create separate SchedulerFeatureFlags instead of using ReactFeatureFl…',\n 'Memoize promise listeners to prevent exponential growth',\n 'Removed Fabric-specific feature flag files and updated Rollup to use',\n 'Updated version incrementing suggestion in release script based on te…',\n 'Implement pauseExecution, continueExecution, dumpQueue for Scheduler',\n 'Added ErrorBoundary tests for useEffect and useLayoutEffect',\n 'Enable hooks in fabric',\n 'Tweaked wording for v8 \"performance cliff\" issue',\n 'Automated fixture tests',\n 'Removed unnecessary externals from Jest bundles',\n 'Fixed scheduler setTimeout fallback',\n 'Fix bug in cloneHook',\n '[Fire] Add initial build infrastructure',\n 'Adding isMemo check to react-is package',\n 'Remove useMutationEffect',\n 'Add a null type test for memo',\n 'Minified Deep Learn ',\n 'Deep net learn',\n 'update credits',\n 'update url',\n 'Update rnnrbm.py',\n 'Update segmentation tutorials and .gitignore.',\n 'Preload data for 1D cortical dataset',\n 'Results for unet',\n 'added per class jaccard',\n 'Formatting',\n 'Remove files that are not used any more',\n 'More small fixes',\n 'data augmentation fixed',\n 'remove loading pretrained weights',\n 'index updated with new links',\n 'removed files .pyc',\n 'small change',\n 'import conv2DDDNlayer',\n 'fix input dim and details',\n 'fix dataset loader for em, name for polyps',\n 'adde training loop link',\n 'small changes',\n 'relative paths',\n 'with BN',\n 'update from pascal comment',\n 'polyps results image',\n 'acknowledgements',\n 'added code',\n 'first commit for fcn1D segmentations',\n 'deleted files',\n 'figure 4 description',\n 'explanation ground truth vs predicted segmentation',\n 'FCN description fixed',\n 'cortical layers imagse'\n]\n\ndef modify():\n file = open('zero.md', 'r')\n flag = file.readline() == '0'\n file.close()\n file = open('zero.md', 'w+')\n if flag:\n file.write('1')\n else:\n file.write('0')\n file.close()\n\n\ndef commit():\n cm = 'git commit -a -m \"' + choice(comments) + '\"'\n print(cm)\n os.system(cm)\n\n\ndef set_sys_time(day, month, year):\n os.system('date %02d%02d1720%04d' % (month, day, year))\n\n\ndef trick_commit(year, month, day):\n set_sys_time(year, month, day)\n modify()\n commit()\n\n\ndef daily_commit(start_date, end_date):\n count = 0\n while (count < 2):\n for i in range((end_date - start_date).days - 5):\n if (i%random.randint(1,2)):\n cur_date = start_date + datetime.timedelta(days=(i+random.randint(1,5)))\n trick_commit(cur_date.day, cur_date.month, cur_date.year)\n count = count + 1\n\n\nif __name__ == '__main__':\n daily_commit(datetime.date(2015, 2, 21), datetime.date(2018, 12, 19))" }, { "path": "init.bat", "content": "cd /d %~dp0\nmkdir .\\output\nmkdir .\\external\\mxnet\nmkdir .\\model\\pretrained_model\npause\ncd lib\\bbox\npython setup_windows.py build_ext --inplace\ncd ..\\nms\npython setup_windows.py build_ext --inplace\npython setup_windows_cuda.py build_ext --inplace\ncd ..\\..\npause\n" }, { "path": "init.sh", "content": "#!/bin/bash\n\nmkdir -p ./output\nmkdir -p ./external/mxnet\nmkdir -p ./model/pretrained_model\n\ncd lib/bbox\npython setup_linux.py build_ext --inplace\ncd ../nms\npython setup_linux.py build_ext --inplace\ncd ../..\n" }, { "path": "lib/Makefile", "content": "all:\n\tcd nms/; python setup.py build_ext --inplace; rm -rf build; cd ../../\n\tcd bbox/; python setup.py build_ext --inplace; rm -rf build; cd ../../\n\tcd dataset/pycocotools/; python setup.py build_ext --inplace; rm -rf build; cd ../../\nclean:\n\tcd nms/; rm *.so *.c *.cpp; cd ../../\n\tcd bbox/; rm *.so *.c *.cpp; cd ../../\n\tcd dataset/pycocotools/; rm *.so; cd ../../\n" }, { "path": "lib/__init__.py", "content": "" }, { "path": "lib/bbox/.gitignore", "content": "*.c\n*.cpp" }, { "path": "lib/bbox/__init__.py", "content": "" }, { "path": "lib/bbox/bbox.pyx", "content": "# --------------------------------------------------------\n# Deep Feature Flow\n# Copyright (c) 2017 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Written by Sergey Karayev\n# Modified by Yuwen Xiong\n# --------------------------------------------------------\n# Based on:\n# py-faster-rcnn\n# Copyright (c) 2015 Microsoft\n# Licence under The MIT License\n# https://github.com/rbgirshick/py-faster-rcnn\n# --------------------------------------------------------\n\ncimport cython\nimport numpy as np\ncimport numpy as np\n\nDTYPE = np.float\nctypedef np.float_t DTYPE_t\n\ndef bbox_overlaps_cython(\n np.ndarray[DTYPE_t, ndim=2] boxes,\n np.ndarray[DTYPE_t, ndim=2] query_boxes):\n \"\"\"\n Parameters\n ----------\n boxes: (N, 4) ndarray of float\n query_boxes: (K, 4) ndarray of float\n Returns\n -------\n overlaps: (N, K) ndarray of overlap between boxes and query_boxes\n \"\"\"\n cdef unsigned int N = boxes.shape[0]\n cdef unsigned int K = query_boxes.shape[0]\n cdef np.ndarray[DTYPE_t, ndim=2] overlaps = np.zeros((N, K), dtype=DTYPE)\n cdef DTYPE_t iw, ih, box_area\n cdef DTYPE_t ua\n cdef unsigned int k, n\n for k in range(K):\n box_area = (\n (query_boxes[k, 2] - query_boxes[k, 0] + 1) *\n (query_boxes[k, 3] - query_boxes[k, 1] + 1)\n )\n for n in range(N):\n iw = (\n min(boxes[n, 2], query_boxes[k, 2]) -\n max(boxes[n, 0], query_boxes[k, 0]) + 1\n )\n if iw > 0:\n ih = (\n min(boxes[n, 3], query_boxes[k, 3]) -\n max(boxes[n, 1], query_boxes[k, 1]) + 1\n )\n if ih > 0:\n ua = float(\n (boxes[n, 2] - boxes[n, 0] + 1) *\n (boxes[n, 3] - boxes[n, 1] + 1) +\n box_area - iw * ih\n )\n overlaps[n, k] = iw * ih / ua\n return overlaps\n" }, { "path": "lib/bbox/bbox_regression.py", "content": "# --------------------------------------------------------\n# Deep Feature Flow\n# Copyright (c) 2017 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Modified by Yuwen Xiong\n# --------------------------------------------------------\n# Based on:\n# py-faster-rcnn\n# Copyright (c) 2016 by Contributors\n# Licence under The MIT License\n# https://github.com/rbgirshick/py-faster-rcnn\n# --------------------------------------------------------\n\n\"\"\"\nThis file has functions about generating bounding box regression targets\n\"\"\"\n\nimport numpy as np\n\nfrom bbox_transform import bbox_overlaps, bbox_transform\n\n\ndef compute_bbox_regression_targets(rois, overlaps, labels, cfg):\n \"\"\"\n given rois, overlaps, gt labels, compute bounding box regression targets\n :param rois: roidb[i]['boxes'] k * 4\n :param overlaps: roidb[i]['max_overlaps'] k * 1\n :param labels: roidb[i]['max_classes'] k * 1\n :return: targets[i][class, dx, dy, dw, dh] k * 5\n \"\"\"\n # Ensure ROIs are floats\n rois = rois.astype(np.float, copy=False)\n\n # Sanity check\n if len(rois) != len(overlaps):\n print 'bbox regression: this should not happen'\n\n # Indices of ground-truth ROIs\n gt_inds = np.where(overlaps == 1)[0]\n if len(gt_inds) == 0:\n print 'something wrong : zero ground truth rois'\n # Indices of examples for which we try to make predictions\n ex_inds = np.where(overlaps >= cfg.TRAIN.BBOX_REGRESSION_THRESH)[0]\n\n # Get IoU overlap between each ex ROI and gt ROI\n ex_gt_overlaps = bbox_overlaps(rois[ex_inds, :], rois[gt_inds, :])\n\n # Find which gt ROI each ex ROI has max overlap with:\n # this will be the ex ROI's gt target\n gt_assignment = ex_gt_overlaps.argmax(axis=1)\n gt_rois = rois[gt_inds[gt_assignment], :]\n ex_rois = rois[ex_inds, :]\n\n targets = np.zeros((rois.shape[0], 5), dtype=np.float32)\n targets[ex_inds, 0] = labels[ex_inds]\n targets[ex_inds, 1:] = bbox_transform(ex_rois, gt_rois)\n return targets\n\n\ndef add_bbox_regression_targets(roidb, cfg):\n \"\"\"\n given roidb, add ['bbox_targets'] and normalize bounding box regression targets\n :param roidb: roidb to be processed. must have gone through imdb.prepare_roidb\n :return: means, std variances of targets\n \"\"\"\n print 'add bounding box regression targets'\n assert len(roidb) > 0\n assert 'max_classes' in roidb[0]\n\n num_images = len(roidb)\n num_classes = 2 if cfg.CLASS_AGNOSTIC else roidb[0]['gt_overlaps'].shape[1]\n\n for im_i in range(num_images):\n rois = roidb[im_i]['boxes']\n max_overlaps = roidb[im_i]['max_overlaps']\n max_classes = roidb[im_i]['max_classes']\n roidb[im_i]['bbox_targets'] = compute_bbox_regression_targets(rois, max_overlaps, max_classes, cfg)\n\n if cfg.TRAIN.BBOX_NORMALIZATION_PRECOMPUTED:\n # use fixed / precomputed means and stds instead of empirical values\n means = np.tile(np.array(cfg.TRAIN.BBOX_MEANS), (num_classes, 1))\n stds = np.tile(np.array(cfg.TRAIN.BBOX_STDS), (num_classes, 1))\n else:\n # compute mean, std values\n class_counts = np.zeros((num_classes, 1)) + 1e-14\n sums = np.zeros((num_classes, 4))\n squared_sums = np.zeros((num_classes, 4))\n for im_i in range(num_images):\n targets = roidb[im_i]['bbox_targets']\n for cls in range(1, num_classes):\n cls_indexes = np.where(targets[:, 0] > 0)[0] if cfg.CLASS_AGNOSTIC else np.where(targets[:, 0] == cls)[0]\n if cls_indexes.size > 0:\n class_counts[cls] += cls_indexes.size\n sums[cls, :] += targets[cls_indexes, 1:].sum(axis=0)\n squared_sums[cls, :] += (targets[cls_indexes, 1:] ** 2).sum(axis=0)\n\n means = sums / class_counts\n # var(x) = E(x^2) - E(x)^2\n stds = np.sqrt(squared_sums / class_counts - means ** 2)\n\n print 'bbox target means:'\n print means\n print means[1:, :].mean(axis=0) # ignore bg class\n print 'bbox target stdevs:'\n print stds\n print stds[1:, :].mean(axis=0) # ignore bg class\n\n\n # normalized targets\n for im_i in range(num_images):\n targets = roidb[im_i]['bbox_targets']\n for cls in range(1, num_classes):\n cls_indexes = np.where(targets[:, 0] > 0) if cfg.CLASS_AGNOSTIC else np.where(targets[:, 0] == cls)[0]\n roidb[im_i]['bbox_targets'][cls_indexes, 1:] -= means[cls, :]\n roidb[im_i]['bbox_targets'][cls_indexes, 1:] /= stds[cls, :]\n\n return means.ravel(), stds.ravel()\n\n\ndef expand_bbox_regression_targets(bbox_targets_data, num_classes, cfg):\n \"\"\"\n expand from 5 to 4 * num_classes; only the right class has non-zero bbox regression targets\n :param bbox_targets_data: [k * 5]\n :param num_classes: number of classes\n :return: bbox target processed [k * 4 num_classes]\n bbox_weights ! only foreground boxes have bbox regression computation!\n \"\"\"\n classes = bbox_targets_data[:, 0]\n if cfg.CLASS_AGNOSTIC:\n num_classes = 2\n bbox_targets = np.zeros((classes.size, 4 * num_classes), dtype=np.float32)\n bbox_weights = np.zeros(bbox_targets.shape, dtype=np.float32)\n indexes = np.where(classes > 0)[0]\n for index in indexes:\n cls = classes[index]\n start = int(4 * 1 if cls > 0 else 0) if cfg.CLASS_AGNOSTIC else int(4 * cls)\n end = start + 4\n bbox_targets[index, start:end] = bbox_targets_data[index, 1:]\n bbox_weights[index, start:end] = cfg.TRAIN.BBOX_WEIGHTS\n return bbox_targets, bbox_weights\n\n" }, { "path": "lib/bbox/bbox_transform.py", "content": "# --------------------------------------------------------\n# Deep Feature Flow\n# Copyright (c) 2017 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Modified by Yuwen Xiong\n# --------------------------------------------------------\n# Based on:\n# py-faster-rcnn\n# Copyright (c) 2016 by Contributors\n# Licence under The MIT License\n# https://github.com/rbgirshick/py-faster-rcnn\n# --------------------------------------------------------\n\nimport numpy as np\nfrom bbox import bbox_overlaps_cython\n\n\ndef bbox_overlaps(boxes, query_boxes):\n return bbox_overlaps_cython(boxes, query_boxes)\n\n\ndef bbox_overlaps_py(boxes, query_boxes):\n \"\"\"\n determine overlaps between boxes and query_boxes\n :param boxes: n * 4 bounding boxes\n :param query_boxes: k * 4 bounding boxes\n :return: overlaps: n * k overlaps\n \"\"\"\n n_ = boxes.shape[0]\n k_ = query_boxes.shape[0]\n overlaps = np.zeros((n_, k_), dtype=np.float)\n for k in range(k_):\n query_box_area = (query_boxes[k, 2] - query_boxes[k, 0] + 1) * (query_boxes[k, 3] - query_boxes[k, 1] + 1)\n for n in range(n_):\n iw = min(boxes[n, 2], query_boxes[k, 2]) - max(boxes[n, 0], query_boxes[k, 0]) + 1\n if iw > 0:\n ih = min(boxes[n, 3], query_boxes[k, 3]) - max(boxes[n, 1], query_boxes[k, 1]) + 1\n if ih > 0:\n box_area = (boxes[n, 2] - boxes[n, 0] + 1) * (boxes[n, 3] - boxes[n, 1] + 1)\n all_area = float(box_area + query_box_area - iw * ih)\n overlaps[n, k] = iw * ih / all_area\n return overlaps\n\n\ndef clip_boxes(boxes, im_shape):\n \"\"\"\n Clip boxes to image boundaries.\n :param boxes: [N, 4* num_classes]\n :param im_shape: tuple of 2\n :return: [N, 4* num_classes]\n \"\"\"\n # x1 >= 0\n boxes[:, 0::4] = np.maximum(np.minimum(boxes[:, 0::4], im_shape[1] - 1), 0)\n # y1 >= 0\n boxes[:, 1::4] = np.maximum(np.minimum(boxes[:, 1::4], im_shape[0] - 1), 0)\n # x2 < im_shape[1]\n boxes[:, 2::4] = np.maximum(np.minimum(boxes[:, 2::4], im_shape[1] - 1), 0)\n # y2 < im_shape[0]\n boxes[:, 3::4] = np.maximum(np.minimum(boxes[:, 3::4], im_shape[0] - 1), 0)\n return boxes\n\ndef filter_boxes(boxes, min_size):\n \"\"\"\n filter small boxes.\n :param boxes: [N, 4* num_classes]\n :param min_size:\n :return: keep:\n \"\"\"\n ws = boxes[:, 2] - boxes[:, 0] + 1\n hs = boxes[:, 3] - boxes[:, 1] + 1\n keep = np.where((ws >= min_size) & (hs >= min_size))[0]\n return keep\n\ndef nonlinear_transform(ex_rois, gt_rois):\n \"\"\"\n compute bounding box regression targets from ex_rois to gt_rois\n :param ex_rois: [N, 4]\n :param gt_rois: [N, 4]\n :return: [N, 4]\n \"\"\"\n assert ex_rois.shape[0] == gt_rois.shape[0], 'inconsistent rois number'\n\n ex_widths = ex_rois[:, 2] - ex_rois[:, 0] + 1.0\n ex_heights = ex_rois[:, 3] - ex_rois[:, 1] + 1.0\n ex_ctr_x = ex_rois[:, 0] + 0.5 * (ex_widths - 1.0)\n ex_ctr_y = ex_rois[:, 1] + 0.5 * (ex_heights - 1.0)\n\n gt_widths = gt_rois[:, 2] - gt_rois[:, 0] + 1.0\n gt_heights = gt_rois[:, 3] - gt_rois[:, 1] + 1.0\n gt_ctr_x = gt_rois[:, 0] + 0.5 * (gt_widths - 1.0)\n gt_ctr_y = gt_rois[:, 1] + 0.5 * (gt_heights - 1.0)\n\n targets_dx = (gt_ctr_x - ex_ctr_x) / (ex_widths + 1e-14)\n targets_dy = (gt_ctr_y - ex_ctr_y) / (ex_heights + 1e-14)\n targets_dw = np.log(gt_widths / ex_widths)\n targets_dh = np.log(gt_heights / ex_heights)\n\n targets = np.vstack(\n (targets_dx, targets_dy, targets_dw, targets_dh)).transpose()\n return targets\n\n\ndef nonlinear_pred(boxes, box_deltas):\n \"\"\"\n Transform the set of class-agnostic boxes into class-specific boxes\n by applying the predicted offsets (box_deltas)\n :param boxes: !important [N 4]\n :param box_deltas: [N, 4 * num_classes]\n :return: [N 4 * num_classes]\n \"\"\"\n if boxes.shape[0] == 0:\n return np.zeros((0, box_deltas.shape[1]))\n\n boxes = boxes.astype(np.float, copy=False)\n widths = boxes[:, 2] - boxes[:, 0] + 1.0\n heights = boxes[:, 3] - boxes[:, 1] + 1.0\n ctr_x = boxes[:, 0] + 0.5 * (widths - 1.0)\n ctr_y = boxes[:, 1] + 0.5 * (heights - 1.0)\n\n dx = box_deltas[:, 0::4]\n dy = box_deltas[:, 1::4]\n dw = box_deltas[:, 2::4]\n dh = box_deltas[:, 3::4]\n\n pred_ctr_x = dx * widths[:, np.newaxis] + ctr_x[:, np.newaxis]\n pred_ctr_y = dy * heights[:, np.newaxis] + ctr_y[:, np.newaxis]\n pred_w = np.exp(dw) * widths[:, np.newaxis]\n pred_h = np.exp(dh) * heights[:, np.newaxis]\n\n pred_boxes = np.zeros(box_deltas.shape)\n # x1\n pred_boxes[:, 0::4] = pred_ctr_x - 0.5 * (pred_w - 1.0)\n # y1\n pred_boxes[:, 1::4] = pred_ctr_y - 0.5 * (pred_h - 1.0)\n # x2\n pred_boxes[:, 2::4] = pred_ctr_x + 0.5 * (pred_w - 1.0)\n # y2\n pred_boxes[:, 3::4] = pred_ctr_y + 0.5 * (pred_h - 1.0)\n\n return pred_boxes\n\n\ndef iou_transform(ex_rois, gt_rois):\n \"\"\" return bbox targets, IoU loss uses gt_rois as gt \"\"\"\n assert ex_rois.shape[0] == gt_rois.shape[0], 'inconsistent rois number'\n return gt_rois\n\n\ndef iou_pred(boxes, box_deltas):\n \"\"\"\n Transform the set of class-agnostic boxes into class-specific boxes\n by applying the predicted offsets (box_deltas)\n :param boxes: !important [N 4]\n :param box_deltas: [N, 4 * num_classes]\n :return: [N 4 * num_classes]\n \"\"\"\n if boxes.shape[0] == 0:\n return np.zeros((0, box_deltas.shape[1]))\n\n boxes = boxes.astype(np.float, copy=False)\n x1 = boxes[:, 0]\n y1 = boxes[:, 1]\n x2 = boxes[:, 2]\n y2 = boxes[:, 3]\n\n dx1 = box_deltas[:, 0::4]\n dy1 = box_deltas[:, 1::4]\n dx2 = box_deltas[:, 2::4]\n dy2 = box_deltas[:, 3::4]\n\n pred_boxes = np.zeros(box_deltas.shape)\n # x1\n pred_boxes[:, 0::4] = dx1 + x1[:, np.newaxis]\n # y1\n pred_boxes[:, 1::4] = dy1 + y1[:, np.newaxis]\n # x2\n pred_boxes[:, 2::4] = dx2 + x2[:, np.newaxis]\n # y2\n pred_boxes[:, 3::4] = dy2 + y2[:, np.newaxis]\n\n return pred_boxes\n\n\n# define bbox_transform and bbox_pred\nbbox_transform = nonlinear_transform\nbbox_pred = nonlinear_pred\n" }, { "path": "lib/bbox/setup_linux.py", "content": "# --------------------------------------------------------\n# Deep Feature Flow\n# Copyright (c) 2017 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Modified by Yuwen Xiong\n# --------------------------------------------------------\n# Based on:\n# py-faster-rcnn\n# Copyright (c) 2015 Microsoft\n# Licence under The MIT License\n# https://github.com/rbgirshick/py-faster-rcnn\n# --------------------------------------------------------\n\n\nimport os\nfrom os.path import join as pjoin\nfrom setuptools import setup\nfrom distutils.extension import Extension\nfrom Cython.Distutils import build_ext\nimport numpy as np\n\n# Obtain the numpy include directory. This logic works across numpy versions.\ntry:\n numpy_include = np.get_include()\nexcept AttributeError:\n numpy_include = np.get_numpy_include()\n\n\ndef customize_compiler_for_nvcc(self):\n \"\"\"inject deep into distutils to customize how the dispatch\n to gcc/nvcc works.\n If you subclass UnixCCompiler, it's not trivial to get your subclass\n injected in, and still have the right customizations (i.e.\n distutils.sysconfig.customize_compiler) run on it. So instead of going\n the OO route, I have this. Note, it's kindof like a wierd functional\n subclassing going on.\"\"\"\n\n # tell the compiler it can processes .cu\n self.src_extensions.append('.cu')\n\n # save references to the default compiler_so and _comple methods\n default_compiler_so = self.compiler_so\n super = self._compile\n\n # now redefine the _compile method. This gets executed for each\n # object but distutils doesn't have the ability to change compilers\n # based on source extension: we add it.\n def _compile(obj, src, ext, cc_args, extra_postargs, pp_opts):\n if os.path.splitext(src)[1] == '.cu':\n # use the cuda for .cu files\n self.set_executable('compiler_so', CUDA['nvcc'])\n # use only a subset of the extra_postargs, which are 1-1 translated\n # from the extra_compile_args in the Extension class\n postargs = extra_postargs['nvcc']\n else:\n postargs = extra_postargs['gcc']\n\n super(obj, src, ext, cc_args, postargs, pp_opts)\n # reset the default compiler_so, which we might have changed for cuda\n self.compiler_so = default_compiler_so\n\n # inject our redefined _compile method into the class\n self._compile = _compile\n\n\n# run the customize_compiler\nclass custom_build_ext(build_ext):\n def build_extensions(self):\n customize_compiler_for_nvcc(self.compiler)\n build_ext.build_extensions(self)\n\n\next_modules = [\n Extension(\n \"bbox\",\n [\"bbox.pyx\"],\n extra_compile_args={'gcc': [\"-Wno-cpp\", \"-Wno-unused-function\"]},\n include_dirs=[numpy_include]\n ),\n]\n\nsetup(\n name='bbox_cython',\n ext_modules=ext_modules,\n # inject our custom trigger\n cmdclass={'build_ext': custom_build_ext},\n)\n" }, { "path": "lib/bbox/setup_windows.py", "content": "# --------------------------------------------------------\n# Deep Feature Flow\n# Copyright (c) 2017 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Modified by Yuwen Xiong\n# --------------------------------------------------------\n# Based on:\n# py-faster-rcnn\n# Copyright (c) 2015 Microsoft\n# Licence under The MIT License\n# https://github.com/rbgirshick/py-faster-rcnn\n# --------------------------------------------------------\n\nimport numpy as np\nimport os\nfrom os.path import join as pjoin\n#from distutils.core import setup\nfrom setuptools import setup\nfrom distutils.extension import Extension\nfrom Cython.Distutils import build_ext\nimport subprocess\n\n#change for windows, by MrX\nnvcc_bin = 'nvcc.exe'\nlib_dir = 'lib/x64'\n\nimport distutils.msvc9compiler\ndistutils.msvc9compiler.VERSION = 14.0\n\n# Obtain the numpy include directory. This logic works across numpy versions.\ntry:\n numpy_include = np.get_include()\nexcept AttributeError:\n numpy_include = np.get_numpy_include()\n\next_modules = [\n # unix _compile: obj, src, ext, cc_args, extra_postargs, pp_opts\n Extension(\n \"bbox\",\n sources=[\"bbox.pyx\"],\n extra_compile_args={},\n include_dirs = [numpy_include]\n ),\n]\n\nsetup(\n name='fast_rcnn',\n ext_modules=ext_modules,\n # inject our custom trigger\n cmdclass={'build_ext': build_ext},\n)\n" }, { "path": "lib/dataset/__init__.py", "content": "from imdb import IMDB\nfrom imagenet_vid import ImageNetVID\n" }, { "path": "lib/dataset/ds_utils.py", "content": "# --------------------------------------------------------\n# Deep Feature Flow\n# Copyright (c) 2017 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Written by Yuwen Xiong\n# --------------------------------------------------------\n\nimport numpy as np\n\n\ndef unique_boxes(boxes, scale=1.0):\n \"\"\" return indices of unique boxes \"\"\"\n v = np.array([1, 1e3, 1e6, 1e9])\n hashes = np.round(boxes * scale).dot(v)\n _, index = np.unique(hashes, return_index=True)\n return np.sort(index)\n\n\ndef filter_small_boxes(boxes, min_size):\n w = boxes[:, 2] - boxes[:, 0]\n h = boxes[:, 3] - boxes[:, 1]\n keep = np.where((w >= min_size) & (h > min_size))[0]\n return keep" }, { "path": "lib/dataset/imagenet_vid.py", "content": "# --------------------------------------------------------\n# Deep Feature Flow\n# Copyright (c) 2017 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Written by Xizhou Zhu\n# --------------------------------------------------------\n\n\"\"\"\nImageNet VID database\nThis class loads ground truth notations from standard ImageNet VID XML data formats\nand transform them into IMDB format. Selective search is used for proposals, see roidb\nfunction. Results are written as the ImageNet VID format. Evaluation is based on mAP\ncriterion.\n\"\"\"\n\nimport cPickle\nimport cv2\nimport os\nimport numpy as np\n\nfrom imdb import IMDB\nfrom imagenet_vid_eval import vid_eval\nfrom ds_utils import unique_boxes, filter_small_boxes\n\n\nclass ImageNetVID(IMDB):\n def __init__(self, image_set, root_path, dataset_path, result_path=None):\n \"\"\"\n fill basic information to initialize imdb\n \"\"\"\n det_vid = image_set.split('_')[0]\n super(ImageNetVID, self).__init__('ImageNetVID', image_set, root_path, dataset_path, result_path) # set self.name\n\n self.det_vid = det_vid\n self.root_path = root_path\n self.data_path = dataset_path\n\n self.classes = ['__background__', # always index 0\n 'airplane', 'antelope', 'bear', 'bicycle',\n 'bird', 'bus', 'car', 'cattle',\n 'dog', 'domestic_cat', 'elephant', 'fox',\n 'giant_panda', 'hamster', 'horse', 'lion',\n 'lizard', 'monkey', 'motorcycle', 'rabbit',\n 'red_panda', 'sheep', 'snake', 'squirrel',\n 'tiger', 'train', 'turtle', 'watercraft',\n 'whale', 'zebra']\n self.classes_map = ['__background__', # always index 0\n 'n02691156', 'n02419796', 'n02131653', 'n02834778',\n 'n01503061', 'n02924116', 'n02958343', 'n02402425',\n 'n02084071', 'n02121808', 'n02503517', 'n02118333',\n 'n02510455', 'n02342885', 'n02374451', 'n02129165',\n 'n01674464', 'n02484322', 'n03790512', 'n02324045',\n 'n02509815', 'n02411705', 'n01726692', 'n02355227',\n 'n02129604', 'n04468005', 'n01662784', 'n04530566',\n 'n02062744', 'n02391049']\n\n self.num_classes = len(self.classes)\n self.load_image_set_index()\n self.num_images = len(self.image_set_index)\n print 'num_images', self.num_images\n\n def load_image_set_index(self):\n \"\"\"\n find out which indexes correspond to given image set (train or val)\n :return:\n \"\"\"\n image_set_index_file = os.path.join(self.data_path, 'ImageSets', self.image_set + '.txt')\n assert os.path.exists(image_set_index_file), 'Path does not exist: {}'.format(image_set_index_file)\n with open(image_set_index_file) as f:\n lines = [x.strip().split(' ') for x in f.readlines()]\n if len(lines[0]) == 2:\n self.image_set_index = [x[0] for x in lines]\n self.frame_id = [int(x[1]) for x in lines]\n else:\n self.image_set_index = ['%s/%06d' % (x[0], int(x[2])) for x in lines]\n self.pattern = [x[0]+'/%06d' for x in lines]\n self.frame_id = [int(x[1]) for x in lines]\n self.frame_seg_id = [int(x[2]) for x in lines]\n self.frame_seg_len = [int(x[3]) for x in lines]\n # return image_set_index, frame_id\n\n def image_path_from_index(self, index):\n \"\"\"\n given image index, find out full path\n :param index: index of a specific image\n :return: full path of this image\n \"\"\"\n if self.det_vid == 'DET':\n image_file = os.path.join(self.data_path, 'Data', 'DET', index + '.JPEG')\n else:\n image_file = os.path.join(self.data_path, 'Data', 'VID', index + '.JPEG')\n\n # assert os.path.exists(image_file), 'Path does not exist: {}'.format(image_file)\n return image_file\n\n def gt_roidb(self):\n \"\"\"\n return ground truth image regions database\n :return: imdb[image_index]['boxes', 'gt_classes', 'gt_overlaps', 'flipped']\n \"\"\"\n cache_file = os.path.join(self.cache_path, self.name + '_gt_roidb.pkl')\n if os.path.exists(cache_file):\n with open(cache_file, 'rb') as fid:\n roidb = cPickle.load(fid)\n print '{} gt roidb loaded from {}'.format(self.name, cache_file)\n return roidb\n\n gt_roidb = [self.load_vid_annotation(index) for index in range(0, len(self.image_set_index))]\n with open(cache_file, 'wb') as fid:\n cPickle.dump(gt_roidb, fid, cPickle.HIGHEST_PROTOCOL)\n print 'wrote gt roidb to {}'.format(cache_file)\n\n return gt_roidb\n\n def load_vid_annotation(self, iindex):\n \"\"\"\n for a given index, load image and bounding boxes info from XML file\n :param index: index of a specific image\n :return: record['boxes', 'gt_classes', 'gt_overlaps', 'flipped']\n \"\"\"\n index = self.image_set_index[iindex]\n\n import xml.etree.ElementTree as ET\n roi_rec = dict()\n roi_rec['image'] = self.image_path_from_index(index)\n roi_rec['frame_id'] = self.frame_id[iindex]\n if hasattr(self,'frame_seg_id'):\n roi_rec['pattern'] = self.image_path_from_index(self.pattern[iindex])\n roi_rec['frame_seg_id'] = self.frame_seg_id[iindex]\n roi_rec['frame_seg_len'] = self.frame_seg_len[iindex]\n\n if self.det_vid == 'DET':\n filename = os.path.join(self.data_path, 'Annotations', 'DET', index + '.xml')\n else:\n filename = os.path.join(self.data_path, 'Annotations', 'VID', index + '.xml')\n\n tree = ET.parse(filename)\n size = tree.find('size')\n roi_rec['height'] = float(size.find('height').text)\n roi_rec['width'] = float(size.find('width').text)\n #im_size = cv2.imread(roi_rec['image'], cv2.IMREAD_COLOR|cv2.IMREAD_IGNORE_ORIENTATION).shape\n #assert im_size[0] == roi_rec['height'] and im_size[1] == roi_rec['width']\n\n objs = tree.findall('object')\n num_objs = len(objs)\n\n boxes = np.zeros((num_objs, 4), dtype=np.uint16)\n gt_classes = np.zeros((num_objs), dtype=np.int32)\n overlaps = np.zeros((num_objs, self.num_classes), dtype=np.float32)\n valid_objs = np.zeros((num_objs), dtype=np.bool)\n\n class_to_index = dict(zip(self.classes_map, range(self.num_classes)))\n # Load object bounding boxes into a data frame.\n for ix, obj in enumerate(objs):\n bbox = obj.find('bndbox')\n # Make pixel indexes 0-based\n x1 = np.maximum(float(bbox.find('xmin').text), 0)\n y1 = np.maximum(float(bbox.find('ymin').text), 0)\n x2 = np.minimum(float(bbox.find('xmax').text), roi_rec['width']-1)\n y2 = np.minimum(float(bbox.find('ymax').text), roi_rec['height']-1)\n if not class_to_index.has_key(obj.find('name').text):\n continue\n valid_objs[ix] = True\n cls = class_to_index[obj.find('name').text.lower().strip()]\n boxes[ix, :] = [x1, y1, x2, y2]\n gt_classes[ix] = cls\n overlaps[ix, cls] = 1.0\n\n boxes = boxes[valid_objs, :]\n gt_classes = gt_classes[valid_objs]\n overlaps = overlaps[valid_objs, :]\n\n assert (boxes[:, 2] >= boxes[:, 0]).all()\n\n roi_rec.update({'boxes': boxes,\n 'gt_classes': gt_classes,\n 'gt_overlaps': overlaps,\n 'max_classes': overlaps.argmax(axis=1),\n 'max_overlaps': overlaps.max(axis=1),\n 'flipped': False})\n return roi_rec\n\n###################################################################################################\n def evaluate_detections(self, detections):\n \"\"\"\n top level evaluations\n :param detections: result matrix, [bbox, confidence]\n :return: None\n \"\"\"\n # make all these folders for results\n result_dir = os.path.join(self.result_path, 'results')\n if not os.path.exists(result_dir):\n os.mkdir(result_dir)\n\n self.write_vid_results(detections)\n info = self.do_python_eval()\n return info\n\n def evaluate_detections_multiprocess(self, detections):\n \"\"\"\n top level evaluations\n :param detections: result matrix, [bbox, confidence]\n :return: None\n \"\"\"\n # make all these folders for results\n result_dir = os.path.join(self.result_path, 'results')\n if not os.path.exists(result_dir):\n os.mkdir(result_dir)\n\n self.write_vid_results_multiprocess(detections)\n info = self.do_python_eval_gen()\n return info\n\n def get_result_file_template(self):\n \"\"\"\n :return: a string template\n \"\"\"\n res_file_folder = os.path.join(self.result_path, 'results')\n filename = 'det_' + self.image_set + '_{:s}.txt'\n path = os.path.join(res_file_folder, filename)\n return path\n\n def write_vid_results(self, all_boxes):\n \"\"\"\n write results files in pascal devkit path\n :param all_boxes: boxes to be processed [bbox, confidence]\n :return: None\n \"\"\"\n print 'Writing {} ImageNetVID results file'.format('all')\n filename = self.get_result_file_template().format('all')\n with open(filename, 'wt') as f:\n for im_ind, index in enumerate(self.image_set_index):\n for cls_ind, cls in enumerate(self.classes):\n if cls == '__background__':\n continue\n dets = all_boxes[cls_ind][im_ind]\n if len(dets) == 0:\n continue\n # the imagenet expects 0-based indices\n for k in range(dets.shape[0]):\n f.write('{:d} {:d} {:.4f} {:.2f} {:.2f} {:.2f} {:.2f}\\n'.\n format(self.frame_id[im_ind], cls_ind, dets[k, -1],\n dets[k, 0], dets[k, 1], dets[k, 2], dets[k, 3]))\n\n def write_vid_results_multiprocess(self, detections):\n \"\"\"\n write results files in pascal devkit path\n :param all_boxes: boxes to be processed [bbox, confidence]\n :return: None\n \"\"\"\n print 'Writing {} ImageNetVID results file'.format('all')\n filename = self.get_result_file_template().format('all')\n with open(filename, 'wt') as f:\n for detection in detections:\n all_boxes = detection[0]\n frame_ids = detection[1]\n for im_ind in range(len(frame_ids)):\n for cls_ind, cls in enumerate(self.classes):\n if cls == '__background__':\n continue\n dets = all_boxes[cls_ind][im_ind]\n if len(dets) == 0:\n continue\n # the imagenet expects 0-based indices\n for k in range(dets.shape[0]):\n f.write('{:d} {:d} {:.4f} {:.2f} {:.2f} {:.2f} {:.2f}\\n'.\n format(frame_ids[im_ind], cls_ind, dets[k, -1],\n dets[k, 0], dets[k, 1], dets[k, 2], dets[k, 3]))\n \n def do_python_eval(self):\n \"\"\"\n python evaluation wrapper\n :return: info_str\n \"\"\"\n info_str = ''\n annopath = os.path.join(self.data_path, 'Annotations', '{0!s}.xml')\n imageset_file = os.path.join(self.data_path, 'ImageSets', self.image_set + '.txt')\n annocache = os.path.join(self.cache_path, self.name + '_annotations.pkl')\n\n filename = self.get_result_file_template().format('all')\n ap = vid_eval(filename, annopath, imageset_file, self.classes_map, annocache, ovthresh=0.5)\n for cls_ind, cls in enumerate(self.classes):\n if cls == '__background__':\n continue\n print('AP for {} = {:.4f}'.format(cls, ap[cls_ind-1]))\n info_str += 'AP for {} = {:.4f}\\n'.format(cls, ap[cls_ind-1])\n print('Mean AP@0.5 = {:.4f}'.format(np.mean(ap)))\n info_str += 'Mean AP@0.5 = {:.4f}\\n\\n'.format(np.mean(ap))\n return info_str\n\n def do_python_eval_gen(self):\n \"\"\"\n python evaluation wrapper\n :return: info_str\n \"\"\"\n info_str = ''\n annopath = os.path.join(self.data_path, 'Annotations', '{0!s}.xml')\n imageset_file = os.path.join(self.data_path, 'ImageSets', self.image_set + '_eval.txt')\n annocache = os.path.join(self.cache_path, self.name + '_annotations.pkl')\n\n with open(imageset_file, 'w') as f:\n for i in range(len(self.pattern)):\n for j in range(self.frame_seg_len[i]):\n f.write((self.pattern[i] % (self.frame_seg_id[i] + j)) + ' ' + str(self.frame_id[i] + j) + '\\n')\n\n filename = self.get_result_file_template().format('all')\n ap = vid_eval(filename, annopath, imageset_file, self.classes_map, annocache, ovthresh=0.5)\n for cls_ind, cls in enumerate(self.classes):\n if cls == '__background__':\n continue\n print('AP for {} = {:.4f}'.format(cls, ap[cls_ind-1]))\n info_str += 'AP for {} = {:.4f}\\n'.format(cls, ap[cls_ind-1])\n print('Mean AP@0.5 = {:.4f}'.format(np.mean(ap)))\n info_str += 'Mean AP@0.5 = {:.4f}\\n\\n'.format(np.mean(ap))\n return info_str\n" }, { "path": "lib/dataset/imagenet_vid_eval.py", "content": "# --------------------------------------------------------\n# Deep Feature Flow\n# Copyright (c) 2017 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Written by Xizhou Zhu\n# --------------------------------------------------------\n\n\"\"\"\ngiven a imagenet vid imdb, compute mAP\n\"\"\"\n\nimport numpy as np\nimport os\nimport cPickle\n\n\ndef parse_vid_rec(filename, classhash, img_ids, defaultIOUthr=0.5, pixelTolerance=10):\n \"\"\"\n parse imagenet vid record into a dictionary\n :param filename: xml file path\n :return: list of dict\n \"\"\"\n import xml.etree.ElementTree as ET\n tree = ET.parse(filename)\n objects = []\n for obj in tree.findall('object'):\n obj_dict = dict()\n obj_dict['label'] = classhash[obj.find('name').text]\n bbox = obj.find('bndbox')\n obj_dict['bbox'] = [float(bbox.find('xmin').text),\n float(bbox.find('ymin').text),\n float(bbox.find('xmax').text),\n float(bbox.find('ymax').text)]\n gt_w = obj_dict['bbox'][2] - obj_dict['bbox'][0] + 1\n gt_h = obj_dict['bbox'][3] - obj_dict['bbox'][1] + 1\n thr = (gt_w*gt_h)/((gt_w+pixelTolerance)*(gt_h+pixelTolerance))\n obj_dict['thr'] = np.min([thr, defaultIOUthr])\n objects.append(obj_dict)\n return {'bbox' : np.array([x['bbox'] for x in objects]),\n 'label': np.array([x['label'] for x in objects]),\n 'thr' : np.array([x['thr'] for x in objects]),\n 'img_ids': img_ids}\n\n\ndef vid_ap(rec, prec):\n \"\"\"\n average precision calculations\n [precision integrated to recall]\n :param rec: recall\n :param prec: precision\n :return: average precision\n \"\"\"\n\n # append sentinel values at both ends\n mrec = np.concatenate(([0.], rec, [1.]))\n mpre = np.concatenate(([0.], prec, [0.]))\n\n # compute precision integration ladder\n for i in range(mpre.size - 1, 0, -1):\n mpre[i - 1] = np.maximum(mpre[i - 1], mpre[i])\n\n # look for recall value changes\n i = np.where(mrec[1:] != mrec[:-1])[0]\n\n # sum (\\delta recall) * prec\n ap = np.sum((mrec[i + 1] - mrec[i]) * mpre[i + 1])\n return ap\n\n\ndef vid_eval(detpath, annopath, imageset_file, classname_map, annocache, ovthresh=0.5):\n \"\"\"\n imagenet vid evaluation\n :param detpath: detection results detpath.format(classname)\n :param annopath: annotations annopath.format(classname)\n :param imageset_file: text file containing list of images\n :param annocache: caching annotations\n :param ovthresh: overlap threshold\n :return: rec, prec, ap\n \"\"\"\n with open(imageset_file, 'r') as f:\n lines = [x.strip().split(' ') for x in f.readlines()]\n img_basenames = [x[0] for x in lines]\n gt_img_ids = [int(x[1]) for x in lines]\n classhash = dict(zip(classname_map, range(0,len(classname_map))))\n\n # load annotations from cache\n if not os.path.isfile(annocache):\n recs = []\n for ind, image_filename in enumerate(img_basenames):\n recs.append(parse_vid_rec(annopath.format('VID/' + image_filename), classhash, gt_img_ids[ind]))\n if ind % 100 == 0:\n print 'reading annotations for {:d}/{:d}'.format(ind + 1, len(img_basenames))\n print 'saving annotations cache to {:s}'.format(annocache)\n with open(annocache, 'wb') as f:\n cPickle.dump(recs, f, protocol=cPickle.HIGHEST_PROTOCOL)\n else:\n with open(annocache, 'rb') as f:\n recs = cPickle.load(f)\n\n # extract objects in :param classname:\n npos = np.zeros(len(classname_map))\n for rec in recs:\n rec_labels = rec['label']\n for x in rec_labels:\n npos[x] += 1\n\n # read detections\n with open(detpath, 'r') as f:\n lines = f.readlines()\n\n splitlines = [x.strip().split(' ') for x in lines]\n img_ids = np.array([int(x[0]) for x in splitlines])\n obj_labels = np.array([int(x[1]) for x in splitlines])\n obj_confs = np.array([float(x[2]) for x in splitlines])\n obj_bboxes = np.array([[float(z) for z in x[3:]] for x in splitlines])\n\n # sort by confidence\n if obj_bboxes.shape[0] > 0:\n sorted_inds = np.argsort(img_ids)\n img_ids = img_ids[sorted_inds]\n obj_labels = obj_labels[sorted_inds]\n obj_confs = obj_confs[sorted_inds]\n obj_bboxes = obj_bboxes[sorted_inds, :]\n\n num_imgs = max(max(gt_img_ids),max(img_ids)) + 1\n obj_labels_cell = [None] * num_imgs\n obj_confs_cell = [None] * num_imgs\n obj_bboxes_cell = [None] * num_imgs\n start_i = 0\n id = img_ids[0]\n for i in range(0, len(img_ids)):\n if i == len(img_ids)-1 or img_ids[i+1] != id:\n conf = obj_confs[start_i:i+1]\n label = obj_labels[start_i:i+1]\n bbox = obj_bboxes[start_i:i+1, :]\n sorted_inds = np.argsort(-conf)\n\n obj_labels_cell[id] = label[sorted_inds]\n obj_confs_cell[id] = conf[sorted_inds]\n obj_bboxes_cell[id] = bbox[sorted_inds, :]\n if i < len(img_ids)-1:\n id = img_ids[i+1]\n start_i = i+1\n\n\n # go down detections and mark true positives and false positives\n tp_cell = [None] * num_imgs\n fp_cell = [None] * num_imgs\n\n for rec in recs:\n id = rec['img_ids']\n gt_labels = rec['label']\n gt_bboxes = rec['bbox']\n gt_thr = rec['thr']\n num_gt_obj = len(gt_labels)\n gt_detected = np.zeros(num_gt_obj)\n\n labels = obj_labels_cell[id]\n bboxes = obj_bboxes_cell[id]\n\n num_obj = 0 if labels is None else len(labels)\n tp = np.zeros(num_obj)\n fp = np.zeros(num_obj)\n\n for j in range(0,num_obj):\n bb = bboxes[j, :]\n ovmax = -1\n kmax = -1\n for k in range(0,num_gt_obj):\n if labels[j] != gt_labels[k]:\n continue\n if gt_detected[k] > 0:\n continue\n bbgt = gt_bboxes[k, :]\n bi=[np.max((bb[0],bbgt[0])), np.max((bb[1],bbgt[1])), np.min((bb[2],bbgt[2])), np.min((bb[3],bbgt[3]))]\n iw=bi[2]-bi[0]+1\n ih=bi[3]-bi[1]+1\n if iw>0 and ih>0: \n # compute overlap as area of intersection / area of union\n ua = (bb[2] - bb[0] + 1.) * (bb[3] - bb[1] + 1.) + \\\n (bbgt[2] - bbgt[0] + 1.) * \\\n (bbgt[3] - bbgt[1] + 1.) - iw*ih\n ov=iw*ih/ua\n # makes sure that this object is detected according\n # to its individual threshold\n if ov >= gt_thr[k] and ov > ovmax:\n ovmax=ov\n kmax=k\n if kmax >= 0:\n tp[j] = 1\n gt_detected[kmax] = 1\n else:\n fp[j] = 1\n\n tp_cell[id] = tp\n fp_cell[id] = fp\n\n tp_all = np.concatenate([x for x in np.array(tp_cell)[gt_img_ids] if x is not None])\n fp_all = np.concatenate([x for x in np.array(fp_cell)[gt_img_ids] if x is not None])\n obj_labels = np.concatenate([x for x in np.array(obj_labels_cell)[gt_img_ids] if x is not None])\n confs = np.concatenate([x for x in np.array(obj_confs_cell)[gt_img_ids] if x is not None])\n\n sorted_inds = np.argsort(-confs)\n tp_all = tp_all[sorted_inds]\n fp_all = fp_all[sorted_inds]\n obj_labels = obj_labels[sorted_inds]\n\n ap = np.zeros(len(classname_map))\n for c in range(1, len(classname_map)):\n # compute precision recall\n fp = np.cumsum(fp_all[obj_labels == c])\n tp = np.cumsum(tp_all[obj_labels == c])\n rec = tp / float(npos[c])\n # avoid division by zero in case first detection matches a difficult ground ruth\n prec = tp / np.maximum(tp + fp, np.finfo(np.float64).eps)\n ap[c] = vid_ap(rec, prec)\n ap = ap[1:]\n return ap\n" }, { "path": "lib/dataset/imdb.py", "content": "# --------------------------------------------------------\n# Deep Feature Flow\n# Copyright (c) 2017 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Written by Yuwen Xiong\n# --------------------------------------------------------\n\n\"\"\"\nGeneral image database\nAn image database creates a list of relative image path called image_set_index and\ntransform index to absolute image path. As to training, it is necessary that ground\ntruth and proposals are mixed together for training.\nroidb\nbasic format [image_index]\n['image', 'height', 'width', 'flipped',\n'boxes', 'gt_classes', 'gt_overlaps', 'max_classes', 'max_overlaps', 'bbox_targets']\n\"\"\"\n\nimport os\nimport cPickle\nimport numpy as np\nfrom PIL import Image\nfrom bbox.bbox_transform import bbox_overlaps\nfrom multiprocessing import Pool, cpu_count\n\ndef get_flipped_entry_outclass_wrapper(IMDB_instance, seg_rec):\n return IMDB_instance.get_flipped_entry(seg_rec)\n\nclass IMDB(object):\n def __init__(self, name, image_set, root_path, dataset_path, result_path=None):\n \"\"\"\n basic information about an image database\n :param name: name of image database will be used for any output\n :param root_path: root path store cache and proposal data\n :param dataset_path: dataset path store images and image lists\n \"\"\"\n self.name = name + '_' + image_set\n self.image_set = image_set\n self.root_path = root_path\n self.data_path = dataset_path\n self._result_path = result_path\n\n # abstract attributes\n self.classes = []\n self.num_classes = 0\n self.image_set_index = []\n self.num_images = 0\n\n self.config = {}\n\n def image_path_from_index(self, index):\n raise NotImplementedError\n\n def gt_roidb(self):\n raise NotImplementedError\n\n def evaluate_detections(self, detections):\n raise NotImplementedError\n\n def evaluate_segmentations(self, segmentations):\n raise NotImplementedError\n\n @property\n def cache_path(self):\n \"\"\"\n make a directory to store all caches\n :return: cache path\n \"\"\"\n cache_path = os.path.join(self.root_path, 'cache')\n if not os.path.exists(cache_path):\n os.mkdir(cache_path)\n return cache_path\n\n @property\n def result_path(self):\n if self._result_path and os.path.exists(self._result_path):\n return self._result_path\n else:\n return self.cache_path\n\n def image_path_at(self, index):\n \"\"\"\n access image at index in image database\n :param index: image index in image database\n :return: image path\n \"\"\"\n return self.image_path_from_index(self.image_set_index[index])\n\n def load_rpn_data(self, full=False):\n if full:\n rpn_file = os.path.join(self.result_path, 'rpn_data', self.name + '_full_rpn.pkl')\n else:\n rpn_file = os.path.join(self.result_path, 'rpn_data', self.name + '_rpn.pkl')\n print 'loading {}'.format(rpn_file)\n assert os.path.exists(rpn_file), 'rpn data not found at {}'.format(rpn_file)\n with open(rpn_file, 'rb') as f:\n box_list = cPickle.load(f)\n return box_list\n\n def load_rpn_roidb(self, gt_roidb):\n \"\"\"\n turn rpn detection boxes into roidb\n :param gt_roidb: [image_index]['boxes', 'gt_classes', 'gt_overlaps', 'flipped']\n :return: roidb: [image_index]['boxes', 'gt_classes', 'gt_overlaps', 'flipped']\n \"\"\"\n box_list = self.load_rpn_data()\n return self.create_roidb_from_box_list(box_list, gt_roidb)\n\n def rpn_roidb(self, gt_roidb, append_gt=False):\n \"\"\"\n get rpn roidb and ground truth roidb\n :param gt_roidb: ground truth roidb\n :param append_gt: append ground truth\n :return: roidb of rpn\n \"\"\"\n if append_gt:\n print 'appending ground truth annotations'\n rpn_roidb = self.load_rpn_roidb(gt_roidb)\n roidb = IMDB.merge_roidbs(gt_roidb, rpn_roidb)\n else:\n roidb = self.load_rpn_roidb(gt_roidb)\n return roidb\n\n def create_roidb_from_box_list(self, box_list, gt_roidb):\n \"\"\"\n given ground truth, prepare roidb\n :param box_list: [image_index] ndarray of [box_index][x1, x2, y1, y2]\n :param gt_roidb: [image_index]['boxes', 'gt_classes', 'gt_overlaps', 'flipped']\n :return: roidb: [image_index]['boxes', 'gt_classes', 'gt_overlaps', 'flipped']\n \"\"\"\n assert len(box_list) == self.num_images, 'number of boxes matrix must match number of images'\n roidb = []\n for i in range(self.num_images):\n roi_rec = dict()\n roi_rec['image'] = gt_roidb[i]['image']\n roi_rec['height'] = gt_roidb[i]['height']\n roi_rec['width'] = gt_roidb[i]['width']\n\n boxes = box_list[i]\n if boxes.shape[1] == 5:\n boxes = boxes[:, :4]\n num_boxes = boxes.shape[0]\n overlaps = np.zeros((num_boxes, self.num_classes), dtype=np.float32)\n if gt_roidb is not None and gt_roidb[i]['boxes'].size > 0:\n gt_boxes = gt_roidb[i]['boxes']\n gt_classes = gt_roidb[i]['gt_classes']\n # n boxes and k gt_boxes => n * k overlap\n gt_overlaps = bbox_overlaps(boxes.astype(np.float), gt_boxes.astype(np.float))\n # for each box in n boxes, select only maximum overlap (must be greater than zero)\n argmaxes = gt_overlaps.argmax(axis=1)\n maxes = gt_overlaps.max(axis=1)\n I = np.where(maxes > 0)[0]\n overlaps[I, gt_classes[argmaxes[I]]] = maxes[I]\n\n roi_rec.update({'boxes': boxes,\n 'gt_classes': np.zeros((num_boxes,), dtype=np.int32),\n 'gt_overlaps': overlaps,\n 'max_classes': overlaps.argmax(axis=1),\n 'max_overlaps': overlaps.max(axis=1),\n 'flipped': False})\n\n # background roi => background class\n zero_indexes = np.where(roi_rec['max_overlaps'] == 0)[0]\n assert all(roi_rec['max_classes'][zero_indexes] == 0)\n # foreground roi => foreground class\n nonzero_indexes = np.where(roi_rec['max_overlaps'] > 0)[0]\n assert all(roi_rec['max_classes'][nonzero_indexes] != 0)\n\n roidb.append(roi_rec)\n\n return roidb\n\n def get_flipped_entry(self, seg_rec):\n return {'image': self.flip_and_save(seg_rec['image']),\n 'seg_cls_path': self.flip_and_save(seg_rec['seg_cls_path']),\n 'height': seg_rec['height'],\n 'width': seg_rec['width'],\n 'flipped': True}\n\n def append_flipped_images_for_segmentation(self, segdb):\n \"\"\"\n append flipped images to an roidb\n flip boxes coordinates, images will be actually flipped when loading into network\n :param segdb: [image_index]['seg_cls_path', 'flipped']\n :return: segdb: [image_index]['seg_cls_path', 'flipped']\n \"\"\"\n print 'append flipped images to segdb'\n assert self.num_images == len(segdb)\n pool = Pool(processes=1)\n pool_result = []\n for i in range(self.num_images):\n seg_rec = segdb[i]\n pool_result.append(pool.apply_async(get_flipped_entry_outclass_wrapper, args=(self, seg_rec, )))\n #self.get_flipped_entry(seg_rec, segdb_flip, i)\n pool.close()\n pool.join()\n segdb_flip = [res_instance.get() for res_instance in pool_result]\n segdb += segdb_flip\n self.image_set_index *= 2\n return segdb\n\n def append_flipped_images(self, roidb):\n \"\"\"\n append flipped images to an roidb\n flip boxes coordinates, images will be actually flipped when loading into network\n :param roidb: [image_index]['boxes', 'gt_classes', 'gt_overlaps', 'flipped']\n :return: roidb: [image_index]['boxes', 'gt_classes', 'gt_overlaps', 'flipped']\n \"\"\"\n print 'append flipped images to roidb'\n assert self.num_images == len(roidb)\n for i in range(self.num_images):\n roi_rec = roidb[i]\n boxes = roi_rec['boxes'].copy()\n oldx1 = boxes[:, 0].copy()\n oldx2 = boxes[:, 2].copy()\n boxes[:, 0] = roi_rec['width'] - oldx2 - 1\n boxes[:, 2] = roi_rec['width'] - oldx1 - 1\n assert (boxes[:, 2] >= boxes[:, 0]).all()\n entry = roi_rec.copy()\n entry['boxes'] = boxes\n entry['flipped'] = True\n\n # if roidb has mask\n if 'cache_seg_inst' in roi_rec:\n [filename, extension] = os.path.splitext(roi_rec['cache_seg_inst'])\n entry['cache_seg_inst'] = os.path.join(filename + '_flip' + extension)\n\n roidb.append(entry)\n\n self.image_set_index *= 2\n return roidb\n\n def flip_and_save(self, image_path):\n \"\"\"\n flip the image by the path and save the flipped image with suffix 'flip'\n :param path: the path of specific image\n :return: the path of saved image\n \"\"\"\n [image_name, image_ext] = os.path.splitext(os.path.basename(image_path))\n image_dir = os.path.dirname(image_path)\n saved_image_path = os.path.join(image_dir, image_name + '_flip' + image_ext)\n try:\n flipped_image = Image.open(saved_image_path)\n except:\n flipped_image = Image.open(image_path)\n flipped_image = flipped_image.transpose(Image.FLIP_LEFT_RIGHT)\n flipped_image.save(saved_image_path, 'png')\n return saved_image_path\n\n def evaluate_recall(self, roidb, candidate_boxes=None, thresholds=None):\n \"\"\"\n evaluate detection proposal recall metrics\n record max overlap value for each gt box; return vector of overlap values\n :param roidb: used to evaluate\n :param candidate_boxes: if not given, use roidb's non-gt boxes\n :param thresholds: array-like recall threshold\n :return: None\n ar: average recall, recalls: vector recalls at each IoU overlap threshold\n thresholds: vector of IoU overlap threshold, gt_overlaps: vector of all ground-truth overlaps\n \"\"\"\n all_log_info = ''\n area_names = ['all', '0-25', '25-50', '50-100',\n '100-200', '200-300', '300-inf']\n area_ranges = [[0**2, 1e5**2], [0**2, 25**2], [25**2, 50**2], [50**2, 100**2],\n [100**2, 200**2], [200**2, 300**2], [300**2, 1e5**2]]\n area_counts = []\n for area_name, area_range in zip(area_names[1:], area_ranges[1:]):\n area_count = 0\n for i in range(self.num_images):\n if candidate_boxes is None:\n # default is use the non-gt boxes from roidb\n non_gt_inds = np.where(roidb[i]['gt_classes'] == 0)[0]\n boxes = roidb[i]['boxes'][non_gt_inds, :]\n else:\n boxes = candidate_boxes[i]\n boxes_areas = (boxes[:, 2] - boxes[:, 0] + 1) * (boxes[:, 3] - boxes[:, 1] + 1)\n valid_range_inds = np.where((boxes_areas >= area_range[0]) & (boxes_areas < area_range[1]))[0]\n area_count += len(valid_range_inds)\n area_counts.append(area_count)\n total_counts = float(sum(area_counts))\n for area_name, area_count in zip(area_names[1:], area_counts):\n log_info = 'percentage of {} {}'.format(area_name, area_count / total_counts)\n print log_info\n all_log_info += log_info\n log_info = 'average number of proposal {}'.format(total_counts / self.num_images)\n print log_info\n all_log_info += log_info\n for area_name, area_range in zip(area_names, area_ranges):\n gt_overlaps = np.zeros(0)\n num_pos = 0\n for i in range(self.num_images):\n # check for max_overlaps == 1 avoids including crowd annotations\n max_gt_overlaps = roidb[i]['gt_overlaps'].max(axis=1)\n gt_inds = np.where((roidb[i]['gt_classes'] > 0) & (max_gt_overlaps == 1))[0]\n gt_boxes = roidb[i]['boxes'][gt_inds, :]\n gt_areas = (gt_boxes[:, 2] - gt_boxes[:, 0] + 1) * (gt_boxes[:, 3] - gt_boxes[:, 1] + 1)\n valid_gt_inds = np.where((gt_areas >= area_range[0]) & (gt_areas < area_range[1]))[0]\n gt_boxes = gt_boxes[valid_gt_inds, :]\n num_pos += len(valid_gt_inds)\n\n if candidate_boxes is None:\n # default is use the non-gt boxes from roidb\n non_gt_inds = np.where(roidb[i]['gt_classes'] == 0)[0]\n boxes = roidb[i]['boxes'][non_gt_inds, :]\n else:\n boxes = candidate_boxes[i]\n if boxes.shape[0] == 0:\n continue\n\n overlaps = bbox_overlaps(boxes.astype(np.float), gt_boxes.astype(np.float))\n\n _gt_overlaps = np.zeros((gt_boxes.shape[0]))\n # choose whatever is smaller to iterate\n rounds = min(boxes.shape[0], gt_boxes.shape[0])\n for j in range(rounds):\n # find which proposal maximally covers each gt box\n argmax_overlaps = overlaps.argmax(axis=0)\n # get the IoU amount of coverage for each gt box\n max_overlaps = overlaps.max(axis=0)\n # find which gt box is covered by most IoU\n gt_ind = max_overlaps.argmax()\n gt_ovr = max_overlaps.max()\n assert (gt_ovr >= 0), '%s\\n%s\\n%s' % (boxes, gt_boxes, overlaps)\n # find the proposal box that covers the best covered gt box\n box_ind = argmax_overlaps[gt_ind]\n # record the IoU coverage of this gt box\n _gt_overlaps[j] = overlaps[box_ind, gt_ind]\n assert (_gt_overlaps[j] == gt_ovr)\n # mark the proposal box and the gt box as used\n overlaps[box_ind, :] = -1\n overlaps[:, gt_ind] = -1\n # append recorded IoU coverage level\n gt_overlaps = np.hstack((gt_overlaps, _gt_overlaps))\n\n gt_overlaps = np.sort(gt_overlaps)\n if thresholds is None:\n step = 0.05\n thresholds = np.arange(0.5, 0.95 + 1e-5, step)\n recalls = np.zeros_like(thresholds)\n\n # compute recall for each IoU threshold\n for i, t in enumerate(thresholds):\n recalls[i] = (gt_overlaps >= t).sum() / float(num_pos)\n ar = recalls.mean()\n\n # print results\n log_info = 'average recall for {}: {:.3f}'.format(area_name, ar)\n print log_info\n all_log_info += log_info\n for threshold, recall in zip(thresholds, recalls):\n log_info = 'recall @{:.2f}: {:.3f}'.format(threshold, recall)\n print log_info\n all_log_info += log_info\n\n return all_log_info\n\n @staticmethod\n def merge_roidbs(a, b):\n \"\"\"\n merge roidbs into one\n :param a: roidb to be merged into\n :param b: roidb to be merged\n :return: merged imdb\n \"\"\"\n assert len(a) == len(b)\n for i in range(len(a)):\n a[i]['boxes'] = np.vstack((a[i]['boxes'], b[i]['boxes']))\n a[i]['gt_classes'] = np.hstack((a[i]['gt_classes'], b[i]['gt_classes']))\n a[i]['gt_overlaps'] = np.vstack((a[i]['gt_overlaps'], b[i]['gt_overlaps']))\n a[i]['max_classes'] = np.hstack((a[i]['max_classes'], b[i]['max_classes']))\n a[i]['max_overlaps'] = np.hstack((a[i]['max_overlaps'], b[i]['max_overlaps']))\n return a\n" }, { "path": "lib/nms/__init__.py", "content": "" }, { "path": "lib/nms/cpu_nms.pyx", "content": "# --------------------------------------------------------\n# Deep Feature Flow\n# Copyright (c) 2017 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Written by Yuwen Xiong\n# --------------------------------------------------------\n# Based on:\n# py-faster-rcnn\n# Copyright (c) 2015 Microsoft\n# Licence under The MIT License\n# https://github.com/rbgirshick/py-faster-rcnn\n# --------------------------------------------------------\n\nimport numpy as np\ncimport numpy as np\n\ncdef inline np.float32_t max(np.float32_t a, np.float32_t b):\n return a if a >= b else b\n\ncdef inline np.float32_t min(np.float32_t a, np.float32_t b):\n return a if a <= b else b\n\ndef cpu_nms(np.ndarray[np.float32_t, ndim=2] dets, np.float thresh):\n cdef np.ndarray[np.float32_t, ndim=1] x1 = dets[:, 0]\n cdef np.ndarray[np.float32_t, ndim=1] y1 = dets[:, 1]\n cdef np.ndarray[np.float32_t, ndim=1] x2 = dets[:, 2]\n cdef np.ndarray[np.float32_t, ndim=1] y2 = dets[:, 3]\n cdef np.ndarray[np.float32_t, ndim=1] scores = dets[:, 4]\n\n cdef np.ndarray[np.float32_t, ndim=1] areas = (x2 - x1 + 1) * (y2 - y1 + 1)\n cdef np.ndarray[np.int_t, ndim=1] order = scores.argsort()[::-1].astype('i')\n\n cdef int ndets = dets.shape[0]\n cdef np.ndarray[np.int_t, ndim=1] suppressed = \\\n np.zeros((ndets), dtype=np.int)\n\n # nominal indices\n cdef int _i, _j\n # sorted indices\n cdef int i, j\n # temp variables for box i's (the box currently under consideration)\n cdef np.float32_t ix1, iy1, ix2, iy2, iarea\n # variables for computing overlap with box j (lower scoring box)\n cdef np.float32_t xx1, yy1, xx2, yy2\n cdef np.float32_t w, h\n cdef np.float32_t inter, ovr\n\n keep = []\n for _i in range(ndets):\n i = order[_i]\n if suppressed[i] == 1:\n continue\n keep.append(i)\n ix1 = x1[i]\n iy1 = y1[i]\n ix2 = x2[i]\n iy2 = y2[i]\n iarea = areas[i]\n for _j in range(_i + 1, ndets):\n j = order[_j]\n if suppressed[j] == 1:\n continue\n xx1 = max(ix1, x1[j])\n yy1 = max(iy1, y1[j])\n xx2 = min(ix2, x2[j])\n yy2 = min(iy2, y2[j])\n w = max(0.0, xx2 - xx1 + 1)\n h = max(0.0, yy2 - yy1 + 1)\n inter = w * h\n ovr = inter / (iarea + areas[j] - inter)\n if ovr >= thresh:\n suppressed[j] = 1\n\n return keep\n" }, { "path": "lib/nms/gpu_nms.cu", "content": "// ------------------------------------------------------------------\n// Deep Feature Flow\n// Copyright (c) 2017 Microsoft\n// Licensed under The MIT License\n// Written by Yuwen Xiong\n// ------------------------------------------------------------------\n// Based on:\n// Faster R-CNN\n// Copyright (c) 2015 Microsoft\n// Licensed under The MIT License\n// https://github.com/shaoqingren/faster_rcnn\n// ------------------------------------------------------------------\n\n//#include \"gpu_nms.hpp\"\n#include \n#include \n\n\n#define CUDA_CHECK(condition) \\\n /* Code block avoids redefinition of cudaError_t error */ \\\n do { \\\n cudaError_t error = condition; \\\n if (error != cudaSuccess) { \\\n std::cout << cudaGetErrorString(error) << std::endl; \\\n } \\\n } while (0)\n\n#define DIVUP(m,n) ((m) / (n) + ((m) % (n) > 0))\nint const threadsPerBlock = sizeof(unsigned long long) * 8;\n\n__device__ inline float devIoU(float const * const a, float const * const b) {\n float left = max(a[0], b[0]), right = min(a[2], b[2]);\n float top = max(a[1], b[1]), bottom = min(a[3], b[3]);\n float width = max(right - left + 1, 0.f), height = max(bottom - top + 1, 0.f);\n float interS = width * height;\n float Sa = (a[2] - a[0] + 1) * (a[3] - a[1] + 1);\n float Sb = (b[2] - b[0] + 1) * (b[3] - b[1] + 1);\n return interS / (Sa + Sb - interS);\n}\n\n__global__ void nms_kernel(const int n_boxes, const float nms_overlap_thresh,\n const float *dev_boxes, unsigned long long *dev_mask) {\n const int row_start = blockIdx.y;\n const int col_start = blockIdx.x;\n\n // if (row_start > col_start) return;\n\n const int row_size =\n min(n_boxes - row_start * threadsPerBlock, threadsPerBlock);\n const int col_size =\n min(n_boxes - col_start * threadsPerBlock, threadsPerBlock);\n\n __shared__ float block_boxes[threadsPerBlock * 5];\n if (threadIdx.x < col_size) {\n block_boxes[threadIdx.x * 5 + 0] =\n dev_boxes[(threadsPerBlock * col_start + threadIdx.x) * 5 + 0];\n block_boxes[threadIdx.x * 5 + 1] =\n dev_boxes[(threadsPerBlock * col_start + threadIdx.x) * 5 + 1];\n block_boxes[threadIdx.x * 5 + 2] =\n dev_boxes[(threadsPerBlock * col_start + threadIdx.x) * 5 + 2];\n block_boxes[threadIdx.x * 5 + 3] =\n dev_boxes[(threadsPerBlock * col_start + threadIdx.x) * 5 + 3];\n block_boxes[threadIdx.x * 5 + 4] =\n dev_boxes[(threadsPerBlock * col_start + threadIdx.x) * 5 + 4];\n }\n __syncthreads();\n\n if (threadIdx.x < row_size) {\n const int cur_box_idx = threadsPerBlock * row_start + threadIdx.x;\n const float *cur_box = dev_boxes + cur_box_idx * 5;\n int i = 0;\n unsigned long long t = 0;\n int start = 0;\n if (row_start == col_start) {\n start = threadIdx.x + 1;\n }\n for (i = start; i < col_size; i++) {\n if (devIoU(cur_box, block_boxes + i * 5) > nms_overlap_thresh) {\n t |= 1ULL << i;\n }\n }\n const int col_blocks = DIVUP(n_boxes, threadsPerBlock);\n dev_mask[cur_box_idx * col_blocks + col_start] = t;\n }\n}\n\nvoid _set_device(int device_id) {\n int current_device;\n CUDA_CHECK(cudaGetDevice(¤t_device));\n if (current_device == device_id) {\n return;\n }\n // The call to cudaSetDevice must come before any calls to Get, which\n // may perform initialization using the GPU.\n CUDA_CHECK(cudaSetDevice(device_id));\n}\n\nvoid _nms(long* keep_out, int* num_out, const float* boxes_host, int boxes_num,\n int boxes_dim, float nms_overlap_thresh, int device_id) {\n _set_device(device_id);\n\n float* boxes_dev = NULL;\n unsigned long long* mask_dev = NULL;\n\n const int col_blocks = DIVUP(boxes_num, threadsPerBlock);\n\n CUDA_CHECK(cudaMalloc(&boxes_dev,\n boxes_num * boxes_dim * sizeof(float)));\n CUDA_CHECK(cudaMemcpy(boxes_dev,\n boxes_host,\n boxes_num * boxes_dim * sizeof(float),\n cudaMemcpyHostToDevice));\n\n CUDA_CHECK(cudaMalloc(&mask_dev,\n boxes_num * col_blocks * sizeof(unsigned long long)));\n\n dim3 blocks(DIVUP(boxes_num, threadsPerBlock),\n DIVUP(boxes_num, threadsPerBlock));\n dim3 threads(threadsPerBlock);\n nms_kernel<<>>(boxes_num,\n nms_overlap_thresh,\n boxes_dev,\n mask_dev);\n\n std::vector mask_host(boxes_num * col_blocks);\n CUDA_CHECK(cudaMemcpy(&mask_host[0],\n mask_dev,\n sizeof(unsigned long long) * boxes_num * col_blocks,\n cudaMemcpyDeviceToHost));\n\n std::vector remv(col_blocks);\n memset(&remv[0], 0, sizeof(unsigned long long) * col_blocks);\n\n int num_to_keep = 0;\n for (int i = 0; i < boxes_num; i++) {\n int nblock = i / threadsPerBlock;\n int inblock = i % threadsPerBlock;\n\n if (!(remv[nblock] & (1ULL << inblock))) {\n keep_out[num_to_keep++] = i;\n unsigned long long *p = &mask_host[0] + i * col_blocks;\n for (int j = nblock; j < col_blocks; j++) {\n remv[j] |= p[j];\n }\n }\n }\n *num_out = num_to_keep;\n\n CUDA_CHECK(cudaFree(boxes_dev));\n CUDA_CHECK(cudaFree(mask_dev));\n}\n\n\n\n\n\n\n\n\n\n\n/* Generated by Cython 0.24 */\n\n#define PY_SSIZE_T_CLEAN\n#include \"Python.h\"\n#ifndef Py_PYTHON_H\n #error Python headers needed to compile C extensions, please install development version of Python.\n#elif PY_VERSION_HEX < 0x02060000 || (0x03000000 <= PY_VERSION_HEX && PY_VERSION_HEX < 0x03020000)\n #error Cython requires Python 2.6+ or Python 3.2+.\n#else\n#define CYTHON_ABI \"0_24\"\n#include \n#ifndef offsetof\n #define offsetof(type, member) ( (size_t) & ((type*)0) -> member )\n#endif\n#if !defined(WIN32) && !defined(MS_WINDOWS)\n #ifndef __stdcall\n #define __stdcall\n #endif\n #ifndef __cdecl\n #define __cdecl\n #endif\n #ifndef __fastcall\n #define __fastcall\n #endif\n#endif\n#ifndef DL_IMPORT\n #define DL_IMPORT(t) t\n#endif\n#ifndef DL_EXPORT\n #define DL_EXPORT(t) t\n#endif\n#ifndef PY_LONG_LONG\n #define PY_LONG_LONG LONG_LONG\n#endif\n#ifndef Py_HUGE_VAL\n #define Py_HUGE_VAL HUGE_VAL\n#endif\n#ifdef PYPY_VERSION\n #define CYTHON_COMPILING_IN_PYPY 1\n #define CYTHON_COMPILING_IN_CPYTHON 0\n#else\n #define CYTHON_COMPILING_IN_PYPY 0\n #define CYTHON_COMPILING_IN_CPYTHON 1\n#endif\n#if !defined(CYTHON_USE_PYLONG_INTERNALS) && CYTHON_COMPILING_IN_CPYTHON && PY_VERSION_HEX >= 0x02070000\n #define CYTHON_USE_PYLONG_INTERNALS 1\n#endif\n#if CYTHON_USE_PYLONG_INTERNALS\n #include \"longintrepr.h\"\n #undef SHIFT\n #undef BASE\n #undef MASK\n#endif\n#if CYTHON_COMPILING_IN_PYPY && PY_VERSION_HEX < 0x02070600 && !defined(Py_OptimizeFlag)\n #define Py_OptimizeFlag 0\n#endif\n#define __PYX_BUILD_PY_SSIZE_T \"n\"\n#define CYTHON_FORMAT_SSIZE_T \"z\"\n#if PY_MAJOR_VERSION < 3\n #define __Pyx_BUILTIN_MODULE_NAME \"__builtin__\"\n #define __Pyx_PyCode_New(a, k, l, s, f, code, c, n, v, fv, cell, fn, name, fline, lnos)\\\n PyCode_New(a+k, l, s, f, code, c, n, v, fv, cell, fn, name, fline, lnos)\n #define __Pyx_DefaultClassType PyClass_Type\n#else\n #define __Pyx_BUILTIN_MODULE_NAME \"builtins\"\n #define __Pyx_PyCode_New(a, k, l, s, f, code, c, n, v, fv, cell, fn, name, fline, lnos)\\\n PyCode_New(a, k, l, s, f, code, c, n, v, fv, cell, fn, name, fline, lnos)\n #define __Pyx_DefaultClassType PyType_Type\n#endif\n#ifndef Py_TPFLAGS_CHECKTYPES\n #define Py_TPFLAGS_CHECKTYPES 0\n#endif\n#ifndef Py_TPFLAGS_HAVE_INDEX\n #define Py_TPFLAGS_HAVE_INDEX 0\n#endif\n#ifndef Py_TPFLAGS_HAVE_NEWBUFFER\n #define Py_TPFLAGS_HAVE_NEWBUFFER 0\n#endif\n#ifndef Py_TPFLAGS_HAVE_FINALIZE\n #define Py_TPFLAGS_HAVE_FINALIZE 0\n#endif\n#if PY_VERSION_HEX > 0x03030000 && defined(PyUnicode_KIND)\n #define CYTHON_PEP393_ENABLED 1\n #define __Pyx_PyUnicode_READY(op) (likely(PyUnicode_IS_READY(op)) ?\\\n 0 : _PyUnicode_Ready((PyObject *)(op)))\n #define __Pyx_PyUnicode_GET_LENGTH(u) PyUnicode_GET_LENGTH(u)\n #define __Pyx_PyUnicode_READ_CHAR(u, i) PyUnicode_READ_CHAR(u, i)\n #define __Pyx_PyUnicode_KIND(u) PyUnicode_KIND(u)\n #define __Pyx_PyUnicode_DATA(u) PyUnicode_DATA(u)\n #define __Pyx_PyUnicode_READ(k, d, i) PyUnicode_READ(k, d, i)\n #define __Pyx_PyUnicode_IS_TRUE(u) (0 != (likely(PyUnicode_IS_READY(u)) ? PyUnicode_GET_LENGTH(u) : PyUnicode_GET_SIZE(u)))\n#else\n #define CYTHON_PEP393_ENABLED 0\n #define __Pyx_PyUnicode_READY(op) (0)\n #define __Pyx_PyUnicode_GET_LENGTH(u) PyUnicode_GET_SIZE(u)\n #define __Pyx_PyUnicode_READ_CHAR(u, i) ((Py_UCS4)(PyUnicode_AS_UNICODE(u)[i]))\n #define __Pyx_PyUnicode_KIND(u) (sizeof(Py_UNICODE))\n #define __Pyx_PyUnicode_DATA(u) ((void*)PyUnicode_AS_UNICODE(u))\n #define __Pyx_PyUnicode_READ(k, d, i) ((void)(k), (Py_UCS4)(((Py_UNICODE*)d)[i]))\n #define __Pyx_PyUnicode_IS_TRUE(u) (0 != PyUnicode_GET_SIZE(u))\n#endif\n#if CYTHON_COMPILING_IN_PYPY\n #define __Pyx_PyUnicode_Concat(a, b) PyNumber_Add(a, b)\n #define __Pyx_PyUnicode_ConcatSafe(a, b) PyNumber_Add(a, b)\n#else\n #define __Pyx_PyUnicode_Concat(a, b) PyUnicode_Concat(a, b)\n #define __Pyx_PyUnicode_ConcatSafe(a, b) ((unlikely((a) == Py_None) || unlikely((b) == Py_None)) ?\\\n PyNumber_Add(a, b) : __Pyx_PyUnicode_Concat(a, b))\n#endif\n#if CYTHON_COMPILING_IN_PYPY && !defined(PyUnicode_Contains)\n #define PyUnicode_Contains(u, s) PySequence_Contains(u, s)\n#endif\n#if CYTHON_COMPILING_IN_PYPY && !defined(PyObject_Format)\n #define PyObject_Format(obj, fmt) PyObject_CallMethod(obj, \"__format__\", \"O\", fmt)\n#endif\n#if CYTHON_COMPILING_IN_PYPY && !defined(PyObject_Malloc)\n #define PyObject_Malloc(s) PyMem_Malloc(s)\n #define PyObject_Free(p) PyMem_Free(p)\n #define PyObject_Realloc(p) PyMem_Realloc(p)\n#endif\n#define __Pyx_PyString_FormatSafe(a, b) ((unlikely((a) == Py_None)) ? PyNumber_Remainder(a, b) : __Pyx_PyString_Format(a, b))\n#define __Pyx_PyUnicode_FormatSafe(a, b) ((unlikely((a) == Py_None)) ? PyNumber_Remainder(a, b) : PyUnicode_Format(a, b))\n#if PY_MAJOR_VERSION >= 3\n #define __Pyx_PyString_Format(a, b) PyUnicode_Format(a, b)\n#else\n #define __Pyx_PyString_Format(a, b) PyString_Format(a, b)\n#endif\n#if PY_MAJOR_VERSION < 3 && !defined(PyObject_ASCII)\n #define PyObject_ASCII(o) PyObject_Repr(o)\n#endif\n#if PY_MAJOR_VERSION >= 3\n #define PyBaseString_Type PyUnicode_Type\n #define PyStringObject PyUnicodeObject\n #define PyString_Type PyUnicode_Type\n #define PyString_Check PyUnicode_Check\n #define PyString_CheckExact PyUnicode_CheckExact\n#endif\n#if PY_MAJOR_VERSION >= 3\n #define __Pyx_PyBaseString_Check(obj) PyUnicode_Check(obj)\n #define __Pyx_PyBaseString_CheckExact(obj) PyUnicode_CheckExact(obj)\n#else\n #define __Pyx_PyBaseString_Check(obj) (PyString_Check(obj) || PyUnicode_Check(obj))\n #define __Pyx_PyBaseString_CheckExact(obj) (PyString_CheckExact(obj) || PyUnicode_CheckExact(obj))\n#endif\n#ifndef PySet_CheckExact\n #define PySet_CheckExact(obj) (Py_TYPE(obj) == &PySet_Type)\n#endif\n#define __Pyx_TypeCheck(obj, type) PyObject_TypeCheck(obj, (PyTypeObject *)type)\n#if PY_MAJOR_VERSION >= 3\n #define PyIntObject PyLongObject\n #define PyInt_Type PyLong_Type\n #define PyInt_Check(op) PyLong_Check(op)\n #define PyInt_CheckExact(op) PyLong_CheckExact(op)\n #define PyInt_FromString PyLong_FromString\n #define PyInt_FromUnicode PyLong_FromUnicode\n #define PyInt_FromLong PyLong_FromLong\n #define PyInt_FromSize_t PyLong_FromSize_t\n #define PyInt_FromSsize_t PyLong_FromSsize_t\n #define PyInt_AsLong PyLong_AsLong\n #define PyInt_AS_LONG PyLong_AS_LONG\n #define PyInt_AsSsize_t PyLong_AsSsize_t\n #define PyInt_AsUnsignedLongMask PyLong_AsUnsignedLongMask\n #define PyInt_AsUnsignedLongLongMask PyLong_AsUnsignedLongLongMask\n #define PyNumber_Int PyNumber_Long\n#endif\n#if PY_MAJOR_VERSION >= 3\n #define PyBoolObject PyLongObject\n#endif\n#if PY_MAJOR_VERSION >= 3 && CYTHON_COMPILING_IN_PYPY\n #ifndef PyUnicode_InternFromString\n #define PyUnicode_InternFromString(s) PyUnicode_FromString(s)\n #endif\n#endif\n#if PY_VERSION_HEX < 0x030200A4\n typedef long Py_hash_t;\n #define __Pyx_PyInt_FromHash_t PyInt_FromLong\n #define __Pyx_PyInt_AsHash_t PyInt_AsLong\n#else\n #define __Pyx_PyInt_FromHash_t PyInt_FromSsize_t\n #define __Pyx_PyInt_AsHash_t PyInt_AsSsize_t\n#endif\n#if PY_MAJOR_VERSION >= 3\n #define __Pyx_PyMethod_New(func, self, klass) ((self) ? PyMethod_New(func, self) : PyInstanceMethod_New(func))\n#else\n #define __Pyx_PyMethod_New(func, self, klass) PyMethod_New(func, self, klass)\n#endif\n#if PY_VERSION_HEX >= 0x030500B1\n#define __Pyx_PyAsyncMethodsStruct PyAsyncMethods\n#define __Pyx_PyType_AsAsync(obj) (Py_TYPE(obj)->tp_as_async)\n#elif CYTHON_COMPILING_IN_CPYTHON && PY_MAJOR_VERSION >= 3\ntypedef struct {\n unaryfunc am_await;\n unaryfunc am_aiter;\n unaryfunc am_anext;\n} __Pyx_PyAsyncMethodsStruct;\n#define __Pyx_PyType_AsAsync(obj) ((__Pyx_PyAsyncMethodsStruct*) (Py_TYPE(obj)->tp_reserved))\n#else\n#define __Pyx_PyType_AsAsync(obj) NULL\n#endif\n#ifndef CYTHON_RESTRICT\n #if defined(__GNUC__)\n #define CYTHON_RESTRICT __restrict__\n #elif defined(_MSC_VER) && _MSC_VER >= 1400\n #define CYTHON_RESTRICT __restrict\n #elif defined (__STDC_VERSION__) && __STDC_VERSION__ >= 199901L\n #define CYTHON_RESTRICT restrict\n #else\n #define CYTHON_RESTRICT\n #endif\n#endif\n#define __Pyx_void_to_None(void_result) ((void)(void_result), Py_INCREF(Py_None), Py_None)\n\n#ifndef __cplusplus\n #error \"Cython files generated with the C++ option must be compiled with a C++ compiler.\"\n#endif\n#ifndef CYTHON_INLINE\n #define CYTHON_INLINE inline\n#endif\ntemplate\nvoid __Pyx_call_destructor(T& x) {\n x.~T();\n}\ntemplate\nclass __Pyx_FakeReference {\n public:\n __Pyx_FakeReference() : ptr(NULL) { }\n __Pyx_FakeReference(const T& ref) : ptr(const_cast(&ref)) { }\n T *operator->() { return ptr; }\n operator T&() { return *ptr; }\n private:\n T *ptr;\n};\n\n#if defined(WIN32) || defined(MS_WINDOWS)\n #define _USE_MATH_DEFINES\n#endif\n#include \n#ifdef NAN\n#define __PYX_NAN() ((float) NAN)\n#else\nstatic CYTHON_INLINE float __PYX_NAN() {\n float value;\n memset(&value, 0xFF, sizeof(value));\n return value;\n}\n#endif\n\n\n#define __PYX_ERR(f_index, lineno, Ln_error) \\\n{ \\\n __pyx_filename = __pyx_f[f_index]; __pyx_lineno = lineno; __pyx_clineno = __LINE__; goto Ln_error; \\\n}\n\n#if PY_MAJOR_VERSION >= 3\n #define __Pyx_PyNumber_Divide(x,y) PyNumber_TrueDivide(x,y)\n #define __Pyx_PyNumber_InPlaceDivide(x,y) PyNumber_InPlaceTrueDivide(x,y)\n#else\n #define __Pyx_PyNumber_Divide(x,y) PyNumber_Divide(x,y)\n #define __Pyx_PyNumber_InPlaceDivide(x,y) PyNumber_InPlaceDivide(x,y)\n#endif\n\n#ifndef __PYX_EXTERN_C\n #ifdef __cplusplus\n #define __PYX_EXTERN_C extern \"C\"\n #else\n #define __PYX_EXTERN_C extern\n #endif\n#endif\n\n#define __PYX_HAVE__nms__gpu_nms\n#define __PYX_HAVE_API__nms__gpu_nms\n#include \"string.h\"\n#include \"stdio.h\"\n#include \"stdlib.h\"\n#include \"numpy/arrayobject.h\"\n#include \"numpy/ufuncobject.h\"\n#include \"gpu_nms.hpp\"\n#ifdef _OPENMP\n#include \n#endif /* _OPENMP */\n\n#ifdef PYREX_WITHOUT_ASSERTIONS\n#define CYTHON_WITHOUT_ASSERTIONS\n#endif\n\n#ifndef CYTHON_UNUSED\n# if defined(__GNUC__)\n# if !(defined(__cplusplus)) || (__GNUC__ > 3 || (__GNUC__ == 3 && __GNUC_MINOR__ >= 4))\n# define CYTHON_UNUSED __attribute__ ((__unused__))\n# else\n# define CYTHON_UNUSED\n# endif\n# elif defined(__ICC) || (defined(__INTEL_COMPILER) && !defined(_MSC_VER))\n# define CYTHON_UNUSED __attribute__ ((__unused__))\n# else\n# define CYTHON_UNUSED\n# endif\n#endif\n#ifndef CYTHON_NCP_UNUSED\n# if CYTHON_COMPILING_IN_CPYTHON\n# define CYTHON_NCP_UNUSED\n# else\n# define CYTHON_NCP_UNUSED CYTHON_UNUSED\n# endif\n#endif\ntypedef struct {PyObject **p; const char *s; const Py_ssize_t n; const char* encoding;\n const char is_unicode; const char is_str; const char intern; } __Pyx_StringTabEntry;\n\n#define __PYX_DEFAULT_STRING_ENCODING_IS_ASCII 0\n#define __PYX_DEFAULT_STRING_ENCODING_IS_DEFAULT 0\n#define __PYX_DEFAULT_STRING_ENCODING \"\"\n#define __Pyx_PyObject_FromString __Pyx_PyBytes_FromString\n#define __Pyx_PyObject_FromStringAndSize __Pyx_PyBytes_FromStringAndSize\n#define __Pyx_uchar_cast(c) ((unsigned char)c)\n#define __Pyx_long_cast(x) ((long)x)\n#define __Pyx_fits_Py_ssize_t(v, type, is_signed) (\\\n (sizeof(type) < sizeof(Py_ssize_t)) ||\\\n (sizeof(type) > sizeof(Py_ssize_t) &&\\\n likely(v < (type)PY_SSIZE_T_MAX ||\\\n v == (type)PY_SSIZE_T_MAX) &&\\\n (!is_signed || likely(v > (type)PY_SSIZE_T_MIN ||\\\n v == (type)PY_SSIZE_T_MIN))) ||\\\n (sizeof(type) == sizeof(Py_ssize_t) &&\\\n (is_signed || likely(v < (type)PY_SSIZE_T_MAX ||\\\n v == (type)PY_SSIZE_T_MAX))) )\n#if defined (__cplusplus) && __cplusplus >= 201103L\n #include \n #define __Pyx_sst_abs(value) std::abs(value)\n#elif SIZEOF_INT >= SIZEOF_SIZE_T\n #define __Pyx_sst_abs(value) abs(value)\n#elif SIZEOF_LONG >= SIZEOF_SIZE_T\n #define __Pyx_sst_abs(value) labs(value)\n#elif defined (_MSC_VER) && defined (_M_X64)\n #define __Pyx_sst_abs(value) _abs64(value)\n#elif defined (__STDC_VERSION__) && __STDC_VERSION__ >= 199901L\n #define __Pyx_sst_abs(value) llabs(value)\n#elif defined (__GNUC__)\n #define __Pyx_sst_abs(value) __builtin_llabs(value)\n#else\n #define __Pyx_sst_abs(value) ((value<0) ? -value : value)\n#endif\nstatic CYTHON_INLINE char* __Pyx_PyObject_AsString(PyObject*);\nstatic CYTHON_INLINE char* __Pyx_PyObject_AsStringAndSize(PyObject*, Py_ssize_t* length);\n#define __Pyx_PyByteArray_FromString(s) PyByteArray_FromStringAndSize((const char*)s, strlen((const char*)s))\n#define __Pyx_PyByteArray_FromStringAndSize(s, l) PyByteArray_FromStringAndSize((const char*)s, l)\n#define __Pyx_PyBytes_FromString PyBytes_FromString\n#define __Pyx_PyBytes_FromStringAndSize PyBytes_FromStringAndSize\nstatic CYTHON_INLINE PyObject* __Pyx_PyUnicode_FromString(const char*);\n#if PY_MAJOR_VERSION < 3\n #define __Pyx_PyStr_FromString __Pyx_PyBytes_FromString\n #define __Pyx_PyStr_FromStringAndSize __Pyx_PyBytes_FromStringAndSize\n#else\n #define __Pyx_PyStr_FromString __Pyx_PyUnicode_FromString\n #define __Pyx_PyStr_FromStringAndSize __Pyx_PyUnicode_FromStringAndSize\n#endif\n#define __Pyx_PyObject_AsSString(s) ((signed char*) __Pyx_PyObject_AsString(s))\n#define __Pyx_PyObject_AsUString(s) ((unsigned char*) __Pyx_PyObject_AsString(s))\n#define __Pyx_PyObject_FromCString(s) __Pyx_PyObject_FromString((const char*)s)\n#define __Pyx_PyBytes_FromCString(s) __Pyx_PyBytes_FromString((const char*)s)\n#define __Pyx_PyByteArray_FromCString(s) __Pyx_PyByteArray_FromString((const char*)s)\n#define __Pyx_PyStr_FromCString(s) __Pyx_PyStr_FromString((const char*)s)\n#define __Pyx_PyUnicode_FromCString(s) __Pyx_PyUnicode_FromString((const char*)s)\n#if PY_MAJOR_VERSION < 3\nstatic CYTHON_INLINE size_t __Pyx_Py_UNICODE_strlen(const Py_UNICODE *u)\n{\n const Py_UNICODE *u_end = u;\n while (*u_end++) ;\n return (size_t)(u_end - u - 1);\n}\n#else\n#define __Pyx_Py_UNICODE_strlen Py_UNICODE_strlen\n#endif\n#define __Pyx_PyUnicode_FromUnicode(u) PyUnicode_FromUnicode(u, __Pyx_Py_UNICODE_strlen(u))\n#define __Pyx_PyUnicode_FromUnicodeAndLength PyUnicode_FromUnicode\n#define __Pyx_PyUnicode_AsUnicode PyUnicode_AsUnicode\n#define __Pyx_NewRef(obj) (Py_INCREF(obj), obj)\n#define __Pyx_Owned_Py_None(b) __Pyx_NewRef(Py_None)\n#define __Pyx_PyBool_FromLong(b) ((b) ? __Pyx_NewRef(Py_True) : __Pyx_NewRef(Py_False))\nstatic CYTHON_INLINE int __Pyx_PyObject_IsTrue(PyObject*);\nstatic CYTHON_INLINE PyObject* __Pyx_PyNumber_IntOrLong(PyObject* x);\nstatic CYTHON_INLINE Py_ssize_t __Pyx_PyIndex_AsSsize_t(PyObject*);\nstatic CYTHON_INLINE PyObject * __Pyx_PyInt_FromSize_t(size_t);\n#if CYTHON_COMPILING_IN_CPYTHON\n#define __pyx_PyFloat_AsDouble(x) (PyFloat_CheckExact(x) ? PyFloat_AS_DOUBLE(x) : PyFloat_AsDouble(x))\n#else\n#define __pyx_PyFloat_AsDouble(x) PyFloat_AsDouble(x)\n#endif\n#define __pyx_PyFloat_AsFloat(x) ((float) __pyx_PyFloat_AsDouble(x))\n#if PY_MAJOR_VERSION >= 3\n#define __Pyx_PyNumber_Int(x) (PyLong_CheckExact(x) ? __Pyx_NewRef(x) : PyNumber_Long(x))\n#else\n#define __Pyx_PyNumber_Int(x) (PyInt_CheckExact(x) ? __Pyx_NewRef(x) : PyNumber_Int(x))\n#endif\n#define __Pyx_PyNumber_Float(x) (PyFloat_CheckExact(x) ? __Pyx_NewRef(x) : PyNumber_Float(x))\n#if PY_MAJOR_VERSION < 3 && __PYX_DEFAULT_STRING_ENCODING_IS_ASCII\nstatic int __Pyx_sys_getdefaultencoding_not_ascii;\nstatic int __Pyx_init_sys_getdefaultencoding_params(void) {\n PyObject* sys;\n PyObject* default_encoding = NULL;\n PyObject* ascii_chars_u = NULL;\n PyObject* ascii_chars_b = NULL;\n const char* default_encoding_c;\n sys = PyImport_ImportModule(\"sys\");\n if (!sys) goto bad;\n default_encoding = PyObject_CallMethod(sys, (char*) \"getdefaultencoding\", NULL);\n Py_DECREF(sys);\n if (!default_encoding) goto bad;\n default_encoding_c = PyBytes_AsString(default_encoding);\n if (!default_encoding_c) goto bad;\n if (strcmp(default_encoding_c, \"ascii\") == 0) {\n __Pyx_sys_getdefaultencoding_not_ascii = 0;\n } else {\n char ascii_chars[128];\n int c;\n for (c = 0; c < 128; c++) {\n ascii_chars[c] = c;\n }\n __Pyx_sys_getdefaultencoding_not_ascii = 1;\n ascii_chars_u = PyUnicode_DecodeASCII(ascii_chars, 128, NULL);\n if (!ascii_chars_u) goto bad;\n ascii_chars_b = PyUnicode_AsEncodedString(ascii_chars_u, default_encoding_c, NULL);\n if (!ascii_chars_b || !PyBytes_Check(ascii_chars_b) || memcmp(ascii_chars, PyBytes_AS_STRING(ascii_chars_b), 128) != 0) {\n PyErr_Format(\n PyExc_ValueError,\n \"This module compiled with c_string_encoding=ascii, but default encoding '%.200s' is not a superset of ascii.\",\n default_encoding_c);\n goto bad;\n }\n Py_DECREF(ascii_chars_u);\n Py_DECREF(ascii_chars_b);\n }\n Py_DECREF(default_encoding);\n return 0;\nbad:\n Py_XDECREF(default_encoding);\n Py_XDECREF(ascii_chars_u);\n Py_XDECREF(ascii_chars_b);\n return -1;\n}\n#endif\n#if __PYX_DEFAULT_STRING_ENCODING_IS_DEFAULT && PY_MAJOR_VERSION >= 3\n#define __Pyx_PyUnicode_FromStringAndSize(c_str, size) PyUnicode_DecodeUTF8(c_str, size, NULL)\n#else\n#define __Pyx_PyUnicode_FromStringAndSize(c_str, size) PyUnicode_Decode(c_str, size, __PYX_DEFAULT_STRING_ENCODING, NULL)\n#if __PYX_DEFAULT_STRING_ENCODING_IS_DEFAULT\nstatic char* __PYX_DEFAULT_STRING_ENCODING;\nstatic int __Pyx_init_sys_getdefaultencoding_params(void) {\n PyObject* sys;\n PyObject* default_encoding = NULL;\n char* default_encoding_c;\n sys = PyImport_ImportModule(\"sys\");\n if (!sys) goto bad;\n default_encoding = PyObject_CallMethod(sys, (char*) (const char*) \"getdefaultencoding\", NULL);\n Py_DECREF(sys);\n if (!default_encoding) goto bad;\n default_encoding_c = PyBytes_AsString(default_encoding);\n if (!default_encoding_c) goto bad;\n __PYX_DEFAULT_STRING_ENCODING = (char*) malloc(strlen(default_encoding_c));\n if (!__PYX_DEFAULT_STRING_ENCODING) goto bad;\n strcpy(__PYX_DEFAULT_STRING_ENCODING, default_encoding_c);\n Py_DECREF(default_encoding);\n return 0;\nbad:\n Py_XDECREF(default_encoding);\n return -1;\n}\n#endif\n#endif\n\n\n/* Test for GCC > 2.95 */\n#if defined(__GNUC__) && (__GNUC__ > 2 || (__GNUC__ == 2 && (__GNUC_MINOR__ > 95)))\n #define likely(x) __builtin_expect(!!(x), 1)\n #define unlikely(x) __builtin_expect(!!(x), 0)\n#else /* !__GNUC__ or GCC < 2.95 */\n #define likely(x) (x)\n #define unlikely(x) (x)\n#endif /* __GNUC__ */\n\nstatic PyObject *__pyx_m;\nstatic PyObject *__pyx_d;\nstatic PyObject *__pyx_b;\nstatic PyObject *__pyx_empty_tuple;\nstatic PyObject *__pyx_empty_bytes;\nstatic PyObject *__pyx_empty_unicode;\nstatic int __pyx_lineno;\nstatic int __pyx_clineno = 0;\nstatic const char * __pyx_cfilenm= __FILE__;\nstatic const char *__pyx_filename;\n\n/* None.proto */\n#if !defined(CYTHON_CCOMPLEX)\n #if defined(__cplusplus)\n #define CYTHON_CCOMPLEX 1\n #elif defined(_Complex_I)\n #define CYTHON_CCOMPLEX 1\n #else\n #define CYTHON_CCOMPLEX 0\n #endif\n#endif\n#if CYTHON_CCOMPLEX\n #ifdef __cplusplus\n #include \n #else\n #include \n #endif\n#endif\n#if CYTHON_CCOMPLEX && !defined(__cplusplus) && defined(__sun__) && defined(__GNUC__)\n #undef _Complex_I\n #define _Complex_I 1.0fj\n#endif\n\n\nstatic const char *__pyx_f[] = {\n \"nms\\\\gpu_nms.pyx\",\n \"__init__.pxd\",\n \"type.pxd\",\n};\n/* BufferFormatStructs.proto */\n#define IS_UNSIGNED(type) (((type) -1) > 0)\nstruct __Pyx_StructField_;\n#define __PYX_BUF_FLAGS_PACKED_STRUCT (1 << 0)\ntypedef struct {\n const char* name;\n struct __Pyx_StructField_* fields;\n size_t size;\n size_t arraysize[8];\n int ndim;\n char typegroup;\n char is_unsigned;\n int flags;\n} __Pyx_TypeInfo;\ntypedef struct __Pyx_StructField_ {\n __Pyx_TypeInfo* type;\n const char* name;\n size_t offset;\n} __Pyx_StructField;\ntypedef struct {\n __Pyx_StructField* field;\n size_t parent_offset;\n} __Pyx_BufFmt_StackElem;\ntypedef struct {\n __Pyx_StructField root;\n __Pyx_BufFmt_StackElem* head;\n size_t fmt_offset;\n size_t new_count, enc_count;\n size_t struct_alignment;\n int is_complex;\n char enc_type;\n char new_packmode;\n char enc_packmode;\n char is_valid_array;\n} __Pyx_BufFmt_Context;\n\n\n/* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":725\n * # in Cython to enable them only on the right systems.\n * \n * ctypedef npy_int8 int8_t # <<<<<<<<<<<<<<\n * ctypedef npy_int16 int16_t\n * ctypedef npy_int32 int32_t\n */\ntypedef npy_int8 __pyx_t_5numpy_int8_t;\n\n/* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":726\n * \n * ctypedef npy_int8 int8_t\n * ctypedef npy_int16 int16_t # <<<<<<<<<<<<<<\n * ctypedef npy_int32 int32_t\n * ctypedef npy_int64 int64_t\n */\ntypedef npy_int16 __pyx_t_5numpy_int16_t;\n\n/* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":727\n * ctypedef npy_int8 int8_t\n * ctypedef npy_int16 int16_t\n * ctypedef npy_int32 int32_t # <<<<<<<<<<<<<<\n * ctypedef npy_int64 int64_t\n * #ctypedef npy_int96 int96_t\n */\ntypedef npy_int32 __pyx_t_5numpy_int32_t;\n\n/* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":728\n * ctypedef npy_int16 int16_t\n * ctypedef npy_int32 int32_t\n * ctypedef npy_int64 int64_t # <<<<<<<<<<<<<<\n * #ctypedef npy_int96 int96_t\n * #ctypedef npy_int128 int128_t\n */\ntypedef npy_int64 __pyx_t_5numpy_int64_t;\n\n/* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":732\n * #ctypedef npy_int128 int128_t\n * \n * ctypedef npy_uint8 uint8_t # <<<<<<<<<<<<<<\n * ctypedef npy_uint16 uint16_t\n * ctypedef npy_uint32 uint32_t\n */\ntypedef npy_uint8 __pyx_t_5numpy_uint8_t;\n\n/* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":733\n * \n * ctypedef npy_uint8 uint8_t\n * ctypedef npy_uint16 uint16_t # <<<<<<<<<<<<<<\n * ctypedef npy_uint32 uint32_t\n * ctypedef npy_uint64 uint64_t\n */\ntypedef npy_uint16 __pyx_t_5numpy_uint16_t;\n\n/* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":734\n * ctypedef npy_uint8 uint8_t\n * ctypedef npy_uint16 uint16_t\n * ctypedef npy_uint32 uint32_t # <<<<<<<<<<<<<<\n * ctypedef npy_uint64 uint64_t\n * #ctypedef npy_uint96 uint96_t\n */\ntypedef npy_uint32 __pyx_t_5numpy_uint32_t;\n\n/* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":735\n * ctypedef npy_uint16 uint16_t\n * ctypedef npy_uint32 uint32_t\n * ctypedef npy_uint64 uint64_t # <<<<<<<<<<<<<<\n * #ctypedef npy_uint96 uint96_t\n * #ctypedef npy_uint128 uint128_t\n */\ntypedef npy_uint64 __pyx_t_5numpy_uint64_t;\n\n/* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":739\n * #ctypedef npy_uint128 uint128_t\n * \n * ctypedef npy_float32 float32_t # <<<<<<<<<<<<<<\n * ctypedef npy_float64 float64_t\n * #ctypedef npy_float80 float80_t\n */\ntypedef npy_float32 __pyx_t_5numpy_float32_t;\n\n/* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":740\n * \n * ctypedef npy_float32 float32_t\n * ctypedef npy_float64 float64_t # <<<<<<<<<<<<<<\n * #ctypedef npy_float80 float80_t\n * #ctypedef npy_float128 float128_t\n */\ntypedef npy_float64 __pyx_t_5numpy_float64_t;\n\n/* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":749\n * # The int types are mapped a bit surprising --\n * # numpy.int corresponds to 'l' and numpy.long to 'q'\n * ctypedef npy_long int_t # <<<<<<<<<<<<<<\n * ctypedef npy_longlong long_t\n * ctypedef npy_longlong longlong_t\n */\ntypedef npy_long __pyx_t_5numpy_int_t;\n\n/* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":750\n * # numpy.int corresponds to 'l' and numpy.long to 'q'\n * ctypedef npy_long int_t\n * ctypedef npy_longlong long_t # <<<<<<<<<<<<<<\n * ctypedef npy_longlong longlong_t\n * \n */\ntypedef npy_longlong __pyx_t_5numpy_long_t;\n\n/* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":751\n * ctypedef npy_long int_t\n * ctypedef npy_longlong long_t\n * ctypedef npy_longlong longlong_t # <<<<<<<<<<<<<<\n * \n * ctypedef npy_ulong uint_t\n */\ntypedef npy_longlong __pyx_t_5numpy_longlong_t;\n\n/* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":753\n * ctypedef npy_longlong longlong_t\n * \n * ctypedef npy_ulong uint_t # <<<<<<<<<<<<<<\n * ctypedef npy_ulonglong ulong_t\n * ctypedef npy_ulonglong ulonglong_t\n */\ntypedef npy_ulong __pyx_t_5numpy_uint_t;\n\n/* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":754\n * \n * ctypedef npy_ulong uint_t\n * ctypedef npy_ulonglong ulong_t # <<<<<<<<<<<<<<\n * ctypedef npy_ulonglong ulonglong_t\n * \n */\ntypedef npy_ulonglong __pyx_t_5numpy_ulong_t;\n\n/* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":755\n * ctypedef npy_ulong uint_t\n * ctypedef npy_ulonglong ulong_t\n * ctypedef npy_ulonglong ulonglong_t # <<<<<<<<<<<<<<\n * \n * ctypedef npy_intp intp_t\n */\ntypedef npy_ulonglong __pyx_t_5numpy_ulonglong_t;\n\n/* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":757\n * ctypedef npy_ulonglong ulonglong_t\n * \n * ctypedef npy_intp intp_t # <<<<<<<<<<<<<<\n * ctypedef npy_uintp uintp_t\n * \n */\ntypedef npy_intp __pyx_t_5numpy_intp_t;\n\n/* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":758\n * \n * ctypedef npy_intp intp_t\n * ctypedef npy_uintp uintp_t # <<<<<<<<<<<<<<\n * \n * ctypedef npy_double float_t\n */\ntypedef npy_uintp __pyx_t_5numpy_uintp_t;\n\n/* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":760\n * ctypedef npy_uintp uintp_t\n * \n * ctypedef npy_double float_t # <<<<<<<<<<<<<<\n * ctypedef npy_double double_t\n * ctypedef npy_longdouble longdouble_t\n */\ntypedef npy_double __pyx_t_5numpy_float_t;\n\n/* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":761\n * \n * ctypedef npy_double float_t\n * ctypedef npy_double double_t # <<<<<<<<<<<<<<\n * ctypedef npy_longdouble longdouble_t\n * \n */\ntypedef npy_double __pyx_t_5numpy_double_t;\n\n/* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":762\n * ctypedef npy_double float_t\n * ctypedef npy_double double_t\n * ctypedef npy_longdouble longdouble_t # <<<<<<<<<<<<<<\n * \n * ctypedef npy_cfloat cfloat_t\n */\ntypedef npy_longdouble __pyx_t_5numpy_longdouble_t;\n/* None.proto */\n#if CYTHON_CCOMPLEX\n #ifdef __cplusplus\n typedef ::std::complex< float > __pyx_t_float_complex;\n #else\n typedef float _Complex __pyx_t_float_complex;\n #endif\n#else\n typedef struct { float real, imag; } __pyx_t_float_complex;\n#endif\n\n/* None.proto */\n#if CYTHON_CCOMPLEX\n #ifdef __cplusplus\n typedef ::std::complex< double > __pyx_t_double_complex;\n #else\n typedef double _Complex __pyx_t_double_complex;\n #endif\n#else\n typedef struct { double real, imag; } __pyx_t_double_complex;\n#endif\n\n\n/*--- Type declarations ---*/\n\n/* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":764\n * ctypedef npy_longdouble longdouble_t\n * \n * ctypedef npy_cfloat cfloat_t # <<<<<<<<<<<<<<\n * ctypedef npy_cdouble cdouble_t\n * ctypedef npy_clongdouble clongdouble_t\n */\ntypedef npy_cfloat __pyx_t_5numpy_cfloat_t;\n\n/* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":765\n * \n * ctypedef npy_cfloat cfloat_t\n * ctypedef npy_cdouble cdouble_t # <<<<<<<<<<<<<<\n * ctypedef npy_clongdouble clongdouble_t\n * \n */\ntypedef npy_cdouble __pyx_t_5numpy_cdouble_t;\n\n/* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":766\n * ctypedef npy_cfloat cfloat_t\n * ctypedef npy_cdouble cdouble_t\n * ctypedef npy_clongdouble clongdouble_t # <<<<<<<<<<<<<<\n * \n * ctypedef npy_cdouble complex_t\n */\ntypedef npy_clongdouble __pyx_t_5numpy_clongdouble_t;\n\n/* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":768\n * ctypedef npy_clongdouble clongdouble_t\n * \n * ctypedef npy_cdouble complex_t # <<<<<<<<<<<<<<\n * \n * cdef inline object PyArray_MultiIterNew1(a):\n */\ntypedef npy_cdouble __pyx_t_5numpy_complex_t;\n\n/* --- Runtime support code (head) --- */\n/* Refnanny.proto */\n#ifndef CYTHON_REFNANNY\n #define CYTHON_REFNANNY 0\n#endif\n#if CYTHON_REFNANNY\n typedef struct {\n void (*INCREF)(void*, PyObject*, int);\n void (*DECREF)(void*, PyObject*, int);\n void (*GOTREF)(void*, PyObject*, int);\n void (*GIVEREF)(void*, PyObject*, int);\n void* (*SetupContext)(const char*, int, const char*);\n void (*FinishContext)(void**);\n } __Pyx_RefNannyAPIStruct;\n static __Pyx_RefNannyAPIStruct *__Pyx_RefNanny = NULL;\n static __Pyx_RefNannyAPIStruct *__Pyx_RefNannyImportAPI(const char *modname);\n #define __Pyx_RefNannyDeclarations void *__pyx_refnanny = NULL;\n#ifdef WITH_THREAD\n #define __Pyx_RefNannySetupContext(name, acquire_gil)\\\n if (acquire_gil) {\\\n PyGILState_STATE __pyx_gilstate_save = PyGILState_Ensure();\\\n __pyx_refnanny = __Pyx_RefNanny->SetupContext((name), __LINE__, __FILE__);\\\n PyGILState_Release(__pyx_gilstate_save);\\\n } else {\\\n __pyx_refnanny = __Pyx_RefNanny->SetupContext((name), __LINE__, __FILE__);\\\n }\n#else\n #define __Pyx_RefNannySetupContext(name, acquire_gil)\\\n __pyx_refnanny = __Pyx_RefNanny->SetupContext((name), __LINE__, __FILE__)\n#endif\n #define __Pyx_RefNannyFinishContext()\\\n __Pyx_RefNanny->FinishContext(&__pyx_refnanny)\n #define __Pyx_INCREF(r) __Pyx_RefNanny->INCREF(__pyx_refnanny, (PyObject *)(r), __LINE__)\n #define __Pyx_DECREF(r) __Pyx_RefNanny->DECREF(__pyx_refnanny, (PyObject *)(r), __LINE__)\n #define __Pyx_GOTREF(r) __Pyx_RefNanny->GOTREF(__pyx_refnanny, (PyObject *)(r), __LINE__)\n #define __Pyx_GIVEREF(r) __Pyx_RefNanny->GIVEREF(__pyx_refnanny, (PyObject *)(r), __LINE__)\n #define __Pyx_XINCREF(r) do { if((r) != NULL) {__Pyx_INCREF(r); }} while(0)\n #define __Pyx_XDECREF(r) do { if((r) != NULL) {__Pyx_DECREF(r); }} while(0)\n #define __Pyx_XGOTREF(r) do { if((r) != NULL) {__Pyx_GOTREF(r); }} while(0)\n #define __Pyx_XGIVEREF(r) do { if((r) != NULL) {__Pyx_GIVEREF(r);}} while(0)\n#else\n #define __Pyx_RefNannyDeclarations\n #define __Pyx_RefNannySetupContext(name, acquire_gil)\n #define __Pyx_RefNannyFinishContext()\n #define __Pyx_INCREF(r) Py_INCREF(r)\n #define __Pyx_DECREF(r) Py_DECREF(r)\n #define __Pyx_GOTREF(r)\n #define __Pyx_GIVEREF(r)\n #define __Pyx_XINCREF(r) Py_XINCREF(r)\n #define __Pyx_XDECREF(r) Py_XDECREF(r)\n #define __Pyx_XGOTREF(r)\n #define __Pyx_XGIVEREF(r)\n#endif\n#define __Pyx_XDECREF_SET(r, v) do {\\\n PyObject *tmp = (PyObject *) r;\\\n r = v; __Pyx_XDECREF(tmp);\\\n } while (0)\n#define __Pyx_DECREF_SET(r, v) do {\\\n PyObject *tmp = (PyObject *) r;\\\n r = v; __Pyx_DECREF(tmp);\\\n } while (0)\n#define __Pyx_CLEAR(r) do { PyObject* tmp = ((PyObject*)(r)); r = NULL; __Pyx_DECREF(tmp);} while(0)\n#define __Pyx_XCLEAR(r) do { if((r) != NULL) {PyObject* tmp = ((PyObject*)(r)); r = NULL; __Pyx_DECREF(tmp);}} while(0)\n\n/* RaiseArgTupleInvalid.proto */\nstatic void __Pyx_RaiseArgtupleInvalid(const char* func_name, int exact,\n Py_ssize_t num_min, Py_ssize_t num_max, Py_ssize_t num_found);\n\n/* RaiseDoubleKeywords.proto */\nstatic void __Pyx_RaiseDoubleKeywordsError(const char* func_name, PyObject* kw_name);\n\n/* ParseKeywords.proto */\nstatic int __Pyx_ParseOptionalKeywords(PyObject *kwds, PyObject **argnames[],\\\n PyObject *kwds2, PyObject *values[], Py_ssize_t num_pos_args,\\\n const char* function_name);\n\n/* ArgTypeTest.proto */\nstatic CYTHON_INLINE int __Pyx_ArgTypeTest(PyObject *obj, PyTypeObject *type, int none_allowed,\n const char *name, int exact);\n\n/* BufferFormatCheck.proto */\nstatic CYTHON_INLINE int __Pyx_GetBufferAndValidate(Py_buffer* buf, PyObject* obj,\n __Pyx_TypeInfo* dtype, int flags, int nd, int cast, __Pyx_BufFmt_StackElem* stack);\nstatic CYTHON_INLINE void __Pyx_SafeReleaseBuffer(Py_buffer* info);\nstatic const char* __Pyx_BufFmt_CheckString(__Pyx_BufFmt_Context* ctx, const char* ts);\nstatic void __Pyx_BufFmt_Init(__Pyx_BufFmt_Context* ctx,\n __Pyx_BufFmt_StackElem* stack,\n __Pyx_TypeInfo* type); // PROTO\n\n/* PyObjectGetAttrStr.proto */\n#if CYTHON_COMPILING_IN_CPYTHON\nstatic CYTHON_INLINE PyObject* __Pyx_PyObject_GetAttrStr(PyObject* obj, PyObject* attr_name) {\n PyTypeObject* tp = Py_TYPE(obj);\n if (likely(tp->tp_getattro))\n return tp->tp_getattro(obj, attr_name);\n#if PY_MAJOR_VERSION < 3\n if (likely(tp->tp_getattr))\n return tp->tp_getattr(obj, PyString_AS_STRING(attr_name));\n#endif\n return PyObject_GetAttr(obj, attr_name);\n}\n#else\n#define __Pyx_PyObject_GetAttrStr(o,n) PyObject_GetAttr(o,n)\n#endif\n\n/* GetBuiltinName.proto */\nstatic PyObject *__Pyx_GetBuiltinName(PyObject *name);\n\n/* GetModuleGlobalName.proto */\nstatic CYTHON_INLINE PyObject *__Pyx_GetModuleGlobalName(PyObject *name);\n\n/* PyObjectCall.proto */\n#if CYTHON_COMPILING_IN_CPYTHON\nstatic CYTHON_INLINE PyObject* __Pyx_PyObject_Call(PyObject *func, PyObject *arg, PyObject *kw);\n#else\n#define __Pyx_PyObject_Call(func, arg, kw) PyObject_Call(func, arg, kw)\n#endif\n\n/* ExtTypeTest.proto */\nstatic CYTHON_INLINE int __Pyx_TypeTest(PyObject *obj, PyTypeObject *type);\n\n/* PyObjectCallMethO.proto */\n#if CYTHON_COMPILING_IN_CPYTHON\nstatic CYTHON_INLINE PyObject* __Pyx_PyObject_CallMethO(PyObject *func, PyObject *arg);\n#endif\n\n/* PyObjectCallOneArg.proto */\nstatic CYTHON_INLINE PyObject* __Pyx_PyObject_CallOneArg(PyObject *func, PyObject *arg);\n\n/* PyObjectCallNoArg.proto */\n#if CYTHON_COMPILING_IN_CPYTHON\nstatic CYTHON_INLINE PyObject* __Pyx_PyObject_CallNoArg(PyObject *func);\n#else\n#define __Pyx_PyObject_CallNoArg(func) __Pyx_PyObject_Call(func, __pyx_empty_tuple, NULL)\n#endif\n\n/* BufferIndexError.proto */\nstatic void __Pyx_RaiseBufferIndexError(int axis);\n\n#define __Pyx_BufPtrStrided1d(type, buf, i0, s0) (type)((char*)buf + i0 * s0)\n#define __Pyx_BufPtrStrided2d(type, buf, i0, s0, i1, s1) (type)((char*)buf + i0 * s0 + i1 * s1)\n/* SliceObject.proto */\nstatic CYTHON_INLINE PyObject* __Pyx_PyObject_GetSlice(\n PyObject* obj, Py_ssize_t cstart, Py_ssize_t cstop,\n PyObject** py_start, PyObject** py_stop, PyObject** py_slice,\n int has_cstart, int has_cstop, int wraparound);\n\n/* BufferFallbackError.proto */\nstatic void __Pyx_RaiseBufferFallbackError(void);\n\n/* PyThreadStateGet.proto */\n#if CYTHON_COMPILING_IN_CPYTHON\n#define __Pyx_PyThreadState_declare PyThreadState *__pyx_tstate;\n#define __Pyx_PyThreadState_assign __pyx_tstate = PyThreadState_GET();\n#else\n#define __Pyx_PyThreadState_declare\n#define __Pyx_PyThreadState_assign\n#endif\n\n/* PyErrFetchRestore.proto */\n#if CYTHON_COMPILING_IN_CPYTHON\n#define __Pyx_ErrRestoreWithState(type, value, tb) __Pyx_ErrRestoreInState(PyThreadState_GET(), type, value, tb)\n#define __Pyx_ErrFetchWithState(type, value, tb) __Pyx_ErrFetchInState(PyThreadState_GET(), type, value, tb)\n#define __Pyx_ErrRestore(type, value, tb) __Pyx_ErrRestoreInState(__pyx_tstate, type, value, tb)\n#define __Pyx_ErrFetch(type, value, tb) __Pyx_ErrFetchInState(__pyx_tstate, type, value, tb)\nstatic CYTHON_INLINE void __Pyx_ErrRestoreInState(PyThreadState *tstate, PyObject *type, PyObject *value, PyObject *tb);\nstatic CYTHON_INLINE void __Pyx_ErrFetchInState(PyThreadState *tstate, PyObject **type, PyObject **value, PyObject **tb);\n#else\n#define __Pyx_ErrRestoreWithState(type, value, tb) PyErr_Restore(type, value, tb)\n#define __Pyx_ErrFetchWithState(type, value, tb) PyErr_Fetch(type, value, tb)\n#define __Pyx_ErrRestore(type, value, tb) PyErr_Restore(type, value, tb)\n#define __Pyx_ErrFetch(type, value, tb) PyErr_Fetch(type, value, tb)\n#endif\n\n/* RaiseException.proto */\nstatic void __Pyx_Raise(PyObject *type, PyObject *value, PyObject *tb, PyObject *cause);\n\n/* DictGetItem.proto */\n#if PY_MAJOR_VERSION >= 3 && !CYTHON_COMPILING_IN_PYPY\nstatic PyObject *__Pyx_PyDict_GetItem(PyObject *d, PyObject* key) {\n PyObject *value;\n value = PyDict_GetItemWithError(d, key);\n if (unlikely(!value)) {\n if (!PyErr_Occurred()) {\n PyObject* args = PyTuple_Pack(1, key);\n if (likely(args))\n PyErr_SetObject(PyExc_KeyError, args);\n Py_XDECREF(args);\n }\n return NULL;\n }\n Py_INCREF(value);\n return value;\n}\n#else\n #define __Pyx_PyDict_GetItem(d, key) PyObject_GetItem(d, key)\n#endif\n\n/* RaiseTooManyValuesToUnpack.proto */\nstatic CYTHON_INLINE void __Pyx_RaiseTooManyValuesError(Py_ssize_t expected);\n\n/* RaiseNeedMoreValuesToUnpack.proto */\nstatic CYTHON_INLINE void __Pyx_RaiseNeedMoreValuesError(Py_ssize_t index);\n\n/* RaiseNoneIterError.proto */\nstatic CYTHON_INLINE void __Pyx_RaiseNoneNotIterableError(void);\n\n/* Import.proto */\nstatic PyObject *__Pyx_Import(PyObject *name, PyObject *from_list, int level);\n\n/* CodeObjectCache.proto */\ntypedef struct {\n PyCodeObject* code_object;\n int code_line;\n} __Pyx_CodeObjectCacheEntry;\nstruct __Pyx_CodeObjectCache {\n int count;\n int max_count;\n __Pyx_CodeObjectCacheEntry* entries;\n};\nstatic struct __Pyx_CodeObjectCache __pyx_code_cache = {0,0,NULL};\nstatic int __pyx_bisect_code_objects(__Pyx_CodeObjectCacheEntry* entries, int count, int code_line);\nstatic PyCodeObject *__pyx_find_code_object(int code_line);\nstatic void __pyx_insert_code_object(int code_line, PyCodeObject* code_object);\n\n/* AddTraceback.proto */\nstatic void __Pyx_AddTraceback(const char *funcname, int c_line,\n int py_line, const char *filename);\n\n/* BufferStructDeclare.proto */\ntypedef struct {\n Py_ssize_t shape, strides, suboffsets;\n} __Pyx_Buf_DimInfo;\ntypedef struct {\n size_t refcount;\n Py_buffer pybuffer;\n} __Pyx_Buffer;\ntypedef struct {\n __Pyx_Buffer *rcbuffer;\n char *data;\n __Pyx_Buf_DimInfo diminfo[8];\n} __Pyx_LocalBuf_ND;\n\n#if PY_MAJOR_VERSION < 3\n static int __Pyx_GetBuffer(PyObject *obj, Py_buffer *view, int flags);\n static void __Pyx_ReleaseBuffer(Py_buffer *view);\n#else\n #define __Pyx_GetBuffer PyObject_GetBuffer\n #define __Pyx_ReleaseBuffer PyBuffer_Release\n#endif\n\n\n/* None.proto */\nstatic Py_ssize_t __Pyx_zeros[] = {0, 0, 0, 0, 0, 0, 0, 0};\nstatic Py_ssize_t __Pyx_minusones[] = {-1, -1, -1, -1, -1, -1, -1, -1};\n\n/* CIntToPy.proto */\nstatic CYTHON_INLINE PyObject* __Pyx_PyInt_From_int(int value);\n\n/* None.proto */\n#if CYTHON_CCOMPLEX\n #ifdef __cplusplus\n #define __Pyx_CREAL(z) ((z).real())\n #define __Pyx_CIMAG(z) ((z).imag())\n #else\n #define __Pyx_CREAL(z) (__real__(z))\n #define __Pyx_CIMAG(z) (__imag__(z))\n #endif\n#else\n #define __Pyx_CREAL(z) ((z).real)\n #define __Pyx_CIMAG(z) ((z).imag)\n#endif\n#if defined(__cplusplus) && CYTHON_CCOMPLEX && (defined(_WIN32) || defined(__clang__) || (defined(__GNUC__) && (__GNUC__ >= 5 || __GNUC__ == 4 && __GNUC_MINOR__ >= 4 )) || __cplusplus >= 201103)\n #define __Pyx_SET_CREAL(z,x) ((z).real(x))\n #define __Pyx_SET_CIMAG(z,y) ((z).imag(y))\n#else\n #define __Pyx_SET_CREAL(z,x) __Pyx_CREAL(z) = (x)\n #define __Pyx_SET_CIMAG(z,y) __Pyx_CIMAG(z) = (y)\n#endif\n\n/* None.proto */\nstatic CYTHON_INLINE __pyx_t_float_complex __pyx_t_float_complex_from_parts(float, float);\n\n/* None.proto */\n#if CYTHON_CCOMPLEX\n #define __Pyx_c_eqf(a, b) ((a)==(b))\n #define __Pyx_c_sumf(a, b) ((a)+(b))\n #define __Pyx_c_difff(a, b) ((a)-(b))\n #define __Pyx_c_prodf(a, b) ((a)*(b))\n #define __Pyx_c_quotf(a, b) ((a)/(b))\n #define __Pyx_c_negf(a) (-(a))\n #ifdef __cplusplus\n #define __Pyx_c_is_zerof(z) ((z)==(float)0)\n #define __Pyx_c_conjf(z) (::std::conj(z))\n #if 1\n #define __Pyx_c_absf(z) (::std::abs(z))\n #define __Pyx_c_powf(a, b) (::std::pow(a, b))\n #endif\n #else\n #define __Pyx_c_is_zerof(z) ((z)==0)\n #define __Pyx_c_conjf(z) (conjf(z))\n #if 1\n #define __Pyx_c_absf(z) (cabsf(z))\n #define __Pyx_c_powf(a, b) (cpowf(a, b))\n #endif\n #endif\n#else\n static CYTHON_INLINE int __Pyx_c_eqf(__pyx_t_float_complex, __pyx_t_float_complex);\n static CYTHON_INLINE __pyx_t_float_complex __Pyx_c_sumf(__pyx_t_float_complex, __pyx_t_float_complex);\n static CYTHON_INLINE __pyx_t_float_complex __Pyx_c_difff(__pyx_t_float_complex, __pyx_t_float_complex);\n static CYTHON_INLINE __pyx_t_float_complex __Pyx_c_prodf(__pyx_t_float_complex, __pyx_t_float_complex);\n static CYTHON_INLINE __pyx_t_float_complex __Pyx_c_quotf(__pyx_t_float_complex, __pyx_t_float_complex);\n static CYTHON_INLINE __pyx_t_float_complex __Pyx_c_negf(__pyx_t_float_complex);\n static CYTHON_INLINE int __Pyx_c_is_zerof(__pyx_t_float_complex);\n static CYTHON_INLINE __pyx_t_float_complex __Pyx_c_conjf(__pyx_t_float_complex);\n #if 1\n static CYTHON_INLINE float __Pyx_c_absf(__pyx_t_float_complex);\n static CYTHON_INLINE __pyx_t_float_complex __Pyx_c_powf(__pyx_t_float_complex, __pyx_t_float_complex);\n #endif\n#endif\n\n/* None.proto */\nstatic CYTHON_INLINE __pyx_t_double_complex __pyx_t_double_complex_from_parts(double, double);\n\n/* None.proto */\n#if CYTHON_CCOMPLEX\n #define __Pyx_c_eq(a, b) ((a)==(b))\n #define __Pyx_c_sum(a, b) ((a)+(b))\n #define __Pyx_c_diff(a, b) ((a)-(b))\n #define __Pyx_c_prod(a, b) ((a)*(b))\n #define __Pyx_c_quot(a, b) ((a)/(b))\n #define __Pyx_c_neg(a) (-(a))\n #ifdef __cplusplus\n #define __Pyx_c_is_zero(z) ((z)==(double)0)\n #define __Pyx_c_conj(z) (::std::conj(z))\n #if 1\n #define __Pyx_c_abs(z) (::std::abs(z))\n #define __Pyx_c_pow(a, b) (::std::pow(a, b))\n #endif\n #else\n #define __Pyx_c_is_zero(z) ((z)==0)\n #define __Pyx_c_conj(z) (conj(z))\n #if 1\n #define __Pyx_c_abs(z) (cabs(z))\n #define __Pyx_c_pow(a, b) (cpow(a, b))\n #endif\n #endif\n#else\n static CYTHON_INLINE int __Pyx_c_eq(__pyx_t_double_complex, __pyx_t_double_complex);\n static CYTHON_INLINE __pyx_t_double_complex __Pyx_c_sum(__pyx_t_double_complex, __pyx_t_double_complex);\n static CYTHON_INLINE __pyx_t_double_complex __Pyx_c_diff(__pyx_t_double_complex, __pyx_t_double_complex);\n static CYTHON_INLINE __pyx_t_double_complex __Pyx_c_prod(__pyx_t_double_complex, __pyx_t_double_complex);\n static CYTHON_INLINE __pyx_t_double_complex __Pyx_c_quot(__pyx_t_double_complex, __pyx_t_double_complex);\n static CYTHON_INLINE __pyx_t_double_complex __Pyx_c_neg(__pyx_t_double_complex);\n static CYTHON_INLINE int __Pyx_c_is_zero(__pyx_t_double_complex);\n static CYTHON_INLINE __pyx_t_double_complex __Pyx_c_conj(__pyx_t_double_complex);\n #if 1\n static CYTHON_INLINE double __Pyx_c_abs(__pyx_t_double_complex);\n static CYTHON_INLINE __pyx_t_double_complex __Pyx_c_pow(__pyx_t_double_complex, __pyx_t_double_complex);\n #endif\n#endif\n\n/* CIntToPy.proto */\nstatic CYTHON_INLINE PyObject* __Pyx_PyInt_From_enum__NPY_TYPES(enum NPY_TYPES value);\n\n/* CIntFromPy.proto */\nstatic CYTHON_INLINE npy_int32 __Pyx_PyInt_As_npy_int32(PyObject *);\n\n/* CIntFromPy.proto */\nstatic CYTHON_INLINE int __Pyx_PyInt_As_int(PyObject *);\n\n/* CIntToPy.proto */\nstatic CYTHON_INLINE PyObject* __Pyx_PyInt_From_long(long value);\n\n/* CIntFromPy.proto */\nstatic CYTHON_INLINE long __Pyx_PyInt_As_long(PyObject *);\n\n/* CheckBinaryVersion.proto */\nstatic int __Pyx_check_binary_version(void);\n\n/* PyIdentifierFromString.proto */\n#if !defined(__Pyx_PyIdentifier_FromString)\n#if PY_MAJOR_VERSION < 3\n #define __Pyx_PyIdentifier_FromString(s) PyString_FromString(s)\n#else\n #define __Pyx_PyIdentifier_FromString(s) PyUnicode_FromString(s)\n#endif\n#endif\n\n/* ModuleImport.proto */\nstatic PyObject *__Pyx_ImportModule(const char *name);\n\n/* TypeImport.proto */\nstatic PyTypeObject *__Pyx_ImportType(const char *module_name, const char *class_name, size_t size, int strict);\n\n/* InitStrings.proto */\nstatic int __Pyx_InitStrings(__Pyx_StringTabEntry *t);\n\n\n/* Module declarations from 'cpython.buffer' */\n\n/* Module declarations from 'libc.string' */\n\n/* Module declarations from 'libc.stdio' */\n\n/* Module declarations from '__builtin__' */\n\n/* Module declarations from 'cpython.type' */\nstatic PyTypeObject *__pyx_ptype_7cpython_4type_type = 0;\n\n/* Module declarations from 'cpython' */\n\n/* Module declarations from 'cpython.object' */\n\n/* Module declarations from 'cpython.ref' */\n\n/* Module declarations from 'libc.stdlib' */\n\n/* Module declarations from 'numpy' */\n\n/* Module declarations from 'numpy' */\nstatic PyTypeObject *__pyx_ptype_5numpy_dtype = 0;\nstatic PyTypeObject *__pyx_ptype_5numpy_flatiter = 0;\nstatic PyTypeObject *__pyx_ptype_5numpy_broadcast = 0;\nstatic PyTypeObject *__pyx_ptype_5numpy_ndarray = 0;\nstatic PyTypeObject *__pyx_ptype_5numpy_ufunc = 0;\nstatic CYTHON_INLINE char *__pyx_f_5numpy__util_dtypestring(PyArray_Descr *, char *, char *, int *); /*proto*/\n\n/* Module declarations from 'nms.gpu_nms' */\nstatic __Pyx_TypeInfo __Pyx_TypeInfo_nn___pyx_t_5numpy_float32_t = { \"float32_t\", NULL, sizeof(__pyx_t_5numpy_float32_t), { 0 }, 0, 'R', 0, 0 };\nstatic __Pyx_TypeInfo __Pyx_TypeInfo_nn___pyx_t_5numpy_int32_t = { \"int32_t\", NULL, sizeof(__pyx_t_5numpy_int32_t), { 0 }, 0, IS_UNSIGNED(__pyx_t_5numpy_int32_t) ? 'U' : 'I', IS_UNSIGNED(__pyx_t_5numpy_int32_t), 0 };\nstatic __Pyx_TypeInfo __Pyx_TypeInfo_nn___pyx_t_5numpy_intp_t = { \"intp_t\", NULL, sizeof(__pyx_t_5numpy_intp_t), { 0 }, 0, IS_UNSIGNED(__pyx_t_5numpy_intp_t) ? 'U' : 'I', IS_UNSIGNED(__pyx_t_5numpy_intp_t), 0 };\n#define __Pyx_MODULE_NAME \"nms.gpu_nms\"\nint __pyx_module_is_main_nms__gpu_nms = 0;\n\n/* Implementation of 'nms.gpu_nms' */\nstatic PyObject *__pyx_builtin_ValueError;\nstatic PyObject *__pyx_builtin_range;\nstatic PyObject *__pyx_builtin_RuntimeError;\nstatic const char __pyx_k_np[] = \"np\";\nstatic const char __pyx_k_dets[] = \"dets\";\nstatic const char __pyx_k_keep[] = \"keep\";\nstatic const char __pyx_k_main[] = \"__main__\";\nstatic const char __pyx_k_test[] = \"__test__\";\nstatic const char __pyx_k_dtype[] = \"dtype\";\nstatic const char __pyx_k_int32[] = \"int32\";\nstatic const char __pyx_k_numpy[] = \"numpy\";\nstatic const char __pyx_k_order[] = \"order\";\nstatic const char __pyx_k_range[] = \"range\";\nstatic const char __pyx_k_zeros[] = \"zeros\";\nstatic const char __pyx_k_import[] = \"__import__\";\nstatic const char __pyx_k_scores[] = \"scores\";\nstatic const char __pyx_k_thresh[] = \"thresh\";\nstatic const char __pyx_k_argsort[] = \"argsort\";\nstatic const char __pyx_k_gpu_nms[] = \"gpu_nms\";\nstatic const char __pyx_k_num_out[] = \"num_out\";\nstatic const char __pyx_k_boxes_dim[] = \"boxes_dim\";\nstatic const char __pyx_k_boxes_num[] = \"boxes_num\";\nstatic const char __pyx_k_device_id[] = \"device_id\";\nstatic const char __pyx_k_ValueError[] = \"ValueError\";\nstatic const char __pyx_k_nms_gpu_nms[] = \"nms.gpu_nms\";\nstatic const char __pyx_k_sorted_dets[] = \"sorted_dets\";\nstatic const char __pyx_k_RuntimeError[] = \"RuntimeError\";\nstatic const char __pyx_k_ndarray_is_not_C_contiguous[] = \"ndarray is not C contiguous\";\nstatic const char __pyx_k_unknown_dtype_code_in_numpy_pxd[] = \"unknown dtype code in numpy.pxd (%d)\";\nstatic const char __pyx_k_D_v_zix_caffe_caffe_win_20160523[] = \"D:\\\\v-zix\\\\caffe\\\\caffe-win-20160523\\\\models\\\\py-faster-rcnn-windows\\\\lib\\\\nms\\\\gpu_nms.pyx\";\nstatic const char __pyx_k_Format_string_allocated_too_shor[] = \"Format string allocated too short, see comment in numpy.pxd\";\nstatic const char __pyx_k_Non_native_byte_order_not_suppor[] = \"Non-native byte order not supported\";\nstatic const char __pyx_k_ndarray_is_not_Fortran_contiguou[] = \"ndarray is not Fortran contiguous\";\nstatic const char __pyx_k_Format_string_allocated_too_shor_2[] = \"Format string allocated too short.\";\nstatic PyObject *__pyx_kp_s_D_v_zix_caffe_caffe_win_20160523;\nstatic PyObject *__pyx_kp_u_Format_string_allocated_too_shor;\nstatic PyObject *__pyx_kp_u_Format_string_allocated_too_shor_2;\nstatic PyObject *__pyx_kp_u_Non_native_byte_order_not_suppor;\nstatic PyObject *__pyx_n_s_RuntimeError;\nstatic PyObject *__pyx_n_s_ValueError;\nstatic PyObject *__pyx_n_s_argsort;\nstatic PyObject *__pyx_n_s_boxes_dim;\nstatic PyObject *__pyx_n_s_boxes_num;\nstatic PyObject *__pyx_n_s_dets;\nstatic PyObject *__pyx_n_s_device_id;\nstatic PyObject *__pyx_n_s_dtype;\nstatic PyObject *__pyx_n_s_gpu_nms;\nstatic PyObject *__pyx_n_s_import;\nstatic PyObject *__pyx_n_s_int32;\nstatic PyObject *__pyx_n_s_keep;\nstatic PyObject *__pyx_n_s_main;\nstatic PyObject *__pyx_kp_u_ndarray_is_not_C_contiguous;\nstatic PyObject *__pyx_kp_u_ndarray_is_not_Fortran_contiguou;\nstatic PyObject *__pyx_n_s_nms_gpu_nms;\nstatic PyObject *__pyx_n_s_np;\nstatic PyObject *__pyx_n_s_num_out;\nstatic PyObject *__pyx_n_s_numpy;\nstatic PyObject *__pyx_n_s_order;\nstatic PyObject *__pyx_n_s_range;\nstatic PyObject *__pyx_n_s_scores;\nstatic PyObject *__pyx_n_s_sorted_dets;\nstatic PyObject *__pyx_n_s_test;\nstatic PyObject *__pyx_n_s_thresh;\nstatic PyObject *__pyx_kp_u_unknown_dtype_code_in_numpy_pxd;\nstatic PyObject *__pyx_n_s_zeros;\nstatic PyObject *__pyx_pf_3nms_7gpu_nms_gpu_nms(CYTHON_UNUSED PyObject *__pyx_self, PyArrayObject *__pyx_v_dets, PyObject *__pyx_v_thresh, __pyx_t_5numpy_int32_t __pyx_v_device_id); /* proto */\nstatic int __pyx_pf_5numpy_7ndarray___getbuffer__(PyArrayObject *__pyx_v_self, Py_buffer *__pyx_v_info, int __pyx_v_flags); /* proto */\nstatic void __pyx_pf_5numpy_7ndarray_2__releasebuffer__(PyArrayObject *__pyx_v_self, Py_buffer *__pyx_v_info); /* proto */\nstatic PyObject *__pyx_int_4;\nstatic PyObject *__pyx_int_neg_1;\nstatic PyObject *__pyx_slice_;\nstatic PyObject *__pyx_slice__3;\nstatic PyObject *__pyx_slice__4;\nstatic PyObject *__pyx_tuple__2;\nstatic PyObject *__pyx_tuple__5;\nstatic PyObject *__pyx_tuple__6;\nstatic PyObject *__pyx_tuple__7;\nstatic PyObject *__pyx_tuple__8;\nstatic PyObject *__pyx_tuple__9;\nstatic PyObject *__pyx_tuple__10;\nstatic PyObject *__pyx_tuple__11;\nstatic PyObject *__pyx_codeobj__12;\n\n/* \"nms/gpu_nms.pyx\":16\n * void _nms(np.int32_t*, int*, np.float32_t*, int, int, float, int)\n * \n * def gpu_nms(np.ndarray[np.float32_t, ndim=2] dets, np.float thresh, # <<<<<<<<<<<<<<\n * np.int32_t device_id=0):\n * cdef int boxes_num = dets.shape[0]\n */\n\n/* Python wrapper */\nstatic PyObject *__pyx_pw_3nms_7gpu_nms_1gpu_nms(PyObject *__pyx_self, PyObject *__pyx_args, PyObject *__pyx_kwds); /*proto*/\nstatic PyMethodDef __pyx_mdef_3nms_7gpu_nms_1gpu_nms = {\"gpu_nms\", (PyCFunction)__pyx_pw_3nms_7gpu_nms_1gpu_nms, METH_VARARGS|METH_KEYWORDS, 0};\nstatic PyObject *__pyx_pw_3nms_7gpu_nms_1gpu_nms(PyObject *__pyx_self, PyObject *__pyx_args, PyObject *__pyx_kwds) {\n PyArrayObject *__pyx_v_dets = 0;\n PyObject *__pyx_v_thresh = 0;\n __pyx_t_5numpy_int32_t __pyx_v_device_id;\n PyObject *__pyx_r = 0;\n __Pyx_RefNannyDeclarations\n __Pyx_RefNannySetupContext(\"gpu_nms (wrapper)\", 0);\n {\n static PyObject **__pyx_pyargnames[] = {&__pyx_n_s_dets,&__pyx_n_s_thresh,&__pyx_n_s_device_id,0};\n PyObject* values[3] = {0,0,0};\n if (unlikely(__pyx_kwds)) {\n Py_ssize_t kw_args;\n const Py_ssize_t pos_args = PyTuple_GET_SIZE(__pyx_args);\n switch (pos_args) {\n case 3: values[2] = PyTuple_GET_ITEM(__pyx_args, 2);\n case 2: values[1] = PyTuple_GET_ITEM(__pyx_args, 1);\n case 1: values[0] = PyTuple_GET_ITEM(__pyx_args, 0);\n case 0: break;\n default: goto __pyx_L5_argtuple_error;\n }\n kw_args = PyDict_Size(__pyx_kwds);\n switch (pos_args) {\n case 0:\n if (likely((values[0] = PyDict_GetItem(__pyx_kwds, __pyx_n_s_dets)) != 0)) kw_args--;\n else goto __pyx_L5_argtuple_error;\n case 1:\n if (likely((values[1] = PyDict_GetItem(__pyx_kwds, __pyx_n_s_thresh)) != 0)) kw_args--;\n else {\n __Pyx_RaiseArgtupleInvalid(\"gpu_nms\", 0, 2, 3, 1); __PYX_ERR(0, 16, __pyx_L3_error)\n }\n case 2:\n if (kw_args > 0) {\n PyObject* value = PyDict_GetItem(__pyx_kwds, __pyx_n_s_device_id);\n if (value) { values[2] = value; kw_args--; }\n }\n }\n if (unlikely(kw_args > 0)) {\n if (unlikely(__Pyx_ParseOptionalKeywords(__pyx_kwds, __pyx_pyargnames, 0, values, pos_args, \"gpu_nms\") < 0)) __PYX_ERR(0, 16, __pyx_L3_error)\n }\n } else {\n switch (PyTuple_GET_SIZE(__pyx_args)) {\n case 3: values[2] = PyTuple_GET_ITEM(__pyx_args, 2);\n case 2: values[1] = PyTuple_GET_ITEM(__pyx_args, 1);\n values[0] = PyTuple_GET_ITEM(__pyx_args, 0);\n break;\n default: goto __pyx_L5_argtuple_error;\n }\n }\n __pyx_v_dets = ((PyArrayObject *)values[0]);\n __pyx_v_thresh = ((PyObject*)values[1]);\n if (values[2]) {\n __pyx_v_device_id = __Pyx_PyInt_As_npy_int32(values[2]); if (unlikely((__pyx_v_device_id == (npy_int32)-1) && PyErr_Occurred())) __PYX_ERR(0, 17, __pyx_L3_error)\n } else {\n __pyx_v_device_id = ((__pyx_t_5numpy_int32_t)0);\n }\n }\n goto __pyx_L4_argument_unpacking_done;\n __pyx_L5_argtuple_error:;\n __Pyx_RaiseArgtupleInvalid(\"gpu_nms\", 0, 2, 3, PyTuple_GET_SIZE(__pyx_args)); __PYX_ERR(0, 16, __pyx_L3_error)\n __pyx_L3_error:;\n __Pyx_AddTraceback(\"nms.gpu_nms.gpu_nms\", __pyx_clineno, __pyx_lineno, __pyx_filename);\n __Pyx_RefNannyFinishContext();\n return NULL;\n __pyx_L4_argument_unpacking_done:;\n if (unlikely(!__Pyx_ArgTypeTest(((PyObject *)__pyx_v_dets), __pyx_ptype_5numpy_ndarray, 1, \"dets\", 0))) __PYX_ERR(0, 16, __pyx_L1_error)\n if (unlikely(!__Pyx_ArgTypeTest(((PyObject *)__pyx_v_thresh), (&PyFloat_Type), 1, \"thresh\", 1))) __PYX_ERR(0, 16, __pyx_L1_error)\n __pyx_r = __pyx_pf_3nms_7gpu_nms_gpu_nms(__pyx_self, __pyx_v_dets, __pyx_v_thresh, __pyx_v_device_id);\n\n /* function exit code */\n goto __pyx_L0;\n __pyx_L1_error:;\n __pyx_r = NULL;\n __pyx_L0:;\n __Pyx_RefNannyFinishContext();\n return __pyx_r;\n}\n\nstatic PyObject *__pyx_pf_3nms_7gpu_nms_gpu_nms(CYTHON_UNUSED PyObject *__pyx_self, PyArrayObject *__pyx_v_dets, PyObject *__pyx_v_thresh, __pyx_t_5numpy_int32_t __pyx_v_device_id) {\n int __pyx_v_boxes_num;\n int __pyx_v_boxes_dim;\n int __pyx_v_num_out;\n PyArrayObject *__pyx_v_keep = 0;\n PyArrayObject *__pyx_v_scores = 0;\n PyArrayObject *__pyx_v_order = 0;\n PyArrayObject *__pyx_v_sorted_dets = 0;\n __Pyx_LocalBuf_ND __pyx_pybuffernd_dets;\n __Pyx_Buffer __pyx_pybuffer_dets;\n __Pyx_LocalBuf_ND __pyx_pybuffernd_keep;\n __Pyx_Buffer __pyx_pybuffer_keep;\n __Pyx_LocalBuf_ND __pyx_pybuffernd_order;\n __Pyx_Buffer __pyx_pybuffer_order;\n __Pyx_LocalBuf_ND __pyx_pybuffernd_scores;\n __Pyx_Buffer __pyx_pybuffer_scores;\n __Pyx_LocalBuf_ND __pyx_pybuffernd_sorted_dets;\n __Pyx_Buffer __pyx_pybuffer_sorted_dets;\n PyObject *__pyx_r = NULL;\n __Pyx_RefNannyDeclarations\n PyObject *__pyx_t_1 = NULL;\n PyObject *__pyx_t_2 = NULL;\n PyObject *__pyx_t_3 = NULL;\n PyObject *__pyx_t_4 = NULL;\n PyObject *__pyx_t_5 = NULL;\n PyArrayObject *__pyx_t_6 = NULL;\n PyArrayObject *__pyx_t_7 = NULL;\n PyArrayObject *__pyx_t_8 = NULL;\n PyArrayObject *__pyx_t_9 = NULL;\n Py_ssize_t __pyx_t_10;\n int __pyx_t_11;\n Py_ssize_t __pyx_t_12;\n Py_ssize_t __pyx_t_13;\n float __pyx_t_14;\n PyObject *__pyx_t_15 = NULL;\n PyObject *__pyx_t_16 = NULL;\n PyObject *__pyx_t_17 = NULL;\n __Pyx_RefNannySetupContext(\"gpu_nms\", 0);\n __pyx_pybuffer_keep.pybuffer.buf = NULL;\n __pyx_pybuffer_keep.refcount = 0;\n __pyx_pybuffernd_keep.data = NULL;\n __pyx_pybuffernd_keep.rcbuffer = &__pyx_pybuffer_keep;\n __pyx_pybuffer_scores.pybuffer.buf = NULL;\n __pyx_pybuffer_scores.refcount = 0;\n __pyx_pybuffernd_scores.data = NULL;\n __pyx_pybuffernd_scores.rcbuffer = &__pyx_pybuffer_scores;\n __pyx_pybuffer_order.pybuffer.buf = NULL;\n __pyx_pybuffer_order.refcount = 0;\n __pyx_pybuffernd_order.data = NULL;\n __pyx_pybuffernd_order.rcbuffer = &__pyx_pybuffer_order;\n __pyx_pybuffer_sorted_dets.pybuffer.buf = NULL;\n __pyx_pybuffer_sorted_dets.refcount = 0;\n __pyx_pybuffernd_sorted_dets.data = NULL;\n __pyx_pybuffernd_sorted_dets.rcbuffer = &__pyx_pybuffer_sorted_dets;\n __pyx_pybuffer_dets.pybuffer.buf = NULL;\n __pyx_pybuffer_dets.refcount = 0;\n __pyx_pybuffernd_dets.data = NULL;\n __pyx_pybuffernd_dets.rcbuffer = &__pyx_pybuffer_dets;\n {\n __Pyx_BufFmt_StackElem __pyx_stack[1];\n if (unlikely(__Pyx_GetBufferAndValidate(&__pyx_pybuffernd_dets.rcbuffer->pybuffer, (PyObject*)__pyx_v_dets, &__Pyx_TypeInfo_nn___pyx_t_5numpy_float32_t, PyBUF_FORMAT| PyBUF_STRIDES, 2, 0, __pyx_stack) == -1)) __PYX_ERR(0, 16, __pyx_L1_error)\n }\n __pyx_pybuffernd_dets.diminfo[0].strides = __pyx_pybuffernd_dets.rcbuffer->pybuffer.strides[0]; __pyx_pybuffernd_dets.diminfo[0].shape = __pyx_pybuffernd_dets.rcbuffer->pybuffer.shape[0]; __pyx_pybuffernd_dets.diminfo[1].strides = __pyx_pybuffernd_dets.rcbuffer->pybuffer.strides[1]; __pyx_pybuffernd_dets.diminfo[1].shape = __pyx_pybuffernd_dets.rcbuffer->pybuffer.shape[1];\n\n /* \"nms/gpu_nms.pyx\":18\n * def gpu_nms(np.ndarray[np.float32_t, ndim=2] dets, np.float thresh,\n * np.int32_t device_id=0):\n * cdef int boxes_num = dets.shape[0] # <<<<<<<<<<<<<<\n * cdef int boxes_dim = dets.shape[1]\n * cdef int num_out\n */\n __pyx_v_boxes_num = (__pyx_v_dets->dimensions[0]);\n\n /* \"nms/gpu_nms.pyx\":19\n * np.int32_t device_id=0):\n * cdef int boxes_num = dets.shape[0]\n * cdef int boxes_dim = dets.shape[1] # <<<<<<<<<<<<<<\n * cdef int num_out\n * cdef np.ndarray[np.int32_t, ndim=1] \\\n */\n __pyx_v_boxes_dim = (__pyx_v_dets->dimensions[1]);\n\n /* \"nms/gpu_nms.pyx\":22\n * cdef int num_out\n * cdef np.ndarray[np.int32_t, ndim=1] \\\n * keep = np.zeros(boxes_num, dtype=np.int32) # <<<<<<<<<<<<<<\n * cdef np.ndarray[np.float32_t, ndim=1] \\\n * scores = dets[:, 4]\n */\n __pyx_t_1 = __Pyx_GetModuleGlobalName(__pyx_n_s_np); if (unlikely(!__pyx_t_1)) __PYX_ERR(0, 22, __pyx_L1_error)\n __Pyx_GOTREF(__pyx_t_1);\n __pyx_t_2 = __Pyx_PyObject_GetAttrStr(__pyx_t_1, __pyx_n_s_zeros); if (unlikely(!__pyx_t_2)) __PYX_ERR(0, 22, __pyx_L1_error)\n __Pyx_GOTREF(__pyx_t_2);\n __Pyx_DECREF(__pyx_t_1); __pyx_t_1 = 0;\n __pyx_t_1 = __Pyx_PyInt_From_int(__pyx_v_boxes_num); if (unlikely(!__pyx_t_1)) __PYX_ERR(0, 22, __pyx_L1_error)\n __Pyx_GOTREF(__pyx_t_1);\n __pyx_t_3 = PyTuple_New(1); if (unlikely(!__pyx_t_3)) __PYX_ERR(0, 22, __pyx_L1_error)\n __Pyx_GOTREF(__pyx_t_3);\n __Pyx_GIVEREF(__pyx_t_1);\n PyTuple_SET_ITEM(__pyx_t_3, 0, __pyx_t_1);\n __pyx_t_1 = 0;\n __pyx_t_1 = PyDict_New(); if (unlikely(!__pyx_t_1)) __PYX_ERR(0, 22, __pyx_L1_error)\n __Pyx_GOTREF(__pyx_t_1);\n __pyx_t_4 = __Pyx_GetModuleGlobalName(__pyx_n_s_np); if (unlikely(!__pyx_t_4)) __PYX_ERR(0, 22, __pyx_L1_error)\n __Pyx_GOTREF(__pyx_t_4);\n __pyx_t_5 = __Pyx_PyObject_GetAttrStr(__pyx_t_4, __pyx_n_s_int32); if (unlikely(!__pyx_t_5)) __PYX_ERR(0, 22, __pyx_L1_error)\n __Pyx_GOTREF(__pyx_t_5);\n __Pyx_DECREF(__pyx_t_4); __pyx_t_4 = 0;\n if (PyDict_SetItem(__pyx_t_1, __pyx_n_s_dtype, __pyx_t_5) < 0) __PYX_ERR(0, 22, __pyx_L1_error)\n __Pyx_DECREF(__pyx_t_5); __pyx_t_5 = 0;\n __pyx_t_5 = __Pyx_PyObject_Call(__pyx_t_2, __pyx_t_3, __pyx_t_1); if (unlikely(!__pyx_t_5)) __PYX_ERR(0, 22, __pyx_L1_error)\n __Pyx_GOTREF(__pyx_t_5);\n __Pyx_DECREF(__pyx_t_2); __pyx_t_2 = 0;\n __Pyx_DECREF(__pyx_t_3); __pyx_t_3 = 0;\n __Pyx_DECREF(__pyx_t_1); __pyx_t_1 = 0;\n if (!(likely(((__pyx_t_5) == Py_None) || likely(__Pyx_TypeTest(__pyx_t_5, __pyx_ptype_5numpy_ndarray))))) __PYX_ERR(0, 22, __pyx_L1_error)\n __pyx_t_6 = ((PyArrayObject *)__pyx_t_5);\n {\n __Pyx_BufFmt_StackElem __pyx_stack[1];\n if (unlikely(__Pyx_GetBufferAndValidate(&__pyx_pybuffernd_keep.rcbuffer->pybuffer, (PyObject*)__pyx_t_6, &__Pyx_TypeInfo_nn___pyx_t_5numpy_int32_t, PyBUF_FORMAT| PyBUF_STRIDES, 1, 0, __pyx_stack) == -1)) {\n __pyx_v_keep = ((PyArrayObject *)Py_None); __Pyx_INCREF(Py_None); __pyx_pybuffernd_keep.rcbuffer->pybuffer.buf = NULL;\n __PYX_ERR(0, 21, __pyx_L1_error)\n } else {__pyx_pybuffernd_keep.diminfo[0].strides = __pyx_pybuffernd_keep.rcbuffer->pybuffer.strides[0]; __pyx_pybuffernd_keep.diminfo[0].shape = __pyx_pybuffernd_keep.rcbuffer->pybuffer.shape[0];\n }\n }\n __pyx_t_6 = 0;\n __pyx_v_keep = ((PyArrayObject *)__pyx_t_5);\n __pyx_t_5 = 0;\n\n /* \"nms/gpu_nms.pyx\":24\n * keep = np.zeros(boxes_num, dtype=np.int32)\n * cdef np.ndarray[np.float32_t, ndim=1] \\\n * scores = dets[:, 4] # <<<<<<<<<<<<<<\n * #cdef np.ndarray[np.int_t, ndim=1] \\ // 20160601, by xzn\n * # order = scores.argsort()[::-1]\n */\n __pyx_t_5 = PyObject_GetItem(((PyObject *)__pyx_v_dets), __pyx_tuple__2); if (unlikely(!__pyx_t_5)) __PYX_ERR(0, 24, __pyx_L1_error)\n __Pyx_GOTREF(__pyx_t_5);\n if (!(likely(((__pyx_t_5) == Py_None) || likely(__Pyx_TypeTest(__pyx_t_5, __pyx_ptype_5numpy_ndarray))))) __PYX_ERR(0, 24, __pyx_L1_error)\n __pyx_t_7 = ((PyArrayObject *)__pyx_t_5);\n {\n __Pyx_BufFmt_StackElem __pyx_stack[1];\n if (unlikely(__Pyx_GetBufferAndValidate(&__pyx_pybuffernd_scores.rcbuffer->pybuffer, (PyObject*)__pyx_t_7, &__Pyx_TypeInfo_nn___pyx_t_5numpy_float32_t, PyBUF_FORMAT| PyBUF_STRIDES, 1, 0, __pyx_stack) == -1)) {\n __pyx_v_scores = ((PyArrayObject *)Py_None); __Pyx_INCREF(Py_None); __pyx_pybuffernd_scores.rcbuffer->pybuffer.buf = NULL;\n __PYX_ERR(0, 23, __pyx_L1_error)\n } else {__pyx_pybuffernd_scores.diminfo[0].strides = __pyx_pybuffernd_scores.rcbuffer->pybuffer.strides[0]; __pyx_pybuffernd_scores.diminfo[0].shape = __pyx_pybuffernd_scores.rcbuffer->pybuffer.shape[0];\n }\n }\n __pyx_t_7 = 0;\n __pyx_v_scores = ((PyArrayObject *)__pyx_t_5);\n __pyx_t_5 = 0;\n\n /* \"nms/gpu_nms.pyx\":28\n * # order = scores.argsort()[::-1]\n * cdef np.ndarray[np.intp_t, ndim=1] \\\n * order = scores.argsort()[::-1] # <<<<<<<<<<<<<<\n * cdef np.ndarray[np.float32_t, ndim=2] \\\n * sorted_dets = dets[order, :]\n */\n __pyx_t_1 = __Pyx_PyObject_GetAttrStr(((PyObject *)__pyx_v_scores), __pyx_n_s_argsort); if (unlikely(!__pyx_t_1)) __PYX_ERR(0, 28, __pyx_L1_error)\n __Pyx_GOTREF(__pyx_t_1);\n __pyx_t_3 = NULL;\n if (CYTHON_COMPILING_IN_CPYTHON && likely(PyMethod_Check(__pyx_t_1))) {\n __pyx_t_3 = PyMethod_GET_SELF(__pyx_t_1);\n if (likely(__pyx_t_3)) {\n PyObject* function = PyMethod_GET_FUNCTION(__pyx_t_1);\n __Pyx_INCREF(__pyx_t_3);\n __Pyx_INCREF(function);\n __Pyx_DECREF_SET(__pyx_t_1, function);\n }\n }\n if (__pyx_t_3) {\n __pyx_t_5 = __Pyx_PyObject_CallOneArg(__pyx_t_1, __pyx_t_3); if (unlikely(!__pyx_t_5)) __PYX_ERR(0, 28, __pyx_L1_error)\n __Pyx_DECREF(__pyx_t_3); __pyx_t_3 = 0;\n } else {\n __pyx_t_5 = __Pyx_PyObject_CallNoArg(__pyx_t_1); if (unlikely(!__pyx_t_5)) __PYX_ERR(0, 28, __pyx_L1_error)\n }\n __Pyx_GOTREF(__pyx_t_5);\n __Pyx_DECREF(__pyx_t_1); __pyx_t_1 = 0;\n __pyx_t_1 = PyObject_GetItem(__pyx_t_5, __pyx_slice__3); if (unlikely(!__pyx_t_1)) __PYX_ERR(0, 28, __pyx_L1_error)\n __Pyx_GOTREF(__pyx_t_1);\n __Pyx_DECREF(__pyx_t_5); __pyx_t_5 = 0;\n if (!(likely(((__pyx_t_1) == Py_None) || likely(__Pyx_TypeTest(__pyx_t_1, __pyx_ptype_5numpy_ndarray))))) __PYX_ERR(0, 28, __pyx_L1_error)\n __pyx_t_8 = ((PyArrayObject *)__pyx_t_1);\n {\n __Pyx_BufFmt_StackElem __pyx_stack[1];\n if (unlikely(__Pyx_GetBufferAndValidate(&__pyx_pybuffernd_order.rcbuffer->pybuffer, (PyObject*)__pyx_t_8, &__Pyx_TypeInfo_nn___pyx_t_5numpy_intp_t, PyBUF_FORMAT| PyBUF_STRIDES, 1, 0, __pyx_stack) == -1)) {\n __pyx_v_order = ((PyArrayObject *)Py_None); __Pyx_INCREF(Py_None); __pyx_pybuffernd_order.rcbuffer->pybuffer.buf = NULL;\n __PYX_ERR(0, 27, __pyx_L1_error)\n } else {__pyx_pybuffernd_order.diminfo[0].strides = __pyx_pybuffernd_order.rcbuffer->pybuffer.strides[0]; __pyx_pybuffernd_order.diminfo[0].shape = __pyx_pybuffernd_order.rcbuffer->pybuffer.shape[0];\n }\n }\n __pyx_t_8 = 0;\n __pyx_v_order = ((PyArrayObject *)__pyx_t_1);\n __pyx_t_1 = 0;\n\n /* \"nms/gpu_nms.pyx\":30\n * order = scores.argsort()[::-1]\n * cdef np.ndarray[np.float32_t, ndim=2] \\\n * sorted_dets = dets[order, :] # <<<<<<<<<<<<<<\n * _nms(&keep[0], &num_out, &sorted_dets[0, 0], boxes_num, boxes_dim, thresh, device_id)\n * keep = keep[:num_out]\n */\n __pyx_t_1 = PyTuple_New(2); if (unlikely(!__pyx_t_1)) __PYX_ERR(0, 30, __pyx_L1_error)\n __Pyx_GOTREF(__pyx_t_1);\n __Pyx_INCREF(((PyObject *)__pyx_v_order));\n __Pyx_GIVEREF(((PyObject *)__pyx_v_order));\n PyTuple_SET_ITEM(__pyx_t_1, 0, ((PyObject *)__pyx_v_order));\n __Pyx_INCREF(__pyx_slice__4);\n __Pyx_GIVEREF(__pyx_slice__4);\n PyTuple_SET_ITEM(__pyx_t_1, 1, __pyx_slice__4);\n __pyx_t_5 = PyObject_GetItem(((PyObject *)__pyx_v_dets), __pyx_t_1); if (unlikely(!__pyx_t_5)) __PYX_ERR(0, 30, __pyx_L1_error)\n __Pyx_GOTREF(__pyx_t_5);\n __Pyx_DECREF(__pyx_t_1); __pyx_t_1 = 0;\n if (!(likely(((__pyx_t_5) == Py_None) || likely(__Pyx_TypeTest(__pyx_t_5, __pyx_ptype_5numpy_ndarray))))) __PYX_ERR(0, 30, __pyx_L1_error)\n __pyx_t_9 = ((PyArrayObject *)__pyx_t_5);\n {\n __Pyx_BufFmt_StackElem __pyx_stack[1];\n if (unlikely(__Pyx_GetBufferAndValidate(&__pyx_pybuffernd_sorted_dets.rcbuffer->pybuffer, (PyObject*)__pyx_t_9, &__Pyx_TypeInfo_nn___pyx_t_5numpy_float32_t, PyBUF_FORMAT| PyBUF_STRIDES, 2, 0, __pyx_stack) == -1)) {\n __pyx_v_sorted_dets = ((PyArrayObject *)Py_None); __Pyx_INCREF(Py_None); __pyx_pybuffernd_sorted_dets.rcbuffer->pybuffer.buf = NULL;\n __PYX_ERR(0, 29, __pyx_L1_error)\n } else {__pyx_pybuffernd_sorted_dets.diminfo[0].strides = __pyx_pybuffernd_sorted_dets.rcbuffer->pybuffer.strides[0]; __pyx_pybuffernd_sorted_dets.diminfo[0].shape = __pyx_pybuffernd_sorted_dets.rcbuffer->pybuffer.shape[0]; __pyx_pybuffernd_sorted_dets.diminfo[1].strides = __pyx_pybuffernd_sorted_dets.rcbuffer->pybuffer.strides[1]; __pyx_pybuffernd_sorted_dets.diminfo[1].shape = __pyx_pybuffernd_sorted_dets.rcbuffer->pybuffer.shape[1];\n }\n }\n __pyx_t_9 = 0;\n __pyx_v_sorted_dets = ((PyArrayObject *)__pyx_t_5);\n __pyx_t_5 = 0;\n\n /* \"nms/gpu_nms.pyx\":31\n * cdef np.ndarray[np.float32_t, ndim=2] \\\n * sorted_dets = dets[order, :]\n * _nms(&keep[0], &num_out, &sorted_dets[0, 0], boxes_num, boxes_dim, thresh, device_id) # <<<<<<<<<<<<<<\n * keep = keep[:num_out]\n * return list(order[keep])\n */\n __pyx_t_10 = 0;\n __pyx_t_11 = -1;\n if (__pyx_t_10 < 0) {\n __pyx_t_10 += __pyx_pybuffernd_keep.diminfo[0].shape;\n if (unlikely(__pyx_t_10 < 0)) __pyx_t_11 = 0;\n } else if (unlikely(__pyx_t_10 >= __pyx_pybuffernd_keep.diminfo[0].shape)) __pyx_t_11 = 0;\n if (unlikely(__pyx_t_11 != -1)) {\n __Pyx_RaiseBufferIndexError(__pyx_t_11);\n __PYX_ERR(0, 31, __pyx_L1_error)\n }\n __pyx_t_12 = 0;\n __pyx_t_13 = 0;\n __pyx_t_11 = -1;\n if (__pyx_t_12 < 0) {\n __pyx_t_12 += __pyx_pybuffernd_sorted_dets.diminfo[0].shape;\n if (unlikely(__pyx_t_12 < 0)) __pyx_t_11 = 0;\n } else if (unlikely(__pyx_t_12 >= __pyx_pybuffernd_sorted_dets.diminfo[0].shape)) __pyx_t_11 = 0;\n if (__pyx_t_13 < 0) {\n __pyx_t_13 += __pyx_pybuffernd_sorted_dets.diminfo[1].shape;\n if (unlikely(__pyx_t_13 < 0)) __pyx_t_11 = 1;\n } else if (unlikely(__pyx_t_13 >= __pyx_pybuffernd_sorted_dets.diminfo[1].shape)) __pyx_t_11 = 1;\n if (unlikely(__pyx_t_11 != -1)) {\n __Pyx_RaiseBufferIndexError(__pyx_t_11);\n __PYX_ERR(0, 31, __pyx_L1_error)\n }\n __pyx_t_14 = __pyx_PyFloat_AsFloat(__pyx_v_thresh); if (unlikely((__pyx_t_14 == (float)-1) && PyErr_Occurred())) __PYX_ERR(0, 31, __pyx_L1_error)\n _nms((&(*__Pyx_BufPtrStrided1d(__pyx_t_5numpy_int32_t *, __pyx_pybuffernd_keep.rcbuffer->pybuffer.buf, __pyx_t_10, __pyx_pybuffernd_keep.diminfo[0].strides))), (&__pyx_v_num_out), (&(*__Pyx_BufPtrStrided2d(__pyx_t_5numpy_float32_t *, __pyx_pybuffernd_sorted_dets.rcbuffer->pybuffer.buf, __pyx_t_12, __pyx_pybuffernd_sorted_dets.diminfo[0].strides, __pyx_t_13, __pyx_pybuffernd_sorted_dets.diminfo[1].strides))), __pyx_v_boxes_num, __pyx_v_boxes_dim, __pyx_t_14, __pyx_v_device_id);\n\n /* \"nms/gpu_nms.pyx\":32\n * sorted_dets = dets[order, :]\n * _nms(&keep[0], &num_out, &sorted_dets[0, 0], boxes_num, boxes_dim, thresh, device_id)\n * keep = keep[:num_out] # <<<<<<<<<<<<<<\n * return list(order[keep])\n */\n __pyx_t_5 = __Pyx_PyObject_GetSlice(((PyObject *)__pyx_v_keep), 0, __pyx_v_num_out, NULL, NULL, NULL, 0, 1, 1); if (unlikely(!__pyx_t_5)) __PYX_ERR(0, 32, __pyx_L1_error)\n __Pyx_GOTREF(__pyx_t_5);\n if (!(likely(((__pyx_t_5) == Py_None) || likely(__Pyx_TypeTest(__pyx_t_5, __pyx_ptype_5numpy_ndarray))))) __PYX_ERR(0, 32, __pyx_L1_error)\n __pyx_t_6 = ((PyArrayObject *)__pyx_t_5);\n {\n __Pyx_BufFmt_StackElem __pyx_stack[1];\n __Pyx_SafeReleaseBuffer(&__pyx_pybuffernd_keep.rcbuffer->pybuffer);\n __pyx_t_11 = __Pyx_GetBufferAndValidate(&__pyx_pybuffernd_keep.rcbuffer->pybuffer, (PyObject*)__pyx_t_6, &__Pyx_TypeInfo_nn___pyx_t_5numpy_int32_t, PyBUF_FORMAT| PyBUF_STRIDES, 1, 0, __pyx_stack);\n if (unlikely(__pyx_t_11 < 0)) {\n PyErr_Fetch(&__pyx_t_15, &__pyx_t_16, &__pyx_t_17);\n if (unlikely(__Pyx_GetBufferAndValidate(&__pyx_pybuffernd_keep.rcbuffer->pybuffer, (PyObject*)__pyx_v_keep, &__Pyx_TypeInfo_nn___pyx_t_5numpy_int32_t, PyBUF_FORMAT| PyBUF_STRIDES, 1, 0, __pyx_stack) == -1)) {\n Py_XDECREF(__pyx_t_15); Py_XDECREF(__pyx_t_16); Py_XDECREF(__pyx_t_17);\n __Pyx_RaiseBufferFallbackError();\n } else {\n PyErr_Restore(__pyx_t_15, __pyx_t_16, __pyx_t_17);\n }\n }\n __pyx_pybuffernd_keep.diminfo[0].strides = __pyx_pybuffernd_keep.rcbuffer->pybuffer.strides[0]; __pyx_pybuffernd_keep.diminfo[0].shape = __pyx_pybuffernd_keep.rcbuffer->pybuffer.shape[0];\n if (unlikely(__pyx_t_11 < 0)) __PYX_ERR(0, 32, __pyx_L1_error)\n }\n __pyx_t_6 = 0;\n __Pyx_DECREF_SET(__pyx_v_keep, ((PyArrayObject *)__pyx_t_5));\n __pyx_t_5 = 0;\n\n /* \"nms/gpu_nms.pyx\":33\n * _nms(&keep[0], &num_out, &sorted_dets[0, 0], boxes_num, boxes_dim, thresh, device_id)\n * keep = keep[:num_out]\n * return list(order[keep]) # <<<<<<<<<<<<<<\n */\n __Pyx_XDECREF(__pyx_r);\n __pyx_t_5 = PyObject_GetItem(((PyObject *)__pyx_v_order), ((PyObject *)__pyx_v_keep)); if (unlikely(!__pyx_t_5)) __PYX_ERR(0, 33, __pyx_L1_error)\n __Pyx_GOTREF(__pyx_t_5);\n __pyx_t_1 = PySequence_List(__pyx_t_5); if (unlikely(!__pyx_t_1)) __PYX_ERR(0, 33, __pyx_L1_error)\n __Pyx_GOTREF(__pyx_t_1);\n __Pyx_DECREF(__pyx_t_5); __pyx_t_5 = 0;\n __pyx_r = __pyx_t_1;\n __pyx_t_1 = 0;\n goto __pyx_L0;\n\n /* \"nms/gpu_nms.pyx\":16\n * void _nms(np.int32_t*, int*, np.float32_t*, int, int, float, int)\n * \n * def gpu_nms(np.ndarray[np.float32_t, ndim=2] dets, np.float thresh, # <<<<<<<<<<<<<<\n * np.int32_t device_id=0):\n * cdef int boxes_num = dets.shape[0]\n */\n\n /* function exit code */\n __pyx_L1_error:;\n __Pyx_XDECREF(__pyx_t_1);\n __Pyx_XDECREF(__pyx_t_2);\n __Pyx_XDECREF(__pyx_t_3);\n __Pyx_XDECREF(__pyx_t_4);\n __Pyx_XDECREF(__pyx_t_5);\n { PyObject *__pyx_type, *__pyx_value, *__pyx_tb;\n __Pyx_PyThreadState_declare\n __Pyx_PyThreadState_assign\n __Pyx_ErrFetch(&__pyx_type, &__pyx_value, &__pyx_tb);\n __Pyx_SafeReleaseBuffer(&__pyx_pybuffernd_dets.rcbuffer->pybuffer);\n __Pyx_SafeReleaseBuffer(&__pyx_pybuffernd_keep.rcbuffer->pybuffer);\n __Pyx_SafeReleaseBuffer(&__pyx_pybuffernd_order.rcbuffer->pybuffer);\n __Pyx_SafeReleaseBuffer(&__pyx_pybuffernd_scores.rcbuffer->pybuffer);\n __Pyx_SafeReleaseBuffer(&__pyx_pybuffernd_sorted_dets.rcbuffer->pybuffer);\n __Pyx_ErrRestore(__pyx_type, __pyx_value, __pyx_tb);}\n __Pyx_AddTraceback(\"nms.gpu_nms.gpu_nms\", __pyx_clineno, __pyx_lineno, __pyx_filename);\n __pyx_r = NULL;\n goto __pyx_L2;\n __pyx_L0:;\n __Pyx_SafeReleaseBuffer(&__pyx_pybuffernd_dets.rcbuffer->pybuffer);\n __Pyx_SafeReleaseBuffer(&__pyx_pybuffernd_keep.rcbuffer->pybuffer);\n __Pyx_SafeReleaseBuffer(&__pyx_pybuffernd_order.rcbuffer->pybuffer);\n __Pyx_SafeReleaseBuffer(&__pyx_pybuffernd_scores.rcbuffer->pybuffer);\n __Pyx_SafeReleaseBuffer(&__pyx_pybuffernd_sorted_dets.rcbuffer->pybuffer);\n __pyx_L2:;\n __Pyx_XDECREF((PyObject *)__pyx_v_keep);\n __Pyx_XDECREF((PyObject *)__pyx_v_scores);\n __Pyx_XDECREF((PyObject *)__pyx_v_order);\n __Pyx_XDECREF((PyObject *)__pyx_v_sorted_dets);\n __Pyx_XGIVEREF(__pyx_r);\n __Pyx_RefNannyFinishContext();\n return __pyx_r;\n}\n\n/* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":197\n * # experimental exception made for __getbuffer__ and __releasebuffer__\n * # -- the details of this may change.\n * def __getbuffer__(ndarray self, Py_buffer* info, int flags): # <<<<<<<<<<<<<<\n * # This implementation of getbuffer is geared towards Cython\n * # requirements, and does not yet fullfill the PEP.\n */\n\n/* Python wrapper */\nstatic CYTHON_UNUSED int __pyx_pw_5numpy_7ndarray_1__getbuffer__(PyObject *__pyx_v_self, Py_buffer *__pyx_v_info, int __pyx_v_flags); /*proto*/\nstatic CYTHON_UNUSED int __pyx_pw_5numpy_7ndarray_1__getbuffer__(PyObject *__pyx_v_self, Py_buffer *__pyx_v_info, int __pyx_v_flags) {\n int __pyx_r;\n __Pyx_RefNannyDeclarations\n __Pyx_RefNannySetupContext(\"__getbuffer__ (wrapper)\", 0);\n __pyx_r = __pyx_pf_5numpy_7ndarray___getbuffer__(((PyArrayObject *)__pyx_v_self), ((Py_buffer *)__pyx_v_info), ((int)__pyx_v_flags));\n\n /* function exit code */\n __Pyx_RefNannyFinishContext();\n return __pyx_r;\n}\n\nstatic int __pyx_pf_5numpy_7ndarray___getbuffer__(PyArrayObject *__pyx_v_self, Py_buffer *__pyx_v_info, int __pyx_v_flags) {\n int __pyx_v_copy_shape;\n int __pyx_v_i;\n int __pyx_v_ndim;\n int __pyx_v_endian_detector;\n int __pyx_v_little_endian;\n int __pyx_v_t;\n char *__pyx_v_f;\n PyArray_Descr *__pyx_v_descr = 0;\n int __pyx_v_offset;\n int __pyx_v_hasfields;\n int __pyx_r;\n __Pyx_RefNannyDeclarations\n int __pyx_t_1;\n int __pyx_t_2;\n PyObject *__pyx_t_3 = NULL;\n int __pyx_t_4;\n int __pyx_t_5;\n PyObject *__pyx_t_6 = NULL;\n char *__pyx_t_7;\n __Pyx_RefNannySetupContext(\"__getbuffer__\", 0);\n if (__pyx_v_info != NULL) {\n __pyx_v_info->obj = Py_None; __Pyx_INCREF(Py_None);\n __Pyx_GIVEREF(__pyx_v_info->obj);\n }\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":203\n * # of flags\n * \n * if info == NULL: return # <<<<<<<<<<<<<<\n * \n * cdef int copy_shape, i, ndim\n */\n __pyx_t_1 = ((__pyx_v_info == NULL) != 0);\n if (__pyx_t_1) {\n __pyx_r = 0;\n goto __pyx_L0;\n }\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":206\n * \n * cdef int copy_shape, i, ndim\n * cdef int endian_detector = 1 # <<<<<<<<<<<<<<\n * cdef bint little_endian = ((&endian_detector)[0] != 0)\n * \n */\n __pyx_v_endian_detector = 1;\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":207\n * cdef int copy_shape, i, ndim\n * cdef int endian_detector = 1\n * cdef bint little_endian = ((&endian_detector)[0] != 0) # <<<<<<<<<<<<<<\n * \n * ndim = PyArray_NDIM(self)\n */\n __pyx_v_little_endian = ((((char *)(&__pyx_v_endian_detector))[0]) != 0);\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":209\n * cdef bint little_endian = ((&endian_detector)[0] != 0)\n * \n * ndim = PyArray_NDIM(self) # <<<<<<<<<<<<<<\n * \n * if sizeof(npy_intp) != sizeof(Py_ssize_t):\n */\n __pyx_v_ndim = PyArray_NDIM(__pyx_v_self);\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":211\n * ndim = PyArray_NDIM(self)\n * \n * if sizeof(npy_intp) != sizeof(Py_ssize_t): # <<<<<<<<<<<<<<\n * copy_shape = 1\n * else:\n */\n __pyx_t_1 = (((sizeof(npy_intp)) != (sizeof(Py_ssize_t))) != 0);\n if (__pyx_t_1) {\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":212\n * \n * if sizeof(npy_intp) != sizeof(Py_ssize_t):\n * copy_shape = 1 # <<<<<<<<<<<<<<\n * else:\n * copy_shape = 0\n */\n __pyx_v_copy_shape = 1;\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":211\n * ndim = PyArray_NDIM(self)\n * \n * if sizeof(npy_intp) != sizeof(Py_ssize_t): # <<<<<<<<<<<<<<\n * copy_shape = 1\n * else:\n */\n goto __pyx_L4;\n }\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":214\n * copy_shape = 1\n * else:\n * copy_shape = 0 # <<<<<<<<<<<<<<\n * \n * if ((flags & pybuf.PyBUF_C_CONTIGUOUS == pybuf.PyBUF_C_CONTIGUOUS)\n */\n /*else*/ {\n __pyx_v_copy_shape = 0;\n }\n __pyx_L4:;\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":216\n * copy_shape = 0\n * \n * if ((flags & pybuf.PyBUF_C_CONTIGUOUS == pybuf.PyBUF_C_CONTIGUOUS) # <<<<<<<<<<<<<<\n * and not PyArray_CHKFLAGS(self, NPY_C_CONTIGUOUS)):\n * raise ValueError(u\"ndarray is not C contiguous\")\n */\n __pyx_t_2 = (((__pyx_v_flags & PyBUF_C_CONTIGUOUS) == PyBUF_C_CONTIGUOUS) != 0);\n if (__pyx_t_2) {\n } else {\n __pyx_t_1 = __pyx_t_2;\n goto __pyx_L6_bool_binop_done;\n }\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":217\n * \n * if ((flags & pybuf.PyBUF_C_CONTIGUOUS == pybuf.PyBUF_C_CONTIGUOUS)\n * and not PyArray_CHKFLAGS(self, NPY_C_CONTIGUOUS)): # <<<<<<<<<<<<<<\n * raise ValueError(u\"ndarray is not C contiguous\")\n * \n */\n __pyx_t_2 = ((!(PyArray_CHKFLAGS(__pyx_v_self, NPY_C_CONTIGUOUS) != 0)) != 0);\n __pyx_t_1 = __pyx_t_2;\n __pyx_L6_bool_binop_done:;\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":216\n * copy_shape = 0\n * \n * if ((flags & pybuf.PyBUF_C_CONTIGUOUS == pybuf.PyBUF_C_CONTIGUOUS) # <<<<<<<<<<<<<<\n * and not PyArray_CHKFLAGS(self, NPY_C_CONTIGUOUS)):\n * raise ValueError(u\"ndarray is not C contiguous\")\n */\n if (__pyx_t_1) {\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":218\n * if ((flags & pybuf.PyBUF_C_CONTIGUOUS == pybuf.PyBUF_C_CONTIGUOUS)\n * and not PyArray_CHKFLAGS(self, NPY_C_CONTIGUOUS)):\n * raise ValueError(u\"ndarray is not C contiguous\") # <<<<<<<<<<<<<<\n * \n * if ((flags & pybuf.PyBUF_F_CONTIGUOUS == pybuf.PyBUF_F_CONTIGUOUS)\n */\n __pyx_t_3 = __Pyx_PyObject_Call(__pyx_builtin_ValueError, __pyx_tuple__5, NULL); if (unlikely(!__pyx_t_3)) __PYX_ERR(1, 218, __pyx_L1_error)\n __Pyx_GOTREF(__pyx_t_3);\n __Pyx_Raise(__pyx_t_3, 0, 0, 0);\n __Pyx_DECREF(__pyx_t_3); __pyx_t_3 = 0;\n __PYX_ERR(1, 218, __pyx_L1_error)\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":216\n * copy_shape = 0\n * \n * if ((flags & pybuf.PyBUF_C_CONTIGUOUS == pybuf.PyBUF_C_CONTIGUOUS) # <<<<<<<<<<<<<<\n * and not PyArray_CHKFLAGS(self, NPY_C_CONTIGUOUS)):\n * raise ValueError(u\"ndarray is not C contiguous\")\n */\n }\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":220\n * raise ValueError(u\"ndarray is not C contiguous\")\n * \n * if ((flags & pybuf.PyBUF_F_CONTIGUOUS == pybuf.PyBUF_F_CONTIGUOUS) # <<<<<<<<<<<<<<\n * and not PyArray_CHKFLAGS(self, NPY_F_CONTIGUOUS)):\n * raise ValueError(u\"ndarray is not Fortran contiguous\")\n */\n __pyx_t_2 = (((__pyx_v_flags & PyBUF_F_CONTIGUOUS) == PyBUF_F_CONTIGUOUS) != 0);\n if (__pyx_t_2) {\n } else {\n __pyx_t_1 = __pyx_t_2;\n goto __pyx_L9_bool_binop_done;\n }\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":221\n * \n * if ((flags & pybuf.PyBUF_F_CONTIGUOUS == pybuf.PyBUF_F_CONTIGUOUS)\n * and not PyArray_CHKFLAGS(self, NPY_F_CONTIGUOUS)): # <<<<<<<<<<<<<<\n * raise ValueError(u\"ndarray is not Fortran contiguous\")\n * \n */\n __pyx_t_2 = ((!(PyArray_CHKFLAGS(__pyx_v_self, NPY_F_CONTIGUOUS) != 0)) != 0);\n __pyx_t_1 = __pyx_t_2;\n __pyx_L9_bool_binop_done:;\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":220\n * raise ValueError(u\"ndarray is not C contiguous\")\n * \n * if ((flags & pybuf.PyBUF_F_CONTIGUOUS == pybuf.PyBUF_F_CONTIGUOUS) # <<<<<<<<<<<<<<\n * and not PyArray_CHKFLAGS(self, NPY_F_CONTIGUOUS)):\n * raise ValueError(u\"ndarray is not Fortran contiguous\")\n */\n if (__pyx_t_1) {\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":222\n * if ((flags & pybuf.PyBUF_F_CONTIGUOUS == pybuf.PyBUF_F_CONTIGUOUS)\n * and not PyArray_CHKFLAGS(self, NPY_F_CONTIGUOUS)):\n * raise ValueError(u\"ndarray is not Fortran contiguous\") # <<<<<<<<<<<<<<\n * \n * info.buf = PyArray_DATA(self)\n */\n __pyx_t_3 = __Pyx_PyObject_Call(__pyx_builtin_ValueError, __pyx_tuple__6, NULL); if (unlikely(!__pyx_t_3)) __PYX_ERR(1, 222, __pyx_L1_error)\n __Pyx_GOTREF(__pyx_t_3);\n __Pyx_Raise(__pyx_t_3, 0, 0, 0);\n __Pyx_DECREF(__pyx_t_3); __pyx_t_3 = 0;\n __PYX_ERR(1, 222, __pyx_L1_error)\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":220\n * raise ValueError(u\"ndarray is not C contiguous\")\n * \n * if ((flags & pybuf.PyBUF_F_CONTIGUOUS == pybuf.PyBUF_F_CONTIGUOUS) # <<<<<<<<<<<<<<\n * and not PyArray_CHKFLAGS(self, NPY_F_CONTIGUOUS)):\n * raise ValueError(u\"ndarray is not Fortran contiguous\")\n */\n }\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":224\n * raise ValueError(u\"ndarray is not Fortran contiguous\")\n * \n * info.buf = PyArray_DATA(self) # <<<<<<<<<<<<<<\n * info.ndim = ndim\n * if copy_shape:\n */\n __pyx_v_info->buf = PyArray_DATA(__pyx_v_self);\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":225\n * \n * info.buf = PyArray_DATA(self)\n * info.ndim = ndim # <<<<<<<<<<<<<<\n * if copy_shape:\n * # Allocate new buffer for strides and shape info.\n */\n __pyx_v_info->ndim = __pyx_v_ndim;\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":226\n * info.buf = PyArray_DATA(self)\n * info.ndim = ndim\n * if copy_shape: # <<<<<<<<<<<<<<\n * # Allocate new buffer for strides and shape info.\n * # This is allocated as one block, strides first.\n */\n __pyx_t_1 = (__pyx_v_copy_shape != 0);\n if (__pyx_t_1) {\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":229\n * # Allocate new buffer for strides and shape info.\n * # This is allocated as one block, strides first.\n * info.strides = stdlib.malloc(sizeof(Py_ssize_t) * ndim * 2) # <<<<<<<<<<<<<<\n * info.shape = info.strides + ndim\n * for i in range(ndim):\n */\n __pyx_v_info->strides = ((Py_ssize_t *)malloc((((sizeof(Py_ssize_t)) * ((size_t)__pyx_v_ndim)) * 2)));\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":230\n * # This is allocated as one block, strides first.\n * info.strides = stdlib.malloc(sizeof(Py_ssize_t) * ndim * 2)\n * info.shape = info.strides + ndim # <<<<<<<<<<<<<<\n * for i in range(ndim):\n * info.strides[i] = PyArray_STRIDES(self)[i]\n */\n __pyx_v_info->shape = (__pyx_v_info->strides + __pyx_v_ndim);\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":231\n * info.strides = stdlib.malloc(sizeof(Py_ssize_t) * ndim * 2)\n * info.shape = info.strides + ndim\n * for i in range(ndim): # <<<<<<<<<<<<<<\n * info.strides[i] = PyArray_STRIDES(self)[i]\n * info.shape[i] = PyArray_DIMS(self)[i]\n */\n __pyx_t_4 = __pyx_v_ndim;\n for (__pyx_t_5 = 0; __pyx_t_5 < __pyx_t_4; __pyx_t_5+=1) {\n __pyx_v_i = __pyx_t_5;\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":232\n * info.shape = info.strides + ndim\n * for i in range(ndim):\n * info.strides[i] = PyArray_STRIDES(self)[i] # <<<<<<<<<<<<<<\n * info.shape[i] = PyArray_DIMS(self)[i]\n * else:\n */\n (__pyx_v_info->strides[__pyx_v_i]) = (PyArray_STRIDES(__pyx_v_self)[__pyx_v_i]);\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":233\n * for i in range(ndim):\n * info.strides[i] = PyArray_STRIDES(self)[i]\n * info.shape[i] = PyArray_DIMS(self)[i] # <<<<<<<<<<<<<<\n * else:\n * info.strides = PyArray_STRIDES(self)\n */\n (__pyx_v_info->shape[__pyx_v_i]) = (PyArray_DIMS(__pyx_v_self)[__pyx_v_i]);\n }\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":226\n * info.buf = PyArray_DATA(self)\n * info.ndim = ndim\n * if copy_shape: # <<<<<<<<<<<<<<\n * # Allocate new buffer for strides and shape info.\n * # This is allocated as one block, strides first.\n */\n goto __pyx_L11;\n }\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":235\n * info.shape[i] = PyArray_DIMS(self)[i]\n * else:\n * info.strides = PyArray_STRIDES(self) # <<<<<<<<<<<<<<\n * info.shape = PyArray_DIMS(self)\n * info.suboffsets = NULL\n */\n /*else*/ {\n __pyx_v_info->strides = ((Py_ssize_t *)PyArray_STRIDES(__pyx_v_self));\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":236\n * else:\n * info.strides = PyArray_STRIDES(self)\n * info.shape = PyArray_DIMS(self) # <<<<<<<<<<<<<<\n * info.suboffsets = NULL\n * info.itemsize = PyArray_ITEMSIZE(self)\n */\n __pyx_v_info->shape = ((Py_ssize_t *)PyArray_DIMS(__pyx_v_self));\n }\n __pyx_L11:;\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":237\n * info.strides = PyArray_STRIDES(self)\n * info.shape = PyArray_DIMS(self)\n * info.suboffsets = NULL # <<<<<<<<<<<<<<\n * info.itemsize = PyArray_ITEMSIZE(self)\n * info.readonly = not PyArray_ISWRITEABLE(self)\n */\n __pyx_v_info->suboffsets = NULL;\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":238\n * info.shape = PyArray_DIMS(self)\n * info.suboffsets = NULL\n * info.itemsize = PyArray_ITEMSIZE(self) # <<<<<<<<<<<<<<\n * info.readonly = not PyArray_ISWRITEABLE(self)\n * \n */\n __pyx_v_info->itemsize = PyArray_ITEMSIZE(__pyx_v_self);\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":239\n * info.suboffsets = NULL\n * info.itemsize = PyArray_ITEMSIZE(self)\n * info.readonly = not PyArray_ISWRITEABLE(self) # <<<<<<<<<<<<<<\n * \n * cdef int t\n */\n __pyx_v_info->readonly = (!(PyArray_ISWRITEABLE(__pyx_v_self) != 0));\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":242\n * \n * cdef int t\n * cdef char* f = NULL # <<<<<<<<<<<<<<\n * cdef dtype descr = self.descr\n * cdef int offset\n */\n __pyx_v_f = NULL;\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":243\n * cdef int t\n * cdef char* f = NULL\n * cdef dtype descr = self.descr # <<<<<<<<<<<<<<\n * cdef int offset\n * \n */\n __pyx_t_3 = ((PyObject *)__pyx_v_self->descr);\n __Pyx_INCREF(__pyx_t_3);\n __pyx_v_descr = ((PyArray_Descr *)__pyx_t_3);\n __pyx_t_3 = 0;\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":246\n * cdef int offset\n * \n * cdef bint hasfields = PyDataType_HASFIELDS(descr) # <<<<<<<<<<<<<<\n * \n * if not hasfields and not copy_shape:\n */\n __pyx_v_hasfields = PyDataType_HASFIELDS(__pyx_v_descr);\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":248\n * cdef bint hasfields = PyDataType_HASFIELDS(descr)\n * \n * if not hasfields and not copy_shape: # <<<<<<<<<<<<<<\n * # do not call releasebuffer\n * info.obj = None\n */\n __pyx_t_2 = ((!(__pyx_v_hasfields != 0)) != 0);\n if (__pyx_t_2) {\n } else {\n __pyx_t_1 = __pyx_t_2;\n goto __pyx_L15_bool_binop_done;\n }\n __pyx_t_2 = ((!(__pyx_v_copy_shape != 0)) != 0);\n __pyx_t_1 = __pyx_t_2;\n __pyx_L15_bool_binop_done:;\n if (__pyx_t_1) {\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":250\n * if not hasfields and not copy_shape:\n * # do not call releasebuffer\n * info.obj = None # <<<<<<<<<<<<<<\n * else:\n * # need to call releasebuffer\n */\n __Pyx_INCREF(Py_None);\n __Pyx_GIVEREF(Py_None);\n __Pyx_GOTREF(__pyx_v_info->obj);\n __Pyx_DECREF(__pyx_v_info->obj);\n __pyx_v_info->obj = Py_None;\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":248\n * cdef bint hasfields = PyDataType_HASFIELDS(descr)\n * \n * if not hasfields and not copy_shape: # <<<<<<<<<<<<<<\n * # do not call releasebuffer\n * info.obj = None\n */\n goto __pyx_L14;\n }\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":253\n * else:\n * # need to call releasebuffer\n * info.obj = self # <<<<<<<<<<<<<<\n * \n * if not hasfields:\n */\n /*else*/ {\n __Pyx_INCREF(((PyObject *)__pyx_v_self));\n __Pyx_GIVEREF(((PyObject *)__pyx_v_self));\n __Pyx_GOTREF(__pyx_v_info->obj);\n __Pyx_DECREF(__pyx_v_info->obj);\n __pyx_v_info->obj = ((PyObject *)__pyx_v_self);\n }\n __pyx_L14:;\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":255\n * info.obj = self\n * \n * if not hasfields: # <<<<<<<<<<<<<<\n * t = descr.type_num\n * if ((descr.byteorder == c'>' and little_endian) or\n */\n __pyx_t_1 = ((!(__pyx_v_hasfields != 0)) != 0);\n if (__pyx_t_1) {\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":256\n * \n * if not hasfields:\n * t = descr.type_num # <<<<<<<<<<<<<<\n * if ((descr.byteorder == c'>' and little_endian) or\n * (descr.byteorder == c'<' and not little_endian)):\n */\n __pyx_t_4 = __pyx_v_descr->type_num;\n __pyx_v_t = __pyx_t_4;\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":257\n * if not hasfields:\n * t = descr.type_num\n * if ((descr.byteorder == c'>' and little_endian) or # <<<<<<<<<<<<<<\n * (descr.byteorder == c'<' and not little_endian)):\n * raise ValueError(u\"Non-native byte order not supported\")\n */\n __pyx_t_2 = ((__pyx_v_descr->byteorder == '>') != 0);\n if (!__pyx_t_2) {\n goto __pyx_L20_next_or;\n } else {\n }\n __pyx_t_2 = (__pyx_v_little_endian != 0);\n if (!__pyx_t_2) {\n } else {\n __pyx_t_1 = __pyx_t_2;\n goto __pyx_L19_bool_binop_done;\n }\n __pyx_L20_next_or:;\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":258\n * t = descr.type_num\n * if ((descr.byteorder == c'>' and little_endian) or\n * (descr.byteorder == c'<' and not little_endian)): # <<<<<<<<<<<<<<\n * raise ValueError(u\"Non-native byte order not supported\")\n * if t == NPY_BYTE: f = \"b\"\n */\n __pyx_t_2 = ((__pyx_v_descr->byteorder == '<') != 0);\n if (__pyx_t_2) {\n } else {\n __pyx_t_1 = __pyx_t_2;\n goto __pyx_L19_bool_binop_done;\n }\n __pyx_t_2 = ((!(__pyx_v_little_endian != 0)) != 0);\n __pyx_t_1 = __pyx_t_2;\n __pyx_L19_bool_binop_done:;\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":257\n * if not hasfields:\n * t = descr.type_num\n * if ((descr.byteorder == c'>' and little_endian) or # <<<<<<<<<<<<<<\n * (descr.byteorder == c'<' and not little_endian)):\n * raise ValueError(u\"Non-native byte order not supported\")\n */\n if (__pyx_t_1) {\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":259\n * if ((descr.byteorder == c'>' and little_endian) or\n * (descr.byteorder == c'<' and not little_endian)):\n * raise ValueError(u\"Non-native byte order not supported\") # <<<<<<<<<<<<<<\n * if t == NPY_BYTE: f = \"b\"\n * elif t == NPY_UBYTE: f = \"B\"\n */\n __pyx_t_3 = __Pyx_PyObject_Call(__pyx_builtin_ValueError, __pyx_tuple__7, NULL); if (unlikely(!__pyx_t_3)) __PYX_ERR(1, 259, __pyx_L1_error)\n __Pyx_GOTREF(__pyx_t_3);\n __Pyx_Raise(__pyx_t_3, 0, 0, 0);\n __Pyx_DECREF(__pyx_t_3); __pyx_t_3 = 0;\n __PYX_ERR(1, 259, __pyx_L1_error)\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":257\n * if not hasfields:\n * t = descr.type_num\n * if ((descr.byteorder == c'>' and little_endian) or # <<<<<<<<<<<<<<\n * (descr.byteorder == c'<' and not little_endian)):\n * raise ValueError(u\"Non-native byte order not supported\")\n */\n }\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":260\n * (descr.byteorder == c'<' and not little_endian)):\n * raise ValueError(u\"Non-native byte order not supported\")\n * if t == NPY_BYTE: f = \"b\" # <<<<<<<<<<<<<<\n * elif t == NPY_UBYTE: f = \"B\"\n * elif t == NPY_SHORT: f = \"h\"\n */\n switch (__pyx_v_t) {\n case NPY_BYTE:\n __pyx_v_f = ((char *)\"b\");\n break;\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":261\n * raise ValueError(u\"Non-native byte order not supported\")\n * if t == NPY_BYTE: f = \"b\"\n * elif t == NPY_UBYTE: f = \"B\" # <<<<<<<<<<<<<<\n * elif t == NPY_SHORT: f = \"h\"\n * elif t == NPY_USHORT: f = \"H\"\n */\n case NPY_UBYTE:\n __pyx_v_f = ((char *)\"B\");\n break;\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":262\n * if t == NPY_BYTE: f = \"b\"\n * elif t == NPY_UBYTE: f = \"B\"\n * elif t == NPY_SHORT: f = \"h\" # <<<<<<<<<<<<<<\n * elif t == NPY_USHORT: f = \"H\"\n * elif t == NPY_INT: f = \"i\"\n */\n case NPY_SHORT:\n __pyx_v_f = ((char *)\"h\");\n break;\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":263\n * elif t == NPY_UBYTE: f = \"B\"\n * elif t == NPY_SHORT: f = \"h\"\n * elif t == NPY_USHORT: f = \"H\" # <<<<<<<<<<<<<<\n * elif t == NPY_INT: f = \"i\"\n * elif t == NPY_UINT: f = \"I\"\n */\n case NPY_USHORT:\n __pyx_v_f = ((char *)\"H\");\n break;\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":264\n * elif t == NPY_SHORT: f = \"h\"\n * elif t == NPY_USHORT: f = \"H\"\n * elif t == NPY_INT: f = \"i\" # <<<<<<<<<<<<<<\n * elif t == NPY_UINT: f = \"I\"\n * elif t == NPY_LONG: f = \"l\"\n */\n case NPY_INT:\n __pyx_v_f = ((char *)\"i\");\n break;\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":265\n * elif t == NPY_USHORT: f = \"H\"\n * elif t == NPY_INT: f = \"i\"\n * elif t == NPY_UINT: f = \"I\" # <<<<<<<<<<<<<<\n * elif t == NPY_LONG: f = \"l\"\n * elif t == NPY_ULONG: f = \"L\"\n */\n case NPY_UINT:\n __pyx_v_f = ((char *)\"I\");\n break;\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":266\n * elif t == NPY_INT: f = \"i\"\n * elif t == NPY_UINT: f = \"I\"\n * elif t == NPY_LONG: f = \"l\" # <<<<<<<<<<<<<<\n * elif t == NPY_ULONG: f = \"L\"\n * elif t == NPY_LONGLONG: f = \"q\"\n */\n case NPY_LONG:\n __pyx_v_f = ((char *)\"l\");\n break;\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":267\n * elif t == NPY_UINT: f = \"I\"\n * elif t == NPY_LONG: f = \"l\"\n * elif t == NPY_ULONG: f = \"L\" # <<<<<<<<<<<<<<\n * elif t == NPY_LONGLONG: f = \"q\"\n * elif t == NPY_ULONGLONG: f = \"Q\"\n */\n case NPY_ULONG:\n __pyx_v_f = ((char *)\"L\");\n break;\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":268\n * elif t == NPY_LONG: f = \"l\"\n * elif t == NPY_ULONG: f = \"L\"\n * elif t == NPY_LONGLONG: f = \"q\" # <<<<<<<<<<<<<<\n * elif t == NPY_ULONGLONG: f = \"Q\"\n * elif t == NPY_FLOAT: f = \"f\"\n */\n case NPY_LONGLONG:\n __pyx_v_f = ((char *)\"q\");\n break;\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":269\n * elif t == NPY_ULONG: f = \"L\"\n * elif t == NPY_LONGLONG: f = \"q\"\n * elif t == NPY_ULONGLONG: f = \"Q\" # <<<<<<<<<<<<<<\n * elif t == NPY_FLOAT: f = \"f\"\n * elif t == NPY_DOUBLE: f = \"d\"\n */\n case NPY_ULONGLONG:\n __pyx_v_f = ((char *)\"Q\");\n break;\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":270\n * elif t == NPY_LONGLONG: f = \"q\"\n * elif t == NPY_ULONGLONG: f = \"Q\"\n * elif t == NPY_FLOAT: f = \"f\" # <<<<<<<<<<<<<<\n * elif t == NPY_DOUBLE: f = \"d\"\n * elif t == NPY_LONGDOUBLE: f = \"g\"\n */\n case NPY_FLOAT:\n __pyx_v_f = ((char *)\"f\");\n break;\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":271\n * elif t == NPY_ULONGLONG: f = \"Q\"\n * elif t == NPY_FLOAT: f = \"f\"\n * elif t == NPY_DOUBLE: f = \"d\" # <<<<<<<<<<<<<<\n * elif t == NPY_LONGDOUBLE: f = \"g\"\n * elif t == NPY_CFLOAT: f = \"Zf\"\n */\n case NPY_DOUBLE:\n __pyx_v_f = ((char *)\"d\");\n break;\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":272\n * elif t == NPY_FLOAT: f = \"f\"\n * elif t == NPY_DOUBLE: f = \"d\"\n * elif t == NPY_LONGDOUBLE: f = \"g\" # <<<<<<<<<<<<<<\n * elif t == NPY_CFLOAT: f = \"Zf\"\n * elif t == NPY_CDOUBLE: f = \"Zd\"\n */\n case NPY_LONGDOUBLE:\n __pyx_v_f = ((char *)\"g\");\n break;\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":273\n * elif t == NPY_DOUBLE: f = \"d\"\n * elif t == NPY_LONGDOUBLE: f = \"g\"\n * elif t == NPY_CFLOAT: f = \"Zf\" # <<<<<<<<<<<<<<\n * elif t == NPY_CDOUBLE: f = \"Zd\"\n * elif t == NPY_CLONGDOUBLE: f = \"Zg\"\n */\n case NPY_CFLOAT:\n __pyx_v_f = ((char *)\"Zf\");\n break;\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":274\n * elif t == NPY_LONGDOUBLE: f = \"g\"\n * elif t == NPY_CFLOAT: f = \"Zf\"\n * elif t == NPY_CDOUBLE: f = \"Zd\" # <<<<<<<<<<<<<<\n * elif t == NPY_CLONGDOUBLE: f = \"Zg\"\n * elif t == NPY_OBJECT: f = \"O\"\n */\n case NPY_CDOUBLE:\n __pyx_v_f = ((char *)\"Zd\");\n break;\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":275\n * elif t == NPY_CFLOAT: f = \"Zf\"\n * elif t == NPY_CDOUBLE: f = \"Zd\"\n * elif t == NPY_CLONGDOUBLE: f = \"Zg\" # <<<<<<<<<<<<<<\n * elif t == NPY_OBJECT: f = \"O\"\n * else:\n */\n case NPY_CLONGDOUBLE:\n __pyx_v_f = ((char *)\"Zg\");\n break;\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":276\n * elif t == NPY_CDOUBLE: f = \"Zd\"\n * elif t == NPY_CLONGDOUBLE: f = \"Zg\"\n * elif t == NPY_OBJECT: f = \"O\" # <<<<<<<<<<<<<<\n * else:\n * raise ValueError(u\"unknown dtype code in numpy.pxd (%d)\" % t)\n */\n case NPY_OBJECT:\n __pyx_v_f = ((char *)\"O\");\n break;\n default:\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":278\n * elif t == NPY_OBJECT: f = \"O\"\n * else:\n * raise ValueError(u\"unknown dtype code in numpy.pxd (%d)\" % t) # <<<<<<<<<<<<<<\n * info.format = f\n * return\n */\n __pyx_t_3 = __Pyx_PyInt_From_int(__pyx_v_t); if (unlikely(!__pyx_t_3)) __PYX_ERR(1, 278, __pyx_L1_error)\n __Pyx_GOTREF(__pyx_t_3);\n __pyx_t_6 = PyUnicode_Format(__pyx_kp_u_unknown_dtype_code_in_numpy_pxd, __pyx_t_3); if (unlikely(!__pyx_t_6)) __PYX_ERR(1, 278, __pyx_L1_error)\n __Pyx_GOTREF(__pyx_t_6);\n __Pyx_DECREF(__pyx_t_3); __pyx_t_3 = 0;\n __pyx_t_3 = PyTuple_New(1); if (unlikely(!__pyx_t_3)) __PYX_ERR(1, 278, __pyx_L1_error)\n __Pyx_GOTREF(__pyx_t_3);\n __Pyx_GIVEREF(__pyx_t_6);\n PyTuple_SET_ITEM(__pyx_t_3, 0, __pyx_t_6);\n __pyx_t_6 = 0;\n __pyx_t_6 = __Pyx_PyObject_Call(__pyx_builtin_ValueError, __pyx_t_3, NULL); if (unlikely(!__pyx_t_6)) __PYX_ERR(1, 278, __pyx_L1_error)\n __Pyx_GOTREF(__pyx_t_6);\n __Pyx_DECREF(__pyx_t_3); __pyx_t_3 = 0;\n __Pyx_Raise(__pyx_t_6, 0, 0, 0);\n __Pyx_DECREF(__pyx_t_6); __pyx_t_6 = 0;\n __PYX_ERR(1, 278, __pyx_L1_error)\n break;\n }\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":279\n * else:\n * raise ValueError(u\"unknown dtype code in numpy.pxd (%d)\" % t)\n * info.format = f # <<<<<<<<<<<<<<\n * return\n * else:\n */\n __pyx_v_info->format = __pyx_v_f;\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":280\n * raise ValueError(u\"unknown dtype code in numpy.pxd (%d)\" % t)\n * info.format = f\n * return # <<<<<<<<<<<<<<\n * else:\n * info.format = stdlib.malloc(_buffer_format_string_len)\n */\n __pyx_r = 0;\n goto __pyx_L0;\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":255\n * info.obj = self\n * \n * if not hasfields: # <<<<<<<<<<<<<<\n * t = descr.type_num\n * if ((descr.byteorder == c'>' and little_endian) or\n */\n }\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":282\n * return\n * else:\n * info.format = stdlib.malloc(_buffer_format_string_len) # <<<<<<<<<<<<<<\n * info.format[0] = c'^' # Native data types, manual alignment\n * offset = 0\n */\n /*else*/ {\n __pyx_v_info->format = ((char *)malloc(0xFF));\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":283\n * else:\n * info.format = stdlib.malloc(_buffer_format_string_len)\n * info.format[0] = c'^' # Native data types, manual alignment # <<<<<<<<<<<<<<\n * offset = 0\n * f = _util_dtypestring(descr, info.format + 1,\n */\n (__pyx_v_info->format[0]) = '^';\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":284\n * info.format = stdlib.malloc(_buffer_format_string_len)\n * info.format[0] = c'^' # Native data types, manual alignment\n * offset = 0 # <<<<<<<<<<<<<<\n * f = _util_dtypestring(descr, info.format + 1,\n * info.format + _buffer_format_string_len,\n */\n __pyx_v_offset = 0;\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":285\n * info.format[0] = c'^' # Native data types, manual alignment\n * offset = 0\n * f = _util_dtypestring(descr, info.format + 1, # <<<<<<<<<<<<<<\n * info.format + _buffer_format_string_len,\n * &offset)\n */\n __pyx_t_7 = __pyx_f_5numpy__util_dtypestring(__pyx_v_descr, (__pyx_v_info->format + 1), (__pyx_v_info->format + 0xFF), (&__pyx_v_offset)); if (unlikely(__pyx_t_7 == NULL)) __PYX_ERR(1, 285, __pyx_L1_error)\n __pyx_v_f = __pyx_t_7;\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":288\n * info.format + _buffer_format_string_len,\n * &offset)\n * f[0] = c'\\0' # Terminate format string # <<<<<<<<<<<<<<\n * \n * def __releasebuffer__(ndarray self, Py_buffer* info):\n */\n (__pyx_v_f[0]) = '\\x00';\n }\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":197\n * # experimental exception made for __getbuffer__ and __releasebuffer__\n * # -- the details of this may change.\n * def __getbuffer__(ndarray self, Py_buffer* info, int flags): # <<<<<<<<<<<<<<\n * # This implementation of getbuffer is geared towards Cython\n * # requirements, and does not yet fullfill the PEP.\n */\n\n /* function exit code */\n __pyx_r = 0;\n goto __pyx_L0;\n __pyx_L1_error:;\n __Pyx_XDECREF(__pyx_t_3);\n __Pyx_XDECREF(__pyx_t_6);\n __Pyx_AddTraceback(\"numpy.ndarray.__getbuffer__\", __pyx_clineno, __pyx_lineno, __pyx_filename);\n __pyx_r = -1;\n if (__pyx_v_info != NULL && __pyx_v_info->obj != NULL) {\n __Pyx_GOTREF(__pyx_v_info->obj);\n __Pyx_DECREF(__pyx_v_info->obj); __pyx_v_info->obj = NULL;\n }\n goto __pyx_L2;\n __pyx_L0:;\n if (__pyx_v_info != NULL && __pyx_v_info->obj == Py_None) {\n __Pyx_GOTREF(Py_None);\n __Pyx_DECREF(Py_None); __pyx_v_info->obj = NULL;\n }\n __pyx_L2:;\n __Pyx_XDECREF((PyObject *)__pyx_v_descr);\n __Pyx_RefNannyFinishContext();\n return __pyx_r;\n}\n\n/* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":290\n * f[0] = c'\\0' # Terminate format string\n * \n * def __releasebuffer__(ndarray self, Py_buffer* info): # <<<<<<<<<<<<<<\n * if PyArray_HASFIELDS(self):\n * stdlib.free(info.format)\n */\n\n/* Python wrapper */\nstatic CYTHON_UNUSED void __pyx_pw_5numpy_7ndarray_3__releasebuffer__(PyObject *__pyx_v_self, Py_buffer *__pyx_v_info); /*proto*/\nstatic CYTHON_UNUSED void __pyx_pw_5numpy_7ndarray_3__releasebuffer__(PyObject *__pyx_v_self, Py_buffer *__pyx_v_info) {\n __Pyx_RefNannyDeclarations\n __Pyx_RefNannySetupContext(\"__releasebuffer__ (wrapper)\", 0);\n __pyx_pf_5numpy_7ndarray_2__releasebuffer__(((PyArrayObject *)__pyx_v_self), ((Py_buffer *)__pyx_v_info));\n\n /* function exit code */\n __Pyx_RefNannyFinishContext();\n}\n\nstatic void __pyx_pf_5numpy_7ndarray_2__releasebuffer__(PyArrayObject *__pyx_v_self, Py_buffer *__pyx_v_info) {\n __Pyx_RefNannyDeclarations\n int __pyx_t_1;\n __Pyx_RefNannySetupContext(\"__releasebuffer__\", 0);\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":291\n * \n * def __releasebuffer__(ndarray self, Py_buffer* info):\n * if PyArray_HASFIELDS(self): # <<<<<<<<<<<<<<\n * stdlib.free(info.format)\n * if sizeof(npy_intp) != sizeof(Py_ssize_t):\n */\n __pyx_t_1 = (PyArray_HASFIELDS(__pyx_v_self) != 0);\n if (__pyx_t_1) {\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":292\n * def __releasebuffer__(ndarray self, Py_buffer* info):\n * if PyArray_HASFIELDS(self):\n * stdlib.free(info.format) # <<<<<<<<<<<<<<\n * if sizeof(npy_intp) != sizeof(Py_ssize_t):\n * stdlib.free(info.strides)\n */\n free(__pyx_v_info->format);\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":291\n * \n * def __releasebuffer__(ndarray self, Py_buffer* info):\n * if PyArray_HASFIELDS(self): # <<<<<<<<<<<<<<\n * stdlib.free(info.format)\n * if sizeof(npy_intp) != sizeof(Py_ssize_t):\n */\n }\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":293\n * if PyArray_HASFIELDS(self):\n * stdlib.free(info.format)\n * if sizeof(npy_intp) != sizeof(Py_ssize_t): # <<<<<<<<<<<<<<\n * stdlib.free(info.strides)\n * # info.shape was stored after info.strides in the same block\n */\n __pyx_t_1 = (((sizeof(npy_intp)) != (sizeof(Py_ssize_t))) != 0);\n if (__pyx_t_1) {\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":294\n * stdlib.free(info.format)\n * if sizeof(npy_intp) != sizeof(Py_ssize_t):\n * stdlib.free(info.strides) # <<<<<<<<<<<<<<\n * # info.shape was stored after info.strides in the same block\n * \n */\n free(__pyx_v_info->strides);\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":293\n * if PyArray_HASFIELDS(self):\n * stdlib.free(info.format)\n * if sizeof(npy_intp) != sizeof(Py_ssize_t): # <<<<<<<<<<<<<<\n * stdlib.free(info.strides)\n * # info.shape was stored after info.strides in the same block\n */\n }\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":290\n * f[0] = c'\\0' # Terminate format string\n * \n * def __releasebuffer__(ndarray self, Py_buffer* info): # <<<<<<<<<<<<<<\n * if PyArray_HASFIELDS(self):\n * stdlib.free(info.format)\n */\n\n /* function exit code */\n __Pyx_RefNannyFinishContext();\n}\n\n/* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":770\n * ctypedef npy_cdouble complex_t\n * \n * cdef inline object PyArray_MultiIterNew1(a): # <<<<<<<<<<<<<<\n * return PyArray_MultiIterNew(1, a)\n * \n */\n\nstatic CYTHON_INLINE PyObject *__pyx_f_5numpy_PyArray_MultiIterNew1(PyObject *__pyx_v_a) {\n PyObject *__pyx_r = NULL;\n __Pyx_RefNannyDeclarations\n PyObject *__pyx_t_1 = NULL;\n __Pyx_RefNannySetupContext(\"PyArray_MultiIterNew1\", 0);\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":771\n * \n * cdef inline object PyArray_MultiIterNew1(a):\n * return PyArray_MultiIterNew(1, a) # <<<<<<<<<<<<<<\n * \n * cdef inline object PyArray_MultiIterNew2(a, b):\n */\n __Pyx_XDECREF(__pyx_r);\n __pyx_t_1 = PyArray_MultiIterNew(1, ((void *)__pyx_v_a)); if (unlikely(!__pyx_t_1)) __PYX_ERR(1, 771, __pyx_L1_error)\n __Pyx_GOTREF(__pyx_t_1);\n __pyx_r = __pyx_t_1;\n __pyx_t_1 = 0;\n goto __pyx_L0;\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":770\n * ctypedef npy_cdouble complex_t\n * \n * cdef inline object PyArray_MultiIterNew1(a): # <<<<<<<<<<<<<<\n * return PyArray_MultiIterNew(1, a)\n * \n */\n\n /* function exit code */\n __pyx_L1_error:;\n __Pyx_XDECREF(__pyx_t_1);\n __Pyx_AddTraceback(\"numpy.PyArray_MultiIterNew1\", __pyx_clineno, __pyx_lineno, __pyx_filename);\n __pyx_r = 0;\n __pyx_L0:;\n __Pyx_XGIVEREF(__pyx_r);\n __Pyx_RefNannyFinishContext();\n return __pyx_r;\n}\n\n/* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":773\n * return PyArray_MultiIterNew(1, a)\n * \n * cdef inline object PyArray_MultiIterNew2(a, b): # <<<<<<<<<<<<<<\n * return PyArray_MultiIterNew(2, a, b)\n * \n */\n\nstatic CYTHON_INLINE PyObject *__pyx_f_5numpy_PyArray_MultiIterNew2(PyObject *__pyx_v_a, PyObject *__pyx_v_b) {\n PyObject *__pyx_r = NULL;\n __Pyx_RefNannyDeclarations\n PyObject *__pyx_t_1 = NULL;\n __Pyx_RefNannySetupContext(\"PyArray_MultiIterNew2\", 0);\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":774\n * \n * cdef inline object PyArray_MultiIterNew2(a, b):\n * return PyArray_MultiIterNew(2, a, b) # <<<<<<<<<<<<<<\n * \n * cdef inline object PyArray_MultiIterNew3(a, b, c):\n */\n __Pyx_XDECREF(__pyx_r);\n __pyx_t_1 = PyArray_MultiIterNew(2, ((void *)__pyx_v_a), ((void *)__pyx_v_b)); if (unlikely(!__pyx_t_1)) __PYX_ERR(1, 774, __pyx_L1_error)\n __Pyx_GOTREF(__pyx_t_1);\n __pyx_r = __pyx_t_1;\n __pyx_t_1 = 0;\n goto __pyx_L0;\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":773\n * return PyArray_MultiIterNew(1, a)\n * \n * cdef inline object PyArray_MultiIterNew2(a, b): # <<<<<<<<<<<<<<\n * return PyArray_MultiIterNew(2, a, b)\n * \n */\n\n /* function exit code */\n __pyx_L1_error:;\n __Pyx_XDECREF(__pyx_t_1);\n __Pyx_AddTraceback(\"numpy.PyArray_MultiIterNew2\", __pyx_clineno, __pyx_lineno, __pyx_filename);\n __pyx_r = 0;\n __pyx_L0:;\n __Pyx_XGIVEREF(__pyx_r);\n __Pyx_RefNannyFinishContext();\n return __pyx_r;\n}\n\n/* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":776\n * return PyArray_MultiIterNew(2, a, b)\n * \n * cdef inline object PyArray_MultiIterNew3(a, b, c): # <<<<<<<<<<<<<<\n * return PyArray_MultiIterNew(3, a, b, c)\n * \n */\n\nstatic CYTHON_INLINE PyObject *__pyx_f_5numpy_PyArray_MultiIterNew3(PyObject *__pyx_v_a, PyObject *__pyx_v_b, PyObject *__pyx_v_c) {\n PyObject *__pyx_r = NULL;\n __Pyx_RefNannyDeclarations\n PyObject *__pyx_t_1 = NULL;\n __Pyx_RefNannySetupContext(\"PyArray_MultiIterNew3\", 0);\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":777\n * \n * cdef inline object PyArray_MultiIterNew3(a, b, c):\n * return PyArray_MultiIterNew(3, a, b, c) # <<<<<<<<<<<<<<\n * \n * cdef inline object PyArray_MultiIterNew4(a, b, c, d):\n */\n __Pyx_XDECREF(__pyx_r);\n __pyx_t_1 = PyArray_MultiIterNew(3, ((void *)__pyx_v_a), ((void *)__pyx_v_b), ((void *)__pyx_v_c)); if (unlikely(!__pyx_t_1)) __PYX_ERR(1, 777, __pyx_L1_error)\n __Pyx_GOTREF(__pyx_t_1);\n __pyx_r = __pyx_t_1;\n __pyx_t_1 = 0;\n goto __pyx_L0;\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":776\n * return PyArray_MultiIterNew(2, a, b)\n * \n * cdef inline object PyArray_MultiIterNew3(a, b, c): # <<<<<<<<<<<<<<\n * return PyArray_MultiIterNew(3, a, b, c)\n * \n */\n\n /* function exit code */\n __pyx_L1_error:;\n __Pyx_XDECREF(__pyx_t_1);\n __Pyx_AddTraceback(\"numpy.PyArray_MultiIterNew3\", __pyx_clineno, __pyx_lineno, __pyx_filename);\n __pyx_r = 0;\n __pyx_L0:;\n __Pyx_XGIVEREF(__pyx_r);\n __Pyx_RefNannyFinishContext();\n return __pyx_r;\n}\n\n/* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":779\n * return PyArray_MultiIterNew(3, a, b, c)\n * \n * cdef inline object PyArray_MultiIterNew4(a, b, c, d): # <<<<<<<<<<<<<<\n * return PyArray_MultiIterNew(4, a, b, c, d)\n * \n */\n\nstatic CYTHON_INLINE PyObject *__pyx_f_5numpy_PyArray_MultiIterNew4(PyObject *__pyx_v_a, PyObject *__pyx_v_b, PyObject *__pyx_v_c, PyObject *__pyx_v_d) {\n PyObject *__pyx_r = NULL;\n __Pyx_RefNannyDeclarations\n PyObject *__pyx_t_1 = NULL;\n __Pyx_RefNannySetupContext(\"PyArray_MultiIterNew4\", 0);\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":780\n * \n * cdef inline object PyArray_MultiIterNew4(a, b, c, d):\n * return PyArray_MultiIterNew(4, a, b, c, d) # <<<<<<<<<<<<<<\n * \n * cdef inline object PyArray_MultiIterNew5(a, b, c, d, e):\n */\n __Pyx_XDECREF(__pyx_r);\n __pyx_t_1 = PyArray_MultiIterNew(4, ((void *)__pyx_v_a), ((void *)__pyx_v_b), ((void *)__pyx_v_c), ((void *)__pyx_v_d)); if (unlikely(!__pyx_t_1)) __PYX_ERR(1, 780, __pyx_L1_error)\n __Pyx_GOTREF(__pyx_t_1);\n __pyx_r = __pyx_t_1;\n __pyx_t_1 = 0;\n goto __pyx_L0;\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":779\n * return PyArray_MultiIterNew(3, a, b, c)\n * \n * cdef inline object PyArray_MultiIterNew4(a, b, c, d): # <<<<<<<<<<<<<<\n * return PyArray_MultiIterNew(4, a, b, c, d)\n * \n */\n\n /* function exit code */\n __pyx_L1_error:;\n __Pyx_XDECREF(__pyx_t_1);\n __Pyx_AddTraceback(\"numpy.PyArray_MultiIterNew4\", __pyx_clineno, __pyx_lineno, __pyx_filename);\n __pyx_r = 0;\n __pyx_L0:;\n __Pyx_XGIVEREF(__pyx_r);\n __Pyx_RefNannyFinishContext();\n return __pyx_r;\n}\n\n/* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":782\n * return PyArray_MultiIterNew(4, a, b, c, d)\n * \n * cdef inline object PyArray_MultiIterNew5(a, b, c, d, e): # <<<<<<<<<<<<<<\n * return PyArray_MultiIterNew(5, a, b, c, d, e)\n * \n */\n\nstatic CYTHON_INLINE PyObject *__pyx_f_5numpy_PyArray_MultiIterNew5(PyObject *__pyx_v_a, PyObject *__pyx_v_b, PyObject *__pyx_v_c, PyObject *__pyx_v_d, PyObject *__pyx_v_e) {\n PyObject *__pyx_r = NULL;\n __Pyx_RefNannyDeclarations\n PyObject *__pyx_t_1 = NULL;\n __Pyx_RefNannySetupContext(\"PyArray_MultiIterNew5\", 0);\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":783\n * \n * cdef inline object PyArray_MultiIterNew5(a, b, c, d, e):\n * return PyArray_MultiIterNew(5, a, b, c, d, e) # <<<<<<<<<<<<<<\n * \n * cdef inline char* _util_dtypestring(dtype descr, char* f, char* end, int* offset) except NULL:\n */\n __Pyx_XDECREF(__pyx_r);\n __pyx_t_1 = PyArray_MultiIterNew(5, ((void *)__pyx_v_a), ((void *)__pyx_v_b), ((void *)__pyx_v_c), ((void *)__pyx_v_d), ((void *)__pyx_v_e)); if (unlikely(!__pyx_t_1)) __PYX_ERR(1, 783, __pyx_L1_error)\n __Pyx_GOTREF(__pyx_t_1);\n __pyx_r = __pyx_t_1;\n __pyx_t_1 = 0;\n goto __pyx_L0;\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":782\n * return PyArray_MultiIterNew(4, a, b, c, d)\n * \n * cdef inline object PyArray_MultiIterNew5(a, b, c, d, e): # <<<<<<<<<<<<<<\n * return PyArray_MultiIterNew(5, a, b, c, d, e)\n * \n */\n\n /* function exit code */\n __pyx_L1_error:;\n __Pyx_XDECREF(__pyx_t_1);\n __Pyx_AddTraceback(\"numpy.PyArray_MultiIterNew5\", __pyx_clineno, __pyx_lineno, __pyx_filename);\n __pyx_r = 0;\n __pyx_L0:;\n __Pyx_XGIVEREF(__pyx_r);\n __Pyx_RefNannyFinishContext();\n return __pyx_r;\n}\n\n/* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":785\n * return PyArray_MultiIterNew(5, a, b, c, d, e)\n * \n * cdef inline char* _util_dtypestring(dtype descr, char* f, char* end, int* offset) except NULL: # <<<<<<<<<<<<<<\n * # Recursive utility function used in __getbuffer__ to get format\n * # string. The new location in the format string is returned.\n */\n\nstatic CYTHON_INLINE char *__pyx_f_5numpy__util_dtypestring(PyArray_Descr *__pyx_v_descr, char *__pyx_v_f, char *__pyx_v_end, int *__pyx_v_offset) {\n PyArray_Descr *__pyx_v_child = 0;\n int __pyx_v_endian_detector;\n int __pyx_v_little_endian;\n PyObject *__pyx_v_fields = 0;\n PyObject *__pyx_v_childname = NULL;\n PyObject *__pyx_v_new_offset = NULL;\n PyObject *__pyx_v_t = NULL;\n char *__pyx_r;\n __Pyx_RefNannyDeclarations\n PyObject *__pyx_t_1 = NULL;\n Py_ssize_t __pyx_t_2;\n PyObject *__pyx_t_3 = NULL;\n PyObject *__pyx_t_4 = NULL;\n int __pyx_t_5;\n int __pyx_t_6;\n int __pyx_t_7;\n long __pyx_t_8;\n char *__pyx_t_9;\n __Pyx_RefNannySetupContext(\"_util_dtypestring\", 0);\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":790\n * \n * cdef dtype child\n * cdef int endian_detector = 1 # <<<<<<<<<<<<<<\n * cdef bint little_endian = ((&endian_detector)[0] != 0)\n * cdef tuple fields\n */\n __pyx_v_endian_detector = 1;\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":791\n * cdef dtype child\n * cdef int endian_detector = 1\n * cdef bint little_endian = ((&endian_detector)[0] != 0) # <<<<<<<<<<<<<<\n * cdef tuple fields\n * \n */\n __pyx_v_little_endian = ((((char *)(&__pyx_v_endian_detector))[0]) != 0);\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":794\n * cdef tuple fields\n * \n * for childname in descr.names: # <<<<<<<<<<<<<<\n * fields = descr.fields[childname]\n * child, new_offset = fields\n */\n if (unlikely(__pyx_v_descr->names == Py_None)) {\n PyErr_SetString(PyExc_TypeError, \"'NoneType' object is not iterable\");\n __PYX_ERR(1, 794, __pyx_L1_error)\n }\n __pyx_t_1 = __pyx_v_descr->names; __Pyx_INCREF(__pyx_t_1); __pyx_t_2 = 0;\n for (;;) {\n if (__pyx_t_2 >= PyTuple_GET_SIZE(__pyx_t_1)) break;\n #if CYTHON_COMPILING_IN_CPYTHON\n __pyx_t_3 = PyTuple_GET_ITEM(__pyx_t_1, __pyx_t_2); __Pyx_INCREF(__pyx_t_3); __pyx_t_2++; if (unlikely(0 < 0)) __PYX_ERR(1, 794, __pyx_L1_error)\n #else\n __pyx_t_3 = PySequence_ITEM(__pyx_t_1, __pyx_t_2); __pyx_t_2++; if (unlikely(!__pyx_t_3)) __PYX_ERR(1, 794, __pyx_L1_error)\n __Pyx_GOTREF(__pyx_t_3);\n #endif\n __Pyx_XDECREF_SET(__pyx_v_childname, __pyx_t_3);\n __pyx_t_3 = 0;\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":795\n * \n * for childname in descr.names:\n * fields = descr.fields[childname] # <<<<<<<<<<<<<<\n * child, new_offset = fields\n * \n */\n if (unlikely(__pyx_v_descr->fields == Py_None)) {\n PyErr_SetString(PyExc_TypeError, \"'NoneType' object is not subscriptable\");\n __PYX_ERR(1, 795, __pyx_L1_error)\n }\n __pyx_t_3 = __Pyx_PyDict_GetItem(__pyx_v_descr->fields, __pyx_v_childname); if (unlikely(!__pyx_t_3)) __PYX_ERR(1, 795, __pyx_L1_error)\n __Pyx_GOTREF(__pyx_t_3);\n if (!(likely(PyTuple_CheckExact(__pyx_t_3))||((__pyx_t_3) == Py_None)||(PyErr_Format(PyExc_TypeError, \"Expected %.16s, got %.200s\", \"tuple\", Py_TYPE(__pyx_t_3)->tp_name), 0))) __PYX_ERR(1, 795, __pyx_L1_error)\n __Pyx_XDECREF_SET(__pyx_v_fields, ((PyObject*)__pyx_t_3));\n __pyx_t_3 = 0;\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":796\n * for childname in descr.names:\n * fields = descr.fields[childname]\n * child, new_offset = fields # <<<<<<<<<<<<<<\n * \n * if (end - f) - (new_offset - offset[0]) < 15:\n */\n if (likely(__pyx_v_fields != Py_None)) {\n PyObject* sequence = __pyx_v_fields;\n #if CYTHON_COMPILING_IN_CPYTHON\n Py_ssize_t size = Py_SIZE(sequence);\n #else\n Py_ssize_t size = PySequence_Size(sequence);\n #endif\n if (unlikely(size != 2)) {\n if (size > 2) __Pyx_RaiseTooManyValuesError(2);\n else if (size >= 0) __Pyx_RaiseNeedMoreValuesError(size);\n __PYX_ERR(1, 796, __pyx_L1_error)\n }\n #if CYTHON_COMPILING_IN_CPYTHON\n __pyx_t_3 = PyTuple_GET_ITEM(sequence, 0); \n __pyx_t_4 = PyTuple_GET_ITEM(sequence, 1); \n __Pyx_INCREF(__pyx_t_3);\n __Pyx_INCREF(__pyx_t_4);\n #else\n __pyx_t_3 = PySequence_ITEM(sequence, 0); if (unlikely(!__pyx_t_3)) __PYX_ERR(1, 796, __pyx_L1_error)\n __Pyx_GOTREF(__pyx_t_3);\n __pyx_t_4 = PySequence_ITEM(sequence, 1); if (unlikely(!__pyx_t_4)) __PYX_ERR(1, 796, __pyx_L1_error)\n __Pyx_GOTREF(__pyx_t_4);\n #endif\n } else {\n __Pyx_RaiseNoneNotIterableError(); __PYX_ERR(1, 796, __pyx_L1_error)\n }\n if (!(likely(((__pyx_t_3) == Py_None) || likely(__Pyx_TypeTest(__pyx_t_3, __pyx_ptype_5numpy_dtype))))) __PYX_ERR(1, 796, __pyx_L1_error)\n __Pyx_XDECREF_SET(__pyx_v_child, ((PyArray_Descr *)__pyx_t_3));\n __pyx_t_3 = 0;\n __Pyx_XDECREF_SET(__pyx_v_new_offset, __pyx_t_4);\n __pyx_t_4 = 0;\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":798\n * child, new_offset = fields\n * \n * if (end - f) - (new_offset - offset[0]) < 15: # <<<<<<<<<<<<<<\n * raise RuntimeError(u\"Format string allocated too short, see comment in numpy.pxd\")\n * \n */\n __pyx_t_4 = __Pyx_PyInt_From_int((__pyx_v_offset[0])); if (unlikely(!__pyx_t_4)) __PYX_ERR(1, 798, __pyx_L1_error)\n __Pyx_GOTREF(__pyx_t_4);\n __pyx_t_3 = PyNumber_Subtract(__pyx_v_new_offset, __pyx_t_4); if (unlikely(!__pyx_t_3)) __PYX_ERR(1, 798, __pyx_L1_error)\n __Pyx_GOTREF(__pyx_t_3);\n __Pyx_DECREF(__pyx_t_4); __pyx_t_4 = 0;\n __pyx_t_5 = __Pyx_PyInt_As_int(__pyx_t_3); if (unlikely((__pyx_t_5 == (int)-1) && PyErr_Occurred())) __PYX_ERR(1, 798, __pyx_L1_error)\n __Pyx_DECREF(__pyx_t_3); __pyx_t_3 = 0;\n __pyx_t_6 = ((((__pyx_v_end - __pyx_v_f) - ((int)__pyx_t_5)) < 15) != 0);\n if (__pyx_t_6) {\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":799\n * \n * if (end - f) - (new_offset - offset[0]) < 15:\n * raise RuntimeError(u\"Format string allocated too short, see comment in numpy.pxd\") # <<<<<<<<<<<<<<\n * \n * if ((child.byteorder == c'>' and little_endian) or\n */\n __pyx_t_3 = __Pyx_PyObject_Call(__pyx_builtin_RuntimeError, __pyx_tuple__8, NULL); if (unlikely(!__pyx_t_3)) __PYX_ERR(1, 799, __pyx_L1_error)\n __Pyx_GOTREF(__pyx_t_3);\n __Pyx_Raise(__pyx_t_3, 0, 0, 0);\n __Pyx_DECREF(__pyx_t_3); __pyx_t_3 = 0;\n __PYX_ERR(1, 799, __pyx_L1_error)\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":798\n * child, new_offset = fields\n * \n * if (end - f) - (new_offset - offset[0]) < 15: # <<<<<<<<<<<<<<\n * raise RuntimeError(u\"Format string allocated too short, see comment in numpy.pxd\")\n * \n */\n }\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":801\n * raise RuntimeError(u\"Format string allocated too short, see comment in numpy.pxd\")\n * \n * if ((child.byteorder == c'>' and little_endian) or # <<<<<<<<<<<<<<\n * (child.byteorder == c'<' and not little_endian)):\n * raise ValueError(u\"Non-native byte order not supported\")\n */\n __pyx_t_7 = ((__pyx_v_child->byteorder == '>') != 0);\n if (!__pyx_t_7) {\n goto __pyx_L8_next_or;\n } else {\n }\n __pyx_t_7 = (__pyx_v_little_endian != 0);\n if (!__pyx_t_7) {\n } else {\n __pyx_t_6 = __pyx_t_7;\n goto __pyx_L7_bool_binop_done;\n }\n __pyx_L8_next_or:;\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":802\n * \n * if ((child.byteorder == c'>' and little_endian) or\n * (child.byteorder == c'<' and not little_endian)): # <<<<<<<<<<<<<<\n * raise ValueError(u\"Non-native byte order not supported\")\n * # One could encode it in the format string and have Cython\n */\n __pyx_t_7 = ((__pyx_v_child->byteorder == '<') != 0);\n if (__pyx_t_7) {\n } else {\n __pyx_t_6 = __pyx_t_7;\n goto __pyx_L7_bool_binop_done;\n }\n __pyx_t_7 = ((!(__pyx_v_little_endian != 0)) != 0);\n __pyx_t_6 = __pyx_t_7;\n __pyx_L7_bool_binop_done:;\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":801\n * raise RuntimeError(u\"Format string allocated too short, see comment in numpy.pxd\")\n * \n * if ((child.byteorder == c'>' and little_endian) or # <<<<<<<<<<<<<<\n * (child.byteorder == c'<' and not little_endian)):\n * raise ValueError(u\"Non-native byte order not supported\")\n */\n if (__pyx_t_6) {\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":803\n * if ((child.byteorder == c'>' and little_endian) or\n * (child.byteorder == c'<' and not little_endian)):\n * raise ValueError(u\"Non-native byte order not supported\") # <<<<<<<<<<<<<<\n * # One could encode it in the format string and have Cython\n * # complain instead, BUT: < and > in format strings also imply\n */\n __pyx_t_3 = __Pyx_PyObject_Call(__pyx_builtin_ValueError, __pyx_tuple__9, NULL); if (unlikely(!__pyx_t_3)) __PYX_ERR(1, 803, __pyx_L1_error)\n __Pyx_GOTREF(__pyx_t_3);\n __Pyx_Raise(__pyx_t_3, 0, 0, 0);\n __Pyx_DECREF(__pyx_t_3); __pyx_t_3 = 0;\n __PYX_ERR(1, 803, __pyx_L1_error)\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":801\n * raise RuntimeError(u\"Format string allocated too short, see comment in numpy.pxd\")\n * \n * if ((child.byteorder == c'>' and little_endian) or # <<<<<<<<<<<<<<\n * (child.byteorder == c'<' and not little_endian)):\n * raise ValueError(u\"Non-native byte order not supported\")\n */\n }\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":813\n * \n * # Output padding bytes\n * while offset[0] < new_offset: # <<<<<<<<<<<<<<\n * f[0] = 120 # \"x\"; pad byte\n * f += 1\n */\n while (1) {\n __pyx_t_3 = __Pyx_PyInt_From_int((__pyx_v_offset[0])); if (unlikely(!__pyx_t_3)) __PYX_ERR(1, 813, __pyx_L1_error)\n __Pyx_GOTREF(__pyx_t_3);\n __pyx_t_4 = PyObject_RichCompare(__pyx_t_3, __pyx_v_new_offset, Py_LT); __Pyx_XGOTREF(__pyx_t_4); if (unlikely(!__pyx_t_4)) __PYX_ERR(1, 813, __pyx_L1_error)\n __Pyx_DECREF(__pyx_t_3); __pyx_t_3 = 0;\n __pyx_t_6 = __Pyx_PyObject_IsTrue(__pyx_t_4); if (unlikely(__pyx_t_6 < 0)) __PYX_ERR(1, 813, __pyx_L1_error)\n __Pyx_DECREF(__pyx_t_4); __pyx_t_4 = 0;\n if (!__pyx_t_6) break;\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":814\n * # Output padding bytes\n * while offset[0] < new_offset:\n * f[0] = 120 # \"x\"; pad byte # <<<<<<<<<<<<<<\n * f += 1\n * offset[0] += 1\n */\n (__pyx_v_f[0]) = 0x78;\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":815\n * while offset[0] < new_offset:\n * f[0] = 120 # \"x\"; pad byte\n * f += 1 # <<<<<<<<<<<<<<\n * offset[0] += 1\n * \n */\n __pyx_v_f = (__pyx_v_f + 1);\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":816\n * f[0] = 120 # \"x\"; pad byte\n * f += 1\n * offset[0] += 1 # <<<<<<<<<<<<<<\n * \n * offset[0] += child.itemsize\n */\n __pyx_t_8 = 0;\n (__pyx_v_offset[__pyx_t_8]) = ((__pyx_v_offset[__pyx_t_8]) + 1);\n }\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":818\n * offset[0] += 1\n * \n * offset[0] += child.itemsize # <<<<<<<<<<<<<<\n * \n * if not PyDataType_HASFIELDS(child):\n */\n __pyx_t_8 = 0;\n (__pyx_v_offset[__pyx_t_8]) = ((__pyx_v_offset[__pyx_t_8]) + __pyx_v_child->elsize);\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":820\n * offset[0] += child.itemsize\n * \n * if not PyDataType_HASFIELDS(child): # <<<<<<<<<<<<<<\n * t = child.type_num\n * if end - f < 5:\n */\n __pyx_t_6 = ((!(PyDataType_HASFIELDS(__pyx_v_child) != 0)) != 0);\n if (__pyx_t_6) {\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":821\n * \n * if not PyDataType_HASFIELDS(child):\n * t = child.type_num # <<<<<<<<<<<<<<\n * if end - f < 5:\n * raise RuntimeError(u\"Format string allocated too short.\")\n */\n __pyx_t_4 = __Pyx_PyInt_From_int(__pyx_v_child->type_num); if (unlikely(!__pyx_t_4)) __PYX_ERR(1, 821, __pyx_L1_error)\n __Pyx_GOTREF(__pyx_t_4);\n __Pyx_XDECREF_SET(__pyx_v_t, __pyx_t_4);\n __pyx_t_4 = 0;\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":822\n * if not PyDataType_HASFIELDS(child):\n * t = child.type_num\n * if end - f < 5: # <<<<<<<<<<<<<<\n * raise RuntimeError(u\"Format string allocated too short.\")\n * \n */\n __pyx_t_6 = (((__pyx_v_end - __pyx_v_f) < 5) != 0);\n if (__pyx_t_6) {\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":823\n * t = child.type_num\n * if end - f < 5:\n * raise RuntimeError(u\"Format string allocated too short.\") # <<<<<<<<<<<<<<\n * \n * # Until ticket #99 is fixed, use integers to avoid warnings\n */\n __pyx_t_4 = __Pyx_PyObject_Call(__pyx_builtin_RuntimeError, __pyx_tuple__10, NULL); if (unlikely(!__pyx_t_4)) __PYX_ERR(1, 823, __pyx_L1_error)\n __Pyx_GOTREF(__pyx_t_4);\n __Pyx_Raise(__pyx_t_4, 0, 0, 0);\n __Pyx_DECREF(__pyx_t_4); __pyx_t_4 = 0;\n __PYX_ERR(1, 823, __pyx_L1_error)\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":822\n * if not PyDataType_HASFIELDS(child):\n * t = child.type_num\n * if end - f < 5: # <<<<<<<<<<<<<<\n * raise RuntimeError(u\"Format string allocated too short.\")\n * \n */\n }\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":826\n * \n * # Until ticket #99 is fixed, use integers to avoid warnings\n * if t == NPY_BYTE: f[0] = 98 #\"b\" # <<<<<<<<<<<<<<\n * elif t == NPY_UBYTE: f[0] = 66 #\"B\"\n * elif t == NPY_SHORT: f[0] = 104 #\"h\"\n */\n __pyx_t_4 = __Pyx_PyInt_From_enum__NPY_TYPES(NPY_BYTE); if (unlikely(!__pyx_t_4)) __PYX_ERR(1, 826, __pyx_L1_error)\n __Pyx_GOTREF(__pyx_t_4);\n __pyx_t_3 = PyObject_RichCompare(__pyx_v_t, __pyx_t_4, Py_EQ); __Pyx_XGOTREF(__pyx_t_3); if (unlikely(!__pyx_t_3)) __PYX_ERR(1, 826, __pyx_L1_error)\n __Pyx_DECREF(__pyx_t_4); __pyx_t_4 = 0;\n __pyx_t_6 = __Pyx_PyObject_IsTrue(__pyx_t_3); if (unlikely(__pyx_t_6 < 0)) __PYX_ERR(1, 826, __pyx_L1_error)\n __Pyx_DECREF(__pyx_t_3); __pyx_t_3 = 0;\n if (__pyx_t_6) {\n (__pyx_v_f[0]) = 98;\n goto __pyx_L15;\n }\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":827\n * # Until ticket #99 is fixed, use integers to avoid warnings\n * if t == NPY_BYTE: f[0] = 98 #\"b\"\n * elif t == NPY_UBYTE: f[0] = 66 #\"B\" # <<<<<<<<<<<<<<\n * elif t == NPY_SHORT: f[0] = 104 #\"h\"\n * elif t == NPY_USHORT: f[0] = 72 #\"H\"\n */\n __pyx_t_3 = __Pyx_PyInt_From_enum__NPY_TYPES(NPY_UBYTE); if (unlikely(!__pyx_t_3)) __PYX_ERR(1, 827, __pyx_L1_error)\n __Pyx_GOTREF(__pyx_t_3);\n __pyx_t_4 = PyObject_RichCompare(__pyx_v_t, __pyx_t_3, Py_EQ); __Pyx_XGOTREF(__pyx_t_4); if (unlikely(!__pyx_t_4)) __PYX_ERR(1, 827, __pyx_L1_error)\n __Pyx_DECREF(__pyx_t_3); __pyx_t_3 = 0;\n __pyx_t_6 = __Pyx_PyObject_IsTrue(__pyx_t_4); if (unlikely(__pyx_t_6 < 0)) __PYX_ERR(1, 827, __pyx_L1_error)\n __Pyx_DECREF(__pyx_t_4); __pyx_t_4 = 0;\n if (__pyx_t_6) {\n (__pyx_v_f[0]) = 66;\n goto __pyx_L15;\n }\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":828\n * if t == NPY_BYTE: f[0] = 98 #\"b\"\n * elif t == NPY_UBYTE: f[0] = 66 #\"B\"\n * elif t == NPY_SHORT: f[0] = 104 #\"h\" # <<<<<<<<<<<<<<\n * elif t == NPY_USHORT: f[0] = 72 #\"H\"\n * elif t == NPY_INT: f[0] = 105 #\"i\"\n */\n __pyx_t_4 = __Pyx_PyInt_From_enum__NPY_TYPES(NPY_SHORT); if (unlikely(!__pyx_t_4)) __PYX_ERR(1, 828, __pyx_L1_error)\n __Pyx_GOTREF(__pyx_t_4);\n __pyx_t_3 = PyObject_RichCompare(__pyx_v_t, __pyx_t_4, Py_EQ); __Pyx_XGOTREF(__pyx_t_3); if (unlikely(!__pyx_t_3)) __PYX_ERR(1, 828, __pyx_L1_error)\n __Pyx_DECREF(__pyx_t_4); __pyx_t_4 = 0;\n __pyx_t_6 = __Pyx_PyObject_IsTrue(__pyx_t_3); if (unlikely(__pyx_t_6 < 0)) __PYX_ERR(1, 828, __pyx_L1_error)\n __Pyx_DECREF(__pyx_t_3); __pyx_t_3 = 0;\n if (__pyx_t_6) {\n (__pyx_v_f[0]) = 0x68;\n goto __pyx_L15;\n }\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":829\n * elif t == NPY_UBYTE: f[0] = 66 #\"B\"\n * elif t == NPY_SHORT: f[0] = 104 #\"h\"\n * elif t == NPY_USHORT: f[0] = 72 #\"H\" # <<<<<<<<<<<<<<\n * elif t == NPY_INT: f[0] = 105 #\"i\"\n * elif t == NPY_UINT: f[0] = 73 #\"I\"\n */\n __pyx_t_3 = __Pyx_PyInt_From_enum__NPY_TYPES(NPY_USHORT); if (unlikely(!__pyx_t_3)) __PYX_ERR(1, 829, __pyx_L1_error)\n __Pyx_GOTREF(__pyx_t_3);\n __pyx_t_4 = PyObject_RichCompare(__pyx_v_t, __pyx_t_3, Py_EQ); __Pyx_XGOTREF(__pyx_t_4); if (unlikely(!__pyx_t_4)) __PYX_ERR(1, 829, __pyx_L1_error)\n __Pyx_DECREF(__pyx_t_3); __pyx_t_3 = 0;\n __pyx_t_6 = __Pyx_PyObject_IsTrue(__pyx_t_4); if (unlikely(__pyx_t_6 < 0)) __PYX_ERR(1, 829, __pyx_L1_error)\n __Pyx_DECREF(__pyx_t_4); __pyx_t_4 = 0;\n if (__pyx_t_6) {\n (__pyx_v_f[0]) = 72;\n goto __pyx_L15;\n }\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":830\n * elif t == NPY_SHORT: f[0] = 104 #\"h\"\n * elif t == NPY_USHORT: f[0] = 72 #\"H\"\n * elif t == NPY_INT: f[0] = 105 #\"i\" # <<<<<<<<<<<<<<\n * elif t == NPY_UINT: f[0] = 73 #\"I\"\n * elif t == NPY_LONG: f[0] = 108 #\"l\"\n */\n __pyx_t_4 = __Pyx_PyInt_From_enum__NPY_TYPES(NPY_INT); if (unlikely(!__pyx_t_4)) __PYX_ERR(1, 830, __pyx_L1_error)\n __Pyx_GOTREF(__pyx_t_4);\n __pyx_t_3 = PyObject_RichCompare(__pyx_v_t, __pyx_t_4, Py_EQ); __Pyx_XGOTREF(__pyx_t_3); if (unlikely(!__pyx_t_3)) __PYX_ERR(1, 830, __pyx_L1_error)\n __Pyx_DECREF(__pyx_t_4); __pyx_t_4 = 0;\n __pyx_t_6 = __Pyx_PyObject_IsTrue(__pyx_t_3); if (unlikely(__pyx_t_6 < 0)) __PYX_ERR(1, 830, __pyx_L1_error)\n __Pyx_DECREF(__pyx_t_3); __pyx_t_3 = 0;\n if (__pyx_t_6) {\n (__pyx_v_f[0]) = 0x69;\n goto __pyx_L15;\n }\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":831\n * elif t == NPY_USHORT: f[0] = 72 #\"H\"\n * elif t == NPY_INT: f[0] = 105 #\"i\"\n * elif t == NPY_UINT: f[0] = 73 #\"I\" # <<<<<<<<<<<<<<\n * elif t == NPY_LONG: f[0] = 108 #\"l\"\n * elif t == NPY_ULONG: f[0] = 76 #\"L\"\n */\n __pyx_t_3 = __Pyx_PyInt_From_enum__NPY_TYPES(NPY_UINT); if (unlikely(!__pyx_t_3)) __PYX_ERR(1, 831, __pyx_L1_error)\n __Pyx_GOTREF(__pyx_t_3);\n __pyx_t_4 = PyObject_RichCompare(__pyx_v_t, __pyx_t_3, Py_EQ); __Pyx_XGOTREF(__pyx_t_4); if (unlikely(!__pyx_t_4)) __PYX_ERR(1, 831, __pyx_L1_error)\n __Pyx_DECREF(__pyx_t_3); __pyx_t_3 = 0;\n __pyx_t_6 = __Pyx_PyObject_IsTrue(__pyx_t_4); if (unlikely(__pyx_t_6 < 0)) __PYX_ERR(1, 831, __pyx_L1_error)\n __Pyx_DECREF(__pyx_t_4); __pyx_t_4 = 0;\n if (__pyx_t_6) {\n (__pyx_v_f[0]) = 73;\n goto __pyx_L15;\n }\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":832\n * elif t == NPY_INT: f[0] = 105 #\"i\"\n * elif t == NPY_UINT: f[0] = 73 #\"I\"\n * elif t == NPY_LONG: f[0] = 108 #\"l\" # <<<<<<<<<<<<<<\n * elif t == NPY_ULONG: f[0] = 76 #\"L\"\n * elif t == NPY_LONGLONG: f[0] = 113 #\"q\"\n */\n __pyx_t_4 = __Pyx_PyInt_From_enum__NPY_TYPES(NPY_LONG); if (unlikely(!__pyx_t_4)) __PYX_ERR(1, 832, __pyx_L1_error)\n __Pyx_GOTREF(__pyx_t_4);\n __pyx_t_3 = PyObject_RichCompare(__pyx_v_t, __pyx_t_4, Py_EQ); __Pyx_XGOTREF(__pyx_t_3); if (unlikely(!__pyx_t_3)) __PYX_ERR(1, 832, __pyx_L1_error)\n __Pyx_DECREF(__pyx_t_4); __pyx_t_4 = 0;\n __pyx_t_6 = __Pyx_PyObject_IsTrue(__pyx_t_3); if (unlikely(__pyx_t_6 < 0)) __PYX_ERR(1, 832, __pyx_L1_error)\n __Pyx_DECREF(__pyx_t_3); __pyx_t_3 = 0;\n if (__pyx_t_6) {\n (__pyx_v_f[0]) = 0x6C;\n goto __pyx_L15;\n }\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":833\n * elif t == NPY_UINT: f[0] = 73 #\"I\"\n * elif t == NPY_LONG: f[0] = 108 #\"l\"\n * elif t == NPY_ULONG: f[0] = 76 #\"L\" # <<<<<<<<<<<<<<\n * elif t == NPY_LONGLONG: f[0] = 113 #\"q\"\n * elif t == NPY_ULONGLONG: f[0] = 81 #\"Q\"\n */\n __pyx_t_3 = __Pyx_PyInt_From_enum__NPY_TYPES(NPY_ULONG); if (unlikely(!__pyx_t_3)) __PYX_ERR(1, 833, __pyx_L1_error)\n __Pyx_GOTREF(__pyx_t_3);\n __pyx_t_4 = PyObject_RichCompare(__pyx_v_t, __pyx_t_3, Py_EQ); __Pyx_XGOTREF(__pyx_t_4); if (unlikely(!__pyx_t_4)) __PYX_ERR(1, 833, __pyx_L1_error)\n __Pyx_DECREF(__pyx_t_3); __pyx_t_3 = 0;\n __pyx_t_6 = __Pyx_PyObject_IsTrue(__pyx_t_4); if (unlikely(__pyx_t_6 < 0)) __PYX_ERR(1, 833, __pyx_L1_error)\n __Pyx_DECREF(__pyx_t_4); __pyx_t_4 = 0;\n if (__pyx_t_6) {\n (__pyx_v_f[0]) = 76;\n goto __pyx_L15;\n }\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":834\n * elif t == NPY_LONG: f[0] = 108 #\"l\"\n * elif t == NPY_ULONG: f[0] = 76 #\"L\"\n * elif t == NPY_LONGLONG: f[0] = 113 #\"q\" # <<<<<<<<<<<<<<\n * elif t == NPY_ULONGLONG: f[0] = 81 #\"Q\"\n * elif t == NPY_FLOAT: f[0] = 102 #\"f\"\n */\n __pyx_t_4 = __Pyx_PyInt_From_enum__NPY_TYPES(NPY_LONGLONG); if (unlikely(!__pyx_t_4)) __PYX_ERR(1, 834, __pyx_L1_error)\n __Pyx_GOTREF(__pyx_t_4);\n __pyx_t_3 = PyObject_RichCompare(__pyx_v_t, __pyx_t_4, Py_EQ); __Pyx_XGOTREF(__pyx_t_3); if (unlikely(!__pyx_t_3)) __PYX_ERR(1, 834, __pyx_L1_error)\n __Pyx_DECREF(__pyx_t_4); __pyx_t_4 = 0;\n __pyx_t_6 = __Pyx_PyObject_IsTrue(__pyx_t_3); if (unlikely(__pyx_t_6 < 0)) __PYX_ERR(1, 834, __pyx_L1_error)\n __Pyx_DECREF(__pyx_t_3); __pyx_t_3 = 0;\n if (__pyx_t_6) {\n (__pyx_v_f[0]) = 0x71;\n goto __pyx_L15;\n }\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":835\n * elif t == NPY_ULONG: f[0] = 76 #\"L\"\n * elif t == NPY_LONGLONG: f[0] = 113 #\"q\"\n * elif t == NPY_ULONGLONG: f[0] = 81 #\"Q\" # <<<<<<<<<<<<<<\n * elif t == NPY_FLOAT: f[0] = 102 #\"f\"\n * elif t == NPY_DOUBLE: f[0] = 100 #\"d\"\n */\n __pyx_t_3 = __Pyx_PyInt_From_enum__NPY_TYPES(NPY_ULONGLONG); if (unlikely(!__pyx_t_3)) __PYX_ERR(1, 835, __pyx_L1_error)\n __Pyx_GOTREF(__pyx_t_3);\n __pyx_t_4 = PyObject_RichCompare(__pyx_v_t, __pyx_t_3, Py_EQ); __Pyx_XGOTREF(__pyx_t_4); if (unlikely(!__pyx_t_4)) __PYX_ERR(1, 835, __pyx_L1_error)\n __Pyx_DECREF(__pyx_t_3); __pyx_t_3 = 0;\n __pyx_t_6 = __Pyx_PyObject_IsTrue(__pyx_t_4); if (unlikely(__pyx_t_6 < 0)) __PYX_ERR(1, 835, __pyx_L1_error)\n __Pyx_DECREF(__pyx_t_4); __pyx_t_4 = 0;\n if (__pyx_t_6) {\n (__pyx_v_f[0]) = 81;\n goto __pyx_L15;\n }\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":836\n * elif t == NPY_LONGLONG: f[0] = 113 #\"q\"\n * elif t == NPY_ULONGLONG: f[0] = 81 #\"Q\"\n * elif t == NPY_FLOAT: f[0] = 102 #\"f\" # <<<<<<<<<<<<<<\n * elif t == NPY_DOUBLE: f[0] = 100 #\"d\"\n * elif t == NPY_LONGDOUBLE: f[0] = 103 #\"g\"\n */\n __pyx_t_4 = __Pyx_PyInt_From_enum__NPY_TYPES(NPY_FLOAT); if (unlikely(!__pyx_t_4)) __PYX_ERR(1, 836, __pyx_L1_error)\n __Pyx_GOTREF(__pyx_t_4);\n __pyx_t_3 = PyObject_RichCompare(__pyx_v_t, __pyx_t_4, Py_EQ); __Pyx_XGOTREF(__pyx_t_3); if (unlikely(!__pyx_t_3)) __PYX_ERR(1, 836, __pyx_L1_error)\n __Pyx_DECREF(__pyx_t_4); __pyx_t_4 = 0;\n __pyx_t_6 = __Pyx_PyObject_IsTrue(__pyx_t_3); if (unlikely(__pyx_t_6 < 0)) __PYX_ERR(1, 836, __pyx_L1_error)\n __Pyx_DECREF(__pyx_t_3); __pyx_t_3 = 0;\n if (__pyx_t_6) {\n (__pyx_v_f[0]) = 0x66;\n goto __pyx_L15;\n }\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":837\n * elif t == NPY_ULONGLONG: f[0] = 81 #\"Q\"\n * elif t == NPY_FLOAT: f[0] = 102 #\"f\"\n * elif t == NPY_DOUBLE: f[0] = 100 #\"d\" # <<<<<<<<<<<<<<\n * elif t == NPY_LONGDOUBLE: f[0] = 103 #\"g\"\n * elif t == NPY_CFLOAT: f[0] = 90; f[1] = 102; f += 1 # Zf\n */\n __pyx_t_3 = __Pyx_PyInt_From_enum__NPY_TYPES(NPY_DOUBLE); if (unlikely(!__pyx_t_3)) __PYX_ERR(1, 837, __pyx_L1_error)\n __Pyx_GOTREF(__pyx_t_3);\n __pyx_t_4 = PyObject_RichCompare(__pyx_v_t, __pyx_t_3, Py_EQ); __Pyx_XGOTREF(__pyx_t_4); if (unlikely(!__pyx_t_4)) __PYX_ERR(1, 837, __pyx_L1_error)\n __Pyx_DECREF(__pyx_t_3); __pyx_t_3 = 0;\n __pyx_t_6 = __Pyx_PyObject_IsTrue(__pyx_t_4); if (unlikely(__pyx_t_6 < 0)) __PYX_ERR(1, 837, __pyx_L1_error)\n __Pyx_DECREF(__pyx_t_4); __pyx_t_4 = 0;\n if (__pyx_t_6) {\n (__pyx_v_f[0]) = 0x64;\n goto __pyx_L15;\n }\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":838\n * elif t == NPY_FLOAT: f[0] = 102 #\"f\"\n * elif t == NPY_DOUBLE: f[0] = 100 #\"d\"\n * elif t == NPY_LONGDOUBLE: f[0] = 103 #\"g\" # <<<<<<<<<<<<<<\n * elif t == NPY_CFLOAT: f[0] = 90; f[1] = 102; f += 1 # Zf\n * elif t == NPY_CDOUBLE: f[0] = 90; f[1] = 100; f += 1 # Zd\n */\n __pyx_t_4 = __Pyx_PyInt_From_enum__NPY_TYPES(NPY_LONGDOUBLE); if (unlikely(!__pyx_t_4)) __PYX_ERR(1, 838, __pyx_L1_error)\n __Pyx_GOTREF(__pyx_t_4);\n __pyx_t_3 = PyObject_RichCompare(__pyx_v_t, __pyx_t_4, Py_EQ); __Pyx_XGOTREF(__pyx_t_3); if (unlikely(!__pyx_t_3)) __PYX_ERR(1, 838, __pyx_L1_error)\n __Pyx_DECREF(__pyx_t_4); __pyx_t_4 = 0;\n __pyx_t_6 = __Pyx_PyObject_IsTrue(__pyx_t_3); if (unlikely(__pyx_t_6 < 0)) __PYX_ERR(1, 838, __pyx_L1_error)\n __Pyx_DECREF(__pyx_t_3); __pyx_t_3 = 0;\n if (__pyx_t_6) {\n (__pyx_v_f[0]) = 0x67;\n goto __pyx_L15;\n }\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":839\n * elif t == NPY_DOUBLE: f[0] = 100 #\"d\"\n * elif t == NPY_LONGDOUBLE: f[0] = 103 #\"g\"\n * elif t == NPY_CFLOAT: f[0] = 90; f[1] = 102; f += 1 # Zf # <<<<<<<<<<<<<<\n * elif t == NPY_CDOUBLE: f[0] = 90; f[1] = 100; f += 1 # Zd\n * elif t == NPY_CLONGDOUBLE: f[0] = 90; f[1] = 103; f += 1 # Zg\n */\n __pyx_t_3 = __Pyx_PyInt_From_enum__NPY_TYPES(NPY_CFLOAT); if (unlikely(!__pyx_t_3)) __PYX_ERR(1, 839, __pyx_L1_error)\n __Pyx_GOTREF(__pyx_t_3);\n __pyx_t_4 = PyObject_RichCompare(__pyx_v_t, __pyx_t_3, Py_EQ); __Pyx_XGOTREF(__pyx_t_4); if (unlikely(!__pyx_t_4)) __PYX_ERR(1, 839, __pyx_L1_error)\n __Pyx_DECREF(__pyx_t_3); __pyx_t_3 = 0;\n __pyx_t_6 = __Pyx_PyObject_IsTrue(__pyx_t_4); if (unlikely(__pyx_t_6 < 0)) __PYX_ERR(1, 839, __pyx_L1_error)\n __Pyx_DECREF(__pyx_t_4); __pyx_t_4 = 0;\n if (__pyx_t_6) {\n (__pyx_v_f[0]) = 90;\n (__pyx_v_f[1]) = 0x66;\n __pyx_v_f = (__pyx_v_f + 1);\n goto __pyx_L15;\n }\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":840\n * elif t == NPY_LONGDOUBLE: f[0] = 103 #\"g\"\n * elif t == NPY_CFLOAT: f[0] = 90; f[1] = 102; f += 1 # Zf\n * elif t == NPY_CDOUBLE: f[0] = 90; f[1] = 100; f += 1 # Zd # <<<<<<<<<<<<<<\n * elif t == NPY_CLONGDOUBLE: f[0] = 90; f[1] = 103; f += 1 # Zg\n * elif t == NPY_OBJECT: f[0] = 79 #\"O\"\n */\n __pyx_t_4 = __Pyx_PyInt_From_enum__NPY_TYPES(NPY_CDOUBLE); if (unlikely(!__pyx_t_4)) __PYX_ERR(1, 840, __pyx_L1_error)\n __Pyx_GOTREF(__pyx_t_4);\n __pyx_t_3 = PyObject_RichCompare(__pyx_v_t, __pyx_t_4, Py_EQ); __Pyx_XGOTREF(__pyx_t_3); if (unlikely(!__pyx_t_3)) __PYX_ERR(1, 840, __pyx_L1_error)\n __Pyx_DECREF(__pyx_t_4); __pyx_t_4 = 0;\n __pyx_t_6 = __Pyx_PyObject_IsTrue(__pyx_t_3); if (unlikely(__pyx_t_6 < 0)) __PYX_ERR(1, 840, __pyx_L1_error)\n __Pyx_DECREF(__pyx_t_3); __pyx_t_3 = 0;\n if (__pyx_t_6) {\n (__pyx_v_f[0]) = 90;\n (__pyx_v_f[1]) = 0x64;\n __pyx_v_f = (__pyx_v_f + 1);\n goto __pyx_L15;\n }\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":841\n * elif t == NPY_CFLOAT: f[0] = 90; f[1] = 102; f += 1 # Zf\n * elif t == NPY_CDOUBLE: f[0] = 90; f[1] = 100; f += 1 # Zd\n * elif t == NPY_CLONGDOUBLE: f[0] = 90; f[1] = 103; f += 1 # Zg # <<<<<<<<<<<<<<\n * elif t == NPY_OBJECT: f[0] = 79 #\"O\"\n * else:\n */\n __pyx_t_3 = __Pyx_PyInt_From_enum__NPY_TYPES(NPY_CLONGDOUBLE); if (unlikely(!__pyx_t_3)) __PYX_ERR(1, 841, __pyx_L1_error)\n __Pyx_GOTREF(__pyx_t_3);\n __pyx_t_4 = PyObject_RichCompare(__pyx_v_t, __pyx_t_3, Py_EQ); __Pyx_XGOTREF(__pyx_t_4); if (unlikely(!__pyx_t_4)) __PYX_ERR(1, 841, __pyx_L1_error)\n __Pyx_DECREF(__pyx_t_3); __pyx_t_3 = 0;\n __pyx_t_6 = __Pyx_PyObject_IsTrue(__pyx_t_4); if (unlikely(__pyx_t_6 < 0)) __PYX_ERR(1, 841, __pyx_L1_error)\n __Pyx_DECREF(__pyx_t_4); __pyx_t_4 = 0;\n if (__pyx_t_6) {\n (__pyx_v_f[0]) = 90;\n (__pyx_v_f[1]) = 0x67;\n __pyx_v_f = (__pyx_v_f + 1);\n goto __pyx_L15;\n }\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":842\n * elif t == NPY_CDOUBLE: f[0] = 90; f[1] = 100; f += 1 # Zd\n * elif t == NPY_CLONGDOUBLE: f[0] = 90; f[1] = 103; f += 1 # Zg\n * elif t == NPY_OBJECT: f[0] = 79 #\"O\" # <<<<<<<<<<<<<<\n * else:\n * raise ValueError(u\"unknown dtype code in numpy.pxd (%d)\" % t)\n */\n __pyx_t_4 = __Pyx_PyInt_From_enum__NPY_TYPES(NPY_OBJECT); if (unlikely(!__pyx_t_4)) __PYX_ERR(1, 842, __pyx_L1_error)\n __Pyx_GOTREF(__pyx_t_4);\n __pyx_t_3 = PyObject_RichCompare(__pyx_v_t, __pyx_t_4, Py_EQ); __Pyx_XGOTREF(__pyx_t_3); if (unlikely(!__pyx_t_3)) __PYX_ERR(1, 842, __pyx_L1_error)\n __Pyx_DECREF(__pyx_t_4); __pyx_t_4 = 0;\n __pyx_t_6 = __Pyx_PyObject_IsTrue(__pyx_t_3); if (unlikely(__pyx_t_6 < 0)) __PYX_ERR(1, 842, __pyx_L1_error)\n __Pyx_DECREF(__pyx_t_3); __pyx_t_3 = 0;\n if (__pyx_t_6) {\n (__pyx_v_f[0]) = 79;\n goto __pyx_L15;\n }\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":844\n * elif t == NPY_OBJECT: f[0] = 79 #\"O\"\n * else:\n * raise ValueError(u\"unknown dtype code in numpy.pxd (%d)\" % t) # <<<<<<<<<<<<<<\n * f += 1\n * else:\n */\n /*else*/ {\n __pyx_t_3 = PyUnicode_Format(__pyx_kp_u_unknown_dtype_code_in_numpy_pxd, __pyx_v_t); if (unlikely(!__pyx_t_3)) __PYX_ERR(1, 844, __pyx_L1_error)\n __Pyx_GOTREF(__pyx_t_3);\n __pyx_t_4 = PyTuple_New(1); if (unlikely(!__pyx_t_4)) __PYX_ERR(1, 844, __pyx_L1_error)\n __Pyx_GOTREF(__pyx_t_4);\n __Pyx_GIVEREF(__pyx_t_3);\n PyTuple_SET_ITEM(__pyx_t_4, 0, __pyx_t_3);\n __pyx_t_3 = 0;\n __pyx_t_3 = __Pyx_PyObject_Call(__pyx_builtin_ValueError, __pyx_t_4, NULL); if (unlikely(!__pyx_t_3)) __PYX_ERR(1, 844, __pyx_L1_error)\n __Pyx_GOTREF(__pyx_t_3);\n __Pyx_DECREF(__pyx_t_4); __pyx_t_4 = 0;\n __Pyx_Raise(__pyx_t_3, 0, 0, 0);\n __Pyx_DECREF(__pyx_t_3); __pyx_t_3 = 0;\n __PYX_ERR(1, 844, __pyx_L1_error)\n }\n __pyx_L15:;\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":845\n * else:\n * raise ValueError(u\"unknown dtype code in numpy.pxd (%d)\" % t)\n * f += 1 # <<<<<<<<<<<<<<\n * else:\n * # Cython ignores struct boundary information (\"T{...}\"),\n */\n __pyx_v_f = (__pyx_v_f + 1);\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":820\n * offset[0] += child.itemsize\n * \n * if not PyDataType_HASFIELDS(child): # <<<<<<<<<<<<<<\n * t = child.type_num\n * if end - f < 5:\n */\n goto __pyx_L13;\n }\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":849\n * # Cython ignores struct boundary information (\"T{...}\"),\n * # so don't output it\n * f = _util_dtypestring(child, f, end, offset) # <<<<<<<<<<<<<<\n * return f\n * \n */\n /*else*/ {\n __pyx_t_9 = __pyx_f_5numpy__util_dtypestring(__pyx_v_child, __pyx_v_f, __pyx_v_end, __pyx_v_offset); if (unlikely(__pyx_t_9 == NULL)) __PYX_ERR(1, 849, __pyx_L1_error)\n __pyx_v_f = __pyx_t_9;\n }\n __pyx_L13:;\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":794\n * cdef tuple fields\n * \n * for childname in descr.names: # <<<<<<<<<<<<<<\n * fields = descr.fields[childname]\n * child, new_offset = fields\n */\n }\n __Pyx_DECREF(__pyx_t_1); __pyx_t_1 = 0;\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":850\n * # so don't output it\n * f = _util_dtypestring(child, f, end, offset)\n * return f # <<<<<<<<<<<<<<\n * \n * \n */\n __pyx_r = __pyx_v_f;\n goto __pyx_L0;\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":785\n * return PyArray_MultiIterNew(5, a, b, c, d, e)\n * \n * cdef inline char* _util_dtypestring(dtype descr, char* f, char* end, int* offset) except NULL: # <<<<<<<<<<<<<<\n * # Recursive utility function used in __getbuffer__ to get format\n * # string. The new location in the format string is returned.\n */\n\n /* function exit code */\n __pyx_L1_error:;\n __Pyx_XDECREF(__pyx_t_1);\n __Pyx_XDECREF(__pyx_t_3);\n __Pyx_XDECREF(__pyx_t_4);\n __Pyx_AddTraceback(\"numpy._util_dtypestring\", __pyx_clineno, __pyx_lineno, __pyx_filename);\n __pyx_r = NULL;\n __pyx_L0:;\n __Pyx_XDECREF((PyObject *)__pyx_v_child);\n __Pyx_XDECREF(__pyx_v_fields);\n __Pyx_XDECREF(__pyx_v_childname);\n __Pyx_XDECREF(__pyx_v_new_offset);\n __Pyx_XDECREF(__pyx_v_t);\n __Pyx_RefNannyFinishContext();\n return __pyx_r;\n}\n\n/* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":966\n * \n * \n * cdef inline void set_array_base(ndarray arr, object base): # <<<<<<<<<<<<<<\n * cdef PyObject* baseptr\n * if base is None:\n */\n\nstatic CYTHON_INLINE void __pyx_f_5numpy_set_array_base(PyArrayObject *__pyx_v_arr, PyObject *__pyx_v_base) {\n PyObject *__pyx_v_baseptr;\n __Pyx_RefNannyDeclarations\n int __pyx_t_1;\n int __pyx_t_2;\n __Pyx_RefNannySetupContext(\"set_array_base\", 0);\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":968\n * cdef inline void set_array_base(ndarray arr, object base):\n * cdef PyObject* baseptr\n * if base is None: # <<<<<<<<<<<<<<\n * baseptr = NULL\n * else:\n */\n __pyx_t_1 = (__pyx_v_base == Py_None);\n __pyx_t_2 = (__pyx_t_1 != 0);\n if (__pyx_t_2) {\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":969\n * cdef PyObject* baseptr\n * if base is None:\n * baseptr = NULL # <<<<<<<<<<<<<<\n * else:\n * Py_INCREF(base) # important to do this before decref below!\n */\n __pyx_v_baseptr = NULL;\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":968\n * cdef inline void set_array_base(ndarray arr, object base):\n * cdef PyObject* baseptr\n * if base is None: # <<<<<<<<<<<<<<\n * baseptr = NULL\n * else:\n */\n goto __pyx_L3;\n }\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":971\n * baseptr = NULL\n * else:\n * Py_INCREF(base) # important to do this before decref below! # <<<<<<<<<<<<<<\n * baseptr = base\n * Py_XDECREF(arr.base)\n */\n /*else*/ {\n Py_INCREF(__pyx_v_base);\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":972\n * else:\n * Py_INCREF(base) # important to do this before decref below!\n * baseptr = base # <<<<<<<<<<<<<<\n * Py_XDECREF(arr.base)\n * arr.base = baseptr\n */\n __pyx_v_baseptr = ((PyObject *)__pyx_v_base);\n }\n __pyx_L3:;\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":973\n * Py_INCREF(base) # important to do this before decref below!\n * baseptr = base\n * Py_XDECREF(arr.base) # <<<<<<<<<<<<<<\n * arr.base = baseptr\n * \n */\n Py_XDECREF(__pyx_v_arr->base);\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":974\n * baseptr = base\n * Py_XDECREF(arr.base)\n * arr.base = baseptr # <<<<<<<<<<<<<<\n * \n * cdef inline object get_array_base(ndarray arr):\n */\n __pyx_v_arr->base = __pyx_v_baseptr;\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":966\n * \n * \n * cdef inline void set_array_base(ndarray arr, object base): # <<<<<<<<<<<<<<\n * cdef PyObject* baseptr\n * if base is None:\n */\n\n /* function exit code */\n __Pyx_RefNannyFinishContext();\n}\n\n/* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":976\n * arr.base = baseptr\n * \n * cdef inline object get_array_base(ndarray arr): # <<<<<<<<<<<<<<\n * if arr.base is NULL:\n * return None\n */\n\nstatic CYTHON_INLINE PyObject *__pyx_f_5numpy_get_array_base(PyArrayObject *__pyx_v_arr) {\n PyObject *__pyx_r = NULL;\n __Pyx_RefNannyDeclarations\n int __pyx_t_1;\n __Pyx_RefNannySetupContext(\"get_array_base\", 0);\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":977\n * \n * cdef inline object get_array_base(ndarray arr):\n * if arr.base is NULL: # <<<<<<<<<<<<<<\n * return None\n * else:\n */\n __pyx_t_1 = ((__pyx_v_arr->base == NULL) != 0);\n if (__pyx_t_1) {\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":978\n * cdef inline object get_array_base(ndarray arr):\n * if arr.base is NULL:\n * return None # <<<<<<<<<<<<<<\n * else:\n * return arr.base\n */\n __Pyx_XDECREF(__pyx_r);\n __Pyx_INCREF(Py_None);\n __pyx_r = Py_None;\n goto __pyx_L0;\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":977\n * \n * cdef inline object get_array_base(ndarray arr):\n * if arr.base is NULL: # <<<<<<<<<<<<<<\n * return None\n * else:\n */\n }\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":980\n * return None\n * else:\n * return arr.base # <<<<<<<<<<<<<<\n */\n /*else*/ {\n __Pyx_XDECREF(__pyx_r);\n __Pyx_INCREF(((PyObject *)__pyx_v_arr->base));\n __pyx_r = ((PyObject *)__pyx_v_arr->base);\n goto __pyx_L0;\n }\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":976\n * arr.base = baseptr\n * \n * cdef inline object get_array_base(ndarray arr): # <<<<<<<<<<<<<<\n * if arr.base is NULL:\n * return None\n */\n\n /* function exit code */\n __pyx_L0:;\n __Pyx_XGIVEREF(__pyx_r);\n __Pyx_RefNannyFinishContext();\n return __pyx_r;\n}\n\nstatic PyMethodDef __pyx_methods[] = {\n {0, 0, 0, 0}\n};\n\n#if PY_MAJOR_VERSION >= 3\nstatic struct PyModuleDef __pyx_moduledef = {\n #if PY_VERSION_HEX < 0x03020000\n { PyObject_HEAD_INIT(NULL) NULL, 0, NULL },\n #else\n PyModuleDef_HEAD_INIT,\n #endif\n \"gpu_nms\",\n 0, /* m_doc */\n -1, /* m_size */\n __pyx_methods /* m_methods */,\n NULL, /* m_reload */\n NULL, /* m_traverse */\n NULL, /* m_clear */\n NULL /* m_free */\n};\n#endif\n\nstatic __Pyx_StringTabEntry __pyx_string_tab[] = {\n {&__pyx_kp_s_D_v_zix_caffe_caffe_win_20160523, __pyx_k_D_v_zix_caffe_caffe_win_20160523, sizeof(__pyx_k_D_v_zix_caffe_caffe_win_20160523), 0, 0, 1, 0},\n {&__pyx_kp_u_Format_string_allocated_too_shor, __pyx_k_Format_string_allocated_too_shor, sizeof(__pyx_k_Format_string_allocated_too_shor), 0, 1, 0, 0},\n {&__pyx_kp_u_Format_string_allocated_too_shor_2, __pyx_k_Format_string_allocated_too_shor_2, sizeof(__pyx_k_Format_string_allocated_too_shor_2), 0, 1, 0, 0},\n {&__pyx_kp_u_Non_native_byte_order_not_suppor, __pyx_k_Non_native_byte_order_not_suppor, sizeof(__pyx_k_Non_native_byte_order_not_suppor), 0, 1, 0, 0},\n {&__pyx_n_s_RuntimeError, __pyx_k_RuntimeError, sizeof(__pyx_k_RuntimeError), 0, 0, 1, 1},\n {&__pyx_n_s_ValueError, __pyx_k_ValueError, sizeof(__pyx_k_ValueError), 0, 0, 1, 1},\n {&__pyx_n_s_argsort, __pyx_k_argsort, sizeof(__pyx_k_argsort), 0, 0, 1, 1},\n {&__pyx_n_s_boxes_dim, __pyx_k_boxes_dim, sizeof(__pyx_k_boxes_dim), 0, 0, 1, 1},\n {&__pyx_n_s_boxes_num, __pyx_k_boxes_num, sizeof(__pyx_k_boxes_num), 0, 0, 1, 1},\n {&__pyx_n_s_dets, __pyx_k_dets, sizeof(__pyx_k_dets), 0, 0, 1, 1},\n {&__pyx_n_s_device_id, __pyx_k_device_id, sizeof(__pyx_k_device_id), 0, 0, 1, 1},\n {&__pyx_n_s_dtype, __pyx_k_dtype, sizeof(__pyx_k_dtype), 0, 0, 1, 1},\n {&__pyx_n_s_gpu_nms, __pyx_k_gpu_nms, sizeof(__pyx_k_gpu_nms), 0, 0, 1, 1},\n {&__pyx_n_s_import, __pyx_k_import, sizeof(__pyx_k_import), 0, 0, 1, 1},\n {&__pyx_n_s_int32, __pyx_k_int32, sizeof(__pyx_k_int32), 0, 0, 1, 1},\n {&__pyx_n_s_keep, __pyx_k_keep, sizeof(__pyx_k_keep), 0, 0, 1, 1},\n {&__pyx_n_s_main, __pyx_k_main, sizeof(__pyx_k_main), 0, 0, 1, 1},\n {&__pyx_kp_u_ndarray_is_not_C_contiguous, __pyx_k_ndarray_is_not_C_contiguous, sizeof(__pyx_k_ndarray_is_not_C_contiguous), 0, 1, 0, 0},\n {&__pyx_kp_u_ndarray_is_not_Fortran_contiguou, __pyx_k_ndarray_is_not_Fortran_contiguou, sizeof(__pyx_k_ndarray_is_not_Fortran_contiguou), 0, 1, 0, 0},\n {&__pyx_n_s_nms_gpu_nms, __pyx_k_nms_gpu_nms, sizeof(__pyx_k_nms_gpu_nms), 0, 0, 1, 1},\n {&__pyx_n_s_np, __pyx_k_np, sizeof(__pyx_k_np), 0, 0, 1, 1},\n {&__pyx_n_s_num_out, __pyx_k_num_out, sizeof(__pyx_k_num_out), 0, 0, 1, 1},\n {&__pyx_n_s_numpy, __pyx_k_numpy, sizeof(__pyx_k_numpy), 0, 0, 1, 1},\n {&__pyx_n_s_order, __pyx_k_order, sizeof(__pyx_k_order), 0, 0, 1, 1},\n {&__pyx_n_s_range, __pyx_k_range, sizeof(__pyx_k_range), 0, 0, 1, 1},\n {&__pyx_n_s_scores, __pyx_k_scores, sizeof(__pyx_k_scores), 0, 0, 1, 1},\n {&__pyx_n_s_sorted_dets, __pyx_k_sorted_dets, sizeof(__pyx_k_sorted_dets), 0, 0, 1, 1},\n {&__pyx_n_s_test, __pyx_k_test, sizeof(__pyx_k_test), 0, 0, 1, 1},\n {&__pyx_n_s_thresh, __pyx_k_thresh, sizeof(__pyx_k_thresh), 0, 0, 1, 1},\n {&__pyx_kp_u_unknown_dtype_code_in_numpy_pxd, __pyx_k_unknown_dtype_code_in_numpy_pxd, sizeof(__pyx_k_unknown_dtype_code_in_numpy_pxd), 0, 1, 0, 0},\n {&__pyx_n_s_zeros, __pyx_k_zeros, sizeof(__pyx_k_zeros), 0, 0, 1, 1},\n {0, 0, 0, 0, 0, 0, 0}\n};\nstatic int __Pyx_InitCachedBuiltins(void) {\n __pyx_builtin_ValueError = __Pyx_GetBuiltinName(__pyx_n_s_ValueError); if (!__pyx_builtin_ValueError) __PYX_ERR(1, 218, __pyx_L1_error)\n __pyx_builtin_range = __Pyx_GetBuiltinName(__pyx_n_s_range); if (!__pyx_builtin_range) __PYX_ERR(1, 231, __pyx_L1_error)\n __pyx_builtin_RuntimeError = __Pyx_GetBuiltinName(__pyx_n_s_RuntimeError); if (!__pyx_builtin_RuntimeError) __PYX_ERR(1, 799, __pyx_L1_error)\n return 0;\n __pyx_L1_error:;\n return -1;\n}\n\nstatic int __Pyx_InitCachedConstants(void) {\n __Pyx_RefNannyDeclarations\n __Pyx_RefNannySetupContext(\"__Pyx_InitCachedConstants\", 0);\n\n /* \"nms/gpu_nms.pyx\":24\n * keep = np.zeros(boxes_num, dtype=np.int32)\n * cdef np.ndarray[np.float32_t, ndim=1] \\\n * scores = dets[:, 4] # <<<<<<<<<<<<<<\n * #cdef np.ndarray[np.int_t, ndim=1] \\ // 20160601, by xzn\n * # order = scores.argsort()[::-1]\n */\n __pyx_slice_ = PySlice_New(Py_None, Py_None, Py_None); if (unlikely(!__pyx_slice_)) __PYX_ERR(0, 24, __pyx_L1_error)\n __Pyx_GOTREF(__pyx_slice_);\n __Pyx_GIVEREF(__pyx_slice_);\n __pyx_tuple__2 = PyTuple_Pack(2, __pyx_slice_, __pyx_int_4); if (unlikely(!__pyx_tuple__2)) __PYX_ERR(0, 24, __pyx_L1_error)\n __Pyx_GOTREF(__pyx_tuple__2);\n __Pyx_GIVEREF(__pyx_tuple__2);\n\n /* \"nms/gpu_nms.pyx\":28\n * # order = scores.argsort()[::-1]\n * cdef np.ndarray[np.intp_t, ndim=1] \\\n * order = scores.argsort()[::-1] # <<<<<<<<<<<<<<\n * cdef np.ndarray[np.float32_t, ndim=2] \\\n * sorted_dets = dets[order, :]\n */\n __pyx_slice__3 = PySlice_New(Py_None, Py_None, __pyx_int_neg_1); if (unlikely(!__pyx_slice__3)) __PYX_ERR(0, 28, __pyx_L1_error)\n __Pyx_GOTREF(__pyx_slice__3);\n __Pyx_GIVEREF(__pyx_slice__3);\n\n /* \"nms/gpu_nms.pyx\":30\n * order = scores.argsort()[::-1]\n * cdef np.ndarray[np.float32_t, ndim=2] \\\n * sorted_dets = dets[order, :] # <<<<<<<<<<<<<<\n * _nms(&keep[0], &num_out, &sorted_dets[0, 0], boxes_num, boxes_dim, thresh, device_id)\n * keep = keep[:num_out]\n */\n __pyx_slice__4 = PySlice_New(Py_None, Py_None, Py_None); if (unlikely(!__pyx_slice__4)) __PYX_ERR(0, 30, __pyx_L1_error)\n __Pyx_GOTREF(__pyx_slice__4);\n __Pyx_GIVEREF(__pyx_slice__4);\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":218\n * if ((flags & pybuf.PyBUF_C_CONTIGUOUS == pybuf.PyBUF_C_CONTIGUOUS)\n * and not PyArray_CHKFLAGS(self, NPY_C_CONTIGUOUS)):\n * raise ValueError(u\"ndarray is not C contiguous\") # <<<<<<<<<<<<<<\n * \n * if ((flags & pybuf.PyBUF_F_CONTIGUOUS == pybuf.PyBUF_F_CONTIGUOUS)\n */\n __pyx_tuple__5 = PyTuple_Pack(1, __pyx_kp_u_ndarray_is_not_C_contiguous); if (unlikely(!__pyx_tuple__5)) __PYX_ERR(1, 218, __pyx_L1_error)\n __Pyx_GOTREF(__pyx_tuple__5);\n __Pyx_GIVEREF(__pyx_tuple__5);\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":222\n * if ((flags & pybuf.PyBUF_F_CONTIGUOUS == pybuf.PyBUF_F_CONTIGUOUS)\n * and not PyArray_CHKFLAGS(self, NPY_F_CONTIGUOUS)):\n * raise ValueError(u\"ndarray is not Fortran contiguous\") # <<<<<<<<<<<<<<\n * \n * info.buf = PyArray_DATA(self)\n */\n __pyx_tuple__6 = PyTuple_Pack(1, __pyx_kp_u_ndarray_is_not_Fortran_contiguou); if (unlikely(!__pyx_tuple__6)) __PYX_ERR(1, 222, __pyx_L1_error)\n __Pyx_GOTREF(__pyx_tuple__6);\n __Pyx_GIVEREF(__pyx_tuple__6);\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":259\n * if ((descr.byteorder == c'>' and little_endian) or\n * (descr.byteorder == c'<' and not little_endian)):\n * raise ValueError(u\"Non-native byte order not supported\") # <<<<<<<<<<<<<<\n * if t == NPY_BYTE: f = \"b\"\n * elif t == NPY_UBYTE: f = \"B\"\n */\n __pyx_tuple__7 = PyTuple_Pack(1, __pyx_kp_u_Non_native_byte_order_not_suppor); if (unlikely(!__pyx_tuple__7)) __PYX_ERR(1, 259, __pyx_L1_error)\n __Pyx_GOTREF(__pyx_tuple__7);\n __Pyx_GIVEREF(__pyx_tuple__7);\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":799\n * \n * if (end - f) - (new_offset - offset[0]) < 15:\n * raise RuntimeError(u\"Format string allocated too short, see comment in numpy.pxd\") # <<<<<<<<<<<<<<\n * \n * if ((child.byteorder == c'>' and little_endian) or\n */\n __pyx_tuple__8 = PyTuple_Pack(1, __pyx_kp_u_Format_string_allocated_too_shor); if (unlikely(!__pyx_tuple__8)) __PYX_ERR(1, 799, __pyx_L1_error)\n __Pyx_GOTREF(__pyx_tuple__8);\n __Pyx_GIVEREF(__pyx_tuple__8);\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":803\n * if ((child.byteorder == c'>' and little_endian) or\n * (child.byteorder == c'<' and not little_endian)):\n * raise ValueError(u\"Non-native byte order not supported\") # <<<<<<<<<<<<<<\n * # One could encode it in the format string and have Cython\n * # complain instead, BUT: < and > in format strings also imply\n */\n __pyx_tuple__9 = PyTuple_Pack(1, __pyx_kp_u_Non_native_byte_order_not_suppor); if (unlikely(!__pyx_tuple__9)) __PYX_ERR(1, 803, __pyx_L1_error)\n __Pyx_GOTREF(__pyx_tuple__9);\n __Pyx_GIVEREF(__pyx_tuple__9);\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":823\n * t = child.type_num\n * if end - f < 5:\n * raise RuntimeError(u\"Format string allocated too short.\") # <<<<<<<<<<<<<<\n * \n * # Until ticket #99 is fixed, use integers to avoid warnings\n */\n __pyx_tuple__10 = PyTuple_Pack(1, __pyx_kp_u_Format_string_allocated_too_shor_2); if (unlikely(!__pyx_tuple__10)) __PYX_ERR(1, 823, __pyx_L1_error)\n __Pyx_GOTREF(__pyx_tuple__10);\n __Pyx_GIVEREF(__pyx_tuple__10);\n\n /* \"nms/gpu_nms.pyx\":16\n * void _nms(np.int32_t*, int*, np.float32_t*, int, int, float, int)\n * \n * def gpu_nms(np.ndarray[np.float32_t, ndim=2] dets, np.float thresh, # <<<<<<<<<<<<<<\n * np.int32_t device_id=0):\n * cdef int boxes_num = dets.shape[0]\n */\n __pyx_tuple__11 = PyTuple_Pack(10, __pyx_n_s_dets, __pyx_n_s_thresh, __pyx_n_s_device_id, __pyx_n_s_boxes_num, __pyx_n_s_boxes_dim, __pyx_n_s_num_out, __pyx_n_s_keep, __pyx_n_s_scores, __pyx_n_s_order, __pyx_n_s_sorted_dets); if (unlikely(!__pyx_tuple__11)) __PYX_ERR(0, 16, __pyx_L1_error)\n __Pyx_GOTREF(__pyx_tuple__11);\n __Pyx_GIVEREF(__pyx_tuple__11);\n __pyx_codeobj__12 = (PyObject*)__Pyx_PyCode_New(3, 0, 10, 0, 0, __pyx_empty_bytes, __pyx_empty_tuple, __pyx_empty_tuple, __pyx_tuple__11, __pyx_empty_tuple, __pyx_empty_tuple, __pyx_kp_s_D_v_zix_caffe_caffe_win_20160523, __pyx_n_s_gpu_nms, 16, __pyx_empty_bytes); if (unlikely(!__pyx_codeobj__12)) __PYX_ERR(0, 16, __pyx_L1_error)\n __Pyx_RefNannyFinishContext();\n return 0;\n __pyx_L1_error:;\n __Pyx_RefNannyFinishContext();\n return -1;\n}\n\nstatic int __Pyx_InitGlobals(void) {\n if (__Pyx_InitStrings(__pyx_string_tab) < 0) __PYX_ERR(0, 1, __pyx_L1_error);\n __pyx_int_4 = PyInt_FromLong(4); if (unlikely(!__pyx_int_4)) __PYX_ERR(0, 1, __pyx_L1_error)\n __pyx_int_neg_1 = PyInt_FromLong(-1); if (unlikely(!__pyx_int_neg_1)) __PYX_ERR(0, 1, __pyx_L1_error)\n return 0;\n __pyx_L1_error:;\n return -1;\n}\n\n#if PY_MAJOR_VERSION < 3\nPyMODINIT_FUNC initgpu_nms(void); /*proto*/\nPyMODINIT_FUNC initgpu_nms(void)\n#else\nPyMODINIT_FUNC PyInit_gpu_nms(void); /*proto*/\nPyMODINIT_FUNC PyInit_gpu_nms(void)\n#endif\n{\n PyObject *__pyx_t_1 = NULL;\n __Pyx_RefNannyDeclarations\n #if CYTHON_REFNANNY\n __Pyx_RefNanny = __Pyx_RefNannyImportAPI(\"refnanny\");\n if (!__Pyx_RefNanny) {\n PyErr_Clear();\n __Pyx_RefNanny = __Pyx_RefNannyImportAPI(\"Cython.Runtime.refnanny\");\n if (!__Pyx_RefNanny)\n Py_FatalError(\"failed to import 'refnanny' module\");\n }\n #endif\n __Pyx_RefNannySetupContext(\"PyMODINIT_FUNC PyInit_gpu_nms(void)\", 0);\n if (__Pyx_check_binary_version() < 0) __PYX_ERR(0, 1, __pyx_L1_error)\n __pyx_empty_tuple = PyTuple_New(0); if (unlikely(!__pyx_empty_tuple)) __PYX_ERR(0, 1, __pyx_L1_error)\n __pyx_empty_bytes = PyBytes_FromStringAndSize(\"\", 0); if (unlikely(!__pyx_empty_bytes)) __PYX_ERR(0, 1, __pyx_L1_error)\n __pyx_empty_unicode = PyUnicode_FromStringAndSize(\"\", 0); if (unlikely(!__pyx_empty_unicode)) __PYX_ERR(0, 1, __pyx_L1_error)\n #ifdef __Pyx_CyFunction_USED\n if (__pyx_CyFunction_init() < 0) __PYX_ERR(0, 1, __pyx_L1_error)\n #endif\n #ifdef __Pyx_FusedFunction_USED\n if (__pyx_FusedFunction_init() < 0) __PYX_ERR(0, 1, __pyx_L1_error)\n #endif\n #ifdef __Pyx_Coroutine_USED\n if (__pyx_Coroutine_init() < 0) __PYX_ERR(0, 1, __pyx_L1_error)\n #endif\n #ifdef __Pyx_Generator_USED\n if (__pyx_Generator_init() < 0) __PYX_ERR(0, 1, __pyx_L1_error)\n #endif\n #ifdef __Pyx_StopAsyncIteration_USED\n if (__pyx_StopAsyncIteration_init() < 0) __PYX_ERR(0, 1, __pyx_L1_error)\n #endif\n /*--- Library function declarations ---*/\n /*--- Threads initialization code ---*/\n #if defined(__PYX_FORCE_INIT_THREADS) && __PYX_FORCE_INIT_THREADS\n #ifdef WITH_THREAD /* Python build with threading support? */\n PyEval_InitThreads();\n #endif\n #endif\n /*--- Module creation code ---*/\n #if PY_MAJOR_VERSION < 3\n __pyx_m = Py_InitModule4(\"gpu_nms\", __pyx_methods, 0, 0, PYTHON_API_VERSION); Py_XINCREF(__pyx_m);\n #else\n __pyx_m = PyModule_Create(&__pyx_moduledef);\n #endif\n if (unlikely(!__pyx_m)) __PYX_ERR(0, 1, __pyx_L1_error)\n __pyx_d = PyModule_GetDict(__pyx_m); if (unlikely(!__pyx_d)) __PYX_ERR(0, 1, __pyx_L1_error)\n Py_INCREF(__pyx_d);\n __pyx_b = PyImport_AddModule(__Pyx_BUILTIN_MODULE_NAME); if (unlikely(!__pyx_b)) __PYX_ERR(0, 1, __pyx_L1_error)\n #if CYTHON_COMPILING_IN_PYPY\n Py_INCREF(__pyx_b);\n #endif\n if (PyObject_SetAttrString(__pyx_m, \"__builtins__\", __pyx_b) < 0) __PYX_ERR(0, 1, __pyx_L1_error);\n /*--- Initialize various global constants etc. ---*/\n if (__Pyx_InitGlobals() < 0) __PYX_ERR(0, 1, __pyx_L1_error)\n #if PY_MAJOR_VERSION < 3 && (__PYX_DEFAULT_STRING_ENCODING_IS_ASCII || __PYX_DEFAULT_STRING_ENCODING_IS_DEFAULT)\n if (__Pyx_init_sys_getdefaultencoding_params() < 0) __PYX_ERR(0, 1, __pyx_L1_error)\n #endif\n if (__pyx_module_is_main_nms__gpu_nms) {\n if (PyObject_SetAttrString(__pyx_m, \"__name__\", __pyx_n_s_main) < 0) __PYX_ERR(0, 1, __pyx_L1_error)\n }\n #if PY_MAJOR_VERSION >= 3\n {\n PyObject *modules = PyImport_GetModuleDict(); if (unlikely(!modules)) __PYX_ERR(0, 1, __pyx_L1_error)\n if (!PyDict_GetItemString(modules, \"nms.gpu_nms\")) {\n if (unlikely(PyDict_SetItemString(modules, \"nms.gpu_nms\", __pyx_m) < 0)) __PYX_ERR(0, 1, __pyx_L1_error)\n }\n }\n #endif\n /*--- Builtin init code ---*/\n if (__Pyx_InitCachedBuiltins() < 0) __PYX_ERR(0, 1, __pyx_L1_error)\n /*--- Constants init code ---*/\n if (__Pyx_InitCachedConstants() < 0) __PYX_ERR(0, 1, __pyx_L1_error)\n /*--- Global init code ---*/\n /*--- Variable export code ---*/\n /*--- Function export code ---*/\n /*--- Type init code ---*/\n /*--- Type import code ---*/\n __pyx_ptype_7cpython_4type_type = __Pyx_ImportType(__Pyx_BUILTIN_MODULE_NAME, \"type\", \n #if CYTHON_COMPILING_IN_PYPY\n sizeof(PyTypeObject),\n #else\n sizeof(PyHeapTypeObject),\n #endif\n 0); if (unlikely(!__pyx_ptype_7cpython_4type_type)) __PYX_ERR(2, 9, __pyx_L1_error)\n __pyx_ptype_5numpy_dtype = __Pyx_ImportType(\"numpy\", \"dtype\", sizeof(PyArray_Descr), 0); if (unlikely(!__pyx_ptype_5numpy_dtype)) __PYX_ERR(1, 155, __pyx_L1_error)\n __pyx_ptype_5numpy_flatiter = __Pyx_ImportType(\"numpy\", \"flatiter\", sizeof(PyArrayIterObject), 0); if (unlikely(!__pyx_ptype_5numpy_flatiter)) __PYX_ERR(1, 168, __pyx_L1_error)\n __pyx_ptype_5numpy_broadcast = __Pyx_ImportType(\"numpy\", \"broadcast\", sizeof(PyArrayMultiIterObject), 0); if (unlikely(!__pyx_ptype_5numpy_broadcast)) __PYX_ERR(1, 172, __pyx_L1_error)\n __pyx_ptype_5numpy_ndarray = __Pyx_ImportType(\"numpy\", \"ndarray\", sizeof(PyArrayObject), 0); if (unlikely(!__pyx_ptype_5numpy_ndarray)) __PYX_ERR(1, 181, __pyx_L1_error)\n __pyx_ptype_5numpy_ufunc = __Pyx_ImportType(\"numpy\", \"ufunc\", sizeof(PyUFuncObject), 0); if (unlikely(!__pyx_ptype_5numpy_ufunc)) __PYX_ERR(1, 861, __pyx_L1_error)\n /*--- Variable import code ---*/\n /*--- Function import code ---*/\n /*--- Execution code ---*/\n #if defined(__Pyx_Generator_USED) || defined(__Pyx_Coroutine_USED)\n if (__Pyx_patch_abc() < 0) __PYX_ERR(0, 1, __pyx_L1_error)\n #endif\n\n /* \"nms/gpu_nms.pyx\":8\n * # --------------------------------------------------------\n * \n * import numpy as np # <<<<<<<<<<<<<<\n * cimport numpy as np\n * \n */\n __pyx_t_1 = __Pyx_Import(__pyx_n_s_numpy, 0, -1); if (unlikely(!__pyx_t_1)) __PYX_ERR(0, 8, __pyx_L1_error)\n __Pyx_GOTREF(__pyx_t_1);\n if (PyDict_SetItem(__pyx_d, __pyx_n_s_np, __pyx_t_1) < 0) __PYX_ERR(0, 8, __pyx_L1_error)\n __Pyx_DECREF(__pyx_t_1); __pyx_t_1 = 0;\n\n /* \"nms/gpu_nms.pyx\":11\n * cimport numpy as np\n * \n * assert sizeof(int) == sizeof(np.int32_t) # <<<<<<<<<<<<<<\n * \n * cdef extern from \"gpu_nms.hpp\":\n */\n #ifndef CYTHON_WITHOUT_ASSERTIONS\n if (unlikely(!Py_OptimizeFlag)) {\n if (unlikely(!(((sizeof(int)) == (sizeof(__pyx_t_5numpy_int32_t))) != 0))) {\n PyErr_SetNone(PyExc_AssertionError);\n __PYX_ERR(0, 11, __pyx_L1_error)\n }\n }\n #endif\n\n /* \"nms/gpu_nms.pyx\":16\n * void _nms(np.int32_t*, int*, np.float32_t*, int, int, float, int)\n * \n * def gpu_nms(np.ndarray[np.float32_t, ndim=2] dets, np.float thresh, # <<<<<<<<<<<<<<\n * np.int32_t device_id=0):\n * cdef int boxes_num = dets.shape[0]\n */\n __pyx_t_1 = PyCFunction_NewEx(&__pyx_mdef_3nms_7gpu_nms_1gpu_nms, NULL, __pyx_n_s_nms_gpu_nms); if (unlikely(!__pyx_t_1)) __PYX_ERR(0, 16, __pyx_L1_error)\n __Pyx_GOTREF(__pyx_t_1);\n if (PyDict_SetItem(__pyx_d, __pyx_n_s_gpu_nms, __pyx_t_1) < 0) __PYX_ERR(0, 16, __pyx_L1_error)\n __Pyx_DECREF(__pyx_t_1); __pyx_t_1 = 0;\n\n /* \"nms/gpu_nms.pyx\":1\n * # -------------------------------------------------------- # <<<<<<<<<<<<<<\n * # Faster R-CNN\n * # Copyright (c) 2015 Microsoft\n */\n __pyx_t_1 = PyDict_New(); if (unlikely(!__pyx_t_1)) __PYX_ERR(0, 1, __pyx_L1_error)\n __Pyx_GOTREF(__pyx_t_1);\n if (PyDict_SetItem(__pyx_d, __pyx_n_s_test, __pyx_t_1) < 0) __PYX_ERR(0, 1, __pyx_L1_error)\n __Pyx_DECREF(__pyx_t_1); __pyx_t_1 = 0;\n\n /* \"C:/Anaconda2/lib/site-packages/Cython/Includes/numpy/__init__.pxd\":976\n * arr.base = baseptr\n * \n * cdef inline object get_array_base(ndarray arr): # <<<<<<<<<<<<<<\n * if arr.base is NULL:\n * return None\n */\n\n /*--- Wrapped vars code ---*/\n\n goto __pyx_L0;\n __pyx_L1_error:;\n __Pyx_XDECREF(__pyx_t_1);\n if (__pyx_m) {\n if (__pyx_d) {\n __Pyx_AddTraceback(\"init nms.gpu_nms\", __pyx_clineno, __pyx_lineno, __pyx_filename);\n }\n Py_DECREF(__pyx_m); __pyx_m = 0;\n } else if (!PyErr_Occurred()) {\n PyErr_SetString(PyExc_ImportError, \"init nms.gpu_nms\");\n }\n __pyx_L0:;\n __Pyx_RefNannyFinishContext();\n #if PY_MAJOR_VERSION < 3\n return;\n #else\n return __pyx_m;\n #endif\n}\n\n/* --- Runtime support code --- */\n/* Refnanny */\n#if CYTHON_REFNANNY\nstatic __Pyx_RefNannyAPIStruct *__Pyx_RefNannyImportAPI(const char *modname) {\n PyObject *m = NULL, *p = NULL;\n void *r = NULL;\n m = PyImport_ImportModule((char *)modname);\n if (!m) goto end;\n p = PyObject_GetAttrString(m, (char *)\"RefNannyAPI\");\n if (!p) goto end;\n r = PyLong_AsVoidPtr(p);\nend:\n Py_XDECREF(p);\n Py_XDECREF(m);\n return (__Pyx_RefNannyAPIStruct *)r;\n}\n#endif\n\n/* RaiseArgTupleInvalid */\nstatic void __Pyx_RaiseArgtupleInvalid(\n const char* func_name,\n int exact,\n Py_ssize_t num_min,\n Py_ssize_t num_max,\n Py_ssize_t num_found)\n{\n Py_ssize_t num_expected;\n const char *more_or_less;\n if (num_found < num_min) {\n num_expected = num_min;\n more_or_less = \"at least\";\n } else {\n num_expected = num_max;\n more_or_less = \"at most\";\n }\n if (exact) {\n more_or_less = \"exactly\";\n }\n PyErr_Format(PyExc_TypeError,\n \"%.200s() takes %.8s %\" CYTHON_FORMAT_SSIZE_T \"d positional argument%.1s (%\" CYTHON_FORMAT_SSIZE_T \"d given)\",\n func_name, more_or_less, num_expected,\n (num_expected == 1) ? \"\" : \"s\", num_found);\n}\n\n/* RaiseDoubleKeywords */\nstatic void __Pyx_RaiseDoubleKeywordsError(\n const char* func_name,\n PyObject* kw_name)\n{\n PyErr_Format(PyExc_TypeError,\n #if PY_MAJOR_VERSION >= 3\n \"%s() got multiple values for keyword argument '%U'\", func_name, kw_name);\n #else\n \"%s() got multiple values for keyword argument '%s'\", func_name,\n PyString_AsString(kw_name));\n #endif\n}\n\n/* ParseKeywords */\nstatic int __Pyx_ParseOptionalKeywords(\n PyObject *kwds,\n PyObject **argnames[],\n PyObject *kwds2,\n PyObject *values[],\n Py_ssize_t num_pos_args,\n const char* function_name)\n{\n PyObject *key = 0, *value = 0;\n Py_ssize_t pos = 0;\n PyObject*** name;\n PyObject*** first_kw_arg = argnames + num_pos_args;\n while (PyDict_Next(kwds, &pos, &key, &value)) {\n name = first_kw_arg;\n while (*name && (**name != key)) name++;\n if (*name) {\n values[name-argnames] = value;\n continue;\n }\n name = first_kw_arg;\n #if PY_MAJOR_VERSION < 3\n if (likely(PyString_CheckExact(key)) || likely(PyString_Check(key))) {\n while (*name) {\n if ((CYTHON_COMPILING_IN_PYPY || PyString_GET_SIZE(**name) == PyString_GET_SIZE(key))\n && _PyString_Eq(**name, key)) {\n values[name-argnames] = value;\n break;\n }\n name++;\n }\n if (*name) continue;\n else {\n PyObject*** argname = argnames;\n while (argname != first_kw_arg) {\n if ((**argname == key) || (\n (CYTHON_COMPILING_IN_PYPY || PyString_GET_SIZE(**argname) == PyString_GET_SIZE(key))\n && _PyString_Eq(**argname, key))) {\n goto arg_passed_twice;\n }\n argname++;\n }\n }\n } else\n #endif\n if (likely(PyUnicode_Check(key))) {\n while (*name) {\n int cmp = (**name == key) ? 0 :\n #if !CYTHON_COMPILING_IN_PYPY && PY_MAJOR_VERSION >= 3\n (PyUnicode_GET_SIZE(**name) != PyUnicode_GET_SIZE(key)) ? 1 :\n #endif\n PyUnicode_Compare(**name, key);\n if (cmp < 0 && unlikely(PyErr_Occurred())) goto bad;\n if (cmp == 0) {\n values[name-argnames] = value;\n break;\n }\n name++;\n }\n if (*name) continue;\n else {\n PyObject*** argname = argnames;\n while (argname != first_kw_arg) {\n int cmp = (**argname == key) ? 0 :\n #if !CYTHON_COMPILING_IN_PYPY && PY_MAJOR_VERSION >= 3\n (PyUnicode_GET_SIZE(**argname) != PyUnicode_GET_SIZE(key)) ? 1 :\n #endif\n PyUnicode_Compare(**argname, key);\n if (cmp < 0 && unlikely(PyErr_Occurred())) goto bad;\n if (cmp == 0) goto arg_passed_twice;\n argname++;\n }\n }\n } else\n goto invalid_keyword_type;\n if (kwds2) {\n if (unlikely(PyDict_SetItem(kwds2, key, value))) goto bad;\n } else {\n goto invalid_keyword;\n }\n }\n return 0;\narg_passed_twice:\n __Pyx_RaiseDoubleKeywordsError(function_name, key);\n goto bad;\ninvalid_keyword_type:\n PyErr_Format(PyExc_TypeError,\n \"%.200s() keywords must be strings\", function_name);\n goto bad;\ninvalid_keyword:\n PyErr_Format(PyExc_TypeError,\n #if PY_MAJOR_VERSION < 3\n \"%.200s() got an unexpected keyword argument '%.200s'\",\n function_name, PyString_AsString(key));\n #else\n \"%s() got an unexpected keyword argument '%U'\",\n function_name, key);\n #endif\nbad:\n return -1;\n}\n\n/* ArgTypeTest */\nstatic void __Pyx_RaiseArgumentTypeInvalid(const char* name, PyObject *obj, PyTypeObject *type) {\n PyErr_Format(PyExc_TypeError,\n \"Argument '%.200s' has incorrect type (expected %.200s, got %.200s)\",\n name, type->tp_name, Py_TYPE(obj)->tp_name);\n}\nstatic CYTHON_INLINE int __Pyx_ArgTypeTest(PyObject *obj, PyTypeObject *type, int none_allowed,\n const char *name, int exact)\n{\n if (unlikely(!type)) {\n PyErr_SetString(PyExc_SystemError, \"Missing type object\");\n return 0;\n }\n if (none_allowed && obj == Py_None) return 1;\n else if (exact) {\n if (likely(Py_TYPE(obj) == type)) return 1;\n #if PY_MAJOR_VERSION == 2\n else if ((type == &PyBaseString_Type) && likely(__Pyx_PyBaseString_CheckExact(obj))) return 1;\n #endif\n }\n else {\n if (likely(PyObject_TypeCheck(obj, type))) return 1;\n }\n __Pyx_RaiseArgumentTypeInvalid(name, obj, type);\n return 0;\n}\n\n/* BufferFormatCheck */\nstatic CYTHON_INLINE int __Pyx_IsLittleEndian(void) {\n unsigned int n = 1;\n return *(unsigned char*)(&n) != 0;\n}\nstatic void __Pyx_BufFmt_Init(__Pyx_BufFmt_Context* ctx,\n __Pyx_BufFmt_StackElem* stack,\n __Pyx_TypeInfo* type) {\n stack[0].field = &ctx->root;\n stack[0].parent_offset = 0;\n ctx->root.type = type;\n ctx->root.name = \"buffer dtype\";\n ctx->root.offset = 0;\n ctx->head = stack;\n ctx->head->field = &ctx->root;\n ctx->fmt_offset = 0;\n ctx->head->parent_offset = 0;\n ctx->new_packmode = '@';\n ctx->enc_packmode = '@';\n ctx->new_count = 1;\n ctx->enc_count = 0;\n ctx->enc_type = 0;\n ctx->is_complex = 0;\n ctx->is_valid_array = 0;\n ctx->struct_alignment = 0;\n while (type->typegroup == 'S') {\n ++ctx->head;\n ctx->head->field = type->fields;\n ctx->head->parent_offset = 0;\n type = type->fields->type;\n }\n}\nstatic int __Pyx_BufFmt_ParseNumber(const char** ts) {\n int count;\n const char* t = *ts;\n if (*t < '0' || *t > '9') {\n return -1;\n } else {\n count = *t++ - '0';\n while (*t >= '0' && *t < '9') {\n count *= 10;\n count += *t++ - '0';\n }\n }\n *ts = t;\n return count;\n}\nstatic int __Pyx_BufFmt_ExpectNumber(const char **ts) {\n int number = __Pyx_BufFmt_ParseNumber(ts);\n if (number == -1)\n PyErr_Format(PyExc_ValueError,\\\n \"Does not understand character buffer dtype format string ('%c')\", **ts);\n return number;\n}\nstatic void __Pyx_BufFmt_RaiseUnexpectedChar(char ch) {\n PyErr_Format(PyExc_ValueError,\n \"Unexpected format string character: '%c'\", ch);\n}\nstatic const char* __Pyx_BufFmt_DescribeTypeChar(char ch, int is_complex) {\n switch (ch) {\n case 'c': return \"'char'\";\n case 'b': return \"'signed char'\";\n case 'B': return \"'unsigned char'\";\n case 'h': return \"'short'\";\n case 'H': return \"'unsigned short'\";\n case 'i': return \"'int'\";\n case 'I': return \"'unsigned int'\";\n case 'l': return \"'long'\";\n case 'L': return \"'unsigned long'\";\n case 'q': return \"'long long'\";\n case 'Q': return \"'unsigned long long'\";\n case 'f': return (is_complex ? \"'complex float'\" : \"'float'\");\n case 'd': return (is_complex ? \"'complex double'\" : \"'double'\");\n case 'g': return (is_complex ? \"'complex long double'\" : \"'long double'\");\n case 'T': return \"a struct\";\n case 'O': return \"Python object\";\n case 'P': return \"a pointer\";\n case 's': case 'p': return \"a string\";\n case 0: return \"end\";\n default: return \"unparseable format string\";\n }\n}\nstatic size_t __Pyx_BufFmt_TypeCharToStandardSize(char ch, int is_complex) {\n switch (ch) {\n case '?': case 'c': case 'b': case 'B': case 's': case 'p': return 1;\n case 'h': case 'H': return 2;\n case 'i': case 'I': case 'l': case 'L': return 4;\n case 'q': case 'Q': return 8;\n case 'f': return (is_complex ? 8 : 4);\n case 'd': return (is_complex ? 16 : 8);\n case 'g': {\n PyErr_SetString(PyExc_ValueError, \"Python does not define a standard format string size for long double ('g')..\");\n return 0;\n }\n case 'O': case 'P': return sizeof(void*);\n default:\n __Pyx_BufFmt_RaiseUnexpectedChar(ch);\n return 0;\n }\n}\nstatic size_t __Pyx_BufFmt_TypeCharToNativeSize(char ch, int is_complex) {\n switch (ch) {\n case 'c': case 'b': case 'B': case 's': case 'p': return 1;\n case 'h': case 'H': return sizeof(short);\n case 'i': case 'I': return sizeof(int);\n case 'l': case 'L': return sizeof(long);\n #ifdef HAVE_LONG_LONG\n case 'q': case 'Q': return sizeof(PY_LONG_LONG);\n #endif\n case 'f': return sizeof(float) * (is_complex ? 2 : 1);\n case 'd': return sizeof(double) * (is_complex ? 2 : 1);\n case 'g': return sizeof(long double) * (is_complex ? 2 : 1);\n case 'O': case 'P': return sizeof(void*);\n default: {\n __Pyx_BufFmt_RaiseUnexpectedChar(ch);\n return 0;\n }\n }\n}\ntypedef struct { char c; short x; } __Pyx_st_short;\ntypedef struct { char c; int x; } __Pyx_st_int;\ntypedef struct { char c; long x; } __Pyx_st_long;\ntypedef struct { char c; float x; } __Pyx_st_float;\ntypedef struct { char c; double x; } __Pyx_st_double;\ntypedef struct { char c; long double x; } __Pyx_st_longdouble;\ntypedef struct { char c; void *x; } __Pyx_st_void_p;\n#ifdef HAVE_LONG_LONG\ntypedef struct { char c; PY_LONG_LONG x; } __Pyx_st_longlong;\n#endif\nstatic size_t __Pyx_BufFmt_TypeCharToAlignment(char ch, CYTHON_UNUSED int is_complex) {\n switch (ch) {\n case '?': case 'c': case 'b': case 'B': case 's': case 'p': return 1;\n case 'h': case 'H': return sizeof(__Pyx_st_short) - sizeof(short);\n case 'i': case 'I': return sizeof(__Pyx_st_int) - sizeof(int);\n case 'l': case 'L': return sizeof(__Pyx_st_long) - sizeof(long);\n#ifdef HAVE_LONG_LONG\n case 'q': case 'Q': return sizeof(__Pyx_st_longlong) - sizeof(PY_LONG_LONG);\n#endif\n case 'f': return sizeof(__Pyx_st_float) - sizeof(float);\n case 'd': return sizeof(__Pyx_st_double) - sizeof(double);\n case 'g': return sizeof(__Pyx_st_longdouble) - sizeof(long double);\n case 'P': case 'O': return sizeof(__Pyx_st_void_p) - sizeof(void*);\n default:\n __Pyx_BufFmt_RaiseUnexpectedChar(ch);\n return 0;\n }\n}\n/* These are for computing the padding at the end of the struct to align\n on the first member of the struct. This will probably the same as above,\n but we don't have any guarantees.\n */\ntypedef struct { short x; char c; } __Pyx_pad_short;\ntypedef struct { int x; char c; } __Pyx_pad_int;\ntypedef struct { long x; char c; } __Pyx_pad_long;\ntypedef struct { float x; char c; } __Pyx_pad_float;\ntypedef struct { double x; char c; } __Pyx_pad_double;\ntypedef struct { long double x; char c; } __Pyx_pad_longdouble;\ntypedef struct { void *x; char c; } __Pyx_pad_void_p;\n#ifdef HAVE_LONG_LONG\ntypedef struct { PY_LONG_LONG x; char c; } __Pyx_pad_longlong;\n#endif\nstatic size_t __Pyx_BufFmt_TypeCharToPadding(char ch, CYTHON_UNUSED int is_complex) {\n switch (ch) {\n case '?': case 'c': case 'b': case 'B': case 's': case 'p': return 1;\n case 'h': case 'H': return sizeof(__Pyx_pad_short) - sizeof(short);\n case 'i': case 'I': return sizeof(__Pyx_pad_int) - sizeof(int);\n case 'l': case 'L': return sizeof(__Pyx_pad_long) - sizeof(long);\n#ifdef HAVE_LONG_LONG\n case 'q': case 'Q': return sizeof(__Pyx_pad_longlong) - sizeof(PY_LONG_LONG);\n#endif\n case 'f': return sizeof(__Pyx_pad_float) - sizeof(float);\n case 'd': return sizeof(__Pyx_pad_double) - sizeof(double);\n case 'g': return sizeof(__Pyx_pad_longdouble) - sizeof(long double);\n case 'P': case 'O': return sizeof(__Pyx_pad_void_p) - sizeof(void*);\n default:\n __Pyx_BufFmt_RaiseUnexpectedChar(ch);\n return 0;\n }\n}\nstatic char __Pyx_BufFmt_TypeCharToGroup(char ch, int is_complex) {\n switch (ch) {\n case 'c':\n return 'H';\n case 'b': case 'h': case 'i':\n case 'l': case 'q': case 's': case 'p':\n return 'I';\n case 'B': case 'H': case 'I': case 'L': case 'Q':\n return 'U';\n case 'f': case 'd': case 'g':\n return (is_complex ? 'C' : 'R');\n case 'O':\n return 'O';\n case 'P':\n return 'P';\n default: {\n __Pyx_BufFmt_RaiseUnexpectedChar(ch);\n return 0;\n }\n }\n}\nstatic void __Pyx_BufFmt_RaiseExpected(__Pyx_BufFmt_Context* ctx) {\n if (ctx->head == NULL || ctx->head->field == &ctx->root) {\n const char* expected;\n const char* quote;\n if (ctx->head == NULL) {\n expected = \"end\";\n quote = \"\";\n } else {\n expected = ctx->head->field->type->name;\n quote = \"'\";\n }\n PyErr_Format(PyExc_ValueError,\n \"Buffer dtype mismatch, expected %s%s%s but got %s\",\n quote, expected, quote,\n __Pyx_BufFmt_DescribeTypeChar(ctx->enc_type, ctx->is_complex));\n } else {\n __Pyx_StructField* field = ctx->head->field;\n __Pyx_StructField* parent = (ctx->head - 1)->field;\n PyErr_Format(PyExc_ValueError,\n \"Buffer dtype mismatch, expected '%s' but got %s in '%s.%s'\",\n field->type->name, __Pyx_BufFmt_DescribeTypeChar(ctx->enc_type, ctx->is_complex),\n parent->type->name, field->name);\n }\n}\nstatic int __Pyx_BufFmt_ProcessTypeChunk(__Pyx_BufFmt_Context* ctx) {\n char group;\n size_t size, offset, arraysize = 1;\n if (ctx->enc_type == 0) return 0;\n if (ctx->head->field->type->arraysize[0]) {\n int i, ndim = 0;\n if (ctx->enc_type == 's' || ctx->enc_type == 'p') {\n ctx->is_valid_array = ctx->head->field->type->ndim == 1;\n ndim = 1;\n if (ctx->enc_count != ctx->head->field->type->arraysize[0]) {\n PyErr_Format(PyExc_ValueError,\n \"Expected a dimension of size %zu, got %zu\",\n ctx->head->field->type->arraysize[0], ctx->enc_count);\n return -1;\n }\n }\n if (!ctx->is_valid_array) {\n PyErr_Format(PyExc_ValueError, \"Expected %d dimensions, got %d\",\n ctx->head->field->type->ndim, ndim);\n return -1;\n }\n for (i = 0; i < ctx->head->field->type->ndim; i++) {\n arraysize *= ctx->head->field->type->arraysize[i];\n }\n ctx->is_valid_array = 0;\n ctx->enc_count = 1;\n }\n group = __Pyx_BufFmt_TypeCharToGroup(ctx->enc_type, ctx->is_complex);\n do {\n __Pyx_StructField* field = ctx->head->field;\n __Pyx_TypeInfo* type = field->type;\n if (ctx->enc_packmode == '@' || ctx->enc_packmode == '^') {\n size = __Pyx_BufFmt_TypeCharToNativeSize(ctx->enc_type, ctx->is_complex);\n } else {\n size = __Pyx_BufFmt_TypeCharToStandardSize(ctx->enc_type, ctx->is_complex);\n }\n if (ctx->enc_packmode == '@') {\n size_t align_at = __Pyx_BufFmt_TypeCharToAlignment(ctx->enc_type, ctx->is_complex);\n size_t align_mod_offset;\n if (align_at == 0) return -1;\n align_mod_offset = ctx->fmt_offset % align_at;\n if (align_mod_offset > 0) ctx->fmt_offset += align_at - align_mod_offset;\n if (ctx->struct_alignment == 0)\n ctx->struct_alignment = __Pyx_BufFmt_TypeCharToPadding(ctx->enc_type,\n ctx->is_complex);\n }\n if (type->size != size || type->typegroup != group) {\n if (type->typegroup == 'C' && type->fields != NULL) {\n size_t parent_offset = ctx->head->parent_offset + field->offset;\n ++ctx->head;\n ctx->head->field = type->fields;\n ctx->head->parent_offset = parent_offset;\n continue;\n }\n if ((type->typegroup == 'H' || group == 'H') && type->size == size) {\n } else {\n __Pyx_BufFmt_RaiseExpected(ctx);\n return -1;\n }\n }\n offset = ctx->head->parent_offset + field->offset;\n if (ctx->fmt_offset != offset) {\n PyErr_Format(PyExc_ValueError,\n \"Buffer dtype mismatch; next field is at offset %\" CYTHON_FORMAT_SSIZE_T \"d but %\" CYTHON_FORMAT_SSIZE_T \"d expected\",\n (Py_ssize_t)ctx->fmt_offset, (Py_ssize_t)offset);\n return -1;\n }\n ctx->fmt_offset += size;\n if (arraysize)\n ctx->fmt_offset += (arraysize - 1) * size;\n --ctx->enc_count;\n while (1) {\n if (field == &ctx->root) {\n ctx->head = NULL;\n if (ctx->enc_count != 0) {\n __Pyx_BufFmt_RaiseExpected(ctx);\n return -1;\n }\n break;\n }\n ctx->head->field = ++field;\n if (field->type == NULL) {\n --ctx->head;\n field = ctx->head->field;\n continue;\n } else if (field->type->typegroup == 'S') {\n size_t parent_offset = ctx->head->parent_offset + field->offset;\n if (field->type->fields->type == NULL) continue;\n field = field->type->fields;\n ++ctx->head;\n ctx->head->field = field;\n ctx->head->parent_offset = parent_offset;\n break;\n } else {\n break;\n }\n }\n } while (ctx->enc_count);\n ctx->enc_type = 0;\n ctx->is_complex = 0;\n return 0;\n}\nstatic CYTHON_INLINE PyObject *\n__pyx_buffmt_parse_array(__Pyx_BufFmt_Context* ctx, const char** tsp)\n{\n const char *ts = *tsp;\n int i = 0, number;\n int ndim = ctx->head->field->type->ndim;\n;\n ++ts;\n if (ctx->new_count != 1) {\n PyErr_SetString(PyExc_ValueError,\n \"Cannot handle repeated arrays in format string\");\n return NULL;\n }\n if (__Pyx_BufFmt_ProcessTypeChunk(ctx) == -1) return NULL;\n while (*ts && *ts != ')') {\n switch (*ts) {\n case ' ': case '\\f': case '\\r': case '\\n': case '\\t': case '\\v': continue;\n default: break;\n }\n number = __Pyx_BufFmt_ExpectNumber(&ts);\n if (number == -1) return NULL;\n if (i < ndim && (size_t) number != ctx->head->field->type->arraysize[i])\n return PyErr_Format(PyExc_ValueError,\n \"Expected a dimension of size %zu, got %d\",\n ctx->head->field->type->arraysize[i], number);\n if (*ts != ',' && *ts != ')')\n return PyErr_Format(PyExc_ValueError,\n \"Expected a comma in format string, got '%c'\", *ts);\n if (*ts == ',') ts++;\n i++;\n }\n if (i != ndim)\n return PyErr_Format(PyExc_ValueError, \"Expected %d dimension(s), got %d\",\n ctx->head->field->type->ndim, i);\n if (!*ts) {\n PyErr_SetString(PyExc_ValueError,\n \"Unexpected end of format string, expected ')'\");\n return NULL;\n }\n ctx->is_valid_array = 1;\n ctx->new_count = 1;\n *tsp = ++ts;\n return Py_None;\n}\nstatic const char* __Pyx_BufFmt_CheckString(__Pyx_BufFmt_Context* ctx, const char* ts) {\n int got_Z = 0;\n while (1) {\n switch(*ts) {\n case 0:\n if (ctx->enc_type != 0 && ctx->head == NULL) {\n __Pyx_BufFmt_RaiseExpected(ctx);\n return NULL;\n }\n if (__Pyx_BufFmt_ProcessTypeChunk(ctx) == -1) return NULL;\n if (ctx->head != NULL) {\n __Pyx_BufFmt_RaiseExpected(ctx);\n return NULL;\n }\n return ts;\n case ' ':\n case '\\r':\n case '\\n':\n ++ts;\n break;\n case '<':\n if (!__Pyx_IsLittleEndian()) {\n PyErr_SetString(PyExc_ValueError, \"Little-endian buffer not supported on big-endian compiler\");\n return NULL;\n }\n ctx->new_packmode = '=';\n ++ts;\n break;\n case '>':\n case '!':\n if (__Pyx_IsLittleEndian()) {\n PyErr_SetString(PyExc_ValueError, \"Big-endian buffer not supported on little-endian compiler\");\n return NULL;\n }\n ctx->new_packmode = '=';\n ++ts;\n break;\n case '=':\n case '@':\n case '^':\n ctx->new_packmode = *ts++;\n break;\n case 'T':\n {\n const char* ts_after_sub;\n size_t i, struct_count = ctx->new_count;\n size_t struct_alignment = ctx->struct_alignment;\n ctx->new_count = 1;\n ++ts;\n if (*ts != '{') {\n PyErr_SetString(PyExc_ValueError, \"Buffer acquisition: Expected '{' after 'T'\");\n return NULL;\n }\n if (__Pyx_BufFmt_ProcessTypeChunk(ctx) == -1) return NULL;\n ctx->enc_type = 0;\n ctx->enc_count = 0;\n ctx->struct_alignment = 0;\n ++ts;\n ts_after_sub = ts;\n for (i = 0; i != struct_count; ++i) {\n ts_after_sub = __Pyx_BufFmt_CheckString(ctx, ts);\n if (!ts_after_sub) return NULL;\n }\n ts = ts_after_sub;\n if (struct_alignment) ctx->struct_alignment = struct_alignment;\n }\n break;\n case '}':\n {\n size_t alignment = ctx->struct_alignment;\n ++ts;\n if (__Pyx_BufFmt_ProcessTypeChunk(ctx) == -1) return NULL;\n ctx->enc_type = 0;\n if (alignment && ctx->fmt_offset % alignment) {\n ctx->fmt_offset += alignment - (ctx->fmt_offset % alignment);\n }\n }\n return ts;\n case 'x':\n if (__Pyx_BufFmt_ProcessTypeChunk(ctx) == -1) return NULL;\n ctx->fmt_offset += ctx->new_count;\n ctx->new_count = 1;\n ctx->enc_count = 0;\n ctx->enc_type = 0;\n ctx->enc_packmode = ctx->new_packmode;\n ++ts;\n break;\n case 'Z':\n got_Z = 1;\n ++ts;\n if (*ts != 'f' && *ts != 'd' && *ts != 'g') {\n __Pyx_BufFmt_RaiseUnexpectedChar('Z');\n return NULL;\n }\n case 'c': case 'b': case 'B': case 'h': case 'H': case 'i': case 'I':\n case 'l': case 'L': case 'q': case 'Q':\n case 'f': case 'd': case 'g':\n case 'O': case 'p':\n if (ctx->enc_type == *ts && got_Z == ctx->is_complex &&\n ctx->enc_packmode == ctx->new_packmode) {\n ctx->enc_count += ctx->new_count;\n ctx->new_count = 1;\n got_Z = 0;\n ++ts;\n break;\n }\n case 's':\n if (__Pyx_BufFmt_ProcessTypeChunk(ctx) == -1) return NULL;\n ctx->enc_count = ctx->new_count;\n ctx->enc_packmode = ctx->new_packmode;\n ctx->enc_type = *ts;\n ctx->is_complex = got_Z;\n ++ts;\n ctx->new_count = 1;\n got_Z = 0;\n break;\n case ':':\n ++ts;\n while(*ts != ':') ++ts;\n ++ts;\n break;\n case '(':\n if (!__pyx_buffmt_parse_array(ctx, &ts)) return NULL;\n break;\n default:\n {\n int number = __Pyx_BufFmt_ExpectNumber(&ts);\n if (number == -1) return NULL;\n ctx->new_count = (size_t)number;\n }\n }\n }\n}\nstatic CYTHON_INLINE void __Pyx_ZeroBuffer(Py_buffer* buf) {\n buf->buf = NULL;\n buf->obj = NULL;\n buf->strides = __Pyx_zeros;\n buf->shape = __Pyx_zeros;\n buf->suboffsets = __Pyx_minusones;\n}\nstatic CYTHON_INLINE int __Pyx_GetBufferAndValidate(\n Py_buffer* buf, PyObject* obj, __Pyx_TypeInfo* dtype, int flags,\n int nd, int cast, __Pyx_BufFmt_StackElem* stack)\n{\n if (obj == Py_None || obj == NULL) {\n __Pyx_ZeroBuffer(buf);\n return 0;\n }\n buf->buf = NULL;\n if (__Pyx_GetBuffer(obj, buf, flags) == -1) goto fail;\n if (buf->ndim != nd) {\n PyErr_Format(PyExc_ValueError,\n \"Buffer has wrong number of dimensions (expected %d, got %d)\",\n nd, buf->ndim);\n goto fail;\n }\n if (!cast) {\n __Pyx_BufFmt_Context ctx;\n __Pyx_BufFmt_Init(&ctx, stack, dtype);\n if (!__Pyx_BufFmt_CheckString(&ctx, buf->format)) goto fail;\n }\n if ((unsigned)buf->itemsize != dtype->size) {\n PyErr_Format(PyExc_ValueError,\n \"Item size of buffer (%\" CYTHON_FORMAT_SSIZE_T \"d byte%s) does not match size of '%s' (%\" CYTHON_FORMAT_SSIZE_T \"d byte%s)\",\n buf->itemsize, (buf->itemsize > 1) ? \"s\" : \"\",\n dtype->name, (Py_ssize_t)dtype->size, (dtype->size > 1) ? \"s\" : \"\");\n goto fail;\n }\n if (buf->suboffsets == NULL) buf->suboffsets = __Pyx_minusones;\n return 0;\nfail:;\n __Pyx_ZeroBuffer(buf);\n return -1;\n}\nstatic CYTHON_INLINE void __Pyx_SafeReleaseBuffer(Py_buffer* info) {\n if (info->buf == NULL) return;\n if (info->suboffsets == __Pyx_minusones) info->suboffsets = NULL;\n __Pyx_ReleaseBuffer(info);\n}\n\n/* GetBuiltinName */\n static PyObject *__Pyx_GetBuiltinName(PyObject *name) {\n PyObject* result = __Pyx_PyObject_GetAttrStr(__pyx_b, name);\n if (unlikely(!result)) {\n PyErr_Format(PyExc_NameError,\n#if PY_MAJOR_VERSION >= 3\n \"name '%U' is not defined\", name);\n#else\n \"name '%.200s' is not defined\", PyString_AS_STRING(name));\n#endif\n }\n return result;\n}\n\n/* GetModuleGlobalName */\n static CYTHON_INLINE PyObject *__Pyx_GetModuleGlobalName(PyObject *name) {\n PyObject *result;\n#if CYTHON_COMPILING_IN_CPYTHON\n result = PyDict_GetItem(__pyx_d, name);\n if (likely(result)) {\n Py_INCREF(result);\n } else {\n#else\n result = PyObject_GetItem(__pyx_d, name);\n if (!result) {\n PyErr_Clear();\n#endif\n result = __Pyx_GetBuiltinName(name);\n }\n return result;\n}\n\n/* PyObjectCall */\n #if CYTHON_COMPILING_IN_CPYTHON\nstatic CYTHON_INLINE PyObject* __Pyx_PyObject_Call(PyObject *func, PyObject *arg, PyObject *kw) {\n PyObject *result;\n ternaryfunc call = func->ob_type->tp_call;\n if (unlikely(!call))\n return PyObject_Call(func, arg, kw);\n if (unlikely(Py_EnterRecursiveCall((char*)\" while calling a Python object\")))\n return NULL;\n result = (*call)(func, arg, kw);\n Py_LeaveRecursiveCall();\n if (unlikely(!result) && unlikely(!PyErr_Occurred())) {\n PyErr_SetString(\n PyExc_SystemError,\n \"NULL result without error in PyObject_Call\");\n }\n return result;\n}\n#endif\n\n/* ExtTypeTest */\n static CYTHON_INLINE int __Pyx_TypeTest(PyObject *obj, PyTypeObject *type) {\n if (unlikely(!type)) {\n PyErr_SetString(PyExc_SystemError, \"Missing type object\");\n return 0;\n }\n if (likely(PyObject_TypeCheck(obj, type)))\n return 1;\n PyErr_Format(PyExc_TypeError, \"Cannot convert %.200s to %.200s\",\n Py_TYPE(obj)->tp_name, type->tp_name);\n return 0;\n}\n\n/* PyObjectCallMethO */\n #if CYTHON_COMPILING_IN_CPYTHON\nstatic CYTHON_INLINE PyObject* __Pyx_PyObject_CallMethO(PyObject *func, PyObject *arg) {\n PyObject *self, *result;\n PyCFunction cfunc;\n cfunc = PyCFunction_GET_FUNCTION(func);\n self = PyCFunction_GET_SELF(func);\n if (unlikely(Py_EnterRecursiveCall((char*)\" while calling a Python object\")))\n return NULL;\n result = cfunc(self, arg);\n Py_LeaveRecursiveCall();\n if (unlikely(!result) && unlikely(!PyErr_Occurred())) {\n PyErr_SetString(\n PyExc_SystemError,\n \"NULL result without error in PyObject_Call\");\n }\n return result;\n}\n#endif\n\n/* PyObjectCallOneArg */\n #if CYTHON_COMPILING_IN_CPYTHON\nstatic PyObject* __Pyx__PyObject_CallOneArg(PyObject *func, PyObject *arg) {\n PyObject *result;\n PyObject *args = PyTuple_New(1);\n if (unlikely(!args)) return NULL;\n Py_INCREF(arg);\n PyTuple_SET_ITEM(args, 0, arg);\n result = __Pyx_PyObject_Call(func, args, NULL);\n Py_DECREF(args);\n return result;\n}\nstatic CYTHON_INLINE PyObject* __Pyx_PyObject_CallOneArg(PyObject *func, PyObject *arg) {\n#ifdef __Pyx_CyFunction_USED\n if (likely(PyCFunction_Check(func) || PyObject_TypeCheck(func, __pyx_CyFunctionType))) {\n#else\n if (likely(PyCFunction_Check(func))) {\n#endif\n if (likely(PyCFunction_GET_FLAGS(func) & METH_O)) {\n return __Pyx_PyObject_CallMethO(func, arg);\n }\n }\n return __Pyx__PyObject_CallOneArg(func, arg);\n}\n#else\nstatic CYTHON_INLINE PyObject* __Pyx_PyObject_CallOneArg(PyObject *func, PyObject *arg) {\n PyObject *result;\n PyObject *args = PyTuple_Pack(1, arg);\n if (unlikely(!args)) return NULL;\n result = __Pyx_PyObject_Call(func, args, NULL);\n Py_DECREF(args);\n return result;\n}\n#endif\n\n/* PyObjectCallNoArg */\n #if CYTHON_COMPILING_IN_CPYTHON\nstatic CYTHON_INLINE PyObject* __Pyx_PyObject_CallNoArg(PyObject *func) {\n#ifdef __Pyx_CyFunction_USED\n if (likely(PyCFunction_Check(func) || PyObject_TypeCheck(func, __pyx_CyFunctionType))) {\n#else\n if (likely(PyCFunction_Check(func))) {\n#endif\n if (likely(PyCFunction_GET_FLAGS(func) & METH_NOARGS)) {\n return __Pyx_PyObject_CallMethO(func, NULL);\n }\n }\n return __Pyx_PyObject_Call(func, __pyx_empty_tuple, NULL);\n}\n#endif\n\n/* BufferIndexError */\n static void __Pyx_RaiseBufferIndexError(int axis) {\n PyErr_Format(PyExc_IndexError,\n \"Out of bounds on buffer access (axis %d)\", axis);\n}\n\n/* SliceObject */\n static CYTHON_INLINE PyObject* __Pyx_PyObject_GetSlice(PyObject* obj,\n Py_ssize_t cstart, Py_ssize_t cstop,\n PyObject** _py_start, PyObject** _py_stop, PyObject** _py_slice,\n int has_cstart, int has_cstop, CYTHON_UNUSED int wraparound) {\n#if CYTHON_COMPILING_IN_CPYTHON\n PyMappingMethods* mp;\n#if PY_MAJOR_VERSION < 3\n PySequenceMethods* ms = Py_TYPE(obj)->tp_as_sequence;\n if (likely(ms && ms->sq_slice)) {\n if (!has_cstart) {\n if (_py_start && (*_py_start != Py_None)) {\n cstart = __Pyx_PyIndex_AsSsize_t(*_py_start);\n if ((cstart == (Py_ssize_t)-1) && PyErr_Occurred()) goto bad;\n } else\n cstart = 0;\n }\n if (!has_cstop) {\n if (_py_stop && (*_py_stop != Py_None)) {\n cstop = __Pyx_PyIndex_AsSsize_t(*_py_stop);\n if ((cstop == (Py_ssize_t)-1) && PyErr_Occurred()) goto bad;\n } else\n cstop = PY_SSIZE_T_MAX;\n }\n if (wraparound && unlikely((cstart < 0) | (cstop < 0)) && likely(ms->sq_length)) {\n Py_ssize_t l = ms->sq_length(obj);\n if (likely(l >= 0)) {\n if (cstop < 0) {\n cstop += l;\n if (cstop < 0) cstop = 0;\n }\n if (cstart < 0) {\n cstart += l;\n if (cstart < 0) cstart = 0;\n }\n } else {\n if (!PyErr_ExceptionMatches(PyExc_OverflowError))\n goto bad;\n PyErr_Clear();\n }\n }\n return ms->sq_slice(obj, cstart, cstop);\n }\n#endif\n mp = Py_TYPE(obj)->tp_as_mapping;\n if (likely(mp && mp->mp_subscript))\n#endif\n {\n PyObject* result;\n PyObject *py_slice, *py_start, *py_stop;\n if (_py_slice) {\n py_slice = *_py_slice;\n } else {\n PyObject* owned_start = NULL;\n PyObject* owned_stop = NULL;\n if (_py_start) {\n py_start = *_py_start;\n } else {\n if (has_cstart) {\n owned_start = py_start = PyInt_FromSsize_t(cstart);\n if (unlikely(!py_start)) goto bad;\n } else\n py_start = Py_None;\n }\n if (_py_stop) {\n py_stop = *_py_stop;\n } else {\n if (has_cstop) {\n owned_stop = py_stop = PyInt_FromSsize_t(cstop);\n if (unlikely(!py_stop)) {\n Py_XDECREF(owned_start);\n goto bad;\n }\n } else\n py_stop = Py_None;\n }\n py_slice = PySlice_New(py_start, py_stop, Py_None);\n Py_XDECREF(owned_start);\n Py_XDECREF(owned_stop);\n if (unlikely(!py_slice)) goto bad;\n }\n#if CYTHON_COMPILING_IN_CPYTHON\n result = mp->mp_subscript(obj, py_slice);\n#else\n result = PyObject_GetItem(obj, py_slice);\n#endif\n if (!_py_slice) {\n Py_DECREF(py_slice);\n }\n return result;\n }\n PyErr_Format(PyExc_TypeError,\n \"'%.200s' object is unsliceable\", Py_TYPE(obj)->tp_name);\nbad:\n return NULL;\n}\n\n/* BufferFallbackError */\n static void __Pyx_RaiseBufferFallbackError(void) {\n PyErr_SetString(PyExc_ValueError,\n \"Buffer acquisition failed on assignment; and then reacquiring the old buffer failed too!\");\n}\n\n/* PyErrFetchRestore */\n #if CYTHON_COMPILING_IN_CPYTHON\nstatic CYTHON_INLINE void __Pyx_ErrRestoreInState(PyThreadState *tstate, PyObject *type, PyObject *value, PyObject *tb) {\n PyObject *tmp_type, *tmp_value, *tmp_tb;\n tmp_type = tstate->curexc_type;\n tmp_value = tstate->curexc_value;\n tmp_tb = tstate->curexc_traceback;\n tstate->curexc_type = type;\n tstate->curexc_value = value;\n tstate->curexc_traceback = tb;\n Py_XDECREF(tmp_type);\n Py_XDECREF(tmp_value);\n Py_XDECREF(tmp_tb);\n}\nstatic CYTHON_INLINE void __Pyx_ErrFetchInState(PyThreadState *tstate, PyObject **type, PyObject **value, PyObject **tb) {\n *type = tstate->curexc_type;\n *value = tstate->curexc_value;\n *tb = tstate->curexc_traceback;\n tstate->curexc_type = 0;\n tstate->curexc_value = 0;\n tstate->curexc_traceback = 0;\n}\n#endif\n\n/* RaiseException */\n #if PY_MAJOR_VERSION < 3\nstatic void __Pyx_Raise(PyObject *type, PyObject *value, PyObject *tb,\n CYTHON_UNUSED PyObject *cause) {\n __Pyx_PyThreadState_declare\n Py_XINCREF(type);\n if (!value || value == Py_None)\n value = NULL;\n else\n Py_INCREF(value);\n if (!tb || tb == Py_None)\n tb = NULL;\n else {\n Py_INCREF(tb);\n if (!PyTraceBack_Check(tb)) {\n PyErr_SetString(PyExc_TypeError,\n \"raise: arg 3 must be a traceback or None\");\n goto raise_error;\n }\n }\n if (PyType_Check(type)) {\n#if CYTHON_COMPILING_IN_PYPY\n if (!value) {\n Py_INCREF(Py_None);\n value = Py_None;\n }\n#endif\n PyErr_NormalizeException(&type, &value, &tb);\n } else {\n if (value) {\n PyErr_SetString(PyExc_TypeError,\n \"instance exception may not have a separate value\");\n goto raise_error;\n }\n value = type;\n type = (PyObject*) Py_TYPE(type);\n Py_INCREF(type);\n if (!PyType_IsSubtype((PyTypeObject *)type, (PyTypeObject *)PyExc_BaseException)) {\n PyErr_SetString(PyExc_TypeError,\n \"raise: exception class must be a subclass of BaseException\");\n goto raise_error;\n }\n }\n __Pyx_PyThreadState_assign\n __Pyx_ErrRestore(type, value, tb);\n return;\nraise_error:\n Py_XDECREF(value);\n Py_XDECREF(type);\n Py_XDECREF(tb);\n return;\n}\n#else\nstatic void __Pyx_Raise(PyObject *type, PyObject *value, PyObject *tb, PyObject *cause) {\n PyObject* owned_instance = NULL;\n if (tb == Py_None) {\n tb = 0;\n } else if (tb && !PyTraceBack_Check(tb)) {\n PyErr_SetString(PyExc_TypeError,\n \"raise: arg 3 must be a traceback or None\");\n goto bad;\n }\n if (value == Py_None)\n value = 0;\n if (PyExceptionInstance_Check(type)) {\n if (value) {\n PyErr_SetString(PyExc_TypeError,\n \"instance exception may not have a separate value\");\n goto bad;\n }\n value = type;\n type = (PyObject*) Py_TYPE(value);\n } else if (PyExceptionClass_Check(type)) {\n PyObject *instance_class = NULL;\n if (value && PyExceptionInstance_Check(value)) {\n instance_class = (PyObject*) Py_TYPE(value);\n if (instance_class != type) {\n int is_subclass = PyObject_IsSubclass(instance_class, type);\n if (!is_subclass) {\n instance_class = NULL;\n } else if (unlikely(is_subclass == -1)) {\n goto bad;\n } else {\n type = instance_class;\n }\n }\n }\n if (!instance_class) {\n PyObject *args;\n if (!value)\n args = PyTuple_New(0);\n else if (PyTuple_Check(value)) {\n Py_INCREF(value);\n args = value;\n } else\n args = PyTuple_Pack(1, value);\n if (!args)\n goto bad;\n owned_instance = PyObject_Call(type, args, NULL);\n Py_DECREF(args);\n if (!owned_instance)\n goto bad;\n value = owned_instance;\n if (!PyExceptionInstance_Check(value)) {\n PyErr_Format(PyExc_TypeError,\n \"calling %R should have returned an instance of \"\n \"BaseException, not %R\",\n type, Py_TYPE(value));\n goto bad;\n }\n }\n } else {\n PyErr_SetString(PyExc_TypeError,\n \"raise: exception class must be a subclass of BaseException\");\n goto bad;\n }\n#if PY_VERSION_HEX >= 0x03030000\n if (cause) {\n#else\n if (cause && cause != Py_None) {\n#endif\n PyObject *fixed_cause;\n if (cause == Py_None) {\n fixed_cause = NULL;\n } else if (PyExceptionClass_Check(cause)) {\n fixed_cause = PyObject_CallObject(cause, NULL);\n if (fixed_cause == NULL)\n goto bad;\n } else if (PyExceptionInstance_Check(cause)) {\n fixed_cause = cause;\n Py_INCREF(fixed_cause);\n } else {\n PyErr_SetString(PyExc_TypeError,\n \"exception causes must derive from \"\n \"BaseException\");\n goto bad;\n }\n PyException_SetCause(value, fixed_cause);\n }\n PyErr_SetObject(type, value);\n if (tb) {\n#if CYTHON_COMPILING_IN_PYPY\n PyObject *tmp_type, *tmp_value, *tmp_tb;\n PyErr_Fetch(&tmp_type, &tmp_value, &tmp_tb);\n Py_INCREF(tb);\n PyErr_Restore(tmp_type, tmp_value, tb);\n Py_XDECREF(tmp_tb);\n#else\n PyThreadState *tstate = PyThreadState_GET();\n PyObject* tmp_tb = tstate->curexc_traceback;\n if (tb != tmp_tb) {\n Py_INCREF(tb);\n tstate->curexc_traceback = tb;\n Py_XDECREF(tmp_tb);\n }\n#endif\n }\nbad:\n Py_XDECREF(owned_instance);\n return;\n}\n#endif\n\n/* RaiseTooManyValuesToUnpack */\n static CYTHON_INLINE void __Pyx_RaiseTooManyValuesError(Py_ssize_t expected) {\n PyErr_Format(PyExc_ValueError,\n \"too many values to unpack (expected %\" CYTHON_FORMAT_SSIZE_T \"d)\", expected);\n}\n\n/* RaiseNeedMoreValuesToUnpack */\n static CYTHON_INLINE void __Pyx_RaiseNeedMoreValuesError(Py_ssize_t index) {\n PyErr_Format(PyExc_ValueError,\n \"need more than %\" CYTHON_FORMAT_SSIZE_T \"d value%.1s to unpack\",\n index, (index == 1) ? \"\" : \"s\");\n}\n\n/* RaiseNoneIterError */\n static CYTHON_INLINE void __Pyx_RaiseNoneNotIterableError(void) {\n PyErr_SetString(PyExc_TypeError, \"'NoneType' object is not iterable\");\n}\n\n/* Import */\n static PyObject *__Pyx_Import(PyObject *name, PyObject *from_list, int level) {\n PyObject *empty_list = 0;\n PyObject *module = 0;\n PyObject *global_dict = 0;\n PyObject *empty_dict = 0;\n PyObject *list;\n #if PY_VERSION_HEX < 0x03030000\n PyObject *py_import;\n py_import = __Pyx_PyObject_GetAttrStr(__pyx_b, __pyx_n_s_import);\n if (!py_import)\n goto bad;\n #endif\n if (from_list)\n list = from_list;\n else {\n empty_list = PyList_New(0);\n if (!empty_list)\n goto bad;\n list = empty_list;\n }\n global_dict = PyModule_GetDict(__pyx_m);\n if (!global_dict)\n goto bad;\n empty_dict = PyDict_New();\n if (!empty_dict)\n goto bad;\n {\n #if PY_MAJOR_VERSION >= 3\n if (level == -1) {\n if (strchr(__Pyx_MODULE_NAME, '.')) {\n #if PY_VERSION_HEX < 0x03030000\n PyObject *py_level = PyInt_FromLong(1);\n if (!py_level)\n goto bad;\n module = PyObject_CallFunctionObjArgs(py_import,\n name, global_dict, empty_dict, list, py_level, NULL);\n Py_DECREF(py_level);\n #else\n module = PyImport_ImportModuleLevelObject(\n name, global_dict, empty_dict, list, 1);\n #endif\n if (!module) {\n if (!PyErr_ExceptionMatches(PyExc_ImportError))\n goto bad;\n PyErr_Clear();\n }\n }\n level = 0;\n }\n #endif\n if (!module) {\n #if PY_VERSION_HEX < 0x03030000\n PyObject *py_level = PyInt_FromLong(level);\n if (!py_level)\n goto bad;\n module = PyObject_CallFunctionObjArgs(py_import,\n name, global_dict, empty_dict, list, py_level, NULL);\n Py_DECREF(py_level);\n #else\n module = PyImport_ImportModuleLevelObject(\n name, global_dict, empty_dict, list, level);\n #endif\n }\n }\nbad:\n #if PY_VERSION_HEX < 0x03030000\n Py_XDECREF(py_import);\n #endif\n Py_XDECREF(empty_list);\n Py_XDECREF(empty_dict);\n return module;\n}\n\n/* CodeObjectCache */\n static int __pyx_bisect_code_objects(__Pyx_CodeObjectCacheEntry* entries, int count, int code_line) {\n int start = 0, mid = 0, end = count - 1;\n if (end >= 0 && code_line > entries[end].code_line) {\n return count;\n }\n while (start < end) {\n mid = start + (end - start) / 2;\n if (code_line < entries[mid].code_line) {\n end = mid;\n } else if (code_line > entries[mid].code_line) {\n start = mid + 1;\n } else {\n return mid;\n }\n }\n if (code_line <= entries[mid].code_line) {\n return mid;\n } else {\n return mid + 1;\n }\n}\nstatic PyCodeObject *__pyx_find_code_object(int code_line) {\n PyCodeObject* code_object;\n int pos;\n if (unlikely(!code_line) || unlikely(!__pyx_code_cache.entries)) {\n return NULL;\n }\n pos = __pyx_bisect_code_objects(__pyx_code_cache.entries, __pyx_code_cache.count, code_line);\n if (unlikely(pos >= __pyx_code_cache.count) || unlikely(__pyx_code_cache.entries[pos].code_line != code_line)) {\n return NULL;\n }\n code_object = __pyx_code_cache.entries[pos].code_object;\n Py_INCREF(code_object);\n return code_object;\n}\nstatic void __pyx_insert_code_object(int code_line, PyCodeObject* code_object) {\n int pos, i;\n __Pyx_CodeObjectCacheEntry* entries = __pyx_code_cache.entries;\n if (unlikely(!code_line)) {\n return;\n }\n if (unlikely(!entries)) {\n entries = (__Pyx_CodeObjectCacheEntry*)PyMem_Malloc(64*sizeof(__Pyx_CodeObjectCacheEntry));\n if (likely(entries)) {\n __pyx_code_cache.entries = entries;\n __pyx_code_cache.max_count = 64;\n __pyx_code_cache.count = 1;\n entries[0].code_line = code_line;\n entries[0].code_object = code_object;\n Py_INCREF(code_object);\n }\n return;\n }\n pos = __pyx_bisect_code_objects(__pyx_code_cache.entries, __pyx_code_cache.count, code_line);\n if ((pos < __pyx_code_cache.count) && unlikely(__pyx_code_cache.entries[pos].code_line == code_line)) {\n PyCodeObject* tmp = entries[pos].code_object;\n entries[pos].code_object = code_object;\n Py_DECREF(tmp);\n return;\n }\n if (__pyx_code_cache.count == __pyx_code_cache.max_count) {\n int new_max = __pyx_code_cache.max_count + 64;\n entries = (__Pyx_CodeObjectCacheEntry*)PyMem_Realloc(\n __pyx_code_cache.entries, (size_t)new_max*sizeof(__Pyx_CodeObjectCacheEntry));\n if (unlikely(!entries)) {\n return;\n }\n __pyx_code_cache.entries = entries;\n __pyx_code_cache.max_count = new_max;\n }\n for (i=__pyx_code_cache.count; i>pos; i--) {\n entries[i] = entries[i-1];\n }\n entries[pos].code_line = code_line;\n entries[pos].code_object = code_object;\n __pyx_code_cache.count++;\n Py_INCREF(code_object);\n}\n\n/* AddTraceback */\n #include \"compile.h\"\n#include \"frameobject.h\"\n#include \"traceback.h\"\nstatic PyCodeObject* __Pyx_CreateCodeObjectForTraceback(\n const char *funcname, int c_line,\n int py_line, const char *filename) {\n PyCodeObject *py_code = 0;\n PyObject *py_srcfile = 0;\n PyObject *py_funcname = 0;\n #if PY_MAJOR_VERSION < 3\n py_srcfile = PyString_FromString(filename);\n #else\n py_srcfile = PyUnicode_FromString(filename);\n #endif\n if (!py_srcfile) goto bad;\n if (c_line) {\n #if PY_MAJOR_VERSION < 3\n py_funcname = PyString_FromFormat( \"%s (%s:%d)\", funcname, __pyx_cfilenm, c_line);\n #else\n py_funcname = PyUnicode_FromFormat( \"%s (%s:%d)\", funcname, __pyx_cfilenm, c_line);\n #endif\n }\n else {\n #if PY_MAJOR_VERSION < 3\n py_funcname = PyString_FromString(funcname);\n #else\n py_funcname = PyUnicode_FromString(funcname);\n #endif\n }\n if (!py_funcname) goto bad;\n py_code = __Pyx_PyCode_New(\n 0,\n 0,\n 0,\n 0,\n 0,\n __pyx_empty_bytes, /*PyObject *code,*/\n __pyx_empty_tuple, /*PyObject *consts,*/\n __pyx_empty_tuple, /*PyObject *names,*/\n __pyx_empty_tuple, /*PyObject *varnames,*/\n __pyx_empty_tuple, /*PyObject *freevars,*/\n __pyx_empty_tuple, /*PyObject *cellvars,*/\n py_srcfile, /*PyObject *filename,*/\n py_funcname, /*PyObject *name,*/\n py_line,\n __pyx_empty_bytes /*PyObject *lnotab*/\n );\n Py_DECREF(py_srcfile);\n Py_DECREF(py_funcname);\n return py_code;\nbad:\n Py_XDECREF(py_srcfile);\n Py_XDECREF(py_funcname);\n return NULL;\n}\nstatic void __Pyx_AddTraceback(const char *funcname, int c_line,\n int py_line, const char *filename) {\n PyCodeObject *py_code = 0;\n PyFrameObject *py_frame = 0;\n py_code = __pyx_find_code_object(c_line ? c_line : py_line);\n if (!py_code) {\n py_code = __Pyx_CreateCodeObjectForTraceback(\n funcname, c_line, py_line, filename);\n if (!py_code) goto bad;\n __pyx_insert_code_object(c_line ? c_line : py_line, py_code);\n }\n py_frame = PyFrame_New(\n PyThreadState_GET(), /*PyThreadState *tstate,*/\n py_code, /*PyCodeObject *code,*/\n __pyx_d, /*PyObject *globals,*/\n 0 /*PyObject *locals*/\n );\n if (!py_frame) goto bad;\n py_frame->f_lineno = py_line;\n PyTraceBack_Here(py_frame);\nbad:\n Py_XDECREF(py_code);\n Py_XDECREF(py_frame);\n}\n\n#if PY_MAJOR_VERSION < 3\nstatic int __Pyx_GetBuffer(PyObject *obj, Py_buffer *view, int flags) {\n if (PyObject_CheckBuffer(obj)) return PyObject_GetBuffer(obj, view, flags);\n if (PyObject_TypeCheck(obj, __pyx_ptype_5numpy_ndarray)) return __pyx_pw_5numpy_7ndarray_1__getbuffer__(obj, view, flags);\n PyErr_Format(PyExc_TypeError, \"'%.200s' does not have the buffer interface\", Py_TYPE(obj)->tp_name);\n return -1;\n}\nstatic void __Pyx_ReleaseBuffer(Py_buffer *view) {\n PyObject *obj = view->obj;\n if (!obj) return;\n if (PyObject_CheckBuffer(obj)) {\n PyBuffer_Release(view);\n return;\n }\n if (PyObject_TypeCheck(obj, __pyx_ptype_5numpy_ndarray)) { __pyx_pw_5numpy_7ndarray_3__releasebuffer__(obj, view); return; }\n Py_DECREF(obj);\n view->obj = NULL;\n}\n#endif\n\n\n /* CIntFromPyVerify */\n #define __PYX_VERIFY_RETURN_INT(target_type, func_type, func_value)\\\n __PYX__VERIFY_RETURN_INT(target_type, func_type, func_value, 0)\n#define __PYX_VERIFY_RETURN_INT_EXC(target_type, func_type, func_value)\\\n __PYX__VERIFY_RETURN_INT(target_type, func_type, func_value, 1)\n#define __PYX__VERIFY_RETURN_INT(target_type, func_type, func_value, exc)\\\n {\\\n func_type value = func_value;\\\n if (sizeof(target_type) < sizeof(func_type)) {\\\n if (unlikely(value != (func_type) (target_type) value)) {\\\n func_type zero = 0;\\\n if (exc && unlikely(value == (func_type)-1 && PyErr_Occurred()))\\\n return (target_type) -1;\\\n if (is_unsigned && unlikely(value < zero))\\\n goto raise_neg_overflow;\\\n else\\\n goto raise_overflow;\\\n }\\\n }\\\n return (target_type) value;\\\n }\n\n/* CIntToPy */\n static CYTHON_INLINE PyObject* __Pyx_PyInt_From_int(int value) {\n const int neg_one = (int) -1, const_zero = (int) 0;\n const int is_unsigned = neg_one > const_zero;\n if (is_unsigned) {\n if (sizeof(int) < sizeof(long)) {\n return PyInt_FromLong((long) value);\n } else if (sizeof(int) <= sizeof(unsigned long)) {\n return PyLong_FromUnsignedLong((unsigned long) value);\n } else if (sizeof(int) <= sizeof(unsigned PY_LONG_LONG)) {\n return PyLong_FromUnsignedLongLong((unsigned PY_LONG_LONG) value);\n }\n } else {\n if (sizeof(int) <= sizeof(long)) {\n return PyInt_FromLong((long) value);\n } else if (sizeof(int) <= sizeof(PY_LONG_LONG)) {\n return PyLong_FromLongLong((PY_LONG_LONG) value);\n }\n }\n {\n int one = 1; int little = (int)*(unsigned char *)&one;\n unsigned char *bytes = (unsigned char *)&value;\n return _PyLong_FromByteArray(bytes, sizeof(int),\n little, !is_unsigned);\n }\n}\n\n/* None */\n #if CYTHON_CCOMPLEX\n #ifdef __cplusplus\n static CYTHON_INLINE __pyx_t_float_complex __pyx_t_float_complex_from_parts(float x, float y) {\n return ::std::complex< float >(x, y);\n }\n #else\n static CYTHON_INLINE __pyx_t_float_complex __pyx_t_float_complex_from_parts(float x, float y) {\n return x + y*(__pyx_t_float_complex)_Complex_I;\n }\n #endif\n#else\n static CYTHON_INLINE __pyx_t_float_complex __pyx_t_float_complex_from_parts(float x, float y) {\n __pyx_t_float_complex z;\n z.real = x;\n z.imag = y;\n return z;\n }\n#endif\n\n/* None */\n #if CYTHON_CCOMPLEX\n#else\n static CYTHON_INLINE int __Pyx_c_eqf(__pyx_t_float_complex a, __pyx_t_float_complex b) {\n return (a.real == b.real) && (a.imag == b.imag);\n }\n static CYTHON_INLINE __pyx_t_float_complex __Pyx_c_sumf(__pyx_t_float_complex a, __pyx_t_float_complex b) {\n __pyx_t_float_complex z;\n z.real = a.real + b.real;\n z.imag = a.imag + b.imag;\n return z;\n }\n static CYTHON_INLINE __pyx_t_float_complex __Pyx_c_difff(__pyx_t_float_complex a, __pyx_t_float_complex b) {\n __pyx_t_float_complex z;\n z.real = a.real - b.real;\n z.imag = a.imag - b.imag;\n return z;\n }\n static CYTHON_INLINE __pyx_t_float_complex __Pyx_c_prodf(__pyx_t_float_complex a, __pyx_t_float_complex b) {\n __pyx_t_float_complex z;\n z.real = a.real * b.real - a.imag * b.imag;\n z.imag = a.real * b.imag + a.imag * b.real;\n return z;\n }\n static CYTHON_INLINE __pyx_t_float_complex __Pyx_c_quotf(__pyx_t_float_complex a, __pyx_t_float_complex b) {\n __pyx_t_float_complex z;\n float denom = b.real * b.real + b.imag * b.imag;\n z.real = (a.real * b.real + a.imag * b.imag) / denom;\n z.imag = (a.imag * b.real - a.real * b.imag) / denom;\n return z;\n }\n static CYTHON_INLINE __pyx_t_float_complex __Pyx_c_negf(__pyx_t_float_complex a) {\n __pyx_t_float_complex z;\n z.real = -a.real;\n z.imag = -a.imag;\n return z;\n }\n static CYTHON_INLINE int __Pyx_c_is_zerof(__pyx_t_float_complex a) {\n return (a.real == 0) && (a.imag == 0);\n }\n static CYTHON_INLINE __pyx_t_float_complex __Pyx_c_conjf(__pyx_t_float_complex a) {\n __pyx_t_float_complex z;\n z.real = a.real;\n z.imag = -a.imag;\n return z;\n }\n #if 1\n static CYTHON_INLINE float __Pyx_c_absf(__pyx_t_float_complex z) {\n #if !defined(HAVE_HYPOT) || defined(_MSC_VER)\n return sqrtf(z.real*z.real + z.imag*z.imag);\n #else\n return hypotf(z.real, z.imag);\n #endif\n }\n static CYTHON_INLINE __pyx_t_float_complex __Pyx_c_powf(__pyx_t_float_complex a, __pyx_t_float_complex b) {\n __pyx_t_float_complex z;\n float r, lnr, theta, z_r, z_theta;\n if (b.imag == 0 && b.real == (int)b.real) {\n if (b.real < 0) {\n float denom = a.real * a.real + a.imag * a.imag;\n a.real = a.real / denom;\n a.imag = -a.imag / denom;\n b.real = -b.real;\n }\n switch ((int)b.real) {\n case 0:\n z.real = 1;\n z.imag = 0;\n return z;\n case 1:\n return a;\n case 2:\n z = __Pyx_c_prodf(a, a);\n return __Pyx_c_prodf(a, a);\n case 3:\n z = __Pyx_c_prodf(a, a);\n return __Pyx_c_prodf(z, a);\n case 4:\n z = __Pyx_c_prodf(a, a);\n return __Pyx_c_prodf(z, z);\n }\n }\n if (a.imag == 0) {\n if (a.real == 0) {\n return a;\n }\n r = a.real;\n theta = 0;\n } else {\n r = __Pyx_c_absf(a);\n theta = atan2f(a.imag, a.real);\n }\n lnr = logf(r);\n z_r = expf(lnr * b.real - theta * b.imag);\n z_theta = theta * b.real + lnr * b.imag;\n z.real = z_r * cosf(z_theta);\n z.imag = z_r * sinf(z_theta);\n return z;\n }\n #endif\n#endif\n\n/* None */\n #if CYTHON_CCOMPLEX\n #ifdef __cplusplus\n static CYTHON_INLINE __pyx_t_double_complex __pyx_t_double_complex_from_parts(double x, double y) {\n return ::std::complex< double >(x, y);\n }\n #else\n static CYTHON_INLINE __pyx_t_double_complex __pyx_t_double_complex_from_parts(double x, double y) {\n return x + y*(__pyx_t_double_complex)_Complex_I;\n }\n #endif\n#else\n static CYTHON_INLINE __pyx_t_double_complex __pyx_t_double_complex_from_parts(double x, double y) {\n __pyx_t_double_complex z;\n z.real = x;\n z.imag = y;\n return z;\n }\n#endif\n\n/* None */\n #if CYTHON_CCOMPLEX\n#else\n static CYTHON_INLINE int __Pyx_c_eq(__pyx_t_double_complex a, __pyx_t_double_complex b) {\n return (a.real == b.real) && (a.imag == b.imag);\n }\n static CYTHON_INLINE __pyx_t_double_complex __Pyx_c_sum(__pyx_t_double_complex a, __pyx_t_double_complex b) {\n __pyx_t_double_complex z;\n z.real = a.real + b.real;\n z.imag = a.imag + b.imag;\n return z;\n }\n static CYTHON_INLINE __pyx_t_double_complex __Pyx_c_diff(__pyx_t_double_complex a, __pyx_t_double_complex b) {\n __pyx_t_double_complex z;\n z.real = a.real - b.real;\n z.imag = a.imag - b.imag;\n return z;\n }\n static CYTHON_INLINE __pyx_t_double_complex __Pyx_c_prod(__pyx_t_double_complex a, __pyx_t_double_complex b) {\n __pyx_t_double_complex z;\n z.real = a.real * b.real - a.imag * b.imag;\n z.imag = a.real * b.imag + a.imag * b.real;\n return z;\n }\n static CYTHON_INLINE __pyx_t_double_complex __Pyx_c_quot(__pyx_t_double_complex a, __pyx_t_double_complex b) {\n __pyx_t_double_complex z;\n double denom = b.real * b.real + b.imag * b.imag;\n z.real = (a.real * b.real + a.imag * b.imag) / denom;\n z.imag = (a.imag * b.real - a.real * b.imag) / denom;\n return z;\n }\n static CYTHON_INLINE __pyx_t_double_complex __Pyx_c_neg(__pyx_t_double_complex a) {\n __pyx_t_double_complex z;\n z.real = -a.real;\n z.imag = -a.imag;\n return z;\n }\n static CYTHON_INLINE int __Pyx_c_is_zero(__pyx_t_double_complex a) {\n return (a.real == 0) && (a.imag == 0);\n }\n static CYTHON_INLINE __pyx_t_double_complex __Pyx_c_conj(__pyx_t_double_complex a) {\n __pyx_t_double_complex z;\n z.real = a.real;\n z.imag = -a.imag;\n return z;\n }\n #if 1\n static CYTHON_INLINE double __Pyx_c_abs(__pyx_t_double_complex z) {\n #if !defined(HAVE_HYPOT) || defined(_MSC_VER)\n return sqrt(z.real*z.real + z.imag*z.imag);\n #else\n return hypot(z.real, z.imag);\n #endif\n }\n static CYTHON_INLINE __pyx_t_double_complex __Pyx_c_pow(__pyx_t_double_complex a, __pyx_t_double_complex b) {\n __pyx_t_double_complex z;\n double r, lnr, theta, z_r, z_theta;\n if (b.imag == 0 && b.real == (int)b.real) {\n if (b.real < 0) {\n double denom = a.real * a.real + a.imag * a.imag;\n a.real = a.real / denom;\n a.imag = -a.imag / denom;\n b.real = -b.real;\n }\n switch ((int)b.real) {\n case 0:\n z.real = 1;\n z.imag = 0;\n return z;\n case 1:\n return a;\n case 2:\n z = __Pyx_c_prod(a, a);\n return __Pyx_c_prod(a, a);\n case 3:\n z = __Pyx_c_prod(a, a);\n return __Pyx_c_prod(z, a);\n case 4:\n z = __Pyx_c_prod(a, a);\n return __Pyx_c_prod(z, z);\n }\n }\n if (a.imag == 0) {\n if (a.real == 0) {\n return a;\n }\n r = a.real;\n theta = 0;\n } else {\n r = __Pyx_c_abs(a);\n theta = atan2(a.imag, a.real);\n }\n lnr = log(r);\n z_r = exp(lnr * b.real - theta * b.imag);\n z_theta = theta * b.real + lnr * b.imag;\n z.real = z_r * cos(z_theta);\n z.imag = z_r * sin(z_theta);\n return z;\n }\n #endif\n#endif\n\n/* CIntToPy */\n static CYTHON_INLINE PyObject* __Pyx_PyInt_From_enum__NPY_TYPES(enum NPY_TYPES value) {\n const enum NPY_TYPES neg_one = (enum NPY_TYPES) -1, const_zero = (enum NPY_TYPES) 0;\n const int is_unsigned = neg_one > const_zero;\n if (is_unsigned) {\n if (sizeof(enum NPY_TYPES) < sizeof(long)) {\n return PyInt_FromLong((long) value);\n } else if (sizeof(enum NPY_TYPES) <= sizeof(unsigned long)) {\n return PyLong_FromUnsignedLong((unsigned long) value);\n } else if (sizeof(enum NPY_TYPES) <= sizeof(unsigned PY_LONG_LONG)) {\n return PyLong_FromUnsignedLongLong((unsigned PY_LONG_LONG) value);\n }\n } else {\n if (sizeof(enum NPY_TYPES) <= sizeof(long)) {\n return PyInt_FromLong((long) value);\n } else if (sizeof(enum NPY_TYPES) <= sizeof(PY_LONG_LONG)) {\n return PyLong_FromLongLong((PY_LONG_LONG) value);\n }\n }\n {\n int one = 1; int little = (int)*(unsigned char *)&one;\n unsigned char *bytes = (unsigned char *)&value;\n return _PyLong_FromByteArray(bytes, sizeof(enum NPY_TYPES),\n little, !is_unsigned);\n }\n}\n\n/* CIntFromPy */\n static CYTHON_INLINE npy_int32 __Pyx_PyInt_As_npy_int32(PyObject *x) {\n const npy_int32 neg_one = (npy_int32) -1, const_zero = (npy_int32) 0;\n const int is_unsigned = neg_one > const_zero;\n#if PY_MAJOR_VERSION < 3\n if (likely(PyInt_Check(x))) {\n if (sizeof(npy_int32) < sizeof(long)) {\n __PYX_VERIFY_RETURN_INT(npy_int32, long, PyInt_AS_LONG(x))\n } else {\n long val = PyInt_AS_LONG(x);\n if (is_unsigned && unlikely(val < 0)) {\n goto raise_neg_overflow;\n }\n return (npy_int32) val;\n }\n } else\n#endif\n if (likely(PyLong_Check(x))) {\n if (is_unsigned) {\n#if CYTHON_USE_PYLONG_INTERNALS\n const digit* digits = ((PyLongObject*)x)->ob_digit;\n switch (Py_SIZE(x)) {\n case 0: return (npy_int32) 0;\n case 1: __PYX_VERIFY_RETURN_INT(npy_int32, digit, digits[0])\n case 2:\n if (8 * sizeof(npy_int32) > 1 * PyLong_SHIFT) {\n if (8 * sizeof(unsigned long) > 2 * PyLong_SHIFT) {\n __PYX_VERIFY_RETURN_INT(npy_int32, unsigned long, (((((unsigned long)digits[1]) << PyLong_SHIFT) | (unsigned long)digits[0])))\n } else if (8 * sizeof(npy_int32) >= 2 * PyLong_SHIFT) {\n return (npy_int32) (((((npy_int32)digits[1]) << PyLong_SHIFT) | (npy_int32)digits[0]));\n }\n }\n break;\n case 3:\n if (8 * sizeof(npy_int32) > 2 * PyLong_SHIFT) {\n if (8 * sizeof(unsigned long) > 3 * PyLong_SHIFT) {\n __PYX_VERIFY_RETURN_INT(npy_int32, unsigned long, (((((((unsigned long)digits[2]) << PyLong_SHIFT) | (unsigned long)digits[1]) << PyLong_SHIFT) | (unsigned long)digits[0])))\n } else if (8 * sizeof(npy_int32) >= 3 * PyLong_SHIFT) {\n return (npy_int32) (((((((npy_int32)digits[2]) << PyLong_SHIFT) | (npy_int32)digits[1]) << PyLong_SHIFT) | (npy_int32)digits[0]));\n }\n }\n break;\n case 4:\n if (8 * sizeof(npy_int32) > 3 * PyLong_SHIFT) {\n if (8 * sizeof(unsigned long) > 4 * PyLong_SHIFT) {\n __PYX_VERIFY_RETURN_INT(npy_int32, unsigned long, (((((((((unsigned long)digits[3]) << PyLong_SHIFT) | (unsigned long)digits[2]) << PyLong_SHIFT) | (unsigned long)digits[1]) << PyLong_SHIFT) | (unsigned long)digits[0])))\n } else if (8 * sizeof(npy_int32) >= 4 * PyLong_SHIFT) {\n return (npy_int32) (((((((((npy_int32)digits[3]) << PyLong_SHIFT) | (npy_int32)digits[2]) << PyLong_SHIFT) | (npy_int32)digits[1]) << PyLong_SHIFT) | (npy_int32)digits[0]));\n }\n }\n break;\n }\n#endif\n#if CYTHON_COMPILING_IN_CPYTHON\n if (unlikely(Py_SIZE(x) < 0)) {\n goto raise_neg_overflow;\n }\n#else\n {\n int result = PyObject_RichCompareBool(x, Py_False, Py_LT);\n if (unlikely(result < 0))\n return (npy_int32) -1;\n if (unlikely(result == 1))\n goto raise_neg_overflow;\n }\n#endif\n if (sizeof(npy_int32) <= sizeof(unsigned long)) {\n __PYX_VERIFY_RETURN_INT_EXC(npy_int32, unsigned long, PyLong_AsUnsignedLong(x))\n } else if (sizeof(npy_int32) <= sizeof(unsigned PY_LONG_LONG)) {\n __PYX_VERIFY_RETURN_INT_EXC(npy_int32, unsigned PY_LONG_LONG, PyLong_AsUnsignedLongLong(x))\n }\n } else {\n#if CYTHON_USE_PYLONG_INTERNALS\n const digit* digits = ((PyLongObject*)x)->ob_digit;\n switch (Py_SIZE(x)) {\n case 0: return (npy_int32) 0;\n case -1: __PYX_VERIFY_RETURN_INT(npy_int32, sdigit, (sdigit) (-(sdigit)digits[0]))\n case 1: __PYX_VERIFY_RETURN_INT(npy_int32, digit, +digits[0])\n case -2:\n if (8 * sizeof(npy_int32) - 1 > 1 * PyLong_SHIFT) {\n if (8 * sizeof(unsigned long) > 2 * PyLong_SHIFT) {\n __PYX_VERIFY_RETURN_INT(npy_int32, long, -(long) (((((unsigned long)digits[1]) << PyLong_SHIFT) | (unsigned long)digits[0])))\n } else if (8 * sizeof(npy_int32) - 1 > 2 * PyLong_SHIFT) {\n return (npy_int32) (((npy_int32)-1)*(((((npy_int32)digits[1]) << PyLong_SHIFT) | (npy_int32)digits[0])));\n }\n }\n break;\n case 2:\n if (8 * sizeof(npy_int32) > 1 * PyLong_SHIFT) {\n if (8 * sizeof(unsigned long) > 2 * PyLong_SHIFT) {\n __PYX_VERIFY_RETURN_INT(npy_int32, unsigned long, (((((unsigned long)digits[1]) << PyLong_SHIFT) | (unsigned long)digits[0])))\n } else if (8 * sizeof(npy_int32) - 1 > 2 * PyLong_SHIFT) {\n return (npy_int32) ((((((npy_int32)digits[1]) << PyLong_SHIFT) | (npy_int32)digits[0])));\n }\n }\n break;\n case -3:\n if (8 * sizeof(npy_int32) - 1 > 2 * PyLong_SHIFT) {\n if (8 * sizeof(unsigned long) > 3 * PyLong_SHIFT) {\n __PYX_VERIFY_RETURN_INT(npy_int32, long, -(long) (((((((unsigned long)digits[2]) << PyLong_SHIFT) | (unsigned long)digits[1]) << PyLong_SHIFT) | (unsigned long)digits[0])))\n } else if (8 * sizeof(npy_int32) - 1 > 3 * PyLong_SHIFT) {\n return (npy_int32) (((npy_int32)-1)*(((((((npy_int32)digits[2]) << PyLong_SHIFT) | (npy_int32)digits[1]) << PyLong_SHIFT) | (npy_int32)digits[0])));\n }\n }\n break;\n case 3:\n if (8 * sizeof(npy_int32) > 2 * PyLong_SHIFT) {\n if (8 * sizeof(unsigned long) > 3 * PyLong_SHIFT) {\n __PYX_VERIFY_RETURN_INT(npy_int32, unsigned long, (((((((unsigned long)digits[2]) << PyLong_SHIFT) | (unsigned long)digits[1]) << PyLong_SHIFT) | (unsigned long)digits[0])))\n } else if (8 * sizeof(npy_int32) - 1 > 3 * PyLong_SHIFT) {\n return (npy_int32) ((((((((npy_int32)digits[2]) << PyLong_SHIFT) | (npy_int32)digits[1]) << PyLong_SHIFT) | (npy_int32)digits[0])));\n }\n }\n break;\n case -4:\n if (8 * sizeof(npy_int32) - 1 > 3 * PyLong_SHIFT) {\n if (8 * sizeof(unsigned long) > 4 * PyLong_SHIFT) {\n __PYX_VERIFY_RETURN_INT(npy_int32, long, -(long) (((((((((unsigned long)digits[3]) << PyLong_SHIFT) | (unsigned long)digits[2]) << PyLong_SHIFT) | (unsigned long)digits[1]) << PyLong_SHIFT) | (unsigned long)digits[0])))\n } else if (8 * sizeof(npy_int32) - 1 > 4 * PyLong_SHIFT) {\n return (npy_int32) (((npy_int32)-1)*(((((((((npy_int32)digits[3]) << PyLong_SHIFT) | (npy_int32)digits[2]) << PyLong_SHIFT) | (npy_int32)digits[1]) << PyLong_SHIFT) | (npy_int32)digits[0])));\n }\n }\n break;\n case 4:\n if (8 * sizeof(npy_int32) > 3 * PyLong_SHIFT) {\n if (8 * sizeof(unsigned long) > 4 * PyLong_SHIFT) {\n __PYX_VERIFY_RETURN_INT(npy_int32, unsigned long, (((((((((unsigned long)digits[3]) << PyLong_SHIFT) | (unsigned long)digits[2]) << PyLong_SHIFT) | (unsigned long)digits[1]) << PyLong_SHIFT) | (unsigned long)digits[0])))\n } else if (8 * sizeof(npy_int32) - 1 > 4 * PyLong_SHIFT) {\n return (npy_int32) ((((((((((npy_int32)digits[3]) << PyLong_SHIFT) | (npy_int32)digits[2]) << PyLong_SHIFT) | (npy_int32)digits[1]) << PyLong_SHIFT) | (npy_int32)digits[0])));\n }\n }\n break;\n }\n#endif\n if (sizeof(npy_int32) <= sizeof(long)) {\n __PYX_VERIFY_RETURN_INT_EXC(npy_int32, long, PyLong_AsLong(x))\n } else if (sizeof(npy_int32) <= sizeof(PY_LONG_LONG)) {\n __PYX_VERIFY_RETURN_INT_EXC(npy_int32, PY_LONG_LONG, PyLong_AsLongLong(x))\n }\n }\n {\n#if CYTHON_COMPILING_IN_PYPY && !defined(_PyLong_AsByteArray)\n PyErr_SetString(PyExc_RuntimeError,\n \"_PyLong_AsByteArray() not available in PyPy, cannot convert large numbers\");\n#else\n npy_int32 val;\n PyObject *v = __Pyx_PyNumber_IntOrLong(x);\n #if PY_MAJOR_VERSION < 3\n if (likely(v) && !PyLong_Check(v)) {\n PyObject *tmp = v;\n v = PyNumber_Long(tmp);\n Py_DECREF(tmp);\n }\n #endif\n if (likely(v)) {\n int one = 1; int is_little = (int)*(unsigned char *)&one;\n unsigned char *bytes = (unsigned char *)&val;\n int ret = _PyLong_AsByteArray((PyLongObject *)v,\n bytes, sizeof(val),\n is_little, !is_unsigned);\n Py_DECREF(v);\n if (likely(!ret))\n return val;\n }\n#endif\n return (npy_int32) -1;\n }\n } else {\n npy_int32 val;\n PyObject *tmp = __Pyx_PyNumber_IntOrLong(x);\n if (!tmp) return (npy_int32) -1;\n val = __Pyx_PyInt_As_npy_int32(tmp);\n Py_DECREF(tmp);\n return val;\n }\nraise_overflow:\n PyErr_SetString(PyExc_OverflowError,\n \"value too large to convert to npy_int32\");\n return (npy_int32) -1;\nraise_neg_overflow:\n PyErr_SetString(PyExc_OverflowError,\n \"can't convert negative value to npy_int32\");\n return (npy_int32) -1;\n}\n\n/* CIntFromPy */\n static CYTHON_INLINE int __Pyx_PyInt_As_int(PyObject *x) {\n const int neg_one = (int) -1, const_zero = (int) 0;\n const int is_unsigned = neg_one > const_zero;\n#if PY_MAJOR_VERSION < 3\n if (likely(PyInt_Check(x))) {\n if (sizeof(int) < sizeof(long)) {\n __PYX_VERIFY_RETURN_INT(int, long, PyInt_AS_LONG(x))\n } else {\n long val = PyInt_AS_LONG(x);\n if (is_unsigned && unlikely(val < 0)) {\n goto raise_neg_overflow;\n }\n return (int) val;\n }\n } else\n#endif\n if (likely(PyLong_Check(x))) {\n if (is_unsigned) {\n#if CYTHON_USE_PYLONG_INTERNALS\n const digit* digits = ((PyLongObject*)x)->ob_digit;\n switch (Py_SIZE(x)) {\n case 0: return (int) 0;\n case 1: __PYX_VERIFY_RETURN_INT(int, digit, digits[0])\n case 2:\n if (8 * sizeof(int) > 1 * PyLong_SHIFT) {\n if (8 * sizeof(unsigned long) > 2 * PyLong_SHIFT) {\n __PYX_VERIFY_RETURN_INT(int, unsigned long, (((((unsigned long)digits[1]) << PyLong_SHIFT) | (unsigned long)digits[0])))\n } else if (8 * sizeof(int) >= 2 * PyLong_SHIFT) {\n return (int) (((((int)digits[1]) << PyLong_SHIFT) | (int)digits[0]));\n }\n }\n break;\n case 3:\n if (8 * sizeof(int) > 2 * PyLong_SHIFT) {\n if (8 * sizeof(unsigned long) > 3 * PyLong_SHIFT) {\n __PYX_VERIFY_RETURN_INT(int, unsigned long, (((((((unsigned long)digits[2]) << PyLong_SHIFT) | (unsigned long)digits[1]) << PyLong_SHIFT) | (unsigned long)digits[0])))\n } else if (8 * sizeof(int) >= 3 * PyLong_SHIFT) {\n return (int) (((((((int)digits[2]) << PyLong_SHIFT) | (int)digits[1]) << PyLong_SHIFT) | (int)digits[0]));\n }\n }\n break;\n case 4:\n if (8 * sizeof(int) > 3 * PyLong_SHIFT) {\n if (8 * sizeof(unsigned long) > 4 * PyLong_SHIFT) {\n __PYX_VERIFY_RETURN_INT(int, unsigned long, (((((((((unsigned long)digits[3]) << PyLong_SHIFT) | (unsigned long)digits[2]) << PyLong_SHIFT) | (unsigned long)digits[1]) << PyLong_SHIFT) | (unsigned long)digits[0])))\n } else if (8 * sizeof(int) >= 4 * PyLong_SHIFT) {\n return (int) (((((((((int)digits[3]) << PyLong_SHIFT) | (int)digits[2]) << PyLong_SHIFT) | (int)digits[1]) << PyLong_SHIFT) | (int)digits[0]));\n }\n }\n break;\n }\n#endif\n#if CYTHON_COMPILING_IN_CPYTHON\n if (unlikely(Py_SIZE(x) < 0)) {\n goto raise_neg_overflow;\n }\n#else\n {\n int result = PyObject_RichCompareBool(x, Py_False, Py_LT);\n if (unlikely(result < 0))\n return (int) -1;\n if (unlikely(result == 1))\n goto raise_neg_overflow;\n }\n#endif\n if (sizeof(int) <= sizeof(unsigned long)) {\n __PYX_VERIFY_RETURN_INT_EXC(int, unsigned long, PyLong_AsUnsignedLong(x))\n } else if (sizeof(int) <= sizeof(unsigned PY_LONG_LONG)) {\n __PYX_VERIFY_RETURN_INT_EXC(int, unsigned PY_LONG_LONG, PyLong_AsUnsignedLongLong(x))\n }\n } else {\n#if CYTHON_USE_PYLONG_INTERNALS\n const digit* digits = ((PyLongObject*)x)->ob_digit;\n switch (Py_SIZE(x)) {\n case 0: return (int) 0;\n case -1: __PYX_VERIFY_RETURN_INT(int, sdigit, (sdigit) (-(sdigit)digits[0]))\n case 1: __PYX_VERIFY_RETURN_INT(int, digit, +digits[0])\n case -2:\n if (8 * sizeof(int) - 1 > 1 * PyLong_SHIFT) {\n if (8 * sizeof(unsigned long) > 2 * PyLong_SHIFT) {\n __PYX_VERIFY_RETURN_INT(int, long, -(long) (((((unsigned long)digits[1]) << PyLong_SHIFT) | (unsigned long)digits[0])))\n } else if (8 * sizeof(int) - 1 > 2 * PyLong_SHIFT) {\n return (int) (((int)-1)*(((((int)digits[1]) << PyLong_SHIFT) | (int)digits[0])));\n }\n }\n break;\n case 2:\n if (8 * sizeof(int) > 1 * PyLong_SHIFT) {\n if (8 * sizeof(unsigned long) > 2 * PyLong_SHIFT) {\n __PYX_VERIFY_RETURN_INT(int, unsigned long, (((((unsigned long)digits[1]) << PyLong_SHIFT) | (unsigned long)digits[0])))\n } else if (8 * sizeof(int) - 1 > 2 * PyLong_SHIFT) {\n return (int) ((((((int)digits[1]) << PyLong_SHIFT) | (int)digits[0])));\n }\n }\n break;\n case -3:\n if (8 * sizeof(int) - 1 > 2 * PyLong_SHIFT) {\n if (8 * sizeof(unsigned long) > 3 * PyLong_SHIFT) {\n __PYX_VERIFY_RETURN_INT(int, long, -(long) (((((((unsigned long)digits[2]) << PyLong_SHIFT) | (unsigned long)digits[1]) << PyLong_SHIFT) | (unsigned long)digits[0])))\n } else if (8 * sizeof(int) - 1 > 3 * PyLong_SHIFT) {\n return (int) (((int)-1)*(((((((int)digits[2]) << PyLong_SHIFT) | (int)digits[1]) << PyLong_SHIFT) | (int)digits[0])));\n }\n }\n break;\n case 3:\n if (8 * sizeof(int) > 2 * PyLong_SHIFT) {\n if (8 * sizeof(unsigned long) > 3 * PyLong_SHIFT) {\n __PYX_VERIFY_RETURN_INT(int, unsigned long, (((((((unsigned long)digits[2]) << PyLong_SHIFT) | (unsigned long)digits[1]) << PyLong_SHIFT) | (unsigned long)digits[0])))\n } else if (8 * sizeof(int) - 1 > 3 * PyLong_SHIFT) {\n return (int) ((((((((int)digits[2]) << PyLong_SHIFT) | (int)digits[1]) << PyLong_SHIFT) | (int)digits[0])));\n }\n }\n break;\n case -4:\n if (8 * sizeof(int) - 1 > 3 * PyLong_SHIFT) {\n if (8 * sizeof(unsigned long) > 4 * PyLong_SHIFT) {\n __PYX_VERIFY_RETURN_INT(int, long, -(long) (((((((((unsigned long)digits[3]) << PyLong_SHIFT) | (unsigned long)digits[2]) << PyLong_SHIFT) | (unsigned long)digits[1]) << PyLong_SHIFT) | (unsigned long)digits[0])))\n } else if (8 * sizeof(int) - 1 > 4 * PyLong_SHIFT) {\n return (int) (((int)-1)*(((((((((int)digits[3]) << PyLong_SHIFT) | (int)digits[2]) << PyLong_SHIFT) | (int)digits[1]) << PyLong_SHIFT) | (int)digits[0])));\n }\n }\n break;\n case 4:\n if (8 * sizeof(int) > 3 * PyLong_SHIFT) {\n if (8 * sizeof(unsigned long) > 4 * PyLong_SHIFT) {\n __PYX_VERIFY_RETURN_INT(int, unsigned long, (((((((((unsigned long)digits[3]) << PyLong_SHIFT) | (unsigned long)digits[2]) << PyLong_SHIFT) | (unsigned long)digits[1]) << PyLong_SHIFT) | (unsigned long)digits[0])))\n } else if (8 * sizeof(int) - 1 > 4 * PyLong_SHIFT) {\n return (int) ((((((((((int)digits[3]) << PyLong_SHIFT) | (int)digits[2]) << PyLong_SHIFT) | (int)digits[1]) << PyLong_SHIFT) | (int)digits[0])));\n }\n }\n break;\n }\n#endif\n if (sizeof(int) <= sizeof(long)) {\n __PYX_VERIFY_RETURN_INT_EXC(int, long, PyLong_AsLong(x))\n } else if (sizeof(int) <= sizeof(PY_LONG_LONG)) {\n __PYX_VERIFY_RETURN_INT_EXC(int, PY_LONG_LONG, PyLong_AsLongLong(x))\n }\n }\n {\n#if CYTHON_COMPILING_IN_PYPY && !defined(_PyLong_AsByteArray)\n PyErr_SetString(PyExc_RuntimeError,\n \"_PyLong_AsByteArray() not available in PyPy, cannot convert large numbers\");\n#else\n int val;\n PyObject *v = __Pyx_PyNumber_IntOrLong(x);\n #if PY_MAJOR_VERSION < 3\n if (likely(v) && !PyLong_Check(v)) {\n PyObject *tmp = v;\n v = PyNumber_Long(tmp);\n Py_DECREF(tmp);\n }\n #endif\n if (likely(v)) {\n int one = 1; int is_little = (int)*(unsigned char *)&one;\n unsigned char *bytes = (unsigned char *)&val;\n int ret = _PyLong_AsByteArray((PyLongObject *)v,\n bytes, sizeof(val),\n is_little, !is_unsigned);\n Py_DECREF(v);\n if (likely(!ret))\n return val;\n }\n#endif\n return (int) -1;\n }\n } else {\n int val;\n PyObject *tmp = __Pyx_PyNumber_IntOrLong(x);\n if (!tmp) return (int) -1;\n val = __Pyx_PyInt_As_int(tmp);\n Py_DECREF(tmp);\n return val;\n }\nraise_overflow:\n PyErr_SetString(PyExc_OverflowError,\n \"value too large to convert to int\");\n return (int) -1;\nraise_neg_overflow:\n PyErr_SetString(PyExc_OverflowError,\n \"can't convert negative value to int\");\n return (int) -1;\n}\n\n/* CIntToPy */\n static CYTHON_INLINE PyObject* __Pyx_PyInt_From_long(long value) {\n const long neg_one = (long) -1, const_zero = (long) 0;\n const int is_unsigned = neg_one > const_zero;\n if (is_unsigned) {\n if (sizeof(long) < sizeof(long)) {\n return PyInt_FromLong((long) value);\n } else if (sizeof(long) <= sizeof(unsigned long)) {\n return PyLong_FromUnsignedLong((unsigned long) value);\n } else if (sizeof(long) <= sizeof(unsigned PY_LONG_LONG)) {\n return PyLong_FromUnsignedLongLong((unsigned PY_LONG_LONG) value);\n }\n } else {\n if (sizeof(long) <= sizeof(long)) {\n return PyInt_FromLong((long) value);\n } else if (sizeof(long) <= sizeof(PY_LONG_LONG)) {\n return PyLong_FromLongLong((PY_LONG_LONG) value);\n }\n }\n {\n int one = 1; int little = (int)*(unsigned char *)&one;\n unsigned char *bytes = (unsigned char *)&value;\n return _PyLong_FromByteArray(bytes, sizeof(long),\n little, !is_unsigned);\n }\n}\n\n/* CIntFromPy */\n static CYTHON_INLINE long __Pyx_PyInt_As_long(PyObject *x) {\n const long neg_one = (long) -1, const_zero = (long) 0;\n const int is_unsigned = neg_one > const_zero;\n#if PY_MAJOR_VERSION < 3\n if (likely(PyInt_Check(x))) {\n if (sizeof(long) < sizeof(long)) {\n __PYX_VERIFY_RETURN_INT(long, long, PyInt_AS_LONG(x))\n } else {\n long val = PyInt_AS_LONG(x);\n if (is_unsigned && unlikely(val < 0)) {\n goto raise_neg_overflow;\n }\n return (long) val;\n }\n } else\n#endif\n if (likely(PyLong_Check(x))) {\n if (is_unsigned) {\n#if CYTHON_USE_PYLONG_INTERNALS\n const digit* digits = ((PyLongObject*)x)->ob_digit;\n switch (Py_SIZE(x)) {\n case 0: return (long) 0;\n case 1: __PYX_VERIFY_RETURN_INT(long, digit, digits[0])\n case 2:\n if (8 * sizeof(long) > 1 * PyLong_SHIFT) {\n if (8 * sizeof(unsigned long) > 2 * PyLong_SHIFT) {\n __PYX_VERIFY_RETURN_INT(long, unsigned long, (((((unsigned long)digits[1]) << PyLong_SHIFT) | (unsigned long)digits[0])))\n } else if (8 * sizeof(long) >= 2 * PyLong_SHIFT) {\n return (long) (((((long)digits[1]) << PyLong_SHIFT) | (long)digits[0]));\n }\n }\n break;\n case 3:\n if (8 * sizeof(long) > 2 * PyLong_SHIFT) {\n if (8 * sizeof(unsigned long) > 3 * PyLong_SHIFT) {\n __PYX_VERIFY_RETURN_INT(long, unsigned long, (((((((unsigned long)digits[2]) << PyLong_SHIFT) | (unsigned long)digits[1]) << PyLong_SHIFT) | (unsigned long)digits[0])))\n } else if (8 * sizeof(long) >= 3 * PyLong_SHIFT) {\n return (long) (((((((long)digits[2]) << PyLong_SHIFT) | (long)digits[1]) << PyLong_SHIFT) | (long)digits[0]));\n }\n }\n break;\n case 4:\n if (8 * sizeof(long) > 3 * PyLong_SHIFT) {\n if (8 * sizeof(unsigned long) > 4 * PyLong_SHIFT) {\n __PYX_VERIFY_RETURN_INT(long, unsigned long, (((((((((unsigned long)digits[3]) << PyLong_SHIFT) | (unsigned long)digits[2]) << PyLong_SHIFT) | (unsigned long)digits[1]) << PyLong_SHIFT) | (unsigned long)digits[0])))\n } else if (8 * sizeof(long) >= 4 * PyLong_SHIFT) {\n return (long) (((((((((long)digits[3]) << PyLong_SHIFT) | (long)digits[2]) << PyLong_SHIFT) | (long)digits[1]) << PyLong_SHIFT) | (long)digits[0]));\n }\n }\n break;\n }\n#endif\n#if CYTHON_COMPILING_IN_CPYTHON\n if (unlikely(Py_SIZE(x) < 0)) {\n goto raise_neg_overflow;\n }\n#else\n {\n int result = PyObject_RichCompareBool(x, Py_False, Py_LT);\n if (unlikely(result < 0))\n return (long) -1;\n if (unlikely(result == 1))\n goto raise_neg_overflow;\n }\n#endif\n if (sizeof(long) <= sizeof(unsigned long)) {\n __PYX_VERIFY_RETURN_INT_EXC(long, unsigned long, PyLong_AsUnsignedLong(x))\n } else if (sizeof(long) <= sizeof(unsigned PY_LONG_LONG)) {\n __PYX_VERIFY_RETURN_INT_EXC(long, unsigned PY_LONG_LONG, PyLong_AsUnsignedLongLong(x))\n }\n } else {\n#if CYTHON_USE_PYLONG_INTERNALS\n const digit* digits = ((PyLongObject*)x)->ob_digit;\n switch (Py_SIZE(x)) {\n case 0: return (long) 0;\n case -1: __PYX_VERIFY_RETURN_INT(long, sdigit, (sdigit) (-(sdigit)digits[0]))\n case 1: __PYX_VERIFY_RETURN_INT(long, digit, +digits[0])\n case -2:\n if (8 * sizeof(long) - 1 > 1 * PyLong_SHIFT) {\n if (8 * sizeof(unsigned long) > 2 * PyLong_SHIFT) {\n __PYX_VERIFY_RETURN_INT(long, long, -(long) (((((unsigned long)digits[1]) << PyLong_SHIFT) | (unsigned long)digits[0])))\n } else if (8 * sizeof(long) - 1 > 2 * PyLong_SHIFT) {\n return (long) (((long)-1)*(((((long)digits[1]) << PyLong_SHIFT) | (long)digits[0])));\n }\n }\n break;\n case 2:\n if (8 * sizeof(long) > 1 * PyLong_SHIFT) {\n if (8 * sizeof(unsigned long) > 2 * PyLong_SHIFT) {\n __PYX_VERIFY_RETURN_INT(long, unsigned long, (((((unsigned long)digits[1]) << PyLong_SHIFT) | (unsigned long)digits[0])))\n } else if (8 * sizeof(long) - 1 > 2 * PyLong_SHIFT) {\n return (long) ((((((long)digits[1]) << PyLong_SHIFT) | (long)digits[0])));\n }\n }\n break;\n case -3:\n if (8 * sizeof(long) - 1 > 2 * PyLong_SHIFT) {\n if (8 * sizeof(unsigned long) > 3 * PyLong_SHIFT) {\n __PYX_VERIFY_RETURN_INT(long, long, -(long) (((((((unsigned long)digits[2]) << PyLong_SHIFT) | (unsigned long)digits[1]) << PyLong_SHIFT) | (unsigned long)digits[0])))\n } else if (8 * sizeof(long) - 1 > 3 * PyLong_SHIFT) {\n return (long) (((long)-1)*(((((((long)digits[2]) << PyLong_SHIFT) | (long)digits[1]) << PyLong_SHIFT) | (long)digits[0])));\n }\n }\n break;\n case 3:\n if (8 * sizeof(long) > 2 * PyLong_SHIFT) {\n if (8 * sizeof(unsigned long) > 3 * PyLong_SHIFT) {\n __PYX_VERIFY_RETURN_INT(long, unsigned long, (((((((unsigned long)digits[2]) << PyLong_SHIFT) | (unsigned long)digits[1]) << PyLong_SHIFT) | (unsigned long)digits[0])))\n } else if (8 * sizeof(long) - 1 > 3 * PyLong_SHIFT) {\n return (long) ((((((((long)digits[2]) << PyLong_SHIFT) | (long)digits[1]) << PyLong_SHIFT) | (long)digits[0])));\n }\n }\n break;\n case -4:\n if (8 * sizeof(long) - 1 > 3 * PyLong_SHIFT) {\n if (8 * sizeof(unsigned long) > 4 * PyLong_SHIFT) {\n __PYX_VERIFY_RETURN_INT(long, long, -(long) (((((((((unsigned long)digits[3]) << PyLong_SHIFT) | (unsigned long)digits[2]) << PyLong_SHIFT) | (unsigned long)digits[1]) << PyLong_SHIFT) | (unsigned long)digits[0])))\n } else if (8 * sizeof(long) - 1 > 4 * PyLong_SHIFT) {\n return (long) (((long)-1)*(((((((((long)digits[3]) << PyLong_SHIFT) | (long)digits[2]) << PyLong_SHIFT) | (long)digits[1]) << PyLong_SHIFT) | (long)digits[0])));\n }\n }\n break;\n case 4:\n if (8 * sizeof(long) > 3 * PyLong_SHIFT) {\n if (8 * sizeof(unsigned long) > 4 * PyLong_SHIFT) {\n __PYX_VERIFY_RETURN_INT(long, unsigned long, (((((((((unsigned long)digits[3]) << PyLong_SHIFT) | (unsigned long)digits[2]) << PyLong_SHIFT) | (unsigned long)digits[1]) << PyLong_SHIFT) | (unsigned long)digits[0])))\n } else if (8 * sizeof(long) - 1 > 4 * PyLong_SHIFT) {\n return (long) ((((((((((long)digits[3]) << PyLong_SHIFT) | (long)digits[2]) << PyLong_SHIFT) | (long)digits[1]) << PyLong_SHIFT) | (long)digits[0])));\n }\n }\n break;\n }\n#endif\n if (sizeof(long) <= sizeof(long)) {\n __PYX_VERIFY_RETURN_INT_EXC(long, long, PyLong_AsLong(x))\n } else if (sizeof(long) <= sizeof(PY_LONG_LONG)) {\n __PYX_VERIFY_RETURN_INT_EXC(long, PY_LONG_LONG, PyLong_AsLongLong(x))\n }\n }\n {\n#if CYTHON_COMPILING_IN_PYPY && !defined(_PyLong_AsByteArray)\n PyErr_SetString(PyExc_RuntimeError,\n \"_PyLong_AsByteArray() not available in PyPy, cannot convert large numbers\");\n#else\n long val;\n PyObject *v = __Pyx_PyNumber_IntOrLong(x);\n #if PY_MAJOR_VERSION < 3\n if (likely(v) && !PyLong_Check(v)) {\n PyObject *tmp = v;\n v = PyNumber_Long(tmp);\n Py_DECREF(tmp);\n }\n #endif\n if (likely(v)) {\n int one = 1; int is_little = (int)*(unsigned char *)&one;\n unsigned char *bytes = (unsigned char *)&val;\n int ret = _PyLong_AsByteArray((PyLongObject *)v,\n bytes, sizeof(val),\n is_little, !is_unsigned);\n Py_DECREF(v);\n if (likely(!ret))\n return val;\n }\n#endif\n return (long) -1;\n }\n } else {\n long val;\n PyObject *tmp = __Pyx_PyNumber_IntOrLong(x);\n if (!tmp) return (long) -1;\n val = __Pyx_PyInt_As_long(tmp);\n Py_DECREF(tmp);\n return val;\n }\nraise_overflow:\n PyErr_SetString(PyExc_OverflowError,\n \"value too large to convert to long\");\n return (long) -1;\nraise_neg_overflow:\n PyErr_SetString(PyExc_OverflowError,\n \"can't convert negative value to long\");\n return (long) -1;\n}\n\n/* CheckBinaryVersion */\n static int __Pyx_check_binary_version(void) {\n char ctversion[4], rtversion[4];\n PyOS_snprintf(ctversion, 4, \"%d.%d\", PY_MAJOR_VERSION, PY_MINOR_VERSION);\n PyOS_snprintf(rtversion, 4, \"%s\", Py_GetVersion());\n if (ctversion[0] != rtversion[0] || ctversion[2] != rtversion[2]) {\n char message[200];\n PyOS_snprintf(message, sizeof(message),\n \"compiletime version %s of module '%.100s' \"\n \"does not match runtime version %s\",\n ctversion, __Pyx_MODULE_NAME, rtversion);\n return PyErr_WarnEx(NULL, message, 1);\n }\n return 0;\n}\n\n/* ModuleImport */\n #ifndef __PYX_HAVE_RT_ImportModule\n#define __PYX_HAVE_RT_ImportModule\nstatic PyObject *__Pyx_ImportModule(const char *name) {\n PyObject *py_name = 0;\n PyObject *py_module = 0;\n py_name = __Pyx_PyIdentifier_FromString(name);\n if (!py_name)\n goto bad;\n py_module = PyImport_Import(py_name);\n Py_DECREF(py_name);\n return py_module;\nbad:\n Py_XDECREF(py_name);\n return 0;\n}\n#endif\n\n/* TypeImport */\n #ifndef __PYX_HAVE_RT_ImportType\n#define __PYX_HAVE_RT_ImportType\nstatic PyTypeObject *__Pyx_ImportType(const char *module_name, const char *class_name,\n size_t size, int strict)\n{\n PyObject *py_module = 0;\n PyObject *result = 0;\n PyObject *py_name = 0;\n char warning[200];\n Py_ssize_t basicsize;\n#ifdef Py_LIMITED_API\n PyObject *py_basicsize;\n#endif\n py_module = __Pyx_ImportModule(module_name);\n if (!py_module)\n goto bad;\n py_name = __Pyx_PyIdentifier_FromString(class_name);\n if (!py_name)\n goto bad;\n result = PyObject_GetAttr(py_module, py_name);\n Py_DECREF(py_name);\n py_name = 0;\n Py_DECREF(py_module);\n py_module = 0;\n if (!result)\n goto bad;\n if (!PyType_Check(result)) {\n PyErr_Format(PyExc_TypeError,\n \"%.200s.%.200s is not a type object\",\n module_name, class_name);\n goto bad;\n }\n#ifndef Py_LIMITED_API\n basicsize = ((PyTypeObject *)result)->tp_basicsize;\n#else\n py_basicsize = PyObject_GetAttrString(result, \"__basicsize__\");\n if (!py_basicsize)\n goto bad;\n basicsize = PyLong_AsSsize_t(py_basicsize);\n Py_DECREF(py_basicsize);\n py_basicsize = 0;\n if (basicsize == (Py_ssize_t)-1 && PyErr_Occurred())\n goto bad;\n#endif\n if (!strict && (size_t)basicsize > size) {\n PyOS_snprintf(warning, sizeof(warning),\n \"%s.%s size changed, may indicate binary incompatibility. Expected %zd, got %zd\",\n module_name, class_name, basicsize, size);\n if (PyErr_WarnEx(NULL, warning, 0) < 0) goto bad;\n }\n else if ((size_t)basicsize != size) {\n PyErr_Format(PyExc_ValueError,\n \"%.200s.%.200s has the wrong size, try recompiling. Expected %zd, got %zd\",\n module_name, class_name, basicsize, size);\n goto bad;\n }\n return (PyTypeObject *)result;\nbad:\n Py_XDECREF(py_module);\n Py_XDECREF(result);\n return NULL;\n}\n#endif\n\n/* InitStrings */\n static int __Pyx_InitStrings(__Pyx_StringTabEntry *t) {\n while (t->p) {\n #if PY_MAJOR_VERSION < 3\n if (t->is_unicode) {\n *t->p = PyUnicode_DecodeUTF8(t->s, t->n - 1, NULL);\n } else if (t->intern) {\n *t->p = PyString_InternFromString(t->s);\n } else {\n *t->p = PyString_FromStringAndSize(t->s, t->n - 1);\n }\n #else\n if (t->is_unicode | t->is_str) {\n if (t->intern) {\n *t->p = PyUnicode_InternFromString(t->s);\n } else if (t->encoding) {\n *t->p = PyUnicode_Decode(t->s, t->n - 1, t->encoding, NULL);\n } else {\n *t->p = PyUnicode_FromStringAndSize(t->s, t->n - 1);\n }\n } else {\n *t->p = PyBytes_FromStringAndSize(t->s, t->n - 1);\n }\n #endif\n if (!*t->p)\n return -1;\n ++t;\n }\n return 0;\n}\n\nstatic CYTHON_INLINE PyObject* __Pyx_PyUnicode_FromString(const char* c_str) {\n return __Pyx_PyUnicode_FromStringAndSize(c_str, (Py_ssize_t)strlen(c_str));\n}\nstatic CYTHON_INLINE char* __Pyx_PyObject_AsString(PyObject* o) {\n Py_ssize_t ignore;\n return __Pyx_PyObject_AsStringAndSize(o, &ignore);\n}\nstatic CYTHON_INLINE char* __Pyx_PyObject_AsStringAndSize(PyObject* o, Py_ssize_t *length) {\n#if CYTHON_COMPILING_IN_CPYTHON && (__PYX_DEFAULT_STRING_ENCODING_IS_ASCII || __PYX_DEFAULT_STRING_ENCODING_IS_DEFAULT)\n if (\n#if PY_MAJOR_VERSION < 3 && __PYX_DEFAULT_STRING_ENCODING_IS_ASCII\n __Pyx_sys_getdefaultencoding_not_ascii &&\n#endif\n PyUnicode_Check(o)) {\n#if PY_VERSION_HEX < 0x03030000\n char* defenc_c;\n PyObject* defenc = _PyUnicode_AsDefaultEncodedString(o, NULL);\n if (!defenc) return NULL;\n defenc_c = PyBytes_AS_STRING(defenc);\n#if __PYX_DEFAULT_STRING_ENCODING_IS_ASCII\n {\n char* end = defenc_c + PyBytes_GET_SIZE(defenc);\n char* c;\n for (c = defenc_c; c < end; c++) {\n if ((unsigned char) (*c) >= 128) {\n PyUnicode_AsASCIIString(o);\n return NULL;\n }\n }\n }\n#endif\n *length = PyBytes_GET_SIZE(defenc);\n return defenc_c;\n#else\n if (__Pyx_PyUnicode_READY(o) == -1) return NULL;\n#if __PYX_DEFAULT_STRING_ENCODING_IS_ASCII\n if (PyUnicode_IS_ASCII(o)) {\n *length = PyUnicode_GET_LENGTH(o);\n return PyUnicode_AsUTF8(o);\n } else {\n PyUnicode_AsASCIIString(o);\n return NULL;\n }\n#else\n return PyUnicode_AsUTF8AndSize(o, length);\n#endif\n#endif\n } else\n#endif\n#if (!CYTHON_COMPILING_IN_PYPY) || (defined(PyByteArray_AS_STRING) && defined(PyByteArray_GET_SIZE))\n if (PyByteArray_Check(o)) {\n *length = PyByteArray_GET_SIZE(o);\n return PyByteArray_AS_STRING(o);\n } else\n#endif\n {\n char* result;\n int r = PyBytes_AsStringAndSize(o, &result, length);\n if (unlikely(r < 0)) {\n return NULL;\n } else {\n return result;\n }\n }\n}\nstatic CYTHON_INLINE int __Pyx_PyObject_IsTrue(PyObject* x) {\n int is_true = x == Py_True;\n if (is_true | (x == Py_False) | (x == Py_None)) return is_true;\n else return PyObject_IsTrue(x);\n}\nstatic CYTHON_INLINE PyObject* __Pyx_PyNumber_IntOrLong(PyObject* x) {\n PyNumberMethods *m;\n const char *name = NULL;\n PyObject *res = NULL;\n#if PY_MAJOR_VERSION < 3\n if (PyInt_Check(x) || PyLong_Check(x))\n#else\n if (PyLong_Check(x))\n#endif\n return __Pyx_NewRef(x);\n m = Py_TYPE(x)->tp_as_number;\n#if PY_MAJOR_VERSION < 3\n if (m && m->nb_int) {\n name = \"int\";\n res = PyNumber_Int(x);\n }\n else if (m && m->nb_long) {\n name = \"long\";\n res = PyNumber_Long(x);\n }\n#else\n if (m && m->nb_int) {\n name = \"int\";\n res = PyNumber_Long(x);\n }\n#endif\n if (res) {\n#if PY_MAJOR_VERSION < 3\n if (!PyInt_Check(res) && !PyLong_Check(res)) {\n#else\n if (!PyLong_Check(res)) {\n#endif\n PyErr_Format(PyExc_TypeError,\n \"__%.4s__ returned non-%.4s (type %.200s)\",\n name, name, Py_TYPE(res)->tp_name);\n Py_DECREF(res);\n return NULL;\n }\n }\n else if (!PyErr_Occurred()) {\n PyErr_SetString(PyExc_TypeError,\n \"an integer is required\");\n }\n return res;\n}\nstatic CYTHON_INLINE Py_ssize_t __Pyx_PyIndex_AsSsize_t(PyObject* b) {\n Py_ssize_t ival;\n PyObject *x;\n#if PY_MAJOR_VERSION < 3\n if (likely(PyInt_CheckExact(b))) {\n if (sizeof(Py_ssize_t) >= sizeof(long))\n return PyInt_AS_LONG(b);\n else\n return PyInt_AsSsize_t(x);\n }\n#endif\n if (likely(PyLong_CheckExact(b))) {\n #if CYTHON_USE_PYLONG_INTERNALS\n const digit* digits = ((PyLongObject*)b)->ob_digit;\n const Py_ssize_t size = Py_SIZE(b);\n if (likely(__Pyx_sst_abs(size) <= 1)) {\n ival = likely(size) ? digits[0] : 0;\n if (size == -1) ival = -ival;\n return ival;\n } else {\n switch (size) {\n case 2:\n if (8 * sizeof(Py_ssize_t) > 2 * PyLong_SHIFT) {\n return (Py_ssize_t) (((((size_t)digits[1]) << PyLong_SHIFT) | (size_t)digits[0]));\n }\n break;\n case -2:\n if (8 * sizeof(Py_ssize_t) > 2 * PyLong_SHIFT) {\n return -(Py_ssize_t) (((((size_t)digits[1]) << PyLong_SHIFT) | (size_t)digits[0]));\n }\n break;\n case 3:\n if (8 * sizeof(Py_ssize_t) > 3 * PyLong_SHIFT) {\n return (Py_ssize_t) (((((((size_t)digits[2]) << PyLong_SHIFT) | (size_t)digits[1]) << PyLong_SHIFT) | (size_t)digits[0]));\n }\n break;\n case -3:\n if (8 * sizeof(Py_ssize_t) > 3 * PyLong_SHIFT) {\n return -(Py_ssize_t) (((((((size_t)digits[2]) << PyLong_SHIFT) | (size_t)digits[1]) << PyLong_SHIFT) | (size_t)digits[0]));\n }\n break;\n case 4:\n if (8 * sizeof(Py_ssize_t) > 4 * PyLong_SHIFT) {\n return (Py_ssize_t) (((((((((size_t)digits[3]) << PyLong_SHIFT) | (size_t)digits[2]) << PyLong_SHIFT) | (size_t)digits[1]) << PyLong_SHIFT) | (size_t)digits[0]));\n }\n break;\n case -4:\n if (8 * sizeof(Py_ssize_t) > 4 * PyLong_SHIFT) {\n return -(Py_ssize_t) (((((((((size_t)digits[3]) << PyLong_SHIFT) | (size_t)digits[2]) << PyLong_SHIFT) | (size_t)digits[1]) << PyLong_SHIFT) | (size_t)digits[0]));\n }\n break;\n }\n }\n #endif\n return PyLong_AsSsize_t(b);\n }\n x = PyNumber_Index(b);\n if (!x) return -1;\n ival = PyInt_AsSsize_t(x);\n Py_DECREF(x);\n return ival;\n}\nstatic CYTHON_INLINE PyObject * __Pyx_PyInt_FromSize_t(size_t ival) {\n return PyInt_FromSize_t(ival);\n}\n\n\n#endif /* Py_PYTHON_H */\n\n" }, { "path": "lib/nms/gpu_nms.hpp", "content": "// ------------------------------------------------------------------\n// Deep Feature Flow\n// Copyright (c) 2017 Microsoft\n// Licensed under The MIT License\n// Modified from MATLAB Faster R-CNN (https://github.com/shaoqingren/faster_rcnn)\n// ------------------------------------------------------------------\n// Based on:\n// Faster R-CNN\n// Copyright (c) 2015 Microsoft\n// Licensed under The MIT License\n// https://github.com/shaoqingren/faster_rcnn\n// ------------------------------------------------------------------\n\nvoid _nms(int* keep_out, int* num_out, const float* boxes_host, int boxes_num,\n int boxes_dim, float nms_overlap_thresh, int device_id);\n" }, { "path": "lib/nms/gpu_nms.pyx", "content": "# ------------------------------------------------------------------\n# Deep Feature Flow\n# Copyright (c) 2015 Microsoft\n# Licensed under The MIT License\n# Written by Yuwen Xiong, Tao Kong\n# ------------------------------------------------------------------\n# Based on:\n# Faster R-CNN\n# Copyright (c) 2015 Microsoft\n# Licensed under The MIT License\n# https://github.com/shaoqingren/faster_rcnn\n# ------------------------------------------------------------------\n\nimport numpy as np\ncimport numpy as np\n\nassert sizeof(int) == sizeof(np.int32_t)\n\ncdef extern from \"gpu_nms.hpp\":\n void _nms(np.int32_t*, int*, np.float32_t*, int, int, float, int)\n\ndef gpu_nms(np.ndarray[np.float32_t, ndim=2] dets, np.float thresh,\n np.int32_t device_id=0):\n cdef int boxes_num = dets.shape[0]\n cdef int boxes_dim = dets.shape[1]\n cdef int num_out\n cdef np.ndarray[np.int32_t, ndim=1] \\\n keep = np.zeros(boxes_num, dtype=np.int32)\n cdef np.ndarray[np.float32_t, ndim=1] \\\n scores = dets[:, 4]\n cdef np.ndarray[np.int32_t, ndim=1] \\\n order = scores.argsort()[::-1].astype(np.int32)\n cdef np.ndarray[np.float32_t, ndim=2] \\\n sorted_dets = dets[order, :]\n _nms(&keep[0], &num_out, &sorted_dets[0, 0], boxes_num, boxes_dim, thresh, device_id)\n keep = keep[:num_out]\n return list(order[keep])\n" }, { "path": "lib/nms/nms.py", "content": "# --------------------------------------------------------\n# Deep Feature Flow\n# Copyright (c) 2017 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Written by Yuwen Xiong\n# --------------------------------------------------------\n# Based on:\n# py-faster-rcnn\n# Copyright (c) 2015 Microsoft\n# Licence under The MIT License\n# https://github.com/rbgirshick/py-faster-rcnn\n# --------------------------------------------------------\n\nimport numpy as np\n\nfrom cpu_nms import cpu_nms\nfrom gpu_nms import gpu_nms\n\ndef py_nms_wrapper(thresh):\n def _nms(dets):\n return nms(dets, thresh)\n return _nms\n\n\ndef cpu_nms_wrapper(thresh):\n def _nms(dets):\n return cpu_nms(dets, thresh)\n return _nms\n\n\ndef gpu_nms_wrapper(thresh, device_id):\n def _nms(dets):\n return gpu_nms(dets, thresh, device_id)\n return _nms\n\n\ndef nms(dets, thresh):\n \"\"\"\n greedily select boxes with high confidence and overlap with current maximum <= thresh\n rule out overlap >= thresh\n :param dets: [[x1, y1, x2, y2 score]]\n :param thresh: retain overlap < thresh\n :return: indexes to keep\n \"\"\"\n if dets.shape[0] == 0:\n return []\n\n x1 = dets[:, 0]\n y1 = dets[:, 1]\n x2 = dets[:, 2]\n y2 = dets[:, 3]\n scores = dets[:, 4]\n\n areas = (x2 - x1 + 1) * (y2 - y1 + 1)\n order = scores.argsort()[::-1]\n\n keep = []\n while order.size > 0:\n i = order[0]\n keep.append(i)\n xx1 = np.maximum(x1[i], x1[order[1:]])\n yy1 = np.maximum(y1[i], y1[order[1:]])\n xx2 = np.minimum(x2[i], x2[order[1:]])\n yy2 = np.minimum(y2[i], y2[order[1:]])\n\n w = np.maximum(0.0, xx2 - xx1 + 1)\n h = np.maximum(0.0, yy2 - yy1 + 1)\n inter = w * h\n ovr = inter / (areas[i] + areas[order[1:]] - inter)\n\n inds = np.where(ovr <= thresh)[0]\n order = order[inds + 1]\n\n return keep\n" }, { "path": "lib/nms/nms_kernel.cu", "content": "// ------------------------------------------------------------------\n// Deep Feature Flow\n// Copyright (c) 2015 Microsoft\n// Licensed under The MIT License\n// Written by Yuwen Xiong\n// ------------------------------------------------------------------\n// Based on:\n// Faster R-CNN\n// Copyright (c) 2015 Microsoft\n// Licensed under The MIT License\n// https://github.com/shaoqingren/faster_rcnn\n// ------------------------------------------------------------------\n\n#include \"gpu_nms.hpp\"\n#include \n#include \n\n#define CUDA_CHECK(condition) \\\n /* Code block avoids redefinition of cudaError_t error */ \\\n do { \\\n cudaError_t error = condition; \\\n if (error != cudaSuccess) { \\\n std::cout << cudaGetErrorString(error) << std::endl; \\\n } \\\n } while (0)\n\n#define DIVUP(m,n) ((m) / (n) + ((m) % (n) > 0))\nint const threadsPerBlock = sizeof(unsigned long long) * 8;\n\n__device__ inline float devIoU(float const * const a, float const * const b) {\n float left = max(a[0], b[0]), right = min(a[2], b[2]);\n float top = max(a[1], b[1]), bottom = min(a[3], b[3]);\n float width = max(right - left + 1, 0.f), height = max(bottom - top + 1, 0.f);\n float interS = width * height;\n float Sa = (a[2] - a[0] + 1) * (a[3] - a[1] + 1);\n float Sb = (b[2] - b[0] + 1) * (b[3] - b[1] + 1);\n return interS / (Sa + Sb - interS);\n}\n\n__global__ void nms_kernel(const int n_boxes, const float nms_overlap_thresh,\n const float *dev_boxes, unsigned long long *dev_mask) {\n const int row_start = blockIdx.y;\n const int col_start = blockIdx.x;\n\n // if (row_start > col_start) return;\n\n const int row_size =\n min(n_boxes - row_start * threadsPerBlock, threadsPerBlock);\n const int col_size =\n min(n_boxes - col_start * threadsPerBlock, threadsPerBlock);\n\n __shared__ float block_boxes[threadsPerBlock * 5];\n if (threadIdx.x < col_size) {\n block_boxes[threadIdx.x * 5 + 0] =\n dev_boxes[(threadsPerBlock * col_start + threadIdx.x) * 5 + 0];\n block_boxes[threadIdx.x * 5 + 1] =\n dev_boxes[(threadsPerBlock * col_start + threadIdx.x) * 5 + 1];\n block_boxes[threadIdx.x * 5 + 2] =\n dev_boxes[(threadsPerBlock * col_start + threadIdx.x) * 5 + 2];\n block_boxes[threadIdx.x * 5 + 3] =\n dev_boxes[(threadsPerBlock * col_start + threadIdx.x) * 5 + 3];\n block_boxes[threadIdx.x * 5 + 4] =\n dev_boxes[(threadsPerBlock * col_start + threadIdx.x) * 5 + 4];\n }\n __syncthreads();\n\n if (threadIdx.x < row_size) {\n const int cur_box_idx = threadsPerBlock * row_start + threadIdx.x;\n const float *cur_box = dev_boxes + cur_box_idx * 5;\n int i = 0;\n unsigned long long t = 0;\n int start = 0;\n if (row_start == col_start) {\n start = threadIdx.x + 1;\n }\n for (i = start; i < col_size; i++) {\n if (devIoU(cur_box, block_boxes + i * 5) > nms_overlap_thresh) {\n t |= 1ULL << i;\n }\n }\n const int col_blocks = DIVUP(n_boxes, threadsPerBlock);\n dev_mask[cur_box_idx * col_blocks + col_start] = t;\n }\n}\n\nvoid _set_device(int device_id) {\n int current_device;\n CUDA_CHECK(cudaGetDevice(¤t_device));\n if (current_device == device_id) {\n return;\n }\n // The call to cudaSetDevice must come before any calls to Get, which\n // may perform initialization using the GPU.\n CUDA_CHECK(cudaSetDevice(device_id));\n}\n\nvoid _nms(int* keep_out, int* num_out, const float* boxes_host, int boxes_num,\n int boxes_dim, float nms_overlap_thresh, int device_id) {\n _set_device(device_id);\n\n float* boxes_dev = NULL;\n unsigned long long* mask_dev = NULL;\n\n const int col_blocks = DIVUP(boxes_num, threadsPerBlock);\n\n CUDA_CHECK(cudaMalloc(&boxes_dev,\n boxes_num * boxes_dim * sizeof(float)));\n CUDA_CHECK(cudaMemcpy(boxes_dev,\n boxes_host,\n boxes_num * boxes_dim * sizeof(float),\n cudaMemcpyHostToDevice));\n\n CUDA_CHECK(cudaMalloc(&mask_dev,\n boxes_num * col_blocks * sizeof(unsigned long long)));\n\n dim3 blocks(DIVUP(boxes_num, threadsPerBlock),\n DIVUP(boxes_num, threadsPerBlock));\n dim3 threads(threadsPerBlock);\n nms_kernel<<>>(boxes_num,\n nms_overlap_thresh,\n boxes_dev,\n mask_dev);\n\n std::vector mask_host(boxes_num * col_blocks);\n CUDA_CHECK(cudaMemcpy(&mask_host[0],\n mask_dev,\n sizeof(unsigned long long) * boxes_num * col_blocks,\n cudaMemcpyDeviceToHost));\n\n std::vector remv(col_blocks);\n memset(&remv[0], 0, sizeof(unsigned long long) * col_blocks);\n\n int num_to_keep = 0;\n for (int i = 0; i < boxes_num; i++) {\n int nblock = i / threadsPerBlock;\n int inblock = i % threadsPerBlock;\n\n if (!(remv[nblock] & (1ULL << inblock))) {\n keep_out[num_to_keep++] = i;\n unsigned long long *p = &mask_host[0] + i * col_blocks;\n for (int j = nblock; j < col_blocks; j++) {\n remv[j] |= p[j];\n }\n }\n }\n *num_out = num_to_keep;\n\n CUDA_CHECK(cudaFree(boxes_dev));\n CUDA_CHECK(cudaFree(mask_dev));\n}\n" }, { "path": "lib/nms/setup_linux.py", "content": "# --------------------------------------------------------\n# Deep Feature Flow\n# Copyright (c) 2017 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Written by Yuwen Xiong\n# --------------------------------------------------------\n# Based on:\n# py-faster-rcnn\n# Copyright (c) 2015 Microsoft\n# Licence under The MIT License\n# https://github.com/rbgirshick/py-faster-rcnn\n# --------------------------------------------------------\n\nimport os\nfrom os.path import join as pjoin\nfrom setuptools import setup\nfrom distutils.extension import Extension\nfrom Cython.Distutils import build_ext\nimport numpy as np\n\n\ndef find_in_path(name, path):\n \"Find a file in a search path\"\n # Adapted fom\n # http://code.activestate.com/recipes/52224-find-a-file-given-a-search-path/\n for dir in path.split(os.pathsep):\n binpath = pjoin(dir, name)\n if os.path.exists(binpath):\n return os.path.abspath(binpath)\n return None\n\n\ndef locate_cuda():\n \"\"\"Locate the CUDA environment on the system\n Returns a dict with keys 'home', 'nvcc', 'include', and 'lib64'\n and values giving the absolute path to each directory.\n Starts by looking for the CUDAHOME env variable. If not found, everything\n is based on finding 'nvcc' in the PATH.\n \"\"\"\n\n # first check if the CUDAHOME env variable is in use\n if 'CUDAHOME' in os.environ:\n home = os.environ['CUDAHOME']\n nvcc = pjoin(home, 'bin', 'nvcc')\n else:\n # otherwise, search the PATH for NVCC\n default_path = pjoin(os.sep, 'usr', 'local', 'cuda', 'bin')\n nvcc = find_in_path('nvcc', os.environ['PATH'] + os.pathsep + default_path)\n if nvcc is None:\n raise EnvironmentError('The nvcc binary could not be '\n 'located in your $PATH. Either add it to your path, or set $CUDAHOME')\n home = os.path.dirname(os.path.dirname(nvcc))\n\n cudaconfig = {'home':home, 'nvcc':nvcc,\n 'include': pjoin(home, 'include'),\n 'lib64': pjoin(home, 'lib64')}\n for k, v in cudaconfig.iteritems():\n if not os.path.exists(v):\n raise EnvironmentError('The CUDA %s path could not be located in %s' % (k, v))\n\n return cudaconfig\nCUDA = locate_cuda()\n\n\n# Obtain the numpy include directory. This logic works across numpy versions.\ntry:\n numpy_include = np.get_include()\nexcept AttributeError:\n numpy_include = np.get_numpy_include()\n\n\ndef customize_compiler_for_nvcc(self):\n \"\"\"inject deep into distutils to customize how the dispatch\n to gcc/nvcc works.\n If you subclass UnixCCompiler, it's not trivial to get your subclass\n injected in, and still have the right customizations (i.e.\n distutils.sysconfig.customize_compiler) run on it. So instead of going\n the OO route, I have this. Note, it's kindof like a wierd functional\n subclassing going on.\"\"\"\n\n # tell the compiler it can processes .cu\n self.src_extensions.append('.cu')\n\n # save references to the default compiler_so and _comple methods\n default_compiler_so = self.compiler_so\n super = self._compile\n\n # now redefine the _compile method. This gets executed for each\n # object but distutils doesn't have the ability to change compilers\n # based on source extension: we add it.\n def _compile(obj, src, ext, cc_args, extra_postargs, pp_opts):\n if os.path.splitext(src)[1] == '.cu':\n # use the cuda for .cu files\n self.set_executable('compiler_so', CUDA['nvcc'])\n # use only a subset of the extra_postargs, which are 1-1 translated\n # from the extra_compile_args in the Extension class\n postargs = extra_postargs['nvcc']\n else:\n postargs = extra_postargs['gcc']\n\n super(obj, src, ext, cc_args, postargs, pp_opts)\n # reset the default compiler_so, which we might have changed for cuda\n self.compiler_so = default_compiler_so\n\n # inject our redefined _compile method into the class\n self._compile = _compile\n\n\n# run the customize_compiler\nclass custom_build_ext(build_ext):\n def build_extensions(self):\n customize_compiler_for_nvcc(self.compiler)\n build_ext.build_extensions(self)\n\n\next_modules = [\n Extension(\n \"cpu_nms\",\n [\"cpu_nms.pyx\"],\n extra_compile_args={'gcc': [\"-Wno-cpp\", \"-Wno-unused-function\"]},\n include_dirs = [numpy_include]\n ),\n Extension('gpu_nms',\n ['nms_kernel.cu', 'gpu_nms.pyx'],\n library_dirs=[CUDA['lib64']],\n libraries=['cudart'],\n language='c++',\n runtime_library_dirs=[CUDA['lib64']],\n # this syntax is specific to this build system\n # we're only going to use certain compiler args with nvcc and not with\n # gcc the implementation of this trick is in customize_compiler() below\n extra_compile_args={'gcc': [\"-Wno-unused-function\"],\n 'nvcc': ['-arch=sm_35',\n '--ptxas-options=-v',\n '-c',\n '--compiler-options',\n \"'-fPIC'\"]},\n include_dirs = [numpy_include, CUDA['include']]\n ),\n]\n\nsetup(\n name='nms',\n ext_modules=ext_modules,\n # inject our custom trigger\n cmdclass={'build_ext': custom_build_ext},\n)\n" }, { "path": "lib/nms/setup_windows.py", "content": "# --------------------------------------------------------\n# Deep Feature Flow\n# Copyright (c) 2017 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Written by Yuwen Xiong\n# --------------------------------------------------------\n# Based on:\n# py-faster-rcnn\n# Copyright (c) 2015 Microsoft\n# Licence under The MIT License\n# https://github.com/rbgirshick/py-faster-rcnn\n# --------------------------------------------------------\n\nimport numpy as np\nimport os\nfrom os.path import join as pjoin\n#from distutils.core import setup\nfrom setuptools import setup\nfrom distutils.extension import Extension\nfrom Cython.Distutils import build_ext\nimport subprocess\n\n#change for windows, by MrX\nnvcc_bin = 'nvcc.exe'\nlib_dir = 'lib/x64'\n\nimport distutils.msvc9compiler\ndistutils.msvc9compiler.VERSION = 14.0\n\n\ndef find_in_path(name, path):\n \"Find a file in a search path\"\n # Adapted fom\n # http://code.activestate.com/recipes/52224-find-a-file-given-a-search-path/\n for dir in path.split(os.pathsep):\n binpath = pjoin(dir, name)\n if os.path.exists(binpath):\n return os.path.abspath(binpath)\n return None\n\n\ndef locate_cuda():\n \"\"\"Locate the CUDA environment on the system\n\n Returns a dict with keys 'home', 'nvcc', 'include', and 'lib64'\n and values giving the absolute path to each directory.\n\n Starts by looking for the CUDAHOME env variable. If not found, everything\n is based on finding 'nvcc' in the PATH.\n \"\"\"\n\n # first check if the CUDAHOME env variable is in use\n if 'CUDA_PATH' in os.environ:\n home = os.environ['CUDA_PATH']\n print(\"home = %s\\n\" % home)\n nvcc = pjoin(home, 'bin', nvcc_bin)\n else:\n # otherwise, search the PATH for NVCC\n default_path = pjoin(os.sep, 'usr', 'local', 'cuda', 'bin')\n nvcc = find_in_path(nvcc_bin, os.environ['PATH'] + os.pathsep + default_path)\n if nvcc is None:\n raise EnvironmentError('The nvcc binary could not be '\n 'located in your $PATH. Either add it to your path, or set $CUDA_PATH')\n home = os.path.dirname(os.path.dirname(nvcc))\n print(\"home = %s, nvcc = %s\\n\" % (home, nvcc))\n\n\n cudaconfig = {'home':home, 'nvcc':nvcc,\n 'include': pjoin(home, 'include'),\n 'lib64': pjoin(home, lib_dir)}\n for k, v in cudaconfig.iteritems():\n if not os.path.exists(v):\n raise EnvironmentError('The CUDA %s path could not be located in %s' % (k, v))\n\n return cudaconfig\nCUDA = locate_cuda()\n\n\n# Obtain the numpy include directory. This logic works across numpy versions.\ntry:\n numpy_include = np.get_include()\nexcept AttributeError:\n numpy_include = np.get_numpy_include()\n\n\ndef customize_compiler_for_nvcc(self):\n \"\"\"inject deep into distutils to customize how the dispatch\n to gcc/nvcc works.\n\n If you subclass UnixCCompiler, it's not trivial to get your subclass\n injected in, and still have the right customizations (i.e.\n distutils.sysconfig.customize_compiler) run on it. So instead of going\n the OO route, I have this. Note, it's kindof like a wierd functional\n subclassing going on.\"\"\"\n\n # tell the compiler it can processes .cu\n #self.src_extensions.append('.cu')\n\n\t\n # save references to the default compiler_so and _comple methods\n #default_compiler_so = self.spawn \n #default_compiler_so = self.rc\n super = self.compile\n\n # now redefine the _compile method. This gets executed for each\n # object but distutils doesn't have the ability to change compilers\n # based on source extension: we add it.\n def compile(sources, output_dir=None, macros=None, include_dirs=None, debug=0, extra_preargs=None, extra_postargs=None, depends=None):\n postfix=os.path.splitext(sources[0])[1]\n \n if postfix == '.cu':\n # use the cuda for .cu files\n #self.set_executable('compiler_so', CUDA['nvcc'])\n # use only a subset of the extra_postargs, which are 1-1 translated\n # from the extra_compile_args in the Extension class\n postargs = extra_postargs['nvcc']\n else:\n postargs = extra_postargs['gcc']\n\n\n return super(sources, output_dir, macros, include_dirs, debug, extra_preargs, postargs, depends)\n # reset the default compiler_so, which we might have changed for cuda\n #self.rc = default_compiler_so\n\n # inject our redefined _compile method into the class\n self.compile = compile\n\n\n# run the customize_compiler\nclass custom_build_ext(build_ext):\n def build_extensions(self):\n customize_compiler_for_nvcc(self.compiler)\n build_ext.build_extensions(self)\n\n\next_modules = [\n # unix _compile: obj, src, ext, cc_args, extra_postargs, pp_opts\n Extension(\n \"cpu_nms\",\n sources=[\"cpu_nms.pyx\"],\n extra_compile_args={'gcc': []},\n include_dirs = [numpy_include],\n ),\n]\n\nsetup(\n name='fast_rcnn',\n ext_modules=ext_modules,\n # inject our custom trigger\n cmdclass={'build_ext': custom_build_ext},\n)\n" }, { "path": "lib/nms/setup_windows_cuda.py", "content": "# --------------------------------------------------------\n# Deep Feature Flow\n# Copyright (c) 2017 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Written by Yuwen Xiong\n# --------------------------------------------------------\n\n#!/usr/bin/env python\n\nimport numpy as np\nimport os\n# on Windows, we need the original PATH without Anaconda's compiler in it:\nPATH = os.environ.get('PATH') + ';C:\\\\Program Files (x86)\\\\Microsoft Visual Studio 14.0\\\\VC\\\\bin'\nfrom distutils.spawn import spawn, find_executable\nfrom setuptools import setup, find_packages, Extension\nfrom setuptools.command.build_ext import build_ext\nimport sys\n\n# CUDA specific config\n# nvcc is assumed to be in user's PATH\nnvcc_compile_args = ['-O', '--ptxas-options=-v', '-arch=compute_35', '-code=sm_35,sm_52,sm_61', '-c', '--compiler-options=-fPIC']\nnvcc_compile_args = os.environ.get('NVCCFLAGS', '').split() + nvcc_compile_args\ncuda_libs = ['cublas']\nnvcc_bin = 'nvcc.exe'\nlib_dir = 'lib/x64'\n\n\nimport distutils.msvc9compiler\ndistutils.msvc9compiler.VERSION = 14.0\n\n# Obtain the numpy include directory. This logic works across numpy versions.\ntry:\n numpy_include = np.get_include()\nexcept AttributeError:\n numpy_include = np.get_numpy_include()\n\n\ncudamat_ext = Extension('gpu_nms',\n sources=[\n 'gpu_nms.cu'\n ],\n language='c++',\n libraries=cuda_libs,\n extra_compile_args=nvcc_compile_args,\n include_dirs = [numpy_include, 'C:\\\\Programming\\\\CUDA\\\\v8.0\\\\include'])\n\n\nclass CUDA_build_ext(build_ext):\n \"\"\"\n Custom build_ext command that compiles CUDA files.\n Note that all extension source files will be processed with this compiler.\n \"\"\"\n def build_extensions(self):\n self.compiler.src_extensions.append('.cu')\n self.compiler.set_executable('compiler_so', 'nvcc')\n self.compiler.set_executable('linker_so', 'nvcc --shared')\n if hasattr(self.compiler, '_c_extensions'):\n self.compiler._c_extensions.append('.cu') # needed for Windows\n self.compiler.spawn = self.spawn\n build_ext.build_extensions(self)\n\n def spawn(self, cmd, search_path=1, verbose=0, dry_run=0):\n \"\"\"\n Perform any CUDA specific customizations before actually launching\n compile/link etc. commands.\n \"\"\"\n if (sys.platform == 'darwin' and len(cmd) >= 2 and cmd[0] == 'nvcc' and\n cmd[1] == '--shared' and cmd.count('-arch') > 0):\n # Versions of distutils on OSX earlier than 2.7.9 inject\n # '-arch x86_64' which we need to strip while using nvcc for\n # linking\n while True:\n try:\n index = cmd.index('-arch')\n del cmd[index:index+2]\n except ValueError:\n break\n elif self.compiler.compiler_type == 'msvc':\n # There are several things we need to do to change the commands\n # issued by MSVCCompiler into one that works with nvcc. In the end,\n # it might have been easier to write our own CCompiler class for\n # nvcc, as we're only interested in creating a shared library to\n # load with ctypes, not in creating an importable Python extension.\n # - First, we replace the cl.exe or link.exe call with an nvcc\n # call. In case we're running Anaconda, we search cl.exe in the\n # original search path we captured further above -- Anaconda\n # inserts a MSVC version into PATH that is too old for nvcc.\n cmd[:1] = ['nvcc', '--compiler-bindir',\n os.path.dirname(find_executable(\"cl.exe\", PATH))\n or cmd[0]]\n # - Secondly, we fix a bunch of command line arguments.\n for idx, c in enumerate(cmd):\n # create .dll instead of .pyd files\n #if '.pyd' in c: cmd[idx] = c = c.replace('.pyd', '.dll') #20160601, by MrX\n # replace /c by -c\n if c == '/c': cmd[idx] = '-c'\n # replace /DLL by --shared\n elif c == '/DLL': cmd[idx] = '--shared'\n # remove --compiler-options=-fPIC\n elif '-fPIC' in c: del cmd[idx]\n # replace /Tc... by ...\n elif c.startswith('/Tc'): cmd[idx] = c[3:]\n # replace /Fo... by -o ...\n elif c.startswith('/Fo'): cmd[idx:idx+1] = ['-o', c[3:]]\n # replace /LIBPATH:... by -L...\n elif c.startswith('/LIBPATH:'): cmd[idx] = '-L' + c[9:]\n # replace /OUT:... by -o ...\n elif c.startswith('/OUT:'): cmd[idx:idx+1] = ['-o', c[5:]]\n # remove /EXPORT:initlibcudamat or /EXPORT:initlibcudalearn\n elif c.startswith('/EXPORT:'): del cmd[idx]\n # replace cublas.lib by -lcublas\n elif c == 'cublas.lib': cmd[idx] = '-lcublas'\n # - Finally, we pass on all arguments starting with a '/' to the\n # compiler or linker, and have nvcc handle all other arguments\n if '--shared' in cmd:\n pass_on = '--linker-options='\n # we only need MSVCRT for a .dll, remove CMT if it sneaks in:\n cmd.append('/NODEFAULTLIB:libcmt.lib')\n else:\n pass_on = '--compiler-options='\n cmd = ([c for c in cmd if c[0] != '/'] +\n [pass_on + ','.join(c for c in cmd if c[0] == '/')])\n # For the future: Apart from the wrongly set PATH by Anaconda, it\n # would suffice to run the following for compilation on Windows:\n # nvcc -c -O -o .obj .cu\n # And the following for linking:\n # nvcc --shared -o .dll .obj .obj -lcublas\n # This could be done by a NVCCCompiler class for all platforms.\n spawn(cmd, search_path, verbose, dry_run)\n\nsetup(name=\"py_fast_rcnn_gpu\",\n description=\"Performs linear algebra computation on the GPU via CUDA\",\n ext_modules=[cudamat_ext],\n cmdclass={'build_ext': CUDA_build_ext},\n)\n" }, { "path": "lib/rpn/__init__.py", "content": "" }, { "path": "lib/rpn/generate_anchor.py", "content": "# --------------------------------------------------------\n# Deep Feature Flow\n# Copyright (c) 2017 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Modified by Yuwen Xiong\n# --------------------------------------------------------\n# Based on:\n# MX-RCNN\n# Copyright (c) 2016 by Contributors\n# Licence under The Apache 2.0 License\n# https://github.com/ijkguo/mx-rcnn/\n# --------------------------------------------------------\n\n\"\"\"\nGenerate base anchors on index 0\n\"\"\"\n\nimport numpy as np\n\n\ndef generate_anchors(base_size=16, ratios=[0.5, 1, 2],\n scales=2 ** np.arange(3, 6)):\n \"\"\"\n Generate anchor (reference) windows by enumerating aspect ratios X\n scales wrt a reference (0, 0, 15, 15) window.\n \"\"\"\n\n base_anchor = np.array([1, 1, base_size, base_size]) - 1\n ratio_anchors = _ratio_enum(base_anchor, ratios)\n anchors = np.vstack([_scale_enum(ratio_anchors[i, :], scales)\n for i in xrange(ratio_anchors.shape[0])])\n return anchors\n\n\ndef _whctrs(anchor):\n \"\"\"\n Return width, height, x center, and y center for an anchor (window).\n \"\"\"\n\n w = anchor[2] - anchor[0] + 1\n h = anchor[3] - anchor[1] + 1\n x_ctr = anchor[0] + 0.5 * (w - 1)\n y_ctr = anchor[1] + 0.5 * (h - 1)\n return w, h, x_ctr, y_ctr\n\n\ndef _mkanchors(ws, hs, x_ctr, y_ctr):\n \"\"\"\n Given a vector of widths (ws) and heights (hs) around a center\n (x_ctr, y_ctr), output a set of anchors (windows).\n \"\"\"\n\n ws = ws[:, np.newaxis]\n hs = hs[:, np.newaxis]\n anchors = np.hstack((x_ctr - 0.5 * (ws - 1),\n y_ctr - 0.5 * (hs - 1),\n x_ctr + 0.5 * (ws - 1),\n y_ctr + 0.5 * (hs - 1)))\n return anchors\n\n\ndef _ratio_enum(anchor, ratios):\n \"\"\"\n Enumerate a set of anchors for each aspect ratio wrt an anchor.\n \"\"\"\n\n w, h, x_ctr, y_ctr = _whctrs(anchor)\n size = w * h\n size_ratios = size / ratios\n ws = np.round(np.sqrt(size_ratios))\n hs = np.round(ws * ratios)\n anchors = _mkanchors(ws, hs, x_ctr, y_ctr)\n return anchors\n\n\ndef _scale_enum(anchor, scales):\n \"\"\"\n Enumerate a set of anchors for each scale wrt an anchor.\n \"\"\"\n\n w, h, x_ctr, y_ctr = _whctrs(anchor)\n ws = w * scales\n hs = h * scales\n anchors = _mkanchors(ws, hs, x_ctr, y_ctr)\n return anchors\n" }, { "path": "lib/rpn/rpn.py", "content": "# --------------------------------------------------------\n# Deep Feature Flow\n# Copyright (c) 2017 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Modified by Haozhi Qi\n# --------------------------------------------------------\n# Based on:\n# MX-RCNN\n# Copyright (c) 2016 by Contributors\n# Licence under The Apache 2.0 License\n# https://github.com/ijkguo/mx-rcnn/\n# --------------------------------------------------------\n\n\"\"\"\nRPN:\ndata =\n {'data': [num_images, c, h, w],\n 'im_info': [num_images, 4] (optional)}\nlabel =\n {'gt_boxes': [num_boxes, 5] (optional),\n 'label': [batch_size, 1] <- [batch_size, num_anchors, feat_height, feat_width],\n 'bbox_target': [batch_size, num_anchors, feat_height, feat_width],\n 'bbox_weight': [batch_size, num_anchors, feat_height, feat_width]}\n\"\"\"\n\nimport numpy as np\nimport numpy.random as npr\n\nfrom utils.image import get_image, get_pair_image, tensor_vstack\nfrom generate_anchor import generate_anchors\nfrom bbox.bbox_transform import bbox_overlaps, bbox_transform\n\n\ndef get_rpn_testbatch(roidb, cfg):\n \"\"\"\n return a dict of testbatch\n :param roidb: ['image', 'flipped']\n :return: data, label, im_info\n \"\"\"\n # assert len(roidb) == 1, 'Single batch only'\n imgs, roidb = get_image(roidb, cfg)\n im_array = imgs\n im_info = [np.array([roidb[i]['im_info']], dtype=np.float32) for i in range(len(roidb))]\n\n data = [{'data': im_array[i],\n 'im_info': im_info[i]} for i in range(len(roidb))]\n label = {}\n\n return data, label, im_info\n\n\ndef get_rpn_batch(roidb, cfg):\n \"\"\"\n prototype for rpn batch: data, im_info, gt_boxes\n :param roidb: ['image', 'flipped'] + ['gt_boxes', 'boxes', 'gt_classes']\n :return: data, label\n \"\"\"\n assert len(roidb) == 1, 'Single batch only'\n imgs, roidb = get_image(roidb, cfg)\n im_array = imgs[0]\n im_info = np.array([roidb[0]['im_info']], dtype=np.float32)\n\n # gt boxes: (x1, y1, x2, y2, cls)\n if roidb[0]['gt_classes'].size > 0:\n gt_inds = np.where(roidb[0]['gt_classes'] != 0)[0]\n gt_boxes = np.empty((roidb[0]['boxes'].shape[0], 5), dtype=np.float32)\n gt_boxes[:, 0:4] = roidb[0]['boxes'][gt_inds, :]\n gt_boxes[:, 4] = roidb[0]['gt_classes'][gt_inds]\n else:\n gt_boxes = np.empty((0, 5), dtype=np.float32)\n\n data = {'data': im_array,\n 'im_info': im_info}\n label = {'gt_boxes': gt_boxes}\n\n return data, label\n\ndef get_rpn_pair_batch(roidb, cfg):\n \"\"\"\n prototype for rpn batch: data, im_info, gt_boxes\n :param roidb: ['image', 'flipped'] + ['gt_boxes', 'boxes', 'gt_classes']\n :return: data, label\n \"\"\"\n assert len(roidb) == 1, 'Single batch only'\n imgs, ref_imgs, eq_flags, roidb = get_pair_image(roidb, cfg)\n im_array = imgs[0]\n ref_im_array = ref_imgs[0]\n eq_flag_array = np.array([eq_flags[0],], dtype=np.float32)\n im_info = np.array([roidb[0]['im_info']], dtype=np.float32)\n\n # gt boxes: (x1, y1, x2, y2, cls)\n if roidb[0]['gt_classes'].size > 0:\n gt_inds = np.where(roidb[0]['gt_classes'] != 0)[0]\n gt_boxes = np.empty((roidb[0]['boxes'].shape[0], 5), dtype=np.float32)\n gt_boxes[:, 0:4] = roidb[0]['boxes'][gt_inds, :]\n gt_boxes[:, 4] = roidb[0]['gt_classes'][gt_inds]\n else:\n gt_boxes = np.empty((0, 5), dtype=np.float32)\n\n data = {'data': im_array,\n 'data_ref': ref_im_array,\n 'eq_flag': eq_flag_array,\n 'im_info': im_info}\n label = {'gt_boxes': gt_boxes}\n\n return data, label\n\ndef assign_anchor(feat_shape, gt_boxes, im_info, cfg, feat_stride=16,\n scales=(8, 16, 32), ratios=(0.5, 1, 2), allowed_border=0,\n normalize_target=False, bbox_mean=(0.0, 0.0, 0.0, 0.0),\n bbox_std=(0.1, 0.1, 0.4, 0.4)):\n \"\"\"\n assign ground truth boxes to anchor positions\n :param feat_shape: infer output shape\n :param gt_boxes: assign ground truth\n :param im_info: filter out anchors overlapped with edges\n :param feat_stride: anchor position step\n :param scales: used to generate anchors, affects num_anchors (per location)\n :param ratios: aspect ratios of generated anchors\n :param allowed_border: filter out anchors with edge overlap > allowed_border\n :param normalize_target: normalize rpn target\n :param bbox_mean: anchor target mean\n :param bbox_std: anchor target std\n :return: dict of label\n 'label': of shape (batch_size, 1) <- (batch_size, num_anchors, feat_height, feat_width)\n 'bbox_target': of shape (batch_size, num_anchors * 4, feat_height, feat_width)\n 'bbox_inside_weight': *todo* mark the assigned anchors\n 'bbox_outside_weight': used to normalize the bbox_loss, all weights sums to RPN_POSITIVE_WEIGHT\n \"\"\"\n def _unmap(data, count, inds, fill=0):\n \"\"\"\" unmap a subset inds of data into original data of size count \"\"\"\n if len(data.shape) == 1:\n ret = np.empty((count,), dtype=np.float32)\n ret.fill(fill)\n ret[inds] = data\n else:\n ret = np.empty((count,) + data.shape[1:], dtype=np.float32)\n ret.fill(fill)\n ret[inds, :] = data\n return ret\n\n DEBUG = False\n im_info = im_info[0]\n scales = np.array(scales, dtype=np.float32)\n base_anchors = generate_anchors(base_size=feat_stride, ratios=list(ratios), scales=scales)\n num_anchors = base_anchors.shape[0]\n feat_height, feat_width = feat_shape[-2:]\n\n if DEBUG:\n print 'anchors:'\n print base_anchors\n print 'anchor shapes:'\n print np.hstack((base_anchors[:, 2::4] - base_anchors[:, 0::4],\n base_anchors[:, 3::4] - base_anchors[:, 1::4]))\n print 'im_info', im_info\n print 'height', feat_height, 'width', feat_width\n print 'gt_boxes shape', gt_boxes.shape\n print 'gt_boxes', gt_boxes\n\n # 1. generate proposals from bbox deltas and shifted anchors\n shift_x = np.arange(0, feat_width) * feat_stride\n shift_y = np.arange(0, feat_height) * feat_stride\n shift_x, shift_y = np.meshgrid(shift_x, shift_y)\n shifts = np.vstack((shift_x.ravel(), shift_y.ravel(), shift_x.ravel(), shift_y.ravel())).transpose()\n # add A anchors (1, A, 4) to\n # cell K shifts (K, 1, 4) to get\n # shift anchors (K, A, 4)\n # reshape to (K*A, 4) shifted anchors\n A = num_anchors\n K = shifts.shape[0]\n all_anchors = base_anchors.reshape((1, A, 4)) + shifts.reshape((1, K, 4)).transpose((1, 0, 2))\n all_anchors = all_anchors.reshape((K * A, 4))\n total_anchors = int(K * A)\n\n # only keep anchors inside the image\n inds_inside = np.where((all_anchors[:, 0] >= -allowed_border) &\n (all_anchors[:, 1] >= -allowed_border) &\n (all_anchors[:, 2] < im_info[1] + allowed_border) &\n (all_anchors[:, 3] < im_info[0] + allowed_border))[0]\n if DEBUG:\n print 'total_anchors', total_anchors\n print 'inds_inside', len(inds_inside)\n\n # keep only inside anchors\n anchors = all_anchors[inds_inside, :]\n if DEBUG:\n print 'anchors shape', anchors.shape\n\n # label: 1 is positive, 0 is negative, -1 is dont care\n labels = np.empty((len(inds_inside),), dtype=np.float32)\n labels.fill(-1)\n\n if gt_boxes.size > 0:\n # overlap between the anchors and the gt boxes\n # overlaps (ex, gt)\n overlaps = bbox_overlaps(anchors.astype(np.float), gt_boxes.astype(np.float))\n argmax_overlaps = overlaps.argmax(axis=1)\n max_overlaps = overlaps[np.arange(len(inds_inside)), argmax_overlaps]\n gt_argmax_overlaps = overlaps.argmax(axis=0)\n gt_max_overlaps = overlaps[gt_argmax_overlaps, np.arange(overlaps.shape[1])]\n gt_argmax_overlaps = np.where(overlaps == gt_max_overlaps)[0]\n\n if not cfg.TRAIN.RPN_CLOBBER_POSITIVES:\n # assign bg labels first so that positive labels can clobber them\n labels[max_overlaps < cfg.TRAIN.RPN_NEGATIVE_OVERLAP] = 0\n\n # fg label: for each gt, anchor with highest overlap\n labels[gt_argmax_overlaps] = 1\n\n # fg label: above threshold IoU\n labels[max_overlaps >= cfg.TRAIN.RPN_POSITIVE_OVERLAP] = 1\n\n if cfg.TRAIN.RPN_CLOBBER_POSITIVES:\n # assign bg labels last so that negative labels can clobber positives\n labels[max_overlaps < cfg.TRAIN.RPN_NEGATIVE_OVERLAP] = 0\n else:\n labels[:] = 0\n\n # subsample positive labels if we have too many\n num_fg = int(cfg.TRAIN.RPN_FG_FRACTION * cfg.TRAIN.RPN_BATCH_SIZE)\n fg_inds = np.where(labels == 1)[0]\n if len(fg_inds) > num_fg:\n disable_inds = npr.choice(fg_inds, size=(len(fg_inds) - num_fg), replace=False)\n if DEBUG:\n disable_inds = fg_inds[:(len(fg_inds) - num_fg)]\n labels[disable_inds] = -1\n\n # subsample negative labels if we have too many\n num_bg = cfg.TRAIN.RPN_BATCH_SIZE - np.sum(labels == 1)\n bg_inds = np.where(labels == 0)[0]\n if len(bg_inds) > num_bg:\n disable_inds = npr.choice(bg_inds, size=(len(bg_inds) - num_bg), replace=False)\n if DEBUG:\n disable_inds = bg_inds[:(len(bg_inds) - num_bg)]\n labels[disable_inds] = -1\n\n bbox_targets = np.zeros((len(inds_inside), 4), dtype=np.float32)\n if gt_boxes.size > 0:\n bbox_targets[:] = bbox_transform(anchors, gt_boxes[argmax_overlaps, :4])\n\n bbox_weights = np.zeros((len(inds_inside), 4), dtype=np.float32)\n bbox_weights[labels == 1, :] = np.array(cfg.TRAIN.RPN_BBOX_WEIGHTS)\n\n if DEBUG:\n _sums = bbox_targets[labels == 1, :].sum(axis=0)\n _squared_sums = (bbox_targets[labels == 1, :] ** 2).sum(axis=0)\n _counts = np.sum(labels == 1)\n means = _sums / (_counts + 1e-14)\n stds = np.sqrt(_squared_sums / _counts - means ** 2)\n print 'means', means\n print 'stdevs', stds\n\n if normalize_target:\n bbox_targets = ((bbox_targets - np.array(bbox_mean))\n / np.array(bbox_std))\n\n # map up to original set of anchors\n labels = _unmap(labels, total_anchors, inds_inside, fill=-1)\n bbox_targets = _unmap(bbox_targets, total_anchors, inds_inside, fill=0)\n bbox_weights = _unmap(bbox_weights, total_anchors, inds_inside, fill=0)\n\n if DEBUG:\n print 'rpn: max max_overlaps', np.max(max_overlaps)\n print 'rpn: num_positives', np.sum(labels == 1)\n print 'rpn: num_negatives', np.sum(labels == 0)\n _fg_sum = np.sum(labels == 1)\n _bg_sum = np.sum(labels == 0)\n _count = 1\n print 'rpn: num_positive avg', _fg_sum / _count\n print 'rpn: num_negative avg', _bg_sum / _count\n\n labels = labels.reshape((1, feat_height, feat_width, A)).transpose(0, 3, 1, 2)\n labels = labels.reshape((1, A * feat_height * feat_width))\n bbox_targets = bbox_targets.reshape((1, feat_height, feat_width, A * 4)).transpose(0, 3, 1, 2)\n bbox_weights = bbox_weights.reshape((1, feat_height, feat_width, A * 4)).transpose((0, 3, 1, 2))\n\n label = {'label': labels,\n 'bbox_target': bbox_targets,\n 'bbox_weight': bbox_weights}\n return label\n" }, { "path": "lib/utils/PrefetchingIter.py", "content": "# --------------------------------------------------------\n# Deep Feature Flow\n# Copyright (c) 2017 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Modified by Yuwen Xiong\n# --------------------------------------------------------\n# Based on:\n# MX-RCNN\n# Copyright (c) 2016 by Contributors\n# Licence under The Apache 2.0 License\n# https://github.com/ijkguo/mx-rcnn/\n# --------------------------------------------------------\n\nimport mxnet as mx\nfrom mxnet.io import DataDesc, DataBatch\nimport threading\n\n\nclass PrefetchingIter(mx.io.DataIter):\n \"\"\"Base class for prefetching iterators. Takes one or more DataIters (\n or any class with \"reset\" and \"next\" methods) and combine them with\n prefetching. For example:\n\n Parameters\n ----------\n iters : DataIter or list of DataIter\n one or more DataIters (or any class with \"reset\" and \"next\" methods)\n rename_data : None or list of dict\n i-th element is a renaming map for i-th iter, in the form of\n {'original_name' : 'new_name'}. Should have one entry for each entry\n in iter[i].provide_data\n rename_label : None or list of dict\n Similar to rename_data\n\n Examples\n --------\n iter = PrefetchingIter([NDArrayIter({'data': X1}), NDArrayIter({'data': X2})],\n rename_data=[{'data': 'data1'}, {'data': 'data2'}])\n \"\"\"\n def __init__(self, iters, rename_data=None, rename_label=None):\n super(PrefetchingIter, self).__init__()\n if not isinstance(iters, list):\n iters = [iters]\n self.n_iter = len(iters)\n assert self.n_iter ==1, \"Our prefetching iter only support 1 DataIter\"\n self.iters = iters\n self.rename_data = rename_data\n self.rename_label = rename_label\n self.batch_size = len(self.provide_data) * self.provide_data[0][0][1][0]\n self.data_ready = [threading.Event() for i in range(self.n_iter)]\n self.data_taken = [threading.Event() for i in range(self.n_iter)]\n for e in self.data_taken:\n e.set()\n self.started = True\n self.current_batch = [None for _ in range(self.n_iter)]\n self.next_batch = [None for _ in range(self.n_iter)]\n def prefetch_func(self, i):\n \"\"\"Thread entry\"\"\"\n while True:\n self.data_taken[i].wait()\n if not self.started:\n break\n try:\n self.next_batch[i] = self.iters[i].next()\n except StopIteration:\n self.next_batch[i] = None\n self.data_taken[i].clear()\n self.data_ready[i].set()\n self.prefetch_threads = [threading.Thread(target=prefetch_func, args=[self, i]) \\\n for i in range(self.n_iter)]\n for thread in self.prefetch_threads:\n thread.setDaemon(True)\n thread.start()\n\n def __del__(self):\n self.started = False\n for e in self.data_taken:\n e.set()\n for thread in self.prefetch_threads:\n thread.join()\n\n @property\n def provide_data(self):\n \"\"\"The name and shape of data provided by this iterator\"\"\"\n if self.rename_data is None:\n return sum([i.provide_data for i in self.iters], [])\n else:\n return sum([[\n DataDesc(r[x.name], x.shape, x.dtype)\n if isinstance(x, DataDesc) else DataDesc(*x)\n for x in i.provide_data\n ] for r, i in zip(self.rename_data, self.iters)], [])\n\n @property\n def provide_label(self):\n \"\"\"The name and shape of label provided by this iterator\"\"\"\n if self.rename_label is None:\n return sum([i.provide_label for i in self.iters], [])\n else:\n return sum([[\n DataDesc(r[x.name], x.shape, x.dtype)\n if isinstance(x, DataDesc) else DataDesc(*x)\n for x in i.provide_label\n ] for r, i in zip(self.rename_label, self.iters)], [])\n\n def reset(self):\n for e in self.data_ready:\n e.wait()\n for i in self.iters:\n i.reset()\n for e in self.data_ready:\n e.clear()\n for e in self.data_taken:\n e.set()\n\n def iter_next(self):\n for e in self.data_ready:\n e.wait()\n if self.next_batch[0] is None:\n return False\n else:\n self.current_batch = self.next_batch[0]\n for e in self.data_ready:\n e.clear()\n for e in self.data_taken:\n e.set()\n return True\n\n def next(self):\n if self.iter_next():\n return self.current_batch\n else:\n raise StopIteration\n\n def getdata(self):\n return self.current_batch.data\n\n def getlabel(self):\n return self.current_batch.label\n\n def getindex(self):\n return self.current_batch.index\n\n def getpad(self):\n return self.current_batch.pad\n" }, { "path": "lib/utils/__init__.py", "content": "" }, { "path": "lib/utils/combine_model.py", "content": "# --------------------------------------------------------\n# Deep Feature Flow\n# Copyright (c) 2017 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Written by Yuwen Xiong\n# --------------------------------------------------------\n\nfrom load_model import load_checkpoint\nfrom save_model import save_checkpoint\n\n\ndef combine_model(prefix1, epoch1, prefix2, epoch2, prefix_out, epoch_out):\n args1, auxs1 = load_checkpoint(prefix1, epoch1)\n args2, auxs2 = load_checkpoint(prefix2, epoch2)\n arg_names = args1.keys() + args2.keys()\n aux_names = auxs1.keys() + auxs2.keys()\n args = dict()\n for arg in arg_names:\n if arg in args1:\n args[arg] = args1[arg]\n if arg in args2:\n args[arg] = args2[arg]\n auxs = dict()\n for aux in aux_names:\n if aux in auxs1:\n auxs[aux] = auxs1[aux]\n if aux in auxs2:\n auxs[aux] = auxs2[aux]\n save_checkpoint(prefix_out, epoch_out, args, auxs)\n" }, { "path": "lib/utils/create_logger.py", "content": "# --------------------------------------------------------\n# Deep Feature Flow\n# Copyright (c) 2017 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Written by Bin Xiao\n# --------------------------------------------------------\n\nimport os\nimport logging\nimport time\n\ndef create_logger(root_output_path, cfg, image_set):\n # set up logger\n if not os.path.exists(root_output_path):\n os.makedirs(root_output_path)\n assert os.path.exists(root_output_path), '{} does not exist'.format(root_output_path)\n\n cfg_name = os.path.basename(cfg).split('.')[0]\n config_output_path = os.path.join(root_output_path, '{}'.format(cfg_name))\n if not os.path.exists(config_output_path):\n os.makedirs(config_output_path)\n\n image_sets = [iset for iset in image_set.split('+')]\n final_output_path = os.path.join(config_output_path, '{}'.format('_'.join(image_sets)))\n if not os.path.exists(final_output_path):\n os.makedirs(final_output_path)\n\n log_file = '{}_{}.log'.format(cfg_name, time.strftime('%Y-%m-%d-%H-%M'))\n head = '%(asctime)-15s %(message)s'\n logging.basicConfig(filename=os.path.join(final_output_path, log_file), format=head)\n logger = logging.getLogger()\n logger.setLevel(logging.INFO)\n\n return logger, final_output_path\n\n" }, { "path": "lib/utils/image.py", "content": "# --------------------------------------------------------\n# Deep Feature Flow\n# Copyright (c) 2017 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Written by Yuwen Xiong\n# --------------------------------------------------------\n\nimport numpy as np\nimport os\nimport cv2\nimport random\nfrom PIL import Image\nfrom bbox.bbox_transform import clip_boxes\n\n\n# TODO: This two functions should be merged with individual data loader\ndef get_image(roidb, config):\n \"\"\"\n preprocess image and return processed roidb\n :param roidb: a list of roidb\n :return: list of img as in mxnet format\n roidb add new item['im_info']\n 0 --- x (width, second dim of im)\n |\n y (height, first dim of im)\n \"\"\"\n num_images = len(roidb)\n processed_ims = []\n processed_roidb = []\n for i in range(num_images):\n roi_rec = roidb[i]\n assert os.path.exists(roi_rec['image']), '%s does not exist'.format(roi_rec['image'])\n im = cv2.imread(roi_rec['image'], cv2.IMREAD_COLOR|cv2.IMREAD_IGNORE_ORIENTATION)\n if roidb[i]['flipped']:\n im = im[:, ::-1, :]\n new_rec = roi_rec.copy()\n scale_ind = random.randrange(len(config.SCALES))\n target_size = config.SCALES[scale_ind][0]\n max_size = config.SCALES[scale_ind][1]\n im, im_scale = resize(im, target_size, max_size, stride=config.network.IMAGE_STRIDE)\n im_tensor = transform(im, config.network.PIXEL_MEANS)\n processed_ims.append(im_tensor)\n im_info = [im_tensor.shape[2], im_tensor.shape[3], im_scale]\n new_rec['boxes'] = clip_boxes(np.round(roi_rec['boxes'].copy() * im_scale), im_info[:2])\n new_rec['im_info'] = im_info\n processed_roidb.append(new_rec)\n return processed_ims, processed_roidb\n\ndef get_pair_image(roidb, config):\n \"\"\"\n preprocess image and return processed roidb\n :param roidb: a list of roidb\n :return: list of img as in mxnet format\n roidb add new item['im_info']\n 0 --- x (width, second dim of im)\n |\n y (height, first dim of im)\n \"\"\"\n num_images = len(roidb)\n processed_ims = []\n processed_ref_ims = []\n processed_eq_flags = []\n processed_roidb = []\n for i in range(num_images):\n roi_rec = roidb[i]\n\n eq_flag = 0 # 0 for unequal, 1 for equal\n assert os.path.exists(roi_rec['image']), '%s does not exist'.format(roi_rec['image'])\n im = cv2.imread(roi_rec['image'], cv2.IMREAD_COLOR|cv2.IMREAD_IGNORE_ORIENTATION)\n\n if roi_rec.has_key('pattern'):\n ref_id = min(max(roi_rec['frame_seg_id'] + np.random.randint(config.TRAIN.MIN_OFFSET, config.TRAIN.MAX_OFFSET+1), 0),roi_rec['frame_seg_len']-1)\n ref_image = roi_rec['pattern'] % ref_id\n assert os.path.exists(ref_image), '%s does not exist'.format(ref_image)\n ref_im = cv2.imread(ref_image, cv2.IMREAD_COLOR|cv2.IMREAD_IGNORE_ORIENTATION)\n if ref_id == roi_rec['frame_seg_id']:\n eq_flag = 1\n else:\n ref_im = im.copy()\n eq_flag = 1\n\n if roidb[i]['flipped']:\n im = im[:, ::-1, :]\n ref_im = ref_im[:, ::-1, :]\n\n new_rec = roi_rec.copy()\n scale_ind = random.randrange(len(config.SCALES))\n target_size = config.SCALES[scale_ind][0]\n max_size = config.SCALES[scale_ind][1]\n\n im, im_scale = resize(im, target_size, max_size, stride=config.network.IMAGE_STRIDE)\n ref_im, im_scale = resize(ref_im, target_size, max_size, stride=config.network.IMAGE_STRIDE)\n im_tensor = transform(im, config.network.PIXEL_MEANS)\n ref_im_tensor = transform(ref_im, config.network.PIXEL_MEANS)\n processed_ims.append(im_tensor)\n processed_ref_ims.append(ref_im_tensor)\n processed_eq_flags.append(eq_flag)\n im_info = [im_tensor.shape[2], im_tensor.shape[3], im_scale]\n new_rec['boxes'] = roi_rec['boxes'].copy() * im_scale\n new_rec['im_info'] = im_info\n processed_roidb.append(new_rec)\n return processed_ims, processed_ref_ims, processed_eq_flags, processed_roidb\n\ndef resize(im, target_size, max_size, stride=0, interpolation = cv2.INTER_LINEAR):\n \"\"\"\n only resize input image to target size and return scale\n :param im: BGR image input by opencv\n :param target_size: one dimensional size (the short side)\n :param max_size: one dimensional max size (the long side)\n :param stride: if given, pad the image to designated stride\n :param interpolation: if given, using given interpolation method to resize image\n :return:\n \"\"\"\n im_shape = im.shape\n im_size_min = np.min(im_shape[0:2])\n im_size_max = np.max(im_shape[0:2])\n im_scale = float(target_size) / float(im_size_min)\n # prevent bigger axis from being more than max_size:\n if np.round(im_scale * im_size_max) > max_size:\n im_scale = float(max_size) / float(im_size_max)\n im = cv2.resize(im, None, None, fx=im_scale, fy=im_scale, interpolation=interpolation)\n\n if stride == 0:\n return im, im_scale\n else:\n # pad to product of stride\n im_height = int(np.ceil(im.shape[0] / float(stride)) * stride)\n im_width = int(np.ceil(im.shape[1] / float(stride)) * stride)\n im_channel = im.shape[2]\n padded_im = np.zeros((im_height, im_width, im_channel))\n padded_im[:im.shape[0], :im.shape[1], :] = im\n return padded_im, im_scale\n\ndef transform(im, pixel_means):\n \"\"\"\n transform into mxnet tensor\n substract pixel size and transform to correct format\n :param im: [height, width, channel] in BGR\n :param pixel_means: [B, G, R pixel means]\n :return: [batch, channel, height, width]\n \"\"\"\n im_tensor = np.zeros((1, 3, im.shape[0], im.shape[1]))\n for i in range(3):\n im_tensor[0, i, :, :] = im[:, :, 2 - i] - pixel_means[2 - i]\n return im_tensor\n\ndef transform_seg_gt(gt):\n \"\"\"\n transform segmentation gt image into mxnet tensor\n :param gt: [height, width, channel = 1]\n :return: [batch, channel = 1, height, width]\n \"\"\"\n gt_tensor = np.zeros((1, 1, gt.shape[0], gt.shape[1]))\n gt_tensor[0, 0, :, :] = gt[:, :]\n\n return gt_tensor\n\ndef transform_inverse(im_tensor, pixel_means):\n \"\"\"\n transform from mxnet im_tensor to ordinary RGB image\n im_tensor is limited to one image\n :param im_tensor: [batch, channel, height, width]\n :param pixel_means: [B, G, R pixel means]\n :return: im [height, width, channel(RGB)]\n \"\"\"\n assert im_tensor.shape[0] == 1\n im_tensor = im_tensor.copy()\n # put channel back\n channel_swap = (0, 2, 3, 1)\n im_tensor = im_tensor.transpose(channel_swap)\n im = im_tensor[0]\n assert im.shape[2] == 3\n im += pixel_means[[2, 1, 0]]\n im = im.astype(np.uint8)\n return im\n\ndef tensor_vstack(tensor_list, pad=0):\n \"\"\"\n vertically stack tensors\n :param tensor_list: list of tensor to be stacked vertically\n :param pad: label to pad with\n :return: tensor with max shape\n \"\"\"\n ndim = len(tensor_list[0].shape)\n dtype = tensor_list[0].dtype\n islice = tensor_list[0].shape[0]\n dimensions = []\n first_dim = sum([tensor.shape[0] for tensor in tensor_list])\n dimensions.append(first_dim)\n for dim in range(1, ndim):\n dimensions.append(max([tensor.shape[dim] for tensor in tensor_list]))\n if pad == 0:\n all_tensor = np.zeros(tuple(dimensions), dtype=dtype)\n elif pad == 1:\n all_tensor = np.ones(tuple(dimensions), dtype=dtype)\n else:\n all_tensor = np.full(tuple(dimensions), pad, dtype=dtype)\n if ndim == 1:\n for ind, tensor in enumerate(tensor_list):\n all_tensor[ind*islice:(ind+1)*islice] = tensor\n elif ndim == 2:\n for ind, tensor in enumerate(tensor_list):\n all_tensor[ind*islice:(ind+1)*islice, :tensor.shape[1]] = tensor\n elif ndim == 3:\n for ind, tensor in enumerate(tensor_list):\n all_tensor[ind*islice:(ind+1)*islice, :tensor.shape[1], :tensor.shape[2]] = tensor\n elif ndim == 4:\n for ind, tensor in enumerate(tensor_list):\n all_tensor[ind*islice:(ind+1)*islice, :tensor.shape[1], :tensor.shape[2], :tensor.shape[3]] = tensor\n else:\n raise Exception('Sorry, unimplemented.')\n return all_tensor\n" }, { "path": "lib/utils/image_processing.py", "content": "# --------------------------------------------------------\n# Deep Feature Flow\n# Copyright (c) 2017 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Written by Yuwen Xiong\n# --------------------------------------------------------\n\nimport numpy as np\nimport cv2\n\n\ndef resize(im, target_size, max_size):\n \"\"\"\n only resize input image to target size and return scale\n :param im: BGR image input by opencv\n :param target_size: one dimensional size (the short side)\n :param max_size: one dimensional max size (the long side)\n :return:\n \"\"\"\n im_shape = im.shape\n im_size_min = np.min(im_shape[0:2])\n im_size_max = np.max(im_shape[0:2])\n im_scale = float(target_size) / float(im_size_min)\n # prevent bigger axis from being more than max_size:\n if np.round(im_scale * im_size_max) > max_size:\n im_scale = float(max_size) / float(im_size_max)\n im = cv2.resize(im, None, None, fx=im_scale, fy=im_scale, interpolation=cv2.INTER_LINEAR)\n return im, im_scale\n\n\ndef transform(im, pixel_means, need_mean=False):\n \"\"\"\n transform into mxnet tensor\n subtract pixel size and transform to correct format\n :param im: [height, width, channel] in BGR\n :param pixel_means: [[[R, G, B pixel means]]]\n :return: [batch, channel, height, width]\n \"\"\"\n assert False, \"shouldn't reach here.\"\n im = im.copy()\n im[:, :, (0, 1, 2)] = im[:, :, (2, 1, 0)]\n im = im.astype(float)\n if need_mean:\n im -= pixel_means\n im_tensor = im[np.newaxis, :]\n # put channel first\n channel_swap = (0, 3, 1, 2)\n im_tensor = im_tensor.transpose(channel_swap)\n return im_tensor\n\n\ndef transform_inverse(im_tensor, pixel_means):\n \"\"\"\n transform from mxnet im_tensor to ordinary RGB image\n im_tensor is limited to one image\n :param im_tensor: [batch, channel, height, width]\n :param pixel_means: [[[R, G, B pixel means]]]\n :return: im [height, width, channel(RGB)]\n \"\"\"\n assert im_tensor.shape[0] == 1\n im_tensor = im_tensor.copy()\n # put channel back\n channel_swap = (0, 2, 3, 1)\n im_tensor = im_tensor.transpose(channel_swap)\n im = im_tensor[0]\n assert im.shape[2] == 3\n im += pixel_means\n im = im.astype(np.uint8)\n return im\n\n\ndef tensor_vstack(tensor_list, pad=0):\n \"\"\"\n vertically stack tensors\n :param tensor_list: list of tensor to be stacked vertically\n :param pad: label to pad with\n :return: tensor with max shape\n \"\"\"\n ndim = len(tensor_list[0].shape)\n if ndim == 1:\n return np.hstack(tensor_list)\n dimensions = [0]\n for dim in range(1, ndim):\n dimensions.append(max([tensor.shape[dim] for tensor in tensor_list]))\n for ind, tensor in enumerate(tensor_list):\n pad_shape = [(0, 0)]\n for dim in range(1, ndim):\n pad_shape.append((0, dimensions[dim] - tensor.shape[dim]))\n tensor_list[ind] = np.lib.pad(tensor, pad_shape, 'constant', constant_values=pad)\n all_tensor = np.vstack(tensor_list)\n return all_tensor\n" }, { "path": "lib/utils/load_data.py", "content": "# --------------------------------------------------------\n# Deep Feature Flow\n# Copyright (c) 2017 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Written by Yuwen Xiong\n# --------------------------------------------------------\n\nimport numpy as np\nfrom dataset import *\n\n\ndef load_gt_roidb(dataset_name, image_set_name, root_path, dataset_path, result_path=None,\n flip=False):\n \"\"\" load ground truth roidb \"\"\"\n imdb = eval(dataset_name)(image_set_name, root_path, dataset_path, result_path)\n roidb = imdb.gt_roidb()\n if flip:\n roidb = imdb.append_flipped_images(roidb)\n return roidb\n\n\ndef load_proposal_roidb(dataset_name, image_set_name, root_path, dataset_path, result_path=None,\n proposal='rpn', append_gt=True, flip=False):\n \"\"\" load proposal roidb (append_gt when training) \"\"\"\n imdb = eval(dataset_name)(image_set_name, root_path, dataset_path, result_path)\n\n gt_roidb = imdb.gt_roidb()\n roidb = eval('imdb.' + proposal + '_roidb')(gt_roidb, append_gt)\n if flip:\n roidb = imdb.append_flipped_images(roidb)\n return roidb\n\n\ndef merge_roidb(roidbs):\n \"\"\" roidb are list, concat them together \"\"\"\n roidb = roidbs[0]\n for r in roidbs[1:]:\n roidb.extend(r)\n return roidb\n\n\ndef filter_roidb(roidb, config):\n \"\"\" remove roidb entries without usable rois \"\"\"\n\n def is_valid(entry):\n \"\"\" valid images have at least 1 fg or bg roi \"\"\"\n overlaps = entry['max_overlaps']\n fg_inds = np.where(overlaps >= config.TRAIN.FG_THRESH)[0]\n bg_inds = np.where((overlaps < config.TRAIN.BG_THRESH_HI) & (overlaps >= config.TRAIN.BG_THRESH_LO))[0]\n valid = len(fg_inds) > 0 or len(bg_inds) > 0\n return valid\n\n num = len(roidb)\n filtered_roidb = [entry for entry in roidb if is_valid(entry)]\n num_after = len(filtered_roidb)\n print 'filtered %d roidb entries: %d -> %d' % (num - num_after, num, num_after)\n\n return filtered_roidb\n\n\ndef load_gt_segdb(dataset_name, image_set_name, root_path, dataset_path, result_path=None,\n flip=False):\n \"\"\" load ground truth segdb \"\"\"\n imdb = eval(dataset_name)(image_set_name, root_path, dataset_path, result_path)\n segdb = imdb.gt_segdb()\n if flip:\n segdb = imdb.append_flipped_images_for_segmentation(segdb)\n return segdb\n\n\ndef merge_segdb(segdbs):\n \"\"\" segdb are list, concat them together \"\"\"\n segdb = segdbs[0]\n for r in segdbs[1:]:\n segdb.extend(r)\n return segdb\n" }, { "path": "lib/utils/load_model.py", "content": "# --------------------------------------------------------\n# Deep Feature Flow\n# Copyright (c) 2017 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Written by Yuwen Xiong\n# --------------------------------------------------------\n\nimport mxnet as mx\n\n\ndef load_checkpoint(prefix, epoch):\n \"\"\"\n Load model checkpoint from file.\n :param prefix: Prefix of model name.\n :param epoch: Epoch number of model we would like to load.\n :return: (arg_params, aux_params)\n arg_params : dict of str to NDArray\n Model parameter, dict of name to NDArray of net's weights.\n aux_params : dict of str to NDArray\n Model parameter, dict of name to NDArray of net's auxiliary states.\n \"\"\"\n save_dict = mx.nd.load('%s-%04d.params' % (prefix, epoch))\n arg_params = {}\n aux_params = {}\n for k, v in save_dict.items():\n tp, name = k.split(':', 1)\n if tp == 'arg':\n arg_params[name] = v\n if tp == 'aux':\n aux_params[name] = v\n return arg_params, aux_params\n\n\ndef convert_context(params, ctx):\n \"\"\"\n :param params: dict of str to NDArray\n :param ctx: the context to convert to\n :return: dict of str of NDArray with context ctx\n \"\"\"\n new_params = dict()\n for k, v in params.items():\n new_params[k] = v.as_in_context(ctx)\n return new_params\n\n\ndef load_param(prefix, epoch, convert=False, ctx=None, process=False):\n \"\"\"\n wrapper for load checkpoint\n :param prefix: Prefix of model name.\n :param epoch: Epoch number of model we would like to load.\n :param convert: reference model should be converted to GPU NDArray first\n :param ctx: if convert then ctx must be designated.\n :param process: model should drop any test\n :return: (arg_params, aux_params)\n \"\"\"\n arg_params, aux_params = load_checkpoint(prefix, epoch)\n if convert:\n if ctx is None:\n ctx = mx.cpu()\n arg_params = convert_context(arg_params, ctx)\n aux_params = convert_context(aux_params, ctx)\n if process:\n tests = [k for k in arg_params.keys() if '_test' in k]\n for test in tests:\n arg_params[test.replace('_test', '')] = arg_params.pop(test)\n return arg_params, aux_params\n" }, { "path": "lib/utils/lr_scheduler.py", "content": "# --------------------------------------------------------\n# Deep Feature Flow\n# Copyright (c) 2017 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Written by Yuwen Xiong\n# --------------------------------------------------------\n\n\nimport logging\nfrom mxnet.lr_scheduler import LRScheduler\n\nclass WarmupMultiFactorScheduler(LRScheduler):\n \"\"\"Reduce learning rate in factor at steps specified in a list\n\n Assume the weight has been updated by n times, then the learning rate will\n be\n\n base_lr * factor^(sum((step/n)<=1)) # step is an array\n\n Parameters\n ----------\n step: list of int\n schedule learning rate after n updates\n factor: float\n the factor for reducing the learning rate\n \"\"\"\n def __init__(self, step, factor=1, warmup=False, warmup_lr=0, warmup_step=0):\n super(WarmupMultiFactorScheduler, self).__init__()\n assert isinstance(step, list) and len(step) >= 1\n for i, _step in enumerate(step):\n if i != 0 and step[i] <= step[i-1]:\n raise ValueError(\"Schedule step must be an increasing integer list\")\n if _step < 1:\n raise ValueError(\"Schedule step must be greater or equal than 1 round\")\n if factor > 1.0:\n raise ValueError(\"Factor must be no more than 1 to make lr reduce\")\n self.step = step\n self.cur_step_ind = 0\n self.factor = factor\n self.count = 0\n self.warmup = warmup\n self.warmup_lr = warmup_lr\n self.warmup_step = warmup_step\n\n def __call__(self, num_update):\n \"\"\"\n Call to schedule current learning rate\n\n Parameters\n ----------\n num_update: int\n the maximal number of updates applied to a weight.\n \"\"\"\n\n # NOTE: use while rather than if (for continuing training via load_epoch)\n if self.warmup and num_update < self.warmup_step:\n return self.warmup_lr\n while self.cur_step_ind <= len(self.step)-1:\n if num_update > self.step[self.cur_step_ind]:\n self.count = self.step[self.cur_step_ind]\n self.cur_step_ind += 1\n self.base_lr *= self.factor\n logging.info(\"Update[%d]: Change learning rate to %0.5e\",\n num_update, self.base_lr)\n else:\n return self.base_lr\n return self.base_lr\n" }, { "path": "lib/utils/roidb.py", "content": "# --------------------------------------------------------\n# Deep Feature Flow\n# Copyright (c) 2017 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Written by Yuwen Xiong\n# --------------------------------------------------------\n\n\"\"\"\nroidb\nbasic format [image_index]['boxes', 'gt_classes', 'gt_overlaps', 'flipped']\nextended ['image', 'max_classes', 'max_overlaps', 'bbox_targets']\n\"\"\"\n\nimport cv2\nimport numpy as np\n\nfrom bbox.bbox_regression import compute_bbox_regression_targets\n\n\ndef prepare_roidb(imdb, roidb, cfg):\n \"\"\"\n add image path, max_classes, max_overlaps to roidb\n :param imdb: image database, provide path\n :param roidb: roidb\n :return: None\n \"\"\"\n print 'prepare roidb'\n for i in range(len(roidb)): # image_index\n roidb[i]['image'] = imdb.image_path_from_index(imdb.image_set_index[i])\n if cfg.TRAIN.ASPECT_GROUPING:\n size = cv2.imread(roidb[i]['image']).shape\n roidb[i]['height'] = size[0]\n roidb[i]['width'] = size[1]\n gt_overlaps = roidb[i]['gt_overlaps'].toarray()\n max_overlaps = gt_overlaps.max(axis=1)\n max_classes = gt_overlaps.argmax(axis=1)\n roidb[i]['max_overlaps'] = max_overlaps\n roidb[i]['max_classes'] = max_classes\n\n # background roi => background class\n zero_indexes = np.where(max_overlaps == 0)[0]\n assert all(max_classes[zero_indexes] == 0)\n # foreground roi => foreground class\n nonzero_indexes = np.where(max_overlaps > 0)[0]\n assert all(max_classes[nonzero_indexes] != 0)\n" }, { "path": "lib/utils/save_model.py", "content": "# --------------------------------------------------------\n# Deep Feature Flow\n# Copyright (c) 2017 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Written by Yuwen Xiong\n# --------------------------------------------------------\n\nimport mxnet as mx\n\n\ndef save_checkpoint(prefix, epoch, arg_params, aux_params):\n \"\"\"Checkpoint the model data into file.\n :param prefix: Prefix of model name.\n :param epoch: The epoch number of the model.\n :param arg_params: dict of str to NDArray\n Model parameter, dict of name to NDArray of net's weights.\n :param aux_params: dict of str to NDArray\n Model parameter, dict of name to NDArray of net's auxiliary states.\n :return: None\n prefix-epoch.params will be saved for parameters.\n \"\"\"\n save_dict = {('arg:%s' % k) : v for k, v in arg_params.items()}\n save_dict.update({('aux:%s' % k) : v for k, v in aux_params.items()})\n param_name = '%s-%04d.params' % (prefix, epoch)\n mx.nd.save(param_name, save_dict)\n" }, { "path": "lib/utils/show_boxes.py", "content": "# --------------------------------------------------------\n# Deep Feature Flow\n# Copyright (c) 2017 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Written by Yi Li, Haocheng Zhang, Xizhou Zhu\n# --------------------------------------------------------\n\nimport matplotlib.pyplot as plt\nimport cv2\nimport random\n\ndef show_boxes(im, dets, classes, scale = 1.0):\n plt.cla()\n plt.axis(\"off\")\n plt.imshow(im)\n for cls_idx, cls_name in enumerate(classes):\n cls_dets = dets[cls_idx]\n for det in cls_dets:\n bbox = det[:4] * scale\n color = (random.random(), random.random(), random.random())\n rect = plt.Rectangle((bbox[0], bbox[1]),\n bbox[2] - bbox[0],\n bbox[3] - bbox[1], fill=False,\n edgecolor=color, linewidth=2.5)\n plt.gca().add_patch(rect)\n\n if cls_dets.shape[1] == 5:\n score = det[-1]\n plt.gca().text(bbox[0], bbox[1],\n '{:s} {:.3f}'.format(cls_name, score),\n bbox=dict(facecolor=color, alpha=0.5), fontsize=9, color='white')\n plt.show()\n return im\n\n\ndef draw_boxes(im, dets, classes, scale = 1.0):\n color_white = (255, 255, 255)\n for cls_idx, cls_name in enumerate(classes):\n cls_dets = dets[cls_idx]\n for det in cls_dets:\n bbox = det[:4] * scale\n bbox = map(int, bbox)\n color = (random.randint(0, 256), random.randint(0, 256), random.randint(0, 256))\n cv2.rectangle(im, (bbox[0], bbox[1]), (bbox[2], bbox[3]), color=color, thickness=3)\n\n if cls_dets.shape[1] == 5:\n score = det[-1]\n cv2.putText(im, '%s %.3f' % (cls_name, score), (bbox[0], bbox[1]+10),\n color=color_white, fontFace=cv2.FONT_HERSHEY_COMPLEX, fontScale=1, thickness=2)\n return im\n" }, { "path": "lib/utils/symbol.py", "content": "# --------------------------------------------------------\n# Deep Feature Flow\n# Copyright (c) 2017 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Written by Yuwen Xiong\n# --------------------------------------------------------\n\nimport numpy as np\nclass Symbol:\n def __init__(self):\n self.arg_shape_dict = None\n self.out_shape_dict = None\n self.aux_shape_dict = None\n self.sym = None\n\n @property\n def symbol(self):\n return self.sym\n\n def get_symbol(self, cfg, is_train=True):\n \"\"\"\n return a generated symbol, it also need to be assigned to self.sym\n \"\"\"\n raise NotImplementedError()\n\n def init_weights(self, cfg, arg_params, aux_params):\n raise NotImplementedError()\n\n def get_msra_std(self, shape):\n fan_in = float(shape[1])\n if len(shape) > 2:\n fan_in *= np.prod(shape[2:])\n print(np.sqrt(2 / fan_in))\n return np.sqrt(2 / fan_in)\n\n def infer_shape(self, data_shape_dict):\n # infer shape\n arg_shape, out_shape, aux_shape = self.sym.infer_shape(**data_shape_dict)\n self.arg_shape_dict = dict(zip(self.sym.list_arguments(), arg_shape))\n self.out_shape_dict = dict(zip(self.sym.list_outputs(), out_shape))\n self.aux_shape_dict = dict(zip(self.sym.list_auxiliary_states(), aux_shape))\n\n def check_parameter_shapes(self, arg_params, aux_params, data_shape_dict, is_train=True):\n for k in self.sym.list_arguments():\n if k in data_shape_dict or (False if is_train else 'label' in k):\n continue\n assert k in arg_params, k + ' not initialized'\n assert arg_params[k].shape == self.arg_shape_dict[k], \\\n 'shape inconsistent for ' + k + ' inferred ' + str(self.arg_shape_dict[k]) + ' provided ' + str(\n arg_params[k].shape)\n for k in self.sym.list_auxiliary_states():\n assert k in aux_params, k + ' not initialized'\n assert aux_params[k].shape == self.aux_shape_dict[k], \\\n 'shape inconsistent for ' + k + ' inferred ' + str(self.aux_shape_dict[k]) + ' provided ' + str(\n aux_params[k].shape)\n" }, { "path": "lib/utils/tictoc.py", "content": "# --------------------------------------------------------\n# Deep Feature Flow\n# Copyright (c) 2017 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Written by Yuwen Xiong\n# --------------------------------------------------------\n\nimport time\n\ndef tic():\n import time\n global startTime_for_tictoc\n startTime_for_tictoc = time.time()\n return startTime_for_tictoc\n\ndef toc():\n if 'startTime_for_tictoc' in globals():\n endTime = time.time()\n return endTime - startTime_for_tictoc\n else:\n return None\n" }, { "path": "rfcn/__init__.py", "content": "" }, { "path": "rfcn/_init_paths.py", "content": "# --------------------------------------------------------\n# Deep Feature Flow\n# Copyright (c) 2017 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Written by Xizhou Zhu\n# --------------------------------------------------------\n\nimport os.path as osp\nimport sys\n\ndef add_path(path):\n if path not in sys.path:\n sys.path.insert(0, path)\n\nthis_dir = osp.dirname(__file__)\n\nlib_path = osp.join(this_dir, '..', 'lib')\nadd_path(lib_path)\n" }, { "path": "rfcn/config/__init__.py", "content": "" }, { "path": "rfcn/config/config.py", "content": "# --------------------------------------------------------\n# Deep Feature Flow\n# Copyright (c) 2017 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Modified by Xizhou Zhu, Yuwen Xiong, Bin Xiao\n# --------------------------------------------------------\n# Based on:\n# MX-RCNN\n# Copyright (c) 2016 by Contributors\n# Licence under The Apache 2.0 License\n# https://github.com/ijkguo/mx-rcnn/\n# --------------------------------------------------------\n\nimport yaml\nimport numpy as np\nfrom easydict import EasyDict as edict\n\nconfig = edict()\n\nconfig.MXNET_VERSION = ''\nconfig.output_path = ''\nconfig.symbol = ''\nconfig.gpus = ''\nconfig.CLASS_AGNOSTIC = True\nconfig.SCALES = [(600, 1000)] # first is scale (the shorter side); second is max size\n\n# default training\nconfig.default = edict()\nconfig.default.frequent = 20\nconfig.default.kvstore = 'device'\n\n# network related params\nconfig.network = edict()\nconfig.network.pretrained = ''\nconfig.network.pretrained_epoch = 0\nconfig.network.PIXEL_MEANS = np.array([0, 0, 0])\nconfig.network.IMAGE_STRIDE = 0\nconfig.network.RPN_FEAT_STRIDE = 16\nconfig.network.RCNN_FEAT_STRIDE = 16\nconfig.network.FIXED_PARAMS = ['gamma', 'beta']\nconfig.network.ANCHOR_SCALES = (8, 16, 32)\nconfig.network.ANCHOR_RATIOS = (0.5, 1, 2)\nconfig.network.NORMALIZE_RPN = True\nconfig.network.ANCHOR_MEANS = (0.0, 0.0, 0.0, 0.0)\nconfig.network.ANCHOR_STDS = (0.1, 0.1, 0.4, 0.4)\nconfig.network.NUM_ANCHORS = len(config.network.ANCHOR_SCALES) * len(config.network.ANCHOR_RATIOS)\n\n# dataset related params\nconfig.dataset = edict()\nconfig.dataset.dataset = 'ImageNetVID'\nconfig.dataset.image_set = 'DET_train_30classes+VID_train_15frames'\nconfig.dataset.test_image_set = 'VID_val_frames'\nconfig.dataset.root_path = './data'\nconfig.dataset.dataset_path = './data/ILSVRC2015'\nconfig.dataset.NUM_CLASSES = 31\n\n\nconfig.TRAIN = edict()\n\nconfig.TRAIN.lr = 0\nconfig.TRAIN.lr_step = ''\nconfig.TRAIN.lr_factor = 0.1\nconfig.TRAIN.warmup = False\nconfig.TRAIN.warmup_lr = 0\nconfig.TRAIN.warmup_step = 0\nconfig.TRAIN.momentum = 0.9\nconfig.TRAIN.wd = 0.0005\nconfig.TRAIN.begin_epoch = 0\nconfig.TRAIN.end_epoch = 0\nconfig.TRAIN.model_prefix = ''\n\n# whether resume training\nconfig.TRAIN.RESUME = False\n# whether flip image\nconfig.TRAIN.FLIP = True\n# whether shuffle image\nconfig.TRAIN.SHUFFLE = True\n# whether use OHEM\nconfig.TRAIN.ENABLE_OHEM = False\n# size of images for each device, 2 for rcnn, 1 for rpn and e2e\nconfig.TRAIN.BATCH_IMAGES = 2\n# e2e changes behavior of anchor loader and metric\nconfig.TRAIN.END2END = False\n# group images with similar aspect ratio\nconfig.TRAIN.ASPECT_GROUPING = True\n\n# R-CNN\n# rcnn rois batch size\nconfig.TRAIN.BATCH_ROIS = 128\nconfig.TRAIN.BATCH_ROIS_OHEM = 128\n# rcnn rois sampling params\nconfig.TRAIN.FG_FRACTION = 0.25\nconfig.TRAIN.FG_THRESH = 0.5\nconfig.TRAIN.BG_THRESH_HI = 0.5\nconfig.TRAIN.BG_THRESH_LO = 0.0\n# rcnn bounding box regression params\nconfig.TRAIN.BBOX_REGRESSION_THRESH = 0.5\nconfig.TRAIN.BBOX_WEIGHTS = np.array([1.0, 1.0, 1.0, 1.0])\n\n# RPN anchor loader\n# rpn anchors batch size\nconfig.TRAIN.RPN_BATCH_SIZE = 256\n# rpn anchors sampling params\nconfig.TRAIN.RPN_FG_FRACTION = 0.5\nconfig.TRAIN.RPN_POSITIVE_OVERLAP = 0.7\nconfig.TRAIN.RPN_NEGATIVE_OVERLAP = 0.3\nconfig.TRAIN.RPN_CLOBBER_POSITIVES = False\n# rpn bounding box regression params\nconfig.TRAIN.RPN_BBOX_WEIGHTS = (1.0, 1.0, 1.0, 1.0)\nconfig.TRAIN.RPN_POSITIVE_WEIGHT = -1.0\n\n# used for end2end training\n# RPN proposal\nconfig.TRAIN.CXX_PROPOSAL = True\nconfig.TRAIN.RPN_NMS_THRESH = 0.7\nconfig.TRAIN.RPN_PRE_NMS_TOP_N = 12000\nconfig.TRAIN.RPN_POST_NMS_TOP_N = 2000\nconfig.TRAIN.RPN_MIN_SIZE = config.network.RPN_FEAT_STRIDE\n# approximate bounding box regression\nconfig.TRAIN.BBOX_NORMALIZATION_PRECOMPUTED = True\nconfig.TRAIN.BBOX_MEANS = (0.0, 0.0, 0.0, 0.0)\nconfig.TRAIN.BBOX_STDS = (0.1, 0.1, 0.2, 0.2)\n\nconfig.TEST = edict()\n\n# R-CNN testing\n# use rpn to generate proposal\nconfig.TEST.HAS_RPN = False\n# size of images for each device\nconfig.TEST.BATCH_IMAGES = 1\n\n# RPN proposal\nconfig.TEST.CXX_PROPOSAL = True\nconfig.TEST.RPN_NMS_THRESH = 0.7\nconfig.TEST.RPN_PRE_NMS_TOP_N = 6000\nconfig.TEST.RPN_POST_NMS_TOP_N = 300\nconfig.TEST.RPN_MIN_SIZE = config.network.RPN_FEAT_STRIDE\n\n# RCNN nms\nconfig.TEST.NMS = 0.3\n\nconfig.TEST.max_per_image = 300\n\n# Test Model Epoch\nconfig.TEST.test_epoch = 0\n\n\ndef update_config(config_file):\n exp_config = None\n with open(config_file) as f:\n exp_config = edict(yaml.load(f))\n for k, v in exp_config.items():\n if k in config:\n if isinstance(v, dict):\n if k == 'TRAIN':\n if 'BBOX_WEIGHTS' in v:\n v['BBOX_WEIGHTS'] = np.array(v['BBOX_WEIGHTS'])\n elif k == 'network':\n if 'PIXEL_MEANS' in v:\n v['PIXEL_MEANS'] = np.array(v['PIXEL_MEANS'])\n for vk, vv in v.items():\n config[k][vk] = vv\n else:\n if k == 'SCALES':\n config[k][0] = (tuple(v))\n else:\n config[k] = v\n else:\n raise ValueError(\"key must exist in config.py\")\n" }, { "path": "rfcn/core/DataParallelExecutorGroup.py", "content": "# --------------------------------------------------------\n# Deep Feature Flow\n# Copyright (c) 2017 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Modified by Yuwen Xiong\n# --------------------------------------------------------\n# Based on:\n# MX-RCNN\n# Copyright (c) 2016 by Contributors\n# Licence under The Apache 2.0 License\n# https://github.com/ijkguo/mx-rcnn/\n# --------------------------------------------------------\n\nimport logging\nimport numpy as np\n\nfrom mxnet import context as ctx\nfrom mxnet import ndarray as nd\nfrom mxnet.io import DataDesc\nfrom mxnet.executor_manager import _split_input_slice\n\n\n\ndef _load_general(data, targets, major_axis):\n \"\"\"Load a list of arrays into a list of arrays specified by slices\"\"\"\n for d_src, d_targets in zip(data, targets):\n if isinstance(d_targets, nd.NDArray):\n d_src.copyto(d_targets)\n elif isinstance(d_src, (list, tuple)):\n for src, dst in zip(d_src, d_targets):\n src.copyto(dst)\n else:\n raise NotImplementedError\n\n\n\ndef _load_data(batch, targets, major_axis):\n \"\"\"Load data into sliced arrays\"\"\"\n _load_general(batch.data, targets, major_axis)\n\n\ndef _load_label(batch, targets, major_axis):\n \"\"\"Load label into sliced arrays\"\"\"\n _load_general(batch.label, targets, major_axis)\n\n\ndef _merge_multi_context(outputs, major_axis):\n \"\"\"Merge outputs that lives on multiple context into one, so that they look\n like living on one context.\n \"\"\"\n rets = []\n for tensors, axis in zip(outputs, major_axis):\n if axis >= 0:\n rets.append(nd.concatenate(tensors, axis=axis, always_copy=False))\n else:\n # negative axis means the there is no batch_size axis, and all the\n # results should be the same on each device. We simply take the\n # first one, without checking they are actually the same\n rets.append(tensors[0])\n return rets\n\n\n\nclass DataParallelExecutorGroup(object):\n \"\"\"DataParallelExecutorGroup is a group of executors that lives on a group of devices.\n This is a helper class used to implement data parallelization. Each mini-batch will\n be split and run on the devices.\n\n Parameters\n ----------\n symbol : Symbol\n The common symbolic computation graph for all executors.\n contexts : list\n A list of contexts.\n workload : list\n If not `None`, could be a list of numbers that specify the workload to be assigned\n to different context. Larger number indicate heavier workload.\n data_shapes : list\n Should be a list of (name, shape) tuples, for the shapes of data. Note the order is\n important and should be the same as the order that the `DataIter` provide the data.\n label_shapes : list\n Should be a list of (name, shape) tuples, for the shapes of label. Note the order is\n important and should be the same as the order that the `DataIter` provide the label.\n param_names : list\n A list of strings, indicating the names of parameters (e.g. weights, filters, etc.)\n in the computation graph.\n for_training : bool\n Indicate whether the executors should be bind for training. When not doing training,\n the memory for gradients will not be allocated.\n inputs_need_grad : bool\n Indicate whether the gradients for the input data should be computed. This is currently\n not used. It will be useful for implementing composition of modules.\n shared_group : DataParallelExecutorGroup\n Default is `None`. This is used in bucketing. When not `None`, it should be a executor\n group corresponding to a different bucket. In other words, it will correspond to a different\n symbol but with the same set of parameters (e.g. unrolled RNNs with different lengths).\n In this case, many memory will be shared.\n logger : Logger\n Default is `logging`.\n fixed_param_names: list of str\n Indicate parameters to be fixed during training. Parameters in this list will not allocate\n space for gradient, nor do gradient calculation.\n grad_req : str, list of str, dict of str to str\n Requirement for gradient accumulation. Can be 'write', 'add', or 'null'\n (default to 'write').\n Can be specified globally (str) or for each argument (list, dict).\n \"\"\"\n def __init__(self, symbol, contexts, workload, data_shapes, label_shapes, param_names,\n for_training, inputs_need_grad, shared_group=None, logger=logging,\n fixed_param_names=None, grad_req='write', state_names=None):\n self.param_names = param_names\n self.arg_names = symbol.list_arguments()\n self.aux_names = symbol.list_auxiliary_states()\n\n self.symbol = symbol\n self.contexts = contexts\n self.workload = workload\n\n self.for_training = for_training\n self.inputs_need_grad = inputs_need_grad\n\n self.logger = logger\n #In the future we should have a better way to profile memory per device (haibin)\n # self._total_exec_bytes = 0\n self.fixed_param_names = fixed_param_names\n if self.fixed_param_names is None:\n self.fixed_param_names = []\n\n self.state_names = state_names\n if self.state_names is None:\n self.state_names = []\n\n if not for_training:\n grad_req = 'null'\n\n # data_shapes = [x if isinstance(x, DataDesc) else DataDesc(*x) for x in data_shapes]\n # if label_shapes is not None:\n # label_shapes = [x if isinstance(x, DataDesc) else DataDesc(*x) for x in label_shapes]\n\n data_names = [x.name for x in data_shapes[0]]\n\n if isinstance(grad_req, str):\n self.grad_req = {}\n for k in self.arg_names:\n if k in self.param_names:\n self.grad_req[k] = 'null' if k in self.fixed_param_names else grad_req\n elif k in data_names:\n self.grad_req[k] = grad_req if self.inputs_need_grad else 'null'\n else:\n self.grad_req[k] = 'null'\n elif isinstance(grad_req, (list, tuple)):\n assert len(grad_req) == len(self.arg_names)\n self.grad_req = dict(zip(self.arg_names, grad_req))\n elif isinstance(grad_req, dict):\n self.grad_req = {}\n for k in self.arg_names:\n if k in self.param_names:\n self.grad_req[k] = 'null' if k in self.fixed_param_names else 'write'\n elif k in data_names:\n self.grad_req[k] = 'write' if self.inputs_need_grad else 'null'\n else:\n self.grad_req[k] = 'null'\n self.grad_req.update(grad_req)\n else:\n raise ValueError(\"grad_req must be one of str, list, tuple, or dict.\")\n\n if shared_group is not None:\n self.shared_data_arrays = shared_group.shared_data_arrays\n else:\n self.shared_data_arrays = [{} for _ in contexts]\n\n # initialize some instance variables\n self.batch_size = len(data_shapes)\n self.slices = None\n self.execs = []\n self._default_execs = None\n self.data_arrays = None\n self.label_arrays = None\n self.param_arrays = None\n self.state_arrays = None\n self.grad_arrays = None\n self.aux_arrays = None\n self.input_grad_arrays = None\n\n self.data_shapes = None\n self.label_shapes = None\n self.data_layouts = None\n self.label_layouts = None\n self.output_layouts = [DataDesc.get_batch_axis(self.symbol[name].attr('__layout__'))\n for name in self.symbol.list_outputs()]\n self.bind_exec(data_shapes, label_shapes, shared_group)\n\n def decide_slices(self, data_shapes):\n \"\"\"Decide the slices for each context according to the workload.\n\n Parameters\n ----------\n data_shapes : list\n list of (name, shape) specifying the shapes for the input data or label.\n \"\"\"\n assert len(data_shapes) > 0\n major_axis = [DataDesc.get_batch_axis(x.layout) for x in data_shapes]\n\n for (name, shape), axis in zip(data_shapes, major_axis):\n if axis == -1:\n continue\n\n batch_size = shape[axis]\n if self.batch_size is not None:\n assert batch_size == self.batch_size, (\"all data must have the same batch size: \"\n + (\"batch_size = %d, but \" % self.batch_size)\n + (\"%s has shape %s\" % (name, shape)))\n else:\n self.batch_size = batch_size\n self.slices = _split_input_slice(self.batch_size, self.workload)\n\n return major_axis\n\n def _collect_arrays(self):\n \"\"\"Collect internal arrays from executors.\"\"\"\n # convenient data structures\n self.data_arrays = [[e.arg_dict[name] for name, _ in self.data_shapes[0]] for e in self.execs]\n\n self.state_arrays = [[e.arg_dict[name] for e in self.execs]\n for name in self.state_names]\n\n if self.label_shapes is not None:\n self.label_arrays = [[e.arg_dict[name] for name, _ in self.label_shapes[0]] for e in self.execs]\n else:\n self.label_arrays = None\n\n self.param_arrays = [[exec_.arg_arrays[i] for exec_ in self.execs]\n for i, name in enumerate(self.arg_names)\n if name in self.param_names]\n if self.for_training:\n self.grad_arrays = [[exec_.grad_arrays[i] for exec_ in self.execs]\n for i, name in enumerate(self.arg_names)\n if name in self.param_names]\n else:\n self.grad_arrays = None\n\n data_names = [x[0] for x in self.data_shapes]\n if self.inputs_need_grad:\n self.input_grad_arrays = [[exec_.grad_arrays[i] for exec_ in self.execs]\n for i, name in enumerate(self.arg_names)\n if name in data_names]\n else:\n self.input_grad_arrays = None\n\n self.aux_arrays = [[exec_.aux_arrays[i] for exec_ in self.execs]\n for i in range(len(self.aux_names))]\n\n def bind_exec(self, data_shapes, label_shapes, shared_group=None, reshape=False):\n \"\"\"Bind executors on their respective devices.\n\n Parameters\n ----------\n data_shapes : list\n label_shapes : list\n shared_group : DataParallelExecutorGroup\n reshape : bool\n \"\"\"\n assert reshape or not self.execs\n\n for i in range(len(self.contexts)):\n data_shapes_i = data_shapes[i]\n if label_shapes is not None:\n label_shapes_i = label_shapes[i]\n else:\n label_shapes_i = []\n\n if reshape:\n self.execs[i] = self._default_execs[i].reshape(\n allow_up_sizing=True, **dict(data_shapes_i + label_shapes_i))\n else:\n self.execs.append(self._bind_ith_exec(i, data_shapes_i, label_shapes_i,\n shared_group))\n\n self.data_shapes = data_shapes\n self.label_shapes = label_shapes\n self._collect_arrays()\n\n def reshape(self, data_shapes, label_shapes):\n \"\"\"Reshape executors.\n\n Parameters\n ----------\n data_shapes : list\n label_shapes : list\n \"\"\"\n if self._default_execs is None:\n self._default_execs = [i for i in self.execs]\n for i in range(len(self.contexts)):\n self.execs[i] = self._default_execs[i].reshape(\n allow_up_sizing=True, **dict(data_shapes[i] + (label_shapes[i] if label_shapes is not None else []))\n )\n self.data_shapes = data_shapes\n self.label_shapes = label_shapes\n self._collect_arrays()\n\n\n def set_params(self, arg_params, aux_params):\n \"\"\"Assign, i.e. copy parameters to all the executors.\n\n Parameters\n ----------\n arg_params : dict\n A dictionary of name to `NDArray` parameter mapping.\n aux_params : dict\n A dictionary of name to `NDArray` auxiliary variable mapping.\n \"\"\"\n for exec_ in self.execs:\n exec_.copy_params_from(arg_params, aux_params)\n\n def get_params(self, arg_params, aux_params):\n \"\"\" Copy data from each executor to `arg_params` and `aux_params`.\n\n Parameters\n ----------\n arg_params : list of NDArray\n target parameter arrays\n aux_params : list of NDArray\n target aux arrays\n\n Notes\n -----\n - This function will inplace update the NDArrays in arg_params and aux_params.\n \"\"\"\n for name, block in zip(self.param_names, self.param_arrays):\n weight = sum(w.copyto(ctx.cpu()) for w in block) / len(block)\n weight.astype(arg_params[name].dtype).copyto(arg_params[name])\n for name, block in zip(self.aux_names, self.aux_arrays):\n weight = sum(w.copyto(ctx.cpu()) for w in block) / len(block)\n weight.astype(aux_params[name].dtype).copyto(aux_params[name])\n\n def forward(self, data_batch, is_train=None):\n \"\"\"Split `data_batch` according to workload and run forward on each devices.\n\n Parameters\n ----------\n data_batch : DataBatch\n Or could be any object implementing similar interface.\n is_train : bool\n The hint for the backend, indicating whether we are during training phase.\n Default is `None`, then the value `self.for_training` will be used.\n Returns\n -------\n\n \"\"\"\n _load_data(data_batch, self.data_arrays, self.data_layouts)\n if is_train is None:\n is_train = self.for_training\n\n if self.label_arrays is not None:\n assert not is_train or data_batch.label\n if data_batch.label:\n _load_label(data_batch, self.label_arrays, self.label_layouts)\n\n for exec_ in self.execs:\n exec_.forward(is_train=is_train)\n\n\n def get_outputs(self, merge_multi_context=True):\n \"\"\"Get outputs of the previous forward computation.\n\n Parameters\n ----------\n merge_multi_context : bool\n Default is `True`. In the case when data-parallelism is used, the outputs\n will be collected from multiple devices. A `True` value indicate that we\n should merge the collected results so that they look like from a single\n executor.\n\n Returns\n -------\n If `merge_multi_context` is `True`, it is like `[out1, out2]`. Otherwise, it\n is like `[[out1_dev1, out1_dev2], [out2_dev1, out2_dev2]]`. All the output\n elements are `NDArray`.\n \"\"\"\n outputs = [[exec_.outputs[i] for exec_ in self.execs]\n for i in range(len(self.execs[0].outputs))]\n if merge_multi_context:\n outputs = _merge_multi_context(outputs, self.output_layouts)\n return outputs\n\n def get_states(self, merge_multi_context=True):\n \"\"\"Get states from all devices\n\n Parameters\n ----------\n merge_multi_context : bool\n Default is `True`. In the case when data-parallelism is used, the states\n will be collected from multiple devices. A `True` value indicate that we\n should merge the collected results so that they look like from a single\n executor.\n\n Returns\n -------\n If `merge_multi_context` is `True`, it is like `[out1, out2]`. Otherwise, it\n is like `[[out1_dev1, out1_dev2], [out2_dev1, out2_dev2]]`. All the output\n elements are `NDArray`.\n \"\"\"\n assert not merge_multi_context, \\\n \"merge_multi_context=True is not supported for get_states yet.\"\n return self.state_arrays\n\n def set_states(self, states=None, value=None):\n \"\"\"Set value for states. Only one of states & value can be specified.\n\n Parameters\n ----------\n states : list of list of NDArrays\n source states arrays formatted like [[state1_dev1, state1_dev2],\n [state2_dev1, state2_dev2]].\n value : number\n a single scalar value for all state arrays.\n \"\"\"\n if states is not None:\n assert value is None, \"Only one of states & value can be specified.\"\n _load_general(states, self.state_arrays, (0,)*len(states))\n else:\n assert value is not None, \"At least one of states & value must be specified.\"\n assert states is None, \"Only one of states & value can be specified.\"\n for d_dst in self.state_arrays:\n for dst in d_dst:\n dst[:] = value\n\n def get_input_grads(self, merge_multi_context=True):\n \"\"\"Get the gradients with respect to the inputs of the module.\n\n Parameters\n ----------\n merge_multi_context : bool\n Default is `True`. In the case when data-parallelism is used, the outputs\n will be collected from multiple devices. A `True` value indicate that we\n should merge the collected results so that they look like from a single\n executor.\n\n Returns\n -------\n If `merge_multi_context` is `True`, it is like `[grad1, grad2]`. Otherwise, it\n is like `[[grad1_dev1, grad1_dev2], [grad2_dev1, grad2_dev2]]`. All the output\n elements are `NDArray`.\n \"\"\"\n assert self.inputs_need_grad\n if merge_multi_context:\n return _merge_multi_context(self.input_grad_arrays, self.data_layouts)\n return self.input_grad_arrays\n\n def backward(self, out_grads=None):\n \"\"\"Run backward on all devices. A backward should be called after\n a call to the forward function. Backward cannot be called unless\n `self.for_training` is `True`.\n\n Parameters\n ----------\n out_grads : NDArray or list of NDArray, optional\n Gradient on the outputs to be propagated back.\n This parameter is only needed when bind is called\n on outputs that are not a loss function.\n \"\"\"\n assert self.for_training, 're-bind with for_training=True to run backward'\n if out_grads is None:\n out_grads = []\n\n for i, exec_ in enumerate(self.execs):\n out_grads_slice = []\n exec_.backward(out_grads=out_grads_slice)\n\n def update_metric(self, eval_metric, labels):\n \"\"\"Accumulate the performance according to `eval_metric` on all devices.\n\n Parameters\n ----------\n eval_metric : EvalMetric\n The metric used for evaluation.\n labels : list of NDArray\n Typically comes from `label` of a `DataBatch`.\n \"\"\"\n for texec, labels in zip(self.execs, labels):\n eval_metric.update(labels, texec.outputs)\n\n def _bind_ith_exec(self, i, data_shapes, label_shapes, shared_group):\n \"\"\"Internal utility function to bind the i-th executor.\n \"\"\"\n shared_exec = None if shared_group is None else shared_group.execs[i]\n context = self.contexts[i]\n shared_data_arrays = self.shared_data_arrays[i]\n\n input_shapes = dict(data_shapes)\n if label_shapes is not None:\n input_shapes.update(dict(label_shapes))\n\n arg_shapes, _, aux_shapes = self.symbol.infer_shape(**input_shapes)\n assert arg_shapes is not None, \"shape inference failed\"\n\n input_types = {x.name: x.dtype for x in data_shapes}\n if label_shapes is not None:\n input_types.update({x.name: x.dtype for x in label_shapes})\n arg_types, _, aux_types = self.symbol.infer_type(**input_types)\n assert arg_types is not None, \"type inference failed\"\n\n arg_arrays = []\n grad_arrays = {} if self.for_training else None\n\n def _get_or_reshape(name, shared_data_arrays, arg_shape, arg_type, context, logger):\n \"\"\"Internal helper to get a memory block or re-use by re-shaping\"\"\"\n if name in shared_data_arrays:\n arg_arr = shared_data_arrays[name]\n\n if np.prod(arg_arr.shape) >= np.prod(arg_shape):\n # nice, we can directly re-use this data blob\n assert arg_arr.dtype == arg_type\n arg_arr = arg_arr.reshape(arg_shape)\n else:\n logger.warning(('bucketing: data \"%s\" has a shape %s' % (name, arg_shape)) +\n (', which is larger than already allocated ') +\n ('shape %s' % (arg_arr.shape,)) +\n ('. Need to re-allocate. Consider putting ') +\n ('default_bucket_key to') +\n (' be the bucket taking the largest input for better ') +\n ('memory sharing.'))\n arg_arr = nd.zeros(arg_shape, context, dtype=arg_type)\n\n # replace existing shared array because the new one is bigger\n shared_data_arrays[name] = arg_arr\n else:\n arg_arr = nd.zeros(arg_shape, context, dtype=arg_type)\n shared_data_arrays[name] = arg_arr\n\n return arg_arr\n\n # create or borrow arguments and gradients\n for j in range(len(self.arg_names)):\n name = self.arg_names[j]\n if name in self.param_names: # model parameters\n if shared_exec is None:\n arg_arr = nd.zeros(arg_shapes[j], context, dtype=arg_types[j])\n if self.grad_req[name] != 'null':\n grad_arr = nd.zeros(arg_shapes[j], context, dtype=arg_types[j])\n grad_arrays[name] = grad_arr\n else:\n arg_arr = shared_exec.arg_dict[name]\n assert arg_arr.shape == arg_shapes[j]\n assert arg_arr.dtype == arg_types[j]\n if self.grad_req[name] != 'null':\n grad_arrays[name] = shared_exec.grad_dict[name]\n else: # data, label, or states\n arg_arr = _get_or_reshape(name, shared_data_arrays, arg_shapes[j], arg_types[j],\n context, self.logger)\n\n # data might also need grad if inputs_need_grad is True\n if self.grad_req[name] != 'null':\n grad_arrays[name] = _get_or_reshape('grad of ' + name, shared_data_arrays,\n arg_shapes[j], arg_types[j], context,\n self.logger)\n\n arg_arrays.append(arg_arr)\n\n # create or borrow aux variables\n if shared_exec is None:\n aux_arrays = [nd.zeros(s, context, dtype=t) for s, t in zip(aux_shapes, aux_types)]\n else:\n for j, arr in enumerate(shared_exec.aux_arrays):\n assert aux_shapes[j] == arr.shape\n assert aux_types[j] == arr.dtype\n aux_arrays = shared_exec.aux_arrays[:]\n\n executor = self.symbol.bind(ctx=context, args=arg_arrays,\n args_grad=grad_arrays, aux_states=aux_arrays,\n grad_req=self.grad_req, shared_exec=shared_exec)\n # Get the total bytes allocated for this executor\n return executor\n\n def _sliced_shape(self, shapes, i, major_axis):\n \"\"\"Get the sliced shapes for the i-th executor.\n\n Parameters\n ----------\n shapes : list of (str, tuple)\n The original (name, shape) pairs.\n i : int\n Which executor we are dealing with.\n \"\"\"\n sliced_shapes = []\n for desc, axis in zip(shapes, major_axis):\n shape = list(desc.shape)\n if axis >= 0:\n shape[axis] = self.slices[i].stop - self.slices[i].start\n sliced_shapes.append(DataDesc(desc.name, tuple(shape), desc.dtype, desc.layout))\n return sliced_shapes\n\n def install_monitor(self, mon):\n \"\"\"Install monitor on all executors\"\"\"\n for exe in self.execs:\n mon.install(exe)\n" }, { "path": "rfcn/core/__init__.py", "content": "" }, { "path": "rfcn/core/callback.py", "content": "# --------------------------------------------------------\n# Deep Feature Flow\n# Copyright (c) 2017 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Modified by Yuwen Xiong\n# --------------------------------------------------------\n# Based on:\n# MX-RCNN\n# Copyright (c) 2016 by Contributors\n# Licence under The Apache 2.0 License\n# https://github.com/ijkguo/mx-rcnn/\n# --------------------------------------------------------\n\nimport time\nimport logging\nimport mxnet as mx\n\n\nclass Speedometer(object):\n def __init__(self, batch_size, frequent=50):\n self.batch_size = batch_size\n self.frequent = frequent\n self.init = False\n self.tic = 0\n self.last_count = 0\n\n def __call__(self, param):\n \"\"\"Callback to Show speed.\"\"\"\n count = param.nbatch\n if self.last_count > count:\n self.init = False\n self.last_count = count\n\n if self.init:\n if count % self.frequent == 0:\n speed = self.frequent * self.batch_size / (time.time() - self.tic)\n s = ''\n if param.eval_metric is not None:\n name, value = param.eval_metric.get()\n s = \"Epoch[%d] Batch [%d]\\tSpeed: %.2f samples/sec\\tTrain-\" % (param.epoch, count, speed)\n for n, v in zip(name, value):\n s += \"%s=%f,\\t\" % (n, v)\n else:\n s = \"Iter[%d] Batch [%d]\\tSpeed: %.2f samples/sec\" % (param.epoch, count, speed)\n\n logging.info(s)\n print(s)\n self.tic = time.time()\n else:\n self.init = True\n self.tic = time.time()\n\n\ndef do_checkpoint(prefix, means, stds):\n def _callback(iter_no, sym, arg, aux):\n weight = arg['rfcn_bbox_weight']\n bias = arg['rfcn_bbox_bias']\n repeat = bias.shape[0] / means.shape[0]\n\n arg['rfcn_bbox_weight_test'] = weight * mx.nd.repeat(mx.nd.array(stds), repeats=repeat).reshape((bias.shape[0], 1, 1, 1))\n arg['rfcn_bbox_bias_test'] = arg['rfcn_bbox_bias'] * mx.nd.repeat(mx.nd.array(stds), repeats=repeat) + mx.nd.repeat(mx.nd.array(means), repeats=repeat)\n mx.model.save_checkpoint(prefix, iter_no + 1, sym, arg, aux)\n arg.pop('rfcn_bbox_weight_test')\n arg.pop('rfcn_bbox_bias_test')\n return _callback" }, { "path": "rfcn/core/loader.py", "content": "# --------------------------------------------------------\n# Deep Feature Flow\n# Copyright (c) 2017 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Modified by Yuwen Xiong\n# --------------------------------------------------------\n# Based on:\n# MX-RCNN\n# Copyright (c) 2016 by Contributors\n# Licence under The Apache 2.0 License\n# https://github.com/ijkguo/mx-rcnn/\n# --------------------------------------------------------\n\nimport numpy as np\nimport mxnet as mx\nfrom mxnet.executor_manager import _split_input_slice\n\nfrom config.config import config\nfrom utils.image import tensor_vstack\nfrom rpn.rpn import get_rpn_testbatch, get_rpn_batch, assign_anchor\nfrom rcnn import get_rcnn_testbatch, get_rcnn_batch\n\nclass TestLoader(mx.io.DataIter):\n def __init__(self, roidb, config, batch_size=1, shuffle=False,\n has_rpn=False):\n super(TestLoader, self).__init__()\n\n # save parameters as properties\n self.cfg = config\n self.roidb = roidb\n self.batch_size = batch_size\n self.shuffle = shuffle\n self.has_rpn = has_rpn\n\n # infer properties from roidb\n self.size = len(self.roidb)\n self.index = np.arange(self.size)\n\n # decide data and label names (only for training)\n if has_rpn:\n self.data_name = ['data', 'im_info']\n else:\n self.data_name = ['data', 'rois']\n self.label_name = None\n\n # status variable for synchronization between get_data and get_label\n self.cur = 0\n self.data = None\n self.label = []\n self.im_info = None\n\n # get first batch to fill in provide_data and provide_label\n self.reset()\n self.get_batch()\n\n @property\n def provide_data(self):\n return [[(k, v.shape) for k, v in zip(self.data_name, idata)] for idata in self.data]\n\n @property\n def provide_label(self):\n return [None for _ in range(len(self.data))]\n\n @property\n def provide_data_single(self):\n return [(k, v.shape) for k, v in zip(self.data_name, self.data[0])]\n\n @property\n def provide_label_single(self):\n return None\n\n def reset(self):\n self.cur = 0\n if self.shuffle:\n np.random.shuffle(self.index)\n\n def iter_next(self):\n return self.cur < self.size\n\n def next(self):\n if self.iter_next():\n self.get_batch()\n self.cur += self.batch_size\n return self.im_info, mx.io.DataBatch(data=self.data, label=self.label,\n pad=self.getpad(), index=self.getindex(),\n provide_data=self.provide_data, provide_label=self.provide_label)\n else:\n raise StopIteration\n\n def getindex(self):\n return self.cur / self.batch_size\n\n def getpad(self):\n if self.cur + self.batch_size > self.size:\n return self.cur + self.batch_size - self.size\n else:\n return 0\n\n def get_batch(self):\n cur_from = self.cur\n cur_to = min(cur_from + self.batch_size, self.size)\n roidb = [self.roidb[self.index[i]] for i in range(cur_from, cur_to)]\n if self.has_rpn:\n data, label, im_info = get_rpn_testbatch(roidb, self.cfg)\n else:\n data, label, im_info = get_rcnn_testbatch(roidb, self.cfg)\n self.data = [[mx.nd.array(idata[name]) for name in self.data_name] for idata in data]\n self.im_info = im_info\n\nclass AnchorLoader(mx.io.DataIter):\n\n def __init__(self, feat_sym, roidb, cfg, batch_size=1, shuffle=False, ctx=None, work_load_list=None,\n feat_stride=16, anchor_scales=(8, 16, 32), anchor_ratios=(0.5, 1, 2), allowed_border=0,\n aspect_grouping=False, normalize_target=False, bbox_mean=(0.0, 0.0, 0.0, 0.0),\n bbox_std=(0.1, 0.1, 0.4, 0.4)):\n \"\"\"\n This Iter will provide roi data to Fast R-CNN network\n :param feat_sym: to infer shape of assign_output\n :param roidb: must be preprocessed\n :param batch_size: must divide BATCH_SIZE(128)\n :param shuffle: bool\n :param ctx: list of contexts\n :param work_load_list: list of work load\n :param aspect_grouping: group images with similar aspects\n :param normalize_target: normalize rpn target\n :param bbox_mean: anchor target mean\n :param bbox_std: anchor target std\n :return: AnchorLoader\n \"\"\"\n super(AnchorLoader, self).__init__()\n\n # save parameters as properties\n self.feat_sym = feat_sym\n self.roidb = roidb\n self.cfg = cfg\n self.batch_size = batch_size\n self.shuffle = shuffle\n self.ctx = ctx\n if self.ctx is None:\n self.ctx = [mx.cpu()]\n self.work_load_list = work_load_list\n self.feat_stride = feat_stride\n self.anchor_scales = anchor_scales\n self.anchor_ratios = anchor_ratios\n self.allowed_border = allowed_border\n self.aspect_grouping = aspect_grouping\n self.normalize_target = normalize_target\n self.bbox_mean = bbox_mean\n self.bbox_std = bbox_std\n\n # infer properties from roidb\n self.size = len(roidb)\n self.index = np.arange(self.size)\n\n # decide data and label names\n if config.TRAIN.END2END:\n self.data_name = ['data', 'im_info', 'gt_boxes']\n else:\n self.data_name = ['data']\n self.label_name = ['label', 'bbox_target', 'bbox_weight']\n\n # status variable for synchronization between get_data and get_label\n self.cur = 0\n self.batch = None\n self.data = None\n self.label = None\n\n # get first batch to fill in provide_data and provide_label\n self.reset()\n self.get_batch_individual()\n\n @property\n def provide_data(self):\n return [[(k, v.shape) for k, v in zip(self.data_name, self.data[i])] for i in xrange(len(self.data))]\n\n @property\n def provide_label(self):\n return [[(k, v.shape) for k, v in zip(self.label_name, self.label[i])] for i in xrange(len(self.data))]\n\n @property\n def provide_data_single(self):\n return [(k, v.shape) for k, v in zip(self.data_name, self.data[0])]\n\n @property\n def provide_label_single(self):\n return [(k, v.shape) for k, v in zip(self.label_name, self.label[0])]\n\n def reset(self):\n self.cur = 0\n if self.shuffle:\n if self.aspect_grouping:\n widths = np.array([r['width'] for r in self.roidb])\n heights = np.array([r['height'] for r in self.roidb])\n horz = (widths >= heights)\n vert = np.logical_not(horz)\n horz_inds = np.where(horz)[0]\n vert_inds = np.where(vert)[0]\n inds = np.hstack((np.random.permutation(horz_inds), np.random.permutation(vert_inds)))\n extra = inds.shape[0] % self.batch_size\n inds_ = np.reshape(inds[:-extra], (-1, self.batch_size))\n row_perm = np.random.permutation(np.arange(inds_.shape[0]))\n inds[:-extra] = np.reshape(inds_[row_perm, :], (-1,))\n self.index = inds\n else:\n np.random.shuffle(self.index)\n\n def iter_next(self):\n return self.cur + self.batch_size <= self.size\n\n def next(self):\n if self.iter_next():\n self.get_batch_individual()\n self.cur += self.batch_size\n return mx.io.DataBatch(data=self.data, label=self.label,\n pad=self.getpad(), index=self.getindex(),\n provide_data=self.provide_data, provide_label=self.provide_label)\n else:\n raise StopIteration\n\n def getindex(self):\n return self.cur / self.batch_size\n\n def getpad(self):\n if self.cur + self.batch_size > self.size:\n return self.cur + self.batch_size - self.size\n else:\n return 0\n\n def infer_shape(self, max_data_shape=None, max_label_shape=None):\n \"\"\" Return maximum data and label shape for single gpu \"\"\"\n if max_data_shape is None:\n max_data_shape = []\n if max_label_shape is None:\n max_label_shape = []\n max_shapes = dict(max_data_shape + max_label_shape)\n input_batch_size = max_shapes['data'][0]\n im_info = [[max_shapes['data'][2], max_shapes['data'][3], 1.0]]\n _, feat_shape, _ = self.feat_sym.infer_shape(**max_shapes)\n label = assign_anchor(feat_shape[0], np.zeros((0, 5)), im_info, self.cfg,\n self.feat_stride, self.anchor_scales, self.anchor_ratios, self.allowed_border,\n self.normalize_target, self.bbox_mean, self.bbox_std)\n label = [label[k] for k in self.label_name]\n label_shape = [(k, tuple([input_batch_size] + list(v.shape[1:]))) for k, v in zip(self.label_name, label)]\n return max_data_shape, label_shape\n\n def get_batch(self):\n # slice roidb\n cur_from = self.cur\n cur_to = min(cur_from + self.batch_size, self.size)\n roidb = [self.roidb[self.index[i]] for i in range(cur_from, cur_to)]\n\n # decide multi device slice\n work_load_list = self.work_load_list\n ctx = self.ctx\n if work_load_list is None:\n work_load_list = [1] * len(ctx)\n assert isinstance(work_load_list, list) and len(work_load_list) == len(ctx), \\\n \"Invalid settings for work load. \"\n slices = _split_input_slice(self.batch_size, work_load_list)\n\n # get testing data for multigpu\n data_list = []\n label_list = []\n for islice in slices:\n iroidb = [roidb[i] for i in range(islice.start, islice.stop)]\n data, label = get_rpn_batch(iroidb, self.cfg)\n data_list.append(data)\n label_list.append(label)\n\n # pad data first and then assign anchor (read label)\n data_tensor = tensor_vstack([batch['data'] for batch in data_list])\n for data, data_pad in zip(data_list, data_tensor):\n data['data'] = data_pad[np.newaxis, :]\n\n new_label_list = []\n for data, label in zip(data_list, label_list):\n # infer label shape\n data_shape = {k: v.shape for k, v in data.items()}\n del data_shape['im_info']\n _, feat_shape, _ = self.feat_sym.infer_shape(**data_shape)\n feat_shape = [int(i) for i in feat_shape[0]]\n\n # add gt_boxes to data for e2e\n data['gt_boxes'] = label['gt_boxes'][np.newaxis, :, :]\n\n # assign anchor for label\n label = assign_anchor(feat_shape, label['gt_boxes'], data['im_info'], self.cfg,\n self.feat_stride, self.anchor_scales,\n self.anchor_ratios, self.allowed_border,\n self.normalize_target, self.bbox_mean, self.bbox_std)\n new_label_list.append(label)\n\n all_data = dict()\n for key in self.data_name:\n all_data[key] = tensor_vstack([batch[key] for batch in data_list])\n\n all_label = dict()\n for key in self.label_name:\n pad = -1 if key == 'label' else 0\n all_label[key] = tensor_vstack([batch[key] for batch in new_label_list], pad=pad)\n\n self.data = [mx.nd.array(all_data[key]) for key in self.data_name]\n self.label = [mx.nd.array(all_label[key]) for key in self.label_name]\n\n def get_batch_individual(self):\n cur_from = self.cur\n cur_to = min(cur_from + self.batch_size, self.size)\n roidb = [self.roidb[self.index[i]] for i in range(cur_from, cur_to)]\n # decide multi device slice\n work_load_list = self.work_load_list\n ctx = self.ctx\n if work_load_list is None:\n work_load_list = [1] * len(ctx)\n assert isinstance(work_load_list, list) and len(work_load_list) == len(ctx), \\\n \"Invalid settings for work load. \"\n slices = _split_input_slice(self.batch_size, work_load_list)\n rst = []\n for idx, islice in enumerate(slices):\n iroidb = [roidb[i] for i in range(islice.start, islice.stop)]\n rst.append(self.parfetch(iroidb))\n all_data = [_['data'] for _ in rst]\n all_label = [_['label'] for _ in rst]\n self.data = [[mx.nd.array(data[key]) for key in self.data_name] for data in all_data]\n self.label = [[mx.nd.array(label[key]) for key in self.label_name] for label in all_label]\n\n def parfetch(self, iroidb):\n # get testing data for multigpu\n data, label = get_rpn_batch(iroidb, self.cfg)\n data_shape = {k: v.shape for k, v in data.items()}\n del data_shape['im_info']\n _, feat_shape, _ = self.feat_sym.infer_shape(**data_shape)\n feat_shape = [int(i) for i in feat_shape[0]]\n\n # add gt_boxes to data for e2e\n data['gt_boxes'] = label['gt_boxes'][np.newaxis, :, :]\n\n # assign anchor for label\n label = assign_anchor(feat_shape, label['gt_boxes'], data['im_info'], self.cfg,\n self.feat_stride, self.anchor_scales,\n self.anchor_ratios, self.allowed_border,\n self.normalize_target, self.bbox_mean, self.bbox_std)\n return {'data': data, 'label': label}\n\n" }, { "path": "rfcn/core/metric.py", "content": "# --------------------------------------------------------\n# Deep Feature Flow\n# Copyright (c) 2017 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Modified by Yuwen Xiong\n# --------------------------------------------------------\n# Based on:\n# MX-RCNN\n# Copyright (c) 2016 by Contributors\n# Licence under The Apache 2.0 License\n# https://github.com/ijkguo/mx-rcnn/\n# --------------------------------------------------------\n\nimport mxnet as mx\nimport numpy as np\n\n\ndef get_rpn_names():\n pred = ['rpn_cls_prob', 'rpn_bbox_loss']\n label = ['rpn_label', 'rpn_bbox_target', 'rpn_bbox_weight']\n return pred, label\n\n\ndef get_rcnn_names(cfg):\n pred = ['rcnn_cls_prob', 'rcnn_bbox_loss']\n label = ['rcnn_label', 'rcnn_bbox_target', 'rcnn_bbox_weight']\n if cfg.TRAIN.ENABLE_OHEM or cfg.TRAIN.END2END:\n pred.append('rcnn_label')\n if cfg.TRAIN.END2END:\n rpn_pred, rpn_label = get_rpn_names()\n pred = rpn_pred + pred\n label = rpn_label\n return pred, label\n\n\nclass RPNAccMetric(mx.metric.EvalMetric):\n def __init__(self):\n super(RPNAccMetric, self).__init__('RPNAcc')\n self.pred, self.label = get_rpn_names()\n\n def update(self, labels, preds):\n pred = preds[self.pred.index('rpn_cls_prob')]\n label = labels[self.label.index('rpn_label')]\n\n # pred (b, c, p) or (b, c, h, w)\n pred_label = mx.ndarray.argmax_channel(pred).asnumpy().astype('int32')\n pred_label = pred_label.reshape((pred_label.shape[0], -1))\n # label (b, p)\n label = label.asnumpy().astype('int32')\n\n # filter with keep_inds\n keep_inds = np.where(label != -1)\n pred_label = pred_label[keep_inds]\n label = label[keep_inds]\n\n self.sum_metric += np.sum(pred_label.flat == label.flat)\n self.num_inst += len(pred_label.flat)\n\n\nclass RCNNAccMetric(mx.metric.EvalMetric):\n def __init__(self, cfg):\n super(RCNNAccMetric, self).__init__('RCNNAcc')\n self.e2e = cfg.TRAIN.END2END\n self.ohem = cfg.TRAIN.ENABLE_OHEM\n self.pred, self.label = get_rcnn_names(cfg)\n\n def update(self, labels, preds):\n pred = preds[self.pred.index('rcnn_cls_prob')]\n if self.ohem or self.e2e:\n label = preds[self.pred.index('rcnn_label')]\n else:\n label = labels[self.label.index('rcnn_label')]\n\n last_dim = pred.shape[-1]\n pred_label = pred.asnumpy().reshape(-1, last_dim).argmax(axis=1).astype('int32')\n label = label.asnumpy().reshape(-1,).astype('int32')\n\n # filter with keep_inds\n keep_inds = np.where(label != -1)\n pred_label = pred_label[keep_inds]\n label = label[keep_inds]\n\n self.sum_metric += np.sum(pred_label.flat == label.flat)\n self.num_inst += len(pred_label.flat)\n\n\nclass RPNLogLossMetric(mx.metric.EvalMetric):\n def __init__(self):\n super(RPNLogLossMetric, self).__init__('RPNLogLoss')\n self.pred, self.label = get_rpn_names()\n\n def update(self, labels, preds):\n pred = preds[self.pred.index('rpn_cls_prob')]\n label = labels[self.label.index('rpn_label')]\n\n # label (b, p)\n label = label.asnumpy().astype('int32').reshape((-1))\n # pred (b, c, p) or (b, c, h, w) --> (b, p, c) --> (b*p, c)\n pred = pred.asnumpy().reshape((pred.shape[0], pred.shape[1], -1)).transpose((0, 2, 1))\n pred = pred.reshape((label.shape[0], -1))\n\n # filter with keep_inds\n keep_inds = np.where(label != -1)[0]\n label = label[keep_inds]\n cls = pred[keep_inds, label]\n\n cls += 1e-14\n cls_loss = -1 * np.log(cls)\n cls_loss = np.sum(cls_loss)\n self.sum_metric += cls_loss\n self.num_inst += label.shape[0]\n\n\nclass RCNNLogLossMetric(mx.metric.EvalMetric):\n def __init__(self, cfg):\n super(RCNNLogLossMetric, self).__init__('RCNNLogLoss')\n self.e2e = cfg.TRAIN.END2END\n self.ohem = cfg.TRAIN.ENABLE_OHEM\n self.pred, self.label = get_rcnn_names(cfg)\n\n def update(self, labels, preds):\n pred = preds[self.pred.index('rcnn_cls_prob')]\n if self.ohem or self.e2e:\n label = preds[self.pred.index('rcnn_label')]\n else:\n label = labels[self.label.index('rcnn_label')]\n\n last_dim = pred.shape[-1]\n pred = pred.asnumpy().reshape(-1, last_dim)\n label = label.asnumpy().reshape(-1,).astype('int32')\n\n # filter with keep_inds\n keep_inds = np.where(label != -1)[0]\n label = label[keep_inds]\n cls = pred[keep_inds, label]\n\n cls += 1e-14\n cls_loss = -1 * np.log(cls)\n cls_loss = np.sum(cls_loss)\n self.sum_metric += cls_loss\n self.num_inst += label.shape[0]\n\n\nclass RPNL1LossMetric(mx.metric.EvalMetric):\n def __init__(self):\n super(RPNL1LossMetric, self).__init__('RPNL1Loss')\n self.pred, self.label = get_rpn_names()\n\n def update(self, labels, preds):\n bbox_loss = preds[self.pred.index('rpn_bbox_loss')].asnumpy()\n\n # calculate num_inst (average on those kept anchors)\n label = labels[self.label.index('rpn_label')].asnumpy()\n num_inst = np.sum(label != -1)\n\n self.sum_metric += np.sum(bbox_loss)\n self.num_inst += num_inst\n\n\nclass RCNNL1LossMetric(mx.metric.EvalMetric):\n def __init__(self, cfg):\n super(RCNNL1LossMetric, self).__init__('RCNNL1Loss')\n self.e2e = cfg.TRAIN.END2END\n self.ohem = cfg.TRAIN.ENABLE_OHEM\n self.pred, self.label = get_rcnn_names(cfg)\n\n def update(self, labels, preds):\n bbox_loss = preds[self.pred.index('rcnn_bbox_loss')].asnumpy()\n if self.ohem:\n label = preds[self.pred.index('rcnn_label')].asnumpy()\n else:\n if self.e2e:\n label = preds[self.pred.index('rcnn_label')].asnumpy()\n else:\n label = labels[self.label.index('rcnn_label')].asnumpy()\n\n # calculate num_inst (average on those kept anchors)\n num_inst = np.sum(label != -1)\n\n self.sum_metric += np.sum(bbox_loss)\n self.num_inst += num_inst\n" }, { "path": "rfcn/core/module.py", "content": "# --------------------------------------------------------\n# Deep Feature Flow\n# Copyright (c) 2017 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Modified by Yuwen Xiong\n# --------------------------------------------------------\n# Based on:\n# MX-RCNN\n# Copyright (c) 2016 by Contributors\n# Licence under The Apache 2.0 License\n# https://github.com/ijkguo/mx-rcnn/\n# --------------------------------------------------------\n\n\"\"\"A `MutableModule` implement the `BaseModule` API, and allows input shape\nvarying with training iterations. If shapes vary, executors will rebind,\nusing shared arrays from the initial module binded with maximum shape.\n\"\"\"\n\nimport time\nimport logging\nimport warnings\n\nfrom mxnet import context as ctx\nfrom mxnet.initializer import Uniform, InitDesc\nfrom mxnet.module.base_module import BaseModule, _check_input_names, _parse_data_desc, _as_list\nfrom mxnet.model import _create_kvstore, _initialize_kvstore, _update_params, _update_params_on_kvstore, load_checkpoint, BatchEndParam\nfrom mxnet import metric\n\nfrom .DataParallelExecutorGroup import DataParallelExecutorGroup\nfrom mxnet import ndarray as nd\nfrom mxnet import optimizer as opt\n\n\nclass Module(BaseModule):\n \"\"\"Module is a basic module that wrap a `Symbol`. It is functionally the same\n as the `FeedForward` model, except under the module API.\n\n Parameters\n ----------\n symbol : Symbol\n data_names : list of str\n Default is `('data')` for a typical model used in image classification.\n label_names : list of str\n Default is `('softmax_label')` for a typical model used in image\n classification.\n logger : Logger\n Default is `logging`.\n context : Context or list of Context\n Default is `cpu()`.\n work_load_list : list of number\n Default `None`, indicating uniform workload.\n fixed_param_names: list of str\n Default `None`, indicating no network parameters are fixed.\n state_names : list of str\n states are similar to data and label, but not provided by data iterator.\n Instead they are initialized to 0 and can be set by set_states()\n \"\"\"\n def __init__(self, symbol, data_names=('data',), label_names=('softmax_label',),\n logger=logging, context=ctx.cpu(), work_load_list=None,\n fixed_param_names=None, state_names=None):\n super(Module, self).__init__(logger=logger)\n\n if isinstance(context, ctx.Context):\n context = [context]\n self._context = context\n if work_load_list is None:\n work_load_list = [1] * len(self._context)\n assert len(work_load_list) == len(self._context)\n self._work_load_list = work_load_list\n\n self._symbol = symbol\n\n data_names = list(data_names) if data_names is not None else []\n label_names = list(label_names) if label_names is not None else []\n state_names = list(state_names) if state_names is not None else []\n fixed_param_names = list(fixed_param_names) if fixed_param_names is not None else []\n\n _check_input_names(symbol, data_names, \"data\", True)\n _check_input_names(symbol, label_names, \"label\", False)\n _check_input_names(symbol, state_names, \"state\", True)\n _check_input_names(symbol, fixed_param_names, \"fixed_param\", True)\n\n arg_names = symbol.list_arguments()\n input_names = data_names + label_names + state_names\n self._param_names = [x for x in arg_names if x not in input_names]\n self._fixed_param_names = fixed_param_names\n self._aux_names = symbol.list_auxiliary_states()\n self._data_names = data_names\n self._label_names = label_names\n self._state_names = state_names\n self._output_names = symbol.list_outputs()\n\n self._arg_params = None\n self._aux_params = None\n self._params_dirty = False\n\n self._optimizer = None\n self._kvstore = None\n self._update_on_kvstore = None\n self._updater = None\n self._preload_opt_states = None\n self._grad_req = None\n\n self._exec_group = None\n self._data_shapes = None\n self._label_shapes = None\n\n @staticmethod\n def load(prefix, epoch, load_optimizer_states=False, **kwargs):\n \"\"\"Create a model from previously saved checkpoint.\n\n Parameters\n ----------\n prefix : str\n path prefix of saved model files. You should have\n \"prefix-symbol.json\", \"prefix-xxxx.params\", and\n optionally \"prefix-xxxx.states\", where xxxx is the\n epoch number.\n epoch : int\n epoch to load.\n load_optimizer_states : bool\n whether to load optimizer states. Checkpoint needs\n to have been made with save_optimizer_states=True.\n data_names : list of str\n Default is `('data')` for a typical model used in image classification.\n label_names : list of str\n Default is `('softmax_label')` for a typical model used in image\n classification.\n logger : Logger\n Default is `logging`.\n context : Context or list of Context\n Default is `cpu()`.\n work_load_list : list of number\n Default `None`, indicating uniform workload.\n fixed_param_names: list of str\n Default `None`, indicating no network parameters are fixed.\n \"\"\"\n sym, args, auxs = load_checkpoint(prefix, epoch)\n mod = Module(symbol=sym, **kwargs)\n mod._arg_params = args\n mod._aux_params = auxs\n mod.params_initialized = True\n if load_optimizer_states:\n mod._preload_opt_states = '%s-%04d.states'%(prefix, epoch)\n return mod\n\n def save_checkpoint(self, prefix, epoch, save_optimizer_states=False):\n \"\"\"Save current progress to checkpoint.\n Use mx.callback.module_checkpoint as epoch_end_callback to save during training.\n\n Parameters\n ----------\n prefix : str\n The file prefix to checkpoint to\n epoch : int\n The current epoch number\n save_optimizer_states : bool\n Whether to save optimizer states for continue training\n \"\"\"\n self._symbol.save('%s-symbol.json'%prefix)\n param_name = '%s-%04d.params' % (prefix, epoch)\n self.save_params(param_name)\n logging.info('Saved checkpoint to \\\"%s\\\"', param_name)\n if save_optimizer_states:\n state_name = '%s-%04d.states' % (prefix, epoch)\n self.save_optimizer_states(state_name)\n logging.info('Saved optimizer state to \\\"%s\\\"', state_name)\n\n def _reset_bind(self):\n \"\"\"Internal function to reset binded state.\"\"\"\n self.binded = False\n self._exec_group = None\n self._data_shapes = None\n self._label_shapes = None\n\n @property\n def data_names(self):\n \"\"\"A list of names for data required by this module.\"\"\"\n return self._data_names\n\n @property\n def label_names(self):\n \"\"\"A list of names for labels required by this module.\"\"\"\n return self._label_names\n\n @property\n def output_names(self):\n \"\"\"A list of names for the outputs of this module.\"\"\"\n return self._output_names\n\n @property\n def data_shapes(self):\n \"\"\"Get data shapes.\n Returns\n -------\n A list of `(name, shape)` pairs.\n \"\"\"\n assert self.binded\n return self._data_shapes\n\n @property\n def label_shapes(self):\n \"\"\"Get label shapes.\n Returns\n -------\n A list of `(name, shape)` pairs. The return value could be `None` if\n the module does not need labels, or if the module is not binded for\n training (in this case, label information is not available).\n \"\"\"\n assert self.binded\n return self._label_shapes\n\n @property\n def output_shapes(self):\n \"\"\"Get output shapes.\n Returns\n -------\n A list of `(name, shape)` pairs.\n \"\"\"\n assert self.binded\n return self._exec_group.get_output_shapes()\n\n def get_params(self):\n \"\"\"Get current parameters.\n Returns\n -------\n `(arg_params, aux_params)`, each a dictionary of name to parameters (in\n `NDArray`) mapping.\n \"\"\"\n assert self.binded and self.params_initialized\n\n if self._params_dirty:\n self._sync_params_from_devices()\n return (self._arg_params, self._aux_params)\n\n def init_params(self, initializer=Uniform(0.01), arg_params=None, aux_params=None,\n allow_missing=False, force_init=False):\n \"\"\"Initialize the parameters and auxiliary states.\n\n Parameters\n ----------\n initializer : Initializer\n Called to initialize parameters if needed.\n arg_params : dict\n If not None, should be a dictionary of existing arg_params. Initialization\n will be copied from that.\n aux_params : dict\n If not None, should be a dictionary of existing aux_params. Initialization\n will be copied from that.\n allow_missing : bool\n If true, params could contain missing values, and the initializer will be\n called to fill those missing params.\n force_init : bool\n If true, will force re-initialize even if already initialized.\n \"\"\"\n if self.params_initialized and not force_init:\n warnings.warn(\"Parameters already initialized and force_init=False. \"\n \"init_params call ignored.\", stacklevel=2)\n return\n assert self.binded, 'call bind before initializing the parameters'\n\n def _impl(name, arr, cache):\n \"\"\"Internal helper for parameter initialization\"\"\"\n if cache is not None:\n if name in cache:\n cache_arr = cache[name]\n\n # just in case the cached array is just the target itself\n if cache_arr is not arr:\n cache_arr.copyto(arr)\n else:\n if not allow_missing:\n raise RuntimeError(\"%s is not presented\" % name)\n if initializer != None:\n initializer(name, arr)\n else:\n initializer(name, arr)\n\n attrs = self._symbol.attr_dict()\n for name, arr in self._arg_params.items():\n desc = InitDesc(name, attrs.get(name, None))\n _impl(desc, arr, arg_params)\n\n for name, arr in self._aux_params.items():\n desc = InitDesc(name, attrs.get(name, None))\n _impl(desc, arr, aux_params)\n\n self.params_initialized = True\n self._params_dirty = False\n\n # copy the initialized parameters to devices\n self._exec_group.set_params(self._arg_params, self._aux_params)\n\n def set_params(self, arg_params, aux_params, allow_missing=False, force_init=True):\n \"\"\"Assign parameter and aux state values.\n\n Parameters\n ----------\n arg_params : dict\n Dictionary of name to value (`NDArray`) mapping.\n aux_params : dict\n Dictionary of name to value (`NDArray`) mapping.\n allow_missing : bool\n If true, params could contain missing values, and the initializer will be\n called to fill those missing params.\n force_init : bool\n If true, will force re-initialize even if already initialized.\n\n Examples\n --------\n An example of setting module parameters::\n >>> sym, arg_params, aux_params = \\\n >>> mx.model.load_checkpoint(model_prefix, n_epoch_load)\n >>> mod.set_params(arg_params=arg_params, aux_params=aux_params)\n \"\"\"\n if not allow_missing:\n self.init_params(initializer=None, arg_params=arg_params, aux_params=aux_params,\n allow_missing=allow_missing, force_init=force_init)\n return\n\n if self.params_initialized and not force_init:\n warnings.warn(\"Parameters already initialized and force_init=False. \"\n \"set_params call ignored.\", stacklevel=2)\n return\n\n self._exec_group.set_params(arg_params, aux_params)\n\n # because we didn't update self._arg_params, they are dirty now.\n self._params_dirty = True\n self.params_initialized = True\n\n def bind(self, data_shapes, label_shapes=None, for_training=True,\n inputs_need_grad=False, force_rebind=False, shared_module=None,\n grad_req='write'):\n \"\"\"Bind the symbols to construct executors. This is necessary before one\n can perform computation with the module.\n\n Parameters\n ----------\n data_shapes : list of (str, tuple)\n Typically is `data_iter.provide_data`.\n label_shapes : list of (str, tuple)\n Typically is `data_iter.provide_label`.\n for_training : bool\n Default is `True`. Whether the executors should be bind for training.\n inputs_need_grad : bool\n Default is `False`. Whether the gradients to the input data need to be computed.\n Typically this is not needed. But this might be needed when implementing composition\n of modules.\n force_rebind : bool\n Default is `False`. This function does nothing if the executors are already\n binded. But with this `True`, the executors will be forced to rebind.\n shared_module : Module\n Default is `None`. This is used in bucketing. When not `None`, the shared module\n essentially corresponds to a different bucket -- a module with different symbol\n but with the same sets of parameters (e.g. unrolled RNNs with different lengths).\n \"\"\"\n # force rebinding is typically used when one want to switch from\n # training to prediction phase.\n if force_rebind:\n self._reset_bind()\n\n if self.binded:\n self.logger.warning('Already binded, ignoring bind()')\n return\n\n self.for_training = for_training\n self.inputs_need_grad = inputs_need_grad\n self.binded = True\n self._grad_req = grad_req\n\n if not for_training:\n assert not inputs_need_grad\n else:\n pass\n # this is not True, as some module might not contains a loss function\n # that consumes the labels\n # assert label_shapes is not None\n\n # self._data_shapes, self._label_shapes = _parse_data_desc(\n # self.data_names, self.label_names, data_shapes, label_shapes)\n self._data_shapes, self._label_shapes = zip(*[_parse_data_desc(self.data_names, self.label_names, data_shape, label_shape)\n for data_shape, label_shape in zip(data_shapes, label_shapes)])\n if self._label_shapes.count(None) == len(self._label_shapes):\n self._label_shapes = None\n\n if shared_module is not None:\n assert isinstance(shared_module, Module) and \\\n shared_module.binded and shared_module.params_initialized\n shared_group = shared_module._exec_group\n else:\n shared_group = None\n\n self._exec_group = DataParallelExecutorGroup(self._symbol, self._context,\n self._work_load_list, self._data_shapes,\n self._label_shapes, self._param_names,\n for_training, inputs_need_grad,\n shared_group, logger=self.logger,\n fixed_param_names=self._fixed_param_names,\n grad_req=grad_req,\n state_names=self._state_names)\n # self._total_exec_bytes = self._exec_group._total_exec_bytes\n if shared_module is not None:\n self.params_initialized = True\n self._arg_params = shared_module._arg_params\n self._aux_params = shared_module._aux_params\n elif self.params_initialized:\n # if the parameters are already initialized, we are re-binding\n # so automatically copy the already initialized params\n self._exec_group.set_params(self._arg_params, self._aux_params)\n else:\n assert self._arg_params is None and self._aux_params is None\n param_arrays = [\n nd.zeros(x[0].shape, dtype=x[0].dtype)\n for x in self._exec_group.param_arrays\n ]\n self._arg_params = {name:arr for name, arr in zip(self._param_names, param_arrays)}\n\n aux_arrays = [\n nd.zeros(x[0].shape, dtype=x[0].dtype)\n for x in self._exec_group.aux_arrays\n ]\n self._aux_params = {name:arr for name, arr in zip(self._aux_names, aux_arrays)}\n\n if shared_module is not None and shared_module.optimizer_initialized:\n self.borrow_optimizer(shared_module)\n\n\n def reshape(self, data_shapes, label_shapes=None):\n \"\"\"Reshape the module for new input shapes.\n\n Parameters\n ----------\n data_shapes : list of (str, tuple)\n Typically is `data_iter.provide_data`.\n label_shapes : list of (str, tuple)\n Typically is `data_iter.provide_label`.\n \"\"\"\n assert self.binded\n # self._data_shapes, self._label_shapes = _parse_data_desc(\n # self.data_names, self.label_names, data_shapes, label_shapes)\n self._data_shapes, self._label_shapes = zip(*[_parse_data_desc(self.data_names, self.label_names, data_shape, label_shape)\n for data_shape, label_shape in zip(data_shapes, label_shapes)])\n\n self._exec_group.reshape(self._data_shapes, self._label_shapes)\n\n\n def init_optimizer(self, kvstore='local', optimizer='sgd',\n optimizer_params=(('learning_rate', 0.01),), force_init=False):\n \"\"\"Install and initialize optimizers.\n\n Parameters\n ----------\n kvstore : str or KVStore\n Default `'local'`.\n optimizer : str or Optimizer\n Default `'sgd'`\n optimizer_params : dict\n Default `(('learning_rate', 0.01),)`. The default value is not a dictionary,\n just to avoid pylint warning of dangerous default values.\n force_init : bool\n Default `False`, indicating whether we should force re-initializing the\n optimizer in the case an optimizer is already installed.\n \"\"\"\n assert self.binded and self.params_initialized\n\n if self.optimizer_initialized and not force_init:\n self.logger.warning('optimizer already initialized, ignoring...')\n return\n\n (kvstore, update_on_kvstore) = \\\n _create_kvstore(kvstore, len(self._context), self._arg_params)\n\n batch_size = self._exec_group.batch_size\n if kvstore and 'dist' in kvstore.type and '_sync' in kvstore.type:\n batch_size *= kvstore.num_workers\n rescale_grad = 1.0/batch_size\n\n if isinstance(optimizer, str):\n idx2name = {}\n if update_on_kvstore:\n idx2name.update(enumerate(self._exec_group.param_names))\n else:\n for k in range(len(self._context)):\n idx2name.update({i*len(self._context)+k: n\n for i, n in enumerate(self._exec_group.param_names)})\n optimizer_params = dict(optimizer_params)\n if 'rescale_grad' not in optimizer_params:\n optimizer_params['rescale_grad'] = rescale_grad\n optimizer = opt.create(optimizer,\n sym=self.symbol, param_idx2name=idx2name,\n **optimizer_params)\n else:\n assert isinstance(optimizer, opt.Optimizer)\n if optimizer.rescale_grad != rescale_grad:\n #pylint: disable=no-member\n warnings.warn(\n \"Optimizer created manually outside Module but rescale_grad \" +\n \"is not normalized to 1.0/batch_size/num_workers (%s vs. %s). \"%(\n optimizer.rescale_grad, rescale_grad) +\n \"Is this intended?\", stacklevel=2)\n\n self._optimizer = optimizer\n self._kvstore = kvstore\n self._update_on_kvstore = update_on_kvstore\n self._updater = None\n\n if kvstore:\n # copy initialized local parameters to kvstore\n _initialize_kvstore(kvstore=kvstore,\n param_arrays=self._exec_group.param_arrays,\n arg_params=self._arg_params,\n param_names=self._param_names,\n update_on_kvstore=update_on_kvstore)\n if update_on_kvstore:\n kvstore.set_optimizer(self._optimizer)\n else:\n self._updater = opt.get_updater(optimizer)\n\n self.optimizer_initialized = True\n\n if self._preload_opt_states is not None:\n self.load_optimizer_states(self._preload_opt_states)\n self._preload_opt_states = None\n\n def borrow_optimizer(self, shared_module):\n \"\"\"Borrow optimizer from a shared module. Used in bucketing, where exactly the same\n optimizer (esp. kvstore) is used.\n\n Parameters\n ----------\n shared_module : Module\n \"\"\"\n assert shared_module.optimizer_initialized\n self._optimizer = shared_module._optimizer\n self._kvstore = shared_module._kvstore\n self._update_on_kvstore = shared_module._update_on_kvstore\n self._updater = shared_module._updater\n self.optimizer_initialized = True\n\n def forward(self, data_batch, is_train=None):\n \"\"\"Forward computation.\n\n Parameters\n ----------\n data_batch : DataBatch\n Could be anything with similar API implemented.\n is_train : bool\n Default is `None`, which means `is_train` takes the value of `self.for_training`.\n \"\"\"\n assert self.binded and self.params_initialized\n self._exec_group.forward(data_batch, is_train)\n\n def backward(self, out_grads=None):\n \"\"\"Backward computation.\n\n Parameters\n ----------\n out_grads : NDArray or list of NDArray, optional\n Gradient on the outputs to be propagated back.\n This parameter is only needed when bind is called\n on outputs that are not a loss function.\n \"\"\"\n assert self.binded and self.params_initialized\n self._exec_group.backward(out_grads=out_grads)\n\n def update(self):\n \"\"\"Update parameters according to the installed optimizer and the gradients computed\n in the previous forward-backward batch.\n \"\"\"\n assert self.binded and self.params_initialized and self.optimizer_initialized\n\n self._params_dirty = True\n if self._update_on_kvstore:\n _update_params_on_kvstore(self._exec_group.param_arrays,\n self._exec_group.grad_arrays,\n self._kvstore)\n else:\n _update_params(self._exec_group.param_arrays,\n self._exec_group.grad_arrays,\n updater=self._updater,\n num_device=len(self._context),\n kvstore=self._kvstore)\n\n def get_outputs(self, merge_multi_context=True):\n \"\"\"Get outputs of the previous forward computation.\n\n Parameters\n ----------\n merge_multi_context : bool\n Default is `True`. In the case when data-parallelism is used, the outputs\n will be collected from multiple devices. A `True` value indicate that we\n should merge the collected results so that they look like from a single\n executor.\n\n Returns\n -------\n If `merge_multi_context` is `True`, it is like `[out1, out2]`. Otherwise, it\n is like `[[out1_dev1, out1_dev2], [out2_dev1, out2_dev2]]`. All the output\n elements are `NDArray`.\n \"\"\"\n assert self.binded and self.params_initialized\n return self._exec_group.get_outputs(merge_multi_context=merge_multi_context)\n\n def get_input_grads(self, merge_multi_context=True):\n \"\"\"Get the gradients with respect to the inputs of the module.\n\n Parameters\n ----------\n merge_multi_context : bool\n Default is `True`. In the case when data-parallelism is used, the outputs\n will be collected from multiple devices. A `True` value indicate that we\n should merge the collected results so that they look like from a single\n executor.\n\n Returns\n -------\n If `merge_multi_context` is `True`, it is like `[grad1, grad2]`. Otherwise, it\n is like `[[grad1_dev1, grad1_dev2], [grad2_dev1, grad2_dev2]]`. All the output\n elements are `NDArray`.\n \"\"\"\n assert self.binded and self.params_initialized and self.inputs_need_grad\n return self._exec_group.get_input_grads(merge_multi_context=merge_multi_context)\n\n def get_states(self, merge_multi_context=True):\n \"\"\"Get states from all devices\n\n Parameters\n ----------\n merge_multi_context : bool\n Default is `True`. In the case when data-parallelism is used, the states\n will be collected from multiple devices. A `True` value indicate that we\n should merge the collected results so that they look like from a single\n executor.\n\n Returns\n -------\n If `merge_multi_context` is `True`, it is like `[out1, out2]`. Otherwise, it\n is like `[[out1_dev1, out1_dev2], [out2_dev1, out2_dev2]]`. All the output\n elements are `NDArray`.\n \"\"\"\n assert self.binded and self.params_initialized\n return self._exec_group.get_states(merge_multi_context=merge_multi_context)\n\n def set_states(self, states=None, value=None):\n \"\"\"Set value for states. Only one of states & value can be specified.\n\n Parameters\n ----------\n states : list of list of NDArrays\n source states arrays formatted like [[state1_dev1, state1_dev2],\n [state2_dev1, state2_dev2]].\n value : number\n a single scalar value for all state arrays.\n \"\"\"\n assert self.binded and self.params_initialized\n self._exec_group.set_states(states, value)\n\n def update_metric(self, eval_metric, labels):\n \"\"\"Evaluate and accumulate evaluation metric on outputs of the last forward computation.\n\n Parameters\n ----------\n eval_metric : EvalMetric\n labels : list of NDArray\n Typically `data_batch.label`.\n \"\"\"\n self._exec_group.update_metric(eval_metric, labels)\n\n def _sync_params_from_devices(self):\n \"\"\"Synchronize parameters from devices to CPU. This function should be called after\n calling `update` that updates the parameters on the devices, before one can read the\n latest parameters from `self._arg_params` and `self._aux_params`.\n \"\"\"\n self._exec_group.get_params(self._arg_params, self._aux_params)\n self._params_dirty = False\n\n def save_optimizer_states(self, fname):\n \"\"\"Save optimizer (updater) state to file\n\n Parameters\n ----------\n fname : str\n Path to output states file.\n \"\"\"\n assert self.optimizer_initialized\n\n if self._update_on_kvstore:\n self._kvstore.save_optimizer_states(fname)\n else:\n with open(fname, 'wb') as fout:\n fout.write(self._updater.get_states())\n\n def load_optimizer_states(self, fname):\n \"\"\"Load optimizer (updater) state from file\n\n Parameters\n ----------\n fname : str\n Path to input states file.\n \"\"\"\n assert self.optimizer_initialized\n\n if self._update_on_kvstore:\n self._kvstore.load_optimizer_states(fname)\n else:\n self._updater.set_states(open(fname, 'rb').read())\n\n def install_monitor(self, mon):\n \"\"\" Install monitor on all executors \"\"\"\n assert self.binded\n self._exec_group.install_monitor(mon)\n\n\nclass MutableModule(BaseModule):\n \"\"\"A mutable module is a module that supports variable input data.\n\n Parameters\n ----------\n symbol : Symbol\n data_names : list of str\n label_names : list of str\n logger : Logger\n context : Context or list of Context\n work_load_list : list of number\n max_data_shapes : list of (name, shape) tuple, designating inputs whose shape vary\n max_label_shapes : list of (name, shape) tuple, designating inputs whose shape vary\n fixed_param_prefix : list of str, indicating fixed parameters\n \"\"\"\n def __init__(self, symbol, data_names, label_names,\n logger=logging, context=ctx.cpu(), work_load_list=None,\n max_data_shapes=None, max_label_shapes=None, fixed_param_prefix=None):\n super(MutableModule, self).__init__(logger=logger)\n self._symbol = symbol\n self._data_names = data_names\n self._label_names = label_names\n self._context = context\n self._work_load_list = work_load_list\n\n self._curr_module = None\n self._max_data_shapes = max_data_shapes\n self._max_label_shapes = max_label_shapes\n self._fixed_param_prefix = fixed_param_prefix\n\n fixed_param_names = list()\n if fixed_param_prefix is not None:\n for name in self._symbol.list_arguments():\n for prefix in self._fixed_param_prefix:\n if name.startswith(prefix):\n fixed_param_names.append(name)\n self._fixed_param_names = fixed_param_names\n self._preload_opt_states = None\n\n def _reset_bind(self):\n self.binded = False\n self._curr_module = None\n\n @property\n def data_names(self):\n return self._data_names\n\n @property\n def output_names(self):\n return self._symbol.list_outputs()\n\n @property\n def data_shapes(self):\n assert self.binded\n return self._curr_module.data_shapes\n\n @property\n def label_shapes(self):\n assert self.binded\n return self._curr_module.label_shapes\n\n @property\n def output_shapes(self):\n assert self.binded\n return self._curr_module.output_shapes\n\n def get_params(self):\n assert self.binded and self.params_initialized\n return self._curr_module.get_params()\n\n def init_params(self, initializer=Uniform(0.01), arg_params=None, aux_params=None,\n allow_missing=False, force_init=False):\n if self.params_initialized and not force_init:\n return\n assert self.binded, 'call bind before initializing the parameters'\n self._curr_module.init_params(initializer=initializer, arg_params=arg_params,\n aux_params=aux_params, allow_missing=allow_missing,\n force_init=force_init)\n self.params_initialized = True\n\n def bind(self, data_shapes, label_shapes=None, for_training=True,\n inputs_need_grad=False, force_rebind=False, shared_module=None, grad_req='write'):\n # in case we already initialized params, keep it\n if self.params_initialized:\n arg_params, aux_params = self.get_params()\n\n # force rebinding is typically used when one want to switch from\n # training to prediction phase.\n if force_rebind:\n self._reset_bind()\n\n if self.binded:\n self.logger.warning('Already binded, ignoring bind()')\n return\n\n assert shared_module is None, 'shared_module for MutableModule is not supported'\n\n self.for_training = for_training\n self.inputs_need_grad = inputs_need_grad\n self.binded = True\n\n max_shapes_dict = dict()\n if self._max_data_shapes is not None:\n max_shapes_dict.update(dict(self._max_data_shapes[0]))\n if self._max_label_shapes is not None:\n max_shapes_dict.update(dict(self._max_label_shapes[0]))\n\n max_data_shapes = list()\n for name, shape in data_shapes[0]:\n if name in max_shapes_dict:\n max_data_shapes.append((name, max_shapes_dict[name]))\n else:\n max_data_shapes.append((name, shape))\n\n max_label_shapes = list()\n if not label_shapes.count(None) == len(label_shapes):\n for name, shape in label_shapes[0]:\n if name in max_shapes_dict:\n max_label_shapes.append((name, max_shapes_dict[name]))\n else:\n max_label_shapes.append((name, shape))\n\n if len(max_label_shapes) == 0:\n max_label_shapes = None\n\n module = Module(self._symbol, self._data_names, self._label_names, logger=self.logger,\n context=self._context, work_load_list=self._work_load_list,\n fixed_param_names=self._fixed_param_names)\n module.bind([max_data_shapes for _ in xrange(len(self._context))], [max_label_shapes for _ in xrange(len(self._context))],\n for_training, inputs_need_grad, force_rebind=False, shared_module=None)\n self._curr_module = module\n\n # copy back saved params, if already initialized\n if self.params_initialized:\n self.set_params(arg_params, aux_params)\n\n def save_checkpoint(self, prefix, epoch, save_optimizer_states=False):\n \"\"\"Save current progress to checkpoint.\n Use mx.callback.module_checkpoint as epoch_end_callback to save during training.\n\n Parameters\n ----------\n prefix : str\n The file prefix to checkpoint to\n epoch : int\n The current epoch number\n save_optimizer_states : bool\n Whether to save optimizer states for continue training\n \"\"\"\n self._curr_module.save_checkpoint(prefix, epoch, save_optimizer_states)\n\n def init_optimizer(self, kvstore='local', optimizer='sgd',\n optimizer_params=(('learning_rate', 0.01),), force_init=False):\n assert self.binded and self.params_initialized\n if self.optimizer_initialized and not force_init:\n self.logger.warning('optimizer already initialized, ignoring.')\n return\n\n self._curr_module._preload_opt_states = self._preload_opt_states\n self._curr_module.init_optimizer(kvstore, optimizer, optimizer_params,\n force_init=force_init)\n self.optimizer_initialized = True\n\n def fit(self, train_data, eval_data=None, eval_metric='acc',\n epoch_end_callback=None, batch_end_callback=None, kvstore='local',\n optimizer='sgd', optimizer_params=(('learning_rate', 0.01),),\n eval_end_callback=None,\n eval_batch_end_callback=None, initializer=Uniform(0.01),\n arg_params=None, aux_params=None, allow_missing=False,\n force_rebind=False, force_init=False, begin_epoch=0, num_epoch=None,\n validation_metric=None, monitor=None, prefix=None):\n \"\"\"Train the module parameters.\n\n Parameters\n ----------\n train_data : DataIter\n eval_data : DataIter\n If not `None`, will be used as validation set and evaluate the performance\n after each epoch.\n eval_metric : str or EvalMetric\n Default `'acc'`. The performance measure used to display during training.\n epoch_end_callback : function or list of function\n Each callback will be called with the current `epoch`, `symbol`, `arg_params`\n and `aux_params`.\n batch_end_callback : function or list of function\n Each callback will be called with a `BatchEndParam`.\n kvstore : str or KVStore\n Default `'local'`.\n optimizer : str or Optimizer\n Default `'sgd'`\n optimizer_params : dict\n Default `(('learning_rate', 0.01),)`. The parameters for the optimizer constructor.\n The default value is not a `dict`, just to avoid pylint warning on dangerous\n default values.\n eval_end_callback : function or list of function\n These will be called at the end of each full evaluation, with the metrics over\n the entire evaluation set.\n eval_batch_end_callback : function or list of function\n These will be called at the end of each minibatch during evaluation\n initializer : Initializer\n Will be called to initialize the module parameters if not already initialized.\n arg_params : dict\n Default `None`, if not `None`, should be existing parameters from a trained\n model or loaded from a checkpoint (previously saved model). In this case,\n the value here will be used to initialize the module parameters, unless they\n are already initialized by the user via a call to `init_params` or `fit`.\n `arg_params` has higher priority to `initializer`.\n aux_params : dict\n Default `None`. Similar to `arg_params`, except for auxiliary states.\n allow_missing : bool\n Default `False`. Indicate whether we allow missing parameters when `arg_params`\n and `aux_params` are not `None`. If this is `True`, then the missing parameters\n will be initialized via the `initializer`.\n force_rebind : bool\n Default `False`. Whether to force rebinding the executors if already binded.\n force_init : bool\n Default `False`. Indicate whether we should force initialization even if the\n parameters are already initialized.\n begin_epoch : int\n Default `0`. Indicate the starting epoch. Usually, if we are resuming from a\n checkpoint saved at a previous training phase at epoch N, then we should specify\n this value as N+1.\n num_epoch : int\n Number of epochs to run training.\n\n Examples\n --------\n An example of using fit for training::\n >>> #Assume training dataIter and validation dataIter are ready\n >>> mod.fit(train_data=train_dataiter, eval_data=val_dataiter,\n optimizer_params={'learning_rate':0.01, 'momentum': 0.9},\n num_epoch=10)\n \"\"\"\n assert num_epoch is not None, 'please specify number of epochs'\n\n self.bind(data_shapes=train_data.provide_data, label_shapes=train_data.provide_label,\n for_training=True, force_rebind=force_rebind)\n if monitor is not None:\n self.install_monitor(monitor)\n self.init_params(initializer=initializer, arg_params=arg_params, aux_params=aux_params,\n allow_missing=allow_missing, force_init=force_init)\n self.init_optimizer(kvstore=kvstore, optimizer=optimizer,\n optimizer_params=optimizer_params)\n\n if validation_metric is None:\n validation_metric = eval_metric\n if not isinstance(eval_metric, metric.EvalMetric):\n eval_metric = metric.create(eval_metric)\n\n ################################################################################\n # training loop\n ################################################################################\n for epoch in range(begin_epoch, num_epoch):\n tic = time.time()\n eval_metric.reset()\n for nbatch, data_batch in enumerate(train_data):\n if monitor is not None:\n monitor.tic()\n self.forward_backward(data_batch)\n self.update()\n self.update_metric(eval_metric, data_batch.label)\n\n if monitor is not None:\n monitor.toc_print()\n\n if batch_end_callback is not None:\n batch_end_params = BatchEndParam(epoch=epoch, nbatch=nbatch,\n eval_metric=eval_metric,\n locals=locals())\n for callback in _as_list(batch_end_callback):\n callback(batch_end_params)\n\n # one epoch of training is finished\n for name, val in eval_metric.get_name_value():\n self.logger.info('Epoch[%d] Train-%s=%f', epoch, name, val)\n toc = time.time()\n self.logger.info('Epoch[%d] Time cost=%.3f', epoch, (toc-tic))\n\n # sync aux params across devices\n arg_params, aux_params = self.get_params()\n self.set_params(arg_params, aux_params)\n\n if epoch_end_callback is not None:\n for callback in _as_list(epoch_end_callback):\n callback(epoch, self.symbol, arg_params, aux_params)\n\n #----------------------------------------\n # evaluation on validation set\n if eval_data:\n res = self.score(eval_data, validation_metric,\n score_end_callback=eval_end_callback,\n batch_end_callback=eval_batch_end_callback, epoch=epoch)\n #TODO: pull this into default\n for name, val in res:\n self.logger.info('Epoch[%d] Validation-%s=%f', epoch, name, val)\n\n # end of 1 epoch, reset the data-iter for another epoch\n train_data.reset()\n\n\n def forward(self, data_batch, is_train=None):\n assert self.binded and self.params_initialized\n\n # get current_shapes\n if self._curr_module.label_shapes is not None:\n current_shapes = [dict(self._curr_module.data_shapes[i] + self._curr_module.label_shapes[i]) for i in xrange(len(self._context))]\n else:\n current_shapes = [dict(self._curr_module.data_shapes[i]) for i in xrange(len(self._context))]\n\n # get input_shapes\n if is_train:\n input_shapes = [dict(data_batch.provide_data[i] + data_batch.provide_label[i]) for i in xrange(len(self._context))]\n else:\n input_shapes = [dict(data_batch.provide_data[i]) for i in xrange(len(data_batch.provide_data))]\n\n # decide if shape changed\n shape_changed = len(current_shapes) != len(input_shapes)\n for pre, cur in zip(current_shapes, input_shapes):\n for k, v in pre.items():\n if v != cur[k]:\n shape_changed = True\n\n if shape_changed:\n # self._curr_module.reshape(data_batch.provide_data, data_batch.provide_label)\n module = Module(self._symbol, self._data_names, self._label_names,\n logger=self.logger, context=[self._context[i] for i in xrange(len(data_batch.provide_data))],\n work_load_list=self._work_load_list,\n fixed_param_names=self._fixed_param_names)\n module.bind(data_batch.provide_data, data_batch.provide_label, self._curr_module.for_training,\n self._curr_module.inputs_need_grad, force_rebind=False,\n shared_module=self._curr_module)\n self._curr_module = module\n\n self._curr_module.forward(data_batch, is_train=is_train)\n\n def backward(self, out_grads=None):\n assert self.binded and self.params_initialized\n self._curr_module.backward(out_grads=out_grads)\n\n def update(self):\n assert self.binded and self.params_initialized and self.optimizer_initialized\n self._curr_module.update()\n\n def get_outputs(self, merge_multi_context=True):\n assert self.binded and self.params_initialized\n return self._curr_module.get_outputs(merge_multi_context=merge_multi_context)\n def get_input_grads(self, merge_multi_context=True):\n assert self.binded and self.params_initialized and self.inputs_need_grad\n return self._curr_module.get_input_grads(merge_multi_context=merge_multi_context)\n\n def update_metric(self, eval_metric, labels):\n assert self.binded and self.params_initialized\n self._curr_module.update_metric(eval_metric, labels)\n\n def install_monitor(self, mon):\n \"\"\" Install monitor on all executors \"\"\"\n assert self.binded\n self._curr_module.install_monitor(mon)\n" }, { "path": "rfcn/core/rcnn.py", "content": "# --------------------------------------------------------\n# Deep Feature Flow\n# Copyright (c) 2017 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Modified by Yuwen Xiong\n# --------------------------------------------------------\n# Based on:\n# MX-RCNN\n# Copyright (c) 2016 by Contributors\n# Licence under The Apache 2.0 License\n# https://github.com/ijkguo/mx-rcnn/\n# --------------------------------------------------------\n\n\"\"\"\nFast R-CNN:\ndata =\n {'data': [num_images, c, h, w],\n 'rois': [num_rois, 5]}\nlabel =\n {'label': [num_rois],\n 'bbox_target': [num_rois, 4 * num_classes],\n 'bbox_weight': [num_rois, 4 * num_classes]}\nroidb extended format [image_index]\n ['image', 'height', 'width', 'flipped',\n 'boxes', 'gt_classes', 'gt_overlaps', 'max_classes', 'max_overlaps', 'bbox_targets']\n\"\"\"\n\nimport numpy as np\nimport numpy.random as npr\n\nfrom utils.image import get_image, tensor_vstack\nfrom bbox.bbox_transform import bbox_overlaps, bbox_transform\nfrom bbox.bbox_regression import expand_bbox_regression_targets\n\n\ndef get_rcnn_testbatch(roidb, cfg):\n \"\"\"\n return a dict of testbatch\n :param roidb: ['image', 'flipped'] + ['boxes']\n :return: data, label, im_info\n \"\"\"\n # assert len(roidb) == 1, 'Single batch only'\n imgs, roidb = get_image(roidb, cfg)\n im_array = imgs\n im_info = [np.array([roidb[i]['im_info']], dtype=np.float32) for i in range(len(roidb))]\n\n im_rois = [roidb[i]['boxes'] for i in range(len(roidb))]\n rois = im_rois\n rois_array = [np.hstack((0 * np.ones((rois[i].shape[0], 1)), rois[i])) for i in range(len(rois))]\n\n data = [{'data': im_array[i],\n 'rois': rois_array[i]} for i in range(len(roidb))]\n label = {}\n\n return data, label, im_info\n\n\ndef get_rcnn_batch(roidb, cfg):\n \"\"\"\n return a dict of multiple images\n :param roidb: a list of dict, whose length controls batch size\n ['images', 'flipped'] + ['gt_boxes', 'boxes', 'gt_overlap'] => ['bbox_targets']\n :return: data, label\n \"\"\"\n num_images = len(roidb)\n imgs, roidb = get_image(roidb, cfg)\n im_array = tensor_vstack(imgs)\n\n assert cfg.TRAIN.BATCH_ROIS == -1 or cfg.TRAIN.BATCH_ROIS % cfg.TRAIN.BATCH_IMAGES == 0, \\\n 'BATCHIMAGES {} must divide BATCH_ROIS {}'.format(cfg.TRAIN.BATCH_IMAGES, cfg.TRAIN.BATCH_ROIS)\n\n if cfg.TRAIN.BATCH_ROIS == -1:\n rois_per_image = np.sum([iroidb['boxes'].shape[0] for iroidb in roidb])\n fg_rois_per_image = rois_per_image\n else:\n rois_per_image = cfg.TRAIN.BATCH_ROIS / cfg.TRAIN.BATCH_IMAGES\n fg_rois_per_image = np.round(cfg.TRAIN.FG_FRACTION * rois_per_image).astype(int)\n\n rois_array = list()\n labels_array = list()\n bbox_targets_array = list()\n bbox_weights_array = list()\n\n for im_i in range(num_images):\n roi_rec = roidb[im_i]\n\n # infer num_classes from gt_overlaps\n num_classes = roi_rec['gt_overlaps'].shape[1]\n\n # label = class RoI has max overlap with\n rois = roi_rec['boxes']\n labels = roi_rec['max_classes']\n overlaps = roi_rec['max_overlaps']\n bbox_targets = roi_rec['bbox_targets']\n\n im_rois, labels, bbox_targets, bbox_weights = \\\n sample_rois(rois, fg_rois_per_image, rois_per_image, num_classes, cfg,\n labels, overlaps, bbox_targets)\n\n # project im_rois\n # do not round roi\n rois = im_rois\n batch_index = im_i * np.ones((rois.shape[0], 1))\n rois_array_this_image = np.hstack((batch_index, rois))\n rois_array.append(rois_array_this_image)\n\n # add labels\n labels_array.append(labels)\n bbox_targets_array.append(bbox_targets)\n bbox_weights_array.append(bbox_weights)\n\n rois_array = np.array(rois_array)\n labels_array = np.array(labels_array)\n bbox_targets_array = np.array(bbox_targets_array)\n bbox_weights_array = np.array(bbox_weights_array)\n\n data = {'data': im_array,\n 'rois': rois_array}\n label = {'label': labels_array,\n 'bbox_target': bbox_targets_array,\n 'bbox_weight': bbox_weights_array}\n\n return data, label\n\n\ndef sample_rois(rois, fg_rois_per_image, rois_per_image, num_classes, cfg,\n labels=None, overlaps=None, bbox_targets=None, gt_boxes=None):\n \"\"\"\n generate random sample of ROIs comprising foreground and background examples\n :param rois: all_rois [n, 4]; e2e: [n, 5] with batch_index\n :param fg_rois_per_image: foreground roi number\n :param rois_per_image: total roi number\n :param num_classes: number of classes\n :param labels: maybe precomputed\n :param overlaps: maybe precomputed (max_overlaps)\n :param bbox_targets: maybe precomputed\n :param gt_boxes: optional for e2e [n, 5] (x1, y1, x2, y2, cls)\n :return: (labels, rois, bbox_targets, bbox_weights)\n \"\"\"\n if labels is None:\n overlaps = bbox_overlaps(rois[:, 1:].astype(np.float), gt_boxes[:, :4].astype(np.float))\n gt_assignment = overlaps.argmax(axis=1)\n overlaps = overlaps.max(axis=1)\n labels = gt_boxes[gt_assignment, 4]\n\n # foreground RoI with FG_THRESH overlap\n fg_indexes = np.where(overlaps >= cfg.TRAIN.FG_THRESH)[0]\n # guard against the case when an image has fewer than fg_rois_per_image foreground RoIs\n fg_rois_per_this_image = np.minimum(fg_rois_per_image, fg_indexes.size)\n # Sample foreground regions without replacement\n if len(fg_indexes) > fg_rois_per_this_image:\n fg_indexes = npr.choice(fg_indexes, size=fg_rois_per_this_image, replace=False)\n\n # Select background RoIs as those within [BG_THRESH_LO, BG_THRESH_HI)\n bg_indexes = np.where((overlaps < cfg.TRAIN.BG_THRESH_HI) & (overlaps >= cfg.TRAIN.BG_THRESH_LO))[0]\n # Compute number of background RoIs to take from this image (guarding against there being fewer than desired)\n bg_rois_per_this_image = rois_per_image - fg_rois_per_this_image\n bg_rois_per_this_image = np.minimum(bg_rois_per_this_image, bg_indexes.size)\n # Sample foreground regions without replacement\n if len(bg_indexes) > bg_rois_per_this_image:\n bg_indexes = npr.choice(bg_indexes, size=bg_rois_per_this_image, replace=False)\n\n # indexes selected\n keep_indexes = np.append(fg_indexes, bg_indexes)\n\n # pad more to ensure a fixed minibatch size\n while keep_indexes.shape[0] < rois_per_image:\n gap = np.minimum(len(rois), rois_per_image - keep_indexes.shape[0])\n gap_indexes = npr.choice(range(len(rois)), size=gap, replace=False)\n keep_indexes = np.append(keep_indexes, gap_indexes)\n\n # select labels\n labels = labels[keep_indexes]\n # set labels of bg_rois to be 0\n labels[fg_rois_per_this_image:] = 0\n rois = rois[keep_indexes]\n\n # load or compute bbox_target\n if bbox_targets is not None:\n bbox_target_data = bbox_targets[keep_indexes, :]\n else:\n targets = bbox_transform(rois[:, 1:], gt_boxes[gt_assignment[keep_indexes], :4])\n if cfg.TRAIN.BBOX_NORMALIZATION_PRECOMPUTED:\n targets = ((targets - np.array(cfg.TRAIN.BBOX_MEANS))\n / np.array(cfg.TRAIN.BBOX_STDS))\n bbox_target_data = np.hstack((labels[:, np.newaxis], targets))\n\n bbox_targets, bbox_weights = \\\n expand_bbox_regression_targets(bbox_target_data, num_classes, cfg)\n\n return rois, labels, bbox_targets, bbox_weights\n\n" }, { "path": "rfcn/core/tester.py", "content": "# --------------------------------------------------------\n# Deep Feature Flow\n# Copyright (c) 2017 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Modified by Yuwen Xiong\n# --------------------------------------------------------\n# Based on:\n# MX-RCNN\n# Copyright (c) 2016 by Contributors\n# Licence under The Apache 2.0 License\n# https://github.com/ijkguo/mx-rcnn/\n# --------------------------------------------------------\n\nimport cPickle\nimport os\nimport time\nimport mxnet as mx\nimport numpy as np\n\nfrom module import MutableModule\nfrom utils import image\nfrom bbox.bbox_transform import bbox_pred, clip_boxes\nfrom nms.nms import py_nms_wrapper, cpu_nms_wrapper, gpu_nms_wrapper\nfrom utils.PrefetchingIter import PrefetchingIter\n\n\nclass Predictor(object):\n def __init__(self, symbol, data_names, label_names,\n context=mx.cpu(), max_data_shapes=None,\n provide_data=None, provide_label=None,\n arg_params=None, aux_params=None):\n self._mod = MutableModule(symbol, data_names, label_names,\n context=context, max_data_shapes=max_data_shapes)\n self._mod.bind(provide_data, provide_label, for_training=False)\n self._mod.init_params(arg_params=arg_params, aux_params=aux_params)\n\n def predict(self, data_batch):\n self._mod.forward(data_batch)\n # [dict(zip(self._mod.output_names, _)) for _ in zip(*self._mod.get_outputs(merge_multi_context=False))]\n return [dict(zip(self._mod.output_names, _)) for _ in zip(*self._mod.get_outputs(merge_multi_context=False))]\n\n\ndef im_proposal(predictor, data_batch, data_names, scales):\n output_all = predictor.predict(data_batch)\n\n data_dict_all = [dict(zip(data_names, data_batch.data[i])) for i in xrange(len(data_batch.data))]\n scores_all = []\n boxes_all = []\n\n for output, data_dict, scale in zip(output_all, data_dict_all, scales):\n # drop the batch index\n boxes = output['rois_output'].asnumpy()[:, 1:]\n scores = output['rois_score'].asnumpy()\n\n # transform to original scale\n boxes = boxes / scale\n scores_all.append(scores)\n boxes_all.append(boxes)\n\n return scores_all, boxes_all, data_dict_all\n\n\ndef generate_proposals(predictor, test_data, imdb, cfg, vis=False, thresh=0.):\n \"\"\"\n Generate detections results using RPN.\n :param predictor: Predictor\n :param test_data: data iterator, must be non-shuffled\n :param imdb: image database\n :param vis: controls visualization\n :param thresh: thresh for valid detections\n :return: list of detected boxes\n \"\"\"\n assert vis or not test_data.shuffle\n data_names = [k[0] for k in test_data.provide_data[0]]\n\n if not isinstance(test_data, PrefetchingIter):\n test_data = PrefetchingIter(test_data)\n\n idx = 0\n t = time.clock()\n imdb_boxes = list()\n original_boxes = list()\n for im_info, data_batch in test_data:\n t1 = time.clock() - t\n t = time.clock()\n\n scales = [iim_info[0, 2] for iim_info in im_info]\n scores_all, boxes_all, data_dict_all = im_proposal(predictor, data_batch, data_names, scales)\n t2 = time.clock() - t\n t = time.clock()\n for delta, (scores, boxes, data_dict, scale) in enumerate(zip(scores_all, boxes_all, data_dict_all, scales)):\n # assemble proposals\n dets = np.hstack((boxes, scores))\n original_boxes.append(dets)\n\n # filter proposals\n keep = np.where(dets[:, 4:] > thresh)[0]\n dets = dets[keep, :]\n imdb_boxes.append(dets)\n\n if vis:\n vis_all_detection(data_dict['data'].asnumpy(), [dets], ['obj'], scale, cfg)\n\n print 'generating %d/%d' % (idx + 1, imdb.num_images), 'proposal %d' % (dets.shape[0]), \\\n 'data %.4fs net %.4fs' % (t1, t2 / test_data.batch_size)\n idx += 1\n\n\n assert len(imdb_boxes) == imdb.num_images, 'calculations not complete'\n\n # save results\n rpn_folder = os.path.join(imdb.result_path, 'rpn_data')\n if not os.path.exists(rpn_folder):\n os.mkdir(rpn_folder)\n\n rpn_file = os.path.join(rpn_folder, imdb.name + '_rpn.pkl')\n with open(rpn_file, 'wb') as f:\n cPickle.dump(imdb_boxes, f, cPickle.HIGHEST_PROTOCOL)\n\n if thresh > 0:\n full_rpn_file = os.path.join(rpn_folder, imdb.name + '_full_rpn.pkl')\n with open(full_rpn_file, 'wb') as f:\n cPickle.dump(original_boxes, f, cPickle.HIGHEST_PROTOCOL)\n\n print 'wrote rpn proposals to {}'.format(rpn_file)\n return imdb_boxes\n\n\ndef im_detect(predictor, data_batch, data_names, scales, cfg):\n output_all = predictor.predict(data_batch)\n\n data_dict_all = [dict(zip(data_names, idata)) for idata in data_batch.data]\n scores_all = []\n pred_boxes_all = []\n for output, data_dict, scale in zip(output_all, data_dict_all, scales):\n if cfg.TEST.HAS_RPN:\n rois = output['rois_output'].asnumpy()[:, 1:]\n else:\n rois = data_dict['rois'].asnumpy().reshape((-1, 5))[:, 1:]\n im_shape = data_dict['data'].shape\n\n # save output\n scores = output['cls_prob_reshape_output'].asnumpy()[0]\n bbox_deltas = output['bbox_pred_reshape_output'].asnumpy()[0]\n\n # post processing\n pred_boxes = bbox_pred(rois, bbox_deltas)\n pred_boxes = clip_boxes(pred_boxes, im_shape[-2:])\n\n # we used scaled image & roi to train, so it is necessary to transform them back\n pred_boxes = pred_boxes / scale\n\n scores_all.append(scores)\n pred_boxes_all.append(pred_boxes)\n return scores_all, pred_boxes_all, data_dict_all\n\ndef im_batch_detect(predictor, data_batch, data_names, scales, cfg):\n output_all = predictor.predict(data_batch)\n\n data_dict_all = [dict(zip(data_names, data_batch.data[i])) for i in xrange(len(data_batch.data))]\n scores_all = []\n pred_boxes_all = []\n for output, data_dict, scale in zip(output_all, data_dict_all, scales):\n im_infos = data_dict['im_info'].asnumpy()\n # save output\n scores = output['cls_prob_reshape_output'].asnumpy()[0]\n bbox_deltas = output['bbox_pred_reshape_output'].asnumpy()[0]\n rois = output['rois_output'].asnumpy()\n for im_idx in xrange(im_infos.shape[0]):\n bb_idxs = np.where(rois[:,0] == im_idx)[0]\n im_shape = im_infos[im_idx, :2].astype(np.int)\n\n # post processing\n pred_boxes = bbox_pred(rois[bb_idxs, 1:], bbox_deltas[bb_idxs, :])\n pred_boxes = clip_boxes(pred_boxes, im_shape)\n\n # we used scaled image & roi to train, so it is necessary to transform them back\n pred_boxes = pred_boxes / scale[im_idx]\n\n scores_all.append(scores[bb_idxs, :])\n pred_boxes_all.append(pred_boxes)\n\n return scores_all, pred_boxes_all, data_dict_all\n\ndef pred_eval(predictor, test_data, imdb, cfg, vis=False, thresh=1e-3, logger=None, ignore_cache=True):\n \"\"\"\n wrapper for calculating offline validation for faster data analysis\n in this example, all threshold are set by hand\n :param predictor: Predictor\n :param test_data: data iterator, must be non-shuffle\n :param imdb: image database\n :param vis: controls visualization\n :param thresh: valid detection threshold\n :return:\n \"\"\"\n\n det_file = os.path.join(imdb.result_path, imdb.name + '_detections.pkl')\n if os.path.exists(det_file) and not ignore_cache:\n with open(det_file, 'rb') as fid:\n all_boxes = cPickle.load(fid)\n info_str = imdb.evaluate_detections(all_boxes)\n if logger:\n logger.info('evaluate detections: \\n{}'.format(info_str))\n return\n\n assert vis or not test_data.shuffle\n data_names = [k[0] for k in test_data.provide_data[0]]\n\n if not isinstance(test_data, PrefetchingIter):\n test_data = PrefetchingIter(test_data)\n\n nms = py_nms_wrapper(cfg.TEST.NMS)\n\n # limit detections to max_per_image over all classes\n max_per_image = cfg.TEST.max_per_image\n\n num_images = imdb.num_images\n # all detections are collected into:\n # all_boxes[cls][image] = N x 5 array of detections in\n # (x1, y1, x2, y2, score)\n all_boxes = [[[] for _ in range(num_images)]\n for _ in range(imdb.num_classes)]\n\n idx = 0\n data_time, net_time, post_time = 0.0, 0.0, 0.0\n t = time.clock()\n for im_info, data_batch in test_data:\n t1 = time.clock() - t\n t = time.clock()\n\n scales = [iim_info[0, 2] for iim_info in im_info]\n scores_all, boxes_all, data_dict_all = im_detect(predictor, data_batch, data_names, scales, cfg)\n\n t2 = time.clock() - t\n t = time.clock()\n for delta, (scores, boxes, data_dict) in enumerate(zip(scores_all, boxes_all, data_dict_all)):\n for j in range(1, imdb.num_classes):\n indexes = np.where(scores[:, j] > thresh)[0]\n cls_scores = scores[indexes, j, np.newaxis]\n cls_boxes = boxes[indexes, 4:8] if cfg.CLASS_AGNOSTIC else boxes[indexes, j * 4:(j + 1) * 4]\n cls_dets = np.hstack((cls_boxes, cls_scores))\n keep = nms(cls_dets)\n all_boxes[j][idx+delta] = cls_dets[keep, :]\n\n if max_per_image > 0:\n image_scores = np.hstack([all_boxes[j][idx+delta][:, -1]\n for j in range(1, imdb.num_classes)])\n if len(image_scores) > max_per_image:\n image_thresh = np.sort(image_scores)[-max_per_image]\n for j in range(1, imdb.num_classes):\n keep = np.where(all_boxes[j][idx+delta][:, -1] >= image_thresh)[0]\n all_boxes[j][idx+delta] = all_boxes[j][idx+delta][keep, :]\n\n if vis:\n boxes_this_image = [[]] + [all_boxes[j][idx+delta] for j in range(1, imdb.num_classes)]\n vis_all_detection(data_dict['data'].asnumpy(), boxes_this_image, imdb.classes, scales[delta], cfg)\n\n idx += test_data.batch_size\n t3 = time.clock() - t\n t = time.clock()\n data_time += t1\n net_time += t2\n post_time += t3\n print 'testing {}/{} data {:.4f}s net {:.4f}s post {:.4f}s'.format(idx, imdb.num_images, data_time / idx * test_data.batch_size, net_time / idx * test_data.batch_size, post_time / idx * test_data.batch_size)\n if logger:\n logger.info('testing {}/{} data {:.4f}s net {:.4f}s post {:.4f}s'.format(idx, imdb.num_images, data_time / idx * test_data.batch_size, net_time / idx * test_data.batch_size, post_time / idx * test_data.batch_size))\n\n with open(det_file, 'wb') as f:\n cPickle.dump(all_boxes, f, protocol=cPickle.HIGHEST_PROTOCOL)\n\n info_str = imdb.evaluate_detections(all_boxes)\n if logger:\n logger.info('evaluate detections: \\n{}'.format(info_str))\n\n\ndef vis_all_detection(im_array, detections, class_names, scale, cfg, threshold=1e-3):\n \"\"\"\n visualize all detections in one image\n :param im_array: [b=1 c h w] in rgb\n :param detections: [ numpy.ndarray([[x1 y1 x2 y2 score]]) for j in classes ]\n :param class_names: list of names in imdb\n :param scale: visualize the scaled image\n :return:\n \"\"\"\n import matplotlib.pyplot as plt\n import random\n im = image.transform_inverse(im_array, cfg.network.PIXEL_MEANS)\n plt.imshow(im)\n for j, name in enumerate(class_names):\n if name == '__background__':\n continue\n color = (random.random(), random.random(), random.random()) # generate a random color\n dets = detections[j]\n for det in dets:\n bbox = det[:4] * scale\n score = det[-1]\n if score < threshold:\n continue\n rect = plt.Rectangle((bbox[0], bbox[1]),\n bbox[2] - bbox[0],\n bbox[3] - bbox[1], fill=False,\n edgecolor=color, linewidth=3.5)\n plt.gca().add_patch(rect)\n plt.gca().text(bbox[0], bbox[1] - 2,\n '{:s} {:.3f}'.format(name, score),\n bbox=dict(facecolor=color, alpha=0.5), fontsize=12, color='white')\n plt.show()\n\n\ndef draw_all_detection(im_array, detections, class_names, scale, cfg, threshold=1e-1):\n \"\"\"\n visualize all detections in one image\n :param im_array: [b=1 c h w] in rgb\n :param detections: [ numpy.ndarray([[x1 y1 x2 y2 score]]) for j in classes ]\n :param class_names: list of names in imdb\n :param scale: visualize the scaled image\n :return:\n \"\"\"\n import cv2\n import random\n color_white = (255, 255, 255)\n im = image.transform_inverse(im_array, cfg.network.PIXEL_MEANS)\n # change to bgr\n im = cv2.cvtColor(im, cv2.COLOR_RGB2BGR)\n for j, name in enumerate(class_names):\n if name == '__background__':\n continue\n color = (random.randint(0, 256), random.randint(0, 256), random.randint(0, 256)) # generate a random color\n dets = detections[j]\n for det in dets:\n bbox = det[:4] * scale\n score = det[-1]\n if score < threshold:\n continue\n bbox = map(int, bbox)\n cv2.rectangle(im, (bbox[0], bbox[1]), (bbox[2], bbox[3]), color=color, thickness=2)\n cv2.putText(im, '%s %.3f' % (class_names[j], score), (bbox[0], bbox[1] + 10),\n color=color_white, fontFace=cv2.FONT_HERSHEY_COMPLEX, fontScale=0.5)\n return im\n" }, { "path": "rfcn/demo.py", "content": "# --------------------------------------------------------\n# Deep Feature Flow\n# Copyright (c) 2017 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Written by Xizhou Zhu, Yi Li, Haochen Zhang\n# --------------------------------------------------------\n\nimport _init_paths\n\nimport argparse\nimport os\nimport glob\nimport sys\nimport logging\nimport pprint\nimport cv2\nfrom config.config import config, update_config\nfrom utils.image import resize, transform\nimport numpy as np\n# get config\nos.environ['PYTHONUNBUFFERED'] = '1'\nos.environ['MXNET_CUDNN_AUTOTUNE_DEFAULT'] = '0'\nos.environ['MXNET_ENABLE_GPU_P2P'] = '0'\ncur_path = os.path.abspath(os.path.dirname(__file__))\nupdate_config(cur_path + '/../experiments/rfcn/cfgs/rfcn_vid_demo.yaml')\n\nsys.path.insert(0, os.path.join(cur_path, '../external/mxnet', config.MXNET_VERSION))\nimport mxnet as mx\nfrom core.tester import im_detect, Predictor\nfrom symbols import *\nfrom utils.load_model import load_param\nfrom utils.show_boxes import show_boxes, draw_boxes\nfrom utils.tictoc import tic, toc\nfrom nms.nms import py_nms_wrapper, cpu_nms_wrapper, gpu_nms_wrapper\n\ndef parse_args():\n parser = argparse.ArgumentParser(description='Show Deep Feature Flow demo')\n args = parser.parse_args()\n return args\n\nargs = parse_args()\n\ndef main():\n # get symbol\n pprint.pprint(config)\n config.symbol = 'resnet_v1_101_rfcn'\n model = '/../model/rfcn_vid'\n sym_instance = eval(config.symbol + '.' + config.symbol)()\n sym = sym_instance.get_test_symbol(config)\n\n # set up class names\n num_classes = 31\n classes = ['airplane', 'antelope', 'bear', 'bicycle',\n 'bird', 'bus', 'car', 'cattle',\n 'dog', 'domestic_cat', 'elephant', 'fox',\n 'giant_panda', 'hamster', 'horse', 'lion',\n 'lizard', 'monkey', 'motorcycle', 'rabbit',\n 'red_panda', 'sheep', 'snake', 'squirrel',\n 'tiger', 'train', 'turtle', 'watercraft',\n 'whale', 'zebra']\n\n # load demo data\n image_names = glob.glob(cur_path + '/../demo/ILSVRC2015_val_00007010/*.JPEG')\n output_dir = cur_path + '/../demo/rfcn/'\n if not os.path.exists(output_dir):\n os.makedirs(output_dir)\n\n #\n\n data = []\n for im_name in image_names:\n assert os.path.exists(im_name), ('%s does not exist'.format(im_name))\n im = cv2.imread(im_name, cv2.IMREAD_COLOR | cv2.IMREAD_IGNORE_ORIENTATION)\n target_size = config.SCALES[0][0]\n max_size = config.SCALES[0][1]\n im, im_scale = resize(im, target_size, max_size, stride=config.network.IMAGE_STRIDE)\n im_tensor = transform(im, config.network.PIXEL_MEANS)\n im_info = np.array([[im_tensor.shape[2], im_tensor.shape[3], im_scale]], dtype=np.float32)\n data.append({'data': im_tensor, 'im_info': im_info})\n\n\n # get predictor\n data_names = ['data', 'im_info']\n label_names = []\n data = [[mx.nd.array(data[i][name]) for name in data_names] for i in xrange(len(data))]\n max_data_shape = [[('data', (1, 3, max([v[0] for v in config.SCALES]), max([v[1] for v in config.SCALES])))]]\n provide_data = [[(k, v.shape) for k, v in zip(data_names, data[i])] for i in xrange(len(data))]\n provide_label = [None for i in xrange(len(data))]\n arg_params, aux_params = load_param(cur_path + model, 0, process=True)\n predictor = Predictor(sym, data_names, label_names,\n context=[mx.gpu(0)], max_data_shapes=max_data_shape,\n provide_data=provide_data, provide_label=provide_label,\n arg_params=arg_params, aux_params=aux_params)\n nms = gpu_nms_wrapper(config.TEST.NMS, 0)\n\n # warm up\n for j in xrange(2):\n data_batch = mx.io.DataBatch(data=[data[0]], label=[], pad=0, index=0,\n provide_data=[[(k, v.shape) for k, v in zip(data_names, data[0])]],\n provide_label=[None])\n scales = [data_batch.data[i][1].asnumpy()[0, 2] for i in xrange(len(data_batch.data))]\n scores, boxes, data_dict = im_detect(predictor, data_batch, data_names, scales, config)\n\n # test\n time = 0\n count = 0\n for idx, im_name in enumerate(image_names):\n data_batch = mx.io.DataBatch(data=[data[idx]], label=[], pad=0, index=idx,\n provide_data=[[(k, v.shape) for k, v in zip(data_names, data[idx])]],\n provide_label=[None])\n scales = [data_batch.data[i][1].asnumpy()[0, 2] for i in xrange(len(data_batch.data))]\n\n tic()\n scores, boxes, data_dict = im_detect(predictor, data_batch, data_names, scales, config)\n time += toc()\n count += 1\n print 'testing {} {:.4f}s'.format(im_name, time/count)\n\n boxes = boxes[0].astype('f')\n scores = scores[0].astype('f')\n dets_nms = []\n for j in range(1, scores.shape[1]):\n cls_scores = scores[:, j, np.newaxis]\n cls_boxes = boxes[:, 4:8] if config.CLASS_AGNOSTIC else boxes[:, j * 4:(j + 1) * 4]\n cls_dets = np.hstack((cls_boxes, cls_scores))\n keep = nms(cls_dets)\n cls_dets = cls_dets[keep, :]\n cls_dets = cls_dets[cls_dets[:, -1] > 0.7, :]\n dets_nms.append(cls_dets)\n\n # visualize\n im = cv2.imread(im_name)\n im = cv2.cvtColor(im, cv2.COLOR_BGR2RGB)\n # show_boxes(im, dets_nms, classes, 1)\n out_im = draw_boxes(im, dets_nms, classes, 1)\n _, filename = os.path.split(im_name)\n cv2.imwrite(output_dir + filename,out_im)\n\n print 'done'\n\nif __name__ == '__main__':\n main()\n" }, { "path": "rfcn/demo_batch.py", "content": "# --------------------------------------------------------\n# Deep Feature Flow\n# Copyright (c) 2017 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Written by Xizhou Zhu, Yi Li, Haochen Zhang\n# --------------------------------------------------------\n\nimport _init_paths\n\nimport argparse\nimport os\nimport glob\nimport sys\nimport logging\nimport pprint\nimport cv2\nfrom config.config import config, update_config\nfrom utils.image import resize, transform\nimport numpy as np\n# get config\nos.environ['PYTHONUNBUFFERED'] = '1'\nos.environ['MXNET_CUDNN_AUTOTUNE_DEFAULT'] = '0'\nos.environ['MXNET_ENABLE_GPU_P2P'] = '0'\ncur_path = os.path.abspath(os.path.dirname(__file__))\nupdate_config(cur_path + '/../experiments/rfcn/cfgs/rfcn_vid_demo.yaml')\n\nsys.path.insert(0, os.path.join(cur_path, '../external/mxnet', config.MXNET_VERSION))\nimport mxnet as mx\nfrom core.tester import im_batch_detect, Predictor\nfrom symbols import *\nfrom utils.load_model import load_param\nfrom utils.show_boxes import show_boxes, draw_boxes\nfrom utils.tictoc import tic, toc\nfrom nms.nms import py_nms_wrapper, cpu_nms_wrapper, gpu_nms_wrapper\n\ndef parse_args():\n parser = argparse.ArgumentParser(description='Show Deep Feature Flow demo')\n args = parser.parse_args()\n return args\n\nargs = parse_args()\n\ndef main():\n # get symbol\n pprint.pprint(config)\n config.symbol = 'resnet_v1_101_rfcn'\n model = '/../model/rfcn_vid'\n sym_instance = eval(config.symbol + '.' + config.symbol)()\n sym = sym_instance.get_test_symbol(config)\n\n # set up class names\n num_classes = 31\n classes = ['airplane', 'antelope', 'bear', 'bicycle',\n 'bird', 'bus', 'car', 'cattle',\n 'dog', 'domestic_cat', 'elephant', 'fox',\n 'giant_panda', 'hamster', 'horse', 'lion',\n 'lizard', 'monkey', 'motorcycle', 'rabbit',\n 'red_panda', 'sheep', 'snake', 'squirrel',\n 'tiger', 'train', 'turtle', 'watercraft',\n 'whale', 'zebra']\n\n # load demo data\n image_names = glob.glob(cur_path + '/../demo/ILSVRC2015_val_00007010/*.JPEG')\n output_dir = cur_path + '/../demo/rfcn_batch/'\n if not os.path.exists(output_dir):\n os.makedirs(output_dir)\n batch_size = 10\n\n #\n\n data = []\n cur_im_tensor = []\n im_info_tensor = []\n image_names_list = []\n image_names_batch = []\n for idx, im_name in enumerate(image_names):\n assert os.path.exists(im_name), ('%s does not exist'.format(im_name))\n im = cv2.imread(im_name, cv2.IMREAD_COLOR | cv2.IMREAD_IGNORE_ORIENTATION)\n target_size = config.SCALES[0][0]\n max_size = config.SCALES[0][1]\n im, im_scale = resize(im, target_size, max_size, stride=config.network.IMAGE_STRIDE)\n im_tensor = transform(im, config.network.PIXEL_MEANS)\n im_info = np.array([[im_tensor.shape[2], im_tensor.shape[3], im_scale]], dtype=np.float32)\n\n cur_im_tensor.append(im_tensor)\n im_info_tensor.append(im_info)\n image_names_batch.append(im_name)\n if (idx+1) % batch_size == 0 or idx == len(image_names) - 1:\n data.append({'data': np.concatenate(cur_im_tensor), 'im_info': np.concatenate(im_info_tensor)})\n cur_im_tensor = []\n im_info_tensor = []\n image_names_list.append(image_names_batch)\n image_names_batch = []\n\n\n # get predictor\n data_names = ['data', 'im_info']\n label_names = []\n data = [[mx.nd.array(data[i][name]) for name in data_names] for i in xrange(len(data))]\n max_data_shape = [[('data', (batch_size, 3, max([v[0] for v in config.SCALES]), max([v[1] for v in config.SCALES])))]]\n provide_data = [[(k, v.shape) for k, v in zip(data_names, data[i])] for i in xrange(len(data))]\n provide_label = [None for i in xrange(len(data))]\n arg_params, aux_params = load_param(cur_path + model, 0, process=True)\n predictor = Predictor(sym, data_names, label_names,\n context=[mx.gpu(0)], max_data_shapes=max_data_shape,\n provide_data=provide_data, provide_label=provide_label,\n arg_params=arg_params, aux_params=aux_params)\n nms = gpu_nms_wrapper(config.TEST.NMS, 0)\n\n # warm up\n for j in xrange(1):\n data_batch = mx.io.DataBatch(data=[data[j]], label=[], pad=0, index=0,\n provide_data=[[(k, v.shape) for k, v in zip(data_names, data[j])]],\n provide_label=[None])\n scales = [data_batch.data[i][1].asnumpy()[:, 2] for i in xrange(len(data_batch.data))]\n scores_all, boxes_all, data_dict = im_batch_detect(predictor, data_batch, data_names, scales, config)\n\n print \"warmup done\"\n # test\n time = 0\n count = 0\n for idx, im_names in enumerate(image_names_list):\n data_batch = mx.io.DataBatch(data=[data[idx]], label=[], pad=0, index=idx,\n provide_data=[[(k, v.shape) for k, v in zip(data_names, data[idx])]],\n provide_label=[None])\n scales = [data_batch.data[i][1].asnumpy()[:, 2] for i in xrange(len(data_batch.data))]\n\n tic()\n scores_all, boxes_all, data_dict = im_batch_detect(predictor, data_batch, data_names, scales, config)\n time += toc()\n count += len(scores_all)\n print 'testing {} {:.4f}s x {:d}'.format(im_names[0], time/count, len(scores_all))\n\n for batch_idx in xrange(len(scores_all)):\n boxes = boxes_all[batch_idx].astype('f')\n scores = scores_all[batch_idx].astype('f')\n dets_nms = []\n for j in range(1, scores.shape[1]):\n cls_scores = scores[:, j, np.newaxis]\n cls_boxes = boxes[:, 4:8] if config.CLASS_AGNOSTIC else boxes[:, j * 4:(j + 1) * 4]\n cls_dets = np.hstack((cls_boxes, cls_scores))\n keep = nms(cls_dets)\n cls_dets = cls_dets[keep, :]\n cls_dets = cls_dets[cls_dets[:, -1] > 0.7, :]\n dets_nms.append(cls_dets)\n # visualize\n im = cv2.imread(im_names[batch_idx])\n im = cv2.cvtColor(im, cv2.COLOR_BGR2RGB)\n # show_boxes(im, dets_nms, classes, 1)\n out_im = draw_boxes(im, dets_nms, classes, 1)\n _, filename = os.path.split(im_names[batch_idx])\n cv2.imwrite(output_dir + filename,out_im)\n\n print 'done'\n\nif __name__ == '__main__':\n main()\n" }, { "path": "rfcn/function/__init__.py", "content": "" }, { "path": "rfcn/function/test_rcnn.py", "content": "# --------------------------------------------------------\n# Deep Feature Flow\n# Copyright (c) 2017 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Modified by Yuwen Xiong\n# --------------------------------------------------------\n# Based on:\n# MX-RCNN\n# Copyright (c) 2016 by Contributors\n# Licence under The Apache 2.0 License\n# https://github.com/ijkguo/mx-rcnn/\n# --------------------------------------------------------\n\nimport argparse\nimport pprint\nimport logging\nimport time\nimport os\nimport mxnet as mx\n\nfrom symbols import *\nfrom dataset import *\nfrom core.loader import TestLoader\nfrom core.tester import Predictor, pred_eval\nfrom utils.load_model import load_param\n\n\ndef test_rcnn(cfg, dataset, image_set, root_path, dataset_path,\n ctx, prefix, epoch,\n vis, ignore_cache, shuffle, has_rpn, proposal, thresh, logger=None, output_path=None):\n if not logger:\n assert False, 'require a logger'\n\n # print cfg\n pprint.pprint(cfg)\n logger.info('testing cfg:{}\\n'.format(pprint.pformat(cfg)))\n\n # load symbol and testing data\n sym_instance = eval(cfg.symbol + '.' + cfg.symbol)()\n sym = sym_instance.get_test_symbol(cfg)\n imdb = eval(dataset)(image_set, root_path, dataset_path, result_path=output_path)\n roidb = imdb.gt_roidb()\n\n # get test data iter\n test_data = TestLoader(roidb, cfg, batch_size=len(ctx), shuffle=shuffle, has_rpn=has_rpn)\n\n # load model\n arg_params, aux_params = load_param(prefix, epoch, process=True)\n\n # infer shape\n data_shape_dict = dict(test_data.provide_data_single)\n sym_instance.infer_shape(data_shape_dict)\n\n sym_instance.check_parameter_shapes(arg_params, aux_params, data_shape_dict, is_train=False)\n\n # decide maximum shape\n data_names = [k[0] for k in test_data.provide_data_single]\n label_names = None\n max_data_shape = [[('data', (1, 3, max([v[0] for v in cfg.SCALES]), max([v[1] for v in cfg.SCALES])))]]\n if not has_rpn:\n max_data_shape.append(('rois', (cfg.TEST.PROPOSAL_POST_NMS_TOP_N + 30, 5)))\n\n # create predictor\n predictor = Predictor(sym, data_names, label_names,\n context=ctx, max_data_shapes=max_data_shape,\n provide_data=test_data.provide_data, provide_label=test_data.provide_label,\n arg_params=arg_params, aux_params=aux_params)\n\n # start detection\n pred_eval(predictor, test_data, imdb, cfg, vis=vis, ignore_cache=ignore_cache, thresh=thresh, logger=logger)\n\n" }, { "path": "rfcn/function/test_rpn.py", "content": "# --------------------------------------------------------\n# Deep Feature Flow\n# Copyright (c) 2017 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Modified by Yuwen Xiong\n# --------------------------------------------------------\n# Based on:\n# MX-RCNN\n# Copyright (c) 2016 by Contributors\n# Licence under The Apache 2.0 License\n# https://github.com/ijkguo/mx-rcnn/\n# --------------------------------------------------------\n\nimport argparse\nimport pprint\nimport logging\nimport mxnet as mx\n\nfrom symbols import *\nfrom dataset import *\nfrom core.loader import TestLoader\nfrom core.tester import Predictor, generate_proposals\nfrom utils.load_model import load_param\n\n\ndef test_rpn(cfg, dataset, image_set, root_path, dataset_path,\n ctx, prefix, epoch,\n vis, shuffle, thresh, logger=None, output_path=None):\n # set up logger\n if not logger:\n logging.basicConfig()\n logger = logging.getLogger()\n logger.setLevel(logging.INFO)\n\n # rpn generate proposal cfg\n cfg.TEST.HAS_RPN = True\n\n # print cfg\n pprint.pprint(cfg)\n logger.info('testing rpn cfg:{}\\n'.format(pprint.pformat(cfg)))\n\n # load symbol\n sym_instance = eval(cfg.symbol + '.' + cfg.symbol)()\n sym = sym_instance.get_symbol_rpn(cfg, is_train=False)\n\n # load dataset and prepare imdb for training\n imdb = eval(dataset)(image_set, root_path, dataset_path, result_path=output_path)\n roidb = imdb.gt_roidb()\n test_data = TestLoader(roidb, cfg, batch_size=len(ctx), shuffle=shuffle, has_rpn=True)\n\n # load model\n arg_params, aux_params = load_param(prefix, epoch)\n\n # infer shape\n data_shape_dict = dict(test_data.provide_data_single)\n sym_instance.infer_shape(data_shape_dict)\n\n # check parameters\n sym_instance.check_parameter_shapes(arg_params, aux_params, data_shape_dict, is_train=False)\n\n # decide maximum shape\n data_names = [k[0] for k in test_data.provide_data[0]]\n label_names = None if test_data.provide_label[0] is None else [k[0] for k in test_data.provide_label[0]]\n max_data_shape = [[('data', (1, 3, max([v[0] for v in cfg.SCALES]), max([v[1] for v in cfg.SCALES])))]]\n\n # create predictor\n predictor = Predictor(sym, data_names, label_names,\n context=ctx, max_data_shapes=max_data_shape,\n provide_data=test_data.provide_data, provide_label=test_data.provide_label,\n arg_params=arg_params, aux_params=aux_params)\n\n # start testing\n imdb_boxes = generate_proposals(predictor, test_data, imdb, cfg, vis=vis, thresh=thresh)\n\n all_log_info = imdb.evaluate_recall(roidb, candidate_boxes=imdb_boxes)\n logger.info(all_log_info)\n" }, { "path": "rfcn/function/train_rcnn.py", "content": "# --------------------------------------------------------\n# Deep Feature Flow\n# Copyright (c) 2017 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Modified by Yuwen Xiong\n# --------------------------------------------------------\n# Based on:\n# MX-RCNN\n# Copyright (c) 2016 by Contributors\n# Licence under The Apache 2.0 License\n# https://github.com/ijkguo/mx-rcnn/\n# --------------------------------------------------------\n\nimport argparse\nimport logging\nimport pprint\nimport os\nimport mxnet as mx\n\nfrom symbols import *\nfrom core import callback, metric\nfrom core.loader import ROIIter\nfrom core.module import MutableModule\nfrom bbox.bbox_regression import add_bbox_regression_targets\nfrom utils.load_data import load_proposal_roidb, merge_roidb, filter_roidb\nfrom utils.load_model import load_param\nfrom utils.PrefetchingIter import PrefetchingIter\nfrom utils.lr_scheduler import WarmupMultiFactorScheduler\n\n\ndef train_rcnn(cfg, dataset, image_set, root_path, dataset_path,\n frequent, kvstore, flip, shuffle, resume,\n ctx, pretrained, epoch, prefix, begin_epoch, end_epoch,\n train_shared, lr, lr_step, proposal, logger=None, output_path=None):\n # set up logger\n if not logger:\n logging.basicConfig()\n logger = logging.getLogger()\n logger.setLevel(logging.INFO)\n\n # load symbol\n sym_instance = eval(cfg.symbol + '.' + cfg.symbol)()\n sym = sym_instance.get_symbol_rfcn(cfg, is_train=True)\n\n # setup multi-gpu\n batch_size = len(ctx)\n input_batch_size = cfg.TRAIN.BATCH_IMAGES * batch_size\n\n # print cfg\n pprint.pprint(cfg)\n logger.info('training rcnn cfg:{}\\n'.format(pprint.pformat(cfg)))\n\n # load dataset and prepare imdb for training\n image_sets = [iset for iset in image_set.split('+')]\n roidbs = [load_proposal_roidb(dataset, image_set, root_path, dataset_path,\n proposal=proposal, append_gt=True, flip=flip, result_path=output_path)\n for image_set in image_sets]\n roidb = merge_roidb(roidbs)\n roidb = filter_roidb(roidb, cfg)\n means, stds = add_bbox_regression_targets(roidb, cfg)\n\n # load training data\n train_data = ROIIter(roidb, cfg, batch_size=input_batch_size, shuffle=shuffle,\n ctx=ctx, aspect_grouping=cfg.TRAIN.ASPECT_GROUPING)\n\n # infer max shape\n max_data_shape = [('data', (cfg.TRAIN.BATCH_IMAGES, 3, max([v[0] for v in cfg.SCALES]), max([v[1] for v in cfg.SCALES])))]\n\n # infer shape\n data_shape_dict = dict(train_data.provide_data_single + train_data.provide_label_single)\n sym_instance.infer_shape(data_shape_dict)\n\n # load and initialize params\n if resume:\n print('continue training from ', begin_epoch)\n arg_params, aux_params = load_param(prefix, begin_epoch, convert=True)\n else:\n arg_params, aux_params = load_param(pretrained, epoch, convert=True)\n sym_instance.init_weight_rfcn(cfg, arg_params, aux_params)\n\n # check parameter shapes\n sym_instance.check_parameter_shapes(arg_params, aux_params, data_shape_dict)\n\n # prepare training\n # create solver\n data_names = [k[0] for k in train_data.provide_data_single]\n label_names = [k[0] for k in train_data.provide_label_single]\n if train_shared:\n fixed_param_prefix = cfg.network.FIXED_PARAMS_SHARED\n else:\n fixed_param_prefix = cfg.network.FIXED_PARAMS\n mod = MutableModule(sym, data_names=data_names, label_names=label_names,\n logger=logger, context=ctx,\n max_data_shapes=[max_data_shape for _ in range(batch_size)], fixed_param_prefix=fixed_param_prefix)\n\n if cfg.TRAIN.RESUME:\n mod._preload_opt_states = '%s-%04d.states'%(prefix, begin_epoch)\n\n\n # decide training params\n # metric\n eval_metric = metric.RCNNAccMetric(cfg)\n cls_metric = metric.RCNNLogLossMetric(cfg)\n bbox_metric = metric.RCNNL1LossMetric(cfg)\n eval_metrics = mx.metric.CompositeEvalMetric()\n for child_metric in [eval_metric, cls_metric, bbox_metric]:\n eval_metrics.add(child_metric)\n # callback\n batch_end_callback = callback.Speedometer(train_data.batch_size, frequent=frequent)\n epoch_end_callback = [mx.callback.module_checkpoint(mod, prefix, period=1, save_optimizer_states=True),\n callback.do_checkpoint(prefix, means, stds)]\n # decide learning rate\n base_lr = lr\n lr_factor = cfg.TRAIN.lr_factor\n lr_epoch = [float(epoch) for epoch in lr_step.split(',')]\n lr_epoch_diff = [epoch - begin_epoch for epoch in lr_epoch if epoch > begin_epoch]\n lr = base_lr * (lr_factor ** (len(lr_epoch) - len(lr_epoch_diff)))\n lr_iters = [int(epoch * len(roidb) / batch_size) for epoch in lr_epoch_diff]\n print('lr', lr, 'lr_epoch_diff', lr_epoch_diff, 'lr_iters', lr_iters)\n lr_scheduler = WarmupMultiFactorScheduler(lr_iters, lr_factor, cfg.TRAIN.warmup, cfg.TRAIN.warmup_lr, cfg.TRAIN.warmup_step)\n # optimizer\n optimizer_params = {'momentum': cfg.TRAIN.momentum,\n 'wd': cfg.TRAIN.wd,\n 'learning_rate': lr,\n 'lr_scheduler': lr_scheduler,\n 'rescale_grad': 1.0,\n 'clip_gradient': None}\n\n # train\n\n if not isinstance(train_data, PrefetchingIter):\n train_data = PrefetchingIter(train_data)\n\n mod.fit(train_data, eval_metric=eval_metrics, epoch_end_callback=epoch_end_callback,\n batch_end_callback=batch_end_callback, kvstore=kvstore,\n optimizer='sgd', optimizer_params=optimizer_params,\n arg_params=arg_params, aux_params=aux_params, begin_epoch=begin_epoch, num_epoch=end_epoch)\n\n" }, { "path": "rfcn/function/train_rpn.py", "content": "# --------------------------------------------------------\n# Deep Feature Flow\n# Copyright (c) 2017 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Modified by Yuwen Xiong\n# --------------------------------------------------------\n# Based on:\n# MX-RCNN\n# Copyright (c) 2016 by Contributors\n# Licence under The Apache 2.0 License\n# https://github.com/ijkguo/mx-rcnn/\n# --------------------------------------------------------\n\nimport argparse\nimport logging\nimport pprint\nimport mxnet as mx\n\nfrom symbols import *\nfrom core import callback, metric\nfrom core.loader import AnchorLoader\nfrom core.module import MutableModule\nfrom utils.load_data import load_gt_roidb, merge_roidb, filter_roidb\nfrom utils.load_model import load_param\nfrom utils.PrefetchingIter import PrefetchingIter\nfrom utils.lr_scheduler import WarmupMultiFactorScheduler\n\n\ndef train_rpn(cfg, dataset, image_set, root_path, dataset_path,\n frequent, kvstore, flip, shuffle, resume,\n ctx, pretrained, epoch, prefix, begin_epoch, end_epoch,\n train_shared, lr, lr_step, logger=None, output_path=None):\n # set up logger\n if not logger:\n logging.basicConfig()\n logger = logging.getLogger()\n logger.setLevel(logging.INFO)\n\n # set up config\n cfg.TRAIN.BATCH_IMAGES = cfg.TRAIN.ALTERNATE.RPN_BATCH_IMAGES\n\n # load symbol\n sym_instance = eval(cfg.symbol + '.' + cfg.symbol)()\n sym = sym_instance.get_symbol_rpn(cfg, is_train=True)\n feat_sym = sym.get_internals()['rpn_cls_score_output']\n\n # setup multi-gpu\n batch_size = len(ctx)\n input_batch_size = cfg.TRAIN.BATCH_IMAGES * batch_size\n\n # print cfg\n pprint.pprint(cfg)\n logger.info('training rpn cfg:{}\\n'.format(pprint.pformat(cfg)))\n\n # load dataset and prepare imdb for training\n image_sets = [iset for iset in image_set.split('+')]\n roidbs = [load_gt_roidb(dataset, image_set, root_path, dataset_path, result_path=output_path,\n flip=flip)\n for image_set in image_sets]\n roidb = merge_roidb(roidbs)\n roidb = filter_roidb(roidb, cfg)\n\n # load training data\n train_data = AnchorLoader(feat_sym, roidb, cfg, batch_size=input_batch_size, shuffle=shuffle,\n ctx=ctx, feat_stride=cfg.network.RPN_FEAT_STRIDE, anchor_scales=cfg.network.ANCHOR_SCALES,\n anchor_ratios=cfg.network.ANCHOR_RATIOS, aspect_grouping=cfg.TRAIN.ASPECT_GROUPING)\n\n # infer max shape\n max_data_shape = [('data', (cfg.TRAIN.BATCH_IMAGES, 3, max([v[0] for v in cfg.SCALES]), max([v[1] for v in cfg.SCALES])))]\n max_data_shape, max_label_shape = train_data.infer_shape(max_data_shape)\n print('providing maximum shape', max_data_shape, max_label_shape)\n\n # infer shape\n data_shape_dict = dict(train_data.provide_data_single + train_data.provide_label_single)\n sym_instance.infer_shape(data_shape_dict)\n\n # load and initialize params\n if resume:\n print('continue training from ', begin_epoch)\n arg_params, aux_params = load_param(prefix, begin_epoch, convert=True)\n else:\n arg_params, aux_params = load_param(pretrained, epoch, convert=True)\n sym_instance.init_weight_rpn(cfg, arg_params, aux_params)\n\n # check parameter shapes\n sym_instance.check_parameter_shapes(arg_params, aux_params, data_shape_dict)\n\n # create solver\n data_names = [k[0] for k in train_data.provide_data_single]\n label_names = [k[0] for k in train_data.provide_label_single]\n if train_shared:\n fixed_param_prefix = cfg.network.FIXED_PARAMS_SHARED\n else:\n fixed_param_prefix = cfg.network.FIXED_PARAMS\n mod = MutableModule(sym, data_names=data_names, label_names=label_names,\n logger=logger, context=ctx, max_data_shapes=[max_data_shape for _ in xrange(batch_size)],\n max_label_shapes=[max_label_shape for _ in xrange(batch_size)], fixed_param_prefix=fixed_param_prefix)\n\n # decide training params\n # metric\n eval_metric = metric.RPNAccMetric()\n cls_metric = metric.RPNLogLossMetric()\n bbox_metric = metric.RPNL1LossMetric()\n eval_metrics = mx.metric.CompositeEvalMetric()\n for child_metric in [eval_metric, cls_metric, bbox_metric]:\n eval_metrics.add(child_metric)\n # callback\n batch_end_callback = callback.Speedometer(train_data.batch_size, frequent=frequent)\n # epoch_end_callback = mx.callback.do_checkpoint(prefix)\n epoch_end_callback = mx.callback.module_checkpoint(mod, prefix, period=1, save_optimizer_states=True)\n # decide learning rate\n base_lr = lr\n lr_factor = cfg.TRAIN.lr_factor\n lr_epoch = [int(epoch) for epoch in lr_step.split(',')]\n lr_epoch_diff = [epoch - begin_epoch for epoch in lr_epoch if epoch > begin_epoch]\n lr = base_lr * (lr_factor ** (len(lr_epoch) - len(lr_epoch_diff)))\n lr_iters = [int(epoch * len(roidb) / batch_size) for epoch in lr_epoch_diff]\n print('lr', lr, 'lr_epoch_diff', lr_epoch_diff, 'lr_iters', lr_iters)\n lr_scheduler = WarmupMultiFactorScheduler(lr_iters, lr_factor, cfg.TRAIN.warmup, cfg.TRAIN.warmup_lr, cfg.TRAIN.warmup_step)\n # optimizer\n optimizer_params = {'momentum': cfg.TRAIN.momentum,\n 'wd': cfg.TRAIN.wd,\n 'learning_rate': lr,\n 'lr_scheduler': lr_scheduler,\n 'rescale_grad': 1.0,\n 'clip_gradient': None}\n\n if not isinstance(train_data, PrefetchingIter):\n train_data = PrefetchingIter(train_data)\n\n # train\n mod.fit(train_data, eval_metric=eval_metrics, epoch_end_callback=epoch_end_callback,\n batch_end_callback=batch_end_callback, kvstore=kvstore,\n optimizer='sgd', optimizer_params=optimizer_params,\n arg_params=arg_params, aux_params=aux_params, begin_epoch=begin_epoch, num_epoch=end_epoch)\n\n" }, { "path": "rfcn/operator_cxx/multi_proposal-inl.h", "content": "/*!\n * Copyright (c) 2015 by Contributors\n * Copyright (c) 2017 Microsoft\n * Licensed under The MIT License [see LICENSE for details]\n * \\file multi_proposal-inl.h\n * \\brief MultiProposal Operator\n * \\author Piotr Teterwak, Bing Xu, Jian Guo, Xizhou Zhu\n*/\n#ifndef MXNET_OPERATOR_CONTRIB_PROPOSAL_INL_H_\n#define MXNET_OPERATOR_CONTRIB_PROPOSAL_INL_H_\n\n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \"../operator_common.h\"\n#include \"../mshadow_op.h\"\n\n// extend NumericalParam\nnamespace mxnet {\nnamespace op {\n\n/*!\n* \\brief structure for numerical tuple input\n* \\tparam VType data type of param\n*/\ntemplate\nstruct NumericalParam {\n NumericalParam() {}\n explicit NumericalParam(VType *begin, VType *end) {\n int32_t size = static_cast(end - begin);\n info.resize(size);\n for (int i = 0; i < size; ++i) {\n info[i] = *(begin + i);\n }\n }\n inline size_t ndim() const {\n return info.size();\n }\n std::vector info;\n};\n\ntemplate\ninline std::istream &operator>>(std::istream &is, NumericalParam ¶m) {\n while (true) {\n char ch = is.get();\n if (ch == '(') break;\n if (!isspace(ch)) {\n is.setstate(std::ios::failbit);\n return is;\n }\n }\n VType idx;\n std::vector tmp;\n // deal with empty case\n size_t pos = is.tellg();\n char ch = is.get();\n if (ch == ')') {\n param.info = tmp;\n return is;\n }\n is.seekg(pos);\n // finish deal\n while (is >> idx) {\n tmp.push_back(idx);\n char ch;\n do {\n ch = is.get();\n } while (isspace(ch));\n if (ch == ',') {\n while (true) {\n ch = is.peek();\n if (isspace(ch)) {\n is.get(); continue;\n }\n if (ch == ')') {\n is.get(); break;\n }\n break;\n }\n if (ch == ')') break;\n } else if (ch == ')') {\n break;\n } else {\n is.setstate(std::ios::failbit);\n return is;\n }\n }\n param.info = tmp;\n return is;\n}\n\ntemplate\ninline std::ostream &operator<<(std::ostream &os, const NumericalParam ¶m) {\n os << '(';\n for (index_t i = 0; i < param.info.size(); ++i) {\n if (i != 0) os << ',';\n os << param.info[i];\n }\n // python style tuple\n if (param.info.size() == 1) os << ',';\n os << ')';\n return os;\n}\n\n} // namespace op\n} // namespace mxnet\n\nnamespace mxnet {\nnamespace op {\n\nnamespace proposal {\nenum MultiProposalOpInputs {kClsProb, kBBoxPred, kImInfo};\nenum MultiProposalOpOutputs {kOut, kScore};\nenum MultiProposalForwardResource {kTempResource};\n} // proposal\n\nstruct MultiProposalParam : public dmlc::Parameter {\n int rpn_pre_nms_top_n;\n int rpn_post_nms_top_n;\n float threshold;\n int rpn_min_size;\n NumericalParam scales;\n NumericalParam ratios;\n int feature_stride;\n bool output_score;\n bool iou_loss;\n DMLC_DECLARE_PARAMETER(MultiProposalParam) {\n float tmp[] = {0, 0, 0, 0};\n DMLC_DECLARE_FIELD(rpn_pre_nms_top_n).set_default(6000)\n .describe(\"Number of top scoring boxes to keep after applying NMS to RPN proposals\");\n DMLC_DECLARE_FIELD(rpn_post_nms_top_n).set_default(300)\n .describe(\"Overlap threshold used for non-maximum\"\n \"suppresion(suppress boxes with IoU >= this threshold\");\n DMLC_DECLARE_FIELD(threshold).set_default(0.7)\n .describe(\"NMS value, below which to suppress.\");\n DMLC_DECLARE_FIELD(rpn_min_size).set_default(16)\n .describe(\"Minimum height or width in proposal\");\n tmp[0] = 4.0f; tmp[1] = 8.0f; tmp[2] = 16.0f; tmp[3] = 32.0f;\n DMLC_DECLARE_FIELD(scales).set_default(NumericalParam(tmp, tmp + 4))\n .describe(\"Used to generate anchor windows by enumerating scales\");\n tmp[0] = 0.5f; tmp[1] = 1.0f; tmp[2] = 2.0f;\n DMLC_DECLARE_FIELD(ratios).set_default(NumericalParam(tmp, tmp + 3))\n .describe(\"Used to generate anchor windows by enumerating ratios\");\n DMLC_DECLARE_FIELD(feature_stride).set_default(16)\n .describe(\"The size of the receptive field each unit in the convolution layer of the rpn,\"\n \"for example the product of all stride's prior to this layer.\");\n DMLC_DECLARE_FIELD(output_score).set_default(false)\n .describe(\"Add score to outputs\");\n DMLC_DECLARE_FIELD(iou_loss).set_default(false)\n .describe(\"Usage of IoU Loss\");\n }\n};\n\ntemplate\nOperator *CreateOp(MultiProposalParam param);\n\n#if DMLC_USE_CXX11\nclass MultiProposalProp : public OperatorProperty {\n public:\n void Init(const std::vector >& kwargs) override {\n param_.Init(kwargs);\n }\n\n std::map GetParams() const override {\n return param_.__DICT__();\n }\n\n bool InferShape(std::vector *in_shape,\n std::vector *out_shape,\n std::vector *aux_shape) const override {\n using namespace mshadow;\n CHECK_EQ(in_shape->size(), 3) << \"Input:[cls_prob, bbox_pred, im_info]\";\n const TShape &dshape = in_shape->at(proposal::kClsProb);\n if (dshape.ndim() == 0) return false;\n Shape<4> bbox_pred_shape;\n bbox_pred_shape = Shape4(dshape[0], dshape[1] * 2, dshape[2], dshape[3]);\n SHAPE_ASSIGN_CHECK(*in_shape, proposal::kBBoxPred,\n bbox_pred_shape);\n Shape<2> im_info_shape;\n im_info_shape = Shape2(dshape[0], 3);\n SHAPE_ASSIGN_CHECK(*in_shape, proposal::kImInfo, im_info_shape);\n out_shape->clear();\n // output\n out_shape->push_back(Shape2(dshape[0] * param_.rpn_post_nms_top_n, 5));\n // score\n out_shape->push_back(Shape2(dshape[0] * param_.rpn_post_nms_top_n, 1));\n return true;\n }\n\n OperatorProperty* Copy() const override {\n auto ptr = new MultiProposalProp();\n ptr->param_ = param_;\n return ptr;\n }\n\n std::string TypeString() const override {\n return \"_contrib_MultiProposal\";\n }\n\n std::vector ForwardResource(\n const std::vector &in_shape) const override {\n return {ResourceRequest::kTempSpace};\n }\n\n std::vector DeclareBackwardDependency(\n const std::vector &out_grad,\n const std::vector &in_data,\n const std::vector &out_data) const override {\n return {};\n }\n\n int NumVisibleOutputs() const override {\n if (param_.output_score) {\n return 2;\n } else {\n return 1;\n }\n }\n\n int NumOutputs() const override {\n return 2;\n }\n\n std::vector ListArguments() const override {\n return {\"cls_prob\", \"bbox_pred\", \"im_info\"};\n }\n\n std::vector ListOutputs() const override {\n return {\"output\", \"score\"};\n }\n\n Operator* CreateOperator(Context ctx) const override;\n\n private:\n MultiProposalParam param_;\n}; // class MultiProposalProp\n\n#endif // DMLC_USE_CXX11\n} // namespace op\n} // namespace mxnet\n\n//========================\n// Anchor Generation Utils\n//========================\nnamespace mxnet {\nnamespace op {\nnamespace utils {\n\ninline void _MakeAnchor(float w,\n float h,\n float x_ctr,\n float y_ctr,\n std::vector *out_anchors) {\n out_anchors->push_back(x_ctr - 0.5f * (w - 1.0f));\n out_anchors->push_back(y_ctr - 0.5f * (h - 1.0f));\n out_anchors->push_back(x_ctr + 0.5f * (w - 1.0f));\n out_anchors->push_back(y_ctr + 0.5f * (h - 1.0f));\n out_anchors->push_back(0.0f);\n}\n\ninline void _Transform(float scale,\n float ratio,\n const std::vector& base_anchor,\n std::vector *out_anchors) {\n float w = base_anchor[2] - base_anchor[1] + 1.0f;\n float h = base_anchor[3] - base_anchor[1] + 1.0f;\n float x_ctr = base_anchor[0] + 0.5 * (w - 1.0f);\n float y_ctr = base_anchor[1] + 0.5 * (h - 1.0f);\n float size = w * h;\n float size_ratios = std::floor(size / ratio);\n float new_w = std::floor(std::sqrt(size_ratios) + 0.5f) * scale;\n float new_h = std::floor((new_w / scale * ratio) + 0.5f) * scale;\n\n _MakeAnchor(new_w, new_h, x_ctr,\n y_ctr, out_anchors);\n}\n\n// out_anchors must have shape (n, 5), where n is ratios.size() * scales.size()\ninline void GenerateAnchors(const std::vector& base_anchor,\n const std::vector& ratios,\n const std::vector& scales,\n std::vector *out_anchors) {\n for (size_t j = 0; j < ratios.size(); ++j) {\n for (size_t k = 0; k < scales.size(); ++k) {\n _Transform(scales[k], ratios[j], base_anchor, out_anchors);\n }\n }\n}\n\n} // namespace utils\n} // namespace op\n} // namespace mxnet\n\n#endif // MXNET_OPERATOR_CONTRIB_PROPOSAL_INL_H_\n" }, { "path": "rfcn/operator_cxx/multi_proposal.cc", "content": "/*!\n * Copyright (c) 2017 Microsoft\n * Licensed under The MIT License [see LICENSE for details]\n * \\file multi_proposal.cc\n * \\brief\n * \\author Xizhou Zhu\n*/\n\n#include \"./multi_proposal-inl.h\"\n\n\nnamespace mxnet {\nnamespace op {\n\ntemplate\nclass MultiProposalOp : public Operator{\n public:\n explicit MultiProposalOp(MultiProposalParam param) {\n this->param_ = param;\n }\n\n virtual void Forward(const OpContext &ctx,\n const std::vector &in_data,\n const std::vector &req,\n const std::vector &out_data,\n const std::vector &aux_states) {\n LOG(FATAL) << \"not implemented\";\n }\n\n virtual void Backward(const OpContext &ctx,\n const std::vector &out_grad,\n const std::vector &in_data,\n const std::vector &out_data,\n const std::vector &req,\n const std::vector &in_grad,\n const std::vector &aux_states) {\n LOG(FATAL) << \"not implemented\";\n }\n\n private:\n MultiProposalParam param_;\n}; // class MultiProposalOp\n\ntemplate<>\nOperator *CreateOp(MultiProposalParam param) {\n return new MultiProposalOp(param);\n}\n\nOperator* MultiProposalProp::CreateOperator(Context ctx) const {\n DO_BIND_DISPATCH(CreateOp, param_);\n}\n\nDMLC_REGISTER_PARAMETER(MultiProposalParam);\n\nMXNET_REGISTER_OP_PROPERTY(_contrib_MultiProposal, MultiProposalProp)\n.describe(\"Generate region proposals via RPN\")\n.add_argument(\"cls_score\", \"NDArray-or-Symbol\", \"Score of how likely proposal is object.\")\n.add_argument(\"bbox_pred\", \"NDArray-or-Symbol\", \"BBox Predicted deltas from anchors for proposals\")\n.add_argument(\"im_info\", \"NDArray-or-Symbol\", \"Image size and scale.\")\n.add_arguments(MultiProposalParam::__FIELDS__());\n\n} // namespace op\n} // namespace mxnet\n" }, { "path": "rfcn/operator_cxx/multi_proposal.cu", "content": "/*!\n * Copyright (c) 2015 by Contributors\n * Copyright (c) 2017 Microsoft\n * Licensed under The MIT License [see LICENSE for details]\n * \\file multi_proposal.cu\n * \\brief MultiProposal Operator\n * \\author Shaoqing Ren, Xizhou Zhu, Jian Guo\n*/\n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n\n#include \n#include \n#include \n#include \n#include \n#include \n\n#include \"../operator_common.h\"\n#include \"../mshadow_op.h\"\n#include \"./multi_proposal-inl.h\"\n\n#define DIVUP(m, n) ((m) / (n) + ((m) % (n) > 0))\n\n#define FRCNN_CUDA_CHECK(condition) \\\n /* Code block avoids redefinition of cudaError_t error */ \\\n do { \\\n cudaError_t error = condition; \\\n CHECK_EQ(error, cudaSuccess) << \" \" << cudaGetErrorString(error); \\\n} while (0)\n\nnamespace mshadow {\nnamespace cuda {\nnamespace multi_proposal {\n\n// scores are (b, 2 * anchor, h, w)\n// workspace_proposals are (b, h * w * anchor, 5)\n// w defines \"x\" and h defines \"y\"\n// count should be total anchors numbers, h * w * anchors\ntemplate\n__global__ void ProposalGridKernel(const int count,\n const int num_anchors,\n const int height,\n const int width,\n const int feature_stride,\n const Dtype* scores,\n Dtype* workspace_proposals) {\n for (int index = blockIdx.x * blockDim.x + threadIdx.x;\n index < count;\n index += blockDim.x * gridDim.x) {\n int a = index % num_anchors;\n int w = (index / num_anchors) % width;\n int h = (index / num_anchors / width) % height;\n\tint b = index / num_anchors / width / height;\n\n workspace_proposals[index * 5 + 0] = workspace_proposals[a * 5 + 0] + w * feature_stride;\n workspace_proposals[index * 5 + 1] = workspace_proposals[a * 5 + 1] + h * feature_stride;\n workspace_proposals[index * 5 + 2] = workspace_proposals[a * 5 + 2] + w * feature_stride;\n workspace_proposals[index * 5 + 3] = workspace_proposals[a * 5 + 3] + h * feature_stride;\n\tworkspace_proposals[index * 5 + 4] = \n\t\tscores[((b * (2 * num_anchors) + a + num_anchors) * height + h) * width + w];\n\t//workspace_proposals[index * 5 + 4] = scores[(a * height + h) * width + w];\n }\n}\n\n// boxes are (b, h * w * anchor, 5)\n// deltas are (b, 4 * anchor, h, w)\n// out_pred_boxes are (b, h * w * anchor, 5)\n// count should be total anchors numbers, b * h * w * anchors\n// in-place write: boxes and out_pred_boxes are the same location\ntemplate\n__global__ void BBoxPredKernel(const int count,\n const int num_anchors,\n const int feat_height,\n const int feat_width,\n\t\t\t\t\t\t\t const int feature_stride,\n\t\t\t\t\t\t\t const Dtype* im_infos,\n const Dtype* boxes,\n const Dtype* deltas,\n Dtype* out_pred_boxes) {\n for (int index = blockIdx.x * blockDim.x + threadIdx.x;\n index < count;\n index += blockDim.x * gridDim.x) {\n int a = index % num_anchors;\n int w = (index / num_anchors) % feat_width;\n\tint h = (index / num_anchors / feat_width) % feat_height;\n\tint b = index / num_anchors / feat_width / feat_height;\n\n\tfloat im_height = im_infos[b * 3];\n\tfloat im_width = im_infos[b * 3 + 1];\n\tint real_height = static_cast(im_height / feature_stride);\n\tint real_width = static_cast(im_width / feature_stride);\n\n float width = boxes[index * 5 + 2] - boxes[index * 5 + 0] + 1.0f;\n float height = boxes[index * 5 + 3] - boxes[index * 5 + 1] + 1.0f;\n float ctr_x = boxes[index * 5 + 0] + 0.5f * (width - 1.0f);\n float ctr_y = boxes[index * 5 + 1] + 0.5f * (height - 1.0f);\n\n\tint ba = (b * num_anchors + a);\n float dx = deltas[((ba * 4) * feat_height + h) * feat_width + w];\n float dy = deltas[((ba * 4 + 1) * feat_height + h) * feat_width + w];\n float dw = deltas[((ba * 4 + 2) * feat_height + h) * feat_width + w];\n float dh = deltas[((ba * 4 + 3) * feat_height + h) * feat_width + w];\n\n float pred_ctr_x = dx * width + ctr_x;\n float pred_ctr_y = dy * height + ctr_y;\n float pred_w = exp(dw) * width;\n float pred_h = exp(dh) * height;\n\n float pred_x1 = pred_ctr_x - 0.5f * (pred_w - 1.0f);\n float pred_y1 = pred_ctr_y - 0.5f * (pred_h - 1.0f);\n float pred_x2 = pred_ctr_x + 0.5f * (pred_w - 1.0f);\n float pred_y2 = pred_ctr_y + 0.5f * (pred_h - 1.0f);\n\n pred_x1 = max(min(pred_x1, im_width - 1.0f), 0.0f);\n pred_y1 = max(min(pred_y1, im_height - 1.0f), 0.0f);\n pred_x2 = max(min(pred_x2, im_width - 1.0f), 0.0f);\n pred_y2 = max(min(pred_y2, im_height - 1.0f), 0.0f);\n\n out_pred_boxes[index * 5 + 0] = pred_x1;\n out_pred_boxes[index * 5 + 1] = pred_y1;\n out_pred_boxes[index * 5 + 2] = pred_x2;\n out_pred_boxes[index * 5 + 3] = pred_y2;\n\n if (h >= real_height || w >= real_width) {\n out_pred_boxes[index * 5 + 4] = -1.0f;\n }\n }\n}\n\n// boxes are (b, h * w * anchor, 5)\n// deltas are (b, 4 * anchor, h, w)\n// out_pred_boxes are (b, h * w * anchor, 5)\n// count should be total anchors numbers, b * h * w * anchors\n// in-place write: boxes and out_pred_boxes are the same location\ntemplate\n__global__ void IoUPredKernel(const int count,\n const int num_anchors,\n const int feat_height,\n const int feat_width,\n const int feature_stride,\n\t\t\t\t\t\t\t const Dtype* im_infos,\n const Dtype* boxes,\n const Dtype* deltas,\n Dtype* out_pred_boxes) {\n for (int index = blockIdx.x * blockDim.x + threadIdx.x;\n index < count;\n index += blockDim.x * gridDim.x) {\n int a = index % num_anchors;\n int w = (index / num_anchors) % feat_width;\n int h = (index / num_anchors / feat_width) % feat_height;\n\tint b = index / num_anchors / feat_width / feat_height;\n\n\tfloat im_height = im_infos[b * 3];\n\tfloat im_width = im_infos[b * 3 + 1];\n\tint real_height = static_cast(im_height / feature_stride);\n\tint real_width = static_cast(im_width / feature_stride);\n\n float x1 = boxes[index * 5 + 0];\n float y1 = boxes[index * 5 + 1];\n float x2 = boxes[index * 5 + 2];\n float y2 = boxes[index * 5 + 3];\n\n\tint ba = (b * num_anchors + a);\n float dx1 = deltas[((ba * 4) * feat_height + h) * feat_width + w];\n float dy1 = deltas[((ba * 4 + 1) * feat_height + h) * feat_width + w];\n float dx2 = deltas[((ba * 4 + 2) * feat_height + h) * feat_width + w];\n float dy2 = deltas[((ba * 4 + 3) * feat_height + h) * feat_width + w];\n\n float pred_x1 = max(min(x1 + dx1, im_width - 1.0f), 0.0f);\n float pred_y1 = max(min(y1 + dy1, im_height - 1.0f), 0.0f);\n float pred_x2 = max(min(x2 + dx2, im_width - 1.0f), 0.0f);\n float pred_y2 = max(min(y2 + dy2, im_height - 1.0f), 0.0f);\n\n out_pred_boxes[index * 5 + 0] = pred_x1;\n out_pred_boxes[index * 5 + 1] = pred_y1;\n out_pred_boxes[index * 5 + 2] = pred_x2;\n out_pred_boxes[index * 5 + 3] = pred_y2;\n\n\tif (h >= real_height || w >= real_width) {\n out_pred_boxes[index * 5 + 4] = -1.0f;\n }\n }\n}\n\n// filter box with stride less than rpn_min_size\n// filter: set score to zero\n// dets (b, n, 5)\ntemplate\n__global__ void FilterBoxKernel(const int count,\n\t\t\t\t\t\t\t\tconst int count_anchors,\n const float original_min_size,\n\t\t\t\t\t\t\t\tconst Dtype* im_infos,\n Dtype* dets) {\n for (int index = blockIdx.x * blockDim.x + threadIdx.x;\n index < count;\n index += blockDim.x * gridDim.x) {\n\tint b = index / count_anchors;\n float iw = dets[index * 5 + 2] - dets[index * 5 + 0] + 1.0f;\n float ih = dets[index * 5 + 3] - dets[index * 5 + 1] + 1.0f;\n\tfloat min_size = original_min_size * im_infos[b * 3 + 2];\n if (iw < min_size || ih < min_size) {\n dets[index * 5 + 0] -= min_size / 2;\n dets[index * 5 + 1] -= min_size / 2;\n dets[index * 5 + 2] += min_size / 2;\n dets[index * 5 + 3] += min_size / 2;\n dets[index * 5 + 4] = -1.0f;\n }\n }\n}\n\n// copy score and init order\n// dets (n, 5); score (n, ); order (n, )\n// count should be n (total anchors or proposals)\ntemplate\n__global__ void CopyScoreKernel(const int count,\n const Dtype* dets,\n Dtype* score,\n int* order) {\n for (int index = blockIdx.x * blockDim.x + threadIdx.x;\n index < count;\n index += blockDim.x * gridDim.x) {\n score[index] = dets[index * 5 + 4];\n order[index] = index;\n }\n}\n\n// reorder proposals according to order and keep the top_n proposals\n// prev_dets (n, 5); order (n, ); dets (n, 5)\n// count should be output anchor numbers (top_n)\ntemplate\n__global__ void ReorderProposalsKernel(const int count,\n const Dtype* prev_dets,\n const int* order,\n Dtype* dets) {\n for (int index = blockIdx.x * blockDim.x + threadIdx.x;\n index < count;\n index += blockDim.x * gridDim.x) {\n const int order_i = order[index];\n for (int j = 0; j < 5; j ++) {\n dets[index * 5 + j] = prev_dets[order_i * 5 + j];\n }\n }\n}\n\n__device__ inline float devIoU(float const * const a, float const * const b) {\n float left = max(a[0], b[0]), right = min(a[2], b[2]);\n float top = max(a[1], b[1]), bottom = min(a[3], b[3]);\n float width = max(right - left + 1, 0.f), height = max(bottom - top + 1, 0.f);\n float interS = width * height;\n float Sa = (a[2] - a[0] + 1) * (a[3] - a[1] + 1);\n float Sb = (b[2] - b[0] + 1) * (b[3] - b[1] + 1);\n return interS / (Sa + Sb - interS);\n}\n\n__global__ void nms_kernel(const int n_boxes, const float nms_overlap_thresh,\n const float *dev_boxes, uint64_t *dev_mask) {\n const int threadsPerBlock = sizeof(uint64_t) * 8;\n const int row_start = blockIdx.y;\n const int col_start = blockIdx.x;\n\n // if (row_start > col_start) return;\n\n const int row_size =\n min(n_boxes - row_start * threadsPerBlock, threadsPerBlock);\n const int col_size =\n min(n_boxes - col_start * threadsPerBlock, threadsPerBlock);\n\n __shared__ float block_boxes[threadsPerBlock * 5];\n if (threadIdx.x < col_size) {\n block_boxes[threadIdx.x * 5 + 0] =\n dev_boxes[(threadsPerBlock * col_start + threadIdx.x) * 5 + 0];\n block_boxes[threadIdx.x * 5 + 1] =\n dev_boxes[(threadsPerBlock * col_start + threadIdx.x) * 5 + 1];\n block_boxes[threadIdx.x * 5 + 2] =\n dev_boxes[(threadsPerBlock * col_start + threadIdx.x) * 5 + 2];\n block_boxes[threadIdx.x * 5 + 3] =\n dev_boxes[(threadsPerBlock * col_start + threadIdx.x) * 5 + 3];\n block_boxes[threadIdx.x * 5 + 4] =\n dev_boxes[(threadsPerBlock * col_start + threadIdx.x) * 5 + 4];\n }\n __syncthreads();\n\n if (threadIdx.x < row_size) {\n const int cur_box_idx = threadsPerBlock * row_start + threadIdx.x;\n const float *cur_box = dev_boxes + cur_box_idx * 5;\n int i = 0;\n uint64_t t = 0;\n int start = 0;\n if (row_start == col_start) {\n start = threadIdx.x + 1;\n }\n for (i = start; i < col_size; i++) {\n if (devIoU(cur_box, block_boxes + i * 5) > nms_overlap_thresh) {\n t |= 1ULL << i;\n }\n }\n const int col_blocks = DIVUP(n_boxes, threadsPerBlock);\n dev_mask[cur_box_idx * col_blocks + col_start] = t;\n }\n}\n\nvoid _nms(const mshadow::Tensor& boxes,\n const float nms_overlap_thresh,\n int *keep,\n int *num_out) {\n const int threadsPerBlock = sizeof(uint64_t) * 8;\n const int boxes_num = boxes.size(0);\n const int boxes_dim = boxes.size(1);\n\n float* boxes_dev = boxes.dptr_;\n uint64_t* mask_dev = NULL;\n\n const int col_blocks = DIVUP(boxes_num, threadsPerBlock);\n FRCNN_CUDA_CHECK(cudaMalloc(&mask_dev,\n boxes_num * col_blocks * sizeof(uint64_t)));\n\n dim3 blocks(DIVUP(boxes_num, threadsPerBlock),\n DIVUP(boxes_num, threadsPerBlock));\n dim3 threads(threadsPerBlock);\n nms_kernel<<>>(boxes_num,\n nms_overlap_thresh,\n boxes_dev,\n mask_dev);\n FRCNN_CUDA_CHECK(cudaPeekAtLastError());\n std::vector mask_host(boxes_num * col_blocks);\n FRCNN_CUDA_CHECK(cudaMemcpy(&mask_host[0],\n mask_dev,\n sizeof(uint64_t) * boxes_num * col_blocks,\n cudaMemcpyDeviceToHost));\n\n std::vector remv(col_blocks);\n memset(&remv[0], 0, sizeof(uint64_t) * col_blocks);\n\n int num_to_keep = 0;\n for (int i = 0; i < boxes_num; i++) {\n int nblock = i / threadsPerBlock;\n int inblock = i % threadsPerBlock;\n\n if (!(remv[nblock] & (1ULL << inblock))) {\n keep[num_to_keep++] = i;\n uint64_t *p = &mask_host[0] + i * col_blocks;\n for (int j = nblock; j < col_blocks; j++) {\n remv[j] |= p[j];\n }\n }\n }\n *num_out = num_to_keep;\n\n FRCNN_CUDA_CHECK(cudaFree(mask_dev));\n}\n\n// copy proposals to output\n// dets (top_n, 5); keep (top_n, ); out (top_n, )\n// count should be top_n (total anchors or proposals)\ntemplate\n__global__ void PrepareOutput(const int count,\n const Dtype* dets,\n const int* keep,\n const int out_size,\n\t\t\t\t\t\t\t const int image_index,\n Dtype* out,\n Dtype* score) {\n for (int index = blockIdx.x * blockDim.x + threadIdx.x;\n index < count;\n index += blockDim.x * gridDim.x) {\n out[index * 5] = image_index;\n if (index < out_size) {\n int keep_i = keep[index];\n for (int j = 0; j < 4; ++j) {\n out[index * 5 + j + 1] = dets[keep_i * 5 + j];\n }\n score[index] = dets[keep_i * 5 + 4];\n } else {\n int keep_i = keep[index % out_size];\n for (int j = 0; j < 4; ++j) {\n out[index * 5 + j + 1] = dets[keep_i * 5 + j];\n }\n score[index] = dets[keep_i * 5 + 4];\n }\n }\n}\n} // namespace multi_proposal\n} // namespace cuda\n} // namespace mshadow\n\nnamespace mxnet {\nnamespace op {\n\ntemplate\nclass MultiProposalGPUOp : public Operator{\n public:\n explicit MultiProposalGPUOp(MultiProposalParam param) {\n this->param_ = param;\n }\n\n virtual void Forward(const OpContext &ctx,\n const std::vector &in_data,\n const std::vector &req,\n const std::vector &out_data,\n const std::vector &aux_states) {\n using namespace mshadow;\n using namespace mshadow::expr;\n\tusing namespace mshadow::cuda;\n using namespace mshadow::cuda::multi_proposal;\n CHECK_EQ(in_data.size(), 3);\n CHECK_EQ(out_data.size(), 2);\n CHECK_GT(req.size(), 1);\n CHECK_EQ(req[proposal::kOut], kWriteTo);\n /*CHECK_EQ(in_data[proposal::kClsProb].shape_[0], 1)\n << \"Sorry, multiple images each device is not implemented.\";*/\n\n Stream *s = ctx.get_stream();\n\n Tensor scores = in_data[proposal::kClsProb].get(s);\n Tensor bbox_deltas = in_data[proposal::kBBoxPred].get(s);\n Tensor im_info = in_data[proposal::kImInfo].get(s);\n\n Tensor out = out_data[proposal::kOut].get(s);\n Tensor out_score = out_data[proposal::kScore].get(s);\n \n\tint num_images = scores.size(0);\n\tint num_anchors = scores.size(1) / 2;\n int height = scores.size(2);\n int width = scores.size(3);\n int count_anchors = num_anchors * height * width; // count of total anchors\n\tint count = num_images * count_anchors;\n // set to -1 for max\n int rpn_pre_nms_top_n = (param_.rpn_pre_nms_top_n > 0) ? param_.rpn_pre_nms_top_n : count_anchors;\n rpn_pre_nms_top_n = std::min(rpn_pre_nms_top_n, count_anchors);\n int rpn_post_nms_top_n = std::min(param_.rpn_post_nms_top_n, rpn_pre_nms_top_n);\n\n // Generate first anchors based on base anchor\n std::vector base_anchor(4);\n base_anchor[0] = 0.0;\n base_anchor[1] = 0.0;\n base_anchor[2] = param_.feature_stride - 1.0;\n base_anchor[3] = param_.feature_stride - 1.0;\n CHECK_EQ(num_anchors, param_.ratios.info.size() * param_.scales.info.size());\n std::vector anchors;\n utils::GenerateAnchors(base_anchor,\n param_.ratios.info,\n param_.scales.info,\n &anchors);\n\n // Copy generated anchors to GPU\n float* workspace_proposals_ptr = NULL;\n FRCNN_CUDA_CHECK(cudaMalloc(&workspace_proposals_ptr, sizeof(float) * num_images * count_anchors * 5));\n Tensor workspace_proposals(workspace_proposals_ptr, Shape3(num_images, count_anchors, 5));\n FRCNN_CUDA_CHECK(cudaMemcpy(workspace_proposals.dptr_,\n &anchors[0], sizeof(float) * anchors.size(),\n cudaMemcpyHostToDevice));\n\n // Copy proposals to a mesh grid\n dim3 dimGrid((count + kMaxThreadsPerBlock - 1) / kMaxThreadsPerBlock);\n dim3 dimBlock(kMaxThreadsPerBlock);\n CheckLaunchParam(dimGrid, dimBlock, \"ProposalGrid\");\n ProposalGridKernel<<>>(\n\t count, num_anchors, height, width, param_.feature_stride,\n scores.dptr_, workspace_proposals.dptr_);\n FRCNN_CUDA_CHECK(cudaPeekAtLastError());\n\n // Transform anchors and bbox_deltas into bboxes\n CheckLaunchParam(dimGrid, dimBlock, \"BBoxPred\");\n if (param_.iou_loss) {\n IoUPredKernel<<>>(\n\t\tcount, num_anchors, height, width, param_.feature_stride, im_info.dptr_,\n workspace_proposals.dptr_, bbox_deltas.dptr_, workspace_proposals.dptr_);\n } else {\n BBoxPredKernel<<>>(\n\t\tcount, num_anchors, height, width, param_.feature_stride, im_info.dptr_,\n workspace_proposals.dptr_, bbox_deltas.dptr_, workspace_proposals.dptr_);\n }\n FRCNN_CUDA_CHECK(cudaPeekAtLastError());\n\n // filter boxes with less than rpn_min_size\n CheckLaunchParam(dimGrid, dimBlock, \"FilterBox\");\n FilterBoxKernel<<>>(\n\t count, count_anchors, param_.rpn_min_size, im_info.dptr_, workspace_proposals.dptr_);\n FRCNN_CUDA_CHECK(cudaPeekAtLastError());\n\n\n\n\tdimGrid = dim3((count_anchors + kMaxThreadsPerBlock - 1) / kMaxThreadsPerBlock);\n\tdimBlock = dim3(kMaxThreadsPerBlock);\n\t// Copy score to a continuous memory\n\tfloat* score_ptr = NULL;\n\tFRCNN_CUDA_CHECK(cudaMalloc(&score_ptr, sizeof(float) * count_anchors));\n\tTensor score(score_ptr, Shape1(count_anchors));\n\tint* order_ptr = NULL;\n\tFRCNN_CUDA_CHECK(cudaMalloc(&order_ptr, sizeof(int) * count_anchors));\n\tTensor order(order_ptr, Shape1(count_anchors));\n\n\tfloat* workspace_ordered_proposals_ptr = NULL;\n\tFRCNN_CUDA_CHECK(cudaMalloc(&workspace_ordered_proposals_ptr,\n\t\tsizeof(float) * rpn_pre_nms_top_n * 5));\n\tTensor workspace_ordered_proposals(workspace_ordered_proposals_ptr,\n\t\tShape2(rpn_pre_nms_top_n, 5));\n\n\tint* keep;\n\tFRCNN_CUDA_CHECK(cudaMalloc(&keep, sizeof(int) * rpn_pre_nms_top_n));\n\n\tfor (int b = 0; b < num_images; b++) {\n\n\t\tCheckLaunchParam(dimGrid, dimBlock, \"CopyScore\");\n\t\tCopyScoreKernel << > >(\n\t\t\tcount_anchors, workspace_proposals.dptr_ + b * count_anchors * 5, score.dptr_, order.dptr_);\n\t\tFRCNN_CUDA_CHECK(cudaPeekAtLastError());\n\n\t\t// argsort score, save order\n\t\tthrust::stable_sort_by_key(thrust::device,\n\t\t\tscore.dptr_,\n\t\t\tscore.dptr_ + score.size(0),\n\t\t\torder.dptr_,\n\t\t\tthrust::greater());\n\t\tFRCNN_CUDA_CHECK(cudaPeekAtLastError());\n\n\t\t// Reorder proposals according to order\n\n\t\tdimGrid.x = (rpn_pre_nms_top_n + kMaxThreadsPerBlock - 1) / kMaxThreadsPerBlock;\n\t\tCheckLaunchParam(dimGrid, dimBlock, \"ReorderProposals\");\n\t\tReorderProposalsKernel << > >(\n\t\t\trpn_pre_nms_top_n, workspace_proposals.dptr_ + b * count_anchors * 5, order.dptr_, workspace_ordered_proposals.dptr_);\n\t\tFRCNN_CUDA_CHECK(cudaPeekAtLastError());\n\n\t\t// perform nms\n\t\tstd::vector _keep(workspace_ordered_proposals.size(0));\n\t\tint out_size = 0;\n\t\t_nms(workspace_ordered_proposals,\n\t\t\tparam_.threshold,\n\t\t\t&_keep[0],\n\t\t\t&out_size);\n\n\t\t// copy nms result to gpu\n\t\tFRCNN_CUDA_CHECK(cudaMemcpy(keep, &_keep[0], sizeof(int) * _keep.size(),\n\t\t\tcudaMemcpyHostToDevice));\n\n\t\t// copy results after nms\n\t\tdimGrid.x = (rpn_post_nms_top_n + kMaxThreadsPerBlock - 1) / kMaxThreadsPerBlock;\n\t\tCheckLaunchParam(dimGrid, dimBlock, \"PrepareOutput\");\n\t\tPrepareOutput << > >(\n\t\t\trpn_post_nms_top_n, workspace_ordered_proposals.dptr_, keep, out_size, b,\n\t\t\tout.dptr_ + b * rpn_post_nms_top_n * 5, out_score.dptr_ + b * rpn_post_nms_top_n);\n\t\tFRCNN_CUDA_CHECK(cudaPeekAtLastError());\n\t}\n\t// free temporary memory\n\tFRCNN_CUDA_CHECK(cudaFree(keep));\n\tFRCNN_CUDA_CHECK(cudaFree(workspace_ordered_proposals_ptr));\n\tFRCNN_CUDA_CHECK(cudaFree(workspace_proposals_ptr));\n\tFRCNN_CUDA_CHECK(cudaFree(score_ptr));\n\tFRCNN_CUDA_CHECK(cudaFree(order_ptr));\n }\n\n virtual void Backward(const OpContext &ctx,\n const std::vector &out_grad,\n const std::vector &in_data,\n const std::vector &out_data,\n const std::vector &req,\n const std::vector &in_grad,\n const std::vector &aux_states) {\n using namespace mshadow;\n using namespace mshadow::expr;\n CHECK_EQ(in_grad.size(), 3);\n\n Stream *s = ctx.get_stream();\n Tensor gscores = in_grad[proposal::kClsProb].get(s);\n Tensor gbbox = in_grad[proposal::kBBoxPred].get(s);\n Tensor ginfo = in_grad[proposal::kImInfo].get(s);\n\n // can not assume the grad would be zero\n Assign(gscores, req[proposal::kClsProb], 0);\n Assign(gbbox, req[proposal::kBBoxPred], 0);\n Assign(ginfo, req[proposal::kImInfo], 0);\n }\n\n private:\n MultiProposalParam param_;\n}; // class MultiProposalGPUOp\n\ntemplate<>\nOperator* CreateOp(MultiProposalParam param) {\n return new MultiProposalGPUOp(param);\n}\n} // namespace op\n} // namespace mxnet\n" }, { "path": "rfcn/operator_cxx/psroi_pooling-inl.h", "content": "/*!\n * Copyright (c) 2017 by Contributors\n * Copyright (c) 2017 Microsoft\n * Licensed under The MIT License [see LICENSE for details]\n * \\file psroi_pooling-inl.h\n * \\brief psroi pooling operator and symbol\n * \\author Yi Li, Tairui Chen, Guodong Zhang, Jifeng Dai\n*/\n#ifndef MXNET_OPERATOR_PSROI_POOLING_INL_H_\n#define MXNET_OPERATOR_PSROI_POOLING_INL_H_\n\n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \"../mshadow_op.h\"\n#include \"../operator_common.h\"\n\n\nnamespace mxnet {\nnamespace op {\n\n// Declare enumeration of input order to make code more intuitive.\n// These enums are only visible within this header\nnamespace psroipool {\nenum PSROIPoolingOpInputs {kData, kBox};\nenum PSROIPoolingOpOutputs {kOut, kMappingChannel};\n} // psroipool\n\nstruct PSROIPoolingParam : public dmlc::Parameter {\n // TShape pooled_size;\n float spatial_scale;\n int output_dim;\n int pooled_size;\n int group_size;\n DMLC_DECLARE_PARAMETER(PSROIPoolingParam) {\n DMLC_DECLARE_FIELD(spatial_scale).set_range(0.0, 1.0)\n .describe(\"Ratio of input feature map height (or w) to raw image height (or w). \"\n \"Equals the reciprocal of total stride in convolutional layers\");\n DMLC_DECLARE_FIELD(output_dim).describe(\"fix output dim\");\n DMLC_DECLARE_FIELD(pooled_size).describe(\"fix pooled size\");\n DMLC_DECLARE_FIELD(group_size).set_default(0).describe(\"fix group size\");\n }\n};\n\ntemplate\nclass PSROIPoolingOp : public Operator {\n public:\n explicit PSROIPoolingOp(PSROIPoolingParam p) {\n this->param_ = p;\n }\n\n virtual void Forward(const OpContext &ctx,\n const std::vector &in_data,\n const std::vector &req,\n const std::vector &out_data,\n const std::vector &aux_args) {\n using namespace mshadow;\n size_t expected = 2;\n CHECK_EQ(in_data.size(), expected);\n CHECK_EQ(out_data.size(), expected);\n CHECK_EQ(out_data[psroipool::kOut].shape_[0], in_data[psroipool::kBox].shape_[0]);\n CHECK_EQ(out_data[psroipool::kMappingChannel].shape_[0], in_data[psroipool::kBox].shape_[0]);\n Stream *s = ctx.get_stream();\n\n Tensor data = in_data[psroipool::kData].get(s);\n Tensor bbox = in_data[psroipool::kBox].get(s);\n Tensor out = out_data[psroipool::kOut].get(s);\n Tensor mapping_channel = out_data[psroipool::kMappingChannel].get(s);\n CHECK_EQ(data.CheckContiguous(), true);\n CHECK_EQ(bbox.CheckContiguous(), true);\n CHECK_EQ(out.CheckContiguous(), true);\n CHECK_EQ(mapping_channel.CheckContiguous(), true);\n out = -FLT_MAX;\n mapping_channel = -1.0f;\n PSROIPoolForward(out, data, bbox, mapping_channel, param_.spatial_scale, param_.output_dim, param_.group_size);\n }\n\n virtual void Backward(const OpContext &ctx,\n const std::vector &out_grad,\n const std::vector &in_data,\n const std::vector &out_data,\n const std::vector &req,\n const std::vector &in_grad,\n const std::vector &aux_args) {\n using namespace mshadow;\n size_t expected = 2;\n CHECK_EQ(in_data.size(), expected);\n CHECK_EQ(out_data.size(), expected);\n CHECK_EQ(out_grad[psroipool::kOut].shape_[0], in_data[psroipool::kBox].shape_[0]);\n CHECK_EQ(out_data[psroipool::kMappingChannel].shape_[0], in_data[psroipool::kBox].shape_[0]);\n CHECK_NE(req[psroipool::kData], kWriteInplace) <<\n \"ROIPooling: Backward doesn't support kWriteInplace.\";\n CHECK_NE(req[psroipool::kBox], kWriteInplace) <<\n \"ROIPooling: Backward doesn't support kWriteInplace.\";\n Stream *s = ctx.get_stream();\n\n Tensor grad_out = out_grad[psroipool::kOut].get(s);\n Tensor bbox = in_data[psroipool::kBox].get(s);\n Tensor mapping_channel = out_data[psroipool::kMappingChannel].get(s);\n Tensor grad_in = in_grad[psroipool::kData].get(s);\n Tensor grad_roi = in_grad[psroipool::kBox].get(s);\n\n CHECK_EQ(grad_out.CheckContiguous(), true);\n CHECK_EQ(bbox.CheckContiguous(), true);\n CHECK_EQ(mapping_channel.CheckContiguous(), true);\n CHECK_EQ(grad_in.CheckContiguous(), true);\n\n if (kAddTo == req[psroipool::kData] || kWriteTo == req[psroipool::kData]) {\n if (kWriteTo == req[psroipool::kData]) {\n grad_in = 0.0f;\n }\n PSROIPoolBackwardAcc(grad_in, grad_out, bbox, mapping_channel, param_.spatial_scale, param_.output_dim);\n }\n if (kWriteTo == req[psroipool::kBox]) {\n grad_roi = 0.0f;\n }\n\n }\n\n private:\n PSROIPoolingParam param_;\n}; // class PSROIPoolingOp\n\n// Decalre Factory function, used for dispatch specialization\ntemplate\nOperator* CreateOp(PSROIPoolingParam param, int dtype);\n\n#if DMLC_USE_CXX11\nclass PSROIPoolingProp : public OperatorProperty {\n public:\n std::vector ListArguments() const override {\n return {\"data\", \"rois\"};\n }\n\n std::vector ListOutputs() const override {\n return {\"output\", \"maxidx\"};\n }\n\n int NumOutputs() const override {\n return 2;\n }\n\n int NumVisibleOutputs() const override {\n return 1;\n }\n\n void Init(const std::vector >& kwargs) override {\n param_.Init(kwargs);\n if (param_.group_size == 0) {\n param_.group_size = param_.pooled_size;\n }\n }\n\n std::map GetParams() const override {\n return param_.__DICT__();\n }\n\n bool InferShape(std::vector *in_shape,\n std::vector *out_shape,\n std::vector *aux_shape) const override {\n using namespace mshadow;\n CHECK_EQ(in_shape->size(), 2) << \"Input:[data, rois]\";\n\n // data: [batch_size, c, h, w]\n TShape dshape = in_shape->at(psroipool::kData);\n CHECK_EQ(dshape.ndim(), 4) << \"data should be a 4D tensor\";\n\n // bbox: [num_rois, 5]\n TShape bshape = in_shape->at(psroipool::kBox);\n CHECK_EQ(bshape.ndim(), 2) << \"bbox should be a 2D tensor of shape [batch, 5]\";\n CHECK_EQ(bshape[1], 5) << \"bbox should be a 2D tensor of shape [batch, 5]\";\n\n // out: [num_rois, c, pooled_h, pooled_w]\n // mapping_channel: [num_rois, c, pooled_h, pooled_w]\n out_shape->clear();\n out_shape->push_back(\n Shape4(bshape[0], param_.output_dim, param_.pooled_size, param_.pooled_size));\n out_shape->push_back(\n Shape4(bshape[0], param_.output_dim, param_.pooled_size, param_.pooled_size));\n return true;\n }\n\n bool InferType(std::vector *in_type,\n std::vector *out_type,\n std::vector *aux_type) const override {\n CHECK_EQ(in_type->size(), 2);\n int dtype = (*in_type)[0];\n CHECK_EQ(dtype, (*in_type)[1]);\n CHECK_NE(dtype, -1) << \"Input must have specified type\";\n\n out_type->clear();\n out_type->push_back(dtype);\n out_type->push_back(dtype);\n return true;\n }\n\n OperatorProperty* Copy() const override {\n PSROIPoolingProp* psroi_pooling_sym = new PSROIPoolingProp();\n psroi_pooling_sym->param_ = this->param_;\n return psroi_pooling_sym;\n }\n\n std::string TypeString() const override {\n return \"_contrib_PSROIPooling\";\n }\n\n // decalre dependency and inplace optimization options\n std::vector DeclareBackwardDependency(\n const std::vector &out_grad,\n const std::vector &in_data,\n const std::vector &out_data) const override {\n return {out_grad[psroipool::kOut], in_data[psroipool::kBox], out_data[psroipool::kMappingChannel]};\n }\n\n\n Operator* CreateOperator(Context ctx) const override {\n LOG(FATAL) << \"Not Implemented.\";\n return NULL;\n }\n\n Operator* CreateOperatorEx(Context ctx, std::vector *in_shape,\n std::vector *in_type) const override;\n\n\n private:\n PSROIPoolingParam param_;\n}; // class PSROIPoolingProp\n#endif\n} // namespace op\n} // namespace mxnet\n#endif // MXNET_OPERATOR_PSROI_POOLING_INL_H_" }, { "path": "rfcn/operator_cxx/psroi_pooling.cc", "content": "/*!\n * Copyright (c) 2017 by Contributors\n * Copyright (c) 2017 Microsoft\n * Licensed under The MIT License [see LICENSE for details]\n * \\file psroi_pooling.cc\n * \\brief psroi pooling operator\n * \\author Yi Li, Tairui Chen, Guodong Zhang, Jifeng Dai\n*/\n#include \"./psroi_pooling-inl.h\"\n#include \n#include \n#include \n#include \n#include \n\nusing std::max;\nusing std::min;\nusing std::floor;\nusing std::ceil;\n\nnamespace mshadow {\ntemplate\ninline void PSROIPoolForward(const Tensor &out,\n const Tensor &data,\n const Tensor &bbox,\n const Tensor &mapping_channel,\n const float spatial_scale_,\n const int output_dim_, \n const int group_size_) {\n // NOT_IMPLEMENTED;\n return;\n}\n\ntemplate\ninline void PSROIPoolBackwardAcc(const Tensor &in_grad,\n const Tensor &out_grad,\n const Tensor &bbox,\n const Tensor &mapping_channel,\n const float spatial_scale_,\n const int output_dim_) {\n // NOT_IMPLEMENTED;\n return;\n}\n} // namespace mshadow\n\nnamespace mxnet {\nnamespace op {\n\ntemplate<>\nOperator *CreateOp(PSROIPoolingParam param, int dtype) {\n Operator* op = NULL;\n MSHADOW_REAL_TYPE_SWITCH(dtype, DType, {\n op = new PSROIPoolingOp(param);\n });\n return op;\n}\n\nOperator *PSROIPoolingProp::CreateOperatorEx(Context ctx, std::vector *in_shape,\n std::vector *in_type) const {\n std::vector out_shape, aux_shape;\n std::vector out_type, aux_type;\n CHECK(InferType(in_type, &out_type, &aux_type));\n CHECK(InferShape(in_shape, &out_shape, &aux_shape));\n DO_BIND_DISPATCH(CreateOp, param_, in_type->at(0));\n}\n\nDMLC_REGISTER_PARAMETER(PSROIPoolingParam);\n\nMXNET_REGISTER_OP_PROPERTY(_contrib_PSROIPooling, PSROIPoolingProp)\n.describe(\"Performs region-of-interest pooling on inputs. Resize bounding box coordinates by \"\n\"spatial_scale and crop input feature maps accordingly. The cropped feature maps are pooled \"\n\"by max pooling to a fixed size output indicated by pooled_size. batch_size will change to \"\n\"the number of region bounding boxes after PSROIPooling\")\n.add_argument(\"data\", \"Symbol\", \"Input data to the pooling operator, a 4D Feature maps\")\n.add_argument(\"rois\", \"Symbol\", \"Bounding box coordinates, a 2D array of \"\n\"[[batch_index, x1, y1, x2, y2]]. (x1, y1) and (x2, y2) are top left and down right corners \"\n\"of designated region of interest. batch_index indicates the index of corresponding image \"\n\"in the input data\")\n.add_arguments(PSROIPoolingParam::__FIELDS__());\n} // namespace op\n} // namespace mxnet" }, { "path": "rfcn/operator_cxx/psroi_pooling.cu", "content": "/*!\n * Copyright (c) 2017 by Contributors\n * Copyright (c) 2017 Microsoft\n * Licensed under The MIT License [see LICENSE for details]\n * \\file psroi_pooling.cu\n * \\brief psroi pooling operator\n * \\author Yi Li, Tairui Chen, Guodong Zhang, Jifeng Dai\n*/\n#include \"./psroi_pooling-inl.h\"\n#include \n#include \n#include \n#include \n#include \"../../common/cuda_utils.h\"\n#include \"../mxnet_op.h\"\n\n#define PSROIPOOLING_CUDA_CHECK(condition) \\\n /* Code block avoids redefinition of cudaError_t error */ \\\n do { \\\n cudaError_t error = condition; \\\n CHECK_EQ(error, cudaSuccess) << \" \" << cudaGetErrorString(error); \\\n } while (0)\n#define CUDA_KERNEL_LOOP(i, n) \\\nfor (int i = blockIdx.x * blockDim.x + threadIdx.x; \\\n i < (n); \\\n i += blockDim.x * gridDim.x)\n\nnamespace mshadow {\nnamespace cuda {\n\ntemplate \n__global__ void PSROIPoolForwardKernel(\n const int count,\n const DType* bottom_data,\n const DType spatial_scale,\n const int channels,\n const int height, const int width,\n const int pooled_height, const int pooled_width,\n const DType* bottom_rois,\n const int output_dim,\n const int group_size,\n DType* top_data,\n DType* mapping_channel) {\n CUDA_KERNEL_LOOP(index, count) {\n // The output is in order (n, ctop, ph, pw)\n int pw = index % pooled_width;\n int ph = (index / pooled_width) % pooled_height;\n int ctop = (index / pooled_width / pooled_height) % output_dim;\n int n = index / pooled_width / pooled_height / output_dim;\n\n // [start, end) interval for spatial sampling\n bottom_rois += n * 5;\n int roi_batch_ind = bottom_rois[0];\n DType roi_start_w = static_cast(round(bottom_rois[1])) * spatial_scale;\n DType roi_start_h = static_cast(round(bottom_rois[2])) * spatial_scale;\n DType roi_end_w = static_cast(round(bottom_rois[3]) + 1.) * spatial_scale;\n DType roi_end_h = static_cast(round(bottom_rois[4]) + 1.) * spatial_scale;\n\n // Force too small ROIs to be 1x1\n DType roi_width = max(roi_end_w - roi_start_w, 0.1); //avoid 0\n DType roi_height = max(roi_end_h - roi_start_h, 0.1);\n\n // Compute w and h at bottom\n DType bin_size_h = roi_height / static_cast(pooled_height);\n DType bin_size_w = roi_width / static_cast(pooled_width);\n\n int hstart = floor(static_cast(ph) * bin_size_h\n + roi_start_h);\n int wstart = floor(static_cast(pw)* bin_size_w\n + roi_start_w);\n int hend = ceil(static_cast(ph + 1) * bin_size_h\n + roi_start_h);\n int wend = ceil(static_cast(pw + 1) * bin_size_w\n + roi_start_w);\n // Add roi offsets and clip to input boundaries\n hstart = min(max(hstart, 0), height);\n hend = min(max(hend, 0), height);\n wstart = min(max(wstart, 0),width);\n wend = min(max(wend, 0), width);\n bool is_empty = (hend <= hstart) || (wend <= wstart);\n\n int gw = floor(static_cast(pw)* group_size / pooled_width);\n int gh = floor(static_cast(ph)* group_size / pooled_height);\n gw = min(max(gw, 0), group_size - 1);\n gh = min(max(gh, 0), group_size - 1);\n int c = (ctop*group_size + gh)*group_size + gw;\n\n bottom_data += (roi_batch_ind * channels + c) * height * width;\n DType out_sum = 0;\n for (int h = hstart; h < hend; ++h){\n for (int w = wstart; w < wend; ++w){\n int bottom_index = h*width + w;\n out_sum += bottom_data[bottom_index];\n }\n }\n\n DType bin_area = (hend - hstart)*(wend - wstart);\n top_data[index] = is_empty? (DType)0. : out_sum/bin_area;\n mapping_channel[index] = c;\n }\n}\n\ntemplate\ninline void PSROIPoolForward(const Tensor &out,\n const Tensor &data,\n const Tensor &bbox,\n const Tensor &mapping_channel,\n const float spatial_scale,\n const int output_dim_,\n const int group_size_) {\n // LOG(INFO) << \"PSROIPoolForward\";\n const DType *bottom_data = data.dptr_;\n const DType *bottom_rois = bbox.dptr_;\n DType *top_data = out.dptr_;\n DType *mapping_channel_ptr = mapping_channel.dptr_;\n const int count = out.shape_.Size();\n const int channels = data.size(1);\n const int height = data.size(2);\n const int width = data.size(3);\n const int pooled_height = out.size(2);\n const int pooled_width = out.size(3);\n cudaStream_t stream = Stream::GetStream(out.stream_);\n PSROIPoolForwardKernel << > >(\n count, bottom_data, spatial_scale, channels, height, width,\n pooled_height, pooled_width, bottom_rois, output_dim_, group_size_, top_data, mapping_channel_ptr);\n PSROIPOOLING_CUDA_CHECK(cudaPeekAtLastError());\n}\n\n\ntemplate \n__global__ void PSROIPoolBackwardAccKernel(\n const int count,\n const DType* top_diff,\n const DType* mapping_channel,\n const int num_rois,\n const DType spatial_scale,\n const int channels,\n const int height, const int width,\n const int pooled_height, const int pooled_width,\n const int output_dim,\n DType* bottom_diff,\n const DType* bottom_rois) {\n CUDA_KERNEL_LOOP(index, count) {\n // The output is in order (n, ctop, ph, pw)\n int pw = index % pooled_width;\n int ph = (index / pooled_width) % pooled_height;\n int n = index / pooled_width / pooled_height / output_dim;\n\n // [start, end) interval for spatial sampling\n bottom_rois += n * 5;\n int roi_batch_ind = bottom_rois[0];\n DType roi_start_w = static_cast(round(bottom_rois[1])) * spatial_scale;\n DType roi_start_h = static_cast(round(bottom_rois[2])) * spatial_scale;\n DType roi_end_w = static_cast(round(bottom_rois[3]) + 1.) * spatial_scale;\n DType roi_end_h = static_cast(round(bottom_rois[4]) + 1.) * spatial_scale;\n\n // Force too small ROIs to be 1x1\n DType roi_width = max(roi_end_w - roi_start_w, 0.1); //avoid 0\n DType roi_height = max(roi_end_h - roi_start_h, 0.1);\n\n // Compute w and h at bottom\n DType bin_size_h = roi_height / static_cast(pooled_height);\n DType bin_size_w = roi_width / static_cast(pooled_width);\n\n int hstart = floor(static_cast(ph)* bin_size_h\n + roi_start_h);\n int wstart = floor(static_cast(pw)* bin_size_w\n + roi_start_w);\n int hend = ceil(static_cast(ph + 1) * bin_size_h\n + roi_start_h);\n int wend = ceil(static_cast(pw + 1) * bin_size_w\n + roi_start_w);\n // Add roi offsets and clip to input boundaries\n hstart = min(max(hstart, 0), height);\n hend = min(max(hend, 0), height);\n wstart = min(max(wstart, 0), width);\n wend = min(max(wend, 0), width);\n bool is_empty = (hend <= hstart) || (wend <= wstart);\n\n // Compute c at bottom\n int c = mapping_channel[index];\n DType* offset_bottom_diff = bottom_diff + (roi_batch_ind * channels + c) * height * width;\n DType bin_area = (hend - hstart)*(wend - wstart);\n DType diff_val = is_empty ? (DType)0. : top_diff[index] / bin_area;\n for (int h = hstart; h < hend; ++h){\n for (int w = wstart; w < wend; ++w){\n int bottom_index = h*width + w;\n // mxnet_gpu_atomic_add(diff_val, offset_bottom_diff + bottom_index);\n atomicAdd(offset_bottom_diff + bottom_index, diff_val);\n }\n }\n }\n}\n\n\ntemplate\ninline void PSROIPoolBackwardAcc(const Tensor &in_grad,\n const Tensor &out_grad,\n const Tensor &bbox,\n const Tensor &mapping_channel,\n const float spatial_scale,\n const int output_dim_) {\n // LOG(INFO) << \"PSROIPoolBackward\";\n const DType *top_diff = out_grad.dptr_;\n const DType *bottom_rois = bbox.dptr_;\n DType *bottom_diff = in_grad.dptr_;\n DType *mapping_channel_ptr = mapping_channel.dptr_;\n const int count = out_grad.shape_.Size();\n const int num_rois = bbox.size(0);\n const int channels = in_grad.size(1);\n const int height = in_grad.size(2);\n const int width = in_grad.size(3);\n const int pooled_height = out_grad.size(2);\n const int pooled_width = out_grad.size(3);\n cudaStream_t stream = Stream::GetStream(in_grad.stream_);\n PSROIPoolBackwardAccKernel << > >(\n count, top_diff, mapping_channel_ptr, num_rois, spatial_scale, channels, height, width,\n pooled_height, pooled_width, output_dim_, bottom_diff, bottom_rois);\n PSROIPOOLING_CUDA_CHECK(cudaPeekAtLastError());\n}\n\n} // namespace cuda\n\ntemplate\ninline void PSROIPoolForward(const Tensor &out,\n const Tensor &data,\n const Tensor &bbox,\n const Tensor &mapping_channel,\n const float spatial_scale,\n const int output_dim_,\n const int group_size_) {\n cuda::PSROIPoolForward(out, data, bbox, mapping_channel, spatial_scale, output_dim_, group_size_);\n}\n\ntemplate\ninline void PSROIPoolBackwardAcc(const Tensor &in_grad,\n const Tensor &out_grad,\n const Tensor &bbox,\n const Tensor &mapping_channel,\n const float spatial_scale,\n const int output_dim_) {\n cuda::PSROIPoolBackwardAcc(in_grad, out_grad, bbox, mapping_channel, spatial_scale, output_dim_);\n}\n\n} // namespace mshadow\n\n\nnamespace mxnet {\nnamespace op {\n\ntemplate<>\nOperator* CreateOp(PSROIPoolingParam param, int dtype) {\n Operator* op = NULL;\n MSHADOW_REAL_TYPE_SWITCH(dtype, DType, {\n op = new PSROIPoolingOp(param);\n });\n return op;\n}\n\n} // namespace op\n} // namespace mxnet" }, { "path": "rfcn/operator_py/__init__.py", "content": "" }, { "path": "rfcn/operator_py/box_annotator_ohem.py", "content": "# --------------------------------------------------------\n# Deep Feature Flow\n# Copyright (c) 2017 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Written by Yuwen Xiong\n# --------------------------------------------------------\n\n\"\"\"\nProposal Target Operator selects foreground and background roi and assigns label, bbox_transform to them.\n\"\"\"\n\nimport mxnet as mx\nimport numpy as np\nfrom distutils.util import strtobool\n\n\n\n\nclass BoxAnnotatorOHEMOperator(mx.operator.CustomOp):\n def __init__(self, num_classes, num_reg_classes, roi_per_img):\n super(BoxAnnotatorOHEMOperator, self).__init__()\n self._num_classes = num_classes\n self._num_reg_classes = num_reg_classes\n self._roi_per_img = roi_per_img\n\n def forward(self, is_train, req, in_data, out_data, aux):\n\n cls_score = in_data[0]\n bbox_pred = in_data[1]\n labels = in_data[2].asnumpy()\n bbox_targets = in_data[3]\n bbox_weights = in_data[4]\n\n per_roi_loss_cls = mx.nd.SoftmaxActivation(cls_score) + 1e-14\n per_roi_loss_cls = per_roi_loss_cls.asnumpy()\n per_roi_loss_cls = per_roi_loss_cls[np.arange(per_roi_loss_cls.shape[0], dtype='int'), labels.astype('int')]\n per_roi_loss_cls = -1 * np.log(per_roi_loss_cls)\n per_roi_loss_cls = np.reshape(per_roi_loss_cls, newshape=(-1,))\n\n per_roi_loss_bbox = bbox_weights * mx.nd.smooth_l1((bbox_pred - bbox_targets), scalar=1.0)\n per_roi_loss_bbox = mx.nd.sum(per_roi_loss_bbox, axis=1).asnumpy()\n\n top_k_per_roi_loss = np.argsort(per_roi_loss_cls + per_roi_loss_bbox)\n labels_ohem = labels\n labels_ohem[top_k_per_roi_loss[::-1][self._roi_per_img:]] = -1\n bbox_weights_ohem = bbox_weights.asnumpy()\n bbox_weights_ohem[top_k_per_roi_loss[::-1][self._roi_per_img:]] = 0\n\n labels_ohem = mx.nd.array(labels_ohem)\n bbox_weights_ohem = mx.nd.array(bbox_weights_ohem)\n\n for ind, val in enumerate([labels_ohem, bbox_weights_ohem]):\n self.assign(out_data[ind], req[ind], val)\n\n\n def backward(self, req, out_grad, in_data, out_data, in_grad, aux):\n for i in range(len(in_grad)):\n self.assign(in_grad[i], req[i], 0)\n\n\n@mx.operator.register('BoxAnnotatorOHEM')\nclass BoxAnnotatorOHEMProp(mx.operator.CustomOpProp):\n def __init__(self, num_classes, num_reg_classes, roi_per_img):\n super(BoxAnnotatorOHEMProp, self).__init__(need_top_grad=False)\n self._num_classes = int(num_classes)\n self._num_reg_classes = int(num_reg_classes)\n self._roi_per_img = int(roi_per_img)\n\n def list_arguments(self):\n return ['cls_score', 'bbox_pred', 'labels', 'bbox_targets', 'bbox_weights']\n\n def list_outputs(self):\n return ['labels_ohem', 'bbox_weights_ohem']\n\n def infer_shape(self, in_shape):\n labels_shape = in_shape[2]\n bbox_weights_shape = in_shape[4]\n\n return in_shape, \\\n [labels_shape, bbox_weights_shape]\n\n def create_operator(self, ctx, shapes, dtypes):\n return BoxAnnotatorOHEMOperator(self._num_classes, self._num_reg_classes, self._roi_per_img)\n\n def declare_backward_dependency(self, out_grad, in_data, out_data):\n return []\n" }, { "path": "rfcn/operator_py/proposal.py", "content": "# --------------------------------------------------------\n# Deep Feature Flow\n# Copyright (c) 2017 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Modified by Yuwen Xiong\n# --------------------------------------------------------\n# Based on:\n# MX-RCNN\n# Copyright (c) 2016 by Contributors\n# Licence under The Apache 2.0 License\n# https://github.com/ijkguo/mx-rcnn/\n# --------------------------------------------------------\n\n\"\"\"\nProposal Operator transform anchor coordinates into ROI coordinates with prediction results on\nclassification probability and bounding box prediction results, and image size and scale information.\n\"\"\"\n\nimport mxnet as mx\nimport numpy as np\nimport numpy.random as npr\nfrom distutils.util import strtobool\n\nfrom bbox.bbox_transform import bbox_pred, clip_boxes\nfrom rpn.generate_anchor import generate_anchors\nfrom nms.nms import py_nms_wrapper, cpu_nms_wrapper, gpu_nms_wrapper\n\nDEBUG = False\n\n\nclass ProposalOperator(mx.operator.CustomOp):\n def __init__(self, feat_stride, scales, ratios, output_score,\n rpn_pre_nms_top_n, rpn_post_nms_top_n, threshold, rpn_min_size):\n super(ProposalOperator, self).__init__()\n self._feat_stride = feat_stride\n self._scales = np.fromstring(scales[1:-1], dtype=float, sep=',')\n self._ratios = np.fromstring(ratios[1:-1], dtype=float, sep=',')\n self._anchors = generate_anchors(base_size=self._feat_stride, scales=self._scales, ratios=self._ratios)\n self._num_anchors = self._anchors.shape[0]\n self._output_score = output_score\n self._rpn_pre_nms_top_n = rpn_pre_nms_top_n\n self._rpn_post_nms_top_n = rpn_post_nms_top_n\n self._threshold = threshold\n self._rpn_min_size = rpn_min_size\n\n if DEBUG:\n print 'feat_stride: {}'.format(self._feat_stride)\n print 'anchors:'\n print self._anchors\n\n def forward(self, is_train, req, in_data, out_data, aux):\n nms = gpu_nms_wrapper(self._threshold, in_data[0].context.device_id)\n\n batch_size = in_data[0].shape[0]\n if batch_size > 1:\n raise ValueError(\"Sorry, multiple images each device is not implemented\")\n\n # for each (H, W) location i\n # generate A anchor boxes centered on cell i\n # apply predicted bbox deltas at cell i to each of the A anchors\n # clip predicted boxes to image\n # remove predicted boxes with either height or width < threshold\n # sort all (proposal, score) pairs by score from highest to lowest\n # take top pre_nms_topN proposals before NMS\n # apply NMS with threshold 0.7 to remaining proposals\n # take after_nms_topN proposals after NMS\n # return the top proposals (-> RoIs top, scores top)\n\n pre_nms_topN = self._rpn_pre_nms_top_n\n post_nms_topN = self._rpn_post_nms_top_n\n min_size = self._rpn_min_size\n\n # the first set of anchors are background probabilities\n # keep the second part\n scores = in_data[0].asnumpy()[:, self._num_anchors:, :, :]\n bbox_deltas = in_data[1].asnumpy()\n im_info = in_data[2].asnumpy()[0, :]\n\n if DEBUG:\n print 'im_size: ({}, {})'.format(im_info[0], im_info[1])\n print 'scale: {}'.format(im_info[2])\n\n # 1. Generate proposals from bbox_deltas and shifted anchors\n # use real image size instead of padded feature map sizes\n height, width = int(im_info[0] / self._feat_stride), int(im_info[1] / self._feat_stride)\n\n if DEBUG:\n print 'score map size: {}'.format(scores.shape)\n print \"resudial: {}\".format((scores.shape[2] - height, scores.shape[3] - width))\n\n # Enumerate all shifts\n shift_x = np.arange(0, width) * self._feat_stride\n shift_y = np.arange(0, height) * self._feat_stride\n shift_x, shift_y = np.meshgrid(shift_x, shift_y)\n shifts = np.vstack((shift_x.ravel(), shift_y.ravel(), shift_x.ravel(), shift_y.ravel())).transpose()\n\n # Enumerate all shifted anchors:\n #\n # add A anchors (1, A, 4) to\n # cell K shifts (K, 1, 4) to get\n # shift anchors (K, A, 4)\n # reshape to (K*A, 4) shifted anchors\n A = self._num_anchors\n K = shifts.shape[0]\n anchors = self._anchors.reshape((1, A, 4)) + shifts.reshape((1, K, 4)).transpose((1, 0, 2))\n anchors = anchors.reshape((K * A, 4))\n\n # Transpose and reshape predicted bbox transformations to get them\n # into the same order as the anchors:\n #\n # bbox deltas will be (1, 4 * A, H, W) format\n # transpose to (1, H, W, 4 * A)\n # reshape to (1 * H * W * A, 4) where rows are ordered by (h, w, a)\n # in slowest to fastest order\n bbox_deltas = self._clip_pad(bbox_deltas, (height, width))\n bbox_deltas = bbox_deltas.transpose((0, 2, 3, 1)).reshape((-1, 4))\n\n # Same story for the scores:\n #\n # scores are (1, A, H, W) format\n # transpose to (1, H, W, A)\n # reshape to (1 * H * W * A, 1) where rows are ordered by (h, w, a)\n scores = self._clip_pad(scores, (height, width))\n scores = scores.transpose((0, 2, 3, 1)).reshape((-1, 1))\n\n # Convert anchors into proposals via bbox transformations\n proposals = bbox_pred(anchors, bbox_deltas)\n\n # 2. clip predicted boxes to image\n proposals = clip_boxes(proposals, im_info[:2])\n\n # 3. remove predicted boxes with either height or width < threshold\n # (NOTE: convert min_size to input image scale stored in im_info[2])\n keep = self._filter_boxes(proposals, min_size * im_info[2])\n proposals = proposals[keep, :]\n scores = scores[keep]\n\n # 4. sort all (proposal, score) pairs by score from highest to lowest\n # 5. take top pre_nms_topN (e.g. 6000)\n order = scores.ravel().argsort()[::-1]\n if pre_nms_topN > 0:\n order = order[:pre_nms_topN]\n proposals = proposals[order, :]\n scores = scores[order]\n\n # 6. apply nms (e.g. threshold = 0.7)\n # 7. take after_nms_topN (e.g. 300)\n # 8. return the top proposals (-> RoIs top)\n det = np.hstack((proposals, scores)).astype(np.float32)\n keep = nms(det)\n if post_nms_topN > 0:\n keep = keep[:post_nms_topN]\n # pad to ensure output size remains unchanged\n if len(keep) < post_nms_topN:\n pad = npr.choice(keep, size=post_nms_topN - len(keep))\n keep = np.hstack((keep, pad))\n proposals = proposals[keep, :]\n scores = scores[keep]\n\n # Output rois array\n # Our RPN implementation only supports a single input image, so all\n # batch inds are 0\n batch_inds = np.zeros((proposals.shape[0], 1), dtype=np.float32)\n blob = np.hstack((batch_inds, proposals.astype(np.float32, copy=False)))\n self.assign(out_data[0], req[0], blob)\n\n if self._output_score:\n self.assign(out_data[1], req[1], scores.astype(np.float32, copy=False))\n\n def backward(self, req, out_grad, in_data, out_data, in_grad, aux):\n self.assign(in_grad[0], req[0], 0)\n self.assign(in_grad[1], req[1], 0)\n self.assign(in_grad[2], req[2], 0)\n\n @staticmethod\n def _filter_boxes(boxes, min_size):\n \"\"\" Remove all boxes with any side smaller than min_size \"\"\"\n ws = boxes[:, 2] - boxes[:, 0] + 1\n hs = boxes[:, 3] - boxes[:, 1] + 1\n keep = np.where((ws >= min_size) & (hs >= min_size))[0]\n return keep\n\n @staticmethod\n def _clip_pad(tensor, pad_shape):\n \"\"\"\n Clip boxes of the pad area.\n :param tensor: [n, c, H, W]\n :param pad_shape: [h, w]\n :return: [n, c, h, w]\n \"\"\"\n H, W = tensor.shape[2:]\n h, w = pad_shape\n\n if h < H or w < W:\n tensor = tensor[:, :, :h, :w].copy()\n\n return tensor\n\n\n@mx.operator.register(\"proposal\")\nclass ProposalProp(mx.operator.CustomOpProp):\n def __init__(self, feat_stride='16', scales='(8, 16, 32)', ratios='(0.5, 1, 2)', output_score='False',\n rpn_pre_nms_top_n='6000', rpn_post_nms_top_n='300', threshold='0.3', rpn_min_size='16'):\n super(ProposalProp, self).__init__(need_top_grad=False)\n self._feat_stride = int(feat_stride)\n self._scales = scales\n self._ratios = ratios\n self._output_score = strtobool(output_score)\n self._rpn_pre_nms_top_n = int(rpn_pre_nms_top_n)\n self._rpn_post_nms_top_n = int(rpn_post_nms_top_n)\n self._threshold = float(threshold)\n self._rpn_min_size = int(rpn_min_size)\n\n def list_arguments(self):\n return ['cls_prob', 'bbox_pred', 'im_info']\n\n def list_outputs(self):\n if self._output_score:\n return ['output', 'score']\n else:\n return ['output']\n\n def infer_shape(self, in_shape):\n cls_prob_shape = in_shape[0]\n bbox_pred_shape = in_shape[1]\n assert cls_prob_shape[0] == bbox_pred_shape[0], 'ROI number does not equal in cls and reg'\n\n batch_size = cls_prob_shape[0]\n im_info_shape = (batch_size, 3)\n output_shape = (self._rpn_post_nms_top_n, 5)\n score_shape = (self._rpn_post_nms_top_n, 1)\n\n if self._output_score:\n return [cls_prob_shape, bbox_pred_shape, im_info_shape], [output_shape, score_shape]\n else:\n return [cls_prob_shape, bbox_pred_shape, im_info_shape], [output_shape]\n\n def create_operator(self, ctx, shapes, dtypes):\n return ProposalOperator(self._feat_stride, self._scales, self._ratios, self._output_score,\n self._rpn_pre_nms_top_n, self._rpn_post_nms_top_n, self._threshold, self._rpn_min_size)\n\n def declare_backward_dependency(self, out_grad, in_data, out_data):\n return []\n" }, { "path": "rfcn/operator_py/proposal_target.py", "content": "# --------------------------------------------------------\n# Deep Feature Flow\n# Copyright (c) 2017 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Modified by Yuwen Xiong\n# --------------------------------------------------------\n# Based on:\n# MX-RCNN\n# Copyright (c) 2016 by Contributors\n# Licence under The Apache 2.0 License\n# https://github.com/ijkguo/mx-rcnn/\n# --------------------------------------------------------\n\n\"\"\"\nProposal Target Operator selects foreground and background roi and assigns label, bbox_transform to them.\n\"\"\"\n\nimport mxnet as mx\nimport numpy as np\nfrom distutils.util import strtobool\nfrom easydict import EasyDict as edict\nimport cPickle\n\n\nfrom core.rcnn import sample_rois\n\nDEBUG = False\n\n\nclass ProposalTargetOperator(mx.operator.CustomOp):\n def __init__(self, num_classes, batch_images, batch_rois, cfg, fg_fraction):\n super(ProposalTargetOperator, self).__init__()\n self._num_classes = num_classes\n self._batch_images = batch_images\n self._batch_rois = batch_rois\n self._cfg = cfg\n self._fg_fraction = fg_fraction\n\n if DEBUG:\n self._count = 0\n self._fg_num = 0\n self._bg_num = 0\n\n def forward(self, is_train, req, in_data, out_data, aux):\n assert self._batch_rois == -1 or self._batch_rois % self._batch_images == 0, \\\n 'batchimages {} must devide batch_rois {}'.format(self._batch_images, self._batch_rois)\n all_rois = in_data[0].asnumpy()\n gt_boxes = in_data[1].asnumpy()\n\n if self._batch_rois == -1:\n rois_per_image = all_rois.shape[0] + gt_boxes.shape[0]\n fg_rois_per_image = rois_per_image\n else:\n rois_per_image = self._batch_rois / self._batch_images\n fg_rois_per_image = np.round(self._fg_fraction * rois_per_image).astype(int)\n\n\n # Include ground-truth boxes in the set of candidate rois\n zeros = np.zeros((gt_boxes.shape[0], 1), dtype=gt_boxes.dtype)\n all_rois = np.vstack((all_rois, np.hstack((zeros, gt_boxes[:, :-1]))))\n # Sanity check: single batch only\n assert np.all(all_rois[:, 0] == 0), 'Only single item batches are supported'\n\n rois, labels, bbox_targets, bbox_weights = \\\n sample_rois(all_rois, fg_rois_per_image, rois_per_image, self._num_classes, self._cfg, gt_boxes=gt_boxes)\n\n if DEBUG:\n print \"labels=\", labels\n print 'num fg: {}'.format((labels > 0).sum())\n print 'num bg: {}'.format((labels == 0).sum())\n self._count += 1\n self._fg_num += (labels > 0).sum()\n self._bg_num += (labels == 0).sum()\n print \"self._count=\", self._count\n print 'num fg avg: {}'.format(self._fg_num / self._count)\n print 'num bg avg: {}'.format(self._bg_num / self._count)\n print 'ratio: {:.3f}'.format(float(self._fg_num) / float(self._bg_num))\n\n for ind, val in enumerate([rois, labels, bbox_targets, bbox_weights]):\n self.assign(out_data[ind], req[ind], val)\n\n def backward(self, req, out_grad, in_data, out_data, in_grad, aux):\n self.assign(in_grad[0], req[0], 0)\n self.assign(in_grad[1], req[1], 0)\n\n\n@mx.operator.register('proposal_target')\nclass ProposalTargetProp(mx.operator.CustomOpProp):\n def __init__(self, num_classes, batch_images, batch_rois, cfg, fg_fraction='0.25'):\n super(ProposalTargetProp, self).__init__(need_top_grad=False)\n self._num_classes = int(num_classes)\n self._batch_images = int(batch_images)\n self._batch_rois = int(batch_rois)\n self._cfg = cPickle.loads(cfg)\n self._fg_fraction = float(fg_fraction)\n\n def list_arguments(self):\n return ['rois', 'gt_boxes']\n\n def list_outputs(self):\n return ['rois_output', 'label', 'bbox_target', 'bbox_weight']\n\n def infer_shape(self, in_shape):\n rpn_rois_shape = in_shape[0]\n gt_boxes_shape = in_shape[1]\n\n rois = rpn_rois_shape[0] + gt_boxes_shape[0] if self._batch_rois == -1 else self._batch_rois\n\n output_rois_shape = (rois, 5)\n label_shape = (rois, )\n bbox_target_shape = (rois, self._num_classes * 4)\n bbox_weight_shape = (rois, self._num_classes * 4)\n\n return [rpn_rois_shape, gt_boxes_shape], \\\n [output_rois_shape, label_shape, bbox_target_shape, bbox_weight_shape]\n\n def create_operator(self, ctx, shapes, dtypes):\n return ProposalTargetOperator(self._num_classes, self._batch_images, self._batch_rois, self._cfg, self._fg_fraction)\n\n def declare_backward_dependency(self, out_grad, in_data, out_data):\n return []\n" }, { "path": "rfcn/operator_py/rpn_inv_normalize.py", "content": "# --------------------------------------------------------\n# Deep Feature Flow\n# Copyright (c) 2017 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Written by Xizhou Zhu\n# --------------------------------------------------------\n\nimport mxnet as mx\nimport numpy as np\nfrom distutils.util import strtobool\n\nclass RPNInvNormalizeOperator(mx.operator.CustomOp):\n def __init__(self, num_anchors, bbox_mean, bbox_std):\n super(RPNInvNormalizeOperator, self).__init__()\n self._num_anchors = num_anchors\n self._bbox_mean = mx.ndarray.Reshape(mx.nd.array(bbox_mean), shape=(1,4,1,1))\n self._bbox_std = mx.ndarray.Reshape(mx.nd.array(bbox_std), shape=(1,4,1,1))\n\n def forward(self, is_train, req, in_data, out_data, aux):\n bbox_pred = in_data[0]\n tile_shape = (bbox_pred.shape[0], self._num_anchors, bbox_pred.shape[2], bbox_pred.shape[3])\n bbox_mean = mx.ndarray.tile(self._bbox_mean.as_in_context(bbox_pred.context), reps=tile_shape)\n bbox_std = mx.ndarray.tile(self._bbox_std.as_in_context(bbox_pred.context), reps=tile_shape)\n bbox_pred = bbox_pred * bbox_std + bbox_mean\n\n self.assign(out_data[0], req[0], bbox_pred)\n\n def backward(self, req, out_grad, in_data, out_data, in_grad, aux):\n self.assign(in_grad[0], req[0], 0)\n\n@mx.operator.register('rpn_inv_normalize')\nclass RPNInvNormalizeProp(mx.operator.CustomOpProp):\n def __init__(self, num_anchors, bbox_mean='(0.0, 0.0, 0.0, 0.0)', bbox_std='0.1, 0.1, 0.2, 0.2'):\n super(RPNInvNormalizeProp, self).__init__(need_top_grad=False)\n self._num_anchors = int(num_anchors)\n self._bbox_mean = np.fromstring(bbox_mean[1:-1], dtype=float, sep=',')\n self._bbox_std = np.fromstring(bbox_std[1:-1], dtype=float, sep=',')\n\n def list_arguments(self):\n return ['bbox_pred']\n\n def list_outputs(self):\n return ['out_bbox_pred']\n\n def infer_shape(self, in_shape):\n\n return [in_shape[0]], \\\n [in_shape[0]]\n\n def create_operator(self, ctx, shapes, dtypes):\n return RPNInvNormalizeOperator(self._num_anchors, self._bbox_mean, self._bbox_std)\n\n def declare_backward_dependency(self, out_grad, in_data, out_data):\n return []\n" }, { "path": "rfcn/symbols/__init__.py", "content": "import resnet_v1_101_rfcn\n" }, { "path": "rfcn/symbols/resnet_v1_101_rfcn.py", "content": "# --------------------------------------------------------\n# Deep Feature Flow\n# Copyright (c) 2017 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Written by Yuwen Xiong, Xizhou Zhu\n# --------------------------------------------------------\n\nimport cPickle\nimport mxnet as mx\nfrom utils.symbol import Symbol\nfrom operator_py.proposal import *\nfrom operator_py.proposal_target import *\nfrom operator_py.box_annotator_ohem import *\nfrom operator_py.rpn_inv_normalize import *\n\nclass resnet_v1_101_rfcn(Symbol):\n \n def __init__(self):\n \"\"\"\n Use __init__ to define parameter network needs\n \"\"\"\n self.eps = 1e-5\n self.use_global_stats = True\n self.workspace = 512\n self.units = (3, 4, 23, 3) # use for 101\n self.filter_list = [256, 512, 1024, 2048]\n \n def get_resnet_v1(self, data):\n conv1 = mx.symbol.Convolution(name='conv1', data=data , num_filter=64, pad=(3,3), kernel=(7,7), stride=(2,2), no_bias=True)\n bn_conv1 = mx.symbol.BatchNorm(name='bn_conv1', data=conv1 , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale_conv1 = bn_conv1\n conv1_relu = mx.symbol.Activation(name='conv1_relu', data=scale_conv1 , act_type='relu')\n pool1 = mx.symbol.Pooling(name='pool1', data=conv1_relu , pad=(1,1), kernel=(3,3), stride=(2,2), pool_type='max')\n res2a_branch1 = mx.symbol.Convolution(name='res2a_branch1', data=pool1 , num_filter=256, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn2a_branch1 = mx.symbol.BatchNorm(name='bn2a_branch1', data=res2a_branch1 , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale2a_branch1 = bn2a_branch1\n res2a_branch2a = mx.symbol.Convolution(name='res2a_branch2a', data=pool1 , num_filter=64, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn2a_branch2a = mx.symbol.BatchNorm(name='bn2a_branch2a', data=res2a_branch2a , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale2a_branch2a = bn2a_branch2a\n res2a_branch2a_relu = mx.symbol.Activation(name='res2a_branch2a_relu', data=scale2a_branch2a , act_type='relu')\n res2a_branch2b = mx.symbol.Convolution(name='res2a_branch2b', data=res2a_branch2a_relu , num_filter=64, pad=(1,1), kernel=(3,3), stride=(1,1), no_bias=True)\n bn2a_branch2b = mx.symbol.BatchNorm(name='bn2a_branch2b', data=res2a_branch2b , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale2a_branch2b = bn2a_branch2b\n res2a_branch2b_relu = mx.symbol.Activation(name='res2a_branch2b_relu', data=scale2a_branch2b , act_type='relu')\n res2a_branch2c = mx.symbol.Convolution(name='res2a_branch2c', data=res2a_branch2b_relu , num_filter=256, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn2a_branch2c = mx.symbol.BatchNorm(name='bn2a_branch2c', data=res2a_branch2c , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale2a_branch2c = bn2a_branch2c\n res2a = mx.symbol.broadcast_add(name='res2a', *[scale2a_branch1,scale2a_branch2c] )\n res2a_relu = mx.symbol.Activation(name='res2a_relu', data=res2a , act_type='relu')\n res2b_branch2a = mx.symbol.Convolution(name='res2b_branch2a', data=res2a_relu , num_filter=64, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn2b_branch2a = mx.symbol.BatchNorm(name='bn2b_branch2a', data=res2b_branch2a , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale2b_branch2a = bn2b_branch2a\n res2b_branch2a_relu = mx.symbol.Activation(name='res2b_branch2a_relu', data=scale2b_branch2a , act_type='relu')\n res2b_branch2b = mx.symbol.Convolution(name='res2b_branch2b', data=res2b_branch2a_relu , num_filter=64, pad=(1,1), kernel=(3,3), stride=(1,1), no_bias=True)\n bn2b_branch2b = mx.symbol.BatchNorm(name='bn2b_branch2b', data=res2b_branch2b , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale2b_branch2b = bn2b_branch2b\n res2b_branch2b_relu = mx.symbol.Activation(name='res2b_branch2b_relu', data=scale2b_branch2b , act_type='relu')\n res2b_branch2c = mx.symbol.Convolution(name='res2b_branch2c', data=res2b_branch2b_relu , num_filter=256, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn2b_branch2c = mx.symbol.BatchNorm(name='bn2b_branch2c', data=res2b_branch2c , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale2b_branch2c = bn2b_branch2c\n res2b = mx.symbol.broadcast_add(name='res2b', *[res2a_relu,scale2b_branch2c] )\n res2b_relu = mx.symbol.Activation(name='res2b_relu', data=res2b , act_type='relu')\n res2c_branch2a = mx.symbol.Convolution(name='res2c_branch2a', data=res2b_relu , num_filter=64, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn2c_branch2a = mx.symbol.BatchNorm(name='bn2c_branch2a', data=res2c_branch2a , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale2c_branch2a = bn2c_branch2a\n res2c_branch2a_relu = mx.symbol.Activation(name='res2c_branch2a_relu', data=scale2c_branch2a , act_type='relu')\n res2c_branch2b = mx.symbol.Convolution(name='res2c_branch2b', data=res2c_branch2a_relu , num_filter=64, pad=(1,1), kernel=(3,3), stride=(1,1), no_bias=True)\n bn2c_branch2b = mx.symbol.BatchNorm(name='bn2c_branch2b', data=res2c_branch2b , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale2c_branch2b = bn2c_branch2b\n res2c_branch2b_relu = mx.symbol.Activation(name='res2c_branch2b_relu', data=scale2c_branch2b , act_type='relu')\n res2c_branch2c = mx.symbol.Convolution(name='res2c_branch2c', data=res2c_branch2b_relu , num_filter=256, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn2c_branch2c = mx.symbol.BatchNorm(name='bn2c_branch2c', data=res2c_branch2c , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale2c_branch2c = bn2c_branch2c\n res2c = mx.symbol.broadcast_add(name='res2c', *[res2b_relu,scale2c_branch2c] )\n res2c_relu = mx.symbol.Activation(name='res2c_relu', data=res2c , act_type='relu')\n res3a_branch1 = mx.symbol.Convolution(name='res3a_branch1', data=res2c_relu , num_filter=512, pad=(0,0), kernel=(1,1), stride=(2,2), no_bias=True)\n bn3a_branch1 = mx.symbol.BatchNorm(name='bn3a_branch1', data=res3a_branch1 , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale3a_branch1 = bn3a_branch1\n res3a_branch2a = mx.symbol.Convolution(name='res3a_branch2a', data=res2c_relu , num_filter=128, pad=(0,0), kernel=(1,1), stride=(2,2), no_bias=True)\n bn3a_branch2a = mx.symbol.BatchNorm(name='bn3a_branch2a', data=res3a_branch2a , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale3a_branch2a = bn3a_branch2a\n res3a_branch2a_relu = mx.symbol.Activation(name='res3a_branch2a_relu', data=scale3a_branch2a , act_type='relu')\n res3a_branch2b = mx.symbol.Convolution(name='res3a_branch2b', data=res3a_branch2a_relu , num_filter=128, pad=(1,1), kernel=(3,3), stride=(1,1), no_bias=True)\n bn3a_branch2b = mx.symbol.BatchNorm(name='bn3a_branch2b', data=res3a_branch2b , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale3a_branch2b = bn3a_branch2b\n res3a_branch2b_relu = mx.symbol.Activation(name='res3a_branch2b_relu', data=scale3a_branch2b , act_type='relu')\n res3a_branch2c = mx.symbol.Convolution(name='res3a_branch2c', data=res3a_branch2b_relu , num_filter=512, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn3a_branch2c = mx.symbol.BatchNorm(name='bn3a_branch2c', data=res3a_branch2c , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale3a_branch2c = bn3a_branch2c\n res3a = mx.symbol.broadcast_add(name='res3a', *[scale3a_branch1,scale3a_branch2c] )\n res3a_relu = mx.symbol.Activation(name='res3a_relu', data=res3a , act_type='relu')\n res3b1_branch2a = mx.symbol.Convolution(name='res3b1_branch2a', data=res3a_relu , num_filter=128, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn3b1_branch2a = mx.symbol.BatchNorm(name='bn3b1_branch2a', data=res3b1_branch2a , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale3b1_branch2a = bn3b1_branch2a\n res3b1_branch2a_relu = mx.symbol.Activation(name='res3b1_branch2a_relu', data=scale3b1_branch2a , act_type='relu')\n res3b1_branch2b = mx.symbol.Convolution(name='res3b1_branch2b', data=res3b1_branch2a_relu , num_filter=128, pad=(1,1), kernel=(3,3), stride=(1,1), no_bias=True)\n bn3b1_branch2b = mx.symbol.BatchNorm(name='bn3b1_branch2b', data=res3b1_branch2b , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale3b1_branch2b = bn3b1_branch2b\n res3b1_branch2b_relu = mx.symbol.Activation(name='res3b1_branch2b_relu', data=scale3b1_branch2b , act_type='relu')\n res3b1_branch2c = mx.symbol.Convolution(name='res3b1_branch2c', data=res3b1_branch2b_relu , num_filter=512, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn3b1_branch2c = mx.symbol.BatchNorm(name='bn3b1_branch2c', data=res3b1_branch2c , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale3b1_branch2c = bn3b1_branch2c\n res3b1 = mx.symbol.broadcast_add(name='res3b1', *[res3a_relu,scale3b1_branch2c] )\n res3b1_relu = mx.symbol.Activation(name='res3b1_relu', data=res3b1 , act_type='relu')\n res3b2_branch2a = mx.symbol.Convolution(name='res3b2_branch2a', data=res3b1_relu , num_filter=128, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn3b2_branch2a = mx.symbol.BatchNorm(name='bn3b2_branch2a', data=res3b2_branch2a , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale3b2_branch2a = bn3b2_branch2a\n res3b2_branch2a_relu = mx.symbol.Activation(name='res3b2_branch2a_relu', data=scale3b2_branch2a , act_type='relu')\n res3b2_branch2b = mx.symbol.Convolution(name='res3b2_branch2b', data=res3b2_branch2a_relu , num_filter=128, pad=(1,1), kernel=(3,3), stride=(1,1), no_bias=True)\n bn3b2_branch2b = mx.symbol.BatchNorm(name='bn3b2_branch2b', data=res3b2_branch2b , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale3b2_branch2b = bn3b2_branch2b\n res3b2_branch2b_relu = mx.symbol.Activation(name='res3b2_branch2b_relu', data=scale3b2_branch2b , act_type='relu')\n res3b2_branch2c = mx.symbol.Convolution(name='res3b2_branch2c', data=res3b2_branch2b_relu , num_filter=512, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn3b2_branch2c = mx.symbol.BatchNorm(name='bn3b2_branch2c', data=res3b2_branch2c , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale3b2_branch2c = bn3b2_branch2c\n res3b2 = mx.symbol.broadcast_add(name='res3b2', *[res3b1_relu,scale3b2_branch2c] )\n res3b2_relu = mx.symbol.Activation(name='res3b2_relu', data=res3b2 , act_type='relu')\n res3b3_branch2a = mx.symbol.Convolution(name='res3b3_branch2a', data=res3b2_relu , num_filter=128, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn3b3_branch2a = mx.symbol.BatchNorm(name='bn3b3_branch2a', data=res3b3_branch2a , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale3b3_branch2a = bn3b3_branch2a\n res3b3_branch2a_relu = mx.symbol.Activation(name='res3b3_branch2a_relu', data=scale3b3_branch2a , act_type='relu')\n res3b3_branch2b = mx.symbol.Convolution(name='res3b3_branch2b', data=res3b3_branch2a_relu , num_filter=128, pad=(1,1), kernel=(3,3), stride=(1,1), no_bias=True)\n bn3b3_branch2b = mx.symbol.BatchNorm(name='bn3b3_branch2b', data=res3b3_branch2b , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale3b3_branch2b = bn3b3_branch2b\n res3b3_branch2b_relu = mx.symbol.Activation(name='res3b3_branch2b_relu', data=scale3b3_branch2b , act_type='relu')\n res3b3_branch2c = mx.symbol.Convolution(name='res3b3_branch2c', data=res3b3_branch2b_relu , num_filter=512, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn3b3_branch2c = mx.symbol.BatchNorm(name='bn3b3_branch2c', data=res3b3_branch2c , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale3b3_branch2c = bn3b3_branch2c\n res3b3 = mx.symbol.broadcast_add(name='res3b3', *[res3b2_relu,scale3b3_branch2c] )\n res3b3_relu = mx.symbol.Activation(name='res3b3_relu', data=res3b3 , act_type='relu')\n res4a_branch1 = mx.symbol.Convolution(name='res4a_branch1', data=res3b3_relu , num_filter=1024, pad=(0,0), kernel=(1,1), stride=(2,2), no_bias=True)\n bn4a_branch1 = mx.symbol.BatchNorm(name='bn4a_branch1', data=res4a_branch1 , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4a_branch1 = bn4a_branch1\n res4a_branch2a = mx.symbol.Convolution(name='res4a_branch2a', data=res3b3_relu , num_filter=256, pad=(0,0), kernel=(1,1), stride=(2,2), no_bias=True)\n bn4a_branch2a = mx.symbol.BatchNorm(name='bn4a_branch2a', data=res4a_branch2a , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4a_branch2a = bn4a_branch2a\n res4a_branch2a_relu = mx.symbol.Activation(name='res4a_branch2a_relu', data=scale4a_branch2a , act_type='relu')\n res4a_branch2b = mx.symbol.Convolution(name='res4a_branch2b', data=res4a_branch2a_relu , num_filter=256, pad=(1,1), kernel=(3,3), stride=(1,1), no_bias=True)\n bn4a_branch2b = mx.symbol.BatchNorm(name='bn4a_branch2b', data=res4a_branch2b , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4a_branch2b = bn4a_branch2b\n res4a_branch2b_relu = mx.symbol.Activation(name='res4a_branch2b_relu', data=scale4a_branch2b , act_type='relu')\n res4a_branch2c = mx.symbol.Convolution(name='res4a_branch2c', data=res4a_branch2b_relu , num_filter=1024, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn4a_branch2c = mx.symbol.BatchNorm(name='bn4a_branch2c', data=res4a_branch2c , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4a_branch2c = bn4a_branch2c\n res4a = mx.symbol.broadcast_add(name='res4a', *[scale4a_branch1,scale4a_branch2c] )\n res4a_relu = mx.symbol.Activation(name='res4a_relu', data=res4a , act_type='relu')\n res4b1_branch2a = mx.symbol.Convolution(name='res4b1_branch2a', data=res4a_relu , num_filter=256, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn4b1_branch2a = mx.symbol.BatchNorm(name='bn4b1_branch2a', data=res4b1_branch2a , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b1_branch2a = bn4b1_branch2a\n res4b1_branch2a_relu = mx.symbol.Activation(name='res4b1_branch2a_relu', data=scale4b1_branch2a , act_type='relu')\n res4b1_branch2b = mx.symbol.Convolution(name='res4b1_branch2b', data=res4b1_branch2a_relu , num_filter=256, pad=(1,1), kernel=(3,3), stride=(1,1), no_bias=True)\n bn4b1_branch2b = mx.symbol.BatchNorm(name='bn4b1_branch2b', data=res4b1_branch2b , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b1_branch2b = bn4b1_branch2b\n res4b1_branch2b_relu = mx.symbol.Activation(name='res4b1_branch2b_relu', data=scale4b1_branch2b , act_type='relu')\n res4b1_branch2c = mx.symbol.Convolution(name='res4b1_branch2c', data=res4b1_branch2b_relu , num_filter=1024, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn4b1_branch2c = mx.symbol.BatchNorm(name='bn4b1_branch2c', data=res4b1_branch2c , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b1_branch2c = bn4b1_branch2c\n res4b1 = mx.symbol.broadcast_add(name='res4b1', *[res4a_relu,scale4b1_branch2c] )\n res4b1_relu = mx.symbol.Activation(name='res4b1_relu', data=res4b1 , act_type='relu')\n res4b2_branch2a = mx.symbol.Convolution(name='res4b2_branch2a', data=res4b1_relu , num_filter=256, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn4b2_branch2a = mx.symbol.BatchNorm(name='bn4b2_branch2a', data=res4b2_branch2a , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b2_branch2a = bn4b2_branch2a\n res4b2_branch2a_relu = mx.symbol.Activation(name='res4b2_branch2a_relu', data=scale4b2_branch2a , act_type='relu')\n res4b2_branch2b = mx.symbol.Convolution(name='res4b2_branch2b', data=res4b2_branch2a_relu , num_filter=256, pad=(1,1), kernel=(3,3), stride=(1,1), no_bias=True)\n bn4b2_branch2b = mx.symbol.BatchNorm(name='bn4b2_branch2b', data=res4b2_branch2b , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b2_branch2b = bn4b2_branch2b\n res4b2_branch2b_relu = mx.symbol.Activation(name='res4b2_branch2b_relu', data=scale4b2_branch2b , act_type='relu')\n res4b2_branch2c = mx.symbol.Convolution(name='res4b2_branch2c', data=res4b2_branch2b_relu , num_filter=1024, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn4b2_branch2c = mx.symbol.BatchNorm(name='bn4b2_branch2c', data=res4b2_branch2c , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b2_branch2c = bn4b2_branch2c\n res4b2 = mx.symbol.broadcast_add(name='res4b2', *[res4b1_relu,scale4b2_branch2c] )\n res4b2_relu = mx.symbol.Activation(name='res4b2_relu', data=res4b2 , act_type='relu')\n res4b3_branch2a = mx.symbol.Convolution(name='res4b3_branch2a', data=res4b2_relu , num_filter=256, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn4b3_branch2a = mx.symbol.BatchNorm(name='bn4b3_branch2a', data=res4b3_branch2a , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b3_branch2a = bn4b3_branch2a\n res4b3_branch2a_relu = mx.symbol.Activation(name='res4b3_branch2a_relu', data=scale4b3_branch2a , act_type='relu')\n res4b3_branch2b = mx.symbol.Convolution(name='res4b3_branch2b', data=res4b3_branch2a_relu , num_filter=256, pad=(1,1), kernel=(3,3), stride=(1,1), no_bias=True)\n bn4b3_branch2b = mx.symbol.BatchNorm(name='bn4b3_branch2b', data=res4b3_branch2b , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b3_branch2b = bn4b3_branch2b\n res4b3_branch2b_relu = mx.symbol.Activation(name='res4b3_branch2b_relu', data=scale4b3_branch2b , act_type='relu')\n res4b3_branch2c = mx.symbol.Convolution(name='res4b3_branch2c', data=res4b3_branch2b_relu , num_filter=1024, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn4b3_branch2c = mx.symbol.BatchNorm(name='bn4b3_branch2c', data=res4b3_branch2c , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b3_branch2c = bn4b3_branch2c\n res4b3 = mx.symbol.broadcast_add(name='res4b3', *[res4b2_relu,scale4b3_branch2c] )\n res4b3_relu = mx.symbol.Activation(name='res4b3_relu', data=res4b3 , act_type='relu')\n res4b4_branch2a = mx.symbol.Convolution(name='res4b4_branch2a', data=res4b3_relu , num_filter=256, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn4b4_branch2a = mx.symbol.BatchNorm(name='bn4b4_branch2a', data=res4b4_branch2a , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b4_branch2a = bn4b4_branch2a\n res4b4_branch2a_relu = mx.symbol.Activation(name='res4b4_branch2a_relu', data=scale4b4_branch2a , act_type='relu')\n res4b4_branch2b = mx.symbol.Convolution(name='res4b4_branch2b', data=res4b4_branch2a_relu , num_filter=256, pad=(1,1), kernel=(3,3), stride=(1,1), no_bias=True)\n bn4b4_branch2b = mx.symbol.BatchNorm(name='bn4b4_branch2b', data=res4b4_branch2b , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b4_branch2b = bn4b4_branch2b\n res4b4_branch2b_relu = mx.symbol.Activation(name='res4b4_branch2b_relu', data=scale4b4_branch2b , act_type='relu')\n res4b4_branch2c = mx.symbol.Convolution(name='res4b4_branch2c', data=res4b4_branch2b_relu , num_filter=1024, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn4b4_branch2c = mx.symbol.BatchNorm(name='bn4b4_branch2c', data=res4b4_branch2c , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b4_branch2c = bn4b4_branch2c\n res4b4 = mx.symbol.broadcast_add(name='res4b4', *[res4b3_relu,scale4b4_branch2c] )\n res4b4_relu = mx.symbol.Activation(name='res4b4_relu', data=res4b4 , act_type='relu')\n res4b5_branch2a = mx.symbol.Convolution(name='res4b5_branch2a', data=res4b4_relu , num_filter=256, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn4b5_branch2a = mx.symbol.BatchNorm(name='bn4b5_branch2a', data=res4b5_branch2a , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b5_branch2a = bn4b5_branch2a\n res4b5_branch2a_relu = mx.symbol.Activation(name='res4b5_branch2a_relu', data=scale4b5_branch2a , act_type='relu')\n res4b5_branch2b = mx.symbol.Convolution(name='res4b5_branch2b', data=res4b5_branch2a_relu , num_filter=256, pad=(1,1), kernel=(3,3), stride=(1,1), no_bias=True)\n bn4b5_branch2b = mx.symbol.BatchNorm(name='bn4b5_branch2b', data=res4b5_branch2b , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b5_branch2b = bn4b5_branch2b\n res4b5_branch2b_relu = mx.symbol.Activation(name='res4b5_branch2b_relu', data=scale4b5_branch2b , act_type='relu')\n res4b5_branch2c = mx.symbol.Convolution(name='res4b5_branch2c', data=res4b5_branch2b_relu , num_filter=1024, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn4b5_branch2c = mx.symbol.BatchNorm(name='bn4b5_branch2c', data=res4b5_branch2c , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b5_branch2c = bn4b5_branch2c\n res4b5 = mx.symbol.broadcast_add(name='res4b5', *[res4b4_relu,scale4b5_branch2c] )\n res4b5_relu = mx.symbol.Activation(name='res4b5_relu', data=res4b5 , act_type='relu')\n res4b6_branch2a = mx.symbol.Convolution(name='res4b6_branch2a', data=res4b5_relu , num_filter=256, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn4b6_branch2a = mx.symbol.BatchNorm(name='bn4b6_branch2a', data=res4b6_branch2a , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b6_branch2a = bn4b6_branch2a\n res4b6_branch2a_relu = mx.symbol.Activation(name='res4b6_branch2a_relu', data=scale4b6_branch2a , act_type='relu')\n res4b6_branch2b = mx.symbol.Convolution(name='res4b6_branch2b', data=res4b6_branch2a_relu , num_filter=256, pad=(1,1), kernel=(3,3), stride=(1,1), no_bias=True)\n bn4b6_branch2b = mx.symbol.BatchNorm(name='bn4b6_branch2b', data=res4b6_branch2b , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b6_branch2b = bn4b6_branch2b\n res4b6_branch2b_relu = mx.symbol.Activation(name='res4b6_branch2b_relu', data=scale4b6_branch2b , act_type='relu')\n res4b6_branch2c = mx.symbol.Convolution(name='res4b6_branch2c', data=res4b6_branch2b_relu , num_filter=1024, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn4b6_branch2c = mx.symbol.BatchNorm(name='bn4b6_branch2c', data=res4b6_branch2c , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b6_branch2c = bn4b6_branch2c\n res4b6 = mx.symbol.broadcast_add(name='res4b6', *[res4b5_relu,scale4b6_branch2c] )\n res4b6_relu = mx.symbol.Activation(name='res4b6_relu', data=res4b6 , act_type='relu')\n res4b7_branch2a = mx.symbol.Convolution(name='res4b7_branch2a', data=res4b6_relu , num_filter=256, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn4b7_branch2a = mx.symbol.BatchNorm(name='bn4b7_branch2a', data=res4b7_branch2a , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b7_branch2a = bn4b7_branch2a\n res4b7_branch2a_relu = mx.symbol.Activation(name='res4b7_branch2a_relu', data=scale4b7_branch2a , act_type='relu')\n res4b7_branch2b = mx.symbol.Convolution(name='res4b7_branch2b', data=res4b7_branch2a_relu , num_filter=256, pad=(1,1), kernel=(3,3), stride=(1,1), no_bias=True)\n bn4b7_branch2b = mx.symbol.BatchNorm(name='bn4b7_branch2b', data=res4b7_branch2b , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b7_branch2b = bn4b7_branch2b\n res4b7_branch2b_relu = mx.symbol.Activation(name='res4b7_branch2b_relu', data=scale4b7_branch2b , act_type='relu')\n res4b7_branch2c = mx.symbol.Convolution(name='res4b7_branch2c', data=res4b7_branch2b_relu , num_filter=1024, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn4b7_branch2c = mx.symbol.BatchNorm(name='bn4b7_branch2c', data=res4b7_branch2c , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b7_branch2c = bn4b7_branch2c\n res4b7 = mx.symbol.broadcast_add(name='res4b7', *[res4b6_relu,scale4b7_branch2c] )\n res4b7_relu = mx.symbol.Activation(name='res4b7_relu', data=res4b7 , act_type='relu')\n res4b8_branch2a = mx.symbol.Convolution(name='res4b8_branch2a', data=res4b7_relu , num_filter=256, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn4b8_branch2a = mx.symbol.BatchNorm(name='bn4b8_branch2a', data=res4b8_branch2a , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b8_branch2a = bn4b8_branch2a\n res4b8_branch2a_relu = mx.symbol.Activation(name='res4b8_branch2a_relu', data=scale4b8_branch2a , act_type='relu')\n res4b8_branch2b = mx.symbol.Convolution(name='res4b8_branch2b', data=res4b8_branch2a_relu , num_filter=256, pad=(1,1), kernel=(3,3), stride=(1,1), no_bias=True)\n bn4b8_branch2b = mx.symbol.BatchNorm(name='bn4b8_branch2b', data=res4b8_branch2b , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b8_branch2b = bn4b8_branch2b\n res4b8_branch2b_relu = mx.symbol.Activation(name='res4b8_branch2b_relu', data=scale4b8_branch2b , act_type='relu')\n res4b8_branch2c = mx.symbol.Convolution(name='res4b8_branch2c', data=res4b8_branch2b_relu , num_filter=1024, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn4b8_branch2c = mx.symbol.BatchNorm(name='bn4b8_branch2c', data=res4b8_branch2c , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b8_branch2c = bn4b8_branch2c\n res4b8 = mx.symbol.broadcast_add(name='res4b8', *[res4b7_relu,scale4b8_branch2c] )\n res4b8_relu = mx.symbol.Activation(name='res4b8_relu', data=res4b8 , act_type='relu')\n res4b9_branch2a = mx.symbol.Convolution(name='res4b9_branch2a', data=res4b8_relu , num_filter=256, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn4b9_branch2a = mx.symbol.BatchNorm(name='bn4b9_branch2a', data=res4b9_branch2a , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b9_branch2a = bn4b9_branch2a\n res4b9_branch2a_relu = mx.symbol.Activation(name='res4b9_branch2a_relu', data=scale4b9_branch2a , act_type='relu')\n res4b9_branch2b = mx.symbol.Convolution(name='res4b9_branch2b', data=res4b9_branch2a_relu , num_filter=256, pad=(1,1), kernel=(3,3), stride=(1,1), no_bias=True)\n bn4b9_branch2b = mx.symbol.BatchNorm(name='bn4b9_branch2b', data=res4b9_branch2b , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b9_branch2b = bn4b9_branch2b\n res4b9_branch2b_relu = mx.symbol.Activation(name='res4b9_branch2b_relu', data=scale4b9_branch2b , act_type='relu')\n res4b9_branch2c = mx.symbol.Convolution(name='res4b9_branch2c', data=res4b9_branch2b_relu , num_filter=1024, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn4b9_branch2c = mx.symbol.BatchNorm(name='bn4b9_branch2c', data=res4b9_branch2c , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b9_branch2c = bn4b9_branch2c\n res4b9 = mx.symbol.broadcast_add(name='res4b9', *[res4b8_relu,scale4b9_branch2c] )\n res4b9_relu = mx.symbol.Activation(name='res4b9_relu', data=res4b9 , act_type='relu')\n res4b10_branch2a = mx.symbol.Convolution(name='res4b10_branch2a', data=res4b9_relu , num_filter=256, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn4b10_branch2a = mx.symbol.BatchNorm(name='bn4b10_branch2a', data=res4b10_branch2a , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b10_branch2a = bn4b10_branch2a\n res4b10_branch2a_relu = mx.symbol.Activation(name='res4b10_branch2a_relu', data=scale4b10_branch2a , act_type='relu')\n res4b10_branch2b = mx.symbol.Convolution(name='res4b10_branch2b', data=res4b10_branch2a_relu , num_filter=256, pad=(1,1), kernel=(3,3), stride=(1,1), no_bias=True)\n bn4b10_branch2b = mx.symbol.BatchNorm(name='bn4b10_branch2b', data=res4b10_branch2b , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b10_branch2b = bn4b10_branch2b\n res4b10_branch2b_relu = mx.symbol.Activation(name='res4b10_branch2b_relu', data=scale4b10_branch2b , act_type='relu')\n res4b10_branch2c = mx.symbol.Convolution(name='res4b10_branch2c', data=res4b10_branch2b_relu , num_filter=1024, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn4b10_branch2c = mx.symbol.BatchNorm(name='bn4b10_branch2c', data=res4b10_branch2c , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b10_branch2c = bn4b10_branch2c\n res4b10 = mx.symbol.broadcast_add(name='res4b10', *[res4b9_relu,scale4b10_branch2c] )\n res4b10_relu = mx.symbol.Activation(name='res4b10_relu', data=res4b10 , act_type='relu')\n res4b11_branch2a = mx.symbol.Convolution(name='res4b11_branch2a', data=res4b10_relu , num_filter=256, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn4b11_branch2a = mx.symbol.BatchNorm(name='bn4b11_branch2a', data=res4b11_branch2a , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b11_branch2a = bn4b11_branch2a\n res4b11_branch2a_relu = mx.symbol.Activation(name='res4b11_branch2a_relu', data=scale4b11_branch2a , act_type='relu')\n res4b11_branch2b = mx.symbol.Convolution(name='res4b11_branch2b', data=res4b11_branch2a_relu , num_filter=256, pad=(1,1), kernel=(3,3), stride=(1,1), no_bias=True)\n bn4b11_branch2b = mx.symbol.BatchNorm(name='bn4b11_branch2b', data=res4b11_branch2b , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b11_branch2b = bn4b11_branch2b\n res4b11_branch2b_relu = mx.symbol.Activation(name='res4b11_branch2b_relu', data=scale4b11_branch2b , act_type='relu')\n res4b11_branch2c = mx.symbol.Convolution(name='res4b11_branch2c', data=res4b11_branch2b_relu , num_filter=1024, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn4b11_branch2c = mx.symbol.BatchNorm(name='bn4b11_branch2c', data=res4b11_branch2c , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b11_branch2c = bn4b11_branch2c\n res4b11 = mx.symbol.broadcast_add(name='res4b11', *[res4b10_relu,scale4b11_branch2c] )\n res4b11_relu = mx.symbol.Activation(name='res4b11_relu', data=res4b11 , act_type='relu')\n res4b12_branch2a = mx.symbol.Convolution(name='res4b12_branch2a', data=res4b11_relu , num_filter=256, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn4b12_branch2a = mx.symbol.BatchNorm(name='bn4b12_branch2a', data=res4b12_branch2a , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b12_branch2a = bn4b12_branch2a\n res4b12_branch2a_relu = mx.symbol.Activation(name='res4b12_branch2a_relu', data=scale4b12_branch2a , act_type='relu')\n res4b12_branch2b = mx.symbol.Convolution(name='res4b12_branch2b', data=res4b12_branch2a_relu , num_filter=256, pad=(1,1), kernel=(3,3), stride=(1,1), no_bias=True)\n bn4b12_branch2b = mx.symbol.BatchNorm(name='bn4b12_branch2b', data=res4b12_branch2b , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b12_branch2b = bn4b12_branch2b\n res4b12_branch2b_relu = mx.symbol.Activation(name='res4b12_branch2b_relu', data=scale4b12_branch2b , act_type='relu')\n res4b12_branch2c = mx.symbol.Convolution(name='res4b12_branch2c', data=res4b12_branch2b_relu , num_filter=1024, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn4b12_branch2c = mx.symbol.BatchNorm(name='bn4b12_branch2c', data=res4b12_branch2c , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b12_branch2c = bn4b12_branch2c\n res4b12 = mx.symbol.broadcast_add(name='res4b12', *[res4b11_relu,scale4b12_branch2c] )\n res4b12_relu = mx.symbol.Activation(name='res4b12_relu', data=res4b12 , act_type='relu')\n res4b13_branch2a = mx.symbol.Convolution(name='res4b13_branch2a', data=res4b12_relu , num_filter=256, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn4b13_branch2a = mx.symbol.BatchNorm(name='bn4b13_branch2a', data=res4b13_branch2a , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b13_branch2a = bn4b13_branch2a\n res4b13_branch2a_relu = mx.symbol.Activation(name='res4b13_branch2a_relu', data=scale4b13_branch2a , act_type='relu')\n res4b13_branch2b = mx.symbol.Convolution(name='res4b13_branch2b', data=res4b13_branch2a_relu , num_filter=256, pad=(1,1), kernel=(3,3), stride=(1,1), no_bias=True)\n bn4b13_branch2b = mx.symbol.BatchNorm(name='bn4b13_branch2b', data=res4b13_branch2b , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b13_branch2b = bn4b13_branch2b\n res4b13_branch2b_relu = mx.symbol.Activation(name='res4b13_branch2b_relu', data=scale4b13_branch2b , act_type='relu')\n res4b13_branch2c = mx.symbol.Convolution(name='res4b13_branch2c', data=res4b13_branch2b_relu , num_filter=1024, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn4b13_branch2c = mx.symbol.BatchNorm(name='bn4b13_branch2c', data=res4b13_branch2c , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b13_branch2c = bn4b13_branch2c\n res4b13 = mx.symbol.broadcast_add(name='res4b13', *[res4b12_relu,scale4b13_branch2c] )\n res4b13_relu = mx.symbol.Activation(name='res4b13_relu', data=res4b13 , act_type='relu')\n res4b14_branch2a = mx.symbol.Convolution(name='res4b14_branch2a', data=res4b13_relu , num_filter=256, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn4b14_branch2a = mx.symbol.BatchNorm(name='bn4b14_branch2a', data=res4b14_branch2a , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b14_branch2a = bn4b14_branch2a\n res4b14_branch2a_relu = mx.symbol.Activation(name='res4b14_branch2a_relu', data=scale4b14_branch2a , act_type='relu')\n res4b14_branch2b = mx.symbol.Convolution(name='res4b14_branch2b', data=res4b14_branch2a_relu , num_filter=256, pad=(1,1), kernel=(3,3), stride=(1,1), no_bias=True)\n bn4b14_branch2b = mx.symbol.BatchNorm(name='bn4b14_branch2b', data=res4b14_branch2b , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b14_branch2b = bn4b14_branch2b\n res4b14_branch2b_relu = mx.symbol.Activation(name='res4b14_branch2b_relu', data=scale4b14_branch2b , act_type='relu')\n res4b14_branch2c = mx.symbol.Convolution(name='res4b14_branch2c', data=res4b14_branch2b_relu , num_filter=1024, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn4b14_branch2c = mx.symbol.BatchNorm(name='bn4b14_branch2c', data=res4b14_branch2c , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b14_branch2c = bn4b14_branch2c\n res4b14 = mx.symbol.broadcast_add(name='res4b14', *[res4b13_relu,scale4b14_branch2c] )\n res4b14_relu = mx.symbol.Activation(name='res4b14_relu', data=res4b14 , act_type='relu')\n res4b15_branch2a = mx.symbol.Convolution(name='res4b15_branch2a', data=res4b14_relu , num_filter=256, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn4b15_branch2a = mx.symbol.BatchNorm(name='bn4b15_branch2a', data=res4b15_branch2a , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b15_branch2a = bn4b15_branch2a\n res4b15_branch2a_relu = mx.symbol.Activation(name='res4b15_branch2a_relu', data=scale4b15_branch2a , act_type='relu')\n res4b15_branch2b = mx.symbol.Convolution(name='res4b15_branch2b', data=res4b15_branch2a_relu , num_filter=256, pad=(1,1), kernel=(3,3), stride=(1,1), no_bias=True)\n bn4b15_branch2b = mx.symbol.BatchNorm(name='bn4b15_branch2b', data=res4b15_branch2b , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b15_branch2b = bn4b15_branch2b\n res4b15_branch2b_relu = mx.symbol.Activation(name='res4b15_branch2b_relu', data=scale4b15_branch2b , act_type='relu')\n res4b15_branch2c = mx.symbol.Convolution(name='res4b15_branch2c', data=res4b15_branch2b_relu , num_filter=1024, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn4b15_branch2c = mx.symbol.BatchNorm(name='bn4b15_branch2c', data=res4b15_branch2c , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b15_branch2c = bn4b15_branch2c\n res4b15 = mx.symbol.broadcast_add(name='res4b15', *[res4b14_relu,scale4b15_branch2c] )\n res4b15_relu = mx.symbol.Activation(name='res4b15_relu', data=res4b15 , act_type='relu')\n res4b16_branch2a = mx.symbol.Convolution(name='res4b16_branch2a', data=res4b15_relu , num_filter=256, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn4b16_branch2a = mx.symbol.BatchNorm(name='bn4b16_branch2a', data=res4b16_branch2a , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b16_branch2a = bn4b16_branch2a\n res4b16_branch2a_relu = mx.symbol.Activation(name='res4b16_branch2a_relu', data=scale4b16_branch2a , act_type='relu')\n res4b16_branch2b = mx.symbol.Convolution(name='res4b16_branch2b', data=res4b16_branch2a_relu , num_filter=256, pad=(1,1), kernel=(3,3), stride=(1,1), no_bias=True)\n bn4b16_branch2b = mx.symbol.BatchNorm(name='bn4b16_branch2b', data=res4b16_branch2b , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b16_branch2b = bn4b16_branch2b\n res4b16_branch2b_relu = mx.symbol.Activation(name='res4b16_branch2b_relu', data=scale4b16_branch2b , act_type='relu')\n res4b16_branch2c = mx.symbol.Convolution(name='res4b16_branch2c', data=res4b16_branch2b_relu , num_filter=1024, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn4b16_branch2c = mx.symbol.BatchNorm(name='bn4b16_branch2c', data=res4b16_branch2c , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b16_branch2c = bn4b16_branch2c\n res4b16 = mx.symbol.broadcast_add(name='res4b16', *[res4b15_relu,scale4b16_branch2c] )\n res4b16_relu = mx.symbol.Activation(name='res4b16_relu', data=res4b16 , act_type='relu')\n res4b17_branch2a = mx.symbol.Convolution(name='res4b17_branch2a', data=res4b16_relu , num_filter=256, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn4b17_branch2a = mx.symbol.BatchNorm(name='bn4b17_branch2a', data=res4b17_branch2a , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b17_branch2a = bn4b17_branch2a\n res4b17_branch2a_relu = mx.symbol.Activation(name='res4b17_branch2a_relu', data=scale4b17_branch2a , act_type='relu')\n res4b17_branch2b = mx.symbol.Convolution(name='res4b17_branch2b', data=res4b17_branch2a_relu , num_filter=256, pad=(1,1), kernel=(3,3), stride=(1,1), no_bias=True)\n bn4b17_branch2b = mx.symbol.BatchNorm(name='bn4b17_branch2b', data=res4b17_branch2b , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b17_branch2b = bn4b17_branch2b\n res4b17_branch2b_relu = mx.symbol.Activation(name='res4b17_branch2b_relu', data=scale4b17_branch2b , act_type='relu')\n res4b17_branch2c = mx.symbol.Convolution(name='res4b17_branch2c', data=res4b17_branch2b_relu , num_filter=1024, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn4b17_branch2c = mx.symbol.BatchNorm(name='bn4b17_branch2c', data=res4b17_branch2c , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b17_branch2c = bn4b17_branch2c\n res4b17 = mx.symbol.broadcast_add(name='res4b17', *[res4b16_relu,scale4b17_branch2c] )\n res4b17_relu = mx.symbol.Activation(name='res4b17_relu', data=res4b17 , act_type='relu')\n res4b18_branch2a = mx.symbol.Convolution(name='res4b18_branch2a', data=res4b17_relu , num_filter=256, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn4b18_branch2a = mx.symbol.BatchNorm(name='bn4b18_branch2a', data=res4b18_branch2a , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b18_branch2a = bn4b18_branch2a\n res4b18_branch2a_relu = mx.symbol.Activation(name='res4b18_branch2a_relu', data=scale4b18_branch2a , act_type='relu')\n res4b18_branch2b = mx.symbol.Convolution(name='res4b18_branch2b', data=res4b18_branch2a_relu , num_filter=256, pad=(1,1), kernel=(3,3), stride=(1,1), no_bias=True)\n bn4b18_branch2b = mx.symbol.BatchNorm(name='bn4b18_branch2b', data=res4b18_branch2b , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b18_branch2b = bn4b18_branch2b\n res4b18_branch2b_relu = mx.symbol.Activation(name='res4b18_branch2b_relu', data=scale4b18_branch2b , act_type='relu')\n res4b18_branch2c = mx.symbol.Convolution(name='res4b18_branch2c', data=res4b18_branch2b_relu , num_filter=1024, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn4b18_branch2c = mx.symbol.BatchNorm(name='bn4b18_branch2c', data=res4b18_branch2c , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b18_branch2c = bn4b18_branch2c\n res4b18 = mx.symbol.broadcast_add(name='res4b18', *[res4b17_relu,scale4b18_branch2c] )\n res4b18_relu = mx.symbol.Activation(name='res4b18_relu', data=res4b18 , act_type='relu')\n res4b19_branch2a = mx.symbol.Convolution(name='res4b19_branch2a', data=res4b18_relu , num_filter=256, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn4b19_branch2a = mx.symbol.BatchNorm(name='bn4b19_branch2a', data=res4b19_branch2a , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b19_branch2a = bn4b19_branch2a\n res4b19_branch2a_relu = mx.symbol.Activation(name='res4b19_branch2a_relu', data=scale4b19_branch2a , act_type='relu')\n res4b19_branch2b = mx.symbol.Convolution(name='res4b19_branch2b', data=res4b19_branch2a_relu , num_filter=256, pad=(1,1), kernel=(3,3), stride=(1,1), no_bias=True)\n bn4b19_branch2b = mx.symbol.BatchNorm(name='bn4b19_branch2b', data=res4b19_branch2b , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b19_branch2b = bn4b19_branch2b\n res4b19_branch2b_relu = mx.symbol.Activation(name='res4b19_branch2b_relu', data=scale4b19_branch2b , act_type='relu')\n res4b19_branch2c = mx.symbol.Convolution(name='res4b19_branch2c', data=res4b19_branch2b_relu , num_filter=1024, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn4b19_branch2c = mx.symbol.BatchNorm(name='bn4b19_branch2c', data=res4b19_branch2c , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b19_branch2c = bn4b19_branch2c\n res4b19 = mx.symbol.broadcast_add(name='res4b19', *[res4b18_relu,scale4b19_branch2c] )\n res4b19_relu = mx.symbol.Activation(name='res4b19_relu', data=res4b19 , act_type='relu')\n res4b20_branch2a = mx.symbol.Convolution(name='res4b20_branch2a', data=res4b19_relu , num_filter=256, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn4b20_branch2a = mx.symbol.BatchNorm(name='bn4b20_branch2a', data=res4b20_branch2a , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b20_branch2a = bn4b20_branch2a\n res4b20_branch2a_relu = mx.symbol.Activation(name='res4b20_branch2a_relu', data=scale4b20_branch2a , act_type='relu')\n res4b20_branch2b = mx.symbol.Convolution(name='res4b20_branch2b', data=res4b20_branch2a_relu , num_filter=256, pad=(1,1), kernel=(3,3), stride=(1,1), no_bias=True)\n bn4b20_branch2b = mx.symbol.BatchNorm(name='bn4b20_branch2b', data=res4b20_branch2b , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b20_branch2b = bn4b20_branch2b\n res4b20_branch2b_relu = mx.symbol.Activation(name='res4b20_branch2b_relu', data=scale4b20_branch2b , act_type='relu')\n res4b20_branch2c = mx.symbol.Convolution(name='res4b20_branch2c', data=res4b20_branch2b_relu , num_filter=1024, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn4b20_branch2c = mx.symbol.BatchNorm(name='bn4b20_branch2c', data=res4b20_branch2c , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b20_branch2c = bn4b20_branch2c\n res4b20 = mx.symbol.broadcast_add(name='res4b20', *[res4b19_relu,scale4b20_branch2c] )\n res4b20_relu = mx.symbol.Activation(name='res4b20_relu', data=res4b20 , act_type='relu')\n res4b21_branch2a = mx.symbol.Convolution(name='res4b21_branch2a', data=res4b20_relu , num_filter=256, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn4b21_branch2a = mx.symbol.BatchNorm(name='bn4b21_branch2a', data=res4b21_branch2a , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b21_branch2a = bn4b21_branch2a\n res4b21_branch2a_relu = mx.symbol.Activation(name='res4b21_branch2a_relu', data=scale4b21_branch2a , act_type='relu')\n res4b21_branch2b = mx.symbol.Convolution(name='res4b21_branch2b', data=res4b21_branch2a_relu , num_filter=256, pad=(1,1), kernel=(3,3), stride=(1,1), no_bias=True)\n bn4b21_branch2b = mx.symbol.BatchNorm(name='bn4b21_branch2b', data=res4b21_branch2b , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b21_branch2b = bn4b21_branch2b\n res4b21_branch2b_relu = mx.symbol.Activation(name='res4b21_branch2b_relu', data=scale4b21_branch2b , act_type='relu')\n res4b21_branch2c = mx.symbol.Convolution(name='res4b21_branch2c', data=res4b21_branch2b_relu , num_filter=1024, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn4b21_branch2c = mx.symbol.BatchNorm(name='bn4b21_branch2c', data=res4b21_branch2c , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b21_branch2c = bn4b21_branch2c\n res4b21 = mx.symbol.broadcast_add(name='res4b21', *[res4b20_relu,scale4b21_branch2c] )\n res4b21_relu = mx.symbol.Activation(name='res4b21_relu', data=res4b21 , act_type='relu')\n res4b22_branch2a = mx.symbol.Convolution(name='res4b22_branch2a', data=res4b21_relu , num_filter=256, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn4b22_branch2a = mx.symbol.BatchNorm(name='bn4b22_branch2a', data=res4b22_branch2a , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b22_branch2a = bn4b22_branch2a\n res4b22_branch2a_relu = mx.symbol.Activation(name='res4b22_branch2a_relu', data=scale4b22_branch2a , act_type='relu')\n res4b22_branch2b = mx.symbol.Convolution(name='res4b22_branch2b', data=res4b22_branch2a_relu , num_filter=256, pad=(1,1), kernel=(3,3), stride=(1,1), no_bias=True)\n bn4b22_branch2b = mx.symbol.BatchNorm(name='bn4b22_branch2b', data=res4b22_branch2b , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b22_branch2b = bn4b22_branch2b\n res4b22_branch2b_relu = mx.symbol.Activation(name='res4b22_branch2b_relu', data=scale4b22_branch2b , act_type='relu')\n res4b22_branch2c = mx.symbol.Convolution(name='res4b22_branch2c', data=res4b22_branch2b_relu , num_filter=1024, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn4b22_branch2c = mx.symbol.BatchNorm(name='bn4b22_branch2c', data=res4b22_branch2c , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale4b22_branch2c = bn4b22_branch2c\n res4b22 = mx.symbol.broadcast_add(name='res4b22', *[res4b21_relu,scale4b22_branch2c] )\n res4b22_relu = mx.symbol.Activation(name='res4b22_relu', data=res4b22 , act_type='relu')\n res5a_branch1 = mx.symbol.Convolution(name='res5a_branch1', data=res4b22_relu , num_filter=2048, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn5a_branch1 = mx.symbol.BatchNorm(name='bn5a_branch1', data=res5a_branch1 , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale5a_branch1 = bn5a_branch1\n res5a_branch2a = mx.symbol.Convolution(name='res5a_branch2a', data=res4b22_relu , num_filter=512, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn5a_branch2a = mx.symbol.BatchNorm(name='bn5a_branch2a', data=res5a_branch2a , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale5a_branch2a = bn5a_branch2a\n res5a_branch2a_relu = mx.symbol.Activation(name='res5a_branch2a_relu', data=scale5a_branch2a , act_type='relu')\n res5a_branch2b = mx.symbol.Convolution(name='res5a_branch2b', data=res5a_branch2a_relu , num_filter=512, pad=(2,2), dilate=(2,2), kernel=(3,3), stride=(1,1), no_bias=True)\n bn5a_branch2b = mx.symbol.BatchNorm(name='bn5a_branch2b', data=res5a_branch2b , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale5a_branch2b = bn5a_branch2b\n res5a_branch2b_relu = mx.symbol.Activation(name='res5a_branch2b_relu', data=scale5a_branch2b , act_type='relu')\n res5a_branch2c = mx.symbol.Convolution(name='res5a_branch2c', data=res5a_branch2b_relu , num_filter=2048, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn5a_branch2c = mx.symbol.BatchNorm(name='bn5a_branch2c', data=res5a_branch2c , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale5a_branch2c = bn5a_branch2c\n res5a = mx.symbol.broadcast_add(name='res5a', *[scale5a_branch1,scale5a_branch2c] )\n res5a_relu = mx.symbol.Activation(name='res5a_relu', data=res5a , act_type='relu')\n res5b_branch2a = mx.symbol.Convolution(name='res5b_branch2a', data=res5a_relu , num_filter=512, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn5b_branch2a = mx.symbol.BatchNorm(name='bn5b_branch2a', data=res5b_branch2a , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale5b_branch2a = bn5b_branch2a\n res5b_branch2a_relu = mx.symbol.Activation(name='res5b_branch2a_relu', data=scale5b_branch2a , act_type='relu')\n res5b_branch2b = mx.symbol.Convolution(name='res5b_branch2b', data=res5b_branch2a_relu , num_filter=512, pad=(2,2), dilate=(2,2), kernel=(3,3), stride=(1,1), no_bias=True)\n bn5b_branch2b = mx.symbol.BatchNorm(name='bn5b_branch2b', data=res5b_branch2b , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale5b_branch2b = bn5b_branch2b\n res5b_branch2b_relu = mx.symbol.Activation(name='res5b_branch2b_relu', data=scale5b_branch2b , act_type='relu')\n res5b_branch2c = mx.symbol.Convolution(name='res5b_branch2c', data=res5b_branch2b_relu , num_filter=2048, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn5b_branch2c = mx.symbol.BatchNorm(name='bn5b_branch2c', data=res5b_branch2c , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale5b_branch2c = bn5b_branch2c\n res5b = mx.symbol.broadcast_add(name='res5b', *[res5a_relu,scale5b_branch2c] )\n res5b_relu = mx.symbol.Activation(name='res5b_relu', data=res5b , act_type='relu')\n res5c_branch2a = mx.symbol.Convolution(name='res5c_branch2a', data=res5b_relu , num_filter=512, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn5c_branch2a = mx.symbol.BatchNorm(name='bn5c_branch2a', data=res5c_branch2a , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale5c_branch2a = bn5c_branch2a\n res5c_branch2a_relu = mx.symbol.Activation(name='res5c_branch2a_relu', data=scale5c_branch2a , act_type='relu')\n res5c_branch2b = mx.symbol.Convolution(name='res5c_branch2b', data=res5c_branch2a_relu , num_filter=512, pad=(2,2), dilate=(2,2), kernel=(3,3), stride=(1,1), no_bias=True)\n bn5c_branch2b = mx.symbol.BatchNorm(name='bn5c_branch2b', data=res5c_branch2b , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale5c_branch2b = bn5c_branch2b\n res5c_branch2b_relu = mx.symbol.Activation(name='res5c_branch2b_relu', data=scale5c_branch2b , act_type='relu')\n res5c_branch2c = mx.symbol.Convolution(name='res5c_branch2c', data=res5c_branch2b_relu , num_filter=2048, pad=(0,0), kernel=(1,1), stride=(1,1), no_bias=True)\n bn5c_branch2c = mx.symbol.BatchNorm(name='bn5c_branch2c', data=res5c_branch2c , use_global_stats=self.use_global_stats, eps=self.eps, fix_gamma=False)\n scale5c_branch2c = bn5c_branch2c\n res5c = mx.symbol.broadcast_add(name='res5c', *[res5b_relu,scale5c_branch2c] )\n res5c_relu = mx.symbol.Activation(name='res5c_relu', data=res5c , act_type='relu')\n\n feat_conv_3x3 = mx.sym.Convolution(\n data=res5c_relu, kernel=(3, 3), pad=(6, 6), dilate=(6, 6), num_filter=1024, name=\"feat_conv_3x3\")\n feat_conv_3x3_relu = mx.sym.Activation(data=feat_conv_3x3, act_type=\"relu\", name=\"feat_conv_3x3_relu\")\n return feat_conv_3x3_relu\n \n def get_train_symbol(self, cfg):\n\n # config alias for convenient\n num_classes = cfg.dataset.NUM_CLASSES\n num_reg_classes = (2 if cfg.CLASS_AGNOSTIC else num_classes)\n num_anchors = cfg.network.NUM_ANCHORS\n\n data = mx.sym.Variable(name=\"data\")\n im_info = mx.sym.Variable(name=\"im_info\")\n gt_boxes = mx.sym.Variable(name=\"gt_boxes\")\n rpn_label = mx.sym.Variable(name='label')\n rpn_bbox_target = mx.sym.Variable(name='bbox_target')\n rpn_bbox_weight = mx.sym.Variable(name='bbox_weight')\n \n # shared convolutional layers\n conv_feat = self.get_resnet_v1(data)\n conv_feats = mx.sym.SliceChannel(conv_feat, axis=1, num_outputs=2)\n \n # RPN layers\n rpn_feat = conv_feats[0]\n rpn_cls_score = mx.sym.Convolution(\n data=rpn_feat, kernel=(1, 1), pad=(0, 0), num_filter=2 * num_anchors, name=\"rpn_cls_score\")\n rpn_bbox_pred = mx.sym.Convolution(\n data=rpn_feat, kernel=(1, 1), pad=(0, 0), num_filter=4 * num_anchors, name=\"rpn_bbox_pred\")\n \n # prepare rpn data\n rpn_cls_score_reshape = mx.sym.Reshape(\n data=rpn_cls_score, shape=(0, 2, -1, 0), name=\"rpn_cls_score_reshape\")\n \n # classification\n rpn_cls_prob = mx.sym.SoftmaxOutput(data=rpn_cls_score_reshape, label=rpn_label, multi_output=True,\n normalization='valid', use_ignore=True, ignore_label=-1, name=\"rpn_cls_prob\")\n # bounding box regression\n if cfg.network.NORMALIZE_RPN:\n rpn_bbox_loss_ = rpn_bbox_weight * mx.sym.smooth_l1(name='rpn_bbox_loss_', scalar=1.0, data=(rpn_bbox_pred - rpn_bbox_target))\n rpn_bbox_pred = mx.sym.Custom(\n bbox_pred=rpn_bbox_pred, op_type='rpn_inv_normalize', num_anchors=num_anchors,\n bbox_mean=cfg.network.ANCHOR_MEANS, bbox_std=cfg.network.ANCHOR_STDS)\n else:\n rpn_bbox_loss_ = rpn_bbox_weight * mx.sym.smooth_l1(name='rpn_bbox_loss_', scalar=3.0, data=(rpn_bbox_pred - rpn_bbox_target))\n rpn_bbox_loss = mx.sym.MakeLoss(name='rpn_bbox_loss', data=rpn_bbox_loss_, grad_scale=1.0 / cfg.TRAIN.RPN_BATCH_SIZE)\n \n # ROI proposal\n rpn_cls_act = mx.sym.SoftmaxActivation(\n data=rpn_cls_score_reshape, mode=\"channel\", name=\"rpn_cls_act\")\n rpn_cls_act_reshape = mx.sym.Reshape(\n data=rpn_cls_act, shape=(0, 2 * num_anchors, -1, 0), name='rpn_cls_act_reshape')\n\n if cfg.TRAIN.CXX_PROPOSAL:\n rois = mx.contrib.sym.Proposal(\n cls_prob=rpn_cls_act_reshape, bbox_pred=rpn_bbox_pred, im_info=im_info, name='rois',\n feature_stride=cfg.network.RPN_FEAT_STRIDE, scales=tuple(cfg.network.ANCHOR_SCALES), ratios=tuple(cfg.network.ANCHOR_RATIOS),\n rpn_pre_nms_top_n=cfg.TRAIN.RPN_PRE_NMS_TOP_N, rpn_post_nms_top_n=cfg.TRAIN.RPN_POST_NMS_TOP_N,\n threshold=cfg.TRAIN.RPN_NMS_THRESH, rpn_min_size=cfg.TRAIN.RPN_MIN_SIZE)\n else:\n rois = mx.sym.Custom(\n cls_prob=rpn_cls_act_reshape, bbox_pred=rpn_bbox_pred, im_info=im_info, name='rois',\n op_type='proposal', feat_stride=cfg.network.RPN_FEAT_STRIDE,\n scales=tuple(cfg.network.ANCHOR_SCALES), ratios=tuple(cfg.network.ANCHOR_RATIOS),\n rpn_pre_nms_top_n=cfg.TRAIN.RPN_PRE_NMS_TOP_N, rpn_post_nms_top_n=cfg.TRAIN.RPN_POST_NMS_TOP_N,\n threshold=cfg.TRAIN.RPN_NMS_THRESH, rpn_min_size=cfg.TRAIN.RPN_MIN_SIZE)\n \n # ROI proposal target\n gt_boxes_reshape = mx.sym.Reshape(data=gt_boxes, shape=(-1, 5), name='gt_boxes_reshape')\n rois, label, bbox_target, bbox_weight = mx.sym.Custom(rois=rois, gt_boxes=gt_boxes_reshape,\n op_type='proposal_target',\n num_classes=num_reg_classes,\n batch_images=cfg.TRAIN.BATCH_IMAGES,\n batch_rois=cfg.TRAIN.BATCH_ROIS,\n cfg=cPickle.dumps(cfg),\n fg_fraction=cfg.TRAIN.FG_FRACTION)\n \n # res5\n rfcn_feat = conv_feats[1]\n rfcn_cls = mx.sym.Convolution(data=rfcn_feat, kernel=(1, 1), num_filter=7*7*num_classes, name=\"rfcn_cls\")\n rfcn_bbox = mx.sym.Convolution(data=rfcn_feat, kernel=(1, 1), num_filter=7*7*4*num_reg_classes, name=\"rfcn_bbox\")\n psroipooled_cls_rois = mx.contrib.sym.PSROIPooling(name='psroipooled_cls_rois', data=rfcn_cls, rois=rois, group_size=7,pooled_size=7,\n output_dim=num_classes, spatial_scale=0.0625)\n psroipooled_loc_rois = mx.contrib.sym.PSROIPooling(name='psroipooled_loc_rois', data=rfcn_bbox, rois=rois, group_size=7,pooled_size=7,\n output_dim=8, spatial_scale=0.0625)\n cls_score = mx.sym.Pooling(name='ave_cls_scors_rois', data=psroipooled_cls_rois, pool_type='avg', global_pool=True, kernel=(7, 7))\n bbox_pred = mx.sym.Pooling(name='ave_bbox_pred_rois', data=psroipooled_loc_rois, pool_type='avg', global_pool=True, kernel=(7, 7))\n cls_score = mx.sym.Reshape(name='cls_score_reshape', data=cls_score, shape=(-1, num_classes))\n bbox_pred = mx.sym.Reshape(name='bbox_pred_reshape', data=bbox_pred, shape=(-1, 4 * num_reg_classes))\n \n \n # classification\n if cfg.TRAIN.ENABLE_OHEM:\n print 'use ohem!'\n labels_ohem, bbox_weights_ohem = mx.sym.Custom(op_type='BoxAnnotatorOHEM', num_classes=num_classes,\n num_reg_classes=num_reg_classes, roi_per_img=cfg.TRAIN.BATCH_ROIS_OHEM,\n cls_score=cls_score, bbox_pred=bbox_pred, labels=label,\n bbox_targets=bbox_target, bbox_weights=bbox_weight)\n cls_prob = mx.sym.SoftmaxOutput(name='cls_prob', data=cls_score, label=labels_ohem, normalization='valid', use_ignore=True, ignore_label=-1)\n bbox_loss_ = bbox_weights_ohem * mx.sym.smooth_l1(name='bbox_loss_', scalar=1.0, data=(bbox_pred - bbox_target))\n bbox_loss = mx.sym.MakeLoss(name='bbox_loss', data=bbox_loss_, grad_scale=1.0 / cfg.TRAIN.BATCH_ROIS_OHEM)\n rcnn_label = labels_ohem\n else:\n cls_prob = mx.sym.SoftmaxOutput(name='cls_prob', data=cls_score, label=label, normalization='valid')\n bbox_loss_ = bbox_weight * mx.sym.smooth_l1(name='bbox_loss_', scalar=1.0, data=(bbox_pred - bbox_target))\n bbox_loss = mx.sym.MakeLoss(name='bbox_loss', data=bbox_loss_, grad_scale=1.0 / cfg.TRAIN.BATCH_ROIS)\n rcnn_label = label\n \n # reshape output\n rcnn_label = mx.sym.Reshape(data=rcnn_label, shape=(cfg.TRAIN.BATCH_IMAGES, -1), name='label_reshape')\n cls_prob = mx.sym.Reshape(data=cls_prob, shape=(cfg.TRAIN.BATCH_IMAGES, -1, num_classes), name='cls_prob_reshape')\n bbox_loss = mx.sym.Reshape(data=bbox_loss, shape=(cfg.TRAIN.BATCH_IMAGES, -1, 4 * num_reg_classes), name='bbox_loss_reshape')\n \n group = mx.sym.Group([rpn_cls_prob, rpn_bbox_loss, cls_prob, bbox_loss, mx.sym.BlockGrad(rcnn_label)])\n self.sym = group\n return group\n\n def get_test_symbol(self, cfg):\n \n # config alias for convenient\n num_classes = cfg.dataset.NUM_CLASSES\n num_reg_classes = (2 if cfg.CLASS_AGNOSTIC else num_classes)\n num_anchors = cfg.network.NUM_ANCHORS\n\n data = mx.sym.Variable(name=\"data\")\n im_info = mx.sym.Variable(name=\"im_info\")\n \n # shared convolutional layers\n conv_feat = self.get_resnet_v1(data)\n conv_feats = mx.sym.SliceChannel(conv_feat, axis=1, num_outputs=2)\n \n # RPN\n rpn_feat = conv_feats[0]\n rpn_cls_score = mx.sym.Convolution(\n data=rpn_feat, kernel=(1, 1), pad=(0, 0), num_filter=2 * num_anchors, name=\"rpn_cls_score\")\n rpn_bbox_pred = mx.sym.Convolution(\n data=rpn_feat, kernel=(1, 1), pad=(0, 0), num_filter=4 * num_anchors, name=\"rpn_bbox_pred\")\n \n if cfg.network.NORMALIZE_RPN:\n rpn_bbox_pred = mx.sym.Custom(\n bbox_pred=rpn_bbox_pred, op_type='rpn_inv_normalize', num_anchors=num_anchors,\n bbox_mean=cfg.network.ANCHOR_MEANS, bbox_std=cfg.network.ANCHOR_STDS)\n \n # ROI Proposal\n rpn_cls_score_reshape = mx.sym.Reshape(\n data=rpn_cls_score, shape=(0, 2, -1, 0), name=\"rpn_cls_score_reshape\")\n rpn_cls_prob = mx.sym.SoftmaxActivation(\n data=rpn_cls_score_reshape, mode=\"channel\", name=\"rpn_cls_prob\")\n rpn_cls_prob_reshape = mx.sym.Reshape(\n data=rpn_cls_prob, shape=(0, 2 * num_anchors, -1, 0), name='rpn_cls_prob_reshape')\n if cfg.TEST.CXX_PROPOSAL:\n rois = mx.contrib.sym.MultiProposal(\n cls_prob=rpn_cls_prob_reshape, bbox_pred=rpn_bbox_pred, im_info=im_info, name='rois',\n feature_stride=cfg.network.RPN_FEAT_STRIDE, scales=tuple(cfg.network.ANCHOR_SCALES),\n ratios=tuple(cfg.network.ANCHOR_RATIOS),\n rpn_pre_nms_top_n=cfg.TEST.RPN_PRE_NMS_TOP_N, rpn_post_nms_top_n=cfg.TEST.RPN_POST_NMS_TOP_N,\n threshold=cfg.TEST.RPN_NMS_THRESH, rpn_min_size=cfg.TEST.RPN_MIN_SIZE)\n else:\n NotImplemented\n \n # res5\n rfcn_feat = conv_feats[1]\n rfcn_cls = mx.sym.Convolution(data=rfcn_feat, kernel=(1, 1), num_filter=7*7*num_classes, name=\"rfcn_cls\")\n rfcn_bbox = mx.sym.Convolution(data=rfcn_feat, kernel=(1, 1), num_filter=7*7*4*num_reg_classes, name=\"rfcn_bbox\")\n psroipooled_cls_rois = mx.contrib.sym.PSROIPooling(name='psroipooled_cls_rois', data=rfcn_cls, rois=rois, group_size=7, pooled_size=7,\n output_dim=num_classes, spatial_scale=0.0625)\n psroipooled_loc_rois = mx.contrib.sym.PSROIPooling(name='psroipooled_loc_rois', data=rfcn_bbox, rois=rois, group_size=7, pooled_size=7,\n output_dim=8, spatial_scale=0.0625)\n cls_score = mx.sym.Pooling(name='ave_cls_scors_rois', data=psroipooled_cls_rois, pool_type='avg', global_pool=True, kernel=(7, 7))\n bbox_pred = mx.sym.Pooling(name='ave_bbox_pred_rois', data=psroipooled_loc_rois, pool_type='avg', global_pool=True, kernel=(7, 7))\n \n # classification\n cls_score = mx.sym.Reshape(name='cls_score_reshape', data=cls_score, shape=(-1, num_classes))\n cls_prob = mx.sym.SoftmaxActivation(name='cls_prob', data=cls_score)\n # bounding box regression\n bbox_pred = mx.sym.Reshape(name='bbox_pred_reshape', data=bbox_pred, shape=(-1, 4 * num_reg_classes))\n \n # reshape output\n cls_prob = mx.sym.Reshape(data=cls_prob, shape=(cfg.TEST.BATCH_IMAGES, -1, num_classes), name='cls_prob_reshape')\n bbox_pred = mx.sym.Reshape(data=bbox_pred, shape=(cfg.TEST.BATCH_IMAGES, -1, 4 * num_reg_classes), name='bbox_pred_reshape')\n \n # group output\n group = mx.sym.Group([rois, cls_prob, bbox_pred])\n self.sym = group\n return group\n\n def init_weight(self, cfg, arg_params, aux_params):\n arg_params['feat_conv_3x3_weight'] = mx.random.normal(0, 0.01, shape=self.arg_shape_dict['feat_conv_3x3_weight'])\n arg_params['feat_conv_3x3_bias'] = mx.nd.zeros(shape=self.arg_shape_dict['feat_conv_3x3_bias'])\n\n arg_params['rpn_cls_score_weight'] = mx.random.normal(0, 0.01, shape=self.arg_shape_dict['rpn_cls_score_weight'])\n arg_params['rpn_cls_score_bias'] = mx.nd.zeros(shape=self.arg_shape_dict['rpn_cls_score_bias'])\n arg_params['rpn_bbox_pred_weight'] = mx.random.normal(0, 0.01, shape=self.arg_shape_dict['rpn_bbox_pred_weight'])\n arg_params['rpn_bbox_pred_bias'] = mx.nd.zeros(shape=self.arg_shape_dict['rpn_bbox_pred_bias'])\n\n arg_params['rfcn_cls_weight'] = mx.random.normal(0, 0.01, shape=self.arg_shape_dict['rfcn_cls_weight'])\n arg_params['rfcn_cls_bias'] = mx.nd.zeros(shape=self.arg_shape_dict['rfcn_cls_bias'])\n arg_params['rfcn_bbox_weight'] = mx.random.normal(0, 0.01, shape=self.arg_shape_dict['rfcn_bbox_weight'])\n arg_params['rfcn_bbox_bias'] = mx.nd.zeros(shape=self.arg_shape_dict['rfcn_bbox_bias'])\n" }, { "path": "rfcn/test.py", "content": "# --------------------------------------------------------\n# Deep Feature Flow\n# Copyright (c) 2017 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Modified by Yuwen Xiong\n# --------------------------------------------------------\n# Based on:\n# MX-RCNN\n# Copyright (c) 2016 by Contributors\n# Licence under The Apache 2.0 License\n# https://github.com/ijkguo/mx-rcnn/\n# --------------------------------------------------------\n\nimport _init_paths\n\nimport cv2\nimport argparse\nimport os\nimport sys\nimport time\nimport logging\nfrom config.config import config, update_config\n\ndef parse_args():\n parser = argparse.ArgumentParser(description='Test a R-FCN network')\n # general\n parser.add_argument('--cfg', help='experiment configure file name', required=True, type=str)\n\n args, rest = parser.parse_known_args()\n update_config(args.cfg)\n\n # rcnn\n parser.add_argument('--vis', help='turn on visualization', action='store_true')\n parser.add_argument('--ignore_cache', help='ignore cached results boxes', action='store_true')\n parser.add_argument('--thresh', help='valid detection threshold', default=1e-3, type=float)\n parser.add_argument('--shuffle', help='shuffle data on visualization', action='store_true')\n args = parser.parse_args()\n return args\n\nargs = parse_args()\ncurr_path = os.path.abspath(os.path.dirname(__file__))\nsys.path.insert(0, os.path.join(curr_path, '../external/mxnet', config.MXNET_VERSION))\n\nimport mxnet as mx\nfrom function.test_rcnn import test_rcnn\nfrom utils.create_logger import create_logger\n\n\ndef main():\n ctx = [mx.gpu(int(i)) for i in config.gpus.split(',')]\n print args\n\n logger, final_output_path = create_logger(config.output_path, args.cfg, config.dataset.test_image_set)\n\n test_rcnn(config, config.dataset.dataset, config.dataset.test_image_set, config.dataset.root_path, config.dataset.dataset_path,\n ctx, os.path.join(final_output_path, '..', '_'.join([iset for iset in config.dataset.image_set.split('+')]), config.TRAIN.model_prefix), config.TEST.test_epoch,\n args.vis, args.ignore_cache, args.shuffle, config.TEST.HAS_RPN, config.dataset.proposal, args.thresh, logger=logger, output_path=final_output_path)\n\nif __name__ == '__main__':\n main()\n" }, { "path": "rfcn/train_end2end.py", "content": "# --------------------------------------------------------\n# Deep Feature Flow\n# Copyright (c) 2017 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Modified by Yuwen Xiong\n# --------------------------------------------------------\n# Based on:\n# MX-RCNN\n# Copyright (c) 2016 by Contributors\n# Licence under The Apache 2.0 License\n# https://github.com/ijkguo/mx-rcnn/\n# --------------------------------------------------------\n\nimport _init_paths\n\nimport cv2\nimport time\nimport argparse\nimport logging\nimport pprint\nimport os\nimport sys\nfrom config.config import config, update_config\n\ndef parse_args():\n parser = argparse.ArgumentParser(description='Train R-FCN network')\n # general\n parser.add_argument('--cfg', help='experiment configure file name', required=True, type=str)\n\n args, rest = parser.parse_known_args()\n # update config\n update_config(args.cfg)\n\n # training\n parser.add_argument('--frequent', help='frequency of logging', default=config.default.frequent, type=int)\n args = parser.parse_args()\n return args\n\nargs = parse_args()\ncurr_path = os.path.abspath(os.path.dirname(__file__))\nsys.path.insert(0, os.path.join(curr_path, '../external/mxnet', config.MXNET_VERSION))\n\nimport shutil\nimport numpy as np\nimport mxnet as mx\n\nfrom symbols import *\nfrom core import callback, metric\nfrom core.loader import AnchorLoader\nfrom core.module import MutableModule\nfrom utils.create_logger import create_logger\nfrom utils.load_data import load_gt_roidb, merge_roidb, filter_roidb\nfrom utils.load_model import load_param\nfrom utils.PrefetchingIter import PrefetchingIter\nfrom utils.lr_scheduler import WarmupMultiFactorScheduler\n\n\ndef train_net(args, ctx, pretrained, epoch, prefix, begin_epoch, end_epoch, lr, lr_step):\n logger, final_output_path = create_logger(config.output_path, args.cfg, config.dataset.image_set)\n prefix = os.path.join(final_output_path, prefix)\n\n # load symbol\n shutil.copy2(os.path.join(curr_path, 'symbols', config.symbol + '.py'), final_output_path)\n sym_instance = eval(config.symbol + '.' + config.symbol)()\n sym = sym_instance.get_train_symbol(config)\n feat_sym = sym.get_internals()['rpn_cls_score_output']\n\n # setup multi-gpu\n batch_size = len(ctx)\n input_batch_size = config.TRAIN.BATCH_IMAGES * batch_size\n\n # print config\n pprint.pprint(config)\n logger.info('training config:{}\\n'.format(pprint.pformat(config)))\n\n # load dataset and prepare imdb for training\n image_sets = [iset for iset in config.dataset.image_set.split('+')]\n roidbs = [load_gt_roidb(config.dataset.dataset, image_set, config.dataset.root_path, config.dataset.dataset_path,\n flip=config.TRAIN.FLIP)\n for image_set in image_sets]\n roidb = merge_roidb(roidbs)\n roidb = filter_roidb(roidb, config)\n # load training data\n train_data = AnchorLoader(feat_sym, roidb, config, batch_size=input_batch_size, shuffle=config.TRAIN.SHUFFLE, ctx=ctx,\n feat_stride=config.network.RPN_FEAT_STRIDE, anchor_scales=config.network.ANCHOR_SCALES,\n anchor_ratios=config.network.ANCHOR_RATIOS, aspect_grouping=config.TRAIN.ASPECT_GROUPING,\n normalize_target=config.network.NORMALIZE_RPN, bbox_mean=config.network.ANCHOR_MEANS,\n bbox_std=config.network.ANCHOR_STDS)\n\n # infer max shape\n max_data_shape = [('data', (config.TRAIN.BATCH_IMAGES, 3, max([v[0] for v in config.SCALES]), max([v[1] for v in config.SCALES])))]\n max_data_shape, max_label_shape = train_data.infer_shape(max_data_shape)\n max_data_shape.append(('gt_boxes', (config.TRAIN.BATCH_IMAGES, 100, 5)))\n print('providing maximum shape', max_data_shape, max_label_shape)\n\n data_shape_dict = dict(train_data.provide_data_single + train_data.provide_label_single)\n pprint.pprint(data_shape_dict)\n sym_instance.infer_shape(data_shape_dict)\n\n # load and initialize params\n if config.TRAIN.RESUME:\n print('continue training from ', begin_epoch)\n arg_params, aux_params = load_param(prefix, begin_epoch, convert=True)\n else:\n arg_params, aux_params = load_param(pretrained, epoch, convert=True)\n sym_instance.init_weight(config, arg_params, aux_params)\n\n # check parameter shapes\n sym_instance.check_parameter_shapes(arg_params, aux_params, data_shape_dict)\n\n # create solver\n fixed_param_prefix = config.network.FIXED_PARAMS\n data_names = [k[0] for k in train_data.provide_data_single]\n label_names = [k[0] for k in train_data.provide_label_single]\n\n mod = MutableModule(sym, data_names=data_names, label_names=label_names,\n logger=logger, context=ctx, max_data_shapes=[max_data_shape for _ in range(batch_size)],\n max_label_shapes=[max_label_shape for _ in range(batch_size)], fixed_param_prefix=fixed_param_prefix)\n\n if config.TRAIN.RESUME:\n mod._preload_opt_states = '%s-%04d.states'%(prefix, begin_epoch)\n\n # decide training params\n # metric\n rpn_eval_metric = metric.RPNAccMetric()\n rpn_cls_metric = metric.RPNLogLossMetric()\n rpn_bbox_metric = metric.RPNL1LossMetric()\n eval_metric = metric.RCNNAccMetric(config)\n cls_metric = metric.RCNNLogLossMetric(config)\n bbox_metric = metric.RCNNL1LossMetric(config)\n eval_metrics = mx.metric.CompositeEvalMetric()\n # rpn_eval_metric, rpn_cls_metric, rpn_bbox_metric, eval_metric, cls_metric, bbox_metric\n for child_metric in [rpn_eval_metric, rpn_cls_metric, rpn_bbox_metric, eval_metric, cls_metric, bbox_metric]:\n eval_metrics.add(child_metric)\n # callback\n batch_end_callback = callback.Speedometer(train_data.batch_size, frequent=args.frequent)\n means = np.tile(np.array(config.TRAIN.BBOX_MEANS), 2 if config.CLASS_AGNOSTIC else config.dataset.NUM_CLASSES)\n stds = np.tile(np.array(config.TRAIN.BBOX_STDS), 2 if config.CLASS_AGNOSTIC else config.dataset.NUM_CLASSES)\n epoch_end_callback = [mx.callback.module_checkpoint(mod, prefix, period=1, save_optimizer_states=True), callback.do_checkpoint(prefix, means, stds)]\n # decide learning rate\n base_lr = lr\n lr_factor = config.TRAIN.lr_factor\n lr_epoch = [float(epoch) for epoch in lr_step.split(',')]\n lr_epoch_diff = [epoch - begin_epoch for epoch in lr_epoch if epoch > begin_epoch]\n lr = base_lr * (lr_factor ** (len(lr_epoch) - len(lr_epoch_diff)))\n lr_iters = [int(epoch * len(roidb) / batch_size) for epoch in lr_epoch_diff]\n print('lr', lr, 'lr_epoch_diff', lr_epoch_diff, 'lr_iters', lr_iters)\n lr_scheduler = WarmupMultiFactorScheduler(lr_iters, lr_factor, config.TRAIN.warmup, config.TRAIN.warmup_lr, config.TRAIN.warmup_step)\n # optimizer\n optimizer_params = {'momentum': config.TRAIN.momentum,\n 'wd': config.TRAIN.wd,\n 'learning_rate': lr,\n 'lr_scheduler': lr_scheduler,\n 'rescale_grad': 1.0,\n 'clip_gradient': None}\n\n if not isinstance(train_data, PrefetchingIter):\n train_data = PrefetchingIter(train_data)\n\n # train\n mod.fit(train_data, eval_metric=eval_metrics, epoch_end_callback=epoch_end_callback,\n batch_end_callback=batch_end_callback, kvstore=config.default.kvstore,\n optimizer='sgd', optimizer_params=optimizer_params,\n arg_params=arg_params, aux_params=aux_params, begin_epoch=begin_epoch, num_epoch=end_epoch)\n\n\ndef main():\n print('Called with argument:', args)\n ctx = [mx.gpu(int(i)) for i in config.gpus.split(',')]\n train_net(args, ctx, config.network.pretrained, config.network.pretrained_epoch, config.TRAIN.model_prefix,\n config.TRAIN.begin_epoch, config.TRAIN.end_epoch, config.TRAIN.lr, config.TRAIN.lr_step)\n\nif __name__ == '__main__':\n main()\n" }, { "path": "zero.md", "content": "0" } ]