[
{
"path": ".gitignore",
"content": "data/*\n\n# READ THIS BEFORE YOU REFACTOR ME\n#\n# setup.py uses the list of patterns in this file to decide\n# what to delete, but it's not 100% sound. So, for example,\n# if you delete aten/build/ because it's redundant with build/,\n# aten/build/ will stop being cleaned. So be careful when\n# refactoring this file!\n\n## PyTorch\n\n.mypy_cache\n*.pyc\n*/*.pyc\n*/*.so*\n*/**/__pycache__\n*/**/*.dylib*\n*/**/*.pyc\n*/**/*.pyd\n*/**/*.so*\n*/**/**/*.pyc\n*/**/**/**/*.pyc\n*/**/**/**/**/*.pyc\naten/build/\naten/src/ATen/Config.h\naten/src/ATen/cuda/CUDAConfig.h\nbuild/\ndist/\ndocs/src/**/*\ntest/.coverage\ntest/cpp/api/mnist\ntest/data/gpu_tensors.pt\ntest/data/legacy_modules.t7\ntest/data/legacy_serialized.pt\ntest/data/linear.pt\ntest/htmlcov\nthird_party/build/\ntools/shared/_utils_internal.py\ntorch.egg-info/\ntorch/csrc/autograd/generated/*\ntorch/csrc/cudnn/cuDNN.cpp\ntorch/csrc/generated\ntorch/csrc/generic/TensorMethods.cpp\ntorch/csrc/jit/generated/*\ntorch/csrc/nn/THCUNN.cpp\ntorch/csrc/nn/THCUNN.cwrap\ntorch/csrc/nn/THNN_generic.cpp\ntorch/csrc/nn/THNN_generic.cwrap\ntorch/csrc/nn/THNN_generic.h\ntorch/csrc/nn/THNN.cpp\ntorch/csrc/nn/THNN.cwrap\ntorch/lib/*.a*\ntorch/lib/*.dll*\ntorch/lib/*.dylib*\ntorch/lib/*.h\ntorch/lib/*.lib\ntorch/lib/*.so*\ntorch/lib/build\ntorch/lib/cmake\ntorch/lib/include\ntorch/lib/pkgconfig\ntorch/lib/protoc\ntorch/lib/tmp_install\ntorch/lib/torch_shm_manager\ntorch/version.py\n\n# IPython notebook checkpoints\n.ipynb_checkpoints\n\n# Editor temporaries\n*.swn\n*.swo\n*.swp\n*.swm\n*~\n\n# macOS dir files\n.DS_Store\n\n# Symbolic files\ntools/shared/cwrap_common.py\n\n# Ninja files\n.ninja_deps\n.ninja_log\ncompile_commands.json\n*.egg-info/\ndocs/source/scripts/activation_images/\n\n## General\n\n# Compiled Object files\n*.slo\n*.lo\n*.o\n*.cuo\n*.obj\n\n# Compiled Dynamic libraries\n*.so\n*.dylib\n*.dll\n\n# Compiled Static libraries\n*.lai\n*.la\n*.a\n*.lib\n\n# Compiled protocol buffers\n*.pb.h\n*.pb.cc\n*_pb2.py\n\n# Compiled python\n*.pyc\n*.pyd\n\n# Compiled MATLAB\n*.mex*\n\n# IPython notebook checkpoints\n.ipynb_checkpoints\n\n# Editor temporaries\n*.swn\n*.swo\n*.swp\n*~\n\n# Sublime Text settings\n*.sublime-workspace\n*.sublime-project\n\n# Eclipse Project settings\n*.*project\n.settings\n\n# QtCreator files\n*.user\n\n# PyCharm files\n.idea\n\n# Visual Studio Code files\n.vscode\n.vs\n\n# OSX dir files\n.DS_Store\n\n## Caffe2\n\n# build, distribute, and bins (+ python proto bindings)\nbuild\nbuild_host_protoc\nbuild_android\nbuild_ios\n/build_*\n.build_debug/*\n.build_release/*\ndistribute/*\n*.testbin\n*.bin\ncmake_build\n.cmake_build\ngen\n.setuptools-cmake-build\n.pytest_cache\naten/build/*\n\n# Bram\nplsdontbreak\n\n# Generated documentation\ndocs/_site\ndocs/gathered\n_site\ndoxygen\ndocs/dev\n\n# LevelDB files\n*.sst\n*.ldb\nLOCK\nLOG*\nCURRENT\nMANIFEST-*\n\n# generated version file\ncaffe2/version.py\n\n# setup.py intermediates\n.eggs\ncaffe2.egg-info\n\n# Atom/Watchman required file\n.watchmanconfig\n\n# cython generated files\nlib/model/utils/bbox.c\nlib/pycocotools/_mask.c"
},
{
"path": "LICENSE",
"content": "MIT License\n\nCopyright (c) 2017 Jianwei Yang\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\nSOFTWARE.\n"
},
{
"path": "README.md",
"content": "# A *Faster* Pytorch Implementation of Faster R-CNN\n\n## Write at the beginning\n\n[05/29/2020] This repo was initaited about two years ago, developed as the first open-sourced object detection code which supports multi-gpu training. It has been integrating tremendous efforts from many people. However, we have seen many high-quality repos emerged in the last years, such as:\n\n* [maskrcnn-benchmark](https://github.com/facebookresearch/maskrcnn-benchmark)\n* [detectron2](https://github.com/facebookresearch/detectron2)\n* [mmdetection](https://github.com/open-mmlab/mmdetection)\n\n**At this point, I think this repo is out-of-data in terms of the pipeline and coding style, and will not maintain actively. Though you can still use this repo as a playground, I highly recommend you move to the above repos to delve into west world of object detection!**\n\n## Introduction\n\n### :boom: Good news! This repo supports pytorch-1.0 now!!! We borrowed some code and techniques from [maskrcnn-benchmark](https://github.com/facebookresearch/maskrcnn-benchmark). Just go to pytorch-1.0 branch!\n\nThis project is a *faster* pytorch implementation of faster R-CNN, aimed to accelerating the training of faster R-CNN object detection models. Recently, there are a number of good implementations:\n\n* [rbgirshick/py-faster-rcnn](https://github.com/rbgirshick/py-faster-rcnn), developed based on Pycaffe + Numpy\n\n* [longcw/faster_rcnn_pytorch](https://github.com/longcw/faster_rcnn_pytorch), developed based on Pytorch + Numpy\n\n* [endernewton/tf-faster-rcnn](https://github.com/endernewton/tf-faster-rcnn), developed based on TensorFlow + Numpy\n\n* [ruotianluo/pytorch-faster-rcnn](https://github.com/ruotianluo/pytorch-faster-rcnn), developed based on Pytorch + TensorFlow + Numpy\n\nDuring our implementing, we referred the above implementations, especailly [longcw/faster_rcnn_pytorch](https://github.com/longcw/faster_rcnn_pytorch). However, our implementation has several unique and new features compared with the above implementations:\n\n* **It is pure Pytorch code**. We convert all the numpy implementations to pytorch!\n\n* **It supports multi-image batch training**. We revise all the layers, including dataloader, rpn, roi-pooling, etc., to support multiple images in each minibatch.\n\n* **It supports multiple GPUs training**. We use a multiple GPU wrapper (nn.DataParallel here) to make it flexible to use one or more GPUs, as a merit of the above two features.\n\n* **It supports three pooling methods**. We integrate three pooling methods: roi pooing, roi align and roi crop. More importantly, we modify all of them to support multi-image batch training.\n\n* **It is memory efficient**. We limit the image aspect ratio, and group images with similar aspect ratios into a minibatch. As such, we can train resnet101 and VGG16 with batchsize = 4 (4 images) on a single Titan X (12 GB). When training with 8 GPU, the maximum batchsize for each GPU is 3 (Res101), totaling 24.\n\n* **It is faster**. Based on the above modifications, the training is much faster. We report the training speed on NVIDIA TITAN Xp in the tables below.\n\n### What we are doing and going to do\n\n- [x] Support both python2 and python3 (great thanks to [cclauss](https://github.com/cclauss)).\n- [x] Add deformable pooling layer (mainly supported by [Xander](https://github.com/xanderchf)).\n- [x] Support pytorch-0.4.0 (this branch).\n- [x] Support tensorboardX.\n- [x] Support pytorch-1.0 (go to pytorch-1.0 branch).\n\n## Other Implementations\n\n* [Feature Pyramid Network (FPN)](https://github.com/jwyang/fpn.pytorch)\n\n* [Mask R-CNN](https://github.com/roytseng-tw/mask-rcnn.pytorch) (~~ongoing~~ already implemented by [roytseng-tw](https://github.com/roytseng-tw))\n\n* [Graph R-CNN](https://github.com/jwyang/graph-rcnn.pytorch) (extension to scene graph generation)\n\n## Tutorial\n\n* [Blog](http://www.telesens.co/2018/03/11/object-detection-and-classification-using-r-cnns/) by [ankur6ue](https://github.com/ankur6ue)\n\n## Benchmarking\n\nWe benchmark our code thoroughly on three datasets: pascal voc, coco and visual genome, using two different network architectures: vgg16 and resnet101. Below are the results:\n\n1). PASCAL VOC 2007 (Train/Test: 07trainval/07test, scale=600, ROI Align)\n\nmodel | #GPUs | batch size | lr | lr_decay | max_epoch | time/epoch | mem/GPU | mAP\n---------|--------|-----|--------|-----|-----|-------|--------|-----\n[VGG-16](https://www.dropbox.com/s/6ief4w7qzka6083/faster_rcnn_1_6_10021.pth?dl=0) | 1 | 1 | 1e-3 | 5 | 6 | 0.76 hr | 3265MB | 70.1\n[VGG-16](https://www.dropbox.com/s/cpj2nu35am0f9hp/faster_rcnn_1_9_2504.pth?dl=0) | 1 | 4 | 4e-3 | 8 | 9 | 0.50 hr | 9083MB | 69.6\n[VGG-16](https://www.dropbox.com/s/1a31y7vicby0kvy/faster_rcnn_1_10_625.pth?dl=0) | 8 | 16| 1e-2 | 8 | 10 | 0.19 hr | 5291MB | 69.4\n[VGG-16](https://www.dropbox.com/s/hkj7i6mbhw9tq4k/faster_rcnn_1_11_416.pth?dl=0) | 8 | 24| 1e-2 | 10 | 11 | 0.16 hr | 11303MB | 69.2\n[Res-101](https://www.dropbox.com/s/4v3or0054kzl19q/faster_rcnn_1_7_10021.pth?dl=0) | 1 | 1 | 1e-3 | 5 | 7 | 0.88 hr | 3200 MB | 75.2\n[Res-101](https://www.dropbox.com/s/8bhldrds3mf0yuj/faster_rcnn_1_10_2504.pth?dl=0) | 1 | 4 | 4e-3 | 8 | 10 | 0.60 hr | 9700 MB | 74.9\n[Res-101](https://www.dropbox.com/s/5is50y01m1l9hbu/faster_rcnn_1_10_625.pth?dl=0) | 8 | 16| 1e-2 | 8 | 10 | 0.23 hr | 8400 MB | 75.2 \n[Res-101](https://www.dropbox.com/s/cn8gneumg4gjo9i/faster_rcnn_1_12_416.pth?dl=0) | 8 | 24| 1e-2 | 10 | 12 | 0.17 hr | 10327MB | 75.1 \n\n\n2). COCO (Train/Test: coco_train+coco_val-minival/minival, scale=800, max_size=1200, ROI Align)\n\nmodel | #GPUs | batch size |lr | lr_decay | max_epoch | time/epoch | mem/GPU | mAP\n---------|--------|-----|--------|-----|-----|-------|--------|-----\nVGG-16 | 8 | 16 |1e-2| 4 | 6 | 4.9 hr | 7192 MB | 29.2\n[Res-101](https://www.dropbox.com/s/5if6l7mqsi4rfk9/faster_rcnn_1_6_14657.pth?dl=0) | 8 | 16 |1e-2| 4 | 6 | 6.0 hr |10956 MB | 36.2\n[Res-101](https://www.dropbox.com/s/be0isevd22eikqb/faster_rcnn_1_10_14657.pth?dl=0) | 8 | 16 |1e-2| 4 | 10 | 6.0 hr |10956 MB | 37.0\n\n**NOTE**. Since the above models use scale=800, you need add \"--ls\" at the end of test command.\n\n3). COCO (Train/Test: coco_train+coco_val-minival/minival, scale=600, max_size=1000, ROI Align)\n\nmodel | #GPUs | batch size |lr | lr_decay | max_epoch | time/epoch | mem/GPU | mAP\n---------|--------|-----|--------|-----|-----|-------|--------|-----\n[Res-101](https://www.dropbox.com/s/y171ze1sdw1o2ph/faster_rcnn_1_6_9771.pth?dl=0) | 8 | 24 |1e-2| 4 | 6 | 5.4 hr |10659 MB | 33.9\n[Res-101](https://www.dropbox.com/s/dpq6qv0efspelr3/faster_rcnn_1_10_9771.pth?dl=0) | 8 | 24 |1e-2| 4 | 10 | 5.4 hr |10659 MB | 34.5\n\n4). Visual Genome (Train/Test: vg_train/vg_test, scale=600, max_size=1000, ROI Align, category=2500)\n\nmodel | #GPUs | batch size |lr | lr_decay | max_epoch | time/epoch | mem/GPU | mAP\n---------|--------|-----|--------|-----|-----|-------|--------|-----\n[VGG-16](http://data.lip6.fr/cadene/faster-rcnn.pytorch/faster_rcnn_1_19_48611.pth) | 1 P100 | 4 |1e-3| 5 | 20 | 3.7 hr |12707 MB | 4.4\n\nThanks to [Remi](https://github.com/Cadene) for providing the pretrained detection model on visual genome!\n\n* Click the links in the above tables to download our pre-trained faster r-cnn models.\n* If not mentioned, the GPU we used is NVIDIA Titan X Pascal (12GB).\n\n## Preparation\n\n\nFirst of all, clone the code\n```\ngit clone https://github.com/jwyang/faster-rcnn.pytorch.git\n```\n\nThen, create a folder:\n```\ncd faster-rcnn.pytorch && mkdir data\n```\n\n### prerequisites\n\n* Python 2.7 or 3.6\n* Pytorch 0.4.0 (**now it does not support 0.4.1 or higher**)\n* CUDA 8.0 or higher\n\n### Data Preparation\n\n* **PASCAL_VOC 07+12**: Please follow the instructions in [py-faster-rcnn](https://github.com/rbgirshick/py-faster-rcnn#beyond-the-demo-installation-for-training-and-testing-models) to prepare VOC datasets. Actually, you can refer to any others. After downloading the data, create softlinks in the folder data/.\n\n* **COCO**: Please also follow the instructions in [py-faster-rcnn](https://github.com/rbgirshick/py-faster-rcnn#beyond-the-demo-installation-for-training-and-testing-models) to prepare the data.\n\n* **Visual Genome**: Please follow the instructions in [bottom-up-attention](https://github.com/peteanderson80/bottom-up-attention) to prepare Visual Genome dataset. You need to download the images and object annotation files first, and then perform proprecessing to obtain the vocabulary and cleansed annotations based on the scripts provided in this repository.\n\n### Pretrained Model\n\nWe used two pretrained models in our experiments, VGG and ResNet101. You can download these two models from:\n\n* VGG16: [Dropbox](https://www.dropbox.com/s/s3brpk0bdq60nyb/vgg16_caffe.pth?dl=0), [VT Server](https://filebox.ece.vt.edu/~jw2yang/faster-rcnn/pretrained-base-models/vgg16_caffe.pth)\n\n* ResNet101: [Dropbox](https://www.dropbox.com/s/iev3tkbz5wyyuz9/resnet101_caffe.pth?dl=0), [VT Server](https://filebox.ece.vt.edu/~jw2yang/faster-rcnn/pretrained-base-models/resnet101_caffe.pth)\n\nDownload them and put them into the data/pretrained_model/.\n\n**NOTE**. We compare the pretrained models from Pytorch and Caffe, and surprisingly find Caffe pretrained models have slightly better performance than Pytorch pretrained. We would suggest to use Caffe pretrained models from the above link to reproduce our results.\n\n**If you want to use pytorch pre-trained models, please remember to transpose images from BGR to RGB, and also use the same data transformer (minus mean and normalize) as used in pretrained model.**\n\n### Compilation\n\nAs pointed out by [ruotianluo/pytorch-faster-rcnn](https://github.com/ruotianluo/pytorch-faster-rcnn), choose the right `-arch` in `make.sh` file, to compile the cuda code:\n\n | GPU model | Architecture |\n | ------------- | ------------- |\n | TitanX (Maxwell/Pascal) | sm_52 |\n | GTX 960M | sm_50 |\n | GTX 1080 (Ti) | sm_61 |\n | Grid K520 (AWS g2.2xlarge) | sm_30 |\n | Tesla K80 (AWS p2.xlarge) | sm_37 |\n\nMore details about setting the architecture can be found [here](https://developer.nvidia.com/cuda-gpus) or [here](http://arnon.dk/matching-sm-architectures-arch-and-gencode-for-various-nvidia-cards/)\n\nInstall all the python dependencies using pip:\n```\npip install -r requirements.txt\n```\n\nCompile the cuda dependencies using following simple commands:\n\n```\ncd lib\nsh make.sh\n```\n\nIt will compile all the modules you need, including NMS, ROI_Pooing, ROI_Align and ROI_Crop. The default version is compiled with Python 2.7, please compile by yourself if you are using a different python version.\n\n**As pointed out in this [issue](https://github.com/jwyang/faster-rcnn.pytorch/issues/16), if you encounter some error during the compilation, you might miss to export the CUDA paths to your environment.**\n\n## Train\n\nBefore training, set the right directory to save and load the trained models. Change the arguments \"save_dir\" and \"load_dir\" in trainval_net.py and test_net.py to adapt to your environment.\n\nTo train a faster R-CNN model with vgg16 on pascal_voc, simply run:\n```\nCUDA_VISIBLE_DEVICES=$GPU_ID python trainval_net.py \\\n --dataset pascal_voc --net vgg16 \\\n --bs $BATCH_SIZE --nw $WORKER_NUMBER \\\n --lr $LEARNING_RATE --lr_decay_step $DECAY_STEP \\\n --cuda\n```\nwhere 'bs' is the batch size with default 1. Alternatively, to train with resnet101 on pascal_voc, simple run:\n```\n CUDA_VISIBLE_DEVICES=$GPU_ID python trainval_net.py \\\n --dataset pascal_voc --net res101 \\\n --bs $BATCH_SIZE --nw $WORKER_NUMBER \\\n --lr $LEARNING_RATE --lr_decay_step $DECAY_STEP \\\n --cuda\n```\nAbove, BATCH_SIZE and WORKER_NUMBER can be set adaptively according to your GPU memory size. **On Titan Xp with 12G memory, it can be up to 4**.\n\nIf you have multiple (say 8) Titan Xp GPUs, then just use them all! Try:\n```\npython trainval_net.py --dataset pascal_voc --net vgg16 \\\n --bs 24 --nw 8 \\\n --lr $LEARNING_RATE --lr_decay_step $DECAY_STEP \\\n --cuda --mGPUs\n\n```\n\nChange dataset to \"coco\" or 'vg' if you want to train on COCO or Visual Genome.\n\n## Test\n\nIf you want to evaluate the detection performance of a pre-trained vgg16 model on pascal_voc test set, simply run\n```\npython test_net.py --dataset pascal_voc --net vgg16 \\\n --checksession $SESSION --checkepoch $EPOCH --checkpoint $CHECKPOINT \\\n --cuda\n```\nSpecify the specific model session, checkepoch and checkpoint, e.g., SESSION=1, EPOCH=6, CHECKPOINT=416.\n\n## Demo\n\nIf you want to run detection on your own images with a pre-trained model, download the pretrained model listed in above tables or train your own models at first, then add images to folder $ROOT/images, and then run\n```\npython demo.py --net vgg16 \\\n --checksession $SESSION --checkepoch $EPOCH --checkpoint $CHECKPOINT \\\n --cuda --load_dir path/to/model/directoy\n```\n\nThen you will find the detection results in folder $ROOT/images.\n\n**Note the default demo.py merely support pascal_voc categories. You need to change the [line](https://github.com/jwyang/faster-rcnn.pytorch/blob/530f3fdccaa60d05fa068bc2148695211586bd88/demo.py#L156) to adapt your own model.**\n\nBelow are some detection results:\n\n\n
![](\"images/img3_det_res101.jpg\")
![](\"images/img4_det_res101.jpg\")
\n
\n#include \n#include \n\n\nvoid ROIAlignForwardCpu(const float* bottom_data, const float spatial_scale, const int num_rois,\n const int height, const int width, const int channels,\n const int aligned_height, const int aligned_width, const float * bottom_rois,\n float* top_data);\n\nvoid ROIAlignBackwardCpu(const float* top_diff, const float spatial_scale, const int num_rois,\n const int height, const int width, const int channels,\n const int aligned_height, const int aligned_width, const float * bottom_rois,\n float* top_data);\n\nint roi_align_forward(int aligned_height, int aligned_width, float spatial_scale,\n THFloatTensor * features, THFloatTensor * rois, THFloatTensor * output)\n{\n //Grab the input tensor\n float * data_flat = THFloatTensor_data(features);\n float * rois_flat = THFloatTensor_data(rois);\n\n float * output_flat = THFloatTensor_data(output);\n\n // Number of ROIs\n int num_rois = THFloatTensor_size(rois, 0);\n int size_rois = THFloatTensor_size(rois, 1);\n if (size_rois != 5)\n {\n return 0;\n }\n\n // data height\n int data_height = THFloatTensor_size(features, 2);\n // data width\n int data_width = THFloatTensor_size(features, 3);\n // Number of channels\n int num_channels = THFloatTensor_size(features, 1);\n\n // do ROIAlignForward\n ROIAlignForwardCpu(data_flat, spatial_scale, num_rois, data_height, data_width, num_channels,\n aligned_height, aligned_width, rois_flat, output_flat);\n\n return 1;\n}\n\nint roi_align_backward(int aligned_height, int aligned_width, float spatial_scale,\n THFloatTensor * top_grad, THFloatTensor * rois, THFloatTensor * bottom_grad)\n{\n //Grab the input tensor\n float * top_grad_flat = THFloatTensor_data(top_grad);\n float * rois_flat = THFloatTensor_data(rois);\n\n float * bottom_grad_flat = THFloatTensor_data(bottom_grad);\n\n // Number of ROIs\n int num_rois = THFloatTensor_size(rois, 0);\n int size_rois = THFloatTensor_size(rois, 1);\n if (size_rois != 5)\n {\n return 0;\n }\n\n // batch size\n // int batch_size = THFloatTensor_size(bottom_grad, 0);\n // data height\n int data_height = THFloatTensor_size(bottom_grad, 2);\n // data width\n int data_width = THFloatTensor_size(bottom_grad, 3);\n // Number of channels\n int num_channels = THFloatTensor_size(bottom_grad, 1);\n\n // do ROIAlignBackward\n ROIAlignBackwardCpu(top_grad_flat, spatial_scale, num_rois, data_height,\n data_width, num_channels, aligned_height, aligned_width, rois_flat, bottom_grad_flat);\n\n return 1;\n}\n\nvoid ROIAlignForwardCpu(const float* bottom_data, const float spatial_scale, const int num_rois,\n const int height, const int width, const int channels,\n const int aligned_height, const int aligned_width, const float * bottom_rois,\n float* top_data)\n{\n const int output_size = num_rois * aligned_height * aligned_width * channels;\n\n int idx = 0;\n for (idx = 0; idx < output_size; ++idx)\n {\n // (n, c, ph, pw) is an element in the aligned output\n int pw = idx % aligned_width;\n int ph = (idx / aligned_width) % aligned_height;\n int c = (idx / aligned_width / aligned_height) % channels;\n int n = idx / aligned_width / aligned_height / channels;\n\n float roi_batch_ind = bottom_rois[n * 5 + 0];\n float roi_start_w = bottom_rois[n * 5 + 1] * spatial_scale;\n float roi_start_h = bottom_rois[n * 5 + 2] * spatial_scale;\n float roi_end_w = bottom_rois[n * 5 + 3] * spatial_scale;\n float roi_end_h = bottom_rois[n * 5 + 4] * spatial_scale;\n\n // Force malformed ROI to be 1x1\n float roi_width = fmaxf(roi_end_w - roi_start_w + 1., 0.);\n float roi_height = fmaxf(roi_end_h - roi_start_h + 1., 0.);\n float bin_size_h = roi_height / (aligned_height - 1.);\n float bin_size_w = roi_width / (aligned_width - 1.);\n\n float h = (float)(ph) * bin_size_h + roi_start_h;\n float w = (float)(pw) * bin_size_w + roi_start_w;\n\n int hstart = fminf(floor(h), height - 2);\n int wstart = fminf(floor(w), width - 2);\n\n int img_start = roi_batch_ind * channels * height * width;\n\n // bilinear interpolation\n if (h < 0 || h >= height || w < 0 || w >= width)\n {\n top_data[idx] = 0.;\n }\n else\n {\n float h_ratio = h - (float)(hstart);\n float w_ratio = w - (float)(wstart);\n int upleft = img_start + (c * height + hstart) * width + wstart;\n int upright = upleft + 1;\n int downleft = upleft + width;\n int downright = downleft + 1;\n\n top_data[idx] = bottom_data[upleft] * (1. - h_ratio) * (1. - w_ratio)\n + bottom_data[upright] * (1. - h_ratio) * w_ratio\n + bottom_data[downleft] * h_ratio * (1. - w_ratio)\n + bottom_data[downright] * h_ratio * w_ratio;\n }\n }\n}\n\nvoid ROIAlignBackwardCpu(const float* top_diff, const float spatial_scale, const int num_rois,\n const int height, const int width, const int channels,\n const int aligned_height, const int aligned_width, const float * bottom_rois,\n float* bottom_diff)\n{\n const int output_size = num_rois * aligned_height * aligned_width * channels;\n\n int idx = 0;\n for (idx = 0; idx < output_size; ++idx)\n {\n // (n, c, ph, pw) is an element in the aligned output\n int pw = idx % aligned_width;\n int ph = (idx / aligned_width) % aligned_height;\n int c = (idx / aligned_width / aligned_height) % channels;\n int n = idx / aligned_width / aligned_height / channels;\n\n float roi_batch_ind = bottom_rois[n * 5 + 0];\n float roi_start_w = bottom_rois[n * 5 + 1] * spatial_scale;\n float roi_start_h = bottom_rois[n * 5 + 2] * spatial_scale;\n float roi_end_w = bottom_rois[n * 5 + 3] * spatial_scale;\n float roi_end_h = bottom_rois[n * 5 + 4] * spatial_scale;\n\n // Force malformed ROI to be 1x1\n float roi_width = fmaxf(roi_end_w - roi_start_w + 1., 0.);\n float roi_height = fmaxf(roi_end_h - roi_start_h + 1., 0.);\n float bin_size_h = roi_height / (aligned_height - 1.);\n float bin_size_w = roi_width / (aligned_width - 1.);\n\n float h = (float)(ph) * bin_size_h + roi_start_h;\n float w = (float)(pw) * bin_size_w + roi_start_w;\n\n int hstart = fminf(floor(h), height - 2);\n int wstart = fminf(floor(w), width - 2);\n\n int img_start = roi_batch_ind * channels * height * width;\n\n // bilinear interpolation\n if (h < 0 || h >= height || w < 0 || w >= width)\n {\n float h_ratio = h - (float)(hstart);\n float w_ratio = w - (float)(wstart);\n int upleft = img_start + (c * height + hstart) * width + wstart;\n int upright = upleft + 1;\n int downleft = upleft + width;\n int downright = downleft + 1;\n\n bottom_diff[upleft] += top_diff[idx] * (1. - h_ratio) * (1. - w_ratio);\n bottom_diff[upright] += top_diff[idx] * (1. - h_ratio) * w_ratio;\n bottom_diff[downleft] += top_diff[idx] * h_ratio * (1. - w_ratio);\n bottom_diff[downright] += top_diff[idx] * h_ratio * w_ratio;\n }\n }\n}\n"
},
{
"path": "lib/model/roi_align/src/roi_align.h",
"content": "int roi_align_forward(int aligned_height, int aligned_width, float spatial_scale,\n THFloatTensor * features, THFloatTensor * rois, THFloatTensor * output);\n\nint roi_align_backward(int aligned_height, int aligned_width, float spatial_scale,\n THFloatTensor * top_grad, THFloatTensor * rois, THFloatTensor * bottom_grad);\n"
},
{
"path": "lib/model/roi_align/src/roi_align_cuda.c",
"content": "#include \n#include \n#include \"roi_align_kernel.h\"\n\nextern THCState *state;\n\nint roi_align_forward_cuda(int aligned_height, int aligned_width, float spatial_scale,\n THCudaTensor * features, THCudaTensor * rois, THCudaTensor * output)\n{\n // Grab the input tensor\n float * data_flat = THCudaTensor_data(state, features);\n float * rois_flat = THCudaTensor_data(state, rois);\n\n float * output_flat = THCudaTensor_data(state, output);\n\n // Number of ROIs\n int num_rois = THCudaTensor_size(state, rois, 0);\n int size_rois = THCudaTensor_size(state, rois, 1);\n if (size_rois != 5)\n {\n return 0;\n }\n\n // data height\n int data_height = THCudaTensor_size(state, features, 2);\n // data width\n int data_width = THCudaTensor_size(state, features, 3);\n // Number of channels\n int num_channels = THCudaTensor_size(state, features, 1);\n\n cudaStream_t stream = THCState_getCurrentStream(state);\n\n ROIAlignForwardLaucher(\n data_flat, spatial_scale, num_rois, data_height,\n data_width, num_channels, aligned_height,\n aligned_width, rois_flat,\n output_flat, stream);\n\n return 1;\n}\n\nint roi_align_backward_cuda(int aligned_height, int aligned_width, float spatial_scale,\n THCudaTensor * top_grad, THCudaTensor * rois, THCudaTensor * bottom_grad)\n{\n // Grab the input tensor\n float * top_grad_flat = THCudaTensor_data(state, top_grad);\n float * rois_flat = THCudaTensor_data(state, rois);\n\n float * bottom_grad_flat = THCudaTensor_data(state, bottom_grad);\n\n // Number of ROIs\n int num_rois = THCudaTensor_size(state, rois, 0);\n int size_rois = THCudaTensor_size(state, rois, 1);\n if (size_rois != 5)\n {\n return 0;\n }\n\n // batch size\n int batch_size = THCudaTensor_size(state, bottom_grad, 0);\n // data height\n int data_height = THCudaTensor_size(state, bottom_grad, 2);\n // data width\n int data_width = THCudaTensor_size(state, bottom_grad, 3);\n // Number of channels\n int num_channels = THCudaTensor_size(state, bottom_grad, 1);\n\n cudaStream_t stream = THCState_getCurrentStream(state);\n ROIAlignBackwardLaucher(\n top_grad_flat, spatial_scale, batch_size, num_rois, data_height,\n data_width, num_channels, aligned_height,\n aligned_width, rois_flat,\n bottom_grad_flat, stream);\n\n return 1;\n}\n"
},
{
"path": "lib/model/roi_align/src/roi_align_cuda.h",
"content": "int roi_align_forward_cuda(int aligned_height, int aligned_width, float spatial_scale,\n THCudaTensor * features, THCudaTensor * rois, THCudaTensor * output);\n\nint roi_align_backward_cuda(int aligned_height, int aligned_width, float spatial_scale,\n THCudaTensor * top_grad, THCudaTensor * rois, THCudaTensor * bottom_grad);\n"
},
{
"path": "lib/model/roi_align/src/roi_align_kernel.cu",
"content": "#ifdef __cplusplus\nextern \"C\" {\n#endif\n\n#include \n#include \n#include \n#include \"roi_align_kernel.h\"\n\n#define CUDA_1D_KERNEL_LOOP(i, n) \\\n for (int i = blockIdx.x * blockDim.x + threadIdx.x; i < n; \\\n i += blockDim.x * gridDim.x)\n\n\n __global__ void ROIAlignForward(const int nthreads, const float* bottom_data, const float spatial_scale, const int height, const int width,\n const int channels, const int aligned_height, const int aligned_width, const float* bottom_rois, float* top_data) {\n CUDA_1D_KERNEL_LOOP(index, nthreads) {\n // (n, c, ph, pw) is an element in the aligned output\n // int n = index;\n // int pw = n % aligned_width;\n // n /= aligned_width;\n // int ph = n % aligned_height;\n // n /= aligned_height;\n // int c = n % channels;\n // n /= channels;\n\n int pw = index % aligned_width;\n int ph = (index / aligned_width) % aligned_height;\n int c = (index / aligned_width / aligned_height) % channels;\n int n = index / aligned_width / aligned_height / channels;\n\n // bottom_rois += n * 5;\n float roi_batch_ind = bottom_rois[n * 5 + 0];\n float roi_start_w = bottom_rois[n * 5 + 1] * spatial_scale;\n float roi_start_h = bottom_rois[n * 5 + 2] * spatial_scale;\n float roi_end_w = bottom_rois[n * 5 + 3] * spatial_scale;\n float roi_end_h = bottom_rois[n * 5 + 4] * spatial_scale;\n\n // Force malformed ROIs to be 1x1\n float roi_width = fmaxf(roi_end_w - roi_start_w + 1., 0.);\n float roi_height = fmaxf(roi_end_h - roi_start_h + 1., 0.);\n float bin_size_h = roi_height / (aligned_height - 1.);\n float bin_size_w = roi_width / (aligned_width - 1.);\n\n float h = (float)(ph) * bin_size_h + roi_start_h;\n float w = (float)(pw) * bin_size_w + roi_start_w;\n\n int hstart = fminf(floor(h), height - 2);\n int wstart = fminf(floor(w), width - 2);\n\n int img_start = roi_batch_ind * channels * height * width;\n\n // bilinear interpolation\n if (h < 0 || h >= height || w < 0 || w >= width) {\n top_data[index] = 0.;\n } else {\n float h_ratio = h - (float)(hstart);\n float w_ratio = w - (float)(wstart);\n int upleft = img_start + (c * height + hstart) * width + wstart;\n int upright = upleft + 1;\n int downleft = upleft + width;\n int downright = downleft + 1;\n\n top_data[index] = bottom_data[upleft] * (1. - h_ratio) * (1. - w_ratio)\n + bottom_data[upright] * (1. - h_ratio) * w_ratio\n + bottom_data[downleft] * h_ratio * (1. - w_ratio)\n + bottom_data[downright] * h_ratio * w_ratio;\n }\n }\n }\n\n\n int ROIAlignForwardLaucher(const float* bottom_data, const float spatial_scale, const int num_rois, const int height, const int width,\n const int channels, const int aligned_height, const int aligned_width, const float* bottom_rois, float* top_data, cudaStream_t stream) {\n const int kThreadsPerBlock = 1024;\n const int output_size = num_rois * aligned_height * aligned_width * channels;\n cudaError_t err;\n\n\n ROIAlignForward<<<(output_size + kThreadsPerBlock - 1) / kThreadsPerBlock, kThreadsPerBlock, 0, stream>>>(\n output_size, bottom_data, spatial_scale, height, width, channels,\n aligned_height, aligned_width, bottom_rois, top_data);\n\n err = cudaGetLastError();\n if(cudaSuccess != err) {\n fprintf( stderr, \"cudaCheckError() failed : %s\\n\", cudaGetErrorString( err ) );\n exit( -1 );\n }\n\n return 1;\n }\n\n\n __global__ void ROIAlignBackward(const int nthreads, const float* top_diff, const float spatial_scale, const int height, const int width,\n const int channels, const int aligned_height, const int aligned_width, float* bottom_diff, const float* bottom_rois) {\n CUDA_1D_KERNEL_LOOP(index, nthreads) {\n\n // (n, c, ph, pw) is an element in the aligned output\n int pw = index % aligned_width;\n int ph = (index / aligned_width) % aligned_height;\n int c = (index / aligned_width / aligned_height) % channels;\n int n = index / aligned_width / aligned_height / channels;\n\n float roi_batch_ind = bottom_rois[n * 5 + 0];\n float roi_start_w = bottom_rois[n * 5 + 1] * spatial_scale;\n float roi_start_h = bottom_rois[n * 5 + 2] * spatial_scale;\n float roi_end_w = bottom_rois[n * 5 + 3] * spatial_scale;\n float roi_end_h = bottom_rois[n * 5 + 4] * spatial_scale;\n /* int roi_start_w = round(bottom_rois[1] * spatial_scale); */\n /* int roi_start_h = round(bottom_rois[2] * spatial_scale); */\n /* int roi_end_w = round(bottom_rois[3] * spatial_scale); */\n /* int roi_end_h = round(bottom_rois[4] * spatial_scale); */\n\n // Force malformed ROIs to be 1x1\n float roi_width = fmaxf(roi_end_w - roi_start_w + 1., 0.);\n float roi_height = fmaxf(roi_end_h - roi_start_h + 1., 0.);\n float bin_size_h = roi_height / (aligned_height - 1.);\n float bin_size_w = roi_width / (aligned_width - 1.);\n\n float h = (float)(ph) * bin_size_h + roi_start_h;\n float w = (float)(pw) * bin_size_w + roi_start_w;\n\n int hstart = fminf(floor(h), height - 2);\n int wstart = fminf(floor(w), width - 2);\n\n int img_start = roi_batch_ind * channels * height * width;\n\n // bilinear interpolation\n if (!(h < 0 || h >= height || w < 0 || w >= width)) {\n float h_ratio = h - (float)(hstart);\n float w_ratio = w - (float)(wstart);\n int upleft = img_start + (c * height + hstart) * width + wstart;\n int upright = upleft + 1;\n int downleft = upleft + width;\n int downright = downleft + 1;\n\n atomicAdd(bottom_diff + upleft, top_diff[index] * (1. - h_ratio) * (1 - w_ratio));\n atomicAdd(bottom_diff + upright, top_diff[index] * (1. - h_ratio) * w_ratio);\n atomicAdd(bottom_diff + downleft, top_diff[index] * h_ratio * (1 - w_ratio));\n atomicAdd(bottom_diff + downright, top_diff[index] * h_ratio * w_ratio);\n }\n }\n }\n\n int ROIAlignBackwardLaucher(const float* top_diff, const float spatial_scale, const int batch_size, const int num_rois, const int height, const int width,\n const int channels, const int aligned_height, const int aligned_width, const float* bottom_rois, float* bottom_diff, cudaStream_t stream) {\n const int kThreadsPerBlock = 1024;\n const int output_size = num_rois * aligned_height * aligned_width * channels;\n cudaError_t err;\n\n ROIAlignBackward<<<(output_size + kThreadsPerBlock - 1) / kThreadsPerBlock, kThreadsPerBlock, 0, stream>>>(\n output_size, top_diff, spatial_scale, height, width, channels,\n aligned_height, aligned_width, bottom_diff, bottom_rois);\n\n err = cudaGetLastError();\n if(cudaSuccess != err) {\n fprintf( stderr, \"cudaCheckError() failed : %s\\n\", cudaGetErrorString( err ) );\n exit( -1 );\n }\n\n return 1;\n }\n\n\n#ifdef __cplusplus\n}\n#endif\n"
},
{
"path": "lib/model/roi_align/src/roi_align_kernel.h",
"content": "#ifndef _ROI_ALIGN_KERNEL\n#define _ROI_ALIGN_KERNEL\n\n#ifdef __cplusplus\nextern \"C\" {\n#endif\n\n__global__ void ROIAlignForward(const int nthreads, const float* bottom_data,\n const float spatial_scale, const int height, const int width,\n const int channels, const int aligned_height, const int aligned_width,\n const float* bottom_rois, float* top_data);\n\nint ROIAlignForwardLaucher(\n const float* bottom_data, const float spatial_scale, const int num_rois, const int height,\n const int width, const int channels, const int aligned_height,\n const int aligned_width, const float* bottom_rois,\n float* top_data, cudaStream_t stream);\n\n__global__ void ROIAlignBackward(const int nthreads, const float* top_diff,\n const float spatial_scale, const int height, const int width,\n const int channels, const int aligned_height, const int aligned_width,\n float* bottom_diff, const float* bottom_rois);\n\nint ROIAlignBackwardLaucher(const float* top_diff, const float spatial_scale, const int batch_size, const int num_rois,\n const int height, const int width, const int channels, const int aligned_height,\n const int aligned_width, const float* bottom_rois,\n float* bottom_diff, cudaStream_t stream);\n\n#ifdef __cplusplus\n}\n#endif\n\n#endif\n\n"
},
{
"path": "lib/model/roi_crop/__init__.py",
"content": ""
},
{
"path": "lib/model/roi_crop/_ext/__init__.py",
"content": ""
},
{
"path": "lib/model/roi_crop/_ext/crop_resize/__init__.py",
"content": "\nfrom torch.utils.ffi import _wrap_function\nfrom ._crop_resize import lib as _lib, ffi as _ffi\n\n__all__ = []\ndef _import_symbols(locals):\n for symbol in dir(_lib):\n fn = getattr(_lib, symbol)\n locals[symbol] = _wrap_function(fn, _ffi)\n __all__.append(symbol)\n\n_import_symbols(locals())\n"
},
{
"path": "lib/model/roi_crop/_ext/roi_crop/__init__.py",
"content": "\nfrom torch.utils.ffi import _wrap_function\nfrom ._roi_crop import lib as _lib, ffi as _ffi\n\n__all__ = []\ndef _import_symbols(locals):\n for symbol in dir(_lib):\n fn = getattr(_lib, symbol)\n if callable(fn):\n locals[symbol] = _wrap_function(fn, _ffi)\n else:\n locals[symbol] = fn\n __all__.append(symbol)\n\n_import_symbols(locals())\n"
},
{
"path": "lib/model/roi_crop/build.py",
"content": "from __future__ import print_function\nimport os\nimport torch\nfrom torch.utils.ffi import create_extension\n\n#this_file = os.path.dirname(__file__)\n\nsources = ['src/roi_crop.c']\nheaders = ['src/roi_crop.h']\ndefines = []\nwith_cuda = False\n\nif torch.cuda.is_available():\n print('Including CUDA code.')\n sources += ['src/roi_crop_cuda.c']\n headers += ['src/roi_crop_cuda.h']\n defines += [('WITH_CUDA', None)]\n with_cuda = True\n\nthis_file = os.path.dirname(os.path.realpath(__file__))\nprint(this_file)\nextra_objects = ['src/roi_crop_cuda_kernel.cu.o']\nextra_objects = [os.path.join(this_file, fname) for fname in extra_objects]\n\nffi = create_extension(\n '_ext.roi_crop',\n headers=headers,\n sources=sources,\n define_macros=defines,\n relative_to=__file__,\n with_cuda=with_cuda,\n extra_objects=extra_objects\n)\n\nif __name__ == '__main__':\n ffi.build()\n"
},
{
"path": "lib/model/roi_crop/functions/__init__.py",
"content": ""
},
{
"path": "lib/model/roi_crop/functions/crop_resize.py",
"content": "# functions/add.py\nimport torch\nfrom torch.autograd import Function\nfrom .._ext import roi_crop\nfrom cffi import FFI\nffi = FFI()\n\nclass RoICropFunction(Function):\n def forward(self, input1, input2):\n self.input1 = input1\n self.input2 = input2\n self.device_c = ffi.new(\"int *\")\n output = torch.zeros(input2.size()[0], input1.size()[1], input2.size()[1], input2.size()[2])\n #print('decice %d' % torch.cuda.current_device())\n if input1.is_cuda:\n self.device = torch.cuda.current_device()\n else:\n self.device = -1\n self.device_c[0] = self.device\n if not input1.is_cuda:\n roi_crop.BilinearSamplerBHWD_updateOutput(input1, input2, output)\n else:\n output = output.cuda(self.device)\n roi_crop.BilinearSamplerBHWD_updateOutput_cuda(input1, input2, output)\n return output\n\n def backward(self, grad_output):\n grad_input1 = torch.zeros(self.input1.size())\n grad_input2 = torch.zeros(self.input2.size())\n #print('backward decice %d' % self.device)\n if not grad_output.is_cuda:\n roi_crop.BilinearSamplerBHWD_updateGradInput(self.input1, self.input2, grad_input1, grad_input2, grad_output)\n else:\n grad_input1 = grad_input1.cuda(self.device)\n grad_input2 = grad_input2.cuda(self.device)\n roi_crop.BilinearSamplerBHWD_updateGradInput_cuda(self.input1, self.input2, grad_input1, grad_input2, grad_output)\n return grad_input1, grad_input2\n"
},
{
"path": "lib/model/roi_crop/functions/gridgen.py",
"content": "# functions/add.py\nimport torch\nfrom torch.autograd import Function\nimport numpy as np\n\n\nclass AffineGridGenFunction(Function):\n def __init__(self, height, width,lr=1):\n super(AffineGridGenFunction, self).__init__()\n self.lr = lr\n self.height, self.width = height, width\n self.grid = np.zeros( [self.height, self.width, 3], dtype=np.float32)\n self.grid[:,:,0] = np.expand_dims(np.repeat(np.expand_dims(np.arange(-1, 1, 2.0/(self.height)), 0), repeats = self.width, axis = 0).T, 0)\n self.grid[:,:,1] = np.expand_dims(np.repeat(np.expand_dims(np.arange(-1, 1, 2.0/(self.width)), 0), repeats = self.height, axis = 0), 0)\n # self.grid[:,:,0] = np.expand_dims(np.repeat(np.expand_dims(np.arange(-1, 1, 2.0/(self.height - 1)), 0), repeats = self.width, axis = 0).T, 0)\n # self.grid[:,:,1] = np.expand_dims(np.repeat(np.expand_dims(np.arange(-1, 1, 2.0/(self.width - 1)), 0), repeats = self.height, axis = 0), 0)\n self.grid[:,:,2] = np.ones([self.height, width])\n self.grid = torch.from_numpy(self.grid.astype(np.float32))\n #print(self.grid)\n\n def forward(self, input1):\n self.input1 = input1\n output = input1.new(torch.Size([input1.size(0)]) + self.grid.size()).zero_()\n self.batchgrid = input1.new(torch.Size([input1.size(0)]) + self.grid.size()).zero_()\n for i in range(input1.size(0)):\n self.batchgrid[i] = self.grid.astype(self.batchgrid[i])\n\n # if input1.is_cuda:\n # self.batchgrid = self.batchgrid.cuda()\n # output = output.cuda()\n\n for i in range(input1.size(0)):\n output = torch.bmm(self.batchgrid.view(-1, self.height*self.width, 3), torch.transpose(input1, 1, 2)).view(-1, self.height, self.width, 2)\n\n return output\n\n def backward(self, grad_output):\n\n grad_input1 = self.input1.new(self.input1.size()).zero_()\n\n # if grad_output.is_cuda:\n # self.batchgrid = self.batchgrid.cuda()\n # grad_input1 = grad_input1.cuda()\n\n grad_input1 = torch.baddbmm(grad_input1, torch.transpose(grad_output.view(-1, self.height*self.width, 2), 1,2), self.batchgrid.view(-1, self.height*self.width, 3))\n return grad_input1\n"
},
{
"path": "lib/model/roi_crop/functions/roi_crop.py",
"content": "# functions/add.py\nimport torch\nfrom torch.autograd import Function\nfrom .._ext import roi_crop\nimport pdb\n\nclass RoICropFunction(Function):\n def forward(self, input1, input2):\n self.input1 = input1.clone()\n self.input2 = input2.clone()\n output = input2.new(input2.size()[0], input1.size()[1], input2.size()[1], input2.size()[2]).zero_()\n assert output.get_device() == input1.get_device(), \"output and input1 must on the same device\"\n assert output.get_device() == input2.get_device(), \"output and input2 must on the same device\"\n roi_crop.BilinearSamplerBHWD_updateOutput_cuda(input1, input2, output)\n return output\n\n def backward(self, grad_output):\n grad_input1 = self.input1.new(self.input1.size()).zero_()\n grad_input2 = self.input2.new(self.input2.size()).zero_()\n roi_crop.BilinearSamplerBHWD_updateGradInput_cuda(self.input1, self.input2, grad_input1, grad_input2, grad_output)\n return grad_input1, grad_input2\n"
},
{
"path": "lib/model/roi_crop/make.sh",
"content": "#!/usr/bin/env bash\n\nCUDA_PATH=/usr/local/cuda/\n\ncd src\necho \"Compiling my_lib kernels by nvcc...\"\nnvcc -c -o roi_crop_cuda_kernel.cu.o roi_crop_cuda_kernel.cu -x cu -Xcompiler -fPIC -arch=sm_52\n\ncd ../\npython build.py\n"
},
{
"path": "lib/model/roi_crop/modules/__init__.py",
"content": ""
},
{
"path": "lib/model/roi_crop/modules/gridgen.py",
"content": "from torch.nn.modules.module import Module\nimport torch\nfrom torch.autograd import Variable\nimport numpy as np\nfrom ..functions.gridgen import AffineGridGenFunction\n\nimport pyximport\npyximport.install(setup_args={\"include_dirs\":np.get_include()},\n reload_support=True)\n\n\nclass _AffineGridGen(Module):\n def __init__(self, height, width, lr = 1, aux_loss = False):\n super(_AffineGridGen, self).__init__()\n self.height, self.width = height, width\n self.aux_loss = aux_loss\n self.f = AffineGridGenFunction(self.height, self.width, lr=lr)\n self.lr = lr\n def forward(self, input):\n # if not self.aux_loss:\n return self.f(input)\n # else:\n # identity = torch.from_numpy(np.array([[1,0,0], [0,1,0]], dtype=np.float32))\n # batch_identity = torch.zeros([input.size(0), 2,3])\n # for i in range(input.size(0)):\n # batch_identity[i] = identity\n # batch_identity = Variable(batch_identity)\n # loss = torch.mul(input - batch_identity, input - batch_identity)\n # loss = torch.sum(loss,1)\n # loss = torch.sum(loss,2)\n\n # return self.f(input), loss.view(-1,1)\n\nclass CylinderGridGen(Module):\n def __init__(self, height, width, lr = 1, aux_loss = False):\n super(CylinderGridGen, self).__init__()\n self.height, self.width = height, width\n self.aux_loss = aux_loss\n self.f = CylinderGridGenFunction(self.height, self.width, lr=lr)\n self.lr = lr\n def forward(self, input):\n\n if not self.aux_loss:\n return self.f(input)\n else:\n return self.f(input), torch.mul(input, input).view(-1,1)\n\n\nclass AffineGridGenV2(Module):\n def __init__(self, height, width, lr = 1, aux_loss = False):\n super(AffineGridGenV2, self).__init__()\n self.height, self.width = height, width\n self.aux_loss = aux_loss\n self.lr = lr\n\n self.grid = np.zeros( [self.height, self.width, 3], dtype=np.float32)\n self.grid[:,:,0] = np.expand_dims(np.repeat(np.expand_dims(np.arange(-1, 1, 2.0/self.height), 0), repeats = self.width, axis = 0).T, 0)\n self.grid[:,:,1] = np.expand_dims(np.repeat(np.expand_dims(np.arange(-1, 1, 2.0/self.width), 0), repeats = self.height, axis = 0), 0)\n self.grid[:,:,2] = np.ones([self.height, width])\n self.grid = torch.from_numpy(self.grid.astype(np.float32))\n\n\n def forward(self, input1):\n self.batchgrid = torch.zeros(torch.Size([input1.size(0)]) + self.grid.size())\n\n for i in range(input1.size(0)):\n self.batchgrid[i] = self.grid\n self.batchgrid = Variable(self.batchgrid)\n\n if input1.is_cuda:\n self.batchgrid = self.batchgrid.cuda()\n\n output = torch.bmm(self.batchgrid.view(-1, self.height*self.width, 3), torch.transpose(input1, 1, 2)).view(-1, self.height, self.width, 2)\n\n return output\n\n\nclass CylinderGridGenV2(Module):\n def __init__(self, height, width, lr = 1):\n super(CylinderGridGenV2, self).__init__()\n self.height, self.width = height, width\n self.lr = lr\n self.grid = np.zeros( [self.height, self.width, 3], dtype=np.float32)\n self.grid[:,:,0] = np.expand_dims(np.repeat(np.expand_dims(np.arange(-1, 1, 2.0/self.height), 0), repeats = self.width, axis = 0).T, 0)\n self.grid[:,:,1] = np.expand_dims(np.repeat(np.expand_dims(np.arange(-1, 1, 2.0/self.width), 0), repeats = self.height, axis = 0), 0)\n self.grid[:,:,2] = np.ones([self.height, width])\n self.grid = torch.from_numpy(self.grid.astype(np.float32))\n def forward(self, input):\n self.batchgrid = torch.zeros(torch.Size([input.size(0)]) + self.grid.size() )\n #print(self.batchgrid.size())\n for i in range(input.size(0)):\n self.batchgrid[i,:,:,:] = self.grid\n self.batchgrid = Variable(self.batchgrid)\n\n #print(self.batchgrid.size())\n\n input_u = input.view(-1,1,1,1).repeat(1,self.height, self.width,1)\n #print(input_u.requires_grad, self.batchgrid)\n\n output0 = self.batchgrid[:,:,:,0:1]\n output1 = torch.atan(torch.tan(np.pi/2.0*(self.batchgrid[:,:,:,1:2] + self.batchgrid[:,:,:,2:] * input_u[:,:,:,:]))) /(np.pi/2)\n #print(output0.size(), output1.size())\n\n output = torch.cat([output0, output1], 3)\n return output\n\n\nclass DenseAffineGridGen(Module):\n def __init__(self, height, width, lr = 1, aux_loss = False):\n super(DenseAffineGridGen, self).__init__()\n self.height, self.width = height, width\n self.aux_loss = aux_loss\n self.lr = lr\n\n self.grid = np.zeros( [self.height, self.width, 3], dtype=np.float32)\n self.grid[:,:,0] = np.expand_dims(np.repeat(np.expand_dims(np.arange(-1, 1, 2.0/self.height), 0), repeats = self.width, axis = 0).T, 0)\n self.grid[:,:,1] = np.expand_dims(np.repeat(np.expand_dims(np.arange(-1, 1, 2.0/self.width), 0), repeats = self.height, axis = 0), 0)\n self.grid[:,:,2] = np.ones([self.height, width])\n self.grid = torch.from_numpy(self.grid.astype(np.float32))\n\n\n def forward(self, input1):\n self.batchgrid = torch.zeros(torch.Size([input1.size(0)]) + self.grid.size())\n\n for i in range(input1.size(0)):\n self.batchgrid[i] = self.grid\n\n self.batchgrid = Variable(self.batchgrid)\n #print self.batchgrid, input1[:,:,:,0:3]\n #print self.batchgrid, input1[:,:,:,4:6]\n x = torch.mul(self.batchgrid, input1[:,:,:,0:3])\n y = torch.mul(self.batchgrid, input1[:,:,:,3:6])\n\n output = torch.cat([torch.sum(x,3),torch.sum(y,3)], 3)\n return output\n\n\n\n\nclass DenseAffine3DGridGen(Module):\n def __init__(self, height, width, lr = 1, aux_loss = False):\n super(DenseAffine3DGridGen, self).__init__()\n self.height, self.width = height, width\n self.aux_loss = aux_loss\n self.lr = lr\n\n self.grid = np.zeros( [self.height, self.width, 3], dtype=np.float32)\n self.grid[:,:,0] = np.expand_dims(np.repeat(np.expand_dims(np.arange(-1, 1, 2.0/self.height), 0), repeats = self.width, axis = 0).T, 0)\n self.grid[:,:,1] = np.expand_dims(np.repeat(np.expand_dims(np.arange(-1, 1, 2.0/self.width), 0), repeats = self.height, axis = 0), 0)\n self.grid[:,:,2] = np.ones([self.height, width])\n self.grid = torch.from_numpy(self.grid.astype(np.float32))\n\n self.theta = self.grid[:,:,0] * np.pi/2 + np.pi/2\n self.phi = self.grid[:,:,1] * np.pi\n\n self.x = torch.sin(self.theta) * torch.cos(self.phi)\n self.y = torch.sin(self.theta) * torch.sin(self.phi)\n self.z = torch.cos(self.theta)\n\n self.grid3d = torch.from_numpy(np.zeros( [self.height, self.width, 4], dtype=np.float32))\n\n self.grid3d[:,:,0] = self.x\n self.grid3d[:,:,1] = self.y\n self.grid3d[:,:,2] = self.z\n self.grid3d[:,:,3] = self.grid[:,:,2]\n\n\n def forward(self, input1):\n self.batchgrid3d = torch.zeros(torch.Size([input1.size(0)]) + self.grid3d.size())\n\n for i in range(input1.size(0)):\n self.batchgrid3d[i] = self.grid3d\n\n self.batchgrid3d = Variable(self.batchgrid3d)\n #print(self.batchgrid3d)\n\n x = torch.sum(torch.mul(self.batchgrid3d, input1[:,:,:,0:4]), 3)\n y = torch.sum(torch.mul(self.batchgrid3d, input1[:,:,:,4:8]), 3)\n z = torch.sum(torch.mul(self.batchgrid3d, input1[:,:,:,8:]), 3)\n #print(x)\n r = torch.sqrt(x**2 + y**2 + z**2) + 1e-5\n\n #print(r)\n theta = torch.acos(z/r)/(np.pi/2) - 1\n #phi = torch.atan(y/x)\n phi = torch.atan(y/(x + 1e-5)) + np.pi * x.lt(0).type(torch.FloatTensor) * (y.ge(0).type(torch.FloatTensor) - y.lt(0).type(torch.FloatTensor))\n phi = phi/np.pi\n\n\n output = torch.cat([theta,phi], 3)\n\n return output\n\n\n\n\n\nclass DenseAffine3DGridGen_rotate(Module):\n def __init__(self, height, width, lr = 1, aux_loss = False):\n super(DenseAffine3DGridGen_rotate, self).__init__()\n self.height, self.width = height, width\n self.aux_loss = aux_loss\n self.lr = lr\n\n self.grid = np.zeros( [self.height, self.width, 3], dtype=np.float32)\n self.grid[:,:,0] = np.expand_dims(np.repeat(np.expand_dims(np.arange(-1, 1, 2.0/self.height), 0), repeats = self.width, axis = 0).T, 0)\n self.grid[:,:,1] = np.expand_dims(np.repeat(np.expand_dims(np.arange(-1, 1, 2.0/self.width), 0), repeats = self.height, axis = 0), 0)\n self.grid[:,:,2] = np.ones([self.height, width])\n self.grid = torch.from_numpy(self.grid.astype(np.float32))\n\n self.theta = self.grid[:,:,0] * np.pi/2 + np.pi/2\n self.phi = self.grid[:,:,1] * np.pi\n\n self.x = torch.sin(self.theta) * torch.cos(self.phi)\n self.y = torch.sin(self.theta) * torch.sin(self.phi)\n self.z = torch.cos(self.theta)\n\n self.grid3d = torch.from_numpy(np.zeros( [self.height, self.width, 4], dtype=np.float32))\n\n self.grid3d[:,:,0] = self.x\n self.grid3d[:,:,1] = self.y\n self.grid3d[:,:,2] = self.z\n self.grid3d[:,:,3] = self.grid[:,:,2]\n\n\n def forward(self, input1, input2):\n self.batchgrid3d = torch.zeros(torch.Size([input1.size(0)]) + self.grid3d.size())\n\n for i in range(input1.size(0)):\n self.batchgrid3d[i] = self.grid3d\n\n self.batchgrid3d = Variable(self.batchgrid3d)\n\n self.batchgrid = torch.zeros(torch.Size([input1.size(0)]) + self.grid.size())\n\n for i in range(input1.size(0)):\n self.batchgrid[i] = self.grid\n\n self.batchgrid = Variable(self.batchgrid)\n\n #print(self.batchgrid3d)\n\n x = torch.sum(torch.mul(self.batchgrid3d, input1[:,:,:,0:4]), 3)\n y = torch.sum(torch.mul(self.batchgrid3d, input1[:,:,:,4:8]), 3)\n z = torch.sum(torch.mul(self.batchgrid3d, input1[:,:,:,8:]), 3)\n #print(x)\n r = torch.sqrt(x**2 + y**2 + z**2) + 1e-5\n\n #print(r)\n theta = torch.acos(z/r)/(np.pi/2) - 1\n #phi = torch.atan(y/x)\n phi = torch.atan(y/(x + 1e-5)) + np.pi * x.lt(0).type(torch.FloatTensor) * (y.ge(0).type(torch.FloatTensor) - y.lt(0).type(torch.FloatTensor))\n phi = phi/np.pi\n\n input_u = input2.view(-1,1,1,1).repeat(1,self.height, self.width,1)\n\n output = torch.cat([theta,phi], 3)\n\n output1 = torch.atan(torch.tan(np.pi/2.0*(output[:,:,:,1:2] + self.batchgrid[:,:,:,2:] * input_u[:,:,:,:]))) /(np.pi/2)\n output2 = torch.cat([output[:,:,:,0:1], output1], 3)\n\n return output2\n\n\nclass Depth3DGridGen(Module):\n def __init__(self, height, width, lr = 1, aux_loss = False):\n super(Depth3DGridGen, self).__init__()\n self.height, self.width = height, width\n self.aux_loss = aux_loss\n self.lr = lr\n\n self.grid = np.zeros( [self.height, self.width, 3], dtype=np.float32)\n self.grid[:,:,0] = np.expand_dims(np.repeat(np.expand_dims(np.arange(-1, 1, 2.0/self.height), 0), repeats = self.width, axis = 0).T, 0)\n self.grid[:,:,1] = np.expand_dims(np.repeat(np.expand_dims(np.arange(-1, 1, 2.0/self.width), 0), repeats = self.height, axis = 0), 0)\n self.grid[:,:,2] = np.ones([self.height, width])\n self.grid = torch.from_numpy(self.grid.astype(np.float32))\n\n self.theta = self.grid[:,:,0] * np.pi/2 + np.pi/2\n self.phi = self.grid[:,:,1] * np.pi\n\n self.x = torch.sin(self.theta) * torch.cos(self.phi)\n self.y = torch.sin(self.theta) * torch.sin(self.phi)\n self.z = torch.cos(self.theta)\n\n self.grid3d = torch.from_numpy(np.zeros( [self.height, self.width, 4], dtype=np.float32))\n\n self.grid3d[:,:,0] = self.x\n self.grid3d[:,:,1] = self.y\n self.grid3d[:,:,2] = self.z\n self.grid3d[:,:,3] = self.grid[:,:,2]\n\n\n def forward(self, depth, trans0, trans1, rotate):\n self.batchgrid3d = torch.zeros(torch.Size([depth.size(0)]) + self.grid3d.size())\n\n for i in range(depth.size(0)):\n self.batchgrid3d[i] = self.grid3d\n\n self.batchgrid3d = Variable(self.batchgrid3d)\n\n self.batchgrid = torch.zeros(torch.Size([depth.size(0)]) + self.grid.size())\n\n for i in range(depth.size(0)):\n self.batchgrid[i] = self.grid\n\n self.batchgrid = Variable(self.batchgrid)\n\n x = self.batchgrid3d[:,:,:,0:1] * depth + trans0.view(-1,1,1,1).repeat(1, self.height, self.width, 1)\n\n y = self.batchgrid3d[:,:,:,1:2] * depth + trans1.view(-1,1,1,1).repeat(1, self.height, self.width, 1)\n z = self.batchgrid3d[:,:,:,2:3] * depth\n #print(x.size(), y.size(), z.size())\n r = torch.sqrt(x**2 + y**2 + z**2) + 1e-5\n\n #print(r)\n theta = torch.acos(z/r)/(np.pi/2) - 1\n #phi = torch.atan(y/x)\n phi = torch.atan(y/(x + 1e-5)) + np.pi * x.lt(0).type(torch.FloatTensor) * (y.ge(0).type(torch.FloatTensor) - y.lt(0).type(torch.FloatTensor))\n phi = phi/np.pi\n\n #print(theta.size(), phi.size())\n\n\n input_u = rotate.view(-1,1,1,1).repeat(1,self.height, self.width,1)\n\n output = torch.cat([theta,phi], 3)\n #print(output.size())\n\n output1 = torch.atan(torch.tan(np.pi/2.0*(output[:,:,:,1:2] + self.batchgrid[:,:,:,2:] * input_u[:,:,:,:]))) /(np.pi/2)\n output2 = torch.cat([output[:,:,:,0:1], output1], 3)\n\n return output2\n\n\n\n\n\nclass Depth3DGridGen_with_mask(Module):\n def __init__(self, height, width, lr = 1, aux_loss = False, ray_tracing = False):\n super(Depth3DGridGen_with_mask, self).__init__()\n self.height, self.width = height, width\n self.aux_loss = aux_loss\n self.lr = lr\n self.ray_tracing = ray_tracing\n\n self.grid = np.zeros( [self.height, self.width, 3], dtype=np.float32)\n self.grid[:,:,0] = np.expand_dims(np.repeat(np.expand_dims(np.arange(-1, 1, 2.0/self.height), 0), repeats = self.width, axis = 0).T, 0)\n self.grid[:,:,1] = np.expand_dims(np.repeat(np.expand_dims(np.arange(-1, 1, 2.0/self.width), 0), repeats = self.height, axis = 0), 0)\n self.grid[:,:,2] = np.ones([self.height, width])\n self.grid = torch.from_numpy(self.grid.astype(np.float32))\n\n self.theta = self.grid[:,:,0] * np.pi/2 + np.pi/2\n self.phi = self.grid[:,:,1] * np.pi\n\n self.x = torch.sin(self.theta) * torch.cos(self.phi)\n self.y = torch.sin(self.theta) * torch.sin(self.phi)\n self.z = torch.cos(self.theta)\n\n self.grid3d = torch.from_numpy(np.zeros( [self.height, self.width, 4], dtype=np.float32))\n\n self.grid3d[:,:,0] = self.x\n self.grid3d[:,:,1] = self.y\n self.grid3d[:,:,2] = self.z\n self.grid3d[:,:,3] = self.grid[:,:,2]\n\n\n def forward(self, depth, trans0, trans1, rotate):\n self.batchgrid3d = torch.zeros(torch.Size([depth.size(0)]) + self.grid3d.size())\n\n for i in range(depth.size(0)):\n self.batchgrid3d[i] = self.grid3d\n\n self.batchgrid3d = Variable(self.batchgrid3d)\n\n self.batchgrid = torch.zeros(torch.Size([depth.size(0)]) + self.grid.size())\n\n for i in range(depth.size(0)):\n self.batchgrid[i] = self.grid\n\n self.batchgrid = Variable(self.batchgrid)\n\n if depth.is_cuda:\n self.batchgrid = self.batchgrid.cuda()\n self.batchgrid3d = self.batchgrid3d.cuda()\n\n\n x_ = self.batchgrid3d[:,:,:,0:1] * depth + trans0.view(-1,1,1,1).repeat(1, self.height, self.width, 1)\n\n y_ = self.batchgrid3d[:,:,:,1:2] * depth + trans1.view(-1,1,1,1).repeat(1, self.height, self.width, 1)\n z = self.batchgrid3d[:,:,:,2:3] * depth\n #print(x.size(), y.size(), z.size())\n\n rotate_z = rotate.view(-1,1,1,1).repeat(1,self.height, self.width,1) * np.pi\n\n x = x_ * torch.cos(rotate_z) - y_ * torch.sin(rotate_z)\n y = x_ * torch.sin(rotate_z) + y_ * torch.cos(rotate_z)\n\n\n r = torch.sqrt(x**2 + y**2 + z**2) + 1e-5\n\n #print(r)\n theta = torch.acos(z/r)/(np.pi/2) - 1\n #phi = torch.atan(y/x)\n\n if depth.is_cuda:\n phi = torch.atan(y/(x + 1e-5)) + np.pi * x.lt(0).type(torch.cuda.FloatTensor) * (y.ge(0).type(torch.cuda.FloatTensor) - y.lt(0).type(torch.cuda.FloatTensor))\n else:\n phi = torch.atan(y/(x + 1e-5)) + np.pi * x.lt(0).type(torch.FloatTensor) * (y.ge(0).type(torch.FloatTensor) - y.lt(0).type(torch.FloatTensor))\n\n\n phi = phi/np.pi\n\n output = torch.cat([theta,phi], 3)\n return output\n"
},
{
"path": "lib/model/roi_crop/modules/roi_crop.py",
"content": "from torch.nn.modules.module import Module\nfrom ..functions.roi_crop import RoICropFunction\n\nclass _RoICrop(Module):\n def __init__(self, layout = 'BHWD'):\n super(_RoICrop, self).__init__()\n def forward(self, input1, input2):\n return RoICropFunction()(input1, input2)\n"
},
{
"path": "lib/model/roi_crop/src/roi_crop.c",
"content": "#include \n#include \n#include \n\n#define real float\n\nint BilinearSamplerBHWD_updateOutput(THFloatTensor *inputImages, THFloatTensor *grids, THFloatTensor *output)\n{\n\n int batchsize = THFloatTensor_size(inputImages, 0);\n int inputImages_height = THFloatTensor_size(inputImages, 1);\n int inputImages_width = THFloatTensor_size(inputImages, 2);\n int output_height = THFloatTensor_size(output, 1);\n int output_width = THFloatTensor_size(output, 2);\n int inputImages_channels = THFloatTensor_size(inputImages, 3);\n\n int output_strideBatch = THFloatTensor_stride(output, 0);\n int output_strideHeight = THFloatTensor_stride(output, 1);\n int output_strideWidth = THFloatTensor_stride(output, 2);\n\n int inputImages_strideBatch = THFloatTensor_stride(inputImages, 0);\n int inputImages_strideHeight = THFloatTensor_stride(inputImages, 1);\n int inputImages_strideWidth = THFloatTensor_stride(inputImages, 2);\n\n int grids_strideBatch = THFloatTensor_stride(grids, 0);\n int grids_strideHeight = THFloatTensor_stride(grids, 1);\n int grids_strideWidth = THFloatTensor_stride(grids, 2);\n\n real *inputImages_data, *output_data, *grids_data;\n inputImages_data = THFloatTensor_data(inputImages);\n output_data = THFloatTensor_data(output);\n grids_data = THFloatTensor_data(grids);\n\n int b, yOut, xOut;\n\n for(b=0; b < batchsize; b++)\n {\n for(yOut=0; yOut < output_height; yOut++)\n {\n for(xOut=0; xOut < output_width; xOut++)\n {\n //read the grid\n real yf = grids_data[b*grids_strideBatch + yOut*grids_strideHeight + xOut*grids_strideWidth];\n real xf = grids_data[b*grids_strideBatch + yOut*grids_strideHeight + xOut*grids_strideWidth + 1];\n\n // get the weights for interpolation\n int yInTopLeft, xInTopLeft;\n real yWeightTopLeft, xWeightTopLeft;\n\n real xcoord = (xf + 1) * (inputImages_width - 1) / 2;\n xInTopLeft = floor(xcoord);\n xWeightTopLeft = 1 - (xcoord - xInTopLeft);\n\n real ycoord = (yf + 1) * (inputImages_height - 1) / 2;\n yInTopLeft = floor(ycoord);\n yWeightTopLeft = 1 - (ycoord - yInTopLeft);\n\n\n\n const int outAddress = output_strideBatch * b + output_strideHeight * yOut + output_strideWidth * xOut;\n const int inTopLeftAddress = inputImages_strideBatch * b + inputImages_strideHeight * yInTopLeft + inputImages_strideWidth * xInTopLeft;\n const int inTopRightAddress = inTopLeftAddress + inputImages_strideWidth;\n const int inBottomLeftAddress = inTopLeftAddress + inputImages_strideHeight;\n const int inBottomRightAddress = inBottomLeftAddress + inputImages_strideWidth;\n\n real v=0;\n real inTopLeft=0;\n real inTopRight=0;\n real inBottomLeft=0;\n real inBottomRight=0;\n\n // we are careful with the boundaries\n bool topLeftIsIn = xInTopLeft >= 0 && xInTopLeft <= inputImages_width-1 && yInTopLeft >= 0 && yInTopLeft <= inputImages_height-1;\n bool topRightIsIn = xInTopLeft+1 >= 0 && xInTopLeft+1 <= inputImages_width-1 && yInTopLeft >= 0 && yInTopLeft <= inputImages_height-1;\n bool bottomLeftIsIn = xInTopLeft >= 0 && xInTopLeft <= inputImages_width-1 && yInTopLeft+1 >= 0 && yInTopLeft+1 <= inputImages_height-1;\n bool bottomRightIsIn = xInTopLeft+1 >= 0 && xInTopLeft+1 <= inputImages_width-1 && yInTopLeft+1 >= 0 && yInTopLeft+1 <= inputImages_height-1;\n\n int t;\n // interpolation happens here\n for(t=0; t= 0 && xInTopLeft <= inputImages_width-1 && yInTopLeft >= 0 && yInTopLeft <= inputImages_height-1;\n bool topRightIsIn = xInTopLeft+1 >= 0 && xInTopLeft+1 <= inputImages_width-1 && yInTopLeft >= 0 && yInTopLeft <= inputImages_height-1;\n bool bottomLeftIsIn = xInTopLeft >= 0 && xInTopLeft <= inputImages_width-1 && yInTopLeft+1 >= 0 && yInTopLeft+1 <= inputImages_height-1;\n bool bottomRightIsIn = xInTopLeft+1 >= 0 && xInTopLeft+1 <= inputImages_width-1 && yInTopLeft+1 >= 0 && yInTopLeft+1 <= inputImages_height-1;\n\n int t;\n\n for(t=0; t= 0 && xInTopLeft <= inputImages_width-1 && yInTopLeft >= 0 && yInTopLeft <= inputImages_height-1;\n bool topRightIsIn = xInTopLeft+1 >= 0 && xInTopLeft+1 <= inputImages_width-1 && yInTopLeft >= 0 && yInTopLeft <= inputImages_height-1;\n bool bottomLeftIsIn = xInTopLeft >= 0 && xInTopLeft <= inputImages_width-1 && yInTopLeft+1 >= 0 && yInTopLeft+1 <= inputImages_height-1;\n bool bottomRightIsIn = xInTopLeft+1 >= 0 && xInTopLeft+1 <= inputImages_width-1 && yInTopLeft+1 >= 0 && yInTopLeft+1 <= inputImages_height-1;\n\n int t;\n // interpolation happens here\n for(t=0; t= 0 && xInTopLeft <= inputImages_width-1 && yInTopLeft >= 0 && yInTopLeft <= inputImages_height-1;\n bool topRightIsIn = xInTopLeft+1 >= 0 && xInTopLeft+1 <= inputImages_width-1 && yInTopLeft >= 0 && yInTopLeft <= inputImages_height-1;\n bool bottomLeftIsIn = xInTopLeft >= 0 && xInTopLeft <= inputImages_width-1 && yInTopLeft+1 >= 0 && yInTopLeft+1 <= inputImages_height-1;\n bool bottomRightIsIn = xInTopLeft+1 >= 0 && xInTopLeft+1 <= inputImages_width-1 && yInTopLeft+1 >= 0 && yInTopLeft+1 <= inputImages_height-1;\n\n int t;\n\n for(t=0; t\n#include \n#include \n#include \"roi_crop_cuda_kernel.h\"\n\n#define real float\n\n// this symbol will be resolved automatically from PyTorch libs\nextern THCState *state;\n\n// Bilinear sampling is done in BHWD (coalescing is not obvious in BDHW)\n// we assume BHWD format in inputImages\n// we assume BHW(YX) format on grids\n\nint BilinearSamplerBHWD_updateOutput_cuda(THCudaTensor *inputImages, THCudaTensor *grids, THCudaTensor *output){\n// THCState *state = getCutorchState(L);\n// THCudaTensor *inputImages = (THCudaTensor *)luaT_checkudata(L, 2, \"torch.CudaTensor\");\n// THCudaTensor *grids = (THCudaTensor *)luaT_checkudata(L, 3, \"torch.CudaTensor\");\n// THCudaTensor *output = (THCudaTensor *)luaT_checkudata(L, 4, \"torch.CudaTensor\");\n\n int success = 0;\n success = BilinearSamplerBHWD_updateOutput_cuda_kernel(THCudaTensor_size(state, output, 1),\n\t\t\t\t\t\t\t THCudaTensor_size(state, output, 3),\n\t\t\t\t\t\t\t THCudaTensor_size(state, output, 2),\n\t\t\t\t\t\t\t THCudaTensor_size(state, output, 0),\n\t\t\t\t\t\t\t THCudaTensor_size(state, inputImages, 1),\n\t\t\t\t\t\t\t THCudaTensor_size(state, inputImages, 2),\n\t\t\t\t\t\t\t THCudaTensor_size(state, inputImages, 3),\n\t\t\t\t\t\t\t THCudaTensor_size(state, inputImages, 0),\n\t\t\t\t\t\t\t THCudaTensor_data(state, inputImages),\n\t\t\t\t\t\t\t THCudaTensor_stride(state, inputImages, 0),\n\t\t\t\t\t\t\t THCudaTensor_stride(state, inputImages, 1),\n\t\t\t\t\t\t\t THCudaTensor_stride(state, inputImages, 2),\n\t\t\t\t\t\t\t THCudaTensor_stride(state, inputImages, 3),\n\t\t\t\t\t\t\t THCudaTensor_data(state, grids),\n\t\t\t\t\t\t\t THCudaTensor_stride(state, grids, 0),\n\t\t\t\t\t\t\t THCudaTensor_stride(state, grids, 3),\n\t\t\t\t\t\t\t THCudaTensor_stride(state, grids, 1),\n\t\t\t\t\t\t\t THCudaTensor_stride(state, grids, 2),\n\t\t\t\t\t\t\t THCudaTensor_data(state, output),\n\t\t\t\t\t\t\t THCudaTensor_stride(state, output, 0),\n\t\t\t\t\t\t\t THCudaTensor_stride(state, output, 1),\n\t\t\t\t\t\t\t THCudaTensor_stride(state, output, 2),\n\t\t\t\t\t\t\t THCudaTensor_stride(state, output, 3),\n\t\t\t\t\t\t\t THCState_getCurrentStream(state));\n\n //check for errors\n if (!success) {\n THError(\"aborting\");\n }\n return 1;\n}\n\nint BilinearSamplerBHWD_updateGradInput_cuda(THCudaTensor *inputImages, THCudaTensor *grids, THCudaTensor *gradInputImages,\n THCudaTensor *gradGrids, THCudaTensor *gradOutput)\n{\n// THCState *state = getCutorchState(L);\n// THCudaTensor *inputImages = (THCudaTensor *)luaT_checkudata(L, 2, \"torch.CudaTensor\");\n// THCudaTensor *grids = (THCudaTensor *)luaT_checkudata(L, 3, \"torch.CudaTensor\");\n// THCudaTensor *gradInputImages = (THCudaTensor *)luaT_checkudata(L, 4, \"torch.CudaTensor\");\n// THCudaTensor *gradGrids = (THCudaTensor *)luaT_checkudata(L, 5, \"torch.CudaTensor\");\n// THCudaTensor *gradOutput = (THCudaTensor *)luaT_checkudata(L, 6, \"torch.CudaTensor\");\n\n int success = 0;\n success = BilinearSamplerBHWD_updateGradInput_cuda_kernel(THCudaTensor_size(state, gradOutput, 1),\n\t\t\t\t\t\t\t THCudaTensor_size(state, gradOutput, 3),\n\t\t\t\t\t\t\t THCudaTensor_size(state, gradOutput, 2),\n\t\t\t\t\t\t\t THCudaTensor_size(state, gradOutput, 0),\n\t\t\t\t\t\t\t THCudaTensor_size(state, inputImages, 1),\n\t\t\t\t\t\t\t THCudaTensor_size(state, inputImages, 2),\n\t\t\t\t\t\t\t THCudaTensor_size(state, inputImages, 3),\n\t\t\t\t\t\t\t THCudaTensor_size(state, inputImages, 0),\n\t\t\t\t\t\t\t THCudaTensor_data(state, inputImages),\n\t\t\t\t\t\t\t THCudaTensor_stride(state, inputImages, 0),\n\t\t\t\t\t\t\t THCudaTensor_stride(state, inputImages, 1),\n\t\t\t\t\t\t\t THCudaTensor_stride(state, inputImages, 2),\n\t\t\t\t\t\t\t THCudaTensor_stride(state, inputImages, 3),\n\t\t\t\t\t\t\t THCudaTensor_data(state, grids),\n\t\t\t\t\t\t\t THCudaTensor_stride(state, grids, 0),\n\t\t\t\t\t\t\t THCudaTensor_stride(state, grids, 3),\n\t\t\t\t\t\t\t THCudaTensor_stride(state, grids, 1),\n\t\t\t\t\t\t\t THCudaTensor_stride(state, grids, 2),\n\t\t\t\t\t\t\t THCudaTensor_data(state, gradInputImages),\n\t\t\t\t\t\t\t THCudaTensor_stride(state, gradInputImages, 0),\n\t\t\t\t\t\t\t THCudaTensor_stride(state, gradInputImages, 1),\n\t\t\t\t\t\t\t THCudaTensor_stride(state, gradInputImages, 2),\n\t\t\t\t\t\t\t THCudaTensor_stride(state, gradInputImages, 3),\n\t\t\t\t\t\t\t THCudaTensor_data(state, gradGrids),\n\t\t\t\t\t\t\t THCudaTensor_stride(state, gradGrids, 0),\n\t\t\t\t\t\t\t THCudaTensor_stride(state, gradGrids, 3),\n\t\t\t\t\t\t\t THCudaTensor_stride(state, gradGrids, 1),\n\t\t\t\t\t\t\t THCudaTensor_stride(state, gradGrids, 2),\n\t\t\t\t\t\t\t THCudaTensor_data(state, gradOutput),\n\t\t\t\t\t\t\t THCudaTensor_stride(state, gradOutput, 0),\n\t\t\t\t\t\t\t THCudaTensor_stride(state, gradOutput, 1),\n\t\t\t\t\t\t\t THCudaTensor_stride(state, gradOutput, 2),\n\t\t\t\t\t\t\t THCudaTensor_stride(state, gradOutput, 3),\n\t\t\t\t\t\t\t THCState_getCurrentStream(state));\n\n //check for errors\n if (!success) {\n THError(\"aborting\");\n }\n return 1;\n}\n"
},
{
"path": "lib/model/roi_crop/src/roi_crop_cuda.h",
"content": "// Bilinear sampling is done in BHWD (coalescing is not obvious in BDHW)\n// we assume BHWD format in inputImages\n// we assume BHW(YX) format on grids\n\nint BilinearSamplerBHWD_updateOutput_cuda(THCudaTensor *inputImages, THCudaTensor *grids, THCudaTensor *output);\n\nint BilinearSamplerBHWD_updateGradInput_cuda(THCudaTensor *inputImages, THCudaTensor *grids, THCudaTensor *gradInputImages,\n THCudaTensor *gradGrids, THCudaTensor *gradOutput);\n"
},
{
"path": "lib/model/roi_crop/src/roi_crop_cuda_kernel.cu",
"content": "#include \n#include \n#include \"roi_crop_cuda_kernel.h\"\n\n#define real float\n\n// Bilinear sampling is done in BHWD (coalescing is not obvious in BDHW)\n// we assume BHWD format in inputImages\n// we assume BHW(YX) format on grids\n\n__device__ void getTopLeft(float x, int width, int& point, float& weight)\n{\n /* for interpolation :\n stores in point and weight :\n - the x-coordinate of the pixel on the left (or y-coordinate of the upper pixel)\n - the weight for interpolating\n */\n\n float xcoord = (x + 1) * (width - 1) / 2;\n point = floor(xcoord);\n weight = 1 - (xcoord - point);\n}\n\n__device__ bool between(int value, int lowerBound, int upperBound)\n{\n return (value >= lowerBound && value <= upperBound);\n}\n\n__device__ void sumReduceShMem(volatile float s[])\n{\n /* obviously only works for 32 elements */\n /* sums up a shared memory array of 32 elements, stores it in s[0] */\n /* whole warp can then read first element (broadcasting) */\n if(threadIdx.x<16) { s[threadIdx.x] = s[threadIdx.x] + s[threadIdx.x+16]; }\n if(threadIdx.x<8) { s[threadIdx.x] = s[threadIdx.x] + s[threadIdx.x+8]; }\n if(threadIdx.x<4) { s[threadIdx.x] = s[threadIdx.x] + s[threadIdx.x+4]; }\n if(threadIdx.x<2) { s[threadIdx.x] = s[threadIdx.x] + s[threadIdx.x+2]; }\n if(threadIdx.x<1) { s[threadIdx.x] = s[threadIdx.x] + s[threadIdx.x+1]; }\n}\n\n// CUDA: grid stride looping\n#define CUDA_KERNEL_LOOP(i, n) \\\n for (int i = blockIdx.x * blockDim.x + threadIdx.x; \\\n i < (n); \\\n i += blockDim.x * gridDim.x)\n\n__global__ void bilinearSamplingFromGrid(const int nthreads, float* inputImages_data, int inputImages_strideBatch, int inputImages_strideChannels, int inputImages_strideHeight, int inputImages_strideWidth,\n float* grids_data, int grids_strideBatch, int grids_strideYX, int grids_strideHeight, int grids_strideWidth,\n float* output_data, int output_strideBatch, int output_strideChannels, int output_strideHeight, int output_strideWidth,\n int inputImages_channels, int inputImages_height, int inputImages_width,\n int output_channels, int output_height, int output_width, int output_batchsize,\n int roiPerImage)\n{\n CUDA_KERNEL_LOOP(index, nthreads)\n {\n const int xOut = index % output_width;\n const int yOut = (index / output_width) % output_height;\n const int cOut = (index / output_width / output_height) % output_channels;\n const int b = index / output_width / output_height / output_channels;\n\n const int width = inputImages_width;\n const int height = inputImages_height;\n\n const int b_input = b / roiPerImage;\n\n float yf = grids_data[b*grids_strideBatch + yOut*grids_strideHeight + xOut*grids_strideWidth];\n float xf = grids_data[b*grids_strideBatch + yOut*grids_strideHeight + xOut*grids_strideWidth + 1];\n\n int yInTopLeft, xInTopLeft;\n float yWeightTopLeft, xWeightTopLeft;\n getTopLeft(xf, inputImages_width, xInTopLeft, xWeightTopLeft);\n getTopLeft(yf, inputImages_height, yInTopLeft, yWeightTopLeft);\n\n // const int outAddress = output_strideBatch * b + output_strideHeight * yOut + output_strideWidth * xOut;\n const int outAddress = output_strideBatch * b + output_strideChannels * cOut + output_strideHeight * yOut + xOut;\n\n const int inTopLeftAddress = inputImages_strideBatch * b_input + inputImages_strideChannels * cOut + inputImages_strideHeight * yInTopLeft + xInTopLeft;\n const int inTopRightAddress = inTopLeftAddress + inputImages_strideWidth;\n const int inBottomLeftAddress = inTopLeftAddress + inputImages_strideHeight;\n const int inBottomRightAddress = inBottomLeftAddress + inputImages_strideWidth;\n\n float v=0;\n float inTopLeft=0;\n float inTopRight=0;\n float inBottomLeft=0;\n float inBottomRight=0;\n\n bool topLeftIsIn = between(xInTopLeft, 0, width-1) && between(yInTopLeft, 0, height-1);\n bool topRightIsIn = between(xInTopLeft+1, 0, width-1) && between(yInTopLeft, 0, height-1);\n bool bottomLeftIsIn = between(xInTopLeft, 0, width-1) && between(yInTopLeft+1, 0, height-1);\n bool bottomRightIsIn = between(xInTopLeft+1, 0, width-1) && between(yInTopLeft+1, 0, height-1);\n\n if (!topLeftIsIn && !topRightIsIn && !bottomLeftIsIn && !bottomRightIsIn)\n continue;\n\n if(topLeftIsIn) inTopLeft = inputImages_data[inTopLeftAddress];\n if(topRightIsIn) inTopRight = inputImages_data[inTopRightAddress];\n if(bottomLeftIsIn) inBottomLeft = inputImages_data[inBottomLeftAddress];\n if(bottomRightIsIn) inBottomRight = inputImages_data[inBottomRightAddress];\n\n v = xWeightTopLeft * yWeightTopLeft * inTopLeft\n + (1 - xWeightTopLeft) * yWeightTopLeft * inTopRight\n + xWeightTopLeft * (1 - yWeightTopLeft) * inBottomLeft\n + (1 - xWeightTopLeft) * (1 - yWeightTopLeft) * inBottomRight;\n\n output_data[outAddress] = v;\n }\n\n}\n\n__global__ void backwardBilinearSampling(const int nthreads, float* inputImages_data, int inputImages_strideBatch, int inputImages_strideChannels, int inputImages_strideHeight, int inputImages_strideWidth,\n float* gradInputImages_data, int gradInputImages_strideBatch, int gradInputImages_strideChannels, int gradInputImages_strideHeight, int gradInputImages_strideWidth,\n float* grids_data, int grids_strideBatch, int grids_strideYX, int grids_strideHeight, int grids_strideWidth,\n float* gradGrids_data, int gradGrids_strideBatch, int gradGrids_strideYX, int gradGrids_strideHeight, int gradGrids_strideWidth,\n float* gradOutput_data, int gradOutput_strideBatch, int gradOutput_strideChannels, int gradOutput_strideHeight, int gradOutput_strideWidth,\n int inputImages_channels, int inputImages_height, int inputImages_width,\n int gradOutput_channels, int gradOutput_height, int gradOutput_width, int gradOutput_batchsize,\n int roiPerImage)\n{\n\n CUDA_KERNEL_LOOP(index, nthreads)\n {\n const int xOut = index % gradOutput_width;\n const int yOut = (index / gradOutput_width) % gradOutput_height;\n const int cOut = (index / gradOutput_width / gradOutput_height) % gradOutput_channels;\n const int b = index / gradOutput_width / gradOutput_height / gradOutput_channels;\n\n const int b_input = b / roiPerImage;\n\n const int width = inputImages_width;\n const int height = inputImages_height;\n\n float yf = grids_data[b*grids_strideBatch + yOut*grids_strideHeight + xOut*grids_strideWidth];\n float xf = grids_data[b*grids_strideBatch + yOut*grids_strideHeight + xOut*grids_strideWidth + 1];\n\n int yInTopLeft, xInTopLeft;\n float yWeightTopLeft, xWeightTopLeft;\n getTopLeft(xf, inputImages_width, xInTopLeft, xWeightTopLeft);\n getTopLeft(yf, inputImages_height, yInTopLeft, yWeightTopLeft);\n\n const int inTopLeftAddress = inputImages_strideBatch * b_input + inputImages_strideChannels * cOut + inputImages_strideHeight * yInTopLeft + xInTopLeft;\n const int inTopRightAddress = inTopLeftAddress + inputImages_strideWidth;\n const int inBottomLeftAddress = inTopLeftAddress + inputImages_strideHeight;\n const int inBottomRightAddress = inBottomLeftAddress + inputImages_strideWidth;\n\n const int gradInputImagesTopLeftAddress = gradInputImages_strideBatch * b_input + gradInputImages_strideChannels * cOut\n + gradInputImages_strideHeight * yInTopLeft + xInTopLeft;\n const int gradInputImagesTopRightAddress = gradInputImagesTopLeftAddress + gradInputImages_strideWidth;\n const int gradInputImagesBottomLeftAddress = gradInputImagesTopLeftAddress + gradInputImages_strideHeight;\n const int gradInputImagesBottomRightAddress = gradInputImagesBottomLeftAddress + gradInputImages_strideWidth;\n\n const int gradOutputAddress = gradOutput_strideBatch * b + gradOutput_strideChannels * cOut + gradOutput_strideHeight * yOut + xOut;\n\n float topLeftDotProduct = 0;\n float topRightDotProduct = 0;\n float bottomLeftDotProduct = 0;\n float bottomRightDotProduct = 0;\n\n bool topLeftIsIn = between(xInTopLeft, 0, width-1) && between(yInTopLeft, 0, height-1);\n bool topRightIsIn = between(xInTopLeft+1, 0, width-1) && between(yInTopLeft, 0, height-1);\n bool bottomLeftIsIn = between(xInTopLeft, 0, width-1) && between(yInTopLeft+1, 0, height-1);\n bool bottomRightIsIn = between(xInTopLeft+1, 0, width-1) && between(yInTopLeft+1, 0, height-1);\n\n float gradOutValue = gradOutput_data[gradOutputAddress];\n // bool between(int value, int lowerBound, int upperBound)\n if(topLeftIsIn)\n {\n float inTopLeft = inputImages_data[inTopLeftAddress];\n topLeftDotProduct += inTopLeft * gradOutValue;\n atomicAdd(&gradInputImages_data[gradInputImagesTopLeftAddress], xWeightTopLeft * yWeightTopLeft * gradOutValue);\n }\n\n if(topRightIsIn)\n {\n float inTopRight = inputImages_data[inTopRightAddress];\n topRightDotProduct += inTopRight * gradOutValue;\n atomicAdd(&gradInputImages_data[gradInputImagesTopRightAddress], (1 - xWeightTopLeft) * yWeightTopLeft * gradOutValue);\n }\n\n if(bottomLeftIsIn)\n {\n float inBottomLeft = inputImages_data[inBottomLeftAddress];\n bottomLeftDotProduct += inBottomLeft * gradOutValue;\n atomicAdd(&gradInputImages_data[gradInputImagesBottomLeftAddress], xWeightTopLeft * (1 - yWeightTopLeft) * gradOutValue);\n }\n\n if(bottomRightIsIn)\n {\n float inBottomRight = inputImages_data[inBottomRightAddress];\n bottomRightDotProduct += inBottomRight * gradOutValue;\n atomicAdd(&gradInputImages_data[gradInputImagesBottomRightAddress], (1 - xWeightTopLeft) * (1 - yWeightTopLeft) * gradOutValue);\n }\n }\n}\n\n\n#ifdef __cplusplus\nextern \"C\" {\n#endif\n\nint BilinearSamplerBHWD_updateOutput_cuda_kernel(/*output->size[1]*/int oc,\n /*output->size[3]*/int ow,\n /*output->size[2]*/int oh,\n /*output->size[0]*/int ob,\n /*THCudaTensor_size(state, inputImages, 1)*/int ic,\n /*THCudaTensor_size(state, inputImages, 2)*/int ih,\n /*THCudaTensor_size(state, inputImages, 3)*/int iw,\n /*THCudaTensor_size(state, inputImages, 0)*/int ib,\n /*THCudaTensor *inputImages*/float *inputImages, int isb, int isc, int ish, int isw,\n /*THCudaTensor *grids*/float *grids, int gsb, int gsc, int gsh, int gsw,\n /*THCudaTensor *output*/float *output, int osb, int osc, int osh, int osw,\n /*THCState_getCurrentStream(state)*/cudaStream_t stream)\n{\n const int kThreadsPerBlock = 1024;\n int output_size = ob * oh * ow * oc;\n cudaError_t err;\n int roiPerImage = ob / ib;\n\n // printf(\"forward pass\\n\");\n\n bilinearSamplingFromGrid<<<(output_size + kThreadsPerBlock - 1) / kThreadsPerBlock, kThreadsPerBlock, 0, stream>>>(\n output_size,\n /*THCudaTensor_data(state, inputImages)*/inputImages,\n /*THCudaTensor_stride(state, inputImages, 0)*/isb,\n /*THCudaTensor_stride(state, inputImages, 3)*/isc,\n /*THCudaTensor_stride(state, inputImages, 1)*/ish,\n /*THCudaTensor_stride(state, inputImages, 2)*/isw,\n /*THCudaTensor_data(state, grids)*/grids,\n /*THCudaTensor_stride(state, grids, 0)*/gsb,\n /*THCudaTensor_stride(state, grids, 3)*/gsc,\n /*THCudaTensor_stride(state, grids, 1)*/gsh,\n /*THCudaTensor_stride(state, grids, 2)*/gsw,\n /*THCudaTensor_data(state, output)*/output,\n /*THCudaTensor_stride(state, output, 0)*/osb,\n /*THCudaTensor_stride(state, output, 3)*/osc,\n /*THCudaTensor_stride(state, output, 1)*/osh,\n /*THCudaTensor_stride(state, output, 2)*/osw,\n /*THCudaTensor_size(state, inputImages, 3)*/ic,\n /*THCudaTensor_size(state, inputImages, 1)*/ih,\n /*THCudaTensor_size(state, inputImages, 2)*/iw,\n /*THCudaTensor_size(state, output, 3)*/oc,\n /*THCudaTensor_size(state, output, 1)*/oh,\n /*THCudaTensor_size(state, output, 2)*/ow,\n /*THCudaTensor_size(state, output, 0)*/ob,\n /*Number of rois per image*/roiPerImage);\n\n // check for errors\n err = cudaGetLastError();\n if (err != cudaSuccess) {\n printf(\"error in BilinearSampler.updateOutput: %s\\n\", cudaGetErrorString(err));\n //THError(\"aborting\");\n return 0;\n }\n return 1;\n}\n\nint BilinearSamplerBHWD_updateGradInput_cuda_kernel(/*gradOutput->size[1]*/int goc,\n /*gradOutput->size[3]*/int gow,\n /*gradOutput->size[2]*/int goh,\n /*gradOutput->size[0]*/int gob,\n /*THCudaTensor_size(state, inputImages, 1)*/int ic,\n /*THCudaTensor_size(state, inputImages, 2)*/int ih,\n /*THCudaTensor_size(state, inputImages, 3)*/int iw,\n /*THCudaTensor_size(state, inputImages, 0)*/int ib,\n /*THCudaTensor *inputImages*/float *inputImages, int isb, int isc, int ish, int isw,\n /*THCudaTensor *grids*/float *grids, int gsb, int gsc, int gsh, int gsw,\n /*THCudaTensor *gradInputImages*/float *gradInputImages, int gisb, int gisc, int gish, int gisw,\n /*THCudaTensor *gradGrids*/float *gradGrids, int ggsb, int ggsc, int ggsh, int ggsw,\n /*THCudaTensor *gradOutput*/float *gradOutput, int gosb, int gosc, int gosh, int gosw,\n /*THCState_getCurrentStream(state)*/cudaStream_t stream)\n{\n\n const int kThreadsPerBlock = 1024;\n int output_size = gob * goh * gow * goc;\n cudaError_t err;\n int roiPerImage = gob / ib;\n\n // printf(\"%d %d %d %d\\n\", gob, goh, gow, goc);\n // printf(\"%d %d %d %d\\n\", ib, ih, iw, ic);\n // printf(\"backward pass\\n\");\n\n backwardBilinearSampling<<<(output_size + kThreadsPerBlock - 1) / kThreadsPerBlock, kThreadsPerBlock, 0, stream>>>(\n output_size,\n /*THCudaTensor_data(state, inputImages)*/inputImages,\n /*THCudaTensor_stride(state, inputImages, 0)*/isb,\n /*THCudaTensor_stride(state, inputImages, 3)*/isc,\n /*THCudaTensor_stride(state, inputImages, 1)*/ish,\n /*THCudaTensor_stride(state, inputImages, 2)*/isw,\n /*THCudaTensor_data(state, gradInputImages)*/gradInputImages,\n /*THCudaTensor_stride(state, gradInputImages, 0)*/gisb,\n /*THCudaTensor_stride(state, gradInputImages, 3)*/gisc,\n /*THCudaTensor_stride(state, gradInputImages, 1)*/gish,\n /*THCudaTensor_stride(state, gradInputImages, 2)*/gisw,\n /*THCudaTensor_data(state, grids)*/grids,\n /*THCudaTensor_stride(state, grids, 0)*/gsb,\n /*THCudaTensor_stride(state, grids, 3)*/gsc,\n /*THCudaTensor_stride(state, grids, 1)*/gsh,\n /*THCudaTensor_stride(state, grids, 2)*/gsw,\n /*THCudaTensor_data(state, gradGrids)*/gradGrids,\n /*THCudaTensor_stride(state, gradGrids, 0)*/ggsb,\n /*THCudaTensor_stride(state, gradGrids, 3)*/ggsc,\n /*THCudaTensor_stride(state, gradGrids, 1)*/ggsh,\n /*THCudaTensor_stride(state, gradGrids, 2)*/ggsw,\n /*THCudaTensor_data(state, gradOutput)*/gradOutput,\n /*THCudaTensor_stride(state, gradOutput, 0)*/gosb,\n /*THCudaTensor_stride(state, gradOutput, 3)*/gosc,\n /*THCudaTensor_stride(state, gradOutput, 1)*/gosh,\n /*THCudaTensor_stride(state, gradOutput, 2)*/gosw,\n /*THCudaTensor_size(state, inputImages, 3)*/ic,\n /*THCudaTensor_size(state, inputImages, 1)*/ih,\n /*THCudaTensor_size(state, inputImages, 2)*/iw,\n /*THCudaTensor_size(state, gradOutput, 3)*/goc,\n /*THCudaTensor_size(state, gradOutput, 1)*/goh,\n /*THCudaTensor_size(state, gradOutput, 2)*/gow,\n /*THCudaTensor_size(state, gradOutput, 0)*/gob,\n /*Number of rois per image*/roiPerImage);\n\n // check for errors\n err = cudaGetLastError();\n if (err != cudaSuccess) {\n printf(\"error in BilinearSampler.updateGradInput: %s\\n\", cudaGetErrorString(err));\n //THError(\"aborting\");\n return 0;\n }\n return 1;\n}\n\n#ifdef __cplusplus\n}\n#endif\n"
},
{
"path": "lib/model/roi_crop/src/roi_crop_cuda_kernel.h",
"content": "#ifdef __cplusplus\nextern \"C\" {\n#endif\n\n\nint BilinearSamplerBHWD_updateOutput_cuda_kernel(/*output->size[3]*/int oc,\n /*output->size[2]*/int ow,\n /*output->size[1]*/int oh,\n /*output->size[0]*/int ob,\n /*THCudaTensor_size(state, inputImages, 3)*/int ic,\n /*THCudaTensor_size(state, inputImages, 1)*/int ih,\n /*THCudaTensor_size(state, inputImages, 2)*/int iw,\n /*THCudaTensor_size(state, inputImages, 0)*/int ib,\n /*THCudaTensor *inputImages*/float *inputImages, int isb, int isc, int ish, int isw,\n /*THCudaTensor *grids*/float *grids, int gsb, int gsc, int gsh, int gsw,\n /*THCudaTensor *output*/float *output, int osb, int osc, int osh, int osw,\n /*THCState_getCurrentStream(state)*/cudaStream_t stream);\n\nint BilinearSamplerBHWD_updateGradInput_cuda_kernel(/*gradOutput->size[3]*/int goc,\n /*gradOutput->size[2]*/int gow,\n /*gradOutput->size[1]*/int goh,\n /*gradOutput->size[0]*/int gob,\n /*THCudaTensor_size(state, inputImages, 3)*/int ic,\n /*THCudaTensor_size(state, inputImages, 1)*/int ih,\n /*THCudaTensor_size(state, inputImages, 2)*/int iw,\n /*THCudaTensor_size(state, inputImages, 0)*/int ib,\n /*THCudaTensor *inputImages*/float *inputImages, int isb, int isc, int ish, int isw,\n /*THCudaTensor *grids*/float *grids, int gsb, int gsc, int gsh, int gsw,\n /*THCudaTensor *gradInputImages*/float *gradInputImages, int gisb, int gisc, int gish, int gisw,\n /*THCudaTensor *gradGrids*/float *gradGrids, int ggsb, int ggsc, int ggsh, int ggsw,\n /*THCudaTensor *gradOutput*/float *gradOutput, int gosb, int gosc, int gosh, int gosw,\n /*THCState_getCurrentStream(state)*/cudaStream_t stream);\n\n\n#ifdef __cplusplus\n}\n#endif\n"
},
{
"path": "lib/model/roi_pooling/__init__.py",
"content": ""
},
{
"path": "lib/model/roi_pooling/_ext/__init__.py",
"content": ""
},
{
"path": "lib/model/roi_pooling/_ext/roi_pooling/__init__.py",
"content": "\nfrom torch.utils.ffi import _wrap_function\nfrom ._roi_pooling import lib as _lib, ffi as _ffi\n\n__all__ = []\ndef _import_symbols(locals):\n for symbol in dir(_lib):\n fn = getattr(_lib, symbol)\n if callable(fn):\n locals[symbol] = _wrap_function(fn, _ffi)\n else:\n locals[symbol] = fn\n __all__.append(symbol)\n\n_import_symbols(locals())\n"
},
{
"path": "lib/model/roi_pooling/build.py",
"content": "from __future__ import print_function\nimport os\nimport torch\nfrom torch.utils.ffi import create_extension\n\n\nsources = ['src/roi_pooling.c']\nheaders = ['src/roi_pooling.h']\nextra_objects = []\ndefines = []\nwith_cuda = False\n\nthis_file = os.path.dirname(os.path.realpath(__file__))\nprint(this_file)\n\nif torch.cuda.is_available():\n print('Including CUDA code.')\n sources += ['src/roi_pooling_cuda.c']\n headers += ['src/roi_pooling_cuda.h']\n defines += [('WITH_CUDA', None)]\n with_cuda = True\n extra_objects = ['src/roi_pooling.cu.o']\n extra_objects = [os.path.join(this_file, fname) for fname in extra_objects]\n\nffi = create_extension(\n '_ext.roi_pooling',\n headers=headers,\n sources=sources,\n define_macros=defines,\n relative_to=__file__,\n with_cuda=with_cuda,\n extra_objects=extra_objects\n)\n\nif __name__ == '__main__':\n ffi.build()\n"
},
{
"path": "lib/model/roi_pooling/functions/__init__.py",
"content": ""
},
{
"path": "lib/model/roi_pooling/functions/roi_pool.py",
"content": "import torch\nfrom torch.autograd import Function\nfrom .._ext import roi_pooling\nimport pdb\n\nclass RoIPoolFunction(Function):\n def __init__(ctx, pooled_height, pooled_width, spatial_scale):\n ctx.pooled_width = pooled_width\n ctx.pooled_height = pooled_height\n ctx.spatial_scale = spatial_scale\n ctx.feature_size = None\n\n def forward(ctx, features, rois): \n ctx.feature_size = features.size() \n batch_size, num_channels, data_height, data_width = ctx.feature_size\n num_rois = rois.size(0)\n output = features.new(num_rois, num_channels, ctx.pooled_height, ctx.pooled_width).zero_()\n ctx.argmax = features.new(num_rois, num_channels, ctx.pooled_height, ctx.pooled_width).zero_().int()\n ctx.rois = rois\n if not features.is_cuda:\n _features = features.permute(0, 2, 3, 1)\n roi_pooling.roi_pooling_forward(ctx.pooled_height, ctx.pooled_width, ctx.spatial_scale,\n _features, rois, output)\n else:\n roi_pooling.roi_pooling_forward_cuda(ctx.pooled_height, ctx.pooled_width, ctx.spatial_scale,\n features, rois, output, ctx.argmax)\n\n return output\n\n def backward(ctx, grad_output):\n assert(ctx.feature_size is not None and grad_output.is_cuda)\n batch_size, num_channels, data_height, data_width = ctx.feature_size\n grad_input = grad_output.new(batch_size, num_channels, data_height, data_width).zero_()\n\n roi_pooling.roi_pooling_backward_cuda(ctx.pooled_height, ctx.pooled_width, ctx.spatial_scale,\n grad_output, ctx.rois, grad_input, ctx.argmax)\n\n return grad_input, None\n"
},
{
"path": "lib/model/roi_pooling/modules/__init__.py",
"content": ""
},
{
"path": "lib/model/roi_pooling/modules/roi_pool.py",
"content": "from torch.nn.modules.module import Module\nfrom ..functions.roi_pool import RoIPoolFunction\n\n\nclass _RoIPooling(Module):\n def __init__(self, pooled_height, pooled_width, spatial_scale):\n super(_RoIPooling, self).__init__()\n\n self.pooled_width = int(pooled_width)\n self.pooled_height = int(pooled_height)\n self.spatial_scale = float(spatial_scale)\n\n def forward(self, features, rois):\n return RoIPoolFunction(self.pooled_height, self.pooled_width, self.spatial_scale)(features, rois)\n"
},
{
"path": "lib/model/roi_pooling/src/roi_pooling.c",
"content": "#include | \n#include \n\nint roi_pooling_forward(int pooled_height, int pooled_width, float spatial_scale,\n THFloatTensor * features, THFloatTensor * rois, THFloatTensor * output)\n{\n // Grab the input tensor\n float * data_flat = THFloatTensor_data(features);\n float * rois_flat = THFloatTensor_data(rois);\n\n float * output_flat = THFloatTensor_data(output);\n\n // Number of ROIs\n int num_rois = THFloatTensor_size(rois, 0);\n int size_rois = THFloatTensor_size(rois, 1);\n // batch size\n int batch_size = THFloatTensor_size(features, 0);\n if(batch_size != 1)\n {\n return 0;\n }\n // data height\n int data_height = THFloatTensor_size(features, 1);\n // data width\n int data_width = THFloatTensor_size(features, 2);\n // Number of channels\n int num_channels = THFloatTensor_size(features, 3);\n\n // Set all element of the output tensor to -inf.\n THFloatStorage_fill(THFloatTensor_storage(output), -1);\n\n // For each ROI R = [batch_index x1 y1 x2 y2]: max pool over R\n int index_roi = 0;\n int index_output = 0;\n int n;\n for (n = 0; n < num_rois; ++n)\n {\n int roi_batch_ind = rois_flat[index_roi + 0];\n int roi_start_w = round(rois_flat[index_roi + 1] * spatial_scale);\n int roi_start_h = round(rois_flat[index_roi + 2] * spatial_scale);\n int roi_end_w = round(rois_flat[index_roi + 3] * spatial_scale);\n int roi_end_h = round(rois_flat[index_roi + 4] * spatial_scale);\n // CHECK_GE(roi_batch_ind, 0);\n // CHECK_LT(roi_batch_ind, batch_size);\n\n int roi_height = fmaxf(roi_end_h - roi_start_h + 1, 1);\n int roi_width = fmaxf(roi_end_w - roi_start_w + 1, 1);\n float bin_size_h = (float)(roi_height) / (float)(pooled_height);\n float bin_size_w = (float)(roi_width) / (float)(pooled_width);\n\n int index_data = roi_batch_ind * data_height * data_width * num_channels;\n const int output_area = pooled_width * pooled_height;\n\n int c, ph, pw;\n for (ph = 0; ph < pooled_height; ++ph)\n {\n for (pw = 0; pw < pooled_width; ++pw)\n {\n int hstart = (floor((float)(ph) * bin_size_h));\n int wstart = (floor((float)(pw) * bin_size_w));\n int hend = (ceil((float)(ph + 1) * bin_size_h));\n int wend = (ceil((float)(pw + 1) * bin_size_w));\n\n hstart = fminf(fmaxf(hstart + roi_start_h, 0), data_height);\n hend = fminf(fmaxf(hend + roi_start_h, 0), data_height);\n wstart = fminf(fmaxf(wstart + roi_start_w, 0), data_width);\n wend = fminf(fmaxf(wend + roi_start_w, 0), data_width);\n\n const int pool_index = index_output + (ph * pooled_width + pw);\n int is_empty = (hend <= hstart) || (wend <= wstart);\n if (is_empty)\n {\n for (c = 0; c < num_channels * output_area; c += output_area)\n {\n output_flat[pool_index + c] = 0;\n }\n }\n else\n {\n int h, w, c;\n for (h = hstart; h < hend; ++h)\n {\n for (w = wstart; w < wend; ++w)\n {\n for (c = 0; c < num_channels; ++c)\n {\n const int index = (h * data_width + w) * num_channels + c;\n if (data_flat[index_data + index] > output_flat[pool_index + c * output_area])\n {\n output_flat[pool_index + c * output_area] = data_flat[index_data + index];\n }\n }\n }\n }\n }\n }\n }\n\n // Increment ROI index\n index_roi += size_rois;\n index_output += pooled_height * pooled_width * num_channels;\n }\n return 1;\n}"
},
{
"path": "lib/model/roi_pooling/src/roi_pooling.h",
"content": "int roi_pooling_forward(int pooled_height, int pooled_width, float spatial_scale,\n THFloatTensor * features, THFloatTensor * rois, THFloatTensor * output);"
},
{
"path": "lib/model/roi_pooling/src/roi_pooling_cuda.c",
"content": "#include \n#include \n#include \"roi_pooling_kernel.h\"\n\nextern THCState *state;\n\nint roi_pooling_forward_cuda(int pooled_height, int pooled_width, float spatial_scale,\n THCudaTensor * features, THCudaTensor * rois, THCudaTensor * output, THCudaIntTensor * argmax)\n{\n // Grab the input tensor\n float * data_flat = THCudaTensor_data(state, features);\n float * rois_flat = THCudaTensor_data(state, rois);\n\n float * output_flat = THCudaTensor_data(state, output);\n int * argmax_flat = THCudaIntTensor_data(state, argmax);\n\n // Number of ROIs\n int num_rois = THCudaTensor_size(state, rois, 0);\n int size_rois = THCudaTensor_size(state, rois, 1);\n if (size_rois != 5)\n {\n return 0;\n }\n\n // batch size\n // int batch_size = THCudaTensor_size(state, features, 0);\n // if (batch_size != 1)\n // {\n // return 0;\n // }\n // data height\n int data_height = THCudaTensor_size(state, features, 2);\n // data width\n int data_width = THCudaTensor_size(state, features, 3);\n // Number of channels\n int num_channels = THCudaTensor_size(state, features, 1);\n\n cudaStream_t stream = THCState_getCurrentStream(state);\n\n ROIPoolForwardLaucher(\n data_flat, spatial_scale, num_rois, data_height,\n data_width, num_channels, pooled_height,\n pooled_width, rois_flat,\n output_flat, argmax_flat, stream);\n\n return 1;\n}\n\nint roi_pooling_backward_cuda(int pooled_height, int pooled_width, float spatial_scale,\n THCudaTensor * top_grad, THCudaTensor * rois, THCudaTensor * bottom_grad, THCudaIntTensor * argmax)\n{\n // Grab the input tensor\n float * top_grad_flat = THCudaTensor_data(state, top_grad);\n float * rois_flat = THCudaTensor_data(state, rois);\n\n float * bottom_grad_flat = THCudaTensor_data(state, bottom_grad);\n int * argmax_flat = THCudaIntTensor_data(state, argmax);\n\n // Number of ROIs\n int num_rois = THCudaTensor_size(state, rois, 0);\n int size_rois = THCudaTensor_size(state, rois, 1);\n if (size_rois != 5)\n {\n return 0;\n }\n\n // batch size\n int batch_size = THCudaTensor_size(state, bottom_grad, 0);\n // if (batch_size != 1)\n // {\n // return 0;\n // }\n // data height\n int data_height = THCudaTensor_size(state, bottom_grad, 2);\n // data width\n int data_width = THCudaTensor_size(state, bottom_grad, 3);\n // Number of channels\n int num_channels = THCudaTensor_size(state, bottom_grad, 1);\n\n cudaStream_t stream = THCState_getCurrentStream(state);\n ROIPoolBackwardLaucher(\n top_grad_flat, spatial_scale, batch_size, num_rois, data_height,\n data_width, num_channels, pooled_height,\n pooled_width, rois_flat,\n bottom_grad_flat, argmax_flat, stream);\n\n return 1;\n}\n"
},
{
"path": "lib/model/roi_pooling/src/roi_pooling_cuda.h",
"content": "int roi_pooling_forward_cuda(int pooled_height, int pooled_width, float spatial_scale,\n THCudaTensor * features, THCudaTensor * rois, THCudaTensor * output, THCudaIntTensor * argmax);\n\nint roi_pooling_backward_cuda(int pooled_height, int pooled_width, float spatial_scale,\n THCudaTensor * top_grad, THCudaTensor * rois, THCudaTensor * bottom_grad, THCudaIntTensor * argmax);"
},
{
"path": "lib/model/roi_pooling/src/roi_pooling_kernel.cu",
"content": "// #ifdef __cplusplus\n// extern \"C\" {\n// #endif\n\n#include \n#include \n#include \n#include \n#include \"roi_pooling_kernel.h\"\n\n\n#define DIVUP(m, n) ((m) / (m) + ((m) % (n) > 0))\n\n#define CUDA_1D_KERNEL_LOOP(i, n) \\\n for (int i = blockIdx.x * blockDim.x + threadIdx.x; i < n; \\\n i += blockDim.x * gridDim.x)\n\n// CUDA: grid stride looping\n#define CUDA_KERNEL_LOOP(i, n) \\\n for (int i = blockIdx.x * blockDim.x + threadIdx.x; \\\n i < (n); \\\n i += blockDim.x * gridDim.x)\n\n__global__ void ROIPoolForward(const int nthreads, const float* bottom_data,\n const float spatial_scale, const int height, const int width,\n const int channels, const int pooled_height, const int pooled_width,\n const float* bottom_rois, float* top_data, int* argmax_data)\n{\n CUDA_KERNEL_LOOP(index, nthreads)\n {\n // (n, c, ph, pw) is an element in the pooled output\n // int n = index;\n // int pw = n % pooled_width;\n // n /= pooled_width;\n // int ph = n % pooled_height;\n // n /= pooled_height;\n // int c = n % channels;\n // n /= channels;\n int pw = index % pooled_width;\n int ph = (index / pooled_width) % pooled_height;\n int c = (index / pooled_width / pooled_height) % channels;\n int n = index / pooled_width / pooled_height / channels;\n\n // bottom_rois += n * 5;\n int roi_batch_ind = bottom_rois[n * 5 + 0];\n int roi_start_w = round(bottom_rois[n * 5 + 1] * spatial_scale);\n int roi_start_h = round(bottom_rois[n * 5 + 2] * spatial_scale);\n int roi_end_w = round(bottom_rois[n * 5 + 3] * spatial_scale);\n int roi_end_h = round(bottom_rois[n * 5 + 4] * spatial_scale);\n\n // Force malformed ROIs to be 1x1\n int roi_width = fmaxf(roi_end_w - roi_start_w + 1, 1);\n int roi_height = fmaxf(roi_end_h - roi_start_h + 1, 1);\n float bin_size_h = (float)(roi_height) / (float)(pooled_height);\n float bin_size_w = (float)(roi_width) / (float)(pooled_width);\n\n int hstart = (int)(floor((float)(ph) * bin_size_h));\n int wstart = (int)(floor((float)(pw) * bin_size_w));\n int hend = (int)(ceil((float)(ph + 1) * bin_size_h));\n int wend = (int)(ceil((float)(pw + 1) * bin_size_w));\n\n // Add roi offsets and clip to input boundaries\n hstart = fminf(fmaxf(hstart + roi_start_h, 0), height);\n hend = fminf(fmaxf(hend + roi_start_h, 0), height);\n wstart = fminf(fmaxf(wstart + roi_start_w, 0), width);\n wend = fminf(fmaxf(wend + roi_start_w, 0), width);\n bool is_empty = (hend <= hstart) || (wend <= wstart);\n\n // Define an empty pooling region to be zero\n float maxval = is_empty ? 0 : -FLT_MAX;\n // If nothing is pooled, argmax = -1 causes nothing to be backprop'd\n int maxidx = -1;\n // bottom_data += roi_batch_ind * channels * height * width;\n\n int bottom_data_batch_offset = roi_batch_ind * channels * height * width;\n int bottom_data_offset = bottom_data_batch_offset + c * height * width;\n\n for (int h = hstart; h < hend; ++h) {\n for (int w = wstart; w < wend; ++w) {\n // int bottom_index = (h * width + w) * channels + c;\n // int bottom_index = (c * height + h) * width + w;\n int bottom_index = h * width + w;\n if (bottom_data[bottom_data_offset + bottom_index] > maxval) {\n maxval = bottom_data[bottom_data_offset + bottom_index];\n maxidx = bottom_data_offset + bottom_index;\n }\n }\n }\n top_data[index] = maxval;\n if (argmax_data != NULL)\n argmax_data[index] = maxidx;\n }\n}\n\nint ROIPoolForwardLaucher(\n const float* bottom_data, const float spatial_scale, const int num_rois, const int height,\n const int width, const int channels, const int pooled_height,\n const int pooled_width, const float* bottom_rois,\n float* top_data, int* argmax_data, cudaStream_t stream)\n{\n const int kThreadsPerBlock = 1024;\n int output_size = num_rois * pooled_height * pooled_width * channels;\n cudaError_t err;\n\n ROIPoolForward<<<(output_size + kThreadsPerBlock - 1) / kThreadsPerBlock, kThreadsPerBlock, 0, stream>>>(\n output_size, bottom_data, spatial_scale, height, width, channels, pooled_height,\n pooled_width, bottom_rois, top_data, argmax_data);\n\n // dim3 blocks(DIVUP(output_size, kThreadsPerBlock),\n // DIVUP(output_size, kThreadsPerBlock));\n // dim3 threads(kThreadsPerBlock);\n //\n // ROIPoolForward<<>>(\n // output_size, bottom_data, spatial_scale, height, width, channels, pooled_height,\n // pooled_width, bottom_rois, top_data, argmax_data);\n\n err = cudaGetLastError();\n if(cudaSuccess != err)\n {\n fprintf( stderr, \"cudaCheckError() failed : %s\\n\", cudaGetErrorString( err ) );\n exit( -1 );\n }\n\n return 1;\n}\n\n\n__global__ void ROIPoolBackward(const int nthreads, const float* top_diff,\n const int* argmax_data, const int num_rois, const float spatial_scale,\n const int height, const int width, const int channels,\n const int pooled_height, const int pooled_width, float* bottom_diff,\n const float* bottom_rois) {\n CUDA_1D_KERNEL_LOOP(index, nthreads)\n {\n\n // (n, c, ph, pw) is an element in the pooled output\n int n = index;\n int w = n % width;\n n /= width;\n int h = n % height;\n n /= height;\n int c = n % channels;\n n /= channels;\n\n float gradient = 0;\n // Accumulate gradient over all ROIs that pooled this element\n for (int roi_n = 0; roi_n < num_rois; ++roi_n)\n {\n const float* offset_bottom_rois = bottom_rois + roi_n * 5;\n int roi_batch_ind = offset_bottom_rois[0];\n // Skip if ROI's batch index doesn't match n\n if (n != roi_batch_ind) {\n continue;\n }\n\n int roi_start_w = round(offset_bottom_rois[1] * spatial_scale);\n int roi_start_h = round(offset_bottom_rois[2] * spatial_scale);\n int roi_end_w = round(offset_bottom_rois[3] * spatial_scale);\n int roi_end_h = round(offset_bottom_rois[4] * spatial_scale);\n\n // Skip if ROI doesn't include (h, w)\n const bool in_roi = (w >= roi_start_w && w <= roi_end_w &&\n h >= roi_start_h && h <= roi_end_h);\n if (!in_roi) {\n continue;\n }\n\n int offset = roi_n * pooled_height * pooled_width * channels;\n const float* offset_top_diff = top_diff + offset;\n const int* offset_argmax_data = argmax_data + offset;\n\n // Compute feasible set of pooled units that could have pooled\n // this bottom unit\n\n // Force malformed ROIs to be 1x1\n int roi_width = fmaxf(roi_end_w - roi_start_w + 1, 1);\n int roi_height = fmaxf(roi_end_h - roi_start_h + 1, 1);\n\n float bin_size_h = (float)(roi_height) / (float)(pooled_height);\n float bin_size_w = (float)(roi_width) / (float)(pooled_width);\n\n int phstart = floor((float)(h - roi_start_h) / bin_size_h);\n int phend = ceil((float)(h - roi_start_h + 1) / bin_size_h);\n int pwstart = floor((float)(w - roi_start_w) / bin_size_w);\n int pwend = ceil((float)(w - roi_start_w + 1) / bin_size_w);\n\n phstart = fminf(fmaxf(phstart, 0), pooled_height);\n phend = fminf(fmaxf(phend, 0), pooled_height);\n pwstart = fminf(fmaxf(pwstart, 0), pooled_width);\n pwend = fminf(fmaxf(pwend, 0), pooled_width);\n\n for (int ph = phstart; ph < phend; ++ph) {\n for (int pw = pwstart; pw < pwend; ++pw) {\n if (offset_argmax_data[(c * pooled_height + ph) * pooled_width + pw] == index)\n {\n gradient += offset_top_diff[(c * pooled_height + ph) * pooled_width + pw];\n }\n }\n }\n }\n bottom_diff[index] = gradient;\n }\n}\n\nint ROIPoolBackwardLaucher(const float* top_diff, const float spatial_scale, const int batch_size, const int num_rois,\n const int height, const int width, const int channels, const int pooled_height,\n const int pooled_width, const float* bottom_rois,\n float* bottom_diff, const int* argmax_data, cudaStream_t stream)\n{\n const int kThreadsPerBlock = 1024;\n int output_size = batch_size * height * width * channels;\n cudaError_t err;\n\n ROIPoolBackward<<<(output_size + kThreadsPerBlock - 1) / kThreadsPerBlock, kThreadsPerBlock, 0, stream>>>(\n output_size, top_diff, argmax_data, num_rois, spatial_scale, height, width, channels, pooled_height,\n pooled_width, bottom_diff, bottom_rois);\n\n // dim3 blocks(DIVUP(output_size, kThreadsPerBlock),\n // DIVUP(output_size, kThreadsPerBlock));\n // dim3 threads(kThreadsPerBlock);\n //\n // ROIPoolBackward<<>>(\n // output_size, top_diff, argmax_data, num_rois, spatial_scale, height, width, channels, pooled_height,\n // pooled_width, bottom_diff, bottom_rois);\n\n err = cudaGetLastError();\n if(cudaSuccess != err)\n {\n fprintf( stderr, \"cudaCheckError() failed : %s\\n\", cudaGetErrorString( err ) );\n exit( -1 );\n }\n\n return 1;\n}\n\n\n// #ifdef __cplusplus\n// }\n// #endif\n"
},
{
"path": "lib/model/roi_pooling/src/roi_pooling_kernel.h",
"content": "#ifndef _ROI_POOLING_KERNEL\n#define _ROI_POOLING_KERNEL\n\n#ifdef __cplusplus\nextern \"C\" {\n#endif\n\nint ROIPoolForwardLaucher(\n const float* bottom_data, const float spatial_scale, const int num_rois, const int height,\n const int width, const int channels, const int pooled_height,\n const int pooled_width, const float* bottom_rois,\n float* top_data, int* argmax_data, cudaStream_t stream);\n\n\nint ROIPoolBackwardLaucher(const float* top_diff, const float spatial_scale, const int batch_size, const int num_rois,\n const int height, const int width, const int channels, const int pooled_height,\n const int pooled_width, const float* bottom_rois,\n float* bottom_diff, const int* argmax_data, cudaStream_t stream);\n\n#ifdef __cplusplus\n}\n#endif\n\n#endif\n\n"
},
{
"path": "lib/model/rpn/__init__.py",
"content": ""
},
{
"path": "lib/model/rpn/anchor_target_layer.py",
"content": "from __future__ import absolute_import\n# --------------------------------------------------------\n# Faster R-CNN\n# Copyright (c) 2015 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Written by Ross Girshick and Sean Bell\n# --------------------------------------------------------\n# --------------------------------------------------------\n# Reorganized and modified by Jianwei Yang and Jiasen Lu\n# --------------------------------------------------------\n\nimport torch\nimport torch.nn as nn\nimport numpy as np\nimport numpy.random as npr\n\nfrom model.utils.config import cfg\nfrom .generate_anchors import generate_anchors\nfrom .bbox_transform import clip_boxes, bbox_overlaps_batch, bbox_transform_batch\n\nimport pdb\n\nDEBUG = False\n\ntry:\n long # Python 2\nexcept NameError:\n long = int # Python 3\n\n\nclass _AnchorTargetLayer(nn.Module):\n \"\"\"\n Assign anchors to ground-truth targets. Produces anchor classification\n labels and bounding-box regression targets.\n \"\"\"\n def __init__(self, feat_stride, scales, ratios):\n super(_AnchorTargetLayer, self).__init__()\n\n self._feat_stride = feat_stride\n self._scales = scales\n anchor_scales = scales\n self._anchors = torch.from_numpy(generate_anchors(scales=np.array(anchor_scales), ratios=np.array(ratios))).float()\n self._num_anchors = self._anchors.size(0)\n\n # allow boxes to sit over the edge by a small amount\n self._allowed_border = 0 # default is 0\n\n def forward(self, input):\n # Algorithm:\n #\n # for each (H, W) location i\n # generate 9 anchor boxes centered on cell i\n # apply predicted bbox deltas at cell i to each of the 9 anchors\n # filter out-of-image anchors\n\n rpn_cls_score = input[0]\n gt_boxes = input[1]\n im_info = input[2]\n num_boxes = input[3]\n\n # map of shape (..., H, W)\n height, width = rpn_cls_score.size(2), rpn_cls_score.size(3)\n\n batch_size = gt_boxes.size(0)\n\n feat_height, feat_width = rpn_cls_score.size(2), rpn_cls_score.size(3)\n shift_x = np.arange(0, feat_width) * self._feat_stride\n shift_y = np.arange(0, feat_height) * self._feat_stride\n shift_x, shift_y = np.meshgrid(shift_x, shift_y)\n shifts = torch.from_numpy(np.vstack((shift_x.ravel(), shift_y.ravel(),\n shift_x.ravel(), shift_y.ravel())).transpose())\n shifts = shifts.contiguous().type_as(rpn_cls_score).float()\n\n A = self._num_anchors\n K = shifts.size(0)\n\n self._anchors = self._anchors.type_as(gt_boxes) # move to specific gpu.\n all_anchors = self._anchors.view(1, A, 4) + shifts.view(K, 1, 4)\n all_anchors = all_anchors.view(K * A, 4)\n\n total_anchors = int(K * A)\n\n keep = ((all_anchors[:, 0] >= -self._allowed_border) &\n (all_anchors[:, 1] >= -self._allowed_border) &\n (all_anchors[:, 2] < long(im_info[0][1]) + self._allowed_border) &\n (all_anchors[:, 3] < long(im_info[0][0]) + self._allowed_border))\n\n inds_inside = torch.nonzero(keep).view(-1)\n\n # keep only inside anchors\n anchors = all_anchors[inds_inside, :]\n\n # label: 1 is positive, 0 is negative, -1 is dont care\n labels = gt_boxes.new(batch_size, inds_inside.size(0)).fill_(-1)\n bbox_inside_weights = gt_boxes.new(batch_size, inds_inside.size(0)).zero_()\n bbox_outside_weights = gt_boxes.new(batch_size, inds_inside.size(0)).zero_()\n\n overlaps = bbox_overlaps_batch(anchors, gt_boxes)\n\n max_overlaps, argmax_overlaps = torch.max(overlaps, 2)\n gt_max_overlaps, _ = torch.max(overlaps, 1)\n\n if not cfg.TRAIN.RPN_CLOBBER_POSITIVES:\n labels[max_overlaps < cfg.TRAIN.RPN_NEGATIVE_OVERLAP] = 0\n\n gt_max_overlaps[gt_max_overlaps==0] = 1e-5\n keep = torch.sum(overlaps.eq(gt_max_overlaps.view(batch_size,1,-1).expand_as(overlaps)), 2)\n\n if torch.sum(keep) > 0:\n labels[keep>0] = 1\n\n # fg label: above threshold IOU\n labels[max_overlaps >= cfg.TRAIN.RPN_POSITIVE_OVERLAP] = 1\n\n if cfg.TRAIN.RPN_CLOBBER_POSITIVES:\n labels[max_overlaps < cfg.TRAIN.RPN_NEGATIVE_OVERLAP] = 0\n\n num_fg = int(cfg.TRAIN.RPN_FG_FRACTION * cfg.TRAIN.RPN_BATCHSIZE)\n\n sum_fg = torch.sum((labels == 1).int(), 1)\n sum_bg = torch.sum((labels == 0).int(), 1)\n\n for i in range(batch_size):\n # subsample positive labels if we have too many\n if sum_fg[i] > num_fg:\n fg_inds = torch.nonzero(labels[i] == 1).view(-1)\n # torch.randperm seems has a bug on multi-gpu setting that cause the segfault.\n # See https://github.com/pytorch/pytorch/issues/1868 for more details.\n # use numpy instead.\n #rand_num = torch.randperm(fg_inds.size(0)).type_as(gt_boxes).long()\n rand_num = torch.from_numpy(np.random.permutation(fg_inds.size(0))).type_as(gt_boxes).long()\n disable_inds = fg_inds[rand_num[:fg_inds.size(0)-num_fg]]\n labels[i][disable_inds] = -1\n\n# num_bg = cfg.TRAIN.RPN_BATCHSIZE - sum_fg[i]\n num_bg = cfg.TRAIN.RPN_BATCHSIZE - torch.sum((labels == 1).int(), 1)[i]\n\n # subsample negative labels if we have too many\n if sum_bg[i] > num_bg:\n bg_inds = torch.nonzero(labels[i] == 0).view(-1)\n #rand_num = torch.randperm(bg_inds.size(0)).type_as(gt_boxes).long()\n\n rand_num = torch.from_numpy(np.random.permutation(bg_inds.size(0))).type_as(gt_boxes).long()\n disable_inds = bg_inds[rand_num[:bg_inds.size(0)-num_bg]]\n labels[i][disable_inds] = -1\n\n offset = torch.arange(0, batch_size)*gt_boxes.size(1)\n\n argmax_overlaps = argmax_overlaps + offset.view(batch_size, 1).type_as(argmax_overlaps)\n bbox_targets = _compute_targets_batch(anchors, gt_boxes.view(-1,5)[argmax_overlaps.view(-1), :].view(batch_size, -1, 5))\n\n # use a single value instead of 4 values for easy index.\n bbox_inside_weights[labels==1] = cfg.TRAIN.RPN_BBOX_INSIDE_WEIGHTS[0]\n\n if cfg.TRAIN.RPN_POSITIVE_WEIGHT < 0:\n num_examples = torch.sum(labels[i] >= 0)\n positive_weights = 1.0 / num_examples.item()\n negative_weights = 1.0 / num_examples.item()\n else:\n assert ((cfg.TRAIN.RPN_POSITIVE_WEIGHT > 0) &\n (cfg.TRAIN.RPN_POSITIVE_WEIGHT < 1))\n\n bbox_outside_weights[labels == 1] = positive_weights\n bbox_outside_weights[labels == 0] = negative_weights\n\n labels = _unmap(labels, total_anchors, inds_inside, batch_size, fill=-1)\n bbox_targets = _unmap(bbox_targets, total_anchors, inds_inside, batch_size, fill=0)\n bbox_inside_weights = _unmap(bbox_inside_weights, total_anchors, inds_inside, batch_size, fill=0)\n bbox_outside_weights = _unmap(bbox_outside_weights, total_anchors, inds_inside, batch_size, fill=0)\n\n outputs = []\n\n labels = labels.view(batch_size, height, width, A).permute(0,3,1,2).contiguous()\n labels = labels.view(batch_size, 1, A * height, width)\n outputs.append(labels)\n\n bbox_targets = bbox_targets.view(batch_size, height, width, A*4).permute(0,3,1,2).contiguous()\n outputs.append(bbox_targets)\n\n anchors_count = bbox_inside_weights.size(1)\n bbox_inside_weights = bbox_inside_weights.view(batch_size,anchors_count,1).expand(batch_size, anchors_count, 4)\n\n bbox_inside_weights = bbox_inside_weights.contiguous().view(batch_size, height, width, 4*A)\\\n .permute(0,3,1,2).contiguous()\n\n outputs.append(bbox_inside_weights)\n\n bbox_outside_weights = bbox_outside_weights.view(batch_size,anchors_count,1).expand(batch_size, anchors_count, 4)\n bbox_outside_weights = bbox_outside_weights.contiguous().view(batch_size, height, width, 4*A)\\\n .permute(0,3,1,2).contiguous()\n outputs.append(bbox_outside_weights)\n\n return outputs\n\n def backward(self, top, propagate_down, bottom):\n \"\"\"This layer does not propagate gradients.\"\"\"\n pass\n\n def reshape(self, bottom, top):\n \"\"\"Reshaping happens during the call to forward.\"\"\"\n pass\n\ndef _unmap(data, count, inds, batch_size, fill=0):\n \"\"\" Unmap a subset of item (data) back to the original set of items (of\n size count) \"\"\"\n\n if data.dim() == 2:\n ret = torch.Tensor(batch_size, count).fill_(fill).type_as(data)\n ret[:, inds] = data\n else:\n ret = torch.Tensor(batch_size, count, data.size(2)).fill_(fill).type_as(data)\n ret[:, inds,:] = data\n return ret\n\n\ndef _compute_targets_batch(ex_rois, gt_rois):\n \"\"\"Compute bounding-box regression targets for an image.\"\"\"\n\n return bbox_transform_batch(ex_rois, gt_rois[:, :, :4])\n"
},
{
"path": "lib/model/rpn/bbox_transform.py",
"content": "# --------------------------------------------------------\n# Fast R-CNN\n# Copyright (c) 2015 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Written by Ross Girshick\n# --------------------------------------------------------\n# --------------------------------------------------------\n# Reorganized and modified by Jianwei Yang and Jiasen Lu\n# --------------------------------------------------------\n\nimport torch\nimport numpy as np\nimport pdb\n\ndef bbox_transform(ex_rois, gt_rois):\n ex_widths = ex_rois[:, 2] - ex_rois[:, 0] + 1.0\n ex_heights = ex_rois[:, 3] - ex_rois[:, 1] + 1.0\n ex_ctr_x = ex_rois[:, 0] + 0.5 * ex_widths\n ex_ctr_y = ex_rois[:, 1] + 0.5 * ex_heights\n\n gt_widths = gt_rois[:, 2] - gt_rois[:, 0] + 1.0\n gt_heights = gt_rois[:, 3] - gt_rois[:, 1] + 1.0\n gt_ctr_x = gt_rois[:, 0] + 0.5 * gt_widths\n gt_ctr_y = gt_rois[:, 1] + 0.5 * gt_heights\n\n targets_dx = (gt_ctr_x - ex_ctr_x) / ex_widths\n targets_dy = (gt_ctr_y - ex_ctr_y) / ex_heights\n targets_dw = torch.log(gt_widths / ex_widths)\n targets_dh = torch.log(gt_heights / ex_heights)\n\n targets = torch.stack(\n (targets_dx, targets_dy, targets_dw, targets_dh),1)\n\n return targets\n\ndef bbox_transform_batch(ex_rois, gt_rois):\n\n if ex_rois.dim() == 2:\n ex_widths = ex_rois[:, 2] - ex_rois[:, 0] + 1.0\n ex_heights = ex_rois[:, 3] - ex_rois[:, 1] + 1.0\n ex_ctr_x = ex_rois[:, 0] + 0.5 * ex_widths\n ex_ctr_y = ex_rois[:, 1] + 0.5 * ex_heights\n\n gt_widths = gt_rois[:, :, 2] - gt_rois[:, :, 0] + 1.0\n gt_heights = gt_rois[:, :, 3] - gt_rois[:, :, 1] + 1.0\n gt_ctr_x = gt_rois[:, :, 0] + 0.5 * gt_widths\n gt_ctr_y = gt_rois[:, :, 1] + 0.5 * gt_heights\n\n targets_dx = (gt_ctr_x - ex_ctr_x.view(1,-1).expand_as(gt_ctr_x)) / ex_widths\n targets_dy = (gt_ctr_y - ex_ctr_y.view(1,-1).expand_as(gt_ctr_y)) / ex_heights\n targets_dw = torch.log(gt_widths / ex_widths.view(1,-1).expand_as(gt_widths))\n targets_dh = torch.log(gt_heights / ex_heights.view(1,-1).expand_as(gt_heights))\n\n elif ex_rois.dim() == 3:\n ex_widths = ex_rois[:, :, 2] - ex_rois[:, :, 0] + 1.0\n ex_heights = ex_rois[:,:, 3] - ex_rois[:,:, 1] + 1.0\n ex_ctr_x = ex_rois[:, :, 0] + 0.5 * ex_widths\n ex_ctr_y = ex_rois[:, :, 1] + 0.5 * ex_heights\n\n gt_widths = gt_rois[:, :, 2] - gt_rois[:, :, 0] + 1.0\n gt_heights = gt_rois[:, :, 3] - gt_rois[:, :, 1] + 1.0\n gt_ctr_x = gt_rois[:, :, 0] + 0.5 * gt_widths\n gt_ctr_y = gt_rois[:, :, 1] + 0.5 * gt_heights\n\n targets_dx = (gt_ctr_x - ex_ctr_x) / ex_widths\n targets_dy = (gt_ctr_y - ex_ctr_y) / ex_heights\n targets_dw = torch.log(gt_widths / ex_widths)\n targets_dh = torch.log(gt_heights / ex_heights)\n else:\n raise ValueError('ex_roi input dimension is not correct.')\n\n targets = torch.stack(\n (targets_dx, targets_dy, targets_dw, targets_dh),2)\n\n return targets\n\ndef bbox_transform_inv(boxes, deltas, batch_size):\n widths = boxes[:, :, 2] - boxes[:, :, 0] + 1.0\n heights = boxes[:, :, 3] - boxes[:, :, 1] + 1.0\n ctr_x = boxes[:, :, 0] + 0.5 * widths\n ctr_y = boxes[:, :, 1] + 0.5 * heights\n\n dx = deltas[:, :, 0::4]\n dy = deltas[:, :, 1::4]\n dw = deltas[:, :, 2::4]\n dh = deltas[:, :, 3::4]\n\n pred_ctr_x = dx * widths.unsqueeze(2) + ctr_x.unsqueeze(2)\n pred_ctr_y = dy * heights.unsqueeze(2) + ctr_y.unsqueeze(2)\n pred_w = torch.exp(dw) * widths.unsqueeze(2)\n pred_h = torch.exp(dh) * heights.unsqueeze(2)\n\n pred_boxes = deltas.clone()\n # x1\n pred_boxes[:, :, 0::4] = pred_ctr_x - 0.5 * pred_w\n # y1\n pred_boxes[:, :, 1::4] = pred_ctr_y - 0.5 * pred_h\n # x2\n pred_boxes[:, :, 2::4] = pred_ctr_x + 0.5 * pred_w\n # y2\n pred_boxes[:, :, 3::4] = pred_ctr_y + 0.5 * pred_h\n\n return pred_boxes\n\ndef clip_boxes_batch(boxes, im_shape, batch_size):\n \"\"\"\n Clip boxes to image boundaries.\n \"\"\"\n num_rois = boxes.size(1)\n\n boxes[boxes < 0] = 0\n # batch_x = (im_shape[:,0]-1).view(batch_size, 1).expand(batch_size, num_rois)\n # batch_y = (im_shape[:,1]-1).view(batch_size, 1).expand(batch_size, num_rois)\n\n batch_x = im_shape[:, 1] - 1\n batch_y = im_shape[:, 0] - 1\n\n boxes[:,:,0][boxes[:,:,0] > batch_x] = batch_x\n boxes[:,:,1][boxes[:,:,1] > batch_y] = batch_y\n boxes[:,:,2][boxes[:,:,2] > batch_x] = batch_x\n boxes[:,:,3][boxes[:,:,3] > batch_y] = batch_y\n\n return boxes\n\ndef clip_boxes(boxes, im_shape, batch_size):\n\n for i in range(batch_size):\n boxes[i,:,0::4].clamp_(0, im_shape[i, 1]-1)\n boxes[i,:,1::4].clamp_(0, im_shape[i, 0]-1)\n boxes[i,:,2::4].clamp_(0, im_shape[i, 1]-1)\n boxes[i,:,3::4].clamp_(0, im_shape[i, 0]-1)\n\n return boxes\n\n\ndef bbox_overlaps(anchors, gt_boxes):\n \"\"\"\n anchors: (N, 4) ndarray of float\n gt_boxes: (K, 4) ndarray of float\n\n overlaps: (N, K) ndarray of overlap between boxes and query_boxes\n \"\"\"\n N = anchors.size(0)\n K = gt_boxes.size(0)\n\n gt_boxes_area = ((gt_boxes[:,2] - gt_boxes[:,0] + 1) *\n (gt_boxes[:,3] - gt_boxes[:,1] + 1)).view(1, K)\n\n anchors_area = ((anchors[:,2] - anchors[:,0] + 1) *\n (anchors[:,3] - anchors[:,1] + 1)).view(N, 1)\n\n boxes = anchors.view(N, 1, 4).expand(N, K, 4)\n query_boxes = gt_boxes.view(1, K, 4).expand(N, K, 4)\n\n iw = (torch.min(boxes[:,:,2], query_boxes[:,:,2]) -\n torch.max(boxes[:,:,0], query_boxes[:,:,0]) + 1)\n iw[iw < 0] = 0\n\n ih = (torch.min(boxes[:,:,3], query_boxes[:,:,3]) -\n torch.max(boxes[:,:,1], query_boxes[:,:,1]) + 1)\n ih[ih < 0] = 0\n\n ua = anchors_area + gt_boxes_area - (iw * ih)\n overlaps = iw * ih / ua\n\n return overlaps\n\ndef bbox_overlaps_batch(anchors, gt_boxes):\n \"\"\"\n anchors: (N, 4) ndarray of float\n gt_boxes: (b, K, 5) ndarray of float\n\n overlaps: (N, K) ndarray of overlap between boxes and query_boxes\n \"\"\"\n batch_size = gt_boxes.size(0)\n\n\n if anchors.dim() == 2:\n\n N = anchors.size(0)\n K = gt_boxes.size(1)\n\n anchors = anchors.view(1, N, 4).expand(batch_size, N, 4).contiguous()\n gt_boxes = gt_boxes[:,:,:4].contiguous()\n\n\n gt_boxes_x = (gt_boxes[:,:,2] - gt_boxes[:,:,0] + 1)\n gt_boxes_y = (gt_boxes[:,:,3] - gt_boxes[:,:,1] + 1)\n gt_boxes_area = (gt_boxes_x * gt_boxes_y).view(batch_size, 1, K)\n\n anchors_boxes_x = (anchors[:,:,2] - anchors[:,:,0] + 1)\n anchors_boxes_y = (anchors[:,:,3] - anchors[:,:,1] + 1)\n anchors_area = (anchors_boxes_x * anchors_boxes_y).view(batch_size, N, 1)\n\n gt_area_zero = (gt_boxes_x == 1) & (gt_boxes_y == 1)\n anchors_area_zero = (anchors_boxes_x == 1) & (anchors_boxes_y == 1)\n\n boxes = anchors.view(batch_size, N, 1, 4).expand(batch_size, N, K, 4)\n query_boxes = gt_boxes.view(batch_size, 1, K, 4).expand(batch_size, N, K, 4)\n\n iw = (torch.min(boxes[:,:,:,2], query_boxes[:,:,:,2]) -\n torch.max(boxes[:,:,:,0], query_boxes[:,:,:,0]) + 1)\n iw[iw < 0] = 0\n\n ih = (torch.min(boxes[:,:,:,3], query_boxes[:,:,:,3]) -\n torch.max(boxes[:,:,:,1], query_boxes[:,:,:,1]) + 1)\n ih[ih < 0] = 0\n ua = anchors_area + gt_boxes_area - (iw * ih)\n overlaps = iw * ih / ua\n\n # mask the overlap here.\n overlaps.masked_fill_(gt_area_zero.view(batch_size, 1, K).expand(batch_size, N, K), 0)\n overlaps.masked_fill_(anchors_area_zero.view(batch_size, N, 1).expand(batch_size, N, K), -1)\n\n elif anchors.dim() == 3:\n N = anchors.size(1)\n K = gt_boxes.size(1)\n\n if anchors.size(2) == 4:\n anchors = anchors[:,:,:4].contiguous()\n else:\n anchors = anchors[:,:,1:5].contiguous()\n\n gt_boxes = gt_boxes[:,:,:4].contiguous()\n\n gt_boxes_x = (gt_boxes[:,:,2] - gt_boxes[:,:,0] + 1)\n gt_boxes_y = (gt_boxes[:,:,3] - gt_boxes[:,:,1] + 1)\n gt_boxes_area = (gt_boxes_x * gt_boxes_y).view(batch_size, 1, K)\n\n anchors_boxes_x = (anchors[:,:,2] - anchors[:,:,0] + 1)\n anchors_boxes_y = (anchors[:,:,3] - anchors[:,:,1] + 1)\n anchors_area = (anchors_boxes_x * anchors_boxes_y).view(batch_size, N, 1)\n\n gt_area_zero = (gt_boxes_x == 1) & (gt_boxes_y == 1)\n anchors_area_zero = (anchors_boxes_x == 1) & (anchors_boxes_y == 1)\n\n boxes = anchors.view(batch_size, N, 1, 4).expand(batch_size, N, K, 4)\n query_boxes = gt_boxes.view(batch_size, 1, K, 4).expand(batch_size, N, K, 4)\n\n iw = (torch.min(boxes[:,:,:,2], query_boxes[:,:,:,2]) -\n torch.max(boxes[:,:,:,0], query_boxes[:,:,:,0]) + 1)\n iw[iw < 0] = 0\n\n ih = (torch.min(boxes[:,:,:,3], query_boxes[:,:,:,3]) -\n torch.max(boxes[:,:,:,1], query_boxes[:,:,:,1]) + 1)\n ih[ih < 0] = 0\n ua = anchors_area + gt_boxes_area - (iw * ih)\n \n # Intersection (iw * ih) divided by Union (ua)\n overlaps = iw * ih / ua\n\n # mask the overlap here.\n overlaps.masked_fill_(gt_area_zero.view(batch_size, 1, K).expand(batch_size, N, K), 0)\n overlaps.masked_fill_(anchors_area_zero.view(batch_size, N, 1).expand(batch_size, N, K), -1)\n else:\n raise ValueError('anchors input dimension is not correct.')\n\n return overlaps\n"
},
{
"path": "lib/model/rpn/generate_anchors.py",
"content": "from __future__ import print_function\n# --------------------------------------------------------\n# Faster R-CNN\n# Copyright (c) 2015 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Written by Ross Girshick and Sean Bell\n# --------------------------------------------------------\n\nimport numpy as np\nimport pdb\n\n# Verify that we compute the same anchors as Shaoqing's matlab implementation:\n#\n# >> load output/rpn_cachedir/faster_rcnn_VOC2007_ZF_stage1_rpn/anchors.mat\n# >> anchors\n#\n# anchors =\n#\n# -83 -39 100 56\n# -175 -87 192 104\n# -359 -183 376 200\n# -55 -55 72 72\n# -119 -119 136 136\n# -247 -247 264 264\n# -35 -79 52 96\n# -79 -167 96 184\n# -167 -343 184 360\n\n#array([[ -83., -39., 100., 56.],\n# [-175., -87., 192., 104.],\n# [-359., -183., 376., 200.],\n# [ -55., -55., 72., 72.],\n# [-119., -119., 136., 136.],\n# [-247., -247., 264., 264.],\n# [ -35., -79., 52., 96.],\n# [ -79., -167., 96., 184.],\n# [-167., -343., 184., 360.]])\n\ntry:\n xrange # Python 2\nexcept NameError:\n xrange = range # Python 3\n\n\ndef generate_anchors(base_size=16, ratios=[0.5, 1, 2],\n scales=2**np.arange(3, 6)):\n \"\"\"\n Generate anchor (reference) windows by enumerating aspect ratios X\n scales wrt a reference (0, 0, 15, 15) window.\n \"\"\"\n\n base_anchor = np.array([1, 1, base_size, base_size]) - 1\n ratio_anchors = _ratio_enum(base_anchor, ratios)\n anchors = np.vstack([_scale_enum(ratio_anchors[i, :], scales)\n for i in xrange(ratio_anchors.shape[0])])\n return anchors\n\ndef _whctrs(anchor):\n \"\"\"\n Return width, height, x center, and y center for an anchor (window).\n \"\"\"\n\n w = anchor[2] - anchor[0] + 1\n h = anchor[3] - anchor[1] + 1\n x_ctr = anchor[0] + 0.5 * (w - 1)\n y_ctr = anchor[1] + 0.5 * (h - 1)\n return w, h, x_ctr, y_ctr\n\ndef _mkanchors(ws, hs, x_ctr, y_ctr):\n \"\"\"\n Given a vector of widths (ws) and heights (hs) around a center\n (x_ctr, y_ctr), output a set of anchors (windows).\n \"\"\"\n\n ws = ws[:, np.newaxis]\n hs = hs[:, np.newaxis]\n anchors = np.hstack((x_ctr - 0.5 * (ws - 1),\n y_ctr - 0.5 * (hs - 1),\n x_ctr + 0.5 * (ws - 1),\n y_ctr + 0.5 * (hs - 1)))\n return anchors\n\ndef _ratio_enum(anchor, ratios):\n \"\"\"\n Enumerate a set of anchors for each aspect ratio wrt an anchor.\n \"\"\"\n\n w, h, x_ctr, y_ctr = _whctrs(anchor)\n size = w * h\n size_ratios = size / ratios\n ws = np.round(np.sqrt(size_ratios))\n hs = np.round(ws * ratios)\n anchors = _mkanchors(ws, hs, x_ctr, y_ctr)\n return anchors\n\ndef _scale_enum(anchor, scales):\n \"\"\"\n Enumerate a set of anchors for each scale wrt an anchor.\n \"\"\"\n\n w, h, x_ctr, y_ctr = _whctrs(anchor)\n ws = w * scales\n hs = h * scales\n anchors = _mkanchors(ws, hs, x_ctr, y_ctr)\n return anchors\n\nif __name__ == '__main__':\n import time\n t = time.time()\n a = generate_anchors()\n print(time.time() - t)\n print(a)\n from IPython import embed; embed()\n"
},
{
"path": "lib/model/rpn/proposal_layer.py",
"content": "from __future__ import absolute_import\n# --------------------------------------------------------\n# Faster R-CNN\n# Copyright (c) 2015 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Written by Ross Girshick and Sean Bell\n# --------------------------------------------------------\n# --------------------------------------------------------\n# Reorganized and modified by Jianwei Yang and Jiasen Lu\n# --------------------------------------------------------\n\nimport torch\nimport torch.nn as nn\nimport numpy as np\nimport math\nimport yaml\nfrom model.utils.config import cfg\nfrom .generate_anchors import generate_anchors\nfrom .bbox_transform import bbox_transform_inv, clip_boxes, clip_boxes_batch\nfrom model.nms.nms_wrapper import nms\n\nimport pdb\n\nDEBUG = False\n\nclass _ProposalLayer(nn.Module):\n \"\"\"\n Outputs object detection proposals by applying estimated bounding-box\n transformations to a set of regular boxes (called \"anchors\").\n \"\"\"\n\n def __init__(self, feat_stride, scales, ratios):\n super(_ProposalLayer, self).__init__()\n\n self._feat_stride = feat_stride\n self._anchors = torch.from_numpy(generate_anchors(scales=np.array(scales), \n ratios=np.array(ratios))).float()\n self._num_anchors = self._anchors.size(0)\n\n # rois blob: holds R regions of interest, each is a 5-tuple\n # (n, x1, y1, x2, y2) specifying an image batch index n and a\n # rectangle (x1, y1, x2, y2)\n # top[0].reshape(1, 5)\n #\n # # scores blob: holds scores for R regions of interest\n # if len(top) > 1:\n # top[1].reshape(1, 1, 1, 1)\n\n def forward(self, input):\n\n # Algorithm:\n #\n # for each (H, W) location i\n # generate A anchor boxes centered on cell i\n # apply predicted bbox deltas at cell i to each of the A anchors\n # clip predicted boxes to image\n # remove predicted boxes with either height or width < threshold\n # sort all (proposal, score) pairs by score from highest to lowest\n # take top pre_nms_topN proposals before NMS\n # apply NMS with threshold 0.7 to remaining proposals\n # take after_nms_topN proposals after NMS\n # return the top proposals (-> RoIs top, scores top)\n\n\n # the first set of _num_anchors channels are bg probs\n # the second set are the fg probs\n scores = input[0][:, self._num_anchors:, :, :]\n bbox_deltas = input[1]\n im_info = input[2]\n cfg_key = input[3]\n\n pre_nms_topN = cfg[cfg_key].RPN_PRE_NMS_TOP_N\n post_nms_topN = cfg[cfg_key].RPN_POST_NMS_TOP_N\n nms_thresh = cfg[cfg_key].RPN_NMS_THRESH\n min_size = cfg[cfg_key].RPN_MIN_SIZE\n\n batch_size = bbox_deltas.size(0)\n\n feat_height, feat_width = scores.size(2), scores.size(3)\n shift_x = np.arange(0, feat_width) * self._feat_stride\n shift_y = np.arange(0, feat_height) * self._feat_stride\n shift_x, shift_y = np.meshgrid(shift_x, shift_y)\n shifts = torch.from_numpy(np.vstack((shift_x.ravel(), shift_y.ravel(),\n shift_x.ravel(), shift_y.ravel())).transpose())\n shifts = shifts.contiguous().type_as(scores).float()\n\n A = self._num_anchors\n K = shifts.size(0)\n\n self._anchors = self._anchors.type_as(scores)\n # anchors = self._anchors.view(1, A, 4) + shifts.view(1, K, 4).permute(1, 0, 2).contiguous()\n anchors = self._anchors.view(1, A, 4) + shifts.view(K, 1, 4)\n anchors = anchors.view(1, K * A, 4).expand(batch_size, K * A, 4)\n\n # Transpose and reshape predicted bbox transformations to get them\n # into the same order as the anchors:\n\n bbox_deltas = bbox_deltas.permute(0, 2, 3, 1).contiguous()\n bbox_deltas = bbox_deltas.view(batch_size, -1, 4)\n\n # Same story for the scores:\n scores = scores.permute(0, 2, 3, 1).contiguous()\n scores = scores.view(batch_size, -1)\n\n # Convert anchors into proposals via bbox transformations\n proposals = bbox_transform_inv(anchors, bbox_deltas, batch_size)\n\n # 2. clip predicted boxes to image\n proposals = clip_boxes(proposals, im_info, batch_size)\n # proposals = clip_boxes_batch(proposals, im_info, batch_size)\n\n # assign the score to 0 if it's non keep.\n # keep = self._filter_boxes(proposals, min_size * im_info[:, 2])\n\n # trim keep index to make it euqal over batch\n # keep_idx = torch.cat(tuple(keep_idx), 0)\n\n # scores_keep = scores.view(-1)[keep_idx].view(batch_size, trim_size)\n # proposals_keep = proposals.view(-1, 4)[keep_idx, :].contiguous().view(batch_size, trim_size, 4)\n \n # _, order = torch.sort(scores_keep, 1, True)\n \n scores_keep = scores\n proposals_keep = proposals\n _, order = torch.sort(scores_keep, 1, True)\n\n output = scores.new(batch_size, post_nms_topN, 5).zero_()\n for i in range(batch_size):\n # # 3. remove predicted boxes with either height or width < threshold\n # # (NOTE: convert min_size to input image scale stored in im_info[2])\n proposals_single = proposals_keep[i]\n scores_single = scores_keep[i]\n\n # # 4. sort all (proposal, score) pairs by score from highest to lowest\n # # 5. take top pre_nms_topN (e.g. 6000)\n order_single = order[i]\n\n if pre_nms_topN > 0 and pre_nms_topN < scores_keep.numel():\n order_single = order_single[:pre_nms_topN]\n\n proposals_single = proposals_single[order_single, :]\n scores_single = scores_single[order_single].view(-1,1)\n\n # 6. apply nms (e.g. threshold = 0.7)\n # 7. take after_nms_topN (e.g. 300)\n # 8. return the top proposals (-> RoIs top)\n\n keep_idx_i = nms(torch.cat((proposals_single, scores_single), 1), nms_thresh, force_cpu=not cfg.USE_GPU_NMS)\n keep_idx_i = keep_idx_i.long().view(-1)\n\n if post_nms_topN > 0:\n keep_idx_i = keep_idx_i[:post_nms_topN]\n proposals_single = proposals_single[keep_idx_i, :]\n scores_single = scores_single[keep_idx_i, :]\n\n # padding 0 at the end.\n num_proposal = proposals_single.size(0)\n output[i,:,0] = i\n output[i,:num_proposal,1:] = proposals_single\n\n return output\n\n def backward(self, top, propagate_down, bottom):\n \"\"\"This layer does not propagate gradients.\"\"\"\n pass\n\n def reshape(self, bottom, top):\n \"\"\"Reshaping happens during the call to forward.\"\"\"\n pass\n\n def _filter_boxes(self, boxes, min_size):\n \"\"\"Remove all boxes with any side smaller than min_size.\"\"\"\n ws = boxes[:, :, 2] - boxes[:, :, 0] + 1\n hs = boxes[:, :, 3] - boxes[:, :, 1] + 1\n keep = ((ws >= min_size.view(-1,1).expand_as(ws)) & (hs >= min_size.view(-1,1).expand_as(hs)))\n return keep\n"
},
{
"path": "lib/model/rpn/proposal_target_layer_cascade.py",
"content": "from __future__ import absolute_import\n# --------------------------------------------------------\n# Faster R-CNN\n# Copyright (c) 2015 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Written by Ross Girshick and Sean Bell\n# --------------------------------------------------------\n# --------------------------------------------------------\n# Reorganized and modified by Jianwei Yang and Jiasen Lu\n# --------------------------------------------------------\n\nimport torch\nimport torch.nn as nn\nimport numpy as np\nimport numpy.random as npr\nfrom ..utils.config import cfg\nfrom .bbox_transform import bbox_overlaps_batch, bbox_transform_batch\nimport pdb\n\nclass _ProposalTargetLayer(nn.Module):\n \"\"\"\n Assign object detection proposals to ground-truth targets. Produces proposal\n classification labels and bounding-box regression targets.\n \"\"\"\n\n def __init__(self, nclasses):\n super(_ProposalTargetLayer, self).__init__()\n self._num_classes = nclasses\n self.BBOX_NORMALIZE_MEANS = torch.FloatTensor(cfg.TRAIN.BBOX_NORMALIZE_MEANS)\n self.BBOX_NORMALIZE_STDS = torch.FloatTensor(cfg.TRAIN.BBOX_NORMALIZE_STDS)\n self.BBOX_INSIDE_WEIGHTS = torch.FloatTensor(cfg.TRAIN.BBOX_INSIDE_WEIGHTS)\n\n def forward(self, all_rois, gt_boxes, num_boxes):\n\n self.BBOX_NORMALIZE_MEANS = self.BBOX_NORMALIZE_MEANS.type_as(gt_boxes)\n self.BBOX_NORMALIZE_STDS = self.BBOX_NORMALIZE_STDS.type_as(gt_boxes)\n self.BBOX_INSIDE_WEIGHTS = self.BBOX_INSIDE_WEIGHTS.type_as(gt_boxes)\n\n gt_boxes_append = gt_boxes.new(gt_boxes.size()).zero_()\n gt_boxes_append[:,:,1:5] = gt_boxes[:,:,:4]\n\n # Include ground-truth boxes in the set of candidate rois\n all_rois = torch.cat([all_rois, gt_boxes_append], 1)\n\n num_images = 1\n rois_per_image = int(cfg.TRAIN.BATCH_SIZE / num_images)\n fg_rois_per_image = int(np.round(cfg.TRAIN.FG_FRACTION * rois_per_image))\n fg_rois_per_image = 1 if fg_rois_per_image == 0 else fg_rois_per_image\n\n labels, rois, bbox_targets, bbox_inside_weights = self._sample_rois_pytorch(\n all_rois, gt_boxes, fg_rois_per_image,\n rois_per_image, self._num_classes)\n\n bbox_outside_weights = (bbox_inside_weights > 0).float()\n\n return rois, labels, bbox_targets, bbox_inside_weights, bbox_outside_weights\n\n def backward(self, top, propagate_down, bottom):\n \"\"\"This layer does not propagate gradients.\"\"\"\n pass\n\n def reshape(self, bottom, top):\n \"\"\"Reshaping happens during the call to forward.\"\"\"\n pass\n\n def _get_bbox_regression_labels_pytorch(self, bbox_target_data, labels_batch, num_classes):\n \"\"\"Bounding-box regression targets (bbox_target_data) are stored in a\n compact form b x N x (class, tx, ty, tw, th)\n\n This function expands those targets into the 4-of-4*K representation used\n by the network (i.e. only one class has non-zero targets).\n\n Returns:\n bbox_target (ndarray): b x N x 4K blob of regression targets\n bbox_inside_weights (ndarray): b x N x 4K blob of loss weights\n \"\"\"\n batch_size = labels_batch.size(0)\n rois_per_image = labels_batch.size(1)\n clss = labels_batch\n bbox_targets = bbox_target_data.new(batch_size, rois_per_image, 4).zero_()\n bbox_inside_weights = bbox_target_data.new(bbox_targets.size()).zero_()\n\n for b in range(batch_size):\n # assert clss[b].sum() > 0\n if clss[b].sum() == 0:\n continue\n inds = torch.nonzero(clss[b] > 0).view(-1)\n for i in range(inds.numel()):\n ind = inds[i]\n bbox_targets[b, ind, :] = bbox_target_data[b, ind, :]\n bbox_inside_weights[b, ind, :] = self.BBOX_INSIDE_WEIGHTS\n\n return bbox_targets, bbox_inside_weights\n\n\n def _compute_targets_pytorch(self, ex_rois, gt_rois):\n \"\"\"Compute bounding-box regression targets for an image.\"\"\"\n\n assert ex_rois.size(1) == gt_rois.size(1)\n assert ex_rois.size(2) == 4\n assert gt_rois.size(2) == 4\n\n batch_size = ex_rois.size(0)\n rois_per_image = ex_rois.size(1)\n\n targets = bbox_transform_batch(ex_rois, gt_rois)\n\n if cfg.TRAIN.BBOX_NORMALIZE_TARGETS_PRECOMPUTED:\n # Optionally normalize targets by a precomputed mean and stdev\n targets = ((targets - self.BBOX_NORMALIZE_MEANS.expand_as(targets))\n / self.BBOX_NORMALIZE_STDS.expand_as(targets))\n\n return targets\n\n\n def _sample_rois_pytorch(self, all_rois, gt_boxes, fg_rois_per_image, rois_per_image, num_classes):\n \"\"\"Generate a random sample of RoIs comprising foreground and background\n examples.\n \"\"\"\n # overlaps: (rois x gt_boxes)\n\n overlaps = bbox_overlaps_batch(all_rois, gt_boxes)\n\n max_overlaps, gt_assignment = torch.max(overlaps, 2)\n\n batch_size = overlaps.size(0)\n num_proposal = overlaps.size(1)\n num_boxes_per_img = overlaps.size(2)\n\n offset = torch.arange(0, batch_size)*gt_boxes.size(1)\n offset = offset.view(-1, 1).type_as(gt_assignment) + gt_assignment\n\n labels = gt_boxes[:,:,4].contiguous().view(-1).index((offset.view(-1),)).view(batch_size, -1)\n \n labels_batch = labels.new(batch_size, rois_per_image).zero_()\n rois_batch = all_rois.new(batch_size, rois_per_image, 5).zero_()\n gt_rois_batch = all_rois.new(batch_size, rois_per_image, 5).zero_()\n # Guard against the case when an image has fewer than max_fg_rois_per_image\n # foreground RoIs\n for i in range(batch_size):\n\n fg_inds = torch.nonzero(max_overlaps[i] >= cfg.TRAIN.FG_THRESH).view(-1)\n fg_num_rois = fg_inds.numel()\n\n # Select background RoIs as those within [BG_THRESH_LO, BG_THRESH_HI)\n bg_inds = torch.nonzero((max_overlaps[i] < cfg.TRAIN.BG_THRESH_HI) &\n (max_overlaps[i] >= cfg.TRAIN.BG_THRESH_LO)).view(-1)\n bg_num_rois = bg_inds.numel()\n\n if fg_num_rois > 0 and bg_num_rois > 0:\n # sampling fg\n fg_rois_per_this_image = min(fg_rois_per_image, fg_num_rois)\n\n # torch.randperm seems has a bug on multi-gpu setting that cause the segfault.\n # See https://github.com/pytorch/pytorch/issues/1868 for more details.\n # use numpy instead.\n #rand_num = torch.randperm(fg_num_rois).long().cuda()\n rand_num = torch.from_numpy(np.random.permutation(fg_num_rois)).type_as(gt_boxes).long()\n fg_inds = fg_inds[rand_num[:fg_rois_per_this_image]]\n\n # sampling bg\n bg_rois_per_this_image = rois_per_image - fg_rois_per_this_image\n\n # Seems torch.rand has a bug, it will generate very large number and make an error.\n # We use numpy rand instead.\n #rand_num = (torch.rand(bg_rois_per_this_image) * bg_num_rois).long().cuda()\n rand_num = np.floor(np.random.rand(bg_rois_per_this_image) * bg_num_rois)\n rand_num = torch.from_numpy(rand_num).type_as(gt_boxes).long()\n bg_inds = bg_inds[rand_num]\n\n elif fg_num_rois > 0 and bg_num_rois == 0:\n # sampling fg\n #rand_num = torch.floor(torch.rand(rois_per_image) * fg_num_rois).long().cuda()\n rand_num = np.floor(np.random.rand(rois_per_image) * fg_num_rois)\n rand_num = torch.from_numpy(rand_num).type_as(gt_boxes).long()\n fg_inds = fg_inds[rand_num]\n fg_rois_per_this_image = rois_per_image\n bg_rois_per_this_image = 0\n elif bg_num_rois > 0 and fg_num_rois == 0:\n # sampling bg\n #rand_num = torch.floor(torch.rand(rois_per_image) * bg_num_rois).long().cuda()\n rand_num = np.floor(np.random.rand(rois_per_image) * bg_num_rois)\n rand_num = torch.from_numpy(rand_num).type_as(gt_boxes).long()\n\n bg_inds = bg_inds[rand_num]\n bg_rois_per_this_image = rois_per_image\n fg_rois_per_this_image = 0\n else:\n raise ValueError(\"bg_num_rois = 0 and fg_num_rois = 0, this should not happen!\")\n\n # The indices that we're selecting (both fg and bg)\n keep_inds = torch.cat([fg_inds, bg_inds], 0)\n\n # Select sampled values from various arrays:\n labels_batch[i].copy_(labels[i][keep_inds])\n\n # Clamp labels for the background RoIs to 0\n if fg_rois_per_this_image < rois_per_image:\n labels_batch[i][fg_rois_per_this_image:] = 0\n\n rois_batch[i] = all_rois[i][keep_inds]\n rois_batch[i,:,0] = i\n\n gt_rois_batch[i] = gt_boxes[i][gt_assignment[i][keep_inds]]\n\n bbox_target_data = self._compute_targets_pytorch(\n rois_batch[:,:,1:5], gt_rois_batch[:,:,:4])\n\n bbox_targets, bbox_inside_weights = \\\n self._get_bbox_regression_labels_pytorch(bbox_target_data, labels_batch, num_classes)\n\n return labels_batch, rois_batch, bbox_targets, bbox_inside_weights\n"
},
{
"path": "lib/model/rpn/rpn.py",
"content": "from __future__ import absolute_import\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nfrom torch.autograd import Variable\n\nfrom model.utils.config import cfg\nfrom .proposal_layer import _ProposalLayer\nfrom .anchor_target_layer import _AnchorTargetLayer\nfrom model.utils.net_utils import _smooth_l1_loss\n\nimport numpy as np\nimport math\nimport pdb\nimport time\n\nclass _RPN(nn.Module):\n \"\"\" region proposal network \"\"\"\n def __init__(self, din):\n super(_RPN, self).__init__()\n \n self.din = din # get depth of input feature map, e.g., 512\n self.anchor_scales = cfg.ANCHOR_SCALES\n self.anchor_ratios = cfg.ANCHOR_RATIOS\n self.feat_stride = cfg.FEAT_STRIDE[0]\n\n # define the convrelu layers processing input feature map\n self.RPN_Conv = nn.Conv2d(self.din, 512, 3, 1, 1, bias=True)\n\n # define bg/fg classifcation score layer\n self.nc_score_out = len(self.anchor_scales) * len(self.anchor_ratios) * 2 # 2(bg/fg) * 9 (anchors)\n self.RPN_cls_score = nn.Conv2d(512, self.nc_score_out, 1, 1, 0)\n\n # define anchor box offset prediction layer\n self.nc_bbox_out = len(self.anchor_scales) * len(self.anchor_ratios) * 4 # 4(coords) * 9 (anchors)\n self.RPN_bbox_pred = nn.Conv2d(512, self.nc_bbox_out, 1, 1, 0)\n\n # define proposal layer\n self.RPN_proposal = _ProposalLayer(self.feat_stride, self.anchor_scales, self.anchor_ratios)\n\n # define anchor target layer\n self.RPN_anchor_target = _AnchorTargetLayer(self.feat_stride, self.anchor_scales, self.anchor_ratios)\n\n self.rpn_loss_cls = 0\n self.rpn_loss_box = 0\n\n @staticmethod\n def reshape(x, d):\n input_shape = x.size()\n x = x.view(\n input_shape[0],\n int(d),\n int(float(input_shape[1] * input_shape[2]) / float(d)),\n input_shape[3]\n )\n return x\n\n def forward(self, base_feat, im_info, gt_boxes, num_boxes):\n\n batch_size = base_feat.size(0)\n\n # return feature map after convrelu layer\n rpn_conv1 = F.relu(self.RPN_Conv(base_feat), inplace=True)\n # get rpn classification score\n rpn_cls_score = self.RPN_cls_score(rpn_conv1)\n\n rpn_cls_score_reshape = self.reshape(rpn_cls_score, 2)\n rpn_cls_prob_reshape = F.softmax(rpn_cls_score_reshape, 1)\n rpn_cls_prob = self.reshape(rpn_cls_prob_reshape, self.nc_score_out)\n\n # get rpn offsets to the anchor boxes\n rpn_bbox_pred = self.RPN_bbox_pred(rpn_conv1)\n\n # proposal layer\n cfg_key = 'TRAIN' if self.training else 'TEST'\n\n rois = self.RPN_proposal((rpn_cls_prob.data, rpn_bbox_pred.data,\n im_info, cfg_key))\n\n self.rpn_loss_cls = 0\n self.rpn_loss_box = 0\n\n # generating training labels and build the rpn loss\n if self.training:\n assert gt_boxes is not None\n\n rpn_data = self.RPN_anchor_target((rpn_cls_score.data, gt_boxes, im_info, num_boxes))\n\n # compute classification loss\n rpn_cls_score = rpn_cls_score_reshape.permute(0, 2, 3, 1).contiguous().view(batch_size, -1, 2)\n rpn_label = rpn_data[0].view(batch_size, -1)\n\n rpn_keep = Variable(rpn_label.view(-1).ne(-1).nonzero().view(-1))\n rpn_cls_score = torch.index_select(rpn_cls_score.view(-1,2), 0, rpn_keep)\n rpn_label = torch.index_select(rpn_label.view(-1), 0, rpn_keep.data)\n rpn_label = Variable(rpn_label.long())\n self.rpn_loss_cls = F.cross_entropy(rpn_cls_score, rpn_label)\n fg_cnt = torch.sum(rpn_label.data.ne(0))\n\n rpn_bbox_targets, rpn_bbox_inside_weights, rpn_bbox_outside_weights = rpn_data[1:]\n\n # compute bbox regression loss\n rpn_bbox_inside_weights = Variable(rpn_bbox_inside_weights)\n rpn_bbox_outside_weights = Variable(rpn_bbox_outside_weights)\n rpn_bbox_targets = Variable(rpn_bbox_targets)\n\n self.rpn_loss_box = _smooth_l1_loss(rpn_bbox_pred, rpn_bbox_targets, rpn_bbox_inside_weights,\n rpn_bbox_outside_weights, sigma=3, dim=[1,2,3])\n\n return rois, self.rpn_loss_cls, self.rpn_loss_box\n"
},
{
"path": "lib/model/utils/.gitignore",
"content": "*.c\n*.cpp\n*.so\n"
},
{
"path": "lib/model/utils/__init__.py",
"content": ""
},
{
"path": "lib/model/utils/bbox.pyx",
"content": "# --------------------------------------------------------\n# Fast R-CNN\n# Copyright (c) 2015 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Written by Sergey Karayev\n# --------------------------------------------------------\n\ncimport cython\nimport numpy as np\ncimport numpy as np\n\nDTYPE = np.float\nctypedef np.float_t DTYPE_t\n\ndef bbox_overlaps(np.ndarray[DTYPE_t, ndim=2] boxes,\n np.ndarray[DTYPE_t, ndim=2] query_boxes):\n return bbox_overlaps_c(boxes, query_boxes)\n\ncdef np.ndarray[DTYPE_t, ndim=2] bbox_overlaps_c(\n np.ndarray[DTYPE_t, ndim=2] boxes,\n np.ndarray[DTYPE_t, ndim=2] query_boxes):\n \"\"\"\n Parameters\n ----------\n boxes: (N, 4) ndarray of float\n query_boxes: (K, 4) ndarray of float\n Returns\n -------\n overlaps: (N, K) ndarray of overlap between boxes and query_boxes\n \"\"\"\n cdef unsigned int N = boxes.shape[0]\n cdef unsigned int K = query_boxes.shape[0]\n cdef np.ndarray[DTYPE_t, ndim=2] overlaps = np.zeros((N, K), dtype=DTYPE)\n cdef DTYPE_t iw, ih, box_area\n cdef DTYPE_t ua\n cdef unsigned int k, n\n for k in range(K):\n box_area = (\n (query_boxes[k, 2] - query_boxes[k, 0] + 1) *\n (query_boxes[k, 3] - query_boxes[k, 1] + 1)\n )\n for n in range(N):\n iw = (\n min(boxes[n, 2], query_boxes[k, 2]) -\n max(boxes[n, 0], query_boxes[k, 0]) + 1\n )\n if iw > 0:\n ih = (\n min(boxes[n, 3], query_boxes[k, 3]) -\n max(boxes[n, 1], query_boxes[k, 1]) + 1\n )\n if ih > 0:\n ua = float(\n (boxes[n, 2] - boxes[n, 0] + 1) *\n (boxes[n, 3] - boxes[n, 1] + 1) +\n box_area - iw * ih\n )\n overlaps[n, k] = iw * ih / ua\n return overlaps\n\n\ndef bbox_intersections(\n np.ndarray[DTYPE_t, ndim=2] boxes,\n np.ndarray[DTYPE_t, ndim=2] query_boxes):\n return bbox_intersections_c(boxes, query_boxes)\n\n\ncdef np.ndarray[DTYPE_t, ndim=2] bbox_intersections_c(\n np.ndarray[DTYPE_t, ndim=2] boxes,\n np.ndarray[DTYPE_t, ndim=2] query_boxes):\n \"\"\"\n For each query box compute the intersection ratio covered by boxes\n ----------\n Parameters\n ----------\n boxes: (N, 4) ndarray of float\n query_boxes: (K, 4) ndarray of float\n Returns\n -------\n overlaps: (N, K) ndarray of intersec between boxes and query_boxes\n \"\"\"\n cdef unsigned int N = boxes.shape[0]\n cdef unsigned int K = query_boxes.shape[0]\n cdef np.ndarray[DTYPE_t, ndim=2] intersec = np.zeros((N, K), dtype=DTYPE)\n cdef DTYPE_t iw, ih, box_area\n cdef DTYPE_t ua\n cdef unsigned int k, n\n for k in range(K):\n box_area = (\n (query_boxes[k, 2] - query_boxes[k, 0] + 1) *\n (query_boxes[k, 3] - query_boxes[k, 1] + 1)\n )\n for n in range(N):\n iw = (\n min(boxes[n, 2], query_boxes[k, 2]) -\n max(boxes[n, 0], query_boxes[k, 0]) + 1\n )\n if iw > 0:\n ih = (\n min(boxes[n, 3], query_boxes[k, 3]) -\n max(boxes[n, 1], query_boxes[k, 1]) + 1\n )\n if ih > 0:\n intersec[n, k] = iw * ih / box_area\n return intersec"
},
{
"path": "lib/model/utils/blob.py",
"content": "# --------------------------------------------------------\n# Fast R-CNN\n# Copyright (c) 2015 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Written by Ross Girshick\n# --------------------------------------------------------\n\n\"\"\"Blob helper functions.\"\"\"\n\nimport numpy as np\n# from scipy.misc import imread, imresize\nimport cv2\n\ntry:\n xrange # Python 2\nexcept NameError:\n xrange = range # Python 3\n\n\ndef im_list_to_blob(ims):\n \"\"\"Convert a list of images into a network input.\n\n Assumes images are already prepared (means subtracted, BGR order, ...).\n \"\"\"\n max_shape = np.array([im.shape for im in ims]).max(axis=0)\n num_images = len(ims)\n blob = np.zeros((num_images, max_shape[0], max_shape[1], 3),\n dtype=np.float32)\n for i in xrange(num_images):\n im = ims[i]\n blob[i, 0:im.shape[0], 0:im.shape[1], :] = im\n\n return blob\n\ndef prep_im_for_blob(im, pixel_means, target_size, max_size):\n \"\"\"Mean subtract and scale an image for use in a blob.\"\"\"\n\n im = im.astype(np.float32, copy=False)\n im -= pixel_means\n # im = im[:, :, ::-1]\n im_shape = im.shape\n im_size_min = np.min(im_shape[0:2])\n im_size_max = np.max(im_shape[0:2])\n im_scale = float(target_size) / float(im_size_min)\n # Prevent the biggest axis from being more than MAX_SIZE\n # if np.round(im_scale * im_size_max) > max_size:\n # im_scale = float(max_size) / float(im_size_max)\n # im = imresize(im, im_scale)\n im = cv2.resize(im, None, None, fx=im_scale, fy=im_scale,\n interpolation=cv2.INTER_LINEAR)\n\n return im, im_scale\n"
},
{
"path": "lib/model/utils/config.py",
"content": "from __future__ import absolute_import\nfrom __future__ import division\nfrom __future__ import print_function\n\nimport os\nimport os.path as osp\nimport numpy as np\n# `pip install easydict` if you don't have it\nfrom easydict import EasyDict as edict\n\n__C = edict()\n# Consumers can get config by:\n# from fast_rcnn_config import cfg\ncfg = __C\n\n#\n# Training options\n#\n__C.TRAIN = edict()\n\n# Initial learning rate\n__C.TRAIN.LEARNING_RATE = 0.001\n\n# Momentum\n__C.TRAIN.MOMENTUM = 0.9\n\n# Weight decay, for regularization\n__C.TRAIN.WEIGHT_DECAY = 0.0005\n\n# Factor for reducing the learning rate\n__C.TRAIN.GAMMA = 0.1\n\n# Step size for reducing the learning rate, currently only support one step\n__C.TRAIN.STEPSIZE = [30000]\n\n# Iteration intervals for showing the loss during training, on command line interface\n__C.TRAIN.DISPLAY = 10\n\n# Whether to double the learning rate for bias\n__C.TRAIN.DOUBLE_BIAS = True\n\n# Whether to initialize the weights with truncated normal distribution\n__C.TRAIN.TRUNCATED = False\n\n# Whether to have weight decay on bias as well\n__C.TRAIN.BIAS_DECAY = False\n\n# Whether to add ground truth boxes to the pool when sampling regions\n__C.TRAIN.USE_GT = False\n\n# Whether to use aspect-ratio grouping of training images, introduced merely for saving\n# GPU memory\n__C.TRAIN.ASPECT_GROUPING = False\n\n# The number of snapshots kept, older ones are deleted to save space\n__C.TRAIN.SNAPSHOT_KEPT = 3\n\n# The time interval for saving tensorflow summaries\n__C.TRAIN.SUMMARY_INTERVAL = 180\n\n# Scale to use during training (can list multiple scales)\n# The scale is the pixel size of an image's shortest side\n__C.TRAIN.SCALES = (600,)\n\n# Max pixel size of the longest side of a scaled input image\n__C.TRAIN.MAX_SIZE = 1000\n\n# Trim size for input images to create minibatch\n__C.TRAIN.TRIM_HEIGHT = 600\n__C.TRAIN.TRIM_WIDTH = 600\n\n# Images to use per minibatch\n__C.TRAIN.IMS_PER_BATCH = 1\n\n# Minibatch size (number of regions of interest [ROIs])\n__C.TRAIN.BATCH_SIZE = 128\n\n# Fraction of minibatch that is labeled foreground (i.e. class > 0)\n__C.TRAIN.FG_FRACTION = 0.25\n\n# Overlap threshold for a ROI to be considered foreground (if >= FG_THRESH)\n__C.TRAIN.FG_THRESH = 0.5\n\n# Overlap threshold for a ROI to be considered background (class = 0 if\n# overlap in [LO, HI))\n__C.TRAIN.BG_THRESH_HI = 0.5\n__C.TRAIN.BG_THRESH_LO = 0.1\n\n# Use horizontally-flipped images during training?\n__C.TRAIN.USE_FLIPPED = True\n\n# Train bounding-box regressors\n__C.TRAIN.BBOX_REG = True\n\n# Overlap required between a ROI and ground-truth box in order for that ROI to\n# be used as a bounding-box regression training example\n__C.TRAIN.BBOX_THRESH = 0.5\n\n# Iterations between snapshots\n__C.TRAIN.SNAPSHOT_ITERS = 5000\n\n# solver.prototxt specifies the snapshot path prefix, this adds an optional\n# infix to yield the path: [_]_iters_XYZ.caffemodel\n__C.TRAIN.SNAPSHOT_PREFIX = 'res101_faster_rcnn'\n# __C.TRAIN.SNAPSHOT_INFIX = ''\n\n# Use a prefetch thread in roi_data_layer.layer\n# So far I haven't found this useful; likely more engineering work is required\n# __C.TRAIN.USE_PREFETCH = False\n\n# Normalize the targets (subtract empirical mean, divide by empirical stddev)\n__C.TRAIN.BBOX_NORMALIZE_TARGETS = True\n# Deprecated (inside weights)\n__C.TRAIN.BBOX_INSIDE_WEIGHTS = (1.0, 1.0, 1.0, 1.0)\n# Normalize the targets using \"precomputed\" (or made up) means and stdevs\n# (BBOX_NORMALIZE_TARGETS must also be True)\n__C.TRAIN.BBOX_NORMALIZE_TARGETS_PRECOMPUTED = True\n__C.TRAIN.BBOX_NORMALIZE_MEANS = (0.0, 0.0, 0.0, 0.0)\n__C.TRAIN.BBOX_NORMALIZE_STDS = (0.1, 0.1, 0.2, 0.2)\n\n# Train using these proposals\n__C.TRAIN.PROPOSAL_METHOD = 'gt'\n\n# Make minibatches from images that have similar aspect ratios (i.e. both\n# tall and thin or both short and wide) in order to avoid wasting computation\n# on zero-padding.\n\n# Use RPN to detect objects\n__C.TRAIN.HAS_RPN = True\n# IOU >= thresh: positive example\n__C.TRAIN.RPN_POSITIVE_OVERLAP = 0.7\n# IOU < thresh: negative example\n__C.TRAIN.RPN_NEGATIVE_OVERLAP = 0.3\n# If an anchor statisfied by positive and negative conditions set to negative\n__C.TRAIN.RPN_CLOBBER_POSITIVES = False\n# Max number of foreground examples\n__C.TRAIN.RPN_FG_FRACTION = 0.5\n# Total number of examples\n__C.TRAIN.RPN_BATCHSIZE = 256\n# NMS threshold used on RPN proposals\n__C.TRAIN.RPN_NMS_THRESH = 0.7\n# Number of top scoring boxes to keep before apply NMS to RPN proposals\n__C.TRAIN.RPN_PRE_NMS_TOP_N = 12000\n# Number of top scoring boxes to keep after applying NMS to RPN proposals\n__C.TRAIN.RPN_POST_NMS_TOP_N = 2000\n# Proposal height and width both need to be greater than RPN_MIN_SIZE (at orig image scale)\n__C.TRAIN.RPN_MIN_SIZE = 8\n# Deprecated (outside weights)\n__C.TRAIN.RPN_BBOX_INSIDE_WEIGHTS = (1.0, 1.0, 1.0, 1.0)\n# Give the positive RPN examples weight of p * 1 / {num positives}\n# and give negatives a weight of (1 - p)\n# Set to -1.0 to use uniform example weighting\n__C.TRAIN.RPN_POSITIVE_WEIGHT = -1.0\n# Whether to use all ground truth bounding boxes for training,\n# For COCO, setting USE_ALL_GT to False will exclude boxes that are flagged as ''iscrowd''\n__C.TRAIN.USE_ALL_GT = True\n\n# Whether to tune the batch normalization parameters during training\n__C.TRAIN.BN_TRAIN = False\n\n#\n# Testing options\n#\n__C.TEST = edict()\n\n# Scale to use during testing (can NOT list multiple scales)\n# The scale is the pixel size of an image's shortest side\n__C.TEST.SCALES = (600,)\n\n# Max pixel size of the longest side of a scaled input image\n__C.TEST.MAX_SIZE = 1000\n\n# Overlap threshold used for non-maximum suppression (suppress boxes with\n# IoU >= this threshold)\n__C.TEST.NMS = 0.3\n\n# Experimental: treat the (K+1) units in the cls_score layer as linear\n# predictors (trained, eg, with one-vs-rest SVMs).\n__C.TEST.SVM = False\n\n# Test using bounding-box regressors\n__C.TEST.BBOX_REG = True\n\n# Propose boxes\n__C.TEST.HAS_RPN = False\n\n# Test using these proposals\n__C.TEST.PROPOSAL_METHOD = 'gt'\n\n## NMS threshold used on RPN proposals\n__C.TEST.RPN_NMS_THRESH = 0.7\n## Number of top scoring boxes to keep before apply NMS to RPN proposals\n__C.TEST.RPN_PRE_NMS_TOP_N = 6000\n\n## Number of top scoring boxes to keep after applying NMS to RPN proposals\n__C.TEST.RPN_POST_NMS_TOP_N = 300\n\n# Proposal height and width both need to be greater than RPN_MIN_SIZE (at orig image scale)\n__C.TEST.RPN_MIN_SIZE = 16\n\n# Testing mode, default to be 'nms', 'top' is slower but better\n# See report for details\n__C.TEST.MODE = 'nms'\n\n# Only useful when TEST.MODE is 'top', specifies the number of top proposals to select\n__C.TEST.RPN_TOP_N = 5000\n\n#\n# ResNet options\n#\n\n__C.RESNET = edict()\n\n# Option to set if max-pooling is appended after crop_and_resize.\n# if true, the region will be resized to a square of 2xPOOLING_SIZE,\n# then 2x2 max-pooling is applied; otherwise the region will be directly\n# resized to a square of POOLING_SIZE\n__C.RESNET.MAX_POOL = False\n\n# Number of fixed blocks during training, by default the first of all 4 blocks is fixed\n# Range: 0 (none) to 3 (all)\n__C.RESNET.FIXED_BLOCKS = 1\n\n#\n# MobileNet options\n#\n\n__C.MOBILENET = edict()\n\n# Whether to regularize the depth-wise filters during training\n__C.MOBILENET.REGU_DEPTH = False\n\n# Number of fixed layers during training, by default the first of all 14 layers is fixed\n# Range: 0 (none) to 12 (all)\n__C.MOBILENET.FIXED_LAYERS = 5\n\n# Weight decay for the mobilenet weights\n__C.MOBILENET.WEIGHT_DECAY = 0.00004\n\n# Depth multiplier\n__C.MOBILENET.DEPTH_MULTIPLIER = 1.\n\n#\n# MISC\n#\n\n# The mapping from image coordinates to feature map coordinates might cause\n# some boxes that are distinct in image space to become identical in feature\n# coordinates. If DEDUP_BOXES > 0, then DEDUP_BOXES is used as the scale factor\n# for identifying duplicate boxes.\n# 1/16 is correct for {Alex,Caffe}Net, VGG_CNN_M_1024, and VGG16\n__C.DEDUP_BOXES = 1. / 16.\n\n# Pixel mean values (BGR order) as a (1, 1, 3) array\n# We use the same pixel mean for all networks even though it's not exactly what\n# they were trained with\n__C.PIXEL_MEANS = np.array([[[102.9801, 115.9465, 122.7717]]])\n\n# For reproducibility\n__C.RNG_SEED = 3\n\n# A small number that's used many times\n__C.EPS = 1e-14\n\n# Root directory of project\n__C.ROOT_DIR = osp.abspath(osp.join(osp.dirname(__file__), '..', '..', '..'))\n\n# Data directory\n__C.DATA_DIR = osp.abspath(osp.join(__C.ROOT_DIR, 'data'))\n\n# Name (or path to) the matlab executable\n__C.MATLAB = 'matlab'\n\n# Place outputs under an experiments directory\n__C.EXP_DIR = 'default'\n\n# Use GPU implementation of non-maximum suppression\n__C.USE_GPU_NMS = True\n\n# Default GPU device id\n__C.GPU_ID = 0\n\n__C.POOLING_MODE = 'crop'\n\n# Size of the pooled region after RoI pooling\n__C.POOLING_SIZE = 7\n\n# Maximal number of gt rois in an image during Training\n__C.MAX_NUM_GT_BOXES = 20\n\n# Anchor scales for RPN\n__C.ANCHOR_SCALES = [8,16,32]\n\n# Anchor ratios for RPN\n__C.ANCHOR_RATIOS = [0.5,1,2]\n\n# Feature stride for RPN\n__C.FEAT_STRIDE = [16, ]\n\n__C.CUDA = False\n\n__C.CROP_RESIZE_WITH_MAX_POOL = True\n\nimport pdb\ndef get_output_dir(imdb, weights_filename):\n \"\"\"Return the directory where experimental artifacts are placed.\n If the directory does not exist, it is created.\n\n A canonical path is built using the name from an imdb and a network\n (if not None).\n \"\"\"\n outdir = osp.abspath(osp.join(__C.ROOT_DIR, 'output', __C.EXP_DIR, imdb.name))\n if weights_filename is None:\n weights_filename = 'default'\n outdir = osp.join(outdir, weights_filename)\n if not os.path.exists(outdir):\n os.makedirs(outdir)\n return outdir\n\n\ndef get_output_tb_dir(imdb, weights_filename):\n \"\"\"Return the directory where tensorflow summaries are placed.\n If the directory does not exist, it is created.\n\n A canonical path is built using the name from an imdb and a network\n (if not None).\n \"\"\"\n outdir = osp.abspath(osp.join(__C.ROOT_DIR, 'tensorboard', __C.EXP_DIR, imdb.name))\n if weights_filename is None:\n weights_filename = 'default'\n outdir = osp.join(outdir, weights_filename)\n if not os.path.exists(outdir):\n os.makedirs(outdir)\n return outdir\n\n\ndef _merge_a_into_b(a, b):\n \"\"\"Merge config dictionary a into config dictionary b, clobbering the\n options in b whenever they are also specified in a.\n \"\"\"\n if type(a) is not edict:\n return\n\n for k, v in a.items():\n # a must specify keys that are in b\n if k not in b:\n raise KeyError('{} is not a valid config key'.format(k))\n\n # the types must match, too\n old_type = type(b[k])\n if old_type is not type(v):\n if isinstance(b[k], np.ndarray):\n v = np.array(v, dtype=b[k].dtype)\n else:\n raise ValueError(('Type mismatch ({} vs. {}) '\n 'for config key: {}').format(type(b[k]),\n type(v), k))\n\n # recursively merge dicts\n if type(v) is edict:\n try:\n _merge_a_into_b(a[k], b[k])\n except:\n print(('Error under config key: {}'.format(k)))\n raise\n else:\n b[k] = v\n\n\ndef cfg_from_file(filename):\n \"\"\"Load a config file and merge it into the default options.\"\"\"\n import yaml\n with open(filename, 'r') as f:\n yaml_cfg = edict(yaml.load(f))\n\n _merge_a_into_b(yaml_cfg, __C)\n\n\ndef cfg_from_list(cfg_list):\n \"\"\"Set config keys via list (e.g., from command line).\"\"\"\n from ast import literal_eval\n assert len(cfg_list) % 2 == 0\n for k, v in zip(cfg_list[0::2], cfg_list[1::2]):\n key_list = k.split('.')\n d = __C\n for subkey in key_list[:-1]:\n assert subkey in d\n d = d[subkey]\n subkey = key_list[-1]\n assert subkey in d\n try:\n value = literal_eval(v)\n except:\n # handle the case when v is a string literal\n value = v\n assert type(value) == type(d[subkey]), \\\n 'type {} does not match original type {}'.format(\n type(value), type(d[subkey]))\n d[subkey] = value\n"
},
{
"path": "lib/model/utils/logger.py",
"content": "# Code referenced from https://gist.github.com/gyglim/1f8dfb1b5c82627ae3efcfbbadb9f514\nimport tensorflow as tf\nimport numpy as np\nimport scipy.misc \ntry:\n from StringIO import StringIO # Python 2.7\nexcept ImportError:\n from io import BytesIO # Python 3.x\n\n\nclass Logger(object):\n \n def __init__(self, log_dir):\n \"\"\"Create a summary writer logging to log_dir.\"\"\"\n self.writer = tf.summary.FileWriter(log_dir)\n\n def scalar_summary(self, tag, value, step):\n \"\"\"Log a scalar variable.\"\"\"\n summary = tf.Summary(value=[tf.Summary.Value(tag=tag, simple_value=value)])\n self.writer.add_summary(summary, step)\n\n def image_summary(self, tag, images, step):\n \"\"\"Log a list of images.\"\"\"\n\n img_summaries = []\n for i, img in enumerate(images):\n # Write the image to a string\n try:\n s = StringIO()\n except:\n s = BytesIO()\n scipy.misc.toimage(img).save(s, format=\"png\")\n\n # Create an Image object\n img_sum = tf.Summary.Image(encoded_image_string=s.getvalue(),\n height=img.shape[0],\n width=img.shape[1])\n # Create a Summary value\n img_summaries.append(tf.Summary.Value(tag='%s/%d' % (tag, i), image=img_sum))\n\n # Create and write Summary\n summary = tf.Summary(value=img_summaries)\n self.writer.add_summary(summary, step)\n \n def histo_summary(self, tag, values, step, bins=1000):\n \"\"\"Log a histogram of the tensor of values.\"\"\"\n\n # Create a histogram using numpy\n counts, bin_edges = np.histogram(values, bins=bins)\n\n # Fill the fields of the histogram proto\n hist = tf.HistogramProto()\n hist.min = float(np.min(values))\n hist.max = float(np.max(values))\n hist.num = int(np.prod(values.shape))\n hist.sum = float(np.sum(values))\n hist.sum_squares = float(np.sum(values**2))\n\n # Drop the start of the first bin\n bin_edges = bin_edges[1:]\n\n # Add bin edges and counts\n for edge in bin_edges:\n hist.bucket_limit.append(edge)\n for c in counts:\n hist.bucket.append(c)\n\n # Create and write Summary\n summary = tf.Summary(value=[tf.Summary.Value(tag=tag, histo=hist)])\n self.writer.add_summary(summary, step)\n self.writer.flush()\n"
},
{
"path": "lib/model/utils/net_utils.py",
"content": "import torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nfrom torch.autograd import Variable\nimport numpy as np\nimport torchvision.models as models\nfrom model.utils.config import cfg\nfrom model.roi_crop.functions.roi_crop import RoICropFunction\nimport cv2\nimport pdb\nimport random\n\ndef save_net(fname, net):\n import h5py\n h5f = h5py.File(fname, mode='w')\n for k, v in net.state_dict().items():\n h5f.create_dataset(k, data=v.cpu().numpy())\n\ndef load_net(fname, net):\n import h5py\n h5f = h5py.File(fname, mode='r')\n for k, v in net.state_dict().items():\n param = torch.from_numpy(np.asarray(h5f[k]))\n v.copy_(param)\n\ndef weights_normal_init(model, dev=0.01):\n if isinstance(model, list):\n for m in model:\n weights_normal_init(m, dev)\n else:\n for m in model.modules():\n if isinstance(m, nn.Conv2d):\n m.weight.data.normal_(0.0, dev)\n elif isinstance(m, nn.Linear):\n m.weight.data.normal_(0.0, dev)\n\n\ndef clip_gradient(model, clip_norm):\n \"\"\"Computes a gradient clipping coefficient based on gradient norm.\"\"\"\n totalnorm = 0\n for p in model.parameters():\n if p.requires_grad:\n modulenorm = p.grad.data.norm()\n totalnorm += modulenorm ** 2\n totalnorm = torch.sqrt(totalnorm).item()\n norm = (clip_norm / max(totalnorm, clip_norm))\n for p in model.parameters():\n if p.requires_grad:\n p.grad.mul_(norm)\n\ndef vis_detections(im, class_name, dets, thresh=0.8):\n \"\"\"Visual debugging of detections.\"\"\"\n for i in range(np.minimum(10, dets.shape[0])):\n bbox = tuple(int(np.round(x)) for x in dets[i, :4])\n score = dets[i, -1]\n if score > thresh:\n cv2.rectangle(im, bbox[0:2], bbox[2:4], (0, 204, 0), 2)\n cv2.putText(im, '%s: %.3f' % (class_name, score), (bbox[0], bbox[1] + 15), cv2.FONT_HERSHEY_PLAIN,\n 1.0, (0, 0, 255), thickness=1)\n return im\n\n\ndef adjust_learning_rate(optimizer, decay=0.1):\n \"\"\"Sets the learning rate to the initial LR decayed by 0.5 every 20 epochs\"\"\"\n for param_group in optimizer.param_groups:\n param_group['lr'] = decay * param_group['lr']\n\n\ndef save_checkpoint(state, filename):\n torch.save(state, filename)\n\ndef _smooth_l1_loss(bbox_pred, bbox_targets, bbox_inside_weights, bbox_outside_weights, sigma=1.0, dim=[1]):\n \n sigma_2 = sigma ** 2\n box_diff = bbox_pred - bbox_targets\n in_box_diff = bbox_inside_weights * box_diff\n abs_in_box_diff = torch.abs(in_box_diff)\n smoothL1_sign = (abs_in_box_diff < 1. / sigma_2).detach().float()\n in_loss_box = torch.pow(in_box_diff, 2) * (sigma_2 / 2.) * smoothL1_sign \\\n + (abs_in_box_diff - (0.5 / sigma_2)) * (1. - smoothL1_sign)\n out_loss_box = bbox_outside_weights * in_loss_box\n loss_box = out_loss_box\n for i in sorted(dim, reverse=True):\n loss_box = loss_box.sum(i)\n loss_box = loss_box.mean()\n return loss_box\n\ndef _crop_pool_layer(bottom, rois, max_pool=True):\n # code modified from \n # https://github.com/ruotianluo/pytorch-faster-rcnn\n # implement it using stn\n # box to affine\n # input (x1,y1,x2,y2)\n \"\"\"\n [ x2-x1 x1 + x2 - W + 1 ]\n [ ----- 0 --------------- ]\n [ W - 1 W - 1 ]\n [ ]\n [ y2-y1 y1 + y2 - H + 1 ]\n [ 0 ----- --------------- ]\n [ H - 1 H - 1 ]\n \"\"\"\n rois = rois.detach()\n batch_size = bottom.size(0)\n D = bottom.size(1)\n H = bottom.size(2)\n W = bottom.size(3)\n roi_per_batch = rois.size(0) / batch_size\n x1 = rois[:, 1::4] / 16.0\n y1 = rois[:, 2::4] / 16.0\n x2 = rois[:, 3::4] / 16.0\n y2 = rois[:, 4::4] / 16.0\n\n height = bottom.size(2)\n width = bottom.size(3)\n\n # affine theta\n zero = Variable(rois.data.new(rois.size(0), 1).zero_())\n theta = torch.cat([\\\n (x2 - x1) / (width - 1),\n zero,\n (x1 + x2 - width + 1) / (width - 1),\n zero,\n (y2 - y1) / (height - 1),\n (y1 + y2 - height + 1) / (height - 1)], 1).view(-1, 2, 3)\n\n if max_pool:\n pre_pool_size = cfg.POOLING_SIZE * 2\n grid = F.affine_grid(theta, torch.Size((rois.size(0), 1, pre_pool_size, pre_pool_size)))\n bottom = bottom.view(1, batch_size, D, H, W).contiguous().expand(roi_per_batch, batch_size, D, H, W)\\\n .contiguous().view(-1, D, H, W)\n crops = F.grid_sample(bottom, grid)\n crops = F.max_pool2d(crops, 2, 2)\n else:\n grid = F.affine_grid(theta, torch.Size((rois.size(0), 1, cfg.POOLING_SIZE, cfg.POOLING_SIZE)))\n bottom = bottom.view(1, batch_size, D, H, W).contiguous().expand(roi_per_batch, batch_size, D, H, W)\\\n .contiguous().view(-1, D, H, W)\n crops = F.grid_sample(bottom, grid)\n \n return crops, grid\n\ndef _affine_grid_gen(rois, input_size, grid_size):\n\n rois = rois.detach()\n x1 = rois[:, 1::4] / 16.0\n y1 = rois[:, 2::4] / 16.0\n x2 = rois[:, 3::4] / 16.0\n y2 = rois[:, 4::4] / 16.0\n\n height = input_size[0]\n width = input_size[1]\n\n zero = Variable(rois.data.new(rois.size(0), 1).zero_())\n theta = torch.cat([\\\n (x2 - x1) / (width - 1),\n zero,\n (x1 + x2 - width + 1) / (width - 1),\n zero,\n (y2 - y1) / (height - 1),\n (y1 + y2 - height + 1) / (height - 1)], 1).view(-1, 2, 3)\n\n grid = F.affine_grid(theta, torch.Size((rois.size(0), 1, grid_size, grid_size)))\n\n return grid\n\ndef _affine_theta(rois, input_size):\n\n rois = rois.detach()\n x1 = rois[:, 1::4] / 16.0\n y1 = rois[:, 2::4] / 16.0\n x2 = rois[:, 3::4] / 16.0\n y2 = rois[:, 4::4] / 16.0\n\n height = input_size[0]\n width = input_size[1]\n\n zero = Variable(rois.data.new(rois.size(0), 1).zero_())\n\n # theta = torch.cat([\\\n # (x2 - x1) / (width - 1),\n # zero,\n # (x1 + x2 - width + 1) / (width - 1),\n # zero,\n # (y2 - y1) / (height - 1),\n # (y1 + y2 - height + 1) / (height - 1)], 1).view(-1, 2, 3)\n\n theta = torch.cat([\\\n (y2 - y1) / (height - 1),\n zero,\n (y1 + y2 - height + 1) / (height - 1),\n zero,\n (x2 - x1) / (width - 1),\n (x1 + x2 - width + 1) / (width - 1)], 1).view(-1, 2, 3)\n\n return theta\n\ndef compare_grid_sample():\n # do gradcheck\n N = random.randint(1, 8)\n C = 2 # random.randint(1, 8)\n H = 5 # random.randint(1, 8)\n W = 4 # random.randint(1, 8)\n input = Variable(torch.randn(N, C, H, W).cuda(), requires_grad=True)\n input_p = input.clone().data.contiguous()\n \n grid = Variable(torch.randn(N, H, W, 2).cuda(), requires_grad=True)\n grid_clone = grid.clone().contiguous()\n\n out_offcial = F.grid_sample(input, grid) \n grad_outputs = Variable(torch.rand(out_offcial.size()).cuda())\n grad_outputs_clone = grad_outputs.clone().contiguous()\n grad_inputs = torch.autograd.grad(out_offcial, (input, grid), grad_outputs.contiguous())\n grad_input_off = grad_inputs[0]\n\n\n crf = RoICropFunction()\n grid_yx = torch.stack([grid_clone.data[:,:,:,1], grid_clone.data[:,:,:,0]], 3).contiguous().cuda()\n out_stn = crf.forward(input_p, grid_yx)\n grad_inputs = crf.backward(grad_outputs_clone.data)\n grad_input_stn = grad_inputs[0]\n pdb.set_trace()\n\n delta = (grad_input_off.data - grad_input_stn).sum()\n"
},
{
"path": "lib/pycocotools/UPSTREAM_REV",
"content": "https://github.com/pdollar/coco/commit/3ac47c77ebd5a1ed4254a98b7fbf2ef4765a3574\n"
},
{
"path": "lib/pycocotools/__init__.py",
"content": "__author__ = 'tylin'\n"
},
{
"path": "lib/pycocotools/_mask.pyx",
"content": "# distutils: language = c\n# distutils: sources = ../MatlabAPI/private/maskApi.c\n\n#**************************************************************************\n# Microsoft COCO Toolbox. version 2.0\n# Data, paper, and tutorials available at: http://mscoco.org/\n# Code written by Piotr Dollar and Tsung-Yi Lin, 2015.\n# Licensed under the Simplified BSD License [see coco/license.txt]\n#**************************************************************************\n\n__author__ = 'tsungyi'\n\n# import both Python-level and C-level symbols of Numpy\n# the API uses Numpy to interface C and Python\nimport numpy as np\ncimport numpy as np\nfrom libc.stdlib cimport malloc, free\n\n# intialized Numpy. must do.\nnp.import_array()\n\n# import numpy C function\n# we use PyArray_ENABLEFLAGS to make Numpy ndarray responsible to memoery management\ncdef extern from \"numpy/arrayobject.h\":\n void PyArray_ENABLEFLAGS(np.ndarray arr, int flags)\n\n# Declare the prototype of the C functions in MaskApi.h\ncdef extern from \"maskApi.h\":\n ctypedef unsigned int uint\n ctypedef unsigned long siz\n ctypedef unsigned char byte\n ctypedef double* BB\n ctypedef struct RLE:\n siz h,\n siz w,\n siz m,\n uint* cnts,\n void rlesInit( RLE **R, siz n )\n void rleEncode( RLE *R, const byte *M, siz h, siz w, siz n )\n void rleDecode( const RLE *R, byte *mask, siz n )\n void rleMerge( const RLE *R, RLE *M, siz n, bint intersect )\n void rleArea( const RLE *R, siz n, uint *a )\n void rleIou( RLE *dt, RLE *gt, siz m, siz n, byte *iscrowd, double *o )\n void bbIou( BB dt, BB gt, siz m, siz n, byte *iscrowd, double *o )\n void rleToBbox( const RLE *R, BB bb, siz n )\n void rleFrBbox( RLE *R, const BB bb, siz h, siz w, siz n )\n void rleFrPoly( RLE *R, const double *xy, siz k, siz h, siz w )\n char* rleToString( const RLE *R )\n void rleFrString( RLE *R, char *s, siz h, siz w )\n\n# python class to wrap RLE array in C\n# the class handles the memory allocation and deallocation\ncdef class RLEs:\n cdef RLE *_R\n cdef siz _n\n\n def __cinit__(self, siz n =0):\n rlesInit(&self._R, n)\n self._n = n\n\n # free the RLE array here\n def __dealloc__(self):\n if self._R is not NULL:\n for i in range(self._n):\n free(self._R[i].cnts)\n free(self._R)\n def __getattr__(self, key):\n if key == 'n':\n return self._n\n raise AttributeError(key)\n\n# python class to wrap Mask array in C\n# the class handles the memory allocation and deallocation\ncdef class Masks:\n cdef byte *_mask\n cdef siz _h\n cdef siz _w\n cdef siz _n\n\n def __cinit__(self, h, w, n):\n self._mask = malloc(h*w*n* sizeof(byte))\n self._h = h\n self._w = w\n self._n = n\n # def __dealloc__(self):\n # the memory management of _mask has been passed to np.ndarray\n # it doesn't need to be freed here\n\n # called when passing into np.array() and return an np.ndarray in column-major order\n def __array__(self):\n cdef np.npy_intp shape[1]\n shape[0] = self._h*self._w*self._n\n # Create a 1D array, and reshape it to fortran/Matlab column-major array\n ndarray = np.PyArray_SimpleNewFromData(1, shape, np.NPY_UINT8, self._mask).reshape((self._h, self._w, self._n), order='F')\n # The _mask allocated by Masks is now handled by ndarray\n PyArray_ENABLEFLAGS(ndarray, np.NPY_OWNDATA)\n return ndarray\n\n# internal conversion from Python RLEs object to compressed RLE format\ndef _toString(RLEs Rs):\n cdef siz n = Rs.n\n cdef bytes py_string\n cdef char* c_string\n objs = []\n for i in range(n):\n c_string = rleToString( &Rs._R[i] )\n py_string = c_string\n objs.append({\n 'size': [Rs._R[i].h, Rs._R[i].w],\n 'counts': py_string\n })\n free(c_string)\n return objs\n\n# internal conversion from compressed RLE format to Python RLEs object\ndef _frString(rleObjs):\n cdef siz n = len(rleObjs)\n Rs = RLEs(n)\n cdef bytes py_string\n cdef char* c_string\n for i, obj in enumerate(rleObjs):\n py_string = str(obj['counts'])\n c_string = py_string\n rleFrString( &Rs._R[i], c_string, obj['size'][0], obj['size'][1] )\n return Rs\n\n# encode mask to RLEs objects\n# list of RLE string can be generated by RLEs member function\ndef encode(np.ndarray[np.uint8_t, ndim=3, mode='fortran'] mask):\n h, w, n = mask.shape[0], mask.shape[1], mask.shape[2]\n cdef RLEs Rs = RLEs(n)\n rleEncode(Rs._R,mask.data,h,w,n)\n objs = _toString(Rs)\n return objs\n\n# decode mask from compressed list of RLE string or RLEs object\ndef decode(rleObjs):\n cdef RLEs Rs = _frString(rleObjs)\n h, w, n = Rs._R[0].h, Rs._R[0].w, Rs._n\n masks = Masks(h, w, n)\n rleDecode( Rs._R, masks._mask, n );\n return np.array(masks)\n\ndef merge(rleObjs, bint intersect=0):\n cdef RLEs Rs = _frString(rleObjs)\n cdef RLEs R = RLEs(1)\n rleMerge(Rs._R, R._R, Rs._n, intersect)\n obj = _toString(R)[0]\n return obj\n\ndef area(rleObjs):\n cdef RLEs Rs = _frString(rleObjs)\n cdef uint* _a = malloc(Rs._n* sizeof(uint))\n rleArea(Rs._R, Rs._n, _a)\n cdef np.npy_intp shape[1]\n shape[0] = Rs._n\n a = np.array((Rs._n, ), dtype=np.uint8)\n a = np.PyArray_SimpleNewFromData(1, shape, np.NPY_UINT32, _a)\n PyArray_ENABLEFLAGS(a, np.NPY_OWNDATA)\n return a\n\n# iou computation. support function overload (RLEs-RLEs and bbox-bbox).\ndef iou( dt, gt, pyiscrowd ):\n def _preproc(objs):\n if len(objs) == 0:\n return objs\n if type(objs) == np.ndarray:\n if len(objs.shape) == 1:\n objs = objs.reshape((objs[0], 1))\n # check if it's Nx4 bbox\n if not len(objs.shape) == 2 or not objs.shape[1] == 4:\n raise Exception('numpy ndarray input is only for *bounding boxes* and should have Nx4 dimension')\n objs = objs.astype(np.double)\n elif type(objs) == list:\n # check if list is in box format and convert it to np.ndarray\n isbox = np.all(np.array([(len(obj)==4) and ((type(obj)==list) or (type(obj)==np.ndarray)) for obj in objs]))\n isrle = np.all(np.array([type(obj) == dict for obj in objs]))\n if isbox:\n objs = np.array(objs, dtype=np.double)\n if len(objs.shape) == 1:\n objs = objs.reshape((1,objs.shape[0]))\n elif isrle:\n objs = _frString(objs)\n else:\n raise Exception('list input can be bounding box (Nx4) or RLEs ([RLE])')\n else:\n raise Exception('unrecognized type. The following type: RLEs (rle), np.ndarray (box), and list (box) are supported.')\n return objs\n def _rleIou(RLEs dt, RLEs gt, np.ndarray[np.uint8_t, ndim=1] iscrowd, siz m, siz n, np.ndarray[np.double_t, ndim=1] _iou):\n rleIou( dt._R, gt._R, m, n, iscrowd.data, _iou.data )\n def _bbIou(np.ndarray[np.double_t, ndim=2] dt, np.ndarray[np.double_t, ndim=2] gt, np.ndarray[np.uint8_t, ndim=1] iscrowd, siz m, siz n, np.ndarray[np.double_t, ndim=1] _iou):\n bbIou( dt.data, gt.data, m, n, iscrowd.data, _iou.data )\n def _len(obj):\n cdef siz N = 0\n if type(obj) == RLEs:\n N = obj.n\n elif len(obj)==0:\n pass\n elif type(obj) == np.ndarray:\n N = obj.shape[0]\n return N\n # convert iscrowd to numpy array\n cdef np.ndarray[np.uint8_t, ndim=1] iscrowd = np.array(pyiscrowd, dtype=np.uint8)\n # simple type checking\n cdef siz m, n\n dt = _preproc(dt)\n gt = _preproc(gt)\n m = _len(dt)\n n = _len(gt)\n if m == 0 or n == 0:\n return []\n if not type(dt) == type(gt):\n raise Exception('The dt and gt should have the same data type, either RLEs, list or np.ndarray')\n\n # define local variables\n cdef double* _iou = 0\n cdef np.npy_intp shape[1]\n # check type and assign iou function\n if type(dt) == RLEs:\n _iouFun = _rleIou\n elif type(dt) == np.ndarray:\n _iouFun = _bbIou\n else:\n raise Exception('input data type not allowed.')\n _iou = malloc(m*n* sizeof(double))\n iou = np.zeros((m*n, ), dtype=np.double)\n shape[0] = m*n\n iou = np.PyArray_SimpleNewFromData(1, shape, np.NPY_DOUBLE, _iou)\n PyArray_ENABLEFLAGS(iou, np.NPY_OWNDATA)\n _iouFun(dt, gt, iscrowd, m, n, iou)\n return iou.reshape((m,n), order='F')\n\ndef toBbox( rleObjs ):\n cdef RLEs Rs = _frString(rleObjs)\n cdef siz n = Rs.n\n cdef BB _bb = malloc(4*n* sizeof(double))\n rleToBbox( Rs._R, _bb, n )\n cdef np.npy_intp shape[1]\n shape[0] = 4*n\n bb = np.array((1,4*n), dtype=np.double)\n bb = np.PyArray_SimpleNewFromData(1, shape, np.NPY_DOUBLE, _bb).reshape((n, 4))\n PyArray_ENABLEFLAGS(bb, np.NPY_OWNDATA)\n return bb\n\ndef frBbox(np.ndarray[np.double_t, ndim=2] bb, siz h, siz w ):\n cdef siz n = bb.shape[0]\n Rs = RLEs(n)\n rleFrBbox( Rs._R, bb.data, h, w, n )\n objs = _toString(Rs)\n return objs\n\ndef frPoly( poly, siz h, siz w ):\n cdef np.ndarray[np.double_t, ndim=1] np_poly\n n = len(poly)\n Rs = RLEs(n)\n for i, p in enumerate(poly):\n np_poly = np.array(p, dtype=np.double, order='F')\n rleFrPoly( &Rs._R[i], np_poly.data, len(np_poly)/2, h, w )\n objs = _toString(Rs)\n return objs\n\ndef frUncompressedRLE(ucRles, siz h, siz w):\n cdef np.ndarray[np.uint32_t, ndim=1] cnts\n cdef RLE R\n cdef uint *data\n n = len(ucRles)\n objs = []\n for i in range(n):\n Rs = RLEs(1)\n cnts = np.array(ucRles[i]['counts'], dtype=np.uint32)\n # time for malloc can be saved here but it's fine\n data = malloc(len(cnts)* sizeof(uint))\n for j in range(len(cnts)):\n data[j] = cnts[j]\n R = RLE(ucRles[i]['size'][0], ucRles[i]['size'][1], len(cnts), data)\n Rs._R[0] = R\n objs.append(_toString(Rs)[0])\n return objs\n\ndef frPyObjects(pyobj, siz h, w):\n if type(pyobj) == np.ndarray:\n objs = frBbox(pyobj, h, w )\n elif type(pyobj) == list and len(pyobj[0]) == 4:\n objs = frBbox(pyobj, h, w )\n elif type(pyobj) == list and len(pyobj[0]) > 4:\n objs = frPoly(pyobj, h, w )\n elif type(pyobj) == list and type(pyobj[0]) == dict:\n objs = frUncompressedRLE(pyobj, h, w)\n else:\n raise Exception('input type is not supported.')\n return objs\n"
},
{
"path": "lib/pycocotools/coco.py",
"content": "from __future__ import print_function\nfrom __future__ import absolute_import\n__author__ = 'tylin'\n__version__ = '1.0.1'\n# Interface for accessing the Microsoft COCO dataset.\n\n# Microsoft COCO is a large image dataset designed for object detection,\n# segmentation, and caption generation. pycocotools is a Python API that\n# assists in loading, parsing and visualizing the annotations in COCO.\n# Please visit http://mscoco.org/ for more information on COCO, including\n# for the data, paper, and tutorials. The exact format of the annotations\n# is also described on the COCO website. For example usage of the pycocotools\n# please see pycocotools_demo.ipynb. In addition to this API, please download both\n# the COCO images and annotations in order to run the demo.\n\n# An alternative to using the API is to load the annotations directly\n# into Python dictionary\n# Using the API provides additional utility functions. Note that this API\n# supports both *instance* and *caption* annotations. In the case of\n# captions not all functions are defined (e.g. categories are undefined).\n\n# The following API functions are defined:\n# COCO - COCO api class that loads COCO annotation file and prepare data structures.\n# decodeMask - Decode binary mask M encoded via run-length encoding.\n# encodeMask - Encode binary mask M using run-length encoding.\n# getAnnIds - Get ann ids that satisfy given filter conditions.\n# getCatIds - Get cat ids that satisfy given filter conditions.\n# getImgIds - Get img ids that satisfy given filter conditions.\n# loadAnns - Load anns with the specified ids.\n# loadCats - Load cats with the specified ids.\n# loadImgs - Load imgs with the specified ids.\n# segToMask - Convert polygon segmentation to binary mask.\n# showAnns - Display the specified annotations.\n# loadRes - Load algorithm results and create API for accessing them.\n# download - Download COCO images from mscoco.org server.\n# Throughout the API \"ann\"=annotation, \"cat\"=category, and \"img\"=image.\n# Help on each functions can be accessed by: \"help COCO>function\".\n\n# See also COCO>decodeMask,\n# COCO>encodeMask, COCO>getAnnIds, COCO>getCatIds,\n# COCO>getImgIds, COCO>loadAnns, COCO>loadCats,\n# COCO>loadImgs, COCO>segToMask, COCO>showAnns\n\n# Microsoft COCO Toolbox. version 2.0\n# Data, paper, and tutorials available at: http://mscoco.org/\n# Code written by Piotr Dollar and Tsung-Yi Lin, 2014.\n# Licensed under the Simplified BSD License [see bsd.txt]\n\nimport json\nimport datetime\nimport time\nimport matplotlib.pyplot as plt\nfrom matplotlib.collections import PatchCollection\nfrom matplotlib.patches import Polygon\nimport numpy as np\n# from skimage.draw import polygon\nimport urllib\nimport copy\nimport itertools\nfrom . import mask\nimport os\ntry:\n unicode # Python 2\nexcept NameError:\n unicode = str # Python 3\n \nclass COCO:\n def __init__(self, annotation_file=None):\n \"\"\"\n Constructor of Microsoft COCO helper class for reading and visualizing annotations.\n :param annotation_file (str): location of annotation file\n :param image_folder (str): location to the folder that hosts images.\n :return:\n \"\"\"\n # load dataset\n self.dataset = {}\n self.anns = []\n self.imgToAnns = {}\n self.catToImgs = {}\n self.imgs = {}\n self.cats = {}\n if not annotation_file == None:\n print('loading annotations into memory...')\n tic = time.time()\n dataset = json.load(open(annotation_file, 'r'))\n print('Done (t=%0.2fs)'%(time.time()- tic))\n self.dataset = dataset\n self.createIndex()\n\n def createIndex(self):\n # create index\n print('creating index...')\n anns = {}\n imgToAnns = {}\n catToImgs = {}\n cats = {}\n imgs = {}\n if 'annotations' in self.dataset:\n imgToAnns = {ann['image_id']: [] for ann in self.dataset['annotations']}\n anns = {ann['id']: [] for ann in self.dataset['annotations']}\n for ann in self.dataset['annotations']:\n imgToAnns[ann['image_id']] += [ann]\n anns[ann['id']] = ann\n\n if 'images' in self.dataset:\n imgs = {im['id']: {} for im in self.dataset['images']}\n for img in self.dataset['images']:\n imgs[img['id']] = img\n\n if 'categories' in self.dataset:\n cats = {cat['id']: [] for cat in self.dataset['categories']}\n for cat in self.dataset['categories']:\n cats[cat['id']] = cat\n catToImgs = {cat['id']: [] for cat in self.dataset['categories']}\n if 'annotations' in self.dataset:\n for ann in self.dataset['annotations']:\n catToImgs[ann['category_id']] += [ann['image_id']]\n\n print('index created!')\n\n # create class members\n self.anns = anns\n self.imgToAnns = imgToAnns\n self.catToImgs = catToImgs\n self.imgs = imgs\n self.cats = cats\n\n def info(self):\n \"\"\"\n Print information about the annotation file.\n :return:\n \"\"\"\n for key, value in self.dataset['info'].items():\n print('%s: %s'%(key, value))\n\n def getAnnIds(self, imgIds=[], catIds=[], areaRng=[], iscrowd=None):\n \"\"\"\n Get ann ids that satisfy given filter conditions. default skips that filter\n :param imgIds (int array) : get anns for given imgs\n catIds (int array) : get anns for given cats\n areaRng (float array) : get anns for given area range (e.g. [0 inf])\n iscrowd (boolean) : get anns for given crowd label (False or True)\n :return: ids (int array) : integer array of ann ids\n \"\"\"\n imgIds = imgIds if type(imgIds) == list else [imgIds]\n catIds = catIds if type(catIds) == list else [catIds]\n\n if len(imgIds) == len(catIds) == len(areaRng) == 0:\n anns = self.dataset['annotations']\n else:\n if not len(imgIds) == 0:\n # this can be changed by defaultdict\n lists = [self.imgToAnns[imgId] for imgId in imgIds if imgId in self.imgToAnns]\n anns = list(itertools.chain.from_iterable(lists))\n else:\n anns = self.dataset['annotations']\n anns = anns if len(catIds) == 0 else [ann for ann in anns if ann['category_id'] in catIds]\n anns = anns if len(areaRng) == 0 else [ann for ann in anns if ann['area'] > areaRng[0] and ann['area'] < areaRng[1]]\n if not iscrowd == None:\n ids = [ann['id'] for ann in anns if ann['iscrowd'] == iscrowd]\n else:\n ids = [ann['id'] for ann in anns]\n return ids\n\n def getCatIds(self, catNms=[], supNms=[], catIds=[]):\n \"\"\"\n filtering parameters. default skips that filter.\n :param catNms (str array) : get cats for given cat names\n :param supNms (str array) : get cats for given supercategory names\n :param catIds (int array) : get cats for given cat ids\n :return: ids (int array) : integer array of cat ids\n \"\"\"\n catNms = catNms if type(catNms) == list else [catNms]\n supNms = supNms if type(supNms) == list else [supNms]\n catIds = catIds if type(catIds) == list else [catIds]\n\n if len(catNms) == len(supNms) == len(catIds) == 0:\n cats = self.dataset['categories']\n else:\n cats = self.dataset['categories']\n cats = cats if len(catNms) == 0 else [cat for cat in cats if cat['name'] in catNms]\n cats = cats if len(supNms) == 0 else [cat for cat in cats if cat['supercategory'] in supNms]\n cats = cats if len(catIds) == 0 else [cat for cat in cats if cat['id'] in catIds]\n ids = [cat['id'] for cat in cats]\n return ids\n\n def getImgIds(self, imgIds=[], catIds=[]):\n '''\n Get img ids that satisfy given filter conditions.\n :param imgIds (int array) : get imgs for given ids\n :param catIds (int array) : get imgs with all given cats\n :return: ids (int array) : integer array of img ids\n '''\n imgIds = imgIds if type(imgIds) == list else [imgIds]\n catIds = catIds if type(catIds) == list else [catIds]\n\n if len(imgIds) == len(catIds) == 0:\n ids = self.imgs.keys()\n else:\n ids = set(imgIds)\n for i, catId in enumerate(catIds):\n if i == 0 and len(ids) == 0:\n ids = set(self.catToImgs[catId])\n else:\n ids &= set(self.catToImgs[catId])\n return list(ids)\n\n def loadAnns(self, ids=[]):\n \"\"\"\n Load anns with the specified ids.\n :param ids (int array) : integer ids specifying anns\n :return: anns (object array) : loaded ann objects\n \"\"\"\n if type(ids) == list:\n return [self.anns[id] for id in ids]\n elif type(ids) == int:\n return [self.anns[ids]]\n\n def loadCats(self, ids=[]):\n \"\"\"\n Load cats with the specified ids.\n :param ids (int array) : integer ids specifying cats\n :return: cats (object array) : loaded cat objects\n \"\"\"\n if type(ids) == list:\n return [self.cats[id] for id in ids]\n elif type(ids) == int:\n return [self.cats[ids]]\n\n def loadImgs(self, ids=[]):\n \"\"\"\n Load anns with the specified ids.\n :param ids (int array) : integer ids specifying img\n :return: imgs (object array) : loaded img objects\n \"\"\"\n if type(ids) == list:\n return [self.imgs[id] for id in ids]\n elif type(ids) == int:\n return [self.imgs[ids]]\n\n def showAnns(self, anns):\n \"\"\"\n Display the specified annotations.\n :param anns (array of object): annotations to display\n :return: None\n \"\"\"\n if len(anns) == 0:\n return 0\n if 'segmentation' in anns[0]:\n datasetType = 'instances'\n elif 'caption' in anns[0]:\n datasetType = 'captions'\n if datasetType == 'instances':\n ax = plt.gca()\n polygons = []\n color = []\n for ann in anns:\n c = np.random.random((1, 3)).tolist()[0]\n if type(ann['segmentation']) == list:\n # polygon\n for seg in ann['segmentation']:\n poly = np.array(seg).reshape((len(seg)/2, 2))\n polygons.append(Polygon(poly, True,alpha=0.4))\n color.append(c)\n else:\n # mask\n t = self.imgs[ann['image_id']]\n if type(ann['segmentation']['counts']) == list:\n rle = mask.frPyObjects([ann['segmentation']], t['height'], t['width'])\n else:\n rle = [ann['segmentation']]\n m = mask.decode(rle)\n img = np.ones( (m.shape[0], m.shape[1], 3) )\n if ann['iscrowd'] == 1:\n color_mask = np.array([2.0,166.0,101.0])/255\n if ann['iscrowd'] == 0:\n color_mask = np.random.random((1, 3)).tolist()[0]\n for i in range(3):\n img[:,:,i] = color_mask[i]\n ax.imshow(np.dstack( (img, m*0.5) ))\n p = PatchCollection(polygons, facecolors=color, edgecolors=(0,0,0,1), linewidths=3, alpha=0.4)\n ax.add_collection(p)\n elif datasetType == 'captions':\n for ann in anns:\n print(ann['caption'])\n\n def loadRes(self, resFile):\n \"\"\"\n Load result file and return a result api object.\n :param resFile (str) : file name of result file\n :return: res (obj) : result api object\n \"\"\"\n res = COCO()\n res.dataset['images'] = [img for img in self.dataset['images']]\n # res.dataset['info'] = copy.deepcopy(self.dataset['info'])\n # res.dataset['licenses'] = copy.deepcopy(self.dataset['licenses'])\n\n print('Loading and preparing results... ')\n tic = time.time()\n anns = json.load(open(resFile))\n assert type(anns) == list, 'results in not an array of objects'\n annsImgIds = [ann['image_id'] for ann in anns]\n assert set(annsImgIds) == (set(annsImgIds) & set(self.getImgIds())), \\\n 'Results do not correspond to current coco set'\n if 'caption' in anns[0]:\n imgIds = set([img['id'] for img in res.dataset['images']]) & set([ann['image_id'] for ann in anns])\n res.dataset['images'] = [img for img in res.dataset['images'] if img['id'] in imgIds]\n for id, ann in enumerate(anns):\n ann['id'] = id+1\n elif 'bbox' in anns[0] and not anns[0]['bbox'] == []:\n res.dataset['categories'] = copy.deepcopy(self.dataset['categories'])\n for id, ann in enumerate(anns):\n bb = ann['bbox']\n x1, x2, y1, y2 = [bb[0], bb[0]+bb[2], bb[1], bb[1]+bb[3]]\n if not 'segmentation' in ann:\n ann['segmentation'] = [[x1, y1, x1, y2, x2, y2, x2, y1]]\n ann['area'] = bb[2]*bb[3]\n ann['id'] = id+1\n ann['iscrowd'] = 0\n elif 'segmentation' in anns[0]:\n res.dataset['categories'] = copy.deepcopy(self.dataset['categories'])\n for id, ann in enumerate(anns):\n # now only support compressed RLE format as segmentation results\n ann['area'] = mask.area([ann['segmentation']])[0]\n if not 'bbox' in ann:\n ann['bbox'] = mask.toBbox([ann['segmentation']])[0]\n ann['id'] = id+1\n ann['iscrowd'] = 0\n print('DONE (t=%0.2fs)'%(time.time()- tic))\n\n res.dataset['annotations'] = anns\n res.createIndex()\n return res\n\n def download( self, tarDir = None, imgIds = [] ):\n '''\n Download COCO images from mscoco.org server.\n :param tarDir (str): COCO results directory name\n imgIds (list): images to be downloaded\n :return:\n '''\n if tarDir is None:\n print('Please specify target directory')\n return -1\n if len(imgIds) == 0:\n imgs = self.imgs.values()\n else:\n imgs = self.loadImgs(imgIds)\n N = len(imgs)\n if not os.path.exists(tarDir):\n os.makedirs(tarDir)\n for i, img in enumerate(imgs):\n tic = time.time()\n fname = os.path.join(tarDir, img['file_name'])\n if not os.path.exists(fname):\n urllib.urlretrieve(img['coco_url'], fname)\n print('downloaded %d/%d images (t=%.1fs)'%(i, N, time.time()- tic))\n"
},
{
"path": "lib/pycocotools/cocoeval.py",
"content": "from __future__ import print_function\nfrom __future__ import absolute_import\n__author__ = 'tsungyi'\n\nimport numpy as np\nimport datetime\nimport time\nfrom collections import defaultdict\nfrom . import mask\nimport copy\n\ntry:\n string_types = (str, unicode) # Python 2\nexcept NameError:\n string_types = (str, ) # Python 3\n\n\nclass COCOeval:\n # Interface for evaluating detection on the Microsoft COCO dataset.\n #\n # The usage for CocoEval is as follows:\n # cocoGt=..., cocoDt=... # load dataset and results\n # E = CocoEval(cocoGt,cocoDt); # initialize CocoEval object\n # E.params.recThrs = ...; # set parameters as desired\n # E.evaluate(); # run per image evaluation\n # E.accumulate(); # accumulate per image results\n # E.summarize(); # display summary metrics of results\n # For example usage see evalDemo.m and http://mscoco.org/.\n #\n # The evaluation parameters are as follows (defaults in brackets):\n # imgIds - [all] N img ids to use for evaluation\n # catIds - [all] K cat ids to use for evaluation\n # iouThrs - [.5:.05:.95] T=10 IoU thresholds for evaluation\n # recThrs - [0:.01:1] R=101 recall thresholds for evaluation\n # areaRng - [...] A=4 object area ranges for evaluation\n # maxDets - [1 10 100] M=3 thresholds on max detections per image\n # useSegm - [1] if true evaluate against ground-truth segments\n # useCats - [1] if true use category labels for evaluation # Note: if useSegm=0 the evaluation is run on bounding boxes.\n # Note: if useCats=0 category labels are ignored as in proposal scoring.\n # Note: multiple areaRngs [Ax2] and maxDets [Mx1] can be specified.\n #\n # evaluate(): evaluates detections on every image and every category and\n # concats the results into the \"evalImgs\" with fields:\n # dtIds - [1xD] id for each of the D detections (dt)\n # gtIds - [1xG] id for each of the G ground truths (gt)\n # dtMatches - [TxD] matching gt id at each IoU or 0\n # gtMatches - [TxG] matching dt id at each IoU or 0\n # dtScores - [1xD] confidence of each dt\n # gtIgnore - [1xG] ignore flag for each gt\n # dtIgnore - [TxD] ignore flag for each dt at each IoU\n #\n # accumulate(): accumulates the per-image, per-category evaluation\n # results in \"evalImgs\" into the dictionary \"eval\" with fields:\n # params - parameters used for evaluation\n # date - date evaluation was performed\n # counts - [T,R,K,A,M] parameter dimensions (see above)\n # precision - [TxRxKxAxM] precision for every evaluation setting\n # recall - [TxKxAxM] max recall for every evaluation setting\n # Note: precision and recall==-1 for settings with no gt objects.\n #\n # See also coco, mask, pycocoDemo, pycocoEvalDemo\n #\n # Microsoft COCO Toolbox. version 2.0\n # Data, paper, and tutorials available at: http://mscoco.org/\n # Code written by Piotr Dollar and Tsung-Yi Lin, 2015.\n # Licensed under the Simplified BSD License [see coco/license.txt]\n def __init__(self, cocoGt=None, cocoDt=None):\n '''\n Initialize CocoEval using coco APIs for gt and dt\n :param cocoGt: coco object with ground truth annotations\n :param cocoDt: coco object with detection results\n :return: None\n '''\n self.cocoGt = cocoGt # ground truth COCO API\n self.cocoDt = cocoDt # detections COCO API\n self.params = {} # evaluation parameters\n self.evalImgs = defaultdict(list) # per-image per-category evaluation results [KxAxI] elements\n self.eval = {} # accumulated evaluation results\n self._gts = defaultdict(list) # gt for evaluation\n self._dts = defaultdict(list) # dt for evaluation\n self.params = Params() # parameters\n self._paramsEval = {} # parameters for evaluation\n self.stats = [] # result summarization\n self.ious = {} # ious between all gts and dts\n if not cocoGt is None:\n self.params.imgIds = sorted(cocoGt.getImgIds())\n self.params.catIds = sorted(cocoGt.getCatIds())\n\n\n def _prepare(self):\n '''\n Prepare ._gts and ._dts for evaluation based on params\n :return: None\n '''\n #\n def _toMask(objs, coco):\n # modify segmentation by reference\n for obj in objs:\n t = coco.imgs[obj['image_id']]\n if type(obj['segmentation']) == list:\n if type(obj['segmentation'][0]) == dict:\n print('debug')\n obj['segmentation'] = mask.frPyObjects(obj['segmentation'],t['height'],t['width'])\n if len(obj['segmentation']) == 1:\n obj['segmentation'] = obj['segmentation'][0]\n else:\n # an object can have multiple polygon regions\n # merge them into one RLE mask\n obj['segmentation'] = mask.merge(obj['segmentation'])\n elif type(obj['segmentation']) == dict and type(obj['segmentation']['counts']) == list:\n obj['segmentation'] = mask.frPyObjects([obj['segmentation']],t['height'],t['width'])[0]\n elif type(obj['segmentation']) == dict and \\\n type(obj['segmentation']['counts']) in string_types:\n pass\n else:\n raise Exception('segmentation format not supported.')\n p = self.params\n if p.useCats:\n gts=self.cocoGt.loadAnns(self.cocoGt.getAnnIds(imgIds=p.imgIds, catIds=p.catIds))\n dts=self.cocoDt.loadAnns(self.cocoDt.getAnnIds(imgIds=p.imgIds, catIds=p.catIds))\n else:\n gts=self.cocoGt.loadAnns(self.cocoGt.getAnnIds(imgIds=p.imgIds))\n dts=self.cocoDt.loadAnns(self.cocoDt.getAnnIds(imgIds=p.imgIds))\n\n if p.useSegm:\n _toMask(gts, self.cocoGt)\n _toMask(dts, self.cocoDt)\n self._gts = defaultdict(list) # gt for evaluation\n self._dts = defaultdict(list) # dt for evaluation\n for gt in gts:\n self._gts[gt['image_id'], gt['category_id']].append(gt)\n for dt in dts:\n self._dts[dt['image_id'], dt['category_id']].append(dt)\n self.evalImgs = defaultdict(list) # per-image per-category evaluation results\n self.eval = {} # accumulated evaluation results\n\n def evaluate(self):\n '''\n Run per image evaluation on given images and store results (a list of dict) in self.evalImgs\n :return: None\n '''\n tic = time.time()\n print('Running per image evaluation... ')\n p = self.params\n p.imgIds = list(np.unique(p.imgIds))\n if p.useCats:\n p.catIds = list(np.unique(p.catIds))\n p.maxDets = sorted(p.maxDets)\n self.params=p\n\n self._prepare()\n # loop through images, area range, max detection number\n catIds = p.catIds if p.useCats else [-1]\n\n computeIoU = self.computeIoU\n self.ious = {(imgId, catId): computeIoU(imgId, catId) \\\n for imgId in p.imgIds\n for catId in catIds}\n\n evaluateImg = self.evaluateImg\n maxDet = p.maxDets[-1]\n self.evalImgs = [evaluateImg(imgId, catId, areaRng, maxDet)\n for catId in catIds\n for areaRng in p.areaRng\n for imgId in p.imgIds\n ]\n self._paramsEval = copy.deepcopy(self.params)\n toc = time.time()\n print('DONE (t=%0.2fs).'%(toc-tic))\n\n def computeIoU(self, imgId, catId):\n p = self.params\n if p.useCats:\n gt = self._gts[imgId,catId]\n dt = self._dts[imgId,catId]\n else:\n gt = [_ for cId in p.catIds for _ in self._gts[imgId,cId]]\n dt = [_ for cId in p.catIds for _ in self._dts[imgId,cId]]\n if len(gt) == 0 and len(dt) ==0:\n return []\n dt = sorted(dt, key=lambda x: -x['score'])\n if len(dt) > p.maxDets[-1]:\n dt=dt[0:p.maxDets[-1]]\n\n if p.useSegm:\n g = [g['segmentation'] for g in gt]\n d = [d['segmentation'] for d in dt]\n else:\n g = [g['bbox'] for g in gt]\n d = [d['bbox'] for d in dt]\n\n # compute iou between each dt and gt region\n iscrowd = [int(o['iscrowd']) for o in gt]\n ious = mask.iou(d,g,iscrowd)\n return ious\n\n def evaluateImg(self, imgId, catId, aRng, maxDet):\n '''\n perform evaluation for single category and image\n :return: dict (single image results)\n '''\n #\n p = self.params\n if p.useCats:\n gt = self._gts[imgId,catId]\n dt = self._dts[imgId,catId]\n else:\n gt = [_ for cId in p.catIds for _ in self._gts[imgId,cId]]\n dt = [_ for cId in p.catIds for _ in self._dts[imgId,cId]]\n if len(gt) == 0 and len(dt) ==0:\n return None\n\n for g in gt:\n if 'ignore' not in g:\n g['ignore'] = 0\n if g['iscrowd'] == 1 or g['ignore'] or (g['area']aRng[1]):\n g['_ignore'] = 1\n else:\n g['_ignore'] = 0\n\n # sort dt highest score first, sort gt ignore last\n # gt = sorted(gt, key=lambda x: x['_ignore'])\n gtind = [ind for (ind, g) in sorted(enumerate(gt), key=lambda ind_g: ind_g[1]['_ignore']) ]\n\n gt = [gt[ind] for ind in gtind]\n dt = sorted(dt, key=lambda x: -x['score'])[0:maxDet]\n iscrowd = [int(o['iscrowd']) for o in gt]\n # load computed ious\n N_iou = len(self.ious[imgId, catId])\n ious = self.ious[imgId, catId][0:maxDet, np.array(gtind)] if N_iou >0 else self.ious[imgId, catId]\n\n T = len(p.iouThrs)\n G = len(gt)\n D = len(dt)\n gtm = np.zeros((T,G))\n dtm = np.zeros((T,D))\n gtIg = np.array([g['_ignore'] for g in gt])\n dtIg = np.zeros((T,D))\n if not len(ious)==0:\n for tind, t in enumerate(p.iouThrs):\n for dind, d in enumerate(dt):\n # information about best match so far (m=-1 -> unmatched)\n iou = min([t,1-1e-10])\n m = -1\n for gind, g in enumerate(gt):\n # if this gt already matched, and not a crowd, continue\n if gtm[tind,gind]>0 and not iscrowd[gind]:\n continue\n # if dt matched to reg gt, and on ignore gt, stop\n if m>-1 and gtIg[m]==0 and gtIg[gind]==1:\n break\n # continue to next gt unless better match made\n if ious[dind,gind] < iou:\n continue\n # match successful and best so far, store appropriately\n iou=ious[dind,gind]\n m=gind\n # if match made store id of match for both dt and gt\n if m ==-1:\n continue\n dtIg[tind,dind] = gtIg[m]\n dtm[tind,dind] = gt[m]['id']\n gtm[tind,m] = d['id']\n # set unmatched detections outside of area range to ignore\n a = np.array([d['area']aRng[1] for d in dt]).reshape((1, len(dt)))\n dtIg = np.logical_or(dtIg, np.logical_and(dtm==0, np.repeat(a,T,0)))\n # store results for given image and category\n return {\n 'image_id': imgId,\n 'category_id': catId,\n 'aRng': aRng,\n 'maxDet': maxDet,\n 'dtIds': [d['id'] for d in dt],\n 'gtIds': [g['id'] for g in gt],\n 'dtMatches': dtm,\n 'gtMatches': gtm,\n 'dtScores': [d['score'] for d in dt],\n 'gtIgnore': gtIg,\n 'dtIgnore': dtIg,\n }\n\n def accumulate(self, p = None):\n '''\n Accumulate per image evaluation results and store the result in self.eval\n :param p: input params for evaluation\n :return: None\n '''\n print('Accumulating evaluation results... ')\n tic = time.time()\n if not self.evalImgs:\n print('Please run evaluate() first')\n # allows input customized parameters\n if p is None:\n p = self.params\n p.catIds = p.catIds if p.useCats == 1 else [-1]\n T = len(p.iouThrs)\n R = len(p.recThrs)\n K = len(p.catIds) if p.useCats else 1\n A = len(p.areaRng)\n M = len(p.maxDets)\n precision = -np.ones((T,R,K,A,M)) # -1 for the precision of absent categories\n recall = -np.ones((T,K,A,M))\n\n # create dictionary for future indexing\n _pe = self._paramsEval\n catIds = _pe.catIds if _pe.useCats else [-1]\n setK = set(catIds)\n setA = set(map(tuple, _pe.areaRng))\n setM = set(_pe.maxDets)\n setI = set(_pe.imgIds)\n # get inds to evaluate\n k_list = [n for n, k in enumerate(p.catIds) if k in setK]\n m_list = [m for n, m in enumerate(p.maxDets) if m in setM]\n a_list = [n for n, a in enumerate(map(lambda x: tuple(x), p.areaRng)) if a in setA]\n i_list = [n for n, i in enumerate(p.imgIds) if i in setI]\n # K0 = len(_pe.catIds)\n I0 = len(_pe.imgIds)\n A0 = len(_pe.areaRng)\n # retrieve E at each category, area range, and max number of detections\n for k, k0 in enumerate(k_list):\n Nk = k0*A0*I0\n for a, a0 in enumerate(a_list):\n Na = a0*I0\n for m, maxDet in enumerate(m_list):\n E = [self.evalImgs[Nk+Na+i] for i in i_list]\n E = filter(None, E)\n if len(E) == 0:\n continue\n dtScores = np.concatenate([e['dtScores'][0:maxDet] for e in E])\n\n # different sorting method generates slightly different results.\n # mergesort is used to be consistent as Matlab implementation.\n inds = np.argsort(-dtScores, kind='mergesort')\n\n dtm = np.concatenate([e['dtMatches'][:,0:maxDet] for e in E], axis=1)[:,inds]\n dtIg = np.concatenate([e['dtIgnore'][:,0:maxDet] for e in E], axis=1)[:,inds]\n gtIg = np.concatenate([e['gtIgnore'] for e in E])\n npig = len([ig for ig in gtIg if ig == 0])\n if npig == 0:\n continue\n tps = np.logical_and( dtm, np.logical_not(dtIg) )\n fps = np.logical_and(np.logical_not(dtm), np.logical_not(dtIg) )\n\n tp_sum = np.cumsum(tps, axis=1).astype(dtype=np.float)\n fp_sum = np.cumsum(fps, axis=1).astype(dtype=np.float)\n for t, (tp, fp) in enumerate(zip(tp_sum, fp_sum)):\n tp = np.array(tp)\n fp = np.array(fp)\n nd = len(tp)\n rc = tp / npig\n pr = tp / (fp+tp+np.spacing(1))\n q = np.zeros((R,))\n\n if nd:\n recall[t,k,a,m] = rc[-1]\n else:\n recall[t,k,a,m] = 0\n\n # numpy is slow without cython optimization for accessing elements\n # use python array gets significant speed improvement\n pr = pr.tolist(); q = q.tolist()\n\n for i in range(nd-1, 0, -1):\n if pr[i] > pr[i-1]:\n pr[i-1] = pr[i]\n\n inds = np.searchsorted(rc, p.recThrs)\n try:\n for ri, pi in enumerate(inds):\n q[ri] = pr[pi]\n except:\n pass\n precision[t,:,k,a,m] = np.array(q)\n self.eval = {\n 'params': p,\n 'counts': [T, R, K, A, M],\n 'date': datetime.datetime.now().strftime(\"%Y-%m-%d %H:%M:%S\"),\n 'precision': precision,\n 'recall': recall,\n }\n toc = time.time()\n print('DONE (t=%0.2fs).'%( toc-tic ))\n\n def summarize(self):\n '''\n Compute and display summary metrics for evaluation results.\n Note this functin can *only* be applied on the default parameter setting\n '''\n def _summarize( ap=1, iouThr=None, areaRng='all', maxDets=100 ):\n p = self.params\n iStr = ' {:<18} {} @[ IoU={:<9} | area={:>6} | maxDets={:>3} ] = {}'\n titleStr = 'Average Precision' if ap == 1 else 'Average Recall'\n typeStr = '(AP)' if ap==1 else '(AR)'\n iouStr = '%0.2f:%0.2f'%(p.iouThrs[0], p.iouThrs[-1]) if iouThr is None else '%0.2f'%(iouThr)\n areaStr = areaRng\n maxDetsStr = '%d'%(maxDets)\n\n aind = [i for i, aRng in enumerate(['all', 'small', 'medium', 'large']) if aRng == areaRng]\n mind = [i for i, mDet in enumerate([1, 10, 100]) if mDet == maxDets]\n if ap == 1:\n # dimension of precision: [TxRxKxAxM]\n s = self.eval['precision']\n # IoU\n if iouThr is not None:\n t = np.where(iouThr == p.iouThrs)[0]\n s = s[t]\n # areaRng\n s = s[:,:,:,aind,mind]\n else:\n # dimension of recall: [TxKxAxM]\n s = self.eval['recall']\n s = s[:,:,aind,mind]\n if len(s[s>-1])==0:\n mean_s = -1\n else:\n mean_s = np.mean(s[s>-1])\n print(iStr.format(titleStr, typeStr, iouStr, areaStr, maxDetsStr, '%.3f'%(float(mean_s))))\n return mean_s\n\n if not self.eval:\n raise Exception('Please run accumulate() first')\n self.stats = np.zeros((12,))\n self.stats[0] = _summarize(1)\n self.stats[1] = _summarize(1,iouThr=.5)\n self.stats[2] = _summarize(1,iouThr=.75)\n self.stats[3] = _summarize(1,areaRng='small')\n self.stats[4] = _summarize(1,areaRng='medium')\n self.stats[5] = _summarize(1,areaRng='large')\n self.stats[6] = _summarize(0,maxDets=1)\n self.stats[7] = _summarize(0,maxDets=10)\n self.stats[8] = _summarize(0,maxDets=100)\n self.stats[9] = _summarize(0,areaRng='small')\n self.stats[10] = _summarize(0,areaRng='medium')\n self.stats[11] = _summarize(0,areaRng='large')\n\n def __str__(self):\n self.summarize()\n\nclass Params:\n '''\n Params for coco evaluation api\n '''\n def __init__(self):\n self.imgIds = []\n self.catIds = []\n # np.arange causes trouble. the data point on arange is slightly larger than the true value\n self.iouThrs = np.linspace(.5, 0.95, np.round((0.95-.5)/.05)+1, endpoint=True)\n self.recThrs = np.linspace(.0, 1.00, np.round((1.00-.0)/.01)+1, endpoint=True)\n self.maxDets = [1,10,100]\n self.areaRng = [ [0**2,1e5**2], [0**2, 32**2], [32**2, 96**2], [96**2, 1e5**2] ]\n self.useSegm = 0\n self.useCats = 1\n"
},
{
"path": "lib/pycocotools/license.txt",
"content": "Copyright (c) 2014, Piotr Dollar and Tsung-Yi Lin\nAll rights reserved.\n\nRedistribution and use in source and binary forms, with or without\nmodification, are permitted provided that the following conditions are met: \n\n1. Redistributions of source code must retain the above copyright notice, this\n list of conditions and the following disclaimer. \n2. Redistributions in binary form must reproduce the above copyright notice,\n this list of conditions and the following disclaimer in the documentation\n and/or other materials provided with the distribution. \n\nTHIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\" AND\nANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED\nWARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE\nDISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR\nANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES\n(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;\nLOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND\nON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT\n(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS\nSOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\n\nThe views and conclusions contained in the software and documentation are those\nof the authors and should not be interpreted as representing official policies, \neither expressed or implied, of the FreeBSD Project.\n"
},
{
"path": "lib/pycocotools/mask.py",
"content": "__author__ = 'tsungyi'\n\nfrom . import _mask\n\n# Interface for manipulating masks stored in RLE format.\n#\n# RLE is a simple yet efficient format for storing binary masks. RLE\n# first divides a vector (or vectorized image) into a series of piecewise\n# constant regions and then for each piece simply stores the length of\n# that piece. For example, given M=[0 0 1 1 1 0 1] the RLE counts would\n# be [2 3 1 1], or for M=[1 1 1 1 1 1 0] the counts would be [0 6 1]\n# (note that the odd counts are always the numbers of zeros). Instead of\n# storing the counts directly, additional compression is achieved with a\n# variable bitrate representation based on a common scheme called LEB128.\n#\n# Compression is greatest given large piecewise constant regions.\n# Specifically, the size of the RLE is proportional to the number of\n# *boundaries* in M (or for an image the number of boundaries in the y\n# direction). Assuming fairly simple shapes, the RLE representation is\n# O(sqrt(n)) where n is number of pixels in the object. Hence space usage\n# is substantially lower, especially for large simple objects (large n).\n#\n# Many common operations on masks can be computed directly using the RLE\n# (without need for decoding). This includes computations such as area,\n# union, intersection, etc. All of these operations are linear in the\n# size of the RLE, in other words they are O(sqrt(n)) where n is the area\n# of the object. Computing these operations on the original mask is O(n).\n# Thus, using the RLE can result in substantial computational savings.\n#\n# The following API functions are defined:\n# encode - Encode binary masks using RLE.\n# decode - Decode binary masks encoded via RLE.\n# merge - Compute union or intersection of encoded masks.\n# iou - Compute intersection over union between masks.\n# area - Compute area of encoded masks.\n# toBbox - Get bounding boxes surrounding encoded masks.\n# frPyObjects - Convert polygon, bbox, and uncompressed RLE to encoded RLE mask.\n#\n# Usage:\n# Rs = encode( masks )\n# masks = decode( Rs )\n# R = merge( Rs, intersect=false )\n# o = iou( dt, gt, iscrowd )\n# a = area( Rs )\n# bbs = toBbox( Rs )\n# Rs = frPyObjects( [pyObjects], h, w )\n#\n# In the API the following formats are used:\n# Rs - [dict] Run-length encoding of binary masks\n# R - dict Run-length encoding of binary mask\n# masks - [hxwxn] Binary mask(s) (must have type np.ndarray(dtype=uint8) in column-major order)\n# iscrowd - [nx1] list of np.ndarray. 1 indicates corresponding gt image has crowd region to ignore\n# bbs - [nx4] Bounding box(es) stored as [x y w h]\n# poly - Polygon stored as [[x1 y1 x2 y2...],[x1 y1 ...],...] (2D list)\n# dt,gt - May be either bounding boxes or encoded masks\n# Both poly and bbs are 0-indexed (bbox=[0 0 1 1] encloses first pixel).\n#\n# Finally, a note about the intersection over union (iou) computation.\n# The standard iou of a ground truth (gt) and detected (dt) object is\n# iou(gt,dt) = area(intersect(gt,dt)) / area(union(gt,dt))\n# For \"crowd\" regions, we use a modified criteria. If a gt object is\n# marked as \"iscrowd\", we allow a dt to match any subregion of the gt.\n# Choosing gt' in the crowd gt that best matches the dt can be done using\n# gt'=intersect(dt,gt). Since by definition union(gt',dt)=dt, computing\n# iou(gt,dt,iscrowd) = iou(gt',dt) = area(intersect(gt,dt)) / area(dt)\n# For crowd gt regions we use this modified criteria above for the iou.\n#\n# To compile run \"python setup.py build_ext --inplace\"\n# Please do not contact us for help with compiling.\n#\n# Microsoft COCO Toolbox. version 2.0\n# Data, paper, and tutorials available at: http://mscoco.org/\n# Code written by Piotr Dollar and Tsung-Yi Lin, 2015.\n# Licensed under the Simplified BSD License [see coco/license.txt]\n\nencode = _mask.encode\ndecode = _mask.decode\niou = _mask.iou\nmerge = _mask.merge\narea = _mask.area\ntoBbox = _mask.toBbox\nfrPyObjects = _mask.frPyObjects"
},
{
"path": "lib/pycocotools/maskApi.c",
"content": "/**************************************************************************\n* Microsoft COCO Toolbox. version 2.0\n* Data, paper, and tutorials available at: http://mscoco.org/\n* Code written by Piotr Dollar and Tsung-Yi Lin, 2015.\n* Licensed under the Simplified BSD License [see coco/license.txt]\n**************************************************************************/\n#include \"maskApi.h\"\n#include \n#include \n\nuint umin( uint a, uint b ) { return (ab) ? a : b; }\n\nvoid rleInit( RLE *R, siz h, siz w, siz m, uint *cnts ) {\n R->h=h; R->w=w; R->m=m; R->cnts=(m==0)?0:malloc(sizeof(uint)*m);\n if(cnts) for(siz j=0; jcnts[j]=cnts[j];\n}\n\nvoid rleFree( RLE *R ) {\n free(R->cnts); R->cnts=0;\n}\n\nvoid rlesInit( RLE **R, siz n ) {\n *R = (RLE*) malloc(sizeof(RLE)*n);\n for(siz i=0; i0 ) {\n c=umin(ca,cb); cc+=c; ct=0;\n ca-=c; if(!ca && a0) {\n crowd=iscrowd!=NULL && iscrowd[g];\n if(dt[d].h!=gt[g].h || dt[d].w!=gt[g].w) { o[g*m+d]=-1; continue; }\n siz ka, kb, a, b; uint c, ca, cb, ct, i, u; bool va, vb;\n ca=dt[d].cnts[0]; ka=dt[d].m; va=vb=0;\n cb=gt[g].cnts[0]; kb=gt[g].m; a=b=1; i=u=0; ct=1;\n while( ct>0 ) {\n c=umin(ca,cb); if(va||vb) { u+=c; if(va&&vb) i+=c; } ct=0;\n ca-=c; if(!ca && ad?1:c=dy && xs>xe) || (dxye);\n if(flip) { t=xs; xs=xe; xe=t; t=ys; ys=ye; ye=t; }\n s = dx>=dy ? (double)(ye-ys)/dx : (double)(xe-xs)/dy;\n if(dx>=dy) for( int d=0; d<=dx; d++ ) {\n t=flip?dx-d:d; u[m]=t+xs; v[m]=(int)(ys+s*t+.5); m++;\n } else for( int d=0; d<=dy; d++ ) {\n t=flip?dy-d:d; v[m]=t+ys; u[m]=(int)(xs+s*t+.5); m++;\n }\n }\n // get points along y-boundary and downsample\n free(x); free(y); k=m; m=0; double xd, yd;\n x=malloc(sizeof(int)*k); y=malloc(sizeof(int)*k);\n for( j=1; jw-1 ) continue;\n yd=(double)(v[j]h) yd=h; yd=ceil(yd);\n x[m]=(int) xd; y[m]=(int) yd; m++;\n }\n // compute rle encoding given y-boundary points\n k=m; a=malloc(sizeof(uint)*(k+1));\n for( j=0; j0) b[m++]=a[j++]; else {\n j++; if(jm, p=0; long x; bool more;\n char *s=malloc(sizeof(char)*m*6);\n for( i=0; icnts[i]; if(i>2) x-=(long) R->cnts[i-2]; more=1;\n while( more ) {\n char c=x & 0x1f; x >>= 5; more=(c & 0x10) ? x!=-1 : x!=0;\n if(more) c |= 0x20; c+=48; s[p++]=c;\n }\n }\n s[p]=0; return s;\n}\n\nvoid rleFrString( RLE *R, char *s, siz h, siz w ) {\n siz m=0, p=0, k; long x; bool more; uint *cnts;\n while( s[m] ) m++; cnts=malloc(sizeof(uint)*m); m=0;\n while( s[p] ) {\n x=0; k=0; more=1;\n while( more ) {\n char c=s[p]-48; x |= (c & 0x1f) << 5*k;\n more = c & 0x20; p++; k++;\n if(!more && (c & 0x10)) x |= -1 << 5*k;\n }\n if(m>2) x+=(long) cnts[m-2]; cnts[m++]=(uint) x;\n }\n rleInit(R,h,w,m,cnts); free(cnts);\n}\n"
},
{
"path": "lib/pycocotools/maskApi.h",
"content": "/**************************************************************************\n* Microsoft COCO Toolbox. version 2.0\n* Data, paper, and tutorials available at: http://mscoco.org/\n* Code written by Piotr Dollar and Tsung-Yi Lin, 2015.\n* Licensed under the Simplified BSD License [see coco/license.txt]\n**************************************************************************/\n#pragma once\n#include \n\ntypedef unsigned int uint;\ntypedef unsigned long siz;\ntypedef unsigned char byte;\ntypedef double* BB;\ntypedef struct { siz h, w, m; uint *cnts; } RLE;\n\n// Initialize/destroy RLE.\nvoid rleInit( RLE *R, siz h, siz w, siz m, uint *cnts );\nvoid rleFree( RLE *R );\n\n// Initialize/destroy RLE array.\nvoid rlesInit( RLE **R, siz n );\nvoid rlesFree( RLE **R, siz n );\n\n// Encode binary masks using RLE.\nvoid rleEncode( RLE *R, const byte *mask, siz h, siz w, siz n );\n\n// Decode binary masks encoded via RLE.\nvoid rleDecode( const RLE *R, byte *mask, siz n );\n\n// Compute union or intersection of encoded masks.\nvoid rleMerge( const RLE *R, RLE *M, siz n, bool intersect );\n\n// Compute area of encoded masks.\nvoid rleArea( const RLE *R, siz n, uint *a );\n\n// Compute intersection over union between masks.\nvoid rleIou( RLE *dt, RLE *gt, siz m, siz n, byte *iscrowd, double *o );\n\n// Compute intersection over union between bounding boxes.\nvoid bbIou( BB dt, BB gt, siz m, siz n, byte *iscrowd, double *o );\n\n// Get bounding boxes surrounding encoded masks.\nvoid rleToBbox( const RLE *R, BB bb, siz n );\n\n// Convert bounding boxes to encoded masks.\nvoid rleFrBbox( RLE *R, const BB bb, siz h, siz w, siz n );\n\n// Convert polygon to encoded mask.\nvoid rleFrPoly( RLE *R, const double *xy, siz k, siz h, siz w );\n\n// Get compressed string representation of encoded mask.\nchar* rleToString( const RLE *R );\n\n// Convert from compressed string representation of encoded mask.\nvoid rleFrString( RLE *R, char *s, siz h, siz w );\n"
},
{
"path": "lib/roi_data_layer/__init__.py",
"content": "# --------------------------------------------------------\n# Fast R-CNN\n# Copyright (c) 2015 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Written by Ross Girshick\n# --------------------------------------------------------\n"
},
{
"path": "lib/roi_data_layer/minibatch.py",
"content": "# --------------------------------------------------------\n# Fast R-CNN\n# Copyright (c) 2015 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Written by Ross Girshick and Xinlei Chen\n# --------------------------------------------------------\n\n\"\"\"Compute minibatch blobs for training a Fast R-CNN network.\"\"\"\nfrom __future__ import absolute_import\nfrom __future__ import division\nfrom __future__ import print_function\n\nimport numpy as np\nimport numpy.random as npr\nfrom scipy.misc import imread\nfrom model.utils.config import cfg\nfrom model.utils.blob import prep_im_for_blob, im_list_to_blob\nimport pdb\ndef get_minibatch(roidb, num_classes):\n \"\"\"Given a roidb, construct a minibatch sampled from it.\"\"\"\n num_images = len(roidb)\n # Sample random scales to use for each image in this batch\n random_scale_inds = npr.randint(0, high=len(cfg.TRAIN.SCALES),\n size=num_images)\n assert(cfg.TRAIN.BATCH_SIZE % num_images == 0), \\\n 'num_images ({}) must divide BATCH_SIZE ({})'. \\\n format(num_images, cfg.TRAIN.BATCH_SIZE)\n\n # Get the input image blob, formatted for caffe\n im_blob, im_scales = _get_image_blob(roidb, random_scale_inds)\n\n blobs = {'data': im_blob}\n\n assert len(im_scales) == 1, \"Single batch only\"\n assert len(roidb) == 1, \"Single batch only\"\n \n # gt boxes: (x1, y1, x2, y2, cls)\n if cfg.TRAIN.USE_ALL_GT:\n # Include all ground truth boxes\n gt_inds = np.where(roidb[0]['gt_classes'] != 0)[0]\n else:\n # For the COCO ground truth boxes, exclude the ones that are ''iscrowd'' \n gt_inds = np.where((roidb[0]['gt_classes'] != 0) & np.all(roidb[0]['gt_overlaps'].toarray() > -1.0, axis=1))[0]\n gt_boxes = np.empty((len(gt_inds), 5), dtype=np.float32)\n gt_boxes[:, 0:4] = roidb[0]['boxes'][gt_inds, :] * im_scales[0]\n gt_boxes[:, 4] = roidb[0]['gt_classes'][gt_inds]\n blobs['gt_boxes'] = gt_boxes\n blobs['im_info'] = np.array(\n [[im_blob.shape[1], im_blob.shape[2], im_scales[0]]],\n dtype=np.float32)\n\n blobs['img_id'] = roidb[0]['img_id']\n\n return blobs\n\ndef _get_image_blob(roidb, scale_inds):\n \"\"\"Builds an input blob from the images in the roidb at the specified\n scales.\n \"\"\"\n num_images = len(roidb)\n\n processed_ims = []\n im_scales = []\n for i in range(num_images):\n #im = cv2.imread(roidb[i]['image'])\n im = imread(roidb[i]['image'])\n\n if len(im.shape) == 2:\n im = im[:,:,np.newaxis]\n im = np.concatenate((im,im,im), axis=2)\n # flip the channel, since the original one using cv2\n # rgb -> bgr\n im = im[:,:,::-1]\n\n if roidb[i]['flipped']:\n im = im[:, ::-1, :]\n target_size = cfg.TRAIN.SCALES[scale_inds[i]]\n im, im_scale = prep_im_for_blob(im, cfg.PIXEL_MEANS, target_size,\n cfg.TRAIN.MAX_SIZE)\n im_scales.append(im_scale)\n processed_ims.append(im)\n\n # Create a blob to hold the input images\n blob = im_list_to_blob(processed_ims)\n\n return blob, im_scales\n"
},
{
"path": "lib/roi_data_layer/roibatchLoader.py",
"content": "\n\"\"\"The data layer used during training to train a Fast R-CNN network.\n\"\"\"\n\nfrom __future__ import absolute_import\nfrom __future__ import division\nfrom __future__ import print_function\n\nimport torch.utils.data as data\nfrom PIL import Image\nimport torch\n\nfrom model.utils.config import cfg\nfrom roi_data_layer.minibatch import get_minibatch, get_minibatch\nfrom model.rpn.bbox_transform import bbox_transform_inv, clip_boxes\n\nimport numpy as np\nimport random\nimport time\nimport pdb\n\nclass roibatchLoader(data.Dataset):\n def __init__(self, roidb, ratio_list, ratio_index, batch_size, num_classes, training=True, normalize=None):\n self._roidb = roidb\n self._num_classes = num_classes\n # we make the height of image consistent to trim_height, trim_width\n self.trim_height = cfg.TRAIN.TRIM_HEIGHT\n self.trim_width = cfg.TRAIN.TRIM_WIDTH\n self.max_num_box = cfg.MAX_NUM_GT_BOXES\n self.training = training\n self.normalize = normalize\n self.ratio_list = ratio_list\n self.ratio_index = ratio_index\n self.batch_size = batch_size\n self.data_size = len(self.ratio_list)\n\n # given the ratio_list, we want to make the ratio same for each batch.\n self.ratio_list_batch = torch.Tensor(self.data_size).zero_()\n num_batch = int(np.ceil(len(ratio_index) / batch_size))\n for i in range(num_batch):\n left_idx = i*batch_size\n right_idx = min((i+1)*batch_size-1, self.data_size-1)\n\n if ratio_list[right_idx] < 1:\n # for ratio < 1, we preserve the leftmost in each batch.\n target_ratio = ratio_list[left_idx]\n elif ratio_list[left_idx] > 1:\n # for ratio > 1, we preserve the rightmost in each batch.\n target_ratio = ratio_list[right_idx]\n else:\n # for ratio cross 1, we make it to be 1.\n target_ratio = 1\n\n self.ratio_list_batch[left_idx:(right_idx+1)] = torch.tensor(target_ratio.astype(np.float64)) # trainset ratio list ,each batch is same number\n\n\n def __getitem__(self, index):\n if self.training:\n index_ratio = int(self.ratio_index[index])\n else:\n index_ratio = index\n\n # get the anchor index for current sample index\n # here we set the anchor index to the last one\n # sample in this group\n minibatch_db = [self._roidb[index_ratio]]\n blobs = get_minibatch(minibatch_db, self._num_classes)\n data = torch.from_numpy(blobs['data'])\n im_info = torch.from_numpy(blobs['im_info'])\n # we need to random shuffle the bounding box.\n data_height, data_width = data.size(1), data.size(2)\n if self.training:\n np.random.shuffle(blobs['gt_boxes'])\n gt_boxes = torch.from_numpy(blobs['gt_boxes'])\n\n ########################################################\n # padding the input image to fixed size for each group #\n ########################################################\n\n # NOTE1: need to cope with the case where a group cover both conditions. (done)\n # NOTE2: need to consider the situation for the tail samples. (no worry)\n # NOTE3: need to implement a parallel data loader. (no worry)\n # get the index range\n\n # if the image need to crop, crop to the target size.\n ratio = self.ratio_list_batch[index]\n\n if self._roidb[index_ratio]['need_crop']:\n if ratio < 1:\n # this means that data_width << data_height, we need to crop the\n # data_height\n min_y = int(torch.min(gt_boxes[:,1]))\n max_y = int(torch.max(gt_boxes[:,3]))\n trim_size = int(np.floor(data_width / ratio))\n if trim_size > data_height:\n trim_size = data_height \n box_region = max_y - min_y + 1\n if min_y == 0:\n y_s = 0\n else:\n if (box_region-trim_size) < 0:\n y_s_min = max(max_y-trim_size, 0)\n y_s_max = min(min_y, data_height-trim_size)\n if y_s_min == y_s_max:\n y_s = y_s_min\n else:\n y_s = np.random.choice(range(y_s_min, y_s_max))\n else:\n y_s_add = int((box_region-trim_size)/2)\n if y_s_add == 0:\n y_s = min_y\n else:\n y_s = np.random.choice(range(min_y, min_y+y_s_add))\n # crop the image\n data = data[:, y_s:(y_s + trim_size), :, :]\n\n # shift y coordiante of gt_boxes\n gt_boxes[:, 1] = gt_boxes[:, 1] - float(y_s)\n gt_boxes[:, 3] = gt_boxes[:, 3] - float(y_s)\n\n # update gt bounding box according the trip\n gt_boxes[:, 1].clamp_(0, trim_size - 1)\n gt_boxes[:, 3].clamp_(0, trim_size - 1)\n\n else:\n # this means that data_width >> data_height, we need to crop the\n # data_width\n min_x = int(torch.min(gt_boxes[:,0]))\n max_x = int(torch.max(gt_boxes[:,2]))\n trim_size = int(np.ceil(data_height * ratio))\n if trim_size > data_width:\n trim_size = data_width \n box_region = max_x - min_x + 1\n if min_x == 0:\n x_s = 0\n else:\n if (box_region-trim_size) < 0:\n x_s_min = max(max_x-trim_size, 0)\n x_s_max = min(min_x, data_width-trim_size)\n if x_s_min == x_s_max:\n x_s = x_s_min\n else:\n x_s = np.random.choice(range(x_s_min, x_s_max))\n else:\n x_s_add = int((box_region-trim_size)/2)\n if x_s_add == 0:\n x_s = min_x\n else:\n x_s = np.random.choice(range(min_x, min_x+x_s_add))\n # crop the image\n data = data[:, :, x_s:(x_s + trim_size), :]\n\n # shift x coordiante of gt_boxes\n gt_boxes[:, 0] = gt_boxes[:, 0] - float(x_s)\n gt_boxes[:, 2] = gt_boxes[:, 2] - float(x_s)\n # update gt bounding box according the trip\n gt_boxes[:, 0].clamp_(0, trim_size - 1)\n gt_boxes[:, 2].clamp_(0, trim_size - 1)\n\n # based on the ratio, padding the image.\n if ratio < 1:\n # this means that data_width < data_height\n trim_size = int(np.floor(data_width / ratio))\n\n padding_data = torch.FloatTensor(int(np.ceil(data_width / ratio)), \\\n data_width, 3).zero_()\n\n padding_data[:data_height, :, :] = data[0]\n # update im_info\n im_info[0, 0] = padding_data.size(0)\n # print(\"height %d %d \\n\" %(index, anchor_idx))\n elif ratio > 1:\n # this means that data_width > data_height\n # if the image need to crop.\n padding_data = torch.FloatTensor(data_height, \\\n int(np.ceil(data_height * ratio)), 3).zero_()\n padding_data[:, :data_width, :] = data[0]\n im_info[0, 1] = padding_data.size(1)\n else:\n trim_size = min(data_height, data_width)\n padding_data = torch.FloatTensor(trim_size, trim_size, 3).zero_()\n padding_data = data[0][:trim_size, :trim_size, :]\n # gt_boxes.clamp_(0, trim_size)\n gt_boxes[:, :4].clamp_(0, trim_size)\n im_info[0, 0] = trim_size\n im_info[0, 1] = trim_size\n\n\n # check the bounding box:\n not_keep = (gt_boxes[:,0] == gt_boxes[:,2]) | (gt_boxes[:,1] == gt_boxes[:,3])\n keep = torch.nonzero(not_keep == 0).view(-1)\n\n gt_boxes_padding = torch.FloatTensor(self.max_num_box, gt_boxes.size(1)).zero_()\n if keep.numel() != 0:\n gt_boxes = gt_boxes[keep]\n num_boxes = min(gt_boxes.size(0), self.max_num_box)\n gt_boxes_padding[:num_boxes,:] = gt_boxes[:num_boxes]\n else:\n num_boxes = 0\n\n # permute trim_data to adapt to downstream processing\n padding_data = padding_data.permute(2, 0, 1).contiguous()\n im_info = im_info.view(3)\n\n return padding_data, im_info, gt_boxes_padding, num_boxes\n else:\n data = data.permute(0, 3, 1, 2).contiguous().view(3, data_height, data_width)\n im_info = im_info.view(3)\n\n gt_boxes = torch.FloatTensor([1,1,1,1,1])\n num_boxes = 0\n\n return data, im_info, gt_boxes, num_boxes\n\n def __len__(self):\n return len(self._roidb)\n"
},
{
"path": "lib/roi_data_layer/roidb.py",
"content": "\"\"\"Transform a roidb into a trainable roidb by adding a bunch of metadata.\"\"\"\nfrom __future__ import absolute_import\nfrom __future__ import division\nfrom __future__ import print_function\n\nimport os\nimport pickle\n\nimport datasets\nimport numpy as np\nfrom model.utils.config import cfg\nfrom datasets.factory import get_imdb\nimport PIL\nimport pdb\n\ndef prepare_roidb(imdb):\n \"\"\"Enrich the imdb's roidb by adding some derived quantities that\n are useful for training. This function precomputes the maximum\n overlap, taken over ground-truth boxes, between each ROI and\n each ground-truth box. The class with maximum overlap is also\n recorded.\n \"\"\"\n\n roidb = imdb.roidb\n if not (imdb.name.startswith('coco')):\n cache_file = os.path.join(imdb.cache_path, imdb.name + '_sizes.pkl')\n if os.path.exists(cache_file):\n print('Image sizes loaded from %s' % cache_file)\n with open(cache_file, 'rb') as f:\n sizes = pickle.load(f)\n else:\n print('Extracting image sizes... (It may take long time)')\n sizes = [PIL.Image.open(imdb.image_path_at(i)).size\n for i in range(imdb.num_images)]\n with open(cache_file, 'wb') as f:\n pickle.dump(sizes, f)\n print('Done!!')\n \n for i in range(len(imdb.image_index)):\n roidb[i]['img_id'] = imdb.image_id_at(i)\n roidb[i]['image'] = imdb.image_path_at(i)\n if not (imdb.name.startswith('coco')):\n roidb[i]['width'] = sizes[i][0]\n roidb[i]['height'] = sizes[i][1]\n # need gt_overlaps as a dense array for argmax\n gt_overlaps = roidb[i]['gt_overlaps'].toarray()\n # max overlap with gt over classes (columns)\n max_overlaps = gt_overlaps.max(axis=1)\n # gt class that had the max overlap\n max_classes = gt_overlaps.argmax(axis=1)\n roidb[i]['max_classes'] = max_classes\n roidb[i]['max_overlaps'] = max_overlaps\n # sanity checks\n # max overlap of 0 => class should be zero (background)\n zero_inds = np.where(max_overlaps == 0)[0]\n assert all(max_classes[zero_inds] == 0)\n # max overlap > 0 => class should not be zero (must be a fg class)\n nonzero_inds = np.where(max_overlaps > 0)[0]\n assert all(max_classes[nonzero_inds] != 0)\n\n\ndef rank_roidb_ratio(roidb):\n # rank roidb based on the ratio between width and height.\n ratio_large = 2 # largest ratio to preserve.\n ratio_small = 0.5 # smallest ratio to preserve. \n \n ratio_list = []\n for i in range(len(roidb)):\n width = roidb[i]['width']\n height = roidb[i]['height']\n ratio = width / float(height)\n\n if ratio > ratio_large:\n roidb[i]['need_crop'] = 1\n ratio = ratio_large\n elif ratio < ratio_small:\n roidb[i]['need_crop'] = 1\n ratio = ratio_small \n else:\n roidb[i]['need_crop'] = 0\n\n ratio_list.append(ratio)\n\n ratio_list = np.array(ratio_list)\n ratio_index = np.argsort(ratio_list)\n return ratio_list[ratio_index], ratio_index\n\ndef filter_roidb(roidb):\n # filter the image without bounding box.\n print('before filtering, there are %d images...' % (len(roidb)))\n i = 0\n while i < len(roidb):\n if len(roidb[i]['boxes']) == 0:\n del roidb[i]\n i -= 1\n i += 1\n\n print('after filtering, there are %d images...' % (len(roidb)))\n return roidb\n\ndef combined_roidb(imdb_names, training=True):\n \"\"\"\n Combine multiple roidbs\n \"\"\"\n\n def get_training_roidb(imdb):\n \"\"\"Returns a roidb (Region of Interest database) for use in training.\"\"\"\n if cfg.TRAIN.USE_FLIPPED:\n print('Appending horizontally-flipped training examples...')\n imdb.append_flipped_images()\n print('done')\n\n print('Preparing training data...')\n\n prepare_roidb(imdb)\n #ratio_index = rank_roidb_ratio(imdb)\n print('done')\n\n return imdb.roidb\n \n def get_roidb(imdb_name):\n imdb = get_imdb(imdb_name)\n print('Loaded dataset `{:s}`'.format(imdb.name))\n imdb.set_proposal_method(cfg.TRAIN.PROPOSAL_METHOD)\n print('Set proposal method: {:s}'.format(cfg.TRAIN.PROPOSAL_METHOD))\n roidb = get_training_roidb(imdb)\n return roidb\n\n roidbs = [get_roidb(s) for s in imdb_names.split('+')]\n roidb = roidbs[0]\n\n if len(roidbs) > 1:\n for r in roidbs[1:]:\n roidb.extend(r)\n tmp = get_imdb(imdb_names.split('+')[1])\n imdb = datasets.imdb.imdb(imdb_names, tmp.classes)\n else:\n imdb = get_imdb(imdb_names)\n\n if training:\n roidb = filter_roidb(roidb)\n\n ratio_list, ratio_index = rank_roidb_ratio(roidb)\n\n return imdb, roidb, ratio_list, ratio_index\n"
},
{
"path": "lib/setup.py",
"content": "from __future__ import print_function\n# --------------------------------------------------------\n# Fast R-CNN\n# Copyright (c) 2015 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Written by Ross Girshick\n# --------------------------------------------------------\n\nimport os\nfrom os.path import join as pjoin\nimport numpy as np\nfrom distutils.core import setup\nfrom distutils.extension import Extension\nfrom Cython.Distutils import build_ext\n\n\ndef find_in_path(name, path):\n \"Find a file in a search path\"\n # adapted fom http://code.activestate.com/recipes/52224-find-a-file-given-a-search-path/\n for dir in path.split(os.pathsep):\n binpath = pjoin(dir, name)\n if os.path.exists(binpath):\n return os.path.abspath(binpath)\n return None\n\n\n# def locate_cuda():\n# \"\"\"Locate the CUDA environment on the system\n#\n# Returns a dict with keys 'home', 'nvcc', 'include', and 'lib64'\n# and values giving the absolute path to each directory.\n#\n# Starts by looking for the CUDAHOME env variable. If not found, everything\n# is based on finding 'nvcc' in the PATH.\n# \"\"\"\n# \n# # first check if the CUDAHOME env variable is in use\n# if 'CUDAHOME' in os.environ:\n# home = os.environ['CUDAHOME']\n# nvcc = pjoin(home, 'bin', 'nvcc')\n# else:\n# # otherwise, search the PATH for NVCC\n# default_path = pjoin(os.sep, 'usr', 'local', 'cuda', 'bin')\n# nvcc = find_in_path('nvcc', os.environ['PATH'] + os.pathsep + default_path)\n# if nvcc is None:\n# raise EnvironmentError('The nvcc binary could not be '\n# 'located in your $PATH. Either add it to your path, or set $CUDAHOME')\n# home = os.path.dirname(os.path.dirname(nvcc))\n#\n# cudaconfig = {'home': home, 'nvcc': nvcc,\n# 'include': pjoin(home, 'include'),\n# 'lib64': pjoin(home, 'lib64')}\n# for k, v in cudaconfig.iteritems():\n# if not os.path.exists(v):\n# raise EnvironmentError('The CUDA %s path could not be located in %s' % (k, v))\n#\n# return cudaconfig\n\n\n# CUDA = locate_cuda()\n\n# Obtain the numpy include directory. This logic works across numpy versions.\ntry:\n numpy_include = np.get_include()\nexcept AttributeError:\n numpy_include = np.get_numpy_include()\n\n\ndef customize_compiler_for_nvcc(self):\n \"\"\"inject deep into distutils to customize how the dispatch\n to gcc/nvcc works.\n\n If you subclass UnixCCompiler, it's not trivial to get your subclass\n injected in, and still have the right customizations (i.e.\n distutils.sysconfig.customize_compiler) run on it. So instead of going\n the OO route, I have this. Note, it's kindof like a wierd functional\n subclassing going on.\"\"\"\n\n # tell the compiler it can processes .cu\n self.src_extensions.append('.cu')\n\n # save references to the default compiler_so and _comple methods\n default_compiler_so = self.compiler_so\n super = self._compile\n\n # now redefine the _compile method. This gets executed for each\n # object but distutils doesn't have the ability to change compilers\n # based on source extension: we add it.\n def _compile(obj, src, ext, cc_args, extra_postargs, pp_opts):\n print(extra_postargs)\n if os.path.splitext(src)[1] == '.cu':\n # use the cuda for .cu files\n self.set_executable('compiler_so', CUDA['nvcc'])\n # use only a subset of the extra_postargs, which are 1-1 translated\n # from the extra_compile_args in the Extension class\n postargs = extra_postargs['nvcc']\n else:\n postargs = extra_postargs['gcc']\n\n super(obj, src, ext, cc_args, postargs, pp_opts)\n # reset the default compiler_so, which we might have changed for cuda\n self.compiler_so = default_compiler_so\n\n # inject our redefined _compile method into the class\n self._compile = _compile\n\n\n# run the customize_compiler\nclass custom_build_ext(build_ext):\n def build_extensions(self):\n customize_compiler_for_nvcc(self.compiler)\n build_ext.build_extensions(self)\n\n\next_modules = [\n Extension(\n \"model.utils.cython_bbox\",\n [\"model/utils/bbox.pyx\"],\n extra_compile_args={'gcc': [\"-Wno-cpp\", \"-Wno-unused-function\"]},\n include_dirs=[numpy_include]\n ),\n Extension(\n 'pycocotools._mask',\n sources=['pycocotools/maskApi.c', 'pycocotools/_mask.pyx'],\n include_dirs=[numpy_include, 'pycocotools'],\n extra_compile_args={\n 'gcc': ['-Wno-cpp', '-Wno-unused-function', '-std=c99']},\n ),\n]\n\nsetup(\n name='faster_rcnn',\n ext_modules=ext_modules,\n # inject our custom trigger\n cmdclass={'build_ext': custom_build_ext},\n)\n"
},
{
"path": "requirements.txt",
"content": "cython\ncffi\nopencv-python\nscipy\nmsgpack\neasydict\nmatplotlib\npyyaml\ntensorboardX\n"
},
{
"path": "test_net.py",
"content": "# --------------------------------------------------------\n# Tensorflow Faster R-CNN\n# Licensed under The MIT License [see LICENSE for details]\n# Written by Jiasen Lu, Jianwei Yang, based on code from Ross Girshick\n# --------------------------------------------------------\nfrom __future__ import absolute_import\nfrom __future__ import division\nfrom __future__ import print_function\n\nimport _init_paths\nimport os\nimport sys\nimport numpy as np\nimport argparse\nimport pprint\nimport pdb\nimport time\n\nimport cv2\n\nimport torch\nfrom torch.autograd import Variable\nimport torch.nn as nn\nimport torch.optim as optim\nimport pickle\nfrom roi_data_layer.roidb import combined_roidb\nfrom roi_data_layer.roibatchLoader import roibatchLoader\nfrom model.utils.config import cfg, cfg_from_file, cfg_from_list, get_output_dir\nfrom model.rpn.bbox_transform import clip_boxes\nfrom model.nms.nms_wrapper import nms\nfrom model.rpn.bbox_transform import bbox_transform_inv\nfrom model.utils.net_utils import save_net, load_net, vis_detections\n\nfrom model.faster_rcnn.vgg16 import vgg16\nfrom model.faster_rcnn.resnet import resnet\n\ntry:\n xrange # Python 2\nexcept NameError:\n xrange = range # Python 3\n\n\ndef parse_args():\n \"\"\"\n Parse input arguments\n \"\"\"\n parser = argparse.ArgumentParser(description='Train a Fast R-CNN network')\n parser.add_argument('--dataset', dest='dataset',\n help='training dataset',\n default='pascal_voc', type=str)\n parser.add_argument('--cfg', dest='cfg_file',\n help='optional config file',\n default='cfgs/vgg16.yml', type=str)\n parser.add_argument('--net', dest='net',\n help='vgg16, res50, res101, res152',\n default='res101', type=str)\n parser.add_argument('--set', dest='set_cfgs',\n help='set config keys', default=None,\n nargs=argparse.REMAINDER)\n parser.add_argument('--load_dir', dest='load_dir',\n help='directory to load models', default=\"models\",\n type=str)\n parser.add_argument('--cuda', dest='cuda',\n help='whether use CUDA',\n action='store_true')\n parser.add_argument('--ls', dest='large_scale',\n help='whether use large imag scale',\n action='store_true')\n parser.add_argument('--mGPUs', dest='mGPUs',\n help='whether use multiple GPUs',\n action='store_true')\n parser.add_argument('--cag', dest='class_agnostic',\n help='whether perform class_agnostic bbox regression',\n action='store_true')\n parser.add_argument('--parallel_type', dest='parallel_type',\n help='which part of model to parallel, 0: all, 1: model before roi pooling',\n default=0, type=int)\n parser.add_argument('--checksession', dest='checksession',\n help='checksession to load model',\n default=1, type=int)\n parser.add_argument('--checkepoch', dest='checkepoch',\n help='checkepoch to load network',\n default=1, type=int)\n parser.add_argument('--checkpoint', dest='checkpoint',\n help='checkpoint to load network',\n default=10021, type=int)\n parser.add_argument('--vis', dest='vis',\n help='visualization mode',\n action='store_true')\n args = parser.parse_args()\n return args\n\nlr = cfg.TRAIN.LEARNING_RATE\nmomentum = cfg.TRAIN.MOMENTUM\nweight_decay = cfg.TRAIN.WEIGHT_DECAY\n\nif __name__ == '__main__':\n\n args = parse_args()\n\n print('Called with args:')\n print(args)\n\n if torch.cuda.is_available() and not args.cuda:\n print(\"WARNING: You have a CUDA device, so you should probably run with --cuda\")\n\n np.random.seed(cfg.RNG_SEED)\n if args.dataset == \"pascal_voc\":\n args.imdb_name = \"voc_2007_trainval\"\n args.imdbval_name = \"voc_2007_test\"\n args.set_cfgs = ['ANCHOR_SCALES', '[8, 16, 32]', 'ANCHOR_RATIOS', '[0.5,1,2]']\n elif args.dataset == \"pascal_voc_0712\":\n args.imdb_name = \"voc_2007_trainval+voc_2012_trainval\"\n args.imdbval_name = \"voc_2007_test\"\n args.set_cfgs = ['ANCHOR_SCALES', '[8, 16, 32]', 'ANCHOR_RATIOS', '[0.5,1,2]']\n elif args.dataset == \"coco\":\n args.imdb_name = \"coco_2014_train+coco_2014_valminusminival\"\n args.imdbval_name = \"coco_2014_minival\"\n args.set_cfgs = ['ANCHOR_SCALES', '[4, 8, 16, 32]', 'ANCHOR_RATIOS', '[0.5,1,2]']\n elif args.dataset == \"imagenet\":\n args.imdb_name = \"imagenet_train\"\n args.imdbval_name = \"imagenet_val\"\n args.set_cfgs = ['ANCHOR_SCALES', '[8, 16, 32]', 'ANCHOR_RATIOS', '[0.5,1,2]']\n elif args.dataset == \"vg\":\n args.imdb_name = \"vg_150-50-50_minitrain\"\n args.imdbval_name = \"vg_150-50-50_minival\"\n args.set_cfgs = ['ANCHOR_SCALES', '[4, 8, 16, 32]', 'ANCHOR_RATIOS', '[0.5,1,2]']\n\n args.cfg_file = \"cfgs/{}_ls.yml\".format(args.net) if args.large_scale else \"cfgs/{}.yml\".format(args.net)\n\n if args.cfg_file is not None:\n cfg_from_file(args.cfg_file)\n if args.set_cfgs is not None:\n cfg_from_list(args.set_cfgs)\n\n print('Using config:')\n pprint.pprint(cfg)\n\n cfg.TRAIN.USE_FLIPPED = False\n imdb, roidb, ratio_list, ratio_index = combined_roidb(args.imdbval_name, False)\n imdb.competition_mode(on=True)\n\n print('{:d} roidb entries'.format(len(roidb)))\n\n input_dir = args.load_dir + \"/\" + args.net + \"/\" + args.dataset\n if not os.path.exists(input_dir):\n raise Exception('There is no input directory for loading network from ' + input_dir)\n load_name = os.path.join(input_dir,\n 'faster_rcnn_{}_{}_{}.pth'.format(args.checksession, args.checkepoch, args.checkpoint))\n\n # initilize the network here.\n if args.net == 'vgg16':\n fasterRCNN = vgg16(imdb.classes, pretrained=False, class_agnostic=args.class_agnostic)\n elif args.net == 'res101':\n fasterRCNN = resnet(imdb.classes, 101, pretrained=False, class_agnostic=args.class_agnostic)\n elif args.net == 'res50':\n fasterRCNN = resnet(imdb.classes, 50, pretrained=False, class_agnostic=args.class_agnostic)\n elif args.net == 'res152':\n fasterRCNN = resnet(imdb.classes, 152, pretrained=False, class_agnostic=args.class_agnostic)\n else:\n print(\"network is not defined\")\n pdb.set_trace()\n\n fasterRCNN.create_architecture()\n\n print(\"load checkpoint %s\" % (load_name))\n checkpoint = torch.load(load_name)\n fasterRCNN.load_state_dict(checkpoint['model'])\n if 'pooling_mode' in checkpoint.keys():\n cfg.POOLING_MODE = checkpoint['pooling_mode']\n\n\n print('load model successfully!')\n # initilize the tensor holder here.\n im_data = torch.FloatTensor(1)\n im_info = torch.FloatTensor(1)\n num_boxes = torch.LongTensor(1)\n gt_boxes = torch.FloatTensor(1)\n\n # ship to cuda\n if args.cuda:\n im_data = im_data.cuda()\n im_info = im_info.cuda()\n num_boxes = num_boxes.cuda()\n gt_boxes = gt_boxes.cuda()\n\n # make variable\n im_data = Variable(im_data)\n im_info = Variable(im_info)\n num_boxes = Variable(num_boxes)\n gt_boxes = Variable(gt_boxes)\n\n if args.cuda:\n cfg.CUDA = True\n\n if args.cuda:\n fasterRCNN.cuda()\n\n start = time.time()\n max_per_image = 100\n\n vis = args.vis\n\n if vis:\n thresh = 0.05\n else:\n thresh = 0.0\n\n save_name = 'faster_rcnn_10'\n num_images = len(imdb.image_index)\n all_boxes = [[[] for _ in xrange(num_images)]\n for _ in xrange(imdb.num_classes)]\n\n output_dir = get_output_dir(imdb, save_name)\n dataset = roibatchLoader(roidb, ratio_list, ratio_index, 1, \\\n imdb.num_classes, training=False, normalize = False)\n dataloader = torch.utils.data.DataLoader(dataset, batch_size=1,\n shuffle=False, num_workers=0,\n pin_memory=True)\n\n data_iter = iter(dataloader)\n\n _t = {'im_detect': time.time(), 'misc': time.time()}\n det_file = os.path.join(output_dir, 'detections.pkl')\n\n fasterRCNN.eval()\n empty_array = np.transpose(np.array([[],[],[],[],[]]), (1,0))\n for i in range(num_images):\n\n data = next(data_iter)\n im_data.data.resize_(data[0].size()).copy_(data[0])\n im_info.data.resize_(data[1].size()).copy_(data[1])\n gt_boxes.data.resize_(data[2].size()).copy_(data[2])\n num_boxes.data.resize_(data[3].size()).copy_(data[3])\n\n det_tic = time.time()\n rois, cls_prob, bbox_pred, \\\n rpn_loss_cls, rpn_loss_box, \\\n RCNN_loss_cls, RCNN_loss_bbox, \\\n rois_label = fasterRCNN(im_data, im_info, gt_boxes, num_boxes)\n\n scores = cls_prob.data\n boxes = rois.data[:, :, 1:5]\n\n if cfg.TEST.BBOX_REG:\n # Apply bounding-box regression deltas\n box_deltas = bbox_pred.data\n if cfg.TRAIN.BBOX_NORMALIZE_TARGETS_PRECOMPUTED:\n # Optionally normalize targets by a precomputed mean and stdev\n if args.class_agnostic:\n box_deltas = box_deltas.view(-1, 4) * torch.FloatTensor(cfg.TRAIN.BBOX_NORMALIZE_STDS).cuda() \\\n + torch.FloatTensor(cfg.TRAIN.BBOX_NORMALIZE_MEANS).cuda()\n box_deltas = box_deltas.view(1, -1, 4)\n else:\n box_deltas = box_deltas.view(-1, 4) * torch.FloatTensor(cfg.TRAIN.BBOX_NORMALIZE_STDS).cuda() \\\n + torch.FloatTensor(cfg.TRAIN.BBOX_NORMALIZE_MEANS).cuda()\n box_deltas = box_deltas.view(1, -1, 4 * len(imdb.classes))\n\n pred_boxes = bbox_transform_inv(boxes, box_deltas, 1)\n pred_boxes = clip_boxes(pred_boxes, im_info.data, 1)\n else:\n # Simply repeat the boxes, once for each class\n _ = torch.from_numpy(np.tile(boxes, (1, scores.shape[1])))\n pred_boxes = _.cuda() if args.cuda > 0 else _\n\n pred_boxes /= data[1][0][2].item()\n\n scores = scores.squeeze()\n pred_boxes = pred_boxes.squeeze()\n det_toc = time.time()\n detect_time = det_toc - det_tic\n misc_tic = time.time()\n if vis:\n im = cv2.imread(imdb.image_path_at(i))\n im2show = np.copy(im)\n for j in xrange(1, imdb.num_classes):\n inds = torch.nonzero(scores[:,j]>thresh).view(-1)\n # if there is det\n if inds.numel() > 0:\n cls_scores = scores[:,j][inds]\n _, order = torch.sort(cls_scores, 0, True)\n if args.class_agnostic:\n cls_boxes = pred_boxes[inds, :]\n else:\n cls_boxes = pred_boxes[inds][:, j * 4:(j + 1) * 4]\n \n cls_dets = torch.cat((cls_boxes, cls_scores.unsqueeze(1)), 1)\n # cls_dets = torch.cat((cls_boxes, cls_scores), 1)\n cls_dets = cls_dets[order]\n keep = nms(cls_dets, cfg.TEST.NMS)\n cls_dets = cls_dets[keep.view(-1).long()]\n if vis:\n im2show = vis_detections(im2show, imdb.classes[j], cls_dets.cpu().numpy(), 0.3)\n all_boxes[j][i] = cls_dets.cpu().numpy()\n else:\n all_boxes[j][i] = empty_array\n\n # Limit to max_per_image detections *over all classes*\n if max_per_image > 0:\n image_scores = np.hstack([all_boxes[j][i][:, -1]\n for j in xrange(1, imdb.num_classes)])\n if len(image_scores) > max_per_image:\n image_thresh = np.sort(image_scores)[-max_per_image]\n for j in xrange(1, imdb.num_classes):\n keep = np.where(all_boxes[j][i][:, -1] >= image_thresh)[0]\n all_boxes[j][i] = all_boxes[j][i][keep, :]\n\n misc_toc = time.time()\n nms_time = misc_toc - misc_tic\n\n sys.stdout.write('im_detect: {:d}/{:d} {:.3f}s {:.3f}s \\r' \\\n .format(i + 1, num_images, detect_time, nms_time))\n sys.stdout.flush()\n\n if vis:\n cv2.imwrite('result.png', im2show)\n pdb.set_trace()\n #cv2.imshow('test', im2show)\n #cv2.waitKey(0)\n\n with open(det_file, 'wb') as f:\n pickle.dump(all_boxes, f, pickle.HIGHEST_PROTOCOL)\n\n print('Evaluating detections')\n imdb.evaluate_detections(all_boxes, output_dir)\n\n end = time.time()\n print(\"test time: %0.4fs\" % (end - start))\n"
},
{
"path": "trainval_net.py",
"content": "# --------------------------------------------------------\n# Pytorch multi-GPU Faster R-CNN\n# Licensed under The MIT License [see LICENSE for details]\n# Written by Jiasen Lu, Jianwei Yang, based on code from Ross Girshick\n# --------------------------------------------------------\nfrom __future__ import absolute_import\nfrom __future__ import division\nfrom __future__ import print_function\n\nimport _init_paths\nimport os\nimport sys\nimport numpy as np\nimport argparse\nimport pprint\nimport pdb\nimport time\n\nimport torch\nfrom torch.autograd import Variable\nimport torch.nn as nn\nimport torch.optim as optim\n\nimport torchvision.transforms as transforms\nfrom torch.utils.data.sampler import Sampler\n\nfrom roi_data_layer.roidb import combined_roidb\nfrom roi_data_layer.roibatchLoader import roibatchLoader\nfrom model.utils.config import cfg, cfg_from_file, cfg_from_list, get_output_dir\nfrom model.utils.net_utils import weights_normal_init, save_net, load_net, \\\n adjust_learning_rate, save_checkpoint, clip_gradient\n\nfrom model.faster_rcnn.vgg16 import vgg16\nfrom model.faster_rcnn.resnet import resnet\n\ndef parse_args():\n \"\"\"\n Parse input arguments\n \"\"\"\n parser = argparse.ArgumentParser(description='Train a Fast R-CNN network')\n parser.add_argument('--dataset', dest='dataset',\n help='training dataset',\n default='pascal_voc', type=str)\n parser.add_argument('--net', dest='net',\n help='vgg16, res101',\n default='vgg16', type=str)\n parser.add_argument('--start_epoch', dest='start_epoch',\n help='starting epoch',\n default=1, type=int)\n parser.add_argument('--epochs', dest='max_epochs',\n help='number of epochs to train',\n default=20, type=int)\n parser.add_argument('--disp_interval', dest='disp_interval',\n help='number of iterations to display',\n default=100, type=int)\n parser.add_argument('--checkpoint_interval', dest='checkpoint_interval',\n help='number of iterations to display',\n default=10000, type=int)\n\n parser.add_argument('--save_dir', dest='save_dir',\n help='directory to save models', default=\"models\",\n type=str)\n parser.add_argument('--nw', dest='num_workers',\n help='number of workers to load data',\n default=0, type=int)\n parser.add_argument('--cuda', dest='cuda',\n help='whether use CUDA',\n action='store_true')\n parser.add_argument('--ls', dest='large_scale',\n help='whether use large imag scale',\n action='store_true') \n parser.add_argument('--mGPUs', dest='mGPUs',\n help='whether use multiple GPUs',\n action='store_true')\n parser.add_argument('--bs', dest='batch_size',\n help='batch_size',\n default=1, type=int)\n parser.add_argument('--cag', dest='class_agnostic',\n help='whether to perform class_agnostic bbox regression',\n action='store_true')\n\n# config optimization\n parser.add_argument('--o', dest='optimizer',\n help='training optimizer',\n default=\"sgd\", type=str)\n parser.add_argument('--lr', dest='lr',\n help='starting learning rate',\n default=0.001, type=float)\n parser.add_argument('--lr_decay_step', dest='lr_decay_step',\n help='step to do learning rate decay, unit is epoch',\n default=5, type=int)\n parser.add_argument('--lr_decay_gamma', dest='lr_decay_gamma',\n help='learning rate decay ratio',\n default=0.1, type=float)\n\n# set training session\n parser.add_argument('--s', dest='session',\n help='training session',\n default=1, type=int)\n\n# resume trained model\n parser.add_argument('--r', dest='resume',\n help='resume checkpoint or not',\n default=False, type=bool)\n parser.add_argument('--checksession', dest='checksession',\n help='checksession to load model',\n default=1, type=int)\n parser.add_argument('--checkepoch', dest='checkepoch',\n help='checkepoch to load model',\n default=1, type=int)\n parser.add_argument('--checkpoint', dest='checkpoint',\n help='checkpoint to load model',\n default=0, type=int)\n# log and display\n parser.add_argument('--use_tfb', dest='use_tfboard',\n help='whether use tensorboard',\n action='store_true')\n\n args = parser.parse_args()\n return args\n\n\nclass sampler(Sampler):\n def __init__(self, train_size, batch_size):\n self.num_data = train_size\n self.num_per_batch = int(train_size / batch_size)\n self.batch_size = batch_size\n self.range = torch.arange(0,batch_size).view(1, batch_size).long()\n self.leftover_flag = False\n if train_size % batch_size:\n self.leftover = torch.arange(self.num_per_batch*batch_size, train_size).long()\n self.leftover_flag = True\n\n def __iter__(self):\n rand_num = torch.randperm(self.num_per_batch).view(-1,1) * self.batch_size\n self.rand_num = rand_num.expand(self.num_per_batch, self.batch_size) + self.range\n\n self.rand_num_view = self.rand_num.view(-1)\n\n if self.leftover_flag:\n self.rand_num_view = torch.cat((self.rand_num_view, self.leftover),0)\n\n return iter(self.rand_num_view)\n\n def __len__(self):\n return self.num_data\n\nif __name__ == '__main__':\n\n args = parse_args()\n\n print('Called with args:')\n print(args)\n\n if args.dataset == \"pascal_voc\":\n args.imdb_name = \"voc_2007_trainval\"\n args.imdbval_name = \"voc_2007_test\"\n args.set_cfgs = ['ANCHOR_SCALES', '[8, 16, 32]', 'ANCHOR_RATIOS', '[0.5,1,2]', 'MAX_NUM_GT_BOXES', '20']\n elif args.dataset == \"pascal_voc_0712\":\n args.imdb_name = \"voc_2007_trainval+voc_2012_trainval\"\n args.imdbval_name = \"voc_2007_test\"\n args.set_cfgs = ['ANCHOR_SCALES', '[8, 16, 32]', 'ANCHOR_RATIOS', '[0.5,1,2]', 'MAX_NUM_GT_BOXES', '20']\n elif args.dataset == \"coco\":\n args.imdb_name = \"coco_2014_train+coco_2014_valminusminival\"\n args.imdbval_name = \"coco_2014_minival\"\n args.set_cfgs = ['ANCHOR_SCALES', '[4, 8, 16, 32]', 'ANCHOR_RATIOS', '[0.5,1,2]', 'MAX_NUM_GT_BOXES', '50']\n elif args.dataset == \"imagenet\":\n args.imdb_name = \"imagenet_train\"\n args.imdbval_name = \"imagenet_val\"\n args.set_cfgs = ['ANCHOR_SCALES', '[4, 8, 16, 32]', 'ANCHOR_RATIOS', '[0.5,1,2]', 'MAX_NUM_GT_BOXES', '30']\n elif args.dataset == \"vg\":\n # train sizes: train, smalltrain, minitrain\n # train scale: ['150-50-20', '150-50-50', '500-150-80', '750-250-150', '1750-700-450', '1600-400-20']\n args.imdb_name = \"vg_150-50-50_minitrain\"\n args.imdbval_name = \"vg_150-50-50_minival\"\n args.set_cfgs = ['ANCHOR_SCALES', '[4, 8, 16, 32]', 'ANCHOR_RATIOS', '[0.5,1,2]', 'MAX_NUM_GT_BOXES', '50']\n\n args.cfg_file = \"cfgs/{}_ls.yml\".format(args.net) if args.large_scale else \"cfgs/{}.yml\".format(args.net)\n\n if args.cfg_file is not None:\n cfg_from_file(args.cfg_file)\n if args.set_cfgs is not None:\n cfg_from_list(args.set_cfgs)\n\n print('Using config:')\n pprint.pprint(cfg)\n np.random.seed(cfg.RNG_SEED)\n\n #torch.backends.cudnn.benchmark = True\n if torch.cuda.is_available() and not args.cuda:\n print(\"WARNING: You have a CUDA device, so you should probably run with --cuda\")\n\n # train set\n # -- Note: Use validation set and disable the flipped to enable faster loading.\n cfg.TRAIN.USE_FLIPPED = True\n cfg.USE_GPU_NMS = args.cuda\n imdb, roidb, ratio_list, ratio_index = combined_roidb(args.imdb_name)\n train_size = len(roidb)\n\n print('{:d} roidb entries'.format(len(roidb)))\n\n output_dir = args.save_dir + \"/\" + args.net + \"/\" + args.dataset\n if not os.path.exists(output_dir):\n os.makedirs(output_dir)\n\n sampler_batch = sampler(train_size, args.batch_size)\n\n dataset = roibatchLoader(roidb, ratio_list, ratio_index, args.batch_size, \\\n imdb.num_classes, training=True)\n\n dataloader = torch.utils.data.DataLoader(dataset, batch_size=args.batch_size,\n sampler=sampler_batch, num_workers=args.num_workers)\n\n # initilize the tensor holder here.\n im_data = torch.FloatTensor(1)\n im_info = torch.FloatTensor(1)\n num_boxes = torch.LongTensor(1)\n gt_boxes = torch.FloatTensor(1)\n\n # ship to cuda\n if args.cuda:\n im_data = im_data.cuda()\n im_info = im_info.cuda()\n num_boxes = num_boxes.cuda()\n gt_boxes = gt_boxes.cuda()\n\n # make variable\n im_data = Variable(im_data)\n im_info = Variable(im_info)\n num_boxes = Variable(num_boxes)\n gt_boxes = Variable(gt_boxes)\n\n if args.cuda:\n cfg.CUDA = True\n\n # initilize the network here.\n if args.net == 'vgg16':\n fasterRCNN = vgg16(imdb.classes, pretrained=True, class_agnostic=args.class_agnostic)\n elif args.net == 'res101':\n fasterRCNN = resnet(imdb.classes, 101, pretrained=True, class_agnostic=args.class_agnostic)\n elif args.net == 'res50':\n fasterRCNN = resnet(imdb.classes, 50, pretrained=True, class_agnostic=args.class_agnostic)\n elif args.net == 'res152':\n fasterRCNN = resnet(imdb.classes, 152, pretrained=True, class_agnostic=args.class_agnostic)\n else:\n print(\"network is not defined\")\n pdb.set_trace()\n\n fasterRCNN.create_architecture()\n\n lr = cfg.TRAIN.LEARNING_RATE\n lr = args.lr\n #tr_momentum = cfg.TRAIN.MOMENTUM\n #tr_momentum = args.momentum\n\n params = []\n for key, value in dict(fasterRCNN.named_parameters()).items():\n if value.requires_grad:\n if 'bias' in key:\n params += [{'params':[value],'lr':lr*(cfg.TRAIN.DOUBLE_BIAS + 1), \\\n 'weight_decay': cfg.TRAIN.BIAS_DECAY and cfg.TRAIN.WEIGHT_DECAY or 0}]\n else:\n params += [{'params':[value],'lr':lr, 'weight_decay': cfg.TRAIN.WEIGHT_DECAY}]\n\n if args.optimizer == \"adam\":\n lr = lr * 0.1\n optimizer = torch.optim.Adam(params)\n\n elif args.optimizer == \"sgd\":\n optimizer = torch.optim.SGD(params, momentum=cfg.TRAIN.MOMENTUM)\n\n if args.cuda:\n fasterRCNN.cuda()\n\n if args.resume:\n load_name = os.path.join(output_dir,\n 'faster_rcnn_{}_{}_{}.pth'.format(args.checksession, args.checkepoch, args.checkpoint))\n print(\"loading checkpoint %s\" % (load_name))\n checkpoint = torch.load(load_name)\n args.session = checkpoint['session']\n args.start_epoch = checkpoint['epoch']\n fasterRCNN.load_state_dict(checkpoint['model'])\n optimizer.load_state_dict(checkpoint['optimizer'])\n lr = optimizer.param_groups[0]['lr']\n if 'pooling_mode' in checkpoint.keys():\n cfg.POOLING_MODE = checkpoint['pooling_mode']\n print(\"loaded checkpoint %s\" % (load_name))\n\n if args.mGPUs:\n fasterRCNN = nn.DataParallel(fasterRCNN)\n\n iters_per_epoch = int(train_size / args.batch_size)\n\n if args.use_tfboard:\n from tensorboardX import SummaryWriter\n logger = SummaryWriter(\"logs\")\n\n for epoch in range(args.start_epoch, args.max_epochs + 1):\n # setting to train mode\n fasterRCNN.train()\n loss_temp = 0\n start = time.time()\n\n if epoch % (args.lr_decay_step + 1) == 0:\n adjust_learning_rate(optimizer, args.lr_decay_gamma)\n lr *= args.lr_decay_gamma\n\n data_iter = iter(dataloader)\n for step in range(iters_per_epoch):\n data = next(data_iter)\n im_data.data.resize_(data[0].size()).copy_(data[0])\n im_info.data.resize_(data[1].size()).copy_(data[1])\n gt_boxes.data.resize_(data[2].size()).copy_(data[2])\n num_boxes.data.resize_(data[3].size()).copy_(data[3])\n\n fasterRCNN.zero_grad()\n rois, cls_prob, bbox_pred, \\\n rpn_loss_cls, rpn_loss_box, \\\n RCNN_loss_cls, RCNN_loss_bbox, \\\n rois_label = fasterRCNN(im_data, im_info, gt_boxes, num_boxes)\n\n loss = rpn_loss_cls.mean() + rpn_loss_box.mean() \\\n + RCNN_loss_cls.mean() + RCNN_loss_bbox.mean()\n loss_temp += loss.item()\n\n # backward\n optimizer.zero_grad()\n loss.backward()\n if args.net == \"vgg16\":\n clip_gradient(fasterRCNN, 10.)\n optimizer.step()\n\n if step % args.disp_interval == 0:\n end = time.time()\n if step > 0:\n loss_temp /= (args.disp_interval + 1)\n\n if args.mGPUs:\n loss_rpn_cls = rpn_loss_cls.mean().item()\n loss_rpn_box = rpn_loss_box.mean().item()\n loss_rcnn_cls = RCNN_loss_cls.mean().item()\n loss_rcnn_box = RCNN_loss_bbox.mean().item()\n fg_cnt = torch.sum(rois_label.data.ne(0))\n bg_cnt = rois_label.data.numel() - fg_cnt\n else:\n loss_rpn_cls = rpn_loss_cls.item()\n loss_rpn_box = rpn_loss_box.item()\n loss_rcnn_cls = RCNN_loss_cls.item()\n loss_rcnn_box = RCNN_loss_bbox.item()\n fg_cnt = torch.sum(rois_label.data.ne(0))\n bg_cnt = rois_label.data.numel() - fg_cnt\n\n print(\"[session %d][epoch %2d][iter %4d/%4d] loss: %.4f, lr: %.2e\" \\\n % (args.session, epoch, step, iters_per_epoch, loss_temp, lr))\n print(\"\\t\\t\\tfg/bg=(%d/%d), time cost: %f\" % (fg_cnt, bg_cnt, end-start))\n print(\"\\t\\t\\trpn_cls: %.4f, rpn_box: %.4f, rcnn_cls: %.4f, rcnn_box %.4f\" \\\n % (loss_rpn_cls, loss_rpn_box, loss_rcnn_cls, loss_rcnn_box))\n if args.use_tfboard:\n info = {\n 'loss': loss_temp,\n 'loss_rpn_cls': loss_rpn_cls,\n 'loss_rpn_box': loss_rpn_box,\n 'loss_rcnn_cls': loss_rcnn_cls,\n 'loss_rcnn_box': loss_rcnn_box\n }\n logger.add_scalars(\"logs_s_{}/losses\".format(args.session), info, (epoch - 1) * iters_per_epoch + step)\n\n loss_temp = 0\n start = time.time()\n\n \n save_name = os.path.join(output_dir, 'faster_rcnn_{}_{}_{}.pth'.format(args.session, epoch, step))\n save_checkpoint({\n 'session': args.session,\n 'epoch': epoch + 1,\n 'model': fasterRCNN.module.state_dict() if args.mGPUs else fasterRCNN.state_dict(),\n 'optimizer': optimizer.state_dict(),\n 'pooling_mode': cfg.POOLING_MODE,\n 'class_agnostic': args.class_agnostic,\n }, save_name)\n print('save model: {}'.format(save_name))\n\n if args.use_tfboard:\n logger.close()\n"
}
] | | |