[
  {
    "path": ".gitignore",
    "content": "snapshots/*.caffemodel\nsvg\n"
  },
  {
    "path": "LICENSE",
    "content": "Copyright the Chinese University of Hong Kong. All rights reserved.\n\nContact persons:\nBo Dai (db014 [at] ie [dot] cuhk [dot] edu [dot] hk)\n\nThis software is being made available for research use only.\nAny commercial use or redistribution of this software requires a license from\nthe Chinese University of Hong Kong.\n\nYou may use this work subject to the following conditions:\n\n1. This work is provided \"as is\" by the copyright holder, with\nabsolutely no warranties of correctness, fitness, intellectual property\nownership, or anything else whatsoever.  You use the work\nentirely at your own risk.  The copyright holder will not be liable for\nany legal damages whatsoever connected with the use of this work.\n\n2. The copyright holder retain all copyright to the work. All copies of\nthe work and all works derived from it must contain (1) this copyright\nnotice, and (2) additional notices describing the content, dates and\ncopyright holder of modifications or additions made to the work, if\nany, including distribution and use conditions and intellectual property\nclaims.  Derived works must be clearly distinguished from the original\nwork, both by name and by the prominent inclusion of explicit\ndescriptions of overlaps and differences.\n\n3. The names and trademarks of the copyright holder may not be used in\nadvertising or publicity related to this work without specific prior\nwritten permission. \n\n4. In return for the free use of this work, you are requested, but not\nlegally required, to do the following:\n\n* If you become aware of factors that may significantly affect other\n  users of the work, for example major bugs or\n  deficiencies or possible intellectual property issues, you are\n  requested to report them to the copyright holder, if possible\n  including redistributable fixes or workarounds.\n\n* If you use the work in scientific research or as part of a larger\n  software system, you are requested to cite the use in any related\n  publications or technical documentation. The work is based upon:\n \n \n  \"Detecting Visual Relationship with Deep Relational Networks\"\n  Bo Dai, Yuqi Zhang, Dahua Lin, Computer Vision and Pattern Recognition,\n  (CVPR 2017), 2017 (oral).\n\nthis copyright notice must be retained with all copies of the software,\nincluding any modified or derived versions.\n"
  },
  {
    "path": "README.md",
    "content": "# Code of [Detecting Visual Relationships with Deep Relational Networks](https://arxiv.org/abs/1704.03114)\n\nThe code is written in python, and all networks are implemented using [Caffe](https://github.com/BVLC/caffe).\n\n## Datasets \n\n* [VRD](http://cs.stanford.edu/people/ranjaykrishna/vrd/dataset.zip)\n* sVG: subset of [Visual Genome](https://visualgenome.org/)\n\t- [Link](https://drive.google.com/file/d/0B5RJWjAhdT04SXRfVHBKZ0dOTzQ/view?usp=sharing&resourcekey=0-bW_W0QVJOfaNs5NyGjDjbQ)\n\t- Images can be downloaded from the website of Visual Genome\n\t- Remarks: eventually I found no time to further clean it. This subset has a manually cleaned list for relationship predicates. The list for objects may need further cleaning, although Faster-RCNN can get a recall@20 around 50%.\n\t- Using our method, you can get the corresponding results reported in the paper on this dataset.\n\n## Networks\n\nThis repo contains three kinds of networks. And all of them get the raw response for predicate based on both appearance cues and spatial cues,\nfollowed by a refinement according to responses of the subject, the object and the predicate.\nThe networks are designed for the task of predicate recognition, \nwhere ground-truth labels of the subject and the object are provided as inputs.\nTherefore, in these networks, responses of the subject and the object are replaced with indicator vectors,\nand only response of the predicate will be refined.\n\nIn these networks, the subnet for appearance cues is VGG16, and the subnet for spatial cues consists of three conv layers.\nAnd outputs of both subnets are combined via a customized concatenate layer,\nfollowed by two fc layers to generate raw response for the predicate.\n\nThe customized concatenate layer is used for combining the output of a fc layer and channels of the output of a conv layer,\nwhich can be replaced with caffe's Concat layer\nif the last conv layer in spatial subnet (conv3_p) is equivalently replaced with a fc layer.\n\nThe details of these networks are\n\n* drnet_8units_softmax: it has 8 inference units with softmax function as the activation function.\n\n* drnet_8units_linear_shareweight: it has 8 inference units with no activation function, and the weights are shared across units.\n\n* drnet_8units_relu_shareweight: it has 8 inference units with relu function as the activation function, and the weights are shared across units.\n\n### Training\n\nThe training procedure is component-by-component. \nSpecifically, a network usually contains three components, \nnamely the subnet for appearance (A), the subnet for spatial cues (S), and the drnet for statistical dependencies (D).\nIn training, we train the network as follow:\n* train A in isolation\n* train S in isolation\n* train A + S in isolation, with weights initialized from previous steps\n* train A + S + D jointly, with weights initialized from previous steps\n\nEach step we use the same loss, and we use dropout to avoid overfit.\n\n### Recalls on Predicate Recognition\n\n| Networks | Recall@50 | Recall@100 |\n| --- | :---: | :---: |\n| drnet_8units_softmax | 75.22 | 77.55 |\n| drnet_8units_linear_shareweight | 78.57 | 79.94 |\n| drnet_8units_relu_shareweight | 80.86 | 81.83 |\n\n## Codes\n\n* lib/: python layers, as well as auxiliary files for evaluation\n* prototxts/: training and testing prototxts\n* tools/: python codes for preparing data and evaluation\n* snapshots/: pretrain models\n\n## Finetune or Evaluate\n\n1. Download the dataset [VRD](https://github.com/Prof-Lu-Cewu/Visual-Relationship-Detection)\n2. Preprocess the dataset using tools/preprare_data.py\n3. Download one pretrain model in snapshots/\n4. Finetune or Evaluate using corresponding prototxts in prototxts/\n\n## Pair Filter\n\n### Structure\n\n![Structure](https://raw.github.com/doubledaibo/drnet/master/imgs/pair_filter.jpg)\n\n### Training\n\nTo train this network, we randomly sample pairs of bounding boxes (with labels) from \neach training image, treating those with 0.5 IoU (or above) with any ground-truth pairs (with same labels)\nas positive samples, and the rest as negative samples.\n\n## Citation\n\nIf you use this code, please cite the following paper(s):\n\n\t@article{dai2017detecting,\n\t\ttitle={Detecting Visual Relationships with Deep Relational Networks},\n\t\tauthor={Dai, Bo and Zhang, Yuqi and Lin, Dahua},\n  \t\tbooktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},\n  \t\tyear={2017}\n\t}\n\n## License\n\nThis code is used for research only. See LICENSE for details.\n"
  },
  {
    "path": "lib/customize_layers/__init__.py",
    "content": ""
  },
  {
    "path": "lib/customize_layers/concat_layer.py",
    "content": "import caffe\nimport numpy as np\nimport json\n\nclass Layer(caffe.Layer):\n\tdef setup(self, bottom, top):\n\t\tself._sum = 0\n\t\tfor i in xrange(len(bottom)):\n\t\t\tself._sum += bottom[i].data.shape[1]\n\n\tdef reshape(self, bottom, top):\n\t\ttop[0].reshape(bottom[0].data.shape[0], self._sum)\n\n\tdef forward(self, bottom, top):\n\t\toffset = 0\n\t\tfor i in xrange(len(bottom)):\n\t\t\tif len(bottom[i].data.shape) == 2:\n\t\t\t\ttop[0].data[:, offset : offset + bottom[i].data.shape[1]] = bottom[i].data\n\t\t\telse:\n\t\t\t\ttop[0].data[:, offset : offset + bottom[i].data.shape[1]] = bottom[i].data[:, :, 0, 0]\n\t\t\toffset += bottom[i].data.shape[1]\n\t\n\tdef backward(self, top, propagate_down, bottom):\n\t\toffset = 0\n\t\tfor i in xrange(len(bottom)):\n\t\t\tif len(bottom[i].data.shape) == 2:\n\t\t\t\tbottom[i].diff[...] = top[0].diff[:, offset : offset + bottom[i].data.shape[1]]\n\t\t\telse:\n\t\t\t\tbottom[i].diff[:, :, 0, 0] = top[0].diff[:, offset : offset + bottom[i].data.shape[1]]\n\t\t\toffset += bottom[i].data.shape[1]\n\n\t\t\n"
  },
  {
    "path": "lib/rel_data_layer/__init__.py",
    "content": ""
  },
  {
    "path": "lib/rel_data_layer/layer.py",
    "content": "import caffe\nimport numpy as np\nimport json\nimport math\nimport os.path as osp\nimport os\nimport cv2\nimport h5py\n\nclass RelDataLayer(caffe.Layer):\n\tdef _shuffle_inds(self):\n\t\tself._perm = np.random.permutation(np.arange(self._num_instance))\n\t\tself._cur = 0\n\n\tdef _get_next_batch_ids(self):\n\t\tif self._cur + self._batch_size > self._num_instance:\t\n\t\t\tself._shuffle_inds()\n\t\tids = self._perm[self._cur : self._cur + self._batch_size]\n\t\tself._cur += self._batch_size\t\n\t\treturn ids\n\n\tdef _getAppr(self, im, bb):\n\t\tsubim = im[bb[1] : bb[3], bb[0] : bb[2], :]\n\t\tsubim = cv2.resize(subim, None, None, 224.0 / subim.shape[1], 224.0 / subim.shape[0], interpolation=cv2.INTER_LINEAR)\n\t\tpixel_means = np.array([[[103.939, 116.779, 123.68]]])\n\t\tsubim -= pixel_means\n\t\tsubim = subim.transpose((2, 0, 1))\n\t\treturn subim\n\n\tdef _getDualMask(self, ih, iw, bb):\n\t\trh = 32.0 / ih\n\t\trw = 32.0 / iw\n\t\tx1 = max(0, int(math.floor(bb[0] * rw)))\n\t\tx2 = min(32, int(math.ceil(bb[2] * rw)))\n\t\ty1 = max(0, int(math.floor(bb[1] * rh)))\n\t\ty2 = min(32, int(math.ceil(bb[3] * rh)))\n\t\tmask = np.zeros((32, 32))\n\t\tmask[y1 : y2, x1 : x2] = 1\n\t\tassert(mask.sum() == (y2 - y1) * (x2 - x1))\n\t\treturn mask\t\t\n\t\n\tdef _get_next_batch(self):\n\t\tids = self._get_next_batch_ids()\n\t\tqas = []\n\t\tqbs = []\n\t\tims = []\n\t\tposes = []\n\t\tlabels = []\n\t\tfor id in ids:\n\t\t\tsample = self._samples[id]\n\t\t\tim = cv2.imread(sample[\"imPath\"]).astype(np.float32, copy=False)\n\t\t\tih = im.shape[0]\t\n\t\t\tiw = im.shape[1]\n\t\t\tqa = np.zeros(self._nclass)\n\t\t\tqa[sample[\"aLabel\"] - 1] = 1\n\t\t\tqas.append(qa)\n\t\t\tqb = np.zeros(self._nclass)\n\t\t\tqb[sample[\"bLabel\"] - 1] = 1\n\t\t\tqbs.append(qb)\n\t\t\tims.append(self._getAppr(im, sample[\"rBBox\"]))\n\t\t\tposes.append([self._getDualMask(ih, iw, sample[\"aBBox\"]), \\\n\t\t\t\t\tself._getDualMask(ih, iw, sample[\"bBBox\"])])\n\t\t\tlabels.append(sample[\"rLabel\"])\n\t\treturn {\"qa\": np.array(qas), \"qb\": np.array(qbs), \"im\": np.array(ims), \"posdata\": np.array(poses), \"labels\": np.array(labels)}\n\t\n\tdef setup(self, bottom, top):\n\t\tlayer_params = json.loads(self.param_str)\n\t\tself._samples = json.load(open(layer_params[\"dataset\"]))\n\t\tself._num_instance = len(self._samples)\n\t\tself._batch_size = layer_params[\"batch_size\"]\n\t\tself._nclass = layer_params[\"nclass\"]\n\t\t\n\t\tself._name_to_top_map = {\"qa\": 0, \"qb\": 1, \"im\": 2, \"posdata\": 3, \"labels\": 4}\n\t\tself._shuffle_inds()\n\t\ttop[0].reshape(self._batch_size, self._nclass)\n\t\ttop[1].reshape(self._batch_size, self._nclass)\n\t\ttop[2].reshape(self._batch_size, 3, 224, 224)\n\t\ttop[3].reshape(self._batch_size, 2, 32, 32)\n\t\ttop[4].reshape(self._batch_size)\n\t\t\n\tdef forward(self, bottom, top):\n\t\tbatch = self._get_next_batch()\n\t\tfor blob_name, blob in batch.iteritems():\n\t\t\tidx = self._name_to_top_map[blob_name]\t\n\t\t\ttop[idx].reshape(*(blob.shape))\n\t\t\ttop[idx].data[...] = blob.astype(np.float32, copy=False)\n\t\n\tdef backward(self, top, propagate_down, bottom):\n\t\tpass\n\n\tdef reshape(self, bottom, top):\n\t\tpass\n\n"
  },
  {
    "path": "lib/utils/__init__.py",
    "content": ""
  },
  {
    "path": "lib/utils/eval_utils.py",
    "content": "def computeArea(bb):\n\treturn max(0, bb[2] - bb[0] + 1) * max(0, bb[3] - bb[1] + 1)\n\ndef computeIoU(bb1, bb2):\n\tibb = [max(bb1[0], bb2[0]), \\\n\t\tmax(bb1[1], bb2[1]), \\\n\t\tmin(bb1[2], bb2[2]), \\\n\t\tmin(bb1[3], bb2[3])]\n\tiArea = computeArea(ibb)\n\tuArea = computeArea(bb1) + computeArea(bb2) - iArea\n\treturn (iArea + 0.0) / uArea\n\n"
  },
  {
    "path": "prototxts/drnet_8units_linear_shareweight.prototxt",
    "content": "layer {\n        name: \"data\"\n        type: \"Python\"\n        top: \"qa\"\n        top: \"qb\"\n        top: \"im\"\n        top: \"posdata\"\n        top: \"labels\"\n        include {\n                phase: TRAIN     \n        }         \n        python_param {                 \n                module: 'rel_data_layer.layer'              \n                layer: 'RelDataLayer'                 \n                param_str: '{\"dataset\": \"reltrain.json\", \"batch_size\": 25, \"nclass\": 100}'   \n        }         \n}                    \nlayer {           \n        name: \"data\"                   \n        type: \"Python\"                 \n        top: \"qa\" \n        top: \"qb\" \n        top: \"im\" \n        top: \"posdata\"                 \n        top: \"labels\"                  \n        include { \n                phase: TEST            \n        }         \n        python_param {                 \n                module: 'rel_data_layer.layer'              \n                layer: 'RelDataLayer'\n                param_str: '{\"dataset\": \"reltest.json\", \"batch_size\": 25, \"nclass\": 100}'\n        }         \n}\n# Appearance Subnet\nlayer {\n\tname: \"conv1_1\"\n\ttype: \"Convolution\"\n\tbottom: \"im\"\n\ttop: \"conv1_1\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t} \n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tconvolution_param {\n\t\tnum_output: 64\n\t\tpad: 1\n\t\tkernel_size: 3\n\t}\n}\nlayer {\n\tname: \"relu1_1\"\n\ttype: \"ReLU\"\n\tbottom: \"conv1_1\"\n\ttop: \"conv1_1\"\n}\nlayer {\n\tname: \"conv1_2\"\n\ttype: \"Convolution\"\n\tbottom: \"conv1_1\"\n\ttop: \"conv1_2\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tconvolution_param {\n\t\tnum_output: 64\n\t\tpad: 1\n\t\tkernel_size: 3\n\t}\n}\nlayer {\n\tname: \"relu1_2\"\n\ttype: \"ReLU\"\n\tbottom: \"conv1_2\"\n\ttop: \"conv1_2\"\n}\nlayer {\n\tname: \"pool1\"\n\ttype: \"Pooling\"\n\tbottom: \"conv1_2\"\n\ttop: \"pool1\"\n\tpooling_param {\n\t\tpool: MAX\n\t\tkernel_size: 2\n\t\tstride: 2\n\t}\n}\nlayer {\n\tname: \"conv2_1\"\n\ttype: \"Convolution\"\n\tbottom: \"pool1\"\n\ttop: \"conv2_1\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tconvolution_param {\n\t\tnum_output: 128\n\t\tpad: 1\n\t\tkernel_size: 3\n\t}\n}\nlayer {\n\tname: \"relu2_1\"\n\ttype: \"ReLU\"\n\tbottom: \"conv2_1\"\n\ttop: \"conv2_1\"\n}\nlayer {\n\tname: \"conv2_2\"\n\ttype: \"Convolution\"\n\tbottom: \"conv2_1\"\n\ttop: \"conv2_2\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tconvolution_param {\n\t\tnum_output: 128\n\t\tpad: 1\n\t\tkernel_size: 3\n\t}\n}\nlayer {\n\tname: \"relu2_2\"\n\ttype: \"ReLU\"\n\tbottom: \"conv2_2\"\n\ttop: \"conv2_2\"\n}\nlayer {\n\tname: \"pool2\"\t\n\ttype: \"Pooling\"\n\tbottom: \"conv2_2\"\n\ttop: \"pool2\"\n\tpooling_param {\n\t\tpool: MAX\n\t\tkernel_size: 2\n\t\tstride: 2\n\t}\n}\nlayer {\n\tname: \"conv3_1\"\n\ttype: \"Convolution\"\n\tbottom: \"pool2\"\n\ttop: \"conv3_1\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tconvolution_param {\n\t\tnum_output: 256\n\t\tpad: 1\n\t\tkernel_size: 3\n\t}\n}\nlayer {\n\tname: \"relu3_1\"\n\ttype: \"ReLU\"\n\tbottom: \"conv3_1\"\n\ttop: \"conv3_1\"\n}\nlayer {\n\tname: \"conv3_2\"\n\ttype: \"Convolution\"\n\tbottom: \"conv3_1\"\n\ttop: \"conv3_2\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tconvolution_param {\n\t\tnum_output: 256\n\t\tpad: 1\n\t\tkernel_size: 3\n\t}\n}\nlayer {\n\tname: \"relu3_2\"\n\ttype: \"ReLU\"\n\tbottom: \"conv3_2\"\n\ttop: \"conv3_2\"\n}\nlayer {\n\tname: \"conv3_3\"\n\ttype: \"Convolution\"\n\tbottom: \"conv3_2\"\n\ttop: \"conv3_3\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tconvolution_param {\n\t\tnum_output: 256\n\t\tpad: 1\n\t\tkernel_size: 3\n\t}\n}\nlayer {\n\tname: \"relu3_3\"\n\ttype: \"ReLU\"\n\tbottom: \"conv3_3\"\n\ttop: \"conv3_3\"\n}\nlayer {\n\tname: \"pool3\"\n\ttype: \"Pooling\"\n\tbottom: \"conv3_3\"\n\ttop: \"pool3\"\n\tpooling_param {\n\t\tpool: MAX\n\t\tkernel_size: 2\n\t\tstride: 2\n\t}\n}\nlayer {\n\tname: \"conv4_1\"\n\ttype: \"Convolution\"\n\tbottom: \"pool3\"\n\ttop: \"conv4_1\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tconvolution_param {\n\t\tnum_output: 512\n\t\tpad: 1\n\t\tkernel_size: 3\n\t}\n}\nlayer {\n\tname: \"relu4_1\"\n\ttype: \"ReLU\"\n\tbottom: \"conv4_1\"\n\ttop: \"conv4_1\"\n}\nlayer {\n\tname: \"conv4_2\"\n\ttype: \"Convolution\"\n\tbottom: \"conv4_1\"\n\ttop: \"conv4_2\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tconvolution_param {\n\t\tnum_output: 512\n\t\tpad: 1\n\t\tkernel_size: 3\n\t}\n}\nlayer {\n\tname: \"relu4_2\"\n\ttype: \"ReLU\"\n\tbottom: \"conv4_2\"\n\ttop: \"conv4_2\"\n}\nlayer {\n\tname: \"conv4_3\"\n\ttype: \"Convolution\"\n\tbottom: \"conv4_2\"\n\ttop: \"conv4_3\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tconvolution_param {\n\t\tnum_output: 512\n\t\tpad: 1\n\t\tkernel_size: 3\n\t}\n}\nlayer {\n\tname: \"relu4_3\"\n\ttype: \"ReLU\"\n\tbottom: \"conv4_3\"\n\ttop: \"conv4_3\"\n}\nlayer {\n\tname: \"pool4\"\n\ttype: \"Pooling\"\n\tbottom: \"conv4_3\"\n\ttop: \"pool4\"\n\tpooling_param {\n\t\tpool: MAX\n\t\tkernel_size: 2\n\t\tstride: 2\n\t}\n}\nlayer {\n\tname: \"conv5_1\"\n\ttype: \"Convolution\"\n\tbottom: \"pool4\"\n\ttop: \"conv5_1\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tconvolution_param {\n\t\tnum_output: 512\n\t\tpad: 1\n\t\tkernel_size: 3\n\t}\n}\nlayer {\n\tname: \"relu5_1\"\n\ttype: \"ReLU\"\n\tbottom: \"conv5_1\"\n\ttop: \"conv5_1\"\n}\nlayer {\n\tname: \"conv5_2\"\n\ttype: \"Convolution\"\n\tbottom: \"conv5_1\"\n\ttop: \"conv5_2\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tconvolution_param {\n\t\tnum_output: 512\n\t\tpad: 1\n\t\tkernel_size: 3\n\t}\n}\nlayer {\n\tname: \"relu5_2\"\n\ttype: \"ReLU\"\n\tbottom: \"conv5_2\"\n\ttop: \"conv5_2\"\n}\nlayer {\n\tname: \"conv5_3\"\n\ttype: \"Convolution\"\n\tbottom: \"conv5_2\"\n\ttop: \"conv5_3\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tconvolution_param {\n\t\tnum_output: 512\n\t\tpad: 1\n\t\tkernel_size: 3\n\t}\n}\nlayer {\n\tname: \"relu5_3\"\n\ttype: \"ReLU\"\n\tbottom: \"conv5_3\"\n\ttop: \"conv5_3\"\n}\nlayer {\n\tname: \"pool5\"\n\ttype: \"Pooling\"\n\tbottom: \"conv5_3\"\n\ttop: \"pool5\"\n\tpooling_param {\n\t\tpool: MAX\n\t\tkernel_size: 2\n\t\tstride: 2\n\t}\n}\nlayer {\n\tname: \"fc6\"\n\ttype: \"InnerProduct\"\n\tbottom: \"pool5\"\n\ttop: \"fc6\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 4096\n\t}\n}\nlayer {\n\tname: \"relu6\"\n\ttype: \"ReLU\"\n\tbottom: \"fc6\"\n\ttop: \"fc6\"\n}\nlayer {\n\tname: \"fc7\"\n\ttype: \"InnerProduct\"\n\tbottom: \"fc6\"\n\ttop: \"fc7\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 4096\n\t}\n}\nlayer {\n\tname: \"relu7\"\n\ttype: \"ReLU\"\n\tbottom: \"fc7\"\n\ttop: \"fc7\"\n}\nlayer {\n\tname: \"fc8\"\n\ttype: \"InnerProduct\"\n\tbottom: \"fc7\"\n\ttop: \"fc8\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 256\n\t\tweight_filler {\n\t\t\ttype: \"xavier\"\n\t\t}\n\t\tbias_filler {\n\t\t\ttype: \"constant\"\n\t\t\tvalue: 0\n\t\t}\n\t}\n}\nlayer {\n\tname: \"relu8\"\n\ttype: \"ReLU\"\n\tbottom: \"fc8\"\n\ttop: \"fc8\"\n}\n# Spatial Cfg Subnet\nlayer {\n\tname: \"conv1_p\"\n\ttype: \"Convolution\"\n\tbottom: \"posdata\"\n\ttop: \"conv1_p\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tconvolution_param {\n\t\tnum_output: 96\n\t\tkernel_size: 5\n\t\tpad: 2\n\t\tstride: 2\n\t}\n}\nlayer {\n\tname: \"relu1_p\"\n\ttype: \"ReLU\"\n\tbottom: \"conv1_p\"\n\ttop: \"conv1_p\"\n}\nlayer {\n\tname: \"conv2_p\"\n\ttype: \"Convolution\"\n\tbottom: \"conv1_p\"\n\ttop: \"conv2_p\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tconvolution_param {\n\t\tnum_output: 128\n\t\tkernel_size: 5\n\t\tpad: 2\n\t\tstride: 2\n\t}\n}\nlayer {\n\tname: \"conv3_p\"\n\ttype: \"Convolution\"\n\tbottom: \"conv2_p\"\n\ttop: \"conv3_p\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tconvolution_param {\n\t\tnum_output: 64\n\t\tkernel_size: 8\n\t}\n}\nlayer {\n\tname: \"relu3_p\"\n\ttype: \"ReLU\"\n\tbottom: \"conv3_p\"\n\ttop: \"conv3_p\"\n}\n# Combine features from subnets\nlayer {\n\tname: \"concat1_c\"\n\ttype: \"Python\"\n\tbottom: \"fc8\"\n\tbottom: \"conv3_p\"\n\ttop: \"concat1_c\"\n\tpython_param {\n\t\tmodule: \"customize_layers.concat_layer\"\n\t\tlayer: \"Layer\"\n\t}\n}\nlayer {\n\tname: \"fc2_c\"\n\ttype: \"InnerProduct\"\n\tbottom: \"concat1_c\"\n\ttop: \"fc2_c\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 128\n\t}\n}\nlayer {\n\tname: \"relu2_c\"\n\ttype: \"ReLU\"\n\tbottom: \"fc2_c\"\n\ttop: \"fc2_c\"\n}\nlayer {\n\tname: \"PhiR_0\"\n\ttype: \"InnerProduct\"\n\tbottom: \"fc2_c\"\n\ttop: \"qr0\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t\tweight_filler {\n\t\t\ttype: \"xavier\"\n\t\t}\n\t\tbias_filler {\n\t\t\ttype: \"constant\"\n\t\t\tvalue: 0\n\t\t}\n\t}\n}\n#DR-Net\nlayer {\n\tname: \"PhiA_1\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qa\"\n\ttop: \"qar1\"\n\tparam {\n\t\tname: \"qar_w\"\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tname: \"qar_b\"\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t\tweight_filler {\n\t\t\ttype: \"xavier\"\n\t\t}\n\t\tbias_filler {\n\t\t\ttype: \"constant\"\n\t\t\tvalue: 0\n\t\t}\n\t}\n}\nlayer {\n\tname: \"PhiB_1\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qb\"\n\ttop: \"qbr1\"\n\tparam {\n\t\tname: \"qbr_w\"\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tname: \"qbr_b\"\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t\tweight_filler {\n\t\t\ttype: \"xavier\"\n\t\t}\n\t\tbias_filler {\n\t\t\ttype: \"constant\"\n\t\t\tvalue: 0\n\t\t}\n\t}\n}\nlayer {\n\tname: \"PhiR_1\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qr0\"\n\ttop: \"q1r\"\n\tparam {\n\t\tname: \"qr_w\"\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\t\n\t\tname: \"qr_b\"\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t\tweight_filler {\n\t\t\ttype: \"xavier\"\n\t\t}\n\t\tbias_filler {\n\t\t\ttype: \"constant\"\n\t\t\tvalue: 0\n\t\t}\n\t}\n}\nlayer {\n\tname: \"QSum_1\"\n\ttype: \"Eltwise\"\n\tbottom: \"qar1\"\n\tbottom: \"qbr1\"\n\tbottom: \"q1r\"\n\ttop: \"qr1\"\n\teltwise_param { operation: SUM }\n}\nlayer {\n\tname: \"PhiA_2\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qa\"\n\ttop: \"qar2\"\n\tparam {\n\t\tname: \"qar_w\"\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tname: \"qar_b\"\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"PhiB_2\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qb\"\n\ttop: \"qbr2\"\n\tparam {\n\t\tname: \"qbr_w\"\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tname: \"qbr_b\"\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"PhiR_2\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qr1\"\n\ttop: \"q2r\"\n\tparam {\n\t\tname: \"qr_w\"\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tname: \"qr_b\"\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"QSum_2\"\n\ttype: \"Eltwise\"\n\tbottom: \"qar2\"\n\tbottom: \"qbr2\"\n\tbottom: \"q2r\"\n\ttop: \"qr2\"\n\teltwise_param { operation: SUM }\n}\nlayer {\n\tname: \"PhiA_3\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qa\"\n\ttop: \"qar3\"\n\tparam {\n\t\tname: \"qar_w\"\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tname: \"qar_b\"\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"PhiB_3\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qb\"\n\ttop: \"qbr3\"\n\tparam {\n\t\tname: \"qbr_w\"\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tname: \"qbr_b\"\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"PhiR_3\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qr2\"\n\ttop: \"q3r\"\n\tparam {\n\t\tname: \"qr_w\"\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tname: \"qr_b\"\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"QSum_3\"\n\ttype: \"Eltwise\"\n\tbottom: \"qar3\"\n\tbottom: \"qbr3\"\n\tbottom: \"q3r\"\n\ttop: \"qr3\"\n\teltwise_param { operation: SUM }\n}\nlayer {\n\tname: \"PhiA_4\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qa\"\n\ttop: \"qar4\"\n\tparam {\n\t\tname: \"qar_w\"\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tname: \"qar_b\"\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"PhiB_4\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qb\"\n\ttop: \"qbr4\"\n\tparam {\n\t\tname: \"qbr_w\"\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tname: \"qbr_b\"\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"PhiR_4\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qr3\"\n\ttop: \"q4r\"\n\tparam {\n\t\tname: \"qr_w\"\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tname: \"qr_b\"\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"QSum_4\"\n\ttype: \"Eltwise\"\n\tbottom: \"qar4\"\n\tbottom: \"qbr4\"\n\tbottom: \"q4r\"\n\ttop: \"qr4\"\n\teltwise_param { operation: SUM }\n}\nlayer {\n\tname: \"PhiA_5\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qa\"\n\ttop: \"qar5\"\n\tparam {\n\t\tname: \"qar_w\"\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tname: \"qar_b\"\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"PhiB_5\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qb\"\n\ttop: \"qbr5\"\n\tparam {\n\t\tname: \"qbr_w\"\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tname: \"qbr_b\"\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"PhiR_5\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qr4\"\n\ttop: \"q5r\"\n\tparam {\n\t\tname: \"qr_w\"\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tname: \"qr_b\"\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"QSum_5\"\n\ttype: \"Eltwise\"\n\tbottom: \"qar5\"\n\tbottom: \"qbr5\"\n\tbottom: \"q5r\"\n\ttop: \"qr5\"\n\teltwise_param { operation: SUM }\n}\nlayer {\n\tname: \"PhiA_6\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qa\"\n\ttop: \"qar6\"\n\tparam {\n\t\tname: \"qar_w\"\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tname: \"qar_b\"\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"PhiB_6\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qb\"\n\ttop: \"qbr6\"\n\tparam {\n\t\tname: \"qbr_w\"\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tname: \"qbr_b\"\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"PhiR_6\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qr5\"\n\ttop: \"q6r\"\n\tparam {\n\t\tname: \"qr_w\"\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tname: \"qr_b\"\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"QSum_6\"\n\ttype: \"Eltwise\"\n\tbottom: \"qar6\"\n\tbottom: \"qbr6\"\n\tbottom: \"q6r\"\n\ttop: \"qr6\"\n\teltwise_param { operation: SUM }\n}\nlayer {\n\tname: \"PhiA_7\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qa\"\n\ttop: \"qar7\"\n\tparam {\n\t\tname: \"qar_w\"\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tname: \"qar_b\"\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"PhiB_7\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qb\"\n\ttop: \"qbr7\"\n\tparam {\n\t\tname: \"qbr_w\"\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tname: \"qbr_b\"\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"PhiR_7\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qr6\"\n\ttop: \"q7r\"\n\tparam {\n\t\tname: \"qr_w\"\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tname: \"qr_b\"\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"QSum_7\"\n\ttype: \"Eltwise\"\n\tbottom: \"qar7\"\n\tbottom: \"qbr7\"\n\tbottom: \"q7r\"\n\ttop: \"qr7\"\n\teltwise_param { operation: SUM }\n}\nlayer {\n\tname: \"PhiA_8\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qa\"\n\ttop: \"qar8\"\n\tparam {\n\t\tname: \"qar_w\"\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tname: \"qar_b\"\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"PhiB_8\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qb\"\n\ttop: \"qbr8\"\n\tparam {\n\t\tname: \"qbr_w\"\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tname: \"qbr_b\"\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"PhiR_8\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qr7\"\n\ttop: \"q8r\"\n\tparam {\n\t\tname: \"qr_w\"\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tname: \"qr_b\"\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"QSum_8\"\n\ttype: \"Eltwise\"\n\tbottom: \"qar8\"\n\tbottom: \"qbr8\"\n\tbottom: \"q8r\"\n\ttop: \"qr8\"\n\teltwise_param { operation: SUM }\n}\n\nlayer {\n\tname: \"loss3\"\n\ttype: \"SoftmaxWithLoss\"\n\ttop: \"loss3\"\n\tbottom: \"qr8\"\n\tbottom: \"labels\"\n\tinclude {\n\t\tphase: TRAIN\n\t}\n\tloss_weight: 1\n}\nlayer {\n\tname: \"softmax\"\n\ttype: \"Softmax\"\n\tbottom: \"qr8\"\n\ttop: \"pred\"\n\tinclude {\n\t\tphase: TEST\n\t}\n}\nlayer {\n\tname: \"accuracy\"\n\ttype: \"Accuracy\"\n\tbottom: \"pred\"\n\tbottom: \"labels\"\n\ttop: \"top-1\"\n\tinclude {\n\t\tphase: TEST\n\t}\n\taccuracy_param {\n\t\ttop_k: 1\n\t}\n}\nlayer {\n\tname: \"accuracy_top5\"\n\ttype: \"Accuracy\"\n\tbottom: \"pred\"\n\tbottom: \"labels\"\n\ttop: \"top-5\"\n\tinclude {\n\t\tphase: TEST\n\t}\n\taccuracy_param {\n\t\ttop_k: 5\n\t}\n}\n"
  },
  {
    "path": "prototxts/drnet_8units_relu_shareweight.prototxt",
    "content": "layer {\n        name: \"data\"\n        type: \"Python\"\n        top: \"qa\"\n        top: \"qb\"\n        top: \"im\"\n        top: \"posdata\"\n        top: \"labels\"\n        include {\n                phase: TRAIN     \n        }         \n        python_param {                 \n                module: 'rel_data_layer.layer'              \n                layer: 'RelDataLayer'                 \n                param_str: '{\"dataset\": \"reltrain.json\", \"batch_size\": 25, \"nclass\": 100}'   \n        }         \n}                    \nlayer {           \n        name: \"data\"                   \n        type: \"Python\"                 \n        top: \"qa\" \n        top: \"qb\" \n        top: \"im\" \n        top: \"posdata\"                 \n        top: \"labels\"                  \n        include { \n                phase: TEST            \n        }         \n        python_param {                 \n                module: 'rel_data_layer.layer'              \n                layer: 'RelDataLayer'\n                param_str: '{\"dataset\": \"reltest.json\", \"batch_size\": 25, \"nclass\": 100}'\n        }         \n}\n# Appearance Subnet\nlayer {\n\tname: \"conv1_1\"\n\ttype: \"Convolution\"\n\tbottom: \"im\"\n\ttop: \"conv1_1\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t} \n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tconvolution_param {\n\t\tnum_output: 64\n\t\tpad: 1\n\t\tkernel_size: 3\n\t}\n}\nlayer {\n\tname: \"relu1_1\"\n\ttype: \"ReLU\"\n\tbottom: \"conv1_1\"\n\ttop: \"conv1_1\"\n}\nlayer {\n\tname: \"conv1_2\"\n\ttype: \"Convolution\"\n\tbottom: \"conv1_1\"\n\ttop: \"conv1_2\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tconvolution_param {\n\t\tnum_output: 64\n\t\tpad: 1\n\t\tkernel_size: 3\n\t}\n}\nlayer {\n\tname: \"relu1_2\"\n\ttype: \"ReLU\"\n\tbottom: \"conv1_2\"\n\ttop: \"conv1_2\"\n}\nlayer {\n\tname: \"pool1\"\n\ttype: \"Pooling\"\n\tbottom: \"conv1_2\"\n\ttop: \"pool1\"\n\tpooling_param {\n\t\tpool: MAX\n\t\tkernel_size: 2\n\t\tstride: 2\n\t}\n}\nlayer {\n\tname: \"conv2_1\"\n\ttype: \"Convolution\"\n\tbottom: \"pool1\"\n\ttop: \"conv2_1\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tconvolution_param {\n\t\tnum_output: 128\n\t\tpad: 1\n\t\tkernel_size: 3\n\t}\n}\nlayer {\n\tname: \"relu2_1\"\n\ttype: \"ReLU\"\n\tbottom: \"conv2_1\"\n\ttop: \"conv2_1\"\n}\nlayer {\n\tname: \"conv2_2\"\n\ttype: \"Convolution\"\n\tbottom: \"conv2_1\"\n\ttop: \"conv2_2\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tconvolution_param {\n\t\tnum_output: 128\n\t\tpad: 1\n\t\tkernel_size: 3\n\t}\n}\nlayer {\n\tname: \"relu2_2\"\n\ttype: \"ReLU\"\n\tbottom: \"conv2_2\"\n\ttop: \"conv2_2\"\n}\nlayer {\n\tname: \"pool2\"\t\n\ttype: \"Pooling\"\n\tbottom: \"conv2_2\"\n\ttop: \"pool2\"\n\tpooling_param {\n\t\tpool: MAX\n\t\tkernel_size: 2\n\t\tstride: 2\n\t}\n}\nlayer {\n\tname: \"conv3_1\"\n\ttype: \"Convolution\"\n\tbottom: \"pool2\"\n\ttop: \"conv3_1\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tconvolution_param {\n\t\tnum_output: 256\n\t\tpad: 1\n\t\tkernel_size: 3\n\t}\n}\nlayer {\n\tname: \"relu3_1\"\n\ttype: \"ReLU\"\n\tbottom: \"conv3_1\"\n\ttop: \"conv3_1\"\n}\nlayer {\n\tname: \"conv3_2\"\n\ttype: \"Convolution\"\n\tbottom: \"conv3_1\"\n\ttop: \"conv3_2\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tconvolution_param {\n\t\tnum_output: 256\n\t\tpad: 1\n\t\tkernel_size: 3\n\t}\n}\nlayer {\n\tname: \"relu3_2\"\n\ttype: \"ReLU\"\n\tbottom: \"conv3_2\"\n\ttop: \"conv3_2\"\n}\nlayer {\n\tname: \"conv3_3\"\n\ttype: \"Convolution\"\n\tbottom: \"conv3_2\"\n\ttop: \"conv3_3\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tconvolution_param {\n\t\tnum_output: 256\n\t\tpad: 1\n\t\tkernel_size: 3\n\t}\n}\nlayer {\n\tname: \"relu3_3\"\n\ttype: \"ReLU\"\n\tbottom: \"conv3_3\"\n\ttop: \"conv3_3\"\n}\nlayer {\n\tname: \"pool3\"\n\ttype: \"Pooling\"\n\tbottom: \"conv3_3\"\n\ttop: \"pool3\"\n\tpooling_param {\n\t\tpool: MAX\n\t\tkernel_size: 2\n\t\tstride: 2\n\t}\n}\nlayer {\n\tname: \"conv4_1\"\n\ttype: \"Convolution\"\n\tbottom: \"pool3\"\n\ttop: \"conv4_1\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tconvolution_param {\n\t\tnum_output: 512\n\t\tpad: 1\n\t\tkernel_size: 3\n\t}\n}\nlayer {\n\tname: \"relu4_1\"\n\ttype: \"ReLU\"\n\tbottom: \"conv4_1\"\n\ttop: \"conv4_1\"\n}\nlayer {\n\tname: \"conv4_2\"\n\ttype: \"Convolution\"\n\tbottom: \"conv4_1\"\n\ttop: \"conv4_2\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tconvolution_param {\n\t\tnum_output: 512\n\t\tpad: 1\n\t\tkernel_size: 3\n\t}\n}\nlayer {\n\tname: \"relu4_2\"\n\ttype: \"ReLU\"\n\tbottom: \"conv4_2\"\n\ttop: \"conv4_2\"\n}\nlayer {\n\tname: \"conv4_3\"\n\ttype: \"Convolution\"\n\tbottom: \"conv4_2\"\n\ttop: \"conv4_3\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tconvolution_param {\n\t\tnum_output: 512\n\t\tpad: 1\n\t\tkernel_size: 3\n\t}\n}\nlayer {\n\tname: \"relu4_3\"\n\ttype: \"ReLU\"\n\tbottom: \"conv4_3\"\n\ttop: \"conv4_3\"\n}\nlayer {\n\tname: \"pool4\"\n\ttype: \"Pooling\"\n\tbottom: \"conv4_3\"\n\ttop: \"pool4\"\n\tpooling_param {\n\t\tpool: MAX\n\t\tkernel_size: 2\n\t\tstride: 2\n\t}\n}\nlayer {\n\tname: \"conv5_1\"\n\ttype: \"Convolution\"\n\tbottom: \"pool4\"\n\ttop: \"conv5_1\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tconvolution_param {\n\t\tnum_output: 512\n\t\tpad: 1\n\t\tkernel_size: 3\n\t}\n}\nlayer {\n\tname: \"relu5_1\"\n\ttype: \"ReLU\"\n\tbottom: \"conv5_1\"\n\ttop: \"conv5_1\"\n}\nlayer {\n\tname: \"conv5_2\"\n\ttype: \"Convolution\"\n\tbottom: \"conv5_1\"\n\ttop: \"conv5_2\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tconvolution_param {\n\t\tnum_output: 512\n\t\tpad: 1\n\t\tkernel_size: 3\n\t}\n}\nlayer {\n\tname: \"relu5_2\"\n\ttype: \"ReLU\"\n\tbottom: \"conv5_2\"\n\ttop: \"conv5_2\"\n}\nlayer {\n\tname: \"conv5_3\"\n\ttype: \"Convolution\"\n\tbottom: \"conv5_2\"\n\ttop: \"conv5_3\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tconvolution_param {\n\t\tnum_output: 512\n\t\tpad: 1\n\t\tkernel_size: 3\n\t}\n}\nlayer {\n\tname: \"relu5_3\"\n\ttype: \"ReLU\"\n\tbottom: \"conv5_3\"\n\ttop: \"conv5_3\"\n}\nlayer {\n\tname: \"pool5\"\n\ttype: \"Pooling\"\n\tbottom: \"conv5_3\"\n\ttop: \"pool5\"\n\tpooling_param {\n\t\tpool: MAX\n\t\tkernel_size: 2\n\t\tstride: 2\n\t}\n}\nlayer {\n\tname: \"fc6\"\n\ttype: \"InnerProduct\"\n\tbottom: \"pool5\"\n\ttop: \"fc6\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 4096\n\t}\n}\nlayer {\n\tname: \"relu6\"\n\ttype: \"ReLU\"\n\tbottom: \"fc6\"\n\ttop: \"fc6\"\n}\nlayer {\n\tname: \"fc7\"\n\ttype: \"InnerProduct\"\n\tbottom: \"fc6\"\n\ttop: \"fc7\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 4096\n\t}\n}\nlayer {\n\tname: \"relu7\"\n\ttype: \"ReLU\"\n\tbottom: \"fc7\"\n\ttop: \"fc7\"\n}\nlayer {\n\tname: \"fc8\"\n\ttype: \"InnerProduct\"\n\tbottom: \"fc7\"\n\ttop: \"fc8\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 256\n\t\tweight_filler {\n\t\t\ttype: \"xavier\"\n\t\t}\n\t\tbias_filler {\n\t\t\ttype: \"constant\"\n\t\t\tvalue: 0\n\t\t}\n\t}\n}\nlayer {\n\tname: \"relu8\"\n\ttype: \"ReLU\"\n\tbottom: \"fc8\"\n\ttop: \"fc8\"\n}\n\n# Spatial Cfg Subnet\nlayer {\n\tname: \"conv1_p\"\n\ttype: \"Convolution\"\n\tbottom: \"posdata\"\n\ttop: \"conv1_p\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tconvolution_param {\n\t\tnum_output: 96\n\t\tkernel_size: 5\n\t\tpad: 2\n\t\tstride: 2\n\t}\n}\nlayer {\n\tname: \"relu1_p\"\n\ttype: \"ReLU\"\n\tbottom: \"conv1_p\"\n\ttop: \"conv1_p\"\n}\nlayer {\n\tname: \"conv2_p\"\n\ttype: \"Convolution\"\n\tbottom: \"conv1_p\"\n\ttop: \"conv2_p\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tconvolution_param {\n\t\tnum_output: 128\n\t\tkernel_size: 5\n\t\tpad: 2\n\t\tstride: 2\n\t}\n}\nlayer {\n\tname: \"conv3_p\"\n\ttype: \"Convolution\"\n\tbottom: \"conv2_p\"\n\ttop: \"conv3_p\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tconvolution_param {\n\t\tnum_output: 64\n\t\tkernel_size: 8\n\t}\n}\nlayer {\n\tname: \"relu3_p\"\n\ttype: \"ReLU\"\n\tbottom: \"conv3_p\"\n\ttop: \"conv3_p\"\n}\n# Combine features from subnets\nlayer {\n\tname: \"concat1_c\"\n\ttype: \"Python\"\n\tbottom: \"fc8\"\n\tbottom: \"conv3_p\"\n\ttop: \"concat1_c\"\n\tpython_param {\n\t\tmodule: \"customize_layers.concat_layer\"\n\t\tlayer: \"Layer\"\n\t}\n}\nlayer {\n\tname: \"fc2_c\"\n\ttype: \"InnerProduct\"\n\tbottom: \"concat1_c\"\n\ttop: \"fc2_c\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 128\n\t}\n}\nlayer {\n\tname: \"relu2_c\"\n\ttype: \"ReLU\"\n\tbottom: \"fc2_c\"\n\ttop: \"fc2_c\"\n}\nlayer {\n\tname: \"PhiR_0\"\n\ttype: \"InnerProduct\"\n\tbottom: \"fc2_c\"\n\ttop: \"q0r\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t\tweight_filler {\n\t\t\ttype: \"xavier\"\n\t\t}\n\t\tbias_filler {\n\t\t\ttype: \"constant\"\n\t\t\tvalue: 0\n\t\t}\n\t}\n}\nlayer {\n\tname: \"relu_0\"\n\ttype: \"ReLU\"\n\tbottom: \"q0r\"\n\ttop: \"qr0\"\n}\n#DR-Net\nlayer {\n\tname: \"PhiA_1\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qa\"\n\ttop: \"qar1\"\n\tparam {\n\t\tname: \"qar_w\"\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tname: \"qar_b\"\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t\tweight_filler {\n\t\t\ttype: \"xavier\"\n\t\t}\n\t\tbias_filler {\n\t\t\ttype: \"constant\"\n\t\t\tvalue: 0\n\t\t}\n\t}\n}\nlayer {\n\tname: \"PhiB_1\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qb\"\n\ttop: \"qbr1\"\n\tparam {\n\t\tname: \"qbr_w\"\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tname: \"qbr_b\"\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t\tweight_filler {\n\t\t\ttype: \"xavier\"\n\t\t}\n\t\tbias_filler {\n\t\t\ttype: \"constant\"\n\t\t\tvalue: 0\n\t\t}\n\t}\n}\nlayer {\n\tname: \"PhiR_1\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qr0\"\n\ttop: \"q1r\"\n\tparam {\n\t\tname: \"qr_w\"\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\t\n\t\tname: \"qr_b\"\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t\tweight_filler {\n\t\t\ttype: \"xavier\"\n\t\t}\n\t\tbias_filler {\n\t\t\ttype: \"constant\"\n\t\t\tvalue: 0\n\t\t}\n\t}\n}\nlayer {\n\tname: \"QSum_1\"\n\ttype: \"Eltwise\"\n\tbottom: \"qar1\"\n\tbottom: \"qbr1\"\n\tbottom: \"q1r\"\n\ttop: \"qr1un\"\n\teltwise_param { operation: SUM }\n}\nlayer {\n\tname: \"relu_1\"\n\ttype: \"ReLU\"\n\tbottom: \"qr1un\"\n\ttop: \"qr1\"\n}\nlayer {\n\tname: \"PhiA_2\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qa\"\n\ttop: \"qar2\"\n\tparam {\n\t\tname: \"qar_w\"\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tname: \"qar_b\"\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"PhiB_2\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qb\"\n\ttop: \"qbr2\"\n\tparam {\n\t\tname: \"qbr_w\"\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tname: \"qbr_b\"\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"PhiR_2\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qr1\"\n\ttop: \"q2r\"\n\tparam {\n\t\tname: \"qr_w\"\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tname: \"qr_b\"\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"QSum_2\"\n\ttype: \"Eltwise\"\n\tbottom: \"qar2\"\n\tbottom: \"qbr2\"\n\tbottom: \"q2r\"\n\ttop: \"qr2un\"\n\teltwise_param { operation: SUM }\n}\nlayer {\n\tname: \"relu_2\"\n\ttype: \"ReLU\"\n\tbottom: \"qr2un\"\n\ttop: \"qr2\"\n}\nlayer {\n\tname: \"PhiA_3\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qa\"\n\ttop: \"qar3\"\n\tparam {\n\t\tname: \"qar_w\"\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tname: \"qar_b\"\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"PhiB_3\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qb\"\n\ttop: \"qbr3\"\n\tparam {\n\t\tname: \"qbr_w\"\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tname: \"qbr_b\"\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"PhiR_3\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qr2\"\n\ttop: \"q3r\"\n\tparam {\n\t\tname: \"qr_w\"\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tname: \"qr_b\"\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"QSum_3\"\n\ttype: \"Eltwise\"\n\tbottom: \"qar3\"\n\tbottom: \"qbr3\"\n\tbottom: \"q3r\"\n\ttop: \"qr3un\"\n\teltwise_param { operation: SUM }\n}\nlayer {\n\tname: \"relu_3\"\n\ttype: \"ReLU\"\n\tbottom: \"qr3un\"\n\ttop: \"qr3\"\n}\nlayer {\n\tname: \"PhiA_4\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qa\"\n\ttop: \"qar4\"\n\tparam {\n\t\tname: \"qar_w\"\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tname: \"qar_b\"\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"PhiB_4\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qb\"\n\ttop: \"qbr4\"\n\tparam {\n\t\tname: \"qbr_w\"\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tname: \"qbr_b\"\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"PhiR_4\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qr3\"\n\ttop: \"q4r\"\n\tparam {\n\t\tname: \"qr_w\"\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tname: \"qr_b\"\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"QSum_4\"\n\ttype: \"Eltwise\"\n\tbottom: \"qar4\"\n\tbottom: \"qbr4\"\n\tbottom: \"q4r\"\n\ttop: \"qr4un\"\n\teltwise_param { operation: SUM }\n}\nlayer {\n\tname: \"relu_4\"\n\ttype: \"ReLU\"\n\tbottom: \"qr4un\"\n\ttop: \"qr4\"\n}\nlayer {\n\tname: \"PhiA_5\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qa\"\n\ttop: \"qar5\"\n\tparam {\n\t\tname: \"qar_w\"\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tname: \"qar_b\"\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"PhiB_5\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qb\"\n\ttop: \"qbr5\"\n\tparam {\n\t\tname: \"qbr_w\"\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tname: \"qbr_b\"\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"PhiR_5\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qr4\"\n\ttop: \"q5r\"\n\tparam {\n\t\tname: \"qr_w\"\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tname: \"qr_b\"\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"QSum_5\"\n\ttype: \"Eltwise\"\n\tbottom: \"qar5\"\n\tbottom: \"qbr5\"\n\tbottom: \"q5r\"\n\ttop: \"qr5un\"\n\teltwise_param { operation: SUM }\n}\nlayer {\n\tname: \"relu_5\"\n\ttype: \"ReLU\"\n\tbottom: \"qr5un\"\n\ttop: \"qr5\"\n}\n\nlayer {\n\tname: \"PhiA_6\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qa\"\n\ttop: \"qar6\"\n\tparam {\n\t\tname: \"qar_w\"\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tname: \"qar_b\"\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"PhiB_6\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qb\"\n\ttop: \"qbr6\"\n\tparam {\n\t\tname: \"qbr_w\"\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tname: \"qbr_b\"\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"PhiR_6\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qr5\"\n\ttop: \"q6r\"\n\tparam {\n\t\tname: \"qr_w\"\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tname: \"qr_b\"\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"QSum_6\"\n\ttype: \"Eltwise\"\n\tbottom: \"qar6\"\n\tbottom: \"qbr6\"\n\tbottom: \"q6r\"\n\ttop: \"qr6un\"\n\teltwise_param { operation: SUM }\n}\nlayer {\n\tname: \"relu_6\"\n\ttype: \"ReLU\"\n\tbottom: \"qr6un\"\n\ttop: \"qr6\"\n}\nlayer {\n\tname: \"PhiA_7\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qa\"\n\ttop: \"qar7\"\n\tparam {\n\t\tname: \"qar_w\"\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tname: \"qar_b\"\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"PhiB_7\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qb\"\n\ttop: \"qbr7\"\n\tparam {\n\t\tname: \"qbr_w\"\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tname: \"qbr_b\"\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"PhiR_7\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qr6\"\n\ttop: \"q7r\"\n\tparam {\n\t\tname: \"qr_w\"\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tname: \"qr_b\"\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"QSum_7\"\n\ttype: \"Eltwise\"\n\tbottom: \"qar7\"\n\tbottom: \"qbr7\"\n\tbottom: \"q7r\"\n\ttop: \"qr7un\"\n\teltwise_param { operation: SUM }\n}\nlayer {\n\tname: \"relu_7\"\n\ttype: \"ReLU\"\n\tbottom: \"qr7un\"\n\ttop: \"qr7\"\n}\nlayer {\n\tname: \"PhiA_8\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qa\"\n\ttop: \"qar8\"\n\tparam {\n\t\tname: \"qar_w\"\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tname: \"qar_b\"\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"PhiB_8\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qb\"\n\ttop: \"qbr8\"\n\tparam {\n\t\tname: \"qbr_w\"\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tname: \"qbr_b\"\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"PhiR_8\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qr7\"\n\ttop: \"q8r\"\n\tparam {\n\t\tname: \"qr_w\"\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tname: \"qr_b\"\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"QSum_8\"\n\ttype: \"Eltwise\"\n\tbottom: \"qar8\"\n\tbottom: \"qbr8\"\n\tbottom: \"q8r\"\n\ttop: \"qr8\"\n\teltwise_param { operation: SUM }\n}\nlayer {\n\tname: \"loss3\"\n\ttype: \"SoftmaxWithLoss\"\n\ttop: \"loss3\"\n\tbottom: \"qr8\"\n\tbottom: \"labels\"\n\tinclude {\n\t\tphase: TRAIN\n\t}\n\tloss_weight: 1\n}\nlayer {\n\tname: \"softmax\"\n\ttype: \"Softmax\"\n\tbottom: \"qr8\"\n\ttop: \"pred\"\n\tinclude {\n\t\tphase: TEST\n\t}\n}\nlayer {\n\tname: \"accuracy\"\n\ttype: \"Accuracy\"\n\tbottom: \"pred\"\n\tbottom: \"labels\"\n\ttop: \"top-1\"\n\tinclude {\n\t\tphase: TEST\n\t}\n\taccuracy_param {\n\t\ttop_k: 1\n\t}\n}\nlayer {\n\tname: \"accuracy_top5\"\n\ttype: \"Accuracy\"\n\tbottom: \"pred\"\n\tbottom: \"labels\"\n\ttop: \"top-5\"\n\tinclude {\n\t\tphase: TEST\n\t}\n\taccuracy_param {\n\t\ttop_k: 5\n\t}\n}\n"
  },
  {
    "path": "prototxts/drnet_8units_softmax.prototxt",
    "content": "layer {\n        name: \"data\"\n        type: \"Python\"\n        top: \"qa\"\n        top: \"qb\"\n        top: \"im\"\n        top: \"posdata\"\n        top: \"labels\"\n        include {\n                phase: TRAIN     \n        }         \n        python_param {                 \n                module: 'rel_data_layer.layer'              \n                layer: 'RelDataLayer'                 \n                param_str: '{\"dataset\": \"reltrain.json\", \"batch_size\": 25, \"nclass\": 100}'   \n        }         \n}                    \nlayer {           \n        name: \"data\"                   \n        type: \"Python\"                 \n        top: \"qa\" \n        top: \"qb\" \n        top: \"im\" \n        top: \"posdata\"                 \n        top: \"labels\"                  \n        include { \n                phase: TEST            \n        }         \n        python_param {                 \n                module: 'rel_data_layer.layer'              \n                layer: 'RelDataLayer'\n                param_str: '{\"dataset\": \"reltest.json\", \"batch_size\": 25, \"nclass\": 100}'\n        }         \n}\n# Appearance Subnet \nlayer {\n\tname: \"conv1_1\"\n\ttype: \"Convolution\"\n\tbottom: \"im\"\n\ttop: \"conv1_1\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t} \n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tconvolution_param {\n\t\tnum_output: 64\n\t\tpad: 1\n\t\tkernel_size: 3\n\t}\n}\nlayer {\n\tname: \"relu1_1\"\n\ttype: \"ReLU\"\n\tbottom: \"conv1_1\"\n\ttop: \"conv1_1\"\n}\nlayer {\n\tname: \"conv1_2\"\n\ttype: \"Convolution\"\n\tbottom: \"conv1_1\"\n\ttop: \"conv1_2\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tconvolution_param {\n\t\tnum_output: 64\n\t\tpad: 1\n\t\tkernel_size: 3\n\t}\n}\nlayer {\n\tname: \"relu1_2\"\n\ttype: \"ReLU\"\n\tbottom: \"conv1_2\"\n\ttop: \"conv1_2\"\n}\nlayer {\n\tname: \"pool1\"\n\ttype: \"Pooling\"\n\tbottom: \"conv1_2\"\n\ttop: \"pool1\"\n\tpooling_param {\n\t\tpool: MAX\n\t\tkernel_size: 2\n\t\tstride: 2\n\t}\n}\nlayer {\n\tname: \"conv2_1\"\n\ttype: \"Convolution\"\n\tbottom: \"pool1\"\n\ttop: \"conv2_1\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tconvolution_param {\n\t\tnum_output: 128\n\t\tpad: 1\n\t\tkernel_size: 3\n\t}\n}\nlayer {\n\tname: \"relu2_1\"\n\ttype: \"ReLU\"\n\tbottom: \"conv2_1\"\n\ttop: \"conv2_1\"\n}\nlayer {\n\tname: \"conv2_2\"\n\ttype: \"Convolution\"\n\tbottom: \"conv2_1\"\n\ttop: \"conv2_2\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tconvolution_param {\n\t\tnum_output: 128\n\t\tpad: 1\n\t\tkernel_size: 3\n\t}\n}\nlayer {\n\tname: \"relu2_2\"\n\ttype: \"ReLU\"\n\tbottom: \"conv2_2\"\n\ttop: \"conv2_2\"\n}\nlayer {\n\tname: \"pool2\"\t\n\ttype: \"Pooling\"\n\tbottom: \"conv2_2\"\n\ttop: \"pool2\"\n\tpooling_param {\n\t\tpool: MAX\n\t\tkernel_size: 2\n\t\tstride: 2\n\t}\n}\nlayer {\n\tname: \"conv3_1\"\n\ttype: \"Convolution\"\n\tbottom: \"pool2\"\n\ttop: \"conv3_1\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tconvolution_param {\n\t\tnum_output: 256\n\t\tpad: 1\n\t\tkernel_size: 3\n\t}\n}\nlayer {\n\tname: \"relu3_1\"\n\ttype: \"ReLU\"\n\tbottom: \"conv3_1\"\n\ttop: \"conv3_1\"\n}\nlayer {\n\tname: \"conv3_2\"\n\ttype: \"Convolution\"\n\tbottom: \"conv3_1\"\n\ttop: \"conv3_2\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tconvolution_param {\n\t\tnum_output: 256\n\t\tpad: 1\n\t\tkernel_size: 3\n\t}\n}\nlayer {\n\tname: \"relu3_2\"\n\ttype: \"ReLU\"\n\tbottom: \"conv3_2\"\n\ttop: \"conv3_2\"\n}\nlayer {\n\tname: \"conv3_3\"\n\ttype: \"Convolution\"\n\tbottom: \"conv3_2\"\n\ttop: \"conv3_3\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tconvolution_param {\n\t\tnum_output: 256\n\t\tpad: 1\n\t\tkernel_size: 3\n\t}\n}\nlayer {\n\tname: \"relu3_3\"\n\ttype: \"ReLU\"\n\tbottom: \"conv3_3\"\n\ttop: \"conv3_3\"\n}\nlayer {\n\tname: \"pool3\"\n\ttype: \"Pooling\"\n\tbottom: \"conv3_3\"\n\ttop: \"pool3\"\n\tpooling_param {\n\t\tpool: MAX\n\t\tkernel_size: 2\n\t\tstride: 2\n\t}\n}\nlayer {\n\tname: \"conv4_1\"\n\ttype: \"Convolution\"\n\tbottom: \"pool3\"\n\ttop: \"conv4_1\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tconvolution_param {\n\t\tnum_output: 512\n\t\tpad: 1\n\t\tkernel_size: 3\n\t}\n}\nlayer {\n\tname: \"relu4_1\"\n\ttype: \"ReLU\"\n\tbottom: \"conv4_1\"\n\ttop: \"conv4_1\"\n}\nlayer {\n\tname: \"conv4_2\"\n\ttype: \"Convolution\"\n\tbottom: \"conv4_1\"\n\ttop: \"conv4_2\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tconvolution_param {\n\t\tnum_output: 512\n\t\tpad: 1\n\t\tkernel_size: 3\n\t}\n}\nlayer {\n\tname: \"relu4_2\"\n\ttype: \"ReLU\"\n\tbottom: \"conv4_2\"\n\ttop: \"conv4_2\"\n}\nlayer {\n\tname: \"conv4_3\"\n\ttype: \"Convolution\"\n\tbottom: \"conv4_2\"\n\ttop: \"conv4_3\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tconvolution_param {\n\t\tnum_output: 512\n\t\tpad: 1\n\t\tkernel_size: 3\n\t}\n}\nlayer {\n\tname: \"relu4_3\"\n\ttype: \"ReLU\"\n\tbottom: \"conv4_3\"\n\ttop: \"conv4_3\"\n}\nlayer {\n\tname: \"pool4\"\n\ttype: \"Pooling\"\n\tbottom: \"conv4_3\"\n\ttop: \"pool4\"\n\tpooling_param {\n\t\tpool: MAX\n\t\tkernel_size: 2\n\t\tstride: 2\n\t}\n}\nlayer {\n\tname: \"conv5_1\"\n\ttype: \"Convolution\"\n\tbottom: \"pool4\"\n\ttop: \"conv5_1\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tconvolution_param {\n\t\tnum_output: 512\n\t\tpad: 1\n\t\tkernel_size: 3\n\t}\n}\nlayer {\n\tname: \"relu5_1\"\n\ttype: \"ReLU\"\n\tbottom: \"conv5_1\"\n\ttop: \"conv5_1\"\n}\nlayer {\n\tname: \"conv5_2\"\n\ttype: \"Convolution\"\n\tbottom: \"conv5_1\"\n\ttop: \"conv5_2\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tconvolution_param {\n\t\tnum_output: 512\n\t\tpad: 1\n\t\tkernel_size: 3\n\t}\n}\nlayer {\n\tname: \"relu5_2\"\n\ttype: \"ReLU\"\n\tbottom: \"conv5_2\"\n\ttop: \"conv5_2\"\n}\nlayer {\n\tname: \"conv5_3\"\n\ttype: \"Convolution\"\n\tbottom: \"conv5_2\"\n\ttop: \"conv5_3\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tconvolution_param {\n\t\tnum_output: 512\n\t\tpad: 1\n\t\tkernel_size: 3\n\t}\n}\nlayer {\n\tname: \"relu5_3\"\n\ttype: \"ReLU\"\n\tbottom: \"conv5_3\"\n\ttop: \"conv5_3\"\n}\nlayer {\n\tname: \"pool5\"\n\ttype: \"Pooling\"\n\tbottom: \"conv5_3\"\n\ttop: \"pool5\"\n\tpooling_param {\n\t\tpool: MAX\n\t\tkernel_size: 2\n\t\tstride: 2\n\t}\n}\nlayer {\n\tname: \"fc6\"\n\ttype: \"InnerProduct\"\n\tbottom: \"pool5\"\n\ttop: \"fc6\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 4096\n\t}\n}\nlayer {\n\tname: \"relu6\"\n\ttype: \"ReLU\"\n\tbottom: \"fc6\"\n\ttop: \"fc6\"\n}\nlayer {\n\tname: \"fc7\"\n\ttype: \"InnerProduct\"\n\tbottom: \"fc6\"\n\ttop: \"fc7\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 4096\n\t}\n}\nlayer {\n\tname: \"relu7\"\n\ttype: \"ReLU\"\n\tbottom: \"fc7\"\n\ttop: \"fc7\"\n}\nlayer {\n\tname: \"fc8\"\n\ttype: \"InnerProduct\"\n\tbottom: \"fc7\"\n\ttop: \"fc8\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 256\n\t\tweight_filler {\n\t\t\ttype: \"xavier\"\n\t\t}\n\t\tbias_filler {\n\t\t\ttype: \"constant\"\n\t\t\tvalue: 0\n\t\t}\n\t}\n}\nlayer {\n\tname: \"relu8\"\n\ttype: \"ReLU\"\n\tbottom: \"fc8\"\n\ttop: \"fc8\"\n}\n\n# Spatial Cfg Subnet\nlayer {\n\tname: \"conv1_p\"\n\ttype: \"Convolution\"\n\tbottom: \"posdata\"\n\ttop: \"conv1_p\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tconvolution_param {\n\t\tnum_output: 96\n\t\tkernel_size: 5\n\t\tpad: 2\n\t\tstride: 2\n\t}\n}\nlayer {\n\tname: \"relu1_p\"\n\ttype: \"ReLU\"\n\tbottom: \"conv1_p\"\n\ttop: \"conv1_p\"\n}\nlayer {\n\tname: \"conv2_p\"\n\ttype: \"Convolution\"\n\tbottom: \"conv1_p\"\n\ttop: \"conv2_p\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tconvolution_param {\n\t\tnum_output: 128\n\t\tkernel_size: 5\n\t\tpad: 2\n\t\tstride: 2\n\t}\n}\nlayer {\n\tname: \"conv3_p\"\n\ttype: \"Convolution\"\n\tbottom: \"conv2_p\"\n\ttop: \"conv3_p\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tconvolution_param {\n\t\tnum_output: 64\n\t\tkernel_size: 8\n\t}\n}\nlayer {\n\tname: \"relu3_p\"\n\ttype: \"ReLU\"\n\tbottom: \"conv3_p\"\n\ttop: \"conv3_p\"\n}\n\n# Combine features from subnets\nlayer {\n\tname: \"concat1_c\"\n\ttype: \"Python\"\n\tbottom: \"fc8\"\n\tbottom: \"conv3_p\"\n\ttop: \"concat1_c\"\n\tpython_param {\n\t\tmodule: \"customize_layers.concat_layer\"\n\t\tlayer: \"Layer\"\n\t}\n}\nlayer {\n\tname: \"fc2_c\"\n\ttype: \"InnerProduct\"\n\tbottom: \"concat1_c\"\n\ttop: \"fc2_c\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 128\n\t}\n}\nlayer {\n\tname: \"relu2_c\"\n\ttype: \"ReLU\"\n\tbottom: \"fc2_c\"\n\ttop: \"fc2_c\"\n}\nlayer {\n\tname: \"PhiR_0\"\n\ttype: \"InnerProduct\"\n\tbottom: \"fc2_c\"\n\ttop: \"q0r\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t\tweight_filler {\n\t\t\ttype: \"xavier\"\n\t\t}\n\t\tbias_filler {\n\t\t\ttype: \"constant\"\n\t\t\tvalue: 0\n\t\t}\n\t}\n}\nlayer {\n\tname: \"Softmax_0\"\n\ttype: \"Softmax\"\n\tbottom: \"q0r\"\n\ttop: \"qr0\"\n}\n\n#DR-Net\nlayer {\n\tname: \"PhiA_1\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qa\"\n\ttop: \"qar1\"\n\tparam {\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t\tweight_filler {\n\t\t\ttype: \"xavier\"\n\t\t}\n\t\tbias_filler {\n\t\t\ttype: \"constant\"\n\t\t\tvalue: 0\n\t\t}\n\t}\n}\nlayer {\n\tname: \"PhiB_1\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qb\"\n\ttop: \"qbr1\"\n\tparam {\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t\tweight_filler {\n\t\t\ttype: \"xavier\"\n\t\t}\n\t\tbias_filler {\n\t\t\ttype: \"constant\"\n\t\t\tvalue: 0\n\t\t}\n\t}\n}\nlayer {\n\tname: \"PhiR_1\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qr0\"\n\ttop: \"q1r\"\n\tparam {\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\t\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t\tweight_filler {\n\t\t\ttype: \"xavier\"\n\t\t}\n\t\tbias_filler {\n\t\t\ttype: \"constant\"\n\t\t\tvalue: 0\n\t\t}\n\t}\n}\nlayer {\n\tname: \"QSum_1\"\n\ttype: \"Eltwise\"\n\tbottom: \"qar1\"\n\tbottom: \"qbr1\"\n\tbottom: \"q1r\"\n\ttop: \"qr1un\"\n\teltwise_param { operation: SUM }\n}\nlayer {\n\tname: \"Softmax_1\"\n\ttype: \"Softmax\"\n\tbottom: \"qr1un\"\n\ttop: \"qr1\"\n}\nlayer {\n\tname: \"PhiA_2\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qa\"\n\ttop: \"qar2\"\n\tparam {\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"PhiB_2\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qb\"\n\ttop: \"qbr2\"\n\tparam {\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"PhiR_2\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qr1\"\n\ttop: \"q2r\"\n\tparam {\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"QSum_2\"\n\ttype: \"Eltwise\"\n\tbottom: \"qar2\"\n\tbottom: \"qbr2\"\n\tbottom: \"q2r\"\n\ttop: \"qr2un\"\n\teltwise_param { operation: SUM }\n}\nlayer {\n\tname: \"Softmax_2\"\n\ttype: \"Softmax\"\n\tbottom: \"qr2un\"\n\ttop: \"qr2\"\n}\nlayer {\n\tname: \"PhiA_3\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qa\"\n\ttop: \"qar3\"\n\tparam {\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"PhiB_3\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qb\"\n\ttop: \"qbr3\"\n\tparam {\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"PhiR_3\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qr2\"\n\ttop: \"q3r\"\n\tparam {\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"QSum_3\"\n\ttype: \"Eltwise\"\n\tbottom: \"qar3\"\n\tbottom: \"qbr3\"\n\tbottom: \"q3r\"\n\ttop: \"qr3un\"\n\teltwise_param { operation: SUM }\n}\nlayer {\n\tname: \"Softmax_3\"\n\ttype: \"Softmax\"\n\tbottom: \"qr3un\"\n\ttop: \"qr3\"\n}\nlayer {\n\tname: \"PhiA_4\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qa\"\n\ttop: \"qar4\"\n\tparam {\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"PhiB_4\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qb\"\n\ttop: \"qbr4\"\n\tparam {\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"PhiR_4\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qr3\"\n\ttop: \"q4r\"\n\tparam {\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"QSum_4\"\n\ttype: \"Eltwise\"\n\tbottom: \"qar4\"\n\tbottom: \"qbr4\"\n\tbottom: \"q4r\"\n\ttop: \"qr4un\"\n\teltwise_param { operation: SUM }\n}\nlayer {\n\tname: \"Softmax_4\"\n\ttype: \"Softmax\"\n\tbottom: \"qr4un\"\n\ttop: \"qr4\"\n}\nlayer {\n\tname: \"PhiA_5\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qa\"\n\ttop: \"qar5\"\n\tparam {\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"PhiB_5\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qb\"\n\ttop: \"qbr5\"\n\tparam {\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"PhiR_5\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qr4\"\n\ttop: \"q5r\"\n\tparam {\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"QSum_5\"\n\ttype: \"Eltwise\"\n\tbottom: \"qar5\"\n\tbottom: \"qbr5\"\n\tbottom: \"q5r\"\n\ttop: \"qr5un\"\n\teltwise_param { operation: SUM }\n}\nlayer {\n\tname: \"Softmax_5\"\n\ttype: \"Softmax\"\n\tbottom: \"qr5un\"\n\ttop: \"qr5\"\n}\n\nlayer {\n\tname: \"PhiA_6\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qa\"\n\ttop: \"qar6\"\n\tparam {\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"PhiB_6\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qb\"\n\ttop: \"qbr6\"\n\tparam {\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"PhiR_6\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qr5\"\n\ttop: \"q6r\"\n\tparam {\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"QSum_6\"\n\ttype: \"Eltwise\"\n\tbottom: \"qar6\"\n\tbottom: \"qbr6\"\n\tbottom: \"q6r\"\n\ttop: \"qr6un\"\n\teltwise_param { operation: SUM }\n}\nlayer {\n\tname: \"Softmax_6\"\n\ttype: \"Softmax\"\n\tbottom: \"qr6un\"\n\ttop: \"qr6\"\n}\nlayer {\n\tname: \"PhiA_7\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qa\"\n\ttop: \"qar7\"\n\tparam {\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"PhiB_7\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qb\"\n\ttop: \"qbr7\"\n\tparam {\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"PhiR_7\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qr6\"\n\ttop: \"q7r\"\n\tparam {\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"QSum_7\"\n\ttype: \"Eltwise\"\n\tbottom: \"qar7\"\n\tbottom: \"qbr7\"\n\tbottom: \"q7r\"\n\ttop: \"qr7un\"\n\teltwise_param { operation: SUM }\n}\nlayer {\n\tname: \"Softmax_7\"\n\ttype: \"Softmax\"\n\tbottom: \"qr7un\"\n\ttop: \"qr7\"\n}\nlayer {\n\tname: \"PhiA_8\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qa\"\n\ttop: \"qar8\"\n\tparam {\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"PhiB_8\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qb\"\n\ttop: \"qbr8\"\n\tparam {\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"PhiR_8\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qr7\"\n\ttop: \"q8r\"\n\tparam {\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"QSum_8\"\n\ttype: \"Eltwise\"\n\tbottom: \"qar8\"\n\tbottom: \"qbr8\"\n\tbottom: \"q8r\"\n\ttop: \"qr8\"\n\teltwise_param { operation: SUM }\n}\nlayer {\n\tname: \"loss3\"\n\ttype: \"SoftmaxWithLoss\"\n\ttop: \"loss3\"\n\tbottom: \"qr8\"\n\tbottom: \"labels\"\n\tinclude {\n\t\tphase: TRAIN\n\t}\n\tloss_weight: 1\n}\nlayer {\n\tname: \"softmax\"\n\ttype: \"Softmax\"\n\tbottom: \"qr8\"\n\ttop: \"pred\"\n\tinclude {\n\t\tphase: TEST\n\t}\n}\nlayer {\n\tname: \"accuracy\"\n\ttype: \"Accuracy\"\n\tbottom: \"pred\"\n\tbottom: \"labels\"\n\ttop: \"top-1\"\n\tinclude {\n\t\tphase: TEST\n\t}\n\taccuracy_param {\n\t\ttop_k: 1\n\t}\n}\nlayer {\n\tname: \"accuracy_top5\"\n\ttype: \"Accuracy\"\n\tbottom: \"pred\"\n\tbottom: \"labels\"\n\ttop: \"top-5\"\n\tinclude {\n\t\tphase: TEST\n\t}\n\taccuracy_param {\n\t\ttop_k: 5\n\t}\n}\n"
  },
  {
    "path": "prototxts/test_drnet_8units_linear_shareweight.prototxt",
    "content": "input: \"qa\"\ninput_shape {\t\n\tdim: 1\n\tdim: 100\n}\n\ninput: \"qb\"\ninput_shape {\n\tdim: 1\n\tdim: 100\n}\n\ninput: \"im\"\ninput_shape {\n\tdim: 1\n\tdim: 3\n\tdim: 224\n\tdim: 224\n}\n\ninput: \"posdata\"\ninput_shape {\n\tdim: 1\n\tdim: 2\n\tdim: 32\n\tdim: 32\n}\n\n# Appearance Subnet\nlayer {\n\tname: \"conv1_1\"\n\ttype: \"Convolution\"\n\tbottom: \"im\"\n\ttop: \"conv1_1\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t} \n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tconvolution_param {\n\t\tnum_output: 64\n\t\tpad: 1\n\t\tkernel_size: 3\n\t}\n}\nlayer {\n\tname: \"relu1_1\"\n\ttype: \"ReLU\"\n\tbottom: \"conv1_1\"\n\ttop: \"conv1_1\"\n}\nlayer {\n\tname: \"conv1_2\"\n\ttype: \"Convolution\"\n\tbottom: \"conv1_1\"\n\ttop: \"conv1_2\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tconvolution_param {\n\t\tnum_output: 64\n\t\tpad: 1\n\t\tkernel_size: 3\n\t}\n}\nlayer {\n\tname: \"relu1_2\"\n\ttype: \"ReLU\"\n\tbottom: \"conv1_2\"\n\ttop: \"conv1_2\"\n}\nlayer {\n\tname: \"pool1\"\n\ttype: \"Pooling\"\n\tbottom: \"conv1_2\"\n\ttop: \"pool1\"\n\tpooling_param {\n\t\tpool: MAX\n\t\tkernel_size: 2\n\t\tstride: 2\n\t}\n}\nlayer {\n\tname: \"conv2_1\"\n\ttype: \"Convolution\"\n\tbottom: \"pool1\"\n\ttop: \"conv2_1\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tconvolution_param {\n\t\tnum_output: 128\n\t\tpad: 1\n\t\tkernel_size: 3\n\t}\n}\nlayer {\n\tname: \"relu2_1\"\n\ttype: \"ReLU\"\n\tbottom: \"conv2_1\"\n\ttop: \"conv2_1\"\n}\nlayer {\n\tname: \"conv2_2\"\n\ttype: \"Convolution\"\n\tbottom: \"conv2_1\"\n\ttop: \"conv2_2\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tconvolution_param {\n\t\tnum_output: 128\n\t\tpad: 1\n\t\tkernel_size: 3\n\t}\n}\nlayer {\n\tname: \"relu2_2\"\n\ttype: \"ReLU\"\n\tbottom: \"conv2_2\"\n\ttop: \"conv2_2\"\n}\nlayer {\n\tname: \"pool2\"\t\n\ttype: \"Pooling\"\n\tbottom: \"conv2_2\"\n\ttop: \"pool2\"\n\tpooling_param {\n\t\tpool: MAX\n\t\tkernel_size: 2\n\t\tstride: 2\n\t}\n}\nlayer {\n\tname: \"conv3_1\"\n\ttype: \"Convolution\"\n\tbottom: \"pool2\"\n\ttop: \"conv3_1\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tconvolution_param {\n\t\tnum_output: 256\n\t\tpad: 1\n\t\tkernel_size: 3\n\t}\n}\nlayer {\n\tname: \"relu3_1\"\n\ttype: \"ReLU\"\n\tbottom: \"conv3_1\"\n\ttop: \"conv3_1\"\n}\nlayer {\n\tname: \"conv3_2\"\n\ttype: \"Convolution\"\n\tbottom: \"conv3_1\"\n\ttop: \"conv3_2\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tconvolution_param {\n\t\tnum_output: 256\n\t\tpad: 1\n\t\tkernel_size: 3\n\t}\n}\nlayer {\n\tname: \"relu3_2\"\n\ttype: \"ReLU\"\n\tbottom: \"conv3_2\"\n\ttop: \"conv3_2\"\n}\nlayer {\n\tname: \"conv3_3\"\n\ttype: \"Convolution\"\n\tbottom: \"conv3_2\"\n\ttop: \"conv3_3\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tconvolution_param {\n\t\tnum_output: 256\n\t\tpad: 1\n\t\tkernel_size: 3\n\t}\n}\nlayer {\n\tname: \"relu3_3\"\n\ttype: \"ReLU\"\n\tbottom: \"conv3_3\"\n\ttop: \"conv3_3\"\n}\nlayer {\n\tname: \"pool3\"\n\ttype: \"Pooling\"\n\tbottom: \"conv3_3\"\n\ttop: \"pool3\"\n\tpooling_param {\n\t\tpool: MAX\n\t\tkernel_size: 2\n\t\tstride: 2\n\t}\n}\nlayer {\n\tname: \"conv4_1\"\n\ttype: \"Convolution\"\n\tbottom: \"pool3\"\n\ttop: \"conv4_1\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tconvolution_param {\n\t\tnum_output: 512\n\t\tpad: 1\n\t\tkernel_size: 3\n\t}\n}\nlayer {\n\tname: \"relu4_1\"\n\ttype: \"ReLU\"\n\tbottom: \"conv4_1\"\n\ttop: \"conv4_1\"\n}\nlayer {\n\tname: \"conv4_2\"\n\ttype: \"Convolution\"\n\tbottom: \"conv4_1\"\n\ttop: \"conv4_2\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tconvolution_param {\n\t\tnum_output: 512\n\t\tpad: 1\n\t\tkernel_size: 3\n\t}\n}\nlayer {\n\tname: \"relu4_2\"\n\ttype: \"ReLU\"\n\tbottom: \"conv4_2\"\n\ttop: \"conv4_2\"\n}\nlayer {\n\tname: \"conv4_3\"\n\ttype: \"Convolution\"\n\tbottom: \"conv4_2\"\n\ttop: \"conv4_3\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tconvolution_param {\n\t\tnum_output: 512\n\t\tpad: 1\n\t\tkernel_size: 3\n\t}\n}\nlayer {\n\tname: \"relu4_3\"\n\ttype: \"ReLU\"\n\tbottom: \"conv4_3\"\n\ttop: \"conv4_3\"\n}\nlayer {\n\tname: \"pool4\"\n\ttype: \"Pooling\"\n\tbottom: \"conv4_3\"\n\ttop: \"pool4\"\n\tpooling_param {\n\t\tpool: MAX\n\t\tkernel_size: 2\n\t\tstride: 2\n\t}\n}\nlayer {\n\tname: \"conv5_1\"\n\ttype: \"Convolution\"\n\tbottom: \"pool4\"\n\ttop: \"conv5_1\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tconvolution_param {\n\t\tnum_output: 512\n\t\tpad: 1\n\t\tkernel_size: 3\n\t}\n}\nlayer {\n\tname: \"relu5_1\"\n\ttype: \"ReLU\"\n\tbottom: \"conv5_1\"\n\ttop: \"conv5_1\"\n}\nlayer {\n\tname: \"conv5_2\"\n\ttype: \"Convolution\"\n\tbottom: \"conv5_1\"\n\ttop: \"conv5_2\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tconvolution_param {\n\t\tnum_output: 512\n\t\tpad: 1\n\t\tkernel_size: 3\n\t}\n}\nlayer {\n\tname: \"relu5_2\"\n\ttype: \"ReLU\"\n\tbottom: \"conv5_2\"\n\ttop: \"conv5_2\"\n}\nlayer {\n\tname: \"conv5_3\"\n\ttype: \"Convolution\"\n\tbottom: \"conv5_2\"\n\ttop: \"conv5_3\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tconvolution_param {\n\t\tnum_output: 512\n\t\tpad: 1\n\t\tkernel_size: 3\n\t}\n}\nlayer {\n\tname: \"relu5_3\"\n\ttype: \"ReLU\"\n\tbottom: \"conv5_3\"\n\ttop: \"conv5_3\"\n}\nlayer {\n\tname: \"pool5\"\n\ttype: \"Pooling\"\n\tbottom: \"conv5_3\"\n\ttop: \"pool5\"\n\tpooling_param {\n\t\tpool: MAX\n\t\tkernel_size: 2\n\t\tstride: 2\n\t}\n}\nlayer {\n\tname: \"fc6\"\n\ttype: \"InnerProduct\"\n\tbottom: \"pool5\"\n\ttop: \"fc6\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 4096\n\t}\n}\nlayer {\n\tname: \"relu6\"\n\ttype: \"ReLU\"\n\tbottom: \"fc6\"\n\ttop: \"fc6\"\n}\nlayer {\n\tname: \"fc7\"\n\ttype: \"InnerProduct\"\n\tbottom: \"fc6\"\n\ttop: \"fc7\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 4096\n\t}\n}\nlayer {\n\tname: \"relu7\"\n\ttype: \"ReLU\"\n\tbottom: \"fc7\"\n\ttop: \"fc7\"\n}\nlayer {\n\tname: \"fc8\"\n\ttype: \"InnerProduct\"\n\tbottom: \"fc7\"\n\ttop: \"fc8\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 256\n\t\tweight_filler {\n\t\t\ttype: \"xavier\"\n\t\t}\n\t\tbias_filler {\n\t\t\ttype: \"constant\"\n\t\t\tvalue: 0\n\t\t}\n\t}\n}\nlayer {\n\tname: \"relu8\"\n\ttype: \"ReLU\"\n\tbottom: \"fc8\"\n\ttop: \"fc8\"\n}\n# Spatial Cfg Subnet\nlayer {\n\tname: \"conv1_p\"\n\ttype: \"Convolution\"\n\tbottom: \"posdata\"\n\ttop: \"conv1_p\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tconvolution_param {\n\t\tnum_output: 96\n\t\tkernel_size: 5\n\t\tpad: 2\n\t\tstride: 2\n\t}\n}\nlayer {\n\tname: \"relu1_p\"\n\ttype: \"ReLU\"\n\tbottom: \"conv1_p\"\n\ttop: \"conv1_p\"\n}\nlayer {\n\tname: \"conv2_p\"\n\ttype: \"Convolution\"\n\tbottom: \"conv1_p\"\n\ttop: \"conv2_p\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tconvolution_param {\n\t\tnum_output: 128\n\t\tkernel_size: 5\n\t\tpad: 2\n\t\tstride: 2\n\t}\n}\nlayer {\n\tname: \"conv3_p\"\n\ttype: \"Convolution\"\n\tbottom: \"conv2_p\"\n\ttop: \"conv3_p\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tconvolution_param {\n\t\tnum_output: 64\n\t\tkernel_size: 8\n\t}\n}\nlayer {\n\tname: \"relu3_p\"\n\ttype: \"ReLU\"\n\tbottom: \"conv3_p\"\n\ttop: \"conv3_p\"\n}\n# Combine features from subnets\nlayer {\n\tname: \"concat1_c\"\n\ttype: \"Python\"\n\tbottom: \"fc8\"\n\tbottom: \"conv3_p\"\n\ttop: \"concat1_c\"\n\tpython_param {\n\t\tmodule: \"customize_layers.concat_layer\"\n\t\tlayer: \"Layer\"\n\t}\n}\nlayer {\n\tname: \"fc2_c\"\n\ttype: \"InnerProduct\"\n\tbottom: \"concat1_c\"\n\ttop: \"fc2_c\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 128\n\t}\n}\nlayer {\n\tname: \"relu2_c\"\n\ttype: \"ReLU\"\n\tbottom: \"fc2_c\"\n\ttop: \"fc2_c\"\n}\nlayer {\n\tname: \"PhiR_0\"\n\ttype: \"InnerProduct\"\n\tbottom: \"fc2_c\"\n\ttop: \"qr0\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t\tweight_filler {\n\t\t\ttype: \"xavier\"\n\t\t}\n\t\tbias_filler {\n\t\t\ttype: \"constant\"\n\t\t\tvalue: 0\n\t\t}\n\t}\n}\n#DR-Net\nlayer {\n\tname: \"PhiA_1\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qa\"\n\ttop: \"qar1\"\n\tparam {\n\t\tname: \"qar_w\"\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tname: \"qar_b\"\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t\tweight_filler {\n\t\t\ttype: \"xavier\"\n\t\t}\n\t\tbias_filler {\n\t\t\ttype: \"constant\"\n\t\t\tvalue: 0\n\t\t}\n\t}\n}\nlayer {\n\tname: \"PhiB_1\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qb\"\n\ttop: \"qbr1\"\n\tparam {\n\t\tname: \"qbr_w\"\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tname: \"qbr_b\"\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t\tweight_filler {\n\t\t\ttype: \"xavier\"\n\t\t}\n\t\tbias_filler {\n\t\t\ttype: \"constant\"\n\t\t\tvalue: 0\n\t\t}\n\t}\n}\nlayer {\n\tname: \"PhiR_1\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qr0\"\n\ttop: \"q1r\"\n\tparam {\n\t\tname: \"qr_w\"\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\t\n\t\tname: \"qr_b\"\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t\tweight_filler {\n\t\t\ttype: \"xavier\"\n\t\t}\n\t\tbias_filler {\n\t\t\ttype: \"constant\"\n\t\t\tvalue: 0\n\t\t}\n\t}\n}\nlayer {\n\tname: \"QSum_1\"\n\ttype: \"Eltwise\"\n\tbottom: \"qar1\"\n\tbottom: \"qbr1\"\n\tbottom: \"q1r\"\n\ttop: \"qr1\"\n\teltwise_param { operation: SUM }\n}\nlayer {\n\tname: \"PhiA_2\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qa\"\n\ttop: \"qar2\"\n\tparam {\n\t\tname: \"qar_w\"\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tname: \"qar_b\"\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"PhiB_2\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qb\"\n\ttop: \"qbr2\"\n\tparam {\n\t\tname: \"qbr_w\"\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tname: \"qbr_b\"\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"PhiR_2\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qr1\"\n\ttop: \"q2r\"\n\tparam {\n\t\tname: \"qr_w\"\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tname: \"qr_b\"\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"QSum_2\"\n\ttype: \"Eltwise\"\n\tbottom: \"qar2\"\n\tbottom: \"qbr2\"\n\tbottom: \"q2r\"\n\ttop: \"qr2\"\n\teltwise_param { operation: SUM }\n}\nlayer {\n\tname: \"PhiA_3\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qa\"\n\ttop: \"qar3\"\n\tparam {\n\t\tname: \"qar_w\"\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tname: \"qar_b\"\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"PhiB_3\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qb\"\n\ttop: \"qbr3\"\n\tparam {\n\t\tname: \"qbr_w\"\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tname: \"qbr_b\"\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"PhiR_3\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qr2\"\n\ttop: \"q3r\"\n\tparam {\n\t\tname: \"qr_w\"\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tname: \"qr_b\"\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"QSum_3\"\n\ttype: \"Eltwise\"\n\tbottom: \"qar3\"\n\tbottom: \"qbr3\"\n\tbottom: \"q3r\"\n\ttop: \"qr3\"\n\teltwise_param { operation: SUM }\n}\nlayer {\n\tname: \"PhiA_4\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qa\"\n\ttop: \"qar4\"\n\tparam {\n\t\tname: \"qar_w\"\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tname: \"qar_b\"\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"PhiB_4\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qb\"\n\ttop: \"qbr4\"\n\tparam {\n\t\tname: \"qbr_w\"\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tname: \"qbr_b\"\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"PhiR_4\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qr3\"\n\ttop: \"q4r\"\n\tparam {\n\t\tname: \"qr_w\"\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tname: \"qr_b\"\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"QSum_4\"\n\ttype: \"Eltwise\"\n\tbottom: \"qar4\"\n\tbottom: \"qbr4\"\n\tbottom: \"q4r\"\n\ttop: \"qr4\"\n\teltwise_param { operation: SUM }\n}\nlayer {\n\tname: \"PhiA_5\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qa\"\n\ttop: \"qar5\"\n\tparam {\n\t\tname: \"qar_w\"\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tname: \"qar_b\"\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"PhiB_5\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qb\"\n\ttop: \"qbr5\"\n\tparam {\n\t\tname: \"qbr_w\"\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tname: \"qbr_b\"\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"PhiR_5\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qr4\"\n\ttop: \"q5r\"\n\tparam {\n\t\tname: \"qr_w\"\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tname: \"qr_b\"\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"QSum_5\"\n\ttype: \"Eltwise\"\n\tbottom: \"qar5\"\n\tbottom: \"qbr5\"\n\tbottom: \"q5r\"\n\ttop: \"qr5\"\n\teltwise_param { operation: SUM }\n}\nlayer {\n\tname: \"PhiA_6\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qa\"\n\ttop: \"qar6\"\n\tparam {\n\t\tname: \"qar_w\"\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tname: \"qar_b\"\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"PhiB_6\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qb\"\n\ttop: \"qbr6\"\n\tparam {\n\t\tname: \"qbr_w\"\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tname: \"qbr_b\"\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"PhiR_6\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qr5\"\n\ttop: \"q6r\"\n\tparam {\n\t\tname: \"qr_w\"\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tname: \"qr_b\"\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"QSum_6\"\n\ttype: \"Eltwise\"\n\tbottom: \"qar6\"\n\tbottom: \"qbr6\"\n\tbottom: \"q6r\"\n\ttop: \"qr6\"\n\teltwise_param { operation: SUM }\n}\nlayer {\n\tname: \"PhiA_7\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qa\"\n\ttop: \"qar7\"\n\tparam {\n\t\tname: \"qar_w\"\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tname: \"qar_b\"\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"PhiB_7\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qb\"\n\ttop: \"qbr7\"\n\tparam {\n\t\tname: \"qbr_w\"\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tname: \"qbr_b\"\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"PhiR_7\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qr6\"\n\ttop: \"q7r\"\n\tparam {\n\t\tname: \"qr_w\"\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tname: \"qr_b\"\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"QSum_7\"\n\ttype: \"Eltwise\"\n\tbottom: \"qar7\"\n\tbottom: \"qbr7\"\n\tbottom: \"q7r\"\n\ttop: \"qr7\"\n\teltwise_param { operation: SUM }\n}\nlayer {\n\tname: \"PhiA_8\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qa\"\n\ttop: \"qar8\"\n\tparam {\n\t\tname: \"qar_w\"\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tname: \"qar_b\"\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"PhiB_8\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qb\"\n\ttop: \"qbr8\"\n\tparam {\n\t\tname: \"qbr_w\"\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tname: \"qbr_b\"\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"PhiR_8\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qr7\"\n\ttop: \"q8r\"\n\tparam {\n\t\tname: \"qr_w\"\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tname: \"qr_b\"\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"QSum_8\"\n\ttype: \"Eltwise\"\n\tbottom: \"qar8\"\n\tbottom: \"qbr8\"\n\tbottom: \"q8r\"\n\ttop: \"qr8\"\n\teltwise_param { operation: SUM }\n}\n\nlayer {\n\tname: \"softmax\"\n\ttype: \"Softmax\"\n\tbottom: \"qr8\"\n\ttop: \"pred\"\n}\n"
  },
  {
    "path": "prototxts/test_drnet_8units_relu_shareweight.prototxt",
    "content": "input: \"qa\"\ninput_shape {\t\n\tdim: 1\n\tdim: 100\n}\n\ninput: \"qb\"\ninput_shape {\n\tdim: 1\n\tdim: 100\n}\n\ninput: \"im\"\ninput_shape {\n\tdim: 1\n\tdim: 3\n\tdim: 224\n\tdim: 224\n}\n\ninput: \"posdata\"\ninput_shape {\n\tdim: 1\n\tdim: 2\n\tdim: 32\n\tdim: 32\n}\n\n# Appearance Subnet\nlayer {\n\tname: \"conv1_1\"\n\ttype: \"Convolution\"\n\tbottom: \"im\"\n\ttop: \"conv1_1\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t} \n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tconvolution_param {\n\t\tnum_output: 64\n\t\tpad: 1\n\t\tkernel_size: 3\n\t}\n}\nlayer {\n\tname: \"relu1_1\"\n\ttype: \"ReLU\"\n\tbottom: \"conv1_1\"\n\ttop: \"conv1_1\"\n}\nlayer {\n\tname: \"conv1_2\"\n\ttype: \"Convolution\"\n\tbottom: \"conv1_1\"\n\ttop: \"conv1_2\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tconvolution_param {\n\t\tnum_output: 64\n\t\tpad: 1\n\t\tkernel_size: 3\n\t}\n}\nlayer {\n\tname: \"relu1_2\"\n\ttype: \"ReLU\"\n\tbottom: \"conv1_2\"\n\ttop: \"conv1_2\"\n}\nlayer {\n\tname: \"pool1\"\n\ttype: \"Pooling\"\n\tbottom: \"conv1_2\"\n\ttop: \"pool1\"\n\tpooling_param {\n\t\tpool: MAX\n\t\tkernel_size: 2\n\t\tstride: 2\n\t}\n}\nlayer {\n\tname: \"conv2_1\"\n\ttype: \"Convolution\"\n\tbottom: \"pool1\"\n\ttop: \"conv2_1\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tconvolution_param {\n\t\tnum_output: 128\n\t\tpad: 1\n\t\tkernel_size: 3\n\t}\n}\nlayer {\n\tname: \"relu2_1\"\n\ttype: \"ReLU\"\n\tbottom: \"conv2_1\"\n\ttop: \"conv2_1\"\n}\nlayer {\n\tname: \"conv2_2\"\n\ttype: \"Convolution\"\n\tbottom: \"conv2_1\"\n\ttop: \"conv2_2\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tconvolution_param {\n\t\tnum_output: 128\n\t\tpad: 1\n\t\tkernel_size: 3\n\t}\n}\nlayer {\n\tname: \"relu2_2\"\n\ttype: \"ReLU\"\n\tbottom: \"conv2_2\"\n\ttop: \"conv2_2\"\n}\nlayer {\n\tname: \"pool2\"\t\n\ttype: \"Pooling\"\n\tbottom: \"conv2_2\"\n\ttop: \"pool2\"\n\tpooling_param {\n\t\tpool: MAX\n\t\tkernel_size: 2\n\t\tstride: 2\n\t}\n}\nlayer {\n\tname: \"conv3_1\"\n\ttype: \"Convolution\"\n\tbottom: \"pool2\"\n\ttop: \"conv3_1\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tconvolution_param {\n\t\tnum_output: 256\n\t\tpad: 1\n\t\tkernel_size: 3\n\t}\n}\nlayer {\n\tname: \"relu3_1\"\n\ttype: \"ReLU\"\n\tbottom: \"conv3_1\"\n\ttop: \"conv3_1\"\n}\nlayer {\n\tname: \"conv3_2\"\n\ttype: \"Convolution\"\n\tbottom: \"conv3_1\"\n\ttop: \"conv3_2\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tconvolution_param {\n\t\tnum_output: 256\n\t\tpad: 1\n\t\tkernel_size: 3\n\t}\n}\nlayer {\n\tname: \"relu3_2\"\n\ttype: \"ReLU\"\n\tbottom: \"conv3_2\"\n\ttop: \"conv3_2\"\n}\nlayer {\n\tname: \"conv3_3\"\n\ttype: \"Convolution\"\n\tbottom: \"conv3_2\"\n\ttop: \"conv3_3\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tconvolution_param {\n\t\tnum_output: 256\n\t\tpad: 1\n\t\tkernel_size: 3\n\t}\n}\nlayer {\n\tname: \"relu3_3\"\n\ttype: \"ReLU\"\n\tbottom: \"conv3_3\"\n\ttop: \"conv3_3\"\n}\nlayer {\n\tname: \"pool3\"\n\ttype: \"Pooling\"\n\tbottom: \"conv3_3\"\n\ttop: \"pool3\"\n\tpooling_param {\n\t\tpool: MAX\n\t\tkernel_size: 2\n\t\tstride: 2\n\t}\n}\nlayer {\n\tname: \"conv4_1\"\n\ttype: \"Convolution\"\n\tbottom: \"pool3\"\n\ttop: \"conv4_1\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tconvolution_param {\n\t\tnum_output: 512\n\t\tpad: 1\n\t\tkernel_size: 3\n\t}\n}\nlayer {\n\tname: \"relu4_1\"\n\ttype: \"ReLU\"\n\tbottom: \"conv4_1\"\n\ttop: \"conv4_1\"\n}\nlayer {\n\tname: \"conv4_2\"\n\ttype: \"Convolution\"\n\tbottom: \"conv4_1\"\n\ttop: \"conv4_2\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tconvolution_param {\n\t\tnum_output: 512\n\t\tpad: 1\n\t\tkernel_size: 3\n\t}\n}\nlayer {\n\tname: \"relu4_2\"\n\ttype: \"ReLU\"\n\tbottom: \"conv4_2\"\n\ttop: \"conv4_2\"\n}\nlayer {\n\tname: \"conv4_3\"\n\ttype: \"Convolution\"\n\tbottom: \"conv4_2\"\n\ttop: \"conv4_3\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tconvolution_param {\n\t\tnum_output: 512\n\t\tpad: 1\n\t\tkernel_size: 3\n\t}\n}\nlayer {\n\tname: \"relu4_3\"\n\ttype: \"ReLU\"\n\tbottom: \"conv4_3\"\n\ttop: \"conv4_3\"\n}\nlayer {\n\tname: \"pool4\"\n\ttype: \"Pooling\"\n\tbottom: \"conv4_3\"\n\ttop: \"pool4\"\n\tpooling_param {\n\t\tpool: MAX\n\t\tkernel_size: 2\n\t\tstride: 2\n\t}\n}\nlayer {\n\tname: \"conv5_1\"\n\ttype: \"Convolution\"\n\tbottom: \"pool4\"\n\ttop: \"conv5_1\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tconvolution_param {\n\t\tnum_output: 512\n\t\tpad: 1\n\t\tkernel_size: 3\n\t}\n}\nlayer {\n\tname: \"relu5_1\"\n\ttype: \"ReLU\"\n\tbottom: \"conv5_1\"\n\ttop: \"conv5_1\"\n}\nlayer {\n\tname: \"conv5_2\"\n\ttype: \"Convolution\"\n\tbottom: \"conv5_1\"\n\ttop: \"conv5_2\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tconvolution_param {\n\t\tnum_output: 512\n\t\tpad: 1\n\t\tkernel_size: 3\n\t}\n}\nlayer {\n\tname: \"relu5_2\"\n\ttype: \"ReLU\"\n\tbottom: \"conv5_2\"\n\ttop: \"conv5_2\"\n}\nlayer {\n\tname: \"conv5_3\"\n\ttype: \"Convolution\"\n\tbottom: \"conv5_2\"\n\ttop: \"conv5_3\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tconvolution_param {\n\t\tnum_output: 512\n\t\tpad: 1\n\t\tkernel_size: 3\n\t}\n}\nlayer {\n\tname: \"relu5_3\"\n\ttype: \"ReLU\"\n\tbottom: \"conv5_3\"\n\ttop: \"conv5_3\"\n}\nlayer {\n\tname: \"pool5\"\n\ttype: \"Pooling\"\n\tbottom: \"conv5_3\"\n\ttop: \"pool5\"\n\tpooling_param {\n\t\tpool: MAX\n\t\tkernel_size: 2\n\t\tstride: 2\n\t}\n}\nlayer {\n\tname: \"fc6\"\n\ttype: \"InnerProduct\"\n\tbottom: \"pool5\"\n\ttop: \"fc6\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 4096\n\t}\n}\nlayer {\n\tname: \"relu6\"\n\ttype: \"ReLU\"\n\tbottom: \"fc6\"\n\ttop: \"fc6\"\n}\nlayer {\n\tname: \"fc7\"\n\ttype: \"InnerProduct\"\n\tbottom: \"fc6\"\n\ttop: \"fc7\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 4096\n\t}\n}\nlayer {\n\tname: \"relu7\"\n\ttype: \"ReLU\"\n\tbottom: \"fc7\"\n\ttop: \"fc7\"\n}\nlayer {\n\tname: \"fc8\"\n\ttype: \"InnerProduct\"\n\tbottom: \"fc7\"\n\ttop: \"fc8\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 256\n\t\tweight_filler {\n\t\t\ttype: \"xavier\"\n\t\t}\n\t\tbias_filler {\n\t\t\ttype: \"constant\"\n\t\t\tvalue: 0\n\t\t}\n\t}\n}\nlayer {\n\tname: \"relu8\"\n\ttype: \"ReLU\"\n\tbottom: \"fc8\"\n\ttop: \"fc8\"\n}\n\n# Spatial Cfg Subnet\nlayer {\n\tname: \"conv1_p\"\n\ttype: \"Convolution\"\n\tbottom: \"posdata\"\n\ttop: \"conv1_p\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tconvolution_param {\n\t\tnum_output: 96\n\t\tkernel_size: 5\n\t\tpad: 2\n\t\tstride: 2\n\t}\n}\nlayer {\n\tname: \"relu1_p\"\n\ttype: \"ReLU\"\n\tbottom: \"conv1_p\"\n\ttop: \"conv1_p\"\n}\nlayer {\n\tname: \"conv2_p\"\n\ttype: \"Convolution\"\n\tbottom: \"conv1_p\"\n\ttop: \"conv2_p\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tconvolution_param {\n\t\tnum_output: 128\n\t\tkernel_size: 5\n\t\tpad: 2\n\t\tstride: 2\n\t}\n}\nlayer {\n\tname: \"conv3_p\"\n\ttype: \"Convolution\"\n\tbottom: \"conv2_p\"\n\ttop: \"conv3_p\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tconvolution_param {\n\t\tnum_output: 64\n\t\tkernel_size: 8\n\t}\n}\nlayer {\n\tname: \"relu3_p\"\n\ttype: \"ReLU\"\n\tbottom: \"conv3_p\"\n\ttop: \"conv3_p\"\n}\n# Combine features from subnets\nlayer {\n\tname: \"concat1_c\"\n\ttype: \"Python\"\n\tbottom: \"fc8\"\n\tbottom: \"conv3_p\"\n\ttop: \"concat1_c\"\n\tpython_param {\n\t\tmodule: \"customize_layers.concat_layer\"\n\t\tlayer: \"Layer\"\n\t}\n}\nlayer {\n\tname: \"fc2_c\"\n\ttype: \"InnerProduct\"\n\tbottom: \"concat1_c\"\n\ttop: \"fc2_c\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 128\n\t}\n}\nlayer {\n\tname: \"relu2_c\"\n\ttype: \"ReLU\"\n\tbottom: \"fc2_c\"\n\ttop: \"fc2_c\"\n}\nlayer {\n\tname: \"PhiR_0\"\n\ttype: \"InnerProduct\"\n\tbottom: \"fc2_c\"\n\ttop: \"q0r\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t\tweight_filler {\n\t\t\ttype: \"xavier\"\n\t\t}\n\t\tbias_filler {\n\t\t\ttype: \"constant\"\n\t\t\tvalue: 0\n\t\t}\n\t}\n}\nlayer {\n\tname: \"relu_0\"\n\ttype: \"ReLU\"\n\tbottom: \"q0r\"\n\ttop: \"qr0\"\n}\n#DR-Net\nlayer {\n\tname: \"PhiA_1\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qa\"\n\ttop: \"qar1\"\n\tparam {\n\t\tname: \"qar_w\"\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tname: \"qar_b\"\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t\tweight_filler {\n\t\t\ttype: \"xavier\"\n\t\t}\n\t\tbias_filler {\n\t\t\ttype: \"constant\"\n\t\t\tvalue: 0\n\t\t}\n\t}\n}\nlayer {\n\tname: \"PhiB_1\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qb\"\n\ttop: \"qbr1\"\n\tparam {\n\t\tname: \"qbr_w\"\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tname: \"qbr_b\"\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t\tweight_filler {\n\t\t\ttype: \"xavier\"\n\t\t}\n\t\tbias_filler {\n\t\t\ttype: \"constant\"\n\t\t\tvalue: 0\n\t\t}\n\t}\n}\nlayer {\n\tname: \"PhiR_1\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qr0\"\n\ttop: \"q1r\"\n\tparam {\n\t\tname: \"qr_w\"\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\t\n\t\tname: \"qr_b\"\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t\tweight_filler {\n\t\t\ttype: \"xavier\"\n\t\t}\n\t\tbias_filler {\n\t\t\ttype: \"constant\"\n\t\t\tvalue: 0\n\t\t}\n\t}\n}\nlayer {\n\tname: \"QSum_1\"\n\ttype: \"Eltwise\"\n\tbottom: \"qar1\"\n\tbottom: \"qbr1\"\n\tbottom: \"q1r\"\n\ttop: \"qr1un\"\n\teltwise_param { operation: SUM }\n}\nlayer {\n\tname: \"relu_1\"\n\ttype: \"ReLU\"\n\tbottom: \"qr1un\"\n\ttop: \"qr1\"\n}\nlayer {\n\tname: \"PhiA_2\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qa\"\n\ttop: \"qar2\"\n\tparam {\n\t\tname: \"qar_w\"\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tname: \"qar_b\"\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"PhiB_2\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qb\"\n\ttop: \"qbr2\"\n\tparam {\n\t\tname: \"qbr_w\"\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tname: \"qbr_b\"\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"PhiR_2\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qr1\"\n\ttop: \"q2r\"\n\tparam {\n\t\tname: \"qr_w\"\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tname: \"qr_b\"\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"QSum_2\"\n\ttype: \"Eltwise\"\n\tbottom: \"qar2\"\n\tbottom: \"qbr2\"\n\tbottom: \"q2r\"\n\ttop: \"qr2un\"\n\teltwise_param { operation: SUM }\n}\nlayer {\n\tname: \"relu_2\"\n\ttype: \"ReLU\"\n\tbottom: \"qr2un\"\n\ttop: \"qr2\"\n}\nlayer {\n\tname: \"PhiA_3\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qa\"\n\ttop: \"qar3\"\n\tparam {\n\t\tname: \"qar_w\"\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tname: \"qar_b\"\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"PhiB_3\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qb\"\n\ttop: \"qbr3\"\n\tparam {\n\t\tname: \"qbr_w\"\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tname: \"qbr_b\"\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"PhiR_3\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qr2\"\n\ttop: \"q3r\"\n\tparam {\n\t\tname: \"qr_w\"\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tname: \"qr_b\"\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"QSum_3\"\n\ttype: \"Eltwise\"\n\tbottom: \"qar3\"\n\tbottom: \"qbr3\"\n\tbottom: \"q3r\"\n\ttop: \"qr3un\"\n\teltwise_param { operation: SUM }\n}\nlayer {\n\tname: \"relu_3\"\n\ttype: \"ReLU\"\n\tbottom: \"qr3un\"\n\ttop: \"qr3\"\n}\nlayer {\n\tname: \"PhiA_4\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qa\"\n\ttop: \"qar4\"\n\tparam {\n\t\tname: \"qar_w\"\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tname: \"qar_b\"\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"PhiB_4\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qb\"\n\ttop: \"qbr4\"\n\tparam {\n\t\tname: \"qbr_w\"\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tname: \"qbr_b\"\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"PhiR_4\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qr3\"\n\ttop: \"q4r\"\n\tparam {\n\t\tname: \"qr_w\"\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tname: \"qr_b\"\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"QSum_4\"\n\ttype: \"Eltwise\"\n\tbottom: \"qar4\"\n\tbottom: \"qbr4\"\n\tbottom: \"q4r\"\n\ttop: \"qr4un\"\n\teltwise_param { operation: SUM }\n}\nlayer {\n\tname: \"relu_4\"\n\ttype: \"ReLU\"\n\tbottom: \"qr4un\"\n\ttop: \"qr4\"\n}\nlayer {\n\tname: \"PhiA_5\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qa\"\n\ttop: \"qar5\"\n\tparam {\n\t\tname: \"qar_w\"\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tname: \"qar_b\"\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"PhiB_5\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qb\"\n\ttop: \"qbr5\"\n\tparam {\n\t\tname: \"qbr_w\"\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tname: \"qbr_b\"\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"PhiR_5\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qr4\"\n\ttop: \"q5r\"\n\tparam {\n\t\tname: \"qr_w\"\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tname: \"qr_b\"\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"QSum_5\"\n\ttype: \"Eltwise\"\n\tbottom: \"qar5\"\n\tbottom: \"qbr5\"\n\tbottom: \"q5r\"\n\ttop: \"qr5un\"\n\teltwise_param { operation: SUM }\n}\nlayer {\n\tname: \"relu_5\"\n\ttype: \"ReLU\"\n\tbottom: \"qr5un\"\n\ttop: \"qr5\"\n}\n\nlayer {\n\tname: \"PhiA_6\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qa\"\n\ttop: \"qar6\"\n\tparam {\n\t\tname: \"qar_w\"\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tname: \"qar_b\"\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"PhiB_6\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qb\"\n\ttop: \"qbr6\"\n\tparam {\n\t\tname: \"qbr_w\"\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tname: \"qbr_b\"\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"PhiR_6\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qr5\"\n\ttop: \"q6r\"\n\tparam {\n\t\tname: \"qr_w\"\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tname: \"qr_b\"\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"QSum_6\"\n\ttype: \"Eltwise\"\n\tbottom: \"qar6\"\n\tbottom: \"qbr6\"\n\tbottom: \"q6r\"\n\ttop: \"qr6un\"\n\teltwise_param { operation: SUM }\n}\nlayer {\n\tname: \"relu_6\"\n\ttype: \"ReLU\"\n\tbottom: \"qr6un\"\n\ttop: \"qr6\"\n}\nlayer {\n\tname: \"PhiA_7\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qa\"\n\ttop: \"qar7\"\n\tparam {\n\t\tname: \"qar_w\"\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tname: \"qar_b\"\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"PhiB_7\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qb\"\n\ttop: \"qbr7\"\n\tparam {\n\t\tname: \"qbr_w\"\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tname: \"qbr_b\"\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"PhiR_7\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qr6\"\n\ttop: \"q7r\"\n\tparam {\n\t\tname: \"qr_w\"\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tname: \"qr_b\"\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"QSum_7\"\n\ttype: \"Eltwise\"\n\tbottom: \"qar7\"\n\tbottom: \"qbr7\"\n\tbottom: \"q7r\"\n\ttop: \"qr7un\"\n\teltwise_param { operation: SUM }\n}\nlayer {\n\tname: \"relu_7\"\n\ttype: \"ReLU\"\n\tbottom: \"qr7un\"\n\ttop: \"qr7\"\n}\nlayer {\n\tname: \"PhiA_8\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qa\"\n\ttop: \"qar8\"\n\tparam {\n\t\tname: \"qar_w\"\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tname: \"qar_b\"\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"PhiB_8\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qb\"\n\ttop: \"qbr8\"\n\tparam {\n\t\tname: \"qbr_w\"\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tname: \"qbr_b\"\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"PhiR_8\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qr7\"\n\ttop: \"q8r\"\n\tparam {\n\t\tname: \"qr_w\"\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tname: \"qr_b\"\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"QSum_8\"\n\ttype: \"Eltwise\"\n\tbottom: \"qar8\"\n\tbottom: \"qbr8\"\n\tbottom: \"q8r\"\n\ttop: \"qr8\"\n\teltwise_param { operation: SUM }\n}\nlayer {\n\tname: \"softmax\"\n\ttype: \"Softmax\"\n\tbottom: \"qr8\"\n\ttop: \"pred\"\n}\n"
  },
  {
    "path": "prototxts/test_drnet_8units_softmax.prototxt",
    "content": "input: \"qa\"\ninput_shape {\t\n\tdim: 1\n\tdim: 100\n}\n\ninput: \"qb\"\ninput_shape {\n\tdim: 1\n\tdim: 100\n}\n\ninput: \"im\"\ninput_shape {\n\tdim: 1\n\tdim: 3\n\tdim: 224\n\tdim: 224\n}\n\ninput: \"posdata\"\ninput_shape {\n\tdim: 1\n\tdim: 2\n\tdim: 32\n\tdim: 32\n}\n\n# Appearance Subnet \nlayer {\n\tname: \"conv1_1\"\n\ttype: \"Convolution\"\n\tbottom: \"im\"\n\ttop: \"conv1_1\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t} \n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tconvolution_param {\n\t\tnum_output: 64\n\t\tpad: 1\n\t\tkernel_size: 3\n\t}\n}\nlayer {\n\tname: \"relu1_1\"\n\ttype: \"ReLU\"\n\tbottom: \"conv1_1\"\n\ttop: \"conv1_1\"\n}\nlayer {\n\tname: \"conv1_2\"\n\ttype: \"Convolution\"\n\tbottom: \"conv1_1\"\n\ttop: \"conv1_2\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tconvolution_param {\n\t\tnum_output: 64\n\t\tpad: 1\n\t\tkernel_size: 3\n\t}\n}\nlayer {\n\tname: \"relu1_2\"\n\ttype: \"ReLU\"\n\tbottom: \"conv1_2\"\n\ttop: \"conv1_2\"\n}\nlayer {\n\tname: \"pool1\"\n\ttype: \"Pooling\"\n\tbottom: \"conv1_2\"\n\ttop: \"pool1\"\n\tpooling_param {\n\t\tpool: MAX\n\t\tkernel_size: 2\n\t\tstride: 2\n\t}\n}\nlayer {\n\tname: \"conv2_1\"\n\ttype: \"Convolution\"\n\tbottom: \"pool1\"\n\ttop: \"conv2_1\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tconvolution_param {\n\t\tnum_output: 128\n\t\tpad: 1\n\t\tkernel_size: 3\n\t}\n}\nlayer {\n\tname: \"relu2_1\"\n\ttype: \"ReLU\"\n\tbottom: \"conv2_1\"\n\ttop: \"conv2_1\"\n}\nlayer {\n\tname: \"conv2_2\"\n\ttype: \"Convolution\"\n\tbottom: \"conv2_1\"\n\ttop: \"conv2_2\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tconvolution_param {\n\t\tnum_output: 128\n\t\tpad: 1\n\t\tkernel_size: 3\n\t}\n}\nlayer {\n\tname: \"relu2_2\"\n\ttype: \"ReLU\"\n\tbottom: \"conv2_2\"\n\ttop: \"conv2_2\"\n}\nlayer {\n\tname: \"pool2\"\t\n\ttype: \"Pooling\"\n\tbottom: \"conv2_2\"\n\ttop: \"pool2\"\n\tpooling_param {\n\t\tpool: MAX\n\t\tkernel_size: 2\n\t\tstride: 2\n\t}\n}\nlayer {\n\tname: \"conv3_1\"\n\ttype: \"Convolution\"\n\tbottom: \"pool2\"\n\ttop: \"conv3_1\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tconvolution_param {\n\t\tnum_output: 256\n\t\tpad: 1\n\t\tkernel_size: 3\n\t}\n}\nlayer {\n\tname: \"relu3_1\"\n\ttype: \"ReLU\"\n\tbottom: \"conv3_1\"\n\ttop: \"conv3_1\"\n}\nlayer {\n\tname: \"conv3_2\"\n\ttype: \"Convolution\"\n\tbottom: \"conv3_1\"\n\ttop: \"conv3_2\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tconvolution_param {\n\t\tnum_output: 256\n\t\tpad: 1\n\t\tkernel_size: 3\n\t}\n}\nlayer {\n\tname: \"relu3_2\"\n\ttype: \"ReLU\"\n\tbottom: \"conv3_2\"\n\ttop: \"conv3_2\"\n}\nlayer {\n\tname: \"conv3_3\"\n\ttype: \"Convolution\"\n\tbottom: \"conv3_2\"\n\ttop: \"conv3_3\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tconvolution_param {\n\t\tnum_output: 256\n\t\tpad: 1\n\t\tkernel_size: 3\n\t}\n}\nlayer {\n\tname: \"relu3_3\"\n\ttype: \"ReLU\"\n\tbottom: \"conv3_3\"\n\ttop: \"conv3_3\"\n}\nlayer {\n\tname: \"pool3\"\n\ttype: \"Pooling\"\n\tbottom: \"conv3_3\"\n\ttop: \"pool3\"\n\tpooling_param {\n\t\tpool: MAX\n\t\tkernel_size: 2\n\t\tstride: 2\n\t}\n}\nlayer {\n\tname: \"conv4_1\"\n\ttype: \"Convolution\"\n\tbottom: \"pool3\"\n\ttop: \"conv4_1\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tconvolution_param {\n\t\tnum_output: 512\n\t\tpad: 1\n\t\tkernel_size: 3\n\t}\n}\nlayer {\n\tname: \"relu4_1\"\n\ttype: \"ReLU\"\n\tbottom: \"conv4_1\"\n\ttop: \"conv4_1\"\n}\nlayer {\n\tname: \"conv4_2\"\n\ttype: \"Convolution\"\n\tbottom: \"conv4_1\"\n\ttop: \"conv4_2\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tconvolution_param {\n\t\tnum_output: 512\n\t\tpad: 1\n\t\tkernel_size: 3\n\t}\n}\nlayer {\n\tname: \"relu4_2\"\n\ttype: \"ReLU\"\n\tbottom: \"conv4_2\"\n\ttop: \"conv4_2\"\n}\nlayer {\n\tname: \"conv4_3\"\n\ttype: \"Convolution\"\n\tbottom: \"conv4_2\"\n\ttop: \"conv4_3\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tconvolution_param {\n\t\tnum_output: 512\n\t\tpad: 1\n\t\tkernel_size: 3\n\t}\n}\nlayer {\n\tname: \"relu4_3\"\n\ttype: \"ReLU\"\n\tbottom: \"conv4_3\"\n\ttop: \"conv4_3\"\n}\nlayer {\n\tname: \"pool4\"\n\ttype: \"Pooling\"\n\tbottom: \"conv4_3\"\n\ttop: \"pool4\"\n\tpooling_param {\n\t\tpool: MAX\n\t\tkernel_size: 2\n\t\tstride: 2\n\t}\n}\nlayer {\n\tname: \"conv5_1\"\n\ttype: \"Convolution\"\n\tbottom: \"pool4\"\n\ttop: \"conv5_1\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tconvolution_param {\n\t\tnum_output: 512\n\t\tpad: 1\n\t\tkernel_size: 3\n\t}\n}\nlayer {\n\tname: \"relu5_1\"\n\ttype: \"ReLU\"\n\tbottom: \"conv5_1\"\n\ttop: \"conv5_1\"\n}\nlayer {\n\tname: \"conv5_2\"\n\ttype: \"Convolution\"\n\tbottom: \"conv5_1\"\n\ttop: \"conv5_2\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tconvolution_param {\n\t\tnum_output: 512\n\t\tpad: 1\n\t\tkernel_size: 3\n\t}\n}\nlayer {\n\tname: \"relu5_2\"\n\ttype: \"ReLU\"\n\tbottom: \"conv5_2\"\n\ttop: \"conv5_2\"\n}\nlayer {\n\tname: \"conv5_3\"\n\ttype: \"Convolution\"\n\tbottom: \"conv5_2\"\n\ttop: \"conv5_3\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tconvolution_param {\n\t\tnum_output: 512\n\t\tpad: 1\n\t\tkernel_size: 3\n\t}\n}\nlayer {\n\tname: \"relu5_3\"\n\ttype: \"ReLU\"\n\tbottom: \"conv5_3\"\n\ttop: \"conv5_3\"\n}\nlayer {\n\tname: \"pool5\"\n\ttype: \"Pooling\"\n\tbottom: \"conv5_3\"\n\ttop: \"pool5\"\n\tpooling_param {\n\t\tpool: MAX\n\t\tkernel_size: 2\n\t\tstride: 2\n\t}\n}\nlayer {\n\tname: \"fc6\"\n\ttype: \"InnerProduct\"\n\tbottom: \"pool5\"\n\ttop: \"fc6\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 4096\n\t}\n}\nlayer {\n\tname: \"relu6\"\n\ttype: \"ReLU\"\n\tbottom: \"fc6\"\n\ttop: \"fc6\"\n}\nlayer {\n\tname: \"fc7\"\n\ttype: \"InnerProduct\"\n\tbottom: \"fc6\"\n\ttop: \"fc7\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 4096\n\t}\n}\nlayer {\n\tname: \"relu7\"\n\ttype: \"ReLU\"\n\tbottom: \"fc7\"\n\ttop: \"fc7\"\n}\nlayer {\n\tname: \"fc8\"\n\ttype: \"InnerProduct\"\n\tbottom: \"fc7\"\n\ttop: \"fc8\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 256\n\t\tweight_filler {\n\t\t\ttype: \"xavier\"\n\t\t}\n\t\tbias_filler {\n\t\t\ttype: \"constant\"\n\t\t\tvalue: 0\n\t\t}\n\t}\n}\nlayer {\n\tname: \"relu8\"\n\ttype: \"ReLU\"\n\tbottom: \"fc8\"\n\ttop: \"fc8\"\n}\n\n# Spatial Cfg Subnet\nlayer {\n\tname: \"conv1_p\"\n\ttype: \"Convolution\"\n\tbottom: \"posdata\"\n\ttop: \"conv1_p\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tconvolution_param {\n\t\tnum_output: 96\n\t\tkernel_size: 5\n\t\tpad: 2\n\t\tstride: 2\n\t}\n}\nlayer {\n\tname: \"relu1_p\"\n\ttype: \"ReLU\"\n\tbottom: \"conv1_p\"\n\ttop: \"conv1_p\"\n}\nlayer {\n\tname: \"conv2_p\"\n\ttype: \"Convolution\"\n\tbottom: \"conv1_p\"\n\ttop: \"conv2_p\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tconvolution_param {\n\t\tnum_output: 128\n\t\tkernel_size: 5\n\t\tpad: 2\n\t\tstride: 2\n\t}\n}\nlayer {\n\tname: \"conv3_p\"\n\ttype: \"Convolution\"\n\tbottom: \"conv2_p\"\n\ttop: \"conv3_p\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tconvolution_param {\n\t\tnum_output: 64\n\t\tkernel_size: 8\n\t}\n}\nlayer {\n\tname: \"relu3_p\"\n\ttype: \"ReLU\"\n\tbottom: \"conv3_p\"\n\ttop: \"conv3_p\"\n}\n\n# Combine features from subnets\nlayer {\n\tname: \"concat1_c\"\n\ttype: \"Python\"\n\tbottom: \"fc8\"\n\tbottom: \"conv3_p\"\n\ttop: \"concat1_c\"\n\tpython_param {\n\t\tmodule: \"customize_layers.concat_layer\"\n\t\tlayer: \"Layer\"\n\t}\n}\nlayer {\n\tname: \"fc2_c\"\n\ttype: \"InnerProduct\"\n\tbottom: \"concat1_c\"\n\ttop: \"fc2_c\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 128\n\t}\n}\nlayer {\n\tname: \"relu2_c\"\n\ttype: \"ReLU\"\n\tbottom: \"fc2_c\"\n\ttop: \"fc2_c\"\n}\nlayer {\n\tname: \"PhiR_0\"\n\ttype: \"InnerProduct\"\n\tbottom: \"fc2_c\"\n\ttop: \"q0r\"\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tparam {\n\t\tlr_mult: 0\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t\tweight_filler {\n\t\t\ttype: \"xavier\"\n\t\t}\n\t\tbias_filler {\n\t\t\ttype: \"constant\"\n\t\t\tvalue: 0\n\t\t}\n\t}\n}\nlayer {\n\tname: \"Softmax_0\"\n\ttype: \"Softmax\"\n\tbottom: \"q0r\"\n\ttop: \"qr0\"\n}\n\n#DR-Net\nlayer {\n\tname: \"PhiA_1\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qa\"\n\ttop: \"qar1\"\n\tparam {\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t\tweight_filler {\n\t\t\ttype: \"xavier\"\n\t\t}\n\t\tbias_filler {\n\t\t\ttype: \"constant\"\n\t\t\tvalue: 0\n\t\t}\n\t}\n}\nlayer {\n\tname: \"PhiB_1\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qb\"\n\ttop: \"qbr1\"\n\tparam {\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t\tweight_filler {\n\t\t\ttype: \"xavier\"\n\t\t}\n\t\tbias_filler {\n\t\t\ttype: \"constant\"\n\t\t\tvalue: 0\n\t\t}\n\t}\n}\nlayer {\n\tname: \"PhiR_1\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qr0\"\n\ttop: \"q1r\"\n\tparam {\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\t\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t\tweight_filler {\n\t\t\ttype: \"xavier\"\n\t\t}\n\t\tbias_filler {\n\t\t\ttype: \"constant\"\n\t\t\tvalue: 0\n\t\t}\n\t}\n}\nlayer {\n\tname: \"QSum_1\"\n\ttype: \"Eltwise\"\n\tbottom: \"qar1\"\n\tbottom: \"qbr1\"\n\tbottom: \"q1r\"\n\ttop: \"qr1un\"\n\teltwise_param { operation: SUM }\n}\nlayer {\n\tname: \"Softmax_1\"\n\ttype: \"Softmax\"\n\tbottom: \"qr1un\"\n\ttop: \"qr1\"\n}\nlayer {\n\tname: \"PhiA_2\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qa\"\n\ttop: \"qar2\"\n\tparam {\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"PhiB_2\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qb\"\n\ttop: \"qbr2\"\n\tparam {\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"PhiR_2\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qr1\"\n\ttop: \"q2r\"\n\tparam {\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"QSum_2\"\n\ttype: \"Eltwise\"\n\tbottom: \"qar2\"\n\tbottom: \"qbr2\"\n\tbottom: \"q2r\"\n\ttop: \"qr2un\"\n\teltwise_param { operation: SUM }\n}\nlayer {\n\tname: \"Softmax_2\"\n\ttype: \"Softmax\"\n\tbottom: \"qr2un\"\n\ttop: \"qr2\"\n}\nlayer {\n\tname: \"PhiA_3\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qa\"\n\ttop: \"qar3\"\n\tparam {\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"PhiB_3\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qb\"\n\ttop: \"qbr3\"\n\tparam {\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"PhiR_3\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qr2\"\n\ttop: \"q3r\"\n\tparam {\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"QSum_3\"\n\ttype: \"Eltwise\"\n\tbottom: \"qar3\"\n\tbottom: \"qbr3\"\n\tbottom: \"q3r\"\n\ttop: \"qr3un\"\n\teltwise_param { operation: SUM }\n}\nlayer {\n\tname: \"Softmax_3\"\n\ttype: \"Softmax\"\n\tbottom: \"qr3un\"\n\ttop: \"qr3\"\n}\nlayer {\n\tname: \"PhiA_4\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qa\"\n\ttop: \"qar4\"\n\tparam {\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"PhiB_4\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qb\"\n\ttop: \"qbr4\"\n\tparam {\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"PhiR_4\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qr3\"\n\ttop: \"q4r\"\n\tparam {\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"QSum_4\"\n\ttype: \"Eltwise\"\n\tbottom: \"qar4\"\n\tbottom: \"qbr4\"\n\tbottom: \"q4r\"\n\ttop: \"qr4un\"\n\teltwise_param { operation: SUM }\n}\nlayer {\n\tname: \"Softmax_4\"\n\ttype: \"Softmax\"\n\tbottom: \"qr4un\"\n\ttop: \"qr4\"\n}\nlayer {\n\tname: \"PhiA_5\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qa\"\n\ttop: \"qar5\"\n\tparam {\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"PhiB_5\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qb\"\n\ttop: \"qbr5\"\n\tparam {\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"PhiR_5\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qr4\"\n\ttop: \"q5r\"\n\tparam {\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"QSum_5\"\n\ttype: \"Eltwise\"\n\tbottom: \"qar5\"\n\tbottom: \"qbr5\"\n\tbottom: \"q5r\"\n\ttop: \"qr5un\"\n\teltwise_param { operation: SUM }\n}\nlayer {\n\tname: \"Softmax_5\"\n\ttype: \"Softmax\"\n\tbottom: \"qr5un\"\n\ttop: \"qr5\"\n}\n\nlayer {\n\tname: \"PhiA_6\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qa\"\n\ttop: \"qar6\"\n\tparam {\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"PhiB_6\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qb\"\n\ttop: \"qbr6\"\n\tparam {\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"PhiR_6\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qr5\"\n\ttop: \"q6r\"\n\tparam {\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"QSum_6\"\n\ttype: \"Eltwise\"\n\tbottom: \"qar6\"\n\tbottom: \"qbr6\"\n\tbottom: \"q6r\"\n\ttop: \"qr6un\"\n\teltwise_param { operation: SUM }\n}\nlayer {\n\tname: \"Softmax_6\"\n\ttype: \"Softmax\"\n\tbottom: \"qr6un\"\n\ttop: \"qr6\"\n}\nlayer {\n\tname: \"PhiA_7\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qa\"\n\ttop: \"qar7\"\n\tparam {\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"PhiB_7\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qb\"\n\ttop: \"qbr7\"\n\tparam {\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"PhiR_7\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qr6\"\n\ttop: \"q7r\"\n\tparam {\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"QSum_7\"\n\ttype: \"Eltwise\"\n\tbottom: \"qar7\"\n\tbottom: \"qbr7\"\n\tbottom: \"q7r\"\n\ttop: \"qr7un\"\n\teltwise_param { operation: SUM }\n}\nlayer {\n\tname: \"Softmax_7\"\n\ttype: \"Softmax\"\n\tbottom: \"qr7un\"\n\ttop: \"qr7\"\n}\nlayer {\n\tname: \"PhiA_8\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qa\"\n\ttop: \"qar8\"\n\tparam {\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"PhiB_8\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qb\"\n\ttop: \"qbr8\"\n\tparam {\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"PhiR_8\"\n\ttype: \"InnerProduct\"\n\tbottom: \"qr7\"\n\ttop: \"q8r\"\n\tparam {\n\t\tlr_mult: 1\n\t\tdecay_mult: 1\n\t}\n\tparam {\n\t\tlr_mult: 2\n\t\tdecay_mult: 0\n\t}\n\tinner_product_param {\n\t\tnum_output: 70\n\t}\n}\nlayer {\n\tname: \"QSum_8\"\n\ttype: \"Eltwise\"\n\tbottom: \"qar8\"\n\tbottom: \"qbr8\"\n\tbottom: \"q8r\"\n\ttop: \"qr8\"\n\teltwise_param { operation: SUM }\n}\nlayer {\n\tname: \"softmax\"\n\ttype: \"Softmax\"\n\tbottom: \"qr8\"\n\ttop: \"pred\"\n}\n"
  },
  {
    "path": "snapshots/README.md",
    "content": "## Download Links\n\n* [drnet_8units_softmax](https://drive.google.com/file/d/0B5RJWjAhdT04Y2pJelpxSEhYWTQ/view?usp=sharing)\n\n* [drnet_8units_linear_shareweight](https://drive.google.com/file/d/0B5RJWjAhdT04X24xWFY1aHhLY0U/view?usp=sharing)\n\n* [drnet_8units_relu_shareweight](https://drive.google.com/file/d/0B5RJWjAhdT04MTVvanRHcWtlODg/view?usp=sharing)\n"
  },
  {
    "path": "tools/_init_paths.py",
    "content": "import os.path as osp\nimport sys\n\ndef add_path(path):\n    if path not in sys.path:\n        sys.path.insert(0, path)\n\nthis_dir = osp.dirname(__file__)\n\n# Add caffe to PYTHONPATH\ncaffe_path = osp.join(this_dir, '..', '..', '..', 'caffes', 'caffe-newest', 'python')\nadd_path(caffe_path)\n\n# Add lib to PYTHONPATH\nlib_path = osp.join(this_dir, '..', 'lib')\nadd_path(lib_path)\n"
  },
  {
    "path": "tools/eval_triplet_recall.py",
    "content": "#!/usr/bin/env python\n\nimport _init_paths\nimport argparse\n\nimport time, os, sys\nimport json\nimport cv2\nimport cPickle as cp\nimport numpy as np\nimport math\nfrom utils.eval_utils import computeIoU\n\ndef parse_args():\n\t\"\"\"\n\tParse input arguments\n\t\"\"\"\n\tparser = argparse.ArgumentParser()\n\t# gt file format: [ gt_label, gt_box ]\n\t# \tgt_label: list [ gt_label(image_i) for image_i in images ]\n\t# \t\tgt_label(image_i): numpy.array of size: num_instance x 3\n\t# \t\t\tinstance: [ label_s, label_r, label_o ]\n\t# \tgt_box: list [ gt_box(image_i) for image_i in images ]\n\t#\t\tgt_box(image_i): numpy.array of size: num_instance x 2 x 4\n\t#\t\t\tinstance: [ [x1_s, y1_s, x2_s, y2_s], \n\t#\t\t\t\t    [x1_o, y1_o, x2_o, y2_o]]\n\tparser.add_argument('--gt_file', dest='gt_file',\n\t\t\t\t\t\thelp='file containing gts',\n\t\t\t\t\t\tdefault=None, type=str)\n\tparser.add_argument('--num_dets', dest='num_dets',\n\t\t\t\t\t\thelp='max number of detections per image',\n\t\t\t\t\t\tdefault=50, type=int)\n\t# det file format: [ det_label, det_box ]\n\t# \tdet_label: list [ det_label(image_i) for image_i in images ]\n\t# \t\tdet_label(image_i): numpy.array of size: num_instance x 6\n\t# \t\t\tinstance: [ prob_s, prob_r, prob_o, label_s, label_r, label_o ]\n\t# \tdet_box: list [ det_box(image_i) for image_i in images ]\n\t#\t\tdet_box(image_i): numpy.array of size: num_instance x 2 x 4\n\t#\t\t\tinstance: [ [x1_s, y1_s, x2_s, y2_s], \n\t#\t\t\t\t    [x1_o, y1_o, x2_o, y2_o]]\n\tparser.add_argument('--det_file', dest='det_file', \n\t\t\t\t\t\thelp='file containing triplet detections',\n\t\t\t\t\t\tdefault=None, type=str)\n\t\n\tparser.add_argument('--min_overlap', dest='ov_thresh',\n\t\t\t\t\t\thelp='minimum overlap for a correct detection',\n\t\t\t\t\t\tdefault=0.5, type=float)\n\n\n\tif len(sys.argv) == 1:\n\t\tparser.print_help()\n\t\tsys.exit(1)\n\n\targs = parser.parse_args()\n\treturn args\n\ndef computeOverlap(detBBs, gtBBs):\n\taIoU = computeIoU(detBBs[0, :], gtBBs[0, :])\n\tbIoU = computeIoU(detBBs[1, :], gtBBs[1, :])\n\treturn min(aIoU, bIoU)\t\t\n\ndef eval_recall(args):\n\tf = open(args.det_file, \"r\")\n\tdets, det_bboxes = cp.load(f)\n\tf.close()\n\tf = open(args.gt_file, \"r\")\n\tall_gts, all_gt_bboxes = cp.load(f)\n\tf.close()\n\tnum_img = len(dets)\n\ttp = []\n\tfp = []\n\tscore = []\n\ttotal_num_gts = 0\n\tfor i in xrange(num_img):\n\t\tgts = all_gts[i]\n\t\tgt_bboxes = all_gt_bboxes[i]\n\t\tnum_gts = gts.shape[0]\n\t\ttotal_num_gts += num_gts\n\t\tgt_detected = np.zeros(num_gts)\n\t\tif isinstance(dets[i], np.ndarray) and dets[i].shape[0] > 0:\n\t\t\tdet_score = np.log(dets[i][:, 0]) + np.log(dets[i][:, 1]) + np.log(dets[i][:, 2])\n\t\t\tinds = np.argsort(det_score)[::-1]\n\t\t\tif args.num_dets > 0 and args.num_dets < len(inds):\n\t\t\t\tinds = inds[:args.num_dets]\n\t\t\ttop_dets = dets[i][inds, 3:]\n\t\t\ttop_scores = det_score[inds]\n\t\t\ttop_det_bboxes = det_bboxes[i][inds, :]\n\t\t\tnum_dets = len(inds)\n\t\t\tfor j in xrange(num_dets):\n\t\t\t\tov_max = 0\n\t\t\t\targ_max = -1\n\t\t\t\tfor k in xrange(num_gts):\n\t\t\t\t\tif gt_detected[k] == 0 and top_dets[j, 0] == gts[k, 0] and top_dets[j, 1] == gts[k, 1] and top_dets[j, 2] == gts[k, 2]:\n\t\t\t\t\t\tov = computeOverlap(top_det_bboxes[j, :, :], gt_bboxes[k, :, :])\n\t\t\t\t\t\tif ov >= args.ov_thresh and ov > ov_max:\n\t\t\t\t\t\t\tov_max = ov\n\t\t\t\t\t\t\targ_max = k\n\t\t\t\tif arg_max != -1:\n\t\t\t\t\tgt_detected[arg_max] = 1\n\t\t\t\t\ttp.append(1)\n\t\t\t\t\tfp.append(0)\n\t\t\t\telse:\n\t\t\t\t\ttp.append(0)\n\t\t\t\t\tfp.append(1)\n\t\t\t\tscore.append(top_scores[j])\n\tscore = np.array(score)\n\ttp = np.array(tp)\n\tfp = np.array(fp)\n\tinds = np.argsort(score)\n\tinds = inds[::-1]\n\ttp = tp[inds]\n\tfp = fp[inds]\n\ttp = np.cumsum(tp)\n\tfp = np.cumsum(fp)\n\trecall = (tp + 0.0) / total_num_gts\n\ttop_recall = recall[-1] \n\tprint top_recall\t\t\t\t\t\t\n\nif __name__ == '__main__':\n\targs = parse_args()\n\n\tprint('Called with args:')\n\tprint(args)\n\n\teval_recall(args)\n\t\n"
  },
  {
    "path": "tools/eval_union_recall.py",
    "content": "#!/usr/bin/env python\n\nimport _init_paths\nimport argparse\n\nimport time, os, sys\nimport json\nimport cv2\nimport cPickle as cp\nimport numpy as np\nimport math\nfrom utils.eval_utils import computeIoU\n\ndef parse_args():\n\t\"\"\"\n\tParse input arguments\n\t\"\"\"\n\tparser = argparse.ArgumentParser()\n\t# gt file format: [ gt_label, gt_box ]\n\t# \tgt_label: list [ gt_label(image_i) for image_i in images ]\n\t# \t\tgt_label(image_i): numpy.array of size: num_instance x 3\n\t# \t\t\tinstance: [ label_s, label_r, label_o ]\n\t# \tgt_box: list [ gt_box(image_i) for image_i in images ]\n\t#\t\tgt_box(image_i): numpy.array of size: num_instance x 2 x 4\n\t#\t\t\tinstance: [ [x1_s, y1_s, x2_s, y2_s], \n\t#\t\t\t\t    [x1_o, y1_o, x2_o, y2_o]]\n\tparser.add_argument('--gt_file', dest='gt_file',\n\t\t\t\t\t\thelp='file containing gts',\n\t\t\t\t\t\tdefault=None, type=str)\n\tparser.add_argument('--num_dets', dest='num_dets',\n\t\t\t\t\t\thelp='max number of detections per image',\n\t\t\t\t\t\tdefault=50, type=int)\n\t# det file format: [ det_label, det_box ]\n\t# \tdet_label: list [ det_label(image_i) for image_i in images ]\n\t# \t\tdet_label(image_i): numpy.array of size: num_instance x 6\n\t# \t\t\tinstance: [ prob_s, prob_r, prob_o, label_s, label_r, label_o ]\n\t# \tdet_box: list [ det_box(image_i) for image_i in images ]\n\t#\t\tdet_box(image_i): numpy.array of size: num_instance x 2 x 4\n\t#\t\t\tinstance: [ [x1_s, y1_s, x2_s, y2_s], \n\t#\t\t\t\t    [x1_o, y1_o, x2_o, y2_o]]\n\tparser.add_argument('--det_file', dest='det_file', \n\t\t\t\t\t\thelp='file containing triplet detections',\n\t\t\t\t\t\tdefault=None, type=str)\n\t\n\tparser.add_argument('--min_overlap', dest='ov_thresh',\n\t\t\t\t\t\thelp='minimum overlap for a correct detection',\n\t\t\t\t\t\tdefault=0.5, type=float)\n\n\n\tif len(sys.argv) == 1:\n\t\tparser.print_help()\n\t\tsys.exit(1)\n\n\targs = parser.parse_args()\n\treturn args\n\ndef getUnionBB(aBB, bBB):\n\treturn [min(aBB[0], bBB[0]), \\\n\t\tmin(aBB[1], bBB[1]), \\\n\t\tmax(aBB[2], bBB[2]), \\\n\t\tmax(aBB[3], bBB[3])]\t\n\ndef computeOverlap(detBBs, gtBBs):\n\taIoU = computeIoU(detBBs[0, :], gtBBs[0, :])\n\tbIoU = computeIoU(detBBs[1, :], gtBBs[1, :])\n\treturn min(aIoU, bIoU)\t\t\n\ndef eval_recall(args):\n\tf = open(args.det_file, \"r\")\n\tdets, det_bboxes = cp.load(f)\n\tf.close()\n\tf = open(args.gt_file, \"r\")\n\tall_gts, all_gt_bboxes = cp.load(f)\n\tf.close()\n\tnum_img = len(dets)\n\ttp = []\n\tfp = []\n\tscore = []\n\ttotal_num_gts = 0\n\tfor i in xrange(num_img):\n\t\tgts = all_gts[i]\n\t\tgt_bboxes = all_gt_bboxes[i]\n\t\tgt_ubbs = [] \t\n\t\tnum_gts = gts.shape[0]\n\t\tfor j in xrange(num_gts):\n\t\t\tgt_ubbs.append(getUnionBB(gt_bboxes[j, 0, :], gt_bboxes[j, 1, :]))\n\t\ttotal_num_gts += num_gts\n\t\tgt_detected = np.zeros(num_gts)\n\t\tif isinstance(dets[i], np.ndarray) and dets[i].shape[0] > 0:\n\t\t\tdet_score = np.log(dets[i][:, 0]) + np.log(dets[i][:, 1]) + np.log(dets[i][:, 2])\n\t\t\tinds = np.argsort(det_score)[::-1]\n\t\t\tif args.num_dets > 0 and args.num_dets < len(inds):\n\t\t\t\tinds = inds[:args.num_dets]\n\t\t\ttop_dets = dets[i][inds, 3:]\n\t\t\ttop_scores = det_score[inds]\n\t\t\ttop_det_bboxes = det_bboxes[i][inds, :]\n\t\t\ttop_det_ubbs = []\n\t\t\tnum_dets = len(inds)\n\t\t\tfor j in xrange(num_dets):\n\t\t\t\ttop_det_ubbs.append(getUnionBB(top_det_bboxes[j, 0, :], top_det_bboxes[j, 1, :]))\n\t\t\tfor j in xrange(num_dets):\n\t\t\t\tov_max = 0\n\t\t\t\targ_max = -1\n\t\t\t\tfor k in xrange(num_gts):\n\t\t\t\t\tif gt_detected[k] == 0 and top_dets[j, 0] == gts[k, 0] and top_dets[j, 1] == gts[k, 1] and top_dets[j, 2] == gts[k, 2]:\n\t\t\t\t\t\tov = computeIoU(top_det_ubbs[j], gt_ubbs[k])\n\t\t\t\t\t\tif ov >= args.ov_thresh and ov > ov_max:\n\t\t\t\t\t\t\tov_max = ov\n\t\t\t\t\t\t\targ_max = k\n\t\t\t\tif arg_max != -1:\n\t\t\t\t\tgt_detected[arg_max] = 1\n\t\t\t\t\ttp.append(1)\n\t\t\t\t\tfp.append(0)\n\t\t\t\telse:\n\t\t\t\t\ttp.append(0)\n\t\t\t\t\tfp.append(1)\n\t\t\t\tscore.append(top_scores[j])\n\tscore = np.array(score)\n\ttp = np.array(tp)\n\tfp = np.array(fp)\n\tinds = np.argsort(score)\n\tinds = inds[::-1]\n\ttp = tp[inds]\n\tfp = fp[inds]\n\ttp = np.cumsum(tp)\n\tfp = np.cumsum(fp)\n\trecall = (tp + 0.0) / total_num_gts\n\ttop_recall = recall[-1] \n\tprint top_recall\t\t\t\t\t\t\n\nif __name__ == '__main__':\n\targs = parse_args()\n\n\tprint('Called with args:')\n\tprint(args)\n\n\teval_recall(args)\n\t\n"
  },
  {
    "path": "tools/prepare_data.py",
    "content": "import os.path as osp\nimport mat4py as mp\nimport cv2\nimport numpy as np\nimport math\nimport json\n\nclass DataLoader:\n\tdef __init__(self, datasetRoot, split):\n\t\tself._root = datasetRoot\n\t\tself._split = split\n\t\tself._loadLabels()\n\t\tself._loadAnnotation(split)\n\n\tdef _loadLabels(self):\n\t\tmat = mp.loadmat(osp.join(self._root, \"predicate.mat\"))\n\t\tself._relList = mat[\"predicate\"]\n\t\tself._numRelClass = len(self._relList)\n\t\tself._relMapping = {}\n\t\tfor i in xrange(len(self._relList)):\n\t\t\tself._relMapping[self._relList[i]] = i\t\n\t\tmat = mp.loadmat(osp.join(self._root, \"objectListN.mat\"))\n\t\tself._objList = mat[\"objectListN\"]\n\t\tself._numObjClass = len(self._objList) + 1\n\t\tself._objMapping = {}\n\t\tself._objMapping[\"__BG\"] = 0\n\t\tfor i in xrange(len(self._objList)):\n\t\t\tself._objMapping[self._objList[i]] = i + 1\t\n\n\tdef _loadAnnotation(self, split):\n\t\tmat = mp.loadmat(osp.join(self._root, \"annotation_\" + split + \".mat\"))\n\t\tself._annotations = mat[\"annotation_\" + split]\n\n\tdef _getNumImgs(self):\n\t\treturn len(self._annotations)\n\n\tdef _getImPath(self, idx):\n\t\treturn osp.join(self._root, \"images\", self._split, self._annotations[idx][\"filename\"])\n\t\n\tdef _getNumRel(self):\n\t\tnumRels = 0\n\t\tn = self._getNumImgs()\n\t\tfor i in xrange(n):\n\t\t\trels = self._getRels(i)\n\t\t\tnumRels += len(rels)\t\n\t\treturn numRels\n\n\tdef _getRels(self, idx):\n\t\tif \"relationship\" in self._annotations[idx]:\n\t\t\trels = self._annotations[idx][\"relationship\"] \n\t\t\tif isinstance(rels, dict):\n\t\t\t\trels = [rels]\t\n\t\t\treturn rels\n\t\telse:\n\t\t\treturn []\n\n\tdef _outputDB(self, type, data):\n\t\tjson.dump(data, open(type + self._split + \".json\", \"w\"))\n\n\tdef _bboxTransform(self, bbox, ih, iw): #[x1, y1, x2, y2]\n\t\treturn [max(bbox[2], 0), max(bbox[0], 0), min(bbox[3] + 1, iw), min(bbox[1] + 1, ih)]\n\t\n\tdef _getRelLabel(self, predicate):\n\t\tif not (predicate in self._relMapping):\n\t\t\treturn -1\n\t\treturn self._relMapping[predicate]\n\n\tdef _getObjLabel(self, predicate):\n\t\tif not (predicate in self._objMapping):\n\t\t\treturn -1\n\t\treturn self._objMapping[predicate]\n\n\tdef _getUnionBBox(self, aBB, bBB, ih, iw, margin = 10):\n\t\treturn [max(0, min(aBB[0], bBB[0]) - margin), \\\n\t\t\tmax(0, min(aBB[1], bBB[1]) - margin), \\\n\t\t\tmin(iw, max(aBB[2], bBB[2]) + margin), \\\n\t\t\tmin(ih, max(aBB[3], bBB[3]) + margin)]\n\n\tdef _getRelSamplesSingle(self):\n\t\tn = self._getNumImgs()\n\t\tself._sampleIdx = 0\n\t\tsamples = []\n\t\tfor i in xrange(n):\n\t\t\trels = self._getRels(i)\n\t\t\tif len(rels) == 0:\n\t\t\t\tcontinue\n\t\t\tpath = self._getImPath(i)\n\t\t\tim = cv2.imread(path)\n\t\t\tih = im.shape[0]\t\n\t\t\tiw = im.shape[1]\n\t\t\tfor rel in rels:\n\t\t\t\tphrase = rel[\"phrase\"]\n\t\t\t\trLabel = self._getRelLabel(phrase[1])\n\t\t\t\taLabel = self._getObjLabel(phrase[0])\n\t\t\t\tbLabel = self._getObjLabel(phrase[2])\n\t\t\t\taBBox = self._bboxTransform(rel[\"subBox\"], ih, iw)\n\t\t\t\tbBBox = self._bboxTransform(rel[\"objBox\"], ih, iw)\n\t\t\t\trBBox = self._getUnionBBox(aBBox, bBBox, ih, iw)\t\n\t\t\t\tsamples.append({\"imPath\": path, \"rLabel\": rLabel, \"aLabel\": aLabel, \"bLabel\": bLabel, \"rBBox\": rBBox, \"aBBox\": aBBox, \"bBBox\": bBBox})\n\t\t\t\tself._sampleIdx += 1\t\n\t\t\t\tif self._sampleIdx % 100 == 0:\n\t\t\t\t\tprint self._sampleIdx\n\t\tself._outputDB(\"rel\", samples)\t\n\t\t\nif __name__ == \"__main__\":\n\tloader = DataLoader(\"/datasets/vrd\", \"test\")\n\tloader._getRelSamplesSingle()\n"
  },
  {
    "path": "tools/test_predicate_recognition.py",
    "content": "#!/usr/bin/env python\n\nimport _init_paths\nimport caffe\nimport argparse\nimport time, os, sys\nimport json\nimport cv2\nimport cPickle as cp\nimport numpy as np\nimport math\n\ndef parse_args():\n\t\"\"\"\n\tParse input arguments\n\t\"\"\"\n\tparser = argparse.ArgumentParser()\n\t# image_paths file format: [ path of image_i for image_i in images ]\n\t# the order of images in image_paths should be the same with gt_file\n\tparser.add_argument('--image_paths', dest='image_paths', help='file containing image paths',\n\t\t\t\t\t\tdefault='', type=str)\n\n\tparser.add_argument('--gpu', dest='gpu_id', help='GPU id to use',\n\t\t\t\t\t\tdefault=0, type=int)\n\tparser.add_argument('--def', dest='prototxt',\n\t\t\t\t\t\thelp='prototxt file defining the network',\n\t\t\t\t\t\tdefault=None, type=str)\n\tparser.add_argument('--net', dest='caffemodel',\n\t\t\t\t\t\thelp='model to test',\n\t\t\t\t\t\tdefault=None, type=str)\n\t# gt file format: [ gt_label, gt_box ]\n\t# \tgt_label: list [ gt_label(image_i) for image_i in images ]\n\t# \t\tgt_label(image_i): numpy.array of size: num_instance x 3\n\t# \t\t\tinstance: [ label_s, label_r, label_o ]\n\t# \tgt_box: list [ gt_box(image_i) for image_i in images ]\n\t#\t\tgt_box(image_i): numpy.array of size: num_instance x 2 x 4\n\t#\t\t\tinstance: [ [x1_s, y1_s, x2_s, y2_s], \n\t#\t\t\t\t    [x1_o, y1_o, x2_o, y2_o]]\n\tparser.add_argument('--gt_file', dest='gt_file', \n\t\t\t\t\t\thelp='file containing ground truth pairs',\n\t\t\t\t\t\tdefault=None, type=str)\n\n\tparser.add_argument('--ncls', dest='ncls', help='number of object classes', default=101, type=int)\n\n\tparser.add_argument('--input_type', dest='type',\n\t\t\t\t\t\thelp='type of input sets',\n\t\t\t\t\t\tdefault=0, type=int)\n\n\tparser.add_argument('--out', dest='out', help='name of output file', default='', type=str)\n\n\tif len(sys.argv) == 1:\n\t\tparser.print_help()\n\t\tsys.exit(1)\n\n\targs = parser.parse_args()\n\treturn args\n\n\ndef getUnionBBox(aBB, bBB, ih, iw):\n\tmargin = 10\n\treturn [max(0, min(aBB[0], bBB[0]) - margin), \\\n\t\tmax(0, min(aBB[1], bBB[1]) - margin), \\\n\t\tmin(iw, max(aBB[2], bBB[2]) + margin), \\\n\t\tmin(ih, max(aBB[3], bBB[3]) + margin)]\n\ndef getAppr(im, bb):\n\tsubim = im[bb[1] : bb[3], bb[0] : bb[2], :]\n\tsubim = cv2.resize(subim, None, None, 224.0 / subim.shape[1], 224.0 / subim.shape[0], interpolation=cv2.INTER_LINEAR)\n\tpixel_means = np.array([[[103.939, 116.779, 123.68]]])\n\tsubim -= pixel_means\n\tsubim = subim.transpose((2, 0, 1))\n\treturn subim\t\n\ndef getDualMask(ih, iw, bb):\n\trh = 32.0 / ih\n\trw = 32.0 / iw\t\n\tx1 = max(0, int(math.floor(bb[0] * rw)))\n\tx2 = min(32, int(math.ceil(bb[2] * rw)))\n\ty1 = max(0, int(math.floor(bb[1] * rh)))\n\ty2 = min(32, int(math.ceil(bb[3] * rh)))\n\tmask = np.zeros((32, 32))\n\tmask[y1 : y2, x1 : x2] = 1\n\tassert(mask.sum() == (y2 - y1) * (x2 - x1))\n\treturn mask\n\ndef forward_batch(net, ims, poses, qas, qbs, args):\n\tforward_args = {}\n\tif args.type != 1:\n\t\tnet.blobs[\"im\"].reshape(*(ims.shape))\n\t\tforward_args[\"im\"] = ims.astype(np.float32, copy=False)\n\tif args.type != 0:\n\t\tnet.blobs[\"posdata\"].reshape(*(poses.shape))\n\t\tforward_args[\"posdata\"] = poses.astype(np.float32, copy=False)\n\tif args.type == 3:\n\t\tnet.blobs[\"qa\"].reshape(*(qas.shape))\n\t\tforward_args[\"qa\"] = qas.astype(np.float32, copy=False)\n\t\tnet.blobs[\"qb\"].reshape(*(qbs.shape))\n\t\tforward_args[\"qb\"] = qbs.astype(np.float32, copy=False)\n\n\tnet_out = net.forward(**forward_args)\n\titr_pred = net_out[\"pred\"].copy()\n\treturn itr_pred\n\ndef test_net(net, image_paths, args):\n\tf = open(args.gt_file, \"r\")\n\tall_gts, all_gt_bboxes = cp.load(f)\n\tf.close()\n\tnum_img = len(image_paths)\n\tnum_class = args.ncls\n\tthresh = 0.05\n\tbatch_size = 20\n\tpred = []\n\tpred_bboxes = []\n\tfor i in xrange(num_img):\n\t\tprint str(i) + \" / \" + str(num_img)\n\t\tim = cv2.imread(image_paths[i]).astype(np.float32, copy=False)\n\t\tih = im.shape[0]\n\t\tiw = im.shape[1]\n\t\tgts = all_gts[i]\n\t\tgt_bboxes = all_gt_bboxes[i]\n\t\tnum_gts = gts.shape[0]\n\t\tpred.append([])\n\t\tpred_bboxes.append([])\n\t\tims = []\n\t\tposes = []\n\t\tqas = []\n\t\tqbs = []\n\t\tfor j in xrange(num_gts):\t\n\t\t\tsub = gt_bboxes[j, 0, :]\n\t\t\tobj = gt_bboxes[j, 1, :]\n\t\t\trBB = getUnionBBox(sub, obj, ih, iw)\n\t\t\trAppr = getAppr(im, rBB)\t\n\t\t\trMask = np.array([getDualMask(ih, iw, sub), getDualMask(ih, iw, obj)])\n\t\t\tims.append(rAppr)\n\t\t\tposes.append(rMask)\n\t\t\tqa = np.zeros(num_class - 1)\n\t\t\tqa[gts[j, 0] - 1] = 1\n\t\t\tqb = np.zeros(num_class - 1)\n\t\t\tqb[gts[j, 2] - 1] = 1\n\t\t\tqas.append(qa)\n\t\t\tqbs.append(qb)\n\t\tif len(ims) == 0:\n\t\t\tcontinue\n\t\tims = np.array(ims)\n\t\tposes = np.array(poses)\n\t\tqas = np.array(qas)\n\t\tqbs = np.array(qbs)\n\t\t_cursor = 0\n\t\titr_pred = None\n\t\tnum_ins = ims.shape[0]\n\t\twhile _cursor < num_ins:\n\t\t\t_end_batch = min(_cursor + batch_size, num_ins)\n\t\t\titr_pred_batch = forward_batch(net, ims[_cursor : _end_batch], poses[_cursor : _end_batch], qas[_cursor : _end_batch], qbs[_cursor : _end_batch], args)\n\t\t\tif itr_pred is None:\n\t\t\t\titr_pred = itr_pred_batch\n\t\t\telse:\n\t\t\t\titr_pred = np.vstack((itr_pred, itr_pred_batch))\n\t\t\t_cursor = _end_batch\n\t\t\n\t\tfor j in xrange(num_gts):\t\n\t\t\tsub = gt_bboxes[j, 0, :]\n\t\t\tobj = gt_bboxes[j, 1, :]\n\t\t\tfor k in xrange(itr_pred.shape[1]):\n\t\t\t\tif itr_pred[j, k] < thresh: \n\t\t\t\t\tcontinue\n\t\t\t\tpred[i].append([itr_pred[j, k], 1, 1, gts[j, 0], k, gts[j, 2]])\n\t\t\t\tpred_bboxes[i].append([sub, obj])\n\t\tpred[i] = np.array(pred[i])\n\t\tpred_bboxes[i] = np.array(pred_bboxes[i])\t\n\tprint \"writing file..\"\n\tf = open(args.out, \"wb\")\n\tcp.dump([pred, pred_bboxes], f, cp.HIGHEST_PROTOCOL)\t\t\t\n\tf.close()\n\nif __name__ == '__main__':\n\targs = parse_args()\n\n\tprint('Called with args:')\n\tprint(args)\n\n\tcaffe.set_mode_gpu()\n\tcaffe.set_device(args.gpu_id)\n\tnet = caffe.Net(args.prototxt, args.caffemodel, caffe.TEST)\n\n\ttest_image_paths = json.load(open(args.image_paths))\n\n\ttest_net(net, test_image_paths, args)\n\t\n"
  },
  {
    "path": "tools/test_triplet_detection.py",
    "content": "#!/usr/bin/env python\n\nimport _init_paths\nimport caffe\nimport argparse\nimport time, os, sys\nimport json\nimport cv2\nimport cPickle as cp\nimport numpy as np\nimport math\n\ndef parse_args():\n\t\"\"\"\n\tParse input arguments\n\t\"\"\"\n\tparser = argparse.ArgumentParser()\n\t# image_paths file format: [ path of image_i for image_i in images ]\n\t# the order of images in image_paths should be the same with obj_dets_file\n\tparser.add_argument('--image_paths', dest='image_paths', help='file containing test dataset',\n\t\t\t\t\t\tdefault='', type=str)\n\tparser.add_argument('--gpu', dest='gpu_id', help='GPU id to use',\n\t\t\t\t\t\tdefault=0, type=int)\n\tparser.add_argument('--def', dest='prototxt',\n\t\t\t\t\t\thelp='prototxt file defining the network',\n\t\t\t\t\t\tdefault=None, type=str)\n\tparser.add_argument('--net', dest='caffemodel',\n\t\t\t\t\t\thelp='model to test',\n\t\t\t\t\t\tdefault=None, type=str)\n\tparser.add_argument('--num_dets', dest='max_det',\n\t\t\t\t\t\thelp='max number of detections per image',\n\t\t\t\t\t\tdefault=100, type=int)\n\t# obj_dets_file format: [ obj_dets of image_i for image_i in images ]\n\t# \tobj_dets: numpy.array of size: num_instance x 5\n\t# \t\tinstance: [x1, y1, x2, y2, prob, label]\n\tparser.add_argument('--obj_dets_file', dest='obj_dets_file', \n\t\t\t\t\t\thelp='file containing object detections',\n\t\t\t\t\t\tdefault=None, type=str)\n\t# type 0: im only\n\t# type 1: pos only\n\t# type 2: im + pos\n\t# type 3: im + pos + qa + qb\n\tparser.add_argument('--input_type', dest='type',\n\t\t\t\t\t\thelp='type of input sets',\n\t\t\t\t\t\tdefault=0, type=int)\n\n\tparser.add_argument('--ncls', dest='num_class', help='number of object classes', default=101, type=int)\n\n\tparser.add_argument('--out', dest='out', help='name of output file', default='', type=str)\n\n\tif len(sys.argv) == 1:\n\t\tparser.print_help()\n\t\tsys.exit(1)\n\n\targs = parser.parse_args()\n\treturn args\n\n\ndef getPred(pred, max_num_det):\n\tif pred.shape[0] == 0:\n\t\treturn pred\n\tinds = np.argsort(pred[:, 4])\n\tinds = inds[::-1]\n\tif len(inds) > max_num_det:\n\t\tinds = inds[:max_num_det]\t\t\t\n\treturn pred[inds, :]\n\n\ndef getUnionBBox(aBB, bBB, ih, iw):\n\tmargin = 10\n\treturn [max(0, min(aBB[0], bBB[0]) - margin), \\\n\t\tmax(0, min(aBB[1], bBB[1]) - margin), \\\n\t\tmin(iw, max(aBB[2], bBB[2]) + margin), \\\n\t\tmin(ih, max(aBB[3], bBB[3]) + margin)]\n\ndef getAppr(im, bb):\n\tsubim = im[bb[1] : bb[3], bb[0] : bb[2], :]\n\tsubim = cv2.resize(subim, None, None, 224.0 / subim.shape[1], 224.0 / subim.shape[0], interpolation=cv2.INTER_LINEAR)\n\tpixel_means = np.array([[[103.939, 116.779, 123.68]]])\n\tsubim -= pixel_means\n\tsubim = subim.transpose((2, 0, 1))\n\treturn subim\t\n\ndef getDualMask(ih, iw, bb):\n\trh = 32.0 / ih\n\trw = 32.0 / iw\t\n\tx1 = max(0, int(math.floor(bb[0] * rw)))\n\tx2 = min(32, int(math.ceil(bb[2] * rw)))\n\ty1 = max(0, int(math.floor(bb[1] * rh)))\n\ty2 = min(32, int(math.ceil(bb[3] * rh)))\n\tmask = np.zeros((32, 32))\n\tmask[y1 : y2, x1 : x2] = 1\n\tassert(mask.sum() == (y2 - y1) * (x2 - x1))\n\treturn mask\n\ndef forward_batch(net, ims, poses, qas, qbs, args):\n\tforward_args = {}\n\tif args.type != 1:\n\t\tnet.blobs[\"im\"].reshape(*(ims.shape))\n\t\tforward_args[\"im\"] = ims.astype(np.float32, copy=False)\n\tif args.type != 0:\n\t\tnet.blobs[\"posdata\"].reshape(*(poses.shape))\n\t\tforward_args[\"posdata\"] = poses.astype(np.float32, copy=False)\n\tif args.type == 3:\t\t\n\t\tnet.blobs[\"qa\"].reshape(*(qas.shape))\n\t\tforward_args[\"qa\"] = qas.astype(np.float32, copy=False)\n\t\tnet.blobs[\"qb\"].reshape(*(qbs.shape))\n\t\tforward_args[\"qb\"] = qbs.astype(np.float32, copy=False)\t\n\tnet_out = net.forward(**forward_args)\n\titr_pred = net_out[\"pred\"].copy()\n\treturn itr_pred\n\ndef test_net(net, image_paths, args):\n\tf = open(args.obj_dets_file, \"r\")\n\tall_dets = cp.load(f)\n\tf.close()\n\tnum_img = len(image_paths)\n\tnum_class = args.num_class\n\tthresh = 0.05\n\tmax_num_det = args.max_det\n\tbatch_size = 30\n\tpred = []\n\tpred_bboxes = []\n\tfor i in xrange(num_img):\n\t\tim = cv2.imread(image_paths[i]).astype(np.float32, copy=False)\n\t\tih = im.shape[0]\n\t\tiw = im.shape[1]\n\t\t\t\n\t\tdets = getPred(all_dets[i], max_num_det)\n\t\tnum_dets = dets.shape[0]\n\t\tpred.append([])\n\t\tpred_bboxes.append([])\n\t\tfor subIdx in xrange(num_dets):\n\t\t\tims = []\n\t\t\tposes = []\n\t\t\tqas = []\n\t\t\tqbs = []\n\t\t\tfor objIdx in xrange(num_dets):\n\t\t\t\tif subIdx != objIdx: \n\t\t\t\t\tsub = dets[subIdx, 0: 4]\n\t\t\t\t\tobj = dets[objIdx, 0: 4]\n\t\t\t\t\trBB = getUnionBBox(sub, obj, ih, iw)\n\t\t\t\t\trAppr = getAppr(im, rBB)\t\n\t\t\t\t\trMask = np.array([getDualMask(ih, iw, sub), getDualMask(ih, iw, obj)])\n\t\t\t\t\tims.append(rAppr)\n\t\t\t\t\tposes.append(rMask)\n\t\t\t\t\tqa = np.zeros(num_class - 1)\n\t\t\t\t\tqa[dets[subIdx, 5] - 1] = 1\n\t\t\t\t\tqb = np.zeros(num_class - 1)\n\t\t\t\t\tqb[dets[objIdx, 5] - 1] = 1\n\t\t\t\t\tqas.append(qa)\n\t\t\t\t\tqbs.append(qb)\n\t\t\tif len(ims) == 0:\n\t\t\t\tbreak\n\t\t\tims = np.array(ims)\n\t\t\tposes = np.array(poses)\n\t\t\tqas = np.array(qas)\n\t\t\tqbs = np.array(qbs)\n\t\t\t_cursor = 0\n\t\t\titr_pred = None\n\t\t\tnum_ins = ims.shape[0]\n\t\t\twhile _cursor < num_ins:\n\t\t\t\t_end_batch = min(_cursor + batch_size, num_ins)\n\t\t\t\titr_pred_batch = forward_batch(net, ims[_cursor : _end_batch] if ims.shape[0] > 0 else None, poses[_cursor : _end_batch] if poses.shape[0] > 0 else None, qas[_cursor : _end_batch] if qas.shape[0] > 0 else None, qbs[_cursor : _end_batch] if qbs.shape[0] > 0 else None, args)\n\t\t\t\tif itr_pred is None:\n\t\t\t\t\titr_pred = itr_pred_batch\n\t\t\t\telse:\n\t\t\t\t\titr_pred = np.vstack((itr_pred, itr_pred_batch))\n\t\t\t\t_cursor = _end_batch\n\n\t\t\t\n\t\t\tcur = 0\n\t\t\tfor objIdx in xrange(num_dets):\t\n\t\t\t\tif subIdx != objIdx:\n\t\t\t\t\tsub = dets[subIdx, 0: 4]\n\t\t\t\t\tobj = dets[objIdx, 0: 4]\n\t\t\t\t\tfor j in xrange(itr_pred.shape[1]):\n\t\t\t\t\t\tif itr_pred[cur, j] < thresh: \n\t\t\t\t\t\t\tcontinue\n\t\t\t\t\t\tpred[i].append([itr_pred[cur, j], dets[subIdx, 4], dets[objIdx, 4], dets[subIdx, 5], j, dets[objIdx, 5]])\n\t\t\t\t\t\tpred_bboxes[i].append([sub, obj])\n\t\t\t\t\tcur += 1\n\t\t\tassert(cur == itr_pred.shape[0])\n\t\tpred[i] = np.array(pred[i])\n\t\tpred_bboxes[i] = np.array(pred_bboxes[i])\t\n\tprint \"writing file..\"\n\tf = open(args.out, \"wb\")\n\tcp.dump([pred, pred_bboxes], f, cp.HIGHEST_PROTOCOL)\t\t\t\n\tf.close()\n\nif __name__ == '__main__':\n\targs = parse_args()\n\n\tprint('Called with args:')\n\tprint(args)\n\n\tcaffe.set_mode_gpu()\n\tcaffe.set_device(args.gpu_id)\n\tnet = caffe.Net(args.prototxt, args.caffemodel, caffe.TEST)\n\n\ttest_image_paths = json.load(open(args.image_paths))\n\n\ttest_net(net, test_image_paths, args)\n\t\n"
  }
]