Repository: ZihaoWang-CV/CAMP_iccv19
Branch: master
Commit: b0ec07908f47
Files: 31
Total size: 46.7 MB
Directory structure:
gitextract_bd4r4mqp/
├── LICENSE
├── README.md
├── data.py
├── embed/
│ ├── cocoword2vec300dim.npy
│ ├── cocoword2vecmask.npy
│ ├── f30kword2vec300dim_3.npy
│ └── f30kword2vecmask_3.npy
├── evaluation.py
├── experiments/
│ ├── f30k_cross_attention/
│ │ ├── config.yaml
│ │ ├── config_256.yaml
│ │ └── config_test.yaml
│ └── f30k_gate_fusion/
│ ├── config_finetune.yaml
│ ├── config_fixatt.yaml
│ └── config_test.yaml
├── fusion_module.py
├── model.py
├── pick_wordvec.py
├── test.py
├── test_modules.py
├── train.py
├── vocab/
│ ├── 10crop_precomp_vocab.pkl
│ ├── CUHK-PEDES_vocab.pkl
│ ├── coco_precomp_vocab.pkl
│ ├── coco_precomp_vocab_1.pkl
│ ├── coco_vocab.pkl
│ ├── f30k_precomp_vocab.pkl
│ ├── f30k_precomp_vocab_1.pkl
│ ├── f30k_vocab.pkl
│ ├── f8k_precomp_vocab.pkl
│ └── f8k_vocab.pkl
└── vocab.py
================================================
FILE CONTENTS
================================================
================================================
FILE: LICENSE
================================================
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
================================================
FILE: README.md
================================================
# Introduction
This repository is for [CAMP: Cross-Modal Adaptive Message Passing for Text-Image Retrieval](https://arxiv.org/abs/1909.05506) from CUHK-SenseTime Joint Lab (appear in ICCV 2019).
It is built on top of the [VSE++](https://github.com/fartashf/vsepp) and [SCAN](https://kuanghuei.github.io/SCANProject/) in PyTorch.
## Requirements and Installation
We recommended the following dependencies.
* Python 3
* [PyTorch](http://pytorch.org/) (>1.0)
* [NumPy](http://www.numpy.org/) (>1.12.1)
* [TensorBoard](https://github.com/TeamHG-Memex/tensorboard_logger)
* [pycocotools](https://github.com/cocodataset/cocoapi)
* [torchvision]()
* [matplotlib]()
* Punkt Sentence Tokenizer:
```python
import nltk
nltk.download()
> d punkt
```
## Download data
Download the dataset files and pre-trained models. We use the same pre-extracted features and splits as [SCAN](https://kuanghuei.github.io/SCANProject/).
The splits are produced by [Andrej Karpathy](http://cs.stanford.edu/people/karpathy/deepimagesent/). The raw images can be downloaded from from their original sources [here](http://nlp.cs.illinois.edu/HockenmaierGroup/Framing_Image_Description/KCCA.html), [here](http://shannon.cs.illinois.edu/DenotationGraph/) and [here](http://mscoco.org/).
The precomputed image features of MS-COCO are from [here](https://github.com/peteanderson80/bottom-up-attention). The precomputed image features of Flickr30K are extracted from the raw Flickr30K images using the bottom-up attention model from [here](https://github.com/peteanderson80/bottom-up-attention).
The pre-extracted image features are from [SCAN](https://kuanghuei.github.io/SCANProject/), produced by [Kuang-Huei Lee](https://kuanghuei.github.io/). The data can be downloaded from:
```bash
wget https://scanproject.blob.core.windows.net/scan-data/data.zip
```
We refer to the path of extracted files for `data.zip` as `./data` directory.
## Training new models
Run `train.py` in the directory of the corresponding config path:
Training the cross-attention model on Flickr30K dataset:
```bash
cd ./experiments/f30k_cross_attention
python python -u ../../train.py --config ./config_256.yaml
```
Training the full CAMP model on Flickr30K dataset:
```bash
cd ./experiments/f30k_gate_fusion
python python -u ../../train.py --config ./config_finetune.yaml
```
**We initialize the network weights from the pretrained cross-attention model to train the full CAMP model. The weights for attention map are fixed for the first several epochs and then we finetune the whole network.**
## Evaluate trained models
Changing the `resume` arguments in the coreesponding config file and running evaluation in the project root directory:
```python
from test_modules import test_CAMP_model
#config_path = "./experiments/f30k_cross_attention/config_test.yaml"
test_CAMP_model(config_path)
```
Pretrained model for Flickr30K could be downloaded [here](https://drive.google.com/drive/folders/1o8rUv78uS_aX4P1hMPELl53cxnZ8UqiF?usp=sharing).
## Reference
If you found this code useful, please cite the following paper:
```
@InProceedings{Wang_2019_ICCV,
author = {Wang, Zihao and Liu, Xihui and Li, Hongsheng and Sheng, Lu and Yan, Junjie and Wang, Xiaogang and Shao, Jing},
title = {CAMP: Cross-Modal Adaptive Message Passing for Text-Image Retrieval},
booktitle = {The IEEE International Conference on Computer Vision (ICCV)},
month = {October},
year = {2019}
}
```
## License
[Apache License 2.0](http://www.apache.org/licenses/LICENSE-2.0)
*I have left CUHK and the email address is deprecated. Please directly open a new issue or contact zihaowang.cv@gmail.com if you have further quetions. Thanks!*
================================================
FILE: data.py
================================================
import torch
import torch.utils.data as data
import torchvision.transforms as transforms
import os
import nltk
from PIL import Image
from pycocotools.coco import COCO
import numpy as np
import json as jsonmod
from random import shuffle, seed, choice, randint
import torch.distributed as dist
from torch.utils.data.distributed import DistributedSampler
import model
from model import max_length
def get_paths(path, name="coco", use_restval=False):
"""
Returns paths to images and annotations for the given datasets. For MSCOCO
indices are also returned to control the data split being used.
The indices are extracted from the Karpathy et al. splits using this
snippet:
>>> import json
>>> dataset=json.load(open("dataset_coco.json","r"))
>>> A=[]
>>> for i in range(len(D["images"])):
... if D["images"][i]["split"] == "val":
... A+=D["images"][i]["sentids"][:5]
...
:param name: Dataset names
:param use_restval: If True, the the `restval` data is included in train.
"""
roots = {}
ids = {}
if "coco" == name:
imgdir = os.path.join(path, "images")
capdir = os.path.join(path, "annotations")
roots["train"] = {
"img": os.path.join(imgdir, "train2014"),
"cap": os.path.join(capdir, "captions_train2014.json")
}
roots["val"] = {
"img": os.path.join(imgdir, "val2014"),
"cap": os.path.join(capdir, "captions_val2014.json")
}
roots["test"] = {
"img": os.path.join(imgdir, "val2014"),
"cap": os.path.join(capdir, "captions_val2014.json")
}
roots["trainrestval"] = {
"img": (roots["train"]["img"], roots["val"]["img"]),
"cap": (roots["train"]["cap"], roots["val"]["cap"])
}
ids["train"] = np.load(os.path.join(capdir, "coco_train_ids.npy"))
ids["val"] = np.load(os.path.join(capdir, "coco_dev_ids.npy"))[:5000]
ids["test"] = np.load(os.path.join(capdir, "coco_test_ids.npy"))
ids["trainrestval"] = (
ids["train"],
np.load(os.path.join(capdir, "coco_restval_ids.npy")))
if use_restval:
roots["train"] = roots["trainrestval"]
ids["train"] = ids["trainrestval"]
elif "f8k" == name:
imgdir = os.path.join(path, "images")
cap = os.path.join(path, "dataset_flickr8k.json")
roots["train"] = {"img": imgdir, "cap": cap}
roots["val"] = {"img": imgdir, "cap": cap}
roots["test"] = {"img": imgdir, "cap": cap}
ids = {"train": None, "val": None, "test": None}
elif "f30k" == name:
imgdir = os.path.join(path, "images")
cap = os.path.join(path, "dataset_flickr30k.json")
roots["train"] = {"img": imgdir, "cap": cap}
roots["val"] = {"img": imgdir, "cap": cap}
roots["test"] = {"img": imgdir, "cap": cap}
ids = {"train": None, "val": None, "test": None}
elif "CUHK-PEDES" == name:
imgdir = os.path.join(path, "imgs")
cap = os.path.join(path, "reid_raw.json")
roots["train"] = {"img": imgdir, "cap": cap}
roots["val"] = {"img": imgdir, "cap": cap}
roots["test"] = {"img": imgdir, "cap": cap}
ids = {"train": None, "val": None, "test": None}
return roots, ids
# CUHK DATASET
class CUHKDataset(data.Dataset):
"""CUHKDataset test on the person retrieval task."""
def __init__(self, root, json, split, vocab, transform=None):
self.root = root
self.vocab = vocab
self.split = split
self.transform = transform
imgs = jsonmod.load(open(json, "r"))
self.imgs = [x for x in imgs if x["split"] == split ]
self.ids = []
for i, d in enumerate(self.imgs):
self.ids += [(i, x) for x in range(len(d["captions"]))]
def __getitem__(self, index):
vocab = self.vocab
ann_id = self.ids[index]
img_id = ann_id[0]
img = self.imgs[img_id]
image = Image.open(os.path.join(self.root, img["file_path"]))
if self.transform is not None:
image= self.transform(image)
tokens = nltk.tokenize.word_tokenize(
str(img["captions"][ann_id[1]]).lower()) # deop decode
caption = []
caption.append(vocab("<start>"))
caption.extend([vocab(token) for token in tokens])
caption.append(vocab("<end>"))
# print len(tokens), len(caption)
# print caption
target = torch.Tensor(caption)
return image, target, index, img_id
def __len__(self):
return len(self.ids)
class CocoDataset(data.Dataset):
"""COCO Custom Dataset compatible with torch.utils.data.DataLoader."""
def __init__(self, root, json, vocab, transform=None, ids=None):
"""
Args:
root: image directory.
json: coco annotation file path.
vocab: vocabulary wrapper.
transform: transformer for image.
"""
self.root = root
# when using `restval`, two json files are needed
if isinstance(json, tuple):
self.coco = (COCO(json[0]), COCO(json[1]))
else:
self.coco = (COCO(json),)
self.root = (root,)
# if ids provided by get_paths, use split-specific ids
if ids is None:
self.ids = list(self.coco.anns.keys())
else:
self.ids = ids
# if `restval` data is to be used, record the break point for ids
if isinstance(self.ids, tuple):
self.bp = len(self.ids[0])
self.ids = list(self.ids[0]) + list(self.ids[1])
else:
self.bp = len(self.ids)
self.vocab = vocab
self.transform = transform
def __getitem__(self, index):
"""This function returns a tuple that is further passed to collate_fn
"""
vocab = self.vocab
root, caption, img_id, path, image = self.get_raw_item(index)
if self.transform is not None:
image = self.transform(image)
# Convert caption (string) to word ids.
tokens = nltk.tokenize.word_tokenize(
str(caption).lower()) # drop decode
#tokens = tokens[:max_length]
caption = []
caption.append(vocab("<start>"))
caption.extend([vocab(token) for token in tokens])
caption.append(vocab("<end>"))
target = torch.Tensor(caption)
return image, target, index, img_id
def get_raw_item(self, index):
if index < self.bp:
coco = self.coco[0]
root = self.root[0]
else:
coco = self.coco[1]
root = self.root[1]
ann_id = self.ids[index]
caption = coco.anns[ann_id]["caption"]
img_id = coco.anns[ann_id]["image_id"]
path = coco.loadImgs(img_id)[0]["file_name"]
image = Image.open(os.path.join(root, path)).convert("RGB")
return root, caption, img_id, path, image
def __len__(self):
return len(self.ids)
class FlickrDataset(data.Dataset):
"""
Dataset loader for Flickr30k and Flickr8k full datasets.
Formats:
"images":[{
"sentids": [0, 1, 2, 3, 4],
"imgid": 0,
"sentences": [{
"tokens": ["a", "sample", "example"],
"raw": "A sample example.",
"imgid": 0,
"sentid": 0
}, ... ]
"split": "train/val/test",
"filename:" "xxx.jpg",
}, ... ]
"""
def __init__(self, root, json, split, vocab, transform=None):
self.root = root
self.vocab = vocab
self.split = split
self.transform = transform
self.dataset = jsonmod.load(open(json, "r"))["images"]
self.ids = []
self.img_num = 0
for i, d in enumerate(self.dataset):
if d["split"] == split:
self.ids += [(i, x, self.img_num) for x in range(len(d["sentences"]))]
self.img_num += 1
def __getitem__(self, index):
"""This function returns a tuple that is further passed to collate_fn
"""
vocab = self.vocab
root = self.root
ann_id = self.ids[index]
img_id = ann_id[0]
img_cls = ann_id[2]
caption = self.dataset[img_id]["sentences"][ann_id[1]]["raw"]
path = self.dataset[img_id]["filename"]
image = Image.open(os.path.join(root, path)).convert("RGB")
if self.transform is not None:
image = self.transform(image)
# Convert caption (string) to word ids.
tokens = nltk.tokenize.word_tokenize(
str(caption).lower()) # drop decode
caption = []
caption.append(vocab("<start>"))
caption.extend([vocab(token) for token in tokens])
caption.append(vocab("<end>"))
target = torch.Tensor(caption)
return image, target, index, img_id, img_cls
def __len__(self):
return len(self.ids)
class PrecompDataset(data.Dataset):
"""
Load precomputed captions and image features
Possible options: f8k, f30k, coco, 10crop
"""
def __init__(self, data_path, data_split, vocab):
self.vocab = vocab
loc = data_path + "/"
# Captions
self.captions = []
with open(loc+"%s_caps.txt" % data_split, "r") as f:
for line in f:
self.captions.append(line.strip())
# Image features
self.images = np.load(loc+"%s_ims.npy" % data_split)
# rkiros data has redundancy in images, we divide by 5, 10crop doesn"t
if self.images.shape[0] != len(self.captions):
self.im_div = 5
else:
self.im_div = 1
self.length = len(self.captions) // self.im_div
# the development set for coco is large and so validation would be slow
if data_split == "dev":
self.length = 5000
def __getitem__(self, index):
# handle the image redundancy
img_id = index
caption_id = index * self.im_div + randint(0, self.im_div-1)
image = torch.Tensor(self.images[img_id])
caption = self.captions[caption_id]
vocab = self.vocab
# Convert caption (string) to word ids.
tokens = nltk.tokenize.word_tokenize(
str(caption).lower()) # drop decode
tokens = tokens[:max_length]
caption = []
caption.append(vocab("<start>"))
caption.extend([vocab(token) for token in tokens])
caption.append(vocab("<end>"))
target = torch.Tensor(caption)
return image, target, caption_id, img_id, img_id
def __len__(self):
return self.length
def collate_fn(data):
"""Build mini-batch tensors from a list of (image, caption) tuples.
Args:
data: list of (image, caption) tuple.
- image: torch tensor of shape (3, 256, 256).
- caption: torch tensor of shape (?); variable length.
Returns:
images: torch tensor of shape (batch_size, 3, 256, 256).
targets: torch tensor of shape (batch_size, padded_length).
lengths: list; valid length for each padded caption.
"""
# Sort a data list by caption length
data.sort(key=lambda x: len(x[1]), reverse=True)
images, captions, ids, img_ids, img_cls = zip(*data)
# Merge images (convert tuple of 3D tensor to 4D tensor)
images = torch.stack(images, 0)
# Merget captions (convert tuple of 1D tensor to 2D tensor)
lengths = [len(cap) for cap in captions]
# targets = torch.zeros(len(captions), max(lengths)).long()
targets = torch.zeros(len(captions), max(lengths)).long()
for i, cap in enumerate(captions):
end = lengths[i]
targets[i, :end] = cap[:end]
lengths = torch.Tensor(lengths)
img_cls = torch.Tensor(img_cls).long()
return images, targets, lengths, ids, img_cls
def get_loader_single(data_name, split, root, json, vocab, transform,
batch_size=100, shuffle=True,
num_workers=2, ids=None, collate_fn=collate_fn,
distributed=False):
"""Returns torch.utils.data.DataLoader for custom coco dataset."""
if "coco" in data_name:
# COCO custom dataset
dataset = CocoDataset(root=root,
json=json,
vocab=vocab,
transform=transform, ids=ids)
elif "f8k" in data_name or "f30k" in data_name:
dataset = FlickrDataset(root=root,
split=split,
json=json,
vocab=vocab,
transform=transform)
elif "CUHK" in data_name:
dataset = CUHKDataset(root=root,
split=split,
json=json,
vocab=vocab,
transform=transform)
# Data loader
if distributed:
data_sampler = DistributedSampler(dataset)
data_loader = torch.utils.data.DataLoader(dataset=dataset,
batch_size=batch_size,
shuffle=False,
pin_memory=False,
num_workers=num_workers,
sampler=data_sampler,
collate_fn=collate_fn)
else:
data_loader = torch.utils.data.DataLoader(dataset=dataset,
batch_size=batch_size,
shuffle=shuffle,
pin_memory=True,
num_workers=num_workers,
collate_fn=collate_fn)
return data_loader
def get_precomp_loader(data_path, data_split, vocab, opt, batch_size=100,
shuffle=True, num_workers=2):
"""Returns torch.utils.data.DataLoader for custom coco dataset."""
dset = PrecompDataset(data_path, data_split, vocab)
data_loader = torch.utils.data.DataLoader(dataset=dset,
batch_size=batch_size,
shuffle=shuffle,
pin_memory=True,
collate_fn=collate_fn)
return data_loader
def get_transform(data_name, split_name, opt):
normalizer = transforms.Normalize(mean=[0.485, 0.456, 0.406],
std=[0.229, 0.224, 0.225])
t_list = []
if split_name == "train":
t_list = [transforms.RandomSizedCrop(opt.crop_size),
transforms.RandomHorizontalFlip()]
elif split_name == "val":
t_list = [transforms.Resize(256), transforms.CenterCrop(224)]
#t_list = [transforms.Resize((224, 224))]
elif split_name == "test":
t_list = [transforms.Resize(256), transforms.CenterCrop(224)]
#t_list = [transforms.Resize((224, 224))]
"""if "CUHK" in data_name:
t_end = [transforms.ToTensor()]
else:"""
t_end = [transforms.ToTensor(), normalizer]
transform = transforms.Compose(t_list + t_end)
return transform
def get_loaders(data_name, vocab, crop_size, batch_size, workers, opt):
dpath = os.path.join(opt.data_path, data_name)
if opt.data_name.endswith("_precomp"):
train_loader = get_precomp_loader(dpath, "train", vocab, opt,
batch_size, True, workers)
val_loader = get_precomp_loader(dpath, "dev", vocab, opt,
batch_size, False, workers)
else:
# Build Dataset Loader
roots, ids = get_paths(dpath, data_name, opt.use_restval)
transform = get_transform(data_name, "train", opt)
train_loader = get_loader_single(opt.data_name, "train", # !!!
roots["train"]["img"],
roots["train"]["cap"],
vocab, transform, ids=ids["train"],
batch_size=batch_size, shuffle=True,
num_workers=workers,
collate_fn=collate_fn,
distributed=opt.distributed)
transform = get_transform(data_name, "val", opt)
val_loader = get_loader_single(opt.data_name, "test", # !!!
roots["val"]["img"],
roots["val"]["cap"],
vocab, transform, ids=ids["val"],
batch_size=16, shuffle=False,
num_workers=workers,
collate_fn=collate_fn,
distributed=False)
return train_loader, val_loader
def get_test_loader(split_name, data_name, vocab, crop_size, batch_size,
workers, opt):
dpath = os.path.join(opt.data_path, data_name)
if opt.data_name.endswith("_precomp"):
test_loader = get_precomp_loader(dpath, split_name, vocab, opt,
batch_size, False, workers)
else:
# Build Dataset Loader
roots, ids = get_paths(dpath, data_name, opt.use_restval)
transform = get_transform(data_name, split_name, opt)
test_loader = get_loader_single(opt.data_name, split_name,
roots[split_name]["img"],
roots[split_name]["cap"],
vocab, transform, ids=ids[split_name],
batch_size=batch_size, shuffle=False,
num_workers=workers,
collate_fn=collate_fn,
distributed=False)
return test_loader
================================================
FILE: embed/cocoword2vec300dim.npy
================================================
[File too large to display: 26.9 MB]
================================================
FILE: embed/f30kword2vec300dim_3.npy
================================================
[File too large to display: 19.7 MB]
================================================
FILE: evaluation.py
================================================
from __future__ import print_function
import os
import pickle
import time
import logging
import numpy
from data import get_test_loader
import time
import numpy as np
from vocab import Vocabulary # NOQA
import torch
from model import CAMP, attention_sim
from collections import OrderedDict
import torch.distributed as dist
from torch.utils.data.distributed import DistributedSampler
from torch.autograd import Variable
import torch.backends.cudnn as cudnn
from model import max_length
class AverageMeter(object):
"""Computes and stores the average and current value"""
def __init__(self):
self.reset()
def reset(self):
self.val = 0
self.avg = 0
self.sum = 0
self.count = 0
def update(self, val, n=0):
self.val = val
self.sum += val * n
self.count += n
self.avg = self.sum / (.0001 + self.count)
def __str__(self):
"""String representation for logging
"""
# for values that should be recorded exactly e.g. iteration number
if self.count == 0:
return str(self.val)
# for stats
return '%.4f (%.4f)' % (self.val, self.avg)
class LogCollector(object):
"""A collection of logging objects that can change from train to val"""
def __init__(self):
# to keep the order of logged variables deterministic
self.meters = OrderedDict()
def update(self, k, v, n=0):
# create a new meter if previously not recorded
if k not in self.meters:
self.meters[k] = AverageMeter()
self.meters[k].update(v, n)
def __str__(self):
"""Concatenate the meters in one log line
"""
s = ''
for i, (k, v) in enumerate(self.meters.items()): # drop iter
if i > 0:
s += ' '
s += k + ' ' + str(v)
return s
def tb_log(self, tb_logger, prefix='', step=None):
"""Log using tensorboard
"""
for k, v in self.meters.items(): # drop iter
tb_logger.log_value(prefix + k, v.val, step=step)
def encode_data(model, data_loader, log_step=10, logging=print):
"""Encode all images and captions loadable by `data_loader`
"""
batch_time = AverageMeter()
val_logger = LogCollector()
# switch to evaluate mode
model.val_start()
end = time.time()
# numpy array to keep all the embeddings
img_embs = None
cap_embs = None
cap_masks = None
print("start loading val data...")
for i, (images, captions, lengths, ids, img_ids) in enumerate(data_loader):
# make sure val logger is used
model.logger = val_logger
# compute the embeddings
img_emb, cap_emb = model.forward_emb(images, captions, lengths,
volatile=True)
# logging("forward finish!")
# initialize the numpy arrays given the size of the embeddings
if img_embs is None:
if model.opt.cross_model:
img_embs = np.zeros((len(data_loader.dataset), img_emb.size(1), img_emb.size(2)))
cap_embs = np.zeros((len(data_loader.dataset), max_length+3, cap_emb.size(2)))
cap_masks = np.zeros((len(data_loader.dataset), max_length+3), dtype=int)
else:
img_embs = np.zeros((len(data_loader.dataset), img_emb.size(1)))
if model.opt.measure == "attention":
#cap_embs = np.zeros((len(data_loader.dataset), max_length+3, cap_emb.size(2)))
cap_embs = np.zeros((len(data_loader.dataset), cap_emb.size(1)))
else:
cap_embs = np.zeros((len(data_loader.dataset), cap_emb.size(1)))
# preserve the embeddings by copying from gpu and converting to numpy
img_embs[ids] = img_emb.data.cpu().numpy().copy()
cap_embs[ids] = cap_emb.data.cpu().numpy().copy()
l_list = [int(l_now) for l_now in lengths]
cur_mask = np.zeros((lengths.size(0), max_length+3), dtype=int)
for mask_idx, mask_l in enumerate(l_list):
cur_mask[mask_idx, :mask_l] = 1
cap_masks[ids] = cur_mask
# measure accuracy and record loss
# model.forward_loss(img_emb, cap_emb)
# measure elapsed time
batch_time.update(time.time() - end)
end = time.time()
if i % log_step == 0:
logging('Test: [{0}/{1}]\t'
'{e_log}\t'
'Time {batch_time.val:.3f} ({batch_time.avg:.3f})\t'
.format(
i, len(data_loader), batch_time=batch_time,
e_log="Unavailable"))
del images, captions
return img_embs, cap_embs, cap_masks
def evalrank(model_path, data_path=None, split='dev', fold5=False, return_ranks=False):
"""
Evaluate a trained model on either dev or test. If `fold5=True`, 5 fold
cross-validation is done (only for MSCOCO). Otherwise, the full data is
used for evaluation.
"""
# load model and options
logging.basicConfig(format='%(asctime)s %(message)s', level=logging.INFO)
checkpoint = torch.load(model_path)
opt = checkpoint['opt']
if data_path is not None:
opt.data_path = data_path
# load vocabulary used by the model
vocab = pickle.load(open(os.path.join(
opt.vocab_path, '%s_vocab.pkl' % opt.data_name), 'rb'))
opt.vocab_size = len(vocab)
opt.distributed = False
opt.use_all = True
opt.instance_loss = False
opt.attention = False
print(opt)
# construct model
model = VSE(opt)
if "cnn.classifier.1.weight" in checkpoint['model'][0]:
checkpoint['model'][0]["cnn.classifier.0.weight"] = checkpoint['model'][0].pop("cnn.classifier.1.weight")
checkpoint['model'][0]["cnn.classifier.0.bias"] = checkpoint['model'][0].pop("cnn.classifier.1.bias")
checkpoint['model'][0]["cnn.classifier.3.weight"] = checkpoint['model'][0].pop("cnn.classifier.4.weight")
checkpoint['model'][0]["cnn.classifier.3.bias"] = checkpoint['model'][0].pop("cnn.classifier.4.bias")
# load model state
model.load_state_dict(checkpoint['model'])
print('Loading dataset')
data_loader = get_test_loader(split, opt.data_name, vocab, opt.crop_size,
opt.batch_size, opt.workers, opt)
print('Computing results...')
img_embs, cap_embs = encode_data(model, data_loader)
print('Images: %d, Captions: %d' %
(img_embs.shape[0] / 5, cap_embs.shape[0]))
if not fold5:
# no cross-validation, full evaluation
r, rt = i2t(img_embs, cap_embs, measure=opt.measure, return_ranks=True)
ri, rti = t2i(img_embs, cap_embs,
measure=opt.measure, return_ranks=True)
ar = (r[0] + r[1] + r[2]) / 3
ari = (ri[0] + ri[1] + ri[2]) / 3
rsum = r[0] + r[1] + r[2] + ri[0] + ri[1] + ri[2]
print("rsum: %.1f" % rsum)
print("Average i2t Recall: %.1f" % ar)
print("Image to text: %.1f %.1f %.1f %.1f %.1f" % r)
print("Average t2i Recall: %.1f" % ari)
print("Text to image: %.1f %.1f %.1f %.1f %.1f" % ri)
else:
# 5fold cross-validation, only for MSCOCO
results = []
for i in range(5):
r, rt0 = i2t(img_embs[i * 5000:(i + 1) * 5000],
cap_embs[i * 5000:(i + 1) *
5000], measure=opt.measure,
return_ranks=True)
print("Image to text: %.1f, %.1f, %.1f, %.1f, %.1f" % r)
ri, rti0 = t2i(img_embs[i * 5000:(i + 1) * 5000],
cap_embs[i * 5000:(i + 1) *
5000], measure=opt.measure,
return_ranks=True)
if i == 0:
rt, rti = rt0, rti0
print("Text to image: %.1f, %.1f, %.1f, %.1f, %.1f" % ri)
ar = (r[0] + r[1] + r[2]) / 3
ari = (ri[0] + ri[1] + ri[2]) / 3
rsum = r[0] + r[1] + r[2] + ri[0] + ri[1] + ri[2]
print("rsum: %.1f ar: %.1f ari: %.1f" % (rsum, ar, ari))
results += [list(r) + list(ri) + [ar, ari, rsum]]
print("-----------------------------------")
print("Mean metrics: ")
mean_metrics = tuple(np.array(results).mean(axis=0).flatten())
print("rsum: %.1f" % (mean_metrics[10] * 6))
print("Average i2t Recall: %.1f" % mean_metrics[11])
print("Image to text: %.1f %.1f %.1f %.1f %.1f" %
mean_metrics[:5])
print("Average t2i Recall: %.1f" % mean_metrics[12])
print("Text to image: %.1f %.1f %.1f %.1f %.1f" %
mean_metrics[5:10])
if return_ranks:
return rt, rti
def i2t(images, captions, masks, npts=None, measure='cosine', return_ranks=False,
model=None):
"""
Images->Text (Image Annotation)
Images: (5N, K) matrix of images
Captions: (5N, K) matrix of captions
"""
if npts is None:
npts = images.shape[0] // 5
index_list = []
gv1_list = []
gv2_list = []
ranks = numpy.zeros(npts)
top1 = numpy.zeros(npts)
score_matrix = numpy.zeros((images.shape[0] // 5, captions.shape[0]))
for index in range(npts):
# Get query image
if model.opt.cross_model:
im = images[5 * index].reshape(1, images.shape[1], images.shape[2])
else:
im = images[5 * index].reshape(1, images.shape[1])
# Compute scores
if measure == 'attention':
bs = 5
if index % bs == 0:
# print ('['+str(index)+'/'+str(npts)+']')
mx = min(images.shape[0], 5 * (index + bs))
im2 = images[5 * index:mx:5]
d2 = attention_sim(Variable(torch.Tensor(im2)).cuda(),
Variable(torch.Tensor(captions)).cuda())
d2 = d2.data.cpu().numpy()
d = d2[index % bs]
elif 'cross_attention' in measure:
bs = 10
if index % bs == 0:
#print ('['+str(index)+'/'+str(npts)+']')
mx = min(images.shape[0], 5 * (index + bs))
im2 = images[5 * index:mx:5]
d2 = model.criterion(Variable(torch.Tensor(im2)).cuda(),
Variable(torch.Tensor(captions)).cuda(),
True, keep="regions",
mask=Variable(torch.ByteTensor(masks)).cuda())
d2 = d2.data.cpu().numpy()
d = d2[index % bs]
elif 'gate_fusion' in measure:
bs = 5
if index % bs == 0:
if index % 50 == 0:
print ('['+str(index)+'/'+str(npts)+']')
mx = min(images.shape[0], 5 * (index + bs))
im2 = images[5 * index:mx:5]
tt1 = time.time()
d2 = model.criterion(Variable(torch.Tensor(im2)).cuda(),
Variable(torch.Tensor(captions)).cuda(),
True, keep="regions",
mask=Variable(torch.ByteTensor(masks)).cuda())
tt2 = time.time()
d2 = d2.data.cpu().numpy()
d = d2[index % bs]
elif measure == 'cosine':
bs = 5
if index % bs == 0:
# print ('['+str(index)+'/'+str(npts)+']')
mx = min(images.shape[0], 5 * (index + bs))
im2 = images[5 * index:mx:5]
d2 = model.criterion(Variable(torch.Tensor(im2)).cuda(),
Variable(torch.Tensor(captions)).cuda(),
True, keep="regions")
d2 = d2.data.cpu().numpy()
d = d2[index % bs]
else:
d = numpy.dot(im, captions.T).flatten()
inds = numpy.argsort(d)[::-1]
index_list.append(inds[0])
score_matrix[index] = d
# Score
rank = 1e20
for i in range(5 * index, 5 * index + 5, 1):
tmp = numpy.where(inds == i)[0][0]
if tmp < rank:
rank = tmp
ranks[index] = rank
top1[index] = inds[0]
#i2t
stat_num = 0
minnum_rank_image = np.array([1e7]*npts)
for i in range(npts):
cur_rank = np.argsort(score_matrix[i])[::-1]
for index, j in enumerate(cur_rank):
if j in range(5*i, 5*i+5):
stat_num += 1
minnum_rank_image[i] = index
break
print ("i2t stat num:", stat_num)
i2t_r1 = 100.0 * len(numpy.where(minnum_rank_image<1)[0]) / len(minnum_rank_image)
i2t_r5 = 100.0 * len(numpy.where(minnum_rank_image<5)[0]) / len(minnum_rank_image)
i2t_r10 = 100.0 * len(numpy.where(minnum_rank_image<10)[0]) / len(minnum_rank_image)
i2t_medr = numpy.floor(numpy.median(minnum_rank_image)) + 1
i2t_meanr = minnum_rank_image.mean() + 1
#print("i2t results:", i2t_r1, i2t_r5, i2t_r10, i2t_medr, i2t_meanr)
#t2i
stat_num = 0
score_matrix = score_matrix.transpose()
minnum_rank_caption = np.array([1e7]*npts*5)
for i in range(5*npts):
img_id = i // 5
cur_rank = np.argsort(score_matrix[i])[::-1]
for index, j in enumerate(cur_rank):
if j == img_id:
stat_num += 1
minnum_rank_caption[i] = index
break
print ("t2i stat num:", stat_num)
t2i_r1 = 100.0 * len(numpy.where(minnum_rank_caption<1)[0]) / len(minnum_rank_caption)
t2i_r5 = 100.0 * len(numpy.where(minnum_rank_caption<5)[0]) / len(minnum_rank_caption)
t2i_r10 = 100.0 * len(numpy.where(minnum_rank_caption<10)[0]) / len(minnum_rank_caption)
t2i_medr = numpy.floor(numpy.median(minnum_rank_caption)) + 1
t2i_meanr = minnum_rank_caption.mean() + 1
# print("t2i results:", t2i_r1, t2i_r5, t2i_r10, t2i_medr, t2i_meanr)
# Compute metrics
r1 = 100.0 * len(numpy.where(ranks < 1)[0]) / len(ranks)
r5 = 100.0 * len(numpy.where(ranks < 5)[0]) / len(ranks)
r10 = 100.0 * len(numpy.where(ranks < 10)[0]) / len(ranks)
medr = numpy.floor(numpy.median(ranks)) + 1
meanr = ranks.mean() + 1
if return_ranks:
return (i2t_r1, i2t_r5, i2t_r10, i2t_medr, i2t_meanr), (t2i_r1, t2i_r5, t2i_r10, t2i_medr, t2i_meanr), score_matrix
else:
return (i2t_r1, i2t_r5, i2t_r10, i2t_medr, i2t_meanr), (t2i_r1, t2i_r5, t2i_r10, t2i_medr, t2i_meanr)
def t2i(images, captions, npts=None, measure='cosine', return_ranks=False,
model=None):
"""
Text->Images (Image Search)
Images: (5N, K) matrix of images
Captions: (5N, K) matrix of captions
"""
if npts is None:
npts = images.shape[0] // 5
ims = numpy.array([images[i] for i in range(0, len(images), 5)])
ranks = numpy.zeros(5 * npts)
top1 = numpy.zeros(5 * npts)
for index in range(npts):
# Get query captions
queries = captions[5 * index:5 * index + 5]
# Compute scores
if measure == 'attention':
bs = 5
if 5 * index % bs == 0:
mx = min(captions.shape[0], 5 * index + bs)
q2 = captions[5 * index:mx]
d2 = attention_sim(Variable(torch.Tensor(ims)).cuda(),
Variable(torch.Tensor(q2)).cuda())
d2 = d2.data.cpu().numpy()
d = d2[:, (5 * index) % bs:(5 * index) % bs + 5].T
elif measure == 'fusion':
bs = 25
if 5 * index % bs == 0:
print ('['+str(index)+'/'+str(npts)+']')
mx = min(captions.shape[0], 5 * index + bs)
q2 = captions[5 * index:mx]
d2 = model.criterion(Variable(torch.Tensor(ims)).cuda(),
Variable(torch.Tensor(q2)).cuda(),
True, keep="words")
d2 = d2.data.cpu().numpy()
d = d2[:, (5 * index) % bs:(5 * index) % bs + 5].T
else:
d = numpy.dot(queries, ims.T)
inds = numpy.zeros(d.shape)
for i in range(len(inds)):
inds[i] = numpy.argsort(d[i])[::-1]
ranks[5 * index + i] = numpy.where(inds[i] == index)[0][0]
top1[5 * index + i] = inds[i][0]
# Compute metrics
r1 = 100.0 * len(numpy.where(ranks < 1)[0]) / len(ranks)
r5 = 100.0 * len(numpy.where(ranks < 5)[0]) / len(ranks)
r10 = 100.0 * len(numpy.where(ranks < 10)[0]) / len(ranks)
medr = numpy.floor(numpy.median(ranks)) + 1
meanr = ranks.mean() + 1
if return_ranks:
return (r1, r5, r10, medr, meanr), (ranks, top1)
else:
return (r1, r5, r10, medr, meanr)
================================================
FILE: experiments/f30k_cross_attention/config.yaml
================================================
common:
data_path: ../../data/
data_name: f30k_precomp
use_restval: True
crop_size: 224
vocab_path: ../../vocab/
logger_name: ./runs/f30k_cros_attn
num_epochs: 300
batch_size: 128
word_dim: 300
img_dim: 2048
embed_size: 1024
grad_clip: 2
learning_rate: 0.0002
lr_update: 75
workers: 3
log_step: 10
val_epoc: 5
cnn_type: None
cross_model: True
max_violation: True
margin: 0.2
measure: cross_attention
word_embed: ../../embed/f30kword2vec300dim_3.npy
embed_mask: False
self_attention: False
bi_gru: True
num_layers: 1
use_abs: False
no_imgnorm: True
no_txtnorm: True
resume: False
finetune: False
lr_scheduler:
type: STEP
lr_steps: [18750, 37500, 56250]
lr_mults: [0.1, 0.1, 0.1]
base_lr: 0.2
warmup_steps: 2500
warmup_lr: 0.8
max_iter: 62500
optimizer:
type: Adam
momentum: 0.9
weight_decay: 0.0001
nesterov: True
================================================
FILE: experiments/f30k_cross_attention/config_256.yaml
================================================
common:
data_path: ../../data/
data_name: f30k_precomp
use_restval: True
crop_size: 224
vocab_path: ../../vocab/
logger_name: ./runs/f30k_cros_attn_256_new_normalLe_0.002
num_epochs: 300
batch_size: 256
word_dim: 300
img_dim: 2048
embed_size: 1024
grad_clip: 2
learning_rate: 0.002
lr_update: 50
workers: 3
log_step: 10
val_epoc: 5
cnn_type: None
cross_model: True
max_violation: True
margin: 0.2
measure: cross_attention_new
fusion_func: concat
word_embed: ../../embed/f30kword2vec300dim_3.npy
embed_mask: False
self_attention: False
bi_gru: True
num_layers: 1
use_abs: False
no_imgnorm: True
no_txtnorm: True
resume: False
finetune: False
finetune_gate: False
lr_scheduler:
type: STEP
lr_steps: [18750, 37500, 56250]
lr_mults: [0.1, 0.1, 0.1]
base_lr: 0.2
warmup_steps: 2500
warmup_lr: 0.8
max_iter: 62500
optimizer:
type: Adam
momentum: 0.9
weight_decay: 0.0001
nesterov: True
================================================
FILE: experiments/f30k_cross_attention/config_test.yaml
================================================
common:
data_path: ./data/
data_name: f30k_precomp
use_restval: True
crop_size: 224
vocab_path: ./vocab/
logger_name: ./experiments/f30k_cross_attention/runs/f30k_cros_attn
num_epochs: 300
batch_size: 128
word_dim: 300
img_dim: 2048
embed_size: 1024
grad_clip: 2
learning_rate: 0.0002
lr_update: 75
workers: 3
log_step: 10
val_epoc: 5
cnn_type: None
cross_model: True
max_violation: True
margin: 0.2
measure: cross_attention_new
word_embed: ./embed/f30kword2vec300dim_3.npy
embed_mask: False
self_attention: False
bi_gru: True
num_layers: 1
use_abs: False
no_imgnorm: True
no_txtnorm: True
resume: ./checkpoint_110.pth.tar
finetune: False
lr_scheduler:
type: STEP
lr_steps: [18750, 37500, 56250]
lr_mults: [0.1, 0.1, 0.1]
base_lr: 0.2
warmup_steps: 2500
warmup_lr: 0.8
max_iter: 62500
optimizer:
type: Adam
momentum: 0.9
weight_decay: 0.0001
nesterov: True
================================================
FILE: experiments/f30k_gate_fusion/config_finetune.yaml
================================================
common:
data_path: ../../data/
data_name: f30k_precomp
use_restval: True
crop_size: 224
vocab_path: ../../vocab/
logger_name: ./runs/f30k_gate_fusion_new_SGD_finetuneall_BCE_loss_0.01lr_nonscale_test_withnorm_25_testdev
num_epochs: 300
batch_size: 128
word_dim: 300
img_dim: 2048
embed_size: 1024
grad_clip: 2
learning_rate: 0.01
lr_update: 50
workers: 3
log_step: 10
val_epoc: 1
cnn_type: None
cross_model: True
max_violation: True
loss_func: BCE
margin: 0.2
measure: gate_fusion_new
fusion_func: concat
word_embed: False
embed_mask: False
self_attention: False
bi_gru: True
num_layers: 1
use_abs: False
no_imgnorm: True
no_txtnorm: True
resume: ../f30k_gate_fusion_new/runs/f30k_gate_fusion_new_SGD_finxco_BCE_loss_0.01lr_noscaleLE/checkpoint_65.pth.tar
finetune: False
finetune_gate: True
lr_scheduler:
type: STEP
lr_steps: [18750, 37500, 56250]
lr_mults: [0.1, 0.1, 0.1]
base_lr: 0.2
warmup_steps: 2500
warmup_lr: 0.8
max_iter: 62500
optimizer:
type: SGD
momentum: 0.9
weight_decay: 0.0001
nesterov: True
================================================
FILE: experiments/f30k_gate_fusion/config_fixatt.yaml
================================================
common:
data_path: ../../data/
data_name: f30k_precomp
use_restval: True
crop_size: 224
vocab_path: ../../vocab/
logger_name: ./runs/f30k_gate_fusion_new_SGD_finxco_BCE_loss_0.01lr_noscaleLE
num_epochs: 300
batch_size: 128
word_dim: 300
img_dim: 2048
embed_size: 1024
grad_clip: 2
learning_rate: 0.01
lr_update: 45
workers: 3
log_step: 10
val_epoc: 5
cnn_type: None
cross_model: True
max_violation: True
loss_func: BCE
margin: 0.2
measure: gate_fusion_new
fusion_func: concat
word_embed: False
embed_mask: False
self_attention: False
bi_gru: True
num_layers: 1
use_abs: False
no_imgnorm: True
no_txtnorm: True
resume: ../f30k_cross_attention_new/runs/f30k_cros_attn_256_new_withnorm/checkpoint_110.pth.tar
finetune: False
finetune_gate: False
lr_scheduler:
type: STEP
lr_steps: [18750, 37500, 56250]
lr_mults: [0.1, 0.1, 0.1]
base_lr: 0.2
warmup_steps: 2500
warmup_lr: 0.8
max_iter: 62500
optimizer:
type: SGD
momentum: 0.9
weight_decay: 0.0001
nesterov: True
================================================
FILE: experiments/f30k_gate_fusion/config_test.yaml
================================================
common:
data_path: ../../data/
data_name: f30k_precomp
use_restval: True
crop_size: 224
vocab_path: ./vocab/
logger_name: ./runs/f30k_gate_fusion_test/
num_epochs: 300
batch_size: 128
word_dim: 300
img_dim: 2048
embed_size: 1024
grad_clip: 2
learning_rate: 0.01
lr_update: 50
workers: 3
log_step: 10
val_epoc: 1
cnn_type: None
cross_model: True
max_violation: True
loss_func: BCE
margin: 0.2
measure: gate_fusion_new
fusion_func: concat
word_embed: False
embed_mask: False
self_attention: False
bi_gru: True
num_layers: 1
use_abs: False
no_imgnorm: True
no_txtnorm: True
resume: ../f30k_gate_fusion/checkpoint_150.pth.tar
finetune: False
finetune_gate: True
lr_scheduler:
type: STEP
lr_steps: [18750, 37500, 56250]
lr_mults: [0.1, 0.1, 0.1]
base_lr: 0.2
warmup_steps: 2500
warmup_lr: 0.8
max_iter: 62500
optimizer:
type: SGD
momentum: 0.9
weight_decay: 0.0001
nesterov: True
================================================
FILE: fusion_module.py
================================================
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.distributed as dist
import torch.nn.init
import torchvision.models as models
from torch.autograd import Variable
from torch.nn.utils.rnn import pack_padded_sequence, pad_packed_sequence
from torch.nn.utils.clip_grad import clip_grad_norm
import numpy as np
from collections import OrderedDict
import torch.nn.functional as F
from torch.utils.data import DataLoader
from torch.utils.data.distributed import DistributedSampler
import logging
import torch.backends.cudnn as cudnn
import pickle
from math import sqrt
def l2norm(X, dim=-1, eps=1e-8):
"""L2-normalize columns of X
"""
norm = torch.pow(X, 2).sum(dim=dim, keepdim=True).sqrt() + eps
X = torch.div(X, norm)
return X
def sum_attention(nnet, query, value, mask=None, dropout=None):
scores = nnet(query).transpose(-2, -1)
if mask is not None:
scores.data.masked_fill_(mask.data.eq(0), -1e9)
p_attn = F.softmax(scores, dim=-1)
if dropout is not None:
p_attn = dropout(p_attn)
return torch.matmul(p_attn, value), p_attn
def qkv_attention(query, key, value, mask=None, dropout=None):
d_k = query.size(-1)
scores = torch.matmul(query, key.transpose(-2, -1)) / sqrt(d_k)
if mask is not None:
scores.data.masked_fill_(mask.data.eq(0), -1e9)
p_attn = F.softmax(scores, dim=-1)
if dropout is not None:
p_attn = dropout(p_attn)
return torch.matmul(p_attn, value), p_attn
class SummaryAttn(nn.Module):
def __init__(self, dim, num_attn, dropout, is_cat=False):
super(SummaryAttn, self).__init__()
self.linear = nn.Sequential(
nn.Linear(dim, dim),
nn.ReLU(inplace=True),
nn.Linear(dim, num_attn),
)
self.h = num_attn
self.is_cat = is_cat
self.attn = None
self.dropout = nn.Dropout(p=dropout) if dropout > 0 else None
def forward(self, query, value, mask=None):
if mask is not None:
mask = mask.unsqueeze(-2)
batch = query.size(0)
weighted, self.attn = sum_attention(self.linear, query, value, mask=mask, dropout=self.dropout)
weighted = weighted if self.is_cat else weighted.mean(dim=-2)
return weighted
class CrossAttention(nn.Module):
""" TBD...
"""
def __init__(self, dim, num_attn, dropout, reduce_func="self_attn"):
super(CrossAttention, self).__init__()
self.dim = dim
self.h = num_attn
self.dropout = nn.Dropout(p=dropout) if dropout > 0 else None
self.reduce_func = reduce_func
self.img_key_fc = nn.Linear(dim, dim, bias=False)
self.txt_key_fc = nn.Linear(dim, dim, bias=False)
if reduce_func == "mean":
self.reduce_layer = torch.mean
elif reduce_func == "self_attn":
self.reduce_layer_1 = SummaryAttn(dim, num_attn, dropout)
self.reduce_layer_2 = SummaryAttn(dim, num_attn, dropout)
self.init_weights()
print("CrossAttention module init success!")
def init_weights(self):
"""Xavier initialization for the fully connected layer
"""
r = np.sqrt(6.) / np.sqrt(self.dim +
self.dim)
self.img_key_fc.weight.data.uniform_(-r, r)
self.txt_key_fc.weight.data.uniform_(-r, r)
def forward(self, v1, v2, get_score=True, keep=None, mask=None):
if keep == "words":
v2 = v2.squeeze(0)
mask = mask.squeeze(0)
elif keep == "regions":
v1 = v1.squeeze(0)
k1 = self.img_key_fc(v1)
k2 = self.txt_key_fc(v2)
batch_size_v1 = v1.size(0)
batch_size_v2 = v2.size(0)
v1 = v1.unsqueeze(1).expand(-1, batch_size_v2, -1, -1)
k1 = k1.unsqueeze(1).expand(-1, batch_size_v2, -1, -1)
v2 = v2.unsqueeze(0).expand(batch_size_v1, -1, -1, -1)
k2 = k2.unsqueeze(0).expand(batch_size_v1, -1, -1, -1)
weighted_v1, attn_1 = qkv_attention(k2, k1, v1)
if mask is not None:
weighted_v2, attn_2 = qkv_attention(k1, k2, v2, mask.unsqueeze(-2))
else:
weighted_v2, attn_2 = qkv_attention(k1, k2, v2)
fused_v1 = weighted_v2
fused_v2 = weighted_v1
if self.reduce_func == "self_attn":
co_v1 = self.reduce_layer_1(fused_v1, fused_v1)
co_v2 = self.reduce_layer_2(fused_v2, fused_v2, mask)
co_v1 = l2norm(co_v1)
co_v2 = l2norm(co_v2)
else:
co_v1 = self.reduce_func(co_v1, dim=-2)
co_v2 = self.reduce_func(co_v2, dim=-2)
co_v1 = l2norm(co_v1)
co_v2 = l2norm(co_v2)
if get_score:
score = (co_v1 * co_v2).sum(dim=-1)
if keep == "regions":
score = score.transpose(0, 1)
return score
else:
return torch.cat((co_v1, co_v2), dim=-1)
class GatedFusion(nn.Module):
def __init__(self, dim, num_attn, dropout=0.01, reduce_func="self_attn", fusion_func="concat"):
super(GatedFusion, self).__init__()
self.dim = dim
self.h = num_attn
self.dropout = nn.Dropout(p=dropout) if dropout > 0 else None
self.reduce_func = reduce_func
self.fusion_func = fusion_func
self.img_key_fc = nn.Linear(dim, dim, bias=False)
self.txt_key_fc = nn.Linear(dim, dim, bias=False)
in_dim = dim
if fusion_func == "sum":
in_dim = dim
elif fusion_func == "concat":
in_dim = 2 * dim
else:
raise NotImplementedError('Only support sum or concat fusion')
self.fc_1 = nn.Sequential(
nn.Linear(in_dim, dim),
nn.ReLU(inplace=True),
nn.Dropout(p=dropout),)
self.fc_2 = nn.Sequential(
nn.Linear(in_dim, dim),
nn.ReLU(inplace=True),
nn.Dropout(p=dropout),)
self.fc_out = nn.Sequential(
nn.Linear(in_dim, dim),
nn.ReLU(inplace=True),
nn.Dropout(p=dropout),
nn.Linear(dim, 1),
nn.Sigmoid(),
)
if reduce_func == "mean":
self.reduce_layer = torch.mean
elif reduce_func == "self_attn":
self.reduce_layer_1 = SummaryAttn(dim, num_attn, dropout)
self.reduce_layer_2 = SummaryAttn(dim, num_attn, dropout)
self.init_weights()
print("GatedFusion module init success!")
def init_weights(self):
"""Xavier initialization for the fully connected layer
"""
r = np.sqrt(6.) / np.sqrt(self.dim +
self.dim)
self.img_key_fc.weight.data.uniform_(-r, r)
self.txt_key_fc.weight.data.uniform_(-r, r)
self.fc_1[0].weight.data.uniform_(-r, r)
self.fc_1[0].bias.data.fill_(0)
self.fc_2[0].weight.data.uniform_(-r, r)
self.fc_2[0].bias.data.fill_(0)
self.fc_out[0].weight.data.uniform_(-r, r)
self.fc_out[0].bias.data.fill_(0)
self.fc_out[3].weight.data.uniform_(-r, r)
self.fc_out[3].bias.data.fill_(0)
def forward(self, v1, v2, get_score=True, keep=None, mask=None):
if keep == "words":
v2 = v2.squeeze(0)
mask = mask.squeeze(0)
elif keep == "regions":
v1 = v1.squeeze(0)
k1 = self.img_key_fc(v1)
k2 = self.txt_key_fc(v2)
batch_size_v1 = v1.size(0)
batch_size_v2 = v2.size(0)
v1 = v1.unsqueeze(1).expand(-1, batch_size_v2, -1, -1)
k1 = k1.unsqueeze(1).expand(-1, batch_size_v2, -1, -1)
v2 = v2.unsqueeze(0).expand(batch_size_v1, -1, -1, -1)
k2 = k2.unsqueeze(0).expand(batch_size_v1, -1, -1, -1)
weighted_v1, attn_1 = qkv_attention(k2, k1, v1)
if mask is not None:
weighted_v2, attn_2 = qkv_attention(k1, k2, v2, mask.unsqueeze(-2))
else:
weighted_v2, attn_2 = qkv_attention(k1, k2, v2)
gate_v1 = F.sigmoid((v1 * weighted_v2).sum(dim=-1)).unsqueeze(-1)
gate_v2 = F.sigmoid((v2 * weighted_v1).sum(dim=-1)).unsqueeze(-1)
#gate_v1 = F.sigmoid((v1 * weighted_v2))
#gate_v2 = F.sigmoid((v2 * weighted_v1))
if self.fusion_func == "sum":
fused_v1 = (v1 + weighted_v2)* gate_v1
fused_v2 = (v2 + weighted_v1)* gate_v2
elif self.fusion_func == "concat":
fused_v1 = torch.cat((v1, weighted_v2), dim=-1)* gate_v1
fused_v2 = torch.cat((v2, weighted_v1), dim=-1)* gate_v2
co_v1 = self.fc_1(fused_v1) + v1
co_v2 = self.fc_2(fused_v2) + v2
if self.reduce_func == "self_attn":
co_v1 = self.reduce_layer_1(co_v1, co_v1)
co_v2 = self.reduce_layer_2(co_v2, co_v2, mask)
#co_v1 = l2norm(co_v1)
#co_v2 = l2norm(co_v2)
else:
co_v1 = self.reduce_func(co_v1, dim=-2)
co_v2 = self.reduce_func(co_v2, dim=-2)
co_v1 = l2norm(co_v1)
co_v2 = l2norm(co_v2)
if get_score:
if self.fusion_func == "sum":
score = self.fc_out(co_v1 + co_v2).squeeze(dim=-1)
elif self.fusion_func == "concat":
score = self.fc_out(torch.cat((co_v1, co_v2), dim=-1)).squeeze(dim=-1)
if keep == "regions":
score = score.transpose(0, 1)
#mean_gate = gate_v1.mean(dim=-1).mean(dim=-1) + gate_v2.mean(dim=-1).mean(dim=-1)
return score
else:
return torch.cat((co_v1, co_v2), dim=-1)
class CrossAttentionNew(nn.Module):
""" TBD...
"""
def __init__(self, dim, num_attn, dropout, reduce_func="mean"):
super(CrossAttentionNew, self).__init__()
self.dim = dim
self.h = num_attn
self.dropout = nn.Dropout(p=dropout) if dropout > 0 else None
self.reduce_func = reduce_func
self.img_key_fc = nn.Linear(dim, dim, bias=False)
self.txt_key_fc = nn.Linear(dim, dim, bias=False)
self.img_query_fc = nn.Linear(dim, dim, bias=False)
self.txt_query_fc = nn.Linear(dim, dim, bias=False)
self.weighted_img_key_fc = nn.Linear(dim, dim, bias=False)
self.weighted_txt_key_fc = nn.Linear(dim, dim, bias=False)
self.weighted_img_query_fc = nn.Linear(dim, dim, bias=False)
self.weighted_txt_query_fc = nn.Linear(dim, dim, bias=False)
if reduce_func == "mean":
self.reduce_layer = torch.mean
elif reduce_func == "self_attn":
self.reduce_layer_1 = SummaryAttn(dim, num_attn, dropout)
self.reduce_layer_2 = SummaryAttn(dim, num_attn, dropout)
self.init_weights()
print("CrossAttention module init success!")
def init_weights(self):
"""Xavier initialization for the fully connected layer
"""
r = np.sqrt(6.) / np.sqrt(self.dim +
self.dim)
self.img_key_fc.weight.data.uniform_(-r, r)
self.txt_key_fc.weight.data.uniform_(-r, r)
def forward(self, v1, v2, get_score=True, keep=None, mask=None):
if keep == "words":
v2 = v2.squeeze(0)
mask = mask.squeeze(0)
elif keep == "regions":
v1 = v1.squeeze(0)
k1 = self.img_key_fc(v1)
k2 = self.txt_key_fc(v2)
q1 = self.img_query_fc(v1)
q2 = self.txt_query_fc(v2)
batch_size_v1 = v1.size(0)
batch_size_v2 = v2.size(0)
v1 = v1.unsqueeze(1).expand(-1, batch_size_v2, -1, -1)
k1 = k1.unsqueeze(1).expand(-1, batch_size_v2, -1, -1)
q1 = q1.unsqueeze(1).expand(-1, batch_size_v2, -1, -1)
v2 = v2.unsqueeze(0).expand(batch_size_v1, -1, -1, -1)
k2 = k2.unsqueeze(0).expand(batch_size_v1, -1, -1, -1)
q2 = q2.unsqueeze(0).expand(batch_size_v1, -1, -1, -1)
weighted_v1, attn_1 = qkv_attention(q2, k1, v1)
if mask is not None:
weighted_v2, attn_2 = qkv_attention(q1, k2, v2, mask.unsqueeze(-2))
else:
weighted_v2, attn_2 = qkv_attention(q1, k2, v2)
weighted_v2_q = self.weighted_txt_query_fc(weighted_v2)
weighted_v2_k = self.weighted_txt_key_fc(weighted_v2)
weighted_v1_q = self.weighted_img_query_fc(weighted_v1)
weighted_v1_k = self.weighted_img_key_fc(weighted_v1)
fused_v1, _ = qkv_attention(weighted_v2_q, weighted_v2_k, weighted_v2)
if mask is not None:
fused_v2, _ = qkv_attention(weighted_v1_q, weighted_v1_k, weighted_v1, mask.unsqueeze(-2))
else:
fused_v2, _ = qkv_attention(weighted_v1_q, weighted_v1_k, weighted_v1)
#fused_v1 = l2norm(fused_v1)
#fused_v2 = l2norm(fused_v2)
if self.reduce_func == "self_attn":
co_v1 = self.reduce_layer_1(fused_v1, fused_v1)
co_v2 = self.reduce_layer_2(fused_v2, fused_v2, mask)
co_v1 = l2norm(co_v1)
co_v2 = l2norm(co_v2)
else:
co_v1 = self.reduce_layer(fused_v1, dim=-2)
co_v2 = self.reduce_layer(fused_v2, dim=-2)
co_v1 = l2norm(co_v1)
co_v2 = l2norm(co_v2)
if get_score:
score = (co_v1 * co_v2).sum(dim=-1)
if keep == "regions":
score = score.transpose(0, 1)
return score
else:
return torch.cat((co_v1, co_v2), dim=-1)
class GatedFusionNew(nn.Module):
def __init__(self, dim, num_attn, dropout=0.01, reduce_func="self_attn", fusion_func="concat"):
super(GatedFusionNew, self).__init__()
self.dim = dim
self.h = num_attn
self.dropout = nn.Dropout(p=dropout) if dropout > 0 else None
self.reduce_func = reduce_func
self.fusion_func = fusion_func
self.img_key_fc = nn.Linear(dim, dim, bias=False)
self.txt_key_fc = nn.Linear(dim, dim, bias=False)
self.img_query_fc = nn.Linear(dim, dim, bias=False)
self.txt_query_fc = nn.Linear(dim, dim, bias=False)
self.weighted_img_key_fc = nn.Linear(dim, dim, bias=False)
self.weighted_txt_key_fc = nn.Linear(dim, dim, bias=False)
self.weighted_img_query_fc = nn.Linear(dim, dim, bias=False)
self.weighted_txt_query_fc = nn.Linear(dim, dim, bias=False)
in_dim = dim
if fusion_func == "sum":
in_dim = dim
elif fusion_func == "concat":
in_dim = 2 * dim
else:
raise NotImplementedError('Only support sum or concat fusion')
"""self.fc_gate_1 = nn.Sequential(
nn.Linear(in_dim, in_dim, bias=False),
#nn.ReLU(inplace=True),
#nn.Dropout(p=dropout),
#nn.Linear(dim, 1),
nn.Sigmoid(),
)
self.fc_gate_2 = nn.Sequential(
nn.Linear(in_dim, in_dim, bias=False),
#nn.ReLU(inplace=True),
#nn.Dropout(p=dropout),
#nn.Linear(dim, 1),
nn.Sigmoid(),
)"""
self.fc_1 = nn.Sequential(
nn.Linear(in_dim, dim, bias=False),
nn.ReLU(inplace=True),
nn.Dropout(p=dropout),)
self.fc_2 = nn.Sequential(
nn.Linear(in_dim, dim, bias=False),
nn.ReLU(inplace=True),
nn.Dropout(p=dropout),)
self.fc_out = nn.Sequential(
nn.Linear(in_dim, dim),
nn.ReLU(inplace=True),
nn.Dropout(p=dropout),
nn.Linear(dim, 1),
nn.Sigmoid(),
)
if reduce_func == "mean":
self.reduce_layer = torch.mean
elif reduce_func == "self_attn":
#self.reduce_layer_1 = SummaryAttn(dim, num_attn, dropout, is_cat=True)
#self.reduce_layer_2 = SummaryAttn(dim, num_attn, dropout, is_cat=True)
self.final_reduce_1 = SummaryAttn(dim, num_attn, dropout)
self.final_reduce_2 = SummaryAttn(dim, num_attn, dropout)
self.init_weights()
print("GatedFusion module init success!")
def init_weights(self):
"""Xavier initialization for the fully connected layer
"""
r = np.sqrt(6.) / np.sqrt(self.dim +
self.dim)
self.img_key_fc.weight.data.uniform_(-r, r)
self.txt_key_fc.weight.data.uniform_(-r, r)
self.fc_1[0].weight.data.uniform_(-r, r)
#self.fc_1[0].bias.data.fill_(0)
self.fc_2[0].weight.data.uniform_(-r, r)
#self.fc_2[0].bias.data.fill_(0)
self.fc_out[0].weight.data.uniform_(-r, r)
self.fc_out[0].bias.data.fill_(0)
self.fc_out[3].weight.data.uniform_(-r, r)
self.fc_out[3].bias.data.fill_(0)
def forward(self, v1, v2, get_score=True, keep=None, mask=None):
if keep == "words":
v2 = v2.squeeze(0)
mask = mask.squeeze(0)
elif keep == "regions":
v1 = v1.squeeze(0)
k1 = self.img_key_fc(v1)
k2 = self.txt_key_fc(v2)
q1 = self.img_query_fc(v1)
q2 = self.txt_query_fc(v2)
batch_size_v1 = v1.size(0)
batch_size_v2 = v2.size(0)
v1 = v1.unsqueeze(1).expand(-1, batch_size_v2, -1, -1)
k1 = k1.unsqueeze(1).expand(-1, batch_size_v2, -1, -1)
q1 = q1.unsqueeze(1).expand(-1, batch_size_v2, -1, -1)
v2 = v2.unsqueeze(0).expand(batch_size_v1, -1, -1, -1)
k2 = k2.unsqueeze(0).expand(batch_size_v1, -1, -1, -1)
q2 = q2.unsqueeze(0).expand(batch_size_v1, -1, -1, -1)
weighted_v1, attn_1 = qkv_attention(q2, k1, v1)
if mask is not None:
weighted_v2, attn_2 = qkv_attention(q1, k2, v2, mask.unsqueeze(-2))
else:
weighted_v2, attn_2 = qkv_attention(q1, k2, v2)
weighted_v2_q = self.weighted_txt_query_fc(weighted_v2)
weighted_v2_k = self.weighted_txt_key_fc(weighted_v2)
weighted_v1_q = self.weighted_img_query_fc(weighted_v1)
weighted_v1_k = self.weighted_img_key_fc(weighted_v1)
fused_v1, _ = qkv_attention(weighted_v2_q, weighted_v2_k, weighted_v2)
if mask is not None:
fused_v2, _ = qkv_attention(weighted_v1_q, weighted_v1_k, weighted_v1, mask.unsqueeze(-2))
else:
fused_v2, _ = qkv_attention(weighted_v1_q, weighted_v1_k, weighted_v1)
fused_v1 = l2norm(fused_v1)
fused_v2 = l2norm(fused_v2)
gate_v1 = F.sigmoid((v1 * fused_v1).sum(dim=-1)).unsqueeze(-1)
gate_v2 = F.sigmoid((v2 * fused_v2).sum(dim=-1)).unsqueeze(-1)
if self.fusion_func == "sum":
#gate_v1 = self.fc_gate_1(v1 + fused_v1)
#gate_v2 = self.fc_gate_2(v2 + fused_v2)
co_v1 = (v1 + fused_v1) * gate_v1
co_v2 = (v2 + fused_v2) * gate_v2
elif self.fusion_func == "concat":
#gate_v1 = self.fc_gate_1(torch.cat((v1, fused_v1), dim=-1))
#gate_v2 = self.fc_gate_2(torch.cat((v2, fused_v2), dim=-1))
co_v1 = torch.cat((v1, fused_v1), dim=-1) * gate_v1
co_v2 = torch.cat((v2, fused_v2), dim=-1) * gate_v2
co_v1 = self.fc_1(co_v1) + v1
co_v2 = self.fc_2(co_v2) + v2
if self.reduce_func == "self_attn":
co_v1 = self.final_reduce_1(co_v1, co_v1)
co_v2 = self.final_reduce_2(co_v2, co_v2, mask)
co_v1 = l2norm(co_v1)
co_v2 = l2norm(co_v2)
else:
co_v1 = self.reduce_func(co_v1, dim=-2)
co_v2 = self.reduce_func(co_v2, dim=-2)
co_v1 = l2norm(co_v1)
co_v2 = l2norm(co_v2)
if get_score:
if self.fusion_func == "sum":
score = self.fc_out(co_v1 + co_v2).squeeze(dim=-1)
elif self.fusion_func == "concat":
score = self.fc_out(torch.cat((co_v1, co_v2), dim=-1)).squeeze(dim=-1)
if keep == "regions":
score = score.transpose(0, 1)
#mean_gate = gate_v1.mean(dim=-1).mean(dim=-1) + gate_v2.mean(dim=-1).mean(dim=-1)
return score
else:
return torch.cat((co_v1, co_v2), dim=-1)
================================================
FILE: model.py
================================================
import torch
import torch.nn as nn
import torch.distributed as dist
import torch.nn.init
import torchvision.models as models
from torch.autograd import Variable
from torch.nn.utils.rnn import pack_padded_sequence, pad_packed_sequence
from torch.nn.utils.clip_grad import clip_grad_norm
import numpy as np
from collections import OrderedDict
#from transformer.Models import Encoder as self_attention_encoder
#from transformer.Layers import EncoderLayer as attention_layer
from resnet import *
from torch.utils.data import DataLoader
from torch.utils.data.distributed import DistributedSampler
import logging
import torch.backends.cudnn as cudnn
import pickle
from fusion_module import *
max_length = 47
def l2norm(X, dim=-1, eps=1e-8):
"""L2-normalize columns of X
"""
norm = torch.pow(X, 2).sum(dim=dim, keepdim=True).sqrt() + eps
X = torch.div(X, norm)
return X
def EncoderImage(data_name, img_dim, embed_size, finetune=False,
cnn_type='resnet152', no_imgnorm=False,
self_attention=False):
"""A wrapper to image encoders. Chooses between an encoder that uses
precomputed image features, `EncoderImagePrecomp`, or an encoder that
computes image features on the fly `EncoderImageFull`.
"""
if data_name.endswith('_precomp'):
img_enc = EncoderImagePrecomp(img_dim, embed_size, no_imgnorm,
self_attention)
else:
img_enc = EncoderImageFull(embed_size, finetune, cnn_type, no_imgnorm,
self_attention, fusion)
return img_enc
class ImageSelfAttention(nn.Module):
""" Self-attention module for CNN's feature map.
Inspired by: Zhang et al., 2018 The self-attention mechanism in SAGAN.
"""
def __init__(self, planes):
super(ImageSelfAttention, self).__init__()
inner = planes // 8
self.conv_f = nn.Conv1d(planes, inner, kernel_size=1, bias=False)
self.conv_g = nn.Conv1d(planes, inner, kernel_size=1, bias=False)
self.conv_h = nn.Conv1d(planes, planes, kernel_size=1, bias=False)
def forward(self, x):
x = x.view(x.size(0), x.size(1), -1)
f = self.conv_f(x)
g = self.conv_g(x)
h = self.conv_h(x)
sim_beta = torch.matmul(f.transpose(1, 2), g)
beta = nn.functional.softmax(sim_beta, dim=1)
o = torch.matmul(h, beta)
return o
# tutorials/09 - Image Captioning
class EncoderImageFull(nn.Module):
def __init__(self, embed_size, finetune=False, cnn_type='resnet152',
no_imgnorm=False, self_attention=False, fusion=False):
"""Load pretrained VGG19 and replace top fc layer."""
super(EncoderImageFull, self).__init__()
self.embed_size = embed_size
self.no_imgnorm = no_imgnorm
self.self_attention = self_attention
self.fusion = fusion
# Load a pre-trained model
self.cnn = self.get_cnn(cnn_type, True, fusion)
# For efficient memory usage.
for param in self.cnn.parameters():
param.requires_grad = finetune
# Replace the last fully connected layer of CNN with new structures
if self_attention:
self.cnn.avgpool = nn.Sequential()
self.attention_layer = ImageSelfAttention(2048)
self.AvgPool2d = nn.AvgPool2d(7, stride=1)
if fusion:
self.cnn.avgpool = nn.Sequential()
self.fc = nn.Linear(2048, embed_size)
else:
self.fc = nn.Linear(self.cnn.fc.in_features, embed_size)
self.cnn.fc = nn.Sequential()
self.init_weights()
def get_cnn(self, arch, pretrained, fusion):
"""Load a pretrained CNN and parallelize over GPUs
"""
if arch == "resnet152":
if pretrained:
print("=> using pre-trained model '{}'".format(arch))
model = resnet152(pretrained=True, fusion=fusion)
else:
print("=> creating model '{}'".format(arch))
model = resnet152(pretrained=False, fusion=fusion)
else:
if pretrained:
print("=> using pre-trained model '{}'".format(arch))
model = models.__dict__[arch](pretrained=True)
else:
print("=> creating model '{}'".format(arch))
model = models.__dict__[arch]()
return model
def init_weights(self):
"""Xavier initialization for the fully connected layer
"""
r = np.sqrt(6.) / np.sqrt(self.fc.in_features +
self.fc.out_features)
self.fc.weight.data.uniform_(-r, r)
self.fc.bias.data.fill_(0)
def forward(self, images):
"""Extract image feature vectors."""
features = self.cnn(images)
if self.self_attention:
features = features.view(images.size(0), -1, 7, 7)
features = self.attention_layer(features)
features = features.view(images.size(0), -1, 7, 7)
features = self.AvgPool2d(features)
# linear projection to the joint embedding space
if self.fusion:
features = features.view(features.size(0), features.size(1), -1)
features = features.transpose(1, 2)
else:
features = features.view(features.size(0), -1)
features = self.fc(features)
# normalization in the joint embedding space
if not self.no_imgnorm:
if self.fusion:
features = l2norm(features, dim=2)
else:
features = l2norm(features, dim=1)
return features
class EncoderImagePrecomp(nn.Module):
def __init__(self, img_dim, embed_size, no_imgnorm=False,
self_attention=False):
super(EncoderImagePrecomp, self).__init__()
self.embed_size = embed_size
self.no_imgnorm = no_imgnorm
self.self_attention = self_attention
self.fc = nn.Linear(img_dim, embed_size)
if self_attention:
self.attention_layer = SummaryAttn(embed_size, 1, -1)
self._init_weights()
def _init_weights(self):
"""Xavier initialization for the fully connected layer
"""
r = np.sqrt(6.) / np.sqrt(self.fc.in_features +
self.fc.out_features)
self.fc.weight.data.uniform_(-r, r)
self.fc.bias.data.fill_(0)
def forward(self, images):
"""Extract image feature vectors."""
# assuming that the precomputed features are already l2-normalized
features = self.fc(images)
if self.self_attention:
features = self.attention_layer(features, features)
# normalize in the joint embedding space
if not self.no_imgnorm:
features = l2norm(features, dim=-1)
return features
def load_state_dict(self, state_dict):
"""Copies parameters. overwritting the default one to
accept state_dict from Full model
"""
own_state = self.state_dict()
new_state = OrderedDict()
for name, param in state_dict.items():
if name in own_state:
new_state[name] = param
super(EncoderImagePrecomp, self).load_state_dict(new_state)
# tutorials/08 - Language Model
# RNN Based Language Model
class EncoderText(nn.Module):
def __init__(self, vocab_size, word_dim, embed_size, num_layers,
bi_gru=False, no_txtnorm=False,
self_attention=False, embed_weights=''):
super(EncoderText, self).__init__()
self.no_txtnorm = no_txtnorm
self.embed_size = embed_size
self.self_attention = self_attention
self.bi_gru = bi_gru
# word embedding
self.embed = nn.Embedding(vocab_size, word_dim)
# caption embedding
self.rnn = nn.GRU(word_dim, embed_size, num_layers,
batch_first=True, bidirectional=bi_gru)
if self_attention:
self.attention_layer = SummaryAttn(embed_size, 1, -1)
self._init_weights(embed_weights)
def _init_weights(self, embed_weights=''):
if embed_weights:
w = np.load(embed_weights)
w = torch.from_numpy(w)
self.embed.load_state_dict({'weight': w})
print("Load Word Embedding Weights Successfully.")
else:
self.embed.weight.data.uniform_(-0.1, 0.1)
def forward(self, x, lengths):
"""Handles variable size captions
"""
# Embed word ids to vectors
x = self.embed(x)
# Mask the attention weights of emtpy token
l_list = [int(i) for i in lengths.data]
mask = Variable(torch.ByteTensor([i*[1] + (max_length+3-i)*[0] for i in l_list])).cuda()
# Forward propagate RNN
packed = pack_padded_sequence(x, l_list, batch_first=True)
self.rnn.flatten_parameters()
out, _ = self.rnn(packed)
# Reshape *final* output to (batch_size, hidden_size)
padded = pad_packed_sequence(out, batch_first=True)
if self.bi_gru:
out, cap_len = padded
out = (out[:,:,:out.size(2)//2] + out[:,:,out.size(2)//2:])/2
I = Variable(torch.zeros(out.size(0),
max_length+3-out.size(1), out.size(2))).cuda()
if not len(I.size()) < 3:
out = torch.cat((out, I), dim=1)
else:
I = torch.LongTensor(l_list).view(-1, 1, 1)
I = Variable(I.expand(x.size(0), 1, self.embed_size)-1).cuda()
out = torch.gather(padded[0], 1, I).squeeze(1)
if self.self_attention:
out = self.attention_layer(out, out, mask=mask)
# normalization in the joint embedding space
if not self.no_txtnorm:
out = l2norm(out, dim=-1)
return out
def cosine_sim(im, s):
"""Cosine similarity between all the image and sentence pairs
"""
return im.mm(s.t())
def attention_sim(im, s):
im_expanded = im.unsqueeze(1).expand(
im.size(0),s.size(0),s.size(1))
no_attention_score = im_expanded * s
im_to_s_attention = nn.functional.softmax(no_attention_score, dim=2)
score = (im_to_s_attention*no_attention_score).sum(dim=2)
return score
class InstanceLoss(nn.Module):
"""
Compute instance loss
"""
def __init__(self):
super(InstanceLoss, self).__init__()
self.loss = nn.CrossEntropyLoss()
def forward(self, img_cls, txt_cls, labels):
cost_im = self.loss(img_cls, labels)
cost_s = self.loss(txt_cls, labels)
return cost_im + cost_s
class SimLoss(nn.Module):
"""
Compute contrastive loss
"""
def __init__(self, margin=0, measure=False, max_violation=False, inner_dim=0, loss_func="BCE"):
super(SimLoss, self).__init__()
self.margin = margin
self.measure = measure
if measure == 'cosine':
self.sim = cosine_sim
elif measure == 'attention':
self.sim = attention_sim
elif measure == 'cross_attention':
self.sim = CrossAttention(inner_dim, 4, -1)
elif measure == 'cross_attention_new':
self.sim = CrossAttentionNew(inner_dim, 4, -1)
elif measure == 'gate_fusion':
self.sim = GatedFusion(inner_dim, 4, 0.0)
elif measure == 'gate_fusion_new':
self.sim = GatedFusionNew(inner_dim, 4, 0.0)
else:
self.sim = cosine_sim
self.loss_func = loss_func
self.max_violation = max_violation
def forward(self, im, s, get_score=False, keep="words", mask=None):
# compute image-sentence score matrix
if self.measure == 'cosine':
cur_im = im
cur_s = s
drive_num = torch.cuda.device_count()
if keep == "words":
cur_s = s.unsqueeze(0).expand(drive_num, -1, -1, -1)
elif keep == "regions":
cur_im = im.unsqueeze(0).expand(drive_num, -1, -1, -1)
scores = self.sim(cur_im, cur_s, keep=keep, ret_dot=True)
if keep == "regions":
scores = scores.transpose(0, 1)
elif self.measure == 'cross_attention' or self.measure == 'cross_attention_new':
cur_im = im
cur_s = s
cur_mask = mask
drive_num = torch.cuda.device_count()
if keep == "words":
cur_s = s.unsqueeze(0).expand(drive_num, -1, -1, -1)
cur_mask = mask.unsqueeze(0).expand(drive_num, -1, -1)
elif keep == "regions":
cur_im = im.unsqueeze(0).expand(drive_num, -1, -1, -1)
scores = self.sim(cur_im, cur_s, keep=keep, mask=cur_mask)
if keep == "regions":
scores = scores.transpose(0, 1)
elif self.measure == 'gate_fusion' or self.measure == 'gate_fusion_new':
cur_im = im
cur_s = s
cur_mask = mask
drive_num = torch.cuda.device_count()
if keep == "words":
cur_s = s.unsqueeze(0).expand(min(im.size(0), drive_num), -1, -1, -1)
cur_mask = mask.unsqueeze(0).expand(min(im.size(0), drive_num), -1, -1)
elif keep == "regions":
cur_im = im.unsqueeze(0).expand(drive_num, -1, -1, -1)
scores = self.sim(cur_im, cur_s, keep=keep, mask=cur_mask)
if keep == "regions":
scores = scores.transpose(0, 1)
else:
scores = self.sim(im, s)
if get_score:
return scores
if self.loss_func == 'BCE':
eps = 0.000001
scores = scores.clamp(min=eps, max=(1.0-eps))
de_scores = 1.0 - scores
label = Variable(torch.eye(scores.size(0))).cuda()
de_label = 1 - label
scores = torch.log(scores) * label
de_scores = torch.log(de_scores) * de_label
if self.max_violation:
le = -(scores.sum() + scores.sum() + de_scores.min(1)[0].sum() + de_scores.min(0)[0].sum())
else:
le = -(scores.diag().mean() + de_scores.mean())
return le
else:
diagonal = scores.diag().view(im.size(0), 1)
d1 = diagonal.expand_as(scores)
d2 = diagonal.t().expand_as(scores)
# compare every diagonal score to scores in its column
# caption retrieval
cost_s = (self.margin + scores - d1).clamp(min=0)
# compare every diagonal score to scores in its row
# image retrieval
cost_im = (self.margin + scores - d2).clamp(min=0)
# clear diagonals
mask = torch.eye(scores.size(0)) > .5
I = Variable(mask)
if torch.cuda.is_available():
I = I.cuda()
cost_s = cost_s.masked_fill_(I, 0)
cost_im = cost_im.masked_fill_(I, 0)
# keep the maximum violating negative for each query
if self.max_violation:
cost_s = cost_s.max(1)[0]
cost_im = cost_im.max(0)[0]
return cost_s.sum() + cost_im.sum()
class CAMP(object):
"""
rkiros/uvs model
"""
def __init__(self, opt):
# Build Models
self.opt = opt
self.grad_clip = opt.grad_clip
self.img_enc = EncoderImage(opt.data_name, opt.img_dim, opt.embed_size,
opt.finetune, opt.cnn_type,
no_imgnorm=opt.no_imgnorm,
self_attention=opt.self_attention)
self.txt_enc = EncoderText(opt.vocab_size, opt.word_dim,
opt.embed_size, opt.num_layers,
no_txtnorm=opt.no_txtnorm,
self_attention=opt.self_attention,
embed_weights=opt.word_embed,
bi_gru=opt.bi_gru)
# Loss and Optimizer
if opt.cross_model:
self.criterion = SimLoss(margin=opt.margin,
measure=opt.measure,
max_violation=opt.max_violation,
inner_dim=opt.embed_size)
else:
self.criterion = SimLoss(margin=opt.margin,
measure=opt.measure,
max_violation=opt.max_violation)
if torch.cuda.is_available():
self.img_enc = nn.DataParallel(self.img_enc)
self.txt_enc = nn.DataParallel(self.txt_enc)
self.img_enc.cuda()
self.txt_enc.cuda()
if opt.cross_model:
self.criterion.sim = nn.DataParallel(self.criterion.sim)
self.criterion.sim.cuda()
cudnn.benchmark = True
print("Encoders init OK!")
params = list(self.txt_enc.parameters())
params += list(self.img_enc.module.fc.parameters())
if opt.self_attention:
params += list(self.img_enc.module.attention_layer.parameters())
if opt.finetune:
params += list(self.img_enc.module.cnn.parameters())
if opt.cross_model:
params += list(self.criterion.sim.parameters())
if opt.measure == "gate_fusion" and not opt.finetune_gate:
print("Only fc layers and final aggregation layers optimized.")
params = list(self.criterion.sim.module.fc_1.parameters())
params += list(self.criterion.sim.module.fc_2.parameters())
params += list(self.criterion.sim.module.fc_out.parameters())
params += list(self.criterion.sim.module.reduce_layer_1.parameters())
params += list(self.criterion.sim.module.reduce_layer_2.parameters())
if opt.measure == "gate_fusion_new" and not opt.finetune_gate:
print("Only fc layers and final aggregation layers optimized.")
params = list(self.criterion.sim.module.fc_1.parameters())
params += list(self.criterion.sim.module.fc_2.parameters())
#params += list(self.criterion.sim.module.fc_gate_1.parameters())
#params += list(self.criterion.sim.module.fc_gate_2.parameters())
params += list(self.criterion.sim.module.fc_out.parameters())
params += list(self.criterion.sim.module.final_reduce_1.parameters())
params += list(self.criterion.sim.module.final_reduce_2.parameters())
if opt.embed_mask:
self.embed_mask = np.load(opt.embed_mask)
else:
self.embed_mask = None
self.params = params
if opt.optimizer.type == "Adam":
self.optimizer = torch.optim.Adam(params, lr=opt.learning_rate)
elif opt.optimizer.type == "SGD":
self.optimizer = torch.optim.SGD(params, lr=opt.learning_rate,
momentum=opt.optimizer.momentum,
weight_decay=opt.optimizer.weight_decay,
nesterov=opt.optimizer.nesterov)
else:
raise NotImplementedError('Only support Adam and SGD optimizer.')
self.Eiters = 0
print("Model init OK!")
def state_dict(self):
state_dict = [self.img_enc.state_dict(), self.txt_enc.state_dict()]
if self.opt.cross_model:
state_dict += [self.criterion.sim.state_dict()]
return state_dict
def load_state_dict(self, state_dict):
new_state_dict = OrderedDict()
for k, v in state_dict[0].items():
new_state_dict[k] = v
self.img_enc.load_state_dict(new_state_dict, strict=True)
new_state_dict = OrderedDict()
for k, v in state_dict[1].items():
#name = k.replace('module.', '') # remove `module.`
new_state_dict[k] = v
self.txt_enc.load_state_dict(new_state_dict, strict=True)
new_state_dict = OrderedDict()
if len(state_dict)>2:
new_state_dict = OrderedDict()
for k, v in state_dict[2].items():
#name = k.replace('module.', '') # remove `module.`
new_state_dict[k] = v
self.criterion.sim.load_state_dict(new_state_dict, strict=False)
new_state_dict = OrderedDict()
def train_start(self):
"""switch to train mode
"""
self.img_enc.train()
self.txt_enc.train()
if self.opt.cross_model:
self.criterion.sim.train()
def val_start(self):
"""switch to evaluate mode
"""
self.img_enc.eval()
self.txt_enc.eval()
if self.opt.cross_model:
self.criterion.sim.eval()
def forward_emb(self, images, captions, lengths, volatile=False):
"""Compute the image and caption embeddings
"""
# Set mini-batch dataset
images = Variable(images, volatile=volatile)
captions = Variable(captions, volatile=volatile)
lengths = Variable(lengths, volatile=volatile)
if torch.cuda.is_available():
images = images.cuda()
captions = captions.cuda()
lengths = lengths.cuda()
# Forward
img_emb = self.img_enc(images)
cap_emb = self.txt_enc(captions, lengths)
return img_emb, cap_emb
def forward_loss(self, img_emb, cap_emb, instance_ids, mask=None, **kwargs):
"""Compute the loss given pairs of image and caption embeddings
"""
loss = self.criterion(img_emb, cap_emb, mask=mask)
loss = loss #/ self.opt.batch_size
self.logger.update('Le', loss.data, img_emb.size(0))
return loss
def train_emb(self, images, captions, lengths, ids=None,
instance_ids=None, *args):
"""One training step given images and captions.
"""
self.Eiters += 1
self.logger.update('Eit', self.Eiters)
self.logger.update('lr', self.optimizer.param_groups[0]['lr'])
# compute the embeddings
img_emb, cap_emb = self.forward_emb(images, captions, lengths)
# measure accuracy and record loss
self.optimizer.zero_grad()
l_list = [int(i) for i in lengths]
mask = Variable(torch.ByteTensor([i*[1] + (max_length+3-i)*[0] for i in l_list])).cuda()
loss = self.forward_loss(img_emb, cap_emb, instance_ids, mask)
# compute gradient and do optimization
loss.backward()
if self.grad_clip > 0:
clip_grad_norm(self.params, self.grad_clip)
if self.embed_mask is not None:
for i, mask in enumerate(self.embed_mask):
if mask:
self.txt_enc.module.embed.weight.grad.data[i].zero_()
self.optimizer.step()
================================================
FILE: pick_wordvec.py
================================================
import pickle
import argparse
import numpy as np
from vocab import Vocabulary
from gensim.models import KeyedVectors
def main(opt):
vocab = pickle.load(open(opt.vocab_path, 'rb'))
num = len(vocab)
print (num)
model = KeyedVectors.load_word2vec_format(opt.embed_weight, binary=True)
matrix_len = num
weights_matrix = np.zeros((num, 300))
words_found = 0
mask = np.zeros(num, dtype=int)
for i, word in enumerate(vocab.idx2word):
try:
weights_matrix[i] = model[vocab.idx2word[i]]
words_found += 1
mask[i] = 1
except KeyError:
weights_matrix[i] = np.random.normal(scale=0.1, size=(300, ))
print (words_found)
np.save("./embed/f30kword2vec300dim_3.npy", weights_matrix)
np.save("./embed/f30kword2vecmask_3.npy", mask)
if __name__ == '__main__':
parser = argparse.ArgumentParser()
parser.add_argument('--vocab_path', default='')
parser.add_argument('--embed_weight', default='')
opt = parser.parse_args()
main(opt)
================================================
FILE: test.py
================================================
from vocab import Vocabulary
import evaluation
import pickle
evaluation.evalrank("",
data_path="./data", split="test",
fold5=True)
"""print (rt,rti)
print(len(rt),len(rti))
dic_now = {}
dic_now["rt_ranks"]=rt[0]
dic_now["rt_top1"]=rt[1]
dic_now["rti_ranks"] =rti[0]
dic_now["rti_top1"]=rti[1]
with open('vsepp' + '.results.pickle', 'wb') as handle:
pickle.dump(dic_now, handle, protocol=pickle.HIGHEST_PROTOCOL)"""
================================================
FILE: test_modules.py
================================================
print("work start!")
import torch
print(torch.__version__)
#import tensorboard_logger as tb_logger
print("import logger OK!")
import torch.nn as nn
import torch.distributed as dist
import torch.nn.init
import torchvision.models as models
import torchvision.transforms as transforms
from torch.autograd import Variable
from torch.nn.utils.rnn import pack_padded_sequence, pad_packed_sequence
from torch.nn.utils.clip_grad import clip_grad_norm
import numpy as np
from collections import OrderedDict
import yaml
from easydict import EasyDict
print("import all torch OK!")
#from transformer.Models import Encoder as self_attention_encoder
#from transformer.Layers import EncoderLayer as attention_layer
#print("import transformer OK!")
from torch.utils.data import DataLoader
from torch.utils.data.distributed import DistributedSampler
import logging
import torch.backends.cudnn as cudnn
import pickle
import os
from evaluation import i2t, t2i, AverageMeter, LogCollector, encode_data
import data
from model import ImageSelfAttention
import model
from vocab import Vocabulary
import argparse
from fusion_module import *
def test_img_self_att():
fake_feature = Variable(torch.randn(16, 32*7*7))
fake_feature = fake_feature.view(16, -1, 7, 7)
img_self_attention = ImageSelfAttention(32)
out = img_self_attention(fake_feature)
print(out.size())
def test_f30k_dataloader():
data_name = "f30k"
data_path = "./data/f30k"
vocab_path = "./vocab/"
vocab = pickle.load(open(os.path.join(vocab_path,
'%s_vocab.pkl' % data_name), 'rb'))
roots, ids = data.get_paths(data_path, data_name, False)
transform = transforms.Compose([transforms.RandomSizedCrop(224),
transforms.ToTensor()])
print (roots, ids)
train_loader = data.get_loader_single(data_name, "train", # !!!
roots["train"]["img"],
roots["train"]["cap"],
vocab, transform, ids=ids["train"],
batch_size=16, shuffle=False,
num_workers=1,
collate_fn=data.collate_fn,
distributed=False)
print ("f30k dataloader output:", train_loader.dataset.img_num)
#for (id, x) in enumerate(train_loader):
#if id > 0 : break
#print (id, x)
def test_text_encoder():
data_name = "f30k_precomp"
data_path = "./data/"
vocab_path = "./vocab/"
vocab = pickle.load(open(os.path.join(vocab_path,
'%s_vocab.pkl' % data_name), 'rb'))
vocab_size = len(vocab)
print(vocab_size)
word_dim = 10
embed_size = 20
num_layers = 1
txt_enc = model.EncoderText(vocab_size, word_dim, embed_size, num_layers,
bi_gru=True, self_attention=True)
txt_enc = nn.DataParallel(txt_enc)
txt_enc.cuda()
fake_text = Variable(torch.ones(16, 50).long())
fake_lengths = Variable(torch.Tensor([16-i for i in range(16)]).long())
out = txt_enc(fake_text, fake_lengths)
print ("txt_enc output:", out.size())
def test_img_encoder():
embed_size = 20
img_enc = model.EncoderImage("f30k_precomp", 20, 20, False, self_attention=True)
img_enc = nn.DataParallel(img_enc)
img_enc.cuda()
fake_img = Variable(torch.ones(16, 3, 20))
out = img_enc(fake_img)
print ("img_enc output:", out.size())
def test_stack_fusion():
fusion_module = CrossAttention(32, 2, -1)
print("CrossAttention init success!")
fake_img = Variable(torch.randn(16, 49, 32))
fake_txt = Variable(torch.randn(8, 14, 32))
score = fusion_module(fake_img, fake_txt, get_score=True)
print(score.size())
print("----CrossAttention module success!----")
def test_stack_fusion_new():
fusion_module = CrossAttentionNew(32, 2, -1)
print("CrossAttention init success!")
fake_img = Variable(torch.randn(16, 49, 32))
fake_txt = Variable(torch.randn(8, 14, 32))
score = fusion_module(fake_img, fake_txt, get_score=True)
print(score.size())
print("----CrossAttention module success!----")
def test_gate_fusion():
fusion_module = GatedFusion(32, 2, 0.0)
print("FusionModule init success!")
fake_img = Variable(torch.randn(16, 49, 32))
fake_txt = Variable(torch.randn(8, 14, 32))
score = fusion_module(fake_img, fake_txt, get_score=True)
print(score.size())
print("----GatedFusion module success!----")
def test_gate_fusion_new():
fusion_module = GatedFusionNew(32, 2, 0.0)
print("FusionModule init success!")
fake_img = Variable(torch.randn(16, 49, 32))
fake_txt = Variable(torch.randn(8, 14, 32))
score = fusion_module(fake_img, fake_txt, get_score=True)
print(score.size())
print("----GatedFusion module success!----")
def test_CAMP_model(config_path):
print("OK!")
logging.basicConfig(format='%(asctime)s %(message)s', level=logging.INFO)
parser = argparse.ArgumentParser()
#config_path = "./experiments/f30k_cross_attention/config_test.yaml"
with open(config_path) as f:
opt = yaml.load(f)
opt = EasyDict(opt['common'])
vocab = pickle.load(open(os.path.join(opt.vocab_path,
'%s_vocab.pkl' % opt.data_name), 'rb'))
opt.vocab_size = len(vocab)
train_logger = LogCollector()
print("----Start init model----")
CAMP = model.CAMP(opt)
CAMP.logger = train_logger
if opt.resume is not None:
ckp = torch.load(opt.resume)
CAMP.load_state_dict(ckp["model"])
CAMP.train_start()
print("----Model init success----")
"""
fake_img = torch.randn(16, 36, opt.img_dim)
fake_text = torch.ones(16, 32).long()
fake_lengths = torch.Tensor([32] * 16)
fake_pos = torch.ones(16, 32).long()
fake_ids = torch.ones(16).long()
CAMP.train_emb(fake_img, fake_text, fake_lengths,
instance_ids=fake_ids)
print("----Test train_emb success----")
"""
train_loader, val_loader = data.get_loaders(
opt.data_name, vocab, opt.crop_size, 128, 4, opt)
test_loader = data.get_test_loader("test", opt.data_name, vocab, opt.crop_size, 128, 4, opt)
CAMP.val_start()
img_embs, cap_embs, cap_masks = encode_data(
CAMP, test_loader, opt.log_step, logging.info)
(r1, r5, r10, medr, meanr), (r1i, r5i, r10i, medri, meanri), score_matrix= i2t(img_embs, cap_embs, cap_masks, measure=opt.measure,
model=CAMP, return_ranks=True)
logging.info("Image to text: %.1f, %.1f, %.1f, %.1f, %.1f" %
(r1, r5, r10, medr, meanr))
logging.info("Text to image: %.1f, %.1f, %.1f, %.1f, %.1f" %
(r1i, r5i, r10i, medri, meanri))
def main():
#test_f30k_dataloader()
#test_text_encoder()
#test_img_encoder()
#test_stack_fusion()
#test_gate_fusion()
#test_stack_fusion_new()
#test_gate_fusion_new()
test_CAMP_model("./experiments/f30k_cross_attention/config_test.yaml")
if __name__ == '__main__':
main()
================================================
FILE: train.py
================================================
import pickle
import os
import time
import shutil
import torch
import yaml
from easydict import EasyDict
import data
from vocab import Vocabulary # NOQA
from model import CAMP
from evaluation import i2t, t2i, AverageMeter, LogCollector, encode_data
import logging
import tensorboard_logger as tb_logger
import argparse
def main():
# Hyper Parameters
parser = argparse.ArgumentParser()
parser.add_argument('--config', default='',
help='Config path.')
args = parser.parse_args()
with open(args.config) as f:
opt = yaml.load(f)
opt = EasyDict(opt['common'])
opt.learning_rate = opt.learning_rate * (128.0/opt.batch_size)
print(opt)
logging.basicConfig(format='%(asctime)s %(message)s', level=logging.INFO)
tb_logger.configure(opt.logger_name, flush_secs=5)
# Load Vocabulary Wrapper
vocab = pickle.load(open(os.path.join(
opt.vocab_path, '%s_vocab.pkl' % opt.data_name), 'rb'))
opt.vocab_size = len(vocab)
opt.distributed = False
# Load data loaders
train_loader, val_loader = data.get_loaders(
opt.data_name, vocab, opt.crop_size, opt.batch_size, opt.workers, opt)
print(len(train_loader), len(val_loader), opt.batch_size)
# Construct the model
model = CAMP(opt)
# Train the Model
best_rsum = 0
# optionally resume from a checkpoint
if opt.resume:
if os.path.isfile(opt.resume):
print("=> loading checkpoint '{}'".format(opt.resume))
checkpoint = torch.load(opt.resume)
start_epoch = checkpoint['epoch']
best_rsum = checkpoint['best_rsum']
model.load_state_dict(checkpoint['model'])
# Eiters is used to show logs as the continuation of another
# training
model.Eiters = checkpoint['Eiters']
print("=> loaded checkpoint '{}' (epoch {}, best_rsum {})"
.format(opt.resume, start_epoch, best_rsum))
else:
print("=> no checkpoint found at '{}'".format(opt.resume))
for epoch in range(opt.num_epochs):
adjust_learning_rate(opt, model.optimizer, epoch)
# train for one epoch
train(opt, train_loader, model, epoch, val_loader, tb_logger)
if epoch % opt.val_epoc == 0:
# evaluate on validation set
rsum = validate(opt, val_loader, model, tb_logger)
# remember best R@ sum and save checkpoint
is_best = rsum > best_rsum
best_rsum = max(rsum, best_rsum)
save_checkpoint({
'epoch': epoch + 1,
'model': model.state_dict(),
'best_rsum': best_rsum,
'opt': opt,
'Eiters': model.Eiters,
}, is_best, filename='checkpoint_'+ str(epoch) +'.pth.tar', prefix=opt.logger_name + '/')
def train(opt, train_loader, model, epoch, val_loader, tb_logger):
print("start to train")
# average meters to record the training statistics
batch_time = AverageMeter()
data_time = AverageMeter()
train_logger = LogCollector()
# switch to train mode
model.train_start()
end = time.time()
print("start loading data...")
for i, train_data in enumerate(train_loader):
# measure data loading time
data_time.update(time.time() - end)
# make sure train logger is used
model.logger = train_logger
# Update the model
model.train_emb(*train_data)
# measure elapsed time
batch_time.update(time.time() - end)
end = time.time()
# Print log info
if model.Eiters % opt.log_step == 0:
logging.info(
'Epoch: [{0}][{1}/{2}]\t'
'{e_log}\t'
'Time {batch_time.val:.3f} ({batch_time.avg:.3f})\t'
'Data {data_time.val:.3f} ({data_time.avg:.3f})\t'
.format(
epoch, i, len(train_loader), batch_time=batch_time,
data_time=data_time, e_log=str(model.logger)))
# Record logs in tensorboard
tb_logger.log_value('epoch', epoch, step=model.Eiters)
tb_logger.log_value('step', i, step=model.Eiters)
tb_logger.log_value('batch_time', batch_time.val, step=model.Eiters)
tb_logger.log_value('data_time', data_time.val, step=model.Eiters)
model.logger.tb_log(tb_logger, step=model.Eiters)
# validate at every val_step
#if model.Eiters % opt.val_step == 0:
# validate(opt, val_loader, model, tb_logger)
# switch to train mode
# model.train_start()
def validate(opt, val_loader, model, tb_logger):
# compute the encoding for all the validation images and captions
print("start validate")
model.val_start()
img_embs, cap_embs, cap_masks = encode_data(
model, val_loader, opt.log_step, logging.info)
# caption retrieval
(i2t_r1, i2t_r5, i2t_r10, i2t_medr, i2t_meanr), (t2i_r1, t2i_r5, t2i_r10, t2i_medr, t2i_meanr) = i2t(img_embs, cap_embs, cap_masks, measure=opt.measure, model=model)
logging.info("Image to text: %.1f, %.1f, %.1f, %.1f, %.1f" %
(i2t_r1, i2t_r5, i2t_r10, i2t_medr, i2t_meanr))
# image retrieval
#(r1i, r5i, r10i, medri, meanr) = t2i(
# img_embs, cap_embs, measure=opt.measure, model=model)
logging.info("Text to image: %.1f, %.1f, %.1f, %.1f, %.1f" %
(t2i_r1, t2i_r5, t2i_r10, t2i_medr, t2i_meanr))
# sum of recalls to be used for early stopping
currscore = i2t_r1 + i2t_r5 + i2t_r10 + t2i_r1 + t2i_r5 + t2i_r10
# record metrics in tensorboard
tb_logger.log_value('i2t_r1', i2t_r1, step=model.Eiters)
tb_logger.log_value('i2t_r5', i2t_r5, step=model.Eiters)
tb_logger.log_value('i2t_r10', i2t_r10, step=model.Eiters)
tb_logger.log_value('i2t_medr', i2t_medr, step=model.Eiters)
tb_logger.log_value('i2t_meanr', i2t_meanr, step=model.Eiters)
tb_logger.log_value('t2i_r1', t2i_r1, step=model.Eiters)
tb_logger.log_value('t2i_r5', t2i_r5, step=model.Eiters)
tb_logger.log_value('t2i_r10', t2i_r10, step=model.Eiters)
tb_logger.log_value('t2i_medr', t2i_medr, step=model.Eiters)
tb_logger.log_value('t2i_meanr', t2i_meanr, step=model.Eiters)
tb_logger.log_value('rsum', currscore, step=model.Eiters)
return currscore
def save_checkpoint(state, is_best, filename='checkpoint.pth.tar', prefix=''):
torch.save(state, prefix + filename)
if is_best:
shutil.copyfile(prefix + filename, prefix + 'model_best.pth.tar')
def adjust_learning_rate(opt, optimizer, epoch):
"""Sets the learning rate to the initial LR
decayed by 10 every 30 epochs"""
lr = opt.learning_rate * (0.1 ** (epoch // opt.lr_update))
for param_group in optimizer.param_groups:
param_group['lr'] = lr
def accuracy(output, target, topk=(1,)):
"""Computes the precision@k for the specified values of k"""
maxk = max(topk)
batch_size = target.size(0)
_, pred = output.topk(maxk, 1, True, True)
pred = pred.t()
correct = pred.eq(target.view(1, -1).expand_as(pred))
res = []
for k in topk:
correct_k = correct[:k].view(-1).float().sum(0)
res.append(correct_k.mul_(100.0 / batch_size))
return res
if __name__ == '__main__':
main()
================================================
FILE: vocab.py
================================================
# Create a vocabulary wrapper
import nltk
import pickle
from collections import Counter
from pycocotools.coco import COCO
import json
import argparse
import os
annotations = {
'coco_precomp': ['train_caps.txt', 'dev_caps.txt'],
'coco': ['annotations/captions_train2014.json',
'annotations/captions_val2014.json'],
'f8k_precomp': ['train_caps.txt', 'dev_caps.txt'],
'10crop_precomp': ['train_caps.txt', 'dev_caps.txt'],
'f30k_precomp': ['train_caps.txt', 'dev_caps.txt'],
'f8k': ['dataset_flickr8k.json'],
'f30k': ['dataset_flickr30k.json'],
'CUHK-PEDES': ['reid_raw.json'],
}
class Vocabulary(object):
"""Simple vocabulary wrapper."""
def __init__(self):
self.word2idx = {}
self.idx2word = {}
self.idx = 0
def add_word(self, word):
if word not in self.word2idx:
self.word2idx[word] = self.idx
self.idx2word[self.idx] = word
self.idx += 1
def __call__(self, word):
if word not in self.word2idx:
return self.word2idx['<unk>']
return self.word2idx[word]
def __len__(self):
return len(self.word2idx)
def from_coco_json(path):
coco = COCO(path)
ids = coco.anns.keys()
captions = []
for i, idx in enumerate(ids):
captions.append(str(coco.anns[idx]['caption']))
return captions
def from_flickr_json(path):
dataset = json.load(open(path, 'r'))['images']
captions = []
for i, d in enumerate(dataset):
captions += [str(x['raw']) for x in d['sentences']]
return captions
def from_CUHK_json(path):
dataset = [x["captions"] for x in json.load(open(path, "r"))]
captions = []
for i, d in enumerate(dataset):
captions += [str(x) for x in d]
return captions
def from_txt(txt):
captions = []
with open(txt, 'rb') as f:
for line in f:
captions.append(line.strip())
return captions
def build_vocab(data_path, data_name, jsons, threshold):
"""Build a simple vocabulary wrapper."""
counter = Counter()
for path in jsons[data_name]:
full_path = os.path.join(os.path.join(data_path, data_name), path)
if data_name == 'coco':
captions = from_coco_json(full_path)
elif data_name == 'f8k' or data_name == 'f30k':
captions = from_flickr_json(full_path)
elif data_name == 'CUHK-PEDES':
captions = from_CUHK_json(full_path)
else:
captions = from_txt(full_path)
for i, caption in enumerate(captions):
tokens = nltk.tokenize.word_tokenize(
caption.lower().decode('utf-8'))
counter.update(tokens)
if i % 1000 == 0:
print("[%d/%d] tokenized the captions." % (i, len(captions)))
# Discard if the occurrence of the word is less than min_word_cnt.
words = [word for word, cnt in counter.items() if cnt >= threshold]
# Create a vocab wrapper and add some special tokens.
vocab = Vocabulary()
vocab.add_word('<pad>')
vocab.add_word('<start>')
vocab.add_word('<end>')
vocab.add_word('<unk>')
# Add words to the vocabulary.
for i, word in enumerate(words):
vocab.add_word(word)
return vocab
def main(data_path, data_name):
vocab = build_vocab(data_path, data_name, jsons=annotations, threshold=4)
with open('./vocab/%s_vocab.pkl' % data_name, 'wb') as f:
pickle.dump(vocab, f, pickle.HIGHEST_PROTOCOL)
print("Saved vocabulary file to ", './vocab/%s_vocab.pkl' % data_name)
if __name__ == '__main__':
parser = argparse.ArgumentParser()
parser.add_argument('--data_path', default='/w/31/faghri/vsepp_data/')
parser.add_argument('--data_name', default='coco',
help='{coco,f8k,f30k,10crop}_precomp|coco|f8k|f30k')
opt = parser.parse_args()
main(opt.data_path, opt.data_name)
gitextract_bd4r4mqp/ ├── LICENSE ├── README.md ├── data.py ├── embed/ │ ├── cocoword2vec300dim.npy │ ├── cocoword2vecmask.npy │ ├── f30kword2vec300dim_3.npy │ └── f30kword2vecmask_3.npy ├── evaluation.py ├── experiments/ │ ├── f30k_cross_attention/ │ │ ├── config.yaml │ │ ├── config_256.yaml │ │ └── config_test.yaml │ └── f30k_gate_fusion/ │ ├── config_finetune.yaml │ ├── config_fixatt.yaml │ └── config_test.yaml ├── fusion_module.py ├── model.py ├── pick_wordvec.py ├── test.py ├── test_modules.py ├── train.py ├── vocab/ │ ├── 10crop_precomp_vocab.pkl │ ├── CUHK-PEDES_vocab.pkl │ ├── coco_precomp_vocab.pkl │ ├── coco_precomp_vocab_1.pkl │ ├── coco_vocab.pkl │ ├── f30k_precomp_vocab.pkl │ ├── f30k_precomp_vocab_1.pkl │ ├── f30k_vocab.pkl │ ├── f8k_precomp_vocab.pkl │ └── f8k_vocab.pkl └── vocab.py
SYMBOL INDEX (124 symbols across 8 files)
FILE: data.py
function get_paths (line 16) | def get_paths(path, name="coco", use_restval=False):
class CUHKDataset (line 90) | class CUHKDataset(data.Dataset):
method __init__ (line 92) | def __init__(self, root, json, split, vocab, transform=None):
method __getitem__ (line 103) | def __getitem__(self, index):
method __len__ (line 123) | def __len__(self):
class CocoDataset (line 126) | class CocoDataset(data.Dataset):
method __init__ (line 129) | def __init__(self, root, json, vocab, transform=None, ids=None):
method __getitem__ (line 159) | def __getitem__(self, index):
method get_raw_item (line 179) | def get_raw_item(self, index):
method __len__ (line 194) | def __len__(self):
class FlickrDataset (line 198) | class FlickrDataset(data.Dataset):
method __init__ (line 216) | def __init__(self, root, json, split, vocab, transform=None):
method __getitem__ (line 229) | def __getitem__(self, index):
method __len__ (line 254) | def __len__(self):
class PrecompDataset (line 258) | class PrecompDataset(data.Dataset):
method __init__ (line 264) | def __init__(self, data_path, data_split, vocab):
method __getitem__ (line 287) | def __getitem__(self, index):
method __len__ (line 306) | def __len__(self):
function collate_fn (line 310) | def collate_fn(data):
function get_loader_single (line 342) | def get_loader_single(data_name, split, root, json, vocab, transform,
function get_precomp_loader (line 387) | def get_precomp_loader(data_path, data_split, vocab, opt, batch_size=100,
function get_transform (line 400) | def get_transform(data_name, split_name, opt):
function get_loaders (line 423) | def get_loaders(data_name, vocab, crop_size, batch_size, workers, opt):
function get_test_loader (line 457) | def get_test_loader(split_name, data_name, vocab, crop_size, batch_size,
FILE: evaluation.py
class AverageMeter (line 21) | class AverageMeter(object):
method __init__ (line 24) | def __init__(self):
method reset (line 27) | def reset(self):
method update (line 33) | def update(self, val, n=0):
method __str__ (line 39) | def __str__(self):
class LogCollector (line 49) | class LogCollector(object):
method __init__ (line 52) | def __init__(self):
method update (line 56) | def update(self, k, v, n=0):
method __str__ (line 62) | def __str__(self):
method tb_log (line 72) | def tb_log(self, tb_logger, prefix='', step=None):
function encode_data (line 79) | def encode_data(model, data_loader, log_step=10, logging=print):
function evalrank (line 144) | def evalrank(model_path, data_path=None, split='dev', fold5=False, retur...
function i2t (line 237) | def i2t(images, captions, masks, npts=None, measure='cosine', return_ran...
function t2i (line 384) | def t2i(images, captions, npts=None, measure='cosine', return_ranks=False,
FILE: fusion_module.py
function l2norm (line 22) | def l2norm(X, dim=-1, eps=1e-8):
function sum_attention (line 30) | def sum_attention(nnet, query, value, mask=None, dropout=None):
function qkv_attention (line 40) | def qkv_attention(query, key, value, mask=None, dropout=None):
class SummaryAttn (line 51) | class SummaryAttn(nn.Module):
method __init__ (line 53) | def __init__(self, dim, num_attn, dropout, is_cat=False):
method forward (line 65) | def forward(self, query, value, mask=None):
class CrossAttention (line 75) | class CrossAttention(nn.Module):
method __init__ (line 78) | def __init__(self, dim, num_attn, dropout, reduce_func="self_attn"):
method init_weights (line 98) | def init_weights(self):
method forward (line 106) | def forward(self, v1, v2, get_score=True, keep=None, mask=None):
class GatedFusion (line 155) | class GatedFusion(nn.Module):
method __init__ (line 156) | def __init__(self, dim, num_attn, dropout=0.01, reduce_func="self_attn...
method init_weights (line 203) | def init_weights(self):
method forward (line 219) | def forward(self, v1, v2, get_score=True, keep=None, mask=None):
class CrossAttentionNew (line 282) | class CrossAttentionNew(nn.Module):
method __init__ (line 285) | def __init__(self, dim, num_attn, dropout, reduce_func="mean"):
method init_weights (line 314) | def init_weights(self):
method forward (line 322) | def forward(self, v1, v2, get_score=True, keep=None, mask=None):
class GatedFusionNew (line 387) | class GatedFusionNew(nn.Module):
method __init__ (line 388) | def __init__(self, dim, num_attn, dropout=0.01, reduce_func="self_attn...
method init_weights (line 462) | def init_weights(self):
method forward (line 478) | def forward(self, v1, v2, get_score=True, keep=None, mask=None):
FILE: model.py
function l2norm (line 25) | def l2norm(X, dim=-1, eps=1e-8):
function EncoderImage (line 32) | def EncoderImage(data_name, img_dim, embed_size, finetune=False,
class ImageSelfAttention (line 50) | class ImageSelfAttention(nn.Module):
method __init__ (line 54) | def __init__(self, planes):
method forward (line 61) | def forward(self, x):
class EncoderImageFull (line 72) | class EncoderImageFull(nn.Module):
method __init__ (line 74) | def __init__(self, embed_size, finetune=False, cnn_type='resnet152',
method get_cnn (line 105) | def get_cnn(self, arch, pretrained, fusion):
method init_weights (line 126) | def init_weights(self):
method forward (line 134) | def forward(self, images):
class EncoderImagePrecomp (line 163) | class EncoderImagePrecomp(nn.Module):
method __init__ (line 165) | def __init__(self, img_dim, embed_size, no_imgnorm=False,
method _init_weights (line 178) | def _init_weights(self):
method forward (line 186) | def forward(self, images):
method load_state_dict (line 200) | def load_state_dict(self, state_dict):
class EncoderText (line 214) | class EncoderText(nn.Module):
method __init__ (line 216) | def __init__(self, vocab_size, word_dim, embed_size, num_layers,
method _init_weights (line 238) | def _init_weights(self, embed_weights=''):
method forward (line 247) | def forward(self, x, lengths):
function cosine_sim (line 290) | def cosine_sim(im, s):
function attention_sim (line 296) | def attention_sim(im, s):
class InstanceLoss (line 304) | class InstanceLoss(nn.Module):
method __init__ (line 309) | def __init__(self):
method forward (line 313) | def forward(self, img_cls, txt_cls, labels):
class SimLoss (line 319) | class SimLoss(nn.Module):
method __init__ (line 324) | def __init__(self, margin=0, measure=False, max_violation=False, inner...
method forward (line 346) | def forward(self, im, s, get_score=False, keep="words", mask=None):
class CAMP (line 446) | class CAMP(object):
method __init__ (line 451) | def __init__(self, opt):
method state_dict (line 539) | def state_dict(self):
method load_state_dict (line 545) | def load_state_dict(self, state_dict):
method train_start (line 566) | def train_start(self):
method val_start (line 574) | def val_start(self):
method forward_emb (line 582) | def forward_emb(self, images, captions, lengths, volatile=False):
method forward_loss (line 601) | def forward_loss(self, img_emb, cap_emb, instance_ids, mask=None, **kw...
method train_emb (line 609) | def train_emb(self, images, captions, lengths, ids=None,
FILE: pick_wordvec.py
function main (line 9) | def main(opt):
FILE: test_modules.py
function test_img_self_att (line 43) | def test_img_self_att():
function test_f30k_dataloader (line 50) | def test_f30k_dataloader():
function test_text_encoder (line 74) | def test_text_encoder():
function test_img_encoder (line 101) | def test_img_encoder():
function test_stack_fusion (line 111) | def test_stack_fusion():
function test_stack_fusion_new (line 122) | def test_stack_fusion_new():
function test_gate_fusion (line 133) | def test_gate_fusion():
function test_gate_fusion_new (line 144) | def test_gate_fusion_new():
function test_CAMP_model (line 154) | def test_CAMP_model(config_path):
function main (line 210) | def main():
FILE: train.py
function main (line 21) | def main():
function train (line 92) | def train(opt, train_loader, model, epoch, val_loader, tb_logger):
function validate (line 143) | def validate(opt, val_loader, model, tb_logger):
function save_checkpoint (line 180) | def save_checkpoint(state, is_best, filename='checkpoint.pth.tar', prefi...
function adjust_learning_rate (line 186) | def adjust_learning_rate(opt, optimizer, epoch):
function accuracy (line 194) | def accuracy(output, target, topk=(1,)):
FILE: vocab.py
class Vocabulary (line 23) | class Vocabulary(object):
method __init__ (line 26) | def __init__(self):
method add_word (line 31) | def add_word(self, word):
method __call__ (line 37) | def __call__(self, word):
method __len__ (line 42) | def __len__(self):
function from_coco_json (line 46) | def from_coco_json(path):
function from_flickr_json (line 56) | def from_flickr_json(path):
function from_CUHK_json (line 64) | def from_CUHK_json(path):
function from_txt (line 73) | def from_txt(txt):
function build_vocab (line 81) | def build_vocab(data_path, data_name, jsons, threshold):
function main (line 118) | def main(data_path, data_name):
Condensed preview — 31 files, each showing path, character count, and a content snippet. Download the .json file or copy for the full structured content (126K chars).
[
{
"path": "LICENSE",
"chars": 11356,
"preview": " Apache License\n Version 2.0, January 2004\n "
},
{
"path": "README.md",
"chars": 3670,
"preview": "# Introduction\n\nThis repository is for [CAMP: Cross-Modal Adaptive Message Passing for Text-Image Retrieval](https://arx"
},
{
"path": "data.py",
"chars": 18404,
"preview": "import torch\nimport torch.utils.data as data\nimport torchvision.transforms as transforms\nimport os\nimport nltk\nfrom PIL "
},
{
"path": "evaluation.py",
"chars": 16923,
"preview": "from __future__ import print_function\nimport os\nimport pickle\n\nimport time\nimport logging\nimport numpy\nfrom data import "
},
{
"path": "experiments/f30k_cross_attention/config.yaml",
"chars": 1061,
"preview": "common:\n data_path: ../../data/\n data_name: f30k_precomp\n use_restval: True\n crop_size: 224\n vocab_path: "
},
{
"path": "experiments/f30k_cross_attention/config_256.yaml",
"chars": 1136,
"preview": "common:\n data_path: ../../data/\n data_name: f30k_precomp\n use_restval: True\n crop_size: 224\n vocab_path: "
},
{
"path": "experiments/f30k_cross_attention/config_test.yaml",
"chars": 1105,
"preview": "common:\n data_path: ./data/\n data_name: f30k_precomp\n use_restval: True\n crop_size: 224\n vocab_path: ./vo"
},
{
"path": "experiments/f30k_gate_fusion/config_finetune.yaml",
"chars": 1268,
"preview": "common:\n data_path: ../../data/\n data_name: f30k_precomp\n use_restval: True\n crop_size: 224\n vocab_path: "
},
{
"path": "experiments/f30k_gate_fusion/config_fixatt.yaml",
"chars": 1219,
"preview": "common:\n data_path: ../../data/\n data_name: f30k_precomp\n use_restval: True\n crop_size: 224\n vocab_path: "
},
{
"path": "experiments/f30k_gate_fusion/config_test.yaml",
"chars": 1134,
"preview": "common:\n data_path: ../../data/\n data_name: f30k_precomp\n use_restval: True\n crop_size: 224\n vocab_path: "
},
{
"path": "fusion_module.py",
"chars": 20668,
"preview": "import torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport torch.distributed as dist\nimport torch.nn.init"
},
{
"path": "model.py",
"chars": 23176,
"preview": "import torch\nimport torch.nn as nn\nimport torch.distributed as dist\nimport torch.nn.init\nimport torchvision.models as mo"
},
{
"path": "pick_wordvec.py",
"chars": 1056,
"preview": "import pickle\nimport argparse\nimport numpy as np\n\nfrom vocab import Vocabulary\nfrom gensim.models import KeyedVectors\n\n\n"
},
{
"path": "test.py",
"chars": 465,
"preview": "from vocab import Vocabulary\nimport evaluation\nimport pickle\n\nevaluation.evalrank(\"\",\n data_path=\"./"
},
{
"path": "test_modules.py",
"chars": 7146,
"preview": "print(\"work start!\")\nimport torch\nprint(torch.__version__)\n#import tensorboard_logger as tb_logger\nprint(\"import logger "
},
{
"path": "train.py",
"chars": 7376,
"preview": "import pickle\nimport os\nimport time\nimport shutil\n\nimport torch\nimport yaml\nfrom easydict import EasyDict\n\nimport data\nf"
},
{
"path": "vocab.py",
"chars": 3927,
"preview": "# Create a vocabulary wrapper\nimport nltk\nimport pickle\nfrom collections import Counter\nfrom pycocotools.coco import COC"
}
]
// ... and 14 more files (download for full content)
About this extraction
This page contains the full source code of the ZihaoWang-CV/CAMP_iccv19 GitHub repository, extracted and formatted as plain text for AI agents and large language models (LLMs). The extraction includes 31 files (46.7 MB), approximately 31.2k tokens, and a symbol index with 124 extracted functions, classes, methods, constants, and types. Use this with OpenClaw, Claude, ChatGPT, Cursor, Windsurf, or any other AI tool that accepts text input. You can copy the full output to your clipboard or download it as a .txt file.
Extracted by GitExtract — free GitHub repo to text converter for AI. Built by Nikandr Surkov.