Showing preview only (2,877K chars total). Download the full file or copy to clipboard to get everything.
Repository: FangShancheng/ABINet
Branch: main
Commit: 789743c56eac
Files: 44
Total size: 2.7 MB
Directory structure:
gitextract_vdjnznth/
├── .gitignore
├── LICENSE
├── README.md
├── callbacks.py
├── configs/
│ ├── pretrain_language_model.yaml
│ ├── pretrain_vision_model.yaml
│ ├── pretrain_vision_model_sv.yaml
│ ├── template.yaml
│ ├── train_abinet.yaml
│ ├── train_abinet_sv.yaml
│ └── train_abinet_wo_iter.yaml
├── dataset.py
├── demo.py
├── demo_onnx.py
├── docker/
│ └── Dockerfile
├── export_onnx.py
├── losses.py
├── main.py
├── modules/
│ ├── __init__.py
│ ├── attention.py
│ ├── backbone.py
│ ├── backbone_v2.py
│ ├── model.py
│ ├── model_abinet.py
│ ├── model_abinet_iter.py
│ ├── model_alignment.py
│ ├── model_language.py
│ ├── model_vision.py
│ ├── resnet.py
│ └── transformer.py
├── notebooks/
│ ├── dataset-text.ipynb
│ ├── dataset.ipynb
│ ├── prepare_wikitext103.ipynb
│ └── transforms.ipynb
├── requirements.txt
├── tools/
│ ├── clean_dictionary.py
│ ├── create_charset.py
│ ├── create_language_data.py
│ ├── create_lmdb_dataset.py
│ ├── crop_by_word_bb_syn90k.py
│ ├── get_max_len.py
│ └── line2word.py
├── transforms.py
└── utils.py
================================================
FILE CONTENTS
================================================
================================================
FILE: .gitignore
================================================
# Created by https://www.gitignore.io/api/python
notebooks/WIP
### Python ###
# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class
# C extensions
*.so
# Distribution / packaging
.Python
env/
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
*.egg-info/
.installed.cfg
*.egg
# PyInstaller
# Usually these files are written by a python script from a template
# before PyInstaller builds the exe, so as to inject date/other infos into it.
*.manifest
*.spec
# Installer logs
pip-log.txt
pip-delete-this-directory.txt
# Unit test / coverage reports
htmlcov/
.tox/
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*,cover
.hypothesis/
# Translations
*.mo
*.pot
# Django stuff:
*.log
local_settings.py
# Flask stuff:
instance/
.webassets-cache
# Scrapy stuff:
.scrapy
# Sphinx documentation
docs/_build/
# PyBuilder
target/
# Jupyter Notebook
.ipynb_checkpoints
# pyenv
.python-version
# celery beat schedule file
celerybeat-schedule
# dotenv
.env
# virtualenv
.venv/
venv/
ENV/
# Spyder project settings
.spyderproject
# Rope project settings
.ropeproject
# Sublime
.sublimelinterrc
# Tensorflow
.tfprof_history.txt
# Mac OS X
.DS_Store
# MkDocs
site/
### Emacs ###
# -*- mode: gitignore; -*-
*~
\#*\#
/.emacs.desktop
/.emacs.desktop.lock
*.elc
auto-save-list
tramp
.\#*
# Org-mode
.org-id-locations
*_archive
# flymake-mode
*_flymake.*
# eshell files
/eshell/history
/eshell/lastdir
# elpa packages
/elpa/
# reftex files
*.rel
# AUCTeX auto folder
/auto/
# cask packages
.cask/
dist/
# Flycheck
flycheck_*.el
# server auth directory
/server/
# projectiles files
.projectile
# directory configuration
.dir-locals.el
# End of https://www.gitignore.io/api/emacs
# pycharm
.idea/
# vscode
.vscode
# data
data
# workdir
workdir*
================================================
FILE: LICENSE
================================================
ABINet for non-commercial purposes
Copyright (c) 2021, USTC
All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:
1. Redistributions of source code must retain the above copyright notice, this
list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright notice,
this list of conditions and the following disclaimer in the documentation
and/or other materials provided with the distribution.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
================================================
FILE: README.md
================================================
# Read Like Humans: Autonomous, Bidirectional and Iterative Language Modeling for Scene Text Recognition
The official code of [ABINet](https://arxiv.org/pdf/2103.06495.pdf) (CVPR 2021, Oral).
ABINet uses a vision model and an explicit language model to recognize text in the wild, which are trained in end-to-end way. The language model (BCN) achieves bidirectional language representation in simulating cloze test, additionally utilizing iterative correction strategy.
## Runtime Environment
- We provide a pre-built docker image using the Dockerfile from `docker/Dockerfile`
- Running in Docker
```
$ git@github.com:FangShancheng/ABINet.git
$ docker run --gpus all --rm -ti --ipc=host -v "$(pwd)"/ABINet:/app fangshancheng/fastai:torch1.1 /bin/bash
```
- (Untested) Or using the dependencies
```
pip install -r requirements.txt
```
## Datasets
- Training datasets
1. [MJSynth](http://www.robots.ox.ac.uk/~vgg/data/text/) (MJ):
- Use `tools/create_lmdb_dataset.py` to convert images into LMDB dataset
- [LMDB dataset BaiduNetdisk(passwd:n23k)](https://pan.baidu.com/s/1mgnTiyoR8f6Cm655rFI4HQ)
2. [SynthText](http://www.robots.ox.ac.uk/~vgg/data/scenetext/) (ST):
- Use `tools/crop_by_word_bb.py` to crop images from original [SynthText](http://www.robots.ox.ac.uk/~vgg/data/scenetext/) dataset, and convert images into LMDB dataset by `tools/create_lmdb_dataset.py`
- [LMDB dataset BaiduNetdisk(passwd:n23k)](https://pan.baidu.com/s/1mgnTiyoR8f6Cm655rFI4HQ)
3. [WikiText103](https://s3.amazonaws.com/research.metamind.io/wikitext/wikitext-103-v1.zip), which is only used for pre-trainig language models:
- Use `notebooks/prepare_wikitext103.ipynb` to convert text into CSV format.
- [CSV dataset BaiduNetdisk(passwd:dk01)](https://pan.baidu.com/s/1yabtnPYDKqhBb_Ie9PGFXA)
- Evaluation datasets, LMDB datasets can be downloaded from [BaiduNetdisk(passwd:1dbv)](https://pan.baidu.com/s/1RUg3Akwp7n8kZYJ55rU5LQ), [GoogleDrive](https://drive.google.com/file/d/1dTI0ipu14Q1uuK4s4z32DqbqF3dJPdkk/view?usp=sharing).
1. ICDAR 2013 (IC13)
2. ICDAR 2015 (IC15)
3. IIIT5K Words (IIIT)
4. Street View Text (SVT)
5. Street View Text-Perspective (SVTP)
6. CUTE80 (CUTE)
- The structure of `data` directory is
```
data
├── charset_36.txt
├── evaluation
│ ├── CUTE80
│ ├── IC13_857
│ ├── IC15_1811
│ ├── IIIT5k_3000
│ ├── SVT
│ └── SVTP
├── training
│ ├── MJ
│ │ ├── MJ_test
│ │ ├── MJ_train
│ │ └── MJ_valid
│ └── ST
├── WikiText-103.csv
└── WikiText-103_eval_d1.csv
```
### Pretrained Models
Get the pretrained models from [BaiduNetdisk(passwd:kwck)](https://pan.baidu.com/s/1b3vyvPwvh_75FkPlp87czQ), [GoogleDrive](https://drive.google.com/file/d/1mYM_26qHUom_5NU7iutHneB_KHlLjL5y/view?usp=sharing). Performances of the pretrained models are summaried as follows:
|Model|IC13|SVT|IIIT|IC15|SVTP|CUTE|AVG|
|-|-|-|-|-|-|-|-|
|ABINet-SV|97.1|92.7|95.2|84.0|86.7|88.5|91.4|
|ABINet-LV|97.0|93.4|96.4|85.9|89.5|89.2|92.7|
## Training
1. Pre-train vision model
```
CUDA_VISIBLE_DEVICES=0,1,2,3 python main.py --config=configs/pretrain_vision_model.yaml
```
2. Pre-train language model
```
CUDA_VISIBLE_DEVICES=0,1,2,3 python main.py --config=configs/pretrain_language_model.yaml
```
3. Train ABINet
```
CUDA_VISIBLE_DEVICES=0,1,2,3 python main.py --config=configs/train_abinet.yaml
```
Note:
- You can set the `checkpoint` path for vision and language models separately for specific pretrained model, or set to `None` to train from scratch
## Evaluation
```
CUDA_VISIBLE_DEVICES=0 python main.py --config=configs/train_abinet.yaml --phase test --image_only
```
Additional flags:
- `--checkpoint /path/to/checkpoint` set the path of evaluation model
- `--test_root /path/to/dataset` set the path of evaluation dataset
- `--model_eval [alignment|vision]` which sub-model to evaluate
- `--image_only` disable dumping visualization of attention masks
## Web Demo
Integrated into [Huggingface Spaces 🤗](https://huggingface.co/spaces) using [Gradio](https://github.com/gradio-app/gradio). Try out the Web Demo: [](https://huggingface.co/spaces/tomofi/ABINet-OCR)
## Run Demo
```
python demo.py --config=configs/train_abinet.yaml --input=figs/test
```
Additional flags:
- `--config /path/to/config` set the path of configuration file
- `--input /path/to/image-directory` set the path of image directory or wildcard path, e.g, `--input='figs/test/*.png'`
- `--checkpoint /path/to/checkpoint` set the path of trained model
- `--cuda [-1|0|1|2|3...]` set the cuda id, by default -1 is set and stands for cpu
- `--model_eval [alignment|vision]` which sub-model to use
- `--image_only` disable dumping visualization of attention masks
## Visualization
Successful and failure cases on low-quality images:
## Citation
If you find our method useful for your reserach, please cite
```bash
@article{fang2021read,
title={Read Like Humans: Autonomous, Bidirectional and Iterative Language Modeling for Scene Text Recognition},
author={Fang, Shancheng and Xie, Hongtao and Wang, Yuxin and Mao, Zhendong and Zhang, Yongdong},
booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
year={2021}
}
```
## License
This project is only free for academic research purposes, licensed under the 2-clause BSD License - see the LICENSE file for details.
Feel free to contact fangsc@ustc.edu.cn if you have any questions.
================================================
FILE: callbacks.py
================================================
import logging
import shutil
import time
import editdistance as ed
import torchvision.utils as vutils
from fastai.callbacks.tensorboard import (LearnerTensorboardWriter,
SummaryWriter, TBWriteRequest,
asyncTBWriter)
from fastai.vision import *
from torch.nn.parallel import DistributedDataParallel
from torchvision import transforms
import dataset
from utils import CharsetMapper, Timer, blend_mask
class IterationCallback(LearnerTensorboardWriter):
"A `TrackerCallback` that monitor in each iteration."
def __init__(self, learn:Learner, name:str='model', checpoint_keep_num=5,
show_iters:int=50, eval_iters:int=1000, save_iters:int=20000,
start_iters:int=0, stats_iters=20000):
#if self.learn.rank is not None: time.sleep(self.learn.rank) # keep all event files
super().__init__(learn, base_dir='.', name=learn.path, loss_iters=show_iters,
stats_iters=stats_iters, hist_iters=stats_iters)
self.name, self.bestname = Path(name).name, f'best-{Path(name).name}'
self.show_iters = show_iters
self.eval_iters = eval_iters
self.save_iters = save_iters
self.start_iters = start_iters
self.checpoint_keep_num = checpoint_keep_num
self.metrics_root = 'metrics/' # rewrite
self.timer = Timer()
self.host = self.learn.rank is None or self.learn.rank == 0
def _write_metrics(self, iteration:int, names:List[str], last_metrics:MetricsList)->None:
"Writes training metrics to Tensorboard."
for i, name in enumerate(names):
if last_metrics is None or len(last_metrics) < i+1: return
scalar_value = last_metrics[i]
self._write_scalar(name=name, scalar_value=scalar_value, iteration=iteration)
def _write_sub_loss(self, iteration:int, last_losses:dict)->None:
"Writes sub loss to Tensorboard."
for name, loss in last_losses.items():
scalar_value = to_np(loss)
tag = self.metrics_root + name
self.tbwriter.add_scalar(tag=tag, scalar_value=scalar_value, global_step=iteration)
def _save(self, name):
if isinstance(self.learn.model, DistributedDataParallel):
tmp = self.learn.model
self.learn.model = self.learn.model.module
self.learn.save(name)
self.learn.model = tmp
else: self.learn.save(name)
def _validate(self, dl=None, callbacks=None, metrics=None, keeped_items=False):
"Validate on `dl` with potential `callbacks` and `metrics`."
dl = ifnone(dl, self.learn.data.valid_dl)
metrics = ifnone(metrics, self.learn.metrics)
cb_handler = CallbackHandler(ifnone(callbacks, []), metrics)
cb_handler.on_train_begin(1, None, metrics); cb_handler.on_epoch_begin()
if keeped_items: cb_handler.state_dict.update(dict(keeped_items=[]))
val_metrics = validate(self.learn.model, dl, self.loss_func, cb_handler)
cb_handler.on_epoch_end(val_metrics)
if keeped_items: return cb_handler.state_dict['keeped_items']
else: return cb_handler.state_dict['last_metrics']
def jump_to_epoch_iter(self, epoch:int, iteration:int)->None:
try:
self.learn.load(f'{self.name}_{epoch}_{iteration}', purge=False)
logging.info(f'Loaded {self.name}_{epoch}_{iteration}')
except: logging.info(f'Model {self.name}_{epoch}_{iteration} not found.')
def on_train_begin(self, n_epochs, **kwargs):
# TODO: can not write graph here
# super().on_train_begin(**kwargs)
self.best = -float('inf')
self.timer.tic()
if self.host:
checkpoint_path = self.learn.path/'checkpoint.yaml'
if checkpoint_path.exists():
os.remove(checkpoint_path)
open(checkpoint_path, 'w').close()
return {'skip_validate': True, 'iteration':self.start_iters} # disable default validate
def on_batch_begin(self, **kwargs:Any)->None:
self.timer.toc_data()
super().on_batch_begin(**kwargs)
def on_batch_end(self, iteration, epoch, last_loss, smooth_loss, train, **kwargs):
super().on_batch_end(last_loss, iteration, train, **kwargs)
if iteration == 0: return
if iteration % self.loss_iters == 0:
last_losses = self.learn.loss_func.last_losses
self._write_sub_loss(iteration=iteration, last_losses=last_losses)
self.tbwriter.add_scalar(tag=self.metrics_root + 'lr',
scalar_value=self.opt.lr, global_step=iteration)
if iteration % self.show_iters == 0:
log_str = f'epoch {epoch} iter {iteration}: loss = {last_loss:6.4f}, ' \
f'smooth loss = {smooth_loss:6.4f}'
logging.info(log_str)
# log_str = f'data time = {self.timer.data_diff:.4f}s, runing time = {self.timer.running_diff:.4f}s'
# logging.info(log_str)
if iteration % self.eval_iters == 0:
# TODO: or remove time to on_epoch_end
# 1. Record time
log_str = f'average data time = {self.timer.average_data_time():.4f}s, ' \
f'average running time = {self.timer.average_running_time():.4f}s'
logging.info(log_str)
# 2. Call validate
last_metrics = self._validate()
self.learn.model.train()
log_str = f'epoch {epoch} iter {iteration}: eval loss = {last_metrics[0]:6.4f}, ' \
f'ccr = {last_metrics[1]:6.4f}, cwr = {last_metrics[2]:6.4f}, ' \
f'ted = {last_metrics[3]:6.4f}, ned = {last_metrics[4]:6.4f}, ' \
f'ted/w = {last_metrics[5]:6.4f}, '
logging.info(log_str)
names = ['eval_loss', 'ccr', 'cwr', 'ted', 'ned', 'ted/w']
self._write_metrics(iteration, names, last_metrics)
# 3. Save best model
current = last_metrics[2]
if current is not None and current > self.best:
logging.info(f'Better model found at epoch {epoch}, '\
f'iter {iteration} with accuracy value: {current:6.4f}.')
self.best = current
self._save(f'{self.bestname}')
if iteration % self.save_iters == 0 and self.host:
logging.info(f'Save model {self.name}_{epoch}_{iteration}')
filename = f'{self.name}_{epoch}_{iteration}'
self._save(filename)
checkpoint_path = self.learn.path/'checkpoint.yaml'
if not checkpoint_path.exists():
open(checkpoint_path, 'w').close()
with open(checkpoint_path, 'r') as file:
checkpoints = yaml.load(file, Loader=yaml.FullLoader) or dict()
checkpoints['all_checkpoints'] = (
checkpoints.get('all_checkpoints') or list())
checkpoints['all_checkpoints'].insert(0, filename)
if len(checkpoints['all_checkpoints']) > self.checpoint_keep_num:
removed_checkpoint = checkpoints['all_checkpoints'].pop()
removed_checkpoint = self.learn.path/self.learn.model_dir/f'{removed_checkpoint}.pth'
os.remove(removed_checkpoint)
checkpoints['current_checkpoint'] = filename
with open(checkpoint_path, 'w') as file:
yaml.dump(checkpoints, file)
self.timer.toc_running()
def on_train_end(self, **kwargs):
#self.learn.load(f'{self.bestname}', purge=False)
pass
def on_epoch_end(self, last_metrics:MetricsList, iteration:int, **kwargs)->None:
self._write_embedding(iteration=iteration)
class TextAccuracy(Callback):
_names = ['ccr', 'cwr', 'ted', 'ned', 'ted/w']
def __init__(self, charset_path, max_length, case_sensitive, model_eval):
self.charset_path = charset_path
self.max_length = max_length
self.case_sensitive = case_sensitive
self.charset = CharsetMapper(charset_path, self.max_length)
self.names = self._names
self.model_eval = model_eval or 'alignment'
assert self.model_eval in ['vision', 'language', 'alignment']
def on_epoch_begin(self, **kwargs):
self.total_num_char = 0.
self.total_num_word = 0.
self.correct_num_char = 0.
self.correct_num_word = 0.
self.total_ed = 0.
self.total_ned = 0.
def _get_output(self, last_output):
if isinstance(last_output, (tuple, list)):
for res in last_output:
if res['name'] == self.model_eval: output = res
else: output = last_output
return output
def _update_output(self, last_output, items):
if isinstance(last_output, (tuple, list)):
for res in last_output:
if res['name'] == self.model_eval: res.update(items)
else: last_output.update(items)
return last_output
def on_batch_end(self, last_output, last_target, **kwargs):
output = self._get_output(last_output)
logits, pt_lengths = output['logits'], output['pt_lengths']
pt_text, pt_scores, pt_lengths_ = self.decode(logits)
assert (pt_lengths == pt_lengths_).all(), f'{pt_lengths} != {pt_lengths_} for {pt_text}'
last_output = self._update_output(last_output, {'pt_text':pt_text, 'pt_scores':pt_scores})
pt_text = [self.charset.trim(t) for t in pt_text]
label = last_target[0]
if label.dim() == 3: label = label.argmax(dim=-1) # one-hot label
gt_text = [self.charset.get_text(l, trim=True) for l in label]
for i in range(len(gt_text)):
if not self.case_sensitive:
gt_text[i], pt_text[i] = gt_text[i].lower(), pt_text[i].lower()
distance = ed.eval(gt_text[i], pt_text[i])
self.total_ed += distance
self.total_ned += float(distance) / max(len(gt_text[i]), 1)
if gt_text[i] == pt_text[i]:
self.correct_num_word += 1
self.total_num_word += 1
for j in range(min(len(gt_text[i]), len(pt_text[i]))):
if gt_text[i][j] == pt_text[i][j]:
self.correct_num_char += 1
self.total_num_char += len(gt_text[i])
return {'last_output': last_output}
def on_epoch_end(self, last_metrics, **kwargs):
mets = [self.correct_num_char / self.total_num_char,
self.correct_num_word / self.total_num_word,
self.total_ed,
self.total_ned,
self.total_ed / self.total_num_word]
return add_metrics(last_metrics, mets)
def decode(self, logit):
""" Greed decode """
# TODO: test running time and decode on GPU
out = F.softmax(logit, dim=2)
pt_text, pt_scores, pt_lengths = [], [], []
for o in out:
text = self.charset.get_text(o.argmax(dim=1), padding=False, trim=False)
text = text.split(self.charset.null_char)[0] # end at end-token
pt_text.append(text)
pt_scores.append(o.max(dim=1)[0])
pt_lengths.append(min(len(text) + 1, self.max_length)) # one for end-token
pt_scores = torch.stack(pt_scores)
pt_lengths = pt_scores.new_tensor(pt_lengths, dtype=torch.long)
return pt_text, pt_scores, pt_lengths
class TopKTextAccuracy(TextAccuracy):
_names = ['ccr', 'cwr']
def __init__(self, k, charset_path, max_length, case_sensitive, model_eval):
self.k = k
self.charset_path = charset_path
self.max_length = max_length
self.case_sensitive = case_sensitive
self.charset = CharsetMapper(charset_path, self.max_length)
self.names = self._names
def on_epoch_begin(self, **kwargs):
self.total_num_char = 0.
self.total_num_word = 0.
self.correct_num_char = 0.
self.correct_num_word = 0.
def on_batch_end(self, last_output, last_target, **kwargs):
logits, pt_lengths = last_output['logits'], last_output['pt_lengths']
gt_labels, gt_lengths = last_target[:]
for logit, pt_length, label, length in zip(logits, pt_lengths, gt_labels, gt_lengths):
word_flag = True
for i in range(length):
char_logit = logit[i].topk(self.k)[1]
char_label = label[i].argmax(-1)
if char_label in char_logit: self.correct_num_char += 1
else: word_flag = False
self.total_num_char += 1
if pt_length == length and word_flag:
self.correct_num_word += 1
self.total_num_word += 1
def on_epoch_end(self, last_metrics, **kwargs):
mets = [self.correct_num_char / self.total_num_char,
self.correct_num_word / self.total_num_word,
0., 0., 0.]
return add_metrics(last_metrics, mets)
class DumpPrediction(LearnerCallback):
def __init__(self, learn, dataset, charset_path, model_eval, image_only=False, debug=False):
super().__init__(learn=learn)
self.debug = debug
self.model_eval = model_eval or 'alignment'
self.image_only = image_only
assert self.model_eval in ['vision', 'language', 'alignment']
self.dataset, self.root = dataset, Path(self.learn.path)/f'{dataset}-{self.model_eval}'
self.attn_root = self.root/'attn'
self.charset = CharsetMapper(charset_path)
if self.root.exists(): shutil.rmtree(self.root)
self.root.mkdir(), self.attn_root.mkdir()
self.pil = transforms.ToPILImage()
self.tensor = transforms.ToTensor()
size = self.learn.data.img_h, self.learn.data.img_w
self.resize = transforms.Resize(size=size, interpolation=0)
self.c = 0
def on_batch_end(self, last_input, last_output, last_target, **kwargs):
if isinstance(last_output, (tuple, list)):
for res in last_output:
if res['name'] == self.model_eval: pt_text = res['pt_text']
if res['name'] == 'vision': attn_scores = res['attn_scores'].detach().cpu()
if res['name'] == self.model_eval: logits = res['logits']
else:
pt_text = last_output['pt_text']
attn_scores = last_output['attn_scores'].detach().cpu()
logits = last_output['logits']
images = last_input[0] if isinstance(last_input, (tuple, list)) else last_input
images = images.detach().cpu()
pt_text = [self.charset.trim(t) for t in pt_text]
gt_label = last_target[0]
if gt_label.dim() == 3: gt_label = gt_label.argmax(dim=-1) # one-hot label
gt_text = [self.charset.get_text(l, trim=True) for l in gt_label]
prediction, false_prediction = [], []
for gt, pt, image, attn, logit in zip(gt_text, pt_text, images, attn_scores, logits):
prediction.append(f'{gt}\t{pt}\n')
if gt != pt:
if self.debug:
scores = torch.softmax(logit, dim=-1)[:max(len(pt), len(gt)) + 1]
logging.info(f'{self.c} gt {gt}, pt {pt}, logit {logit.shape}, scores {scores.topk(5, dim=-1)}')
false_prediction.append(f'{gt}\t{pt}\n')
image = self.learn.data.denorm(image)
if not self.image_only:
image_np = np.array(self.pil(image))
attn_pil = [self.pil(a) for a in attn[:, None, :, :]]
attn = [self.tensor(self.resize(a)).repeat(3, 1, 1) for a in attn_pil]
attn_sum = np.array([np.array(a) for a in attn_pil[:len(pt)]]).sum(axis=0)
blended_sum = self.tensor(blend_mask(image_np, attn_sum))
blended = [self.tensor(blend_mask(image_np, np.array(a))) for a in attn_pil]
save_image = torch.stack([image] + attn + [blended_sum] + blended)
save_image = save_image.view(2, -1, *save_image.shape[1:])
save_image = save_image.permute(1, 0, 2, 3, 4).flatten(0, 1)
vutils.save_image(save_image, self.attn_root/f'{self.c}_{gt}_{pt}.jpg',
nrow=2, normalize=True, scale_each=True)
else:
self.pil(image).save(self.attn_root/f'{self.c}_{gt}_{pt}.jpg')
self.c += 1
with open(self.root/f'{self.model_eval}.txt', 'a') as f: f.writelines(prediction)
with open(self.root/f'{self.model_eval}-false.txt', 'a') as f: f.writelines(false_prediction)
================================================
FILE: configs/pretrain_language_model.yaml
================================================
global:
name: pretrain-language-model
phase: train
stage: pretrain-language
workdir: workdir
seed: ~
dataset:
train: {
roots: ['data/WikiText-103.csv'],
batch_size: 4096
}
test: {
roots: ['data/WikiText-103_eval_d1.csv'],
batch_size: 4096
}
training:
epochs: 80
show_iters: 50
eval_iters: 6000
save_iters: 3000
optimizer:
type: Adam
true_wd: False
wd: 0.0
bn_wd: False
clip_grad: 20
lr: 0.0001
args: {
betas: !!python/tuple [0.9, 0.999], # for default Adam
}
scheduler: {
periods: [70, 10],
gamma: 0.1,
}
model:
name: 'modules.model_language.BCNLanguage'
language: {
num_layers: 4,
loss_weight: 1.,
use_self_attn: False
}
================================================
FILE: configs/pretrain_vision_model.yaml
================================================
global:
name: pretrain-vision-model
phase: train
stage: pretrain-vision
workdir: workdir
seed: ~
dataset:
train: {
roots: ['data/training/MJ/MJ_train/',
'data/training/MJ/MJ_test/',
'data/training/MJ/MJ_valid/',
'data/training/ST'],
batch_size: 384
}
test: {
roots: ['data/evaluation/IIIT5k_3000',
'data/evaluation/SVT',
'data/evaluation/SVTP',
'data/evaluation/IC13_857',
'data/evaluation/IC15_1811',
'data/evaluation/CUTE80'],
batch_size: 384
}
data_aug: True
multiscales: False
num_workers: 14
training:
epochs: 8
show_iters: 50
eval_iters: 3000
save_iters: 3000
optimizer:
type: Adam
true_wd: False
wd: 0.0
bn_wd: False
clip_grad: 20
lr: 0.0001
args: {
betas: !!python/tuple [0.9, 0.999], # for default Adam
}
scheduler: {
periods: [6, 2],
gamma: 0.1,
}
model:
name: 'modules.model_vision.BaseVision'
checkpoint: ~
vision: {
loss_weight: 1.,
attention: 'position',
backbone: 'transformer',
backbone_ln: 3,
}
================================================
FILE: configs/pretrain_vision_model_sv.yaml
================================================
global:
name: pretrain-vision-model-sv
phase: train
stage: pretrain-vision
workdir: workdir
seed: ~
dataset:
train: {
roots: ['data/training/MJ/MJ_train/',
'data/training/MJ/MJ_test/',
'data/training/MJ/MJ_valid/',
'data/training/ST'],
batch_size: 384
}
test: {
roots: ['data/evaluation/IIIT5k_3000',
'data/evaluation/SVT',
'data/evaluation/SVTP',
'data/evaluation/IC13_857',
'data/evaluation/IC15_1811',
'data/evaluation/CUTE80'],
batch_size: 384
}
data_aug: True
multiscales: False
num_workers: 14
training:
epochs: 8
show_iters: 50
eval_iters: 3000
save_iters: 3000
optimizer:
type: Adam
true_wd: False
wd: 0.0
bn_wd: False
clip_grad: 20
lr: 0.0001
args: {
betas: !!python/tuple [0.9, 0.999], # for default Adam
}
scheduler: {
periods: [6, 2],
gamma: 0.1,
}
model:
name: 'modules.model_vision.BaseVision'
checkpoint: ~
vision: {
loss_weight: 1.,
attention: 'attention',
backbone: 'transformer',
backbone_ln: 2,
}
================================================
FILE: configs/template.yaml
================================================
global:
name: exp
phase: train
stage: pretrain-vision
workdir: /tmp/workdir
seed: ~
dataset:
train: {
roots: ['data/training/MJ/MJ_train/',
'data/training/MJ/MJ_test/',
'data/training/MJ/MJ_valid/',
'data/training/ST'],
batch_size: 128
}
test: {
roots: ['data/evaluation/IIIT5k_3000',
'data/evaluation/SVT',
'data/evaluation/SVTP',
'data/evaluation/IC13_857',
'data/evaluation/IC15_1811',
'data/evaluation/CUTE80'],
batch_size: 128
}
charset_path: data/charset_36.txt
num_workers: 4
max_length: 25 # 30
image_height: 32
image_width: 128
case_sensitive: False
eval_case_sensitive: False
data_aug: True
multiscales: False
pin_memory: True
smooth_label: False
smooth_factor: 0.1
one_hot_y: True
use_sm: False
training:
epochs: 6
show_iters: 50
eval_iters: 3000
save_iters: 20000
start_iters: 0
stats_iters: 100000
optimizer:
type: Adadelta # Adadelta, Adam
true_wd: False
wd: 0. # 0.001
bn_wd: False
args: {
# betas: !!python/tuple [0.9, 0.99], # betas=(0.9,0.99) for AdamW
# betas: !!python/tuple [0.9, 0.999], # for default Adam
}
clip_grad: 20
lr: [1.0, 1.0, 1.0] # lr: [0.005, 0.005, 0.005]
scheduler: {
periods: [3, 2, 1],
gamma: 0.1,
}
model:
name: 'modules.model_abinet.ABINetModel'
checkpoint: ~
strict: True
================================================
FILE: configs/train_abinet.yaml
================================================
global:
name: train-abinet
phase: train
stage: train-super
workdir: workdir
seed: ~
dataset:
train: {
roots: ['data/training/MJ/MJ_train/',
'data/training/MJ/MJ_test/',
'data/training/MJ/MJ_valid/',
'data/training/ST'],
batch_size: 384
}
test: {
roots: ['data/evaluation/IIIT5k_3000',
'data/evaluation/SVT',
'data/evaluation/SVTP',
'data/evaluation/IC13_857',
'data/evaluation/IC15_1811',
'data/evaluation/CUTE80'],
batch_size: 384
}
data_aug: True
multiscales: False
num_workers: 14
training:
epochs: 10
show_iters: 50
eval_iters: 3000
save_iters: 3000
optimizer:
type: Adam
true_wd: False
wd: 0.0
bn_wd: False
clip_grad: 20
lr: 0.0001
args: {
betas: !!python/tuple [0.9, 0.999], # for default Adam
}
scheduler: {
periods: [6, 4],
gamma: 0.1,
}
model:
name: 'modules.model_abinet_iter.ABINetIterModel'
iter_size: 3
ensemble: ''
use_vision: False
vision: {
checkpoint: workdir/pretrain-vision-model/best-pretrain-vision-model.pth,
loss_weight: 1.,
attention: 'position',
backbone: 'transformer',
backbone_ln: 3,
}
language: {
checkpoint: workdir/pretrain-language-model/pretrain-language-model.pth,
num_layers: 4,
loss_weight: 1.,
detach: True,
use_self_attn: False
}
alignment: {
loss_weight: 1.,
}
================================================
FILE: configs/train_abinet_sv.yaml
================================================
global:
name: train-abinet-sv
phase: train
stage: train-super
workdir: workdir
seed: ~
dataset:
train: {
roots: ['data/training/MJ/MJ_train/',
'data/training/MJ/MJ_test/',
'data/training/MJ/MJ_valid/',
'data/training/ST'],
batch_size: 384
}
test: {
roots: ['data/evaluation/IIIT5k_3000',
'data/evaluation/SVT',
'data/evaluation/SVTP',
'data/evaluation/IC13_857',
'data/evaluation/IC15_1811',
'data/evaluation/CUTE80'],
batch_size: 384
}
data_aug: True
multiscales: False
num_workers: 14
training:
epochs: 10
show_iters: 50
eval_iters: 3000
save_iters: 3000
optimizer:
type: Adam
true_wd: False
wd: 0.0
bn_wd: False
clip_grad: 20
lr: 0.0001
args: {
betas: !!python/tuple [0.9, 0.999], # for default Adam
}
scheduler: {
periods: [6, 4],
gamma: 0.1,
}
model:
name: 'modules.model_abinet_iter.ABINetIterModel'
iter_size: 3
ensemble: ''
use_vision: False
vision: {
checkpoint: workdir/pretrain-vision-model-sv/best-pretrain-vision-model-sv.pth,
loss_weight: 1.,
attention: 'attention',
backbone: 'transformer',
backbone_ln: 2,
}
language: {
checkpoint: workdir/pretrain-language-model/pretrain-language-model.pth,
num_layers: 4,
loss_weight: 1.,
detach: True,
use_self_attn: False
}
alignment: {
loss_weight: 1.,
}
================================================
FILE: configs/train_abinet_wo_iter.yaml
================================================
global:
name: train-abinet-wo-iter
phase: train
stage: train-super
workdir: workdir
seed: ~
dataset:
train: {
roots: ['data/training/MJ/MJ_train/',
'data/training/MJ/MJ_test/',
'data/training/MJ/MJ_valid/',
'data/training/ST'],
batch_size: 384
}
test: {
roots: ['data/evaluation/IIIT5k_3000',
'data/evaluation/SVT',
'data/evaluation/SVTP',
'data/evaluation/IC13_857',
'data/evaluation/IC15_1811',
'data/evaluation/CUTE80'],
batch_size: 384
}
data_aug: True
multiscales: False
num_workers: 14
training:
epochs: 10
show_iters: 50
eval_iters: 3000
save_iters: 3000
optimizer:
type: Adam
true_wd: False
wd: 0.0
bn_wd: False
clip_grad: 20
lr: 0.0001
args: {
betas: !!python/tuple [0.9, 0.999], # for default Adam
}
scheduler: {
periods: [6, 4],
gamma: 0.1,
}
model:
name: 'modules.model_abinet.ABINetModel'
vision: {
checkpoint: workdir/pretrain-vision-model/best-pretrain-vision-model.pth,
loss_weight: 1.,
attention: 'position',
backbone: 'transformer',
backbone_ln: 3,
}
language: {
checkpoint: workdir/pretrain-language-model/pretrain-language-model.pth,
num_layers: 4,
loss_weight: 1.,
detach: True,
use_self_attn: False
}
alignment: {
loss_weight: 1.,
}
================================================
FILE: dataset.py
================================================
import logging
# import re
import cv2
import lmdb
import six
from fastai.vision import *
from torchvision import transforms
from transforms import CVColorJitter, CVDeterioration, CVGeometry
from utils import CharsetMapper, onehot
class ImageDataset(Dataset):
"`ImageDataset` read data from LMDB database."
def __init__(self,
path: PathOrStr,
is_training: bool = True,
img_h: int = 32,
img_w: int = 100,
max_length: int = 25,
check_length: bool = True,
case_sensitive: bool = False,
charset_path: str = 'data/charset_vn_with_space.txt',
convert_mode: str = 'RGB',
data_aug: bool = True,
deteriorate_ratio: float = 0.,
multiscales: bool = True,
one_hot_y: bool = True,
return_idx: bool = False,
return_raw: bool = False,
**kwargs):
self.path, self.name = Path(path), Path(path).name
assert self.path.is_dir() and self.path.exists(), f"{path} is not a valid directory."
self.convert_mode, self.check_length = convert_mode, check_length
self.img_h, self.img_w = img_h, img_w
self.max_length, self.one_hot_y = max_length, one_hot_y
self.return_idx, self.return_raw = return_idx, return_raw
self.case_sensitive, self.is_training = case_sensitive, is_training
self.data_aug, self.multiscales = data_aug, multiscales
self.charset = CharsetMapper(charset_path, max_length=max_length + 1)
self.character = self.charset.label_to_char.values()
self.c = self.charset.num_classes
self.env = lmdb.open(str(path), readonly=True, lock=False, readahead=False, meminit=False)
assert self.env, f'Cannot open LMDB dataset from {path}.'
with self.env.begin(write=False) as txn:
self.length = int(txn.get('num-samples'.encode()))
if self.is_training and self.data_aug:
self.augment_tfs = transforms.Compose([
CVGeometry(degrees=45, translate=(0.0, 0.0), scale=(0.5, 2.), shear=(45, 15), distortion=0.5, p=0.5),
CVDeterioration(var=20, degrees=6, factor=4, p=0.25),
CVColorJitter(brightness=0.5, contrast=0.5, saturation=0.5, hue=0.1, p=0.25)
])
self.totensor = transforms.ToTensor()
def __len__(self):
return self.length
def _next_image(self, index):
next_index = random.randint(0, len(self) - 1)
return self.get(next_index)
def _check_image(self, x, pixels=6):
if x.size[0] <= pixels or x.size[1] <= pixels:
return False
else:
return True
def resize_multiscales(self, img, borderType=cv2.BORDER_CONSTANT):
def _resize_ratio(img, ratio, fix_h=True):
if ratio * self.img_w < self.img_h:
if fix_h:
trg_h = self.img_h
else:
trg_h = int(ratio * self.img_w)
trg_w = self.img_w
else:
trg_h, trg_w = self.img_h, int(self.img_h / ratio)
img = cv2.resize(img, (trg_w, trg_h))
pad_h, pad_w = (self.img_h - trg_h) / 2, (self.img_w - trg_w) / 2
top, bottom = math.ceil(pad_h), math.floor(pad_h)
left, right = math.ceil(pad_w), math.floor(pad_w)
img = cv2.copyMakeBorder(img, top, bottom, left, right, borderType)
return img
if self.is_training:
if random.random() < 0.5:
base, maxh, maxw = self.img_h, self.img_h, self.img_w
h, w = random.randint(base, maxh), random.randint(base, maxw)
return _resize_ratio(img, h / w)
else:
return _resize_ratio(img, img.shape[0] / img.shape[1]) # keep aspect ratio
else:
return _resize_ratio(img, img.shape[0] / img.shape[1]) # keep aspect ratio
def resize(self, img):
if self.multiscales:
return self.resize_multiscales(img, cv2.BORDER_REPLICATE)
else:
return cv2.resize(img, (self.img_w, self.img_h))
def get(self, idx):
with self.env.begin(write=False) as txn:
image_key, label_key = f'image-{idx + 1:09d}', f'label-{idx + 1:09d}'
try:
label = str(txn.get(label_key.encode()), 'utf-8').strip() # label
if not set(label).issubset(self.character):
return self._next_image(idx)
# label = re.sub('[^0-9a-zA-Z]+', '', label)
if self.check_length and self.max_length > 0:
if len(label) > self.max_length or len(label) <= 0:
# logging.info(f'Long or short text image is found: {self.name}, {idx}, {label}, {len(label)}')
return self._next_image(idx)
label = label[:self.max_length]
imgbuf = txn.get(image_key.encode()) # image
buf = six.BytesIO()
buf.write(imgbuf)
buf.seek(0)
with warnings.catch_warnings():
warnings.simplefilter("ignore", UserWarning) # EXIF warning from TiffPlugin
image = PIL.Image.open(buf).convert(self.convert_mode)
if self.is_training and not self._check_image(image):
# logging.info(f'Invalid image is found: {self.name}, {idx}, {label}, {len(label)}')
return self._next_image(idx)
except:
import traceback
traceback.print_exc()
logging.info(f'Corrupted image is found: {self.name}, {idx}, {label}, {len(label)}')
return self._next_image(idx)
return image, label, idx
def _process_training(self, image):
if self.data_aug: image = self.augment_tfs(image)
image = self.resize(np.array(image))
return image
def _process_test(self, image):
return self.resize(np.array(image)) # TODO:move is_training to here
def __getitem__(self, idx):
image, text, idx_new = self.get(idx)
# print(image, text, idx_new, idx)
if not self.is_training: assert idx == idx_new, f'idx {idx} != idx_new {idx_new} during testing.'
if self.is_training:
image = self._process_training(image)
else:
image = self._process_test(image)
if self.return_raw: return image, text
image = self.totensor(image)
length = tensor(len(text) + 1).to(dtype=torch.long) # one for end token
label = self.charset.get_labels(text, case_sensitive=self.case_sensitive)
label = tensor(label).to(dtype=torch.long)
if self.one_hot_y: label = onehot(label, self.charset.num_classes)
if self.return_idx:
y = [label, length, idx_new]
else:
y = [label, length]
return image, y
class TextDataset(Dataset):
def __init__(self,
path: PathOrStr,
delimiter: str = '\t',
max_length: int = 25,
charset_path: str = 'data/charset_vn.txt',
case_sensitive=False,
one_hot_x=True,
one_hot_y=True,
is_training=True,
smooth_label=False,
smooth_factor=0.2,
use_sm=False,
**kwargs):
self.path = Path(path)
self.case_sensitive, self.use_sm = case_sensitive, use_sm
self.smooth_factor, self.smooth_label = smooth_factor, smooth_label
self.charset = CharsetMapper(charset_path, max_length=max_length + 1)
self.one_hot_x, self.one_hot_y, self.is_training = one_hot_x, one_hot_y, is_training
if self.is_training and self.use_sm: self.sm = SpellingMutation(charset=self.charset)
dtype = {'inp': str, 'gt': str}
self.df = pd.read_csv(self.path, dtype=dtype, delimiter=delimiter, na_filter=False)
self.inp_col, self.gt_col = 0, 1
def __len__(self):
return len(self.df)
def __getitem__(self, idx):
text_x = self.df.iloc[idx, self.inp_col].strip()
# text_x = re.sub('[^0-9a-zA-Z]+', '', text_x)
if not self.case_sensitive: text_x = text_x.lower()
if self.is_training and self.use_sm: text_x = self.sm(text_x)
length_x = tensor(len(text_x) + 1).to(dtype=torch.long) # one for end token
label_x = self.charset.get_labels(text_x, case_sensitive=self.case_sensitive)
label_x = tensor(label_x)
if self.one_hot_x:
label_x = onehot(label_x, self.charset.num_classes)
if self.is_training and self.smooth_label:
label_x = torch.stack([self.prob_smooth_label(l) for l in label_x])
x = [label_x, length_x]
text_y = self.df.iloc[idx, self.gt_col]
# text_y = re.sub('[^0-9a-zA-Z]+', '', text_y)
if not self.case_sensitive: text_y = text_y.lower()
length_y = tensor(len(text_y) + 1).to(dtype=torch.long) # one for end token
label_y = self.charset.get_labels(text_y, case_sensitive=self.case_sensitive)
label_y = tensor(label_y)
if self.one_hot_y: label_y = onehot(label_y, self.charset.num_classes)
y = [label_y, length_y]
return x, y
def prob_smooth_label(self, one_hot):
one_hot = one_hot.float()
delta = torch.rand([]) * self.smooth_factor
num_classes = len(one_hot)
noise = torch.rand(num_classes)
noise = noise / noise.sum() * delta
one_hot = one_hot * (1 - delta) + noise
return one_hot
class SpellingMutation(object):
def __init__(self, pn0=0.7, pn1=0.85, pn2=0.95, pt0=0.7, pt1=0.85, charset=None):
"""
Args:
pn0: the prob of not modifying characters is (pn0)
pn1: the prob of modifying one characters is (pn1 - pn0)
pn2: the prob of modifying two characters is (pn2 - pn1),
and three (1 - pn2)
pt0: the prob of replacing operation is pt0.
pt1: the prob of inserting operation is (pt1 - pt0),
and deleting operation is (1 - pt1)
"""
super().__init__()
self.pn0, self.pn1, self.pn2 = pn0, pn1, pn2
self.pt0, self.pt1 = pt0, pt1
self.charset = charset
logging.info(f'the probs: pn0={self.pn0}, pn1={self.pn1} ' +
f'pn2={self.pn2}, pt0={self.pt0}, pt1={self.pt1}')
def is_digit(self, text, ratio=0.5):
length = max(len(text), 1)
digit_num = sum([t in self.charset.digits for t in text])
if digit_num / length < ratio: return False
return True
def is_unk_char(self, char):
# return char == self.charset.unk_char
return (char not in self.charset.digits) and (char not in self.charset.alphabets)
def get_num_to_modify(self, length):
prob = random.random()
if prob < self.pn0:
num_to_modify = 0
elif prob < self.pn1:
num_to_modify = 1
elif prob < self.pn2:
num_to_modify = 2
else:
num_to_modify = 3
if length <= 1:
num_to_modify = 0
elif length >= 2 and length <= 4:
num_to_modify = min(num_to_modify, 1)
else:
num_to_modify = min(num_to_modify, length // 2) # smaller than length // 2
return num_to_modify
def __call__(self, text, debug=False):
if self.is_digit(text): return text
length = len(text)
num_to_modify = self.get_num_to_modify(length)
if num_to_modify <= 0: return text
chars = []
index = np.arange(0, length)
random.shuffle(index)
index = index[: num_to_modify]
if debug: self.index = index
for i, t in enumerate(text):
if i not in index:
chars.append(t)
elif self.is_unk_char(t):
chars.append(t)
else:
prob = random.random()
if prob < self.pt0: # replace
chars.append(random.choice(self.charset.alphabets))
elif prob < self.pt1: # insert
chars.append(random.choice(self.charset.alphabets))
chars.append(t)
else: # delete
continue
new_text = ''.join(chars[: self.charset.max_length - 1])
return new_text if len(new_text) >= 1 else text
================================================
FILE: demo.py
================================================
import argparse
import logging
import os
import glob
import tqdm
import torch
import PIL
import cv2
import numpy as np
import torch.nn.functional as F
from torchvision import transforms
from utils import Config, Logger, CharsetMapper
def get_model(config):
import importlib
names = config.model_name.split('.')
module_name, class_name = '.'.join(names[:-1]), names[-1]
cls = getattr(importlib.import_module(module_name), class_name)
model = cls(config)
logging.info(model)
model = model.eval()
return model
def preprocess(img, width, height):
img = cv2.resize(np.array(img), (width, height))
img = transforms.ToTensor()(img).unsqueeze(0)
mean = torch.tensor([0.485, 0.456, 0.406])
std = torch.tensor([0.229, 0.224, 0.225])
return (img-mean[...,None,None]) / std[...,None,None]
def postprocess(output, charset, model_eval):
def _get_output(last_output, model_eval):
if isinstance(last_output, (tuple, list)):
for res in last_output:
if res['name'] == model_eval: output = res
else: output = last_output
return output
def _decode(logit):
""" Greed decode """
out = F.softmax(logit, dim=2)
pt_text, pt_scores, pt_lengths = [], [], []
for o in out:
text = charset.get_text(o.argmax(dim=1), padding=False, trim=False)
text = text.split(charset.null_char)[0] # end at end-token
pt_text.append(text)
pt_scores.append(o.max(dim=1)[0])
pt_lengths.append(min(len(text) + 1, charset.max_length)) # one for end-token
return pt_text, pt_scores, pt_lengths
output = _get_output(output, model_eval)
logits, pt_lengths = output['logits'], output['pt_lengths']
pt_text, pt_scores, pt_lengths_ = _decode(logits)
return pt_text, pt_scores, pt_lengths_
def load(model, file, device=None, strict=True):
if device is None: device = 'cpu'
elif isinstance(device, int): device = torch.device('cuda', device)
assert os.path.isfile(file)
state = torch.load(file, map_location=device)
if set(state.keys()) == {'model', 'opt'}:
state = state['model']
model.load_state_dict(state, strict=strict)
return model
def main():
parser = argparse.ArgumentParser()
parser.add_argument('--config', type=str, default='configs/train_abinet.yaml',
help='path to config file')
parser.add_argument('--input', type=str, default='figs/test')
parser.add_argument('--cuda', type=int, default=-1)
parser.add_argument('--checkpoint', type=str, default='workdir/train-abinet/best-train-abinet.pth')
parser.add_argument('--model_eval', type=str, default='alignment',
choices=['alignment', 'vision', 'language'])
args = parser.parse_args()
config = Config(args.config)
if args.checkpoint is not None: config.model_checkpoint = args.checkpoint
if args.model_eval is not None: config.model_eval = args.model_eval
config.global_phase = 'test'
config.model_vision_checkpoint, config.model_language_checkpoint = None, None
device = 'cpu' if args.cuda < 0 else f'cuda:{args.cuda}'
Logger.init(config.global_workdir, config.global_name, config.global_phase)
Logger.enable_file()
logging.info(config)
logging.info('Construct model.')
model = get_model(config).to(device)
model = load(model, config.model_checkpoint, device=device)
charset = CharsetMapper(filename=config.dataset_charset_path,
max_length=config.dataset_max_length + 1)
if os.path.isdir(args.input):
paths = [os.path.join(args.input, fname) for fname in os.listdir(args.input)]
else:
paths = glob.glob(os.path.expanduser(args.input))
assert paths, "The input path(s) was not found"
paths = sorted(paths)
for path in tqdm.tqdm(paths):
img = PIL.Image.open(path).convert('RGB')
img = preprocess(img, config.dataset_image_width, config.dataset_image_height)
img = img.to(device)
res = model(img)
pt_text, _, __ = postprocess(res, charset, config.model_eval)
logging.info(f'{path}: {pt_text[0]}')
if __name__ == '__main__':
main()
================================================
FILE: demo_onnx.py
================================================
""" Created by MrBBS """
# 10/6/2022
# -*-encoding:utf-8-*-
import onnxruntime
import torch
import cv2
import numpy as np
from torchvision import transforms
from utils import CharsetMapper
from torch.nn import functional as F
from typing import *
import tqdm
sess = onnxruntime.InferenceSession('abinet.onnx', providers=['CUDAExecutionProvider'])
charset = CharsetMapper(filename='data/charset_vn.txt',
max_length=51)
class WordAccuary:
def __init__(self, case_sensitive=False):
super(WordAccuary, self).__init__()
self.total = 1e-10
self.correct = 0
self.case_sensitive = case_sensitive
def update(self, y_pred: List[str], y: List[str]):
self.total += len(y_pred)
for pred, gt in zip(y_pred, y):
if not self.case_sensitive:
pred = pred.lower()
gt = gt.lower()
if pred == gt:
self.correct += 1
def compute(self):
return self.correct / self.total
def reset(self):
self.total = 1e-10
self.correct = 0
def _decode(logit):
""" Greed decode """
out = F.softmax(logit, dim=2)
pt_text, pt_scores, pt_lengths = [], [], []
for o in out:
text = charset.get_text(o.argmax(dim=1), padding=False, trim=False)
text = text.split(charset.null_char)[0] # end at end-token
pt_text.append(text)
pt_scores.append(o.max(dim=1)[0])
pt_lengths.append(min(len(text) + 1, charset.max_length)) # one for end-token
return pt_text, pt_scores, pt_lengths
def preprocess(img, width, height):
img = cv2.resize(np.array(img), (width, height))
img = transforms.ToTensor()(img).unsqueeze(0)
mean = torch.tensor([0.485, 0.456, 0.406])
std = torch.tensor([0.229, 0.224, 0.225])
return (img - mean[..., None, None]) / std[..., None, None]
imgs = r'E:\text_recognize_data\syntext_line'
# cv2.namedWindow('cc', cv2.WINDOW_NORMAL)
with open(r'E:\text_recognize_data\syntext_line\anno_real.txt', 'r', encoding='utf8') as f:
data = f.readlines()
metric = WordAccuary()
# for i in tqdm.tqdm(data[:4000]):
for i in data:
img_path, text = i.strip().split('\t')
image = cv2.imread(imgs + f'/{img_path}')[:, :, ::-1]
image = preprocess(image, 128, 32)
image = image.cpu().numpy()
inp = {'input': image}
logits, lengths = sess.run(["logits", "lengths"], inp)
# print(logits)
pt_text, pt_scores, pt_lengths = _decode(torch.from_numpy(logits))
metric.update(pt_text[0], text)
print(text, pt_text[0])
# break
print(metric.compute())
================================================
FILE: docker/Dockerfile
================================================
FROM anibali/pytorch:cuda-9.0
MAINTAINER fangshancheng <fangsc@ustc.edu.cn>
RUN sudo rm -rf /etc/apt/sources.list.d && \
sudo apt update && \
sudo apt install -y build-essential vim && \
conda config --add channels https://mirrors.ustc.edu.cn/anaconda/pkgs/free/ && \
conda config --add channels https://mirrors.ustc.edu.cn/anaconda/pkgs/main/ && \
conda config --set show_channel_urls yes && \
pip config set global.index-url https://mirrors.aliyun.com/pypi/simple/ && \
pip install torch==1.1.0 torchvision==0.3.0 && \
pip install fastai==1.0.60 && \
pip install ipdb jupyter ipython lmdb editdistance tensorboardX natsort nltk && \
conda uninstall -y --force pillow pil jpeg libtiff libjpeg-turbo && \
pip uninstall -y pillow pil jpeg libtiff libjpeg-turbo && \
conda install -yc conda-forge libjpeg-turbo && \
CFLAGS="${CFLAGS} -mavx2" pip install --no-cache-dir --force-reinstall --no-binary :all: --compile pillow-simd==6.2.2.post1 && \
conda install -y jpeg libtiff opencv && \
sudo rm -rf /var/lib/apt/lists/* && \
sudo rm -rf /tmp/* && \
sudo rm -rf ~/.cache && \
sudo apt clean all && \
conda clean -y -a
EXPOSE 8888
ENV LANG C.UTF-8
ENV LC_ALL C.UTF-8
================================================
FILE: export_onnx.py
================================================
""" Created by MrBBS """
import os
import torch
from utils import Config
def get_model(config):
import importlib
names = config.model_name.split('.')
module_name, class_name = '.'.join(names[:-1]), names[-1]
cls = getattr(importlib.import_module(module_name), class_name)
model = cls(config)
model = model.eval()
return model
def load(model, file, device=None, strict=True):
if device is None:
device = 'cpu'
elif isinstance(device, int):
device = torch.device('cuda', device)
assert os.path.isfile(file)
state = torch.load(file, map_location=device)
if set(state.keys()) == {'model', 'opt'}:
state = state['model']
model.load_state_dict(state, strict=strict)
return model
if __name__ == '__main__':
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('--config', type=str, default='configs/train_abinet.yaml',
help='path to config file')
parser.add_argument('--input', type=str, default='figs/test')
parser.add_argument('--cuda', type=int, default=-1)
parser.add_argument('--checkpoint', type=str, default='workdir_bbs/train-abinet/best-train-abinet.pth')
parser.add_argument('--model_eval', type=str, default='alignment',
choices=['alignment', 'vision', 'language'])
args = parser.parse_args()
config = Config(args.config)
if args.checkpoint is not None: config.model_checkpoint = args.checkpoint
if args.model_eval is not None: config.model_eval = args.model_eval
config.global_phase = 'test'
config.model_vision_checkpoint, config.model_language_checkpoint = None, None
device = 'cpu' if args.cuda < 0 else f'cuda:{args.cuda}'
config.export = True
# Logger.init(config.global_workdir, config.global_name, config.global_phase)
# Logger.enable_file()
print(config)
# logging.info('Construct model.')
model = get_model(config).to(device)
model = load(model, config.model_checkpoint, device=device)
x = torch.rand((1, 3, config.dataset_image_height, config.dataset_image_width), requires_grad=True)
torch.onnx.export(model, x, 'abinet.onnx',
verbose=True, opset_version=13,
do_constant_folding=True,
export_params=True,
input_names=["input"],
output_names=["logits", "lengths"],
dynamic_axes={"input": {0: "batch"}})
================================================
FILE: losses.py
================================================
from fastai.vision import *
from modules.model import Model
class MultiLosses(nn.Module):
def __init__(self, one_hot=True):
super().__init__()
self.ce = SoftCrossEntropyLoss() if one_hot else torch.nn.CrossEntropyLoss()
self.bce = torch.nn.BCELoss()
@property
def last_losses(self):
return self.losses
def _flatten(self, sources, lengths):
return torch.cat([t[:l] for t, l in zip(sources, lengths)])
def _merge_list(self, all_res):
if not isinstance(all_res, (list, tuple)):
return all_res
def merge(items):
if isinstance(items[0], torch.Tensor): return torch.cat(items, dim=0)
else: return items[0]
res = dict()
for key in all_res[0].keys():
items = [r[key] for r in all_res]
res[key] = merge(items)
return res
def _ce_loss(self, output, gt_labels, gt_lengths, idx=None, record=True):
loss_name = output.get('name')
pt_logits, weight = output['logits'], output['loss_weight']
assert pt_logits.shape[0] % gt_labels.shape[0] == 0
iter_size = pt_logits.shape[0] // gt_labels.shape[0]
if iter_size > 1:
gt_labels = gt_labels.repeat(3, 1, 1)
gt_lengths = gt_lengths.repeat(3)
flat_gt_labels = self._flatten(gt_labels, gt_lengths)
flat_pt_logits = self._flatten(pt_logits, gt_lengths)
nll = output.get('nll')
if nll is not None:
loss = self.ce(flat_pt_logits, flat_gt_labels, softmax=False) * weight
else:
loss = self.ce(flat_pt_logits, flat_gt_labels) * weight
if record and loss_name is not None: self.losses[f'{loss_name}_loss'] = loss
return loss
def forward(self, outputs, *args):
self.losses = {}
if isinstance(outputs, (tuple, list)):
outputs = [self._merge_list(o) for o in outputs]
return sum([self._ce_loss(o, *args) for o in outputs if o['loss_weight'] > 0.])
else:
return self._ce_loss(outputs, *args, record=False)
class SoftCrossEntropyLoss(nn.Module):
def __init__(self, reduction="mean"):
super().__init__()
self.reduction = reduction
def forward(self, input, target, softmax=True):
if softmax: log_prob = F.log_softmax(input, dim=-1)
else: log_prob = torch.log(input)
loss = -(target * log_prob).sum(dim=-1)
if self.reduction == "mean": return loss.mean()
elif self.reduction == "sum": return loss.sum()
else: return loss
================================================
FILE: main.py
================================================
import argparse
import logging
import os
import random
import torch
from fastai.callbacks.general_sched import GeneralScheduler, TrainingPhase
from fastai.distributed import *
from fastai.vision import *
from torch.backends import cudnn
from callbacks import DumpPrediction, IterationCallback, TextAccuracy, TopKTextAccuracy
from dataset import ImageDataset, TextDataset
from losses import MultiLosses
from utils import Config, Logger, MyDataParallel, MyConcatDataset
def _set_random_seed(seed):
if seed is not None:
random.seed(seed)
torch.manual_seed(seed)
cudnn.deterministic = True
logging.warning('You have chosen to seed training. '
'This will slow down your training!')
def _get_training_phases(config, n):
lr = np.array(config.optimizer_lr)
periods = config.optimizer_scheduler_periods
sigma = [config.optimizer_scheduler_gamma ** i for i in range(len(periods))]
phases = [TrainingPhase(n * periods[i]).schedule_hp('lr', lr * sigma[i])
for i in range(len(periods))]
return phases
def _get_dataset(ds_type, paths, is_training, config, **kwargs):
kwargs.update({
'img_h': config.dataset_image_height,
'img_w': config.dataset_image_width,
'max_length': config.dataset_max_length,
'case_sensitive': config.dataset_case_sensitive,
'charset_path': config.dataset_charset_path,
'data_aug': config.dataset_data_aug,
'deteriorate_ratio': config.dataset_deteriorate_ratio,
'is_training': is_training,
'multiscales': config.dataset_multiscales,
'one_hot_y': config.dataset_one_hot_y,
})
datasets = [ds_type(p, **kwargs) for p in paths]
if len(datasets) > 1: return MyConcatDataset(datasets)
else: return datasets[0]
def _get_language_databaunch(config):
kwargs = {
'max_length': config.dataset_max_length,
'case_sensitive': config.dataset_case_sensitive,
'charset_path': config.dataset_charset_path,
'smooth_label': config.dataset_smooth_label,
'smooth_factor': config.dataset_smooth_factor,
'one_hot_y': config.dataset_one_hot_y,
'use_sm': config.dataset_use_sm,
}
train_ds = TextDataset(config.dataset_train_roots[0], is_training=True, **kwargs)
valid_ds = TextDataset(config.dataset_test_roots[0], is_training=False, **kwargs)
data = DataBunch.create(
path=train_ds.path,
train_ds=train_ds,
valid_ds=valid_ds,
bs=config.dataset_train_batch_size,
val_bs=config.dataset_test_batch_size,
num_workers=config.dataset_num_workers,
pin_memory=config.dataset_pin_memory)
logging.info(f'{len(data.train_ds)} training items found.')
if not data.empty_val:
logging.info(f'{len(data.valid_ds)} valid items found.')
return data
def _get_databaunch(config):
# An awkward way to reduce loadding data time during test
if config.global_phase == 'test': config.dataset_train_roots = config.dataset_test_roots
train_ds = _get_dataset(ImageDataset, config.dataset_train_roots, True, config)
valid_ds = _get_dataset(ImageDataset, config.dataset_test_roots, False, config)
data = ImageDataBunch.create(
train_ds=train_ds,
valid_ds=valid_ds,
bs=config.dataset_train_batch_size,
val_bs=config.dataset_test_batch_size,
num_workers=config.dataset_num_workers,
pin_memory=config.dataset_pin_memory).normalize(imagenet_stats)
ar_tfm = lambda x: ((x[0], x[1]), x[1]) # auto-regression only for dtd
data.add_tfm(ar_tfm)
logging.info(f'{len(data.train_ds)} training items found.')
if not data.empty_val:
logging.info(f'{len(data.valid_ds)} valid items found.')
return data
def _get_model(config):
import importlib
names = config.model_name.split('.')
module_name, class_name = '.'.join(names[:-1]), names[-1]
cls = getattr(importlib.import_module(module_name), class_name)
model = cls(config)
logging.info(model)
return model
def _get_learner(config, data, model, local_rank=None):
strict = ifnone(config.model_strict, True)
if config.global_stage == 'pretrain-language':
metrics = [TopKTextAccuracy(
k=ifnone(config.model_k, 5),
charset_path=config.dataset_charset_path,
max_length=config.dataset_max_length + 1,
case_sensitive=config.dataset_eval_case_sensisitves,
model_eval=config.model_eval)]
else:
metrics = [TextAccuracy(
charset_path=config.dataset_charset_path,
max_length=config.dataset_max_length + 1,
case_sensitive=config.dataset_eval_case_sensisitves,
model_eval=config.model_eval)]
opt_type = getattr(torch.optim, config.optimizer_type)
learner = Learner(data, model, silent=True, model_dir='.',
true_wd=config.optimizer_true_wd,
wd=config.optimizer_wd,
bn_wd=config.optimizer_bn_wd,
path=config.global_workdir,
metrics=metrics,
opt_func=partial(opt_type, **config.optimizer_args or dict()),
loss_func=MultiLosses(one_hot=config.dataset_one_hot_y))
learner.split(lambda m: children(m))
if config.global_phase == 'train':
num_replicas = 1 if local_rank is None else torch.distributed.get_world_size()
phases = _get_training_phases(config, len(learner.data.train_dl)//num_replicas)
learner.callback_fns += [
partial(GeneralScheduler, phases=phases),
partial(GradientClipping, clip=config.optimizer_clip_grad),
partial(IterationCallback, name=config.global_name,
show_iters=config.training_show_iters,
eval_iters=config.training_eval_iters,
save_iters=config.training_save_iters,
start_iters=config.training_start_iters,
stats_iters=config.training_stats_iters)]
else:
learner.callbacks += [
DumpPrediction(learn=learner,
dataset='-'.join([Path(p).name for p in config.dataset_test_roots]),charset_path=config.dataset_charset_path,
model_eval=config.model_eval,
debug=config.global_debug,
image_only=config.global_image_only)]
learner.rank = local_rank
if local_rank is not None:
logging.info(f'Set model to distributed with rank {local_rank}.')
learner.model = torch.nn.SyncBatchNorm.convert_sync_batchnorm(learner.model)
learner.model.to(local_rank)
learner = learner.to_distributed(local_rank)
if torch.cuda.device_count() > 1 and local_rank is None:
logging.info(f'Use {torch.cuda.device_count()} GPUs.')
learner.model = MyDataParallel(learner.model)
if config.model_checkpoint:
if Path(config.model_checkpoint).exists():
with open(config.model_checkpoint, 'rb') as f:
buffer = io.BytesIO(f.read())
learner.load(buffer, strict=strict)
else:
from distutils.dir_util import copy_tree
src = Path('/data/fangsc/model')/config.global_name
trg = Path('/output')/config.global_name
if src.exists(): copy_tree(str(src), str(trg))
learner.load(config.model_checkpoint, strict=strict)
logging.info(f'Read model from {config.model_checkpoint}')
elif config.global_phase == 'test':
learner.load(f'best-{config.global_name}', strict=strict)
logging.info(f'Read model from best-{config.global_name}')
if learner.opt_func.func.__name__ == 'Adadelta': # fastai bug, fix after 1.0.60
learner.fit(epochs=0, lr=config.optimizer_lr)
learner.opt.mom = 0.
return learner
def main():
parser = argparse.ArgumentParser()
parser.add_argument('--config', type=str, required=True,
help='path to config file')
parser.add_argument('--phase', type=str, default=None, choices=['train', 'test'])
parser.add_argument('--name', type=str, default=None)
parser.add_argument('--checkpoint', type=str, default=None)
parser.add_argument('--test_root', type=str, default=None)
parser.add_argument("--local_rank", type=int, default=None)
parser.add_argument('--debug', action='store_true', default=None)
parser.add_argument('--image_only', action='store_true', default=None)
parser.add_argument('--model_strict', action='store_false', default=None)
parser.add_argument('--model_eval', type=str, default=None,
choices=['alignment', 'vision', 'language'])
args = parser.parse_args()
config = Config(args.config)
if args.name is not None: config.global_name = args.name
if args.phase is not None: config.global_phase = args.phase
if args.test_root is not None: config.dataset_test_roots = [args.test_root]
if args.checkpoint is not None: config.model_checkpoint = args.checkpoint
if args.debug is not None: config.global_debug = args.debug
if args.image_only is not None: config.global_image_only = args.image_only
if args.model_eval is not None: config.model_eval = args.model_eval
if args.model_strict is not None: config.model_strict = args.model_strict
Logger.init(config.global_workdir, config.global_name, config.global_phase)
Logger.enable_file()
_set_random_seed(config.global_seed)
logging.info(config)
if args.local_rank is not None:
logging.info(f'Init distribution training at device {args.local_rank}.')
torch.cuda.set_device(args.local_rank)
torch.distributed.init_process_group(backend='nccl', init_method='env://')
logging.info('Construct dataset.')
if config.global_stage == 'pretrain-language': data = _get_language_databaunch(config)
else: data = _get_databaunch(config)
logging.info('Construct model.')
model = _get_model(config)
logging.info('Construct learner.')
learner = _get_learner(config, data, model, args.local_rank)
if config.global_phase == 'train':
logging.info('Start training.')
learner.fit(epochs=config.training_epochs,
lr=config.optimizer_lr)
else:
logging.info('Start validate')
last_metrics = learner.validate()
log_str = f'eval loss = {last_metrics[0]:6.3f}, ' \
f'ccr = {last_metrics[1]:6.3f}, cwr = {last_metrics[2]:6.3f}, ' \
f'ted = {last_metrics[3]:6.3f}, ned = {last_metrics[4]:6.0f}, ' \
f'ted/w = {last_metrics[5]:6.3f}, '
logging.info(log_str)
if __name__ == '__main__':
main()
================================================
FILE: modules/__init__.py
================================================
================================================
FILE: modules/attention.py
================================================
import torch
import torch.nn as nn
from .transformer import PositionalEncoding
class Attention(nn.Module):
def __init__(self, in_channels=512, max_length=25, n_feature=256):
super().__init__()
self.max_length = max_length
self.f0_embedding = nn.Embedding(max_length, in_channels)
self.w0 = nn.Linear(max_length, n_feature)
self.wv = nn.Linear(in_channels, in_channels)
self.we = nn.Linear(in_channels, max_length)
self.active = nn.Tanh()
self.softmax = nn.Softmax(dim=2)
def forward(self, enc_output):
enc_output = enc_output.permute(0, 2, 3, 1).flatten(1, 2)
reading_order = torch.arange(self.max_length, dtype=torch.long, device=enc_output.device)
reading_order = reading_order.unsqueeze(0).expand(enc_output.size(0), -1) # (S,) -> (B, S)
reading_order_embed = self.f0_embedding(reading_order) # b,25,512
t = self.w0(reading_order_embed.permute(0, 2, 1)) # b,512,256
t = self.active(t.permute(0, 2, 1) + self.wv(enc_output)) # b,256,512
attn = self.we(t) # b,256,25
attn = self.softmax(attn.permute(0, 2, 1)) # b,25,256
g_output = torch.bmm(attn, enc_output) # b,25,512
return g_output, attn.view(*attn.shape[:2], 8, 32)
def encoder_layer(in_c, out_c, k=3, s=2, p=1):
return nn.Sequential(nn.Conv2d(in_c, out_c, k, s, p),
nn.BatchNorm2d(out_c),
nn.ReLU(True))
def decoder_layer(in_c, out_c, k=3, s=1, p=1, mode='nearest', scale_factor=None, size=None):
align_corners = None if mode=='nearest' else True
return nn.Sequential(nn.Upsample(size=size, scale_factor=scale_factor,
mode=mode, align_corners=align_corners),
nn.Conv2d(in_c, out_c, k, s, p),
nn.BatchNorm2d(out_c),
nn.ReLU(True))
class PositionAttention(nn.Module):
def __init__(self, max_length, in_channels=512, num_channels=64,
h=8, w=32, mode='nearest', **kwargs):
super().__init__()
self.max_length = max_length
self.k_encoder = nn.Sequential(
encoder_layer(in_channels, num_channels, s=(1, 2)),
encoder_layer(num_channels, num_channels, s=(2, 2)),
encoder_layer(num_channels, num_channels, s=(2, 2)),
encoder_layer(num_channels, num_channels, s=(2, 2))
)
self.k_decoder = nn.Sequential(
decoder_layer(num_channels, num_channels, scale_factor=2, mode=mode),
decoder_layer(num_channels, num_channels, scale_factor=2, mode=mode),
decoder_layer(num_channels, num_channels, scale_factor=2, mode=mode),
decoder_layer(num_channels, in_channels, size=(h, w), mode=mode)
)
self.pos_encoder = PositionalEncoding(in_channels, dropout=0, max_len=max_length)
self.project = nn.Linear(in_channels, in_channels)
def forward(self, x):
N, E, H, W = x.size()
k, v = x, x # (N, E, H, W)
# calculate key vector
features = []
for i in range(0, len(self.k_encoder)):
k = self.k_encoder[i](k)
features.append(k)
for i in range(0, len(self.k_decoder) - 1):
k = self.k_decoder[i](k)
k = k + features[len(self.k_decoder) - 2 - i]
k = self.k_decoder[-1](k)
# calculate query vector
# TODO q=f(q,k)
zeros = x.new_zeros((self.max_length, N, E)) # (T, N, E)
q = self.pos_encoder(zeros) # (T, N, E)
q = q.permute(1, 0, 2) # (N, T, E)
q = self.project(q) # (N, T, E)
# calculate attention
attn_scores = torch.bmm(q, k.flatten(2, 3)) # (N, T, (H*W))
attn_scores = attn_scores / (E ** 0.5)
attn_scores = torch.softmax(attn_scores, dim=-1)
v = v.permute(0, 2, 3, 1).view(N, -1, E) # (N, (H*W), E)
attn_vecs = torch.bmm(attn_scores, v) # (N, T, E)
return attn_vecs, attn_scores.view(N, -1, H, W)
================================================
FILE: modules/backbone.py
================================================
from fastai.vision import *
from modules.model import _default_tfmer_cfg
from modules.resnet import resnet45
from modules.transformer import (PositionalEncoding,
TransformerEncoder,
TransformerEncoderLayer)
class ResTranformer(nn.Module):
def __init__(self, config):
super().__init__()
self.resnet = resnet45()
self.d_model = ifnone(config.model_vision_d_model, _default_tfmer_cfg['d_model'])
nhead = ifnone(config.model_vision_nhead, _default_tfmer_cfg['nhead'])
d_inner = ifnone(config.model_vision_d_inner, _default_tfmer_cfg['d_inner'])
dropout = ifnone(config.model_vision_dropout, _default_tfmer_cfg['dropout'])
activation = ifnone(config.model_vision_activation, _default_tfmer_cfg['activation'])
num_layers = ifnone(config.model_vision_backbone_ln, 2)
self.pos_encoder = PositionalEncoding(self.d_model, max_len=8*32)
encoder_layer = TransformerEncoderLayer(d_model=self.d_model, nhead=nhead,
dim_feedforward=d_inner, dropout=dropout, activation=activation)
self.transformer = TransformerEncoder(encoder_layer, num_layers)
def forward(self, images):
feature = self.resnet(images)
n, c, h, w = feature.shape
feature = feature.view(n, c, -1).permute(2, 0, 1)
feature = self.pos_encoder(feature)
feature = self.transformer(feature)
feature = feature.permute(1, 2, 0).view(n, c, h, w)
return feature
================================================
FILE: modules/backbone_v2.py
================================================
""" Created by MrBBS """
# 10/11/2022
# -*-encoding:utf-8-*-
================================================
FILE: modules/model.py
================================================
import torch
import torch.nn as nn
import numpy as np
from utils import CharsetMapper
_default_tfmer_cfg = dict(d_model=512, nhead=8, d_inner=2048, # 1024
dropout=0.1, activation='relu')
class Model(nn.Module):
def __init__(self, config):
super().__init__()
self.max_length = config.dataset_max_length + 1
self.charset = CharsetMapper(config.dataset_charset_path, max_length=self.max_length)
def load(self, source, device=None, strict=True):
state = torch.load(source, map_location=device)
self.load_state_dict(state['model'], strict=strict)
def _get_length(self, logit):
""" Greed decoder to obtain length from logit"""
out = (logit.argmax(dim=-1) == self.charset.null_label)
out = self.first_nonzero(out.int()) + 1
return out
@staticmethod
def first_nonzero(x):
non_zero_mask = x != 0
mask_max_values, mask_max_indices = torch.max(non_zero_mask.int(), dim=-1)
mask_max_indices[mask_max_values == 0] = -1
return mask_max_indices
@staticmethod
def _get_padding_mask(length, max_length):
length = length.unsqueeze(-1)
grid = torch.arange(0, max_length, device=length.device).unsqueeze(0)
return grid >= length
@staticmethod
def _get_square_subsequent_mask(sz, device, diagonal=0, fw=True):
r"""Generate a square mask for the sequence. The masked positions are filled with float('-inf').
Unmasked positions are filled with float(0.0).
"""
mask = (torch.triu(torch.ones(sz, sz, device=device), diagonal=diagonal) == 1)
if fw: mask = mask.transpose(0, 1)
mask = mask.float().masked_fill(mask == 0, float('-inf')).masked_fill(mask == 1, float(0.0))
return mask
@staticmethod
def _get_location_mask(sz, device=None):
mask = torch.eye(sz, device=device)
mask = mask.float().masked_fill(mask == 1, float('-inf'))
return mask
================================================
FILE: modules/model_abinet.py
================================================
from fastai.vision import *
from .model_alignment import BaseAlignment
from .model_language import BCNLanguage
from .model_vision import BaseVision
class ABINetModel(nn.Module):
def __init__(self, config):
super().__init__()
self.use_alignment = ifnone(config.model_use_alignment, True)
self.max_length = config.dataset_max_length + 1 # additional stop token
self.vision = BaseVision(config)
self.language = BCNLanguage(config)
if self.use_alignment: self.alignment = BaseAlignment(config)
def forward(self, images, *args):
v_res = self.vision(images)
v_tokens = torch.softmax(v_res['logits'], dim=-1)
v_lengths = v_res['pt_lengths'].clamp_(2, self.max_length) # TODO:move to langauge model
l_res = self.language(v_tokens, v_lengths)
if not self.use_alignment:
return l_res, v_res
l_feature, v_feature = l_res['feature'], v_res['feature']
a_res = self.alignment(l_feature, v_feature)
return a_res, l_res, v_res
================================================
FILE: modules/model_abinet_iter.py
================================================
import torch
import torch.nn.functional as F
from fastai.vision import *
from .model_vision import BaseVision
from .model_language import BCNLanguage
from .model_alignment import BaseAlignment
class ABINetIterModel(nn.Module):
def __init__(self, config):
super().__init__()
self.iter_size = ifnone(config.model_iter_size, 1)
self.max_length = config.dataset_max_length + 1 # additional stop token
self.vision = BaseVision(config)
self.language = BCNLanguage(config)
self.alignment = BaseAlignment(config)
self.export = config.export
def forward(self, images, *args):
v_res = self.vision(images)
a_res = v_res
all_l_res, all_a_res = [], []
for _ in range(self.iter_size):
tokens = torch.softmax(a_res['logits'], dim=-1)
lengths = a_res['pt_lengths']
lengths.clamp_(2, self.max_length) # TODO:move to langauge model
l_res = self.language(tokens, lengths)
all_l_res.append(l_res)
a_res = self.alignment(l_res['feature'], v_res['feature'])
all_a_res.append(a_res)
if self.export:
return F.softmax(a_res['logits'], dim=2), a_res['pt_lengths']
if self.training:
return all_a_res, all_l_res, v_res
else:
return a_res, all_l_res[-1], v_res
================================================
FILE: modules/model_alignment.py
================================================
import torch
import torch.nn as nn
from fastai.vision import *
from modules.model import Model, _default_tfmer_cfg
class BaseAlignment(Model):
def __init__(self, config):
super().__init__(config)
d_model = ifnone(config.model_alignment_d_model, _default_tfmer_cfg['d_model'])
self.loss_weight = ifnone(config.model_alignment_loss_weight, 1.0)
self.max_length = config.dataset_max_length + 1 # additional stop token
self.w_att = nn.Linear(2 * d_model, d_model)
self.cls = nn.Linear(d_model, self.charset.num_classes)
def forward(self, l_feature, v_feature):
"""
Args:
l_feature: (N, T, E) where T is length, N is batch size and d is dim of model
v_feature: (N, T, E) shape the same as l_feature
l_lengths: (N,)
v_lengths: (N,)
"""
f = torch.cat((l_feature, v_feature), dim=2)
f_att = torch.sigmoid(self.w_att(f))
output = f_att * v_feature + (1 - f_att) * l_feature
logits = self.cls(output) # (N, T, C)
pt_lengths = self._get_length(logits)
return {'logits': logits, 'pt_lengths': pt_lengths, 'loss_weight':self.loss_weight,
'name': 'alignment'}
================================================
FILE: modules/model_language.py
================================================
import logging
import torch.nn as nn
from fastai.vision import *
from modules.model import _default_tfmer_cfg
from modules.model import Model
from modules.transformer import (PositionalEncoding,
TransformerDecoder,
TransformerDecoderLayer)
class BCNLanguage(Model):
def __init__(self, config):
super().__init__(config)
d_model = ifnone(config.model_language_d_model, _default_tfmer_cfg['d_model'])
nhead = ifnone(config.model_language_nhead, _default_tfmer_cfg['nhead'])
d_inner = ifnone(config.model_language_d_inner, _default_tfmer_cfg['d_inner'])
dropout = ifnone(config.model_language_dropout, _default_tfmer_cfg['dropout'])
activation = ifnone(config.model_language_activation, _default_tfmer_cfg['activation'])
num_layers = ifnone(config.model_language_num_layers, 4)
self.d_model = d_model
self.detach = ifnone(config.model_language_detach, True)
self.use_self_attn = ifnone(config.model_language_use_self_attn, False)
self.loss_weight = ifnone(config.model_language_loss_weight, 1.0)
self.max_length = config.dataset_max_length + 1 # additional stop token
self.debug = ifnone(config.global_debug, False)
self.proj = nn.Linear(self.charset.num_classes, d_model, False)
self.token_encoder = PositionalEncoding(d_model, max_len=self.max_length)
self.pos_encoder = PositionalEncoding(d_model, dropout=0, max_len=self.max_length)
decoder_layer = TransformerDecoderLayer(d_model, nhead, d_inner, dropout,
activation, self_attn=self.use_self_attn, debug=self.debug)
self.model = TransformerDecoder(decoder_layer, num_layers)
self.cls = nn.Linear(d_model, self.charset.num_classes)
if config.model_language_checkpoint is not None:
logging.info(f'Read language model from {config.model_language_checkpoint}.')
self.load(config.model_language_checkpoint)
def forward(self, tokens, lengths):
"""
Args:
tokens: (N, T, C) where T is length, N is batch size and C is classes number
lengths: (N,)
"""
if self.detach: tokens = tokens.detach()
embed = self.proj(tokens) # (N, T, E)
embed = embed.permute(1, 0, 2) # (T, N, E)
embed = self.token_encoder(embed) # (T, N, E)
padding_mask = self._get_padding_mask(lengths, self.max_length)
zeros = embed.new_zeros(*embed.shape)
qeury = self.pos_encoder(zeros)
location_mask = self._get_location_mask(self.max_length, tokens.device)
output = self.model(qeury, embed,
tgt_key_padding_mask=padding_mask,
memory_mask=location_mask,
memory_key_padding_mask=padding_mask) # (T, N, E)
output = output.permute(1, 0, 2) # (N, T, E)
logits = self.cls(output) # (N, T, C)
pt_lengths = self._get_length(logits)
res = {'feature': output, 'logits': logits, 'pt_lengths': pt_lengths,
'loss_weight':self.loss_weight, 'name': 'language'}
return res
================================================
FILE: modules/model_vision.py
================================================
import logging
import torch.nn as nn
from fastai.vision import *
from modules.attention import *
from modules.backbone import ResTranformer
from modules.model import Model
from modules.resnet import resnet45
class BaseVision(Model):
def __init__(self, config):
super().__init__(config)
self.loss_weight = ifnone(config.model_vision_loss_weight, 1.0)
self.out_channels = ifnone(config.model_vision_d_model, 512)
if config.model_vision_backbone == 'transformer':
self.backbone = ResTranformer(config)
else: self.backbone = resnet45()
if config.model_vision_attention == 'position':
mode = ifnone(config.model_vision_attention_mode, 'nearest')
self.attention = PositionAttention(
max_length=config.dataset_max_length + 1, # additional stop token
mode=mode,
)
elif config.model_vision_attention == 'attention':
self.attention = Attention(
max_length=config.dataset_max_length + 1, # additional stop token
n_feature=8*32,
)
else:
raise Exception(f'{config.model_vision_attention} is not valid.')
self.cls = nn.Linear(self.out_channels, self.charset.num_classes)
if config.model_vision_checkpoint is not None:
logging.info(f'Read vision model from {config.model_vision_checkpoint}.')
self.load(config.model_vision_checkpoint)
def forward(self, images, *args):
features = self.backbone(images) # (N, E, H, W)
attn_vecs, attn_scores = self.attention(features) # (N, T, E), (N, T, H, W)
logits = self.cls(attn_vecs) # (N, T, C)
pt_lengths = self._get_length(logits)
return {'feature': attn_vecs, 'logits': logits, 'pt_lengths': pt_lengths,
'attn_scores': attn_scores, 'loss_weight':self.loss_weight, 'name': 'vision'}
================================================
FILE: modules/resnet.py
================================================
import math
import torch.nn as nn
import torch.nn.functional as F
import torch.utils.model_zoo as model_zoo
def conv1x1(in_planes, out_planes, stride=1):
return nn.Conv2d(in_planes, out_planes, kernel_size=1, stride=stride, bias=False)
def conv3x3(in_planes, out_planes, stride=1):
"3x3 convolution with padding"
return nn.Conv2d(in_planes, out_planes, kernel_size=3, stride=stride,
padding=1, bias=False)
class BasicBlock(nn.Module):
expansion = 1
def __init__(self, inplanes, planes, stride=1, downsample=None):
super(BasicBlock, self).__init__()
self.conv1 = conv1x1(inplanes, planes)
self.bn1 = nn.BatchNorm2d(planes)
self.relu = nn.ReLU(inplace=True)
self.conv2 = conv3x3(planes, planes, stride)
self.bn2 = nn.BatchNorm2d(planes)
self.downsample = downsample
self.stride = stride
def forward(self, x):
residual = x
out = self.conv1(x)
out = self.bn1(out)
out = self.relu(out)
out = self.conv2(out)
out = self.bn2(out)
if self.downsample is not None:
residual = self.downsample(x)
out += residual
out = self.relu(out)
return out
class ResNet(nn.Module):
def __init__(self, block, layers):
self.inplanes = 32
super(ResNet, self).__init__()
self.conv1 = nn.Conv2d(3, 32, kernel_size=3, stride=1, padding=1,
bias=False)
self.bn1 = nn.BatchNorm2d(32)
self.relu = nn.ReLU(inplace=True)
self.layer1 = self._make_layer(block, 32, layers[0], stride=2)
self.layer2 = self._make_layer(block, 64, layers[1], stride=1)
self.layer3 = self._make_layer(block, 128, layers[2], stride=2)
self.layer4 = self._make_layer(block, 256, layers[3], stride=1)
self.layer5 = self._make_layer(block, 512, layers[4], stride=1)
for m in self.modules():
if isinstance(m, nn.Conv2d):
n = m.kernel_size[0] * m.kernel_size[1] * m.out_channels
m.weight.data.normal_(0, math.sqrt(2. / n))
elif isinstance(m, nn.BatchNorm2d):
m.weight.data.fill_(1)
m.bias.data.zero_()
def _make_layer(self, block, planes, blocks, stride=1):
downsample = None
if stride != 1 or self.inplanes != planes * block.expansion:
downsample = nn.Sequential(
nn.Conv2d(self.inplanes, planes * block.expansion,
kernel_size=1, stride=stride, bias=False),
nn.BatchNorm2d(planes * block.expansion),
)
layers = []
layers.append(block(self.inplanes, planes, stride, downsample))
self.inplanes = planes * block.expansion
for i in range(1, blocks):
layers.append(block(self.inplanes, planes))
return nn.Sequential(*layers)
def forward(self, x):
x = self.conv1(x)
x = self.bn1(x)
x = self.relu(x)
x = self.layer1(x)
x = self.layer2(x)
x = self.layer3(x)
x = self.layer4(x)
x = self.layer5(x)
return x
def resnet45():
return ResNet(BasicBlock, [3, 4, 6, 6, 3])
================================================
FILE: modules/transformer.py
================================================
# pytorch 1.5.0
import copy
import math
import warnings
from typing import Optional
import torch
import torch.nn as nn
from torch import Tensor
from torch.nn import Dropout, LayerNorm, Linear, Module, ModuleList, Parameter
from torch.nn import functional as F
from torch.nn.init import constant_, xavier_uniform_
def multi_head_attention_forward(query, # type: Tensor
key, # type: Tensor
value, # type: Tensor
embed_dim_to_check, # type: int
num_heads, # type: int
in_proj_weight, # type: Tensor
in_proj_bias, # type: Tensor
bias_k, # type: Optional[Tensor]
bias_v, # type: Optional[Tensor]
add_zero_attn, # type: bool
dropout_p, # type: float
out_proj_weight, # type: Tensor
out_proj_bias, # type: Tensor
training=True, # type: bool
key_padding_mask=None, # type: Optional[Tensor]
need_weights=True, # type: bool
attn_mask=None, # type: Optional[Tensor]
use_separate_proj_weight=False, # type: bool
q_proj_weight=None, # type: Optional[Tensor]
k_proj_weight=None, # type: Optional[Tensor]
v_proj_weight=None, # type: Optional[Tensor]
static_k=None, # type: Optional[Tensor]
static_v=None # type: Optional[Tensor]
):
# type: (...) -> Tuple[Tensor, Optional[Tensor]]
r"""
Args:
query, key, value: map a query and a set of key-value pairs to an output.
See "Attention Is All You Need" for more details.
embed_dim_to_check: total dimension of the model.
num_heads: parallel attention heads.
in_proj_weight, in_proj_bias: input projection weight and bias.
bias_k, bias_v: bias of the key and value sequences to be added at dim=0.
add_zero_attn: add a new batch of zeros to the key and
value sequences at dim=1.
dropout_p: probability of an element to be zeroed.
out_proj_weight, out_proj_bias: the output projection weight and bias.
training: apply dropout if is ``True``.
key_padding_mask: if provided, specified padding elements in the key will
be ignored by the attention. This is an binary mask. When the value is True,
the corresponding value on the attention layer will be filled with -inf.
need_weights: output attn_output_weights.
attn_mask: 2D or 3D mask that prevents attention to certain positions. A 2D mask will be broadcasted for all
the batches while a 3D mask allows to specify a different mask for the entries of each batch.
use_separate_proj_weight: the function accept the proj. weights for query, key,
and value in different forms. If false, in_proj_weight will be used, which is
a combination of q_proj_weight, k_proj_weight, v_proj_weight.
q_proj_weight, k_proj_weight, v_proj_weight, in_proj_bias: input projection weight and bias.
static_k, static_v: static key and value used for attention operators.
Shape:
Inputs:
- query: :math:`(L, N, E)` where L is the target sequence length, N is the batch size, E is
the embedding dimension.
- key: :math:`(S, N, E)`, where S is the source sequence length, N is the batch size, E is
the embedding dimension.
- value: :math:`(S, N, E)` where S is the source sequence length, N is the batch size, E is
the embedding dimension.
- key_padding_mask: :math:`(N, S)` where N is the batch size, S is the source sequence length.
If a ByteTensor is provided, the non-zero positions will be ignored while the zero positions
will be unchanged. If a BoolTensor is provided, the positions with the
value of ``True`` will be ignored while the position with the value of ``False`` will be unchanged.
- attn_mask: 2D mask :math:`(L, S)` where L is the target sequence length, S is the source sequence length.
3D mask :math:`(N*num_heads, L, S)` where N is the batch size, L is the target sequence length,
S is the source sequence length. attn_mask ensures that position i is allowed to attend the unmasked
positions. If a ByteTensor is provided, the non-zero positions are not allowed to attend
while the zero positions will be unchanged. If a BoolTensor is provided, positions with ``True``
are not allowed to attend while ``False`` values will be unchanged. If a FloatTensor
is provided, it will be added to the attention weight.
- static_k: :math:`(N*num_heads, S, E/num_heads)`, where S is the source sequence length,
N is the batch size, E is the embedding dimension. E/num_heads is the head dimension.
- static_v: :math:`(N*num_heads, S, E/num_heads)`, where S is the source sequence length,
N is the batch size, E is the embedding dimension. E/num_heads is the head dimension.
Outputs:
- attn_output: :math:`(L, N, E)` where L is the target sequence length, N is the batch size,
E is the embedding dimension.
- attn_output_weights: :math:`(N, L, S)` where N is the batch size,
L is the target sequence length, S is the source sequence length.
"""
# if not torch.jit.is_scripting():
# tens_ops = (query, key, value, in_proj_weight, in_proj_bias, bias_k, bias_v,
# out_proj_weight, out_proj_bias)
# if any([type(t) is not Tensor for t in tens_ops]) and has_torch_function(tens_ops):
# return handle_torch_function(
# multi_head_attention_forward, tens_ops, query, key, value,
# embed_dim_to_check, num_heads, in_proj_weight, in_proj_bias,
# bias_k, bias_v, add_zero_attn, dropout_p, out_proj_weight,
# out_proj_bias, training=training, key_padding_mask=key_padding_mask,
# need_weights=need_weights, attn_mask=attn_mask,
# use_separate_proj_weight=use_separate_proj_weight,
# q_proj_weight=q_proj_weight, k_proj_weight=k_proj_weight,
# v_proj_weight=v_proj_weight, static_k=static_k, static_v=static_v)
tgt_len, bsz, embed_dim = query.size()
assert embed_dim == embed_dim_to_check
assert key.size() == value.size()
head_dim = embed_dim // num_heads
assert head_dim * num_heads == embed_dim, "embed_dim must be divisible by num_heads"
scaling = float(head_dim) ** -0.5
if not use_separate_proj_weight:
if torch.equal(query, key) and torch.equal(key, value):
# self-attention
q, k, v = F.linear(query, in_proj_weight, in_proj_bias).chunk(3, dim=-1)
elif torch.equal(key, value):
# encoder-decoder attention
# This is inline in_proj function with in_proj_weight and in_proj_bias
_b = in_proj_bias
_start = 0
_end = embed_dim
_w = in_proj_weight[_start:_end, :]
if _b is not None:
_b = _b[_start:_end]
q = F.linear(query, _w, _b)
if key is None:
assert value is None
k = None
v = None
else:
# This is inline in_proj function with in_proj_weight and in_proj_bias
_b = in_proj_bias
_start = embed_dim
_end = None
_w = in_proj_weight[_start:, :]
if _b is not None:
_b = _b[_start:]
k, v = F.linear(key, _w, _b).chunk(2, dim=-1)
else:
# This is inline in_proj function with in_proj_weight and in_proj_bias
_b = in_proj_bias
_start = 0
_end = embed_dim
_w = in_proj_weight[_start:_end, :]
if _b is not None:
_b = _b[_start:_end]
q = F.linear(query, _w, _b)
# This is inline in_proj function with in_proj_weight and in_proj_bias
_b = in_proj_bias
_start = embed_dim
_end = embed_dim * 2
_w = in_proj_weight[_start:_end, :]
if _b is not None:
_b = _b[_start:_end]
k = F.linear(key, _w, _b)
# This is inline in_proj function with in_proj_weight and in_proj_bias
_b = in_proj_bias
_start = embed_dim * 2
_end = None
_w = in_proj_weight[_start:, :]
if _b is not None:
_b = _b[_start:]
v = F.linear(value, _w, _b)
else:
q_proj_weight_non_opt = torch.jit._unwrap_optional(q_proj_weight)
len1, len2 = q_proj_weight_non_opt.size()
assert len1 == embed_dim and len2 == query.size(-1)
k_proj_weight_non_opt = torch.jit._unwrap_optional(k_proj_weight)
len1, len2 = k_proj_weight_non_opt.size()
assert len1 == embed_dim and len2 == key.size(-1)
v_proj_weight_non_opt = torch.jit._unwrap_optional(v_proj_weight)
len1, len2 = v_proj_weight_non_opt.size()
assert len1 == embed_dim and len2 == value.size(-1)
if in_proj_bias is not None:
q = F.linear(query, q_proj_weight_non_opt, in_proj_bias[0:embed_dim])
k = F.linear(key, k_proj_weight_non_opt, in_proj_bias[embed_dim:(embed_dim * 2)])
v = F.linear(value, v_proj_weight_non_opt, in_proj_bias[(embed_dim * 2):])
else:
q = F.linear(query, q_proj_weight_non_opt, in_proj_bias)
k = F.linear(key, k_proj_weight_non_opt, in_proj_bias)
v = F.linear(value, v_proj_weight_non_opt, in_proj_bias)
q = q * scaling
if attn_mask is not None:
assert attn_mask.dtype == torch.float32 or attn_mask.dtype == torch.float64 or \
attn_mask.dtype == torch.float16 or attn_mask.dtype == torch.uint8 or attn_mask.dtype == torch.bool, \
'Only float, byte, and bool types are supported for attn_mask, not {}'.format(attn_mask.dtype)
if attn_mask.dtype == torch.uint8:
warnings.warn("Byte tensor for attn_mask in nn.MultiheadAttention is deprecated. Use bool tensor instead.")
attn_mask = attn_mask.to(torch.bool)
if attn_mask.dim() == 2:
attn_mask = attn_mask.unsqueeze(0)
if list(attn_mask.size()) != [1, query.size(0), key.size(0)]:
raise RuntimeError('The size of the 2D attn_mask is not correct.')
elif attn_mask.dim() == 3:
if list(attn_mask.size()) != [bsz * num_heads, query.size(0), key.size(0)]:
raise RuntimeError('The size of the 3D attn_mask is not correct.')
else:
raise RuntimeError("attn_mask's dimension {} is not supported".format(attn_mask.dim()))
# attn_mask's dim is 3 now.
# # convert ByteTensor key_padding_mask to bool
# if key_padding_mask is not None and key_padding_mask.dtype == torch.uint8:
# warnings.warn("Byte tensor for key_padding_mask in nn.MultiheadAttention is deprecated. Use bool tensor instead.")
# key_padding_mask = key_padding_mask.to(torch.bool)
if bias_k is not None and bias_v is not None:
if static_k is None and static_v is None:
k = torch.cat([k, bias_k.repeat(1, bsz, 1)])
v = torch.cat([v, bias_v.repeat(1, bsz, 1)])
if attn_mask is not None:
attn_mask = pad(attn_mask, (0, 1))
if key_padding_mask is not None:
key_padding_mask = pad(key_padding_mask, (0, 1))
else:
assert static_k is None, "bias cannot be added to static key."
assert static_v is None, "bias cannot be added to static value."
else:
assert bias_k is None
assert bias_v is None
q = q.contiguous().view(tgt_len, bsz * num_heads, head_dim).transpose(0, 1)
if k is not None:
k = k.contiguous().view(-1, bsz * num_heads, head_dim).transpose(0, 1)
if v is not None:
v = v.contiguous().view(-1, bsz * num_heads, head_dim).transpose(0, 1)
if static_k is not None:
assert static_k.size(0) == bsz * num_heads
assert static_k.size(2) == head_dim
k = static_k
if static_v is not None:
assert static_v.size(0) == bsz * num_heads
assert static_v.size(2) == head_dim
v = static_v
src_len = k.size(1)
if key_padding_mask is not None:
assert key_padding_mask.size(0) == bsz
assert key_padding_mask.size(1) == src_len
if add_zero_attn:
src_len += 1
k = torch.cat([k, torch.zeros((k.size(0), 1) + k.size()[2:], dtype=k.dtype, device=k.device)], dim=1)
v = torch.cat([v, torch.zeros((v.size(0), 1) + v.size()[2:], dtype=v.dtype, device=v.device)], dim=1)
if attn_mask is not None:
attn_mask = pad(attn_mask, (0, 1))
if key_padding_mask is not None:
key_padding_mask = pad(key_padding_mask, (0, 1))
attn_output_weights = torch.bmm(q, k.transpose(1, 2))
assert list(attn_output_weights.size()) == [bsz * num_heads, tgt_len, src_len]
if attn_mask is not None:
if attn_mask.dtype == torch.bool:
attn_output_weights.masked_fill_(attn_mask, float('-inf'))
else:
attn_output_weights += attn_mask
if key_padding_mask is not None:
attn_output_weights = attn_output_weights.view(bsz, num_heads, tgt_len, src_len)
attn_output_weights = attn_output_weights.masked_fill(
key_padding_mask.unsqueeze(1).unsqueeze(2),
float('-inf'),
)
attn_output_weights = attn_output_weights.view(bsz * num_heads, tgt_len, src_len)
attn_output_weights = F.softmax(
attn_output_weights, dim=-1)
attn_output_weights = F.dropout(attn_output_weights, p=dropout_p, training=training)
attn_output = torch.bmm(attn_output_weights, v)
assert list(attn_output.size()) == [bsz * num_heads, tgt_len, head_dim]
attn_output = attn_output.transpose(0, 1).contiguous().view(tgt_len, bsz, embed_dim)
attn_output = F.linear(attn_output, out_proj_weight, out_proj_bias)
if need_weights:
# average attention weights over heads
attn_output_weights = attn_output_weights.view(bsz, num_heads, tgt_len, src_len)
return attn_output, attn_output_weights.sum(dim=1) / num_heads
else:
return attn_output, None
class MultiheadAttention(Module):
r"""Allows the model to jointly attend to information
from different representation subspaces.
See reference: Attention Is All You Need
.. math::
\text{MultiHead}(Q, K, V) = \text{Concat}(head_1,\dots,head_h)W^O
\text{where} head_i = \text{Attention}(QW_i^Q, KW_i^K, VW_i^V)
Args:
embed_dim: total dimension of the model.
num_heads: parallel attention heads.
dropout: a Dropout layer on attn_output_weights. Default: 0.0.
bias: add bias as module parameter. Default: True.
add_bias_kv: add bias to the key and value sequences at dim=0.
add_zero_attn: add a new batch of zeros to the key and
value sequences at dim=1.
kdim: total number of features in key. Default: None.
vdim: total number of features in value. Default: None.
Note: if kdim and vdim are None, they will be set to embed_dim such that
query, key, and value have the same number of features.
Examples::
>>> multihead_attn = nn.MultiheadAttention(embed_dim, num_heads)
>>> attn_output, attn_output_weights = multihead_attn(query, key, value)
"""
# __annotations__ = {
# 'bias_k': torch._jit_internal.Optional[torch.Tensor],
# 'bias_v': torch._jit_internal.Optional[torch.Tensor],
# }
__constants__ = ['q_proj_weight', 'k_proj_weight', 'v_proj_weight', 'in_proj_weight']
def __init__(self, embed_dim, num_heads, dropout=0., bias=True, add_bias_kv=False, add_zero_attn=False, kdim=None, vdim=None):
super(MultiheadAttention, self).__init__()
self.embed_dim = embed_dim
self.kdim = kdim if kdim is not None else embed_dim
self.vdim = vdim if vdim is not None else embed_dim
self._qkv_same_embed_dim = self.kdim == embed_dim and self.vdim == embed_dim
self.num_heads = num_heads
self.dropout = dropout
self.head_dim = embed_dim // num_heads
assert self.head_dim * num_heads == self.embed_dim, "embed_dim must be divisible by num_heads"
if self._qkv_same_embed_dim is False:
self.q_proj_weight = Parameter(torch.Tensor(embed_dim, embed_dim))
self.k_proj_weight = Parameter(torch.Tensor(embed_dim, self.kdim))
self.v_proj_weight = Parameter(torch.Tensor(embed_dim, self.vdim))
self.register_parameter('in_proj_weight', None)
else:
self.in_proj_weight = Parameter(torch.empty(3 * embed_dim, embed_dim))
self.register_parameter('q_proj_weight', None)
self.register_parameter('k_proj_weight', None)
self.register_parameter('v_proj_weight', None)
if bias:
self.in_proj_bias = Parameter(torch.empty(3 * embed_dim))
else:
self.register_parameter('in_proj_bias', None)
self.out_proj = Linear(embed_dim, embed_dim, bias=bias)
if add_bias_kv:
self.bias_k = Parameter(torch.empty(1, 1, embed_dim))
self.bias_v = Parameter(torch.empty(1, 1, embed_dim))
else:
self.bias_k = self.bias_v = None
self.add_zero_attn = add_zero_attn
self._reset_parameters()
def _reset_parameters(self):
if self._qkv_same_embed_dim:
xavier_uniform_(self.in_proj_weight)
else:
xavier_uniform_(self.q_proj_weight)
xavier_uniform_(self.k_proj_weight)
xavier_uniform_(self.v_proj_weight)
if self.in_proj_bias is not None:
constant_(self.in_proj_bias, 0.)
constant_(self.out_proj.bias, 0.)
if self.bias_k is not None:
xavier_normal_(self.bias_k)
if self.bias_v is not None:
xavier_normal_(self.bias_v)
def __setstate__(self, state):
# Support loading old MultiheadAttention checkpoints generated by v1.1.0
if '_qkv_same_embed_dim' not in state:
state['_qkv_same_embed_dim'] = True
super(MultiheadAttention, self).__setstate__(state)
def forward(self, query, key, value, key_padding_mask=None,
need_weights=True, attn_mask=None):
# type: (Tensor, Tensor, Tensor, Optional[Tensor], bool, Optional[Tensor]) -> Tuple[Tensor, Optional[Tensor]]
r"""
Args:
query, key, value: map a query and a set of key-value pairs to an output.
See "Attention Is All You Need" for more details.
key_padding_mask: if provided, specified padding elements in the key will
be ignored by the attention. This is an binary mask. When the value is True,
the corresponding value on the attention layer will be filled with -inf.
need_weights: output attn_output_weights.
attn_mask: 2D or 3D mask that prevents attention to certain positions. A 2D mask will be broadcasted for all
the batches while a 3D mask allows to specify a different mask for the entries of each batch.
Shape:
- Inputs:
- query: :math:`(L, N, E)` where L is the target sequence length, N is the batch size, E is
the embedding dimension.
- key: :math:`(S, N, E)`, where S is the source sequence length, N is the batch size, E is
the embedding dimension.
- value: :math:`(S, N, E)` where S is the source sequence length, N is the batch size, E is
the embedding dimension.
- key_padding_mask: :math:`(N, S)` where N is the batch size, S is the source sequence length.
If a ByteTensor is provided, the non-zero positions will be ignored while the position
with the zero positions will be unchanged. If a BoolTensor is provided, the positions with the
value of ``True`` will be ignored while the position with the value of ``False`` will be unchanged.
- attn_mask: 2D mask :math:`(L, S)` where L is the target sequence length, S is the source sequence length.
3D mask :math:`(N*num_heads, L, S)` where N is the batch size, L is the target sequence length,
S is the source sequence length. attn_mask ensure that position i is allowed to attend the unmasked
positions. If a ByteTensor is provided, the non-zero positions are not allowed to attend
while the zero positions will be unchanged. If a BoolTensor is provided, positions with ``True``
is not allowed to attend while ``False`` values will be unchanged. If a FloatTensor
is provided, it will be added to the attention weight.
- Outputs:
- attn_output: :math:`(L, N, E)` where L is the target sequence length, N is the batch size,
E is the embedding dimension.
- attn_output_weights: :math:`(N, L, S)` where N is the batch size,
L is the target sequence length, S is the source sequence length.
"""
if not self._qkv_same_embed_dim:
return multi_head_attention_forward(
query, key, value, self.embed_dim, self.num_heads,
self.in_proj_weight, self.in_proj_bias,
self.bias_k, self.bias_v, self.add_zero_attn,
self.dropout, self.out_proj.weight, self.out_proj.bias,
training=self.training,
key_padding_mask=key_padding_mask, need_weights=need_weights,
attn_mask=attn_mask, use_separate_proj_weight=True,
q_proj_weight=self.q_proj_weight, k_proj_weight=self.k_proj_weight,
v_proj_weight=self.v_proj_weight)
else:
return multi_head_attention_forward(
query, key, value, self.embed_dim, self.num_heads,
self.in_proj_weight, self.in_proj_bias,
self.bias_k, self.bias_v, self.add_zero_attn,
self.dropout, self.out_proj.weight, self.out_proj.bias,
training=self.training,
key_padding_mask=key_padding_mask, need_weights=need_weights,
attn_mask=attn_mask)
class Transformer(Module):
r"""A transformer model. User is able to modify the attributes as needed. The architecture
is based on the paper "Attention Is All You Need". Ashish Vaswani, Noam Shazeer,
Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and
Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information
Processing Systems, pages 6000-6010. Users can build the BERT(https://arxiv.org/abs/1810.04805)
model with corresponding parameters.
Args:
d_model: the number of expected features in the encoder/decoder inputs (default=512).
nhead: the number of heads in the multiheadattention models (default=8).
num_encoder_layers: the number of sub-encoder-layers in the encoder (default=6).
num_decoder_layers: the number of sub-decoder-layers in the decoder (default=6).
dim_feedforward: the dimension of the feedforward network model (default=2048).
dropout: the dropout value (default=0.1).
activation: the activation function of encoder/decoder intermediate layer, relu or gelu (default=relu).
custom_encoder: custom encoder (default=None).
custom_decoder: custom decoder (default=None).
Examples::
>>> transformer_model = nn.Transformer(nhead=16, num_encoder_layers=12)
>>> src = torch.rand((10, 32, 512))
>>> tgt = torch.rand((20, 32, 512))
>>> out = transformer_model(src, tgt)
Note: A full example to apply nn.Transformer module for the word language model is available in
https://github.com/pytorch/examples/tree/master/word_language_model
"""
def __init__(self, d_model=512, nhead=8, num_encoder_layers=6,
num_decoder_layers=6, dim_feedforward=2048, dropout=0.1,
activation="relu", custom_encoder=None, custom_decoder=None):
super(Transformer, self).__init__()
if custom_encoder is not None:
self.encoder = custom_encoder
else:
encoder_layer = TransformerEncoderLayer(d_model, nhead, dim_feedforward, dropout, activation)
encoder_norm = LayerNorm(d_model)
self.encoder = TransformerEncoder(encoder_layer, num_encoder_layers, encoder_norm)
if custom_decoder is not None:
self.decoder = custom_decoder
else:
decoder_layer = TransformerDecoderLayer(d_model, nhead, dim_feedforward, dropout, activation)
decoder_norm = LayerNorm(d_model)
self.decoder = TransformerDecoder(decoder_layer, num_decoder_layers, decoder_norm)
self._reset_parameters()
self.d_model = d_model
self.nhead = nhead
def forward(self, src, tgt, src_mask=None, tgt_mask=None,
memory_mask=None, src_key_padding_mask=None,
tgt_key_padding_mask=None, memory_key_padding_mask=None):
# type: (Tensor, Tensor, Optional[Tensor], Optional[Tensor], Optional[Tensor], Optional[Tensor], Optional[Tensor], Optional[Tensor]) -> Tensor # noqa
r"""Take in and process masked source/target sequences.
Args:
src: the sequence to the encoder (required).
tgt: the sequence to the decoder (required).
src_mask: the additive mask for the src sequence (optional).
tgt_mask: the additive mask for the tgt sequence (optional).
memory_mask: the additive mask for the encoder output (optional).
src_key_padding_mask: the ByteTensor mask for src keys per batch (optional).
tgt_key_padding_mask: the ByteTensor mask for tgt keys per batch (optional).
memory_key_padding_mask: the ByteTensor mask for memory keys per batch (optional).
Shape:
- src: :math:`(S, N, E)`.
- tgt: :math:`(T, N, E)`.
- src_mask: :math:`(S, S)`.
- tgt_mask: :math:`(T, T)`.
- memory_mask: :math:`(T, S)`.
- src_key_padding_mask: :math:`(N, S)`.
- tgt_key_padding_mask: :math:`(N, T)`.
- memory_key_padding_mask: :math:`(N, S)`.
Note: [src/tgt/memory]_mask ensures that position i is allowed to attend the unmasked
positions. If a ByteTensor is provided, the non-zero positions are not allowed to attend
while the zero positions will be unchanged. If a BoolTensor is provided, positions with ``True``
are not allowed to attend while ``False`` values will be unchanged. If a FloatTensor
is provided, it will be added to the attention weight.
[src/tgt/memory]_key_padding_mask provides specified elements in the key to be ignored by
the attention. If a ByteTensor is provided, the non-zero positions will be ignored while the zero
positions will be unchanged. If a BoolTensor is provided, the positions with the
value of ``True`` will be ignored while the position with the value of ``False`` will be unchanged.
- output: :math:`(T, N, E)`.
Note: Due to the multi-head attention architecture in the transformer model,
the output sequence length of a transformer is same as the input sequence
(i.e. target) length of the decode.
where S is the source sequence length, T is the target sequence length, N is the
batch size, E is the feature number
Examples:
>>> output = transformer_model(src, tgt, src_mask=src_mask, tgt_mask=tgt_mask)
"""
if src.size(1) != tgt.size(1):
raise RuntimeError("the batch number of src and tgt must be equal")
if src.size(2) != self.d_model or tgt.size(2) != self.d_model:
raise RuntimeError("the feature number of src and tgt must be equal to d_model")
memory = self.encoder(src, mask=src_mask, src_key_padding_mask=src_key_padding_mask)
output = self.decoder(tgt, memory, tgt_mask=tgt_mask, memory_mask=memory_mask,
tgt_key_padding_mask=tgt_key_padding_mask,
memory_key_padding_mask=memory_key_padding_mask)
return output
def generate_square_subsequent_mask(self, sz):
r"""Generate a square mask for the sequence. The masked positions are filled with float('-inf').
Unmasked positions are filled with float(0.0).
"""
mask = (torch.triu(torch.ones(sz, sz)) == 1).transpose(0, 1)
mask = mask.float().masked_fill(mask == 0, float('-inf')).masked_fill(mask == 1, float(0.0))
return mask
def _reset_parameters(self):
r"""Initiate parameters in the transformer model."""
for p in self.parameters():
if p.dim() > 1:
xavier_uniform_(p)
class TransformerEncoder(Module):
r"""TransformerEncoder is a stack of N encoder layers
Args:
encoder_layer: an instance of the TransformerEncoderLayer() class (required).
num_layers: the number of sub-encoder-layers in the encoder (required).
norm: the layer normalization component (optional).
Examples::
>>> encoder_layer = nn.TransformerEncoderLayer(d_model=512, nhead=8)
>>> transformer_encoder = nn.TransformerEncoder(encoder_layer, num_layers=6)
>>> src = torch.rand(10, 32, 512)
>>> out = transformer_encoder(src)
"""
__constants__ = ['norm']
def __init__(self, encoder_layer, num_layers, norm=None):
super(TransformerEncoder, self).__init__()
self.layers = _get_clones(encoder_layer, num_layers)
self.num_layers = num_layers
self.norm = norm
def forward(self, src, mask=None, src_key_padding_mask=None):
# type: (Tensor, Optional[Tensor], Optional[Tensor]) -> Tensor
r"""Pass the input through the encoder layers in turn.
Args:
src: the sequence to the encoder (required).
mask: the mask for the src sequence (optional).
src_key_padding_mask: the mask for the src keys per batch (optional).
Shape:
see the docs in Transformer class.
"""
output = src
for i, mod in enumerate(self.layers):
output = mod(output, src_mask=mask, src_key_padding_mask=src_key_padding_mask)
if self.norm is not None:
output = self.norm(output)
return output
class TransformerDecoder(Module):
r"""TransformerDecoder is a stack of N decoder layers
Args:
decoder_layer: an instance of the TransformerDecoderLayer() class (required).
num_layers: the number of sub-decoder-layers in the decoder (required).
norm: the layer normalization component (optional).
Examples::
>>> decoder_layer = nn.TransformerDecoderLayer(d_model=512, nhead=8)
>>> transformer_decoder = nn.TransformerDecoder(decoder_layer, num_layers=6)
>>> memory = torch.rand(10, 32, 512)
>>> tgt = torch.rand(20, 32, 512)
>>> out = transformer_decoder(tgt, memory)
"""
__constants__ = ['norm']
def __init__(self, decoder_layer, num_layers, norm=None):
super(TransformerDecoder, self).__init__()
self.layers = _get_clones(decoder_layer, num_layers)
self.num_layers = num_layers
self.norm = norm
def forward(self, tgt, memory, memory2=None, tgt_mask=None,
memory_mask=None, memory_mask2=None, tgt_key_padding_mask=None,
memory_key_padding_mask=None, memory_key_padding_mask2=None):
# type: (Tensor, Tensor, Optional[Tensor], Optional[Tensor], Optional[Tensor], Optional[Tensor]) -> Tensor
r"""Pass the inputs (and mask) through the decoder layer in turn.
Args:
tgt: the sequence to the decoder (required).
memory: the sequence from the last layer of the encoder (required).
tgt_mask: the mask for the tgt sequence (optional).
memory_mask: the mask for the memory sequence (optional).
tgt_key_padding_mask: the mask for the tgt keys per batch (optional).
memory_key_padding_mask: the mask for the memory keys per batch (optional).
Shape:
see the docs in Transformer class.
"""
output = tgt
for mod in self.layers:
output = mod(output, memory, memory2=memory2, tgt_mask=tgt_mask,
memory_mask=memory_mask, memory_mask2=memory_mask2,
tgt_key_padding_mask=tgt_key_padding_mask,
memory_key_padding_mask=memory_key_padding_mask,
memory_key_padding_mask2=memory_key_padding_mask2)
if self.norm is not None:
output = self.norm(output)
return output
class TransformerEncoderLayer(Module):
r"""TransformerEncoderLayer is made up of self-attn and feedforward network.
This standard encoder layer is based on the paper "Attention Is All You Need".
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez,
Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in
Neural Information Processing Systems, pages 6000-6010. Users may modify or implement
in a different way during application.
Args:
d_model: the number of expected features in the input (required).
nhead: the number of heads in the multiheadattention models (required).
dim_feedforward: the dimension of the feedforward network model (default=2048).
dropout: the dropout value (default=0.1).
activation: the activation function of intermediate layer, relu or gelu (default=relu).
Examples::
>>> encoder_layer = nn.TransformerEncoderLayer(d_model=512, nhead=8)
>>> src = torch.rand(10, 32, 512)
>>> out = encoder_layer(src)
"""
def __init__(self, d_model, nhead, dim_feedforward=2048, dropout=0.1,
activation="relu", debug=False):
super(TransformerEncoderLayer, self).__init__()
self.debug = debug
self.self_attn = MultiheadAttention(d_model, nhead, dropout=dropout)
# Implementation of Feedforward model
self.linear1 = Linear(d_model, dim_feedforward)
self.dropout = Dropout(dropout)
self.linear2 = Linear(dim_feedforward, d_model)
self.norm1 = LayerNorm(d_model)
self.norm2 = LayerNorm(d_model)
self.dropout1 = Dropout(dropout)
self.dropout2 = Dropout(dropout)
self.activation = _get_activation_fn(activation)
def __setstate__(self, state):
if 'activation' not in state:
state['activation'] = F.relu
super(TransformerEncoderLayer, self).__setstate__(state)
def forward(self, src, src_mask=None, src_key_padding_mask=None):
# type: (Tensor, Optional[Tensor], Optional[Tensor]) -> Tensor
r"""Pass the input through the encoder layer.
Args:
src: the sequence to the encoder layer (required).
src_mask: the mask for the src sequence (optional).
src_key_padding_mask: the mask for the src keys per batch (optional).
Shape:
see the docs in Transformer class.
"""
src2, attn = self.self_attn(src, src, src, attn_mask=src_mask,
key_padding_mask=src_key_padding_mask)
if self.debug: self.attn = attn
src = src + self.dropout1(src2)
src = self.norm1(src)
src2 = self.linear2(self.dropout(self.activation(self.linear1(src))))
src = src + self.dropout2(src2)
src = self.norm2(src)
return src
class TransformerDecoderLayer(Module):
r"""TransformerDecoderLayer is made up of self-attn, multi-head-attn and feedforward network.
This standard decoder layer is based on the paper "Attention Is All You Need".
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez,
Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in
Neural Information Processing Systems, pages 6000-6010. Users may modify or implement
in a different way during application.
Args:
d_model: the number of expected features in the input (required).
nhead: the number of heads in the multiheadattention models (required).
dim_feedforward: the dimension of the feedforward network model (default=2048).
dropout: the dropout value (default=0.1).
activation: the activation function of intermediate layer, relu or gelu (default=relu).
Examples::
>>> decoder_layer = nn.TransformerDecoderLayer(d_model=512, nhead=8)
>>> memory = torch.rand(10, 32, 512)
>>> tgt = torch.rand(20, 32, 512)
>>> out = decoder_layer(tgt, memory)
"""
def __init__(self, d_model, nhead, dim_feedforward=2048, dropout=0.1,
activation="relu", self_attn=True, siamese=False, debug=False):
super(TransformerDecoderLayer, self).__init__()
self.has_self_attn, self.siamese = self_attn, siamese
self.debug = debug
if self.has_self_attn:
self.self_attn = MultiheadAttention(d_model, nhead, dropout=dropout)
self.norm1 = LayerNorm(d_model)
self.dropout1 = Dropout(dropout)
self.multihead_attn = MultiheadAttention(d_model, nhead, dropout=dropout)
# Implementation of Feedforward model
self.linear1 = Linear(d_model, dim_feedforward)
self.dropout = Dropout(dropout)
self.linear2 = Linear(dim_feedforward, d_model)
self.norm2 = LayerNorm(d_model)
self.norm3 = LayerNorm(d_model)
self.dropout2 = Dropout(dropout)
self.dropout3 = Dropout(dropout)
if self.siamese:
self.multihead_attn2 = MultiheadAttention(d_model, nhead, dropout=dropout)
self.activation = _get_activation_fn(activation)
def __setstate__(self, state):
if 'activation' not in state:
state['activation'] = F.relu
super(TransformerDecoderLayer, self).__setstate__(state)
def forward(self, tgt, memory, tgt_mask=None, memory_mask=None,
tgt_key_padding_mask=None, memory_key_padding_mask=None,
memory2=None, memory_mask2=None, memory_key_padding_mask2=None):
# type: (Tensor, Tensor, Optional[Tensor], Optional[Tensor], Optional[Tensor], Optional[Tensor]) -> Tensor
r"""Pass the inputs (and mask) through the decoder layer.
Args:
tgt: the sequence to the decoder layer (required).
memory: the sequence from the last layer of the encoder (required).
tgt_mask: the mask for the tgt sequence (optional).
memory_mask: the mask for the memory sequence (optional).
tgt_key_padding_mask: the mask for the tgt keys per batch (optional).
memory_key_padding_mask: the mask for the memory keys per batch (optional).
Shape:
see the docs in Transformer class.
"""
if self.has_self_attn:
tgt2, attn = self.self_attn(tgt, tgt, tgt, attn_mask=tgt_mask,
key_padding_mask=tgt_key_padding_mask)
tgt = tgt + self.dropout1(tgt2)
tgt = self.norm1(tgt)
if self.debug: self.attn = attn
tgt2, attn2 = self.multihead_attn(tgt, memory, memory, attn_mask=memory_mask,
key_padding_mask=memory_key_padding_mask)
if self.debug: self.attn2 = attn2
if self.siamese:
tgt3, attn3 = self.multihead_attn2(tgt, memory2, memory2, attn_mask=memory_mask2,
key_padding_mask=memory_key_padding_mask2)
tgt = tgt + self.dropout2(tgt3)
if self.debug: self.attn3 = attn3
tgt = tgt + self.dropout2(tgt2)
tgt = self.norm2(tgt)
tgt2 = self.linear2(self.dropout(self.activation(self.linear1(tgt))))
tgt = tgt + self.dropout3(tgt2)
tgt = self.norm3(tgt)
return tgt
def _get_clones(module, N):
return ModuleList([copy.deepcopy(module) for i in range(N)])
def _get_activation_fn(activation):
if activation == "relu":
return F.relu
elif activation == "gelu":
return F.gelu
raise RuntimeError("activation should be relu/gelu, not {}".format(activation))
class PositionalEncoding(nn.Module):
r"""Inject some information about the relative or absolute position of the tokens
in the sequence. The positional encodings have the same dimension as
the embeddings, so that the two can be summed. Here, we use sine and cosine
functions of different frequencies.
.. math::
\text{PosEncoder}(pos, 2i) = sin(pos/10000^(2i/d_model))
\text{PosEncoder}(pos, 2i+1) = cos(pos/10000^(2i/d_model))
\text{where pos is the word position and i is the embed idx)
Args:
d_model: the embed dim (required).
dropout: the dropout value (default=0.1).
max_len: the max. length of the incoming sequence (default=5000).
Examples:
>>> pos_encoder = PositionalEncoding(d_model)
"""
def __init__(self, d_model, dropout=0.1, max_len=5000):
super(PositionalEncoding, self).__init__()
self.dropout = nn.Dropout(p=dropout)
pe = torch.zeros(max_len, d_model)
position = torch.arange(0, max_len, dtype=torch.float).unsqueeze(1)
div_term = torch.exp(torch.arange(0, d_model, 2).float() * (-math.log(10000.0) / d_model))
pe[:, 0::2] = torch.sin(position * div_term)
pe[:, 1::2] = torch.cos(position * div_term)
pe = pe.unsqueeze(0).transpose(0, 1)
self.register_buffer('pe', pe)
def forward(self, x):
r"""Inputs of forward function
Args:
x: the sequence fed to the positional encoder model (required).
Shape:
x: [sequence length, batch size, embed dim]
output: [sequence length, batch size, embed dim]
Examples:
>>> output = pos_encoder(x)
"""
x = x + self.pe[:x.size(0), :]
return self.dropout(x)
if __name__ == '__main__':
transformer_model = Transformer(nhead=16, num_encoder_layers=12)
src = torch.rand((10, 32, 512))
tgt = torch.rand((20, 32, 512))
out = transformer_model(src, tgt)
print(out)
================================================
FILE: notebooks/dataset-text.ipynb
================================================
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import os\n",
"os.chdir('..')\n",
"from dataset import *\n",
"torch.set_printoptions(sci_mode=False)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Construct dataset"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"data = TextDataset('data/Vocabulary_train_v2.csv', is_training=False, smooth_label=True, smooth_factor=0.1)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"data = DataBunch.create(train_ds=data, valid_ds=None, bs=6)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"x, y = data.one_batch(); x, y"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"x[0].shape, x[1].shape"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"y[0].shape, y[1].shape"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"x[0].argmax(-1) - y[0].argmax(-1)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"x[0].argmax(-1)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"y[0].argmax(-1)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"x[0][0,0]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# test SpellingMutation"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"probs = {'pn0': 0., 'pn1': 0., 'pn2': 0., 'pt0': 1.0, 'pt1': 1.0}\n",
"charset = CharsetMapper('data/charset_36.txt')\n",
"sm = SpellingMutation(charset=charset, **probs)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"sm('*a-aa')"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.4"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
================================================
FILE: notebooks/dataset.ipynb
================================================
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import os\n",
"os.chdir('..')\n",
"from dataset import *"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": false
},
"outputs": [],
"source": [
"import logging\n",
"from torchvision.transforms import ToPILImage\n",
"from torchvision.utils import make_grid\n",
"from IPython.display import display\n",
"from torch.utils.data import ConcatDataset\n",
"charset = CharsetMapper('data/charset_36.txt')"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def show_all(dl, iter_size=None):\n",
" if iter_size is None: iter_size = len(dl)\n",
" for i, item in enumerate(dl):\n",
" if i >= iter_size:\n",
" break\n",
" image = item[0]\n",
" label = item[1][0]\n",
" length = item[1][1]\n",
" print(f'iter {i}:', [charset.get_text(label[j][0: length[j]].argmax(-1), padding=False) for j in range(bs)])\n",
" display(ToPILImage()(make_grid(item[0].cpu())))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Construct dataset"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"data1 = ImageDataset('data/training/ST', is_training=True);data1 # is_training"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"bs=64\n",
"data2 = ImageDataBunch.create(train_ds=data1, valid_ds=None, bs=bs, num_workers=1);data2"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"#data3 = data2.normalize(imagenet_stats);data3\n",
"data3 = data2"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"show_all(data3.train_dl, 4)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Add dataset"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"kwargs = {'data_aug': False, 'is_training': False}"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"data1 = ImageDataset('data/evaluation/IIIT5k_3000', **kwargs);data1"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"data2 = ImageDataset('data/evaluation/SVT', **kwargs);data2"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"data3 = ConcatDataset([data1, data2])"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"bs=64\n",
"data4 = ImageDataBunch.create(train_ds=data1, valid_ds=data3, bs=bs, num_workers=1);data4"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"len(data4.train_dl), len(data4.valid_dl)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"show_all(data4.train_dl, 4)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# TEST"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"len(data4.valid_dl)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import time\n",
"niter = 1000\n",
"start = time.time()\n",
"for i, item in enumerate(progress_bar(data4.valid_dl)):\n",
" if i % niter == 0 and i > 0:\n",
" print(i, (time.time() - start) / niter)\n",
" start = time.time()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"num = 20\n",
"index = 6\n",
"plt.figure(figsize=(20, 10))\n",
"for i in range(num):\n",
" plt.subplot(num // 4, 4, i+1)\n",
" plt.imshow(data4.train_ds[i][0].data.numpy().transpose(1,2,0))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def show(path, image_key):\n",
" with lmdb.open(str(path), readonly=True, lock=False, readahead=False, meminit=False).begin(write=False) as txn:\n",
" imgbuf = txn.get(image_key.encode()) # image\n",
" buf = six.BytesIO()\n",
" buf.write(imgbuf)\n",
" buf.seek(0)\n",
" with warnings.catch_warnings():\n",
" warnings.simplefilter(\"ignore\", UserWarning) # EXIF warning from TiffPlugin\n",
" x = PIL.Image.open(buf).convert('RGB')\n",
" print(x.size)\n",
" plt.imshow(x)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"image_key = 'image-003118258'\n",
"image_key = 'image-002780217'\n",
"image_key = 'image-002780218'\n",
"path = 'data/CVPR2016'\n",
"show(path, image_key)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"image_key = 'image-004668347'\n",
"image_key = 'image-006128516'\n",
"path = 'data/NIPS2014'\n",
"show(path, image_key)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"image_key = 'image-004668347'\n",
"image_key = 'image-000002420'\n",
"path = 'data/IIIT5K_3000'\n",
"show(path, image_key)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.4"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
================================================
FILE: notebooks/prepare_wikitext103.ipynb
================================================
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# 82841986 is_char and is_digit"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# 82075350 regrex non-ascii and none-digit"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 86460763 left"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"import os\n",
"import random\n",
"import re\n",
"import pandas as pd"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"max_length = 25\n",
"min_length = 1\n",
"root = '../data'\n",
"charset = 'abcdefghijklmnopqrstuvwxyz'\n",
"digits = '0123456789'"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"def is_char(text, ratio=0.5):\n",
" text = text.lower()\n",
" length = max(len(text), 1)\n",
" char_num = sum([t in charset for t in text])\n",
" if char_num < min_length: return False\n",
" if char_num / length < ratio: return False\n",
" return True\n",
"\n",
"def is_digit(text, ratio=0.5):\n",
" length = max(len(text), 1)\n",
" digit_num = sum([t in digits for t in text])\n",
" if digit_num / length < ratio: return False\n",
" return True"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# generate training dataset"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
"with open('/tmp/wikitext-103/wiki.train.tokens', 'r') as file:\n",
" lines = file.readlines()"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [],
"source": [
"inp, gt = [], []\n",
"for line in lines:\n",
" token = line.lower().split()\n",
" for text in token:\n",
" text = re.sub('[^0-9a-zA-Z]+', '', text)\n",
" if len(text) < min_length:\n",
" # print('short-text', text)\n",
" continue\n",
" if len(text) > max_length:\n",
" # print('long-text', text)\n",
" continue\n",
" inp.append(text)\n",
" gt.append(text)"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [],
"source": [
"train_voc = os.path.join(root, 'WikiText-103.csv')\n",
"pd.DataFrame({'inp':inp, 'gt':gt}).to_csv(train_voc, index=None, sep='\\t')"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"86460763"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"len(inp)"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"['valkyria',\n",
" 'chronicles',\n",
" 'iii',\n",
" 'senj',\n",
" 'no',\n",
" 'valkyria',\n",
" '3',\n",
" 'unk',\n",
" 'chronicles',\n",
" 'japanese',\n",
" '3',\n",
" 'lit',\n",
" 'valkyria',\n",
" 'of',\n",
" 'the',\n",
" 'battlefield',\n",
" '3',\n",
" 'commonly',\n",
" 'referred',\n",
" 'to',\n",
" 'as',\n",
" 'valkyria',\n",
" 'chronicles',\n",
" 'iii',\n",
" 'outside',\n",
" 'japan',\n",
" 'is',\n",
" 'a',\n",
" 'tactical',\n",
" 'role',\n",
" 'playing',\n",
" 'video',\n",
" 'game',\n",
" 'developed',\n",
" 'by',\n",
" 'sega',\n",
" 'and',\n",
" 'mediavision',\n",
" 'for',\n",
" 'the',\n",
" 'playstation',\n",
" 'portable',\n",
" 'released',\n",
" 'in',\n",
" 'january',\n",
" '2011',\n",
" 'in',\n",
" 'japan',\n",
" 'it',\n",
" 'is',\n",
" 'the',\n",
" 'third',\n",
" 'game',\n",
" 'in',\n",
" 'the',\n",
" 'valkyria',\n",
" 'series',\n",
" 'employing',\n",
" 'the',\n",
" 'same',\n",
" 'fusion',\n",
" 'of',\n",
" 'tactical',\n",
" 'and',\n",
" 'real',\n",
" 'time',\n",
" 'gameplay',\n",
" 'as',\n",
" 'its',\n",
" 'predecessors',\n",
" 'the',\n",
" 'story',\n",
" 'runs',\n",
" 'parallel',\n",
" 'to',\n",
" 'the',\n",
" 'first',\n",
" 'game',\n",
" 'and',\n",
" 'follows',\n",
" 'the',\n",
" 'nameless',\n",
" 'a',\n",
" 'penal',\n",
" 'military',\n",
" 'unit',\n",
" 'serving',\n",
" 'the',\n",
" 'nation',\n",
" 'of',\n",
" 'gallia',\n",
" 'during',\n",
" 'the',\n",
" 'second',\n",
" 'europan',\n",
" 'war',\n",
" 'who',\n",
" 'perform',\n",
" 'secret',\n",
" 'black']"
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"inp[:100]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# generate evaluation dataset"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [],
"source": [
"def disturb(word, degree, p=0.3):\n",
" if len(word) // 2 < degree: return word\n",
" if is_digit(word): return word\n",
" if random.random() < p: return word\n",
" else:\n",
" index = list(range(len(word)))\n",
" random.shuffle(index)\n",
" index = index[:degree]\n",
" new_word = []\n",
" for i in range(len(word)):\n",
" if i not in index: \n",
" new_word.append(word[i])\n",
" continue\n",
" if (word[i] not in charset) and (word[i] not in digits):\n",
" # special token\n",
" new_word.append(word[i])\n",
" continue\n",
" op = random.random()\n",
" if op < 0.1: # add\n",
" new_word.append(random.choice(charset))\n",
" new_word.append(word[i])\n",
" elif op < 0.2: continue # remove\n",
" else: new_word.append(random.choice(charset)) # replace\n",
" return ''.join(new_word)"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [],
"source": [
"lines = inp\n",
"degree = 1\n",
"keep_num = 50000\n",
"\n",
"random.shuffle(lines)\n",
"part_lines = lines[:keep_num]\n",
"inp, gt = [], []\n",
"\n",
"for w in part_lines:\n",
" w = w.strip().lower()\n",
" new_w = disturb(w, degree)\n",
" inp.append(new_w)\n",
" gt.append(w)\n",
" \n",
"eval_voc = os.path.join(root, f'WikiText-103_eval_d{degree}.csv')\n",
"pd.DataFrame({'inp':inp, 'gt':gt}).to_csv(eval_voc, index=None, sep='\\t')"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[('high', 'high'),\n",
" ('vctoria', 'victoria'),\n",
" ('mains', 'mains'),\n",
" ('bi', 'by'),\n",
" ('13', '13'),\n",
" ('ticnet', 'ticket'),\n",
" ('basil', 'basic'),\n",
" ('cut', 'cut'),\n",
" ('aqarky', 'anarky'),\n",
" ('the', 'the'),\n",
" ('tqe', 'the'),\n",
" ('oc', 'of'),\n",
" ('diwpersal', 'dispersal'),\n",
" ('traffic', 'traffic'),\n",
" ('in', 'in'),\n",
" ('the', 'the'),\n",
" ('ti', 'to'),\n",
" ('professionalms', 'professionals'),\n",
" ('747', '747'),\n",
" ('in', 'in'),\n",
" ('and', 'and'),\n",
" ('exezutive', 'executive'),\n",
" ('n400', 'n400'),\n",
" ('yusic', 'music'),\n",
" ('s', 's'),\n",
" ('henri', 'henry'),\n",
" ('heard', 'heard'),\n",
" ('thousand', 'thousand'),\n",
" ('to', 'to'),\n",
" ('arhy', 'army'),\n",
" ('td', 'to'),\n",
" ('a', 'a'),\n",
" ('oall', 'hall'),\n",
" ('qind', 'kind'),\n",
" ('od', 'on'),\n",
" ('samfria', 'samaria'),\n",
" ('driveway', 'driveway'),\n",
" ('which', 'which'),\n",
" ('wotk', 'work'),\n",
" ('ak', 'as'),\n",
" ('persona', 'persona'),\n",
" ('s', 's'),\n",
" ('melbourne', 'melbourne'),\n",
" ('apong', 'along'),\n",
" ('fas', 'was'),\n",
" ('thea', 'then'),\n",
" ('permcy', 'percy'),\n",
" ('nnd', 'and'),\n",
" ('alan', 'alan'),\n",
" ('13', '13'),\n",
" ('matteos', 'matters'),\n",
" ('against', 'against'),\n",
" ('nefion', 'nexion'),\n",
" ('held', 'held'),\n",
" ('negative', 'negative'),\n",
" ('gogd', 'good'),\n",
" ('the', 'the'),\n",
" ('thd', 'the'),\n",
" ('groening', 'groening'),\n",
" ('tqe', 'the'),\n",
" ('cwould', 'would'),\n",
" ('fb', 'ft'),\n",
" ('uniten', 'united'),\n",
" ('kone', 'one'),\n",
" ('thiy', 'this'),\n",
" ('lanren', 'lauren'),\n",
" ('s', 's'),\n",
" ('thhe', 'the'),\n",
" ('is', 'is'),\n",
" ('modep', 'model'),\n",
" ('weird', 'weird'),\n",
" ('angwer', 'answer'),\n",
" ('imprisxnment', 'imprisonment'),\n",
" ('marpery', 'margery'),\n",
" ('eventuanly', 'eventually'),\n",
" ('in', 'in'),\n",
" ('donnoa', 'donna'),\n",
" ('ik', 'it'),\n",
" ('reached', 'reached'),\n",
" ('at', 'at'),\n",
" ('excxted', 'excited'),\n",
" ('ws', 'was'),\n",
" ('raes', 'rates'),\n",
" ('the', 'the'),\n",
" ('firsq', 'first'),\n",
" ('concluyed', 'concluded'),\n",
" ('recdorded', 'recorded'),\n",
" ('fhe', 'the'),\n",
" ('uegiment', 'regiment'),\n",
" ('a', 'a'),\n",
" ('glanes', 'planes'),\n",
" ('conyrol', 'control'),\n",
" ('thr', 'the'),\n",
" ('arrext', 'arrest'),\n",
" ('bth', 'both'),\n",
" ('forward', 'forward'),\n",
" ('allowdd', 'allowed'),\n",
" ('revealed', 'revealed'),\n",
" ('mayagement', 'management'),\n",
" ('normal', 'normal')]"
]
},
"execution_count": 11,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"list(zip(inp, gt))[:100]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.4"
}
},
"nbformat": 4,
"nbformat_minor": 4
}
================================================
FILE: notebooks/transforms.ipynb
================================================
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Preparation"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"%matplotlib inline\n",
"from matplotlib import pyplot as plt\n",
"from PIL import Image"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"import random\n",
"import numbers\n",
"import cv2\n",
"import math\n",
"import numpy as np\n",
"from torchvision.transforms import Compose"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"img = Image.open('../data/test1.jpg')\n",
"img2 = Image.open('../data/test2.jpg')"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"scrolled": true
},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAQIAAAAiCAIAAAA79kCFAAA/mklEQVR4nGW915JkSZIlpszsEmdBk1dldVeTYTtLIRABvg8PeMIf4RUQ2ZXd2emdmZ7u4pU0MpizS8yU4MGzZncFLiESHi7u193vNVM9es5RDfzDi//LzMIqERFRzjkCzcwBiKhpulormqqquxMRACAiM6oqEZkZIouIOSCihiNioJsZqBGRCBFREKoqEgGAFhMRciulNF1LSdy91EoMRFRNmRkMAaDWCgDCHBHuioiMUEoJRBJGEHUL4sxCwHUuKbGZ1Xnu+z4QAQC0mhmaSk7MaTZ1hAAwQ2ZGJjPDACJCYvBACPTQOru7ZGZmdQAAZgYAm4u7BwLR6RGvHohIgkRkDgBwOieqXt1SYsopAqdaNBwI1aGUghEYsGybxCIQIqLqEeEGzIwUtdaMiIhlHokoIpgZkM0rIqYkyDTVUtyqW3Wv4YGATExNrdWLM+KiXbQiEMFoLafwalrAFJhOX9zCwxkRRQQAmMUB5nmmJlVVI6ims6m7VfCIQMREHGpompEXeZElkbu7A2FEBAURJUYzA1VOEgiz6aRhDiIJgNzA3cOVCNy97RoEMjMg5pTmeeZWaq3VKkREBKq7O2r0fd/nBgDGMiNjACgomOaczeJ0jTR0HMcKKk2u4czsCKiAEWFAgIlShDFRZkos4QpqESbMKSKKRSAEgjGWUkpVRwjHBaIQOxIgIwOKqOpUhtNqEBFEFKRqlUiyCFQ/7RMAMHQFHyatXpkTCQvkWgo6diyBRLkpbuPuWEznOlGS0/UgotBYdEuWJMSMpKoWhgjIAuKHcbDZOTUgpHMBgJayVW2CGem0gR0gzEopRERMiGgREeERSIQYqsU1hnmqtbJIztkRvGpOSUuJsBYbdmZOjOTuiTl+iQIO5mrIQAHqxtJAYJgiYoQTEUqUYa5IHjMIK7tCUJMQODJ7BJoPoGiV3JIlLbVr+rZpa602l5QSRlSdiQgRETEQAIyEU5KgGMs0u86uc5gzYxLKYggawX0bczHDiXSqM6i1RJ68YSIUTuwIDgHkFBQARKco4wBASaiRWct2ODhhJKaWU9MSo7oDACN5VSukiiOHoaEZIzY5QwSAm5l7ICLnhEz7cVCMSm5MTjLXydyYEmCEacMCIq5GTYoIjTrYBIWAwiTMLCI4IwYD+XY6DtN4tt70fT/Ns0VpsxghIkrDRetYx0CfszmRN+jEsugTEXgQsM82D+NcK6qTQg1aIjKQh0KEiCSHMK/HMoJgci9ejEAkh0FozeSMlHJizo4+RR3J51DpknslEvY6D2OXu+u2SZAiYq7TMB0rOyWeRWdTZmhzkyWNcxFiBEyIKcvj9j4EvGGADpMYAAAiJQl+2B433RKd5zKHW0oJKAZVI4qudQbIDQirFVcjZB1qkiTIZGBmZhZhKaXTwg0khzAIjwAzIgKKAOsWOVNSCGUnYeF+KjUyk8nkwe4do7CgOjgSSaIAgGKOEG5ALKCKyAAOEQAQYUCUGmbKE9SDj3m57M7WqU+574AFEUOjHEcdpnl/jCGqmVu4TUFICIDortXU3VJKiKiqESAiKGikYy37+QBZaJ0Xi1VaLNJ6QYvWEWq1MKizxlTm/XF42NPshARmodEgMiVwq7UgIyWOQACY54qI7gDmhrEr+1ksb/q0WawuN/35CjPVcAtPIPMw6ljKfvKhHscZjpYCCQMjhBAdqykRUeISquyFQ1Z927eUM5bqDm3bY4HDw1ZCippF9Emmeayg6byJhHnZU8NjLe6Qc06eqKIdZhyLcPaqwmwVEDxcq8U866CTZ15cbtbnV7xs8rLnPjWLJRBZMQSwSY/3u7Idxu1xvj/MwxyB66ZhTmgoFhYCqcmGjIvEXQILlJRTEwVg1HF7EAunQLTZSvTYPbnsW06LbiozOoF6vXmss09kTB5aCyosZHG+aM4XmqCYu1HXdGKCd1sfVItGeJBNOdbXm/ZsAZ0A4Vhm0+hT12PDHx5j0uNhZsSUkoUWrSqYN/3l1RkvW00UjA7gqjJ7fRxgX3Q3WViO7ABACIgAgESAGAAMTADEQMLb/V7ZVhebzbNLaHmwYoLStjo51rBBx9vd/HhksEwZANw9wkAEwNHR3EkSMydEd1ctCECC6n6ch+LhHct5f76+PHt5vXh6CZm4bwDRLDBIx7kep+lhNz0e7TDPd8Pt24/NPJ4vNsu+tXny8CQSEQAAhJwIBfbTfjseLSOvmuXTs+XT882z6+ZsRX0TbTIEs7BiYQGz7u+2u4+3tptgPx/vjrvdYQF50y0yJwoPciIxMwAQkdSIht4fdoeYraXzV9ebV9f5crV+ctGdr1GwgptZGNSpxGTzfpzuD/tPD/Onw/7+sDvet8bLnBdNCx4ArmaDTppw9ezi4ovr9vqcu2Y004Au9THA7uPD4ePD/dsbDa06HcqWO3768tX5q6vufI19M2l1h05aMeaR6FimT4dPP7x5uL3dLBdNlgAnwYfDbrQavVy9fvr0N1/0L85lvaBFTsuOcnIInwM8oPq0G8rjsHt/f//jh8d3n6YxEkQvgsgy1GN7tjh7erH56gltmll8Bghmomz7enh3O/zk4+NhrpUYeCVnL6+vv3pGq9YarAFejSbYvb29//7dfjuS1SzcPVmevbzsX17RWVczzBE66jqf2W5Ob24+/fCh7oq4R8LzL69+9be/zU9WsOBKMNdiFg3kZpb2p5v7725mrb00wrTf3lWy5bOLi18/W79+mi9WR1ZjRCYflfc+vH/cff/xcDio1YUsMDgMqzkACAUCIAYzAiEJURI2adb9k9++fPb7r+S83XMpBMHJ5/CDTZ8On755s59HGC17FQiMiLA4oX9GAwZCQEImh+poROSCh3ne4xh9071cXfzuy9UXV5uX183ZqpJRZgtXdUS06lRi3k3D/X662R3e3O+hjnd70CElZiYIBkQENzfOCTNNdXqsh7nDzReX6y+fPP36y/xk3V6uqG89nYopiECRDOaosNoe1p+u5/vh8aeb/TdvhzpMY1GVcxFJGcFPEA4AcuYKNkEdeObz7vrr50/+4vXZ62vctOlsQTk5QgY0M9cQ4tCwwcr2uPzweHh7/+mbn2+/eTePs1tkbBCBmdVtstJebK6/fvHkL1/nZ2fRy4xeDBtq9FHz03P9x5/mh4dZx0Pde+OXL86f/M3L5797HauMXa4BYSDGPAHvfPfTfbnf7adhPw7r5UJEgtSZ66TdVX/19asXf/XrzVfP6LL3ZbJE0KSIQIjsHOZgwGXdH619ddm+uEx/fnP86eZwP5RSWxKZbFoszi5eX23+4gVcdSXHjI4po6f6OEJPD4ftNIRHWS37xfPzZ3/16uI3L2KZJ7ZIqQ4zD5E2zWHY3x53TGWAeHV1ef1XX7Svr+qCtaWVpCjBE803x6bO85s3hUorxB0/+92Lq794AZdN7SIJtsThiHP4XdXtQ21ilmgxwm2OSgtuLpsXf/mKX53FWRvi0Agg10OhbZ3qhLeNZY7EhgAG4R4QGBCf4YoDIXiEgaGWqIuLzeb1ZXq1irMkuYOGDSVFLg+TJoJPXc3kx9JLFqFQd4Bwd4RAAvJAcIiISEko0ex1X/cPvqeL7uzr56//3V+c/+4FbFpcZstkoCFSSmFOEWi1InL7dNGMm3p3zFfr6OXuT28fvn1Xd/Xl+kKAa62LJqNqUBSfHvVQFnj56xdf/uvfrb96tnx5Xjv2lqERoCA8pY0cEW6GDs16LVcdDC6XXc0whg03+yhz8twGUXhihMAAr+Gf9ndzE3GVX/7rX7/+t7/vX13gWRdL1kQl3CGEEkJicw/U6rSR5rxJl6v180taZyUffrqbBjt65WJ92wSTC/UXq/Z61b44o+vF3GEIhodjC7kSInxofUVFscy6frJ88tevnv7t6+bZ2ZyjCiKJOOMRbFtxjsfj9uOnD3Odmr5BxnEelcpD2eEmXfz++a/+w19efP3Kz1pbECySISCj1kokzFlVXY094QbSebu42jTr7meEm+OPx+O4zp1gQielJcMZwxV7B8CoRFgx9/2qnuOPzXTjzKm0ka77/otzuG7jIjlb9RBf6t0w7YEvu+lN6DDnTHVJdN3B884XUHMoBFfxfXBtS3LsxA8Q6NBac5nhHOCCsKdgcOJAwikCovQ+sc1h94ftpmul5YIF1wyXCa5TnGVKWBjCAZvkAHVB2xgHn4XAAdEhDGpVZmRBYo5AUwPHQFcuyuYd64rskv2igY5CGALVyEP0UXwhEzq5GpI5EgIRB0IEVlf0AABGYOGplrxszeetDfikefmvv/7yP/z18vWVrZvSogtSx+Y+a5GeIhAopOWizhjcEzbt5dkrWTbNsi/meHsYwnLVngQMI0zVH+1YOly+vPrqf/ubJ3/9OlaNb7KiekZiUHdHQCL1SkQEQR7RAC86Dlx3xKse+2b754+HH27ud/uni/OYHSvklGafd9Nuj3O6WHz1v/7+9b///fLLS181c0JvZA5TBADUKFnE1QAIG4hA7JlXHD0867/KvXwL/xDvx+PRmkA22B8PvGmgl5JCG6Aea4eayRCrIanEBO11qyvfbvcXX25e/auvX/2b3/KLzbQk76SGAQAWSBUc7f7m5ttvvz2+v2nUWWgoB2R/HHa+oud/+eVv//e/3Xz9oi7FV8larmgeGKHBAWCzz4HBLUOAilGiIFz71eugaT/P/DhsR9HQUacp5lUOb6J2YC0bgE1uAbogWCc5b22YSgOxTOmiizUfUp1btPCMaDPBWRPrNlYCkGXRNFdLPG98QXPvYxOApJMnJ70vu3IctYSgE5hYkQKtWx+lDxVUBg9HRG9ibmKWKKCIgQycRDrPq6Zk5ca01ZrJGC2QELHFWbSyKztxAGFKCVnalN1dTQE0pQSOtVZFQ0bOCVuOBVmPtoDaYOWwANSwDmrjljESYmISRgB3hQA4YfWgU54hJiIQ5mM9fprufMOv/vbXr/79bze/fVZWYgtRBgUNqCe6TkDqXBARmFgoEBUgiK2RNT8FgPX9/QRUtzNXMvOiXr0eYRxzXb589vzffn3x+5f8ZOkrGVCpyYiupsCAJ2oRA5CAAIXQUd3UkS+bhVw8g2DAYbs3HQarMY7rfkE56TgW9uZyefXXXzz961+1r8/rWS7iltkFHAkMwBUdzSzcESOYSKiaUUOGlNrFmb94+jg82PvDD7dmkYlBaNZ6KON1Im8QMmpGy6QQTkrZjjTd1vstDs3zxbO/eP38b163ry78rKnJI6EbQA0GGI/Duz999/N/+vP253fpUDtp2yRtw5NNsmye/e7lq7/6zerLK7psvRVtqXAEEijUySBOyNIjDJ05pdQwZQokCZZ6+eRXz7dKH4Y3QoJTnadpClOHAKJIHCfqt8HoBZfNxDbHeLE4S2edd6wpaoZZQgkNAvsEyxbWTSwzgvOq5fPWlwJNWAPQsgJiMM6wm8e77eN+HFq3YC4YM0VJaMm0QUusiO4YxYLdhSqrsiaOEAgMk1AJJQcOE6jixhwRxs4S0BAkNEFDUHQEyHiKjMGAGIiODKSoiGgQBqEEhmACyqESKuAe4ebkwaBgio5gGormGRE8TrUyOCAgMQngNI/RUqUZ1vn6L199+b/85ea3L8qaSkeaolJoBbQgQzC3qDHNmJkSQeKK7oTYYkXvkqzq+fP9F7cGuz++tYO6hmiVHp2ovVg++/2rL//1b9Pzs2NjIVTAhdBUwYCRYjadZ2RyYRfEzEBsYSgsfWbJawOcbHfzsD2+eXzcNw5LwqEMD/PeVnD5q+ev/81fnP/2hZ41pQtLUsIijIKjqE/VzcONiIiZWwkm50CGAIZEzZPV099+aVsbH0bTQQkcACmIPdgd1UmDKRgcAACdK/QwpUoXzcXZsy/+9jdnv3qK6zwlwywWgUFUwB+Hhx8//Pj3//zxj9+3h9g0qwVKncaCiVvpF8unv3p+/vopbro5R8kwkxuAacSk026ohxnN+zb3fZs7wjB1QxEPQIpm2V68uILtdHvzSXLOo+rhcBgOR6ktQA50B/IEaGENUM/eYj0Gr1LetCWFs1eByl4ZzJQ5eJlxmb3jMIaF4DpFA5XD0AoyEAVoGByPx8N+X8epATbycDIEJVeGQmEERuAQQWAUihYIKETJHVxDzWyuE1AYhZEaoqKbIyE6KCbBTMiAp9oSwszA3cwaYRRW9YBomoaxHmAycgdXDsOoZIqgiIaACEhhFArqEICIiGYGJIh4UpEAASEITvVlGnEqbMvnmye/e7F4de2bZk5exJwYiBiJFGIouh/H/TGhS+a06vKqxQwFgDh5jjqXZpOuXz/Bh3J8d+e76kFG7FBok89+/fTq96/al+fzkmZ2zICUalWvzgrzcTw+bOtUc5bUtbJIuFpAiyQYhBoG4bTm9auLZ3/xRRymUe8ibIYKrtjg6tnm+e++uPzVc1g3Y/KZgRkoUhhggXo/DveP9TByQNs3/WaF65aRNIx7colwgIzN2eL82ZPj28fjoSqE5AQJmNG9OLiHapAieEAQhcS+HGJBZ19cvHj1avXVFWxybdGZ1CtVEiXfz7fffvzxv/zx/R+/T0N5vnl6nRc21IfDYRw158xtJ31u1l3NWDNoAmOwALMow/z+h/fDp0cbptVy8eTq8ur6rF/2SYjVbXYoQBp9boYmZSEhYJ/naT8M2+OyrCUaVXdxIgKGEJdFWlwvPKZm0+VV5l6oZcOCQkgRESAhHedlS4tcbJJNx+vOGzECJzxVbF4UptDdhJNlxxyA5IiMTEF4+jEChdMic8QAcEcnIj6lJnAMwHAmAsBT4RsBgBT4uRZwcIAT+sCTYoYeVioKM2ANCwQWdmQwTDlzK5QoBI38tA0COQmDAAtKYmYkBsQQAnADAAwCPOkEjgjuig0UqNjT+ZfXl189o022FNRxhEYgGXGF+f64e3u7ffNhur3vkvSrbnW1Ov/yWXu1ogYCXSMIkYSaZbu5OlusF3VhiCpIJrB8tnr2+y/XX17qAqbs0LJiREBYkPJws333px/2725zYErSrrrF87OL18/TWR8ZndyRgqBdNT77xa+elofjh92oZb+bjss+by4uzr9+cvXrZ+miObbgDUXCuVpDzJNPd8e7P7+9+e6n6W6bibtFe/3li/NXV5tX18aKCFmYAEBYVv3Z9fnjZn2Qe6u1FXGykw8gwgAckAECEQIcGSvV7mrVni/Onz3ls6ZkB0InDENQn+7n/Td3P/zHP737+++nd7tNXl50S64B4et+cdRjrXOLLSTCliKhCzqHgwcimHuxhw+fDm/vpofDHdJ4eVdfPH3+4nqxXlUzDJLCvvN4POJQeC5ixcxrHafDfr8o1+AejhgUGEAAArwQ7sUy5LNWVp2SOjOkAA5CQAJiwEx51aV1V2zCZeZl50JOiMgRBgFhMB2O8/4o1TMlqYYBwgiIQBhAAeAQDgAACICIIpmZK4JaKGiAIlJKTZwkHwAA+MyDEgKTuplZoCMFImI4eri7MLr7STILhGpWyYgBCBAREIMCAAIDkQBPnFJEBAESBgGe7oc7ICJ+dpQAgoMj+W6/rytsz8/PXj1przfWYhGfXYPZ1cFsvp9u/vzm4z989/DdW73brppmddYfrjdJ8YpTu2nMjQETSNRA53W/OD87Gx5iKo/m3p0t1s8vz19d46oZSYtgSCpjQWecw3f1w5/f/vP/8wf9dLxse4Gw7Fdfv+xImJ/EMhkCpiQNezFvob1cNlcr7TkambUse6I1L5+vF0/W1mJBVSbkREYx+fww3vzp7Zu//+7Tn36st9tl26ZGxnf39le/vuiWy03rxG4AEAQkIv16db9ePSaxaXKEkwfkZNIxQkcwgurOyMGxuDrrl0pE7XrpnThYYJRpbiDNh/rp27fv/vMP7/7+W7qfLvPmarGaDgMGgbuIeHEmatqUEkeYk2ugehiCGwUCEq3PzvBgflQb58e7PagPj/uUUpiTYeMMu6h3Y7k7cHXBIHCcxzIexmkcU+m8EUSGE0OepVl10ZAmw17SMs1oEJWIIjwi5IS/CblLzdlirBMtF9y1TgxIRICh4CSG27vd8fYRRmuD0I2QTtYJi/++8k5qFwABEDEDEABGhAYwEQCBAQZFQASe9ktEYJCHOwAKAxOQeyiYoRuGp5QC/eShOK3dCAMAVTUzdwdzAD7txQhCj1CDamRBQQTBGEKAjIgIQEgQEYEBAE7uCbBP7dkqnfXWUmXQBI4Ip6fXmB+PN9+8uflvP8LNYV1jmY0Ox8PdsO0Wl6uNWHCfDZEC4wg8Bpt01LjI4BbkhV1WWdbZUkA6pT9IKGigoz/8/PDhT++PP283A68XRG4P43YIPq5Xi/UyNStIBAlESKtKK7DAxfmqv1jpwZ3DOvEFwzr7AmsKJTdANE/QQKn1cb755v3j9zfwcVwOfAmdjvOw/7iXVJ487fWKepvFIzBGwAPEUWEeGT0iXD0SiGRE9JM0TqAACugEklj6NhonYu7bE2pgIkH32fY3Dz/+0/cP37wvd9OzvD7v2x6pHA8unasDgBtkZiIyrxGBHhTAyEZoEdLkdM6vfvvVcX1+vHoy70eYFc1341xuHmKuuh+keGwNjtpVSoGSJUskm9SGapNKNYmkgI6ODKltFmfL5dXqcd9Dx9By6hKkNFohQEAUJqRARszEXYKW87qjRYMtG6NDgCNWg2Lz47HsRpiKOIMDM4oIoCMigAPQ5wAPgIQBnw0RGCDEGRHJg/FkjMP/4RaIgGAQJ82YiPAkdEEkREpCxAbgp+1GREiE4b8klNNB+JcdFREQfNIZEE9WQGAkZjwR84GACBHgDg5u6JBIuiavusX5Oi2b2iAKm6uZUwWao2yn4ePj/HF3NvPLxWXjMY/17vF+WH+qT57zHLBIxNltiglhdL2djjePx4ftNBzaVYKEzWrRrPsxEwhXK1AiYUtA4/bx0w/vH368TQdaR+4nSUHTMeqH/eHnT2cvr5fnXTSplHk2I5ISFUOXl+vl5ebTu0djTymWT89l2VQKR+ecJHGtoXOh4mU779/el4/HdkqXuHgiZ8O43Q7T/NN298/vfa+4aiyzQ8Rkjeb5zXb/7kMMY8MkBIMVVQ2kQHKkQAaCQAci88CGCYVIQqicnC8WZlYP4/bu4eHjrY/aU7uQLiOXeeyaHiymUpZ93/d98XnYH3ScoSgUZZOEZIglVMFyK/3luu0XZ0+v9Vjn3XHeH+0wlt043e32wzxX81L0MIRJL62go8+RXebH4/RwWHxxrm5hHiRuhsgKntpmdXWWl70KQmCoMpMDhJuzowczSYPSY7NO3JNJKLkxqxt6sKGNtd4fhptHPsw5rVxkLmMZIMKBAQhP8g8iBoCFuukpTntVq8FNDvAARAwPZQh3xeAI8CBmQgZKwjkRkVlxBEFBhjBXDHd3RAQi5NNqN6uIRPTZH/nZn4joSCdvECfBhI7gAEUrVmyQ4fN2CmQMCGACEvdCCft1L10CAqcoYY7EIj5WHefpcU+jXXWbc5DGGq66kkQpxjePN3/3w/L5WSSerAImnRQmP7zbf/j+3YrahqTrunbZY+ZJa3AGAkl51iBznz0Gr49TmqCBtjVZUCOuUkBHG24ey/2hrc9mAxUwdHOlJKlr5lyhYWmTDmDomGhxtuYmV2ILn2dlThiB6lx8iT3ls2UX3WRlO6G7jIh386c//Oz31Ts51Lno7LMtcze8e3x8c9NFYiJkhHBDqCcVE4gCwAMQnIMJjAI1DNQcCBGYqrlBkOTlYr1cLncwAvjxeBROPTEFiEjXN9UroEfEcDh++PHD1W+/kPPONepcpUsNQ2AghLeISajnfN6mulhP1Y8THurwaRu/+uLm+3fbH2+HY7FdWSw2guoSFLNND8P8eNTj7GuOzEAeREDBWaAVagUaRmEgRCA0h6oRHuxogW6YsVl0nRbqJFKYgEEEAgZFNd3P08NBHPqmTSTFigg7g4afjISfQ3tYICPiydhMREm4SUiIjmhm1QogMiIFICIBMJyQPZxskn6qbU5YjdABqlWkE+Mf4H4K8EQE6ACnXAQO5L+kGYdwBEdXM/WK4eqQAtUDMSiRMBs4qHkAoVTTRNT0TW6T86nC+YwUARCcoKgfiw8GhYkxexZi1bL/tPtkbx7f3znBUItaYFBDGWbMNbo+GRck6JfLbt1LlwqFu0dggAEkL2Zz1cOMU/TY9JxY0ea6SN2MBpOW7aEME20WfMq5SQQTlAAGaaVZtj61sqB2tZC+hSxBjARIenJlgNrhYX942ONYk7dRlRAzN6uzbj8d7r+72X/cR+LBZ06SEXfFOpc8GQFKl1hSQykiOAkw/VLMUVA4uLsToTOAA0Y4OjAxE0O2aWrW3cWTa7iuh23VwQIbZGFis+ruJ8cpWcQUup0+fvfuapFyc26Myu4EzuGmqW3UAszdgFqWBdMq5SkWF8t6UGxS1y/dYPfNh+08CjgmJK9QhqL7ycaZrTuduKBwDmoltUmanNuGmRHRDcJ92B1QuFmIBBGRiHArNJC0CRI7QiAQMGKEWhlmKybBmQA8wNzdI8jd45cbBgD+skaJPrv8mQHc3U+FLAB9fjJ+LiMiggPilCkizIzdDSKdXuZ6MpMigAWoG5pXUGUFoBMrdfpAEewYiOgYQeRIiuERwETE4CevswMRELl5qUpCSbLPQYzEAOhm1S3A2QMEOTxqrWWc61zZgIAREoDrXMm55wUNNA3HECIgQXDzipWdEiVEdNeiAAKQEBkM3E6uISTwQA92QHVUp9M3QCDAxAIpOfNn0AyESExS3MEdwoupgSlYcaXIlJtgUgwNNziRC+HuZZjdLMx11hpVEBkpzIERaqBDYASbBDmqQZRhbJolKVQvuRFAp1O4AmBABYgIIPp8qS2YhPgXNRIdABRDMuV130Xz5Fcv4FH37x6mcZrD1tKWqmh2OhoAJsjTdNy9uSvgkeWKWZ6sY4lNS5bIwus4O2MEEuPnz5HEs/NKaBNP1t3Z1WXb9j853f/8UcijoVzNyqi6n2KobOiAGoCIzs4tUZ/SouEmB5MHgLvPdX/30C0WlDtmQnQUlC6lVUudQCJDQAAEwog6e9kXPcxlGNOIyyQnG72f7JP/843wf3owImqtjIBiRJxSikAHiPhM4Pz3ouJE7xCRIAlDIAYEAhISoUNEQARwEOOpuvq8nf7Hlwd+pqFOjyPiSS9iRgiPwNO+1VrNjISIqGtbEfksUwCIiBG6w4nRPfUq5ZylcZ6Te1hgBLA0LaCCalFGTrkx9wKuc1UHIppqUTfCdBgPZ/NczRwoEJEJNBCCmHKTROR0xlzDw4S4VgMQEck5M7OGm4UZAdKpOYaZT8n2lHWL1mKKqpolkN2dgE8JNufctu3EgysAMjJCECBKapBAAdU9mCHAwxebM6xgqESEnEqZi5cFMwZEOHgAIPi/tGwImDMKAgBgAFZ3QHdiIKeMq+dn9lju3nway12ZQwkoQgCY2WoJwq7p6qyPj7vH8Sh9qxaLr67y1cpXTVovpKFwDaZIyQEiwsAc3BIkEGRkyVz94lfPYrBxmoQABSks/DDXx6E+Du2gskiVDJgCwRNSK7JqsMuKgQDo4LPWYZYQ78qJuMWE7aaLBrlrIRMmhsBQoII21Hl7vHt/0+wqREccjCRIfoLnRI6nfhg4BdV/WZMR9hnnIBHwiaw5baEgPEnDBJ8hUyml1mrh+JkujUAH4rnOzGgQEcjMIhIAAjRZcXfwz6X254r51ImGZIAUgAAn9AUAYI4nmiacEVgkSRKk0xb6H7KaA1CEmSMiNk0jOZOwBSiEE5ZqkqSi78poHGnVd+seSXbHQzhRI2GBFgXcEIIl4lTRf+6/IQIXJscgd4CTOAMEcFL0HYCwmqIbMKEkJAmMOAFJC6aEIkQEHkzUZrFa0CMxK7MjBiIiIUHTtgeAEoaZhFKAg5AwuxeVoDY3OeXElkjDE3ErbQwVDgNbEON8PGiuEogen2OVBwFgkIAQgNdKEBTgaCQYiA7IgA5gGburFf8arz/cfhymw4+PtN0+XWyinGILns65UEqR61y3P97VamfbfffkXM777skmrTtqBNscHTqhkztHUABhhVN3lPiC2iebi4Mt3r+TiCAA8hAgHUrdTjFWtAaBADwIjVwWqY0lL7IxcgB4aLH5OIkTnAcTOSIQUiucGkxijADBRBBgVW2s436Moo2kJXcY4FVdDDGd1vT/Pyf8dwr1FNmQkMI/S1gQv/CqERAIn9ES0WcYyhCEcUI4ENLkE7F54v0DITzsJDKY/w+kEBERAuFJSMY4GQdQghlTIjUVOj2TIwI8EMOsTsPIc2dm4Kfi2yKQiNExCIjIMIYy6zwsUfKii6nsjtvdvNNsz798dfbisj9bB9PieETgBBKjHx+2+08P81EtgjkJnupLcPfqYR6ABBgmIa240LGOOE1Nt0pN8qApioT7iXggYUFHdEc3Tw5afdwN0+Fo46wjozmHJyAjNAiAMDAkRHJjrKFzmVqnxOxCk03746NmfPr86umXX3aXK2U8lkko+Vj5COPtbvj0sH+4r6af6exfsiud3CgYYM6GGVnHejwcA6Fb95LZMJAxpXRaeumyPfvqsj4OH+8O+8NwaUudS5sSMxvU/X6YogbRYrEoU7l792koc3P7wOu2vz7jZddfrLhv86qVZYN9lo68IWAEQgvnRLxsxam5qN3lmbg7UIADOpRDmfZjnaoUxw6JUdExUV72kVLTt8Ac6hBQpzrsBnHwor+0XAIk5gyck0l4KIWQQ511PE7j/tDnphGiCPBTqRdgTgH/Uh//C4MZYKeo5O4n0M/MgR5hEeGnDQpxktsMABEAoVv0khMSBZ7slhHhwdBIgs+90fHfexp/edPP4Q/RP9fZlogrqII6uqOrW3ETcKCYtSLGCVGEmhtwkCCFWhQ/AcPMCEyK7BqBbuTUCLZigrP5aEVQ92W/t8P50+uX/+7Xq1dXsEiWubWQkOzJH8rNt+/28+TzqBZlqPNYo7q7G4B5VMSTeU76tjvf5M3yeHu0aWyJg9LjuIdOFutFu14qY/HikM2AiSMwzHUow+OhHiaeqg+zHwcbxphnaBMmDojAIASnaDZ9d7YY+OFYRuEEVodhN8a4vrq6/MtXl795yWe9Nszz1EDDleG23n77bnc4bt9PhMwin2k4D3CjEEaSCFbHClFif3P34d17pHj68sXmak1y4lM1Ap0CWjh7dZ4msPtj1YdhO6VqLbELqNmxjNHQ8mrTXq4s4UQBnSCxFdp92td395STZM6Lpr1Y9E/X3bOz5mpB3GCScFAPBHfwkoLXrQCAOwBGKTMdqBwmOxYpDoaYGQFZSChTw5wzGoaBm3l1HWulotPsc/WWQSB1WRKjAIF49UBX9TqX+TCNjyNVKodxNGv6Zdu2ITF/bmX5HJJPbA/AKS0TIn4eDuAA/EvlcIrZiID/s4ZAMJXpMB23036YLQGJhjiAwX4+ChEAhEFiCWELMwgRQSZEjgiIz2ogRDiEhhe3OepORy6TGE2GKbDU0d0bSYjoGgTYUUMkddbD7rjej7l2YBwaCircFtLUpfa8Xz7d1NtxdzvCUaPMNWm6WC5eXyx+c9V/cakdK1MmiSnyzAUP9pZqgIV70f32MO4PZZjJUkNJyU9VCzQInaTztrtaTvfb6XC8KY+fxmJJ16vrs1fPNi+eyKKdxCExeJgZVMMZ9DiXwwDFMzCXKLvRDpMdJ+gQElDyk4OWWsnrvr3e5Ovd/Gl/c9i2NRSr9379xdnV33zRvrwoGbBLXDJFkwac5sOIPkdoRJMEBSxOwctOl5jxhJHYq+pueHx7+/HPP4jIWeq4b2nROgFyQBKnAPX2cpkm/HS9qh/302HmxJBYa9Ewymn9ZHP165fLJxfpYmEtudDoWj2O0/hwc//Dt997LSHQnXVnX10/La+v0tNGMMyciDFREsjAKVVwQWQGr6UKYJPy9uah//S4/Oq6lghxzGTgQYGZQKCqSkBolONUtmM1sqGSIzI7OwUFUQQUs5zSPNZGJNSGh70dih10I4uWEgAZ6KlxXogZUH+xB52wEBh4dTdLnBUYAUSk2KSqJ6rKvBhkhwgSRPKqEYYS0eHcGK5wTNhT8smxGoVM5gQcBcZ5LkpNS5wo5DPHGigWEEiBGADVlQRDYM4Qq+yzFUAxsQBv2iypmtlshBHFyziFQBtp2B73j9uz4xLbxDkzQqDnNrtpc9Fffv3MDnqET9tD6VYcQsuXZ1f/6jU/X+pFqhmUKBxz4uJWyYfpWMpkVbuWfJrKfuIKVkNHbdou0E093HMvV6+flO1hPGwJFMdK3izOuu7V2ebrl3S+1MyQZTZFEnBvKNfjvmwHGBRGtWqUZX4YH98/9M8v2QHU2kacuFJAQlql818/06L7JZVPUU2Xq3U+a776X3+fX63qGdcEIyovch3QC3gjg9X7xwdKElgCgJkjggGDIMKKORADoJXQg9tD+fTP71fLLq6e45WlJqp4NQ8IAMhdxmBdlM3r6+OHx3osw93UBndNd9wepU/Pvnxx/ZsvNl89KQv2ZVaCJSEg7x73eb3aTePjze00jcfDaB/u+6eba31K6iBMgB6GKGqVKDBCwDwouradq2IJdmKFepy9Y8Yc8Nk44AgVFNAF2KrbUMt+0mA/FpsKd0wNMqMTza7IfKpHsQQF6XH2Q1lyt0xtoxBa1YqTYUhYgEM4okMEBhKcRmIAEMqJZkGMUkqgC1K4ohvgyfGGcMLjEEgBHS6ebC5/86KdaSldF4LVxBAtiCRjM+/K4/vbeXdACGA6WSoYGP2E0wIE3SJLA41HK+myv/ztS3mqy8hLzClwnmchEk8pmEbffrg/3D0A2jzO5dPj+nZ3NT9HRZst92lSQwTuZXG1OH99hSUO69b3c5nm5aZ98qvnF3/xKj1baU/KboioqAaIsd3vHx4exsORA7AaVzjePN7//PH8/KvVuj+WCKg5NcAZwJrL7snvXorw4/lNHKdEuL7eXPzq+uxXz2GZrZVIgAHg2HLGsew/3N9+/07vhzYkB+puniTqw+D7KdeFtTRbAFhilkW2SOevrxF8ebko949UtVs2V68uN799HlddyVAYggWCdHIi2h8Ot7e30zQ1AG1uZhtcAzwgggLwF99kOGGQTsX3lfcx73bv25/6pnmav2RCzhAEgAgemKm5WGxeP7FdeXeYffbtYY7IebGIFmZXyJQ2bazIlimSOIU6NN36qmsONq2uzuo8QYq0TpdPrnPTEFEEEBIagWMCKcfD8LAVRjLXlFIn7TDOtj2U/ajDJLFk5OIW7AYIgBaRiIgoivpYY6iGkx9nrs4W6g58snmGELk6O3lRP5Txfii7sZmYAQRQHeOzjhJePQIB5DO6Of0CxqgEgH5C826mgICIXtVU0Rw9CJAAIhwFsaPuYnn1u+ebJ+e5Ys+tAJIFG7gaQ+qxnz8N8I98/71N48QBHkiApyRMbgkJ4CQbG7eSz7vLr59dv3jCE6TiC8nkMM9VghpsOkuwq/zHH/VPOm53bF630/7d/fTp0PSJCSiTkjsGivMqr16epZTWz85gMtPSny/XV5v2elV6nqAqBAVkFDejotPhOO4P0+GwwbygtHt4rLey/+lTd7Xuu8SOTQOUwchNgle8eX3VrJvzV1dYLSfObVo/21hLo1Qk1AhwSAa2n+3T8f679+//+MP44f46FktpCP3wcHj8+WbxdH2xSomXJQJbScwWlRpZPVn2rfjrpzoM5Ja71G766OWQzJNQFnTQyXUY54d68+bd7tNd1JIhtUjTXNmCFVA93DAQ3A0Jg9GjDIVmWEJbj8e7b96u++Z8vUp43m5az1gAFAojyVJWLzY8w8OH+2Gw+4fdfjherJfg+un+bnO43tgTx+wMni1EgtBzcLt43nylx1nnCQi45+aiy5slNGiIEQjVcdYYve6n+ngUIgpjLZZFplqn7TzdHX1XO6NSDdmBT5orhYcrQI06zmU/xaG4s+0mnIyquSImPGFuCHCLKO6jHe8Px4ddTOrFLUqggDsQuLtbnCyhGIBBp4VJARinYGCIwRgBnycMgVEEgiNEkBN6nIw+QYGJ2stFv1zQ80hOAsgOpyDETjZHqpm7bvG429/vZq0nSwU6nlLB6U0/KxboRiiLfPbyssWGikG1TBzmrUXCxDOnCeKhNI8P/d3ZeDgSYAxl+9P93fN3zzZd067nyXKXalhAgERatzlnv15DDUTETNzKlCJQQZiBxJAL+mi7m/ubH99Oj8dksGzygpuxBOzr/ue79nLdni2yLDBxnRUZpWFnCvBEi2a9IA90c3RbZ2Xj3AYT1JqCeII04Ye395/+9GZ4d98rr1KTDFxVIIaPu9tv3zVn/fmiWaRmqmHgiIQSvGBJHVq45ghLWUB49HKya1ULqE5zNC4///DDx+9/oqlmFK5WBiXARIwBBIxBfJL/AQkRgRfcz5CbyDRPPs3D24fju/vLzZK61lusYAjh6CYuy3b5FJ9+/eXtPo63+3K/r+DsMQ/j/nFfjqNsEmNUQEgIgsBsYm1eki5dKwBwougoEgJzYvY5MIjG2H94uP/x7fHjg5zsNu4olNYtR4ywr7Gd8aDRQLNJDoBAgGjmZGhzzI/TvB3rbjal+jjEodB5x0YMLAhB4sUERTxQcXo4+LGQYyctKqqqgQfGSYqC0yr04AAJ1gAMIEQAZKQm56N7rTURO4SrMnObcmCqgApogIAWQBVNFllaBIWoZojgAB4eIdL6scRIfqDasTeETaIAA/2FxPjMzJ54KmQ2M2qEE+ps0mFaJvMIQwZCTDaEosMYpcUqAEw2apge3t1//KefFufr875N2ESoZ3ABRIwcQMINm32WQBTDIYCAgcjAR7ChTjf7n//hu5/+/pv8qFd5seaWS6zSctxOj8Nbb2l5sW4SErXRAHQJ5LPESQ1JK2FeqyFLFQBhEq6z+mQtJjzY7Tfv3v/h2/2Pn5YmT/rNGvJpboeVcbg93P/5fbdcLFarxMRLjk6xyRYaAgYnt7lEYGVAdApmSWFgc+EKvje4L5/+9NP22zfntWtAwEopE68kkIEYLMKADNlJgtgJzVrKYkJTZGXhtt5PN39+L6v+rM2GQRkgESJYKBFS31x9+QzvZr0/jiDswB4w2+P7u7ff/HiRXi77a0QGAqAEFCaADNEgmCAAMDu6B5CilfDJu4r2MH/64w/v//Bt+fggRWvTNMzEzMJ4qLPuiu1KrmwFyMTJFQiBzU2cuCgcK+41VchT6HbWQ1lVVCM3CERzt+odJFbXIXQ74xhZJQGDI5zSBbtBNQV2yiHmhEEQyMERQB4Q2EpKQGhBFsyn8T6QkDImd3LFLAAIwRiACicbEQY7IkEAnoZQuJeokJkwSWskYaEeCgCNJAERoAYEIYUjGhJGGLpHw5mZqs0ESMK1Vk6EwB6IFVIj0TogVtMIRMVVXg46PH7/8YcmBdHmN6+aaCXIAA3dwh0ghJAwCE9dyGiGpzedgfc6vt+/+fs///x3f7KH4aI5X3uDxRCjoRwe+/3w8Z9+4kQv43dn8oLP20pugQpIyCAEECTctguAcIiwcHOu0Fj27bD9/vYP//d/fPznt/KgZ3m9ljamih6EKCA0lv2bh/fph5zzWX0hLzbN1dLBZkSAYKRfBDAGAAQQFAyyqS4swajv/vjT2//63cd/+H5+93jeXzfUpLZ7OBwxuM5zAk7BaBKVgBgB2BkLaOXOuY2sZYDw/Xhf65yX7fn1VSM5ITliEM9qaCEQy67Fqwt/ttsf9fjpdh4mmP3D928ehocvbXotnJ6csYMDiYSHIfPJ7U7AIkLuEYHKMLvvdbzbHb+7ffOf/3j3zz+lGmLkJWqWBgC0mM513o7Dpx0cNS3aGEMcHYyJkqFMEEfX+6HeDwtLeSZ9OOr9kcZInVeM1DBC0GwcAvt6/HBX7vZwrDQ7E3KA4GmSImlAWIi61OCKUCECFDUccHIYKw5F9wMVbUlaFCD2cCwRY8GxkYxmIBLAEOhiQVER6dQu/FkZAKeIqE4maApavcyuM2ihgMREs9EQPEEaYzanFIJQvGYiZiXnHIzgMVc0AyOLEACqwMZeDObZS41qiWXTdl3gp+3d/T/8FEWf7+ar37+isyyrbBkKQWQMRmeqtZ7GG5IjVcBq/jiV2/ntf/nzu7/7c3mze57PzmUhs510MuG0bhoWvLn/ePN3fxZwrKV9fZ2uV5Hws0R98lIAksgpzVIAWvDs0+327T/8+PDHd9M3N/jhsIBugczoZl61Ho/H1WYpOXE5zG+27+Cb4/5w9tvni6+eyNWSE0iTgkUxAl0h3CGxmBYqHjuzSQ/v73/6f//hp7/7Jh1iESwWFJGaxIC1VC8ao8K+YJ6pJMnhAKEBDxUeRtwXPM6+nzEqozoNw0+385cPfUCa28JuBByIWuFIMsEypUGS5TQ71KqJ2fZ10IeP6YcW0vnXL+VqCZsmLRtpyNTcNQKRJEDBI6rXyXmG+Wb79g/fDT/fHL9/gw/7RV4IZ1ZTDtaCQcHOMPj+436+mXKXEzl3DGGESDXSzOVuLh/304fHdIygWvBY70fca0qQKnCOzAKVqVS9LcPP9/gww2HG2ZoeWyZQ1aIFZpdKNTWV+WCxtxQBfBoDBDiYbYs+HKf7bcylpUVCskBV9H3hY+DeCSPEkT3AgYkDAQxPZhgwh4Bwc+eAPnV1mPUwT7f7uj3gVNkhEdtc/Fh8N+FjIYAk1QWMMUOkJtc61+JZBBFVCxO5OwAFpBjBhtDbsT4O9TCiOQaRQQN5Q4uHx8PtP/5cB4NB22fr5fONrNvUoCfWpIDUCLsamEcFmH26P9z98PH40/3HP/44vntczbzOjR3mJCwpaTEMrLM2nBbQPnx8/Ok//3m/PZz95vnF1y95s2jP19K3IaQQCginbim1KDZuj+Pt7sOf37z5+2/wbqJtObP26fJ8yY3NVbU0TWNm4bharInofn+//fbjYb8/PuzO7vdnr69x2eSzVepbJjRGIXQPMvWp2EEP7x92b+5uvnvz/p++T3u9bjZn50ueA8JrnVW1gvXY7N8/dqulDKW2HE0qEVDJHub9z/eP33+sd0OjdLa54hyP5fHxu083Z99fDxU3bQFXAhJmSDhAWxIdih6PPk3gxkibZunSb8ftw7fvyzA+u90tX13mq1X/5ExWrbQy22lyDWmgFpsPU9nP091x+PD45//0B96VRcHzxfkytyJCOefQAMSq5k7zbsbb6e5PNxdItiLrTgPUyMYouzJ92Nef7uFh5NmtFtUY3zxOFw/1gXdUoE0pJVDwg9pj2f7jzx/+8P3S8rJpM6CbCgMgZpCq5oNuf/p0+eQqjsMR6wgWQsKZCx7ePLz/h+9jP3bACQPNojiEP755ePzuJh1mWDczqjKOPk9aOUmE/eI/tc82OgD0mHLvo+u2HN5tdz98Kg/H1ij3zVx0ujvef/c+ImCdjuwTqhIFU4CB+ali/uzpIHQN4czGMNQ28vBhf/PHn+AwtUSL3GiZIWzTrDH4OE7xfvj+8b+1T1dXr58tn1/iglUAupwWnRNp9TpWO8z1cTh8fLz/4f3h3T3u9SmvV8uGinGQayBzt0geOlVT9fXijJt2fzjsvr2ftzrfjN3Fpr/eNKtFZKrkQcwsrjEPetweysNhutvv39y1t/PCmkRNzsHFIBmdXJ9aFovFNE3H/YEEL7uzoQ7DzXSst7Cz3fef+utNe7n2xNhy07YGUaeZHOfdYEc9fto/vr19fHtH9+WyXZ/TMiuEW0RoYN8uGonxdnp4+HF7v7t4eT3YVMAlt+VYuuhuvv/w8NMtj3G+Pms4m8+NpU9v7x7XH3U7y7Iba5nNOQkixuRSYXwY5vu97+cstDrbMJG7Plme7+phvjl+HL47fLhdXm0O50tcNdQ3xY2E29xZtf3D4/FhwCl2N/c0xMWU66RJY5nbhgT/6/P/kzkRfk6mQ5l2dfAOz764Xn95QWd5plnRVL2x3EH78OPH99+8EcWWBD2q1/ZscfbldXO1HEkrmeQ2I9tYdTvdfvd+vNk+WVz0mH1UAQSA6lVSco6jzbSk1YsLWLImmNCRiEigeLkfpps9HWprLBaJBdC34/7o89PffLF4cg6dHMo0hzmHEwIHEbqHq8KppTLotBm0KDvBDL7TupvhWBrkLiUiNyjRUjpv28u1JT7qvBtHEhIRJgpTV61mEUGSiIghUVDMJsZ1Ow/3+zW32aAh8rmCR9M0GvUwHaOhx3LgdZfPF4urVV730IAJuRCy1GLzMOlu8mOJQ/F9gWNdStdhIyAcp1GkVVWJoOubWouGkpw0Ca2kkbGiybpZnG/SuoMsRm7hEFSKeY16nO0ww1hpCK7QBkHxTrIAhp2G0ZOB1fp55jERNYk1dNK5knvH0HLNYRmpy91m0S56d9Wxeqk2aUxeDkWPlSagEuu8ECQOB3AAcIhq4QTSpd10iAabRRrqaBBJGp29xSamqIeSMS37nhmcVH0e6qDslBMkNIcQ4iRmVqbakKACaeSQBlkAXS3UKJFiLV4VTTmwQVq0ssgzRQBQEpHss03HQYcCc2DBlpIEooIANiQIgP/pyf8hnHNu53kWSoFg4ZONmgI6nLhy38w6oRMZsTGOVo8TGTZNt1mtgWI/HSoYNowtp0bU3d3rOFGFepyz0cViQ4Y2F0ax8KKamgwJ98fDbEUWCVJQw4ph7oioxcQQZ+8hZ2BXa3Nq+maYxof9Li/7frPiJh/KdCyDk3NKLIiIXquq4snMSAQA7q5uQgIKWJGVMhCYo1nTskYpoY6AbZY2T1oP40DCbW5SYoxwNQcDAAeyAHcnIA72Obx6drpcbbTU0yVBj9Ok+1lLjapoFU0lIlHqM2Uy8qJOwhFoxaMaliDDBlNPDVigBzgKkrvP41RrJaLcNsSACEFooe5eQwN9shkSYSbKSdr8LzNMj/sDAEowO5Bjck4g6JGAhYgAfzlsnrWO45ibxlQRMTMRkUFMNivBbjpEYswUgg7m6CmlLuXt/UNmoaCYI1O7aHp2ysi1FHSjJIhRTc0MmJDoNG0X+DQ8k0REiyVswiDR547900S2oDCvRYuZFVU8/RcLptOfzCzISSRjYiQ011K9arvogQIIplrmOlXQECBmZ/STkd4x3ME8imuxVbvIuU0s7hAnXQ9RIsK8WmRkqqWevJY9t4o6TwW0oqKoI6BPc53trNvkZWcFzMyKRQRW5AAfa8VRmaSRUidX63PfS0uEoOoVGOlk4W1zDiKtxih95vE4EiG2GFYAAomwVMZmkbsGhZECNCKsmBBfrM63+900IbcZ0WEuZgq5SpMAALSgGVkw82eYHJGTAAUGkXErWYjDVKeSkJvUt6HDPJXDXAfFiB445pBpJmYOZGaSbOGqNpbZ1FGkaZZMCIxCiOag9rndLNCqA7oQJ2bMZAizT6Vq3U2nNqYENNcBAgmRIpEHOjIq5YwB6GRmgYAAKSVmTimp6mcPuPqpEbWhhAzoYdVLqQpV00xEJ3foMqU6a5u5bxdQzT93rKMIu1r5ZULHPM/V7TQDIaUUEVoU0ZGp4TYzYeJCFhFkZFCrqR/LDKWL1GFGIA/LKDkE3E0rIwYyBQQCeCCiEFc3dEmBApKbfJorWahgCCVCxFprKXNqMiNCgKCklILV5TRpOAIBGuKezf3UQc5OoXbq7iASd8AACEzAkntErKC11mkoEKFmCaVr20VujWy2smpXQAgAxc3txPTh/wemajs7Gm3p4wAAAABJRU5ErkJggg==\n",
"text/plain": [
"<PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=258x34 at 0x7F8821F4C4A8>"
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"img"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"scrolled": false
},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAEQAAAAfCAIAAADROPzRAAANwUlEQVR4nGVYS28bVRs+c5+xZ+zxJbHjuk2TNCU0TUOgSRs2CFWw5Q+w41cgseGPwII9bCoh1EJVKlBEQ0shF8VJncgkjuPYzsRjz8yZy5lv8cCh0udFFM3MOee9PO/zPucVvvjiC0IIIUSWZUmSBEFIkiRJElEUkySJ41gQBEmS8IoQIkkSY4wxRggRBAEP0zR1XdeyLE3ToiiKogjPkyQxDCNNUzxUFMUwDMaY7/uiKMZxnKYpNk/TNEmSNE1lWWaMGYYhCMJgMIjj2DRNRVEkSUqSJAzDJEkURVFVFdsqikIpFUVR13VZ0zQYHYYhIUQURVEUBUHI5XKMsTiO4Rv+YYxJksS/4VEQRZGfF8cxHsLV8Xisqiq+EQQhiiJKqed5+Xxe13U8ZIzBDUmScFAQBIIgmKZp27YgCOfn52ma8ojHcTwajURRRGQlSYqiaDgcyjgbm6ZpKooiFpyensqyjP+RENiHpAmCEMdxFEVIHdxL0xS76bqOYGNDhBDB0jQNJrbbbVVVEWDGmCAIiqLgdMaYoiiEkPF43Ol0ZFkOw1CW5SiKXNdtt9vNZtN13UqlMjMzk8vlSqVSPp+nlMpRFMFoDrAwDBEVJAEASJIERiOKwB5WAW+wHm/xWRAEQRBIkgR0EUIopZRSx3EYY6VSCXFFRAghQIdlWa7ryrJcKBQ0TfN9X1GUfD5/fn7+9OnTjY0NQoiiKJqmHRwcPHv27OrVq7du3VpaWrJtW0Y8CCHsjR9ymiQJgATrYbcsy0ACdwD1oGkaMgywATaWZQVBQCmN4xgZVlU1k8moqjoYDHBEmqa6rmcymTRNgyB48uRJu90Ow7BSqczOzs7MzCiK0ul0vv766yAIVFVNkgQnEkIymczff/99eXmZpul7770n42wkWpZlXddhtO/7cAn1gBTBVU4JyIyiKHEc84LmIZBlWdO0XC7neV4QBFgYRVEQBISQMAx93z8/P+92u57nRVE0GAx6vd54PMbyvb29Fy9erK+vZ7PZvb095JMXAgLheZ4gCMPhcGtrK5PJCF9++SWMANgYY+Px2Pd927ZxPJAN93zfj+NY0zRN01A2jDHsDnyiYIDyMAxxNmwF7RwdHe3u7h4fH6Pw8A0hxDTNNE3hia7ruq47jkMIMQwjn8/3+32kAmyJomKMJUmiaRqllBAyMTEhAxjAfRAE3W738PCw3W47jqNpWrFYvHbt2vXr14vFIiKdzWYlSfJ9v9Vq7ezsHB4eDodD4FBV1Xq9fvv27fn5eZRckiQ///zz69evW60WMMZRGsexqqpoAISQIAiAdsMwfN8PgkDTNGTP931CSD6fv7y8pJRWq9UHDx5cv379xYsX33//fRiGgMnl5aUMdB4fH+/t7e3s7DQaDYQKHjebzVevXhUKhY8++mhlZcXzPMuyjo+Pt7a2fvrpJ7AkjNY0LQiCvb29ZrO5urp69+7dYrEIg7rdLpATRZGu62gyoiiGYVgoFBzHAS4AP0ppoVCIomg0GiFjo9EIKECRJEkyMTGhadq9e/eePXs2Go2SJJEkKZvNykmSuK57eHj4+++/I366riOnSFcURd1u99GjR4Ig3L59u9Vq/fbbb/AZ1aUoSpIklFK0rCiKDg4OJEn64IMPYCVvEUmSDIdDwB0Jubi4wEGAaKFQ+OSTTwRB6HQ6jx8/jqIIaUnTFFjyPI9S+urVq1qtVq1W5+bmKKWqqmazWcMw5Gw2u7Oz8+OPP/Z6PU6yQRDgMEEQEMtut7uxsVGtVtvtdqPRgBHwGXbAK0qpoijdbjeO4/X1dV3XC4UCjwu8IoSUSqVcLuc4DvbRdR1pWV5evnPnThzHhUJhY2PDcRzeElCrKOMnT55omjY7O0spnZmZmZiYmJ6erlQq8uHh4a+//gpPMpmM53mEEFSI53mj0SiOY9u2HcdptVrb29vcE13XKaX5fP7mzZuSJPV6vXa7DdADJGdnZ9PT09VqFQnk2b5x48bi4mKtVjs+Pn78+DFQh4paWlrq9/uyLBuGkclkhsMhxAtXFePxOJfLhWFIKd3d3RVFsdlsZrPZxcXFt99+W3748OHZ2Rmi1e/3cdjKyookSScnJ0+fPvV9HyQThiFInXdPSZKWl5fX1tYEQfA876uvvgJBIQ+NRuPatWv5fJ4nBEQ/Pz+/vLwMhsArpCVN00wmg4LWdb1cLne7XVAFpCBHmqqqxWLRdV3P81CrL1++3NrakjudjiiKqqq6rksIqVQqDx48mJmZYYzV6/V+v//8+XMe1G63Ox6PuYJSVXVubq5er3ueV6/Xs9lsHMc4UhTFXq+naRpEDSgVajWXy6G1i6KInWErIaTf79dqNZRZuVx+861pmjMzM0dHR3Ec+77f6XR4saG5RVEkQ7Nwvq/X65ZloQdD8yJaWOA4ThiGYBhJkorFoq7rFxcXnucdHR2hKvA3DEOAR1EUNFZKKWSEKIooa03TVFVljEH2iqJ4cnJSq9UgNIvFIhcc6Dyrq6srKyuNRmNnZ8fzPOyPVi6KoqZpMhf8hmF4ngfapZQGQRCG4f7+PqoIwSaEzM/P37p1KwgCz/MGg0Gz2dzc3PR9/+LiwnVdKD14Dojbts2vDzgIkiybzUJicf5gjHW7XTjMGANzqKqKnh6GoWEYpVIpk8ksLS0xxhzHaTabL1++xM5RFMm8pXieh9bz7bffcpRDsCDehmFQStfX1+/evdtoNDY3N3u93vb2Npaj2/BYEkIcxxFFcTQa1Wq1g4MD8u/9Ynt7e3FxcTQa6bpeq9UODw8RYNBGEATlcpkQMh6P0eYJIZIkUUq/+eYbtLVqtfrZZ58RQmq12pUrVx4+fAjikQE7ZJ93LkmSVFX1fR+FAZhx1bS5ufnLL7+0222chGzAZ7ACdsMqtBpOAFxNI1f5fB6ecyaAEEEX5nmDskQ1gl1OT0+r1Wq9XhcEAULb9335/yOKH+9WKFy8kiRpd3f39PT0+PgYwSD/yu03F6KN+r6PawWkHXYTRZFSiu4hCEK1Wo3jWFGUKIpEUXRdV1EUaNxMJsNbBWcgGOn7/v7+PuoKFx7s9p8z3G/DMBRFgd4GOyG6uJpubW3BT3iCX6VSQTvrdrtwDM7AAV3X33QGdiOT1WoVmyO9o9FI07TxeIxVlmVxKcTRjh02NjY2Nzd1XT87O4PliqLIbwaVEHLz5s3p6ekwDFVVrVQqnU5nc3NzOBwiuuhWiBBsnZycXFhYmJmZ0TSt3W4/ffoUNIAK0TQtjuN8Ps9DhtQxxqBubNvG5QLmImnokpIk2bbN3TBNE80DBuM6xH//XEmAHyTaMIy5ubm1tbU4juM4rtfrf/7558uXL7Ee92QeJ7i0trZ2586dXC6n6/p4PObzDUII+IoxZpomLhdgOTS7fD4fhiEIDdvC216vNzk5iWIrlUrc3Dd35k9Q21CGvu+L/N4ryzKlFIMC27ahGh3HgTTEcOPNSMDcqakpy7LCMGy1Wo1GAx9jQzwcjUacGJAuSikoB5yr6zrygL9HR0eAE2MMqgoOYGcYhgkBnuN+hlf/DURg8f7+/o0bN6ampmRZvry87Ha7CBufTuDihaJyXdd13fF4jIVHR0fwEMMkz/P++OMPNCWUBK9gfi3n9MDnWKenpwACoMWFGSHENM21tTXI08PDQ1yQdV1fWFgolUrLy8syYJ0kia7rSZJ0Op0ffvjh/v37uVyu0Wi8evWK166u6+jrIBw4/+jRo7/++itN052dHUIIeJyjcWNj4/nz55VKBcIPtGsYxnA4LBaL5XL5+PjYtu2zszNwWhiGyACahKIopmniLELIxMREoVBYWVlJ03Q4HEJSlMtlLizk+fn5169fu66bpmkulxuNRu12+7vvvlNVFbUOfR5FkaqqUBO4FeJK2O/3HceRJKlQKIzHY97O+ZAEvZmHNgzDOI57vV6lUrEsK5/Pl8vl/f19ZCOXy0GSgeImJibu378/GAwsy4qiqFKp1Go1XdcZY7ZtY4DIWZ4QIr///vuKomxublJKYXEYhtBLsIDT3cLCwocffhgEwfb2NqzEx3xABSJRVZUP+8i/dJzJZAaDAehI1/WJiQnobtM033rrrTRNPc+zbbtYLFYqFVwuZFlWVfXWrVsodGg8finClZvLvH/6zOzsLC50W1tbaM+Y0WAIhLmELMvvvPPOvXv3pqamPv74Y4AKz/mAk1Jq27Zt277v406BovQ8zzRN0zSHwyGmBeVy2TRNTKEYY1euXKlWq7BP0zTUMOre933DMBAyKDrUG0yHhTwtgiDIYRhevXpVkiTLsnZ3dx3HAS9zUiqXy5iyTU5OjsfjarW6tLQkCMLJyQkGKJiOlkqld999d3V1NZvNNpvNra2t09PTKIo+/fRTpCuKolwuJ4ri5eVlpVLBoCgMw2w2a1kWDEWAXNfF3YFSilE1HOOBwxPUCe9RhBDh888/z2azgiD0+/1Op9Pr9QaDwXA4xHeVSmVhYaFWq4F8JEkyTRMjlU6n02q1II0YY7Ozs6qqQhGORiMgFlM4UFkYhplMRpblfr8P9YBaQi+CzsAO0NqArmEYb9YhKBvcqygK2sl/NYOGgHZbKBQALYxVQXH5fF4QBNd1GWOqqjabTcuyCoVCoVDAeMk0TfDhcDjENFDTNMuycDBaCkTK2dkZQp7JZEajEWbt3FsUCU6BwIG5w+EQw2veIWA6Rx0H0f8AjH4V96gqofYAAAAASUVORK5CYII=\n",
"text/plain": [
"<PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=68x31 at 0x7F87EA2B2128>"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"img2"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Geometry transformations"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [],
"source": [
"def sample_asym(magnitude, size=None):\n",
" return np.random.beta(1, 2, size) * magnitude\n",
"\n",
"def sample_sym(magnitude, size=None):\n",
" return (np.random.beta(2, 2, size=size) - 0.5) * 2 * magnitude\n",
"\n",
"def sample_uniform(low, high, size=None):\n",
" return np.random.uniform(low, high, size=size)\n",
"\n",
"def get_interpolation(type='random'):\n",
" if type == 'random':\n",
" choice = [cv2.INTER_NEAREST, cv2.INTER_LINEAR, cv2.INTER_CUBIC, cv2.INTER_AREA]\n",
" interpolation = choice[random.randint(0, len(choice)-1)]\n",
" elif type == 'nearest': interpolation = cv2.INTER_NEAREST\n",
" elif type == 'linear': interpolation = cv2.INTER_LINEAR\n",
" elif type == 'cubic': interpolation = cv2.INTER_CUBIC\n",
" elif type == 'area': interpolation = cv2.INTER_AREA\n",
" else: raise TypeError('Interpolation types only nearest, linear, cubic, area are supported!')\n",
" return interpolation"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# CVRandomRotation"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [],
"source": [
"class CVRandomRotation(object):\n",
" def __init__(self, degrees=15):\n",
" assert isinstance(degrees, numbers.Number), \"degree should be a single number.\"\n",
" assert degrees >= 0, \"degree must be positive.\"\n",
" self.degrees = degrees\n",
"\n",
" @staticmethod\n",
" def get_params(degrees):\n",
" return sample_sym(degrees)\n",
" \n",
" def __call__(self, img):\n",
" angle = self.get_params(self.degrees)\n",
" src_h, src_w = img.shape[:2]\n",
" M = cv2.getRotationMatrix2D(center=(src_w/2, src_h/2), angle=angle, scale=1.0)\n",
" abs_cos, abs_sin = abs(M[0,0]), abs(M[0,1])\n",
" dst_w = int(src_h * abs_sin + src_w * abs_cos)\n",
" dst_h = int(src_h * abs_cos + src_w * abs_sin)\n",
" M[0, 2] += (dst_w - src_w)/2\n",
" M[1, 2] += (dst_h - src_h)/2\n",
" \n",
" flags = get_interpolation()\n",
" return cv2.warpAffine(img, M, (dst_w, dst_h), flags=flags, borderMode=cv2.BORDER_REPLICATE)\n"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAPQAAACVCAIAAADg5UBWAACcyUlEQVR4nNT917IkSZKuiykxc/egiycr0nSm9waO4Dwnng4iwMWePaRJkaSLBnV3M1XFhZpHxGKZWd3VZ4CSkZGurMyVEe5mn6mp6v8r/r9e/j/NBBEZoe97Q0QmQlZDBUPiQEzAqU+B0cxySoRYVZUhAgDkJCKowkwcogIkVUMwABFkZiRSUzBDQCRCJFBDMFSTnFQFCTkQEokCADAzAGjKKmJmQECERASgSQ0RkZEIAVANAICIREREkyoRcgzIJGKdZDEFQlFIOZsIAkTipoqRmMFCCCKqaqaASEgIoJJzRETE1HcABgCISMSGqJoRMcaARElzLzmpJpVslk0NAZmIoopJEhCLIYyrUcUMZoxSUUBQk6SSwAwIkUlE1FSNEMj/8b9OAfq+x8iimsEENIkkzaKaQc3MHwOagQirVRTGcRRDJFUzMwADAwR/eoFQRECEmIBQTDvRPpsBMgcAUgVTUxUEBTBErOqIQCICRMRBVBUNCHLOWTOYmRmKmhqIBgqjpqlDNLM29choAAqqmgmBOaiav1kx6fqulwQBgUkBiEgRMAOamQIaMhIjARgjRqbIwVRAxFQA/MuVRWHEhiyEGSGBJbMElsESgLIJASEiAAKAATAzmoGagREimKmpmhkYM6kaIjJzztn/jrK4iAxJRVWFkQh81QMRqKqZEZEBmBn4/yEYoCESMyKqPyMrm0VVAQCZERHA/4R/JWREMzM1G34zAPibRkQCVDUT8Q/lnwQMfOEYogEk2a1nUFWi8idVFQwQEYn8EXII/pN9qSAAIgGAiJoZExMSITIiIZI/QDUzISJ/uyKSJIsZIKpRNkEkJAYkGL4YlU9gZuZf3L8sAjCzZSUiMiAkRqLChvIH1AyJAEkBxMrSL8/FDBHBwMxUTQ2ICIhMFA0YiRGYAMD/UkMEfxBmgIiqqqIARkT+I8qHQd9/5HveCP3/55xTzmqKiJGDqSEAAgLR7uf7VyOkEAIBapayJswQwAgMy0dWU4CyDMVMVBEJCAHQzAgoYEAKylXm0BO2aFvTrekWtAXrGFI0qziMKhq+lakBM5cnbQqghIRmYmqgxOzrGYmIOaWE/u/EiAiIBmCqZspETo3hJ6k/a0Pfd1YWPzovyZ+mmdHwCAAAyTGPWHaEASBh2YX+K7vf74+cCQFA1UwU0QITIvqeYSIiAgBD6HMWUwNUVUQsnwV9HyIRqbOBmIj8w2j5CuRL0FQJMFBZcAxA6N/fn5v5+lazlFP2JQsoDg0k8vVtYH4QAfpm321sM/XFDWZo5YcTISMhIkPZ1gZlfRuhmCVJolI2uv9If+gATqXhHRkABKLh8QyLDwERYVjfOYuB7xAzUzBjZvXNhgO+hj0Bvr5FoPARwAABCQiIRMSfsqoiIBMHDgSkooRoZmgABIDl/wxATMoGVlNTATNkYGaKSEE5CseesEXYgG5Mt6BbsI6gj2YVhVEV6hgDl11YMAlARAULoLRjipmBEbOIISKHkHP2N+Hr+z68kXAH74LdsrYew5sc3vYEvMMT8Kb78MYB3mV9+z4RAQNCCuTwNkJgZv/pWTSLyCN4D/tuD2/ip+HtJ38gLtjeAdWXGwzwRsgqWbKoAqIZZtMCb/pqeMtDePv6HuCtQAhIipBNsubd+gZHj8PbDuCtBd4BkfFZeJuZiBzCO4RgooS4W9+IaOQIB4e3qCJA8OcGQECA9+Ctqg5vBlK5D2+8B2/fIL3KKvXLvttKVmaLlYTYE7UEG9CtyRZsC9YSpGhaITdlWUcEzpnUjzC4B28oi0nR4a0Ob/LFvFvfA7zJ14QB+PoOSFjgDU/A2w7gTTSs70fwRkTew1tV78FbYRec7CMTRPTgRAXRgwdnGPhbcXgnj3T38MYH8JY9vNl/uMObDuCNAFzgXYKT8lF0gDehmfW5P4S3OLyRwPfTHt7wHLzBoKxvIvYTbQ9vKPBGUrNeelHV8nX28DYzeQRvJgyEiKAqw+Ler2+/TO3hrerrW7Mxsz/JQ3gbYUopqwBAYEYnsK/vA3ib+d2EmZl8fT8D74S6Jekq3AZYQlqiLFEXICvQLehmWNYdWQqmFVITY1PFSIGAJVNKnCSoKpUAdx95W86maoyEwdS0nGxIgSVLCGTMqe+DKjKX4MRMkVSFjJiIyprcwdv85mdm5b8gmCEAIDGqqCqqMRohqjlTiZhNtGy0YRETgph5cFL+mgHeZkaGWZR8LSMxkYqpGSEyc1JFM4c3ISmiw9sMgUylwNvXN5sxh5S6Xajmy9hMVRWJArEvJjVkRDXU8l2FOaqqqolqksxEgcgMxZQpACmYgugO3qaCoKZlv5X1jcTMIkKBVYEImUhMDZENxVcdKBGDoiKIaZLEvrEATNVDHsCyjzzoMlUiZiRGYzTDcoKVZYpk5ZmYZOFQ/pSqcgjSCwAysaoqqIfLaGhkIpJyDsRMFIiTZkQkIEVSET84bXizIQS/k3j+oJyQBGDIowAxWMXNpMEqAGKXelBLvXSrNi3bpqozgBAIGQSiGMjvQAAkgqIkhmoEEADA3z0AqYoCMnk8AqYGpL5+RI3YiFhRdjfLlFJNZIhAhOaXd1VRRGUkVfEto+qIMiLy8BpAkdiXOBI6IE3UCJHKTajw2O+4/mdUmZmwHFqmYGjlow7nOyERmqqhKAYs688Mh6yLqRpYnyUQK7P/TF++frMkJiLKOZcjmNhMy28wIyQxVTVSJaZALGaERuDr2z+6+s3S2CxbkhQ5BCIDFDMxZWQiXx5iaMhESGZq6IGbnxXGDMy8v5SbenCiaOSB8ABv39iimjRFjRRoCF0Kwv0NRt7DGxmZKPhnVfGndxB5q0feHNhPb4c3M0sWCsTE6ukthGFXQEopMDNVgTmr+L4iICUVEeZQMAfExCEEzWoiHEJWZWSometYn8zHF8dhPo7jOo4bZVSkftOub5bLd9d819om9dstBKIQKOyWtaIIqpEYAXgCIZRIiNk/IwB4cGI5mykCIgZU8/eAiMSsA7xz3+sO3kiI6tuUeA/vfeSthjxE3n4fRjT//sRa4K1+xOvwjonJxKNe2L34Am8FIyBCHP4TEZnJHt5G6PBWUzOCAd4AoppUiFCRUJWIPfDzfejbQD32+By8kZlYUREVkBAZUPbwrlRV8SG8sypz8G3oS7QEJ4gIasa7m+4O3kkyx6BZD+FNVsI8cXgbmhV4e1iMiHt4wz14qyoTMyITspoiqEoJ/gd4e5zl8EZEU1MRjlF6KZGS0mN4Z5HMEogDc5IDeKtQYcQe3jnnJJnZkFHHYXRxNP3mfHRxPHlxPD470prCuDEwIs5tn5bd9tPq8s/v0qe1vPugpoxIhCSC+eGy9rt28Lfof5/DGxA9vSAehlO51aqqMRGxoqoaDWmTh/A2O4S34RB5O/mGRWPm8PaLZ4G3ihEZUVncn4O3B50Ob8BdZEJEpPAZePtBD2ApSyRWBvL74gBvh4tvAyqpzgfw5gHeRoiBWf3ShkiGAnt4M7OpWbaUU+DARACoUOCNpKYKKoCGzgLzCLl8TVXl8ibQb5YO73Af3rCDt5qoJEmBA/P+orqLdx3e6MeXR1ZIQiXjRqSAjM/Be8h9FXjH+/AmREOjAu9AHIhFxHbwRlPVEIKIrwT0m6WoJlKejWa/uTj7l+/G35w250c8rWhSWUVUscdUVQp1N5qdzZtJs/rxisiu3n1EEeoE/UXYwbIGxBANcVjcZlTuigXeRKQ5m6pf0BE876iIyEyScwhMzLnvRYRCKDdLVSVUESIi3sN7eDTPw5sLvJ1b9+Ad2LygcwBvBshPwRt9NxtlkR28A3GBt8fTIuDwlmfg7Tk4Ir/+cwipP4Q3ErKaiCgiBmZBESQG0z28QU2YKiRUNAdqeAhvRpIn4b3b2Ht458xVgTeVe66R2T14E5mpgCZJjI5mNDVA2MFbFAjRiFTET9dAKAXeOqRuHN5eG4CcJTyAtxR4s7GoeK7D764l8vb1zaGXNMBbdy+3kArRGCTi6NXx7LcvT/7lzdHvX/HJCGaRm0ojGhkwmikjobKtBRFP5DQmwLvN5sNl2naoRgZ8uKw5GFObcyfZS2JU0pDlsuwXwSE37DUdQDQVVQNDQkTawXufNtmnxUhVyseycqsu5SVnTGG5Dvd+gMO04EHaxNc0BX6QNkGE8nsOajr7xY1IiCLqOW9m8lQdQoG3IRpCL1lUFUDNAOwwLWiGzKwAVj7aQVrQ1HOCHoh6DnhIC5aCDviVxYSZkdGGnLdnuBRgnxZEgpIGLdmew7Sgn3LkXBdlIvIqjCeohnoZAIgpEhqSbyTx9NMQ3+zSJmql6AqI+7QgIZVkkQLu0oIlZJecYfhJYKYi+wIT7tImnu0vaZOS8yYioH3aBEmkBD+K1kqfI4xfnZz/6dtX/4/fHf3pVfh+Dhc1zINNWaNZMEWhgGYCmJEEQRikRmCAcaxYLCoGRfAPzgGrqgNbdt3K+k0QAihJaL/o+VffRd4ejgBoOa1KWtCIS2hIzFpyolDgXcCgplK+MhzWdMopCbu0N+JQsAwDHh6lBZlxSD8dpAV9FXiCCR6sb0ZSNZVS+Q9UUpye8/YfpWpZvEC6T1MMpT0D84qxmZlfhg72mxFyCdXUPEVAQL7EuexDUysnGNBBTQfRDEXFo/SyvocnQ4iAevAXwRD6l5y3l3W8Rl0W6VCw9ODQEAUsaSoX/yHmBhjS3v4KiDyoII+8CajUdMCz2IBk5tTCnPOQFizXSk/2ERGXu2spWAKCmu5qOoHZhpz3jggcKFva5A1Mwvkf31z8376b/fGi+maORxVMWIJl7VWz9B2mTL1gl2krtE2wajfXN1fv319/uszbjg09nEAqy3rRtyvt15zziOPxOACWdHOpvZXIO5ecZ4m8FQgJyVTFkMDJQYeRd+PJnpKuMEBSUSQNz0feHpvuI28E4mCWVA096DyIvInJ8+Wlkg+EYITgd/8HaRPP7pGhqpIIBgzMoiW28fXtkXefJTATE5oRGBGW5WflJyUR8xXAQVWGyFsR2U8NVWOyQCSqCkNOsHzQkjZRUk+bBA0BEQAVMJuGcq1EEEem0w8B1Yw88jZT8nN0gHdJm5CJKRmqd5IAqCkTmpJ65E2Bieww8rbdtRLRr/mqSMRIAU3Qa91aPgYiIAHILvImLGklh3fOmauSE7TC53uRd/SUuJKZAQIhhqpSyalP27TtOR9fTE9+ezH//gwvxnJcJzRBFVEQy5uOxHpRAGIByLC6vLt5e735cHf31+vF22tb9aG3uhphHXrJXd/2oD0pVBzG4zhp4qwJgAaGu8i7bFXPTAEys6ZkCkhOx3KzJEIKnPsUIxNRFsk5c4wAAMSePVbJpaEKyUA98i7Bz1D83Z3y+7QJipbUtB1G3siMouBxQLlxIyFkNVA0tAHN+5oOIyUVv6QCIjOrmEDJY/ruUrMkQkhINFzjzCNfj7x9GyAAcRDJQ+RtiEZIaqKqpMjM7HxAY0SFUklVzEw1cXm8OedMHInUKKswRyzbwWDYooQooIf5UE+rM3PKiapIYobAiIzkWQrz9wJDGdJMwJKkQIwcaBd57+E9RN45e+TNhMFsuMv6BRIN0RT9ti1ZMJYMjClyCAhoYkRERqKlRQQVDU1VU86BQ2QOxOq7XDMyIUKvqYd+cjY9/c3F8fcX8WyaJiGzKYJkaDdde7tZX93ldRuT1YbjGPttf3t5t7xcrt7f2EYS5N4SZJnFSvucQHtSqyiMRnHSxNkoTpowagKUJpf9+mYiADoIz56ENzIycYG37+MQQonXHN5EJmokTCzid47Cb1DzwAt8OxEcwJut/B3OrUN4s6juqjYAhACENhRxjRDlPrzZSFVJpaRNRNXE4T1UIi1licwKpGZkT8DbY+WvgDeLqp9c5PAGADO17DlvTdrnPnj2wWuWXs0lNlMQD22fhTcTKbG3ZKg3IT0Hb0MRyZrzkMB+At4lliAVJSZGYrKs5skFRNvB20yIHsFblZmzPAvvnHMKOUvmEGKsTJXE0JQDNVW92Kyr4+bom5Nw2sgIpSIJkNVSlsXt6uqv7+/efmqvVyHppB5VMTDH1c2GAbGp60lVHVlcdd31erNJImqRwngUxnWcjuJsFEd1aBqsQ/ByPxju6gX2HLzRvI6tXm8hIx7gzSwi+/VNTGZippJR1QNEvz46vBlLX9ST8DYSLdmNfbBReC/0CN64gzd4q8Nh2kQxqaqUPFdgUlEx82vlDt59FkLS+/AegqiSYPG0iXSy/08P4U3Bc+pO4wHegMLESl4MAU+bELMaZRCmgESoZEhfA2/pe6qj70vvn0lm7KmNAm8EJNinaJiehLcVeFvOfmNlxEBPwNvb8Tw4iZFKpVmRCrw9gNQH8M5oy34bYjWJDYSwWSxGdUWsxtLnLTY8uZhPX53wvJaahL1V1ZJp1/aL28XyaiGbHrJVzbRuplSFk+msCqEKMYTYb3vrtF906w+3i4832cSXdRg1oampDlQHJQjm4ESA4Qh+Et5qZqbe8AEqaqiGhExMIuprJaUUPCfowTdqAQMpUxDxn/Y8vKHkJpAYVFW8SvIQ3mqGuqM1IiKh8xO8K3sH73KzNPKct6dmqeybEnlnh7fkKrACqsEO3laCKHZ4KwADcggq2Vcbe+xT4K2EyMRsKk/AW3bwTjlFjt6fYYBZNT6Ed8mE6H14o6E3EWgWYlIVchKgJ/FRhgS73yBEctIcvwbeOqTPH8CbvB+owFuyBPZuGrThZply5poJSbGkJCSAAIVxM5rOJuPJrJm0i017d5u3m+m46XPfSYIxjWaTOKmhQmMTNCXwvpPJbDKaTeglsFLTjMaTST1uQsXNuB7Figg5VirKQu3tNl1vFz9fv/3p5zBqwqimOmIVgCEDGFjg4bvBgKjPwlsduWpesCSHNzMRs96DN5GRmJmIqjLql+EN5NnvHbxVDVEJSYa6BjKh4JCNcEL757GhX+AJeGdVL1sAciBWK/Cm/QeAJ+HtFYRyJxEhAOagknfZHgTFAm8jMmZiJcYvwLvPPRNVzGokIMHhbWTm8EYiMjHwe2m55Pg9kgu8mcn7m70Ea0aGOqRNvFEEkNS0l54fwbt0UxnwHt6IhIwUSvuJMGPZEvfgnWNVDfBWCqFcc5nVNJEmU2u4OZ3NTk/PLl4ez47ypk8/vqfLqt+ss6mCKdj86GgymSKSmhmqMRkqIEDEOBu9+Zff2PcCGaqqCXWEimNFzFAREwJGNjFUqmexOZ2Pjo9yE1arNdYM7F075v3ggZBEs5+Gn4G3/7qpASshq6frTJ1WDm8K4R68iVDVhppO+DK8hy5NRKJglrzbhAh2cTYiYmBMh/C2Q3gDAhE+gDcZiRiSw9uTFw/h3eccgz9jQFWHN5R9eAhvID6ANyMi+/3T4R2IxVSRFI130bCZDgVLTZolZ5VQ4E1ZNRITqqCCZlDzHg8/4Xes2d0rmNl7HLwQo0aCBmBeH30Ab2+75fvwLj/Udj19pGrDtRJEQdRfbglOAMhUiFBEWYWJ78M7WcRMahXX8zEfjU++e3nx7ZvpfD7m5upvH1tKGgECddJXkUdxhIBVDAHKOUugaKZmwFQdj5vJyHpFQ0WAQIaAARFBENVyqcSoAkcacVVVp/Jy/ecfFUxB0QwNUteltg1m5lsfh7TJk/De/Tqawn14M1PqpVS+iVJKMUZPCw6XUfGCfCBKqoBAZCJ+rB/C24Z+YithqPnfcR/eVLKNh/DmAd5QdAwlzULeKqjeBzvUnIm92noIbwBIOXNERSYzZhre/h7e8hy8gbxg6VltVhY073BiGI4UlCE3jaqWcgrEFbMaCAgbIXmE47SGr4C3V+wd3qrim2GAt1e5jLybKmjYwxvuwTv4O8rZE9UO79J/wrstgTBsDMlCFR3AmwW0z5nn1fTF0fy3F/WLo9GL46MXZwGDttq/17tus+nbEaFrowIGzba6Xsxvl9OXY8oE2TAQE2UAAcOaqCIzoIF9RihgoglJkRQAQJG5ygioNHpxNL1b3F1eI0Bqu9x20iXtUrCSdZbn4W2fgbdfnzhwud4xZ4e389lfiJmqoBJzEFApFUlvXcd78MZSKzb0bhNVVbLSwnzvZqn6AN6MIObCT6ChtbAUpQjJ0OGN3qatqH5a3IO3RGZCcnh7tuAQ3jrAm4e04CG8oTTjY2ASI/XSvyEBiN9bMTMHl+5lyVkKvNVITKN3kSiBCiiA3ywHeA+Lu8DbBaD8FLzVC/JoAcnQFHLWnCXzkL02uAdvdXUCkakSMhOxKdnn4a3sLZYMAmIRwqiafXNy8a/fTn//on59hJOaq6gJs2431t1ubiV1E65NpOuTBuMQ7t5eHX9/PnlzxCOMIQAhMSGDqCIZKMYQsihIEc2gqQ0STyRUAlHBOhAwpnj+5vzq53ep7aXrtU1526V1S0PhA82K2InKjWEHb//fsGu9tb0IzRV6SswuAiUiJHqgY/AGEvOCPNFQBh9oPTS0lL/Ry+ToxwMhoIqpKpWaopfQij5tt+JdxwDlX32r72qm5EkVMNBdQb7oGIwORGgA0Oe9joGG4upwtOCuYEkcPAwd6qyKyGYmouA6PyIuUmZvz/Ovp4BGXERovfSlYAkopmoAXrCk8vcREQICPi1CM1EcFG5cGnlwr2MwUzAgNCJxHYMdFCxtD29vugIqKwGh6HSw/I27ajzdE6ERKgKGIGCddTTl49+cn//p9eQ3x3hR22noZ5RHcNct316+27ZrQB039aQZBaCgoJteV+3q3fX6/Q2vUtjm2ElMUoPVAMHzP12PfYqiUSAkpS6HVqlT7MV750WTsWBl0GDVVNPxJC3W/c16e7XYfrrrLu/Crqp3CG8/dDwM2MHbdQwi4r09hJ4X92jNDuGd+r7AGw7gLYokzFGgtL
gitextract_vdjnznth/ ├── .gitignore ├── LICENSE ├── README.md ├── callbacks.py ├── configs/ │ ├── pretrain_language_model.yaml │ ├── pretrain_vision_model.yaml │ ├── pretrain_vision_model_sv.yaml │ ├── template.yaml │ ├── train_abinet.yaml │ ├── train_abinet_sv.yaml │ └── train_abinet_wo_iter.yaml ├── dataset.py ├── demo.py ├── demo_onnx.py ├── docker/ │ └── Dockerfile ├── export_onnx.py ├── losses.py ├── main.py ├── modules/ │ ├── __init__.py │ ├── attention.py │ ├── backbone.py │ ├── backbone_v2.py │ ├── model.py │ ├── model_abinet.py │ ├── model_abinet_iter.py │ ├── model_alignment.py │ ├── model_language.py │ ├── model_vision.py │ ├── resnet.py │ └── transformer.py ├── notebooks/ │ ├── dataset-text.ipynb │ ├── dataset.ipynb │ ├── prepare_wikitext103.ipynb │ └── transforms.ipynb ├── requirements.txt ├── tools/ │ ├── clean_dictionary.py │ ├── create_charset.py │ ├── create_language_data.py │ ├── create_lmdb_dataset.py │ ├── crop_by_word_bb_syn90k.py │ ├── get_max_len.py │ └── line2word.py ├── transforms.py └── utils.py
SYMBOL INDEX (231 symbols across 21 files)
FILE: callbacks.py
class IterationCallback (line 18) | class IterationCallback(LearnerTensorboardWriter):
method __init__ (line 20) | def __init__(self, learn:Learner, name:str='model', checpoint_keep_num=5,
method _write_metrics (line 36) | def _write_metrics(self, iteration:int, names:List[str], last_metrics:...
method _write_sub_loss (line 43) | def _write_sub_loss(self, iteration:int, last_losses:dict)->None:
method _save (line 50) | def _save(self, name):
method _validate (line 58) | def _validate(self, dl=None, callbacks=None, metrics=None, keeped_item...
method jump_to_epoch_iter (line 70) | def jump_to_epoch_iter(self, epoch:int, iteration:int)->None:
method on_train_begin (line 76) | def on_train_begin(self, n_epochs, **kwargs):
method on_batch_begin (line 88) | def on_batch_begin(self, **kwargs:Any)->None:
method on_batch_end (line 92) | def on_batch_end(self, iteration, epoch, last_loss, smooth_loss, train...
method on_train_end (line 159) | def on_train_end(self, **kwargs):
method on_epoch_end (line 163) | def on_epoch_end(self, last_metrics:MetricsList, iteration:int, **kwar...
class TextAccuracy (line 167) | class TextAccuracy(Callback):
method __init__ (line 169) | def __init__(self, charset_path, max_length, case_sensitive, model_eval):
method on_epoch_begin (line 179) | def on_epoch_begin(self, **kwargs):
method _get_output (line 187) | def _get_output(self, last_output):
method _update_output (line 194) | def _update_output(self, last_output, items):
method on_batch_end (line 201) | def on_batch_end(self, last_output, last_target, **kwargs):
method on_epoch_end (line 231) | def on_epoch_end(self, last_metrics, **kwargs):
method decode (line 239) | def decode(self, logit):
class TopKTextAccuracy (line 255) | class TopKTextAccuracy(TextAccuracy):
method __init__ (line 257) | def __init__(self, k, charset_path, max_length, case_sensitive, model_...
method on_epoch_begin (line 265) | def on_epoch_begin(self, **kwargs):
method on_batch_end (line 271) | def on_batch_end(self, last_output, last_target, **kwargs):
method on_epoch_end (line 287) | def on_epoch_end(self, last_metrics, **kwargs):
class DumpPrediction (line 294) | class DumpPrediction(LearnerCallback):
method __init__ (line 296) | def __init__(self, learn, dataset, charset_path, model_eval, image_onl...
method on_batch_end (line 315) | def on_batch_end(self, last_input, last_output, last_target, **kwargs):
FILE: dataset.py
class ImageDataset (line 14) | class ImageDataset(Dataset):
method __init__ (line 17) | def __init__(self,
method __len__ (line 59) | def __len__(self):
method _next_image (line 62) | def _next_image(self, index):
method _check_image (line 66) | def _check_image(self, x, pixels=6):
method resize_multiscales (line 72) | def resize_multiscales(self, img, borderType=cv2.BORDER_CONSTANT):
method resize (line 99) | def resize(self, img):
method get (line 105) | def get(self, idx):
method _process_training (line 136) | def _process_training(self, image):
method _process_test (line 141) | def _process_test(self, image):
method __getitem__ (line 144) | def __getitem__(self, idx):
class TextDataset (line 168) | class TextDataset(Dataset):
method __init__ (line 169) | def __init__(self,
method __len__ (line 193) | def __len__(self):
method __getitem__ (line 196) | def __getitem__(self, idx):
method prob_smooth_label (line 222) | def prob_smooth_label(self, one_hot):
class SpellingMutation (line 232) | class SpellingMutation(object):
method __init__ (line 233) | def __init__(self, pn0=0.7, pn1=0.85, pn2=0.95, pt0=0.7, pt1=0.85, cha...
method is_digit (line 251) | def is_digit(self, text, ratio=0.5):
method is_unk_char (line 257) | def is_unk_char(self, char):
method get_num_to_modify (line 261) | def get_num_to_modify(self, length):
method __call__ (line 280) | def __call__(self, text, debug=False):
FILE: demo.py
function get_model (line 14) | def get_model(config):
function preprocess (line 24) | def preprocess(img, width, height):
function postprocess (line 31) | def postprocess(output, charset, model_eval):
function load (line 57) | def load(model, file, device=None, strict=True):
function main (line 67) | def main():
FILE: demo_onnx.py
class WordAccuary (line 22) | class WordAccuary:
method __init__ (line 24) | def __init__(self, case_sensitive=False):
method update (line 30) | def update(self, y_pred: List[str], y: List[str]):
method compute (line 41) | def compute(self):
method reset (line 44) | def reset(self):
function _decode (line 49) | def _decode(logit):
function preprocess (line 62) | def preprocess(img, width, height):
FILE: export_onnx.py
function get_model (line 10) | def get_model(config):
function load (line 20) | def load(model, file, device=None, strict=True):
FILE: losses.py
class MultiLosses (line 6) | class MultiLosses(nn.Module):
method __init__ (line 7) | def __init__(self, one_hot=True):
method last_losses (line 13) | def last_losses(self):
method _flatten (line 16) | def _flatten(self, sources, lengths):
method _merge_list (line 19) | def _merge_list(self, all_res):
method _ce_loss (line 31) | def _ce_loss(self, output, gt_labels, gt_lengths, idx=None, record=True):
method forward (line 52) | def forward(self, outputs, *args):
class SoftCrossEntropyLoss (line 61) | class SoftCrossEntropyLoss(nn.Module):
method __init__ (line 62) | def __init__(self, reduction="mean"):
method forward (line 66) | def forward(self, input, target, softmax=True):
FILE: main.py
function _set_random_seed (line 18) | def _set_random_seed(seed):
function _get_training_phases (line 26) | def _get_training_phases(config, n):
function _get_dataset (line 34) | def _get_dataset(ds_type, paths, is_training, config, **kwargs):
function _get_language_databaunch (line 52) | def _get_language_databaunch(config):
function _get_databaunch (line 77) | def _get_databaunch(config):
function _get_model (line 98) | def _get_model(config):
function _get_learner (line 108) | def _get_learner(config, data, model, local_rank=None):
function main (line 187) | def main():
FILE: modules/attention.py
class Attention (line 5) | class Attention(nn.Module):
method __init__ (line 6) | def __init__(self, in_channels=512, max_length=25, n_feature=256):
method forward (line 18) | def forward(self, enc_output):
function encoder_layer (line 33) | def encoder_layer(in_c, out_c, k=3, s=2, p=1):
function decoder_layer (line 38) | def decoder_layer(in_c, out_c, k=3, s=1, p=1, mode='nearest', scale_fact...
class PositionAttention (line 47) | class PositionAttention(nn.Module):
method __init__ (line 48) | def __init__(self, max_length, in_channels=512, num_channels=64,
method forward (line 68) | def forward(self, x):
FILE: modules/backbone.py
class ResTranformer (line 10) | class ResTranformer(nn.Module):
method __init__ (line 11) | def __init__(self, config):
method forward (line 27) | def forward(self, images):
FILE: modules/model.py
class Model (line 11) | class Model(nn.Module):
method __init__ (line 13) | def __init__(self, config):
method load (line 18) | def load(self, source, device=None, strict=True):
method _get_length (line 22) | def _get_length(self, logit):
method first_nonzero (line 29) | def first_nonzero(x):
method _get_padding_mask (line 36) | def _get_padding_mask(length, max_length):
method _get_square_subsequent_mask (line 42) | def _get_square_subsequent_mask(sz, device, diagonal=0, fw=True):
method _get_location_mask (line 52) | def _get_location_mask(sz, device=None):
FILE: modules/model_abinet.py
class ABINetModel (line 8) | class ABINetModel(nn.Module):
method __init__ (line 9) | def __init__(self, config):
method forward (line 17) | def forward(self, images, *args):
FILE: modules/model_abinet_iter.py
class ABINetIterModel (line 10) | class ABINetIterModel(nn.Module):
method __init__ (line 11) | def __init__(self, config):
method forward (line 20) | def forward(self, images, *args):
FILE: modules/model_alignment.py
class BaseAlignment (line 8) | class BaseAlignment(Model):
method __init__ (line 9) | def __init__(self, config):
method forward (line 18) | def forward(self, l_feature, v_feature):
FILE: modules/model_language.py
class BCNLanguage (line 12) | class BCNLanguage(Model):
method __init__ (line 13) | def __init__(self, config):
method forward (line 41) | def forward(self, tokens, lengths):
FILE: modules/model_vision.py
class BaseVision (line 11) | class BaseVision(Model):
method __init__ (line 12) | def __init__(self, config):
method forward (line 40) | def forward(self, images, *args):
FILE: modules/resnet.py
function conv1x1 (line 8) | def conv1x1(in_planes, out_planes, stride=1):
function conv3x3 (line 12) | def conv3x3(in_planes, out_planes, stride=1):
class BasicBlock (line 18) | class BasicBlock(nn.Module):
method __init__ (line 21) | def __init__(self, inplanes, planes, stride=1, downsample=None):
method forward (line 31) | def forward(self, x):
class ResNet (line 50) | class ResNet(nn.Module):
method __init__ (line 52) | def __init__(self, block, layers):
method _make_layer (line 74) | def _make_layer(self, block, planes, blocks, stride=1):
method forward (line 91) | def forward(self, x):
function resnet45 (line 103) | def resnet45():
FILE: modules/transformer.py
function multi_head_attention_forward (line 15) | def multi_head_attention_forward(query, # type...
class MultiheadAttention (line 299) | class MultiheadAttention(Module):
method __init__ (line 328) | def __init__(self, embed_dim, num_heads, dropout=0., bias=True, add_bi...
method _reset_parameters (line 367) | def _reset_parameters(self):
method __setstate__ (line 383) | def __setstate__(self, state):
method forward (line 390) | def forward(self, query, key, value, key_padding_mask=None,
class Transformer (line 450) | class Transformer(Module):
method __init__ (line 479) | def __init__(self, d_model=512, nhead=8, num_encoder_layers=6,
method forward (line 503) | def forward(self, src, tgt, src_mask=None, tgt_mask=None,
method generate_square_subsequent_mask (line 564) | def generate_square_subsequent_mask(self, sz):
method _reset_parameters (line 572) | def _reset_parameters(self):
class TransformerEncoder (line 580) | class TransformerEncoder(Module):
method __init__ (line 596) | def __init__(self, encoder_layer, num_layers, norm=None):
method forward (line 602) | def forward(self, src, mask=None, src_key_padding_mask=None):
class TransformerDecoder (line 625) | class TransformerDecoder(Module):
method __init__ (line 642) | def __init__(self, decoder_layer, num_layers, norm=None):
method forward (line 648) | def forward(self, tgt, memory, memory2=None, tgt_mask=None,
class TransformerEncoderLayer (line 679) | class TransformerEncoderLayer(Module):
method __init__ (line 700) | def __init__(self, d_model, nhead, dim_feedforward=2048, dropout=0.1,
method __setstate__ (line 717) | def __setstate__(self, state):
method forward (line 722) | def forward(self, src, src_mask=None, src_key_padding_mask=None):
class TransformerDecoderLayer (line 746) | class TransformerDecoderLayer(Module):
method __init__ (line 768) | def __init__(self, d_model, nhead, dim_feedforward=2048, dropout=0.1,
method __setstate__ (line 792) | def __setstate__(self, state):
method forward (line 797) | def forward(self, tgt, memory, tgt_mask=None, memory_mask=None,
function _get_clones (line 839) | def _get_clones(module, N):
function _get_activation_fn (line 843) | def _get_activation_fn(activation):
class PositionalEncoding (line 852) | class PositionalEncoding(nn.Module):
method __init__ (line 869) | def __init__(self, d_model, dropout=0.1, max_len=5000):
method forward (line 881) | def forward(self, x):
FILE: tools/create_language_data.py
function is_char (line 20) | def is_char(text, ratio=0.5):
function is_digit (line 29) | def is_digit(text, ratio=0.5):
function disturb (line 60) | def disturb(word, degree, p=0.3):
FILE: tools/create_lmdb_dataset.py
function checkImageIsValid (line 11) | def checkImageIsValid(imageBin):
function writeCache (line 22) | def writeCache(env, cache):
function createDataset (line 28) | def createDataset(inputPath, gtFile, outputPath, checkValid=True, map_si...
FILE: transforms.py
function sample_asym (line 12) | def sample_asym(magnitude, size=None):
function sample_sym (line 15) | def sample_sym(magnitude, size=None):
function sample_uniform (line 18) | def sample_uniform(low, high, size=None):
function get_interpolation (line 21) | def get_interpolation(type='random'):
class CVRandomRotation (line 32) | class CVRandomRotation(object):
method __init__ (line 33) | def __init__(self, degrees=15):
method get_params (line 39) | def get_params(degrees):
method __call__ (line 42) | def __call__(self, img):
class CVRandomAffine (line 55) | class CVRandomAffine(object):
method __init__ (line 56) | def __init__(self, degrees, translate=None, scale=None, shear=None):
method _get_inverse_affine_matrix (line 89) | def _get_inverse_affine_matrix(self, center, angle, translate, scale, ...
method get_params (line 129) | def get_params(degrees, translate, scale_ranges, shears, height):
method __call__ (line 154) | def __call__(self, img):
class CVRandomPerspective (line 185) | class CVRandomPerspective(object):
method __init__ (line 186) | def __init__(self, distortion=0.5):
method get_params (line 189) | def get_params(self, width, height, distortion):
method __call__ (line 201) | def __call__(self, img):
class CVRescale (line 218) | class CVRescale(object):
method __init__ (line 220) | def __init__(self, factor=4, base_size=(128, 512)):
method __call__ (line 236) | def __call__(self, img):
class CVGaussianNoise (line 246) | class CVGaussianNoise(object):
method __init__ (line 247) | def __init__(self, mean=0, var=20):
method __call__ (line 256) | def __call__(self, img):
class CVMotionBlur (line 261) | class CVMotionBlur(object):
method __init__ (line 262) | def __init__(self, degrees=12, angle=90):
method __call__ (line 271) | def __call__(self, img):
class CVGeometry (line 281) | class CVGeometry(object):
method __init__ (line 282) | def __init__(self, degrees=15, translate=(0.3, 0.3), scale=(0.5, 2.),
method __call__ (line 293) | def __call__(self, img):
class CVDeterioration (line 299) | class CVDeterioration(object):
method __init__ (line 300) | def __init__(self, var, degrees, factor, p=0.5):
method __call__ (line 314) | def __call__(self, img):
class CVColorJitter (line 321) | class CVColorJitter(object):
method __init__ (line 322) | def __init__(self, brightness=0.5, contrast=0.5, saturation=0.5, hue=0...
method __call__ (line 327) | def __call__(self, img):
FILE: utils.py
class CharsetMapper (line 14) | class CharsetMapper(object):
method __init__ (line 21) | def __init__(self,
method _read_charset (line 40) | def _read_charset(self, filename):
method trim (line 64) | def trim(self, text):
method get_text (line 68) | def get_text(self, labels, length=None, padding=True, trim=False):
method get_labels (line 79) | def get_labels(self, text, length=None, padding=True, case_sensitive=F...
method pad_labels (line 90) | def pad_labels(self, labels, length=None):
method digits (line 96) | def digits(self):
method digit_labels (line 100) | def digit_labels(self):
method alphabets (line 104) | def alphabets(self):
method alphabet_labels (line 113) | def alphabet_labels(self):
class Timer (line 117) | class Timer(object):
method __init__ (line 119) | def __init__(self):
method tic (line 129) | def tic(self):
method toc_data (line 133) | def toc_data(self):
method toc_running (line 139) | def toc_running(self):
method total_time (line 145) | def total_time(self):
method average_time (line 148) | def average_time(self):
method average_data_time (line 151) | def average_data_time(self):
method average_running_time (line 154) | def average_running_time(self):
class Logger (line 158) | class Logger(object):
method init (line 163) | def init(output_dir, name, phase):
method enable_file (line 175) | def enable_file():
method disable_file (line 181) | def disable_file():
class Config (line 187) | class Config(object):
method __init__ (line 189) | def __init__(self, config_path, host=True):
method __getattr__ (line 211) | def __getattr__(self, item):
method __repr__ (line 224) | def __repr__(self):
function blend_mask (line 231) | def blend_mask(image, mask, alpha=0.5, cmap='jet', color='b', color_alph...
function onehot (line 253) | def onehot(label, depth, device=None):
class MyDataParallel (line 269) | class MyDataParallel(nn.DataParallel):
method gather (line 271) | def gather(self, outputs, target_device):
class MyConcatDataset (line 302) | class MyConcatDataset(ConcatDataset):
method __getattr__ (line 303) | def __getattr__(self, k):
Condensed preview — 44 files, each showing path, character count, and a content snippet. Download the .json file or copy for the full structured content (2,885K chars).
[
{
"path": ".gitignore",
"chars": 1822,
"preview": "# Created by https://www.gitignore.io/api/python\n\nnotebooks/WIP\n\n### Python ###\n# Byte-compiled / optimized / DLL files\n"
},
{
"path": "LICENSE",
"chars": 1328,
"preview": "ABINet for non-commercial purposes\n\nCopyright (c) 2021, USTC\nAll rights reserved.\n\nRedistribution and use in source and "
},
{
"path": "README.md",
"chars": 5765,
"preview": "# Read Like Humans: Autonomous, Bidirectional and Iterative Language Modeling for Scene Text Recognition\n\nThe official c"
},
{
"path": "callbacks.py",
"chars": 16721,
"preview": "import logging\nimport shutil\nimport time\n\nimport editdistance as ed\nimport torchvision.utils as vutils\nfrom fastai.callb"
},
{
"path": "configs/pretrain_language_model.yaml",
"chars": 720,
"preview": "global:\n name: pretrain-language-model\n phase: train\n stage: pretrain-language\n workdir: workdir\n seed: ~\n \ndataset"
},
{
"path": "configs/pretrain_vision_model.yaml",
"chars": 1123,
"preview": "global:\n name: pretrain-vision-model\n phase: train\n stage: pretrain-vision\n workdir: workdir\n seed: ~\n \ndataset:\n "
},
{
"path": "configs/pretrain_vision_model_sv.yaml",
"chars": 1127,
"preview": "global:\n name: pretrain-vision-model-sv\n phase: train\n stage: pretrain-vision\n workdir: workdir\n seed: ~\n \ndataset:"
},
{
"path": "configs/template.yaml",
"chars": 1442,
"preview": "global:\n name: exp\n phase: train\n stage: pretrain-vision\n workdir: /tmp/workdir\n seed: ~\n \ndataset:\n train: {\n "
},
{
"path": "configs/train_abinet.yaml",
"chars": 1452,
"preview": "global:\n name: train-abinet\n phase: train\n stage: train-super\n workdir: workdir\n seed: ~\n \ndataset:\n train: {\n "
},
{
"path": "configs/train_abinet_sv.yaml",
"chars": 1462,
"preview": "global:\n name: train-abinet-sv\n phase: train\n stage: train-super\n workdir: workdir\n seed: ~\n \ndataset:\n train: {\n "
},
{
"path": "configs/train_abinet_wo_iter.yaml",
"chars": 1401,
"preview": "global:\n name: train-abinet-wo-iter\n phase: train\n stage: train-super\n workdir: workdir\n seed: ~\n \ndataset:\n train"
},
{
"path": "dataset.py",
"chars": 12687,
"preview": "import logging\n# import re\n\nimport cv2\nimport lmdb\nimport six\nfrom fastai.vision import *\nfrom torchvision import transf"
},
{
"path": "demo.py",
"chars": 4264,
"preview": "import argparse\nimport logging\nimport os\nimport glob\nimport tqdm\nimport torch\nimport PIL\nimport cv2\nimport numpy as np\ni"
},
{
"path": "demo_onnx.py",
"chars": 2718,
"preview": "\"\"\" Created by MrBBS \"\"\"\r\n# 10/6/2022\r\n# -*-encoding:utf-8-*-\r\n\r\n\r\nimport onnxruntime\r\nimport torch\r\nimport cv2\r\nimport "
},
{
"path": "docker/Dockerfile",
"chars": 1239,
"preview": "FROM anibali/pytorch:cuda-9.0\nMAINTAINER fangshancheng <fangsc@ustc.edu.cn>\nRUN sudo rm -rf /etc/apt/sources.list.d && \\"
},
{
"path": "export_onnx.py",
"chars": 2571,
"preview": "\"\"\" Created by MrBBS \"\"\"\r\n\r\nimport os\r\n\r\nimport torch\r\n\r\nfrom utils import Config\r\n\r\n\r\ndef get_model(config):\r\n impor"
},
{
"path": "losses.py",
"chars": 2592,
"preview": "from fastai.vision import *\n\nfrom modules.model import Model\n\n\nclass MultiLosses(nn.Module):\n def __init__(self, one_"
},
{
"path": "main.py",
"chars": 10726,
"preview": "import argparse\nimport logging\nimport os\nimport random\n\nimport torch\nfrom fastai.callbacks.general_sched import GeneralS"
},
{
"path": "modules/__init__.py",
"chars": 0,
"preview": ""
},
{
"path": "modules/attention.py",
"chars": 4086,
"preview": "import torch\nimport torch.nn as nn\nfrom .transformer import PositionalEncoding\n\nclass Attention(nn.Module):\n def __in"
},
{
"path": "modules/backbone.py",
"chars": 1545,
"preview": "from fastai.vision import *\n\nfrom modules.model import _default_tfmer_cfg\nfrom modules.resnet import resnet45\nfrom modul"
},
{
"path": "modules/backbone_v2.py",
"chars": 66,
"preview": "\"\"\" Created by MrBBS \"\"\"\r\n# 10/11/2022\r\n# -*-encoding:utf-8-*-\r\n\r\n"
},
{
"path": "modules/model.py",
"chars": 2011,
"preview": "import torch\nimport torch.nn as nn\nimport numpy as np\n\nfrom utils import CharsetMapper\n\n_default_tfmer_cfg = dict(d_mode"
},
{
"path": "modules/model_abinet.py",
"chars": 1059,
"preview": "from fastai.vision import *\n\nfrom .model_alignment import BaseAlignment\nfrom .model_language import BCNLanguage\nfrom .mo"
},
{
"path": "modules/model_abinet_iter.py",
"chars": 1378,
"preview": "import torch\nimport torch.nn.functional as F\nfrom fastai.vision import *\n\nfrom .model_vision import BaseVision\nfrom .mod"
},
{
"path": "modules/model_alignment.py",
"chars": 1248,
"preview": "import torch\nimport torch.nn as nn\nfrom fastai.vision import *\n\nfrom modules.model import Model, _default_tfmer_cfg\n\n\ncl"
},
{
"path": "modules/model_language.py",
"chars": 3186,
"preview": "import logging\nimport torch.nn as nn\nfrom fastai.vision import *\n\nfrom modules.model import _default_tfmer_cfg\nfrom modu"
},
{
"path": "modules/model_vision.py",
"chars": 1945,
"preview": "import logging\nimport torch.nn as nn\nfrom fastai.vision import *\n\nfrom modules.attention import *\nfrom modules.backbone "
},
{
"path": "modules/resnet.py",
"chars": 3263,
"preview": "import math\n\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport torch.utils.model_zoo as model_zoo\n\n\ndef conv1"
},
{
"path": "modules/transformer.py",
"chars": 43920,
"preview": "# pytorch 1.5.0\nimport copy\nimport math\nimport warnings\nfrom typing import Optional\n\nimport torch\nimport torch.nn as nn\n"
},
{
"path": "notebooks/dataset-text.ipynb",
"chars": 2875,
"preview": "{\n \"cells\": [\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": "
},
{
"path": "notebooks/dataset.ipynb",
"chars": 6547,
"preview": "{\n \"cells\": [\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": "
},
{
"path": "notebooks/prepare_wikitext103.ipynb",
"chars": 11518,
"preview": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"# 82841986 is_char and is_digit\"\n "
},
{
"path": "notebooks/transforms.ipynb",
"chars": 2674103,
"preview": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"# Preparation\"\n ]\n },\n {\n \"ce"
},
{
"path": "requirements.txt",
"chars": 85,
"preview": "torch==1.1.0\ntorchvision==0.3.0\nfastai==1.0.60\nLMDB\nPillow\nopencv-python\ntensorboardX"
},
{
"path": "tools/clean_dictionary.py",
"chars": 66,
"preview": "\"\"\" Created by MrBBS \"\"\"\r\n# 10/11/2022\r\n# -*-encoding:utf-8-*-\r\n\r\n"
},
{
"path": "tools/create_charset.py",
"chars": 269,
"preview": "chars = open(r'D:\\SourceCode\\bill_extract\\crawl_data\\char.txt', 'r', encoding='utf-8').read().strip().split('\\t')\r\n\r\nwit"
},
{
"path": "tools/create_language_data.py",
"chars": 3266,
"preview": "\"\"\" Created by MrBBS \"\"\"\r\n# 10/3/2022\r\n# -*-encoding:utf-8-*-\r\n\r\nimport os\r\nimport random\r\nimport numpy as np\r\n\r\nimport "
},
{
"path": "tools/create_lmdb_dataset.py",
"chars": 3751,
"preview": "\"\"\" a modified version of CRNN torch repository https://github.com/bgshih/crnn/blob/master/tool/create_dataset.py \"\"\"\n\ni"
},
{
"path": "tools/crop_by_word_bb_syn90k.py",
"chars": 6477,
"preview": "# Crop by word bounding box\n# Locate script with gt.mat\n# $ python crop_by_word_bb.py\n\nimport os\nimport re\nimport cv2\nim"
},
{
"path": "tools/get_max_len.py",
"chars": 179,
"preview": "\"\"\" Created by MrBBS \"\"\"\r\n# 10/7/2022\r\n# -*-encoding:utf-8-*-\r\n\r\nprint(max(open(r'E:\\NLP_data\\words_with_space.txt', 'r'"
},
{
"path": "tools/line2word.py",
"chars": 1226,
"preview": "\"\"\" Created by MrBBS \"\"\"\r\n# 10/4/2022\r\n# -*-encoding:utf-8-*-\r\n\r\nfrom tqdm import tqdm\r\nimport random\r\n\r\nlines = list(se"
},
{
"path": "transforms.py",
"chars": 13355,
"preview": "import math\nimport numbers\nimport random\n\nimport cv2\nimport numpy as np\nfrom PIL import Image\nfrom torchvision import tr"
},
{
"path": "utils.py",
"chars": 10313,
"preview": "import logging\nimport os\nimport time\n\nimport cv2\nimport numpy as np\nimport torch\nimport yaml\nfrom matplotlib import colo"
}
]
About this extraction
This page contains the full source code of the FangShancheng/ABINet GitHub repository, extracted and formatted as plain text for AI agents and large language models (LLMs). The extraction includes 44 files (2.7 MB), approximately 719.1k tokens, and a symbol index with 231 extracted functions, classes, methods, constants, and types. Use this with OpenClaw, Claude, ChatGPT, Cursor, Windsurf, or any other AI tool that accepts text input. You can copy the full output to your clipboard or download it as a .txt file.
Extracted by GitExtract — free GitHub repo to text converter for AI. Built by Nikandr Surkov.