Repository: ivanpanshin/SupCon-Framework
Branch: main
Commit: 4a934d4a19b7
Files: 25
Total size: 97.5 KB
Directory structure:
gitextract_cf8_gd7q/
├── .gitattributes
├── LICENSE.md
├── README.md
├── configs/
│ └── train/
│ ├── lr_finder_supcon_resnet18_cifar100_stage2.yml
│ ├── lr_finder_supcon_resnet18_cifar10_stage2.yml
│ ├── swa_supcon_resnet18_cifar100_stage1.yml
│ ├── swa_supcon_resnet18_cifar100_stage2.yml
│ ├── swa_supcon_resnet18_cifar10_stage1.yml
│ ├── swa_supcon_resnet18_cifar10_stage2.yml
│ ├── train_supcon_resnet18_cifar100_stage1.yml
│ ├── train_supcon_resnet18_cifar100_stage2.yml
│ ├── train_supcon_resnet18_cifar10_stage1.yml
│ └── train_supcon_resnet18_cifar10_stage2.yml
├── learning_rate_finder.py
├── requirements.txt
├── swa.py
├── t-SNE.ipynb
├── tools/
│ ├── backbones.py
│ ├── datasets.py
│ ├── losses.py
│ ├── models.py
│ ├── optimizers.py
│ ├── schedulers.py
│ └── utils.py
└── train.py
================================================
FILE CONTENTS
================================================
================================================
FILE: .gitattributes
================================================
*.ipynb linguist-vendored
================================================
FILE: LICENSE.md
================================================
MIT License
Copyright (c) 2022 Ivan Panshin
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
================================================
FILE: README.md
================================================
# SupCon-Framework
The repo is an implementation of [Supervised Contrastive Learning](https://arxiv.org/abs/2004.11362). It's based on another [implementation](https://github.com/HobbitLong/SupContrast), but with several differencies:
- Fixed bugs (incorrect ResNet implementations, which leads to a very small max batch size),
- Offers a lot of additional functionality (first of all, rich validation).
To be more precise, in this implementations you will find:
- Augmentations with [albumentations](https://github.com/albumentations-team/albumentations)
- Hyperparameters are moved to .yml configs
- [t-SNE](https://github.com/DmitryUlyanov/Multicore-TSNE) visualizations
- 2-step validation (for features before and after the projection head) using metrics like AMI, NMI, mAP, precision_at_1, etc with [PyTorch Metric Learning](https://github.com/KevinMusgrave/pytorch-metric-learning).
- [Exponential Moving Average](https://github.com/fadel/pytorch_ema) for a more stable training, and Stochastic Moving Average for a better generalization and just overall performance.
- Automatic Mixed Precision (torch version) training in order to be able to train with a bigger batch size (roughly by a factor of 2).
- LabelSmoothing loss, and [LRFinder](https://github.com/davidtvs/pytorch-lr-finder) for the second stage of the training (FC).
- TensorBoard logs, checkpoints
- Support of [timm models](https://github.com/rwightman/pytorch-image-models), and [pytorch-optimizer](https://github.com/jettify/pytorch-optimizer)
## Install
1. Clone the repo:
```
git clone https://github.com/ivanpanshin/SupCon-Framework && cd SupCon-Framework/
```
2. Create a clean virtual environment
```
python3 -m venv venv
source venv/bin/activate
```
3. Install dependencies
````
python -m pip install --upgrade pip
pip install -r requirements.txt
````
## Training
In order to execute Cifar10 training run:
```
python train.py --config_name configs/train/train_supcon_resnet18_cifar10_stage1.yml
python swa.py --config_name configs/train/swa_supcon_resnet18_cifar10_stage1.yml
python train.py --config_name configs/train/train_supcon_resnet18_cifar10_stage2.yml
python swa.py --config_name configs/train/swa_supcon_resnet18_cifar10_stage2.yml
```
In order to run LRFinder on the second stage of the training, run:
```
python learning_rate_finder.py --config_name configs/train/lr_finder_supcon_resnet18_cifar10_stage2.yml
```
The process of training Cifar100 is exactly the same, just change config names from cifar10 to cifar100.
After that you can check the results of the training either in `logs` or `runs` directory. For example, in order to check tensorboard logs for the first stage of Cifar10 training, run:
```
tensorboard --logdir runs/supcon_first_stage_cifar10
```
## Visualizations
This repo is supplied with t-SNE visualizations so that you can check embeddings you get after the training. Check `t-SNE.ipynb` for details.
Those are t-SNE visualizations for Cifar10 for validation and train with SupCon (top), and validation and train with CE (bottom).
Those are t-SNE visualizations for Cifar100 for validation and train with SupCon (top), and validation and train with CE (bottom).
## Results
| Model | Stage | Dataset | Accuracy |
| ------------- | ------------- | ------------- | ------------- |
| ResNet18 | Frist | CIFAR10 | 95.9 |
| ResNet18 | Second | CIFAR10 | 94.9 |
| ResNet18 | Frist | CIFAR100 | 79.0 |
| ResNet18 | Second | CIFAR100 | 77.9 |
Note that even though the accuracy on the second stage is lower, it's not always the case. In my experience, the difference between stages is usually around 1 percent, including the difference that favors the second stage.
Training time for the whole pipeline (without any early stopping) on CIFAR10 or CIFAR100 is around 4 hours (single 2080Ti with AMP). However, with reasonable early stopping that value goes down to around 2.5-3 hours.
## Custom datasets
It's fairly easy to adapt this pipeline to custom datasets. First, you need to check `tools/datasets.py` for that. Second, add a new class for your dataset. The only guideline here is to follow the same augmentation logic, that is
```
if self.second_stage:
image = self.transform(image=image)['image']
else:
image = self.transform(image)
```
Third, add your dataset to `DATASETS` dict still inside `tools/datasets.py`, and you're good to go.
## FAQ
- Q: What hyperparameters should I try to change?
A: First of all, learning rate. Second of all, try to change the augmentation policy. SupCon is built around "cropping + color jittering" scheme, so you can try changing the cropping size or the intensity of jittering. Check `tools.utils.build_transforms` for that.
- Q: What backbone and batch size should I use?
A: This is quite simple. Take the biggest backbone that you can, and after that take the highest batch size your GPU can offer. The reason for that: SupCon is more prone (than regular classification training with CE/LabelSmoothing/etc) to improving with stronger backbones. Moverover, it has a property of explicit hard positive and negative mining. It means that the higher the batch size - the more difficult and helpful samples you supply to the model.
- Q: Do I need the second stage of the training?
A: Not necessarily. You can do classification based only on embeddings. In order to do that compute embeddings for the train set, and at inference time do the following: take a sample, compute its embedding, take the closest one from the training, take its class. To make this fast and efficient, you something like [faiss](https://github.com/facebookresearch/faiss) for similarity search. Note that this is actually how validation is done in this repo. Moveover, during training you will see a metric `precision_at_1`. This is actually just an accuracy based solely on embeddings.
- Q: Should I use AMP?
A: If your GPU has tensor cores (like 2080Ti) - yes. If it doesn't (like 1080Ti) - check the speed with AMP and without. If the speed dropped slightly (or even increased by a bit) - use it, since SupCon works better with bigger batch sizes.
- Q: How should I use EMA?
A: You only need to choose the `ema_decay_per_epoch` parameter in the config. The heuristic is fairly simple. If your dataset is big, then something as small as 0.3 will do just fine. And as your dataset gets smaller, you can increase `ema_decay_per_epoch`. Thanks to bonlime for this idea. I advice you to check his great [pytorch tools](https://github.com/bonlime/pytorch-tools) repo, it's a hidden gem.
- Q: Is it better than training with Cross Entropy/Label Smoothing/etc?
A: Unfortunately, in my experience, it's much easier to get better results with something like CE. It's more stable, faster to train, and simply produces better or the same results. For instance, in case on CIFAR10/100 it's trivial to train ResNet18 up tp 96/81 percent respectively. Of cource, I've seen cases where SupCon performs better, but it takes quite a bit of work to make it outperform CE.
- Q: How long should I train with SupCon?
A: The answer is tricky. On one hand, authors of the original paper claim that the longer you train with SupCon, the better it gets. However, I did not observe such a behavior in my tests. So the only recommendation I can give is the following: start with 100 epochs for easy datasets (like CIFAR10/100), and 1000 for more industrial ones. Then - monitor the training process. If the validaton metric (such as `precision_at_1`) doesn't impove for several dozens of epochs - you can stop the training. You might incorporate early stopping for this reason into the pipeline.
================================================
FILE: configs/train/lr_finder_supcon_resnet18_cifar100_stage2.yml
================================================
model:
backbone: resnet18
ckpt_pretrained:
num_classes: 100
dataset: data/cifar100
dataloaders:
train_batch_size: 200 # the higher - the better
valid_batch_size: 200
num_workers: 16 # set this to num of threads in your CPU
optimizer:
name: SGD
params:
lr: 0.001
criterion:
name: 'CrossEntropy'
================================================
FILE: configs/train/lr_finder_supcon_resnet18_cifar10_stage2.yml
================================================
model:
backbone: resnet18
ckpt_pretrained:
num_classes: 10
dataset: data/cifar10
dataloaders:
train_batch_size: 200 # the higher - the better
valid_batch_size: 200
num_workers: 16 # set this to num of threads in your CPU
optimizer:
name: SGD
params:
lr: 0.001
criterion:
name: 'CrossEntropy'
================================================
FILE: configs/train/swa_supcon_resnet18_cifar100_stage1.yml
================================================
model:
backbone: resnet18
num_classes:
top_k_checkoints: 3
train:
amp: True # set this to True, if your GPU supports FP16. 2080Ti - okay, 1080Ti - not okay
weights_dir: weights/supcon_first_stage_cifar100
stage: first
dataset: data/cifar100
dataloaders:
train_batch_size: 200 # the higher - the better
valid_batch_size: 200
num_workers: 12 # set this to num of threads in your CPU
================================================
FILE: configs/train/swa_supcon_resnet18_cifar100_stage2.yml
================================================
model:
backbone: resnet18
num_classes: 100
top_k_checkoints: 3
train:
amp: True # set this to True, if your GPU supports FP16. 2080Ti - okay, 1080Ti - not okay
weights_dir: weights/supcon_second_stage_cifar100
stage: second
dataset: data/cifar100
dataloaders:
train_batch_size: 200 # the higher - the better
valid_batch_size: 200
num_workers: 12 # set this to num of threads in your CPU
================================================
FILE: configs/train/swa_supcon_resnet18_cifar10_stage1.yml
================================================
model:
backbone: resnet18
num_classes:
top_k_checkoints: 3
train:
amp: True # set this to True, if your GPU supports FP16. 2080Ti - okay, 1080Ti - not okay
weights_dir: weights/supcon_first_stage_cifar10
stage: first
dataset: data/cifar10
dataloaders:
train_batch_size: 200 # the higher - the better
valid_batch_size: 200
num_workers: 12 # set this to num of threads in your CPU
================================================
FILE: configs/train/swa_supcon_resnet18_cifar10_stage2.yml
================================================
model:
backbone: resnet18
num_classes: 10
top_k_checkoints: 3
train:
amp: True # set this to True, if your GPU supports FP16. 2080Ti - okay, 1080Ti - not okay
weights_dir: weights/supcon_second_stage_cifar10
stage: second
dataset: data/cifar10
dataloaders:
train_batch_size: 200 # the higher - the better
valid_batch_size: 200
num_workers: 12 # set this to num of threads in your CPU
================================================
FILE: configs/train/train_supcon_resnet18_cifar100_stage1.yml
================================================
model:
backbone: resnet18
ckpt_pretrained:
num_classes: # in the first stage of training we don't need num_classes, since we don't have a FC head
train:
n_epochs: 100
amp: True # set this to True, if your GPU supports FP16. 2080Ti - okay, 1080Ti - not okay
ema: True # optional, but I recommend it, since the training might get unstable otherwise
ema_decay_per_epoch: 0.3 # 0.3 for middle/big datasets. Increase, if you have low amount of samples
logging_name: supcon_first_stage_cifar100
target_metric: precision_at_1
stage: first # first = Supcon, second = FC finetuning for classification
dataset: data/cifar100
dataloaders:
train_batch_size: 200 # the higher - the better
valid_batch_size: 200
num_workers: 12 # set this to num of threads in your CPU
optimizer:
name: SGD
params:
lr: 0.1
scheduler:
name: CosineAnnealingLR
params:
T_max: 100
eta_min: 0.01
criterion:
name: 'SupCon'
params:
temperature: 0.1
================================================
FILE: configs/train/train_supcon_resnet18_cifar100_stage2.yml
================================================
model:
backbone: resnet18
ckpt_pretrained: weights/supcon_first_stage_cifar100/swa
num_classes: 100
train:
n_epochs: 20
amp: True # set this to True, if your GPU supports FP16. 2080Ti - okay, 1080Ti - not okay
ema: True # optional, but I recommend it, since the training might get unstable otherwise
ema_decay_per_epoch: 0.3 # for middle/big datasets. Increase, if you have low amount of samples
logging_name: supcon_second_stage_cifar100
target_metric: accuracy
stage: second # first = Supcon, second = FC finetuning for classification
dataset: data/cifar100
dataloaders:
train_batch_size: 20 # the higher - the better
valid_batch_size: 20
num_workers: 12 # set this to num of threads in your CPU
optimizer:
name: SGD
params:
lr: 0.01
scheduler:
name: CosineAnnealingLR
params:
T_max: 20
eta_min: 0.001
criterion:
name: 'LabelSmoothing'
params:
classes: 100
smoothing: 0.01
================================================
FILE: configs/train/train_supcon_resnet18_cifar10_stage1.yml
================================================
model:
backbone: resnet18
ckpt_pretrained:
num_classes: # in the first stage of training we don't need num_classes, since we don't have a FC head
train:
n_epochs: 100
amp: True # set this to True, if your GPU supports FP16. 2080Ti - okay, 1080Ti - not okay
ema: True # optional, but I recommend it, since the training might get unstable otherwise
ema_decay_per_epoch: 0.3 # 0.3 for middle/big datasets. Increase, if you have low amount of samples
logging_name: supcon_first_stage_cifar10
target_metric: precision_at_1
stage: first # first = Supcon, second = FC finetuning for classification
dataset: data/cifar10
dataloaders:
train_batch_size: 200 # the higher - the better
valid_batch_size: 200
num_workers: 12 # set this to num of threads in your CPU
optimizer:
name: SGD
params:
lr: 0.1
scheduler:
name: CosineAnnealingLR
params:
T_max: 100
eta_min: 0.01
criterion:
name: 'SupCon'
params:
temperature: 0.1
================================================
FILE: configs/train/train_supcon_resnet18_cifar10_stage2.yml
================================================
model:
backbone: resnet18
ckpt_pretrained: weights/supcon_first_stage_cifar10/swa
num_classes: 10
train:
n_epochs: 20
amp: True # set this to True, if your GPU supports FP16. 2080Ti - okay, 1080Ti - not okay
ema: True # optional, but I recommend it, since the training might get unstable otherwise
ema_decay_per_epoch: 0.3 # for middle/big datasets. Increase, if you have low amount of samples
logging_name: supcon_second_stage_cifar10
target_metric: accuracy
stage: second # first = Supcon, second = FC finetuning for classification
dataset: data/cifar10
dataloaders:
train_batch_size: 20 # the higher - the better
valid_batch_size: 20
num_workers: 12 # set this to num of threads in your CPU
optimizer:
name: SGD
params:
lr: 0.01
scheduler:
name: CosineAnnealingLR
params:
T_max: 20
eta_min: 0.001
criterion:
name: 'LabelSmoothing'
params:
classes: 10
smoothing: 0.01
================================================
FILE: learning_rate_finder.py
================================================
import argparse
import os
import matplotlib.pyplot as plt
import yaml
from torch_lr_finder import LRFinder
from tools import utils
def parse_config():
parser = argparse.ArgumentParser()
parser.add_argument(
"--config_name",
type=str,
default="configs/train/lr_finder_supcon_resnet18_cifar10_stage2.yml",
)
parser_args = parser.parse_args()
with open(vars(parser_args)["config_name"], "r") as config_file:
hyperparams = yaml.full_load(config_file)
return hyperparams
if __name__ == "__main__":
os.makedirs("lr_finder_plots", exist_ok=True)
hyperparams = parse_config()
backbone = hyperparams["model"]["backbone"]
ckpt_pretrained = hyperparams["model"]["ckpt_pretrained"]
num_classes = hyperparams['model']['num_classes']
optimizer_params = hyperparams["optimizer"]
scheduler_params = None
criterion_params = hyperparams["criterion"]
data_dir = hyperparams["dataset"]
batch_sizes = {
"train_batch_size": hyperparams["dataloaders"]["train_batch_size"],
"valid_batch_size": hyperparams["dataloaders"]["valid_batch_size"]
}
num_workers = hyperparams["dataloaders"]["num_workers"]
transforms = utils.build_transforms(second_stage=True)
loaders = utils.build_loaders(data_dir, transforms, batch_sizes, num_workers, second_stage=True)
model = utils.build_model(backbone, second_stage=True, num_classes=num_classes,
ckpt_pretrained=ckpt_pretrained).cuda()
optim = utils.build_optim(model, optimizer_params, scheduler_params, criterion_params)
criterion, optimizer, scheduler = (
optim["criterion"],
optim["optimizer"],
optim["scheduler"],
)
lr_finder = LRFinder(model, optimizer, criterion, device="cuda")
lr_finder.range_test(loaders["train_features_loader"], end_lr=1, num_iter=300)
fig, ax = plt.subplots()
lr_finder.plot(ax=ax)
fig.savefig(
"lr_finder_plots/supcon_{}_{}_bs_{}_stage_{}_lr_finder.png".format(
optimizer_params["name"],
data_dir.split("/")[-1],
batch_sizes["train_batch_size"],
'second'
)
)
================================================
FILE: requirements.txt
================================================
absl-py==0.11.0
albumentations==0.5.2
argon2-cffi==20.1.0
async-generator==1.10
attrs==20.3.0
backcall==0.2.0
bleach==3.2.1
cachetools==4.2.0
certifi==2020.12.5
cffi==1.14.4
chardet==4.0.0
cycler==0.10.0
dataclasses==0.8
decorator==4.4.2
defusedxml==0.6.0
entrypoints==0.3
faiss-gpu==1.6.5
google-auth==1.24.0
google-auth-oauthlib==0.4.2
grpcio==1.34.0
idna==2.10
imageio==2.9.0
imgaug==0.4.0
importlib-metadata==3.3.0
ipykernel==5.4.3
ipython==7.16.1
ipython-genutils==0.2.0
ipywidgets==7.6.3
jedi==0.18.0
Jinja2==2.11.2
joblib==1.0.0
jsonschema==3.2.0
jupyter==1.0.0
jupyter-client==6.1.11
jupyter-console==6.2.0
jupyter-core==4.7.0
jupyterlab-pygments==0.1.2
jupyterlab-widgets==1.0.0
kiwisolver==1.3.1
Markdown==3.3.3
MarkupSafe==1.1.1
matplotlib==3.3.3
mistune==0.8.4
MulticoreTSNE==0.1
nbclient==0.5.1
nbconvert==6.0.7
nbformat==5.0.8
nest-asyncio==1.4.3
networkx==2.5
notebook==6.1.6
numpy==1.19.4
oauthlib==3.1.0
opencv-python==4.4.0.46
opencv-python-headless==4.4.0.46
packaging==20.8
pandas==1.1.5
pandocfilters==1.4.3
parso==0.8.1
pexpect==4.8.0
pickleshare==0.7.5
Pillow==8.0.1
prometheus-client==0.9.0
prompt-toolkit==3.0.10
protobuf==3.14.0
ptyprocess==0.7.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycparser==2.20
Pygments==2.7.4
pyparsing==2.4.7
pyrsistent==0.17.3
python-dateutil==2.8.1
pytorch-metric-learning==0.9.95
pytorch-ranger==0.1.1
pytz==2020.4
PyWavelets==1.1.1
PyYAML==5.3.1
pyzmq==20.0.0
qtconsole==5.0.1
QtPy==1.9.0
requests==2.25.1
requests-oauthlib==1.3.0
rsa==4.6
scikit-image==0.17.2
scikit-learn==0.23.2
scipy==1.5.4
Send2Trash==1.5.0
Shapely==1.7.1
six==1.15.0
tensorboard==2.4.0
tensorboard-plugin-wit==1.7.0
terminado==0.9.2
testpath==0.4.4
threadpoolctl==2.1.0
tifffile==2020.9.3
timm==0.3.4
torch==1.7.1
torch-ema @ git+https://github.com/fadel/pytorch_ema@27afe25b9fb9f0d05a87ae94e4e4ad9e92d70a85
torch-lr-finder==0.2.1
torch-optimizer==0.0.1a17
torchvision==0.8.2
tornado==6.1
tqdm==4.54.1
traitlets==4.3.3
typing-extensions==3.7.4.3
urllib3==1.26.2
wcwidth==0.2.5
webencodings==0.5.1
Werkzeug==1.0.1
widgetsnbextension==3.5.1
zipp==3.4.0
================================================
FILE: swa.py
================================================
import os
from collections import OrderedDict
import torch
import argparse
import yaml
from tools import utils
scaler = torch.cuda.amp.GradScaler()
def swa(paths):
state_dicts = []
for path in paths:
state_dicts.append(torch.load(path)["model_state_dict"])
average_dict = OrderedDict()
for k in state_dicts[0].keys():
average_dict[k] = sum([state_dict[k] for state_dict in state_dicts]) / len(state_dicts)
return average_dict
def parse_config():
parser = argparse.ArgumentParser()
parser.add_argument(
"--config_name",
type=str,
default="configs/train/swa_supcon_resnet18_cifar10_stage2.yml"
)
parser_args = parser.parse_args()
with open(vars(parser_args)["config_name"], "r") as config_file:
hyperparams = yaml.full_load(config_file)
return hyperparams
if __name__ == "__main__":
hyperparams = parse_config()
backbone = hyperparams["model"]["backbone"]
num_classes = hyperparams['model']['num_classes']
top_k_checkoints = hyperparams['model']['top_k_checkoints']
amp = hyperparams['train']['amp']
weights_dir = hyperparams['train']['weights_dir']
stage = hyperparams['train']['stage']
data_dir = hyperparams["dataset"]
batch_sizes = {
"train_batch_size": hyperparams["dataloaders"]["train_batch_size"],
'valid_batch_size': hyperparams['dataloaders']['valid_batch_size']
}
num_workers = hyperparams["dataloaders"]["num_workers"]
if not amp: scaler = None
utils.seed_everything()
if os.path.exists(os.path.join(weights_dir, "swa")):
os.remove(os.path.join(weights_dir, "swa"))
transforms = utils.build_transforms(second_stage=(stage == 'second'))
loaders = utils.build_loaders(data_dir, transforms, batch_sizes, num_workers, second_stage=(stage == 'second'))
model = utils.build_model(backbone, second_stage=(stage == 'second'), num_classes=num_classes, ckpt_pretrained=None).cuda()
list_of_epochs = sorted([int(x.split('epoch')[1]) for x in os.listdir(weights_dir)])
best_epochs = list_of_epochs[-top_k_checkoints::]
model_prefix = "epoch"
checkpoints_paths = ["{}/{}{}".format(weights_dir, model_prefix, epoch) for epoch in best_epochs]
average_dict = swa(checkpoints_paths)
torch.save({"model_state_dict": average_dict}, os.path.join(weights_dir, "swa"))
model.load_state_dict(torch.load(os.path.join(weights_dir, "swa"))['model_state_dict'])
if stage == 'first':
valid_metrics = utils.validation_constructive(loaders['valid_loader'], loaders['train_features_loader'], model,
scaler)
else:
valid_metrics = utils.validation_ce(model, None, loaders['valid_loader'], scaler)
print('swa stage {} validation metrics: {}'.format(stage, valid_metrics))
================================================
FILE: t-SNE.ipynb
================================================
{
"cells": [
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"from MulticoreTSNE import MulticoreTSNE as TSNE\n",
"from matplotlib import pyplot as plt\n",
"from tools import utils\n",
"import torch\n",
"import torch.functional as F\n",
"\n",
"scaler = torch.cuda.amp.GradScaler()"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Files already downloaded and verified\n",
"Files already downloaded and verified\n",
"Files already downloaded and verified\n"
]
}
],
"source": [
"ckpt_pretrained = 'weights/supcon_first_stage_cifar10/swa'\n",
"data_dir = 'data/cifar10'\n",
"num_classes = 10\n",
"batch_sizes = {\n",
" \"train_batch_size\": 20,\n",
" 'valid_batch_size': 20\n",
"}\n",
"num_workers = 16\n",
"backbone = 'resnet18'\n",
"stage = 'first'\n",
"\n",
"transforms = utils.build_transforms(second_stage=(stage == 'second'))\n",
"loaders = utils.build_loaders(data_dir, transforms, batch_sizes, num_workers, second_stage=(stage == 'second'))\n",
" "
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"pycharm": {
"name": "#%%\n"
}
},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAV8AAAD8CAYAAADQSqd1AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAACBe0lEQVR4nOydd3gU1deA3zuzLb0TAgkQeu9SBKRjo9gFu6JYULGLXVF/9u5nAeyiIlZEVEAERHrvvQcI6T3b5n5/zKb3ZFOAeZ9nn+zO3HJ2s3vmzrmnCCklBgYGBgZ1i1LfAhgYGBicjRjK18DAwKAeMJSvgYGBQT1gKF8DAwODesBQvgYGBgb1gKF8DQwMDOoBQ/kaGBgYeAEhxBQhxDYhxHYhxH0VtTeUr4GBgUENEUJ0Bm4D+gDdgNFCiNbl9TGUr4GBgUHN6QCsllJmSyldwFLgsvI6mOpErEoSHh4uW7RoUd9iGBgYnAasX78+UUoZUZMxWgshsyvZ9gRsB3ILHZoupZzueb4NeFEIEQbkABcB68obz2vKVwiheiaLk1KOFkLEAt8BYcB64HoppaO8MVq0aMG6deXKa2BgYACAEOJwTcfIBm6vZNtnIVdK2bu0c1LKnUKIV4AFQBawCXCXN543zQ5TgJ2FXr8CvCWlbA2kABO9OJeBgYFBg0JK+YmUspeU8jx0nbenvPZeUb5CiGjgYmCm57UAhgE/eJp8AVzijbkMDAwMGiJCiEaev83Q7b3flNfeW2aHt4FHgADP6zAg1WN4BjgGNC2toxBiEjAJoFmzZl4Sx8DAwKDO+dFj83UCk6WUqeU1rrHyFUKMBk5JKdcLIYZUtb/HYD0doHfv3kZ+SwMDg9MSKeWgqrT3xsp3ADBWCHERYAMCgXeAYCGEybP6jQbivDCXgYGBwRlBjW2+UsrHpJTRUsoWwHhgsZTyWuAf4ApPsxuBX2s6l4GBgcGZQm0GWTwKPCCE2IduA/6kFucyMPAq8WxhEY+wglewk17f4hicgXg1yEJKuQRY4nl+AD3UzsDgtMJOBkt5BtBIBnJIZjiv1LdYBmcYDSrCzcCgPtBwsYHpnGQj/kTRguGAln8+mb31J5zBGYuhfA3Oatw4+Y2JOEgDIJtTnGJbkTYSNxuYjhs7nZhANgkksJ1IuhFKm/oQ2+AMwFC+Bmcdx1jLSl5G4kbBilYkXB8Kr3rz2MfvABxkEfpWicZWZtGDW4mmHz6E1brcBmcWRlYzg7MKDTcreAGJC5ClKF6AitzNtfy/G5nOb9zCX9yHg0zvCmtwRmMoX4MznjUHYM46SM+BbBJrZY40DrKd2bUytsGZiWF2MDij+XQ5TP01nejGh1ipLWJ4lwzwqZ259jIXB1n04R4EonYmMThjMJSvwRnHHuaxj98IIpY/Do/kjRdexmLRs5lmAtItUNTaiWQ/zGLacDGhtKqV8Q3OHAyzg8EZxSEWs4kZZHKSOFYy+tI3sFodCEHBQ6nNFCKSxTxKLqm1OIfBmYChfA3OKI6wvMhrH58sZF2la/LM45ZOfnd8WUeTGpyuGMrX4IxAItnDXOwef918SjG9itoyxxYad36GXy1NYnCmYNh8Dc4IDsg/2MpXuEXJSlW1pmxLIW+VHWM9hEQaG28GZWKsfA1Oe6Q7i2PZrxYo3jqzM5Qkz67c0W9brbm1GTRMhBD3CyG2CyG2CSG+FULYymtvKF+D054dGfNI2+eHw2HGYTcjtPpRvkV0vgBrfmEXgzMdIURT4F6gt5SyM6Cip9gtE8PsYHDak+FozPiP5hAYnEazpgfxVzK5467/q7P5pSxp2rAQgIlyFz4GDYAmJngmuHJtn634RsYE+AghnIAvcLyixgYGpzUtrINxai7iE6KIT4jihccfqsPZBVaCyZVpgERIFVWoxHAuLnKZx0QcZKJgYTQzsBFch7IZ1BVSyjghxOvAESAHWCClXFBeH8PsYFBjlnCId1nDOk7Uy/yNg+DaviZsZg2L2UmLmIN1NrdAYYh4lsvFLC4Q7xIuWhNCa9owhiU8nZ/vQcPBAu6vM7kMaoVwIcS6Qo9JeSeEECHAOCAWaAL4CSGuK28wbxTQtAHLAKtnvB+klM8IIWKB79CrWKwHrpdSltyKNjit+Y+jfMB67LhZzlGeZwjt6iHD12c3w2MXu9lgewbV7Kq4g5eQuDnJRkJoxXL+h5scABYwBTO+RdoaiXdOexKllL3LODcCOCilTAAQQvwEnAt8XdZg3lj52oFhUspuQHfgAiFEP+AV4C0pZWsgBZjohbkMGhi7ScKOG9BjDA6QUi9yCAHhkQcxBe2u87m38DlLeSpf8YKeoL2ZHIkmBVKCJgXNtavqXDaDOuMI0E8I4SuEEMBwYGd5HbxRQFNKKfMu6WbPQwLDgB88x78ALqnpXAYNjwHEYEXFhgkVQQ8a55/LznayevUxEhOz60SWDI55UkU2DLZldSbbrWfxyXT78196p3qWyKC2kFKuRtd3G4Ct6Lp1enl9vLLhJoRQ0U0LrYH/A/YDqZ6y8QDHgKbemMugbvieHfzOXhrhxzV0phuRKKUEDLQjjDcZyQ4S2UI8b7Oaq+lIi5Rgeg+fgc8YM84UN9/dejk9ukbVqsxZFfrVCirO1estJCbfN7Eq+oUn0JRBQNDj7OJG2nGJEXxxBiKlfAZ4prLtvaJ8pZRuoLsQIhj4GWhf2b4eo/UkgGbNmnlDHIMasockfmAndtykYuc5luWrLBsmHmcAXWnE3xxiC/F0phHrOc4qj2fNs/xL7MkgOq6MRJhBc0he3bCSb7kMiSQXFzZMXldAzRnCDr5DeswgJalb/19FpBdxQxNCspWv8SeKaPrVqSwGDQ9vVy9OFUL8A/QHgoUQJs/qNxqIK6PPdDzL8969e9dfaNJZTi4ufmYXJ8liNXH5dlwoqrJycTGNfzmXJizjGABLOVJivIPt0xCAEALVJnB10dhOAk+xBDeSMHz4gAuxefEr6E8jxvApB/mb3e7pOBULUiilO+LWAUKAwIKUjvy8DxIXm/iEJvRGMTw9z2pqbPMVQkR4VrwIIXyAkeiG5n+AKzzNbgR+relcBt4lnix2kIATN6+xkh/ZxRIOk1OB3dSFlq94y0ToihcACWF+vjzpUbwAiVoO49Lmc/nJhfwkd2H3kq3WRjAduJyhyWuwanZUzYWPu7RSQXWDxIFNhBc5ZieNRHbVk0QGDQVvXHqjgC88dl8F+F5KOU8IsQP4TgjxArAR+MQLcxl4iTUc5zVWIgA3Gi5v35JLCrJ8CTgh9D1ZoWm0OHEI/+wMtrbuijMwly9I4Xt28CXjsKB6YW5JoCuTi0/9TY5qw+q283PUxTUft5rklrBFS2wE1YssBg2HGitfKeUWoEcpxw8AfWo6vkHt8BO7cJRpG60mhZIbhKYlkRwUXjSlo5RIReFgk1hUt17AMm91nIOLPSTRmUY1l0MIhN9UlKzX8HXnsN2/bc3H9BIKZnpwG4HE1LcoBvWMEeF2lhJNAKba2HH3pPXK9vGjxAZXwc4TbtWEKJZ9rDH+3pMj8HlExA6SQ/+Pnf7tdFEawI7CCF6jJaPqWwyDBoChfM8yMrBzO7+zkIPeNzXk23gluWZr6ZtceQpXCKTQndfMKNxJT8KLRYTVGFNLjint9Xcp9IcmlXwRyss8adfMHM5tjkt6wQyCLsAAHiOYWC+NZ3C6Y2y3niVIJK+xkv8q2ijzBuV5FhQ+J3RnsxlcTEgtlRT+07GIluaC1wdyWvJ9wjUcs0dzS+MZ9ApYW0JcKWFZ6lA+PXkHZuHgoZgX6eq/tYaSCI7wLz6EEUqbGo5lcCZgrHzPElZzvG4Ubx6VdO3SQPcj1tzg9n502lFnI9ya/lxKWJRwJzuyu9AnYDWd/beUKeaviZehoWKXPrx5bKoX8rNrHGU5i3mC1fI9NsuvcFI3kX8GDRND+Z4lpFN/7lYVcVCm8tqOh9Ge9yFu5Qs13giUSOLZxBH+xZ5zLTuze5DkiGBb2m0EqiZc0kRnvy3YFHvxjkgJK9LOJdHVKP/gXU3erZE8UGDi+C+tF4N23U7vXeN5ItXwvjybMZTvWYAbjRV1ueqtKkKwum0PPh85nvcbu3gs+VOcNVDA2/mW5bzEOt7nysYvMICnaeWYyWNBAxka8Q6+ShYr0wfg0Mwlqk8A9AlYw4DAZSi4iDIfo41tV2FzNqcSIsi1W5AS3O6yf0LFV8vpGf58fOJunNKCS5p568TlJMsD1X6fBqc3hvI9C/iF3Wwkvr7FKJewjGT+OGckO2I7csTfxj+Za6s91iEW4yYXF7lkijgG+icxyh+yxHFirCd4reUUjtqb8+qRJ1iZfm4JJWlWXdzV9B0eiXmBlj77eevYo6Q6db/chKRGPPD0+3w66w7m/nkpb330cJlyuKSKlAVK+Lufr0MVBRcVFY14NlX7fRqc3hgbbmcYKVngcEGkx4f/e3bwHdvrV6hKkBDcCM2zvBRIVqSvJsK/OSoqXYioUh6IUNqSSyoaLlRs+dUjQmiJGT9+ThzPMXsz3JjYk9OeGOthYmxFo98VJF38NtHBdwdmxYkqdMOxzZoLKCxdMRwhNNq32VFifinBjcrME3fyX/pg7mv6CvGrGhGX3JM7oj5g+ok70FCYHPU+4WJc9T4wg9MeY+V7BvHqikzCH3TT5FE3Y39MYCMnmMW2/JDehkjeqlBTFH2TTkrsJgtbIsL5n3sJL7gW8t7JmZCdXOkx+zCF9lxOS85nOK+iors7mLAxgBdwSBuaR5lfFTGLaGuB4i28AWdSJDbVjoKWfywwIJ07bnyXsJAEWjbfx123vFNifiHAJNxcEDofp7Tw6ck7GDn0bw4MieStYw/SN2Atc1vN4f7Ai4jASDN5tmKsfM8gnvnejObS/VLn/RWGNvZHVHMFnRoAUqKvbPPDkQVuzwNUlob7cs/r7RFT9oBPcIXjmbDSmQlFji3kAP9yBMkCLotIYGtWB2xKDheH/VZE4RZyQy5ijiicm2dgv38Z2O/fcmVwSZUoyzHeaHUX38bfAKrAYQEwsyp9JARBpPHrq38igFsr2fZ5705t/PvPIKz+DnIzLYBAMbsRSsNd8eYhBGhuUULW4MxU0n0DAUmTpBPgyICjK6HthZUeW0Mylz2s4Si7SMKNQKUzfS2r+aDNRJxSLeFqVswNucrkKWwVNybVTRMljinRr+Pv7o+vAEVArBn6+7hJYDe+ROBHRNUnMjjtMZTvGYB95098r+zl1rujmb+hE7HD9qNa3IjT1agkodWxAwTkZqBqGtf9PRsAe0QTJGmVTkrzM7v4Vm7BKUR+oh83KikEE6b5sze3LT18N6II712kiq+YFQFmnBzMiWVbSzjlhm4+LhaIieSSCkA3bqadUejltEYI0Q6YXehQS+BpKeXbZfUxlO/pzr4FvOtezurWPXGpKm0u3F0vuWurQ/4tfvEVuoD1HXrRbe8mblz+PX5R3Tg6aBRrQqYh0YjmXPryAEoFGdA2uLfSTN1PK/axWAxDemKM97pb897+h3FoFmIsh3k85jUCzd7zBhFCEChbkSb3AXDU3oypx8Yy0wLrW0KiWJuveAG28Y2hfE9zpJS70WtY5lX2iUMvLFEmhvI93Ylbx74usTjNFv11zUOx6oyKrhGb23TngTbd6UUUHXmH44TxL4Nwo3IhM7mN28vt3yNpNY6IA5iExlg5l2QtBBTBrMzrsWtW3JqZg7ltWHTqFS5vOhFNuislV8UoNJOXMmxfa2xqFkfssUgUjmRn8u+nV9K1W2s4p6C1qZZCqw3qjeHAfinl4fIana43pgZ5tBvNqA3LsDpysdlzPKkazyzWc4IMQlnBubjdKjf+MYvBH79G9vo3yu3Xw6cLFpcTAB9hp0XGETZndsfPkg1ScF7Q3zzd/An6hD2BJt15CdkqpOLrm5ttymu82uoucjQ/pOdn5kbQOmkXYX9+SOv0nggUzPgxmGcrntSgIRAuhFhX6DGpjHbjgW8rGsxY+Z7uNO7K5aYXaX9oNb8EtGNNWCLILGojW2R9oeLkMMF0YxPR609y/tpF2FwOXKceZWejeDrEvFqijxsHCQHJmLJcOBWB4tbo88sGhsf/y3uTbqdzzFH6+87HojiLuDLkPS3s9VAYKcEtFUxCoyJsqpu3W93N0pM/sO3Ich7b+D9isw7htKh0Tx1Gz8BK11o0aBgkSil7l9dACGEBxgKPVTSYoXzPBMLb0im8LZ0AJ27u1n7nJDmnje23IvqzkmjiUNBQummY/tBXs05hZkfaLlrHZGMulo7yX54nge1IPyt/Hr2IucmX0bLdfp645Fna+e+mjbqvRKJ30A+tSe9La9seAELMqUVCi/Xnlf9cVcXJR01gv7KCmMWL0BRBZqvOhEQPqOanUUtICTkpYAuCfX/CgX+g583QyPBDriIXAhuklBVuItRY+QohYoAvgUj0PeXpUsp3hBCh6Lt/LYBDwFVSypSazmdQPmZUHk9pygOBO3GZLWUWj7SioKKQ7aXaabWDG1CIIBE1L9DBopAeHIgtw0GGrz8/tb6EczlKAqsJpgUxDAIkp9gKSA7nNmdW5o04zDZaRB0g098Ps6nQe5YSBY3O6TsxSxcpWaFsCOxFqCUVAIdmwoKGRMv/GPOCLoKIJY2D5b6D9lwJQKvGj5D+6JVIexrBvt1oULcmbie81QrSj4Jqg7yadyvehDvWQ5MShWoMymYClTA5gHdWvi7gQSnlBiFEALBeCLEQuAn4W0r5shBiKjAVeNQL8xlUQPOQHjywbDpfd+7I8ZBGoBZ4BSjoCjoYKyelntKwIS6Q+9EUzXEQ1bwSByo2BFII/EUTnrv1HSzJRzkU2Ry3xcq/PAtko2LFTiZtuIhgYknnCC5py1dzJ51Rpe5yaAjSzYH0SN/K7rBWdLVuKjgnFboq15Lqgv3KN6jCRYKjEY0s8aSJ8hVvLBfTlevyXweqsXg7X7xXeKu1rnihQPECIGHddBj7Yb2IdbohhPBDLyBc/k6wB2/UcDsBnPA8zxBC7ASaAuOAIZ5mXwBLMJRv3aAoDDjvQ85N2M5vTjuz1WNk4SAcX55iEAJIx85U1zJUs76KK8vGWR/YULmfvnxunkGQSGUn7UmQjTBLN4miOff7DeR9v7WouJlAEHpBdnBjJ56NtOEihvA8+5hPR5vCjkCVb9PAocVgcQ9AKCvRcOiTCd397JBPDIds0UihoKKhafopq+JgK1/RXD7GpN3fEmqOI8pygr6B/9HFfyt7c9oSaztAuLl4kUw4wiKi6EY0fevy46sa6z+B9CNln49uwLI3MKSUWUBYZdt71dtBCNECvZjmaiDSo5gBTqKbJUrrMylv9zAhIcGb4pzdKAoisgtjbb2ZxSXc4/Ft+pj1hONDZxqRcTwAqemKV3MJ3I560rx53gNSYrPn8Or797F7/v0Ek4qK5ARNOC6iOSyaIzFhxcQXjOM7LuMieiFQULGgYiWGgfz99wFefGYtvodG0VFcwVdNzWS0lyxrO5dg83HaMgZz8XpxeWWNBEW8HvT4DI19yoe08d3Biy0f4a6m79DMdpj7933A0tShmIQzP4NZYdzYWcH/OMbKUt6zHVKugfimkHYvyIo38LzO4eXwawWxtb/crK+Mf70d/n0VUg7ViWhnA15TvkIIf+BH4D4pZXrhc1LKMssXSimnSyl7Syl7R0QYYZa1QRwZvMtaEshmO4k8wmKSyMGZaUFKoSsaKVDUevIR9ngXjP3vd2a8MZnmp47hjvuPY1o0TqkSLY+hShcmQEEhtlCEm41gRvAGLTmfntzOT29ZGDHiK6ZNW0arVu9y4IC+zXBCWcQC53y+TW7Dr9nbiaQbzRhCMC0JpZ0uRqFrjxtR9C5AZDI4eAEZrkB2Z3dkfcY5OKWJyU3eIdiUhkm49SCOUj7CgywseTDrA8j9GbTjkP2p/ryuWV1Jc0LKflg/HRY+Cm/FwvpPTyt/8oaKV7wdhBBmdMU7S0r5k+dwvBAiSkp5QggRBZzyxlwGVWdLsVy+J8lgK6eI7JiI5gmtVcxauSYHTXpqUIoy9/BqjMXpINPHn8DcLDqe2M6zRx7HLyiLIDWV9uoO+vpPYACxBGDN7yPRWMd7pLAfiWRlfHPwrGo1TfLBB2t5/fVRbHIcZOqBt3BJFYEkN/oDXgy4n3TtFAfknySru/XxJCQ6wzmQ24pe/msxKRqahA3pvfBVs3hw//sowo1A36gziYLNO6c0011OYaf6JniSwQsUwuhQygeaDDjz3wV1sRftssPS52HLN+DXGI6VsiKvDL9OhF1zYcJPoBihAtXFG94OAvgE2CmlfLPQqbnAjcDLnr9GzZR6Ijv/R67TlEBiCUYVAs2zVKtQmUryc0VIACkRXtbAPwy5lF8GjWHsit/pcnA73313HQc7NcOiOWgcdTMBfUu6PWVximT24vbYcEfdc5w5r7TNP3/uuTEAHMkZiJQCp9QV96r0c/gtfiepMbqvrcWT0VIApxyRzDxxF67GM2huPcTGjJ7MSriJrn4bsUsbSLCJHIYE/cvilBGMDP0LiWDm8TvYmxvLYzERRFpPAoJOXEMqB/mecahYGMzzhNMe/O6EnM9ASwK1Bdiu9upnWQIp4aNz4JSnEGhK+ZuFFbL7d4jfClHdai7bWYo3LlsDgOuBYUKITZ7HRehKd6QQYi8wwvPaoB5ohB9WTx4EEwqjaUNzgniG8ziHKJRKuD1phb8qUqDJgj41vgPN6y8ELpOZn867hIdHvcS9w99jn7ktfm0mE9C39IAEK4EIz3sTmGjbuD0jRrQkvKk/N07rx9hLdZNCN1uWJ7cDWEUunf3Wszz9d6xWO1arHU0rSL7TwW8nbX12sThlJACjw+fyTYfLiDTHY/asdDUE5wX/xflhf5Lp9uWWXV+zLH04Jx2N+TT+tvw3dph/OMZ/gB748R8v6qfUJtDoEDTaDxFbQQmo4YdYNnGJX5Hzuh/aqa1oXrteumDWaEg/7q0Bzzq84e2wnLKdFofXdHyDmjOQGPaTwmri6EFjhtECgC40oiPh3M2fHCezzP7h2Nhvt+JvTUcREkWR+QrXG6Y/s1AIwcYpTzVfKWHfwjb8t2koi7uO4PMLVZoU6+MiFyc5+BDCYKaxnW/xIYxu6k3c/Fs2Nx0K5iup8u/RJHbGhHGutR1Pt3iS1en9ae2zlwsCE5h1sj+59jXYrHY0TUVRXAhgdVo/tmX2wInCE4de541WdxNuTuSmxjMIlLmE+Bynf9AK/FRdXj81m3fb3MlX8Tfxb9oQXFrBzyqDE0XktpPOf7xEfx5BESZdCdciLnJR5z+ELSOHI52bELXvFEKTCE3iMivYcmrg5515Av59Gc65E/bOh5bDIaq712Q/0zEi3M4CBIKb6MZNlLxFVFF4nRH8wX6WcpgjpBfrCxPpwVbbSX52ZmO1FJgwNAm5Th8OJLeic+NtnqOyUM+yUV0Kx7c3wpllZWpsOE2jLLzKyvzeXa/dhH3MTv555nyirs2FQknhT7GFf3kBiTs/w9l5FKyMnz6VjVMzA4Ij2cH8kJ3JBN9AkjKu4bygmezM7sSfybFc1GocC5Y5iI7ZhcmURdtWexGaZGNSb3I9dmULTvbmtCXcnIiqSNpnbaJzky1FQpAVJEGmNCZFfcC2zG70C1jOtEPPc1/0awSq6SU+ijhWc4h/aMmIcj8jbyDRkEIggWOdm7D5gs40OphIZogv7ZfvI3J3ImZZTQUs0SuM/F8X8CQlYtxM6DXRW+Kf0RjK1wA/LFxBBy6iNdfzK65CZXMk8AorQYC1UOI0l6aSnB2GKdON+E+wP6g1LUfs89iBJQI3spyUjy4JER1PEbcumptfasS817disajY8SS4USUmm4um3U4SaYrib/4ljs/xJxMrQbjRy77HsYoM4ggkOn/sEFWgW7NVpBQ0VS0szIIM0wqePPQGTs2MKlx8Y4rn/YGh+NiSyUAPMug3ex2n/Oawqs0AHNKMJgWtrLvRXLD1nwD2mZvQvuV2zCZd2eRoPviqOQCYhZNrwhYy49StOKQNq5ILAg7mxPLh8XuQKFwQOo8sdwDSz0EL3XyMWosefmZ8ybroSXK+eIQmO09ysnUjjnaKQnVphMWlIKqreEFXuEdXFChegMXPGMq3khjK1yAfX8zM4GLeYBU7SMzfjCuOEKAqbiL8ElACJEFjUjm0pCXSLRAm3auwFYcYSkdmeJRkHucQRRwZHDdnIoConnEcWtCK42krGR4xlPnsy/emUFSNiO5HuY7VCClRxXCGsITGnELfrtCQSMzFUjL+EBnMcGcWJ+1Wbgmz098ayLlxsNVxK26p4saEW5roHbCGXOu3uIWDvNKda67oSWzqXu61vcHh3BacF7iMXVkdeT7pWsJbJ3CV8300oSClG7u08vnJW7kl6mNMwoXdbaGVJQBFaCDhx4SrSXKGsyJ9IJrnpzb9xN2ouJkjVG6UIAV81BhuDvHyP7MQrcPuxv3A7TQ/tQvx4yXsiG5J3w1r8cmwU2Pv4qxiTkwhLWo64lmD4SdiUIRQfJjGYC6mNf5YSpzX3PpDESBEnqeEJCfZl+xEP/J2reJoTCQtSxgf7qQXkfjlWyeEkIQHJtEuLJTb6cmPXM5o0ZooLYA9v3fA3NqJW1P1um6Y2ENbBCbCaIsvEfRmMj6EccyVxQMpv/JM5sskm75gb4sAMttZeTc8kOuOwXo7OKQNNyb0ehYuWvnsw6w4isjnNqk4fVSuUr7nSdsLaFLj4/h7OOWMYldOR36xTsKJCSHgpcNPcSi3FZpUcWlmUt1hPBY3jFzNBkh+Tbqc5emD8xVv/hyYcEiwAw4Jd5zUTTi4dkHmS5A73wv/yaKomFHCO9Fi1yE2H+3FrNDrSbCG466pCojoCP2mgC0EmpwD43+quI8BYKx8DUpBReFWenA9XXiZFWzkJBLww0SaXWDyLbD7unJV3E6VpJ2RtBt+wHNUkIMvszjOe4zif6zAiZu76E0YvkzmHF4VK9jrTCPjuD/DJ2+hnTIeABMqk+jJTe75JMeHEZ/YhL6+y9iX1A5NKrQM20df0xSaMYhTZPGeXMd2928s3D8Eh3YBgvM52uhbXghbTwgtsRHC+twi0RL4qS76NF9MlikApBlZyBUv3tGYEY4ltLEfREFjle85CM+VQsNEkiuc1ekDGBC4nEvCfyLT7Y9NyUEREEoS3fw3sSajv2c0j/uE57niuZeQxRSeAuA+ConngMwBrBD0Ifje4JX/Z8FECvS6jcfWvYKGQOT7f1R3PDOMfBlaDYOL3vaSkGcPhvI9mzm1A+ZN1u/xR38AEe2LnLZi4hnOK3Jsws7tZHTZiZSCpF0RXCbaMS4mhBZPWfjRHMNsdnhaCvywEEMwH3JRkTEi8GU4sRwybyK4RRoJKCzmKGMp8M/NNucQ2ukU+3M7sWT/Bbg0MxJ4J2kqXaMUbgjRuEP+gQuNFEcoTmnG4fHh/TdtIEuCX2BJWn/CFCtX+V/LyykWj1QwImYd0pbBKcJYIM+nf1YTFmdnc9DRhA0ZPdgT2B6zottCL83+jbdte9iT2x4QnHJEMuPEZDZl9sJPyeLqRl/lKzBFaKS4Qgu90wLFC3hUb95PzkWAmo1JBvJsOKxKX0s/KVBwA9lg/63qynfzN/DzjXqo8oBHYNRL+afSOELu1o8Ii9+ASTGhaDWw9dqC9ZwPPW7WFe/pTARwZyXbGtWLDbzGl6MK/DS/HAUPlpNgxcN7bTtx3jvNOZbpoqtPIPc+qGDzeCJcTUfWcJxDpBKElUfoX/5g5TCGNuQO3EXy4UYkZUeSp8gkkjtOQKLvHlwWPSrP35rp8UPWsAgH3fx28dCBF0lyhSOAcNMpLEQzOugI10d+xlqTynZaI1E4cCKGT94/HxwChgEt4CfHpdxq/QQhNX51XMqunI5oqIDA5XG7WJPRn34By/FVc3BLBbc0MSv+BvbmtKXoipdCz/N+bpJGpkR+aD2H3Ox7uOQoNFPOYW2AxE+oCKxgHVP1D+3nmyBPqS5/Gc57DGyBJLOX7ftu5dwf/0XVZAnpqoRigcFPwYAHqjuCgQdD+Z7NZMaTb3zNrFwByfAA2PGwPw4XWIp9e1QU3mZUpcYZRgv+4yjbSKANoYyiZZHzV9GRjSIemi/j730Xku30R1cZEkW4OVGoXJJZcdC32RJ2HhvI8xEqQVYX85LDcUgbAKdcEahKNhc3nopDzaILAhtpbKQP//3fEDjlUUVzgTvgAd5itasfm91d2al18MxbdBULsCpjICcPNQEpOWRvWaxd+SS6GjFk190AaMAudwyDM9fyeegvdPHvCraLyh+gNAp7HYAeTgwcZy2BJxMRWkHCeIQJCns6WEPBnlzxHCEt9CTrBjXG2HA7mxnwMJh89MeAR6rUtbjirSoWVJ5nCJ9kXsmyl4bRaLKJu74q8J1dwAGOkIYQMKDFYnzNmQjcNAs6gJ8C55lXY8OBlLo63BXfjQdCfLkzxMoQ8wW4pUKey5tTWog0n8zPw6Agick5Sp81rXFk2gqE0oAjIN0K3zknsNPdCf0nUnwVW/A4lNuSQ/bWVO2nJNBQ0BBFvA12udtjCphaPcULMLDQ/7DV+eCvJ6oKox0n2zdDUwX5Ga5UMyh5/0QBYS0pFb9IvV14e5i8Fe7ZCT616JpxFnHWrnydTnjzTdi3DwYOhNhY6NoV/PzAbC7a9tAh6N8f0tLgxhvh/feL5Cc/fRn5P+hxk/48vG25Tb1FPFlk46QFQQgE0+bB2kPgdMOXq+DSnjCyU9HkYD5mOyPa/AESWh/rh831GbnKX1wunORKf+bH/Y/Gu07xd4/1LDziy87c/rg8nhoSBSvZnHBE5ocGO10mvvnxOhYv6wRBLshV9Qm7CvgbOB8IRk98XsTho+gGWslVbvW3r3pa4ZVI6GCtuG2ZjHwJBk4Ftx38G+UfjqIXWviz7Lh3EC1X78RfNIZet0FuKmyZBVE9ISQWPh1cdDzfRnDLkhJ7AQalI4QIBmYCndG/ILdIKcvMXnRWKt/jx2HSJFi0COx2mDmz4JzJBE8+Cc94AqZcLmjZsmBF9tFH4OsLl10GY8dCTg689x5MPF39yutI6QIsYD8fsxENSQwBvMUocp0Kmmf5JwC75064A2HY7SAsRZP+fPRSMx6+bz+Kqnso+OLi8PYEmg8/hGrRcLuyiE5dT/yp/uDZ0Xd77LTfHHuPSTFP8fRrD7B7fyd9xhRJ2LgT2No4OeGKQXMpsBDog/4TKkLxFXDeOrJmURIqML8ZRJorbFoxPkGlHm5KX5qG9IULip2I7lPw/LLPYeHjkHECkODMhM1fw4gXvCDYWcE7wJ9Syis8hTTLrVtyRpgd0tP1R2U4dAg6dIA//tAVb3FcLnj2Wfj9d4iPhzFjSuYvmDcPhgyB5GRd+d5+O2Rl1fBNnAV8w3ZcaGhIDpPOd2zniYshOkT3Gz63NQxorbf9/NRR/njgErJO+emmBQkWt4n+T/7BD2lXMuHuH7n27jl8uHwiCS5fpCdjjGqS+HhyLujbcwIXFlzSwuacpozhU/Yd6UyBwhSIPQptY3egKUIvAnMz0BzKCdDL71vwEyqcsrrYF6bMbNZuTEhmNwWfhvBL7H4jXP4VWD1JfpzZcGxV/cp0miCECALOQ8/wiJTSIaVMLa9PQ/iX14jGjSEoSH+8845+7PBhuOIKePzxom23bdOVZno6+autsrjhBn1lu2BByXN79uhKOg+328gtXRn8MCM1OLmpCXFrozngyiAmFA6+Ag+Mgr93QqP74Y2/IGF/CCBZ/OSFHFneAs0NDpMLNcLO9s96ouWacOVa+OebkRwIb0vSgXBcDhWXQ2VXat6SNU/Baqi46OvjUeLtAaVAUWqNFA4mtSroYkI3O1QSNT+Kr5QVsAv4FNgIJHheA2NDfyRITUdD4/oTWTTeDQvLzm1UdzQ9B6yBYAkAsy/0rlQ5srOF8LyqO57HpELnYtH/w58JITYKIWZ6arqVyWltdrjmGn11msd998H110OLFgXHpk+H3bvhq6/g/vsrP3ZyMqxZU/n2/v4Vtznb6btjIHP+zCRpbwSaU2GbRTLkfjhwCl7/q6Ddoz/AumlN+EzqCXbdThXV803NcfgVXUVKAQEmVmuD8NuVhcNiwSqtgCTEJxG3pqC6JBN9j3JdwJdcvvEucoa2glYCNkrwheQeESSnVqWKSlHbrxsLZZoesoB09AqGJmA4RHY/QYIrgnR3EBLdTc0BPBwPm+ryeyQlpBzQo9N8Pf7J1gB9Y+3AIghtBVE96lCgBk+ilLJ3GedMQE/gHinlaiHEO+hFg58qa7DTWvmWphy/+qro66QkiIio/ZXpkSPQrFntznE6o2lw5/sBZDsK8tbaHXDeKxBSzDJmUqFbYzMbHoeZm7MIbh7ETlQkGmHWBPzGpJM1N1DXgeeDCYFUFHKUAIbYYFoETHGsIzJQ91tuGXeUdsFbeMZ/FJsaA6kCWgItSy/7UzGl+fCWQSBwC/Cl57UZst0+HM5pnh9f5pC6sdfXW/eh9ky9GnFIKzCVDBEH9B/EBz0gfrP++vzXYcCD+nOfYOh0Rck+6z+FpD3Q714IrN1UmKchx4BjUsrVntc/oCvfMjmtle+LL8L48QWvL70UzjuvZLu6MAksXgw33VT0mNvpZO/8+Zh9fWk5YoTXKz+cTkjAUUZQVUp20dePXaRvsnWJhnei/YDWbMaPZ/gXaTIx7MK/SDovlMSklvRUYnksAroW8hhby3Ga+h7M14vHoyPIET0QQtI06CiHU1t5Fq+iBntl5W20iaJPA4GxwB6gDWRowWRoQRS2OwPstnuhRFPSXpjeF1wOXUHevg5sgSXbHV9foHgBFj1eoHxL47srYccP+vMVb8Gj8bqSNgBASnlSCHFUCNFOSrkbPZf5jvL6eKuG26fAaOCUlLKz51goMBtoARwCrpLSu4Wqrr4aAgPhtdd0E8Stt+ouZD4++kZYXSGEbnt2OiE1FcLD9WOzLryQuNWrkVLSc+JELsgzSjcw7E54fzH4WuBwe3g7FSJVWNgc2tbE9cmDpsGxlIrt7HncPbTksf2k0taxh8lp03Gj8G7wXYjmGm8VC85w4uZVTwpMoWmAxFdkEZ19nBx7AP7+6fSJ/pdNh8/BodqovvYtz+WslKbNPY8S/Quep2q6SbhGTg+r34ecVF2u9OOw61fofr1HUkkiO5BIImzBRaUWxZbd6XGw4yfd9ND2Itj5S8E5zQGHl0H7sQXHNnwOv9+t+w9f/QO0OivrKNwDzPJ4OhxA37otE2+tfD8H3qfg5gr0JfffUsqXhRBTPa8f9dJ8+Vx4of6QUnf5euQRyM319iy6ki/No8JiAYcDRo/WfX8dDl35btuQyeGlS9E8O3Obv/qqQSpfpwuaPgRJed4aFwM+cCQM2i+GyIUwpB28cgU0C6v6+Au3wyX/B9mOitvmcf5bcHUfeLiQW1R7Gcao5Jfxl7qgTyS/SkLkzhJ9XWi4PaEL/jmZRLlPoCkqLx19Rr/NPwWDGv+DnyUTh9uzXC6y3KyM65jEhANFSILVJE65mlb+zZXDrcFgrunNUUA0mGzg8qw+AqLyT63nQw6zFAE0DTuXvp3Hw7bv9JOKGdKOQlCMrrw/6A6ODD0S7rzHi0bDAQQ201fXUgOh6iXmQa8J+tWF8GwV/uFnCFLKTUBZNuESeEX5SimXCSFaFDs8Dhjief4F+paD15UvwNatMHSobt+tLcpyZXN4vmNuN7jd+g83MVEysO02LgsdgF/ScpxKAGnNr+P4cWjSwExlqw4UUrwAv3v+qiBbwcl0+G6t/hjdBe4ZAaNK1rEsk4mfV03xAqw/oj+SMuFlj+mxI+G4tdx8vRjiziWCkptkPpi5RLbnjWQnh1JjGR/yPTuy2+aHGgupkZgawbCQhezI7srO7M7FdG3ltJ+CG4e0etJHuqmEX1qZBAr4vRkMKNcrtJL0nwLJe+DQv9DjRmhVUC3jEH9jysrinF8345fyJ7LxOIRiAs3Fbp9mPLonBWvGVt7c9gRNndme8GQ7rP+k5Dy7f4MZffX8or0mFT2nOUu2NyhBbbqaRUop8wpYnQQiS2skhJiU57qRkJBQpQm2b4fOnaFbt9pVvNXBlZvFwoz7aHXtvXxs3c87e96mbVtY1cDcJhuX7pOv65M9RQ/N26qvSs/9H/yzq3K2dFcNsnX/3xJ45RT8mA5uTeGVpVOxu0zkuiw8/XfJeqwL94K1J9wY3YUtn/Qg3R7CZ/E3oeDGLPQrQBNLHA82e4XLIn7gsWbPMTRoAbc2/oBhQX9Rpp9uCQQOfACVdC2Umv6MnBL2O2to683DZIFxM2DKLj2xTiH8aEzv37bQeE88wfHpiG2zQejZ4gYPWchcn0782GgUF/f6TPfxBT30XJbyT/znaXA79HwS6z4sek61waavSvYxKEKd+PlKKct0M5dSTpdS9pZS9o6IqIq7D1x1la6AG5aPrYbAxSk6sytnAI/98yYJmaFkZytkZcHbb9e3fEVpU+olsXxWHoBhr8ONpSyIQLftPv0LnPM89GpeepvKkOmAJ1bAhCPw2sZPeGzwi1hUF7lOH37acVWRtrlOuPB6cGwGjgNPC0gEp7SyOWMEUuq7/u18d+KSKlszu3E4N5ZboqYzPGQBN0fNYHDQIirv/iCK/a0+dbUNO5hpBKeqqFpe8TkztL0Ap2Lm3MT/mPPfFVx76GsO+LcEFN3X1+2AtMNVm8idC79OguWvQXYDWxU1IGpT+cYLIaIAPH9PVdC+yqSleXvEmmEhnfb8hMRENpHkEM7Ro0Xb+JXrdn16MWt1yWMpWfD5CnhjAaw7DPO2gLUGxi23A5wK3Bt1t+6cICDIlsZ3N34PwPwt0PVZCJgM7n1QJFNNlq7Ynm0E3Tzm3d3Z7XntyJO8Hfcwzx+exq+Jl6EIsChOroj4nsqvfgtTM/XZxwcmlHUH4kWSMbNj+LNoZk8ypcCmkHoUi+bkx/+u5PK4X/h8zc18u2I8+EfpNl/pplr+eO5cPVT57TaQWnGq0rOR2lS+c4EbPc9vBH711sAOh57gJjFRf60oejIcUz07zjkIYhdXUN6qaNkyePBB3UbcEPhiefX7+pjh1T8gx6Gvdi/7P4h8AO76uqid116Gi1mFaMAGIB2ypA/LQ/qwMOw84q3hmJuvZONRFxe/C1uPecwbfdB3McygjILLOsKKFnBfGIR6vulxjmbsyulAruaLQ9r4K+Victw2ct1WFqeOoCCLWSkKtfAtlhdvt5bkwHXH9JJCtcUx0nmQhbzaJohJU97lUKOmus/uiXVF2gng4hN/QGZczSeVLj15zxfnw+ejYPfvFXY5m/CWq9m36Jtr4UKIY8AzwMvA90KIicBh4KqyR6gar76qB1Pkff81rfJuTPXNvn26V0bLljB5cv3K8uTP8GINfg9ZDnj0R3jtT5h9OyzYoWcn8yYiCZgNX907gRjrEaRQWBbcH3Cyc98OoGtB4yZgugFeuBce6gKqApkaNN0Nx4vIVbDmyHT789GJe8hwBbI9u0sFwgiQEiHdSFHVDbY8zVr6KnlOBvRMgqnhVRy2AjRcrOdj/iERF+1xa5LJP79H9MmDRdrVnulDQtIu/XF0OQx5Wk9P2W6svmo6i/GWt8OEMk7VirPfgQMNzc5bNZxOPcFPffM/Ly1EErNg+JtV62NS9P+hu4L/o1kBLQsUHw3p8UV1uK0kJjZhaKcsni3U1scCn98MV3UrODY9pbjiBSvkZ2OQqKxKH1jq3ApuEBKBxM+SSbo9GIRAVsuzoWJXti214Ju+kx854pjPqHmHuSouDcXtIjr5ZP0kdXHmwN9P67bmnjfD6PfrQ4oGw2l56bn33vo3MdSU226rbwlqnhC9JvhYIMAG+ANd0KPASsHhBpsZZv98LXaHhVy7lS07uhIpm9A/pDv/PgJju8Fz4yD7A7jqnKL9bcX0nJ+Azyrp7icRaNKEW5pJtwcT5uvZtsgzPld6BVC4XdlrzEQX/C8BNnpRCcexmo6Ld9Ft03ZaJBwjJvlknW3wlYrmBFc2bPu+PqVoEJyWyrd7dxg5sr6lqD5CwIABsLqUDau65OfJdbfTXpyMXEhVgBvQDVZXUaarrFmFrVsHcv/jH/H6m69wZIqDWRfHc93Ap+joe4Jf74GnSyl5JiVEqBDtucgECFjWAiYEwxAfPfoNZNF86YUwKQ5PrQmJSXHichcSsEpxwIVXvWUr7IU58EQC9DkIa7PLbFZ5jm+g36e/0XJdgbdCGdbsuieqV31LUO+ctuvHI6fxBqqU+mbhrbfqASL1Rc9mutnNXV/28ibomsCsP2wXQe5vJZulZIM2A4QIY+1HPzA6YgJJsbEoaCS8sp/F7xVEcUkkGcRhwsYXKeE8dNKNXUoCNSe7W9hp7BMMWhY9te9Zwg2Aik0kEyvjyA4M4Gh6c/LUk0szoZebVHBpJtLshUL8quWUW7k+LuCG47CzdTWmyENzw+cjCMxNKZoErsrS1BIn1ukJgKz1mw4wxRzE91GDK24I6D4E3uO0XPlC0Xy6pyuOeo7A1KSexLzeOEXBN9AJPVV9lVsaea6p7sBGJAa2wW3ywWnyY72jTZF2a3iHhdzPfO5kdno82ai4hQncDjYueU9v5FhAC2UbVuHELByc22IpbdrvpEvUetqGb88fS2KiYDlevJZb7bKvpt8Nlx3s6aT7BvDt0Cv5YdA4csx6og6HauJkUHi1nOq8Rm4axK2tj5kbDKet8j1woL4lqDm33FK/80cFwxMXFz3mU0ZWFwGllNWpPqoC5kxgDrAeWARb/oABrUq2fWq03h6gz9XjaKImobrtWKSdUd0KjAZ20jnKv7hxoOGgU8A8fF1ZmN0OQNLzuJ4Zf05WJz6y30qQmkpzy14EEkUBk+qmkf9J773JGuBf01+mxZe0Htfw6K3T+OG8S/hm+NXc8OgMkv2DSfcL4rshBSkj6+X6qzl1N7SzmNNW+ZaWOvJ0w+qFjGE15ekx4F9IjlA/CPIp2ibQB16/Cjp3BwoFiYQHl1w5d2gM1/QBP0vRIjvFcWv6ht/5YSCWAjv1zbVBbcBSaPV7TguYdknBa0VV2PJ2Y14Zb+XdG6zMmlTQ2IQNJd+SJrg0dCff7XuBF3Y8z4a/B/BT40cJejiTW9+O4HBiU065GnHS1QTNrOLWFFwulRNHoyv+0GqB4gv+J6qRxKg4W8e9wPGwKNyqCU1RCUtPwua0E56exJRfP6r5BDXl76cgo2Fc7OqD09Lmm5HR8HIkVIffftOrb9QnQsCvd+sJcMwm+HoixIbD/bP1kN0pw6F7c90z4WI79AmG3B0Q6g+rh0CLB4qOt/cUbH1O9x9++Q/I8eRYaRkBhxOLupZJqWdMW75X9xk2K9C/NbxwKTw7F4JscPtgPQ9wnmeGwwVfroS0HJjQt6jHhoqFQTzNJj7Fgj+9xd34jQthzPF1JJkmMPnHtsgbdDXnszELQhUyXcEsOTCKJoFHybb7E++IqkmOnGozzA/GB8APGXBJAEwKrfmYvUQUUOCVcdfcGfg4chvOplvCdnizOXS9Bi751EvJLU4fTkvlu3Nn6cUvTzeWLq1vCXSGddDrqBXm61Jc4dpZIa4rHOsArSx6+sPeLfTS73m4NNgdD5f3glf+1I9ZTRKrkoQmw8j72ZtV+HYS9IjRs5dtOgZX9oILu+iPMH+4exZM+Q4+Xgrf3w5bj8Or82H5fn3c1/+ChLfAr1Ai9Qg6MZI3Cg6oQEw/VuyIRzkP3Ko+f04PPz15kCKwu3w4mNxWN37mKV6PO26MCkdrORpRAZ4Kh0F+cIsXlG4ePpjpSiO2aCdAUbC4nSgNzUHe7YCt3+nBFyGx9S1NnXJamh3attUTpp/uqOrpFyzir0B7a0He2RWPlWwTEwKdmupRb/1aarg1JzvjCzZ4FAGRgbDrBDR/VM8DsS8ebh6gn7c74ZE5+qo5yw7rD0PsYzD2vQLFC5DjlNz8WcUy/7MLxr/fCFORqqeAkKXHPkiweIQ96i7fPcwbaNRe9eLJ6VH0OLCVwOwMpl98Cxk2P9xCsKNpTVwpvIyUYDn9iyAKIQ4JIbYKITYJIdZV1P60VL7BwbB5s14Q83RWwooCc+bUtxQ1w6TCpqd1ZRriqyvcAM//ZHQ3uLZPKialYOkY7qtnuUrJ0kOT81TboSRoORUWbgOfO4vmGNbK1H2CHzZqHM9xsC8e1hyAz5brfwuzcAdkOwQ3HfkMkeyGZIltfg44hK75XGAukj9G4lDyJMu7Sa9dBbzGG369xTm1nch3e/Pw92/z4Tv3keYbyHWPfcJVT33NMzc/hVMpsK/U6xqgywTwq1pGwwbMUCll93IKbeZzWpodAGJj4c03G16KxqrgdOpl6OuDbDt8s1q3mU7oo9t7q0u3ZnCyjPDiYR0CsKg5WFQHbk1lQOxhft0eRlYprlTHUuGqj0tRBIXjgYHCeRKkJuj5fymc2t0IKQUCPXrum9tgXA+PDO3hnUXw5/KLmdr1VUzmplx7wyC22WN5LwV6hsEtITB4YwaZkSYcIq+8UFGvWAs5njy+3uf+U3BjCPh5czm05VuEIws/JA7VTL+da1jQcxhXL/2JsPQkMnz8CM3SqwQcszSiiSMBxfOe69T6mnwGuC5Vg9NW+ebRrh3s2uXtUSW+ZJBNANX9Gio48SObDErmChQCAjxFfK++ugZi1oCRb8Kmo/q7+20zzLnT+3MkZ8Kot8yE+KbSp+m/3DfCwZaEK1m4V896ZjGVLKpZaq6HUu37Bf+X+F2N8l9LwO52MWdzJmN76J/9oI4ZzLvXj//2NWNUx8fo4yn71g64vNCIa3sNZdip2Rx2FK4Ll6eMJOEkcpyYyn8AVUDKWlh9RnYGsw84szErZgbv3Ezng9vptXczZreziMUlyJ3JSx0e49GdL2OijqNujiyDzATwb/Cr3/Bi5oTpUsrphV5LYIEQQgIfFztXgtNe+f7zD0ybpgcsPPcctGhRswCM9soOpvtPoolynPPT/2K/1pqqK2BJGMmsCOpP27R9yGLWHSn1vL6bN+tl7esahwtW7pd6TTPNza9fr+Gh/RnccUdvWrf23o7Pb5shNRuyHBEcTrmMiAh4/xo989nyfTC0Pdz5ddE+M2+Aq8v9ykLBqrSkSUBRnDxx/7O0an6Av4khh0RySEK0V7mr/SuE0qaMMSHUHcBNjWYy7diLnrg2jfetd5El/Fjp7M9PrisLpvcy/X284NtbnM5XQ+Yp2PsHIn4rbQ9vK1mLDU+BZXc2j+x8BXNdK9481BqVDa0rEiswJwyUUsYJIRoBC4UQu6SUy8pqfFrafAvTuDF88AHMnAkrV9Y84c7b/vczyPIfN2d+Xm3Fq+IimTA+dtxBa2Vvqa1OnIAwL/hyVgfL1rV0St+OxW3H9N9c5Kq/ePONFfTt+AZZjf3g+nG6TaSGxEYUqEWzCp//B22fgCt6w+w74NxSAir2J0DLSqVVLFn9F0ViNrt49b0nmPLk/3E06wQ56DZmiZvVvF3uiMGBXzNZbmRxzFgG+m9AQ+Uu+8c8nP2mrnhr8V78aG1EbAoB/e+FSz+D7MRSFW9hTLjrx/arWM6IMvRSyjjP31PAz+gZpsvktFe+hcnOrnnl4p3uDuRIG9tcnajur82NGTcm3smZwj6t9F1li6Ue3BqlhMk3wqg+LFs+iGd2P0fzA8twaQKJwOGUHLb7wZ9zYUBH2LFF7/fPAvhyBuRULd3WeW3hzauha7S+2s12wL5TEP0w/LYJRr2lp5YszOM/w4HEary3AKAPOJwWcu2+ZGb5s2dfuyJNREVfd7UpEUF/MiRgHhsyCyV+qQPH2O62ittUG99w8A3TqwybbHpKx1IoGUBdFzfGil5qfs2HsHX26ZOYuxhCCD8hREDec2AUsK28Pqe92aEwmZk1H+OxrJcw40Cr9q+toJ8TC8W/zooCjRrB11/Xg/K9biwsmAdAiDOVx/e+xPiAYPql3MqttvX0NJ+gjeqpuXVwH4zoDecOhqWL9GPTHoFdCVW6vbh9sO4FcfXHBcc0CWPfrwUfAlUgFQU0UBUNX3snfDlENgkomOjHg5Ue6q5QeD1Zfy4o5BJYzv9MgWrftPeoTeWrmuD2NbDqfV0JH1oGe4pnMCpN+tpOoCKg+/Xw+2TIiNMvDgeXwNgPK+zZAIkEfhb6j9oEfCOl/LO8DrWufIUQFwDvoLuvz5RSliw76yWKKt+yk1aXRy4+3J31gZckKjq/2QyXXgqzZ3tp+MqQmQmvPA0nj8Pikt+FWCWV42GvAwKTKKYKnc4CxQuQlgqDOsPM76FTVyrLmDKaelXxZgArJVHd4kjfE4TTYWHG7z2YLMOI7fM2UkhOsZVgWuCWMD9T//Kf7196cqHXGsNQP1iZA+MDIV2D5dkw0g9eT9L7t7HA+lxdZQUqeoBJVsmhKkVKbeu5wKYw6iX9eZ874cPeenWJfDT0n2gd1rcKawvdboBNX+ivpQabvzotla+U8gDQrcKGhahVs4MQQgX+D7gQ6AhMEEJ0rK35brkFQkLyXpXmHC+Bmtsyi45XORViNsNLL8GXX3px+spw783w2Qfwy+xSdyKFAJOgpOIti327YfSgKkWH+Fjhq1vrwH1JEyTvCMNht+BwWjka15wnPunPz/MvReJmC/qP/IpjcPVRuOQoXHm07OEuCoDnG0EnG/T3hYfDobsPfB0Nye1hdUs40AYWN4fjbfXjxZO3V5YaWsuqhsUPWg4r5UQdFxbsfj0sLxZa6T4DQlcrSW3bfPsA+6SUB6SUDuA7YFxtTRYeDnFxsHEjzDh/AhFsp+itlEBf7xRPple9NZhSZGwJxAOJhV7r+PnpnhgTJ9ZDMp2tG70fi52ZXmXb3HX9oF1j74pRGna7Dacrz6YpcGtmVm/oD4DZnYU75TZ+SZPk/AmOD+GnObC/Eqbs+DQ4938QPgVemFdwvJkZhviBrwLjAiComsp3UkjFbbzKufeBra4nLcaSaboJpDBVro13+lLbyrcpUHhtccxzLB8hxCQhxDohxLqEhIQaT+jjo1e6GPPak0wOvpgYVlBUyRY1R3TnE+6iM8N5nKoqYa1IBpb5wCfAx8BS/DlOgJ8TPz/97v3ZZ2FwZXM2e5OuPb0/ZrQnC/v3X8GkCfDNp5VSxj2beV+UkhR2P5OARqsWe0hNDGFQyipU+7eY17phB5AD7ITbfqh41Ifm6DkskrLgpfmwsZRk/pkaxFfD6Lu4WS1vuJVGWBt4OA4Ca8dvuVK4HaAVi7bpdkP9yFIP1Lu3g5RyupSyt5Syd4QXnV4ju3ThyVP72H6yPdERaVhJJZaFRLEWPKVhLKTRl7fJIJKW/EUbqltR0o2elNYJuFD5h3tpw5xJjyKl7oPscOhVK7zgwVV5MjJgbi3ELzdtBn3bwuQb4Ofv4P7b4KWnKuw28yawldhl0BVk9beqSkN4htXwd2YRMiuNDnsPEepMBQRBe10UvtDuq0SQTkZuQcUPReg5JwojJYw4XLJfRSS2haH1ldbA7AOXfl47Y6uVvJrIQv/3TlfCuI/LbnuGUdsbbnFQJCQo2nOsTlDNZgIiw+kaNI9zEvQiXxomUmiBRCWcPUgE3/ETPiRzWAwrdfHr41O2l5XZrH9/XG4fIBuBRghJ+PkKovv14/xD8Ndf+o+zWze9fZ2x/O/aGXf18qKvNQ3++BWeeLHcbj4WSHhbD75oHAQ3fALHUgqvVDW8th4QAoSKcEvuP/Au/VMPAwpucz9G9PqX7/4Ynj/vNT0rthW8eKme+jItB0Z2LOmjnKLBxioabgUQVt/+Rq2GQYvBcMjLKfbc1bBix60/q9JK1va/fi3QRggRi650xwPX1PKcJWiXo6e+EoCKi3D2FYqPklzA/bzHHqQs3d5Ulsk0MBAyMwWapiLEjXTt+idB5mSubXGCvmM/otNVVzH7Upg1Sx/j+utr5e2VzW8/1t1coy+vuA3gb9Pz8D707TGu7DCbw6kt+HnHJYXKsVfPS6UEUiKkhkVz8G/oAK75aS7NtoTw6f3ruXTcO6Q6ctixqwujeh/i5TFDKxyuU1OIf0tf8QaWkt4hWIHGJjju0n0GAgQkVWDFimwo5s2RL8OMc6nn9Dq6R8ZZRK0qXymlSwhxN/AX+nfyUynl9gq6eZ3mPttJKfRao2gmQSe+qDg9Xo36R2IljeYsZS8Xo2ml/0p69oQlSzzjyEYEBd1QIkev2Qw33eSlN1JVNqypm3lMJrjjvko3X7knhefObYfVZMfhtvDS0kd5Yekz6P+Vwv+dmjHm5FxGnlrAo51fJ1v6cWgv3DE9hjvudnPzhE9QMNGGsZUeT1VKV7ygmyJWxcLbyRCkwH1h4CP044/Fw5tJUNi6KYDpUaWPVef4R4Ji0kv71AtCDwS5+uwqJ1/rNz1Syvnou1H1Rq9bbmTR40/iQkWgsZ8RhHKQMPbiwsrvvEMTVnMEfUfMRC530IMT9GAfFxRalemmAxtpPPiAi/U7wvKVL+h5JRoUA4fCgb21nzTY5YJ2EXDbPTDtDX0zrhycmTPw9c/W3dyUHMZ1mMsLS5/2rkxCMDdqHPOixhYJmNm0L4o2XMwh/iaYlnTkSq9N2cQMr0aWPD6tkZ7/eHOuXqUiR0JfH+jVUNKhhsTCsGnw95Mgi7ub1X46TS5+H3pP0oNB6pgkwpjFtZVs7d3qxfVtcaoTBkydijkkjIlTojng6EsOYbSMlWxZk8x/L7+E+Y0RnKQLn/IfmmKhuW0nESIRW9bfaBQ20krk5q8ZyW3suEUy9OHH+Lnts+zZA82bw7vv1ttbLJ3n3wKhwBcfF93YqA00N3z8tq7oX3y73KZdYxohnQKXW8HutjJv92gKfuRe3AMWSoltvNsGCbpyA63tN/B//8ASJ0weqlfOqC3MQlfADZrzpsJ/r0NOUt3P3fPmelG89Y2QDaiUQu/eveW6dRUmgK828+bBDTfovrbz50OPHiClZN4dd3Bg0SIaXzyJ4IsfpUubVL7s0Rx7ejrvsptkWpGnFB4mAj9PshahKIRP20PrntGMurABVMMsi3bhkFxHP6qY5rDhULlNVh+QHNp3FYqQ/LLzEr7Zci11kUH2gZHw4PnQJBgueAuW7NZDnVtFwI7nz6q9ntJ5JRKyTtXtnN1vgssqUY6kFIQQ6yuTtLw8gnu3koPXvVSptnPF1TWerzBnzeXG6dRz52Z7KgZMmKDnARZCMObj4u4twdyyYgVrH32ImctG8ZPjOYRqZlb2VeQSgi9JunVS01j85BPcxye88kIm93uj5GxtUHYpCO+TlaV7P5RjejiVLhj//RwUXB5f6brRem8uhA+XwM4XYMV+sHsC/vbE68VCfSzldj/zuWo2fDGqwPZr9gWnF0psBLWA9CMFd1+Nu0Pn8RAQCd3qehe64VDvfr51hdOp+9rmkZJSdluARp06cfG8P7j054/46vlDfPFfO3r2NvELBVdpAbTkb5z48fH/1UYdGC/RrVfFbbxFanKFq+z+rfRKxRom6rqObo4Tft0IF3fVy9v7WqBPrKF4AYgdAs86YJrUH7etArMfmHzAEgDnPVHQVvWt3JhXzYEHDsD5r4PquTtM3A0+QdDjJlAaistH3XPWKF9fX5g6VU/laLXCW29VsuPwC+DhZxDde7FyJUyffy5+0a0RJhNOfDjIUMxk07N9zaPzaoVT8fBvLfn7loamlZrApzDhAfDm+DqSpxSiguGrifDxDfDuBFj8UOX6pbmh9wEI2gX3nahVERsGjbvAQ0fh+vlw3z4Y8QI8mQVTk+CZLBg6TVfM1iAYOLX0VJVtL9TtOWYfff8B9Mi29Dpz92+wnDXKd+usWYR+0YyXOo1iz9rDXFMNb2OTCS64UGHyplUMe/55ws8dzQn/i7muzzw++72L94X2BpUM/fUqk2+AeT+V26R4Ht+6IswPruilF/68th9MHFT5Ve9VR/UsZukavJMCP6dXXw63hAWZsCSrgVew9gnRV8T+nh1Diy/4eqqdDH0Kns6GJ1Jh6DN628KM+0RP4gN69JpfhF6l2CcEek6sozfQcDkrbL5Zp04x99ZbceXmQlwci++9iZv++afa4/mGhTFw6lQGToX7vCem95ES5v9SP3PfehXE2UEt/bayLpLsFOfxi+C5cdXfWNtdLA3B6hy4NLB6Y115DBZm6v4dtwTDuw3F57e6mG1w+1pYNwMCGntcxwqthH3DYMoeSN4Pwc0LlPJZzFmx8nVkZRX84jSN3OTk+hWoLjh5Aq4bA5vW1s/8bne5iSyiQ0rL81BFqhCqHRUEL16mr3iry5RC5e0UYGJw9cZxSPglAzIlZEn4LLX6MjUogpvBiOeh7+TSa7KZrNCo4xmteIUQqhBioxBiXkVtzwrlG9yiBV2uuQbVYsHs68uoN8uoc34mcfPlsOiP+runDQwGW9nJVdpE6m5fZhWahsAnN1ZjjiokIL+ufzXGty+Dky3gZDTkLuT+cJgfA0+Fw7aW0Maq24Efj4d7T0JcJQPEzECMSf/xmYAODdhL0aDKTAF2VqbhWWF2EEIwduZMRrz8MmY/P8w+DSW0qBY5crCe62FVrPRfuFR/gH6NWLYHvljpvSkE+g1Pr+bwxEVVGDeP5LFAmv485XxoFMeFAVHsWQ3D/oSwSLBcBtsdug3313S40B8WZOmRbK9Hll4lQwhY1gKeTtCTr7/Q0AMwDCqFECIauBh4EXigovZnhfLNwze8UmVxS0VqGjt+/BF7ejqdr74ai5aG/PIiSNkP/e5FjPwfTqeuRCwNwW1pymPw7EN1nMOyEJV0IcrI1X1vAd69BkL94Z1F3nFN7hMLq56ouB3A8pTNfGDagnTBuMShjG8TDRTeUZOQPZM/Dj7Ffd/pR06mopcv8vy/j7ngy1Q9TfD0FOjjA+ODSpnMuZ3m2hG+aDIYhC9kPAuZ/wMCIew3sFRnmW5QB4QLIQpHgU2XUk4v9Ppt4BH0cq4VclYp35rw+113senLL8HtZuWbb3LLlBiUY1twu8C85BV+3HszNz3QBk2Dt9+Gu+6qR2GPH4P9u+t25auqup03j959K9XtgrdgvScP7k8bYOVjesrGAwlwz7dVsZqUzIZ2Q3+4+B3YcBiu7QuvX116T7fbydv+W3Ga9fv/uSxg6P7PiPQtOfnsYgGY0fGHiY/Rd8taqinsd+vJHdxAQt7HId0gU9BEMKdynqZR2hsITAjcgD+Q5xedBCk3QOTeyr5pg7olsawINyHEaOCUlHK9EGJIZQYzlG8l2TZ7Nm5PUt/EHTtY9vVR1q/S9Vv7dhp3HIrOTz358JRMLGs/YOWGQNpNmMiDD5vL2vT3Pnt3w8COdW9yKKx4m0TDax9Vqtuag3rhSYB1B/Vb8gs9Xnsn0vSqEVYzPDACVu6HJXtKWxVrKBY3msOEYnbjH2Tnf6P8+HEDLPYkSn9jISRkwheleDhpuclotkLbHxY4nppLZJE4Ahv4TuG28+CLFfqRUJ9EYpsf4qTWBA2BS0vjJstiAsQplmnjuCawBdjXQsqFINPJVUMJd59CQQIOz+WieL7SOq3mZuA9BgBjhRAXATYgUAjxtZTyurI6nBUbbt5ALZYFfe3qXJxOXefs2g2NLXs8ZyTXu87jyOdTidlyJ1mPWXlx0L3UWQ6NLz+uP1uv1Qqh4bBgrV5qqBIM76BHmflaYGj7oudeuBRyPoRHL4AX58Pi3WWZIwSNexyj3djt9Lh5LWPu3MrkYbC1mB//TxtKl8Hs24gr/l2O6nZhdjrpOm8/XSK35r0psF0PjfaAGsiA1vDZzdCj2WGeumIacbIpLsxomLjG8i0f+17D6z73s9Y/lrDEAZDcB2QS4MTHHY9akaHadkXFH5pBg0NK+ZiUMlpK2QI9b/ni8hQvGMq3UjgyMwmMKVrryq9xgWOmFCb+75NQoqOhaUQ2jdmIikQAChJt5Xvs/u039pPCWo5jr8o2fVXpWE/BHhYrPPkSrNgJkZV34v31bnjvGv0x795ShjXB+4srHsdtN9Fu7A6a9ozjwua6K9OU4UXb9CirXJkQXDPgTd5ea+KpFaE8eMkUTNb2gBlsV0Lgm+BYDg7d5nBJr838e2snbmnzOZuCetBB2YY/Gdxo/QJFgCL0/z1yRcmpPH8L6l6LIqaVxMxZFb9ZgzMCw+xQCeZcfTWntm3Lfy1UFd/wcPwiIsg8cZyh0x6j+6XR/OH4ju0//sgOT0HGwhbIReajLJV/o6DQSPjxFiMx1ca1b/xNsGs7fPCG98cuTmxreO4NuH8ipCTDtKnwxvPw0ntwReVypFrNcMvA8tu43MVXi8VdCASPXig5RgitlSCuSrJAYC5PjLbhZ4X3/obuMfDtpHImMdto1u8q8tfrEZsg41XIfBZyvy40ZwBS+uNDFoqAHGnlPZ97SCCSGOVw/rq2rDiOggoqnja2iZAzM/98mEjgl+0buaRTj3KENWjISCmXAEsqandWpZSsDlJKXg4KwpGRUeLcsCdiOHfSMZCwe+m5/HzHBlweu3DhT1UAe3a+Q2Z7z4ow043j2kZcH/QbiSvmEd2/Pxe9/z62oNK2xqvBkUPQK9Y7Y1WE1Qb2YnZKixV2J4B/pTZ9KyT24VMcSsnzx5L4WzLIdBSEln17G4zvC2Qlwke9ITsRrAFwx7rql6Zx7YaELuhFUYuiSdAQmIQkS/qw192KbqYDaNhRcZdTCMkfyCx6yDwY6ViaHwMkJXSdtZMfrm9fL1GApxOne0rJGi29hBBXCiG2CyE0IUTvYuceE0LsE0LsFkKcXzMx648Z/fqVqngB+t58FNUsSUhtRKPu2/MVLxRb3QC2hRtpdf6LdGoxmUbv/EqfeaPY89U7pOzfz9avv2Zmv341lvX777fTps17DLviNxyyjixKxRUvVBjdVlVGd1hD4ctZ96g9XNoDRneFdU96FC/Atu8gKx6cWboC3lC9PLEAyBzKqqisCP2Hs9LZhyPuGLqp2xBko6BvOhYuXA+A2gOCZ1NyM00B5+r8L4mUsCO7PYcOt2fPSc8xJPFyHUflP7hKbM4ZnM7U1OywDbgMKJIQVwjREd3o3AloAiwSQrSVskSNkgbN7Msv58Sasuug2TPh8dde4/3P7gYko5Xb6KKVbrOLmfI5KALhljR9cnaJlVHSrl04c3KqHQBy8GAKV1+t2zv2AT0DHmSb9bVqjVUmNh/co68gZ/ZX+CqlBxAAernn/Xugd80vKKBnHnO7PmbhvmFc2eknnh1/I5bSosL8GoHwuJWoFr02WXUxdUP/eZT+lRVI+pn070Z+5Dq6UhboqWvfXnkfyw8P5OaenzOm08ue8fLs/f6gxoB7p95ewvHMxgx8byuBNhjUVm+1LecG9liTEYC/24+Rpu8QxlbNGUGN/otSyp1Syt2lnBoHfCeltEspD6Lrgz41mauuceXmsuvnn8tt89l1Abwz817sDht2hw9/aa+W3Viih0GVdVooTLr7CXZVc3GzevWxIq+PZ5bRsCbk5sAPX+GjVJCFNzMDRg/Sbc9eQPiM5oOberH32eX877rrsFjLyELT6UrocxeEttIrJPS8pQaTCihnpSlEyQQ9hesvf7j2TvYktuXabrP4ZedYNhyPRcMPiR/gC/5Twb2zyHhh/qHMujWTvU+8TzBfgebgkCUet2LCpZhIV7PJkYnVf08GDYra2nBrCqwq9PqY51gJhBCTgEkAzZpVzj2pLlDMZsw+PjjzSl+UQtK2TM7nHlbwMBk0wTffWb4kJW5Fi+GyWFjW/wK+2g/3hcJrkVXLvnXRRW0wmxWcTjfDzQf4I7B2ds1VAKGv1KQsR0a3CwZ11ksKxTSv+cSWc/RHeQgB57+qP2qKLOqRIiVkO6z4WuxF7LPJOcE43VZ8zFkE2TLzL0qaVHj9wofwt2RzQZs/OZjShlxnNmbFgapoKJlPlpjSJhK5qFGIbp3IBdLuJjS4LSdsFjQUzJoLqxJc8/dmkE9abghzd1xVydZlROlUkwqVrxBiEVCa6f8JKeWvNRXAE543HfQNt5qO5y0OHFTZO2IzKctm0TX1BdRS3MMUJL2ZTi9m4sSXbMovI1TeDvh/Nz/EwT5DkcBHKTDUDy6uwn5VYKCNQ4em8HrPSTzv/BGzqB1f3zyFm6+AqGAV/OYL8NYMr8vx5JfJvPFjKuF+km0zWxLkW7OKGNvi4Jt/V9M1cg1x2SOISwvijSEUUbSFFW8eoT6p+vlCn4JbKoxt/xs2k27jNatOOjXaUomLafH6aen0Td3ILv/W5CoW2ueCGlqFVG4GDZoKla+UckQ1xo0DCntVRnuOnRZkZkLvni7SMmIx8TBxRDCayaW21TfVNKxkYi2+k10Gpbkj5QYEFzmSUtjU6HbrNtSISAgplNewGE2aBPLmdxPhytlFjmtSoAjvXNeKK5AK9UnHrl6ZtzCLNmTx4iI/CA/hmMtJzJU7Sf+9Y7XHszvhie8W880VY1CFCyHggT/ewOFSsZjd+Ql6CpN3Ecpf/RfzGmoefAS3JshxW5FCYBGu/H6UMl5ZmHDTOTPPsqfqG3QW79jSDeqX2rLczwXGCyGsQohYoA1Q9s5VA+PgATe5WQ5AxYUvhxns1fHzvCAKmyH6ffUWfikJmOw5RKXGc0mAZOPbb/Fh82ZkNLYiB3TUqxD/MrvsgQGyMov8sk+46zFlVvfecOvdXh921u+eGj5CgNlChqlmLnop2TA89jf8LNnYzA4sqoM3L3yQH3eOB1m6lixxESryWiNHsyKERAqBSehZ2Gvu1ekG4dmQlfWUMMnAa9TU1exSIcQxoD/wuxDiLwAp5Xbge2AH8Ccw+XTydFhwSRd8tBOYycRMJl35qlbmKeyOFnwyjvtHNGPKqJZcMyCKbS3C6fLCA0zKOoq/dHsipiRMvafM8bJX/Iv75suLhDLnSBuiEukda4KUkCOVoqG/jZvAwrW1Uo/9gasiwJ4NTjs4HTTK2F+j8SIDId4+lBynNX8lazU5mNBlFnjuGErbYMs7XhxVgE3YUQT4KrlYPCagvDGq/5EE4Da1IjVpBPKkFRK6g5ZW3cEM6pmaejv87IlntkopI6WU5xc696KUspWUsp2U8o+ai1o3LHnuOTIO7uR2enER9zCBMQzkFa/PU9rmm+pyEZB4EkVK2mclYxL6D7lwH4q5oiW74ZIjcMHS/aiXDUPRZBFTQKx6pMAVStZObnUh9AvIayHDSLX46kEWvSqX1aw6dGkXxPfXpdL06AoG5f7H0T8H1Gg8IeDZK8eyz/4pEnORzcTq6slauOaQoLRmopzDxNCbeTDsReyu/ZD9ccUdDRokhsNgIZzZ2Sx97jkE4EMaPficliypleLmhQMwSuO9FFjpcbRwSo/itfmwpt8ofp04kfQ43YQ++QTMz4QO/8zVxy20QeSWRV+XZrv0Bm5V4fmfn2DVllu4dedHHHzmcfi/2rlbyOPKMbEcWzyUZbOHYrHUPGWc2QRdWk9AUSw1XJ3WHnP9upAmbLiEmThTE1bZuoFjG7hOeIJCDE4nDOVbiBMbykh7VQ9owKJsWJ4Nay+/hWNzV/BpTHf++Ggmmz79lLeio/m/Tp2w/zoHJ5DuG4DmKc2tmwHUfOf9PCpl9zFXraaNJmFT/07s694KzceM09/MdxOHgl8Fdbp2boOvZsK+0tzE6wHHf5A0CmTZroV5SKlvYiZrwSRqIXUgnM6I7L+xyrwoOYGfzAb7V5DQBE76Qe7vdSaLQc0xEusUIqRlS4SqIl2uil2o6gAN+DsbmPEppq+/LRK+DHpe4W4PXE+H65dz97yP2ZDtxiR8+cQ+jOGWA1xt3QHoysIu4Zs0uD4IzOVdcp1Vi/Kwo/DJqe7kKroLlMmlEWSpIEpvwxq4ZKj+XAj4azW071Sleb2KlgjJHsVbUQ4fQEMhVQYSKDJQK3dJ8wrN3HFMTPuSrwMncF7Ov/Sybyx0VkLK1RBVG9E1BrWBoXwLEdCkCYHR0aQdOlTvirc4xRVvHnaH5PLZH/JllhOnBIVsdpkVkG0ZI/dgw4UL+CgVcjU46YKY4mWOAoJISrVzU8Y49mrhrA6eSZBSOSVsBobt34l8OZU/br+A5ocTuaH3GChvAb3wd8jxrDDNFv11fSpf9/FCPmCeY+XYxhU0wpTU2paqBAIYmbuEkblLymhx2uxpG2CYHUpQkzpvtY0oltBdKiYSCeeE5o+G/tNzAlf1TOX8z19EvPsJh53wRSqkafpKOqgU86hMT+O2rDHMc3VgtxbOXZnlV5vM25ByA++4BtLafowrP/qNT3tM5onxr+C/e1/pHY8ehiULoWUb8PGUiXA64NVnYd2q0vvUBaaOYPIkU6goJyRF7cGyljYxq0XAy/UtgUEVMJRvIZzZ2cRv2lTfYpSJ9GQK63DllYyZOZPtzd/hS+7l4ZxLcHmymEVYzQzs2ZbLL++IzzU30Dwmiq5W6GmFG4MgsBTlKwR85J9nLxQ87bu06LzoiluiK5q59ja0TbmbRzJHsr7JuSx3RpMtTXqqRaer9HDit/4HPVvAlaPgruuLZkPLzYEL+8N5XeC/pSX71jbCBOGrIeDNokq3HAVcOFii/jfnVPB/E/yn1LcgZy1CCJsQYo0QYrMn0+NzFfUxzA6FcOXmornKrzJRiYVRrSKBlb/u4Nb5s2iUtRAXAzkgjzKHvvwQcjetFSfi+5mwbwssWI3y3Xz6juhdtMZaobHy3keoyOZS8w4WOlsRo6SXaJO3eScEjLQe5EhmEO/k9uPN5i6e2TWMZM2XdmoiFwYmYfHzLyn4G88XfV1aqaOd2+Cq82HdfoiqZh7e6iKsYGpX+eb1rnALEfgJ+IyvbynOduzAMCllphDCDCwXQvwhpSzzls5Y+RbCJzSUoOYVJ4GphFmw1hCA25FDRpaZI5xHf96lC8u41fIBbVQXSt5KbOMaXVt27g6bjsLsP6Fj13z/4hzVTKJfcP57MAn4MfB7UsNewke4Cm6lPX8LJ9ExoWETbsxoXLrqfXYFv8/5lv2Mse4l0JkOq/4tKXhpCrk0HHY9GXx9kDOnfuYtk0rmcUi/CeIDwHWoNoUxKAepk7fbafY8ylURhvItxtU//VTu+cJhwfWx+NGAJTzpkUUSzRrGcBf+yo6iDQvfDzeO0nPr7tiSv4pNCQoj8rfEEl1UUXKYwn81CdNzepIlzQSLXBqrWUSo2fQ1x+ErnLoJIaKUPLo/LASLhWI6vXS69Sr/Q6gNnDsh95u6n7dcqhJC7ISMx2pNEgMAwoUQ6wo9ihSmEkKoQohN6BmSFkopV5c3mKF8ixHVsydN+hZEZ6llJDevr7tOAXTkFwQuollFG/4AITia6+I/u8fxyWSGT4qt4pSixt40/2DMLifr2vQosWEkSokAKewM0MGUxIM+K1kTPKNk9jRFgQ6dSwrepTs5l91Y8ecmFLDZKmrlfcQZ8FNQapA83qAyJEopexd6TC98UkrpllJ2R08k1kcIUcoPoYAz4BvnXU5s3MipLVvyl3ruUly86tPcJ4D2zOVRgmlkO8bcVkfZbx4HwN8Zbj5p3gtOOGDM5UU7+vvDtDdACOKVAK497xkcX6zh/p192RvYqtR5StvJFwKGWw7yit8iotV0fSOucLtzzi1T9mXn3EqW6ke2KN0PTQLcO7UyH4P3MbUDv0fRMxarYLkAgv+pH1kqhQrhp0BtpT9Xu0KAlyuXGFQLKWUq8A9wQXntDOVbjCPLlzcw/6GSCMBCDufkvsiO/YF855hFGtGYfHxoMWRI2R3vfABOafzxylJ2Tz+AePcf9jga0TrjQImm7uatuDF9LA7PVyQvfaKrUJYvcc0tKHMWIP7dirh9il46/vu/ypw+YnAfelywgwc7v1HE7CAlZGhmvg0eCVMr3CSuPQKnQZRLf4T9ATK1/mSpEDekXQERe3V5G20Gxcj1W18IISKEEMGe5z7ASGBXeX0Mb4ditBg8GKEoKGYzSJnv/WAJCiK8QweOr6pHf9RCKGgEcJKmrCHBdxDNxjxA/3NVzrnrrgr73nRTD4KCbGzadJLxoYdRXiikCq1WOJaLIiVfKc8xxrGXiyz7EEjuyriIP51t2R7yf4QpObopY8hIvd+Lb1c4b8/m8NCEMO55/TwmqS3p4DyEWUjmOVpzSca1mNIVrjE1oK+kY3F9S1A+zmWQ9a7hYtYwiAK+EEKo6Iva76WU88rr0IC+6Q2DyK5duXn5cg4sWsSOH37guKeApiMtrcEo3jxM5DCcqaBEss0ZytfKOC5MNvF8JVL4XnppBy69tIOeOf7V28Gh55xltG6uEELw+OMDufp/GgNNR9inhXFCCyQAO4udsVxp3VG6bbcCLuslmLL0Z/o7JzDEfIggk4t5ogu+voKBA2MqHqAu0U6D/P8Zj4LfPWeGzfo0Rkq5BehRlT6G8i2FqB49iOrRgw0zZ9a3KOUi0IhhNTITxE8QOf97fvCdy9gJwzmnskWQ/f1hSxx89Ba0bg9XX09urourr/6Bf/45yJzAHznkCuTp7GEAuBG0VxMhLAKuubnKMoeH+/LZl5cx9aG/SA2P5N3ZExi1/AgA113n/aoXNcLvdrD/gR7L56hS16pWrKgQ//fBFASZM8FVOBDFgf4FCPTSRAZ1haF8y+GcyZP5a0rDvaUTxf6qTgdNtq8lWxtetYHCwuGJF/NfzpixngUL9pOb62Rc2E4UM7hR+M/VjGGv3kGX8ydAu07V1izjx3dm/PiCVXPbdg00pNs6CsLXg2snZH8KjspnDcuzkRfeOqi+IhaQ6fkeWoaCOgC3ewUbAruQZAknlt9px4TqDm5QT9RI+QohXgPGoC8L9gM3e3b6EEI8BkxEXzbcK6UseyemgdL9xhv55+mncaQ17GoBEnBZbUhFJd1l44aAW3H5RzLzuxu58KK2VR4vK8uJpmmAYI87jDZqMvf5rOIC936G/TueKVO8W1apQWPuoD8sI+BU1coV5SngmiPJT5rjXAVSZXVQD475NAUh2Cy/w0IksQzzxmRnF/HAm/UzdU0NRQuBzlLKrsAe4DEAIURHYDzQCd3d4gOPIfq0IXn/ft6KiWnwihf0la9qz0VKjf3vHuSIjOZ4hoWLL/6WhISsKo83aVIvYmNDMJkEl6l3MNPVmxm5PRmZcQMZ2Wdp7TA1EHyrfhdU89JBRXFLyf3ZT5NoCSsUAQMnaTi5qA0qR41WvlLKBYVergKu8DwfB3wnpbQDB4UQ+4A+wMqazFeXbPz0UxwZGfUtRqVRAGtuDmkEkmeIkMDq1XGMHl211W9oqA87d04mM9OBr6+ZVmO7cPjPXRDqxzPTzuLVVdDbIEIgaxp6rGFd48sn9jt4134vwzL/gqCCTCMtxFn8fzlN8abN9xYgr7RuU3RlnMcxz7ESeEL0JgE0a9bMi+LUjOBK5HhoaAyYOpX2755kR7b+UZtMCv36RVdrLCEEAQF6MMTBeVexMc1FiI9KrLUhZZSpB/zvhuz3QCbV/dyB7/HS8VvQgHHH/+DKzO+5r8kxWikX0Iiqe54Y1C8Vmh2EEIuEENtKeYwr1OYJwAXMqqoAUsrpeeF6ERERVe1ea/SYOJFmgwbVtxhV4uA//7A1YzovPD+UG27oxubNdxAe7luzQaVEfPQkAVMGs/3Zl0lNza24z5mMEgaNDgKhdT+3uRNfNYVwFXyEwii/8fRXHjIU72lKhStfKeWI8s4LIW4CRgPDZUHN8jigsNNmtOfYaYOiqty8bBm/3XknGz76qL7FqRQ5iYkoiuCJJ724ITbjGZxHXqJJlEJTuYbx5x1n9uo38fEpI5rKsQ5Sr9ZL8gR+BD7jSm93OqMEQFQS5C6CzPfAObdu5k06j4E4SAg/H0L/aGB5LQ2qSo023IQQFwCPAGOlLFJ5cC4wXghhFULEAm2ANTWZq74Y8+GHPJqezl3bt/Ok00njHgV+1CabjW633NJgfgQXf/ih9wfd+xdSgJ/Fhc3son/EVrZvTyi7ferV4D4A2klIHQ+yiht0m9eTdvXlHJ88BZnRwOuR2UaA6sVkNhV6Rnh8jR1/gf03781rUC/U1NvhfSAAWCiE2CSE+AhASrkd+B7YAfwJTJZSnrYFpmwBAUR07IhqMtF88GBMnkxnQtUrBDeEPBChbdrQauRI7w885IZ8pZDjNLM2vgUtW5ZTsVcW9q5wo1ujKklGOr/e+CKzDkTi+O0X5o+4tdSc6w0K25iaj5GXo7QqaBVXWTZo2NTU26F1OedeBF4s6/zpyoiXX8bi50fizp30u/9+Di1dqlc8LqVSRF0hTCY6XXVV7Qx+8WQUHysbfpvF6tw+PPftA4SGFg2fKxLNFfgRpE4A3BDwIoiyQ+3suFjOUcyoDCCaJz9P4ZVu3yKFwLfdiyxdNYJNR/WcEA0WnzHgfh8ynkD3Mkmt+hhVzc4vmoDPlVWfx6BBYUS4VRGT1cqwF17If92kd28O/v03h/6p2/SDUb17E9m1K9mJiXS8/HK6Xn+9V8f/9K0U3ng+m66hR5gxbxQ937qVnqW0W5QJD6/cyzW/zaRf2+YMumMSjtBkVFVDNfmVPnhuLpjNPK0u5aBHWa3lOB+sbom06R4W2ZrGo80eY0YF+4Ur98N1M8CpweC2EBEAD50PTYKr/darjv9k/QHg3AuJHahWJeHyrFemHiBTwDYBAv9XHSkNGhiG8q0hJpuNGxcv5uDixXw3bhyOTO/ZKU2UcdMuBAIYO2MGQvF+QpVt2+DOBxNwyBXsSvHD2m87n6feWmrbe/ak8d+kPgRlpZFr8eHbOZu4NvUKbAc28PMPV3D++UVvjk7dfTehsz8kw8eX3Yc/IS9D5VqOE5JtI10NBk9ms5aXDKNlOQ4wOQ4Y/Co4PXrua49z43drIO71ejLFm1pRLcVbZmkUfwidC9ahNZPLoMFhpELyErHDhiG9ZKDMCwUUwN3BpfwmpeTU9u2kHzvmlfmKs3dvDg45BziMxg5+SktCEc9w222/IaVk7ZqjLFlyCLdbo8XxA5g0N6qUHMvyYdvGI8jwaHK6juCWW4p5ARw/RuDsjzGhEZyTSdi+eFQEZhTaEca855sSvGsxYutyJgU/y4w7dpcrZ3w6uErRcyfSIMvuvc+jSggF1NKCWiq4EhQrR5/l8CHJfT1EHjUU7xmKoXy9SQ2XWmpTK+2Oj6arn0obM1wfBKE2M6pa7N8kBGYfH/waVSJ3JOCWeu21ytKmTTaFI7gyyEGi8MXnG5l3cx96/daMjOkXccVl33J1/4Nk+fix1dWIXqm380pGH1g8BxSFU6eykIU3I82WfAOxlHDdBf/HFbI94+nEYwygXdtQbuoxmwHiCwa3/x2SR4EsWUkkj2ah0Dys9HP+9VCJKJ+ITWCbAoQAwWC5CNTK+eK6NdgW34Gxs+by4r/vgBJce3Ia1CuG8vUiF73/fpUVsFBVfJqFEXhdNG0OjMQUZaXHX9O4pnkoMYG+iDc+5oIPPyI4NpbWF15I3/vvp/tNN3HLihWYKlHr7IGFxzC1+xA14CUGXf9rpWTq2DGUqChfCnaA9PfkdGkkpzhQBAyJ2Yd6cB7xh37gv8HRPJw1khxMuPPW7ct/w+XSyMpy8MILy2jc+HWGXDmfR8xXstsVylpnE9YOe4hrRBeCFlpoGvYm/v4v8f4n0Sxf04zbHhrL+s2hoKWWKaeiwMwbSx5vV7lrUu0hfCDkbYhKhqgUCPsdAp6h4J6mePvGgIW03ADm7R7L8M+XsOroubSLCq47mQ1qhBAiRgjxjxBihxBiuxCiwkQghs3Xi3S/6SaS9u5l5Ztv4s6tXCSY2ceHsd/OZMe53wECgULogNtg7+P5bXoBvW67rcryuCW89doK2J8AmmT5D9tYc08v+vQpP+RYUQSffjqOsWO/xeksakrZcDKKG7ttAQR2u8ozg1txjtNGSyUZrci2vUBRBC+//B9vvbWK7GwniYnZLHW35z3a6/P8mMuFz8CECT+SnJy3wjV5ZJCs3tyHniMjy71hn/RlyWM7Xih5rN7xuRzEH5ByKVAs2ZGlB4T8TlDagyRuz6FDxCGGd27HbefVMDrRoC5xAQ9KKTcIIQKA9UKIhVLKHWV1MJSvlxn63HOoFgsnNmwgNzmZuHXrkG43/o0bk370KACKxYJiMiGEILJbN9r2GU0LhpDGIYJpiRXvJMYWAHZnIT9kQVZWGUEPUhZZtZ9/fiueeWYwb7yxkpQU/UJisygMCLXjXKuwIiea3/e0QqKwnOY0s6Ryk2UD+7QwVrlicKFiNiukp9tRFE8xUndR24eWksQ54xbiSC++raiRnWNm8qOdePx/r7Jnz900auRfqthHkkseq4U9SO9gGwmN0yH5Ak+JIjeIGAh8R//sg99k4miYOLq+BTWoKlLKE8AJz/MMIcRO9Hw2hvKtKxSTiSHPPAOAlJKUAwewBQdjCw5m3ccfk370KL3vuAOLvz9Z8fGEtWuHoqrYCMZGd+/KIuCW67ry6dbjkJpDy66NGTKkRdFGibvh8xGQeRKa9oW0wxDVA3HFLKZOHci86f/QNHMbB90hXGvawuWbNyEdkoEcZaT5AAucrTGbFVZFD2ZN6mOoSAan3cQ2dyNatghl2rSh7N6dyD//HCIgwFpohatj37oeBl0CS38GtwtfXxN+fmYSEvR2aWl27r77D77/vqRf64f/SFwaFN6tWvukVz9C7yMUCFtQcTuDhki4EGJdodfTi5ePBxBCtEAvKbS6vMEM5VuLCCEIbVVQlr1PseKWvmFl7BZ5kU9u68K717QjO9tJREQpfre/3wvpcYDkf98KZm68nM4Rp/hIPsnUBUNYe8TNKjoB8LhYhuZwYhYSE05GeJSv1WpiwoQuyDmNCEg6zOaQj0jCj5BtaSgmlT//vI6kpByCgqy88spynnpqScH8foEQ2xGiW8P37zJ0aBP8/Mx8/33BgkErxYtk9epj3PVZhL6Jp4cZ0qWpoHcLb356BgZFSJRS9i6vgRDCH/gRuE9KmV5e24Z6g2bgRfz8LKUrXtBXYgIWHojlxeXncTA1hN/2tiXmhmDmzNnh2UATgOCB7AtwSgWHVMiSJv5y6heWzEwHb7+9is0PfoSzRz+yWnUheN5fKCZ9g0kIQXi4L2azytGj6ZhMnq+drz+MmgBOO6QlQW4WwcEWOneOwGpVPbKbue++/vTv/wmdO3/AkiWHAJh0x++wez1sXg452SDh/kEVJ44/eeIQm1ZPJzWpzLtBA4NqIYQwoyveWVLKnypqb6x8z3Yufpcplz7JB/+1Rc13LxNoUpCb66Kw9/8RGUyz5Pu5xWcjKxwx/OcuiPt1ONzM3S4ZvnAl6el2+gz9gs2b/2bo0Fh+//0aLBYVp9PNzJkb0Tx+b5bcdBz/zQOrDxzWfXpnzdqeP6avr4no6ACGDPk83148cuSXPPXUeWzZcgrEP7qtev9WGHsr2Xt2wIhzynyrO7ZtI4J+tAx1oaZpJNjnE9Gk3KR9BgaVQgghgE+AnVLKShUmMpTvWc72k0HMXN8Nl+bElZ/hpbB/QWF3M0ESvryWMzD/rKqCpoHTqfHRR+ux2UxkZzvZsOEEAIsWHeCjj9YxefI5fP/9Nvz8zGRmOhACosNU2pz8g7/cbYvNqZOd7WL37qI7ai6XZNq0ZR7HZc/FIi0RTh6hQ4fy80FvXvM1owc5CfDXs4MdOfyeoXwNvMUA4HpgqxBik+fY41LK+WV1MJTvWU7O8cMMsC8kkUA20oOSSrB41pei51VVRVE0nE6Jw+HmtddW0LVr0TSLmzadpGvXD9mxIxHQTQkdOoSzbt0JDpAXDVZmfG0JintNYPOD/ZvZ6Li83BKSfkHdUFS9b1a2GcVaWrYKA4OqI6VcTmW/wB6EbADpEPPo3bu3XLduXcUNDbxCbmoq78TGkpOahhMTP3IZuz0+uDolv0tCQMeOEezYkYCUYLWq2O3l5zJQFEqkhvT1NZPtrWKc4U2gzyjCdy2iQ1OFr766lObNg0s0k1Ly87evEBk4B7PfefQZ8oZu8zY4LRFCrK9oA6ziMXpLqKzOqfl8hTG+eWcxSXv3IjUNgcSCkzSCyDMvlHURlxJ27UpEVQUhITaef77ivAOlpbzwmuIFaNoS5n9O4oFj/PvvETp3/qDUZkIILrtmKgNGr6fP0LcMxWtQrxjfvrOYiA4dMPn44MKEAzNaJb8ObrfE5ZJYLIL33y/XlbFuOFw0AU9mppPHH1+Ew3Ha5u83OAuoaRmh54UQWzxVLBYIIZp4jgshxLtCiH2e84ZxrQFi8ffnjk2b6HDf0/zAFZwiggLbbkXmKEl8fDZHjpTrylj7CAVSS5Y1euml/7j88tmldDAwaBjUdOX7mpSyq5SyOzAPeNpz/EL0um1t0MvC10JxMQNv4N+4Mfauozlgag/5Pr1Q8d5BZdvVMlIjKMha6qmFCw/UsTAGBpWnRsq3WASHHwXLpXHAl1JnFRAshIiqyVwGtcPevUlMmjQPl6s6G69lK966TGQ+eXKf/PwRhXG7JceP1/PK3MCgDGps8xVCvCiEOApcS8HKtylwtFCzY55jpfWfJIRYJ4RYl5BQTlVcg1ph375kVLWqmrK4oi6puOvSiWbZskMIUXJCl0ujT58Z2O1VKOJpYFBHVKh8hRCLhBDbSnmMA5BSPiGljAFmAXdXVQAp5XQpZW8pZe+IiPKd5A28z8CBzYiMLD1jWNkUD8KoX9PD8uVHKat+aVxcJoGBL7FqVe1U/TAwqC4VKl8p5QgpZedSHsUzc88CLvc8jwNiCp2L9hwzaGAEBFh5550LajBC3StetYyc5GXhcGhccYWx+WbQsKipt0ObQi/HAbs8z+cCN3i8HvoBaZ58lwYNkCuvnFPfIlSJsla55REXl8m5587wvjAGBtWkpjbflz0miC3AKCCvdMZ84ACwD5gB3FVGf4MGgMvlncKfeTTUZOYrVx6nffv36lsMAwOghrkdpJSXl3FcApNrMrZB3dGjR2M2bjzptfG8VMS5ViieqMfAoL5ooGsUg7pk/fpJXHFF+4obGhgYeA1D+RoghGDOnKsZNCgGIfQCmvfccw533eW1HCLeQQFCalZUcsWKm7wiioFBTTFSShrks3TpzWzeHI+fn5k2bcJ45ZXl9S1SAU2CYPZt4G+Fzcfgtq/AWbp9w9fXRHZ2Sd/efv2a0r9/81J6GBjUPcbK1yAfIQTduzemTZswpJQ8+eQ/NRiLagRvlMNdgyHIBywm6BgFA1qX2VSvwFGS99+/0HvyGBgUQwjxqRDilBBiW2XaG8rXoExqojylLCXpeU1IzwG3Z6UrBGTkltm0tA2/sDAbvXqVGmRpYOAtPgcq7TRvKF+DUtHtwFfi42NCVQXNmgVWWxn7+FQxKqI0PlgGqw9BYia8OB/WH6lS98ceG1RzGQwMykFKuQyotDuNYfM1KJMxY9qRnf0EAIMGfZafPtJkEthsZlwurcxb/MLk5Hghr26mHe6YVe3uo0a1qrkMBgZexFj5GlSKqVMH4OtrIiDAQufOkSQlPUJm5mO1Pm91s6MJoV8k/PzMvP32+XTpEllxJwOD8gnPSwLmeUyqyWDGytegUlx8cVu2b5/MsWPpnHNOEywWld27ExGi9AxmMTGB9O4dxc8/7y55sgqMHNmShQsPVJglzWRScLs1pIQnnxzE8897SmlqWeBcD+7joDapkSwGZz2J3qzhZihfg0rTokUwLVoE578OC/PFalXJzdXNCnmrVCnh6NF0jh6tTC7d4lWRi2ZJq4ziBdi1azKtWoUipUR4BNFcqSTvaYvFlI7ZDO6gBfiHnlcJmQwMah9D+RpUm/BwX3766WqeeGIxMTGBjBvXjokTf6viKAVKt3FEBmGhOWzfXWAiqIziDQiw0LJlCFde+T05OS7mzp2AogjWrfiCDlFpBPg7ANi4eRo9hi6qonwGZzbHgee8MpIQ4ltgCLp54hjwjJTyk7LaG8rXoEZceGEbLrxQT27311/7Sm1TlmnCbFZwu11omoLV4qJ39zh27a16TudzzmmCokzLf62q05DyGVIzmqBG6xNn5Zg4kRBDjyqPbmBQOaSUE6rS3thwM/Aao0a1YvjwFkWONW7sV2aWM6dTQ9MEAg27w8S8hR3Ydyis3Dl69y5ZjcrPz1Li2LZt8QwZeRlvfz6RZauaMevnIfQ6781KvxcDg9rGUL4GXkMIwaJFNzJz5hgaN/and+8mrFlzG8HBPuX1QqJQ2YKcr702En9/c77PcVCQleefH1qiXUxMIBaLyuPTPqT/6ANMenAhkY1DqvfGDAxqAcPsYOB1Jk7sycSJPfNf33xzd15/fWWNx+3VK4ohQ2LZs+cetm8/RfPm+gag2axy3nnNWLZMD7zo3DmCoKAChW82eyHIw8DAyxjK16DWefXVkSQn5zBnzg4yMhzVGiM6OpB163S3yqioAKKiAoqcX7r05hrLaWBQlxhmB4NaRwjBJ5+MIz39MZz/3969hlhVhWEc/z8etQQpS+lCaipZMJaQDCb4RUrQTNQgUokyCqIwKBAqk6APEd3oRiVoBgrhhS4oUpiZSF+stJtZVlLaBc2EzDAoJ58+7CWe1JnpzDkz++w97w/Es/beZ896Ofi6Z5211nv0IRYsmFDzPZYund4NPQshPw1JvpIWSLKkIaktSc9L2i3pc0njOrtH6B369u3DU09NYdKk2rZ2nDKl/V3MQiiiupOvpGFk9duqdzq5Fhid/twBLK7354RysM3hw3+xYsX1DBjw/0a9sk3e8y1PH0KjNeLJ9xngPk4sVYKskvEKZ7YCgySdOkco9Cq//nqESy99gcGDn2DWrNWcc86JL8UqFTFv3lguvvjs/7ynpWUI69ff1NNdDaHb1fWFm6SZwM+2PzvpyeQi4Meq9k/p2Cnl49PmFHcADB8+vJ7uhCa3ePE29u49RFvbMXbtOsjs2WNYufILJJg793KWLZuZdxdD6DGdJl9J7wIXnObUIuBBsiGHLrO9BFgC0Nra2sDdt0OzGTiwP5VKH44ePYYEkyeP4tFHr+HIkb8ZNSrm4IbepdPka3vy6Y5LugIYCRx/6h0KfCxpPPAzMKzq8qHpWOjF7rqrlS1b9vD++z8wY8ZlzJ49hkolJtyE3qnLww62dwDnHW9L2gO02j4oaR1wt6RVwFXA77ZPGXIIvcuAAf1Yu7am5e8hlFZ3LbJ4C5gG7Ab+BGIGfAghVGlY8rU9ouq1gfmNuncIIZRNDLiFEEIOIvmGEEIOIvmGEEIDSJoq6eu0rcIDnV0fyTeEEOokqQK8SLa1QgswV1JLR++J5BtCCPUbD+y2/Z3tv4FVZNsstKup9vPdvn37QUl767zNEOBgI/rTxCLGcogY61Pb1nintW8DPDzkf158pqRtVe0laYUunH5Lhas6ullTJV/btVdPPImkbbZbG9GfZhUxlkPEmD/bU/P62THsEEII9at5S4VIviGEUL+PgNGSRkrqD8wB1nX0hqYadmiQJZ1fUngRYzlEjCVhu03S3cAGoAK8YntnR+9RthI4hBBCT4phhxBCyEEk3xBCyEHpkm+ZKylLelLSrhTHm5IGVZ1bmGL8WtKUHLtZl1qXaBaBpGGSNkv6UtJOSfek4+dK2ijp2/R34ct5SKpI+kTS+tQeKemD9HmuTl9GBUqWfHtBJeWNwOW2xwLfAAsB0jLGOcAYYCrwUlruWChdWaJZEG3AAtstwARgforrAWCT7dHAptQuunuAr6rajwPP2L4E+A24PZdeNaFSJV9KXknZ9ju221JzK9lcQshiXGX7L9vfk21iPz6PPtap5iWaRWB7n+2P0+s/yJLTRWSxLU+XLQdm5dLBBpE0FLgOeDm1BVwNvJYuKXyMjVSa5FtdSfmkU+1VUi6624C30+uyxFiWONolaQRwJfABcH5Vea39wPl59atBniV7+DmW2oOBQ1UPDKX7POtRqHm+3V1JuRl0FKPttemaRWS/yr7ak30L9ZE0EHgduNf24VR4Fsiqv0gq7LxPSdOBA7a3S5qUc3cKoVDJtzdUUm4vxuMk3QpMB67xiUnahYqxA2WJ4xSS+pEl3ldtv5EO/yLpQtv70lDYgfx6WLeJwAxJ04AzgbOA58iG+fqmp9/SfJ6NUIphB9s7bJ9ne0SqJfcTMM72frIlfrekWQ8TKHAlZUlTyX6tm2H7z6pT64A5ks6QNJLsy8UP8+hjnWpeolkEaexzGfCV7aerTq0D5qXX84C1Pd23RrG90PbQ9O9vDvCe7ZuAzcAN6bJCx9hohXry7aIyVVJ+ATgD2Jie8LfavtP2TklrgC/JhiPm2/4nx352SVeWaBbEROBmYIekT9OxB4HHgDWSbgf2Ajfm071udT+wStIjwCdk/wkFYnlxCCHkohTDDiGEUDSRfEMIIQeRfEMIIQeRfEMIIQeRfEMIIQeRfEMIIQeRfEMIIQf/AnOS2zJBfDHFAAAAAElFTkSuQmCC\n",
"text/plain": [
""
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"model = utils.build_model(backbone, second_stage=(stage == 'second'), num_classes=num_classes, ckpt_pretrained=ckpt_pretrained).cuda()\n",
"model.use_projection_head(False)\n",
"model.eval()\n",
"\n",
"embeddings, labels = utils.compute_embeddings(loaders['valid_loader'], model, scaler)\n",
"embeddings_tsne = TSNE(n_jobs=num_workers).fit_transform(embeddings)\n",
"vis_x = embeddings_tsne[:, 0]\n",
"vis_y = embeddings_tsne[:, 1]\n",
"plt.scatter(vis_x, vis_y, c=labels, cmap=plt.cm.get_cmap(\"jet\", num_classes), marker='.')\n",
"plt.colorbar(ticks=range(num_classes))\n",
"plt.show()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"embeddings, labels = utils.compute_embeddings(loaders['train_features_loader'], model, scaler)\n",
"embeddings_tsne = TSNE(n_jobs=num_workers).fit_transform(embeddings)\n",
"vis_x = embeddings_tsne[:, 0]\n",
"vis_y = embeddings_tsne[:, 1]\n",
"plt.scatter(vis_x, vis_y, c=labels, cmap=plt.cm.get_cmap(\"jet\", num_classes), marker='.')\n",
"plt.colorbar(ticks=range(num_classes))\n",
"plt.show()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.9"
},
"pycharm": {
"stem_cell": {
"cell_type": "raw",
"metadata": {
"collapsed": false
},
"source": []
}
}
},
"nbformat": 4,
"nbformat_minor": 4
}
================================================
FILE: tools/backbones.py
================================================
import torchvision.models as models
# for timm models we don't have such files, since it provides a simple wrapper timm.create_model. Check tools.models.py
BACKBONES = {
"alexnet": models.alexnet,
"resnet18": models.resnet18,
"resnet34": models.resnet34,
"resnet50": models.resnet50,
"resnet101": models.resnet101,
"resnet152": models.resnet152,
"mobilenet_v2": models.mobilenet_v2,
"vgg11": models.vgg11,
"vgg11_bn": models.vgg11_bn,
"vgg13": models.vgg13,
"vgg13_bn": models.vgg13_bn,
"vgg16": models.vgg16,
"vgg16_bn": models.vgg16_bn,
"vgg19": models.vgg19,
"vgg19_bn": models.vgg19_bn,
"densenet121": models.densenet121,
"densenet169": models.densenet169,
"densenet161": models.densenet161,
"densenet201": models.densenet201,
"inception_v3": models.inception_v3,
"resnext50_32x4d": models.resnext50_32x4d,
"resnext101_32x8d": models.resnext101_32x8d,
"wide_resnet50": models.wide_resnet50_2,
"wide_resnet101": models.wide_resnet101_2,
}
================================================
FILE: tools/datasets.py
================================================
import torchvision
class SupConDatasetCifar10(torchvision.datasets.CIFAR10):
def __init__(self, data_dir, train, transform, second_stage):
super().__init__(root=data_dir, train=train, download=True, transform=transform)
self.second_stage = second_stage
self.transform = transform
def __getitem__(self, idx):
image, label = self.data[idx], self.targets[idx]
# leave this part unchanged. The reason for this implementation - in the first stage of training
# you have TwoCropTransform(actual transforms), so you have to call it by self.transform(img)
# on the other hard, in the second stage of training there is no wrapper, so it's a regular
# albumentation trans block, so it's called by self.transform(image=img)['image']
if self.second_stage:
image = self.transform(image=image)['image']
else:
image = self.transform(image)
return image, label
class SupConDatasetCifar100(torchvision.datasets.CIFAR100):
def __init__(self, data_dir, train, transform, second_stage):
super().__init__(root=data_dir, train=train, download=True, transform=transform)
self.second_stage = second_stage
self.transform = transform
def __getitem__(self, idx):
image, label = self.data[idx], self.targets[idx]
# leave this part unchanged. The reason for this implementation - in the first stage of training
# you have TwoCropTransform(actual transforms), so you have to call it by self.transform(img)
# on the other hard, in the second stage of training there is no wrapper, so it's a regular
# albumentation trans block, so it's called by self.transform(image=img)['image']
if self.second_stage:
image = self.transform(image=image)['image']
else:
image = self.transform(image)
return image, label
DATASETS = {'cifar10': SupConDatasetCifar10,
'cifar100': SupConDatasetCifar100}
def create_supcon_dataset(dataset_name, data_dir, train, transform, second_stage):#, csv, second_stage):
try:
return DATASETS[dataset_name](data_dir, train, transform, second_stage)#, csv, second_stage)
except KeyError:
Exception('Can\'t find such a dataset. Either use cifar10 or cifar100, or write your own one in tools.datasets')
================================================
FILE: tools/losses.py
================================================
import torch.nn as nn
import torch
class SupConLoss(nn.Module):
"""Supervised Contrastive Learning: https://arxiv.org/pdf/2004.11362.pdf.
It also supports the unsupervised contrastive loss in SimCLR"""
def __init__(self, temperature=0.07, contrast_mode='all',
base_temperature=0.07):
super(SupConLoss, self).__init__()
self.temperature = temperature
self.contrast_mode = contrast_mode
self.base_temperature = base_temperature
def forward(self, features, labels=None, mask=None):
"""Compute loss for model. If both `labels` and `mask` are None,
it degenerates to SimCLR unsupervised loss:
https://arxiv.org/pdf/2002.05709.pdf
Args:
features: hidden vector of shape [bsz, n_views, ...].
labels: ground truth of shape [bsz].
mask: contrastive mask of shape [bsz, bsz], mask_{i,j}=1 if sample j
has the same class as sample i. Can be asymmetric.
Returns:
A loss scalar.
"""
device = (torch.device('cuda')
if features.is_cuda
else torch.device('cpu'))
if len(features.shape) < 3:
raise ValueError('`features` needs to be [bsz, n_views, ...],'
'at least 3 dimensions are required')
if len(features.shape) > 3:
features = features.view(features.shape[0], features.shape[1], -1)
batch_size = features.shape[0]
if labels is not None and mask is not None:
raise ValueError('Cannot define both `labels` and `mask`')
elif labels is None and mask is None:
mask = torch.eye(batch_size, dtype=torch.float32).to(device)
elif labels is not None:
labels = labels.contiguous().view(-1, 1)
if labels.shape[0] != batch_size:
raise ValueError('Num of labels does not match num of features')
mask = torch.eq(labels, labels.T).float().to(device)
else:
mask = mask.float().to(device)
contrast_count = features.shape[1]
contrast_feature = torch.cat(torch.unbind(features, dim=1), dim=0)
if self.contrast_mode == 'one':
anchor_feature = features[:, 0]
anchor_count = 1
elif self.contrast_mode == 'all':
anchor_feature = contrast_feature
anchor_count = contrast_count
else:
raise ValueError('Unknown mode: {}'.format(self.contrast_mode))
# compute logits
anchor_dot_contrast = torch.div(
torch.matmul(anchor_feature, contrast_feature.T),
self.temperature)
# for numerical stability
logits_max, _ = torch.max(anchor_dot_contrast, dim=1, keepdim=True)
logits = anchor_dot_contrast - logits_max.detach()
# tile mask
mask = mask.repeat(anchor_count, contrast_count)
# mask-out self-contrast cases
logits_mask = torch.scatter(
torch.ones_like(mask),
1,
torch.arange(batch_size * anchor_count).view(-1, 1).to(device),
0
)
mask = mask * logits_mask
# compute log_prob
exp_logits = torch.exp(logits) * logits_mask
log_prob = logits - torch.log(exp_logits.sum(1, keepdim=True))
# compute mean of log-likelihood over positive
mean_log_prob_pos = (mask * log_prob).sum(1) / mask.sum(1)
# loss
loss = - (self.temperature / self.base_temperature) * mean_log_prob_pos
loss = loss.view(anchor_count, batch_size).mean()
return loss
class LabelSmoothingLoss(nn.Module):
def __init__(self, classes, smoothing=0, dim=-1):
super(LabelSmoothingLoss, self).__init__()
self.confidence = 1.0 - smoothing
self.smoothing = smoothing
self.cls = classes
self.dim = dim
def forward(self, pred, target):
pred = pred.log_softmax(dim=self.dim)
with torch.no_grad():
# true_dist = pred.data.clone()
true_dist = torch.zeros_like(pred)
true_dist.fill_(self.smoothing / (self.cls - 1))
true_dist.scatter_(1, target.data.unsqueeze(1), self.confidence)
return torch.mean(torch.sum(-true_dist * pred, dim=self.dim))
LOSSES = {'SupCon': SupConLoss,
'LabelSmoothing': LabelSmoothingLoss,
'CrossEntropy': nn.CrossEntropyLoss}
================================================
FILE: tools/models.py
================================================
import torch
import torch.nn as nn
import torch.nn.functional as F
import timm
from .backbones import BACKBONES
def create_encoder(backbone):
try:
if 'timm_' in backbone:
backbone = backbone.split('_')[-1]
timm.create_model(model_name=backbone, pretrained=True)
else:
model = BACKBONES[backbone](pretrained=True)
except RuntimeError or KeyError:
raise RuntimeError('Specify the correct backbone name. Either one of torchvision backbones, or a timm backbone.'
'For timm - add prefix \'timm_\'. For instance, timm_resnet18')
layers = torch.nn.Sequential(*list(model.children()))
try:
potential_last_layer = layers[-1]
while not isinstance(potential_last_layer, nn.Linear):
potential_last_layer = potential_last_layer[-1]
except TypeError:
raise TypeError('Can\'t find the linear layer of the model')
features_dim = potential_last_layer.in_features
model = torch.nn.Sequential(*list(model.children())[:-1])
return model, features_dim
class SupConModel(nn.Module):
def __init__(self, backbone='resnet50', projection_dim=128, second_stage=False, num_classes=None):
super(SupConModel, self).__init__()
self.encoder, self.features_dim = create_encoder(backbone)
self.second_stage = second_stage
self.projection_head = True
self.projection_dim = projection_dim
self.embed_dim = projection_dim
if self.second_stage:
for param in self.encoder.parameters():
param.requires_grad = False
self.classifier = nn.Linear(self.features_dim, num_classes)
else:
self.head = nn.Sequential(
nn.Linear(self.features_dim, self.features_dim),
nn.ReLU(inplace=True),
nn.Linear(self.features_dim, self.projection_dim))
def use_projection_head(self, mode):
self.projection_head = mode
if mode:
self.embed_dim = self.projection_dim
else:
self.embed_dim = self.features_dim
def forward(self, x):
if self.second_stage:
feat = self.encoder(x).squeeze()
return self.classifier(feat)
else:
feat = self.encoder(x).squeeze()
if self.projection_head:
return F.normalize(self.head(feat), dim=1)
else:
return F.normalize(feat, dim=1)
================================================
FILE: tools/optimizers.py
================================================
import torch.optim as optim
import torch_optimizer as jettify_optim
OPTIMIZERS = {
"Adam": optim.Adam,
'AdamW': optim.AdamW,
"SGD": optim.SGD,
'LookAhead': jettify_optim.Lookahead,
'Ranger': jettify_optim.Ranger,
'RAdam': jettify_optim.RAdam,
}
================================================
FILE: tools/schedulers.py
================================================
import torch.optim as optim
SCHEDULERS = {
"ReduceLROnPlateau": optim.lr_scheduler.ReduceLROnPlateau,
"CosineAnnealingLR": optim.lr_scheduler.CosineAnnealingLR,
}
================================================
FILE: tools/utils.py
================================================
import torch
import albumentations as A
from albumentations import pytorch as AT
from torch.utils.data import DataLoader
import random
import os
import numpy as np
import torch.backends.cudnn as cudnn
from sklearn.metrics import f1_score
from pytorch_metric_learning.utils.accuracy_calculator import AccuracyCalculator
from .losses import LOSSES
from .optimizers import OPTIMIZERS
from .schedulers import SCHEDULERS
from .models import SupConModel
from .datasets import create_supcon_dataset
def seed_everything(seed=42):
random.seed(seed)
os.environ["PYHTONHASHSEED"] = str(seed)
np.random.seed(seed)
torch.manual_seed(seed)
torch.cuda.manual_seed(seed)
torch.backends.cudnn.deterministic = True
def add_to_logs(logging, message):
logging.info(message)
def add_to_tensorboard_logs(writer, message, tag, index):
writer.add_scalar(tag, message, index)
class TwoCropTransform:
"""Create two crops of the same image"""
def __init__(self, crop_transform):
self.crop_transform = crop_transform
def __call__(self, x):
return [self.crop_transform(image=x), self.crop_transform(image=x)]
def build_transforms(second_stage):
if second_stage:
train_transforms = A.Compose([
#A.Flip(),
#A.Rotate(),
A.Resize(224, 224),
A.Normalize(),
AT.ToTensorV2()
])
valid_transforms = A.Compose([A.Resize(224, 224), A.Normalize(), AT.ToTensorV2()])
transforms_dict = {
"train_transforms": train_transforms,
"valid_transforms": valid_transforms,
}
else:
train_transforms = A.Compose([
A.RandomResizedCrop(height=224, width=224, scale=(0.15, 1.)),
A.Rotate(),
A.ColorJitter(0.4, 0.4, 0.4, 0.1, p=0.9),
A.ToGray(p=0.2),
A.Normalize(),
AT.ToTensorV2(),
])
valid_transforms = A.Compose([A.Resize(224, 224), A.Normalize(), AT.ToTensorV2()])
transforms_dict = {
"train_transforms": train_transforms,
'valid_transforms': valid_transforms,
}
return transforms_dict
def build_loaders(data_dir, transforms, batch_sizes, num_workers, second_stage=False):
dataset_name = data_dir.split('/')[-1]
if second_stage:
train_features_dataset = create_supcon_dataset(dataset_name, data_dir=data_dir, train=True,
transform=transforms['train_transforms'], second_stage=True)
else:
# train_features_dataset is used for evaluation -> hence, we don't need TwoCropTransform
train_features_dataset = create_supcon_dataset(dataset_name, data_dir=data_dir, train=True,
transform=transforms['valid_transforms'], second_stage=True)
train_supcon_dataset = create_supcon_dataset(dataset_name, data_dir=data_dir, train=True,
transform=TwoCropTransform(transforms['train_transforms']), second_stage=False)
valid_dataset = create_supcon_dataset(dataset_name, data_dir=data_dir, train=False,
transform=transforms['valid_transforms'], second_stage=True)
if not second_stage:
train_supcon_loader = DataLoader(
train_supcon_dataset, batch_size=batch_sizes['train_batch_size'], shuffle=True,
num_workers=num_workers, pin_memory=True)
train_features_loader = DataLoader(
train_features_dataset, batch_size=batch_sizes['train_batch_size'], shuffle=True,
num_workers=num_workers, pin_memory=True, drop_last=True)
valid_loader = DataLoader(
valid_dataset, batch_size=batch_sizes['valid_batch_size'], shuffle=True,
num_workers=num_workers, pin_memory=True, drop_last=True)
if second_stage:
return {'train_features_loader': train_features_loader, 'valid_loader': valid_loader}
return {'train_supcon_loader': train_supcon_loader, 'train_features_loader': train_features_loader, 'valid_loader': valid_loader}
def build_model(backbone, second_stage=False, num_classes=None, ckpt_pretrained=None):
model = SupConModel(backbone=backbone, second_stage=second_stage, num_classes=num_classes)
if ckpt_pretrained:
model.load_state_dict(torch.load(ckpt_pretrained)['model_state_dict'], strict=False)
return model
def build_optim(model, optimizer_params, scheduler_params, loss_params):
if 'params' in loss_params:
criterion = LOSSES[loss_params['name']](**loss_params['params'])
else:
criterion = LOSSES[loss_params['name']]()
optimizer = OPTIMIZERS[optimizer_params["name"]](model.parameters(), **optimizer_params["params"])
if scheduler_params:
scheduler = SCHEDULERS[scheduler_params["name"]](optimizer, **scheduler_params["params"])
else:
scheduler = None
return {"criterion": criterion, "optimizer": optimizer, "scheduler": scheduler}
def compute_embeddings(loader, model, scaler):
# note that it's okay to do len(loader) * bs, since drop_last=True is enabled
total_embeddings = np.zeros((len(loader)*loader.batch_size, model.embed_dim))
total_labels = np.zeros(len(loader)*loader.batch_size)
for idx, (images, labels) in enumerate(loader):
images = images.cuda()
bsz = labels.shape[0]
if scaler:
with torch.cuda.amp.autocast():
embed = model(images)
total_embeddings[idx * bsz: (idx + 1) * bsz] = embed.detach().cpu().numpy()
total_labels[idx * bsz: (idx + 1) * bsz] = labels.detach().numpy()
else:
embed = model(images)
total_embeddings[idx * bsz: (idx + 1) * bsz] = embed.detach().cpu().numpy()
total_labels[idx * bsz: (idx + 1) * bsz] = labels.detach().numpy()
del images, labels, embed
torch.cuda.empty_cache()
return np.float32(total_embeddings), total_labels.astype(int)
def train_epoch_constructive(train_loader, model, criterion, optimizer, scaler, ema):
model.train()
train_loss = []
for idx, (images, labels) in enumerate(train_loader):
images = torch.cat([images[0]['image'], images[1]['image']], dim=0)
images = images.cuda()
labels = labels.cuda()
bsz = labels.shape[0]
if scaler:
with torch.cuda.amp.autocast():
embed = model(images)
f1, f2 = torch.split(embed, [bsz, bsz], dim=0)
embed = torch.cat([f1.unsqueeze(1), f2.unsqueeze(1)], dim=1)
loss = criterion(embed, labels)
else:
embed = model(images)
f1, f2 = torch.split(embed, [bsz, bsz], dim=0)
embed = torch.cat([f1.unsqueeze(1), f2.unsqueeze(1)], dim=1)
loss = criterion(embed, labels)
del images, labels, embed
torch.cuda.empty_cache()
train_loss.append(loss.item())
optimizer.zero_grad()
if scaler:
scaler.scale(loss).backward()
scaler.step(optimizer)
scaler.update()
else:
loss.backward()
optimizer.step()
if ema:
ema.update(model.parameters())
return {'loss': np.mean(train_loss)}
def validation_constructive(valid_loader, train_loader, model, scaler):
calculator = AccuracyCalculator(k=1)
model.eval()
query_embeddings, query_labels = compute_embeddings(valid_loader, model, scaler)
reference_embeddings, reference_labels = compute_embeddings(train_loader, model, scaler)
acc_dict = calculator.get_accuracy(
query_embeddings,
reference_embeddings,
query_labels,
reference_labels,
embeddings_come_from_same_source=False
)
del query_embeddings, query_labels, reference_embeddings, reference_labels
torch.cuda.empty_cache()
return acc_dict
def train_epoch_ce(train_loader, model, criterion, optimizer, scaler, ema):
model.train()
train_loss = []
for batch_i, (data, target) in enumerate(train_loader):
data, target = data.cuda(), target.cuda()
optimizer.zero_grad()
if scaler:
with torch.cuda.amp.autocast():
output = model(data)
loss = criterion(output, target)
train_loss.append(loss.item())
scaler.scale(loss).backward()
scaler.step(optimizer)
scaler.update()
else:
output = model(data)
loss = criterion(output, target)
train_loss.append(loss.item())
loss.backward()
optimizer.step()
if ema:
ema.update(model.parameters())
del data, target, output
torch.cuda.empty_cache()
return {"loss": np.mean(train_loss)}
def validation_ce(model, criterion, valid_loader, scaler):
model.eval()
val_loss = []
valid_bs = valid_loader.batch_size
# note that it's okay to do len(loader) * bs, since drop_last=True is enabled
y_pred, y_true = np.zeros(len(valid_loader)*valid_bs), np.zeros(len(valid_loader)*valid_bs)
correct_samples = 0
for batch_i, (data, target) in enumerate(valid_loader):
with torch.no_grad():
data, target = data.cuda(), target.cuda()
if scaler:
with torch.cuda.amp.autocast():
output = model(data)
if criterion:
loss = criterion(output, target)
val_loss.append(loss.item())
else:
output = model(data)
if criterion:
loss = criterion(output, target)
val_loss.append(loss.item())
correct_samples += (
target.detach().cpu().numpy() == np.argmax(output.detach().cpu().numpy(), axis=1)
).sum()
y_pred[batch_i * valid_bs : (batch_i + 1) * valid_bs] = np.argmax(output.detach().cpu().numpy(), axis=1)
y_true[batch_i * valid_bs : (batch_i + 1) * valid_bs] = target.detach().cpu().numpy()
del data, target, output
torch.cuda.empty_cache()
valid_loss = np.mean(val_loss)
f1_scores = f1_score(y_true, y_pred, average=None)
f1_score_macro = f1_score(y_true, y_pred, average='macro')
accuracy_score = correct_samples / (len(valid_loader)*valid_bs)
metrics = {"loss": valid_loss, "accuracy": accuracy_score, "f1_scores": f1_scores, 'f1_score_macro': f1_score_macro}
return metrics
def copy_parameters_from_model(model):
copy_of_model_parameters = [p.clone().detach() for p in model.parameters() if p.requires_grad]
return copy_of_model_parameters
def copy_parameters_to_model(copy_of_model_parameters, model):
for s_param, param in zip(copy_of_model_parameters, model.parameters()):
if param.requires_grad:
param.data.copy_(s_param.data)
================================================
FILE: train.py
================================================
import argparse
import logging
import os
import time
import torch
import yaml
import shutil
from torch.utils.tensorboard import SummaryWriter
from torch_ema import ExponentialMovingAverage
from tools import utils
scaler = torch.cuda.amp.GradScaler()
def parse_config():
parser = argparse.ArgumentParser()
parser.add_argument(
"--config_name",
type=str,
default="configs/train/train_supcon_resnet18_cifar10_stage2.yml",
)
parser_args = parser.parse_args()
with open(vars(parser_args)["config_name"], "r") as config_file:
hyperparams = yaml.full_load(config_file)
return hyperparams
if __name__ == "__main__":
# parse hyperparameters
hyperparams = parse_config()
print(hyperparams)
backbone = hyperparams["model"]["backbone"]
ckpt_pretrained = hyperparams['model']['ckpt_pretrained']
num_classes = hyperparams['model']['num_classes']
amp = hyperparams['train']['amp']
ema = hyperparams['train']['ema']
ema_decay_per_epoch = hyperparams['train']['ema_decay_per_epoch']
n_epochs = hyperparams["train"]["n_epochs"]
logging_name = hyperparams['train']['logging_name']
target_metric = hyperparams['train']['target_metric']
stage = hyperparams['train']['stage']
data_dir = hyperparams["dataset"]
optimizer_params = hyperparams["optimizer"]
scheduler_params = hyperparams["scheduler"]
criterion_params = hyperparams["criterion"]
batch_sizes = {
"train_batch_size": hyperparams["dataloaders"]["train_batch_size"],
'valid_batch_size': hyperparams['dataloaders']['valid_batch_size']
}
num_workers = hyperparams["dataloaders"]["num_workers"]
if not amp: scaler = None
utils.seed_everything()
# create model, loaders, optimizer, etc
transforms = utils.build_transforms(second_stage=(stage == 'second'))
loaders = utils.build_loaders(data_dir, transforms, batch_sizes, num_workers, second_stage=(stage == 'second'))
model = utils.build_model(backbone, second_stage=(stage == 'second'), num_classes=num_classes, ckpt_pretrained=ckpt_pretrained).cuda()
if ema:
iters = len(loaders['train_features_loader'])
ema_decay = ema_decay_per_epoch**(1/iters)
ema = ExponentialMovingAverage(model.parameters(), decay=ema_decay)
optim = utils.build_optim(model, optimizer_params, scheduler_params, criterion_params)
criterion, optimizer, scheduler = (
optim["criterion"],
optim["optimizer"],
optim["scheduler"],
)
# handle logging (regular logs, tensorboard, and weights)
if logging_name is None:
logging_name = "stage_{}_model_{}_dataset_{}".format(stage, backbone, data_dir.split("/")[-1])
shutil.rmtree("weights/{}".format(logging_name), ignore_errors=True)
shutil.rmtree(
"runs/{}".format(logging_name),
ignore_errors=True,
)
shutil.rmtree(
"logs/{}".format(logging_name),
ignore_errors=True,
)
os.makedirs(
"logs/{}".format(logging_name),
exist_ok=True,
)
writer = SummaryWriter("runs/{}".format(logging_name))
logging_dir = "logs/{}".format(logging_name)
logging_path = os.path.join(logging_dir, "train.log")
logging.basicConfig(filename=logging_path, level=logging.INFO, filemode="w+")
# epoch loop
metric_best = 0
for epoch in range(n_epochs):
utils.add_to_logs(logging, "{}, epoch {}".format(time.ctime(), epoch))
start_training_time = time.time()
if stage == 'first':
train_metrics = utils.train_epoch_constructive(loaders['train_supcon_loader'], model, criterion, optimizer, scaler, ema)
else:
train_metrics = utils.train_epoch_ce(loaders['train_features_loader'], model, criterion, optimizer, scaler, ema)
end_training_time = time.time()
if ema:
copy_of_model_parameters = utils.copy_parameters_from_model(model)
ema.copy_to(model.parameters())
start_validation_time = time.time()
if stage == 'first':
valid_metrics_projection_head = utils.validation_constructive(loaders['valid_loader'], loaders['train_features_loader'], model, scaler)
model.use_projection_head(False)
valid_metrics_encoder = utils.validation_constructive(loaders['valid_loader'], loaders['train_features_loader'], model, scaler)
model.use_projection_head(True)
print(
'epoch {}, train time {:.2f} valid time {:.2f} train loss {:.2f}\nvalid acc dict projection head {}\nvalid acc dict encoder {}'.format(
epoch,
end_training_time - start_training_time,
time.time() - start_validation_time,
train_metrics['loss'], valid_metrics_projection_head, valid_metrics_encoder))
valid_metrics = valid_metrics_projection_head
else:
valid_metrics = utils.validation_ce(model, criterion, loaders['valid_loader'], scaler)
print(
'epoch {}, train time {:.2f} valid time {:.2f} train loss {:.2f}\n valid acc dict {}\n'.format(
epoch,
end_training_time - start_training_time,
time.time() - start_validation_time,
train_metrics['loss'], valid_metrics))
# write train and valid metrics to the logs
utils.add_to_tensorboard_logs(writer, train_metrics['loss'], "Loss/train", epoch)
for valid_metric in valid_metrics:
try:
utils.add_to_tensorboard_logs(writer, valid_metrics[valid_metric],
'{}/validation'.format(valid_metric), epoch)
except AssertionError:
# in case valid metric is a list
pass
if stage == 'first':
utils.add_to_logs(
logging,
"Epoch {}, train loss: {:.4f}\nvalid metrics projection head: {}\nvalid metric encoder: {}".format(
epoch,
train_metrics['loss'],
valid_metrics_projection_head,
valid_metrics_encoder
),
)
else:
utils.add_to_logs(
logging,
"Epoch {}, train loss: {:.4f} valid metrics: {}".format(
epoch,
train_metrics['loss'],
valid_metrics
),
)
# check if the best value of metric changed. If so -> save the model
if valid_metrics[target_metric] > metric_best:
utils.add_to_logs(
logging,
"{} increased ({:.6f} --> {:.6f}). Saving model ...".format(target_metric,
metric_best, valid_metrics[target_metric]
),
)
os.makedirs(
"weights/{}".format(logging_name),
exist_ok=True,
)
torch.save(
{
"epoch": epoch,
"model_state_dict": model.state_dict(),
"optimizer_state_dict": optimizer.state_dict(),
},
"weights/{}/epoch{}".format(
logging_name, epoch
),
)
metric_best = valid_metrics[target_metric]
# if ema is used, go back to regular weights without ema
if ema:
utils.copy_parameters_to_model(copy_of_model_parameters, model)
scheduler.step()
writer.close()
