Repository: ivanpanshin/SupCon-Framework
Branch: main
Commit: 4a934d4a19b7
Files: 25
Total size: 97.5 KB
Directory structure:
gitextract_cf8_gd7q/
├── .gitattributes
├── LICENSE.md
├── README.md
├── configs/
│ └── train/
│ ├── lr_finder_supcon_resnet18_cifar100_stage2.yml
│ ├── lr_finder_supcon_resnet18_cifar10_stage2.yml
│ ├── swa_supcon_resnet18_cifar100_stage1.yml
│ ├── swa_supcon_resnet18_cifar100_stage2.yml
│ ├── swa_supcon_resnet18_cifar10_stage1.yml
│ ├── swa_supcon_resnet18_cifar10_stage2.yml
│ ├── train_supcon_resnet18_cifar100_stage1.yml
│ ├── train_supcon_resnet18_cifar100_stage2.yml
│ ├── train_supcon_resnet18_cifar10_stage1.yml
│ └── train_supcon_resnet18_cifar10_stage2.yml
├── learning_rate_finder.py
├── requirements.txt
├── swa.py
├── t-SNE.ipynb
├── tools/
│ ├── backbones.py
│ ├── datasets.py
│ ├── losses.py
│ ├── models.py
│ ├── optimizers.py
│ ├── schedulers.py
│ └── utils.py
└── train.py
================================================
FILE CONTENTS
================================================
================================================
FILE: .gitattributes
================================================
*.ipynb linguist-vendored
================================================
FILE: LICENSE.md
================================================
MIT License
Copyright (c) 2022 Ivan Panshin
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
================================================
FILE: README.md
================================================
# SupCon-Framework
<p align="center"><img src="https://github.com/ivanpanshin/SupCon-Framework/blob/main/images/logo.png?raw=true" width="800"></p>
The repo is an implementation of [Supervised Contrastive Learning](https://arxiv.org/abs/2004.11362). It's based on another [implementation](https://github.com/HobbitLong/SupContrast), but with several differencies:
- Fixed bugs (incorrect ResNet implementations, which leads to a very small max batch size),
- Offers a lot of additional functionality (first of all, rich validation).
To be more precise, in this implementations you will find:
- Augmentations with [albumentations](https://github.com/albumentations-team/albumentations)
- Hyperparameters are moved to .yml configs
- [t-SNE](https://github.com/DmitryUlyanov/Multicore-TSNE) visualizations
- 2-step validation (for features before and after the projection head) using metrics like AMI, NMI, mAP, precision_at_1, etc with [PyTorch Metric Learning](https://github.com/KevinMusgrave/pytorch-metric-learning).
- [Exponential Moving Average](https://github.com/fadel/pytorch_ema) for a more stable training, and Stochastic Moving Average for a better generalization and just overall performance.
- Automatic Mixed Precision (torch version) training in order to be able to train with a bigger batch size (roughly by a factor of 2).
- LabelSmoothing loss, and [LRFinder](https://github.com/davidtvs/pytorch-lr-finder) for the second stage of the training (FC).
- TensorBoard logs, checkpoints
- Support of [timm models](https://github.com/rwightman/pytorch-image-models), and [pytorch-optimizer](https://github.com/jettify/pytorch-optimizer)
## Install
1. Clone the repo:
```
git clone https://github.com/ivanpanshin/SupCon-Framework && cd SupCon-Framework/
```
2. Create a clean virtual environment
```
python3 -m venv venv
source venv/bin/activate
```
3. Install dependencies
````
python -m pip install --upgrade pip
pip install -r requirements.txt
````
## Training
In order to execute Cifar10 training run:
```
python train.py --config_name configs/train/train_supcon_resnet18_cifar10_stage1.yml
python swa.py --config_name configs/train/swa_supcon_resnet18_cifar10_stage1.yml
python train.py --config_name configs/train/train_supcon_resnet18_cifar10_stage2.yml
python swa.py --config_name configs/train/swa_supcon_resnet18_cifar10_stage2.yml
```
In order to run LRFinder on the second stage of the training, run:
```
python learning_rate_finder.py --config_name configs/train/lr_finder_supcon_resnet18_cifar10_stage2.yml
```
The process of training Cifar100 is exactly the same, just change config names from cifar10 to cifar100.
After that you can check the results of the training either in `logs` or `runs` directory. For example, in order to check tensorboard logs for the first stage of Cifar10 training, run:
```
tensorboard --logdir runs/supcon_first_stage_cifar10
```
## Visualizations
This repo is supplied with t-SNE visualizations so that you can check embeddings you get after the training. Check `t-SNE.ipynb` for details.
<p align="center"><img src="https://github.com/ivanpanshin/SupCon-Framework/blob/main/images/t-SNE-cifar10.png?raw=true" width="600"></p>
Those are t-SNE visualizations for Cifar10 for validation and train with SupCon (top), and validation and train with CE (bottom).
<p align="center"><img src="https://github.com/ivanpanshin/SupCon-Framework/blob/main/images/t-SNE-cifar100.png?raw=true" width="600"></p>
Those are t-SNE visualizations for Cifar100 for validation and train with SupCon (top), and validation and train with CE (bottom).
## Results
| Model | Stage | Dataset | Accuracy |
| ------------- | ------------- | ------------- | ------------- |
| ResNet18 | Frist | CIFAR10 | 95.9 |
| ResNet18 | Second | CIFAR10 | 94.9 |
| ResNet18 | Frist | CIFAR100 | 79.0 |
| ResNet18 | Second | CIFAR100 | 77.9 |
Note that even though the accuracy on the second stage is lower, it's not always the case. In my experience, the difference between stages is usually around 1 percent, including the difference that favors the second stage.
Training time for the whole pipeline (without any early stopping) on CIFAR10 or CIFAR100 is around 4 hours (single 2080Ti with AMP). However, with reasonable early stopping that value goes down to around 2.5-3 hours.
## Custom datasets
It's fairly easy to adapt this pipeline to custom datasets. First, you need to check `tools/datasets.py` for that. Second, add a new class for your dataset. The only guideline here is to follow the same augmentation logic, that is
```
if self.second_stage:
image = self.transform(image=image)['image']
else:
image = self.transform(image)
```
Third, add your dataset to `DATASETS` dict still inside `tools/datasets.py`, and you're good to go.
## FAQ
- Q: What hyperparameters should I try to change?
A: First of all, learning rate. Second of all, try to change the augmentation policy. SupCon is built around "cropping + color jittering" scheme, so you can try changing the cropping size or the intensity of jittering. Check `tools.utils.build_transforms` for that.
- Q: What backbone and batch size should I use?
A: This is quite simple. Take the biggest backbone that you can, and after that take the highest batch size your GPU can offer. The reason for that: SupCon is more prone (than regular classification training with CE/LabelSmoothing/etc) to improving with stronger backbones. Moverover, it has a property of explicit hard positive and negative mining. It means that the higher the batch size - the more difficult and helpful samples you supply to the model.
- Q: Do I need the second stage of the training?
A: Not necessarily. You can do classification based only on embeddings. In order to do that compute embeddings for the train set, and at inference time do the following: take a sample, compute its embedding, take the closest one from the training, take its class. To make this fast and efficient, you something like [faiss](https://github.com/facebookresearch/faiss) for similarity search. Note that this is actually how validation is done in this repo. Moveover, during training you will see a metric `precision_at_1`. This is actually just an accuracy based solely on embeddings.
- Q: Should I use AMP?
A: If your GPU has tensor cores (like 2080Ti) - yes. If it doesn't (like 1080Ti) - check the speed with AMP and without. If the speed dropped slightly (or even increased by a bit) - use it, since SupCon works better with bigger batch sizes.
- Q: How should I use EMA?
A: You only need to choose the `ema_decay_per_epoch` parameter in the config. The heuristic is fairly simple. If your dataset is big, then something as small as 0.3 will do just fine. And as your dataset gets smaller, you can increase `ema_decay_per_epoch`. Thanks to bonlime for this idea. I advice you to check his great [pytorch tools](https://github.com/bonlime/pytorch-tools) repo, it's a hidden gem.
- Q: Is it better than training with Cross Entropy/Label Smoothing/etc?
A: Unfortunately, in my experience, it's much easier to get better results with something like CE. It's more stable, faster to train, and simply produces better or the same results. For instance, in case on CIFAR10/100 it's trivial to train ResNet18 up tp 96/81 percent respectively. Of cource, I've seen cases where SupCon performs better, but it takes quite a bit of work to make it outperform CE.
- Q: How long should I train with SupCon?
A: The answer is tricky. On one hand, authors of the original paper claim that the longer you train with SupCon, the better it gets. However, I did not observe such a behavior in my tests. So the only recommendation I can give is the following: start with 100 epochs for easy datasets (like CIFAR10/100), and 1000 for more industrial ones. Then - monitor the training process. If the validaton metric (such as `precision_at_1`) doesn't impove for several dozens of epochs - you can stop the training. You might incorporate early stopping for this reason into the pipeline.
================================================
FILE: configs/train/lr_finder_supcon_resnet18_cifar100_stage2.yml
================================================
model:
backbone: resnet18
ckpt_pretrained:
num_classes: 100
dataset: data/cifar100
dataloaders:
train_batch_size: 200 # the higher - the better
valid_batch_size: 200
num_workers: 16 # set this to num of threads in your CPU
optimizer:
name: SGD
params:
lr: 0.001
criterion:
name: 'CrossEntropy'
================================================
FILE: configs/train/lr_finder_supcon_resnet18_cifar10_stage2.yml
================================================
model:
backbone: resnet18
ckpt_pretrained:
num_classes: 10
dataset: data/cifar10
dataloaders:
train_batch_size: 200 # the higher - the better
valid_batch_size: 200
num_workers: 16 # set this to num of threads in your CPU
optimizer:
name: SGD
params:
lr: 0.001
criterion:
name: 'CrossEntropy'
================================================
FILE: configs/train/swa_supcon_resnet18_cifar100_stage1.yml
================================================
model:
backbone: resnet18
num_classes:
top_k_checkoints: 3
train:
amp: True # set this to True, if your GPU supports FP16. 2080Ti - okay, 1080Ti - not okay
weights_dir: weights/supcon_first_stage_cifar100
stage: first
dataset: data/cifar100
dataloaders:
train_batch_size: 200 # the higher - the better
valid_batch_size: 200
num_workers: 12 # set this to num of threads in your CPU
================================================
FILE: configs/train/swa_supcon_resnet18_cifar100_stage2.yml
================================================
model:
backbone: resnet18
num_classes: 100
top_k_checkoints: 3
train:
amp: True # set this to True, if your GPU supports FP16. 2080Ti - okay, 1080Ti - not okay
weights_dir: weights/supcon_second_stage_cifar100
stage: second
dataset: data/cifar100
dataloaders:
train_batch_size: 200 # the higher - the better
valid_batch_size: 200
num_workers: 12 # set this to num of threads in your CPU
================================================
FILE: configs/train/swa_supcon_resnet18_cifar10_stage1.yml
================================================
model:
backbone: resnet18
num_classes:
top_k_checkoints: 3
train:
amp: True # set this to True, if your GPU supports FP16. 2080Ti - okay, 1080Ti - not okay
weights_dir: weights/supcon_first_stage_cifar10
stage: first
dataset: data/cifar10
dataloaders:
train_batch_size: 200 # the higher - the better
valid_batch_size: 200
num_workers: 12 # set this to num of threads in your CPU
================================================
FILE: configs/train/swa_supcon_resnet18_cifar10_stage2.yml
================================================
model:
backbone: resnet18
num_classes: 10
top_k_checkoints: 3
train:
amp: True # set this to True, if your GPU supports FP16. 2080Ti - okay, 1080Ti - not okay
weights_dir: weights/supcon_second_stage_cifar10
stage: second
dataset: data/cifar10
dataloaders:
train_batch_size: 200 # the higher - the better
valid_batch_size: 200
num_workers: 12 # set this to num of threads in your CPU
================================================
FILE: configs/train/train_supcon_resnet18_cifar100_stage1.yml
================================================
model:
backbone: resnet18
ckpt_pretrained:
num_classes: # in the first stage of training we don't need num_classes, since we don't have a FC head
train:
n_epochs: 100
amp: True # set this to True, if your GPU supports FP16. 2080Ti - okay, 1080Ti - not okay
ema: True # optional, but I recommend it, since the training might get unstable otherwise
ema_decay_per_epoch: 0.3 # 0.3 for middle/big datasets. Increase, if you have low amount of samples
logging_name: supcon_first_stage_cifar100
target_metric: precision_at_1
stage: first # first = Supcon, second = FC finetuning for classification
dataset: data/cifar100
dataloaders:
train_batch_size: 200 # the higher - the better
valid_batch_size: 200
num_workers: 12 # set this to num of threads in your CPU
optimizer:
name: SGD
params:
lr: 0.1
scheduler:
name: CosineAnnealingLR
params:
T_max: 100
eta_min: 0.01
criterion:
name: 'SupCon'
params:
temperature: 0.1
================================================
FILE: configs/train/train_supcon_resnet18_cifar100_stage2.yml
================================================
model:
backbone: resnet18
ckpt_pretrained: weights/supcon_first_stage_cifar100/swa
num_classes: 100
train:
n_epochs: 20
amp: True # set this to True, if your GPU supports FP16. 2080Ti - okay, 1080Ti - not okay
ema: True # optional, but I recommend it, since the training might get unstable otherwise
ema_decay_per_epoch: 0.3 # for middle/big datasets. Increase, if you have low amount of samples
logging_name: supcon_second_stage_cifar100
target_metric: accuracy
stage: second # first = Supcon, second = FC finetuning for classification
dataset: data/cifar100
dataloaders:
train_batch_size: 20 # the higher - the better
valid_batch_size: 20
num_workers: 12 # set this to num of threads in your CPU
optimizer:
name: SGD
params:
lr: 0.01
scheduler:
name: CosineAnnealingLR
params:
T_max: 20
eta_min: 0.001
criterion:
name: 'LabelSmoothing'
params:
classes: 100
smoothing: 0.01
================================================
FILE: configs/train/train_supcon_resnet18_cifar10_stage1.yml
================================================
model:
backbone: resnet18
ckpt_pretrained:
num_classes: # in the first stage of training we don't need num_classes, since we don't have a FC head
train:
n_epochs: 100
amp: True # set this to True, if your GPU supports FP16. 2080Ti - okay, 1080Ti - not okay
ema: True # optional, but I recommend it, since the training might get unstable otherwise
ema_decay_per_epoch: 0.3 # 0.3 for middle/big datasets. Increase, if you have low amount of samples
logging_name: supcon_first_stage_cifar10
target_metric: precision_at_1
stage: first # first = Supcon, second = FC finetuning for classification
dataset: data/cifar10
dataloaders:
train_batch_size: 200 # the higher - the better
valid_batch_size: 200
num_workers: 12 # set this to num of threads in your CPU
optimizer:
name: SGD
params:
lr: 0.1
scheduler:
name: CosineAnnealingLR
params:
T_max: 100
eta_min: 0.01
criterion:
name: 'SupCon'
params:
temperature: 0.1
================================================
FILE: configs/train/train_supcon_resnet18_cifar10_stage2.yml
================================================
model:
backbone: resnet18
ckpt_pretrained: weights/supcon_first_stage_cifar10/swa
num_classes: 10
train:
n_epochs: 20
amp: True # set this to True, if your GPU supports FP16. 2080Ti - okay, 1080Ti - not okay
ema: True # optional, but I recommend it, since the training might get unstable otherwise
ema_decay_per_epoch: 0.3 # for middle/big datasets. Increase, if you have low amount of samples
logging_name: supcon_second_stage_cifar10
target_metric: accuracy
stage: second # first = Supcon, second = FC finetuning for classification
dataset: data/cifar10
dataloaders:
train_batch_size: 20 # the higher - the better
valid_batch_size: 20
num_workers: 12 # set this to num of threads in your CPU
optimizer:
name: SGD
params:
lr: 0.01
scheduler:
name: CosineAnnealingLR
params:
T_max: 20
eta_min: 0.001
criterion:
name: 'LabelSmoothing'
params:
classes: 10
smoothing: 0.01
================================================
FILE: learning_rate_finder.py
================================================
import argparse
import os
import matplotlib.pyplot as plt
import yaml
from torch_lr_finder import LRFinder
from tools import utils
def parse_config():
parser = argparse.ArgumentParser()
parser.add_argument(
"--config_name",
type=str,
default="configs/train/lr_finder_supcon_resnet18_cifar10_stage2.yml",
)
parser_args = parser.parse_args()
with open(vars(parser_args)["config_name"], "r") as config_file:
hyperparams = yaml.full_load(config_file)
return hyperparams
if __name__ == "__main__":
os.makedirs("lr_finder_plots", exist_ok=True)
hyperparams = parse_config()
backbone = hyperparams["model"]["backbone"]
ckpt_pretrained = hyperparams["model"]["ckpt_pretrained"]
num_classes = hyperparams['model']['num_classes']
optimizer_params = hyperparams["optimizer"]
scheduler_params = None
criterion_params = hyperparams["criterion"]
data_dir = hyperparams["dataset"]
batch_sizes = {
"train_batch_size": hyperparams["dataloaders"]["train_batch_size"],
"valid_batch_size": hyperparams["dataloaders"]["valid_batch_size"]
}
num_workers = hyperparams["dataloaders"]["num_workers"]
transforms = utils.build_transforms(second_stage=True)
loaders = utils.build_loaders(data_dir, transforms, batch_sizes, num_workers, second_stage=True)
model = utils.build_model(backbone, second_stage=True, num_classes=num_classes,
ckpt_pretrained=ckpt_pretrained).cuda()
optim = utils.build_optim(model, optimizer_params, scheduler_params, criterion_params)
criterion, optimizer, scheduler = (
optim["criterion"],
optim["optimizer"],
optim["scheduler"],
)
lr_finder = LRFinder(model, optimizer, criterion, device="cuda")
lr_finder.range_test(loaders["train_features_loader"], end_lr=1, num_iter=300)
fig, ax = plt.subplots()
lr_finder.plot(ax=ax)
fig.savefig(
"lr_finder_plots/supcon_{}_{}_bs_{}_stage_{}_lr_finder.png".format(
optimizer_params["name"],
data_dir.split("/")[-1],
batch_sizes["train_batch_size"],
'second'
)
)
================================================
FILE: requirements.txt
================================================
absl-py==0.11.0
albumentations==0.5.2
argon2-cffi==20.1.0
async-generator==1.10
attrs==20.3.0
backcall==0.2.0
bleach==3.2.1
cachetools==4.2.0
certifi==2020.12.5
cffi==1.14.4
chardet==4.0.0
cycler==0.10.0
dataclasses==0.8
decorator==4.4.2
defusedxml==0.6.0
entrypoints==0.3
faiss-gpu==1.6.5
google-auth==1.24.0
google-auth-oauthlib==0.4.2
grpcio==1.34.0
idna==2.10
imageio==2.9.0
imgaug==0.4.0
importlib-metadata==3.3.0
ipykernel==5.4.3
ipython==7.16.1
ipython-genutils==0.2.0
ipywidgets==7.6.3
jedi==0.18.0
Jinja2==2.11.2
joblib==1.0.0
jsonschema==3.2.0
jupyter==1.0.0
jupyter-client==6.1.11
jupyter-console==6.2.0
jupyter-core==4.7.0
jupyterlab-pygments==0.1.2
jupyterlab-widgets==1.0.0
kiwisolver==1.3.1
Markdown==3.3.3
MarkupSafe==1.1.1
matplotlib==3.3.3
mistune==0.8.4
MulticoreTSNE==0.1
nbclient==0.5.1
nbconvert==6.0.7
nbformat==5.0.8
nest-asyncio==1.4.3
networkx==2.5
notebook==6.1.6
numpy==1.19.4
oauthlib==3.1.0
opencv-python==4.4.0.46
opencv-python-headless==4.4.0.46
packaging==20.8
pandas==1.1.5
pandocfilters==1.4.3
parso==0.8.1
pexpect==4.8.0
pickleshare==0.7.5
Pillow==8.0.1
prometheus-client==0.9.0
prompt-toolkit==3.0.10
protobuf==3.14.0
ptyprocess==0.7.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycparser==2.20
Pygments==2.7.4
pyparsing==2.4.7
pyrsistent==0.17.3
python-dateutil==2.8.1
pytorch-metric-learning==0.9.95
pytorch-ranger==0.1.1
pytz==2020.4
PyWavelets==1.1.1
PyYAML==5.3.1
pyzmq==20.0.0
qtconsole==5.0.1
QtPy==1.9.0
requests==2.25.1
requests-oauthlib==1.3.0
rsa==4.6
scikit-image==0.17.2
scikit-learn==0.23.2
scipy==1.5.4
Send2Trash==1.5.0
Shapely==1.7.1
six==1.15.0
tensorboard==2.4.0
tensorboard-plugin-wit==1.7.0
terminado==0.9.2
testpath==0.4.4
threadpoolctl==2.1.0
tifffile==2020.9.3
timm==0.3.4
torch==1.7.1
torch-ema @ git+https://github.com/fadel/pytorch_ema@27afe25b9fb9f0d05a87ae94e4e4ad9e92d70a85
torch-lr-finder==0.2.1
torch-optimizer==0.0.1a17
torchvision==0.8.2
tornado==6.1
tqdm==4.54.1
traitlets==4.3.3
typing-extensions==3.7.4.3
urllib3==1.26.2
wcwidth==0.2.5
webencodings==0.5.1
Werkzeug==1.0.1
widgetsnbextension==3.5.1
zipp==3.4.0
================================================
FILE: swa.py
================================================
import os
from collections import OrderedDict
import torch
import argparse
import yaml
from tools import utils
scaler = torch.cuda.amp.GradScaler()
def swa(paths):
state_dicts = []
for path in paths:
state_dicts.append(torch.load(path)["model_state_dict"])
average_dict = OrderedDict()
for k in state_dicts[0].keys():
average_dict[k] = sum([state_dict[k] for state_dict in state_dicts]) / len(state_dicts)
return average_dict
def parse_config():
parser = argparse.ArgumentParser()
parser.add_argument(
"--config_name",
type=str,
default="configs/train/swa_supcon_resnet18_cifar10_stage2.yml"
)
parser_args = parser.parse_args()
with open(vars(parser_args)["config_name"], "r") as config_file:
hyperparams = yaml.full_load(config_file)
return hyperparams
if __name__ == "__main__":
hyperparams = parse_config()
backbone = hyperparams["model"]["backbone"]
num_classes = hyperparams['model']['num_classes']
top_k_checkoints = hyperparams['model']['top_k_checkoints']
amp = hyperparams['train']['amp']
weights_dir = hyperparams['train']['weights_dir']
stage = hyperparams['train']['stage']
data_dir = hyperparams["dataset"]
batch_sizes = {
"train_batch_size": hyperparams["dataloaders"]["train_batch_size"],
'valid_batch_size': hyperparams['dataloaders']['valid_batch_size']
}
num_workers = hyperparams["dataloaders"]["num_workers"]
if not amp: scaler = None
utils.seed_everything()
if os.path.exists(os.path.join(weights_dir, "swa")):
os.remove(os.path.join(weights_dir, "swa"))
transforms = utils.build_transforms(second_stage=(stage == 'second'))
loaders = utils.build_loaders(data_dir, transforms, batch_sizes, num_workers, second_stage=(stage == 'second'))
model = utils.build_model(backbone, second_stage=(stage == 'second'), num_classes=num_classes, ckpt_pretrained=None).cuda()
list_of_epochs = sorted([int(x.split('epoch')[1]) for x in os.listdir(weights_dir)])
best_epochs = list_of_epochs[-top_k_checkoints::]
model_prefix = "epoch"
checkpoints_paths = ["{}/{}{}".format(weights_dir, model_prefix, epoch) for epoch in best_epochs]
average_dict = swa(checkpoints_paths)
torch.save({"model_state_dict": average_dict}, os.path.join(weights_dir, "swa"))
model.load_state_dict(torch.load(os.path.join(weights_dir, "swa"))['model_state_dict'])
if stage == 'first':
valid_metrics = utils.validation_constructive(loaders['valid_loader'], loaders['train_features_loader'], model,
scaler)
else:
valid_metrics = utils.validation_ce(model, None, loaders['valid_loader'], scaler)
print('swa stage {} validation metrics: {}'.format(stage, valid_metrics))
================================================
FILE: t-SNE.ipynb
================================================
{
"cells": [
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"from MulticoreTSNE import MulticoreTSNE as TSNE\n",
"from matplotlib import pyplot as plt\n",
"from tools import utils\n",
"import torch\n",
"import torch.functional as F\n",
"\n",
"scaler = torch.cuda.amp.GradScaler()"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Files already downloaded and verified\n",
"Files already downloaded and verified\n",
"Files already downloaded and verified\n"
]
}
],
"source": [
"ckpt_pretrained = 'weights/supcon_first_stage_cifar10/swa'\n",
"data_dir = 'data/cifar10'\n",
"num_classes = 10\n",
"batch_sizes = {\n",
" \"train_batch_size\": 20,\n",
" 'valid_batch_size': 20\n",
"}\n",
"num_workers = 16\n",
"backbone = 'resnet18'\n",
"stage = 'first'\n",
"\n",
"transforms = utils.build_transforms(second_stage=(stage == 'second'))\n",
"loaders = utils.build_loaders(data_dir, transforms, batch_sizes, num_workers, second_stage=(stage == 'second'))\n",
" "
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"pycharm": {
"name": "#%%\n"
}
},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAV8AAAD8CAYAAADQSqd1AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAACBe0lEQVR4nOydd3gU1deA3zuzLb0TAgkQeu9SBKRjo9gFu6JYULGLXVF/9u5nAeyiIlZEVEAERHrvvQcI6T3b5n5/zKb3ZFOAeZ9nn+zO3HJ2s3vmzrmnCCklBgYGBgZ1i1LfAhgYGBicjRjK18DAwKAeMJSvgYGBQT1gKF8DAwODesBQvgYGBgb1gKF8DQwMDOoBQ/kaGBgYeAEhxBQhxDYhxHYhxH0VtTeUr4GBgUENEUJ0Bm4D+gDdgNFCiNbl9TGUr4GBgUHN6QCsllJmSyldwFLgsvI6mOpErEoSHh4uW7RoUd9iGBgYnAasX78+UUoZUZMxWgshsyvZ9gRsB3ILHZoupZzueb4NeFEIEQbkABcB68obz2vKVwiheiaLk1KOFkLEAt8BYcB64HoppaO8MVq0aMG6deXKa2BgYACAEOJwTcfIBm6vZNtnIVdK2bu0c1LKnUKIV4AFQBawCXCXN543zQ5TgJ2FXr8CvCWlbA2kABO9OJeBgYFBg0JK+YmUspeU8jx0nbenvPZeUb5CiGjgYmCm57UAhgE/eJp8AVzijbkMDAwMGiJCiEaev83Q7b3flNfeW2aHt4FHgADP6zAg1WN4BjgGNC2toxBiEjAJoFmzZl4Sx8DAwKDO+dFj83UCk6WUqeU1rrHyFUKMBk5JKdcLIYZUtb/HYD0doHfv3kZ+SwMDg9MSKeWgqrT3xsp3ADBWCHERYAMCgXeAYCGEybP6jQbivDCXgYGBwRlBjW2+UsrHpJTRUsoWwHhgsZTyWuAf4ApPsxuBX2s6l4GBgcGZQm0GWTwKPCCE2IduA/6kFucyMPAq8WxhEY+wglewk17f4hicgXg1yEJKuQRY4nl+AD3UzsDgtMJOBkt5BtBIBnJIZjiv1LdYBmcYDSrCzcCgPtBwsYHpnGQj/kTRguGAln8+mb31J5zBGYuhfA3Oatw4+Y2JOEgDIJtTnGJbkTYSNxuYjhs7nZhANgkksJ1IuhFKm/oQ2+AMwFC+Bmcdx1jLSl5G4kbBilYkXB8Kr3rz2MfvABxkEfpWicZWZtGDW4mmHz6E1brcBmcWRlYzg7MKDTcreAGJC5ClKF6AitzNtfy/G5nOb9zCX9yHg0zvCmtwRmMoX4MznjUHYM46SM+BbBJrZY40DrKd2bUytsGZiWF2MDij+XQ5TP01nejGh1ipLWJ4lwzwqZ259jIXB1n04R4EonYmMThjMJSvwRnHHuaxj98IIpY/Do/kjRdexmLRs5lmAtItUNTaiWQ/zGLacDGhtKqV8Q3OHAyzg8EZxSEWs4kZZHKSOFYy+tI3sFodCEHBQ6nNFCKSxTxKLqm1OIfBmYChfA3OKI6wvMhrH58sZF2la/LM45ZOfnd8WUeTGpyuGMrX4IxAItnDXOwef918SjG9itoyxxYad36GXy1NYnCmYNh8Dc4IDsg/2MpXuEXJSlW1pmxLIW+VHWM9hEQaG28GZWKsfA1Oe6Q7i2PZrxYo3jqzM5Qkz67c0W9brbm1GTRMhBD3CyG2CyG2CSG+FULYymtvKF+D054dGfNI2+eHw2HGYTcjtPpRvkV0vgBrfmEXgzMdIURT4F6gt5SyM6Cip9gtE8PsYHDak+FozPiP5hAYnEazpgfxVzK5467/q7P5pSxp2rAQgIlyFz4GDYAmJngmuHJtn634RsYE+AghnIAvcLyixgYGpzUtrINxai7iE6KIT4jihccfqsPZBVaCyZVpgERIFVWoxHAuLnKZx0QcZKJgYTQzsBFch7IZ1BVSyjghxOvAESAHWCClXFBeH8PsYFBjlnCId1nDOk7Uy/yNg+DaviZsZg2L2UmLmIN1NrdAYYh4lsvFLC4Q7xIuWhNCa9owhiU8nZ/vQcPBAu6vM7kMaoVwIcS6Qo9JeSeEECHAOCAWaAL4CSGuK28wbxTQtAHLAKtnvB+klM8IIWKB79CrWKwHrpdSltyKNjit+Y+jfMB67LhZzlGeZwjt6iHD12c3w2MXu9lgewbV7Kq4g5eQuDnJRkJoxXL+h5scABYwBTO+RdoaiXdOexKllL3LODcCOCilTAAQQvwEnAt8XdZg3lj52oFhUspuQHfgAiFEP+AV4C0pZWsgBZjohbkMGhi7ScKOG9BjDA6QUi9yCAHhkQcxBe2u87m38DlLeSpf8YKeoL2ZHIkmBVKCJgXNtavqXDaDOuMI0E8I4SuEEMBwYGd5HbxRQFNKKfMu6WbPQwLDgB88x78ALqnpXAYNjwHEYEXFhgkVQQ8a55/LznayevUxEhOz60SWDI55UkU2DLZldSbbrWfxyXT78196p3qWyKC2kFKuRtd3G4Ct6Lp1enl9vLLhJoRQ0U0LrYH/A/YDqZ6y8QDHgKbemMugbvieHfzOXhrhxzV0phuRKKUEDLQjjDcZyQ4S2UI8b7Oaq+lIi5Rgeg+fgc8YM84UN9/dejk9ukbVqsxZFfrVCirO1estJCbfN7Eq+oUn0JRBQNDj7OJG2nGJEXxxBiKlfAZ4prLtvaJ8pZRuoLsQIhj4GWhf2b4eo/UkgGbNmnlDHIMasockfmAndtykYuc5luWrLBsmHmcAXWnE3xxiC/F0phHrOc4qj2fNs/xL7MkgOq6MRJhBc0he3bCSb7kMiSQXFzZMXldAzRnCDr5DeswgJalb/19FpBdxQxNCspWv8SeKaPrVqSwGDQ9vVy9OFUL8A/QHgoUQJs/qNxqIK6PPdDzL8969e9dfaNJZTi4ufmYXJ8liNXH5dlwoqrJycTGNfzmXJizjGABLOVJivIPt0xCAEALVJnB10dhOAk+xBDeSMHz4gAuxefEr6E8jxvApB/mb3e7pOBULUiilO+LWAUKAwIKUjvy8DxIXm/iEJvRGMTw9z2pqbPMVQkR4VrwIIXyAkeiG5n+AKzzNbgR+relcBt4lnix2kIATN6+xkh/ZxRIOk1OB3dSFlq94y0ToihcACWF+vjzpUbwAiVoO49Lmc/nJhfwkd2H3kq3WRjAduJyhyWuwanZUzYWPu7RSQXWDxIFNhBc5ZieNRHbVk0QGDQVvXHqjgC88dl8F+F5KOU8IsQP4TgjxArAR+MQLcxl4iTUc5zVWIgA3Gi5v35JLCrJ8CTgh9D1ZoWm0OHEI/+wMtrbuijMwly9I4Xt28CXjsKB6YW5JoCuTi0/9TY5qw+q283PUxTUft5rklrBFS2wE1YssBg2HGitfKeUWoEcpxw8AfWo6vkHt8BO7cJRpG60mhZIbhKYlkRwUXjSlo5RIReFgk1hUt17AMm91nIOLPSTRmUY1l0MIhN9UlKzX8HXnsN2/bc3H9BIKZnpwG4HE1LcoBvWMEeF2lhJNAKba2HH3pPXK9vGjxAZXwc4TbtWEKJZ9rDH+3pMj8HlExA6SQ/+Pnf7tdFEawI7CCF6jJaPqWwyDBoChfM8yMrBzO7+zkIPeNzXk23gluWZr6ZtceQpXCKTQndfMKNxJT8KLRYTVGFNLjint9Xcp9IcmlXwRyss8adfMHM5tjkt6wQyCLsAAHiOYWC+NZ3C6Y2y3niVIJK+xkv8q2ijzBuV5FhQ+J3RnsxlcTEgtlRT+07GIluaC1wdyWvJ9wjUcs0dzS+MZ9ApYW0JcKWFZ6lA+PXkHZuHgoZgX6eq/tYaSCI7wLz6EEUqbGo5lcCZgrHzPElZzvG4Ubx6VdO3SQPcj1tzg9n502lFnI9ya/lxKWJRwJzuyu9AnYDWd/beUKeaviZehoWKXPrx5bKoX8rNrHGU5i3mC1fI9NsuvcFI3kX8GDRND+Z4lpFN/7lYVcVCm8tqOh9Ge9yFu5Qs13giUSOLZxBH+xZ5zLTuze5DkiGBb2m0EqiZc0kRnvy3YFHvxjkgJK9LOJdHVKP/gXU3erZE8UGDi+C+tF4N23U7vXeN5ItXwvjybMZTvWYAbjRV1ueqtKkKwum0PPh85nvcbu3gs+VOcNVDA2/mW5bzEOt7nysYvMICnaeWYyWNBAxka8Q6+ShYr0wfg0Mwlqk8A9AlYw4DAZSi4iDIfo41tV2FzNqcSIsi1W5AS3O6yf0LFV8vpGf58fOJunNKCS5p568TlJMsD1X6fBqc3hvI9C/iF3Wwkvr7FKJewjGT+OGckO2I7csTfxj+Za6s91iEW4yYXF7lkijgG+icxyh+yxHFirCd4reUUjtqb8+qRJ1iZfm4JJWlWXdzV9B0eiXmBlj77eevYo6Q6db/chKRGPPD0+3w66w7m/nkpb330cJlyuKSKlAVK+Lufr0MVBRcVFY14NlX7fRqc3hgbbmcYKVngcEGkx4f/e3bwHdvrV6hKkBDcCM2zvBRIVqSvJsK/OSoqXYioUh6IUNqSSyoaLlRs+dUjQmiJGT9+ThzPMXsz3JjYk9OeGOthYmxFo98VJF38NtHBdwdmxYkqdMOxzZoLKCxdMRwhNNq32VFifinBjcrME3fyX/pg7mv6CvGrGhGX3JM7oj5g+ok70FCYHPU+4WJc9T4wg9MeY+V7BvHqikzCH3TT5FE3Y39MYCMnmMW2/JDehkjeqlBTFH2TTkrsJgtbIsL5n3sJL7gW8t7JmZCdXOkx+zCF9lxOS85nOK+iors7mLAxgBdwSBuaR5lfFTGLaGuB4i28AWdSJDbVjoKWfywwIJ07bnyXsJAEWjbfx123vFNifiHAJNxcEDofp7Tw6ck7GDn0bw4MieStYw/SN2Atc1vN4f7Ai4jASDN5tmKsfM8gnvnejObS/VLn/RWGNvZHVHMFnRoAUqKvbPPDkQVuzwNUlob7cs/r7RFT9oBPcIXjmbDSmQlFji3kAP9yBMkCLotIYGtWB2xKDheH/VZE4RZyQy5ijiicm2dgv38Z2O/fcmVwSZUoyzHeaHUX38bfAKrAYQEwsyp9JARBpPHrq38igFsr2fZ5705t/PvPIKz+DnIzLYBAMbsRSsNd8eYhBGhuUULW4MxU0n0DAUmTpBPgyICjK6HthZUeW0Mylz2s4Si7SMKNQKUzfS2r+aDNRJxSLeFqVswNucrkKWwVNybVTRMljinRr+Pv7o+vAEVArBn6+7hJYDe+ROBHRNUnMjjtMZTvGYB95098r+zl1rujmb+hE7HD9qNa3IjT1agkodWxAwTkZqBqGtf9PRsAe0QTJGmVTkrzM7v4Vm7BKUR+oh83KikEE6b5sze3LT18N6II712kiq+YFQFmnBzMiWVbSzjlhm4+LhaIieSSCkA3bqadUejltEYI0Q6YXehQS+BpKeXbZfUxlO/pzr4FvOtezurWPXGpKm0u3F0vuWurQ/4tfvEVuoD1HXrRbe8mblz+PX5R3Tg6aBRrQqYh0YjmXPryAEoFGdA2uLfSTN1PK/axWAxDemKM97pb897+h3FoFmIsh3k85jUCzd7zBhFCEChbkSb3AXDU3oypx8Yy0wLrW0KiWJuveAG28Y2hfE9zpJS70WtY5lX2iUMvLFEmhvI93Ylbx74usTjNFv11zUOx6oyKrhGb23TngTbd6UUUHXmH44TxL4Nwo3IhM7mN28vt3yNpNY6IA5iExlg5l2QtBBTBrMzrsWtW3JqZg7ltWHTqFS5vOhFNuislV8UoNJOXMmxfa2xqFkfssUgUjmRn8u+nV9K1W2s4p6C1qZZCqw3qjeHAfinl4fIana43pgZ5tBvNqA3LsDpysdlzPKkazyzWc4IMQlnBubjdKjf+MYvBH79G9vo3yu3Xw6cLFpcTAB9hp0XGETZndsfPkg1ScF7Q3zzd/An6hD2BJt15CdkqpOLrm5ttymu82uoucjQ/pOdn5kbQOmkXYX9+SOv0nggUzPgxmGcrntSgIRAuhFhX6DGpjHbjgW8rGsxY+Z7uNO7K5aYXaX9oNb8EtGNNWCLILGojW2R9oeLkMMF0YxPR609y/tpF2FwOXKceZWejeDrEvFqijxsHCQHJmLJcOBWB4tbo88sGhsf/y3uTbqdzzFH6+87HojiLuDLkPS3s9VAYKcEtFUxCoyJsqpu3W93N0pM/sO3Ich7b+D9isw7htKh0Tx1Gz8BK11o0aBgkSil7l9dACGEBxgKPVTSYoXzPBMLb0im8LZ0AJ27u1n7nJDmnje23IvqzkmjiUNBQummY/tBXs05hZkfaLlrHZGMulo7yX54nge1IPyt/Hr2IucmX0bLdfp645Fna+e+mjbqvRKJ30A+tSe9La9seAELMqUVCi/Xnlf9cVcXJR01gv7KCmMWL0BRBZqvOhEQPqOanUUtICTkpYAuCfX/CgX+g583QyPBDriIXAhuklBVuItRY+QohYoAvgUj0PeXpUsp3hBCh6Lt/LYBDwFVSypSazmdQPmZUHk9pygOBO3GZLWUWj7SioKKQ7aXaabWDG1CIIBE1L9DBopAeHIgtw0GGrz8/tb6EczlKAqsJpgUxDAIkp9gKSA7nNmdW5o04zDZaRB0g098Ps6nQe5YSBY3O6TsxSxcpWaFsCOxFqCUVAIdmwoKGRMv/GPOCLoKIJY2D5b6D9lwJQKvGj5D+6JVIexrBvt1oULcmbie81QrSj4Jqg7yadyvehDvWQ5MShWoMymYClTA5gHdWvi7gQSnlBiFEALBeCLEQuAn4W0r5shBiKjAVeNQL8xlUQPOQHjywbDpfd+7I8ZBGoBZ4BSjoCjoYKyelntKwIS6Q+9EUzXEQ1bwSByo2BFII/EUTnrv1HSzJRzkU2Ry3xcq/PAtko2LFTiZtuIhgYknnCC5py1dzJ51Rpe5yaAjSzYH0SN/K7rBWdLVuKjgnFboq15Lqgv3KN6jCRYKjEY0s8aSJ8hVvLBfTlevyXweqsXg7X7xXeKu1rnihQPECIGHddBj7Yb2IdbohhPBDLyBc/k6wB2/UcDsBnPA8zxBC7ASaAuOAIZ5mXwBLMJRv3aAoDDjvQ85N2M5vTjuz1WNk4SAcX55iEAJIx85U1zJUs76KK8vGWR/YULmfvnxunkGQSGUn7UmQjTBLN4miOff7DeR9v7WouJlAEHpBdnBjJ56NtOEihvA8+5hPR5vCjkCVb9PAocVgcQ9AKCvRcOiTCd397JBPDIds0UihoKKhafopq+JgK1/RXD7GpN3fEmqOI8pygr6B/9HFfyt7c9oSaztAuLl4kUw4wiKi6EY0fevy46sa6z+B9CNln49uwLI3MKSUWUBYZdt71dtBCNECvZjmaiDSo5gBTqKbJUrrMylv9zAhIcGb4pzdKAoisgtjbb2ZxSXc4/Ft+pj1hONDZxqRcTwAqemKV3MJ3I560rx53gNSYrPn8Or797F7/v0Ek4qK5ARNOC6iOSyaIzFhxcQXjOM7LuMieiFQULGgYiWGgfz99wFefGYtvodG0VFcwVdNzWS0lyxrO5dg83HaMgZz8XpxeWWNBEW8HvT4DI19yoe08d3Biy0f4a6m79DMdpj7933A0tShmIQzP4NZYdzYWcH/OMbKUt6zHVKugfimkHYvyIo38LzO4eXwawWxtb/crK+Mf70d/n0VUg7ViWhnA15TvkIIf+BH4D4pZXrhc1LKMssXSimnSyl7Syl7R0QYYZa1QRwZvMtaEshmO4k8wmKSyMGZaUFKoSsaKVDUevIR9ngXjP3vd2a8MZnmp47hjvuPY1o0TqkSLY+hShcmQEEhtlCEm41gRvAGLTmfntzOT29ZGDHiK6ZNW0arVu9y4IC+zXBCWcQC53y+TW7Dr9nbiaQbzRhCMC0JpZ0uRqFrjxtR9C5AZDI4eAEZrkB2Z3dkfcY5OKWJyU3eIdiUhkm49SCOUj7CgywseTDrA8j9GbTjkP2p/ryuWV1Jc0LKflg/HRY+Cm/FwvpPTyt/8oaKV7wdhBBmdMU7S0r5k+dwvBAiSkp5QggRBZzyxlwGVWdLsVy+J8lgK6eI7JiI5gmtVcxauSYHTXpqUIoy9/BqjMXpINPHn8DcLDqe2M6zRx7HLyiLIDWV9uoO+vpPYACxBGDN7yPRWMd7pLAfiWRlfHPwrGo1TfLBB2t5/fVRbHIcZOqBt3BJFYEkN/oDXgy4n3TtFAfknySru/XxJCQ6wzmQ24pe/msxKRqahA3pvfBVs3hw//sowo1A36gziYLNO6c0011OYaf6JniSwQsUwuhQygeaDDjz3wV1sRftssPS52HLN+DXGI6VsiKvDL9OhF1zYcJPoBihAtXFG94OAvgE2CmlfLPQqbnAjcDLnr9GzZR6Ijv/R67TlEBiCUYVAs2zVKtQmUryc0VIACkRXtbAPwy5lF8GjWHsit/pcnA73313HQc7NcOiOWgcdTMBfUu6PWVximT24vbYcEfdc5w5r7TNP3/uuTEAHMkZiJQCp9QV96r0c/gtfiepMbqvrcWT0VIApxyRzDxxF67GM2huPcTGjJ7MSriJrn4bsUsbSLCJHIYE/cvilBGMDP0LiWDm8TvYmxvLYzERRFpPAoJOXEMqB/mecahYGMzzhNMe/O6EnM9ASwK1Bdiu9upnWQIp4aNz4JSnEGhK+ZuFFbL7d4jfClHdai7bWYo3LlsDgOuBYUKITZ7HRehKd6QQYi8wwvPaoB5ohB9WTx4EEwqjaUNzgniG8ziHKJRKuD1phb8qUqDJgj41vgPN6y8ELpOZn867hIdHvcS9w99jn7ktfm0mE9C39IAEK4EIz3sTmGjbuD0jRrQkvKk/N07rx9hLdZNCN1uWJ7cDWEUunf3Wszz9d6xWO1arHU0rSL7TwW8nbX12sThlJACjw+fyTYfLiDTHY/asdDUE5wX/xflhf5Lp9uWWXV+zLH04Jx2N+TT+tvw3dph/OMZ/gB748R8v6qfUJtDoEDTaDxFbQQmo4YdYNnGJX5Hzuh/aqa1oXrteumDWaEg/7q0Bzzq84e2wnLKdFofXdHyDmjOQGPaTwmri6EFjhtECgC40oiPh3M2fHCezzP7h2Nhvt+JvTUcREkWR+QrXG6Y/s1AIwcYpTzVfKWHfwjb8t2koi7uO4PMLVZoU6+MiFyc5+BDCYKaxnW/xIYxu6k3c/Fs2Nx0K5iup8u/RJHbGhHGutR1Pt3iS1en9ae2zlwsCE5h1sj+59jXYrHY0TUVRXAhgdVo/tmX2wInCE4de541WdxNuTuSmxjMIlLmE+Bynf9AK/FRdXj81m3fb3MlX8Tfxb9oQXFrBzyqDE0XktpPOf7xEfx5BESZdCdciLnJR5z+ELSOHI52bELXvFEKTCE3iMivYcmrg5515Av59Gc65E/bOh5bDIaq712Q/0zEi3M4CBIKb6MZNlLxFVFF4nRH8wX6WcpgjpBfrCxPpwVbbSX52ZmO1FJgwNAm5Th8OJLeic+NtnqOyUM+yUV0Kx7c3wpllZWpsOE2jLLzKyvzeXa/dhH3MTv555nyirs2FQknhT7GFf3kBiTs/w9l5FKyMnz6VjVMzA4Ij2cH8kJ3JBN9AkjKu4bygmezM7sSfybFc1GocC5Y5iI7ZhcmURdtWexGaZGNSb3I9dmULTvbmtCXcnIiqSNpnbaJzky1FQpAVJEGmNCZFfcC2zG70C1jOtEPPc1/0awSq6SU+ijhWc4h/aMmIcj8jbyDRkEIggWOdm7D5gs40OphIZogv7ZfvI3J3ImZZTQUs0SuM/F8X8CQlYtxM6DXRW+Kf0RjK1wA/LFxBBy6iNdfzK65CZXMk8AorQYC1UOI0l6aSnB2GKdON+E+wP6g1LUfs89iBJQI3spyUjy4JER1PEbcumptfasS817disajY8SS4USUmm4um3U4SaYrib/4ljs/xJxMrQbjRy77HsYoM4ggkOn/sEFWgW7NVpBQ0VS0szIIM0wqePPQGTs2MKlx8Y4rn/YGh+NiSyUAPMug3ex2n/Oawqs0AHNKMJgWtrLvRXLD1nwD2mZvQvuV2zCZd2eRoPviqOQCYhZNrwhYy49StOKQNq5ILAg7mxPLh8XuQKFwQOo8sdwDSz0EL3XyMWosefmZ8ybroSXK+eIQmO09ysnUjjnaKQnVphMWlIKqreEFXuEdXFChegMXPGMq3khjK1yAfX8zM4GLeYBU7SMzfjCuOEKAqbiL8ElACJEFjUjm0pCXSLRAm3auwFYcYSkdmeJRkHucQRRwZHDdnIoConnEcWtCK42krGR4xlPnsy/emUFSNiO5HuY7VCClRxXCGsITGnELfrtCQSMzFUjL+EBnMcGcWJ+1Wbgmz098ayLlxsNVxK26p4saEW5roHbCGXOu3uIWDvNKda67oSWzqXu61vcHh3BacF7iMXVkdeT7pWsJbJ3CV8300oSClG7u08vnJW7kl6mNMwoXdbaGVJQBFaCDhx4SrSXKGsyJ9IJrnpzb9xN2ouJkjVG6UIAV81BhuDvHyP7MQrcPuxv3A7TQ/tQvx4yXsiG5J3w1r8cmwU2Pv4qxiTkwhLWo64lmD4SdiUIRQfJjGYC6mNf5YSpzX3PpDESBEnqeEJCfZl+xEP/J2reJoTCQtSxgf7qQXkfjlWyeEkIQHJtEuLJTb6cmPXM5o0ZooLYA9v3fA3NqJW1P1um6Y2ENbBCbCaIsvEfRmMj6EccyVxQMpv/JM5sskm75gb4sAMttZeTc8kOuOwXo7OKQNNyb0ehYuWvnsw6w4isjnNqk4fVSuUr7nSdsLaFLj4/h7OOWMYldOR36xTsKJCSHgpcNPcSi3FZpUcWlmUt1hPBY3jFzNBkh+Tbqc5emD8xVv/hyYcEiwAw4Jd5zUTTi4dkHmS5A73wv/yaKomFHCO9Fi1yE2H+3FrNDrSbCG466pCojoCP2mgC0EmpwD43+quI8BYKx8DUpBReFWenA9XXiZFWzkJBLww0SaXWDyLbD7unJV3E6VpJ2RtBt+wHNUkIMvszjOe4zif6zAiZu76E0YvkzmHF4VK9jrTCPjuD/DJ2+hnTIeABMqk+jJTe75JMeHEZ/YhL6+y9iX1A5NKrQM20df0xSaMYhTZPGeXMd2928s3D8Eh3YBgvM52uhbXghbTwgtsRHC+twi0RL4qS76NF9MlikApBlZyBUv3tGYEY4ltLEfREFjle85CM+VQsNEkiuc1ekDGBC4nEvCfyLT7Y9NyUEREEoS3fw3sSajv2c0j/uE57niuZeQxRSeAuA+ConngMwBrBD0Ifje4JX/Z8FECvS6jcfWvYKGQOT7f1R3PDOMfBlaDYOL3vaSkGcPhvI9mzm1A+ZN1u/xR38AEe2LnLZi4hnOK3Jsws7tZHTZiZSCpF0RXCbaMS4mhBZPWfjRHMNsdnhaCvywEEMwH3JRkTEi8GU4sRwybyK4RRoJKCzmKGMp8M/NNucQ2ukU+3M7sWT/Bbg0MxJ4J2kqXaMUbgjRuEP+gQuNFEcoTmnG4fHh/TdtIEuCX2BJWn/CFCtX+V/LyykWj1QwImYd0pbBKcJYIM+nf1YTFmdnc9DRhA0ZPdgT2B6zottCL83+jbdte9iT2x4QnHJEMuPEZDZl9sJPyeLqRl/lKzBFaKS4Qgu90wLFC3hUb95PzkWAmo1JBvJsOKxKX0s/KVBwA9lg/63qynfzN/DzjXqo8oBHYNRL+afSOELu1o8Ii9+ASTGhaDWw9dqC9ZwPPW7WFe/pTARwZyXbGtWLDbzGl6MK/DS/HAUPlpNgxcN7bTtx3jvNOZbpoqtPIPc+qGDzeCJcTUfWcJxDpBKElUfoX/5g5TCGNuQO3EXy4UYkZUeSp8gkkjtOQKLvHlwWPSrP35rp8UPWsAgH3fx28dCBF0lyhSOAcNMpLEQzOugI10d+xlqTynZaI1E4cCKGT94/HxwChgEt4CfHpdxq/QQhNX51XMqunI5oqIDA5XG7WJPRn34By/FVc3BLBbc0MSv+BvbmtKXoipdCz/N+bpJGpkR+aD2H3Ox7uOQoNFPOYW2AxE+oCKxgHVP1D+3nmyBPqS5/Gc57DGyBJLOX7ftu5dwf/0XVZAnpqoRigcFPwYAHqjuCgQdD+Z7NZMaTb3zNrFwByfAA2PGwPw4XWIp9e1QU3mZUpcYZRgv+4yjbSKANoYyiZZHzV9GRjSIemi/j730Xku30R1cZEkW4OVGoXJJZcdC32RJ2HhvI8xEqQVYX85LDcUgbAKdcEahKNhc3nopDzaILAhtpbKQP//3fEDjlUUVzgTvgAd5itasfm91d2al18MxbdBULsCpjICcPNQEpOWRvWaxd+SS6GjFk190AaMAudwyDM9fyeegvdPHvCraLyh+gNAp7HYAeTgwcZy2BJxMRWkHCeIQJCns6WEPBnlzxHCEt9CTrBjXG2HA7mxnwMJh89MeAR6rUtbjirSoWVJ5nCJ9kXsmyl4bRaLKJu74q8J1dwAGOkIYQMKDFYnzNmQjcNAs6gJ8C55lXY8OBlLo63BXfjQdCfLkzxMoQ8wW4pUKey5tTWog0n8zPw6Agick5Sp81rXFk2gqE0oAjIN0K3zknsNPdCf0nUnwVW/A4lNuSQ/bWVO2nJNBQ0BBFvA12udtjCphaPcULMLDQ/7DV+eCvJ6oKox0n2zdDUwX5Ga5UMyh5/0QBYS0pFb9IvV14e5i8Fe7ZCT616JpxFnHWrnydTnjzTdi3DwYOhNhY6NoV/PzAbC7a9tAh6N8f0tLgxhvh/feL5Cc/fRn5P+hxk/48vG25Tb1FPFlk46QFQQgE0+bB2kPgdMOXq+DSnjCyU9HkYD5mOyPa/AESWh/rh831GbnKX1wunORKf+bH/Y/Gu07xd4/1LDziy87c/rg8nhoSBSvZnHBE5ocGO10mvvnxOhYv6wRBLshV9Qm7CvgbOB8IRk98XsTho+gGWslVbvW3r3pa4ZVI6GCtuG2ZjHwJBk4Ftx38G+UfjqIXWviz7Lh3EC1X78RfNIZet0FuKmyZBVE9ISQWPh1cdDzfRnDLkhJ7AQalI4QIBmYCndG/ILdIKcvMXnRWKt/jx2HSJFi0COx2mDmz4JzJBE8+Cc94AqZcLmjZsmBF9tFH4OsLl10GY8dCTg689x5MPF39yutI6QIsYD8fsxENSQwBvMUocp0Kmmf5JwC75064A2HY7SAsRZP+fPRSMx6+bz+Kqnso+OLi8PYEmg8/hGrRcLuyiE5dT/yp/uDZ0Xd77LTfHHuPSTFP8fRrD7B7fyd9xhRJ2LgT2No4OeGKQXMpsBDog/4TKkLxFXDeOrJmURIqML8ZRJorbFoxPkGlHm5KX5qG9IULip2I7lPw/LLPYeHjkHECkODMhM1fw4gXvCDYWcE7wJ9Syis8hTTLrVtyRpgd0tP1R2U4dAg6dIA//tAVb3FcLnj2Wfj9d4iPhzFjSuYvmDcPhgyB5GRd+d5+O2Rl1fBNnAV8w3ZcaGhIDpPOd2zniYshOkT3Gz63NQxorbf9/NRR/njgErJO+emmBQkWt4n+T/7BD2lXMuHuH7n27jl8uHwiCS5fpCdjjGqS+HhyLujbcwIXFlzSwuacpozhU/Yd6UyBwhSIPQptY3egKUIvAnMz0BzKCdDL71vwEyqcsrrYF6bMbNZuTEhmNwWfhvBL7H4jXP4VWD1JfpzZcGxV/cp0miCECALOQ8/wiJTSIaVMLa9PQ/iX14jGjSEoSH+8845+7PBhuOIKePzxom23bdOVZno6+autsrjhBn1lu2BByXN79uhKOg+328gtXRn8MCM1OLmpCXFrozngyiAmFA6+Ag+Mgr93QqP74Y2/IGF/CCBZ/OSFHFneAs0NDpMLNcLO9s96ouWacOVa+OebkRwIb0vSgXBcDhWXQ2VXat6SNU/Baqi46OvjUeLtAaVAUWqNFA4mtSroYkI3O1QSNT+Kr5QVsAv4FNgIJHheA2NDfyRITUdD4/oTWTTeDQvLzm1UdzQ9B6yBYAkAsy/0rlQ5srOF8LyqO57HpELnYtH/w58JITYKIWZ6arqVyWltdrjmGn11msd998H110OLFgXHpk+H3bvhq6/g/vsrP3ZyMqxZU/n2/v4Vtznb6btjIHP+zCRpbwSaU2GbRTLkfjhwCl7/q6Ddoz/AumlN+EzqCXbdThXV803NcfgVXUVKAQEmVmuD8NuVhcNiwSqtgCTEJxG3pqC6JBN9j3JdwJdcvvEucoa2glYCNkrwheQeESSnVqWKSlHbrxsLZZoesoB09AqGJmA4RHY/QYIrgnR3EBLdTc0BPBwPm+ryeyQlpBzQo9N8Pf7J1gB9Y+3AIghtBVE96lCgBk+ilLJ3GedMQE/gHinlaiHEO+hFg58qa7DTWvmWphy/+qro66QkiIio/ZXpkSPQrFntznE6o2lw5/sBZDsK8tbaHXDeKxBSzDJmUqFbYzMbHoeZm7MIbh7ETlQkGmHWBPzGpJM1N1DXgeeDCYFUFHKUAIbYYFoETHGsIzJQ91tuGXeUdsFbeMZ/FJsaA6kCWgItSy/7UzGl+fCWQSBwC/Cl57UZst0+HM5pnh9f5pC6sdfXW/eh9ky9GnFIKzCVDBEH9B/EBz0gfrP++vzXYcCD+nOfYOh0Rck+6z+FpD3Q714IrN1UmKchx4BjUsrVntc/oCvfMjmtle+LL8L48QWvL70UzjuvZLu6MAksXgw33VT0mNvpZO/8+Zh9fWk5YoTXKz+cTkjAUUZQVUp20dePXaRvsnWJhnei/YDWbMaPZ/gXaTIx7MK/SDovlMSklvRUYnksAroW8hhby3Ga+h7M14vHoyPIET0QQtI06CiHU1t5Fq+iBntl5W20iaJPA4GxwB6gDWRowWRoQRS2OwPstnuhRFPSXpjeF1wOXUHevg5sgSXbHV9foHgBFj1eoHxL47srYccP+vMVb8Gj8bqSNgBASnlSCHFUCNFOSrkbPZf5jvL6eKuG26fAaOCUlLKz51goMBtoARwCrpLSu4Wqrr4aAgPhtdd0E8Stt+ouZD4++kZYXSGEbnt2OiE1FcLD9WOzLryQuNWrkVLSc+JELsgzSjcw7E54fzH4WuBwe3g7FSJVWNgc2tbE9cmDpsGxlIrt7HncPbTksf2k0taxh8lp03Gj8G7wXYjmGm8VC85w4uZVTwpMoWmAxFdkEZ19nBx7AP7+6fSJ/pdNh8/BodqovvYtz+WslKbNPY8S/Quep2q6SbhGTg+r34ecVF2u9OOw61fofr1HUkkiO5BIImzBRaUWxZbd6XGw4yfd9ND2Itj5S8E5zQGHl0H7sQXHNnwOv9+t+w9f/QO0OivrKNwDzPJ4OhxA37otE2+tfD8H3qfg5gr0JfffUsqXhRBTPa8f9dJ8+Vx4of6QUnf5euQRyM319iy6ki/No8JiAYcDRo/WfX8dDl35btuQyeGlS9E8O3Obv/qqQSpfpwuaPgRJed4aFwM+cCQM2i+GyIUwpB28cgU0C6v6+Au3wyX/B9mOitvmcf5bcHUfeLiQW1R7Gcao5Jfxl7qgTyS/SkLkzhJ9XWi4PaEL/jmZRLlPoCkqLx19Rr/NPwWDGv+DnyUTh9uzXC6y3KyM65jEhANFSILVJE65mlb+zZXDrcFgrunNUUA0mGzg8qw+AqLyT63nQw6zFAE0DTuXvp3Hw7bv9JOKGdKOQlCMrrw/6A6ODD0S7rzHi0bDAQQ201fXUgOh6iXmQa8J+tWF8GwV/uFnCFLKTUBZNuESeEX5SimXCSFaFDs8Dhjief4F+paD15UvwNatMHSobt+tLcpyZXN4vmNuN7jd+g83MVEysO02LgsdgF/ScpxKAGnNr+P4cWjSwExlqw4UUrwAv3v+qiBbwcl0+G6t/hjdBe4ZAaNK1rEsk4mfV03xAqw/oj+SMuFlj+mxI+G4tdx8vRjiziWCkptkPpi5RLbnjWQnh1JjGR/yPTuy2+aHGgupkZgawbCQhezI7srO7M7FdG3ltJ+CG4e0etJHuqmEX1qZBAr4vRkMKNcrtJL0nwLJe+DQv9DjRmhVUC3jEH9jysrinF8345fyJ7LxOIRiAs3Fbp9mPLonBWvGVt7c9gRNndme8GQ7rP+k5Dy7f4MZffX8or0mFT2nOUu2NyhBbbqaRUop8wpYnQQiS2skhJiU57qRkJBQpQm2b4fOnaFbt9pVvNXBlZvFwoz7aHXtvXxs3c87e96mbVtY1cDcJhuX7pOv65M9RQ/N26qvSs/9H/yzq3K2dFcNsnX/3xJ45RT8mA5uTeGVpVOxu0zkuiw8/XfJeqwL94K1J9wY3YUtn/Qg3R7CZ/E3oeDGLPQrQBNLHA82e4XLIn7gsWbPMTRoAbc2/oBhQX9Rpp9uCQQOfACVdC2Umv6MnBL2O2to683DZIFxM2DKLj2xTiH8aEzv37bQeE88wfHpiG2zQejZ4gYPWchcn0782GgUF/f6TPfxBT30XJbyT/znaXA79HwS6z4sek61waavSvYxKEKd+PlKKct0M5dSTpdS9pZS9o6IqIq7D1x1la6AG5aPrYbAxSk6sytnAI/98yYJmaFkZytkZcHbb9e3fEVpU+olsXxWHoBhr8ONpSyIQLftPv0LnPM89GpeepvKkOmAJ1bAhCPw2sZPeGzwi1hUF7lOH37acVWRtrlOuPB6cGwGjgNPC0gEp7SyOWMEUuq7/u18d+KSKlszu3E4N5ZboqYzPGQBN0fNYHDQIirv/iCK/a0+dbUNO5hpBKeqqFpe8TkztL0Ap2Lm3MT/mPPfFVx76GsO+LcEFN3X1+2AtMNVm8idC79OguWvQXYDWxU1IGpT+cYLIaIAPH9PVdC+yqSleXvEmmEhnfb8hMRENpHkEM7Ro0Xb+JXrdn16MWt1yWMpWfD5CnhjAaw7DPO2gLUGxi23A5wK3Bt1t+6cICDIlsZ3N34PwPwt0PVZCJgM7n1QJFNNlq7Ynm0E3Tzm3d3Z7XntyJO8Hfcwzx+exq+Jl6EIsChOroj4nsqvfgtTM/XZxwcmlHUH4kWSMbNj+LNoZk8ypcCmkHoUi+bkx/+u5PK4X/h8zc18u2I8+EfpNl/pplr+eO5cPVT57TaQWnGq0rOR2lS+c4EbPc9vBH711sAOh57gJjFRf60oejIcUz07zjkIYhdXUN6qaNkyePBB3UbcEPhiefX7+pjh1T8gx6Gvdi/7P4h8AO76uqid116Gi1mFaMAGIB2ypA/LQ/qwMOw84q3hmJuvZONRFxe/C1uPecwbfdB3McygjILLOsKKFnBfGIR6vulxjmbsyulAruaLQ9r4K+Victw2ct1WFqeOoCCLWSkKtfAtlhdvt5bkwHXH9JJCtcUx0nmQhbzaJohJU97lUKOmus/uiXVF2gng4hN/QGZczSeVLj15zxfnw+ejYPfvFXY5m/CWq9m36Jtr4UKIY8AzwMvA90KIicBh4KqyR6gar76qB1Pkff81rfJuTPXNvn26V0bLljB5cv3K8uTP8GINfg9ZDnj0R3jtT5h9OyzYoWcn8yYiCZgNX907gRjrEaRQWBbcH3Cyc98OoGtB4yZgugFeuBce6gKqApkaNN0Nx4vIVbDmyHT789GJe8hwBbI9u0sFwgiQEiHdSFHVDbY8zVr6KnlOBvRMgqnhVRy2AjRcrOdj/iERF+1xa5LJP79H9MmDRdrVnulDQtIu/XF0OQx5Wk9P2W6svmo6i/GWt8OEMk7VirPfgQMNzc5bNZxOPcFPffM/Ly1EErNg+JtV62NS9P+hu4L/o1kBLQsUHw3p8UV1uK0kJjZhaKcsni3U1scCn98MV3UrODY9pbjiBSvkZ2OQqKxKH1jq3ApuEBKBxM+SSbo9GIRAVsuzoWJXti214Ju+kx854pjPqHmHuSouDcXtIjr5ZP0kdXHmwN9P67bmnjfD6PfrQ4oGw2l56bn33vo3MdSU226rbwlqnhC9JvhYIMAG+ANd0KPASsHhBpsZZv98LXaHhVy7lS07uhIpm9A/pDv/PgJju8Fz4yD7A7jqnKL9bcX0nJ+Azyrp7icRaNKEW5pJtwcT5uvZtsgzPld6BVC4XdlrzEQX/C8BNnpRCcexmo6Ld9Ft03ZaJBwjJvlknW3wlYrmBFc2bPu+PqVoEJyWyrd7dxg5sr6lqD5CwIABsLqUDau65OfJdbfTXpyMXEhVgBvQDVZXUaarrFmFrVsHcv/jH/H6m69wZIqDWRfHc93Ap+joe4Jf74GnSyl5JiVEqBDtucgECFjWAiYEwxAfPfoNZNF86YUwKQ5PrQmJSXHichcSsEpxwIVXvWUr7IU58EQC9DkIa7PLbFZ5jm+g36e/0XJdgbdCGdbsuieqV31LUO+ctuvHI6fxBqqU+mbhrbfqASL1Rc9mutnNXV/28ibomsCsP2wXQe5vJZulZIM2A4QIY+1HPzA6YgJJsbEoaCS8sp/F7xVEcUkkGcRhwsYXKeE8dNKNXUoCNSe7W9hp7BMMWhY9te9Zwg2Aik0kEyvjyA4M4Gh6c/LUk0szoZebVHBpJtLshUL8quWUW7k+LuCG47CzdTWmyENzw+cjCMxNKZoErsrS1BIn1ukJgKz1mw4wxRzE91GDK24I6D4E3uO0XPlC0Xy6pyuOeo7A1KSexLzeOEXBN9AJPVV9lVsaea6p7sBGJAa2wW3ywWnyY72jTZF2a3iHhdzPfO5kdno82ai4hQncDjYueU9v5FhAC2UbVuHELByc22IpbdrvpEvUetqGb88fS2KiYDlevJZb7bKvpt8Nlx3s6aT7BvDt0Cv5YdA4csx6og6HauJkUHi1nOq8Rm4axK2tj5kbDKet8j1woL4lqDm33FK/80cFwxMXFz3mU0ZWFwGllNWpPqoC5kxgDrAeWARb/oABrUq2fWq03h6gz9XjaKImobrtWKSdUd0KjAZ20jnKv7hxoOGgU8A8fF1ZmN0OQNLzuJ4Zf05WJz6y30qQmkpzy14EEkUBk+qmkf9J773JGuBf01+mxZe0Htfw6K3T+OG8S/hm+NXc8OgMkv2DSfcL4rshBSkj6+X6qzl1N7SzmNNW+ZaWOvJ0w+qFjGE15ekx4F9IjlA/CPIp2ibQB16/Cjp3BwoFiYQHl1w5d2gM1/QBP0vRIjvFcWv6ht/5YSCWAjv1zbVBbcBSaPV7TguYdknBa0VV2PJ2Y14Zb+XdG6zMmlTQ2IQNJd+SJrg0dCff7XuBF3Y8z4a/B/BT40cJejiTW9+O4HBiU065GnHS1QTNrOLWFFwulRNHoyv+0GqB4gv+J6qRxKg4W8e9wPGwKNyqCU1RCUtPwua0E56exJRfP6r5BDXl76cgo2Fc7OqD09Lmm5HR8HIkVIffftOrb9QnQsCvd+sJcMwm+HoixIbD/bP1kN0pw6F7c90z4WI79AmG3B0Q6g+rh0CLB4qOt/cUbH1O9x9++Q/I8eRYaRkBhxOLupZJqWdMW75X9xk2K9C/NbxwKTw7F4JscPtgPQ9wnmeGwwVfroS0HJjQt6jHhoqFQTzNJj7Fgj+9xd34jQthzPF1JJkmMPnHtsgbdDXnszELQhUyXcEsOTCKJoFHybb7E++IqkmOnGozzA/GB8APGXBJAEwKrfmYvUQUUOCVcdfcGfg4chvOplvCdnizOXS9Bi751EvJLU4fTkvlu3Nn6cUvTzeWLq1vCXSGddDrqBXm61Jc4dpZIa4rHOsArSx6+sPeLfTS73m4NNgdD5f3glf+1I9ZTRKrkoQmw8j72ZtV+HYS9IjRs5dtOgZX9oILu+iPMH+4exZM+Q4+Xgrf3w5bj8Or82H5fn3c1/+ChLfAr1Ai9Qg6MZI3Cg6oQEw/VuyIRzkP3Ko+f04PPz15kCKwu3w4mNxWN37mKV6PO26MCkdrORpRAZ4Kh0F+cIsXlG4ePpjpSiO2aCdAUbC4nSgNzUHe7YCt3+nBFyGx9S1NnXJamh3attUTpp/uqOrpFyzir0B7a0He2RWPlWwTEwKdmupRb/1aarg1JzvjCzZ4FAGRgbDrBDR/VM8DsS8ebh6gn7c74ZE5+qo5yw7rD0PsYzD2vQLFC5DjlNz8WcUy/7MLxr/fCFORqqeAkKXHPkiweIQ96i7fPcwbaNRe9eLJ6VH0OLCVwOwMpl98Cxk2P9xCsKNpTVwpvIyUYDn9iyAKIQ4JIbYKITYJIdZV1P60VL7BwbB5s14Q83RWwooCc+bUtxQ1w6TCpqd1ZRriqyvcAM//ZHQ3uLZPKialYOkY7qtnuUrJ0kOT81TboSRoORUWbgOfO4vmGNbK1H2CHzZqHM9xsC8e1hyAz5brfwuzcAdkOwQ3HfkMkeyGZIltfg44hK75XGAukj9G4lDyJMu7Sa9dBbzGG369xTm1nch3e/Pw92/z4Tv3keYbyHWPfcJVT33NMzc/hVMpsK/U6xqgywTwq1pGwwbMUCll93IKbeZzWpodAGJj4c03G16KxqrgdOpl6OuDbDt8s1q3mU7oo9t7q0u3ZnCyjPDiYR0CsKg5WFQHbk1lQOxhft0eRlYprlTHUuGqj0tRBIXjgYHCeRKkJuj5fymc2t0IKQUCPXrum9tgXA+PDO3hnUXw5/KLmdr1VUzmplx7wyC22WN5LwV6hsEtITB4YwaZkSYcIq+8UFGvWAs5njy+3uf+U3BjCPh5czm05VuEIws/JA7VTL+da1jQcxhXL/2JsPQkMnz8CM3SqwQcszSiiSMBxfOe69T6mnwGuC5Vg9NW+ebRrh3s2uXtUSW+ZJBNANX9Gio48SObDErmChQCAjxFfK++ugZi1oCRb8Kmo/q7+20zzLnT+3MkZ8Kot8yE+KbSp+m/3DfCwZaEK1m4V896ZjGVLKpZaq6HUu37Bf+X+F2N8l9LwO52MWdzJmN76J/9oI4ZzLvXj//2NWNUx8fo4yn71g64vNCIa3sNZdip2Rx2FK4Ll6eMJOEkcpyYyn8AVUDKWlh9RnYGsw84szErZgbv3Ezng9vptXczZreziMUlyJ3JSx0e49GdL2OijqNujiyDzATwb/Cr3/Bi5oTpUsrphV5LYIEQQgIfFztXgtNe+f7zD0ybpgcsPPcctGhRswCM9soOpvtPoolynPPT/2K/1pqqK2BJGMmsCOpP27R9yGLWHSn1vL6bN+tl7esahwtW7pd6TTPNza9fr+Gh/RnccUdvWrf23o7Pb5shNRuyHBEcTrmMiAh4/xo989nyfTC0Pdz5ddE+M2+Aq8v9ykLBqrSkSUBRnDxx/7O0an6Av4khh0RySEK0V7mr/SuE0qaMMSHUHcBNjWYy7diLnrg2jfetd5El/Fjp7M9PrisLpvcy/X284NtbnM5XQ+Yp2PsHIn4rbQ9vK1mLDU+BZXc2j+x8BXNdK9481BqVDa0rEiswJwyUUsYJIRoBC4UQu6SUy8pqfFrafAvTuDF88AHMnAkrV9Y84c7b/vczyPIfN2d+Xm3Fq+IimTA+dtxBa2Vvqa1OnIAwL/hyVgfL1rV0St+OxW3H9N9c5Kq/ePONFfTt+AZZjf3g+nG6TaSGxEYUqEWzCp//B22fgCt6w+w74NxSAir2J0DLSqVVLFn9F0ViNrt49b0nmPLk/3E06wQ56DZmiZvVvF3uiMGBXzNZbmRxzFgG+m9AQ+Uu+8c8nP2mrnhr8V78aG1EbAoB/e+FSz+D7MRSFW9hTLjrx/arWM6IMvRSyjjP31PAz+gZpsvktFe+hcnOrnnl4p3uDuRIG9tcnajur82NGTcm3smZwj6t9F1li6Ue3BqlhMk3wqg+LFs+iGd2P0fzA8twaQKJwOGUHLb7wZ9zYUBH2LFF7/fPAvhyBuRULd3WeW3hzauha7S+2s12wL5TEP0w/LYJRr2lp5YszOM/w4HEary3AKAPOJwWcu2+ZGb5s2dfuyJNREVfd7UpEUF/MiRgHhsyCyV+qQPH2O62ittUG99w8A3TqwybbHpKx1IoGUBdFzfGil5qfs2HsHX26ZOYuxhCCD8hREDec2AUsK28Pqe92aEwmZk1H+OxrJcw40Cr9q+toJ8TC8W/zooCjRrB11/Xg/K9biwsmAdAiDOVx/e+xPiAYPql3MqttvX0NJ+gjeqpuXVwH4zoDecOhqWL9GPTHoFdCVW6vbh9sO4FcfXHBcc0CWPfrwUfAlUgFQU0UBUNX3snfDlENgkomOjHg5Ue6q5QeD1Zfy4o5BJYzv9MgWrftPeoTeWrmuD2NbDqfV0JH1oGe4pnMCpN+tpOoCKg+/Xw+2TIiNMvDgeXwNgPK+zZAIkEfhb6j9oEfCOl/LO8DrWufIUQFwDvoLuvz5RSliw76yWKKt+yk1aXRy4+3J31gZckKjq/2QyXXgqzZ3tp+MqQmQmvPA0nj8Pikt+FWCWV42GvAwKTKKYKnc4CxQuQlgqDOsPM76FTVyrLmDKaelXxZgArJVHd4kjfE4TTYWHG7z2YLMOI7fM2UkhOsZVgWuCWMD9T//Kf7196cqHXGsNQP1iZA+MDIV2D5dkw0g9eT9L7t7HA+lxdZQUqeoBJVsmhKkVKbeu5wKYw6iX9eZ874cPeenWJfDT0n2gd1rcKawvdboBNX+ivpQabvzotla+U8gDQrcKGhahVs4MQQgX+D7gQ6AhMEEJ0rK35brkFQkLyXpXmHC+Bmtsyi45XORViNsNLL8GXX3px+spw783w2Qfwy+xSdyKFAJOgpOIti327YfSgKkWH+Fjhq1vrwH1JEyTvCMNht+BwWjka15wnPunPz/MvReJmC/qP/IpjcPVRuOQoXHm07OEuCoDnG0EnG/T3hYfDobsPfB0Nye1hdUs40AYWN4fjbfXjxZO3V5YaWsuqhsUPWg4r5UQdFxbsfj0sLxZa6T4DQlcrSW3bfPsA+6SUB6SUDuA7YFxtTRYeDnFxsHEjzDh/AhFsp+itlEBf7xRPple9NZhSZGwJxAOJhV7r+PnpnhgTJ9ZDMp2tG70fi52ZXmXb3HX9oF1j74pRGna7Dacrz6YpcGtmVm/oD4DZnYU75TZ+SZPk/AmOD+GnObC/Eqbs+DQ4938QPgVemFdwvJkZhviBrwLjAiComsp3UkjFbbzKufeBra4nLcaSaboJpDBVro13+lLbyrcpUHhtccxzLB8hxCQhxDohxLqEhIQaT+jjo1e6GPPak0wOvpgYVlBUyRY1R3TnE+6iM8N5nKoqYa1IBpb5wCfAx8BS/DlOgJ8TPz/97v3ZZ2FwZXM2e5OuPb0/ZrQnC/v3X8GkCfDNp5VSxj2beV+UkhR2P5OARqsWe0hNDGFQyipU+7eY17phB5AD7ITbfqh41Ifm6DkskrLgpfmwsZRk/pkaxFfD6Lu4WS1vuJVGWBt4OA4Ca8dvuVK4HaAVi7bpdkP9yFIP1Lu3g5RyupSyt5Syd4QXnV4ju3ThyVP72H6yPdERaVhJJZaFRLEWPKVhLKTRl7fJIJKW/EUbqltR0o2elNYJuFD5h3tpw5xJjyKl7oPscOhVK7zgwVV5MjJgbi3ELzdtBn3bwuQb4Ofv4P7b4KWnKuw28yawldhl0BVk9beqSkN4htXwd2YRMiuNDnsPEepMBQRBe10UvtDuq0SQTkZuQcUPReg5JwojJYw4XLJfRSS2haH1ldbA7AOXfl47Y6uVvJrIQv/3TlfCuI/LbnuGUdsbbnFQJCQo2nOsTlDNZgIiw+kaNI9zEvQiXxomUmiBRCWcPUgE3/ETPiRzWAwrdfHr41O2l5XZrH9/XG4fIBuBRghJ+PkKovv14/xD8Ndf+o+zWze9fZ2x/O/aGXf18qKvNQ3++BWeeLHcbj4WSHhbD75oHAQ3fALHUgqvVDW8th4QAoSKcEvuP/Au/VMPAwpucz9G9PqX7/4Ynj/vNT0rthW8eKme+jItB0Z2LOmjnKLBxioabgUQVt/+Rq2GQYvBcMjLKfbc1bBix60/q9JK1va/fi3QRggRi650xwPX1PKcJWiXo6e+EoCKi3D2FYqPklzA/bzHHqQs3d5Ulsk0MBAyMwWapiLEjXTt+idB5mSubXGCvmM/otNVVzH7Upg1Sx/j+utr5e2VzW8/1t1coy+vuA3gb9Pz8D707TGu7DCbw6kt+HnHJYXKsVfPS6UEUiKkhkVz8G/oAK75aS7NtoTw6f3ruXTcO6Q6ctixqwujeh/i5TFDKxyuU1OIf0tf8QaWkt4hWIHGJjju0n0GAgQkVWDFimwo5s2RL8OMc6nn9Dq6R8ZZRK0qXymlSwhxN/AX+nfyUynl9gq6eZ3mPttJKfRao2gmQSe+qDg9Xo36R2IljeYsZS8Xo2ml/0p69oQlSzzjyEYEBd1QIkev2Qw33eSlN1JVNqypm3lMJrjjvko3X7knhefObYfVZMfhtvDS0kd5Yekz6P+Vwv+dmjHm5FxGnlrAo51fJ1v6cWgv3DE9hjvudnPzhE9QMNGGsZUeT1VKV7ygmyJWxcLbyRCkwH1h4CP044/Fw5tJUNi6KYDpUaWPVef4R4Ji0kv71AtCDwS5+uwqJ1/rNz1Syvnou1H1Rq9bbmTR40/iQkWgsZ8RhHKQMPbiwsrvvEMTVnMEfUfMRC530IMT9GAfFxRalemmAxtpPPiAi/U7wvKVL+h5JRoUA4fCgb21nzTY5YJ2EXDbPTDtDX0zrhycmTPw9c/W3dyUHMZ1mMsLS5/2rkxCMDdqHPOixhYJmNm0L4o2XMwh/iaYlnTkSq9N2cQMr0aWPD6tkZ7/eHOuXqUiR0JfH+jVUNKhhsTCsGnw95Mgi7ub1X46TS5+H3pP0oNB6pgkwpjFtZVs7d3qxfVtcaoTBkydijkkjIlTojng6EsOYbSMlWxZk8x/L7+E+Y0RnKQLn/IfmmKhuW0nESIRW9bfaBQ20krk5q8ZyW3suEUy9OHH+Lnts+zZA82bw7vv1ttbLJ3n3wKhwBcfF93YqA00N3z8tq7oX3y73KZdYxohnQKXW8HutjJv92gKfuRe3AMWSoltvNsGCbpyA63tN/B//8ASJ0weqlfOqC3MQlfADZrzpsJ/r0NOUt3P3fPmelG89Y2QDaiUQu/eveW6dRUmgK828+bBDTfovrbz50OPHiClZN4dd3Bg0SIaXzyJ4IsfpUubVL7s0Rx7ejrvsptkWpGnFB4mAj9PshahKIRP20PrntGMurABVMMsi3bhkFxHP6qY5rDhULlNVh+QHNp3FYqQ/LLzEr7Zci11kUH2gZHw4PnQJBgueAuW7NZDnVtFwI7nz6q9ntJ5JRKyTtXtnN1vgssqUY6kFIQQ6yuTtLw8gnu3koPXvVSptnPF1TWerzBnzeXG6dRz52Z7KgZMmKDnARZCMObj4u4twdyyYgVrH32ImctG8ZPjOYRqZlb2VeQSgi9JunVS01j85BPcxye88kIm93uj5GxtUHYpCO+TlaV7P5RjejiVLhj//RwUXB5f6brRem8uhA+XwM4XYMV+sHsC/vbE68VCfSzldj/zuWo2fDGqwPZr9gWnF0psBLWA9CMFd1+Nu0Pn8RAQCd3qehe64VDvfr51hdOp+9rmkZJSdluARp06cfG8P7j054/46vlDfPFfO3r2NvELBVdpAbTkb5z48fH/1UYdGC/RrVfFbbxFanKFq+z+rfRKxRom6rqObo4Tft0IF3fVy9v7WqBPrKF4AYgdAs86YJrUH7etArMfmHzAEgDnPVHQVvWt3JhXzYEHDsD5r4PquTtM3A0+QdDjJlAaistH3XPWKF9fX5g6VU/laLXCW29VsuPwC+DhZxDde7FyJUyffy5+0a0RJhNOfDjIUMxk07N9zaPzaoVT8fBvLfn7loamlZrApzDhAfDm+DqSpxSiguGrifDxDfDuBFj8UOX6pbmh9wEI2gX3nahVERsGjbvAQ0fh+vlw3z4Y8QI8mQVTk+CZLBg6TVfM1iAYOLX0VJVtL9TtOWYfff8B9Mi29Dpz92+wnDXKd+usWYR+0YyXOo1iz9rDXFMNb2OTCS64UGHyplUMe/55ws8dzQn/i7muzzw++72L94X2BpUM/fUqk2+AeT+V26R4Ht+6IswPruilF/68th9MHFT5Ve9VR/UsZukavJMCP6dXXw63hAWZsCSrgVew9gnRV8T+nh1Diy/4eqqdDH0Kns6GJ1Jh6DN628KM+0RP4gN69JpfhF6l2CcEek6sozfQcDkrbL5Zp04x99ZbceXmQlwci++9iZv++afa4/mGhTFw6lQGToX7vCem95ES5v9SP3PfehXE2UEt/bayLpLsFOfxi+C5cdXfWNtdLA3B6hy4NLB6Y115DBZm6v4dtwTDuw3F57e6mG1w+1pYNwMCGntcxwqthH3DYMoeSN4Pwc0LlPJZzFmx8nVkZRX84jSN3OTk+hWoLjh5Aq4bA5vW1s/8bne5iSyiQ0rL81BFqhCqHRUEL16mr3iry5RC5e0UYGJw9cZxSPglAzIlZEn4LLX6MjUogpvBiOeh7+TSa7KZrNCo4xmteIUQqhBioxBiXkVtzwrlG9yiBV2uuQbVYsHs68uoN8uoc34mcfPlsOiP+runDQwGW9nJVdpE6m5fZhWahsAnN1ZjjiokIL+ufzXGty+Dky3gZDTkLuT+cJgfA0+Fw7aW0Maq24Efj4d7T0JcJQPEzECMSf/xmYAODdhL0aDKTAF2VqbhWWF2EEIwduZMRrz8MmY/P8w+DSW0qBY5crCe62FVrPRfuFR/gH6NWLYHvljpvSkE+g1Pr+bwxEVVGDeP5LFAmv485XxoFMeFAVHsWQ3D/oSwSLBcBtsdug3313S40B8WZOmRbK9Hll4lQwhY1gKeTtCTr7/Q0AMwDCqFECIauBh4EXigovZnhfLNwze8UmVxS0VqGjt+/BF7ejqdr74ai5aG/PIiSNkP/e5FjPwfTqeuRCwNwW1pymPw7EN1nMOyEJV0IcrI1X1vAd69BkL94Z1F3nFN7hMLq56ouB3A8pTNfGDagnTBuMShjG8TDRTeUZOQPZM/Dj7Ffd/pR06mopcv8vy/j7ngy1Q9TfD0FOjjA+ODSpnMuZ3m2hG+aDIYhC9kPAuZ/wMCIew3sFRnmW5QB4QLIQpHgU2XUk4v9Ppt4BH0cq4VclYp35rw+113senLL8HtZuWbb3LLlBiUY1twu8C85BV+3HszNz3QBk2Dt9+Gu+6qR2GPH4P9u+t25auqup03j959K9XtgrdgvScP7k8bYOVjesrGAwlwz7dVsZqUzIZ2Q3+4+B3YcBiu7QuvX116T7fbydv+W3Ga9fv/uSxg6P7PiPQtOfnsYgGY0fGHiY/Rd8taqinsd+vJHdxAQt7HId0gU9BEMKdynqZR2hsITAjcgD+Q5xedBCk3QOTeyr5pg7olsawINyHEaOCUlHK9EGJIZQYzlG8l2TZ7Nm5PUt/EHTtY9vVR1q/S9Vv7dhp3HIrOTz358JRMLGs/YOWGQNpNmMiDD5vL2vT3Pnt3w8COdW9yKKx4m0TDax9Vqtuag3rhSYB1B/Vb8gs9Xnsn0vSqEVYzPDACVu6HJXtKWxVrKBY3msOEYnbjH2Tnf6P8+HEDLPYkSn9jISRkwheleDhpuclotkLbHxY4nppLZJE4Ahv4TuG28+CLFfqRUJ9EYpsf4qTWBA2BS0vjJstiAsQplmnjuCawBdjXQsqFINPJVUMJd59CQQIOz+WieL7SOq3mZuA9BgBjhRAXATYgUAjxtZTyurI6nBUbbt5ALZYFfe3qXJxOXefs2g2NLXs8ZyTXu87jyOdTidlyJ1mPWXlx0L3UWQ6NLz+uP1uv1Qqh4bBgrV5qqBIM76BHmflaYGj7oudeuBRyPoRHL4AX58Pi3WWZIwSNexyj3djt9Lh5LWPu3MrkYbC1mB//TxtKl8Hs24gr/l2O6nZhdjrpOm8/XSK35r0psF0PjfaAGsiA1vDZzdCj2WGeumIacbIpLsxomLjG8i0f+17D6z73s9Y/lrDEAZDcB2QS4MTHHY9akaHadkXFH5pBg0NK+ZiUMlpK2QI9b/ni8hQvGMq3UjgyMwmMKVrryq9xgWOmFCb+75NQoqOhaUQ2jdmIikQAChJt5Xvs/u039pPCWo5jr8o2fVXpWE/BHhYrPPkSrNgJkZV34v31bnjvGv0x795ShjXB+4srHsdtN9Fu7A6a9ozjwua6K9OU4UXb9CirXJkQXDPgTd5ea+KpFaE8eMkUTNb2gBlsV0Lgm+BYDg7d5nBJr838e2snbmnzOZuCetBB2YY/Gdxo/QJFgCL0/z1yRcmpPH8L6l6LIqaVxMxZFb9ZgzMCw+xQCeZcfTWntm3Lfy1UFd/wcPwiIsg8cZyh0x6j+6XR/OH4ju0//sgOT0HGwhbIReajLJV/o6DQSPjxFiMx1ca1b/xNsGs7fPCG98cuTmxreO4NuH8ipCTDtKnwxvPw0ntwReVypFrNcMvA8tu43MVXi8VdCASPXig5RgitlSCuSrJAYC5PjLbhZ4X3/obuMfDtpHImMdto1u8q8tfrEZsg41XIfBZyvy40ZwBS+uNDFoqAHGnlPZ97SCCSGOVw/rq2rDiOggoqnja2iZAzM/98mEjgl+0buaRTj3KENWjISCmXAEsqandWpZSsDlJKXg4KwpGRUeLcsCdiOHfSMZCwe+m5/HzHBlweu3DhT1UAe3a+Q2Z7z4ow043j2kZcH/QbiSvmEd2/Pxe9/z62oNK2xqvBkUPQK9Y7Y1WE1Qb2YnZKixV2J4B/pTZ9KyT24VMcSsnzx5L4WzLIdBSEln17G4zvC2Qlwke9ITsRrAFwx7rql6Zx7YaELuhFUYuiSdAQmIQkS/qw192KbqYDaNhRcZdTCMkfyCx6yDwY6ViaHwMkJXSdtZMfrm9fL1GApxOne0rJGi29hBBXCiG2CyE0IUTvYuceE0LsE0LsFkKcXzMx648Z/fqVqngB+t58FNUsSUhtRKPu2/MVLxRb3QC2hRtpdf6LdGoxmUbv/EqfeaPY89U7pOzfz9avv2Zmv341lvX777fTps17DLviNxyyjixKxRUvVBjdVlVGd1hD4ctZ96g9XNoDRneFdU96FC/Atu8gKx6cWboC3lC9PLEAyBzKqqisCP2Hs9LZhyPuGLqp2xBko6BvOhYuXA+A2gOCZ1NyM00B5+r8L4mUsCO7PYcOt2fPSc8xJPFyHUflP7hKbM4ZnM7U1OywDbgMKJIQVwjREd3o3AloAiwSQrSVskSNkgbN7Msv58Sasuug2TPh8dde4/3P7gYko5Xb6KKVbrOLmfI5KALhljR9cnaJlVHSrl04c3KqHQBy8GAKV1+t2zv2AT0DHmSb9bVqjVUmNh/co68gZ/ZX+CqlBxAAernn/Xugd80vKKBnHnO7PmbhvmFc2eknnh1/I5bSosL8GoHwuJWoFr02WXUxdUP/eZT+lRVI+pn070Z+5Dq6UhboqWvfXnkfyw8P5OaenzOm08ue8fLs/f6gxoB7p95ewvHMxgx8byuBNhjUVm+1LecG9liTEYC/24+Rpu8QxlbNGUGN/otSyp1Syt2lnBoHfCeltEspD6Lrgz41mauuceXmsuvnn8tt89l1Abwz817sDht2hw9/aa+W3Viih0GVdVooTLr7CXZVc3GzevWxIq+PZ5bRsCbk5sAPX+GjVJCFNzMDRg/Sbc9eQPiM5oOberH32eX877rrsFjLyELT6UrocxeEttIrJPS8pQaTCihnpSlEyQQ9hesvf7j2TvYktuXabrP4ZedYNhyPRcMPiR/gC/5Twb2zyHhh/qHMujWTvU+8TzBfgebgkCUet2LCpZhIV7PJkYnVf08GDYra2nBrCqwq9PqY51gJhBCTgEkAzZpVzj2pLlDMZsw+PjjzSl+UQtK2TM7nHlbwMBk0wTffWb4kJW5Fi+GyWFjW/wK+2g/3hcJrkVXLvnXRRW0wmxWcTjfDzQf4I7B2ds1VAKGv1KQsR0a3CwZ11ksKxTSv+cSWc/RHeQgB57+qP2qKLOqRIiVkO6z4WuxF7LPJOcE43VZ8zFkE2TLzL0qaVHj9wofwt2RzQZs/OZjShlxnNmbFgapoKJlPlpjSJhK5qFGIbp3IBdLuJjS4LSdsFjQUzJoLqxJc8/dmkE9abghzd1xVydZlROlUkwqVrxBiEVCa6f8JKeWvNRXAE543HfQNt5qO5y0OHFTZO2IzKctm0TX1BdRS3MMUJL2ZTi9m4sSXbMovI1TeDvh/Nz/EwT5DkcBHKTDUDy6uwn5VYKCNQ4em8HrPSTzv/BGzqB1f3zyFm6+AqGAV/OYL8NYMr8vx5JfJvPFjKuF+km0zWxLkW7OKGNvi4Jt/V9M1cg1x2SOISwvijSEUUbSFFW8eoT6p+vlCn4JbKoxt/xs2k27jNatOOjXaUomLafH6aen0Td3ILv/W5CoW2ueCGlqFVG4GDZoKla+UckQ1xo0DCntVRnuOnRZkZkLvni7SMmIx8TBxRDCayaW21TfVNKxkYi2+k10Gpbkj5QYEFzmSUtjU6HbrNtSISAgplNewGE2aBPLmdxPhytlFjmtSoAjvXNeKK5AK9UnHrl6ZtzCLNmTx4iI/CA/hmMtJzJU7Sf+9Y7XHszvhie8W880VY1CFCyHggT/ewOFSsZjd+Ql6CpN3Ecpf/RfzGmoefAS3JshxW5FCYBGu/H6UMl5ZmHDTOTPPsqfqG3QW79jSDeqX2rLczwXGCyGsQohYoA1Q9s5VA+PgATe5WQ5AxYUvhxns1fHzvCAKmyH6ffUWfikJmOw5RKXGc0mAZOPbb/Fh82ZkNLYiB3TUqxD/MrvsgQGyMov8sk+46zFlVvfecOvdXh921u+eGj5CgNlChqlmLnop2TA89jf8LNnYzA4sqoM3L3yQH3eOB1m6lixxESryWiNHsyKERAqBSehZ2Gvu1ekG4dmQlfWUMMnAa9TU1exSIcQxoD/wuxDiLwAp5Xbge2AH8Ccw+XTydFhwSRd8tBOYycRMJl35qlbmKeyOFnwyjvtHNGPKqJZcMyCKbS3C6fLCA0zKOoq/dHsipiRMvafM8bJX/Iv75suLhDLnSBuiEukda4KUkCOVoqG/jZvAwrW1Uo/9gasiwJ4NTjs4HTTK2F+j8SIDId4+lBynNX8lazU5mNBlFnjuGErbYMs7XhxVgE3YUQT4KrlYPCagvDGq/5EE4Da1IjVpBPKkFRK6g5ZW3cEM6pmaejv87IlntkopI6WU5xc696KUspWUsp2U8o+ai1o3LHnuOTIO7uR2enER9zCBMQzkFa/PU9rmm+pyEZB4EkVK2mclYxL6D7lwH4q5oiW74ZIjcMHS/aiXDUPRZBFTQKx6pMAVStZObnUh9AvIayHDSLX46kEWvSqX1aw6dGkXxPfXpdL06AoG5f7H0T8H1Gg8IeDZK8eyz/4pEnORzcTq6slauOaQoLRmopzDxNCbeTDsReyu/ZD9ccUdDRokhsNgIZzZ2Sx97jkE4EMaPficliypleLmhQMwSuO9FFjpcbRwSo/itfmwpt8ofp04kfQ43YQ++QTMz4QO/8zVxy20QeSWRV+XZrv0Bm5V4fmfn2DVllu4dedHHHzmcfi/2rlbyOPKMbEcWzyUZbOHYrHUPGWc2QRdWk9AUSw1XJ3WHnP9upAmbLiEmThTE1bZuoFjG7hOeIJCDE4nDOVbiBMbykh7VQ9owKJsWJ4Nay+/hWNzV/BpTHf++Ggmmz79lLeio/m/Tp2w/zoHJ5DuG4DmKc2tmwHUfOf9PCpl9zFXraaNJmFT/07s694KzceM09/MdxOHgl8Fdbp2boOvZsK+0tzE6wHHf5A0CmTZroV5SKlvYiZrwSRqIXUgnM6I7L+xyrwoOYGfzAb7V5DQBE76Qe7vdSaLQc0xEusUIqRlS4SqIl2uil2o6gAN+DsbmPEppq+/LRK+DHpe4W4PXE+H65dz97yP2ZDtxiR8+cQ+jOGWA1xt3QHoysIu4Zs0uD4IzOVdcp1Vi/Kwo/DJqe7kKroLlMmlEWSpIEpvwxq4ZKj+XAj4azW071Sleb2KlgjJHsVbUQ4fQEMhVQYSKDJQK3dJ8wrN3HFMTPuSrwMncF7Ov/Sybyx0VkLK1RBVG9E1BrWBoXwLEdCkCYHR0aQdOlTvirc4xRVvHnaH5PLZH/JllhOnBIVsdpkVkG0ZI/dgw4UL+CgVcjU46YKY4mWOAoJISrVzU8Y49mrhrA6eSZBSOSVsBobt34l8OZU/br+A5ocTuaH3GChvAb3wd8jxrDDNFv11fSpf9/FCPmCeY+XYxhU0wpTU2paqBAIYmbuEkblLymhx2uxpG2CYHUpQkzpvtY0oltBdKiYSCeeE5o+G/tNzAlf1TOX8z19EvPsJh53wRSqkafpKOqgU86hMT+O2rDHMc3VgtxbOXZnlV5vM25ByA++4BtLafowrP/qNT3tM5onxr+C/e1/pHY8ehiULoWUb8PGUiXA64NVnYd2q0vvUBaaOYPIkU6goJyRF7cGyljYxq0XAy/UtgUEVMJRvIZzZ2cRv2lTfYpSJ9GQK63DllYyZOZPtzd/hS+7l4ZxLcHmymEVYzQzs2ZbLL++IzzU30Dwmiq5W6GmFG4MgsBTlKwR85J9nLxQ87bu06LzoiluiK5q59ja0TbmbRzJHsr7JuSx3RpMtTXqqRaer9HDit/4HPVvAlaPgruuLZkPLzYEL+8N5XeC/pSX71jbCBOGrIeDNokq3HAVcOFii/jfnVPB/E/yn1LcgZy1CCJsQYo0QYrMn0+NzFfUxzA6FcOXmornKrzJRiYVRrSKBlb/u4Nb5s2iUtRAXAzkgjzKHvvwQcjetFSfi+5mwbwssWI3y3Xz6juhdtMZaobHy3keoyOZS8w4WOlsRo6SXaJO3eScEjLQe5EhmEO/k9uPN5i6e2TWMZM2XdmoiFwYmYfHzLyn4G88XfV1aqaOd2+Cq82HdfoiqZh7e6iKsYGpX+eb1rnALEfgJ+IyvbynOduzAMCllphDCDCwXQvwhpSzzls5Y+RbCJzSUoOYVJ4GphFmw1hCA25FDRpaZI5xHf96lC8u41fIBbVQXSt5KbOMaXVt27g6bjsLsP6Fj13z/4hzVTKJfcP57MAn4MfB7UsNewke4Cm6lPX8LJ9ExoWETbsxoXLrqfXYFv8/5lv2Mse4l0JkOq/4tKXhpCrk0HHY9GXx9kDOnfuYtk0rmcUi/CeIDwHWoNoUxKAepk7fbafY8ylURhvItxtU//VTu+cJhwfWx+NGAJTzpkUUSzRrGcBf+yo6iDQvfDzeO0nPr7tiSv4pNCQoj8rfEEl1UUXKYwn81CdNzepIlzQSLXBqrWUSo2fQ1x+ErnLoJIaKUPLo/LASLhWI6vXS69Sr/Q6gNnDsh95u6n7dcqhJC7ISMx2pNEgMAwoUQ6wo9ihSmEkKoQohN6BmSFkopV5c3mKF8ixHVsydN+hZEZ6llJDevr7tOAXTkFwQuollFG/4AITia6+I/u8fxyWSGT4qt4pSixt40/2DMLifr2vQosWEkSokAKewM0MGUxIM+K1kTPKNk9jRFgQ6dSwrepTs5l91Y8ecmFLDZKmrlfcQZ8FNQapA83qAyJEopexd6TC98UkrpllJ2R08k1kcIUcoPoYAz4BvnXU5s3MipLVvyl3ruUly86tPcJ4D2zOVRgmlkO8bcVkfZbx4HwN8Zbj5p3gtOOGDM5UU7+vvDtDdACOKVAK497xkcX6zh/p192RvYqtR5StvJFwKGWw7yit8iotV0fSOucLtzzi1T9mXn3EqW6ke2KN0PTQLcO7UyH4P3MbUDv0fRMxarYLkAgv+pH1kqhQrhp0BtpT9Xu0KAlyuXGFQLKWUq8A9wQXntDOVbjCPLlzcw/6GSCMBCDufkvsiO/YF855hFGtGYfHxoMWRI2R3vfABOafzxylJ2Tz+AePcf9jga0TrjQImm7uatuDF9LA7PVyQvfaKrUJYvcc0tKHMWIP7dirh9il46/vu/ypw+YnAfelywgwc7v1HE7CAlZGhmvg0eCVMr3CSuPQKnQZRLf4T9ATK1/mSpEDekXQERe3V5G20Gxcj1W18IISKEEMGe5z7ASGBXeX0Mb4ditBg8GKEoKGYzSJnv/WAJCiK8QweOr6pHf9RCKGgEcJKmrCHBdxDNxjxA/3NVzrnrrgr73nRTD4KCbGzadJLxoYdRXiikCq1WOJaLIiVfKc8xxrGXiyz7EEjuyriIP51t2R7yf4QpObopY8hIvd+Lb1c4b8/m8NCEMO55/TwmqS3p4DyEWUjmOVpzSca1mNIVrjE1oK+kY3F9S1A+zmWQ9a7hYtYwiAK+EEKo6Iva76WU88rr0IC+6Q2DyK5duXn5cg4sWsSOH37guKeApiMtrcEo3jxM5DCcqaBEss0ZytfKOC5MNvF8JVL4XnppBy69tIOeOf7V28Gh55xltG6uEELw+OMDufp/GgNNR9inhXFCCyQAO4udsVxp3VG6bbcCLuslmLL0Z/o7JzDEfIggk4t5ogu+voKBA2MqHqAu0U6D/P8Zj4LfPWeGzfo0Rkq5BehRlT6G8i2FqB49iOrRgw0zZ9a3KOUi0IhhNTITxE8QOf97fvCdy9gJwzmnskWQ/f1hSxx89Ba0bg9XX09urourr/6Bf/45yJzAHznkCuTp7GEAuBG0VxMhLAKuubnKMoeH+/LZl5cx9aG/SA2P5N3ZExi1/AgA113n/aoXNcLvdrD/gR7L56hS16pWrKgQ//fBFASZM8FVOBDFgf4FCPTSRAZ1haF8y+GcyZP5a0rDvaUTxf6qTgdNtq8lWxtetYHCwuGJF/NfzpixngUL9pOb62Rc2E4UM7hR+M/VjGGv3kGX8ydAu07V1izjx3dm/PiCVXPbdg00pNs6CsLXg2snZH8KjspnDcuzkRfeOqi+IhaQ6fkeWoaCOgC3ewUbAruQZAknlt9px4TqDm5QT9RI+QohXgPGoC8L9gM3e3b6EEI8BkxEXzbcK6UseyemgdL9xhv55+mncaQ17GoBEnBZbUhFJd1l44aAW3H5RzLzuxu58KK2VR4vK8uJpmmAYI87jDZqMvf5rOIC936G/TueKVO8W1apQWPuoD8sI+BU1coV5SngmiPJT5rjXAVSZXVQD475NAUh2Cy/w0IksQzzxmRnF/HAm/UzdU0NRQuBzlLKrsAe4DEAIURHYDzQCd3d4gOPIfq0IXn/ft6KiWnwihf0la9qz0VKjf3vHuSIjOZ4hoWLL/6WhISsKo83aVIvYmNDMJkEl6l3MNPVmxm5PRmZcQMZ2Wdp7TA1EHyrfhdU89JBRXFLyf3ZT5NoCSsUAQMnaTi5qA0qR41WvlLKBYVergKu8DwfB3wnpbQDB4UQ+4A+wMqazFeXbPz0UxwZGfUtRqVRAGtuDmkEkmeIkMDq1XGMHl211W9oqA87d04mM9OBr6+ZVmO7cPjPXRDqxzPTzuLVVdDbIEIgaxp6rGFd48sn9jt4134vwzL/gqCCTCMtxFn8fzlN8abN9xYgr7RuU3RlnMcxz7ESeEL0JgE0a9bMi+LUjOBK5HhoaAyYOpX2755kR7b+UZtMCv36RVdrLCEEAQF6MMTBeVexMc1FiI9KrLUhZZSpB/zvhuz3QCbV/dyB7/HS8VvQgHHH/+DKzO+5r8kxWikX0Iiqe54Y1C8Vmh2EEIuEENtKeYwr1OYJwAXMqqoAUsrpeeF6ERERVe1ea/SYOJFmgwbVtxhV4uA//7A1YzovPD+UG27oxubNdxAe7luzQaVEfPQkAVMGs/3Zl0lNza24z5mMEgaNDgKhdT+3uRNfNYVwFXyEwii/8fRXHjIU72lKhStfKeWI8s4LIW4CRgPDZUHN8jigsNNmtOfYaYOiqty8bBm/3XknGz76qL7FqRQ5iYkoiuCJJ724ITbjGZxHXqJJlEJTuYbx5x1n9uo38fEpI5rKsQ5Sr9ZL8gR+BD7jSm93OqMEQFQS5C6CzPfAObdu5k06j4E4SAg/H0L/aGB5LQ2qSo023IQQFwCPAGOlLFJ5cC4wXghhFULEAm2ANTWZq74Y8+GHPJqezl3bt/Ok00njHgV+1CabjW633NJgfgQXf/ih9wfd+xdSgJ/Fhc3son/EVrZvTyi7ferV4D4A2klIHQ+yiht0m9eTdvXlHJ88BZnRwOuR2UaA6sVkNhV6Rnh8jR1/gf03781rUC/U1NvhfSAAWCiE2CSE+AhASrkd+B7YAfwJTJZSnrYFpmwBAUR07IhqMtF88GBMnkxnQtUrBDeEPBChbdrQauRI7w885IZ8pZDjNLM2vgUtW5ZTsVcW9q5wo1ujKklGOr/e+CKzDkTi+O0X5o+4tdSc6w0K25iaj5GXo7QqaBVXWTZo2NTU26F1OedeBF4s6/zpyoiXX8bi50fizp30u/9+Di1dqlc8LqVSRF0hTCY6XXVV7Qx+8WQUHysbfpvF6tw+PPftA4SGFg2fKxLNFfgRpE4A3BDwIoiyQ+3suFjOUcyoDCCaJz9P4ZVu3yKFwLfdiyxdNYJNR/WcEA0WnzHgfh8ynkD3Mkmt+hhVzc4vmoDPlVWfx6BBYUS4VRGT1cqwF17If92kd28O/v03h/6p2/SDUb17E9m1K9mJiXS8/HK6Xn+9V8f/9K0U3ng+m66hR5gxbxQ937qVnqW0W5QJD6/cyzW/zaRf2+YMumMSjtBkVFVDNfmVPnhuLpjNPK0u5aBHWa3lOB+sbom06R4W2ZrGo80eY0YF+4Ur98N1M8CpweC2EBEAD50PTYKr/darjv9k/QHg3AuJHahWJeHyrFemHiBTwDYBAv9XHSkNGhiG8q0hJpuNGxcv5uDixXw3bhyOTO/ZKU2UcdMuBAIYO2MGQvF+QpVt2+DOBxNwyBXsSvHD2m87n6feWmrbe/ak8d+kPgRlpZFr8eHbOZu4NvUKbAc28PMPV3D++UVvjk7dfTehsz8kw8eX3Yc/IS9D5VqOE5JtI10NBk9ms5aXDKNlOQ4wOQ4Y/Co4PXrua49z43drIO71ejLFm1pRLcVbZmkUfwidC9ahNZPLoMFhpELyErHDhiG9ZKDMCwUUwN3BpfwmpeTU9u2kHzvmlfmKs3dvDg45BziMxg5+SktCEc9w222/IaVk7ZqjLFlyCLdbo8XxA5g0N6qUHMvyYdvGI8jwaHK6juCWW4p5ARw/RuDsjzGhEZyTSdi+eFQEZhTaEca855sSvGsxYutyJgU/y4w7dpcrZ3w6uErRcyfSIMvuvc+jSggF1NKCWiq4EhQrR5/l8CHJfT1EHjUU7xmKoXy9SQ2XWmpTK+2Oj6arn0obM1wfBKE2M6pa7N8kBGYfH/waVSJ3JOCWeu21ytKmTTaFI7gyyEGi8MXnG5l3cx96/daMjOkXccVl33J1/4Nk+fix1dWIXqm380pGH1g8BxSFU6eykIU3I82WfAOxlHDdBf/HFbI94+nEYwygXdtQbuoxmwHiCwa3/x2SR4EsWUkkj2ah0Dys9HP+9VCJKJ+ITWCbAoQAwWC5CNTK+eK6NdgW34Gxs+by4r/vgBJce3Ia1CuG8vUiF73/fpUVsFBVfJqFEXhdNG0OjMQUZaXHX9O4pnkoMYG+iDc+5oIPPyI4NpbWF15I3/vvp/tNN3HLihWYKlHr7IGFxzC1+xA14CUGXf9rpWTq2DGUqChfCnaA9PfkdGkkpzhQBAyJ2Yd6cB7xh37gv8HRPJw1khxMuPPW7ct/w+XSyMpy8MILy2jc+HWGXDmfR8xXstsVylpnE9YOe4hrRBeCFlpoGvYm/v4v8f4n0Sxf04zbHhrL+s2hoKWWKaeiwMwbSx5vV7lrUu0hfCDkbYhKhqgUCPsdAp6h4J6mePvGgIW03ADm7R7L8M+XsOroubSLCq47mQ1qhBAiRgjxjxBihxBiuxCiwkQghs3Xi3S/6SaS9u5l5Ztv4s6tXCSY2ceHsd/OZMe53wECgULogNtg7+P5bXoBvW67rcryuCW89doK2J8AmmT5D9tYc08v+vQpP+RYUQSffjqOsWO/xeksakrZcDKKG7ttAQR2u8ozg1txjtNGSyUZrci2vUBRBC+//B9vvbWK7GwniYnZLHW35z3a6/P8mMuFz8CECT+SnJy3wjV5ZJCs3tyHniMjy71hn/RlyWM7Xih5rN7xuRzEH5ByKVAs2ZGlB4T8TlDagyRuz6FDxCGGd27HbefVMDrRoC5xAQ9KKTcIIQKA9UKIhVLKHWV1MJSvlxn63HOoFgsnNmwgNzmZuHXrkG43/o0bk370KACKxYJiMiGEILJbN9r2GU0LhpDGIYJpiRXvJMYWAHZnIT9kQVZWGUEPUhZZtZ9/fiueeWYwb7yxkpQU/UJisygMCLXjXKuwIiea3/e0QqKwnOY0s6Ryk2UD+7QwVrlicKFiNiukp9tRFE8xUndR24eWksQ54xbiSC++raiRnWNm8qOdePx/r7Jnz900auRfqthHkkseq4U9SO9gGwmN0yH5Ak+JIjeIGAh8R//sg99k4miYOLq+BTWoKlLKE8AJz/MMIcRO9Hw2hvKtKxSTiSHPPAOAlJKUAwewBQdjCw5m3ccfk370KL3vuAOLvz9Z8fGEtWuHoqrYCMZGd+/KIuCW67ry6dbjkJpDy66NGTKkRdFGibvh8xGQeRKa9oW0wxDVA3HFLKZOHci86f/QNHMbB90hXGvawuWbNyEdkoEcZaT5AAucrTGbFVZFD2ZN6mOoSAan3cQ2dyNatghl2rSh7N6dyD//HCIgwFpohatj37oeBl0CS38GtwtfXxN+fmYSEvR2aWl27r77D77/vqRf64f/SFwaFN6tWvukVz9C7yMUCFtQcTuDhki4EGJdodfTi5ePBxBCtEAvKbS6vMEM5VuLCCEIbVVQlr1PseKWvmFl7BZ5kU9u68K717QjO9tJREQpfre/3wvpcYDkf98KZm68nM4Rp/hIPsnUBUNYe8TNKjoB8LhYhuZwYhYSE05GeJSv1WpiwoQuyDmNCEg6zOaQj0jCj5BtaSgmlT//vI6kpByCgqy88spynnpqScH8foEQ2xGiW8P37zJ0aBP8/Mx8/33BgkErxYtk9epj3PVZhL6Jp4cZ0qWpoHcLb356BgZFSJRS9i6vgRDCH/gRuE9KmV5e24Z6g2bgRfz8LKUrXtBXYgIWHojlxeXncTA1hN/2tiXmhmDmzNnh2UATgOCB7AtwSgWHVMiSJv5y6heWzEwHb7+9is0PfoSzRz+yWnUheN5fKCZ9g0kIQXi4L2azytGj6ZhMnq+drz+MmgBOO6QlQW4WwcEWOneOwGpVPbKbue++/vTv/wmdO3/AkiWHAJh0x++wez1sXg452SDh/kEVJ44/eeIQm1ZPJzWpzLtBA4NqIYQwoyveWVLKnypqb6x8z3Yufpcplz7JB/+1Rc13LxNoUpCb66Kw9/8RGUyz5Pu5xWcjKxwx/OcuiPt1ONzM3S4ZvnAl6el2+gz9gs2b/2bo0Fh+//0aLBYVp9PNzJkb0Tx+b5bcdBz/zQOrDxzWfXpnzdqeP6avr4no6ACGDPk83148cuSXPPXUeWzZcgrEP7qtev9WGHsr2Xt2wIhzynyrO7ZtI4J+tAx1oaZpJNjnE9Gk3KR9BgaVQgghgE+AnVLKShUmMpTvWc72k0HMXN8Nl+bElZ/hpbB/QWF3M0ESvryWMzD/rKqCpoHTqfHRR+ux2UxkZzvZsOEEAIsWHeCjj9YxefI5fP/9Nvz8zGRmOhACosNU2pz8g7/cbYvNqZOd7WL37qI7ai6XZNq0ZR7HZc/FIi0RTh6hQ4fy80FvXvM1owc5CfDXs4MdOfyeoXwNvMUA4HpgqxBik+fY41LK+WV1MJTvWU7O8cMMsC8kkUA20oOSSrB41pei51VVRVE0nE6Jw+HmtddW0LVr0TSLmzadpGvXD9mxIxHQTQkdOoSzbt0JDpAXDVZmfG0JintNYPOD/ZvZ6Li83BKSfkHdUFS9b1a2GcVaWrYKA4OqI6VcTmW/wB6EbADpEPPo3bu3XLduXcUNDbxCbmoq78TGkpOahhMTP3IZuz0+uDolv0tCQMeOEezYkYCUYLWq2O3l5zJQFEqkhvT1NZPtrWKc4U2gzyjCdy2iQ1OFr766lObNg0s0k1Ly87evEBk4B7PfefQZ8oZu8zY4LRFCrK9oA6ziMXpLqKzOqfl8hTG+eWcxSXv3IjUNgcSCkzSCyDMvlHURlxJ27UpEVQUhITaef77ivAOlpbzwmuIFaNoS5n9O4oFj/PvvETp3/qDUZkIILrtmKgNGr6fP0LcMxWtQrxjfvrOYiA4dMPn44MKEAzNaJb8ObrfE5ZJYLIL33y/XlbFuOFw0AU9mppPHH1+Ew3Ha5u83OAuoaRmh54UQWzxVLBYIIZp4jgshxLtCiH2e84ZxrQFi8ffnjk2b6HDf0/zAFZwiggLbbkXmKEl8fDZHjpTrylj7CAVSS5Y1euml/7j88tmldDAwaBjUdOX7mpSyq5SyOzAPeNpz/EL0um1t0MvC10JxMQNv4N+4Mfauozlgag/5Pr1Q8d5BZdvVMlIjKMha6qmFCw/UsTAGBpWnRsq3WASHHwXLpXHAl1JnFRAshIiqyVwGtcPevUlMmjQPl6s6G69lK966TGQ+eXKf/PwRhXG7JceP1/PK3MCgDGps8xVCvCiEOApcS8HKtylwtFCzY55jpfWfJIRYJ4RYl5BQTlVcg1ph375kVLWqmrK4oi6puOvSiWbZskMIUXJCl0ujT58Z2O1VKOJpYFBHVKh8hRCLhBDbSnmMA5BSPiGljAFmAXdXVQAp5XQpZW8pZe+IiPKd5A28z8CBzYiMLD1jWNkUD8KoX9PD8uVHKat+aVxcJoGBL7FqVe1U/TAwqC4VKl8p5QgpZedSHsUzc88CLvc8jwNiCp2L9hwzaGAEBFh5550LajBC3StetYyc5GXhcGhccYWx+WbQsKipt0ObQi/HAbs8z+cCN3i8HvoBaZ58lwYNkCuvnFPfIlSJsla55REXl8m5587wvjAGBtWkpjbflz0miC3AKCCvdMZ84ACwD5gB3FVGf4MGgMvlncKfeTTUZOYrVx6nffv36lsMAwOghrkdpJSXl3FcApNrMrZB3dGjR2M2bjzptfG8VMS5ViieqMfAoL5ooGsUg7pk/fpJXHFF+4obGhgYeA1D+RoghGDOnKsZNCgGIfQCmvfccw533eW1HCLeQQFCalZUcsWKm7wiioFBTTFSShrks3TpzWzeHI+fn5k2bcJ45ZXl9S1SAU2CYPZt4G+Fzcfgtq/AWbp9w9fXRHZ2Sd/efv2a0r9/81J6GBjUPcbK1yAfIQTduzemTZswpJQ8+eQ/NRiLagRvlMNdgyHIBywm6BgFA1qX2VSvwFGS99+/0HvyGBgUQwjxqRDilBBiW2XaG8rXoExqojylLCXpeU1IzwG3Z6UrBGTkltm0tA2/sDAbvXqVGmRpYOAtPgcq7TRvKF+DUtHtwFfi42NCVQXNmgVWWxn7+FQxKqI0PlgGqw9BYia8OB/WH6lS98ceG1RzGQwMykFKuQyotDuNYfM1KJMxY9qRnf0EAIMGfZafPtJkEthsZlwurcxb/MLk5Hghr26mHe6YVe3uo0a1qrkMBgZexFj5GlSKqVMH4OtrIiDAQufOkSQlPUJm5mO1Pm91s6MJoV8k/PzMvP32+XTpEllxJwOD8gnPSwLmeUyqyWDGytegUlx8cVu2b5/MsWPpnHNOEywWld27ExGi9AxmMTGB9O4dxc8/7y55sgqMHNmShQsPVJglzWRScLs1pIQnnxzE8897SmlqWeBcD+7joDapkSwGZz2J3qzhZihfg0rTokUwLVoE578OC/PFalXJzdXNCnmrVCnh6NF0jh6tTC7d4lWRi2ZJq4ziBdi1azKtWoUipUR4BNFcqSTvaYvFlI7ZDO6gBfiHnlcJmQwMah9D+RpUm/BwX3766WqeeGIxMTGBjBvXjokTf6viKAVKt3FEBmGhOWzfXWAiqIziDQiw0LJlCFde+T05OS7mzp2AogjWrfiCDlFpBPg7ANi4eRo9hi6qonwGZzbHgee8MpIQ4ltgCLp54hjwjJTyk7LaG8rXoEZceGEbLrxQT27311/7Sm1TlmnCbFZwu11omoLV4qJ39zh27a16TudzzmmCokzLf62q05DyGVIzmqBG6xNn5Zg4kRBDjyqPbmBQOaSUE6rS3thwM/Aao0a1YvjwFkWONW7sV2aWM6dTQ9MEAg27w8S8hR3Ydyis3Dl69y5ZjcrPz1Li2LZt8QwZeRlvfz6RZauaMevnIfQ6781KvxcDg9rGUL4GXkMIwaJFNzJz5hgaN/and+8mrFlzG8HBPuX1QqJQ2YKcr702En9/c77PcVCQleefH1qiXUxMIBaLyuPTPqT/6ANMenAhkY1DqvfGDAxqAcPsYOB1Jk7sycSJPfNf33xzd15/fWWNx+3VK4ohQ2LZs+cetm8/RfPm+gag2axy3nnNWLZMD7zo3DmCoKAChW82eyHIw8DAyxjK16DWefXVkSQn5zBnzg4yMhzVGiM6OpB163S3yqioAKKiAoqcX7r05hrLaWBQlxhmB4NaRwjBJ5+MIz39MZz/3969hlhVhWEc/z8etQQpS+lCaipZMJaQDCb4RUrQTNQgUokyCqIwKBAqk6APEd3oRiVoBgrhhS4oUpiZSF+stJtZVlLaBc2EzDAoJ58+7CWe1JnpzDkz++w97w/Es/beZ896Ofi6Z5211nv0IRYsmFDzPZYund4NPQshPw1JvpIWSLKkIaktSc9L2i3pc0njOrtH6B369u3DU09NYdKk2rZ2nDKl/V3MQiiiupOvpGFk9duqdzq5Fhid/twBLK7354RysM3hw3+xYsX1DBjw/0a9sk3e8y1PH0KjNeLJ9xngPk4sVYKskvEKZ7YCgySdOkco9Cq//nqESy99gcGDn2DWrNWcc86JL8UqFTFv3lguvvjs/7ynpWUI69ff1NNdDaHb1fWFm6SZwM+2PzvpyeQi4Meq9k/p2Cnl49PmFHcADB8+vJ7uhCa3ePE29u49RFvbMXbtOsjs2WNYufILJJg793KWLZuZdxdD6DGdJl9J7wIXnObUIuBBsiGHLrO9BFgC0Nra2sDdt0OzGTiwP5VKH44ePYYEkyeP4tFHr+HIkb8ZNSrm4IbepdPka3vy6Y5LugIYCRx/6h0KfCxpPPAzMKzq8qHpWOjF7rqrlS1b9vD++z8wY8ZlzJ49hkolJtyE3qnLww62dwDnHW9L2gO02j4oaR1wt6RVwFXA77ZPGXIIvcuAAf1Yu7am5e8hlFZ3LbJ4C5gG7Ab+BGIGfAghVGlY8rU9ouq1gfmNuncIIZRNDLiFEEIOIvmGEEIOIvmGEEIDSJoq6eu0rcIDnV0fyTeEEOokqQK8SLa1QgswV1JLR++J5BtCCPUbD+y2/Z3tv4FVZNsstKup9vPdvn37QUl767zNEOBgI/rTxCLGcogY61Pb1nintW8DPDzkf158pqRtVe0laYUunH5Lhas6ullTJV/btVdPPImkbbZbG9GfZhUxlkPEmD/bU/P62THsEEII9at5S4VIviGEUL+PgNGSRkrqD8wB1nX0hqYadmiQJZ1fUngRYzlEjCVhu03S3cAGoAK8YntnR+9RthI4hBBCT4phhxBCyEEk3xBCyEHpkm+ZKylLelLSrhTHm5IGVZ1bmGL8WtKUHLtZl1qXaBaBpGGSNkv6UtJOSfek4+dK2ijp2/R34ct5SKpI+kTS+tQeKemD9HmuTl9GBUqWfHtBJeWNwOW2xwLfAAsB0jLGOcAYYCrwUlruWChdWaJZEG3AAtstwARgforrAWCT7dHAptQuunuAr6rajwPP2L4E+A24PZdeNaFSJV9KXknZ9ju221JzK9lcQshiXGX7L9vfk21iPz6PPtap5iWaRWB7n+2P0+s/yJLTRWSxLU+XLQdm5dLBBpE0FLgOeDm1BVwNvJYuKXyMjVSa5FtdSfmkU+1VUi6624C30+uyxFiWONolaQRwJfABcH5Vea39wPl59atBniV7+DmW2oOBQ1UPDKX7POtRqHm+3V1JuRl0FKPttemaRWS/yr7ak30L9ZE0EHgduNf24VR4Fsiqv0gq7LxPSdOBA7a3S5qUc3cKoVDJtzdUUm4vxuMk3QpMB67xiUnahYqxA2WJ4xSS+pEl3ldtv5EO/yLpQtv70lDYgfx6WLeJwAxJ04AzgbOA58iG+fqmp9/SfJ6NUIphB9s7bJ9ne0SqJfcTMM72frIlfrekWQ8TKHAlZUlTyX6tm2H7z6pT64A5ks6QNJLsy8UP8+hjnWpeolkEaexzGfCV7aerTq0D5qXX84C1Pd23RrG90PbQ9O9vDvCe7ZuAzcAN6bJCx9hohXry7aIyVVJ+ATgD2Jie8LfavtP2TklrgC/JhiPm2/4nx352SVeWaBbEROBmYIekT9OxB4HHgDWSbgf2Ajfm071udT+wStIjwCdk/wkFYnlxCCHkohTDDiGEUDSRfEMIIQeRfEMIIQeRfEMIIQeRfEMIIQeRfEMIIQeRfEMIIQf/AnOS2zJBfDHFAAAAAElFTkSuQmCC\n",
"text/plain": [
"<Figure size 432x288 with 2 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"model = utils.build_model(backbone, second_stage=(stage == 'second'), num_classes=num_classes, ckpt_pretrained=ckpt_pretrained).cuda()\n",
"model.use_projection_head(False)\n",
"model.eval()\n",
"\n",
"embeddings, labels = utils.compute_embeddings(loaders['valid_loader'], model, scaler)\n",
"embeddings_tsne = TSNE(n_jobs=num_workers).fit_transform(embeddings)\n",
"vis_x = embeddings_tsne[:, 0]\n",
"vis_y = embeddings_tsne[:, 1]\n",
"plt.scatter(vis_x, vis_y, c=labels, cmap=plt.cm.get_cmap(\"jet\", num_classes), marker='.')\n",
"plt.colorbar(ticks=range(num_classes))\n",
"plt.show()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"embeddings, labels = utils.compute_embeddings(loaders['train_features_loader'], model, scaler)\n",
"embeddings_tsne = TSNE(n_jobs=num_workers).fit_transform(embeddings)\n",
"vis_x = embeddings_tsne[:, 0]\n",
"vis_y = embeddings_tsne[:, 1]\n",
"plt.scatter(vis_x, vis_y, c=labels, cmap=plt.cm.get_cmap(\"jet\", num_classes), marker='.')\n",
"plt.colorbar(ticks=range(num_classes))\n",
"plt.show()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.9"
},
"pycharm": {
"stem_cell": {
"cell_type": "raw",
"metadata": {
"collapsed": false
},
"source": []
}
}
},
"nbformat": 4,
"nbformat_minor": 4
}
================================================
FILE: tools/backbones.py
================================================
import torchvision.models as models
# for timm models we don't have such files, since it provides a simple wrapper timm.create_model. Check tools.models.py
BACKBONES = {
"alexnet": models.alexnet,
"resnet18": models.resnet18,
"resnet34": models.resnet34,
"resnet50": models.resnet50,
"resnet101": models.resnet101,
"resnet152": models.resnet152,
"mobilenet_v2": models.mobilenet_v2,
"vgg11": models.vgg11,
"vgg11_bn": models.vgg11_bn,
"vgg13": models.vgg13,
"vgg13_bn": models.vgg13_bn,
"vgg16": models.vgg16,
"vgg16_bn": models.vgg16_bn,
"vgg19": models.vgg19,
"vgg19_bn": models.vgg19_bn,
"densenet121": models.densenet121,
"densenet169": models.densenet169,
"densenet161": models.densenet161,
"densenet201": models.densenet201,
"inception_v3": models.inception_v3,
"resnext50_32x4d": models.resnext50_32x4d,
"resnext101_32x8d": models.resnext101_32x8d,
"wide_resnet50": models.wide_resnet50_2,
"wide_resnet101": models.wide_resnet101_2,
}
================================================
FILE: tools/datasets.py
================================================
import torchvision
class SupConDatasetCifar10(torchvision.datasets.CIFAR10):
def __init__(self, data_dir, train, transform, second_stage):
super().__init__(root=data_dir, train=train, download=True, transform=transform)
self.second_stage = second_stage
self.transform = transform
def __getitem__(self, idx):
image, label = self.data[idx], self.targets[idx]
# leave this part unchanged. The reason for this implementation - in the first stage of training
# you have TwoCropTransform(actual transforms), so you have to call it by self.transform(img)
# on the other hard, in the second stage of training there is no wrapper, so it's a regular
# albumentation trans block, so it's called by self.transform(image=img)['image']
if self.second_stage:
image = self.transform(image=image)['image']
else:
image = self.transform(image)
return image, label
class SupConDatasetCifar100(torchvision.datasets.CIFAR100):
def __init__(self, data_dir, train, transform, second_stage):
super().__init__(root=data_dir, train=train, download=True, transform=transform)
self.second_stage = second_stage
self.transform = transform
def __getitem__(self, idx):
image, label = self.data[idx], self.targets[idx]
# leave this part unchanged. The reason for this implementation - in the first stage of training
# you have TwoCropTransform(actual transforms), so you have to call it by self.transform(img)
# on the other hard, in the second stage of training there is no wrapper, so it's a regular
# albumentation trans block, so it's called by self.transform(image=img)['image']
if self.second_stage:
image = self.transform(image=image)['image']
else:
image = self.transform(image)
return image, label
DATASETS = {'cifar10': SupConDatasetCifar10,
'cifar100': SupConDatasetCifar100}
def create_supcon_dataset(dataset_name, data_dir, train, transform, second_stage):#, csv, second_stage):
try:
return DATASETS[dataset_name](data_dir, train, transform, second_stage)#, csv, second_stage)
except KeyError:
Exception('Can\'t find such a dataset. Either use cifar10 or cifar100, or write your own one in tools.datasets')
================================================
FILE: tools/losses.py
================================================
import torch.nn as nn
import torch
class SupConLoss(nn.Module):
"""Supervised Contrastive Learning: https://arxiv.org/pdf/2004.11362.pdf.
It also supports the unsupervised contrastive loss in SimCLR"""
def __init__(self, temperature=0.07, contrast_mode='all',
base_temperature=0.07):
super(SupConLoss, self).__init__()
self.temperature = temperature
self.contrast_mode = contrast_mode
self.base_temperature = base_temperature
def forward(self, features, labels=None, mask=None):
"""Compute loss for model. If both `labels` and `mask` are None,
it degenerates to SimCLR unsupervised loss:
https://arxiv.org/pdf/2002.05709.pdf
Args:
features: hidden vector of shape [bsz, n_views, ...].
labels: ground truth of shape [bsz].
mask: contrastive mask of shape [bsz, bsz], mask_{i,j}=1 if sample j
has the same class as sample i. Can be asymmetric.
Returns:
A loss scalar.
"""
device = (torch.device('cuda')
if features.is_cuda
else torch.device('cpu'))
if len(features.shape) < 3:
raise ValueError('`features` needs to be [bsz, n_views, ...],'
'at least 3 dimensions are required')
if len(features.shape) > 3:
features = features.view(features.shape[0], features.shape[1], -1)
batch_size = features.shape[0]
if labels is not None and mask is not None:
raise ValueError('Cannot define both `labels` and `mask`')
elif labels is None and mask is None:
mask = torch.eye(batch_size, dtype=torch.float32).to(device)
elif labels is not None:
labels = labels.contiguous().view(-1, 1)
if labels.shape[0] != batch_size:
raise ValueError('Num of labels does not match num of features')
mask = torch.eq(labels, labels.T).float().to(device)
else:
mask = mask.float().to(device)
contrast_count = features.shape[1]
contrast_feature = torch.cat(torch.unbind(features, dim=1), dim=0)
if self.contrast_mode == 'one':
anchor_feature = features[:, 0]
anchor_count = 1
elif self.contrast_mode == 'all':
anchor_feature = contrast_feature
anchor_count = contrast_count
else:
raise ValueError('Unknown mode: {}'.format(self.contrast_mode))
# compute logits
anchor_dot_contrast = torch.div(
torch.matmul(anchor_feature, contrast_feature.T),
self.temperature)
# for numerical stability
logits_max, _ = torch.max(anchor_dot_contrast, dim=1, keepdim=True)
logits = anchor_dot_contrast - logits_max.detach()
# tile mask
mask = mask.repeat(anchor_count, contrast_count)
# mask-out self-contrast cases
logits_mask = torch.scatter(
torch.ones_like(mask),
1,
torch.arange(batch_size * anchor_count).view(-1, 1).to(device),
0
)
mask = mask * logits_mask
# compute log_prob
exp_logits = torch.exp(logits) * logits_mask
log_prob = logits - torch.log(exp_logits.sum(1, keepdim=True))
# compute mean of log-likelihood over positive
mean_log_prob_pos = (mask * log_prob).sum(1) / mask.sum(1)
# loss
loss = - (self.temperature / self.base_temperature) * mean_log_prob_pos
loss = loss.view(anchor_count, batch_size).mean()
return loss
class LabelSmoothingLoss(nn.Module):
def __init__(self, classes, smoothing=0, dim=-1):
super(LabelSmoothingLoss, self).__init__()
self.confidence = 1.0 - smoothing
self.smoothing = smoothing
self.cls = classes
self.dim = dim
def forward(self, pred, target):
pred = pred.log_softmax(dim=self.dim)
with torch.no_grad():
# true_dist = pred.data.clone()
true_dist = torch.zeros_like(pred)
true_dist.fill_(self.smoothing / (self.cls - 1))
true_dist.scatter_(1, target.data.unsqueeze(1), self.confidence)
return torch.mean(torch.sum(-true_dist * pred, dim=self.dim))
LOSSES = {'SupCon': SupConLoss,
'LabelSmoothing': LabelSmoothingLoss,
'CrossEntropy': nn.CrossEntropyLoss}
================================================
FILE: tools/models.py
================================================
import torch
import torch.nn as nn
import torch.nn.functional as F
import timm
from .backbones import BACKBONES
def create_encoder(backbone):
try:
if 'timm_' in backbone:
backbone = backbone.split('_')[-1]
timm.create_model(model_name=backbone, pretrained=True)
else:
model = BACKBONES[backbone](pretrained=True)
except RuntimeError or KeyError:
raise RuntimeError('Specify the correct backbone name. Either one of torchvision backbones, or a timm backbone.'
'For timm - add prefix \'timm_\'. For instance, timm_resnet18')
layers = torch.nn.Sequential(*list(model.children()))
try:
potential_last_layer = layers[-1]
while not isinstance(potential_last_layer, nn.Linear):
potential_last_layer = potential_last_layer[-1]
except TypeError:
raise TypeError('Can\'t find the linear layer of the model')
features_dim = potential_last_layer.in_features
model = torch.nn.Sequential(*list(model.children())[:-1])
return model, features_dim
class SupConModel(nn.Module):
def __init__(self, backbone='resnet50', projection_dim=128, second_stage=False, num_classes=None):
super(SupConModel, self).__init__()
self.encoder, self.features_dim = create_encoder(backbone)
self.second_stage = second_stage
self.projection_head = True
self.projection_dim = projection_dim
self.embed_dim = projection_dim
if self.second_stage:
for param in self.encoder.parameters():
param.requires_grad = False
self.classifier = nn.Linear(self.features_dim, num_classes)
else:
self.head = nn.Sequential(
nn.Linear(self.features_dim, self.features_dim),
nn.ReLU(inplace=True),
nn.Linear(self.features_dim, self.projection_dim))
def use_projection_head(self, mode):
self.projection_head = mode
if mode:
self.embed_dim = self.projection_dim
else:
self.embed_dim = self.features_dim
def forward(self, x):
if self.second_stage:
feat = self.encoder(x).squeeze()
return self.classifier(feat)
else:
feat = self.encoder(x).squeeze()
if self.projection_head:
return F.normalize(self.head(feat), dim=1)
else:
return F.normalize(feat, dim=1)
================================================
FILE: tools/optimizers.py
================================================
import torch.optim as optim
import torch_optimizer as jettify_optim
OPTIMIZERS = {
"Adam": optim.Adam,
'AdamW': optim.AdamW,
"SGD": optim.SGD,
'LookAhead': jettify_optim.Lookahead,
'Ranger': jettify_optim.Ranger,
'RAdam': jettify_optim.RAdam,
}
================================================
FILE: tools/schedulers.py
================================================
import torch.optim as optim
SCHEDULERS = {
"ReduceLROnPlateau": optim.lr_scheduler.ReduceLROnPlateau,
"CosineAnnealingLR": optim.lr_scheduler.CosineAnnealingLR,
}
================================================
FILE: tools/utils.py
================================================
import torch
import albumentations as A
from albumentations import pytorch as AT
from torch.utils.data import DataLoader
import random
import os
import numpy as np
import torch.backends.cudnn as cudnn
from sklearn.metrics import f1_score
from pytorch_metric_learning.utils.accuracy_calculator import AccuracyCalculator
from .losses import LOSSES
from .optimizers import OPTIMIZERS
from .schedulers import SCHEDULERS
from .models import SupConModel
from .datasets import create_supcon_dataset
def seed_everything(seed=42):
random.seed(seed)
os.environ["PYHTONHASHSEED"] = str(seed)
np.random.seed(seed)
torch.manual_seed(seed)
torch.cuda.manual_seed(seed)
torch.backends.cudnn.deterministic = True
def add_to_logs(logging, message):
logging.info(message)
def add_to_tensorboard_logs(writer, message, tag, index):
writer.add_scalar(tag, message, index)
class TwoCropTransform:
"""Create two crops of the same image"""
def __init__(self, crop_transform):
self.crop_transform = crop_transform
def __call__(self, x):
return [self.crop_transform(image=x), self.crop_transform(image=x)]
def build_transforms(second_stage):
if second_stage:
train_transforms = A.Compose([
#A.Flip(),
#A.Rotate(),
A.Resize(224, 224),
A.Normalize(),
AT.ToTensorV2()
])
valid_transforms = A.Compose([A.Resize(224, 224), A.Normalize(), AT.ToTensorV2()])
transforms_dict = {
"train_transforms": train_transforms,
"valid_transforms": valid_transforms,
}
else:
train_transforms = A.Compose([
A.RandomResizedCrop(height=224, width=224, scale=(0.15, 1.)),
A.Rotate(),
A.ColorJitter(0.4, 0.4, 0.4, 0.1, p=0.9),
A.ToGray(p=0.2),
A.Normalize(),
AT.ToTensorV2(),
])
valid_transforms = A.Compose([A.Resize(224, 224), A.Normalize(), AT.ToTensorV2()])
transforms_dict = {
"train_transforms": train_transforms,
'valid_transforms': valid_transforms,
}
return transforms_dict
def build_loaders(data_dir, transforms, batch_sizes, num_workers, second_stage=False):
dataset_name = data_dir.split('/')[-1]
if second_stage:
train_features_dataset = create_supcon_dataset(dataset_name, data_dir=data_dir, train=True,
transform=transforms['train_transforms'], second_stage=True)
else:
# train_features_dataset is used for evaluation -> hence, we don't need TwoCropTransform
train_features_dataset = create_supcon_dataset(dataset_name, data_dir=data_dir, train=True,
transform=transforms['valid_transforms'], second_stage=True)
train_supcon_dataset = create_supcon_dataset(dataset_name, data_dir=data_dir, train=True,
transform=TwoCropTransform(transforms['train_transforms']), second_stage=False)
valid_dataset = create_supcon_dataset(dataset_name, data_dir=data_dir, train=False,
transform=transforms['valid_transforms'], second_stage=True)
if not second_stage:
train_supcon_loader = DataLoader(
train_supcon_dataset, batch_size=batch_sizes['train_batch_size'], shuffle=True,
num_workers=num_workers, pin_memory=True)
train_features_loader = DataLoader(
train_features_dataset, batch_size=batch_sizes['train_batch_size'], shuffle=True,
num_workers=num_workers, pin_memory=True, drop_last=True)
valid_loader = DataLoader(
valid_dataset, batch_size=batch_sizes['valid_batch_size'], shuffle=True,
num_workers=num_workers, pin_memory=True, drop_last=True)
if second_stage:
return {'train_features_loader': train_features_loader, 'valid_loader': valid_loader}
return {'train_supcon_loader': train_supcon_loader, 'train_features_loader': train_features_loader, 'valid_loader': valid_loader}
def build_model(backbone, second_stage=False, num_classes=None, ckpt_pretrained=None):
model = SupConModel(backbone=backbone, second_stage=second_stage, num_classes=num_classes)
if ckpt_pretrained:
model.load_state_dict(torch.load(ckpt_pretrained)['model_state_dict'], strict=False)
return model
def build_optim(model, optimizer_params, scheduler_params, loss_params):
if 'params' in loss_params:
criterion = LOSSES[loss_params['name']](**loss_params['params'])
else:
criterion = LOSSES[loss_params['name']]()
optimizer = OPTIMIZERS[optimizer_params["name"]](model.parameters(), **optimizer_params["params"])
if scheduler_params:
scheduler = SCHEDULERS[scheduler_params["name"]](optimizer, **scheduler_params["params"])
else:
scheduler = None
return {"criterion": criterion, "optimizer": optimizer, "scheduler": scheduler}
def compute_embeddings(loader, model, scaler):
# note that it's okay to do len(loader) * bs, since drop_last=True is enabled
total_embeddings = np.zeros((len(loader)*loader.batch_size, model.embed_dim))
total_labels = np.zeros(len(loader)*loader.batch_size)
for idx, (images, labels) in enumerate(loader):
images = images.cuda()
bsz = labels.shape[0]
if scaler:
with torch.cuda.amp.autocast():
embed = model(images)
total_embeddings[idx * bsz: (idx + 1) * bsz] = embed.detach().cpu().numpy()
total_labels[idx * bsz: (idx + 1) * bsz] = labels.detach().numpy()
else:
embed = model(images)
total_embeddings[idx * bsz: (idx + 1) * bsz] = embed.detach().cpu().numpy()
total_labels[idx * bsz: (idx + 1) * bsz] = labels.detach().numpy()
del images, labels, embed
torch.cuda.empty_cache()
return np.float32(total_embeddings), total_labels.astype(int)
def train_epoch_constructive(train_loader, model, criterion, optimizer, scaler, ema):
model.train()
train_loss = []
for idx, (images, labels) in enumerate(train_loader):
images = torch.cat([images[0]['image'], images[1]['image']], dim=0)
images = images.cuda()
labels = labels.cuda()
bsz = labels.shape[0]
if scaler:
with torch.cuda.amp.autocast():
embed = model(images)
f1, f2 = torch.split(embed, [bsz, bsz], dim=0)
embed = torch.cat([f1.unsqueeze(1), f2.unsqueeze(1)], dim=1)
loss = criterion(embed, labels)
else:
embed = model(images)
f1, f2 = torch.split(embed, [bsz, bsz], dim=0)
embed = torch.cat([f1.unsqueeze(1), f2.unsqueeze(1)], dim=1)
loss = criterion(embed, labels)
del images, labels, embed
torch.cuda.empty_cache()
train_loss.append(loss.item())
optimizer.zero_grad()
if scaler:
scaler.scale(loss).backward()
scaler.step(optimizer)
scaler.update()
else:
loss.backward()
optimizer.step()
if ema:
ema.update(model.parameters())
return {'loss': np.mean(train_loss)}
def validation_constructive(valid_loader, train_loader, model, scaler):
calculator = AccuracyCalculator(k=1)
model.eval()
query_embeddings, query_labels = compute_embeddings(valid_loader, model, scaler)
reference_embeddings, reference_labels = compute_embeddings(train_loader, model, scaler)
acc_dict = calculator.get_accuracy(
query_embeddings,
reference_embeddings,
query_labels,
reference_labels,
embeddings_come_from_same_source=False
)
del query_embeddings, query_labels, reference_embeddings, reference_labels
torch.cuda.empty_cache()
return acc_dict
def train_epoch_ce(train_loader, model, criterion, optimizer, scaler, ema):
model.train()
train_loss = []
for batch_i, (data, target) in enumerate(train_loader):
data, target = data.cuda(), target.cuda()
optimizer.zero_grad()
if scaler:
with torch.cuda.amp.autocast():
output = model(data)
loss = criterion(output, target)
train_loss.append(loss.item())
scaler.scale(loss).backward()
scaler.step(optimizer)
scaler.update()
else:
output = model(data)
loss = criterion(output, target)
train_loss.append(loss.item())
loss.backward()
optimizer.step()
if ema:
ema.update(model.parameters())
del data, target, output
torch.cuda.empty_cache()
return {"loss": np.mean(train_loss)}
def validation_ce(model, criterion, valid_loader, scaler):
model.eval()
val_loss = []
valid_bs = valid_loader.batch_size
# note that it's okay to do len(loader) * bs, since drop_last=True is enabled
y_pred, y_true = np.zeros(len(valid_loader)*valid_bs), np.zeros(len(valid_loader)*valid_bs)
correct_samples = 0
for batch_i, (data, target) in enumerate(valid_loader):
with torch.no_grad():
data, target = data.cuda(), target.cuda()
if scaler:
with torch.cuda.amp.autocast():
output = model(data)
if criterion:
loss = criterion(output, target)
val_loss.append(loss.item())
else:
output = model(data)
if criterion:
loss = criterion(output, target)
val_loss.append(loss.item())
correct_samples += (
target.detach().cpu().numpy() == np.argmax(output.detach().cpu().numpy(), axis=1)
).sum()
y_pred[batch_i * valid_bs : (batch_i + 1) * valid_bs] = np.argmax(output.detach().cpu().numpy(), axis=1)
y_true[batch_i * valid_bs : (batch_i + 1) * valid_bs] = target.detach().cpu().numpy()
del data, target, output
torch.cuda.empty_cache()
valid_loss = np.mean(val_loss)
f1_scores = f1_score(y_true, y_pred, average=None)
f1_score_macro = f1_score(y_true, y_pred, average='macro')
accuracy_score = correct_samples / (len(valid_loader)*valid_bs)
metrics = {"loss": valid_loss, "accuracy": accuracy_score, "f1_scores": f1_scores, 'f1_score_macro': f1_score_macro}
return metrics
def copy_parameters_from_model(model):
copy_of_model_parameters = [p.clone().detach() for p in model.parameters() if p.requires_grad]
return copy_of_model_parameters
def copy_parameters_to_model(copy_of_model_parameters, model):
for s_param, param in zip(copy_of_model_parameters, model.parameters()):
if param.requires_grad:
param.data.copy_(s_param.data)
================================================
FILE: train.py
================================================
import argparse
import logging
import os
import time
import torch
import yaml
import shutil
from torch.utils.tensorboard import SummaryWriter
from torch_ema import ExponentialMovingAverage
from tools import utils
scaler = torch.cuda.amp.GradScaler()
def parse_config():
parser = argparse.ArgumentParser()
parser.add_argument(
"--config_name",
type=str,
default="configs/train/train_supcon_resnet18_cifar10_stage2.yml",
)
parser_args = parser.parse_args()
with open(vars(parser_args)["config_name"], "r") as config_file:
hyperparams = yaml.full_load(config_file)
return hyperparams
if __name__ == "__main__":
# parse hyperparameters
hyperparams = parse_config()
print(hyperparams)
backbone = hyperparams["model"]["backbone"]
ckpt_pretrained = hyperparams['model']['ckpt_pretrained']
num_classes = hyperparams['model']['num_classes']
amp = hyperparams['train']['amp']
ema = hyperparams['train']['ema']
ema_decay_per_epoch = hyperparams['train']['ema_decay_per_epoch']
n_epochs = hyperparams["train"]["n_epochs"]
logging_name = hyperparams['train']['logging_name']
target_metric = hyperparams['train']['target_metric']
stage = hyperparams['train']['stage']
data_dir = hyperparams["dataset"]
optimizer_params = hyperparams["optimizer"]
scheduler_params = hyperparams["scheduler"]
criterion_params = hyperparams["criterion"]
batch_sizes = {
"train_batch_size": hyperparams["dataloaders"]["train_batch_size"],
'valid_batch_size': hyperparams['dataloaders']['valid_batch_size']
}
num_workers = hyperparams["dataloaders"]["num_workers"]
if not amp: scaler = None
utils.seed_everything()
# create model, loaders, optimizer, etc
transforms = utils.build_transforms(second_stage=(stage == 'second'))
loaders = utils.build_loaders(data_dir, transforms, batch_sizes, num_workers, second_stage=(stage == 'second'))
model = utils.build_model(backbone, second_stage=(stage == 'second'), num_classes=num_classes, ckpt_pretrained=ckpt_pretrained).cuda()
if ema:
iters = len(loaders['train_features_loader'])
ema_decay = ema_decay_per_epoch**(1/iters)
ema = ExponentialMovingAverage(model.parameters(), decay=ema_decay)
optim = utils.build_optim(model, optimizer_params, scheduler_params, criterion_params)
criterion, optimizer, scheduler = (
optim["criterion"],
optim["optimizer"],
optim["scheduler"],
)
# handle logging (regular logs, tensorboard, and weights)
if logging_name is None:
logging_name = "stage_{}_model_{}_dataset_{}".format(stage, backbone, data_dir.split("/")[-1])
shutil.rmtree("weights/{}".format(logging_name), ignore_errors=True)
shutil.rmtree(
"runs/{}".format(logging_name),
ignore_errors=True,
)
shutil.rmtree(
"logs/{}".format(logging_name),
ignore_errors=True,
)
os.makedirs(
"logs/{}".format(logging_name),
exist_ok=True,
)
writer = SummaryWriter("runs/{}".format(logging_name))
logging_dir = "logs/{}".format(logging_name)
logging_path = os.path.join(logging_dir, "train.log")
logging.basicConfig(filename=logging_path, level=logging.INFO, filemode="w+")
# epoch loop
metric_best = 0
for epoch in range(n_epochs):
utils.add_to_logs(logging, "{}, epoch {}".format(time.ctime(), epoch))
start_training_time = time.time()
if stage == 'first':
train_metrics = utils.train_epoch_constructive(loaders['train_supcon_loader'], model, criterion, optimizer, scaler, ema)
else:
train_metrics = utils.train_epoch_ce(loaders['train_features_loader'], model, criterion, optimizer, scaler, ema)
end_training_time = time.time()
if ema:
copy_of_model_parameters = utils.copy_parameters_from_model(model)
ema.copy_to(model.parameters())
start_validation_time = time.time()
if stage == 'first':
valid_metrics_projection_head = utils.validation_constructive(loaders['valid_loader'], loaders['train_features_loader'], model, scaler)
model.use_projection_head(False)
valid_metrics_encoder = utils.validation_constructive(loaders['valid_loader'], loaders['train_features_loader'], model, scaler)
model.use_projection_head(True)
print(
'epoch {}, train time {:.2f} valid time {:.2f} train loss {:.2f}\nvalid acc dict projection head {}\nvalid acc dict encoder {}'.format(
epoch,
end_training_time - start_training_time,
time.time() - start_validation_time,
train_metrics['loss'], valid_metrics_projection_head, valid_metrics_encoder))
valid_metrics = valid_metrics_projection_head
else:
valid_metrics = utils.validation_ce(model, criterion, loaders['valid_loader'], scaler)
print(
'epoch {}, train time {:.2f} valid time {:.2f} train loss {:.2f}\n valid acc dict {}\n'.format(
epoch,
end_training_time - start_training_time,
time.time() - start_validation_time,
train_metrics['loss'], valid_metrics))
# write train and valid metrics to the logs
utils.add_to_tensorboard_logs(writer, train_metrics['loss'], "Loss/train", epoch)
for valid_metric in valid_metrics:
try:
utils.add_to_tensorboard_logs(writer, valid_metrics[valid_metric],
'{}/validation'.format(valid_metric), epoch)
except AssertionError:
# in case valid metric is a list
pass
if stage == 'first':
utils.add_to_logs(
logging,
"Epoch {}, train loss: {:.4f}\nvalid metrics projection head: {}\nvalid metric encoder: {}".format(
epoch,
train_metrics['loss'],
valid_metrics_projection_head,
valid_metrics_encoder
),
)
else:
utils.add_to_logs(
logging,
"Epoch {}, train loss: {:.4f} valid metrics: {}".format(
epoch,
train_metrics['loss'],
valid_metrics
),
)
# check if the best value of metric changed. If so -> save the model
if valid_metrics[target_metric] > metric_best:
utils.add_to_logs(
logging,
"{} increased ({:.6f} --> {:.6f}). Saving model ...".format(target_metric,
metric_best, valid_metrics[target_metric]
),
)
os.makedirs(
"weights/{}".format(logging_name),
exist_ok=True,
)
torch.save(
{
"epoch": epoch,
"model_state_dict": model.state_dict(),
"optimizer_state_dict": optimizer.state_dict(),
},
"weights/{}/epoch{}".format(
logging_name, epoch
),
)
metric_best = valid_metrics[target_metric]
# if ema is used, go back to regular weights without ema
if ema:
utils.copy_parameters_to_model(copy_of_model_parameters, model)
scheduler.step()
writer.close()
gitextract_cf8_gd7q/ ├── .gitattributes ├── LICENSE.md ├── README.md ├── configs/ │ └── train/ │ ├── lr_finder_supcon_resnet18_cifar100_stage2.yml │ ├── lr_finder_supcon_resnet18_cifar10_stage2.yml │ ├── swa_supcon_resnet18_cifar100_stage1.yml │ ├── swa_supcon_resnet18_cifar100_stage2.yml │ ├── swa_supcon_resnet18_cifar10_stage1.yml │ ├── swa_supcon_resnet18_cifar10_stage2.yml │ ├── train_supcon_resnet18_cifar100_stage1.yml │ ├── train_supcon_resnet18_cifar100_stage2.yml │ ├── train_supcon_resnet18_cifar10_stage1.yml │ └── train_supcon_resnet18_cifar10_stage2.yml ├── learning_rate_finder.py ├── requirements.txt ├── swa.py ├── t-SNE.ipynb ├── tools/ │ ├── backbones.py │ ├── datasets.py │ ├── losses.py │ ├── models.py │ ├── optimizers.py │ ├── schedulers.py │ └── utils.py └── train.py
SYMBOL INDEX (39 symbols across 7 files)
FILE: learning_rate_finder.py
function parse_config (line 10) | def parse_config():
FILE: swa.py
function swa (line 11) | def swa(paths):
function parse_config (line 23) | def parse_config():
FILE: tools/datasets.py
class SupConDatasetCifar10 (line 3) | class SupConDatasetCifar10(torchvision.datasets.CIFAR10):
method __init__ (line 4) | def __init__(self, data_dir, train, transform, second_stage):
method __getitem__ (line 10) | def __getitem__(self, idx):
class SupConDatasetCifar100 (line 25) | class SupConDatasetCifar100(torchvision.datasets.CIFAR100):
method __init__ (line 26) | def __init__(self, data_dir, train, transform, second_stage):
method __getitem__ (line 32) | def __getitem__(self, idx):
function create_supcon_dataset (line 51) | def create_supcon_dataset(dataset_name, data_dir, train, transform, seco...
FILE: tools/losses.py
class SupConLoss (line 5) | class SupConLoss(nn.Module):
method __init__ (line 8) | def __init__(self, temperature=0.07, contrast_mode='all',
method forward (line 15) | def forward(self, features, labels=None, mask=None):
class LabelSmoothingLoss (line 95) | class LabelSmoothingLoss(nn.Module):
method __init__ (line 96) | def __init__(self, classes, smoothing=0, dim=-1):
method forward (line 103) | def forward(self, pred, target):
FILE: tools/models.py
function create_encoder (line 9) | def create_encoder(backbone):
class SupConModel (line 34) | class SupConModel(nn.Module):
method __init__ (line 35) | def __init__(self, backbone='resnet50', projection_dim=128, second_sta...
method use_projection_head (line 53) | def use_projection_head(self, mode):
method forward (line 60) | def forward(self, x):
FILE: tools/utils.py
function seed_everything (line 20) | def seed_everything(seed=42):
function add_to_logs (line 29) | def add_to_logs(logging, message):
function add_to_tensorboard_logs (line 33) | def add_to_tensorboard_logs(writer, message, tag, index):
class TwoCropTransform (line 37) | class TwoCropTransform:
method __init__ (line 39) | def __init__(self, crop_transform):
method __call__ (line 42) | def __call__(self, x):
function build_transforms (line 46) | def build_transforms(second_stage):
function build_loaders (line 81) | def build_loaders(data_dir, transforms, batch_sizes, num_workers, second...
function build_model (line 114) | def build_model(backbone, second_stage=False, num_classes=None, ckpt_pre...
function build_optim (line 123) | def build_optim(model, optimizer_params, scheduler_params, loss_params):
function compute_embeddings (line 139) | def compute_embeddings(loader, model, scaler):
function train_epoch_constructive (line 163) | def train_epoch_constructive(train_loader, model, criterion, optimizer, ...
function validation_constructive (line 206) | def validation_constructive(valid_loader, train_loader, model, scaler):
function train_epoch_ce (line 227) | def train_epoch_ce(train_loader, model, criterion, optimizer, scaler, ema):
function validation_ce (line 258) | def validation_ce(model, criterion, valid_loader, scaler):
function copy_parameters_from_model (line 299) | def copy_parameters_from_model(model):
function copy_parameters_to_model (line 304) | def copy_parameters_to_model(copy_of_model_parameters, model):
FILE: train.py
function parse_config (line 16) | def parse_config():
Condensed preview — 25 files, each showing path, character count, and a content snippet. Download the .json file or copy for the full structured content (104K chars).
[
{
"path": ".gitattributes",
"chars": 26,
"preview": "*.ipynb linguist-vendored\n"
},
{
"path": "LICENSE.md",
"chars": 1069,
"preview": "MIT License\n\nCopyright (c) 2022 Ivan Panshin\n\nPermission is hereby granted, free of charge, to any person obtaining a co"
},
{
"path": "README.md",
"chars": 8151,
"preview": "\n\n# SupCon-Framework\n<p align=\"center\"><img src=\"https://github.com/ivanpanshin/SupCon-Framework/blob/main/images/logo.p"
},
{
"path": "configs/train/lr_finder_supcon_resnet18_cifar100_stage2.yml",
"chars": 321,
"preview": "model:\n backbone: resnet18\n ckpt_pretrained:\n num_classes: 100\n\ndataset: data/cifar100\n\ndataloaders:\n train_batch_si"
},
{
"path": "configs/train/lr_finder_supcon_resnet18_cifar10_stage2.yml",
"chars": 319,
"preview": "model:\n backbone: resnet18\n ckpt_pretrained:\n num_classes: 10\n\ndataset: data/cifar10\n\ndataloaders:\n train_batch_size"
},
{
"path": "configs/train/swa_supcon_resnet18_cifar100_stage1.yml",
"chars": 403,
"preview": "model:\n backbone: resnet18\n num_classes:\n top_k_checkoints: 3\n\ntrain:\n amp: True # set this to True, if your GPU sup"
},
{
"path": "configs/train/swa_supcon_resnet18_cifar100_stage2.yml",
"chars": 409,
"preview": "model:\n backbone: resnet18\n num_classes: 100\n top_k_checkoints: 3\n\ntrain:\n amp: True # set this to True, if your GPU"
},
{
"path": "configs/train/swa_supcon_resnet18_cifar10_stage1.yml",
"chars": 401,
"preview": "model:\n backbone: resnet18\n num_classes:\n top_k_checkoints: 3\n\ntrain:\n amp: True # set this to True, if your GPU sup"
},
{
"path": "configs/train/swa_supcon_resnet18_cifar10_stage2.yml",
"chars": 406,
"preview": "model:\n backbone: resnet18\n num_classes: 10\n top_k_checkoints: 3\n\ntrain:\n amp: True # set this to True, if your GPU "
},
{
"path": "configs/train/train_supcon_resnet18_cifar100_stage1.yml",
"chars": 972,
"preview": "model:\n backbone: resnet18\n ckpt_pretrained:\n num_classes: # in the first stage of training we don't need num_classes"
},
{
"path": "configs/train/train_supcon_resnet18_cifar100_stage2.yml",
"chars": 940,
"preview": "model:\n backbone: resnet18\n ckpt_pretrained: weights/supcon_first_stage_cifar100/swa\n num_classes: 100\n\ntrain:\n n_ep"
},
{
"path": "configs/train/train_supcon_resnet18_cifar10_stage1.yml",
"chars": 970,
"preview": "model:\n backbone: resnet18\n ckpt_pretrained:\n num_classes: # in the first stage of training we don't need num_classes"
},
{
"path": "configs/train/train_supcon_resnet18_cifar10_stage2.yml",
"chars": 935,
"preview": "model:\n backbone: resnet18\n ckpt_pretrained: weights/supcon_first_stage_cifar10/swa\n num_classes: 10\n\ntrain:\n n_epoc"
},
{
"path": "learning_rate_finder.py",
"chars": 2200,
"preview": "import argparse\nimport os\n\nimport matplotlib.pyplot as plt\nimport yaml\nfrom torch_lr_finder import LRFinder\nfrom tools i"
},
{
"path": "requirements.txt",
"chars": 2077,
"preview": "absl-py==0.11.0\nalbumentations==0.5.2\nargon2-cffi==20.1.0\nasync-generator==1.10\nattrs==20.3.0\nbackcall==0.2.0\nbleach==3."
},
{
"path": "swa.py",
"chars": 2864,
"preview": "import os\nfrom collections import OrderedDict\nimport torch\nimport argparse\nimport yaml\n\nfrom tools import utils\n\nscaler "
},
{
"path": "t-SNE.ipynb",
"chars": 47914,
"preview": "{\n \"cells\": [\n {\n \"cell_type\": \"code\",\n \"execution_count\": 1,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n "
},
{
"path": "tools/backbones.py",
"chars": 1039,
"preview": "import torchvision.models as models\n\n# for timm models we don't have such files, since it provides a simple wrapper timm"
},
{
"path": "tools/datasets.py",
"chars": 2379,
"preview": "import torchvision\n\nclass SupConDatasetCifar10(torchvision.datasets.CIFAR10):\n def __init__(self, data_dir, train, tr"
},
{
"path": "tools/losses.py",
"chars": 4463,
"preview": "import torch.nn as nn\nimport torch\n\n\nclass SupConLoss(nn.Module):\n \"\"\"Supervised Contrastive Learning: https://arxiv."
},
{
"path": "tools/models.py",
"chars": 2492,
"preview": "import torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport timm\n\nfrom .backbones import BACKBONES\n\n\ndef c"
},
{
"path": "tools/optimizers.py",
"chars": 270,
"preview": "import torch.optim as optim\nimport torch_optimizer as jettify_optim\n\n\nOPTIMIZERS = {\n \"Adam\": optim.Adam,\n 'AdamW'"
},
{
"path": "tools/schedulers.py",
"chars": 174,
"preview": "import torch.optim as optim\n\n\nSCHEDULERS = {\n \"ReduceLROnPlateau\": optim.lr_scheduler.ReduceLROnPlateau,\n \"CosineA"
},
{
"path": "tools/utils.py",
"chars": 11036,
"preview": "import torch\nimport albumentations as A\nfrom albumentations import pytorch as AT\nfrom torch.utils.data import DataLoader"
},
{
"path": "train.py",
"chars": 7639,
"preview": "import argparse\nimport logging\nimport os\nimport time\nimport torch\nimport yaml\nimport shutil\nfrom torch.utils.tensorboard"
}
]
About this extraction
This page contains the full source code of the ivanpanshin/SupCon-Framework GitHub repository, extracted and formatted as plain text for AI agents and large language models (LLMs). The extraction includes 25 files (97.5 KB), approximately 44.9k tokens, and a symbol index with 39 extracted functions, classes, methods, constants, and types. Use this with OpenClaw, Claude, ChatGPT, Cursor, Windsurf, or any other AI tool that accepts text input. You can copy the full output to your clipboard or download it as a .txt file.
Extracted by GitExtract — free GitHub repo to text converter for AI. Built by Nikandr Surkov.