[
  {
    "path": ".gitignore",
    "content": "__pycache__\n*.pyc\nTraining_Results\nwandb\ndiva_main.py\n"
  },
  {
    "path": "LICENSE",
    "content": "MIT License\n\nCopyright (c) 2020 Karsten Roth\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\nSOFTWARE.\n"
  },
  {
    "path": "README.md",
    "content": "# Deep Metric Learning Research in PyTorch\n\n---\n## What can I find here?\n\nThis repository contains all code and implementations used in:\n\n```\nRevisiting Training Strategies and Generalization Performance in Deep Metric Learning\n```\n\naccepted to **ICML 2020**.\n\n**Link**: https://arxiv.org/abs/2002.08473\n\nThe code is meant to serve as a research starting point in Deep Metric Learning.\nBy implementing key baselines under a consistent setting and logging a vast set of metrics, it should be easier to ensure that method gains are not due to implementational variations, while better understanding driving factors.\n\nIt is set up in a modular way to allow for fast and detailed prototyping, but with key elements written in a way that allows the code to be directly copied into other pipelines. In addition, multiple training and test metrics are logged in W&B to allow for easy and large-scale evaluation.\n\nFinally, please find a public W&B repo with key runs performed in the paper here: https://app.wandb.ai/confusezius/RevisitDML.\n\n**Contact**: Karsten Roth, karsten.rh1@gmail.com  \n\n*Suggestions are always welcome!*\n\n---\n## Some Notes:\n\nIf you use this code in your research, please cite\n```\n@misc{roth2020revisiting,\n    title={Revisiting Training Strategies and Generalization Performance in Deep Metric Learning},\n    author={Karsten Roth and Timo Milbich and Samarth Sinha and Prateek Gupta and Björn Ommer and Joseph Paul Cohen},\n    year={2020},\n    eprint={2002.08473},\n    archivePrefix={arXiv},\n    primaryClass={cs.CV}\n}\n```\n\nThis repository contains (in parts) code that has been adapted from:\n* https://github.com/idstcv/SoftTriple\n* https://github.com/bnu-wangxun/Deep_Metric\n* https://github.com/valerystrizh/pytorch-histogram-loss\n* https://github.com/Confusezius/Deep-Metric-Learning-Baselines\n\nMake sure to also check out the following repo with a great plug-and-play implementation of DML methods:\n* https://github.com/KevinMusgrave/pytorch-metric-learning\n\n---\n\n**[All implemented methods and metrics are listed at the bottom!](#-implemented-methods)**\n\n---\n\n## Paper-related Information\n\n#### Reproduce results from our paper **[Revisiting Training Strategies and Generalization Performance in Deep Metric Learning](https://arxiv.org/pdf/2002.08473.pdf)**\n\n* *ALL* standardized Runs that were used are available in `Revisit_Runs.sh`.\n* These runs are also logged in this public W&B repo: https://app.wandb.ai/confusezius/RevisitDML.\n* All Runs and their respective metrics can be downloaded and evaluated to generate the plots in our paper by following `Result_Evaluations.py`. This also allows for potential introspection of other relations. It also converts results directly into Latex-table format with mean and standard deviations.\n* To utilize different batch-creation methods, simply set the flag `--data_sampler` to the method of choice. Allowed flags are listed in `datasampler/__init__.py`.\n* To use the proposed spectral regularization for tuple-based methods, set `--batch_mining rho_distance` with flip probability `--miner_rho_distance_cp e.g. 0.2`.\n* A script to run the toy experiments in the paper is provided in `toy_experiments`.\n\n**Note**: There may be small deviations in results based on the Hardware (e.g. between P100 and RTX GPUs) and Software (different PyTorch/Cuda versions) used to run these experiments, but they should be covered in the standard deviations reported in the paper.\n\n---\n\n## How to use this Repo\n\n### Requirements:\n\n* PyTorch 1.2.0+ & Faiss-Gpu\n* Python 3.6+\n* pretrainedmodels, torchvision 0.3.0+\n\nAn exemplary setup of a virtual environment containing everything needed:\n```\n(1) wget  https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh\n(2) bash Miniconda3-latest-Linux-x86_64.sh (say yes to append path to bashrc)\n(3) source .bashrc\n(4) conda create -n DL python=3.6\n(5) conda activate DL\n(6) conda install matplotlib scipy scikit-learn scikit-image tqdm pandas pillow\n(7) conda install pytorch torchvision faiss-gpu cudatoolkit=10.0 -c pytorch\n(8) pip install wandb pretrainedmodels\n(9) Run the scripts!\n```\n\n### Datasets:\nData for\n* CUB200-2011 (http://www.vision.caltech.edu/visipedia/CUB-200.html)\n* CARS196 (https://ai.stanford.edu/~jkrause/cars/car_dataset.html)\n* Stanford Online Products (http://cvgl.stanford.edu/projects/lifted_struct/)\n\ncan be downloaded either from the respective project sites or directly via Dropbox:\n\n* CUB200-2011 (1.08 GB): https://www.dropbox.com/s/tjhf7fbxw5f9u0q/cub200.tar?dl=0\n* CARS196 (1.86 GB): https://www.dropbox.com/s/zi2o92hzqekbmef/cars196.tar?dl=0\n* SOP (2.84 GB): https://www.dropbox.com/s/fu8dgxulf10hns9/online_products.tar?dl=0\n\n**The latter ensures that the folder structure is already consistent with this pipeline and the dataloaders**.   \n\nOtherwise, please make sure that the datasets have the following internal structure:\n\n* For CUB200-2011/CARS196:\n```\ncub200/cars196\n└───images\n|    └───001.Black_footed_Albatross\n|           │   Black_Footed_Albatross_0001_796111\n|           │   ...\n|    ...\n```\n\n* For Stanford Online Products:\n```\nonline_products\n└───images\n|    └───bicycle_final\n|           │   111085122871_0.jpg\n|    ...\n|\n└───Info_Files\n|    │   bicycle.txt\n|    │   ...\n```\n\nAssuming your folder is placed in e.g. `<$datapath/cub200>`, pass `$datapath` as input to `--source`.\n\n### Training:\nTraining is done by using `main.py` and setting the respective flags, all of which are listed and explained in `parameters.py`. A vast set of exemplary runs is provided in `Revisit_Runs.sh`.\n\n**[I.]** **A basic sample run using default parameters would like this**:\n\n```\npython main.py --loss margin --batch_mining distance --log_online \\\n              --project DML_Project --group Margin_with_Distance --seed 0 \\\n              --gpu 0 --bs 112 --data_sampler class_random --samples_per_class 2 \\\n              --arch resnet50_frozen_normalize --source $datapath --n_epochs 150 \\\n              --lr 0.00001 --embed_dim 128 --evaluate_on_gpu\n```\n\nThe purpose of each flag explained:\n\n* `--loss <loss_name>`: Name of the training objective used. See folder `criteria` for implementations of these methods.\n* `--batch_mining <batchminer_name>`: Name of the batch-miner to use (for tuple-based ranking methods). See folder `batch_mining` for implementations of these methods.\n* `--log_online`: Log metrics online via either W&B (Default) or CometML. Regardless, plots, weights and parameters are all stored offline as well.\n*  `--project`, `--group`: Project name as well as name of the run. Different seeds will be logged into the same `--group` online. The group as well as the used seed also define the local savename.\n* `--seed`, `--gpu`, `--source`: Basic Parameters setting the training seed, the used GPU and the path to the parent folder containing the respective Datasets.\n* `--arch`: The utilized backbone, e.g. ResNet50. You can append `_frozen` and `_normalize` to the name to ensure that BatchNorm layers are frozen and embeddings are normalized, respectively.\n* `--data_sampler`, `--samples_per_class`: How to construct a batch. The default method, `class_random`, selects classes at random and places `<samples_per_class>` samples into the batch until the batch is filled.\n* `--lr`, `--n_epochs`, `--bs` ,`--embed_dim`: Learning rate, number of training epochs, the batchsize and the embedding dimensionality.  \n* `--evaluate_on_gpu`: If set, all metrics are computed using the gpu - requires Faiss-GPU and may need additional GPU memory.\n\n#### Some Notes:\n* During training, metrics listed in `--evaluation_metrics` will be logged for both training and validation/test set. If you do not care about detailed training metric logging, simply set the flag `--no_train_metrics`. A checkpoint is saved for improvements in metrics listed in `--storage_metrics` on training, validation or test sets. Detailed information regarding the available metrics can be found at the bottom of this `README`.\n* If one wishes to use a training/validation split, simply set `--use_tv_split` and `--tv_split_perc <train/val split percentage>`.\n\n\n**[II.]** **Advanced Runs**:\n\n```\npython main.py --loss margin --batch_mining distance --loss_margin_beta 0.6 --miner_distance_lower_cutoff 0.5 ... (basic parameters)\n```\n\n* To use specific parameters that are loss, batchminer or e.g. datasampler-related, simply set the respective flag.\n* For structure and ease of use, parameters relating to a specifc loss function/batchminer etc. are marked as e.g. `--loss_<lossname>_<parameter_name>`, see `parameters.py`.\n* However, every parameter can be called from every class, as all parameters are stored in a shared namespace that is passed to all methods. This makes it easy to create novel fusion losses and the likes.\n\n\n### Evaluating Results with W&B\nHere some information on using W&B (highly encouraged!)\n\n* Create an account here (free): https://wandb.ai\n* After the account is set, make sure to include your API key in `parameters.py` under `--wandb_key`.\n* To make sure that W&B data can be stored, ensure to run `wandb on` in the folder pointed to by `--save_path`.\n* When data is logged online to W&B, one can use `Result_Evaluations.py` to download all data, create named metric and correlation plots and output a summary in the form of a latex-ready table with mean and standard deviations of all metrics. **This ensures that there are no errors between computed and reported results.**\n\n\n### Creating custom methods:\n\n1. **Create custom objectives**: Simply take a look at e.g. `criteria/margin.py`, and ensure that the used methods has the following properties:\n  * Inherit from `torch.nn.Module` and define a custom `forward()` function.\n  * When using trainable parameters, make sure to either provide a `self.lr` to set the learning rate of the loss-specific parameters, or set `self.optim_dict_list`, which is a list containing optimization dictionaries passed to the optimizer (see e.g `criteria/proxynca.py`). If both are set, `self.optim_dict_list` has priority.\n  * Depending on the loss, remember to set the variables `ALLOWED_MINING_OPS  = None or list of allowed mining operations`, `REQUIRES_BATCHMINER = False or True`, `REQUIRES_OPTIM = False or True` to denote if the method needs a batchminer or optimization of internal parameters.\n\n\n2. **Create custom batchminer**: Simply take a look at e.g. `batch_mining/distance.py` - The miner needs to be a class with a defined `__call__()`-function, taking in a batch and labels and returning e.g. a list of triplets.\n\n3. **Create custom datasamplers**:Simply take a look at e.g. `datasampler/class_random_sampler.py`. The sampler needs to inherit from `torch.utils.data.sampler.Sampler` and has to provide a `__iter__()` and a `__len__()` function. It has to yield a set of indices that are used to create the batch.\n\n\n---\n\n# Implemented Methods\n\nFor a detailed explanation of everything, please refer to the supplementary of our paper!\n\n### DML criteria\n\n* **Angular** [[Deep Metric Learning with Angular Loss](https://arxiv.org/pdf/1708.01682.pdf)] `--loss angular`\n* **ArcFace** [[ArcFace: Additive Angular Margin Loss for Deep Face Recognition](https://arxiv.org/pdf/1801.07698.pdf)] `--loss arcface`\n* **Contrastive** [[Dimensionality Reduction by Learning an Invariant Mapping](http://yann.lecun.com/exdb/publis/pdf/hadsell-chopra-lecun-06.pdf)] `--loss contrastive`\n* **Generalized Lifted Structure** [[In Defense of the Triplet Loss for Person Re-Identification](https://arxiv.org/abs/1703.07737)] `--loss lifted`\n* **Histogram** [[Learning Deep Embeddings with Histogram Loss](https://arxiv.org/pdf/1611.00822.pdf)] `--loss histogram`\n* **Marginloss** [[Sampling Matters in Deep Embeddings Learning](https://arxiv.org/abs/1706.07567)] `--loss margin`\n* **MultiSimilarity** [[Multi-Similarity Loss with General Pair Weighting for Deep Metric Learning](https://arxiv.org/abs/1904.06627)] `--loss multisimilarity`\n* **N-Pair** [[Improved Deep Metric Learning with Multi-class N-pair Loss Objective](https://papers.nips.cc/paper/6200-improved-deep-metric-learning-with-multi-class-n-pair-loss-objective)] `--loss npair`\n* **ProxyNCA** [[No Fuss Distance Metric Learning using Proxies](https://arxiv.org/pdf/1703.07464.pdf)] `--loss proxynca`\n* **Quadruplet** [[Beyond triplet loss: a deep quadruplet network for person re-identification](https://arxiv.org/abs/1704.01719)] `--loss quadruplet`\n* **Signal-to-Noise Ratio (SNR)** [[Signal-to-Noise Ratio: A Robust Distance Metric for Deep Metric Learning](https://arxiv.org/pdf/1904.02616.pdf)] `--loss snr`\n* **SoftTriple** [[SoftTriple Loss: Deep Metric Learning Without Triplet Sampling](https://arxiv.org/abs/1909.05235)] `--loss softtriplet`\n* **Normalized Softmax** [[Classification is a Strong Baseline for Deep Metric Learning](https://arxiv.org/abs/1811.12649)] `--loss softmax`\n* **Triplet** [[Facenet: A unified embedding for face recognition and clustering](https://arxiv.org/abs/1503.03832)] `--loss triplet`\n\n### DML batchminer\n\n* **Random** [[Facenet: A unified embedding for face recognition and clustering](https://arxiv.org/abs/1503.03832)] `--batch_mining random`\n* **Semihard** [[Facenet: A unified embedding for face recognition and clustering](https://arxiv.org/abs/1503.03832)] `--batch_mining semihard`\n* **Softhard** [https://github.com/Confusezius/Deep-Metric-Learning-Baselines] `--batch_mining softhard`\n* **Distance-based** [[Sampling Matters in Deep Embeddings Learning](https://arxiv.org/abs/1706.07567)] `--batch_mining distance`\n* **Rho-Distance** [[Revisiting Training Strategies and Generalization Performance in Deep Metric Learning](https://arxiv.org/abs/2002.08473)] `--batch_mining rho_distance`\n* **Parametric** [[PADS: Policy-Adapted Sampling for Visual Similarity Learning](https://arxiv.org/abs/2003.11113)] `--batch_mining parametric`\n\n### Architectures\n\n* **ResNet50** [[Deep Residual Learning for Image Recognition](https://arxiv.org/abs/1512.03385)] e.g. `--arch resnet50_frozen_normalize`.\n* **Inception-BN** [[Going Deeper with Convolutions](https://arxiv.org/abs/1409.4842)] e.g. `--arch bninception_normalize_frozen`.\n* **GoogLeNet** (torchvision variant w/ BN) [[Going Deeper with Convolutions](https://arxiv.org/abs/1409.4842)] e.g. `--arch googlenet`.\n\n### Datasets\n* **CUB200-2011** [[Caltech-UCSD Birds-200-2011](http://www.vision.caltech.edu/visipedia/CUB-200-2011.html)]  `--dataset cub200`.\n* **CARS196** [[Cars Dataset](https://ai.stanford.edu/~jkrause/cars/car_dataset.html)] `--dataset cars196`.\n* **Stanford Online Products** [[Deep Metric Learning via Lifted Structured Feature Embedding](https://cvgl.stanford.edu/projects/lifted_struct/)] `--dataset online_products`.\n\n\n\n### Evaluation Metrics\n**Metrics based on Euclidean Distances**\n* **Recall@k**: Include R@1 e.g. with `e_recall@1` into the list of evaluation metrics `--evaluation_metrics`.\n* **Normalized Mutual Information (NMI)**: Include with `nmi`.\n* **F1**: include with `f1`.\n* **mAP (class-averaged)**: Include standard mAP at Recall with `mAP_lim`. You may also include `mAP_1000` for mAP limited to Recall@1000, and `mAP_c` limited to mAP at Recall@Max_Num_Samples_Per_Class. Note that all of these are heavily correlated.\n\n**Metrics based on Cosine Similarities** *(not included by default)*\n* **Cosine Recall@k**: Cosine-Similarity variant of Recall@k. Include with `c_recall@k` in `--evaluation_metrics`.\n* **Cosine Normalized Mutual Information (NMI)**: Include with `c_nmi`.\n* **Cosine F1**: include with `c_f1`.\n* **Cosine mAP (class-averaged)**: Include cosine similarity mAP at Recall variants with `c_mAP_lim`. You may also include `c_mAP_1000` for mAP limited to Recall@1000, and `c_mAP_c` limited to mAP at Recall@Max_Num_Samples_Per_Class.\n\n**Embedding Space Metrics**\n* **Spectral Variance**: This metric refers to the spectral decay metric used in our ICML paper. Include it with `rho_spectrum@1`. To exclude the `k` largest spectral values for a more robust estimate, simply include `rho_spectrum@k+1`. Adding `rho_spectrum@0` logs the whole singular value distribution, and `rho_spectrum@-1` computes KL(q,p) instead of KL(p,q).\n* **Mean Intraclass Distance**: Include the mean intraclass distance via `dists@intra`.\n* **Mean Interclass Distance**: Include the mean interlcass distance via `dists@inter`.\n* **Ratio Intra- to Interclass Distance**: Include the ratio of distances via `dists@intra_over_inter`.\n"
  },
  {
    "path": "Result_Evaluations.py",
    "content": "\"\"\"\nThis scripts downloads and evaluates W&B run data to produce plots and tables used in the original paper.\n\"\"\"\nimport numpy as np\nimport wandb\nimport matplotlib.pyplot as plt\n\n\ndef get_data(project):\n    from tqdm import tqdm\n    api = wandb.Api()\n    # Project is specified by <entity/project-name>\n    runs = api.runs(project)\n\n    info_list = []\n\n    # history_list = []\n    for run in tqdm(runs, desc='Downloading data...'):\n        config = {k:v for k,v in run.config.items() if not k.startswith('_')}\n        info_dict = {'metrics':run.history(), 'config':config}\n        info_list.append((run.name,info_dict))\n    return info_list\n\nall_df  = get_data(\"confusezius/RevisitDML\")\n\nnames_to_check = list(np.unique(['_s'.join(x[0].split('_s')[:-1]) for x in all_df]))\nmetrics        = ['Test: discriminative_e_recall: e_recall@1', 'Test: discriminative_e_recall: e_recall@2', \\\n                  'Test: discriminative_e_recall: e_recall@4', 'Test: discriminative_nmi: nmi', \\\n                  'Test: discriminative_f1: f1', 'Test: discriminative_mAP: mAP']\nmetric_names   = ['R@1', 'R@2', 'R@4', 'NMI', 'F1', 'mAP']\n\nidxs = {x:[i for i,y in enumerate(all_df) if x=='_s'.join(y[0].split('_s')[:-1])] for x in names_to_check}\nvals = {}\nfor group, runs in idxs.items():\n    if 'CUB' in group:\n        min_len = 40\n    elif 'CAR' in group:\n        min_len = 40\n    elif 'SOP' in group:\n        min_len = 40\n\n    vals[group] = {metric_name:[] for metric_name in metric_names}\n    vals[group]['Max_Epoch'] = []\n    vals[group]['Intra_over_Inter'] = []\n    vals[group]['Intra'] = []\n    vals[group]['Inter'] = []\n    vals[group]['Rho1'] = []\n    vals[group]['Rho2'] = []\n    vals[group]['Rho3'] = []\n    vals[group]['Rho4'] = []\n    for i,run in enumerate(runs):\n        name, data = all_df[run]\n        for metric,metric_name in zip(metrics, metric_names):\n            if len(data['metrics']):\n                sub_data = list(data['metrics'][metric])\n                if len(sub_data)>min_len:\n                    vals[group][metric_name].append(np.nanmax(sub_data))\n                    if metric_name=='R@1':\n                        r_argmax = np.nanargmax(sub_data)\n                        vals[group]['Max_Epoch'].append(r_argmax)\n                        vals[group]['Intra_over_Inter'].append(data['metrics']['Train: discriminative_dists: dists@intra_over_inter'][r_argmax])\n                        vals[group]['Intra'].append(data['metrics']['Train: discriminative_dists: dists@intra'][r_argmax])\n                        vals[group]['Inter'].append(data['metrics']['Train: discriminative_dists: dists@inter'][r_argmax])\n                        vals[group]['Rho1'].append(data['metrics']['Train: discriminative_rho_spectrum: rho_spectrum@-1'][r_argmax])\n                        vals[group]['Rho2'].append(data['metrics']['Train: discriminative_rho_spectrum: rho_spectrum@1'][r_argmax])\n                        vals[group]['Rho3'].append(data['metrics']['Train: discriminative_rho_spectrum: rho_spectrum@2'][r_argmax])\n                        vals[group]['Rho4'].append(data['metrics']['Train: discriminative_rho_spectrum: rho_spectrum@10'][r_argmax])\n    vals[group] = {metric_name:(np.mean(metric_vals),np.std(metric_vals)) for metric_name, metric_vals in vals[group].items()}\n\n\n###\ncub_vals = {key:item for key,item in vals.items() if 'CUB' in key}\ncar_vals = {key:item for key,item in vals.items() if 'CAR' in key}\nsop_vals = {key:item for key,item in vals.items() if 'SOP' in key}\n\n\n\n\n##########\ndef name_filter(n):\n    n = '_'.join(n.split('_')[1:])\n    return n\n\ndef name_adjust(n, prep='', app='', for_plot=True):\n    if 'Margin_b06_Distance' in n:\n        t = 'Margin (D), \\\\beta=0.6' if for_plot else 'Margin (D, \\\\beta=0.6)'\n    elif 'Margin_b12_Distance' in n:\n        t = 'Margin (D), \\\\beta=1.2' if for_plot else 'Margin (D, \\\\beta=1.2)'\n    elif 'ArcFace' in n:\n        t = 'ArcFace'\n    elif 'Histogram' in n:\n        t = 'Histogram'\n    elif 'SoftTriple' in n:\n        t = 'SoftTriple'\n    elif 'Contrastive' in n:\n        t = 'Contrastive (D)'\n    elif 'Triplet_Distance' in n:\n        t = 'Triplet (D)'\n    elif 'Quadruplet_Distance' in n:\n        t = 'Quadruplet (D)'\n    elif 'SNR_Distance' in n:\n        t = 'SNR (D)'\n    elif 'Triplet_Random' in n:\n        t = 'Triplet (R)'\n    elif 'Triplet_Semihard' in n:\n        t = 'Triplet (S)'\n    elif 'Triplet_Softhard' in n:\n        t = 'Triplet (H)'\n    elif 'Softmax' in n:\n        t = 'Softmax'\n    elif 'MS' in n:\n        t = 'Multisimilarity'\n    else:\n        t = '_'.join(n.split('_')[1:])\n\n    if for_plot:\n        t = r'${0}$'.format(t)\n\n    return prep+t+app\n\n\n########\ndef single_table(vals):\n    print_str = ''\n    for name,metrics in vals.items():\n        prep = 'R-' if 'reg_' in name else ''\n        name = name_adjust(name, for_plot=False, prep=prep)\n        add = '{0} & ${1:2.2f}\\\\pm{2:2.2f}$ & ${3:2.2f}\\\\pm{4:2.2f}$ & ${5:2.2f}\\\\pm{6:2.2f}$ & ${7:2.2f}\\\\pm{8:2.2f}$ & ${9:2.2f}\\\\pm{10:2.2f}$ & ${11:2.2f}\\\\pm{12:2.2f}$'.format(name,\n                                                                                                                                     metrics['R@1'][0]*100, metrics['R@1'][1]*100,\n                                                                                                                                     metrics['R@2'][0]*100, metrics['R@2'][1]*100,\n                                                                                                                                     metrics['F1'][0]*100, metrics['F1'][1]*100,\n                                                                                                                                     metrics['mAP'][0]*100, metrics['mAP'][1]*100,\n                                                                                                                                     metrics['NMI'][0]*100, metrics['NMI'][1]*100,\n                                                                                                                                     metrics['Max_Epoch'][0], metrics['Max_Epoch'][1])\n        print_str += add\n        print_str += '\\\\'\n        print_str += '\\\\'\n        print_str += '\\n'\n    return print_str\n\nprint(single_table(cub_vals))\nprint(single_table(car_vals))\nprint(single_table(sop_vals))\n\n\n\n########\ndef shared_table():\n    cub_names, car_names, sop_names = list(cub_vals.keys()), list(car_vals.keys()), list(sop_vals.keys())\n    cub_names  = [name_adjust(n, for_plot=False, prep='R-' if 'reg_' in n else '') for n in cub_names]\n    cub_vals_2 = {name_adjust(n, for_plot=False, prep='R-' if 'reg_' in n else ''):item for n,item in cub_vals.items()}\n    car_names = [name_adjust(n, for_plot=False, prep='R-' if 'reg_' in n else '') for n in car_names]\n    car_vals_2 = {name_adjust(n, for_plot=False, prep='R-' if 'reg_' in n else ''):item for n,item in car_vals.items()}\n    sop_names = [name_adjust(n, for_plot=False, prep='R-' if 'reg_' in n else '') for n in sop_names]\n    sop_vals_2 = {name_adjust(n, for_plot=False, prep='R-' if 'reg_' in n else ''):item for n,item in sop_vals.items()}\n    cub_vvals, car_vvals, sop_vvals = list(cub_vals.values()), list(car_vals.values()), list(sop_vals.values())\n    unique_names = np.unique(np.concatenate([cub_names, car_names, sop_names], axis=0).reshape(-1))\n    unique_names = sorted([x for x in unique_names if 'R-' not in x]) + sorted([x for x in unique_names if 'R-' in x])\n\n    print_str = ''\n\n    for name in unique_names:\n        cub_rm, cub_rs = ('{0:2.2f}'.format(cub_vals_2[name]['R@1'][0]*100), '{0:2.2f}'.format(cub_vals_2[name]['R@1'][1]*100)) if name in cub_vals_2 else ('-', '-')\n        cub_nm, cub_ns = ('{0:2.2f}'.format(cub_vals_2[name]['NMI'][0]*100), '{0:2.2f}'.format(cub_vals_2[name]['NMI'][1]*100)) if name in cub_vals_2 else ('-', '-')\n        car_rm, car_rs = ('{0:2.2f}'.format(car_vals_2[name]['R@1'][0]*100), '{0:2.2f}'.format(car_vals_2[name]['R@1'][1]*100)) if name in car_vals_2 else ('-', '-')\n        car_nm, car_ns = ('{0:2.2f}'.format(car_vals_2[name]['NMI'][0]*100), '{0:2.2f}'.format(car_vals_2[name]['NMI'][1]*100)) if name in car_vals_2 else ('-', '-')\n        sop_rm, sop_rs = ('{0:2.2f}'.format(sop_vals_2[name]['R@1'][0]*100), '{0:2.2f}'.format(sop_vals_2[name]['R@1'][1]*100)) if name in sop_vals_2 else ('-', '-')\n        sop_nm, sop_ns = ('{0:2.2f}'.format(sop_vals_2[name]['NMI'][0]*100), '{0:2.2f}'.format(sop_vals_2[name]['NMI'][1]*100)) if name in sop_vals_2 else ('-', '-')\n\n        add = '{0} & ${1}\\\\pm{2}$ & ${3}\\\\pm{4}$ & ${5}\\\\pm{6}$ & ${7}\\\\pm{8}$ & ${9}\\\\pm{10}$ & ${11}\\\\pm{12}$'.format(name,\n                                                                                                                        cub_rm, cub_rs,\n                                                                                                                        cub_nm, cub_ns,\n                                                                                                                        car_rm, car_rs,\n                                                                                                                        car_nm, car_ns,\n                                                                                                                        sop_rm, sop_rs,\n                                                                                                                        sop_nm, sop_ns)\n\n        print_str += add\n        print_str += '\\\\'\n        print_str += '\\\\'\n        print_str += '\\n'\n    return print_str\n\nprint(shared_table())\n\n\n\n\n\"\"\"===================================================\"\"\"\ndef give_basic_metr(vals, key='CUB'):\n    if key=='CUB':\n        Basic  = sorted(list(filter(lambda x: '{}_'.format(key) in x, list(vals.keys()))))\n    elif key=='CARS':\n        Basic  = sorted(list(filter(lambda x: 'CARS_' in x, list(vals.keys()))))\n    elif key=='SOP':\n        Basic  = sorted(list(filter(lambda x: 'SOP_' in x, list(vals.keys()))))\n\n    basic_recall      = np.array([vals[k]['R@1'][0] for k in Basic])\n    basic_recall_err  = np.array([vals[k]['R@1'][1] for k in Basic])\n    #\n    basic_recall2 = np.array([vals[k]['R@2'][0] for k in Basic])\n    basic_recall4 = np.array([vals[k]['R@4'][0] for k in Basic])\n    basic_nmi     = np.array([vals[k]['NMI'][0] for k in Basic])\n    basic_f1      = np.array([vals[k]['F1'][0] for k in Basic])\n    basic_map     = np.array([vals[k]['mAP'][0] for k in Basic])\n\n    mets = [basic_recall, basic_recall2, basic_recall4, basic_nmi, basic_f1, basic_map]\n\n    return Basic, mets, basic_recall, basic_recall_err\n\n\ndef give_reg_metr(vals, key='CUB'):\n    if key=='CUB':\n        RhoReg = sorted(list(filter(lambda x: '{}reg_'.format(key) in x, list(vals.keys()))))\n    elif key=='CARS':\n        RhoReg = sorted(list(filter(lambda x: 'CARreg_' in x, list(vals.keys()))))\n    elif key=='SOP':\n        RhoReg = sorted(list(filter(lambda x: 'SOPreg_' in x, list(vals.keys()))))\n\n    rho_recall      = np.array([vals[k]['R@1'][0] for k in RhoReg])\n    rho_recall_err  = np.array([vals[k]['R@1'][1] for k in RhoReg])\n    #\n    rho_recall2 = np.array([vals[k]['R@2'][0] for k in RhoReg])\n    rho_recall4 = np.array([vals[k]['R@4'][0] for k in RhoReg])\n    rho_nmi     = np.array([vals[k]['NMI'][0] for k in RhoReg])\n    rho_f1      = np.array([vals[k]['F1'][0] for k in RhoReg])\n    rho_map     = np.array([vals[k]['mAP'][0] for k in RhoReg])\n\n    mets = [rho_recall, rho_recall2, rho_recall4, rho_nmi, rho_f1, rho_map]\n\n    return RhoReg, mets, rho_recall, rho_recall_err\n\ncub_basic_names, cub_mets, cub_basic_recall, cub_basic_recall_err = give_basic_metr(cub_vals, key='CUB')\ncar_basic_names, car_mets, car_basic_recall, car_basic_recall_err = give_basic_metr(car_vals, key='CARS')\nsop_basic_names, sop_mets, sop_basic_recall, sop_basic_recall_err = give_basic_metr(sop_vals, key='SOP')\ncub_reg_names, cub_reg_mets, cub_reg_recall, cub_reg_recall_err = give_reg_metr(cub_vals, key='CUB')\ncar_reg_names, car_reg_mets, car_reg_recall, car_reg_recall_err = give_reg_metr(car_vals, key='CARS')\nsop_reg_names, sop_reg_mets, sop_reg_recall, sop_reg_recall_err = give_reg_metr(sop_vals, key='SOP')\n\n\n\n\n\n\n\n\n\"\"\"=============================================================\"\"\"\n# def produce_plot(basic_recall, basic_recall_err, BasicLosses, vals, ylim=[0.58, 0.635]):\n#\n#     intra  = np.array([vals[k]['Intra'][0] for k in BasicLosses])\n#     inter  = np.array([vals[k]['Inter'][0] for k in BasicLosses])\n#     ratio  = np.array([vals[k]['Intra_over_Inter'][0] for k in BasicLosses])\n#     rho1  = np.array([vals[k]['Rho1'][0] for k in BasicLosses])\n#     rho2  = np.array([vals[k]['Rho2'][0] for k in BasicLosses])\n#     rho3  = np.array([vals[k]['Rho3'][0] for k in BasicLosses])\n#     rho4  = np.array([vals[k]['Rho4'][0] for k in BasicLosses])\n#\n#     def comp(met):\n#         sort = np.argsort(met)\n#         corr = np.corrcoef(met[sort],basic_recall[sort])[0,1]\n#         m,b  = np.polyfit(met[sort], basic_recall[sort], 1)\n#         lim  = [np.min(met)*0.9, np.max(met)*1.1]\n#         x    = np.linspace(lim[0], lim[1], 50)\n#         linfit = m*x + b\n#         return sort, corr, linfit, x, lim\n#\n#     intra_sort, intra_corr, intra_linfit, intra_x, intra_lim = comp(intra)\n#     inter_sort, inter_corr, inter_linfit, inter_x, inter_lim = comp(inter)\n#     ratio_sort, ratio_corr, ratio_linfit, ratio_x, ratio_lim = comp(ratio)\n#     rho1_sort, rho1_corr, rho1_linfit, rho1_x, rho1_lim = comp(rho1)\n#     rho2_sort, rho2_corr, rho2_linfit, rho2_x, rho2_lim = comp(rho2)\n#     rho3_sort, rho3_corr, rho3_linfit, rho3_x, rho3_lim = comp(rho3)\n#     rho4_sort, rho4_corr, rho4_linfit, rho4_x, rho4_lim = comp(rho4)\n#\n#\n#\n#     f,ax = plt.subplots(1,4)\n#     # f,ax = plt.subplots(1,7)\n#     colors = np.array([np.random.rand(3,) for _ in range(len(basic_recall))])\n#     for i in range(len(colors)):\n#         ax[0].errorbar(intra[intra_sort][i], basic_recall[intra_sort][i], yerr=basic_recall_err[intra_sort][i], fmt='o', color=colors[intra_sort][i], ecolor='gray', elinewidth=3, capsize=0, label='Basic Criteria', markersize=8)\n#         ax[1].errorbar(inter[inter_sort][i], basic_recall[inter_sort][i], yerr=basic_recall_err[inter_sort][i], fmt='o', color=colors[inter_sort][i], ecolor='gray', elinewidth=3, capsize=0, label='Basic Criteria', markersize=8)\n#         ax[2].errorbar(ratio[ratio_sort][i], basic_recall[ratio_sort][i], yerr=basic_recall_err[ratio_sort][i], fmt='o', color=colors[ratio_sort][i], ecolor='gray', elinewidth=3, capsize=0, label='Basic Criteria', markersize=8)\n#         # ax[3].errorbar(rho1[rho1_sort][i], basic_recall[rho1_sort][i], yerr=basic_recall_err[rho1_sort][i], fmt='o', color=colors[rho1_sort][i], ecolor='gray', elinewidth=3, capsize=0, label='Basic Criteria', markersize=8)\n#         # ax[4].errorbar(rho2[rho2_sort][i], basic_recall[rho2_sort][i], yerr=basic_recall_err[rho2_sort][i], fmt='o', color=colors[rho2_sort][i], ecolor='gray', elinewidth=3, capsize=0, label='Basic Criteria', markersize=8)\n#         ax[3].errorbar(rho3[rho3_sort][i], basic_recall[rho3_sort][i], yerr=basic_recall_err[rho3_sort][i], fmt='o', color=colors[rho3_sort][i], ecolor='gray', elinewidth=3, capsize=0, label='Basic Criteria', markersize=8)\n#         # ax[6].errorbar(rho4[rho4_sort][i], basic_recall[rho4_sort][i], yerr=basic_recall_err[rho4_sort][i], fmt='o', color=colors[rho4_sort][i], ecolor='gray', elinewidth=3, capsize=0, label='Basic Criteria', markersize=8)\n#     ax[1].set_yticks([])\n#     ax[2].set_yticks([])\n#     ax[3].set_yticks([])\n#     # ax[4].set_yticks([])\n#     # ax[5].set_yticks([])\n#     # ax[6].set_yticks([])\n#     ax[0].plot(intra_x, intra_linfit, 'k--', alpha=0.5, linewidth=3)\n#     ax[1].plot(inter_x, inter_linfit, 'k--', alpha=0.5, linewidth=3)\n#     ax[2].plot(ratio_x, ratio_linfit, 'k--', alpha=0.5, linewidth=3)\n#     # ax[3].plot(rho1_x, rho1_linfit, 'k--', alpha=0.5, linewidth=3)\n#     ax[3].plot(rho2_x, rho2_linfit, 'k--', alpha=0.5, linewidth=3)\n#     # ax[5].plot(rho3_x, rho3_linfit, 'k--', alpha=0.5, linewidth=3)\n#     # ax[6].plot(rho4_x, rho4_linfit, 'k--', alpha=0.5, linewidth=3)\n#     ax[0].text('Correlation: {0:2.2f}'.format(intra_corr), fontsize=18)\n#     ax[1].text('Correlation: {0:2.2f}'.format(inter_corr), fontsize=18)\n#     ax[2].text('Correlation: {0:2.2f}'.format(ratio_corr), fontsize=18)\n#     # ax[3].text('Correlation: {0:2.2f}'.format(rho1_corr), fontsize=18)\n#     ax[3].text('Correlation: {0:2.2f}'.format(rho2_corr), fontsize=18)\n#     # ax[5].set_title('Correlation: {0:2.2f}'.format(rho3_corr), fontsize=18)\n#     # ax[6].set_title('Correlation: {0:2.2f}'.format(rho4_corr), fontsize=18)\n#     ax[0].set_title(r'$\\pi_{intra}$', fontsize=18)\n#     ax[1].set_title(r'$\\pi_{inter}$', fontsize=18)\n#     ax[2].set_title(r'$\\pi_{ratio}$', fontsize=18)\n#     ax[3].set_title(r'$\\rho(\\Phi)$', fontsize=18)\n#     ax[0].set_ylabel('Recall Performance', fontsize=18)\n#     for a in ax.reshape(-1):\n#         a.tick_params(axis='both', which='major', labelsize=16)\n#         a.tick_params(axis='both', which='minor', labelsize=16)\n#         a.set_ylim(ylim)\n#     f.set_size_inches(22,8)\n#     f.tight_layout()\n\n\n# produce_plot(cub_basic_recall, cub_basic_recall_err, cub_basic_names, cub_vals, ylim=[0.581,0.635])\n# produce_plot(car_basic_recall, car_basic_recall_err, car_basic_names, car_vals, ylim=[0.70,0.82])\n# produce_plot(sop_basic_recall, sop_basic_recall_err, sop_basic_names, sop_vals, ylim=[0.67,0.79])\n\n\ndef full_rel_plot():\n    recallss= [cub_basic_recall, car_basic_recall, sop_basic_recall]\n    rerrss  = [cub_basic_recall_err, car_basic_recall_err, sop_basic_recall_err]\n    namess  = [cub_basic_names, car_basic_names, sop_basic_names]\n    valss   = [cub_vals, car_vals, sop_vals]\n    ylims   = [[0.581, 0.638],[0.70,0.82],[0.67,0.79]]\n\n    f,axes = plt.subplots(3,4)\n    for k,(ax, recalls, rerrs, names, vals, ylim) in enumerate(zip(axes, recallss, rerrss, namess, valss, ylims)):\n        col = 'red' if k==3 else 'gray'\n\n        intra  = np.array([vals[k]['Intra'][0] for k in names])\n        inter  = np.array([vals[k]['Inter'][0] for k in names])\n        ratio  = np.array([vals[k]['Intra_over_Inter'][0] for k in names])\n        rho    = np.array([vals[k]['Rho3'][0] for k in names])\n\n        def comp(met):\n            sort = np.argsort(met)\n            corr = np.corrcoef(met[sort],recalls[sort])[0,1]\n            m,b  = np.polyfit(met[sort], recalls[sort], 1)\n            lim  = [np.min(met)*0.9, np.max(met)*1.1]\n            x    = np.linspace(lim[0], lim[1], 50)\n            linfit = m*x + b\n            return sort, corr, linfit, x, lim\n\n        intra_sort, intra_corr, intra_linfit, intra_x, intra_lim = comp(intra)\n        inter_sort, inter_corr, inter_linfit, inter_x, inter_lim = comp(inter)\n        ratio_sort, ratio_corr, ratio_linfit, ratio_x, ratio_lim = comp(ratio)\n        rho_sort, rho_corr, rho_linfit, rho_x, rho_lim = comp(rho)\n\n        # f,ax = plt.subplots(1,7)\n        colors = np.array([np.random.rand(3,) for _ in range(len(recalls))])\n        for i in range(len(colors)):\n            ax[0].errorbar(intra[intra_sort][i], recalls[intra_sort][i], yerr=rerrs[intra_sort][i], fmt='o', color=colors[intra_sort][i], ecolor='gray', elinewidth=3, capsize=0, label='Basic Criteria', markersize=8)\n            ax[1].errorbar(inter[inter_sort][i], recalls[inter_sort][i], yerr=rerrs[inter_sort][i], fmt='o', color=colors[inter_sort][i], ecolor='gray', elinewidth=3, capsize=0, label='Basic Criteria', markersize=8)\n            ax[2].errorbar(ratio[ratio_sort][i], recalls[ratio_sort][i], yerr=rerrs[ratio_sort][i], fmt='o', color=colors[ratio_sort][i], ecolor='gray', elinewidth=3, capsize=0, label='Basic Criteria', markersize=8)\n            ax[3].errorbar(rho[rho_sort][i], recalls[rho_sort][i], yerr=rerrs[rho_sort][i], fmt='o', color=colors[rho_sort][i], ecolor='gray', elinewidth=3, capsize=0, label='Basic Criteria', markersize=8)\n        ax[1].set_yticks([])\n        ax[2].set_yticks([])\n        ax[3].set_yticks([])\n        ax[0].plot(intra_x, intra_linfit, 'k--', alpha=0.5, linewidth=3)\n        ax[1].plot(inter_x, inter_linfit, 'k--', alpha=0.5, linewidth=3)\n        ax[2].plot(ratio_x, ratio_linfit, 'k--', alpha=0.5, linewidth=3)\n        ax[3].plot(rho_x, rho_linfit, 'r--', alpha=0.5, linewidth=3)\n\n        ax[0].text(intra_lim[1]-0.7*(intra_lim[1]-intra_lim[0]),ylim[0]+0.05*(ylim[1]-ylim[0]),'Corr: {0:1.2f}'.format(intra_corr), bbox=dict(facecolor='gray', alpha=0.5), fontsize=26)\n        ax[1].text(inter_lim[1]-0.7*(inter_lim[1]-inter_lim[0]),ylim[0]+0.05*(ylim[1]-ylim[0]),'Corr: {0:1.2f}'.format(inter_corr), bbox=dict(facecolor='gray', alpha=0.5), fontsize=26)\n        ax[2].text(ratio_lim[1]-0.7*(ratio_lim[1]-ratio_lim[0]),ylim[0]+0.05*(ylim[1]-ylim[0]),'Corr: {0:1.2f}'.format(ratio_corr), bbox=dict(facecolor='gray', alpha=0.5), fontsize=26)\n        ax[3].text(rho_lim[1]-0.7*(rho_lim[1]-rho_lim[0]),ylim[0]+0.05*(ylim[1]-ylim[0]),'Corr: {0:1.2f}'.format(rho_corr),   bbox=dict(facecolor='red', alpha=0.5), fontsize=26)\n\n        if k==0:\n            ax[0].set_title(r'$\\pi_{intra}$', fontsize=26)\n            ax[1].set_title(r'$\\pi_{inter}$', fontsize=26)\n            ax[2].set_title(r'$\\pi_{ratio}$', fontsize=26)\n            ax[3].set_title(r'$\\rho(\\Phi)$', fontsize=26, color='red')\n        if k==0:\n            ax[0].set_ylabel('CUB200-2011 R@1', fontsize=23)\n        elif k==1:\n            ax[0].set_ylabel('CARS196 R@1', fontsize=23)\n        elif k==2:\n            ax[0].set_ylabel('SOP R@1', fontsize=23)\n        for a in ax.reshape(-1):\n            a.tick_params(axis='both', which='major', labelsize=20)\n            a.tick_params(axis='both', which='minor', labelsize=20)\n            a.set_ylim(ylim)\n    f.set_size_inches(21,15)\n    f.tight_layout()\n\n    f.savefig('comp_metric_relation.pdf')\n    f.savefig('comp_metric_relation.png')\n\nfull_rel_plot()\n\n\n\n\n\n\n\n\n\n\n\n\"\"\"================================================\"\"\"\nimport itertools as it\ncub_corr_mat = np.corrcoef(cub_mets)\nf,ax         = plt.subplots(1,3)\nax[0].imshow(cub_corr_mat, vmin=0, vmax=1, cmap='plasma')\ncorr_x = [0,1,2,3,4,5]\nax[0].set_xticklabels(metric_names)\nax[0].set_yticklabels(metric_names)\nax[0].set_xticks(corr_x)\nax[0].set_yticks(corr_x)\nax[0].set_xlim([-0.5,5.5])\nax[0].set_ylim([-0.5,5.5])\ncs = list(it.product(corr_x, corr_x))\nfor c in cs:\n    ax[0].text(c[0]-0.2, c[1]-0.11, '{0:1.2f}'.format(cub_corr_mat[c[0], c[1]]), fontsize=18)\nax[0].tick_params(axis='both', which='major', labelsize=18)\nax[0].tick_params(axis='both', which='minor', labelsize=18)\ncar_corr_mat = np.corrcoef(car_mets)\nax[1].imshow(car_corr_mat, vmin=0, vmax=1, cmap='plasma')\ncorr_x = [0,1,2,3,4,5]\nax[1].set_xticklabels(metric_names)\nax[1].set_yticklabels(metric_names)\nax[1].set_xticks(corr_x)\nax[1].set_yticks(corr_x)\nax[1].set_xlim([-0.5,5.5])\nax[1].set_ylim([-0.5,5.5])\ncs = list(it.product(corr_x, corr_x))\nfor c in cs:\n    ax[1].text(c[0]-0.2, c[1]-0.11, '{0:1.2f}'.format(car_corr_mat[c[0], c[1]]), fontsize=18)\nax[1].tick_params(axis='both', which='major', labelsize=18)\nax[1].tick_params(axis='both', which='minor', labelsize=18)\nsop_corr_mat = np.corrcoef(sop_mets)\nax[2].imshow(sop_corr_mat, vmin=0, vmax=1, cmap='plasma')\ncorr_x = [0,1,2,3,4,5]\nax[2].set_xticklabels(metric_names)\nax[2].set_yticklabels(metric_names)\nax[2].set_xticks(corr_x)\nax[2].set_yticks(corr_x)\nax[2].set_xlim([-0.5,5.5])\nax[2].set_ylim([-0.5,5.5])\ncs = list(it.product(corr_x, corr_x))\nfor c in cs:\n    ax[2].text(c[0]-0.2, c[1]-0.11, '{0:1.2f}'.format(sop_corr_mat[c[0], c[1]]), fontsize=18)\nax[2].tick_params(axis='both', which='major', labelsize=18)\nax[2].tick_params(axis='both', which='minor', labelsize=18)\nax[0].set_title('CUB200-2011', fontsize=22)\nax[1].set_title('CARS196', fontsize=22)\nax[2].set_title('Stanford Online Products', fontsize=22)\nf.set_size_inches(22,8)\nf.tight_layout()\nf.savefig('metric_correlation_matrix.pdf')\nf.savefig('metric_correlation_matrix.png')\n\n\n\n\n\n\n\n\n\n\n\"\"\"==================================================\"\"\"\n####\nrecallss, valss = [cub_basic_recall, car_basic_recall, sop_basic_recall], [cub_vals, car_vals, sop_vals]\nerrss           = [cub_basic_recall_err, car_basic_recall_err, sop_basic_recall_err]\nnamess          = [cub_basic_names, car_basic_names, sop_basic_names]\nreg_recallss, reg_valss = [cub_reg_recall, car_reg_recall, sop_reg_recall], [cub_vals, car_vals, sop_vals]\nreg_errss           = [cub_reg_recall_err, car_reg_recall_err, sop_reg_recall_err]\nreg_namess          = [cub_reg_names, car_reg_names, sop_reg_names]\n\n####\ndef plot(vals, recalls, errs, names, reg_vals=None, reg_recalls=None, reg_errs=None, reg_names=None, xlab=None, ylab=None, xlim=[0,1], ylim=[0,1], savename=None):\n    from adjustText import adjust_text\n    f, ax = plt.subplots(1)\n    texts = []\n    rho       = np.array([vals[k]['Rho3'][0] for k in names])\n    adj_names = names\n\n    nnames    = []\n    for n in adj_names:\n        nnames.append(name_adjust(n, prep='', app=''))\n    print(nnames)\n    ax.errorbar(rho, recalls, yerr=errs,color='deepskyblue',fmt='o',ecolor='deepskyblue',elinewidth=5,capsize=0,markersize=16,mec='k')\n    recalls = np.array(recalls)\n    for rho_v, rec_v, n in zip(rho, recalls, nnames):\n        r = ax.text(rho_v, rec_v, n, fontsize=17, va='top', ha='left')\n        # r = ax.text(rho_v, rec_v, n, fontsize=15, bbox=dict(facecolor='gray', alpha=0.5), va='left', ha='left')\n        texts.append(r)\n\n    if reg_names is not None:\n        rho       = np.array([vals[k]['Rho3'][0] for k in reg_names])\n        adj_names = ['_'.join(x.split('_')[1:]) for x in reg_names]\n\n        nnames    = []\n        for n in adj_names:\n            nnames.append(name_adjust(n, prep='R-', app=''))\n        ax.errorbar(rho, reg_recalls, yerr=reg_errs,color='orange',fmt='o',ecolor='gray',elinewidth=5,capsize=0,markersize=16,mec='k')\n        for rho_v, rec_v, n in zip(rho, reg_recalls, nnames):\n            r = ax.text(rho_v, rec_v, n, fontsize=17, va='top', ha='left', color='chocolate')\n            texts.append(r)\n    ax.tick_params(axis='both', which='major', labelsize=20)\n    ax.tick_params(axis='both', which='minor', labelsize=20)\n    if xlab is not None:\n        ax.set_xlabel(xlab, fontsize=20)\n    if ylab is not None:\n        ax.set_ylabel(ylab, fontsize=20)\n    ax.set_xlim(xlim)\n    ax.set_ylim(ylim)\n    ax.grid()\n    f.set_size_inches(25,5)\n    f.tight_layout()\n    adjust_text(texts, arrowprops=dict(arrowstyle=\"->\", color='k', lw=1))\n    f.savefig('{}.png'.format(savename))\n    f.savefig('{}.pdf'.format(savename))\n\nplot(valss[0], recallss[0], errss[0], namess[0], reg_valss[0], reg_recallss[0], reg_errss[0], reg_namess[0], xlab=r'$\\rho(\\Phi)$', ylab=r'$CUB200-2011, R@1$', xlim=[0,0.59], ylim=[0.58, 0.66], savename='Detailed_Rel_Recall_Rho_CUB')\nplot(valss[1], recallss[1], errss[1], namess[1], reg_valss[1], reg_recallss[1], reg_errss[1], reg_namess[1], xlab=r'$\\rho(\\Phi)$', ylab=r'$CARS196, R@1$',     xlim=[0,0.59], ylim=[0.7, 0.84], savename='Detailed_Rel_Recall_Rho_CAR')\nplot(valss[2], recallss[2], errss[2], namess[2], reg_valss[2], reg_recallss[2], reg_errss[2], reg_namess[2], xlab=r'$\\rho(\\Phi)$', ylab=r'$SOP, R@1$', xlim=[0,0.59], ylim=[0.67, 0.81], savename='Detailed_Rel_Recall_Rho_SOP')\n\n\n\n\n\n\"\"\"==================================================\"\"\"\n#### First Page Figure\nplt.style.use('seaborn')\ntotal_recall = np.array(cub_basic_recall.tolist() + cub_reg_recall.tolist())\ntotal_err    = np.array(cub_basic_recall_err.tolist() + cub_reg_recall_err.tolist())\ntotal_names  = np.array(cub_basic_names+cub_reg_names)\nsort_idx = np.argsort(total_recall)\nf, ax = plt.subplots(1)\nbasic_label, reg_label = False, False\nfor i,idx in enumerate(sort_idx):\n    if 'reg_' not in total_names[idx]:\n        if basic_label:\n            ax.barh(i,total_recall[idx], xerr=total_err[idx], color='orange', alpha=0.6)\n        else:\n            ax.barh(i,total_recall[idx], xerr=total_err[idx], color='orange', alpha=0.6, label='Basic DML Criteria')\n            basic_label = True\n        ax.text(0.5703,i-0.2,name_adjust(total_names[idx]), fontsize=17)\n    else:\n        if reg_label:\n            ax.barh(i,total_recall[idx], xerr=total_err[idx], color='forestgreen', alpha=0.8)\n        else:\n            ax.barh(i,total_recall[idx], xerr=total_err[idx], color='forestgreen', alpha=0.8, label='Regularized Variant')\n            reg_label = True\n        ax.text(0.5703,i-0.2,name_adjust(total_names[idx], prep='R-'), fontsize=17)\nax.legend(fontsize=20)\nax.set_yticks([])\nax.set_yticklabels([])\nax.set_xticks([0.58, 0.6, 0.62, 0.64])\nax.tick_params(axis='both', which='major', labelsize=22)\nax.tick_params(axis='both', which='minor', labelsize=22)\nax.set_title('CUB200-2011, R@1', fontsize=20)\nax.set_ylim([-0.5,22.5])\nax.set_xlim([0.57, 0.655])\nf.set_size_inches(15,8)\nf.tight_layout()\nf.savefig('FirstPage.png')\nf.savefig('FirstPage.pdf')\n"
  },
  {
    "path": "Sample_Runs/ICML2020_RevisitDML_SampleRuns.sh",
    "content": "python main.py --kernels 6 --source /home/karsten_dl/Dropbox/Projects/Datasets --n_epochs 150 --seed 0 --gpu 1 --bs 112 --samples_per_class 2 --loss npair --batch_mining npair --arch resnet50_frozen\n\n\n\"\"\"============= Baseline Runs --- CUB200-2011 ====================\"\"\"\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUB_Npair --seed 0 --gpu 0 --bs 112 --samples_per_class 2 --loss npair --batch_mining npair --arch resnet50_frozen\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUB_Npair --seed 1 --gpu 0 --bs 112 --samples_per_class 2 --loss npair --batch_mining npair --arch resnet50_frozen\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUB_Npair --seed 2 --gpu 0 --bs 112 --samples_per_class 2 --loss npair --batch_mining npair --arch resnet50_frozen\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUB_Npair --seed 3 --gpu 0 --bs 112 --samples_per_class 2 --loss npair --batch_mining npair --arch resnet50_frozen\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUB_Npair --seed 4 --gpu 0 --bs 112 --samples_per_class 2 --loss npair --batch_mining npair --arch resnet50_frozen\n\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUB_GenLifted --seed 0 --gpu 0 --bs 112 --samples_per_class 2 --loss lifted --batch_mining lifted --arch resnet50_frozen\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUB_GenLifted --seed 1 --gpu 0 --bs 112 --samples_per_class 2 --loss lifted --batch_mining lifted --arch resnet50_frozen\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUB_GenLifted --seed 2 --gpu 0 --bs 112 --samples_per_class 2 --loss lifted --batch_mining lifted --arch resnet50_frozen\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUB_GenLifted --seed 3 --gpu 0 --bs 112 --samples_per_class 2 --loss lifted --batch_mining lifted --arch resnet50_frozen\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUB_GenLifted --seed 4 --gpu 0 --bs 112 --samples_per_class 2 --loss lifted --batch_mining lifted --arch resnet50_frozen\n\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUB_ProxyNCA --seed 0 --gpu 0 --bs 112 --samples_per_class 2 --loss proxynca --arch resnet50_frozen_normalize\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUB_ProxyNCA --seed 1 --gpu 0 --bs 112 --samples_per_class 2 --loss proxynca --arch resnet50_frozen_normalize\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUB_ProxyNCA --seed 2 --gpu 0 --bs 112 --samples_per_class 2 --loss proxynca --arch resnet50_frozen_normalize\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUB_ProxyNCA --seed 3 --gpu 0 --bs 112 --samples_per_class 2 --loss proxynca --arch resnet50_frozen_normalize\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUB_ProxyNCA --seed 4 --gpu 0 --bs 112 --samples_per_class 2 --loss proxynca --arch resnet50_frozen_normalize\n\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUB_Histogram --seed 0 --gpu 0 --bs 112 --samples_per_class 2 --loss histogram --arch resnet50_frozen_normalize --loss_histogram_nbins 65\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUB_Histogram --seed 1 --gpu 0 --bs 112 --samples_per_class 2 --loss histogram --arch resnet50_frozen_normalize --loss_histogram_nbins 65\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUB_Histogram --seed 2 --gpu 0 --bs 112 --samples_per_class 2 --loss histogram --arch resnet50_frozen_normalize --loss_histogram_nbins 65\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUB_Histogram --seed 3 --gpu 0 --bs 112 --samples_per_class 2 --loss histogram --arch resnet50_frozen_normalize --loss_histogram_nbins 65\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUB_Histogram --seed 4 --gpu 0 --bs 112 --samples_per_class 2 --loss histogram --arch resnet50_frozen_normalize --loss_histogram_nbins 65\n\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUB_Contrastive --seed 0 --gpu 0 --bs 112 --samples_per_class 2 --loss contrastive --batch_mining distance --arch resnet50_frozen_normalize\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUB_Contrastive --seed 1 --gpu 0 --bs 112 --samples_per_class 2 --loss contrastive --batch_mining distance --arch resnet50_frozen_normalize\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUB_Contrastive --seed 2 --gpu 0 --bs 112 --samples_per_class 2 --loss contrastive --batch_mining distance --arch resnet50_frozen_normalize\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUB_Contrastive --seed 3 --gpu 0 --bs 112 --samples_per_class 2 --loss contrastive --batch_mining distance --arch resnet50_frozen_normalize\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUB_Contrastive --seed 4 --gpu 0 --bs 112 --samples_per_class 2 --loss contrastive --batch_mining distance --arch resnet50_frozen_normalize\n\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUB_SoftTriple --seed 0 --gpu 0 --bs 112 --samples_per_class 2 --loss softtriplet --arch resnet50_frozen_normalize\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUB_SoftTriple --seed 1 --gpu 0 --bs 112 --samples_per_class 2 --loss softtriplet --arch resnet50_frozen_normalize\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUB_SoftTriple --seed 2 --gpu 0 --bs 112 --samples_per_class 2 --loss softtriplet --arch resnet50_frozen_normalize\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUB_SoftTriple --seed 3 --gpu 0 --bs 112 --samples_per_class 2 --loss softtriplet --arch resnet50_frozen_normalize\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUB_SoftTriple --seed 4 --gpu 0 --bs 112 --samples_per_class 2 --loss softtriplet --arch resnet50_frozen_normalize\n\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUB_Angular --seed 0 --gpu 0 --bs 112 --samples_per_class 2 --loss angular --batch_mining npair --arch resnet50_frozen\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUB_Angular --seed 1 --gpu 0 --bs 112 --samples_per_class 2 --loss angular --batch_mining npair --arch resnet50_frozen\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUB_Angular --seed 2 --gpu 0 --bs 112 --samples_per_class 2 --loss angular --batch_mining npair --arch resnet50_frozen\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUB_Angular --seed 3 --gpu 0 --bs 112 --samples_per_class 2 --loss angular --batch_mining npair --arch resnet50_frozen\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUB_Angular --seed 4 --gpu 0 --bs 112 --samples_per_class 2 --loss angular --batch_mining npair --arch resnet50_frozen\n\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUB_ArcFace --seed 0 --gpu 0 --bs 112 --samples_per_class 2 --loss arcface --arch resnet50_frozen_normalize\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUB_ArcFace --seed 1 --gpu 0 --bs 112 --samples_per_class 2 --loss arcface --arch resnet50_frozen_normalize\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUB_ArcFace --seed 2 --gpu 0 --bs 112 --samples_per_class 2 --loss arcface --arch resnet50_frozen_normalize\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUB_ArcFace --seed 3 --gpu 0 --bs 112 --samples_per_class 2 --loss arcface --arch resnet50_frozen_normalize\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUB_ArcFace --seed 4 --gpu 0 --bs 112 --samples_per_class 2 --loss arcface --arch resnet50_frozen_normalize\n\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUB_Triplet_Random --seed 0 --gpu 0 --bs 112 --samples_per_class 2 --loss triplet --batch_mining random --arch resnet50_frozen_normalize\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUB_Triplet_Random --seed 1 --gpu 0 --bs 112 --samples_per_class 2 --loss triplet --batch_mining random --arch resnet50_frozen_normalize\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUB_Triplet_Random --seed 2 --gpu 0 --bs 112 --samples_per_class 2 --loss triplet --batch_mining random --arch resnet50_frozen_normalize\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUB_Triplet_Random --seed 3 --gpu 0 --bs 112 --samples_per_class 2 --loss triplet --batch_mining random --arch resnet50_frozen_normalize\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUB_Triplet_Random --seed 4 --gpu 0 --bs 112 --samples_per_class 2 --loss triplet --batch_mining random --arch resnet50_frozen_normalize\n\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUB_Triplet_Semihard --seed 0 --gpu 0 --bs 112 --samples_per_class 2 --loss triplet --batch_mining semihard --arch resnet50_frozen_normalize\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUB_Triplet_Semihard --seed 1 --gpu 0 --bs 112 --samples_per_class 2 --loss triplet --batch_mining semihard --arch resnet50_frozen_normalize\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUB_Triplet_Semihard --seed 2 --gpu 0 --bs 112 --samples_per_class 2 --loss triplet --batch_mining semihard --arch resnet50_frozen_normalize\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUB_Triplet_Semihard --seed 3 --gpu 0 --bs 112 --samples_per_class 2 --loss triplet --batch_mining semihard --arch resnet50_frozen_normalize\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUB_Triplet_Semihard --seed 4 --gpu 0 --bs 112 --samples_per_class 2 --loss triplet --batch_mining semihard --arch resnet50_frozen_normalize\n\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUB_Triplet_Softhard --seed 0 --gpu 0 --bs 112 --samples_per_class 2 --loss triplet --batch_mining softhard --arch resnet50_frozen_normalize\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUB_Triplet_Softhard --seed 1 --gpu 0 --bs 112 --samples_per_class 2 --loss triplet --batch_mining softhard --arch resnet50_frozen_normalize\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUB_Triplet_Softhard --seed 2 --gpu 0 --bs 112 --samples_per_class 2 --loss triplet --batch_mining softhard --arch resnet50_frozen_normalize\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUB_Triplet_Softhard --seed 3 --gpu 0 --bs 112 --samples_per_class 2 --loss triplet --batch_mining softhard --arch resnet50_frozen_normalize\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUB_Triplet_Softhard --seed 4 --gpu 0 --bs 112 --samples_per_class 2 --loss triplet --batch_mining softhard --arch resnet50_frozen_normalize\n\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUB_Triplet_Distance --seed 0 --gpu 0 --bs 112 --samples_per_class 2 --loss triplet --batch_mining distance --arch resnet50_frozen_normalize\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUB_Triplet_Distance --seed 1 --gpu 0 --bs 112 --samples_per_class 2 --loss triplet --batch_mining distance --arch resnet50_frozen_normalize\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUB_Triplet_Distance --seed 2 --gpu 0 --bs 112 --samples_per_class 2 --loss triplet --batch_mining distance --arch resnet50_frozen_normalize\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUB_Triplet_Distance --seed 3 --gpu 0 --bs 112 --samples_per_class 2 --loss triplet --batch_mining distance --arch resnet50_frozen_normalize\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUB_Triplet_Distance --seed 4 --gpu 0 --bs 112 --samples_per_class 2 --loss triplet --batch_mining distance --arch resnet50_frozen_normalize\n\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUB_Quadruplet_Distance --seed 0 --gpu 0 --bs 112 --samples_per_class 2 --loss quadruplet --batch_mining distance --arch resnet50_frozen_normalize\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUB_Quadruplet_Distance --seed 1 --gpu 0 --bs 112 --samples_per_class 2 --loss quadruplet --batch_mining distance --arch resnet50_frozen_normalize\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUB_Quadruplet_Distance --seed 2 --gpu 0 --bs 112 --samples_per_class 2 --loss quadruplet --batch_mining distance --arch resnet50_frozen_normalize\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUB_Quadruplet_Distance --seed 3 --gpu 0 --bs 112 --samples_per_class 2 --loss quadruplet --batch_mining distance --arch resnet50_frozen_normalize\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUB_Quadruplet_Distance --seed 4 --gpu 0 --bs 112 --samples_per_class 2 --loss quadruplet --batch_mining distance --arch resnet50_frozen_normalize\n\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUB_Margin_b06_Distance --loss_margin_beta 0.6 --seed 0 --gpu 0 --bs 112 --samples_per_class 2 --loss margin --batch_mining distance --arch resnet50_frozen_normalize\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUB_Margin_b06_Distance --loss_margin_beta 0.6 --seed 1 --gpu 0 --bs 112 --samples_per_class 2 --loss margin --batch_mining distance --arch resnet50_frozen_normalize\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUB_Margin_b06_Distance --loss_margin_beta 0.6 --seed 2 --gpu 0 --bs 112 --samples_per_class 2 --loss margin --batch_mining distance --arch resnet50_frozen_normalize\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUB_Margin_b06_Distance --loss_margin_beta 0.6 --seed 3 --gpu 0 --bs 112 --samples_per_class 2 --loss margin --batch_mining distance --arch resnet50_frozen_normalize\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUB_Margin_b06_Distance --loss_margin_beta 0.6 --seed 4 --gpu 0 --bs 112 --samples_per_class 2 --loss margin --batch_mining distance --arch resnet50_frozen_normalize\n\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUB_Margin_b12_Distance --seed 0 --gpu 0 --bs 112 --samples_per_class 2 --loss margin --batch_mining distance --arch resnet50_frozen_normalize\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUB_Margin_b12_Distance --seed 1 --gpu 0 --bs 112 --samples_per_class 2 --loss margin --batch_mining distance --arch resnet50_frozen_normalize\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUB_Margin_b12_Distance --seed 2 --gpu 0 --bs 112 --samples_per_class 2 --loss margin --batch_mining distance --arch resnet50_frozen_normalize\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUB_Margin_b12_Distance --seed 3 --gpu 0 --bs 112 --samples_per_class 2 --loss margin --batch_mining distance --arch resnet50_frozen_normalize\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUB_Margin_b12_Distance --seed 4 --gpu 0 --bs 112 --samples_per_class 2 --loss margin --batch_mining distance --arch resnet50_frozen_normalize\n\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUB_SNR_Distance  --seed 0 --gpu 0 --bs 112 --samples_per_class 2 --loss snr --batch_mining distance --arch resnet50_frozen_normalize\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUB_SNR_Distance  --seed 1 --gpu 0 --bs 112 --samples_per_class 2 --loss snr --batch_mining distance --arch resnet50_frozen_normalize\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUB_SNR_Distance  --seed 2 --gpu 0 --bs 112 --samples_per_class 2 --loss snr --batch_mining distance --arch resnet50_frozen_normalize\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUB_SNR_Distance  --seed 3 --gpu 0 --bs 112 --samples_per_class 2 --loss snr --batch_mining distance --arch resnet50_frozen_normalize\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUB_SNR_Distance  --seed 4 --gpu 0 --bs 112 --samples_per_class 2 --loss snr --batch_mining distance --arch resnet50_frozen_normalize\n\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUB_MS --seed 0 --gpu 0 --bs 112 --samples_per_class 2 --loss multisimilarity --arch resnet50_frozen_normalize\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUB_MS --seed 1 --gpu 0 --bs 112 --samples_per_class 2 --loss multisimilarity --arch resnet50_frozen_normalize\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUB_MS --seed 2 --gpu 0 --bs 112 --samples_per_class 2 --loss multisimilarity --arch resnet50_frozen_normalize\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUB_MS --seed 3 --gpu 0 --bs 112 --samples_per_class 2 --loss multisimilarity --arch resnet50_frozen_normalize\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUB_MS --seed 4 --gpu 0 --bs 112 --samples_per_class 2 --loss multisimilarity --arch resnet50_frozen_normalize\n\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUB_Softmax --seed 0 --gpu 0 --bs 112 --samples_per_class 2 --loss softmax --batch_mining distance --arch resnet50_frozen_normalize\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUB_Softmax --seed 1 --gpu 0 --bs 112 --samples_per_class 2 --loss softmax --batch_mining distance --arch resnet50_frozen_normalize\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUB_Softmax --seed 2 --gpu 0 --bs 112 --samples_per_class 2 --loss softmax --batch_mining distance --arch resnet50_frozen_normalize\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUB_Softmax --seed 3 --gpu 0 --bs 112 --samples_per_class 2 --loss softmax --batch_mining distance --arch resnet50_frozen_normalize\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUB_Softmax --seed 4 --gpu 0 --bs 112 --samples_per_class 2 --loss softmax --batch_mining distance --arch resnet50_frozen_normalize\n\n\n### Specturm-Regularized Ranking Losses\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUBreg_Contrastive --seed 0 --gpu 0 --bs 112 --samples_per_class 2 --loss contrastive --batch_mining rho_distance --arch resnet50_frozen_normalize --miner_rho_distance_cp 0.4\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUBreg_Contrastive --seed 1 --gpu 0 --bs 112 --samples_per_class 2 --loss contrastive --batch_mining rho_distance --arch resnet50_frozen_normalize --miner_rho_distance_cp 0.4\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUBreg_Contrastive --seed 2 --gpu 0 --bs 112 --samples_per_class 2 --loss contrastive --batch_mining rho_distance --arch resnet50_frozen_normalize --miner_rho_distance_cp 0.4\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUBreg_Contrastive --seed 3 --gpu 0 --bs 112 --samples_per_class 2 --loss contrastive --batch_mining rho_distance --arch resnet50_frozen_normalize --miner_rho_distance_cp 0.4\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUBreg_Contrastive --seed 4 --gpu 0 --bs 112 --samples_per_class 2 --loss contrastive --batch_mining rho_distance --arch resnet50_frozen_normalize --miner_rho_distance_cp 0.4\n\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUBreg_Margin_b06_Distance --loss_margin_beta 0.6 --seed 0 --gpu 0 --bs 112 --samples_per_class 2 --loss margin --batch_mining rho_distance --arch resnet50_frozen_normalize --miner_rho_distance_cp 0.4\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUBreg_Margin_b06_Distance --loss_margin_beta 0.6 --seed 1 --gpu 0 --bs 112 --samples_per_class 2 --loss margin --batch_mining rho_distance --arch resnet50_frozen_normalize --miner_rho_distance_cp 0.4\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUBreg_Margin_b06_Distance --loss_margin_beta 0.6 --seed 2 --gpu 0 --bs 112 --samples_per_class 2 --loss margin --batch_mining rho_distance --arch resnet50_frozen_normalize --miner_rho_distance_cp 0.4\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUBreg_Margin_b06_Distance --loss_margin_beta 0.6 --seed 3 --gpu 0 --bs 112 --samples_per_class 2 --loss margin --batch_mining rho_distance --arch resnet50_frozen_normalize --miner_rho_distance_cp 0.4\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUBreg_Margin_b06_Distance --loss_margin_beta 0.6 --seed 4 --gpu 0 --bs 112 --samples_per_class 2 --loss margin --batch_mining rho_distance --arch resnet50_frozen_normalize --miner_rho_distance_cp 0.4\n\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUBreg_Margin_b12_Distance_3 --seed 0 --gpu 0 --bs 112 --samples_per_class 2 --loss margin --batch_mining rho_distance --arch resnet50_frozen_normalize --miner_rho_distance_cp 0.35\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUBreg_Margin_b12_Distance_3 --seed 1 --gpu 0 --bs 112 --samples_per_class 2 --loss margin --batch_mining rho_distance --arch resnet50_frozen_normalize --miner_rho_distance_cp 0.35\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUBreg_Margin_b12_Distance_3 --seed 2 --gpu 0 --bs 112 --samples_per_class 2 --loss margin --batch_mining rho_distance --arch resnet50_frozen_normalize --miner_rho_distance_cp 0.35\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUBreg_Margin_b12_Distance_3 --seed 3 --gpu 0 --bs 112 --samples_per_class 2 --loss margin --batch_mining rho_distance --arch resnet50_frozen_normalize --miner_rho_distance_cp 0.35\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUBreg_Margin_b12_Distance_3 --seed 4 --gpu 0 --bs 112 --samples_per_class 2 --loss margin --batch_mining rho_distance --arch resnet50_frozen_normalize --miner_rho_distance_cp 0.35\n\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUBreg_Triplet_Distance_3 --seed 0 --gpu 0 --bs 112 --samples_per_class 2 --loss triplet --batch_mining rho_distance --arch resnet50_frozen_normalize --miner_rho_distance_cp 0.4\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUBreg_Triplet_Distance_3 --seed 1 --gpu 0 --bs 112 --samples_per_class 2 --loss triplet --batch_mining rho_distance --arch resnet50_frozen_normalize --miner_rho_distance_cp 0.4\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUBreg_Triplet_Distance_3 --seed 2 --gpu 0 --bs 112 --samples_per_class 2 --loss triplet --batch_mining rho_distance --arch resnet50_frozen_normalize --miner_rho_distance_cp 0.4\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUBreg_Triplet_Distance_3 --seed 3 --gpu 0 --bs 112 --samples_per_class 2 --loss triplet --batch_mining rho_distance --arch resnet50_frozen_normalize --miner_rho_distance_cp 0.4\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUBreg_Triplet_Distance_3 --seed 4 --gpu 0 --bs 112 --samples_per_class 2 --loss triplet --batch_mining rho_distance --arch resnet50_frozen_normalize --miner_rho_distance_cp 0.4\n\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUBreg_SNR_Distance _3 --seed 0 --gpu 0 --bs 112 --samples_per_class 2 --loss snr --batch_mining rho_distance --arch resnet50_frozen_normalize --miner_rho_distance_cp 0.3\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUBreg_SNR_Distance _3 --seed 1 --gpu 0 --bs 112 --samples_per_class 2 --loss snr --batch_mining rho_distance --arch resnet50_frozen_normalize --miner_rho_distance_cp 0.3\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUBreg_SNR_Distance _3 --seed 2 --gpu 0 --bs 112 --samples_per_class 2 --loss snr --batch_mining rho_distance --arch resnet50_frozen_normalize --miner_rho_distance_cp 0.3\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUBreg_SNR_Distance _3 --seed 3 --gpu 0 --bs 112 --samples_per_class 2 --loss snr --batch_mining rho_distance --arch resnet50_frozen_normalize --miner_rho_distance_cp 0.3\npython main.py --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CUBreg_SNR_Distance _3 --seed 4 --gpu 0 --bs 112 --samples_per_class 2 --loss snr --batch_mining rho_distance --arch resnet50_frozen_normalize --miner_rho_distance_cp 0.3\n\n\n\n\n\n\"\"\"============= Baseline Runs --- CARS196 ====================\"\"\"\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARS_Npair --seed 0 --gpu 0 --bs 112 --samples_per_class 2 --loss npair --batch_mining npair --arch resnet50_frozen\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARS_Npair --seed 1 --gpu 0 --bs 112 --samples_per_class 2 --loss npair --batch_mining npair --arch resnet50_frozen\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARS_Npair --seed 2 --gpu 0 --bs 112 --samples_per_class 2 --loss npair --batch_mining npair --arch resnet50_frozen\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARS_Npair --seed 3 --gpu 0 --bs 112 --samples_per_class 2 --loss npair --batch_mining npair --arch resnet50_frozen\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARS_Npair --seed 4 --gpu 0 --bs 112 --samples_per_class 2 --loss npair --batch_mining npair --arch resnet50_frozen\n\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARS_GenLifted --seed 0 --gpu 0 --bs 112 --samples_per_class 2 --loss lifted --batch_mining lifted --arch resnet50_frozen\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARS_GenLifted --seed 1 --gpu 0 --bs 112 --samples_per_class 2 --loss lifted --batch_mining lifted --arch resnet50_frozen\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARS_GenLifted --seed 2 --gpu 0 --bs 112 --samples_per_class 2 --loss lifted --batch_mining lifted --arch resnet50_frozen\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARS_GenLifted --seed 3 --gpu 0 --bs 112 --samples_per_class 2 --loss lifted --batch_mining lifted --arch resnet50_frozen\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARS_GenLifted --seed 4 --gpu 0 --bs 112 --samples_per_class 2 --loss lifted --batch_mining lifted --arch resnet50_frozen\n\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARS_ProxyNCA --seed 0 --gpu 0 --bs 112 --samples_per_class 2 --loss proxynca --arch resnet50_frozen_normalize\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARS_ProxyNCA --seed 1 --gpu 0 --bs 112 --samples_per_class 2 --loss proxynca --arch resnet50_frozen_normalize\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARS_ProxyNCA --seed 2 --gpu 0 --bs 112 --samples_per_class 2 --loss proxynca --arch resnet50_frozen_normalize\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARS_ProxyNCA --seed 3 --gpu 0 --bs 112 --samples_per_class 2 --loss proxynca --arch resnet50_frozen_normalize\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARS_ProxyNCA --seed 4 --gpu 0 --bs 112 --samples_per_class 2 --loss proxynca --arch resnet50_frozen_normalize\n\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARS_Histogram --seed 0 --gpu 0 --bs 112 --samples_per_class 2 --loss histogram --arch resnet50_frozen_normalize --loss_histogram_nbins 65\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARS_Histogram --seed 1 --gpu 0 --bs 112 --samples_per_class 2 --loss histogram --arch resnet50_frozen_normalize --loss_histogram_nbins 65\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARS_Histogram --seed 2 --gpu 0 --bs 112 --samples_per_class 2 --loss histogram --arch resnet50_frozen_normalize --loss_histogram_nbins 65\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARS_Histogram --seed 3 --gpu 0 --bs 112 --samples_per_class 2 --loss histogram --arch resnet50_frozen_normalize --loss_histogram_nbins 65\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARS_Histogram --seed 4 --gpu 0 --bs 112 --samples_per_class 2 --loss histogram --arch resnet50_frozen_normalize --loss_histogram_nbins 65\n\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARS_Contrastive --seed 0 --gpu 0 --bs 112 --samples_per_class 2 --loss contrastive --batch_mining distance --arch resnet50_frozen_normalize\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARS_Contrastive --seed 1 --gpu 0 --bs 112 --samples_per_class 2 --loss contrastive --batch_mining distance --arch resnet50_frozen_normalize\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARS_Contrastive --seed 2 --gpu 0 --bs 112 --samples_per_class 2 --loss contrastive --batch_mining distance --arch resnet50_frozen_normalize\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARS_Contrastive --seed 3 --gpu 0 --bs 112 --samples_per_class 2 --loss contrastive --batch_mining distance --arch resnet50_frozen_normalize\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARS_Contrastive --seed 4 --gpu 0 --bs 112 --samples_per_class 2 --loss contrastive --batch_mining distance --arch resnet50_frozen_normalize\n\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARS_SoftTriple --seed 0 --gpu 0 --bs 112 --samples_per_class 2 --loss softtriplet --arch resnet50_frozen_normalize\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARS_SoftTriple --seed 1 --gpu 0 --bs 112 --samples_per_class 2 --loss softtriplet --arch resnet50_frozen_normalize\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARS_SoftTriple --seed 2 --gpu 0 --bs 112 --samples_per_class 2 --loss softtriplet --arch resnet50_frozen_normalize\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARS_SoftTriple --seed 3 --gpu 0 --bs 112 --samples_per_class 2 --loss softtriplet --arch resnet50_frozen_normalize\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARS_SoftTriple --seed 4 --gpu 0 --bs 112 --samples_per_class 2 --loss softtriplet --arch resnet50_frozen_normalize\n\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARS_Angular --seed 0 --gpu 0 --bs 112 --samples_per_class 2 --loss angular --batch_mining npair --arch resnet50_frozen\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARS_Angular --seed 1 --gpu 0 --bs 112 --samples_per_class 2 --loss angular --batch_mining npair --arch resnet50_frozen\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARS_Angular --seed 2 --gpu 0 --bs 112 --samples_per_class 2 --loss angular --batch_mining npair --arch resnet50_frozen\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARS_Angular --seed 3 --gpu 0 --bs 112 --samples_per_class 2 --loss angular --batch_mining npair --arch resnet50_frozen\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARS_Angular --seed 4 --gpu 0 --bs 112 --samples_per_class 2 --loss angular --batch_mining npair --arch resnet50_frozen\n\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARS_ArcFace --seed 0 --gpu 0 --bs 112 --samples_per_class 2 --loss arcface --arch resnet50_frozen_normalize\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARS_ArcFace --seed 1 --gpu 0 --bs 112 --samples_per_class 2 --loss arcface --arch resnet50_frozen_normalize\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARS_ArcFace --seed 2 --gpu 0 --bs 112 --samples_per_class 2 --loss arcface --arch resnet50_frozen_normalize\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARS_ArcFace --seed 3 --gpu 0 --bs 112 --samples_per_class 2 --loss arcface --arch resnet50_frozen_normalize\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARS_ArcFace --seed 4 --gpu 0 --bs 112 --samples_per_class 2 --loss arcface --arch resnet50_frozen_normalize\n\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARS_Triplet_Random --seed 0 --gpu 0 --bs 112 --samples_per_class 2 --loss triplet --batch_mining random --arch resnet50_frozen_normalize\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARS_Triplet_Random --seed 1 --gpu 0 --bs 112 --samples_per_class 2 --loss triplet --batch_mining random --arch resnet50_frozen_normalize\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARS_Triplet_Random --seed 2 --gpu 0 --bs 112 --samples_per_class 2 --loss triplet --batch_mining random --arch resnet50_frozen_normalize\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARS_Triplet_Random --seed 3 --gpu 0 --bs 112 --samples_per_class 2 --loss triplet --batch_mining random --arch resnet50_frozen_normalize\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARS_Triplet_Random --seed 4 --gpu 0 --bs 112 --samples_per_class 2 --loss triplet --batch_mining random --arch resnet50_frozen_normalize\n\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARS_Triplet_Semihard --seed 0 --gpu 0 --bs 112 --samples_per_class 2 --loss triplet --batch_mining semihard --arch resnet50_frozen_normalize\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARS_Triplet_Semihard --seed 1 --gpu 0 --bs 112 --samples_per_class 2 --loss triplet --batch_mining semihard --arch resnet50_frozen_normalize\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARS_Triplet_Semihard --seed 2 --gpu 0 --bs 112 --samples_per_class 2 --loss triplet --batch_mining semihard --arch resnet50_frozen_normalize\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARS_Triplet_Semihard --seed 3 --gpu 0 --bs 112 --samples_per_class 2 --loss triplet --batch_mining semihard --arch resnet50_frozen_normalize\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARS_Triplet_Semihard --seed 4 --gpu 0 --bs 112 --samples_per_class 2 --loss triplet --batch_mining semihard --arch resnet50_frozen_normalize\n\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARS_Triplet_Softhard --seed 0 --gpu 0 --bs 112 --samples_per_class 2 --loss triplet --batch_mining softhard --arch resnet50_frozen_normalize\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARS_Triplet_Softhard --seed 1 --gpu 0 --bs 112 --samples_per_class 2 --loss triplet --batch_mining softhard --arch resnet50_frozen_normalize\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARS_Triplet_Softhard --seed 2 --gpu 0 --bs 112 --samples_per_class 2 --loss triplet --batch_mining softhard --arch resnet50_frozen_normalize\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARS_Triplet_Softhard --seed 3 --gpu 0 --bs 112 --samples_per_class 2 --loss triplet --batch_mining softhard --arch resnet50_frozen_normalize\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARS_Triplet_Softhard --seed 4 --gpu 0 --bs 112 --samples_per_class 2 --loss triplet --batch_mining softhard --arch resnet50_frozen_normalize\n\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARS_Triplet_Distance --seed 0 --gpu 0 --bs 112 --samples_per_class 2 --loss triplet --batch_mining distance --arch resnet50_frozen_normalize\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARS_Triplet_Distance --seed 1 --gpu 0 --bs 112 --samples_per_class 2 --loss triplet --batch_mining distance --arch resnet50_frozen_normalize\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARS_Triplet_Distance --seed 2 --gpu 0 --bs 112 --samples_per_class 2 --loss triplet --batch_mining distance --arch resnet50_frozen_normalize\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARS_Triplet_Distance --seed 3 --gpu 0 --bs 112 --samples_per_class 2 --loss triplet --batch_mining distance --arch resnet50_frozen_normalize\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARS_Triplet_Distance --seed 4 --gpu 0 --bs 112 --samples_per_class 2 --loss triplet --batch_mining distance --arch resnet50_frozen_normalize\n\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARS_Quadruplet_Distance --seed 0 --gpu 0 --bs 112 --samples_per_class 2 --loss quadruplet --batch_mining distance --arch resnet50_frozen_normalize\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARS_Quadruplet_Distance --seed 1 --gpu 0 --bs 112 --samples_per_class 2 --loss quadruplet --batch_mining distance --arch resnet50_frozen_normalize\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARS_Quadruplet_Distance --seed 2 --gpu 0 --bs 112 --samples_per_class 2 --loss quadruplet --batch_mining distance --arch resnet50_frozen_normalize\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARS_Quadruplet_Distance --seed 3 --gpu 0 --bs 112 --samples_per_class 2 --loss quadruplet --batch_mining distance --arch resnet50_frozen_normalize\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARS_Quadruplet_Distance --seed 4 --gpu 0 --bs 112 --samples_per_class 2 --loss quadruplet --batch_mining distance --arch resnet50_frozen_normalize\n\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARS_Margin_b06_Distance --loss_margin_beta 0.6 --seed 0 --gpu 0 --bs 112 --samples_per_class 2 --loss margin --batch_mining distance --arch resnet50_frozen_normalize\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARS_Margin_b06_Distance --loss_margin_beta 0.6 --seed 1 --gpu 0 --bs 112 --samples_per_class 2 --loss margin --batch_mining distance --arch resnet50_frozen_normalize\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARS_Margin_b06_Distance --loss_margin_beta 0.6 --seed 2 --gpu 0 --bs 112 --samples_per_class 2 --loss margin --batch_mining distance --arch resnet50_frozen_normalize\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARS_Margin_b06_Distance --loss_margin_beta 0.6 --seed 3 --gpu 0 --bs 112 --samples_per_class 2 --loss margin --batch_mining distance --arch resnet50_frozen_normalize\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARS_Margin_b06_Distance --loss_margin_beta 0.6 --seed 4 --gpu 0 --bs 112 --samples_per_class 2 --loss margin --batch_mining distance --arch resnet50_frozen_normalize\n\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARS_Margin_b12_Distance --seed 0 --gpu 0 --bs 112 --samples_per_class 2 --loss margin --batch_mining distance --arch resnet50_frozen_normalize\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARS_Margin_b12_Distance --seed 1 --gpu 0 --bs 112 --samples_per_class 2 --loss margin --batch_mining distance --arch resnet50_frozen_normalize\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARS_Margin_b12_Distance --seed 2 --gpu 0 --bs 112 --samples_per_class 2 --loss margin --batch_mining distance --arch resnet50_frozen_normalize\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARS_Margin_b12_Distance --seed 3 --gpu 0 --bs 112 --samples_per_class 2 --loss margin --batch_mining distance --arch resnet50_frozen_normalize\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARS_Margin_b12_Distance --seed 4 --gpu 0 --bs 112 --samples_per_class 2 --loss margin --batch_mining distance --arch resnet50_frozen_normalize\n\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARS_SNR_Distance  --seed 0 --gpu 0 --bs 112 --samples_per_class 2 --loss snr --batch_mining distance --arch resnet50_frozen_normalize\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARS_SNR_Distance  --seed 1 --gpu 0 --bs 112 --samples_per_class 2 --loss snr --batch_mining distance --arch resnet50_frozen_normalize\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARS_SNR_Distance  --seed 2 --gpu 0 --bs 112 --samples_per_class 2 --loss snr --batch_mining distance --arch resnet50_frozen_normalize\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARS_SNR_Distance  --seed 3 --gpu 0 --bs 112 --samples_per_class 2 --loss snr --batch_mining distance --arch resnet50_frozen_normalize\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARS_SNR_Distance  --seed 4 --gpu 0 --bs 112 --samples_per_class 2 --loss snr --batch_mining distance --arch resnet50_frozen_normalize\n\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARS_MS --seed 0 --gpu 0 --bs 112 --samples_per_class 2 --loss multisimilarity --arch resnet50_frozen_normalize\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARS_MS --seed 1 --gpu 0 --bs 112 --samples_per_class 2 --loss multisimilarity --arch resnet50_frozen_normalize\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARS_MS --seed 2 --gpu 0 --bs 112 --samples_per_class 2 --loss multisimilarity --arch resnet50_frozen_normalize\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARS_MS --seed 3 --gpu 0 --bs 112 --samples_per_class 2 --loss multisimilarity --arch resnet50_frozen_normalize\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARS_MS --seed 4 --gpu 0 --bs 112 --samples_per_class 2 --loss multisimilarity --arch resnet50_frozen_normalize\n\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARS_Softmax --seed 0 --gpu 0 --bs 112 --samples_per_class 2 --loss softmax --batch_mining distance --arch resnet50_frozen_normalize\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARS_Softmax --seed 1 --gpu 0 --bs 112 --samples_per_class 2 --loss softmax --batch_mining distance --arch resnet50_frozen_normalize\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARS_Softmax --seed 2 --gpu 0 --bs 112 --samples_per_class 2 --loss softmax --batch_mining distance --arch resnet50_frozen_normalize\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARS_Softmax --seed 3 --gpu 0 --bs 112 --samples_per_class 2 --loss softmax --batch_mining distance --arch resnet50_frozen_normalize\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARS_Softmax --seed 4 --gpu 0 --bs 112 --samples_per_class 2 --loss softmax --batch_mining distance --arch resnet50_frozen_normalize\n\n\n\n### Specturm-Regularized Ranking Losses\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARreg_Contrastive --seed 0 --gpu 0 --bs 112 --samples_per_class 2 --loss contrastive --batch_mining rho_distance --arch resnet50_frozen_normalize --miner_rho_distance_cp 0.35\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARreg_Contrastive --seed 1 --gpu 0 --bs 112 --samples_per_class 2 --loss contrastive --batch_mining rho_distance --arch resnet50_frozen_normalize --miner_rho_distance_cp 0.35\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARreg_Contrastive --seed 2 --gpu 0 --bs 112 --samples_per_class 2 --loss contrastive --batch_mining rho_distance --arch resnet50_frozen_normalize --miner_rho_distance_cp 0.35\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARreg_Contrastive --seed 3 --gpu 0 --bs 112 --samples_per_class 2 --loss contrastive --batch_mining rho_distance --arch resnet50_frozen_normalize --miner_rho_distance_cp 0.35\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARreg_Contrastive --seed 4 --gpu 0 --bs 112 --samples_per_class 2 --loss contrastive --batch_mining rho_distance --arch resnet50_frozen_normalize --miner_rho_distance_cp 0.35\n\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARreg_Margin_b06_Distance --loss_margin_beta 0.6 --seed 0 --gpu 0 --bs 112 --samples_per_class 2 --loss margin --batch_mining rho_distance --arch resnet50_frozen_normalize --miner_rho_distance_cp 0.35\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARreg_Margin_b06_Distance --loss_margin_beta 0.6 --seed 1 --gpu 0 --bs 112 --samples_per_class 2 --loss margin --batch_mining rho_distance --arch resnet50_frozen_normalize --miner_rho_distance_cp 0.35\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARreg_Margin_b06_Distance --loss_margin_beta 0.6 --seed 2 --gpu 0 --bs 112 --samples_per_class 2 --loss margin --batch_mining rho_distance --arch resnet50_frozen_normalize --miner_rho_distance_cp 0.35\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARreg_Margin_b06_Distance --loss_margin_beta 0.6 --seed 3 --gpu 0 --bs 112 --samples_per_class 2 --loss margin --batch_mining rho_distance --arch resnet50_frozen_normalize --miner_rho_distance_cp 0.35\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARreg_Margin_b06_Distance --loss_margin_beta 0.6 --seed 4 --gpu 0 --bs 112 --samples_per_class 2 --loss margin --batch_mining rho_distance --arch resnet50_frozen_normalize --miner_rho_distance_cp 0.35\n\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARreg_Margin_b12_Distance --seed 0 --gpu 0 --bs 112 --samples_per_class 2 --loss margin --batch_mining rho_distance --arch resnet50_frozen_normalize --miner_rho_distance_cp 0.35\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARreg_Margin_b12_Distance --seed 1 --gpu 0 --bs 112 --samples_per_class 2 --loss margin --batch_mining rho_distance --arch resnet50_frozen_normalize --miner_rho_distance_cp 0.35\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARreg_Margin_b12_Distance --seed 2 --gpu 0 --bs 112 --samples_per_class 2 --loss margin --batch_mining rho_distance --arch resnet50_frozen_normalize --miner_rho_distance_cp 0.35\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARreg_Margin_b12_Distance --seed 3 --gpu 0 --bs 112 --samples_per_class 2 --loss margin --batch_mining rho_distance --arch resnet50_frozen_normalize --miner_rho_distance_cp 0.35\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARreg_Margin_b12_Distance --seed 4 --gpu 0 --bs 112 --samples_per_class 2 --loss margin --batch_mining rho_distance --arch resnet50_frozen_normalize --miner_rho_distance_cp 0.35\n\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARreg_Triplet_Distance --seed 0 --gpu 0 --bs 112 --samples_per_class 2 --loss triplet --batch_mining rho_distance --arch resnet50_frozen_normalize --miner_rho_distance_cp 0.35\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARreg_Triplet_Distance --seed 1 --gpu 0 --bs 112 --samples_per_class 2 --loss triplet --batch_mining rho_distance --arch resnet50_frozen_normalize --miner_rho_distance_cp 0.35\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARreg_Triplet_Distance --seed 2 --gpu 0 --bs 112 --samples_per_class 2 --loss triplet --batch_mining rho_distance --arch resnet50_frozen_normalize --miner_rho_distance_cp 0.35\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARreg_Triplet_Distance --seed 3 --gpu 0 --bs 112 --samples_per_class 2 --loss triplet --batch_mining rho_distance --arch resnet50_frozen_normalize --miner_rho_distance_cp 0.35\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARreg_Triplet_Distance --seed 4 --gpu 0 --bs 112 --samples_per_class 2 --loss triplet --batch_mining rho_distance --arch resnet50_frozen_normalize --miner_rho_distance_cp 0.35\n\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARreg_SNR_Distance  --seed 0 --gpu 0 --bs 112 --samples_per_class 2 --loss snr --batch_mining rho_distance --arch resnet50_frozen_normalize --miner_rho_distance_cp 0.35\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARreg_SNR_Distance  --seed 1 --gpu 0 --bs 112 --samples_per_class 2 --loss snr --batch_mining rho_distance --arch resnet50_frozen_normalize --miner_rho_distance_cp 0.35\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARreg_SNR_Distance  --seed 2 --gpu 0 --bs 112 --samples_per_class 2 --loss snr --batch_mining rho_distance --arch resnet50_frozen_normalize --miner_rho_distance_cp 0.35\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARreg_SNR_Distance  --seed 3 --gpu 0 --bs 112 --samples_per_class 2 --loss snr --batch_mining rho_distance --arch resnet50_frozen_normalize --miner_rho_distance_cp 0.35\npython main.py --dataset cars196 --kernels 6 --source $datapath --n_epochs 150 --log_online --project RevisitDML --group CARreg_SNR_Distance  --seed 4 --gpu 0 --bs 112 --samples_per_class 2 --loss snr --batch_mining rho_distance --arch resnet50_frozen_normalize --miner_rho_distance_cp 0.35\n\n\n\n\n\n\n\n\n\n\n\"\"\"============= Baseline Runs --- Online Products ====================\"\"\"\npython main.py --dataset online_products --kernels 6 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOP_Npair --seed 0 --gpu 0 --bs 112 --samples_per_class 2 --loss npair --batch_mining npair --arch resnet50_frozen\npython main.py --dataset online_products --kernels 6 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOP_Npair --seed 1 --gpu 0 --bs 112 --samples_per_class 2 --loss npair --batch_mining npair --arch resnet50_frozen\npython main.py --dataset online_products --kernels 6 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOP_Npair --seed 2 --gpu 0 --bs 112 --samples_per_class 2 --loss npair --batch_mining npair --arch resnet50_frozen\npython main.py --dataset online_products --kernels 6 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOP_Npair --seed 3 --gpu 0 --bs 112 --samples_per_class 2 --loss npair --batch_mining npair --arch resnet50_frozen\npython main.py --dataset online_products --kernels 6 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOP_Npair --seed 4 --gpu 0 --bs 112 --samples_per_class 2 --loss npair --batch_mining npair --arch resnet50_frozen\n\n\npython main.py --dataset online_products --kernels 6 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOP_GenLifted --seed 0 --gpu 0 --bs 112 --samples_per_class 2 --loss lifted --batch_mining lifted --arch resnet50_frozen\npython main.py --dataset online_products --kernels 6 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOP_GenLifted --seed 1 --gpu 0 --bs 112 --samples_per_class 2 --loss lifted --batch_mining lifted --arch resnet50_frozen\npython main.py --dataset online_products --kernels 6 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOP_GenLifted --seed 2 --gpu 0 --bs 112 --samples_per_class 2 --loss lifted --batch_mining lifted --arch resnet50_frozen\npython main.py --dataset online_products --kernels 6 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOP_GenLifted --seed 3 --gpu 0 --bs 112 --samples_per_class 2 --loss lifted --batch_mining lifted --arch resnet50_frozen\npython main.py --dataset online_products --kernels 6 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOP_GenLifted --seed 4 --gpu 0 --bs 112 --samples_per_class 2 --loss lifted --batch_mining lifted --arch resnet50_frozen\n\n\npython main.py --dataset online_products --kernels 6 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOP_Histogram --seed 0 --gpu 0 --bs 112 --samples_per_class 2 --loss histogram --arch resnet50_frozen_normalize --loss_histogram_nbins 11\npython main.py --dataset online_products --kernels 6 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOP_Histogram --seed 1 --gpu 0 --bs 112 --samples_per_class 2 --loss histogram --arch resnet50_frozen_normalize --loss_histogram_nbins 11\npython main.py --dataset online_products --kernels 6 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOP_Histogram --seed 2 --gpu 0 --bs 112 --samples_per_class 2 --loss histogram --arch resnet50_frozen_normalize --loss_histogram_nbins 11\npython main.py --dataset online_products --kernels 6 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOP_Histogram --seed 3 --gpu 0 --bs 112 --samples_per_class 2 --loss histogram --arch resnet50_frozen_normalize --loss_histogram_nbins 11\npython main.py --dataset online_products --kernels 6 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOP_Histogram --seed 4 --gpu 0 --bs 112 --samples_per_class 2 --loss histogram --arch resnet50_frozen_normalize --loss_histogram_nbins 11\n\n\npython main.py --dataset online_products --kernels 6 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOP_Contrastive --seed 0 --gpu 0 --bs 112 --samples_per_class 2 --loss contrastive --batch_mining distance --arch resnet50_frozen_normalize\npython main.py --dataset online_products --kernels 6 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOP_Contrastive --seed 1 --gpu 0 --bs 112 --samples_per_class 2 --loss contrastive --batch_mining distance --arch resnet50_frozen_normalize\npython main.py --dataset online_products --kernels 6 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOP_Contrastive --seed 2 --gpu 0 --bs 112 --samples_per_class 2 --loss contrastive --batch_mining distance --arch resnet50_frozen_normalize\npython main.py --dataset online_products --kernels 6 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOP_Contrastive --seed 3 --gpu 0 --bs 112 --samples_per_class 2 --loss contrastive --batch_mining distance --arch resnet50_frozen_normalize\npython main.py --dataset online_products --kernels 6 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOP_Contrastive --seed 4 --gpu 0 --bs 112 --samples_per_class 2 --loss contrastive --batch_mining distance --arch resnet50_frozen_normalize\n\n\npython main.py --dataset online_products --kernels 6 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOP_Angular --seed 0 --gpu 0 --bs 112 --samples_per_class 2 --loss angular --batch_mining npair --arch resnet50_frozen\npython main.py --dataset online_products --kernels 6 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOP_Angular --seed 1 --gpu 0 --bs 112 --samples_per_class 2 --loss angular --batch_mining npair --arch resnet50_frozen\npython main.py --dataset online_products --kernels 6 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOP_Angular --seed 2 --gpu 0 --bs 112 --samples_per_class 2 --loss angular --batch_mining npair --arch resnet50_frozen\npython main.py --dataset online_products --kernels 6 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOP_Angular --seed 3 --gpu 0 --bs 112 --samples_per_class 2 --loss angular --batch_mining npair --arch resnet50_frozen\npython main.py --dataset online_products --kernels 6 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOP_Angular --seed 4 --gpu 0 --bs 112 --samples_per_class 2 --loss angular --batch_mining npair --arch resnet50_frozen\n\npython main.py --dataset online_products --kernels 6 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOP_ArcFace --seed 0 --gpu 0 --bs 112 --samples_per_class 2 --loss arcface --arch resnet50_frozen_normalize\npython main.py --dataset online_products --kernels 6 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOP_ArcFace --seed 1 --gpu 0 --bs 112 --samples_per_class 2 --loss arcface --arch resnet50_frozen_normalize\npython main.py --dataset online_products --kernels 6 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOP_ArcFace --seed 2 --gpu 0 --bs 112 --samples_per_class 2 --loss arcface --arch resnet50_frozen_normalize\npython main.py --dataset online_products --kernels 6 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOP_ArcFace --seed 3 --gpu 0 --bs 112 --samples_per_class 2 --loss arcface --arch resnet50_frozen_normalize\npython main.py --dataset online_products --kernels 6 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOP_ArcFace --seed 4 --gpu 0 --bs 112 --samples_per_class 2 --loss arcface --arch resnet50_frozen_normalize\n\npython main.py --dataset online_products --kernels 6 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOP_Triplet_Random --seed 0 --gpu 0 --bs 112 --samples_per_class 2 --loss triplet --batch_mining random --arch resnet50_frozen_normalize\npython main.py --dataset online_products --kernels 6 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOP_Triplet_Random --seed 1 --gpu 0 --bs 112 --samples_per_class 2 --loss triplet --batch_mining random --arch resnet50_frozen_normalize\npython main.py --dataset online_products --kernels 6 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOP_Triplet_Random --seed 2 --gpu 0 --bs 112 --samples_per_class 2 --loss triplet --batch_mining random --arch resnet50_frozen_normalize\npython main.py --dataset online_products --kernels 6 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOP_Triplet_Random --seed 3 --gpu 0 --bs 112 --samples_per_class 2 --loss triplet --batch_mining random --arch resnet50_frozen_normalize\npython main.py --dataset online_products --kernels 6 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOP_Triplet_Random --seed 4 --gpu 0 --bs 112 --samples_per_class 2 --loss triplet --batch_mining random --arch resnet50_frozen_normalize\n\npython main.py --dataset online_products --kernels 6 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOP_Triplet_Semihard --seed 0 --gpu 0 --bs 112 --samples_per_class 2 --loss triplet --batch_mining semihard --arch resnet50_frozen_normalize\npython main.py --dataset online_products --kernels 6 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOP_Triplet_Semihard --seed 1 --gpu 0 --bs 112 --samples_per_class 2 --loss triplet --batch_mining semihard --arch resnet50_frozen_normalize\npython main.py --dataset online_products --kernels 6 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOP_Triplet_Semihard --seed 2 --gpu 0 --bs 112 --samples_per_class 2 --loss triplet --batch_mining semihard --arch resnet50_frozen_normalize\npython main.py --dataset online_products --kernels 6 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOP_Triplet_Semihard --seed 3 --gpu 0 --bs 112 --samples_per_class 2 --loss triplet --batch_mining semihard --arch resnet50_frozen_normalize\npython main.py --dataset online_products --kernels 6 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOP_Triplet_Semihard --seed 4 --gpu 0 --bs 112 --samples_per_class 2 --loss triplet --batch_mining semihard --arch resnet50_frozen_normalize\n\npython main.py --dataset online_products --kernels 6 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOP_Triplet_Softhard --seed 0 --gpu 0 --bs 112 --samples_per_class 2 --loss triplet --batch_mining softhard --arch resnet50_frozen_normalize\npython main.py --dataset online_products --kernels 6 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOP_Triplet_Softhard --seed 1 --gpu 0 --bs 112 --samples_per_class 2 --loss triplet --batch_mining softhard --arch resnet50_frozen_normalize\npython main.py --dataset online_products --kernels 6 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOP_Triplet_Softhard --seed 2 --gpu 0 --bs 112 --samples_per_class 2 --loss triplet --batch_mining softhard --arch resnet50_frozen_normalize\npython main.py --dataset online_products --kernels 6 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOP_Triplet_Softhard --seed 3 --gpu 0 --bs 112 --samples_per_class 2 --loss triplet --batch_mining softhard --arch resnet50_frozen_normalize\npython main.py --dataset online_products --kernels 6 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOP_Triplet_Softhard --seed 4 --gpu 0 --bs 112 --samples_per_class 2 --loss triplet --batch_mining softhard --arch resnet50_frozen_normalize\n\npython main.py --dataset online_products --kernels 6 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOP_Triplet_Distance --seed 0 --gpu 0 --bs 112 --samples_per_class 2 --loss triplet --batch_mining distance --arch resnet50_frozen_normalize\npython main.py --dataset online_products --kernels 6 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOP_Triplet_Distance --seed 1 --gpu 0 --bs 112 --samples_per_class 2 --loss triplet --batch_mining distance --arch resnet50_frozen_normalize\npython main.py --dataset online_products --kernels 6 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOP_Triplet_Distance --seed 2 --gpu 0 --bs 112 --samples_per_class 2 --loss triplet --batch_mining distance --arch resnet50_frozen_normalize\npython main.py --dataset online_products --kernels 6 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOP_Triplet_Distance --seed 3 --gpu 0 --bs 112 --samples_per_class 2 --loss triplet --batch_mining distance --arch resnet50_frozen_normalize\npython main.py --dataset online_products --kernels 6 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOP_Triplet_Distance --seed 4 --gpu 0 --bs 112 --samples_per_class 2 --loss triplet --batch_mining distance --arch resnet50_frozen_normalize\n\npython main.py --dataset online_products --kernels 6 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOP_Quadruplet_Distance --seed 0 --gpu 0 --bs 112 --samples_per_class 2 --loss quadruplet --batch_mining distance --arch resnet50_frozen_normalize\npython main.py --dataset online_products --kernels 6 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOP_Quadruplet_Distance --seed 1 --gpu 0 --bs 112 --samples_per_class 2 --loss quadruplet --batch_mining distance --arch resnet50_frozen_normalize\npython main.py --dataset online_products --kernels 6 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOP_Quadruplet_Distance --seed 2 --gpu 0 --bs 112 --samples_per_class 2 --loss quadruplet --batch_mining distance --arch resnet50_frozen_normalize\npython main.py --dataset online_products --kernels 6 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOP_Quadruplet_Distance --seed 3 --gpu 0 --bs 112 --samples_per_class 2 --loss quadruplet --batch_mining distance --arch resnet50_frozen_normalize\npython main.py --dataset online_products --kernels 6 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOP_Quadruplet_Distance --seed 4 --gpu 0 --bs 112 --samples_per_class 2 --loss quadruplet --batch_mining distance --arch resnet50_frozen_normalize\n\npython main.py --dataset online_products --kernels 6 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOP_Margin_b06_Distance --loss_margin_beta 0.6 --seed 0 --gpu 0 --bs 112 --samples_per_class 2 --loss margin --batch_mining distance --arch resnet50_frozen_normalize\npython main.py --dataset online_products --kernels 6 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOP_Margin_b06_Distance --loss_margin_beta 0.6 --seed 1 --gpu 0 --bs 112 --samples_per_class 2 --loss margin --batch_mining distance --arch resnet50_frozen_normalize\npython main.py --dataset online_products --kernels 6 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOP_Margin_b06_Distance --loss_margin_beta 0.6 --seed 2 --gpu 0 --bs 112 --samples_per_class 2 --loss margin --batch_mining distance --arch resnet50_frozen_normalize\npython main.py --dataset online_products --kernels 6 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOP_Margin_b06_Distance --loss_margin_beta 0.6 --seed 3 --gpu 0 --bs 112 --samples_per_class 2 --loss margin --batch_mining distance --arch resnet50_frozen_normalize\npython main.py --dataset online_products --kernels 6 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOP_Margin_b06_Distance --loss_margin_beta 0.6 --seed 4 --gpu 0 --bs 112 --samples_per_class 2 --loss margin --batch_mining distance --arch resnet50_frozen_normalize\n\npython main.py --dataset online_products --kernels 2 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOP_Margin_b12_Distance --seed 0 --gpu 0 --bs 112 --samples_per_class 2 --loss margin --batch_mining distance --arch resnet50_frozen_normalize\npython main.py --dataset online_products --kernels 2 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOP_Margin_b12_Distance --seed 1 --gpu 0 --bs 112 --samples_per_class 2 --loss margin --batch_mining distance --arch resnet50_frozen_normalize\npython main.py --dataset online_products --kernels 2 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOP_Margin_b12_Distance --seed 2 --gpu 0 --bs 112 --samples_per_class 2 --loss margin --batch_mining distance --arch resnet50_frozen_normalize\npython main.py --dataset online_products --kernels 2 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOP_Margin_b12_Distance --seed 3 --gpu 0 --bs 112 --samples_per_class 2 --loss margin --batch_mining distance --arch resnet50_frozen_normalize\npython main.py --dataset online_products --kernels 2 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOP_Margin_b12_Distance --seed 4 --gpu 0 --bs 112 --samples_per_class 2 --loss margin --batch_mining distance --arch resnet50_frozen_normalize\n\npython main.py --dataset online_products --kernels 2 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOP_SNR_Distance  --seed 0 --gpu 0 --bs 112 --samples_per_class 2 --loss snr --batch_mining distance --arch resnet50_frozen_normalize\npython main.py --dataset online_products --kernels 2 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOP_SNR_Distance  --seed 1 --gpu 0 --bs 112 --samples_per_class 2 --loss snr --batch_mining distance --arch resnet50_frozen_normalize\npython main.py --dataset online_products --kernels 2 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOP_SNR_Distance  --seed 2 --gpu 0 --bs 112 --samples_per_class 2 --loss snr --batch_mining distance --arch resnet50_frozen_normalize\npython main.py --dataset online_products --kernels 2 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOP_SNR_Distance  --seed 3 --gpu 0 --bs 112 --samples_per_class 2 --loss snr --batch_mining distance --arch resnet50_frozen_normalize\npython main.py --dataset online_products --kernels 2 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOP_SNR_Distance  --seed 4 --gpu 0 --bs 112 --samples_per_class 2 --loss snr --batch_mining distance --arch resnet50_frozen_normalize\n\npython main.py --dataset online_products --kernels 2 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOP_MS --seed 0 --gpu 0 --bs 112 --samples_per_class 2 --loss multisimilarity --arch resnet50_frozen_normalize\npython main.py --dataset online_products --kernels 2 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOP_MS --seed 1 --gpu 0 --bs 112 --samples_per_class 2 --loss multisimilarity --arch resnet50_frozen_normalize\npython main.py --dataset online_products --kernels 2 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOP_MS --seed 2 --gpu 0 --bs 112 --samples_per_class 2 --loss multisimilarity --arch resnet50_frozen_normalize\npython main.py --dataset online_products --kernels 2 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOP_MS --seed 3 --gpu 0 --bs 112 --samples_per_class 2 --loss multisimilarity --arch resnet50_frozen_normalize\npython main.py --dataset online_products --kernels 2 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOP_MS --seed 4 --gpu 0 --bs 112 --samples_per_class 2 --loss multisimilarity --arch resnet50_frozen_normalize\n\npython main.py --dataset online_products --kernels 2 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOP_Softmax --seed 0 --gpu 0 --bs 112 --samples_per_class 2 --loss softmax --batch_mining distance --arch resnet50_frozen_normalize --loss_softmax_lr 0.002\npython main.py --dataset online_products --kernels 2 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOP_Softmax --seed 1 --gpu 0 --bs 112 --samples_per_class 2 --loss softmax --batch_mining distance --arch resnet50_frozen_normalize --loss_softmax_lr 0.002\npython main.py --dataset online_products --kernels 2 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOP_Softmax --seed 2 --gpu 0 --bs 112 --samples_per_class 2 --loss softmax --batch_mining distance --arch resnet50_frozen_normalize --loss_softmax_lr 0.002\npython main.py --dataset online_products --kernels 2 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOP_Softmax --seed 3 --gpu 0 --bs 112 --samples_per_class 2 --loss softmax --batch_mining distance --arch resnet50_frozen_normalize --loss_softmax_lr 0.002\npython main.py --dataset online_products --kernels 2 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOP_Softmax --seed 4 --gpu 0 --bs 112 --samples_per_class 2 --loss softmax --batch_mining distance --arch resnet50_frozen_normalize --loss_softmax_lr 0.002\n\n\n### Specturm-Regularized Ranking Losses\npython main.py --dataset online_products --kernels 6 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOPreg_Contrastive --seed 0 --gpu 0 --bs 112 --samples_per_class 2 --loss contrastive --batch_mining rho_distance --arch resnet50_frozen_normalize --miner_rho_distance_cp 0.15\npython main.py --dataset online_products --kernels 6 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOPreg_Contrastive --seed 1 --gpu 0 --bs 112 --samples_per_class 2 --loss contrastive --batch_mining rho_distance --arch resnet50_frozen_normalize --miner_rho_distance_cp 0.15\npython main.py --dataset online_products --kernels 6 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOPreg_Contrastive --seed 2 --gpu 0 --bs 112 --samples_per_class 2 --loss contrastive --batch_mining rho_distance --arch resnet50_frozen_normalize --miner_rho_distance_cp 0.15\npython main.py --dataset online_products --kernels 6 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOPreg_Contrastive --seed 3 --gpu 0 --bs 112 --samples_per_class 2 --loss contrastive --batch_mining rho_distance --arch resnet50_frozen_normalize --miner_rho_distance_cp 0.15\npython main.py --dataset online_products --kernels 6 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOPreg_Contrastive --seed 4 --gpu 0 --bs 112 --samples_per_class 2 --loss contrastive --batch_mining rho_distance --arch resnet50_frozen_normalize --miner_rho_distance_cp 0.15\n\npython main.py --dataset online_products --kernels 6 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOPreg_Margin_b06_Distance --loss_margin_beta 0.6 --seed 0 --gpu 0 --bs 112 --samples_per_class 2 --loss margin --batch_mining rho_distance --arch resnet50_frozen_normalize --miner_rho_distance_cp 0.15\npython main.py --dataset online_products --kernels 6 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOPreg_Margin_b06_Distance --loss_margin_beta 0.6 --seed 1 --gpu 0 --bs 112 --samples_per_class 2 --loss margin --batch_mining rho_distance --arch resnet50_frozen_normalize --miner_rho_distance_cp 0.15\npython main.py --dataset online_products --kernels 6 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOPreg_Margin_b06_Distance --loss_margin_beta 0.6 --seed 2 --gpu 0 --bs 112 --samples_per_class 2 --loss margin --batch_mining rho_distance --arch resnet50_frozen_normalize --miner_rho_distance_cp 0.15\npython main.py --dataset online_products --kernels 6 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOPreg_Margin_b06_Distance --loss_margin_beta 0.6 --seed 3 --gpu 0 --bs 112 --samples_per_class 2 --loss margin --batch_mining rho_distance --arch resnet50_frozen_normalize --miner_rho_distance_cp 0.15\npython main.py --dataset online_products --kernels 6 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOPreg_Margin_b06_Distance --loss_margin_beta 0.6 --seed 4 --gpu 0 --bs 112 --samples_per_class 2 --loss margin --batch_mining rho_distance --arch resnet50_frozen_normalize --miner_rho_distance_cp 0.15\n\npython main.py --dataset online_products --kernels 6 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOPreg_Margin_b12_Distance --seed 0 --gpu 0 --bs 112 --samples_per_class 2 --loss margin --batch_mining rho_distance --arch resnet50_frozen_normalize --miner_rho_distance_cp 0.15\npython main.py --dataset online_products --kernels 6 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOPreg_Margin_b12_Distance --seed 1 --gpu 0 --bs 112 --samples_per_class 2 --loss margin --batch_mining rho_distance --arch resnet50_frozen_normalize --miner_rho_distance_cp 0.15\npython main.py --dataset online_products --kernels 6 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOPreg_Margin_b12_Distance --seed 2 --gpu 0 --bs 112 --samples_per_class 2 --loss margin --batch_mining rho_distance --arch resnet50_frozen_normalize --miner_rho_distance_cp 0.15\npython main.py --dataset online_products --kernels 6 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOPreg_Margin_b12_Distance --seed 3 --gpu 0 --bs 112 --samples_per_class 2 --loss margin --batch_mining rho_distance --arch resnet50_frozen_normalize --miner_rho_distance_cp 0.15\npython main.py --dataset online_products --kernels 6 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOPreg_Margin_b12_Distance --seed 4 --gpu 0 --bs 112 --samples_per_class 2 --loss margin --batch_mining rho_distance --arch resnet50_frozen_normalize --miner_rho_distance_cp 0.15\n\npython main.py --dataset online_products --kernels 6 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOPreg_Triplet_Distance --seed 0 --gpu 0 --bs 112 --samples_per_class 2 --loss triplet --batch_mining rho_distance --arch resnet50_frozen_normalize --miner_rho_distance_cp 0.15\npython main.py --dataset online_products --kernels 6 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOPreg_Triplet_Distance --seed 1 --gpu 0 --bs 112 --samples_per_class 2 --loss triplet --batch_mining rho_distance --arch resnet50_frozen_normalize --miner_rho_distance_cp 0.15\npython main.py --dataset online_products --kernels 6 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOPreg_Triplet_Distance --seed 2 --gpu 0 --bs 112 --samples_per_class 2 --loss triplet --batch_mining rho_distance --arch resnet50_frozen_normalize --miner_rho_distance_cp 0.15\npython main.py --dataset online_products --kernels 6 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOPreg_Triplet_Distance --seed 3 --gpu 0 --bs 112 --samples_per_class 2 --loss triplet --batch_mining rho_distance --arch resnet50_frozen_normalize --miner_rho_distance_cp 0.15\npython main.py --dataset online_products --kernels 6 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOPreg_Triplet_Distance --seed 4 --gpu 0 --bs 112 --samples_per_class 2 --loss triplet --batch_mining rho_distance --arch resnet50_frozen_normalize --miner_rho_distance_cp 0.15\n\npython main.py --dataset online_products --kernels 6 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOPreg_SNR_Distance  --seed 0 --gpu 0 --bs 112 --samples_per_class 2 --loss snr --batch_mining rho_distance --arch resnet50_frozen_normalize --miner_rho_distance_cp 0.15\npython main.py --dataset online_products --kernels 6 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOPreg_SNR_Distance  --seed 1 --gpu 0 --bs 112 --samples_per_class 2 --loss snr --batch_mining rho_distance --arch resnet50_frozen_normalize --miner_rho_distance_cp 0.15\npython main.py --dataset online_products --kernels 6 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOPreg_SNR_Distance  --seed 2 --gpu 0 --bs 112 --samples_per_class 2 --loss snr --batch_mining rho_distance --arch resnet50_frozen_normalize --miner_rho_distance_cp 0.15\npython main.py --dataset online_products --kernels 6 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOPreg_SNR_Distance  --seed 3 --gpu 0 --bs 112 --samples_per_class 2 --loss snr --batch_mining rho_distance --arch resnet50_frozen_normalize --miner_rho_distance_cp 0.15\npython main.py --dataset online_products --kernels 6 --source $datapath --n_epochs 100 --log_online --project RevisitDML --group SOPreg_SNR_Distance  --seed 4 --gpu 0 --bs 112 --samples_per_class 2 --loss snr --batch_mining rho_distance --arch resnet50_frozen_normalize --miner_rho_distance_cp 0.15\n"
  },
  {
    "path": "architectures/__init__.py",
    "content": "import architectures.resnet50\nimport architectures.googlenet\nimport architectures.bninception\n\ndef select(arch, opt):\n    if 'resnet50' in arch:\n        return resnet50.Network(opt)\n    if 'googlenet' in arch:\n        return googlenet.Network(opt)\n    if 'bninception' in arch:\n        return bninception.Network(opt)\n"
  },
  {
    "path": "architectures/bninception.py",
    "content": "\"\"\"\nThe network architectures and weights are adapted and used from the great https://github.com/Cadene/pretrained-models.pytorch.\n\"\"\"\nimport torch, torch.nn as nn, torch.nn.functional as F\nimport pretrainedmodels as ptm\n\n\n\n\"\"\"=============================================================\"\"\"\nclass Network(torch.nn.Module):\n    def __init__(self, opt, return_embed_dict=False):\n        super(Network, self).__init__()\n\n        self.pars  = opt\n        self.model = ptm.__dict__['bninception'](num_classes=1000, pretrained='imagenet')\n        self.model.last_linear = torch.nn.Linear(self.model.last_linear.in_features, opt.embed_dim)\n        if '_he' in opt.arch:\n            torch.nn.init.kaiming_normal_(self.model.last_linear.weight, mode='fan_out')\n            torch.nn.init.constant_(self.model.last_linear.bias, 0)\n\n        if 'frozen' in opt.arch:\n            for module in filter(lambda m: type(m) == nn.BatchNorm2d, self.model.modules()):\n                module.eval()\n                module.train = lambda _: None\n\n        self.return_embed_dict = return_embed_dict\n\n        self.pool_base = torch.nn.AdaptiveAvgPool2d(1)\n        self.pool_aux = torch.nn.AdaptiveMaxPool2d(1) if 'double' in opt.arch else None\n\n        self.name = opt.arch\n\n        self.out_adjust = None\n\n    def forward(self, x, warmup=False, **kwargs):\n        x = self.model.features(x)\n        y = self.pool_base(x)\n        if self.pool_aux is not None:\n            y += self.pool_aux(x)\n        if warmup:\n            y,x = y.detach(), x.detach()\n        z = self.model.last_linear(y.view(len(x),-1))\n        if 'normalize' in self.name:\n            z = F.normalize(z, dim=-1)\n        if self.out_adjust and not self.training:\n            z = self.out_adjust(z)\n        return z,(y,x)\n\n    def functional_forward(self, x):\n        pass\n"
  },
  {
    "path": "architectures/googlenet.py",
    "content": "\"\"\"\nThe network architectures and weights are adapted and used from the great https://github.com/Cadene/pretrained-models.pytorch.\n\"\"\"\nimport torch, torch.nn as nn\nimport torchvision.models as mod\n\n\n\n\n\n\"\"\"=============================================================\"\"\"\nclass Network(torch.nn.Module):\n    def __init__(self, opt):\n        super(Network, self).__init__()\n\n        self.pars  = opt\n        self.model = mod.googlenet(pretrained=True)\n\n        self.model.last_linear = torch.nn.Linear(self.model.fc.in_features, opt.embed_dim)\n        self.model.fc = self.model.last_linear\n\n        self.name = opt.arch\n\n    def forward(self, x):\n        x = self.model(x)\n        if not 'normalize' in self.pars.arch:\n            return x\n        return torch.nn.functional.normalize(x, dim=-1)\n"
  },
  {
    "path": "architectures/resnet50.py",
    "content": "\"\"\"\nThe network architectures and weights are adapted and used from the great https://github.com/Cadene/pretrained-models.pytorch.\n\"\"\"\nimport torch, torch.nn as nn\nimport pretrainedmodels as ptm\n\n\n\n\n\n\"\"\"=============================================================\"\"\"\nclass Network(torch.nn.Module):\n    def __init__(self, opt):\n        super(Network, self).__init__()\n\n        self.pars  = opt\n        self.model = ptm.__dict__['resnet50'](num_classes=1000, pretrained='imagenet' if not opt.not_pretrained else None)\n\n        self.name = opt.arch\n\n        if 'frozen' in opt.arch:\n            for module in filter(lambda m: type(m) == nn.BatchNorm2d, self.model.modules()):\n                module.eval()\n                module.train = lambda _: None\n\n        self.model.last_linear = torch.nn.Linear(self.model.last_linear.in_features, opt.embed_dim)\n\n        self.layer_blocks = nn.ModuleList([self.model.layer1, self.model.layer2, self.model.layer3, self.model.layer4])\n\n        self.out_adjust = None\n\n\n    def forward(self, x, **kwargs):\n        x = self.model.maxpool(self.model.relu(self.model.bn1(self.model.conv1(x))))\n        for layerblock in self.layer_blocks:\n            x = layerblock(x)\n        no_avg_feat = x\n        x = self.model.avgpool(x)\n        enc_out = x = x.view(x.size(0),-1)\n\n        x = self.model.last_linear(x)\n\n        if 'normalize' in self.pars.arch:\n            x = torch.nn.functional.normalize(x, dim=-1)\n        if self.out_adjust and not self.train:\n            x = self.out_adjust(x)\n\n        return x, (enc_out, no_avg_feat)\n"
  },
  {
    "path": "batchminer/__init__.py",
    "content": "from batchminer import random_distance, intra_random\nfrom batchminer import lifted, rho_distance, softhard, npair, parametric, random, semihard, distance\n\nBATCHMINING_METHODS = {'random':random,\n                       'semihard':semihard,\n                       'softhard':softhard,\n                       'distance':distance,\n                       'rho_distance':rho_distance,\n                       'npair':npair,\n                       'parametric':parametric,\n                       'lifted':lifted,\n                       'random_distance': random_distance,\n                       'intra_random': intra_random}\n\n\ndef select(batchminername, opt):\n    #####\n    if batchminername not in BATCHMINING_METHODS: raise NotImplementedError('Batchmining {} not available!'.format(batchminername))\n\n    batchmine_lib = BATCHMINING_METHODS[batchminername]\n\n    return batchmine_lib.BatchMiner(opt)\n"
  },
  {
    "path": "batchminer/distance.py",
    "content": "import numpy as np\nimport torch, torch.nn as nn, torch.nn.functional as F\nimport batchminer\n\n\nclass BatchMiner():\n    def __init__(self, opt):\n        self.par          = opt\n        self.lower_cutoff = opt.miner_distance_lower_cutoff\n        self.upper_cutoff = opt.miner_distance_upper_cutoff\n        self.name         = 'distance'\n\n    def __call__(self, batch, labels, tar_labels=None, return_distances=False, distances=None):\n        if isinstance(labels, torch.Tensor): labels = labels.detach().cpu().numpy()\n        bs, dim = batch.shape\n\n        if distances is None:\n            distances = self.pdist(batch.detach()).clamp(min=self.lower_cutoff)\n        sel_d = distances.shape[-1]\n\n        positives, negatives = [],[]\n        labels_visited       = []\n        anchors              = []\n\n        tar_labels = labels if tar_labels is None else tar_labels\n\n        for i in range(bs):\n            neg = tar_labels!=labels[i]; pos = tar_labels==labels[i]\n\n            anchors.append(i)\n            q_d_inv = self.inverse_sphere_distances(dim, bs, distances[i], tar_labels, labels[i])\n            negatives.append(np.random.choice(sel_d,p=q_d_inv))\n\n            if np.sum(pos)>0:\n                #Sample positives randomly\n                if np.sum(pos)>1: pos[i] = 0\n                positives.append(np.random.choice(np.where(pos)[0]))\n                #Sample negatives by distance\n\n        sampled_triplets = [[a,p,n] for a,p,n in zip(anchors, positives, negatives)]\n\n        if return_distances:\n            return sampled_triplets, distances\n        else:\n            return sampled_triplets\n\n\n    def inverse_sphere_distances(self, dim, bs, anchor_to_all_dists, labels, anchor_label):\n            dists  = anchor_to_all_dists\n\n            #negated log-distribution of distances of unit sphere in dimension <dim>\n            log_q_d_inv = ((2.0 - float(dim)) * torch.log(dists) - (float(dim-3) / 2) * torch.log(1.0 - 0.25 * (dists.pow(2))))\n            log_q_d_inv[np.where(labels==anchor_label)[0]] = 0\n\n            q_d_inv     = torch.exp(log_q_d_inv - torch.max(log_q_d_inv)) # - max(log) for stability\n            q_d_inv[np.where(labels==anchor_label)[0]] = 0\n\n            ### NOTE: Cutting of values with high distances made the results slightly worse. It can also lead to\n            # errors where there are no available negatives (for high samples_per_class cases).\n            # q_d_inv[np.where(dists.detach().cpu().numpy()>self.upper_cutoff)[0]]    = 0\n\n            q_d_inv = q_d_inv/q_d_inv.sum()\n            return q_d_inv.detach().cpu().numpy()\n\n\n    def pdist(self, A):\n        prod = torch.mm(A, A.t())\n        norm = prod.diag().unsqueeze(1).expand_as(prod)\n        res = (norm + norm.t() - 2 * prod).clamp(min = 0)\n        return res.sqrt()\n"
  },
  {
    "path": "batchminer/intra_random.py",
    "content": "import numpy as np, torch\nimport itertools as it\nimport random\n\nclass BatchMiner():\n    def __init__(self, opt):\n        self.par          = opt\n        self.name         = 'random'\n\n    def __call__(self, batch, labels):\n        if isinstance(labels, torch.Tensor): labels = labels.detach().cpu().numpy()\n        unique_classes   = np.unique(labels)\n        indices          = np.arange(len(batch))\n        class_dict       = {i:indices[labels==i] for i in unique_classes}\n\n        sampled_triplets = []\n        for cls in np.random.choice(list(class_dict.keys()), len(labels), replace=True):\n            a,p,n = np.random.choice(class_dict[cls], 3, replace=True)\n            sampled_triplets.append((a,p,n))\n\n        return sampled_triplets\n"
  },
  {
    "path": "batchminer/lifted.py",
    "content": "import numpy as np, torch\n\nclass BatchMiner():\n    def __init__(self, opt):\n        self.par          = opt\n        self.name         = 'lifted'\n\n    def __call__(self, batch, labels):\n        if isinstance(labels, torch.Tensor): labels = labels.detach().cpu().numpy()\n\n        ###\n        anchors, positives, negatives = [], [], []\n        list(range(len(batch)))\n\n        for i in range(len(batch)):\n            anchor = i\n            pos    = labels==labels[anchor]\n\n            ###\n            if np.sum(pos)>1:\n                anchors.append(anchor)\n                positive_set = np.where(pos)[0]\n                positive_set = positive_set[positive_set!=anchor]\n                positives.append(positive_set)\n\n        ###\n        negatives = []\n        for anchor,positive_set in zip(anchors, positives):\n            neg_idxs = [i for i in range(len(batch)) if i not in [anchor]+list(positive_set)]\n            negative_set = np.arange(len(batch))[neg_idxs]\n            negatives.append(negative_set)\n\n        return anchors, positives, negatives\n"
  },
  {
    "path": "batchminer/npair.py",
    "content": "import numpy as np, torch\nclass BatchMiner():\n    def __init__(self, opt):\n        self.par          = opt\n        self.name         = 'npair'\n\n    def __call__(self, batch, labels):\n        if isinstance(labels, torch.Tensor): labels = labels.detach().cpu().numpy()\n\n        anchors, positives, negatives = [],[],[]\n\n        for i in range(len(batch)):\n            anchor = i\n            pos    = labels==labels[anchor]\n\n            if np.sum(pos)>1:\n                anchors.append(anchor)\n                avail_positive = np.where(pos)[0]\n                avail_positive = avail_positive[avail_positive!=anchor]\n                positive       = np.random.choice(avail_positive)\n                positives.append(positive)\n\n        ###\n        negatives = []\n        for anchor,positive in zip(anchors, positives):\n            neg_idxs = [i for i in range(len(batch)) if i not in [anchor, positive] and labels[i] != labels[anchor]]\n            # neg_idxs = [i for i in range(len(batch)) if i not in [anchor, positive]]\n            negative_set = np.arange(len(batch))[neg_idxs]\n            negatives.append(negative_set)\n\n        return anchors, positives, negatives\n"
  },
  {
    "path": "batchminer/parametric.py",
    "content": "import numpy as np, torch\n\n\nclass BatchMiner():\n    def __init__(self, opt):\n        self.par          = opt\n        self.mode         = opt.miner_parametric_mode\n        self.n_support    = opt.miner_parametric_n_support\n        self.support_lim  = opt.miner_parametric_support_lim\n        self.name         = 'parametric'\n\n        ###\n        self.set_sample_distr()\n\n\n\n    def __call__(self, batch, labels):\n        bs           = batch.shape[0]\n        sample_distr = self.sample_distr\n\n        if isinstance(labels, torch.Tensor): labels = labels.detach().cpu().numpy()\n\n        ###\n        distances = self.pdist(batch.detach())\n\n        p_assigns           = np.sum((distances.cpu().numpy().reshape(-1)>self.support[1:-1].reshape(-1,1)).T,axis=1).reshape(distances.shape)\n        outside_support_lim = (distances.cpu().numpy().reshape(-1)<self.support_lim[0]) * (distances.cpu().numpy().reshape(-1)>self.support_lim[1])\n        outside_support_lim = outside_support_lim.reshape(distances.shape)\n\n        sample_ps                      = sample_distr[p_assigns]\n        sample_ps[outside_support_lim] = 0\n\n        ###\n        anchors, labels_visited = [], []\n        positives, negatives = [],[]\n\n        ###\n        for i in range(bs):\n            neg = labels!=labels[i]; pos = labels==labels[i]\n\n            if np.sum(pos)>1:\n                anchors.append(i)\n\n                #Sample positives randomly\n                pos[i] = 0\n                positives.append(np.random.choice(np.where(pos)[0]))\n\n                #Sample negatives by distance\n                sample_p = sample_ps[i][neg]\n                sample_p = sample_p/sample_p.sum()\n                negatives.append(np.random.choice(np.arange(bs)[neg],p=sample_p))\n\n        sampled_triplets = [[a,p,n] for a,p,n in zip(anchors, positives, negatives)]\n        return sampled_triplets\n\n\n\n    def pdist(self, A, eps=1e-4):\n        prod = torch.mm(A, A.t())\n        norm = prod.diag().unsqueeze(1).expand_as(prod)\n        res = (norm + norm.t() - 2 * prod).clamp(min = 0)\n        return res.clamp(min = eps).sqrt()\n\n\n    def set_sample_distr(self):\n        self.support = np.linspace(self.support_lim[0], self.support_lim[1], self.n_support)\n\n        if self.mode == 'uniform':\n            self.sample_distr = np.array([1.] * (self.n_support-1))\n\n        if self.mode == 'hards':\n            self.sample_distr = self.support.copy()\n            self.sample_distr[self.support<=0.5] = 1\n            self.sample_distr[self.support>0.5]  = 0\n\n        if self.mode == 'semihards':\n            self.sample_distr = self.support.copy()\n            from IPython import embed; embed()\n            self.sample_distr[(self.support<=0.7) * (self.support>=0.3)] = 1\n            self.sample_distr[(self.support<0.3)  * (self.support>0.7)]  = 0\n\n        if self.mode == 'veryhards':\n            self.sample_distr = self.support.copy()\n            self.sample_distr[self.support<=0.3] = 1\n            self.sample_distr[self.support>0.3]  = 0\n\n        self.sample_distr = np.clip(self.sample_distr, 1e-15, 1)\n        self.sample_distr = self.sample_distr/self.sample_distr.sum()\n"
  },
  {
    "path": "batchminer/random.py",
    "content": "import numpy as np, torch\nimport itertools as it\nimport random\n\nclass BatchMiner():\n    def __init__(self, opt):\n        self.par          = opt\n        self.name         = 'random'\n\n    def __call__(self, batch, labels):\n        if isinstance(labels, torch.Tensor): labels = labels.detach().cpu().numpy()\n        unique_classes = np.unique(labels)\n        indices        = np.arange(len(batch))\n        class_dict     = {i:indices[labels==i] for i in unique_classes}\n\n        sampled_triplets = [list(it.product([x],[x],[y for y in unique_classes if x!=y])) for x in unique_classes]\n        sampled_triplets = [x for y in sampled_triplets for x in y]\n\n        sampled_triplets = [[x for x in list(it.product(*[class_dict[j] for j in i])) if x[0]!=x[1]] for i in sampled_triplets]\n        sampled_triplets = [x for y in sampled_triplets for x in y]\n\n        #NOTE: The number of possible triplets is given by #unique_classes*(2*(samples_per_class-1)!)*(#unique_classes-1)*samples_per_class\n        sampled_triplets = random.sample(sampled_triplets, batch.shape[0])\n        return sampled_triplets\n"
  },
  {
    "path": "batchminer/random_distance.py",
    "content": "import numpy as np, torch\n\n\nclass BatchMiner():\n    def __init__(self, opt):\n        self.par          = opt\n        self.lower_cutoff = opt.miner_distance_lower_cutoff\n        self.upper_cutoff = opt.miner_distance_upper_cutoff\n        self.name         = 'distance'\n\n    def __call__(self, batch, labels):\n        if isinstance(labels, torch.Tensor): labels = labels.detach().cpu().numpy()\n        labels = labels[np.random.choice(len(labels), len(labels), replace=False)]\n\n        bs = batch.shape[0]\n        distances    = self.pdist(batch.detach()).clamp(min=self.lower_cutoff)\n\n        positives, negatives = [],[]\n        labels_visited = []\n        anchors = []\n\n        for i in range(bs):\n            neg = labels!=labels[i]; pos = labels==labels[i]\n\n            if np.sum(pos)>1:\n                anchors.append(i)\n                q_d_inv = self.inverse_sphere_distances(batch, distances[i], labels, labels[i])\n                #Sample positives randomly\n                pos[i] = 0\n                positives.append(np.random.choice(np.where(pos)[0]))\n                #Sample negatives by distance\n                negatives.append(np.random.choice(bs,p=q_d_inv))\n\n        sampled_triplets = [[a,p,n] for a,p,n in zip(anchors, positives, negatives)]\n        return sampled_triplets\n\n\n    def inverse_sphere_distances(self, batch, anchor_to_all_dists, labels, anchor_label):\n            dists        = anchor_to_all_dists\n            bs,dim       = len(dists),batch.shape[-1]\n\n            #negated log-distribution of distances of unit sphere in dimension <dim>\n            log_q_d_inv = ((2.0 - float(dim)) * torch.log(dists) - (float(dim-3) / 2) * torch.log(1.0 - 0.25 * (dists.pow(2))))\n            log_q_d_inv[np.where(labels==anchor_label)[0]] = 0\n\n            q_d_inv     = torch.exp(log_q_d_inv - torch.max(log_q_d_inv)) # - max(log) for stability\n            q_d_inv[np.where(labels==anchor_label)[0]] = 0\n\n            ### NOTE: Cutting of values with high distances made the results slightly worse. It can also lead to\n            # errors where there are no available negatives (for high samples_per_class cases).\n            # q_d_inv[np.where(dists.detach().cpu().numpy()>self.upper_cutoff)[0]]    = 0\n\n            q_d_inv = q_d_inv/q_d_inv.sum()\n            return q_d_inv.detach().cpu().numpy()\n\n\n    def pdist(self, A):\n        prod = torch.mm(A, A.t())\n        norm = prod.diag().unsqueeze(1).expand_as(prod)\n        res = (norm + norm.t() - 2 * prod).clamp(min = 0)\n        return res.sqrt()\n"
  },
  {
    "path": "batchminer/rho_distance.py",
    "content": "import numpy as np, torch\n\n\nclass BatchMiner():\n    def __init__(self, opt):\n        self.par          = opt\n        self.lower_cutoff  = opt.miner_rho_distance_lower_cutoff\n        self.upper_cutoff  = opt.miner_rho_distance_upper_cutoff\n        self.contrastive_p = opt.miner_rho_distance_cp\n\n        self.name         = 'rho_distance'\n\n    def __call__(self, batch, labels, return_distances=False):\n        if isinstance(labels, torch.Tensor): labels = labels.detach().cpu().numpy()\n        bs = batch.shape[0]\n        distances    = self.pdist(batch.detach()).clamp(min=self.lower_cutoff)\n\n        positives, negatives = [],[]\n        labels_visited = []\n        anchors = []\n\n        for i in range(bs):\n            neg = labels!=labels[i]; pos = labels==labels[i]\n\n            use_contr = np.random.choice(2, p=[1-self.contrastive_p, self.contrastive_p])\n            if np.sum(pos)>1:\n                anchors.append(i)\n                if use_contr:\n                    positives.append(i)\n                    #Sample negatives by distance\n                    pos[i] = 0\n                    negatives.append(np.random.choice(np.where(pos)[0]))\n                else:\n                    q_d_inv = self.inverse_sphere_distances(batch, distances[i], labels, labels[i])\n                    #Sample positives randomly\n                    pos[i] = 0\n                    positives.append(np.random.choice(np.where(pos)[0]))\n                    #Sample negatives by distance\n                    negatives.append(np.random.choice(bs,p=q_d_inv))\n\n        sampled_triplets   = [[a,p,n] for a,p,n in zip(anchors, positives, negatives)]\n        self.push_triplets = np.sum([m[1]==m[2] for m in labels[sampled_triplets]])\n\n        if return_distances:\n            return sampled_triplets, distances\n        else:\n            return sampled_triplets\n\n\n    def inverse_sphere_distances(self, batch, anchor_to_all_dists, labels, anchor_label):\n            dists        = anchor_to_all_dists\n            bs,dim       = len(dists),batch.shape[-1]\n\n            #negated log-distribution of distances of unit sphere in dimension <dim>\n            log_q_d_inv = ((2.0 - float(dim)) * torch.log(dists) - (float(dim-3) / 2) * torch.log(1.0 - 0.25 * (dists.pow(2))))\n            log_q_d_inv[np.where(labels==anchor_label)[0]] = 0\n\n            q_d_inv     = torch.exp(log_q_d_inv - torch.max(log_q_d_inv)) # - max(log) for stability\n            q_d_inv[np.where(labels==anchor_label)[0]] = 0\n\n            ### NOTE: Cutting of values with high distances made the results slightly worse. It can also lead to\n            # errors where there are no available negatives (for high samples_per_class cases).\n            # q_d_inv[np.where(dists.detach().cpu().numpy()>self.upper_cutoff)[0]]    = 0\n\n            q_d_inv = q_d_inv/q_d_inv.sum()\n            return q_d_inv.detach().cpu().numpy()\n\n\n    def pdist(self, A, eps=1e-4):\n        prod = torch.mm(A, A.t())\n        norm = prod.diag().unsqueeze(1).expand_as(prod)\n        res = (norm + norm.t() - 2 * prod).clamp(min = 0)\n        return res.clamp(min = eps).sqrt()\n"
  },
  {
    "path": "batchminer/semihard.py",
    "content": "import numpy as np, torch\n\n\nclass BatchMiner():\n    def __init__(self, opt):\n        self.par          = opt\n        self.name         = 'semihard'\n        self.margin       = vars(opt)['loss_'+opt.loss+'_margin']\n\n    def __call__(self, batch, labels, return_distances=False):\n        if isinstance(labels, torch.Tensor): labels = labels.detach().numpy()\n        bs = batch.size(0)\n        #Return distance matrix for all elements in batch (BSxBS)\n        distances = self.pdist(batch.detach()).detach().cpu().numpy()\n\n        positives, negatives = [], []\n        anchors = []\n        for i in range(bs):\n            l, d = labels[i], distances[i]\n            neg = labels!=l; pos = labels==l\n\n            anchors.append(i)\n            pos[i] = 0\n            p      = np.random.choice(np.where(pos)[0])\n            positives.append(p)\n\n            #Find negatives that violate tripet constraint semi-negatives\n            neg_mask = np.logical_and(neg,d>d[p])\n            neg_mask = np.logical_and(neg_mask,d<self.margin+d[p])\n            if neg_mask.sum()>0:\n                negatives.append(np.random.choice(np.where(neg_mask)[0]))\n            else:\n                negatives.append(np.random.choice(np.where(neg)[0]))\n\n        sampled_triplets = [[a, p, n] for a, p, n in zip(anchors, positives, negatives)]\n\n        if return_distances:\n            return sampled_triplets, distances\n        else:\n            return sampled_triplets\n\n\n    def pdist(self, A):\n        prod = torch.mm(A, A.t())\n        norm = prod.diag().unsqueeze(1).expand_as(prod)\n        res = (norm + norm.t() - 2 * prod).clamp(min = 0)\n        return res.clamp(min = 0).sqrt()\n"
  },
  {
    "path": "batchminer/softhard.py",
    "content": "import numpy as np, torch\n\n\nclass BatchMiner():\n    def __init__(self, opt):\n        self.par          = opt\n        self.name         = 'softhard'\n\n    def __call__(self, batch, labels, return_distances=False):\n        if isinstance(labels, torch.Tensor): labels = labels.detach().numpy()\n        bs = batch.size(0)\n        #Return distance matrix for all elements in batch (BSxBS)\n        distances = self.pdist(batch.detach()).detach().cpu().numpy()\n\n        positives, negatives = [], []\n        anchors = []\n        for i in range(bs):\n            l, d = labels[i], distances[i]\n            neg = labels!=l; pos = labels==l\n\n            if np.sum(pos)>1:\n                anchors.append(i)\n                #1 for batchelements with label l\n                #0 for current anchor\n                pos[i] = False\n\n                #Find negatives that violate triplet constraint in a hard fashion\n                neg_mask = np.logical_and(neg,d<d[np.where(pos)[0]].max())\n                #Find positives that violate triplet constraint in a hard fashion\n                pos_mask = np.logical_and(pos,d>d[np.where(neg)[0]].min())\n\n                if pos_mask.sum()>0:\n                    positives.append(np.random.choice(np.where(pos_mask)[0]))\n                else:\n                    positives.append(np.random.choice(np.where(pos)[0]))\n\n                if neg_mask.sum()>0:\n                    negatives.append(np.random.choice(np.where(neg_mask)[0]))\n                else:\n                    negatives.append(np.random.choice(np.where(neg)[0]))\n\n        sampled_triplets = [[a, p, n] for a, p, n in zip(anchors, positives, negatives)]\n        if return_distances:\n            return sampled_triplets, distances\n        else:\n            return sampled_triplets\n\n\n\n    def pdist(self, A):\n        prod = torch.mm(A, A.t())\n        norm = prod.diag().unsqueeze(1).expand_as(prod)\n        res = (norm + norm.t() - 2 * prod).clamp(min = 0)\n        return res.clamp(min = 0).sqrt()\n"
  },
  {
    "path": "criteria/__init__.py",
    "content": "### Standard DML criteria\nfrom criteria import triplet, margin, proxynca, npair\nfrom criteria import lifted, contrastive, softmax\nfrom criteria import angular, snr, histogram, arcface\nfrom criteria import softtriplet, multisimilarity, quadruplet\n### Non-Standard Criteria\nfrom criteria import adversarial_separation\n### Basic Libs\nimport copy\n\n\n\"\"\"=================================================================================================\"\"\"\ndef select(loss, opt, to_optim, batchminer=None):\n    #####\n    losses = {'triplet': triplet,\n              'margin':margin,\n              'proxynca':proxynca,\n              'npair':npair,\n              'angular':angular,\n              'contrastive':contrastive,\n              'lifted':lifted,\n              'snr':snr,\n              'multisimilarity':multisimilarity,\n              'histogram':histogram,\n              'softmax':softmax,\n              'softtriplet':softtriplet,\n              'arcface':arcface,\n              'quadruplet':quadruplet,\n              'adversarial_separation':adversarial_separation}\n\n\n    if loss not in losses: raise NotImplementedError('Loss {} not implemented!'.format(loss))\n\n    loss_lib = losses[loss]\n    if loss_lib.REQUIRES_BATCHMINER:\n        if batchminer is None:\n            raise Exception('Loss {} requires one of the following batch mining methods: {}'.format(loss, loss_lib.ALLOWED_MINING_OPS))\n        else:\n            if batchminer.name not in loss_lib.ALLOWED_MINING_OPS:\n                raise Exception('{}-mining not allowed for {}-loss!'.format(batchminer.name, loss))\n\n\n    loss_par_dict  = {'opt':opt}\n    if loss_lib.REQUIRES_BATCHMINER:\n        loss_par_dict['batchminer'] = batchminer\n\n    criterion = loss_lib.Criterion(**loss_par_dict)\n\n    if loss_lib.REQUIRES_OPTIM:\n        if hasattr(criterion,'optim_dict_list') and criterion.optim_dict_list is not None:\n            to_optim += criterion.optim_dict_list\n        else:\n            to_optim    += [{'params':criterion.parameters(), 'lr':criterion.lr}]\n\n    return criterion, to_optim\n"
  },
  {
    "path": "criteria/adversarial_separation.py",
    "content": "import numpy as np\nimport torch, torch.nn as nn, torch.nn.functional as F\nimport batchminer\n\n\n\"\"\"=================================================================================================\"\"\"\nALLOWED_MINING_OPS  = list(batchminer.BATCHMINING_METHODS.keys())\nREQUIRES_BATCHMINER = False\nREQUIRES_OPTIM      = True\n\n### MarginLoss with trainable class separation margin beta. Runs on Mini-batches as well.\nclass Criterion(torch.nn.Module):\n    def __init__(self, opt):\n        \"\"\"\n        Args:\n            margin:             Triplet Margin.\n            nu:                 Regularisation Parameter for beta values if they are learned.\n            beta:               Class-Margin values.\n            n_classes:          Number of different classes during training.\n        \"\"\"\n        super().__init__()\n\n        ####\n        self.embed_dim  = opt.embed_dim\n        self.proj_dim   = opt.diva_decorrnet_dim\n\n        self.directions = opt.diva_decorrelations\n        self.weights    = opt.diva_rho_decorrelation\n\n        self.name       = 'adversarial_separation'\n\n        #Projection network\n        self.regressors = nn.ModuleDict()\n        for direction in self.directions:\n            self.regressors[direction] = torch.nn.Sequential(torch.nn.Linear(self.embed_dim, self.proj_dim), torch.nn.ReLU(), torch.nn.Linear(self.proj_dim, self.embed_dim)).to(torch.float).to(opt.device)\n\n        #Learning Rate for Projection Network\n        self.lr        = opt.diva_decorrnet_lr\n\n\n        ####\n        self.ALLOWED_MINING_OPS  = ALLOWED_MINING_OPS\n        self.REQUIRES_BATCHMINER = REQUIRES_BATCHMINER\n        self.REQUIRES_OPTIM      = REQUIRES_OPTIM\n\n\n        \n\n    def forward(self, feature_dict):\n        #Apply gradient reversal on input embeddings.\n        adj_feature_dict = {key:torch.nn.functional.normalize(grad_reverse(features),dim=-1) for key, features in feature_dict.items()}\n        #Project one embedding to the space of the other (with normalization), then compute the correlation.\n        sim_loss = 0\n        for weight, direction in zip(self.weights, self.directions):\n            source, target = direction.split('-')\n            sim_loss += -1.*weight*torch.mean(torch.mean((adj_feature_dict[target]*torch.nn.functional.normalize(self.regressors[direction](adj_feature_dict[source]),dim=-1))**2,dim=-1))\n        return sim_loss\n\n\n\n### Gradient Reversal Layer\nclass GradRev(torch.autograd.Function):\n    \"\"\"\n    Implements an autograd class to flip gradients during backward pass.\n    \"\"\"\n    def forward(self, x):\n        \"\"\"\n        Container which applies a simple identity function.\n\n        Input:\n            x: any torch tensor input.\n        \"\"\"\n        return x.view_as(x)\n\n    def backward(self, grad_output):\n        \"\"\"\n        Container to reverse gradient signal during backward pass.\n\n        Input:\n            grad_output: any computed gradient.\n        \"\"\"\n        return (grad_output * -1.)\n\n### Gradient reverse function\ndef grad_reverse(x):\n    \"\"\"\n    Applies gradient reversal on input.\n\n    Input:\n        x: any torch tensor input.\n    \"\"\"\n    return GradRev()(x)\n"
  },
  {
    "path": "criteria/angular.py",
    "content": "import numpy as np\nimport torch, torch.nn as nn, torch.nn.functional as F\nimport batchminer\n\n\n\"\"\"=================================================================================================\"\"\"\nALLOWED_MINING_OPS = ['npair']\nREQUIRES_BATCHMINER = True\nREQUIRES_OPTIM      = False\n\nclass Criterion(torch.nn.Module):\n    def __init__(self, opt, batchminer):\n        super(Criterion, self).__init__()\n\n        self.tan_angular_margin = np.tan(np.pi/180*opt.loss_angular_alpha)\n        self.lam            = opt.loss_angular_npair_ang_weight\n        self.l2_weight      = opt.loss_angular_npair_l2\n        self.batchminer     = batchminer\n\n        self.name           = 'angular'\n\n        ####\n        self.ALLOWED_MINING_OPS  = ALLOWED_MINING_OPS\n        self.REQUIRES_BATCHMINER = REQUIRES_BATCHMINER\n        self.REQUIRES_OPTIM      = REQUIRES_OPTIM\n\n        \n\n    def forward(self, batch, labels, **kwargs):\n        ####NOTE: Normalize Angular Loss, but dont normalize npair loss!\n        anchors, positives, negatives = self.batchminer(batch, labels)\n        anchors, positives, negatives = batch[anchors], batch[positives], batch[negatives]\n        n_anchors, n_positives, n_negatives = F.normalize(anchors, dim=1), F.normalize(positives, dim=1), F.normalize(negatives, dim=-1)\n\n        is_term1 = 4*self.tan_angular_margin**2*(n_anchors + n_positives)[:,None,:].bmm(n_negatives.permute(0,2,1))\n        is_term2 = 2*(1+self.tan_angular_margin**2)*n_anchors[:,None,:].bmm(n_positives[:,None,:].permute(0,2,1))\n        is_term1 = is_term1.view(is_term1.shape[0], is_term1.shape[-1])\n        is_term2 = is_term2.view(-1, 1)\n\n        inner_sum_ang = is_term1 - is_term2\n        angular_loss = torch.mean(torch.log(torch.sum(torch.exp(inner_sum_ang), dim=1) + 1))\n\n\n        inner_sum_npair = anchors[:,None,:].bmm((negatives - positives[:,None,:]).permute(0,2,1))\n        inner_sum_npair = inner_sum_npair.view(inner_sum_npair.shape[0], inner_sum_npair.shape[-1])\n        npair_loss = torch.mean(torch.log(torch.sum(torch.exp(inner_sum_npair.clamp(max=50,min=-50)), dim=1) + 1))\n\n        loss = npair_loss + self.lam*angular_loss + self.l2_weight*torch.mean(torch.norm(batch, p=2, dim=1))\n        return loss\n"
  },
  {
    "path": "criteria/arcface.py",
    "content": "import numpy as np\nimport torch, torch.nn as nn, torch.nn.functional as F\nimport batchminer\n\n\"\"\"=================================================================================================\"\"\"\nALLOWED_MINING_OPS  = None\nREQUIRES_BATCHMINER = False\nREQUIRES_OPTIM      = True\n\n### This implementation follows the pseudocode provided in the original paper.\nclass Criterion(torch.nn.Module):\n    def __init__(self, opt):\n        super(Criterion, self).__init__()\n        self.par = opt\n\n        ####\n        self.angular_margin = opt.loss_arcface_angular_margin\n        self.feature_scale  = opt.loss_arcface_feature_scale\n\n        self.class_map = torch.nn.Parameter(torch.Tensor(opt.n_classes, opt.embed_dim))\n        stdv = 1. / np.sqrt(self.class_map.size(1))\n        self.class_map.data.uniform_(-stdv, stdv)\n\n        self.name  = 'arcface'\n\n        self.lr    = opt.loss_arcface_lr\n\n        ####\n        self.ALLOWED_MINING_OPS  = ALLOWED_MINING_OPS\n        self.REQUIRES_BATCHMINER = REQUIRES_BATCHMINER\n        self.REQUIRES_OPTIM      = REQUIRES_OPTIM\n\n\n\n\n    def forward(self, batch, labels, **kwargs):\n        bs, labels = len(batch), labels.to(self.par.device)\n\n        class_map      = torch.nn.functional.normalize(self.class_map, dim=1)\n        #Note that the similarity becomes the cosine for normalized embeddings. Denoted as 'fc7' in the paper pseudocode.\n        cos_similarity = batch.mm(class_map.T).clamp(min=1e-10, max=1-1e-10)\n\n        pick = torch.zeros(bs, self.par.n_classes).bool().to(self.par.device)\n        pick[torch.arange(bs), labels] = 1\n\n        original_target_logit  = cos_similarity[pick]\n\n        theta                 = torch.acos(original_target_logit)\n        marginal_target_logit = torch.cos(theta + self.angular_margin)\n\n        class_pred = self.feature_scale * (cos_similarity + pick * (marginal_target_logit-original_target_logit).unsqueeze(1))\n        loss       = torch.nn.CrossEntropyLoss()(class_pred, labels)\n\n        return loss\n"
  },
  {
    "path": "criteria/contrastive.py",
    "content": "import numpy as np\nimport torch, torch.nn as nn, torch.nn.functional as F\nimport batchminer\n\n\"\"\"=================================================================================================\"\"\"\nALLOWED_MINING_OPS  = list(batchminer.BATCHMINING_METHODS.keys())\nREQUIRES_BATCHMINER = True\nREQUIRES_OPTIM      = False\n\n\nclass Criterion(torch.nn.Module):\n    def __init__(self, opt, batchminer):\n        super(Criterion, self).__init__()\n        self.pos_margin = opt.loss_contrastive_pos_margin\n        self.neg_margin = opt.loss_contrastive_neg_margin\n        self.batchminer = batchminer\n\n        self.name           = 'contrastive'\n\n        ####\n        self.ALLOWED_MINING_OPS  = ALLOWED_MINING_OPS\n        self.REQUIRES_BATCHMINER = REQUIRES_BATCHMINER\n        self.REQUIRES_OPTIM      = REQUIRES_OPTIM\n\n\n\n    def forward(self, batch, labels, **kwargs):\n        sampled_triplets = self.batchminer(batch, labels)\n\n        anchors   = [triplet[0] for triplet in sampled_triplets]\n        positives = [triplet[1] for triplet in sampled_triplets]\n        negatives = [triplet[2] for triplet in sampled_triplets]\n\n        pos_dists = torch.mean(F.relu(nn.PairwiseDistance(p=2)(batch[anchors,:], batch[positives,:]) -  self.pos_margin))\n        neg_dists = torch.mean(F.relu(self.neg_margin - nn.PairwiseDistance(p=2)(batch[anchors,:], batch[negatives,:])))\n\n        loss      = pos_dists + neg_dists\n\n        return loss\n"
  },
  {
    "path": "criteria/histogram.py",
    "content": "import numpy as np\nimport torch, torch.nn as nn, torch.nn.functional as F\nimport batchminer\n\n\"\"\"=================================================================================================\"\"\"\nALLOWED_MINING_OPS  = None\nREQUIRES_BATCHMINER = False\nREQUIRES_OPTIM      = False\n\n\n#NOTE: This implementation follows: https://github.com/valerystrizh/pytorch-histogram-loss\nclass Criterion(torch.nn.Module):\n    def __init__(self, opt):\n        \"\"\"\n        Args:\n            margin:             Triplet Margin.\n        \"\"\"\n        super(Criterion, self).__init__()\n        self.par       = opt\n\n        self.nbins     = opt.loss_histogram_nbins\n        self.bin_width = 2/(self.nbins - 1)\n\n        # We require a numpy and torch support as parts of the computation require numpy.\n        self.support        = np.linspace(-1,1,self.nbins).reshape(-1,1)\n        self.support_torch  = torch.linspace(-1,1,self.nbins).reshape(-1,1).to(opt.device)\n\n        self.name           = 'histogram'\n\n        ####\n        self.ALLOWED_MINING_OPS  = ALLOWED_MINING_OPS\n        self.REQUIRES_BATCHMINER = REQUIRES_BATCHMINER\n        self.REQUIRES_OPTIM      = REQUIRES_OPTIM\n\n\n    def forward(self, batch, labels, **kwargs):\n        #The original paper utilizes similarities instead of distances.\n        similarity = batch.mm(batch.T)\n\n        bs         = labels.size()[0]\n\n        ### We create a equality matrix for labels occuring in the batch\n        label_eqs = (labels.repeat(bs, 1)  == labels.view(-1, 1).repeat(1, bs))\n\n        ### Because the similarity matrix is symmetric, we will only utilise the upper triangular.\n        ### These values are indexed by sim_inds\n        sim_inds = torch.triu(torch.ones(similarity.size()), 1).bool().to(self.par.device)\n\n        ### For the upper triangular similarity matrix, we want to know where our positives/anchors and negatives are:\n        pos_inds = label_eqs[sim_inds].repeat(self.nbins, 1)\n        neg_inds = ~label_eqs[sim_inds].repeat(self.nbins, 1)\n\n        ###\n        n_pos = pos_inds[0].sum()\n        n_neg = neg_inds[0].sum()\n\n        ### Extract upper triangular from the similarity matrix. (produces a one-dim vector)\n        unique_sim = similarity[sim_inds].view(1, -1)\n\n        ### We broadcast this vector to each histogram bin. Each bin entry requires a different summation in self.histogram()\n        unique_sim_rep = unique_sim.repeat(self.nbins, 1)\n\n        ### This assigns bin-values for float-similarities. The conversion to numpy is important to avoid rounding errors in torch.\n        assigned_bin_values = ((unique_sim_rep.detach().cpu().numpy() + 1) / self.bin_width).astype(int) * self.bin_width - 1\n\n        ### We now compute the histogram over distances\n        hist_pos_sim = self.histogram(unique_sim_rep, assigned_bin_values, pos_inds, n_pos)\n        hist_neg_sim = self.histogram(unique_sim_rep, assigned_bin_values, neg_inds, n_neg)\n\n        ### Compute the CDF for the positive similarity histogram\n        hist_pos_rep  = hist_pos_sim.view(-1, 1).repeat(1, hist_pos_sim.size()[0])\n        hist_pos_inds = torch.tril(torch.ones(hist_pos_rep.size()), -1).bool()\n        hist_pos_rep[hist_pos_inds] = 0\n        hist_pos_cdf  = hist_pos_rep.sum(0)\n\n        loss = torch.sum(hist_neg_sim * hist_pos_cdf)\n\n        return loss\n\n\n    def histogram(self, unique_sim_rep, assigned_bin_values, idxs, n_elem):\n        \"\"\"\n        Compute the histogram over similarities.\n        Args:\n            unique_sim_rep:      torch tensor of shape nbins x n_unique_neg_similarities.\n            assigned_bin_values: Bin value for each similarity value in unique_sim_rep.\n            idxs:                positive/negative entry indices in unique_sim_rep\n            n_elem:              number of elements in unique_sim_rep.\n        \"\"\"\n        # Cloning is required because we change the similarity matrix in-place, but need it for the\n        # positive AND negative histogram. Note that clone() allows for backprop.\n        usr = unique_sim_rep.clone()\n        # For each bin (and its lower neighbour bin) we find the distance values that belong.\n        indsa = torch.tensor((assigned_bin_values==(self.support-self.bin_width) ) & idxs.detach().cpu().numpy())\n        indsb = torch.tensor((assigned_bin_values==self.support) & idxs.detach().cpu().numpy())\n        # Set all irrelevant similarities to 0\n        usr[~(indsb|indsa)]=0\n        #\n        usr[indsa] = (usr  - self.support_torch + self.bin_width)[indsa] / self.bin_width\n        usr[indsb] = (-usr + self.support_torch + self.bin_width)[indsb] / self.bin_width\n\n        return usr.sum(1)/n_elem\n"
  },
  {
    "path": "criteria/lifted.py",
    "content": "import numpy as np\nimport torch, torch.nn as nn, torch.nn.functional as F\nimport batchminer\n\n\"\"\"=================================================================================================\"\"\"\nALLOWED_MINING_OPS = ['lifted']\nREQUIRES_BATCHMINER = True\nREQUIRES_OPTIM      = False\n\n\nclass Criterion(torch.nn.Module):\n    def __init__(self, opt, batchminer):\n\n        super(Criterion, self).__init__()\n        self.margin     = opt.loss_lifted_neg_margin\n        self.l2_weight  = opt.loss_lifted_l2\n        self.batchminer = batchminer\n\n        self.name           = 'lifted'\n\n        ####\n        self.ALLOWED_MINING_OPS  = ALLOWED_MINING_OPS\n        self.REQUIRES_BATCHMINER = REQUIRES_BATCHMINER\n        self.REQUIRES_OPTIM      = REQUIRES_OPTIM\n\n\n        \n    def forward(self, batch, labels, **kwargs):\n        anchors, positives, negatives = self.batchminer(batch, labels)\n\n        loss = []\n        for anchor, positive_set, negative_set in zip(anchors, positives, negatives):\n            anchor, positive_set, negative_set = batch[anchor, :].view(1,-1), batch[positive_set, :].view(1,len(positive_set),-1), batch[negative_set, :].view(1,len(negative_set),-1)\n            pos_term = torch.logsumexp(nn.PairwiseDistance(p=2)(anchor[:,:,None], positive_set.permute(0,2,1)), dim=1)\n            neg_term = torch.logsumexp(self.margin - nn.PairwiseDistance(p=2)(anchor[:,:,None], negative_set.permute(0,2,1)), dim=1)\n            loss.append(F.relu(pos_term + neg_term))\n\n        loss = torch.mean(torch.stack(loss)) + self.l2_weight*torch.mean(torch.norm(batch, p=2, dim=1))\n        return loss\n"
  },
  {
    "path": "criteria/margin.py",
    "content": "import numpy as np\nimport torch, torch.nn as nn, torch.nn.functional as F\nimport batchminer\n\n\"\"\"=================================================================================================\"\"\"\nALLOWED_MINING_OPS  = list(batchminer.BATCHMINING_METHODS.keys())\nREQUIRES_BATCHMINER = True\nREQUIRES_OPTIM      = True\n\n### MarginLoss with trainable class separation margin beta. Runs on Mini-batches as well.\nclass Criterion(torch.nn.Module):\n    def __init__(self, opt, batchminer):\n        super(Criterion, self).__init__()\n        self.n_classes          = opt.n_classes\n\n        self.margin             = opt.loss_margin_margin\n        self.nu                 = opt.loss_margin_nu\n        self.beta_constant      = opt.loss_margin_beta_constant\n        self.beta_val           = opt.loss_margin_beta\n\n        if opt.loss_margin_beta_constant:\n            self.beta = opt.loss_margin_beta\n        else:\n            self.beta = torch.nn.Parameter(torch.ones(opt.n_classes)*opt.loss_margin_beta)\n\n        self.batchminer = batchminer\n\n        self.name  = 'margin'\n\n        self.lr    = opt.loss_margin_beta_lr\n\n        ####\n        self.ALLOWED_MINING_OPS  = ALLOWED_MINING_OPS\n        self.REQUIRES_BATCHMINER = REQUIRES_BATCHMINER\n        self.REQUIRES_OPTIM      = REQUIRES_OPTIM\n\n\n\n    def forward(self, batch, labels, **kwargs):\n        sampled_triplets = self.batchminer(batch, labels)\n\n        if len(sampled_triplets):\n            d_ap, d_an = [],[]\n            for triplet in sampled_triplets:\n                train_triplet = {'Anchor': batch[triplet[0],:], 'Positive':batch[triplet[1],:], 'Negative':batch[triplet[2]]}\n\n                pos_dist = ((train_triplet['Anchor']-train_triplet['Positive']).pow(2).sum()+1e-8).pow(1/2)\n                neg_dist = ((train_triplet['Anchor']-train_triplet['Negative']).pow(2).sum()+1e-8).pow(1/2)\n\n                d_ap.append(pos_dist)\n                d_an.append(neg_dist)\n            d_ap, d_an = torch.stack(d_ap), torch.stack(d_an)\n\n            if self.beta_constant:\n                beta = self.beta\n            else:\n                beta = torch.stack([self.beta[labels[triplet[0]]] for triplet in sampled_triplets]).to(torch.float).to(d_ap.device)\n\n            pos_loss = torch.nn.functional.relu(d_ap-beta+self.margin)\n            neg_loss = torch.nn.functional.relu(beta-d_an+self.margin)\n\n            pair_count = torch.sum((pos_loss>0.)+(neg_loss>0.)).to(torch.float).to(d_ap.device)\n\n            if pair_count == 0.:\n                loss = torch.sum(pos_loss+neg_loss)\n            else:\n                loss = torch.sum(pos_loss+neg_loss)/pair_count\n\n            if self.nu: \n                beta_regularization_loss = torch.sum(beta)\n                loss += self.nu * beta_regularisation_loss.to(torch.float).to(d_ap.device)\n        else:\n            loss = torch.tensor(0.).to(torch.float).to(batch.device)\n\n        return loss\n"
  },
  {
    "path": "criteria/multisimilarity.py",
    "content": "import torch, torch.nn as nn\n\n\n\n\"\"\"=================================================================================================\"\"\"\nALLOWED_MINING_OPS  = None\nREQUIRES_BATCHMINER = False\nREQUIRES_OPTIM      = False\n\nclass Criterion(torch.nn.Module):\n    def __init__(self, opt):\n        super(Criterion, self).__init__()\n        self.n_classes          = opt.n_classes\n\n        self.pos_weight = opt.loss_multisimilarity_pos_weight\n        self.neg_weight = opt.loss_multisimilarity_neg_weight\n        self.margin     = opt.loss_multisimilarity_margin\n        self.thresh     = opt.loss_multisimilarity_thresh\n\n        self.name           = 'multisimilarity'\n\n        ####\n        self.ALLOWED_MINING_OPS  = ALLOWED_MINING_OPS\n        self.REQUIRES_BATCHMINER = REQUIRES_BATCHMINER\n        self.REQUIRES_OPTIM      = REQUIRES_OPTIM\n\n\n    def forward(self, batch, labels, **kwargs):\n        similarity = batch.mm(batch.T)\n\n        loss = []\n        for i in range(len(batch)):\n            pos_idxs       = labels==labels[i]\n            pos_idxs[i]    = 0\n            neg_idxs       = labels!=labels[i]\n\n            anchor_pos_sim = similarity[i][pos_idxs]\n            anchor_neg_sim = similarity[i][neg_idxs]\n\n            ### This part doesn't really work, especially when you dont have a lot of positives in the batch...\n            neg_idxs = (anchor_neg_sim + self.margin) > torch.min(anchor_pos_sim)\n            pos_idxs = (anchor_pos_sim - self.margin) < torch.max(anchor_neg_sim)\n            if not torch.sum(neg_idxs) or not torch.sum(pos_idxs):\n                continue\n            anchor_neg_sim = anchor_neg_sim[neg_idxs]\n            anchor_pos_sim = anchor_pos_sim[pos_idxs]\n\n            pos_term = 1./self.pos_weight * torch.log(1+torch.sum(torch.exp(-self.pos_weight* (anchor_pos_sim - self.thresh))))\n            neg_term = 1./self.neg_weight * torch.log(1+torch.sum(torch.exp(self.neg_weight * (anchor_neg_sim - self.thresh))))\n\n            loss.append(pos_term + neg_term)\n\n        loss = torch.mean(torch.stack(loss))\n        return loss\n"
  },
  {
    "path": "criteria/npair.py",
    "content": "import numpy as np\nimport torch, torch.nn as nn, torch.nn.functional as F\nimport batchminer\n\n\n\"\"\"=================================================================================================\"\"\"\nALLOWED_MINING_OPS = ['npair']\nREQUIRES_BATCHMINER = True\nREQUIRES_OPTIM      = False\n\nclass Criterion(torch.nn.Module):\n    def __init__(self, opt, batchminer):\n        \"\"\"\n        Args:\n        \"\"\"\n        super(Criterion, self).__init__()\n        self.pars = opt\n        self.l2_weight = opt.loss_npair_l2\n        self.batchminer = batchminer\n\n        self.name           = 'npair'\n\n        ####\n        self.ALLOWED_MINING_OPS  = ALLOWED_MINING_OPS\n        self.REQUIRES_BATCHMINER = REQUIRES_BATCHMINER\n        self.REQUIRES_OPTIM      = REQUIRES_OPTIM\n\n\n    def forward(self, batch, labels, **kwargs):\n        anchors, positives, negatives = self.batchminer(batch, labels)\n\n        ##\n        loss  = 0\n        if 'bninception' in self.pars.arch:\n            ### clamping/value reduction to avoid initial overflow for high embedding dimensions!\n            batch = batch/4\n        for anchor, positive, negative_set in zip(anchors, positives, negatives):\n            a_embs, p_embs, n_embs = batch[anchor:anchor+1], batch[positive:positive+1], batch[negative_set]\n            inner_sum = a_embs[:,None,:].bmm((n_embs - p_embs[:,None,:]).permute(0,2,1))\n            inner_sum = inner_sum.view(inner_sum.shape[0], inner_sum.shape[-1])\n            loss  = loss + torch.mean(torch.log(torch.sum(torch.exp(inner_sum), dim=1) + 1))/len(anchors)\n            loss  = loss + self.l2_weight*torch.mean(torch.norm(batch, p=2, dim=1))/len(anchors)\n\n\n        return loss\n"
  },
  {
    "path": "criteria/proxynca.py",
    "content": "import numpy as np\nimport torch, torch.nn as nn, torch.nn.functional as F\nimport batchminer\n\n\n\"\"\"=================================================================================================\"\"\"\nALLOWED_MINING_OPS  = None\nREQUIRES_BATCHMINER = False\nREQUIRES_OPTIM      = True\n\n\nclass Criterion(torch.nn.Module):\n    def __init__(self, opt):\n        \"\"\"\n        Args:\n            opt: Namespace containing all relevant parameters.\n        \"\"\"\n        super(Criterion, self).__init__()\n\n        ####\n        self.num_proxies        = opt.n_classes\n        self.embed_dim          = opt.embed_dim\n\n        self.proxies            = torch.nn.Parameter(torch.randn(self.num_proxies, self.embed_dim)/8)\n        self.class_idxs         = torch.arange(self.num_proxies)\n\n        self.name           = 'proxynca'\n\n        self.optim_dict_list = [{'params':self.proxies, 'lr':opt.lr * opt.loss_proxynca_lrmulti}]\n\n\n        ####\n        self.ALLOWED_MINING_OPS  = ALLOWED_MINING_OPS\n        self.REQUIRES_BATCHMINER = REQUIRES_BATCHMINER\n        self.REQUIRES_OPTIM      = REQUIRES_OPTIM\n\n        \n\n    def forward(self, batch, labels, **kwargs):\n        #Empirically, multiplying the embeddings during the computation of the loss seem to allow for more stable training;\n        #Acts as a temperature in the NCA objective.\n        batch   = 3*torch.nn.functional.normalize(batch, dim=1)\n        proxies = 3*torch.nn.functional.normalize(self.proxies, dim=1)\n        #Group required proxies\n        pos_proxies = torch.stack([proxies[pos_label:pos_label+1,:] for pos_label in labels])\n        neg_proxies = torch.stack([torch.cat([self.class_idxs[:class_label],self.class_idxs[class_label+1:]]) for class_label in labels])\n        neg_proxies = torch.stack([proxies[neg_labels,:] for neg_labels in neg_proxies])\n        #Compute Proxy-distances\n        dist_to_neg_proxies = torch.sum((batch[:,None,:]-neg_proxies).pow(2),dim=-1)\n        dist_to_pos_proxies = torch.sum((batch[:,None,:]-pos_proxies).pow(2),dim=-1)\n        #Compute final proxy-based NCA loss\n        loss = torch.mean(dist_to_pos_proxies[:,0] + torch.logsumexp(-dist_to_neg_proxies, dim=1))\n\n        return loss\n"
  },
  {
    "path": "criteria/quadruplet.py",
    "content": "import numpy as np\nimport torch, torch.nn as nn, torch.nn.functional as F\nimport batchminer\n\"\"\"=================================================================================================\"\"\"\nALLOWED_MINING_OPS  = list(batchminer.BATCHMINING_METHODS.keys())\nREQUIRES_BATCHMINER = True\nREQUIRES_OPTIM      = False\n\n\nclass Criterion(torch.nn.Module):\n    def __init__(self, opt, batchminer):\n        super(Criterion, self).__init__()\n        self.batchminer = batchminer\n\n        self.name           = 'quadruplet'\n\n        self.margin_alpha_1 = opt.loss_quadruplet_margin_alpha_1\n        self.margin_alpha_2 = opt.loss_quadruplet_margin_alpha_2\n\n        ####\n        self.ALLOWED_MINING_OPS  = ALLOWED_MINING_OPS\n        self.REQUIRES_BATCHMINER = REQUIRES_BATCHMINER\n        self.REQUIRES_OPTIM      = REQUIRES_OPTIM\n\n\n\n    def triplet_distance(self, anchor, positive, negative):\n        return torch.nn.functional.relu(torch.norm(anchor-positive, p=2, dim=-1)-torch.norm(anchor-negative, p=2, dim=-1)+self.margin_alpha_1)\n\n    def quadruplet_distance(self, anchor, positive, negative, fourth_negative):\n        return torch.nn.functional.relu(torch.norm(anchor-positive, p=2, dim=-1)-torch.norm(negative-fourth_negative, p=2, dim=-1)+self.margin_alpha_2)\n\n    def forward(self, batch, labels, **kwargs):\n        sampled_triplets    = self.batchminer(batch, labels)\n\n        anchors   = np.array([triplet[0] for triplet in sampled_triplets]).reshape(-1,1)\n        positives = np.array([triplet[1] for triplet in sampled_triplets]).reshape(-1,1)\n        negatives = np.array([triplet[2] for triplet in sampled_triplets]).reshape(-1,1)\n\n        fourth_negatives = negatives!=negatives.T\n        fourth_negatives = [np.random.choice(np.arange(len(batch))[idxs]) for idxs in fourth_negatives]\n\n        triplet_loss     = self.triplet_distance(batch[anchors,:],batch[positives,:],batch[negatives,:])\n        quadruplet_loss  = self.quadruplet_distance(batch[anchors,:],batch[positives,:],batch[negatives,:],batch[fourth_negatives,:])\n\n        return torch.mean(triplet_loss) + torch.mean(quadruplet_loss)\n"
  },
  {
    "path": "criteria/snr.py",
    "content": "import numpy as np\nimport torch, torch.nn as nn, torch.nn.functional as F\nimport batchminer\n\n\"\"\"=================================================================================================\"\"\"\nALLOWED_MINING_OPS  = list(batchminer.BATCHMINING_METHODS.keys())\nREQUIRES_BATCHMINER = True\nREQUIRES_OPTIM      = False\n\n### This implements the Signal-To-Noise Ratio Triplet Loss\nclass Criterion(torch.nn.Module):\n    def __init__(self, opt, batchminer):\n        super(Criterion, self).__init__()\n        self.margin     = opt.loss_snr_margin\n        self.reg_lambda = opt.loss_snr_reg_lambda\n        self.batchminer = batchminer\n\n        if self.batchminer.name=='distance': self.reg_lambda = 0\n\n        self.name = 'snr'\n\n        ####\n        self.ALLOWED_MINING_OPS  = ALLOWED_MINING_OPS\n        self.REQUIRES_BATCHMINER = REQUIRES_BATCHMINER\n        self.REQUIRES_OPTIM      = REQUIRES_OPTIM\n\n\n\n    def forward(self, batch, labels, **kwargs):\n        sampled_triplets = self.batchminer(batch, labels)\n        anchors   = [triplet[0] for triplet in sampled_triplets]\n        positives = [triplet[1] for triplet in sampled_triplets]\n        negatives = [triplet[2] for triplet in sampled_triplets]\n\n        pos_snr  = torch.var(batch[anchors,:]-batch[positives,:], dim=1)/torch.var(batch[anchors,:], dim=1)\n        neg_snr  = torch.var(batch[anchors,:]-batch[negatives,:], dim=1)/torch.var(batch[anchors,:], dim=1)\n\n        reg_loss = torch.mean(torch.abs(torch.sum(batch[anchors,:],dim=1)))\n\n        snr_loss = torch.nn.functional.relu(pos_snr - neg_snr + self.margin)\n        snr_loss = torch.sum(snr_loss)/torch.sum(snr_loss>0)\n\n        loss = snr_loss + self.reg_lambda * reg_loss\n\n        return loss\n"
  },
  {
    "path": "criteria/softmax.py",
    "content": "import numpy as np\nimport torch, torch.nn as nn, torch.nn.functional as F\nimport batchminer\n\n\"\"\"=================================================================================================\"\"\"\nALLOWED_MINING_OPS  = None\nREQUIRES_BATCHMINER = False\nREQUIRES_OPTIM      = True\n\n### This Implementation follows: https://github.com/azgo14/classification_metric_learning\n\nclass Criterion(torch.nn.Module):\n    def __init__(self, opt):\n        super(Criterion, self).__init__()\n        self.par         = opt\n\n        self.temperature = opt.loss_softmax_temperature\n\n        self.class_map = torch.nn.Parameter(torch.Tensor(opt.n_classes, opt.embed_dim))\n        stdv = 1. / np.sqrt(self.class_map.size(1))\n        self.class_map.data.uniform_(-stdv, stdv)\n\n        self.name           = 'softmax'\n\n        self.lr = opt.loss_softmax_lr\n\n        ####\n        self.ALLOWED_MINING_OPS  = ALLOWED_MINING_OPS\n        self.REQUIRES_BATCHMINER = REQUIRES_BATCHMINER\n        self.REQUIRES_OPTIM      = REQUIRES_OPTIM\n\n\n    def forward(self, batch, labels, **kwargs):\n        class_mapped_batch = torch.nn.functional.linear(batch, torch.nn.functional.normalize(self.class_map, dim=1))\n\n        loss = torch.nn.CrossEntropyLoss()(class_mapped_batch/self.temperature, labels.to(torch.long).to(self.par.device))\n\n        return loss\n"
  },
  {
    "path": "criteria/softtriplet.py",
    "content": "import numpy as np\nimport torch, torch.nn as nn, torch.nn.functional as F\nimport batchminer\n\n\"\"\"=================================================================================================\"\"\"\nALLOWED_MINING_OPS  = None\nREQUIRES_BATCHMINER = False\nREQUIRES_OPTIM      = True\n\n### This implementation follows https://github.com/idstcv/SoftTriple\nclass Criterion(torch.nn.Module):\n    def __init__(self, opt):\n        super(Criterion, self).__init__()\n\n        ####\n        self.par         = opt\n        self.n_classes   = opt.n_classes\n\n        ####\n        self.n_centroids  = opt.loss_softtriplet_n_centroids\n        self.margin_delta = opt.loss_softtriplet_margin_delta\n        self.gamma        = opt.loss_softtriplet_gamma\n        self.lam          = opt.loss_softtriplet_lambda\n        self.reg_weight   = opt.loss_softtriplet_reg_weight\n\n\n        ####\n        self.reg_norm    = self.n_classes*self.n_centroids*(self.n_centroids-1)\n        self.reg_indices = torch.zeros((self.n_classes*self.n_centroids, self.n_classes*self.n_centroids), dtype=torch.bool).to(opt.device)\n        for i in range(0, self.n_classes):\n            for j in range(0, self.n_centroids):\n                self.reg_indices[i*self.n_centroids+j, i*self.n_centroids+j+1:(i+1)*self.n_centroids] = 1\n\n\n        ####\n        self.intra_class_centroids = torch.nn.Parameter(torch.Tensor(opt.embed_dim, self.n_classes*self.n_centroids))\n        stdv = 1. / np.sqrt(self.intra_class_centroids.size(1))\n        self.intra_class_centroids.data.uniform_(-stdv, stdv)\n\n        self.name = 'softtriplet'\n\n        self.lr   = opt.lr*opt.loss_softtriplet_lr\n\n        ####\n        self.ALLOWED_MINING_OPS  = ALLOWED_MINING_OPS\n        self.REQUIRES_BATCHMINER = REQUIRES_BATCHMINER\n        self.REQUIRES_OPTIM      = REQUIRES_OPTIM\n\n\n    def forward(self, batch, labels, **kwargs):\n        bs = batch.size(0)\n\n        intra_class_centroids     = torch.nn.functional.normalize(self.intra_class_centroids, dim=1)\n        similarities_to_centroids = batch.mm(intra_class_centroids).reshape(-1, self.n_classes, self.n_centroids)\n\n        soft_weight_over_centroids = torch.nn.Softmax(dim=1)(self.gamma*similarities_to_centroids)\n        per_class_embed            = torch.sum(soft_weight_over_centroids * similarities_to_centroids, dim=2)\n\n        margin_delta = torch.zeros(per_class_embed.shape).to(self.par.device)\n        margin_delta[torch.arange(0, bs), labels] = self.margin_delta\n\n        centroid_classification_loss = torch.nn.CrossEntropyLoss()(self.lam*(per_class_embed-margin_delta), labels.to(torch.long).to(self.par.device))\n\n        inter_centroid_similarity = intra_class_centroids.T.mm(intra_class_centroids)\n        regularisation_loss = torch.sum(torch.sqrt(2.00001-2*inter_centroid_similarity[self.reg_indices]))/self.reg_norm\n\n        return centroid_classification_loss + self.reg_weight * regularisation_loss\n"
  },
  {
    "path": "criteria/triplet.py",
    "content": "import numpy as np\nimport torch, torch.nn as nn, torch.nn.functional as F\nimport batchminer\n\n\"\"\"=================================================================================================\"\"\"\nALLOWED_MINING_OPS  = list(batchminer.BATCHMINING_METHODS.keys())\nREQUIRES_BATCHMINER = True\nREQUIRES_OPTIM      = False\n\n### Standard Triplet Loss, finds triplets in Mini-batches.\nclass Criterion(torch.nn.Module):\n    def __init__(self, opt, batchminer):\n        super(Criterion, self).__init__()\n        self.margin     = opt.loss_triplet_margin\n        self.batchminer = batchminer\n        self.name           = 'triplet'\n\n        ####\n        self.ALLOWED_MINING_OPS  = ALLOWED_MINING_OPS\n        self.REQUIRES_BATCHMINER = REQUIRES_BATCHMINER\n        self.REQUIRES_OPTIM      = REQUIRES_OPTIM\n\n\n    def triplet_distance(self, anchor, positive, negative):\n        return torch.nn.functional.relu((anchor-positive).pow(2).sum()-(anchor-negative).pow(2).sum()+self.margin)\n\n    def forward(self, batch, labels, **kwargs):\n        if isinstance(labels, torch.Tensor): labels = labels.cpu().numpy()\n        sampled_triplets = self.batchminer(batch, labels)\n        loss             = torch.stack([self.triplet_distance(batch[triplet[0],:],batch[triplet[1],:],batch[triplet[2],:]) for triplet in sampled_triplets])\n\n        return torch.mean(loss)\n"
  },
  {
    "path": "datasampler/__init__.py",
    "content": "import datasampler.class_random_sampler\nimport datasampler.random_sampler\nimport datasampler.greedy_coreset_sampler\nimport datasampler.fid_batchmatch_sampler\nimport datasampler.disthist_batchmatch_sampler\nimport datasampler.d2_coreset_sampler\n\n\ndef select(sampler, opt, image_dict, image_list=None, **kwargs):\n    if 'batchmatch' in sampler:\n        if sampler=='disthist_batchmatch':\n            sampler_lib = disthist_batchmatch_sampler\n        elif sampler=='fid_batchmatch':\n            sampler_lib = spc_fid_batchmatch_sampler\n    elif 'random' in sampler:\n        if 'class' in sampler:\n            sampler_lib = class_random_sampler\n        elif 'full' in sampler:\n            sampler_lib = random_sampler\n    elif 'coreset' in sampler:\n        if 'greedy' in sampler:\n            sampler_lib = greedy_coreset_sampler\n        elif 'd2' in sampler:\n            sampler_lib = d2_coreset_sampler\n    else:\n        raise Exception('Minibatch sampler <{}> not available!'.format(sampler))\n\n    sampler = sampler_lib.Sampler(opt,image_dict=image_dict,image_list=image_list)\n\n    return sampler\n"
  },
  {
    "path": "datasampler/class_random_sampler.py",
    "content": "import numpy as np\nimport torch, torch.nn as nn, torch.nn.functional as F\nfrom tqdm import tqdm\nimport random\n\n\n\n\"\"\"======================================================\"\"\"\nREQUIRES_STORAGE = False\n\n###\nclass Sampler(torch.utils.data.sampler.Sampler):\n    \"\"\"\n    Plugs into PyTorch Batchsampler Package.\n    \"\"\"\n    def __init__(self, opt, image_dict, image_list, **kwargs):\n        self.pars = opt\n\n        #####\n        self.image_dict         = image_dict\n        self.image_list         = image_list\n\n        #####\n        self.classes        = list(self.image_dict.keys())\n\n        ####\n        self.batch_size         = opt.bs\n        self.samples_per_class  = opt.samples_per_class\n        self.sampler_length     = len(image_list)//opt.bs\n        assert self.batch_size%self.samples_per_class==0, '#Samples per class must divide batchsize!'\n\n        self.name             = 'class_random_sampler'\n        self.requires_storage = False\n\n    def __iter__(self):\n        for _ in range(self.sampler_length):\n            subset = []\n            ### Random Subset from Random classes\n            draws = self.batch_size//self.samples_per_class\n\n            for _ in range(draws):\n                class_key = random.choice(self.classes)\n                class_ix_list = [random.choice(self.image_dict[class_key])[-1] for _ in range(self.samples_per_class)]\n                subset.extend(class_ix_list)\n\n            yield subset\n\n    def __len__(self):\n        return self.sampler_length\n"
  },
  {
    "path": "datasampler/d2_coreset_sampler.py",
    "content": "import numpy as np\nimport torch, torch.nn as nn, torch.nn.functional as F\nfrom tqdm import tqdm\nimport random\nfrom scipy import linalg\nfrom scipy.stats import multivariate_normal\n\n\"\"\"======================================================\"\"\"\nREQUIRES_STORAGE = True\n\n###\nclass Sampler(torch.utils.data.sampler.Sampler):\n    \"\"\"\n    Plugs into PyTorch Batchsampler Package.\n    \"\"\"\n    def __init__(self, opt, image_dict, image_list):\n        self.image_dict = image_dict\n        self.image_list = image_list\n\n        self.batch_size         = opt.bs\n        self.samples_per_class  = opt.samples_per_class\n        self.sampler_length     = len(image_list)//opt.bs\n        assert self.batch_size%self.samples_per_class==0, '#Samples per class must divide batchsize!'\n\n        self.name             = 'greedy_coreset_sampler'\n        self.requires_storage = True\n\n        self.bigbs           = opt.data_batchmatch_bigbs\n        self.update_storage  = not opt.data_storage_no_update\n        self.num_batch_comps = opt.data_batchmatch_ncomps\n\n        self.low_proj_dim    = opt.data_sampler_lowproj_dim\n\n        self.lam             = opt.data_d2_coreset_lambda\n\n        self.n_jobs = 16\n\n    def __iter__(self):\n        for i in range(self.sampler_length):\n            yield self.epoch_indices[i]\n\n\n    def precompute_indices(self):\n        from joblib import Parallel, delayed\n        import time\n\n        ### Random Subset from Random classes\n        bigb_idxs = np.random.choice(len(self.storage), self.bigbs, replace=True)\n        bigbatch  = self.storage[bigb_idxs]\n\n        print('Precomputing Indices... ', end='')\n        start = time.time()\n        def batchfinder(n_calls, pos):\n            idx_sets         = self.d2_coreset(n_calls, pos)\n            structured_batches = [list(bigb_idxs[idx_set]) for idx_set in idx_sets]\n            # structured_batch = list(bigb_idxs[self.fid_match(bigbatch, batch_size=self.batch_size//self.samples_per_class)])\n            #Add random per-class fillers to ensure that the batch is build up correctly.\n            for i in range(len(structured_batches)):\n                class_idxs = [self.image_list[idx][-1] for idx in structured_batches[i]]\n                for class_idx in class_idxs:\n                    structured_batches[i].extend([random.choice(self.image_dict[class_idx])[-1] for _ in range(self.samples_per_class-1)])\n\n            return structured_batches\n\n        n_calls            = int(np.ceil(self.sampler_length/self.n_jobs))\n        # self.epoch_indices = batchfinder(n_calls, 0)\n        self.epoch_indices = Parallel(n_jobs = self.n_jobs)(delayed(batchfinder)(n_calls, i) for i in range(self.n_jobs))\n        self.epoch_indices = [x for y in self.epoch_indices for x in y]\n        # self.epoch_indices = Parallel(n_jobs = self.n_jobs)(delayed(batchfinder)(self.storage[np.random.choice(len(self.storage), self.bigbs, replace=True)]) for _ in tqdm(range(self.sampler_length), desc='Precomputing Indices...'))\n\n        print('Done in {0:3.4f}s.'.format(time.time()-start))\n    def replace_storage_entries(self, embeddings, indices):\n        self.storage[indices] = embeddings\n\n    def create_storage(self, dataloader, model, device):\n        with torch.no_grad():\n            _ = model.eval()\n            _ = model.to(device)\n\n            embed_collect = []\n            for i,input_tuple in enumerate(tqdm(dataloader, 'Creating data storage...')):\n                embed = model(input_tuple[1].type(torch.FloatTensor).to(device))\n                if isinstance(embed, tuple): embed = embed[0]\n                embed = embed.cpu()\n                embed_collect.append(embed)\n            embed_collect = torch.cat(embed_collect, dim=0)\n            self.storage = embed_collect\n\n\n    def d2_coreset(self, calls, pos):\n        \"\"\"\n        \"\"\"\n        coll = []\n\n        for _ in range(calls):\n            bigbatch   = self.storage[np.random.choice(len(self.storage), self.bigbs, replace=False)]\n            batch_size = self.batch_size//self.samples_per_class\n\n            if self.low_proj_dim>0:\n                low_dim_proj = nn.Linear(bigbatch.shape[-1],self.low_proj_dim,bias=False)\n                with torch.no_grad(): bigbatch = low_dim_proj(bigbatch)\n\n            bigbatch = bigbatch.numpy()\n            # emp_mean, emp_std = np.mean(bigbatch, axis=0), np.std(bigbatch, axis=0)\n            emp_mean, emp_cov = np.mean(bigbatch, axis=0), np.cov(bigbatch.T)\n\n            prod        = np.matmul(bigbatch, bigbatch.T)\n            sq          = prod.diagonal().reshape(bigbatch.shape[0], 1)\n            dist_matrix = np.clip(-2*prod + sq + sq.T, 0, None)\n\n            start_anchor = np.random.multivariate_normal(emp_mean, emp_cov, 1).reshape(-1)\n            start_dists  = np.linalg.norm(bigbatch-start_anchor,axis=1)\n            start_point  = np.argmin(start_dists, axis=0)\n\n            idxs = list(range(len(bigbatch)))\n            del idxs[start_point]\n\n            k, sampled_indices = 1, [start_point]\n            dist_weights = dist_matrix[:,start_point]\n\n            normal_weights = multivariate_normal.pdf(bigbatch,emp_mean,emp_cov)\n            while k<batch_size:\n                normal_weights_to_use = normal_weights[idxs]/normal_weights[idxs].sum()\n                dim = bigbatch.shape[-1]\n\n                sampling_p = normal_weights_to_use*dist_weights[idxs]**self.lam\n                sampling_p/= np.sum(sampling_p)\n\n                dm_idx = np.random.choice(range(len(dist_matrix)-k),p=sampling_p.reshape(-1))\n                sample = idxs[dm_idx]\n\n                del idxs[dm_idx]\n\n                sampled_indices.append(sample)\n\n                dist_weights = dist_weights + dist_matrix[:,sample]\n                k += 1\n\n            coll.append(sampled_indices)\n\n        return coll\n\n\n    def __len__(self):\n        return self.sampler_length\n"
  },
  {
    "path": "datasampler/disthist_batchmatch_sampler.py",
    "content": "import numpy as np\nimport torch, torch.nn as nn, torch.nn.functional as F\nfrom tqdm import tqdm\nimport random\nfrom scipy import linalg\nfrom scipy.stats import wasserstein_distance\n\n\"\"\"======================================================\"\"\"\nREQUIRES_STORAGE = True\n\n###\nclass Sampler(torch.utils.data.sampler.Sampler):\n    \"\"\"\n    Plugs into PyTorch Batchsampler Package.\n    \"\"\"\n    def __init__(self, opt, image_dict, image_list):\n        self.image_dict = image_dict\n        self.image_list = image_list\n\n        self.batch_size         = opt.bs\n        self.samples_per_class  = opt.samples_per_class\n        self.sampler_length     = len(image_list)//opt.bs\n        assert self.batch_size%self.samples_per_class==0, '#Samples per class must divide batchsize!'\n\n        self.name             = 'distmoment_batchmatch_sampler'\n        self.requires_storage = True\n\n        self.bigbs           = opt.data_batchmatch_bigbs\n        self.update_storage  = not opt.data_storage_no_update\n        self.num_batch_comps = opt.data_batchmatch_ncomps\n\n        self.low_proj_dim    = opt.data_sampler_lowproj_dim\n\n        self.n_jobs = 16\n\n        self.internal_image_dict = {self.image_list[i]:i for i in range(len(self.image_list))}\n\n\n    def __iter__(self):\n        for i in range(self.sampler_length):\n            # ### Random Subset from Random classes\n            # bigb_idxs = np.random.choice(len(self.storage), self.bigbs, replace=True)\n            # bigbatch  = self.storage[bigb_idxs]\n            #\n            # structured_batch = list(bigb_idxs[self.fid_match(bigbatch, batch_size=self.batch_size//self.samples_per_class)])\n            # #Add random per-class fillers to ensure that the batch is build up correctly.\n            #\n            # class_idxs = [self.image_list[idx][-1] for idx in structured_batch]\n            # for class_idx in class_idxs:\n            #     structured_batch.extend([random.choice(self.image_dict[class_idx])[-1] for _ in range(self.samples_per_class-1)])\n\n            yield self.epoch_indices[i]\n\n\n\n    def precompute_indices(self):\n        from joblib import Parallel, delayed\n        import time\n        ### Random Subset from Random classes\n        # self.disthist_match()\n        print('Precomputing Indices... ', end='')\n        start = time.time()\n        n_calls            = int(np.ceil(self.sampler_length/self.n_jobs))\n        self.epoch_indices = Parallel(n_jobs = self.n_jobs)(delayed(self.disthist_match)(n_calls, i) for i in range(self.n_jobs))\n        self.epoch_indices = [x for y in self.epoch_indices for x in y]\n        print('Done in {0:3.4f}s.'.format(time.time()-start))\n\n\n    def replace_storage_entries(self, embeddings, indices):\n        self.storage[indices] = embeddings\n\n    def create_storage(self, dataloader, model, device):\n        with torch.no_grad():\n            _ = model.eval()\n            _ = model.to(device)\n\n            embed_collect = []\n            for i,input_tuple in enumerate(tqdm(dataloader, 'Creating data storage...')):\n                embed = model(input_tuple[1].type(torch.FloatTensor).to(device))\n                if isinstance(embed, tuple): embed = embed[0]\n                embed = embed.cpu()\n                embed_collect.append(embed)\n            embed_collect = torch.cat(embed_collect, dim=0)\n            self.storage = embed_collect\n\n\n    def spc_batchfinder(self, n_samples):\n        ### SpC-Sample big batch:\n        subset, classes = [], []\n        ### Random Subset from Random classes\n        for _ in range(n_samples//self.samples_per_class):\n            class_key = random.choice(list(self.image_dict.keys()))\n            # subset.extend([(class_key, random.choice(len(self.image_dict[class_key])) for _ in range(self.samples_per_class)])\n            subset.extend([random.choice(self.image_dict[class_key])[-1] for _ in range(self.samples_per_class)])\n            classes.extend([class_key]*self.samples_per_class)\n        return np.array(subset), np.array(classes)\n\n\n    def get_distmat(self, arr):\n        prod = np.matmul(arr, arr.T)\n        sq   = prod.diagonal().reshape(arr.shape[0], 1)\n        dist_matrix = np.sqrt(np.clip(-2*prod + sq + sq.T, 0 , None))\n        return dist_matrix\n\n    def disthist_match(self, calls, pos):\n        \"\"\"\n        \"\"\"\n        coll = []\n        for _ in range(calls):\n            bigb_data_idxs, bigb_data_classes = self.spc_batchfinder(self.bigbs)\n            bigb_dict = {}\n            for i, bigb_cls in enumerate(bigb_data_classes):\n                if bigb_cls not in bigb_dict: bigb_dict[bigb_cls] = []\n                bigb_dict[bigb_cls].append(i)\n\n            bigbatch = self.storage[bigb_data_idxs]\n            if self.low_proj_dim>0:\n                low_dim_proj = nn.Linear(bigbatch.shape[-1],self.low_proj_dim,bias=False)\n                with torch.no_grad(): bigbatch = low_dim_proj(bigbatch)\n            bigbatch = bigbatch.numpy()\n\n            bigb_distmat_triu_idxs = np.triu_indices(len(bigbatch),1)\n            bigb_distvals          = self.get_distmat(bigbatch)[bigb_distmat_triu_idxs]\n\n            bigb_disthist_range, bigb_disthist_bins = (np.min(bigb_distvals), np.max(bigb_distvals)), 50\n            bigb_disthist, _                        = np.histogram(bigb_distvals, bins=bigb_disthist_bins, range=bigb_disthist_range)\n            bigb_disthist = bigb_disthist/np.sum(bigb_disthist)\n\n            bigb_mu  = np.mean(bigbatch, axis=0)\n            bigb_std = np.std(bigbatch, axis=0)\n\n\n            cost_collect, bigb_idxs = [], []\n\n            for _ in range(self.num_batch_comps):\n                subset_idxs = [np.random.choice(bigb_dict[np.random.choice(list(bigb_dict.keys()))], self.samples_per_class, replace=False) for _ in range(self.batch_size//self.samples_per_class)]\n                subset_idxs = [x for y in subset_idxs for x in y]\n                # subset_idxs = sorted(np.random.choice(len(bigbatch), batch_size, replace=False))\n                bigb_idxs.append(subset_idxs)\n                subset         = bigbatch[subset_idxs,:]\n                subset_distmat = self.get_distmat(subset)\n\n                subset_distmat_triu_idxs = np.triu_indices(len(subset_distmat),1)\n                subset_distvals          = self.get_distmat(subset)[subset_distmat_triu_idxs]\n\n                subset_disthist_range, subset_disthist_bins = (np.min(subset_distvals), np.max(subset_distvals)), 50\n                subset_disthist, _                          = np.histogram(subset_distvals, bins=bigb_disthist_bins, range=bigb_disthist_range)\n                subset_disthist = subset_disthist/np.sum(subset_disthist)\n\n                subset_mu  = np.mean(subset, axis=0)\n                subset_std = np.std(subset, axis=0)\n\n\n                dist_wd = wasserstein_distance(bigb_disthist, subset_disthist)+wasserstein_distance(subset_disthist, bigb_disthist)\n                cost    = np.linalg.norm(bigb_mu - subset_mu) + np.linalg.norm(bigb_std - subset_std) + 75*dist_wd\n                cost_collect.append(cost)\n\n            bigb_ix      = bigb_idxs[np.argmin(cost_collect)]\n            bigb_data_ix = bigb_data_idxs[bigb_ix]\n            coll.append(bigb_data_ix)\n\n        return coll\n\n    def __len__(self):\n        return self.sampler_length\n"
  },
  {
    "path": "datasampler/fid_batchmatch_sampler.py",
    "content": "import numpy as np\nimport torch, torch.nn as nn, torch.nn.functional as F\nfrom tqdm import tqdm\nimport random\nfrom scipy import linalg\n\n\n\"\"\"======================================================\"\"\"\nREQUIRES_STORAGE = True\n\n###\nclass Sampler(torch.utils.data.sampler.Sampler):\n    \"\"\"\n    Plugs into PyTorch Batchsampler Package.\n    \"\"\"\n    def __init__(self, opt, image_dict, image_list):\n        self.image_dict = image_dict\n        self.image_list = image_list\n\n        self.batch_size         = opt.bs\n        self.samples_per_class  = opt.samples_per_class\n        self.sampler_length     = len(image_list)//opt.bs\n        assert self.batch_size%self.samples_per_class==0, '#Samples per class must divide batchsize!'\n\n        self.name             = 'spc_fid_batchmatch_sampler'\n        self.requires_storage = True\n\n        self.bigbs           = opt.data_batchmatch_bigbs\n        self.update_storage  = not opt.data_storage_no_update\n        self.num_batch_comps = opt.data_batchmatch_ncomps\n        self.low_proj_dim    = opt.data_sampler_lowproj_dim\n\n        self.n_jobs = 16\n\n        self.internal_image_dict = {self.image_list[i]:i for i in range(len(self.image_list))}\n\n\n    def __iter__(self):\n        for i in range(self.sampler_length):\n            # ### Random Subset from Random classes\n            # bigb_idxs = np.random.choice(len(self.storage), self.bigbs, replace=True)\n            # bigbatch  = self.storage[bigb_idxs]\n            #\n            # structured_batch = list(bigb_idxs[self.fid_match(bigbatch, batch_size=self.batch_size//self.samples_per_class)])\n            # #Add random per-class fillers to ensure that the batch is build up correctly.\n            #\n            # class_idxs = [self.image_list[idx][-1] for idx in structured_batch]\n            # for class_idx in class_idxs:\n            #     structured_batch.extend([random.choice(self.image_dict[class_idx])[-1] for _ in range(self.samples_per_class-1)])\n\n            yield self.epoch_indices[i]\n\n\n    def precompute_indices(self):\n        from joblib import Parallel, delayed\n        import time\n        ### Random Subset from Random classes\n        # self.disthist_match()\n        print('Precomputing Indices... ', end='')\n        start = time.time()\n        n_calls            = int(np.ceil(self.sampler_length/self.n_jobs))\n        self.epoch_indices = Parallel(n_jobs = self.n_jobs)(delayed(self.spc_fid_match)(n_calls, i) for i in range(self.n_jobs))\n        self.epoch_indices = [x for y in self.epoch_indices for x in y]\n        print('Done in {0:3.4f}s.'.format(time.time()-start))\n\n\n    def replace_storage_entries(self, embeddings, indices):\n        self.storage[indices] = embeddings\n\n    def create_storage(self, dataloader, model, device):\n        with torch.no_grad():\n            _ = model.eval()\n            _ = model.to(device)\n\n            embed_collect = []\n            for i,input_tuple in enumerate(tqdm(dataloader, 'Creating data storage...')):\n                embed = model(input_tuple[1].type(torch.FloatTensor).to(device))\n                if isinstance(embed, tuple): embed = embed[0]\n                embed = embed.cpu()\n                embed_collect.append(embed)\n            embed_collect = torch.cat(embed_collect, dim=0)\n            self.storage = embed_collect\n\n\n    def spc_batchfinder(self, n_samples):\n        ### SpC-Sample big batch:\n        subset, classes = [], []\n        ### Random Subset from Random classes\n        for _ in range(n_samples//self.samples_per_class):\n            class_key = random.choice(list(self.image_dict.keys()))\n            # subset.extend([(class_key, random.choice(len(self.image_dict[class_key])) for _ in range(self.samples_per_class)])\n            subset.extend([random.choice(self.image_dict[class_key])[-1] for _ in range(self.samples_per_class)])\n            classes.extend([class_key]*self.samples_per_class)\n        return np.array(subset), np.array(classes)\n\n\n    def spc_fid_match(self, calls, pos):\n        \"\"\"\n        \"\"\"\n        coll = []\n\n        for _ in range(calls):\n            bigb_data_idxs, bigb_data_classes = self.spc_batchfinder(self.bigbs)\n            bigb_dict = {}\n            for i, bigb_cls in enumerate(bigb_data_classes):\n                if bigb_cls not in bigb_dict: bigb_dict[bigb_cls] = []\n                bigb_dict[bigb_cls].append(i)\n\n            bigbatch      = self.storage[bigb_data_idxs]\n            if self.low_proj_dim>0:\n                low_dim_proj = nn.Linear(bigbatch.shape[-1],self.low_proj_dim,bias=False)\n                with torch.no_grad(): bigbatch = low_dim_proj(bigbatch)\n            bigbatch  = bigbatch.numpy()\n\n            bigbatch_mean = np.mean(bigbatch, axis=0).reshape(-1,1)\n            bigbatch_cov  = np.cov(bigbatch.T)\n\n\n            fid_collect, bigb_idxs = [], []\n\n            for _ in range(self.num_batch_comps):\n                subset_idxs = [np.random.choice(bigb_dict[np.random.choice(list(bigb_dict.keys()))], self.samples_per_class, replace=False) for _ in range(self.batch_size//self.samples_per_class)]\n                subset_idxs = [x for y in subset_idxs for x in y]\n                # subset_idxs = sorted(np.random.choice(len(bigbatch), batch_size, replace=False))\n                bigb_idxs.append(subset_idxs)\n                subset      = bigbatch[subset_idxs,:]\n\n                subset_mean = np.mean(subset, axis=0).reshape(-1,1)\n                subset_cov  = np.cov(subset.T)\n\n                diag_offset = np.eye(subset_cov.shape[0])*1e-8\n                cov_sqrt    = linalg.sqrtm((bigbatch_cov+diag_offset).dot((subset_cov+diag_offset)), disp=False)[0].real\n\n                diff = bigbatch_mean-subset_mean\n                fid  = diff.T.dot(diff) + np.trace(bigbatch_cov) + np.trace(subset_cov) - 2*np.trace(cov_sqrt)\n\n                fid_collect.append(fid)\n\n            bigb_ix      = bigb_idxs[np.argmin(fid_collect)]\n            bigb_data_ix = bigb_data_idxs[bigb_ix]\n            coll.append(bigb_data_ix)\n\n        return coll\n\n\n\n    def __len__(self):\n        return self.sampler_length\n"
  },
  {
    "path": "datasampler/greedy_coreset_sampler.py",
    "content": "import numpy as np\nimport torch, torch.nn as nn, torch.nn.functional as F\nfrom tqdm import tqdm\nimport random\nfrom scipy import linalg\n\n\n\"\"\"======================================================\"\"\"\nREQUIRES_STORAGE = True\n\n###\nclass Sampler(torch.utils.data.sampler.Sampler):\n    \"\"\"\n    Plugs into PyTorch Batchsampler Package.\n    \"\"\"\n    def __init__(self, opt, image_dict, image_list):\n        self.image_dict = image_dict\n        self.image_list = image_list\n\n        self.batch_size         = opt.bs\n        self.samples_per_class  = opt.samples_per_class\n        self.sampler_length     = len(image_list)//opt.bs\n        assert self.batch_size%self.samples_per_class==0, '#Samples per class must divide batchsize!'\n\n        self.name             = 'greedy_coreset_sampler'\n        self.requires_storage = True\n\n        self.bigbs           = opt.data_batchmatch_bigbs\n        self.update_storage  = not opt.data_storage_no_update\n        self.num_batch_comps = opt.data_batchmatch_ncomps\n        self.dist_lim        = opt.data_gc_coreset_lim\n\n        self.low_proj_dim    = opt.data_sampler_lowproj_dim\n\n        self.softened = opt.data_gc_softened\n\n        self.n_jobs = 16\n\n    def __iter__(self):\n        for i in range(self.sampler_length):\n            yield self.epoch_indices[i]\n\n\n    def precompute_indices(self):\n        from joblib import Parallel, delayed\n        import time\n\n        ### Random Subset from Random classes\n        bigb_idxs = np.random.choice(len(self.storage), self.bigbs, replace=True)\n        bigbatch  = self.storage[bigb_idxs]\n\n        print('Precomputing Indices... ', end='')\n        start = time.time()\n        def batchfinder(n_calls, pos):\n            idx_sets         = self.greedy_coreset(n_calls, pos)\n            structured_batches = [list(bigb_idxs[idx_set]) for idx_set in idx_sets]\n            # structured_batch = list(bigb_idxs[self.fid_match(bigbatch, batch_size=self.batch_size//self.samples_per_class)])\n            #Add random per-class fillers to ensure that the batch is build up correctly.\n            for i in range(len(structured_batches)):\n                class_idxs = [self.image_list[idx][-1] for idx in structured_batches[i]]\n                for class_idx in class_idxs:\n                    structured_batches[i].extend([random.choice(self.image_dict[class_idx])[-1] for _ in range(self.samples_per_class-1)])\n\n            return structured_batches\n\n        n_calls            = int(np.ceil(self.sampler_length/self.n_jobs))\n        # self.epoch_indices = batchfinder(n_calls, 0)\n        self.epoch_indices = Parallel(n_jobs = self.n_jobs)(delayed(batchfinder)(n_calls, i) for i in range(self.n_jobs))\n        self.epoch_indices = [x for y in self.epoch_indices for x in y]\n        # self.epoch_indices = Parallel(n_jobs = self.n_jobs)(delayed(batchfinder)(self.storage[np.random.choice(len(self.storage), self.bigbs, replace=True)]) for _ in tqdm(range(self.sampler_length), desc='Precomputing Indices...'))\n\n        print('Done in {0:3.4f}s.'.format(time.time()-start))\n\n\n    def replace_storage_entries(self, embeddings, indices):\n        self.storage[indices] = embeddings\n\n    def create_storage(self, dataloader, model, device):\n        with torch.no_grad():\n            _ = model.eval()\n            _ = model.to(device)\n\n            embed_collect = []\n            for i,input_tuple in enumerate(tqdm(dataloader, 'Creating data storage...')):\n                embed = model(input_tuple[1].type(torch.FloatTensor).to(device))\n                if isinstance(embed, tuple): embed = embed[0]\n                embed = embed.cpu()\n                embed_collect.append(embed)\n            embed_collect = torch.cat(embed_collect, dim=0)\n            self.storage = embed_collect\n\n\n    def full_storage_update(self, dataloader, model, device):\n        with torch.no_grad():\n            _ = model.eval()\n            _ = model.to(device)\n\n            embed_collect = []\n            for i,input_tuple in enumerate(tqdm(dataloader, 'Creating data storage...')):\n                embed = model(input_tuple[1].type(torch.FloatTensor).to(device))\n                if isinstance(embed, tuple): embed = embed[0]\n                embed = embed.cpu()\n                embed_collect.append(embed)\n            embed_collect = torch.cat(embed_collect, dim=0)\n            if self.mb_mom>0:\n                self.delta_storage = self.mb_mom*self.delta_storage + (1-self.mb_mom)*(embed_collect-self.storage)\n                self.storage       = embed_collect + self.mb_lr*self.delta_storage\n            else:\n                self.storage = embed_collect\n                \n    def greedy_coreset(self, calls, pos):\n        \"\"\"\n        \"\"\"\n        coll = []\n\n\n        for _ in range(calls):\n            bigbatch   = self.storage[np.random.choice(len(self.storage), self.bigbs, replace=False)]\n            batch_size = self.batch_size//self.samples_per_class\n\n            if self.low_proj_dim>0:\n                low_dim_proj = nn.Linear(bigbatch.shape[-1],self.low_proj_dim,bias=False)\n                with torch.no_grad(): bigbatch = low_dim_proj(bigbatch)\n\n            bigbatch = bigbatch.numpy()\n\n            prod        = np.matmul(bigbatch, bigbatch.T)\n            sq          = prod.diagonal().reshape(bigbatch.shape[0], 1)\n            dist_matrix = np.clip(-2*prod + sq + sq.T, 0, None)\n            coreset_anchor_dists = np.linalg.norm(dist_matrix, axis=1)\n\n            k, sampled_indices = 0, []\n\n            while k<batch_size:\n                if k==0:\n                    no    = np.random.randint(len(coreset_anchor_dists))\n                else:\n                    if self.softened:\n                        no = np.random.choice(np.where(coreset_anchor_dists>=np.percentile(coreset_anchor_dists,97))[0])\n                    else:\n                        no    = np.argmax(coreset_anchor_dists)\n\n                sampled_indices.append(no)\n                add_d  = dist_matrix[:, no:no+1]\n                #If its closer to the remaining points than the new addition/additions, sample it.\n                new_dj = np.concatenate([np.expand_dims(coreset_anchor_dists,-1), add_d], axis=1)\n                coreset_anchor_dists = np.min(new_dj, axis=1)\n                k += 1\n\n            coll.append(sampled_indices)\n\n        return coll\n\n\n    def __len__(self):\n        return self.sampler_length\n"
  },
  {
    "path": "datasampler/random_sampler.py",
    "content": "import numpy as np\nimport torch, torch.nn as nn, torch.nn.functional as F\nfrom tqdm import tqdm\nimport random\n\n\n\n\"\"\"======================================================\"\"\"\nREQUIRES_STORAGE = False\n\n###\nclass Sampler(torch.utils.data.sampler.Sampler):\n    \"\"\"\n    Plugs into PyTorch Batchsampler Package.\n    \"\"\"\n    def __init__(self, opt, image_dict, image_list=None):\n        self.image_dict         = image_dict\n        self.image_list         = image_list\n\n        self.batch_size         = opt.bs\n        self.samples_per_class  = opt.samples_per_class\n        self.sampler_length     = len(image_list)//opt.bs\n        assert self.batch_size%self.samples_per_class==0, '#Samples per class must divide batchsize!'\n\n        self.name             = 'random_sampler'\n        self.requires_storage = False\n\n    def __iter__(self):\n        for _ in range(self.sampler_length):\n            subset = []\n            ### Random Subset from Random classes\n            for _ in range(self.batch_size-1):\n                class_key  = random.choice(list(self.image_dict.keys()))\n                sample_idx = np.random.choice(len(self.image_dict[class_key]))\n                subset.append(self.image_dict[class_key][sample_idx][-1])\n            #\n            subset.append(random.choice(self.image_dict[self.image_list[random.choice(subset)][-1]])[-1])\n            yield subset\n\n    def __len__(self):\n        return self.sampler_length\n"
  },
  {
    "path": "datasampler/samplers.py",
    "content": "import numpy as np\nimport torch, torch.nn as nn, torch.nn.functional as F\nfrom tqdm import tqdm\nimport random\n\n\n\"\"\"======================================================\"\"\"\ndef sampler_parse_args(parser):\n    parser.add_argument('--batch_selection',     default='class_random', type=str,   help='Selection of the data batch: Modes of Selection: random, greedy_coreset')\n    parser.add_argument('--primary_subset_perc', default=0.1,            type=float, help='Size of the randomly selected subset before application of coreset selection.')\n    return parser\n\n\n\n\"\"\"======================================================\"\"\"\n###\n# Methods: Full random, Per-Class-Random, CoreSet\nclass AdvancedSampler(torch.utils.data.sampler.Sampler):\n    \"\"\"\n    Plugs into PyTorch Batchsampler Package.\n    \"\"\"\n    def __init__(self, method='class_random', random_subset_perc=0.1, batch_size=128, samples_per_class=4):\n        self.random_subset_perc = random_subset_perc\n        self.batch_size = batch_size\n        self.samples_per_class = samples_per_class\n\n        self.method         = method\n\n        self.storage = None\n        self.sampler_length = None\n\n        self.methods_requiring_storage = ['greedy_class_coreset', 'greedy_semi_class_coreset', 'presampled_infobatch']\n\n    def create_storage(self, dataloader, model, device):\n        self.image_dict = dataloader.dataset.image_dict\n        self.image_list = dataloader.dataset.image_list\n\n        self.sampler_length = len(dataloader.dataset)//self.batch_size\n\n        if self.method in self.methods_requiring_storage:\n            with torch.no_grad():\n                _ = model.eval()\n                _ = model.to(device)\n\n                embed_collect = []\n                for i,input_tuple in enumerate(tqdm(dataloader, 'Creating data storage...')):\n                    embed = model(input_tuple[1].type(torch.FloatTensor).to(device)).cpu()\n                    embed_collect.append(embed)\n                embed_collect = torch.cat(embed_collect, dim=0)\n                self.storage = embed_collect\n\n            self.random_subset_len = int(self.random_subset_perc*len(self.storage))\n\n    def update_storage(self, embeddings, indices):\n        if 'coreset' in self.method:\n            self.storage[indices] = embeddings\n\n    def __iter__(self):\n        for _ in range(self.sampler_length):\n            subset = []\n            if self.method=='greedy_class_coreset':\n                for _ in range(self.batch_size//self.samples_per_class):\n                    class_key     = random.choice(list(self.image_dict.keys()))\n                    class_indices = np.array([x[1] for x in self.image_dict[class_key]])\n                    # print(class_indices)\n                    ### Coreset subset of subset\n                    subset.extend(class_indices[self.greedy_coreset(self.storage[class_indices], self.samples_per_class)])\n                # print([self.image_list[x][1] for x in subset])\n            elif self.method=='greedy_semi_class_coreset':\n                ### Big random subset\n                subset = np.random.randint(0,len(self.storage),self.random_subset_len)\n                ### Coreset subset of subset of half the batch size\n                subset  = subset[self.greedy_coreset(self.storage[subset], self.batch_size//2)]\n                ### Fill the rest of the batch with random samples from each coreset member class\n                subset = list(subset)+[random.choice(self.image_dict[self.image_list[idx][-1]])[-1] for idx in subset]\n            elif self.method=='presampled_infobatch':\n                ### Big random subset\n                subset  = np.random.randint(0,len(self.storage),self.random_subset_len)\n                classes = torch.tensor([self.image_list[idx][-1] for idx in subset])\n                ### Presampled Infobatch for subset of data.\n                subset = subset[self.presample_infobatch(classes, self.storage[subset], self.batch_size//2)]\n                ### Fill the rest of the batch with random samples from each member class\n                subset = list(subset)+[random.choice(self.image_dict[self.image_list[idx][-1]])[-1] for idx in subset]\n            elif self.method=='class_random':\n                ### Random Subset from Random classes\n                for _ in range(self.batch_size//self.samples_per_class):\n                    class_key = random.choice(list(self.image_dict.keys()))\n                    subset.extend([random.choice(self.image_dict[class_key])[-1] for _ in range(self.samples_per_class)])\n            elif self.method=='semi_class_random':\n                ### Select half of the indices completely at random, and the other half corresponding to the classes.\n                for _ in range(self.batch_size//2):\n                    rand_idx       = np.random.randint(len(self.image_list))\n                    class_idx      = self.image_list[rand_idx][-1]\n                    rand_class_idx = random.choice(self.image_dict[class_idx])[-1]\n                    subset.extend([rand_idx, rand_class_idx])\n            else:\n                raise NotImplementedError('Batch selection method {} not available!'.format(self.method))\n            yield subset\n\n    def __len__(self):\n        return self.sampler_length\n\n    def pdistsq(self, A):\n        prod = torch.mm(A, A.t())\n        diag = prod.diag().unsqueeze(1).expand_as(prod)\n        return (-2*prod + diag + diag.T)\n\n    def greedy_coreset(self, A, samples):\n        dist_matrix          = self.pdistsq(A)\n        coreset_anchor_dists = torch.norm(dist_matrix, dim=1)\n\n        sampled_indices, i = [], 0\n\n        while i<samples:\n            if i==0:\n                sample_idx = np.random.randint(len(coreset_anchor_dists))\n            else:\n                sample_idx = torch.argmax(coreset_anchor_dists).item()\n            sampled_indices.append(sample_idx)\n            sample_anchor_dists  = dist_matrix[:, sample_idx:sample_idx+1]\n            new_search_dists     = torch.cat([coreset_anchor_dists.unsqueeze(-1), sample_anchor_dists], dim=1)\n            coreset_anchor_dists = torch.min(new_search_dists, dim=1)[0]\n            i += 1\n\n        return sampled_indices\n\n    def presample_infobatch(self, classes, A, samples):\n        equiv_classes = ((classes.reshape(-1,1)-classes.reshape(-1,1).T)==0).type(torch.BoolTensor)\n\n        dim  = A.shape[-1]\n\n        dist = self.pdistsq(A).clamp(min=0.5)\n        dist = ((2.0 - float(dim)) * torch.log(dist) - (float(dim-3) / 2) * torch.log(1.0 - 0.25 * (dist.pow(2))))\n        dist[equiv_classes] = 0\n        dist = torch.exp(dist - torch.max(dist))\n        dist[equiv_classes] = 0\n\n        dist = dist/torch.sum(dist)\n        dist = dist.flatten().detach().cpu().numpy()\n\n        sampled_idxs = set()\n        while len(sampled_idxs)<samples:\n            index = np.random.choice(len(dist),p=dist)\n            ### Ensure that we do not continously sample the same one in the case of high prob. imbalances!\n            dist[index] = 0\n            dist = dist/np.sum(dist)\n            sample_a, sample_b = index//equiv_classes.shape[0], index%equiv_classes.shape[1]\n            sampled_idxs = sampled_idxs.union(set([sample_a, sample_b]))\n\n        sampled_idxs = list(sampled_idxs)\n        sampled_idxs = sampled_idxs[:samples]\n\n        return sampled_idxs\n        # dist = dist/torch.sum(dist, dim=1).view(-1,1)\n\n        # Divergence of dist from uniform:, normalize with number if NON-ZERO ELEMENTS!\n        # non_equiv_classes = (1-equiv_classes.type(torch.LongTensor)).type(torch.BoolTensor)\n        # uniform_reference = torch.ones(dist.shape)/torch.sum(non_equiv_classes, dim=1).view(-1,1)\n        #\n        # kl_divs = []\n        # for i in range(len(dist)):\n        #     nec_idxs = non_equiv_classes[i,:].type(torch.BoolTensor)\n        #     u        = uniform_reference[i,nec_idxs]\n        #     d        = dist[i,nec_idxs]\n        #     kl_div   = -((u*(d/u).log()).sum())/len(u)\n        #     kl_divs.append(kl_div)\n        # kl_divs      = torch.stack(kl_divs, dim=0)\n        # sample_probs = (kl_divs/kl_divs.sum()).detach().cpu().numpy()\n\n\n        # return np.random.choice(len(sample_probs), samples, p=sample_probs)\n"
  },
  {
    "path": "datasets/__init__.py",
    "content": "import datasets.cub200\nimport datasets.cars196\nimport datasets.stanford_online_products\n\n\ndef select(dataset, opt, data_path):\n    if 'cub200' in dataset:\n        return cub200.Give(opt, data_path)\n\n    if 'cars196' in dataset:\n        return cars196.Give(opt, data_path)\n\n    if 'online_products' in dataset:\n        return stanford_online_products.Give(opt, data_path)\n\n    raise NotImplementedError('A dataset for {} is currently not implemented.\\n\\\n                               Currently available are : cub200, cars196 & online_products!'.format(dataset))\n"
  },
  {
    "path": "datasets/basic_dataset_scaffold.py",
    "content": "from torch.utils.data import Dataset\nimport torchvision.transforms as transforms\nimport numpy as np\nfrom PIL import Image\n\n\n\"\"\"===================================================================================================\"\"\"\n################## BASIC PYTORCH DATASET USED FOR ALL DATASETS ##################################\nclass BaseDataset(Dataset):\n    def __init__(self, image_dict, opt, is_validation=False):\n        self.is_validation = is_validation\n        self.pars          = opt\n\n        #####\n        self.image_dict = image_dict\n\n        #####\n        self.init_setup()\n\n\n        #####\n        if 'bninception' not in opt.arch:\n            self.f_norm = normalize = transforms.Normalize(mean=[0.485, 0.456, 0.406],std=[0.229, 0.224, 0.225])\n        else:\n            # normalize = transforms.Normalize(mean=[0.502, 0.4588, 0.4078],std=[1., 1., 1.])\n            self.f_norm = normalize = transforms.Normalize(mean=[0.502, 0.4588, 0.4078],std=[0.0039, 0.0039, 0.0039])\n\n        transf_list = []\n\n        self.crop_size = crop_im_size = 224 if 'googlenet' not in opt.arch else 227\n        if opt.augmentation=='big':\n            crop_im_size = 256\n\n        #############\n        self.normal_transform = []\n        if not self.is_validation:\n            if opt.augmentation=='base' or opt.augmentation=='big':\n                self.normal_transform.extend([transforms.RandomResizedCrop(size=crop_im_size), transforms.RandomHorizontalFlip(0.5)])\n            elif opt.augmentation=='adv':\n                self.normal_transform.extend([transforms.RandomResizedCrop(size=crop_im_size), transforms.RandomGrayscale(p=0.2),\n                                              transforms.ColorJitter(0.2, 0.2, 0.2, 0.2), transforms.RandomHorizontalFlip(0.5)])\n            elif opt.augmentation=='red':\n                self.normal_transform.extend([transforms.Resize(size=256), transforms.RandomCrop(crop_im_size), transforms.RandomHorizontalFlip(0.5)])\n        else:\n            self.normal_transform.extend([transforms.Resize(256), transforms.CenterCrop(crop_im_size)])\n        self.normal_transform.extend([transforms.ToTensor(), normalize])\n        self.normal_transform = transforms.Compose(self.normal_transform)\n\n\n    def init_setup(self):\n        self.n_files       = np.sum([len(self.image_dict[key]) for key in self.image_dict.keys()])\n        self.avail_classes = sorted(list(self.image_dict.keys()))\n\n\n        counter = 0\n        temp_image_dict = {}\n        for i,key in enumerate(self.avail_classes):\n            temp_image_dict[key] = []\n            for path in self.image_dict[key]:\n                temp_image_dict[key].append([path, counter])\n                counter += 1\n\n        self.image_dict = temp_image_dict\n        self.image_list = [[(x[0],key) for x in self.image_dict[key]] for key in self.image_dict.keys()]\n        self.image_list = [x for y in self.image_list for x in y]\n\n        self.image_paths = self.image_list\n\n        self.is_init = True\n\n\n    def ensure_3dim(self, img):\n        if len(img.size)==2:\n            img = img.convert('RGB')\n        return img\n\n\n    def __getitem__(self, idx):\n        input_image = self.ensure_3dim(Image.open(self.image_list[idx][0]))\n\n        ### Basic preprocessing.\n        im_a = self.normal_transform(input_image)\n        if 'bninception' in self.pars.arch:\n            im_a = im_a[range(3)[::-1],:]\n        return self.image_list[idx][-1], im_a, idx\n\n\n    def __len__(self):\n        return self.n_files\n"
  },
  {
    "path": "datasets/cars196.py",
    "content": "from datasets.basic_dataset_scaffold import BaseDataset\nimport os\n\n\ndef Give(opt, datapath):\n    image_sourcepath  = datapath+'/images'\n    image_classes     = sorted([x for x in os.listdir(image_sourcepath)])\n    total_conversion  = {i:x for i,x in enumerate(image_classes)}\n    image_list    = {i:sorted([image_sourcepath+'/'+key+'/'+x for x in os.listdir(image_sourcepath+'/'+key)]) for i,key in enumerate(image_classes)}\n    image_list    = [[(key,img_path) for img_path in image_list[key]] for key in image_list.keys()]\n    image_list    = [x for y in image_list for x in y]\n\n    ### Dictionary of structure class:list_of_samples_with_said_class\n    image_dict    = {}\n    for key, img_path in image_list:\n        if not key in image_dict.keys():\n            image_dict[key] = []\n        image_dict[key].append(img_path)\n\n    ### Use the first half of the sorted data as training and the second half as test set\n    keys = sorted(list(image_dict.keys()))\n    train,test      = keys[:len(keys)//2], keys[len(keys)//2:]\n\n    ### If required, split the training data into a train/val setup either by or per class.\n    if opt.use_tv_split:\n        if not opt.tv_split_by_samples:\n            train_val_split = int(len(train)*opt.tv_split_perc)\n            train, val      = train[:train_val_split], train[train_val_split:]\n            ###\n            train_image_dict = {i:image_dict[key] for i,key in enumerate(train)}\n            val_image_dict   = {i:image_dict[key] for i,key in enumerate(val)}\n            test_image_dict  = {i:image_dict[key] for i,key in enumerate(test)}\n        else:\n            val = train\n            train_image_dict, val_image_dict = {},{}\n            for key in train:\n                train_ixs = np.random.choice(len(image_dict[key]), int(len(image_dict[key])*opt.tv_split_perc), replace=False)\n                val_ixs   = np.array([x for x in range(len(image_dict[key])) if x not in train_ixs])\n                train_image_dict[key] = np.array(image_dict[key])[train_ixs]\n                val_image_dict[key]   = np.array(image_dict[key])[val_ixs]\n        val_dataset   = BaseDataset(val_image_dict,   opt, is_validation=True)\n        val_conversion = {i:total_conversion[key] for i,key in enumerate(val)}\n        ###\n        val_dataset.conversion   = val_conversion\n    else:\n        train_image_dict = {key:image_dict[key] for key in train}\n        val_image_dict   = None\n        val_dataset      = None\n\n    ###\n    train_conversion = {i:total_conversion[key] for i,key in enumerate(train)}\n    test_conversion  = {i:total_conversion[key] for i,key in enumerate(test)}\n\n    ###\n    test_image_dict = {key:image_dict[key] for key in test}\n\n    ###\n    print('\\nDataset Setup:\\nUsing Train-Val Split: {0}\\n#Classes: Train ({1}) | Val ({2}) | Test ({3})\\n'.format(opt.use_tv_split, len(train_image_dict), len(val_image_dict) if val_image_dict else 'X', len(test_image_dict)))\n\n    ###\n    train_dataset       = BaseDataset(train_image_dict, opt)\n    test_dataset        = BaseDataset(test_image_dict,  opt, is_validation=True)\n    eval_dataset        = BaseDataset(train_image_dict, opt, is_validation=True)\n    eval_train_dataset  = BaseDataset(train_image_dict, opt, is_validation=False)\n    train_dataset.conversion       = train_conversion\n    test_dataset.conversion        = test_conversion\n    eval_dataset.conversion        = test_conversion\n    eval_train_dataset.conversion  = train_conversion\n\n    return {'training':train_dataset, 'validation':val_dataset, 'testing':test_dataset, 'evaluation':eval_dataset, 'evaluation_train':eval_train_dataset}\n"
  },
  {
    "path": "datasets/cub200.py",
    "content": "from datasets.basic_dataset_scaffold import BaseDataset\nimport os\n\ndef Give(opt, datapath):\n    image_sourcepath  = datapath+'/images'\n    image_classes     = sorted([x for x in os.listdir(image_sourcepath) if '._' not in x], key=lambda x: int(x.split('.')[0]))\n    total_conversion  = {int(x.split('.')[0])-1:x.split('.')[-1] for x in image_classes}\n    image_list        = {int(key.split('.')[0])-1:sorted([image_sourcepath+'/'+key+'/'+x for x in os.listdir(image_sourcepath+'/'+key) if '._' not in x]) for key in image_classes}\n    image_list        = [[(key,img_path) for img_path in image_list[key]] for key in image_list.keys()]\n    image_list        = [x for y in image_list for x in y]\n\n    ### Dictionary of structure class:list_of_samples_with_said_class\n    image_dict    = {}\n    for key, img_path in image_list:\n        if not key in image_dict.keys():\n            image_dict[key] = []\n        image_dict[key].append(img_path)\n\n    ### Use the first half of the sorted data as training and the second half as test set\n    keys = sorted(list(image_dict.keys()))\n    train,test      = keys[:len(keys)//2], keys[len(keys)//2:]\n\n    ### If required, split the training data into a train/val setup either by or per class.\n    if opt.use_tv_split:\n        if not opt.tv_split_by_samples:\n            train_val_split = int(len(train)*opt.tv_split_perc)\n            train, val      = train[:train_val_split], train[train_val_split:]\n            ###\n            train_image_dict = {i:image_dict[key] for i,key in enumerate(train)}\n            val_image_dict   = {i:image_dict[key] for i,key in enumerate(val)}\n            test_image_dict  = {i:image_dict[key] for i,key in enumerate(test)}\n        else:\n            val = train\n            train_image_dict, val_image_dict = {},{}\n            for key in train:\n                train_ixs   = np.array(list(set(np.round(np.linspace(0,len(image_dict[key])-1,int(len(image_dict[key])*opt.tv_split_perc)))))).astype(int)\n                val_ixs     = np.array([x for x in range(len(image_dict[key])) if x not in train_ixs])\n                train_image_dict[key] = np.array(image_dict[key])[train_ixs]\n                val_image_dict[key]   = np.array(image_dict[key])[val_ixs]\n        val_dataset    = BaseDataset(val_image_dict, opt, is_validation=True)\n        val_conversion = {i:total_conversion[key] for i,key in enumerate(val)}\n        ###\n        val_dataset.conversion   = val_conversion\n    else:\n        train_image_dict = {key:image_dict[key] for key in train}\n        val_image_dict   = None\n        val_dataset      = None\n\n    ###\n    train_conversion = {i:total_conversion[key] for i,key in enumerate(train)}\n    test_conversion  = {i:total_conversion[key] for i,key in enumerate(test)}\n\n    ###\n    test_image_dict = {key:image_dict[key] for key in test}\n\n    ###\n    print('\\nDataset Setup:\\nUsing Train-Val Split: {0}\\n#Classes: Train ({1}) | Val ({2}) | Test ({3})\\n'.format(opt.use_tv_split, len(train_image_dict), len(val_image_dict) if val_image_dict else 'X', len(test_image_dict)))\n\n    ###\n    train_dataset       = BaseDataset(train_image_dict, opt)\n    test_dataset        = BaseDataset(test_image_dict,  opt, is_validation=True)\n    eval_dataset        = BaseDataset(train_image_dict, opt, is_validation=True)\n    eval_train_dataset  = BaseDataset(train_image_dict, opt, is_validation=False)\n    train_dataset.conversion       = train_conversion\n    test_dataset.conversion        = test_conversion\n    eval_dataset.conversion        = test_conversion\n    eval_train_dataset.conversion  = train_conversion\n\n    return {'training':train_dataset, 'validation':val_dataset, 'testing':test_dataset, 'evaluation':eval_dataset, 'evaluation_train':eval_train_dataset}\n"
  },
  {
    "path": "datasets/stanford_online_products.py",
    "content": "from datasets.basic_dataset_scaffold import BaseDataset\nimport os, numpy as np\nimport pandas as pd\n\n\ndef Give(opt, datapath):\n    image_sourcepath  = opt.source_path+'/images'\n    training_files = pd.read_table(opt.source_path+'/Info_Files/Ebay_train.txt', header=0, delimiter=' ')\n    test_files     = pd.read_table(opt.source_path+'/Info_Files/Ebay_test.txt', header=0, delimiter=' ')\n\n    spi   = np.array([(a,b) for a,b in zip(training_files['super_class_id'], training_files['class_id'])])\n    super_dict       = {}\n    super_conversion = {}\n    for i,(super_ix, class_ix, image_path) in enumerate(zip(training_files['super_class_id'],training_files['class_id'],training_files['path'])):\n        if super_ix not in super_dict: super_dict[super_ix] = {}\n        if class_ix not in super_dict[super_ix]: super_dict[super_ix][class_ix] = []\n        super_dict[super_ix][class_ix].append(image_sourcepath+'/'+image_path)\n\n    if opt.use_tv_split:\n        if not opt.tv_split_by_samples:\n            train_image_dict, val_image_dict = {},{}\n            train_count, val_count = 0, 0\n            for super_ix in super_dict.keys():\n                class_ixs       = sorted(list(super_dict[super_ix].keys()))\n                train_val_split = int(len(super_dict[super_ix])*opt.tv_split_perc)\n                train_image_dict[super_ix] = {}\n                for _,class_ix in enumerate(class_ixs[:train_val_split]):\n                    train_image_dict[super_ix][train_count] = super_dict[super_ix][class_ix]\n                    train_count += 1\n                val_image_dict[super_ix] = {}\n                for _,class_ix in enumerate(class_ixs[train_val_split:]):\n                    val_image_dict[super_ix][val_count]     = super_dict[super_ix][class_ix]\n                    val_count += 1\n        else:\n            train_image_dict, val_image_dict = {},{}\n            for super_ix in super_dict.keys():\n                class_ixs       = sorted(list(super_dict[super_ix].keys()))\n                train_image_dict[super_ix] = {}\n                val_image_dict[super_ix]   = {}\n                for class_ix in class_ixs:\n                    train_val_split = int(len(super_dict[super_ix][class_ix])*opt.tv_split_perc)\n                    train_image_dict[super_ix][class_ix] = super_dict[super_ix][class_ix][:train_val_split]\n                    val_image_dict[super_ix][class_ix]   = super_dict[super_ix][class_ix][train_val_split:]\n    else:\n        train_image_dict = super_dict\n        val_image_dict   = None\n\n    ####\n    test_image_dict        = {}\n    train_image_dict_temp  = {}\n    val_image_dict_temp    = {}\n    super_train_image_dict = {}\n    super_val_image_dict   = {}\n    train_conversion       = {}\n    super_train_conversion = {}\n    val_conversion         = {}\n    super_val_conversion   = {}\n    test_conversion        = {}\n    super_test_conversion  = {}\n\n    ## Create Training Dictionaries\n    i = 0\n    for super_ix,super_set in train_image_dict.items():\n        super_ix -= 1\n        counter   = 0\n        super_train_image_dict[super_ix] = []\n        for class_ix,class_set in super_set.items():\n            class_ix -= 1\n            super_train_image_dict[super_ix].extend(class_set)\n            train_image_dict_temp[class_ix] = class_set\n            if class_ix not in train_conversion:\n                train_conversion[class_ix] = class_set[0].split('/')[-1].split('_')[0]\n                super_conversion[class_ix] = class_set[0].split('/')[-2]\n            counter += 1\n            i       += 1\n    train_image_dict = train_image_dict_temp\n\n    ## Create Validation Dictionaries\n    if opt.use_tv_split:\n        i = 0\n        for super_ix,super_set in val_image_dict.items():\n            super_ix -= 1\n            counter   = 0\n            super_val_image_dict[super_ix] = []\n            for class_ix,class_set in super_set.items():\n                class_ix -= 1\n                super_val_image_dict[super_ix].extend(class_set)\n                val_image_dict_temp[class_ix] = class_set\n                if class_ix not in val_conversion:\n                    val_conversion[class_ix] = class_set[0].split('/')[-1].split('_')[0]\n                    super_conversion[class_ix] = class_set[0].split('/')[-2]\n                counter += 1\n                i       += 1\n        val_image_dict = val_image_dict_temp\n    else:\n        val_image_dict = None\n\n    ## Create Test Dictioniaries\n    for class_ix, img_path in zip(test_files['class_id'],test_files['path']):\n        class_ix = class_ix-1\n        if not class_ix in test_image_dict.keys():\n            test_image_dict[class_ix] = []\n        test_image_dict[class_ix].append(image_sourcepath+'/'+img_path)\n        test_conversion[class_ix]       = img_path.split('/')[-1].split('_')[0]\n        super_test_conversion[class_ix] = img_path.split('/')[-2]\n\n    ##\n    if val_image_dict:\n        val_dataset            = BaseDataset(val_image_dict,   opt, is_validation=True)\n        val_dataset.conversion = val_conversion\n    else:\n        val_dataset = None\n\n    print('\\nDataset Setup:\\nUsing Train-Val Split: {0}\\n#Classes: Train ({1}) | Val ({2}) | Test ({3})\\n'.format(opt.use_tv_split, len(train_image_dict), len(val_image_dict) if val_image_dict else 'X', len(test_image_dict)))\n\n    super_train_dataset = BaseDataset(super_train_image_dict, opt, is_validation=True)\n    train_dataset       = BaseDataset(train_image_dict, opt)\n    test_dataset        = BaseDataset(test_image_dict,  opt, is_validation=True)\n    eval_dataset        = BaseDataset(train_image_dict, opt, is_validation=True)\n    eval_train_dataset  = BaseDataset(train_image_dict, opt)\n\n    super_train_dataset.conversion = super_train_conversion\n    train_dataset.conversion       = train_conversion\n    test_dataset.conversion        = test_conversion\n    eval_dataset.conversion        = train_conversion\n\n    return {'training':train_dataset, 'validation':val_dataset, 'testing':test_dataset, 'evaluation':eval_dataset, 'evaluation_train':eval_train_dataset, 'super_evaluation':super_train_dataset}\n"
  },
  {
    "path": "evaluation/__init__.py",
    "content": "import faiss, matplotlib.pyplot as plt, os, numpy as np, torch\nfrom PIL import Image\n\n\n\n#######################\ndef evaluate(dataset, LOG, metric_computer, dataloaders, model, opt, evaltypes, device,\n             aux_store=None, make_recall_plot=False, store_checkpoints=True, log_key='Test'):\n    \"\"\"\n    Parent-Function to compute evaluation metrics, print summary string and store checkpoint files/plot sample recall plots.\n    \"\"\"\n    computed_metrics, extra_infos = metric_computer.compute_standard(opt, model, dataloaders[0], evaltypes, device)\n\n    numeric_metrics = {}\n    histogr_metrics = {}\n    for main_key in computed_metrics.keys():\n        for name,value in computed_metrics[main_key].items():\n            if isinstance(value, np.ndarray):\n                if main_key not in histogr_metrics: histogr_metrics[main_key] = {}\n                histogr_metrics[main_key][name] = value\n            else:\n                if main_key not in numeric_metrics: numeric_metrics[main_key] = {}\n                numeric_metrics[main_key][name] = value\n\n    ###\n    full_result_str = ''\n    for evaltype in numeric_metrics.keys():\n        full_result_str += 'Embed-Type: {}:\\n'.format(evaltype)\n        for i,(metricname, metricval) in enumerate(numeric_metrics[evaltype].items()):\n            full_result_str += '{0}{1}: {2:4.4f}'.format(' | ' if i>0 else '',metricname, metricval)\n        full_result_str += '\\n'\n\n    print(full_result_str)\n\n\n    ###\n    for evaltype in evaltypes:\n        for storage_metric in opt.storage_metrics:\n            parent_metric = evaltype+'_{}'.format(storage_metric.split('@')[0])\n            if parent_metric not in LOG.progress_saver[log_key].groups.keys() or \\\n               numeric_metrics[evaltype][storage_metric]>np.max(LOG.progress_saver[log_key].groups[parent_metric][storage_metric]['content']):\n               print('Saved weights for best {}: {}\\n'.format(log_key, parent_metric))\n               set_checkpoint(model, opt, LOG.progress_saver, LOG.prop.save_path+'/checkpoint_{}_{}_{}.pth.tar'.format(log_key, evaltype, storage_metric), aux=aux_store)\n\n\n    ###\n    if opt.log_online:\n        for evaltype in histogr_metrics.keys():\n            for eval_metric, hist in histogr_metrics[evaltype].items():\n                import wandb, numpy\n                wandb.log({log_key+': '+evaltype+'_{}'.format(eval_metric): wandb.Histogram(np_histogram=(list(hist),list(np.arange(len(hist)+1))))}, step=opt.epoch)\n                wandb.log({log_key+': '+evaltype+'_LOG-{}'.format(eval_metric): wandb.Histogram(np_histogram=(list(np.log(hist)+20),list(np.arange(len(hist)+1))))}, step=opt.epoch)\n\n    ###\n    for evaltype in numeric_metrics.keys():\n        for eval_metric in numeric_metrics[evaltype].keys():\n            parent_metric = evaltype+'_{}'.format(eval_metric.split('@')[0])\n            LOG.progress_saver[log_key].log(eval_metric, numeric_metrics[evaltype][eval_metric],  group=parent_metric)\n\n        ###\n        if make_recall_plot:\n            recover_closest_standard(extra_infos[evaltype]['features'],\n                                     extra_infos[evaltype]['image_paths'],\n                                     LOG.prop.save_path+'/sample_recoveries.png')\n\n\n###########################\ndef set_checkpoint(model, opt, progress_saver, savepath, aux=None):\n    if 'experiment' in vars(opt):\n        import argparse\n        save_opt = {key:item for key,item in vars(opt).items() if key!='experiment'}\n        save_opt = argparse.Namespace(**save_opt)\n    else:\n        save_opt = opt\n\n    torch.save({'state_dict':model.state_dict(), 'opt':save_opt, 'progress':progress_saver, 'aux':aux}, savepath)\n\n\n\n\n##########################\ndef recover_closest_standard(feature_matrix_all, image_paths, save_path, n_image_samples=10, n_closest=3):\n    image_paths = np.array([x[0] for x in image_paths])\n    sample_idxs = np.random.choice(np.arange(len(feature_matrix_all)), n_image_samples)\n\n    faiss_search_index = faiss.IndexFlatL2(feature_matrix_all.shape[-1])\n    faiss_search_index.add(feature_matrix_all)\n    _, closest_feature_idxs = faiss_search_index.search(feature_matrix_all, n_closest+1)\n\n    sample_paths = image_paths[closest_feature_idxs][sample_idxs]\n\n    f,axes = plt.subplots(n_image_samples, n_closest+1)\n    for i,(ax,plot_path) in enumerate(zip(axes.reshape(-1), sample_paths.reshape(-1))):\n        ax.imshow(np.array(Image.open(plot_path)))\n        ax.set_xticks([])\n        ax.set_yticks([])\n        if i%(n_closest+1):\n            ax.axvline(x=0, color='g', linewidth=13)\n        else:\n            ax.axvline(x=0, color='r', linewidth=13)\n    f.set_size_inches(10,20)\n    f.tight_layout()\n    f.savefig(save_path)\n    plt.close()\n"
  },
  {
    "path": "main.py",
    "content": "\"\"\"===================================================================================================\"\"\"\n################### LIBRARIES ###################\n### Basic Libraries\nimport warnings\nwarnings.filterwarnings(\"ignore\")\n\nimport os, sys, numpy as np, argparse, imp, datetime, pandas as pd, copy\nimport time, pickle as pkl, random, json, collections\nimport matplotlib\nmatplotlib.use('agg')\nimport matplotlib.pyplot as plt\n\nfrom tqdm import tqdm\n\nimport parameters    as par\n\n\n\"\"\"===================================================================================================\"\"\"\n################### INPUT ARGUMENTS ###################\nparser = argparse.ArgumentParser()\n\nparser = par.basic_training_parameters(parser)\nparser = par.batch_creation_parameters(parser)\nparser = par.batchmining_specific_parameters(parser)\nparser = par.loss_specific_parameters(parser)\nparser = par.wandb_parameters(parser)\n\n##### Read in parameters\nopt = parser.parse_args()\n\n\n\"\"\"===================================================================================================\"\"\"\n### The following setting is useful when logging to wandb and running multiple seeds per setup:\n### By setting the savename to <group_plus_seed>, the savename will instead comprise the group and the seed!\nif opt.savename=='group_plus_seed':\n    if opt.log_online:\n        opt.savename = opt.group+'_s{}'.format(opt.seed)\n    else:\n        opt.savename = ''\n\n### If wandb-logging is turned on, initialize the wandb-run here:\nif opt.log_online:\n    import wandb\n    _ = os.system('wandb login {}'.format(opt.wandb_key))\n    os.environ['WANDB_API_KEY'] = opt.wandb_key\n    wandb.init(project=opt.project, group=opt.group, name=opt.savename, dir=opt.save_path)\n    wandb.config.update(opt)\n\n\n\n\"\"\"===================================================================================================\"\"\"\n### Load Remaining Libraries that neeed to be loaded after comet_ml\nimport torch, torch.nn as nn\nimport torch.multiprocessing\ntorch.multiprocessing.set_sharing_strategy('file_system')\nimport architectures as archs\nimport datasampler   as dsamplers\nimport datasets      as datasets\nimport criteria      as criteria\nimport metrics       as metrics\nimport batchminer    as bmine\nimport evaluation    as eval\nfrom utilities import misc\nfrom utilities import logger\n\n\n\n\"\"\"===================================================================================================\"\"\"\nfull_training_start_time = time.time()\n\n\n\n\"\"\"===================================================================================================\"\"\"\nopt.source_path += '/'+opt.dataset\nopt.save_path   += '/'+opt.dataset\n\n#Assert that the construction of the batch makes sense, i.e. the division into class-subclusters.\nassert not opt.bs%opt.samples_per_class, 'Batchsize needs to fit number of samples per class for distance sampling and margin/triplet loss!'\n\nopt.pretrained = not opt.not_pretrained\n\n\n\n\n\"\"\"===================================================================================================\"\"\"\n################### GPU SETTINGS ###########################\nos.environ[\"CUDA_DEVICE_ORDER\"]   =\"PCI_BUS_ID\"\n# if not opt.use_data_parallel:\nos.environ[\"CUDA_VISIBLE_DEVICES\"]= str(opt.gpu[0])\n\n\n\n\"\"\"===================================================================================================\"\"\"\n#################### SEEDS FOR REPROD. #####################\ntorch.backends.cudnn.deterministic=True; np.random.seed(opt.seed); random.seed(opt.seed)\ntorch.manual_seed(opt.seed); torch.cuda.manual_seed(opt.seed); torch.cuda.manual_seed_all(opt.seed)\n\n\n\n\"\"\"===================================================================================================\"\"\"\n##################### NETWORK SETUP ##################\nopt.device = torch.device('cuda')\nmodel      = archs.select(opt.arch, opt)\n\nif opt.fc_lr<0:\n    to_optim   = [{'params':model.parameters(),'lr':opt.lr,'weight_decay':opt.decay}]\nelse:\n    all_but_fc_params = [x[-1] for x in list(filter(lambda x: 'last_linear' not in x[0], model.named_parameters()))]\n    fc_params         = model.model.last_linear.parameters()\n    to_optim          = [{'params':all_but_fc_params,'lr':opt.lr,'weight_decay':opt.decay},\n                         {'params':fc_params,'lr':opt.fc_lr,'weight_decay':opt.decay}]\n\n_  = model.to(opt.device)\n\n\n\n\n\"\"\"============================================================================\"\"\"\n#################### DATALOADER SETUPS ##################\ndataloaders = {}\ndatasets    = datasets.select(opt.dataset, opt, opt.source_path)\n\ndataloaders['evaluation'] = torch.utils.data.DataLoader(datasets['evaluation'], num_workers=opt.kernels, batch_size=opt.bs, shuffle=False)\ndataloaders['testing']    = torch.utils.data.DataLoader(datasets['testing'],    num_workers=opt.kernels, batch_size=opt.bs, shuffle=False)\nif opt.use_tv_split:\n    dataloaders['validation'] = torch.utils.data.DataLoader(datasets['validation'], num_workers=opt.kernels, batch_size=opt.bs,shuffle=False)\n\ntrain_data_sampler      = dsamplers.select(opt.data_sampler, opt, datasets['training'].image_dict, datasets['training'].image_list)\nif train_data_sampler.requires_storage:\n    train_data_sampler.create_storage(dataloaders['evaluation'], model, opt.device)\n\ndataloaders['training'] = torch.utils.data.DataLoader(datasets['training'], num_workers=opt.kernels, batch_sampler=train_data_sampler)\n\nopt.n_classes  = len(dataloaders['training'].dataset.avail_classes)\n\n\n\n\n\"\"\"============================================================================\"\"\"\n#################### CREATE LOGGING FILES ###############\nsub_loggers = ['Train', 'Test', 'Model Grad']\nif opt.use_tv_split: sub_loggers.append('Val')\nLOG = logger.LOGGER(opt, sub_loggers=sub_loggers, start_new=True, log_online=opt.log_online)\n\n\n\n\n\n\"\"\"============================================================================\"\"\"\n#################### LOSS SETUP ####################\nbatchminer   = bmine.select(opt.batch_mining, opt)\ncriterion, to_optim = criteria.select(opt.loss, opt, to_optim, batchminer)\n_ = criterion.to(opt.device)\n\nif 'criterion' in train_data_sampler.name:\n    train_data_sampler.internal_criterion = criterion\n\n\n\n\n\"\"\"============================================================================\"\"\"\n#################### OPTIM SETUP ####################\nif opt.optim == 'adam':\n    optimizer    = torch.optim.Adam(to_optim)\nelif opt.optim == 'sgd':\n    optimizer    = torch.optim.SGD(to_optim, momentum=0.9)\nelse:\n    raise Exception('Optimizer <{}> not available!'.format(opt.optim))\nscheduler    = torch.optim.lr_scheduler.MultiStepLR(optimizer, milestones=opt.tau, gamma=opt.gamma)\n\n\n\n\n\n\"\"\"============================================================================\"\"\"\n#################### METRIC COMPUTER ####################\nopt.rho_spectrum_embed_dim = opt.embed_dim\nmetric_computer = metrics.MetricComputer(opt.evaluation_metrics, opt)\n\n\n\n\n\n\"\"\"============================================================================\"\"\"\n################### Summary #########################3\ndata_text  = 'Dataset:\\t {}'.format(opt.dataset.upper())\nsetup_text = 'Objective:\\t {}'.format(opt.loss.upper())\nminer_text = 'Batchminer:\\t {}'.format(opt.batch_mining if criterion.REQUIRES_BATCHMINER else 'N/A')\narch_text  = 'Backbone:\\t {} (#weights: {})'.format(opt.arch.upper(), misc.gimme_params(model))\nsummary    = data_text+'\\n'+setup_text+'\\n'+miner_text+'\\n'+arch_text\nprint(summary)\n\n\n\n\n\"\"\"============================================================================\"\"\"\n################### SCRIPT MAIN ##########################\nprint('\\n-----\\n')\n\niter_count = 0\nloss_args  = {'batch':None, 'labels':None, 'batch_features':None, 'f_embed':None}\n\n\nfor epoch in range(opt.n_epochs):\n    epoch_start_time = time.time()\n\n    if epoch>0 and opt.data_idx_full_prec and train_data_sampler.requires_storage:\n        train_data_sampler.full_storage_update(dataloaders['evaluation'], model, opt.device)\n\n    opt.epoch = epoch\n    ### Scheduling Changes specifically for cosine scheduling\n    if opt.scheduler!='none': print('Running with learning rates {}...'.format(' | '.join('{}'.format(x) for x in scheduler.get_lr())))\n\n    \"\"\"=======================================\"\"\"\n    if train_data_sampler.requires_storage:\n        train_data_sampler.precompute_indices()\n\n\n    \"\"\"=======================================\"\"\"\n    ### Train one epoch\n    start = time.time()\n    _ = model.train()\n\n\n    loss_collect = []\n    data_iterator = tqdm(dataloaders['training'], desc='Epoch {} Training...'.format(epoch))\n\n\n    for i,out in enumerate(data_iterator):\n        class_labels, input, input_indices = out\n\n        ### Compute Embedding\n        input      = input.to(opt.device)\n        model_args = {'x':input.to(opt.device)}\n        # Needed for MixManifold settings.\n        if 'mix' in opt.arch: model_args['labels'] = class_labels\n        embeds  = model(**model_args)\n        if isinstance(embeds, tuple): embeds, (avg_features, features) = embeds\n\n        ### Compute Loss\n        loss_args['batch']          = embeds\n        loss_args['labels']         = class_labels\n        loss_args['f_embed']        = model.model.last_linear\n        loss_args['batch_features'] = features\n        loss      = criterion(**loss_args)\n\n        ###\n        optimizer.zero_grad()\n        loss.backward()\n\n        ### Compute Model Gradients and log them!\n        grads              = np.concatenate([p.grad.detach().cpu().numpy().flatten() for p in model.parameters() if p.grad is not None])\n        grad_l2, grad_max  = np.mean(np.sqrt(np.mean(np.square(grads)))), np.mean(np.max(np.abs(grads)))\n        LOG.progress_saver['Model Grad'].log('Grad L2',  grad_l2,  group='L2')\n        LOG.progress_saver['Model Grad'].log('Grad Max', grad_max, group='Max')\n\n        ### Update network weights!\n        optimizer.step()\n\n        ###\n        loss_collect.append(loss.item())\n\n        ###\n        iter_count += 1\n\n        if i==len(dataloaders['training'])-1: data_iterator.set_description('Epoch (Train) {0}: Mean Loss [{1:.4f}]'.format(epoch, np.mean(loss_collect)))\n\n        \"\"\"=======================================\"\"\"\n        if train_data_sampler.requires_storage and train_data_sampler.update_storage:\n            train_data_sampler.replace_storage_entries(embeds.detach().cpu(), input_indices)\n\n    result_metrics = {'loss': np.mean(loss_collect)}\n\n    ####\n    LOG.progress_saver['Train'].log('epochs', epoch)\n    for metricname, metricval in result_metrics.items():\n        LOG.progress_saver['Train'].log(metricname, metricval)\n    LOG.progress_saver['Train'].log('time', np.round(time.time()-start, 4))\n\n\n\n    \"\"\"=======================================\"\"\"\n    ### Evaluate Metric for Training & Test (& Validation)\n    _ = model.eval()\n    print('\\nComputing Testing Metrics...')\n    eval.evaluate(opt.dataset, LOG, metric_computer, [dataloaders['testing']],    model, opt, opt.evaltypes, opt.device, log_key='Test')\n    if opt.use_tv_split:\n        print('\\nComputing Validation Metrics...')\n        eval.evaluate(opt.dataset, LOG, metric_computer, [dataloaders['validation']], model, opt, opt.evaltypes, opt.device, log_key='Val')\n    print('\\nComputing Training Metrics...')\n    eval.evaluate(opt.dataset, LOG, metric_computer, [dataloaders['evaluation']], model, opt, opt.evaltypes, opt.device, log_key='Train')\n\n\n    LOG.update(all=True)\n\n\n    \"\"\"=======================================\"\"\"\n    ### Learning Rate Scheduling Step\n    if opt.scheduler != 'none':\n        scheduler.step()\n\n    print('Total Epoch Runtime: {0:4.2f}s'.format(time.time()-epoch_start_time))\n    print('\\n-----\\n')\n\n\n\n\n\"\"\"=======================================================\"\"\"\n### CREATE A SUMMARY TEXT FILE\nsummary_text = ''\nfull_training_time = time.time()-full_training_start_time\nsummary_text += 'Training Time: {} min.\\n'.format(np.round(full_training_time/60,2))\n\nsummary_text += '---------------\\n'\nfor sub_logger in LOG.sub_loggers:\n    metrics       = LOG.graph_writer[sub_logger].ov_title\n    summary_text += '{} metrics: {}\\n'.format(sub_logger.upper(), metrics)\n\nwith open(opt.save_path+'/training_summary.txt','w') as summary_file:\n    summary_file.write(summary_text)\n"
  },
  {
    "path": "metrics/__init__.py",
    "content": "from metrics import e_recall, nmi, f1, mAP, mAP_c, mAP_1000, mAP_lim\nfrom metrics import dists, rho_spectrum\nfrom metrics import c_recall, c_nmi, c_f1, c_mAP_c, c_mAP_1000, c_mAP_lim\nimport numpy as np\nimport faiss\nimport torch\nfrom sklearn.preprocessing import normalize\nfrom tqdm import tqdm\nimport copy\n\n\ndef select(metricname, opt):\n    #### Metrics based on euclidean distances\n    if 'e_recall' in metricname:\n        k = int(metricname.split('@')[-1])\n        return e_recall.Metric(k)\n    elif metricname=='nmi':\n        return nmi.Metric()\n    elif metricname=='mAP':\n        return mAP.Metric()\n    elif metricname=='mAP_c':\n        return mAP_c.Metric()\n    elif metricname=='mAP_lim':\n        return mAP_lim.Metric()\n    elif metricname=='mAP_1000':\n        return mAP_1000.Metric()\n    elif metricname=='f1':\n        return f1.Metric()\n\n    #### Metrics based on cosine similarity\n    elif 'c_recall' in metricname:\n        k = int(metricname.split('@')[-1])\n        return c_recall.Metric(k)\n    elif metricname=='c_nmi':\n        return c_nmi.Metric()\n    elif metricname=='c_mAP':\n        return c_mAP.Metric()\n    elif metricname=='c_mAP_c':\n        return c_mAP_c.Metric()\n    elif metricname=='c_mAP_lim':\n        return c_mAP_lim.Metric()\n    elif metricname=='c_mAP_1000':\n        return c_mAP_1000.Metric()\n    elif metricname=='c_f1':\n        return c_f1.Metric()\n\n    #### Generic Embedding space metrics\n    elif 'dists' in metricname:\n        mode = metricname.split('@')[-1]\n        return dists.Metric(mode)\n    elif 'rho_spectrum' in metricname:\n        mode = int(metricname.split('@')[-1])\n        embed_dim = opt.rho_spectrum_embed_dim\n        return rho_spectrum.Metric(embed_dim, mode=mode, opt=opt)\n    else:\n        raise NotImplementedError(\"Metric {} not available!\".format(metricname))\n\n\n\n\nclass MetricComputer():\n    def __init__(self, metric_names, opt):\n        self.pars            = opt\n        self.metric_names    = metric_names\n        self.list_of_metrics = [select(metricname, opt) for metricname in metric_names]\n        self.requires        = [metric.requires for metric in self.list_of_metrics]\n        self.requires        = list(set([x for y in self.requires for x in y]))\n\n    def compute_standard(self, opt, model, dataloader, evaltypes, device, **kwargs):\n        evaltypes = copy.deepcopy(evaltypes)\n\n        n_classes = opt.n_classes\n        image_paths     = np.array([x[0] for x in dataloader.dataset.image_list])\n        _ = model.eval()\n\n        ###\n        feature_colls  = {key:[] for key in evaltypes}\n\n        ###\n        with torch.no_grad():\n            target_labels = []\n            final_iter = tqdm(dataloader, desc='Embedding Data...'.format(len(evaltypes)))\n            image_paths= [x[0] for x in dataloader.dataset.image_list]\n            for idx,inp in enumerate(final_iter):\n                input_img,target = inp[1], inp[0]\n                target_labels.extend(target.numpy().tolist())\n                out = model(input_img.to(device))\n                if isinstance(out, tuple): out, aux_f = out\n\n                ### Include embeddings of all output features\n                for evaltype in evaltypes:\n                    if isinstance(out, dict):\n                        feature_colls[evaltype].extend(out[evaltype].cpu().detach().numpy().tolist())\n                    else:\n                        feature_colls[evaltype].extend(out.cpu().detach().numpy().tolist())\n\n\n            target_labels = np.hstack(target_labels).reshape(-1,1)\n\n\n        computed_metrics = {evaltype:{} for evaltype in evaltypes}\n        extra_infos      = {evaltype:{} for evaltype in evaltypes}\n\n\n        ###\n        faiss.omp_set_num_threads(self.pars.kernels)\n        # faiss.omp_set_num_threads(self.pars.kernels)\n        res = None\n        torch.cuda.empty_cache()\n        if self.pars.evaluate_on_gpu:\n            res = faiss.StandardGpuResources()\n\n\n        import time\n        for evaltype in evaltypes:\n            features        = np.vstack(feature_colls[evaltype]).astype('float32')\n            features_cosine = normalize(features, axis=1)\n\n            start = time.time()\n\n            \"\"\"============ Compute k-Means ===============\"\"\"\n            if 'kmeans' in self.requires:\n                ### Set CPU Cluster index\n                cluster_idx = faiss.IndexFlatL2(features.shape[-1])\n                if res is not None: cluster_idx = faiss.index_cpu_to_gpu(res, 0, cluster_idx)\n                kmeans            = faiss.Clustering(features.shape[-1], n_classes)\n                kmeans.niter = 20\n                kmeans.min_points_per_centroid = 1\n                kmeans.max_points_per_centroid = 1000000000\n                ### Train Kmeans\n                kmeans.train(features, cluster_idx)\n                centroids = faiss.vector_float_to_array(kmeans.centroids).reshape(n_classes, features.shape[-1])\n\n            if 'kmeans_cosine' in self.requires:\n                ### Set CPU Cluster index\n                cluster_idx = faiss.IndexFlatL2(features_cosine.shape[-1])\n                if res is not None: cluster_idx = faiss.index_cpu_to_gpu(res, 0, cluster_idx)\n                kmeans            = faiss.Clustering(features_cosine.shape[-1], n_classes)\n                kmeans.niter = 20\n                kmeans.min_points_per_centroid = 1\n                kmeans.max_points_per_centroid = 1000000000\n                ### Train Kmeans\n                kmeans.train(features_cosine, cluster_idx)\n                centroids_cosine = faiss.vector_float_to_array(kmeans.centroids).reshape(n_classes, features_cosine.shape[-1])\n                centroids_cosine = normalize(centroids,axis=1)\n\n\n            \"\"\"============ Compute Cluster Labels ===============\"\"\"\n            if 'kmeans_nearest' in self.requires:\n                faiss_search_index = faiss.IndexFlatL2(centroids.shape[-1])\n                if res is not None: faiss_search_index = faiss.index_cpu_to_gpu(res, 0, faiss_search_index)\n                faiss_search_index.add(centroids)\n                _, computed_cluster_labels = faiss_search_index.search(features, 1)\n\n            if 'kmeans_nearest_cosine' in self.requires:\n                faiss_search_index = faiss.IndexFlatIP(centroids_cosine.shape[-1])\n                if res is not None: faiss_search_index = faiss.index_cpu_to_gpu(res, 0, faiss_search_index)\n                faiss_search_index.add(centroids_cosine)\n                _, computed_cluster_labels_cosine = faiss_search_index.search(features_cosine, 1)\n\n\n\n            \"\"\"============ Compute Nearest Neighbours ===============\"\"\"\n            if 'nearest_features' in self.requires:\n                faiss_search_index  = faiss.IndexFlatL2(features.shape[-1])\n                if res is not None: faiss_search_index = faiss.index_cpu_to_gpu(res, 0, faiss_search_index)\n                faiss_search_index.add(features)\n\n                max_kval            = np.max([int(x.split('@')[-1]) for x in self.metric_names if 'recall' in x])\n                _, k_closest_points = faiss_search_index.search(features, int(max_kval+1))\n                k_closest_classes   = target_labels.reshape(-1)[k_closest_points[:,1:]]\n\n            if 'nearest_features_cosine' in self.requires:\n                faiss_search_index  = faiss.IndexFlatIP(features_cosine.shape[-1])\n                if res is not None: faiss_search_index = faiss.index_cpu_to_gpu(res, 0, faiss_search_index)\n                faiss_search_index.add(normalize(features_cosine,axis=1))\n\n                max_kval                   = np.max([int(x.split('@')[-1]) for x in self.metric_names if 'recall' in x])\n                _, k_closest_points_cosine = faiss_search_index.search(normalize(features_cosine,axis=1), int(max_kval+1))\n                k_closest_classes_cosine   = target_labels.reshape(-1)[k_closest_points_cosine[:,1:]]\n\n\n\n            ###\n            if self.pars.evaluate_on_gpu:\n                features        = torch.from_numpy(features).to(self.pars.device)\n                features_cosine = torch.from_numpy(features_cosine).to(self.pars.device)\n\n            start = time.time()\n            for metric in self.list_of_metrics:\n                input_dict = {}\n                if 'features' in metric.requires:         input_dict['features'] = features\n                if 'target_labels' in metric.requires:    input_dict['target_labels'] = target_labels\n\n                if 'kmeans' in metric.requires:           input_dict['centroids'] = centroids\n                if 'kmeans_nearest' in metric.requires:   input_dict['computed_cluster_labels'] = computed_cluster_labels\n                if 'nearest_features' in metric.requires: input_dict['k_closest_classes'] = k_closest_classes\n\n                if 'features_cosine' in metric.requires:         input_dict['features_cosine'] = features_cosine\n\n                if 'kmeans_cosine' in metric.requires:           input_dict['centroids_cosine'] = centroids_cosine\n                if 'kmeans_nearest_cosine' in metric.requires:   input_dict['computed_cluster_labels_cosine'] = computed_cluster_labels_cosine\n                if 'nearest_features_cosine' in metric.requires: input_dict['k_closest_classes_cosine'] = k_closest_classes_cosine\n\n                computed_metrics[evaltype][metric.name] = metric(**input_dict)\n\n            extra_infos[evaltype] = {'features':features, 'target_labels':target_labels,\n                                     'image_paths': dataloader.dataset.image_paths,\n                                     'query_image_paths':None, 'gallery_image_paths':None}\n\n        torch.cuda.empty_cache()\n        return computed_metrics, extra_infos\n"
  },
  {
    "path": "metrics/c_f1.py",
    "content": "import numpy as np\nfrom scipy.special import comb, binom\nimport torch\n\nclass Metric():\n    def __init__(self, **kwargs):\n        self.requires = ['kmeans_cosine', 'kmeans_nearest_cosine', 'features_cosine', 'target_labels']\n        self.name     = 'c_f1'\n\n    def __call__(self, target_labels, computed_cluster_labels_cosine, features_cosine, centroids_cosine):\n        import time\n        start = time.time()\n        if isinstance(features_cosine, torch.Tensor):\n            features_cosine = features_cosine.detach().cpu().numpy()\n        d = np.zeros(len(features_cosine))\n        for i in range(len(features_cosine)):\n            d[i] = np.linalg.norm(features_cosine[i,:] - centroids_cosine[computed_cluster_labels_cosine[i],:])\n\n        start = time.time()\n        labels_pred = np.zeros(len(features_cosine))\n        for i in np.unique(computed_cluster_labels_cosine):\n            index = np.where(computed_cluster_labels_cosine == i)[0]\n            ind = np.argmin(d[index])\n            cid = index[ind]\n            labels_pred[index] = cid\n\n\n        start = time.time()\n        N = len(target_labels)\n\n        # cluster n_labels\n        avail_labels = np.unique(target_labels)\n        n_labels     = len(avail_labels)\n\n        # count the number of objects in each cluster\n        count_cluster = np.zeros(n_labels)\n        for i in range(n_labels):\n            count_cluster[i] = len(np.where(target_labels == avail_labels[i])[0])\n\n        # build a mapping from item_id to item index\n        keys     = np.unique(labels_pred)\n        num_item = len(keys)\n        values   = range(num_item)\n        item_map = dict()\n        for i in range(len(keys)):\n            item_map.update([(keys[i], values[i])])\n\n\n        # count the number of objects of each item\n        count_item = np.zeros(num_item)\n        for i in range(N):\n            index = item_map[labels_pred[i]]\n            count_item[index] = count_item[index] + 1\n\n        # compute True Positive (TP) plus False Positive (FP)\n        # tp_fp = 0\n        tp_fp = comb(count_cluster, 2).sum()\n        # for k in range(n_labels):\n        #     if count_cluster[k] > 1:\n        #         tp_fp = tp_fp + comb(count_cluster[k], 2)\n\n        # compute True Positive (TP)\n        tp     = 0\n        start = time.time()\n        for k in range(n_labels):\n            member     = np.where(target_labels == avail_labels[k])[0]\n            member_ids = labels_pred[member]\n            count = np.zeros(num_item)\n            for j in range(len(member)):\n                index = item_map[member_ids[j]]\n                count[index] = count[index] + 1\n            # for i in range(num_item):\n            #     if count[i] > 1:\n            #         tp = tp + comb(count[i], 2)\n            tp += comb(count,2).sum()\n        # False Positive (FP)\n        fp = tp_fp - tp\n\n        # Compute False Negative (FN)\n        count = comb(count_item, 2).sum()\n        # count = 0\n        # for j in range(num_item):\n            # if count_item[j] > 1:\n            #     count = count + comb(count_item[j], 2)\n        fn = count - tp\n\n        # compute F measure\n        P = tp / (tp + fp)\n        R = tp / (tp + fn)\n        beta = 1\n        F = (beta*beta + 1) * P * R / (beta*beta * P + R)\n        return F\n"
  },
  {
    "path": "metrics/c_mAP_1000.py",
    "content": "import torch\nimport numpy as np\nimport faiss\n\n\n\nclass Metric():\n    def __init__(self, **kwargs):\n        self.requires = ['features_cosine', 'target_labels']\n        self.name     = 'c_mAP_1000'\n\n    def __call__(self, target_labels, features_cosine):\n        labels, freqs = np.unique(target_labels, return_counts=True)\n        R             = 1000\n\n        faiss_search_index  = faiss.IndexFlatIP(features_cosine.shape[-1])\n        if isinstance(features_cosine, torch.Tensor):\n            features_cosine = features_cosine.detach().cpu().numpy()\n            res = faiss.StandardGpuResources()\n            faiss_search_index = faiss.index_cpu_to_gpu(res, 0, faiss_search_index)\n        faiss_search_index.add(features_cosine)\n        nearest_neighbours  = faiss_search_index.search(features_cosine, int(R+1))[1][:,1:]\n\n        target_labels = target_labels.reshape(-1)\n        nn_labels = target_labels[nearest_neighbours]\n\n        avg_r_precisions = []\n        for label, freq in zip(labels, freqs):\n            rows_with_label = np.where(target_labels==label)[0]\n            for row in rows_with_label:\n                n_recalled_samples           = np.arange(1,R+1)\n                target_label_occ_in_row      = nn_labels[row,:]==label\n                cumsum_target_label_freq_row = np.cumsum(target_label_occ_in_row)\n                avg_r_pr_row = np.sum(cumsum_target_label_freq_row*target_label_occ_in_row/n_recalled_samples)/freq\n                avg_r_precisions.append(avg_r_pr_row)\n\n        return np.mean(avg_r_precisions)\n"
  },
  {
    "path": "metrics/c_mAP_c.py",
    "content": "import torch\nimport numpy as np\nimport faiss\n\n\n\nclass Metric():\n    def __init__(self, **kwargs):\n        self.requires = ['features_cosine', 'target_labels']\n        self.name     = 'c_mAP_c'\n\n    def __call__(self, target_labels, features_cosine):\n        labels, freqs = np.unique(target_labels, return_counts=True)\n        R             = np.max(freqs)\n\n        faiss_search_index  = faiss.IndexFlatIP(features_cosine.shape[-1])\n        if isinstance(features_cosine, torch.Tensor):\n            features_cosine = features_cosine.detach().cpu().numpy()\n            res = faiss.StandardGpuResources()\n            faiss_search_index = faiss.index_cpu_to_gpu(res, 0, faiss_search_index)\n        faiss_search_index.add(features_cosine)\n        nearest_neighbours  = faiss_search_index.search(features_cosine, int(R+1))[1][:,1:]\n\n        target_labels = target_labels.reshape(-1)\n        nn_labels = target_labels[nearest_neighbours]\n\n        avg_r_precisions = []\n        for label, freq in zip(labels, freqs):\n            rows_with_label = np.where(target_labels==label)[0]\n            for row in rows_with_label:\n                n_recalled_samples           = np.arange(1,freq+1)\n                target_label_occ_in_row      = nn_labels[row,:freq]==label\n                cumsum_target_label_freq_row = np.cumsum(target_label_occ_in_row)\n                avg_r_pr_row = np.sum(cumsum_target_label_freq_row*target_label_occ_in_row/n_recalled_samples)/freq\n                avg_r_precisions.append(avg_r_pr_row)\n\n        return np.mean(avg_r_precisions)\n"
  },
  {
    "path": "metrics/c_mAP_lim.py",
    "content": "import torch\nimport numpy as np\nimport faiss\n\n\n\nclass Metric():\n    def __init__(self, **kwargs):\n        self.requires = ['features_cosine', 'target_labels']\n        self.name     = 'c_mAP_lim'\n\n    def __call__(self, target_labels, features_cosine):\n        labels, freqs = np.unique(target_labels, return_counts=True)\n        ## Account for faiss-limit at k=1023\n        R             = min(1023,len(features_cosine))\n\n        faiss_search_index  = faiss.IndexFlatIP(features_cosine.shape[-1])\n        if isinstance(features_cosine, torch.Tensor):\n            features_cosine = features_cosine.detach().cpu().numpy()\n            res = faiss.StandardGpuResources()\n            faiss_search_index = faiss.index_cpu_to_gpu(res, 0, faiss_search_index)\n        faiss_search_index.add(features_cosine)\n        nearest_neighbours  = faiss_search_index.search(features_cosine, int(R+1))[1][:,1:]\n\n        target_labels = target_labels.reshape(-1)\n        nn_labels = target_labels[nearest_neighbours]\n\n        avg_r_precisions = []\n        for label, freq in zip(labels, freqs):\n            rows_with_label = np.where(target_labels==label)[0]\n            for row in rows_with_label:\n                n_recalled_samples           = np.arange(1,R+1)\n                target_label_occ_in_row      = nn_labels[row,:]==label\n                cumsum_target_label_freq_row = np.cumsum(target_label_occ_in_row)\n                avg_r_pr_row = np.sum(cumsum_target_label_freq_row*target_label_occ_in_row/n_recalled_samples)/freq\n                avg_r_precisions.append(avg_r_pr_row)\n\n        return np.mean(avg_r_precisions)\n"
  },
  {
    "path": "metrics/c_nmi.py",
    "content": "from sklearn import metrics\n\nclass Metric():\n    def __init__(self, **kwargs):\n        self.requires = ['kmeans_nearest_cosine', 'target_labels']\n        self.name     = 'c_nmi'\n\n    def __call__(self, target_labels, computed_cluster_labels_cosine):\n        NMI = metrics.cluster.normalized_mutual_info_score(computed_cluster_labels_cosine.reshape(-1), target_labels.reshape(-1))\n        return NMI\n"
  },
  {
    "path": "metrics/c_recall.py",
    "content": "import numpy as np\n\nclass Metric():\n    def __init__(self, k, **kwargs):\n        self.k        = k\n        self.requires = ['nearest_features_cosine', 'target_labels']\n        self.name     = 'c_recall@{}'.format(k)\n\n    def __call__(self, target_labels, k_closest_classes_cosine, **kwargs):\n        recall_at_k = np.sum([1 for target, recalled_predictions in zip(target_labels, k_closest_classes_cosine) if target in recalled_predictions[:self.k]])/len(target_labels)\n        return recall_at_k\n"
  },
  {
    "path": "metrics/compute_stack.py",
    "content": "\n"
  },
  {
    "path": "metrics/dists.py",
    "content": "from scipy.spatial import distance\nfrom sklearn.preprocessing import normalize\nimport numpy as np\nimport torch\n\nclass Metric():\n    def __init__(self, mode, **kwargs):\n        self.mode        = mode\n        self.requires = ['features', 'target_labels']\n        self.name     = 'dists@{}'.format(mode)\n\n    def __call__(self, features, target_labels):\n        features_locs = []\n        for lab in np.unique(target_labels):\n            features_locs.append(np.where(target_labels==lab)[0])\n\n        if 'intra' in self.mode:\n            if isinstance(features, torch.Tensor):\n                intrafeatures = features.detach().cpu().numpy()\n            else:\n                intrafeatures = features\n\n            intra_dists = []\n            for loc in features_locs:\n                c_dists = distance.cdist(intrafeatures[loc], intrafeatures[loc], 'cosine')\n                c_dists = np.sum(c_dists)/(len(c_dists)**2-len(c_dists))\n                intra_dists.append(c_dists)\n            intra_dists = np.array(intra_dists)\n            maxval      = np.max(intra_dists[1-np.isnan(intra_dists)])\n            intra_dists[np.isnan(intra_dists)] = maxval\n            intra_dists[np.isinf(intra_dists)] = maxval\n            dist_metric = dist_metric_intra = np.mean(intra_dists)\n\n        if 'inter' in self.mode:\n            if not isinstance(features, torch.Tensor):\n                coms = []\n                for loc in features_locs:\n                    com   = normalize(np.mean(features[loc],axis=0).reshape(1,-1)).reshape(-1)\n                    coms.append(com)\n                mean_inter_dist = distance.cdist(np.array(coms), np.array(coms), 'cosine')\n                dist_metric = dist_metric_inter = np.sum(mean_inter_dist)/(len(mean_inter_dist)**2-len(mean_inter_dist))\n            else:\n                coms = []\n                for loc in features_locs:\n                    com   = torch.nn.functional.normalize(torch.mean(features[loc],dim=0).reshape(1,-1), dim=-1).reshape(1,-1)\n                    coms.append(com)\n                mean_inter_dist = 1-torch.cat(coms,dim=0).mm(torch.cat(coms,dim=0).T).detach().cpu().numpy()\n                dist_metric = dist_metric_inter = np.sum(mean_inter_dist)/(len(mean_inter_dist)**2-len(mean_inter_dist))\n\n        if self.mode=='intra_over_inter':\n            dist_metric = dist_metric_intra/np.clip(dist_metric_inter, 1e-8, None)\n\n        return dist_metric\n"
  },
  {
    "path": "metrics/e_recall.py",
    "content": "import numpy as np\n\nclass Metric():\n    def __init__(self, k, **kwargs):\n        self.k        = k\n        self.requires = ['nearest_features', 'target_labels']\n        self.name     = 'e_recall@{}'.format(k)\n\n    def __call__(self, target_labels, k_closest_classes, **kwargs):\n        recall_at_k = np.sum([1 for target, recalled_predictions in zip(target_labels, k_closest_classes) if target in recalled_predictions[:self.k]])/len(target_labels)\n        return recall_at_k\n"
  },
  {
    "path": "metrics/f1.py",
    "content": "import numpy as np\nfrom scipy.special import comb, binom\nimport torch\n\nclass Metric():\n    def __init__(self, **kwargs):\n        self.requires = ['kmeans', 'kmeans_nearest', 'features', 'target_labels']\n        self.name     = 'f1'\n\n    def __call__(self, target_labels, computed_cluster_labels, features, centroids):\n        import time\n        start = time.time()\n        if isinstance(features, torch.Tensor):\n            features = features.detach().cpu().numpy()\n        d = np.zeros(len(features))\n        for i in range(len(features)):\n            d[i] = np.linalg.norm(features[i,:] - centroids[computed_cluster_labels[i],:])\n\n        start = time.time()\n        labels_pred = np.zeros(len(features))\n        for i in np.unique(computed_cluster_labels):\n            index = np.where(computed_cluster_labels == i)[0]\n            ind = np.argmin(d[index])\n            cid = index[ind]\n            labels_pred[index] = cid\n\n\n        start = time.time()\n        N = len(target_labels)\n\n        # cluster n_labels\n        avail_labels = np.unique(target_labels)\n        n_labels     = len(avail_labels)\n\n        # count the number of objects in each cluster\n        count_cluster = np.zeros(n_labels)\n        for i in range(n_labels):\n            count_cluster[i] = len(np.where(target_labels == avail_labels[i])[0])\n\n        # build a mapping from item_id to item index\n        keys     = np.unique(labels_pred)\n        num_item = len(keys)\n        values   = range(num_item)\n        item_map = dict()\n        for i in range(len(keys)):\n            item_map.update([(keys[i], values[i])])\n\n\n        # count the number of objects of each item\n        count_item = np.zeros(num_item)\n        for i in range(N):\n            index = item_map[labels_pred[i]]\n            count_item[index] = count_item[index] + 1\n\n        # compute True Positive (TP) plus False Positive (FP)\n        # tp_fp = 0\n        tp_fp = comb(count_cluster, 2).sum()\n        # for k in range(n_labels):\n        #     if count_cluster[k] > 1:\n        #         tp_fp = tp_fp + comb(count_cluster[k], 2)\n\n        # compute True Positive (TP)\n        tp     = 0\n        start = time.time()\n        for k in range(n_labels):\n            member     = np.where(target_labels == avail_labels[k])[0]\n            member_ids = labels_pred[member]\n            count = np.zeros(num_item)\n            for j in range(len(member)):\n                index = item_map[member_ids[j]]\n                count[index] = count[index] + 1\n            # for i in range(num_item):\n            #     if count[i] > 1:\n            #         tp = tp + comb(count[i], 2)\n            tp += comb(count,2).sum()\n        # False Positive (FP)\n        fp = tp_fp - tp\n\n        # Compute False Negative (FN)\n        count = comb(count_item, 2).sum()\n        # count = 0\n        # for j in range(num_item):\n            # if count_item[j] > 1:\n            #     count = count + comb(count_item[j], 2)\n        fn = count - tp\n\n        # compute F measure\n        P = tp / (tp + fp)\n        R = tp / (tp + fn)\n        beta = 1\n        F = (beta*beta + 1) * P * R / (beta*beta * P + R)\n        return F\n"
  },
  {
    "path": "metrics/mAP.py",
    "content": "import torch\nimport numpy as np\nimport faiss\n\n\n\nclass Metric():\n    def __init__(self, **kwargs):\n        self.requires = ['features', 'target_labels']\n        self.name     = 'mAP'\n\n    def __call__(self, target_labels, features):\n        labels, freqs = np.unique(target_labels, return_counts=True)\n        R             = len(features)\n\n        faiss_search_index  = faiss.IndexFlatL2(features.shape[-1])\n        if isinstance(features, torch.Tensor):\n            features = features.detach().cpu().numpy()\n            res = faiss.StandardGpuResources()\n            faiss_search_index = faiss.index_cpu_to_gpu(res, 0, faiss_search_index)        \n        faiss_search_index.add(features)\n        nearest_neighbours  = faiss_search_index.search(features, int(R+1))[1][:,1:]\n\n        target_labels = target_labels.reshape(-1)\n        nn_labels = target_labels[nearest_neighbours]\n\n        avg_r_precisions = []\n        for label, freq in zip(labels, freqs):\n            rows_with_label = np.where(target_labels==label)[0]\n            for row in rows_with_label:\n                n_recalled_samples           = np.arange(1,R+1)\n                target_label_occ_in_row      = nn_labels[row,:]==label\n                cumsum_target_label_freq_row = np.cumsum(target_label_occ_in_row)\n                avg_r_pr_row = np.sum(cumsum_target_label_freq_row*target_label_occ_in_row/n_recalled_samples)/freq\n                avg_r_precisions.append(avg_r_pr_row)\n\n        return np.mean(avg_r_precisions)\n"
  },
  {
    "path": "metrics/mAP_1000.py",
    "content": "import torch\nimport numpy as np\nimport faiss\n\n\n\nclass Metric():\n    def __init__(self, **kwargs):\n        self.requires = ['features', 'target_labels']\n        self.name     = 'mAP_1000'\n\n    def __call__(self, target_labels, features):\n        labels, freqs = np.unique(target_labels, return_counts=True)\n        R             = 1000\n\n        faiss_search_index  = faiss.IndexFlatL2(features.shape[-1])\n        if isinstance(features, torch.Tensor):\n            features = features.detach().cpu().numpy()\n            res = faiss.StandardGpuResources()\n            faiss_search_index = faiss.index_cpu_to_gpu(res, 0, faiss_search_index)\n        faiss_search_index.add(features)\n        nearest_neighbours  = faiss_search_index.search(features, int(R+1))[1][:,1:]\n\n        target_labels = target_labels.reshape(-1)\n        nn_labels = target_labels[nearest_neighbours]\n\n        avg_r_precisions = []\n        for label, freq in zip(labels, freqs):\n            rows_with_label = np.where(target_labels==label)[0]\n            for row in rows_with_label:\n                n_recalled_samples           = np.arange(1,R+1)\n                target_label_occ_in_row      = nn_labels[row,:]==label\n                cumsum_target_label_freq_row = np.cumsum(target_label_occ_in_row)\n                avg_r_pr_row = np.sum(cumsum_target_label_freq_row*target_label_occ_in_row/n_recalled_samples)/freq\n                avg_r_precisions.append(avg_r_pr_row)\n\n        return np.mean(avg_r_precisions)\n"
  },
  {
    "path": "metrics/mAP_c.py",
    "content": "import torch\nimport numpy as np\nimport faiss\n\n\n\nclass Metric():\n    def __init__(self, **kwargs):\n        self.requires = ['features', 'target_labels']\n        self.name     = 'mAP_c'\n\n    def __call__(self, target_labels, features):\n        labels, freqs = np.unique(target_labels, return_counts=True)\n        R             = np.max(freqs)\n\n        faiss_search_index  = faiss.IndexFlatL2(features.shape[-1])\n        if isinstance(features, torch.Tensor):\n            features = features.detach().cpu().numpy()\n            res = faiss.StandardGpuResources()\n            faiss_search_index = faiss.index_cpu_to_gpu(res, 0, faiss_search_index)\n        faiss_search_index.add(features)\n        nearest_neighbours  = faiss_search_index.search(features, int(R+1))[1][:,1:]\n\n        target_labels = target_labels.reshape(-1)\n        nn_labels = target_labels[nearest_neighbours]\n\n        avg_r_precisions = []\n        for label, freq in zip(labels, freqs):\n            rows_with_label = np.where(target_labels==label)[0]\n            for row in rows_with_label:\n                n_recalled_samples           = np.arange(1,freq+1)\n                target_label_occ_in_row      = nn_labels[row,:freq]==label\n                cumsum_target_label_freq_row = np.cumsum(target_label_occ_in_row)\n                avg_r_pr_row = np.sum(cumsum_target_label_freq_row*target_label_occ_in_row/n_recalled_samples)/freq\n                avg_r_precisions.append(avg_r_pr_row)\n\n        return np.mean(avg_r_precisions)\n"
  },
  {
    "path": "metrics/mAP_lim.py",
    "content": "import torch\nimport numpy as np\nimport faiss\n\n\n\nclass Metric():\n    def __init__(self, **kwargs):\n        self.requires = ['features', 'target_labels']\n        self.name     = 'mAP_lim'\n\n    def __call__(self, target_labels, features):\n        labels, freqs = np.unique(target_labels, return_counts=True)\n        ## Account for faiss-limit at k=1023\n        R             = min(1023,len(features))\n\n        faiss_search_index  = faiss.IndexFlatL2(features.shape[-1])\n        if isinstance(features, torch.Tensor):\n            features = features.detach().cpu().numpy()\n            res = faiss.StandardGpuResources()\n            faiss_search_index = faiss.index_cpu_to_gpu(res, 0, faiss_search_index)        \n        faiss_search_index.add(features)\n        nearest_neighbours  = faiss_search_index.search(features, int(R+1))[1][:,1:]\n\n        target_labels = target_labels.reshape(-1)\n        nn_labels = target_labels[nearest_neighbours]\n\n        avg_r_precisions = []\n        for label, freq in zip(labels, freqs):\n            rows_with_label = np.where(target_labels==label)[0]\n            for row in rows_with_label:\n                n_recalled_samples           = np.arange(1,R+1)\n                target_label_occ_in_row      = nn_labels[row,:]==label\n                cumsum_target_label_freq_row = np.cumsum(target_label_occ_in_row)\n                avg_r_pr_row = np.sum(cumsum_target_label_freq_row*target_label_occ_in_row/n_recalled_samples)/freq\n                avg_r_precisions.append(avg_r_pr_row)\n\n        return np.mean(avg_r_precisions)\n"
  },
  {
    "path": "metrics/nmi.py",
    "content": "from sklearn import metrics\n\nclass Metric():\n    def __init__(self, **kwargs):\n        self.requires = ['kmeans_nearest', 'target_labels']\n        self.name     = 'nmi'\n\n    def __call__(self, target_labels, computed_cluster_labels):\n        NMI = metrics.cluster.normalized_mutual_info_score(computed_cluster_labels.reshape(-1), target_labels.reshape(-1))\n        return NMI\n"
  },
  {
    "path": "metrics/rho_spectrum.py",
    "content": "from scipy.spatial import distance\nfrom sklearn.preprocessing import normalize\nimport numpy as np\n\n\nclass Metric():\n    def __init__(self, embed_dim, mode,  **kwargs):\n        self.mode      = mode\n        self.embed_dim = embed_dim\n        self.requires = ['features']\n        self.name     = 'rho_spectrum@'+str(mode)\n\n    def __call__(self, features):\n        from sklearn.decomposition import TruncatedSVD\n        from scipy.stats import entropy\n        import torch\n\n        if isinstance(features, torch.Tensor):\n            _,s,_ = torch.svd(features)\n            s     = s.cpu().numpy()\n        else:\n            svd = TruncatedSVD(n_components=self.embed_dim-1, n_iter=7, random_state=42)\n            svd.fit(features)\n            s = svd.singular_values_\n\n        if self.mode!=0:\n            s = s[np.abs(self.mode)-1:]\n        s_norm  = s/np.sum(s)\n        uniform = np.ones(len(s))/(len(s))\n\n        if self.mode<0:\n            kl = entropy(s_norm, uniform)\n        if self.mode>0:\n            kl = entropy(uniform, s_norm)\n        if self.mode==0:\n            kl = s_norm\n\n        return kl\n"
  },
  {
    "path": "parameters.py",
    "content": "import argparse, os\n\n\n#######################################\ndef basic_training_parameters(parser):\n    ##### Dataset-related Parameters\n    parser.add_argument('--dataset',              default='cub200',   type=str,   help='Dataset to use. Currently supported: cub200, cars196, online_products.')\n    parser.add_argument('--use_tv_split',         action='store_true',            help='Flag. If set, split the training set into a training/validation set.')\n    parser.add_argument('--tv_split_by_samples',  action='store_true',            help='Flag. If set, create the validation set by taking a percentage of samples PER class. \\\n                                                                                        Otherwise, the validation set is create by taking a percentage of classes.')\n    parser.add_argument('--tv_split_perc',        default=0.8,      type=float, help='Percentage with which the training dataset is split into training/validation.')\n    parser.add_argument('--augmentation',         default='base',   type=str,   help='Type of preprocessing/augmentation to use on the data.  \\\n                                                                                      Available: base (standard), adv (with color/brightness changes), big (Images of size 256x256), red (No RandomResizedCrop).')\n\n    ### General Training Parameters\n    parser.add_argument('--lr',                default=0.00001,  type=float,        help='Learning Rate for network parameters.')\n    parser.add_argument('--fc_lr',             default=-1,       type=float,        help='Optional. If not -1, sets the learning rate for the final linear embedding layer.')\n    parser.add_argument('--decay',             default=0.0004,   type=float,        help='Weight decay placed on network weights.')\n    parser.add_argument('--n_epochs',          default=150,      type=int,          help='Number of training epochs.')\n    parser.add_argument('--kernels',           default=6,        type=int,          help='Number of workers for pytorch dataloader.')\n    parser.add_argument('--bs',                default=112 ,     type=int,          help='Mini-Batchsize to use.')\n    parser.add_argument('--seed',              default=1,        type=int,          help='Random seed for reproducibility.')\n    parser.add_argument('--scheduler',         default='step',   type=str,          help='Type of learning rate scheduling. Currently supported: step')\n    parser.add_argument('--gamma',             default=0.3,      type=float,        help='Learning rate reduction after tau epochs.')\n    parser.add_argument('--tau',               default=[1000], nargs='+',type=int , help='Stepsize before reducing learning rate.')\n\n    ##### Loss-specific Settings\n    parser.add_argument('--optim',           default='adam',        type=str,   help='Optimization method to use. Currently supported: adam & sgd.')\n    parser.add_argument('--loss',            default='margin',      type=str,   help='Training criteria: For supported methods, please check criteria/__init__.py')\n    parser.add_argument('--batch_mining',    default='distance',    type=str,   help='Batchminer for tuple-based losses: For supported methods, please check batch_mining/__init__.py')\n\n    ##### Network-related Flags\n    parser.add_argument('--embed_dim',        default=128,         type=int,                    help='Embedding dimensionality of the network. Note: dim = 64, 128 or 512 is used in most papers, depending on the architecture.')\n    parser.add_argument('--not_pretrained',   action='store_true',                              help='Flag. If set, no ImageNet pretraining is used to initialize the network.')\n    parser.add_argument('--arch',             default='resnet50_frozen_normalize',  type=str,   help='Underlying network architecture. Frozen denotes that \\\n                                                                                                  exisiting pretrained batchnorm layers are frozen, and normalize denotes normalization of the output embedding.')\n\n    ##### Evaluation Parameters\n    parser.add_argument('--no_train_metrics', action='store_true',   help='Flag. If set, evaluation metrics are not computed for the training data. Saves a forward pass over the full training dataset.')\n    parser.add_argument('--evaluate_on_gpu',  action='store_true',   help='Flag. If set, all metrics, when possible, are computed on the GPU (requires Faiss-GPU).')\n    parser.add_argument('--evaluation_metrics', nargs='+', default=['e_recall@1', 'e_recall@2', 'e_recall@4', 'nmi', 'f1', 'mAP_1000', 'mAP_lim', 'mAP_c', \\\n                                                                    'dists@intra', 'dists@inter', 'dists@intra_over_inter', 'rho_spectrum@0', \\\n                                                                    'rho_spectrum@-1', 'rho_spectrum@1', 'rho_spectrum@2', 'rho_spectrum@10'], type=str, help='Metrics to evaluate performance by.')\n\n    parser.add_argument('--storage_metrics',    nargs='+', default=['e_recall@1'],     type=str, help='Improvement in these metrics on a dataset trigger checkpointing.')\n    parser.add_argument('--evaltypes',          nargs='+', default=['discriminative'], type=str, help='The network may produce multiple embeddings (ModuleDict, relevant for e.g. DiVA). If the key is listed here, the entry will be evaluated on the evaluation metrics.\\\n                                                                                                       Note: One may use Combined_embed1_embed2_..._embedn-w1-w1-...-wn to compute evaluation metrics on weighted (normalized) combinations.')\n\n\n    ##### Setup Parameters\n    parser.add_argument('--gpu',          default=[0], nargs='+',                  type=int, help='Gpu to use.')\n    parser.add_argument('--savename',     default='group_plus_seed',               type=str, help='Run savename - if default, the savename will comprise the project and group name (see wandb_parameters()).')\n    parser.add_argument('--source_path',  default=os.getcwd()+'/../../Datasets',   type=str, help='Path to training data.')\n    parser.add_argument('--save_path',    default=os.getcwd()+'/Training_Results', type=str, help='Where to save everything.')\n\n    return parser\n\n\n\n#######################################\ndef wandb_parameters(parser):\n    ### Online Logging/Wandb Log Arguments\n    parser.add_argument('--log_online',      action='store_true',            help='Flag. If set, run metrics are stored online in addition to offline logging. Should generally be set.')\n    parser.add_argument('--wandb_key',       default='<your_api_key_here>',  type=str,   help='API key for W&B.')\n    parser.add_argument('--project',         default='Sample_Project',       type=str,   help='Name of the project - relates to W&B project names. In --savename default setting part of the savename.')\n    parser.add_argument('--group',           default='Sample_Group',         type=str,   help='Name of the group - relates to W&B group names - all runs with same setup but different seeds are logged into one group. \\\n                                                                                               In --savename default setting part of the savename.')\n    return parser\n\n\n\n#######################################\ndef loss_specific_parameters(parser):\n    ### Contrastive Loss\n    parser.add_argument('--loss_contrastive_pos_margin', default=0, type=float, help='positive margin for contrastive pairs.')\n    parser.add_argument('--loss_contrastive_neg_margin', default=1, type=float, help='negative margin for contrastive pairs.')\n\n    ### Triplet-based Losses\n    parser.add_argument('--loss_triplet_margin',       default=0.2,         type=float, help='Margin for Triplet Loss')\n\n    ### MarginLoss\n    parser.add_argument('--loss_margin_margin',       default=0.2,          type=float, help='Triplet margin.')\n    parser.add_argument('--loss_margin_beta_lr',      default=0.0005,       type=float, help='Learning Rate for learnable class margin parameters in MarginLoss')\n    parser.add_argument('--loss_margin_beta',         default=1.2,          type=float, help='Initial Class Margin Parameter in Margin Loss')\n    parser.add_argument('--loss_margin_nu',           default=0,            type=float, help='Regularisation value on betas in Margin Loss. Generally not needed.')\n    parser.add_argument('--loss_margin_beta_constant',action='store_true',              help='Flag. If set, beta-values are left untrained.')\n\n    ### ProxyNCA\n    parser.add_argument('--loss_proxynca_lrmulti',      default=50,     type=float, help='Learning Rate multiplier for Proxies in proxynca.')\n    #NOTE: The number of proxies is determined by the number of data classes.\n\n    ### NPair\n    parser.add_argument('--loss_npair_l2',     default=0.005,        type=float, help='L2 weight in NPair. Note: Set to 0.02 in paper, but multiplied with 0.25 in their implementation.')\n\n    ### Angular Loss\n    parser.add_argument('--loss_angular_alpha',             default=45, type=float, help='Angular margin in degrees.')\n    parser.add_argument('--loss_angular_npair_ang_weight',  default=2,  type=float, help='Relative weighting between angular and npair contribution.')\n    parser.add_argument('--loss_angular_npair_l2',          default=0.005,  type=float, help='L2 weight on NPair (as embeddings are not normalized).')\n\n    ### Multisimilary Loss\n    parser.add_argument('--loss_multisimilarity_pos_weight', default=2,         type=float, help='Weighting on positive similarities.')\n    parser.add_argument('--loss_multisimilarity_neg_weight', default=40,        type=float, help='Weighting on negative similarities.')\n    parser.add_argument('--loss_multisimilarity_margin',     default=0.1,       type=float, help='Distance margin for both positive and negative similarities.')\n    parser.add_argument('--loss_multisimilarity_thresh',     default=0.5,       type=float, help='Exponential thresholding.')\n\n    ### Lifted Structure Loss\n    parser.add_argument('--loss_lifted_neg_margin', default=1,     type=float, help='Margin placed on similarities.')\n    parser.add_argument('--loss_lifted_l2',         default=0.005, type=float, help='As embeddings are not normalized, they need to be placed under penalty.')\n\n    ### Quadruplet Loss\n    parser.add_argument('--loss_quadruplet_margin_alpha_1',  default=0.2, type=float, help='Quadruplet Loss requires two margins. This is the first one.')\n    parser.add_argument('--loss_quadruplet_margin_alpha_2',  default=0.2, type=float, help='This is the second.')\n\n    ### Soft-Triple Loss\n    parser.add_argument('--loss_softtriplet_n_centroids',   default=2,    type=int,   help='Number of proxies per class.')\n    parser.add_argument('--loss_softtriplet_margin_delta',  default=0.01, type=float, help='Margin placed on sample-proxy similarities.')\n    parser.add_argument('--loss_softtriplet_gamma',         default=0.1,  type=float, help='Weight over sample-proxies within a class.')\n    parser.add_argument('--loss_softtriplet_lambda',        default=8,    type=float, help='Serves as a temperature.')\n    parser.add_argument('--loss_softtriplet_reg_weight',    default=0.2,  type=float, help='Regularization weight on the number of proxies.')\n    parser.add_argument('--loss_softtriplet_lrmulti',       default=1,    type=float, help='Learning Rate multiplier for proxies.')\n\n    ### Normalized Softmax Loss\n    parser.add_argument('--loss_softmax_lr',           default=0.00001, type=float, help='Learning rate on class proxies.')\n    parser.add_argument('--loss_softmax_temperature',  default=0.05,    type=float, help='Temperature for NCA objective.')\n\n    ### Histogram Loss\n    parser.add_argument('--loss_histogram_nbins',  default=65, type=int, help='Number of bins for histogram discretization.')\n\n    ### SNR Triplet (with learnable margin) Loss\n    parser.add_argument('--loss_snr_margin',      default=0.2,   type=float, help='Triplet margin.')\n    parser.add_argument('--loss_snr_reg_lambda',  default=0.005, type=float, help='Regularization of in-batch element sum.')\n\n    ### ArcFace\n    parser.add_argument('--loss_arcface_lr',             default=0.0005,  type=float, help='Learning rate on class proxies.')\n    parser.add_argument('--loss_arcface_angular_margin', default=0.5,     type=float, help='Angular margin in radians.')\n    parser.add_argument('--loss_arcface_feature_scale',  default=16,      type=float, help='Inverse Temperature for NCA objective.')\n    return parser\n\n\n\n#######################################\ndef batchmining_specific_parameters(parser):\n    ### Distance-based Batchminer\n    parser.add_argument('--miner_distance_lower_cutoff', default=0.5, type=float, help='Lower cutoff on distances - values below are sampled with equal prob.')\n    parser.add_argument('--miner_distance_upper_cutoff', default=1.4, type=float, help='Upper cutoff on distances - values above are IGNORED.')\n    ### Spectrum-Regularized Miner (as proposed in our paper) - utilizes a distance-based sampler that is regularized.\n    parser.add_argument('--miner_rho_distance_lower_cutoff', default=0.5, type=float, help='Lower cutoff on distances - values below are sampled with equal prob.')\n    parser.add_argument('--miner_rho_distance_upper_cutoff', default=1.4, type=float, help='Upper cutoff on distances - values above are IGNORED.')\n    parser.add_argument('--miner_rho_distance_cp',           default=0.2, type=float, help='Probability to replace a negative with a positive.')\n    return parser\n\n\n#######################################\ndef batch_creation_parameters(parser):\n    parser.add_argument('--data_sampler',              default='class_random', type=str, help='How the batch is created. Available options: See datasampler/__init__.py.')\n    parser.add_argument('--samples_per_class',         default=2,              type=int, help='Number of samples in one class drawn before choosing the next class. Set to >1 for tuple-based loss.')\n    ### Batch-Sample Flags - Have no relevance to default SPC-N sampling\n    parser.add_argument('--data_batchmatch_bigbs',     default=512,            type=int, help='Size of batch to be summarized into a smaller batch. For distillation/coreset-based methods.')\n    parser.add_argument('--data_batchmatch_ncomps',    default=10,             type=int, help='Number of batch candidates that are evaluated, from which the best one is chosen.')\n    parser.add_argument('--data_storage_no_update',    action='store_true',              help='Flag for methods that need a sample storage. If set, storage entries are NOT updated.')\n    parser.add_argument('--data_d2_coreset_lambda',    default=1, type=float,            help='Regularisation for D2-coreset.')\n    parser.add_argument('--data_gc_coreset_lim',       default=1e-9, type=float,         help='D2-coreset value limit.')\n    parser.add_argument('--data_sampler_lowproj_dim',  default=-1, type=int,             help='Optionally project embeddings into a lower dimension to ensure that greedy coreset works better. Only makes a difference for large embedding dims.')\n    parser.add_argument('--data_sim_measure',          default='euclidean', type=str,    help='Distance measure to use for batch selection.')\n    parser.add_argument('--data_gc_softened',          action='store_true', help='Flag. If set, use a soft version of greedy coreset.')\n    parser.add_argument('--data_idx_full_prec',        action='store_true', help='Deprecated.')\n    parser.add_argument('--data_mb_mom',               default=-1, type=float, help='For memory-bank based samplers - momentum term on storage entry updates.')\n    parser.add_argument('--data_mb_lr',                default=1,  type=float, help='Deprecated.')\n\n    return parser\n"
  },
  {
    "path": "toy_experiments/toy_example_diagonal_lines.py",
    "content": "import os, numpy as np, matplotlib.pyplot as plt\nimport torch, torch.nn as nn, torchvision as tv\nimport numpy as np\nimport random\n\n\"\"\"===================================================\"\"\"\nseed = 1\ntorch.backends.cudnn.deterministic=True; np.random.seed(seed); random.seed(seed)\ntorch.manual_seed(seed); torch.cuda.manual_seed(seed); torch.cuda.manual_seed_all(seed)\n\n\n\"\"\"===================================================\"\"\"\nppline     = 100\nn_lines    = 4\nnoise_perc = 0.15\nintervals  = [(0.1,0.3), (0.35,0.55), (0.6,0.8), (0.85,1.05)]\nlines = [np.stack([np.linspace(intv[0],intv[1],ppline), np.linspace(intv[0],intv[1],ppline)])[:,np.random.choice(ppline, int(ppline*noise_perc), replace=False)] for intv in intervals]\ncls   = [x*np.ones(int(ppline*noise_perc)) for x in range(n_lines)]\ntrain_lines = np.concatenate(lines, axis=1).T\ntrain_cls   = np.concatenate(cls)\n\nx_test_line1 = np.stack([0.2*np.ones(ppline), np.linspace(0.2,0.4,ppline)])[:,np.random.choice(ppline, int(ppline*noise_perc), replace=False)]\nx_test_line2 = np.stack([0.2*np.ones(ppline), np.linspace(0.55,0.85,ppline)])[:,np.random.choice(ppline, int(ppline*noise_perc), replace=False)]\ny_test_line1 = np.stack([np.linspace(0.4,0.6,ppline), 0.2*np.ones(ppline)])[:,np.random.choice(ppline, int(ppline*noise_perc), replace=False)]\ny_test_line2 = np.stack([np.linspace(0.7,0.9,ppline), 0.2*np.ones(ppline)])[:,np.random.choice(ppline, int(ppline*noise_perc), replace=False)]\n# for line in lines:\n#     plt.plot(line[0,:], line[1,:], '.', markersize=6)\n# plt.plot(x_test_line1[0,:], x_test_line1[1,:])\n# plt.plot(x_test_line2[0,:], x_test_line2[1,:])\n# plt.plot(y_test_line1[0,:], y_test_line1[1,:])\n# plt.plot(y_test_line2[0,:], y_test_line2[1,:])\n\n\n###############\nos.environ[\"CUDA_DEVICE_ORDER\"]   = \"PCI_BUS_ID\"\nos.environ[\"CUDA_VISIBLE_DEVICES\"]= '1'\n\n###############\nimport itertools as it\nfrom tqdm import tqdm\nimport torch.nn.functional as F\nbs         = 24\nlr         = 0.03\nneg_margin = 0.1\ntrain_iter = 200\ndevice = torch.device('cpu')\n\n\n###############\nclass Backbone(nn.Module):\n    def __init__(self):\n        super().__init__()\n        self.backbone = nn.Sequential(nn.Linear(2,30), nn.ReLU(), nn.Linear(30,30), nn.ReLU(), nn.Linear(30,2))\n\n    def forward(self, x):\n        return torch.nn.functional.normalize(self.backbone(x),dim=1)\n\n\n\n###############\nbase_net     = Backbone()\nmain_reg_net = Backbone()\n\n\n\n###############\ndef train(net2train, p_switch=0):\n    device = torch.device('cpu')\n    _ = net2train.train()\n    _ = net2train.to(device)\n    optim = torch.optim.Adam(net2train.parameters(), lr=lr)\n\n    loss_collect = []\n    for i in range(train_iter):\n        idxs   = np.random.choice(len(train_lines), bs, replace=False)\n        batch  = torch.from_numpy(train_lines[idxs,:]).to(torch.float).to(device)\n        train_labels = train_cls[idxs]\n        embed        = net2train(batch)\n\n        unique_cls   = np.unique(train_labels)\n        indices      = np.arange(len(batch))\n        class_dict   = {i:indices[train_labels==i] for i in unique_cls}\n\n        sampled_triplets = [list(it.product([x],[x],[y for y in unique_cls if x!=y])) for x in unique_cls]\n        sampled_triplets = [x for y in sampled_triplets for x in y]\n        sampled_triplets = [[x for x in list(it.product(*[class_dict[j] for j in i])) if x[0]!=x[1]] for i in sampled_triplets]\n        sampled_triplets = [x for y in sampled_triplets for x in y]\n\n        anchors   = [triplet[0] for triplet in sampled_triplets]\n        positives = [triplet[1] for triplet in sampled_triplets]\n        negatives = [triplet[2] for triplet in sampled_triplets]\n\n        if p_switch>0:\n            negatives = [p if np.random.choice(2, p=[1-p_switch, p_switch]) else n for n,p in zip(negatives, positives)]\n            neg_dists = torch.mean(F.relu(neg_margin - nn.PairwiseDistance(p=2)(embed[anchors,:], embed[negatives,:])))\n            loss      = neg_dists\n        else:\n            pos_dists = torch.mean(F.relu(nn.PairwiseDistance(p=2)(embed[anchors,:], embed[positives,:])))\n            neg_dists = torch.mean(F.relu(neg_margin - nn.PairwiseDistance(p=2)(embed[anchors,:], embed[negatives,:])))\n            loss      = pos_dists + neg_dists\n\n        optim.zero_grad()\n        loss.backward()\n        optim.step()\n\n        loss_collect.append(loss.item())\n\n    return loss_collect\n\n\n\n###############\nbase_loss    = train(base_net)\n_ = train(main_reg_net, p_switch=0.001)\n\n\n\n###############\ndef get_embeds(net):\n    _ = net.eval()\n    with torch.no_grad():\n        train_embed      = net(torch.from_numpy(train_lines).to(torch.float).to(device)).cpu().detach().numpy()\n        x_embed_test_line1 = net(torch.from_numpy(x_test_line1.T).to(torch.float).to(device)).cpu().detach().numpy()\n        x_embed_test_line2 = net(torch.from_numpy(x_test_line2.T).to(torch.float).to(device)).cpu().detach().numpy()\n        y_embed_test_line1 = net(torch.from_numpy(y_test_line1.T).to(torch.float).to(device)).cpu().detach().numpy()\n        y_embed_test_line2 = net(torch.from_numpy(y_test_line2.T).to(torch.float).to(device)).cpu().detach().numpy()\n    _, s, _ = np.linalg.svd(train_embed)\n    s = s/np.sum(s)\n    return train_embed, x_embed_test_line1, x_embed_test_line2, y_embed_test_line1, y_embed_test_line2, s\n\n\n\n\n###############\nbase_embed, x_base_t1, x_base_t2, y_base_t1, y_base_t2, base_s = get_embeds(base_net)\nsp                                                             = get_embeds(main_reg_net)\n\n###\ntheta = np.radians(np.linspace(0,360,300))\nx_2 = np.cos(theta)\ny_2 = np.sin(theta)\n\n\n###\nplt.style.use('default')\nimport matplotlib.cm as cm\ncolors = cm.rainbow(np.linspace(0, 0.5, n_lines))\ncolors = np.array([colors[int(sample_cls)] for sample_cls in train_cls][::-1])\nf,ax = plt.subplots(1,4)\nfor i in range(len(lines)):\n    loc = np.where(train_cls==i)[0]\n    ax[0].scatter(train_lines[loc,0], train_lines[loc,1], color=list(colors[loc,:]), label='Train Cls {}'.format(i), s=40)\nax[0].scatter(x_test_line1[0,:], x_test_line1[1,:], marker='x', label='Test Cls 1', color='r', s=40)\nax[0].scatter(x_test_line2[0,:], x_test_line2[1,:], marker='x', label='Test Cls 2', color='black', s=60)\nax[0].scatter(y_test_line1[0,:], y_test_line1[1,:], marker='^', label='Test Cls 3', color='brown', s=40)\nax[0].scatter(y_test_line2[0,:], y_test_line2[1,:], marker='^', label='Test Cls 4', color='magenta', s=60)\nax[1].plot(x_2, y_2, '--', color='gray', label='Unit Circle')\nfor i in range(len(lines)):\n    loc = np.where(train_cls==i)[0]\n    ax[1].scatter(base_embed[loc,0], base_embed[loc,1], color=list(colors[loc,:]), s=60)\nax[1].scatter(x_base_t1[:,0], x_base_t1[:,1], marker='x', color='r', s=60)\nax[1].scatter(x_base_t2[:,0], x_base_t2[:,1], marker='x', color='black', s=60)\nax[1].scatter(y_base_t1[:,0], y_base_t1[:,1], marker='^', color='brown', s=60)\nax[1].scatter(y_base_t2[:,0], y_base_t2[:,1], marker='^', color='magenta', s=60)\nax[1].set_xlim([np.min(base_embed[:,0])*0.85,np.max(base_embed[:,0]*1.15)])\nax[1].set_ylim([np.min(base_embed[:,1])*1.15,np.max(base_embed[:,1]*0.85)])\nax[2].plot(x_2, y_2, '--', color='gray', label='Unit Circle')\nfor i in range(len(lines)):\n    loc = np.where(train_cls==i)[0]\n    ax[2].scatter(sp[0][loc,0], sp[0][loc,1], color=list(colors[loc,:]), alpha=0.4, s=60)\nax[2].scatter(sp[1][:,0], sp[1][:,1], marker='x', color='r', s=60)\nax[2].scatter(sp[2][:,0], sp[2][:,1], marker='x', color='black', s=60)\nax[2].scatter(sp[3][:,0], sp[3][:,1], marker='^', color='brown', s=60)\nax[2].scatter(sp[4][:,0], sp[4][:,1], marker='^', color='magenta', s=60)\nax[2].set_xlim([np.min(sp[0][:,0])*1.15,np.max(sp[0][:,0]*1.15)])\nax[2].set_ylim([np.min(sp[0][:,1])*1.15,np.max(sp[0][:,1]*1.15)])\nax[3].bar(np.array([0,0.25]), base_s, width=0.25, alpha=0.5,edgecolor='k', label=r'$Base$')\nax[3].bar(np.array([0.6,0.85]), sp[5],width=0.25, alpha=0.5,edgecolor='k', label=r'$Reg.~Emb.$')\nax[3].set_xticks([0,0.25,0.6,0.85])\nax[3].set_xticklabels([1,2,1,2])\n# ax[1].text(0.5, -1.005, r'$Test~Classes$', fontsize=17, bbox=dict(boxstyle='round', facecolor='white', alpha=0.5))\n# ax[2].text(-0.45, -0.05, r'$Test~Classes$', fontsize=17, bbox=dict(boxstyle='round', facecolor='white', alpha=0.5))\n# ax[1].annotate(\"\", xy=(0.5, -0.9),   xytext=(0.55, -0.98), arrowprops=dict(facecolor='k', headwidth=5, width=2, shrink=0, alpha=1))\n# ax[1].annotate(\"\", xy=(0.55, -0.85), xytext=(0.55, -0.98), arrowprops=dict(facecolor='k', headwidth=5, width=2, shrink=0, alpha=1))\n# ax[1].annotate(\"\", xy=(0.56, -0.86), xytext=(0.55, -0.98), arrowprops=dict(facecolor='k', headwidth=5, width=2, shrink=0, alpha=1))\n# ax[1].annotate(\"\", xy=(0.59, -0.82), xytext=(0.55, -0.98), arrowprops=dict(facecolor='k', headwidth=5, width=2, shrink=0, alpha=1))\n# ax[2].annotate(\"\", xy=(0.2, 0.8),  xytext=(0.2, -0.02), arrowprops=dict(facecolor='k', headwidth=5, width=2, shrink=0, alpha=1))\n# ax[2].annotate(\"\", xy=(0.9, 0.1), xytext=(0.2, -0.02), arrowprops=dict(facecolor='k', headwidth=5, width=2, shrink=0, alpha=1))\n# ax[2].annotate(\"\", xy=(0.8, -0.5), xytext=(0.2, -0.02), arrowprops=dict(facecolor='k', headwidth=5, width=2, shrink=0, alpha=1))\n# ax[2].annotate(\"\", xy=(0.7, -0.52), xytext=(0.2, -0.02), arrowprops=dict(facecolor='k', headwidth=5, width=2, shrink=0, alpha=1))\nax[0].set_title(r'$Train/Test~Data$', fontsize=22)\nax[1].set_title(r'$Base~Embed.$', fontsize=22)\nax[2].set_title(r'$Regularized~Embed.$', fontsize=22)\nax[3].set_title(r'$SV~Spectrum$', fontsize=22)\nax[0].legend(loc='upper center',fontsize=16)\nax[1].legend(fontsize=16)\nax[2].legend(loc='center left', fontsize=16)\n# ax[1].legend(fontsize=16)\n# ax[2].legend(loc=4,fontsize=16)\nax[3].legend(loc=1,fontsize=16)\nfor a in ax.reshape(-1):\n    a.tick_params(axis='both', which='major', labelsize=20)\n    a.tick_params(axis='both', which='minor', labelsize=20)\nf.set_size_inches(22,6)\nf.tight_layout()\nf.savefig('diag_line_toy_ex_save.png')\nf.savefig('diag_line_toy_ex_save.pdf')\nplt.close()\n"
  },
  {
    "path": "utilities/__init__.py",
    "content": ""
  },
  {
    "path": "utilities/logger.py",
    "content": "import datetime, csv, os, numpy as np\nfrom matplotlib import pyplot as plt\nimport pickle as pkl\nfrom utilities.misc import gimme_save_string\n\n\"\"\"=============================================================================================================\"\"\"\n################## WRITE TO CSV FILE #####################\nclass CSV_Writer():\n    def __init__(self, save_path):\n        self.save_path = save_path\n        self.written         = []\n        self.n_written_lines = {}\n\n    def log(self, group, segments, content):\n        if group not in self.n_written_lines.keys():\n            self.n_written_lines[group] = 0\n\n        with open(self.save_path+'_'+group+'.csv', \"a\") as csv_file:\n            writer = csv.writer(csv_file, delimiter=\",\")\n            if group not in self.written: writer.writerow(segments)\n            for line in content:\n                writer.writerow(line)\n                self.n_written_lines[group] += 1\n\n        self.written.append(group)\n\n\n\n################## PLOT SUMMARY IMAGE #####################\nclass InfoPlotter():\n    def __init__(self, save_path, title='Training Log', figsize=(25,19)):\n        self.save_path = save_path\n        self.title     = title\n        self.figsize   = figsize\n        self.colors    = ['r','g','b','y','m','c','orange','darkgreen','lightblue']\n\n    def make_plot(self, base_title, title_append, sub_plots, sub_plots_data):\n        sub_plots = list(sub_plots)\n        if 'epochs' not in sub_plots:\n            x_data = range(len(sub_plots_data[0]))\n        else:\n            x_data = range(sub_plots_data[np.where(np.array(sub_plots)=='epochs')[0][0]][-1]+1)\n\n        self.ov_title = [(sub_plot,sub_plot_data) for sub_plot, sub_plot_data in zip(sub_plots,sub_plots_data) if sub_plot not in ['epoch','epochs','time']]\n        self.ov_title = [(x[0],np.max(x[1])) if 'loss' not in x[0] else (x[0],np.min(x[1])) for x in self.ov_title]\n        self.ov_title = title_append +': '+ '  |  '.join('{0}: {1:.4f}'.format(x[0],x[1]) for x in self.ov_title)\n        sub_plots_data = [x for x,y in zip(sub_plots_data, sub_plots)]\n        sub_plots      = [x for x in sub_plots]\n\n        plt.style.use('ggplot')\n        f,ax = plt.subplots(1)\n        ax.set_title(self.ov_title, fontsize=22)\n        for i,(data, title) in enumerate(zip(sub_plots_data, sub_plots)):\n            ax.plot(x_data, data, '-{}'.format(self.colors[i]), linewidth=1.7, label=base_title+' '+title)\n        ax.tick_params(axis='both', which='major', labelsize=18)\n        ax.tick_params(axis='both', which='minor', labelsize=18)\n        ax.legend(loc=2, prop={'size': 16})\n        f.set_size_inches(self.figsize[0], self.figsize[1])\n        f.savefig(self.save_path+'_'+title_append+'.svg')\n        plt.close()\n\n\n################## GENERATE LOGGING FOLDER/FILES #######################\ndef set_logging(opt):\n    checkfolder = opt.save_path+'/'+opt.savename\n    if opt.savename == '':\n        date = datetime.datetime.now()\n        time_string = '{}-{}-{}-{}-{}-{}'.format(date.year, date.month, date.day, date.hour, date.minute, date.second)\n        checkfolder = opt.save_path+'/{}_{}_'.format(opt.dataset.upper(), opt.arch.upper())+time_string\n    counter     = 1\n    while os.path.exists(checkfolder):\n        checkfolder = opt.save_path+'/'+opt.savename+'_'+str(counter)\n        counter += 1\n    os.makedirs(checkfolder)\n    opt.save_path = checkfolder\n\n    if 'experiment' in vars(opt):\n        import argparse\n        save_opt = {key:item for key,item in vars(opt).items() if key!='experiment'}\n        save_opt = argparse.Namespace(**save_opt)\n    else:\n        save_opt = opt\n\n    with open(save_opt.save_path+'/Parameter_Info.txt','w') as f:\n        f.write(gimme_save_string(save_opt))\n    pkl.dump(save_opt,open(save_opt.save_path+\"/hypa.pkl\",\"wb\"))\n\n\nclass Progress_Saver():\n    def __init__(self):\n        self.groups = {}\n\n    def log(self, segment, content, group=None):\n        if group is None: group = segment\n        if group not in self.groups.keys():\n            self.groups[group] = {}\n\n        if segment not in self.groups[group].keys():\n            self.groups[group][segment] = {'content':[],'saved_idx':0}\n\n        self.groups[group][segment]['content'].append(content)\n\n\nclass LOGGER():\n    def __init__(self, opt, sub_loggers=[], prefix=None, start_new=True, log_online=False):\n        \"\"\"\n        LOGGER Internal Structure:\n\n        self.progress_saver: Contains multiple Progress_Saver instances to log metrics for main metric subsets (e.g. \"Train\" for training metrics)\n            ['main_subset_name']: Name of each main subset (-> e.g. \"Train\")\n                .groups: Dictionary of subsets belonging to one of the main subsets, e.g. [\"Recall\", \"NMI\", ...]\n                    ['specific_metric_name']: Specific name of the metric of interest, e.g. Recall@1.\n        \"\"\"\n        self.prop        = opt\n        self.prefix      = '{}_'.format(prefix) if prefix is not None else ''\n        self.sub_loggers = sub_loggers\n\n        ### Make Logging Directories\n        if start_new: set_logging(opt)\n\n        ### Set Graph and CSV writer\n        self.csv_writer, self.graph_writer, self.progress_saver = {},{},{}\n        for sub_logger in sub_loggers:\n            csv_savepath = opt.save_path+'/CSV_Logs'\n            if not os.path.exists(csv_savepath): os.makedirs(csv_savepath)\n            self.csv_writer[sub_logger]     = CSV_Writer(csv_savepath+'/Data_{}{}'.format(self.prefix, sub_logger))\n\n            prgs_savepath = opt.save_path+'/Progression_Plots'\n            if not os.path.exists(prgs_savepath): os.makedirs(prgs_savepath)\n            self.graph_writer[sub_logger]   = InfoPlotter(prgs_savepath+'/Graph_{}{}'.format(self.prefix, sub_logger))\n            self.progress_saver[sub_logger] = Progress_Saver()\n\n\n        ### WandB Init\n        self.save_path   = opt.save_path\n        self.log_online  = log_online\n\n\n    def update(self, *sub_loggers, all=False):\n        online_content = []\n\n        if all: sub_loggers = self.sub_loggers\n\n        for sub_logger in list(sub_loggers):\n            for group in self.progress_saver[sub_logger].groups.keys():\n                pgs      = self.progress_saver[sub_logger].groups[group]\n                segments = pgs.keys()\n                per_seg_saved_idxs   = [pgs[segment]['saved_idx'] for segment in segments]\n                per_seg_contents     = [pgs[segment]['content'][idx:] for segment,idx in zip(segments, per_seg_saved_idxs)]\n                per_seg_contents_all = [pgs[segment]['content'] for segment,idx in zip(segments, per_seg_saved_idxs)]\n\n                #Adjust indexes\n                for content,segment in zip(per_seg_contents, segments):\n                    self.progress_saver[sub_logger].groups[group][segment]['saved_idx'] += len(content)\n\n                tupled_seg_content = [list(seg_content_slice) for seg_content_slice in zip(*per_seg_contents)]\n\n                self.csv_writer[sub_logger].log(group, segments, tupled_seg_content)\n                self.graph_writer[sub_logger].make_plot(sub_logger, group, segments, per_seg_contents_all)\n\n                for i,segment in enumerate(segments):\n                    if group == segment:\n                        name = sub_logger+': '+group\n                    else:\n                        name = sub_logger+': '+group+': '+segment\n                    online_content.append((name,per_seg_contents[i]))\n\n        if self.log_online:\n            if self.prop.online_backend=='wandb':\n                import wandb\n                for i,item in enumerate(online_content):\n                    if isinstance(item[1], list):\n                        wandb.log({item[0]:np.mean(item[1])}, step=self.prop.epoch)\n                    else:\n                        wandb.log({item[0]:item[1]}, step=self.prop.epoch)\n            elif self.prop.online_backend=='comet_ml':\n                for i,item in enumerate(online_content):\n                    if isinstance(item[1], list):\n                        self.prop.experiment.log_metric(item[0],np.mean(item[1]), self.prop.epoch)\n                    else:\n                        self.prop.experiment.log_metric(item[0],item[1],self.prop.epoch)\n"
  },
  {
    "path": "utilities/misc.py",
    "content": "\"\"\"=============================================================================================================\"\"\"\n######## LIBRARIES #####################\nimport numpy as np\n\n\n\n\"\"\"=============================================================================================================\"\"\"\n################# ACQUIRE NUMBER OF WEIGHTS #################\ndef gimme_params(model):\n    model_parameters = filter(lambda p: p.requires_grad, model.parameters())\n    params = sum([np.prod(p.size()) for p in model_parameters])\n    return params\n\n\n################# SAVE TRAINING PARAMETERS IN NICE STRING #################\ndef gimme_save_string(opt):\n    varx = vars(opt)\n    base_str = ''\n    for key in varx:\n        base_str += str(key)\n        if isinstance(varx[key],dict):\n            for sub_key, sub_item in varx[key].items():\n                base_str += '\\n\\t'+str(sub_key)+': '+str(sub_item)\n        else:\n            base_str += '\\n\\t'+str(varx[key])\n        base_str+='\\n\\n'\n    return base_str\n\n\n#############################################################################\nimport torch, torch.nn as nn\n\nclass DataParallel(nn.Module):\n    def __init__(self, model, device_ids, dim):\n        super().__init__()\n        self.model    = model.model\n        self.network  = nn.DataParallel(model, device_ids, dim)\n\n    def forward(self, x):\n        return self.network(x)\n"
  }
]