Repository: NegatioN/OnlineMiningTripletLoss
Branch: master
Commit: 4c22eac01ee8
Files: 67
Total size: 135.4 KB
Directory structure:
gitextract_53kablsc/
├── .github/
│ └── workflows/
│ └── main.yml
├── .gitignore
├── CONTRIBUTING.md
├── LICENSE
├── Makefile
├── README.md
├── docs/
│ ├── .gitignore
│ ├── Gemfile
│ ├── _config.yml
│ ├── _data/
│ │ ├── alerts.yml
│ │ ├── definitions.yml
│ │ ├── glossary.yml
│ │ ├── sidebars/
│ │ │ └── home_sidebar.yml
│ │ ├── tags.yml
│ │ ├── terms.yml
│ │ └── topnav.yml
│ ├── _includes/
│ │ ├── archive.html
│ │ ├── callout.html
│ │ ├── footer.html
│ │ ├── google_analytics.html
│ │ ├── head.html
│ │ ├── head_print.html
│ │ ├── image.html
│ │ ├── important.html
│ │ ├── initialize_shuffle.html
│ │ ├── inline_image.html
│ │ ├── links.html
│ │ ├── note.html
│ │ ├── search_google_custom.html
│ │ ├── search_simple_jekyll.html
│ │ ├── sidebar.html
│ │ ├── tip.html
│ │ ├── toc.html
│ │ ├── topnav.html
│ │ └── warning.html
│ ├── _layouts/
│ │ ├── default.html
│ │ ├── default_print.html
│ │ ├── none.html
│ │ ├── page.html
│ │ └── page_print.html
│ ├── css/
│ │ ├── boxshadowproperties.css
│ │ ├── customstyles.css
│ │ ├── fonts/
│ │ │ └── FontAwesome.otf
│ │ ├── modern-business.css
│ │ ├── printstyles.css
│ │ ├── syntax.css
│ │ ├── theme-blue.css
│ │ └── theme-green.css
│ ├── feed.xml
│ ├── fonts/
│ │ └── FontAwesome.otf
│ ├── index.html
│ ├── js/
│ │ ├── customscripts.js
│ │ ├── jekyll-search.js
│ │ └── toc.js
│ ├── licenses/
│ │ ├── LICENSE
│ │ └── LICENSE-BSD-NAVGOCO.txt
│ ├── sidebar.json
│ ├── sitemap.xml
│ ├── tooltips.json
│ └── triplet_loss.html
├── index.ipynb
├── online_triplet_loss/
│ ├── __init__.py
│ ├── _nbdev.py
│ └── losses.py
├── settings.ini
├── setup.py
└── triplet_loss.ipynb
================================================
FILE CONTENTS
================================================
================================================
FILE: .github/workflows/main.yml
================================================
name: CI
on: [push]
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v1
- uses: actions/setup-python@v1
with:
python-version: '3.6'
architecture: 'x64'
- name: Install the library
run: |
pip install nbdev jupyter
pip install -e .
- name: Read all notebooks
run: |
nbdev_read_nbs
- name: Check if all notebooks are cleaned
run: |
echo "Check we are starting with clean git checkout"
if [ -n "$(git status -uno -s)" ]; then echo "git status is not clean"; false; fi
echo "Trying to strip out notebooks"
nbdev_clean_nbs
echo "Check that strip out was unnecessary"
git status -s # display the status to see which nbs need cleaning up
if [ -n "$(git status -uno -s)" ]; then echo -e "!!! Detected unstripped out notebooks\n!!!Remember to run nbdev_install_git_hooks"; false; fi
- name: Check if there is no diff library/notebooks
run: |
if [ -n "$(nbdev_diff_nbs)" ]; then echo -e "!!! Detected difference between the notebooks and the library"; false; fi
- name: Run tests
run: |
nbdev_test_nbs
================================================
FILE: .gitignore
================================================
.last_checked
.idea/
# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class
# C extensions
*.so
.gitconfig
.gitattributes
# Distribution / packaging
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
*.egg-info/
.installed.cfg
*.egg
MANIFEST
# PyInstaller
# Usually these files are written by a python script from a template
# before PyInstaller builds the exe, so as to inject date/other infos into it.
*.manifest
*.spec
# Installer logs
pip-log.txt
pip-delete-this-directory.txt
# Unit test / coverage reports
htmlcov/
.tox/
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*.cover
.hypothesis/
.pytest_cache/
# Translations
*.mo
*.pot
# Django stuff:
*.log
local_settings.py
db.sqlite3
# Flask stuff:
instance/
.webassets-cache
# Scrapy stuff:
.scrapy
# Sphinx documentation
docs/_build/
# PyBuilder
target/
# Jupyter Notebook
.ipynb_checkpoints
# pyenv
.python-version
# celery beat schedule file
celerybeat-schedule
# SageMath parsed files
*.sage.py
# Environments
.env
.venv
env/
venv/
ENV/
env.bak/
venv.bak/
# Spyder project settings
.spyderproject
.spyproject
# Rope project settings
.ropeproject
# mkdocs documentation
/site
# mypy
.mypy_cache/
================================================
FILE: CONTRIBUTING.md
================================================
# How to contribute
## How to get started
Before anything else, please install the git hooks that run automatic scripts during each commit and merge to strip the notebooks of superfluous metadata (and avoid merge conflicts). After cloning the repository, run the following command inside it:
```
nbdev_install_git_hooks
```
## Did you find a bug?
* Ensure the bug was not already reported by searching on GitHub under Issues.
* If you're unable to find an open issue addressing the problem, open a new one. Be sure to include a title and clear description, as much relevant information as possible, and a code sample or an executable test case demonstrating the expected behavior that is not occurring.
* Be sure to add the complete error messages.
#### Did you write a patch that fixes a bug?
* Open a new GitHub pull request with the patch.
* Ensure that your PR includes a test that fails without your patch, and pass with it.
* Ensure the PR description clearly describes the problem and solution. Include the relevant issue number if applicable.
## PR submission guidelines
* Keep each PR focused. While it's more convenient, do not combine several unrelated fixes together. Create as many branches as needing to keep each PR focused.
* Do not mix style changes/fixes with "functional" changes. It's very difficult to review such PRs and it most likely get rejected.
* Do not add/remove vertical whitespace. Preserve the original style of the file you edit as much as you can.
* Do not turn an already submitted PR into your development playground. If after you submitted PR, you discovered that more work is needed - close the PR, do the required work and then submit a new PR. Otherwise each of your commits requires attention from maintainers of the project.
* If, however, you submitted a PR and received a request for changes, you should proceed with commits inside that PR, so that the maintainer can see the incremental fixes and won't need to review the whole PR again. In the exception case where you realize it'll take many many commits to complete the requests, then it's probably best to close the PR, do the work and then submit it again. Use common sense where you'd choose one way over another.
## Do you want to contribute to the documentation?
* Docs are automatically created from the notebooks in the nbs folder.
================================================
FILE: LICENSE
================================================
MIT License
Copyright (c) [2019] [Joakim Rishaug]
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
================================================
FILE: Makefile
================================================
SRC = $(wildcard lib/*.ipynb)
all: "online_triplet_loss" docs
"online_triplet_loss": $(SRC)
nbdev_build_lib
touch "online_triplet_loss"
docs_serve: docs
cd docs && bundle exec jekyll serve
docs: $(SRC)
nbdev_build_docs
touch docs
test:
nbdev_test_nbs
release: pypi
nbdev_bump_version
pypi: dist
twine upload --repository pypi dist/*
dist: clean
python setup.py sdist bdist_wheel
clean:
rm -rf dist
================================================
FILE: README.md
================================================
# online_triplet_loss
> PyTorch conversion of the excellent post on the same topic in Tensorflow. Simply an implementation of a triple loss with online mining of candidate triplets used in semi-supervised learning.
## Install
`pip install online_triplet_loss`
Then import with:
`from online_triplet_loss.losses import *`
PS: Requires Pytorch version 1.1.0 or above to use.
## How to use
In these examples I use a really large margin, since the embedding space is so small. A more realistic margins seems to be between `0.1 and 2.0`
```
from torch import nn
import torch
model = nn.Embedding(10, 10)
```
```
#from online_triplet_loss.losses import *
labels = torch.randint(high=10, size=(5,)) # our five labels
embeddings = model(labels)
print('Labels:', labels)
print('Embeddings:', embeddings)
loss = batch_hard_triplet_loss(labels, embeddings, margin=100)
print('Loss:', loss)
loss.backward()
```
Labels: tensor([6, 1, 3, 6, 6])
Embeddings: tensor([[-1.1335, 0.3364, -3.0174, -0.8732, -0.9301, 1.3619, 0.3746, 0.0457,
0.0180, -0.4500],
[ 1.0757, -0.8420, -0.7630, -0.0746, 1.1545, 0.4017, 0.5587, 1.7947,
0.1992, -2.2288],
[ 0.2646, 1.2383, 0.1949, 0.5743, -0.8460, -0.9929, -2.0350, 0.2095,
0.2129, -0.4855],
[-1.1335, 0.3364, -3.0174, -0.8732, -0.9301, 1.3619, 0.3746, 0.0457,
0.0180, -0.4500],
[-1.1335, 0.3364, -3.0174, -0.8732, -0.9301, 1.3619, 0.3746, 0.0457,
0.0180, -0.4500]], grad_fn=)
Loss: tensor(95.1271, grad_fn=)
```
#from online_triplet_loss.losses import *
embeddings = model(labels)
print('Labels:', labels)
print('Embeddings:', embeddings)
loss, fraction_pos = batch_all_triplet_loss(labels, embeddings, squared=False, margin=100)
print('Loss:', loss)
loss.backward()
```
Labels: tensor([6, 1, 3, 6, 6])
Embeddings: tensor([[-1.1335, 0.3364, -3.0174, -0.8732, -0.9301, 1.3619, 0.3746, 0.0457,
0.0180, -0.4500],
[ 1.0757, -0.8420, -0.7630, -0.0746, 1.1545, 0.4017, 0.5587, 1.7947,
0.1992, -2.2288],
[ 0.2646, 1.2383, 0.1949, 0.5743, -0.8460, -0.9929, -2.0350, 0.2095,
0.2129, -0.4855],
[-1.1335, 0.3364, -3.0174, -0.8732, -0.9301, 1.3619, 0.3746, 0.0457,
0.0180, -0.4500],
[-1.1335, 0.3364, -3.0174, -0.8732, -0.9301, 1.3619, 0.3746, 0.0457,
0.0180, -0.4500]], grad_fn=)
tensor(94.9947, grad_fn=) tensor(1.)
Loss: tensor(94.9947, grad_fn=)
## References
* [Triplet Loss and Online Triplet Mining in Tensorflow](https://github.com/omoindrot/tensorflow-triplet-loss)
* [Facenet paper](https://arxiv.org/abs/1503.03832)
* [adambielski's nice implementation](https://github.com/adambielski/siamese-triplet) (unfortunately context switches between CPU / GPU)
================================================
FILE: docs/.gitignore
================================================
_site/
================================================
FILE: docs/Gemfile
================================================
source "https://rubygems.org"
gem 'github-pages', group: :jekyll_plugins
# Added at 2019-11-25 10:11:40 -0800 by jhoward:
gem "jekyll", "~> 3.7"
================================================
FILE: docs/_config.yml
================================================
repository: "NegatioN/OnlineMiningTripletLoss"
output: web
topnav_title: "online_triplet_loss"
site_title: "online_triplet_loss"
company_name: "Joakim Rishaug"
description: "Online mining triplet losses for Pytorch"
# Set to false to disable KaTeX math
use_math: true
# Add Google analytics id if you have one and want to use it here
google_analytics:
# See http://nbdev.fast.ai/search for help with adding Search
google_search:
host: 127.0.0.1
# the preview server used. Leave as is.
port: 4000
# the port where the preview is rendered.
exclude:
- .idea/
- .gitignore
- vendor
exclude: [vendor]
highlighter: rouge
markdown: kramdown
kramdown:
input: GFM
auto_ids: true
hard_wrap: false
syntax_highlighter: rouge
collections:
tooltips:
output: false
defaults:
-
scope:
path: ""
type: "pages"
values:
layout: "page"
comments: true
search: true
sidebar: home_sidebar
topnav: topnav
-
scope:
path: ""
type: "tooltips"
values:
layout: "page"
comments: true
search: true
tooltip: true
sidebars:
- home_sidebar
permalink: pretty
theme: jekyll-theme-cayman
baseurl: /OnlineMiningTripletLoss/
================================================
FILE: docs/_data/alerts.yml
================================================
tip: '
================================================
FILE: docs/js/customscripts.js
================================================
$('#mysidebar').height($(".nav").height());
$( document ).ready(function() {
//this script says, if the height of the viewport is greater than 800px, then insert affix class, which makes the nav bar float in a fixed
// position as your scroll. if you have a lot of nav items, this height may not work for you.
var h = $(window).height();
//console.log (h);
if (h > 800) {
$( "#mysidebar" ).attr("class", "nav affix");
}
// activate tooltips. although this is a bootstrap js function, it must be activated this way in your theme.
$('[data-toggle="tooltip"]').tooltip({
placement : 'top'
});
/**
* AnchorJS
*/
anchors.add('h2,h3,h4,h5');
});
// needed for nav tabs on pages. See Formatting > Nav tabs for more details.
// script from http://stackoverflow.com/questions/10523433/how-do-i-keep-the-current-tab-active-with-twitter-bootstrap-after-a-page-reload
$(function() {
var json, tabsState;
$('a[data-toggle="pill"], a[data-toggle="tab"]').on('shown.bs.tab', function(e) {
var href, json, parentId, tabsState;
tabsState = localStorage.getItem("tabs-state");
json = JSON.parse(tabsState || "{}");
parentId = $(e.target).parents("ul.nav.nav-pills, ul.nav.nav-tabs").attr("id");
href = $(e.target).attr('href');
json[parentId] = href;
return localStorage.setItem("tabs-state", JSON.stringify(json));
});
tabsState = localStorage.getItem("tabs-state");
json = JSON.parse(tabsState || "{}");
$.each(json, function(containerId, href) {
return $("#" + containerId + " a[href=" + href + "]").tab('show');
});
$("ul.nav.nav-pills, ul.nav.nav-tabs").each(function() {
var $this = $(this);
if (!json[$this.attr("id")]) {
return $this.find("a[data-toggle=tab]:first, a[data-toggle=pill]:first").tab("show");
}
});
});
================================================
FILE: docs/js/jekyll-search.js
================================================
!function e(t,n,r){function s(o,u){if(!n[o]){if(!t[o]){var a="function"==typeof require&&require;if(!u&&a)return a(o,!0);if(i)return i(o,!0);throw new Error("Cannot find module '"+o+"'")}var f=n[o]={exports:{}};t[o][0].call(f.exports,function(e){var n=t[o][1][e];return s(n?n:e)},f,f.exports,e,t,n,r)}return n[o].exports}for(var i="function"==typeof require&&require,o=0;o=0}var self=this;self.matches=function(string,crit){return"string"!=typeof string?!1:(string=string.trim(),doMatch(string,crit))}}module.exports=new LiteralSearchStrategy},{}],4:[function(require,module){module.exports=function(){function findMatches(store,crit,strategy){for(var data=store.get(),i=0;i{title}',noResultsText:"No results found",limit:10,fuzzy:!1};self.init=function(_opt){validateOptions(_opt),assignOptions(_opt),isJSON(opt.dataSource)?initWithJSON(opt.dataSource):initWithURL(opt.dataSource)}}var Searcher=require("./Searcher"),Templater=require("./Templater"),Store=require("./Store"),JSONLoader=require("./JSONLoader"),searcher=new Searcher,templater=new Templater,store=new Store,jsonLoader=new JSONLoader;window.SimpleJekyllSearch=new SimpleJekyllSearch}(window,document)},{"./JSONLoader":1,"./Searcher":4,"./Store":5,"./Templater":6}]},{},[7]);
================================================
FILE: docs/js/toc.js
================================================
// https://github.com/ghiculescu/jekyll-table-of-contents
// this library modified by fastai to:
// - update the location.href with the correct anchor when a toc item is clicked on
(function($){
$.fn.toc = function(options) {
var defaults = {
noBackToTopLinks: false,
title: '',
minimumHeaders: 3,
headers: 'h1, h2, h3, h4',
listType: 'ol', // values: [ol|ul]
showEffect: 'show', // values: [show|slideDown|fadeIn|none]
showSpeed: 'slow' // set to 0 to deactivate effect
},
settings = $.extend(defaults, options);
var headers = $(settings.headers).filter(function() {
// get all headers with an ID
var previousSiblingName = $(this).prev().attr( "name" );
if (!this.id && previousSiblingName) {
this.id = $(this).attr( "id", previousSiblingName.replace(/\./g, "-") );
}
return this.id;
}), output = $(this);
if (!headers.length || headers.length < settings.minimumHeaders || !output.length) {
return;
}
if (0 === settings.showSpeed) {
settings.showEffect = 'none';
}
var render = {
show: function() { output.hide().html(html).show(settings.showSpeed); },
slideDown: function() { output.hide().html(html).slideDown(settings.showSpeed); },
fadeIn: function() { output.hide().html(html).fadeIn(settings.showSpeed); },
none: function() { output.html(html); }
};
var get_level = function(ele) { return parseInt(ele.nodeName.replace("H", ""), 10); }
var highest_level = headers.map(function(_, ele) { return get_level(ele); }).get().sort()[0];
//var return_to_top = '';
// other nice icons that can be used instead: glyphicon-upload glyphicon-hand-up glyphicon-chevron-up glyphicon-menu-up glyphicon-triangle-top
var level = get_level(headers[0]),
this_level,
html = settings.title + " <"+settings.listType+">";
headers.on('click', function() {
if (!settings.noBackToTopLinks) {
var pos = $(window).scrollTop();
window.location.hash = this.id;
$(window).scrollTop(pos);
}
})
.addClass('clickable-header')
.each(function(_, header) {
base_url = window.location.href;
base_url = base_url.replace(/#.*$/, "");
this_level = get_level(header);
//if (!settings.noBackToTopLinks && this_level > 1) {
// $(header).addClass('top-level-header').before(return_to_top);
//}
txt = header.textContent.split('¶')[0].split(/\[(test|source)\]/)[0];
if (!txt) {return;}
if (this_level === level) // same level as before; same indenting
html += "
" + txt + "";
else if (this_level <= level){ // higher level than before; end parent ol
for(i = this_level; i < level; i++) {
html += "
"
}
html += "
" + txt + "";
}
else if (this_level > level) { // lower level than before; expand the previous to contain a ol
for(i = this_level; i > level; i--) {
html += "<"+settings.listType+">"+((i-level == 2) ? "
" : "
")
}
html += "" + txt + "";
}
level = this_level; // update for the next one
});
html += "";
if (!settings.noBackToTopLinks) {
$(document).on('click', '.back-to-top', function() {
$(window).scrollTop(0);
window.location.hash = '';
});
}
render[settings.showEffect]();
};
})(jQuery);
================================================
FILE: docs/licenses/LICENSE
================================================
/* This license pertains to the docs template, except for the Navgoco jQuery component. */
The MIT License (MIT)
Original theme: Copyright (c) 2016 Tom Johnson
Modifications: Copyright (c) 2017 onwards fast.ai, Inc
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
================================================
FILE: docs/licenses/LICENSE-BSD-NAVGOCO.txt
================================================
/* This license pertains to the Navgoco jQuery component used for the sidebar. */
Copyright (c) 2013, Christodoulos Tsoulloftas, http://www.komposta.net
All rights reserved.
Redistribution and use in source and binary forms, with or without modification,
are permitted provided that the following conditions are met:
* Redistributions of source code must retain the above copyright notice,
this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above copyright notice,
this list of conditions and the following disclaimer in the documentation
and/or other materials provided with the distribution.
* Neither the name of the nor the names of its
contributors may be used to endorse or promote products derived from this
software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE
OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED
OF THE POSSIBILITY OF SUCH DAMAGE.
================================================
FILE: docs/sidebar.json
================================================
{
"online_triplet_loss": {
"Overview": "/",
"Losses": "/triplet_loss"
}
}
================================================
FILE: docs/sitemap.xml
================================================
---
layout: none
search: exclude
---
{% for post in site.posts %}
{% unless post.search == "exclude" %}
{{site.url}}{{post.url}}
{% endunless %}
{% endfor %}
{% for page in site.pages %}
{% unless page.search == "exclude" %}
{{site.url}}{{ page.url}}
{% endunless %}
{% endfor %}
================================================
FILE: docs/tooltips.json
================================================
---
layout: null
search: exclude
---
{
"entries":
[
{% for page in site.tooltips %}
{
"doc_id": "{{ page.doc_id }}",
"body": "{{ page.content | strip_newlines | replace: '\', '\\\\' | replace: '"', '\\"' }}"
} {% unless forloop.last %},{% endunless %}
{% endfor %}
]
}
================================================
FILE: docs/triplet_loss.html
================================================
---
title: Losses
keywords: fastai
sidebar: home_sidebar
summary: "Where all of the losses are situated."
description: "Where all of the losses are situated."
---
Build the triplet loss over a batch of embeddings.
For each anchor, we get the hardest positive and hardest negative to form a triplet.
Args:
labels: labels of the batch, of size (batch_size,)
embeddings: tensor of shape (batch_size, embed_dim)
margin: margin for triplet loss
squared: Boolean. If true, output is the pairwise squared euclidean distance matrix.
If false, output is the pairwise euclidean distance matrix.
Returns:
triplet_loss: scalar tensor containing the triplet loss
Build the triplet loss over a batch of embeddings.
We generate all the valid triplets and average the loss over the positive ones.
Args:
labels: labels of the batch, of size (batch_size,)
embeddings: tensor of shape (batch_size, embed_dim)
margin: margin for triplet loss
squared: Boolean. If true, output is the pairwise squared euclidean distance matrix.
If false, output is the pairwise euclidean distance matrix.
Returns:
triplet_loss: scalar tensor containing the triplet loss
{% endraw %}
================================================
FILE: index.ipynb
================================================
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"#hide\n",
"from online_triplet_loss.losses import *"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# online_triplet_loss\n",
"\n",
"> PyTorch conversion of the excellent post on the [same topic in Tensorflow](https://omoindrot.github.io/triplet-loss). Simply an implementation of a triple loss with online mining of candidate triplets used in semi-supervised learning."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Install"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"`pip install online_triplet_loss`\n",
"\n",
"Then import with:\n",
"`from online_triplet_loss.losses import *`\n",
"\n",
"PS: Requires Pytorch version 1.1.0 or above to use."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## How to use\n",
"\n",
"In these examples I use a really large margin, since the embedding space is so small. A more realistic margins seems to be between `0.1 and 2.0`"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from torch import nn\n",
"import torch\n",
"\n",
"model = nn.Embedding(10, 10)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": "Labels: tensor([6, 1, 3, 6, 6])\nEmbeddings: tensor([[-1.1335, 0.3364, -3.0174, -0.8732, -0.9301, 1.3619, 0.3746, 0.0457,\n 0.0180, -0.4500],\n [ 1.0757, -0.8420, -0.7630, -0.0746, 1.1545, 0.4017, 0.5587, 1.7947,\n 0.1992, -2.2288],\n [ 0.2646, 1.2383, 0.1949, 0.5743, -0.8460, -0.9929, -2.0350, 0.2095,\n 0.2129, -0.4855],\n [-1.1335, 0.3364, -3.0174, -0.8732, -0.9301, 1.3619, 0.3746, 0.0457,\n 0.0180, -0.4500],\n [-1.1335, 0.3364, -3.0174, -0.8732, -0.9301, 1.3619, 0.3746, 0.0457,\n 0.0180, -0.4500]], grad_fn=)\nLoss: tensor(95.1271, grad_fn=)\n"
}
],
"source": [
"#from online_triplet_loss.losses import *\n",
"labels = torch.randint(high=10, size=(5,)) # our five labels\n",
"\n",
"embeddings = model(labels)\n",
"print('Labels:', labels)\n",
"print('Embeddings:', embeddings)\n",
"loss = batch_hard_triplet_loss(labels, embeddings, margin=100)\n",
"print('Loss:', loss)\n",
"loss.backward()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": "Labels: tensor([6, 1, 3, 6, 6])\nEmbeddings: tensor([[-1.1335, 0.3364, -3.0174, -0.8732, -0.9301, 1.3619, 0.3746, 0.0457,\n 0.0180, -0.4500],\n [ 1.0757, -0.8420, -0.7630, -0.0746, 1.1545, 0.4017, 0.5587, 1.7947,\n 0.1992, -2.2288],\n [ 0.2646, 1.2383, 0.1949, 0.5743, -0.8460, -0.9929, -2.0350, 0.2095,\n 0.2129, -0.4855],\n [-1.1335, 0.3364, -3.0174, -0.8732, -0.9301, 1.3619, 0.3746, 0.0457,\n 0.0180, -0.4500],\n [-1.1335, 0.3364, -3.0174, -0.8732, -0.9301, 1.3619, 0.3746, 0.0457,\n 0.0180, -0.4500]], grad_fn=)\ntensor(94.9947, grad_fn=) tensor(1.)\nLoss: tensor(94.9947, grad_fn=)\n"
}
],
"source": [
"#from online_triplet_loss.losses import *\n",
"embeddings = model(labels)\n",
"print('Labels:', labels)\n",
"print('Embeddings:', embeddings)\n",
"loss, fraction_pos = batch_all_triplet_loss(labels, embeddings, squared=False, margin=100)\n",
"print('Loss:', loss)\n",
"loss.backward()"
]
},
{
"cell_type": "markdown",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"## References\n",
"* [Triplet Loss and Online Triplet Mining in Tensorflow](https://github.com/omoindrot/tensorflow-triplet-loss)\n",
"* [Facenet paper](https://arxiv.org/abs/1503.03832)\n",
"* [adambielski's nice implementation](https://github.com/adambielski/siamese-triplet) (unfortunately context switches between CPU / GPU)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
================================================
FILE: online_triplet_loss/__init__.py
================================================
__version__ = "0.0.6"
================================================
FILE: online_triplet_loss/_nbdev.py
================================================
# AUTOGENERATED BY NBDEV! DO NOT EDIT!
__all__ = ["index", "modules", "custom_doc_links", "git_url"]
index = {"batch_hard_triplet_loss": "triplet_loss.ipynb",
"batch_all_triplet_loss": "triplet_loss.ipynb"}
modules = ["losses.py"]
doc_url = "https://NegatioN.github.io/OnlineMiningTripletLoss/"
git_url = "https://github.com/NegatioN/OnlineMiningTripletLoss/tree/master/"
def custom_doc_links(name): return None
================================================
FILE: online_triplet_loss/losses.py
================================================
# AUTOGENERATED! DO NOT EDIT! File to edit: triplet_loss.ipynb (unless otherwise specified).
__all__ = ['batch_hard_triplet_loss', 'batch_all_triplet_loss']
# Cell
import torch
import torch.nn.functional as F
def _pairwise_distances(embeddings, squared=False):
"""Compute the 2D matrix of distances between all the embeddings.
Args:
embeddings: tensor of shape (batch_size, embed_dim)
squared: Boolean. If true, output is the pairwise squared euclidean distance matrix.
If false, output is the pairwise euclidean distance matrix.
Returns:
pairwise_distances: tensor of shape (batch_size, batch_size)
"""
dot_product = torch.matmul(embeddings, embeddings.t())
# Get squared L2 norm for each embedding. We can just take the diagonal of `dot_product`.
# This also provides more numerical stability (the diagonal of the result will be exactly 0).
# shape (batch_size,)
square_norm = torch.diag(dot_product)
# Compute the pairwise distance matrix as we have:
# ||a - b||^2 = ||a||^2 - 2 + ||b||^2
# shape (batch_size, batch_size)
distances = square_norm.unsqueeze(0) - 2.0 * dot_product + square_norm.unsqueeze(1)
# Because of computation errors, some distances might be negative so we put everything >= 0.0
distances[distances < 0] = 0
if not squared:
# Because the gradient of sqrt is infinite when distances == 0.0 (ex: on the diagonal)
# we need to add a small epsilon where distances == 0.0
mask = distances.eq(0).float()
distances = distances + mask * 1e-16
distances = (1.0 -mask) * torch.sqrt(distances)
return distances
def _get_triplet_mask(labels):
"""Return a 3D mask where mask[a, p, n] is True iff the triplet (a, p, n) is valid.
A triplet (i, j, k) is valid if:
- i, j, k are distinct
- labels[i] == labels[j] and labels[i] != labels[k]
Args:
labels: tf.int32 `Tensor` with shape [batch_size]
"""
# Check that i, j and k are distinct
indices_equal = torch.eye(labels.size(0), device=labels.device).bool()
indices_not_equal = ~indices_equal
i_not_equal_j = indices_not_equal.unsqueeze(2)
i_not_equal_k = indices_not_equal.unsqueeze(1)
j_not_equal_k = indices_not_equal.unsqueeze(0)
distinct_indices = (i_not_equal_j & i_not_equal_k) & j_not_equal_k
label_equal = labels.unsqueeze(0) == labels.unsqueeze(1)
i_equal_j = label_equal.unsqueeze(2)
i_equal_k = label_equal.unsqueeze(1)
valid_labels = ~i_equal_k & i_equal_j
return valid_labels & distinct_indices
def _get_anchor_positive_triplet_mask(labels):
"""Return a 2D mask where mask[a, p] is True iff a and p are distinct and have same label.
Args:
labels: tf.int32 `Tensor` with shape [batch_size]
Returns:
mask: tf.bool `Tensor` with shape [batch_size, batch_size]
"""
# Check that i and j are distinct
indices_equal = torch.eye(labels.size(0), device=labels.device).bool()
indices_not_equal = ~indices_equal
# Check if labels[i] == labels[j]
# Uses broadcasting where the 1st argument has shape (1, batch_size) and the 2nd (batch_size, 1)
labels_equal = labels.unsqueeze(0) == labels.unsqueeze(1)
return labels_equal & indices_not_equal
def _get_anchor_negative_triplet_mask(labels):
"""Return a 2D mask where mask[a, n] is True iff a and n have distinct labels.
Args:
labels: tf.int32 `Tensor` with shape [batch_size]
Returns:
mask: tf.bool `Tensor` with shape [batch_size, batch_size]
"""
# Check if labels[i] != labels[k]
# Uses broadcasting where the 1st argument has shape (1, batch_size) and the 2nd (batch_size, 1)
return ~(labels.unsqueeze(0) == labels.unsqueeze(1))
# Cell
def batch_hard_triplet_loss(labels, embeddings, margin, squared=False):
"""Build the triplet loss over a batch of embeddings.
For each anchor, we get the hardest positive and hardest negative to form a triplet.
Args:
labels: labels of the batch, of size (batch_size,)
embeddings: tensor of shape (batch_size, embed_dim)
margin: margin for triplet loss
squared: Boolean. If true, output is the pairwise squared euclidean distance matrix.
If false, output is the pairwise euclidean distance matrix.
Returns:
triplet_loss: scalar tensor containing the triplet loss
"""
# Get the pairwise distance matrix
pairwise_dist = _pairwise_distances(embeddings, squared=squared)
# For each anchor, get the hardest positive
# First, we need to get a mask for every valid positive (they should have same label)
mask_anchor_positive = _get_anchor_positive_triplet_mask(labels).float()
# We put to 0 any element where (a, p) is not valid (valid if a != p and label(a) == label(p))
anchor_positive_dist = mask_anchor_positive * pairwise_dist
# shape (batch_size, 1)
hardest_positive_dist, _ = anchor_positive_dist.max(1, keepdim=True)
# For each anchor, get the hardest negative
# First, we need to get a mask for every valid negative (they should have different labels)
mask_anchor_negative = _get_anchor_negative_triplet_mask(labels).float()
# We add the maximum value in each row to the invalid negatives (label(a) == label(n))
max_anchor_negative_dist, _ = pairwise_dist.max(1, keepdim=True)
anchor_negative_dist = pairwise_dist + max_anchor_negative_dist * (1.0 - mask_anchor_negative)
# shape (batch_size,)
hardest_negative_dist, _ = anchor_negative_dist.min(1, keepdim=True)
# Combine biggest d(a, p) and smallest d(a, n) into final triplet loss
tl = hardest_positive_dist - hardest_negative_dist + margin
tl = F.relu(tl)
triplet_loss = tl.mean()
return triplet_loss
# Cell
def batch_all_triplet_loss(labels, embeddings, margin, squared=False):
"""Build the triplet loss over a batch of embeddings.
We generate all the valid triplets and average the loss over the positive ones.
Args:
labels: labels of the batch, of size (batch_size,)
embeddings: tensor of shape (batch_size, embed_dim)
margin: margin for triplet loss
squared: Boolean. If true, output is the pairwise squared euclidean distance matrix.
If false, output is the pairwise euclidean distance matrix.
Returns:
triplet_loss: scalar tensor containing the triplet loss
"""
# Get the pairwise distance matrix
pairwise_dist = _pairwise_distances(embeddings, squared=squared)
anchor_positive_dist = pairwise_dist.unsqueeze(2)
anchor_negative_dist = pairwise_dist.unsqueeze(1)
# Compute a 3D tensor of size (batch_size, batch_size, batch_size)
# triplet_loss[i, j, k] will contain the triplet loss of anchor=i, positive=j, negative=k
# Uses broadcasting where the 1st argument has shape (batch_size, batch_size, 1)
# and the 2nd (batch_size, 1, batch_size)
triplet_loss = anchor_positive_dist - anchor_negative_dist + margin
# Put to zero the invalid triplets
# (where label(a) != label(p) or label(n) == label(a) or a == p)
mask = _get_triplet_mask(labels)
triplet_loss = mask.float() * triplet_loss
# Remove negative losses (i.e. the easy triplets)
triplet_loss = F.relu(triplet_loss)
# Count number of positive triplets (where triplet_loss > 0)
valid_triplets = triplet_loss[triplet_loss > 1e-16]
num_positive_triplets = valid_triplets.size(0)
num_valid_triplets = mask.sum()
fraction_positive_triplets = num_positive_triplets / (num_valid_triplets.float() + 1e-16)
# Get final mean triplet loss over the positive valid triplets
triplet_loss = triplet_loss.sum() / (num_positive_triplets + 1e-16)
return triplet_loss, fraction_positive_triplets
================================================
FILE: settings.ini
================================================
[DEFAULT]
lib_name = online_triplet_loss
user = NegatioN
description = "Online mining triplet losses for Pytorch"
keywords = pytorch loss online triplet mining
author = "Joakim Rishaug"
author_email = "joakimrishaug@notmyrealemail.com"
copyright = "Joakim Rishaug"
branch = master
version = 0.0.6
min_python = 3.6
audience = Developers
language = English
custom_sidebar = False
license = apache2
status = 2
nbs_path = .
doc_path = docs
requirements = numpy>=1.15.4 torch>=1.1.0
repo_name = OnlineMiningTripletLoss
doc_host = https://NegatioN.github.io
doc_baseurl = /OnlineMiningTripletLoss/
git_url = https://github.com/NegatioN/OnlineMiningTripletLoss/tree/master/
lib_path = online_triplet_loss
title = OnlineMiningTripletLoss
host = github
================================================
FILE: setup.py
================================================
from packaging.version import parse
from configparser import ConfigParser
import setuptools
assert parse(setuptools.__version__)>=parse('36.2')
# note: all settings are in settings.ini; edit there, not here
config = ConfigParser(delimiters=['='])
config.read('settings.ini')
cfg = config['DEFAULT']
cfg_keys = 'version description keywords author author_email'.split()
expected = cfg_keys + "lib_name user branch license status min_python audience language".split()
for o in expected: assert o in cfg, "missing expected setting: {}".format(o)
setup_cfg = {o:cfg[o] for o in cfg_keys}
licenses = {
'apache2': ('Apache Software License 2.0','OSI Approved :: Apache Software License'),
}
statuses = [ '1 - Planning', '2 - Pre-Alpha', '3 - Alpha',
'4 - Beta', '5 - Production/Stable', '6 - Mature', '7 - Inactive' ]
py_versions = '2.0 2.1 2.2 2.3 2.4 2.5 2.6 2.7 3.0 3.1 3.2 3.3 3.4 3.5 3.6 3.7 3.8'.split()
requirements = cfg.get('requirements','').split()
lic = licenses[cfg['license']]
min_python = cfg['min_python']
setuptools.setup(
name = cfg['lib_name'],
license = lic[0],
classifiers = [
'Development Status :: ' + statuses[int(cfg['status'])],
'Intended Audience :: ' + cfg['audience'].title(),
'License :: ' + lic[1],
'Natural Language :: ' + cfg['language'].title(),
] + ['Programming Language :: Python :: '+o for o in py_versions[py_versions.index(min_python):]],
url = 'https://github.com/{}/{}'.format(cfg['user'],cfg['repo_name']),
packages = setuptools.find_packages(),
include_package_data = True,
install_requires = requirements,
python_requires = '>=' + cfg['min_python'],
long_description = open('README.md').read(),
long_description_content_type = 'text/markdown',
zip_safe = False,
entry_points = { 'console_scripts': cfg.get('console_scripts','').split() },
**setup_cfg)
================================================
FILE: triplet_loss.ipynb
================================================
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# default_exp losses"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Losses\n",
"> Where all of the losses are situated."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"#hide\n",
"from nbdev.showdoc import *"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"#export\n",
"import torch\n",
"import torch.nn.functional as F\n",
"def _pairwise_distances(embeddings, squared=False):\n",
" \"\"\"Compute the 2D matrix of distances between all the embeddings.\n",
"\n",
" Args:\n",
" embeddings: tensor of shape (batch_size, embed_dim)\n",
" squared: Boolean. If true, output is the pairwise squared euclidean distance matrix.\n",
" If false, output is the pairwise euclidean distance matrix.\n",
"\n",
" Returns:\n",
" pairwise_distances: tensor of shape (batch_size, batch_size)\n",
" \"\"\"\n",
" dot_product = torch.matmul(embeddings, embeddings.t())\n",
"\n",
" # Get squared L2 norm for each embedding. We can just take the diagonal of `dot_product`.\n",
" # This also provides more numerical stability (the diagonal of the result will be exactly 0).\n",
" # shape (batch_size,)\n",
" square_norm = torch.diag(dot_product)\n",
"\n",
" # Compute the pairwise distance matrix as we have:\n",
" # ||a - b||^2 = ||a||^2 - 2 + ||b||^2\n",
" # shape (batch_size, batch_size)\n",
" distances = square_norm.unsqueeze(0) - 2.0 * dot_product + square_norm.unsqueeze(1)\n",
"\n",
" # Because of computation errors, some distances might be negative so we put everything >= 0.0\n",
" distances[distances < 0] = 0\n",
"\n",
" if not squared:\n",
" # Because the gradient of sqrt is infinite when distances == 0.0 (ex: on the diagonal)\n",
" # we need to add a small epsilon where distances == 0.0\n",
" mask = distances.eq(0).float()\n",
" distances = distances + mask * 1e-16\n",
"\n",
" distances = (1.0 -mask) * torch.sqrt(distances)\n",
"\n",
" return distances\n",
"\n",
"def _get_triplet_mask(labels):\n",
" \"\"\"Return a 3D mask where mask[a, p, n] is True iff the triplet (a, p, n) is valid.\n",
" A triplet (i, j, k) is valid if:\n",
" - i, j, k are distinct\n",
" - labels[i] == labels[j] and labels[i] != labels[k]\n",
" Args:\n",
" labels: tf.int32 `Tensor` with shape [batch_size]\n",
" \"\"\"\n",
" # Check that i, j and k are distinct\n",
" indices_equal = torch.eye(labels.size(0), device=labels.device).bool()\n",
" indices_not_equal = ~indices_equal\n",
" i_not_equal_j = indices_not_equal.unsqueeze(2)\n",
" i_not_equal_k = indices_not_equal.unsqueeze(1)\n",
" j_not_equal_k = indices_not_equal.unsqueeze(0)\n",
"\n",
" distinct_indices = (i_not_equal_j & i_not_equal_k) & j_not_equal_k\n",
"\n",
"\n",
" label_equal = labels.unsqueeze(0) == labels.unsqueeze(1)\n",
" i_equal_j = label_equal.unsqueeze(2)\n",
" i_equal_k = label_equal.unsqueeze(1)\n",
"\n",
" valid_labels = ~i_equal_k & i_equal_j\n",
"\n",
" return valid_labels & distinct_indices\n",
"\n",
"\n",
"def _get_anchor_positive_triplet_mask(labels):\n",
" \"\"\"Return a 2D mask where mask[a, p] is True iff a and p are distinct and have same label.\n",
" Args:\n",
" labels: tf.int32 `Tensor` with shape [batch_size]\n",
" Returns:\n",
" mask: tf.bool `Tensor` with shape [batch_size, batch_size]\n",
" \"\"\"\n",
" # Check that i and j are distinct\n",
" indices_equal = torch.eye(labels.size(0), device=labels.device).bool()\n",
" indices_not_equal = ~indices_equal\n",
"\n",
" # Check if labels[i] == labels[j]\n",
" # Uses broadcasting where the 1st argument has shape (1, batch_size) and the 2nd (batch_size, 1)\n",
" labels_equal = labels.unsqueeze(0) == labels.unsqueeze(1)\n",
"\n",
" return labels_equal & indices_not_equal\n",
"\n",
"\n",
"def _get_anchor_negative_triplet_mask(labels):\n",
" \"\"\"Return a 2D mask where mask[a, n] is True iff a and n have distinct labels.\n",
" Args:\n",
" labels: tf.int32 `Tensor` with shape [batch_size]\n",
" Returns:\n",
" mask: tf.bool `Tensor` with shape [batch_size, batch_size]\n",
" \"\"\"\n",
" # Check if labels[i] != labels[k]\n",
" # Uses broadcasting where the 1st argument has shape (1, batch_size) and the 2nd (batch_size, 1)\n",
"\n",
" return ~(labels.unsqueeze(0) == labels.unsqueeze(1))\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"#hide\n",
"import numpy as np\n",
"\n",
"# Test-suite from https://github.com/omoindrot/tensorflow-triplet-loss/blob/master/model/tests/test_triplet_loss.py\n",
"# Skipped the `test_gradients_pairwise_distances()` test since it's trivial to see if your model loss turns NaN\n",
"# and porting it proved more difficult than expected.\n",
"\n",
"def pairwise_distance_np(feature, squared=False):\n",
" \"\"\"Computes the pairwise distance matrix in numpy.\n",
" Args:\n",
" feature: 2-D numpy array of size [number of data, feature dimension]\n",
" squared: Boolean. If true, output is the pairwise squared euclidean\n",
" distance matrix; else, output is the pairwise euclidean distance matrix.\n",
" Returns:\n",
" pairwise_distances: 2-D numpy array of size\n",
" [number of data, number of data].\n",
" \"\"\"\n",
" triu = np.triu_indices(feature.shape[0], 1)\n",
" upper_tri_pdists = np.linalg.norm(feature[triu[1]] - feature[triu[0]], axis=1)\n",
" if squared:\n",
" upper_tri_pdists **= 2.\n",
" num_data = feature.shape[0]\n",
" pairwise_distances = np.zeros((num_data, num_data))\n",
" pairwise_distances[np.triu_indices(num_data, 1)] = upper_tri_pdists\n",
" # Make symmetrical.\n",
" pairwise_distances = pairwise_distances + pairwise_distances.T - np.diag(\n",
" pairwise_distances.diagonal())\n",
" return pairwise_distances\n",
"\n",
"def test_pairwise_distances():\n",
" \"\"\"Test the pairwise distances function.\"\"\"\n",
" num_data = 64\n",
" feat_dim = 6\n",
" np.random.seed(42)\n",
"\n",
" embeddings = np.random.randn(num_data, feat_dim).astype(np.float32)\n",
" embeddings[1] = embeddings[0] # to get distance 0\n",
"\n",
" for squared in [True, False]:\n",
" res_np = pairwise_distance_np(embeddings, squared=squared)\n",
" res_pt = _pairwise_distances(torch.from_numpy(embeddings), squared=squared)\n",
" assert np.allclose(res_np, res_pt)\n",
"\n",
"def test_pairwise_distances_are_positive():\n",
" \"\"\"Test that the pairwise distances are always positive.\n",
" Use a tricky case where numerical errors are common.\n",
" \"\"\"\n",
" num_data = 64\n",
" feat_dim = 6\n",
" np.random.seed(42)\n",
"\n",
" # Create embeddings very close to each other in [1.0 - 2e-7, 1.0 + 2e-7]\n",
" # This will encourage errors in the computation\n",
" embeddings = 1.0 + 2e-7 * np.random.randn(num_data, feat_dim).astype(np.float32)\n",
" embeddings[1] = embeddings[0] # to get distance 0\n",
"\n",
" for squared in [True, False]:\n",
" res_tf = _pairwise_distances(torch.from_numpy(embeddings), squared=squared)\n",
" assert res_tf[res_tf < 0].sum() == 0\n",
"\n",
"\n",
"def test_triplet_mask():\n",
" \"\"\"Test function _get_triplet_mask.\"\"\"\n",
" num_data = 64\n",
" num_classes = 10\n",
" np.random.seed(42)\n",
"\n",
" labels = np.random.randint(0, num_classes, size=(num_data)).astype(np.float32)\n",
"\n",
" mask_np = np.zeros((num_data, num_data, num_data))\n",
" for i in range(num_data):\n",
" for j in range(num_data):\n",
" for k in range(num_data):\n",
" distinct = (i != j and i != k and j != k)\n",
" valid = (labels[i] == labels[j]) and (labels[i] != labels[k])\n",
" mask_np[i, j, k] = (distinct and valid)\n",
"\n",
" mask_tf_val = _get_triplet_mask(torch.from_numpy(labels))\n",
" assert np.allclose(mask_np, mask_tf_val)\n",
"\n",
"def test_anchor_positive_triplet_mask():\n",
" \"\"\"Test function _get_anchor_positive_triplet_mask.\"\"\"\n",
" num_data = 64\n",
" num_classes = 10\n",
" np.random.seed(42)\n",
"\n",
" labels = np.random.randint(0, num_classes, size=(num_data)).astype(np.float32)\n",
"\n",
" mask_np = np.zeros((num_data, num_data))\n",
" for i in range(num_data):\n",
" for j in range(num_data):\n",
" distinct = (i != j)\n",
" valid = labels[i] == labels[j]\n",
" mask_np[i, j] = (distinct and valid)\n",
"\n",
" mask_tf_val = _get_anchor_positive_triplet_mask(torch.from_numpy(labels))\n",
"\n",
" assert np.allclose(mask_np, mask_tf_val)\n",
"\n",
"def test_anchor_negative_triplet_mask():\n",
" \"\"\"Test function _get_anchor_negative_triplet_mask.\"\"\"\n",
" num_data = 64\n",
" num_classes = 10\n",
" np.random.seed(42)\n",
"\n",
" labels = np.random.randint(0, num_classes, size=(num_data)).astype(np.float32)\n",
"\n",
" mask_np = np.zeros((num_data, num_data))\n",
" for i in range(num_data):\n",
" for k in range(num_data):\n",
" distinct = (i != k)\n",
" valid = (labels[i] != labels[k])\n",
" mask_np[i, k] = (distinct and valid)\n",
"\n",
" mask_tf_val = _get_anchor_negative_triplet_mask(torch.from_numpy(labels))\n",
"\n",
" assert np.allclose(mask_np, mask_tf_val)\n",
"\n",
"test_pairwise_distances()\n",
"test_pairwise_distances_are_positive()\n",
"test_triplet_mask()\n",
"test_anchor_positive_triplet_mask()\n",
"test_anchor_negative_triplet_mask()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"#export\n",
"def batch_hard_triplet_loss(labels, embeddings, margin, squared=False):\n",
" \"\"\"Build the triplet loss over a batch of embeddings.\n",
"\n",
" For each anchor, we get the hardest positive and hardest negative to form a triplet.\n",
"\n",
" Args:\n",
" labels: labels of the batch, of size (batch_size,)\n",
" embeddings: tensor of shape (batch_size, embed_dim)\n",
" margin: margin for triplet loss\n",
" squared: Boolean. If true, output is the pairwise squared euclidean distance matrix.\n",
" If false, output is the pairwise euclidean distance matrix.\n",
"\n",
" Returns:\n",
" triplet_loss: scalar tensor containing the triplet loss\n",
" \"\"\"\n",
" # Get the pairwise distance matrix\n",
" pairwise_dist = _pairwise_distances(embeddings, squared=squared)\n",
"\n",
" # For each anchor, get the hardest positive\n",
" # First, we need to get a mask for every valid positive (they should have same label)\n",
" mask_anchor_positive = _get_anchor_positive_triplet_mask(labels).float()\n",
"\n",
" # We put to 0 any element where (a, p) is not valid (valid if a != p and label(a) == label(p))\n",
" anchor_positive_dist = mask_anchor_positive * pairwise_dist\n",
"\n",
" # shape (batch_size, 1)\n",
" hardest_positive_dist, _ = anchor_positive_dist.max(1, keepdim=True)\n",
"\n",
" # For each anchor, get the hardest negative\n",
" # First, we need to get a mask for every valid negative (they should have different labels)\n",
" mask_anchor_negative = _get_anchor_negative_triplet_mask(labels).float()\n",
"\n",
" # We add the maximum value in each row to the invalid negatives (label(a) == label(n))\n",
" max_anchor_negative_dist, _ = pairwise_dist.max(1, keepdim=True)\n",
" anchor_negative_dist = pairwise_dist + max_anchor_negative_dist * (1.0 - mask_anchor_negative)\n",
"\n",
" # shape (batch_size,)\n",
" hardest_negative_dist, _ = anchor_negative_dist.min(1, keepdim=True)\n",
"\n",
" # Combine biggest d(a, p) and smallest d(a, n) into final triplet loss\n",
" tl = hardest_positive_dist - hardest_negative_dist + margin\n",
" tl = F.relu(tl)\n",
" triplet_loss = tl.mean()\n",
"\n",
" return triplet_loss"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"#export\n",
"def batch_all_triplet_loss(labels, embeddings, margin, squared=False):\n",
" \"\"\"Build the triplet loss over a batch of embeddings.\n",
"\n",
" We generate all the valid triplets and average the loss over the positive ones.\n",
"\n",
" Args:\n",
" labels: labels of the batch, of size (batch_size,)\n",
" embeddings: tensor of shape (batch_size, embed_dim)\n",
" margin: margin for triplet loss\n",
" squared: Boolean. If true, output is the pairwise squared euclidean distance matrix.\n",
" If false, output is the pairwise euclidean distance matrix.\n",
"\n",
" Returns:\n",
" triplet_loss: scalar tensor containing the triplet loss\n",
" \"\"\"\n",
" # Get the pairwise distance matrix\n",
" pairwise_dist = _pairwise_distances(embeddings, squared=squared)\n",
"\n",
" anchor_positive_dist = pairwise_dist.unsqueeze(2)\n",
" anchor_negative_dist = pairwise_dist.unsqueeze(1)\n",
"\n",
" # Compute a 3D tensor of size (batch_size, batch_size, batch_size)\n",
" # triplet_loss[i, j, k] will contain the triplet loss of anchor=i, positive=j, negative=k\n",
" # Uses broadcasting where the 1st argument has shape (batch_size, batch_size, 1)\n",
" # and the 2nd (batch_size, 1, batch_size)\n",
" triplet_loss = anchor_positive_dist - anchor_negative_dist + margin\n",
"\n",
"\n",
"\n",
" # Put to zero the invalid triplets\n",
" # (where label(a) != label(p) or label(n) == label(a) or a == p)\n",
" mask = _get_triplet_mask(labels)\n",
" triplet_loss = mask.float() * triplet_loss\n",
"\n",
" # Remove negative losses (i.e. the easy triplets)\n",
" triplet_loss = F.relu(triplet_loss)\n",
"\n",
" # Count number of positive triplets (where triplet_loss > 0)\n",
" valid_triplets = triplet_loss[triplet_loss > 1e-16]\n",
" num_positive_triplets = valid_triplets.size(0)\n",
" num_valid_triplets = mask.sum()\n",
"\n",
" fraction_positive_triplets = num_positive_triplets / (num_valid_triplets.float() + 1e-16)\n",
"\n",
" # Get final mean triplet loss over the positive valid triplets\n",
" triplet_loss = triplet_loss.sum() / (num_positive_triplets + 1e-16)\n",
"\n",
" return triplet_loss, fraction_positive_triplets\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"tensor(0.) tensor(0.)\n",
"tensor(0.) tensor(0.)\n",
"tensor(0.7060) tensor(0.7155)\n",
"tensor(0.3978) tensor(0.8276)\n"
]
}
],
"source": [
"#hide\n",
"def test_simple_batch_all_triplet_loss():\n",
" \"\"\"Test the triplet loss with batch all triplet mining in a simple case.\n",
" There is just one class in this super simple edge case, and we want to make sure that\n",
" the loss is 0.\n",
" \"\"\"\n",
" num_data = 10\n",
" feat_dim = 6\n",
" margin = 0.2\n",
" num_classes = 1\n",
" np.random.seed(42)\n",
"\n",
" embeddings = np.random.rand(num_data, feat_dim).astype(np.float32)\n",
" labels = np.random.randint(0, num_classes, size=(num_data)).astype(np.float32)\n",
" labels, embeddings = torch.from_numpy(labels), torch.from_numpy(embeddings)\n",
"\n",
" for squared in [True, False]:\n",
" loss_np = 0.0\n",
"\n",
" # Compute the loss in TF.\n",
" loss_tf_val, fraction_val = batch_all_triplet_loss(labels, embeddings, margin, squared=squared)\n",
" print(loss_tf_val, fraction_val)\n",
" assert np.allclose(loss_np, loss_tf_val)\n",
" assert np.allclose(fraction_val, 0.0)\n",
"\n",
"\n",
"def test_batch_all_triplet_loss():\n",
" \"\"\"Test the triplet loss with batch all triplet mining\"\"\"\n",
" num_data = 10\n",
" feat_dim = 6\n",
" margin = 0.2\n",
" num_classes = 5\n",
" np.random.seed(42)\n",
"\n",
" embeddings = np.random.rand(num_data, feat_dim).astype(np.float32)\n",
" labels = np.random.randint(0, num_classes, size=(num_data)).astype(np.float32)\n",
"\n",
" for squared in [True, False]:\n",
" pdist_matrix = pairwise_distance_np(embeddings, squared=squared)\n",
"\n",
" loss_np = 0.0\n",
" num_positives = 0.0\n",
" num_valid = 0.0\n",
" for i in range(num_data):\n",
" for j in range(num_data):\n",
" for k in range(num_data):\n",
" distinct = (i != j and i != k and j != k)\n",
" valid = (labels[i] == labels[j]) and (labels[i] != labels[k])\n",
" if distinct and valid:\n",
" num_valid += 1.0\n",
"\n",
" pos_distance = pdist_matrix[i][j]\n",
" neg_distance = pdist_matrix[i][k]\n",
"\n",
" loss = np.maximum(0.0, pos_distance - neg_distance + margin)\n",
" loss_np += loss\n",
"\n",
" num_positives += (loss > 0)\n",
"\n",
" loss_np /= num_positives\n",
"\n",
" # Compute the loss in TF.\n",
" loss_tf_val, fraction_val = batch_all_triplet_loss(torch.from_numpy(labels), torch.from_numpy(embeddings), margin, squared=squared)\n",
" print(loss_tf_val, fraction_val)\n",
" assert np.allclose(loss_np, loss_tf_val)\n",
" assert np.allclose(num_positives / num_valid, fraction_val)\n",
"\n",
"def test_batch_hard_triplet_loss():\n",
" \"\"\"Test the triplet loss with batch hard triplet mining\"\"\"\n",
" num_data = 50\n",
" feat_dim = 6\n",
" margin = 0.2\n",
" num_classes = 5\n",
" np.random.seed(42)\n",
"\n",
" embeddings = np.random.rand(num_data, feat_dim).astype(np.float32)\n",
" labels = np.random.randint(0, num_classes, size=(num_data)).astype(np.float32)\n",
"\n",
" for squared in [True, False]:\n",
" pdist_matrix = pairwise_distance_np(embeddings, squared=squared)\n",
"\n",
" loss_np = 0.0\n",
" for i in range(num_data):\n",
" # Select the hardest positive\n",
" max_pos_dist = np.max(pdist_matrix[i][labels == labels[i]])\n",
"\n",
" # Select the hardest negative\n",
" min_neg_dist = np.min(pdist_matrix[i][labels != labels[i]])\n",
"\n",
" loss = np.maximum(0.0, max_pos_dist - min_neg_dist + margin)\n",
" loss_np += loss\n",
"\n",
" loss_np /= num_data\n",
"\n",
" # Compute the loss in TF.\n",
" loss_tf_val = batch_hard_triplet_loss(torch.from_numpy(labels), torch.from_numpy(embeddings), margin, squared=squared)\n",
" assert np.allclose(loss_np, loss_tf_val)\n",
"\n",
"test_simple_batch_all_triplet_loss()\n",
"test_batch_all_triplet_loss()\n",
"test_batch_hard_triplet_loss()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
}
},
"nbformat": 4,
"nbformat_minor": 4
}