Repository: facebookresearch/SING
Branch: main
Commit: 72054bdb23b4
Files: 28
Total size: 91.5 KB
Directory structure:
gitextract_6jkoz_29/
├── CODE_OF_CONDUCT.md
├── CONTRIBUTING.md
├── LICENSE
├── MANIFEST.in
├── README.md
├── environment.yml
├── nsynth_100_test.txt
├── requirements.txt
├── setup.py
└── sing/
├── __init__.py
├── ae/
│ ├── __init__.py
│ ├── models.py
│ ├── trainer.py
│ └── utils.py
├── dsp.py
├── fondation/
│ ├── __init__.py
│ ├── batch.py
│ ├── datasets.py
│ ├── trainer.py
│ └── utils.py
├── generate.py
├── nsynth/
│ └── __init__.py
├── parser.py
├── sequence/
│ ├── __init__.py
│ ├── models.py
│ ├── trainer.py
│ └── utils.py
└── train.py
================================================
FILE CONTENTS
================================================
================================================
FILE: CODE_OF_CONDUCT.md
================================================
# Code of Conduct
Facebook has adopted a Code of Conduct that we expect project participants to adhere to.
Please read the [full text](https://code.fb.com/codeofconduct/)
so that you can understand what actions will and will not be tolerated.
================================================
FILE: CONTRIBUTING.md
================================================
# Contributing to SING
## Pull Requests
In order to accept your pull request, we need you to submit a CLA. You only need
to do this once to work on any of Facebook's open source projects.
Complete your CLA here: <https://code.facebook.com/cla>
SING is the implementation of a research paper.
Therefore, we do not plan on accepting many pull requests for new features.
We certainly welcome them for bug fixes.
## Issues
We use GitHub issues to track public bugs. Please ensure your description is
clear and has sufficient instructions to be able to reproduce the issue.
## License
By contributing to this repository, you agree that your contributions will be licensed
under the LICENSE file in the root directory of this source tree.
================================================
FILE: LICENSE
================================================
Attribution-NonCommercial 4.0 International
=======================================================================
Creative Commons Corporation ("Creative Commons") is not a law firm and
does not provide legal services or legal advice. Distribution of
Creative Commons public licenses does not create a lawyer-client or
other relationship. Creative Commons makes its licenses and related
information available on an "as-is" basis. Creative Commons gives no
warranties regarding its licenses, any material licensed under their
terms and conditions, or any related information. Creative Commons
disclaims all liability for damages resulting from their use to the
fullest extent possible.
Using Creative Commons Public Licenses
Creative Commons public licenses provide a standard set of terms and
conditions that creators and other rights holders may use to share
original works of authorship and other material subject to copyright
and certain other rights specified in the public license below. The
following considerations are for informational purposes only, are not
exhaustive, and do not form part of our licenses.
Considerations for licensors: Our public licenses are
intended for use by those authorized to give the public
permission to use material in ways otherwise restricted by
copyright and certain other rights. Our licenses are
irrevocable. Licensors should read and understand the terms
and conditions of the license they choose before applying it.
Licensors should also secure all rights necessary before
applying our licenses so that the public can reuse the
material as expected. Licensors should clearly mark any
material not subject to the license. This includes other CC-
licensed material, or material used under an exception or
limitation to copyright. More considerations for licensors:
wiki.creativecommons.org/Considerations_for_licensors
Considerations for the public: By using one of our public
licenses, a licensor grants the public permission to use the
licensed material under specified terms and conditions. If
the licensor's permission is not necessary for any reason--for
example, because of any applicable exception or limitation to
copyright--then that use is not regulated by the license. Our
licenses grant only permissions under copyright and certain
other rights that a licensor has authority to grant. Use of
the licensed material may still be restricted for other
reasons, including because others have copyright or other
rights in the material. A licensor may make special requests,
such as asking that all changes be marked or described.
Although not required by our licenses, you are encouraged to
respect those requests where reasonable. More_considerations
for the public:
wiki.creativecommons.org/Considerations_for_licensees
=======================================================================
Creative Commons Attribution-NonCommercial 4.0 International Public
License
By exercising the Licensed Rights (defined below), You accept and agree
to be bound by the terms and conditions of this Creative Commons
Attribution-NonCommercial 4.0 International Public License ("Public
License"). To the extent this Public License may be interpreted as a
contract, You are granted the Licensed Rights in consideration of Your
acceptance of these terms and conditions, and the Licensor grants You
such rights in consideration of benefits the Licensor receives from
making the Licensed Material available under these terms and
conditions.
Section 1 -- Definitions.
a. Adapted Material means material subject to Copyright and Similar
Rights that is derived from or based upon the Licensed Material
and in which the Licensed Material is translated, altered,
arranged, transformed, or otherwise modified in a manner requiring
permission under the Copyright and Similar Rights held by the
Licensor. For purposes of this Public License, where the Licensed
Material is a musical work, performance, or sound recording,
Adapted Material is always produced where the Licensed Material is
synched in timed relation with a moving image.
b. Adapter's License means the license You apply to Your Copyright
and Similar Rights in Your contributions to Adapted Material in
accordance with the terms and conditions of this Public License.
c. Copyright and Similar Rights means copyright and/or similar rights
closely related to copyright including, without limitation,
performance, broadcast, sound recording, and Sui Generis Database
Rights, without regard to how the rights are labeled or
categorized. For purposes of this Public License, the rights
specified in Section 2(b)(1)-(2) are not Copyright and Similar
Rights.
d. Effective Technological Measures means those measures that, in the
absence of proper authority, may not be circumvented under laws
fulfilling obligations under Article 11 of the WIPO Copyright
Treaty adopted on December 20, 1996, and/or similar international
agreements.
e. Exceptions and Limitations means fair use, fair dealing, and/or
any other exception or limitation to Copyright and Similar Rights
that applies to Your use of the Licensed Material.
f. Licensed Material means the artistic or literary work, database,
or other material to which the Licensor applied this Public
License.
g. Licensed Rights means the rights granted to You subject to the
terms and conditions of this Public License, which are limited to
all Copyright and Similar Rights that apply to Your use of the
Licensed Material and that the Licensor has authority to license.
h. Licensor means the individual(s) or entity(ies) granting rights
under this Public License.
i. NonCommercial means not primarily intended for or directed towards
commercial advantage or monetary compensation. For purposes of
this Public License, the exchange of the Licensed Material for
other material subject to Copyright and Similar Rights by digital
file-sharing or similar means is NonCommercial provided there is
no payment of monetary compensation in connection with the
exchange.
j. Share means to provide material to the public by any means or
process that requires permission under the Licensed Rights, such
as reproduction, public display, public performance, distribution,
dissemination, communication, or importation, and to make material
available to the public including in ways that members of the
public may access the material from a place and at a time
individually chosen by them.
k. Sui Generis Database Rights means rights other than copyright
resulting from Directive 96/9/EC of the European Parliament and of
the Council of 11 March 1996 on the legal protection of databases,
as amended and/or succeeded, as well as other essentially
equivalent rights anywhere in the world.
l. You means the individual or entity exercising the Licensed Rights
under this Public License. Your has a corresponding meaning.
Section 2 -- Scope.
a. License grant.
1. Subject to the terms and conditions of this Public License,
the Licensor hereby grants You a worldwide, royalty-free,
non-sublicensable, non-exclusive, irrevocable license to
exercise the Licensed Rights in the Licensed Material to:
a. reproduce and Share the Licensed Material, in whole or
in part, for NonCommercial purposes only; and
b. produce, reproduce, and Share Adapted Material for
NonCommercial purposes only.
2. Exceptions and Limitations. For the avoidance of doubt, where
Exceptions and Limitations apply to Your use, this Public
License does not apply, and You do not need to comply with
its terms and conditions.
3. Term. The term of this Public License is specified in Section
6(a).
4. Media and formats; technical modifications allowed. The
Licensor authorizes You to exercise the Licensed Rights in
all media and formats whether now known or hereafter created,
and to make technical modifications necessary to do so. The
Licensor waives and/or agrees not to assert any right or
authority to forbid You from making technical modifications
necessary to exercise the Licensed Rights, including
technical modifications necessary to circumvent Effective
Technological Measures. For purposes of this Public License,
simply making modifications authorized by this Section 2(a)
(4) never produces Adapted Material.
5. Downstream recipients.
a. Offer from the Licensor -- Licensed Material. Every
recipient of the Licensed Material automatically
receives an offer from the Licensor to exercise the
Licensed Rights under the terms and conditions of this
Public License.
b. No downstream restrictions. You may not offer or impose
any additional or different terms or conditions on, or
apply any Effective Technological Measures to, the
Licensed Material if doing so restricts exercise of the
Licensed Rights by any recipient of the Licensed
Material.
6. No endorsement. Nothing in this Public License constitutes or
may be construed as permission to assert or imply that You
are, or that Your use of the Licensed Material is, connected
with, or sponsored, endorsed, or granted official status by,
the Licensor or others designated to receive attribution as
provided in Section 3(a)(1)(A)(i).
b. Other rights.
1. Moral rights, such as the right of integrity, are not
licensed under this Public License, nor are publicity,
privacy, and/or other similar personality rights; however, to
the extent possible, the Licensor waives and/or agrees not to
assert any such rights held by the Licensor to the limited
extent necessary to allow You to exercise the Licensed
Rights, but not otherwise.
2. Patent and trademark rights are not licensed under this
Public License.
3. To the extent possible, the Licensor waives any right to
collect royalties from You for the exercise of the Licensed
Rights, whether directly or through a collecting society
under any voluntary or waivable statutory or compulsory
licensing scheme. In all other cases the Licensor expressly
reserves any right to collect such royalties, including when
the Licensed Material is used other than for NonCommercial
purposes.
Section 3 -- License Conditions.
Your exercise of the Licensed Rights is expressly made subject to the
following conditions.
a. Attribution.
1. If You Share the Licensed Material (including in modified
form), You must:
a. retain the following if it is supplied by the Licensor
with the Licensed Material:
i. identification of the creator(s) of the Licensed
Material and any others designated to receive
attribution, in any reasonable manner requested by
the Licensor (including by pseudonym if
designated);
ii. a copyright notice;
iii. a notice that refers to this Public License;
iv. a notice that refers to the disclaimer of
warranties;
v. a URI or hyperlink to the Licensed Material to the
extent reasonably practicable;
b. indicate if You modified the Licensed Material and
retain an indication of any previous modifications; and
c. indicate the Licensed Material is licensed under this
Public License, and include the text of, or the URI or
hyperlink to, this Public License.
2. You may satisfy the conditions in Section 3(a)(1) in any
reasonable manner based on the medium, means, and context in
which You Share the Licensed Material. For example, it may be
reasonable to satisfy the conditions by providing a URI or
hyperlink to a resource that includes the required
information.
3. If requested by the Licensor, You must remove any of the
information required by Section 3(a)(1)(A) to the extent
reasonably practicable.
4. If You Share Adapted Material You produce, the Adapter's
License You apply must not prevent recipients of the Adapted
Material from complying with this Public License.
Section 4 -- Sui Generis Database Rights.
Where the Licensed Rights include Sui Generis Database Rights that
apply to Your use of the Licensed Material:
a. for the avoidance of doubt, Section 2(a)(1) grants You the right
to extract, reuse, reproduce, and Share all or a substantial
portion of the contents of the database for NonCommercial purposes
only;
b. if You include all or a substantial portion of the database
contents in a database in which You have Sui Generis Database
Rights, then the database in which You have Sui Generis Database
Rights (but not its individual contents) is Adapted Material; and
c. You must comply with the conditions in Section 3(a) if You Share
all or a substantial portion of the contents of the database.
For the avoidance of doubt, this Section 4 supplements and does not
replace Your obligations under this Public License where the Licensed
Rights include other Copyright and Similar Rights.
Section 5 -- Disclaimer of Warranties and Limitation of Liability.
a. UNLESS OTHERWISE SEPARATELY UNDERTAKEN BY THE LICENSOR, TO THE
EXTENT POSSIBLE, THE LICENSOR OFFERS THE LICENSED MATERIAL AS-IS
AND AS-AVAILABLE, AND MAKES NO REPRESENTATIONS OR WARRANTIES OF
ANY KIND CONCERNING THE LICENSED MATERIAL, WHETHER EXPRESS,
IMPLIED, STATUTORY, OR OTHER. THIS INCLUDES, WITHOUT LIMITATION,
WARRANTIES OF TITLE, MERCHANTABILITY, FITNESS FOR A PARTICULAR
PURPOSE, NON-INFRINGEMENT, ABSENCE OF LATENT OR OTHER DEFECTS,
ACCURACY, OR THE PRESENCE OR ABSENCE OF ERRORS, WHETHER OR NOT
KNOWN OR DISCOVERABLE. WHERE DISCLAIMERS OF WARRANTIES ARE NOT
ALLOWED IN FULL OR IN PART, THIS DISCLAIMER MAY NOT APPLY TO YOU.
b. TO THE EXTENT POSSIBLE, IN NO EVENT WILL THE LICENSOR BE LIABLE
TO YOU ON ANY LEGAL THEORY (INCLUDING, WITHOUT LIMITATION,
NEGLIGENCE) OR OTHERWISE FOR ANY DIRECT, SPECIAL, INDIRECT,
INCIDENTAL, CONSEQUENTIAL, PUNITIVE, EXEMPLARY, OR OTHER LOSSES,
COSTS, EXPENSES, OR DAMAGES ARISING OUT OF THIS PUBLIC LICENSE OR
USE OF THE LICENSED MATERIAL, EVEN IF THE LICENSOR HAS BEEN
ADVISED OF THE POSSIBILITY OF SUCH LOSSES, COSTS, EXPENSES, OR
DAMAGES. WHERE A LIMITATION OF LIABILITY IS NOT ALLOWED IN FULL OR
IN PART, THIS LIMITATION MAY NOT APPLY TO YOU.
c. The disclaimer of warranties and limitation of liability provided
above shall be interpreted in a manner that, to the extent
possible, most closely approximates an absolute disclaimer and
waiver of all liability.
Section 6 -- Term and Termination.
a. This Public License applies for the term of the Copyright and
Similar Rights licensed here. However, if You fail to comply with
this Public License, then Your rights under this Public License
terminate automatically.
b. Where Your right to use the Licensed Material has terminated under
Section 6(a), it reinstates:
1. automatically as of the date the violation is cured, provided
it is cured within 30 days of Your discovery of the
violation; or
2. upon express reinstatement by the Licensor.
For the avoidance of doubt, this Section 6(b) does not affect any
right the Licensor may have to seek remedies for Your violations
of this Public License.
c. For the avoidance of doubt, the Licensor may also offer the
Licensed Material under separate terms or conditions or stop
distributing the Licensed Material at any time; however, doing so
will not terminate this Public License.
d. Sections 1, 5, 6, 7, and 8 survive termination of this Public
License.
Section 7 -- Other Terms and Conditions.
a. The Licensor shall not be bound by any additional or different
terms or conditions communicated by You unless expressly agreed.
b. Any arrangements, understandings, or agreements regarding the
Licensed Material not stated herein are separate from and
independent of the terms and conditions of this Public License.
Section 8 -- Interpretation.
a. For the avoidance of doubt, this Public License does not, and
shall not be interpreted to, reduce, limit, restrict, or impose
conditions on any use of the Licensed Material that could lawfully
be made without permission under this Public License.
b. To the extent possible, if any provision of this Public License is
deemed unenforceable, it shall be automatically reformed to the
minimum extent necessary to make it enforceable. If the provision
cannot be reformed, it shall be severed from this Public License
without affecting the enforceability of the remaining terms and
conditions.
c. No term or condition of this Public License will be waived and no
failure to comply consented to unless expressly agreed to by the
Licensor.
d. Nothing in this Public License constitutes or may be interpreted
as a limitation upon, or waiver of, any privileges and immunities
that apply to the Licensor or You, including from the legal
processes of any jurisdiction or authority.
=======================================================================
Creative Commons is not a party to its public
licenses. Notwithstanding, Creative Commons may elect to apply one of
its public licenses to material it publishes and in those instances
will be considered the “Licensor.” The text of the Creative Commons
public licenses is dedicated to the public domain under the CC0 Public
Domain Dedication. Except for the limited purpose of indicating that
material is shared under a Creative Commons public license or as
otherwise permitted by the Creative Commons policies published at
creativecommons.org/policies, Creative Commons does not authorize the
use of the trademark "Creative Commons" or any other trademark or logo
of Creative Commons without its prior written consent including,
without limitation, in connection with any unauthorized modifications
to any of its public licenses or any other arrangements,
understandings, or agreements concerning use of licensed material. For
the avoidance of doubt, this paragraph does not form part of the
public licenses.
Creative Commons may be contacted at creativecommons.org.
================================================
FILE: MANIFEST.in
================================================
include nsynth/examples.json.gz
================================================
FILE: README.md
================================================
# SING: Symbol-to-Instrument Neural Generator
SING is a deep learning based music notes synthetizer that can be trained on the
[NSynth dataset][nsynth].
Despite being 32 times faster to train and 2,500 faster for inference,
SING produces audio with significantly improved perceptual quality compared to
the NSynth wavenet-like autoencoder [[1]](#ref_nsynth) as measured by
Mean Opinion Scores based on human evaluations.
The architecture and results obtained are detailed in our paper
[SING: Symbol-to-Instrument Neural Generator][sing_nips].
SING is based on a LSTM based sequence generator and a
convolutional decoder:
<p align="center">
<img src="./images/sing.png" alt="Schema representing the structure of SING. A LSTM is followed by a convolutional decoder" width="700px"></p>
## Requirements
SING works with python3.6 and newest.
To use SING, you must have decently recent version of the following
package installed:
- numpy
- requests
- pytorch (needs to be >= 4.1.0 as we use torch.stft)
- scipy
- tqdm
If you have anaconda installed, you can run from the root of this repository:
conda env update
conda activate sing
This will create a `sing` environmnent with all the dependencies installed.
Alternatively, you can use pip to install those:
pip3 install -r requirements.txt
SING can optionally be installed using the usual `setup.py`
although this is not required.
### Obtaining the NSynth dataset
If you want to train SING from scratch, you will need a copy of the NSynth
dataset [[1]](#ref_nsynth). To download it, you use the following instructions
(**WARNING**, NSynth is 30GB so this will take a bit of time):
mkdir data && cd data &&\
wget http://download.magenta.tensorflow.org/datasets/nsynth/nsynth-train.jsonwav.tar.gz &&\
tar xf nsynth-train.jsonwav.tar.gz
## Using SING
Once installed or from the root of this repository, you can use a family
of commands detailed hereafter of the form
python3 -m sing.*
### Common flags
For either training or generation, use the `--cuda` flag for GPU acceleration
and `--parallel` flag to use all available GPUs. Depending on the memory
and number of GPUs available, consider tweaking the batch size using the
`--batch-size` flag. The default is 64 but 256 was used in the paper.
### Training
If you already have the NSynth dataset downloaded somewhere, run
python3 -m sing.train [--cuda [--parallel]] --data PATH_TO_NSYNTH \
--output PATH_TO_SING_MODEL [--checkpoint PATH_TO_CHECKPOINTS]
`PATH_TO_NSYNTH` is by default set to `data/nsynth-train`.
The final model will be saved at `PATH_TO_SING_MODEL` (default is `models/sing.th`). If you want
to save checkpoints after each epoch, or to resume a previously interrupted
training, use the `--checkpoint` option.
### Generation
For generation, you do not need the NSynth dataset but you should have a trained SING model.
python3 -m sing.generate [--cuda [--parallel]] \
--model PATH_TO_SING_MODEL PATH_TO_ITEM_LIST
`PATH_TO_ITEM_LIST` should be a file with one dataset item name per list,
for instance `organ_electronic_044-055-127`.
Alternatively, you can download a pretrained model using
python3 -m sing.generate [--cuda [--parallel]] --dl PATH_TO_ITEM_LIST
By default, the model will be downloaded under `models/sing.th` but a
different path can be provided using the `--model` option.
The pretrained model can be directly download [here](https://dl.fbaipublicfiles.com/sing/sing.th).
### Results reproduction
To reproduce the results of Table 1 in our paper, simply run
```bash
# For the L1 spectral losss
python3 -m sing.train [--cuda [--parallel]] --l1
# For the L1 spectral loss without time embeddings
python3 -m sing.train [--cuda [--parallel]] --l1 --time-dim=0
# For the Wav loss
python3 -m sing.train [--cuda [--parallel]] --wav
```
To reproduce the audio samples used for the human evaluations, simply run
from the root of the git repository
python3 -m sing.generate [--cuda [--parallel]] --dl nsynth_100_test.txt
The file `nsynth_100_test.txt` has been generated using the following code:
```python
from sing import nsynth
from sing.fondation.datasets import RandomSubset
dset = nsynth.get_nsynth_metadata()
train, valid, test = nsynth.make_datasets(dset)
evaluation = RandomSubset(test, 100)
open("nsynth_100_test.txt", "w").write("\n".join(
evaluation[i].metadata['name'] for i in range(len(evaluation))))
```
## Generated audio
A comparison of audio samples generated by SING and the NSynth Wavenet based autoencoder [[1]](#ref_nsynth)
is available on [the paper webpage](https://research.fb.com/wp-content/themes/fb-research/research/sing-paper/).
## Thanks
We thank the Magenta team for their inspiring work on NSynth.
## License
For conveniance we have included a copy of the metadata of the NSynth dataset
in this repository. The dataset has been released by Google Inc
under the Creative Commons Attribution 4.0 International (CC BY 4.0) license.
SING is released under Creative Commons Attribution 4.0 International
(CC BY 4.0) license, as found in the LICENSE file.
## Bibliography
<a name="ref_nsynth"></a>[1]: Jesse Engel, Cinjon Resnick, Adam Roberts,
Sander Dieleman, Douglas Eck,
Karen Simonyan, and Mohammad Norouzi. [Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders](https://arxiv.org/pdf/1704.01279.pdf). 2017.
[nsynth]: https://magenta.tensorflow.org/datasets/nsynth
[sing_nips]: https://research.fb.com/publications/sing-symbol-to-instrument-neural-generator
================================================
FILE: environment.yml
================================================
name: sing
channels:
- pytorch
dependencies:
- numpy>=1.15
- python>=3.6
- pytorch>=0.4.1
- requests>=2.19
- scipy>=1.1
- tqdm>=4.26
================================================
FILE: nsynth_100_test.txt
================================================
bass_synthetic_126-025-025
synth_lead_synthetic_006-045-050
bass_synthetic_065-070-025
keyboard_electronic_063-087-100
reed_acoustic_031-057-050
bass_synthetic_044-094-025
bass_synthetic_095-091-050
bass_synthetic_087-091-127
mallet_electronic_011-074-127
mallet_electronic_007-059-100
brass_acoustic_049-082-050
keyboard_electronic_000-091-075
string_acoustic_048-062-075
string_acoustic_008-033-075
keyboard_electronic_026-055-127
keyboard_electronic_070-049-025
organ_electronic_050-105-127
string_acoustic_044-060-025
bass_synthetic_064-092-075
organ_electronic_085-083-127
mallet_acoustic_013-071-075
keyboard_electronic_026-098-100
mallet_acoustic_002-060-127
keyboard_electronic_100-039-100
bass_synthetic_120-071-100
organ_electronic_061-044-127
vocal_acoustic_023-057-127
string_acoustic_030-051-025
brass_acoustic_003-041-050
bass_synthetic_093-052-075
organ_electronic_016-026-100
organ_electronic_080-025-100
brass_acoustic_001-054-127
guitar_acoustic_009-069-100
brass_acoustic_014-061-050
keyboard_acoustic_001-061-100
organ_electronic_077-047-100
bass_synthetic_130-056-075
guitar_electronic_032-081-127
mallet_synthetic_001-066-100
keyboard_electronic_042-057-050
bass_synthetic_052-059-025
keyboard_acoustic_002-062-075
guitar_acoustic_019-074-025
bass_synthetic_128-097-100
guitar_electronic_004-038-050
bass_synthetic_086-048-100
keyboard_electronic_055-068-127
guitar_electronic_023-046-100
guitar_electronic_015-065-025
bass_synthetic_015-098-127
brass_acoustic_035-058-127
bass_synthetic_063-036-025
reed_acoustic_033-044-050
organ_electronic_065-041-100
bass_synthetic_111-036-100
organ_electronic_037-047-050
bass_synthetic_140-108-050
brass_acoustic_040-059-025
organ_electronic_092-042-025
keyboard_electronic_019-099-025
reed_acoustic_029-077-127
string_acoustic_043-027-025
bass_synthetic_121-043-127
string_acoustic_007-060-025
keyboard_electronic_015-089-050
organ_electronic_050-057-025
bass_synthetic_078-045-075
keyboard_electronic_088-081-075
brass_acoustic_000-039-127
guitar_electronic_017-093-075
bass_synthetic_117-068-127
mallet_electronic_013-087-050
flute_acoustic_012-090-127
bass_synthetic_136-073-025
mallet_electronic_006-085-127
mallet_acoustic_016-082-050
organ_electronic_044-025-025
mallet_acoustic_046-032-127
guitar_acoustic_032-067-100
organ_electronic_014-084-127
organ_electronic_098-030-025
mallet_acoustic_042-103-050
keyboard_electronic_065-093-127
mallet_acoustic_058-108-050
mallet_electronic_007-036-127
keyboard_electronic_092-090-127
string_acoustic_059-070-025
guitar_electronic_009-054-100
bass_synthetic_044-075-050
mallet_acoustic_051-065-075
bass_electronic_030-047-100
flute_acoustic_027-077-050
bass_synthetic_123-086-050
bass_synthetic_117-086-100
mallet_acoustic_004-065-100
bass_synthetic_094-023-025
organ_electronic_020-054-025
brass_acoustic_011-056-050
keyboard_electronic_058-038-127
================================================
FILE: requirements.txt
================================================
numpy
requests
scipy
torch>=0.4.1
tqdm
================================================
FILE: setup.py
================================================
#!/usr/bin/env python
# -*- coding: utf-8 -*-
# Copyright (c) 2018-present, Facebook, Inc.
# All rights reserved.
#
# This source code is licensed under the license found in the
# LICENSE file in the root directory of this source tree.
# Inspired from https://github.com/kennethreitz/setup.py
from pathlib import Path
from setuptools import find_packages, setup
NAME = 'sing'
DESCRIPTION = 'SING: Symbol-to-Instrument Neural Generator'
URL = 'https://github.com/facebookresearch/SING'
EMAIL = 'defossez@fb.com'
AUTHOR = 'Alexandre Defossez'
REQUIRES_PYTHON = '>=3.6.0'
VERSION = "1.0"
HERE = Path(__file__).parent
REQUIRED = [i.strip() for i in open(HERE / "requirements.txt").readlines()]
try:
with open(HERE / "README.md", encoding='utf-8') as f:
long_description = '\n' + f.read()
except FileNotFoundError:
long_description = DESCRIPTION
setup(
name=NAME,
version=VERSION,
description=DESCRIPTION,
long_description=long_description,
long_description_content_type='text/markdown',
author=AUTHOR,
author_email=EMAIL,
python_requires=REQUIRES_PYTHON,
url=URL,
packages=find_packages(),
install_requires=REQUIRED,
include_package_data=True,
license='Creative Common Attribution-NonCommercial 4.0 International',
classifiers=[
# Trove classifiers
# Full list: https://pypi.python.org/pypi?%3Aaction=list_classifiers
'Programming Language :: Python',
'Programming Language :: Python :: 3',
'Programming Language :: Python :: 3.6',
'Programming Language :: Python :: Implementation :: CPython',
'Topic :: Scientific/Engineering :: Artificial Intelligence',
],
)
================================================
FILE: sing/__init__.py
================================================
# -*- coding: utf-8 -*-
# Copyright (c) 2018-present, Facebook, Inc.
# All rights reserved.
#
# This source code is licensed under the license found in the
# LICENSE file in the root directory of this source tree.
#
================================================
FILE: sing/ae/__init__.py
================================================
# -*- coding: utf-8 -*-
# Copyright (c) 2018-present, Facebook, Inc.
# All rights reserved.
#
# This source code is licensed under the license found in the
# LICENSE file in the root directory of this source tree.
#
================================================
FILE: sing/ae/models.py
================================================
# -*- coding: utf-8 -*-
# Copyright (c) 2018-present, Facebook, Inc.
# All rights reserved.
#
# This source code is licensed under the license found in the
# LICENSE file in the root directory of this source tree.
#
from torch import nn
from .utils import WindowedConv1d, WindowedConvTranpose1d
class ConvolutionalDecoder(nn.Module):
"""
Convolutional decoder that takes a downsampled embedding and turns it
into a waveform.
Together with :class:`ConvolutionalEncoder`, it forms a
:class:`ConvolutionalAE`
Arguments:
channels (int): number of channels accross all the inner layers
stride (int): stride of the final :class:`nn.ConvTranspose1d`
dimension (int): dimension of the embedding
kernel_size (int): size of the kernel of the final
:class:`nn.ConvTranspose1d`
context_size (int): kernel size of the first convolution,
this is called a context as one can see it as providing
information about the previous and following embeddings
rewrite_layers (int): after the first convolution, perform
`rewrite_layers` `1x1` convolutions
window_name (str or None): name of the window used to smooth
the convolutions. See :func:`sing.dsp.get_window`
squared_window (bool): if `True`, square the smoothing window
"""
def __init__(self,
channels=4096,
stride=256,
dimension=128,
kernel_size=1024,
context_size=9,
rewrite_layers=2,
window_name="hann",
squared_window=True):
super(ConvolutionalDecoder, self).__init__()
layers = []
layers.extend([
nn.Conv1d(
in_channels=dimension,
out_channels=channels,
kernel_size=context_size),
nn.ReLU()
])
for rewrite in range(rewrite_layers):
layers.extend([
nn.Conv1d(
in_channels=channels, out_channels=channels,
kernel_size=1),
nn.ReLU()
])
conv_tr = nn.ConvTranspose1d(
in_channels=channels,
out_channels=1,
kernel_size=kernel_size,
stride=stride,
padding=kernel_size - stride)
if window_name is not None:
conv_tr = WindowedConvTranpose1d(conv_tr, window_name,
squared_window)
layers.append(conv_tr)
self.layers = nn.Sequential(*layers)
self.context_size = context_size
self.stride = stride
self.kernel_size = kernel_size
self.strip = kernel_size - stride + (context_size - 1) * stride // 2
def __repr__(self):
return "ConvolutionalDecoder({})".format(repr(self.layers))
def forward(self, embeddings):
return self.layers.forward(embeddings).squeeze(1)
def wav_length(self, embedding_length):
"""
Given an embedding of a certain size `embedding_length`,
returns the length of the wav that would be generated from it.
"""
return (embedding_length - self.context_size + 2
) * self.stride - self.kernel_size
def embedding_length(self, wav_length):
"""
Return the embedding length necessary to generate a wav of length
`wav_length`.
"""
return self.context_size - 2 + (
wav_length + self.kernel_size) // self.stride
class ConvolutionalEncoder(nn.Module):
"""
Convolutional encoder that takes a waveform and turns it
into a downsampled embedding.
Together with :class:`ConvolutionalDecoder`, it forms a
:class:`ConvolutionalAE`
Arguments:
channels (int): number of channels accross all the inner layers
stride (int): stride of the initial :class:`nn.Conv1d`
dimension (int): dimension of the embedding
kernel_size (int): size of the kernel of the initial
:class:`nn.Conv1d`
rewrite_layers (int): after the first convolution, perform
`rewrite_layers` `1x1` convolutions.
window_name (str or None): name of the window used to smooth
the convolutions. See :func:`sing.dsp.get_window`
squared_window (bool): if `True`, square the smoothing window
"""
def __init__(self,
channels=4096,
stride=256,
dimension=128,
kernel_size=1024,
rewrite_layers=2,
window_name="hann",
squared_window=True):
super(ConvolutionalEncoder, self).__init__()
layers = []
conv = nn.Conv1d(
in_channels=1,
out_channels=channels,
kernel_size=kernel_size,
stride=stride)
if window_name is not None:
conv = WindowedConv1d(conv, window_name, squared_window)
layers.extend([conv, nn.ReLU()])
for rewrite in range(rewrite_layers):
layers.extend([
nn.Conv1d(
in_channels=channels, out_channels=channels,
kernel_size=1),
nn.ReLU()
])
layers.append(
nn.Conv1d(
in_channels=channels, out_channels=dimension, kernel_size=1))
self.layers = nn.Sequential(*layers)
def __repr__(self):
return "ConvolutionalEncoder({!r})".format(self.layers)
def forward(self, signal):
return self.layers.forward(signal.unsqueeze(1))
class ConvolutionalAE(nn.Module):
"""
Convolutional autoencoder made from :class:`ConvolutionalEncoder` and
:class:`ConvolutionalDecoder`.
Arguments:
channels (int): number of channels accross all the inner layers
stride (int): downsampling stride going from the waveform
to the embedding
dimension (int): dimension of the embedding
kernel_size (int): kernel size of the initial convolution
and last conv transpose
context_size (int): kernel size of the first
convolution of the decoder
rewrite_layers (int): after the first convolution, perform
`rewrite_layers` `1x1` convolutions, both in the encoder
and decoder.
window_name (str or None): name of the window used to smooth
the convolutions. See :func:`sing.dsp.get_window`
squared_window (bool): if `True`, square the smoothing window
"""
def __init__(self,
channels=4096,
stride=256,
dimension=128,
kernel_size=1024,
context_size=9,
rewrite_layers=2,
window_name="hann",
squared_window=True):
super(ConvolutionalAE, self).__init__()
self.encoder = ConvolutionalEncoder(
channels=channels,
stride=stride,
dimension=dimension,
kernel_size=kernel_size,
rewrite_layers=rewrite_layers,
window_name=window_name,
squared_window=squared_window)
self.decoder = ConvolutionalDecoder(
channels=channels,
stride=stride,
dimension=dimension,
kernel_size=kernel_size,
context_size=context_size,
rewrite_layers=rewrite_layers,
window_name=window_name,
squared_window=squared_window)
print(self)
def encode(self, signal):
"""
Returns the embedding for the waveform `signal`.
"""
return self.encoder.forward(signal)
def decode(self, embeddings):
"""
Return the waveforms from `embeddings`
"""
return self.decoder.forward(embeddings)
def forward(self, signal):
return self.decode(self.encode(signal))
def __repr__(self):
return "ConvolutionalAE(encoder={!r},decoder={!r})".format(
self.encoder, self.decoder)
================================================
FILE: sing/ae/trainer.py
================================================
# -*- coding: utf-8 -*-
# Copyright (c) 2018-present, Facebook, Inc.
# All rights reserved.
#
# This source code is licensed under the license found in the
# LICENSE file in the root directory of this source tree.
#
from ..fondation import utils, trainer
class AutoencoderTrainer(trainer.BaseTrainer):
"""
Trainer for the autoencoder.
"""
def _train_batch(self, batch):
rebuilt, target = self._get_rebuilt_target(batch)
self.optimizer.zero_grad()
loss = self.train_loss(rebuilt, target)
loss.backward()
self.optimizer.step()
return loss.item()
def _get_rebuilt_target(self, batch):
wav = batch.tensors['wav']
target = utils.unpad1d(wav, self.model.decoder.strip)
rebuilt = self.parallel.forward(wav)
return rebuilt, target
================================================
FILE: sing/ae/utils.py
================================================
# -*- coding: utf-8 -*-
# Copyright (c) 2018-present, Facebook, Inc.
# All rights reserved.
#
# This source code is licensed under the license found in the
# LICENSE file in the root directory of this source tree.
#
from torch import nn
from torch.nn import functional as F
from .. import dsp
class WindowedConv1d(nn.Module):
"""
Smooth a convolution using a window.
Arguments:
conv (nn.Conv1d): convolution module to wrap
window_name (str or None): name of the window used to smooth
the convolutions. See :func:`sing.dsp.get_window`
squared (bool): if `True`, square the smoothing window
"""
def __init__(self, conv, window_name='hann', squared=True):
super(WindowedConv1d, self).__init__()
self.window_name = window_name
if squared:
self.window_name += "**2"
self.register_buffer('window',
dsp.get_window(
window_name,
conv.weight.size(-1),
squared=squared))
self.conv = conv
def forward(self, input):
weight = self.window * self.conv.weight
return F.conv1d(
input,
weight,
bias=self.conv.bias,
stride=self.conv.stride,
dilation=self.conv.dilation,
groups=self.conv.groups,
padding=self.conv.padding)
def __repr__(self):
return "WindowedConv1d(window={},conv={})".format(
self.window_name, self.conv)
class WindowedConvTranpose1d(nn.Module):
"""
Smooth a transposed convolution using a window.
Arguments:
conv (nn.Conv1d): convolution module to wrap
window_name (str or None): name of the window used to smooth
the convolutions. See :func:`sing.dsp.get_window`
squared (bool): if `True`, square the smoothing window
"""
def __init__(self, conv_tr, window_name='hann', squared=True):
super(WindowedConvTranpose1d, self).__init__()
self.window_name = window_name
if squared:
self.window_name += "**2"
self.register_buffer('window',
dsp.get_window(
window_name,
conv_tr.weight.size(-1),
squared=squared))
self.conv_tr = conv_tr
def forward(self, input):
weight = self.window * self.conv_tr.weight
return F.conv_transpose1d(
input,
weight,
bias=self.conv_tr.bias,
stride=self.conv_tr.stride,
padding=self.conv_tr.padding,
output_padding=self.conv_tr.output_padding,
groups=self.conv_tr.groups,
dilation=self.conv_tr.dilation)
def __repr__(self):
return "WindowedConvTranpose1d(window={},conv_tr={})".format(
self.window_name, self.conv_tr)
================================================
FILE: sing/dsp.py
================================================
# -*- coding: utf-8 -*-
# Copyright (c) 2018-present, Facebook, Inc.
# All rights reserved.
#
# This source code is licensed under the license found in the
# LICENSE file in the root directory of this source tree.
#
import torch
from torch import nn
import torch.nn.functional as F
def power(spec):
"""
Given a complex spectrogram, return the power spectrum.
Shape:
- `spec`: `(*, 2, F, T)`
- Output: `(*, F, T)`
"""
return spec[..., 0]**2 + spec[..., 1]**2
def get_window(name, window_length, squared=False):
"""
Returns a windowing function.
Arguments:
window (str): name of the window, currently only 'hann' is available
window_length (int): length of the window
squared (bool): if true, square the window
Returns:
torch.FloatTensor: window of size `window_length`
"""
if name == "hann":
window = torch.hann_window(window_length)
else:
raise ValueError("Invalid window name {}".format(name))
if squared:
window *= window
return window
class STFT(nn.Module):
"""
Compute the STFT.
See :mod:`torch.stft` for a definition of the parameters.
Arguments:
n_fft (int): performs a FFT over `n_fft` samples
hop_length (int or None): stride of the STFT transform. If `None`
uses `n_fft // 4`
window_name (str or None): name of the window used for the STFT.
No window is used if `None`.
"""
def __init__(self, n_fft=1024, hop_length=None, window_name='hann'):
super(STFT, self).__init__()
assert n_fft % 2 == 0
window = None
if window_name is not None:
window = get_window(window_name, n_fft)
self.register_buffer("window", window)
self.hop_length = hop_length or n_fft // 4
self.n_fft = n_fft
def forward(self, input):
return torch.stft(
input,
window=self.window,
n_fft=self.n_fft,
hop_length=self.hop_length,
center=False)
class SpectralLoss(nn.Module):
"""
Compute a loss between two log power-spectrograms.
Arguments:
base_loss (function): loss used to compare the log power-spectrograms.
For instance :func:`F.mse_loss`
epsilon (float): offset for the log, i.e. `log(epsilon + ...)`
**kwargs (dict): see :class:`STFT`
"""
def __init__(self, base_loss=F.mse_loss, epsilon=1, **kwargs):
super(SpectralLoss, self).__init__()
self.base_loss = base_loss
self.epsilon = epsilon
self.stft = STFT(**kwargs)
def _log_spectrogram(self, signal):
return torch.log(self.epsilon + power(self.stft.forward(signal)))
def forward(self, a, b):
spec_a = self._log_spectrogram(a)
spec_b = self._log_spectrogram(b)
return self.base_loss(spec_a, spec_b)
def float_wav_to_short(wav):
"""
Given a float waveform, return a short waveform.
The input waveform will be clamped between -1 and 1.
"""
return (wav.clamp(-1, 1) * (2**15 - 1)).short()
================================================
FILE: sing/fondation/__init__.py
================================================
# -*- coding: utf-8 -*-
# Copyright (c) 2018-present, Facebook, Inc.
# All rights reserved.
#
# This source code is licensed under the license found in the
# LICENSE file in the root directory of this source tree.
#
================================================
FILE: sing/fondation/batch.py
================================================
# -*- coding: utf-8 -*-
# Copyright (c) 2018-present, Facebook, Inc.
# All rights reserved.
#
# This source code is licensed under the license found in the
# LICENSE file in the root directory of this source tree.
#
from torch.utils.data.dataloader import default_collate
class BatchItem:
"""
Reprensents a single batch item. A :class:`Batch` can be built
from multiple :class:`BatchItem` using :func:`collate`.
Attributes:
metadata (dict[str, object]): Contains all the metadata
about the batch item. Those elements will not be
collated together
when building a batch
tensors (dict[str, tensor]): Contains all the tensors
for the batch item. Those elements will be collated
together when building a batch using :func:`default_collate`.
"""
def __init__(self, metadata=None, tensors=None):
self.metadata = dict(metadata) if metadata else {}
self.tensors = dict(tensors) if tensors else {}
def collate(items):
"""
Collate together all the items into a :class:`Batch`.
The metadata dictionaries will be added to a list
and the tensors will be collated using
:func:`torch.utils.data.dataloader.default_collate`.
Args:
items (list[BatchItem]): list of the items in the batch
Returns:
Batch: a batch made from `items`.
"""
metadata = [item.metadata for item in items]
tensors = default_collate([item.tensors for item in items])
return Batch(metadata=metadata, tensors=tensors)
class Batch:
"""
Represents a batch. Supports iteration
(yields individual :class:`BatchItem`) and indexing. Slice
indexing will return another :class:`Batch`.
Attributes:
metadata (list[dict[str, object]]): a list of dictionaries
for each element in the batch.
Each dictionary contains information
about the corresponding item.
tensors (dict[str, tensor]): a dictionary of collated tensors.
The first dimension of each tensor will always be `B`,
the batch size.
"""
def __init__(self, metadata, tensors):
self.metadata = metadata
self.tensors = tensors
def __len__(self):
return len(self.metadata)
def __iter__(self):
for i in range(len(self)):
yield self[i]
def __getitem__(self, index):
if isinstance(index, slice):
if index.step is not None:
raise IndexError("Does not support slice with step")
metadata = self.metadata[index]
tensors = {
name: tensor[index]
for name, tensor in self.tensors.items()
}
return Batch(metadata=metadata, tensors=tensors)
else:
return BatchItem(
metadata=self.metadata[index],
tensors={
name: tensor[index]
for name, tensor in self.tensors.items()
})
def apply(self, function):
"""
Apply a function to all tensors.
Arguments:
function: callable to be applied to all tensors.
Returns:
Batch: A new batch
"""
tensors = {
name: function(tensor)
for name, tensor in self.tensors.items()
}
return Batch(metadata=self.metadata, tensors=tensors)
def apply_(self, function):
"""
Inplace variance of :meth:`apply`.
"""
other = self.apply(function)
self.tensors = other.tensors
return self
def cuda(self, *args, **kwargs):
"""
Returns a new batch on GPU.
"""
return self.apply(lambda x: x.cuda())
def cuda_(self, *args, **kwargs):
"""
Move the batch inplace to GPU.
"""
return self.apply_(lambda x: x.cuda())
def cpu(self, *args, **kwargs):
"""
Returns a new batch on CPU.
"""
return self.apply(lambda x: x.cpu())
def cpu_(self, *args, **kwargs):
"""
Move the batch inplace to CPU.
"""
return self.apply_(lambda x: x.cpu_())
================================================
FILE: sing/fondation/datasets.py
================================================
# -*- coding: utf-8 -*-
# Copyright (c) 2018-present, Facebook, Inc.
# All rights reserved.
#
# This source code is licensed under the license found in the
# LICENSE file in the root directory of this source tree.
#
import random
import torch
from .utils import random_seed_manager
class DatasetSubset:
"""
Represents a subset of a dataset.
Arguments:
dataset (Dataset): dataset to take a subset of.
indexes (list[int]): list of indexes to keep.
"""
def __init__(self, dataset, indexes):
self.dataset = dataset
self.indexes = torch.LongTensor(indexes)
def __len__(self):
return len(self.indexes)
def __getitem__(self, index):
return self.dataset[self.indexes[index]]
class RandomSubset(DatasetSubset):
"""
A random subset of a given size built from another dataset.
Arguments:
dataset (Dataset): dataset to take a random subset of.
size (int): size of the random subset
random_seed (int): random seed used to select the indexes.
"""
def __init__(self, dataset, size, random_seed=42):
indexes = list(range(len(dataset)))
with random_seed_manager(random_seed):
random.shuffle(indexes)
super(RandomSubset, self).__init__(
dataset=dataset, indexes=indexes[:size])
================================================
FILE: sing/fondation/trainer.py
================================================
# -*- coding: utf-8 -*-
# Copyright (c) 2018-present, Facebook, Inc.
# All rights reserved.
#
# This source code is licensed under the license found in the
# LICENSE file in the root directory of this source tree.
#
import torch
from torch import nn
from torch import optim
from torch.utils.data import DataLoader
import tqdm
from . import utils
from .batch import collate
class BaseTrainer:
"""
Base class for all the epoch-based trainers. Takes care of various
task common to all training like checkpointing,
iterating over the different datasets, computing evaluation metrics etc.
Arguments:
model (nn.Module): model to train
train_loss (nn.Module): loss used for training
eval_losses (dict[str, nn.Module]): dictionary of evaluation losses
train_dataset (Dataset): dataset used for training
eval_datasets (dict[str, Dataset]): dictionary of datasets
on which each evaluation loss will be computed
epochs (int): number of epochs to train for
suffix (str): suffix used for logging, for instance
if the suffix is `"_phase1"`, during training, `"train_phase1"`
will be displayed
batch_size (int): batch size
cuda (bool): if true, runs on GPU
parallel (bool): if true, use all available GPUs
lr (float): learning rate for :class:`optim.Adam`
checkpoint_path (Path): path to save checkpoint to. If `None`, no
checkpointing is performed. Otherwise, a checkpoint is saved
at the end of each epoch and overwrites the previous one.
"""
def __init__(self,
model,
train_loss,
eval_losses,
train_dataset,
eval_datasets,
epochs,
suffix="",
batch_size=32,
cuda=True,
parallel=False,
lr=0.0001,
checkpoint_path=None):
self.model = model
self.parallel = nn.DataParallel(model) if parallel else model
self.is_parallel = parallel
self.train_loss = train_loss
self.eval_losses = nn.ModuleDict(eval_losses)
self.batch_size = batch_size
self.cuda = cuda
self.suffix = suffix
self.train_dataset = train_dataset
self.eval_datasets = eval_datasets
self.epochs = epochs
self.checkpoint_path = checkpoint_path
parameters = [p for p in self.model.parameters() if p.requires_grad]
self.optimizer = optim.Adam(parameters, lr=lr)
if self.cuda:
self.model.cuda()
self.train_loss.cuda()
self.eval_losses.cuda()
else:
self.model.cpu()
self.train_loss.cpu()
self.eval_losses.cpu()
def _train_epoch(self, dataset, epoch):
"""
Train a single epoch on the given dataset and displays
statistics from time to time.
"""
loader = DataLoader(
dataset,
batch_size=self.batch_size,
shuffle=True,
collate_fn=collate)
iterator = utils.progress_iterator(loader, divisions=20)
total_loss = 0
with tqdm.tqdm(total=len(dataset), unit="ex") as bar:
for idx, (progress, batch) in enumerate(iterator):
if self.cuda:
batch.cuda_()
total_loss += self._train_batch(batch)
bar.update(len(batch))
if progress:
tqdm.tqdm.write(
"[train{}][{:03d}] {:.1f}%, loss {:.6f}".format(
self.suffix, epoch, progress,
total_loss / (idx + 1)))
return total_loss
def _eval_dataset(self, dataset_name, dataset, epoch):
"""
Evaluate all the losses `eval_lossers` on the given dataset
and reports the metrics averaged over the entire dataset.
"""
loader = DataLoader(
dataset, batch_size=self.batch_size, collate_fn=collate)
total_losses = {loss_name: 0 for loss_name in self.eval_losses}
with tqdm.tqdm(total=len(dataset), unit="ex") as bar:
for batch in loader:
if self.cuda:
batch.cuda_()
rebuilt, target = self._get_rebuilt_target(batch)
for name, loss in self.eval_losses.items():
total_losses[name] += loss(rebuilt,
target).item() * len(batch)
bar.update(len(batch))
print("[{}{}][{:03d}] Evaluation: \n{}\n".format(
dataset_name, self.suffix, epoch, "\n".join(
"\t{}={:.6f}".format(name, loss / len(dataset))
for name, loss in total_losses.items())))
return total_losses
def _train_batch(self, batch):
"""
Given a batch, call :meth:`_get_rebuilt_target`
to obtain the `target` and `rebuilt` tensors and call
:attr:`train_loss` on them, compute the gradient and perform
one optimizer step.
This method can be overriden in subclasses.
"""
rebuilt, target = self._get_rebuilt_target(batch)
self.optimizer.zero_grad()
loss = self.train_loss(rebuilt, target)
loss.backward()
self.optimizer.step()
return loss.item()
def _get_rebuilt_target(self, batch):
"""
Should be implemenented in subclasses.
Given a batch, returns a tuple (rebuilt, target).
This tuple will be passed to all the losses in `eval_losses`.
"""
raise NotImplementedError()
def train(self):
"""
Train :attr:`model` for :attr:`epochs`
"""
last_epoch, state = utils.load_checkpoint(self.checkpoint_path)
if state is not None:
self.model.load_state_dict(state, strict=False)
start_epoch = last_epoch + 1
if start_epoch > self.epochs:
raise ValueError(("Checkpoint has been trained for {} "
"epochs but we aim for {} epochs").format(
start_epoch, self.epochs))
if start_epoch > 0:
print("Resuming training at epoch {}".format(start_epoch))
for epoch in range(start_epoch, self.epochs):
self._train_epoch(self.train_dataset, epoch)
utils.save_checkpoint(self.checkpoint_path, epoch,
self.model.state_dict())
with torch.no_grad():
for name, dataset in self.eval_datasets.items():
self._eval_dataset(name, dataset, epoch)
================================================
FILE: sing/fondation/utils.py
================================================
# -*- coding: utf-8 -*-
# Copyright (c) 2018-present, Facebook, Inc.
# All rights reserved.
#
# This source code is licensed under the license found in the
# LICENSE file in the root directory of this source tree.
#
import contextlib
import hashlib
from pathlib import Path
import random
import requests
import sys
import torch
import tqdm
@contextlib.contextmanager
def random_seed_manager(seed):
"""
Context manager that will save the python RNG state,
set the seed to `seed` and on exit, set back the python RNG state.
"""
state = random.getstate()
try:
random.seed(seed)
yield None
finally:
random.setstate(state)
def progress_iterator(iterator, divisions=100):
"""
Wraps an iterator of known length and yield a tuple `(progress, item)`
for each `item` in `iterator`. `progress` will be None except `divisions`
times that are evenly spaced. When `progress` is not None
it will contain the current percentage of items that have been seen.
Arguments:
iterator (iterator): source iterator, should support :func:`len`.
divisions (int): progress will be every 1/divisions of the
iterator length.
Examples::
>>> for (progress, element) in progress_iterator(range(500)):
... if progress:
... print("{:.0f}% done".format(progress))
"""
length = len(iterator)
division_width = length / divisions
next_division = division_width
for idx, element in enumerate(iterator):
progress = None
if (idx + 1) >= next_division or idx + 1 == length:
next_division += division_width
progress = (idx + 1) / length * 100
yield progress, element
def unpad1d(tensor, pad):
"""
Opposite of padding, will remove `pad` items on each side
of the last dimension of `tensor`.
Arguments:
tensor (tensor): tensor to unpad
pad (int): amount of padding to remove on each side.
"""
if pad > 0:
return tensor[..., pad:-pad]
return tensor
def load_checkpoint(path):
"""
Arguments:
path (str or Path): path to load
Returns:
(int, object): returns a tuple (epoch, state).
"""
if path is None or not Path(path).exists():
return -1, None
return torch.load(path)
def save_checkpoint(path, epoch, state):
"""
Save a new checkpoint. A temporary file is created
and then renamed to the target path.
Arguments:
path (str or Path): path to write to
epoch (int): current epoch
state (object): state to save
"""
if path is None:
return
path = Path(path)
tmp_path = path.parent / (path.name + ".tmp")
torch.save((epoch, state), str(tmp_path))
tmp_path.rename(path)
def download_file(target, url, sha256=None):
"""
Download a file with a progress bar.
Arguments:
target (Path): target path to write to
url (str): url to download
sha256 (str or None): expected sha256 hexdigest of the file
"""
response = requests.get(url, stream=True)
total_length = int(response.headers.get('content-length', 0))
if sha256 is not None:
sha = hashlib.sha256()
update = sha.update
else:
update = lambda x: None
with tqdm.tqdm(total=total_length, unit="B", unit_scale=True) as bar:
with open(target, "wb") as output:
for data in response.iter_content(chunk_size=4096):
output.write(data)
update(data)
bar.update(len(data))
if sha256 is not None:
signature = sha.hexdigest()
if sha256 != signature:
target.unlink()
raise ValueError("Invalid sha256 signature when downloading {}. "
"Expected {} but got {}".format(
url, sha256, signature))
def fatal(message, error_code=1):
"""
Print `message` to stderr and exit with the code `error_code`.
"""
print(message, file=sys.stderr)
sys.exit(1)
================================================
FILE: sing/generate.py
================================================
# -*- coding: utf-8 -*-
# Copyright (c) 2018-present, Facebook, Inc.
# All rights reserved.
#
# This source code is licensed under the license found in the
# LICENSE file in the root directory of this source tree.
#
import argparse
from pathlib import Path
import sys
from scipy.io import wavfile
import torch
from torch import nn
from torch.utils.data import DataLoader
import tqdm
from . import dsp, nsynth
from .fondation.batch import collate
from .fondation.datasets import DatasetSubset
from .fondation import utils
from .sequence.models import download_pretrained_model
def get_parser():
parser = argparse.ArgumentParser(
"sing.generate",
description="Generate audio samples from a trained SING model",
formatter_class=argparse.ArgumentDefaultsHelpFormatter)
parser.add_argument(
"--model",
type=Path,
default="models/sing.th",
help="Path to the trained SING model as outputted by sing.main")
parser.add_argument(
"--dl",
action="store_true",
help="Download if necessary a pretrained SING model.")
parser.add_argument(
"--output",
type=Path,
default="generated",
help="Path where the generated samples will be saved")
parser.add_argument(
"--metadata",
default=nsynth.get_metadata_path(),
type=Path,
help="path to the dataset metadata file")
parser.add_argument(
"list",
type=Path,
help="File containing a list of names from the nsynth dataset. "
"Those notes will be generated by SING")
parser.add_argument(
"--batch-size", type=int, default=32, help="Batch size")
parser.add_argument("--cuda", action="store_true", help="Use cuda")
parser.add_argument(
"--parallel", action="store_true", help="Use multiple gpus")
parser.add_argument(
"--unpad",
default=512,
type=int,
help="Amount of unpadding to perform")
return parser
def main():
args = get_parser().parse_args()
if not args.model.exists():
if args.dl:
print("Downloading pretrained SING model")
args.model.parent.mkdir(parents=True, exist_ok=True)
download_pretrained_model(args.model)
else:
utils.fatal("No model found for path {}. To download "
"a pretrained model, use --dl".format(args.model))
elif args.dl:
print(
"WARNING: --dl is set but {} already exist.".format(args.model),
file=sys.stderr)
model = torch.load(args.model)
if args.cuda:
model.cuda()
if args.parallel:
model = nn.DataParallel(model)
args.output.mkdir(exist_ok=True, parents=True)
dataset = nsynth.NSynthMetadata(args.metadata)
names = [name.strip() for name in open(args.list)]
indexes = [dataset.names.index(name) for name in names]
to_generate = DatasetSubset(dataset, indexes)
loader = DataLoader(
to_generate, batch_size=args.batch_size, collate_fn=collate)
with tqdm.tqdm(total=len(to_generate), unit="ex") as bar:
for batch in loader:
if args.cuda:
batch.cuda_()
with torch.no_grad():
rebuilt = model.forward(**batch.tensors)
rebuilt = utils.unpad1d(rebuilt, args.unpad)
for metadata, wav in zip(batch.metadata, rebuilt):
path = args.output / (metadata['name'] + ".wav")
wavfile.write(
str(path), metadata['sample_rate'],
dsp.float_wav_to_short(wav).cpu().detach().numpy())
bar.update(len(batch))
if __name__ == "__main__":
main()
================================================
FILE: sing/nsynth/__init__.py
================================================
# -*- coding: utf-8 -*-
# Copyright (c) 2018-present, Facebook, Inc.
# All rights reserved.
#
# This source code is licensed under the license found in the
# LICENSE file in the root directory of this source tree.
#
from collections import defaultdict
import gzip
import json
from pathlib import Path
import random
from scipy.io import wavfile
import torch
from torch.nn import functional as F
from ..fondation.batch import BatchItem
from ..fondation.datasets import DatasetSubset
from ..fondation import utils
class NSynthMetadata:
"""
NSynth metadata without the wavforms.
Arguments:
path (Path): path to the NSynth dataset.
This path should contain a `examples.json` file.
An item of the nsynth metadata dataset will contain the follow tensors:
- instrument (LongTensor)
- pitch (LongTensor)
- velocity (LongTensor)
- instrument_family (LongTensor)
- index (LongTensor)
Attributes:
cardinalities (dict[str, int]): cardinality of
instrument, instrument_family, pitch and velocity
instruments (dict[str, int]): mapping from instrument
name to instrument index
"""
_json_cache = {}
_FEATURES = ['instrument', 'instrument_family', 'pitch', 'velocity']
def _map_velocity(self, metadata):
velocity_mapping = {
25: 0,
50: 1,
75: 2,
100: 4,
127: 5,
}
for meta in self._metadata.values():
meta["velocity"] = velocity_mapping[meta['velocity']]
def __init__(self, path):
self.path = Path(path)
# Cache the json to avoid reparsing it everytime
if self.path in self._json_cache:
self._metadata = self._json_cache[self.path]
else:
if self.path.suffix == ".gz":
file = gzip.open(self.path)
else:
file = open(self.path, "rb")
self._metadata = json.load(file)
self._map_velocity(self._metadata)
self._json_cache[self.path] = self._metadata
self.names = sorted(self._metadata.keys())
# Compute the mapping instrument_name -> instrument id
self.instruments = {}
for meta in self._metadata.values():
self.instruments[meta["instrument_str"]] = meta["instrument"]
# Compute the cardinality for the features velocity, instrument,
# pitch and instrument_family
self.cardinalities = {}
for feature in self._FEATURES:
self.cardinalities[feature] = 1 + max(
i[feature] for i in self._metadata.values())
def __len__(self):
return len(self.names)
def __getitem__(self, index):
if hasattr(index, "item"):
index = index.item()
name = self.names[index]
metadata = self._metadata[name]
tensors = {}
metadata['name'] = name
metadata['index'] = index
for feature in self._FEATURES:
tensors[feature] = torch.LongTensor([metadata[feature]])
return BatchItem(metadata=metadata, tensors=tensors)
class NSynthDataset:
"""
NSynth dataset.
Arguments:
path (Path): path to the NSynth dataset.
This path should contain a `examples.json` file
and an `audio` folder containing the wav files.
pad (int): amount of padding to add to the waveforms.
Items from this dataset will contain all the information
coming from :class:`NSynthMetadata` as well as a `'wav'`
tensor containing the waveform.
Attributes:
metadata (NSynthMetadata): metadata only dataset
"""
def __init__(self, path, pad=0):
self.metadata = NSynthMetadata(Path(path) / "examples.json")
self.pad = pad
def __len__(self):
return len(self.metadata)
def __getitem__(self, index):
item = self.metadata[index]
path = self.metadata.path.parent / "audio" / "{}.wav".format(
item.metadata['name'])
item.metadata['path'] = path
_, wav = wavfile.read(str(path), mmap=True)
wav = torch.as_tensor(wav, dtype=torch.float)
wav /= 2**15 - 1
item.tensors['wav'] = F.pad(wav, (self.pad, self.pad))
return item
def make_datasets(dataset, valid_ratio=0.1, test_ratio=0.1, random_seed=42):
"""
Take the original NSynth training dataset and split it into
a train, valid and test set making sure that for a given instrument,
a pitch is present in only one dataset (each pair of instrument and pitch
has multiple occurences, one for each velocity).
"""
per_pitch_instrument = defaultdict(list)
if isinstance(dataset, NSynthDataset):
metadata = dataset.metadata
elif isinstance(dataset, NSynthMetadata):
metadata = dataset
else:
raise ValueError(
"Invalid dataset {}, should be an instance of "
"either NSynthDataset or NSynthMetadata.".format(dataset))
for index in range(len(metadata)):
item = metadata[index]
per_pitch_instrument[(item.metadata['instrument'],
item.metadata['pitch'])].append(index)
with utils.random_seed_manager(random_seed):
train = []
valid = []
test = []
for indexes in per_pitch_instrument.values():
score = random.random()
if score < valid_ratio:
valid.extend(indexes)
elif score < valid_ratio + test_ratio:
test.extend(indexes)
else:
train.extend(indexes)
return DatasetSubset(dataset, train), DatasetSubset(
dataset, valid), DatasetSubset(dataset, test)
def get_metadata_path():
"""
Get the path to the nsynth-train metadata included with SING.
"""
return Path(__file__).parent / "examples.json.gz"
def get_nsynth_metadata():
return NSynthMetadata(get_metadata_path())
================================================
FILE: sing/parser.py
================================================
# -*- coding: utf-8 -*-
# Copyright (c) 2018-present, Facebook, Inc.
# All rights reserved.
#
# This source code is licensed under the license found in the
# LICENSE file in the root directory of this source tree.
#
import argparse
from pathlib import Path
def get_parser():
"""
Returns:
argparse.ArgumentParser: parser with all the options
for the training of a SING model.
"""
parser = argparse.ArgumentParser(
"sing.train",
description="Train a SING model on the NSynth dataset",
formatter_class=argparse.ArgumentDefaultsHelpFormatter)
# Datasets arguments
parser.add_argument(
"--data",
default="data/nsynth-train",
type=Path,
help="path to the dataset, e.g. .../nsynth-train")
parser.add_argument(
"--pad",
type=int,
default=2304,
help="Extra padding added to the waveforms",
)
# Loss arguments
parser.add_argument("--wav", action="store_true", help="Use a Wav loss")
parser.add_argument(
"--epsilon",
default=1,
type=float,
help="Offset for power spectrum before taking the log")
parser.add_argument(
"--l1",
action="store_true",
help="Use L1 loss instead of mse",
)
# Misc arguments
parser.add_argument("--cuda", action="store_true", help="Use cuda")
parser.add_argument(
"--parallel", action="store_true", help="Use multiple gpus")
parser.add_argument(
"--checkpoint",
type=Path,
default=None,
help="Path to the checkpoint folder")
parser.add_argument(
"--output",
type=Path,
default="models/sing.th",
help="Path to output final SING model")
parser.add_argument(
"-d", "--debug", action="store_true", help="Debug flag")
parser.add_argument(
"-f", "--debug-fast", action="store_true", help="Debug fast flag")
# Common arguments
parser.add_argument(
"--lr", type=float, default=0.0003, help="Learning rate for Adam")
parser.add_argument(
"--batch-size", type=int, default=64, help="Batch size")
# Autoencoder arguments
parser.add_argument(
"--ae-epochs",
type=int,
default=50,
help="Number of epochs for the autoencoder")
parser.add_argument(
"--ae-channels",
type=int,
default=4096,
help="Number of channels in the autoencoder")
parser.add_argument(
"--ae-stride", type=int, default=256, help="Stride of the autoencoder")
parser.add_argument(
"--ae-dimension",
type=int,
default=128,
help="Dimension of the autoencoder embedding")
parser.add_argument(
"--ae-kernel",
type=int,
default=1024,
help="Kernel size of the autoencoder")
parser.add_argument(
"--ae-rewrite",
type=int,
default=2,
help="Number of rewrite layers in the autoencoder")
parser.add_argument(
"--ae-context",
type=int,
default=9,
help="Context size of the decoder")
parser.add_argument(
"--ae-window",
default="hann",
help="Window to use to smooth convolutions. Default to 'hann'. "
"To deactivate, use --ae-no-window")
parser.add_argument(
"--ae-no-window", dest="ae_window", action="store_const", const=None)
parser.add_argument(
"--ae-squared-window",
action="store_true",
default=True,
help="Square the window used to smooth convolutions. "
"To deactivate, use --ae-no-squared-window.")
parser.add_argument(
"--ae-no-squared-window",
action="store_false",
dest="ae_squared_window")
# Sequence generator arguments
parser.add_argument(
"--seq-hidden-size",
type=int,
default=1024,
help="Size of the LSTM hidden layers")
parser.add_argument(
"--seq-layers",
type=int,
default=3,
help="Number of layers in the LSTM")
parser.add_argument(
"--seq-epochs",
type=int,
default=50,
help="Number of epochs for the sequence generator")
parser.add_argument(
"--seq-truncated",
type=int,
default=32,
help="Truncated gradient for the sequence generator. "
"0 means using the full sequence.")
parser.add_argument(
"--sing-epochs",
type=int,
default=20,
help="Number of fine tuning epochs for the full SING model")
# Lookup tables arguments
parser.add_argument(
"--time-dim",
type=int,
default=4,
help="Dimension of the time step lookup table")
parser.add_argument(
"--instrument-dim",
type=int,
default=16,
help="Dimension of the instrument embedding")
parser.add_argument(
"--pitch-dim",
type=int,
default=8,
help="Dimension of the pitch embedding")
parser.add_argument(
"--velocity-dim",
type=int,
default=2,
help="Dimension of the velocity embedding")
return parser
================================================
FILE: sing/sequence/__init__.py
================================================
# Copyright (c) 2018-present, Facebook, Inc.
# All rights reserved.
#
# This source code is licensed under the license found in the
# LICENSE file in the root directory of this source tree.
#
================================================
FILE: sing/sequence/models.py
================================================
# -*- coding: utf-8 -*-
# Copyright (c) 2018-present, Facebook, Inc.
# All rights reserved.
#
# This source code is licensed under the license found in the
# LICENSE file in the root directory of this source tree.
#
import torch
from torch import nn
from ..fondation import utils
class SequenceGenerator(nn.Module):
"""
LSTM part of the SING model.
Arguments:
embeddings (dict[str, (int, int)]):
represents the lookup tables used by the model.
For each entry under the key `name`, with value
`(cardinality, dimension)`, the tensor named `name` will be
retrieved. Its values should be in `[0, cardinality - 1]`
and the lookup table will have dimension `dimension`
length (int): length of the generated sequence
output_dimension (int): dimension of each generated sequence item
hidden_size (int): size of each layer, see the documentation of
:class:`nn.LSTM`
num_layers (int): number of layers, see the documentation of
:class:`nn.LSTM`
"""
def __init__(self,
embeddings,
length,
time_dimension=4,
output_dimension=128,
hidden_size=1024,
num_layers=3):
super(SequenceGenerator, self).__init__()
self.tables = nn.ModuleList()
self.inputs = []
input_size = time_dimension
for name, (cardinality, dimension) in sorted(embeddings.items()):
input_size += dimension
self.inputs.append(name)
self.tables.append(
nn.Embedding(
num_embeddings=cardinality, embedding_dim=dimension))
if time_dimension == 0:
self.time_table = None
else:
self.time_table = nn.Embedding(
num_embeddings=length, embedding_dim=time_dimension)
self.length = length
self.lstm = nn.LSTM(
input_size=input_size,
hidden_size=hidden_size,
num_layers=num_layers)
self.decoder = nn.Linear(hidden_size, output_dimension)
def forward(self, start=0, length=None, hidden=None, **tensors):
"""
Arguments:
start (int): first time step to generate
length (int): length of the sequence to generate. If `None`,
will be taken to be `self.length - start`
hidden ((torch.FloatTensor, torch.FloatTensor)):
hidden state of the LSTM or `None` to start
from a blank one
**tensors (dict[str, torch.LongTensor]):
dictionary containing the tensors used as inputs
to the lookup tables specified by the `embeddings`
parameter of the constructor
"""
length = self.length - start if length is None else length
inputs = []
for name, table in zip(self.inputs, self.tables):
value = tensors[name].transpose(0, 1)
embedding = table.forward(value)
inputs.append(embedding.expand(length, -1, -1))
reference = inputs[0]
if self.time_table is not None:
times = torch.arange(
start, start + length,
device=reference.device).view(-1, 1).expand(
-1, reference.size(1))
inputs.append(self.time_table.forward(times))
input = torch.cat(inputs, dim=-1)
if hidden is not None:
hidden = [h.transpose(0, 1).contiguous() for h in hidden]
self.lstm.flatten_parameters()
output, hidden = self.lstm.forward(input, hidden)
decoded = self.decoder(output.view(-1, output.size(-1))).view(
output.size(0), output.size(1), -1)
hidden = [h.transpose(0, 1) for h in hidden]
return decoded.transpose(0, 1).transpose(1, 2), hidden
class SING(nn.Module):
"""
Complete SING model.
Arguments:
sequence_generator (SequenceGenerator): the LSTM based
sequence generator part of SING
decoder (sing.ae.models.ConvolutionalDecoder):
the convolutional decoder part of SING
"""
def __init__(self, sequence_generator, decoder):
super(SING, self).__init__()
self.sequence_generator = sequence_generator
self.decoder = decoder
def forward(self, **tensors):
"""
Arguments:
**tensors (dict[str, torch.LongTensor]):
Tensors used as inputs
to the lookup tables specified by the `embeddings`
parameter of :class:`SequenceGenerator`
"""
return self.decoder.forward(self.sequence_generator(**tensors)[0])
def download_pretrained_model(target):
"""
Download a pretrained version of SING.
"""
url = "https://dl.fbaipublicfiles.com/sing/sing.th"
sha256 = "eda8a7ce66f1ccf31cdd34a920290d80aabf96584c4d53df866b744f2862dc1c"
utils.download_file(target, url, sha256=sha256)
================================================
FILE: sing/sequence/trainer.py
================================================
# -*- coding: utf-8 -*-
# Copyright (c) 2018-present, Facebook, Inc.
# All rights reserved.
#
# This source code is licensed under the license found in the
# LICENSE file in the root directory of this source tree.
#
from torch import nn
from ..fondation import utils, trainer
class SequenceGeneratorTrainer(trainer.BaseTrainer):
"""
Trainer for the sequence generator (LSTM) part of SING.
Arguments:
decoder (sing.ae.models.ConvolutionalDecoder):
decoder, used to compute the metrics on the waveforms
truncated_gradient (int): size of sequence to compute
the gradients over. If `None`, the whole sequence is used
"""
def __init__(self, decoder, truncated_gradient=32, **kwargs):
super(SequenceGeneratorTrainer, self).__init__(**kwargs)
self.truncated_gradient = truncated_gradient
self.decoder = decoder
if self.is_parallel:
self.parallel_decoder = nn.DataParallel(decoder)
else:
self.parallel_decoder = decoder
def _train_batch(self, batch):
embeddings = batch.tensors['embeddings']
assert embeddings.size(-1) == self.model.length
total_length = self.model.length
hidden = None
if self.truncated_gradient:
truncated_gradient = self.truncated_gradient
else:
truncated_gradient = total_length
steps = list(range(0, total_length, truncated_gradient))
total_loss = 0
for start_time in steps:
sequence_length = min(truncated_gradient,
total_length - start_time)
target = embeddings[..., start_time:start_time + sequence_length]
rebuilt, hidden = self.parallel.forward(
start=start_time,
length=sequence_length,
hidden=hidden,
**batch.tensors)
hidden = tuple([h.detach() for h in hidden])
self.optimizer.zero_grad()
loss = self.train_loss(rebuilt, target)
loss.backward()
self.optimizer.step()
total_loss += loss.item() / len(steps)
return total_loss
def _get_rebuilt_target(self, batch):
wav = batch.tensors['wav']
target = utils.unpad1d(wav, self.decoder.strip)
embeddings, _ = self.parallel.forward(**batch.tensors)
rebuilt = self.parallel_decoder.forward(embeddings)
return rebuilt, target
class SINGTrainer(trainer.BaseTrainer):
"""
Trainer for the entire SING model.
"""
def _get_rebuilt_target(self, batch):
wav = batch.tensors['wav']
rebuilt = self.parallel.forward(**batch.tensors)
target = utils.unpad1d(wav, self.model.decoder.strip)
return rebuilt, target
================================================
FILE: sing/sequence/utils.py
================================================
# -*- coding: utf-8 -*-
# Copyright (c) 2018-present, Facebook, Inc.
# All rights reserved.
#
# This source code is licensed under the license found in the
# LICENSE file in the root directory of this source tree.
#
import torch
from torch import nn
from torch.utils.data import DataLoader
import tqdm
from ..fondation.batch import collate
def generate_embeddings_dataset(dataset, encoder, batch_size, cuda, parallel):
"""
Pre-compute all the embeddings for a given dataset.
Arguments:
dataset (Dataset): dataset to compute the embeddings for. It should
contain a `'wav'` tensor
encoder (sing.ae.models.ConvolutionalEncoder):
encoder to use to generate the embedding
batch_size (int): batch size to use
cuda (bool): if `True`, performs the computation on GPU
parallel (bool): if `True`, use all available GPUs
Returns:
Dataset: dataset of the same size as `dataset` but with the `'wav'`
tensor replaced by an `'embeddings'` tensor.
"""
loader = DataLoader(
dataset, batch_size=batch_size, shuffle=False, collate_fn=collate)
embeddings_dataset = [None] * len(dataset)
if cuda:
encoder.cuda()
if parallel:
encoder = nn.DataParallel(encoder)
row = 0
with tqdm.tqdm(total=len(dataset), unit="ex") as bar:
for batch in loader:
if cuda:
batch.cuda_()
with torch.no_grad():
batch.tensors['embeddings'] = encoder.forward(
batch.tensors['wav'])
del batch.tensors['wav']
for item in batch.cpu():
embeddings_dataset[row] = item
row += 1
bar.update(len(batch))
return embeddings_dataset
================================================
FILE: sing/train.py
================================================
# -*- coding: utf-8 -*-
# Copyright (c) 2018-present, Facebook, Inc.
# All rights reserved.
#
# This source code is licensed under the license found in the
# LICENSE file in the root directory of this source tree.
#
import functools
from torch import nn
import torch
from .parser import get_parser
from . import nsynth, dsp
from .ae.models import ConvolutionalAE
from .ae.trainer import AutoencoderTrainer
from .fondation import utils, datasets
from .sequence.models import SequenceGenerator, SING
from .sequence.trainer import SequenceGeneratorTrainer, SINGTrainer
from .sequence.utils import generate_embeddings_dataset
def train_autoencoder(args, **kwargs):
checkpoint_path = args.checkpoint / "ae.torch" if args.checkpoint else None
model = ConvolutionalAE(
channels=args.ae_channels,
stride=args.ae_stride,
dimension=args.ae_dimension,
kernel_size=args.ae_kernel,
context_size=args.ae_context,
rewrite_layers=args.ae_rewrite,
window_name=args.ae_window,
squared_window=args.ae_squared_window)
advised_pad = model.decoder.strip + 512
if args.pad != advised_pad:
print("Warning, best padding for the current settings is {}, "
"current value is {}.".format(advised_pad, args.pad))
if args.ae_epochs:
print("Training autoencoder")
AutoencoderTrainer(
suffix="_ae",
model=model,
epochs=args.ae_epochs,
checkpoint_path=checkpoint_path,
**kwargs).train()
return model
def train_sequence_generator(args, autoencoder, cardinalities, train_dataset,
eval_datasets, train_loss, eval_losses, **kwargs):
checkpoint_path = (args.checkpoint / "seq.torch"
if args.checkpoint else None)
wav_length = train_dataset[0].tensors['wav'].size(-1)
embedding_length = autoencoder.decoder.embedding_length(
wav_length - 2 * autoencoder.decoder.strip)
embeddings = {
name: (cardinalities[name], getattr(args, '{}_dim'.format(name)))
for name in ['velocity', 'instrument', 'pitch']
}
model = SequenceGenerator(
embeddings=embeddings,
length=embedding_length,
time_dimension=args.time_dim,
output_dimension=args.ae_dimension,
hidden_size=args.seq_hidden_size,
num_layers=args.seq_layers)
if args.seq_epochs:
print("Precomputing embeddings for all datasets")
generate_embeddings = functools.partial(
generate_embeddings_dataset,
encoder=autoencoder.encoder,
batch_size=args.batch_size,
cuda=args.cuda,
parallel=args.parallel)
train_dataset = generate_embeddings(train_dataset)
print("Training sequence generator")
SequenceGeneratorTrainer(
suffix="_seq",
model=model,
decoder=autoencoder.decoder,
epochs=args.seq_epochs,
train_loss=nn.MSELoss(),
eval_losses=eval_losses,
train_dataset=train_dataset,
eval_datasets=eval_datasets,
truncated_gradient=args.seq_truncated,
checkpoint_path=checkpoint_path,
**kwargs).train()
return model
def fine_tune_sing(args, sequence_generator, decoder, **kwargs):
print("Fine tuning SING")
checkpoint_path = (args.checkpoint / "sing.torch"
if args.checkpoint else None)
model = SING(sequence_generator=sequence_generator, decoder=decoder)
if args.sing_epochs:
SINGTrainer(
suffix="_sing",
epochs=args.sing_epochs,
model=model,
checkpoint_path=checkpoint_path,
**kwargs).train()
return model
def main():
args = get_parser().parse_args()
if args.debug:
args.ae_epochs = 1
args.seq_epochs = 1
args.sing_epochs = 1
if args.debug_fast:
args.ae_channels = 128
args.ae_dimension = 16
args.ae_rewrite = 1
args.seq_hidden_size = 128
args.seq_layers = 1
if args.checkpoint:
args.checkpoint.mkdir(exist_ok=True, parents=True)
if not args.data.exists():
utils.fatal("Could not find the nsynth dataset. "
"To download it, follow the instructions at "
"https://github.com/facebookresearch/SING")
nsynth_dataset = nsynth.NSynthDataset(args.data, pad=args.pad)
cardinalities = nsynth_dataset.metadata.cardinalities
train_dataset, valid, test = nsynth.make_datasets(nsynth_dataset)
if args.debug:
train_dataset = datasets.RandomSubset(train_dataset, size=100)
eval_train = datasets.RandomSubset(train_dataset, size=10000)
if args.debug:
eval_datasets = {
'eval_train': eval_train,
}
else:
eval_datasets = {
'eval_train': eval_train,
'valid': valid,
'test': test,
}
base_loss = nn.L1Loss() if args.l1 else nn.MSELoss()
train_loss = base_loss if args.wav else dsp.SpectralLoss(
base_loss, epsilon=args.epsilon)
eval_losses = {
'wav_l1': nn.L1Loss(),
'wav_mse': nn.MSELoss(),
'spec_l1': dsp.SpectralLoss(nn.L1Loss(), epsilon=args.epsilon),
'spec_mse': dsp.SpectralLoss(nn.MSELoss(), epsilon=args.epsilon),
}
kwargs = {
'train_dataset': train_dataset,
'eval_datasets': eval_datasets,
'train_loss': train_loss,
'eval_losses': eval_losses,
'batch_size': args.batch_size,
'lr': args.lr,
'cuda': args.cuda,
'parallel': args.parallel,
}
autoencoder = train_autoencoder(args, **kwargs)
sequence_generator = train_sequence_generator(args, autoencoder,
cardinalities, **kwargs)
sing = fine_tune_sing(args, sequence_generator, autoencoder.decoder,
**kwargs)
torch.save(sing.cpu(), str(args.output))
if __name__ == "__main__":
main()
gitextract_6jkoz_29/
├── CODE_OF_CONDUCT.md
├── CONTRIBUTING.md
├── LICENSE
├── MANIFEST.in
├── README.md
├── environment.yml
├── nsynth_100_test.txt
├── requirements.txt
├── setup.py
└── sing/
├── __init__.py
├── ae/
│ ├── __init__.py
│ ├── models.py
│ ├── trainer.py
│ └── utils.py
├── dsp.py
├── fondation/
│ ├── __init__.py
│ ├── batch.py
│ ├── datasets.py
│ ├── trainer.py
│ └── utils.py
├── generate.py
├── nsynth/
│ └── __init__.py
├── parser.py
├── sequence/
│ ├── __init__.py
│ ├── models.py
│ ├── trainer.py
│ └── utils.py
└── train.py
SYMBOL INDEX (104 symbols across 15 files)
FILE: sing/ae/models.py
class ConvolutionalDecoder (line 14) | class ConvolutionalDecoder(nn.Module):
method __init__ (line 37) | def __init__(self,
method __repr__ (line 79) | def __repr__(self):
method forward (line 82) | def forward(self, embeddings):
method wav_length (line 85) | def wav_length(self, embedding_length):
method embedding_length (line 93) | def embedding_length(self, wav_length):
class ConvolutionalEncoder (line 102) | class ConvolutionalEncoder(nn.Module):
method __init__ (line 122) | def __init__(self,
method __repr__ (line 153) | def __repr__(self):
method forward (line 156) | def forward(self, signal):
class ConvolutionalAE (line 160) | class ConvolutionalAE(nn.Module):
method __init__ (line 182) | def __init__(self,
method encode (line 212) | def encode(self, signal):
method decode (line 218) | def decode(self, embeddings):
method forward (line 224) | def forward(self, signal):
method __repr__ (line 227) | def __repr__(self):
FILE: sing/ae/trainer.py
class AutoencoderTrainer (line 12) | class AutoencoderTrainer(trainer.BaseTrainer):
method _train_batch (line 17) | def _train_batch(self, batch):
method _get_rebuilt_target (line 25) | def _get_rebuilt_target(self, batch):
FILE: sing/ae/utils.py
class WindowedConv1d (line 15) | class WindowedConv1d(nn.Module):
method __init__ (line 26) | def __init__(self, conv, window_name='hann', squared=True):
method forward (line 38) | def forward(self, input):
method __repr__ (line 49) | def __repr__(self):
class WindowedConvTranpose1d (line 54) | class WindowedConvTranpose1d(nn.Module):
method __init__ (line 65) | def __init__(self, conv_tr, window_name='hann', squared=True):
method forward (line 77) | def forward(self, input):
method __repr__ (line 89) | def __repr__(self):
FILE: sing/dsp.py
function power (line 14) | def power(spec):
function get_window (line 25) | def get_window(name, window_length, squared=False):
class STFT (line 46) | class STFT(nn.Module):
method __init__ (line 60) | def __init__(self, n_fft=1024, hop_length=None, window_name='hann'):
method forward (line 70) | def forward(self, input):
class SpectralLoss (line 79) | class SpectralLoss(nn.Module):
method __init__ (line 90) | def __init__(self, base_loss=F.mse_loss, epsilon=1, **kwargs):
method _log_spectrogram (line 96) | def _log_spectrogram(self, signal):
method forward (line 99) | def forward(self, a, b):
function float_wav_to_short (line 105) | def float_wav_to_short(wav):
FILE: sing/fondation/batch.py
class BatchItem (line 12) | class BatchItem:
method __init__ (line 28) | def __init__(self, metadata=None, tensors=None):
function collate (line 33) | def collate(items):
class Batch (line 51) | class Batch:
method __init__ (line 67) | def __init__(self, metadata, tensors):
method __len__ (line 71) | def __len__(self):
method __iter__ (line 74) | def __iter__(self):
method __getitem__ (line 78) | def __getitem__(self, index):
method apply (line 96) | def apply(self, function):
method apply_ (line 112) | def apply_(self, function):
method cuda (line 120) | def cuda(self, *args, **kwargs):
method cuda_ (line 126) | def cuda_(self, *args, **kwargs):
method cpu (line 132) | def cpu(self, *args, **kwargs):
method cpu_ (line 138) | def cpu_(self, *args, **kwargs):
FILE: sing/fondation/datasets.py
class DatasetSubset (line 16) | class DatasetSubset:
method __init__ (line 25) | def __init__(self, dataset, indexes):
method __len__ (line 29) | def __len__(self):
method __getitem__ (line 32) | def __getitem__(self, index):
class RandomSubset (line 36) | class RandomSubset(DatasetSubset):
method __init__ (line 46) | def __init__(self, dataset, size, random_seed=42):
FILE: sing/fondation/trainer.py
class BaseTrainer (line 19) | class BaseTrainer:
method __init__ (line 46) | def __init__(self,
method _train_epoch (line 85) | def _train_epoch(self, dataset, epoch):
method _eval_dataset (line 111) | def _eval_dataset(self, dataset_name, dataset, epoch):
method _train_batch (line 135) | def _train_batch(self, batch):
method _get_rebuilt_target (line 152) | def _get_rebuilt_target(self, batch):
method train (line 160) | def train(self):
FILE: sing/fondation/utils.py
function random_seed_manager (line 21) | def random_seed_manager(seed):
function progress_iterator (line 34) | def progress_iterator(iterator, divisions=100):
function unpad1d (line 63) | def unpad1d(tensor, pad):
function load_checkpoint (line 77) | def load_checkpoint(path):
function save_checkpoint (line 89) | def save_checkpoint(path, epoch, state):
function download_file (line 108) | def download_file(target, url, sha256=None):
function fatal (line 141) | def fatal(message, error_code=1):
FILE: sing/generate.py
function get_parser (line 26) | def get_parser():
function main (line 69) | def main():
FILE: sing/nsynth/__init__.py
class NSynthMetadata (line 24) | class NSynthMetadata:
method _map_velocity (line 49) | def _map_velocity(self, metadata):
method __init__ (line 60) | def __init__(self, path):
method __len__ (line 89) | def __len__(self):
method __getitem__ (line 92) | def __getitem__(self, index):
class NSynthDataset (line 107) | class NSynthDataset:
method __init__ (line 125) | def __init__(self, path, pad=0):
method __len__ (line 129) | def __len__(self):
method __getitem__ (line 132) | def __getitem__(self, index):
function make_datasets (line 147) | def make_datasets(dataset, valid_ratio=0.1, test_ratio=0.1, random_seed=...
function get_metadata_path (line 188) | def get_metadata_path():
function get_nsynth_metadata (line 195) | def get_nsynth_metadata():
FILE: sing/parser.py
function get_parser (line 13) | def get_parser():
FILE: sing/sequence/models.py
class SequenceGenerator (line 15) | class SequenceGenerator(nn.Module):
method __init__ (line 35) | def __init__(self,
method forward (line 67) | def forward(self, start=0, length=None, hidden=None, **tensors):
class SING (line 108) | class SING(nn.Module):
method __init__ (line 119) | def __init__(self, sequence_generator, decoder):
method forward (line 124) | def forward(self, **tensors):
function download_pretrained_model (line 135) | def download_pretrained_model(target):
FILE: sing/sequence/trainer.py
class SequenceGeneratorTrainer (line 14) | class SequenceGeneratorTrainer(trainer.BaseTrainer):
method __init__ (line 26) | def __init__(self, decoder, truncated_gradient=32, **kwargs):
method _train_batch (line 35) | def _train_batch(self, batch):
method _get_rebuilt_target (line 65) | def _get_rebuilt_target(self, batch):
class SINGTrainer (line 73) | class SINGTrainer(trainer.BaseTrainer):
method _get_rebuilt_target (line 78) | def _get_rebuilt_target(self, batch):
FILE: sing/sequence/utils.py
function generate_embeddings_dataset (line 17) | def generate_embeddings_dataset(dataset, encoder, batch_size, cuda, para...
FILE: sing/train.py
function train_autoencoder (line 24) | def train_autoencoder(args, **kwargs):
function train_sequence_generator (line 50) | def train_sequence_generator(args, autoencoder, cardinalities, train_dat...
function fine_tune_sing (line 97) | def fine_tune_sing(args, sequence_generator, decoder, **kwargs):
function main (line 113) | def main():
Condensed preview — 28 files, each showing path, character count, and a content snippet. Download the .json file or copy for the full structured content (98K chars).
[
{
"path": "CODE_OF_CONDUCT.md",
"chars": 243,
"preview": "# Code of Conduct\n\nFacebook has adopted a Code of Conduct that we expect project participants to adhere to.\nPlease read "
},
{
"path": "CONTRIBUTING.md",
"chars": 741,
"preview": "# Contributing to SING\n\n## Pull Requests\n\nIn order to accept your pull request, we need you to submit a CLA. You only ne"
},
{
"path": "LICENSE",
"chars": 19331,
"preview": "Attribution-NonCommercial 4.0 International\n\n=======================================================================\n\nCr"
},
{
"path": "MANIFEST.in",
"chars": 31,
"preview": "include nsynth/examples.json.gz"
},
{
"path": "README.md",
"chars": 5542,
"preview": "# SING: Symbol-to-Instrument Neural Generator\n\nSING is a deep learning based music notes synthetizer that can be trained"
},
{
"path": "environment.yml",
"chars": 146,
"preview": "name: sing\nchannels:\n - pytorch\ndependencies:\n - numpy>=1.15\n - python>=3.6\n - pytorch>=0.4.1\n - requests>=2.19\n -"
},
{
"path": "nsynth_100_test.txt",
"chars": 2860,
"preview": "bass_synthetic_126-025-025\nsynth_lead_synthetic_006-045-050\nbass_synthetic_065-070-025\nkeyboard_electronic_063-087-100\nr"
},
{
"path": "requirements.txt",
"chars": 38,
"preview": "numpy\nrequests\nscipy\ntorch>=0.4.1\ntqdm"
},
{
"path": "setup.py",
"chars": 1697,
"preview": "#!/usr/bin/env python\n# -*- coding: utf-8 -*-\n# Copyright (c) 2018-present, Facebook, Inc.\n# All rights reserved.\n#\n# Th"
},
{
"path": "sing/__init__.py",
"chars": 216,
"preview": "# -*- coding: utf-8 -*-\n# Copyright (c) 2018-present, Facebook, Inc.\n# All rights reserved.\n#\n# This source code is lice"
},
{
"path": "sing/ae/__init__.py",
"chars": 216,
"preview": "# -*- coding: utf-8 -*-\n# Copyright (c) 2018-present, Facebook, Inc.\n# All rights reserved.\n#\n# This source code is lice"
},
{
"path": "sing/ae/models.py",
"chars": 8123,
"preview": "# -*- coding: utf-8 -*-\n# Copyright (c) 2018-present, Facebook, Inc.\n# All rights reserved.\n#\n# This source code is lice"
},
{
"path": "sing/ae/trainer.py",
"chars": 828,
"preview": "# -*- coding: utf-8 -*-\n# Copyright (c) 2018-present, Facebook, Inc.\n# All rights reserved.\n#\n# This source code is lice"
},
{
"path": "sing/ae/utils.py",
"chars": 3016,
"preview": "# -*- coding: utf-8 -*-\n# Copyright (c) 2018-present, Facebook, Inc.\n# All rights reserved.\n#\n# This source code is lice"
},
{
"path": "sing/dsp.py",
"chars": 3128,
"preview": "# -*- coding: utf-8 -*-\n# Copyright (c) 2018-present, Facebook, Inc.\n# All rights reserved.\n#\n# This source code is lice"
},
{
"path": "sing/fondation/__init__.py",
"chars": 216,
"preview": "# -*- coding: utf-8 -*-\n# Copyright (c) 2018-present, Facebook, Inc.\n# All rights reserved.\n#\n# This source code is lice"
},
{
"path": "sing/fondation/batch.py",
"chars": 4218,
"preview": "# -*- coding: utf-8 -*-\n# Copyright (c) 2018-present, Facebook, Inc.\n# All rights reserved.\n#\n# This source code is lice"
},
{
"path": "sing/fondation/datasets.py",
"chars": 1342,
"preview": "# -*- coding: utf-8 -*-\n# Copyright (c) 2018-present, Facebook, Inc.\n# All rights reserved.\n#\n# This source code is lice"
},
{
"path": "sing/fondation/trainer.py",
"chars": 6776,
"preview": "# -*- coding: utf-8 -*-\n# Copyright (c) 2018-present, Facebook, Inc.\n# All rights reserved.\n#\n# This source code is lice"
},
{
"path": "sing/fondation/utils.py",
"chars": 4104,
"preview": "# -*- coding: utf-8 -*-\n# Copyright (c) 2018-present, Facebook, Inc.\n# All rights reserved.\n#\n# This source code is lice"
},
{
"path": "sing/generate.py",
"chars": 3735,
"preview": "# -*- coding: utf-8 -*-\n# Copyright (c) 2018-present, Facebook, Inc.\n# All rights reserved.\n#\n# This source code is lice"
},
{
"path": "sing/nsynth/__init__.py",
"chars": 6017,
"preview": "# -*- coding: utf-8 -*-\n# Copyright (c) 2018-present, Facebook, Inc.\n# All rights reserved.\n#\n# This source code is lice"
},
{
"path": "sing/parser.py",
"chars": 5199,
"preview": "# -*- coding: utf-8 -*-\n# Copyright (c) 2018-present, Facebook, Inc.\n# All rights reserved.\n#\n# This source code is lice"
},
{
"path": "sing/sequence/__init__.py",
"chars": 192,
"preview": "# Copyright (c) 2018-present, Facebook, Inc.\n# All rights reserved.\n#\n# This source code is licensed under the license f"
},
{
"path": "sing/sequence/models.py",
"chars": 5052,
"preview": "# -*- coding: utf-8 -*-\n# Copyright (c) 2018-present, Facebook, Inc.\n# All rights reserved.\n#\n# This source code is lice"
},
{
"path": "sing/sequence/trainer.py",
"chars": 2807,
"preview": "# -*- coding: utf-8 -*-\n# Copyright (c) 2018-present, Facebook, Inc.\n# All rights reserved.\n#\n# This source code is lice"
},
{
"path": "sing/sequence/utils.py",
"chars": 1794,
"preview": "# -*- coding: utf-8 -*-\n# Copyright (c) 2018-present, Facebook, Inc.\n# All rights reserved.\n#\n# This source code is lice"
},
{
"path": "sing/train.py",
"chars": 6103,
"preview": "# -*- coding: utf-8 -*-\n# Copyright (c) 2018-present, Facebook, Inc.\n# All rights reserved.\n#\n# This source code is lice"
}
]
About this extraction
This page contains the full source code of the facebookresearch/SING GitHub repository, extracted and formatted as plain text for AI agents and large language models (LLMs). The extraction includes 28 files (91.5 KB), approximately 21.3k tokens, and a symbol index with 104 extracted functions, classes, methods, constants, and types. Use this with OpenClaw, Claude, ChatGPT, Cursor, Windsurf, or any other AI tool that accepts text input. You can copy the full output to your clipboard or download it as a .txt file.
Extracted by GitExtract — free GitHub repo to text converter for AI. Built by Nikandr Surkov.