Repository: facebookresearch/SING Branch: main Commit: 72054bdb23b4 Files: 28 Total size: 91.5 KB Directory structure: gitextract_6jkoz_29/ ├── CODE_OF_CONDUCT.md ├── CONTRIBUTING.md ├── LICENSE ├── MANIFEST.in ├── README.md ├── environment.yml ├── nsynth_100_test.txt ├── requirements.txt ├── setup.py └── sing/ ├── __init__.py ├── ae/ │ ├── __init__.py │ ├── models.py │ ├── trainer.py │ └── utils.py ├── dsp.py ├── fondation/ │ ├── __init__.py │ ├── batch.py │ ├── datasets.py │ ├── trainer.py │ └── utils.py ├── generate.py ├── nsynth/ │ └── __init__.py ├── parser.py ├── sequence/ │ ├── __init__.py │ ├── models.py │ ├── trainer.py │ └── utils.py └── train.py ================================================ FILE CONTENTS ================================================ ================================================ FILE: CODE_OF_CONDUCT.md ================================================ # Code of Conduct Facebook has adopted a Code of Conduct that we expect project participants to adhere to. Please read the [full text](https://code.fb.com/codeofconduct/) so that you can understand what actions will and will not be tolerated. ================================================ FILE: CONTRIBUTING.md ================================================ # Contributing to SING ## Pull Requests In order to accept your pull request, we need you to submit a CLA. You only need to do this once to work on any of Facebook's open source projects. Complete your CLA here: SING is the implementation of a research paper. Therefore, we do not plan on accepting many pull requests for new features. We certainly welcome them for bug fixes. ## Issues We use GitHub issues to track public bugs. Please ensure your description is clear and has sufficient instructions to be able to reproduce the issue. ## License By contributing to this repository, you agree that your contributions will be licensed under the LICENSE file in the root directory of this source tree. ================================================ FILE: LICENSE ================================================ Attribution-NonCommercial 4.0 International ======================================================================= Creative Commons Corporation ("Creative Commons") is not a law firm and does not provide legal services or legal advice. Distribution of Creative Commons public licenses does not create a lawyer-client or other relationship. Creative Commons makes its licenses and related information available on an "as-is" basis. Creative Commons gives no warranties regarding its licenses, any material licensed under their terms and conditions, or any related information. Creative Commons disclaims all liability for damages resulting from their use to the fullest extent possible. Using Creative Commons Public Licenses Creative Commons public licenses provide a standard set of terms and conditions that creators and other rights holders may use to share original works of authorship and other material subject to copyright and certain other rights specified in the public license below. The following considerations are for informational purposes only, are not exhaustive, and do not form part of our licenses. Considerations for licensors: Our public licenses are intended for use by those authorized to give the public permission to use material in ways otherwise restricted by copyright and certain other rights. Our licenses are irrevocable. Licensors should read and understand the terms and conditions of the license they choose before applying it. Licensors should also secure all rights necessary before applying our licenses so that the public can reuse the material as expected. Licensors should clearly mark any material not subject to the license. This includes other CC- licensed material, or material used under an exception or limitation to copyright. More considerations for licensors: wiki.creativecommons.org/Considerations_for_licensors Considerations for the public: By using one of our public licenses, a licensor grants the public permission to use the licensed material under specified terms and conditions. If the licensor's permission is not necessary for any reason--for example, because of any applicable exception or limitation to copyright--then that use is not regulated by the license. Our licenses grant only permissions under copyright and certain other rights that a licensor has authority to grant. Use of the licensed material may still be restricted for other reasons, including because others have copyright or other rights in the material. A licensor may make special requests, such as asking that all changes be marked or described. Although not required by our licenses, you are encouraged to respect those requests where reasonable. More_considerations for the public: wiki.creativecommons.org/Considerations_for_licensees ======================================================================= Creative Commons Attribution-NonCommercial 4.0 International Public License By exercising the Licensed Rights (defined below), You accept and agree to be bound by the terms and conditions of this Creative Commons Attribution-NonCommercial 4.0 International Public License ("Public License"). To the extent this Public License may be interpreted as a contract, You are granted the Licensed Rights in consideration of Your acceptance of these terms and conditions, and the Licensor grants You such rights in consideration of benefits the Licensor receives from making the Licensed Material available under these terms and conditions. Section 1 -- Definitions. a. Adapted Material means material subject to Copyright and Similar Rights that is derived from or based upon the Licensed Material and in which the Licensed Material is translated, altered, arranged, transformed, or otherwise modified in a manner requiring permission under the Copyright and Similar Rights held by the Licensor. For purposes of this Public License, where the Licensed Material is a musical work, performance, or sound recording, Adapted Material is always produced where the Licensed Material is synched in timed relation with a moving image. b. Adapter's License means the license You apply to Your Copyright and Similar Rights in Your contributions to Adapted Material in accordance with the terms and conditions of this Public License. c. Copyright and Similar Rights means copyright and/or similar rights closely related to copyright including, without limitation, performance, broadcast, sound recording, and Sui Generis Database Rights, without regard to how the rights are labeled or categorized. For purposes of this Public License, the rights specified in Section 2(b)(1)-(2) are not Copyright and Similar Rights. d. Effective Technological Measures means those measures that, in the absence of proper authority, may not be circumvented under laws fulfilling obligations under Article 11 of the WIPO Copyright Treaty adopted on December 20, 1996, and/or similar international agreements. e. Exceptions and Limitations means fair use, fair dealing, and/or any other exception or limitation to Copyright and Similar Rights that applies to Your use of the Licensed Material. f. Licensed Material means the artistic or literary work, database, or other material to which the Licensor applied this Public License. g. Licensed Rights means the rights granted to You subject to the terms and conditions of this Public License, which are limited to all Copyright and Similar Rights that apply to Your use of the Licensed Material and that the Licensor has authority to license. h. Licensor means the individual(s) or entity(ies) granting rights under this Public License. i. NonCommercial means not primarily intended for or directed towards commercial advantage or monetary compensation. For purposes of this Public License, the exchange of the Licensed Material for other material subject to Copyright and Similar Rights by digital file-sharing or similar means is NonCommercial provided there is no payment of monetary compensation in connection with the exchange. j. Share means to provide material to the public by any means or process that requires permission under the Licensed Rights, such as reproduction, public display, public performance, distribution, dissemination, communication, or importation, and to make material available to the public including in ways that members of the public may access the material from a place and at a time individually chosen by them. k. Sui Generis Database Rights means rights other than copyright resulting from Directive 96/9/EC of the European Parliament and of the Council of 11 March 1996 on the legal protection of databases, as amended and/or succeeded, as well as other essentially equivalent rights anywhere in the world. l. You means the individual or entity exercising the Licensed Rights under this Public License. Your has a corresponding meaning. Section 2 -- Scope. a. License grant. 1. Subject to the terms and conditions of this Public License, the Licensor hereby grants You a worldwide, royalty-free, non-sublicensable, non-exclusive, irrevocable license to exercise the Licensed Rights in the Licensed Material to: a. reproduce and Share the Licensed Material, in whole or in part, for NonCommercial purposes only; and b. produce, reproduce, and Share Adapted Material for NonCommercial purposes only. 2. Exceptions and Limitations. For the avoidance of doubt, where Exceptions and Limitations apply to Your use, this Public License does not apply, and You do not need to comply with its terms and conditions. 3. Term. The term of this Public License is specified in Section 6(a). 4. Media and formats; technical modifications allowed. The Licensor authorizes You to exercise the Licensed Rights in all media and formats whether now known or hereafter created, and to make technical modifications necessary to do so. The Licensor waives and/or agrees not to assert any right or authority to forbid You from making technical modifications necessary to exercise the Licensed Rights, including technical modifications necessary to circumvent Effective Technological Measures. For purposes of this Public License, simply making modifications authorized by this Section 2(a) (4) never produces Adapted Material. 5. Downstream recipients. a. Offer from the Licensor -- Licensed Material. Every recipient of the Licensed Material automatically receives an offer from the Licensor to exercise the Licensed Rights under the terms and conditions of this Public License. b. No downstream restrictions. You may not offer or impose any additional or different terms or conditions on, or apply any Effective Technological Measures to, the Licensed Material if doing so restricts exercise of the Licensed Rights by any recipient of the Licensed Material. 6. No endorsement. Nothing in this Public License constitutes or may be construed as permission to assert or imply that You are, or that Your use of the Licensed Material is, connected with, or sponsored, endorsed, or granted official status by, the Licensor or others designated to receive attribution as provided in Section 3(a)(1)(A)(i). b. Other rights. 1. Moral rights, such as the right of integrity, are not licensed under this Public License, nor are publicity, privacy, and/or other similar personality rights; however, to the extent possible, the Licensor waives and/or agrees not to assert any such rights held by the Licensor to the limited extent necessary to allow You to exercise the Licensed Rights, but not otherwise. 2. Patent and trademark rights are not licensed under this Public License. 3. To the extent possible, the Licensor waives any right to collect royalties from You for the exercise of the Licensed Rights, whether directly or through a collecting society under any voluntary or waivable statutory or compulsory licensing scheme. In all other cases the Licensor expressly reserves any right to collect such royalties, including when the Licensed Material is used other than for NonCommercial purposes. Section 3 -- License Conditions. Your exercise of the Licensed Rights is expressly made subject to the following conditions. a. Attribution. 1. If You Share the Licensed Material (including in modified form), You must: a. retain the following if it is supplied by the Licensor with the Licensed Material: i. identification of the creator(s) of the Licensed Material and any others designated to receive attribution, in any reasonable manner requested by the Licensor (including by pseudonym if designated); ii. a copyright notice; iii. a notice that refers to this Public License; iv. a notice that refers to the disclaimer of warranties; v. a URI or hyperlink to the Licensed Material to the extent reasonably practicable; b. indicate if You modified the Licensed Material and retain an indication of any previous modifications; and c. indicate the Licensed Material is licensed under this Public License, and include the text of, or the URI or hyperlink to, this Public License. 2. You may satisfy the conditions in Section 3(a)(1) in any reasonable manner based on the medium, means, and context in which You Share the Licensed Material. For example, it may be reasonable to satisfy the conditions by providing a URI or hyperlink to a resource that includes the required information. 3. If requested by the Licensor, You must remove any of the information required by Section 3(a)(1)(A) to the extent reasonably practicable. 4. If You Share Adapted Material You produce, the Adapter's License You apply must not prevent recipients of the Adapted Material from complying with this Public License. Section 4 -- Sui Generis Database Rights. Where the Licensed Rights include Sui Generis Database Rights that apply to Your use of the Licensed Material: a. for the avoidance of doubt, Section 2(a)(1) grants You the right to extract, reuse, reproduce, and Share all or a substantial portion of the contents of the database for NonCommercial purposes only; b. if You include all or a substantial portion of the database contents in a database in which You have Sui Generis Database Rights, then the database in which You have Sui Generis Database Rights (but not its individual contents) is Adapted Material; and c. You must comply with the conditions in Section 3(a) if You Share all or a substantial portion of the contents of the database. For the avoidance of doubt, this Section 4 supplements and does not replace Your obligations under this Public License where the Licensed Rights include other Copyright and Similar Rights. Section 5 -- Disclaimer of Warranties and Limitation of Liability. a. UNLESS OTHERWISE SEPARATELY UNDERTAKEN BY THE LICENSOR, TO THE EXTENT POSSIBLE, THE LICENSOR OFFERS THE LICENSED MATERIAL AS-IS AND AS-AVAILABLE, AND MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND CONCERNING THE LICENSED MATERIAL, WHETHER EXPRESS, IMPLIED, STATUTORY, OR OTHER. THIS INCLUDES, WITHOUT LIMITATION, WARRANTIES OF TITLE, MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, NON-INFRINGEMENT, ABSENCE OF LATENT OR OTHER DEFECTS, ACCURACY, OR THE PRESENCE OR ABSENCE OF ERRORS, WHETHER OR NOT KNOWN OR DISCOVERABLE. WHERE DISCLAIMERS OF WARRANTIES ARE NOT ALLOWED IN FULL OR IN PART, THIS DISCLAIMER MAY NOT APPLY TO YOU. b. TO THE EXTENT POSSIBLE, IN NO EVENT WILL THE LICENSOR BE LIABLE TO YOU ON ANY LEGAL THEORY (INCLUDING, WITHOUT LIMITATION, NEGLIGENCE) OR OTHERWISE FOR ANY DIRECT, SPECIAL, INDIRECT, INCIDENTAL, CONSEQUENTIAL, PUNITIVE, EXEMPLARY, OR OTHER LOSSES, COSTS, EXPENSES, OR DAMAGES ARISING OUT OF THIS PUBLIC LICENSE OR USE OF THE LICENSED MATERIAL, EVEN IF THE LICENSOR HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH LOSSES, COSTS, EXPENSES, OR DAMAGES. WHERE A LIMITATION OF LIABILITY IS NOT ALLOWED IN FULL OR IN PART, THIS LIMITATION MAY NOT APPLY TO YOU. c. The disclaimer of warranties and limitation of liability provided above shall be interpreted in a manner that, to the extent possible, most closely approximates an absolute disclaimer and waiver of all liability. Section 6 -- Term and Termination. a. This Public License applies for the term of the Copyright and Similar Rights licensed here. However, if You fail to comply with this Public License, then Your rights under this Public License terminate automatically. b. Where Your right to use the Licensed Material has terminated under Section 6(a), it reinstates: 1. automatically as of the date the violation is cured, provided it is cured within 30 days of Your discovery of the violation; or 2. upon express reinstatement by the Licensor. For the avoidance of doubt, this Section 6(b) does not affect any right the Licensor may have to seek remedies for Your violations of this Public License. c. For the avoidance of doubt, the Licensor may also offer the Licensed Material under separate terms or conditions or stop distributing the Licensed Material at any time; however, doing so will not terminate this Public License. d. Sections 1, 5, 6, 7, and 8 survive termination of this Public License. Section 7 -- Other Terms and Conditions. a. The Licensor shall not be bound by any additional or different terms or conditions communicated by You unless expressly agreed. b. Any arrangements, understandings, or agreements regarding the Licensed Material not stated herein are separate from and independent of the terms and conditions of this Public License. Section 8 -- Interpretation. a. For the avoidance of doubt, this Public License does not, and shall not be interpreted to, reduce, limit, restrict, or impose conditions on any use of the Licensed Material that could lawfully be made without permission under this Public License. b. To the extent possible, if any provision of this Public License is deemed unenforceable, it shall be automatically reformed to the minimum extent necessary to make it enforceable. If the provision cannot be reformed, it shall be severed from this Public License without affecting the enforceability of the remaining terms and conditions. c. No term or condition of this Public License will be waived and no failure to comply consented to unless expressly agreed to by the Licensor. d. Nothing in this Public License constitutes or may be interpreted as a limitation upon, or waiver of, any privileges and immunities that apply to the Licensor or You, including from the legal processes of any jurisdiction or authority. ======================================================================= Creative Commons is not a party to its public licenses. Notwithstanding, Creative Commons may elect to apply one of its public licenses to material it publishes and in those instances will be considered the “Licensor.” The text of the Creative Commons public licenses is dedicated to the public domain under the CC0 Public Domain Dedication. Except for the limited purpose of indicating that material is shared under a Creative Commons public license or as otherwise permitted by the Creative Commons policies published at creativecommons.org/policies, Creative Commons does not authorize the use of the trademark "Creative Commons" or any other trademark or logo of Creative Commons without its prior written consent including, without limitation, in connection with any unauthorized modifications to any of its public licenses or any other arrangements, understandings, or agreements concerning use of licensed material. For the avoidance of doubt, this paragraph does not form part of the public licenses. Creative Commons may be contacted at creativecommons.org. ================================================ FILE: MANIFEST.in ================================================ include nsynth/examples.json.gz ================================================ FILE: README.md ================================================ # SING: Symbol-to-Instrument Neural Generator SING is a deep learning based music notes synthetizer that can be trained on the [NSynth dataset][nsynth]. Despite being 32 times faster to train and 2,500 faster for inference, SING produces audio with significantly improved perceptual quality compared to the NSynth wavenet-like autoencoder [[1]](#ref_nsynth) as measured by Mean Opinion Scores based on human evaluations. The architecture and results obtained are detailed in our paper [SING: Symbol-to-Instrument Neural Generator][sing_nips]. SING is based on a LSTM based sequence generator and a convolutional decoder:

Schema representing the structure of SING. A LSTM is followed by a convolutional decoder

## Requirements SING works with python3.6 and newest. To use SING, you must have decently recent version of the following package installed: - numpy - requests - pytorch (needs to be >= 4.1.0 as we use torch.stft) - scipy - tqdm If you have anaconda installed, you can run from the root of this repository: conda env update conda activate sing This will create a `sing` environmnent with all the dependencies installed. Alternatively, you can use pip to install those: pip3 install -r requirements.txt SING can optionally be installed using the usual `setup.py` although this is not required. ### Obtaining the NSynth dataset If you want to train SING from scratch, you will need a copy of the NSynth dataset [[1]](#ref_nsynth). To download it, you use the following instructions (**WARNING**, NSynth is 30GB so this will take a bit of time): mkdir data && cd data &&\ wget http://download.magenta.tensorflow.org/datasets/nsynth/nsynth-train.jsonwav.tar.gz &&\ tar xf nsynth-train.jsonwav.tar.gz ## Using SING Once installed or from the root of this repository, you can use a family of commands detailed hereafter of the form python3 -m sing.* ### Common flags For either training or generation, use the `--cuda` flag for GPU acceleration and `--parallel` flag to use all available GPUs. Depending on the memory and number of GPUs available, consider tweaking the batch size using the `--batch-size` flag. The default is 64 but 256 was used in the paper. ### Training If you already have the NSynth dataset downloaded somewhere, run python3 -m sing.train [--cuda [--parallel]] --data PATH_TO_NSYNTH \ --output PATH_TO_SING_MODEL [--checkpoint PATH_TO_CHECKPOINTS] `PATH_TO_NSYNTH` is by default set to `data/nsynth-train`. The final model will be saved at `PATH_TO_SING_MODEL` (default is `models/sing.th`). If you want to save checkpoints after each epoch, or to resume a previously interrupted training, use the `--checkpoint` option. ### Generation For generation, you do not need the NSynth dataset but you should have a trained SING model. python3 -m sing.generate [--cuda [--parallel]] \ --model PATH_TO_SING_MODEL PATH_TO_ITEM_LIST `PATH_TO_ITEM_LIST` should be a file with one dataset item name per list, for instance `organ_electronic_044-055-127`. Alternatively, you can download a pretrained model using python3 -m sing.generate [--cuda [--parallel]] --dl PATH_TO_ITEM_LIST By default, the model will be downloaded under `models/sing.th` but a different path can be provided using the `--model` option. The pretrained model can be directly download [here](https://dl.fbaipublicfiles.com/sing/sing.th). ### Results reproduction To reproduce the results of Table 1 in our paper, simply run ```bash # For the L1 spectral losss python3 -m sing.train [--cuda [--parallel]] --l1 # For the L1 spectral loss without time embeddings python3 -m sing.train [--cuda [--parallel]] --l1 --time-dim=0 # For the Wav loss python3 -m sing.train [--cuda [--parallel]] --wav ``` To reproduce the audio samples used for the human evaluations, simply run from the root of the git repository python3 -m sing.generate [--cuda [--parallel]] --dl nsynth_100_test.txt The file `nsynth_100_test.txt` has been generated using the following code: ```python from sing import nsynth from sing.fondation.datasets import RandomSubset dset = nsynth.get_nsynth_metadata() train, valid, test = nsynth.make_datasets(dset) evaluation = RandomSubset(test, 100) open("nsynth_100_test.txt", "w").write("\n".join( evaluation[i].metadata['name'] for i in range(len(evaluation)))) ``` ## Generated audio A comparison of audio samples generated by SING and the NSynth Wavenet based autoencoder [[1]](#ref_nsynth) is available on [the paper webpage](https://research.fb.com/wp-content/themes/fb-research/research/sing-paper/). ## Thanks We thank the Magenta team for their inspiring work on NSynth. ## License For conveniance we have included a copy of the metadata of the NSynth dataset in this repository. The dataset has been released by Google Inc under the Creative Commons Attribution 4.0 International (CC BY 4.0) license. SING is released under Creative Commons Attribution 4.0 International (CC BY 4.0) license, as found in the LICENSE file. ## Bibliography [1]: Jesse Engel, Cinjon Resnick, Adam Roberts, Sander Dieleman, Douglas Eck, Karen Simonyan, and Mohammad Norouzi. [Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders](https://arxiv.org/pdf/1704.01279.pdf). 2017. [nsynth]: https://magenta.tensorflow.org/datasets/nsynth [sing_nips]: https://research.fb.com/publications/sing-symbol-to-instrument-neural-generator ================================================ FILE: environment.yml ================================================ name: sing channels: - pytorch dependencies: - numpy>=1.15 - python>=3.6 - pytorch>=0.4.1 - requests>=2.19 - scipy>=1.1 - tqdm>=4.26 ================================================ FILE: nsynth_100_test.txt ================================================ bass_synthetic_126-025-025 synth_lead_synthetic_006-045-050 bass_synthetic_065-070-025 keyboard_electronic_063-087-100 reed_acoustic_031-057-050 bass_synthetic_044-094-025 bass_synthetic_095-091-050 bass_synthetic_087-091-127 mallet_electronic_011-074-127 mallet_electronic_007-059-100 brass_acoustic_049-082-050 keyboard_electronic_000-091-075 string_acoustic_048-062-075 string_acoustic_008-033-075 keyboard_electronic_026-055-127 keyboard_electronic_070-049-025 organ_electronic_050-105-127 string_acoustic_044-060-025 bass_synthetic_064-092-075 organ_electronic_085-083-127 mallet_acoustic_013-071-075 keyboard_electronic_026-098-100 mallet_acoustic_002-060-127 keyboard_electronic_100-039-100 bass_synthetic_120-071-100 organ_electronic_061-044-127 vocal_acoustic_023-057-127 string_acoustic_030-051-025 brass_acoustic_003-041-050 bass_synthetic_093-052-075 organ_electronic_016-026-100 organ_electronic_080-025-100 brass_acoustic_001-054-127 guitar_acoustic_009-069-100 brass_acoustic_014-061-050 keyboard_acoustic_001-061-100 organ_electronic_077-047-100 bass_synthetic_130-056-075 guitar_electronic_032-081-127 mallet_synthetic_001-066-100 keyboard_electronic_042-057-050 bass_synthetic_052-059-025 keyboard_acoustic_002-062-075 guitar_acoustic_019-074-025 bass_synthetic_128-097-100 guitar_electronic_004-038-050 bass_synthetic_086-048-100 keyboard_electronic_055-068-127 guitar_electronic_023-046-100 guitar_electronic_015-065-025 bass_synthetic_015-098-127 brass_acoustic_035-058-127 bass_synthetic_063-036-025 reed_acoustic_033-044-050 organ_electronic_065-041-100 bass_synthetic_111-036-100 organ_electronic_037-047-050 bass_synthetic_140-108-050 brass_acoustic_040-059-025 organ_electronic_092-042-025 keyboard_electronic_019-099-025 reed_acoustic_029-077-127 string_acoustic_043-027-025 bass_synthetic_121-043-127 string_acoustic_007-060-025 keyboard_electronic_015-089-050 organ_electronic_050-057-025 bass_synthetic_078-045-075 keyboard_electronic_088-081-075 brass_acoustic_000-039-127 guitar_electronic_017-093-075 bass_synthetic_117-068-127 mallet_electronic_013-087-050 flute_acoustic_012-090-127 bass_synthetic_136-073-025 mallet_electronic_006-085-127 mallet_acoustic_016-082-050 organ_electronic_044-025-025 mallet_acoustic_046-032-127 guitar_acoustic_032-067-100 organ_electronic_014-084-127 organ_electronic_098-030-025 mallet_acoustic_042-103-050 keyboard_electronic_065-093-127 mallet_acoustic_058-108-050 mallet_electronic_007-036-127 keyboard_electronic_092-090-127 string_acoustic_059-070-025 guitar_electronic_009-054-100 bass_synthetic_044-075-050 mallet_acoustic_051-065-075 bass_electronic_030-047-100 flute_acoustic_027-077-050 bass_synthetic_123-086-050 bass_synthetic_117-086-100 mallet_acoustic_004-065-100 bass_synthetic_094-023-025 organ_electronic_020-054-025 brass_acoustic_011-056-050 keyboard_electronic_058-038-127 ================================================ FILE: requirements.txt ================================================ numpy requests scipy torch>=0.4.1 tqdm ================================================ FILE: setup.py ================================================ #!/usr/bin/env python # -*- coding: utf-8 -*- # Copyright (c) 2018-present, Facebook, Inc. # All rights reserved. # # This source code is licensed under the license found in the # LICENSE file in the root directory of this source tree. # Inspired from https://github.com/kennethreitz/setup.py from pathlib import Path from setuptools import find_packages, setup NAME = 'sing' DESCRIPTION = 'SING: Symbol-to-Instrument Neural Generator' URL = 'https://github.com/facebookresearch/SING' EMAIL = 'defossez@fb.com' AUTHOR = 'Alexandre Defossez' REQUIRES_PYTHON = '>=3.6.0' VERSION = "1.0" HERE = Path(__file__).parent REQUIRED = [i.strip() for i in open(HERE / "requirements.txt").readlines()] try: with open(HERE / "README.md", encoding='utf-8') as f: long_description = '\n' + f.read() except FileNotFoundError: long_description = DESCRIPTION setup( name=NAME, version=VERSION, description=DESCRIPTION, long_description=long_description, long_description_content_type='text/markdown', author=AUTHOR, author_email=EMAIL, python_requires=REQUIRES_PYTHON, url=URL, packages=find_packages(), install_requires=REQUIRED, include_package_data=True, license='Creative Common Attribution-NonCommercial 4.0 International', classifiers=[ # Trove classifiers # Full list: https://pypi.python.org/pypi?%3Aaction=list_classifiers 'Programming Language :: Python', 'Programming Language :: Python :: 3', 'Programming Language :: Python :: 3.6', 'Programming Language :: Python :: Implementation :: CPython', 'Topic :: Scientific/Engineering :: Artificial Intelligence', ], ) ================================================ FILE: sing/__init__.py ================================================ # -*- coding: utf-8 -*- # Copyright (c) 2018-present, Facebook, Inc. # All rights reserved. # # This source code is licensed under the license found in the # LICENSE file in the root directory of this source tree. # ================================================ FILE: sing/ae/__init__.py ================================================ # -*- coding: utf-8 -*- # Copyright (c) 2018-present, Facebook, Inc. # All rights reserved. # # This source code is licensed under the license found in the # LICENSE file in the root directory of this source tree. # ================================================ FILE: sing/ae/models.py ================================================ # -*- coding: utf-8 -*- # Copyright (c) 2018-present, Facebook, Inc. # All rights reserved. # # This source code is licensed under the license found in the # LICENSE file in the root directory of this source tree. # from torch import nn from .utils import WindowedConv1d, WindowedConvTranpose1d class ConvolutionalDecoder(nn.Module): """ Convolutional decoder that takes a downsampled embedding and turns it into a waveform. Together with :class:`ConvolutionalEncoder`, it forms a :class:`ConvolutionalAE` Arguments: channels (int): number of channels accross all the inner layers stride (int): stride of the final :class:`nn.ConvTranspose1d` dimension (int): dimension of the embedding kernel_size (int): size of the kernel of the final :class:`nn.ConvTranspose1d` context_size (int): kernel size of the first convolution, this is called a context as one can see it as providing information about the previous and following embeddings rewrite_layers (int): after the first convolution, perform `rewrite_layers` `1x1` convolutions window_name (str or None): name of the window used to smooth the convolutions. See :func:`sing.dsp.get_window` squared_window (bool): if `True`, square the smoothing window """ def __init__(self, channels=4096, stride=256, dimension=128, kernel_size=1024, context_size=9, rewrite_layers=2, window_name="hann", squared_window=True): super(ConvolutionalDecoder, self).__init__() layers = [] layers.extend([ nn.Conv1d( in_channels=dimension, out_channels=channels, kernel_size=context_size), nn.ReLU() ]) for rewrite in range(rewrite_layers): layers.extend([ nn.Conv1d( in_channels=channels, out_channels=channels, kernel_size=1), nn.ReLU() ]) conv_tr = nn.ConvTranspose1d( in_channels=channels, out_channels=1, kernel_size=kernel_size, stride=stride, padding=kernel_size - stride) if window_name is not None: conv_tr = WindowedConvTranpose1d(conv_tr, window_name, squared_window) layers.append(conv_tr) self.layers = nn.Sequential(*layers) self.context_size = context_size self.stride = stride self.kernel_size = kernel_size self.strip = kernel_size - stride + (context_size - 1) * stride // 2 def __repr__(self): return "ConvolutionalDecoder({})".format(repr(self.layers)) def forward(self, embeddings): return self.layers.forward(embeddings).squeeze(1) def wav_length(self, embedding_length): """ Given an embedding of a certain size `embedding_length`, returns the length of the wav that would be generated from it. """ return (embedding_length - self.context_size + 2 ) * self.stride - self.kernel_size def embedding_length(self, wav_length): """ Return the embedding length necessary to generate a wav of length `wav_length`. """ return self.context_size - 2 + ( wav_length + self.kernel_size) // self.stride class ConvolutionalEncoder(nn.Module): """ Convolutional encoder that takes a waveform and turns it into a downsampled embedding. Together with :class:`ConvolutionalDecoder`, it forms a :class:`ConvolutionalAE` Arguments: channels (int): number of channels accross all the inner layers stride (int): stride of the initial :class:`nn.Conv1d` dimension (int): dimension of the embedding kernel_size (int): size of the kernel of the initial :class:`nn.Conv1d` rewrite_layers (int): after the first convolution, perform `rewrite_layers` `1x1` convolutions. window_name (str or None): name of the window used to smooth the convolutions. See :func:`sing.dsp.get_window` squared_window (bool): if `True`, square the smoothing window """ def __init__(self, channels=4096, stride=256, dimension=128, kernel_size=1024, rewrite_layers=2, window_name="hann", squared_window=True): super(ConvolutionalEncoder, self).__init__() layers = [] conv = nn.Conv1d( in_channels=1, out_channels=channels, kernel_size=kernel_size, stride=stride) if window_name is not None: conv = WindowedConv1d(conv, window_name, squared_window) layers.extend([conv, nn.ReLU()]) for rewrite in range(rewrite_layers): layers.extend([ nn.Conv1d( in_channels=channels, out_channels=channels, kernel_size=1), nn.ReLU() ]) layers.append( nn.Conv1d( in_channels=channels, out_channels=dimension, kernel_size=1)) self.layers = nn.Sequential(*layers) def __repr__(self): return "ConvolutionalEncoder({!r})".format(self.layers) def forward(self, signal): return self.layers.forward(signal.unsqueeze(1)) class ConvolutionalAE(nn.Module): """ Convolutional autoencoder made from :class:`ConvolutionalEncoder` and :class:`ConvolutionalDecoder`. Arguments: channels (int): number of channels accross all the inner layers stride (int): downsampling stride going from the waveform to the embedding dimension (int): dimension of the embedding kernel_size (int): kernel size of the initial convolution and last conv transpose context_size (int): kernel size of the first convolution of the decoder rewrite_layers (int): after the first convolution, perform `rewrite_layers` `1x1` convolutions, both in the encoder and decoder. window_name (str or None): name of the window used to smooth the convolutions. See :func:`sing.dsp.get_window` squared_window (bool): if `True`, square the smoothing window """ def __init__(self, channels=4096, stride=256, dimension=128, kernel_size=1024, context_size=9, rewrite_layers=2, window_name="hann", squared_window=True): super(ConvolutionalAE, self).__init__() self.encoder = ConvolutionalEncoder( channels=channels, stride=stride, dimension=dimension, kernel_size=kernel_size, rewrite_layers=rewrite_layers, window_name=window_name, squared_window=squared_window) self.decoder = ConvolutionalDecoder( channels=channels, stride=stride, dimension=dimension, kernel_size=kernel_size, context_size=context_size, rewrite_layers=rewrite_layers, window_name=window_name, squared_window=squared_window) print(self) def encode(self, signal): """ Returns the embedding for the waveform `signal`. """ return self.encoder.forward(signal) def decode(self, embeddings): """ Return the waveforms from `embeddings` """ return self.decoder.forward(embeddings) def forward(self, signal): return self.decode(self.encode(signal)) def __repr__(self): return "ConvolutionalAE(encoder={!r},decoder={!r})".format( self.encoder, self.decoder) ================================================ FILE: sing/ae/trainer.py ================================================ # -*- coding: utf-8 -*- # Copyright (c) 2018-present, Facebook, Inc. # All rights reserved. # # This source code is licensed under the license found in the # LICENSE file in the root directory of this source tree. # from ..fondation import utils, trainer class AutoencoderTrainer(trainer.BaseTrainer): """ Trainer for the autoencoder. """ def _train_batch(self, batch): rebuilt, target = self._get_rebuilt_target(batch) self.optimizer.zero_grad() loss = self.train_loss(rebuilt, target) loss.backward() self.optimizer.step() return loss.item() def _get_rebuilt_target(self, batch): wav = batch.tensors['wav'] target = utils.unpad1d(wav, self.model.decoder.strip) rebuilt = self.parallel.forward(wav) return rebuilt, target ================================================ FILE: sing/ae/utils.py ================================================ # -*- coding: utf-8 -*- # Copyright (c) 2018-present, Facebook, Inc. # All rights reserved. # # This source code is licensed under the license found in the # LICENSE file in the root directory of this source tree. # from torch import nn from torch.nn import functional as F from .. import dsp class WindowedConv1d(nn.Module): """ Smooth a convolution using a window. Arguments: conv (nn.Conv1d): convolution module to wrap window_name (str or None): name of the window used to smooth the convolutions. See :func:`sing.dsp.get_window` squared (bool): if `True`, square the smoothing window """ def __init__(self, conv, window_name='hann', squared=True): super(WindowedConv1d, self).__init__() self.window_name = window_name if squared: self.window_name += "**2" self.register_buffer('window', dsp.get_window( window_name, conv.weight.size(-1), squared=squared)) self.conv = conv def forward(self, input): weight = self.window * self.conv.weight return F.conv1d( input, weight, bias=self.conv.bias, stride=self.conv.stride, dilation=self.conv.dilation, groups=self.conv.groups, padding=self.conv.padding) def __repr__(self): return "WindowedConv1d(window={},conv={})".format( self.window_name, self.conv) class WindowedConvTranpose1d(nn.Module): """ Smooth a transposed convolution using a window. Arguments: conv (nn.Conv1d): convolution module to wrap window_name (str or None): name of the window used to smooth the convolutions. See :func:`sing.dsp.get_window` squared (bool): if `True`, square the smoothing window """ def __init__(self, conv_tr, window_name='hann', squared=True): super(WindowedConvTranpose1d, self).__init__() self.window_name = window_name if squared: self.window_name += "**2" self.register_buffer('window', dsp.get_window( window_name, conv_tr.weight.size(-1), squared=squared)) self.conv_tr = conv_tr def forward(self, input): weight = self.window * self.conv_tr.weight return F.conv_transpose1d( input, weight, bias=self.conv_tr.bias, stride=self.conv_tr.stride, padding=self.conv_tr.padding, output_padding=self.conv_tr.output_padding, groups=self.conv_tr.groups, dilation=self.conv_tr.dilation) def __repr__(self): return "WindowedConvTranpose1d(window={},conv_tr={})".format( self.window_name, self.conv_tr) ================================================ FILE: sing/dsp.py ================================================ # -*- coding: utf-8 -*- # Copyright (c) 2018-present, Facebook, Inc. # All rights reserved. # # This source code is licensed under the license found in the # LICENSE file in the root directory of this source tree. # import torch from torch import nn import torch.nn.functional as F def power(spec): """ Given a complex spectrogram, return the power spectrum. Shape: - `spec`: `(*, 2, F, T)` - Output: `(*, F, T)` """ return spec[..., 0]**2 + spec[..., 1]**2 def get_window(name, window_length, squared=False): """ Returns a windowing function. Arguments: window (str): name of the window, currently only 'hann' is available window_length (int): length of the window squared (bool): if true, square the window Returns: torch.FloatTensor: window of size `window_length` """ if name == "hann": window = torch.hann_window(window_length) else: raise ValueError("Invalid window name {}".format(name)) if squared: window *= window return window class STFT(nn.Module): """ Compute the STFT. See :mod:`torch.stft` for a definition of the parameters. Arguments: n_fft (int): performs a FFT over `n_fft` samples hop_length (int or None): stride of the STFT transform. If `None` uses `n_fft // 4` window_name (str or None): name of the window used for the STFT. No window is used if `None`. """ def __init__(self, n_fft=1024, hop_length=None, window_name='hann'): super(STFT, self).__init__() assert n_fft % 2 == 0 window = None if window_name is not None: window = get_window(window_name, n_fft) self.register_buffer("window", window) self.hop_length = hop_length or n_fft // 4 self.n_fft = n_fft def forward(self, input): return torch.stft( input, window=self.window, n_fft=self.n_fft, hop_length=self.hop_length, center=False) class SpectralLoss(nn.Module): """ Compute a loss between two log power-spectrograms. Arguments: base_loss (function): loss used to compare the log power-spectrograms. For instance :func:`F.mse_loss` epsilon (float): offset for the log, i.e. `log(epsilon + ...)` **kwargs (dict): see :class:`STFT` """ def __init__(self, base_loss=F.mse_loss, epsilon=1, **kwargs): super(SpectralLoss, self).__init__() self.base_loss = base_loss self.epsilon = epsilon self.stft = STFT(**kwargs) def _log_spectrogram(self, signal): return torch.log(self.epsilon + power(self.stft.forward(signal))) def forward(self, a, b): spec_a = self._log_spectrogram(a) spec_b = self._log_spectrogram(b) return self.base_loss(spec_a, spec_b) def float_wav_to_short(wav): """ Given a float waveform, return a short waveform. The input waveform will be clamped between -1 and 1. """ return (wav.clamp(-1, 1) * (2**15 - 1)).short() ================================================ FILE: sing/fondation/__init__.py ================================================ # -*- coding: utf-8 -*- # Copyright (c) 2018-present, Facebook, Inc. # All rights reserved. # # This source code is licensed under the license found in the # LICENSE file in the root directory of this source tree. # ================================================ FILE: sing/fondation/batch.py ================================================ # -*- coding: utf-8 -*- # Copyright (c) 2018-present, Facebook, Inc. # All rights reserved. # # This source code is licensed under the license found in the # LICENSE file in the root directory of this source tree. # from torch.utils.data.dataloader import default_collate class BatchItem: """ Reprensents a single batch item. A :class:`Batch` can be built from multiple :class:`BatchItem` using :func:`collate`. Attributes: metadata (dict[str, object]): Contains all the metadata about the batch item. Those elements will not be collated together when building a batch tensors (dict[str, tensor]): Contains all the tensors for the batch item. Those elements will be collated together when building a batch using :func:`default_collate`. """ def __init__(self, metadata=None, tensors=None): self.metadata = dict(metadata) if metadata else {} self.tensors = dict(tensors) if tensors else {} def collate(items): """ Collate together all the items into a :class:`Batch`. The metadata dictionaries will be added to a list and the tensors will be collated using :func:`torch.utils.data.dataloader.default_collate`. Args: items (list[BatchItem]): list of the items in the batch Returns: Batch: a batch made from `items`. """ metadata = [item.metadata for item in items] tensors = default_collate([item.tensors for item in items]) return Batch(metadata=metadata, tensors=tensors) class Batch: """ Represents a batch. Supports iteration (yields individual :class:`BatchItem`) and indexing. Slice indexing will return another :class:`Batch`. Attributes: metadata (list[dict[str, object]]): a list of dictionaries for each element in the batch. Each dictionary contains information about the corresponding item. tensors (dict[str, tensor]): a dictionary of collated tensors. The first dimension of each tensor will always be `B`, the batch size. """ def __init__(self, metadata, tensors): self.metadata = metadata self.tensors = tensors def __len__(self): return len(self.metadata) def __iter__(self): for i in range(len(self)): yield self[i] def __getitem__(self, index): if isinstance(index, slice): if index.step is not None: raise IndexError("Does not support slice with step") metadata = self.metadata[index] tensors = { name: tensor[index] for name, tensor in self.tensors.items() } return Batch(metadata=metadata, tensors=tensors) else: return BatchItem( metadata=self.metadata[index], tensors={ name: tensor[index] for name, tensor in self.tensors.items() }) def apply(self, function): """ Apply a function to all tensors. Arguments: function: callable to be applied to all tensors. Returns: Batch: A new batch """ tensors = { name: function(tensor) for name, tensor in self.tensors.items() } return Batch(metadata=self.metadata, tensors=tensors) def apply_(self, function): """ Inplace variance of :meth:`apply`. """ other = self.apply(function) self.tensors = other.tensors return self def cuda(self, *args, **kwargs): """ Returns a new batch on GPU. """ return self.apply(lambda x: x.cuda()) def cuda_(self, *args, **kwargs): """ Move the batch inplace to GPU. """ return self.apply_(lambda x: x.cuda()) def cpu(self, *args, **kwargs): """ Returns a new batch on CPU. """ return self.apply(lambda x: x.cpu()) def cpu_(self, *args, **kwargs): """ Move the batch inplace to CPU. """ return self.apply_(lambda x: x.cpu_()) ================================================ FILE: sing/fondation/datasets.py ================================================ # -*- coding: utf-8 -*- # Copyright (c) 2018-present, Facebook, Inc. # All rights reserved. # # This source code is licensed under the license found in the # LICENSE file in the root directory of this source tree. # import random import torch from .utils import random_seed_manager class DatasetSubset: """ Represents a subset of a dataset. Arguments: dataset (Dataset): dataset to take a subset of. indexes (list[int]): list of indexes to keep. """ def __init__(self, dataset, indexes): self.dataset = dataset self.indexes = torch.LongTensor(indexes) def __len__(self): return len(self.indexes) def __getitem__(self, index): return self.dataset[self.indexes[index]] class RandomSubset(DatasetSubset): """ A random subset of a given size built from another dataset. Arguments: dataset (Dataset): dataset to take a random subset of. size (int): size of the random subset random_seed (int): random seed used to select the indexes. """ def __init__(self, dataset, size, random_seed=42): indexes = list(range(len(dataset))) with random_seed_manager(random_seed): random.shuffle(indexes) super(RandomSubset, self).__init__( dataset=dataset, indexes=indexes[:size]) ================================================ FILE: sing/fondation/trainer.py ================================================ # -*- coding: utf-8 -*- # Copyright (c) 2018-present, Facebook, Inc. # All rights reserved. # # This source code is licensed under the license found in the # LICENSE file in the root directory of this source tree. # import torch from torch import nn from torch import optim from torch.utils.data import DataLoader import tqdm from . import utils from .batch import collate class BaseTrainer: """ Base class for all the epoch-based trainers. Takes care of various task common to all training like checkpointing, iterating over the different datasets, computing evaluation metrics etc. Arguments: model (nn.Module): model to train train_loss (nn.Module): loss used for training eval_losses (dict[str, nn.Module]): dictionary of evaluation losses train_dataset (Dataset): dataset used for training eval_datasets (dict[str, Dataset]): dictionary of datasets on which each evaluation loss will be computed epochs (int): number of epochs to train for suffix (str): suffix used for logging, for instance if the suffix is `"_phase1"`, during training, `"train_phase1"` will be displayed batch_size (int): batch size cuda (bool): if true, runs on GPU parallel (bool): if true, use all available GPUs lr (float): learning rate for :class:`optim.Adam` checkpoint_path (Path): path to save checkpoint to. If `None`, no checkpointing is performed. Otherwise, a checkpoint is saved at the end of each epoch and overwrites the previous one. """ def __init__(self, model, train_loss, eval_losses, train_dataset, eval_datasets, epochs, suffix="", batch_size=32, cuda=True, parallel=False, lr=0.0001, checkpoint_path=None): self.model = model self.parallel = nn.DataParallel(model) if parallel else model self.is_parallel = parallel self.train_loss = train_loss self.eval_losses = nn.ModuleDict(eval_losses) self.batch_size = batch_size self.cuda = cuda self.suffix = suffix self.train_dataset = train_dataset self.eval_datasets = eval_datasets self.epochs = epochs self.checkpoint_path = checkpoint_path parameters = [p for p in self.model.parameters() if p.requires_grad] self.optimizer = optim.Adam(parameters, lr=lr) if self.cuda: self.model.cuda() self.train_loss.cuda() self.eval_losses.cuda() else: self.model.cpu() self.train_loss.cpu() self.eval_losses.cpu() def _train_epoch(self, dataset, epoch): """ Train a single epoch on the given dataset and displays statistics from time to time. """ loader = DataLoader( dataset, batch_size=self.batch_size, shuffle=True, collate_fn=collate) iterator = utils.progress_iterator(loader, divisions=20) total_loss = 0 with tqdm.tqdm(total=len(dataset), unit="ex") as bar: for idx, (progress, batch) in enumerate(iterator): if self.cuda: batch.cuda_() total_loss += self._train_batch(batch) bar.update(len(batch)) if progress: tqdm.tqdm.write( "[train{}][{:03d}] {:.1f}%, loss {:.6f}".format( self.suffix, epoch, progress, total_loss / (idx + 1))) return total_loss def _eval_dataset(self, dataset_name, dataset, epoch): """ Evaluate all the losses `eval_lossers` on the given dataset and reports the metrics averaged over the entire dataset. """ loader = DataLoader( dataset, batch_size=self.batch_size, collate_fn=collate) total_losses = {loss_name: 0 for loss_name in self.eval_losses} with tqdm.tqdm(total=len(dataset), unit="ex") as bar: for batch in loader: if self.cuda: batch.cuda_() rebuilt, target = self._get_rebuilt_target(batch) for name, loss in self.eval_losses.items(): total_losses[name] += loss(rebuilt, target).item() * len(batch) bar.update(len(batch)) print("[{}{}][{:03d}] Evaluation: \n{}\n".format( dataset_name, self.suffix, epoch, "\n".join( "\t{}={:.6f}".format(name, loss / len(dataset)) for name, loss in total_losses.items()))) return total_losses def _train_batch(self, batch): """ Given a batch, call :meth:`_get_rebuilt_target` to obtain the `target` and `rebuilt` tensors and call :attr:`train_loss` on them, compute the gradient and perform one optimizer step. This method can be overriden in subclasses. """ rebuilt, target = self._get_rebuilt_target(batch) self.optimizer.zero_grad() loss = self.train_loss(rebuilt, target) loss.backward() self.optimizer.step() return loss.item() def _get_rebuilt_target(self, batch): """ Should be implemenented in subclasses. Given a batch, returns a tuple (rebuilt, target). This tuple will be passed to all the losses in `eval_losses`. """ raise NotImplementedError() def train(self): """ Train :attr:`model` for :attr:`epochs` """ last_epoch, state = utils.load_checkpoint(self.checkpoint_path) if state is not None: self.model.load_state_dict(state, strict=False) start_epoch = last_epoch + 1 if start_epoch > self.epochs: raise ValueError(("Checkpoint has been trained for {} " "epochs but we aim for {} epochs").format( start_epoch, self.epochs)) if start_epoch > 0: print("Resuming training at epoch {}".format(start_epoch)) for epoch in range(start_epoch, self.epochs): self._train_epoch(self.train_dataset, epoch) utils.save_checkpoint(self.checkpoint_path, epoch, self.model.state_dict()) with torch.no_grad(): for name, dataset in self.eval_datasets.items(): self._eval_dataset(name, dataset, epoch) ================================================ FILE: sing/fondation/utils.py ================================================ # -*- coding: utf-8 -*- # Copyright (c) 2018-present, Facebook, Inc. # All rights reserved. # # This source code is licensed under the license found in the # LICENSE file in the root directory of this source tree. # import contextlib import hashlib from pathlib import Path import random import requests import sys import torch import tqdm @contextlib.contextmanager def random_seed_manager(seed): """ Context manager that will save the python RNG state, set the seed to `seed` and on exit, set back the python RNG state. """ state = random.getstate() try: random.seed(seed) yield None finally: random.setstate(state) def progress_iterator(iterator, divisions=100): """ Wraps an iterator of known length and yield a tuple `(progress, item)` for each `item` in `iterator`. `progress` will be None except `divisions` times that are evenly spaced. When `progress` is not None it will contain the current percentage of items that have been seen. Arguments: iterator (iterator): source iterator, should support :func:`len`. divisions (int): progress will be every 1/divisions of the iterator length. Examples:: >>> for (progress, element) in progress_iterator(range(500)): ... if progress: ... print("{:.0f}% done".format(progress)) """ length = len(iterator) division_width = length / divisions next_division = division_width for idx, element in enumerate(iterator): progress = None if (idx + 1) >= next_division or idx + 1 == length: next_division += division_width progress = (idx + 1) / length * 100 yield progress, element def unpad1d(tensor, pad): """ Opposite of padding, will remove `pad` items on each side of the last dimension of `tensor`. Arguments: tensor (tensor): tensor to unpad pad (int): amount of padding to remove on each side. """ if pad > 0: return tensor[..., pad:-pad] return tensor def load_checkpoint(path): """ Arguments: path (str or Path): path to load Returns: (int, object): returns a tuple (epoch, state). """ if path is None or not Path(path).exists(): return -1, None return torch.load(path) def save_checkpoint(path, epoch, state): """ Save a new checkpoint. A temporary file is created and then renamed to the target path. Arguments: path (str or Path): path to write to epoch (int): current epoch state (object): state to save """ if path is None: return path = Path(path) tmp_path = path.parent / (path.name + ".tmp") torch.save((epoch, state), str(tmp_path)) tmp_path.rename(path) def download_file(target, url, sha256=None): """ Download a file with a progress bar. Arguments: target (Path): target path to write to url (str): url to download sha256 (str or None): expected sha256 hexdigest of the file """ response = requests.get(url, stream=True) total_length = int(response.headers.get('content-length', 0)) if sha256 is not None: sha = hashlib.sha256() update = sha.update else: update = lambda x: None with tqdm.tqdm(total=total_length, unit="B", unit_scale=True) as bar: with open(target, "wb") as output: for data in response.iter_content(chunk_size=4096): output.write(data) update(data) bar.update(len(data)) if sha256 is not None: signature = sha.hexdigest() if sha256 != signature: target.unlink() raise ValueError("Invalid sha256 signature when downloading {}. " "Expected {} but got {}".format( url, sha256, signature)) def fatal(message, error_code=1): """ Print `message` to stderr and exit with the code `error_code`. """ print(message, file=sys.stderr) sys.exit(1) ================================================ FILE: sing/generate.py ================================================ # -*- coding: utf-8 -*- # Copyright (c) 2018-present, Facebook, Inc. # All rights reserved. # # This source code is licensed under the license found in the # LICENSE file in the root directory of this source tree. # import argparse from pathlib import Path import sys from scipy.io import wavfile import torch from torch import nn from torch.utils.data import DataLoader import tqdm from . import dsp, nsynth from .fondation.batch import collate from .fondation.datasets import DatasetSubset from .fondation import utils from .sequence.models import download_pretrained_model def get_parser(): parser = argparse.ArgumentParser( "sing.generate", description="Generate audio samples from a trained SING model", formatter_class=argparse.ArgumentDefaultsHelpFormatter) parser.add_argument( "--model", type=Path, default="models/sing.th", help="Path to the trained SING model as outputted by sing.main") parser.add_argument( "--dl", action="store_true", help="Download if necessary a pretrained SING model.") parser.add_argument( "--output", type=Path, default="generated", help="Path where the generated samples will be saved") parser.add_argument( "--metadata", default=nsynth.get_metadata_path(), type=Path, help="path to the dataset metadata file") parser.add_argument( "list", type=Path, help="File containing a list of names from the nsynth dataset. " "Those notes will be generated by SING") parser.add_argument( "--batch-size", type=int, default=32, help="Batch size") parser.add_argument("--cuda", action="store_true", help="Use cuda") parser.add_argument( "--parallel", action="store_true", help="Use multiple gpus") parser.add_argument( "--unpad", default=512, type=int, help="Amount of unpadding to perform") return parser def main(): args = get_parser().parse_args() if not args.model.exists(): if args.dl: print("Downloading pretrained SING model") args.model.parent.mkdir(parents=True, exist_ok=True) download_pretrained_model(args.model) else: utils.fatal("No model found for path {}. To download " "a pretrained model, use --dl".format(args.model)) elif args.dl: print( "WARNING: --dl is set but {} already exist.".format(args.model), file=sys.stderr) model = torch.load(args.model) if args.cuda: model.cuda() if args.parallel: model = nn.DataParallel(model) args.output.mkdir(exist_ok=True, parents=True) dataset = nsynth.NSynthMetadata(args.metadata) names = [name.strip() for name in open(args.list)] indexes = [dataset.names.index(name) for name in names] to_generate = DatasetSubset(dataset, indexes) loader = DataLoader( to_generate, batch_size=args.batch_size, collate_fn=collate) with tqdm.tqdm(total=len(to_generate), unit="ex") as bar: for batch in loader: if args.cuda: batch.cuda_() with torch.no_grad(): rebuilt = model.forward(**batch.tensors) rebuilt = utils.unpad1d(rebuilt, args.unpad) for metadata, wav in zip(batch.metadata, rebuilt): path = args.output / (metadata['name'] + ".wav") wavfile.write( str(path), metadata['sample_rate'], dsp.float_wav_to_short(wav).cpu().detach().numpy()) bar.update(len(batch)) if __name__ == "__main__": main() ================================================ FILE: sing/nsynth/__init__.py ================================================ # -*- coding: utf-8 -*- # Copyright (c) 2018-present, Facebook, Inc. # All rights reserved. # # This source code is licensed under the license found in the # LICENSE file in the root directory of this source tree. # from collections import defaultdict import gzip import json from pathlib import Path import random from scipy.io import wavfile import torch from torch.nn import functional as F from ..fondation.batch import BatchItem from ..fondation.datasets import DatasetSubset from ..fondation import utils class NSynthMetadata: """ NSynth metadata without the wavforms. Arguments: path (Path): path to the NSynth dataset. This path should contain a `examples.json` file. An item of the nsynth metadata dataset will contain the follow tensors: - instrument (LongTensor) - pitch (LongTensor) - velocity (LongTensor) - instrument_family (LongTensor) - index (LongTensor) Attributes: cardinalities (dict[str, int]): cardinality of instrument, instrument_family, pitch and velocity instruments (dict[str, int]): mapping from instrument name to instrument index """ _json_cache = {} _FEATURES = ['instrument', 'instrument_family', 'pitch', 'velocity'] def _map_velocity(self, metadata): velocity_mapping = { 25: 0, 50: 1, 75: 2, 100: 4, 127: 5, } for meta in self._metadata.values(): meta["velocity"] = velocity_mapping[meta['velocity']] def __init__(self, path): self.path = Path(path) # Cache the json to avoid reparsing it everytime if self.path in self._json_cache: self._metadata = self._json_cache[self.path] else: if self.path.suffix == ".gz": file = gzip.open(self.path) else: file = open(self.path, "rb") self._metadata = json.load(file) self._map_velocity(self._metadata) self._json_cache[self.path] = self._metadata self.names = sorted(self._metadata.keys()) # Compute the mapping instrument_name -> instrument id self.instruments = {} for meta in self._metadata.values(): self.instruments[meta["instrument_str"]] = meta["instrument"] # Compute the cardinality for the features velocity, instrument, # pitch and instrument_family self.cardinalities = {} for feature in self._FEATURES: self.cardinalities[feature] = 1 + max( i[feature] for i in self._metadata.values()) def __len__(self): return len(self.names) def __getitem__(self, index): if hasattr(index, "item"): index = index.item() name = self.names[index] metadata = self._metadata[name] tensors = {} metadata['name'] = name metadata['index'] = index for feature in self._FEATURES: tensors[feature] = torch.LongTensor([metadata[feature]]) return BatchItem(metadata=metadata, tensors=tensors) class NSynthDataset: """ NSynth dataset. Arguments: path (Path): path to the NSynth dataset. This path should contain a `examples.json` file and an `audio` folder containing the wav files. pad (int): amount of padding to add to the waveforms. Items from this dataset will contain all the information coming from :class:`NSynthMetadata` as well as a `'wav'` tensor containing the waveform. Attributes: metadata (NSynthMetadata): metadata only dataset """ def __init__(self, path, pad=0): self.metadata = NSynthMetadata(Path(path) / "examples.json") self.pad = pad def __len__(self): return len(self.metadata) def __getitem__(self, index): item = self.metadata[index] path = self.metadata.path.parent / "audio" / "{}.wav".format( item.metadata['name']) item.metadata['path'] = path _, wav = wavfile.read(str(path), mmap=True) wav = torch.as_tensor(wav, dtype=torch.float) wav /= 2**15 - 1 item.tensors['wav'] = F.pad(wav, (self.pad, self.pad)) return item def make_datasets(dataset, valid_ratio=0.1, test_ratio=0.1, random_seed=42): """ Take the original NSynth training dataset and split it into a train, valid and test set making sure that for a given instrument, a pitch is present in only one dataset (each pair of instrument and pitch has multiple occurences, one for each velocity). """ per_pitch_instrument = defaultdict(list) if isinstance(dataset, NSynthDataset): metadata = dataset.metadata elif isinstance(dataset, NSynthMetadata): metadata = dataset else: raise ValueError( "Invalid dataset {}, should be an instance of " "either NSynthDataset or NSynthMetadata.".format(dataset)) for index in range(len(metadata)): item = metadata[index] per_pitch_instrument[(item.metadata['instrument'], item.metadata['pitch'])].append(index) with utils.random_seed_manager(random_seed): train = [] valid = [] test = [] for indexes in per_pitch_instrument.values(): score = random.random() if score < valid_ratio: valid.extend(indexes) elif score < valid_ratio + test_ratio: test.extend(indexes) else: train.extend(indexes) return DatasetSubset(dataset, train), DatasetSubset( dataset, valid), DatasetSubset(dataset, test) def get_metadata_path(): """ Get the path to the nsynth-train metadata included with SING. """ return Path(__file__).parent / "examples.json.gz" def get_nsynth_metadata(): return NSynthMetadata(get_metadata_path()) ================================================ FILE: sing/parser.py ================================================ # -*- coding: utf-8 -*- # Copyright (c) 2018-present, Facebook, Inc. # All rights reserved. # # This source code is licensed under the license found in the # LICENSE file in the root directory of this source tree. # import argparse from pathlib import Path def get_parser(): """ Returns: argparse.ArgumentParser: parser with all the options for the training of a SING model. """ parser = argparse.ArgumentParser( "sing.train", description="Train a SING model on the NSynth dataset", formatter_class=argparse.ArgumentDefaultsHelpFormatter) # Datasets arguments parser.add_argument( "--data", default="data/nsynth-train", type=Path, help="path to the dataset, e.g. .../nsynth-train") parser.add_argument( "--pad", type=int, default=2304, help="Extra padding added to the waveforms", ) # Loss arguments parser.add_argument("--wav", action="store_true", help="Use a Wav loss") parser.add_argument( "--epsilon", default=1, type=float, help="Offset for power spectrum before taking the log") parser.add_argument( "--l1", action="store_true", help="Use L1 loss instead of mse", ) # Misc arguments parser.add_argument("--cuda", action="store_true", help="Use cuda") parser.add_argument( "--parallel", action="store_true", help="Use multiple gpus") parser.add_argument( "--checkpoint", type=Path, default=None, help="Path to the checkpoint folder") parser.add_argument( "--output", type=Path, default="models/sing.th", help="Path to output final SING model") parser.add_argument( "-d", "--debug", action="store_true", help="Debug flag") parser.add_argument( "-f", "--debug-fast", action="store_true", help="Debug fast flag") # Common arguments parser.add_argument( "--lr", type=float, default=0.0003, help="Learning rate for Adam") parser.add_argument( "--batch-size", type=int, default=64, help="Batch size") # Autoencoder arguments parser.add_argument( "--ae-epochs", type=int, default=50, help="Number of epochs for the autoencoder") parser.add_argument( "--ae-channels", type=int, default=4096, help="Number of channels in the autoencoder") parser.add_argument( "--ae-stride", type=int, default=256, help="Stride of the autoencoder") parser.add_argument( "--ae-dimension", type=int, default=128, help="Dimension of the autoencoder embedding") parser.add_argument( "--ae-kernel", type=int, default=1024, help="Kernel size of the autoencoder") parser.add_argument( "--ae-rewrite", type=int, default=2, help="Number of rewrite layers in the autoencoder") parser.add_argument( "--ae-context", type=int, default=9, help="Context size of the decoder") parser.add_argument( "--ae-window", default="hann", help="Window to use to smooth convolutions. Default to 'hann'. " "To deactivate, use --ae-no-window") parser.add_argument( "--ae-no-window", dest="ae_window", action="store_const", const=None) parser.add_argument( "--ae-squared-window", action="store_true", default=True, help="Square the window used to smooth convolutions. " "To deactivate, use --ae-no-squared-window.") parser.add_argument( "--ae-no-squared-window", action="store_false", dest="ae_squared_window") # Sequence generator arguments parser.add_argument( "--seq-hidden-size", type=int, default=1024, help="Size of the LSTM hidden layers") parser.add_argument( "--seq-layers", type=int, default=3, help="Number of layers in the LSTM") parser.add_argument( "--seq-epochs", type=int, default=50, help="Number of epochs for the sequence generator") parser.add_argument( "--seq-truncated", type=int, default=32, help="Truncated gradient for the sequence generator. " "0 means using the full sequence.") parser.add_argument( "--sing-epochs", type=int, default=20, help="Number of fine tuning epochs for the full SING model") # Lookup tables arguments parser.add_argument( "--time-dim", type=int, default=4, help="Dimension of the time step lookup table") parser.add_argument( "--instrument-dim", type=int, default=16, help="Dimension of the instrument embedding") parser.add_argument( "--pitch-dim", type=int, default=8, help="Dimension of the pitch embedding") parser.add_argument( "--velocity-dim", type=int, default=2, help="Dimension of the velocity embedding") return parser ================================================ FILE: sing/sequence/__init__.py ================================================ # Copyright (c) 2018-present, Facebook, Inc. # All rights reserved. # # This source code is licensed under the license found in the # LICENSE file in the root directory of this source tree. # ================================================ FILE: sing/sequence/models.py ================================================ # -*- coding: utf-8 -*- # Copyright (c) 2018-present, Facebook, Inc. # All rights reserved. # # This source code is licensed under the license found in the # LICENSE file in the root directory of this source tree. # import torch from torch import nn from ..fondation import utils class SequenceGenerator(nn.Module): """ LSTM part of the SING model. Arguments: embeddings (dict[str, (int, int)]): represents the lookup tables used by the model. For each entry under the key `name`, with value `(cardinality, dimension)`, the tensor named `name` will be retrieved. Its values should be in `[0, cardinality - 1]` and the lookup table will have dimension `dimension` length (int): length of the generated sequence output_dimension (int): dimension of each generated sequence item hidden_size (int): size of each layer, see the documentation of :class:`nn.LSTM` num_layers (int): number of layers, see the documentation of :class:`nn.LSTM` """ def __init__(self, embeddings, length, time_dimension=4, output_dimension=128, hidden_size=1024, num_layers=3): super(SequenceGenerator, self).__init__() self.tables = nn.ModuleList() self.inputs = [] input_size = time_dimension for name, (cardinality, dimension) in sorted(embeddings.items()): input_size += dimension self.inputs.append(name) self.tables.append( nn.Embedding( num_embeddings=cardinality, embedding_dim=dimension)) if time_dimension == 0: self.time_table = None else: self.time_table = nn.Embedding( num_embeddings=length, embedding_dim=time_dimension) self.length = length self.lstm = nn.LSTM( input_size=input_size, hidden_size=hidden_size, num_layers=num_layers) self.decoder = nn.Linear(hidden_size, output_dimension) def forward(self, start=0, length=None, hidden=None, **tensors): """ Arguments: start (int): first time step to generate length (int): length of the sequence to generate. If `None`, will be taken to be `self.length - start` hidden ((torch.FloatTensor, torch.FloatTensor)): hidden state of the LSTM or `None` to start from a blank one **tensors (dict[str, torch.LongTensor]): dictionary containing the tensors used as inputs to the lookup tables specified by the `embeddings` parameter of the constructor """ length = self.length - start if length is None else length inputs = [] for name, table in zip(self.inputs, self.tables): value = tensors[name].transpose(0, 1) embedding = table.forward(value) inputs.append(embedding.expand(length, -1, -1)) reference = inputs[0] if self.time_table is not None: times = torch.arange( start, start + length, device=reference.device).view(-1, 1).expand( -1, reference.size(1)) inputs.append(self.time_table.forward(times)) input = torch.cat(inputs, dim=-1) if hidden is not None: hidden = [h.transpose(0, 1).contiguous() for h in hidden] self.lstm.flatten_parameters() output, hidden = self.lstm.forward(input, hidden) decoded = self.decoder(output.view(-1, output.size(-1))).view( output.size(0), output.size(1), -1) hidden = [h.transpose(0, 1) for h in hidden] return decoded.transpose(0, 1).transpose(1, 2), hidden class SING(nn.Module): """ Complete SING model. Arguments: sequence_generator (SequenceGenerator): the LSTM based sequence generator part of SING decoder (sing.ae.models.ConvolutionalDecoder): the convolutional decoder part of SING """ def __init__(self, sequence_generator, decoder): super(SING, self).__init__() self.sequence_generator = sequence_generator self.decoder = decoder def forward(self, **tensors): """ Arguments: **tensors (dict[str, torch.LongTensor]): Tensors used as inputs to the lookup tables specified by the `embeddings` parameter of :class:`SequenceGenerator` """ return self.decoder.forward(self.sequence_generator(**tensors)[0]) def download_pretrained_model(target): """ Download a pretrained version of SING. """ url = "https://dl.fbaipublicfiles.com/sing/sing.th" sha256 = "eda8a7ce66f1ccf31cdd34a920290d80aabf96584c4d53df866b744f2862dc1c" utils.download_file(target, url, sha256=sha256) ================================================ FILE: sing/sequence/trainer.py ================================================ # -*- coding: utf-8 -*- # Copyright (c) 2018-present, Facebook, Inc. # All rights reserved. # # This source code is licensed under the license found in the # LICENSE file in the root directory of this source tree. # from torch import nn from ..fondation import utils, trainer class SequenceGeneratorTrainer(trainer.BaseTrainer): """ Trainer for the sequence generator (LSTM) part of SING. Arguments: decoder (sing.ae.models.ConvolutionalDecoder): decoder, used to compute the metrics on the waveforms truncated_gradient (int): size of sequence to compute the gradients over. If `None`, the whole sequence is used """ def __init__(self, decoder, truncated_gradient=32, **kwargs): super(SequenceGeneratorTrainer, self).__init__(**kwargs) self.truncated_gradient = truncated_gradient self.decoder = decoder if self.is_parallel: self.parallel_decoder = nn.DataParallel(decoder) else: self.parallel_decoder = decoder def _train_batch(self, batch): embeddings = batch.tensors['embeddings'] assert embeddings.size(-1) == self.model.length total_length = self.model.length hidden = None if self.truncated_gradient: truncated_gradient = self.truncated_gradient else: truncated_gradient = total_length steps = list(range(0, total_length, truncated_gradient)) total_loss = 0 for start_time in steps: sequence_length = min(truncated_gradient, total_length - start_time) target = embeddings[..., start_time:start_time + sequence_length] rebuilt, hidden = self.parallel.forward( start=start_time, length=sequence_length, hidden=hidden, **batch.tensors) hidden = tuple([h.detach() for h in hidden]) self.optimizer.zero_grad() loss = self.train_loss(rebuilt, target) loss.backward() self.optimizer.step() total_loss += loss.item() / len(steps) return total_loss def _get_rebuilt_target(self, batch): wav = batch.tensors['wav'] target = utils.unpad1d(wav, self.decoder.strip) embeddings, _ = self.parallel.forward(**batch.tensors) rebuilt = self.parallel_decoder.forward(embeddings) return rebuilt, target class SINGTrainer(trainer.BaseTrainer): """ Trainer for the entire SING model. """ def _get_rebuilt_target(self, batch): wav = batch.tensors['wav'] rebuilt = self.parallel.forward(**batch.tensors) target = utils.unpad1d(wav, self.model.decoder.strip) return rebuilt, target ================================================ FILE: sing/sequence/utils.py ================================================ # -*- coding: utf-8 -*- # Copyright (c) 2018-present, Facebook, Inc. # All rights reserved. # # This source code is licensed under the license found in the # LICENSE file in the root directory of this source tree. # import torch from torch import nn from torch.utils.data import DataLoader import tqdm from ..fondation.batch import collate def generate_embeddings_dataset(dataset, encoder, batch_size, cuda, parallel): """ Pre-compute all the embeddings for a given dataset. Arguments: dataset (Dataset): dataset to compute the embeddings for. It should contain a `'wav'` tensor encoder (sing.ae.models.ConvolutionalEncoder): encoder to use to generate the embedding batch_size (int): batch size to use cuda (bool): if `True`, performs the computation on GPU parallel (bool): if `True`, use all available GPUs Returns: Dataset: dataset of the same size as `dataset` but with the `'wav'` tensor replaced by an `'embeddings'` tensor. """ loader = DataLoader( dataset, batch_size=batch_size, shuffle=False, collate_fn=collate) embeddings_dataset = [None] * len(dataset) if cuda: encoder.cuda() if parallel: encoder = nn.DataParallel(encoder) row = 0 with tqdm.tqdm(total=len(dataset), unit="ex") as bar: for batch in loader: if cuda: batch.cuda_() with torch.no_grad(): batch.tensors['embeddings'] = encoder.forward( batch.tensors['wav']) del batch.tensors['wav'] for item in batch.cpu(): embeddings_dataset[row] = item row += 1 bar.update(len(batch)) return embeddings_dataset ================================================ FILE: sing/train.py ================================================ # -*- coding: utf-8 -*- # Copyright (c) 2018-present, Facebook, Inc. # All rights reserved. # # This source code is licensed under the license found in the # LICENSE file in the root directory of this source tree. # import functools from torch import nn import torch from .parser import get_parser from . import nsynth, dsp from .ae.models import ConvolutionalAE from .ae.trainer import AutoencoderTrainer from .fondation import utils, datasets from .sequence.models import SequenceGenerator, SING from .sequence.trainer import SequenceGeneratorTrainer, SINGTrainer from .sequence.utils import generate_embeddings_dataset def train_autoencoder(args, **kwargs): checkpoint_path = args.checkpoint / "ae.torch" if args.checkpoint else None model = ConvolutionalAE( channels=args.ae_channels, stride=args.ae_stride, dimension=args.ae_dimension, kernel_size=args.ae_kernel, context_size=args.ae_context, rewrite_layers=args.ae_rewrite, window_name=args.ae_window, squared_window=args.ae_squared_window) advised_pad = model.decoder.strip + 512 if args.pad != advised_pad: print("Warning, best padding for the current settings is {}, " "current value is {}.".format(advised_pad, args.pad)) if args.ae_epochs: print("Training autoencoder") AutoencoderTrainer( suffix="_ae", model=model, epochs=args.ae_epochs, checkpoint_path=checkpoint_path, **kwargs).train() return model def train_sequence_generator(args, autoencoder, cardinalities, train_dataset, eval_datasets, train_loss, eval_losses, **kwargs): checkpoint_path = (args.checkpoint / "seq.torch" if args.checkpoint else None) wav_length = train_dataset[0].tensors['wav'].size(-1) embedding_length = autoencoder.decoder.embedding_length( wav_length - 2 * autoencoder.decoder.strip) embeddings = { name: (cardinalities[name], getattr(args, '{}_dim'.format(name))) for name in ['velocity', 'instrument', 'pitch'] } model = SequenceGenerator( embeddings=embeddings, length=embedding_length, time_dimension=args.time_dim, output_dimension=args.ae_dimension, hidden_size=args.seq_hidden_size, num_layers=args.seq_layers) if args.seq_epochs: print("Precomputing embeddings for all datasets") generate_embeddings = functools.partial( generate_embeddings_dataset, encoder=autoencoder.encoder, batch_size=args.batch_size, cuda=args.cuda, parallel=args.parallel) train_dataset = generate_embeddings(train_dataset) print("Training sequence generator") SequenceGeneratorTrainer( suffix="_seq", model=model, decoder=autoencoder.decoder, epochs=args.seq_epochs, train_loss=nn.MSELoss(), eval_losses=eval_losses, train_dataset=train_dataset, eval_datasets=eval_datasets, truncated_gradient=args.seq_truncated, checkpoint_path=checkpoint_path, **kwargs).train() return model def fine_tune_sing(args, sequence_generator, decoder, **kwargs): print("Fine tuning SING") checkpoint_path = (args.checkpoint / "sing.torch" if args.checkpoint else None) model = SING(sequence_generator=sequence_generator, decoder=decoder) if args.sing_epochs: SINGTrainer( suffix="_sing", epochs=args.sing_epochs, model=model, checkpoint_path=checkpoint_path, **kwargs).train() return model def main(): args = get_parser().parse_args() if args.debug: args.ae_epochs = 1 args.seq_epochs = 1 args.sing_epochs = 1 if args.debug_fast: args.ae_channels = 128 args.ae_dimension = 16 args.ae_rewrite = 1 args.seq_hidden_size = 128 args.seq_layers = 1 if args.checkpoint: args.checkpoint.mkdir(exist_ok=True, parents=True) if not args.data.exists(): utils.fatal("Could not find the nsynth dataset. " "To download it, follow the instructions at " "https://github.com/facebookresearch/SING") nsynth_dataset = nsynth.NSynthDataset(args.data, pad=args.pad) cardinalities = nsynth_dataset.metadata.cardinalities train_dataset, valid, test = nsynth.make_datasets(nsynth_dataset) if args.debug: train_dataset = datasets.RandomSubset(train_dataset, size=100) eval_train = datasets.RandomSubset(train_dataset, size=10000) if args.debug: eval_datasets = { 'eval_train': eval_train, } else: eval_datasets = { 'eval_train': eval_train, 'valid': valid, 'test': test, } base_loss = nn.L1Loss() if args.l1 else nn.MSELoss() train_loss = base_loss if args.wav else dsp.SpectralLoss( base_loss, epsilon=args.epsilon) eval_losses = { 'wav_l1': nn.L1Loss(), 'wav_mse': nn.MSELoss(), 'spec_l1': dsp.SpectralLoss(nn.L1Loss(), epsilon=args.epsilon), 'spec_mse': dsp.SpectralLoss(nn.MSELoss(), epsilon=args.epsilon), } kwargs = { 'train_dataset': train_dataset, 'eval_datasets': eval_datasets, 'train_loss': train_loss, 'eval_losses': eval_losses, 'batch_size': args.batch_size, 'lr': args.lr, 'cuda': args.cuda, 'parallel': args.parallel, } autoencoder = train_autoencoder(args, **kwargs) sequence_generator = train_sequence_generator(args, autoencoder, cardinalities, **kwargs) sing = fine_tune_sing(args, sequence_generator, autoencoder.decoder, **kwargs) torch.save(sing.cpu(), str(args.output)) if __name__ == "__main__": main()