Repository: chomeyama/SiFiGAN
Branch: main
Commit: cdb20daeb4da
Files: 56
Total size: 20.8 MB
Directory structure:
gitextract_jyc_yf0t/
├── .gitignore
├── LICENSE
├── README.md
├── egs/
│ ├── libritts-r-clean+nus-48e_all/
│ │ └── data/
│ │ └── scp/
│ │ ├── libritts_r_clean+nus-48e_dev.list
│ │ ├── libritts_r_clean+nus-48e_dev.scp
│ │ ├── libritts_r_clean+nus-48e_eval.list
│ │ ├── libritts_r_clean+nus-48e_eval.scp
│ │ ├── libritts_r_clean+nus-48e_train_no_dev.list
│ │ └── libritts_r_clean+nus-48e_train_no_dev.scp
│ └── namine_ritsu/
│ └── data/
│ └── scp/
│ ├── namine_ritsu_all.scp
│ ├── namine_ritsu_dev.scp
│ ├── namine_ritsu_eval.scp
│ └── namine_ritsu_train_no_dev.scp
├── setup.py
└── sifigan/
├── __init__.py
├── bin/
│ ├── __init__.py
│ ├── anasyn.py
│ ├── compute_statistics.py
│ ├── config/
│ │ ├── __init__.py
│ │ ├── anasyn.yaml
│ │ ├── compute_statistics.yaml
│ │ ├── data/
│ │ │ ├── libritts-r-clean+nus-48e.yaml
│ │ │ └── namine_ritsu.yaml
│ │ ├── decode.yaml
│ │ ├── discriminator/
│ │ │ ├── hifigan.yaml
│ │ │ └── univnet.yaml
│ │ ├── extract_features.yaml
│ │ ├── generator/
│ │ │ ├── hifigan.yaml
│ │ │ ├── sifigan.direct.yaml
│ │ │ └── sifigan.yaml
│ │ ├── param_count.yaml
│ │ ├── train/
│ │ │ ├── hifigan.yaml
│ │ │ ├── sifigan.yaml
│ │ │ └── sifigan_1000k.yaml
│ │ └── train.yaml
│ ├── decode.py
│ ├── extract_features.py
│ ├── param_count.py
│ └── train.py
├── datasets/
│ ├── __init__.py
│ └── audio_feat_dataset.py
├── layers/
│ ├── __init__.py
│ ├── cheaptrick.py
│ ├── residual_block.py
│ └── snake.py
├── losses/
│ ├── __init__.py
│ ├── adv.py
│ ├── mel.py
│ └── reg.py
├── models/
│ ├── __init__.py
│ ├── discriminator.py
│ └── generator.py
└── utils/
├── __init__.py
├── features.py
├── index.py
└── utils.py
================================================
FILE CONTENTS
================================================
================================================
FILE: .gitignore
================================================
__pycache__
.eggs
*.egg-info
*.log
*.out
*.wav
*.h5
.venv
egs/*/exp
================================================
FILE: LICENSE
================================================
MIT License
Copyright (c) 2022 Reo YONEYAMA
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
================================================
FILE: README.md
================================================
# Source-Filter HiFi-GAN (SiFi-GAN)
This repo provides official PyTorch implementation of [SiFi-GAN](https://arxiv.org/abs/2210.15533), a fast and pitch controllable high-fidelity neural vocoder.
For more information, please see our [DEMO](https://chomeyama.github.io/SiFiGAN-Demo/).
## Environment setup
```bash
$ cd SiFiGAN
$ pip install -e .
```
Please refer to the [Parallel WaveGAN](https://github.com/kan-bayashi/ParallelWaveGAN) repo for more details.
## Folder architecture
- **egs**:
The folder for projects.
- **egs/namine_ritsu**:
The folder of the [Namine Ritsu](https://www.youtube.com/watch?v=pKeo9IE_L1I) project example.
- **sifigan**:
The folder of the source codes.
The dataset preparation of Namine Ritsu database is based on [NNSVS](https://github.com/nnsvs/nnsvs/).
Please refer to it for the procedure and details.
## Run
In this repo, hyperparameters are managed using [Hydra](https://hydra.cc/docs/intro/).
Hydra provides an easy way to dynamically create a hierarchical configuration by composition and override it through config files and the command line.
### Dataset preparation
Make dataset and scp files denoting paths to each audio files according to your own dataset (e.g., `egs/namine_ritsu/data/scp/namine_ritsu.scp`).
List files denoting paths to the extracted features are automatically created in the next step (e.g., `egs/namine_ritsu/data/scp/namine_ritsu.list`).
Note that scp/list files for training/validation/evaluation are needed.
### Preprocessing
```bash
# Move to the project directory
$ cd egs/namine_ritsu
# Extract acoustic features (F0, mel-cepstrum, and etc.)
# You can customize parameters according to sifigan/bin/config/extract_features.yaml
$ sifigan-extract-features audio=data/scp/namine_ritsu_all.scp
# Compute statistics of training data
$ sifigan-compute-statistics feats=data/scp/namine_ritsu_train.list stats=data/stats/namine_ritsu_train.joblib
```
### Training
```bash
# Train a model customizing the hyperparameters as you like
$ sifigan-train generator=sifigan discriminator=univnet train=sifigan data=namine_ritsu out_dir=exp/sifigan
```
### Inference
```bash
# Decode with several F0 scaling factors
$ sifigan-decode generator=sifigan data=namine_ritsu out_dir=exp/sifigan checkpoint_steps=400000 f0_factors=[0.5,1.0,2.0]
```
### Analysis-Synthesis
```bash
# WORLD analysis + Neural vocoder synthesis
$ sifigan-anasyn generator=sifigan in_dir=your_own_input_wav_dir out_dir=your_own_output_wav_dir stats=pretrained_sifigan/namine_ritsu_train_no_dev.joblib checkpoint_path=pretrained_sifigan/checkpoint-400000steps.pkl f0_factors=[1.0]
```
### Pretrained model
~~I provide a pretrained SiFiGAN model [HERE](https://www.dropbox.com/s/akofngycxxz1dg5/pretrained_sifigan.tar.gz?dl=0) which is trained on the Namine Ritsu corpus in the same training manner described in the paper.
You can download and place it in your own directory. Then set the appropriate path to the pretrained model and the command should work.~~
~~However, since the Namine Ritsu corpus includes a single female Japanese singer, there is a possibility that the model would not work well especially for male singers.
I am planning to publish another pretrained model trained on larger dataset including many speakers.~~
Due to being trained on the code before bug fixes, I have decided to cancel the release of the model trained on the Namine Ritsu database. Instead, a model trained on the following large-scale dataset is available.
A pretrained model on 24 kHz speech + singing datasets is available [HERE](https://drive.google.com/file/d/1uzqTeumvkPQpfdK_D4U41MDL5-s-hs0l/view?usp=sharing). We used train-clean-100 and train-clean-360 in [LibriTTS-R](https://google.github.io/df-conformer/librittsr/), and [NUS-48E](https://www.smcnus.org/wp-content/uploads/2013/09/05-Pub-NUS-48E.pdf) for training.
Two speakers, ADIZ and JLEE in NUS-48E, were excluded from the training data for evaluation. Also, the wav data of NUS-48E were divided into clips of approximately one second each before the feature extraction step.
The feature preprocessing and training commands are as follows:
```bash
sifigan-extract-features audio=data/scp/libritts_r_clean+nus-48e_train_no_dev.scp minf0=60 maxf0=1000
sifigan-extract-features audio=data/scp/libritts_r_clean+nus-48e_dev.scp minf0=60 maxf0=1000
sifigan-extract-features audio=data/scp/libritts_r_clean+nus-48e_eval.scp minf0=60 maxf0=1000
sifigan-compute-statistics feats=data/scp/libritts_r_clean+nus-48e_train_no_dev.list stats=data/stats/libritts_r_clean+nus-48e_train_no_dev.joblib
sifigan-train out_dir=test/sifigan generator=sifigan data=libritts_r_clean+nus-48e train=sifigan_1000k
```
### Monitor training progress
```bash
$ tensorboard --logdir exp
```
## Citation
If you find the code is helpful, please cite the following article.
```
@INPROCEEDINGS{10095298,
author={Yoneyama, Reo and Wu, Yi-Chiao and Toda, Tomoki},
booktitle={ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
title={{Source-Filter HiFi-GAN: Fast and Pitch Controllable High-Fidelity Neural Vocoder}},
year={2023},
volume={},
number={},
pages={1-5},
doi={10.1109/ICASSP49357.2023.10095298}
}
```
## Authors
Development:
[Reo Yoneyama](https://chomeyama.github.io/Profile/) @ Nagoya University, Japan
E-mail: `yoneyama.reo@g.sp.m.is.nagoya-u.ac.jp`
Advisors:
[Yi-Chiao Wu](https://bigpon.github.io/) @ Meta Reality Labs Research, USA
E-mail: `yichiaowu@fb.com`
[Tomoki Toda](https://sites.google.com/site/tomokitoda/) @ Nagoya University, Japan