Repository: jackaduma/CycleGAN-VC3
Branch: main
Commit: 4bdbddab205e
Files: 8
Total size: 36.8 KB
Directory structure:
gitextract_v07ir58a/
├── LICENSE
├── README.md
├── datasets.py
├── feature_utils.py
├── melgan_vocoder.py
├── model.py
├── tfan_module.py
└── train.py
================================================
FILE CONTENTS
================================================
================================================
FILE: LICENSE
================================================
MIT License
Copyright (c) 2020 Kun Ma
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
================================================
FILE: README.md
================================================
# **CycleGAN-VC3-PyTorch**
[](https://github.com/jackaduma/CycleGAN-VC2)
[](https://paypal.me/jackaduma?locale.x=zh_XC)
[**中文说明**](./README.zh-CN.md) | [**English**](./README.md)
------
This code is a **PyTorch** implementation for paper: [CycleGAN-VC3: Examining and Improving CycleGAN-VCs for Mel-spectrogram Conversion](https://arxiv.org/abs/2010.11672]), a nice work on **Voice-Conversion/Voice Cloning**.
- [x] Dataset
- [ ] VC
- [x] Usage
- [x] Training
- [x] Example
- [ ] Demo
- [x] Reference
------
## **CycleGAN-VC3**
### [**Project Page**](http://www.kecl.ntt.co.jp/people/kaneko.takuhiro/projects/cyclegan-vc3/index.html)
Non-parallel voice conversion (VC) is a technique for learning mappings between source and target speeches without using a parallel corpus. Recently, CycleGAN-VC [3] and CycleGAN-VC2 [2] have shown promising results regarding this problem and have been widely used as benchmark methods. However, owing to the ambiguity of the effectiveness of CycleGAN-VC/VC2 for **mel-spectrogram conversion**, they are typically used for mel-cepstrum conversion even when comparative methods employ mel-spectrogram as a conversion target. To address this, we examined the applicability of CycleGAN-VC/VC2 to **mel-spectrogram conversion**. Through initial experiments, we discovered that their direct applications compromised the time-frequency structure that should be preserved during conversion. To remedy this, we propose CycleGAN-VC3, an improvement of CycleGAN-VC2 that incorporates **time-frequency adaptive normalization (TFAN)**. Using TFAN, we can adjust the scale and bias of the converted features while reflecting the time-frequency structure of the source mel-spectrogram. We evaluated CycleGAN-VC3 on inter-gender and intra-gender non-parallel VC. A subjective evaluation of naturalness and similarity showed that for every VC pair, CycleGAN-VC3 outperforms or is competitive with the two types of CycleGAN-VC2, one of which was applied to mel-cepstrum and the other to mel-spectrogram.
 _Figure 1. We developed time-frequency adaptive normalization (TFAN), which extends instance normalization [5] so that the affine parameters become element-dependent and are determined according to an entire input mel-spectrogram._
------
**This repository contains:**
1. [TFAN module code](tfan_module.py) which implemented the TFAN module
1. [model code](model.py) which implemented the model network.
2. [audio preprocessing script](preprocess_training.py) you can use to create cache for [training data](data).
3. [training scripts](train.py) to train the model.
------
## **Table of Contents**
- [**CycleGAN-VC3-PyTorch**](#cyclegan-vc3-pytorch)
- [**CycleGAN-VC3**](#cyclegan-vc3)
- [**Project Page**](#project-page)
- [**Table of Contents**](#table-of-contents)
- [**Requirement**](#requirement)
- [**Usage**](#usage)
- [**Star-History**](#star-history)
- [**Reference**](#reference)
- [Donation](#donation)
- [**License**](#license)
------
## **Requirement**
```bash
pip install -r requirements.txt
```
## **Usage**
------
## **Star-History**
------
## **Reference**
1. **CycleGAN-VC3: Examining and Improving CycleGAN-VCs for Mel-spectrogram Conversion.** [Paper](https://arxiv.org/abs/2010.11672), [Project](http://www.kecl.ntt.co.jp/people/kaneko.takuhiro/projects/cyclegan-vc3/index.html)
2. CycleGAN-VC2: Improved CycleGAN-based Non-parallel Voice Conversion. [Paper](https://arxiv.org/abs/1904.04631), [Project](http://www.kecl.ntt.co.jp/people/kaneko.takuhiro/projects/cyclegan-vc2/index.html)
3. Parallel-Data-Free Voice Conversion Using Cycle-Consistent Adversarial Networks. [Paper](https://arxiv.org/abs/1711.11293), [Project](http://www.kecl.ntt.co.jp/people/kaneko.takuhiro/projects/cyclegan-vc/)
4. Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks. [Paper](https://arxiv.org/abs/1703.10593), [Project](https://junyanz.github.io/CycleGAN/), [Code](https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix)
5. Image-to-Image Translation with Conditional Adversarial Nets. [Paper](https://arxiv.org/abs/1611.07004), [Project](https://phillipi.github.io/pix2pix/), [Code](https://github.com/phillipi/pix2pix)
------
## Donation
If this project help you reduce time to develop, you can give me a cup of coffee :)
AliPay(支付宝)
WechatPay(微信)
[](https://paypal.me/jackaduma?locale.x=zh_XC)
------
## **License**
[MIT](LICENSE) © Kun
================================================
FILE: datasets.py
================================================
================================================
FILE: feature_utils.py
================================================
#!python
# -*- coding: utf-8 -*-
import torch
import torch.nn as nn
import torch.nn.functional as F
from librosa.filters import mel as librosa_mel_fn
class Audio2Mel(nn.Module):
def __init__(
self,
n_fft=1024,
hop_length=256,
win_length=1024,
sampling_rate=22050,
n_mel_channels=80,
mel_fmin=0.0,
mel_fmax=None,
):
super().__init__()
##############################################
# FFT Parameters #
##############################################
window = torch.hann_window(win_length).float()
mel_basis = librosa_mel_fn(
sampling_rate, n_fft, n_mel_channels, mel_fmin, mel_fmax
)
mel_basis = torch.from_numpy(mel_basis).float()
self.register_buffer("mel_basis", mel_basis)
self.register_buffer("window", window)
self.n_fft = n_fft
self.hop_length = hop_length
self.win_length = win_length
self.sampling_rate = sampling_rate
self.n_mel_channels = n_mel_channels
def forward(self, audio):
p = (self.n_fft - self.hop_length) // 2
audio = F.pad(audio, (p, p), "reflect").squeeze(1)
fft = torch.stft(
audio,
n_fft=self.n_fft,
hop_length=self.hop_length,
win_length=self.win_length,
window=self.window,
center=False,
)
real_part, imag_part = fft.unbind(-1)
magnitude = torch.sqrt(real_part ** 2 + imag_part ** 2)
mel_output = torch.matmul(self.mel_basis, magnitude)
log_mel_spec = torch.log10(torch.clamp(mel_output, min=1e-5))
return log_mel_spec
================================================
FILE: melgan_vocoder.py
================================================
#!python
# -*- coding: utf-8 -*-
import os
import yaml
from pathlib import Path
import torch
import torch.nn as nn
from torch.nn.utils import weight_norm
from feature_utils import Audio2Mel
def weights_init(m):
classname = m.__class__.__name__
if classname.find("Conv") != -1:
m.weight.data.normal_(0.0, 0.02)
elif classname.find("BatchNorm2d") != -1:
m.weight.data.normal_(1.0, 0.02)
m.bias.data.fill_(0)
def WNConv1d(*args, **kwargs):
return weight_norm(nn.Conv1d(*args, **kwargs))
def WNConvTranspose1d(*args, **kwargs):
return weight_norm(nn.ConvTranspose1d(*args, **kwargs))
class ResnetBlock(nn.Module):
def __init__(self, dim, dilation=1):
super().__init__()
self.block = nn.Sequential(
nn.LeakyReLU(0.2),
nn.ReflectionPad1d(dilation),
WNConv1d(dim, dim, kernel_size=3, dilation=dilation),
nn.LeakyReLU(0.2),
WNConv1d(dim, dim, kernel_size=1),
)
self.shortcut = WNConv1d(dim, dim, kernel_size=1)
def forward(self, x):
return self.shortcut(x) + self.block(x)
class Generator(nn.Module):
def __init__(self, input_size, ngf, n_residual_layers):
super().__init__()
ratios = [8, 8, 2, 2]
self.hop_length = np.prod(ratios)
mult = int(2 ** len(ratios))
model = [
nn.ReflectionPad1d(3),
WNConv1d(input_size, mult * ngf, kernel_size=7, padding=0),
]
# Upsample to raw audio scale
for i, r in enumerate(ratios):
model += [
nn.LeakyReLU(0.2),
WNConvTranspose1d(
mult * ngf,
mult * ngf // 2,
kernel_size=r * 2,
stride=r,
padding=r // 2 + r % 2,
output_padding=r % 2,
),
]
for j in range(n_residual_layers):
model += [ResnetBlock(mult * ngf // 2, dilation=3 ** j)]
mult //= 2
model += [
nn.LeakyReLU(0.2),
nn.ReflectionPad1d(3),
WNConv1d(ngf, 1, kernel_size=7, padding=0),
nn.Tanh(),
]
self.model = nn.Sequential(*model)
self.apply(weights_init)
def forward(self, x):
return self.model(x)
def get_default_device():
if torch.cuda.is_available():
return "cuda"
else:
return "cpu"
def load_model(mel2wav_path, device=get_default_device()):
"""
Args:
mel2wav_path (str or Path): path to the root folder of dumped text2mel
device (str or torch.device): device to load the model
"""
root = Path(mel2wav_path)
with open(root / "args.yml", "r") as f:
args = yaml.load(f, Loader=yaml.FullLoader)
netG = Generator(args.n_mel_channels, args.ngf, args.n_residual_layers).to(device)
netG.load_state_dict(torch.load(root / "best_netG.pt", map_location=device))
return netG
class MelVocoder:
def __init__(
self,
path,
device=get_default_device(),
github=False,
model_name="multi_speaker",
):
self.fft = Audio2Mel().to(device)
if github:
netG = Generator(80, 32, 3).to(device)
root = Path(os.path.dirname(__file__)).parent
netG.load_state_dict(
torch.load(root / f"models/{model_name}.pt", map_location=device)
)
self.mel2wav = netG
else:
self.mel2wav = load_model(path, device)
self.device = device
def __call__(self, audio):
"""
Performs audio to mel conversion (See Audio2Mel in mel2wav/modules.py)
Args:
audio (torch.tensor): PyTorch tensor containing audio (batch_size, timesteps)
Returns:
torch.tensor: log-mel-spectrogram computed on input audio (batch_size, 80, timesteps)
"""
return self.fft(audio.unsqueeze(1).to(self.device))
def inverse(self, mel):
"""
Performs mel2audio conversion
Args:
mel (torch.tensor): PyTorch tensor containing log-mel spectrograms (batch_size, 80, timesteps)
Returns:
torch.tensor: Inverted raw audio (batch_size, timesteps)
"""
with torch.no_grad():
return self.mel2wav(mel.to(self.device)).squeeze(1)
================================================
FILE: model.py
================================================
#! python
# -*- coding: utf-8 -*-
# Author: kun
# @Time: 2020-11-17 14:35
import torch.nn as nn
import torch
from tfan_module import TFAN_1D, TFAN_2D
class GLU(nn.Module):
def __init__(self):
super(GLU, self).__init__()
# Custom Implementation because the Voice Conversion Cycle GAN
# paper assumes GLU won't reduce the dimension of tensor by 2.
def forward(self, input):
return input * torch.sigmoid(input)
class PixelShuffle(nn.Module):
def __init__(self, upscale_factor):
super(PixelShuffle, self).__init__()
# Custom Implementation because PyTorch PixelShuffle requires,
# 4D input. Whereas, in this case we have have 3D array
self.upscale_factor = upscale_factor
def forward(self, input):
n = input.shape[0]
c_out = input.shape[1] // 2
w_new = input.shape[2] * 2
return input.view(n, c_out, w_new)
##########################################################################################
class ResidualLayer(nn.Module):
def __init__(self, in_channels, out_channels, kernel_size, stride, padding):
super(ResidualLayer, self).__init__()
# self.residualLayer = nn.Sequential(nn.Conv1d(in_channels=in_channels,
# out_channels=out_channels,
# kernel_size=kernel_size,
# stride=1,
# padding=padding),
# nn.InstanceNorm1d(
# num_features=out_channels,
# affine=True),
# GLU(),
# nn.Conv1d(in_channels=out_channels,
# out_channels=in_channels,
# kernel_size=kernel_size,
# stride=1,
# padding=padding),
# nn.InstanceNorm1d(
# num_features=in_channels,
# affine=True)
# )
self.conv1d_layer = nn.Sequential(nn.Conv1d(in_channels=in_channels,
out_channels=out_channels,
kernel_size=kernel_size,
stride=1,
padding=padding),
nn.InstanceNorm1d(num_features=out_channels,
affine=True))
self.conv_layer_gates = nn.Sequential(nn.Conv1d(in_channels=in_channels,
out_channels=out_channels,
kernel_size=kernel_size,
stride=1,
padding=padding),
nn.InstanceNorm1d(num_features=out_channels,
affine=True))
self.conv1d_out_layer = nn.Sequential(nn.Conv1d(in_channels=out_channels,
out_channels=in_channels,
kernel_size=kernel_size,
stride=1,
padding=padding),
nn.InstanceNorm1d(num_features=in_channels,
affine=True))
def forward(self, input):
h1_norm = self.conv1d_layer(input)
h1_gates_norm = self.conv_layer_gates(input)
# GLU
h1_glu = h1_norm * torch.sigmoid(h1_gates_norm)
h2_norm = self.conv1d_out_layer(h1_glu)
return input + h2_norm
##########################################################################################
class downSample_Generator(nn.Module):
def __init__(self, in_channels, out_channels, kernel_size, stride, padding):
super(downSample_Generator, self).__init__()
self.convLayer = nn.Sequential(nn.Conv2d(in_channels=in_channels,
out_channels=out_channels,
kernel_size=kernel_size,
stride=stride,
padding=padding),
nn.InstanceNorm2d(num_features=out_channels,
affine=True))
self.convLayer_gates = nn.Sequential(nn.Conv2d(in_channels=in_channels,
out_channels=out_channels,
kernel_size=kernel_size,
stride=stride,
padding=padding),
nn.InstanceNorm2d(num_features=out_channels,
affine=True))
def forward(self, input):
# GLU
return self.convLayer(input) * torch.sigmoid(self.convLayer_gates(input))
##########################################################################################
class Generator(nn.Module):
def __init__(self):
super(Generator, self).__init__()
# 2D Conv Layer
self.conv1 = nn.Conv2d(in_channels=1, # TODO 1 ?
out_channels=128,
kernel_size=(5, 15),
stride=(1, 1),
padding=(2, 7))
self.conv1_gates = nn.Conv2d(in_channels=1, # TODO 1 ?
out_channels=128,
kernel_size=(5, 15),
stride=1,
padding=(2, 7))
# 2D Downsample Layer
self.downSample1 = downSample_Generator(in_channels=128,
out_channels=256,
kernel_size=5,
stride=2,
padding=2)
self.downSample2 = downSample_Generator(in_channels=256,
out_channels=256,
kernel_size=5,
stride=2,
padding=2)
# 2D -> 1D Conv
# self.conv2dto1dLayer = nn.Sequential(nn.Conv1d(in_channels=2304,
# out_channels=256,
# kernel_size=1,
# stride=1,
# padding=0),
# nn.InstanceNorm1d(num_features=256,
# affine=True))
self.conv2dto1dLayer = nn.Conv1d(in_channels=2304,
out_channels=256,
kernel_size=1,
stride=1,
padding=0)
self.conv2dto1dLayer_tfan = TFAN_1D(256)
# Residual Blocks
self.residualLayer1 = ResidualLayer(in_channels=256,
out_channels=512,
kernel_size=3,
stride=1,
padding=1)
self.residualLayer2 = ResidualLayer(in_channels=256,
out_channels=512,
kernel_size=3,
stride=1,
padding=1)
self.residualLayer3 = ResidualLayer(in_channels=256,
out_channels=512,
kernel_size=3,
stride=1,
padding=1)
self.residualLayer4 = ResidualLayer(in_channels=256,
out_channels=512,
kernel_size=3,
stride=1,
padding=1)
self.residualLayer5 = ResidualLayer(in_channels=256,
out_channels=512,
kernel_size=3,
stride=1,
padding=1)
self.residualLayer6 = ResidualLayer(in_channels=256,
out_channels=512,
kernel_size=3,
stride=1,
padding=1)
# 1D -> 2D Conv
# self.conv1dto2dLayer = nn.Sequential(nn.Conv1d(in_channels=256,
# out_channels=2304,
# kernel_size=1,
# stride=1,
# padding=0),
# nn.InstanceNorm1d(num_features=2304,
# affine=True))
self.conv1dto2dLayer = nn.Conv1d(in_channels=256,
out_channels=2304,
kernel_size=1,
stride=1,
padding=0)
self.conv1dto2dLayer_tfan = TFAN_1D(2304)
# UpSample Layer
self.upSample1 = self.upSample(in_channels=256,
out_channels=1024,
kernel_size=5,
stride=1,
padding=2)
self.upSample1_tfan = TFAN_2D(1024 // 4)
self.glu = GLU()
self.upSample2 = self.upSample(in_channels=256,
out_channels=512,
kernel_size=5,
stride=1,
padding=2)
self.upSample2_tfan = TFAN_2D(512 // 4)
self.lastConvLayer = nn.Conv2d(in_channels=128,
out_channels=1,
kernel_size=(5, 15),
stride=(1, 1),
padding=(2, 7))
def downSample(self, in_channels, out_channels, kernel_size, stride, padding):
self.ConvLayer = nn.Sequential(nn.Conv1d(in_channels=in_channels,
out_channels=out_channels,
kernel_size=kernel_size,
stride=stride,
padding=padding),
nn.InstanceNorm1d(
num_features=out_channels,
affine=True),
GLU())
return self.ConvLayer
# def upSample(self, in_channels, out_channels, kernel_size, stride, padding):
# self.convLayer = nn.Sequential(nn.Conv2d(in_channels=in_channels,
# out_channels=out_channels,
# kernel_size=kernel_size,
# stride=stride,
# padding=padding),
# nn.PixelShuffle(upscale_factor=2),
# nn.InstanceNorm2d(
# num_features=out_channels // 4,
# affine=True),
# GLU())
# return self.convLayer
def upSample(self, in_channels, out_channels, kernel_size, stride, padding):
self.convLayer = nn.Sequential(nn.Conv2d(in_channels=in_channels,
out_channels=out_channels,
kernel_size=kernel_size,
stride=stride,
padding=padding),
nn.PixelShuffle(upscale_factor=2))
return self.convLayer
def forward(self, input):
# GLU
print("Generator forward input: ", input.shape)
input = input.unsqueeze(1)
print("Generator forward input: ", input.shape)
seg_1d = input # for TFAN module
conv1 = self.conv1(input) * torch.sigmoid(self.conv1_gates(input))
print("Generator forward conv1: ", conv1.shape)
# DownloadSample
downsample1 = self.downSample1(conv1)
print("Generator forward downsample1: ", downsample1.shape)
downsample2 = self.downSample2(downsample1)
print("Generator forward downsample2: ", downsample2.shape)
# 2D -> 1D
# reshape
reshape2dto1d = downsample2.view(downsample2.size(0), 2304, 1, -1)
reshape2dto1d = reshape2dto1d.squeeze(2)
# print("Generator forward reshape2dto1d: ", reshape2dto1d.shape)
conv2dto1d_layer = self.conv2dto1dLayer(reshape2dto1d)
# print("Generator forward conv2dto1d_layer: ", conv2dto1d_layer.shape)
conv2dto1d_layer = self.conv2dto1dLayer_tfan(conv2dto1d_layer, seg_1d)
residual_layer_1 = self.residualLayer1(conv2dto1d_layer)
residual_layer_2 = self.residualLayer2(residual_layer_1)
residual_layer_3 = self.residualLayer3(residual_layer_2)
residual_layer_4 = self.residualLayer4(residual_layer_3)
residual_layer_5 = self.residualLayer5(residual_layer_4)
residual_layer_6 = self.residualLayer6(residual_layer_5)
# print("Generator forward residual_layer_6: ", residual_layer_6.shape)
# 1D -> 2D
conv1dto2d_layer = self.conv1dto2dLayer(residual_layer_6)
# print("Generator forward conv1dto2d_layer: ", conv1dto2d_layer.shape)
conv1dto2d_layer = self.conv1dto2dLayer_tfan(conv1dto2d_layer, seg_1d)
# reshape
reshape1dto2d = conv1dto2d_layer.unsqueeze(2)
reshape1dto2d = reshape1dto2d.view(reshape1dto2d.size(0), 256, 9, -1)
# print("Generator forward reshape1dto2d: ", reshape1dto2d.shape)
seg_2d = reshape1dto2d
# UpSample
upsample_layer_1 = self.upSample1(reshape1dto2d)
# print("Generator forward upsample_layer_1: ", upsample_layer_1.shape)
upsample_layer_1 = self.upSample1_tfan(upsample_layer_1, seg_2d)
upsample_layer_1 = self.glu(upsample_layer_1)
upsample_layer_2 = self.upSample2(upsample_layer_1)
# print("Generator forward upsample_layer_2: ", upsample_layer_2.shape)
upsample_layer_2 = self.upSample2_tfan(upsample_layer_2, seg_2d)
upsample_layer_2 = self.glu(upsample_layer_2)
output = self.lastConvLayer(upsample_layer_2)
# print("Generator forward output: ", output.shape)
output = output.squeeze(1)
# print("Generator forward output: ", output.shape)
return output
##########################################################################################
# 鉴别器 PatchGAN
class Discriminator(nn.Module):
def __init__(self):
super(Discriminator, self).__init__()
self.convLayer1 = nn.Sequential(nn.Conv2d(in_channels=1,
out_channels=128,
kernel_size=(3, 3),
stride=(1, 1),
padding=(1, 1)),
GLU())
# DownSample Layer
self.downSample1 = self.downSample(in_channels=128,
out_channels=256,
kernel_size=(3, 3),
stride=(2, 2),
padding=1)
self.downSample2 = self.downSample(in_channels=256,
out_channels=512,
kernel_size=(3, 3),
stride=[2, 2],
padding=1)
self.downSample3 = self.downSample(in_channels=512,
out_channels=1024,
kernel_size=[3, 3],
stride=[2, 2],
padding=1)
self.downSample4 = self.downSample(in_channels=1024,
out_channels=1024,
kernel_size=[1, 10], # [1, 5] for cyclegan-vc2
stride=(1, 1),
padding=(0, 2))
# Conv Layer
self.outputConvLayer = nn.Sequential(nn.Conv2d(in_channels=1024,
out_channels=1,
kernel_size=(1, 3),
stride=[1, 1],
padding=[0, 1]))
def downSample(self, in_channels, out_channels, kernel_size, stride, padding):
convLayer = nn.Sequential(nn.Conv2d(in_channels=in_channels,
out_channels=out_channels,
kernel_size=kernel_size,
stride=stride,
padding=padding),
nn.InstanceNorm2d(num_features=out_channels,
affine=True),
GLU())
return convLayer
def forward(self, input):
# input has shape [batch_size, num_features, time]
# discriminator requires shape [batchSize, 1, num_features, time]
input = input.unsqueeze(1)
# print("Discriminator forward input: ", input.shape)
conv_layer_1 = self.convLayer1(input)
# print("Discriminator forward conv_layer_1: ", conv_layer_1.shape)
downsample1 = self.downSample1(conv_layer_1)
# print("Discriminator forward downsample1: ", downsample1.shape)
downsample2 = self.downSample2(downsample1)
# print("Discriminator forward downsample2: ", downsample2.shape)
downsample3 = self.downSample3(downsample2)
# print("Discriminator forward downsample3: ", downsample3.shape)
# downsample3 = downsample3.contiguous().permute(0, 2, 3, 1).contiguous()
# print("Discriminator forward downsample3: ", downsample3.shape)
output = torch.sigmoid(self.outputConvLayer(downsample3))
# print("Discriminator forward output: ", output.shape)
return output
if __name__ == '__main__':
import sys
import numpy as np
args = sys.argv
print(args)
if len(args) > 1:
if args[1] == "g":
generator = Generator()
print(generator)
elif args[1] == "d":
discriminator = Discriminator()
print(discriminator)
sys.exit(0)
# Generator Dimensionality Testing
input = torch.randn(10, 36, 1100) # (N, C_in, Width) For Conv1d
np.random.seed(0)
# print(np.random.randn(10))
input = np.random.randn(2, 80, 64)
input = torch.from_numpy(input).float()
print("Generator input: ", input.shape)
generator = Generator()
output = generator(input)
print("Generator output shape: ", output.shape)
# Discriminator Dimensionality Testing
# input = torch.randn(32, 1, 24, 128) # (N, C_in, height, width) For Conv2d
discriminator = Discriminator()
output = discriminator(output)
print("Discriminator output shape ", output.shape)
================================================
FILE: tfan_module.py
================================================
#! python
# -*- coding: utf-8 -*-
# Author: kun
# @Time: 2020-11-17 14:35
import re
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.nn.utils.spectral_norm as spectral_norm
# Returns a function that creates a normalization function
def get_norm_layer(opt):
# helper function to get # output channels of the previous layer
def get_out_channel(layer):
if hasattr(layer, 'out_channels'):
return getattr(layer, 'out_channels')
return layer.weight.size(0)
# this function will be returned
def add_norm_layer(layer):
layer = spectral_norm(layer)
# remove bias in the previous layer, which is meaningless
# since it has no effect after normalization
if getattr(layer, 'bias', None) is not None:
delattr(layer, 'bias')
layer.register_parameter('bias', None)
norm_layer = nn.InstanceNorm2d(get_out_channel(layer), affine=False)
return nn.Sequential(layer, norm_layer)
return add_norm_layer
class TFAN_1D(nn.Module):
"""
as paper said, it has best performance when N=3, kernal_size in h is 5
"""
def __init__(self, norm_nc, ks=5, label_nc=128, N=3):
super().__init__()
self.param_free_norm = nn.InstanceNorm1d(norm_nc, affine=False)
self.repeat_N = N
# The dimension of the intermediate embedding space. Yes, hardcoded.
nhidden = 128
pw = ks // 2
self.mlp_shared = nn.Sequential(
nn.Conv1d(label_nc, nhidden, kernel_size=ks, padding=pw),
nn.ReLU()
)
self.mlp_gamma = nn.Conv1d(nhidden, norm_nc, kernel_size=ks, padding=pw)
self.mlp_beta = nn.Conv1d(nhidden, norm_nc, kernel_size=ks, padding=pw)
def forward(self, x, segmap):
# Part 1. generate parameter-free normalized activations
normalized = self.param_free_norm(x)
# Part 2. produce scaling and bias conditioned on semantic map
segmap = F.interpolate(segmap, size=x.size()[2:], mode='nearest')
# actv = self.mlp_shared(segmap)
temp = segmap
for i in range(self.repeat_N):
temp = self.mlp_shared(temp)
actv = temp
gamma = self.mlp_gamma(actv)
beta = self.mlp_beta(actv)
# apply scale and bias
out = normalized * (1 + gamma) + beta
return out
class TFAN_2D(nn.Module):
"""
as paper said, it has best performance when N=3, kernal_size in h is 5
"""
def __init__(self, norm_nc, ks=5, label_nc=128, N=3):
super().__init__()
self.param_free_norm = nn.InstanceNorm2d(norm_nc, affine=False)
self.repeat_N = N
# The dimension of the intermediate embedding space. Yes, hardcoded.
nhidden = 128
pw = ks // 2
self.mlp_shared = nn.Sequential(
nn.Conv2d(label_nc, nhidden, kernel_size=ks, padding=pw),
nn.ReLU()
)
self.mlp_gamma = nn.Conv2d(nhidden, norm_nc, kernel_size=ks, padding=pw)
self.mlp_beta = nn.Conv2d(nhidden, norm_nc, kernel_size=ks, padding=pw)
def forward(self, x, segmap):
# Part 1. generate parameter-free normalized activations
normalized = self.param_free_norm(x)
# Part 2. produce scaling and bias conditioned on semantic map
segmap = F.interpolate(segmap, size=x.size()[2:], mode='nearest')
# actv = self.mlp_shared(segmap)
temp = segmap
for i in range(self.repeat_N):
temp = self.mlp_shared(temp)
actv = temp
gamma = self.mlp_gamma(actv)
beta = self.mlp_beta(actv)
# apply scale and bias
out = normalized * (1 + gamma) + beta
return out
================================================
FILE: train.py
================================================
