![\"flowchart\"](\"https://raw.githubusercontent.com/mtkresearch/BreezyVoice/main/images/flowchart.png\")
\n\n🚀 **Try out our interactive [UI playground](https://huggingface.co/spaces/Splend1dchan/BreezyVoice-Playground) now!** 🚀 \n\n🚀 **[立即體驗 BreezyVoice 語音合成](https://huggingface.co/spaces/Splend1dchan/BreezyVoice-Playground) !** 🚀 \n\nOr visit one of these resources: \n- [Playground (CLI Inference)](https://www.kaggle.com/code/a24998667/breezyvoice-playground) \n- [Model](https://huggingface.co/MediaTek-Research/BreezyVoice/tree/main) \n- [Paper](https://arxiv.org/abs/2501.17790) \n\n\nRepo Main Contributors: Chia-Chun Lin, Chan-Jan Hsu\n\n## Features\n🔥 BreezyVoice outperforms competing commercial services in terms of naturalness.\n\n\n\n![\"comparisons\"](\"https://raw.githubusercontent.com/mtkresearch/BreezyVoice/main/images/comparisons.png\")
\n\n 🔥 BreezyVoice is highly competitive at code-switching scenarios.\n\n| Code-Switching Term Category | **BreezyVoice** | Z | Y | U | M |\n|-------------|--------------|---|---|---|---|\n| **General Words** | **8** | 5 | **8** | **8** | 7 |\n| **Entities**| **9** | 6 | 4 | 7 | 4 |\n| **Abbreviations** | **9** | 8 | 6 | 6 | 7 |\n| **Toponyms**| 3 | 3 | **7** | 3 | 4 |\n| **Full Sentences**| 7 | 7 | **8** | 5 | 3 |\n\n🔥 BreezyVoice supports automatic 注音 annotation, as well as manual 注音 correction (See Inference).\n\n\n## Install\n\n**Clone and install**\n\n- Clone the repo\n``` sh\ngit clone https://github.com/mtkresearch/BreezyVoice.git\n# If you failed to clone submodule due to network failures, please run following command until success\ncd BreezyVoice\n```\n\n- Install Requirements (requires Python3.10)\n```\npip uninstall onnxruntime # use onnxruntime-gpu instead of onnxruntime\npip install -r requirements.txt\n```\n(The model is runnable on CPU, please change onnxruntime-gpu to onnxruntime in `requirements.txt` if you do not have GPU in your environment)\n\nYou might need to install cudnn depending on cuda version\n```\nsudo apt-get -y install cudnn9-cuda-11\n```\n## Inference\n\nUTF8 encoding is required:\n\n``` sh\nexport PYTHONUTF8=1\n```\n---\n**Run single_inference.py with the following arguments:**\n\n- `--content_to_synthesize`:\n - **Description**: Specifies the content that will be synthesized into speech. Phonetic symbols can optionally be included but should be used sparingly, as shown in the examples below:\n - Simple text: `\"今天天氣真好\"`\n - Text with phonetic symbols: `\"今天天氣真好[:ㄏㄠ3]\"`\n\n- `--speaker_prompt_audio_path`:\n - **Description**: Specifies the path to the prompt speech audio file for setting the style of the speaker. Use your custom audio file or our example file:\n - Example audio: `./data/tc_speaker.wav`\n\n- `--speaker_prompt_text_transcription` (optional):\n - **Description**: Specifies the transcription of the speaker prompt audio. Providing this input is highly recommended for better accuracy. If not provided, the system will automatically transcribe the audio using Whisper.\n - Example text for the audio file: `\"在密碼學中，加密是將明文資訊改變為難以讀取的密文內容，使之不可讀的方法。只有擁有解密方法的對象,經由解密過程才能將密文還原為正常可讀的內容。\"`\n\n- `--output_path` (optional):\n - **Description**: Specifies the name and path for the output `.wav` file. If not provided, the default path is used.\n - **Default Value**: `results/output.wav`\n - Example: `[your_file_name].wav`\n\n- `--model_path` (optional):\n - **Description**: Specifies the pre-trained model used for speech synthesis.\n - **Default Value**: `MediaTek-Research/BreezyVoice`\n\n**Example Usage:**\n\n``` bash\nbash run_single_inference.sh\n```\n\n``` python\n# python single_inference.py --text_to_speech [text to be converted into audio] --text_prompt [the prompt of that audio file] --audio_path [reference audio file]\npython single_inference.py --content_to_synthesize \"今天天氣真好\" --speaker_prompt_text_transcription \"在密碼學中，加密是將明文資訊改變為難以讀取的密文內容，使之不可讀的方法。只有擁有解密方法的對象,經由解密過程才能將密文還原為正常可讀的內容。\" --speaker_prompt_audio_path \"./data/example.wav\"\n```\n\n``` python\n# python single_inference.py --text_to_speech [text to be converted into audio] --audio_path [reference audio file]\npython single_inference.py --content_to_synthesize \"今天天氣真好[:ㄏㄠ3]\" --speaker_prompt_audio_path \"./data/example.wav\"\n```\n\n---\n\n**Run `batch_inference.py` with the following arguments:**\n\n- `--csv_file`:\n - **Description**: Path to the CSV file that contains the input data for batch processing.\n - **Example**: `./data/batch_files.csv`\n\n- `--speaker_prompt_audio_folder`:\n - **Description**: Path to the folder containing the speaker prompt audio files. The files in this folder are used to set the style of the speaker for each synthesis task.\n - **Example**: `./data`\n\n- `--output_audio_folder`:\n - **Description**: Path to the folder where the output audio files will be saved. Each processed row in the CSV will result in a synthesized audio file stored in this folder.\n - **Example**: `./results`\n\n**CSV File Structure:**\n\nThe CSV file should contain the following columns:\n\n- **`speaker_prompt_audio_filename`**:\n - **Description**: The filename (without extension) of the speaker prompt audio file that will be used to guide the style of the generated speech.\n - **Example**: `example`\n\n- **`speaker_prompt_text_transcription`**:\n - **Description**: The transcription of the speaker prompt audio. This field is optional but highly recommended to improve transcription accuracy. If not provided, the system will attempt to transcribe the audio using Whisper.\n - **Example**: `\"在密碼學中，加密是將明文資訊改變為難以讀取的密文內容，使之不可讀的方法。\"`\n\n- **`content_to_synthesize`**:\n - **Description**: The content that will be synthesized into speech. You can include phonetic symbols if needed, though they should be used sparingly.\n - **Example**: `\"今天天氣真好\"`\n\n- **`output_audio_filename`**:\n - **Description**: The filename (without extension) for the generated output audio. The audio will be saved as a `.wav` file in the output folder.\n - **Example**: `output`\n\n**Example Usage:**\n\n``` bash\nbash run_batch_inference.sh\n```\n```bash\npython batch_inference.py \\\n --csv_file ./data/batch_files.csv \\\n --speaker_prompt_audio_folder ./data \\\n --output_audio_folder ./results\n```\n\n### Docker and OpenAI Compatible API\n\n``` bash\n$ docker compose up -d --build\n# after the container is up\n$ pip install openai\n$ python openai_api_inference.py\n```\n\n---\n\nIf you like our work, please cite:\n\n```\n@article{hsu2025breezyvoice,\n title={BreezyVoice: Adapting TTS for Taiwanese Mandarin with Enhanced Polyphone Disambiguation--Challenges and Insights},\n author={Hsu, Chan-Jan and Lin, Yi-Cheng and Lin, Chia-Chun and Chen, Wei-Chih and Chung, Ho Lam and Li, Chen-An and Chen, Yi-Chang and Yu, Chien-Yu and Lee, Ming-Ji and Chen, Chien-Cheng and others},\n journal={arXiv preprint arXiv:2501.17790},\n year={2025}\n}\n@article{hsu2025breeze,\n title={The Breeze 2 Herd of Models: Traditional Chinese LLMs Based on Llama with Vision-Aware and Function-Calling Capabilities},\n author={Hsu, Chan-Jan and Liu, Chia-Sheng and Chen, Meng-Hsi and Chen, Muxi and Hsu, Po-Chun and Chen, Yi-Chang and Shiu, Da-Shan},\n journal={arXiv preprint arXiv:2501.13921},\n year={2025}\n}\n@article{du2024cosyvoice,\n title={Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens},\n author={Du, Zhihao and Chen, Qian and Zhang, Shiliang and Hu, Kai and Lu, Heng and Yang, Yexin and Hu, Hangrui and Zheng, Siqi and Gu, Yue and Ma, Ziyang and others},\n journal={arXiv preprint arXiv:2407.05407},\n year={2024}\n}\n```\n"
},
{
"path": "api.py",
"content": "# OpenAI API Spec. Reference: https://platform.openai.com/docs/api-reference/audio/createSpeech\n\nfrom contextlib import asynccontextmanager\nfrom io import BytesIO\n\nimport torchaudio\nfrom fastapi import FastAPI, Request\nfrom fastapi.responses import StreamingResponse\nfrom g2pw import G2PWConverter\nfrom pydantic import BaseModel, Field\nfrom pydantic_settings import BaseSettings\n\nfrom cosyvoice.utils.file_utils import load_wav\nfrom single_inference import CustomCosyVoice, get_bopomofo_rare\n\n\nclass Settings(BaseSettings):\n api_key: str = Field(\n default=\"\", description=\"Specifies the API key used to authenticate the user.\"\n )\n\n model_path: str = Field(\n default=\"MediaTek-Research/BreezyVoice\",\n description=\"Specifies the model used for speech synthesis.\",\n )\n speaker_prompt_audio_path: str = Field(\n default=\"./data/example.wav\",\n description=\"Specifies the path to the prompt speech audio file of the speaker.\",\n )\n speaker_prompt_text_transcription: str = Field(\n default=\"在密碼學中，加密是將明文資訊改變為難以讀取的密文內容，使之不可讀的方法。只有擁有解密方法的對象，經由解密過程，才能將密文還原為正常可讀的內容。\",\n description=\"Specifies the transcription of the speaker prompt audio.\",\n )\n\n\nclass SpeechRequest(BaseModel):\n model: str = \"\"\n input: str = Field(\n description=\"The content that will be synthesized into speech. You can include phonetic symbols if needed, though they should be used sparingly.\",\n examples=[\"今天天氣真好\"],\n )\n response_format: str = \"\"\n speed: float = 1.0\n\n\n@asynccontextmanager\nasync def lifespan(app: FastAPI):\n app.state.settings = Settings()\n app.state.cosyvoice = CustomCosyVoice(app.state.settings.model_path)\n app.state.bopomofo_converter = G2PWConverter()\n app.state.prompt_speech_16k = load_wav(\n app.state.settings.speaker_prompt_audio_path, 16000\n )\n yield\n del app.state.cosyvoice\n del app.state.bopomofo_converter\n\n\napp = FastAPI(lifespan=lifespan, root_path=\"/v1\")\n\n\n@app.get(\"/models\")\nasync def get_models(request: Request):\n return {\n \"object\": \"list\",\n \"data\": [\n {\n \"id\": request.app.state.settings.model_path,\n \"object\": \"model\",\n \"created\": 0,\n \"owned_by\": \"local\",\n }\n ],\n }\n\n\n@app.post(\"/audio/speech\")\nasync def speach_endpoint(request: Request, payload: SpeechRequest):\n # normalization\n speaker_prompt_text_transcription = (\n request.app.state.cosyvoice.frontend.text_normalize_new(\n request.app.state.settings.speaker_prompt_text_transcription, split=False\n )\n )\n content_to_synthesize = request.app.state.cosyvoice.frontend.text_normalize_new(\n payload.input, split=False\n )\n speaker_prompt_text_transcription_bopomo = get_bopomofo_rare(\n speaker_prompt_text_transcription, request.app.state.bopomofo_converter\n )\n\n content_to_synthesize_bopomo = get_bopomofo_rare(\n content_to_synthesize, request.app.state.bopomofo_converter\n )\n output = request.app.state.cosyvoice.inference_zero_shot_no_normalize(\n content_to_synthesize_bopomo,\n speaker_prompt_text_transcription_bopomo,\n request.app.state.prompt_speech_16k,\n )\n audio_buffer = BytesIO()\n torchaudio.save(audio_buffer, output[\"tts_speech\"], 22050, format=\"wav\")\n audio_buffer.seek(0)\n return StreamingResponse(\n audio_buffer,\n media_type=\"audio/wav\",\n headers={\"Content-Disposition\": \"attachment; filename=output.wav\"},\n )\n\n\nif __name__ == \"__main__\":\n import uvicorn\n\n uvicorn.run(\"api:app\", host=\"0.0.0.0\", port=8080)\n"
},
{
"path": "batch_inference.py",
"content": "import os\r\nimport time\r\nimport subprocess\r\nimport argparse\r\nimport pandas as pd\r\nfrom datasets import Dataset\r\nfrom single_inference import single_inference, CustomCosyVoice\r\nfrom g2pw import G2PWConverter\r\n\r\n\r\ndef process_batch(csv_file, speaker_prompt_audio_folder, output_audio_folder, model):\r\n # Load CSV with pandas\r\n data = pd.read_csv(csv_file)\r\n\r\n # Transform pandas DataFrame to HuggingFace Dataset\r\n dataset = Dataset.from_pandas(data)\r\n dataset = dataset.shuffle(seed = int(time.time()*1000))\r\n\r\n cosyvoice, bopomofo_converter = model\r\n\r\n def gen_audio(row):\r\n speaker_prompt_audio_path = os.path.join(speaker_prompt_audio_folder, f\"{row['speaker_prompt_audio_filename']}.wav\")\r\n speaker_prompt_text_transcription = row['speaker_prompt_text_transcription']\r\n content_to_synthesize = row['content_to_synthesize']\r\n output_audio_path = os.path.join(output_audio_folder, f\"{row['output_audio_filename']}.wav\")\r\n\r\n if not os.path.exists(speaker_prompt_audio_path):\r\n print(f\"File {speaker_prompt_audio_path} does not exist\")\r\n return row #{\"status\": \"failed\", \"reason\": \"file not found\"}\r\n if not os.path.exists(output_audio_path):\r\n single_inference(speaker_prompt_audio_path, content_to_synthesize, output_audio_path, cosyvoice, bopomofo_converter, speaker_prompt_text_transcription)\r\n else:\r\n pass\r\n # command = [\r\n # \"python\", \"single_inference.py\",\r\n # \"--speaker_prompt_audio_path\", speaker_prompt_audio_path,\r\n # \"--speaker_prompt_text_transcription\", speaker_prompt_text_transcription,\r\n # \"--content_to_synthesize\", content_to_synthesize,\r\n # \"--output_path\", output_audio_path\r\n # ]\r\n\r\n # try:\r\n # print(f\"Processing: {speaker_prompt_audio_path}\")\r\n # subprocess.run(command, check=True)\r\n # print(f\"Generated: {output_audio_path}\")\r\n # return row #{\"status\": \"success\", \"output\": gen_voice_file_name}\r\n # except subprocess.CalledProcessError as e:\r\n # print(f\"Failed to generate {speaker_prompt_audio_path}, error: {e}\")\r\n # return row #{\"status\": \"failed\", \"reason\": str(e)}\r\n\r\n dataset = dataset.map(gen_audio, num_proc = 1)\r\n\r\ndef main():\r\n parser = argparse.ArgumentParser(description=\"Batch process audio generation.\")\r\n parser.add_argument(\"--csv_file\", required=True, help=\"Path to the CSV file containing input data.\")\r\n parser.add_argument(\"--speaker_prompt_audio_folder\", required=True, help=\"Path to the folder containing speaker prompt audio files.\")\r\n parser.add_argument(\"--output_audio_folder\", required=True, help=\"Path to the folder where results will be stored.\")\r\n parser.add_argument(\"--model_path\", type=str, required=False, default = \"MediaTek-Research/BreezyVoice-300M\",help=\"Specifies the model used for speech synthesis.\")\r\n\r\n args = parser.parse_args()\r\n\r\n cosyvoice = CustomCosyVoice(args.model_path)\r\n bopomofo_converter = G2PWConverter()\r\n\r\n os.makedirs(args.output_audio_folder, exist_ok=True)\r\n\r\n process_batch(\r\n csv_file=args.csv_file,\r\n speaker_prompt_audio_folder=args.speaker_prompt_audio_folder,\r\n output_audio_folder=args.output_audio_folder,\r\n model = (cosyvoice, bopomofo_converter),\r\n\r\n )\r\n\r\nif __name__ == \"__main__\":\r\n main()\r\n\r\n"
},
{
"path": "compose.yaml",
"content": "services:\n app:\n image: breezyvoice:latest\n build: .\n ports:\n - \"8080:8080\"\n volumes:\n - hf-cache:/root/.cache/huggingface/\n command: api.py\n init: true\n deploy:\n resources:\n reservations:\n devices:\n - driver: nvidia\n count: 1\n capabilities: [gpu]\nvolumes:\n hf-cache:"
},
{
"path": "cosyvoice/__init__.py",
"content": ""
},
{
"path": "cosyvoice/bin/inference.py",
"content": "# Copyright (c) 2024 Alibaba Inc (authors: Xiang Lyu)\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nfrom __future__ import print_function\n\nimport argparse\nimport logging\nlogging.getLogger('matplotlib').setLevel(logging.WARNING)\nimport os\n\nimport torch\nfrom torch.utils.data import DataLoader\nimport torchaudio\nfrom hyperpyyaml import load_hyperpyyaml\nfrom tqdm import tqdm\nfrom cosyvoice.cli.model import CosyVoiceModel\n\nfrom cosyvoice.dataset.dataset import Dataset\n\ndef get_args():\n parser = argparse.ArgumentParser(description='inference with your model')\n parser.add_argument('--config', required=True, help='config file')\n parser.add_argument('--prompt_data', required=True, help='prompt data file')\n parser.add_argument('--prompt_utt2data', required=True, help='prompt data file')\n parser.add_argument('--tts_text', required=True, help='tts input file')\n parser.add_argument('--llm_model', required=True, help='llm model file')\n parser.add_argument('--flow_model', required=True, help='flow model file')\n parser.add_argument('--hifigan_model', required=True, help='hifigan model file')\n parser.add_argument('--gpu',\n type=int,\n default=-1,\n help='gpu id for this rank, -1 for cpu')\n parser.add_argument('--mode',\n default='sft',\n choices=['sft', 'zero_shot'],\n help='inference mode')\n parser.add_argument('--result_dir', required=True, help='asr result file')\n args = parser.parse_args()\n print(args)\n return args\n\n\ndef main():\n args = get_args()\n logging.basicConfig(level=logging.DEBUG,\n format='%(asctime)s %(levelname)s %(message)s')\n os.environ['CUDA_VISIBLE_DEVICES'] = str(args.gpu)\n\n # Init cosyvoice models from configs\n use_cuda = args.gpu >= 0 and torch.cuda.is_available()\n device = torch.device('cuda' if use_cuda else 'cpu')\n with open(args.config, 'r') as f:\n configs = load_hyperpyyaml(f)\n\n model = CosyVoiceModel(configs['llm'], configs['flow'], configs['hift'])\n model.load(args.llm_model, args.flow_model, args.hifigan_model)\n\n test_dataset = Dataset(args.prompt_data, data_pipeline=configs['data_pipeline'], mode='inference', shuffle=False, partition=False, tts_file=args.tts_text, prompt_utt2data=args.prompt_utt2data)\n test_data_loader = DataLoader(test_dataset, batch_size=None, num_workers=0)\n\n del configs\n os.makedirs(args.result_dir, exist_ok=True)\n fn = os.path.join(args.result_dir, 'wav.scp')\n f = open(fn, 'w')\n with torch.no_grad():\n for batch_idx, batch in tqdm(enumerate(test_data_loader)):\n utts = batch[\"utts\"]\n assert len(utts) == 1, \"inference mode only support batchsize 1\"\n text = batch[\"text\"]\n text_token = batch[\"text_token\"].to(device)\n text_token_len = batch[\"text_token_len\"].to(device)\n tts_text = batch[\"tts_text\"]\n tts_index = batch[\"tts_index\"]\n tts_text_token = batch[\"tts_text_token\"].to(device)\n tts_text_token_len = batch[\"tts_text_token_len\"].to(device)\n speech_token = batch[\"speech_token\"].to(device)\n speech_token_len = batch[\"speech_token_len\"].to(device)\n speech_feat = batch[\"speech_feat\"].to(device)\n speech_feat_len = batch[\"speech_feat_len\"].to(device)\n utt_embedding = batch[\"utt_embedding\"].to(device)\n spk_embedding = batch[\"spk_embedding\"].to(device)\n if args.mode == 'sft':\n model_input = {'text': tts_text_token, 'text_len': tts_text_token_len,\n 'llm_embedding': spk_embedding, 'flow_embedding': spk_embedding}\n else:\n model_input = {'text': tts_text_token, 'text_len': tts_text_token_len,\n 'prompt_text': text_token, 'prompt_text_len': text_token_len,\n 'llm_prompt_speech_token': speech_token, 'llm_prompt_speech_token_len': speech_token_len,\n 'flow_prompt_speech_token': speech_token, 'flow_prompt_speech_token_len': speech_token_len,\n 'prompt_speech_feat': speech_feat, 'prompt_speech_feat_len': speech_feat_len,\n 'llm_embedding': utt_embedding, 'flow_embedding': utt_embedding}\n model_output = model.inference(**model_input)\n tts_key = '{}_{}'.format(utts[0], tts_index[0])\n tts_fn = os.path.join(args.result_dir, '{}.wav'.format(tts_key))\n torchaudio.save(tts_fn, model_output['tts_speech'], sample_rate=22050)\n f.write('{} {}\\n'.format(tts_key, tts_fn))\n f.flush()\n f.close()\n logging.info('Result wav.scp saved in {}'.format(fn))\n\n\nif __name__ == '__main__':\n main()\n"
},
{
"path": "cosyvoice/bin/train.py",
"content": "# Copyright (c) 2024 Alibaba Inc (authors: Xiang Lyu)\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nfrom __future__ import print_function\nimport argparse\nimport datetime\nimport logging\nlogging.getLogger('matplotlib').setLevel(logging.WARNING)\nfrom copy import deepcopy\nimport torch\nimport torch.distributed as dist\nimport deepspeed\n\nfrom hyperpyyaml import load_hyperpyyaml\n\nfrom torch.distributed.elastic.multiprocessing.errors import record\n\nfrom cosyvoice.utils.executor import Executor\nfrom cosyvoice.utils.train_utils import (\n init_distributed,\n init_dataset_and_dataloader,\n init_optimizer_and_scheduler,\n init_summarywriter, save_model,\n wrap_cuda_model, check_modify_and_save_config)\n\n\ndef get_args():\n parser = argparse.ArgumentParser(description='training your network')\n parser.add_argument('--train_engine',\n default='torch_ddp',\n choices=['torch_ddp', 'deepspeed'],\n help='Engine for paralleled training')\n parser.add_argument('--model', required=True, help='model which will be trained')\n parser.add_argument('--config', required=True, help='config file')\n parser.add_argument('--train_data', required=True, help='train data file')\n parser.add_argument('--cv_data', required=True, help='cv data file')\n parser.add_argument('--checkpoint', help='checkpoint model')\n parser.add_argument('--model_dir', required=True, help='save model dir')\n parser.add_argument('--tensorboard_dir',\n default='tensorboard',\n help='tensorboard log dir')\n parser.add_argument('--ddp.dist_backend',\n dest='dist_backend',\n default='nccl',\n choices=['nccl', 'gloo'],\n help='distributed backend')\n parser.add_argument('--num_workers',\n default=0,\n type=int,\n help='num of subprocess workers for reading')\n parser.add_argument('--prefetch',\n default=100,\n type=int,\n help='prefetch number')\n parser.add_argument('--pin_memory',\n action='store_true',\n default=False,\n help='Use pinned memory buffers used for reading')\n parser.add_argument('--deepspeed.save_states',\n dest='save_states',\n default='model_only',\n choices=['model_only', 'model+optimizer'],\n help='save model/optimizer states')\n parser.add_argument('--timeout',\n default=30,\n type=int,\n help='timeout (in seconds) of cosyvoice_join.')\n parser = deepspeed.add_config_arguments(parser)\n args = parser.parse_args()\n return args\n\n\n@record\ndef main():\n args = get_args()\n logging.basicConfig(level=logging.DEBUG,\n format='%(asctime)s %(levelname)s %(message)s')\n\n override_dict = {k: None for k in ['llm', 'flow', 'hift'] if k != args.model}\n with open(args.config, 'r') as f:\n configs = load_hyperpyyaml(f, overrides=override_dict)\n configs['train_conf'].update(vars(args))\n\n # Init env for ddp\n init_distributed(args)\n\n # Get dataset & dataloader\n train_dataset, cv_dataset, train_data_loader, cv_data_loader = \\\n init_dataset_and_dataloader(args, configs)\n\n # Do some sanity checks and save config to arsg.model_dir\n configs = check_modify_and_save_config(args, configs)\n\n # Tensorboard summary\n writer = init_summarywriter(args)\n\n # load checkpoint\n model = configs[args.model]\n if args.checkpoint is not None:\n model.load_state_dict(torch.load(args.checkpoint, map_location='cpu'))\n\n # Dispatch model from cpu to gpu\n model = wrap_cuda_model(args, model)\n\n # Get optimizer & scheduler\n model, optimizer, scheduler = init_optimizer_and_scheduler(args, configs, model)\n\n # Save init checkpoints\n info_dict = deepcopy(configs['train_conf'])\n save_model(model, 'init', info_dict)\n\n # Get executor\n executor = Executor()\n\n # Start training loop\n for epoch in range(info_dict['max_epoch']):\n executor.epoch = epoch\n train_dataset.set_epoch(epoch)\n dist.barrier()\n group_join = dist.new_group(backend=\"gloo\", timeout=datetime.timedelta(seconds=args.timeout))\n executor.train_one_epoc(model, optimizer, scheduler, train_data_loader, cv_data_loader, writer, info_dict, group_join)\n dist.destroy_process_group(group_join)\n\nif __name__ == '__main__':\n main()\n"
},
{
"path": "cosyvoice/cli/__init__.py",
"content": ""
},
{
"path": "cosyvoice/cli/cosyvoice.py",
"content": "# Copyright (c) 2024 Alibaba Inc (authors: Xiang Lyu)\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport os\nimport torch\nfrom hyperpyyaml import load_hyperpyyaml\nfrom huggingface_hub import snapshot_download\nfrom cosyvoice.cli.frontend import CosyVoiceFrontEnd\nfrom cosyvoice.cli.model import CosyVoiceModel\n\nclass CosyVoice:\n\n def __init__(self, model_dir):\n instruct = True if '-Instruct' in model_dir else False\n self.model_dir = model_dir\n if not os.path.exists(model_dir):\n model_dir = snapshot_download(model_dir)\n with open('{}/cosyvoice.yaml'.format(model_dir), 'r') as f:\n configs = load_hyperpyyaml(f)\n self.frontend = CosyVoiceFrontEnd(configs['get_tokenizer'],\n configs['feat_extractor'],\n '{}/campplus.onnx'.format(model_dir),\n '{}/speech_tokenizer_v1.onnx'.format(model_dir),\n '{}/spk2info.pt'.format(model_dir),\n instruct,\n configs['allowed_special'])\n self.model = CosyVoiceModel(configs['llm'], configs['flow'], configs['hift'])\n self.model.load('{}/llm.pt'.format(model_dir),\n '{}/flow.pt'.format(model_dir),\n '{}/hift.pt'.format(model_dir))\n del configs\n\n def list_avaliable_spks(self):\n spks = list(self.frontend.spk2info.keys())\n return spks\n\n def inference_sft(self, tts_text, spk_id):\n tts_speeches = []\n for i in self.frontend.text_normalize(tts_text, split=True):\n model_input = self.frontend.frontend_sft(i, spk_id)\n model_output = self.model.inference(**model_input)\n tts_speeches.append(model_output['tts_speech'])\n return {'tts_speech': torch.concat(tts_speeches, dim=1)}\n\n def inference_zero_shot(self, tts_text, prompt_text, prompt_speech_16k):\n prompt_text = self.frontend.text_normalize(prompt_text, split=False)\n tts_speeches = []\n for i in self.frontend.text_normalize(tts_text, split=True):\n model_input = self.frontend.frontend_zero_shot(i, prompt_text, prompt_speech_16k)\n model_output = self.model.inference(**model_input)\n tts_speeches.append(model_output['tts_speech'])\n return {'tts_speech': torch.concat(tts_speeches, dim=1)}\n\n def inference_cross_lingual(self, tts_text, prompt_speech_16k):\n if self.frontend.instruct is True:\n raise ValueError('{} do not support cross_lingual inference'.format(self.model_dir))\n tts_speeches = []\n for i in self.frontend.text_normalize(tts_text, split=True):\n model_input = self.frontend.frontend_cross_lingual(i, prompt_speech_16k)\n model_output = self.model.inference(**model_input)\n tts_speeches.append(model_output['tts_speech'])\n return {'tts_speech': torch.concat(tts_speeches, dim=1)}\n\n def inference_instruct(self, tts_text, spk_id, instruct_text):\n if self.frontend.instruct is False:\n raise ValueError('{} do not support instruct inference'.format(self.model_dir))\n instruct_text = self.frontend.text_normalize(instruct_text, split=False)\n tts_speeches = []\n for i in self.frontend.text_normalize(tts_text, split=True):\n model_input = self.frontend.frontend_instruct(i, spk_id, instruct_text)\n model_output = self.model.inference(**model_input)\n tts_speeches.append(model_output['tts_speech'])\n return {'tts_speech': torch.concat(tts_speeches, dim=1)}\n"
},
{
"path": "cosyvoice/cli/frontend.py",
"content": "# Copyright (c) 2024 Alibaba Inc (authors: Xiang Lyu)\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nfrom functools import partial\nimport onnxruntime\nimport torch\nimport numpy as np\nimport whisper\nfrom typing import Callable\nimport torchaudio.compliance.kaldi as kaldi\nimport torchaudio\nimport os\nimport re\nimport inflect\ntry:\n import ttsfrd\n use_ttsfrd = True\nexcept ImportError:\n print(\"failed to import ttsfrd, use WeTextProcessing instead\")\n from tn.chinese.normalizer import Normalizer as ZhNormalizer\n from tn.english.normalizer import Normalizer as EnNormalizer\n use_ttsfrd = False\nfrom cosyvoice.utils.frontend_utils import contains_chinese, replace_blank, replace_corner_mark, remove_bracket, spell_out_number, split_paragraph\n\n\nclass CosyVoiceFrontEnd:\n\n def __init__(self,\n get_tokenizer: Callable,\n feat_extractor: Callable,\n model_dir: str,\n campplus_model: str,\n speech_tokenizer_model: str,\n spk2info: str = '',\n instruct: bool = False,\n allowed_special: str = 'all'):\n self.tokenizer = get_tokenizer()\n self.feat_extractor = feat_extractor\n self.device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n option = onnxruntime.SessionOptions()\n option.graph_optimization_level = onnxruntime.GraphOptimizationLevel.ORT_ENABLE_ALL\n option.intra_op_num_threads = 1\n self.campplus_session = onnxruntime.InferenceSession(campplus_model, sess_options=option, providers=[\"CPUExecutionProvider\"])\n self.speech_tokenizer_session = onnxruntime.InferenceSession(speech_tokenizer_model, sess_options=option, providers=[\"CUDAExecutionProvider\"if torch.cuda.is_available() else \"CPUExecutionProvider\"])\n if os.path.exists(spk2info):\n self.spk2info = torch.load(spk2info, map_location=self.device)\n self.instruct = instruct\n self.allowed_special = allowed_special\n self.inflect_parser = inflect.engine()\n self.use_ttsfrd = use_ttsfrd\n if self.use_ttsfrd:\n self.frd = ttsfrd.TtsFrontendEngine()\n ROOT_DIR = os.path.dirname(os.path.abspath(__file__))\n assert self.frd.initialize('{}/CosyVoice-ttsfrd/resource'.format(model_dir)) is True, 'failed to initialize ttsfrd resource'\n self.frd.set_lang_type('pinyin')\n self.frd.enable_pinyin_mix(True)\n self.frd.set_breakmodel_index(1)\n else:\n self.zh_tn_model = ZhNormalizer(remove_erhua=False, full_to_half=False)\n self.en_tn_model = EnNormalizer()\n\n def _extract_text_token(self, text):\n text_token = self.tokenizer.encode(text, allowed_special=self.allowed_special)\n text_token = torch.tensor([text_token], dtype=torch.int32).to(self.device)\n text_token_len = torch.tensor([text_token.shape[1]], dtype=torch.int32).to(self.device)\n return text_token, text_token_len\n\n def _extract_speech_token(self, speech):\n feat = whisper.log_mel_spectrogram(speech, n_mels=128)\n speech_token = self.speech_tokenizer_session.run(None, {self.speech_tokenizer_session.get_inputs()[0].name: feat.detach().cpu().numpy(),\n self.speech_tokenizer_session.get_inputs()[1].name: np.array([feat.shape[2]], dtype=np.int32)})[0].flatten().tolist()\n speech_token = torch.tensor([speech_token], dtype=torch.int32).to(self.device)\n speech_token_len = torch.tensor([speech_token.shape[1]], dtype=torch.int32).to(self.device)\n return speech_token, speech_token_len\n\n def _extract_spk_embedding(self, speech):\n feat = kaldi.fbank(speech,\n num_mel_bins=80,\n dither=0,\n sample_frequency=16000)\n feat = feat - feat.mean(dim=0, keepdim=True)\n embedding = self.campplus_session.run(None, {self.campplus_session.get_inputs()[0].name: feat.unsqueeze(dim=0).cpu().numpy()})[0].flatten().tolist()\n embedding = torch.tensor([embedding]).to(self.device)\n return embedding\n\n def _extract_speech_feat(self, speech):\n speech_feat = self.feat_extractor(speech).squeeze(dim=0).transpose(0, 1).to(self.device)\n speech_feat = speech_feat.unsqueeze(dim=0)\n speech_feat_len = torch.tensor([speech_feat.shape[1]], dtype=torch.int32).to(self.device)\n return speech_feat, speech_feat_len\n\n def text_normalize(self, text, split=True):\n text = text.strip()\n if contains_chinese(text):\n if self.use_ttsfrd:\n text = self.frd.get_frd_extra_info(text, 'input')\n else:\n text = self.zh_tn_model.normalize(text)\n text = text.replace(\"\\n\", \"\")\n text = replace_blank(text)\n text = replace_corner_mark(text)\n text = text.replace(\".\", \"、\")\n text = text.replace(\" - \", \"，\")\n text = remove_bracket(text)\n text = re.sub(r'[，,]+$', '。', text)\n texts = [i for i in split_paragraph(text, partial(self.tokenizer.encode, allowed_special=self.allowed_special), \"zh\", token_max_n=80,\n token_min_n=60, merge_len=20,\n comma_split=False)]\n else:\n if self.use_ttsfrd:\n text = self.frd.get_frd_extra_info(text, 'input')\n else:\n text = self.en_tn_model.normalize(text)\n text = spell_out_number(text, self.inflect_parser)\n texts = [i for i in split_paragraph(text, partial(self.tokenizer.encode, allowed_special=self.allowed_special), \"en\", token_max_n=80,\n token_min_n=60, merge_len=20,\n comma_split=False)]\n if split is False:\n return text\n return texts\n\n def frontend_sft(self, tts_text, spk_id):\n tts_text_token, tts_text_token_len = self._extract_text_token(tts_text)\n embedding = self.spk2info[spk_id]['embedding']\n model_input = {'text': tts_text_token, 'text_len': tts_text_token_len, 'llm_embedding': embedding, 'flow_embedding': embedding}\n return model_input\n\n def frontend_zero_shot(self, tts_text, prompt_text, prompt_speech_16k):\n tts_text_token, tts_text_token_len = self._extract_text_token(tts_text)\n prompt_text_token, prompt_text_token_len = self._extract_text_token(prompt_text)\n prompt_speech_22050 = torchaudio.transforms.Resample(orig_freq=16000, new_freq=22050)(prompt_speech_16k)\n speech_feat, speech_feat_len = self._extract_speech_feat(prompt_speech_22050)\n speech_token, speech_token_len = self._extract_speech_token(prompt_speech_16k)\n embedding = self._extract_spk_embedding(prompt_speech_16k)\n model_input = {'text': tts_text_token, 'text_len': tts_text_token_len,\n 'prompt_text': prompt_text_token, 'prompt_text_len': prompt_text_token_len,\n 'llm_prompt_speech_token': speech_token, 'llm_prompt_speech_token_len': speech_token_len,\n 'flow_prompt_speech_token': speech_token, 'flow_prompt_speech_token_len': speech_token_len,\n 'prompt_speech_feat': speech_feat, 'prompt_speech_feat_len': speech_feat_len,\n 'llm_embedding': embedding, 'flow_embedding': embedding}\n return model_input\n\n def frontend_cross_lingual(self, tts_text, prompt_speech_16k):\n model_input = self.frontend_zero_shot(tts_text, '', prompt_speech_16k)\n # in cross lingual mode, we remove prompt in llm\n del model_input['prompt_text']\n del model_input['prompt_text_len']\n del model_input['llm_prompt_speech_token']\n del model_input['llm_prompt_speech_token_len']\n return model_input\n\n def frontend_instruct(self, tts_text, spk_id, instruct_text):\n model_input = self.frontend_sft(tts_text, spk_id)\n # in instruct mode, we remove spk_embedding in llm due to information leakage\n del model_input['llm_embedding']\n instruct_text_token, instruct_text_token_len = self._extract_text_token(instruct_text + '\n\n# 🍵 Matcha-TTS: A fast TTS architecture with conditional flow matching\n\n### [Shivam Mehta](https://www.kth.se/profile/smehta), [Ruibo Tu](https://www.kth.se/profile/ruibo), [Jonas Beskow](https://www.kth.se/profile/beskow), [Éva Székely](https://www.kth.se/profile/szekely), and [Gustav Eje Henter](https://people.kth.se/~ghe/)\n\n[](https://www.python.org/downloads/release/python-3100/)\n[](https://pytorch.org/get-started/locally/)\n[](https://pytorchlightning.ai/)\n[](https://hydra.cc/)\n[](https://black.readthedocs.io/en/stable/)\n[](https://pycqa.github.io/isort/)\n\n
\n\n> This is the official code implementation of 🍵 Matcha-TTS [ICASSP 2024].\n\nWe propose 🍵 Matcha-TTS, a new approach to non-autoregressive neural TTS, that uses [conditional flow matching](https://arxiv.org/abs/2210.02747) (similar to [rectified flows](https://arxiv.org/abs/2209.03003)) to speed up ODE-based speech synthesis. Our method:\n\n- Is probabilistic\n- Has compact memory footprint\n- Sounds highly natural\n- Is very fast to synthesise from\n\nCheck out our [demo page](https://shivammehta25.github.io/Matcha-TTS) and read [our ICASSP 2024 paper](https://arxiv.org/abs/2309.03199) for more details.\n\n[Pre-trained models](https://drive.google.com/drive/folders/17C_gYgEHOxI5ZypcfE_k1piKCtyR0isJ?usp=sharing) will be automatically downloaded with the CLI or gradio interface.\n\nYou can also [try 🍵 Matcha-TTS in your browser on HuggingFace 🤗 spaces](https://huggingface.co/spaces/shivammehta25/Matcha-TTS).\n\n## Teaser video\n\n[](https://youtu.be/xmvJkz3bqw0)\n\n## Installation\n\n1. Create an environment (suggested but optional)\n\n```\nconda create -n matcha-tts python=3.10 -y\nconda activate matcha-tts\n```\n\n2. Install Matcha TTS using pip or from source\n\n```bash\npip install matcha-tts\n```\n\nfrom source\n\n```bash\npip install git+https://github.com/shivammehta25/Matcha-TTS.git\ncd Matcha-TTS\npip install -e .\n```\n\n3. Run CLI / gradio app / jupyter notebook\n\n```bash\n# This will download the required models\nmatcha-tts --text \"\"\n```\n\nor\n\n```bash\nmatcha-tts-app\n```\n\nor open `synthesis.ipynb` on jupyter notebook\n\n### CLI Arguments\n\n- To synthesise from given text, run:\n\n```bash\nmatcha-tts --text \"\"\n```\n\n- To synthesise from a file, run:\n\n```bash\nmatcha-tts --file \n ![](\"https://shivammehta25.github.io/Matcha-TTS/images/logo.png\"){kind=logo}
\n
'\n gr.HTML(html)\n\n with gr.Box():\n radio_options = list(RADIO_OPTIONS.keys())\n model_type = gr.Radio(\n radio_options, value=radio_options[0], label=\"Choose a Model\", interactive=True, container=False\n )\n\n with gr.Row():\n gr.Markdown(\"# Text Input\")\n with gr.Row():\n text = gr.Textbox(value=\"\", lines=2, label=\"Text to synthesise\", scale=3)\n spk_slider = gr.Slider(\n minimum=0, maximum=107, step=1, value=args.spk, label=\"Speaker ID\", interactive=True, scale=1\n )\n\n with gr.Row():\n gr.Markdown(\"### Hyper parameters\")\n with gr.Row():\n n_timesteps = gr.Slider(\n label=\"Number of ODE steps\",\n minimum=1,\n maximum=100,\n step=1,\n value=10,\n interactive=True,\n )\n length_scale = gr.Slider(\n label=\"Length scale (Speaking rate)\",\n minimum=0.5,\n maximum=1.5,\n step=0.05,\n value=1.0,\n interactive=True,\n )\n mel_temp = gr.Slider(\n label=\"Sampling temperature\",\n minimum=0.00,\n maximum=2.001,\n step=0.16675,\n value=0.667,\n interactive=True,\n )\n\n synth_btn = gr.Button(\"Synthesise\")\n\n with gr.Box():\n with gr.Row():\n gr.Markdown(\"### Phonetised text\")\n phonetised_text = gr.Textbox(interactive=False, scale=10, label=\"Phonetised text\")\n\n with gr.Box():\n with gr.Row():\n mel_spectrogram = gr.Image(interactive=False, label=\"mel spectrogram\")\n\n # with gr.Row():\n audio = gr.Audio(interactive=False, label=\"Audio\")\n\n with gr.Row(visible=False) as example_row_lj_speech:\n examples = gr.Examples( # pylint: disable=unused-variable\n examples=[\n [\n \"We propose Matcha-TTS, a new approach to non-autoregressive neural TTS, that uses conditional flow matching (similar to rectified flows) to speed up O D E-based speech synthesis.\",\n 50,\n 0.677,\n 0.95,\n ],\n [\n \"The Secret Service believed that it was very doubtful that any President would ride regularly in a vehicle with a fixed top, even though transparent.\",\n 2,\n 0.677,\n 0.95,\n ],\n [\n \"The Secret Service believed that it was very doubtful that any President would ride regularly in a vehicle with a fixed top, even though transparent.\",\n 4,\n 0.677,\n 0.95,\n ],\n [\n \"The Secret Service believed that it was very doubtful that any President would ride regularly in a vehicle with a fixed top, even though transparent.\",\n 10,\n 0.677,\n 0.95,\n ],\n [\n \"The Secret Service believed that it was very doubtful that any President would ride regularly in a vehicle with a fixed top, even though transparent.\",\n 50,\n 0.677,\n 0.95,\n ],\n [\n \"The narrative of these events is based largely on the recollections of the participants.\",\n 10,\n 0.677,\n 0.95,\n ],\n [\n \"The jury did not believe him, and the verdict was for the defendants.\",\n 10,\n 0.677,\n 0.95,\n ],\n ],\n fn=ljspeech_example_cacher,\n inputs=[text, n_timesteps, mel_temp, length_scale],\n outputs=[phonetised_text, audio, mel_spectrogram],\n cache_examples=True,\n )\n\n with gr.Row() as example_row_multispeaker:\n multi_speaker_examples = gr.Examples( # pylint: disable=unused-variable\n examples=[\n [\n \"Hello everyone! I am speaker 0 and I am here to tell you that Matcha-TTS is amazing!\",\n 10,\n 0.677,\n 0.85,\n 0,\n ],\n [\n \"Hello everyone! I am speaker 16 and I am here to tell you that Matcha-TTS is amazing!\",\n 10,\n 0.677,\n 0.85,\n 16,\n ],\n [\n \"Hello everyone! I am speaker 44 and I am here to tell you that Matcha-TTS is amazing!\",\n 50,\n 0.677,\n 0.85,\n 44,\n ],\n [\n \"Hello everyone! I am speaker 45 and I am here to tell you that Matcha-TTS is amazing!\",\n 50,\n 0.677,\n 0.85,\n 45,\n ],\n [\n \"Hello everyone! I am speaker 58 and I am here to tell you that Matcha-TTS is amazing!\",\n 4,\n 0.677,\n 0.85,\n 58,\n ],\n ],\n fn=multispeaker_example_cacher,\n inputs=[text, n_timesteps, mel_temp, length_scale, spk_slider],\n outputs=[phonetised_text, audio, mel_spectrogram],\n cache_examples=True,\n label=\"Multi Speaker Examples\",\n )\n\n model_type.change(lambda x: gr.update(interactive=False), inputs=[synth_btn], outputs=[synth_btn]).then(\n load_model_ui,\n inputs=[model_type, text],\n outputs=[text, synth_btn, spk_slider, example_row_lj_speech, example_row_multispeaker, length_scale],\n )\n\n synth_btn.click(\n fn=process_text_gradio,\n inputs=[\n text,\n ],\n outputs=[phonetised_text, processed_text, processed_text_len],\n api_name=\"matcha_tts\",\n queue=True,\n ).then(\n fn=synthesise_mel,\n inputs=[processed_text, processed_text_len, n_timesteps, mel_temp, length_scale, spk_slider],\n outputs=[audio, mel_spectrogram],\n )\n\n demo.queue().launch(share=True)\n\n\nif __name__ == \"__main__\":\n main()\n" }, { "path": "third_party/Matcha-TTS/matcha/cli.py", "content": "import argparse\nimport datetime as dt\nimport os\nimport warnings\nfrom pathlib import Path\n\nimport matplotlib.pyplot as plt\nimport numpy as np\nimport soundfile as sf\nimport torch\n\nfrom matcha.hifigan.config import v1\nfrom matcha.hifigan.denoiser import Denoiser\nfrom matcha.hifigan.env import AttrDict\nfrom matcha.hifigan.models import Generator as HiFiGAN\nfrom matcha.models.matcha_tts import MatchaTTS\nfrom matcha.text import sequence_to_text, text_to_sequence\nfrom matcha.utils.utils import assert_model_downloaded, get_user_data_dir, intersperse\n\nMATCHA_URLS = {\n \"matcha_ljspeech\": \"https://github.com/shivammehta25/Matcha-TTS-checkpoints/releases/download/v1.0/matcha_ljspeech.ckpt\",\n \"matcha_vctk\": \"https://github.com/shivammehta25/Matcha-TTS-checkpoints/releases/download/v1.0/matcha_vctk.ckpt\",\n}\n\nVOCODER_URLS = {\n \"hifigan_T2_v1\": \"https://github.com/shivammehta25/Matcha-TTS-checkpoints/releases/download/v1.0/generator_v1\", # Old url: https://drive.google.com/file/d/14NENd4equCBLyyCSke114Mv6YR_j_uFs/view?usp=drive_link\n \"hifigan_univ_v1\": \"https://github.com/shivammehta25/Matcha-TTS-checkpoints/releases/download/v1.0/g_02500000\", # Old url: https://drive.google.com/file/d/1qpgI41wNXFcH-iKq1Y42JlBC9j0je8PW/view?usp=drive_link\n}\n\nMULTISPEAKER_MODEL = {\n \"matcha_vctk\": {\"vocoder\": \"hifigan_univ_v1\", \"speaking_rate\": 0.85, \"spk\": 0, \"spk_range\": (0, 107)}\n}\n\nSINGLESPEAKER_MODEL = {\"matcha_ljspeech\": {\"vocoder\": \"hifigan_T2_v1\", \"speaking_rate\": 0.95, \"spk\": None}}\n\n\ndef plot_spectrogram_to_numpy(spectrogram, filename):\n fig, ax = plt.subplots(figsize=(12, 3))\n im = ax.imshow(spectrogram, aspect=\"auto\", origin=\"lower\", interpolation=\"none\")\n plt.colorbar(im, ax=ax)\n plt.xlabel(\"Frames\")\n plt.ylabel(\"Channels\")\n plt.title(\"Synthesised Mel-Spectrogram\")\n fig.canvas.draw()\n plt.savefig(filename)\n\n\ndef process_text(i: int, text: str, device: torch.device):\n print(f\"[{i}] - Input text: {text}\")\n x = torch.tensor(\n intersperse(text_to_sequence(text, [\"english_cleaners2\"]), 0),\n dtype=torch.long,\n device=device,\n )[None]\n x_lengths = torch.tensor([x.shape[-1]], dtype=torch.long, device=device)\n x_phones = sequence_to_text(x.squeeze(0).tolist())\n print(f\"[{i}] - Phonetised text: {x_phones[1::2]}\")\n\n return {\"x_orig\": text, \"x\": x, \"x_lengths\": x_lengths, \"x_phones\": x_phones}\n\n\ndef get_texts(args):\n if args.text:\n texts = [args.text]\n else:\n with open(args.file, encoding=\"utf-8\") as f:\n texts = f.readlines()\n return texts\n\n\ndef assert_required_models_available(args):\n save_dir = get_user_data_dir()\n if not hasattr(args, \"checkpoint_path\") and args.checkpoint_path is None:\n model_path = args.checkpoint_path\n else:\n model_path = save_dir / f\"{args.model}.ckpt\"\n assert_model_downloaded(model_path, MATCHA_URLS[args.model])\n\n vocoder_path = save_dir / f\"{args.vocoder}\"\n assert_model_downloaded(vocoder_path, VOCODER_URLS[args.vocoder])\n return {\"matcha\": model_path, \"vocoder\": vocoder_path}\n\n\ndef load_hifigan(checkpoint_path, device):\n h = AttrDict(v1)\n hifigan = HiFiGAN(h).to(device)\n hifigan.load_state_dict(torch.load(checkpoint_path, map_location=device)[\"generator\"])\n _ = hifigan.eval()\n hifigan.remove_weight_norm()\n return hifigan\n\n\ndef load_vocoder(vocoder_name, checkpoint_path, device):\n print(f\"[!] Loading {vocoder_name}!\")\n vocoder = None\n if vocoder_name in (\"hifigan_T2_v1\", \"hifigan_univ_v1\"):\n vocoder = load_hifigan(checkpoint_path, device)\n else:\n raise NotImplementedError(\n f\"Vocoder {vocoder_name} not implemented! define a load_<
\nWe provide our implementation and pretrained models as open source in this repository.\n\n**Abstract :**\nSeveral recent work on speech synthesis have employed generative adversarial networks (GANs) to produce raw waveforms.\nAlthough such methods improve the sampling efficiency and memory usage,\ntheir sample quality has not yet reached that of autoregressive and flow-based generative models.\nIn this work, we propose HiFi-GAN, which achieves both efficient and high-fidelity speech synthesis.\nAs speech audio consists of sinusoidal signals with various periods,\nwe demonstrate that modeling periodic patterns of an audio is crucial for enhancing sample quality.\nA subjective human evaluation (mean opinion score, MOS) of a single speaker dataset indicates that our proposed method\ndemonstrates similarity to human quality while generating 22.05 kHz high-fidelity audio 167.9 times faster than\nreal-time on a single V100 GPU. We further show the generality of HiFi-GAN to the mel-spectrogram inversion of unseen\nspeakers and end-to-end speech synthesis. Finally, a small footprint version of HiFi-GAN generates samples 13.4 times\nfaster than real-time on CPU with comparable quality to an autoregressive counterpart.\n\nVisit our [demo website](https://jik876.github.io/hifi-gan-demo/) for audio samples.\n\n## Pre-requisites\n\n1. Python >= 3.6\n2. Clone this repository.\n3. Install python requirements. Please refer [requirements.txt](requirements.txt)\n4. Download and extract the [LJ Speech dataset](https://keithito.com/LJ-Speech-Dataset/).\n And move all wav files to `LJSpeech-1.1/wavs`\n\n## Training\n\n```\npython train.py --config config_v1.json\n```\n\nTo train V2 or V3 Generator, replace `config_v1.json` with `config_v2.json` or `config_v3.json`.
\nCheckpoints and copy of the configuration file are saved in `cp_hifigan` directory by default.
\nYou can change the path by adding `--checkpoint_path` option.\n\nValidation loss during training with V1 generator.
\n\n\n## Pretrained Model\n\nYou can also use pretrained models we provide.
\n[Download pretrained models](https://drive.google.com/drive/folders/1-eEYTB5Av9jNql0WGBlRoi-WH2J7bp5Y?usp=sharing)
\nDetails of each folder are as in follows:\n\n| Folder Name | Generator | Dataset | Fine-Tuned |\n| ------------ | --------- | --------- | ------------------------------------------------------ |\n| LJ_V1 | V1 | LJSpeech | No |\n| LJ_V2 | V2 | LJSpeech | No |\n| LJ_V3 | V3 | LJSpeech | No |\n| LJ_FT_T2_V1 | V1 | LJSpeech | Yes ([Tacotron2](https://github.com/NVIDIA/tacotron2)) |\n| LJ_FT_T2_V2 | V2 | LJSpeech | Yes ([Tacotron2](https://github.com/NVIDIA/tacotron2)) |\n| LJ_FT_T2_V3 | V3 | LJSpeech | Yes ([Tacotron2](https://github.com/NVIDIA/tacotron2)) |\n| VCTK_V1 | V1 | VCTK | No |\n| VCTK_V2 | V2 | VCTK | No |\n| VCTK_V3 | V3 | VCTK | No |\n| UNIVERSAL_V1 | V1 | Universal | No |\n\nWe provide the universal model with discriminator weights that can be used as a base for transfer learning to other datasets.\n\n## Fine-Tuning\n\n1. Generate mel-spectrograms in numpy format using [Tacotron2](https://github.com/NVIDIA/tacotron2) with teacher-forcing.
\n The file name of the generated mel-spectrogram should match the audio file and the extension should be `.npy`.
\n Example:\n ` Audio File : LJ001-0001.wav\nMel-Spectrogram File : LJ001-0001.npy`\n2. Create `ft_dataset` folder and copy the generated mel-spectrogram files into it.
\n3. Run the following command.\n ```\n python train.py --fine_tuning True --config config_v1.json\n ```\n For other command line options, please refer to the training section.\n\n## Inference from wav file\n\n1. Make `test_files` directory and copy wav files into the directory.\n2. Run the following command.\n ` python inference.py --checkpoint_file [generator checkpoint file path]`\n Generated wav files are saved in `generated_files` by default.
\n You can change the path by adding `--output_dir` option.\n\n## Inference for end-to-end speech synthesis\n\n1. Make `test_mel_files` directory and copy generated mel-spectrogram files into the directory.
\n You can generate mel-spectrograms using [Tacotron2](https://github.com/NVIDIA/tacotron2),\n [Glow-TTS](https://github.com/jaywalnut310/glow-tts) and so forth.\n2. Run the following command.\n ` python inference_e2e.py --checkpoint_file [generator checkpoint file path]`\n Generated wav files are saved in `generated_files_from_mel` by default.
\n You can change the path by adding `--output_dir` option.\n\n## Acknowledgements\n\nWe referred to [WaveGlow](https://github.com/NVIDIA/waveglow), [MelGAN](https://github.com/descriptinc/melgan-neurips)\nand [Tacotron2](https://github.com/NVIDIA/tacotron2) to implement this.\n" }, { "path": "third_party/Matcha-TTS/matcha/hifigan/__init__.py", "content": "" }, { "path": "third_party/Matcha-TTS/matcha/hifigan/config.py", "content": "v1 = {\n \"resblock\": \"1\",\n \"num_gpus\": 0,\n \"batch_size\": 16,\n \"learning_rate\": 0.0004,\n \"adam_b1\": 0.8,\n \"adam_b2\": 0.99,\n \"lr_decay\": 0.999,\n \"seed\": 1234,\n \"upsample_rates\": [8, 8, 2, 2],\n \"upsample_kernel_sizes\": [16, 16, 4, 4],\n \"upsample_initial_channel\": 512,\n \"resblock_kernel_sizes\": [3, 7, 11],\n \"resblock_dilation_sizes\": [[1, 3, 5], [1, 3, 5], [1, 3, 5]],\n \"resblock_initial_channel\": 256,\n \"segment_size\": 8192,\n \"num_mels\": 80,\n \"num_freq\": 1025,\n \"n_fft\": 1024,\n \"hop_size\": 256,\n \"win_size\": 1024,\n \"sampling_rate\": 22050,\n \"fmin\": 0,\n \"fmax\": 8000,\n \"fmax_loss\": None,\n \"num_workers\": 4,\n \"dist_config\": {\"dist_backend\": \"nccl\", \"dist_url\": \"tcp://localhost:54321\", \"world_size\": 1},\n}\n" }, { "path": "third_party/Matcha-TTS/matcha/hifigan/denoiser.py", "content": "# Code modified from Rafael Valle's implementation https://github.com/NVIDIA/waveglow/blob/5bc2a53e20b3b533362f974cfa1ea0267ae1c2b1/denoiser.py\n\n\"\"\"Waveglow style denoiser can be used to remove the artifacts from the HiFiGAN generated audio.\"\"\"\nimport torch\n\n\nclass Denoiser(torch.nn.Module):\n \"\"\"Removes model bias from audio produced with waveglow\"\"\"\n\n def __init__(self, vocoder, filter_length=1024, n_overlap=4, win_length=1024, mode=\"zeros\"):\n super().__init__()\n self.filter_length = filter_length\n self.hop_length = int(filter_length / n_overlap)\n self.win_length = win_length\n\n dtype, device = next(vocoder.parameters()).dtype, next(vocoder.parameters()).device\n self.device = device\n if mode == \"zeros\":\n mel_input = torch.zeros((1, 80, 88), dtype=dtype, device=device)\n elif mode == \"normal\":\n mel_input = torch.randn((1, 80, 88), dtype=dtype, device=device)\n else:\n raise Exception(f\"Mode {mode} if not supported\")\n\n def stft_fn(audio, n_fft, hop_length, win_length, window):\n spec = torch.stft(\n audio,\n n_fft=n_fft,\n hop_length=hop_length,\n win_length=win_length,\n window=window,\n return_complex=True,\n )\n spec = torch.view_as_real(spec)\n return torch.sqrt(spec.pow(2).sum(-1)), torch.atan2(spec[..., -1], spec[..., 0])\n\n self.stft = lambda x: stft_fn(\n audio=x,\n n_fft=self.filter_length,\n hop_length=self.hop_length,\n win_length=self.win_length,\n window=torch.hann_window(self.win_length, device=device),\n )\n self.istft = lambda x, y: torch.istft(\n torch.complex(x * torch.cos(y), x * torch.sin(y)),\n n_fft=self.filter_length,\n hop_length=self.hop_length,\n win_length=self.win_length,\n window=torch.hann_window(self.win_length, device=device),\n )\n\n with torch.no_grad():\n bias_audio = vocoder(mel_input).float().squeeze(0)\n bias_spec, _ = self.stft(bias_audio)\n\n self.register_buffer(\"bias_spec\", bias_spec[:, :, 0][:, :, None])\n\n @torch.inference_mode()\n def forward(self, audio, strength=0.0005):\n audio_spec, audio_angles = self.stft(audio)\n audio_spec_denoised = audio_spec - self.bias_spec.to(audio.device) * strength\n audio_spec_denoised = torch.clamp(audio_spec_denoised, 0.0)\n audio_denoised = self.istft(audio_spec_denoised, audio_angles)\n return audio_denoised\n" }, { "path": "third_party/Matcha-TTS/matcha/hifigan/env.py", "content": "\"\"\" from https://github.com/jik876/hifi-gan \"\"\"\n\nimport os\nimport shutil\n\n\nclass AttrDict(dict):\n def __init__(self, *args, **kwargs):\n super().__init__(*args, **kwargs)\n self.__dict__ = self\n\n\ndef build_env(config, config_name, path):\n t_path = os.path.join(path, config_name)\n if config != t_path:\n os.makedirs(path, exist_ok=True)\n shutil.copyfile(config, os.path.join(path, config_name))\n" }, { "path": "third_party/Matcha-TTS/matcha/hifigan/meldataset.py", "content": "\"\"\" from https://github.com/jik876/hifi-gan \"\"\"\n\nimport math\nimport os\nimport random\n\nimport numpy as np\nimport torch\nimport torch.utils.data\nfrom librosa.filters import mel as librosa_mel_fn\nfrom librosa.util import normalize\nfrom scipy.io.wavfile import read\n\nMAX_WAV_VALUE = 32768.0\n\n\ndef load_wav(full_path):\n sampling_rate, data = read(full_path)\n return data, sampling_rate\n\n\ndef dynamic_range_compression(x, C=1, clip_val=1e-5):\n return np.log(np.clip(x, a_min=clip_val, a_max=None) * C)\n\n\ndef dynamic_range_decompression(x, C=1):\n return np.exp(x) / C\n\n\ndef dynamic_range_compression_torch(x, C=1, clip_val=1e-5):\n return torch.log(torch.clamp(x, min=clip_val) * C)\n\n\ndef dynamic_range_decompression_torch(x, C=1):\n return torch.exp(x) / C\n\n\ndef spectral_normalize_torch(magnitudes):\n output = dynamic_range_compression_torch(magnitudes)\n return output\n\n\ndef spectral_de_normalize_torch(magnitudes):\n output = dynamic_range_decompression_torch(magnitudes)\n return output\n\n\nmel_basis = {}\nhann_window = {}\n\n\ndef mel_spectrogram(y, n_fft, num_mels, sampling_rate, hop_size, win_size, fmin, fmax, center=False):\n if torch.min(y) < -1.0:\n print(\"min value is \", torch.min(y))\n if torch.max(y) > 1.0:\n print(\"max value is \", torch.max(y))\n\n global mel_basis, hann_window # pylint: disable=global-statement\n if fmax not in mel_basis:\n mel = librosa_mel_fn(sampling_rate, n_fft, num_mels, fmin, fmax)\n mel_basis[str(fmax) + \"_\" + str(y.device)] = torch.from_numpy(mel).float().to(y.device)\n hann_window[str(y.device)] = torch.hann_window(win_size).to(y.device)\n\n y = torch.nn.functional.pad(\n y.unsqueeze(1), (int((n_fft - hop_size) / 2), int((n_fft - hop_size) / 2)), mode=\"reflect\"\n )\n y = y.squeeze(1)\n\n spec = torch.view_as_real(\n torch.stft(\n y,\n n_fft,\n hop_length=hop_size,\n win_length=win_size,\n window=hann_window[str(y.device)],\n center=center,\n pad_mode=\"reflect\",\n normalized=False,\n onesided=True,\n return_complex=True,\n )\n )\n\n spec = torch.sqrt(spec.pow(2).sum(-1) + (1e-9))\n\n spec = torch.matmul(mel_basis[str(fmax) + \"_\" + str(y.device)], spec)\n spec = spectral_normalize_torch(spec)\n\n return spec\n\n\ndef get_dataset_filelist(a):\n with open(a.input_training_file, encoding=\"utf-8\") as fi:\n training_files = [\n os.path.join(a.input_wavs_dir, x.split(\"|\")[0] + \".wav\") for x in fi.read().split(\"\\n\") if len(x) > 0\n ]\n\n with open(a.input_validation_file, encoding=\"utf-8\") as fi:\n validation_files = [\n os.path.join(a.input_wavs_dir, x.split(\"|\")[0] + \".wav\") for x in fi.read().split(\"\\n\") if len(x) > 0\n ]\n return training_files, validation_files\n\n\nclass MelDataset(torch.utils.data.Dataset):\n def __init__(\n self,\n training_files,\n segment_size,\n n_fft,\n num_mels,\n hop_size,\n win_size,\n sampling_rate,\n fmin,\n fmax,\n split=True,\n shuffle=True,\n n_cache_reuse=1,\n device=None,\n fmax_loss=None,\n fine_tuning=False,\n base_mels_path=None,\n ):\n self.audio_files = training_files\n random.seed(1234)\n if shuffle:\n random.shuffle(self.audio_files)\n self.segment_size = segment_size\n self.sampling_rate = sampling_rate\n self.split = split\n self.n_fft = n_fft\n self.num_mels = num_mels\n self.hop_size = hop_size\n self.win_size = win_size\n self.fmin = fmin\n self.fmax = fmax\n self.fmax_loss = fmax_loss\n self.cached_wav = None\n self.n_cache_reuse = n_cache_reuse\n self._cache_ref_count = 0\n self.device = device\n self.fine_tuning = fine_tuning\n self.base_mels_path = base_mels_path\n\n def __getitem__(self, index):\n filename = self.audio_files[index]\n if self._cache_ref_count == 0:\n audio, sampling_rate = load_wav(filename)\n audio = audio / MAX_WAV_VALUE\n if not self.fine_tuning:\n audio = normalize(audio) * 0.95\n self.cached_wav = audio\n if sampling_rate != self.sampling_rate:\n raise ValueError(f\"{sampling_rate} SR doesn't match target {self.sampling_rate} SR\")\n self._cache_ref_count = self.n_cache_reuse\n else:\n audio = self.cached_wav\n self._cache_ref_count -= 1\n\n audio = torch.FloatTensor(audio)\n audio = audio.unsqueeze(0)\n\n if not self.fine_tuning:\n if self.split:\n if audio.size(1) >= self.segment_size:\n max_audio_start = audio.size(1) - self.segment_size\n audio_start = random.randint(0, max_audio_start)\n audio = audio[:, audio_start : audio_start + self.segment_size]\n else:\n audio = torch.nn.functional.pad(audio, (0, self.segment_size - audio.size(1)), \"constant\")\n\n mel = mel_spectrogram(\n audio,\n self.n_fft,\n self.num_mels,\n self.sampling_rate,\n self.hop_size,\n self.win_size,\n self.fmin,\n self.fmax,\n center=False,\n )\n else:\n mel = np.load(os.path.join(self.base_mels_path, os.path.splitext(os.path.split(filename)[-1])[0] + \".npy\"))\n mel = torch.from_numpy(mel)\n\n if len(mel.shape) < 3:\n mel = mel.unsqueeze(0)\n\n if self.split:\n frames_per_seg = math.ceil(self.segment_size / self.hop_size)\n\n if audio.size(1) >= self.segment_size:\n mel_start = random.randint(0, mel.size(2) - frames_per_seg - 1)\n mel = mel[:, :, mel_start : mel_start + frames_per_seg]\n audio = audio[:, mel_start * self.hop_size : (mel_start + frames_per_seg) * self.hop_size]\n else:\n mel = torch.nn.functional.pad(mel, (0, frames_per_seg - mel.size(2)), \"constant\")\n audio = torch.nn.functional.pad(audio, (0, self.segment_size - audio.size(1)), \"constant\")\n\n mel_loss = mel_spectrogram(\n audio,\n self.n_fft,\n self.num_mels,\n self.sampling_rate,\n self.hop_size,\n self.win_size,\n self.fmin,\n self.fmax_loss,\n center=False,\n )\n\n return (mel.squeeze(), audio.squeeze(0), filename, mel_loss.squeeze())\n\n def __len__(self):\n return len(self.audio_files)\n" }, { "path": "third_party/Matcha-TTS/matcha/hifigan/models.py", "content": "\"\"\" from https://github.com/jik876/hifi-gan \"\"\"\n\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nfrom torch.nn import AvgPool1d, Conv1d, Conv2d, ConvTranspose1d\nfrom torch.nn.utils import remove_weight_norm, spectral_norm, weight_norm\n\nfrom .xutils import get_padding, init_weights\n\nLRELU_SLOPE = 0.1\n\n\nclass ResBlock1(torch.nn.Module):\n def __init__(self, h, channels, kernel_size=3, dilation=(1, 3, 5)):\n super().__init__()\n self.h = h\n self.convs1 = nn.ModuleList(\n [\n weight_norm(\n Conv1d(\n channels,\n channels,\n kernel_size,\n 1,\n dilation=dilation[0],\n padding=get_padding(kernel_size, dilation[0]),\n )\n ),\n weight_norm(\n Conv1d(\n channels,\n channels,\n kernel_size,\n 1,\n dilation=dilation[1],\n padding=get_padding(kernel_size, dilation[1]),\n )\n ),\n weight_norm(\n Conv1d(\n channels,\n channels,\n kernel_size,\n 1,\n dilation=dilation[2],\n padding=get_padding(kernel_size, dilation[2]),\n )\n ),\n ]\n )\n self.convs1.apply(init_weights)\n\n self.convs2 = nn.ModuleList(\n [\n weight_norm(\n Conv1d(\n channels,\n channels,\n kernel_size,\n 1,\n dilation=1,\n padding=get_padding(kernel_size, 1),\n )\n ),\n weight_norm(\n Conv1d(\n channels,\n channels,\n kernel_size,\n 1,\n dilation=1,\n padding=get_padding(kernel_size, 1),\n )\n ),\n weight_norm(\n Conv1d(\n channels,\n channels,\n kernel_size,\n 1,\n dilation=1,\n padding=get_padding(kernel_size, 1),\n )\n ),\n ]\n )\n self.convs2.apply(init_weights)\n\n def forward(self, x):\n for c1, c2 in zip(self.convs1, self.convs2):\n xt = F.leaky_relu(x, LRELU_SLOPE)\n xt = c1(xt)\n xt = F.leaky_relu(xt, LRELU_SLOPE)\n xt = c2(xt)\n x = xt + x\n return x\n\n def remove_weight_norm(self):\n for l in self.convs1:\n remove_weight_norm(l)\n for l in self.convs2:\n remove_weight_norm(l)\n\n\nclass ResBlock2(torch.nn.Module):\n def __init__(self, h, channels, kernel_size=3, dilation=(1, 3)):\n super().__init__()\n self.h = h\n self.convs = nn.ModuleList(\n [\n weight_norm(\n Conv1d(\n channels,\n channels,\n kernel_size,\n 1,\n dilation=dilation[0],\n padding=get_padding(kernel_size, dilation[0]),\n )\n ),\n weight_norm(\n Conv1d(\n channels,\n channels,\n kernel_size,\n 1,\n dilation=dilation[1],\n padding=get_padding(kernel_size, dilation[1]),\n )\n ),\n ]\n )\n self.convs.apply(init_weights)\n\n def forward(self, x):\n for c in self.convs:\n xt = F.leaky_relu(x, LRELU_SLOPE)\n xt = c(xt)\n x = xt + x\n return x\n\n def remove_weight_norm(self):\n for l in self.convs:\n remove_weight_norm(l)\n\n\nclass Generator(torch.nn.Module):\n def __init__(self, h):\n super().__init__()\n self.h = h\n self.num_kernels = len(h.resblock_kernel_sizes)\n self.num_upsamples = len(h.upsample_rates)\n self.conv_pre = weight_norm(Conv1d(80, h.upsample_initial_channel, 7, 1, padding=3))\n resblock = ResBlock1 if h.resblock == \"1\" else ResBlock2\n\n self.ups = nn.ModuleList()\n for i, (u, k) in enumerate(zip(h.upsample_rates, h.upsample_kernel_sizes)):\n self.ups.append(\n weight_norm(\n ConvTranspose1d(\n h.upsample_initial_channel // (2**i),\n h.upsample_initial_channel // (2 ** (i + 1)),\n k,\n u,\n padding=(k - u) // 2,\n )\n )\n )\n\n self.resblocks = nn.ModuleList()\n for i in range(len(self.ups)):\n ch = h.upsample_initial_channel // (2 ** (i + 1))\n for _, (k, d) in enumerate(zip(h.resblock_kernel_sizes, h.resblock_dilation_sizes)):\n self.resblocks.append(resblock(h, ch, k, d))\n\n self.conv_post = weight_norm(Conv1d(ch, 1, 7, 1, padding=3))\n self.ups.apply(init_weights)\n self.conv_post.apply(init_weights)\n\n def forward(self, x):\n x = self.conv_pre(x)\n for i in range(self.num_upsamples):\n x = F.leaky_relu(x, LRELU_SLOPE)\n x = self.ups[i](x)\n xs = None\n for j in range(self.num_kernels):\n if xs is None:\n xs = self.resblocks[i * self.num_kernels + j](x)\n else:\n xs += self.resblocks[i * self.num_kernels + j](x)\n x = xs / self.num_kernels\n x = F.leaky_relu(x)\n x = self.conv_post(x)\n x = torch.tanh(x)\n\n return x\n\n def remove_weight_norm(self):\n print(\"Removing weight norm...\")\n for l in self.ups:\n remove_weight_norm(l)\n for l in self.resblocks:\n l.remove_weight_norm()\n remove_weight_norm(self.conv_pre)\n remove_weight_norm(self.conv_post)\n\n\nclass DiscriminatorP(torch.nn.Module):\n def __init__(self, period, kernel_size=5, stride=3, use_spectral_norm=False):\n super().__init__()\n self.period = period\n norm_f = weight_norm if use_spectral_norm is False else spectral_norm\n self.convs = nn.ModuleList(\n [\n norm_f(Conv2d(1, 32, (kernel_size, 1), (stride, 1), padding=(get_padding(5, 1), 0))),\n norm_f(Conv2d(32, 128, (kernel_size, 1), (stride, 1), padding=(get_padding(5, 1), 0))),\n norm_f(Conv2d(128, 512, (kernel_size, 1), (stride, 1), padding=(get_padding(5, 1), 0))),\n norm_f(Conv2d(512, 1024, (kernel_size, 1), (stride, 1), padding=(get_padding(5, 1), 0))),\n norm_f(Conv2d(1024, 1024, (kernel_size, 1), 1, padding=(2, 0))),\n ]\n )\n self.conv_post = norm_f(Conv2d(1024, 1, (3, 1), 1, padding=(1, 0)))\n\n def forward(self, x):\n fmap = []\n\n # 1d to 2d\n b, c, t = x.shape\n if t % self.period != 0: # pad first\n n_pad = self.period - (t % self.period)\n x = F.pad(x, (0, n_pad), \"reflect\")\n t = t + n_pad\n x = x.view(b, c, t // self.period, self.period)\n\n for l in self.convs:\n x = l(x)\n x = F.leaky_relu(x, LRELU_SLOPE)\n fmap.append(x)\n x = self.conv_post(x)\n fmap.append(x)\n x = torch.flatten(x, 1, -1)\n\n return x, fmap\n\n\nclass MultiPeriodDiscriminator(torch.nn.Module):\n def __init__(self):\n super().__init__()\n self.discriminators = nn.ModuleList(\n [\n DiscriminatorP(2),\n DiscriminatorP(3),\n DiscriminatorP(5),\n DiscriminatorP(7),\n DiscriminatorP(11),\n ]\n )\n\n def forward(self, y, y_hat):\n y_d_rs = []\n y_d_gs = []\n fmap_rs = []\n fmap_gs = []\n for _, d in enumerate(self.discriminators):\n y_d_r, fmap_r = d(y)\n y_d_g, fmap_g = d(y_hat)\n y_d_rs.append(y_d_r)\n fmap_rs.append(fmap_r)\n y_d_gs.append(y_d_g)\n fmap_gs.append(fmap_g)\n\n return y_d_rs, y_d_gs, fmap_rs, fmap_gs\n\n\nclass DiscriminatorS(torch.nn.Module):\n def __init__(self, use_spectral_norm=False):\n super().__init__()\n norm_f = weight_norm if use_spectral_norm is False else spectral_norm\n self.convs = nn.ModuleList(\n [\n norm_f(Conv1d(1, 128, 15, 1, padding=7)),\n norm_f(Conv1d(128, 128, 41, 2, groups=4, padding=20)),\n norm_f(Conv1d(128, 256, 41, 2, groups=16, padding=20)),\n norm_f(Conv1d(256, 512, 41, 4, groups=16, padding=20)),\n norm_f(Conv1d(512, 1024, 41, 4, groups=16, padding=20)),\n norm_f(Conv1d(1024, 1024, 41, 1, groups=16, padding=20)),\n norm_f(Conv1d(1024, 1024, 5, 1, padding=2)),\n ]\n )\n self.conv_post = norm_f(Conv1d(1024, 1, 3, 1, padding=1))\n\n def forward(self, x):\n fmap = []\n for l in self.convs:\n x = l(x)\n x = F.leaky_relu(x, LRELU_SLOPE)\n fmap.append(x)\n x = self.conv_post(x)\n fmap.append(x)\n x = torch.flatten(x, 1, -1)\n\n return x, fmap\n\n\nclass MultiScaleDiscriminator(torch.nn.Module):\n def __init__(self):\n super().__init__()\n self.discriminators = nn.ModuleList(\n [\n DiscriminatorS(use_spectral_norm=True),\n DiscriminatorS(),\n DiscriminatorS(),\n ]\n )\n self.meanpools = nn.ModuleList([AvgPool1d(4, 2, padding=2), AvgPool1d(4, 2, padding=2)])\n\n def forward(self, y, y_hat):\n y_d_rs = []\n y_d_gs = []\n fmap_rs = []\n fmap_gs = []\n for i, d in enumerate(self.discriminators):\n if i != 0:\n y = self.meanpools[i - 1](y)\n y_hat = self.meanpools[i - 1](y_hat)\n y_d_r, fmap_r = d(y)\n y_d_g, fmap_g = d(y_hat)\n y_d_rs.append(y_d_r)\n fmap_rs.append(fmap_r)\n y_d_gs.append(y_d_g)\n fmap_gs.append(fmap_g)\n\n return y_d_rs, y_d_gs, fmap_rs, fmap_gs\n\n\ndef feature_loss(fmap_r, fmap_g):\n loss = 0\n for dr, dg in zip(fmap_r, fmap_g):\n for rl, gl in zip(dr, dg):\n loss += torch.mean(torch.abs(rl - gl))\n\n return loss * 2\n\n\ndef discriminator_loss(disc_real_outputs, disc_generated_outputs):\n loss = 0\n r_losses = []\n g_losses = []\n for dr, dg in zip(disc_real_outputs, disc_generated_outputs):\n r_loss = torch.mean((1 - dr) ** 2)\n g_loss = torch.mean(dg**2)\n loss += r_loss + g_loss\n r_losses.append(r_loss.item())\n g_losses.append(g_loss.item())\n\n return loss, r_losses, g_losses\n\n\ndef generator_loss(disc_outputs):\n loss = 0\n gen_losses = []\n for dg in disc_outputs:\n l = torch.mean((1 - dg) ** 2)\n gen_losses.append(l)\n loss += l\n\n return loss, gen_losses\n" }, { "path": "third_party/Matcha-TTS/matcha/hifigan/xutils.py", "content": "\"\"\" from https://github.com/jik876/hifi-gan \"\"\"\n\nimport glob\nimport os\n\nimport matplotlib\nimport torch\nfrom torch.nn.utils import weight_norm\n\nmatplotlib.use(\"Agg\")\nimport matplotlib.pylab as plt\n\n\ndef plot_spectrogram(spectrogram):\n fig, ax = plt.subplots(figsize=(10, 2))\n im = ax.imshow(spectrogram, aspect=\"auto\", origin=\"lower\", interpolation=\"none\")\n plt.colorbar(im, ax=ax)\n\n fig.canvas.draw()\n plt.close()\n\n return fig\n\n\ndef init_weights(m, mean=0.0, std=0.01):\n classname = m.__class__.__name__\n if classname.find(\"Conv\") != -1:\n m.weight.data.normal_(mean, std)\n\n\ndef apply_weight_norm(m):\n classname = m.__class__.__name__\n if classname.find(\"Conv\") != -1:\n weight_norm(m)\n\n\ndef get_padding(kernel_size, dilation=1):\n return int((kernel_size * dilation - dilation) / 2)\n\n\ndef load_checkpoint(filepath, device):\n assert os.path.isfile(filepath)\n print(f\"Loading '{filepath}'\")\n checkpoint_dict = torch.load(filepath, map_location=device)\n print(\"Complete.\")\n return checkpoint_dict\n\n\ndef save_checkpoint(filepath, obj):\n print(f\"Saving checkpoint to {filepath}\")\n torch.save(obj, filepath)\n print(\"Complete.\")\n\n\ndef scan_checkpoint(cp_dir, prefix):\n pattern = os.path.join(cp_dir, prefix + \"????????\")\n cp_list = glob.glob(pattern)\n if len(cp_list) == 0:\n return None\n return sorted(cp_list)[-1]\n" }, { "path": "third_party/Matcha-TTS/matcha/models/__init__.py", "content": "" }, { "path": "third_party/Matcha-TTS/matcha/models/baselightningmodule.py", "content": "\"\"\"\nThis is a base lightning module that can be used to train a model.\nThe benefit of this abstraction is that all the logic outside of model definition can be reused for different models.\n\"\"\"\nimport inspect\nfrom abc import ABC\nfrom typing import Any, Dict\n\nimport torch\nfrom lightning import LightningModule\nfrom lightning.pytorch.utilities import grad_norm\n\nfrom matcha import utils\nfrom matcha.utils.utils import plot_tensor\n\nlog = utils.get_pylogger(__name__)\n\n\nclass BaseLightningClass(LightningModule, ABC):\n def update_data_statistics(self, data_statistics):\n if data_statistics is None:\n data_statistics = {\n \"mel_mean\": 0.0,\n \"mel_std\": 1.0,\n }\n\n self.register_buffer(\"mel_mean\", torch.tensor(data_statistics[\"mel_mean\"]))\n self.register_buffer(\"mel_std\", torch.tensor(data_statistics[\"mel_std\"]))\n\n def configure_optimizers(self) -> Any:\n optimizer = self.hparams.optimizer(params=self.parameters())\n if self.hparams.scheduler not in (None, {}):\n scheduler_args = {}\n # Manage last epoch for exponential schedulers\n if \"last_epoch\" in inspect.signature(self.hparams.scheduler.scheduler).parameters:\n if hasattr(self, \"ckpt_loaded_epoch\"):\n current_epoch = self.ckpt_loaded_epoch - 1\n else:\n current_epoch = -1\n\n scheduler_args.update({\"optimizer\": optimizer})\n scheduler = self.hparams.scheduler.scheduler(**scheduler_args)\n scheduler.last_epoch = current_epoch\n return {\n \"optimizer\": optimizer,\n \"lr_scheduler\": {\n \"scheduler\": scheduler,\n \"interval\": self.hparams.scheduler.lightning_args.interval,\n \"frequency\": self.hparams.scheduler.lightning_args.frequency,\n \"name\": \"learning_rate\",\n },\n }\n\n return {\"optimizer\": optimizer}\n\n def get_losses(self, batch):\n x, x_lengths = batch[\"x\"], batch[\"x_lengths\"]\n y, y_lengths = batch[\"y\"], batch[\"y_lengths\"]\n spks = batch[\"spks\"]\n\n dur_loss, prior_loss, diff_loss = self(\n x=x,\n x_lengths=x_lengths,\n y=y,\n y_lengths=y_lengths,\n spks=spks,\n out_size=self.out_size,\n )\n return {\n \"dur_loss\": dur_loss,\n \"prior_loss\": prior_loss,\n \"diff_loss\": diff_loss,\n }\n\n def on_load_checkpoint(self, checkpoint: Dict[str, Any]) -> None:\n self.ckpt_loaded_epoch = checkpoint[\"epoch\"] # pylint: disable=attribute-defined-outside-init\n\n def training_step(self, batch: Any, batch_idx: int):\n loss_dict = self.get_losses(batch)\n self.log(\n \"step\",\n float(self.global_step),\n on_step=True,\n prog_bar=True,\n logger=True,\n sync_dist=True,\n )\n\n self.log(\n \"sub_loss/train_dur_loss\",\n loss_dict[\"dur_loss\"],\n on_step=True,\n on_epoch=True,\n logger=True,\n sync_dist=True,\n )\n self.log(\n \"sub_loss/train_prior_loss\",\n loss_dict[\"prior_loss\"],\n on_step=True,\n on_epoch=True,\n logger=True,\n sync_dist=True,\n )\n self.log(\n \"sub_loss/train_diff_loss\",\n loss_dict[\"diff_loss\"],\n on_step=True,\n on_epoch=True,\n logger=True,\n sync_dist=True,\n )\n\n total_loss = sum(loss_dict.values())\n self.log(\n \"loss/train\",\n total_loss,\n on_step=True,\n on_epoch=True,\n logger=True,\n prog_bar=True,\n sync_dist=True,\n )\n\n return {\"loss\": total_loss, \"log\": loss_dict}\n\n def validation_step(self, batch: Any, batch_idx: int):\n loss_dict = self.get_losses(batch)\n self.log(\n \"sub_loss/val_dur_loss\",\n loss_dict[\"dur_loss\"],\n on_step=True,\n on_epoch=True,\n logger=True,\n sync_dist=True,\n )\n self.log(\n \"sub_loss/val_prior_loss\",\n loss_dict[\"prior_loss\"],\n on_step=True,\n on_epoch=True,\n logger=True,\n sync_dist=True,\n )\n self.log(\n \"sub_loss/val_diff_loss\",\n loss_dict[\"diff_loss\"],\n on_step=True,\n on_epoch=True,\n logger=True,\n sync_dist=True,\n )\n\n total_loss = sum(loss_dict.values())\n self.log(\n \"loss/val\",\n total_loss,\n on_step=True,\n on_epoch=True,\n logger=True,\n prog_bar=True,\n sync_dist=True,\n )\n\n return total_loss\n\n def on_validation_end(self) -> None:\n if self.trainer.is_global_zero:\n one_batch = next(iter(self.trainer.val_dataloaders))\n if self.current_epoch == 0:\n log.debug(\"Plotting original samples\")\n for i in range(2):\n y = one_batch[\"y\"][i].unsqueeze(0).to(self.device)\n self.logger.experiment.add_image(\n f\"original/{i}\",\n plot_tensor(y.squeeze().cpu()),\n self.current_epoch,\n dataformats=\"HWC\",\n )\n\n log.debug(\"Synthesising...\")\n for i in range(2):\n x = one_batch[\"x\"][i].unsqueeze(0).to(self.device)\n x_lengths = one_batch[\"x_lengths\"][i].unsqueeze(0).to(self.device)\n spks = one_batch[\"spks\"][i].unsqueeze(0).to(self.device) if one_batch[\"spks\"] is not None else None\n output = self.synthesise(x[:, :x_lengths], x_lengths, n_timesteps=10, spks=spks)\n y_enc, y_dec = output[\"encoder_outputs\"], output[\"decoder_outputs\"]\n attn = output[\"attn\"]\n self.logger.experiment.add_image(\n f\"generated_enc/{i}\",\n plot_tensor(y_enc.squeeze().cpu()),\n self.current_epoch,\n dataformats=\"HWC\",\n )\n self.logger.experiment.add_image(\n f\"generated_dec/{i}\",\n plot_tensor(y_dec.squeeze().cpu()),\n self.current_epoch,\n dataformats=\"HWC\",\n )\n self.logger.experiment.add_image(\n f\"alignment/{i}\",\n plot_tensor(attn.squeeze().cpu()),\n self.current_epoch,\n dataformats=\"HWC\",\n )\n\n def on_before_optimizer_step(self, optimizer):\n self.log_dict({f\"grad_norm/{k}\": v for k, v in grad_norm(self, norm_type=2).items()})\n" }, { "path": "third_party/Matcha-TTS/matcha/models/components/__init__.py", "content": "" }, { "path": "third_party/Matcha-TTS/matcha/models/components/decoder.py", "content": "import math\nfrom typing import Optional\n\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nfrom conformer import ConformerBlock\nfrom diffusers.models.activations import get_activation\nfrom einops import pack, rearrange, repeat\n\nfrom matcha.models.components.transformer import BasicTransformerBlock\n\n\nclass SinusoidalPosEmb(torch.nn.Module):\n def __init__(self, dim):\n super().__init__()\n self.dim = dim\n assert self.dim % 2 == 0, \"SinusoidalPosEmb requires dim to be even\"\n\n def forward(self, x, scale=1000):\n if x.ndim < 1:\n x = x.unsqueeze(0)\n device = x.device\n half_dim = self.dim // 2\n emb = math.log(10000) / (half_dim - 1)\n emb = torch.exp(torch.arange(half_dim, device=device).float() * -emb)\n emb = scale * x.unsqueeze(1) * emb.unsqueeze(0)\n emb = torch.cat((emb.sin(), emb.cos()), dim=-1)\n return emb\n\n\nclass Block1D(torch.nn.Module):\n def __init__(self, dim, dim_out, groups=8):\n super().__init__()\n self.block = torch.nn.Sequential(\n torch.nn.Conv1d(dim, dim_out, 3, padding=1),\n torch.nn.GroupNorm(groups, dim_out),\n nn.Mish(),\n )\n\n def forward(self, x, mask):\n output = self.block(x * mask)\n return output * mask\n\n\nclass ResnetBlock1D(torch.nn.Module):\n def __init__(self, dim, dim_out, time_emb_dim, groups=8):\n super().__init__()\n self.mlp = torch.nn.Sequential(nn.Mish(), torch.nn.Linear(time_emb_dim, dim_out))\n\n self.block1 = Block1D(dim, dim_out, groups=groups)\n self.block2 = Block1D(dim_out, dim_out, groups=groups)\n\n self.res_conv = torch.nn.Conv1d(dim, dim_out, 1)\n\n def forward(self, x, mask, time_emb):\n h = self.block1(x, mask)\n h += self.mlp(time_emb).unsqueeze(-1)\n h = self.block2(h, mask)\n output = h + self.res_conv(x * mask)\n return output\n\n\nclass Downsample1D(nn.Module):\n def __init__(self, dim):\n super().__init__()\n self.conv = torch.nn.Conv1d(dim, dim, 3, 2, 1)\n\n def forward(self, x):\n return self.conv(x)\n\n\nclass TimestepEmbedding(nn.Module):\n def __init__(\n self,\n in_channels: int,\n time_embed_dim: int,\n act_fn: str = \"silu\",\n out_dim: int = None,\n post_act_fn: Optional[str] = None,\n cond_proj_dim=None,\n ):\n super().__init__()\n\n self.linear_1 = nn.Linear(in_channels, time_embed_dim)\n\n if cond_proj_dim is not None:\n self.cond_proj = nn.Linear(cond_proj_dim, in_channels, bias=False)\n else:\n self.cond_proj = None\n\n self.act = get_activation(act_fn)\n\n if out_dim is not None:\n time_embed_dim_out = out_dim\n else:\n time_embed_dim_out = time_embed_dim\n self.linear_2 = nn.Linear(time_embed_dim, time_embed_dim_out)\n\n if post_act_fn is None:\n self.post_act = None\n else:\n self.post_act = get_activation(post_act_fn)\n\n def forward(self, sample, condition=None):\n if condition is not None:\n sample = sample + self.cond_proj(condition)\n sample = self.linear_1(sample)\n\n if self.act is not None:\n sample = self.act(sample)\n\n sample = self.linear_2(sample)\n\n if self.post_act is not None:\n sample = self.post_act(sample)\n return sample\n\n\nclass Upsample1D(nn.Module):\n \"\"\"A 1D upsampling layer with an optional convolution.\n\n Parameters:\n channels (`int`):\n number of channels in the inputs and outputs.\n use_conv (`bool`, default `False`):\n option to use a convolution.\n use_conv_transpose (`bool`, default `False`):\n option to use a convolution transpose.\n out_channels (`int`, optional):\n number of output channels. Defaults to `channels`.\n \"\"\"\n\n def __init__(self, channels, use_conv=False, use_conv_transpose=True, out_channels=None, name=\"conv\"):\n super().__init__()\n self.channels = channels\n self.out_channels = out_channels or channels\n self.use_conv = use_conv\n self.use_conv_transpose = use_conv_transpose\n self.name = name\n\n self.conv = None\n if use_conv_transpose:\n self.conv = nn.ConvTranspose1d(channels, self.out_channels, 4, 2, 1)\n elif use_conv:\n self.conv = nn.Conv1d(self.channels, self.out_channels, 3, padding=1)\n\n def forward(self, inputs):\n assert inputs.shape[1] == self.channels\n if self.use_conv_transpose:\n return self.conv(inputs)\n\n outputs = F.interpolate(inputs, scale_factor=2.0, mode=\"nearest\")\n\n if self.use_conv:\n outputs = self.conv(outputs)\n\n return outputs\n\n\nclass ConformerWrapper(ConformerBlock):\n def __init__( # pylint: disable=useless-super-delegation\n self,\n *,\n dim,\n dim_head=64,\n heads=8,\n ff_mult=4,\n conv_expansion_factor=2,\n conv_kernel_size=31,\n attn_dropout=0,\n ff_dropout=0,\n conv_dropout=0,\n conv_causal=False,\n ):\n super().__init__(\n dim=dim,\n dim_head=dim_head,\n heads=heads,\n ff_mult=ff_mult,\n conv_expansion_factor=conv_expansion_factor,\n conv_kernel_size=conv_kernel_size,\n attn_dropout=attn_dropout,\n ff_dropout=ff_dropout,\n conv_dropout=conv_dropout,\n conv_causal=conv_causal,\n )\n\n def forward(\n self,\n hidden_states,\n attention_mask,\n encoder_hidden_states=None,\n encoder_attention_mask=None,\n timestep=None,\n ):\n return super().forward(x=hidden_states, mask=attention_mask.bool())\n\n\nclass Decoder(nn.Module):\n def __init__(\n self,\n in_channels,\n out_channels,\n channels=(256, 256),\n dropout=0.05,\n attention_head_dim=64,\n n_blocks=1,\n num_mid_blocks=2,\n num_heads=4,\n act_fn=\"snake\",\n down_block_type=\"transformer\",\n mid_block_type=\"transformer\",\n up_block_type=\"transformer\",\n ):\n super().__init__()\n channels = tuple(channels)\n self.in_channels = in_channels\n self.out_channels = out_channels\n\n self.time_embeddings = SinusoidalPosEmb(in_channels)\n time_embed_dim = channels[0] * 4\n self.time_mlp = TimestepEmbedding(\n in_channels=in_channels,\n time_embed_dim=time_embed_dim,\n act_fn=\"silu\",\n )\n\n self.down_blocks = nn.ModuleList([])\n self.mid_blocks = nn.ModuleList([])\n self.up_blocks = nn.ModuleList([])\n\n output_channel = in_channels\n for i in range(len(channels)): # pylint: disable=consider-using-enumerate\n input_channel = output_channel\n output_channel = channels[i]\n is_last = i == len(channels) - 1\n resnet = ResnetBlock1D(dim=input_channel, dim_out=output_channel, time_emb_dim=time_embed_dim)\n transformer_blocks = nn.ModuleList(\n [\n self.get_block(\n down_block_type,\n output_channel,\n attention_head_dim,\n num_heads,\n dropout,\n act_fn,\n )\n for _ in range(n_blocks)\n ]\n )\n downsample = (\n Downsample1D(output_channel) if not is_last else nn.Conv1d(output_channel, output_channel, 3, padding=1)\n )\n\n self.down_blocks.append(nn.ModuleList([resnet, transformer_blocks, downsample]))\n\n for i in range(num_mid_blocks):\n input_channel = channels[-1]\n out_channels = channels[-1]\n\n resnet = ResnetBlock1D(dim=input_channel, dim_out=output_channel, time_emb_dim=time_embed_dim)\n\n transformer_blocks = nn.ModuleList(\n [\n self.get_block(\n mid_block_type,\n output_channel,\n attention_head_dim,\n num_heads,\n dropout,\n act_fn,\n )\n for _ in range(n_blocks)\n ]\n )\n\n self.mid_blocks.append(nn.ModuleList([resnet, transformer_blocks]))\n\n channels = channels[::-1] + (channels[0],)\n for i in range(len(channels) - 1):\n input_channel = channels[i]\n output_channel = channels[i + 1]\n is_last = i == len(channels) - 2\n\n resnet = ResnetBlock1D(\n dim=2 * input_channel,\n dim_out=output_channel,\n time_emb_dim=time_embed_dim,\n )\n transformer_blocks = nn.ModuleList(\n [\n self.get_block(\n up_block_type,\n output_channel,\n attention_head_dim,\n num_heads,\n dropout,\n act_fn,\n )\n for _ in range(n_blocks)\n ]\n )\n upsample = (\n Upsample1D(output_channel, use_conv_transpose=True)\n if not is_last\n else nn.Conv1d(output_channel, output_channel, 3, padding=1)\n )\n\n self.up_blocks.append(nn.ModuleList([resnet, transformer_blocks, upsample]))\n\n self.final_block = Block1D(channels[-1], channels[-1])\n self.final_proj = nn.Conv1d(channels[-1], self.out_channels, 1)\n\n self.initialize_weights()\n # nn.init.normal_(self.final_proj.weight)\n\n @staticmethod\n def get_block(block_type, dim, attention_head_dim, num_heads, dropout, act_fn):\n if block_type == \"conformer\":\n block = ConformerWrapper(\n dim=dim,\n dim_head=attention_head_dim,\n heads=num_heads,\n ff_mult=1,\n conv_expansion_factor=2,\n ff_dropout=dropout,\n attn_dropout=dropout,\n conv_dropout=dropout,\n conv_kernel_size=31,\n )\n elif block_type == \"transformer\":\n block = BasicTransformerBlock(\n dim=dim,\n num_attention_heads=num_heads,\n attention_head_dim=attention_head_dim,\n dropout=dropout,\n activation_fn=act_fn,\n )\n else:\n raise ValueError(f\"Unknown block type {block_type}\")\n\n return block\n\n def initialize_weights(self):\n for m in self.modules():\n if isinstance(m, nn.Conv1d):\n nn.init.kaiming_normal_(m.weight, nonlinearity=\"relu\")\n\n if m.bias is not None:\n nn.init.constant_(m.bias, 0)\n\n elif isinstance(m, nn.GroupNorm):\n nn.init.constant_(m.weight, 1)\n nn.init.constant_(m.bias, 0)\n\n elif isinstance(m, nn.Linear):\n nn.init.kaiming_normal_(m.weight, nonlinearity=\"relu\")\n\n if m.bias is not None:\n nn.init.constant_(m.bias, 0)\n\n def forward(self, x, mask, mu, t, spks=None, cond=None):\n \"\"\"Forward pass of the UNet1DConditional model.\n\n Args:\n x (torch.Tensor): shape (batch_size, in_channels, time)\n mask (_type_): shape (batch_size, 1, time)\n t (_type_): shape (batch_size)\n spks (_type_, optional): shape: (batch_size, condition_channels). Defaults to None.\n cond (_type_, optional): placeholder for future use. Defaults to None.\n\n Raises:\n ValueError: _description_\n ValueError: _description_\n\n Returns:\n _type_: _description_\n \"\"\"\n\n t = self.time_embeddings(t)\n t = self.time_mlp(t)\n\n x = pack([x, mu], \"b * t\")[0]\n\n if spks is not None:\n spks = repeat(spks, \"b c -> b c t\", t=x.shape[-1])\n x = pack([x, spks], \"b * t\")[0]\n\n hiddens = []\n masks = [mask]\n for resnet, transformer_blocks, downsample in self.down_blocks:\n mask_down = masks[-1]\n x = resnet(x, mask_down, t)\n x = rearrange(x, \"b c t -> b t c\")\n mask_down = rearrange(mask_down, \"b 1 t -> b t\")\n for transformer_block in transformer_blocks:\n x = transformer_block(\n hidden_states=x,\n attention_mask=mask_down,\n timestep=t,\n )\n x = rearrange(x, \"b t c -> b c t\")\n mask_down = rearrange(mask_down, \"b t -> b 1 t\")\n hiddens.append(x) # Save hidden states for skip connections\n x = downsample(x * mask_down)\n masks.append(mask_down[:, :, ::2])\n\n masks = masks[:-1]\n mask_mid = masks[-1]\n\n for resnet, transformer_blocks in self.mid_blocks:\n x = resnet(x, mask_mid, t)\n x = rearrange(x, \"b c t -> b t c\")\n mask_mid = rearrange(mask_mid, \"b 1 t -> b t\")\n for transformer_block in transformer_blocks:\n x = transformer_block(\n hidden_states=x,\n attention_mask=mask_mid,\n timestep=t,\n )\n x = rearrange(x, \"b t c -> b c t\")\n mask_mid = rearrange(mask_mid, \"b t -> b 1 t\")\n\n for resnet, transformer_blocks, upsample in self.up_blocks:\n mask_up = masks.pop()\n x = resnet(pack([x, hiddens.pop()], \"b * t\")[0], mask_up, t)\n x = rearrange(x, \"b c t -> b t c\")\n mask_up = rearrange(mask_up, \"b 1 t -> b t\")\n for transformer_block in transformer_blocks:\n x = transformer_block(\n hidden_states=x,\n attention_mask=mask_up,\n timestep=t,\n )\n x = rearrange(x, \"b t c -> b c t\")\n mask_up = rearrange(mask_up, \"b t -> b 1 t\")\n x = upsample(x * mask_up)\n\n x = self.final_block(x, mask_up)\n output = self.final_proj(x * mask_up)\n\n return output * mask\n" }, { "path": "third_party/Matcha-TTS/matcha/models/components/flow_matching.py", "content": "from abc import ABC\n\nimport torch\nimport torch.nn.functional as F\n\nfrom matcha.models.components.decoder import Decoder\nfrom matcha.utils.pylogger import get_pylogger\n\nlog = get_pylogger(__name__)\n\n\nclass BASECFM(torch.nn.Module, ABC):\n def __init__(\n self,\n n_feats,\n cfm_params,\n n_spks=1,\n spk_emb_dim=128,\n ):\n super().__init__()\n self.n_feats = n_feats\n self.n_spks = n_spks\n self.spk_emb_dim = spk_emb_dim\n self.solver = cfm_params.solver\n if hasattr(cfm_params, \"sigma_min\"):\n self.sigma_min = cfm_params.sigma_min\n else:\n self.sigma_min = 1e-4\n\n self.estimator = None\n\n @torch.inference_mode()\n def forward(self, mu, mask, n_timesteps, temperature=1.0, spks=None, cond=None):\n \"\"\"Forward diffusion\n\n Args:\n mu (torch.Tensor): output of encoder\n shape: (batch_size, n_feats, mel_timesteps)\n mask (torch.Tensor): output_mask\n shape: (batch_size, 1, mel_timesteps)\n n_timesteps (int): number of diffusion steps\n temperature (float, optional): temperature for scaling noise. Defaults to 1.0.\n spks (torch.Tensor, optional): speaker ids. Defaults to None.\n shape: (batch_size, spk_emb_dim)\n cond: Not used but kept for future purposes\n\n Returns:\n sample: generated mel-spectrogram\n shape: (batch_size, n_feats, mel_timesteps)\n \"\"\"\n z = torch.randn_like(mu) * temperature\n t_span = torch.linspace(0, 1, n_timesteps + 1, device=mu.device)\n return self.solve_euler(z, t_span=t_span, mu=mu, mask=mask, spks=spks, cond=cond)\n\n def solve_euler(self, x, t_span, mu, mask, spks, cond):\n \"\"\"\n Fixed euler solver for ODEs.\n Args:\n x (torch.Tensor): random noise\n t_span (torch.Tensor): n_timesteps interpolated\n shape: (n_timesteps + 1,)\n mu (torch.Tensor): output of encoder\n shape: (batch_size, n_feats, mel_timesteps)\n mask (torch.Tensor): output_mask\n shape: (batch_size, 1, mel_timesteps)\n spks (torch.Tensor, optional): speaker ids. Defaults to None.\n shape: (batch_size, spk_emb_dim)\n cond: Not used but kept for future purposes\n \"\"\"\n t, _, dt = t_span[0], t_span[-1], t_span[1] - t_span[0]\n\n # I am storing this because I can later plot it by putting a debugger here and saving it to a file\n # Or in future might add like a return_all_steps flag\n sol = []\n\n for step in range(1, len(t_span)):\n dphi_dt = self.estimator(x, mask, mu, t, spks, cond)\n\n x = x + dt * dphi_dt\n t = t + dt\n sol.append(x)\n if step < len(t_span) - 1:\n dt = t_span[step + 1] - t\n\n return sol[-1]\n\n def compute_loss(self, x1, mask, mu, spks=None, cond=None):\n \"\"\"Computes diffusion loss\n\n Args:\n x1 (torch.Tensor): Target\n shape: (batch_size, n_feats, mel_timesteps)\n mask (torch.Tensor): target mask\n shape: (batch_size, 1, mel_timesteps)\n mu (torch.Tensor): output of encoder\n shape: (batch_size, n_feats, mel_timesteps)\n spks (torch.Tensor, optional): speaker embedding. Defaults to None.\n shape: (batch_size, spk_emb_dim)\n\n Returns:\n loss: conditional flow matching loss\n y: conditional flow\n shape: (batch_size, n_feats, mel_timesteps)\n \"\"\"\n b, _, t = mu.shape\n\n # random timestep\n t = torch.rand([b, 1, 1], device=mu.device, dtype=mu.dtype)\n # sample noise p(x_0)\n z = torch.randn_like(x1)\n\n y = (1 - (1 - self.sigma_min) * t) * z + t * x1\n u = x1 - (1 - self.sigma_min) * z\n\n loss = F.mse_loss(self.estimator(y, mask, mu, t.squeeze(), spks), u, reduction=\"sum\") / (\n torch.sum(mask) * u.shape[1]\n )\n return loss, y\n\n\nclass CFM(BASECFM):\n def __init__(self, in_channels, out_channel, cfm_params, decoder_params, n_spks=1, spk_emb_dim=64):\n super().__init__(\n n_feats=in_channels,\n cfm_params=cfm_params,\n n_spks=n_spks,\n spk_emb_dim=spk_emb_dim,\n )\n\n in_channels = in_channels + (spk_emb_dim if n_spks > 1 else 0)\n # Just change the architecture of the estimator here\n self.estimator = Decoder(in_channels=in_channels, out_channels=out_channel, **decoder_params)\n" }, { "path": "third_party/Matcha-TTS/matcha/models/components/text_encoder.py", "content": "\"\"\" from https://github.com/jaywalnut310/glow-tts \"\"\"\n\nimport math\n\nimport torch\nimport torch.nn as nn\nfrom einops import rearrange\n\nimport matcha.utils as utils\nfrom matcha.utils.model import sequence_mask\n\nlog = utils.get_pylogger(__name__)\n\n\nclass LayerNorm(nn.Module):\n def __init__(self, channels, eps=1e-4):\n super().__init__()\n self.channels = channels\n self.eps = eps\n\n self.gamma = torch.nn.Parameter(torch.ones(channels))\n self.beta = torch.nn.Parameter(torch.zeros(channels))\n\n def forward(self, x):\n n_dims = len(x.shape)\n mean = torch.mean(x, 1, keepdim=True)\n variance = torch.mean((x - mean) ** 2, 1, keepdim=True)\n\n x = (x - mean) * torch.rsqrt(variance + self.eps)\n\n shape = [1, -1] + [1] * (n_dims - 2)\n x = x * self.gamma.view(*shape) + self.beta.view(*shape)\n return x\n\n\nclass ConvReluNorm(nn.Module):\n def __init__(self, in_channels, hidden_channels, out_channels, kernel_size, n_layers, p_dropout):\n super().__init__()\n self.in_channels = in_channels\n self.hidden_channels = hidden_channels\n self.out_channels = out_channels\n self.kernel_size = kernel_size\n self.n_layers = n_layers\n self.p_dropout = p_dropout\n\n self.conv_layers = torch.nn.ModuleList()\n self.norm_layers = torch.nn.ModuleList()\n self.conv_layers.append(torch.nn.Conv1d(in_channels, hidden_channels, kernel_size, padding=kernel_size // 2))\n self.norm_layers.append(LayerNorm(hidden_channels))\n self.relu_drop = torch.nn.Sequential(torch.nn.ReLU(), torch.nn.Dropout(p_dropout))\n for _ in range(n_layers - 1):\n self.conv_layers.append(\n torch.nn.Conv1d(hidden_channels, hidden_channels, kernel_size, padding=kernel_size // 2)\n )\n self.norm_layers.append(LayerNorm(hidden_channels))\n self.proj = torch.nn.Conv1d(hidden_channels, out_channels, 1)\n self.proj.weight.data.zero_()\n self.proj.bias.data.zero_()\n\n def forward(self, x, x_mask):\n x_org = x\n for i in range(self.n_layers):\n x = self.conv_layers[i](x * x_mask)\n x = self.norm_layers[i](x)\n x = self.relu_drop(x)\n x = x_org + self.proj(x)\n return x * x_mask\n\n\nclass DurationPredictor(nn.Module):\n def __init__(self, in_channels, filter_channels, kernel_size, p_dropout):\n super().__init__()\n self.in_channels = in_channels\n self.filter_channels = filter_channels\n self.p_dropout = p_dropout\n\n self.drop = torch.nn.Dropout(p_dropout)\n self.conv_1 = torch.nn.Conv1d(in_channels, filter_channels, kernel_size, padding=kernel_size // 2)\n self.norm_1 = LayerNorm(filter_channels)\n self.conv_2 = torch.nn.Conv1d(filter_channels, filter_channels, kernel_size, padding=kernel_size // 2)\n self.norm_2 = LayerNorm(filter_channels)\n self.proj = torch.nn.Conv1d(filter_channels, 1, 1)\n\n def forward(self, x, x_mask):\n x = self.conv_1(x * x_mask)\n x = torch.relu(x)\n x = self.norm_1(x)\n x = self.drop(x)\n x = self.conv_2(x * x_mask)\n x = torch.relu(x)\n x = self.norm_2(x)\n x = self.drop(x)\n x = self.proj(x * x_mask)\n return x * x_mask\n\n\nclass RotaryPositionalEmbeddings(nn.Module):\n \"\"\"\n ## RoPE module\n\n Rotary encoding transforms pairs of features by rotating in the 2D plane.\n That is, it organizes the $d$ features as $\\frac{d}{2}$ pairs.\n Each pair can be considered a coordinate in a 2D plane, and the encoding will rotate it\n by an angle depending on the position of the token.\n \"\"\"\n\n def __init__(self, d: int, base: int = 10_000):\n r\"\"\"\n * `d` is the number of features $d$\n * `base` is the constant used for calculating $\\Theta$\n \"\"\"\n super().__init__()\n\n self.base = base\n self.d = int(d)\n self.cos_cached = None\n self.sin_cached = None\n\n def _build_cache(self, x: torch.Tensor):\n r\"\"\"\n Cache $\\cos$ and $\\sin$ values\n \"\"\"\n # Return if cache is already built\n if self.cos_cached is not None and x.shape[0] <= self.cos_cached.shape[0]:\n return\n\n # Get sequence length\n seq_len = x.shape[0]\n\n # $\\Theta = {\\theta_i = 10000^{-\\frac{2(i-1)}{d}}, i \\in [1, 2, ..., \\frac{d}{2}]}$\n theta = 1.0 / (self.base ** (torch.arange(0, self.d, 2).float() / self.d)).to(x.device)\n\n # Create position indexes `[0, 1, ..., seq_len - 1]`\n seq_idx = torch.arange(seq_len, device=x.device).float().to(x.device)\n\n # Calculate the product of position index and $\\theta_i$\n idx_theta = torch.einsum(\"n,d->nd\", seq_idx, theta)\n\n # Concatenate so that for row $m$ we have\n # $[m \\theta_0, m \\theta_1, ..., m \\theta_{\\frac{d}{2}}, m \\theta_0, m \\theta_1, ..., m \\theta_{\\frac{d}{2}}]$\n idx_theta2 = torch.cat([idx_theta, idx_theta], dim=1)\n\n # Cache them\n self.cos_cached = idx_theta2.cos()[:, None, None, :]\n self.sin_cached = idx_theta2.sin()[:, None, None, :]\n\n def _neg_half(self, x: torch.Tensor):\n # $\\frac{d}{2}$\n d_2 = self.d // 2\n\n # Calculate $[-x^{(\\frac{d}{2} + 1)}, -x^{(\\frac{d}{2} + 2)}, ..., -x^{(d)}, x^{(1)}, x^{(2)}, ..., x^{(\\frac{d}{2})}]$\n return torch.cat([-x[:, :, :, d_2:], x[:, :, :, :d_2]], dim=-1)\n\n def forward(self, x: torch.Tensor):\n \"\"\"\n * `x` is the Tensor at the head of a key or a query with shape `[seq_len, batch_size, n_heads, d]`\n \"\"\"\n # Cache $\\cos$ and $\\sin$ values\n x = rearrange(x, \"b h t d -> t b h d\")\n\n self._build_cache(x)\n\n # Split the features, we can choose to apply rotary embeddings only to a partial set of features.\n x_rope, x_pass = x[..., : self.d], x[..., self.d :]\n\n # Calculate\n # $[-x^{(\\frac{d}{2} + 1)}, -x^{(\\frac{d}{2} + 2)}, ..., -x^{(d)}, x^{(1)}, x^{(2)}, ..., x^{(\\frac{d}{2})}]$\n neg_half_x = self._neg_half(x_rope)\n\n x_rope = (x_rope * self.cos_cached[: x.shape[0]]) + (neg_half_x * self.sin_cached[: x.shape[0]])\n\n return rearrange(torch.cat((x_rope, x_pass), dim=-1), \"t b h d -> b h t d\")\n\n\nclass MultiHeadAttention(nn.Module):\n def __init__(\n self,\n channels,\n out_channels,\n n_heads,\n heads_share=True,\n p_dropout=0.0,\n proximal_bias=False,\n proximal_init=False,\n ):\n super().__init__()\n assert channels % n_heads == 0\n\n self.channels = channels\n self.out_channels = out_channels\n self.n_heads = n_heads\n self.heads_share = heads_share\n self.proximal_bias = proximal_bias\n self.p_dropout = p_dropout\n self.attn = None\n\n self.k_channels = channels // n_heads\n self.conv_q = torch.nn.Conv1d(channels, channels, 1)\n self.conv_k = torch.nn.Conv1d(channels, channels, 1)\n self.conv_v = torch.nn.Conv1d(channels, channels, 1)\n\n # from https://nn.labml.ai/transformers/rope/index.html\n self.query_rotary_pe = RotaryPositionalEmbeddings(self.k_channels * 0.5)\n self.key_rotary_pe = RotaryPositionalEmbeddings(self.k_channels * 0.5)\n\n self.conv_o = torch.nn.Conv1d(channels, out_channels, 1)\n self.drop = torch.nn.Dropout(p_dropout)\n\n torch.nn.init.xavier_uniform_(self.conv_q.weight)\n torch.nn.init.xavier_uniform_(self.conv_k.weight)\n if proximal_init:\n self.conv_k.weight.data.copy_(self.conv_q.weight.data)\n self.conv_k.bias.data.copy_(self.conv_q.bias.data)\n torch.nn.init.xavier_uniform_(self.conv_v.weight)\n\n def forward(self, x, c, attn_mask=None):\n q = self.conv_q(x)\n k = self.conv_k(c)\n v = self.conv_v(c)\n\n x, self.attn = self.attention(q, k, v, mask=attn_mask)\n\n x = self.conv_o(x)\n return x\n\n def attention(self, query, key, value, mask=None):\n b, d, t_s, t_t = (*key.size(), query.size(2))\n query = rearrange(query, \"b (h c) t-> b h t c\", h=self.n_heads)\n key = rearrange(key, \"b (h c) t-> b h t c\", h=self.n_heads)\n value = rearrange(value, \"b (h c) t-> b h t c\", h=self.n_heads)\n\n query = self.query_rotary_pe(query)\n key = self.key_rotary_pe(key)\n\n scores = torch.matmul(query, key.transpose(-2, -1)) / math.sqrt(self.k_channels)\n\n if self.proximal_bias:\n assert t_s == t_t, \"Proximal bias is only available for self-attention.\"\n scores = scores + self._attention_bias_proximal(t_s).to(device=scores.device, dtype=scores.dtype)\n if mask is not None:\n scores = scores.masked_fill(mask == 0, -1e4)\n p_attn = torch.nn.functional.softmax(scores, dim=-1)\n p_attn = self.drop(p_attn)\n output = torch.matmul(p_attn, value)\n output = output.transpose(2, 3).contiguous().view(b, d, t_t)\n return output, p_attn\n\n @staticmethod\n def _attention_bias_proximal(length):\n r = torch.arange(length, dtype=torch.float32)\n diff = torch.unsqueeze(r, 0) - torch.unsqueeze(r, 1)\n return torch.unsqueeze(torch.unsqueeze(-torch.log1p(torch.abs(diff)), 0), 0)\n\n\nclass FFN(nn.Module):\n def __init__(self, in_channels, out_channels, filter_channels, kernel_size, p_dropout=0.0):\n super().__init__()\n self.in_channels = in_channels\n self.out_channels = out_channels\n self.filter_channels = filter_channels\n self.kernel_size = kernel_size\n self.p_dropout = p_dropout\n\n self.conv_1 = torch.nn.Conv1d(in_channels, filter_channels, kernel_size, padding=kernel_size // 2)\n self.conv_2 = torch.nn.Conv1d(filter_channels, out_channels, kernel_size, padding=kernel_size // 2)\n self.drop = torch.nn.Dropout(p_dropout)\n\n def forward(self, x, x_mask):\n x = self.conv_1(x * x_mask)\n x = torch.relu(x)\n x = self.drop(x)\n x = self.conv_2(x * x_mask)\n return x * x_mask\n\n\nclass Encoder(nn.Module):\n def __init__(\n self,\n hidden_channels,\n filter_channels,\n n_heads,\n n_layers,\n kernel_size=1,\n p_dropout=0.0,\n **kwargs,\n ):\n super().__init__()\n self.hidden_channels = hidden_channels\n self.filter_channels = filter_channels\n self.n_heads = n_heads\n self.n_layers = n_layers\n self.kernel_size = kernel_size\n self.p_dropout = p_dropout\n\n self.drop = torch.nn.Dropout(p_dropout)\n self.attn_layers = torch.nn.ModuleList()\n self.norm_layers_1 = torch.nn.ModuleList()\n self.ffn_layers = torch.nn.ModuleList()\n self.norm_layers_2 = torch.nn.ModuleList()\n for _ in range(self.n_layers):\n self.attn_layers.append(MultiHeadAttention(hidden_channels, hidden_channels, n_heads, p_dropout=p_dropout))\n self.norm_layers_1.append(LayerNorm(hidden_channels))\n self.ffn_layers.append(\n FFN(\n hidden_channels,\n hidden_channels,\n filter_channels,\n kernel_size,\n p_dropout=p_dropout,\n )\n )\n self.norm_layers_2.append(LayerNorm(hidden_channels))\n\n def forward(self, x, x_mask):\n attn_mask = x_mask.unsqueeze(2) * x_mask.unsqueeze(-1)\n for i in range(self.n_layers):\n x = x * x_mask\n y = self.attn_layers[i](x, x, attn_mask)\n y = self.drop(y)\n x = self.norm_layers_1[i](x + y)\n y = self.ffn_layers[i](x, x_mask)\n y = self.drop(y)\n x = self.norm_layers_2[i](x + y)\n x = x * x_mask\n return x\n\n\nclass TextEncoder(nn.Module):\n def __init__(\n self,\n encoder_type,\n encoder_params,\n duration_predictor_params,\n n_vocab,\n n_spks=1,\n spk_emb_dim=128,\n ):\n super().__init__()\n self.encoder_type = encoder_type\n self.n_vocab = n_vocab\n self.n_feats = encoder_params.n_feats\n self.n_channels = encoder_params.n_channels\n self.spk_emb_dim = spk_emb_dim\n self.n_spks = n_spks\n\n self.emb = torch.nn.Embedding(n_vocab, self.n_channels)\n torch.nn.init.normal_(self.emb.weight, 0.0, self.n_channels**-0.5)\n\n if encoder_params.prenet:\n self.prenet = ConvReluNorm(\n self.n_channels,\n self.n_channels,\n self.n_channels,\n kernel_size=5,\n n_layers=3,\n p_dropout=0.5,\n )\n else:\n self.prenet = lambda x, x_mask: x\n\n self.encoder = Encoder(\n encoder_params.n_channels + (spk_emb_dim if n_spks > 1 else 0),\n encoder_params.filter_channels,\n encoder_params.n_heads,\n encoder_params.n_layers,\n encoder_params.kernel_size,\n encoder_params.p_dropout,\n )\n\n self.proj_m = torch.nn.Conv1d(self.n_channels + (spk_emb_dim if n_spks > 1 else 0), self.n_feats, 1)\n self.proj_w = DurationPredictor(\n self.n_channels + (spk_emb_dim if n_spks > 1 else 0),\n duration_predictor_params.filter_channels_dp,\n duration_predictor_params.kernel_size,\n duration_predictor_params.p_dropout,\n )\n\n def forward(self, x, x_lengths, spks=None):\n \"\"\"Run forward pass to the transformer based encoder and duration predictor\n\n Args:\n x (torch.Tensor): text input\n shape: (batch_size, max_text_length)\n x_lengths (torch.Tensor): text input lengths\n shape: (batch_size,)\n spks (torch.Tensor, optional): speaker ids. Defaults to None.\n shape: (batch_size,)\n\n Returns:\n mu (torch.Tensor): average output of the encoder\n shape: (batch_size, n_feats, max_text_length)\n logw (torch.Tensor): log duration predicted by the duration predictor\n shape: (batch_size, 1, max_text_length)\n x_mask (torch.Tensor): mask for the text input\n shape: (batch_size, 1, max_text_length)\n \"\"\"\n x = self.emb(x) * math.sqrt(self.n_channels)\n x = torch.transpose(x, 1, -1)\n x_mask = torch.unsqueeze(sequence_mask(x_lengths, x.size(2)), 1).to(x.dtype)\n\n x = self.prenet(x, x_mask)\n if self.n_spks > 1:\n x = torch.cat([x, spks.unsqueeze(-1).repeat(1, 1, x.shape[-1])], dim=1)\n x = self.encoder(x, x_mask)\n mu = self.proj_m(x) * x_mask\n\n x_dp = torch.detach(x)\n logw = self.proj_w(x_dp, x_mask)\n\n return mu, logw, x_mask\n" }, { "path": "third_party/Matcha-TTS/matcha/models/components/transformer.py", "content": "from typing import Any, Dict, Optional\n\nimport torch\nimport torch.nn as nn\nfrom diffusers.models.attention import (\n GEGLU,\n GELU,\n AdaLayerNorm,\n AdaLayerNormZero,\n ApproximateGELU,\n)\nfrom diffusers.models.attention_processor import Attention\nfrom diffusers.models.lora import LoRACompatibleLinear\nfrom diffusers.utils.torch_utils import maybe_allow_in_graph\n\n\nclass SnakeBeta(nn.Module):\n \"\"\"\n A modified Snake function which uses separate parameters for the magnitude of the periodic components\n Shape:\n - Input: (B, C, T)\n - Output: (B, C, T), same shape as the input\n Parameters:\n - alpha - trainable parameter that controls frequency\n - beta - trainable parameter that controls magnitude\n References:\n - This activation function is a modified version based on this paper by Liu Ziyin, Tilman Hartwig, Masahito Ueda:\n https://arxiv.org/abs/2006.08195\n Examples:\n >>> a1 = snakebeta(256)\n >>> x = torch.randn(256)\n >>> x = a1(x)\n \"\"\"\n\n def __init__(self, in_features, out_features, alpha=1.0, alpha_trainable=True, alpha_logscale=True):\n \"\"\"\n Initialization.\n INPUT:\n - in_features: shape of the input\n - alpha - trainable parameter that controls frequency\n - beta - trainable parameter that controls magnitude\n alpha is initialized to 1 by default, higher values = higher-frequency.\n beta is initialized to 1 by default, higher values = higher-magnitude.\n alpha will be trained along with the rest of your model.\n \"\"\"\n super().__init__()\n self.in_features = out_features if isinstance(out_features, list) else [out_features]\n self.proj = LoRACompatibleLinear(in_features, out_features)\n\n # initialize alpha\n self.alpha_logscale = alpha_logscale\n if self.alpha_logscale: # log scale alphas initialized to zeros\n self.alpha = nn.Parameter(torch.zeros(self.in_features) * alpha)\n self.beta = nn.Parameter(torch.zeros(self.in_features) * alpha)\n else: # linear scale alphas initialized to ones\n self.alpha = nn.Parameter(torch.ones(self.in_features) * alpha)\n self.beta = nn.Parameter(torch.ones(self.in_features) * alpha)\n\n self.alpha.requires_grad = alpha_trainable\n self.beta.requires_grad = alpha_trainable\n\n self.no_div_by_zero = 0.000000001\n\n def forward(self, x):\n \"\"\"\n Forward pass of the function.\n Applies the function to the input elementwise.\n SnakeBeta ∶= x + 1/b * sin^2 (xa)\n \"\"\"\n x = self.proj(x)\n if self.alpha_logscale:\n alpha = torch.exp(self.alpha)\n beta = torch.exp(self.beta)\n else:\n alpha = self.alpha\n beta = self.beta\n\n x = x + (1.0 / (beta + self.no_div_by_zero)) * torch.pow(torch.sin(x * alpha), 2)\n\n return x\n\n\nclass FeedForward(nn.Module):\n r\"\"\"\n A feed-forward layer.\n\n Parameters:\n dim (`int`): The number of channels in the input.\n dim_out (`int`, *optional*): The number of channels in the output. If not given, defaults to `dim`.\n mult (`int`, *optional*, defaults to 4): The multiplier to use for the hidden dimension.\n dropout (`float`, *optional*, defaults to 0.0): The dropout probability to use.\n activation_fn (`str`, *optional*, defaults to `\"geglu\"`): Activation function to be used in feed-forward.\n final_dropout (`bool` *optional*, defaults to False): Apply a final dropout.\n \"\"\"\n\n def __init__(\n self,\n dim: int,\n dim_out: Optional[int] = None,\n mult: int = 4,\n dropout: float = 0.0,\n activation_fn: str = \"geglu\",\n final_dropout: bool = False,\n ):\n super().__init__()\n inner_dim = int(dim * mult)\n dim_out = dim_out if dim_out is not None else dim\n\n if activation_fn == \"gelu\":\n act_fn = GELU(dim, inner_dim)\n if activation_fn == \"gelu-approximate\":\n act_fn = GELU(dim, inner_dim, approximate=\"tanh\")\n elif activation_fn == \"geglu\":\n act_fn = GEGLU(dim, inner_dim)\n elif activation_fn == \"geglu-approximate\":\n act_fn = ApproximateGELU(dim, inner_dim)\n elif activation_fn == \"snakebeta\":\n act_fn = SnakeBeta(dim, inner_dim)\n\n self.net = nn.ModuleList([])\n # project in\n self.net.append(act_fn)\n # project dropout\n self.net.append(nn.Dropout(dropout))\n # project out\n self.net.append(LoRACompatibleLinear(inner_dim, dim_out))\n # FF as used in Vision Transformer, MLP-Mixer, etc. have a final dropout\n if final_dropout:\n self.net.append(nn.Dropout(dropout))\n\n def forward(self, hidden_states):\n for module in self.net:\n hidden_states = module(hidden_states)\n return hidden_states\n\n\n@maybe_allow_in_graph\nclass BasicTransformerBlock(nn.Module):\n r\"\"\"\n A basic Transformer block.\n\n Parameters:\n dim (`int`): The number of channels in the input and output.\n num_attention_heads (`int`): The number of heads to use for multi-head attention.\n attention_head_dim (`int`): The number of channels in each head.\n dropout (`float`, *optional*, defaults to 0.0): The dropout probability to use.\n cross_attention_dim (`int`, *optional*): The size of the encoder_hidden_states vector for cross attention.\n only_cross_attention (`bool`, *optional*):\n Whether to use only cross-attention layers. In this case two cross attention layers are used.\n double_self_attention (`bool`, *optional*):\n Whether to use two self-attention layers. In this case no cross attention layers are used.\n activation_fn (`str`, *optional*, defaults to `\"geglu\"`): Activation function to be used in feed-forward.\n num_embeds_ada_norm (:\n obj: `int`, *optional*): The number of diffusion steps used during training. See `Transformer2DModel`.\n attention_bias (:\n obj: `bool`, *optional*, defaults to `False`): Configure if the attentions should contain a bias parameter.\n \"\"\"\n\n def __init__(\n self,\n dim: int,\n num_attention_heads: int,\n attention_head_dim: int,\n dropout=0.0,\n cross_attention_dim: Optional[int] = None,\n activation_fn: str = \"geglu\",\n num_embeds_ada_norm: Optional[int] = None,\n attention_bias: bool = False,\n only_cross_attention: bool = False,\n double_self_attention: bool = False,\n upcast_attention: bool = False,\n norm_elementwise_affine: bool = True,\n norm_type: str = \"layer_norm\",\n final_dropout: bool = False,\n ):\n super().__init__()\n self.only_cross_attention = only_cross_attention\n\n self.use_ada_layer_norm_zero = (num_embeds_ada_norm is not None) and norm_type == \"ada_norm_zero\"\n self.use_ada_layer_norm = (num_embeds_ada_norm is not None) and norm_type == \"ada_norm\"\n\n if norm_type in (\"ada_norm\", \"ada_norm_zero\") and num_embeds_ada_norm is None:\n raise ValueError(\n f\"`norm_type` is set to {norm_type}, but `num_embeds_ada_norm` is not defined. Please make sure to\"\n f\" define `num_embeds_ada_norm` if setting `norm_type` to {norm_type}.\"\n )\n\n # Define 3 blocks. Each block has its own normalization layer.\n # 1. Self-Attn\n if self.use_ada_layer_norm:\n self.norm1 = AdaLayerNorm(dim, num_embeds_ada_norm)\n elif self.use_ada_layer_norm_zero:\n self.norm1 = AdaLayerNormZero(dim, num_embeds_ada_norm)\n else:\n self.norm1 = nn.LayerNorm(dim, elementwise_affine=norm_elementwise_affine)\n self.attn1 = Attention(\n query_dim=dim,\n heads=num_attention_heads,\n dim_head=attention_head_dim,\n dropout=dropout,\n bias=attention_bias,\n cross_attention_dim=cross_attention_dim if only_cross_attention else None,\n upcast_attention=upcast_attention,\n )\n\n # 2. Cross-Attn\n if cross_attention_dim is not None or double_self_attention:\n # We currently only use AdaLayerNormZero for self attention where there will only be one attention block.\n # I.e. the number of returned modulation chunks from AdaLayerZero would not make sense if returned during\n # the second cross attention block.\n self.norm2 = (\n AdaLayerNorm(dim, num_embeds_ada_norm)\n if self.use_ada_layer_norm\n else nn.LayerNorm(dim, elementwise_affine=norm_elementwise_affine)\n )\n self.attn2 = Attention(\n query_dim=dim,\n cross_attention_dim=cross_attention_dim if not double_self_attention else None,\n heads=num_attention_heads,\n dim_head=attention_head_dim,\n dropout=dropout,\n bias=attention_bias,\n upcast_attention=upcast_attention,\n # scale_qk=False, # uncomment this to not to use flash attention\n ) # is self-attn if encoder_hidden_states is none\n else:\n self.norm2 = None\n self.attn2 = None\n\n # 3. Feed-forward\n self.norm3 = nn.LayerNorm(dim, elementwise_affine=norm_elementwise_affine)\n self.ff = FeedForward(dim, dropout=dropout, activation_fn=activation_fn, final_dropout=final_dropout)\n\n # let chunk size default to None\n self._chunk_size = None\n self._chunk_dim = 0\n\n def set_chunk_feed_forward(self, chunk_size: Optional[int], dim: int):\n # Sets chunk feed-forward\n self._chunk_size = chunk_size\n self._chunk_dim = dim\n\n def forward(\n self,\n hidden_states: torch.FloatTensor,\n attention_mask: Optional[torch.FloatTensor] = None,\n encoder_hidden_states: Optional[torch.FloatTensor] = None,\n encoder_attention_mask: Optional[torch.FloatTensor] = None,\n timestep: Optional[torch.LongTensor] = None,\n cross_attention_kwargs: Dict[str, Any] = None,\n class_labels: Optional[torch.LongTensor] = None,\n ):\n # Notice that normalization is always applied before the real computation in the following blocks.\n # 1. Self-Attention\n if self.use_ada_layer_norm:\n norm_hidden_states = self.norm1(hidden_states, timestep)\n elif self.use_ada_layer_norm_zero:\n norm_hidden_states, gate_msa, shift_mlp, scale_mlp, gate_mlp = self.norm1(\n hidden_states, timestep, class_labels, hidden_dtype=hidden_states.dtype\n )\n else:\n norm_hidden_states = self.norm1(hidden_states)\n\n cross_attention_kwargs = cross_attention_kwargs if cross_attention_kwargs is not None else {}\n\n attn_output = self.attn1(\n norm_hidden_states,\n encoder_hidden_states=encoder_hidden_states if self.only_cross_attention else None,\n attention_mask=encoder_attention_mask if self.only_cross_attention else attention_mask,\n **cross_attention_kwargs,\n )\n if self.use_ada_layer_norm_zero:\n attn_output = gate_msa.unsqueeze(1) * attn_output\n hidden_states = attn_output + hidden_states\n\n # 2. Cross-Attention\n if self.attn2 is not None:\n norm_hidden_states = (\n self.norm2(hidden_states, timestep) if self.use_ada_layer_norm else self.norm2(hidden_states)\n )\n\n attn_output = self.attn2(\n norm_hidden_states,\n encoder_hidden_states=encoder_hidden_states,\n attention_mask=encoder_attention_mask,\n **cross_attention_kwargs,\n )\n hidden_states = attn_output + hidden_states\n\n # 3. Feed-forward\n norm_hidden_states = self.norm3(hidden_states)\n\n if self.use_ada_layer_norm_zero:\n norm_hidden_states = norm_hidden_states * (1 + scale_mlp[:, None]) + shift_mlp[:, None]\n\n if self._chunk_size is not None:\n # \"feed_forward_chunk_size\" can be used to save memory\n if norm_hidden_states.shape[self._chunk_dim] % self._chunk_size != 0:\n raise ValueError(\n f\"`hidden_states` dimension to be chunked: {norm_hidden_states.shape[self._chunk_dim]} has to be divisible by chunk size: {self._chunk_size}. Make sure to set an appropriate `chunk_size` when calling `unet.enable_forward_chunking`.\"\n )\n\n num_chunks = norm_hidden_states.shape[self._chunk_dim] // self._chunk_size\n ff_output = torch.cat(\n [self.ff(hid_slice) for hid_slice in norm_hidden_states.chunk(num_chunks, dim=self._chunk_dim)],\n dim=self._chunk_dim,\n )\n else:\n ff_output = self.ff(norm_hidden_states)\n\n if self.use_ada_layer_norm_zero:\n ff_output = gate_mlp.unsqueeze(1) * ff_output\n\n hidden_states = ff_output + hidden_states\n\n return hidden_states\n" }, { "path": "third_party/Matcha-TTS/matcha/models/matcha_tts.py", "content": "import datetime as dt\nimport math\nimport random\n\nimport torch\n\nimport matcha.utils.monotonic_align as monotonic_align\nfrom matcha import utils\nfrom matcha.models.baselightningmodule import BaseLightningClass\nfrom matcha.models.components.flow_matching import CFM\nfrom matcha.models.components.text_encoder import TextEncoder\nfrom matcha.utils.model import (\n denormalize,\n duration_loss,\n fix_len_compatibility,\n generate_path,\n sequence_mask,\n)\n\nlog = utils.get_pylogger(__name__)\n\n\nclass MatchaTTS(BaseLightningClass): # 🍵\n def __init__(\n self,\n n_vocab,\n n_spks,\n spk_emb_dim,\n n_feats,\n encoder,\n decoder,\n cfm,\n data_statistics,\n out_size,\n optimizer=None,\n scheduler=None,\n prior_loss=True,\n ):\n super().__init__()\n\n self.save_hyperparameters(logger=False)\n\n self.n_vocab = n_vocab\n self.n_spks = n_spks\n self.spk_emb_dim = spk_emb_dim\n self.n_feats = n_feats\n self.out_size = out_size\n self.prior_loss = prior_loss\n\n if n_spks > 1:\n self.spk_emb = torch.nn.Embedding(n_spks, spk_emb_dim)\n\n self.encoder = TextEncoder(\n encoder.encoder_type,\n encoder.encoder_params,\n encoder.duration_predictor_params,\n n_vocab,\n n_spks,\n spk_emb_dim,\n )\n\n self.decoder = CFM(\n in_channels=2 * encoder.encoder_params.n_feats,\n out_channel=encoder.encoder_params.n_feats,\n cfm_params=cfm,\n decoder_params=decoder,\n n_spks=n_spks,\n spk_emb_dim=spk_emb_dim,\n )\n\n self.update_data_statistics(data_statistics)\n\n @torch.inference_mode()\n def synthesise(self, x, x_lengths, n_timesteps, temperature=1.0, spks=None, length_scale=1.0):\n \"\"\"\n Generates mel-spectrogram from text. Returns:\n 1. encoder outputs\n 2. decoder outputs\n 3. generated alignment\n\n Args:\n x (torch.Tensor): batch of texts, converted to a tensor with phoneme embedding ids.\n shape: (batch_size, max_text_length)\n x_lengths (torch.Tensor): lengths of texts in batch.\n shape: (batch_size,)\n n_timesteps (int): number of steps to use for reverse diffusion in decoder.\n temperature (float, optional): controls variance of terminal distribution.\n spks (bool, optional): speaker ids.\n shape: (batch_size,)\n length_scale (float, optional): controls speech pace.\n Increase value to slow down generated speech and vice versa.\n\n Returns:\n dict: {\n \"encoder_outputs\": torch.Tensor, shape: (batch_size, n_feats, max_mel_length),\n # Average mel spectrogram generated by the encoder\n \"decoder_outputs\": torch.Tensor, shape: (batch_size, n_feats, max_mel_length),\n # Refined mel spectrogram improved by the CFM\n \"attn\": torch.Tensor, shape: (batch_size, max_text_length, max_mel_length),\n # Alignment map between text and mel spectrogram\n \"mel\": torch.Tensor, shape: (batch_size, n_feats, max_mel_length),\n # Denormalized mel spectrogram\n \"mel_lengths\": torch.Tensor, shape: (batch_size,),\n # Lengths of mel spectrograms\n \"rtf\": float,\n # Real-time factor\n \"\"\"\n # For RTF computation\n t = dt.datetime.now()\n\n if self.n_spks > 1:\n # Get speaker embedding\n spks = self.spk_emb(spks.long())\n\n # Get encoder_outputs `mu_x` and log-scaled token durations `logw`\n mu_x, logw, x_mask = self.encoder(x, x_lengths, spks)\n\n w = torch.exp(logw) * x_mask\n w_ceil = torch.ceil(w) * length_scale\n y_lengths = torch.clamp_min(torch.sum(w_ceil, [1, 2]), 1).long()\n y_max_length = y_lengths.max()\n y_max_length_ = fix_len_compatibility(y_max_length)\n\n # Using obtained durations `w` construct alignment map `attn`\n y_mask = sequence_mask(y_lengths, y_max_length_).unsqueeze(1).to(x_mask.dtype)\n attn_mask = x_mask.unsqueeze(-1) * y_mask.unsqueeze(2)\n attn = generate_path(w_ceil.squeeze(1), attn_mask.squeeze(1)).unsqueeze(1)\n\n # Align encoded text and get mu_y\n mu_y = torch.matmul(attn.squeeze(1).transpose(1, 2), mu_x.transpose(1, 2))\n mu_y = mu_y.transpose(1, 2)\n encoder_outputs = mu_y[:, :, :y_max_length]\n\n # Generate sample tracing the probability flow\n decoder_outputs = self.decoder(mu_y, y_mask, n_timesteps, temperature, spks)\n decoder_outputs = decoder_outputs[:, :, :y_max_length]\n\n t = (dt.datetime.now() - t).total_seconds()\n rtf = t * 22050 / (decoder_outputs.shape[-1] * 256)\n\n return {\n \"encoder_outputs\": encoder_outputs,\n \"decoder_outputs\": decoder_outputs,\n \"attn\": attn[:, :, :y_max_length],\n \"mel\": denormalize(decoder_outputs, self.mel_mean, self.mel_std),\n \"mel_lengths\": y_lengths,\n \"rtf\": rtf,\n }\n\n def forward(self, x, x_lengths, y, y_lengths, spks=None, out_size=None, cond=None):\n \"\"\"\n Computes 3 losses:\n 1. duration loss: loss between predicted token durations and those extracted by Monotinic Alignment Search (MAS).\n 2. prior loss: loss between mel-spectrogram and encoder outputs.\n 3. flow matching loss: loss between mel-spectrogram and decoder outputs.\n\n Args:\n x (torch.Tensor): batch of texts, converted to a tensor with phoneme embedding ids.\n shape: (batch_size, max_text_length)\n x_lengths (torch.Tensor): lengths of texts in batch.\n shape: (batch_size,)\n y (torch.Tensor): batch of corresponding mel-spectrograms.\n shape: (batch_size, n_feats, max_mel_length)\n y_lengths (torch.Tensor): lengths of mel-spectrograms in batch.\n shape: (batch_size,)\n out_size (int, optional): length (in mel's sampling rate) of segment to cut, on which decoder will be trained.\n Should be divisible by 2^{num of UNet downsamplings}. Needed to increase batch size.\n spks (torch.Tensor, optional): speaker ids.\n shape: (batch_size,)\n \"\"\"\n if self.n_spks > 1:\n # Get speaker embedding\n spks = self.spk_emb(spks)\n\n # Get encoder_outputs `mu_x` and log-scaled token durations `logw`\n mu_x, logw, x_mask = self.encoder(x, x_lengths, spks)\n y_max_length = y.shape[-1]\n\n y_mask = sequence_mask(y_lengths, y_max_length).unsqueeze(1).to(x_mask)\n attn_mask = x_mask.unsqueeze(-1) * y_mask.unsqueeze(2)\n\n # Use MAS to find most likely alignment `attn` between text and mel-spectrogram\n with torch.no_grad():\n const = -0.5 * math.log(2 * math.pi) * self.n_feats\n factor = -0.5 * torch.ones(mu_x.shape, dtype=mu_x.dtype, device=mu_x.device)\n y_square = torch.matmul(factor.transpose(1, 2), y**2)\n y_mu_double = torch.matmul(2.0 * (factor * mu_x).transpose(1, 2), y)\n mu_square = torch.sum(factor * (mu_x**2), 1).unsqueeze(-1)\n log_prior = y_square - y_mu_double + mu_square + const\n\n attn = monotonic_align.maximum_path(log_prior, attn_mask.squeeze(1))\n attn = attn.detach()\n\n # Compute loss between predicted log-scaled durations and those obtained from MAS\n # refered to as prior loss in the paper\n logw_ = torch.log(1e-8 + torch.sum(attn.unsqueeze(1), -1)) * x_mask\n dur_loss = duration_loss(logw, logw_, x_lengths)\n\n # Cut a small segment of mel-spectrogram in order to increase batch size\n # - \"Hack\" taken from Grad-TTS, in case of Grad-TTS, we cannot train batch size 32 on a 24GB GPU without it\n # - Do not need this hack for Matcha-TTS, but it works with it as well\n if not isinstance(out_size, type(None)):\n max_offset = (y_lengths - out_size).clamp(0)\n offset_ranges = list(zip([0] * max_offset.shape[0], max_offset.cpu().numpy()))\n out_offset = torch.LongTensor(\n [torch.tensor(random.choice(range(start, end)) if end > start else 0) for start, end in offset_ranges]\n ).to(y_lengths)\n attn_cut = torch.zeros(attn.shape[0], attn.shape[1], out_size, dtype=attn.dtype, device=attn.device)\n y_cut = torch.zeros(y.shape[0], self.n_feats, out_size, dtype=y.dtype, device=y.device)\n\n y_cut_lengths = []\n for i, (y_, out_offset_) in enumerate(zip(y, out_offset)):\n y_cut_length = out_size + (y_lengths[i] - out_size).clamp(None, 0)\n y_cut_lengths.append(y_cut_length)\n cut_lower, cut_upper = out_offset_, out_offset_ + y_cut_length\n y_cut[i, :, :y_cut_length] = y_[:, cut_lower:cut_upper]\n attn_cut[i, :, :y_cut_length] = attn[i, :, cut_lower:cut_upper]\n\n y_cut_lengths = torch.LongTensor(y_cut_lengths)\n y_cut_mask = sequence_mask(y_cut_lengths).unsqueeze(1).to(y_mask)\n\n attn = attn_cut\n y = y_cut\n y_mask = y_cut_mask\n\n # Align encoded text with mel-spectrogram and get mu_y segment\n mu_y = torch.matmul(attn.squeeze(1).transpose(1, 2), mu_x.transpose(1, 2))\n mu_y = mu_y.transpose(1, 2)\n\n # Compute loss of the decoder\n diff_loss, _ = self.decoder.compute_loss(x1=y, mask=y_mask, mu=mu_y, spks=spks, cond=cond)\n\n if self.prior_loss:\n prior_loss = torch.sum(0.5 * ((y - mu_y) ** 2 + math.log(2 * math.pi)) * y_mask)\n prior_loss = prior_loss / (torch.sum(y_mask) * self.n_feats)\n else:\n prior_loss = 0\n\n return dur_loss, prior_loss, diff_loss\n" }, { "path": "third_party/Matcha-TTS/matcha/onnx/__init__.py", "content": "" }, { "path": "third_party/Matcha-TTS/matcha/onnx/export.py", "content": "import argparse\nimport random\nfrom pathlib import Path\n\nimport numpy as np\nimport torch\nfrom lightning import LightningModule\n\nfrom matcha.cli import VOCODER_URLS, load_matcha, load_vocoder\n\nDEFAULT_OPSET = 15\n\nSEED = 1234\nrandom.seed(SEED)\nnp.random.seed(SEED)\ntorch.manual_seed(SEED)\ntorch.cuda.manual_seed(SEED)\ntorch.backends.cudnn.deterministic = True\ntorch.backends.cudnn.benchmark = False\n\n\nclass MatchaWithVocoder(LightningModule):\n def __init__(self, matcha, vocoder):\n super().__init__()\n self.matcha = matcha\n self.vocoder = vocoder\n\n def forward(self, x, x_lengths, scales, spks=None):\n mel, mel_lengths = self.matcha(x, x_lengths, scales, spks)\n wavs = self.vocoder(mel).clamp(-1, 1)\n lengths = mel_lengths * 256\n return wavs.squeeze(1), lengths\n\n\ndef get_exportable_module(matcha, vocoder, n_timesteps):\n \"\"\"\n Return an appropriate `LighteningModule` and output-node names\n based on whether the vocoder is embedded in the final graph\n \"\"\"\n\n def onnx_forward_func(x, x_lengths, scales, spks=None):\n \"\"\"\n Custom forward function for accepting\n scaler parameters as tensors\n \"\"\"\n # Extract scaler parameters from tensors\n temperature = scales[0]\n length_scale = scales[1]\n output = matcha.synthesise(x, x_lengths, n_timesteps, temperature, spks, length_scale)\n return output[\"mel\"], output[\"mel_lengths\"]\n\n # Monkey-patch Matcha's forward function\n matcha.forward = onnx_forward_func\n\n if vocoder is None:\n model, output_names = matcha, [\"mel\", \"mel_lengths\"]\n else:\n model = MatchaWithVocoder(matcha, vocoder)\n output_names = [\"wav\", \"wav_lengths\"]\n return model, output_names\n\n\ndef get_inputs(is_multi_speaker):\n \"\"\"\n Create dummy inputs for tracing\n \"\"\"\n dummy_input_length = 50\n x = torch.randint(low=0, high=20, size=(1, dummy_input_length), dtype=torch.long)\n x_lengths = torch.LongTensor([dummy_input_length])\n\n # Scales\n temperature = 0.667\n length_scale = 1.0\n scales = torch.Tensor([temperature, length_scale])\n\n model_inputs = [x, x_lengths, scales]\n input_names = [\n \"x\",\n \"x_lengths\",\n \"scales\",\n ]\n\n if is_multi_speaker:\n spks = torch.LongTensor([1])\n model_inputs.append(spks)\n input_names.append(\"spks\")\n\n return tuple(model_inputs), input_names\n\n\ndef main():\n parser = argparse.ArgumentParser(description=\"Export 🍵 Matcha-TTS to ONNX\")\n\n parser.add_argument(\n \"checkpoint_path\",\n type=str,\n help=\"Path to the model checkpoint\",\n )\n parser.add_argument(\"output\", type=str, help=\"Path to output `.onnx` file\")\n parser.add_argument(\n \"--n-timesteps\", type=int, default=5, help=\"Number of steps to use for reverse diffusion in decoder (default 5)\"\n )\n parser.add_argument(\n \"--vocoder-name\",\n type=str,\n choices=list(VOCODER_URLS.keys()),\n default=None,\n help=\"Name of the vocoder to embed in the ONNX graph\",\n )\n parser.add_argument(\n \"--vocoder-checkpoint-path\",\n type=str,\n default=None,\n help=\"Vocoder checkpoint to embed in the ONNX graph for an `e2e` like experience\",\n )\n parser.add_argument(\"--opset\", type=int, default=DEFAULT_OPSET, help=\"ONNX opset version to use (default 15\")\n\n args = parser.parse_args()\n\n print(f\"[🍵] Loading Matcha checkpoint from {args.checkpoint_path}\")\n print(f\"Setting n_timesteps to {args.n_timesteps}\")\n\n checkpoint_path = Path(args.checkpoint_path)\n matcha = load_matcha(checkpoint_path.stem, checkpoint_path, \"cpu\")\n\n if args.vocoder_name or args.vocoder_checkpoint_path:\n assert (\n args.vocoder_name and args.vocoder_checkpoint_path\n ), \"Both vocoder_name and vocoder-checkpoint are required when embedding the vocoder in the ONNX graph.\"\n vocoder, _ = load_vocoder(args.vocoder_name, args.vocoder_checkpoint_path, \"cpu\")\n else:\n vocoder = None\n\n is_multi_speaker = matcha.n_spks > 1\n\n dummy_input, input_names = get_inputs(is_multi_speaker)\n model, output_names = get_exportable_module(matcha, vocoder, args.n_timesteps)\n\n # Set dynamic shape for inputs/outputs\n dynamic_axes = {\n \"x\": {0: \"batch_size\", 1: \"time\"},\n \"x_lengths\": {0: \"batch_size\"},\n }\n\n if vocoder is None:\n dynamic_axes.update(\n {\n \"mel\": {0: \"batch_size\", 2: \"time\"},\n \"mel_lengths\": {0: \"batch_size\"},\n }\n )\n else:\n print(\"Embedding the vocoder in the ONNX graph\")\n dynamic_axes.update(\n {\n \"wav\": {0: \"batch_size\", 1: \"time\"},\n \"wav_lengths\": {0: \"batch_size\"},\n }\n )\n\n if is_multi_speaker:\n dynamic_axes[\"spks\"] = {0: \"batch_size\"}\n\n # Create the output directory (if not exists)\n Path(args.output).parent.mkdir(parents=True, exist_ok=True)\n\n model.to_onnx(\n args.output,\n dummy_input,\n input_names=input_names,\n output_names=output_names,\n dynamic_axes=dynamic_axes,\n opset_version=args.opset,\n export_params=True,\n do_constant_folding=True,\n )\n print(f\"[🍵] ONNX model exported to {args.output}\")\n\n\nif __name__ == \"__main__\":\n main()\n" }, { "path": "third_party/Matcha-TTS/matcha/onnx/infer.py", "content": "import argparse\nimport os\nimport warnings\nfrom pathlib import Path\nfrom time import perf_counter\n\nimport numpy as np\nimport onnxruntime as ort\nimport soundfile as sf\nimport torch\n\nfrom matcha.cli import plot_spectrogram_to_numpy, process_text\n\n\ndef validate_args(args):\n assert (\n args.text or args.file\n ), \"Either text or file must be provided Matcha-T(ea)TTS need sometext to whisk the waveforms.\"\n assert args.temperature >= 0, \"Sampling temperature cannot be negative\"\n assert args.speaking_rate >= 0, \"Speaking rate must be greater than 0\"\n return args\n\n\ndef write_wavs(model, inputs, output_dir, external_vocoder=None):\n if external_vocoder is None:\n print(\"The provided model has the vocoder embedded in the graph.\\nGenerating waveform directly\")\n t0 = perf_counter()\n wavs, wav_lengths = model.run(None, inputs)\n infer_secs = perf_counter() - t0\n mel_infer_secs = vocoder_infer_secs = None\n else:\n print(\"[🍵] Generating mel using Matcha\")\n mel_t0 = perf_counter()\n mels, mel_lengths = model.run(None, inputs)\n mel_infer_secs = perf_counter() - mel_t0\n print(\"Generating waveform from mel using external vocoder\")\n vocoder_inputs = {external_vocoder.get_inputs()[0].name: mels}\n vocoder_t0 = perf_counter()\n wavs = external_vocoder.run(None, vocoder_inputs)[0]\n vocoder_infer_secs = perf_counter() - vocoder_t0\n wavs = wavs.squeeze(1)\n wav_lengths = mel_lengths * 256\n infer_secs = mel_infer_secs + vocoder_infer_secs\n\n output_dir = Path(output_dir)\n output_dir.mkdir(parents=True, exist_ok=True)\n for i, (wav, wav_length) in enumerate(zip(wavs, wav_lengths)):\n output_filename = output_dir.joinpath(f\"output_{i + 1}.wav\")\n audio = wav[:wav_length]\n print(f\"Writing audio to {output_filename}\")\n sf.write(output_filename, audio, 22050, \"PCM_24\")\n\n wav_secs = wav_lengths.sum() / 22050\n print(f\"Inference seconds: {infer_secs}\")\n print(f\"Generated wav seconds: {wav_secs}\")\n rtf = infer_secs / wav_secs\n if mel_infer_secs is not None:\n mel_rtf = mel_infer_secs / wav_secs\n print(f\"Matcha RTF: {mel_rtf}\")\n if vocoder_infer_secs is not None:\n vocoder_rtf = vocoder_infer_secs / wav_secs\n print(f\"Vocoder RTF: {vocoder_rtf}\")\n print(f\"Overall RTF: {rtf}\")\n\n\ndef write_mels(model, inputs, output_dir):\n t0 = perf_counter()\n mels, mel_lengths = model.run(None, inputs)\n infer_secs = perf_counter() - t0\n\n output_dir = Path(output_dir)\n output_dir.mkdir(parents=True, exist_ok=True)\n for i, mel in enumerate(mels):\n output_stem = output_dir.joinpath(f\"output_{i + 1}\")\n plot_spectrogram_to_numpy(mel.squeeze(), output_stem.with_suffix(\".png\"))\n np.save(output_stem.with_suffix(\".numpy\"), mel)\n\n wav_secs = (mel_lengths * 256).sum() / 22050\n print(f\"Inference seconds: {infer_secs}\")\n print(f\"Generated wav seconds: {wav_secs}\")\n rtf = infer_secs / wav_secs\n print(f\"RTF: {rtf}\")\n\n\ndef main():\n parser = argparse.ArgumentParser(\n description=\" 🍵 Matcha-TTS: A fast TTS architecture with conditional flow matching\"\n )\n parser.add_argument(\n \"model\",\n type=str,\n help=\"ONNX model to use\",\n )\n parser.add_argument(\"--vocoder\", type=str, default=None, help=\"Vocoder to use (defaults to None)\")\n parser.add_argument(\"--text\", type=str, default=None, help=\"Text to synthesize\")\n parser.add_argument(\"--file\", type=str, default=None, help=\"Text file to synthesize\")\n parser.add_argument(\"--spk\", type=int, default=None, help=\"Speaker ID\")\n parser.add_argument(\n \"--temperature\",\n type=float,\n default=0.667,\n help=\"Variance of the x0 noise (default: 0.667)\",\n )\n parser.add_argument(\n \"--speaking-rate\",\n type=float,\n default=1.0,\n help=\"change the speaking rate, a higher value means slower speaking rate (default: 1.0)\",\n )\n parser.add_argument(\"--gpu\", action=\"store_true\", help=\"Use CPU for inference (default: use GPU if available)\")\n parser.add_argument(\n \"--output-dir\",\n type=str,\n default=os.getcwd(),\n help=\"Output folder to save results (default: current dir)\",\n )\n\n args = parser.parse_args()\n args = validate_args(args)\n\n if args.gpu:\n providers = [\"GPUExecutionProvider\"]\n else:\n providers = [\"CPUExecutionProvider\"]\n model = ort.InferenceSession(args.model, providers=providers)\n\n model_inputs = model.get_inputs()\n model_outputs = list(model.get_outputs())\n\n if args.text:\n text_lines = args.text.splitlines()\n else:\n with open(args.file, encoding=\"utf-8\") as file:\n text_lines = file.read().splitlines()\n\n processed_lines = [process_text(0, line, \"cpu\") for line in text_lines]\n x = [line[\"x\"].squeeze() for line in processed_lines]\n # Pad\n x = torch.nn.utils.rnn.pad_sequence(x, batch_first=True)\n x = x.detach().cpu().numpy()\n x_lengths = np.array([line[\"x_lengths\"].item() for line in processed_lines], dtype=np.int64)\n inputs = {\n \"x\": x,\n \"x_lengths\": x_lengths,\n \"scales\": np.array([args.temperature, args.speaking_rate], dtype=np.float32),\n }\n is_multi_speaker = len(model_inputs) == 4\n if is_multi_speaker:\n if args.spk is None:\n args.spk = 0\n warn = \"[!] Speaker ID not provided! Using speaker ID 0\"\n warnings.warn(warn, UserWarning)\n inputs[\"spks\"] = np.repeat(args.spk, x.shape[0]).astype(np.int64)\n\n has_vocoder_embedded = model_outputs[0].name == \"wav\"\n if has_vocoder_embedded:\n write_wavs(model, inputs, args.output_dir)\n elif args.vocoder:\n external_vocoder = ort.InferenceSession(args.vocoder, providers=providers)\n write_wavs(model, inputs, args.output_dir, external_vocoder=external_vocoder)\n else:\n warn = \"[!] A vocoder is not embedded in the graph nor an external vocoder is provided. The mel output will be written as numpy arrays to `*.npy` files in the output directory\"\n warnings.warn(warn, UserWarning)\n write_mels(model, inputs, args.output_dir)\n\n\nif __name__ == \"__main__\":\n main()\n" }, { "path": "third_party/Matcha-TTS/matcha/text/__init__.py", "content": "\"\"\" from https://github.com/keithito/tacotron \"\"\"\nfrom matcha.text import cleaners\nfrom matcha.text.symbols import symbols\n\n# Mappings from symbol to numeric ID and vice versa:\n_symbol_to_id = {s: i for i, s in enumerate(symbols)}\n_id_to_symbol = {i: s for i, s in enumerate(symbols)} # pylint: disable=unnecessary-comprehension\n\n\ndef text_to_sequence(text, cleaner_names):\n \"\"\"Converts a string of text to a sequence of IDs corresponding to the symbols in the text.\n Args:\n text: string to convert to a sequence\n cleaner_names: names of the cleaner functions to run the text through\n Returns:\n List of integers corresponding to the symbols in the text\n \"\"\"\n sequence = []\n\n clean_text = _clean_text(text, cleaner_names)\n for symbol in clean_text:\n symbol_id = _symbol_to_id[symbol]\n sequence += [symbol_id]\n return sequence\n\n\ndef cleaned_text_to_sequence(cleaned_text):\n \"\"\"Converts a string of text to a sequence of IDs corresponding to the symbols in the text.\n Args:\n text: string to convert to a sequence\n Returns:\n List of integers corresponding to the symbols in the text\n \"\"\"\n sequence = [_symbol_to_id[symbol] for symbol in cleaned_text]\n return sequence\n\n\ndef sequence_to_text(sequence):\n \"\"\"Converts a sequence of IDs back to a string\"\"\"\n result = \"\"\n for symbol_id in sequence:\n s = _id_to_symbol[symbol_id]\n result += s\n return result\n\n\ndef _clean_text(text, cleaner_names):\n for name in cleaner_names:\n cleaner = getattr(cleaners, name)\n if not cleaner:\n raise Exception(\"Unknown cleaner: %s\" % name)\n text = cleaner(text)\n return text\n" }, { "path": "third_party/Matcha-TTS/matcha/text/cleaners.py", "content": "\"\"\" from https://github.com/keithito/tacotron\n\nCleaners are transformations that run over the input text at both training and eval time.\n\nCleaners can be selected by passing a comma-delimited list of cleaner names as the \"cleaners\"\nhyperparameter. Some cleaners are English-specific. You'll typically want to use:\n 1. \"english_cleaners\" for English text\n 2. \"transliteration_cleaners\" for non-English text that can be transliterated to ASCII using\n the Unidecode library (https://pypi.python.org/pypi/Unidecode)\n 3. \"basic_cleaners\" if you do not want to transliterate (in this case, you should also update\n the symbols in symbols.py to match your data).\n\"\"\"\n\nimport logging\nimport re\n\nimport phonemizer\nimport piper_phonemize\nfrom unidecode import unidecode\n\n# To avoid excessive logging we set the log level of the phonemizer package to Critical\ncritical_logger = logging.getLogger(\"phonemizer\")\ncritical_logger.setLevel(logging.CRITICAL)\n\n# Intializing the phonemizer globally significantly reduces the speed\n# now the phonemizer is not initialising at every call\n# Might be less flexible, but it is much-much faster\nglobal_phonemizer = phonemizer.backend.EspeakBackend(\n language=\"en-us\",\n preserve_punctuation=True,\n with_stress=True,\n language_switch=\"remove-flags\",\n logger=critical_logger,\n)\n\n\n# Regular expression matching whitespace:\n_whitespace_re = re.compile(r\"\\s+\")\n\n# List of (regular expression, replacement) pairs for abbreviations:\n_abbreviations = [\n (re.compile(\"\\\\b%s\\\\.\" % x[0], re.IGNORECASE), x[1])\n for x in [\n (\"mrs\", \"misess\"),\n (\"mr\", \"mister\"),\n (\"dr\", \"doctor\"),\n (\"st\", \"saint\"),\n (\"co\", \"company\"),\n (\"jr\", \"junior\"),\n (\"maj\", \"major\"),\n (\"gen\", \"general\"),\n (\"drs\", \"doctors\"),\n (\"rev\", \"reverend\"),\n (\"lt\", \"lieutenant\"),\n (\"hon\", \"honorable\"),\n (\"sgt\", \"sergeant\"),\n (\"capt\", \"captain\"),\n (\"esq\", \"esquire\"),\n (\"ltd\", \"limited\"),\n (\"col\", \"colonel\"),\n (\"ft\", \"fort\"),\n ]\n]\n\n\ndef expand_abbreviations(text):\n for regex, replacement in _abbreviations:\n text = re.sub(regex, replacement, text)\n return text\n\n\ndef lowercase(text):\n return text.lower()\n\n\ndef collapse_whitespace(text):\n return re.sub(_whitespace_re, \" \", text)\n\n\ndef convert_to_ascii(text):\n return unidecode(text)\n\n\ndef basic_cleaners(text):\n \"\"\"Basic pipeline that lowercases and collapses whitespace without transliteration.\"\"\"\n text = lowercase(text)\n text = collapse_whitespace(text)\n return text\n\n\ndef transliteration_cleaners(text):\n \"\"\"Pipeline for non-English text that transliterates to ASCII.\"\"\"\n text = convert_to_ascii(text)\n text = lowercase(text)\n text = collapse_whitespace(text)\n return text\n\n\ndef english_cleaners2(text):\n \"\"\"Pipeline for English text, including abbreviation expansion. + punctuation + stress\"\"\"\n text = convert_to_ascii(text)\n text = lowercase(text)\n text = expand_abbreviations(text)\n phonemes = global_phonemizer.phonemize([text], strip=True, njobs=1)[0]\n phonemes = collapse_whitespace(phonemes)\n return phonemes\n\n\ndef english_cleaners_piper(text):\n \"\"\"Pipeline for English text, including abbreviation expansion. + punctuation + stress\"\"\"\n text = convert_to_ascii(text)\n text = lowercase(text)\n text = expand_abbreviations(text)\n phonemes = \"\".join(piper_phonemize.phonemize_espeak(text=text, voice=\"en-US\")[0])\n phonemes = collapse_whitespace(phonemes)\n return phonemes\n" }, { "path": "third_party/Matcha-TTS/matcha/text/numbers.py", "content": "\"\"\" from https://github.com/keithito/tacotron \"\"\"\n\nimport re\n\nimport inflect\n\n_inflect = inflect.engine()\n_comma_number_re = re.compile(r\"([0-9][0-9\\,]+[0-9])\")\n_decimal_number_re = re.compile(r\"([0-9]+\\.[0-9]+)\")\n_pounds_re = re.compile(r\"£([0-9\\,]*[0-9]+)\")\n_dollars_re = re.compile(r\"\\$([0-9\\.\\,]*[0-9]+)\")\n_ordinal_re = re.compile(r\"[0-9]+(st|nd|rd|th)\")\n_number_re = re.compile(r\"[0-9]+\")\n\n\ndef _remove_commas(m):\n return m.group(1).replace(\",\", \"\")\n\n\ndef _expand_decimal_point(m):\n return m.group(1).replace(\".\", \" point \")\n\n\ndef _expand_dollars(m):\n match = m.group(1)\n parts = match.split(\".\")\n if len(parts) > 2:\n return match + \" dollars\"\n dollars = int(parts[0]) if parts[0] else 0\n cents = int(parts[1]) if len(parts) > 1 and parts[1] else 0\n if dollars and cents:\n dollar_unit = \"dollar\" if dollars == 1 else \"dollars\"\n cent_unit = \"cent\" if cents == 1 else \"cents\"\n return f\"{dollars} {dollar_unit}, {cents} {cent_unit}\"\n elif dollars:\n dollar_unit = \"dollar\" if dollars == 1 else \"dollars\"\n return f\"{dollars} {dollar_unit}\"\n elif cents:\n cent_unit = \"cent\" if cents == 1 else \"cents\"\n return f\"{cents} {cent_unit}\"\n else:\n return \"zero dollars\"\n\n\ndef _expand_ordinal(m):\n return _inflect.number_to_words(m.group(0))\n\n\ndef _expand_number(m):\n num = int(m.group(0))\n if num > 1000 and num < 3000:\n if num == 2000:\n return \"two thousand\"\n elif num > 2000 and num < 2010:\n return \"two thousand \" + _inflect.number_to_words(num % 100)\n elif num % 100 == 0:\n return _inflect.number_to_words(num // 100) + \" hundred\"\n else:\n return _inflect.number_to_words(num, andword=\"\", zero=\"oh\", group=2).replace(\", \", \" \")\n else:\n return _inflect.number_to_words(num, andword=\"\")\n\n\ndef normalize_numbers(text):\n text = re.sub(_comma_number_re, _remove_commas, text)\n text = re.sub(_pounds_re, r\"\\1 pounds\", text)\n text = re.sub(_dollars_re, _expand_dollars, text)\n text = re.sub(_decimal_number_re, _expand_decimal_point, text)\n text = re.sub(_ordinal_re, _expand_ordinal, text)\n text = re.sub(_number_re, _expand_number, text)\n return text\n" }, { "path": "third_party/Matcha-TTS/matcha/text/symbols.py", "content": "\"\"\" from https://github.com/keithito/tacotron\n\nDefines the set of symbols used in text input to the model.\n\"\"\"\n_pad = \"_\"\n_punctuation = ';:,.!?¡¿—…\"«»“” '\n_letters = \"ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz\"\n_letters_ipa = (\n \"ɑɐɒæɓʙβɔɕçɗɖðʤəɘɚɛɜɝɞɟʄɡɠɢʛɦɧħɥʜɨɪʝɭɬɫɮʟɱɯɰŋɳɲɴøɵɸθœɶʘɹɺɾɻʀʁɽʂʃʈʧʉʊʋⱱʌɣɤʍχʎʏʑʐʒʔʡʕʢǀǁǂǃˈˌːˑʼʴʰʱʲʷˠˤ˞↓↑→↗↘'̩'ᵻ\"\n)\n\n\n# Export all symbols:\nsymbols = [_pad] + list(_punctuation) + list(_letters) + list(_letters_ipa)\n\n# Special symbol ids\nSPACE_ID = symbols.index(\" \")\n" }, { "path": "third_party/Matcha-TTS/matcha/train.py", "content": "from typing import Any, Dict, List, Optional, Tuple\n\nimport hydra\nimport lightning as L\nimport rootutils\nfrom lightning import Callback, LightningDataModule, LightningModule, Trainer\nfrom lightning.pytorch.loggers import Logger\nfrom omegaconf import DictConfig\n\nfrom matcha import utils\n\nrootutils.setup_root(__file__, indicator=\".project-root\", pythonpath=True)\n# ------------------------------------------------------------------------------------ #\n# the setup_root above is equivalent to:\n# - adding project root dir to PYTHONPATH\n# (so you don't need to force user to install project as a package)\n# (necessary before importing any local modules e.g. `from src import utils`)\n# - setting up PROJECT_ROOT environment variable\n# (which is used as a base for paths in \"configs/paths/default.yaml\")\n# (this way all filepaths are the same no matter where you run the code)\n# - loading environment variables from \".env\" in root dir\n#\n# you can remove it if you:\n# 1. either install project as a package or move entry files to project root dir\n# 2. set `root_dir` to \".\" in \"configs/paths/default.yaml\"\n#\n# more info: https://github.com/ashleve/rootutils\n# ------------------------------------------------------------------------------------ #\n\n\nlog = utils.get_pylogger(__name__)\n\n\n@utils.task_wrapper\ndef train(cfg: DictConfig) -> Tuple[Dict[str, Any], Dict[str, Any]]:\n \"\"\"Trains the model. Can additionally evaluate on a testset, using best weights obtained during\n training.\n\n This method is wrapped in optional @task_wrapper decorator, that controls the behavior during\n failure. Useful for multiruns, saving info about the crash, etc.\n\n :param cfg: A DictConfig configuration composed by Hydra.\n :return: A tuple with metrics and dict with all instantiated objects.\n \"\"\"\n # set seed for random number generators in pytorch, numpy and python.random\n if cfg.get(\"seed\"):\n L.seed_everything(cfg.seed, workers=True)\n\n log.info(f\"Instantiating datamodule <{cfg.data._target_}>\") # pylint: disable=protected-access\n datamodule: LightningDataModule = hydra.utils.instantiate(cfg.data)\n\n log.info(f\"Instantiating model <{cfg.model._target_}>\") # pylint: disable=protected-access\n model: LightningModule = hydra.utils.instantiate(cfg.model)\n\n log.info(\"Instantiating callbacks...\")\n callbacks: List[Callback] = utils.instantiate_callbacks(cfg.get(\"callbacks\"))\n\n log.info(\"Instantiating loggers...\")\n logger: List[Logger] = utils.instantiate_loggers(cfg.get(\"logger\"))\n\n log.info(f\"Instantiating trainer <{cfg.trainer._target_}>\") # pylint: disable=protected-access\n trainer: Trainer = hydra.utils.instantiate(cfg.trainer, callbacks=callbacks, logger=logger)\n\n object_dict = {\n \"cfg\": cfg,\n \"datamodule\": datamodule,\n \"model\": model,\n \"callbacks\": callbacks,\n \"logger\": logger,\n \"trainer\": trainer,\n }\n\n if logger:\n log.info(\"Logging hyperparameters!\")\n utils.log_hyperparameters(object_dict)\n\n if cfg.get(\"train\"):\n log.info(\"Starting training!\")\n trainer.fit(model=model, datamodule=datamodule, ckpt_path=cfg.get(\"ckpt_path\"))\n\n train_metrics = trainer.callback_metrics\n\n if cfg.get(\"test\"):\n log.info(\"Starting testing!\")\n ckpt_path = trainer.checkpoint_callback.best_model_path\n if ckpt_path == \"\":\n log.warning(\"Best ckpt not found! Using current weights for testing...\")\n ckpt_path = None\n trainer.test(model=model, datamodule=datamodule, ckpt_path=ckpt_path)\n log.info(f\"Best ckpt path: {ckpt_path}\")\n\n test_metrics = trainer.callback_metrics\n\n # merge train and test metrics\n metric_dict = {**train_metrics, **test_metrics}\n\n return metric_dict, object_dict\n\n\n@hydra.main(version_base=\"1.3\", config_path=\"../configs\", config_name=\"train.yaml\")\ndef main(cfg: DictConfig) -> Optional[float]:\n \"\"\"Main entry point for training.\n\n :param cfg: DictConfig configuration composed by Hydra.\n :return: Optional[float] with optimized metric value.\n \"\"\"\n # apply extra utilities\n # (e.g. ask for tags if none are provided in cfg, print cfg tree, etc.)\n utils.extras(cfg)\n\n # train the model\n metric_dict, _ = train(cfg)\n\n # safely retrieve metric value for hydra-based hyperparameter optimization\n metric_value = utils.get_metric_value(metric_dict=metric_dict, metric_name=cfg.get(\"optimized_metric\"))\n\n # return optimized metric\n return metric_value\n\n\nif __name__ == \"__main__\":\n main() # pylint: disable=no-value-for-parameter\n" }, { "path": "third_party/Matcha-TTS/matcha/utils/__init__.py", "content": "from matcha.utils.instantiators import instantiate_callbacks, instantiate_loggers\nfrom matcha.utils.logging_utils import log_hyperparameters\nfrom matcha.utils.pylogger import get_pylogger\nfrom matcha.utils.rich_utils import enforce_tags, print_config_tree\nfrom matcha.utils.utils import extras, get_metric_value, task_wrapper\n" }, { "path": "third_party/Matcha-TTS/matcha/utils/audio.py", "content": "import numpy as np\nimport torch\nimport torch.utils.data\nfrom librosa.filters import mel as librosa_mel_fn\nfrom scipy.io.wavfile import read\n\nMAX_WAV_VALUE = 32768.0\n\n\ndef load_wav(full_path):\n sampling_rate, data = read(full_path)\n return data, sampling_rate\n\n\ndef dynamic_range_compression(x, C=1, clip_val=1e-5):\n return np.log(np.clip(x, a_min=clip_val, a_max=None) * C)\n\n\ndef dynamic_range_decompression(x, C=1):\n return np.exp(x) / C\n\n\ndef dynamic_range_compression_torch(x, C=1, clip_val=1e-5):\n return torch.log(torch.clamp(x, min=clip_val) * C)\n\n\ndef dynamic_range_decompression_torch(x, C=1):\n return torch.exp(x) / C\n\n\ndef spectral_normalize_torch(magnitudes):\n output = dynamic_range_compression_torch(magnitudes)\n return output\n\n\ndef spectral_de_normalize_torch(magnitudes):\n output = dynamic_range_decompression_torch(magnitudes)\n return output\n\n\nmel_basis = {}\nhann_window = {}\n\n\ndef mel_spectrogram(y, n_fft, num_mels, sampling_rate, hop_size, win_size, fmin, fmax, center=False):\n if torch.min(y) < -1.0:\n print(\"min value is \", torch.min(y))\n if torch.max(y) > 1.0:\n print(\"max value is \", torch.max(y))\n\n global mel_basis, hann_window # pylint: disable=global-statement\n if f\"{str(fmax)}_{str(y.device)}\" not in mel_basis:\n mel = librosa_mel_fn(sr=sampling_rate, n_fft=n_fft, n_mels=num_mels, fmin=fmin, fmax=fmax)\n mel_basis[str(fmax) + \"_\" + str(y.device)] = torch.from_numpy(mel).float().to(y.device)\n hann_window[str(y.device)] = torch.hann_window(win_size).to(y.device)\n\n y = torch.nn.functional.pad(\n y.unsqueeze(1), (int((n_fft - hop_size) / 2), int((n_fft - hop_size) / 2)), mode=\"reflect\"\n )\n y = y.squeeze(1)\n\n spec = torch.view_as_real(\n torch.stft(\n y,\n n_fft,\n hop_length=hop_size,\n win_length=win_size,\n window=hann_window[str(y.device)],\n center=center,\n pad_mode=\"reflect\",\n normalized=False,\n onesided=True,\n return_complex=True,\n )\n )\n\n spec = torch.sqrt(spec.pow(2).sum(-1) + (1e-9))\n\n spec = torch.matmul(mel_basis[str(fmax) + \"_\" + str(y.device)], spec)\n spec = spectral_normalize_torch(spec)\n\n return spec\n" }, { "path": "third_party/Matcha-TTS/matcha/utils/generate_data_statistics.py", "content": "r\"\"\"\nThe file creates a pickle file where the values needed for loading of dataset is stored and the model can load it\nwhen needed.\n\nParameters from hparam.py will be used\n\"\"\"\nimport argparse\nimport json\nimport os\nimport sys\nfrom pathlib import Path\n\nimport rootutils\nimport torch\nfrom hydra import compose, initialize\nfrom omegaconf import open_dict\nfrom tqdm.auto import tqdm\n\nfrom matcha.data.text_mel_datamodule import TextMelDataModule\nfrom matcha.utils.logging_utils import pylogger\n\nlog = pylogger.get_pylogger(__name__)\n\n\ndef compute_data_statistics(data_loader: torch.utils.data.DataLoader, out_channels: int):\n \"\"\"Generate data mean and standard deviation helpful in data normalisation\n\n Args:\n data_loader (torch.utils.data.Dataloader): _description_\n out_channels (int): mel spectrogram channels\n \"\"\"\n total_mel_sum = 0\n total_mel_sq_sum = 0\n total_mel_len = 0\n\n for batch in tqdm(data_loader, leave=False):\n mels = batch[\"y\"]\n mel_lengths = batch[\"y_lengths\"]\n\n total_mel_len += torch.sum(mel_lengths)\n total_mel_sum += torch.sum(mels)\n total_mel_sq_sum += torch.sum(torch.pow(mels, 2))\n\n data_mean = total_mel_sum / (total_mel_len * out_channels)\n data_std = torch.sqrt((total_mel_sq_sum / (total_mel_len * out_channels)) - torch.pow(data_mean, 2))\n\n return {\"mel_mean\": data_mean.item(), \"mel_std\": data_std.item()}\n\n\ndef main():\n parser = argparse.ArgumentParser()\n\n parser.add_argument(\n \"-i\",\n \"--input-config\",\n type=str,\n default=\"vctk.yaml\",\n help=\"The name of the yaml config file under configs/data\",\n )\n\n parser.add_argument(\n \"-b\",\n \"--batch-size\",\n type=int,\n default=\"256\",\n help=\"Can have increased batch size for faster computation\",\n )\n\n parser.add_argument(\n \"-f\",\n \"--force\",\n action=\"store_true\",\n default=False,\n required=False,\n help=\"force overwrite the file\",\n )\n args = parser.parse_args()\n output_file = Path(args.input_config).with_suffix(\".json\")\n\n if os.path.exists(output_file) and not args.force:\n print(\"File already exists. Use -f to force overwrite\")\n sys.exit(1)\n\n with initialize(version_base=\"1.3\", config_path=\"../../configs/data\"):\n cfg = compose(config_name=args.input_config, return_hydra_config=True, overrides=[])\n\n root_path = rootutils.find_root(search_from=__file__, indicator=\".project-root\")\n\n with open_dict(cfg):\n del cfg[\"hydra\"]\n del cfg[\"_target_\"]\n cfg[\"data_statistics\"] = None\n cfg[\"seed\"] = 1234\n cfg[\"batch_size\"] = args.batch_size\n cfg[\"train_filelist_path\"] = str(os.path.join(root_path, cfg[\"train_filelist_path\"]))\n cfg[\"valid_filelist_path\"] = str(os.path.join(root_path, cfg[\"valid_filelist_path\"]))\n\n text_mel_datamodule = TextMelDataModule(**cfg)\n text_mel_datamodule.setup()\n data_loader = text_mel_datamodule.train_dataloader()\n log.info(\"Dataloader loaded! Now computing stats...\")\n params = compute_data_statistics(data_loader, cfg[\"n_feats\"])\n print(params)\n json.dump(\n params,\n open(output_file, \"w\"),\n )\n\n\nif __name__ == \"__main__\":\n main()\n" }, { "path": "third_party/Matcha-TTS/matcha/utils/instantiators.py", "content": "from typing import List\n\nimport hydra\nfrom lightning import Callback\nfrom lightning.pytorch.loggers import Logger\nfrom omegaconf import DictConfig\n\nfrom matcha.utils import pylogger\n\nlog = pylogger.get_pylogger(__name__)\n\n\ndef instantiate_callbacks(callbacks_cfg: DictConfig) -> List[Callback]:\n \"\"\"Instantiates callbacks from config.\n\n :param callbacks_cfg: A DictConfig object containing callback configurations.\n :return: A list of instantiated callbacks.\n \"\"\"\n callbacks: List[Callback] = []\n\n if not callbacks_cfg:\n log.warning(\"No callback configs found! Skipping..\")\n return callbacks\n\n if not isinstance(callbacks_cfg, DictConfig):\n raise TypeError(\"Callbacks config must be a DictConfig!\")\n\n for _, cb_conf in callbacks_cfg.items():\n if isinstance(cb_conf, DictConfig) and \"_target_\" in cb_conf:\n log.info(f\"Instantiating callback <{cb_conf._target_}>\") # pylint: disable=protected-access\n callbacks.append(hydra.utils.instantiate(cb_conf))\n\n return callbacks\n\n\ndef instantiate_loggers(logger_cfg: DictConfig) -> List[Logger]:\n \"\"\"Instantiates loggers from config.\n\n :param logger_cfg: A DictConfig object containing logger configurations.\n :return: A list of instantiated loggers.\n \"\"\"\n logger: List[Logger] = []\n\n if not logger_cfg:\n log.warning(\"No logger configs found! Skipping...\")\n return logger\n\n if not isinstance(logger_cfg, DictConfig):\n raise TypeError(\"Logger config must be a DictConfig!\")\n\n for _, lg_conf in logger_cfg.items():\n if isinstance(lg_conf, DictConfig) and \"_target_\" in lg_conf:\n log.info(f\"Instantiating logger <{lg_conf._target_}>\") # pylint: disable=protected-access\n logger.append(hydra.utils.instantiate(lg_conf))\n\n return logger\n" }, { "path": "third_party/Matcha-TTS/matcha/utils/logging_utils.py", "content": "from typing import Any, Dict\n\nfrom lightning.pytorch.utilities import rank_zero_only\nfrom omegaconf import OmegaConf\n\nfrom matcha.utils import pylogger\n\nlog = pylogger.get_pylogger(__name__)\n\n\n@rank_zero_only\ndef log_hyperparameters(object_dict: Dict[str, Any]) -> None:\n \"\"\"Controls which config parts are saved by Lightning loggers.\n\n Additionally saves:\n - Number of model parameters\n\n :param object_dict: A dictionary containing the following objects:\n - `\"cfg\"`: A DictConfig object containing the main config.\n - `\"model\"`: The Lightning model.\n - `\"trainer\"`: The Lightning trainer.\n \"\"\"\n hparams = {}\n\n cfg = OmegaConf.to_container(object_dict[\"cfg\"])\n model = object_dict[\"model\"]\n trainer = object_dict[\"trainer\"]\n\n if not trainer.logger:\n log.warning(\"Logger not found! Skipping hyperparameter logging...\")\n return\n\n hparams[\"model\"] = cfg[\"model\"]\n\n # save number of model parameters\n hparams[\"model/params/total\"] = sum(p.numel() for p in model.parameters())\n hparams[\"model/params/trainable\"] = sum(p.numel() for p in model.parameters() if p.requires_grad)\n hparams[\"model/params/non_trainable\"] = sum(p.numel() for p in model.parameters() if not p.requires_grad)\n\n hparams[\"data\"] = cfg[\"data\"]\n hparams[\"trainer\"] = cfg[\"trainer\"]\n\n hparams[\"callbacks\"] = cfg.get(\"callbacks\")\n hparams[\"extras\"] = cfg.get(\"extras\")\n\n hparams[\"task_name\"] = cfg.get(\"task_name\")\n hparams[\"tags\"] = cfg.get(\"tags\")\n hparams[\"ckpt_path\"] = cfg.get(\"ckpt_path\")\n hparams[\"seed\"] = cfg.get(\"seed\")\n\n # send hparams to all loggers\n for logger in trainer.loggers:\n logger.log_hyperparams(hparams)\n" }, { "path": "third_party/Matcha-TTS/matcha/utils/model.py", "content": "\"\"\" from https://github.com/jaywalnut310/glow-tts \"\"\"\n\nimport numpy as np\nimport torch\n\n\ndef sequence_mask(length, max_length=None):\n if max_length is None:\n max_length = length.max()\n x = torch.arange(max_length, dtype=length.dtype, device=length.device)\n return x.unsqueeze(0) < length.unsqueeze(1)\n\n\ndef fix_len_compatibility(length, num_downsamplings_in_unet=2):\n factor = torch.scalar_tensor(2).pow(num_downsamplings_in_unet)\n length = (length / factor).ceil() * factor\n if not torch.onnx.is_in_onnx_export():\n return length.int().item()\n else:\n return length\n\n\ndef convert_pad_shape(pad_shape):\n inverted_shape = pad_shape[::-1]\n pad_shape = [item for sublist in inverted_shape for item in sublist]\n return pad_shape\n\n\ndef generate_path(duration, mask):\n device = duration.device\n\n b, t_x, t_y = mask.shape\n cum_duration = torch.cumsum(duration, 1)\n path = torch.zeros(b, t_x, t_y, dtype=mask.dtype).to(device=device)\n\n cum_duration_flat = cum_duration.view(b * t_x)\n path = sequence_mask(cum_duration_flat, t_y).to(mask.dtype)\n path = path.view(b, t_x, t_y)\n path = path - torch.nn.functional.pad(path, convert_pad_shape([[0, 0], [1, 0], [0, 0]]))[:, :-1]\n path = path * mask\n return path\n\n\ndef duration_loss(logw, logw_, lengths):\n loss = torch.sum((logw - logw_) ** 2) / torch.sum(lengths)\n return loss\n\n\ndef normalize(data, mu, std):\n if not isinstance(mu, (float, int)):\n if isinstance(mu, list):\n mu = torch.tensor(mu, dtype=data.dtype, device=data.device)\n elif isinstance(mu, torch.Tensor):\n mu = mu.to(data.device)\n elif isinstance(mu, np.ndarray):\n mu = torch.from_numpy(mu).to(data.device)\n mu = mu.unsqueeze(-1)\n\n if not isinstance(std, (float, int)):\n if isinstance(std, list):\n std = torch.tensor(std, dtype=data.dtype, device=data.device)\n elif isinstance(std, torch.Tensor):\n std = std.to(data.device)\n elif isinstance(std, np.ndarray):\n std = torch.from_numpy(std).to(data.device)\n std = std.unsqueeze(-1)\n\n return (data - mu) / std\n\n\ndef denormalize(data, mu, std):\n if not isinstance(mu, float):\n if isinstance(mu, list):\n mu = torch.tensor(mu, dtype=data.dtype, device=data.device)\n elif isinstance(mu, torch.Tensor):\n mu = mu.to(data.device)\n elif isinstance(mu, np.ndarray):\n mu = torch.from_numpy(mu).to(data.device)\n mu = mu.unsqueeze(-1)\n\n if not isinstance(std, float):\n if isinstance(std, list):\n std = torch.tensor(std, dtype=data.dtype, device=data.device)\n elif isinstance(std, torch.Tensor):\n std = std.to(data.device)\n elif isinstance(std, np.ndarray):\n std = torch.from_numpy(std).to(data.device)\n std = std.unsqueeze(-1)\n\n return data * std + mu\n" }, { "path": "third_party/Matcha-TTS/matcha/utils/monotonic_align/__init__.py", "content": "import numpy as np\nimport torch\n\nfrom matcha.utils.monotonic_align.core import maximum_path_c\n\n\ndef maximum_path(value, mask):\n \"\"\"Cython optimised version.\n value: [b, t_x, t_y]\n mask: [b, t_x, t_y]\n \"\"\"\n value = value * mask\n device = value.device\n dtype = value.dtype\n value = value.data.cpu().numpy().astype(np.float32)\n path = np.zeros_like(value).astype(np.int32)\n mask = mask.data.cpu().numpy()\n\n t_x_max = mask.sum(1)[:, 0].astype(np.int32)\n t_y_max = mask.sum(2)[:, 0].astype(np.int32)\n maximum_path_c(path, value, t_x_max, t_y_max)\n return torch.from_numpy(path).to(device=device, dtype=dtype)\n" }, { "path": "third_party/Matcha-TTS/matcha/utils/monotonic_align/core.c", "content": "/* Generated by Cython 0.29.35 */\n\n/* BEGIN: Cython Metadata\n{\n \"distutils\": {\n \"depends\": [],\n \"name\": \"matcha.utils.monotonic_align.core\",\n \"sources\": [\n \"matcha/utils/monotonic_align/core.pyx\"\n ]\n },\n \"module_name\": \"matcha.utils.monotonic_align.core\"\n}\nEND: Cython Metadata */\n\n#ifndef PY_SSIZE_T_CLEAN\n#define PY_SSIZE_T_CLEAN\n#endif /* PY_SSIZE_T_CLEAN */\n#include \"Python.h\"\n#ifndef Py_PYTHON_H\n #error Python headers needed to compile C extensions, please install development version of Python.\n#elif PY_VERSION_HEX < 0x02060000 || (0x03000000 <= PY_VERSION_HEX && PY_VERSION_HEX < 0x03030000)\n #error Cython requires Python 2.6+ or Python 3.3+.\n#else\n#define CYTHON_ABI \"0_29_35\"\n#define CYTHON_HEX_VERSION 0x001D23F0\n#define CYTHON_FUTURE_DIVISION 1\n#include