[ { "path": ".github/ISSUE_TEMPLATE/ask_questions.md", "content": "---\nname: ❓ Questions/Help\nabout: If you have questions, please first search existing issues and docs\nlabels: 'question, needs triage'\n---\n\nNotice: In order to resolve issues more efficiently, please raise issue following the template.\n（注意：为了更加高效率解决您遇到的问题，请按照模板提问，补充细节）\n\n## ❓ Questions and Help\n\n\n### Before asking:\n1. search the issues.\n2. search the docs.\n\n\n\n#### What is your question?\n\n#### Code\n\n\n\n#### What have you tried?\n\n#### What's your environment?\n\n - OS (e.g., Linux):\n - FunASR Version (e.g., 1.0.0):\n - ModelScope Version (e.g., 1.11.0):\n - PyTorch Version (e.g., 2.0.0):\n - How you installed funasr (`pip`, source):\n - Python version:\n - GPU (e.g., V100M32)\n - CUDA/cuDNN version (e.g., cuda11.7):\n - Docker version (e.g., funasr-runtime-sdk-cpu-0.4.1)\n - Any other relevant information:" }, { "path": ".github/ISSUE_TEMPLATE/bug_report.md", "content": "---\nname: 🐛 Bug Report\nabout: Submit a bug report to help us improve\nlabels: 'bug, needs triage'\n---\n\nNotice: In order to resolve issues more efficiently, please raise issue following the template.\n（注意：为了更加高效率解决您遇到的问题，请按照模板提问，补充细节）\n\n## 🐛 Bug\n\n\n\n### To Reproduce\n\nSteps to reproduce the behavior (**always include the command you ran**):\n\n1. Run cmd '....'\n2. See error\n\n\n\n\n#### Code sample\n\n\n### Expected behavior\n\n\n\n### Environment\n\n - OS (e.g., Linux):\n - FunASR Version (e.g., 1.0.0):\n - ModelScope Version (e.g., 1.11.0):\n - PyTorch Version (e.g., 2.0.0):\n - How you installed funasr (`pip`, source):\n - Python version:\n - GPU (e.g., V100M32)\n - CUDA/cuDNN version (e.g., cuda11.7):\n - Docker version (e.g., funasr-runtime-sdk-cpu-0.4.1)\n - Any other relevant information:\n\n### Additional context\n\n" }, { "path": ".github/ISSUE_TEMPLATE/config.yaml", "content": "blank_issues_enabled: false" }, { "path": ".github/ISSUE_TEMPLATE/error_docs.md", "content": "---\nname: 📚 Documentation/Typos\nabout: Report an issue related to documentation or a typo\nlabels: 'documentation, needs triage'\n---\n\n## 📚 Documentation\n\nFor typos and doc fixes, please go ahead and:\n\n1. Create an issue.\n2. Fix the typo.\n3. Submit a PR.\n\nThanks!" }, { "path": "Dockerfile", "content": "# ======================================================\n# FunASR SenseVoiceSmall Inference Server\n# ======================================================\nFROM pytorch/pytorch:2.3.1-cuda12.1-cudnn8-runtime\n\n# Install system dependencies\nRUN apt-get update && apt-get install -y \\\n\tffmpeg libsndfile1 git && \\\n\trm -rf /var/lib/apt/lists/*\n\nWORKDIR /app\nRUN ls -la /app\n# Copy only requirements first\nCOPY requirements.txt /app/\n\n# Install dependencies (cached if requirements.txt didn't change)\nRUN pip install --no-cache-dir -r requirements.txt\n\n# Now copy the rest of your code\nCOPY . /app\n\n\n# Optional: preload model weights during build (saves runtime download)\n# RUN python -c \"from funasr import AutoModel; AutoModel(model='iic/SenseVoiceSmall')\"\n\n# Expose FastAPI port\nEXPOSE 50000\n\n# Environment variables\nENV SENSEVOICE_DEVICE=auto\nENV PYTHONUNBUFFERED=1\nENV MODELSCOPE_CACHE=/models\n\n# Create model cache directory (helps reuse between restarts)\nRUN mkdir -p /models\n\n# Start FastAPI app\nCMD [\"uvicorn\", \"api:app\", \"--host\", \"0.0.0.0\", \"--port\", \"50000\"]\n" }, { "path": "LICENSE", "content": "Ref to https://github.com/modelscope/FunASR?tab=readme-ov-file#license\n" }, { "path": "README.md", "content": "([简体中文](./README_zh.md)|English|[日本語](./README_ja.md))\n\n\n# Introduction\n\nSenseVoice is a speech foundation model with multiple speech understanding capabilities, including automatic speech recognition (ASR), spoken language identification (LID), speech emotion recognition (SER), and audio event detection (AED). \n\n
\n\n
\n\n[//]: # (
)\n\n
\n

\n Homepage \n｜ What's News \n｜ Benchmarks \n｜ Install \n｜ Usage \n｜ Community \n

\n\nModel Zoo:\n[modelscope](https://www.modelscope.cn/models/iic/SenseVoiceSmall), [huggingface](https://huggingface.co/FunAudioLLM/SenseVoiceSmall)\n\nOnline Demo:\n[modelscope demo](https://www.modelscope.cn/studios/iic/SenseVoice), [huggingface space](https://huggingface.co/spaces/FunAudioLLM/SenseVoice)\n\n\n
\n\n\n\n# Highlights 🎯\n**SenseVoice** focuses on high-accuracy multilingual speech recognition, speech emotion recognition, and audio event detection.\n- **Multilingual Speech Recognition:** Trained with over 400,000 hours of data, supporting more than 50 languages, the recognition performance surpasses that of the Whisper model.\n- **Rich transcribe:** \n - Possess excellent emotion recognition capabilities, achieving and surpassing the effectiveness of the current best emotion recognition models on test data.\n - Offer sound event detection capabilities, supporting the detection of various common human-computer interaction events such as bgm, applause, laughter, crying, coughing, and sneezing.\n- **Efficient Inference:** The SenseVoice-Small model utilizes a non-autoregressive end-to-end framework, leading to exceptionally low inference latency. It requires only 70ms to process 10 seconds of audio, which is 15 times faster than Whisper-Large.\n- **Convenient Finetuning:** Provide convenient finetuning scripts and strategies, allowing users to easily address long-tail sample issues according to their business scenarios.\n- **Service Deployment:** Offer service deployment pipeline, supporting multi-concurrent requests, with client-side languages including Python, C++, HTML, Java, and C#, among others.\n\n\n# What's New 🔥\n- 2024/11: Add support for timestamp based on the CTC alignment.\n- 2024/7: Added Export Features for [ONNX](./demo_onnx.py) and [libtorch](./demo_libtorch.py), as well as Python Version Runtimes: [funasr-onnx-0.4.0](https://pypi.org/project/funasr-onnx/), [funasr-torch-0.1.1](https://pypi.org/project/funasr-torch/)\n- 2024/7: The [SenseVoice-Small](https://www.modelscope.cn/models/iic/SenseVoiceSmall) voice understanding model is open-sourced, which offers high-precision multilingual speech recognition, emotion recognition, and audio event detection capabilities for Mandarin, Cantonese, English, Japanese, and Korean and leads to exceptionally low inference latency. \n- 2024/7: The CosyVoice for natural speech generation with multi-language, timbre, and emotion control. CosyVoice excels in multi-lingual voice generation, zero-shot voice generation, cross-lingual voice cloning, and instruction-following capabilities. [CosyVoice repo](https://github.com/FunAudioLLM/CosyVoice) and [CosyVoice space](https://www.modelscope.cn/studios/iic/CosyVoice-300M).\n- 2024/7: [FunASR](https://github.com/modelscope/FunASR) is a fundamental speech recognition toolkit that offers a variety of features, including speech recognition (ASR), Voice Activity Detection (VAD), Punctuation Restoration, Language Models, Speaker Verification, Speaker Diarization and multi-talker ASR.\n\n\n# Benchmarks 📝\n\n## Multilingual Speech Recognition\nWe compared the performance of multilingual speech recognition between SenseVoice and Whisper on open-source benchmark datasets, including AISHELL-1, AISHELL-2, Wenetspeech, LibriSpeech, and Common Voice. In terms of Chinese and Cantonese recognition, the SenseVoice-Small model has advantages.\n\n
\n\n
\n\n## Speech Emotion Recognition\n\nDue to the current lack of widely-used benchmarks and methods for speech emotion recognition, we conducted evaluations across various metrics on multiple test sets and performed a comprehensive comparison with numerous results from recent benchmarks. The selected test sets encompass data in both Chinese and English, and include multiple styles such as performances, films, and natural conversations. Without finetuning on the target data, SenseVoice was able to achieve and exceed the performance of the current best speech emotion recognition models.\n\n
\n\n
\n\nFurthermore, we compared multiple open-source speech emotion recognition models on the test sets, and the results indicate that the SenseVoice-Large model achieved the best performance on nearly all datasets, while the SenseVoice-Small model also surpassed other open-source models on the majority of the datasets.\n\n
\n\n
\n\n## Audio Event Detection\n\nAlthough trained exclusively on speech data, SenseVoice can still function as a standalone event detection model. We compared its performance on the environmental sound classification ESC-50 dataset against the widely used industry models BEATS and PANN. The SenseVoice model achieved commendable results on these tasks. However, due to limitations in training data and methodology, its event classification performance has some gaps compared to specialized AED models.\n\n
\n\n
\n\n## Computational Efficiency\n\nThe SenseVoice-Small model deploys a non-autoregressive end-to-end architecture, resulting in extremely low inference latency. With a similar number of parameters to the Whisper-Small model, it infers more than 5 times faster than Whisper-Small and 15 times faster than Whisper-Large. \n\n
\n\n
\n\n\n# Requirements\n\n```shell\npip install -r requirements.txt\n```\n\n\n# Usage\n\n## Inference\n\nSupports input of audio in any format and of any duration.\n\n```python\nfrom funasr import AutoModel\nfrom funasr.utils.postprocess_utils import rich_transcription_postprocess\n\nmodel_dir = \"iic/SenseVoiceSmall\"\n\n\nmodel = AutoModel(\n model=model_dir,\n trust_remote_code=True,\n remote_code=\"./model.py\", \n vad_model=\"fsmn-vad\",\n vad_kwargs={\"max_single_segment_time\": 30000},\n device=\"cuda:0\",\n)\n\n# en\nres = model.generate(\n input=f\"{model.model_path}/example/en.mp3\",\n cache={},\n language=\"auto\", # \"zh\", \"en\", \"yue\", \"ja\", \"ko\", \"nospeech\"\n use_itn=True,\n batch_size_s=60,\n merge_vad=True, #\n merge_length_s=15,\n)\ntext = rich_transcription_postprocess(res[0][\"text\"])\nprint(text)\n```\n\n
Parameter Description (Click to Expand)\n\n- `model_dir`: The name of the model, or the path to the model on the local disk.\n- `trust_remote_code`:\n - When `True`, it means that the model's code implementation is loaded from `remote_code`, which specifies the exact location of the `model` code (for example, `model.py` in the current directory). It supports absolute paths, relative paths, and network URLs.\n - When `False`, it indicates that the model's code implementation is the integrated version within [FunASR](https://github.com/modelscope/FunASR). At this time, modifications made to `model.py` in the current directory will not be effective, as the version loaded is the internal one from FunASR. For the model code, [click here to view](https://github.com/modelscope/FunASR/tree/main/funasr/models/sense_voice).\n- `vad_model`: This indicates the activation of VAD (Voice Activity Detection). The purpose of VAD is to split long audio into shorter clips. In this case, the inference time includes both VAD and SenseVoice total consumption, and represents the end-to-end latency. If you wish to test the SenseVoice model's inference time separately, the VAD model can be disabled.\n- `vad_kwargs`: Specifies the configurations for the VAD model. `max_single_segment_time`: denotes the maximum duration for audio segmentation by the `vad_model`, with the unit being milliseconds (ms).\n- `use_itn`: Whether the output result includes punctuation and inverse text normalization.\n- `batch_size_s`: Indicates the use of dynamic batching, where the total duration of audio in the batch is measured in seconds (s).\n- `merge_vad`: Whether to merge short audio fragments segmented by the VAD model, with the merged length being `merge_length_s`, in seconds (s).\n- `ban_emo_unk`: Whether to ban the output of the `emo_unk` token.\n
\n\nIf all inputs are short audios (<30s), and batch inference is needed to speed up inference efficiency, the VAD model can be removed, and `batch_size` can be set accordingly.\n```python\nmodel = AutoModel(model=model_dir, trust_remote_code=True, device=\"cuda:0\")\n\nres = model.generate(\n input=f\"{model.model_path}/example/en.mp3\",\n cache={},\n language=\"zh\", # \"zh\", \"en\", \"yue\", \"ja\", \"ko\", \"nospeech\"\n use_itn=False,\n batch_size=64, \n)\n```\n\nFor more usage, please refer to [docs](https://github.com/modelscope/FunASR/blob/main/docs/tutorial/README.md)\n\n### Inference directly\n\nSupports input of audio in any format, with an input duration limit of 30 seconds or less.\n\n```python\nfrom model import SenseVoiceSmall\nfrom funasr.utils.postprocess_utils import rich_transcription_postprocess\n\nmodel_dir = \"iic/SenseVoiceSmall\"\nm, kwargs = SenseVoiceSmall.from_pretrained(model=model_dir, device=\"cuda:0\")\nm.eval()\n\nres = m.inference(\n data_in=f\"{kwargs['model_path']}/example/en.mp3\",\n language=\"auto\", # \"zh\", \"en\", \"yue\", \"ja\", \"ko\", \"nospeech\"\n use_itn=False,\n ban_emo_unk=False,\n **kwargs,\n)\n\ntext = rich_transcription_postprocess(res[0][0][\"text\"])\nprint(text)\n```\n\n### Export and Test\n
ONNX and Libtorch Export\n\n#### ONNX\n```python\n# pip3 install -U funasr funasr-onnx\nfrom pathlib import Path\nfrom funasr_onnx import SenseVoiceSmall\nfrom funasr_onnx.utils.postprocess_utils import rich_transcription_postprocess\n\n\nmodel_dir = \"iic/SenseVoiceSmall\"\n\nmodel = SenseVoiceSmall(model_dir, batch_size=10, quantize=True)\n\n# inference\nwav_or_scp = [\"{}/.cache/modelscope/hub/{}/example/en.mp3\".format(Path.home(), model_dir)]\n\nres = model(wav_or_scp, language=\"auto\", use_itn=True)\nprint([rich_transcription_postprocess(i) for i in res])\n```\nNote: ONNX model is exported to the original model directory.\n\n#### Libtorch\n```python\nfrom pathlib import Path\nfrom funasr_torch import SenseVoiceSmall\nfrom funasr_torch.utils.postprocess_utils import rich_transcription_postprocess\n\n\nmodel_dir = \"iic/SenseVoiceSmall\"\n\nmodel = SenseVoiceSmall(model_dir, batch_size=10, device=\"cuda:0\")\n\nwav_or_scp = [\"{}/.cache/modelscope/hub/{}/example/en.mp3\".format(Path.home(), model_dir)]\n\nres = model(wav_or_scp, language=\"auto\", use_itn=True)\nprint([rich_transcription_postprocess(i) for i in res])\n```\nNote: Libtorch model is exported to the original model directory.\n
\n\n## Service\n\n### Deployment with FastAPI\n```shell\nexport SENSEVOICE_DEVICE=cuda:0\nfastapi run --port 50000\n```\n\n## Finetune\n\n### Requirements\n\n```shell\ngit clone https://github.com/alibaba/FunASR.git && cd FunASR\npip3 install -e ./\n```\n## 🐳 Docker Support\n\nSenseVoice can be built and run using Docker to simplify setup, ensure reproducibility, and support both CPU and GPU inference.\n\n### Build with Docker\n```bash\ndocker build -t sensevoice .\n```\n\n### Run (GPU – default)\n```bash\ndocker run --gpus all -p 50000:50000 sensevoice\n```\n### Run (CPU-only)\n```bash\ndocker run -e SENSEVOICE_DEVICE=cpu -p 50000:50000 sensevoice\n```\n### Docker Compose\nDocker Compose provides an easier way to run SenseVoice with persistent model caching, networking etc. \n\n### Start Stack\n```bash\ndocker compose up --build\n```\n### Data prepare\n\nData examples\n\n```text\n{\"key\": \"YOU0000008470_S0000238_punc_itn\", \"text_language\": \"<|en|>\", \"emo_target\": \"<|NEUTRAL|>\", \"event_target\": \"<|Speech|>\", \"with_or_wo_itn\": \"<|withitn|>\", \"target\": \"Including legal due diligence, subscription agreement, negotiation.\", \"source\": \"/cpfs01/shared/Group-speech/beinian.lzr/data/industrial_data/english_all/audio/YOU0000008470_S0000238.wav\", \"target_len\": 7, \"source_len\": 140}\n{\"key\": \"AUD0000001556_S0007580\", \"text_language\": \"<|en|>\", \"emo_target\": \"<|NEUTRAL|>\", \"event_target\": \"<|Speech|>\", \"with_or_wo_itn\": \"<|woitn|>\", \"target\": \"there is a tendency to identify the self or take interest in what one has got used to\", \"source\": \"/cpfs01/shared/Group-speech/beinian.lzr/data/industrial_data/english_all/audio/AUD0000001556_S0007580.wav\", \"target_len\": 18, \"source_len\": 360}\n```\n\nFull ref to `data/train_example.jsonl`\n\n
Data Prepare Details\n\nDescription：\n- `key`: audio file unique ID\n- `source`：path to the audio file\n- `source_len`：number of fbank frames of the audio file\n- `target`：transcription\n- `target_len`：length of target\n- `text_language`：language id of the audio file\n- `emo_target`：emotion label of the audio file\n- `event_target`：event label of the audio file\n- `with_or_wo_itn`：whether includes punctuation and inverse text normalization\n\n\n`train_text.txt`\n\n\n```bash\nBAC009S0764W0121 甚至出现交易几乎停滞的情况\nBAC009S0916W0489 湖北一公司以员工名义贷款数十员工负债千万\nasr_example_cn_en 所有只要处理 data 不管你是做 machine learning 做 deep learning 做 data analytics 做 data science 也好 scientist 也好通通都要都做的基本功啊那 again 先先对有一些>也许对\nID0012W0014 he tried to think how it could be\n```\n\n`train_wav.scp`\n\n\n\n```bash\nBAC009S0764W0121 https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/BAC009S0764W0121.wav\nBAC009S0916W0489 https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/BAC009S0916W0489.wav\nasr_example_cn_en https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/asr_example_cn_en.wav\nID0012W0014 https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/asr_example_en.wav\n```\n\n`train_text_language.txt`\n\nThe language ids include `<|zh|>`、`<|en|>`、`<|yue|>`、`<|ja|>` and `<|ko|>`.\n\n```bash\nBAC009S0764W0121 <|zh|>\nBAC009S0916W0489 <|zh|>\nasr_example_cn_en <|zh|>\nID0012W0014 <|en|>\n```\n\n`train_emo.txt`\n\nThe emotion labels include`<|HAPPY|>`、`<|SAD|>`、`<|ANGRY|>`、`<|NEUTRAL|>`、`<|FEARFUL|>`、`<|DISGUSTED|>` and `<|SURPRISED|>`.\n\n```bash\nBAC009S0764W0121 <|NEUTRAL|>\nBAC009S0916W0489 <|NEUTRAL|>\nasr_example_cn_en <|NEUTRAL|>\nID0012W0014 <|NEUTRAL|>\n```\n\n`train_event.txt`\n\nThe event labels include`<|BGM|>`、`<|Speech|>`、`<|Applause|>`、`<|Laughter|>`、`<|Cry|>`、`<|Sneeze|>`、`<|Breath|>` and `<|Cough|>`.\n\n```bash\nBAC009S0764W0121 <|Speech|>\nBAC009S0916W0489 <|Speech|>\nasr_example_cn_en <|Speech|>\nID0012W0014 <|Speech|>\n```\n\n`Command`\n```shell\n# generate train.jsonl and val.jsonl from wav.scp, text.txt, text_language.txt, emo_target.txt, event_target.txt\nsensevoice2jsonl \\\n++scp_file_list='[\"../../../data/list/train_wav.scp\", \"../../../data/list/train_text.txt\", \"../../../data/list/train_text_language.txt\", \"../../../data/list/train_emo.txt\", \"../../../data/list/train_event.txt\"]' \\\n++data_type_list='[\"source\", \"target\", \"text_language\", \"emo_target\", \"event_target\"]' \\\n++jsonl_file_out=\"../../../data/list/train.jsonl\"\n```\n\nIf there is no `train_text_language.txt`, `train_emo_target.txt` and `train_event_target.txt`, the language, emotion and event label will be predicted automatically by using the `SenseVoice` model.\n```shell\n# generate train.jsonl and val.jsonl from wav.scp and text.txt\nsensevoice2jsonl \\\n++scp_file_list='[\"../../../data/list/train_wav.scp\", \"../../../data/list/train_text.txt\"]' \\\n++data_type_list='[\"source\", \"target\"]' \\\n++jsonl_file_out=\"../../../data/list/train.jsonl\" \\\n++model_dir='iic/SenseVoiceSmall'\n```\n
\n\n### Finetune\n\nEnsure to modify the train_tool in finetune.sh to the absolute path of `funasr/bin/train_ds.py` from the FunASR installation directory you have set up earlier.\n\n```shell\nbash finetune.sh\n```\n\n## WebUI\n\n```shell\npython webui.py\n```\n\n
\n\n\n## Remarkable Third-Party Work\n- Triton (GPU) Deployment Best Practices: Using Triton + TensorRT, tested with FP32, achieving an acceleration ratio of 526 on V100 GPU. FP16 support is in progress. [Repository](https://github.com/modelscope/FunASR/blob/main/runtime/triton_gpu/README.md)\n- Sherpa-onnx Deployment Best Practices: Supports using SenseVoice in 10 programming languages: C++, C, Python, C#, Go, Swift, Kotlin, Java, JavaScript, and Dart. Also supports deploying SenseVoice on platforms like iOS, Android, and Raspberry Pi. [Repository](https://k2-fsa.github.io/sherpa/onnx/sense-voice/index.html)\n- [SenseVoice.cpp](https://github.com/lovemefan/SenseVoice.cpp). Inference of SenseVoice in pure C/C++ based on GGML, supporting 3-bit, 4-bit, 5-bit, 8-bit quantization, etc. with no third-party dependencies.\n- [streaming-sensevoice](https://github.com/pengzhendong/streaming-sensevoice) processes inference in chunks. To achieve pseudo-streaming, it employs a truncated attention mechanism, sacrificing some accuracy. Additionally, this technology supports CTC prefix beam search and hot-word boosting features.\n- [OmniSenseVoice](https://github.com/lifeiteng/OmniSenseVoice) is optimized for lightning-fast inference and batching process. \n- [SenseVoice Hotword](https://www.modelscope.cn/models/dengcunqin/SenseVoiceSmall_hotword)，Neural Network Hotword Enhancement，[Contextualized End-to-End Speech Recognition with Contextual Phrase Prediction Network](https://mp.weixin.qq.com/s/1QkIvh8j7rrUjRyWOgAvdA)。\n\n# Community\nIf you encounter problems in use, you can directly raise Issues on the github page.\n\nYou can also scan the following DingTalk group QR code to join the community group for communication and discussion.\n\n| FunASR |\n|:--------------------------------------------------------:|\n| |\n\n\n" }, { "path": "README_ja.md", "content": "# SenseVoice\n\n「[简体中文](./README_zh.md)」|「[English](./README.md)」|「日本語」\n\nSenseVoiceは、音声認識（ASR）、言語識別（LID）、音声感情認識（SER）、および音響イベント分類（AEC）または音響イベント検出（AED）を含む音声理解能力を備えた音声基盤モデルです。本プロジェクトでは、SenseVoiceモデルの紹介と、複数のタスクテストセットでのベンチマーク、およびモデルの体験に必要な環境のインストールと推論方法を提供します。\n\n
\n\n
\n[//]: # (
)\n \n
\n

\n ホームページ \n｜ 最新情報 \n｜ 性能評価 \n｜ 環境インストール \n｜ 使用方法チュートリアル \n｜ お問い合わせ \n

\n\nモデルリポジトリ：[modelscope](https://www.modelscope.cn/models/iic/SenseVoiceSmall)，[huggingface](https://huggingface.co/FunAudioLLM/SenseVoiceSmall)\n\nオンライン体験：\n[modelscope demo](https://www.modelscope.cn/studios/iic/SenseVoice), [huggingface space](https://huggingface.co/spaces/FunAudioLLM/SenseVoice)\n\n
\n\n\n# コア機能 🎯\n**SenseVoice**は、高精度な多言語音声認識、感情認識、および音声イベント検出に焦点を当てています。\n- **多言語認識：** 40万時間以上のデータを使用してトレーニングされ、50以上の言語をサポートし、認識性能はWhisperモデルを上回ります。\n- **リッチテキスト認識：** \n - 優れた感情認識能力を持ち、テストデータで現在の最良の感情認識モデルの効果を達成および上回ります。\n - 音声イベント検出能力を提供し、音楽、拍手、笑い声、泣き声、咳、くしゃみなどのさまざまな一般的な人間とコンピュータのインタラクションイベントを検出します。\n- **効率的な推論：** SenseVoice-Smallモデルは非自己回帰エンドツーエンドフレームワークを採用しており、推論遅延が非常に低く、10秒の音声の推論に70msしかかかりません。Whisper-Largeより15倍高速です。\n- **簡単な微調整：** 便利な微調整スクリプトと戦略を提供し、ユーザーがビジネスシナリオに応じてロングテールサンプルの問題を簡単に解決できるようにします。\n- **サービス展開：** マルチコンカレントリクエストをサポートする完全なサービス展開パイプラインを提供し、クライアントサイドの言語にはPython、C++、HTML、Java、C#などがあります。\n\n\n# 最新情報 🔥\n- 2024/7：新しく[ONNX](./demo_onnx.py)と[libtorch](./demo_libtorch.py)のエクスポート機能を追加し、Pythonバージョンのランタイム：[funasr-onnx-0.4.0](https://pypi.org/project/funasr-onnx/)、[funasr-torch-0.1.1](https://pypi.org/project/funasr-torch/)も提供開始。\n- 2024/7: [SenseVoice-Small](https://www.modelscope.cn/models/iic/SenseVoiceSmall) 多言語音声理解モデルがオープンソース化されました。中国語、広東語、英語、日本語、韓国語の多言語音声認識、感情認識、およびイベント検出能力をサポートし、非常に低い推論遅延を実現しています。\n- 2024/7: CosyVoiceは自然な音声生成に取り組んでおり、多言語、音色、感情制御をサポートします。多言語音声生成、ゼロショット音声生成、クロスランゲージ音声クローン、および指示に従う能力に優れています。[CosyVoice repo](https://github.com/FunAudioLLM/CosyVoice) and [CosyVoice オンライン体験](https://www.modelscope.cn/studios/iic/CosyVoice-300M).\n- 2024/7: [FunASR](https://github.com/modelscope/FunASR) は、音声認識（ASR）、音声活動検出（VAD）、句読点復元、言語モデル、話者検証、話者分離、およびマルチトーカーASRなどの機能を提供する基本的な音声認識ツールキットです。\n\n\n# ベンチマーク 📝\n\n## 多言語音声認識\n\nオープンソースのベンチマークデータセット（AISHELL-1、AISHELL-2、Wenetspeech、Librispeech、Common Voiceを含む）でSenseVoiceとWhisperの多言語音声認識性能と推論効率を比較しました。中国語と広東語の認識効果において、SenseVoice-Smallモデルは明らかな効果の優位性を持っています。\n\n
\n\n
\n\n## 感情認識\n\n現在、広く使用されている感情認識のテスト指標と方法が不足しているため、複数のテストセットでさまざまな指標をテストし、最近のベンチマークの複数の結果と包括的に比較しました。選択されたテストセットには、中国語/英語の両方の言語と、パフォーマンス、映画、自然な会話などのさまざまなスタイルのデータが含まれています。ターゲットデータの微調整を行わない前提で、SenseVoiceはテストデータで現在の最良の感情認識モデルの効果を達成および上回ることができました。\n\n
\n\n
\n\nさらに、テストセットで複数のオープンソースの感情認識モデルを比較し、結果はSenseVoice-Largeモデルがほぼすべてのデータで最良の効果を達成し、SenseVoice-Smallモデルも多数のデータセットで他のオープンソースモデルを上回る効果を達成したことを示しています。\n\n
\n\n
\n\n## イベント検出\n\nSenseVoiceは音声データのみでトレーニングされていますが、イベント検出モデルとして単独で使用することもできます。環境音分類ESC-50データセットで、現在業界で広く使用されているBEATSおよびPANNモデルの効果と比較しました。SenseVoiceモデルはこれらのタスクで良好な効果を達成しましたが、トレーニングデータとトレーニング方法の制約により、イベント分類の効果は専門のイベント検出モデルと比較してまだ一定の差があります。\n\n
\n\n
\n\n## 推論効率\n\nSenseVoice-smallモデルは非自己回帰エンドツーエンドアーキテクチャを採用しており、推論遅延が非常に低いです。Whisper-Smallモデルと同等のパラメータ量で、Whisper-Smallモデルより5倍高速で、Whisper-Largeモデルより15倍高速です。同時に、SenseVoice-smallモデルは音声の長さが増加しても、推論時間に明らかな増加はありません。\n\n
\n\n
\n\n\n# 環境インストール 🐍\n\n```shell\npip install -r requirements.txt\n```\n\n\n# 使用方法 🛠️\n\n## 推論\n\n任意の形式の音声入力をサポートし、任意の長さの入力をサポートします。\n\n```python\nfrom funasr import AutoModel\nfrom funasr.utils.postprocess_utils import rich_transcription_postprocess\n\nmodel_dir = \"iic/SenseVoiceSmall\"\n\n\nmodel = AutoModel(\n model=model_dir,\n trust_remote_code=True,\n remote_code=\"./model.py\", \n vad_model=\"fsmn-vad\",\n vad_kwargs={\"max_single_segment_time\": 30000},\n device=\"cuda:0\",\n)\n\n# en\nres = model.generate(\n input=f\"{model.model_path}/example/en.mp3\",\n cache={},\n language=\"auto\", # \"zh\", \"en\", \"yue\", \"ja\", \"ko\", \"nospeech\"\n use_itn=True,\n batch_size_s=60,\n merge_vad=True, #\n merge_length_s=15,\n)\ntext = rich_transcription_postprocess(res[0][\"text\"])\nprint(text)\n```\n\n
パラメータの説明（クリックして展開）\n\n- `model_dir`：モデル名、またはローカルディスク上のモデルパス。\n- `trust_remote_code`：\n - `True`は、modelコードの実装が`remote_code`からロードされることを意味し、`remote_code`は`model`コードの正確な位置を指定します（例：現在のディレクトリの`model.py`）。絶対パス、相対パス、およびネットワークURLをサポートします。\n - `False`は、modelコードの実装が[FunASR](https://github.com/modelscope/FunASR)内部に統合されたバージョンであることを意味し、この場合、現在のディレクトリの`model.py`を変更しても効果がありません。FunASR内部バージョンがロードされるためです。モデルコード[こちらを参照](https://github.com/modelscope/FunASR/tree/main/funasr/models/sense_voice)。\n- `vad_model`：VAD（音声活動検出）を有効にすることを示します。VADの目的は、長い音声を短いクリップに分割することです。この場合、推論時間にはVADとSenseVoiceの合計消費が含まれ、エンドツーエンドの遅延を表します。SenseVoiceモデルの推論時間を個別にテストする場合は、VADモデルを無効にできます。\n- `vad_kwargs`：VADモデルの設定を指定します。`max_single_segment_time`：`vad_model`による音声セグメントの最大長を示し、単位はミリ秒（ms）です。\n- `use_itn`：出力結果に句読点と逆テキスト正規化が含まれるかどうか。\n- `batch_size_s`：動的バッチの使用を示し、バッチ内の音声の合計長を秒（s）で測定します。\n- `merge_vad`：VADモデルによって分割された短い音声フラグメントをマージするかどうか。マージ後の長さは`merge_length_s`で、単位は秒（s）です。\n- `ban_emo_unk`：emo_unkラベルを無効にする。\n
\n\nすべての入力が短い音声（30秒未満）であり、バッチ推論が必要な場合、推論効率を向上させるためにVADモデルを削除し、`batch_size`を設定できます。\n\n```python\nmodel = AutoModel(model=model_dir, trust_remote_code=True, device=\"cuda:0\")\n\nres = model.generate(\n input=f\"{model.model_path}/example/en.mp3\",\n cache={},\n language=\"auto\", # \"zh\", \"en\", \"yue\", \"ja\", \"ko\", \"nospeech\"\n use_itn=True,\n batch_size=64, \n)\n```\n\n詳細な使用方法については、[ドキュメント](https://github.com/modelscope/FunASR/blob/main/docs/tutorial/README.md)を参照してください。\n\n### 直接推論\n\n任意の形式の音声入力をサポートし、入力音声の長さは30秒以下に制限されます。\n\n```python\nfrom model import SenseVoiceSmall\nfrom funasr.utils.postprocess_utils import rich_transcription_postprocess\n\nmodel_dir = \"iic/SenseVoiceSmall\"\nm, kwargs = SenseVoiceSmall.from_pretrained(model=model_dir, device=\"cuda:0\")\nm.eval()\n\nres = m.inference(\n data_in=f\"{kwargs['model_path']}/example/en.mp3\",\n language=\"auto\", # \"zh\", \"en\", \"yue\", \"ja\", \"ko\", \"nospeech\"\n use_itn=False,\n ban_emo_unk=False,\n **kwargs,\n)\n\ntext = rich_transcription_postprocess(res[0][0][\"text\"])\nprint(text)\n```\n\n## サービス展開\n\n未完了\n\n### エクスポートとテスト\n
ONNXとLibtorchのエクスポート\n\n#### ONNX\n```python\n# pip3 install -U funasr funasr-onnx\nfrom pathlib import Path\nfrom funasr_onnx import SenseVoiceSmall\nfrom funasr_onnx.utils.postprocess_utils import rich_transcription_postprocess\n\n\nmodel_dir = \"iic/SenseVoiceSmall\"\n\nmodel = SenseVoiceSmall(model_dir, batch_size=10, quantize=True)\n\n# inference\nwav_or_scp = [\"{}/.cache/modelscope/hub/{}/example/en.mp3\".format(Path.home(), model_dir)]\n\nres = model(wav_or_scp, language=\"auto\", use_itn=True)\nprint([rich_transcription_postprocess(i) for i in res])\n```\n備考：ONNXモデルは元のモデルディレクトリにエクスポートされます。\n\n#### Libtorch\n```python\nfrom pathlib import Path\nfrom funasr_torch import SenseVoiceSmall\nfrom funasr_torch.utils.postprocess_utils import rich_transcription_postprocess\n\n\nmodel_dir = \"iic/SenseVoiceSmall\"\n\nmodel = SenseVoiceSmall(model_dir, batch_size=10, device=\"cuda:0\")\n\nwav_or_scp = [\"{}/.cache/modelscope/hub/{}/example/en.mp3\".format(Path.home(), model_dir)]\n\nres = model(wav_or_scp, language=\"auto\", use_itn=True)\nprint([rich_transcription_postprocess(i) for i in res])\n```\n備考：Libtorchモデルは元のモデルディレクトリにエクスポートされます。\n\n
\n\n### 展開\n\n### FastAPIを使った展開\n```shell\nexport SENSEVOICE_DEVICE=cuda:0\nfastapi run --port 50000\n```\n\n## 微調整\n\n### トレーニング環境のインストール\n\n```shell\ngit clone https://github.com/alibaba/FunASR.git && cd FunASR\npip3 install -e ./\n```\n\n### データ準備\n\nデータ例\n```text\n{\"key\": \"YOU0000008470_S0000238_punc_itn\", \"text_language\": \"<|en|>\", \"emo_target\": \"<|NEUTRAL|>\", \"event_target\": \"<|Speech|>\", \"with_or_wo_itn\": \"<|withitn|>\", \"target\": \"Including legal due diligence, subscription agreement, negotiation.\", \"source\": \"/cpfs01/shared/Group-speech/beinian.lzr/data/industrial_data/english_all/audio/YOU0000008470_S0000238.wav\", \"target_len\": 7, \"source_len\": 140}\n{\"key\": \"AUD0000001556_S0007580\", \"text_language\": \"<|en|>\", \"emo_target\": \"<|NEUTRAL|>\", \"event_target\": \"<|Speech|>\", \"with_or_wo_itn\": \"<|woitn|>\", \"target\": \"there is a tendency to identify the self or take interest in what one has got used to\", \"source\": \"/cpfs01/shared/Group-speech/beinian.lzr/data/industrial_data/english_all/audio/AUD0000001556_S0007580.wav\", \"target_len\": 18, \"source_len\": 360}\n```\n詳細は `data/train_example.jsonl` を参照してください。\n\n
データ準備の詳細\n\n説明：\n- `key`：音声ファイルのユニークID\n- `source`：音声ファイルのパス\n- `source_len`：音声ファイルのfbankフレーム数\n- `target`：文字起こし結果\n- `target_len`：target（文字起こし）の長さ\n- `text_language`：音声ファイルの言語ID\n- `emo_target`：音声ファイルの感情ラベル\n- `event_target`：音声ファイルのイベントラベル\n- `with_or_wo_itn`：句読点と逆テキスト正規化を含むかどうか\n\n`train_text.txt`\n```bash\nBAC009S0764W0121 甚至出现交易几乎停滞的情况\nBAC009S0916W0489 湖北一公司以员工名义贷款数十员工负债千万\nasr_example_cn_en 所有只要处理 data 不管你是做 machine learning 做 deep learning 做 data analytics 做 data science 也好 scientist 也好通通都要都做的基本功啊那 again 先先对有一些>也许对\nID0012W0014 he tried to think how it could be\n```\n`train_wav.scp`\n```bash\nBAC009S0764W0121 https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/BAC009S0764W0121.wav\nBAC009S0916W0489 https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/BAC009S0916W0489.wav\nasr_example_cn_en https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/asr_example_cn_en.wav\nID0012W0014 https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/asr_example_en.wav\n```\n`train_text_language.txt`\n言語IDは `<|zh|>`、`<|en|>`、`<|yue|>`、`<|ja|>`、および `<|ko|>`を含みます。\n```bash\nBAC009S0764W0121 <|zh|>\nBAC009S0916W0489 <|zh|>\nasr_example_cn_en <|zh|>\nID0012W0014 <|en|>\n```\n`train_emo.txt`\n感情ラベルは、`<|HAPPY|>`、`<|SAD|>`、`<|ANGRY|>`、`<|NEUTRAL|>`、`<|FEARFUL|>`、`<|DISGUSTED|>` および `<|SURPRISED|>`を含みます。\n```bash\nBAC009S0764W0121 <|NEUTRAL|>\nBAC009S0916W0489 <|NEUTRAL|>\nasr_example_cn_en <|NEUTRAL|>\nID0012W0014 <|NEUTRAL|>\n```\n`train_event.txt`\nイベントラベルは、 `<|BGM|>`、`<|Speech|>`、`<|Applause|>`、`<|Laughter|>`、`<|Cry|>`、`<|Sneeze|>`、`<|Breath|>` および `<|Cough|>`を含みます。\n```bash\nBAC009S0764W0121 <|Speech|>\nBAC009S0916W0489 <|Speech|>\nasr_example_cn_en <|Speech|>\nID0012W0014 <|Speech|>\n```\n`コマンド`\n```shell\n# wav.scp、text.txt、text_language.txt、emo_target.txt、event_target.txt から train.jsonl と val.jsonl を生成します\nsensevoice2jsonl \\\n++scp_file_list='[\"../../../data/list/train_wav.scp\", \"../../../data/list/train_text.txt\", \"../../../data/list/train_text_language.txt\", \"../../../data/list/train_emo.txt\", \"../../../data/list/train_event.txt\"]' \\\n++data_type_list='[\"source\", \"target\", \"text_language\", \"emo_target\", \"event_target\"]' \\\n++jsonl_file_out=\"../../../data/list/train.jsonl\"\n```\n`train_text_language.txt`、`train_emo_target.txt`、`train_event_target.txt` がない場合、`SenseVoice` モデルを使用して言語、感情、およびイベントラベルが自動的に予測されます。\n```shell\n# wav.scp と text.txt から train.jsonl と val.jsonl を生成します\nsensevoice2jsonl \\\n++scp_file_list='[\"../../../data/list/train_wav.scp\", \"../../../data/list/train_text.txt\"]' \\\n++data_type_list='[\"source\", \"target\"]' \\\n++jsonl_file_out=\"../../../data/list/train.jsonl\"\n```\n
\n\n### トレーニングの開始\n\n`finetune.sh`の`train_tool`を、前述のFunASRパス内の`funasr/bin/train_ds.py`の絶対パスに変更することを忘れないでください。\n\n```shell\nbash finetune.sh\n```\n\n## WebUI\n\n```shell\npython webui.py\n```\n\n
\n\n## 注目すべきサードパーティの取り組み\n- Triton (GPU) デプロイメントのベストプラクティス：Triton + TensorRT を使用し、FP32 でテスト。V100 GPU で加速比 526 を達成。FP16 のサポートは進行中です。[リポジトリ](https://github.com/modelscope/FunASR/blob/main/runtime/triton_gpu/README.md)\n- Sherpa-onnx デプロイメントのベストプラクティス：SenseVoice を10種類のプログラミング言語（C++, C, Python, C#, Go, Swift, Kotlin, Java, JavaScript, Dart）で使用可能。また、iOS, Android, Raspberry Pi などのプラットフォームでも SenseVoice をデプロイできます。[リポジトリ](https://k2-fsa.github.io/sherpa/onnx/sense-voice/index.html)\n- [SenseVoice.cpp](https://github.com/lovemefan/SenseVoice.cpp) GGMLに基づいて純粋なC/C++でSenseVoiceを推測し、3ビット、4ビット、5ビット、8ビット量子化などをサポートし、サードパーティの依存関係はありません。\n- [streaming-sensevoice](https://github.com/pengzhendong/streaming-sensevoice) ストリーム型SenseVoiceは、チャンク（chunk）方式で推論を行います。擬似ストリーミング処理を実現するために、一部の精度を犠牲にして切り捨て注意機構（truncated attention）を採用しています。さらに、この技術はCTCプレフィックスビームサーチ（CTC prefix beam search）とホットワード強化機能もサポートしています。\n- [OmniSenseVoice](https://github.com/lifeiteng/OmniSenseVoice) は、超高速推論とバッチ処理のために最適化されています。\n- [SenseVoice Hotword](https://www.modelscope.cn/models/dengcunqin/SenseVoiceSmall_hotword)，ニューラルネットワークホットワード強化，[WeNetにおけるCPPNベースのニューラルネットワークホットワード強化のオープンソース](https://mp.weixin.qq.com/s/1QkIvh8j7rrUjRyWOgAvdA)。\n# お問い合わせ\n\n使用中に問題が発生した場合は、githubページで直接Issuesを提起できます。音声に興味のある方は、以下のDingTalkグループQRコードをスキャンしてコミュニティグループに参加し、交流と議論を行ってください。\n\n| FunASR |\n|:--------------------------------------------------------:|\n| |\n\n" }, { "path": "README_zh.md", "content": "# SenseVoice\n\n「简体中文」|「[English](./README.md)」|「[日本語](./README_ja.md)」\n\nSenseVoice 是具有音频理解能力的音频基础模型，包括语音识别（ASR）、语种识别（LID）、语音情感识别（SER）和声学事件分类（AEC）或声学事件检测（AED）。本项目提供 SenseVoice 模型的介绍以及在多个任务测试集上的 benchmark，以及体验模型所需的环境安装的与推理方式。\n\n
\n\n
\n\n
\n

\n Homepage \n｜ 最新动态 \n｜ 性能评测 \n｜ 环境安装 \n｜ 用法教程 \n｜ 联系我们 \n\n

\n\n模型仓库：[modelscope](https://www.modelscope.cn/models/iic/SenseVoiceSmall)，[huggingface](https://huggingface.co/FunAudioLLM/SenseVoiceSmall)\n\n在线体验：\n[modelscope demo](https://www.modelscope.cn/studios/iic/SenseVoice), [huggingface space](https://huggingface.co/spaces/FunAudioLLM/SenseVoice)\n\n
\n\n\n\n# 核心功能 🎯\n\n**SenseVoice** 专注于高精度多语言语音识别、情感辨识和音频事件检测\n\n- **多语言识别：** 采用超过 40 万小时数据训练，支持超过 50 种语言，识别效果上优于 Whisper 模型。\n- **富文本识别：**\n - 具备优秀的情感识别，能够在测试数据上达到和超过目前最佳情感识别模型的效果。\n - 支持声音事件检测能力，支持音乐、掌声、笑声、哭声、咳嗽、喷嚏等多种常见人机交互事件进行检测。\n- **高效推理：** SenseVoice-Small 模型采用非自回归端到端框架，推理延迟极低，10s 音频推理仅耗时 70ms，15 倍优于 Whisper-Large。\n- **微调定制：** 具备便捷的微调脚本与策略，方便用户根据业务场景修复长尾样本问题。\n- **服务部署：** 具有完整的服务部署链路，支持多并发请求，支持客户端语言有，python、c++、html、java 与 c# 等。\n\n\n\n# 最新动态 🔥\n\n- 2024/7：新增加导出 [ONNX](./demo_onnx.py) 与 [libtorch](./demo_libtorch.py) 功能，以及 python 版本 runtime：[funasr-onnx-0.4.0](https://pypi.org/project/funasr-onnx/)，[funasr-torch-0.1.1](https://pypi.org/project/funasr-torch/)\n- 2024/7: [SenseVoice-Small](https://www.modelscope.cn/models/iic/SenseVoiceSmall) 多语言音频理解模型开源，支持中、粤、英、日、韩语的多语言语音识别，情感识别和事件检测能力，具有极低的推理延迟。。\n- 2024/7: CosyVoice 致力于自然语音生成，支持多语言、音色和情感控制，擅长多语言语音生成、零样本语音生成、跨语言语音克隆以及遵循指令的能力。[CosyVoice repo](https://github.com/FunAudioLLM/CosyVoice) and [CosyVoice 在线体验](https://www.modelscope.cn/studios/iic/CosyVoice-300M).\n- 2024/7: [FunASR](https://github.com/modelscope/FunASR) 是一个基础语音识别工具包，提供多种功能，包括语音识别（ASR）、语音端点检测（VAD）、标点恢复、语言模型、说话人验证、说话人分离和多人对话语音识别等。\n\n\n\n# 性能评测 📝\n\n## 多语言语音识别\n\n我们在开源基准数据集（包括 AISHELL-1、AISHELL-2、Wenetspeech、Librispeech 和 Common Voice）上比较了 SenseVoice 与 Whisper 的多语言语音识别性能和推理效率。在中文和粤语识别效果上，SenseVoice-Small 模型具有明显的效果优势。\n\n
\n\n
\n\n## 情感识别\n\n由于目前缺乏被广泛使用的情感识别测试指标和方法，我们在多个测试集的多种指标进行测试，并与近年来 Benchmark 上的多个结果进行了全面的对比。所选取的测试集同时包含中文 / 英文两种语言以及表演、影视剧、自然对话等多种风格的数据，在不进行目标数据微调的前提下，SenseVoice 能够在测试数据上达到和超过目前最佳情感识别模型的效果。\n\n
\n\n
\n\n同时，我们还在测试集上对多个开源情感识别模型进行对比，结果表明，SenseVoice-Large 模型可以在几乎所有数据上都达到了最佳效果，而 SenseVoice-Small 模型同样可以在多数数据集上取得超越其他开源模型的效果。\n\n
\n\n
\n\n## 事件检测\n\n尽管 SenseVoice 只在语音数据上进行训练，它仍然可以作为事件检测模型进行单独使用。我们在环境音分类 ESC-50 数据集上与目前业内广泛使用的 BEATS 与 PANN 模型的效果进行了对比。SenseVoice 模型能够在这些任务上取得较好的效果，但受限于训练数据与训练方式，其事件分类效果专业的事件检测模型相比仍然有一定的差距。\n\n
\n\n
\n\n## 推理效率\n\nSenseVoice-small 模型采用非自回归端到端架构，推理延迟极低。在参数量与 Whisper-Small 模型相当的情况下，比 Whisper-Small 模型推理速度快 5 倍，比 Whisper-Large 模型快 15 倍。同时 SenseVoice-small 模型在音频时长增加的情况下，推理耗时也无明显增加。\n\n
\n\n
\n\n\n\n# 安装依赖环境 🐍\n\n```shell\npip install -r requirements.txt\n```\n\n\n\n# 用法 🛠️\n\n## 推理\n\n### 使用 funasr 推理\n\n支持任意格式音频输入，支持任意时长输入\n\n```python\nfrom funasr import AutoModel\nfrom funasr.utils.postprocess_utils import rich_transcription_postprocess\n\nmodel_dir = \"iic/SenseVoiceSmall\"\n\n\nmodel = AutoModel(\n model=model_dir,\n trust_remote_code=True,\n remote_code=\"./model.py\", \n vad_model=\"fsmn-vad\",\n vad_kwargs={\"max_single_segment_time\": 30000},\n device=\"cuda:0\",\n)\n\n# en\nres = model.generate(\n input=f\"{model.model_path}/example/en.mp3\",\n cache={},\n language=\"auto\", # \"zh\", \"en\", \"yue\", \"ja\", \"ko\", \"nospeech\"\n use_itn=True,\n batch_size_s=60,\n merge_vad=True,\n merge_length_s=15,\n)\ntext = rich_transcription_postprocess(res[0][\"text\"])\nprint(text)\n```\n\n
参数说明（点击展开）\n\n- `model_dir`：模型名称，或本地磁盘中的模型路径。\n- `trust_remote_code`：\n - `True` 表示 model 代码实现从 `remote_code` 处加载，`remote_code` 指定 `model` 具体代码的位置（例如，当前目录下的 `model.py`），支持绝对路径与相对路径，以及网络 url。\n - `False` 表示，model 代码实现为 [FunASR](https://github.com/modelscope/FunASR) 内部集成版本，此时修改当前目录下的 `model.py` 不会生效，因为加载的是 funasr 内部版本，模型代码 [点击查看](https://github.com/modelscope/FunASR/tree/main/funasr/models/sense_voice)。\n- `vad_model`：表示开启 VAD，VAD 的作用是将长音频切割成短音频，此时推理耗时包括了 VAD 与 SenseVoice 总耗时，为链路耗时，如果需要单独测试 SenseVoice 模型耗时，可以关闭 VAD 模型。\n- `vad_kwargs`：表示 VAD 模型配置，`max_single_segment_time`: 表示 `vad_model` 最大切割音频时长，单位是毫秒 ms。\n- `use_itn`：输出结果中是否包含标点与逆文本正则化。\n- `batch_size_s` 表示采用动态 batch，batch 中总音频时长，单位为秒 s。\n- `merge_vad`：是否将 vad 模型切割的短音频碎片合成，合并后长度为 `merge_length_s`，单位为秒 s。\n- `ban_emo_unk`：禁用 emo_unk 标签，禁用后所有的句子都会被赋与情感标签。默认 `False`\n\n
\n\n如果输入均为短音频（小于 30s），并且需要批量化推理，为了加快推理效率，可以移除 vad 模型，并设置 `batch_size`\n\n```python\nmodel = AutoModel(model=model_dir, trust_remote_code=True, device=\"cuda:0\")\n\nres = model.generate(\n input=f\"{model.model_path}/example/en.mp3\",\n cache={},\n language=\"auto\", # \"zh\", \"en\", \"yue\", \"ja\", \"ko\", \"nospeech\"\n use_itn=True,\n batch_size=64, \n)\n```\n\n更多详细用法，请参考 [文档](https://github.com/modelscope/FunASR/blob/main/docs/tutorial/README.md)\n\n### 直接推理\n\n支持任意格式音频输入，输入音频时长限制在 30s 以下\n\n```python\nfrom model import SenseVoiceSmall\nfrom funasr.utils.postprocess_utils import rich_transcription_postprocess\n\nmodel_dir = \"iic/SenseVoiceSmall\"\nm, kwargs = SenseVoiceSmall.from_pretrained(model=model_dir, device=\"cuda:0\")\nm.eval()\n\nres = m.inference(\n data_in=f\"{kwargs ['model_path']}/example/en.mp3\",\n language=\"auto\", # \"zh\", \"en\", \"yue\", \"ja\", \"ko\", \"nospeech\"\n use_itn=False,\n ban_emo_unk=False,\n **kwargs,\n)\n\ntext = rich_transcription_postprocess(res [0][0][\"text\"])\nprint(text)\n```\n\n## 服务部署\n\nUndo\n\n### 导出与测试\n\n
ONNX 与 Libtorch 导出 \n\n#### ONNX\n\n```python\n# pip3 install -U funasr funasr-onnx\nfrom pathlib import Path\nfrom funasr_onnx import SenseVoiceSmall\nfrom funasr_onnx.utils.postprocess_utils import rich_transcription_postprocess\n\n\nmodel_dir = \"iic/SenseVoiceSmall\"\n\nmodel = SenseVoiceSmall(model_dir, batch_size=10, quantize=True)\n\n# inference\nwav_or_scp = [\"{}/.cache/modelscope/hub/{}/example/en.mp3\".format(Path.home(), model_dir)]\n\nres = model(wav_or_scp, language=\"auto\", use_itn=True)\nprint([rich_transcription_postprocess(i) for i in res])\n```\n\n备注：ONNX 模型导出到原模型目录中\n\n#### Libtorch\n\n```python\nfrom pathlib import Path\nfrom funasr_torch import SenseVoiceSmall\nfrom funasr_torch.utils.postprocess_utils import rich_transcription_postprocess\n\n\nmodel_dir = \"iic/SenseVoiceSmall\"\n\nmodel = SenseVoiceSmall(model_dir, batch_size=10, device=\"cuda:0\")\n\nwav_or_scp = [\"{}/.cache/modelscope/hub/{}/example/en.mp3\".format(Path.home(), model_dir)]\n\nres = model(wav_or_scp, language=\"auto\", use_itn=True)\nprint([rich_transcription_postprocess (i) for i in res])\n```\n\n备注：Libtorch 模型导出到原模型目录中\n\n
\n\n### 部署\n\n### 使用 FastAPI 部署\n\n```shell\nexport SENSEVOICE_DEVICE=cuda:0\nfastapi run --port 50000\n```\n\n## 微调\n\n### 安装训练环境\n\n```shell\ngit clone https://github.com/alibaba/FunASR.git && cd FunASR\npip3 install -e ./\n```\n\n### 数据准备\n\n数据格式需要包括如下几个字段：\n\n```text\n{\"key\": \"YOU0000008470_S0000238_punc_itn\", \"text_language\": \"<|en|>\", \"emo_target\": \"<|NEUTRAL|>\", \"event_target\": \"<|Speech|>\", \"with_or_wo_itn\": \"<|withitn|>\", \"target\": \"Including legal due diligence, subscription agreement, negotiation.\", \"source\": \"/cpfs01/shared/Group-speech/beinian.lzr/data/industrial_data/english_all/audio/YOU0000008470_S0000238.wav\", \"target_len\": 7, \"source_len\": 140}\n{\"key\": \"AUD0000001556_S0007580\", \"text_language\": \"<|en|>\", \"emo_target\": \"<|NEUTRAL|>\", \"event_target\": \"<|Speech|>\", \"with_or_wo_itn\": \"<|woitn|>\", \"target\": \"there is a tendency to identify the self or take interest in what one has got used to\", \"source\": \"/cpfs01/shared/Group-speech/beinian.lzr/data/industrial_data/english_all/audio/AUD0000001556_S0007580.wav\", \"target_len\": 18, \"source_len\": 360}\n```\n\n详细可以参考：`data/train_example.jsonl`\n\n
数据准备细节介绍 \n\n- `key`: 数据唯一 ID\n- `source`：音频文件的路径\n- `source_len`：音频文件的 fbank 帧数\n- `target`：音频文件标注文本\n- `target_len`：音频文件标注文本长度\n- `text_language`：音频文件的语种标签\n- `emo_target`：音频文件的情感标签\n- `event_target`：音频文件的事件标签\n- `with_or_wo_itn`：标注文本中是否包含标点与逆文本正则化\n\n可以用指令 `sensevoice2jsonl` 从 train_wav.scp、train_text.txt、train_text_language.txt、train_emo_target.txt 和 train_event_target.txt 生成，准备过程如下：\n\n`train_text.txt`\n\n左边为数据唯一 ID，需与 `train_wav.scp` 中的 `ID` 一一对应\n右边为音频文件标注文本，格式如下：\n\n```bash\nBAC009S0764W0121 甚至出现交易几乎停滞的情况\nBAC009S0916W0489 湖北一公司以员工名义贷款数十员工负债千万\nasr_example_cn_en 所有只要处理 data 不管你是做 machine learning 做 deep learning 做 data analytics 做 data science 也好 scientist 也好通通都要都做的基本功啊那 again 先先对有一些 > 也许对\nID0012W0014 he tried to think how it could be\n```\n\n`train_wav.scp`\n\n左边为数据唯一 ID，需与 `train_text.txt` 中的 `ID` 一一对应\n右边为音频文件的路径，格式如下\n\n```bash\nBAC009S0764W0121 https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/BAC009S0764W0121.wav\nBAC009S0916W0489 https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/BAC009S0916W0489.wav\nasr_example_cn_en https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/asr_example_cn_en.wav\nID0012W0014 https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/asr_example_en.wav\n```\n\n`train_text_language.txt`\n\n左边为数据唯一 ID，需与 `train_text_language.txt` 中的 `ID` 一一对应\n右边为音频文件的语种标签，支持 `<|zh|>`、`<|en|>`、`<|yue|>`、`<|ja|>` 和 `<|ko|>`，格式如下\n\n```bash\nBAC009S0764W0121 <|zh|>\nBAC009S0916W0489 <|zh|>\nasr_example_cn_en <|zh|>\nID0012W0014 <|en|>\n```\n\n`train_emo.txt`\n\n左边为数据唯一 ID，需与 `train_emo.txt` 中的 `ID` 一一对应\n右边为音频文件的情感标签，支持 `<|HAPPY|>`、`<|SAD|>`、`<|ANGRY|>`、`<|NEUTRAL|>`、`<|FEARFUL|>`、`<|DISGUSTED|>` 和 `<|SURPRISED|>`，格式如下\n\n```bash\nBAC009S0764W0121 <|NEUTRAL|>\nBAC009S0916W0489 <|NEUTRAL|>\nasr_example_cn_en <|NEUTRAL|>\nID0012W0014 <|NEUTRAL|>\n```\n\n`train_event.txt`\n\n左边为数据唯一 ID，需与 `train_event.txt` 中的 `ID` 一一对应\n右边为音频文件的事件标签，支持 `<|BGM|>`、`<|Speech|>`、`<|Applause|>`、`<|Laughter|>`、`<|Cry|>`、`<|Sneeze|>`、`<|Breath|>` 和 `<|Cough|>`，格式如下\n\n```bash\nBAC009S0764W0121 <|Speech|>\nBAC009S0916W0489 <|Speech|>\nasr_example_cn_en <|Speech|>\nID0012W0014 <|Speech|>\n```\n\n`生成指令`\n\n```shell\n# generate train.jsonl and val.jsonl from wav.scp, text.txt, text_language.txt, emo_target.txt, event_target.txt\nsensevoice2jsonl \\\n++scp_file_list='[\"../../../data/list/train_wav.scp\", \"../../../data/list/train_text.txt\", \"../../../data/list/train_text_language.txt\", \"../../../data/list/train_emo.txt\", \"../../../data/list/train_event.txt\"]' \\\n++data_type_list='[\"source\", \"target\", \"text_language\", \"emo_target\", \"event_target\"]' \\\n++jsonl_file_out=\"../../../data/list/train.jsonl\"\n```\n\n若无 train_text_language.txt、train_emo_target.txt 和 train_event_target.txt，则自动通过使用 `SenseVoice` 模型对语种、情感和事件打标。\n\n```shell\n# generate train.jsonl and val.jsonl from wav.scp and text.txt\nsensevoice2jsonl \\\n++scp_file_list='[\"../../../data/list/train_wav.scp\", \"../../../data/list/train_text.txt\"]' \\\n++data_type_list='[\"source\", \"target\"]' \\\n++jsonl_file_out=\"../../../data/list/train.jsonl\" \\\n++model_dir='iic/SenseVoiceSmall'\n```\n\n
\n\n### 启动训练\n\n注意修改 `finetune.sh` 中 `train_tool` 为你前面安装 FunASR 路径中 `funasr/bin/train_ds.py` 绝对路径\n\n```shell\nbash finetune.sh\n```\n\n## WebUI\n\n```shell\npython webui.py\n```\n\n
\n\n## 优秀三方工作\n\n- Triton（GPU）部署最佳实践，triton + tensorrt，fp32 测试，V100 GPU 上加速比 526，fp16 支持中，[repo](https://github.com/modelscope/FunASR/blob/main/runtime/triton_gpu/README.md)\n- sherpa-onnx 部署最佳实践，支持在 10 种编程语言里面使用 SenseVoice, 即 C++, C, Python, C#, Go, Swift, Kotlin, Java, JavaScript, Dart. 支持在 iOS, Android, Raspberry Pi 等平台使用 SenseVoice，[repo](https://k2-fsa.github.io/sherpa/onnx/sense-voice/index.html)\n- [SenseVoice.cpp](https://github.com/lovemefan/SenseVoice.cpp) 基于GGML，在纯C/C++中推断SenseVoice，支持3位、4位、5位、8位量化等，无需第三方依赖。\n- [流式SenseVoice](https://github.com/pengzhendong/streaming-sensevoice)，通过分块（chunk）的方式进行推理，为了实现伪流式处理，采用了截断注意力机制（truncated attention），牺牲了部分精度。此外，该技术还支持CTC前缀束搜索（CTC prefix beam search）以及热词增强功能。\n- [OmniSenseVoice](https://github.com/lifeiteng/OmniSenseVoice) 轻量化推理库，支持batch推理。\n- [SenseVoice Hotword](https://www.modelscope.cn/models/dengcunqin/SenseVoiceSmall_hotword)，神经网络热词增强，[WeNet 中开源基于 CPPN 的神经网络热词增强](https://mp.weixin.qq.com/s/1QkIvh8j7rrUjRyWOgAvdA)。\n# 联系我们\n\n如果您在使用中遇到问题，可以直接在 github 页面提 Issues。欢迎语音兴趣爱好者扫描以下的钉钉群二维码加入社区群，进行交流和讨论。\n\n| FunASR |\n|:--------------------------------------------------------:|\n| |\n" }, { "path": "api.py", "content": "# Set the device with environment, default is cuda:0\n# export SENSEVOICE_DEVICE=cuda:1\n\nimport os, re\nfrom fastapi import FastAPI, File, Form, UploadFile\nfrom fastapi.responses import HTMLResponse\nfrom typing_extensions import Annotated\nfrom typing import List\nfrom enum import Enum\nimport torchaudio\nfrom model import SenseVoiceSmall\nfrom funasr.utils.postprocess_utils import rich_transcription_postprocess\nfrom io import BytesIO\n\nTARGET_FS = 16000\n\n\nclass Language(str, Enum):\n auto = \"auto\"\n zh = \"zh\"\n en = \"en\"\n yue = \"yue\"\n ja = \"ja\"\n ko = \"ko\"\n nospeech = \"nospeech\"\n\n\nmodel_dir = \"iic/SenseVoiceSmall\"\nm, kwargs = SenseVoiceSmall.from_pretrained(model=model_dir, device=os.getenv(\"SENSEVOICE_DEVICE\", \"cuda:0\"))\nm.eval()\n\nregex = r\"<\\|.*\\|>\"\n\napp = FastAPI()\n\n\n@app.get(\"/\", response_class=HTMLResponse)\nasync def root():\n return \"\"\"\n \n \n \n \n Api information\n \n \n Documents of API\n \n \n \"\"\"\n\n\n@app.post(\"/api/v1/asr\")\nasync def turn_audio_to_text(\n files: Annotated[List[UploadFile], File(description=\"wav or mp3 audios in 16KHz\")],\n keys: Annotated[str, Form(description=\"name of each audio joined with comma\")] = None,\n lang: Annotated[Language, Form(description=\"language of audio content\")] = \"auto\",\n):\n audios = []\n for file in files:\n file_io = BytesIO(await file.read())\n data_or_path_or_list, audio_fs = torchaudio.load(file_io)\n\n # transform to target sample\n if audio_fs != TARGET_FS:\n resampler = torchaudio.transforms.Resample(orig_freq=audio_fs, new_freq=TARGET_FS)\n data_or_path_or_list = resampler(data_or_path_or_list)\n\n data_or_path_or_list = data_or_path_or_list.mean(0)\n audios.append(data_or_path_or_list)\n\n if lang == \"\":\n lang = \"auto\"\n\n if not keys:\n key = [f.filename for f in files]\n else:\n key = keys.split(\",\")\n\n res = m.inference(\n data_in=audios,\n language=lang, # \"zh\", \"en\", \"yue\", \"ja\", \"ko\", \"nospeech\"\n use_itn=False,\n ban_emo_unk=False,\n key=key,\n fs=TARGET_FS,\n **kwargs,\n )\n if len(res) == 0:\n return {\"result\": []}\n for it in res[0]:\n it[\"raw_text\"] = it[\"text\"]\n it[\"clean_text\"] = re.sub(regex, \"\", it[\"text\"], 0, re.MULTILINE)\n it[\"text\"] = rich_transcription_postprocess(it[\"text\"])\n return {\"result\": res[0]}\n\n\nif __name__ == \"__main__\":\n import uvicorn\n\n uvicorn.run(app, host=\"0.0.0.0\", port=50000)\n" }, { "path": "data/train_example.jsonl", "content": "{\"key\": \"YOU0000008470_S0000238_punc_itn\", \"text_language\": \"<|en|>\", \"emo_target\": \"<|NEUTRAL|>\", \"event_target\": \"<|Speech|>\", \"with_or_wo_itn\": \"<|withitn|>\", \"target\": \"Including legal due diligence, subscription agreement, negotiation.\", \"source\": \"/cpfs01/shared/Group-speech/beinian.lzr/data/industrial_data/english_all/audio/YOU0000008470_S0000238.wav\", \"target_len\": 7, \"source_len\": 140}\n{\"key\": \"AUD0000001556_S0007580\", \"text_language\": \"<|en|>\", \"emo_target\": \"<|NEUTRAL|>\", \"event_target\": \"<|Speech|>\", \"with_or_wo_itn\": \"<|woitn|>\", \"target\": \"there is a tendency to identify the self or take interest in what one has got used to\", \"source\": \"/cpfs01/shared/Group-speech/beinian.lzr/data/industrial_data/english_all/audio/AUD0000001556_S0007580.wav\", \"target_len\": 18, \"source_len\": 360}\n{\"key\": \"19208207_HJwKrcFJ8_o_segment720\", \"text_language\": \"<|en|>\", \"emo_target\": \"<|EMO_UNKNOWN|>\", \"event_target\": \"<|Speech|>\", \"with_or_wo_itn\": \"<|woitn|>\", \"target\": \"fourth foul up top now to austin three for leonard and in and out no good rebounded by murray looking for some help and he almost throws it out of bounds\", \"source\": \"/cpfs_speech/data/shared/Group-speech/beinian.lzr/data/multilingual/lcd_data/english/production/20240222/wer_0_5/taskid_19208207/wav/19208207_HJwKrcFJ8_o_segment720.wav\", \"target_len\": 31, \"source_len\": 620}\n{\"key\": \"wav000_0872_bb6f4a79bb9f49249083465445b1cafa_01c38131ed3a4c609e8270a09170525d\", \"text_language\": \"<|zh|>\", \"emo_target\": \"<|NEUTRAL|>\", \"event_target\": \"<|Speech|>\", \"with_or_wo_itn\": \"<|woitn|>\", \"target\": \"案件受理费减半收取计一千六百三十一元赵会龙已预交由赵会龙负担一百零七元\", \"source\": \"/cpfs01/shared/Group-speech/beinian.lzr/data/industrial_data/16k_common/audio/wav000_0872_bb6f4a79bb9f49249083465445b1cafa_01c38131ed3a4c609e8270a09170525d.wav\", \"target_len\": 35, \"source_len\": 700}\n{\"key\": \"Speaker0244_iPhone_s0_227\", \"text_language\": \"<|ko|>\", \"emo_target\": \"<|NEUTRAL|>\", \"event_target\": \"<|Speech|>\", \"with_or_wo_itn\": \"<|woitn|>\", \"target\": \"저녁 다 해결합니다\", \"source\": \"/cpfs_speech/data/shared/Group-speech/beinian.lzr/data/multilingual/korean/audio/Speaker0244_iPhone_s0_227.wav\", \"target_len\": 10, \"source_len\": 200}\n{\"key\": \"data2sim_speed_part1_channel0_CHANNEL0_SPEAKER0948_SESSION1_009481629_speed12\", \"text_language\": \"<|en|>\", \"emo_target\": \"<|NEUTRAL|>\", \"event_target\": \"<|Speech|>\", \"with_or_wo_itn\": \"<|woitn|>\", \"target\": \"the money was entrust to him in february this year before he resign in june according to the documents\", \"source\": \"/cpfs01/shared/Group-speech/beinian.lzr/data/industrial_data/english_all/audio/data2sim_speed_part1_channel0_CHANNEL0_SPEAKER0948_SESSION1_009481629_speed12.wav\", \"target_len\": 19, \"source_len\": 380}\n{\"key\": \"SPEAKER0272_SESSION1_002721613_punc_itn\", \"text_language\": \"<|en|>\", \"emo_target\": \"<|EMO_UNKNOWN|>\", \"event_target\": \"<|Speech|>\", \"with_or_wo_itn\": \"<|withitn|>\", \"target\": \"Current proposals don't go far enough.\", \"source\": \"/cpfs01/shared/Group-speech/beinian.lzr/data/industrial_data/english_all/audio/SPEAKER0272_SESSION1_002721613.wav\", \"target_len\": 6, \"source_len\": 120}\n{\"key\": \"wav004_0490_04c4f9cb2cb347a2a156c5cad1a903aa\", \"text_language\": \"<|zh|>\", \"emo_target\": \"<|NEUTRAL|>\", \"event_target\": \"<|Speech|>\", \"with_or_wo_itn\": \"<|woitn|>\", \"target\": \"小凳子\", \"source\": \"/cpfs01/shared/Group-speech/beinian.lzr/data/industrial_data/16k_common/audio/wav004_0490_04c4f9cb2cb347a2a156c5cad1a903aa.wav\", \"target_len\": 3, \"source_len\": 60}\n{\"key\": \"18874657_MSnA4nfDC7Q_segment680\", \"text_language\": \"<|en|>\", \"emo_target\": \"<|NEUTRAL|>\", \"event_target\": \"<|Speech|>\", \"with_or_wo_itn\": \"<|woitn|>\", \"target\": \"and anything that you would like to know please just put it in the email\", \"source\": \"/cpfs_speech/data/shared/Group-speech/beinian.lzr/data/multilingual/lcd_data/english/production/20240105/wer_0_5/taskid_18874657/wav/18874657_MSnA4nfDC7Q_segment680.wav\", \"target_len\": 15, \"source_len\": 300}\n{\"key\": \"POD0000007250_S0000518\", \"text_language\": \"<|en|>\", \"emo_target\": \"<|EMO_UNKNOWN|>\", \"event_target\": \"<|Speech|>\", \"with_or_wo_itn\": \"<|woitn|>\", \"target\": \"i use netflix but and that's not an app\", \"source\": \"/cpfs01/shared/Group-speech/beinian.lzr/data/industrial_data/english_all/audio/POD0000007250_S0000518.wav\", \"target_len\": 9, \"source_len\": 180}\n" }, { "path": "data/val_example.jsonl", "content": "{\"key\": \"datasim_speed_Speaker0129_winPhone_s0_149_speed-10_punc_itn\", \"text_language\": \"<|ko|>\", \"emo_target\": \"<|NEUTRAL|>\", \"event_target\": \"<|Speech|>\", \"with_or_wo_itn\": \"<|withitn|>\", \"target\": \"이거 올리고 리베옹과 안 좋은 사이가 되었다는.\", \"source\": \"/cpfs_speech/data/shared/Group-speech/beinian.lzr/data/multilingual/korean/audio/datasim_speed_Speaker0129_winPhone_s0_149_speed-10.wav\", \"target_len\": 26, \"source_len\": 520}\n{\"key\": \"data2sim_noise_rir_new_Speaker0048_winPhone_s0_102_punc_itn\", \"text_language\": \"<|ko|>\", \"emo_target\": \"<|NEUTRAL|>\", \"event_target\": \"<|Speech|>\", \"with_or_wo_itn\": \"<|withitn|>\", \"target\": \"개통 대리점이랑 얘길 해봐야 합니다.\", \"source\": \"/cpfs_speech/data/shared/Group-speech/beinian.lzr/data/multilingual/korean/audio/data2sim_noise_rir_new_Speaker0048_winPhone_s0_102.wav\", \"target_len\": 20, \"source_len\": 400}\n{\"key\": \"wav005_0655_1225906248786196892_punc_itn\", \"text_language\": \"<|yue|>\", \"emo_target\": \"<|EMO_UNKNOWN|>\", \"event_target\": \"<|Speech|>\", \"with_or_wo_itn\": \"<|withitn|>\", \"target\": \"万科租售中心。\", \"source\": \"/cpfs01/shared/Group-speech/beinian.lzr/data/multilingual/cantonese/audio/wav005_0655_1225906248786196892.wav\", \"target_len\": 7, \"source_len\": 140}\n{\"key\": \"datasim_speed_Speaker0732_S0_Android_533_speed-10\", \"text_language\": \"<|ko|>\", \"emo_target\": \"<|NEUTRAL|>\", \"event_target\": \"<|Speech|>\", \"with_or_wo_itn\": \"<|woitn|>\", \"target\": \"댁 공개 석상에서 에이즈 환자랑 껴안고 뭐 키스하고\", \"source\": \"/cpfs_speech/data/shared/Group-speech/beinian.lzr/data/multilingual/korean/audio/datasim_speed_Speaker0732_S0_Android_533_speed-10.wav\", \"target_len\": 28, \"source_len\": 560}\n{\"key\": \"wav001_0437_lATPJxDjwp5xwcnOQp9lTs51zRmz\", \"text_language\": \"<|zh|>\", \"emo_target\": \"<|NEUTRAL|>\", \"event_target\": \"<|Speech|>\", \"with_or_wo_itn\": \"<|woitn|>\", \"target\": \"郑超啊你到我这儿来一下\", \"source\": \"/cpfs01/shared/Group-speech/beinian.lzr/data/industrial_data/16k_common/audio/wav001_0437_lATPJxDjwp5xwcnOQp9lTs51zRmz.wav\", \"target_len\": 11, \"source_len\": 220}\n{\"key\": \"wav010_0212_Speaker0045_iOS_s0_088_punc_itn\", \"text_language\": \"<|en|>\", \"emo_target\": \"<|EMO_UNKNOWN|>\", \"event_target\": \"<|Speech|>\", \"with_or_wo_itn\": \"<|withitn|>\", \"target\": \"In dark moments, I speak with her.\", \"source\": \"/cpfs01/shared/Group-speech/beinian.lzr/data/industrial_data/english_all/audio/wav010_0212_Speaker0045_iOS_s0_088.wav\", \"target_len\": 7, \"source_len\": 140}\n{\"key\": \"18934860_GSD17-Sz1vw_segment107\", \"text_language\": \"<|en|>\", \"emo_target\": \"<|NEUTRAL|>\", \"event_target\": \"<|Speech|>\", \"with_or_wo_itn\": \"<|woitn|>\", \"target\": \"some states also include suspected domestic violence within mandatory reporting laws\", \"source\": \"/cpfs_speech/data/shared/Group-speech/beinian.lzr/data/multilingual/lcd_data/english/production/20240114/wer_0_5/taskid_18934860/wav/18934860_GSD17-Sz1vw_segment107.wav\", \"target_len\": 11, \"source_len\": 220}\n{\"key\": \"wav003_0734_XSG5MY6Eym.mp4_160\", \"text_language\": \"<|zh|>\", \"emo_target\": \"<|NEUTRAL|>\", \"event_target\": \"<|Speech|>\", \"with_or_wo_itn\": \"<|woitn|>\", \"target\": \"四万多四万多\", \"source\": \"/cpfs01/shared/Group-speech/beinian.lzr/data/industrial_data/dialect/audio/wav003_0734_XSG5MY6Eym.mp4_160.wav\", \"target_len\": 6, \"source_len\": 120}\n{\"key\": \"19208463_4_dWQ34YNU4_segment311\", \"text_language\": \"<|en|>\", \"emo_target\": \"<|EMO_UNKNOWN|>\", \"event_target\": \"<|Speech|>\", \"with_or_wo_itn\": \"<|woitn|>\", \"target\": \"i said well you have to see that movie history of the world\", \"source\": \"/cpfs_speech/data/shared/Group-speech/beinian.lzr/data/multilingual/lcd_data/english/production/20240222/wer_0_5/taskid_19208463/wav/19208463_4_dWQ34YNU4_segment311.wav\", \"target_len\": 13, \"source_len\": 260}\n{\"key\": \"wav005_0682_lATPJv8gRtSVeIDOUGfqsc5DF8yn_13_punc_itn\", \"text_language\": \"<|zh|>\", \"emo_target\": \"<|NEUTRAL|>\", \"event_target\": \"<|Speech|>\", \"with_or_wo_itn\": \"<|withitn|>\", \"target\": \"这也是我们呃。\", \"source\": \"/cpfs01/shared/Group-speech/beinian.lzr/data/industrial_data/16k_common/audio/wav005_0682_lATPJv8gRtSVeIDOUGfqsc5DF8yn_13.wav\", \"target_len\": 7, \"source_len\": 140}\n" }, { "path": "deepspeed_conf/ds_stage1.json", "content": "{\n \"train_micro_batch_size_per_gpu\": 1,\n \"gradient_accumulation_steps\": 1,\n \"steps_per_print\": 100,\n \"gradient_clipping\": 5,\n \"fp16\": {\n \"enabled\": false,\n \"auto_cast\": false,\n \"loss_scale\": 0,\n \"initial_scale_power\": 16,\n \"loss_scale_window\": 1000,\n \"hysteresis\": 2,\n \"consecutive_hysteresis\": false,\n \"min_loss_scale\": 1\n },\n \"bf16\": {\n \"enabled\": true\n },\n \"zero_force_ds_cpu_optimizer\": false,\n \"zero_optimization\": {\n \"stage\": 1,\n \"offload_optimizer\": {\n \"device\": \"none\",\n \"pin_memory\": true\n },\n \"allgather_partitions\": true,\n \"allgather_bucket_size\": 5e8,\n \"overlap_comm\": true,\n \"reduce_scatter\": true,\n \"reduce_bucket_size\": 5e8,\n \"contiguous_gradients\" : true\n }\n}\n" }, { "path": "demo1.py", "content": "#!/usr/bin/env python3\n# -*- encoding: utf-8 -*-\n# Copyright FunASR (https://github.com/FunAudioLLM/SenseVoice). All Rights Reserved.\n# MIT License (https://opensource.org/licenses/MIT)\n\nfrom funasr import AutoModel\nfrom funasr.utils.postprocess_utils import rich_transcription_postprocess\n\nmodel_dir = \"iic/SenseVoiceSmall\"\n\n\nmodel = AutoModel(\n model=model_dir,\n trust_remote_code=True,\n remote_code=\"./model.py\",\n vad_model=\"fsmn-vad\",\n vad_kwargs={\"max_single_segment_time\": 30000},\n device=\"cuda:0\",\n)\n\n# en\nres = model.generate(\n input=f\"{model.model_path}/example/en.mp3\",\n cache={},\n language=\"auto\", # \"zh\", \"en\", \"yue\", \"ja\", \"ko\", \"nospeech\"\n use_itn=True,\n batch_size_s=60,\n merge_vad=True, #\n merge_length_s=15,\n)\ntext = rich_transcription_postprocess(res[0][\"text\"])\nprint(text)\n\n# zh\nres = model.generate(\n input=f\"{model.model_path}/example/zh.mp3\",\n cache={},\n language=\"auto\", # \"zh\", \"en\", \"yue\", \"ja\", \"ko\", \"nospeech\"\n use_itn=True,\n batch_size_s=60,\n merge_vad=True, #\n merge_length_s=15,\n)\ntext = rich_transcription_postprocess(res[0][\"text\"])\nprint(text)\n\n# yue\nres = model.generate(\n input=f\"{model.model_path}/example/yue.mp3\",\n cache={},\n language=\"auto\", # \"zh\", \"en\", \"yue\", \"ja\", \"ko\", \"nospeech\"\n use_itn=True,\n batch_size_s=60,\n merge_vad=True, #\n merge_length_s=15,\n)\ntext = rich_transcription_postprocess(res[0][\"text\"])\nprint(text)\n\n# ja\nres = model.generate(\n input=f\"{model.model_path}/example/ja.mp3\",\n cache={},\n language=\"auto\", # \"zh\", \"en\", \"yue\", \"ja\", \"ko\", \"nospeech\"\n use_itn=True,\n batch_size_s=60,\n merge_vad=True, #\n merge_length_s=15,\n)\ntext = rich_transcription_postprocess(res[0][\"text\"])\nprint(text)\n\n\n# ko\nres = model.generate(\n input=f\"{model.model_path}/example/ko.mp3\",\n cache={},\n language=\"auto\", # \"zh\", \"en\", \"yue\", \"ja\", \"ko\", \"nospeech\"\n use_itn=True,\n batch_size_s=60,\n merge_vad=True, #\n merge_length_s=15,\n)\ntext = rich_transcription_postprocess(res[0][\"text\"])\nprint(text)\n" }, { "path": "demo2.py", "content": "#!/usr/bin/env python3\n# -*- encoding: utf-8 -*-\n# Copyright FunASR (https://github.com/FunAudioLLM/SenseVoice). All Rights Reserved.\n# MIT License (https://opensource.org/licenses/MIT)\n\nfrom model import SenseVoiceSmall\nfrom funasr.utils.postprocess_utils import rich_transcription_postprocess\n\n\nmodel_dir = \"iic/SenseVoiceSmall\"\nm, kwargs = SenseVoiceSmall.from_pretrained(model=model_dir, device=\"cuda:0\")\nm.eval()\n\nres = m.inference(\n data_in=f\"{kwargs['model_path']}/example/en.mp3\",\n language=\"auto\", # \"zh\", \"en\", \"yue\", \"ja\", \"ko\", \"nospeech\"\n use_itn=False,\n ban_emo_unk=False,\n **kwargs,\n)\n\ntext = rich_transcription_postprocess(res[0][0][\"text\"])\nprint(text)\n\nres = m.inference(\n data_in=f\"{kwargs['model_path']}/example/en.mp3\",\n language=\"auto\", # \"zh\", \"en\", \"yue\", \"ja\", \"ko\", \"nospeech\"\n use_itn=False,\n ban_emo_unk=False,\n output_timestamp=True,\n **kwargs,\n)\n\ntimestamp = res[0][0][\"timestamp\"]\ntext = rich_transcription_postprocess(res[0][0][\"text\"])\nprint(text)\nprint(timestamp)\n" }, { "path": "demo_libtorch.py", "content": "#!/usr/bin/env python3\n# -*- encoding: utf-8 -*-\n# Copyright FunASR (https://github.com/FunAudioLLM/SenseVoice). All Rights Reserved.\n# MIT License (https://opensource.org/licenses/MIT)\n\nfrom pathlib import Path\nfrom funasr_torch import SenseVoiceSmall\nfrom funasr_torch.utils.postprocess_utils import rich_transcription_postprocess\n\n\nmodel_dir = \"iic/SenseVoiceSmall\"\n\nmodel = SenseVoiceSmall(model_dir, batch_size=10, device=\"cuda:0\")\n\nwav_or_scp = [\"{}/.cache/modelscope/hub/{}/example/en.mp3\".format(Path.home(), model_dir)]\n\nres = model(wav_or_scp, language=\"auto\", use_itn=True)\nprint([rich_transcription_postprocess(i) for i in res])\n" }, { "path": "demo_onnx.py", "content": "#!/usr/bin/env python3\n# -*- encoding: utf-8 -*-\n# Copyright FunASR (https://github.com/FunAudioLLM/SenseVoice). All Rights Reserved.\n# MIT License (https://opensource.org/licenses/MIT)\n\nfrom pathlib import Path\nfrom funasr_onnx import SenseVoiceSmall\nfrom funasr_onnx.utils.postprocess_utils import rich_transcription_postprocess\n\n\nmodel_dir = \"iic/SenseVoiceSmall\"\n\nmodel = SenseVoiceSmall(model_dir, batch_size=10, quantize=True)\n\n# inference\nwav_or_scp = [\"{}/.cache/modelscope/hub/{}/example/en.mp3\".format(Path.home(), model_dir)]\n\nres = model(wav_or_scp, language=\"auto\", textnorm=\"withitn\")\nprint([rich_transcription_postprocess(i) for i in res])\n" }, { "path": "docker-compose.yaml", "content": "name: sense\n\nservices:\n sensevoice:\n container_name: sensevoice\n build:\n context: .\n dockerfile: Dockerfile\n environment:\n MODELSCOPE_CACHE: /models\n SENSEVOICE_DEVICE: ${SENSEVOICE_DEVICE:-auto}\n volumes:\n - sensevoice-models:/models\n ports:\n - \"50000:50000\"\n restart: on-failure:5\n logging:\n options:\n max-size: \"5m\"\n max-file: \"3\"\n gpus: all\n\nvolumes:\n sensevoice-models:\n" }, { "path": "export.py", "content": "#!/usr/bin/env python3\n# -*- encoding: utf-8 -*-\n# Copyright FunASR (https://github.com/FunAudioLLM/SenseVoice). All Rights Reserved.\n# MIT License (https://opensource.org/licenses/MIT)\n\nimport os\nimport torch\nfrom model import SenseVoiceSmall\nfrom utils import export_utils\nfrom utils.model_bin import SenseVoiceSmallONNX\nfrom funasr.utils.postprocess_utils import rich_transcription_postprocess\n\nquantize = False\n\nmodel_dir = \"iic/SenseVoiceSmall\"\nmodel, kwargs = SenseVoiceSmall.from_pretrained(model=model_dir, device=\"cuda:0\")\n\nrebuilt_model = model.export(type=\"onnx\", quantize=False)\nmodel_path = kwargs.get(\"output_dir\", os.path.dirname(kwargs.get(\"init_param\")))\n\nmodel_file = os.path.join(model_path, \"model.onnx\")\nif quantize:\n model_file = os.path.join(model_path, \"model_quant.onnx\")\n\n# export model\nif not os.path.exists(model_file):\n with torch.no_grad():\n del kwargs['model']\n export_dir = export_utils.export(model=rebuilt_model, **kwargs)\n print(\"Export model onnx to {}\".format(model_file))\n \n# export model init\nmodel_bin = SenseVoiceSmallONNX(model_path)\n\n# build tokenizer\ntry:\n from funasr.tokenizer.sentencepiece_tokenizer import SentencepiecesTokenizer\n tokenizer = SentencepiecesTokenizer(bpemodel=os.path.join(model_path, \"chn_jpn_yue_eng_ko_spectok.bpe.model\"))\nexcept:\n tokenizer = None\n\n# inference\nwav_or_scp = \"/Users/shixian/Downloads/asr_example_hotword.wav\"\nlanguage_list = [0]\ntextnorm_list = [15]\nres = model_bin(wav_or_scp, language_list, textnorm_list, tokenizer=tokenizer)\nprint([rich_transcription_postprocess(i) for i in res])\n" }, { "path": "export_meta.py", "content": "#!/usr/bin/env python3\n# -*- encoding: utf-8 -*-\n# Copyright FunASR (https://github.com/alibaba-damo-academy/FunASR). All Rights Reserved.\n# MIT License (https://opensource.org/licenses/MIT)\n\nimport types\nimport torch\nfrom funasr.utils.torch_function import sequence_mask\n\n\ndef export_rebuild_model(model, **kwargs):\n model.device = kwargs.get(\"device\")\n model.make_pad_mask = sequence_mask(kwargs[\"max_seq_len\"], flip=False)\n model.forward = types.MethodType(export_forward, model)\n model.export_dummy_inputs = types.MethodType(export_dummy_inputs, model)\n model.export_input_names = types.MethodType(export_input_names, model)\n model.export_output_names = types.MethodType(export_output_names, model)\n model.export_dynamic_axes = types.MethodType(export_dynamic_axes, model)\n model.export_name = types.MethodType(export_name, model)\n return model\n\ndef export_forward(\n self,\n speech: torch.Tensor,\n speech_lengths: torch.Tensor,\n language: torch.Tensor,\n textnorm: torch.Tensor,\n **kwargs,\n):\n # speech = speech.to(device=\"cuda\")\n # speech_lengths = speech_lengths.to(device=\"cuda\")\n language_query = self.embed(language.to(speech.device)).unsqueeze(1)\n textnorm_query = self.embed(textnorm.to(speech.device)).unsqueeze(1)\n print(textnorm_query.shape, speech.shape)\n speech = torch.cat((textnorm_query, speech), dim=1)\n speech_lengths += 1\n \n event_emo_query = self.embed(torch.LongTensor([[1, 2]]).to(speech.device)).repeat(\n speech.size(0), 1, 1\n )\n input_query = torch.cat((language_query, event_emo_query), dim=1)\n speech = torch.cat((input_query, speech), dim=1)\n speech_lengths += 3\n \n encoder_out, encoder_out_lens = self.encoder(speech, speech_lengths)\n if isinstance(encoder_out, tuple):\n encoder_out = encoder_out[0]\n\n ctc_logits = self.ctc.ctc_lo(encoder_out)\n \n return ctc_logits, encoder_out_lens\n\ndef export_dummy_inputs(self):\n speech = torch.randn(2, 30, 560)\n speech_lengths = torch.tensor([6, 30], dtype=torch.int32)\n language = torch.tensor([0, 0], dtype=torch.int32)\n textnorm = torch.tensor([15, 15], dtype=torch.int32)\n return (speech, speech_lengths, language, textnorm)\n\ndef export_input_names(self):\n return [\"speech\", \"speech_lengths\", \"language\", \"textnorm\"]\n\ndef export_output_names(self):\n return [\"ctc_logits\", \"encoder_out_lens\"]\n\ndef export_dynamic_axes(self):\n return {\n \"speech\": {0: \"batch_size\", 1: \"feats_length\"},\n \"speech_lengths\": {0: \"batch_size\"},\n \"language\": {0: \"batch_size\"},\n \"textnorm\": {0: \"batch_size\"},\n \"ctc_logits\": {0: \"batch_size\", 1: \"logits_length\"},\n \"encoder_out_lens\": {0: \"batch_size\"},\n }\n\ndef export_name(self):\n return \"model.onnx\"\n\n" }, { "path": "finetune.sh", "content": "# Copyright FunASR (https://github.com/alibaba-damo-academy/FunASR). All Rights Reserved.\n# MIT License (https://opensource.org/licenses/MIT)\n\nworkspace=`pwd`\n\n# which gpu to train or finetune\nexport CUDA_VISIBLE_DEVICES=\"0,1\"\ngpu_num=$(echo $CUDA_VISIBLE_DEVICES | awk -F \",\" '{print NF}')\n\n# model_name from model_hub, or model_dir in local path\n\n## option 1, download model automatically\nmodel_name_or_model_dir=\"iic/SenseVoiceSmall\"\n\n## option 2, download model by git\n#local_path_root=${workspace}/modelscope_models\n#mkdir -p ${local_path_root}/${model_name_or_model_dir}\n#git clone https://www.modelscope.cn/${model_name_or_model_dir}.git ${local_path_root}/${model_name_or_model_dir}\n#model_name_or_model_dir=${local_path_root}/${model_name_or_model_dir}\n\n\n# data dir, which contains: train.json, val.json\ntrain_data=${workspace}/data/train_example.jsonl\nval_data=${workspace}/data/val_example.jsonl\n\n# exp output dir\noutput_dir=\"./outputs\"\nlog_file=\"${output_dir}/log.txt\"\n\ndeepspeed_config=${workspace}/deepspeed_conf/ds_stage1.json\n\nmkdir -p ${output_dir}\necho \"log_file: ${log_file}\"\n\nDISTRIBUTED_ARGS=\"\n --nnodes ${WORLD_SIZE:-1} \\\n --nproc_per_node $gpu_num \\\n --node_rank ${RANK:-0} \\\n --master_addr ${MASTER_ADDR:-127.0.0.1} \\\n --master_port ${MASTER_PORT:-26669}\n\"\n\necho $DISTRIBUTED_ARGS\n\n# funasr trainer path\nif [ -f `dirname $(which funasr)`/train_ds.py ]; then\n train_tool=`dirname $(which funasr)`/train_ds.py\nelif [ -f `dirname $(which funasr)`/../lib/python*/site-packages/funasr/bin/train_ds.py ]; then\n train_tool=`dirname $(which funasr)`/../lib/python*/site-packages/funasr/bin/train_ds.py\nelse\n echo \"Error: train_ds.py not found in funasr bin directory.\"\n exit 1\nfi\nABSOLUTE_PATH=$(cd $(dirname $train_tool); pwd)\ntrain_tool=${ABSOLUTE_PATH}/train_ds.py\necho \"Using funasr trainer: ${train_tool}\"\n\ntorchrun $DISTRIBUTED_ARGS \\\n${train_tool} \\\n++model=\"${model_name_or_model_dir}\" \\\n++trust_remote_code=true \\\n++train_data_set_list=\"${train_data}\" \\\n++valid_data_set_list=\"${val_data}\" \\\n++dataset_conf.data_split_num=1 \\\n++dataset_conf.batch_sampler=\"BatchSampler\" \\\n++dataset_conf.batch_size=6000 \\\n++dataset_conf.sort_size=1024 \\\n++dataset_conf.batch_type=\"token\" \\\n++dataset_conf.num_workers=4 \\\n++train_conf.max_epoch=50 \\\n++train_conf.log_interval=1 \\\n++train_conf.resume=true \\\n++train_conf.validate_interval=2000 \\\n++train_conf.save_checkpoint_interval=2000 \\\n++train_conf.keep_nbest_models=20 \\\n++train_conf.avg_nbest_model=10 \\\n++train_conf.use_deepspeed=false \\\n++train_conf.deepspeed_config=${deepspeed_config} \\\n++optim_conf.lr=0.0002 \\\n++output_dir=\"${output_dir}\" &> ${log_file}" }, { "path": "model.py", "content": "\nimport time\nimport torch\nfrom torch import nn\nimport torch.nn.functional as F\nfrom typing import Iterable, Optional\n\nfrom funasr.register import tables\nfrom funasr.models.ctc.ctc import CTC\nfrom funasr.utils.datadir_writer import DatadirWriter\nfrom funasr.models.paraformer.search import Hypothesis\nfrom funasr.train_utils.device_funcs import force_gatherable\nfrom funasr.losses.label_smoothing_loss import LabelSmoothingLoss\nfrom funasr.metrics.compute_acc import compute_accuracy, th_accuracy\nfrom funasr.utils.load_utils import load_audio_text_image_video, extract_fbank\nfrom utils.ctc_alignment import ctc_forced_align\n\nclass SinusoidalPositionEncoder(torch.nn.Module):\n \"\"\" \"\"\"\n\n def __int__(self, d_model=80, dropout_rate=0.1):\n pass\n\n def encode(\n self, positions: torch.Tensor = None, depth: int = None, dtype: torch.dtype = torch.float32\n ):\n batch_size = positions.size(0)\n positions = positions.type(dtype)\n device = positions.device\n log_timescale_increment = torch.log(torch.tensor([10000], dtype=dtype, device=device)) / (\n depth / 2 - 1\n )\n inv_timescales = torch.exp(\n torch.arange(depth / 2, device=device).type(dtype) * (-log_timescale_increment)\n )\n inv_timescales = torch.reshape(inv_timescales, [batch_size, -1])\n scaled_time = torch.reshape(positions, [1, -1, 1]) * torch.reshape(\n inv_timescales, [1, 1, -1]\n )\n encoding = torch.cat([torch.sin(scaled_time), torch.cos(scaled_time)], dim=2)\n return encoding.type(dtype)\n\n def forward(self, x):\n batch_size, timesteps, input_dim = x.size()\n positions = torch.arange(1, timesteps + 1, device=x.device)[None, :]\n position_encoding = self.encode(positions, input_dim, x.dtype).to(x.device)\n\n return x + position_encoding\n\n\nclass PositionwiseFeedForward(torch.nn.Module):\n \"\"\"Positionwise feed forward layer.\n\n Args:\n idim (int): Input dimenstion.\n hidden_units (int): The number of hidden units.\n dropout_rate (float): Dropout rate.\n\n \"\"\"\n\n def __init__(self, idim, hidden_units, dropout_rate, activation=torch.nn.ReLU()):\n \"\"\"Construct an PositionwiseFeedForward object.\"\"\"\n super(PositionwiseFeedForward, self).__init__()\n self.w_1 = torch.nn.Linear(idim, hidden_units)\n self.w_2 = torch.nn.Linear(hidden_units, idim)\n self.dropout = torch.nn.Dropout(dropout_rate)\n self.activation = activation\n\n def forward(self, x):\n \"\"\"Forward function.\"\"\"\n return self.w_2(self.dropout(self.activation(self.w_1(x))))\n\n\nclass MultiHeadedAttentionSANM(nn.Module):\n \"\"\"Multi-Head Attention layer.\n\n Args:\n n_head (int): The number of heads.\n n_feat (int): The number of features.\n dropout_rate (float): Dropout rate.\n\n \"\"\"\n\n def __init__(\n self,\n n_head,\n in_feat,\n n_feat,\n dropout_rate,\n kernel_size,\n sanm_shfit=0,\n lora_list=None,\n lora_rank=8,\n lora_alpha=16,\n lora_dropout=0.1,\n ):\n \"\"\"Construct an MultiHeadedAttention object.\"\"\"\n super().__init__()\n assert n_feat % n_head == 0\n # We assume d_v always equals d_k\n self.d_k = n_feat // n_head\n self.h = n_head\n # self.linear_q = nn.Linear(n_feat, n_feat)\n # self.linear_k = nn.Linear(n_feat, n_feat)\n # self.linear_v = nn.Linear(n_feat, n_feat)\n\n self.linear_out = nn.Linear(n_feat, n_feat)\n self.linear_q_k_v = nn.Linear(in_feat, n_feat * 3)\n self.attn = None\n self.dropout = nn.Dropout(p=dropout_rate)\n\n self.fsmn_block = nn.Conv1d(\n n_feat, n_feat, kernel_size, stride=1, padding=0, groups=n_feat, bias=False\n )\n # padding\n left_padding = (kernel_size - 1) // 2\n if sanm_shfit > 0:\n left_padding = left_padding + sanm_shfit\n right_padding = kernel_size - 1 - left_padding\n self.pad_fn = nn.ConstantPad1d((left_padding, right_padding), 0.0)\n\n def forward_fsmn(self, inputs, mask, mask_shfit_chunk=None):\n b, t, d = inputs.size()\n if mask is not None:\n mask = torch.reshape(mask, (b, -1, 1))\n if mask_shfit_chunk is not None:\n mask = mask * mask_shfit_chunk\n inputs = inputs * mask\n\n x = inputs.transpose(1, 2)\n x = self.pad_fn(x)\n x = self.fsmn_block(x)\n x = x.transpose(1, 2)\n x += inputs\n x = self.dropout(x)\n if mask is not None:\n x = x * mask\n return x\n\n def forward_qkv(self, x):\n \"\"\"Transform query, key and value.\n\n Args:\n query (torch.Tensor): Query tensor (#batch, time1, size).\n key (torch.Tensor): Key tensor (#batch, time2, size).\n value (torch.Tensor): Value tensor (#batch, time2, size).\n\n Returns:\n torch.Tensor: Transformed query tensor (#batch, n_head, time1, d_k).\n torch.Tensor: Transformed key tensor (#batch, n_head, time2, d_k).\n torch.Tensor: Transformed value tensor (#batch, n_head, time2, d_k).\n\n \"\"\"\n b, t, d = x.size()\n q_k_v = self.linear_q_k_v(x)\n q, k, v = torch.split(q_k_v, int(self.h * self.d_k), dim=-1)\n q_h = torch.reshape(q, (b, t, self.h, self.d_k)).transpose(\n 1, 2\n ) # (batch, head, time1, d_k)\n k_h = torch.reshape(k, (b, t, self.h, self.d_k)).transpose(\n 1, 2\n ) # (batch, head, time2, d_k)\n v_h = torch.reshape(v, (b, t, self.h, self.d_k)).transpose(\n 1, 2\n ) # (batch, head, time2, d_k)\n\n return q_h, k_h, v_h, v\n\n def forward_attention(self, value, scores, mask, mask_att_chunk_encoder=None):\n \"\"\"Compute attention context vector.\n\n Args:\n value (torch.Tensor): Transformed value (#batch, n_head, time2, d_k).\n scores (torch.Tensor): Attention score (#batch, n_head, time1, time2).\n mask (torch.Tensor): Mask (#batch, 1, time2) or (#batch, time1, time2).\n\n Returns:\n torch.Tensor: Transformed value (#batch, time1, d_model)\n weighted by the attention score (#batch, time1, time2).\n\n \"\"\"\n n_batch = value.size(0)\n if mask is not None:\n if mask_att_chunk_encoder is not None:\n mask = mask * mask_att_chunk_encoder\n\n mask = mask.unsqueeze(1).eq(0) # (batch, 1, *, time2)\n\n min_value = -float(\n \"inf\"\n ) # float(numpy.finfo(torch.tensor(0, dtype=scores.dtype).numpy().dtype).min)\n scores = scores.masked_fill(mask, min_value)\n attn = torch.softmax(scores, dim=-1).masked_fill(\n mask, 0.0\n ) # (batch, head, time1, time2)\n else:\n attn = torch.softmax(scores, dim=-1) # (batch, head, time1, time2)\n\n p_attn = self.dropout(attn)\n x = torch.matmul(p_attn, value) # (batch, head, time1, d_k)\n x = (\n x.transpose(1, 2).contiguous().view(n_batch, -1, self.h * self.d_k)\n ) # (batch, time1, d_model)\n\n return self.linear_out(x) # (batch, time1, d_model)\n\n def forward(self, x, mask, mask_shfit_chunk=None, mask_att_chunk_encoder=None):\n \"\"\"Compute scaled dot product attention.\n\n Args:\n query (torch.Tensor): Query tensor (#batch, time1, size).\n key (torch.Tensor): Key tensor (#batch, time2, size).\n value (torch.Tensor): Value tensor (#batch, time2, size).\n mask (torch.Tensor): Mask tensor (#batch, 1, time2) or\n (#batch, time1, time2).\n\n Returns:\n torch.Tensor: Output tensor (#batch, time1, d_model).\n\n \"\"\"\n q_h, k_h, v_h, v = self.forward_qkv(x)\n fsmn_memory = self.forward_fsmn(v, mask, mask_shfit_chunk)\n q_h = q_h * self.d_k ** (-0.5)\n scores = torch.matmul(q_h, k_h.transpose(-2, -1))\n att_outs = self.forward_attention(v_h, scores, mask, mask_att_chunk_encoder)\n return att_outs + fsmn_memory\n\n def forward_chunk(self, x, cache=None, chunk_size=None, look_back=0):\n \"\"\"Compute scaled dot product attention.\n\n Args:\n query (torch.Tensor): Query tensor (#batch, time1, size).\n key (torch.Tensor): Key tensor (#batch, time2, size).\n value (torch.Tensor): Value tensor (#batch, time2, size).\n mask (torch.Tensor): Mask tensor (#batch, 1, time2) or\n (#batch, time1, time2).\n\n Returns:\n torch.Tensor: Output tensor (#batch, time1, d_model).\n\n \"\"\"\n q_h, k_h, v_h, v = self.forward_qkv(x)\n if chunk_size is not None and look_back > 0 or look_back == -1:\n if cache is not None:\n k_h_stride = k_h[:, :, : -(chunk_size[2]), :]\n v_h_stride = v_h[:, :, : -(chunk_size[2]), :]\n k_h = torch.cat((cache[\"k\"], k_h), dim=2)\n v_h = torch.cat((cache[\"v\"], v_h), dim=2)\n\n cache[\"k\"] = torch.cat((cache[\"k\"], k_h_stride), dim=2)\n cache[\"v\"] = torch.cat((cache[\"v\"], v_h_stride), dim=2)\n if look_back != -1:\n cache[\"k\"] = cache[\"k\"][:, :, -(look_back * chunk_size[1]) :, :]\n cache[\"v\"] = cache[\"v\"][:, :, -(look_back * chunk_size[1]) :, :]\n else:\n cache_tmp = {\n \"k\": k_h[:, :, : -(chunk_size[2]), :],\n \"v\": v_h[:, :, : -(chunk_size[2]), :],\n }\n cache = cache_tmp\n fsmn_memory = self.forward_fsmn(v, None)\n q_h = q_h * self.d_k ** (-0.5)\n scores = torch.matmul(q_h, k_h.transpose(-2, -1))\n att_outs = self.forward_attention(v_h, scores, None)\n return att_outs + fsmn_memory, cache\n\n\nclass LayerNorm(nn.LayerNorm):\n def __init__(self, *args, **kwargs):\n super().__init__(*args, **kwargs)\n\n def forward(self, input):\n output = F.layer_norm(\n input.float(),\n self.normalized_shape,\n self.weight.float() if self.weight is not None else None,\n self.bias.float() if self.bias is not None else None,\n self.eps,\n )\n return output.type_as(input)\n\n\ndef sequence_mask(lengths, maxlen=None, dtype=torch.float32, device=None):\n if maxlen is None:\n maxlen = lengths.max()\n row_vector = torch.arange(0, maxlen, 1).to(lengths.device)\n matrix = torch.unsqueeze(lengths, dim=-1)\n mask = row_vector < matrix\n mask = mask.detach()\n\n return mask.type(dtype).to(device) if device is not None else mask.type(dtype)\n\n\nclass EncoderLayerSANM(nn.Module):\n def __init__(\n self,\n in_size,\n size,\n self_attn,\n feed_forward,\n dropout_rate,\n normalize_before=True,\n concat_after=False,\n stochastic_depth_rate=0.0,\n ):\n \"\"\"Construct an EncoderLayer object.\"\"\"\n super(EncoderLayerSANM, self).__init__()\n self.self_attn = self_attn\n self.feed_forward = feed_forward\n self.norm1 = LayerNorm(in_size)\n self.norm2 = LayerNorm(size)\n self.dropout = nn.Dropout(dropout_rate)\n self.in_size = in_size\n self.size = size\n self.normalize_before = normalize_before\n self.concat_after = concat_after\n if self.concat_after:\n self.concat_linear = nn.Linear(size + size, size)\n self.stochastic_depth_rate = stochastic_depth_rate\n self.dropout_rate = dropout_rate\n\n def forward(self, x, mask, cache=None, mask_shfit_chunk=None, mask_att_chunk_encoder=None):\n \"\"\"Compute encoded features.\n\n Args:\n x_input (torch.Tensor): Input tensor (#batch, time, size).\n mask (torch.Tensor): Mask tensor for the input (#batch, time).\n cache (torch.Tensor): Cache tensor of the input (#batch, time - 1, size).\n\n Returns:\n torch.Tensor: Output tensor (#batch, time, size).\n torch.Tensor: Mask tensor (#batch, time).\n\n \"\"\"\n skip_layer = False\n # with stochastic depth, residual connection `x + f(x)` becomes\n # `x <- x + 1 / (1 - p) * f(x)` at training time.\n stoch_layer_coeff = 1.0\n if self.training and self.stochastic_depth_rate > 0:\n skip_layer = torch.rand(1).item() < self.stochastic_depth_rate\n stoch_layer_coeff = 1.0 / (1 - self.stochastic_depth_rate)\n\n if skip_layer:\n if cache is not None:\n x = torch.cat([cache, x], dim=1)\n return x, mask\n\n residual = x\n if self.normalize_before:\n x = self.norm1(x)\n\n if self.concat_after:\n x_concat = torch.cat(\n (\n x,\n self.self_attn(\n x,\n mask,\n mask_shfit_chunk=mask_shfit_chunk,\n mask_att_chunk_encoder=mask_att_chunk_encoder,\n ),\n ),\n dim=-1,\n )\n if self.in_size == self.size:\n x = residual + stoch_layer_coeff * self.concat_linear(x_concat)\n else:\n x = stoch_layer_coeff * self.concat_linear(x_concat)\n else:\n if self.in_size == self.size:\n x = residual + stoch_layer_coeff * self.dropout(\n self.self_attn(\n x,\n mask,\n mask_shfit_chunk=mask_shfit_chunk,\n mask_att_chunk_encoder=mask_att_chunk_encoder,\n )\n )\n else:\n x = stoch_layer_coeff * self.dropout(\n self.self_attn(\n x,\n mask,\n mask_shfit_chunk=mask_shfit_chunk,\n mask_att_chunk_encoder=mask_att_chunk_encoder,\n )\n )\n if not self.normalize_before:\n x = self.norm1(x)\n\n residual = x\n if self.normalize_before:\n x = self.norm2(x)\n x = residual + stoch_layer_coeff * self.dropout(self.feed_forward(x))\n if not self.normalize_before:\n x = self.norm2(x)\n\n return x, mask, cache, mask_shfit_chunk, mask_att_chunk_encoder\n\n def forward_chunk(self, x, cache=None, chunk_size=None, look_back=0):\n \"\"\"Compute encoded features.\n\n Args:\n x_input (torch.Tensor): Input tensor (#batch, time, size).\n mask (torch.Tensor): Mask tensor for the input (#batch, time).\n cache (torch.Tensor): Cache tensor of the input (#batch, time - 1, size).\n\n Returns:\n torch.Tensor: Output tensor (#batch, time, size).\n torch.Tensor: Mask tensor (#batch, time).\n\n \"\"\"\n\n residual = x\n if self.normalize_before:\n x = self.norm1(x)\n\n if self.in_size == self.size:\n attn, cache = self.self_attn.forward_chunk(x, cache, chunk_size, look_back)\n x = residual + attn\n else:\n x, cache = self.self_attn.forward_chunk(x, cache, chunk_size, look_back)\n\n if not self.normalize_before:\n x = self.norm1(x)\n\n residual = x\n if self.normalize_before:\n x = self.norm2(x)\n x = residual + self.feed_forward(x)\n if not self.normalize_before:\n x = self.norm2(x)\n\n return x, cache\n\n\n@tables.register(\"encoder_classes\", \"SenseVoiceEncoderSmall\")\nclass SenseVoiceEncoderSmall(nn.Module):\n \"\"\"\n Author: Speech Lab of DAMO Academy, Alibaba Group\n SCAMA: Streaming chunk-aware multihead attention for online end-to-end speech recognition\n https://arxiv.org/abs/2006.01713\n \"\"\"\n\n def __init__(\n self,\n input_size: int,\n output_size: int = 256,\n attention_heads: int = 4,\n linear_units: int = 2048,\n num_blocks: int = 6,\n tp_blocks: int = 0,\n dropout_rate: float = 0.1,\n positional_dropout_rate: float = 0.1,\n attention_dropout_rate: float = 0.0,\n stochastic_depth_rate: float = 0.0,\n input_layer: Optional[str] = \"conv2d\",\n pos_enc_class=SinusoidalPositionEncoder,\n normalize_before: bool = True,\n concat_after: bool = False,\n positionwise_layer_type: str = \"linear\",\n positionwise_conv_kernel_size: int = 1,\n padding_idx: int = -1,\n kernel_size: int = 11,\n sanm_shfit: int = 0,\n selfattention_layer_type: str = \"sanm\",\n **kwargs,\n ):\n super().__init__()\n self._output_size = output_size\n\n self.embed = SinusoidalPositionEncoder()\n\n self.normalize_before = normalize_before\n\n positionwise_layer = PositionwiseFeedForward\n positionwise_layer_args = (\n output_size,\n linear_units,\n dropout_rate,\n )\n\n encoder_selfattn_layer = MultiHeadedAttentionSANM\n encoder_selfattn_layer_args0 = (\n attention_heads,\n input_size,\n output_size,\n attention_dropout_rate,\n kernel_size,\n sanm_shfit,\n )\n encoder_selfattn_layer_args = (\n attention_heads,\n output_size,\n output_size,\n attention_dropout_rate,\n kernel_size,\n sanm_shfit,\n )\n\n self.encoders0 = nn.ModuleList(\n [\n EncoderLayerSANM(\n input_size,\n output_size,\n encoder_selfattn_layer(*encoder_selfattn_layer_args0),\n positionwise_layer(*positionwise_layer_args),\n dropout_rate,\n )\n for i in range(1)\n ]\n )\n self.encoders = nn.ModuleList(\n [\n EncoderLayerSANM(\n output_size,\n output_size,\n encoder_selfattn_layer(*encoder_selfattn_layer_args),\n positionwise_layer(*positionwise_layer_args),\n dropout_rate,\n )\n for i in range(num_blocks - 1)\n ]\n )\n\n self.tp_encoders = nn.ModuleList(\n [\n EncoderLayerSANM(\n output_size,\n output_size,\n encoder_selfattn_layer(*encoder_selfattn_layer_args),\n positionwise_layer(*positionwise_layer_args),\n dropout_rate,\n )\n for i in range(tp_blocks)\n ]\n )\n\n self.after_norm = LayerNorm(output_size)\n\n self.tp_norm = LayerNorm(output_size)\n\n def output_size(self) -> int:\n return self._output_size\n\n def forward(\n self,\n xs_pad: torch.Tensor,\n ilens: torch.Tensor,\n ):\n \"\"\"Embed positions in tensor.\"\"\"\n masks = sequence_mask(ilens, device=ilens.device)[:, None, :]\n\n xs_pad *= self.output_size() ** 0.5\n\n xs_pad = self.embed(xs_pad)\n\n # forward encoder1\n for layer_idx, encoder_layer in enumerate(self.encoders0):\n encoder_outs = encoder_layer(xs_pad, masks)\n xs_pad, masks = encoder_outs[0], encoder_outs[1]\n\n for layer_idx, encoder_layer in enumerate(self.encoders):\n encoder_outs = encoder_layer(xs_pad, masks)\n xs_pad, masks = encoder_outs[0], encoder_outs[1]\n\n xs_pad = self.after_norm(xs_pad)\n\n # forward encoder2\n olens = masks.squeeze(1).sum(1).int()\n\n for layer_idx, encoder_layer in enumerate(self.tp_encoders):\n encoder_outs = encoder_layer(xs_pad, masks)\n xs_pad, masks = encoder_outs[0], encoder_outs[1]\n\n xs_pad = self.tp_norm(xs_pad)\n return xs_pad, olens\n\n\n@tables.register(\"model_classes\", \"SenseVoiceSmall\")\nclass SenseVoiceSmall(nn.Module):\n \"\"\"CTC-attention hybrid Encoder-Decoder model\"\"\"\n\n def __init__(\n self,\n specaug: str = None,\n specaug_conf: dict = None,\n normalize: str = None,\n normalize_conf: dict = None,\n encoder: str = None,\n encoder_conf: dict = None,\n ctc_conf: dict = None,\n input_size: int = 80,\n vocab_size: int = -1,\n ignore_id: int = -1,\n blank_id: int = 0,\n sos: int = 1,\n eos: int = 2,\n length_normalized_loss: bool = False,\n **kwargs,\n ):\n\n super().__init__()\n\n if specaug is not None:\n specaug_class = tables.specaug_classes.get(specaug)\n specaug = specaug_class(**specaug_conf)\n if normalize is not None:\n normalize_class = tables.normalize_classes.get(normalize)\n normalize = normalize_class(**normalize_conf)\n encoder_class = tables.encoder_classes.get(encoder)\n encoder = encoder_class(input_size=input_size, **encoder_conf)\n encoder_output_size = encoder.output_size()\n\n if ctc_conf is None:\n ctc_conf = {}\n ctc = CTC(odim=vocab_size, encoder_output_size=encoder_output_size, **ctc_conf)\n\n self.blank_id = blank_id\n self.sos = sos if sos is not None else vocab_size - 1\n self.eos = eos if eos is not None else vocab_size - 1\n self.vocab_size = vocab_size\n self.ignore_id = ignore_id\n self.specaug = specaug\n self.normalize = normalize\n self.encoder = encoder\n self.error_calculator = None\n\n self.ctc = ctc\n\n self.length_normalized_loss = length_normalized_loss\n self.encoder_output_size = encoder_output_size\n\n self.lid_dict = {\"auto\": 0, \"zh\": 3, \"en\": 4, \"yue\": 7, \"ja\": 11, \"ko\": 12, \"nospeech\": 13}\n self.lid_int_dict = {24884: 3, 24885: 4, 24888: 7, 24892: 11, 24896: 12, 24992: 13}\n self.textnorm_dict = {\"withitn\": 14, \"woitn\": 15}\n self.textnorm_int_dict = {25016: 14, 25017: 15}\n self.embed = torch.nn.Embedding(7 + len(self.lid_dict) + len(self.textnorm_dict), input_size)\n self.emo_dict = {\"unk\": 25009, \"happy\": 25001, \"sad\": 25002, \"angry\": 25003, \"neutral\": 25004}\n \n self.criterion_att = LabelSmoothingLoss(\n size=self.vocab_size,\n padding_idx=self.ignore_id,\n smoothing=kwargs.get(\"lsm_weight\", 0.0),\n normalize_length=self.length_normalized_loss,\n )\n \n @staticmethod\n def from_pretrained(model:str=None, **kwargs):\n from funasr import AutoModel\n model, kwargs = AutoModel.build_model(model=model, trust_remote_code=True, **kwargs)\n \n return model, kwargs\n\n def forward(\n self,\n speech: torch.Tensor,\n speech_lengths: torch.Tensor,\n text: torch.Tensor,\n text_lengths: torch.Tensor,\n **kwargs,\n ):\n \"\"\"Encoder + Decoder + Calc loss\n Args:\n speech: (Batch, Length, ...)\n speech_lengths: (Batch, )\n text: (Batch, Length)\n text_lengths: (Batch,)\n \"\"\"\n # import pdb;\n # pdb.set_trace()\n if len(text_lengths.size()) > 1:\n text_lengths = text_lengths[:, 0]\n if len(speech_lengths.size()) > 1:\n speech_lengths = speech_lengths[:, 0]\n\n batch_size = speech.shape[0]\n\n # 1. Encoder\n encoder_out, encoder_out_lens = self.encode(speech, speech_lengths, text)\n\n loss_ctc, cer_ctc = None, None\n loss_rich, acc_rich = None, None\n stats = dict()\n\n loss_ctc, cer_ctc = self._calc_ctc_loss(\n encoder_out[:, 4:, :], encoder_out_lens - 4, text[:, 4:], text_lengths - 4\n )\n\n loss_rich, acc_rich = self._calc_rich_ce_loss(\n encoder_out[:, :4, :], text[:, :4]\n )\n\n loss = loss_ctc + loss_rich\n # Collect total loss stats\n stats[\"loss_ctc\"] = torch.clone(loss_ctc.detach()) if loss_ctc is not None else None\n stats[\"loss_rich\"] = torch.clone(loss_rich.detach()) if loss_rich is not None else None\n stats[\"loss\"] = torch.clone(loss.detach()) if loss is not None else None\n stats[\"acc_rich\"] = acc_rich\n\n # force_gatherable: to-device and to-tensor if scalar for DataParallel\n if self.length_normalized_loss:\n batch_size = int((text_lengths + 1).sum())\n loss, stats, weight = force_gatherable((loss, stats, batch_size), loss.device)\n return loss, stats, weight\n\n def encode(\n self,\n speech: torch.Tensor,\n speech_lengths: torch.Tensor,\n text: torch.Tensor,\n **kwargs,\n ):\n \"\"\"Frontend + Encoder. Note that this method is used by asr_inference.py\n Args:\n speech: (Batch, Length, ...)\n speech_lengths: (Batch, )\n ind: int\n \"\"\"\n\n # Data augmentation\n if self.specaug is not None and self.training:\n speech, speech_lengths = self.specaug(speech, speech_lengths)\n\n # Normalization for feature: e.g. Global-CMVN, Utterance-CMVN\n if self.normalize is not None:\n speech, speech_lengths = self.normalize(speech, speech_lengths)\n\n\n lids = torch.LongTensor([[self.lid_int_dict[int(lid)] if torch.rand(1) > 0.2 and int(lid) in self.lid_int_dict else 0 ] for lid in text[:, 0]]).to(speech.device)\n language_query = self.embed(lids)\n \n styles = torch.LongTensor([[self.textnorm_int_dict[int(style)]] for style in text[:, 3]]).to(speech.device)\n style_query = self.embed(styles)\n speech = torch.cat((style_query, speech), dim=1)\n speech_lengths += 1\n\n event_emo_query = self.embed(torch.LongTensor([[1, 2]]).to(speech.device)).repeat(speech.size(0), 1, 1)\n input_query = torch.cat((language_query, event_emo_query), dim=1)\n speech = torch.cat((input_query, speech), dim=1)\n speech_lengths += 3\n\n encoder_out, encoder_out_lens = self.encoder(speech, speech_lengths)\n\n return encoder_out, encoder_out_lens\n\n def _calc_ctc_loss(\n self,\n encoder_out: torch.Tensor,\n encoder_out_lens: torch.Tensor,\n ys_pad: torch.Tensor,\n ys_pad_lens: torch.Tensor,\n ):\n # Calc CTC loss\n loss_ctc = self.ctc(encoder_out, encoder_out_lens, ys_pad, ys_pad_lens)\n\n # Calc CER using CTC\n cer_ctc = None\n if not self.training and self.error_calculator is not None:\n ys_hat = self.ctc.argmax(encoder_out).data\n cer_ctc = self.error_calculator(ys_hat.cpu(), ys_pad.cpu(), is_ctc=True)\n return loss_ctc, cer_ctc\n\n def _calc_rich_ce_loss(\n self,\n encoder_out: torch.Tensor,\n ys_pad: torch.Tensor,\n ):\n decoder_out = self.ctc.ctc_lo(encoder_out)\n # 2. Compute attention loss\n loss_rich = self.criterion_att(decoder_out, ys_pad.contiguous())\n acc_rich = th_accuracy(\n decoder_out.view(-1, self.vocab_size),\n ys_pad.contiguous(),\n ignore_label=self.ignore_id,\n )\n\n return loss_rich, acc_rich\n\n\n def inference(\n self,\n data_in,\n data_lengths=None,\n key: list = [\"wav_file_tmp_name\"],\n tokenizer=None,\n frontend=None,\n **kwargs,\n ):\n\n\n meta_data = {}\n if (\n isinstance(data_in, torch.Tensor) and kwargs.get(\"data_type\", \"sound\") == \"fbank\"\n ): # fbank\n speech, speech_lengths = data_in, data_lengths\n if len(speech.shape) < 3:\n speech = speech[None, :, :]\n if speech_lengths is None:\n speech_lengths = speech.shape[1]\n else:\n # extract fbank feats\n time1 = time.perf_counter()\n audio_sample_list = load_audio_text_image_video(\n data_in,\n fs=frontend.fs,\n audio_fs=kwargs.get(\"fs\", 16000),\n data_type=kwargs.get(\"data_type\", \"sound\"),\n tokenizer=tokenizer,\n )\n time2 = time.perf_counter()\n meta_data[\"load_data\"] = f\"{time2 - time1:0.3f}\"\n speech, speech_lengths = extract_fbank(\n audio_sample_list, data_type=kwargs.get(\"data_type\", \"sound\"), frontend=frontend\n )\n time3 = time.perf_counter()\n meta_data[\"extract_feat\"] = f\"{time3 - time2:0.3f}\"\n meta_data[\"batch_data_time\"] = (\n speech_lengths.sum().item() * frontend.frame_shift * frontend.lfr_n / 1000\n )\n\n speech = speech.to(device=kwargs[\"device\"])\n speech_lengths = speech_lengths.to(device=kwargs[\"device\"])\n\n language = kwargs.get(\"language\", \"auto\")\n language_query = self.embed(\n torch.LongTensor(\n [[self.lid_dict[language] if language in self.lid_dict else 0]]\n ).to(speech.device)\n ).repeat(speech.size(0), 1, 1)\n \n use_itn = kwargs.get(\"use_itn\", False)\n output_timestamp = kwargs.get(\"output_timestamp\", False)\n\n textnorm = kwargs.get(\"text_norm\", None)\n if textnorm is None:\n textnorm = \"withitn\" if use_itn else \"woitn\"\n textnorm_query = self.embed(\n torch.LongTensor([[self.textnorm_dict[textnorm]]]).to(speech.device)\n ).repeat(speech.size(0), 1, 1)\n speech = torch.cat((textnorm_query, speech), dim=1)\n speech_lengths += 1\n\n event_emo_query = self.embed(torch.LongTensor([[1, 2]]).to(speech.device)).repeat(\n speech.size(0), 1, 1\n )\n input_query = torch.cat((language_query, event_emo_query), dim=1)\n speech = torch.cat((input_query, speech), dim=1)\n speech_lengths += 3\n\n # Encoder\n encoder_out, encoder_out_lens = self.encoder(speech, speech_lengths)\n if isinstance(encoder_out, tuple):\n encoder_out = encoder_out[0]\n\n # c. Passed the encoder result and the beam search\n ctc_logits = self.ctc.log_softmax(encoder_out)\n if kwargs.get(\"ban_emo_unk\", False):\n ctc_logits[:, :, self.emo_dict[\"unk\"]] = -float(\"inf\")\n\n results = []\n b, n, d = encoder_out.size()\n if isinstance(key[0], (list, tuple)):\n key = key[0]\n if len(key) < b:\n key = key * b\n for i in range(b):\n x = ctc_logits[i, : encoder_out_lens[i].item(), :]\n yseq = x.argmax(dim=-1)\n yseq = torch.unique_consecutive(yseq, dim=-1)\n\n ibest_writer = None\n if kwargs.get(\"output_dir\") is not None:\n if not hasattr(self, \"writer\"):\n self.writer = DatadirWriter(kwargs.get(\"output_dir\"))\n ibest_writer = self.writer[f\"1best_recog\"]\n\n mask = yseq != self.blank_id\n token_int = yseq[mask].tolist()\n\n # Change integer-ids to tokens\n text = tokenizer.decode(token_int)\n if ibest_writer is not None:\n ibest_writer[\"text\"][key[i]] = text\n\n if output_timestamp:\n from itertools import groupby\n timestamp = []\n tokens = tokenizer.text2tokens(text)[4:]\n\n logits_speech = self.ctc.softmax(encoder_out)[i, 4:encoder_out_lens[i].item(), :]\n\n pred = logits_speech.argmax(-1).cpu()\n logits_speech[pred==self.blank_id, self.blank_id] = 0\n\n align = ctc_forced_align(\n logits_speech.unsqueeze(0).float(),\n torch.Tensor(token_int[4:]).unsqueeze(0).long().to(logits_speech.device),\n (encoder_out_lens-4).long()[i],\n torch.tensor(len(token_int)-4).unsqueeze(0).long().to(logits_speech.device),\n ignore_id=self.ignore_id,\n )\n\n pred = groupby(align[0, :encoder_out_lens[0]])\n _start = 0\n token_id = 0\n ts_max = encoder_out_lens[i] - 4\n for pred_token, pred_frame in pred:\n _end = _start + len(list(pred_frame))\n if pred_token != 0 and token_id < len(tokens):\n ts_left = max((_start*60-30)/1000, 0)\n ts_right = min((_end*60-30)/1000, (ts_max*60-30)/1000)\n timestamp.append([tokens[token_id], ts_left, ts_right])\n token_id += 1\n _start = _end\n\n result_i = {\"key\": key[i], \"text\": text, \"timestamp\": timestamp}\n results.append(result_i)\n else:\n result_i = {\"key\": key[i], \"text\": text}\n results.append(result_i)\n return results, meta_data\n\n def export(self, **kwargs):\n from export_meta import export_rebuild_model\n\n if \"max_seq_len\" not in kwargs:\n kwargs[\"max_seq_len\"] = 512\n models = export_rebuild_model(model=self, **kwargs)\n return models\n" }, { "path": "requirements.txt", "content": "torch<=2.3\ntorchaudio\nmodelscope\nhuggingface\nhuggingface_hub\nfunasr>=1.1.3\nnumpy<=1.26.4\ngradio\nfastapi>=0.111.1\n" }, { "path": "utils/__init__.py", "content": "" }, { "path": "utils/ctc_alignment.py", "content": "import torch\n\ndef ctc_forced_align(\n log_probs: torch.Tensor,\n targets: torch.Tensor,\n input_lengths: torch.Tensor,\n target_lengths: torch.Tensor,\n blank: int = 0,\n ignore_id: int = -1,\n) -> torch.Tensor:\n \"\"\"Align a CTC label sequence to an emission.\n\n Args:\n log_probs (Tensor): log probability of CTC emission output.\n Tensor of shape `(B, T, C)`. where `B` is the batch size, `T` is the input length,\n `C` is the number of characters in alphabet including blank.\n targets (Tensor): Target sequence. Tensor of shape `(B, L)`,\n where `L` is the target length.\n input_lengths (Tensor):\n Lengths of the inputs (max value must each be <= `T`). 1-D Tensor of shape `(B,)`.\n target_lengths (Tensor):\n Lengths of the targets. 1-D Tensor of shape `(B,)`.\n blank_id (int, optional): The index of blank symbol in CTC emission. (Default: 0)\n ignore_id (int, optional): The index of ignore symbol in CTC emission. (Default: -1)\n \"\"\"\n targets[targets == ignore_id] = blank\n\n batch_size, input_time_size, _ = log_probs.size()\n bsz_indices = torch.arange(batch_size, device=input_lengths.device)\n\n _t_a_r_g_e_t_s_ = torch.cat(\n (\n torch.stack((torch.full_like(targets, blank), targets), dim=-1).flatten(start_dim=1),\n torch.full_like(targets[:, :1], blank),\n ),\n dim=-1,\n )\n diff_labels = torch.cat(\n (\n torch.as_tensor([[False, False]], device=targets.device).expand(batch_size, -1),\n _t_a_r_g_e_t_s_[:, 2:] != _t_a_r_g_e_t_s_[:, :-2],\n ),\n dim=1,\n )\n\n neg_inf = torch.tensor(float(\"-inf\"), device=log_probs.device, dtype=log_probs.dtype)\n padding_num = 2\n padded_t = padding_num + _t_a_r_g_e_t_s_.size(-1)\n best_score = torch.full((batch_size, padded_t), neg_inf, device=log_probs.device, dtype=log_probs.dtype)\n best_score[:, padding_num + 0] = log_probs[:, 0, blank]\n best_score[:, padding_num + 1] = log_probs[bsz_indices, 0, _t_a_r_g_e_t_s_[:, 1]]\n\n backpointers = torch.zeros((batch_size, input_time_size, padded_t), device=log_probs.device, dtype=targets.dtype)\n\n for t in range(1, input_time_size):\n prev = torch.stack(\n (best_score[:, 2:], best_score[:, 1:-1], torch.where(diff_labels, best_score[:, :-2], neg_inf))\n )\n prev_max_value, prev_max_idx = prev.max(dim=0)\n best_score[:, padding_num:] = log_probs[:, t].gather(-1, _t_a_r_g_e_t_s_) + prev_max_value\n backpointers[:, t, padding_num:] = prev_max_idx\n\n l1l2 = best_score.gather(\n -1, torch.stack((padding_num + target_lengths * 2 - 1, padding_num + target_lengths * 2), dim=-1)\n )\n\n path = torch.zeros((batch_size, input_time_size), device=best_score.device, dtype=torch.long)\n path[bsz_indices, input_lengths - 1] = padding_num + target_lengths * 2 - 1 + l1l2.argmax(dim=-1)\n\n for t in range(input_time_size - 1, 0, -1):\n target_indices = path[:, t]\n prev_max_idx = backpointers[bsz_indices, t, target_indices]\n path[:, t - 1] += target_indices - prev_max_idx\n\n alignments = _t_a_r_g_e_t_s_.gather(dim=-1, index=(path - padding_num).clamp(min=0))\n return alignments\n" }, { "path": "utils/export_utils.py", "content": "import os\nimport torch\n\n\ndef export(\n model, quantize: bool = False, opset_version: int = 14, type=\"onnx\", **kwargs\n):\n model_scripts = model.export(**kwargs)\n export_dir = kwargs.get(\"output_dir\", os.path.dirname(kwargs.get(\"init_param\")))\n os.makedirs(export_dir, exist_ok=True)\n\n if not isinstance(model_scripts, (list, tuple)):\n model_scripts = (model_scripts,)\n for m in model_scripts:\n m.eval()\n if type == \"onnx\":\n _onnx(\n m,\n quantize=quantize,\n opset_version=opset_version,\n export_dir=export_dir,\n **kwargs,\n )\n print(\"output dir: {}\".format(export_dir))\n\n return export_dir\n\n\ndef _onnx(\n model,\n quantize: bool = False,\n opset_version: int = 14,\n export_dir: str = None,\n **kwargs,\n):\n\n dummy_input = model.export_dummy_inputs()\n\n verbose = kwargs.get(\"verbose\", False)\n\n export_name = model.export_name()\n model_path = os.path.join(export_dir, export_name)\n torch.onnx.export(\n model,\n dummy_input,\n model_path,\n verbose=verbose,\n opset_version=opset_version,\n input_names=model.export_input_names(),\n output_names=model.export_output_names(),\n dynamic_axes=model.export_dynamic_axes(),\n )\n\n if quantize:\n from onnxruntime.quantization import QuantType, quantize_dynamic\n import onnx\n\n quant_model_path = model_path.replace(\".onnx\", \"_quant.onnx\")\n if not os.path.exists(quant_model_path):\n onnx_model = onnx.load(model_path)\n nodes = [n.name for n in onnx_model.graph.node]\n nodes_to_exclude = [\n m for m in nodes if \"output\" in m or \"bias_encoder\" in m or \"bias_decoder\" in m\n ]\n quantize_dynamic(\n model_input=model_path,\n model_output=quant_model_path,\n op_types_to_quantize=[\"MatMul\"],\n per_channel=True,\n reduce_range=False,\n weight_type=QuantType.QUInt8,\n nodes_to_exclude=nodes_to_exclude,\n )\n" }, { "path": "utils/frontend.py", "content": "# -*- encoding: utf-8 -*-\nfrom pathlib import Path\nfrom typing import Any, Dict, Iterable, List, NamedTuple, Set, Tuple, Union\nimport copy\n\nimport numpy as np\nimport kaldi_native_fbank as knf\n\nroot_dir = Path(__file__).resolve().parent\n\nlogger_initialized = {}\n\n\nclass WavFrontend:\n \"\"\"Conventional frontend structure for ASR.\"\"\"\n\n def __init__(\n self,\n cmvn_file: str = None,\n fs: int = 16000,\n window: str = \"hamming\",\n n_mels: int = 80,\n frame_length: int = 25,\n frame_shift: int = 10,\n lfr_m: int = 1,\n lfr_n: int = 1,\n dither: float = 1.0,\n **kwargs,\n ) -> None:\n\n opts = knf.FbankOptions()\n opts.frame_opts.samp_freq = fs\n opts.frame_opts.dither = dither\n opts.frame_opts.window_type = window\n opts.frame_opts.frame_shift_ms = float(frame_shift)\n opts.frame_opts.frame_length_ms = float(frame_length)\n opts.mel_opts.num_bins = n_mels\n opts.energy_floor = 0\n opts.frame_opts.snip_edges = True\n opts.mel_opts.debug_mel = False\n self.opts = opts\n\n self.lfr_m = lfr_m\n self.lfr_n = lfr_n\n self.cmvn_file = cmvn_file\n\n if self.cmvn_file:\n self.cmvn = self.load_cmvn()\n self.fbank_fn = None\n self.fbank_beg_idx = 0\n self.reset_status()\n\n def fbank(self, waveform: np.ndarray) -> Tuple[np.ndarray, np.ndarray]:\n waveform = waveform * (1 << 15)\n self.fbank_fn = knf.OnlineFbank(self.opts)\n self.fbank_fn.accept_waveform(self.opts.frame_opts.samp_freq, waveform.tolist())\n frames = self.fbank_fn.num_frames_ready\n mat = np.empty([frames, self.opts.mel_opts.num_bins])\n for i in range(frames):\n mat[i, :] = self.fbank_fn.get_frame(i)\n feat = mat.astype(np.float32)\n feat_len = np.array(mat.shape[0]).astype(np.int32)\n return feat, feat_len\n\n def fbank_online(self, waveform: np.ndarray) -> Tuple[np.ndarray, np.ndarray]:\n waveform = waveform * (1 << 15)\n # self.fbank_fn = knf.OnlineFbank(self.opts)\n self.fbank_fn.accept_waveform(self.opts.frame_opts.samp_freq, waveform.tolist())\n frames = self.fbank_fn.num_frames_ready\n mat = np.empty([frames, self.opts.mel_opts.num_bins])\n for i in range(self.fbank_beg_idx, frames):\n mat[i, :] = self.fbank_fn.get_frame(i)\n # self.fbank_beg_idx += (frames-self.fbank_beg_idx)\n feat = mat.astype(np.float32)\n feat_len = np.array(mat.shape[0]).astype(np.int32)\n return feat, feat_len\n\n def reset_status(self):\n self.fbank_fn = knf.OnlineFbank(self.opts)\n self.fbank_beg_idx = 0\n\n def lfr_cmvn(self, feat: np.ndarray) -> Tuple[np.ndarray, np.ndarray]:\n if self.lfr_m != 1 or self.lfr_n != 1:\n feat = self.apply_lfr(feat, self.lfr_m, self.lfr_n)\n\n if self.cmvn_file:\n feat = self.apply_cmvn(feat)\n\n feat_len = np.array(feat.shape[0]).astype(np.int32)\n return feat, feat_len\n\n @staticmethod\n def apply_lfr(inputs: np.ndarray, lfr_m: int, lfr_n: int) -> np.ndarray:\n LFR_inputs = []\n\n T = inputs.shape[0]\n T_lfr = int(np.ceil(T / lfr_n))\n left_padding = np.tile(inputs[0], ((lfr_m - 1) // 2, 1))\n inputs = np.vstack((left_padding, inputs))\n T = T + (lfr_m - 1) // 2\n for i in range(T_lfr):\n if lfr_m <= T - i * lfr_n:\n LFR_inputs.append((inputs[i * lfr_n : i * lfr_n + lfr_m]).reshape(1, -1))\n else:\n # process last LFR frame\n num_padding = lfr_m - (T - i * lfr_n)\n frame = inputs[i * lfr_n :].reshape(-1)\n for _ in range(num_padding):\n frame = np.hstack((frame, inputs[-1]))\n\n LFR_inputs.append(frame)\n LFR_outputs = np.vstack(LFR_inputs).astype(np.float32)\n return LFR_outputs\n\n def apply_cmvn(self, inputs: np.ndarray) -> np.ndarray:\n \"\"\"\n Apply CMVN with mvn data\n \"\"\"\n frame, dim = inputs.shape\n means = np.tile(self.cmvn[0:1, :dim], (frame, 1))\n vars = np.tile(self.cmvn[1:2, :dim], (frame, 1))\n inputs = (inputs + means) * vars\n return inputs\n\n def load_cmvn(\n self,\n ) -> np.ndarray:\n with open(self.cmvn_file, \"r\", encoding=\"utf-8\") as f:\n lines = f.readlines()\n\n means_list = []\n vars_list = []\n for i in range(len(lines)):\n line_item = lines[i].split()\n if line_item[0] == \"\":\n line_item = lines[i + 1].split()\n if line_item[0] == \"\":\n add_shift_line = line_item[3 : (len(line_item) - 1)]\n means_list = list(add_shift_line)\n continue\n elif line_item[0] == \"\":\n line_item = lines[i + 1].split()\n if line_item[0] == \"\":\n rescale_line = line_item[3 : (len(line_item) - 1)]\n vars_list = list(rescale_line)\n continue\n\n means = np.array(means_list).astype(np.float64)\n vars = np.array(vars_list).astype(np.float64)\n cmvn = np.array([means, vars])\n return cmvn\n\n\nclass WavFrontendOnline(WavFrontend):\n def __init__(self, **kwargs):\n super().__init__(**kwargs)\n # self.fbank_fn = knf.OnlineFbank(self.opts)\n # add variables\n self.frame_sample_length = int(\n self.opts.frame_opts.frame_length_ms * self.opts.frame_opts.samp_freq / 1000\n )\n self.frame_shift_sample_length = int(\n self.opts.frame_opts.frame_shift_ms * self.opts.frame_opts.samp_freq / 1000\n )\n self.waveform = None\n self.reserve_waveforms = None\n self.input_cache = None\n self.lfr_splice_cache = []\n\n @staticmethod\n # inputs has catted the cache\n def apply_lfr(\n inputs: np.ndarray, lfr_m: int, lfr_n: int, is_final: bool = False\n ) -> Tuple[np.ndarray, np.ndarray, int]:\n \"\"\"\n Apply lfr with data\n \"\"\"\n\n LFR_inputs = []\n T = inputs.shape[0] # include the right context\n T_lfr = int(\n np.ceil((T - (lfr_m - 1) // 2) / lfr_n)\n ) # minus the right context: (lfr_m - 1) // 2\n splice_idx = T_lfr\n for i in range(T_lfr):\n if lfr_m <= T - i * lfr_n:\n LFR_inputs.append((inputs[i * lfr_n : i * lfr_n + lfr_m]).reshape(1, -1))\n else: # process last LFR frame\n if is_final:\n num_padding = lfr_m - (T - i * lfr_n)\n frame = (inputs[i * lfr_n :]).reshape(-1)\n for _ in range(num_padding):\n frame = np.hstack((frame, inputs[-1]))\n LFR_inputs.append(frame)\n else:\n # update splice_idx and break the circle\n splice_idx = i\n break\n splice_idx = min(T - 1, splice_idx * lfr_n)\n lfr_splice_cache = inputs[splice_idx:, :]\n LFR_outputs = np.vstack(LFR_inputs)\n return LFR_outputs.astype(np.float32), lfr_splice_cache, splice_idx\n\n @staticmethod\n def compute_frame_num(\n sample_length: int, frame_sample_length: int, frame_shift_sample_length: int\n ) -> int:\n frame_num = int((sample_length - frame_sample_length) / frame_shift_sample_length + 1)\n return frame_num if frame_num >= 1 and sample_length >= frame_sample_length else 0\n\n def fbank(\n self, input: np.ndarray, input_lengths: np.ndarray\n ) -> Tuple[np.ndarray, np.ndarray, np.ndarray]:\n self.fbank_fn = knf.OnlineFbank(self.opts)\n batch_size = input.shape[0]\n if self.input_cache is None:\n self.input_cache = np.empty((batch_size, 0), dtype=np.float32)\n input = np.concatenate((self.input_cache, input), axis=1)\n frame_num = self.compute_frame_num(\n input.shape[-1], self.frame_sample_length, self.frame_shift_sample_length\n )\n # update self.in_cache\n self.input_cache = input[\n :, -(input.shape[-1] - frame_num * self.frame_shift_sample_length) :\n ]\n waveforms = np.empty(0, dtype=np.float32)\n feats_pad = np.empty(0, dtype=np.float32)\n feats_lens = np.empty(0, dtype=np.int32)\n if frame_num:\n waveforms = []\n feats = []\n feats_lens = []\n for i in range(batch_size):\n waveform = input[i]\n waveforms.append(\n waveform[\n : (\n (frame_num - 1) * self.frame_shift_sample_length\n + self.frame_sample_length\n )\n ]\n )\n waveform = waveform * (1 << 15)\n\n self.fbank_fn.accept_waveform(self.opts.frame_opts.samp_freq, waveform.tolist())\n frames = self.fbank_fn.num_frames_ready\n mat = np.empty([frames, self.opts.mel_opts.num_bins])\n for i in range(frames):\n mat[i, :] = self.fbank_fn.get_frame(i)\n feat = mat.astype(np.float32)\n feat_len = np.array(mat.shape[0]).astype(np.int32)\n feats.append(feat)\n feats_lens.append(feat_len)\n\n waveforms = np.stack(waveforms)\n feats_lens = np.array(feats_lens)\n feats_pad = np.array(feats)\n self.fbanks = feats_pad\n self.fbanks_lens = copy.deepcopy(feats_lens)\n return waveforms, feats_pad, feats_lens\n\n def get_fbank(self) -> Tuple[np.ndarray, np.ndarray]:\n return self.fbanks, self.fbanks_lens\n\n def lfr_cmvn(\n self, input: np.ndarray, input_lengths: np.ndarray, is_final: bool = False\n ) -> Tuple[np.ndarray, np.ndarray, List[int]]:\n batch_size = input.shape[0]\n feats = []\n feats_lens = []\n lfr_splice_frame_idxs = []\n for i in range(batch_size):\n mat = input[i, : input_lengths[i], :]\n lfr_splice_frame_idx = -1\n if self.lfr_m != 1 or self.lfr_n != 1:\n # update self.lfr_splice_cache in self.apply_lfr\n mat, self.lfr_splice_cache[i], lfr_splice_frame_idx = self.apply_lfr(\n mat, self.lfr_m, self.lfr_n, is_final\n )\n if self.cmvn_file is not None:\n mat = self.apply_cmvn(mat)\n feat_length = mat.shape[0]\n feats.append(mat)\n feats_lens.append(feat_length)\n lfr_splice_frame_idxs.append(lfr_splice_frame_idx)\n\n feats_lens = np.array(feats_lens)\n feats_pad = np.array(feats)\n return feats_pad, feats_lens, lfr_splice_frame_idxs\n\n def extract_fbank(\n self, input: np.ndarray, input_lengths: np.ndarray, is_final: bool = False\n ) -> Tuple[np.ndarray, np.ndarray]:\n batch_size = input.shape[0]\n assert (\n batch_size == 1\n ), \"we support to extract feature online only when the batch size is equal to 1 now\"\n waveforms, feats, feats_lengths = self.fbank(input, input_lengths) # input shape: B T D\n if feats.shape[0]:\n self.waveforms = (\n waveforms\n if self.reserve_waveforms is None\n else np.concatenate((self.reserve_waveforms, waveforms), axis=1)\n )\n if not self.lfr_splice_cache:\n for i in range(batch_size):\n self.lfr_splice_cache.append(\n np.expand_dims(feats[i][0, :], axis=0).repeat((self.lfr_m - 1) // 2, axis=0)\n )\n\n if feats_lengths[0] + self.lfr_splice_cache[0].shape[0] >= self.lfr_m:\n lfr_splice_cache_np = np.stack(self.lfr_splice_cache) # B T D\n feats = np.concatenate((lfr_splice_cache_np, feats), axis=1)\n feats_lengths += lfr_splice_cache_np[0].shape[0]\n frame_from_waveforms = int(\n (self.waveforms.shape[1] - self.frame_sample_length)\n / self.frame_shift_sample_length\n + 1\n )\n minus_frame = (self.lfr_m - 1) // 2 if self.reserve_waveforms is None else 0\n feats, feats_lengths, lfr_splice_frame_idxs = self.lfr_cmvn(\n feats, feats_lengths, is_final\n )\n if self.lfr_m == 1:\n self.reserve_waveforms = None\n else:\n reserve_frame_idx = lfr_splice_frame_idxs[0] - minus_frame\n # print('reserve_frame_idx: ' + str(reserve_frame_idx))\n # print('frame_frame: ' + str(frame_from_waveforms))\n self.reserve_waveforms = self.waveforms[\n :,\n reserve_frame_idx\n * self.frame_shift_sample_length : frame_from_waveforms\n * self.frame_shift_sample_length,\n ]\n sample_length = (\n frame_from_waveforms - 1\n ) * self.frame_shift_sample_length + self.frame_sample_length\n self.waveforms = self.waveforms[:, :sample_length]\n else:\n # update self.reserve_waveforms and self.lfr_splice_cache\n self.reserve_waveforms = self.waveforms[\n :, : -(self.frame_sample_length - self.frame_shift_sample_length)\n ]\n for i in range(batch_size):\n self.lfr_splice_cache[i] = np.concatenate(\n (self.lfr_splice_cache[i], feats[i]), axis=0\n )\n return np.empty(0, dtype=np.float32), feats_lengths\n else:\n if is_final:\n self.waveforms = (\n waveforms if self.reserve_waveforms is None else self.reserve_waveforms\n )\n feats = np.stack(self.lfr_splice_cache)\n feats_lengths = np.zeros(batch_size, dtype=np.int32) + feats.shape[1]\n feats, feats_lengths, _ = self.lfr_cmvn(feats, feats_lengths, is_final)\n if is_final:\n self.cache_reset()\n return feats, feats_lengths\n\n def get_waveforms(self):\n return self.waveforms\n\n def cache_reset(self):\n self.fbank_fn = knf.OnlineFbank(self.opts)\n self.reserve_waveforms = None\n self.input_cache = None\n self.lfr_splice_cache = []\n\n\ndef load_bytes(input):\n middle_data = np.frombuffer(input, dtype=np.int16)\n middle_data = np.asarray(middle_data)\n if middle_data.dtype.kind not in \"iu\":\n raise TypeError(\"'middle_data' must be an array of integers\")\n dtype = np.dtype(\"float32\")\n if dtype.kind != \"f\":\n raise TypeError(\"'dtype' must be a floating point type\")\n\n i = np.iinfo(middle_data.dtype)\n abs_max = 2 ** (i.bits - 1)\n offset = i.min + abs_max\n array = np.frombuffer((middle_data.astype(dtype) - offset) / abs_max, dtype=np.float32)\n return array\n\n\nclass SinusoidalPositionEncoderOnline:\n \"\"\"Streaming Positional encoding.\"\"\"\n\n def encode(self, positions: np.ndarray = None, depth: int = None, dtype: np.dtype = np.float32):\n batch_size = positions.shape[0]\n positions = positions.astype(dtype)\n log_timescale_increment = np.log(np.array([10000], dtype=dtype)) / (depth / 2 - 1)\n inv_timescales = np.exp(np.arange(depth / 2).astype(dtype) * (-log_timescale_increment))\n inv_timescales = np.reshape(inv_timescales, [batch_size, -1])\n scaled_time = np.reshape(positions, [1, -1, 1]) * np.reshape(inv_timescales, [1, 1, -1])\n encoding = np.concatenate((np.sin(scaled_time), np.cos(scaled_time)), axis=2)\n return encoding.astype(dtype)\n\n def forward(self, x, start_idx=0):\n batch_size, timesteps, input_dim = x.shape\n positions = np.arange(1, timesteps + 1 + start_idx)[None, :]\n position_encoding = self.encode(positions, input_dim, x.dtype)\n\n return x + position_encoding[:, start_idx : start_idx + timesteps]\n\n\ndef test():\n path = \"/nfs/zhifu.gzf/export/damo/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-pytorch/example/asr_example.wav\"\n import librosa\n\n cmvn_file = \"/nfs/zhifu.gzf/export/damo/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-pytorch/am.mvn\"\n config_file = \"/nfs/zhifu.gzf/export/damo/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-pytorch/config.yaml\"\n from funasr.runtime.python.onnxruntime.rapid_paraformer.utils.utils import read_yaml\n\n config = read_yaml(config_file)\n waveform, _ = librosa.load(path, sr=None)\n frontend = WavFrontend(\n cmvn_file=cmvn_file,\n **config[\"frontend_conf\"],\n )\n speech, _ = frontend.fbank_online(waveform) # 1d, (sample,), numpy\n feat, feat_len = frontend.lfr_cmvn(\n speech\n ) # 2d, (frame, 450), np.float32 -> torch, torch.from_numpy(), dtype, (1, frame, 450)\n\n frontend.reset_status() # clear cache\n return feat, feat_len\n\n\nif __name__ == \"__main__\":\n test()\n" }, { "path": "utils/infer_utils.py", "content": "# -*- encoding: utf-8 -*-\n\nimport functools\nimport logging\nfrom pathlib import Path\nfrom typing import Any, Dict, Iterable, List, NamedTuple, Set, Tuple, Union\n\nimport re\nimport numpy as np\nimport yaml\n\ntry:\n from onnxruntime import (\n GraphOptimizationLevel,\n InferenceSession,\n SessionOptions,\n get_available_providers,\n get_device,\n )\nexcept:\n print(\"please pip3 install onnxruntime\")\nimport jieba\nimport warnings\n\nroot_dir = Path(__file__).resolve().parent\n\nlogger_initialized = {}\n\n\ndef pad_list(xs, pad_value, max_len=None):\n n_batch = len(xs)\n if max_len is None:\n max_len = max(x.size(0) for x in xs)\n # pad = xs[0].new(n_batch, max_len, *xs[0].size()[1:]).fill_(pad_value)\n # numpy format\n pad = (np.zeros((n_batch, max_len)) + pad_value).astype(np.int32)\n for i in range(n_batch):\n pad[i, : xs[i].shape[0]] = xs[i]\n\n return pad\n\n\n\"\"\"\ndef make_pad_mask(lengths, xs=None, length_dim=-1, maxlen=None):\n if length_dim == 0:\n raise ValueError(\"length_dim cannot be 0: {}\".format(length_dim))\n\n if not isinstance(lengths, list):\n lengths = lengths.tolist()\n bs = int(len(lengths))\n if maxlen is None:\n if xs is None:\n maxlen = int(max(lengths))\n else:\n maxlen = xs.size(length_dim)\n else:\n assert xs is None\n assert maxlen >= int(max(lengths))\n\n seq_range = torch.arange(0, maxlen, dtype=torch.int64)\n seq_range_expand = seq_range.unsqueeze(0).expand(bs, maxlen)\n seq_length_expand = seq_range_expand.new(lengths).unsqueeze(-1)\n mask = seq_range_expand >= seq_length_expand\n\n if xs is not None:\n assert xs.size(0) == bs, (xs.size(0), bs)\n\n if length_dim < 0:\n length_dim = xs.dim() + length_dim\n # ind = (:, None, ..., None, :, , None, ..., None)\n ind = tuple(\n slice(None) if i in (0, length_dim) else None for i in range(xs.dim())\n )\n mask = mask[ind].expand_as(xs).to(xs.device)\n return mask\n\"\"\"\n\n\nclass TokenIDConverter:\n def __init__(\n self,\n token_list: Union[List, str],\n ):\n\n self.token_list = token_list\n self.unk_symbol = token_list[-1]\n self.token2id = {v: i for i, v in enumerate(self.token_list)}\n self.unk_id = self.token2id[self.unk_symbol]\n\n def get_num_vocabulary_size(self) -> int:\n return len(self.token_list)\n\n def ids2tokens(self, integers: Union[np.ndarray, Iterable[int]]) -> List[str]:\n if isinstance(integers, np.ndarray) and integers.ndim != 1:\n raise TokenIDConverterError(f\"Must be 1 dim ndarray, but got {integers.ndim}\")\n return [self.token_list[i] for i in integers]\n\n def tokens2ids(self, tokens: Iterable[str]) -> List[int]:\n\n return [self.token2id.get(i, self.unk_id) for i in tokens]\n\n\nclass CharTokenizer:\n def __init__(\n self,\n symbol_value: Union[Path, str, Iterable[str]] = None,\n space_symbol: str = \"\",\n remove_non_linguistic_symbols: bool = False,\n ):\n\n self.space_symbol = space_symbol\n self.non_linguistic_symbols = self.load_symbols(symbol_value)\n self.remove_non_linguistic_symbols = remove_non_linguistic_symbols\n\n @staticmethod\n def load_symbols(value: Union[Path, str, Iterable[str]] = None) -> Set:\n if value is None:\n return set()\n\n if isinstance(value, Iterable[str]):\n return set(value)\n\n file_path = Path(value)\n if not file_path.exists():\n logging.warning(\"%s doesn't exist.\", file_path)\n return set()\n\n with file_path.open(\"r\", encoding=\"utf-8\") as f:\n return set(line.rstrip() for line in f)\n\n def text2tokens(self, line: Union[str, list]) -> List[str]:\n tokens = []\n while len(line) != 0:\n for w in self.non_linguistic_symbols:\n if line.startswith(w):\n if not self.remove_non_linguistic_symbols:\n tokens.append(line[: len(w)])\n line = line[len(w) :]\n break\n else:\n t = line[0]\n if t == \" \":\n t = \"\"\n tokens.append(t)\n line = line[1:]\n return tokens\n\n def tokens2text(self, tokens: Iterable[str]) -> str:\n tokens = [t if t != self.space_symbol else \" \" for t in tokens]\n return \"\".join(tokens)\n\n def __repr__(self):\n return (\n f\"{self.__class__.__name__}(\"\n f'space_symbol=\"{self.space_symbol}\"'\n f'non_linguistic_symbols=\"{self.non_linguistic_symbols}\"'\n f\")\"\n )\n\n\nclass Hypothesis(NamedTuple):\n \"\"\"Hypothesis data type.\"\"\"\n\n yseq: np.ndarray\n score: Union[float, np.ndarray] = 0\n scores: Dict[str, Union[float, np.ndarray]] = dict()\n states: Dict[str, Any] = dict()\n\n def asdict(self) -> dict:\n \"\"\"Convert data to JSON-friendly dict.\"\"\"\n return self._replace(\n yseq=self.yseq.tolist(),\n score=float(self.score),\n scores={k: float(v) for k, v in self.scores.items()},\n )._asdict()\n\n\nclass TokenIDConverterError(Exception):\n pass\n\n\nclass ONNXRuntimeError(Exception):\n pass\n\n\nclass OrtInferSession:\n def __init__(self, model_file, device_id=-1, intra_op_num_threads=4):\n device_id = str(device_id)\n sess_opt = SessionOptions()\n sess_opt.intra_op_num_threads = intra_op_num_threads\n sess_opt.log_severity_level = 4\n sess_opt.enable_cpu_mem_arena = False\n sess_opt.graph_optimization_level = GraphOptimizationLevel.ORT_ENABLE_ALL\n\n cuda_ep = \"CUDAExecutionProvider\"\n cuda_provider_options = {\n \"device_id\": device_id,\n \"arena_extend_strategy\": \"kNextPowerOfTwo\",\n \"cudnn_conv_algo_search\": \"EXHAUSTIVE\",\n \"do_copy_in_default_stream\": \"true\",\n }\n cpu_ep = \"CPUExecutionProvider\"\n cpu_provider_options = {\n \"arena_extend_strategy\": \"kSameAsRequested\",\n }\n\n EP_list = []\n if device_id != \"-1\" and get_device() == \"GPU\" and cuda_ep in get_available_providers():\n EP_list = [(cuda_ep, cuda_provider_options)]\n EP_list.append((cpu_ep, cpu_provider_options))\n\n self._verify_model(model_file)\n self.session = InferenceSession(model_file, sess_options=sess_opt, providers=EP_list)\n\n if device_id != \"-1\" and cuda_ep not in self.session.get_providers():\n warnings.warn(\n f\"{cuda_ep} is not avaiable for current env, the inference part is automatically shifted to be executed under {cpu_ep}.\\n\"\n \"Please ensure the installed onnxruntime-gpu version matches your cuda and cudnn version, \"\n \"you can check their relations from the offical web site: \"\n \"https://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html\",\n RuntimeWarning,\n )\n\n def __call__(self, input_content: List[Union[np.ndarray, np.ndarray]]) -> np.ndarray:\n input_dict = dict(zip(self.get_input_names(), input_content))\n try:\n return self.session.run(self.get_output_names(), input_dict)\n except Exception as e:\n raise ONNXRuntimeError(\"ONNXRuntime inferece failed.\") from e\n\n def get_input_names(\n self,\n ):\n return [v.name for v in self.session.get_inputs()]\n\n def get_output_names(\n self,\n ):\n return [v.name for v in self.session.get_outputs()]\n\n def get_character_list(self, key: str = \"character\"):\n return self.meta_dict[key].splitlines()\n\n def have_key(self, key: str = \"character\") -> bool:\n self.meta_dict = self.session.get_modelmeta().custom_metadata_map\n if key in self.meta_dict.keys():\n return True\n return False\n\n @staticmethod\n def _verify_model(model_path):\n model_path = Path(model_path)\n if not model_path.exists():\n raise FileNotFoundError(f\"{model_path} does not exists.\")\n if not model_path.is_file():\n raise FileExistsError(f\"{model_path} is not a file.\")\n\n\ndef split_to_mini_sentence(words: list, word_limit: int = 20):\n assert word_limit > 1\n if len(words) <= word_limit:\n return [words]\n sentences = []\n length = len(words)\n sentence_len = length // word_limit\n for i in range(sentence_len):\n sentences.append(words[i * word_limit : (i + 1) * word_limit])\n if length % word_limit > 0:\n sentences.append(words[sentence_len * word_limit :])\n return sentences\n\n\ndef code_mix_split_words(text: str):\n words = []\n segs = text.split()\n for seg in segs:\n # There is no space in seg.\n current_word = \"\"\n for c in seg:\n if len(c.encode()) == 1:\n # This is an ASCII char.\n current_word += c\n else:\n # This is a Chinese char.\n if len(current_word) > 0:\n words.append(current_word)\n current_word = \"\"\n words.append(c)\n if len(current_word) > 0:\n words.append(current_word)\n return words\n\n\ndef isEnglish(text: str):\n if re.search(\"^[a-zA-Z']+$\", text):\n return True\n else:\n return False\n\n\ndef join_chinese_and_english(input_list):\n line = \"\"\n for token in input_list:\n if isEnglish(token):\n line = line + \" \" + token\n else:\n line = line + token\n\n line = line.strip()\n return line\n\n\ndef code_mix_split_words_jieba(seg_dict_file: str):\n jieba.load_userdict(seg_dict_file)\n\n def _fn(text: str):\n input_list = text.split()\n token_list_all = []\n langauge_list = []\n token_list_tmp = []\n language_flag = None\n for token in input_list:\n if isEnglish(token) and language_flag == \"Chinese\":\n token_list_all.append(token_list_tmp)\n langauge_list.append(\"Chinese\")\n token_list_tmp = []\n elif not isEnglish(token) and language_flag == \"English\":\n token_list_all.append(token_list_tmp)\n langauge_list.append(\"English\")\n token_list_tmp = []\n\n token_list_tmp.append(token)\n\n if isEnglish(token):\n language_flag = \"English\"\n else:\n language_flag = \"Chinese\"\n\n if token_list_tmp:\n token_list_all.append(token_list_tmp)\n langauge_list.append(language_flag)\n\n result_list = []\n for token_list_tmp, language_flag in zip(token_list_all, langauge_list):\n if language_flag == \"English\":\n result_list.extend(token_list_tmp)\n else:\n seg_list = jieba.cut(join_chinese_and_english(token_list_tmp), HMM=False)\n result_list.extend(seg_list)\n\n return result_list\n\n return _fn\n\n\ndef read_yaml(yaml_path: Union[str, Path]) -> Dict:\n if not Path(yaml_path).exists():\n raise FileExistsError(f\"The {yaml_path} does not exist.\")\n\n with open(str(yaml_path), \"rb\") as f:\n data = yaml.load(f, Loader=yaml.Loader)\n return data\n\n\n@functools.lru_cache()\ndef get_logger(name=\"funasr_onnx\"):\n \"\"\"Initialize and get a logger by name.\n If the logger has not been initialized, this method will initialize the\n logger by adding one or two handlers, otherwise the initialized logger will\n be directly returned. During initialization, a StreamHandler will always be\n added.\n Args:\n name (str): Logger name.\n Returns:\n logging.Logger: The expected logger.\n \"\"\"\n logger = logging.getLogger(name)\n if name in logger_initialized:\n return logger\n\n for logger_name in logger_initialized:\n if name.startswith(logger_name):\n return logger\n\n formatter = logging.Formatter(\n \"[%(asctime)s] %(name)s %(levelname)s: %(message)s\", datefmt=\"%Y/%m/%d %H:%M:%S\"\n )\n\n sh = logging.StreamHandler()\n sh.setFormatter(formatter)\n logger.addHandler(sh)\n logger_initialized[name] = True\n logger.propagate = False\n logging.basicConfig(level=logging.ERROR)\n return logger\n" }, { "path": "utils/model_bin.py", "content": "#!/usr/bin/env python3\n# -*- encoding: utf-8 -*-\n# Copyright FunASR (https://github.com/FunAudioLLM/SenseVoice). All Rights Reserved.\n# MIT License (https://opensource.org/licenses/MIT)\n\nimport os.path\nfrom pathlib import Path\nfrom typing import List, Union, Tuple\nimport torch\nimport librosa\nimport numpy as np\n\nfrom utils.infer_utils import (\n CharTokenizer,\n Hypothesis,\n ONNXRuntimeError,\n OrtInferSession,\n TokenIDConverter,\n get_logger,\n read_yaml,\n)\nfrom utils.frontend import WavFrontend\nfrom utils.infer_utils import pad_list\n\nlogging = get_logger()\n\n\nclass SenseVoiceSmallONNX:\n \"\"\"\n Author: Speech Lab of DAMO Academy, Alibaba Group\n Paraformer: Fast and Accurate Parallel Transformer for Non-autoregressive End-to-End Speech Recognition\n https://arxiv.org/abs/2206.08317\n \"\"\"\n\n def __init__(\n self,\n model_dir: Union[str, Path] = None,\n batch_size: int = 1,\n device_id: Union[str, int] = \"-1\",\n plot_timestamp_to: str = \"\",\n quantize: bool = False,\n intra_op_num_threads: int = 4,\n cache_dir: str = None,\n **kwargs,\n ):\n if quantize:\n model_file = os.path.join(model_dir, \"model_quant.onnx\")\n else:\n model_file = os.path.join(model_dir, \"model.onnx\")\n\n config_file = os.path.join(model_dir, \"config.yaml\")\n cmvn_file = os.path.join(model_dir, \"am.mvn\")\n config = read_yaml(config_file)\n # token_list = os.path.join(model_dir, \"tokens.json\")\n # with open(token_list, \"r\", encoding=\"utf-8\") as f:\n # token_list = json.load(f)\n\n # self.converter = TokenIDConverter(token_list)\n self.tokenizer = CharTokenizer()\n config[\"frontend_conf\"]['cmvn_file'] = cmvn_file\n self.frontend = WavFrontend(**config[\"frontend_conf\"])\n self.ort_infer = OrtInferSession(\n model_file, device_id, intra_op_num_threads=intra_op_num_threads\n )\n self.batch_size = batch_size\n self.blank_id = 0\n\n def __call__(self, \n wav_content: Union[str, np.ndarray, List[str]], \n language: List, \n textnorm: List,\n tokenizer=None,\n **kwargs) -> List:\n waveform_list = self.load_data(wav_content, self.frontend.opts.frame_opts.samp_freq)\n waveform_nums = len(waveform_list)\n asr_res = []\n for beg_idx in range(0, waveform_nums, self.batch_size):\n end_idx = min(waveform_nums, beg_idx + self.batch_size)\n feats, feats_len = self.extract_feat(waveform_list[beg_idx:end_idx])\n ctc_logits, encoder_out_lens = self.infer(feats, \n feats_len, \n np.array(language, dtype=np.int32), \n np.array(textnorm, dtype=np.int32)\n )\n # back to torch.Tensor\n ctc_logits = torch.from_numpy(ctc_logits).float()\n # support batch_size=1 only currently\n x = ctc_logits[0, : encoder_out_lens[0].item(), :]\n yseq = x.argmax(dim=-1)\n yseq = torch.unique_consecutive(yseq, dim=-1)\n\n mask = yseq != self.blank_id\n token_int = yseq[mask].tolist()\n \n if tokenizer is not None:\n asr_res.append(tokenizer.tokens2text(token_int))\n else:\n asr_res.append(token_int)\n return asr_res\n\n def load_data(self, wav_content: Union[str, np.ndarray, List[str]], fs: int = None) -> List:\n def load_wav(path: str) -> np.ndarray:\n waveform, _ = librosa.load(path, sr=fs)\n return waveform\n\n if isinstance(wav_content, np.ndarray):\n return [wav_content]\n\n if isinstance(wav_content, str):\n return [load_wav(wav_content)]\n\n if isinstance(wav_content, list):\n return [load_wav(path) for path in wav_content]\n\n raise TypeError(f\"The type of {wav_content} is not in [str, np.ndarray, list]\")\n\n def extract_feat(self, waveform_list: List[np.ndarray]) -> Tuple[np.ndarray, np.ndarray]:\n feats, feats_len = [], []\n for waveform in waveform_list:\n speech, _ = self.frontend.fbank(waveform)\n feat, feat_len = self.frontend.lfr_cmvn(speech)\n feats.append(feat)\n feats_len.append(feat_len)\n\n feats = self.pad_feats(feats, np.max(feats_len))\n feats_len = np.array(feats_len).astype(np.int32)\n return feats, feats_len\n\n @staticmethod\n def pad_feats(feats: List[np.ndarray], max_feat_len: int) -> np.ndarray:\n def pad_feat(feat: np.ndarray, cur_len: int) -> np.ndarray:\n pad_width = ((0, max_feat_len - cur_len), (0, 0))\n return np.pad(feat, pad_width, \"constant\", constant_values=0)\n\n feat_res = [pad_feat(feat, feat.shape[0]) for feat in feats]\n feats = np.array(feat_res).astype(np.float32)\n return feats\n\n def infer(self, \n feats: np.ndarray, \n feats_len: np.ndarray,\n language: np.ndarray,\n textnorm: np.ndarray,) -> Tuple[np.ndarray, np.ndarray]:\n outputs = self.ort_infer([feats, feats_len, language, textnorm])\n return outputs\n" }, { "path": "webui.py", "content": "# coding=utf-8\n\nimport os\nimport librosa\nimport base64\nimport io\nimport gradio as gr\nimport re\n\nimport numpy as np\nimport torch\nimport torchaudio\n\n\nfrom funasr import AutoModel\n\nmodel = \"iic/SenseVoiceSmall\"\nmodel = AutoModel(model=model,\n\t\t\t\t vad_model=\"iic/speech_fsmn_vad_zh-cn-16k-common-pytorch\",\n\t\t\t\t vad_kwargs={\"max_single_segment_time\": 30000},\n\t\t\t\t trust_remote_code=True,\n\t\t\t\t )\n\nimport re\n\nemo_dict = {\n\t\"<|HAPPY|>\": \"😊\",\n\t\"<|SAD|>\": \"😔\",\n\t\"<|ANGRY|>\": \"😡\",\n\t\"<|NEUTRAL|>\": \"\",\n\t\"<|FEARFUL|>\": \"😰\",\n\t\"<|DISGUSTED|>\": \"🤢\",\n\t\"<|SURPRISED|>\": \"😮\",\n}\n\nevent_dict = {\n\t\"<|BGM|>\": \"🎼\",\n\t\"<|Speech|>\": \"\",\n\t\"<|Applause|>\": \"👏\",\n\t\"<|Laughter|>\": \"😀\",\n\t\"<|Cry|>\": \"😭\",\n\t\"<|Sneeze|>\": \"🤧\",\n\t\"<|Breath|>\": \"\",\n\t\"<|Cough|>\": \"🤧\",\n}\n\nemoji_dict = {\n\t\"<|nospeech|><|Event_UNK|>\": \"❓\",\n\t\"<|zh|>\": \"\",\n\t\"<|en|>\": \"\",\n\t\"<|yue|>\": \"\",\n\t\"<|ja|>\": \"\",\n\t\"<|ko|>\": \"\",\n\t\"<|nospeech|>\": \"\",\n\t\"<|HAPPY|>\": \"😊\",\n\t\"<|SAD|>\": \"😔\",\n\t\"<|ANGRY|>\": \"😡\",\n\t\"<|NEUTRAL|>\": \"\",\n\t\"<|BGM|>\": \"🎼\",\n\t\"<|Speech|>\": \"\",\n\t\"<|Applause|>\": \"👏\",\n\t\"<|Laughter|>\": \"😀\",\n\t\"<|FEARFUL|>\": \"😰\",\n\t\"<|DISGUSTED|>\": \"🤢\",\n\t\"<|SURPRISED|>\": \"😮\",\n\t\"<|Cry|>\": \"😭\",\n\t\"<|EMO_UNKNOWN|>\": \"\",\n\t\"<|Sneeze|>\": \"🤧\",\n\t\"<|Breath|>\": \"\",\n\t\"<|Cough|>\": \"😷\",\n\t\"<|Sing|>\": \"\",\n\t\"<|Speech_Noise|>\": \"\",\n\t\"<|withitn|>\": \"\",\n\t\"<|woitn|>\": \"\",\n\t\"<|GBG|>\": \"\",\n\t\"<|Event_UNK|>\": \"\",\n}\n\nlang_dict = {\n \"<|zh|>\": \"<|lang|>\",\n \"<|en|>\": \"<|lang|>\",\n \"<|yue|>\": \"<|lang|>\",\n \"<|ja|>\": \"<|lang|>\",\n \"<|ko|>\": \"<|lang|>\",\n \"<|nospeech|>\": \"<|lang|>\",\n}\n\nemo_set = {\"😊\", \"😔\", \"😡\", \"😰\", \"🤢\", \"😮\"}\nevent_set = {\"🎼\", \"👏\", \"😀\", \"😭\", \"🤧\", \"😷\",}\n\ndef format_str(s):\n\tfor sptk in emoji_dict:\n\t\ts = s.replace(sptk, emoji_dict[sptk])\n\treturn s\n\n\ndef format_str_v2(s):\n\tsptk_dict = {}\n\tfor sptk in emoji_dict:\n\t\tsptk_dict[sptk] = s.count(sptk)\n\t\ts = s.replace(sptk, \"\")\n\temo = \"<|NEUTRAL|>\"\n\tfor e in emo_dict:\n\t\tif sptk_dict[e] > sptk_dict[emo]:\n\t\t\temo = e\n\tfor e in event_dict:\n\t\tif sptk_dict[e] > 0:\n\t\t\ts = event_dict[e] + s\n\ts = s + emo_dict[emo]\n\n\tfor emoji in emo_set.union(event_set):\n\t\ts = s.replace(\" \" + emoji, emoji)\n\t\ts = s.replace(emoji + \" \", emoji)\n\treturn s.strip()\n\ndef format_str_v3(s):\n\tdef get_emo(s):\n\t\treturn s[-1] if s[-1] in emo_set else None\n\tdef get_event(s):\n\t\treturn s[0] if s[0] in event_set else None\n\n\ts = s.replace(\"<|nospeech|><|Event_UNK|>\", \"❓\")\n\tfor lang in lang_dict:\n\t\ts = s.replace(lang, \"<|lang|>\")\n\ts_list = [format_str_v2(s_i).strip(\" \") for s_i in s.split(\"<|lang|>\")]\n\tnew_s = \" \" + s_list[0]\n\tcur_ent_event = get_event(new_s)\n\tfor i in range(1, len(s_list)):\n\t\tif len(s_list[i]) == 0:\n\t\t\tcontinue\n\t\tif get_event(s_list[i]) == cur_ent_event and get_event(s_list[i]) != None:\n\t\t\ts_list[i] = s_list[i][1:]\n\t\t#else:\n\t\tcur_ent_event = get_event(s_list[i])\n\t\tif get_emo(s_list[i]) != None and get_emo(s_list[i]) == get_emo(new_s):\n\t\t\tnew_s = new_s[:-1]\n\t\tnew_s += s_list[i].strip().lstrip()\n\tnew_s = new_s.replace(\"The.\", \" \")\n\treturn new_s.strip()\n\ndef model_inference(input_wav, language, fs=16000):\n\t# task_abbr = {\"Speech Recognition\": \"ASR\", \"Rich Text Transcription\": (\"ASR\", \"AED\", \"SER\")}\n\tlanguage_abbr = {\"auto\": \"auto\", \"zh\": \"zh\", \"en\": \"en\", \"yue\": \"yue\", \"ja\": \"ja\", \"ko\": \"ko\",\n\t\t\t\t\t \"nospeech\": \"nospeech\"}\n\t\n\t# task = \"Speech Recognition\" if task is None else task\n\tlanguage = \"auto\" if len(language) < 1 else language\n\tselected_language = language_abbr[language]\n\t# selected_task = task_abbr.get(task)\n\t\n\t# print(f\"input_wav: {type(input_wav)}, {input_wav[1].shape}, {input_wav}\")\n\t\n\tif isinstance(input_wav, tuple):\n\t\tfs, input_wav = input_wav\n\t\tinput_wav = input_wav.astype(np.float32) / np.iinfo(np.int16).max\n\t\tif len(input_wav.shape) > 1:\n\t\t\tinput_wav = input_wav.mean(-1)\n\t\tif fs != 16000:\n\t\t\tprint(f\"audio_fs: {fs}\")\n\t\t\tresampler = torchaudio.transforms.Resample(fs, 16000)\n\t\t\tinput_wav_t = torch.from_numpy(input_wav).to(torch.float32)\n\t\t\tinput_wav = resampler(input_wav_t[None, :])[0, :].numpy()\n\t\n\t\n\tmerge_vad = True #False if selected_task == \"ASR\" else True\n\tprint(f\"language: {language}, merge_vad: {merge_vad}\")\n\ttext = model.generate(input=input_wav,\n\t\t\t\t\t\t cache={},\n\t\t\t\t\t\t language=language,\n\t\t\t\t\t\t use_itn=True,\n\t\t\t\t\t\t batch_size_s=60, merge_vad=merge_vad)\n\t\n\tprint(text)\n\ttext = text[0][\"text\"]\n\ttext = format_str_v3(text)\n\t\n\tprint(text)\n\t\n\treturn text\n\n\naudio_examples = [\n [\"example/zh.mp3\", \"zh\"],\n [\"example/yue.mp3\", \"yue\"],\n [\"example/en.mp3\", \"en\"],\n [\"example/ja.mp3\", \"ja\"],\n [\"example/ko.mp3\", \"ko\"],\n [\"example/emo_1.wav\", \"auto\"],\n [\"example/emo_2.wav\", \"auto\"],\n [\"example/emo_3.wav\", \"auto\"],\n #[\"example/emo_4.wav\", \"auto\"],\n #[\"example/event_1.wav\", \"auto\"],\n #[\"example/event_2.wav\", \"auto\"],\n #[\"example/event_3.wav\", \"auto\"],\n [\"example/rich_1.wav\", \"auto\"],\n [\"example/rich_2.wav\", \"auto\"],\n #[\"example/rich_3.wav\", \"auto\"],\n [\"example/longwav_1.wav\", \"auto\"],\n [\"example/longwav_2.wav\", \"auto\"],\n [\"example/longwav_3.wav\", \"auto\"],\n #[\"example/longwav_4.wav\", \"auto\"],\n]\n\n\n\nhtml_content = \"\"\"\n
\n

Voice Understanding Model: SenseVoice-Small

\n

SenseVoice-Small is an encoder-only speech foundation model designed for rapid voice understanding. It encompasses a variety of features including automatic speech recognition (ASR), spoken language identification (LID), speech emotion recognition (SER), and acoustic event detection (AED). SenseVoice-Small supports multilingual recognition for Chinese, English, Cantonese, Japanese, and Korean. Additionally, it offers exceptionally low inference latency, performing 7 times faster than Whisper-small and 17 times faster than Whisper-large.

\n

Usage

Upload an audio file or input through a microphone, then select the task and language. the audio is transcribed into corresponding text along with associated emotions (😊 happy, 😡 angry/exicting, 😔 sad) and types of sound events (😀 laughter, 🎼 music, 👏 applause, 🤧 cough&sneeze, 😭 cry). The event labels are placed in the front of the text and the emotion are in the back of the text.

\n\t

Recommended audio input duration is below 30 seconds. For audio longer than 30 seconds, local deployment is recommended.

\n\t

Repo

\n\t

SenseVoice: multilingual speech understanding model

\n\t

FunASR: fundamental speech recognition toolkit

\n\t

CosyVoice: high-quality multilingual TTS model

\n
\n\"\"\"\n\n\ndef launch():\n\twith gr.Blocks(theme=gr.themes.Soft()) as demo:\n\t\t# gr.Markdown(description)\n\t\tgr.HTML(html_content)\n\t\twith gr.Row():\n\t\t\twith gr.Column():\n\t\t\t\taudio_inputs = gr.Audio(label=\"Upload audio or use the microphone\")\n\t\t\t\t\n\t\t\t\twith gr.Accordion(\"Configuration\"):\n\t\t\t\t\tlanguage_inputs = gr.Dropdown(choices=[\"auto\", \"zh\", \"en\", \"yue\", \"ja\", \"ko\", \"nospeech\"],\n\t\t\t\t\t\t\t\t\t\t\t\t value=\"auto\",\n\t\t\t\t\t\t\t\t\t\t\t\t label=\"Language\")\n\t\t\t\tfn_button = gr.Button(\"Start\", variant=\"primary\")\n\t\t\t\ttext_outputs = gr.Textbox(label=\"Results\")\n\t\t\tgr.Examples(examples=audio_examples, inputs=[audio_inputs, language_inputs], examples_per_page=20)\n\t\t\n\t\tfn_button.click(model_inference, inputs=[audio_inputs, language_inputs], outputs=text_outputs)\n\n\tdemo.launch()\n\n\nif __name__ == \"__main__\":\n\t# iface.launch()\n\tlaunch()\n\n\n" } ]