[
  {
    "path": ".gitignore",
    "content": "# Byte-compiled / optimized / DLL files\n__pycache__/\n*.py[cod]\n*$py.class\n\n# C extensions\n*.so\n\n# Distribution / packaging\n.Python\nbuild/\ndevelop-eggs/\ndist/\ndownloads/\neggs/\n.eggs/\nlib/\nlib64/\nparts/\nsdist/\nvar/\nwheels/\npip-wheel-metadata/\nshare/python-wheels/\n*.egg-info/\n.installed.cfg\n*.egg\nMANIFEST\n\n# PyInstaller\n#  Usually these files are written by a python script from a template\n#  before PyInstaller builds the exe, so as to inject date/other infos into it.\n*.manifest\n*.spec\n\n# Installer logs\npip-log.txt\npip-delete-this-directory.txt\n\n# Unit test / coverage reports\nhtmlcov/\n.tox/\n.nox/\n.coverage\n.coverage.*\n.cache\nnosetests.xml\ncoverage.xml\n*.cover\n*.py,cover\n.hypothesis/\n.pytest_cache/\n\n# Translations\n*.mo\n*.pot\n\n# Django stuff:\n*.log\nlocal_settings.py\ndb.sqlite3\ndb.sqlite3-journal\n\n# Flask stuff:\ninstance/\n.webassets-cache\n\n# Scrapy stuff:\n.scrapy\n\n# Sphinx documentation\ndocs/_build/\n\n# PyBuilder\ntarget/\n\n# Jupyter Notebook\n.ipynb_checkpoints\n\n# IPython\nprofile_default/\nipython_config.py\n\n# pyenv\n.python-version\n\n# pipenv\n#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.\n#   However, in case of collaboration, if having platform-specific dependencies or dependencies\n#   having no cross-platform support, pipenv may install dependencies that don't work, or not\n#   install all needed dependencies.\n#Pipfile.lock\n\n# PEP 582; used by e.g. github.com/David-OConnor/pyflow\n__pypackages__/\n\n# Celery stuff\ncelerybeat-schedule\ncelerybeat.pid\n\n# SageMath parsed files\n*.sage.py\n\n# Environments\n.env\n.venv\nenv/\nvenv/\nENV/\nenv.bak/\nvenv.bak/\n\n# Spyder project settings\n.spyderproject\n.spyproject\n\n# Rope project settings\n.ropeproject\n\n# mkdocs documentation\n/site\n\n# mypy\n.mypy_cache/\n.dmypy.json\ndmypy.json\n\n# Pyre type checker\n.pyre/\n"
  },
  {
    "path": ".gitmodules",
    "content": "[submodule \"whisper\"]\n\tpath = whisper\n\turl = https://github.com/MiscellaneousStuff/whisper"
  },
  {
    "path": "Dockerfile",
    "content": "FROM python:3.9.14-bullseye\n\n# Install dependencies\nRUN apt-get update && apt-get install -y \\\n    ffmpeg \\\n && apt-get clean \\\n && rm -rf /var/lib/apt/lists/*\n\n# Install Whisper\nRUN git clone https://github.com/MiscellaneousStuff/openai-whisper-cpu.git \\\n && cd openai-whisper-cpu \\\n && git submodule init \\\n && git submodule update \\\n && pip install -e ./whisper\n\n# Install model files\nRUN whisper --model tiny dummy.wav; exit 0\nRUN whisper --model base dummy.wav; exit 0\nRUN whisper --model small dummy.wav; exit 0\nRUN whisper --model medium dummy.wav; exit 0\nRUN whisper --model large dummy.wav; exit 0\nRUN whisper --model tiny.en dummy.wav; exit 0\nRUN whisper --model base.en dummy.wav; exit 0\nRUN whisper --model small.en dummy.wav; exit 0\nRUN whisper --model medium.en dummy.wav; exit 0\n\nWORKDIR /usr/src/app\n\nCMD [\"whisper\",\"python3\"]\n"
  },
  {
    "path": "LICENSE",
    "content": "MIT License\n\nCopyright (c) 2022 MiscellaneousStuff\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\nSOFTWARE."
  },
  {
    "path": "README.md",
    "content": "# OpenAI Whisper - CPU\n\n## About\n\nExperiments applying quantization methods to OpenAI Whisper ASR model\nto improve the inference speed and throughput on CPU-based deployments.\nThis is motivated by the fact that, although the Whisper model greatly\nimproves the accessibility of SOTA ASR and doesn't require depending\non the cloud for high quality transcription, many end users can not\nrun this model out-of-the-box as most consumer computers only contain\nCPUs and do not contain high performance GPUs.\n\nThis could lead to allowing the larger Whisper models to run faster\non laptops without a GPU.\n\nHardware for experiments: \\\nCPU - AMD Ryzen 5 5600X \\\nRAM - 32GB DDR4 \\\nGPU - Nvidia GeForce RTX 3060 Ti \\\nHDD - M.2 SSD \n\n## Usage\n\nFirstly, get the fork of the OpenAI Whisper repo with the\nmodifications needed for CPU dynamic quantization:\n\n```bash\ngit submodule init\ngit submodule update\n```\n\nAnd then install the module using:\n\n```bash\npip install -e ./whisper\n```\n\n### Explanation\n\nQuantization of the Whisper model requires changing the `Linear()`\nlayers within the model to `nn.Linear()`. This is because you need\nto specifiy which layer types to dynamically quantize, such as:\n\n```python\nquantized_model = torch.quantization.quantize_dynamic(\n    model_fp32, {torch.nn.Linear}, dtype=torch.qint8\n)\n```\n\nHowever the whisper model is designed to be adaptable, i.e.\nit can run at different precisions, so the `Linear()` layer contains\ncustom code to account for this. However, this is not required for\nthe quantized model. You can either change the `Linear()` layers in\n\"/whisper/whisper/model.py\" yourself, or you can just use the above\ninstallation instructions.\n\n## Results\n\nTest audio is the first 30 seconds of: \\\nhttps://www.youtube.com/watch?v=oKOtzIo-uYw\n\n| Device | Whisper Model | Data Type | Linear Layer | Inference Time |\n| --- | --- | ----------- | --- | --- |\n| GPU | tiny | fp32 | Linear | 0.5 |\n| CPU | tiny  | fp32 | nn.Linear | 2.3 |\n| CPU | tiny  | qint8 (quant) | nn.Linear | 3.1 (0.74x slowdown) |\n\nTiny quantized model is 9.67x faster than real time. \\\nTiny quantized model is 0.74x slower than the original model.\n\n| Device | Whisper Model | Data Type | Linear Layer | Inference Time |\n| --- | --- | ----------- | --- | --- |\n| GPU | base | fp32 | Linear | 0.6 |\n| CPU | base  | fp32 | nn.Linear | 5.2 |\n| CPU | base  | qint8 (quant) | nn.Linear | 3.2 (1.62x speedup) |\n\nBase quantized model is 9.37x faster than real time. \\\nBase quantized model is 1.62x faster than the original model.\n\n| Device | Whisper Model | Data Type | Linear Layer | Inference Time |\n| --- | --- | ----------- | --- | --- |\n| GPU | small | fp32 | Linear | 0.7 |\n| CPU | small | fp32 | nn.Linear | 19.1s |\n| CPU | small | qint8 (quant) | nn.Linear | 6.9s (2.76x speedup) |\n\nSmall quantized model is 4.34x faster than real time. \\\nSmall quantized model is 2.76x faster than the original model.\n\n| Device | Whisper Model | Data Type | Linear Layer | Inference Time |\n| --- | --- | ----------- | --- | --- \n| GPU | medium | fp32 | Linear | 1.7s |\n| CPU | medium | fp32 | nn.Linear | 60.7 |\n| CPU | medium | qint8 (quant) | nn.Linear | 23.1 (2.62x speedup) |\n\nMedium quantized model is 1.29x faster than real time. \\\nMedium quantized model is 2.62x faster than the original model.\n\n# Docker\n\nBuild the docker image.   \n\n``` \ndocker build -t whisper-cpu . \n```\nRun the quantized model.   \n\n```\ndocker run --rm -v \"$(pwd)/audio\":/usr/src/app/audio -v \"$(pwd)/script\":/usr/src/app/script whisper-cpu python3 ./script/custom_whisper.py audio/path_to_dir_or_audio_file --language English --model medium.en \n```\n\n- ```-v \"$(pwd)/audio\":/usr/src/app/audio``` this creates a volume to give docker access to your audio files.\n- ```-v \"$(pwd)/script\":/usr/src/app/script``` this volume gives docker access to the custom start script. Transcription results are also stored here.\n\n- Note: you might want to adjust ```./script/custom_whisper.py``` for your own needs.\n\n"
  },
  {
    "path": "main.ipynb",
    "content": "{\n \"cells\": [\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"# OpenAI Whisper - CPU\\n\",\n    \"Improving CPU-deployment performance of OpenAI Whisper model, following this procedure:\\n\",\n    \"https://pytorch.org/assets/images/quantization-practice/quantization-flowchart2.png\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"## Load Model\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 1,\n   \"metadata\": {},\n   \"outputs\": [],\n   \"source\": [\n    \"import whisper\\n\",\n    \"import torch\\n\",\n    \"\\n\",\n    \"test_path  = \\\"C:\\\\\\\\Users\\\\\\\\win8t\\\\\\\\Music\\\\\\\\\\\"\\n\",\n    \"test_path += \\\"Fugees - Killing Me Softly With His Song (Official Video).mp3\\\"\\n\",\n    \"\\n\",\n    \"model_fp32 = whisper.load_model(\\n\",\n    \"    name=\\\"base\\\",\\n\",\n    \"    device=\\\"cpu\\\")\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"## Dynamically Quantize Model\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 2,\n   \"metadata\": {},\n   \"outputs\": [],\n   \"source\": [\n    \"quantized_model = torch.quantization.quantize_dynamic(\\n\",\n    \"    model_fp32, {torch.nn.Linear}, dtype=torch.qint8\\n\",\n    \")\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 3,\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"Size (MB): 290.459479\\n\",\n      \"Size (MB): 158.410839\\n\"\n     ]\n    },\n    {\n     \"data\": {\n      \"text/plain\": [\n       \"158.410839\"\n      ]\n     },\n     \"execution_count\": 3,\n     \"metadata\": {},\n     \"output_type\": \"execute_result\"\n    }\n   ],\n   \"source\": [\n    \"import os\\n\",\n    \"\\n\",\n    \"def print_size_of_model(model):\\n\",\n    \"    torch.save(model.state_dict(), \\\"temp.p\\\")\\n\",\n    \"    size = os.path.getsize(\\\"temp.p\\\")/1e6\\n\",\n    \"    print('Size (MB):', size)\\n\",\n    \"    os.remove('temp.p')\\n\",\n    \"    return size\\n\",\n    \"\\n\",\n    \"print_size_of_model(model_fp32)\\n\",\n    \"print_size_of_model(quantized_model)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"## Run Dynamically Quantized Model\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 4,\n   \"metadata\": {},\n   \"outputs\": [],\n   \"source\": [\n    \"audio = whisper.load_audio(test_path)\\n\",\n    \"audio = whisper.pad_or_trim(audio)\\n\",\n    \"\\n\",\n    \"mel   = whisper.log_mel_spectrogram(audio).to(model_fp32.device)\\n\",\n    \"options = whisper.DecodingOptions(fp16=False)\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 5,\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"Detected language: en\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"# regular\\n\",\n    \"_, probs = model_fp32.detect_language(mel)\\n\",\n    \"print(f\\\"Detected language: {max(probs, key=probs.get)}\\\")\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 6,\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"Detected language: en\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"# quantized\\n\",\n    \"_, probs = quantized_model.detect_language(mel)\\n\",\n    \"print(f\\\"Detected language: {max(probs, key=probs.get)}\\\")\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 7,\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stderr\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"c:\\\\users\\\\win8t\\\\onedrive\\\\desktop\\\\projects\\\\openai-whisper-cpu\\\\whisper\\\\whisper\\\\transcribe.py:76: UserWarning: Performing inference on CPU when CUDA is available\\n\",\n      \"  warnings.warn(\\\"Performing inference on CPU when CUDA is available\\\")\\n\",\n      \"c:\\\\users\\\\win8t\\\\onedrive\\\\desktop\\\\projects\\\\openai-whisper-cpu\\\\whisper\\\\whisper\\\\transcribe.py:78: UserWarning: FP16 is not supported on CPU; using FP32 instead\\n\",\n      \"  warnings.warn(\\\"FP16 is not supported on CPU; using FP32 instead\\\")\\n\"\n     ]\n    },\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \" Strum in my pain with his fingers, singing my life with his words. Killing me softly with his song, killing me softly with his song, telling my whole life. With his words killing me softly with his song. This is why I clap for refuge. I'll help you up in the prize where you sit on the base, sit on the beat. While I'm on this road, I got my girl, El. One time, one time, pay your El. You know you got the lyrics. I heard he sang a good song. I heard he had a style. And so I came to see him and listen for a while. And there he was, this young boy, straightened to my eyes. Strumming my pain with his finger, singing my life with his words. Killing me softly with his song, killing me softly with his song, telling my whole life. With his words killing me softly with his song. I felt all flush with the rust, and merrised by the crown. I felt he found my letter, and read each one out loud. I prayed that he would finish, but he just kept writing on. Strumming my pain with his finger, singing my life with his words. Killing me softly with his song, killing me softly with his song, telling my whole life. With his words killing me softly with his song, taking to the best of the world. La la la la la la la la la low, low. I'm alive He's throwing a pain with his finger Yes, he was singing my line with his wife He let me softly with his soul He let me softly hear his song telling my whole life\\n\",\n      \"Evaluate total time (seconds): 129.8\\n\"\n     ]\n    },\n    {\n     \"ename\": \"KeyboardInterrupt\",\n     \"evalue\": \"\",\n     \"output_type\": \"error\",\n     \"traceback\": [\n      \"\\u001b[1;31m---------------------------------------------------------------------------\\u001b[0m\",\n      \"\\u001b[1;31mKeyboardInterrupt\\u001b[0m                         Traceback (most recent call last)\",\n      \"\\u001b[1;32m~\\\\AppData\\\\Local\\\\Temp/ipykernel_9024/408565071.py\\u001b[0m in \\u001b[0;36m<module>\\u001b[1;34m\\u001b[0m\\n\\u001b[0;32m     13\\u001b[0m \\u001b[1;33m\\u001b[0m\\u001b[0m\\n\\u001b[0;32m     14\\u001b[0m \\u001b[1;31m# Evaluate the INT8 BERT model after the dynamic quantization\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[0m\\n\\u001b[1;32m---> 15\\u001b[1;33m \\u001b[0mtime_model_evaluation\\u001b[0m\\u001b[1;33m(\\u001b[0m\\u001b[0mquantized_model\\u001b[0m\\u001b[1;33m,\\u001b[0m \\u001b[0mmel\\u001b[0m\\u001b[1;33m,\\u001b[0m \\u001b[0moptions\\u001b[0m\\u001b[1;33m)\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[0m\\n\\u001b[0m\",\n      \"\\u001b[1;32m~\\\\AppData\\\\Local\\\\Temp/ipykernel_9024/408565071.py\\u001b[0m in \\u001b[0;36mtime_model_evaluation\\u001b[1;34m(model, mel, options)\\u001b[0m\\n\\u001b[0;32m      3\\u001b[0m     \\u001b[0meval_start_time\\u001b[0m \\u001b[1;33m=\\u001b[0m \\u001b[0mtime\\u001b[0m\\u001b[1;33m.\\u001b[0m\\u001b[0mtime\\u001b[0m\\u001b[1;33m(\\u001b[0m\\u001b[1;33m)\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[0m\\n\\u001b[0;32m      4\\u001b[0m     \\u001b[1;31m# result = whisper.decode(model, mel, options)\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[0m\\n\\u001b[1;32m----> 5\\u001b[1;33m     \\u001b[0mresult\\u001b[0m \\u001b[1;33m=\\u001b[0m \\u001b[0mwhisper\\u001b[0m\\u001b[1;33m.\\u001b[0m\\u001b[0mtranscribe\\u001b[0m\\u001b[1;33m(\\u001b[0m\\u001b[0mmodel\\u001b[0m\\u001b[1;33m,\\u001b[0m \\u001b[0mtest_path\\u001b[0m\\u001b[1;33m)\\u001b[0m \\u001b[1;31m# , options)\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[0m\\n\\u001b[0m\\u001b[0;32m      6\\u001b[0m     \\u001b[0meval_end_time\\u001b[0m \\u001b[1;33m=\\u001b[0m \\u001b[0mtime\\u001b[0m\\u001b[1;33m.\\u001b[0m\\u001b[0mtime\\u001b[0m\\u001b[1;33m(\\u001b[0m\\u001b[1;33m)\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[0m\\n\\u001b[0;32m      7\\u001b[0m     \\u001b[0meval_duration_time\\u001b[0m \\u001b[1;33m=\\u001b[0m \\u001b[0meval_end_time\\u001b[0m \\u001b[1;33m-\\u001b[0m \\u001b[0meval_start_time\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[0m\\n\",\n      \"\\u001b[1;32mc:\\\\users\\\\win8t\\\\onedrive\\\\desktop\\\\projects\\\\openai-whisper-cpu\\\\whisper\\\\whisper\\\\transcribe.py\\u001b[0m in \\u001b[0;36mtranscribe\\u001b[1;34m(model, audio, verbose, temperature, compression_ratio_threshold, logprob_threshold, no_speech_threshold, condition_on_previous_text, **decode_options)\\u001b[0m\\n\\u001b[0;32m    180\\u001b[0m \\u001b[1;33m\\u001b[0m\\u001b[0m\\n\\u001b[0;32m    181\\u001b[0m             \\u001b[0mdecode_options\\u001b[0m\\u001b[1;33m[\\u001b[0m\\u001b[1;34m\\\"prompt\\\"\\u001b[0m\\u001b[1;33m]\\u001b[0m \\u001b[1;33m=\\u001b[0m \\u001b[0mall_tokens\\u001b[0m\\u001b[1;33m[\\u001b[0m\\u001b[0mprompt_reset_since\\u001b[0m\\u001b[1;33m:\\u001b[0m\\u001b[1;33m]\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[0m\\n\\u001b[1;32m--> 182\\u001b[1;33m             \\u001b[0mresult\\u001b[0m \\u001b[1;33m=\\u001b[0m \\u001b[0mdecode_with_fallback\\u001b[0m\\u001b[1;33m(\\u001b[0m\\u001b[0msegment\\u001b[0m\\u001b[1;33m)\\u001b[0m\\u001b[1;33m[\\u001b[0m\\u001b[1;36m0\\u001b[0m\\u001b[1;33m]\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[0m\\n\\u001b[0m\\u001b[0;32m    183\\u001b[0m             \\u001b[0mtokens\\u001b[0m \\u001b[1;33m=\\u001b[0m \\u001b[0mtorch\\u001b[0m\\u001b[1;33m.\\u001b[0m\\u001b[0mtensor\\u001b[0m\\u001b[1;33m(\\u001b[0m\\u001b[0mresult\\u001b[0m\\u001b[1;33m.\\u001b[0m\\u001b[0mtokens\\u001b[0m\\u001b[1;33m)\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[0m\\n\\u001b[0;32m    184\\u001b[0m \\u001b[1;33m\\u001b[0m\\u001b[0m\\n\",\n      \"\\u001b[1;32mc:\\\\users\\\\win8t\\\\onedrive\\\\desktop\\\\projects\\\\openai-whisper-cpu\\\\whisper\\\\whisper\\\\transcribe.py\\u001b[0m in \\u001b[0;36mdecode_with_fallback\\u001b[1;34m(segment)\\u001b[0m\\n\\u001b[0;32m    123\\u001b[0m             \\u001b[1;32mif\\u001b[0m \\u001b[0many\\u001b[0m\\u001b[1;33m(\\u001b[0m\\u001b[0mneeds_fallback\\u001b[0m\\u001b[1;33m)\\u001b[0m\\u001b[1;33m:\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[0m\\n\\u001b[0;32m    124\\u001b[0m                 \\u001b[0moptions\\u001b[0m \\u001b[1;33m=\\u001b[0m \\u001b[0mDecodingOptions\\u001b[0m\\u001b[1;33m(\\u001b[0m\\u001b[1;33m**\\u001b[0m\\u001b[0mkwargs\\u001b[0m\\u001b[1;33m,\\u001b[0m \\u001b[0mtemperature\\u001b[0m\\u001b[1;33m=\\u001b[0m\\u001b[0mt\\u001b[0m\\u001b[1;33m)\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[0m\\n\\u001b[1;32m--> 125\\u001b[1;33m                 \\u001b[0mretries\\u001b[0m \\u001b[1;33m=\\u001b[0m \\u001b[0mmodel\\u001b[0m\\u001b[1;33m.\\u001b[0m\\u001b[0mdecode\\u001b[0m\\u001b[1;33m(\\u001b[0m\\u001b[0msegment\\u001b[0m\\u001b[1;33m[\\u001b[0m\\u001b[0mneeds_fallback\\u001b[0m\\u001b[1;33m]\\u001b[0m\\u001b[1;33m,\\u001b[0m \\u001b[0moptions\\u001b[0m\\u001b[1;33m)\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[0m\\n\\u001b[0m\\u001b[0;32m    126\\u001b[0m                 \\u001b[1;32mfor\\u001b[0m \\u001b[0mretry_index\\u001b[0m\\u001b[1;33m,\\u001b[0m \\u001b[0moriginal_index\\u001b[0m \\u001b[1;32min\\u001b[0m \\u001b[0menumerate\\u001b[0m\\u001b[1;33m(\\u001b[0m\\u001b[0mnp\\u001b[0m\\u001b[1;33m.\\u001b[0m\\u001b[0mnonzero\\u001b[0m\\u001b[1;33m(\\u001b[0m\\u001b[0mneeds_fallback\\u001b[0m\\u001b[1;33m)\\u001b[0m\\u001b[1;33m[\\u001b[0m\\u001b[1;36m0\\u001b[0m\\u001b[1;33m]\\u001b[0m\\u001b[1;33m)\\u001b[0m\\u001b[1;33m:\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[0m\\n\\u001b[0;32m    127\\u001b[0m                     \\u001b[0mresults\\u001b[0m\\u001b[1;33m[\\u001b[0m\\u001b[0moriginal_index\\u001b[0m\\u001b[1;33m]\\u001b[0m \\u001b[1;33m=\\u001b[0m \\u001b[0mretries\\u001b[0m\\u001b[1;33m[\\u001b[0m\\u001b[0mretry_index\\u001b[0m\\u001b[1;33m]\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[0m\\n\",\n      \"\\u001b[1;32mc:\\\\Users\\\\win8t\\\\AppData\\\\Local\\\\Programs\\\\Python\\\\Python39\\\\lib\\\\site-packages\\\\torch\\\\autograd\\\\grad_mode.py\\u001b[0m in \\u001b[0;36mdecorate_context\\u001b[1;34m(*args, **kwargs)\\u001b[0m\\n\\u001b[0;32m     26\\u001b[0m         \\u001b[1;32mdef\\u001b[0m \\u001b[0mdecorate_context\\u001b[0m\\u001b[1;33m(\\u001b[0m\\u001b[1;33m*\\u001b[0m\\u001b[0margs\\u001b[0m\\u001b[1;33m,\\u001b[0m \\u001b[1;33m**\\u001b[0m\\u001b[0mkwargs\\u001b[0m\\u001b[1;33m)\\u001b[0m\\u001b[1;33m:\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[0m\\n\\u001b[0;32m     27\\u001b[0m             \\u001b[1;32mwith\\u001b[0m \\u001b[0mself\\u001b[0m\\u001b[1;33m.\\u001b[0m\\u001b[0m__class__\\u001b[0m\\u001b[1;33m(\\u001b[0m\\u001b[1;33m)\\u001b[0m\\u001b[1;33m:\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[0m\\n\\u001b[1;32m---> 28\\u001b[1;33m                 \\u001b[1;32mreturn\\u001b[0m \\u001b[0mfunc\\u001b[0m\\u001b[1;33m(\\u001b[0m\\u001b[1;33m*\\u001b[0m\\u001b[0margs\\u001b[0m\\u001b[1;33m,\\u001b[0m \\u001b[1;33m**\\u001b[0m\\u001b[0mkwargs\\u001b[0m\\u001b[1;33m)\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[0m\\n\\u001b[0m\\u001b[0;32m     29\\u001b[0m         \\u001b[1;32mreturn\\u001b[0m \\u001b[0mcast\\u001b[0m\\u001b[1;33m(\\u001b[0m\\u001b[0mF\\u001b[0m\\u001b[1;33m,\\u001b[0m \\u001b[0mdecorate_context\\u001b[0m\\u001b[1;33m)\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[0m\\n\\u001b[0;32m     30\\u001b[0m \\u001b[1;33m\\u001b[0m\\u001b[0m\\n\",\n      \"\\u001b[1;32mc:\\\\users\\\\win8t\\\\onedrive\\\\desktop\\\\projects\\\\openai-whisper-cpu\\\\whisper\\\\whisper\\\\decoding.py\\u001b[0m in \\u001b[0;36mdecode\\u001b[1;34m(model, mel, options)\\u001b[0m\\n\\u001b[0;32m    697\\u001b[0m         \\u001b[0mmel\\u001b[0m \\u001b[1;33m=\\u001b[0m \\u001b[0mmel\\u001b[0m\\u001b[1;33m.\\u001b[0m\\u001b[0munsqueeze\\u001b[0m\\u001b[1;33m(\\u001b[0m\\u001b[1;36m0\\u001b[0m\\u001b[1;33m)\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[0m\\n\\u001b[0;32m    698\\u001b[0m \\u001b[1;33m\\u001b[0m\\u001b[0m\\n\\u001b[1;32m--> 699\\u001b[1;33m     \\u001b[0mresult\\u001b[0m \\u001b[1;33m=\\u001b[0m \\u001b[0mDecodingTask\\u001b[0m\\u001b[1;33m(\\u001b[0m\\u001b[0mmodel\\u001b[0m\\u001b[1;33m,\\u001b[0m \\u001b[0moptions\\u001b[0m\\u001b[1;33m)\\u001b[0m\\u001b[1;33m.\\u001b[0m\\u001b[0mrun\\u001b[0m\\u001b[1;33m(\\u001b[0m\\u001b[0mmel\\u001b[0m\\u001b[1;33m)\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[0m\\n\\u001b[0m\\u001b[0;32m    700\\u001b[0m \\u001b[1;33m\\u001b[0m\\u001b[0m\\n\\u001b[0;32m    701\\u001b[0m     \\u001b[1;32mif\\u001b[0m \\u001b[0msingle\\u001b[0m\\u001b[1;33m:\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[0m\\n\",\n      \"\\u001b[1;32mc:\\\\Users\\\\win8t\\\\AppData\\\\Local\\\\Programs\\\\Python\\\\Python39\\\\lib\\\\site-packages\\\\torch\\\\autograd\\\\grad_mode.py\\u001b[0m in \\u001b[0;36mdecorate_context\\u001b[1;34m(*args, **kwargs)\\u001b[0m\\n\\u001b[0;32m     26\\u001b[0m         \\u001b[1;32mdef\\u001b[0m \\u001b[0mdecorate_context\\u001b[0m\\u001b[1;33m(\\u001b[0m\\u001b[1;33m*\\u001b[0m\\u001b[0margs\\u001b[0m\\u001b[1;33m,\\u001b[0m \\u001b[1;33m**\\u001b[0m\\u001b[0mkwargs\\u001b[0m\\u001b[1;33m)\\u001b[0m\\u001b[1;33m:\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[0m\\n\\u001b[0;32m     27\\u001b[0m             \\u001b[1;32mwith\\u001b[0m \\u001b[0mself\\u001b[0m\\u001b[1;33m.\\u001b[0m\\u001b[0m__class__\\u001b[0m\\u001b[1;33m(\\u001b[0m\\u001b[1;33m)\\u001b[0m\\u001b[1;33m:\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[0m\\n\\u001b[1;32m---> 28\\u001b[1;33m                 \\u001b[1;32mreturn\\u001b[0m \\u001b[0mfunc\\u001b[0m\\u001b[1;33m(\\u001b[0m\\u001b[1;33m*\\u001b[0m\\u001b[0margs\\u001b[0m\\u001b[1;33m,\\u001b[0m \\u001b[1;33m**\\u001b[0m\\u001b[0mkwargs\\u001b[0m\\u001b[1;33m)\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[0m\\n\\u001b[0m\\u001b[0;32m     29\\u001b[0m         \\u001b[1;32mreturn\\u001b[0m \\u001b[0mcast\\u001b[0m\\u001b[1;33m(\\u001b[0m\\u001b[0mF\\u001b[0m\\u001b[1;33m,\\u001b[0m \\u001b[0mdecorate_context\\u001b[0m\\u001b[1;33m)\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[0m\\n\\u001b[0;32m     30\\u001b[0m \\u001b[1;33m\\u001b[0m\\u001b[0m\\n\",\n      \"\\u001b[1;32mc:\\\\users\\\\win8t\\\\onedrive\\\\desktop\\\\projects\\\\openai-whisper-cpu\\\\whisper\\\\whisper\\\\decoding.py\\u001b[0m in \\u001b[0;36mrun\\u001b[1;34m(self, mel)\\u001b[0m\\n\\u001b[0;32m    629\\u001b[0m \\u001b[1;33m\\u001b[0m\\u001b[0m\\n\\u001b[0;32m    630\\u001b[0m         \\u001b[1;31m# call the main sampling loop\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[0m\\n\\u001b[1;32m--> 631\\u001b[1;33m         \\u001b[0mtokens\\u001b[0m\\u001b[1;33m,\\u001b[0m \\u001b[0msum_logprobs\\u001b[0m\\u001b[1;33m,\\u001b[0m \\u001b[0mno_speech_probs\\u001b[0m \\u001b[1;33m=\\u001b[0m \\u001b[0mself\\u001b[0m\\u001b[1;33m.\\u001b[0m\\u001b[0m_main_loop\\u001b[0m\\u001b[1;33m(\\u001b[0m\\u001b[0maudio_features\\u001b[0m\\u001b[1;33m,\\u001b[0m \\u001b[0mtokens\\u001b[0m\\u001b[1;33m)\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[0m\\n\\u001b[0m\\u001b[0;32m    632\\u001b[0m \\u001b[1;33m\\u001b[0m\\u001b[0m\\n\\u001b[0;32m    633\\u001b[0m         \\u001b[1;31m# reshape the tensors to have (n_audio, n_group) as the first two dimensions\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[0m\\n\",\n      \"\\u001b[1;32mc:\\\\users\\\\win8t\\\\onedrive\\\\desktop\\\\projects\\\\openai-whisper-cpu\\\\whisper\\\\whisper\\\\decoding.py\\u001b[0m in \\u001b[0;36m_main_loop\\u001b[1;34m(self, audio_features, tokens)\\u001b[0m\\n\\u001b[0;32m    584\\u001b[0m         \\u001b[1;32mtry\\u001b[0m\\u001b[1;33m:\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[0m\\n\\u001b[0;32m    585\\u001b[0m             \\u001b[1;32mfor\\u001b[0m \\u001b[0mi\\u001b[0m \\u001b[1;32min\\u001b[0m \\u001b[0mrange\\u001b[0m\\u001b[1;33m(\\u001b[0m\\u001b[0mself\\u001b[0m\\u001b[1;33m.\\u001b[0m\\u001b[0msample_len\\u001b[0m\\u001b[1;33m)\\u001b[0m\\u001b[1;33m:\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[0m\\n\\u001b[1;32m--> 586\\u001b[1;33m                 \\u001b[0mlogits\\u001b[0m \\u001b[1;33m=\\u001b[0m \\u001b[0mself\\u001b[0m\\u001b[1;33m.\\u001b[0m\\u001b[0minference\\u001b[0m\\u001b[1;33m.\\u001b[0m\\u001b[0mlogits\\u001b[0m\\u001b[1;33m(\\u001b[0m\\u001b[0mtokens\\u001b[0m\\u001b[1;33m,\\u001b[0m \\u001b[0maudio_features\\u001b[0m\\u001b[1;33m)\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[0m\\n\\u001b[0m\\u001b[0;32m    587\\u001b[0m \\u001b[1;33m\\u001b[0m\\u001b[0m\\n\\u001b[0;32m    588\\u001b[0m                 \\u001b[1;32mif\\u001b[0m \\u001b[0mi\\u001b[0m \\u001b[1;33m==\\u001b[0m \\u001b[1;36m0\\u001b[0m \\u001b[1;32mand\\u001b[0m \\u001b[0mself\\u001b[0m\\u001b[1;33m.\\u001b[0m\\u001b[0mtokenizer\\u001b[0m\\u001b[1;33m.\\u001b[0m\\u001b[0mno_speech\\u001b[0m \\u001b[1;32mis\\u001b[0m \\u001b[1;32mnot\\u001b[0m \\u001b[1;32mNone\\u001b[0m\\u001b[1;33m:\\u001b[0m  \\u001b[1;31m# save no_speech_probs\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[0m\\n\",\n      \"\\u001b[1;32mc:\\\\users\\\\win8t\\\\onedrive\\\\desktop\\\\projects\\\\openai-whisper-cpu\\\\whisper\\\\whisper\\\\decoding.py\\u001b[0m in \\u001b[0;36mlogits\\u001b[1;34m(self, tokens, audio_features)\\u001b[0m\\n\\u001b[0;32m    143\\u001b[0m             \\u001b[0mtokens\\u001b[0m \\u001b[1;33m=\\u001b[0m \\u001b[0mtokens\\u001b[0m\\u001b[1;33m[\\u001b[0m\\u001b[1;33m:\\u001b[0m\\u001b[1;33m,\\u001b[0m \\u001b[1;33m-\\u001b[0m\\u001b[1;36m1\\u001b[0m\\u001b[1;33m:\\u001b[0m\\u001b[1;33m]\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[0m\\n\\u001b[0;32m    144\\u001b[0m \\u001b[1;33m\\u001b[0m\\u001b[0m\\n\\u001b[1;32m--> 145\\u001b[1;33m         \\u001b[1;32mreturn\\u001b[0m \\u001b[0mself\\u001b[0m\\u001b[1;33m.\\u001b[0m\\u001b[0mmodel\\u001b[0m\\u001b[1;33m.\\u001b[0m\\u001b[0mdecoder\\u001b[0m\\u001b[1;33m(\\u001b[0m\\u001b[0mtokens\\u001b[0m\\u001b[1;33m,\\u001b[0m \\u001b[0maudio_features\\u001b[0m\\u001b[1;33m,\\u001b[0m \\u001b[0mkv_cache\\u001b[0m\\u001b[1;33m=\\u001b[0m\\u001b[0mself\\u001b[0m\\u001b[1;33m.\\u001b[0m\\u001b[0mkv_cache\\u001b[0m\\u001b[1;33m)\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[0m\\n\\u001b[0m\\u001b[0;32m    146\\u001b[0m \\u001b[1;33m\\u001b[0m\\u001b[0m\\n\\u001b[0;32m    147\\u001b[0m     \\u001b[1;32mdef\\u001b[0m \\u001b[0mcleanup_caching\\u001b[0m\\u001b[1;33m(\\u001b[0m\\u001b[0mself\\u001b[0m\\u001b[1;33m)\\u001b[0m\\u001b[1;33m:\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[0m\\n\",\n      \"\\u001b[1;32mc:\\\\Users\\\\win8t\\\\AppData\\\\Local\\\\Programs\\\\Python\\\\Python39\\\\lib\\\\site-packages\\\\torch\\\\nn\\\\modules\\\\module.py\\u001b[0m in \\u001b[0;36m_call_impl\\u001b[1;34m(self, *input, **kwargs)\\u001b[0m\\n\\u001b[0;32m   1100\\u001b[0m         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks\\n\\u001b[0;32m   1101\\u001b[0m                 or _global_forward_hooks or _global_forward_pre_hooks):\\n\\u001b[1;32m-> 1102\\u001b[1;33m             \\u001b[1;32mreturn\\u001b[0m \\u001b[0mforward_call\\u001b[0m\\u001b[1;33m(\\u001b[0m\\u001b[1;33m*\\u001b[0m\\u001b[0minput\\u001b[0m\\u001b[1;33m,\\u001b[0m \\u001b[1;33m**\\u001b[0m\\u001b[0mkwargs\\u001b[0m\\u001b[1;33m)\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[0m\\n\\u001b[0m\\u001b[0;32m   1103\\u001b[0m         \\u001b[1;31m# Do not call functions when jit is used\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[0m\\n\\u001b[0;32m   1104\\u001b[0m         \\u001b[0mfull_backward_hooks\\u001b[0m\\u001b[1;33m,\\u001b[0m \\u001b[0mnon_full_backward_hooks\\u001b[0m \\u001b[1;33m=\\u001b[0m \\u001b[1;33m[\\u001b[0m\\u001b[1;33m]\\u001b[0m\\u001b[1;33m,\\u001b[0m \\u001b[1;33m[\\u001b[0m\\u001b[1;33m]\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[0m\\n\",\n      \"\\u001b[1;32mc:\\\\users\\\\win8t\\\\onedrive\\\\desktop\\\\projects\\\\openai-whisper-cpu\\\\whisper\\\\whisper\\\\model.py\\u001b[0m in \\u001b[0;36mforward\\u001b[1;34m(self, x, xa, kv_cache)\\u001b[0m\\n\\u001b[0;32m    187\\u001b[0m \\u001b[1;33m\\u001b[0m\\u001b[0m\\n\\u001b[0;32m    188\\u001b[0m         \\u001b[1;32mfor\\u001b[0m \\u001b[0mblock\\u001b[0m \\u001b[1;32min\\u001b[0m \\u001b[0mself\\u001b[0m\\u001b[1;33m.\\u001b[0m\\u001b[0mblocks\\u001b[0m\\u001b[1;33m:\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[0m\\n\\u001b[1;32m--> 189\\u001b[1;33m             \\u001b[0mx\\u001b[0m \\u001b[1;33m=\\u001b[0m \\u001b[0mblock\\u001b[0m\\u001b[1;33m(\\u001b[0m\\u001b[0mx\\u001b[0m\\u001b[1;33m,\\u001b[0m \\u001b[0mxa\\u001b[0m\\u001b[1;33m,\\u001b[0m \\u001b[0mmask\\u001b[0m\\u001b[1;33m=\\u001b[0m\\u001b[0mself\\u001b[0m\\u001b[1;33m.\\u001b[0m\\u001b[0mmask\\u001b[0m\\u001b[1;33m,\\u001b[0m \\u001b[0mkv_cache\\u001b[0m\\u001b[1;33m=\\u001b[0m\\u001b[0mkv_cache\\u001b[0m\\u001b[1;33m)\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[0m\\n\\u001b[0m\\u001b[0;32m    190\\u001b[0m \\u001b[1;33m\\u001b[0m\\u001b[0m\\n\\u001b[0;32m    191\\u001b[0m         \\u001b[0mx\\u001b[0m \\u001b[1;33m=\\u001b[0m \\u001b[0mself\\u001b[0m\\u001b[1;33m.\\u001b[0m\\u001b[0mln\\u001b[0m\\u001b[1;33m(\\u001b[0m\\u001b[0mx\\u001b[0m\\u001b[1;33m)\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[0m\\n\",\n      \"\\u001b[1;32mc:\\\\Users\\\\win8t\\\\AppData\\\\Local\\\\Programs\\\\Python\\\\Python39\\\\lib\\\\site-packages\\\\torch\\\\nn\\\\modules\\\\module.py\\u001b[0m in \\u001b[0;36m_call_impl\\u001b[1;34m(self, *input, **kwargs)\\u001b[0m\\n\\u001b[0;32m   1100\\u001b[0m         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks\\n\\u001b[0;32m   1101\\u001b[0m                 or _global_forward_hooks or _global_forward_pre_hooks):\\n\\u001b[1;32m-> 1102\\u001b[1;33m             \\u001b[1;32mreturn\\u001b[0m \\u001b[0mforward_call\\u001b[0m\\u001b[1;33m(\\u001b[0m\\u001b[1;33m*\\u001b[0m\\u001b[0minput\\u001b[0m\\u001b[1;33m,\\u001b[0m \\u001b[1;33m**\\u001b[0m\\u001b[0mkwargs\\u001b[0m\\u001b[1;33m)\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[0m\\n\\u001b[0m\\u001b[0;32m   1103\\u001b[0m         \\u001b[1;31m# Do not call functions when jit is used\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[0m\\n\\u001b[0;32m   1104\\u001b[0m         \\u001b[0mfull_backward_hooks\\u001b[0m\\u001b[1;33m,\\u001b[0m \\u001b[0mnon_full_backward_hooks\\u001b[0m \\u001b[1;33m=\\u001b[0m \\u001b[1;33m[\\u001b[0m\\u001b[1;33m]\\u001b[0m\\u001b[1;33m,\\u001b[0m \\u001b[1;33m[\\u001b[0m\\u001b[1;33m]\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[0m\\n\",\n      \"\\u001b[1;32mc:\\\\users\\\\win8t\\\\onedrive\\\\desktop\\\\projects\\\\openai-whisper-cpu\\\\whisper\\\\whisper\\\\model.py\\u001b[0m in \\u001b[0;36mforward\\u001b[1;34m(self, x, xa, mask, kv_cache)\\u001b[0m\\n\\u001b[0;32m    124\\u001b[0m         \\u001b[0mx\\u001b[0m \\u001b[1;33m=\\u001b[0m \\u001b[0mx\\u001b[0m \\u001b[1;33m+\\u001b[0m \\u001b[0mself\\u001b[0m\\u001b[1;33m.\\u001b[0m\\u001b[0mattn\\u001b[0m\\u001b[1;33m(\\u001b[0m\\u001b[0mself\\u001b[0m\\u001b[1;33m.\\u001b[0m\\u001b[0mattn_ln\\u001b[0m\\u001b[1;33m(\\u001b[0m\\u001b[0mx\\u001b[0m\\u001b[1;33m)\\u001b[0m\\u001b[1;33m,\\u001b[0m \\u001b[0mmask\\u001b[0m\\u001b[1;33m=\\u001b[0m\\u001b[0mmask\\u001b[0m\\u001b[1;33m,\\u001b[0m \\u001b[0mkv_cache\\u001b[0m\\u001b[1;33m=\\u001b[0m\\u001b[0mkv_cache\\u001b[0m\\u001b[1;33m)\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[0m\\n\\u001b[0;32m    125\\u001b[0m         \\u001b[1;32mif\\u001b[0m \\u001b[0mself\\u001b[0m\\u001b[1;33m.\\u001b[0m\\u001b[0mcross_attn\\u001b[0m\\u001b[1;33m:\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[0m\\n\\u001b[1;32m--> 126\\u001b[1;33m             \\u001b[0mx\\u001b[0m \\u001b[1;33m=\\u001b[0m \\u001b[0mx\\u001b[0m \\u001b[1;33m+\\u001b[0m \\u001b[0mself\\u001b[0m\\u001b[1;33m.\\u001b[0m\\u001b[0mcross_attn\\u001b[0m\\u001b[1;33m(\\u001b[0m\\u001b[0mself\\u001b[0m\\u001b[1;33m.\\u001b[0m\\u001b[0mcross_attn_ln\\u001b[0m\\u001b[1;33m(\\u001b[0m\\u001b[0mx\\u001b[0m\\u001b[1;33m)\\u001b[0m\\u001b[1;33m,\\u001b[0m \\u001b[0mxa\\u001b[0m\\u001b[1;33m,\\u001b[0m \\u001b[0mkv_cache\\u001b[0m\\u001b[1;33m=\\u001b[0m\\u001b[0mkv_cache\\u001b[0m\\u001b[1;33m)\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[0m\\n\\u001b[0m\\u001b[0;32m    127\\u001b[0m         \\u001b[0mx\\u001b[0m \\u001b[1;33m=\\u001b[0m \\u001b[0mx\\u001b[0m \\u001b[1;33m+\\u001b[0m \\u001b[0mself\\u001b[0m\\u001b[1;33m.\\u001b[0m\\u001b[0mmlp\\u001b[0m\\u001b[1;33m(\\u001b[0m\\u001b[0mself\\u001b[0m\\u001b[1;33m.\\u001b[0m\\u001b[0mmlp_ln\\u001b[0m\\u001b[1;33m(\\u001b[0m\\u001b[0mx\\u001b[0m\\u001b[1;33m)\\u001b[0m\\u001b[1;33m)\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[0m\\n\\u001b[0;32m    128\\u001b[0m         \\u001b[1;32mreturn\\u001b[0m \\u001b[0mx\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[0m\\n\",\n      \"\\u001b[1;32mc:\\\\Users\\\\win8t\\\\AppData\\\\Local\\\\Programs\\\\Python\\\\Python39\\\\lib\\\\site-packages\\\\torch\\\\nn\\\\modules\\\\module.py\\u001b[0m in \\u001b[0;36m_call_impl\\u001b[1;34m(self, *input, **kwargs)\\u001b[0m\\n\\u001b[0;32m   1100\\u001b[0m         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks\\n\\u001b[0;32m   1101\\u001b[0m                 or _global_forward_hooks or _global_forward_pre_hooks):\\n\\u001b[1;32m-> 1102\\u001b[1;33m             \\u001b[1;32mreturn\\u001b[0m \\u001b[0mforward_call\\u001b[0m\\u001b[1;33m(\\u001b[0m\\u001b[1;33m*\\u001b[0m\\u001b[0minput\\u001b[0m\\u001b[1;33m,\\u001b[0m \\u001b[1;33m**\\u001b[0m\\u001b[0mkwargs\\u001b[0m\\u001b[1;33m)\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[0m\\n\\u001b[0m\\u001b[0;32m   1103\\u001b[0m         \\u001b[1;31m# Do not call functions when jit is used\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[0m\\n\\u001b[0;32m   1104\\u001b[0m         \\u001b[0mfull_backward_hooks\\u001b[0m\\u001b[1;33m,\\u001b[0m \\u001b[0mnon_full_backward_hooks\\u001b[0m \\u001b[1;33m=\\u001b[0m \\u001b[1;33m[\\u001b[0m\\u001b[1;33m]\\u001b[0m\\u001b[1;33m,\\u001b[0m \\u001b[1;33m[\\u001b[0m\\u001b[1;33m]\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[0m\\n\",\n      \"\\u001b[1;32mc:\\\\users\\\\win8t\\\\onedrive\\\\desktop\\\\projects\\\\openai-whisper-cpu\\\\whisper\\\\whisper\\\\model.py\\u001b[0m in \\u001b[0;36mforward\\u001b[1;34m(self, x, xa, mask, kv_cache)\\u001b[0m\\n\\u001b[0;32m     71\\u001b[0m         \\u001b[0mkv_cache\\u001b[0m\\u001b[1;33m:\\u001b[0m \\u001b[0mOptional\\u001b[0m\\u001b[1;33m[\\u001b[0m\\u001b[0mdict\\u001b[0m\\u001b[1;33m]\\u001b[0m \\u001b[1;33m=\\u001b[0m \\u001b[1;32mNone\\u001b[0m\\u001b[1;33m,\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[0m\\n\\u001b[0;32m     72\\u001b[0m     ):\\n\\u001b[1;32m---> 73\\u001b[1;33m         \\u001b[0mq\\u001b[0m \\u001b[1;33m=\\u001b[0m \\u001b[0mself\\u001b[0m\\u001b[1;33m.\\u001b[0m\\u001b[0mquery\\u001b[0m\\u001b[1;33m(\\u001b[0m\\u001b[0mx\\u001b[0m\\u001b[1;33m)\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[0m\\n\\u001b[0m\\u001b[0;32m     74\\u001b[0m \\u001b[1;33m\\u001b[0m\\u001b[0m\\n\\u001b[0;32m     75\\u001b[0m         \\u001b[1;32mif\\u001b[0m \\u001b[0mkv_cache\\u001b[0m \\u001b[1;32mis\\u001b[0m \\u001b[1;32mNone\\u001b[0m \\u001b[1;32mor\\u001b[0m \\u001b[0mxa\\u001b[0m \\u001b[1;32mis\\u001b[0m \\u001b[1;32mNone\\u001b[0m\\u001b[1;33m:\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[0m\\n\",\n      \"\\u001b[1;32mc:\\\\Users\\\\win8t\\\\AppData\\\\Local\\\\Programs\\\\Python\\\\Python39\\\\lib\\\\site-packages\\\\torch\\\\nn\\\\modules\\\\module.py\\u001b[0m in \\u001b[0;36m_call_impl\\u001b[1;34m(self, *input, **kwargs)\\u001b[0m\\n\\u001b[0;32m   1100\\u001b[0m         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks\\n\\u001b[0;32m   1101\\u001b[0m                 or _global_forward_hooks or _global_forward_pre_hooks):\\n\\u001b[1;32m-> 1102\\u001b[1;33m             \\u001b[1;32mreturn\\u001b[0m \\u001b[0mforward_call\\u001b[0m\\u001b[1;33m(\\u001b[0m\\u001b[1;33m*\\u001b[0m\\u001b[0minput\\u001b[0m\\u001b[1;33m,\\u001b[0m \\u001b[1;33m**\\u001b[0m\\u001b[0mkwargs\\u001b[0m\\u001b[1;33m)\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[0m\\n\\u001b[0m\\u001b[0;32m   1103\\u001b[0m         \\u001b[1;31m# Do not call functions when jit is used\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[0m\\n\\u001b[0;32m   1104\\u001b[0m         \\u001b[0mfull_backward_hooks\\u001b[0m\\u001b[1;33m,\\u001b[0m \\u001b[0mnon_full_backward_hooks\\u001b[0m \\u001b[1;33m=\\u001b[0m \\u001b[1;33m[\\u001b[0m\\u001b[1;33m]\\u001b[0m\\u001b[1;33m,\\u001b[0m \\u001b[1;33m[\\u001b[0m\\u001b[1;33m]\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[0m\\n\",\n      \"\\u001b[1;32mc:\\\\Users\\\\win8t\\\\AppData\\\\Local\\\\Programs\\\\Python\\\\Python39\\\\lib\\\\site-packages\\\\torch\\\\nn\\\\quantized\\\\dynamic\\\\modules\\\\linear.py\\u001b[0m in \\u001b[0;36mforward\\u001b[1;34m(self, x)\\u001b[0m\\n\\u001b[0;32m     46\\u001b[0m                     x, self._packed_params._packed_params)\\n\\u001b[0;32m     47\\u001b[0m             \\u001b[1;32melse\\u001b[0m\\u001b[1;33m:\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[0m\\n\\u001b[1;32m---> 48\\u001b[1;33m                 Y = torch.ops.quantized.linear_dynamic(\\n\\u001b[0m\\u001b[0;32m     49\\u001b[0m                     x, self._packed_params._packed_params, reduce_range=True)\\n\\u001b[0;32m     50\\u001b[0m         \\u001b[1;32melif\\u001b[0m \\u001b[0mself\\u001b[0m\\u001b[1;33m.\\u001b[0m\\u001b[0m_packed_params\\u001b[0m\\u001b[1;33m.\\u001b[0m\\u001b[0mdtype\\u001b[0m \\u001b[1;33m==\\u001b[0m \\u001b[0mtorch\\u001b[0m\\u001b[1;33m.\\u001b[0m\\u001b[0mfloat16\\u001b[0m\\u001b[1;33m:\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[1;33m\\u001b[0m\\u001b[0m\\n\",\n      \"\\u001b[1;31mKeyboardInterrupt\\u001b[0m: \"\n     ]\n    }\n   ],\n   \"source\": [\n    \"import time\\n\",\n    \"def time_model_evaluation(model, mel, options):\\n\",\n    \"    eval_start_time = time.time()\\n\",\n    \"    # result = whisper.decode(model, mel, options)\\n\",\n    \"    result = whisper.transcribe(model, test_path) # , options)\\n\",\n    \"    eval_end_time = time.time()\\n\",\n    \"    eval_duration_time = eval_end_time - eval_start_time\\n\",\n    \"    print(result[\\\"text\\\"])\\n\",\n    \"    print(\\\"Evaluate total time (seconds): {0:.1f}\\\".format(eval_duration_time))\\n\",\n    \"\\n\",\n    \"# Evaluate the original FP32 BERT model\\n\",\n    \"time_model_evaluation(model_fp32, mel, options)\\n\",\n    \"\\n\",\n    \"# Evaluate the INT8 BERT model after the dynamic quantization\\n\",\n    \"time_model_evaluation(quantized_model, mel, options)\"\n   ]\n  }\n ],\n \"metadata\": {\n  \"kernelspec\": {\n   \"display_name\": \"Python 3.9.9 64-bit\",\n   \"language\": \"python\",\n   \"name\": \"python3\"\n  },\n  \"language_info\": {\n   \"codemirror_mode\": {\n    \"name\": \"ipython\",\n    \"version\": 3\n   },\n   \"file_extension\": \".py\",\n   \"mimetype\": \"text/x-python\",\n   \"name\": \"python\",\n   \"nbconvert_exporter\": \"python\",\n   \"pygments_lexer\": \"ipython3\",\n   \"version\": \"3.9.9\"\n  },\n  \"orig_nbformat\": 4,\n  \"vscode\": {\n   \"interpreter\": {\n    \"hash\": \"28453d1081d3c550fce4dd227bac61cebcdf565b50505afc80cae3c0cf61cf22\"\n   }\n  }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 2\n}\n"
  },
  {
    "path": "script/custom_whisper.py",
    "content": "#!/usr/bin/python3\n\nimport sys\n\n#audio/Byron_Katie_Podcast/Byron_Katie_KICK_OFF_FINAL_MIX.mp3 --language English --model large\naudio_path = str(sys.argv[1])\nprint ('Audio:', audio_path)\nprint ('Language Tag', str(sys.argv[2]))\nlanguage = str(sys.argv[3])\nprint ('Language:', language)\nprint ('Model Tag:', str(sys.argv[4]))\nmodel_name = str(sys.argv[5])\nprint ('Model:', model_name)\n\nimport whisper\nimport torch\n\n\nmodel_fp32 = whisper.load_model(\n    name=model_name,\n    device=\"cpu\"\n#   ,in_memory=True\n)\n\nprint(torch.__version__)\n\nquantized_model = torch.quantization.quantize_dynamic(\n    model_fp32, {torch.nn.Linear}, dtype=torch.qint8\n)\n\n#print(quantized_model)\n#print(model_fp32)\n\nimport os\n\ndef print_size_of_model(model):\n    path = \"temp.p\"\n    torch.save(model.state_dict(), path)\n    size = os.path.getsize(path)/1e6\n    print('Size (MB):', size)\n    os.remove(path)\n    return size\n\nprint_size_of_model(model_fp32)\nprint_size_of_model(quantized_model)\n\n#audio = whisper.load_audio(audio_file)\n#audio = whisper.pad_or_trim(audio)\n\n#mel   = whisper.log_mel_spectrogram(audio).to(model_fp32.device)\n#options = whisper.DecodingOptions(language=language,fp16=False)\n\n# regular\n#_, probs = model_fp32.detect_language(mel)\n#print(f\"Detected language: {max(probs, key=probs.get)}\")\n\n# quantized\n#_, probs = quantized_model.detect_language(mel)\n#print(f\"Detected language: {max(probs, key=probs.get)}\")\n\nfrom pathlib import Path\nfrom whisper.utils import write_srt\nimport json\n\nimport time\ndef time_model_evaluation(model,audio_file):\n    eval_start_time = time.time()\n    # result = whisper.decode(model, mel, options)\n    result = whisper.transcribe(model, audio_file)\n    eval_end_time = time.time()\n    eval_duration_time = eval_end_time - eval_start_time\n\n    # save SRT\n    audio_basename = Path(audio_file).stem\n    with open(Path(\"./script\") / (audio_basename + \".srt\"), \"w\", encoding=\"utf-8\") as srt:\n        write_srt(result[\"segments\"], file=srt)\n    # save JSON\n    json_object = json.dumps(result, indent=4)\n    with open(Path(\"./script\") / (audio_basename + \".json\"), \"w\", encoding=\"utf-8\") as output:\n        output.write(json_object)\n\n    print(\"Evaluate total time (seconds): {0:.1f}\".format(eval_duration_time))\n\n\n# check if audio_path is a dir or a file\nif os.path.isdir(audio_path):\n    # is dir\n    files = [f for f in os.listdir(audio_path) if os.path.isfile(os.path.join(audio_path, f))]\n    for audio_file in files:\n        time_model_evaluation(quantized_model,os.path.join(audio_path, audio_file))\nelse:\n    # is file\n    time_model_evaluation(quantized_model,audio_path)\n\n\n\n\n# Evaluate the original FP32 BERT model\n# time_model_evaluation(model_fp32, mel, options)\n\n# Evaluate the INT8 BERT model after the dynamic quantization\n#time_model_evaluation(quantized_model)\n\n#torch.save(quantized_model.state_dict(), \"./script/quantized_model.p\")\n"
  }
]