[ { "path": ".dockerignore", "content": "__pycache__\n*.so\nbuild\n.coverage_*\n*.egg-info\n*~\n.vscode/\n.idea/\n.git/\n.github/\n.DS_Store\n\n/private/\n/README-docker.md\n" }, { "path": ".github/ISSUE_TEMPLATE/bug_report.yaml", "content": "name: 🐞 Bug\ndescription: 提交错误报告 | File a bug/issue\ntitle: \"[BUG] \"\nlabels: []\nbody:\n - type: checkboxes\n attributes:\n label: 是否已有关于该错误的issue或讨论？ | Is there an existing issue / discussion for this?\n description: |\n 请先搜索您遇到的错误是否在已有的issues或讨论中提到过。\n Please search to see if an issue / discussion already exists for the bug you encountered.\n [Issues](https://github.com/QwenLM/Qwen-7B/issues)\n [Discussions](https://github.com/QwenLM/Qwen-7B/discussions)\n options:\n - label: 我已经搜索过已有的issues和讨论 | I have searched the existing issues / discussions\n required: true\n - type: checkboxes\n attributes:\n label: 该问题是否在FAQ中有解答？ | Is there an existing answer for this in FAQ?\n description: |\n 请先搜索您遇到的错误是否已在FAQ中有相关解答。\n Please search to see if an answer already exists in FAQ for the bug you encountered.\n [FAQ-en](https://github.com/QwenLM/Qwen-7B/blob/main/FAQ.md)\n [FAQ-zh](https://github.com/QwenLM/Qwen-7B/blob/main/FAQ_zh.md)\n options:\n - label: 我已经搜索过FAQ | I have searched FAQ\n required: true\n - type: textarea\n attributes:\n label: 当前行为 | Current Behavior\n description: |\n 准确描述遇到的行为。\n A concise description of what you're experiencing.\n validations:\n required: false\n - type: textarea\n attributes:\n label: 期望行为 | Expected Behavior\n description: |\n 准确描述预期的行为。\n A concise description of what you expected to happen.\n validations:\n required: false\n - type: textarea\n attributes:\n label: 复现方法 | Steps To Reproduce\n description: |\n 复现当前行为的详细步骤。\n Steps to reproduce the behavior.\n placeholder: |\n 1. In this environment...\n 2. With this config...\n 3. Run '...'\n 4. See error...\n validations:\n required: false\n - type: textarea\n attributes:\n label: 运行环境 | Environment\n description: |\n examples:\n - **OS**: Ubuntu 20.04\n - **Python**: 3.8\n - **Transformers**: 4.31.0\n - **PyTorch**: 2.0.1\n - **CUDA**: 11.4\n value: |\n - OS:\n - Python:\n - Transformers:\n - PyTorch:\n - CUDA (`python -c 'import torch; print(torch.version.cuda)'`):\n render: Markdown\n validations:\n required: false\n - type: textarea\n attributes:\n label: 备注 | Anything else?\n description: |\n 您可以在这里补充其他关于该问题背景信息的描述、链接或引用等。\n \n 您可以通过点击高亮此区域然后拖动文件的方式上传图片或日志文件。\n \n Links? References? Anything that will give us more context about the issue you are encountering!\n \n Tip: You can attach images or log files by clicking this area to highlight it and then dragging files in.\n validations:\n required: false\n" }, { "path": ".github/ISSUE_TEMPLATE/config.yaml", "content": "blank_issues_enabled: true\n" }, { "path": ".github/ISSUE_TEMPLATE/feature_request.yaml", "content": "name: \"💡 Feature Request\"\ndescription: 创建新功能请求 | Create a new ticket for a new feature request\ntitle: \"💡 [REQUEST] - <title>\"\nlabels: [\n \"question\"\n]\nbody:\n - type: input\n id: start_date\n attributes:\n label: \"起始日期 | Start Date\"\n description: |\n 起始开发日期\n Start of development\n placeholder: \"month/day/year\"\n validations:\n required: false\n - type: textarea\n id: implementation_pr\n attributes:\n label: \"实现PR | Implementation PR\"\n description: |\n 实现该功能的Pull request\n Pull request used\n placeholder: \"#Pull Request ID\"\n validations:\n required: false\n - type: textarea\n id: reference_issues\n attributes:\n label: \"相关Issues | Reference Issues\"\n description: |\n 与该功能相关的issues\n Common issues\n placeholder: \"#Issues IDs\"\n validations:\n required: false\n - type: textarea\n id: summary\n attributes:\n label: \"摘要 | Summary\"\n description: |\n 简要描述新功能的特点\n Provide a brief explanation of the feature\n placeholder: |\n Describe in a few lines your feature request\n validations:\n required: true\n - type: textarea\n id: basic_example\n attributes:\n label: \"基本示例 | Basic Example\"\n description: Indicate here some basic examples of your feature.\n placeholder: A few specific words about your feature request.\n validations:\n required: true\n - type: textarea\n id: drawbacks\n attributes:\n label: \"缺陷 | Drawbacks\"\n description: |\n 该新功能有哪些缺陷/可能造成哪些影响？\n What are the drawbacks/impacts of your feature request ?\n placeholder: |\n Identify the drawbacks and impacts while being neutral on your feature request\n validations:\n required: true\n - type: textarea\n id: unresolved_question\n attributes:\n label: \"未解决问题 | Unresolved questions\"\n description: |\n 有哪些尚未解决的问题？\n What questions still remain unresolved ?\n placeholder: |\n Identify any unresolved issues.\n validations:\n required: false" }, { "path": ".github/workflows/stale.yml", "content": "name: Close stale issues\non:\n schedule:\n - cron: \"0 8 * * *\"\n\njobs:\n close-issues:\n runs-on: ubuntu-latest\n permissions:\n actions: write\n issues: write\n steps:\n - uses: actions/stale@v9\n with:\n days-before-issue-stale: 30\n days-before-issue-close: 7\n stale-issue-label: inactive\n stale-issue-message: This issue has been automatically marked as inactive due to lack of recent activity. \n Should you believe it remains unresolved and warrants attention, kindly leave a comment on this thread. \n \n 此问题由于长期未有新进展而被系统自动标记为不活跃。如果您认为它仍有待解决，请在此帖下方留言以补充信息。\n days-before-pr-stale: -1\n days-before-pr-close: -1\n operations-per-run: 128\n repo-token: ${{ secrets.GITHUB_TOKEN }}\n" }, { "path": ".gitignore", "content": "__pycache__\n*.so\nbuild\n.coverage_*\n*.egg-info\n*~\n.vscode/\n.idea/\n.DS_Store\n\n/private/\n" }, { "path": "FAQ.md", "content": "# FAQ\n\n## Installation & Environment\n\n#### Failure in installing flash attention\n\nFlash attention is an option for accelerating training and inference. Only NVIDIA GPUs of Turing, Ampere, Ada, and Hopper architecture, e.g., H100, A100, RTX 3090, T4, RTX 2080, can support flash attention. **You can use our models without installing it.**\n\n#### Which version of transformers should I use?\n\n4.32.0 is preferred.\n\n#### I downloaded the codes and checkpoints but I can't load the model locally. What should I do?\n\nPlease check if you have updated the code to the latest, and correctly downloaded all the sharded checkpoint files.\n\n#### `qwen.tiktoken` is not found. What is it?\n\nThis is the merge file of the tokenizer. You have to download it. Note that if you just git clone the repo without [git-lfs](https://git-lfs.com), you cannot download this file.\n\n#### transformers_stream_generator/tiktoken/accelerate not found\n\nRun the command `pip install -r requirements.txt`. You can find the file at [https://github.com/QwenLM/Qwen-7B/blob/main/requirements.txt](https://github.com/QwenLM/Qwen/blob/main/requirements.txt).\n<br><br>\n\n\n\n## Demo & Inference\n\n#### Is there any demo? CLI demo and Web UI demo?\n\nYes, see `web_demo.py` for web demo and `cli_demo.py` for CLI demo. See README for more information.\n\n\n#### Can I use CPU only?\n\nYes, run `python cli_demo.py --cpu-only` will load the model and inference on CPU only.\n\n#### Can Qwen support streaming?\n\nYes. See the function `chat_stream` in `modeling_qwen.py`.\n\n#### Gibberish in result when using chat_stream().\n\nThis is because tokens represent bytes and a single token may be a meaningless string. We have updated the default setting of our tokenizer to avoid such decoding results. Please update the code to the latest version.\n\n#### It seems that the generation is not related to the instruction...\n\nPlease check if you are loading Qwen-Chat instead of Qwen. Qwen is the base model without alignment, which behaves differently from the SFT/Chat model.\n\n#### Is quantization supported?\n\nYes, the quantization is supported by AutoGPTQ. \n\n\n#### Slow when processing long sequences\n\nUpdating the code to the latest version can help.\n\n#### Unsatisfactory performance in processing long sequences\n\nPlease ensure that NTK is applied. `use_dynamc_ntk` and `use_logn_attn` in `config.json` should be set to `true` (`true` by default).\n<br><br>\n\n\n\n## Finetuning\n\n#### Can Qwen support SFT or even RLHF?\n\nYes, we now support SFT, including full-parameter finetuning, LoRA, and Q-LoRA. Also you can check other projects like [FastChat](**[https://github.com/lm-sys/FastChat](https://github.com/lm-sys/FastChat)), [Firefly]([https://github.com/yangjianxin1/Firefly](https://github.com/yangjianxin1/Firefly)), [**LLaMA Efficient Tuning**]([https://github.com/hiyouga/LLaMA-Efficient-Tuning](https://github.com/hiyouga/LLaMA-Efficient-Tuning)), etc.\n\nHowever, temporarily we do not support RLHF. We will provide the code in the near future.\n<br><br>\n\n\n\n## Tokenizer\n\n#### bos_id/eos_id/pad_id not found\n\nIn our training, we only use `<|endoftext|>` as the separator and padding token. You can set bos_id, eos_id, and pad_id to tokenizer.eod_id. Learn more about our tokenizer from our documents about the tokenizer.\n\n\n\n## Docker\n\n#### Download official docker image is very slow\n\nWhen downloading our official docker image, you may have a slow download speed due to some network issues. You can refer to [Alibaba Cloud Container Image Service](https://help.aliyun.com/zh/acr/user-guide/accelerate-the-pulls-of-docker-official-images) to accelerate the download of official images.\n" }, { "path": "FAQ_ja.md", "content": "# FAQ\n\n## インストールと環境\n\n#### Flash attention 導入の失敗例\n\nFlash attention は、トレーニングと推論を加速するオプションです。H100、A100、RTX 3090、T4、RTX 2080 などの Turing、Ampere、Ada、および Hopper アーキテクチャの NVIDIA GPU だけが、flash attention をサポートできます。それをインストールせずに私たちのモデルを使用することができます。\n\n#### transformers のバージョンは？\n\n4.32.0 が望ましいです。\n\n#### コードとチェックポイントをダウンロードしましたが、モデルをローカルにロードできません。どうすればよいでしょうか？\n\nコードを最新のものに更新し、すべてのシャードされたチェックポイントファイルを正しくダウンロードしたかどうか確認してください。\n\n#### `qwen.tiktoken` が見つかりません。これは何ですか？\n\nこれはトークナイザーのマージファイルです。ダウンロードする必要があります。[git-lfs](https://git-lfs.com) を使わずにリポジトリを git clone しただけでは、このファイルをダウンロードできないことに注意してください。\n\n#### transformers_stream_generator/tiktoken/accelerate が見つかりません。\n\nコマンド `pip install -r requirements.txt` を実行してください。このファイルは [https://github.com/QwenLM/Qwen/blob/main/requirements.txt](https://github.com/QwenLM/Qwen/blob/main/requirements.txt) にあります。\n<br><br>\n\n\n\n## デモと推論\n\n#### デモはありますか？CLI と Web UI のデモはありますか？\n\nはい、Web デモは `web_demo.py` を、CLI デモは `cli_demo.py` を参照してください。詳しくは README を参照してください。\n\n\n\n#### CPU のみを使うことはできますか？\n\nはい、`python cli_demo.py --cpu-only` を実行すると、CPU のみでモデルと推論をロードします。\n\n#### Qwen はストリーミングに対応していますか？\n\n`modeling_qwen.py` の `chat_stream` 関数を参照してください。\n\n#### chat_stream() を使用すると、結果に文字化けが発生します。\n\nこれは、トークンがバイトを表し、単一のトークンが無意味な文字列である可能性があるためです。このようなデコード結果を避けるため、トークナイザのデフォルト設定を更新しました。コードを最新版に更新してください。\n\n#### インストラクションとは関係ないようですが...\n\nQwen ではなく Qwen-Chat を読み込んでいないか確認してください。Qwen はアライメントなしのベースモデルで、SFT/Chat モデルとは挙動が異なります。\n\n#### 量子化はサポートされていますか？\n\nはい、量子化は AutoGPTQ でサポートされています。\n\n\n#### 長いシーケンスの処理に時間がかかる\n\nコードを最新版に更新することで解決します。\n\n#### 長いシーケンスの処理で不満足なパフォーマンス\n\nNTK が適用されていることを確認してください。`config.json` の `use_dynamc_ntk` と `use_logn_attn` を `true` に設定する必要があります（デフォルトでは `true`）。\n<br><br>\n\n\n\n## ファインチューニング\n\n#### Qwen は SFT、あるいは RLHF に対応できますか？\n\nSFTのコードは提供します。[FastChat](**[https://github.com/lm-sys/FastChat](https://github.com/lm-sys/FastChat))、[Firefly]([https://github.com/yangjianxin1/Firefly](https://github.com/yangjianxin1/Firefly))、[**LLaMA Efficient Tuning**]([https://github.com/hiyouga/LLaMA-Efficient-Tuning](https://github.com/hiyouga/LLaMA-Efficient-Tuning))など、いくつかのプロジェクトではファインチューニングをサポートしています。近日中に関連コードを更新する予定です。\n<br><br>\n\n\n\n## トークナイザー\n\n#### bos_id/eos_id/pad_id が見つかりません。\n\n私たちのトレーニングでは、セパレータとパディングトークンとして `<|endoftext|>` のみを使用しています。bos_id、eos_id、pad_id は tokenizer.eod_id に設定できます。私たちのトークナイザーについて詳しくは、トークナイザーについてのドキュメントをご覧ください。\n\n" }, { "path": "FAQ_zh.md", "content": "# FAQ\n\n## 安装&环境\n\n#### flash attention 安装失败\n\nflash attention是一个用于加速模型训练推理的可选项，且仅适用于Turing、Ampere、Ada、Hopper架构的Nvidia GPU显卡（如H100、A100、RTX 3090、T4、RTX 2080），您可以在不安装flash attention的情况下正常使用模型进行推理。\n\n#### 我应该用哪个transformers版本？\n\n建议使用4.32.0。\n\n#### 我把模型和代码下到本地，按照教程无法使用，该怎么办？\n\n答：别着急，先检查你的代码是不是更新到最新版本，然后确认你是否完整地将模型checkpoint下到本地。\n\n#### `qwen.tiktoken`这个文件找不到，怎么办？\n\n这个是我们的tokenizer的merge文件，你必须下载它才能使用我们的tokenizer。注意，如果你使用git clone却没有使用git-lfs，这个文件不会被下载。如果你不了解git-lfs，可点击[官网](https://git-lfs.com/)了解。\n\n#### transformers_stream_generator/tiktoken/accelerate，这几个库提示找不到，怎么办？\n\n运行如下命令：`pip install -r requirements.txt`。相关依赖库在[https://github.com/QwenLM/Qwen-7B/blob/main/requirements.txt](https://github.com/QwenLM/Qwen/blob/main/requirements.txt) 可以找到。\n<br><br>\n\n\n## Demo & 推理\n\n#### 是否提供Demo？CLI Demo及Web UI Demo？\n\n`web_demo.py`和`cli_demo.py`分别提供了Web UI以及CLI的Demo。请查看README相关内容了解更多。\n\n#### 我没有GPU，只用CPU运行CLI demo可以吗？\n\n可以的，运行`python cli_demo.py --cpu-only`命令即可将模型读取到CPU并使用CPU进行推理。\n\n#### Qwen支持流式推理吗？\n\nQwen当前支持流式推理。见位于`modeling_qwen.py`的`chat_stream`函数。\n\n#### 使用`chat_stream()`生成混乱的内容及乱码，为什么？\n\n这是由于模型生成过程中输出的部分token需要与后续token一起解码才能输出正常文本，单个token解码结果是无意义字符串，我们已经更新了tokenizer解码时的默认设置，避免这些字符串在生成结果中出现，如果仍有类似问题请更新模型至最新版本。\n\n#### 模型的输出看起来与输入无关/没有遵循指令/看起来呆呆的\n\n请检查是否加载的是Qwen-Chat模型进行推理，Qwen模型是未经align的预训练基模型，不期望具备响应用户指令的能力。我们在模型最新版本已经对`chat`及`chat_stream`接口内进行了检查，避免您误将预训练模型作为SFT/Chat模型使用。\n\n#### 是否有量化版本模型\n\n目前Qwen支持基于AutoGPTQ的4-bit的量化推理。\n\n#### 生成序列较长后速度显著变慢\n\n请更新到最新代码。\n\n#### 处理长序列时效果有问题\n\n请确认是否开启ntk。若要启用这些技巧，请将`config.json`里的`use_dynamc_ntk`和`use_logn_attn`设置为`true`。最新代码默认为`true`。\n<br><br>\n\n\n## 微调\n\n#### 当前是否支持SFT和RLHF？\n\n我们目前提供了SFT的代码，支持全参数微调、LoRA和Q-LoRA。此外，当前有多个外部项目也已实现支持，如[FastChat](**[https://github.com/lm-sys/FastChat](https://github.com/lm-sys/FastChat))、[Firefly]([https://github.com/yangjianxin1/Firefly](https://github.com/yangjianxin1/Firefly))、[**LLaMA Efficient Tuning**]([https://github.com/hiyouga/LLaMA-Efficient-Tuning](https://github.com/hiyouga/LLaMA-Efficient-Tuning))等。我们会尽快更新这部分代码和说明。\n\n我们还没提供对RLHF训练的支持，敬请期待。\n<br><br>\n\n\n## Tokenizer\n\n#### bos_id/eos_id/pad_id，这些token id不存在，为什么？\n\n在训练过程中，我们仅使用<|endoftext|>这一token作为sample/document之间的分隔符及padding位置占位符，你可以将bos_id, eos_id, pad_id均指向tokenizer.eod_id。请阅读我们关于tokenizer的文档，了解如何设置这些id。\n\n\n## Docker\n\n#### 下载官方Docker镜像速度很慢\n\n在下载官方镜像时，您可能由于某些网络原因导致下载速度变慢。可以参考[阿里云容器镜像服务](https://help.aliyun.com/zh/acr/user-guide/accelerate-the-pulls-of-docker-official-images)加速官方镜像的下载。" }, { "path": "LICENSE", "content": " Apache License\n Version 2.0, January 2004\n http://www.apache.org/licenses/\n\n TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION\n\n 1. Definitions.\n\n \"License\" shall mean the terms and conditions for use, reproduction,\n and distribution as defined by Sections 1 through 9 of this document.\n\n \"Licensor\" shall mean the copyright owner or entity authorized by\n the copyright owner that is granting the License.\n\n \"Legal Entity\" shall mean the union of the acting entity and all\n other entities that control, are controlled by, or are under common\n control with that entity. For the purposes of this definition,\n \"control\" means (i) the power, direct or indirect, to cause the\n direction or management of such entity, whether by contract or\n otherwise, or (ii) ownership of fifty percent (50%) or more of the\n outstanding shares, or (iii) beneficial ownership of such entity.\n\n \"You\" (or \"Your\") shall mean an individual or Legal Entity\n exercising permissions granted by this License.\n\n \"Source\" form shall mean the preferred form for making modifications,\n including but not limited to software source code, documentation\n source, and configuration files.\n\n \"Object\" form shall mean any form resulting from mechanical\n transformation or translation of a Source form, including but\n not limited to compiled object code, generated documentation,\n and conversions to other media types.\n\n \"Work\" shall mean the work of authorship, whether in Source or\n Object form, made available under the License, as indicated by a\n copyright notice that is included in or attached to the work\n (an example is provided in the Appendix below).\n\n \"Derivative Works\" shall mean any work, whether in Source or Object\n form, that is based on (or derived from) the Work and for which the\n editorial revisions, annotations, elaborations, or other modifications\n represent, as a whole, an original work of authorship. For the purposes\n of this License, Derivative Works shall not include works that remain\n separable from, or merely link (or bind by name) to the interfaces of,\n the Work and Derivative Works thereof.\n\n \"Contribution\" shall mean any work of authorship, including\n the original version of the Work and any modifications or additions\n to that Work or Derivative Works thereof, that is intentionally\n submitted to Licensor for inclusion in the Work by the copyright owner\n or by an individual or Legal Entity authorized to submit on behalf of\n the copyright owner. For the purposes of this definition, \"submitted\"\n means any form of electronic, verbal, or written communication sent\n to the Licensor or its representatives, including but not limited to\n communication on electronic mailing lists, source code control systems,\n and issue tracking systems that are managed by, or on behalf of, the\n Licensor for the purpose of discussing and improving the Work, but\n excluding communication that is conspicuously marked or otherwise\n designated in writing by the copyright owner as \"Not a Contribution.\"\n\n \"Contributor\" shall mean Licensor and any individual or Legal Entity\n on behalf of whom a Contribution has been received by Licensor and\n subsequently incorporated within the Work.\n\n 2. Grant of Copyright License. Subject to the terms and conditions of\n this License, each Contributor hereby grants to You a perpetual,\n worldwide, non-exclusive, no-charge, royalty-free, irrevocable\n copyright license to reproduce, prepare Derivative Works of,\n publicly display, publicly perform, sublicense, and distribute the\n Work and such Derivative Works in Source or Object form.\n\n 3. Grant of Patent License. Subject to the terms and conditions of\n this License, each Contributor hereby grants to You a perpetual,\n worldwide, non-exclusive, no-charge, royalty-free, irrevocable\n (except as stated in this section) patent license to make, have made,\n use, offer to sell, sell, import, and otherwise transfer the Work,\n where such license applies only to those patent claims licensable\n by such Contributor that are necessarily infringed by their\n Contribution(s) alone or by combination of their Contribution(s)\n with the Work to which such Contribution(s) was submitted. If You\n institute patent litigation against any entity (including a\n cross-claim or counterclaim in a lawsuit) alleging that the Work\n or a Contribution incorporated within the Work constitutes direct\n or contributory patent infringement, then any patent licenses\n granted to You under this License for that Work shall terminate\n as of the date such litigation is filed.\n\n 4. Redistribution. You may reproduce and distribute copies of the\n Work or Derivative Works thereof in any medium, with or without\n modifications, and in Source or Object form, provided that You\n meet the following conditions:\n\n (a) You must give any other recipients of the Work or\n Derivative Works a copy of this License; and\n\n (b) You must cause any modified files to carry prominent notices\n stating that You changed the files; and\n\n (c) You must retain, in the Source form of any Derivative Works\n that You distribute, all copyright, patent, trademark, and\n attribution notices from the Source form of the Work,\n excluding those notices that do not pertain to any part of\n the Derivative Works; and\n\n (d) If the Work includes a \"NOTICE\" text file as part of its\n distribution, then any Derivative Works that You distribute must\n include a readable copy of the attribution notices contained\n within such NOTICE file, excluding those notices that do not\n pertain to any part of the Derivative Works, in at least one\n of the following places: within a NOTICE text file distributed\n as part of the Derivative Works; within the Source form or\n documentation, if provided along with the Derivative Works; or,\n within a display generated by the Derivative Works, if and\n wherever such third-party notices normally appear. The contents\n of the NOTICE file are for informational purposes only and\n do not modify the License. You may add Your own attribution\n notices within Derivative Works that You distribute, alongside\n or as an addendum to the NOTICE text from the Work, provided\n that such additional attribution notices cannot be construed\n as modifying the License.\n\n You may add Your own copyright statement to Your modifications and\n may provide additional or different license terms and conditions\n for use, reproduction, or distribution of Your modifications, or\n for any such Derivative Works as a whole, provided Your use,\n reproduction, and distribution of the Work otherwise complies with\n the conditions stated in this License.\n\n 5. Submission of Contributions. Unless You explicitly state otherwise,\n any Contribution intentionally submitted for inclusion in the Work\n by You to the Licensor shall be under the terms and conditions of\n this License, without any additional terms or conditions.\n Notwithstanding the above, nothing herein shall supersede or modify\n the terms of any separate license agreement you may have executed\n with Licensor regarding such Contributions.\n\n 6. Trademarks. This License does not grant permission to use the trade\n names, trademarks, service marks, or product names of the Licensor,\n except as required for reasonable and customary use in describing the\n origin of the Work and reproducing the content of the NOTICE file.\n\n 7. Disclaimer of Warranty. Unless required by applicable law or\n agreed to in writing, Licensor provides the Work (and each\n Contributor provides its Contributions) on an \"AS IS\" BASIS,\n WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or\n implied, including, without limitation, any warranties or conditions\n of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A\n PARTICULAR PURPOSE. You are solely responsible for determining the\n appropriateness of using or redistributing the Work and assume any\n risks associated with Your exercise of permissions under this License.\n\n 8. Limitation of Liability. In no event and under no legal theory,\n whether in tort (including negligence), contract, or otherwise,\n unless required by applicable law (such as deliberate and grossly\n negligent acts) or agreed to in writing, shall any Contributor be\n liable to You for damages, including any direct, indirect, special,\n incidental, or consequential damages of any character arising as a\n result of this License or out of the use or inability to use the\n Work (including but not limited to damages for loss of goodwill,\n work stoppage, computer failure or malfunction, or any and all\n other commercial damages or losses), even if such Contributor\n has been advised of the possibility of such damages.\n\n 9. Accepting Warranty or Additional Liability. While redistributing\n the Work or Derivative Works thereof, You may choose to offer,\n and charge a fee for, acceptance of support, warranty, indemnity,\n or other liability obligations and/or rights consistent with this\n License. However, in accepting such obligations, You may act only\n on Your own behalf and on Your sole responsibility, not on behalf\n of any other Contributor, and only if You agree to indemnify,\n defend, and hold each Contributor harmless for any liability\n incurred by, or claims asserted against, such Contributor by reason\n of your accepting any such warranty or additional liability.\n\n END OF TERMS AND CONDITIONS\n\n APPENDIX: How to apply the Apache License to your work.\n\n To apply the Apache License to your work, attach the following\n boilerplate notice, with the fields enclosed by brackets \"[]\"\n replaced with your own identifying information. (Don't include\n the brackets!) The text should be enclosed in the appropriate\n comment syntax for the file format. We also recommend that a\n file or class name and description of purpose be included on the\n same \"printed page\" as the copyright notice for easier\n identification within third-party archives.\n\n Copyright 2023 Alibaba Cloud\n\n Licensed under the Apache License, Version 2.0 (the \"License\");\n you may not use this file except in compliance with the License.\n You may obtain a copy of the License at\n\n http://www.apache.org/licenses/LICENSE-2.0\n\n Unless required by applicable law or agreed to in writing, software\n distributed under the License is distributed on an \"AS IS\" BASIS,\n WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n See the License for the specific language governing permissions and\n limitations under the License." }, { "path": "NOTICE", "content": "------------- LICENSE FOR NVIDIA Megatron-LM code --------------\n\nCopyright (c) 2022, NVIDIA CORPORATION. All rights reserved.\n\nRedistribution and use in source and binary forms, with or without\nmodification, are permitted provided that the following conditions\nare met:\n * Redistributions of source code must retain the above copyright\n notice, this list of conditions and the following disclaimer.\n * Redistributions in binary form must reproduce the above copyright\n notice, this list of conditions and the following disclaimer in the\n documentation and/or other materials provided with the distribution.\n * Neither the name of NVIDIA CORPORATION nor the names of its\n contributors may be used to endorse or promote products derived\n from this software without specific prior written permission.\n\nTHIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ``AS IS'' AND ANY\nEXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\nIMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR\nPURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR\nCONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,\nEXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,\nPROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR\nPROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY\nOF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT\n(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE\nOF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\n\n\n------------- LICENSE FOR OpenAI tiktoken code --------------\n\nMIT License\n\nCopyright (c) 2022 OpenAI, Shantanu Jain\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\nSOFTWARE.\n\n------------- LICENSE FOR stanford_alpaca code --------------\n\n Apache License\n Version 2.0, January 2004\n http://www.apache.org/licenses/\n\n TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION\n\n 1. Definitions.\n\n \"License\" shall mean the terms and conditions for use, reproduction,\n and distribution as defined by Sections 1 through 9 of this document.\n\n \"Licensor\" shall mean the copyright owner or entity authorized by\n the copyright owner that is granting the License.\n\n \"Legal Entity\" shall mean the union of the acting entity and all\n other entities that control, are controlled by, or are under common\n control with that entity. For the purposes of this definition,\n \"control\" means (i) the power, direct or indirect, to cause the\n direction or management of such entity, whether by contract or\n otherwise, or (ii) ownership of fifty percent (50%) or more of the\n outstanding shares, or (iii) beneficial ownership of such entity.\n\n \"You\" (or \"Your\") shall mean an individual or Legal Entity\n exercising permissions granted by this License.\n\n \"Source\" form shall mean the preferred form for making modifications,\n including but not limited to software source code, documentation\n source, and configuration files.\n\n \"Object\" form shall mean any form resulting from mechanical\n transformation or translation of a Source form, including but\n not limited to compiled object code, generated documentation,\n and conversions to other media types.\n\n \"Work\" shall mean the work of authorship, whether in Source or\n Object form, made available under the License, as indicated by a\n copyright notice that is included in or attached to the work\n (an example is provided in the Appendix below).\n\n \"Derivative Works\" shall mean any work, whether in Source or Object\n form, that is based on (or derived from) the Work and for which the\n editorial revisions, annotations, elaborations, or other modifications\n represent, as a whole, an original work of authorship. For the purposes\n of this License, Derivative Works shall not include works that remain\n separable from, or merely link (or bind by name) to the interfaces of,\n the Work and Derivative Works thereof.\n\n \"Contribution\" shall mean any work of authorship, including\n the original version of the Work and any modifications or additions\n to that Work or Derivative Works thereof, that is intentionally\n submitted to Licensor for inclusion in the Work by the copyright owner\n or by an individual or Legal Entity authorized to submit on behalf of\n the copyright owner. For the purposes of this definition, \"submitted\"\n means any form of electronic, verbal, or written communication sent\n to the Licensor or its representatives, including but not limited to\n communication on electronic mailing lists, source code control systems,\n and issue tracking systems that are managed by, or on behalf of, the\n Licensor for the purpose of discussing and improving the Work, but\n excluding communication that is conspicuously marked or otherwise\n designated in writing by the copyright owner as \"Not a Contribution.\"\n\n \"Contributor\" shall mean Licensor and any individual or Legal Entity\n on behalf of whom a Contribution has been received by Licensor and\n subsequently incorporated within the Work.\n\n 2. Grant of Copyright License. Subject to the terms and conditions of\n this License, each Contributor hereby grants to You a perpetual,\n worldwide, non-exclusive, no-charge, royalty-free, irrevocable\n copyright license to reproduce, prepare Derivative Works of,\n publicly display, publicly perform, sublicense, and distribute the\n Work and such Derivative Works in Source or Object form.\n\n 3. Grant of Patent License. Subject to the terms and conditions of\n this License, each Contributor hereby grants to You a perpetual,\n worldwide, non-exclusive, no-charge, royalty-free, irrevocable\n (except as stated in this section) patent license to make, have made,\n use, offer to sell, sell, import, and otherwise transfer the Work,\n where such license applies only to those patent claims licensable\n by such Contributor that are necessarily infringed by their\n Contribution(s) alone or by combination of their Contribution(s)\n with the Work to which such Contribution(s) was submitted. If You\n institute patent litigation against any entity (including a\n cross-claim or counterclaim in a lawsuit) alleging that the Work\n or a Contribution incorporated within the Work constitutes direct\n or contributory patent infringement, then any patent licenses\n granted to You under this License for that Work shall terminate\n as of the date such litigation is filed.\n\n 4. Redistribution. You may reproduce and distribute copies of the\n Work or Derivative Works thereof in any medium, with or without\n modifications, and in Source or Object form, provided that You\n meet the following conditions:\n\n (a) You must give any other recipients of the Work or\n Derivative Works a copy of this License; and\n\n (b) You must cause any modified files to carry prominent notices\n stating that You changed the files; and\n\n (c) You must retain, in the Source form of any Derivative Works\n that You distribute, all copyright, patent, trademark, and\n attribution notices from the Source form of the Work,\n excluding those notices that do not pertain to any part of\n the Derivative Works; and\n\n (d) If the Work includes a \"NOTICE\" text file as part of its\n distribution, then any Derivative Works that You distribute must\n include a readable copy of the attribution notices contained\n within such NOTICE file, excluding those notices that do not\n pertain to any part of the Derivative Works, in at least one\n of the following places: within a NOTICE text file distributed\n as part of the Derivative Works; within the Source form or\n documentation, if provided along with the Derivative Works; or,\n within a display generated by the Derivative Works, if and\n wherever such third-party notices normally appear. The contents\n of the NOTICE file are for informational purposes only and\n do not modify the License. You may add Your own attribution\n notices within Derivative Works that You distribute, alongside\n or as an addendum to the NOTICE text from the Work, provided\n that such additional attribution notices cannot be construed\n as modifying the License.\n\n You may add Your own copyright statement to Your modifications and\n may provide additional or different license terms and conditions\n for use, reproduction, or distribution of Your modifications, or\n for any such Derivative Works as a whole, provided Your use,\n reproduction, and distribution of the Work otherwise complies with\n the conditions stated in this License.\n\n 5. Submission of Contributions. Unless You explicitly state otherwise,\n any Contribution intentionally submitted for inclusion in the Work\n by You to the Licensor shall be under the terms and conditions of\n this License, without any additional terms or conditions.\n Notwithstanding the above, nothing herein shall supersede or modify\n the terms of any separate license agreement you may have executed\n with Licensor regarding such Contributions.\n\n 6. Trademarks. This License does not grant permission to use the trade\n names, trademarks, service marks, or product names of the Licensor,\n except as required for reasonable and customary use in describing the\n origin of the Work and reproducing the content of the NOTICE file.\n\n 7. Disclaimer of Warranty. Unless required by applicable law or\n agreed to in writing, Licensor provides the Work (and each\n Contributor provides its Contributions) on an \"AS IS\" BASIS,\n WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or\n implied, including, without limitation, any warranties or conditions\n of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A\n PARTICULAR PURPOSE. You are solely responsible for determining the\n appropriateness of using or redistributing the Work and assume any\n risks associated with Your exercise of permissions under this License.\n\n 8. Limitation of Liability. In no event and under no legal theory,\n whether in tort (including negligence), contract, or otherwise,\n unless required by applicable law (such as deliberate and grossly\n negligent acts) or agreed to in writing, shall any Contributor be\n liable to You for damages, including any direct, indirect, special,\n incidental, or consequential damages of any character arising as a\n result of this License or out of the use or inability to use the\n Work (including but not limited to damages for loss of goodwill,\n work stoppage, computer failure or malfunction, or any and all\n other commercial damages or losses), even if such Contributor\n has been advised of the possibility of such damages.\n\n 9. Accepting Warranty or Additional Liability. While redistributing\n the Work or Derivative Works thereof, You may choose to offer,\n and charge a fee for, acceptance of support, warranty, indemnity,\n or other liability obligations and/or rights consistent with this\n License. However, in accepting such obligations, You may act only\n on Your own behalf and on Your sole responsibility, not on behalf\n of any other Contributor, and only if You agree to indemnify,\n defend, and hold each Contributor harmless for any liability\n incurred by, or claims asserted against, such Contributor by reason\n of your accepting any such warranty or additional liability.\n\n END OF TERMS AND CONDITIONS\n\n APPENDIX: How to apply the Apache License to your work.\n\n To apply the Apache License to your work, attach the following\n boilerplate notice, with the fields enclosed by brackets \"[]\"\n replaced with your own identifying information. (Don't include\n the brackets!) The text should be enclosed in the appropriate\n comment syntax for the file format. We also recommend that a\n file or class name and description of purpose be included on the\n same \"printed page\" as the copyright notice for easier\n identification within third-party archives.\n\n Copyright 2023 Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li\n\n Licensed under the Apache License, Version 2.0 (the \"License\");\n you may not use this file except in compliance with the License.\n You may obtain a copy of the License at\n\n http://www.apache.org/licenses/LICENSE-2.0\n\n Unless required by applicable law or agreed to in writing, software\n distributed under the License is distributed on an \"AS IS\" BASIS,\n WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n See the License for the specific language governing permissions and\n limitations under the License.\n \n------------- LICENSE FOR PanQiWei AutoGPTQ code --------------\n\nMIT License\n\nCopyright (c) 2023 潘其威(William)\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\nSOFTWARE.\n" }, { "path": "README.md", "content": "<p align=\"left\">\n <a href=\"README_CN.md\">中文</a>  ｜  English  ｜  <a href=\"README_JA.md\">日本語</a> ｜  <a href=\"README_FR.md\">Français</a> ｜  <a href=\"README_ES.md\">Español</a>\n</p>\n<br><br>\n\n<p align=\"center\">\n <img src=\"https://qianwen-res.oss-cn-beijing.aliyuncs.com/logo_qwen.jpg\" width=\"400\"/>\n<p>\n<br>\n\n<p align=\"center\">\n 🤗 <a href=\"https://huggingface.co/Qwen\">Hugging Face</a>   |   🤖 <a href=\"https://modelscope.cn/organization/qwen\">ModelScope</a>   |    📑 <a href=\"https://arxiv.org/abs/2309.16609\">Paper</a>    ｜   🖥️ <a href=\"https://modelscope.cn/studios/qwen/Qwen-72B-Chat-Demo/summary\">Demo</a>\n<br>\n<a href=\"assets/wechat.png\">WeChat (微信)</a>   |   <a href=\"https://discord.gg/CV4E9rpNSD\">Discord</a>   ｜   <a href=\"https://dashscope.aliyun.com\">API</a> \n</p>\n<br><br>\n\n> [!Important]\n> Qwen2 is here! You are welcome to follow [QwenLM/Qwen2](https://github.com/QwenLM/Qwen2) and share your experience there.\n>\n> This repo ([QwenLM/Qwen](https://github.com/QwenLM/Qwen)) is no longer actively maintained, due to substantial codebase differences.\n\n<br>\n\n| | Qwen-Chat | Qwen-Chat (Int4) | Qwen-Chat (Int8) | Qwen |\n|-----|:------------------------------------------------------------------------------------------------------------------------------------:|:----------------------------------------------------------------------------------------------------------------------------------------------:|:---------------------------------------------------------------:|:--------------------------------------------------------------------------------------------------------------------------:|\n| 1.8B | <a href=\"https://modelscope.cn/models/qwen/Qwen-1_8B-Chat/summary\">🤖</a> <a href=\"https://huggingface.co/Qwen/Qwen-1_8B-Chat\">🤗</a> | <a href=\"https://modelscope.cn/models/qwen/Qwen-1_8B-Chat-Int4/summary\">🤖</a> <a href=\"https://huggingface.co/Qwen/Qwen-1_8B-Chat-Int4\">🤗</a> | <a href=\"https://modelscope.cn/models/qwen/Qwen-1_8B-Chat-Int8/summary\">🤖</a> <a href=\"https://huggingface.co/Qwen/Qwen-1_8B-Chat-Int8\">🤗</a> | <a href=\"https://modelscope.cn/models/qwen/Qwen-1_8B/summary\">🤖</a> <a href=\"https://huggingface.co/Qwen/Qwen-1_8B\">🤗</a> |\n| 7B | <a href=\"https://modelscope.cn/models/qwen/Qwen-7B-Chat/summary\">🤖</a> <a href=\"https://huggingface.co/Qwen/Qwen-7B-Chat\">🤗</a> | <a href=\"https://modelscope.cn/models/qwen/Qwen-7B-Chat-Int4/summary\">🤖</a> <a href=\"https://huggingface.co/Qwen/Qwen-7B-Chat-Int4\">🤗</a> | <a href=\"https://modelscope.cn/models/qwen/Qwen-7B-Chat-Int8/summary\">🤖</a> <a href=\"https://huggingface.co/Qwen/Qwen-7B-Chat-Int8\">🤗</a> | <a href=\"https://modelscope.cn/models/qwen/Qwen-7B/summary\">🤖</a> <a href=\"https://huggingface.co/Qwen/Qwen-7B\">🤗</a> |\n| 14B | <a href=\"https://modelscope.cn/models/qwen/Qwen-14B-Chat/summary\">🤖</a> <a href=\"https://huggingface.co/Qwen/Qwen-14B-Chat\">🤗</a> | <a href=\"https://modelscope.cn/models/qwen/Qwen-14B-Chat-Int4/summary\">🤖</a> <a href=\"https://huggingface.co/Qwen/Qwen-14B-Chat-Int4\">🤗</a> | <a href=\"https://modelscope.cn/models/qwen/Qwen-14B-Chat-Int8/summary\">🤖</a> <a href=\"https://huggingface.co/Qwen/Qwen-14B-Chat-Int8\">🤗</a> | <a href=\"https://modelscope.cn/models/qwen/Qwen-14B/summary\">🤖</a> <a href=\"https://huggingface.co/Qwen/Qwen-14B\">🤗</a> |\n| 72B | <a href=\"https://modelscope.cn/models/qwen/Qwen-72B-Chat/summary\">🤖</a> <a href=\"https://huggingface.co/Qwen/Qwen-72B-Chat\">🤗</a> | <a href=\"https://modelscope.cn/models/qwen/Qwen-72B-Chat-Int4/summary\">🤖</a> <a href=\"https://huggingface.co/Qwen/Qwen-72B-Chat-Int4\">🤗</a> | <a href=\"https://modelscope.cn/models/qwen/Qwen-72B-Chat-Int8/summary\">🤖</a> <a href=\"https://huggingface.co/Qwen/Qwen-72B-Chat-Int8\">🤗</a> | <a href=\"https://modelscope.cn/models/qwen/Qwen-72B/summary\">🤖</a> <a href=\"https://huggingface.co/Qwen/Qwen-72B\">🤗</a> |\n\n\n\nWe opensource our **Qwen** series, now including **Qwen**, the base language models, namely **Qwen-1.8B**, **Qwen-7B**, **Qwen-14B**, and **Qwen-72B**, as well as **Qwen-Chat**, the chat models, namely **Qwen-1.8B-Chat**, **Qwen-7B-Chat**, **Qwen-14B-Chat**, and **Qwen-72B-Chat**. Links are on the above table. Click them and check the model cards. Also, we release the **[technical report](https://arxiv.org/abs/2309.16609)**. Please click the paper link and check it out!\n\nIn brief, we have strong base language models, which have been stably pretrained for up to 3 trillion tokens of multilingual data with a wide coverage of domains, languages (with a focus on Chinese and English), etc. They are able to achieve competitive performance on benchmark datasets. Additionally, we have chat models that are aligned with human preference based on SFT and RLHF (not released yet), which are able to chat, create content, extract information, summarize, translate, code, solve math problems, and so on, and are able to use tools, play as agents, or even play as code interpreters, etc.\n\n| Model | Release Date | Max Length | System Prompt Enhancement | # of Pretrained Tokens | Minimum GPU Memory Usage of Finetuning (Q-Lora) | Minimum GPU Usage of Generating 2048 Tokens (Int4) | Tool Usage |\n|:----------|:------------:|:----------:|:-------------------------:|:----------------------:|:-----------------------------------------------:|:--------------------------------------------------:|:----------:|\n| Qwen-1.8B | 23.11.30 | 32K | ✅ | 2.2T | 5.8GB | 2.9GB | ✅ | \n| Qwen-7B | 23.08.03 | 32K | ❎ | 2.4T | 11.5GB | 8.2GB | ✅ | \n| Qwen-14B | 23.09.25 | 8K | ❎ | 3.0T | 18.7GB | 13.0GB | ✅ |\n| Qwen-72B | 23.11.30 | 32K | ✅ | 3.0T | 61.4GB | 48.9GB | ✅ | \n\nIn this repo, you can figure out:\n\n* Quickstart with Qwen, and enjoy the simple inference.\n* Details about the quantization models, including GPTQ and KV cache quantization.\n* Statistics of inference performance, including speed and memory.\n* Tutorials on finetuning, including full-parameter tuning, LoRA, and Q-LoRA.\n* Instructions on deployment, with the example of vLLM and FastChat.\n* Instructions on building demos, including WebUI, CLI demo, etc.\n* Introduction to DashScope API service, as well as the instructions on building an OpenAI-style API for your model.\n* Information about Qwen for tool use, agent, and code interpreter\n* Statistics of long-context understanding evaluation\n* License agreement\n* ...\n\nAlso, if you meet problems, turn to [FAQ](FAQ.md) for help first. Still feeling struggled? Feel free to shoot us issues (better in English so that more people can understand you)! If you would like to help us, send us pull requests with no hesitation! We are always excited about PR! \n\nWould like to chat with us or date us coffee time? Welcome to our Discord or WeChat! \n<br><br>\n\n## News and Updates\n* 2023.11.30 🔥 We release **Qwen-72B** and **Qwen-72B-Chat**, which are trained on 3T tokens and support 32k context, along with **Qwen-1.8B**, and **Qwen-1.8B-Chat**, on ModelScope and Hugging Face. We have also strengthened the System Prompt capabilities of the Qwen-72B-Chat and Qwen-1.8B-Chat, see [example documentation](examples/system_prompt.md). Additionally, support the inference on **Ascend 910** and **Hygon DCU**. Check `ascend-support` and `dcu-support` for more details.\n* 2023.10.17 We release the Int8 quantized model **Qwen-7B-Chat-Int8** and **Qwen-14B-Chat-Int8**. \n* 2023.9.25 🔥 We release **Qwen-14B** and **Qwen-14B-Chat** on ModelScope and Hugging Face, along with [qwen.cpp](https://github.com/QwenLM/qwen.cpp) and [Qwen-Agent](https://github.com/QwenLM/Qwen-Agent). Codes and checkpoints of **Qwen-7B** and **Qwen-7B-Chat** are also updated. **PLEASE PULL THE LATEST VERSION!**\n - Compared to **Qwen-7B** (original), **Qwen-7B** uses more training tokens, increasing from 2.2T tokens to 2.4T tokens, while the context length extends from 2048 to 8192. The Chinese knowledge and coding ability of **Qwen-7B** have been further improved.\n* 2023.9.12 We now support finetuning on the Qwen-7B models, including full-parameter finetuning, LoRA and Q-LoRA.\n* 2023.8.21 We release the Int4 quantized model for Qwen-7B-Chat, **Qwen-7B-Chat-Int4**, which requires low memory costs but achieves improved inference speed. Besides, there is no significant performance degradation on the benchmark evaluation.\n* 2023.8.3 We release both **Qwen-7B** and **Qwen-7B-Chat** on ModelScope and Hugging Face. We also provide a technical memo for more details about the model, including training details and model performance.\n<br>\n\n## Performance\nQwen models outperform the baseline models of similar model sizes on a series of benchmark datasets, e.g., MMLU, C-Eval, GSM8K, MATH, HumanEval, MBPP, BBH, etc., which evaluate the models’ capabilities on natural language understanding, mathematic problem solving, coding, etc. Qwen-72B achieves better performance than LLaMA2-70B on all tasks and outperforms GPT-3.5 on 7 out of 10 tasks. \n\n<p align=\"left\">\n <img src=\"assets/radar_72b.jpg\" width=600px/>\n<p>\n \n\n<br>\n\n| Model | MMLU | C-Eval | GSM8K | MATH | HumanEval | MBPP | BBH | CMMLU |\n|:------------------|:--------:|:--------:|:--------:|:--------:|:---------:|:--------:|:--------:|:--------:|\n| | 5-shot | 5-shot | 8-shot | 4-shot | 0-shot | 3-shot | 3-shot | 5-shot |\n| LLaMA2-7B | 46.8 | 32.5 | 16.7 | 3.3 | 12.8 | 20.8 | 38.2 | 31.8 |\n| LLaMA2-13B | 55.0 | 41.4 | 29.6 | 5.0 | 18.9 | 30.3 | 45.6 | 38.4 |\n| LLaMA2-34B | 62.6 | - | 42.2 | 6.2 | 22.6 | 33.0 | 44.1 | - |\n| ChatGLM2-6B | 47.9 | 51.7 | 32.4 | 6.5 | - | - | 33.7 | - |\n| InternLM-7B | 51.0 | 53.4 | 31.2 | 6.3 | 10.4 | 14.0 | 37.0 | 51.8 |\n| InternLM-20B | 62.1 | 58.8 | 52.6 | 7.9 | 25.6 | 35.6 | 52.5 | 59.0 |\n| Baichuan2-7B | 54.7 | 56.3 | 24.6 | 5.6 | 18.3 | 24.2 | 41.6 | 57.1 |\n| Baichuan2-13B | 59.5 | 59.0 | 52.8 | 10.1 | 17.1 | 30.2 | 49.0 | 62.0 |\n| Yi-34B \t \t | 76.3 | 81.8 | 67.9 | 15.9 | 26.2 | 38.2 | 66.4 | 82.6 |\n| XVERSE-65B \t | 70.8 | 68.6 | 60.3 | - | 26.3 | - | - | - |\n| **Qwen-1.8B** | 45.3 | 56.1 | 32.3 | 2.3 | 15.2 | 14.2 | 22.3 | 52.1 |\n| **Qwen-7B** | 58.2 | 63.5 | 51.7 | 11.6 | 29.9 | 31.6 | 45.0 | 62.2 |\n| **Qwen-14B** | 66.3 | 72.1 | 61.3 | 24.8 | 32.3 | 40.8 | 53.4 | 71.0 |\n| **Qwen-72B** | **77.4** | **83.3** | **78.9** | **35.2** | **35.4** | **52.2** | **67.7** | **83.6** |\n\nFor all compared models, we report the best scores between their official reported results and [OpenCompass](https://opencompass.org.cn/leaderboard-llm). \n\nFor more experimental results (detailed model performance on more benchmark datasets) and details, please refer to our technical report by clicking [here](https://qianwen-res.oss-cn-beijing.aliyuncs.com/QWEN_TECHNICAL_REPORT.pdf).\n<br><br>\n\n## Requirements\n\n* python 3.8 and above\n* pytorch 1.12 and above, 2.0 and above are recommended\n* transformers 4.32 and above\n* CUDA 11.4 and above are recommended (this is for GPU users, flash-attention users, etc.)\n<br>\n\n## Quickstart\n\nBelow, we provide simple examples to show how to use Qwen-Chat with 🤖 ModelScope and 🤗 Transformers.\n\nYou can use our pre-built docker images to skip most of the environment setup steps, see Section [\"Using Pre-built Docker Images\"](#-docker) for more details. \n\nIf not using docker, please make sure you have setup the environment and installed the required packages. Make sure you meet the above requirements, and then install the dependent libraries.\n\n```bash\npip install -r requirements.txt\n```\n\nIf your device supports fp16 or bf16, we recommend installing [flash-attention](https://github.com/Dao-AILab/flash-attention) (**we support flash attention 2 now.**) for higher efficiency and lower memory usage. (**flash-attention is optional and the project can run normally without installing it**)\n\n```bash\ngit clone https://github.com/Dao-AILab/flash-attention\ncd flash-attention && pip install .\n# Below are optional. Installing them might be slow.\n# pip install csrc/layer_norm\n# If the version of flash-attn is higher than 2.1.1, the following is not needed.\n# pip install csrc/rotary\n```\n\nNow you can start with ModelScope or Transformers.\n\n### 🤗 Transformers\n\nTo use Qwen-Chat for the inference, all you need to do is to input a few lines of codes as demonstrated below. Remember to pass in the correct model names or paths, such as \"Qwen/Qwen-7B-Chat\" and \"Qwen/Qwen-14B-Chat\". However, **please make sure that you are using the latest code.**\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\nfrom transformers.generation import GenerationConfig\n\n# Model names: \"Qwen/Qwen-7B-Chat\", \"Qwen/Qwen-14B-Chat\"\ntokenizer = AutoTokenizer.from_pretrained(\"Qwen/Qwen-7B-Chat\", trust_remote_code=True)\n\n# use bf16\n# model = AutoModelForCausalLM.from_pretrained(\"Qwen/Qwen-7B-Chat\", device_map=\"auto\", trust_remote_code=True, bf16=True).eval()\n# use fp16\n# model = AutoModelForCausalLM.from_pretrained(\"Qwen/Qwen-7B-Chat\", device_map=\"auto\", trust_remote_code=True, fp16=True).eval()\n# use cpu only\n# model = AutoModelForCausalLM.from_pretrained(\"Qwen/Qwen-7B-Chat\", device_map=\"cpu\", trust_remote_code=True).eval()\n# use auto mode, automatically select precision based on the device.\nmodel = AutoModelForCausalLM.from_pretrained(\n \"Qwen/Qwen-7B-Chat\",\n device_map=\"auto\",\n trust_remote_code=True\n).eval()\n\n# Specify hyperparameters for generation. But if you use transformers>=4.32.0, there is no need to do this.\n# model.generation_config = GenerationConfig.from_pretrained(\"Qwen/Qwen-7B-Chat\", trust_remote_code=True)\n\n# 1st dialogue turn\nresponse, history = model.chat(tokenizer, \"你好\", history=None)\nprint(response)\n# 你好！很高兴为你提供帮助。\n\n# 2nd dialogue turn\nresponse, history = model.chat(tokenizer, \"给我讲一个年轻人奋斗创业最终取得成功的故事。\", history=history)\nprint(response)\n# 这是一个关于一个年轻人奋斗创业最终取得成功的故事。\n# 故事的主人公叫李明，他来自一个普通的家庭，父母都是普通的工人。从小，李明就立下了一个目标：要成为一名成功的企业家。\n# 为了实现这个目标，李明勤奋学习，考上了大学。在大学期间，他积极参加各种创业比赛，获得了不少奖项。他还利用课余时间去实习，积累了宝贵的经验。\n# 毕业后，李明决定开始自己的创业之路。他开始寻找投资机会，但多次都被拒绝了。然而，他并没有放弃。他继续努力，不断改进自己的创业计划，并寻找新的投资机会。\n# 最终，李明成功地获得了一笔投资，开始了自己的创业之路。他成立了一家科技公司，专注于开发新型软件。在他的领导下，公司迅速发展起来，成为了一家成功的科技企业。\n# 李明的成功并不是偶然的。他勤奋、坚韧、勇于冒险，不断学习和改进自己。他的成功也证明了，只要努力奋斗，任何人都有可能取得成功。\n\n# 3rd dialogue turn\nresponse, history = model.chat(tokenizer, \"给这个故事起一个标题\", history=history)\nprint(response)\n# 《奋斗创业：一个年轻人的成功之路》\n```\n\nRunning Qwen, the base language model, is also simple.\n\n<details>\n <summary>Running Qwen</summary>\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\nfrom transformers.generation import GenerationConfig\n\n# Model names: \"Qwen/Qwen-7B\", \"Qwen/Qwen-14B\" \ntokenizer = AutoTokenizer.from_pretrained(\"Qwen/Qwen-7B\", trust_remote_code=True)\n# use bf16\n# model = AutoModelForCausalLM.from_pretrained(\"Qwen/Qwen-7B\", device_map=\"auto\", trust_remote_code=True, bf16=True).eval()\n# use fp16\n# model = AutoModelForCausalLM.from_pretrained(\"Qwen/Qwen-7B\", device_map=\"auto\", trust_remote_code=True, fp16=True).eval()\n# use cpu only\n# model = AutoModelForCausalLM.from_pretrained(\"Qwen/Qwen-7B\", device_map=\"cpu\", trust_remote_code=True).eval()\n# use auto mode, automatically select precision based on the device.\nmodel = AutoModelForCausalLM.from_pretrained(\n \"Qwen/Qwen-7B\",\n device_map=\"auto\",\n trust_remote_code=True\n).eval()\n\n# Specify hyperparameters for generation. But if you use transformers>=4.32.0, there is no need to do this.\n# model.generation_config = GenerationConfig.from_pretrained(\"Qwen/Qwen-7B\", trust_remote_code=True)\n\ninputs = tokenizer('蒙古国的首都是乌兰巴托（Ulaanbaatar）\\n冰岛的首都是雷克雅未克（Reykjavik）\\n埃塞俄比亚的首都是', return_tensors='pt')\ninputs = inputs.to(model.device)\npred = model.generate(**inputs)\nprint(tokenizer.decode(pred.cpu()[0], skip_special_tokens=True))\n# 蒙古国的首都是乌兰巴托（Ulaanbaatar）\\n冰岛的首都是雷克雅未克（Reykjavik）\\n埃塞俄比亚的首都是亚的斯亚贝巴（Addis Ababa）...\n```\n\n</details>\n\n<p id=\"DownloadModel\">\nIn the event of a network issue while attempting to download model checkpoints and codes from HuggingFace, an alternative approach is to initially fetch the checkpoint from ModelScope and then load it from the local directory as outlined below:\n</p>\n\n```python\nfrom modelscope import snapshot_download\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\n# Downloading model checkpoint to a local dir model_dir\n# model_dir = snapshot_download('qwen/Qwen-7B')\n# model_dir = snapshot_download('qwen/Qwen-7B-Chat')\n# model_dir = snapshot_download('qwen/Qwen-14B')\nmodel_dir = snapshot_download('qwen/Qwen-14B-Chat')\n\n# Loading local checkpoints\n# trust_remote_code is still set as True since we still load codes from local dir instead of transformers\ntokenizer = AutoTokenizer.from_pretrained(model_dir, trust_remote_code=True)\nmodel = AutoModelForCausalLM.from_pretrained(\n model_dir,\n device_map=\"auto\",\n trust_remote_code=True\n).eval()\n```\n\n### 🤖 ModelScope\n\nModelScope is an open-source platform for Model-as-a-Service (MaaS), which provides flexible and cost-effective model service to AI developers. Similarly, you can run the models with ModelScope as shown below:\n\n```python\nfrom modelscope import AutoModelForCausalLM, AutoTokenizer\nfrom modelscope import GenerationConfig\n\n# Model names: \"qwen/Qwen-7B-Chat\", \"qwen/Qwen-14B-Chat\"\ntokenizer = AutoTokenizer.from_pretrained(\"qwen/Qwen-7B-Chat\", trust_remote_code=True)\nmodel = AutoModelForCausalLM.from_pretrained(\"qwen/Qwen-7B-Chat\", device_map=\"auto\", trust_remote_code=True, fp16=True).eval()\nmodel.generation_config = GenerationConfig.from_pretrained(\"Qwen/Qwen-7B-Chat\", trust_remote_code=True) # 可指定不同的生成长度、top_p等相关超参\n\nresponse, history = model.chat(tokenizer, \"你好\", history=None)\nprint(response)\nresponse, history = model.chat(tokenizer, \"浙江的省会在哪里？\", history=history) \nprint(response)\nresponse, history = model.chat(tokenizer, \"它有什么好玩的景点\", history=history)\nprint(response)\n```\n\n### Batch Inference\nQwen supports batch inference. With flash attention enabled, using batch inference can bring a 40% speedup. The example code is shown below:\n```python\nimport torch\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\nfrom transformers import GenerationConfig\nfrom qwen_generation_utils import make_context, decode_tokens, get_stop_words_ids\n\n# To generate attention masks automatically, it is necessary to assign distinct\n# token_ids to pad_token and eos_token, and set pad_token_id in the generation_config.\ntokenizer = AutoTokenizer.from_pretrained(\n './',\n pad_token='<|extra_0|>',\n eos_token='<|endoftext|>',\n padding_side='left',\n trust_remote_code=True\n)\nmodel = AutoModelForCausalLM.from_pretrained(\n './',\n pad_token_id=tokenizer.pad_token_id,\n device_map=\"auto\",\n trust_remote_code=True\n).eval()\nmodel.generation_config = GenerationConfig.from_pretrained('./', pad_token_id=tokenizer.pad_token_id)\n\nall_raw_text = [\"我想听你说爱我。\", \"今天我想吃点啥，甜甜的，推荐下\", \"我马上迟到了，怎么做才能不迟到\"]\nbatch_raw_text = []\nfor q in all_raw_text:\n raw_text, _ = make_context(\n tokenizer,\n q,\n system=\"You are a helpful assistant.\",\n max_window_size=model.generation_config.max_window_size,\n chat_format=model.generation_config.chat_format,\n )\n batch_raw_text.append(raw_text)\n\nbatch_input_ids = tokenizer(batch_raw_text, padding='longest')\nbatch_input_ids = torch.LongTensor(batch_input_ids['input_ids']).to(model.device)\nbatch_out_ids = model.generate(\n batch_input_ids,\n return_dict_in_generate=False,\n generation_config=model.generation_config\n)\npadding_lens = [batch_input_ids[i].eq(tokenizer.pad_token_id).sum().item() for i in range(batch_input_ids.size(0))]\n\nbatch_response = [\n decode_tokens(\n batch_out_ids[i][padding_lens[i]:],\n tokenizer,\n raw_text_len=len(batch_raw_text[i]),\n context_length=(batch_input_ids[i].size(0)-padding_lens[i]),\n chat_format=\"chatml\",\n verbose=False,\n errors='replace'\n ) for i in range(len(all_raw_text))\n]\nprint(batch_response)\n\nresponse, _ = model.chat(tokenizer, \"我想听你说爱我。\", history=None)\nprint(response)\n\nresponse, _ = model.chat(tokenizer, \"今天我想吃点啥，甜甜的，推荐下\", history=None)\nprint(response)\n\nresponse, _ = model.chat(tokenizer, \"我马上迟到了，怎么做才能不迟到\", history=None)\nprint(response)\n```\n\n### CPU\n\nTo deploy our models on CPU, we strongly advise you to use [qwen.cpp](https://github.com/QwenLM/qwen.cpp), which is a pure C++ implementation of Qwen and tiktoken. Check the repo for more details!\n\nAlso, it is also simple to directly run the model on CPU, which requires your specification of device:\n\n```python\nmodel = AutoModelForCausalLM.from_pretrained(\"Qwen/Qwen-7B-Chat\", device_map=\"cpu\", trust_remote_code=True).eval()\n```\n\nHowever, it is likely that you suffer from extremely low inference efficiency.\n\n### Multiple GPUs\n\nIf you suffer from lack of GPU memory and you would like to run the model on more than 1 GPU, you can directly use the default loading method, which is now supported by Transformers. The previous method based on `utils.py` is deprecated.\n\nHowever, though this method is simple, the efficiency of the native pipeline parallelism is low. We advise you to use vLLM with FastChat and please read the section for deployment.\n\n### x86 Platforms\nWhen deploy on Core™/Xeon® Scalable Processors or with Arc™ GPU, [OpenVINO™ Toolkit](https://docs.openvino.ai/2023.3/gen_ai_guide.html) is recommended. You can install and run this [example notebook](https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/254-llm-chatbot). For related issues, you are welcome to file an issue at [OpenVINO repo](https://github.com/openvinotoolkit/openvino_notebooks/issues). \n\n### DashScope\nThe most simple way to use Qwen through APIs is DashScope API service through Alibaba Cloud. We give an introduction to the usage. Additionally, we provide a script for you to deploy an OpenAI-style API on your own servers.\n\nDashScope is the large language model API service provided by Alibaba Cloud, which now supports Qwen. Note that the models behind DashScope are in-house versions temporarily without details provided. The services include `qwen-turbo` and `qwen-plus`, where the former one runs faster and the latter achieves better performance. For more information, visit the documentation [here](https://dashscope.aliyun.com).\n\nPlease head to the official website [link](https://help.aliyun.com/zh/dashscope/developer-reference/activate-dashscope-and-create-an-api-key?spm=a2c4g.11186623.0.0.6c2774fahtfXdn) to create a DashScope account and obtain the API key (AK). We recommend setting the AK with an environment variable:\n```bash\nexport DASHSCOPE_API_KEY=\"YOUR_DASHSCOPE_API_KEY\"\n```\nThen please install the packages and click [here](https://help.aliyun.com/zh/dashscope/developer-reference/install-dashscope-sdk) for the documentation. If you use Python, you can install DashScope with pip:\n```bash\npip install dashscope\n```\nIf you use JAVA SDK, you can install it in this way:\n```xml\n<!-- https://mvnrepository.com/artifact/com.alibaba/dashscope-sdk-java -->\n<dependency>\n <groupId>com.alibaba</groupId>\n <artifactId>dashscope-sdk-java</artifactId>\n <version>the-latest-version</version>\n</dependency>\n```\nThe simplest way to use DashScope is the usage with messages, which is similar to OpenAI API. The example is demonstrated below:\n```python\nimport random\nfrom http import HTTPStatus\nfrom dashscope import Generation\n\n\ndef call_with_messages():\n messages = [{'role': 'system', 'content': 'You are a helpful assistant.'},\n {'role': 'user', 'content': '如何做西红柿鸡蛋？'}]\n gen = Generation()\n response = gen.call(\n Generation.Models.qwen_turbo,\n messages=messages,\n seed=random.randint(1, 10000), # set the random seed, optional, default to 1234 if not set\n result_format='message', # set the result to be \"message\" format.\n )\n return response\n\n\nif __name__ == '__main__':\n response = call_with_messages()\n if response.status_code == HTTPStatus.OK:\n print(response)\n else:\n print('Request id: %s, Status code: %s, error code: %s, error message: %s' % (\n response.request_id, response.status_code,\n response.code, response.message\n ))\n```\nFor more usages, please visit the official website for more details.\n<br><br>\n\n## Quantization\n\n### GPTQ\n\nWe provide a solution based on [AutoGPTQ](https://github.com/PanQiWei/AutoGPTQ), and release the Int4 and Int8 quantized models, which achieve nearly lossless model effects but improved performance on both memory costs and inference speed.\n\nHere we demonstrate how to use our provided quantized models for inference. Before you start, make sure you meet the requirements of auto-gptq (e.g., torch 2.0 and above, transformers 4.32.0 and above, etc.) and install the required packages:\n\n```bash\npip install auto-gptq optimum\n```\n\nIf you meet problems installing `auto-gptq`, we advise you to check out the official [repo](https://github.com/PanQiWei/AutoGPTQ) to find a wheel.\n\n> Note: The pre-compiled `auto-gptq` packages strongly depend on the version of `torch` and its CUDA version. Moreover, due to recent update, \n> you may also encounter unsupported version errors from `transformers`, `optimum`, or `peft`.\n> We recommend using the latest versions meeting the following requirements:\n> - torch==2.1 auto-gptq>=0.5.1 transformers>=4.35.0 optimum>=1.14.0 peft>=0.6.1\n> - torch>=2.0,<2.1 auto-gptq<0.5.0 transformers<4.35.0 optimum<1.14.0 peft>=0.5.0,<0.6.0\n\nThen you can load the quantized model easily and run inference as same as usual:\n\n```python\n# Model names: \"Qwen/Qwen-7B-Chat-Int4\", \"Qwen/Qwen-14B-Chat-Int4\"\nmodel = AutoModelForCausalLM.from_pretrained(\n \"Qwen/Qwen-7B-Chat-Int4\",\n device_map=\"auto\",\n trust_remote_code=True\n).eval()\nresponse, history = model.chat(tokenizer, \"Hi\", history=None)\n```\n\nWe illustrate the model performance of both BF16, Int8 and Int4 models on the benchmark, and we find that the quantized model does not suffer from significant performance degradation. Results are shown below:\n\n| Quantization | MMLU | CEval (val) | GSM8K | Humaneval |\n|----------------------|:----:|:-----------:|:-----:|:---------:|\n| Qwen-1.8B-Chat (BF16)| 43.3 | 55.6 | 33.7 | 26.2 |\n| Qwen-1.8B-Chat (Int8)| 43.1 | 55.8 | 33.0 | 27.4 |\n| Qwen-1.8B-Chat (Int4)| 42.9 | 52.8 | 31.2 | 25.0 |\n| Qwen-7B-Chat (BF16) | 55.8 | 59.7 | 50.3 | 37.2 |\n| Qwen-7B-Chat (Int8) | 55.4 | 59.4 | 48.3 | 34.8 |\n| Qwen-7B-Chat (Int4) | 55.1 | 59.2 | 49.7 | 29.9 |\n| Qwen-14B-Chat (BF16) | 64.6 | 69.8 | 60.1 | 43.9 |\n| Qwen-14B-Chat (Int8) | 63.6 | 68.6 | 60.0 | 48.2 |\n| Qwen-14B-Chat (Int4) | 63.3 | 69.0 | 59.8 | 45.7 |\n| Qwen-72B-Chat (BF16) | 74.4 | 80.1 | 76.4 | 64.6 |\n| Qwen-72B-Chat (Int8) | 73.5 | 80.1 | 73.5 | 62.2 |\n| Qwen-72B-Chat (Int4) | 73.4 | 80.1 | 75.3 | 61.6 |\n\n### Quantization of KV cache\n\n> NOTE: Please be aware that due to the internal mechanism of Hugging Face, the support files for this functionality \n> (i.e., `cache_autogptq_cuda_256.cpp` and `cache_autogptq_cuda_kernel_256.cu`) may be missing. Please manually download\n> them from the Hugging Face Hub and place them into the same folder as the other module files.\n\nThe attention KV cache can be quantized and compressed for storage, to get a higher sample throughput. The arguments `use_cache_quantization` and `use_cache_kernel` in `config.json` are provided to enable KV cache quantization. The specific use method is as follows:\n```python\nmodel = AutoModelForCausalLM.from_pretrained(\n \"Qwen/Qwen-7B-Chat\",\n device_map=\"auto\",\n trust_remote_code=True,\n use_cache_quantization=True,\n use_cache_kernel=True,\n use_flash_attn=False\n)\n```\nAttention: Currently, KV cache quantization and flash attention cannot be used at the same time.\nIf you enable KV cache quantization and flash attention at the same time (`use_flash_attn=True`, `use_cache_quantization=True`, `use_cache_kernel=True`), `use_flash_attn` is disabled by default (`use_flash_attn=false`).\n\nWe have verified that the use of the quantized Int8-KV-Cache model does not suffer from significant performance degradation in downstream evaluation. In the following, we focus on profiling its memory footprint in different conditions. \nThe profiling runs on a single A100-SXM4-80G GPU with PyTorch 2.0.1 and CUDA 11.4. \nWe use BF16 models to generate 1024 tokens by default, and \"OOM\" indicates out-of-memory error.\n\nWith KV cache quantization, the model can infer with a larger batch size (bs).\n\n| USE KV Cache | bs=1 | bs=4 | bs=16 | bs=32 | bs=64 | bs=100 |\n|--------------|:------:|:------:|:------:|:------:|:------:|:------:|\n| No | 16.3GB | 24.1GB | 31.7GB | 48.7GB | OOM | OOM |\n| Yes | 15.5GB | 17.2GB | 22.3GB | 30.2GB | 48.2GB | 72.4GB |\n\nWith KV cache quantization the model can save more memory when generating longer sequence (`sl`, sequence length, referring to the number of tokens generated) at the stage of inference.\n\n| USE KV Cache | sl=512 | sl=1024 | sl=2048 | sl=4096 | sl=8192 |\n|--------------|:------:|:-------:|:-------:|:-------:|:-------:|\n| No | 15.2GB | 16.3GB | 17.6GB | 19.5GB | 23.2GB |\n| Yes | 15GB | 15.5GB | 15.8GB | 16.6GB | 17.6GB |\n\nThe model with KV cache quantization will convert the format of `layer_past` from float to int8, and meanwhile the quantized `layer-past` will also store the quantization parameters.\n\nSpecific steps are as follows:\n\n1. Quantize key/value\n```\n qv,scale,zero_point=quantize_cache_v(v)\n```\n2. Store into layer_past\n\nThe following is the format of quantized `layer_past`:\n```\n layer_past=((q_key,key_scale,key_zero_point),\n (q_value,value_scale,value_zero_point))\n```\n\nThe original format of `layer_past` is shown below:\n```\n layer_past=(key,value)\n```\n\nIf you want to use the attention KV which is quantized, you can use the dequantization operation to convert the Int8 key/value back to the float format as follows:\n```\n v=dequantize_cache_torch(qv,scale,zero_point)\n```\n<br>\n\n\n## Inference Performance\n\nThis section provides the statistics of speed and memory of models in different precisions. The speed and memory profiling are conducted using [this script](https://qianwen-res.oss-cn-beijing.aliyuncs.com/profile.py). \n\nWe measured the average inference speed (tokens/s) and GPU memory usage of generating 2048 with the models in BF16, Int8, and Int4. \n\n<table>\n <tr>\n <td>Model Size</td>\n <td>Quantization</td>\n <td>Speed (Tokens/s)</td>\n <td>GPU Memory Usage</td>\n </tr>\n <tr>\n <td rowspan=\"3\">1.8B</td>\n <td>BF16</td>\n <td>54.09</td>\n <td>4.23GB</td>\n </tr>\n <tr>\n <td>Int8</td>\n <td>55.56</td>\n <td>3.48GB</td>\n </tr>\n <tr>\n <td>Int4</td>\n <td>71.07</td>\n <td>2.91GB</td>\n </tr>\n <tr>\n <td rowspan=\"3\">7B</td>\n <td>BF16</td>\n <td>40.93</td>\n <td>16.99GB</td>\n </tr>\n <tr>\n <td>Int8</td>\n <td>37.47</td>\n <td>11.20GB</td>\n </tr>\n <tr>\n <td>Int4</td>\n <td>50.09</td>\n <td>8.21GB</td>\n </tr>\n <tr>\n <td rowspan=\"3\">14B</td>\n <td>BF16</td>\n <td>32.22</td>\n <td>30.15GB</td>\n </tr>\n <tr>\n <td>Int8</td>\n <td>29.28</td>\n <td>18.81GB</td>\n </tr>\n <tr>\n <td>Int4</td>\n <td>38.72</td>\n <td>13.01GB</td>\n </tr>\n <tr>\n <td rowspan=\"3\">72B</td>\n <td>BF16</td>\n <td>8.48</td>\n <td>144.69GB (2xA100)</td>\n </tr>\n <tr>\n <td>Int8</td>\n <td>9.05</td>\n <td>81.27GB (2xA100)</td>\n </tr>\n <tr>\n <td>Int4</td>\n <td>11.32</td>\n <td>48.86GB</td>\n </tr>\n <tr>\n <td>72B + vLLM</td>\n <td>BF16</td>\n <td>17.60</td>\n <td>2xA100</td>\n </tr>\n</table>\n\nThe profiling runs on a single A100-SXM4-80G GPU (except 2xA100 is mentioned) with PyTorch 2.0.1, CUDA 11.8, and Flash-Attention 2. (72B + vLLM uses PyTorch 2.1.0 and Cuda 11.8.) The inference speed is averaged over the encoded and generated tokens.\n\nNote: The generation speed of the Int4/Int8 models mentioned above is provided by the autogptq library. The current speed of the model loaded using ``AutoModelForCausalLM.from_pretrained`` will be approximately 20% slower. We have reported this issue to the HuggingFace team and will update it promptly if a solution is available.\n\nWe also measure the inference speed and GPU memory usage with different settings of context and generation lengths, Flash-Attention version. You can find the results in the according modelcards on Hugging Face or ModelScope.\n\n## Finetuning\n\n### Usage\nNow we provide the official training script, `finetune.py`, for users to finetune the pretrained model for downstream applications in a simple fashion. Additionally, we provide shell scripts to launch finetuning with no worries. This script supports the training with [DeepSpeed](https://github.com/microsoft/DeepSpeed) and [FSDP](https://engineering.fb.com/2021/07/15/open-source/fsdp/). The shell scripts that we provide use DeepSpeed (Note: this may have conflicts with the latest version of pydantic and you should use make sure `pydantic<2.0`) and Peft. You can install them by:\n```bash\npip install \"peft<0.8.0\" deepspeed\n```\n\nTo prepare your training data, you need to put all the samples into a list and save it to a json file. Each sample is a dictionary consisting of an id and a list for conversation. Below is a simple example list with 1 sample:\n```json\n[\n {\n \"id\": \"identity_0\",\n \"conversations\": [\n {\n \"from\": \"user\",\n \"value\": \"你好\"\n },\n {\n \"from\": \"assistant\",\n \"value\": \"我是一个语言模型，我叫通义千问。\"\n }\n ]\n }\n]\n```\n\nAfter data preparation, you can use the provided shell scripts to run finetuning. Remember to specify the path to the data file, `$DATA`.\n\nThe finetuning scripts allow you to perform:\n- Full-parameter finetuning\n- LoRA\n- Q-LoRA\n\nFull-parameter finetuning requires updating all parameters in the whole training process. To launch your training, run the following script:\n\n```bash\n# Distributed training. We do not provide single-GPU training script as the insufficient GPU memory will break down the training.\nbash finetune/finetune_ds.sh\n```\n\nRemember to specify the correct model name or path, the data path, as well as the output directory in the shell scripts. Another thing to notice is that we use DeepSpeed ZeRO 3 in this script. If you want to make changes, just remove the argument `--deepspeed` or make changes in the DeepSpeed configuration json file based on your requirements. Additionally, this script supports mixed-precision training, and thus you can use `--bf16 True` or `--fp16 True`. Remember to use DeepSpeed when you use fp16 due to mixed precision training. Empirically we advise you to use bf16 to make your training consistent with our pretraining and alignment if your machine supports bf16, and thus we use it by default.\n\nSimilarly, to run LoRA, use another script to run as shown below. Before you start, make sure that you have installed `peft`. Also, you need to specify your paths to your model, data, and output. We advise you to use absolute path for your pretrained model. This is because LoRA only saves the adapter and the absolute path in the adapter configuration json file is used for finding out the pretrained model to load. Also, this script support both bf16 and fp16.\n\n```bash\n# Single GPU training\nbash finetune/finetune_lora_single_gpu.sh\n# Distributed training\nbash finetune/finetune_lora_ds.sh\n```\n\nIn comparison with full-parameter finetuning, LoRA ([paper](https://arxiv.org/abs/2106.09685)) only updates the parameters of adapter layers but keeps the original large language model layers frozen. This allows much fewer memory costs and thus fewer computation costs. \n\nNote that if you use LoRA to finetune the base language model, e.g., Qwen-7B, instead of chat models, e.g., Qwen-7B-Chat, the script automatically switches the embedding and output layer as trainable parameters. This is because the base language model has no knowledge of special tokens brought by ChatML format. Thus these layers should be updated for the model to understand and predict the tokens. Or in another word, if your training brings in special tokens in LoRA, you should set the layers to trainable parameters by setting `modules_to_save` inside the code. Also, if we have these parameters trainable, it is not available to use ZeRO 3, and this is why we use ZeRO 2 in the script by default. If you do not have new trainable parameters, you can switch to ZeRO 3 by changing the DeepSpeed configuration file. Additionally, we find that there is a significant gap between the memory footprint of LoRA with and without these trainable parameters. Therefore, if you have trouble with memory, we advise you to LoRA finetune the chat models. Check the profile below for more information. \n\nIf you still suffer from insufficient memory, you can consider Q-LoRA ([paper](https://arxiv.org/abs/2305.14314)), which uses the quantized large language model and other techniques such as paged attention to allow even fewer memory costs. \n\nNote: to run single-GPU Q-LoRA training, you may need to install `mpi4py` through `pip` or `conda`.\n\nTo run Q-LoRA, directly run the following script:\n\n```bash\n# Single GPU training\nbash finetune/finetune_qlora_single_gpu.sh\n# Distributed training\nbash finetune/finetune_qlora_ds.sh\n```\n\nFor Q-LoRA, we advise you to load our provided quantized model, e.g., Qwen-7B-Chat-Int4. You **SHOULD NOT** use the bf16 models. Different from full-parameter finetuning and LoRA, only fp16 is supported for Q-LoRA. For single-GPU training, we have to use DeepSpeed for mixed-precision training due to our observation of errors caused by torch amp. Besides, for Q-LoRA, the troubles with the special tokens in LoRA still exist. However, as we only provide the Int4 models for chat models, which means the language model has learned the special tokens of ChatML format, you have no worry about the layers. Note that the layers of the Int4 model should not be trainable, and thus if you introduce special tokens in your training, Q-LoRA might not work.\n\n> NOTE: Please be aware that due to the internal mechanisms of Hugging Face, certain non-Python files (e.g., `*.cpp` and `*.cu`) \n> may be missing from the saved checkpoint. You may need to manually copy them to the directory containing other files.\n\nDifferent from full-parameter finetuning, the training of both LoRA and Q-LoRA only saves the adapter parameters. Suppose your training starts from Qwen-7B, you can load the finetuned model for inference as shown below:\n\n```python\nfrom peft import AutoPeftModelForCausalLM\n\nmodel = AutoPeftModelForCausalLM.from_pretrained(\n path_to_adapter, # path to the output directory\n device_map=\"auto\",\n trust_remote_code=True\n).eval()\n```\n\n> NOTE: If `peft>=0.8.0`, it will try to load the tokenizer as well, however, initialized without `trust_remote_code=True`, leading to `ValueError: Tokenizer class QWenTokenizer does not exist or is not currently imported.` Currently, you could downgrade `peft<0.8.0` or move tokenizer files elsewhere to workaround this issue.\n\nIf you want to merge the adapters and save the finetuned model as a standalone model (you can only do this with LoRA, and you CANNOT merge the parameters from Q-LoRA), you can run the following codes:\n\n```python\nfrom peft import AutoPeftModelForCausalLM\n\nmodel = AutoPeftModelForCausalLM.from_pretrained(\n path_to_adapter, # path to the output directory\n device_map=\"auto\",\n trust_remote_code=True\n).eval()\n\nmerged_model = model.merge_and_unload()\n# max_shard_size and safe serialization are not necessary. \n# They respectively work for sharding checkpoint and save the model to safetensors\nmerged_model.save_pretrained(new_model_directory, max_shard_size=\"2048MB\", safe_serialization=True)\n```\n\nThe `new_model_directory` directory will contain the merged model weights and module files. Please note that `*.cu` and `*.cpp` files may be missing in the saved files. If you wish to use the KV cache functionality, please manually copy them. Besides, the tokenizer files are not saved in the new directory in this step. You can copy the tokenizer files or use the following code\n```python\nfrom transformers import AutoTokenizer\n\ntokenizer = AutoTokenizer.from_pretrained(\n path_to_adapter, # path to the output directory\n trust_remote_code=True\n)\n\ntokenizer.save_pretrained(new_model_directory)\n```\n\n\nNote: For multi-GPU training, you need to specify the proper hyperparameters for distributed training based on your machine. Besides, we advise you to specify your maximum sequence length with the argument `--model_max_length`, based on your consideration of data, memory footprint, and training speed.\n\n### Quantize Fine-tuned Models\n\nThis section applies to full-parameter/LoRA fine-tuned models. (Note: You do not need to quantize the Q-LoRA fine-tuned model because it is already quantized.)\nIf you use LoRA, please follow the above instructions to merge your model before quantization. \n\nWe recommend using [auto_gptq](https://github.com/PanQiWei/AutoGPTQ) to quantize the finetuned model. \n\n```bash\npip install auto-gptq optimum\n```\n\nNote: Currently AutoGPTQ has a bug referred in [this issue](https://github.com/PanQiWei/AutoGPTQ/issues/370). Here is a [workaround PR](https://github.com/PanQiWei/AutoGPTQ/pull/495), and you can pull this branch and install from the source.\n\nFirst, prepare the calibration data. You can reuse the fine-tuning data, or use other data following the same format.\n\nSecond, run the following script:\n\n```bash\npython run_gptq.py \\\n --model_name_or_path $YOUR_LORA_MODEL_PATH \\\n --data_path $DATA \\\n --out_path $OUTPUT_PATH \\\n --bits 4 # 4 for int4; 8 for int8\n```\n\nThis step requires GPUs and may costs a few hours according to your data size and model size.\n\nThen, copy all `*.py`, `*.cu`, `*.cpp` files and `generation_config.json` to the output path. And we recommend you to overwrite `config.json` by copying the file from the coresponding official quantized model\n(for example, if you are fine-tuning `Qwen-7B-Chat` and use `--bits 4`, you can find the `config.json` from [Qwen-7B-Chat-Int4](https://huggingface.co/Qwen/Qwen-7B-Chat-Int4/blob/main/config.json)).\nYou should also rename the ``gptq.safetensors`` into ``model.safetensors``.\n\nFinally, test the model by the same method to load the official quantized model. For example,\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\nfrom transformers.generation import GenerationConfig\n\ntokenizer = AutoTokenizer.from_pretrained(\"/path/to/your/model\", trust_remote_code=True)\n\nmodel = AutoModelForCausalLM.from_pretrained(\n \"/path/to/your/model\",\n device_map=\"auto\",\n trust_remote_code=True\n).eval()\n\nresponse, history = model.chat(tokenizer, \"你好\", history=None)\nprint(response)\n```\n\n### Multinode Finetuning\n\nOur provided scripts support multinode finetuning. You can refer to the comments in [script](./finetune/finetune_lora_ds.sh) to correctly set corresponding arguments and launch the script on each node. For more information about multinode distributed training, please refer to [torchrun](https://pytorch.org/docs/stable/elastic/run.html).\n\nNote: DeepSpeed ZeRO 3 requires much greater inter-node communication rate than ZeRO 2, which will significantly reduce the training speed in the case of multinode finetuning. Therefore, we do not recommend using DeepSpeed ZeRO 3 configurations in multinode finetuning scripts.\n\n### Profiling of Memory and Speed\nWe profile the GPU memory and training speed of both LoRA (LoRA (emb) refers to training the embedding and output layer, while LoRA has no trainable embedding and output layer) and Q-LoRA in the setup of single-GPU training. In this test, we experiment on a single A100-SXM4-80G GPU, and we use CUDA 11.8 and Pytorch 2.0. Flash attention 2 is applied. We uniformly use a batch size of 1 and gradient accumulation of 8. We profile the memory (GB) and speed (s/iter) of inputs of different lengths, namely 256, 512, 1024, 2048, 4096, and 8192. We also report the statistics of full-parameter finetuning with Qwen-7B on 2 A100 GPUs. We only report the statistics of 256, 512, and 1024 tokens due to the limitation of GPU memory. \n\nFor Qwen-7B, we also test the performance of multinode finetuning. We experiment using two servers, each containing two A100-SXM4-80G GPUs, and the rest of configurations are the same as other Qwen-7B experiments. The results of multinode finetuning are marked as LoRA (multinode) in the table.\n\nFor Qwen-72B, we experiment in two ways: 1) Lora fintuning + DeepSpeed ZeRO 3 on 4 A100-SXM4-80G GPUs and 2) QLora (int4) fine-tuning on a single A100-SXM4-80G GPU. Note that OOM occurs on 4 A100-SXM4-80G GPUs both with LoRA (emb) fine-tuning and LoRA fine-tuning without Deepspeed ZeRO 3 (you can pass `--deepspeed finetune/ds_config_zero3.json` to [`finetune/finetune_lora_ds.sh`](finetune/finetune_lora_ds.sh) to enable DeepSpeed ZeRO 3).\n\nThe statistics are listed below:\n\n<table>\n <tr>\n <th rowspan=\"2\">Model Size</th><th rowspan=\"2\">Method</th><th rowspan=\"2\">#Nodes</th><th rowspan=\"2\">#GPUs per node</th><th colspan=\"6\" align=\"center\">Sequence Length</th>\n </tr>\n <tr>\n <th align=\"center\">256</th><th align=\"center\">512</th><th align=\"center\">1024</th><th align=\"center\">2048</th><th align=\"center\">4096</th><th align=\"center\">8192</th>\n </tr>\n <tr>\n <th rowspan=\"4\">1.8B</th><td>LoRA</td>\n <td>1</td><td>1</td>\n <td align=\"center\">6.7G / 1.0s/it</td><td align=\"center\">7.4G / 1.0s/it</td><td align=\"center\">8.4G / 1.1s/it</td><td align=\"center\">11.0G / 1.7s/it</td><td align=\"center\">16.2G / 3.3s/it</td><td align=\"center\">21.8G / 6.8s/it</td>\n </tr>\n <tr>\n <td>LoRA (emb)</td>\n <td>1</td><td>1</td>\n <td align=\"center\">13.7G / 1.0s/it</td><td align=\"center\">14.0G / 1.0s/it</td><td align=\"center\">14.0G / 1.1s/it</td><td align=\"center\">15.1G / 1.8s/it</td><td align=\"center\">19.7G / 3.4s/it</td><td align=\"center\">27.7G / 7.0s/it</td>\n </tr>\n <tr>\n <td>Q-LoRA</td>\n <td>1</td><td>1</td>\n <td align=\"center\">5.8G / 1.4s/it</td><td align=\"center\">6.0G / 1.4s/it</td><td align=\"center\">6.6G / 1.4s/it</td><td align=\"center\">7.8G / 2.0s/it</td><td align=\"center\">10.2G / 3.4s/it</td><td align=\"center\">15.8G / 6.5s/it</td>\n </tr>\n <tr>\n <td>Full-parameter</td>\n <td>1</td><td>1</td>\n <td align=\"center\">43.5G / 2.1s/it</td><td align=\"center\">43.5G / 2.2s/it</td><td align=\"center\">43.5G / 2.2s/it</td><td align=\"center\">43.5G / 2.3s/it</td><td align=\"center\">47.1G / 2.8s/it</td><td align=\"center\">48.3G / 5.6s/it</td>\n </tr>\n <tr>\n <th rowspan=\"5\">7B</th>\n <td>LoRA</td>\n <td>1</td><td>1</td>\n <td align=\"center\">20.1G / 1.2s/it</td><td align=\"center\">20.4G / 1.5s/it</td><td align=\"center\">21.5G / 2.8s/it</td><td align=\"center\">23.8G / 5.2s/it</td><td align=\"center\">29.7G / 10.1s/it</td><td align=\"center\">36.6G / 21.3s/it</td>\n </tr>\n <tr>\n <td>LoRA (emb)</td>\n <td>1</td><td>1</td>\n <td align=\"center\">33.7G / 1.4s/it</td><td align=\"center\">34.1G / 1.6s/it</td><td align=\"center\">35.2G / 2.9s/it</td><td align=\"center\">35.1G / 5.3s/it</td><td align=\"center\">39.2G / 10.3s/it</td><td align=\"center\">48.5G / 21.7s/it</td>\n </tr>\n <tr>\n <td>Q-LoRA</td>\n <td>1</td><td>1</td>\n <td align=\"center\">11.5G / 3.0s/it</td><td align=\"center\">11.5G / 3.0s/it</td><td align=\"center\">12.3G / 3.5s/it</td><td align=\"center\">13.9G / 7.0s/it</td><td align=\"center\">16.9G / 11.6s/it</td><td align=\"center\">23.5G / 22.3s/it</td>\n </tr>\n <tr>\n <td>Full-parameter</td>\n<td>1</td><td>2</td>\n<td align=\"center\">139.2G / 4.0s/it</td><td align=\"center\">148.0G / 4.0s/it</td><td align=\"center\">162.0G / 4.5s/it</td><td align=\"center\">-</td><td align=\"center\">-</td><td align=\"center\">-</td>\n </tr>\n <tr>\n <td>LoRA (multinode)</td>\n <td>2</td><td>2</td>\n <td align=\"center\">74.7G / 2.09s/it</td><td align=\"center\">77.6G / 3.16s/it</td><td align=\"center\">84.9G / 5.17s/it</td><td align=\"center\">95.1G / 9.25s/it</td><td align=\"center\">121.1G / 18.1s/it</td><td align=\"center\">155.5G / 37.4s/it</td>\n </tr>\n <tr>\n <th rowspan=\"3\">14B</th>\n <td>LoRA</td>\n <td>1</td><td>1</td>\n <td align=\"center\">34.6G / 1.6s/it</td><td align=\"center\">35.1G / 2.4s/it</td><td align=\"center\">35.3G / 4.4s/it</td><td align=\"center\">37.4G / 8.4s/it</td><td align=\"center\">42.5G / 17.0s/it</td><td align=\"center\">55.2G / 36.0s/it</td>\n </tr>\n <tr>\n <td>LoRA (emb)</td>\n <td>1</td><td>1</td>\n <td align=\"center\">51.2 / 1.7s/it</td><td align=\"center\">51.1G / 2.6s/it</td><td align=\"center\">51.5G / 4.6s/it</td><td align=\"center\">54.1G / 8.6s/it</td><td align=\"center\">56.8G / 17.2s/it</td><td align=\"center\">67.7G / 36.3s/it</td>\n </tr>\n <tr>\n <td>Q-LoRA</td>\n <td>1</td><td>1</td>\n <td align=\"center\">18.7G / 5.3s/it</td><td align=\"center\">18.4G / 6.3s/it</td><td align=\"center\">18.9G / 8.2s/it</td><td align=\"center\">19.9G / 11.8s/it</td><td align=\"center\">23.0G / 20.1s/it</td><td align=\"center\">27.9G / 38.3s/it</td>\n </tr>\n <tr>\n <th rowspan=\"2\">72B</th>\n <td>LoRA + Deepspeed Zero3</td>\n <td>1</td><td>4</td>\n <td align=\"center\">215.4G / 17.6s/it</td><td align=\"center\">217.7G / 20.5s/it</td><td align=\"center\">222.6G / 29.4s/it</td><td align=\"center\">228.8G / 45.7s/it</td><td align=\"center\">249.0G / 83.4s/it</td><td align=\"center\">289.2G / 161.5s/it</td>\n </tr>\n <tr>\n <td>Q-LoRA</td>\n <td>1</td><td>1</td>\n <td align=\"center\">61.4G / 27.4s/it</td><td align=\"center\">61.4G / 31.5s/it</td><td align=\"center\">62.9G / 41.4s/it</td><td align=\"center\">64.1G / 59.5s/it</td><td align=\"center\">68.0G / 97.7s/it</td><td align=\"center\">75.6G / 179.8s/it</td>\n </tr>\n</table>\n\n<br>\n\n## Deployment\n\n### vLLM \n\nFor deployment and fast inference, we suggest using vLLM. \n\nIf you use **CUDA 12.1 and PyTorch 2.1**, you can directly use the following command to install vLLM.\n\n```bash\npip install vllm\n```\n\nOtherwise, please refer to the official vLLM [Installation Instructions](https://docs.vllm.ai/en/latest/getting_started/installation.html).\n\n#### vLLM + Transformer-like Wrapper\n\nYou can download the [wrapper codes](examples/vllm_wrapper.py) and execute the following commands for multiple rounds of dialogue interaction. (Note: It currently only supports the ``model.chat()`` method.)\n\n```python\nfrom vllm_wrapper import vLLMWrapper\n\nmodel = vLLMWrapper('Qwen/Qwen-7B-Chat', tensor_parallel_size=1)\n# model = vLLMWrapper('Qwen/Qwen-7B-Chat-Int4', tensor_parallel_size=1, dtype=\"float16\")\n\nresponse, history = model.chat(query=\"你好\", history=None)\nprint(response)\nresponse, history = model.chat(query=\"给我讲一个年轻人奋斗创业最终取得成功的故事。\", history=history)\nprint(response)\nresponse, history = model.chat(query=\"给这个故事起一个标题\", history=history)\nprint(response)\n```\n\n#### vLLM + Web Demo / OpenAI-like API\n\nYou can use FastChat to lauch a web demo or an OpenAI API server. First, install FastChat:\n\n```bash\npip install \"fschat[model_worker,webui]\"\n```\n\nTo run Qwen with vLLM and FastChat, you need launch a controller by:\n```bash\npython -m fastchat.serve.controller\n```\n\nThen you can launch the model worker, which means loading your model for inference. For single GPU inference, you can directly run:\n```bash\npython -m fastchat.serve.vllm_worker --model-path $model_path --trust-remote-code --dtype bfloat16\n# python -m fastchat.serve.vllm_worker --model-path $model_path --trust-remote-code --dtype float16 # run int4 model\n```\nHowever, if you hope to run the model on multiple GPUs for faster inference or larger memory, you can use tensor parallelism supported by vLLM. Suppose you run the model on 4 GPUs, the command is shown below:\n```bash\npython -m fastchat.serve.vllm_worker --model-path $model_path --trust-remote-code --tensor-parallel-size 4 --dtype bfloat16\n# python -m fastchat.serve.vllm_worker --model-path $model_path --trust-remote-code --tensor-parallel-size 4 --dtype float16 # run int4 model\n```\n\nAfter launching your model worker, you can launch a:\n\n* Web UI Demo\n```bash\npython -m fastchat.serve.gradio_web_server\n```\n\n* OpenAI API\n```bash\npython -m fastchat.serve.openai_api_server --host localhost --port 8000\n```\n\nHowever, if you find it difficult to use vLLM and FastChat, you can try our provided simplest methods to deploy a web demo, CLI demo, and API.\n\n\n### Web UI\n\nWe provide code for users to build a web UI demo (thanks to @wysaid). Before you start, make sure you install the following packages:\n\n```\npip install -r requirements_web_demo.txt\n```\n\nThen run the command below and click on the generated link:\n\n```bash\npython web_demo.py\n```\n\n<p align=\"center\">\n <br>\n <img src=\"assets/web_demo.gif\" width=\"600\" />\n <br>\n<p>\n\n### CLI Demo\n\nWe provide a CLI demo example in `cli_demo.py`, which supports streaming output for the generation. Users can interact with Qwen-7B-Chat by inputting prompts, and the model returns model outputs in the streaming mode. Run the command below:\n\n```bash\npython cli_demo.py\n```\n\n<p align=\"center\">\n <br>\n <img src=\"assets/cli_demo.gif\" width=\"600\" />\n <br>\n<p>\n<br>\n\n### API\n\nWe provide methods to deploy local API based on OpenAI API (thanks to @hanpenggit). Before you start, install the required packages:\n\n```bash\npip install fastapi uvicorn \"openai<1.0\" pydantic sse_starlette\n```\n\nThen run the command to deploy your API:\n\n```bash\npython openai_api.py\n```\n\nYou can change your arguments, e.g., `-c` for checkpoint name or path, `--cpu-only` for CPU deployment, etc. If you meet problems launching your API deployment, updating the packages to the latest version can probably solve them.\n\nUsing the API is also simple. See the example below:\n\n```python\nimport openai\nopenai.api_base = \"http://localhost:8000/v1\"\nopenai.api_key = \"none\"\n\n# create a request activating streaming response\nfor chunk in openai.ChatCompletion.create(\n model=\"Qwen\",\n messages=[\n {\"role\": \"user\", \"content\": \"你好\"}\n ],\n stream=True \n # Specifying stop words in streaming output format is not yet supported and is under development.\n):\n if hasattr(chunk.choices[0].delta, \"content\"):\n print(chunk.choices[0].delta.content, end=\"\", flush=True)\n\n# create a request not activating streaming response\nresponse = openai.ChatCompletion.create(\n model=\"Qwen\",\n messages=[\n {\"role\": \"user\", \"content\": \"你好\"}\n ],\n stream=False,\n stop=[] # You can add custom stop words here, e.g., stop=[\"Observation:\"] for ReAct prompting.\n)\nprint(response.choices[0].message.content)\n```\n\n<p align=\"center\">\n <br>\n <img src=\"assets/openai_api.gif\" width=\"600\" />\n <br>\n<p>\n\n**Function calling** is also supported (but only when `stream=False` for the moment). See the [example usage](examples/function_call_examples.py) here.\n<br><br>\n\n## 🐳 Docker\n\nTo simplify the deployment process, we provide docker images with pre-built environments: [qwenllm/qwen](https://hub.docker.com/r/qwenllm/qwen). You only need to install the driver and download model files to launch demos, deploy OpenAI API, and finetune the model.\n\n### Preparation\n\n1. Install the correct version of Nvidia driver depending on the image to use:\n - `qwenllm/qwen:cu117` (**recommend**): `>= 515.48.07`\n - `qwenllm/qwen:cu114` (w/o flash-attention): `>= 470.82.01`\n - `qwenllm/qwen:cu121`: `>= 530.30.02`\n - `qwenllm/qwen:latest`: same as `qwenllm/qwen:cu117`\n\n2. Install and configure [docker](https://docs.docker.com/engine/install/) and [nvidia-container-toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html):\n\n```bash\n# configure docker\nsudo systemctl start docker\n# test if docker is correctly installed\nsudo docker run hello-world\n\n# configure nvidia-container-toolkit\nsudo nvidia-ctk runtime configure --runtime=docker\nsudo systemctl restart docker\n# test if nvidia-container-toolkit is correctly installed\nsudo docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi\n```\n\n3. Download model checkpoints and codes to your environment (see [here](#DownloadModel)).\n\n### Deployment\n\nHere we use Qwen-7B-Chat as an example. Before launching a web demo or API, you can setup the configuration as shown below:\n\n```bash\nIMAGE_NAME=qwenllm/qwen:cu117\nPORT=8901\nCHECKPOINT_PATH=/path/to/Qwen-7B-Chat # Path to downloaded model checkpoints and codes\n```\nThe following scripts can help you build:\n\n* OpenAI API\n```bash\nbash docker/docker_openai_api.sh -i ${IMAGE_NAME} -c ${CHECKPOINT_PATH} --port ${PORT}\n```\n\n* Web UI\n```bash\nbash docker/docker_web_demo.sh -i ${IMAGE_NAME} -c ${CHECKPOINT_PATH} --port ${PORT}\n```\n\n* CLI Demo\n```bash\nbash docker/docker_cli_demo.sh -i ${IMAGE_NAME} -c ${CHECKPOINT_PATH}\n```\n\nThe commands above will automatically download the required image and launch a Web UI demo in background (the service will auto-restart). You can open `http://localhost:${PORT}` on the host to use the demo.\n\nThe demo is successfully launched if you see the following output:\n\n```text\nSuccessfully started web demo. Open '...' to try!\nRun `docker logs ...` to check demo status.\nRun `docker rm -f ...` to stop and remove the demo.\n```\n\nIf you want to check the status of the demo, you can use `docker logs qwen` to display outputs.\n\nYou can use `docker rm -f qwen` to stop the service and remove the container.\n\n\n### Finetuning\n\nThe method of finetuning using the pre-built Docker image is basically the same as [the above chapter](#Finetuning) (we have already installed dependencies in the image):\n\nThe following is an example of single-GPU LoRA:\n```bash\nIMAGE_NAME=qwenllm/qwen:cu117\nCHECKPOINT_PATH=/path/to/Qwen-7B # Path to downloaded model checkpoints and codes\n#CHECKPOINT_PATH=/path/to/Qwen-7B-Chat-Int4 # Path to downloaded model checkpoints and codes (Q-LoRA)\nDATA_PATH=/path/to/data/root # Prepare finetune data at ${DATA_PATH}/example.json\nOUTPUT_PATH=/path/to/output/checkpoint # Path to finetune outputs\n\n# Use all host devices by default\nDEVICE=all\n# If you need to specify GPUs for training, set device as follow (NOTE: internal quotation marks cannot be omitted)\n#DEVICE='\"device=0,1,2,3\"'\n\nmkdir -p ${OUTPUT_PATH}\n\n# Single-GPU LoRA finetuning\ndocker run --gpus ${DEVICE} --rm --name qwen \\\n --mount type=bind,source=${CHECKPOINT_PATH},target=/data/shared/Qwen/Qwen-7B \\\n --mount type=bind,source=${DATA_PATH},target=/data/shared/Qwen/data \\\n --mount type=bind,source=${OUTPUT_PATH},target=/data/shared/Qwen/output_qwen \\\n --shm-size=2gb \\\n -it ${IMAGE_NAME} \\\n bash finetune/finetune_lora_single_gpu.sh -m /data/shared/Qwen/Qwen-7B/ -d /data/shared/Qwen/data/example.json\n```\n\nTo make a change to single-GPU Q-LoRA for example, you just need to modify the bash command inside `docker run`:\n```bash\nbash finetune/finetune_qlora_single_gpu.sh -m /data/shared/Qwen/Qwen-7B-Chat-Int4/ -d /data/shared/Qwen/data/example.json\n```\n<br>\n\n## 🔥 System Prompt\nQwen-1.8-Chat and Qwen-72B-Chat have been fully trained on diverse system prompts with multiple rounds of complex interactions, so that they can follow a variety of system prompts and realize model customization in context, further improving the scalability of Qwen-chat.\n\nWith System Prompt, Qwen-Chat can realize **roly playing**, **language style transfer**, **task setting**, and **behavior setting**.\n\n![](assets/system_prompt_language_style.png)\n\n![](assets/system_prompt_role_play_en.png)\n\nFor more information, please refer to the [example documentation](examples/system_prompt.md).\n\n## Tool Usage\n\nQwen-Chat has been optimized for tool usage and function calling capabilities. Users can develop agents, LangChain applications, and even augment Qwen with a Python Code Interpreter.\n\nWe provide documentation on how to implement tool calls based on the principle of ReAct Prompting, please refer to [the ReAct example](examples/react_prompt.md). Based on this principle, we provide support for function calling in [openai_api.py](openai_api.py).\n\nWe have tested the model's tool calling capabilities on our open-source Chinese evaluation benchmark and found that Qwen-Chat consistently performs well:\n\n<table>\n <tr>\n <th colspan=\"4\" align=\"center\">Chinese Tool-Use Benchmark (Version 20231206)</th>\n </tr>\n <tr>\n <th align=\"center\">Model</th><th align=\"center\">Tool Selection (Acc.↑)</th><th align=\"center\">Tool Input (Rouge-L↑)</th><th align=\"center\">False Positive Error↓</th>\n </tr>\n <tr>\n <td>GPT-4</td><td align=\"center\">98.0%</td><td align=\"center\">0.953</td><td align=\"center\">23.9%</td>\n </tr>\n <tr>\n <td>GPT-3.5</td><td align=\"center\">74.5%</td><td align=\"center\">0.807</td><td align=\"center\">80.6%</td>\n </tr>\n <tr>\n <td>Qwen-1_8B-Chat</td><td align=\"center\">85.0%</td><td align=\"center\">0.839</td><td align=\"center\">27.6%</td>\n </tr>\n <tr>\n <td>Qwen-7B-Chat</td><td align=\"center\">95.5%</td><td align=\"center\">0.900</td><td align=\"center\">11.6%</td>\n </tr>\n <tr>\n <td>Qwen-14B-Chat</td><td align=\"center\">96.9%</td><td align=\"center\">0.917</td><td align=\"center\">5.6%</td>\n </tr>\n <tr>\n <td>Qwen-72B-Chat</td><td align=\"center\">98.2%</td><td align=\"center\">0.927</td><td align=\"center\">1.1%</td>\n </tr>\n</table>\n\nTo assess Qwen's ability to use the Python Code Interpreter for tasks such as mathematical problem solving, data visualization, and other general-purpose tasks such as file handling and web scraping, we have created and open-sourced a benchmark specifically designed for evaluating these capabilities. You can find the benchmark at this [link](https://github.com/QwenLM/Qwen-Agent/tree/main/benchmark).\n\nWe have observed that Qwen performs well in terms of code executability and result accuracy when generating code:\n\n<table>\n <tr>\n <th colspan=\"5\" align=\"center\">Code Interpreter Benchmark (Version 20231206)</th>\n </tr>\n <tr>\n <th rowspan=\"2\" align=\"center\">Model</th>\n <th colspan=\"3\" align=\"center\">Accuracy of Code Execution Results (%)</th>\n <th colspan=\"1\" align=\"center\">Executable Rate of Code (%)</th>\n </tr>\n <tr>\n <th align=\"center\">Math↑</th><th align=\"center\">Visualization-Hard↑</th><th align=\"center\">Visualization-Easy↑</th><th align=\"center\">General↑</th>\n </tr>\n <tr>\n <td>GPT-4</td>\n <td align=\"center\">82.8</td>\n <td align=\"center\">66.7</td>\n <td align=\"center\">60.8</td>\n <td align=\"center\">82.8</td>\n </tr>\n <tr>\n <td>GPT-3.5</td>\n <td align=\"center\">47.3</td>\n <td align=\"center\">33.3</td>\n <td align=\"center\">55.7</td>\n <td align=\"center\">74.1</td>\n </tr>\n <tr>\n <td>LLaMA2-13B-Chat</td>\n <td align=\"center\">8.3</td>\n <td align=\"center\">1.2</td>\n <td align=\"center\">15.2</td>\n <td align=\"center\">48.3</td>\n </tr>\n <tr>\n <td>CodeLLaMA-13B-Instruct</td>\n <td align=\"center\">28.2</td>\n <td align=\"center\">15.5</td>\n <td align=\"center\">21.5</td>\n <td align=\"center\">74.1</td>\n </tr>\n <tr>\n <td>InternLM-20B-Chat</td>\n <td align=\"center\">34.6</td>\n <td align=\"center\">10.7</td>\n <td align=\"center\">25.1</td>\n <td align=\"center\">65.5</td>\n </tr>\n <tr>\n <td>ChatGLM3-6B</td>\n <td align=\"center\">54.2</td>\n <td align=\"center\">4.8</td>\n <td align=\"center\">15.2</td>\n <td align=\"center\">67.1</td>\n </tr>\n <tr>\n <td>Qwen-1.8B-Chat</td>\n <td align=\"center\">25.6</td>\n <td align=\"center\">21.4</td>\n <td align=\"center\">22.8</td>\n <td align=\"center\">65.5</td>\n </tr>\n <tr>\n <td>Qwen-7B-Chat</td>\n <td align=\"center\">41.9</td>\n <td align=\"center\">23.8</td>\n <td align=\"center\">38.0</td>\n <td align=\"center\">67.2</td>\n </tr>\n <tr>\n <td>Qwen-14B-Chat</td>\n <td align=\"center\">58.4</td>\n <td align=\"center\">31.0</td>\n <td align=\"center\">45.6</td>\n <td align=\"center\">65.5</td>\n </tr>\n <tr>\n <td>Qwen-72B-Chat</td>\n <td align=\"center\">72.7</td>\n <td align=\"center\">41.7</td>\n <td align=\"center\">43.0</td>\n <td align=\"center\">82.8</td>\n </tr>\n</table>\n\n<p align=\"center\">\n <br>\n <img src=\"assets/code_interpreter_showcase_001.jpg\" />\n <br>\n<p>\n\n<br>\n\n## Long-Context Understanding\n\nTo extend the context length and break the bottleneck of training sequence length, we introduce several techniques, including NTK-aware interpolation, window attention, and LogN attention scaling, to extend the context length of Qwen-14B from 2K to over 8K tokens, and Qwen-1.8B/7B from 8K to 32K tokens. \n\nFor Qwen-72B, we adapt RoPE to longer contexts with a larger rotary base. Qwen-72B supports the max context length of 32K tokens.\n\nWe conduct language modeling experiments on the arXiv dataset with the PPL evaluation and find that Qwen can reach outstanding performance in the scenario of long context. Results are demonstrated below:\n\n<table>\n <tr>\n <th rowspan=\"2\">Model</th><th colspan=\"6\" align=\"center\">Sequence Length</th>\n </tr>\n <tr>\n <th align=\"center\">1024</th><th align=\"center\">2048</th><th align=\"center\">4096</th><th align=\"center\">8192</th><th align=\"center\">16384</th><th align=\"center\">32768</th>\n </tr>\n <tr>\n <td>Qwen-7B (original)</td><td align=\"center\">4.23</td><td align=\"center\">3.78</td><td align=\"center\">39.35</td><td align=\"center\">469.81</td><td align=\"center\">2645.09</td><td align=\"center\">-</td>\n </tr>\n <tr>\n <td>+ dynamic_ntk</td><td align=\"center\">4.23</td><td align=\"center\">3.78</td><td align=\"center\">3.59</td><td align=\"center\">3.66</td><td align=\"center\">5.71</td><td align=\"center\">-</td>\n </tr>\n <tr>\n <td>+ dynamic_ntk + logn</td><td align=\"center\">4.23</td><td align=\"center\">3.78</td><td align=\"center\">3.58</td><td align=\"center\">3.56</td><td align=\"center\">4.62</td><td align=\"center\">-</td>\n </tr>\n <tr>\n <td>+ dynamic_ntk + logn + window_attn</td><td align=\"center\">4.23</td><td align=\"center\">3.78</td><td align=\"center\">3.58</td><td align=\"center\">3.49</td><td align=\"center\">4.32</td><td align=\"center\">-</td>\n </tr>\n <tr>\n <tr>\n <td>Qwen-1.8B</td><td align=\"center\"><b>5.00</b></td><td align=\"center\"><b>4.48</b></td><td align=\"center\"><b>4.13</b></td><td align=\"center\"><b>3.89</b></td><td align=\"center\">17.42</td><td align=\"center\">433.85</td>\n </tr>\n <tr>\n <td>+ dynamic_ntk + logn + window_attn</td><td align=\"center\"><b>5.00</b></td><td align=\"center\"><b>4.48</b></td><td align=\"center\"><b>4.14</b></td><td align=\"center\"><b>3.93</b></td><td align=\"center\"><b>3.82</b></td><td align=\"center\"><b>3.83</b></td>\n </tr>\n <tr>\n <td>Qwen-7B</td><td align=\"center\"><b>4.23</b></td><td align=\"center\"><b>3.81</b></td><td align=\"center\"><b>3.52</b></td><td align=\"center\"><b>3.31</b></td><td align=\"center\">7.27</td><td align=\"center\">181.49</td>\n </tr>\n <tr>\n <td>+ dynamic_ntk + logn + window_attn</td><td align=\"center\"><b>4.23</b></td><td align=\"center\"><b>3.81</b></td><td align=\"center\"><b>3.52</b></td><td align=\"center\"><b>3.33</b></td><td align=\"center\"><b>3.22</b></td><td align=\"center\"><b>3.17</b></td>\n </tr>\n <tr>\n <td>Qwen-14B</td><td align=\"center\"><b>-</b></td><td align=\"center\"><b>3.46</b></td><td align=\"center\">22.79</td><td align=\"center\">334.65</td><td align=\"center\">3168.35</td><td align=\"center\">-</td>\n </tr>\n <tr>\n <td>+ dynamic_ntk + logn + window_attn</td><td align=\"center\"><b>-</b></td><td align=\"center\"><b>3.46</b></td><td align=\"center\"><b>3.29</b></td><td align=\"center\"><b>3.18</b></td><td align=\"center\">3.42</td><td align=\"center\">-</td>\n </tr>\n <tr>\n <td>Qwen-72B</td><td align=\"center\"><b>-</b></td><td align=\"center\"><b>-</b></td><td align=\"center\">-</td><td align=\"center\"><b>2.83</b></td><td align=\"center\"><b>2.73</b></td><td align=\"center\"><b>2.72</b></td>\n </tr>\n </tr>\n</table>\n\nFurthermore, to verify the ability of Qwen-72B-Chat on long text understanding, we tested it on [L-Eval](https://arxiv.org/abs/2307.11088) (closed-ended tasks). The results are as follows:\n\n| Model | Input Length | Average | Coursera | GSM | QuALITY | TOEFL | CodeU | SFcition |\n|:------------------|:------------:|:---------:|:----------:|:----------:|:----------:|:----------:|:----------:|:----------:|\n| ChatGPT-3.5-16k | 16K | 60.73 | **63.51** | **84.00** | 61.38 | 78.43 | **12.22** | 64.84 |\n| **Qwen-72B-Chat** | 32K | **62.30** | 58.13 | 76.00 | **77.22** | **86.24** | 6.66 | **69.53** |\n\nWe conducted the \"needle in a haystack\" experiment (the idea came from [@Greg Kamradt](https://twitter.com/GregKamradt/status/1727018183608193393)) to test whether the model can retrieve information at different positions in the inputs of different lengths, the result is as follows:\n\n![](assets/qwen_72b_needle_in_a_haystack.png)\n\nThe above results show that Qwen-72B-Chat can accurately retrieve information placed in various positions within an input length of 32k, proving its excellent long text understanding capabilities.\n\n## Tokenizer\n\nOur tokenizer based on tiktoken is different from other tokenizers, e.g., sentencepiece tokenizer. You need to pay attention to special tokens, especially in finetuning. For more detailed information on the tokenizer and related use in fine-tuning, please refer to the [documentation](tokenization_note.md).\n<br><br>\n\n## Reproduction\n\nFor your reproduction of the model performance on benchmark datasets, we provide scripts for you to reproduce the results. Check [eval/EVALUATION.md](eval/EVALUATION.md) for more information. Note that the reproduction may lead to slight differences from our reported results.\n<br><br>\n\n## FAQ\n\nIf you meet problems, please refer to [FAQ](FAQ.md) and the issues first to search a solution before you launch a new issue.\n<br><br>\n\n## Citation\nIf you find our work helpful, feel free to give us a cite.\n\n```\n@article{qwen,\n title={Qwen Technical Report},\n author={Jinze Bai and Shuai Bai and Yunfei Chu and Zeyu Cui and Kai Dang and Xiaodong Deng and Yang Fan and Wenbin Ge and Yu Han and Fei Huang and Binyuan Hui and Luo Ji and Mei Li and Junyang Lin and Runji Lin and Dayiheng Liu and Gao Liu and Chengqiang Lu and Keming Lu and Jianxin Ma and Rui Men and Xingzhang Ren and Xuancheng Ren and Chuanqi Tan and Sinan Tan and Jianhong Tu and Peng Wang and Shijie Wang and Wei Wang and Shengguang Wu and Benfeng Xu and Jin Xu and An Yang and Hao Yang and Jian Yang and Shusheng Yang and Yang Yao and Bowen Yu and Hongyi Yuan and Zheng Yuan and Jianwei Zhang and Xingxuan Zhang and Yichang Zhang and Zhenru Zhang and Chang Zhou and Jingren Zhou and Xiaohuan Zhou and Tianhang Zhu},\n journal={arXiv preprint arXiv:2309.16609},\n year={2023}\n}\n```\n<br>\n\n## License Agreement\n\nThe source code provided at <https://github.com/QwenLM/Qwen> is licensed under the [Apache 2.0 License](./LICENSE) that can be found at the root directory.\n\nResearchers and developers are free to use the codes and model weights of both Qwen and Qwen-Chat. For their commercial use, please check the License Agreement accompanying each model.\n\n- Qwen-72B, Qwen-14B, and Qwen-7B are licensed under the [Tongyi Qianwen LICENSE AGREEMENT](./Tongyi%20Qianwen%20LICENSE%20AGREEMENT) that can be found at the corresponding HuggingFace and ModelScope repository. For commercial use, please fill out the form ([72B](https://dashscope.console.aliyun.com/openModelApply/Qwen-72B-Chat), [14B](https://dashscope.console.aliyun.com/openModelApply/Qwen-14B-Chat), and [7B](https://dashscope.console.aliyun.com/openModelApply/qianwen)) to apply.\n\n- Qwen-1.8B is licensed under the [Tongyi Qianwen RESEARCH LICENSE AGREEMENT](./Tongyi%20Qianwen%20RESEARCH%20LICENSE%20AGREEMENT) that can be found at the corresponding HuggingFace and ModelScope repository. For commercial use, please contact us.\n<br><br>\n\n## Contact Us\n\nIf you are interested to leave a message to either our research team or product team, join our Discord or WeChat groups! Also, feel free to send an email to qianwen_opensource@alibabacloud.com.\n\n" }, { "path": "README_CN.md", "content": "<p align=\"left\">\n 中文</a>  ｜  <a href=\"README.md\">English</a>  ｜  <a href=\"README_JA.md\">日本語</a> ｜  <a href=\"README_FR.md\">Français</a> ｜  <a href=\"README_ES.md\">Español</a>\n</p>\n<br><br>\n\n<p align=\"center\">\n <img src=\"https://qianwen-res.oss-cn-beijing.aliyuncs.com/logo_qwen.jpg\" width=\"400\"/>\n<p>\n<br>\n\n<p align=\"center\">\n 🤗 <a href=\"https://huggingface.co/Qwen\">Hugging Face</a>   |   🤖 <a href=\"https://modelscope.cn/organization/qwen\">ModelScope</a>   |    📑 <a href=\"https://arxiv.org/abs/2309.16609\">Paper</a>    ｜   🖥️ <a href=\"https://modelscope.cn/studios/qwen/Qwen-72B-Chat-Demo/summary\">Demo</a>\n<br>\n<a href=\"assets/wechat.png\">WeChat (微信)</a>   |   <a href=\"https://discord.gg/z3GAxXZ9Ce\">Discord</a>   ｜   <a href=\"https://dashscope.aliyun.com\">API</a>   |   <a href=\"https://qianwen.aliyun.com\">Web</a>   |   <a href=\"https://apps.apple.com/cn/app/%E9%80%9A%E4%B9%89%E5%8D%83%E9%97%AE/id6466733523\">APP</a>\n</p>\n<br><br>\n\n> [!Important]\n> Qwen2已开，欢迎关注！看这里：[QwenLM/Qwen2](https://github.com/QwenLM/Qwen2)\n>\n> Qwen2模型代码和用法相比此前版本有较大不同，因此我们使用新的repo进行维护。此repo ([QwenLM/Qwen](https://github.com/QwenLM/Qwen)) 已停止主要更新维护。\n\n> [!Warning]\n> 请勿混用[Qwen](https://github.com/QwenLM/Qwen)和[Qwen2](https://github.com/QwenLM/Qwen2)代码，两者并不兼容。\n<br>\n\n| | Qwen-Chat | Qwen-Chat (Int4) | Qwen-Chat (Int8) | Qwen |\n|-----|:------------------------------------------------------------------------------------------------------------------------------------:|:----------------------------------------------------------------------------------------------------------------------------------------------:|:---------------------------------------------------------------:|:--------------------------------------------------------------------------------------------------------------------------:|\n| 1.8B | <a href=\"https://modelscope.cn/models/qwen/Qwen-1_8B-Chat/summary\">🤖</a> <a href=\"https://huggingface.co/Qwen/Qwen-1_8B-Chat\">🤗</a> | <a href=\"https://modelscope.cn/models/qwen/Qwen-1_8B-Chat-Int4/summary\">🤖</a> <a href=\"https://huggingface.co/Qwen/Qwen-1_8B-Chat-Int4\">🤗</a> | <a href=\"https://modelscope.cn/models/qwen/Qwen-1_8B-Chat-Int8/summary\">🤖</a> <a href=\"https://huggingface.co/Qwen/Qwen-1_8B-Chat-Int8\">🤗</a> | <a href=\"https://modelscope.cn/models/qwen/Qwen-1_8B/summary\">🤖</a> <a href=\"https://huggingface.co/Qwen/Qwen-1_8B\">🤗</a> |\n| 7B | <a href=\"https://modelscope.cn/models/qwen/Qwen-7B-Chat/summary\">🤖</a> <a href=\"https://huggingface.co/Qwen/Qwen-7B-Chat\">🤗</a> | <a href=\"https://modelscope.cn/models/qwen/Qwen-7B-Chat-Int4/summary\">🤖</a> <a href=\"https://huggingface.co/Qwen/Qwen-7B-Chat-Int4\">🤗</a> | <a href=\"https://modelscope.cn/models/qwen/Qwen-7B-Chat-Int8/summary\">🤖</a> <a href=\"https://huggingface.co/Qwen/Qwen-7B-Chat-Int8\">🤗</a> | <a href=\"https://modelscope.cn/models/qwen/Qwen-7B/summary\">🤖</a> <a href=\"https://huggingface.co/Qwen/Qwen-7B\">🤗</a> |\n| 14B | <a href=\"https://modelscope.cn/models/qwen/Qwen-14B-Chat/summary\">🤖</a> <a href=\"https://huggingface.co/Qwen/Qwen-14B-Chat\">🤗</a> | <a href=\"https://modelscope.cn/models/qwen/Qwen-14B-Chat-Int4/summary\">🤖</a> <a href=\"https://huggingface.co/Qwen/Qwen-14B-Chat-Int4\">🤗</a> | <a href=\"https://modelscope.cn/models/qwen/Qwen-14B-Chat-Int8/summary\">🤖</a> <a href=\"https://huggingface.co/Qwen/Qwen-14B-Chat-Int8\">🤗</a> | <a href=\"https://modelscope.cn/models/qwen/Qwen-14B/summary\">🤖</a> <a href=\"https://huggingface.co/Qwen/Qwen-14B\">🤗</a> |\n| 72B | <a href=\"https://modelscope.cn/models/qwen/Qwen-72B-Chat/summary\">🤖</a> <a href=\"https://huggingface.co/Qwen/Qwen-72B-Chat\">🤗</a> | <a href=\"https://modelscope.cn/models/qwen/Qwen-72B-Chat-Int4/summary\">🤖</a> <a href=\"https://huggingface.co/Qwen/Qwen-72B-Chat-Int4\">🤗</a> | <a href=\"https://modelscope.cn/models/qwen/Qwen-72B-Chat-Int8/summary\">🤖</a> <a href=\"https://huggingface.co/Qwen/Qwen-72B-Chat-Int8\">🤗</a> | <a href=\"https://modelscope.cn/models/qwen/Qwen-72B/summary\">🤖</a> <a href=\"https://huggingface.co/Qwen/Qwen-72B\">🤗</a> |\n\n\n \n我们开源了**Qwen**（通义千问）系列工作，当前开源模型的参数规模为18亿（1.8B）、70亿（7B）、140亿（14B）和720亿（72B）。本次开源包括基础模型**Qwen**，即**Qwen-1.8B**、**Qwen-7B**、**Qwen-14B**、**Qwen-72B**，以及对话模型**Qwen-Chat**，即**Qwen-1.8B-Chat**、**Qwen-7B-Chat**、**Qwen-14B-Chat**和**Qwen-72B-Chat**。模型链接在表格中，请点击了解详情。同时，我们公开了我们的<b><a href=\"https://arxiv.org/abs/2309.16609\">技术报告</a></b>，请点击上方论文链接查看。\n\n当前基础模型已经稳定训练了大规模高质量且多样化的数据，覆盖多语言（当前以中文和英文为主），总量高达3万亿token。在相关基准评测中，Qwen系列模型拿出非常有竞争力的表现，显著超出同规模模型并紧追一系列最强的闭源模型。此外，我们利用SFT和RLHF技术实现对齐，从基座模型训练得到对话模型。Qwen-Chat具备聊天、文字创作、摘要、信息抽取、翻译等能力，同时还具备一定的代码生成和简单数学推理的能力。在此基础上，我们针对LLM对接外部系统等方面针对性地做了优化，当前具备较强的工具调用能力，以及最近备受关注的Code Interpreter的能力和扮演Agent的能力。我们将各个大小模型的特点列到了下表。\n\n| 模型 | 开源日期 | 最大上下文长度 | System Prompt强化 | 预训练token数 | 微调（Q-Lora）最小GPU用量 | 生成2048个token的最小显存占用（Int4） | 工具调用 |\n|:----------|:--------:|:-------:|:---------------:|:---------:|:-----------------:|:-------------------:|:----:|\n| Qwen-1.8B | 23.11.30 | 32K | ✅ | 2.2T | 5.8GB | 2.9GB | ✅ | \n| Qwen-7B | 23.08.03 | 32K | ❎ | 2.4T | 11.5GB | 8.2GB | ✅ | \n| Qwen-14B | 23.09.25 | 8K | ❎ | 3.0T | 18.7GB | 13.0GB | ✅ |\n| Qwen-72B | 23.11.30 | 32K | ✅ | 3.0T | 61.4GB | 48.9GB | ✅ | \n\n \n在这个项目中，你可以了解到以下内容\n\n* 快速上手Qwen-Chat教程，玩转大模型推理\n* 量化模型相关细节，包括GPTQ和KV cache量化\n* 推理性能数据，包括推理速度和显存占用\n* 微调的教程，帮你实现全参数微调、LoRA以及Q-LoRA\n* 部署教程，以vLLM和FastChat为例\n* 搭建Demo的方法，包括WebUI和CLI Demo\n* 搭建API的方法，我们提供的示例为OpenAI风格的API\n* 更多关于Qwen在工具调用、Code Interpreter、Agent方面的内容\n* 长序列理解能力及评测\n* 使用协议\n* ...\n\n如果遇到问题，请优先考虑查询[FAQ](FAQ.md)。如仍未解决，随时提出issue（但建议使用英语或提供翻译，有助于帮助更多用户）。如果想帮助我们提升，欢迎提交Pull Requests！\n\n想和我们一起讨论和聊天的话，赶紧加入我们的微信群和Discord server（入口见文档开头部分）！\n<br><br>\n\n## 新闻\n\n* 2023.11.30 🔥 我们推出 **Qwen-72B** 和 **Qwen-72B-Chat**，它们在 3T tokens上进行训练，并支持 32k 上下文。同时也发布了 **Qwen-1.8B** 和 **Qwen-1.8B-Chat**。我们还增强了 Qwen-72B-Chat 和 Qwen-1.8B-Chat 的系统指令（System Prompt）功能，请参阅[示例文档](examples/system_prompt.md)。此外，我们还对**昇腾910**以及**海光DCU**实现了推理的支持，详情请查看`ascend-support`及`dcu-support`文件夹。\n* 2023年10月17日 我们推出了Int8量化模型**Qwen-7B-Chat-Int8**和**Qwen-14B-Chat-Int8**。\n* 2023年9月25日 在魔搭社区（ModelScope）和Hugging Face推出**Qwen-14B**和**Qwen-14B-Chat**模型，并开源 [qwen.cpp](https://github.com/QwenLM/qwen.cpp) 和 [Qwen-Agent](https://github.com/QwenLM/Qwen-Agent)。**Qwen-7B**和**Qwen-7B-Chat**的代码和模型也同步得到更新。**请使用最新的代码和模型！**\n - 相比原版Qwen-7B，新版用了更多训练数据（从2.2T增加到2.4T tokens），序列长度从2048扩展至8192。整体中文能力以及代码能力均有所提升。\n* 2023年9月12日 支持Qwen-7B和Qwen-7B-Chat的微调，其中包括全参数微调、LoRA以及Q-LoRA。\n* 2023年8月21日 发布Qwen-7B-Chat的Int4量化模型，Qwen-7B-Chat-Int4。该模型显存占用低，推理速度相比半精度模型显著提升，在基准评测上效果损失较小。\n* 2023年8月3日 在魔搭社区（ModelScope）和Hugging Face同步推出Qwen-7B和Qwen-7B-Chat模型。同时，我们发布了技术备忘录，介绍了相关的训练细节和模型表现。\n<br>\n\n## 评测表现\n\nQwen系列模型相比同规模模型均实现了效果的显著提升。我们评测的数据集包括MMLU、C-Eval、 GSM8K、 MATH、HumanEval、MBPP、BBH等数据集，考察的能力包括自然语言理解、知识、数学计算和推理、代码生成、逻辑推理等。Qwen-72B在所有任务上均超越了LLaMA2-70B的性能，同时在10项任务中的7项任务中超越GPT-3.5.\n\n<p align=\"left\">\n <img src=\"assets/radar_72b.jpg\" width=\"600\"/>\n<p>\n<br>\n\n| Model | MMLU | C-Eval | GSM8K | MATH | HumanEval | MBPP | BBH | CMMLU |\n|:-------------------|:--------:|:--------:|:--------:|:--------:|:---------:|:--------:|:--------:|:--------:|\n| | 5-shot | 5-shot | 8-shot | 4-shot | 0-shot | 3-shot | 3-shot | 5-shot |\n| LLaMA2-7B | 46.8 | 32.5 | 16.7 | 3.3 | 12.8 | 20.8 | 38.2 | 31.8 |\n| LLaMA2-13B | 55.0 | 41.4 | 29.6 | 5.0 | 18.9 | 30.3 | 45.6 | 38.4 |\n| LLaMA2-34B | 62.6 | - | 42.2 | 6.2 | 22.6 | 33.0 | 44.1 | - |\n| ChatGLM2-6B | 47.9 | 51.7 | 32.4 | 6.5 | - | - | 33.7 | - |\n| InternLM-7B | 51.0 | 53.4 | 31.2 | 6.3 | 10.4 | 14.0 | 37.0 | 51.8 |\n| InternLM-20B | 62.1 | 58.8 | 52.6 | 7.9 | 25.6 | 35.6 | 52.5 | 59.0 |\n| Baichuan2-7B | 54.7 | 56.3 | 24.6 | 5.6 | 18.3 | 24.2 | 41.6 | 57.1 |\n| Baichuan2-13B | 59.5 | 59.0 | 52.8 | 10.1 | 17.1 | 30.2 | 49.0 | 62.0 |\n| Yi-34B \t \t | 76.3 | 81.8 | 67.9 | 15.9 | 26.2 | 38.2 | 66.4 | 82.6 |\n| XVERSE-65B \t | 70.8 | 68.6 | 60.3 | - | 26.3 | - | - | - |\n| **Qwen-1.8B** | 45.3 | 56.1 | 32.3 | 2.3 | 15.2 | 14.2 | 22.3 | 52.1 |\n| **Qwen-7B** | 58.2 | 63.5 | 51.7 | 11.6 | 29.9 | 31.6 | 45.0 | 62.2 |\n| **Qwen-14B** | 66.3 | 72.1 | 61.3 | 24.8 | 32.3 | 40.8 | 53.4 | 71.0 |\n| **Qwen-72B** | **77.4** | **83.3** | **78.9** | **35.2** | **35.4** | **52.2** | **67.7** | **83.6** |\n\n\n对于以上所有对比模型，我们列出了其官方汇报结果与[OpenCompass](https://opencompass.org.cn/leaderboard-llm)结果之间的最佳分数。\n\n更多的实验结果和细节请查看我们的技术备忘录。点击[这里](https://qianwen-res.oss-cn-beijing.aliyuncs.com/QWEN_TECHNICAL_REPORT.pdf)。\n<br><br>\n\n## 要求\n\n* python 3.8及以上版本\n* pytorch 1.12及以上版本，推荐2.0及以上版本\n* transformers 4.32及以上版本\n* 建议使用CUDA 11.4及以上（GPU用户、flash-attention用户等需考虑此选项）\n<br>\n\n## 快速使用\n\n我们提供简单的示例来说明如何利用🤖 ModelScope和🤗 Transformers快速使用Qwen-7B和Qwen-7B-Chat。\n\n你可以使用我们预构建好的Docker镜像，省去大部分配置环境的操作，详情见[“使用预构建的docker镜像”](#-使用预构建的docker镜像)一节。\n\n如不使用Docker，请确保你已经配置好环境并安装好相关的代码包。最重要的是，确保你满足上述要求，然后安装相关的依赖库。\n\n```bash\npip install -r requirements.txt\n```\n\n如果你的显卡支持fp16或bf16精度，我们还推荐安装[flash-attention](https://github.com/Dao-AILab/flash-attention)（**当前已支持flash attention 2**）来提高你的运行效率以及降低显存占用。(**flash-attention只是可选项，不安装也可正常运行该项目**)\n\n```bash\ngit clone https://github.com/Dao-AILab/flash-attention\ncd flash-attention && pip install .\n# 下方安装可选，安装可能比较缓慢。\n# pip install csrc/layer_norm\n# 如果flash-attn版本高于2.1.1，下方无需安装。\n# pip install csrc/rotary\n```\n\n接下来你可以开始使用Transformers或者ModelScope来使用我们的模型。\n\n### 🤗 Transformers\n\n如希望使用Qwen-chat进行推理，所需要写的只是如下所示的数行代码。**请确保你使用的是最新代码，并指定正确的模型名称和路径，如`Qwen/Qwen-7B-Chat`和`Qwen/Qwen-14B-Chat`**\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\nfrom transformers.generation import GenerationConfig\n\n# 可选的模型包括: \"Qwen/Qwen-7B-Chat\", \"Qwen/Qwen-14B-Chat\"\ntokenizer = AutoTokenizer.from_pretrained(\"Qwen/Qwen-7B-Chat\", trust_remote_code=True)\n\n# 打开bf16精度，A100、H100、RTX3060、RTX3070等显卡建议启用以节省显存\n# model = AutoModelForCausalLM.from_pretrained(\"Qwen/Qwen-7B-Chat\", device_map=\"auto\", trust_remote_code=True, bf16=True).eval()\n# 打开fp16精度，V100、P100、T4等显卡建议启用以节省显存\n# model = AutoModelForCausalLM.from_pretrained(\"Qwen/Qwen-7B-Chat\", device_map=\"auto\", trust_remote_code=True, fp16=True).eval()\n# 使用CPU进行推理，需要约32GB内存\n# model = AutoModelForCausalLM.from_pretrained(\"Qwen/Qwen-7B-Chat\", device_map=\"cpu\", trust_remote_code=True).eval()\n# 默认使用自动模式，根据设备自动选择精度\nmodel = AutoModelForCausalLM.from_pretrained(\"Qwen/Qwen-7B-Chat\", device_map=\"auto\", trust_remote_code=True).eval()\n\n# 可指定不同的生成长度、top_p等相关超参\nmodel.generation_config = GenerationConfig.from_pretrained(\"Qwen/Qwen-7B-Chat\", trust_remote_code=True)\n\n# 第一轮对话\nresponse, history = model.chat(tokenizer, \"你好\", history=None)\nprint(response)\n# 你好！很高兴为你提供帮助。\n\n# 第二轮对话\nresponse, history = model.chat(tokenizer, \"给我讲一个年轻人奋斗创业最终取得成功的故事。\", history=history)\nprint(response)\n# 这是一个关于一个年轻人奋斗创业最终取得成功的故事。\n# 故事的主人公叫李明，他来自一个普通的家庭，父母都是普通的工人。从小，李明就立下了一个目标：要成为一名成功的企业家。\n# 为了实现这个目标，李明勤奋学习，考上了大学。在大学期间，他积极参加各种创业比赛，获得了不少奖项。他还利用课余时间去实习，积累了宝贵的经验。\n# 毕业后，李明决定开始自己的创业之路。他开始寻找投资机会，但多次都被拒绝了。然而，他并没有放弃。他继续努力，不断改进自己的创业计划，并寻找新的投资机会。\n# 最终，李明成功地获得了一笔投资，开始了自己的创业之路。他成立了一家科技公司，专注于开发新型软件。在他的领导下，公司迅速发展起来，成为了一家成功的科技企业。\n# 李明的成功并不是偶然的。他勤奋、坚韧、勇于冒险，不断学习和改进自己。他的成功也证明了，只要努力奋斗，任何人都有可能取得成功。\n\n# 第三轮对话\nresponse, history = model.chat(tokenizer, \"给这个故事起一个标题\", history=history)\nprint(response)\n# 《奋斗创业：一个年轻人的成功之路》\n```\n\n运行Qwen同样非常简单。\n\n<details>\n <summary>运行Qwen</summary>\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\nfrom transformers.generation import GenerationConfig\n\n# 可选的模型包括: \"Qwen/Qwen-7B\", \"Qwen/Qwen-14B\"\ntokenizer = AutoTokenizer.from_pretrained(\"Qwen/Qwen-7B\", trust_remote_code=True)\n\n# 打开bf16精度，A100、H100、RTX3060、RTX3070等显卡建议启用以节省显存\n# model = AutoModelForCausalLM.from_pretrained(\"Qwen/Qwen-7B\", device_map=\"auto\", trust_remote_code=True, bf16=True).eval()\n# 打开fp16精度，V100、P100、T4等显卡建议启用以节省显存\n# model = AutoModelForCausalLM.from_pretrained(\"Qwen/Qwen-7B\", device_map=\"auto\", trust_remote_code=True, fp16=True).eval()\n# 使用CPU进行推理，需要约32GB内存\n# model = AutoModelForCausalLM.from_pretrained(\"Qwen/Qwen-7B\", device_map=\"cpu\", trust_remote_code=True).eval()\n# 默认使用自动模式，根据设备自动选择精度\nmodel = AutoModelForCausalLM.from_pretrained(\"Qwen/Qwen-7B\", device_map=\"auto\", trust_remote_code=True).eval()\n\n# 可指定不同的生成长度、top_p等相关超参\nmodel.generation_config = GenerationConfig.from_pretrained(\"Qwen/Qwen-7B\", trust_remote_code=True)\n\ninputs = tokenizer('蒙古国的首都是乌兰巴托（Ulaanbaatar）\\n冰岛的首都是雷克雅未克（Reykjavik）\\n埃塞俄比亚的首都是', return_tensors='pt')\ninputs = inputs.to(model.device)\npred = model.generate(**inputs)\nprint(tokenizer.decode(pred.cpu()[0], skip_special_tokens=True))\n# 蒙古国的首都是乌兰巴托（Ulaanbaatar）\\n冰岛的首都是雷克雅未克（Reykjavik）\\n埃塞俄比亚的首都是亚的斯亚贝巴（Addis Ababa）...\n```\n\n</details>\n\n<p id=\"DownloadModel\">\n若在使用上述代码时由于各种原因无法从 HuggingFace 拉取模型和代码，可以先从 ModelScope 下载模型及代码至本地，再从本地加载模型：\n</p>\n\n```python\nfrom modelscope import snapshot_download\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\n# Downloading model checkpoint to a local dir model_dir\n# model_dir = snapshot_download('qwen/Qwen-7B')\n# model_dir = snapshot_download('qwen/Qwen-7B-Chat')\n# model_dir = snapshot_download('qwen/Qwen-14B')\nmodel_dir = snapshot_download('qwen/Qwen-14B-Chat')\n\n# Loading local checkpoints\n# trust_remote_code is still set as True since we still load codes from local dir instead of transformers\ntokenizer = AutoTokenizer.from_pretrained(model_dir, trust_remote_code=True)\nmodel = AutoModelForCausalLM.from_pretrained(\n model_dir,\n device_map=\"auto\",\n trust_remote_code=True\n).eval()\n```\n\n### 🤖 ModelScope\n\n魔搭（ModelScope）是开源的模型即服务共享平台，为泛AI开发者提供灵活、易用、低成本的一站式模型服务产品。使用ModelScope同样非常简单，代码如下所示：\n\n```python\nfrom modelscope import AutoModelForCausalLM, AutoTokenizer\nfrom modelscope import GenerationConfig\n\n# 可选的模型包括: \"qwen/Qwen-7B-Chat\", \"qwen/Qwen-14B-Chat\"\ntokenizer = AutoTokenizer.from_pretrained(\"qwen/Qwen-7B-Chat\", trust_remote_code=True)\nmodel = AutoModelForCausalLM.from_pretrained(\"qwen/Qwen-7B-Chat\", device_map=\"auto\", trust_remote_code=True, fp16=True).eval()\nmodel.generation_config = GenerationConfig.from_pretrained(\"Qwen/Qwen-7B-Chat\", trust_remote_code=True) # 可指定不同的生成长度、top_p等相关超参\n\nresponse, history = model.chat(tokenizer, \"你好\", history=None)\nprint(response)\nresponse, history = model.chat(tokenizer, \"浙江的省会在哪里？\", history=history) \nprint(response)\nresponse, history = model.chat(tokenizer, \"它有什么好玩的景点\", history=history)\nprint(response)\n```\n\n### Batch推理\n千问支持batch批量推理。在开启flash-attention的状态下，使用batch推理可以约40%的提速。示例代码如下所示：\n```python\nimport torch\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\nfrom transformers import GenerationConfig\nfrom qwen_generation_utils import make_context, decode_tokens, get_stop_words_ids\n\ntokenizer = AutoTokenizer.from_pretrained(\n './',\n pad_token='<|extra_0|>',\n eos_token='<|endoftext|>',\n padding_side='left',\n trust_remote_code=True\n)\nmodel = AutoModelForCausalLM.from_pretrained(\n './',\n pad_token_id=tokenizer.pad_token_id,\n device_map=\"auto\",\n trust_remote_code=True\n).eval()\nmodel.generation_config = GenerationConfig.from_pretrained('./', pad_token_id=tokenizer.pad_token_id)\n\nall_raw_text = [\"我想听你说爱我。\", \"今天我想吃点啥，甜甜的，推荐下\", \"我马上迟到了，怎么做才能不迟到\"]\nbatch_raw_text = []\nfor q in all_raw_text:\n raw_text, _ = make_context(\n tokenizer,\n q,\n system=\"You are a helpful assistant.\",\n max_window_size=model.generation_config.max_window_size,\n chat_format=model.generation_config.chat_format,\n )\n batch_raw_text.append(raw_text)\n\nbatch_input_ids = tokenizer(batch_raw_text, padding='longest')\nbatch_input_ids = torch.LongTensor(batch_input_ids['input_ids']).to(model.device)\nbatch_out_ids = model.generate(\n batch_input_ids,\n return_dict_in_generate=False,\n generation_config=model.generation_config\n)\npadding_lens = [batch_input_ids[i].eq(tokenizer.pad_token_id).sum().item() for i in range(batch_input_ids.size(0))]\n\nbatch_response = [\n decode_tokens(\n batch_out_ids[i][padding_lens[i]:],\n tokenizer,\n raw_text_len=len(batch_raw_text[i]),\n context_length=(batch_input_ids[i].size(0)-padding_lens[i]),\n chat_format=\"chatml\",\n verbose=False,\n errors='replace'\n ) for i in range(len(all_raw_text))\n]\nprint(batch_response)\n\nresponse, _ = model.chat(tokenizer, \"我想听你说爱我。\", history=None)\nprint(response)\n\nresponse, _ = model.chat(tokenizer, \"今天我想吃点啥，甜甜的，推荐下\", history=None)\nprint(response)\n\nresponse, _ = model.chat(tokenizer, \"我马上迟到了，怎么做才能不迟到\", history=None)\nprint(response)\n```\n\n### CPU\n\n我们推荐你使用 [qwen.cpp](https://github.com/QwenLM/qwen.cpp) 来实现CPU部署和推理。qwen.cpp是Qwen和tiktoken的C++实现。你可以点击链接进入repo了解详情。\n\n当然，直接在CPU上运行模型也是可以的，示例如下：\n\n```python\nmodel = AutoModelForCausalLM.from_pretrained(\"Qwen/Qwen-7B-Chat\", device_map=\"cpu\", trust_remote_code=True).eval()\n```\n\n但是，这样的推理效率大概率会非常低。\n\n### 多GPU\n\n如果你遇到显存不足的问题而希望使用多张GPU进行推理，可以使用上述的默认的使用方法读取模型。此前提供的脚本`utils.py`已停止维护。\n\n尽管这个方法很简单，但它的效率相对较低。我们建议使用vLLM和FastChat并请阅读部署章节。\n\n### x86 平台\n在 酷睿™/至强® 可扩展处理器或 Arc™ GPU 上部署量化模型时，建议使用 [OpenVINO™ Toolkit](https://docs.openvino.ai/2023.3/gen_ai_guide.html) 以充分利用硬件，实现更好的推理性能。您可以安装并运行此[example notebook](https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/254-llm-chatbot)。相关问题，您可在 [OpenVINO repo](https://github.com/openvinotoolkit/openvino_notebooks/issues)中提交。\n\n\n### 阿里云灵积（DashScope）API服务\n最简单的使用Qwen模型API服务的方法就是通过DashScope（阿里云灵积API模型服务）。我们提供了简单介绍说明使用方法。同时，我们还提供了自己部署OpenAI格式的API的方法。\n\nDashScope是阿里云提供的大语言模型的API服务，目前支持Qwen。但请注意，目前提供服务的Qwen模型为内部模型，暂无更多具体细节对外透露。模型服务包括`qwen-turbo`、`qwen-plus`和`qwen-max`，`qwen-turbo`速度更快，`qwen-plus`效果更优，`qwen-max`是最新发布的千亿级通义千问2.0模型。详情请查看[文档](https://dashscope.aliyun.com)。\n\n请首先前往[官网](https://help.aliyun.com/zh/dashscope/developer-reference/activate-dashscope-and-create-an-api-key?spm=a2c4g.11186623.0.0.6c2774fahtfXdn)开通DashScope，获得API Key（AK）。建议通过环境变量设置AK：\n```bash\nexport DASHSCOPE_API_KEY=\"YOUR_DASHSCOPE_API_KEY\"\n```\n随后安装相关代码包，点击[此处](https://help.aliyun.com/zh/dashscope/developer-reference/install-dashscope-sdk)查看安装文档。如使用python，则直接通过pip安装：\n```bash\npip install dashscope\n```\n如安装JAVA SDK，则通过如下命令安装：\n```xml\n<!-- https://mvnrepository.com/artifact/com.alibaba/dashscope-sdk-java -->\n<dependency>\n <groupId>com.alibaba</groupId>\n <artifactId>dashscope-sdk-java</artifactId>\n <version>the-latest-version</version>\n</dependency>\n```\n最简单的使用方法就是通过messages调用，用法类似OpenAI API。示例如下：\n```python\nimport random\nfrom http import HTTPStatus\nfrom dashscope import Generation\n\n\ndef call_with_messages():\n messages = [{'role': 'system', 'content': 'You are a helpful assistant.'},\n {'role': 'user', 'content': '如何做西红柿鸡蛋？'}]\n gen = Generation()\n response = gen.call(\n Generation.Models.qwen_turbo,\n messages=messages,\n seed=random.randint(1, 10000), # set the random seed, optional, default to 1234 if not set\n result_format='message', # set the result to be \"message\" format.\n )\n return response\n\n\nif __name__ == '__main__':\n response = call_with_messages()\n if response.status_code == HTTPStatus.OK:\n print(response)\n else:\n print('Request id: %s, Status code: %s, error code: %s, error message: %s' % (\n response.request_id, response.status_code,\n response.code, response.message\n ))\n```\n更多用法请查看官方文档了解详情。\n<br><br>\n\n\n## 量化\n\n### GPTQ\n\n我们提供了基于[AutoGPTQ](https://github.com/PanQiWei/AutoGPTQ)的量化方案，并开源了Int4和Int8量化模型。量化模型的效果损失很小，但能显著降低显存占用并提升推理速度。\n\n以下我们提供示例说明如何使用Int4量化模型。在开始使用前，请先保证满足要求（如torch 2.0及以上，transformers版本为4.32.0及以上，等等），并安装所需安装包：\n\n```bash\npip install auto-gptq optimum\n```\n\n如安装`auto-gptq`遇到问题，我们建议您到官方[repo](https://github.com/PanQiWei/AutoGPTQ)搜索合适的wheel。\n\n> 注意：预编译的`auto-gptq`版本对`torch`版本及其CUDA版本要求严格。同时，由于\n> 其近期更新，你可能会遇到`transformers`、`optimum`或`peft`抛出的版本错误。\n> 我们建议使用符合以下要求的最新版本：\n> - torch==2.1 auto-gptq>=0.5.1 transformers>=4.35.0 optimum>=1.14.0 peft>=0.6.1\n> - torch>=2.0,<2.1 auto-gptq<0.5.0 transformers<4.35.0 optimum<1.14.0 peft>=0.5.0,<0.6.0\n\n随后即可使用和上述一致的用法调用量化模型：\n\n```python\n# 可选模型包括：\"Qwen/Qwen-7B-Chat-Int4\", \"Qwen/Qwen-14B-Chat-Int4\"\nmodel = AutoModelForCausalLM.from_pretrained(\n \"Qwen/Qwen-7B-Chat-Int4\",\n device_map=\"auto\",\n trust_remote_code=True\n).eval()\nresponse, history = model.chat(tokenizer, \"Hi\", history=None)\n```\n\n我们对BF16，Int8和Int4模型在基准评测上做了测试，发现量化模型效果损失较小，结果如下所示：\n\n| Quantization | MMLU | CEval (val) | GSM8K | Humaneval |\n|----------------------|:----:|:-----------:|:-----:|:---------:|\n| Qwen-1.8B-Chat (BF16)| 43.3 | 55.6 | 33.7 | 26.2 |\n| Qwen-1.8B-Chat (Int8)| 43.1 | 55.8 | 33.0 | 27.4 |\n| Qwen-1.8B-Chat (Int4)| 42.9 | 52.8 | 31.2 | 25.0 |\n| Qwen-7B-Chat (BF16) | 55.8 | 59.7 | 50.3 | 37.2 |\n| Qwen-7B-Chat (Int8) | 55.4 | 59.4 | 48.3 | 34.8 |\n| Qwen-7B-Chat (Int4) | 55.1 | 59.2 | 49.7 | 29.9 |\n| Qwen-14B-Chat (BF16) | 64.6 | 69.8 | 60.1 | 43.9 |\n| Qwen-14B-Chat (Int8) | 63.6 | 68.6 | 60.0 | 48.2 |\n| Qwen-14B-Chat (Int4) | 63.3 | 69.0 | 59.8 | 45.7 |\n| Qwen-72B-Chat (BF16) | 74.4 | 80.1 | 76.4 | 64.6 |\n| Qwen-72B-Chat (Int8) | 73.5 | 80.1 | 73.5 | 62.2 |\n| Qwen-72B-Chat (Int4) | 73.4 | 80.1 | 75.3 | 61.6 |\n<br>\n\n\n### KV cache量化\n\n> 注意：由于Hugging Face的内部实现，本功能的支持文件`cache_autogptq_cuda_256.cpp`与`cache_autogptq_cuda_kernel_256.cu`可能没被下载。如需开启使用，请手动从相关位置下载，并放置到相应文件中。\n\n在模型推理时，我们可以将中间结果key以及value的值量化后压缩存储，这样便可以在相同的卡上存储更多的key以及value，增加样本吞吐。\n\n我们在`config.json`里提供了`use_cache_quantization`和`use_cache_kernel`两个参数来控制是否启用KV cache量化，具体使用方法如下：\n```python\nmodel = AutoModelForCausalLM.from_pretrained(\n \"Qwen/Qwen-7B-Chat\",\n device_map=\"auto\",\n trust_remote_code=True,\n use_cache_quantization=True,\n use_cache_kernel=True,\n use_flash_attn=False\n)\n```\n注意：当前该功能不支持与flash attention同时开启，如果你开了KV cache量化的同时又开了flash attention（`use_flash_attn=True`， `use_cache_quantization=True`, `use_cache_kernel=True`），程序默认将关闭`use_flash_attn`。\n\n效果方面，我们验证过Int8 KV Cache的使用对模型整体的精度指标基本无损。我们做了针对显存占用的性能测试。评测运行于单张A100-SXM4-80G GPU，模型默认使用BF16格式，默认生成1024个token，其中OOM表示内存不足。\n\n开启了KV cache量化之后，模型在推理的时候可以开启更大的batch size (bs)。\n\n| USE KV Cache | bs=1 | bs=4 | bs=16 | bs=32 | bs=64 | bs=100 |\n|--------------|:------:|:------:|:------:|:------:|:------:|:------:|\n| No | 16.3GB | 24.1GB | 31.7GB | 48.7GB | oom | oom |\n| Yes | 15.5GB | 17.2GB | 22.3GB | 30.2GB | 48.2GB | 72.4GB |\n\n\n开启了KV cache量化之后，模型在推理时可在生成更长的序列（sl，生成的token数）时，节约更多的显存。\n\n| USE KV Cache | sl=512 | sl=1024 | sl=2048 | sl=4096 | sl=8192 |\n|--------------|:------:|:-------:|:-------:|:-------:|:-------:|\n| no | 15.2GB | 16.3GB | 17.6GB | 19.5GB | 23.2GB |\n| yes | 15GB | 15.5GB | 15.8GB | 16.6GB | 17.6GB |\n\n\n开启KV cache量化后，模型在推理时会将原始存进`layer-past`的float格式的key/value转换成int8格式，同时存储量化部分的参数。\n\n具体操作如下：\n\n1. 将key/value进行量化操作\n```\n qv,scale,zero_point=quantize_cache_v(v)\n```\n2. 存入`layer_past`中:\n\n量化格式的`layer-past`:\n```\n layer_past=((q_key,key_scale,key_zero_point),\n (q_value,value_scale,value_zero_point))\n```\n原始格式的`layer-past`:\n```\n layer_past=(key,value)\n```\n如果需要将`layer-past`中存好的key，value直接取出使用，可以使用反量化操作将Int8格式的key/value转回float格式：\n```\n v=dequantize_cache_torch(qv,scale,zero_point)\n```\n<br>\n\n### 推理性能\n这一部分将介绍模型推理的速度和显存占用的相关数据。下文的性能测算使用 [此脚本](https://qianwen-res.oss-cn-beijing.aliyuncs.com/profile.py) 完成。\n\n我们测算了BF16、Int8和Int4模型在生成2048个token时的平均推理速度（tokens/s）和显存使用。结果如下所示：\n\n<table>\n <tr>\n <td>Model Size</td>\n <td>Quantization</td>\n <td>Speed (Tokens/s)</td>\n <td>GPU Memory Usage</td>\n </tr>\n <tr>\n <td rowspan=\"3\">1.8B</td>\n <td>BF16</td>\n <td>54.09</td>\n <td>4.23GB</td>\n </tr>\n <tr>\n <td>Int8</td>\n <td>55.56</td>\n <td>3.48GB</td>\n </tr>\n <tr>\n <td>Int4</td>\n <td>71.07</td>\n <td>2.91GB</td>\n </tr>\n <tr>\n <td rowspan=\"3\">7B</td>\n <td>BF16</td>\n <td>40.93</td>\n <td>16.99GB</td>\n </tr>\n <tr>\n <td>Int8</td>\n <td>37.47</td>\n <td>11.20GB</td>\n </tr>\n <tr>\n <td>Int4</td>\n <td>50.09</td>\n <td>8.21GB</td>\n </tr>\n <tr>\n <td rowspan=\"3\">14B</td>\n <td>BF16</td>\n <td>32.22</td>\n <td>30.15GB</td>\n </tr>\n <tr>\n <td>Int8</td>\n <td>29.28</td>\n <td>18.81GB</td>\n </tr>\n <tr>\n <td>Int4</td>\n <td>38.72</td>\n <td>13.01GB</td>\n </tr>\n <tr>\n <td rowspan=\"3\">72B</td>\n <td>BF16</td>\n <td>8.48</td>\n <td>144.69GB (2xA100)</td>\n </tr>\n <tr>\n <td>Int8</td>\n <td>9.05</td>\n <td>81.27GB (2xA100)</td>\n </tr>\n <tr>\n <td>Int4</td>\n <td>11.32</td>\n <td>48.86GB</td>\n </tr>\n <tr>\n <td>72B + vLLM</td>\n <td>BF16</td>\n <td>17.60</td>\n <td>2xA100</td>\n </tr>\n</table>\n\n评测运行于单张A100-SXM4-80G GPU（除非提到使用2xA100），使用PyTorch 2.0.1、CUDA 11.8和Flash-Attention2。(72B + vLLM 使用 PyTorch 2.1.0和Cuda 11.8.)推理速度是生成2048个token的速度均值。\n\n注意：以上Int4/Int8模型生成速度使用autogptq库给出，当前``AutoModelForCausalLM.from_pretrained``载入的模型生成速度会慢大约20%。我们已经将该问题汇报给HuggingFace团队，若有解决方案将即时更新。\n\n我们还测量了不同上下文长度、生成长度、Flash-Attention版本的推理速度和 GPU 内存使用情况。可以在 Hugging Face 或 ModelScope 上的相应的模型介绍页面找到结果。\n\n## 微调\n\n### 使用方法\n我们提供了`finetune.py`这个脚本供用户实现在自己的数据上进行微调的功能，以接入下游任务。此外，我们还提供了shell脚本减少用户的工作量。这个脚本支持 [DeepSpeed](https://github.com/microsoft/DeepSpeed) 和 [FSDP](https://engineering.fb.com/2021/07/15/open-source/fsdp/) 。我们提供的shell脚本使用了DeepSpeed，因此建议您确保已经安装DeepSpeed和Peft（注意：DeepSpeed可能不兼容最新的pydantic版本，请确保`pydantic<2.0`）。你可以使用如下命令安装：\n```bash\npip install \"peft<0.8.0\" deepspeed\n```\n\n首先，你需要准备你的训练数据。你需要将所有样本放到一个列表中并存入json文件中。每个样本对应一个字典，包含id和conversation，其中后者为一个列表。示例如下所示：\n```json\n[\n {\n \"id\": \"identity_0\",\n \"conversations\": [\n {\n \"from\": \"user\",\n \"value\": \"你好\"\n },\n {\n \"from\": \"assistant\",\n \"value\": \"我是一个语言模型，我叫通义千问。\"\n }\n ]\n }\n]\n```\n\n准备好数据后，你可以使用我们提供的shell脚本实现微调。注意，你需要在脚本中指定你的数据的路径。\n\n微调脚本能够帮你实现：\n- 全参数微调\n- LoRA\n- Q-LoRA\n\n全参数微调在训练过程中更新所有参数。你可以运行这个脚本开始训练：\n\n```bash\n# 分布式训练。由于显存限制将导致单卡训练失败，我们不提供单卡训练脚本。\nbash finetune/finetune_ds.sh\n```\n\n尤其注意，你需要在脚本中指定正确的模型名称或路径、数据路径、以及模型输出的文件夹路径。在这个脚本中我们使用了DeepSpeed ZeRO 3。如果你想修改这个配置，可以删除掉`--deepspeed`这个输入或者自行根据需求修改DeepSpeed配置json文件。此外，我们支持混合精度训练，因此你可以设置`--bf16 True`或者`--fp16 True`。在使用fp16时，请使用DeepSpeed支持混合精度训练。经验上，如果你的机器支持bf16，我们建议使用bf16，这样可以和我们的预训练和对齐训练保持一致，这也是为什么我们把默认配置设为它的原因。\n\n运行LoRA的方法类似全参数微调。但在开始前，请确保已经安装`peft`代码库。另外，记住要设置正确的模型、数据和输出路径。我们建议你为模型路径使用绝对路径。这是因为LoRA仅存储adapter部分参数，而adapter配置json文件记录了预训练模型的路径，用于读取预训练模型权重。同样，你可以设置bf16或者fp16。\n\n```bash\n# 单卡训练\nbash finetune/finetune_lora_single_gpu.sh\n# 分布式训练\nbash finetune/finetune_lora_ds.sh\n```\n\n与全参数微调不同，LoRA ([论文](https://arxiv.org/abs/2106.09685)) 只更新adapter层的参数而无需更新原有语言模型的参数。这种方法允许用户用更低的显存开销来训练模型，也意味着更小的计算开销。\n\n注意，如果你使用预训练模型进行LoRA微调，而非chat模型，模型的embedding和输出层的参数将被设为可训练的参数。这是因为预训练模型没有学习过ChatML格式中的特殊token，因此需要将这部分参数设为可训练才能让模型学会理解和预测这些token。这也意味着，假如你的训练引入新的特殊token，你需要通过代码中的`modules_to_save`将这些参数设为可训练的参数。此外，这部分训练参数的引入会影响ZeRO 3的使用，因此我们默认推荐使用ZeRO 2。当然，如果你不需要引入这部分训练参数，你可以通过替换DeepSpeed的配置文件来使用ZeRO 3。如果你想节省显存占用，可以考虑使用chat模型进行LoRA微调，显存占用将大幅度降低。下文的显存占用和训练速度的记录将详细介绍这部分细节。\n\n如果你依然遇到显存不足的问题，可以考虑使用Q-LoRA ([论文](https://arxiv.org/abs/2305.14314)) 。该方法使用4比特量化模型以及paged attention等技术实现更小的显存开销。\n\n注意：如你使用单卡Q-LoRA，你可能需要安装`mpi4py`。你可以通过`pip`或者`conda`来安装。\n\n运行Q-LoRA你只需运行如下脚本：\n\n```bash\n# 单卡训练\nbash finetune/finetune_qlora_single_gpu.sh\n# 分布式训练\nbash finetune/finetune_qlora_ds.sh\n```\n\n我们建议你使用我们提供的Int4量化模型进行训练，即Qwen-7B-Chat-Int4。请**不要使用**非量化模型！与全参数微调以及LoRA不同，Q-LoRA仅支持fp16。注意，由于我们发现torch amp支持的fp16混合精度训练存在问题，因此当前的单卡训练Q-LoRA必须使用DeepSpeed。此外，上述LoRA关于特殊token的问题在Q-LoRA依然存在。并且，Int4模型的参数无法被设为可训练的参数。所幸的是，我们只提供了Chat模型的Int4模型，因此你不用担心这个问题。但是，如果你执意要在Q-LoRA中引入新的特殊token，很抱歉，我们无法保证你能成功训练。\n\n> 注意：由于Hugging Face的内部实现，模型在保存时，一些非Python文件未保存（例如`*.cpp`与`*.cu`），如需要支持相关功能，请手动复制有关文件。\n\n与全参数微调不同，LoRA和Q-LoRA的训练只需存储adapter部分的参数。假如你需要使用LoRA训练后的模型，你需要使用如下方法。假设你使用Qwen-7B训练模型，你可以用如下代码读取模型：\n\n```python\nfrom peft import AutoPeftModelForCausalLM\n\nmodel = AutoPeftModelForCausalLM.from_pretrained(\n path_to_adapter, # path to the output directory\n device_map=\"auto\",\n trust_remote_code=True\n).eval()\n```\n\n> 注意: 如果`peft>=0.8.0`，加载模型同时会尝试加载tokenizer，但peft内部未相应设置`trust_remote_code=True`，导致`ValueError: Tokenizer class QWenTokenizer does not exist or is not currently imported.`要避过这一问题，你可以降级`peft<0.8.0`或将tokenizer相关文件移到其它文件夹。\n\n\n如果你觉得这样一步到位的方式让你很不安心或者影响你接入下游应用，你可以选择先合并并存储模型（LoRA支持合并，Q-LoRA不支持），再用常规方式读取你的新模型，示例如下：\n\n```python\nfrom peft import AutoPeftModelForCausalLM\n\nmodel = AutoPeftModelForCausalLM.from_pretrained(\n path_to_adapter, # path to the output directory\n device_map=\"auto\",\n trust_remote_code=True\n).eval()\n\nmerged_model = model.merge_and_unload()\n# max_shard_size and safe serialization are not necessary. \n# They respectively work for sharding checkpoint and save the model to safetensors\nmerged_model.save_pretrained(new_model_directory, max_shard_size=\"2048MB\", safe_serialization=True)\n```\n\n`new_model_directory`目录将包含合并后的模型参数与相关模型代码。请注意`*.cu`和`*.cpp`文件可能没被保存，请手动复制。另外，`merge_and_unload`仅保存模型，并未保存tokenizer，如有需要，请复制相关文件或使用以以下代码保存\n```python\nfrom transformers import AutoTokenizer\ntokenizer = AutoTokenizer.from_pretrained(\n path_to_adapter, # path to the output directory\n trust_remote_code=True\n)\ntokenizer.save_pretrained(new_model_directory)\n```\n\n\n注意：分布式训练需要根据你的需求和机器指定正确的分布式训练超参数。此外，你需要根据你的数据、显存情况和训练速度预期，使用`--model_max_length`设定你的数据长度。\n\n### 量化微调后模型\n\n这一小节用于量化全参/LoRA微调后的模型。（注意：你不需要量化Q-LoRA模型因为它本身就是量化过的。）\n如果你需要量化LoRA微调后的模型，请先根据上方说明去合并你的模型权重。\n\n我们推荐使用[auto_gptq](https://github.com/PanQiWei/AutoGPTQ)去量化你的模型。\n\n```bash\npip install auto-gptq optimum\n```\n\n注意: 当前AutoGPTQ有个bug，可以在该[issue](https://github.com/PanQiWei/AutoGPTQ/issues/370)查看。这里有个[修改PR](https://github.com/PanQiWei/AutoGPTQ/pull/495)，你可以使用该分支从代码进行安装。\n\n首先，准备校准集。你可以重用微调你的数据，或者按照微调相同的方式准备其他数据。\n\n第二步，运行以下命令：\n\n```bash\npython run_gptq.py \\\n --model_name_or_path $YOUR_LORA_MODEL_PATH \\\n --data_path $DATA \\\n --out_path $OUTPUT_PATH \\\n --bits 4 # 4 for int4; 8 for int8\n```\n\n这一步需要使用GPU，根据你的校准集大小和模型大小，可能会消耗数个小时。\n\n接下来, 将原模型中所有 `*.py`, `*.cu`, `*.cpp` 文件和 `generation_config.json` 文件复制到输出模型目录下。同时，使用官方对应版本的量化模型的 `config.json` 文件覆盖输出模型目录下的文件\n(例如, 如果你微调了 `Qwen-7B-Chat`和`--bits 4`, 那么你可以从 [Qwen-7B-Chat-Int4](https://huggingface.co/Qwen/Qwen-7B-Chat-Int4/blob/main/config.json) 仓库中找到对应的`config.json` )。\n并且，你需要将 ``gptq.safetensors`` 重命名为 ``model.safetensors``。\n\n最后，像官方量化模型一样测试你的模型。例如：\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\nfrom transformers.generation import GenerationConfig\n\ntokenizer = AutoTokenizer.from_pretrained(\"/path/to/your/model\", trust_remote_code=True)\n\nmodel = AutoModelForCausalLM.from_pretrained(\n \"/path/to/your/model\",\n device_map=\"auto\",\n trust_remote_code=True\n).eval()\n\nresponse, history = model.chat(tokenizer, \"你好\", history=None)\nprint(response)\n```\n\n### 多机微调\n\n我们提供的脚本支持多机微调，可以参考[脚本](./finetune/finetune_lora_ds.sh)中的注释，在每个节点上正确设置相应的参数并启动训练脚本。关于多机分布式训练的更多信息，请参考[torchrun](https://pytorch.org/docs/stable/elastic/run.html)。\n\n注意： DeepSpeed ZeRO 3 对节点间通信速率的要求远大于 ZeRO 2，在多机微调的情况下会大幅降低训练速度。因此，我们不建议在多机微调的情况下使用 DeepSpeed ZeRO 3 配置。\n\n### 显存占用及训练速度\n\n下面记录7B和14B模型在单GPU使用LoRA（LoRA (emb)指的是embedding和输出层参与训练，而LoRA则不优化这部分参数）和QLoRA时处理不同长度输入的显存占用和训练速度的情况。本次评测运行于单张A100-SXM4-80G GPU，使用CUDA 11.8和Pytorch 2.0，并使用了flash attention 2。我们统一使用batch size为1，gradient accumulation为8的训练配置，记录输入长度分别为256、512、1024、2048、4096和8192的显存占用（GB）和训练速度（s/iter）。我们还使用2张A100测了Qwen-7B的全参数微调。受限于显存大小，我们仅测试了256、512和1024token的性能。\n\n对于 Qwen-7B，我们额外测试了多机微调的性能。我们在两台服务器上运行评测，每台服务器包含两张A100-SXM4-80G GPU，其余配置与Qwen-7B的其他评测相同。多机微调的结果在表中以 LoRA (multinode) 标示。\n\n对于 Qwen-72B，我们测试了两种方案：1）使用4个 A100-SXM4-80G GPUs，通过 Lora + DeepSpeed ZeRO 3 微调和2）使用单张A100-SXM4-80G GPU，通过 QLora (int4) 微调。请注意，使用 LoRA (emb) 微调和不带 DeepSpeed ZeRO 3 的 LoRA 微调在4个A100-SXM4-80G GPUs 上都会出现OOM（你可以通过将`--deepspeed finetune/ds_config_zero3.json`参数传给[`finetune/finetune_lora_ds.sh`](finetune/finetune_lora_ds.sh)来打开 DeepSpeed ZeRO 3 配置）。\n\n具体数值如下所示：\n\n\n<table>\n <tr>\n <th rowspan=\"2\">Model Size</th><th rowspan=\"2\">Method</th><th rowspan=\"2\">#Nodes</th><th rowspan=\"2\">#GPUs per node</th><th colspan=\"6\" align=\"center\">Sequence Length</th>\n </tr>\n <tr>\n <th align=\"center\">256</th><th align=\"center\">512</th><th align=\"center\">1024</th><th align=\"center\">2048</th><th align=\"center\">4096</th><th align=\"center\">8192</th>\n </tr>\n <tr>\n <th rowspan=\"4\">1.8B</th><td>LoRA</td>\n <td>1</td><td>1</td>\n <td align=\"center\">6.7G / 1.0s/it</td><td align=\"center\">7.4G / 1.0s/it</td><td align=\"center\">8.4G / 1.1s/it</td><td align=\"center\">11.0G / 1.7s/it</td><td align=\"center\">16.2G / 3.3s/it</td><td align=\"center\">21.8G / 6.8s/it</td>\n </tr>\n <tr>\n <td>LoRA (emb)</td>\n <td>1</td><td>1</td>\n <td align=\"center\">13.7G / 1.0s/it</td><td align=\"center\">14.0G / 1.0s/it</td><td align=\"center\">14.0G / 1.1s/it</td><td align=\"center\">15.1G / 1.8s/it</td><td align=\"center\">19.7G / 3.4s/it</td><td align=\"center\">27.7G / 7.0s/it</td>\n </tr>\n <tr>\n <td>Q-LoRA</td>\n <td>1</td><td>1</td>\n <td align=\"center\">5.8G / 1.4s/it</td><td align=\"center\">6.0G / 1.4s/it</td><td align=\"center\">6.6G / 1.4s/it</td><td align=\"center\">7.8G / 2.0s/it</td><td align=\"center\">10.2G / 3.4s/it</td><td align=\"center\">15.8G / 6.5s/it</td>\n </tr>\n <tr>\n <td>Full-parameter</td>\n <td>1</td><td>1</td>\n <td align=\"center\">43.5G / 2.1s/it</td><td align=\"center\">43.5G / 2.2s/it</td><td align=\"center\">43.5G / 2.2s/it</td><td align=\"center\">43.5G / 2.3s/it</td><td align=\"center\">47.1G / 2.8s/it</td><td align=\"center\">48.3G / 5.6s/it</td>\n </tr>\n <tr>\n <th rowspan=\"5\">7B</th>\n <td>LoRA</td>\n <td>1</td><td>1</td>\n <td align=\"center\">20.1G / 1.2s/it</td><td align=\"center\">20.4G / 1.5s/it</td><td align=\"center\">21.5G / 2.8s/it</td><td align=\"center\">23.8G / 5.2s/it</td><td align=\"center\">29.7G / 10.1s/it</td><td align=\"center\">36.6G / 21.3s/it</td>\n </tr>\n <tr>\n <td>LoRA (emb)</td>\n <td>1</td><td>1</td>\n <td align=\"center\">33.7G / 1.4s/it</td><td align=\"center\">34.1G / 1.6s/it</td><td align=\"center\">35.2G / 2.9s/it</td><td align=\"center\">35.1G / 5.3s/it</td><td align=\"center\">39.2G / 10.3s/it</td><td align=\"center\">48.5G / 21.7s/it</td>\n </tr>\n <tr>\n <td>Q-LoRA</td>\n <td>1</td><td>1</td>\n <td align=\"center\">11.5G / 3.0s/it</td><td align=\"center\">11.5G / 3.0s/it</td><td align=\"center\">12.3G / 3.5s/it</td><td align=\"center\">13.9G / 7.0s/it</td><td align=\"center\">16.9G / 11.6s/it</td><td align=\"center\">23.5G / 22.3s/it</td>\n </tr>\n <tr>\n <td>Full-parameter</td>\n<td>1</td><td>2</td>\n<td align=\"center\">139.2G / 4.0s/it</td><td align=\"center\">148.0G / 4.0s/it</td><td align=\"center\">162.0G / 4.5s/it</td><td align=\"center\">-</td><td align=\"center\">-</td><td align=\"center\">-</td>\n </tr>\n <tr>\n <td>LoRA (multinode)</td>\n <td>2</td><td>2</td>\n <td align=\"center\">74.7G / 2.09s/it</td><td align=\"center\">77.6G / 3.16s/it</td><td align=\"center\">84.9G / 5.17s/it</td><td align=\"center\">95.1G / 9.25s/it</td><td align=\"center\">121.1G / 18.1s/it</td><td align=\"center\">155.5G / 37.4s/it</td>\n </tr>\n <tr>\n <th rowspan=\"3\">14B</th>\n <td>LoRA</td>\n <td>1</td><td>1</td>\n <td align=\"center\">34.6G / 1.6s/it</td><td align=\"center\">35.1G / 2.4s/it</td><td align=\"center\">35.3G / 4.4s/it</td><td align=\"center\">37.4G / 8.4s/it</td><td align=\"center\">42.5G / 17.0s/it</td><td align=\"center\">55.2G / 36.0s/it</td>\n </tr>\n <tr>\n <td>LoRA (emb)</td>\n <td>1</td><td>1</td>\n <td align=\"center\">51.2 / 1.7s/it</td><td align=\"center\">51.1G / 2.6s/it</td><td align=\"center\">51.5G / 4.6s/it</td><td align=\"center\">54.1G / 8.6s/it</td><td align=\"center\">56.8G / 17.2s/it</td><td align=\"center\">67.7G / 36.3s/it</td>\n </tr>\n <tr>\n <td>Q-LoRA</td>\n <td>1</td><td>1</td>\n <td align=\"center\">18.7G / 5.3s/it</td><td align=\"center\">18.4G / 6.3s/it</td><td align=\"center\">18.9G / 8.2s/it</td><td align=\"center\">19.9G / 11.8s/it</td><td align=\"center\">23.0G / 20.1s/it</td><td align=\"center\">27.9G / 38.3s/it</td>\n </tr>\n <tr>\n <th rowspan=\"2\">72B</th>\n <td>LoRA + Deepspeed Zero3</td>\n <td>1</td><td>4</td>\n <td align=\"center\">215.4G / 17.6s/it</td><td align=\"center\">217.7G / 20.5s/it</td><td align=\"center\">222.6G / 29.4s/it</td><td align=\"center\">228.8G / 45.7s/it</td><td align=\"center\">249.0G / 83.4s/it</td><td align=\"center\">289.2G / 161.5s/it</td>\n </tr>\n <tr>\n <td>Q-LoRA</td>\n <td>1</td><td>1</td>\n <td align=\"center\">61.4G / 27.4s/it</td><td align=\"center\">61.4G / 31.5s/it</td><td align=\"center\">62.9G / 41.4s/it</td><td align=\"center\">64.1G / 59.5s/it</td><td align=\"center\">68.0G / 97.7s/it</td><td align=\"center\">75.6G / 179.8s/it</td>\n </tr>\n</table>\n\n<br>\n\n## 部署\n\n### vLLM\n如希望部署及加速推理，我们建议你使用vLLM。\n\n如果你使用**CUDA 12.1和PyTorch 2.1**，可以直接使用以下命令安装vLLM。\n\n```bash\npip install vllm\n```\n\n否则请参考vLLM官方的[安装说明](https://docs.vllm.ai/en/latest/getting_started/installation.html)。\n\n#### vLLM + 类Transformer接口\n\n请下载[接口封装代码](examples/vllm_wrapper.py)到当前文件夹，并执行以下命令进行多轮对话交互。（注意：该方法当前只支持``model.chat()``接口。）\n\n```python\nfrom vllm_wrapper import vLLMWrapper\n\nmodel = vLLMWrapper('Qwen/Qwen-7B-Chat', tensor_parallel_size=1)\n\nresponse, history = model.chat(query=\"你好\", history=None)\nprint(response)\nresponse, history = model.chat(query=\"给我讲一个年轻人奋斗创业最终取得成功的故事。\", history=history)\nprint(response)\nresponse, history = model.chat(query=\"给这个故事起一个标题\", history=history)\nprint(response)\n```\n\n#### vLLM + 网页Demo / 类OpenAI API\n\n你可以使用FastChat去搭建一个网页Demo或类OpenAI API服务器。首先，请安装FastChat：\n\n```bash\npip install \"fschat[model_worker,webui]\"\n```\n\n使用vLLM和FastChat运行Qwen之前，首先启动一个controller：\n```bash\npython -m fastchat.serve.controller\n```\n\n然后启动model worker读取模型。如使用单卡推理，运行如下命令：\n```bash\npython -m fastchat.serve.vllm_worker --model-path $model_path --trust-remote-code --dtype bfloat16\n# python -m fastchat.serve.vllm_worker --model-path $model_path --trust-remote-code --dtype float16 # 运行int4模型\n```\n然而，如果你希望使用多GPU加速推理或者增大显存，你可以使用vLLM支持的模型并行机制。假设你需要在4张GPU上运行你的模型，命令如下所示：\n```bash\npython -m fastchat.serve.vllm_worker --model-path $model_path --trust-remote-code --tensor-parallel-size 4 --dtype bfloat16\n# python -m fastchat.serve.vllm_worker --model-path $model_path --trust-remote-code --tensor-parallel-size 4 --dtype float16 # 运行int4模型\n```\n\n启动model worker后，你可以启动一个：\n\n* Web UI Demo\n```bash\npython -m fastchat.serve.gradio_web_server\n```\n\n* OpenAI API\n\n使用OpenAI API前，请阅读我们的API章节配置好环境，然后运行如下命令：\n```bash\npython -m fastchat.serve.openai_api_server --host localhost --port 8000\n```\n\n然而，如果你觉得使用vLLM和FastChat比较困难，你也可以尝试以下我们提供的最简单的方式部署Web Demo、CLI Demo和OpenAI API。\n<br>\n\n\n### Web UI\n\n我们提供了Web UI的demo供用户使用 (感谢 @wysaid 支持)。在开始前，确保已经安装如下代码库：\n\n```bash\npip install -r requirements_web_demo.txt\n```\n\n随后运行如下命令，并点击生成链接：\n\n```bash\npython web_demo.py\n```\n\n<p align=\"center\">\n <br>\n <img src=\"assets/web_demo.gif\" width=\"600\" />\n <br>\n<p>\n\n### 交互式Demo\n\n我们提供了一个简单的交互式Demo示例，请查看`cli_demo.py`。当前模型已经支持流式输出，用户可通过输入文字的方式和Qwen-7B-Chat交互，模型将流式输出返回结果。运行如下命令：\n\n```bash\npython cli_demo.py\n```\n\n<p align=\"center\">\n <br>\n <img src=\"assets/cli_demo.gif\" width=\"600\" />\n <br>\n<p>\n<br>\n\n### API\n\n我们提供了OpenAI API格式的本地API部署方法（感谢@hanpenggit）。在开始之前先安装必要的代码库：\n\n```bash\npip install fastapi uvicorn \"openai<1.0\" pydantic sse_starlette\n```\n\n随后即可运行以下命令部署你的本地API：\n\n```bash\npython openai_api.py\n```\n\n你也可以修改参数，比如`-c`来修改模型名称或路径, `--cpu-only`改为CPU部署等等。如果部署出现问题，更新上述代码库往往可以解决大多数问题。\n\n使用API同样非常简单，示例如下：\n\n```python\nimport openai\nopenai.api_base = \"http://localhost:8000/v1\"\nopenai.api_key = \"none\"\n\n# 使用流式回复的请求\nfor chunk in openai.ChatCompletion.create(\n model=\"Qwen\",\n messages=[\n {\"role\": \"user\", \"content\": \"你好\"}\n ],\n stream=True\n # 流式输出的自定义stopwords功能尚未支持，正在开发中\n):\n if hasattr(chunk.choices[0].delta, \"content\"):\n print(chunk.choices[0].delta.content, end=\"\", flush=True)\n\n# 不使用流式回复的请求\nresponse = openai.ChatCompletion.create(\n model=\"Qwen\",\n messages=[\n {\"role\": \"user\", \"content\": \"你好\"}\n ],\n stream=False,\n stop=[] # 在此处添加自定义的stop words 例如ReAct prompting时需要增加： stop=[\"Observation:\"]。\n)\nprint(response.choices[0].message.content)\n```\n\n<p align=\"center\">\n <br>\n <img src=\"assets/openai_api.gif\" width=\"600\" />\n <br>\n<p>\n\n该接口也支持函数调用（**Function Calling**），但暂时仅限 `stream=False` 时能生效。用法见[函数调用示例](examples/function_call_examples.py)。\n<br><br>\n\n## 🐳 使用预构建的Docker镜像\n\n为简化部署流程，我们提供了预配置好相应环境的Docker镜像：[qwenllm/qwen](https://hub.docker.com/r/qwenllm/qwen)，只需安装驱动、下载模型文件即可启动Demo、部署OpenAI API以及进行微调。\n\n### 准备操作\n\n1. 根据需要使用的镜像版本，安装相应版本的Nvidia驱动：\n - `qwenllm/qwen:cu117`（**推荐**）：`>= 515.48.07`\n - `qwenllm/qwen:cu114`（不支持flash-attention）：`>= 470.82.01`\n - `qwenllm/qwen:cu121`：`>= 530.30.02`\n - `qwenllm/qwen:latest`：与`qwenllm/qwen:cu117`相同\n\n2. 安装并配置[docker](https://docs.docker.com/engine/install/)和[nvidia-container-toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html)：\n\n```bash\n# 配置docker\nsudo systemctl start docker\n# 测试docker是否安装正确\nsudo docker run hello-world\n\n# 配置nvidia-container-toolkit\nsudo nvidia-ctk runtime configure --runtime=docker\nsudo systemctl restart docker\n# 测试nvidia-container-toolkit是否安装正确\nsudo docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi\n```\n\n3. 下载模型及代码至本地（参考[此处说明](#DownloadModel)）\n\n### 部署\n\n下面我们以Qwen-7B-Chat为例。在启动Web Demo或者部署API前，请先参照下方代码完成配置工作：\n\n```bash\nIMAGE_NAME=qwenllm/qwen:cu117\nPORT=8901\nCHECKPOINT_PATH=/path/to/Qwen-7B-Chat # 下载到本地的模型及代码路径\n```\n\n如下脚本可以帮你部署:\n\n* OpenAI API\n```bash\nbash docker/docker_openai_api.sh -i ${IMAGE_NAME} -c ${CHECKPOINT_PATH} --port ${PORT}\n```\n\n* Web UI\n```bash\nbash docker/docker_web_demo.sh -i ${IMAGE_NAME} -c ${CHECKPOINT_PATH} --port ${PORT}\n```\n\n* 交互式Demo\n```bash\nbash docker/docker_cli_demo.sh -i ${IMAGE_NAME} -c ${CHECKPOINT_PATH}\n```\n\n这些命令将自动下载所需镜像以及后台启动Web UI Demo。你可以打开`http://localhost:${PORT}` 来使用该Demo。\n\n如果输出如下内容，则说明Demo启动成功：\n\n```text\nSuccessfully started web demo. Open '...' to try!\nRun `docker logs ...` to check demo status.\nRun `docker rm -f ...` to stop and remove the demo.\n```\n\n如果你想查看Demo的状态，你可以使用这个命令来展示输出结果：`docker logs qwen`。\n\n你可以使用这个命令`docker rm -f qwen`来停止服务并删除容器。\n\n### 微调\n\n使用预配置好的Docker镜像进行微调的方法与[上一章](#微调)基本一致（我们已经在镜像中安装了相关依赖）：\n\n以下是一个单卡LoRA微调的示例：\n```bash\nIMAGE_NAME=qwenllm/qwen:cu117\nCHECKPOINT_PATH=/path/to/Qwen-7B # 下载的模型和代码路径\n#CHECKPOINT_PATH=/path/to/Qwen-7B-Chat-Int4 # 下载的模型和代码路径 (Q-LoRA)\nDATA_PATH=/path/to/data/root # 准备微调数据放在 ${DATA_PATH}/example.json\nOUTPUT_PATH=/path/to/output/checkpoint # 微调输出路径\n\n# 默认使用主机所有GPU\nDEVICE=all\n# 如果需要指定用于训练的GPU，按照以下方式设置device（注意：内层的引号不可省略）\n#DEVICE='\"device=0,1,2,3\"'\n\nmkdir -p ${OUTPUT_PATH}\n\n# 单卡LoRA微调\ndocker run --gpus ${DEVICE} --rm --name qwen \\\n --mount type=bind,source=${CHECKPOINT_PATH},target=/data/shared/Qwen/Qwen-7B \\\n --mount type=bind,source=${DATA_PATH},target=/data/shared/Qwen/data \\\n --mount type=bind,source=${OUTPUT_PATH},target=/data/shared/Qwen/output_qwen \\\n --shm-size=2gb \\\n -it ${IMAGE_NAME} \\\n bash finetune/finetune_lora_single_gpu.sh -m /data/shared/Qwen/Qwen-7B/ -d /data/shared/Qwen/data/example.json\n```\n\n如需修改为单卡Q-LoRA微调示例，只要修改`docker run`中的bash命令：\n```bash\nbash finetune/finetune_qlora_single_gpu.sh -m /data/shared/Qwen/Qwen-7B-Chat-Int4/ -d /data/shared/Qwen/data/example.json\n```\n<br>\n\n## 🔥 系统指令 (System Prompt)\nQwen-1.8-Chat 和 Qwen-72B-Chat 通义千问在多样且存在多轮复杂交互的系统指令上进行了充分训练，使模型可以跟随多样的系统指令，实现上下文(in-context)中的模型定制化，进一步提升了通义千问的可扩展性。\n\n通过系统指令，Qwen-Chat能够实现**角色扮演**，**语言风格迁移**，**任务设定**，和**行为设定**等能力。\n\n![](assets/system_prompt_language_style.png)\n\n![](assets/system_prompt_role_play_en.png)\n\n更多关于系统指令的介绍信息可以参考[示例文档](examples/system_prompt.md).\n\n\n## 工具调用\n\nQwen-Chat针对工具使用、函数调用能力进行了优化。用户可以开发基于Qwen的Agent、LangChain应用、甚至Code Interpreter。\n\n我们提供了文档说明如何根据ReAct Prompting的原理实现工具调用，请参见[ReAct示例](examples/react_prompt.md)。基于该原理，我们在 [openai_api.py](openai_api.py) 里提供了函数调用（Function Calling）的支持。\n我们在已开源的中文[评测数据集](eval/EVALUATION.md)上测试模型的工具调用能力，并发现Qwen-Chat能够取得稳定的表现：\n\n<table>\n <tr>\n <th colspan=\"4\" align=\"center\">中文工具调用评测基准（版本 20231206）</th>\n </tr>\n <tr>\n <th align=\"center\">Model</th><th align=\"center\">Tool Selection (Acc.↑)</th><th align=\"center\">Tool Input (Rouge-L↑)</th><th align=\"center\">False Positive Error↓</th>\n </tr>\n <tr>\n <td>GPT-4</td><td align=\"center\">98.0%</td><td align=\"center\">0.953</td><td align=\"center\">23.9%</td>\n </tr>\n <tr>\n <td>GPT-3.5</td><td align=\"center\">74.5%</td><td align=\"center\">0.807</td><td align=\"center\">80.6%</td>\n </tr>\n <tr>\n <td>Qwen-1_8B-Chat</td><td align=\"center\">85.0%</td><td align=\"center\">0.839</td><td align=\"center\">27.6%</td>\n </tr>\n <tr>\n <td>Qwen-7B-Chat</td><td align=\"center\">95.5%</td><td align=\"center\">0.900</td><td align=\"center\">11.6%</td>\n </tr>\n <tr>\n <td>Qwen-14B-Chat</td><td align=\"center\">96.9%</td><td align=\"center\">0.917</td><td align=\"center\">5.6%</td>\n </tr>\n <tr>\n <td>Qwen-72B-Chat</td><td align=\"center\">98.2%</td><td align=\"center\">0.927</td><td align=\"center\">1.1%</td>\n </tr>\n</table>\n\n为了考察Qwen使用Python Code Interpreter完成数学解题、数据可视化、及文件处理与爬虫等任务的能力，我们专门建设并开源了一个评测这方面能力的[评测基准](https://github.com/QwenLM/Qwen-Agent/tree/main/benchmark)。\n我们发现Qwen在生成代码的可执行率、结果正确性上均表现较好：\n\n<table>\n <tr>\n <th colspan=\"5\" align=\"center\">Code Interpreter Benchmark (Version 20231206)</th>\n </tr>\n <tr>\n <th rowspan=\"2\" align=\"center\">Model</th>\n <th colspan=\"3\" align=\"center\">代码执行结果正确性 (%)</th>\n <th colspan=\"1\" align=\"center\">生成代码的可执行率 (%)</th>\n </tr>\n <tr>\n <th align=\"center\">Math↑</th><th align=\"center\">Visualization-Hard↑</th><th align=\"center\">Visualization-Easy↑</th><th align=\"center\">General↑</th>\n </tr>\n <tr>\n <td>GPT-4</td>\n <td align=\"center\">82.8</td>\n <td align=\"center\">66.7</td>\n <td align=\"center\">60.8</td>\n <td align=\"center\">82.8</td>\n </tr>\n <tr>\n <td>GPT-3.5</td>\n <td align=\"center\">47.3</td>\n <td align=\"center\">33.3</td>\n <td align=\"center\">55.7</td>\n <td align=\"center\">74.1</td>\n </tr>\n <tr>\n <td>LLaMA2-13B-Chat</td>\n <td align=\"center\">8.3</td>\n <td align=\"center\">1.2</td>\n <td align=\"center\">15.2</td>\n <td align=\"center\">48.3</td>\n </tr>\n <tr>\n <td>CodeLLaMA-13B-Instruct</td>\n <td align=\"center\">28.2</td>\n <td align=\"center\">15.5</td>\n <td align=\"center\">21.5</td>\n <td align=\"center\">74.1</td>\n </tr>\n <tr>\n <td>InternLM-20B-Chat</td>\n <td align=\"center\">34.6</td>\n <td align=\"center\">10.7</td>\n <td align=\"center\">25.1</td>\n <td align=\"center\">65.5</td>\n </tr>\n <tr>\n <td>ChatGLM3-6B</td>\n <td align=\"center\">54.2</td>\n <td align=\"center\">4.8</td>\n <td align=\"center\">15.2</td>\n <td align=\"center\">67.1</td>\n </tr>\n <tr>\n <td>Qwen-1.8B-Chat</td>\n <td align=\"center\">25.6</td>\n <td align=\"center\">21.4</td>\n <td align=\"center\">22.8</td>\n <td align=\"center\">65.5</td>\n </tr>\n <tr>\n <td>Qwen-7B-Chat</td>\n <td align=\"center\">41.9</td>\n <td align=\"center\">23.8</td>\n <td align=\"center\">38.0</td>\n <td align=\"center\">67.2</td>\n </tr>\n <tr>\n <td>Qwen-14B-Chat</td>\n <td align=\"center\">58.4</td>\n <td align=\"center\">31.0</td>\n <td align=\"center\">45.6</td>\n <td align=\"center\">65.5</td>\n </tr>\n <tr>\n <td>Qwen-72B-Chat</td>\n <td align=\"center\">72.7</td>\n <td align=\"center\">41.7</td>\n <td align=\"center\">43.0</td>\n <td align=\"center\">82.8</td>\n </tr>\n</table>\n\n<p align=\"center\">\n <br>\n <img src=\"assets/code_interpreter_showcase_001.jpg\" />\n <br>\n<p>\n\n<br>\n\n## 长文本理解\n\n我们引入了NTK插值、窗口注意力、LogN注意力缩放等技术来提升模型的上下文长度并突破训练序列长度的限制，原生长度为2K的Qwen-14B可以扩展到8K的序列长度，而原生长度8K的Qwen-1.8B/7B能够在32K长序列的设置下取得不错的表现。\n\n对于Qwen-72B，我们基于RoPE采用更大的旋转Base来适应更长的上下文。Qwen-72B支持32K的上下文长度。\n\n通过arXiv数据集上的语言模型实验，发现 Qwen 在长上下文场景下可以达到出色的性能。结果如下：\n\n<table>\n <tr>\n <th rowspan=\"2\">Model</th><th colspan=\"6\" align=\"center\">Sequence Length</th>\n </tr>\n <tr>\n <th align=\"center\">1024</th><th align=\"center\">2048</th><th align=\"center\">4096</th><th align=\"center\">8192</th><th align=\"center\">16384</th><th align=\"center\">32768</th>\n </tr>\n <tr>\n <td>Qwen-7B (original)</td><td align=\"center\">4.23</td><td align=\"center\">3.78</td><td align=\"center\">39.35</td><td align=\"center\">469.81</td><td align=\"center\">2645.09</td><td align=\"center\">-</td>\n </tr>\n <tr>\n <td>+ dynamic_ntk</td><td align=\"center\">4.23</td><td align=\"center\">3.78</td><td align=\"center\">3.59</td><td align=\"center\">3.66</td><td align=\"center\">5.71</td><td align=\"center\">-</td>\n </tr>\n <tr>\n <td>Qwen-1.8B</td><td align=\"center\"><b>5.00</b></td><td align=\"center\"><b>4.48</b></td><td align=\"center\"><b>4.13</b></td><td align=\"center\"><b>3.89</b></td><td align=\"center\">17.42</td><td align=\"center\">433.85</td>\n </tr>\n <tr>\n <td>+ dynamic_ntk + logn + window_attn</td><td align=\"center\"><b>5.00</b></td><td align=\"center\"><b>4.48</b></td><td align=\"center\"><b>4.14</b></td><td align=\"center\"><b>3.93</b></td><td align=\"center\"><b>3.82</b></td><td align=\"center\"><b>3.83</b></td>\n </tr>\n <tr>\n <td>Qwen-7B</td><td align=\"center\"><b>4.23</b></td><td align=\"center\"><b>3.81</b></td><td align=\"center\"><b>3.52</b></td><td align=\"center\"><b>3.31</b></td><td align=\"center\">7.27</td><td align=\"center\">181.49</td>\n </tr>\n <tr>\n <td>+ dynamic_ntk</td><td align=\"center\"><b>4.23</b></td><td align=\"center\"><b>3.81</b></td><td align=\"center\"><b>3.52</b></td><td align=\"center\"><b>3.31</b></td><td align=\"center\"><b>3.23</b></td><td align=\"center\">3.33</td>\n </tr>\n <tr>\n <td>+ dynamic_ntk + logn + window_attn</td><td align=\"center\"><b>4.23</b></td><td align=\"center\"><b>3.81</b></td><td align=\"center\"><b>3.52</b></td><td align=\"center\"><b>3.33</b></td><td align=\"center\"><b>3.22</b></td><td align=\"center\"><b>3.17</b></td>\n </tr>\n <tr>\n <td>Qwen-14B</td><td align=\"center\"><b>-</b></td><td align=\"center\"><b>3.46</b></td><td align=\"center\">22.79</td><td align=\"center\">334.65</td><td align=\"center\">3168.35</td><td align=\"center\">-</td>\n </tr>\n <tr>\n <td>+ dynamic_ntk + logn + window_attn</td><td align=\"center\"><b>-</b></td><td align=\"center\"><b>3.46</b></td><td align=\"center\"><b>3.29</b></td><td align=\"center\"><b>3.18</b></td><td align=\"center\">3.42</td><td align=\"center\">-</td>\n </tr>\n <tr>\n <td>Qwen-72B</td><td align=\"center\"><b>-</b></td><td align=\"center\"><b>-</b></td><td align=\"center\">-</td><td align=\"center\"><b>2.83</b></td><td align=\"center\"><b>2.73</b></td><td align=\"center\"><b>2.72</b></td>\n </tr>\n</table>\n\n进一步，我们为了验证Qwen-72B-Chat在长文本任务上的能力，在[L-Eval](https://arxiv.org/abs/2307.11088)客观题上进行了测试，评分结果如下：\n\n| Model | Input Length | Average | Coursera | GSM | QuALITY | TOEFL | CodeU | SFcition |\n|:------------------|:------------:|:---------:|:----------:|:----------:|:----------:|:----------:|:----------:|:----------:|\n| ChatGPT-3.5-16k | 16K | 60.73 | **63.51** | **84.00** | 61.38 | 78.43 | **12.22** | 64.84 |\n| **Qwen-72B-Chat** | 32K | **62.30** | 58.13 | 76.00 | **77.22** | **86.24** | 6.66 | **69.53** |\n\n\n我们进一步进行了“大海捞针”实验（想法来自于[@Greg Kamradt](https://twitter.com/GregKamradt/status/1727018183608193393)），测试模型在不同长度的输入下，是否能检索到文章不同位置的信息，结果如下：\n\n![](assets/qwen_72b_needle_in_a_haystack.png)\n\n以上结果说明，Qwen-72B-Chat可以能准确检索到32K以内的输入长度中放在各种位置的信息，证明了其具有优秀的长文本处理能力。\n\n## Tokenizer\n\n> 注：作为术语的“tokenizer”在中文中尚无共识的概念对应，本文档采用英文表达以利说明。\n\n基于tiktoken的tokenizer有别于其他分词器，比如sentencepiece tokenizer。尤其在微调阶段，需要特别注意特殊token的使用。关于tokenizer的更多信息，以及微调时涉及的相关使用，请参阅[文档](tokenization_note_zh.md)。\n<br><br>\n\n## 复现\n\n我们提供了评测脚本以供复现我们的实验结果。注意，由于内部代码和开源代码存在少许差异，评测结果可能与汇报结果存在细微的结果不一致。请阅读[eval/EVALUATION.md](eval/EVALUATION.md)了解更多信息。\n<br><br>\n\n## FAQ\n\n如遇到问题，敬请查阅[FAQ](FAQ_zh.md)以及issue区，如仍无法解决再提交issue。\n<br><br>\n\n## 引用\n如果你觉得我们的工作对你有帮助，欢迎引用！\n\n```\n@article{qwen,\n title={Qwen Technical Report},\n author={Jinze Bai and Shuai Bai and Yunfei Chu and Zeyu Cui and Kai Dang and Xiaodong Deng and Yang Fan and Wenbin Ge and Yu Han and Fei Huang and Binyuan Hui and Luo Ji and Mei Li and Junyang Lin and Runji Lin and Dayiheng Liu and Gao Liu and Chengqiang Lu and Keming Lu and Jianxin Ma and Rui Men and Xingzhang Ren and Xuancheng Ren and Chuanqi Tan and Sinan Tan and Jianhong Tu and Peng Wang and Shijie Wang and Wei Wang and Shengguang Wu and Benfeng Xu and Jin Xu and An Yang and Hao Yang and Jian Yang and Shusheng Yang and Yang Yao and Bowen Yu and Hongyi Yuan and Zheng Yuan and Jianwei Zhang and Xingxuan Zhang and Yichang Zhang and Zhenru Zhang and Chang Zhou and Jingren Zhou and Xiaohuan Zhou and Tianhang Zhu},\n journal={arXiv preprint arXiv:2309.16609},\n year={2023}\n}\n```\n<br>\n\n## 使用协议\n\n<https://github.com/QwenLM/Qwen>中的源代码采用[Apache 2.0协议](./LICENSE)授权，您可在该仓库根目录找到协议全文。\n\n研究人员与开发者可使用Qwen和Qwen-Chat或进行二次开发。对于商业使用，请查看模型各自的LICENSE。\n\n- Qwen-72B、Qwen-14B和Qwen-7B采用[Tongyi Qianwen LICENSE AGREEMENT](./Tongyi%20Qianwen%20LICENSE%20AGREEMENT)授权，您可在相应模型的HuggingFace或ModelScope仓库找到协议原文。如需商用，您只需遵循使用协议进行商用即可，我们欢迎您填写问卷([72B](https://dashscope.console.aliyun.com/openModelApply/Qwen-72B-Chat)、[14B](https://dashscope.console.aliyun.com/openModelApply/Qwen-14B-Chat)、[7B](https://dashscope.console.aliyun.com/openModelApply/qianwen))。\n\n- Qwen-1.8B采用[Tongyi Qianwen RESEARCH LICENSE AGREEMENT](./Tongyi%20Qianwen%20RESEARCH%20LICENSE%20AGREEMENT)授权，您可在相应模型的HuggingFace或ModelScope仓库找到协议原文。如需商用，请联系我们。\n\n<br><br>\n\n## 联系我们\n\n如果你想给我们的研发团队和产品团队留言，欢迎加入我们的微信群和Discord server。当然也可以通过邮件（qianwen_opensource@alibabacloud.com）联系我们。\n\n" }, { "path": "README_ES.md", "content": "<p align=\"left\">\n <a href=\"README_CN.md\">中文</a>  ｜  <a href=\"README.md\">English</a>  ｜  <a href=\"README_JA.md\">日本語</a> ｜  <a href=\"README_FR.md\">Français</a> ｜  Español\n</p>\n<br><br>\n\n<p align=\"center\">\n <img src=\"https://qianwen-res.oss-cn-beijing.aliyuncs.com/logo_qwen.jpg\" width=\"400\"/>\n<p>\n<br>\n\n<p align=\"center\">\n 🤗 <a href=\"https://huggingface.co/Qwen\">Hugging Face</a>   |   🤖 <a href=\"https://modelscope.cn/organization/qwen\">ModelScope</a>   |    📑 <a href=\"https://arxiv.org/abs/2309.16609\">Paper</a>    ｜   🖥️ <a href=\"https://modelscope.cn/studios/qwen/Qwen-72B-Chat-Demo/summary\">Demo</a>\n<br>\n<a href=\"assets/wechat.png\">WeChat (微信)</a>   |   <a href=\"https://discord.gg/z3GAxXZ9Ce\">Discord</a>   ｜   <a href=\"https://dashscope.aliyun.com\">API</a> \n</p>\n<br><br>\n\n> [!Important]\n> ¡Qwen2 está aquí! Estás invitado a seguir [QwenLM/Qwen2](https://github.com/QwenLM/Qwen2) y compartir tu experiencia allí.\n>\n> Este repositorio ([QwenLM/Qwen](https://github.com/QwenLM/Qwen)) ya no se mantiene activamente, debido a diferencias sustanciales en la base de código.\n<br>\n\n\n| | Qwen-Chat | Qwen-Chat (Int4) | Qwen-Chat (Int8) | Qwen |\n|-----|:------------------------------------------------------------------------------------------------------------------------------------:|:----------------------------------------------------------------------------------------------------------------------------------------------:|:---------------------------------------------------------------:|:--------------------------------------------------------------------------------------------------------------------------:|\n| 1.8B | <a href=\"https://modelscope.cn/models/qwen/Qwen-1_8B-Chat/summary\">🤖</a> <a href=\"https://huggingface.co/Qwen/Qwen-1_8B-Chat\">🤗</a> | <a href=\"https://modelscope.cn/models/qwen/Qwen-1_8B-Chat-Int4/summary\">🤖</a> <a href=\"https://huggingface.co/Qwen/Qwen-1_8B-Chat-Int4\">🤗</a> | <a href=\"https://modelscope.cn/models/qwen/Qwen-1_8B-Chat-Int8/summary\">🤖</a> <a href=\"https://huggingface.co/Qwen/Qwen-1_8B-Chat-Int8\">🤗</a> | <a href=\"https://modelscope.cn/models/qwen/Qwen-1_8B/summary\">🤖</a> <a href=\"https://huggingface.co/Qwen/Qwen-1_8B\">🤗</a> |\n| 7B | <a href=\"https://modelscope.cn/models/qwen/Qwen-7B-Chat/summary\">🤖</a> <a href=\"https://huggingface.co/Qwen/Qwen-7B-Chat\">🤗</a> | <a href=\"https://modelscope.cn/models/qwen/Qwen-7B-Chat-Int4/summary\">🤖</a> <a href=\"https://huggingface.co/Qwen/Qwen-7B-Chat-Int4\">🤗</a> | <a href=\"https://modelscope.cn/models/qwen/Qwen-7B-Chat-Int8/summary\">🤖</a> <a href=\"https://huggingface.co/Qwen/Qwen-7B-Chat-Int8\">🤗</a> | <a href=\"https://modelscope.cn/models/qwen/Qwen-7B/summary\">🤖</a> <a href=\"https://huggingface.co/Qwen/Qwen-7B\">🤗</a> |\n| 14B | <a href=\"https://modelscope.cn/models/qwen/Qwen-14B-Chat/summary\">🤖</a> <a href=\"https://huggingface.co/Qwen/Qwen-14B-Chat\">🤗</a> | <a href=\"https://modelscope.cn/models/qwen/Qwen-14B-Chat-Int4/summary\">🤖</a> <a href=\"https://huggingface.co/Qwen/Qwen-14B-Chat-Int4\">🤗</a> | <a href=\"https://modelscope.cn/models/qwen/Qwen-14B-Chat-Int8/summary\">🤖</a> <a href=\"https://huggingface.co/Qwen/Qwen-14B-Chat-Int8\">🤗</a> | <a href=\"https://modelscope.cn/models/qwen/Qwen-14B/summary\">🤖</a> <a href=\"https://huggingface.co/Qwen/Qwen-14B\">🤗</a> |\n| 72B | <a href=\"https://modelscope.cn/models/qwen/Qwen-72B-Chat/summary\">🤖</a> <a href=\"https://huggingface.co/Qwen/Qwen-72B-Chat\">🤗</a> | <a href=\"https://modelscope.cn/models/qwen/Qwen-72B-Chat-Int4/summary\">🤖</a> <a href=\"https://huggingface.co/Qwen/Qwen-72B-Chat-Int4\">🤗</a> | <a href=\"https://modelscope.cn/models/qwen/Qwen-72B-Chat-Int8/summary\">🤖</a> <a href=\"https://huggingface.co/Qwen/Qwen-72B-Chat-Int8\">🤗</a> | <a href=\"https://modelscope.cn/models/qwen/Qwen-72B/summary\">🤖</a> <a href=\"https://huggingface.co/Qwen/Qwen-72B\">🤗</a> |\n\n\n\nAbrimos nuestra serie **Qwen**, que ahora incluye **Qwen**, los modelos de lenguaje, es decir **Qwen-7B** y **Qwen-14B**, así como **Qwen-Chat**, los modelos de chat, es decir **Qwen-7B-Chat** y **Qwen-14B-Chat**. Los enlaces se encuentran en la tabla anterior. Haz clic en ellos y comprueba las fichas de los modelos. Además, publicamos el **[informe técnico](https://arxiv.org/abs/2309.16609)**. Haz clic en el enlace y compruébalo.\n\nEn resumen, disponemos de modelos lingüísticos sólidos, que han sido preentrenados de forma estable para hasta 3 billones de tokens de datos multilingües con una amplia cobertura de dominios, idiomas (con especial atención al chino y al inglés), etc. Son capaces de lograr un rendimiento competitivo en conjuntos de datos de referencia. Además, disponemos de modelos de chat alineados con las preferencias humanas basados en SFT y RLHF (aún no publicados), que son capaces de chatear, crear contenidos, extraer información, resumir, traducir, codificar, resolver problemas matemáticos, etc., y son capaces de utilizar herramientas, jugar como agentes o incluso jugar como intérpretes de código, etc.\n\n| Modelo | Fecha de Publicación | Longitud Máx. | Mejora del Sistema de Avisos | # de Fichas Preentrenadas | Uso Mínimo de Memoria GPU de Finetuning (Q-Lora) | Uso Mínimo de la GPU para Generar 2048 Tokens (Int4) | Uso de Herramientas |\n|:----------|:--------------------:|:-------------:|:----------------------------:|:-------------------------:|:------------------------------------------------:|:----------------------------------------------------:|:-------------------:|\n| Qwen-1.8B | 23.11.30 | 32K | ✅ | 2.2T | 5.8GB | 2.9GB | ✅ | \n| Qwen-7B | 23.08.03 | 32K | ❎ | 2.4T | 11.5GB | 8.2GB | ✅ | \n| Qwen-14B | 23.09.25 | 8K | ❎ | 3.0T | 18.7GB | 13.0GB | ✅ |\n| Qwen-72B | 23.11.30 | 32K | ✅ | 3.0T | 61.4GB | 48.9GB | ✅ | \n\nEn este repo, usted puede averiguar:\n\n* Inicio rápido con Qwen, y disfrute de la simple inferencia.\n* Detalles sobre los modelos de cuantificación, incluyendo GPTQ y cuantización de caché KV.\n* Estadísticas de rendimiento de la inferencia, incluyendo velocidad y memoria.\n* Tutoriales sobre ajuste fino, incluyendo ajuste de parámetros completos, LoRA y Q-LoRA.\n* Instrucciones de despliegue, con el ejemplo de vLLM y FastChat.\n* Instrucciones para construir demos, incluyendo WebUI, CLI demo, etc.\n* Introducción al servicio API de DashScope, así como instrucciones para crear una API de estilo OpenAI para tu modelo.\n* Información sobre Qwen para el uso de herramientas, agente e intérprete de código.\n* Estadísticas de la evaluación de la comprensión del contexto largo\n* Acuerdo de licencia\n* ...\n\nAdemás, si tienes problemas, consulta primero [FAQ](FAQ.md) para obtener ayuda. ¿Sigues teniendo problemas? No dudes en plantearnos tus problemas (mejor en inglés para que te entienda más gente). Si quieres ayudarnos, ¡envíanos pull requests sin dudarlo! ¡Siempre nos entusiasman los PR!\n\n¿Quieres charlar con nosotros o quedar para tomar un café? ¡Bienvenido a nuestro Discord o WeChat!\n<br><br>\n\n## Noticias y Actualizaciones\n\n* 2023.11.30 🔥 Lanzamos **Qwen-72B** y **Qwen-72B-Chat**, que están entrenados en tokens 3T y soportan 32k contextos, junto con **Qwen-1.8B**, y **Qwen-1.8B-Chat**, en ModelScope y Hugging Face. También hemos reforzado las capacidades de System Prompt de Qwen-72B-Chat y Qwen-1.8B-Chat, ver [documentación de ejemplo](examples/system_prompt.md). Adicionalmente, soporta la inferencia en **Ascend 910** y **Hygon DCU**. Consulta `ascend-support` y `dcu-support` para más detalles.\n* 2023.10.17 Publicamos el modelo cuantizado Int8 **Qwen-7B-Chat-Int8** y **Qwen-14B-Chat-Int8**.\n* 2023.9.25 Publicamos **Qwen-14B** y **Qwen-14B-Chat** en ModelScope y Hugging Face, junto con [qwen.cpp](https://github.com/QwenLM/qwen.cpp) y [Qwen-Agent](https://github.com/QwenLM/Qwen-Agent). También se actualizan los códigos y pesos de **Qwen-7B** y **Qwen-7B-Chat**. **POR FAVOR, DESCARGA LA ÚLTIMA VERSIÓN!**\n - En comparación con **Qwen-7B** (original), **Qwen-7B** utiliza más tokens de entrenamiento, pasando de 2,2T tokens a 2,4T tokens, mientras que la longitud del contexto se amplía de 2048 a 8192. El conocimiento del chino y la capacidad de codificación de **Qwen-7B** se han mejorado aún más.\n* 2023.9.12 Ahora es posible el ajuste fino de los modelos Qwen-7B, incluido el ajuste fino de parámetros completos, LoRA y Q-LoRA.\n* 2023.8.21 Publicamos el modelo cuantizado Int4 para Qwen-7B-Chat, **Qwen-7B-Chat-Int4**, que requiere bajos costes de memoria pero consigue mejorar la velocidad de inferencia. Además, no se produce una degradación significativa del rendimiento en la evaluación comparativa.\n* 2023.8.3 Publicamos **Qwen-7B** y **Qwen-7B-Chat** en ModelScope y Hugging Face. También proporcionamos una nota técnica para más detalles sobre el modelo, incluidos los detalles de entrenamiento y el rendimiento del modelo.\n<br>\n\n## Rendimiento\n\nLos modelos Qwen superan a los modelos de referencia de tamaños de modelo similares en una serie de conjuntos de datos de referencia, como MMLU, C-Eval, GSM8K, MATH, HumanEval, MBPP, BBH, etc., que evalúan las capacidades de los modelos en comprensión del lenguaje natural, resolución de problemas matemáticos, codificación, etc. Qwen-72B obtiene mejores resultados que LLaMA2-70B en todas las tareas y supera a GPT-3.5 en 7 de cada 10 tareas.\n\n<p align=\"left\">\n <img src=\"assets/radar_72b.jpg\" width=600px/>\n<p>\n<br>\n\n| Model | MMLU | C-Eval | GSM8K | MATH | HumanEval | MBPP | BBH | CMMLU |\n|:------------------|:--------:|:--------:|:--------:|:--------:|:---------:|:--------:|:--------:|:--------:|\n| | 5-shot | 5-shot | 8-shot | 4-shot | 0-shot | 3-shot | 3-shot | 5-shot |\n| LLaMA2-7B | 46.8 | 32.5 | 16.7 | 3.3 | 12.8 | 20.8 | 38.2 | 31.8 |\n| LLaMA2-13B | 55.0 | 41.4 | 29.6 | 5.0 | 18.9 | 30.3 | 45.6 | 38.4 |\n| LLaMA2-34B | 62.6 | - | 42.2 | 6.2 | 22.6 | 33.0 | 44.1 | - |\n| ChatGLM2-6B | 47.9 | 51.7 | 32.4 | 6.5 | - | - | 33.7 | - |\n| InternLM-7B | 51.0 | 53.4 | 31.2 | 6.3 | 10.4 | 14.0 | 37.0 | 51.8 |\n| InternLM-20B | 62.1 | 58.8 | 52.6 | 7.9 | 25.6 | 35.6 | 52.5 | 59.0 |\n| Baichuan2-7B | 54.7 | 56.3 | 24.6 | 5.6 | 18.3 | 24.2 | 41.6 | 57.1 |\n| Baichuan2-13B | 59.5 | 59.0 | 52.8 | 10.1 | 17.1 | 30.2 | 49.0 | 62.0 |\n| Yi-34B \t \t | 76.3 | 81.8 | 67.9 | 15.9 | 26.2 | 38.2 | 66.4 | 82.6 |\n| XVERSE-65B \t | 70.8 | 68.6 | 60.3 | - | 26.3 | - | - | - |\n| **Qwen-1.8B** | 45.3 | 56.1 | 32.3 | 2.3 | 15.2 | 14.2 | 22.3 | 52.1 |\n| **Qwen-7B** | 58.2 | 63.5 | 51.7 | 11.6 | 29.9 | 31.6 | 45.0 | 62.2 |\n| **Qwen-14B** | 66.3 | 72.1 | 61.3 | 24.8 | 32.3 | 40.8 | 53.4 | 71.0 |\n| **Qwen-72B** | **77.4** | **83.3** | **78.9** | **35.2** | **35.4** | **52.2** | **67.7** | **83.6** |\n\nPara todos los modelos comparados, presentamos las mejores puntuaciones entre sus resultados oficiales y [OpenCompass](https://opencompass.org.cn/leaderboard-llm).\n\nPara más resultados experimentales (rendimiento detallado del modelo en más conjuntos de datos de referencia) y detalles, consulte nuestro informe técnico haciendo clic [aquí](https://qianwen-res.oss-cn-beijing.aliyuncs.com/QWEN_TECHNICAL_REPORT.pdf).\n<br><br>\n\n## Requisitos\n\n* python 3.8 y superior\n* pytorch 1.12 y superior, se recomienda 2.0 y superior\n* transformers 4.32 y superiores\n* Se recomienda CUDA 11.4 y superior (esto es para usuarios de GPU, usuarios de flash-attention, etc.)\n<br>\n\n## Inicio rápido\n\nA continuación, proporcionamos ejemplos sencillos para mostrar cómo utilizar Qwen-Chat con 🤖 ModelScope y 🤗 Transformers.\n\nPuedes usar nuestras imágenes docker pre-construidas para saltarte la mayoría de los pasos de configuración del entorno, mira la Sección [\"Usando Imágenes Docker Pre-construidas\"](#-docker) para más detalles. \n\nSi no utiliza Docker, asegúrese de haber configurado el entorno e instalado los paquetes necesarios. Asegúrese de que cumple los requisitos anteriores y, a continuación, instale las bibliotecas dependientes.\n\n```bash\npip install -r requirements.txt\n```\n\nSi tu dispositivo soporta fp16 o bf16, te recomendamos instalar [flash-attention](https://github.com/Dao-AILab/flash-attention) (**ahora soportamos flash attention 2.**) para una mayor eficiencia y un menor uso de memoria. (**flash-attention es opcional y el proyecto puede ejecutarse normalmente sin instalarlo**)\n\n```bash\ngit clone https://github.com/Dao-AILab/flash-attention\ncd flash-attention && pip install .\n# Below are optional. Installing them might be slow.\n# pip install csrc/layer_norm\n# pip install csrc/rotary\n```\n\nAhora puedes empezar con ModelScope o Transformers.\n\n### 🤗 Transformers\n\nPara utilizar Qwen-Chat para la inferencia, todo lo que tienes que hacer es introducir unas pocas líneas de código como se demuestra a continuación. Recuerda introducir los nombres o rutas correctos de los modelos, como \"Qwen/Qwen-7B-Chat\" y \"Qwen/Qwen-14B-Chat\". Sin embargo, **por favor, asegúrese de que está utilizando el código más reciente.**\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\nfrom transformers.generation import GenerationConfig\n\n# Model names: \"Qwen/Qwen-7B-Chat\", \"Qwen/Qwen-14B-Chat\"\ntokenizer = AutoTokenizer.from_pretrained(\"Qwen/Qwen-7B-Chat\", trust_remote_code=True)\n\n# use bf16\n# model = AutoModelForCausalLM.from_pretrained(\"Qwen/Qwen-7B-Chat\", device_map=\"auto\", trust_remote_code=True, bf16=True).eval()\n# use fp16\n# model = AutoModelForCausalLM.from_pretrained(\"Qwen/Qwen-7B-Chat\", device_map=\"auto\", trust_remote_code=True, fp16=True).eval()\n# use cpu only\n# model = AutoModelForCausalLM.from_pretrained(\"Qwen/Qwen-7B-Chat\", device_map=\"cpu\", trust_remote_code=True).eval()\n# use auto mode, automatically select precision based on the device.\nmodel = AutoModelForCausalLM.from_pretrained(\n \"Qwen/Qwen-7B-Chat\",\n device_map=\"auto\",\n trust_remote_code=True\n).eval()\n\n# Specify hyperparameters for generation. But if you use transformers>=4.32.0, there is no need to do this.\n# model.generation_config = GenerationConfig.from_pretrained(\"Qwen/Qwen-7B-Chat\", trust_remote_code=True)\n\n# 1st dialogue turn\nresponse, history = model.chat(tokenizer, \"你好\", history=None)\nprint(response)\n# 你好！很高兴为你提供帮助。\n\n# 2nd dialogue turn\nresponse, history = model.chat(tokenizer, \"给我讲一个年轻人奋斗创业最终取得成功的故事。\", history=history)\nprint(response)\n# 这是一个关于一个年轻人奋斗创业最终取得成功的故事。\n# 故事的主人公叫李明，他来自一个普通的家庭，父母都是普通的工人。从小，李明就立下了一个目标：要成为一名成功的企业家。\n# 为了实现这个目标，李明勤奋学习，考上了大学。在大学期间，他积极参加各种创业比赛，获得了不少奖项。他还利用课余时间去实习，积累了宝贵的经验。\n# 毕业后，李明决定开始自己的创业之路。他开始寻找投资机会，但多次都被拒绝了。然而，他并没有放弃。他继续努力，不断改进自己的创业计划，并寻找新的投资机会。\n# 最终，李明成功地获得了一笔投资，开始了自己的创业之路。他成立了一家科技公司，专注于开发新型软件。在他的领导下，公司迅速发展起来，成为了一家成功的科技企业。\n# 李明的成功并不是偶然的。他勤奋、坚韧、勇于冒险，不断学习和改进自己。他的成功也证明了，只要努力奋斗，任何人都有可能取得成功。\n\n# 3rd dialogue turn\nresponse, history = model.chat(tokenizer, \"给这个故事起一个标题\", history=history)\nprint(response)\n# 《奋斗创业：一个年轻人的成功之路》\n```\n\nEjecutar Qwen, el modelo lingüístico base, también es sencillo.\n\n<details>\n <summary>Ejecutar Qwen</summary>\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\nfrom transformers.generation import GenerationConfig\n\n# Model names: \"Qwen/Qwen-7B\", \"Qwen/Qwen-14B\" \ntokenizer = AutoTokenizer.from_pretrained(\"Qwen/Qwen-7B\", trust_remote_code=True)\n# use bf16\n# model = AutoModelForCausalLM.from_pretrained(\"Qwen/Qwen-7B\", device_map=\"auto\", trust_remote_code=True, bf16=True).eval()\n# use fp16\n# model = AutoModelForCausalLM.from_pretrained(\"Qwen/Qwen-7B\", device_map=\"auto\", trust_remote_code=True, fp16=True).eval()\n# use cpu only\n# model = AutoModelForCausalLM.from_pretrained(\"Qwen/Qwen-7B\", device_map=\"cpu\", trust_remote_code=True).eval()\n# use auto mode, automatically select precision based on the device.\nmodel = AutoModelForCausalLM.from_pretrained(\n \"Qwen/Qwen-7B\",\n device_map=\"auto\",\n trust_remote_code=True\n).eval()\n\n# Specify hyperparameters for generation. But if you use transformers>=4.32.0, there is no need to do this.\n# model.generation_config = GenerationConfig.from_pretrained(\"Qwen/Qwen-7B\", trust_remote_code=True)\n\ninputs = tokenizer('蒙古国的首都是乌兰巴托（Ulaanbaatar）\\n冰岛的首都是雷克雅未克（Reykjavik）\\n埃塞俄比亚的首都是', return_tensors='pt')\ninputs = inputs.to(model.device)\npred = model.generate(**inputs)\nprint(tokenizer.decode(pred.cpu()[0], skip_special_tokens=True))\n# 蒙古国的首都是乌兰巴托（Ulaanbaatar）\\n冰岛的首都是雷克雅未克（Reykjavik）\\n埃塞俄比亚的首都是亚的斯亚贝巴（Addis Ababa）...\n```\n\n</details>\n\nEn caso de que se produzca un problema de red al intentar descargar puntos de control y códigos de modelos desde Hugging Face, un método alternativo consiste en obtener inicialmente el punto de control desde ModelScope y luego cargarlo desde el directorio local como se indica a continuación:\n\n```python\nfrom modelscope import snapshot_download\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\n# Downloading model checkpoint to a local dir model_dir\n# model_dir = snapshot_download('qwen/Qwen-7B', revision='v1.1.4')\n# model_dir = snapshot_download('qwen/Qwen-7B-Chat', revision='v1.1.4')\n# model_dir = snapshot_download('qwen/Qwen-14B', revision='v1.0.4')\nmodel_dir = snapshot_download('qwen/Qwen-14B-Chat', revision='v1.0.4')\n\n# Loading local checkpoints\n# trust_remote_code is still set as True since we still load codes from local dir instead of transformers\ntokenizer = AutoTokenizer.from_pretrained(model_dir, trust_remote_code=True)\nmodel = AutoModelForCausalLM.from_pretrained(\n model_dir,\n device_map=\"auto\",\n trust_remote_code=True\n).eval()\n```\n\n### 🤖 ModelScope\n\nModelScope es una plataforma de código abierto para Model-as-a-Service (MaaS), que proporciona un servicio de modelos flexible y rentable a los desarrolladores de IA. Del mismo modo, puede ejecutar los modelos con ModelScope como se muestra a continuación:\n\n```python\nfrom modelscope import AutoModelForCausalLM, AutoTokenizer\nfrom modelscope import GenerationConfig\n\n# Model names: \"qwen/Qwen-7B-Chat\", \"qwen/Qwen-14B-Chat\"\ntokenizer = AutoTokenizer.from_pretrained(\"qwen/Qwen-7B-Chat\", revision='v1.0.5', trust_remote_code=True)\nmodel = AutoModelForCausalLM.from_pretrained(\"qwen/Qwen-7B-Chat\", revision='v1.0.5', device_map=\"auto\", trust_remote_code=True, fp16=True).eval()\nmodel.generation_config = GenerationConfig.from_pretrained(\"Qwen/Qwen-7B-Chat\", revision='v1.0.5', trust_remote_code=True) # 可指定不同的生成长度、top_p等相关超参\n\nresponse, history = model.chat(tokenizer, \"你好\", history=None)\nprint(response)\nresponse, history = model.chat(tokenizer, \"浙江的省会在哪里？\", history=history) \nprint(response)\nresponse, history = model.chat(tokenizer, \"它有什么好玩的景点\", history=history)\nprint(response)\n```\n\n### Inferencia por lotes\nQwen admite la inferencia por lotes. Con la atención flash activada, el uso de la inferencia por lotes puede suponer un aumento de velocidad del 40%. El código de ejemplo se muestra a continuación:\n\n```python\nimport torch\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\nfrom transformers import GenerationConfig\nfrom qwen_generation_utils import make_context, decode_tokens, get_stop_words_ids\n\ntokenizer = AutoTokenizer.from_pretrained(\n './',\n pad_token='<|extra_0|>',\n eos_token='<|endoftext|>',\n padding_side='left',\n trust_remote_code=True\n)\nmodel = AutoModelForCausalLM.from_pretrained(\n './',\n pad_token_id=tokenizer.pad_token_id,\n device_map=\"auto\",\n trust_remote_code=True\n).eval()\nmodel.generation_config = GenerationConfig.from_pretrained('./', pad_token_id=tokenizer.pad_token_id)\n\nall_raw_text = [\"我想听你说爱我。\", \"今天我想吃点啥，甜甜的，推荐下\", \"我马上迟到了，怎么做才能不迟到\"]\nbatch_raw_text = []\nfor q in all_raw_text:\n raw_text, _ = make_context(\n tokenizer,\n q,\n system=\"You are a helpful assistant.\",\n max_window_size=model.generation_config.max_window_size,\n chat_format=model.generation_config.chat_format,\n )\n batch_raw_text.append(raw_text)\n\nbatch_input_ids = tokenizer(batch_raw_text, padding='longest')\nbatch_input_ids = torch.LongTensor(batch_input_ids['input_ids']).to(model.device)\nbatch_out_ids = model.generate(\n batch_input_ids,\n return_dict_in_generate=False,\n generation_config=model.generation_config\n)\npadding_lens = [batch_input_ids[i].eq(tokenizer.pad_token_id).sum().item() for i in range(batch_input_ids.size(0))]\n\nbatch_response = [\n decode_tokens(\n batch_out_ids[i][padding_lens[i]:],\n tokenizer,\n raw_text_len=len(batch_raw_text[i]),\n context_length=(batch_input_ids[i].size(0)-padding_lens[i]),\n chat_format=\"chatml\",\n verbose=False,\n errors='replace'\n ) for i in range(len(all_raw_text))\n]\nprint(batch_response)\n\nresponse, _ = model.chat(tokenizer, \"我想听你说爱我。\", history=None)\nprint(response)\n\nresponse, _ = model.chat(tokenizer, \"今天我想吃点啥，甜甜的，推荐下\", history=None)\nprint(response)\n\nresponse, _ = model.chat(tokenizer, \"我马上迟到了，怎么做才能不迟到\", history=None)\nprint(response)\n```\n\n### CPU\n\nPara desplegar nuestros modelos en la CPU, le recomendamos encarecidamente que utilice [qwen.cpp](https://github.com/QwenLM/qwen.cpp), que es una implementación C++ pura de Qwen y tiktoken. Comprueba el repositorio para más detalles.\n\nAdemás, también es sencillo ejecutar directamente el modelo en la CPU, lo que requiere que especifiques el dispositivo:\n\n```python\nmodel = AutoModelForCausalLM.from_pretrained(\"Qwen/Qwen-7B-Chat\", device_map=\"cpu\", trust_remote_code=True).eval()\n```\n\nPero es probable que sufra una eficacia de inferencia extremadamente baja.\n\n### Múltiples GPU\n\nSi sufres de falta de memoria en la GPU y quieres ejecutar el modelo en más de 1 GPU, puedes utilizar directamente el método de carga por defecto, que ahora es soportado por Transformers. El método anterior basado en `utils.py` está obsoleto.\n\nSin embargo, aunque este método es sencillo, la eficiencia del paralelismo del pipeline nativo es baja. Le aconsejamos que utilice vLLM con FastChat y por favor lea la sección para el despliegue.\n\n### DashScope\n\nLa forma más sencilla de utilizar Qwen a través de APIs es el servicio DashScope API a través de Alibaba Cloud. Damos una introducción al uso. Además, proporcionamos un script para que despliegues una API estilo OpenAI en tus propios servidores.\n\nDashScope es el gran servicio de API de modelos lingüísticos proporcionado por Alibaba Cloud, que ahora es compatible con Qwen. Tenga en cuenta que los modelos detrás de DashScope son versiones internas temporalmente sin detalles proporcionados. Los servicios incluyen `qwen-turbo` y `qwen-plus`, donde el primero se ejecuta más rápido y el segundo consigue un mejor rendimiento. Para más información, visita la documentación [aquí](https://dashscope.aliyun.com).\n\nDirígete al sitio web oficial [enlace](https://help.aliyun.com/zh/dashscope/developer-reference/activate-dashscope-and-create-an-api-key?spm=a2c4g.11186623.0.0.6c2774fahtfXdn) para crear una cuenta DashScope y obtener la clave API (AK). Recomendamos configurar la AK con una variable de entorno:\n```bash\nexport DASHSCOPE_API_KEY=\"YOUR_DASHSCOPE_API_KEY\"\n```\nA continuación, instala los paquetes y haz clic [aquí](https://help.aliyun.com/zh/dashscope/developer-reference/install-dashscope-sdk) para consultar la documentación. Si utilizas Python, puedes instalar DashScope con pip:\n```bash\npip install dashscope\n```\nSi utiliza JAVA SDK, puede instalarlo de esta forma:\n```xml\n<!-- https://mvnrepository.com/artifact/com.alibaba/dashscope-sdk-java -->\n<dependency>\n <groupId>com.alibaba</groupId>\n <artifactId>dashscope-sdk-java</artifactId>\n <version>the-latest-version</version>\n</dependency>\n```\nLa forma más sencilla de utilizar DashScope es el uso con mensajes, que es similar a la API OpenAI. El ejemplo se muestra a continuación:\n```python\nimport random\nfrom http import HTTPStatus\nfrom dashscope import Generation\n\n\ndef call_with_messages():\n messages = [{'role': 'system', 'content': 'You are a helpful assistant.'},\n {'role': 'user', 'content': '如何做西红柿鸡蛋？'}]\n gen = Generation()\n response = gen.call(\n Generation.Models.qwen_turbo,\n messages=messages,\n seed=random.randint(1, 10000), # set the random seed, optional, default to 1234 if not set\n result_format='message', # set the result to be \"message\" format.\n )\n return response\n\n\nif __name__ == '__main__':\n response = call_with_messages()\n if response.status_code == HTTPStatus.OK:\n print(response)\n else:\n print('Request id: %s, Status code: %s, error code: %s, error message: %s' % (\n response.request_id, response.status_code,\n response.code, response.message\n ))\n```\nPara más usos, visite el sitio web oficial.\n<br><br>\n\n## Cuantización\n\n### GPTQ\n\nProporcionamos una solución basada en [AutoGPTQ](https://github.com/PanQiWei/AutoGPTQ), y liberamos los modelos cuantificados Int4 e Int8, que consiguen efectos de modelo casi sin pérdidas pero un rendimiento mejorado tanto en costes de memoria como en velocidad de inferencia.\n\nAquí demostramos cómo utilizar los modelos cuantizados que proporcionamos para la inferencia. Antes de empezar, asegúrese de que cumple los requisitos de auto-gptq (por ejemplo, torch 2.0 y superior, transformers 4.32.0 y superior, etc.) e instale los paquetes necesarios:\n\n```bash\npip install auto-gptq optimum\n```\n\nSi tiene problemas para instalar `auto-gptq`, le aconsejamos que consulte el [repo] oficial (https://github.com/PanQiWei/AutoGPTQ) para encontrar una rueda.\n\n> Nota: Los paquetes `auto-gptq` precompilados dependen en gran medida de la versión de `torch` y de su versión CUDA. Además, debido a la reciente actualización \n> también puede encontrar errores de versión no soportada de `transformers`, `optimum`, o `peft`.\n> Recomendamos utilizar las últimas versiones que cumplan los siguientes requisitos:\n> - torch==2.1 auto-gptq>=0.5.1 transformers>=4.35.0 optimum>=1.14.0 peft>=0.6.1\n> - antorcha>=2.0,<2.1 auto-gptq<0.5.0 transformadores<4.35.0 óptimo<1.14.0 peft>=0.5.0,<0.6.0\n\nA continuación, puede cargar el modelo cuantizado fácilmente y ejecutar la inferencia como de costumbre:\n\n```python\n# Model names: \"Qwen/Qwen-7B-Chat-Int4\", \"Qwen/Qwen-14B-Chat-Int4\"\nmodel = AutoModelForCausalLM.from_pretrained(\n \"Qwen/Qwen-7B-Chat-Int4\",\n device_map=\"auto\",\n trust_remote_code=True\n).eval()\nresponse, history = model.chat(tokenizer, \"Hi\", history=None)\n```\n\nIlustramos el rendimiento de los modelos BF16, Int8 e Int4 en la prueba de referencia, y observamos que el modelo cuantizado no sufre una degradación significativa del rendimiento. Los resultados se muestran a continuación:\n\n| Quantization | MMLU | CEval (val) | GSM8K | Humaneval |\n|----------------------|:----:|:-----------:|:-----:|:---------:|\n| Qwen-1.8B-Chat (BF16)| 43.3 | 55.6 | 33.7 | 26.2 |\n| Qwen-1.8B-Chat (Int8)| 43.1 | 55.8 | 33.0 | 27.4 |\n| Qwen-1.8B-Chat (Int4)| 42.9 | 52.8 | 31.2 | 25.0 |\n| Qwen-7B-Chat (BF16) | 55.8 | 59.7 | 50.3 | 37.2 |\n| Qwen-7B-Chat (Int8) | 55.4 | 59.4 | 48.3 | 34.8 |\n| Qwen-7B-Chat (Int4) | 55.1 | 59.2 | 49.7 | 29.9 |\n| Qwen-14B-Chat (BF16) | 64.6 | 69.8 | 60.1 | 43.9 |\n| Qwen-14B-Chat (Int8) | 63.6 | 68.6 | 60.0 | 48.2 |\n| Qwen-14B-Chat (Int4) | 63.3 | 69.0 | 59.8 | 45.7 |\n| Qwen-72B-Chat (BF16) | 74.4 | 80.1 | 76.4 | 64.6 |\n| Qwen-72B-Chat (Int8) | 73.5 | 80.1 | 73.5 | 62.2 |\n| Qwen-72B-Chat (Int4) | 73.4 | 80.1 | 75.3 | 61.6 |\n\n### Cuantización de la caché KV\n\n> NOTA: Por favor, ten en cuenta que debido al mecanismo interno de Hugging Face, los archivos de soporte para esta funcionalidad\n> (es decir, `cache_autogptq_cuda_256.cpp` y `cache_autogptq_cuda_kernel_256.cu`). \n> Por favor, descárguelos manualmente del Hugging Face Hub y colóquelos en la misma carpeta que los demás archivos del módulo.\n\nLa caché KV de atención puede cuantificarse y comprimirse para su almacenamiento, con el fin de obtener un mayor rendimiento de la muestra. Los argumentos `use_cache_quantization` y `use_cache_kernel` en `config.json` se proporcionan para habilitar la cuantización de la caché KV. \nEl método de uso específico es el siguiente:\n\n```python\nmodel = AutoModelForCausalLM.from_pretrained(\n \"Qwen/Qwen-7B-Chat\",\n device_map=\"auto\",\n trust_remote_code=True,\n use_cache_quantization=True,\n use_cache_kernel=True,\n use_flash_attn=False\n)\n```\nAtención: Actualmente, la cuantización de caché KV y flash attention no se pueden utilizar al mismo tiempo.\nSi habilita la cuantización de caché KV y flash attention al mismo tiempo (`use_flash_attn=True`, `use_cache_quantization=True`, `use_cache_kernel=True`), `use_flash_attn` está deshabilitado por defecto (`use_flash_attn=false`).\n\nHemos comprobado que el uso del modelo int8-kvcache cuantizado no sufre una degradación significativa del rendimiento en la evaluación posterior. A continuación, nos centraremos en el análisis de su huella de memoria en diferentes condiciones. \nEl perfil se ejecuta en una única GPU A100-SXM4-80G con PyTorch 2.0.1 y CUDA 11.4. \nUtilizamos modelos BF16 para generar 1024 tokens por defecto, y \"OOM\" indica error de memoria insuficiente.\n\nCon la cuantización de la caché KV, el modelo puede inferir con un tamaño de lote (bs) mayor.\n\n| Utilizar la caché KV | bs=1 | bs=4 | bs=16 | bs=32 | bs=64 | bs=100 |\n|----------------------|:------:|:------:|:------:|:------:|:------:|:------:|\n| No | 16.3GB | 24.1GB | 31.7GB | 48.7GB | OOM | OOM |\n| Yes | 15.5GB | 17.2GB | 22.3GB | 30.2GB | 48.2GB | 72.4GB |\n\nCon la cuantización kv-cache activada, el modelo puede ahorrar más memoria cuando genera seq-length más largos (sl, número de tokens generados) en infer.\n\n| Utilizar la caché KV | sl=512 | sl=1024 | sl=2048 | sl=4096 | sl=8192 |\n|----------------------|:------:|:-------:|:-------:|:-------:|:-------:|\n| No | 15.2GB | 16.3GB | 17.6GB | 19.5GB | 23.2GB |\n| Yes | 15GB | 15.5GB | 15.8GB | 16.6GB | 17.6GB |\n\nEl modelo con cuantificación de caché KV convertirá el formato de `layer_past` de float a int8, y mientras tanto el `layer-past` cuantificado también almacenará los parámetros de cuantificación.\n\nLos pasos específicos son los siguientes\n\n1. Cuantificar clave/valor\n```\n qv,scale,zero_point=quantize_cache_v(v)\n```\n2. Almacenar en layer_past\n\nA continuación se muestra el formato de `layer_past` cuantificado:\n```\n layer_past=((q_key,key_scale,key_zero_point),\n (q_value,value_scale,value_zero_point))\n```\nA continuación se muestra el formato original de `layer_past`:\n```\n layer_past=(key,value)\n```\nSi desea utilizar la atención KV que se cuantiza, \npuede utilizar la operación de decuantización para convertir la clave/valor int8 de nuevo al formato float de la siguiente manera:\n```\n v=dequantize_cache_torch(qv,scale,zero_point)\n```\n<br>\n\n\n## Rendimiento de Inferencia\n\nEsta sección proporciona las estadísticas de velocidad y memoria de los modelos en diferentes precisiones. Los perfiles de velocidad y memoria se realizan utilizando [este script](https://qianwen-res.oss-cn-beijing.aliyuncs.com/profile.py).\n\nMedimos la velocidad media de inferencia (tokens/s) y el uso de memoria de la GPU al generar 2048 con los modelos en BF16, Int8 e Int4.\n\n<table>\n <tr>\n <td>Model Size</td>\n <td>Quantization</td>\n <td>Speed (Tokens/s)</td>\n <td>GPU Memory Usage</td>\n </tr>\n <tr>\n <td rowspan=\"3\">1.8B</td>\n <td>BF16</td>\n <td>54.09</td>\n <td>4.23GB</td>\n </tr>\n <tr>\n <td>Int8</td>\n <td>55.56</td>\n <td>3.48GB</td>\n </tr>\n <tr>\n <td>Int4</td>\n <td>71.07</td>\n <td>2.91GB</td>\n </tr>\n <tr>\n <td rowspan=\"3\">7B</td>\n <td>BF16</td>\n <td>40.93</td>\n <td>16.99GB</td>\n </tr>\n <tr>\n <td>Int8</td>\n <td>37.47</td>\n <td>11.20GB</td>\n </tr>\n <tr>\n <td>Int4</td>\n <td>50.09</td>\n <td>8.21GB</td>\n </tr>\n <tr>\n <td rowspan=\"3\">14B</td>\n <td>BF16</td>\n <td>32.22</td>\n <td>30.15GB</td>\n </tr>\n <tr>\n <td>Int8</td>\n <td>29.28</td>\n <td>18.81GB</td>\n </tr>\n <tr>\n <td>Int4</td>\n <td>38.72</td>\n <td>13.01GB</td>\n </tr>\n <tr>\n <td rowspan=\"3\">72B</td>\n <td>BF16</td>\n <td>8.48</td>\n <td>144.69GB (2xA100)</td>\n </tr>\n <tr>\n <td>Int8</td>\n <td>9.05</td>\n <td>81.27GB (2xA100)</td>\n </tr>\n <tr>\n <td>Int4</td>\n <td>11.32</td>\n <td>48.86GB</td>\n </tr>\n <tr>\n <td>72B + vLLM</td>\n <td>BF16</td>\n <td>17.60</td>\n <td>2xA100</td>\n </tr>\n</table>\n\nEl perfil se ejecuta en una única GPU A100-SXM4-80G (salvo que se mencione 2xA100) con PyTorch 2.0.1, CUDA 11.8 y Flash-Attention 2. (72B + vLLM utiliza PyTorch 2.1.0 y Cuda 11.8.) La velocidad de inferencia se promedia sobre los tokens codificados y generados.\n\nNota: La velocidad de generación de los modelos Int4/Int8 mencionados anteriormente es proporcionada por la librería autogptq. La velocidad actual del modelo cargado utilizando ``AutoModelForCausalLM.from_pretrained`` será aproximadamente un 20% más lenta. Hemos informado de este problema al equipo de HuggingFace y lo actualizaremos rápidamente si se encuentra una solución.\n\nTambién medimos la velocidad de inferencia y el uso de memoria de la GPU con diferentes configuraciones de contexto y longitudes de generación, versión Flash-Attention. Puedes encontrar los resultados en las modelcards correspondientes en Hugging Face o ModelScope.\n\n\n\n## Finetuning\n\n### Utilización\nAhora proporcionamos el script de entrenamiento oficial, `finetune.py`, para que los usuarios puedan ajustar el modelo preentrenado para aplicaciones posteriores de forma sencilla. Además, proporcionamos scripts de shell para lanzar el ajuste fino sin preocupaciones. Este script soporta el entrenamiento con [DeepSpeed](https://github.com/microsoft/DeepSpeed) y [FSDP](https://engineering.fb.com/2021/07/15/open-source/fsdp/). Los shell scripts que proporcionamos utilizan DeepSpeed (Nota: esto puede tener conflictos con la última versión de pydantic y debe utilizar make sure `pydantic<2.0`) y Peft. Puede instalarlos de la siguiente manera:\n```bash\npip install \"peft<0.8.0\" deepspeed\n```\n\nPara preparar tus datos de entrenamiento, necesitas poner todas las muestras en una lista y guardarla en un archivo json. Cada muestra es un diccionario que consiste en un id y una lista para la conversación. A continuación se muestra una lista de ejemplo simple con 1 muestra:\n```json\n[\n {\n \"id\": \"identity_0\",\n \"conversations\": [\n {\n \"from\": \"user\",\n \"value\": \"你好\"\n },\n {\n \"from\": \"assistant\",\n \"value\": \"我是一个语言模型，我叫通义千问。\"\n }\n ]\n }\n]\n```\n\nUna vez preparados los datos, puede utilizar los scripts de shell suministrados para ejecutar el ajuste fino. Recuerde especificar la ruta al archivo de datos, `$DATA`.\n\nLos guiones de finetuning permiten realizar:\n- Finetuning de todos los parámetros\n- LoRA\n- Q-LoRA\n\nFull-parameter finetuning requires updating all parameters in the whole training process. To launch your training, run the following script:\n\n```bash\n# Entrenamiento distribuido. No proporcionamos un script de entrenamiento para una sola GPU, ya que la insuficiente memoria de la GPU interrumpiría el entrenamiento.\nbash finetune/finetune_ds.sh\n```\n\nRecuerde especificar el nombre correcto del modelo o ruta, la ruta de datos, así como el directorio de salida en los scripts de shell. Otra cosa a notar es que usamos DeepSpeed ZeRO 3 en este script. Si desea realizar cambios, basta con eliminar el argumento `--deepspeed` o realizar cambios en el archivo json de configuración de DeepSpeed en función de sus necesidades. Además, este script soporta entrenamiento de precisión mixta, por lo que puedes usar `--bf16 True` o `--fp16 True`. Recuerde utilizar DeepSpeed cuando utilice fp16 debido al entrenamiento de precisión mixta. \nEmpíricamente le aconsejamos que utilice bf16 para que su entrenamiento sea coherente con nuestro preentrenamiento y alineación si su máquina soporta bf16, y por lo tanto lo utilizamos por defecto.\n\nPara ejecutar LoRA, utilice otro script para ejecutar como se muestra a continuación. Antes de empezar, asegúrese de que ha instalado `peft`. Además, es necesario especificar las rutas a su modelo, los datos y la salida. Le aconsejamos que utilice la ruta absoluta para su modelo pre-entrenado. Esto se debe a que LoRA sólo guarda el adaptador y la ruta absoluta en el archivo json de configuración del adaptador se utiliza para encontrar el modelo preentrenado para cargar. Además, este script soporta tanto bf16 como fp16.\n\n```bash\n# Single GPU training\nbash finetune/finetune_lora_single_gpu.sh\n# Distributed training\nbash finetune/finetune_lora_ds.sh\n```\n\nEn comparación con el ajuste fino de parámetros completos, LoRA ([artículo](https://arxiv.org/abs/2106.09685)) sólo actualiza los parámetros de las capas adaptadoras, pero mantiene congeladas las grandes capas originales del modelo de lenguaje. Esto permite muchos menos costes de memoria y, por tanto, de computación.\n\nTenga en cuenta que si utiliza LoRA para ajustar el modelo de lenguaje base, por ejemplo, Qwen-7B, en lugar de los modelos de chat, por ejemplo, Qwen-7B-Chat, el script cambia automáticamente la incrustación y la capa de salida como parámetros entrenables. Esto se debe a que el modelo de lenguaje base no tiene conocimiento de los tokens especiales que aporta el formato ChatML. Por lo tanto, estas capas deben actualizarse para que el modelo comprenda y prediga los tokens. O en otras palabras, si tu entrenamiento trae tokens especiales en LoRA, deberías poner las capas como parámetros entrenables poniendo `modules_to_save` dentro del código. Además, si tenemos estos parámetros entrenables, no está disponible para usar ZeRO 3, y es por esto que usamos ZeRO 2 en el script por defecto. Si no tenemos nuevos parámetros entrenables, podemos cambiar a ZeRO 3 cambiando el fichero de configuración de DeepSpeed. Además, encontramos que hay una brecha significativa entre la huella de memoria de LoRA con y sin estos parámetros entrenables. Por lo tanto, si usted tiene problemas con la memoria, le aconsejamos LoRA finetune los modelos de chat. Compruebe el perfil de abajo para obtener más información.\n\nSi sigue sufriendo de memoria insuficiente, puede considerar Q-LoRA ([artículo](https://arxiv.org/abs/2305.14314)), que utiliza el modelo de lenguaje cuantizado de gran tamaño y otras técnicas como la atención paginada para permitir incluso menos costes de memoria.\n\nNota: para ejecutar el entrenamiento Q-LoRA con una sola GPU, puede que necesites instalar `mpi4py` a través de `pip` o `conda`.\n\nPara ejecutar Q-LoRA, ejecute directamente el siguiente script:\n\n```bash\n# Entrenamiento con una sola GPU\nbash finetune/finetune_qlora_single_gpu.sh\n# Entrenamiento distribuida\nbash finetune/finetune_qlora_ds.sh\n```\n\nPara Q-LoRA, le aconsejamos que cargue nuestro modelo cuantizado proporcionado, por ejemplo, Qwen-7B-Chat-Int4. **NO DEBE** utilizar los modelos bf16. A diferencia del finetuning de parámetros completos y LoRA, sólo fp16 es compatible con Q-LoRA. Para el entrenamiento con una sola GPU, tenemos que utilizar DeepSpeed para el entrenamiento de precisión mixta debido a nuestra observación de errores causados por el amplificador de antorcha. Además, para Q-LoRA, los problemas con los tokens especiales en LoRA siguen existiendo. Sin embargo, como sólo proporcionamos los modelos Int4 para los modelos de chat, lo que significa que el modelo lingüístico ha aprendido los tokens especiales del formato ChatML, no hay que preocuparse por las capas. Ten en cuenta que las capas del modelo Int4 no deben ser entrenables, por lo que si introduces tokens especiales en tu entrenamiento, Q-LoRA podría no funcionar.\n\n> NOTA: Tenga en cuenta que debido a los mecanismos internos de Hugging Face, ciertos archivos que no son de Python (por ejemplo, `*.cpp` y `*.cu`) pueden faltar en el punto de control guardado. \n> pueden faltar en el punto de control guardado. Es posible que tenga que copiarlos manualmente en el directorio que contiene otros archivos.\n\nA diferencia del finetuning de parámetros completo, el entrenamiento de LoRA y Q-LoRA sólo guarda los parámetros del adaptador. Supongamos que su entrenamiento comienza desde Qwen-7B, puede cargar el modelo ajustado para la inferencia como se muestra a continuación:\n\n```python\nfrom peft import AutoPeftModelForCausalLM\n\nmodel = AutoPeftModelForCausalLM.from_pretrained(\n path_to_adapter, # path to the output directory\n device_map=\"auto\",\n trust_remote_code=True\n).eval()\n```\n\nSi quieres fusionar los adaptadores y guardar el modelo ajustado como un modelo independiente (sólo puedes hacer esto con LoRA, y NO puedes fusionar los parámetros desde Q-LoRA), puedes ejecutar los siguientes códigos:\n\n```python\nfrom peft import AutoPeftModelForCausalLM\n\nmodel = AutoPeftModelForCausalLM.from_pretrained(\n path_to_adapter, # path to the output directory\n device_map=\"auto\",\n trust_remote_code=True\n).eval()\n\nmerged_model = model.merge_and_unload()\n# max_shard_size and safe serialization are not necessary. \n# They respectively work for sharding checkpoint and save the model to safetensors\nmerged_model.save_pretrained(new_model_directory, max_shard_size=\"2048MB\", safe_serialization=True)\n```\n\nNota: Para el entrenamiento multi-GPU, es necesario especificar los hiperparámetros adecuados para el entrenamiento distribuido basado en su máquina. Además, le aconsejamos que especifique la longitud máxima de la secuencia con el argumento `--model_max_length`, en función de los datos, el espacio de memoria y la velocidad de entrenamiento.\n\n\n### Perfiles de Memoria y Velocidad\nPerfilamos la memoria de la GPU y la velocidad de entrenamiento tanto de LoRA (LoRA (emb) se refiere al entrenamiento de la capa de incrustación y salida, mientras que LoRA no tiene capa de incrustación y salida entrenables) como de Q-LoRA en la configuración de entrenamiento en una sola GPU. En esta prueba, experimentamos con una única GPU A100-SXM4-80G, y utilizamos CUDA 11.8 y Pytorch 2.0. Se aplica Flash attention 2. Utilizamos uniformemente un tamaño de lote de 1 y una acumulación de gradiente de 8. Perfilamos la memoria (GB) y la velocidad (s/iter) de entradas de distintas longitudes, a saber, 256, 512, 1024, 2048, 4096 y 8192. También presentamos las estadísticas del ajuste fino de todos los parámetros con Qwen-7B en 2 GPU A100. Sólo se presentan las estadísticas de 256, 512 y 1024 tokens debido a la limitación de memoria de la GPU. \n\nPara Qwen-72B, experimentamos de dos formas: 1) Ajuste fino de Lora + DeepSpeed ZeRO 3 en 4 GPUs A100-SXM4-80G y 2) Ajuste fino de QLora (int4) en una sola GPU A100-SXM4-80G. Ten en cuenta que la OOM se produce en 4 GPUs A100-SXM4-80G tanto con ajuste fino LoRA (emb) como con ajuste fino LoRA sin Deepspeed ZeRO 3 (puedes pasar `--deepspeed finetune/ds_config_zero3.json` a [`finetune/finetune_lora_ds.sh`](finetune/finetune_lora_ds.sh) para activar DeepSpeed ZeRO 3).\n\nLas estadísticas se enumeran a continuación:\n\n<table>\n <tr>\n <th rowspan=\"2\">Model Size</th><th rowspan=\"2\">Method</th><th colspan=\"6\" align=\"center\">Sequence Length</th>\n </tr>\n <tr>\n <th align=\"center\">256</th><th align=\"center\">512</th><th align=\"center\">1024</th><th align=\"center\">2048</th><th align=\"center\">4096</th><th align=\"center\">8192</th>\n </tr>\n </tr>\n </tr>\n\t\t<tr>\n <th rowspan=\"4\">1.8B</th><td>LoRA</td><td align=\"center\">6.7G / 1.0s/it</td><td align=\"center\">7.4G / 1.0s/it</td><td align=\"center\">8.4G / 1.1s/it</td><td align=\"center\">11.0G / 1.7s/it</td><td align=\"center\">16.2G / 3.3s/it</td><td align=\"center\">21.8G / 6.8s/it</td>\n </tr>\n <tr>\n <td>LoRA (emb)</td><td align=\"center\">13.7G / 1.0s/it</td><td align=\"center\">14.0G / 1.0s/it</td><td align=\"center\">14.0G / 1.1s/it</td><td align=\"center\">15.1G / 1.8s/it</td><td align=\"center\">19.7G / 3.4s/it</td><td align=\"center\">27.7G / 7.0s/it</td>\n </tr>\n <tr>\n <td>Q-LoRA</td><td align=\"center\">5.8G / 1.4s/it</td><td align=\"center\">6.0G / 1.4s/it</td><td align=\"center\">6.6G / 1.4s/it</td><td align=\"center\">7.8G / 2.0s/it</td><td align=\"center\">10.2G / 3.4s/it</td><td align=\"center\">15.8G / 6.5s/it</td>\n </tr>\n <tr>\n <td>Full-parameter</td><td align=\"center\">43.5G / 2.1s/it</td><td align=\"center\">43.5G / 2.2s/it</td><td align=\"center\">43.5G / 2.2s/it</td><td align=\"center\">43.5G / 2.3s/it</td><td align=\"center\">47.1G / 2.8s/it</td><td align=\"center\">48.3G / 5.6s/it</td>\n </tr>\n <tr>\n <th rowspan=\"4\">7B</th><td>LoRA</td><td align=\"center\">20.1G / 1.2s/it</td><td align=\"center\">20.4G / 1.5s/it</td><td align=\"center\">21.5G / 2.8s/it</td><td align=\"center\">23.8G / 5.2s/it</td><td align=\"center\">29.7G / 10.1s/it</td><td align=\"center\">36.6G / 21.3s/it</td>\n </tr>\n <tr>\n <td>LoRA (emb)</td><td align=\"center\">33.7G / 1.4s/it</td><td align=\"center\">34.1G / 1.6s/it</td><td align=\"center\">35.2G / 2.9s/it</td><td align=\"center\">35.1G / 5.3s/it</td><td align=\"center\">39.2G / 10.3s/it</td><td align=\"center\">48.5G / 21.7s/it</td>\n </tr>\n <tr>\n <td>Q-LoRA</td><td align=\"center\">11.5G / 3.0s/it</td><td align=\"center\">11.5G / 3.0s/it</td><td align=\"center\">12.3G / 3.5s/it</td><td align=\"center\">13.9G / 7.0s/it</td><td align=\"center\">16.9G / 11.6s/it</td><td align=\"center\">23.5G / 22.3s/it</td>\n </tr>\n <tr>\n <td>Full-parameter</td><td align=\"center\">139.2G / 4.0s/it</td><td align=\"center\">148.0G / 4.0s/it</td><td align=\"center\">162.0G / 4.5s/it</td><td align=\"center\">-</td><td align=\"center\">-</td><td align=\"center\">-</td>\n </tr>\n <tr>\n <th rowspan=\"3\">14B</th><td>LoRA</td><td align=\"center\">34.6G / 1.6s/it</td><td align=\"center\">35.1G / 2.4s/it</td><td align=\"center\">35.3G / 4.4s/it</td><td align=\"center\">37.4G / 8.4s/it</td><td align=\"center\">42.5G / 17.0s/it</td><td align=\"center\">55.2G / 36.0s/it</td>\n </tr>\n <tr>\n <td>LoRA (emb)</td><td align=\"center\">51.2 / 1.7s/it</td><td align=\"center\">51.1G / 2.6s/it</td><td align=\"center\">51.5G / 4.6s/it</td><td align=\"center\">54.1G / 8.6s/it</td><td align=\"center\">56.8G / 17.2s/it</td><td align=\"center\">67.7G / 36.3s/it</td>\n </tr>\n <tr>\n <td>Q-LoRA</td><td align=\"center\">18.7G / 5.3s/it</td><td align=\"center\">18.4G / 6.3s/it</td><td align=\"center\">18.9G / 8.2s/it</td><td align=\"center\">19.9G / 11.8s/it</td><td align=\"center\">23.0G / 20.1s/it</td><td align=\"center\">27.9G / 38.3s/it</td>\n </tr>\n\t<tr>\n <th rowspan=\"2\">72B</th><td>LoRA + Deepspeed Zero3</td><td align=\"center\">215.4G / 17.6s/it</td><td align=\"center\">217.7G / 20.5s/it</td><td align=\"center\">222.6G / 29.4s/it</td><td align=\"center\">228.8G / 45.7s/it</td><td align=\"center\">249.0G / 83.4s/it</td><td align=\"center\">289.2G / 161.5s/it</td>\n </tr>\n <tr>\n <td>Q-LoRA</td><td align=\"center\">61.4G / 27.4s/it</td><td align=\"center\">61.4G / 31.5s/it</td><td align=\"center\">62.9G / 41.4s/it</td><td align=\"center\">64.1G / 59.5s/it</td><td align=\"center\">68.0G / 97.7s/it</td><td align=\"center\">75.6G / 179.8s/it</td>\n </tr>\n</table>\n<br>\n\n## Despliegue\n\n### vLLM \nPara el despliegue y la inferencia rápida, sugerimos utilizar vLLM con FastChat. Instale primero los paquetes:\n```bash\npip install vllm fastchat\n```\nO puede instalarlos desde el código fuente mediante `git clone` y `pip install -e .`. Le aconsejamos que lea sus documentos si encuentra problemas en la instalación.\n\nPara ejecutar Qwen con vLLM y FastChat, primero necesitas lanzar un controlador por:\n```bash\npython -m fastchat.serve.controller\n```\n\nA continuación, puede iniciar el model worker, lo que significa cargar su modelo para la inferencia. Para la inferencia de una sola GPU, puede ejecutar directamente:\n\n```bash\npython -m fastchat.serve.vllm_worker --model-path $model_path --trust-remote-code\n```\nSin embargo, si desea ejecutar el modelo en varias GPU para acelerar la inferencia o disponer de más memoria, puede utilizar el paralelismo tensorial soportado por vLLM. Supongamos que ejecutas el modelo en 4 GPUs, el comando se muestra a continuación:\n```bash\npython -m fastchat.serve.vllm_worker --model-path $model_path --trust-remote-code --tensor-parallel-size 4\n```\n\nDespués de lanzar tu model worker, puedes lanzar:\n\n* Web UI Demo\n```bash\npython -m fastchat.serve.gradio_web_server\n```\n\n* API OpenAI\n```bash\npython -m fastchat.serve.openai_api_server --host localhost --port 8000\n```\n\n### Interfaz Web\n\nProporcionamos código para que los usuarios construyan una web UI demo (gracias a @wysaid). Antes de empezar, asegúrate de instalar los siguientes paquetes:\n```\npip install -r requirements_web_demo.txt\n```\n\nA continuación, ejecute el siguiente comando y haga clic en el enlace generado:\n\n```bash\npython web_demo.py\n```\n\n<p align=\"center\">\n <br>\n <img src=\"assets/web_demo.gif\" width=\"600\" />\n <br>\n<p>\n\nSin embargo, si le resulta difícil utilizar vLLM y FastChat, puede probar los métodos más sencillos que le proporcionamos para desplegar una demo web, una demo CLI y una API.\n\n### Demo CLI\n\nProporcionamos un ejemplo de demostración CLI en `cli_demo.py`, que soporta la salida de streaming para la generación. Los usuarios pueden interactuar con Qwen-7B-Chat introduciendo mensajes, y el modelo devuelve los resultados del modelo en modo streaming. Ejecute el siguiente comando:\n\n```bash\npython cli_demo.py\n```\n\n<p align=\"center\">\n <br>\n <img src=\"assets/cli_demo.gif\" width=\"600\" />\n <br>\n<p>\n<br>\n\n### API\n\nProporcionamos métodos para desplegar la API local basada en la API de OpenAI (gracias a @hanpenggit). Antes de empezar, instala los paquetes necesarios:\n\n```bash\npip install fastapi uvicorn \"openai<1.0\" pydantic sse_starlette\n```\n\nA continuación, ejecute el comando para desplegar su API:\n\n```bash\npython openai_api.py\n```\n\nPuede cambiar sus argumentos, por ejemplo, `-c` para el nombre o la ruta del punto de control, `--cpu-only` para el despliegue en CPU, etc. Si tienes problemas al iniciar el despliegue de tu API, probablemente puedas solucionarlos actualizando los paquetes a la última versión.\n\nUtilizar la API también es sencillo. Vea el siguiente ejemplo:\n\n```python\nimport openai\nopenai.api_base = \"http://localhost:8000/v1\"\nopenai.api_key = \"none\"\n\n# create a request activating streaming response\nfor chunk in openai.ChatCompletion.create(\n model=\"Qwen\",\n messages=[\n {\"role\": \"user\", \"content\": \"你好\"}\n ],\n stream=True \n # Specifying stop words in streaming output format is not yet supported and is under development.\n):\n if hasattr(chunk.choices[0].delta, \"content\"):\n print(chunk.choices[0].delta.content, end=\"\", flush=True)\n\n# create a request not activating streaming response\nresponse = openai.ChatCompletion.create(\n model=\"Qwen\",\n messages=[\n {\"role\": \"user\", \"content\": \"你好\"}\n ],\n stream=False,\n stop=[] # You can add custom stop words here, e.g., stop=[\"Observation:\"] for ReAct prompting.\n)\nprint(response.choices[0].message.content)\n```\n\n<p align=\"center\">\n <br>\n <img src=\"assets/openai_api.gif\" width=\"600\" />\n <br>\n<p>\n\n**Function calling** también está soportada (pero sólo cuando `stream=False` por el momento). Ver el [ejemplo de uso](examples/function_call_examples.py) aquí.\n<br><br>\n\n## 🐳 Docker\n\nPara simplificar el proceso de despliegue, proporcionamos imágenes Docker con entornos preconstruidos: [qwenllm/qwen](https://hub.docker.com/r/qwenllm/qwen). Solo tienes que instalar el controlador y descargar los archivos del modelo para lanzar demos, desplegar la API de OpenAI y ajustar el modelo.\n\n### Preparación\n\n1. Instale la versión correcta del controlador Nvidia en función de la imagen que vaya a utilizar:\n - `qwenllm/qwen:cu117` (**recomendado**): `>= 515.48.07`\n - `qwenllm/qwen:cu114` (w/o flash-attention): `>= 470.82.01`\n - `qwenllm/qwen:cu121`: `>= 530.30.02`\n - `qwenllm/qwen:latest`: igual que `qwenllm/qwen:cu117`\n\n2. Instale y configure [docker](https://docs.docker.com/engine/install/) y [nvidia-container-toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html):\n\n```bash\n# configure docker\nsudo systemctl start docker\n# test if docker is correctly installed\nsudo docker run hello-world\n\n# configure nvidia-container-toolkit\nsudo nvidia-ctk runtime configure --runtime=docker\nsudo systemctl restart docker\n# test if nvidia-container-toolkit is correctly installed\nsudo docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi\n```\n\n3. Descargue los checkpoints y los códigos del modelo a su entorno (véase [aquí](#DownloadModel)).\n\n### Despliegue\n\nAquí usamos Qwen-7B-Chat como ejemplo. Antes de lanzar una demo web o API, puede establecer la configuración como se muestra a continuación:\n\n```bash\nIMAGE_NAME=qwenllm/qwen:cu117\nPORT=8901\nCHECKPOINT_PATH=/path/to/Qwen-7B-Chat # Path to downloaded model checkpoints and codes\n```\nLos siguientes scripts pueden ayudarte a construir:\n\n* API OpenAI\n```bash\nbash docker/docker_openai_api.sh -i ${IMAGE_NAME} -c ${CHECKPOINT_PATH} --port ${PORT}\n```\n\n* Interfaz Web\n```bash\nbash docker/docker_web_demo.sh -i ${IMAGE_NAME} -c ${CHECKPOINT_PATH} --port ${PORT}\n```\n\n* Demo CLI\n```bash\nbash docker/docker_cli_demo.sh -i ${IMAGE_NAME} -c ${CHECKPOINT_PATH}\n```\n\nLos comandos anteriores descargarán automáticamente la imagen requerida y lanzarán una demo Web UI en segundo plano (el servicio se reiniciará automáticamente). Puede abrir `http://localhost:${PORT}` en el host para utilizar la demo.\n\nLa demostración se ha iniciado correctamente si ve la siguiente salida:\n\n```text\nSuccessfully started web demo. Open '...' to try!\nRun `docker logs ...` to check demo status.\nRun `docker rm -f ...` to stop and remove the demo.\n```\n\nSi quieres comprobar el estado de la demo, puedes usar `docker logs qwen` para mostrar los resultados.\n\nPuede utilizar `docker rm -f qwen` para detener el servicio y eliminar el contenedor.\n\n\n### Finetuning\n\nEl método de finetuning utilizando la imagen Docker pre-construida es básicamente el mismo que [el capítulo anterior](#Finetuning) (ya hemos instalado dependencias en la imagen):\n\nA continuación se muestra un ejemplo de LoRA de GPU única:\n```bash\nIMAGE_NAME=qwenllm/qwen:cu117\nCHECKPOINT_PATH=/path/to/Qwen-7B # Path to downloaded model checkpoints and codes\n#CHECKPOINT_PATH=/path/to/Qwen-7B-Chat-Int4 # Path to downloaded model checkpoints and codes (Q-LoRA)\nDATA_PATH=/path/to/data/root # Prepare finetune data at ${DATA_PATH}/example.json\nOUTPUT_PATH=/path/to/output/checkpoint # Path to finetune outputs\n\n# Use all host devices by default\nDEVICE=all\n# If you need to specify GPUs for training, set device as follow (NOTE: internal quotation marks cannot be omitted)\n#DEVICE='\"device=0,1,2,3\"'\n\nmkdir -p ${OUTPUT_PATH}\n\n# Single-GPU LoRA finetuning\ndocker run --gpus ${DEVICE} --rm --name qwen \\\n --mount type=bind,source=${CHECKPOINT_PATH},target=/data/shared/Qwen/Qwen-7B \\\n --mount type=bind,source=${DATA_PATH},target=/data/shared/Qwen/data \\\n --mount type=bind,source=${OUTPUT_PATH},target=/data/shared/Qwen/output_qwen \\\n --shm-size=2gb \\\n -it ${IMAGE_NAME} \\\n bash finetune/finetune_lora_single_gpu.sh -m /data/shared/Qwen/Qwen-7B/ -d /data/shared/Qwen/data/example.json\n```\n\nPara realizar un cambio a Q-LoRA de una sola GPU, por ejemplo, basta con modificar el comando bash dentro de `docker run`:\n```bash\nbash finetune/finetune_qlora_single_gpu.sh -m /data/shared/Qwen/Qwen-7B-Chat-Int4/ -d /data/shared/Qwen/data/example.json\n```\n<br>\n\n## 🔥 Indicaciones del sistema\nQwen-1.8-Chat y Qwen-72B-Chat han sido completamente entrenados en diversas indicaciones del sistema con múltiples rondas de interacciones complejas, para que puedan seguir una variedad de indicaciones del sistema y realizar la personalización del modelo en contexto, mejorando aún más la escalabilidad de Qwen-chat.\n\nGracias a las instrucciones del sistema, Qwen-Chat puede realizar **juegos de rol**, **transferencia de estilos de lenguaje**, **configuración de tareas** y **configuración de comportamientos**.\n\n![](assets/system_prompt_language_style.png)\n\n![](assets/system_prompt_role_play_en.png)\n\nPara más información, consulta la [documentación de ejemplo](examples/system_prompt.md).\n\n\n## Uso de Herramientas\n\nQwen-Chat ha sido optimizado para el uso de herramientas y capacidades de llamada a funciones. Los usuarios pueden desarrollar agentes, aplicaciones LangChain e incluso aumentar Qwen con un intérprete de código Python.\n\nProporcionamos documentación sobre cómo implementar llamadas a herramientas basadas en el principio de ReAct Prompting, por favor consulte [the ReAct example](examples/react_prompt.md). Basándonos en este principio, proporcionamos soporte para llamadas a funciones en [openai_api.py](openai_api.py).\n\nHemos probado las capacidades de llamada de la herramienta del modelo en nuestro punto de referencia de evaluación chino de código abierto y hemos descubierto que Qwen-Chat obtiene siempre buenos resultados:\n\n<table>\n <tr>\n <th colspan=\"4\" align=\"center\">Chinese Tool-Use Benchmark (Version 20231206)</th>\n </tr>\n <tr>\n <th align=\"center\">Model</th><th align=\"center\">Tool Selection (Acc.↑)</th><th align=\"center\">Tool Input (Rouge-L↑)</th><th align=\"center\">False Positive Error↓</th>\n </tr>\n <tr>\n <td>GPT-4</td><td align=\"center\">98.0%</td><td align=\"center\">0.953</td><td align=\"center\">23.9%</td>\n </tr>\n <tr>\n <td>GPT-3.5</td><td align=\"center\">74.5%</td><td align=\"center\">0.807</td><td align=\"center\">80.6%</td>\n </tr>\n <tr>\n <td>Qwen-1_8B-Chat</td><td align=\"center\">85.0%</td><td align=\"center\">0.839</td><td align=\"center\">27.6%</td>\n </tr>\n <tr>\n <td>Qwen-7B-Chat</td><td align=\"center\">95.5%</td><td align=\"center\">0.900</td><td align=\"center\">11.6%</td>\n </tr>\n <tr>\n <td>Qwen-14B-Chat</td><td align=\"center\">96.9%</td><td align=\"center\">0.917</td><td align=\"center\">5.6%</td>\n </tr>\n <tr>\n <td>Qwen-72B-Chat</td><td align=\"center\">98.2%</td><td align=\"center\">0.927</td><td align=\"center\">1.1%</td>\n </tr>\n</table>\n\nPara evaluar la capacidad de Qwen para utilizar el intérprete de código Python en tareas como la resolución de problemas matemáticos, la visualización de datos y otras tareas de propósito general como el manejo de archivos y el web scraping, hemos creado y puesto a disposición del público un benchmark específicamente diseñado para evaluar estas capacidades. Puede encontrar el punto de referencia en este [enlace](https://github.com/QwenLM/Qwen-Agent/tree/main/benchmark).\n\nHemos observado que Qwen funciona bien en términos de ejecutabilidad del código y precisión de los resultados al generar código:\n\n<table>\n <tr>\n <th colspan=\"5\" align=\"center\">Code Interpreter Benchmark (Version 20231206)</th>\n </tr>\n <tr>\n <th rowspan=\"2\" align=\"center\">Model</th>\n <th colspan=\"3\" align=\"center\">Accuracy of Code Execution Results (%)</th>\n <th colspan=\"1\" align=\"center\">Executable Rate of Code (%)</th>\n </tr>\n <tr>\n <th align=\"center\">Math↑</th><th align=\"center\">Visualization-Hard↑</th><th align=\"center\">Visualization-Easy↑</th><th align=\"center\">General↑</th>\n </tr>\n <tr>\n <td>GPT-4</td>\n <td align=\"center\">82.8</td>\n <td align=\"center\">66.7</td>\n <td align=\"center\">60.8</td>\n <td align=\"center\">82.8</td>\n </tr>\n <tr>\n <td>GPT-3.5</td>\n <td align=\"center\">47.3</td>\n <td align=\"center\">33.3</td>\n <td align=\"center\">55.7</td>\n <td align=\"center\">74.1</td>\n </tr>\n <tr>\n <td>LLaMA2-13B-Chat</td>\n <td align=\"center\">8.3</td>\n <td align=\"center\">1.2</td>\n <td align=\"center\">15.2</td>\n <td align=\"center\">48.3</td>\n </tr>\n <tr>\n <td>CodeLLaMA-13B-Instruct</td>\n <td align=\"center\">28.2</td>\n <td align=\"center\">15.5</td>\n <td align=\"center\">21.5</td>\n <td align=\"center\">74.1</td>\n </tr>\n <tr>\n <td>InternLM-20B-Chat</td>\n <td align=\"center\">34.6</td>\n <td align=\"center\">10.7</td>\n <td align=\"center\">25.1</td>\n <td align=\"center\">65.5</td>\n </tr>\n <tr>\n <td>ChatGLM3-6B</td>\n <td align=\"center\">54.2</td>\n <td align=\"center\">4.8</td>\n <td align=\"center\">15.2</td>\n <td align=\"center\">67.1</td>\n </tr>\n <tr>\n <td>Qwen-1.8B-Chat</td>\n <td align=\"center\">25.6</td>\n <td align=\"center\">21.4</td>\n <td align=\"center\">22.8</td>\n <td align=\"center\">65.5</td>\n </tr>\n <tr>\n <td>Qwen-7B-Chat</td>\n <td align=\"center\">41.9</td>\n <td align=\"center\">23.8</td>\n <td align=\"center\">38.0</td>\n <td align=\"center\">67.2</td>\n </tr>\n <tr>\n <td>Qwen-14B-Chat</td>\n <td align=\"center\">58.4</td>\n <td align=\"center\">31.0</td>\n <td align=\"center\">45.6</td>\n <td align=\"center\">65.5</td>\n </tr>\n <tr>\n <td>Qwen-72B-Chat</td>\n <td align=\"center\">72.7</td>\n <td align=\"center\">41.7</td>\n <td align=\"center\">43.0</td>\n <td align=\"center\">82.8</td>\n </tr>\n</table>\n\n<p align=\"center\">\n <br>\n <img src=\"assets/code_interpreter_showcase_001.jpg\" />\n <br>\n<p>\n\n<br>\n\n## Comprensión del Contexto Largo\n\nPara ampliar la longitud del contexto y romper el cuello de botella de la longitud de la secuencia de entrenamiento, introducimos varias técnicas, como la interpolación NTK, la atención de ventana y el escalado de atención LogN, para ampliar la longitud del contexto de Qwen-14B de 2K a más de 8K tokens, y Qwen-1.8B/7B de 8K a 32K tokens. \n\nPara Qwen-72B, adaptamos RoPE a contextos más largos con una base rotatoria mayor. Qwen-72B admite una longitud máxima de contexto de 32K tokens.\n\nRealizamos experimentos de modelado lingüístico en el conjunto de datos arXiv con la evaluación PPL y descubrimos que Qwen puede alcanzar un rendimiento sobresaliente en el escenario de contextos largos. Los resultados se muestran a continuación:\n\n<table>\n <tr>\n <th rowspan=\"2\">Model</th><th colspan=\"6\" align=\"center\">Sequence Length</th>\n </tr>\n <tr>\n <th align=\"center\">1024</th><th align=\"center\">2048</th><th align=\"center\">4096</th><th align=\"center\">8192</th><th align=\"center\">16384</th><th align=\"center\">32768</th>\n </tr>\n <tr>\n <td>Qwen-7B (original)</td><td align=\"center\">4.23</td><td align=\"center\">3.78</td><td align=\"center\">39.35</td><td align=\"center\">469.81</td><td align=\"center\">2645.09</td><td align=\"center\">-</td>\n </tr>\n <tr>\n <td>+ dynamic_ntk</td><td align=\"center\">4.23</td><td align=\"center\">3.78</td><td align=\"center\">3.59</td><td align=\"center\">3.66</td><td align=\"center\">5.71</td><td align=\"center\">-</td>\n </tr>\n <tr>\n <td>+ dynamic_ntk + logn</td><td align=\"center\">4.23</td><td align=\"center\">3.78</td><td align=\"center\">3.58</td><td align=\"center\">3.56</td><td align=\"center\">4.62</td><td align=\"center\">-</td>\n </tr>\n <tr>\n <td>+ dynamic_ntk + logn + window_attn</td><td align=\"center\">4.23</td><td align=\"center\">3.78</td><td align=\"center\">3.58</td><td align=\"center\">3.49</td><td align=\"center\">4.32</td><td align=\"center\">-</td>\n </tr>\n <tr>\n <tr>\n <td>Qwen-1.8B</td><td align=\"center\"><b>5.00</b></td><td align=\"center\"><b>4.48</b></td><td align=\"center\"><b>4.13</b></td><td align=\"center\"><b>3.89</b></td><td align=\"center\">17.42</td><td align=\"center\">433.85</td>\n </tr>\n <tr>\n <td>+ dynamic_ntk + logn + window_attn</td><td align=\"center\"><b>5.00</b></td><td align=\"center\"><b>4.48</b></td><td align=\"center\"><b>4.14</b></td><td align=\"center\"><b>3.93</b></td><td align=\"center\"><b>3.82</b></td><td align=\"center\"><b>3.83</b></td>\n </tr>\n <tr>\n <td>Qwen-7B</td><td align=\"center\"><b>4.23</b></td><td align=\"center\"><b>3.81</b></td><td align=\"center\"><b>3.52</b></td><td align=\"center\"><b>3.31</b></td><td align=\"center\">7.27</td><td align=\"center\">181.49</td>\n </tr>\n <tr>\n <td>+ dynamic_ntk + logn + window_attn</td><td align=\"center\"><b>4.23</b></td><td align=\"center\"><b>3.81</b></td><td align=\"center\"><b>3.52</b></td><td align=\"center\"><b>3.33</b></td><td align=\"center\"><b>3.22</b></td><td align=\"center\"><b>3.17</b></td>\n </tr>\n <tr>\n <td>Qwen-14B</td><td align=\"center\"><b>-</b></td><td align=\"center\"><b>3.46</b></td><td align=\"center\">22.79</td><td align=\"center\">334.65</td><td align=\"center\">3168.35</td><td align=\"center\">-</td>\n </tr>\n <tr>\n <td>+ dynamic_ntk + logn + window_attn</td><td align=\"center\"><b>-</b></td><td align=\"center\"><b>3.46</b></td><td align=\"center\"><b>3.29</b></td><td align=\"center\"><b>3.18</b></td><td align=\"center\">3.42</td><td align=\"center\">-</td>\n </tr>\n <tr>\n <td>Qwen-72B</td><td align=\"center\"><b>-</b></td><td align=\"center\"><b>-</b></td><td align=\"center\">-</td><td align=\"center\"><b>2.83</b></td><td align=\"center\"><b>2.73</b></td><td align=\"center\"><b>2.72</b></td>\n </tr>\n </tr>\n</table>\n\nFurthermore, to verify the ability of Qwen-72B-Chat on long text understanding, we tested it on [L-Eval](https://arxiv.org/abs/2307.11088) (closed-ended tasks). The results are as follows:\n\n| Model | Input Length | Average | Coursera | GSM | QuALITY | TOEFL | CodeU | SFcition |\n|:------------------|:------------:|:---------:|:----------:|:----------:|:----------:|:----------:|:----------:|:----------:|\n| ChatGPT-3.5-16k | 16K | 60.73 | **63.51** | **84.00** | 61.38 | 78.43 | **12.22** | 64.84 |\n| **Qwen-72B-Chat** | 32K | **62.30** | 58.13 | 76.00 | **77.22** | **86.24** | 6.66 | **69.53** |\n\nHemos realizado el experimento de la \"aguja en el pajar\" (la idea procede de [@Greg Kamradt](https://twitter.com/GregKamradt/status/1727018183608193393)) para comprobar si el modelo puede recuperar información en distintas posiciones de las entradas de distintas longitudes, el resultado es el siguiente:\n\n![](assets/qwen_72b_needle_in_a_haystack.png)\n\nLos resultados anteriores muestran que Qwen-72B-Chat puede recuperar con precisión información situada en varias posiciones dentro de una longitud de entrada de 32K, lo que demuestra su excelente capacidad de comprensión de textos largos.\n\n\n## Tokenizador\n\nNuestro tokenizador basado en tiktoken es diferente de otros tokenizadores, por ejemplo, el tokenizador sentencepiece. Es necesario prestar atención a los tokens especiales, especialmente en el finetuning. Para obtener información más detallada sobre el tokenizador y su uso en el ajuste fino, consulte la [documentación](tokenization_note.md).\n<br><br>\n\n## Reproducción\n\nPara que pueda reproducir el rendimiento del modelo en conjuntos de datos de referencia, le proporcionamos secuencias de comandos para que reproduzca los resultados. Consulte [eval/EVALUATION.md](eval/EVALUATION.md) para obtener más información. Tenga en cuenta que la reproducción puede dar lugar a ligeras diferencias con respecto a nuestros resultados.\n<br><br>\n\n## FAQ\n\nSi tiene problemas, consulte primero [FAQ](FAQ.md) y las incidencias para buscar una solución antes de lanzar una nueva incidencia.\n<br><br>\n\n## Cita\nSi nuestro trabajo le resulta útil, no dude en citarnos.\n\n```\n@article{qwen,\n title={Qwen Technical Report},\n author={Jinze Bai and Shuai Bai and Yunfei Chu and Zeyu Cui and Kai Dang and Xiaodong Deng and Yang Fan and Wenbin Ge and Yu Han and Fei Huang and Binyuan Hui and Luo Ji and Mei Li and Junyang Lin and Runji Lin and Dayiheng Liu and Gao Liu and Chengqiang Lu and Keming Lu and Jianxin Ma and Rui Men and Xingzhang Ren and Xuancheng Ren and Chuanqi Tan and Sinan Tan and Jianhong Tu and Peng Wang and Shijie Wang and Wei Wang and Shengguang Wu and Benfeng Xu and Jin Xu and An Yang and Hao Yang and Jian Yang and Shusheng Yang and Yang Yao and Bowen Yu and Hongyi Yuan and Zheng Yuan and Jianwei Zhang and Xingxuan Zhang and Yichang Zhang and Zhenru Zhang and Chang Zhou and Jingren Zhou and Xiaohuan Zhou and Tianhang Zhu},\n journal={arXiv preprint arXiv:2309.16609},\n year={2023}\n}\n```\n<br>\n\n## Acuerdo de Licencia\n\nEl código fuente proporcionado en <https://github.com/QwenLM/Qwen> está licenciado bajo la [Licencia Apache 2.0](./LICENSE) que puede encontrarse en el directorio raíz.\n\nLos investigadores y desarrolladores son libres de utilizar los códigos y los pesos de los modelos tanto de Qwen como de Qwen-Chat. Para su uso comercial, consulte el Acuerdo de Licencia que acompaña a cada modelo.\n\n- Qwen-72B, Qwen-14B, y Qwen-7B están licenciados bajo el [Tongyi Qianwen LICENSE AGREEMENT](./Tongyi%20Qianwen%20LICENSE%20AGREEMENT) que se puede encontrar en el repositorio correspondiente de HuggingFace y ModelScope. Para uso comercial, rellene el formulario ([72B](https://dashscope.console.aliyun.com/openModelApply/Qwen-72B-Chat), [14B](https://dashscope.console.aliyun.com/openModelApply/Qwen-14B-Chat), y [7B](https://dashscope.console.aliyun.com/openModelApply/qianwen)) para solicitarlo.\n\n- Qwen-1.8B está licenciado bajo el [Tongyi Qianwen RESEARCH LICENSE AGREEMENT](./Tongyi%20Qianwen%20RESEARCH%20LICENSE%20AGREEMENT) que puede encontrarse en el repositorio correspondiente de HuggingFace y ModelScope. Para uso comercial, póngase en contacto con nosotros.\n<br><br>\n\n## Contacte con Nosotros\n\nSi estás interesado en dejar un mensaje a nuestro equipo de investigación o de producto, únete a nuestros grupos de Discord o WeChat. También puedes enviar un correo electrónico a qianwen_opensource@alibabacloud.com.\n\n" }, { "path": "README_FR.md", "content": "<p align=\"left\">\n <a href=\"README_CN.md\">中文</a>  ｜  <a href=\"README.md\">English</a>  ｜  <a href=\"README_JA.md\">日本語</a>  ｜  Français ｜  <a href=\"README_ES.md\">Español</a>\n</p>\n<br><br>\n\n<p align=\"center\">\n <img src=\"https://qianwen-res.oss-cn-beijing.aliyuncs.com/logo_qwen.jpg\" width=\"400\"/>\n<p>\n<br>\n\n<p align=\"center\">\n 🤗 <a href=\"https://huggingface.co/Qwen\">Hugging Face</a>   |   🤖 <a href=\"https://modelscope.cn/organization/qwen\">ModelScope</a>   |    📑 <a href=\"https://arxiv.org/abs/2309.16609\">Paper</a>    ｜   🖥️ <a href=\"https://modelscope.cn/studios/qwen/Qwen-72B-Chat-Demo/summary\">Demo</a>\n<br>\n<a href=\"assets/wechat.png\">WeChat (微信)</a>   |   <a href=\"https://discord.gg/z3GAxXZ9Ce\">Discord</a>   ｜   <a href=\"https://dashscope.aliyun.com\">API</a> \n</p>\n<br><br>\n\n> [!Important]\n> Qwen2 est là ! Vous êtes invité à suivre [QwenLM/Qwen2](https://github.com/QwenLM/Qwen2) et à partager vos expériences là-bas.\n>\n> Ce repo ([QwenLM/Qwen](https://github.com/QwenLM/Qwen)) n'est plus activement maintenu, en raison de différences substantielles dans le code source.\n<br>\n\n| | Qwen-Chat | Qwen-Chat (Int4) | Qwen-Chat (Int8) | Qwen |\n|-----|:------------------------------------------------------------------------------------------------------------------------------------:|:----------------------------------------------------------------------------------------------------------------------------------------------:|:---------------------------------------------------------------:|:--------------------------------------------------------------------------------------------------------------------------:|\n| 1.8B | <a href=\"https://modelscope.cn/models/qwen/Qwen-1_8B-Chat/summary\">🤖</a> <a href=\"https://huggingface.co/Qwen/Qwen-1_8B-Chat\">🤗</a> | <a href=\"https://modelscope.cn/models/qwen/Qwen-1_8B-Chat-Int4/summary\">🤖</a> <a href=\"https://huggingface.co/Qwen/Qwen-1_8B-Chat-Int4\">🤗</a> | <a href=\"https://modelscope.cn/models/qwen/Qwen-1_8B-Chat-Int8/summary\">🤖</a> <a href=\"https://huggingface.co/Qwen/Qwen-1_8B-Chat-Int8\">🤗</a> | <a href=\"https://modelscope.cn/models/qwen/Qwen-1_8B/summary\">🤖</a> <a href=\"https://huggingface.co/Qwen/Qwen-1_8B\">🤗</a> |\n| 7B | <a href=\"https://modelscope.cn/models/qwen/Qwen-7B-Chat/summary\">🤖</a> <a href=\"https://huggingface.co/Qwen/Qwen-7B-Chat\">🤗</a> | <a href=\"https://modelscope.cn/models/qwen/Qwen-7B-Chat-Int4/summary\">🤖</a> <a href=\"https://huggingface.co/Qwen/Qwen-7B-Chat-Int4\">🤗</a> | <a href=\"https://modelscope.cn/models/qwen/Qwen-7B-Chat-Int8/summary\">🤖</a> <a href=\"https://huggingface.co/Qwen/Qwen-7B-Chat-Int8\">🤗</a> | <a href=\"https://modelscope.cn/models/qwen/Qwen-7B/summary\">🤖</a> <a href=\"https://huggingface.co/Qwen/Qwen-7B\">🤗</a> |\n| 14B | <a href=\"https://modelscope.cn/models/qwen/Qwen-14B-Chat/summary\">🤖</a> <a href=\"https://huggingface.co/Qwen/Qwen-14B-Chat\">🤗</a> | <a href=\"https://modelscope.cn/models/qwen/Qwen-14B-Chat-Int4/summary\">🤖</a> <a href=\"https://huggingface.co/Qwen/Qwen-14B-Chat-Int4\">🤗</a> | <a href=\"https://modelscope.cn/models/qwen/Qwen-14B-Chat-Int8/summary\">🤖</a> <a href=\"https://huggingface.co/Qwen/Qwen-14B-Chat-Int8\">🤗</a> | <a href=\"https://modelscope.cn/models/qwen/Qwen-14B/summary\">🤖</a> <a href=\"https://huggingface.co/Qwen/Qwen-14B\">🤗</a> |\n| 72B | <a href=\"https://modelscope.cn/models/qwen/Qwen-72B-Chat/summary\">🤖</a> <a href=\"https://huggingface.co/Qwen/Qwen-72B-Chat\">🤗</a> | <a href=\"https://modelscope.cn/models/qwen/Qwen-72B-Chat-Int4/summary\">🤖</a> <a href=\"https://huggingface.co/Qwen/Qwen-72B-Chat-Int4\">🤗</a> | <a href=\"https://modelscope.cn/models/qwen/Qwen-72B-Chat-Int8/summary\">🤖</a> <a href=\"https://huggingface.co/Qwen/Qwen-72B-Chat-Int8\">🤗</a> | <a href=\"https://modelscope.cn/models/qwen/Qwen-72B/summary\">🤖</a> <a href=\"https://huggingface.co/Qwen/Qwen-72B\">🤗</a> |\n\n\n\nNous ouvrons notre série **Qwen**, qui comprend désormais **Qwen**, les modèles de langue de base, à savoir **Qwen-7B** et **Qwen-14B**, ainsi que **Qwen-Chat**, les modèles de chat, à savoir **Qwen-7B-Chat** et **Qwen-14B-Chat**. Les liens se trouvent dans le tableau ci-dessus. Cliquez dessus et consultez les fiches des modèles. Nous publions également le **[rapport technique](https://arxiv.org/abs/2309.16609)**. Cliquez sur le lien du document et consultez-le !\n\nEn bref, nous disposons de modèles linguistiques solides, qui ont été pré-entraîné de manière stable pour 3 000 milliards de tokens de données multilingues avec une large couverture de domaines, de langues (en particulier le chinois et l'anglais), etc. Ils sont capables d'atteindre des performances compétitives sur des ensembles de données de référence. En outre, nous disposons de modèles de chat alignés sur les préférences humaines basées sur SFT et RLHF (pas encore publiés), qui sont capables de chatter, de créer du contenu, d'extraire des informations, de résumer, de traduire, de coder, de résoudre des problèmes mathématiques, etc. et d'utiliser des outils, de jouer le rôle d'agents ou même code interpreter, etc.\n\n| Modèle | Date de sortie | Longueur maximale | Amélioration de l'invite du système | # de tokens pré-formés | Utilisation minimale de la mémoire du GPU pour Finetuning (Q-Lora) | Utilisation minimale du GPU pour générer 2048 jetons (Int4) | Utilisation des outils |\n|:----------|:--------------:|:-----------------:|:-----------------------------------:|:----------------------:|:------------------------------------------------------------------:|:-----------------------------------------------------------:|:----------------------:|\n| Qwen-1.8B | 23.11.30 | 32K | ✅ | 2.2T | 5.8GB | 2.9GB | ✅ | \n| Qwen-7B | 23.08.03 | 32K | ❎ | 2.4T | 11.5GB | 8.2GB | ✅ | \n| Qwen-14B | 23.09.25 | 8K | ❎ | 3.0T | 18.7GB | 13.0GB | ✅ |\n| Qwen-72B | 23.11.30 | 32K | ✅ | 3.0T | 61.4GB | 48.9GB | ✅ | \n\n\nDans la repo, vous pouvez trouver:\n\n* Comment utiliser Qwen, et profiter de l'inférence simple.\n* Détails sur les modèles de quantization, y compris GPTQ et la quantization de KV cache.\n* Statistiques sur les performances de l'inférence, y compris la vitesse et la mémoire.\n* Tutoriels sur le finetuning, y compris le finetuning de paramètres complets, LoRA, et Q-LoRA.\n* Instructions de déploiement, avec l'exemple de vLLM et FastChat.\n* Instructions sur la création de démos, y compris WebUI, démo CLI, etc.\n* Introduction au service API de DashScope, ainsi que les instructions pour construire une API de type OpenAI pour votre modèle.\n* Informations sur Qwen pour l'utilisation d'outils, d'agents et code interpreter.\n* Statistiques de l'évaluation de la compréhension du contexte long.\n* Contrat de licence.\n* ...\n\nEn outre, si vous rencontrez des problèmes, consultez d'abord la [FAQ](FAQ.md) pour obtenir de l'aide. Vous vous sentez toujours en difficulté ? N'hésitez pas à nous envoyer des questions (de préférence en anglais pour que plus de gens puissent vous comprendre) ! Si vous souhaitez nous aider, envoyez-nous des demandes d'extension sans hésitation ! Nous sommes toujours enthousiastes à propos des relations publiques ! \n\nVous voulez discuter avec nous ou prendre un café avec nous ? Bienvenue sur notre Discord ou WeChat !\n<br><br>\n\n## Nouvelles et mises à jour\n\n* 2023.11.30 🔥 Nous publions **Qwen-72B** et **Qwen-72B-Chat**, qui sont entraînés sur des tokens 3T et prennent en charge 32k contextes, ainsi que **Qwen-1.8B** et **Qwen-1.8B-Chat**, sur ModelScope et Hugging Face. Nous avons également renforcé les capacités de l'invite système du Qwen-72B-Chat et du Qwen-1.8B-Chat, voir la [documentation d'exemple](examples/system_prompt.md). De plus, nous supportons l'inférence sur **Ascend 910** et **Hygon DCU**. Consultez `ascend-support` et `dcu-support` pour plus de détails.\n* 2023.10.17 Nous publions le modèle quantifié Int8 **Qwen-7B-Chat-Int8** et **Qwen-14B-Chat-Int8**.\n* 2023.9.25 🔥 Nous publions **Qwen-14B** et **Qwen-14B-Chat** sur ModelScope et Hugging Face, ainsi que [qwen.cpp](https://github.com/QwenLM/qwen.cpp) et [Qwen-Agent](https://github.com/QwenLM/Qwen-Agent). Les codes et les poids de **Qwen-7B** et **Qwen-7B-Chat** ont également été mis à jour. **S'IL VOUS PLAÎT, TIREZ LA DERNIÈRE VERSION!**\n - Par rapport à **Qwen-7B** (original), **Qwen-7B** utilise davantage de jetons d'entraînement, passant de 2,2 à 2,4T de jetons, tandis que la longueur du contexte passe de 2048 à 8192. La connaissance du chinois et la capacité de codage de **Qwen-7B** ont été encore améliorées.\n* 2023.9.12 Nous prenons désormais en charge le finetuning sur les modèles Qwen-7B, y compris le finetuning de tous les paramètres, LoRA et Q-LoRA.\n* 2023.8.21 Nous publions le modèle quantifié Int4 pour Qwen-7B-Chat, **Qwen-7B-Chat-Int4**, qui nécessite de faibles coûts de mémoire mais permet d'améliorer la vitesse d'inférence. En outre, il n'y a pas de dégradation significative des performances lors de l'évaluation de référence.\n* 2023.8.3 Nous publions **Qwen-7B** et **Qwen-7B-Chat** sur ModelScope et Hugging Face. Nous fournissons également un mémo technique pour plus de détails sur le modèle, y compris les détails de l'entraînement et les performances du modèle.\n<br>\n\n## Performance\n\nLes modèles Qwen surpassent les modèles de base de taille similaire sur une série de données de référence, par exemple MMLU, C-Eval, GSM8K, MATH, HumanEval, MBPP, BBH, etc., qui évaluent les capacités des modèles sur la compréhension du langage naturel, la résolution de problèmes mathématiques, le codage, etc. Qwen-72B obtient de meilleures performances que LLaMA2-70B dans toutes les tâches et surpasse GPT-3.5 dans 7 tâches sur 10.\n\n<p align=\"left\">\n <img src=\"assets/radar_72b.jpg\" width=600px/>\n<p>\n<br>\n\n| Model | MMLU | C-Eval | GSM8K | MATH | HumanEval | MBPP | BBH | CMMLU |\n|:------------------|:--------:|:--------:|:--------:|:--------:|:---------:|:--------:|:--------:|:--------:|\n| | 5-shot | 5-shot | 8-shot | 4-shot | 0-shot | 3-shot | 3-shot | 5-shot |\n| LLaMA2-7B | 46.8 | 32.5 | 16.7 | 3.3 | 12.8 | 20.8 | 38.2 | 31.8 |\n| LLaMA2-13B | 55.0 | 41.4 | 29.6 | 5.0 | 18.9 | 30.3 | 45.6 | 38.4 |\n| LLaMA2-34B | 62.6 | - | 42.2 | 6.2 | 22.6 | 33.0 | 44.1 | - |\n| ChatGLM2-6B | 47.9 | 51.7 | 32.4 | 6.5 | - | - | 33.7 | - |\n| InternLM-7B | 51.0 | 53.4 | 31.2 | 6.3 | 10.4 | 14.0 | 37.0 | 51.8 |\n| InternLM-20B | 62.1 | 58.8 | 52.6 | 7.9 | 25.6 | 35.6 | 52.5 | 59.0 |\n| Baichuan2-7B | 54.7 | 56.3 | 24.6 | 5.6 | 18.3 | 24.2 | 41.6 | 57.1 |\n| Baichuan2-13B | 59.5 | 59.0 | 52.8 | 10.1 | 17.1 | 30.2 | 49.0 | 62.0 |\n| Yi-34B \t \t | 76.3 | 81.8 | 67.9 | 15.9 | 26.2 | 38.2 | 66.4 | 82.6 |\n| XVERSE-65B \t | 70.8 | 68.6 | 60.3 | - | 26.3 | - | - | - |\n| **Qwen-1.8B** | 45.3 | 56.1 | 32.3 | 2.3 | 15.2 | 14.2 | 22.3 | 52.1 |\n| **Qwen-7B** | 58.2 | 63.5 | 51.7 | 11.6 | 29.9 | 31.6 | 45.0 | 62.2 |\n| **Qwen-14B** | 66.3 | 72.1 | 61.3 | 24.8 | 32.3 | 40.8 | 53.4 | 71.0 |\n| **Qwen-72B** | **77.4** | **83.3** | **78.9** | **35.2** | **35.4** | **52.2** | **67.7** | **83.6** |\n\nPour tous les modèles comparés, nous indiquons les meilleurs scores entre leurs résultats officiels et [OpenCompass] (https://opencompass.org.cn/leaderboard-llm). \n\nPour plus de résultats expérimentaux (performances détaillées des modèles sur d'autres ensembles de données de référence) et de détails, veuillez vous référer à notre rapport technique en cliquant [ici](https://qianwen-res.oss-cn-beijing.aliyuncs.com/QWEN_TECHNICAL_REPORT.pdf).\n<br><br>\n\n## Besoins\n\n* python 3.8 et plus\n* pytorch 1.12 et plus, 2.0 et plus sont recommandés\n* transformers 4.32 et plus\n* CUDA 11.4 et plus sont recommandés (pour les utilisateurs de GPU, les utilisateurs de flash, etc.)\n<br>\n\n## Démarrage Rapide\n\nCi-dessous, nous fournissons des exemples simples pour montrer comment utiliser Qwen-Chat avec 🤖 ModelScope et 🤗 Transformers.\n\nVous pouvez utiliser nos images docker pré-construites pour sauter la plupart des étapes de configuration de l'environnement, voir la section [\"Utiliser des images docker pré-construites\"](#-docker) pour plus de détails. \n\nSi vous n'utilisez pas Docker, assurez-vous d'avoir configuré l'environnement et installé les paquets requis. Assurez-vous de répondre aux exigences ci-dessus, puis installez les bibliothèques dépendantes.\n\n```bash\npip install -r requirements.txt\n```\n\nSi votre appareil supporte fp16 ou bf16, nous vous recommandons d'installer [flash-attention](https://github.com/Dao-AILab/flash-attention) (**nous supportons flash-attention 2 maintenant.**) pour une meilleure efficacité et une moindre utilisation de la mémoire. (**flash-attention est optionnel et le projet peut fonctionner normalement sans l'installer**)\n\n```bash\ngit clone https://github.com/Dao-AILab/flash-attention\ncd flash-attention && pip install .\n# Below are optional. Installing them might be slow.\n# pip install csrc/layer_norm\n# pip install csrc/rotary\n```\n\nVous pouvez maintenant commencer avec ModelScope ou Transformers.\n\n### 🤗 Transformers\n\nPour utiliser Qwen-Chat pour l'inférence, il vous suffit de saisir quelques lignes de code, comme indiqué ci-dessous. N'oubliez pas de transmettre les noms de modèles ou les chemins corrects, tels que \"Qwen/Qwen-7B-Chat\" et \"Qwen/Qwen-14B-Chat\". Cependant, **veuillez vous assurer que vous utilisez le code le plus récent**.\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\nfrom transformers.generation import GenerationConfig\n\n# Model names: \"Qwen/Qwen-7B-Chat\", \"Qwen/Qwen-14B-Chat\"\ntokenizer = AutoTokenizer.from_pretrained(\"Qwen/Qwen-7B-Chat\", trust_remote_code=True)\n\n# use bf16\n# model = AutoModelForCausalLM.from_pretrained(\"Qwen/Qwen-7B-Chat\", device_map=\"auto\", trust_remote_code=True, bf16=True).eval()\n# use fp16\n# model = AutoModelForCausalLM.from_pretrained(\"Qwen/Qwen-7B-Chat\", device_map=\"auto\", trust_remote_code=True, fp16=True).eval()\n# use cpu only\n# model = AutoModelForCausalLM.from_pretrained(\"Qwen/Qwen-7B-Chat\", device_map=\"cpu\", trust_remote_code=True).eval()\n# use auto mode, automatically select precision based on the device.\nmodel = AutoModelForCausalLM.from_pretrained(\n \"Qwen/Qwen-7B-Chat\",\n device_map=\"auto\",\n trust_remote_code=True\n).eval()\n\n# Specify hyperparameters for generation. But if you use transformers>=4.32.0, there is no need to do this.\n# model.generation_config = GenerationConfig.from_pretrained(\"Qwen/Qwen-7B-Chat\", trust_remote_code=True)\n\n# 1st dialogue turn\nresponse, history = model.chat(tokenizer, \"你好\", history=None)\nprint(response)\n# 你好！很高兴为你提供帮助。\n\n# 2nd dialogue turn\nresponse, history = model.chat(tokenizer, \"给我讲一个年轻人奋斗创业最终取得成功的故事。\", history=history)\nprint(response)\n# 这是一个关于一个年轻人奋斗创业最终取得成功的故事。\n# 故事的主人公叫李明，他来自一个普通的家庭，父母都是普通的工人。从小，李明就立下了一个目标：要成为一名成功的企业家。\n# 为了实现这个目标，李明勤奋学习，考上了大学。在大学期间，他积极参加各种创业比赛，获得了不少奖项。他还利用课余时间去实习，积累了宝贵的经验。\n# 毕业后，李明决定开始自己的创业之路。他开始寻找投资机会，但多次都被拒绝了。然而，他并没有放弃。他继续努力，不断改进自己的创业计划，并寻找新的投资机会。\n# 最终，李明成功地获得了一笔投资，开始了自己的创业之路。他成立了一家科技公司，专注于开发新型软件。在他的领导下，公司迅速发展起来，成为了一家成功的科技企业。\n# 李明的成功并不是偶然的。他勤奋、坚韧、勇于冒险，不断学习和改进自己。他的成功也证明了，只要努力奋斗，任何人都有可能取得成功。\n\n# 3rd dialogue turn\nresponse, history = model.chat(tokenizer, \"给这个故事起一个标题\", history=history)\nprint(response)\n# 《奋斗创业：一个年轻人的成功之路》\n```\n\nL'exécution du modèle pré-entraîné de Qwen est également simple.\n\n<details>\n <summary>Running Qwen</summary>\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\nfrom transformers.generation import GenerationConfig\n\n# Model names: \"Qwen/Qwen-7B\", \"Qwen/Qwen-14B\" \ntokenizer = AutoTokenizer.from_pretrained(\"Qwen/Qwen-7B\", trust_remote_code=True)\n# use bf16\n# model = AutoModelForCausalLM.from_pretrained(\"Qwen/Qwen-7B\", device_map=\"auto\", trust_remote_code=True, bf16=True).eval()\n# use fp16\n# model = AutoModelForCausalLM.from_pretrained(\"Qwen/Qwen-7B\", device_map=\"auto\", trust_remote_code=True, fp16=True).eval()\n# use cpu only\n# model = AutoModelForCausalLM.from_pretrained(\"Qwen/Qwen-7B\", device_map=\"cpu\", trust_remote_code=True).eval()\n# use auto mode, automatically select precision based on the device.\nmodel = AutoModelForCausalLM.from_pretrained(\n \"Qwen/Qwen-7B\",\n device_map=\"auto\",\n trust_remote_code=True\n).eval()\n\n# Specify hyperparameters for generation. But if you use transformers>=4.32.0, there is no need to do this.\n# model.generation_config = GenerationConfig.from_pretrained(\"Qwen/Qwen-7B\", trust_remote_code=True)\n\ninputs = tokenizer('蒙古国的首都是乌兰巴托（Ulaanbaatar）\\n冰岛的首都是雷克雅未克（Reykjavik）\\n埃塞俄比亚的首都是', return_tensors='pt')\ninputs = inputs.to(model.device)\npred = model.generate(**inputs)\nprint(tokenizer.decode(pred.cpu()[0], skip_special_tokens=True))\n# 蒙古国的首都是乌兰巴托（Ulaanbaatar）\\n冰岛的首都是雷克雅未克（Reykjavik）\\n埃塞俄比亚的首都是亚的斯亚贝巴（Addis Ababa）...\n```\n\n</details>\n\nEn cas de problème de réseau lors de la tentative de téléchargement des poids et des codes du modèle à partir de HuggingFace, une autre approche consiste à récupérer le point de contrôle à partir de ModelScope, puis à le charger à partir du répertoire local, comme indiqué ci-dessous:\n\n```python\nfrom modelscope import snapshot_download\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\n# Downloading model checkpoint to a local dir model_dir\n# model_dir = snapshot_download('qwen/Qwen-7B')\n# model_dir = snapshot_download('qwen/Qwen-7B-Chat')\n# model_dir = snapshot_download('qwen/Qwen-14B')\nmodel_dir = snapshot_download('qwen/Qwen-14B-Chat')\n\n# Loading local checkpoints\n# trust_remote_code is still set as True since we still load codes from local dir instead of transformers\ntokenizer = AutoTokenizer.from_pretrained(model_dir, trust_remote_code=True)\nmodel = AutoModelForCausalLM.from_pretrained(\n model_dir,\n device_map=\"auto\",\n trust_remote_code=True\n).eval()\n```\n\n### 🤖 ModelScope\n\nModelScope est une plateforme opensource pour Model-as-a-Service (MaaS), qui fournit un service de modèle flexible et rentable aux développeurs d'IA. De même, vous pouvez exécuter les modèles avec ModelScope comme indiqué ci-dessous:\n\n```python\nfrom modelscope import AutoModelForCausalLM, AutoTokenizer\nfrom modelscope import GenerationConfig\n\n# Model names: \"qwen/Qwen-7B-Chat\", \"qwen/Qwen-14B-Chat\"\ntokenizer = AutoTokenizer.from_pretrained(\"qwen/Qwen-7B-Chat\", trust_remote_code=True)\nmodel = AutoModelForCausalLM.from_pretrained(\"qwen/Qwen-7B-Chat\", device_map=\"auto\", trust_remote_code=True, fp16=True).eval()\nmodel.generation_config = GenerationConfig.from_pretrained(\"Qwen/Qwen-7B-Chat\", trust_remote_code=True) # 可指定不同的生成长度、top_p等相关超参\n\nresponse, history = model.chat(tokenizer, \"你好\", history=None)\nprint(response)\nresponse, history = model.chat(tokenizer, \"浙江的省会在哪里？\", history=history) \nprint(response)\nresponse, history = model.chat(tokenizer, \"它有什么好玩的景点\", history=history)\nprint(response)\n```\n\n### Inférence par lots\nQwen prend en charge l'inférence par lots. Lorsque flash attention est activée, l'utilisation de l'inférence par lots peut entraîner une accélération de 40 %. Le code d'exemple est présenté ci-dessous:\n```python\nimport torch\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\nfrom transformers import GenerationConfig\nfrom qwen_generation_utils import make_context, decode_tokens, get_stop_words_ids\n\ntokenizer = AutoTokenizer.from_pretrained(\n './',\n pad_token='<|extra_0|>',\n eos_token='<|endoftext|>',\n padding_side='left',\n trust_remote_code=True\n)\nmodel = AutoModelForCausalLM.from_pretrained(\n './',\n pad_token_id=tokenizer.pad_token_id,\n device_map=\"auto\",\n trust_remote_code=True\n).eval()\nmodel.generation_config = GenerationConfig.from_pretrained('./', pad_token_id=tokenizer.pad_token_id)\n\nall_raw_text = [\"我想听你说爱我。\", \"今天我想吃点啥，甜甜的，推荐下\", \"我马上迟到了，怎么做才能不迟到\"]\nbatch_raw_text = []\nfor q in all_raw_text:\n raw_text, _ = make_context(\n tokenizer,\n q,\n system=\"You are a helpful assistant.\",\n max_window_size=model.generation_config.max_window_size,\n chat_format=model.generation_config.chat_format,\n )\n batch_raw_text.append(raw_text)\n\nbatch_input_ids = tokenizer(batch_raw_text, padding='longest')\nbatch_input_ids = torch.LongTensor(batch_input_ids['input_ids']).to(model.device)\nbatch_out_ids = model.generate(\n batch_input_ids,\n return_dict_in_generate=False,\n generation_config=model.generation_config\n)\npadding_lens = [batch_input_ids[i].eq(tokenizer.pad_token_id).sum().item() for i in range(batch_input_ids.size(0))]\n\nbatch_response = [\n decode_tokens(\n batch_out_ids[i][padding_lens[i]:],\n tokenizer,\n raw_text_len=len(batch_raw_text[i]),\n context_length=(batch_input_ids[i].size(0)-padding_lens[i]),\n chat_format=\"chatml\",\n verbose=False,\n errors='replace'\n ) for i in range(len(all_raw_text))\n]\nprint(batch_response)\n\nresponse, _ = model.chat(tokenizer, \"我想听你说爱我。\", history=None)\nprint(response)\n\nresponse, _ = model.chat(tokenizer, \"今天我想吃点啥，甜甜的，推荐下\", history=None)\nprint(response)\n\nresponse, _ = model.chat(tokenizer, \"我马上迟到了，怎么做才能不迟到\", history=None)\nprint(response)\n```\n\n### CPU\n\nPour déployer nos modèles sur CPU, nous vous conseillons vivement d'utiliser [qwen.cpp](https://github.com/QwenLM/qwen.cpp), qui est une implémentation purement C++ de Qwen et de tiktoken. Consultez le repo pour plus de détails!\n\nIl est simple d'exécuter directement le modèle sur le CPU, ce qui nécessite la spécification de votre appareil:\n\n```python\nmodel = AutoModelForCausalLM.from_pretrained(\"Qwen/Qwen-7B-Chat\", device_map=\"cpu\", trust_remote_code=True).eval()\n```\n\nCependant, il est probable que vous souffriez d'une efficacité d'inférence extrêmement faible.\n\n### Plusieurs GPU\n\nSi vous souffrez d'un manque de mémoire GPU et que vous souhaitez exécuter le modèle sur plus d'un GPU, vous pouvez utiliser directement la méthode de chargement par défaut, qui est maintenant supportée par Transformers. La méthode précédente basée sur `utils.py` est obsolète.\n\nCependant, bien que cette méthode soit simple, l'efficacité du parallélisme natif du pipeline est faible. Nous vous conseillons d'utiliser vLLM avec FastChat et de lire la section relative au déploiement.\n\n\n### DashScope\n\nLe moyen le plus simple d'utiliser Qwen via les API est le service API DashScope via Alibaba Cloud. Nous présentons une introduction à l'utilisation. De plus, nous fournissons un script pour vous permettre de déployer une API de type OpenAI sur vos propres serveurs.\n\nDashScope est le service API de grands modèles linguistiques fourni par Alibaba Cloud, qui prend désormais en charge Qwen. Notez que les modèles derrière DashScope sont des versions internes temporairement sans détails fournis. Les services comprennent `qwen-turbo` et `qwen-plus`, le premier fonctionnant plus rapidement et le second atteignant de meilleures performances. Pour plus d'informations, consultez la documentation [ici] (https://dashscope.aliyun.com).\n\nVeuillez vous rendre sur le site officiel [lien](https://help.aliyun.com/zh/dashscope/developer-reference/activate-dashscope-and-create-an-api-key?spm=a2c4g.11186623.0.0.6c2774fahtfXdn) pour créer un compte DashScope et obtenir la clé API (AK). Nous recommandons de définir l'AK à l'aide d'une variable d'environnement:\n```bash\nexport DASHSCOPE_API_KEY=\"YOUR_DASHSCOPE_API_KEY\"\n```\nInstallez ensuite les paquets et cliquez sur [ici](https://help.aliyun.com/zh/dashscope/developer-reference/install-dashscope-sdk) pour obtenir la documentation. Si vous utilisez Python, vous pouvez installer DashScope avec pip:\n```bash\npip install dashscope\n```\nSi vous utilisez JAVA SDK, vous pouvez l'installer de cette manière:\n```xml\n<!-- https://mvnrepository.com/artifact/com.alibaba/dashscope-sdk-java -->\n<dependency>\n <groupId>com.alibaba</groupId>\n <artifactId>dashscope-sdk-java</artifactId>\n <version>the-latest-version</version>\n</dependency>\n```\nLa manière la plus simple d'utiliser DashScope est l'utilisation de messages, qui est similaire à l'API OpenAI. L'exemple est présenté ci-dessous:\n```python\nimport random\nfrom http import HTTPStatus\nfrom dashscope import Generation\n\n\ndef call_with_messages():\n messages = [{'role': 'system', 'content': 'You are a helpful assistant.'},\n {'role': 'user', 'content': '如何做西红柿鸡蛋？'}]\n gen = Generation()\n response = gen.call(\n Generation.Models.qwen_turbo,\n messages=messages,\n seed=random.randint(1, 10000), # set the random seed, optional, default to 1234 if not set\n result_format='message', # set the result to be \"message\" format.\n )\n return response\n\n\nif __name__ == '__main__':\n response = call_with_messages()\n if response.status_code == HTTPStatus.OK:\n print(response)\n else:\n print('Request id: %s, Status code: %s, error code: %s, error message: %s' % (\n response.request_id, response.status_code,\n response.code, response.message\n ))\n```\nPour d'autres utilisations, veuillez consulter le site web officiel pour plus de détails.\n<br><br>\n\n## Quantization\n\n### GPTQ\n\nNous proposons une solution basée sur [AutoGPTQ](https://github.com/PanQiWei/AutoGPTQ), et publions les modèles quantifiés Int4 et Int8, qui permettent d'obtenir des effets de modèle presque sans perte mais des performances améliorées en termes de coûts de mémoire et de vitesse d'inférence.\n\nNous démontrons ici comment utiliser les modèles quantifiés que nous fournissons pour l'inférence. Avant de commencer, assurez-vous que vous répondez aux exigences d'auto-gptq (par exemple, torch 2.0 et plus, transformers 4.32.0 et plus, etc.) et installez les paquets requis:\n\n```bash\npip install auto-gptq optimum\n```\n\nSi vous rencontrez des problèmes pour installer `auto-gptq`, nous vous conseillons de consulter le [repo](https://github.com/PanQiWei/AutoGPTQ) officiel pour trouver une roue.\n\n> Note : Les paquets `auto-gptq` précompilés dépendent fortement de la version de `torch` et de sa version CUDA. De plus, en raison d'une récente mise à jour,\n> vous pouvez aussi rencontrer des erreurs de version non supportée avec `transformers`, `optimum`, ou `peft`.\n> Nous recommandons d'utiliser les dernières versions répondant aux exigences suivantes :\n> - torch==2.1 auto-gptq>=0.5.1 transformers>=4.35.0 optimum>=1.14.0 peft>=0.6.1\n> - torch>=2.0,<2.1 auto-gptq<0.5.0 transformers<4.35.0 optimum<1.14.0 peft>=0.5.0,<0.6.0\n\nVous pouvez ensuite charger facilement le modèle quantifié et lancer l'inférence comme d'habitude:\n\n```python\n# Model names: \"Qwen/Qwen-7B-Chat-Int4\", \"Qwen/Qwen-14B-Chat-Int4\"\nmodel = AutoModelForCausalLM.from_pretrained(\n \"Qwen/Qwen-7B-Chat-Int4\",\n device_map=\"auto\",\n trust_remote_code=True\n).eval()\nresponse, history = model.chat(tokenizer, \"Hi\", history=None)\n```\n\nNous illustrons les performances des modèles BF16, Int8 et Int4 sur le benchmark, et nous constatons que le modèle quantifié ne souffre pas d'une dégradation significative des performances. Les résultats sont présentés ci-dessous:\n\n| Quantization | MMLU | CEval (val) | GSM8K | Humaneval |\n|----------------------|:----:|:-----------:|:-----:|:---------:|\n| Qwen-1.8B-Chat (BF16)| 43.3 | 55.6 | 33.7 | 26.2 |\n| Qwen-1.8B-Chat (Int8)| 43.1 | 55.8 | 33.0 | 27.4 |\n| Qwen-1.8B-Chat (Int4)| 42.9 | 52.8 | 31.2 | 25.0 |\n| Qwen-7B-Chat (BF16) | 55.8 | 59.7 | 50.3 | 37.2 |\n| Qwen-7B-Chat (Int8) | 55.4 | 59.4 | 48.3 | 34.8 |\n| Qwen-7B-Chat (Int4) | 55.1 | 59.2 | 49.7 | 29.9 |\n| Qwen-14B-Chat (BF16) | 64.6 | 69.8 | 60.1 | 43.9 |\n| Qwen-14B-Chat (Int8) | 63.6 | 68.6 | 60.0 | 48.2 |\n| Qwen-14B-Chat (Int4) | 63.3 | 69.0 | 59.8 | 45.7 |\n| Qwen-72B-Chat (BF16) | 74.4 | 80.1 | 76.4 | 64.6 |\n| Qwen-72B-Chat (Int8) | 73.5 | 80.1 | 73.5 | 62.2 |\n| Qwen-72B-Chat (Int4) | 73.4 | 80.1 | 75.3 | 61.6 |\n\n### Quantization du cache KV\n\n> NOTE : Veuillez noter qu'en raison du mécanisme interne de Hugging Face, les fichiers de support pour cette fonctionnalité \n> (i.e., `cache_autogptq_cuda_256.cpp` et `cache_autogptq_cuda_kernel_256.cu`) peuvent être manquants. \n> Veuillez les télécharger manuellement manuellement depuis le Hugging Face Hub et placez-les dans le même dossier que les autres fichiers du module.\n\nLe cache KV de l'attention peut être quantifié et compressé pour le stockage, afin d'obtenir un débit d'échantillonnage plus élevé. Les arguments `use_cache_quantization` et `use_cache_kernel` dans `config.json` sont fournis pour activer la quantification du cache KV. \nLa méthode d'utilisation spécifique est la suivante:\n\n```python\nmodel = AutoModelForCausalLM.from_pretrained(\n \"Qwen/Qwen-7B-Chat\",\n device_map=\"auto\",\n trust_remote_code=True,\n use_cache_quantization=True,\n use_cache_kernel=True,\n use_flash_attn=False\n)\n```\nAttention : Actuellement, la quantification du cache KV et flash attention ne peuvent pas être utilisées en même temps.\nSi vous activez la quantification du cache KV et flash attention en même temps (`use_flash_attn=True`, `use_cache_quantization=True`, `use_cache_kernel=True`), `use_flash_attn` est désactivé par défaut (`use_flash_attn=false`).\n\nNous avons vérifié que l'utilisation du modèle int8-kvcache quantifié ne souffre pas d'une dégradation significative des performances dans l'évaluation en aval. Dans ce qui suit, nous nous concentrons sur le profilage de son empreinte mémoire dans différentes conditions. \nLe profilage s'exécute sur un seul GPU A100-SXM4-80G avec PyTorch 2.0.1 et CUDA 11.4. \nNous utilisons des modèles BF16 pour générer 1024 jetons par défaut, et \"OOM\" indique une erreur de mémoire insuffisante.\n\nAvec la quantification du cache KV, le modèle peut inférer avec une taille de lot (bs) plus grande.\n\n| Utilisation du cache KV | bs=1 | bs=4 | bs=16 | bs=32 | bs=64 | bs=100 |\n|--------------|:------:|:------:|:------:|:------:|:------:|:------:|\n| Non | 16.3GB | 24.1GB | 31.7GB | 48.7GB | OOM | OOM |\n| Oui | 15.5GB | 17.2GB | 22.3GB | 30.2GB | 48.2GB | 72.4GB |\n\nAvec la quantification du cache KV, le modèle peut économiser plus de mémoire lorsqu'il génère des séquences plus longues (`sl`, se référant au nombre de jetons générés) à l'étape de l'inférence.\n\n| Utilisation du cache KV | sl=512 | sl=1024 | sl=2048 | sl=4096 | sl=8192 |\n|-------------------------|:------:|:-------:|:-------:|:-------:|:-------:|\n| Non | 15.2GB | 16.3GB | 17.6GB | 19.5GB | 23.2GB |\n| Oui | 15.0GB | 15.5GB | 15.8GB | 16.6GB | 17.6GB |\n\nLe modèle avec quantification du cache KV convertira le format de `layer_past` de float à int8, et pendant ce temps le `layer-past` quantifié stockera également les paramètres de quantification.\n\nLes étapes spécifiques sont les suivantes:\n\n1. Quantifier clé/valeur\n```\n qv,scale,zero_point=quantize_cache_v(v)\n```\n2. Stocker dans `layer_past`\n\nVoici le format de `layer_past` quantifié:\n```\n layer_past=((q_key,key_scale,key_zero_point),\n (q_value,value_scale,value_zero_point))\n```\n\nLe format original de `layer_past` est illustré ci-dessous:\n```\n layer_past=(key,value)\n```\n\nSi vous souhaitez utiliser l'attention KV qui est quantifiée, vous pouvez utiliser l'opération de déquantification pour reconvertir la clé/valeur int8 au format float comme suit \nvous pouvez utiliser l'opération de déquantification pour reconvertir la clé/valeur int8 au format float comme suit:\n```\n v=dequantize_cache_torch(qv,scale,zero_point)\n```\n<br>\n\n\n## Performance de l'inférence\n\nCette section fournit les statistiques de vitesse et de mémoire des modèles dans différentes précisions. Le profilage de la vitesse et de la mémoire est effectué à l'aide de [ce script] (https://qianwen-res.oss-cn-beijing.aliyuncs.com/profile.py).\n\nNous avons mesuré la vitesse moyenne d'inférence (tokens/s) et l'utilisation de la mémoire GPU pour générer 2048 avec les modèles en BF16, Int8 et Int4.\n\n<table>\n <tr>\n <td>Model Size</td>\n <td>Quantization</td>\n <td>Speed (Tokens/s)</td>\n <td>GPU Memory Usage</td>\n </tr>\n <tr>\n <td rowspan=\"3\">1.8B</td>\n <td>BF16</td>\n <td>54.09</td>\n <td>4.23GB</td>\n </tr>\n <tr>\n <td>Int8</td>\n <td>55.56</td>\n <td>3.48GB</td>\n </tr>\n <tr>\n <td>Int4</td>\n <td>71.07</td>\n <td>2.91GB</td>\n </tr>\n <tr>\n <td rowspan=\"3\">7B</td>\n <td>BF16</td>\n <td>40.93</td>\n <td>16.99GB</td>\n </tr>\n <tr>\n <td>Int8</td>\n <td>37.47</td>\n <td>11.20GB</td>\n </tr>\n <tr>\n <td>Int4</td>\n <td>50.09</td>\n <td>8.21GB</td>\n </tr>\n <tr>\n <td rowspan=\"3\">14B</td>\n <td>BF16</td>\n <td>32.22</td>\n <td>30.15GB</td>\n </tr>\n <tr>\n <td>Int8</td>\n <td>29.28</td>\n <td>18.81GB</td>\n </tr>\n <tr>\n <td>Int4</td>\n <td>38.72</td>\n <td>13.01GB</td>\n </tr>\n <tr>\n <td rowspan=\"3\">72B</td>\n <td>BF16</td>\n <td>8.48</td>\n <td>144.69GB (2xA100)</td>\n </tr>\n <tr>\n <td>Int8</td>\n <td>9.05</td>\n <td>81.27GB (2xA100)</td>\n </tr>\n <tr>\n <td>Int4</td>\n <td>11.32</td>\n <td>48.86GB</td>\n </tr>\n <tr>\n <td>72B + vLLM</td>\n <td>BF16</td>\n <td>17.60</td>\n <td>2xA100</td>\n </tr>\n</table>\n\nLe profilage s'exécute sur un seul GPU A100-SXM4-80G (sauf si 2xA100 est mentionné) avec PyTorch 2.0.1, CUDA 11.8, et Flash-Attention 2. (72B + vLLM utilise PyTorch 2.1.0 et Cuda 11.8.) La vitesse d'inférence est calculée en moyenne sur les tokens encodés et générés.\n\nNote : La vitesse de génération des modèles Int4/Int8 mentionnés ci-dessus est fournie par la bibliothèque autogptq. La vitesse actuelle du modèle chargé en utilisant ``AutoModelForCausalLM.from_pretrained`` sera environ 20% plus lente. Nous avons signalé ce problème à l'équipe HuggingFace et nous le mettrons à jour rapidement si une solution est disponible.\n\nNous mesurons également la vitesse d'inférence et l'utilisation de la mémoire du GPU avec différents paramètres de contexte et de longueur de génération, version Flash-Attention. Vous pouvez trouver les résultats dans les cartes modèles correspondantes sur Hugging Face ou ModelScope.\n\n\n## Finetuning\n\n### Utilisation\nNous fournissons maintenant le script d'entraînement officiel, `finetune.py`, pour que les utilisateurs puissent ajuster le modèle pré-entraîné pour les applications en aval de manière simple. De plus, nous fournissons des scripts shell pour lancer le finetune sans soucis. Ce script prend en charge l'entraînement avec [DeepSpeed](https://github.com/microsoft/DeepSpeed) et [FSDP](https://engineering.fb.com/2021/07/15/open-source/fsdp/). Les scripts que nous fournissons utilisent DeepSpeed (Note : il peut y avoir des conflits avec la dernière version de pydantic et vous devriez utiliser make sure `pydantic<2.0`) et Peft. Vous pouvez les installer en procédant comme suit :\n```bash\npip install \"peft<0.8.0\" deepspeed\n```\n\nPour préparer vos données d'entraînement, vous devez rassembler tous les échantillons dans une liste et l'enregistrer dans un fichier json. Chaque échantillon est un dictionnaire composé d'un identifiant et d'une liste de conversation. Voici un exemple simple de liste avec 1 échantillon :\n```json\n[\n {\n \"id\": \"identity_0\",\n \"conversations\": [\n {\n \"from\": \"user\",\n \"value\": \"你好\"\n },\n {\n \"from\": \"assistant\",\n \"value\": \"我是一个语言模型，我叫通义千问。\"\n }\n ]\n }\n]\n```\n\nAprès la préparation des données, vous pouvez utiliser les scripts shell fournis pour lancer le finetuning. N'oubliez pas de spécifier le chemin d'accès au fichier de données, `$DATA`.\n\nLes scripts de finetuning vous permettent d'effectuer les opérations suivantes\n- Finetuning de tous les paramètres\n- LoRA\n- Q-LoRA\n\nLe finetuning de tous les paramètres nécessite la mise à jour de tous les paramètres au cours de l'ensemble du processus de formation. Pour lancer votre formation, exécutez le script suivant:\n\n```bash\n# Distributed training. We do not provide single-GPU training script as the insufficient GPU memory will break down the training.\nbash finetune/finetune_ds.sh\n```\n\nN'oubliez pas de spécifier le nom ou le chemin d'accès au modèle, le chemin d'accès aux données, ainsi que le répertoire de sortie dans les scripts shell. Une autre chose à noter est que nous utilisons DeepSpeed ZeRO 3 dans ce script. Si vous voulez faire des changements, il suffit de supprimer l'argument `--deepspeed` ou de faire des changements dans le fichier json de configuration de DeepSpeed en fonction de vos besoins. De plus, ce script supporte l'entraînement en précision mixte, et donc vous pouvez utiliser `--bf16 True` ou `--fp16 True`. N'oubliez pas d'utiliser DeepSpeed lorsque vous utilisez fp16 en raison de l'entraînement de précision mixte. Empiriquement, nous vous conseillons d'utiliser bf16 pour rendre votre apprentissage cohérent avec notre pré-entraînement et notre alignement si votre machine supporte bf16, et nous l'utilisons donc par défaut.\n\nPour exécuter LoRA, utilisez un autre script à exécuter comme indiqué ci-dessous. Avant de commencer, assurez-vous que vous avez installé `peft`. Vous devez spécifier les chemins d'accès à votre modèle, à vos données et à vos résultats. Nous vous conseillons d'utiliser des chemins absolus pour votre modèle pré-entraîné. En effet, LoRA ne sauvegarde que l'adaptateur et le chemin absolu dans le fichier json de configuration de l'adaptateur est utilisé pour trouver le modèle pré-entraîné à charger. De plus, ce script supporte à la fois bf16 et fp16.\n\n```bash\n# Single GPU training\nbash finetune/finetune_lora_single_gpu.sh\n# Distributed training\nbash finetune/finetune_lora_ds.sh\n```\n\nPar rapport au finetuning de tous les paramètres, LoRA ([paper](https://arxiv.org/abs/2106.09685)) ne met à jour que les paramètres des couches d'adaptateurs, tout en gelant les couches originales du grand modèle de langage. Cela permet de réduire considérablement les coûts de mémoire et donc les coûts de calcul.\n\nNotez que si vous utilisez LoRA pour affiner le modèle linguistique de base, par exemple Qwen-7B, au lieu des modèles de chat, par exemple Qwen-7B-Chat, le script change automatiquement l'intégration et la couche de sortie en tant que paramètres entraînables. En effet, le modèle linguistique de base n'a aucune connaissance des jetons spéciaux apportés par le format ChatML. Ces couches doivent donc être mises à jour pour que le modèle comprenne et prédise les jetons. En d'autres termes, si votre formation apporte des tokens spéciaux dans LoRA, vous devez définir les couches comme des paramètres entraînables en définissant `modules_to_save` à l'intérieur du code. De plus, si ces paramètres sont entraînables, il n'est pas possible d'utiliser ZeRO 3, et c'est pourquoi nous utilisons ZeRO 2 par défaut dans le script. Si vous n'avez pas de nouveaux paramètres entraînables, vous pouvez passer à ZeRO 3 en modifiant le fichier de configuration de DeepSpeed. En outre, nous constatons qu'il existe un écart important entre l'empreinte mémoire de LoRA avec et sans ces paramètres d'entraînement. Par conséquent, si vous avez des problèmes de mémoire, nous vous conseillons d'affiner les modèles de chat de LoRA. Consultez le profil ci-dessous pour plus d'informations.\n\nSi vous souffrez toujours d'un manque de mémoire, vous pouvez envisager Q-LoRA ([paper](https://arxiv.org/abs/2305.14314)), qui utilise le modèle de langage quantifié et d'autres techniques telles que l'attention paginée pour réduire encore les coûts de mémoire.\n\nNote : pour exécuter l'entraînement Q-LoRA sur un seul GPU, vous pouvez avoir besoin d'installer `mpi4py` via `pip` ou `conda`.\n\nPour lancer Q-LoRA, exécutez directement le script suivant :\n\n```bash\n# Single GPU training\nbash finetune/finetune_qlora_single_gpu.sh\n# Distributed training\nbash finetune/finetune_qlora_ds.sh\n```\n\nPour Q-LoRA, nous vous conseillons de charger le modèle quantifié que nous fournissons, par exemple Qwen-7B-Chat-Int4. Vous **NE DEVRIEZ PAS** utiliser les modèles bf16. Contrairement au finetuning de tous les paramètres et à la LoRA, seul le modèle fp16 est pris en charge pour la Q-LoRA. Pour l'entraînement sur un seul GPU, nous devons utiliser DeepSpeed pour l'entraînement en précision mixte en raison de notre observation des erreurs causées par torch amp. En outre, pour Q-LoRA, les problèmes avec les jetons spéciaux dans LoRA existent toujours. Cependant, comme nous ne fournissons que les modèles Int4 pour les modèles de chat, ce qui signifie que le modèle de langage a appris les tokens spéciaux du format ChatML, vous n'avez pas à vous soucier des couches. Notez que les couches du modèle Int4 ne doivent pas être entraînables, et donc si vous introduisez des tokens spéciaux dans votre entraînement, Q-LoRA risque de ne pas fonctionner.\n\n> NOTE : Veuillez noter qu'en raison des mécanismes internes de Hugging Face, certains fichiers non-Python (par exemple, `*.cpp` et `*.cu`) \n> peuvent être absents du point de contrôle sauvegardé. Vous devrez peut-être les copier manuellement dans le répertoire contenant les autres fichiers.\n\nContrairement au finetuning des paramètres complets, l'entraînement de LoRA et de Q-LoRA n'enregistre que les paramètres de l'adaptateur. Supposons que votre entraînement commence à partir de Qwen-7B, vous pouvez charger le modèle finalisé pour l'inférence comme indiqué ci-dessous:\n\n```python\nfrom peft import AutoPeftModelForCausalLM\n\nmodel = AutoPeftModelForCausalLM.from_pretrained(\n path_to_adapter, # path to the output directory\n device_map=\"auto\",\n trust_remote_code=True\n).eval()\n```\n\nSi vous souhaitez fusionner les adaptateurs et enregistrer le modèle affiné en tant que modèle autonome (vous ne pouvez le faire qu'avec LoRA, et vous **NE POUVEZ PAS** fusionner les paramètres de Q-LoRA), vous pouvez exécuter les codes suivants :\n\n```python\nfrom peft import AutoPeftModelForCausalLM\n\nmodel = AutoPeftModelForCausalLM.from_pretrained(\n path_to_adapter, # path to the output directory\n device_map=\"auto\",\n trust_remote_code=True\n).eval()\n\nmerged_model = model.merge_and_unload()\n# max_shard_size and safe serialization are not necessary. \n# They respectively work for sharding checkpoint and save the model to safetensors\nmerged_model.save_pretrained(new_model_directory, max_shard_size=\"2048MB\", safe_serialization=True)\n```\n\nNote : Pour l'entraînement multi-GPU, vous devez spécifier les hyperparamètres appropriés pour l'entraînement distribué en fonction de votre machine. De plus, nous vous conseillons de spécifier votre longueur maximale de séquence avec l'argument `--model_max_length`, en fonction de votre considération des données, de l'empreinte mémoire, et de la vitesse d'apprentissage.\n\n\n### Profilage de la mémoire et de la vitesse\nNous profilons la mémoire du GPU et la vitesse d'apprentissage de LoRA (LoRA (emb) se réfère à l'apprentissage de la couche d'intégration et de sortie, tandis que LoRA n'a pas de couche d'intégration et de sortie pouvant être entraînée) et de Q-LoRA dans la configuration de l'apprentissage sur un seul GPU. Dans ce test, nous expérimentons sur un seul GPU A100-SXM4-80G, et nous utilisons CUDA 11.8 et Pytorch 2.0. Flash attention 2 est appliqué. Nous utilisons uniformément une taille de lot de 1 et une accumulation de gradient de 8. Nous profilons la mémoire (GB) et la vitesse (s/iter) des entrées de différentes longueurs, à savoir 256, 512, 1024, 2048, 4096, et 8192. Nous présentons également les statistiques du réglage fin de tous les paramètres avec Qwen-7B sur 2 GPU A100. Nous ne présentons que les statistiques de 256, 512 et 1024 jetons en raison de la limitation de la mémoire du GPU. \n\nPour Qwen-72B, nous expérimentons de deux manières : 1) Lora fintuning + DeepSpeed ZeRO 3 sur 4 GPU A100-SXM4-80G et 2) QLora (int4) fintuning sur un seul GPU A100-SXM4-80G. Notez que l'OOM se produit sur 4 GPUs A100-SXM4-80G à la fois avec le réglage fin LoRA (emb) et le réglage fin LoRA sans Deepspeed ZeRO 3 (vous pouvez passer `--deepspeed finetune/ds_config_zero3.json` à [`finetune/finetune_lora_ds.sh`](finetune/finetune_lora_ds.sh) afin d'activer DeepSpeed ZeRO 3).\n\nLes statistiques sont listées ci-dessous :\n\n<table>\n <tr>\n <th rowspan=\"2\">Model Size</th><th rowspan=\"2\">Method</th><th colspan=\"6\" align=\"center\">Sequence Length</th>\n </tr>\n <tr>\n <th align=\"center\">256</th><th align=\"center\">512</th><th align=\"center\">1024</th><th align=\"center\">2048</th><th align=\"center\">4096</th><th align=\"center\">8192</th>\n </tr>\n </tr>\n </tr>\n\t\t<tr>\n <th rowspan=\"4\">1.8B</th><td>LoRA</td><td align=\"center\">6.7G / 1.0s/it</td><td align=\"center\">7.4G / 1.0s/it</td><td align=\"center\">8.4G / 1.1s/it</td><td align=\"center\">11.0G / 1.7s/it</td><td align=\"center\">16.2G / 3.3s/it</td><td align=\"center\">21.8G / 6.8s/it</td>\n </tr>\n <tr>\n <td>LoRA (emb)</td><td align=\"center\">13.7G / 1.0s/it</td><td align=\"center\">14.0G / 1.0s/it</td><td align=\"center\">14.0G / 1.1s/it</td><td align=\"center\">15.1G / 1.8s/it</td><td align=\"center\">19.7G / 3.4s/it</td><td align=\"center\">27.7G / 7.0s/it</td>\n </tr>\n <tr>\n <td>Q-LoRA</td><td align=\"center\">5.8G / 1.4s/it</td><td align=\"center\">6.0G / 1.4s/it</td><td align=\"center\">6.6G / 1.4s/it</td><td align=\"center\">7.8G / 2.0s/it</td><td align=\"center\">10.2G / 3.4s/it</td><td align=\"center\">15.8G / 6.5s/it</td>\n </tr>\n <tr>\n <td>Full-parameter</td><td align=\"center\">43.5G / 2.1s/it</td><td align=\"center\">43.5G / 2.2s/it</td><td align=\"center\">43.5G / 2.2s/it</td><td align=\"center\">43.5G / 2.3s/it</td><td align=\"center\">47.1G / 2.8s/it</td><td align=\"center\">48.3G / 5.6s/it</td>\n </tr>\n <tr>\n <th rowspan=\"4\">7B</th><td>LoRA</td><td align=\"center\">20.1G / 1.2s/it</td><td align=\"center\">20.4G / 1.5s/it</td><td align=\"center\">21.5G / 2.8s/it</td><td align=\"center\">23.8G / 5.2s/it</td><td align=\"center\">29.7G / 10.1s/it</td><td align=\"center\">36.6G / 21.3s/it</td>\n </tr>\n <tr>\n <td>LoRA (emb)</td><td align=\"center\">33.7G / 1.4s/it</td><td align=\"center\">34.1G / 1.6s/it</td><td align=\"center\">35.2G / 2.9s/it</td><td align=\"center\">35.1G / 5.3s/it</td><td align=\"center\">39.2G / 10.3s/it</td><td align=\"center\">48.5G / 21.7s/it</td>\n </tr>\n <tr>\n <td>Q-LoRA</td><td align=\"center\">11.5G / 3.0s/it</td><td align=\"center\">11.5G / 3.0s/it</td><td align=\"center\">12.3G / 3.5s/it</td><td align=\"center\">13.9G / 7.0s/it</td><td align=\"center\">16.9G / 11.6s/it</td><td align=\"center\">23.5G / 22.3s/it</td>\n </tr>\n <tr>\n <td>Full-parameter</td><td align=\"center\">139.2G / 4.0s/it</td><td align=\"center\">148.0G / 4.0s/it</td><td align=\"center\">162.0G / 4.5s/it</td><td align=\"center\">-</td><td align=\"center\">-</td><td align=\"center\">-</td>\n </tr>\n <tr>\n <th rowspan=\"3\">14B</th><td>LoRA</td><td align=\"center\">34.6G / 1.6s/it</td><td align=\"center\">35.1G / 2.4s/it</td><td align=\"center\">35.3G / 4.4s/it</td><td align=\"center\">37.4G / 8.4s/it</td><td align=\"center\">42.5G / 17.0s/it</td><td align=\"center\">55.2G / 36.0s/it</td>\n </tr>\n <tr>\n <td>LoRA (emb)</td><td align=\"center\">51.2 / 1.7s/it</td><td align=\"center\">51.1G / 2.6s/it</td><td align=\"center\">51.5G / 4.6s/it</td><td align=\"center\">54.1G / 8.6s/it</td><td align=\"center\">56.8G / 17.2s/it</td><td align=\"center\">67.7G / 36.3s/it</td>\n </tr>\n <tr>\n <td>Q-LoRA</td><td align=\"center\">18.7G / 5.3s/it</td><td align=\"center\">18.4G / 6.3s/it</td><td align=\"center\">18.9G / 8.2s/it</td><td align=\"center\">19.9G / 11.8s/it</td><td align=\"center\">23.0G / 20.1s/it</td><td align=\"center\">27.9G / 38.3s/it</td>\n </tr>\n\t<tr>\n <th rowspan=\"2\">72B</th><td>LoRA + Deepspeed Zero3</td><td align=\"center\">215.4G / 17.6s/it</td><td align=\"center\">217.7G / 20.5s/it</td><td align=\"center\">222.6G / 29.4s/it</td><td align=\"center\">228.8G / 45.7s/it</td><td align=\"center\">249.0G / 83.4s/it</td><td align=\"center\">289.2G / 161.5s/it</td>\n </tr>\n <tr>\n <td>Q-LoRA</td><td align=\"center\">61.4G / 27.4s/it</td><td align=\"center\">61.4G / 31.5s/it</td><td align=\"center\">62.9G / 41.4s/it</td><td align=\"center\">64.1G / 59.5s/it</td><td align=\"center\">68.0G / 97.7s/it</td><td align=\"center\">75.6G / 179.8s/it</td>\n </tr>\n</table>\n<br>\n\n## Déploiement\n\n### vLLM \nPour le déploiement et l'inférence rapide, nous suggérons d'utiliser vLLM avec FastChat. Installez d'abord les paquets:\n```bash\npip install vllm\npip install \"fschat[model_worker,webui]\"\n```\nOu vous pouvez les installer à partir des sources par `git clone` et `pip install -e .`. Nous vous conseillons de lire leurs documents si vous rencontrez des problèmes lors de l'installation.\n\nPour faire fonctionner Qwen avec vLLM et FastChat, vous devez d'abord lancer un contrôleur par:\n```bash\npython -m fastchat.serve.controller\n```\n\nEnsuite, vous pouvez lancer le travailleur de modèle, ce qui signifie charger votre modèle pour l'inférence. Pour l'inférence sur un seul GPU, vous pouvez directement lancer:\n```bash\npython -m fastchat.serve.vllm_worker --model-path $model_path --trust-remote-code\n```\nCependant, si vous souhaitez exécuter le modèle sur plusieurs GPU pour une inférence plus rapide ou une mémoire plus importante, vous pouvez utiliser le parallélisme tensoriel pris en charge par vLLM. Supposons que vous exécutiez le modèle sur 4 GPU, la commande est présentée ci-dessous:\n```bash\npython -m fastchat.serve.vllm_worker --model-path $model_path --trust-remote-code --tensor-parallel-size 4\n```\n\nAprès avoir lancé votre model worker, vous pouvez lancer :\n\n* Démonstration de l'interface web\n```bash\npython -m fastchat.serve.gradio_web_server\n```\n\n* API OpenAI\n```bash\npython -m fastchat.serve.openai_api_server --host localhost --port 8000\n```\n\nCependant, si vous avez des difficultés à utiliser vLLM et FastChat, vous pouvez essayer nos méthodes les plus simples pour déployer une démo web, une démo CLI et une API.\n\n### Interface Web\n\nNous fournissons du code pour que les utilisateurs puissent construire une démo d'interface web (merci à @wysaid). Avant de commencer, assurez-vous d'installer les paquets suivants:\n\n```\npip install -r requirements_web_demo.txt\n```\n\nExécutez ensuite la commande ci-dessous et cliquez sur le lien généré:\n\n```bash\npython web_demo.py\n```\n\n<p align=\"center\">\n <br>\n <img src=\"assets/web_demo.gif\" width=\"600\" />\n <br>\n<p>\n\n### Démo CLI\n\nNous fournissons un exemple de démonstration CLI dans `cli_demo.py`, qui prend en charge la sortie en continu pour la génération. Les utilisateurs peuvent interagir avec Qwen-7B-Chat en saisissant des invites, et le modèle renvoie les sorties du modèle en mode streaming. Exécutez la commande ci-dessous:\n\n```bash\npython cli_demo.py\n```\n\n<p align=\"center\">\n <br>\n <img src=\"assets/cli_demo.gif\" width=\"600\" />\n <br>\n<p>\n<br>\n\n### API\n\nNous fournissons des méthodes pour déployer une API locale basée sur l'API OpenAI (merci à @hanpenggit). Avant de commencer, installez les paquets nécessaires:\n\n```bash\npip install fastapi uvicorn \"openai<1.0\" pydantic sse_starlette\n```\n\nExécutez ensuite la commande pour déployer votre API:\n\n```bash\npython openai_api.py\n```\n\nVous pouvez modifier vos arguments, par exemple, `-c` pour le nom ou le chemin du poids, `--cpu-only` pour le déploiement CPU, etc. Si vous rencontrez des problèmes lors du lancement du déploiement de l'API, la mise à jour des paquets vers la dernière version peut probablement les résoudre.\n\nL'utilisation de l'API est simple. Voir l'exemple ci-dessous:\n\n```python\nimport openai\nopenai.api_base = \"http://localhost:8000/v1\"\nopenai.api_key = \"none\"\n\n# create a request activating streaming response\nfor chunk in openai.ChatCompletion.create(\n model=\"Qwen\",\n messages=[\n {\"role\": \"user\", \"content\": \"你好\"}\n ],\n stream=True \n # Specifying stop words in streaming output format is not yet supported and is under development.\n):\n if hasattr(chunk.choices[0].delta, \"content\"):\n print(chunk.choices[0].delta.content, end=\"\", flush=True)\n\n# create a request not activating streaming response\nresponse = openai.ChatCompletion.create(\n model=\"Qwen\",\n messages=[\n {\"role\": \"user\", \"content\": \"你好\"}\n ],\n stream=False,\n stop=[] # You can add custom stop words here, e.g., stop=[\"Observation:\"] for ReAct prompting.\n)\nprint(response.choices[0].message.content)\n```\n\n<p align=\"center\">\n <br>\n <img src=\"assets/openai_api.gif\" width=\"600\" />\n <br>\n<p>\n\n**Function calling** est aussi supporté (mais seulement quand `stream=False` pour le moment). Voir [l'exemple d'utilisation](examples/function_call_examples.py) ici.\n<br><br>\n\n\n## 🐳 Docker\n\nPour simplifier le processus de déploiement, nous fournissons des images docker avec des environnements préconstruits : [qwenllm/qwen] (https://hub.docker.com/r/qwenllm/qwen). Il vous suffit d'installer le pilote et de télécharger les fichiers de modèle pour lancer les démonstrations, déployer l'API OpenAI et affiner le modèle.\n\n### Préparation\n\n1. Installez la version correcte du pilote Nvidia en fonction de l'image à utiliser :\n - `qwenllm/qwen:cu117` (**recommandé**): `>= 515.48.07`\n - `qwenllm/qwen:cu114` (w/o flash-attention): `>= 470.82.01`\n - `qwenllm/qwen:cu121`: `>= 530.30.02`\n - `qwenllm/qwen:latest`: même que `qwenllm/qwen:cu117`\n\n2. Installer et configurer [docker](https://docs.docker.com/engine/install/) et [nvidia-container-toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html) :\n\n```bash\n# configure docker\nsudo systemctl start docker\n# test if docker is correctly installed\nsudo docker run hello-world\n\n# configure nvidia-container-toolkit\nsudo nvidia-ctk runtime configure --runtime=docker\nsudo systemctl restart docker\n# test if nvidia-container-toolkit is correctly installed\nsudo docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi\n```\n\n3. Téléchargez les checkpoints et les codes du modèle dans votre environnement (voir [ici](#DownloadModel)).\n\n### Déploiement\n\nNous utilisons ici Qwen-7B-Chat comme exemple. Avant de lancer une démo web ou une API, vous pouvez établir la configuration comme indiqué ci-dessous :\n\n```bash\nIMAGE_NAME=qwenllm/qwen:cu117\nPORT=8901\nCHECKPOINT_PATH=/path/to/Qwen-7B-Chat # Path to downloaded model checkpoints and codes\n```\nLes scripts suivants peuvent vous aider à construire :\n\n* API OpenAI\n```bash\nbash docker/docker_openai_api.sh -i ${IMAGE_NAME} -c ${CHECKPOINT_PATH} --port ${PORT}\n```\n\n* Interface Web\n```bash\nbash docker/docker_web_demo.sh -i ${IMAGE_NAME} -c ${CHECKPOINT_PATH} --port ${PORT}\n```\n\n* Démo CLI\n```bash\nbash docker/docker_cli_demo.sh -i ${IMAGE_NAME} -c ${CHECKPOINT_PATH}\n```\n\nLes commandes ci-dessus téléchargeront automatiquement l'image requise et lanceront une démo d'interface Web en arrière-plan (le service redémarrera automatiquement). Vous pouvez ouvrir `http://localhost:${PORT}` sur l'hôte pour utiliser la démo.\n\nLa démo est lancée avec succès si vous obtenez le résultat suivant :\n\n```text\nSuccessfully started web demo. Open '...' to try!\nRun `docker logs ...` to check demo status.\nRun `docker rm -f ...` to stop and remove the demo.\n```\n\nSi vous voulez vérifier le statut de la démo, vous pouvez utiliser `docker logs qwen` pour afficher les résultats.\n\nVous pouvez utiliser `docker rm -f qwen` pour arrêter le service et supprimer le conteneur.\n\n\n### Finetuning\n\nLa méthode de finetuning utilisant l'image Docker préconstruite est fondamentalement la même que [le chapitre ci-dessus](#Finetuning) (nous avons déjà installé les dépendances dans l'image) :\n\nVoici un exemple de LoRA à une seule GPU :\n```bash\nIMAGE_NAME=qwenllm/qwen:cu117\nCHECKPOINT_PATH=/path/to/Qwen-7B # Path to downloaded model checkpoints and codes\n#CHECKPOINT_PATH=/path/to/Qwen-7B-Chat-Int4 # Path to downloaded model checkpoints and codes (Q-LoRA)\nDATA_PATH=/path/to/data/root # Prepare finetune data at ${DATA_PATH}/example.json\nOUTPUT_PATH=/path/to/output/checkpoint # Path to finetune outputs\n\n# Use all host devices by default\nDEVICE=all\n# If you need to specify GPUs for training, set device as follow (NOTE: internal quotation marks cannot be omitted)\n#DEVICE='\"device=0,1,2,3\"'\n\nmkdir -p ${OUTPUT_PATH}\n\n# Single-GPU LoRA finetuning\ndocker run --gpus ${DEVICE} --rm --name qwen \\\n --mount type=bind,source=${CHECKPOINT_PATH},target=/data/shared/Qwen/Qwen-7B \\\n --mount type=bind,source=${DATA_PATH},target=/data/shared/Qwen/data \\\n --mount type=bind,source=${OUTPUT_PATH},target=/data/shared/Qwen/output_qwen \\\n --shm-size=2gb \\\n -it ${IMAGE_NAME} \\\n bash finetune/finetune_lora_single_gpu.sh -m /data/shared/Qwen/Qwen-7B/ -d /data/shared/Qwen/data/example.json\n```\n\nPour faire un changement vers Q-LoRA à GPU unique par exemple, il suffit de modifier la commande bash à l'intérieur de `docker run` :\n```bash\nbash finetune/finetune_qlora_single_gpu.sh -m /data/shared/Qwen/Qwen-7B-Chat-Int4/ -d /data/shared/Qwen/data/example.json\n```\n<br>\n\n## 🔥 Invite du système\nQwen-1.8-Chat et Qwen-72B-Chat ont été entièrement formés à diverses invites de système avec plusieurs séries d'interactions complexes, de sorte qu'ils peuvent suivre une variété d'invites de système et réaliser la personnalisation du modèle dans le contexte, améliorant ainsi l'évolutivité de Qwen-chat.\n\nGrâce aux messages-guides du système, Qwen-Chat peut **jouer avec enthousiasme**, **transférer le style de langage**, **fixer des tâches** et **fixer des comportements**.\n\n![](assets/system_prompt_language_style.png)\n\n![](assets/system_prompt_role_play_en.png)\n\nPour plus d'informations, veuillez vous référer à la [documentation d'exemple](examples/system_prompt.md).\n\n\n## Utilisation des outils\n\nQwen-Chat a été optimisé pour l'utilisation d'outils et les capacités d'appel de fonctions. Les utilisateurs peuvent développer des agents, des applications LangChain, et même augmenter Qwen avec un Code Interpreter.\n\nNous fournissons une documentation sur la manière d'implémenter les appels d'outils basés sur le principe de ReAct Prompting, veuillez vous référer à [l'exemple ReAct](examples/react_prompt.md). Sur la base de ce principe, nous fournissons un support pour function calling dans [openai_api.py](openai_api.py).\n\nNous avons testé les capacités d'appel d'outil du modèle sur notre benchmark d'évaluation chinois à source ouverte et nous avons constaté que Qwen-Chat obtient systématiquement de bons résultats:\n\n<table>\n <tr>\n <th colspan=\"4\" align=\"center\">Chinese Tool-Use Benchmark (Version 20231206)</th>\n </tr>\n <tr>\n <th align=\"center\">Model</th><th align=\"center\">Tool Selection (Acc.↑)</th><th align=\"center\">Tool Input (Rouge-L↑)</th><th align=\"center\">False Positive Error↓</th>\n </tr>\n <tr>\n <td>GPT-4</td><td align=\"center\">98.0%</td><td align=\"center\">0.953</td><td align=\"center\">23.9%</td>\n </tr>\n <tr>\n <td>GPT-3.5</td><td align=\"center\">74.5%</td><td align=\"center\">0.807</td><td align=\"center\">80.6%</td>\n </tr>\n <tr>\n <td>Qwen-1_8B-Chat</td><td align=\"center\">85.0%</td><td align=\"center\">0.839</td><td align=\"center\">27.6%</td>\n </tr>\n <tr>\n <td>Qwen-7B-Chat</td><td align=\"center\">95.5%</td><td align=\"center\">0.900</td><td align=\"center\">11.6%</td>\n </tr>\n <tr>\n <td>Qwen-14B-Chat</td><td align=\"center\">96.9%</td><td align=\"center\">0.917</td><td align=\"center\">5.6%</td>\n </tr>\n <tr>\n <td>Qwen-72B-Chat</td><td align=\"center\">98.2%</td><td align=\"center\">0.927</td><td align=\"center\">1.1%</td>\n </tr>\n</table>\n\nPour évaluer la capacité de Qwen à utiliser l'interpréteur de code Python pour des tâches telles que la résolution de problèmes mathématiques, la visualisation de données et d'autres tâches générales telles que la manipulation de fichiers et l'exploration du Web, nous avons créé et mis en libre accès un test de référence spécialement conçu pour évaluer ces capacités. Vous pouvez trouver le benchmark sur ce [lien](https://github.com/QwenLM/Qwen-Agent/tree/main/benchmark).\n\nNous avons observé que Qwen est performant en termes d'exécutabilité du code et de précision des résultats lors de la génération du code:\n\n<table>\n <tr>\n <th colspan=\"5\" align=\"center\">Code Interpreter Benchmark (Version 20231206)</th>\n </tr>\n <tr>\n <th rowspan=\"2\" align=\"center\">Model</th>\n <th colspan=\"3\" align=\"center\">Accuracy of Code Execution Results (%)</th>\n <th colspan=\"1\" align=\"center\">Executable Rate of Code (%)</th>\n </tr>\n <tr>\n <th align=\"center\">Math↑</th><th align=\"center\">Visualization-Hard↑</th><th align=\"center\">Visualization-Easy↑</th><th align=\"center\">General↑</th>\n </tr>\n <tr>\n <td>GPT-4</td>\n <td align=\"center\">82.8</td>\n <td align=\"center\">66.7</td>\n <td align=\"center\">60.8</td>\n <td align=\"center\">82.8</td>\n </tr>\n <tr>\n <td>GPT-3.5</td>\n <td align=\"center\">47.3</td>\n <td align=\"center\">33.3</td>\n <td align=\"center\">55.7</td>\n <td align=\"center\">74.1</td>\n </tr>\n <tr>\n <td>LLaMA2-13B-Chat</td>\n <td align=\"center\">8.3</td>\n <td align=\"center\">1.2</td>\n <td align=\"center\">15.2</td>\n <td align=\"center\">48.3</td>\n </tr>\n <tr>\n <td>CodeLLaMA-13B-Instruct</td>\n <td align=\"center\">28.2</td>\n <td align=\"center\">15.5</td>\n <td align=\"center\">21.5</td>\n <td align=\"center\">74.1</td>\n </tr>\n <tr>\n <td>InternLM-20B-Chat</td>\n <td align=\"center\">34.6</td>\n <td align=\"center\">10.7</td>\n <td align=\"center\">25.1</td>\n <td align=\"center\">65.5</td>\n </tr>\n <tr>\n <td>ChatGLM3-6B</td>\n <td align=\"center\">54.2</td>\n <td align=\"center\">4.8</td>\n <td align=\"center\">15.2</td>\n <td align=\"center\">67.1</td>\n </tr>\n <tr>\n <td>Qwen-1.8B-Chat</td>\n <td align=\"center\">25.6</td>\n <td align=\"center\">21.4</td>\n <td align=\"center\">22.8</td>\n <td align=\"center\">65.5</td>\n </tr>\n <tr>\n <td>Qwen-7B-Chat</td>\n <td align=\"center\">41.9</td>\n <td align=\"center\">23.8</td>\n <td align=\"center\">38.0</td>\n <td align=\"center\">67.2</td>\n </tr>\n <tr>\n <td>Qwen-14B-Chat</td>\n <td align=\"center\">58.4</td>\n <td align=\"center\">31.0</td>\n <td align=\"center\">45.6</td>\n <td align=\"center\">65.5</td>\n </tr>\n <tr>\n <td>Qwen-72B-Chat</td>\n <td align=\"center\">72.7</td>\n <td align=\"center\">41.7</td>\n <td align=\"center\">43.0</td>\n <td align=\"center\">82.8</td>\n </tr>\n</table>\n\n<p align=\"center\">\n <br>\n <img src=\"assets/code_interpreter_showcase_001.jpg\" />\n <br>\n<p>\n\n<br>\n\n## Compréhension du Contexte Long\n\nPour augmenter la longueur du contexte et éliminer le goulot d'étranglement que constitue la longueur de la séquence d'entraînement, nous introduisons plusieurs techniques, notamment l'interpolation tenant compte des NTK, l'attention par fenêtre et la mise à l'échelle de l'attention LogN, afin d'augmenter la longueur du contexte de Qwen-14B de 2K à plus de 8K tokens, et de Qwen-1.8B/7B de 8K à 32K tokens. \n\nPour Qwen-72B, nous adaptons RoPE à des contextes plus longs avec une base rotative plus importante. Qwen-72B prend en charge la longueur de contexte maximale de 32K tokens.\n\nNous menons des expériences de modélisation du langage sur l'ensemble de données arXiv avec l'évaluation PPL et nous constatons que Qwen peut atteindre des performances exceptionnelles dans le scénario d'un contexte long. Les résultats sont présentés ci-dessous :\n\n<table>\n <tr>\n <th rowspan=\"2\">Model</th><th colspan=\"6\" align=\"center\">Sequence Length</th>\n </tr>\n <tr>\n <th align=\"center\">1024</th><th align=\"center\">2048</th><th align=\"center\">4096</th><th align=\"center\">8192</th><th align=\"center\">16384</th><th align=\"center\">32768</th>\n </tr>\n <tr>\n <td>Qwen-7B (original)</td><td align=\"center\">4.23</td><td align=\"center\">3.78</td><td align=\"center\">39.35</td><td align=\"center\">469.81</td><td align=\"center\">2645.09</td><td align=\"center\">-</td>\n </tr>\n <tr>\n <td>+ dynamic_ntk</td><td align=\"center\">4.23</td><td align=\"center\">3.78</td><td align=\"center\">3.59</td><td align=\"center\">3.66</td><td align=\"center\">5.71</td><td align=\"center\">-</td>\n </tr>\n <tr>\n <td>+ dynamic_ntk + logn</td><td align=\"center\">4.23</td><td align=\"center\">3.78</td><td align=\"center\">3.58</td><td align=\"center\">3.56</td><td align=\"center\">4.62</td><td align=\"center\">-</td>\n </tr>\n <tr>\n <td>+ dynamic_ntk + logn + window_attn</td><td align=\"center\">4.23</td><td align=\"center\">3.78</td><td align=\"center\">3.58</td><td align=\"center\">3.49</td><td align=\"center\">4.32</td><td align=\"center\">-</td>\n </tr>\n <tr>\n <tr>\n <td>Qwen-1.8B</td><td align=\"center\"><b>5.00</b></td><td align=\"center\"><b>4.48</b></td><td align=\"center\"><b>4.13</b></td><td align=\"center\"><b>3.89</b></td><td align=\"center\">17.42</td><td align=\"center\">433.85</td>\n </tr>\n <tr>\n <td>+ dynamic_ntk + logn + window_attn</td><td align=\"center\"><b>5.00</b></td><td align=\"center\"><b>4.48</b></td><td align=\"center\"><b>4.14</b></td><td align=\"center\"><b>3.93</b></td><td align=\"center\"><b>3.82</b></td><td align=\"center\"><b>3.83</b></td>\n </tr>\n <tr>\n <td>Qwen-7B</td><td align=\"center\"><b>4.23</b></td><td align=\"center\"><b>3.81</b></td><td align=\"center\"><b>3.52</b></td><td align=\"center\"><b>3.31</b></td><td align=\"center\">7.27</td><td align=\"center\">181.49</td>\n </tr>\n <tr>\n <td>+ dynamic_ntk + logn + window_attn</td><td align=\"center\"><b>4.23</b></td><td align=\"center\"><b>3.81</b></td><td align=\"center\"><b>3.52</b></td><td align=\"center\"><b>3.33</b></td><td align=\"center\"><b>3.22</b></td><td align=\"center\"><b>3.17</b></td>\n </tr>\n <tr>\n <td>Qwen-14B</td><td align=\"center\"><b>-</b></td><td align=\"center\"><b>3.46</b></td><td align=\"center\">22.79</td><td align=\"center\">334.65</td><td align=\"center\">3168.35</td><td align=\"center\">-</td>\n </tr>\n <tr>\n <td>+ dynamic_ntk + logn + window_attn</td><td align=\"center\"><b>-</b></td><td align=\"center\"><b>3.46</b></td><td align=\"center\"><b>3.29</b></td><td align=\"center\"><b>3.18</b></td><td align=\"center\">3.42</td><td align=\"center\">-</td>\n </tr>\n <tr>\n <td>Qwen-72B</td><td align=\"center\"><b>-</b></td><td align=\"center\"><b>-</b></td><td align=\"center\">-</td><td align=\"center\"><b>2.83</b></td><td align=\"center\"><b>2.73</b></td><td align=\"center\"><b>2.72</b></td>\n </tr>\n </tr>\n</table>\n\nEn outre, pour vérifier la capacité de Qwen-72B-Chat à comprendre des textes longs, nous l'avons testé sur [L-Eval] (https://arxiv.org/abs/2307.11088) (tâches fermées). Les résultats sont les suivants :\n\n| Model | Input Length | Average | Coursera | GSM | QuALITY | TOEFL | CodeU | SFcition |\n|:------------------|:------------:|:---------:|:----------:|:----------:|:----------:|:----------:|:----------:|:----------:|\n| ChatGPT-3.5-16k | 16K | 60.73 | **63.51** | **84.00** | 61.38 | 78.43 | **12.22** | 64.84 |\n| **Qwen-72B-Chat** | 32K | **62.30** | 58.13 | 76.00 | **77.22** | **86.24** | 6.66 | **69.53** |\n\nNous avons réalisé l'expérience de \"l'aiguille dans une botte de foin\" (l'idée vient de [@Greg Kamradt](https://twitter.com/GregKamradt/status/1727018183608193393)) pour tester si le modèle peut récupérer des informations à différentes positions dans les entrées de différentes longueurs, le résultat est le suivant :\n\n![](assets/qwen_72b_needle_in_a_haystack.png)\n\nLes résultats ci-dessus montrent que Qwen-72B-Chat peut récupérer avec précision des informations placées dans différentes positions dans une longueur d'entrée de 32K, ce qui prouve ses excellentes capacités de compréhension de textes longs.\n\n\n## Tokenizer\n\nNotre tokenizer basé sur tiktoken est différent des autres tokenizers, par exemple le tokenizer sentencepiece. Vous devez faire attention aux tokens spéciaux, en particulier lors de la mise au point. Pour des informations plus détaillées sur le tokenizer et son utilisation dans le cadre du finetuning, veuillez vous référer à la [documentation](tokenization_note.md).\n<br><br>\n\n## Reproduction\n\nPour reproduire les performances du modèle sur des ensembles de données de référence, nous fournissons des scripts permettant de reproduire les résultats. Consultez [eval/EVALUATION.md](eval/EVALUATION.md) pour plus d'informations. Notez que la reproduction peut entraîner de légères différences par rapport à nos résultats.\n<br><br>\n\n## FAQ\n\nSi vous rencontrez des problèmes, veuillez vous référer à la [FAQ](FAQ.md) et aux problèmes pour trouver une solution avant de lancer un nouveau problème.\n<br><br>\n\n## Citation\nSi vous trouvez notre travail utile, n'hésitez pas à nous citer.\n\n```\n@article{qwen,\n title={Qwen Technical Report},\n author={Jinze Bai and Shuai Bai and Yunfei Chu and Zeyu Cui and Kai Dang and Xiaodong Deng and Yang Fan and Wenbin Ge and Yu Han and Fei Huang and Binyuan Hui and Luo Ji and Mei Li and Junyang Lin and Runji Lin and Dayiheng Liu and Gao Liu and Chengqiang Lu and Keming Lu and Jianxin Ma and Rui Men and Xingzhang Ren and Xuancheng Ren and Chuanqi Tan and Sinan Tan and Jianhong Tu and Peng Wang and Shijie Wang and Wei Wang and Shengguang Wu and Benfeng Xu and Jin Xu and An Yang and Hao Yang and Jian Yang and Shusheng Yang and Yang Yao and Bowen Yu and Hongyi Yuan and Zheng Yuan and Jianwei Zhang and Xingxuan Zhang and Yichang Zhang and Zhenru Zhang and Chang Zhou and Jingren Zhou and Xiaohuan Zhou and Tianhang Zhu},\n journal={arXiv preprint arXiv:2309.16609},\n year={2023}\n}\n```\n<br>\n\n## Accord de Licence\n\nLe code source fourni à l'adresse <https://github.com/QwenLM/Qwen> est soumis à la licence [Apache 2.0 License](./LICENSE) qui se trouve dans le répertoire racine.\n\nLes chercheurs et les développeurs sont libres d'utiliser les codes et les poids des modèles de Qwen et de Qwen-Chat. Pour leur utilisation commerciale, veuillez consulter l'accord de licence accompagnant chaque modèle.\n\n- Qwen-72B, Qwen-14B et Qwen-7B sont sous licence [Tongyi Qianwen LICENSE AGREEMENT](./Tongyi%20Qianwen%20LICENSE%20AGREEMENT) que l'on peut trouver dans les dépôts HuggingFace et ModelScope correspondants. Pour une utilisation commerciale, veuillez remplir le formulaire ([72B](https://dashscope.console.aliyun.com/openModelApply/Qwen-72B-Chat), [14B](https://dashscope.console.aliyun.com/openModelApply/Qwen-14B-Chat), et [7B](https://dashscope.console.aliyun.com/openModelApply/qianwen)) pour en faire la demande.\n\n- Qwen-1.8B est sous licence [Tongyi Qianwen RESEARCH LICENSE AGREEMENT](./Tongyi%20Qianwen%20RESEARCH%20LICENSE%20AGREEMENT) qui peut être trouvé dans les dépôts HuggingFace et ModelScope correspondants. Pour une utilisation commerciale, veuillez nous contacter.\n<br><br>\n\n## Contactez-nous\n\nSi vous souhaitez laisser un message à notre équipe de recherche ou à notre équipe produit, rejoignez nos groupes Discord ou WeChat! N'hésitez pas non plus à envoyer un courriel à qianwen_opensource@alibabacloud.com.\n\n" }, { "path": "README_JA.md", "content": "<p align=\"left\">\n <a href=\"README_CN.md\">中文</a>  ｜  <a href=\"README.md\">English</a>  ｜  日本語 ｜  <a href=\"README_FR.md\">Français</a> ｜  <a href=\"README_ES.md\">Español</a>\n</p>\n<br><br>\n\n<p align=\"center\">\n <img src=\"https://qianwen-res.oss-cn-beijing.aliyuncs.com/logo_qwen.jpg\" width=\"400\"/>\n<p>\n<br>\n\n<p align=\"center\">\n 🤗 <a href=\"https://huggingface.co/Qwen\">Hugging Face</a>   |   🤖 <a href=\"https://modelscope.cn/organization/qwen\">ModelScope</a>   |    📑 <a href=\"https://arxiv.org/abs/2309.16609\">Paper</a>    ｜   🖥️ <a href=\"https://modelscope.cn/studios/qwen/Qwen-72B-Chat-Demo/summary\">Demo</a>\n<br>\n<a href=\"assets/wechat.png\">WeChat (微信)</a>   |   <a href=\"https://discord.gg/z3GAxXZ9Ce\">Discord</a>   ｜   <a href=\"https://dashscope.aliyun.com\">API</a> \n</p>\n<br><br>\n\n> [!Important]\n> Qwen2 が登場しました！[QwenLM/Qwen2](https://github.com/QwenLM/Qwen2) のフォローと、そこであなたの体験を共有することをお待ちしております。\n>\n> このリポジトリ ([QwenLM/Qwen](https://github.com/QwenLM/Qwen)) は、コードベースに大幅な違いがあるため、今後アクティブにメンテナンスされなくなります。\n<br>\n\n| | Qwen-Chat | Qwen-Chat (Int4) | Qwen-Chat (Int8) | Qwen |\n|-----|:------------------------------------------------------------------------------------------------------------------------------------:|:----------------------------------------------------------------------------------------------------------------------------------------------:|:---------------------------------------------------------------:|:--------------------------------------------------------------------------------------------------------------------------:|\n| 1.8B | <a href=\"https://modelscope.cn/models/qwen/Qwen-1_8B-Chat/summary\">🤖</a> <a href=\"https://huggingface.co/Qwen/Qwen-1_8B-Chat\">🤗</a> | <a href=\"https://modelscope.cn/models/qwen/Qwen-1_8B-Chat-Int4/summary\">🤖</a> <a href=\"https://huggingface.co/Qwen/Qwen-1_8B-Chat-Int4\">🤗</a> | <a href=\"https://modelscope.cn/models/qwen/Qwen-1_8B-Chat-Int8/summary\">🤖</a> <a href=\"https://huggingface.co/Qwen/Qwen-1_8B-Chat-Int8\">🤗</a> | <a href=\"https://modelscope.cn/models/qwen/Qwen-1_8B/summary\">🤖</a> <a href=\"https://huggingface.co/Qwen/Qwen-1_8B\">🤗</a> |\n| 7B | <a href=\"https://modelscope.cn/models/qwen/Qwen-7B-Chat/summary\">🤖</a> <a href=\"https://huggingface.co/Qwen/Qwen-7B-Chat\">🤗</a> | <a href=\"https://modelscope.cn/models/qwen/Qwen-7B-Chat-Int4/summary\">🤖</a> <a href=\"https://huggingface.co/Qwen/Qwen-7B-Chat-Int4\">🤗</a> | <a href=\"https://modelscope.cn/models/qwen/Qwen-7B-Chat-Int8/summary\">🤖</a> <a href=\"https://huggingface.co/Qwen/Qwen-7B-Chat-Int8\">🤗</a> | <a href=\"https://modelscope.cn/models/qwen/Qwen-7B/summary\">🤖</a> <a href=\"https://huggingface.co/Qwen/Qwen-7B\">🤗</a> |\n| 14B | <a href=\"https://modelscope.cn/models/qwen/Qwen-14B-Chat/summary\">🤖</a> <a href=\"https://huggingface.co/Qwen/Qwen-14B-Chat\">🤗</a> | <a href=\"https://modelscope.cn/models/qwen/Qwen-14B-Chat-Int4/summary\">🤖</a> <a href=\"https://huggingface.co/Qwen/Qwen-14B-Chat-Int4\">🤗</a> | <a href=\"https://modelscope.cn/models/qwen/Qwen-14B-Chat-Int8/summary\">🤖</a> <a href=\"https://huggingface.co/Qwen/Qwen-14B-Chat-Int8\">🤗</a> | <a href=\"https://modelscope.cn/models/qwen/Qwen-14B/summary\">🤖</a> <a href=\"https://huggingface.co/Qwen/Qwen-14B\">🤗</a> |\n| 72B | <a href=\"https://modelscope.cn/models/qwen/Qwen-72B-Chat/summary\">🤖</a> <a href=\"https://huggingface.co/Qwen/Qwen-72B-Chat\">🤗</a> | <a href=\"https://modelscope.cn/models/qwen/Qwen-72B-Chat-Int4/summary\">🤖</a> <a href=\"https://huggingface.co/Qwen/Qwen-72B-Chat-Int4\">🤗</a> | <a href=\"https://modelscope.cn/models/qwen/Qwen-72B-Chat-Int8/summary\">🤖</a> <a href=\"https://huggingface.co/Qwen/Qwen-72B-Chat-Int8\">🤗</a> | <a href=\"https://modelscope.cn/models/qwen/Qwen-72B/summary\">🤖</a> <a href=\"https://huggingface.co/Qwen/Qwen-72B\">🤗</a> |\n\n\n\n**Qwen-1.8B**、**Qwen-7B**、**Qwen-14B**、**Qwen-72B**の基本言語モデルである**Qwen**と、**Qwen-1.8B-Chat**、**Qwen-7B-Chat**、**Qwen-14B-Chat**、**Qwen-72B-Chat**のチャットモデルである**Qwen-Chat**をオープンソース化します。上の表にリンクがあります。リンクをクリックして、モデルカードをご確認ください。また、**[テクニカルレポート](https://arxiv.org/abs/2309.16609)**も公開しています。論文リンクをクリックしてご覧ください！\n\n簡単に説明すると、私たちは、ドメインや言語（中国語と英語を中心に）などを幅広くカバーする最大3兆トークンの多言語データに対して安定的に事前学習された強力なベース言語モデルを持っています。これらのモデルは、ベンチマークデータセットにおいて競争力のあるパフォーマンスを達成することができます。さらに、SFTとRLHFに基づく人間の嗜好に沿ったチャットモデル（まだリリースされていません）があり、チャット、コンテンツ作成、情報抽出、要約、翻訳、コーディング、数学の問題を解くなどが可能で、ツールを使ったり、エージェントとして遊んだり、コードインタプリタとして遊んだりすることもできます。\n\n\n| モデル | 発行日 | コンテキストの最大長 | システムプロンプトの強化 | 预训练されたトークンの数 | Finetuning（Q-Lora）の最小GPUメモリ使用量 | 2048トークン生成時の最小GPUメモリ使用量（Int4） | ツールの使用能力 |\n|:----------|:--------:|:----------:|:------------:|:------------:|:------------------------------:|:-----------------------------:|:--------:|\n| Qwen-1.8B | 23.11.30 | 32K | ✅ | 2.2T | 5.8GB | 2.9GB | ✅ | \n| Qwen-7B | 23.08.03 | 32K | ❎ | 2.4T | 11.5GB | 8.2GB | ✅ | \n| Qwen-14B | 23.09.25 | 8K | ❎ | 3.0T | 18.7GB | 13.0GB | ✅ |\n| Qwen-72B | 23.11.30 | 32K | ✅ | 3.0T | 61.4GB | 48.9GB | ✅ | \n\n\nこのレポでは、それを把握することができる：\n\n* Qwenのクイックスタート。\n* 量子化モデルの詳細（使用量、メモリ、推論速度など）。比較のために、BF16モデルの統計も提供します。\n* フルパラメーターチューニング、LoRA、Q-LoRAを含む、微調整に関するチュートリアル。\n* vLLMとFastChatを例に、デプロイメントについて説明します。\n* WebUI、CLIデモなど、デモの構築に関する説明。\n* あなたのモデルのためのOpenAIスタイルのAPIを構築する手順。\n* ツール使用、エージェント、コードインタプリタの Qwen の詳細。\n* ロングコンテクスト理解評価の統計\n* ライセンス契約\n* ...\n\nまた、困ったことがあれば、まずは[FAQ](FAQ.md)を参照してほしい。まだ悩んでいますか？遠慮なく私たちに問題を送ってください（より多くの人が理解できるように、英語で）！私たちを助けたいなら、遠慮なくプルリクエストを送ってください！\n\n私たちとチャットしたいですか？私たちのDiscordまたはWeChatグループへようこそ！\n<br><br>\n\n## ニュースとアップデート\n\n* 2023.11.30 🔥 3T トークンで学習し、32k コンテキストをサポートする **Qwen-72B** と **Qwen-72B-Chat** を、 **Qwen-1.8B** と **Qwen-1.8B-Chat** とともに、ModelScope と Hugging Face 上でリリースしました。また、Qwen-72B-ChatとQwen-1.8B-Chatのシステム・プロンプト機能を強化しました。[サンプル・ドキュメント](examples/system_prompt.md)を参照してください。さらに、**Ascend 910** と **Hygon DCU** での推論をサポートしました。詳細は `ascend-support` と `dcu-support` を参照してください。\n* 2023.10.17 Int8量子化モデル**Qwen-7B-Chat-Int8**と**Qwen-14B-Chat-Int8**をリリースしました。\n* 2023.9.25 🔥 Qwen-14BとQwen-14B-ChatをModelScopeとHugging Faceでリリースしました。[qwen.cpp](https://github.com/QwenLM/qwen.cpp) と [Qwen-Agent](https://github.com/QwenLM/Qwen-Agent) もリリースされました。同時に、Qwen-7B と Qwen-7B-Chat も更新しました。Qwen-7B（オリジナル）と比較して、Qwen-7Bはより多くの学習トークンを使用し、2.2Tトークンから2.4Tトークンに増加し、コンテキスト長は2048から8192に拡張された。Qwen-7Bの中国語知識とコーディング能力はさらに向上しています。最新のコードとチェックポイントをお使いください！\n* 2023.9.12 Qwen-7Bモデルにおいて、フルパラメーター・ファインチューニング、LoRA、Q-LoRAを含むファインチューニングをサポートしました。\n* 2023.8.21 Qwen-7B-Chat 用 Int4 量子化モデル **Qwen-7B-Chat-Int4** をリリースしました。また、ベンチマーク評価においても大きな性能低下は見られませんでした。\n* 2023.8.3 ModelScope と Hugging Face 上で **Qwen-7B** と **Qwen-7B-Chat** をリリースしました。また、トレーニングの詳細やモデルの性能など、モデルの詳細については技術メモを提供しています。\n<br>\n\n## 性能\n\nQwenモデルは、MMLU、C-Eval、GSM8K、MATH、HumanEval、MBPP、BBHなど、自然言語理解、数学的問題解決、コーディングなどに関するモデルの能力を評価する一連のベンチマークデータセットにおいて、同様のモデルサイズを持つベースラインモデルを上回る性能を発揮する。Qwen-72Bは全てのタスクでLLaMA2-70Bを上回り、10タスク中7タスクでGPT-3.5を上回った。\n\n\n<p align=\"left\">\n <img src=\"assets/radar_72b.jpg\" width=600px/>\n<p>\n<br>\n\n| Model | MMLU | C-Eval | GSM8K | MATH | HumanEval | MBPP | BBH | CMMLU |\n|:------------------|:--------:|:--------:|:--------:|:--------:|:---------:|:--------:|:--------:|:--------:|\n| | 5-shot | 5-shot | 8-shot | 4-shot | 0-shot | 3-shot | 3-shot | 5-shot |\n| LLaMA2-7B | 46.8 | 32.5 | 16.7 | 3.3 | 12.8 | 20.8 | 38.2 | 31.8 |\n| LLaMA2-13B | 55.0 | 41.4 | 29.6 | 5.0 | 18.9 | 30.3 | 45.6 | 38.4 |\n| LLaMA2-34B | 62.6 | - | 42.2 | 6.2 | 22.6 | 33.0 | 44.1 | - |\n| ChatGLM2-6B | 47.9 | 51.7 | 32.4 | 6.5 | - | - | 33.7 | - |\n| InternLM-7B | 51.0 | 53.4 | 31.2 | 6.3 | 10.4 | 14.0 | 37.0 | 51.8 |\n| InternLM-20B | 62.1 | 58.8 | 52.6 | 7.9 | 25.6 | 35.6 | 52.5 | 59.0 |\n| Baichuan2-7B | 54.7 | 56.3 | 24.6 | 5.6 | 18.3 | 24.2 | 41.6 | 57.1 |\n| Baichuan2-13B | 59.5 | 59.0 | 52.8 | 10.1 | 17.1 | 30.2 | 49.0 | 62.0 |\n| Yi-34B \t \t | 76.3 | 81.8 | 67.9 | 15.9 | 26.2 | 38.2 | 66.4 | 82.6 |\n| XVERSE-65B \t | 70.8 | 68.6 | 60.3 | - | 26.3 | - | - | - |\n| **Qwen-1.8B** | 45.3 | 56.1 | 32.3 | 2.3 | 15.2 | 14.2 | 22.3 | 52.1 |\n| **Qwen-7B** | 58.2 | 63.5 | 51.7 | 11.6 | 29.9 | 31.6 | 45.0 | 62.2 |\n| **Qwen-14B** | 66.3 | 72.1 | 61.3 | 24.8 | 32.3 | 40.8 | 53.4 | 71.0 |\n| **Qwen-72B** | **77.4** | **83.3** | **78.9** | **35.2** | **35.4** | **52.2** | **67.7** | **83.6** |\n\n\n比較されたすべてのモデルについて、公式に報告された結果と[OpenCompass](https://opencompass.org.cn/leaderboard-llm) の間の最高スコアを報告します。\n\nより詳細な実験結果（より多くのベンチマークデータセットでの詳細なモデル性能）や詳細については、[こちら](TODO)をクリックして技術メモを参照してください。\n<br><br>\n\n## 必要条件\n\n* python 3.8 以上\n* pytorch 1.12 以上、2.0 以上を推奨\n* transformers 4.32 以上\n* CUDA 11.4 以上を推奨（GPU ユーザー、フラッシュアテンションユーザー向けなど）\n<br>\n\n## クイックスタート\n\n以下では、Qwen-Chat と 🤖 ModelScope と 🤗 Transformers の簡単な使用例を示します。\n\n詳しくはセクション[\"ビルド済みDockerイメージの使用\"](#-docker)を参照してください。\n\nDockerを使用しない場合は、環境のセットアップと必要なパッケージのインストールが済んでいることを確認してください。上記の要件を満たしていることを確認してから、依存するライブラリをインストールしてください。\n\n```bash\npip install -r requirements.txt\n```\n\nお使いのデバイスが fp16 または bf16 をサポートしている場合、[flash-attention](https://github.com/Dao-AILab/flash-attention) （flash attention 2に対応しました）をインストールすることで、より高い効率とメモリ使用量を抑えることができます。(**flash-attention はオプションであり、インストールしなくてもプロジェクトは正常に実行できます**)\n\n```bash\ngit clone https://github.com/Dao-AILab/flash-attention\ncd flash-attention && pip install .\n# 以下はオプションです。インストールに時間がかかる場合があります。\n# pip install csrc/layer_norm\n# flash-attn のバージョンが 2.1.1 以降の場合、以下は必要ありません。\n# pip install csrc/rotary\n```\n\nこれで ModelScope か Transformers で始めることができます。\n\n### 🤗 Transformers\n\nQwen-Chat を推論に使用するには、以下のように数行のコードを入力するだけです。Qwen/Qwen-7B-Chat \"や \"Qwen/Qwen-14B-Chat \"のように、正しいモデル名やパスを渡すことを忘れないでください。**最新のコードを使用していることを確認してください。**\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\nfrom transformers.generation import GenerationConfig\n\n# Model names：\"Qwen/Qwen-7B-Chat\"、\"Qwen/Qwen-14B-Chat\"\ntokenizer = AutoTokenizer.from_pretrained(\"Qwen/Qwen-7B-Chat\", trust_remote_code=True)\n\n# bf16 を使用\n# model = AutoModelForCausalLM.from_pretrained(\"Qwen/Qwen-7B-Chat\", device_map=\"auto\", trust_remote_code=True, bf16=True).eval()\n# fp16 を使用\n# model = AutoModelForCausalLM.from_pretrained(\"Qwen/Qwen-7B-Chat\", device_map=\"auto\", trust_remote_code=True, fp16=True).eval()\n# CPU のみ使用\n# model = AutoModelForCausalLM.from_pretrained(\"Qwen/Qwen-7B-Chat\", device_map=\"cpu\", trust_remote_code=True).eval()\n# オートモードを使用すると、デバイスに応じて自動的に精度が選択されます。\nmodel = AutoModelForCausalLM.from_pretrained(\"Qwen/Qwen-7B-Chat\", device_map=\"auto\", trust_remote_code=True).eval()\n\n# 生成のためのハイパーパラメータを指定。ただし、4.32.0 以上のトTransformerを使用している場合は、これを行う必要はありません。\n# model.generation_config = GenerationConfig.from_pretrained(\"Qwen/Qwen-7B-Chat\", trust_remote_code=True)\n\n# 第一回対話ターン\nresponse, history = model.chat(tokenizer, \"你好\", history=None)\nprint(response)\n# 你好！很高兴为你提供帮助。\n\n# 第二回対話ターン\nresponse, history = model.chat(tokenizer, \"给我讲一个年轻人奋斗创业最终取得成功的故事。\", history=history)\nprint(response)\n# 这是一个关于一个年轻人奋斗创业最终取得成功的故事。\n# 故事的主人公叫李明，他来自一个普通的家庭，父母都是普通的工人。从小，李明就立下了一个目标：要成为一名成功的企业家。\n# 为了实现这个目标，李明勤奋学习，考上了大学。在大学期间，他积极参加各种创业比赛，获得了不少奖项。他还利用课余时间去实习，积累了宝贵的经验。\n# 毕业后，李明决定开始自己的创业之路。他开始寻找投资机会，但多次都被拒绝了。然而，他并没有放弃。他继续努力，不断改进自己的创业计划，并寻找新的投资机会。\n# 最终，李明成功地获得了一笔投资，开始了自己的创业之路。他成立了一家科技公司，专注于开发新型软件。在他的领导下，公司迅速发展起来，成为了一家成功的科技企业。\n# 李明的成功并不是偶然的。他勤奋、坚韧、勇于冒险，不断学习和改进自己。他的成功也证明了，只要努力奋斗，任何人都有可能取得成功。\n\n# 第三轮对话 第三回対話ターン\nresponse, history = model.chat(tokenizer, \"给这个故事起一个标题\", history=history)\nprint(response)\n# 《奋斗创业：一个年轻人的成功之路》\n```\n\nQwen の学習済みベースモデルの実行も簡単です。\n\n<details>\n <summary>Qwen の実行</summary>\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\nfrom transformers.generation import GenerationConfig\n\n# Model names：\"Qwen/Qwen-7B\"、\"Qwen/Qwen-14B\"\ntokenizer = AutoTokenizer.from_pretrained(\"Qwen/Qwen-7B\", trust_remote_code=True)\n# bf16 を使用\n# model = AutoModelForCausalLM.from_pretrained(\"Qwen/Qwen-7B\", device_map=\"auto\", trust_remote_code=True, bf16=True).eval()\n# fp16 を使用\n# model = AutoModelForCausalLM.from_pretrained(\"Qwen/Qwen-7B\", device_map=\"auto\", trust_remote_code=True, fp16=True).eval()\n# CPU のみ使用\n# model = AutoModelForCausalLM.from_pretrained(\"Qwen/Qwen-7B\", device_map=\"cpu\", trust_remote_code=True).eval()\n# オートモードを使用すると、デバイスに応じて自動的に精度が選択されます。\nmodel = AutoModelForCausalLM.from_pretrained(\"Qwen/Qwen-7B\", device_map=\"auto\", trust_remote_code=True).eval()\n\n# 生成のためのハイパーパラメータを指定。ただし、4.32.0 以上のトTransformerを使用している場合は、これを行う必要はありません。\n# model.generation_config = GenerationConfig.from_pretrained(\"Qwen/Qwen-7B-Chat\", trust_remote_code=True)\n\ninputs = tokenizer('蒙古国的首都是乌兰巴托（Ulaanbaatar）\\n冰岛的首都是雷克雅未克（Reykjavik）\\n埃塞俄比亚的首都是', return_tensors='pt')\ninputs = inputs.to(model.device)\npred = model.generate(**inputs)\nprint(tokenizer.decode(pred.cpu()[0], skip_special_tokens=True))\n# 蒙古国的首都是乌兰巴托（Ulaanbaatar）\\n冰岛的首都是雷克雅未克（Reykjavik）\\n埃塞俄比亚的首都是亚的斯亚贝巴（Addis Ababa）...\n```\n\n</details>\n\n<p id=\"DownloadModel\">\nHuggingFaceからモデルのチェックポイントとコードをダウンロードする際にネットワークの問題が発生した場合、ModelScopeからチェックポイントをダウンロードする方法はこちらでございます。\n</p>\n\n```python\nfrom modelscope import snapshot_download\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\n# Downloading model checkpoint to a local dir model_dir\n# model_dir = snapshot_download('qwen/Qwen-7B')\n# model_dir = snapshot_download('qwen/Qwen-7B-Chat')\n# model_dir = snapshot_download('qwen/Qwen-14B')\nmodel_dir = snapshot_download('qwen/Qwen-14B-Chat')\n\n# Loading local checkpoints\n# trust_remote_code is still set as True since we still load codes from local dir instead of transformers\ntokenizer = AutoTokenizer.from_pretrained(model_dir, trust_remote_code=True)\nmodel = AutoModelForCausalLM.from_pretrained(\n model_dir,\n device_map=\"auto\",\n trust_remote_code=True\n).eval()\n```\n\n### 🤖 ModelScope\n\nModelScope は、MaaS（Model-as-a-Service） のためのオープンソースプラットフォームであり、AI 開発者に柔軟で費用対効果の高いモデルサービスを提供します。同様に、以下のように ModelScope でモデルを実行することができます:\n\n```python\nfrom modelscope import AutoModelForCausalLM, AutoTokenizer\nfrom modelscope import GenerationConfig\n\n# Model names：\"Qwen/Qwen-7B-Chat\"、\"Qwen/Qwen-14B-Chat\"\ntokenizer = AutoTokenizer.from_pretrained(\"qwen/Qwen-7B-Chat\", trust_remote_code=True)\nmodel = AutoModelForCausalLM.from_pretrained(\"qwen/Qwen-7B-Chat\", device_map=\"auto\", trust_remote_code=True, fp16=True).eval()\nmodel.generation_config = GenerationConfig.from_pretrained(\"Qwen/Qwen-7B-Chat\", trust_remote_code=True) # 可指定不同的生成长度、top_p等相关超参\n\nresponse, history = model.chat(tokenizer, \"你好\", history=None)\nprint(response)\nresponse, history = model.chat(tokenizer, \"浙江的省会在哪里？\", history=history) \nprint(response)\nresponse, history = model.chat(tokenizer, \"它有什么好玩的景点\", history=history)\nprint(response)\n```\n\n### バッチ推論\nQwenはバッチ推論をサポートしている。フラッシュ・アテンションを有効にした場合、バッチ推論を使用することで40%のスピードアップが期待できる。以下にコード例を示す：\n```python\nimport torch\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\nfrom transformers import GenerationConfig\nfrom qwen_generation_utils import make_context, decode_tokens, get_stop_words_ids\n\ntokenizer = AutoTokenizer.from_pretrained(\n './',\n pad_token='<|extra_0|>',\n eos_token='<|endoftext|>',\n padding_side='left',\n trust_remote_code=True\n)\nmodel = AutoModelForCausalLM.from_pretrained(\n './',\n pad_token_id=tokenizer.pad_token_id,\n device_map=\"auto\",\n trust_remote_code=True\n).eval()\nmodel.generation_config = GenerationConfig.from_pretrained('./', pad_token_id=tokenizer.pad_token_id)\n\nall_raw_text = [\"我想听你说爱我。\", \"今天我想吃点啥，甜甜的，推荐下\", \"我马上迟到了，怎么做才能不迟到\"]\nbatch_raw_text = []\nfor q in all_raw_text:\n raw_text, _ = make_context(\n tokenizer,\n q,\n system=\"You are a helpful assistant.\",\n max_window_size=model.generation_config.max_window_size,\n chat_format=model.generation_config.chat_format,\n )\n batch_raw_text.append(raw_text)\n\nbatch_input_ids = tokenizer(batch_raw_text, padding='longest')\nbatch_input_ids = torch.LongTensor(batch_input_ids['input_ids']).to(model.device)\nbatch_out_ids = model.generate(\n batch_input_ids,\n return_dict_in_generate=False,\n generation_config=model.generation_config\n)\npadding_lens = [batch_input_ids[i].eq(tokenizer.pad_token_id).sum().item() for i in range(batch_input_ids.size(0))]\n\nbatch_response = [\n decode_tokens(\n batch_out_ids[i][padding_lens[i]:],\n tokenizer,\n raw_text_len=len(batch_raw_text[i]),\n context_length=(batch_input_ids[i].size(0)-padding_lens[i]),\n chat_format=\"chatml\",\n verbose=False,\n errors='replace'\n ) for i in range(len(all_raw_text))\n]\nprint(batch_response)\n\nresponse, _ = model.chat(tokenizer, \"我想听你说爱我。\", history=None)\nprint(response)\n\nresponse, _ = model.chat(tokenizer, \"今天我想吃点啥，甜甜的，推荐下\", history=None)\nprint(response)\n\nresponse, _ = model.chat(tokenizer, \"我马上迟到了，怎么做才能不迟到\", history=None)\nprint(response)\n```\n\n### CPU\n\nQwenとtiktokenの純粋なC++実装である [qwen.cpp](https://github.com/QwenLM/qwen.cpp) を使用することを強くお勧めします。詳細はレポを確認してください！\n\nまた、CPU上でモデルを直接実行することも簡単ですが、その場合はデバイスの指定が必要です：\n\n\n```python\nmodel = AutoModelForCausalLM.from_pretrained(\"Qwen/Qwen-7B-Chat\", device_map=\"cpu\", trust_remote_code=True).eval()\n```\n\nただし、推論効率が極端に低下する可能性があります。\n\n### 複数のGPU\n\nGPUメモリ不足に悩まされ、1つ以上のGPUでモデルを実行したい場合、Transformersでサポートされるようになったデフォルトのロード方法を直接使うことができます。以前の `utils.py` に基づく方法は非推奨です。\n\nしかし、この方法は簡単ですが、ネイティブ・パイプライン並列の効率は低いです。FastChatでvLLMを使用することをお勧めします。\n\n### DashScope\n\nAPIを通じてQwenを利用する最も簡単な方法は、Alibaba Cloudを通じたDashScope APIサービスです。その使い方を紹介します。さらに、OpenAIスタイルのAPIをご自身のサーバーにデプロイするためのスクリプトも提供しています。\n\nDashScopeはAlibaba Cloudが提供する大規模言語モデルAPIサービスで、今回Qwenに対応した。DashScopeの背後にあるモデルは、詳細が提供されていない一時的な社内バージョンであることに注意してください。サービスには `qwen-turbo` と `qwen-plus` があり、前者はより高速に動作し、後者はより優れたパフォーマンスを実現している。詳細はドキュメント [こちら](https://dashscope.aliyun.com) を参照。\n\n公式サイト [link](https://help.aliyun.com/zh/dashscope/developer-reference/activate-dashscope-and-create-an-api-key?spm=a2c4g.11186623.0.0.6c2774fahtfXdn) で DashScope アカウントを作成し、API キー (AK) を取得してください。AK は環境変数で設定することをお勧めします：\n```bash\nexport DASHSCOPE_API_KEY=\"YOUR_DASHSCOPE_API_KEY\"\n```\nその後、パッケージをインストールし、ドキュメントは [こちら](https://help.aliyun.com/zh/dashscope/developer-reference/install-dashscope-sdk) をクリックしてください。Python をお使いの場合は、pip で DashScope をインストールできます：\n```bash\npip install dashscope\n```\nJAVA SDKを使用する場合は、この方法でインストールできます：\n```xml\n<!-- https://mvnrepository.com/artifact/com.alibaba/dashscope-sdk-java -->\n<dependency>\n <groupId>com.alibaba</groupId>\n <artifactId>dashscope-sdk-java</artifactId>\n <version>the-latest-version</version>\n</dependency>\n```\nDashScope を使用する最も簡単な方法は、OpenAI API と同様のメッセージを使用する方法です。以下にその例を示す：\n```python\nimport random\nfrom http import HTTPStatus\nfrom dashscope import Generation\n\n\ndef call_with_messages():\n messages = [{'role': 'system', 'content': 'You are a helpful assistant.'},\n {'role': 'user', 'content': '如何做西红柿鸡蛋？'}]\n gen = Generation()\n response = gen.call(\n Generation.Models.qwen_turbo,\n messages=messages,\n seed=random.randint(1, 10000), # set the random seed, optional, default to 1234 if not set\n result_format='message', # set the result to be \"message\" format.\n )\n return response\n\n\nif __name__ == '__main__':\n response = call_with_messages()\n if response.status_code == HTTPStatus.OK:\n print(response)\n else:\n print('Request id: %s, Status code: %s, error code: %s, error message: %s' % (\n response.request_id, response.status_code,\n response.code, response.message\n ))\n```\n詳しい使い方は公式サイトをご覧ください。\n<br><br>\n\n\n## 量子化\n\n### GPTQ\n\n我々は、[AutoGPTQ](https://github.com/PanQiWei/AutoGPTQ)に基づいた解決策を提供し、Int4とInt8の量子化モデルをリリースすることで、ほぼ無損失なモデル効果を達成しつつ、メモリコストと推論速度の両方で性能を向上させた。\n\nここでは、量子化されたモデルを推論に使用する方法を説明する。始める前に、auto-gptqの要件を満たしていることを確認し（例：torch 2.0以上、transformers 4.32.0以上など）、必要なパッケージをインストールしてください：\n\n```bash\npip install auto-gptq optimum\n```\n\nauto-gptq`のインストールに問題がある場合は、公式の[repo](https://github.com/PanQiWei/AutoGPTQ)をチェックして、ホイールを見つけることをお勧めする。\n\n> 注意：コンパイル済みの `auto-gptq` パッケージは `torch` のバージョンと CUDA バージョンに強く依存しています。さらに、最近のアップデートにより \n> さらに、最近のアップデートにより、`transformers`、`optimum`、`peft` でサポートされていないバージョンのエラーが発生する可能性があります。\n> 以下の要件を満たす最新バージョンの使用をお勧めします：\n> - torch==2.1 auto-gptq>=0.5.1 transformers>=4.35.0 optimum>=1.14.0 peft>=0.6.1 > - torch==2.1 auto-gptq>=0.5.1 transformers>=4.35.0 optimum>=1.14.0 peft>=0.6.1\n> - torch>=2.0, <2.1 auto-gptq<0.5.0 transformers<4.35.0 optimum<1.14.0 peft>=0.5.0,<0.6.0\n\nそうすれば、量子化されたモデルを簡単にロードすることができ、いつもと同じように推論を実行することができる：\n\n```python\nmodel = AutoModelForCausalLM.from_pretrained(\n \"Qwen/Qwen-7B-Chat-Int4\",\n device_map=\"auto\",\n trust_remote_code=True\n).eval()\nresponse, history = model.chat(tokenizer, \"Hi\", history=None)\n```\n\nベンチマークにおける BF16 モデルと Int8、Int4 モデルの性能について説明します。その結果は以下に示します：\n\n| Quantization | MMLU | CEval (val) | GSM8K | Humaneval |\n|----------------------|:----:|:-----------:|:-----:|:---------:|\n| Qwen-1.8B-Chat (BF16)| 43.3 | 55.6 | 33.7 | 26.2 |\n| Qwen-1.8B-Chat (Int8)| 43.1 | 55.8 | 33.0 | 27.4 |\n| Qwen-1.8B-Chat (Int4)| 42.9 | 52.8 | 31.2 | 25.0 |\n| Qwen-7B-Chat (BF16) | 55.8 | 59.7 | 50.3 | 37.2 |\n| Qwen-7B-Chat (Int8) | 55.4 | 59.4 | 48.3 | 34.8 |\n| Qwen-7B-Chat (Int4) | 55.1 | 59.2 | 49.7 | 29.9 |\n| Qwen-14B-Chat (BF16) | 64.6 | 69.8 | 60.1 | 43.9 |\n| Qwen-14B-Chat (Int8) | 63.6 | 68.6 | 60.0 | 48.2 |\n| Qwen-14B-Chat (Int4) | 63.3 | 69.0 | 59.8 | 45.7 |\n| Qwen-72B-Chat (BF16) | 74.4 | 80.1 | 76.4 | 64.6 |\n| Qwen-72B-Chat (Int8) | 73.5 | 80.1 | 73.5 | 62.2 |\n| Qwen-72B-Chat (Int4) | 73.4 | 80.1 | 75.3 | 61.6 |\n\n### KVキャッシュ量子化\n\n> 注意: Hugging Faceの内部メカニズムにより、この機能のサポートファイル \n> (すなわち、`cache_autogptq_cuda_256.cpp`と`cache_autogptq_cuda_kernel_256.cu`)が欠落している可能性があります。以下を手動でダウンロードしてください。\n> Hugging Face Hubから手動でダウンロードし、他のモジュールファイルと同じフォルダに入れてください。\n\nアテンション KV キャッシュを量子化して圧縮して保存すると、サンプルのスループットが向上する。この機能を有効にするには、`config.json` に `use_cache_quantization` と `use_cache_kernel` という引数を指定する。\n具体的な使用方法は以下の通りである：\n\n```python\nmodel = AutoModelForCausalLM.from_pretrained(\n \"Qwen/Qwen-7B-Chat\",\n device_map=\"auto\",\n trust_remote_code=True,\n use_cache_quantization=True,\n use_cache_kernel=True,\n use_flash_attn=False\n)\n```\n\n注意： 現在、KVキャッシュの量子化とフラッシュ・アテンションを同時に使用することはできない。\nKV キャッシュの量子化とフラッシュ・アテンションを同時に有効にした場合（`use_flash_attn=True`, `use_cache_quantization=True`, `use_cache_kernel=True`）、デフォルトでは `use_flash_attn` は無効になる（`use_flash_attn=false`）。\n\n量子化されたint8-kvcacheモデルを使用しても、下流の評価で大幅な性能低下がないことを確認しました。以下では、さまざまな条件下でのメモリフットプリントのプロファイリングに焦点を当てます。\nプロファイリングは、PyTorch 2.0.1とCUDA 11.4を搭載したシングルA100-SXM4-80G GPUで実行しました。\nデフォルトで1024トークンを生成するためにBF16モデルを使用し、\"OOM \"はメモリ不足エラーを示します。\n\nKVキャッシュの量子化により、モデルはより大きなバッチサイズ（bs）で推論することができる。\n\n| USE KV Cache | bs=1 | bs=4 | bs=16 | bs=32 | bs=64 | bs=100 |\n|--------------|:------:|:------:|:------:|:------:|:------:|:------:|\n| No | 16.3GB | 24.1GB | 31.7GB | 48.7GB | OOM | OOM |\n| Yes | 15.5GB | 17.2GB | 22.3GB | 30.2GB | 48.2GB | 72.4GB |\n\nKVキャッシュ量子化により、推論段階でより長いシーケンス（`sl`, シーケンス長、生成されるトークン数を指す）を生成する際、モデルはより多くのメモリを節約することができる。\n\n| USE KV Cache | sl=512 | sl=1024 | sl=2048 | sl=4096 | sl=8192 |\n|--------------|:------:|:-------:|:-------:|:-------:|:-------:|\n| No | 15.2GB | 16.3GB | 17.6GB | 19.5GB | 23.2GB |\n| Yes | 15GB | 15.5GB | 15.8GB | 16.6GB | 17.6GB |\n\nKVキャッシュ量子化モデルでは、layer-pastのフォーマットをfloatからint8に変換し、量子化された `layer-past` には量子化パラメータも格納される。\n\n具体的な手順は以下の通り：\n\n1. key/valueの量子化を行います。\n```\n qv,scale,zero_point=quantize_cache_v(v)\n```\n\n2. `layer_past`に保存します。\n\n量子化されたの`layer-past`は:\n```\n layer_past=((q_key,key_scale,key_zero_point),\n (q_value,value_scale,value_zero_point))\n```\n`layer_past`の元のフォーマットは以下の通りである：\n```\n layer_past=(key,value)\n```\n量子化されたアテンションKVを使用したい場合、 \nInt8のkey/valueをfloatフォーマットに戻すには、以下のように逆量子化操作を使用します：\n```\n v=dequantize_cache_torch(qv,scale,zero_point)\n```\n<br>\n\n## 推論パフォーマンス\n\nこのセクションでは、さまざまな精度のモデルのスピードとメモリの統計情報を提供する。スピードとメモリーのプロファイリングは[このスクリプト](https://qianwen-res.oss-cn-beijing.aliyuncs.com/profile.py)を使用しています。\n\nBF16、Int8、および Int4 のモデルを使用して 2048 を生成する際の平均推論速度 (トークン/秒) と GPU メモリ使用量を測定しました。\n\n<table>\n <tr>\n <td>Model Size</td>\n <td>Quantization</td>\n <td>Speed (Tokens/s)</td>\n <td>GPU Memory Usage</td>\n </tr>\n <tr>\n <td rowspan=\"3\">1.8B</td>\n <td>BF16</td>\n <td>54.09</td>\n <td>4.23GB</td>\n </tr>\n <tr>\n <td>Int8</td>\n <td>55.56</td>\n <td>3.48GB</td>\n </tr>\n <tr>\n <td>Int4</td>\n <td>71.07</td>\n <td>2.91GB</td>\n </tr>\n <tr>\n <td rowspan=\"3\">7B</td>\n <td>BF16</td>\n <td>40.93</td>\n <td>16.99GB</td>\n </tr>\n <tr>\n <td>Int8</td>\n <td>37.47</td>\n <td>11.20GB</td>\n </tr>\n <tr>\n <td>Int4</td>\n <td>50.09</td>\n <td>8.21GB</td>\n </tr>\n <tr>\n <td rowspan=\"3\">14B</td>\n <td>BF16</td>\n <td>32.22</td>\n <td>30.15GB</td>\n </tr>\n <tr>\n <td>Int8</td>\n <td>29.28</td>\n <td>18.81GB</td>\n </tr>\n <tr>\n <td>Int4</td>\n <td>38.72</td>\n <td>13.01GB</td>\n </tr>\n <tr>\n <td rowspan=\"3\">72B</td>\n <td>BF16</td>\n <td>8.48</td>\n <td>144.69GB (2xA100)</td>\n </tr>\n <tr>\n <td>Int8</td>\n <td>9.05</td>\n <td>81.27GB (2xA100)</td>\n </tr>\n <tr>\n <td>Int4</td>\n <td>11.32</td>\n <td>48.86GB</td>\n </tr>\n <tr>\n <td>72B + vLLM</td>\n <td>BF16</td>\n <td>17.60</td>\n <td>2xA100</td>\n </tr>\n</table>\n\nプロファイリングは、PyTorch 2.0.1、CUDA 11.8、および Flash-Attendant 2 を備えた単一の A100-SXM4-80G GPU (2xA100 について言及されている場合を除く) で実行されます。(72B + vLLM は PyTorch 2.1.0 および Cuda 11.8 を使用します。) 推論速度 は、エンコードされ生成されたトークンの平均である。\n\n注意：上記のInt4/Int8モデルの推論速度は、autogptqを使用しています。現在、``AutoModelForCausalLM.from_pretrained``で読み込まれるモデルの推論速度は約20%遅くなります。この問題はHuggingFaceチームに報告済みであり、解決策があれば即座に更新されます。\n\nまた、コンテキストと生成の長さ、Flash Attention バージョンのさまざまな設定で推論速度と GPU メモリ使用量も測定します。 結果は、Hugging Face または ModelScope の対応するモデルカードで確認できます。\n\n## ファインチューニング\n\n### 使用方法\n現在、公式のトレーニングスクリプト `finetune.py` を提供しています。さらに、finetune.pyのシェルスクリプトを提供し、finetune.pyを実行することで、finetune.pyを起動することができる。さらに、安心してファインチューニングを開始するためのシェルスクリプトも提供しています。このスクリプトは、[DeepSpeed](https://github.com/microsoft/DeepSpeed) (注意：これはpydanticの最新バージョンとコンフリクトする可能性があるので、`pydantic<2.0`にする必要があります) および [FSDP](https://engineering.fb.com/2021/07/15/open-source/fsdp/) を使用したトレーニングをサポートします。弊社が提供するシェル・スクリプトは DeepSpeed と Peft を使用するため、事前に DeepSpeed と Peft をインストールすることをお勧めします：\n```bash\npip install \"peft<0.8.0\" deepspeed\n```\n\n学習データを準備するには、すべてのサンプルをリストにまとめ、jsonファイルに保存する必要があります。各サンプルはidと会話リストで構成される辞書です。以下は1つのサンプルを含む単純なリストの例です：\n\n```json\n[\n {\n \"id\": \"identity_0\",\n \"conversations\": [\n {\n \"from\": \"user\",\n \"value\": \"你好\"\n },\n {\n \"from\": \"assistant\",\n \"value\": \"我是一个语言模型，我叫通义千问。\"\n }\n ]\n }\n]\n```\n\nデータ準備の後、提供されているシェルスクリプトを使って微調整を実行することができる。データファイルのパス `$DATA` を忘れずに指定してください。\n\nファインチューニングのスクリプトを使用することで、以下のことが可能になる：\n- フルパラメーター・ファインチューニング\n- LoRA\n- Q-LoRA\n\nフルパラメータパラメータのファインチューニングを行うには、トレーニングプロセス全体ですべてのパラメータを更新する必要があります。トレーニングを開始するには、以下のスクリプトを実行します：\n\n```bash\n# 分散トレーニング。GPUメモリが不足するとトレーニングが破綻するため、シングルGPUのトレーニングスクリプトは提供していません。\nbash finetune/finetune_ds.sh\n```\n\nシェルスクリプトでは、正しいモデル名またはパス、データパス、出力ディレクトリを指定することを忘れないでください。このスクリプトでは DeepSpeed ZeRO 3 を使用しています。変更したい場合は、引数 `--deepspeed` を削除するか、要件に基づいて DeepSpeed 設定 json ファイルを変更してください。さらに、このスクリプトは混合精度のトレーニングに対応しており、`--bf16 True` または `--fp16 True` を使用することができます。fp16を使用する場合は、混合精度のトレーニングのため、DeepSpeedを使用することを忘れないこと。経験的に、あなたのマシンがbf16をサポートしている場合、私たちのプリトレーニングとアライメントを整合させるためにbf16を使用することをお勧めします。\n\n同様に、LoRAを実行するには、以下のように別のスクリプトを使って実行する。始める前に、`peft`がインストールされていることを確認してください。また、モデル、データ、出力へのパスを指定する必要があります。学習済みモデルには絶対パスを使用することをお勧めします。なぜなら、LoRAはアダプタのみを保存し、アダプタ設定jsonファイルの絶対パスは、ロードする事前学習済みモデルを見つけるために使用されるからです。また、このスクリプトはbf16とfp16の両方をサポートしている。\n\n```bash\n# シングルGPUトレーニング\nbash finetune/finetune_lora_single_gpu.sh\n# 分散トレーニング\nbash finetune/finetune_lora_ds.sh\n```\n\nLoRA ([論文](https://arxiv.org/abs/2106.09685)) は、フルパラメーターによるファインチューニングと比較して、adapterのパラメーターを更新するだけで、元の大きな言語モデル層は凍結されたままである。そのため、メモリコストが大幅に削減でき、計算コストも削減できる。\n\nなお、チャットモデル（Qwen-7B-Chatなど）ではなく、ベース言語モデル（Qwen-7Bなど）の微調整にLoRAを使用した場合、スクリプトは自動的に学習可能なパラメータとして埋め込み層と出力層を切り替えます。これは、ベースとなる言語モデルには、ChatMLフォーマットによってもたらされる特殊なトークンに関する知識がないためです。したがって、これらのレイヤーは、モデルがトークンを理解し予測するために更新される必要があります。別の言い方をすれば、もしLoRAで特殊なトークンを学習するのであれば、コード内で `modules_to_save` を設定することで、レイヤーを学習可能なパラメータに設定する必要があります。また、これらのパラメータが学習可能な場合、ZeRO 3 を使用することはできません。新しいトレーニング可能なパラメータがない場合は、DeepSpeed 設定ファイルを変更することで ZeRO 3 に切り替えることができます。さらに、LoRAのメモリフットプリントは、このような学習可能なパラメータがある場合とない場合で、大きな開きがあることがわかります。そのため、メモリに問題がある場合は、LoRAのChatモデルを微調整することをお勧めします。詳細は以下のプロファイルを参照してください。\n\nしかし、それでもメモリ不足に悩む場合は、Q-LoRA（[論文](https://arxiv.org/abs/2305.14314)）を検討することができます。これは、量子化されたラージ言語モデルと、ページド・アテンションなどの他のテクニックを使用し、さらに少ないメモリコストで実行することができます。\n\n注：シングル GPU Q-LoRA トレーニングを実行するには、`pip` または `conda` を使って `mpi4py` をインストールする必要があるかもしれない。\n\nQ-LoRAを実行するには、以下のスクリプトを直接実行してください：\n\n```bash\n# シングルGPUトレーニング\nbash finetune/finetune_qlora_single_gpu.sh\n# 分散トレーニング\nbash finetune/finetune_qlora_ds.sh\n```\n\nQ-LoRAについては、弊社が提供する量子化モデル、例えばQwen-7B-Chat-Int4をロードすることをお勧めします。BF16モデルは使用し**ない**でください！フルパラメータ・ファインチューニングやLoRAとは異なり、Q-LoRAではfp16のみがサポートされる。シングルGPUのトレーニングでは、トーチアンプによるエラーが観測されたため、混合精度のトレーニングにはDeepSpeedを使用する必要がある。また、Q-LoRAの場合、LoRAの特殊トークンの問題が残っています。しかし、Q-LoRAではチャットモデルとしてInt4モデルのみを提供しており、言語モデルはChatML形式の特殊トークンを学習しているため、レイヤーの心配はありません。なお、Int4モデルのレイヤーは学習できないはずなので、学習で特殊なトークンを導入すると、Q-LoRAが動作しなくなる可能性があります。\n\nLoRAとQ-LoRAの学習は、フルパラメータによるファインチューニングとは異なり、アダプターパラメータのみを保存する。仮にQwen-7Bから学習を開始したとすると、以下のようにファインチューニングされたモデルを読み込んで推論を行うことができる：\n\n```python\nfrom peft import AutoPeftModelForCausalLM\n\nmodel = AutoPeftModelForCausalLM.from_pretrained(\n path_to_adapter, # path to the output directory\n device_map=\"auto\",\n trust_remote_code=True\n).eval()\n```\n\nアダプターをマージし、微調整したモデルをスタンドアロンモデルとして保存したい場合は（これは LoRA でのみ可能で、Q-LoRA からパラメータをマージすることはできません）、以下のコードを実行します：\n\n```python\nfrom peft import AutoPeftModelForCausalLM\n\nmodel = AutoPeftModelForCausalLM.from_pretrained(\n path_to_adapter, # path to the output directory\n device_map=\"auto\",\n trust_remote_code=True\n).eval()\n\nmerged_model = model.merge_and_unload()\n# max_shard_size and safe serialization are not necessary. \n# They respectively work for sharding checkpoint and save the model to safetensors\nmerged_model.save_pretrained(new_model_directory, max_shard_size=\"2048MB\", safe_serialization=True)\n```\n\n`new_model_directory` ディレクトリには、マージされたモデルの重みとモジュール ファイルが含まれます。 保存されたファイルに `*.cu` および `*.cpp` ファイルが存在しない可能性があることに注意してください。 KVキャッシュ機能を使用したい場合は、手動でコピーしてください。 また、このステップではトークナイザー ファイルは新しいディレクトリに保存されません。 トークナイザー ファイルをコピーするか、次のコードを使用できます。\n```python\nfrom transformers import AutoTokenizer\n\ntokenizer = AutoTokenizer.from_pretrained(\n path_to_adapter, # path to the output directory\n trust_remote_code=True\n)\n\ntokenizer.save_pretrained(new_model_directory)\n```\n\n注意：マルチGPUトレーニングの場合、分散トレーニング用の適切なハイパーパラメータをマシンに応じて指定する必要があります。また、データ、メモリフットプリント、トレーニング速度を考慮して、引数 `--model_max_length` で最大シーケンス長を指定することをお勧めします。\n\n### メモリと速度のプロファイリング\nシングルGPUトレーニングのセットアップにおいて、LoRA (LoRA(emb)はembeddingと出力層を学習させるが、LoRAはembeddingと出力層を学習させない) とQ-LoRAのGPUメモリとトレーニング速度をプロファイリングする。このテストでは、シングルA100-SXM4-80G GPUで実験し、CUDA 11.8とPytorch 2.0を使用します。Flash attention 2を使用します。256、512、1024、2048、4096、8192という異なる長さの入力のメモリ（GB）と速度（s/iter）をプロファイリングします。また、2台のA100 GPUを用いたQwen-7Bによるフルパラメータ・ファインチューニングの統計量も報告する。GPUメモリの制限のため、256、512、1024トークンの統計のみを報告する。\n\n\nQwen-72B については、2 つの方法で実験します。1) 4 つの A100-SXM4-80G GPU での Lora 微調整 + DeepSpeed ZeRO 3、および 2) 1 つの A100-SXM4-80G GPU での QLora (int4) 微調整。 OOM は、LoRA (emb) 微調整と Deepspeed ZeRO 3 を使用しない LoRA 微調整の両方で 4 つの A100-SXM4-80G GPU で発生することに注意してください (`--deepspeedfinetune/ds_config_zero3.json` を [`finetune/finetune_lora_ds に渡すことができます) .sh`](finetune/finetune_lora_ds.sh) を使用して DeepSpeed ZeRO 3 を有効にします)。\n\n統計量を以下に示す：\n\n<table>\n <tr>\n <th rowspan=\"2\">Model Size</th><th rowspan=\"2\">Method</th><th colspan=\"6\" align=\"center\">Sequence Length</th>\n </tr>\n <tr>\n <th align=\"center\">256</th><th align=\"center\">512</th><th align=\"center\">1024</th><th align=\"center\">2048</th><th align=\"center\">4096</th><th align=\"center\">8192</th>\n </tr>\n </tr>\n </tr>\n\t\t<tr>\n <th rowspan=\"4\">1.8B</th><td>LoRA</td><td align=\"center\">6.7G / 1.0s/it</td><td align=\"center\">7.4G / 1.0s/it</td><td align=\"center\">8.4G / 1.1s/it</td><td align=\"center\">11.0G / 1.7s/it</td><td align=\"center\">16.2G / 3.3s/it</td><td align=\"center\">21.8G / 6.8s/it</td>\n </tr>\n <tr>\n <td>LoRA (emb)</td><td align=\"center\">13.7G / 1.0s/it</td><td align=\"center\">14.0G / 1.0s/it</td><td align=\"center\">14.0G / 1.1s/it</td><td align=\"center\">15.1G / 1.8s/it</td><td align=\"center\">19.7G / 3.4s/it</td><td align=\"center\">27.7G / 7.0s/it</td>\n </tr>\n <tr>\n <td>Q-LoRA</td><td align=\"center\">5.8G / 1.4s/it</td><td align=\"center\">6.0G / 1.4s/it</td><td align=\"center\">6.6G / 1.4s/it</td><td align=\"center\">7.8G / 2.0s/it</td><td align=\"center\">10.2G / 3.4s/it</td><td align=\"center\">15.8G / 6.5s/it</td>\n </tr>\n <tr>\n <td>Full-parameter</td><td align=\"center\">43.5G / 2.1s/it</td><td align=\"center\">43.5G / 2.2s/it</td><td align=\"center\">43.5G / 2.2s/it</td><td align=\"center\">43.5G / 2.3s/it</td><td align=\"center\">47.1G / 2.8s/it</td><td align=\"center\">48.3G / 5.6s/it</td>\n </tr>\n <tr>\n <th rowspan=\"4\">7B</th><td>LoRA</td><td align=\"center\">20.1G / 1.2s/it</td><td align=\"center\">20.4G / 1.5s/it</td><td align=\"center\">21.5G / 2.8s/it</td><td align=\"center\">23.8G / 5.2s/it</td><td align=\"center\">29.7G / 10.1s/it</td><td align=\"center\">36.6G / 21.3s/it</td>\n </tr>\n <tr>\n <td>LoRA (emb)</td><td align=\"center\">33.7G / 1.4s/it</td><td align=\"center\">34.1G / 1.6s/it</td><td align=\"center\">35.2G / 2.9s/it</td><td align=\"center\">35.1G / 5.3s/it</td><td align=\"center\">39.2G / 10.3s/it</td><td align=\"center\">48.5G / 21.7s/it</td>\n </tr>\n <tr>\n <td>Q-LoRA</td><td align=\"center\">11.5G / 3.0s/it</td><td align=\"center\">11.5G / 3.0s/it</td><td align=\"center\">12.3G / 3.5s/it</td><td align=\"center\">13.9G / 7.0s/it</td><td align=\"center\">16.9G / 11.6s/it</td><td align=\"center\">23.5G / 22.3s/it</td>\n </tr>\n <tr>\n <td>Full-parameter</td><td align=\"center\">139.2G / 4.0s/it</td><td align=\"center\">148.0G / 4.0s/it</td><td align=\"center\">162.0G / 4.5s/it</td><td align=\"center\">-</td><td align=\"center\">-</td><td align=\"center\">-</td>\n </tr>\n <tr>\n <th rowspan=\"3\">14B</th><td>LoRA</td><td align=\"center\">34.6G / 1.6s/it</td><td align=\"center\">35.1G / 2.4s/it</td><td align=\"center\">35.3G / 4.4s/it</td><td align=\"center\">37.4G / 8.4s/it</td><td align=\"center\">42.5G / 17.0s/it</td><td align=\"center\">55.2G / 36.0s/it</td>\n </tr>\n <tr>\n <td>LoRA (emb)</td><td align=\"center\">51.2 / 1.7s/it</td><td align=\"center\">51.1G / 2.6s/it</td><td align=\"center\">51.5G / 4.6s/it</td><td align=\"center\">54.1G / 8.6s/it</td><td align=\"center\">56.8G / 17.2s/it</td><td align=\"center\">67.7G / 36.3s/it</td>\n </tr>\n <tr>\n <td>Q-LoRA</td><td align=\"center\">18.7G / 5.3s/it</td><td align=\"center\">18.4G / 6.3s/it</td><td align=\"center\">18.9G / 8.2s/it</td><td align=\"center\">19.9G / 11.8s/it</td><td align=\"center\">23.0G / 20.1s/it</td><td align=\"center\">27.9G / 38.3s/it</td>\n </tr>\n\t<tr>\n <th rowspan=\"2\">72B</th><td>LoRA + Deepspeed Zero3</td><td align=\"center\">215.4G / 17.6s/it</td><td align=\"center\">217.7G / 20.5s/it</td><td align=\"center\">222.6G / 29.4s/it</td><td align=\"center\">228.8G / 45.7s/it</td><td align=\"center\">249.0G / 83.4s/it</td><td align=\"center\">289.2G / 161.5s/it</td>\n </tr>\n <tr>\n <td>Q-LoRA</td><td align=\"center\">61.4G / 27.4s/it</td><td align=\"center\">61.4G / 31.5s/it</td><td align=\"center\">62.9G / 41.4s/it</td><td align=\"center\">64.1G / 59.5s/it</td><td align=\"center\">68.0G / 97.7s/it</td><td align=\"center\">75.6G / 179.8s/it</td>\n </tr>\n</table>\n<br>\n\n## デプロイ\n\n### vLLM \nデプロイメントと高速推論のためには、vLLMを使用することをお勧めします。\n\n**CUDA 12.1** および **PyTorch 2.1** を使用している場合は、次のコマンドを直接使用して vLLM をインストールできます。\n```bash\npip install vllm\n```\n\nそれ以外の場合は、公式 vLLM [インストール手順](https://docs.vllm.ai/en/latest/getting_started/installation.html) を参照してください。\n\n#### vLLM + Transformer Wrapper\n\n[ラッパー コード](examples/vllm_wrapper.py) をダウンロードし、複数ラウンドの対話対話のために次のコマンドを実行できます。 (注: 現在は ``model.chat()`` メソッドのみをサポートしています。)\n\n```python\nfrom vllm_wrapper import vLLMWrapper\n\nmodel = vLLMWrapper('Qwen/Qwen-7B-Chat', tensor_parallel_size=1)\n# model = vLLMWrapper('Qwen/Qwen-7B-Chat-Int4', tensor_parallel_size=1, dtype=\"float16\")\n\nresponse, history = model.chat(query=\"你好\", history=None)\nprint(response)\nresponse, history = model.chat(query=\"给我讲一个年轻人奋斗创业最终取得成功的故事。\", history=history)\nprint(response)\nresponse, history = model.chat(query=\"给这个故事起一个标题\", history=history)\nprint(response)\n```\n#### vLLM + Web デモ / OpenAI API\nFastChat を使用して、Web デモまたは OpenAI API サーバーを起動できます。 まず、FastChat をインストールします。\n```\npip install \"fschat[model_worker,webui]\"\n```\n\nvLLM および FastChat で Qwen を実行するには、次の方法でコントローラーを起動する必要があります。\n```bash\npython -m fastchat.serve.controller\n```\n\nそれからmodel workerを起動し、推論のためにモデルをロードします。シングルGPU推論の場合は、直接実行できます：\n```bash\npython -m fastchat.serve.vllm_worker --model-path $model_path --trust-remote-code --dtype bfloat16\n# python -m fastchat.serve.vllm_worker --model-path $model_path --trust-remote-code --dtype float16 # INT4モデルを実行します\n```\nしかし、より高速な推論や大容量メモリーのために複数のGPUでモデルを実行したい場合は、vLLMがサポートするテンソル並列を使用することができます。モデルを4GPUで実行するとすると、コマンドは以下のようになります：\n```bash\npython -m fastchat.serve.vllm_worker --model-path $model_path --trust-remote-code --tensor-parallel-size 4 --dtype bfloat16\n# python -m fastchat.serve.vllm_worker --model-path $model_path --trust-remote-code --tensor-parallel-size 4 --dtype float16 # run int4 model # INT4モデルを実行します\n```\n\nモデルワーカーを起動した後、起動することができます：\n\n* Web UI Demo\n```bash\npython -m fastchat.serve.gradio_web_server\n```\n\n* OpenAI API\n```bash\npython -m fastchat.serve.openai_api_server --host localhost --port 8000\n```\n\nただし、vLLM と FastChat の使用が難しい場合は、Web デモ、CLI デモ、および API をデプロイするために提供されている最も簡単な方法を試すことができます。\n\n\n### ウェブ UI\n\nウェブ UI デモを構築するためのコードを提供します（@wysaid に感謝）。これを始める前に、以下のパッケージがインストールされていることを確認してください:\n\n```bash\npip install -r requirements_web_demo.txt\n```\n\nそして、以下のコマンドを実行し、生成されたリンクをクリックします:\n\n```bash\npython web_demo.py\n```\n\n<p align=\"center\">\n <br>\n <img src=\"assets/web_demo.gif\" width=\"600\" />\n <br>\n<p>\n\n### CLI デモ\n\n`cli_demo.py` に CLI のデモ例を用意しています。ユーザはプロンプトを入力することで Qwen-7B-Chat と対話することができ、モデルはストリーミングモードでモデルの出力を返します。以下のコマンドを実行する:\n\n```\npython cli_demo.py\n```\n\n<p align=\"center\">\n <br>\n <img src=\"assets/cli_demo.gif\" width=\"600\" />\n <br>\n<p>\n<br>\n\n### API\n\nOpenAI API をベースにローカルAPIをデプロイする方法を提供する（@hanpenggit に感謝）。始める前に、必要なパッケージをインストールしてください:\n\n```bash\npip install fastapi uvicorn \"openai<1.0\" pydantic sse_starlette\n```\n\nそれから、API をデプロイするコマンドを実行します:\n\n```bash\npython openai_api.py\n```\n\nチェックポイント名やパスには `-c`、CPU デプロイメントには `--cpu-only` など、引数を変更できます。API デプロイメントを起動する際に問題が発生した場合は、パッケージを最新バージョンに更新することで解決できる可能性があります。\n\nAPI の使い方も簡単です。以下の例をご覧ください:\n\n```python\nimport openai\nopenai.api_base = \"http://localhost:8000/v1\"\nopenai.api_key = \"none\"\n\n# ストリーミングレスポンスを有効化するリクエストを作成する\nfor chunk in openai.ChatCompletion.create(\n model=\"Qwen\",\n messages=[\n {\"role\": \"user\", \"content\": \"你好\"}\n ],\n stream=True\n # ストリーミング出力形式でのストップワードの指定はまだサポートされておらず、開発中です。\n):\n if hasattr(chunk.choices[0].delta, \"content\"):\n print(chunk.choices[0].delta.content, end=\"\", flush=True)\n\n# ストリーミングレスポンスを有効化しないリクエストを作成する\nresponse = openai.ChatCompletion.create(\n model=\"Qwen\",\n messages=[\n {\"role\": \"user\", \"content\": \"你好\"}\n ],\n stream=False,\n stop=[] # 例えば、stop=[\"Observation:\"] (ReAct プロンプトの場合)。\n)\nprint(response.choices[0].message.content)\n```\n\n<p align=\"center\">\n <br>\n <img src=\"assets/openai_api.gif\" width=\"600\" />\n <br>\n<p>\n\n**Function Calling** もサポートされています(ただし、今のところ `stream=False` の場合のみ)。使用例](examples/function_call_examples.py) を参照してください。\n<br><br>\n\n## 🐳 Docker\n\nデプロイプロセスを簡素化するために、あらかじめ環境を構築した docker イメージを提供しています： [qwenllm/qwen](https://hub.docker.com/r/qwenllm/qwen)。ドライバを導入し、モデルファイルをダウンロードするだけで、デモを起動し、OpenAI APIをデプロイし、モデルを微調整することができます。\n\n### 準備\n\n1. 使用するイメージに応じて、正しいバージョンのNvidiaドライバをインストールしてください：\n - `qwenllm/qwen:cu117` (**recommend**): `>= 515.48.07`\n - `qwenllm/qwen:cu114` (w/o flash-attention): `>= 470.82.01`\n - `qwenllm/qwen:cu121`: `>= 530.30.02`\n - `qwenllm/qwen:latest`: same as `qwenllm/qwen:cu117`\n\n2. [Docker](https://docs.docker.com/engine/install/) と [nvidia-container-toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html) をインストールして設定します：\n\n```bash\n# configure docker\nsudo systemctl start docker\n# test if docker is correctly installed\nsudo docker run hello-world\n\n# configure nvidia-container-toolkit\nsudo nvidia-ctk runtime configure --runtime=docker\nsudo systemctl restart docker\n# test if nvidia-container-toolkit is correctly installed\nsudo docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi\n```\n\n3. モデルのチェックポイントとコードを環境にダウンロードします（[こちら](#DownloadModel)を参照）。\n\n### デプロイ\n\nここでは例として Qwen-7B-Chat を使用する。ウェブ・デモや API を起動する前に、以下のように設定を行います：\n\n```bash\nIMAGE_NAME=qwenllm/qwen:cu117\nPORT=8901\nCHECKPOINT_PATH=/path/to/Qwen-7B-Chat # Path to downloaded model checkpoints and codes\n```\n以下のスクリプトがビルドに役立つ：\n\n* OpenAI API\n```bash\nbash docker/docker_openai_api.sh -i ${IMAGE_NAME} -c ${CHECKPOINT_PATH} --port ${PORT}\n```\n\n* Web UI\n```bash\nbash docker/docker_web_demo.sh -i ${IMAGE_NAME} -c ${CHECKPOINT_PATH} --port ${PORT}\n```\n\n* CLI Demo\n```bash\nbash docker/docker_cli_demo.sh -i ${IMAGE_NAME} -c ${CHECKPOINT_PATH}\n```\n\n上記のコマンドは自動的に必要なイメージをダウンロードし、バックグラウンドでWeb UIデモを起動します（サービスは自動で再起動します）。デモを使用するには、ホスト上で `http://localhost:${PORT}` を開いてください。\n\n以下の出力が表示されれば、デモは正常に起動しています：\n\n```text\nSuccessfully started web demo. Open '...' to try!\nRun `docker logs ...` to check demo status.\nRun `docker rm -f ...` to stop and remove the demo.\n```\n\nデモの状態を確認したい場合は、`docker logs qwen` を使って出力を表示できる。\n\ndocker rm -f qwen` でサービスを停止し、コンテナを削除できる。\n\n\n### ファインチューニング\n\nビルド済みのDockerイメージを利用したファインチューニングの方法は、基本的に[前章](#Finetuning)と同じです(すでにイメージに依存関係がインストールされています)：\n\n以下はシングルGPUのLoRAの例です：\n```bash\nIMAGE_NAME=qwenllm/qwen:cu117\nCHECKPOINT_PATH=/path/to/Qwen-7B # Path to downloaded model checkpoints and codes\n#CHECKPOINT_PATH=/path/to/Qwen-7B-Chat-Int4 # Path to downloaded model checkpoints and codes (Q-LoRA)\nDATA_PATH=/path/to/data/root # Prepare finetune data at ${DATA_PATH}/example.json\nOUTPUT_PATH=/path/to/output/checkpoint # Path to finetune outputs\n\n# Use all host devices by default\nDEVICE=all\n# If you need to specify GPUs for training, set device as follow (NOTE: internal quotation marks cannot be omitted)\n#DEVICE='\"device=0,1,2,3\"'\n\nmkdir -p ${OUTPUT_PATH}\n\n# Single-GPU LoRA finetuning\ndocker run --gpus ${DEVICE} --rm --name qwen \\\n --mount type=bind,source=${CHECKPOINT_PATH},target=/data/shared/Qwen/Qwen-7B \\\n --mount type=bind,source=${DATA_PATH},target=/data/shared/Qwen/data \\\n --mount type=bind,source=${OUTPUT_PATH},target=/data/shared/Qwen/output_qwen \\\n --shm-size=2gb \\\n -it ${IMAGE_NAME} \\\n bash finetune/finetune_lora_single_gpu.sh -m /data/shared/Qwen/Qwen-7B/ -d /data/shared/Qwen/data/example.json\n```\n\n例えばシングルGPUのQ-LoRAに変更するには、`docker run`内のbashコマンドを変更するだけでいい：\n```bash\nbash finetune/finetune_qlora_single_gpu.sh -m /data/shared/Qwen/Qwen-7B-Chat-Int4/ -d /data/shared/Qwen/data/example.json\n```\n<br>\n\n## 🔥 システムプロンプト\nQwen-1.8-Chat と Qwen-72B-Chat は、複数回の複雑な対話を伴う多様なシステム プロンプトで完全にトレーニングされているため、さまざまなシステム プロンプトに従い、コンテキストに応じたモデルのカスタマイズを実現し、Qwen-Chat のスケーラビリティをさらに向上させることができます。\n\nシステム プロンプトを使用すると、Qwen-Chat は **ローリー プレイ**、**言語スタイルの転送**、**タスク設定**、**動作設定**を実現できます。\n\n![](assets/system_prompt_ language_style.png)\n\n![](assets/system_prompt_role_play_en.png)\n\n詳細については、[サンプルドキュメント](examples/system_prompt.md)を参照してください。\n\n## ツールの使用\n\nQwen-Chat は、ツールの使用法と関数呼び出し機能に合わせて最適化されています。 ユーザーはエージェント、LangChain アプリケーションを開発し、Python コード インタープリターで Qwen を拡張することもできます。\n\nReAct プロンプトの原則に基づいてツール呼び出しを実装する方法に関するドキュメントを提供しています。[ReAct の例](examples/react_prompt.md) を参照してください。 この原則に基づいて、[openai_api.py](openai_api.py) で関数呼び出しのサポートを提供します。\n\nオープンソースの中国語評価ベンチマークでモデルのツール呼び出し機能をテストしたところ、Qwen-Chat が一貫して良好なパフォーマンスを発揮することがわかりました。\n\n<table>\n <tr>\n <th colspan=\"4\" align=\"center\">Chinese Tool-Use Benchmark (Version 20231206)</th>\n </tr>\n <tr>\n <th align=\"center\">Model</th><th align=\"center\">Tool Selection (Acc.↑)</th><th align=\"center\">Tool Input (Rouge-L↑)</th><th align=\"center\">False Positive Error↓</th>\n </tr>\n <tr>\n <td>GPT-4</td><td align=\"center\">98.0%</td><td align=\"center\">0.953</td><td align=\"center\">23.9%</td>\n </tr>\n <tr>\n <td>GPT-3.5</td><td align=\"center\">74.5%</td><td align=\"center\">0.807</td><td align=\"center\">80.6%</td>\n </tr>\n <tr>\n <td>Qwen-1_8B-Chat</td><td align=\"center\">85.0%</td><td align=\"center\">0.839</td><td align=\"center\">27.6%</td>\n </tr>\n <tr>\n <td>Qwen-7B-Chat</td><td align=\"center\">95.5%</td><td align=\"center\">0.900</td><td align=\"center\">11.6%</td>\n </tr>\n <tr>\n <td>Qwen-14B-Chat</td><td align=\"center\">96.9%</td><td align=\"center\">0.917</td><td align=\"center\">5.6%</td>\n </tr>\n <tr>\n <td>Qwen-72B-Chat</td><td align=\"center\">98.2%</td><td align=\"center\">0.927</td><td align=\"center\">1.1%</td>\n </tr>\n</table>\n\n数学的問題解決、データ視覚化、ファイル処理や Web スクレイピングなどのその他の汎用タスクに Python コード インタープリターを使用する Qwen の能力を評価するために、これらの能力を評価するために特別に設計されたベンチマークを作成し、オープンソース化しました。 。 ベンチマークはこの [リンク](https://github.com/QwenLM/Qwen-Agent/tree/main/benchmark) で見つけることができます。\n\nQwen は、コード生成時のコードの実行可能性と結果の精度の点で優れたパフォーマンスを発揮することがわかりました。\n\n<table>\n <tr>\n <th colspan=\"5\" align=\"center\">Code Interpreter Benchmark (Version 20231206)</th>\n </tr>\n <tr>\n <th rowspan=\"2\" align=\"center\">Model</th>\n <th colspan=\"3\" align=\"center\">Accuracy of Code Execution Results (%)</th>\n <th colspan=\"1\" align=\"center\">Executable Rate of Code (%)</th>\n </tr>\n <tr>\n <th align=\"center\">Math↑</th><th align=\"center\">Visualization-Hard↑</th><th align=\"center\">Visualization-Easy↑</th><th align=\"center\">General↑</th>\n </tr>\n <tr>\n <td>GPT-4</td>\n <td align=\"center\">82.8</td>\n <td align=\"center\">66.7</td>\n <td align=\"center\">60.8</td>\n <td align=\"center\">82.8</td>\n </tr>\n <tr>\n <td>GPT-3.5</td>\n <td align=\"center\">47.3</td>\n <td align=\"center\">33.3</td>\n <td align=\"center\">55.7</td>\n <td align=\"center\">74.1</td>\n </tr>\n <tr>\n <td>LLaMA2-13B-Chat</td>\n <td align=\"center\">8.3</td>\n <td align=\"center\">1.2</td>\n <td align=\"center\">15.2</td>\n <td align=\"center\">48.3</td>\n </tr>\n <tr>\n <td>CodeLLaMA-13B-Instruct</td>\n <td align=\"center\">28.2</td>\n <td align=\"center\">15.5</td>\n <td align=\"center\">21.5</td>\n <td align=\"center\">74.1</td>\n </tr>\n <tr>\n <td>InternLM-20B-Chat</td>\n <td align=\"center\">34.6</td>\n <td align=\"center\">10.7</td>\n <td align=\"center\">25.1</td>\n <td align=\"center\">65.5</td>\n </tr>\n <tr>\n <td>ChatGLM3-6B</td>\n <td align=\"center\">54.2</td>\n <td align=\"center\">4.8</td>\n <td align=\"center\">15.2</td>\n <td align=\"center\">67.1</td>\n </tr>\n <tr>\n <td>Qwen-1.8B-Chat</td>\n <td align=\"center\">25.6</td>\n <td align=\"center\">21.4</td>\n <td align=\"center\">22.8</td>\n <td align=\"center\">65.5</td>\n </tr>\n <tr>\n <td>Qwen-7B-Chat</td>\n <td align=\"center\">41.9</td>\n <td align=\"center\">23.8</td>\n <td align=\"center\">38.0</td>\n <td align=\"center\">67.2</td>\n </tr>\n <tr>\n <td>Qwen-14B-Chat</td>\n <td align=\"center\">58.4</td>\n <td align=\"center\">31.0</td>\n <td align=\"center\">45.6</td>\n <td align=\"center\">65.5</td>\n </tr>\n <tr>\n <td>Qwen-72B-Chat</td>\n <td align=\"center\">72.7</td>\n <td align=\"center\">41.7</td>\n <td align=\"center\">43.0</td>\n <td align=\"center\">82.8</td>\n </tr>\n</table>\n\n<p align=\"center\">\n <br>\n <img src=\"assets/code_interpreter_showcase_001.jpg\" />\n <br>\n<p>\n\n<br>\n\n## 長い文脈の理解\n\nコンテキスト長を拡張し、トレーニング シーケンス長のボトルネックを解消するために、NTK 対応補間、ウィンドウ アテンション、LogN アテンション スケーリングなどのいくつかの技術を導入し、Qwen-14B のコンテキスト長を 2K から 8K 以上に拡張します。 トークン、および Qwen-1.8B/7B は 8K から 32K トークンまで。\n\nQwen-72B では、より大きな回転ベースを備えたより長いコンテキストに RoPE を適応させます。 Qwen-72B は、32K トークンの最大コンテキスト長をサポートします。\n\n私たちは、PPL 評価を使用して arXiv データセットで言語モデリング実験を実施し、Qwen が長いコンテキストのシナリオで優れたパフォーマンスを達成できることを発見しました。 結果を以下に示します。\n\n<table>\n <tr>\n <th rowspan=\"2\">Model</th><th colspan=\"6\" align=\"center\">Sequence Length</th>\n </tr>\n <tr>\n <th align=\"center\">1024</th><th align=\"center\">2048</th><th align=\"center\">4096</th><th align=\"center\">8192</th><th align=\"center\">16384</th><th align=\"center\">32768</th>\n </tr>\n <tr>\n <td>Qwen-7B (original)</td><td align=\"center\">4.23</td><td align=\"center\">3.78</td><td align=\"center\">39.35</td><td align=\"center\">469.81</td><td align=\"center\">2645.09</td><td align=\"center\">-</td>\n </tr>\n <tr>\n <td>+ dynamic_ntk</td><td align=\"center\">4.23</td><td align=\"center\">3.78</td><td align=\"center\">3.59</td><td align=\"center\">3.66</td><td align=\"center\">5.71</td><td align=\"center\">-</td>\n </tr>\n <tr>\n <td>+ dynamic_ntk + logn</td><td align=\"center\">4.23</td><td align=\"center\">3.78</td><td align=\"center\">3.58</td><td align=\"center\">3.56</td><td align=\"center\">4.62</td><td align=\"center\">-</td>\n </tr>\n <tr>\n <td>+ dynamic_ntk + logn + window_attn</td><td align=\"center\">4.23</td><td align=\"center\">3.78</td><td align=\"center\">3.58</td><td align=\"center\">3.49</td><td align=\"center\">4.32</td><td align=\"center\">-</td>\n </tr>\n <tr>\n <tr>\n <td>Qwen-1.8B</td><td align=\"center\"><b>5.00</b></td><td align=\"center\"><b>4.48</b></td><td align=\"center\"><b>4.13</b></td><td align=\"center\"><b>3.89</b></td><td align=\"center\">17.42</td><td align=\"center\">433.85</td>\n </tr>\n <tr>\n <td>+ dynamic_ntk + logn + window_attn</td><td align=\"center\"><b>5.00</b></td><td align=\"center\"><b>4.48</b></td><td align=\"center\"><b>4.14</b></td><td align=\"center\"><b>3.93</b></td><td align=\"center\"><b>3.82</b></td><td align=\"center\"><b>3.83</b></td>\n </tr>\n <tr>\n <td>Qwen-7B</td><td align=\"center\"><b>4.23</b></td><td align=\"center\"><b>3.81</b></td><td align=\"center\"><b>3.52</b></td><td align=\"center\"><b>3.31</b></td><td align=\"center\">7.27</td><td align=\"center\">181.49</td>\n </tr>\n <tr>\n <td>+ dynamic_ntk + logn + window_attn</td><td align=\"center\"><b>4.23</b></td><td align=\"center\"><b>3.81</b></td><td align=\"center\"><b>3.52</b></td><td align=\"center\"><b>3.33</b></td><td align=\"center\"><b>3.22</b></td><td align=\"center\"><b>3.17</b></td>\n </tr>\n <tr>\n <td>Qwen-14B</td><td align=\"center\"><b>-</b></td><td align=\"center\"><b>3.46</b></td><td align=\"center\">22.79</td><td align=\"center\">334.65</td><td align=\"center\">3168.35</td><td align=\"center\">-</td>\n </tr>\n <tr>\n <td>+ dynamic_ntk + logn + window_attn</td><td align=\"center\"><b>-</b></td><td align=\"center\"><b>3.46</b></td><td align=\"center\"><b>3.29</b></td><td align=\"center\"><b>3.18</b></td><td align=\"center\">3.42</td><td align=\"center\">-</td>\n </tr>\n <tr>\n <td>Qwen-72B</td><td align=\"center\"><b>-</b></td><td align=\"center\"><b>-</b></td><td align=\"center\">-</td><td align=\"center\"><b>2.83</b></td><td align=\"center\"><b>2.73</b></td><td align=\"center\"><b>2.72</b></td>\n </tr>\n </tr>\n</table>\n\nさらに、Qwen-72B-Chat の長文理解能力を検証するために、[L-Eval](https://arxiv.org/abs/2307.11088) (クローズドエンド タスク) でテストしました。 結果は次のとおりです。\n\n| Model | Input Length | Average | Coursera | GSM | QuALITY | TOEFL | CodeU | SFcition |\n|:------------------|:------------:|:---------:|:----------:|:----------:|:----------:|:----------:|:----------:|:----------:|\n| ChatGPT-3.5-16k | 16K | 60.73 | **63.51** | **84.00** | 61.38 | 78.43 | **12.22** | 64.84 |\n| **Qwen-72B-Chat** | 32K | **62.30** | 58.13 | 76.00 | **77.22** | **86.24** | 6.66 | **69.53** |\n\n私たちは、モデルが入力内のさまざまな位置で情報を取得できるかどうかをテストするために、「干し草の山の中の針」実験 (このアイデアは [@Greg Kamradt](https://twitter.com/GregKamradt/status/1727018183608193393) から来ました) を実施しました。 異なる長さの場合、結果は次のようになります。\n![](assets/qwen_72b_needle_in_a_haystack.png)\n\n上記の結果は、Qwen-72B-Chat が 32K の入力長内でさまざまな位置に配置された情報を正確に取得できることを示しており、その優れた長文理解能力を証明しています。\n\n## トークナイザー\n\ntiktoken に基づくトークナイザーは、他のトークナイザー、例えばセンテンスピーストークナイザーとは異なります。特にファインチューニングの際には、特殊なトークンに注意を払う必要があります。トークナイザに関する詳細な情報や、ファインチューニングにおける使用方法については、[ドキュメント](tokenization_note_ja.md)を参照してください。\n<br><br>\n\n## 再現\n\nベンチマークデータセットでのモデル性能の再現のために、結果を再現するスクリプトを提供しています。詳しくは [eval/EVALUATION.md](eval/EVALUATION.md) を確認してください。なお、再現の結果、我々の報告結果と若干異なる場合があります。\n<br><br>\n\n## FAQ\n\n問題が発生した場合は、まずは [FAQ](FAQ_ja.md) や issue を参照し、新しい issue を立ち上げる前に解決策を探してください。\n<br><br>\n\n## 引用\n私たちの仕事が役に立ったと思ったら、遠慮なく引用してください。\n\n```\n@article{qwen,\n title={Qwen Technical Report},\n author={Jinze Bai and Shuai Bai and Yunfei Chu and Zeyu Cui and Kai Dang and Xiaodong Deng and Yang Fan and Wenbin Ge and Yu Han and Fei Huang and Binyuan Hui and Luo Ji and Mei Li and Junyang Lin and Runji Lin and Dayiheng Liu and Gao Liu and Chengqiang Lu and Keming Lu and Jianxin Ma and Rui Men and Xingzhang Ren and Xuancheng Ren and Chuanqi Tan and Sinan Tan and Jianhong Tu and Peng Wang and Shijie Wang and Wei Wang and Shengguang Wu and Benfeng Xu and Jin Xu and An Yang and Hao Yang and Jian Yang and Shusheng Yang and Yang Yao and Bowen Yu and Hongyi Yuan and Zheng Yuan and Jianwei Zhang and Xingxuan Zhang and Yichang Zhang and Zhenru Zhang and Chang Zhou and Jingren Zhou and Xiaohuan Zhou and Tianhang Zhu},\n journal={arXiv preprint arXiv:2309.16609},\n year={2023}\n}\n```\n<br>\n\n## ライセンス契約\n\n<https://github.com/QwenLM/Qwen>で提供されるソースコードは、ルートディレクトリにある[Apache 2.0 License](./LICENSE)の下でライセンスされています。\n\n研究者や開発者は、QwenとQwen-Chatのコードとモデルウェイトを自由に使用することができます。商用利用については、各モデルに添付されている使用許諾契約書をご確認ください。\n\n- Qwen-72B、Qwen-14B、Qwen-7Bは、対応するHuggingFaceとModelScopeのリポジトリにある[Tongyi Qianwen LICENSE AGREEMENT](./Tongyi%20Qianwen%20LICENSE%20AGREEMENT)に基づいてライセンスされています。商用利用の場合は、フォーム([72B](https://dashscope.console.aliyun.com/openModelApply/Qwen-72B-Chat), [14B](https://dashscope.console.aliyun.com/openModelApply/Qwen-14B-Chat), [7B](https://dashscope.console.aliyun.com/openModelApply/qianwen))に記入して申請してください。\n\n- Qwen-1.8Bは、対応するHuggingFaceとModelScopeのリポジトリにある[Tongyi Qianwen RESEARCH LICENSE AGREEMENT](./Tongyi%20Qianwen%20RESEARCH%20LICENSE%20AGREEMENT)に基づいてライセンスされています。商用利用については、私たちにご連絡ください。\n<br><br>\n\n## お問い合わせ\n\n研究チームまたは製品チームへのメッセージは、qianwen_opensource@alibabacloud.com までお気軽にお送りください。\n\n" }, { "path": "Tongyi Qianwen LICENSE AGREEMENT", "content": "Tongyi Qianwen LICENSE AGREEMENT\n\nTongyi Qianwen Release Date: August 3, 2023\n\nBy clicking to agree or by using or distributing any portion or element of the Tongyi Qianwen Materials, you will be deemed to have recognized and accepted the content of this Agreement, which is effective immediately.\n\n1. Definitions\n a. This Tongyi Qianwen LICENSE AGREEMENT (this \"Agreement\") shall mean the terms and conditions for use, reproduction, distribution and modification of the Materials as defined by this Agreement.\n b. \"We\"(or \"Us\") shall mean Alibaba Cloud.\n c. \"You\" (or \"Your\") shall mean a natural person or legal entity exercising the rights granted by this Agreement and/or using the Materials for any purpose and in any field of use.\n d. \"Third Parties\" shall mean individuals or legal entities that are not under common control with Us or You.\n e. \"Tongyi Qianwen\" shall mean the large language models (including Qwen model and Qwen-Chat model), and software and algorithms, consisting of trained model weights, parameters (including optimizer states), machine-learning model code, inference-enabling code, training-enabling code, fine-tuning enabling code and other elements of the foregoing distributed by Us.\n f. \"Materials\" shall mean, collectively, Alibaba Cloud's proprietary Tongyi Qianwen and Documentation (and any portion thereof) made available under this Agreement.\n g. \"Source\" form shall mean the preferred form for making modifications, including but not limited to model source code, documentation source, and configuration files.\n h. \"Object\" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation,\n and conversions to other media types.\n\n2. Grant of Rights\nYou are granted a non-exclusive, worldwide, non-transferable and royalty-free limited license under Alibaba Cloud's intellectual property or other rights owned by Us embodied in the Materials to use, reproduce, distribute, copy, create derivative works of, and make modifications to the Materials.\n\n3. Redistribution\nYou may reproduce and distribute copies of the Materials or derivative works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions:\n a. You shall give any other recipients of the Materials or derivative works a copy of this Agreement;\n b. You shall cause any modified files to carry prominent notices stating that You changed the files;\n c. You shall retain in all copies of the Materials that You distribute the following attribution notices within a \"Notice\" text file distributed as a part of such copies: \"Tongyi Qianwen is licensed under the Tongyi Qianwen LICENSE AGREEMENT, Copyright (c) Alibaba Cloud. All Rights Reserved.\"; and\n d. You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such derivative works as a whole, provided Your use, reproduction, and distribution of the work otherwise complies with the terms and conditions of this Agreement.\n\n4. Restrictions\nIf you are commercially using the Materials, and your product or service has more than 100 million monthly active users, You shall request a license from Us. You cannot exercise your rights under this Agreement without our express authorization.\n\n5. Rules of use\n a. The Materials may be subject to export controls or restrictions in China, the United States or other countries or regions. You shall comply with applicable laws and regulations in your use of the Materials.\n b. You can not use the Materials or any output therefrom to improve any other large language model (excluding Tongyi Qianwen or derivative works thereof).\n\n6. Intellectual Property\n a. We retain ownership of all intellectual property rights in and to the Materials and derivatives made by or for Us. Conditioned upon compliance with the terms and conditions of this Agreement, with respect to any derivative works and modifications of the Materials that are made by you, you are and will be the owner of such derivative works and modifications.\n b. No trademark license is granted to use the trade names, trademarks, service marks, or product names of Us, except as required to fulfill notice requirements under this Agreement or as required for reasonable and customary use in describing and redistributing the Materials.\n c. If you commence a lawsuit or other proceedings (including a cross-claim or counterclaim in a lawsuit) against Us or any entity alleging that the Materials or any output therefrom, or any part of the foregoing, infringe any intellectual property or other right owned or licensable by you, then all licences granted to you under this Agreement shall terminate as of the date such lawsuit or other proceeding is commenced or brought.\n\n7. Disclaimer of Warranty and Limitation of Liability\n\n a. We are not obligated to support, update, provide training for, or develop any further version of the Tongyi Qianwen Materials or to grant any license thereto.\n b. THE MATERIALS ARE PROVIDED \"AS IS\" WITHOUT ANY EXPRESS OR IMPLIED WARRANTY OF ANY KIND INCLUDING WARRANTIES OF MERCHANTABILITY, NONINFRINGEMENT, OR FITNESS FOR A PARTICULAR PURPOSE. WE MAKE NO WARRANTY AND ASSUME NO RESPONSIBILITY FOR THE SAFETY OR STABILITY OF THE MATERIALS AND ANY OUTPUT THEREFROM.\n c. IN NO EVENT SHALL WE BE LIABLE TO YOU FOR ANY DAMAGES, INCLUDING, BUT NOT LIMITED TO ANY DIRECT, OR INDIRECT, SPECIAL OR CONSEQUENTIAL DAMAGES ARISING FROM YOUR USE OR INABILITY TO USE THE MATERIALS OR ANY OUTPUT OF IT, NO MATTER HOW IT’S CAUSED.\n d. You will defend, indemnify and hold harmless Us from and against any claim by any third party arising out of or related to your use or distribution of the Materials.\n\n8. Survival and Termination.\n a. The term of this Agreement shall commence upon your acceptance of this Agreement or access to the Materials and will continue in full force and effect until terminated in accordance with the terms and conditions herein.\n b. We may terminate this Agreement if you breach any of the terms or conditions of this Agreement. Upon termination of this Agreement, you must delete and cease use of the Materials. Sections 7 and 9 shall survive the termination of this Agreement.\n\n9. Governing Law and Jurisdiction.\n a. This Agreement and any dispute arising out of or relating to it will be governed by the laws of China, without regard to conflict of law principles, and the UN Convention on Contracts for the International Sale of Goods does not apply to this Agreement.\n b. The People's Courts in Hangzhou City shall have exclusive jurisdiction over any dispute arising out of this Agreement." }, { "path": "Tongyi Qianwen RESEARCH LICENSE AGREEMENT", "content": "Tongyi Qianwen RESEARCH LICENSE AGREEMENT\n\nTongyi Qianwen Release Date: November 30, 2023\n\nBy clicking to agree or by using or distributing any portion or element of the Tongyi Qianwen Materials, you will be deemed to have recognized and accepted the content of this Agreement, which is effective immediately.\n\n1. Definitions\n a. This Tongyi Qianwen RESEARCH LICENSE AGREEMENT (this \"Agreement\") shall mean the terms and conditions for use, reproduction, distribution and modification of the Materials as defined by this Agreement.\n b. \"We\"(or \"Us\") shall mean Alibaba Cloud.\n c. \"You\" (or \"Your\") shall mean a natural person or legal entity exercising the rights granted by this Agreement and/or using the Materials for any purpose and in any field of use.\n d. \"Third Parties\" shall mean individuals or legal entities that are not under common control with Us or You.\n e. \"Tongyi Qianwen\" shall mean the large language models, and software and algorithms, consisting of trained model weights, parameters (including optimizer states), machine-learning model code, inference-enabling code, training-enabling code, fine-tuning enabling code and other elements of the foregoing distributed by Us.\n f. \"Materials\" shall mean, collectively, Alibaba Cloud's proprietary Tongyi Qianwen and Documentation (and any portion thereof) made available under this Agreement.\n g. \"Source\" form shall mean the preferred form for making modifications, including but not limited to model source code, documentation source, and configuration files.\n h. \"Object\" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation,\n and conversions to other media types.\n i. \"Non-Commercial\" shall mean for research or evaluation purposes only.\n\n2. Grant of Rights\n a. You are granted a non-exclusive, worldwide, non-transferable and royalty-free limited license under Alibaba Cloud's intellectual property or other rights owned by Us embodied in the Materials to use, reproduce, distribute, copy, create derivative works of, and make modifications to the Materials FOR NON-COMMERCIAL PURPOSES ONLY.\n b. If you are commercially using the Materials, You shall request a license from Us.\n\n3. Redistribution\nYou may reproduce and distribute copies of the Materials or derivative works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions:\n a. You shall give any other recipients of the Materials or derivative works a copy of this Agreement;\n b. You shall cause any modified files to carry prominent notices stating that You changed the files;\n c. You shall retain in all copies of the Materials that You distribute the following attribution notices within a \"Notice\" text file distributed as a part of such copies: \"Tongyi Qianwen is licensed under the Tongyi Qianwen RESEARCH LICENSE AGREEMENT, Copyright (c) Alibaba Cloud. All Rights Reserved.\"; and\n d. You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such derivative works as a whole, provided Your use, reproduction, and distribution of the work otherwise complies with the terms and conditions of this Agreement.\n\n4. Rules of use\n a. The Materials may be subject to export controls or restrictions in China, the United States or other countries or regions. You shall comply with applicable laws and regulations in your use of the Materials.\n b. You can not use the Materials or any output therefrom to improve any other large language model (excluding Tongyi Qianwen or derivative works thereof).\n\n5. Intellectual Property\n a. We retain ownership of all intellectual property rights in and to the Materials and derivatives made by or for Us. Conditioned upon compliance with the terms and conditions of this Agreement, with respect to any derivative works and modifications of the Materials that are made by you, you are and will be the owner of such derivative works and modifications.\n b. No trademark license is granted to use the trade names, trademarks, service marks, or product names of Us, except as required to fulfill notice requirements under this Agreement or as required for reasonable and customary use in describing and redistributing the Materials.\n c. If you commence a lawsuit or other proceedings (including a cross-claim or counterclaim in a lawsuit) against Us or any entity alleging that the Materials or any output therefrom, or any part of the foregoing, infringe any intellectual property or other right owned or licensable by you, then all licences granted to you under this Agreement shall terminate as of the date such lawsuit or other proceeding is commenced or brought.\n\n6. Disclaimer of Warranty and Limitation of Liability\n a. We are not obligated to support, update, provide training for, or develop any further version of the Tongyi Qianwen Materials or to grant any license thereto.\n b. THE MATERIALS ARE PROVIDED \"AS IS\" WITHOUT ANY EXPRESS OR IMPLIED WARRANTY OF ANY KIND INCLUDING WARRANTIES OF MERCHANTABILITY, NONINFRINGEMENT, OR FITNESS FOR A PARTICULAR PURPOSE. WE MAKE NO WARRANTY AND ASSUME NO RESPONSIBILITY FOR THE SAFETY OR STABILITY OF THE MATERIALS AND ANY OUTPUT THEREFROM.\n c. IN NO EVENT SHALL WE BE LIABLE TO YOU FOR ANY DAMAGES, INCLUDING, BUT NOT LIMITED TO ANY DIRECT, OR INDIRECT, SPECIAL OR CONSEQUENTIAL DAMAGES ARISING FROM YOUR USE OR INABILITY TO USE THE MATERIALS OR ANY OUTPUT OF IT, NO MATTER HOW IT’S CAUSED.\n d. You will defend, indemnify and hold harmless Us from and against any claim by any third party arising out of or related to your use or distribution of the Materials.\n\n7. Survival and Termination.\n a. The term of this Agreement shall commence upon your acceptance of this Agreement or access to the Materials and will continue in full force and effect until terminated in accordance with the terms and conditions herein.\n b. We may terminate this Agreement if you breach any of the terms or conditions of this Agreement. Upon termination of this Agreement, you must delete and cease use of the Materials. Sections 6 and 8 shall survive the termination of this Agreement.\n\n8. Governing Law and Jurisdiction.\n a. This Agreement and any dispute arising out of or relating to it will be governed by the laws of China, without regard to conflict of law principles, and the UN Convention on Contracts for the International Sale of Goods does not apply to this Agreement.\n b. The People's Courts in Hangzhou City shall have exclusive jurisdiction over any dispute arising out of this Agreement.\n\n9. Other Terms and Conditions.\n a. Any arrangements, understandings, or agreements regarding the Material not stated herein are separate from and independent of the terms and conditions of this Agreement. You shall request a seperate license from Us, if You use the Materials in ways not expressly agreed to in this Agreement.\n b. We shall not be bound by any additional or different terms or conditions communicated by You unless expressly agreed.\n" }, { "path": "ascend-support/README.md", "content": "# 昇腾910架构基于mindformers推理Qwen-7B-Chat模型\n\n## 环境要求\n\n- 硬件：Ascend 910A/B\n\n## 运行步骤\n\n首先参考Qwen README下载官方模型到`/path/to/Qwen-7B-Chat`。\n\n### 下载并启动镜像\n\n```bash\ndocker pull qwenllm/qwen-mindspore:latest\n\ncd /path/to/Qwen/ascend-support\n\n# 下载模型到此处\nCHECKPOINT_PATH=/path/to/Qwen-7B-Chat\n\ncd ascend-support\n\n# 启动docker容器\nbash docker_qwen.sh -c ${CHECKPOINT_PATH}\n```\n\n### 执行权重转换\n\n在容器内执行下面的命令，将Qwen模型转换为适配`mindformers`的格式：\n\n```bash\npython3 /data/qwen/mindformers/research/qwen/convert_weight.py\n```\n\n转换后模型的输出位置为`${CHECKPOINT_PATH}/qwen-7b-chat.ckpt`。\n\n### 执行推理\n\n在容器内执行下面的命令，进行推理：\n\n```bash\ncd /data/qwen/mindformers/research/qwen\nexport PYTHONPATH=/data/qwen/mindformers:$PYTHONPATH\npython3 infer_qwen.py\n```\n" }, { "path": "ascend-support/docker_qwen.sh", "content": "#!/bin/bash\n\nIMAGE_NAME=qwenllm/qwen-mindspore:v23.0.RC3\nCONTAINER_NAME=qwen-mindspore\nCHECKPOINT_PATH='NOT_SET'\n\nDOCKER_CHECKPOINT_PATH=/data/qwen/models/Qwen-7B-Chat\n\nfunction usage() {\n echo '\nUsage: bash ascend-support/docker_qwen.sh [-i IMAGE_NAME] -c [/path/to/Qwen-7B-Chat] [-n CONTAINER_NAME]\n'\n}\n\nwhile [[ \"$1\" != \"\" ]]; do\n case $1 in\n -i | --image )\n shift\n IMAGE_NAME=$1\n ;;\n -c | --checkpoint )\n shift\n CHECKPOINT_PATH=$1\n ;;\n -n | --name )\n shift\n CONTAINER_NAME=$1\n ;;\n -h )\n usage\n exit\n ;;\n * )\n echo \"Unknown argument ${1}\"\n exit 1\n ;;\n esac\n shift\ndone\n\ndocker run -it --rm -u root --network=host --ipc=host \\\n --device=/dev/davinci0 \\\n --device=/dev/davinci1 \\\n --device=/dev/davinci2 \\\n --device=/dev/davinci3 \\\n --device=/dev/davinci4 \\\n --device=/dev/davinci5 \\\n --device=/dev/davinci6 \\\n --device=/dev/davinci7 \\\n --name=${CONTAINER_NAME} \\\n --device=/dev/davinci_manager \\\n --device=/dev/devmm_svm \\\n --device=/dev/hisi_hdc \\\n -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \\\n -v /usr/local/Ascend/add-ons/:/usr/local/Ascend/add-ons/ \\\n -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \\\n -v /usr/local/sbin/npu-smi:/usr/local/sbin/npu-smi \\\n -v /etc/ascend_install.info:/etc/ascend_install.info \\\n -v ${CHECKPOINT_PATH}:${DOCKER_CHECKPOINT_PATH} \\\n -v /var/log/npu/:/usr/slog \\\n ${IMAGE_NAME} /bin/bash\n" }, { "path": "cli_demo.py", "content": "# Copyright (c) Alibaba Cloud.\n#\n# This source code is licensed under the license found in the\n# LICENSE file in the root directory of this source tree.\n\n\"\"\"A simple command-line interactive chat demo.\"\"\"\n\nimport argparse\nimport os\nimport platform\nimport shutil\nfrom copy import deepcopy\n\nimport torch\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\nfrom transformers.generation import GenerationConfig\nfrom transformers.trainer_utils import set_seed\n\nDEFAULT_CKPT_PATH = 'Qwen/Qwen-7B-Chat'\n\n_WELCOME_MSG = '''\\\nWelcome to use Qwen-Chat model, type text to start chat, type :h to show command help.\n(欢迎使用 Qwen-Chat 模型，输入内容即可进行对话，:h 显示命令帮助。)\n\nNote: This demo is governed by the original license of Qwen.\nWe strongly advise users not to knowingly generate or allow others to knowingly generate harmful content, including hate speech, violence, pornography, deception, etc.\n(注：本演示受Qwen的许可协议限制。我们强烈建议，用户不应传播及不应允许他人传播以下内容，包括但不限于仇恨言论、暴力、色情、欺诈相关的有害信息。)\n'''\n_HELP_MSG = '''\\\nCommands:\n :help / :h Show this help message 显示帮助信息\n :exit / :quit / :q Exit the demo 退出Demo\n :clear / :cl Clear screen 清屏\n :clear-his / :clh Clear history 清除对话历史\n :history / :his Show history 显示对话历史\n :seed Show current random seed 显示当前随机种子\n :seed <N> Set random seed to <N> 设置随机种子\n :conf Show current generation config 显示生成配置\n :conf <key>=<value> Change generation config 修改生成配置\n :reset-conf Reset generation config 重置生成配置\n'''\n\n\ndef _load_model_tokenizer(args):\n tokenizer = AutoTokenizer.from_pretrained(\n args.checkpoint_path, trust_remote_code=True, resume_download=True,\n )\n\n if args.cpu_only:\n device_map = \"cpu\"\n else:\n device_map = \"auto\"\n\n model = AutoModelForCausalLM.from_pretrained(\n args.checkpoint_path,\n device_map=device_map,\n trust_remote_code=True,\n resume_download=True,\n ).eval()\n\n config = GenerationConfig.from_pretrained(\n args.checkpoint_path, trust_remote_code=True, resume_download=True,\n )\n\n return model, tokenizer, config\n\n\ndef _gc():\n import gc\n gc.collect()\n if torch.cuda.is_available():\n torch.cuda.empty_cache()\n\n\ndef _clear_screen():\n if platform.system() == \"Windows\":\n os.system(\"cls\")\n else:\n os.system(\"clear\")\n\n\ndef _print_history(history):\n terminal_width = shutil.get_terminal_size()[0]\n print(f'History ({len(history)})'.center(terminal_width, '='))\n for index, (query, response) in enumerate(history):\n print(f'User[{index}]: {query}')\n print(f'QWen[{index}]: {response}')\n print('=' * terminal_width)\n\n\ndef _get_input() -> str:\n while True:\n try:\n message = input('User> ').strip()\n except UnicodeDecodeError:\n print('[ERROR] Encoding error in input')\n continue\n except KeyboardInterrupt:\n exit(1)\n if message:\n return message\n print('[ERROR] Query is empty')\n\n\ndef main():\n parser = argparse.ArgumentParser(\n description='QWen-Chat command-line interactive chat demo.')\n parser.add_argument(\"-c\", \"--checkpoint-path\", type=str, default=DEFAULT_CKPT_PATH,\n help=\"Checkpoint name or path, default to %(default)r\")\n parser.add_argument(\"-s\", \"--seed\", type=int, default=1234, help=\"Random seed\")\n parser.add_argument(\"--cpu-only\", action=\"store_true\", help=\"Run demo with CPU only\")\n args = parser.parse_args()\n\n history, response = [], ''\n\n model, tokenizer, config = _load_model_tokenizer(args)\n orig_gen_config = deepcopy(model.generation_config)\n\n _clear_screen()\n print(_WELCOME_MSG)\n\n seed = args.seed\n\n while True:\n query = _get_input()\n\n # Process commands.\n if query.startswith(':'):\n command_words = query[1:].strip().split()\n if not command_words:\n command = ''\n else:\n command = command_words[0]\n\n if command in ['exit', 'quit', 'q']:\n break\n elif command in ['clear', 'cl']:\n _clear_screen()\n print(_WELCOME_MSG)\n _gc()\n continue\n elif command in ['clear-his', 'clh']:\n print(f'[INFO] All {len(history)} history cleared')\n history.clear()\n _gc()\n continue\n elif command in ['help', 'h']:\n print(_HELP_MSG)\n continue\n elif command in ['history', 'his']:\n _print_history(history)\n continue\n elif command in ['seed']:\n if len(command_words) == 1:\n print(f'[INFO] Current random seed: {seed}')\n continue\n else:\n new_seed_s = command_words[1]\n try:\n new_seed = int(new_seed_s)\n except ValueError:\n print(f'[WARNING] Fail to change random seed: {new_seed_s!r} is not a valid number')\n else:\n print(f'[INFO] Random seed changed to {new_seed}')\n seed = new_seed\n continue\n elif command in ['conf']:\n if len(command_words) == 1:\n print(model.generation_config)\n else:\n for key_value_pairs_str in command_words[1:]:\n eq_idx = key_value_pairs_str.find('=')\n if eq_idx == -1:\n print('[WARNING] format: <key>=<value>')\n continue\n conf_key, conf_value_str = key_value_pairs_str[:eq_idx], key_value_pairs_str[eq_idx + 1:]\n try:\n conf_value = eval(conf_value_str)\n except Exception as e:\n print(e)\n continue\n else:\n print(f'[INFO] Change config: model.generation_config.{conf_key} = {conf_value}')\n setattr(model.generation_config, conf_key, conf_value)\n continue\n elif command in ['reset-conf']:\n print('[INFO] Reset generation config')\n model.generation_config = deepcopy(orig_gen_config)\n print(model.generation_config)\n continue\n else:\n # As normal query.\n pass\n\n # Run chat.\n set_seed(seed)\n try:\n for response in model.chat_stream(tokenizer, query, history=history, generation_config=config):\n _clear_screen()\n print(f\"\\nUser: {query}\")\n print(f\"\\nQwen-Chat: {response}\")\n except KeyboardInterrupt:\n print('[WARNING] Generation interrupted')\n continue\n\n history.append((query, response))\n\n\nif __name__ == \"__main__\":\n main()\n" }, { "path": "dcu-support/README.md", "content": "# DCU 架构基于 fastllm 推理 Qwen 模型\n\n\n## 环境配置\n\n### 环境准备\n\n```\ndocker pull image.sourcefind.cn:5000/dcu/admin/base/pytorch:1.13.1-centos7.6-dtk-23.04-py38-latest\n```\n\n### 容器启动\n\n根据如下命令启动推理容器，其中需自定义一个容器名<container_name>，<project_path>即为本目录的路径：\n```\n# <container_name> 自定义容器名\n# <project_path> 当前工程所在路径\ndocker run -it --name=<container_name> -v <project_path>:/work --device=/dev/kfd --device=/dev/dri --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --shm-size=16G --group-add 39 image.sourcefind.cn:5000/dcu/admin/base/pytorch:1.13.1-centos7.6-dtk-23.04-py38-latest /bin/bash\n```\n\n### 加载环境\n\n进入容器后执行如下命令，加载运行环境变量\n\n```\nsource /opt/dtk-23.04/cuda/env.sh\n```\n\n### 安装方法\n\n```\n#进入本工程目录\ncd package\npython setup.py install\n```\n\n## 推理\n\n### 模型转换\n\n首先参考Qwen README下载官方模型，并通过如下方式将模型转换为 fastllm 用于推理的形式：\n\n- 通过`pip install -r requirements.txt`安装模型转换所需依赖\n\n- 如果使用已经下载完成的模型或者自己finetune的模型需要修改qwen2flm.py文件中创建tokenizer, model时的模型存放路径\n\n```\n# 在本工程目录下执行：\npython3 qwen2flm.py qwen-7b-fp16.bin float16 # 导出fp16模型，参数为导出的模型路径\n```\n\n\n### 模型推理\n\n```\n# 命令行聊天程序，使用了模型创建以及流式对话效果\npython cli_demo.py -p qwen-7b-fp16.bin\n\n# batch推理程序\npython cli_demo_batch.py -p qwen-7b-fp16.bin\n\n# 简易webui，需要先安装streamlit-chat\nstreamlit run web_demo.py qwen-7b-fp16.bin \n```\n" }, { "path": "dcu-support/cli_demo.py", "content": "# coding=utf-8\nimport argparse\nfrom fastllm_pytools import llm\n\ndef args_parser():\n parser = argparse.ArgumentParser(description = 'qwen_chat_demo')\n parser.add_argument('-p', '--path', type = str, required = True, default = '', help = '模型文件的路径')\n args = parser.parse_args()\n return args\n\nif __name__ == \"__main__\":\n args = args_parser()\n model = llm.model(args.path)\n\n history = []\n print(\"输入内容即可进行对话，clear 清空对话历史，stop 终止程序\")\n while True:\n query = input(\"\\n用户：\")\n if query.strip() == \"stop\":\n break\n if query.strip() == \"clear\":\n history = []\n print(\"输入内容即可进行对话，clear 清空对话历史，stop 终止程序\")\n continue\n print(\"AI:\", end = \"\")\n curResponse = \"\"\n for response in model.stream_response(query, history = history, do_sample = True, top_p = 0.8, top_k = 1, temperature = 1.0, repeat_penalty = 1.0):\n curResponse += response\n print(response, flush = True, end = \"\")\n history.append((query, curResponse))" }, { "path": "dcu-support/cli_demo_batch.py", "content": "import argparse\nfrom fastllm_pytools import llm\nimport time\n\ndef args_parser():\n parser = argparse.ArgumentParser(description = 'fastllm_chat_demo')\n parser.add_argument('-p', '--path', type = str, required = True, default = '', help = '模型文件的路径')\n args = parser.parse_args()\n return args\n\nif __name__ == \"__main__\":\n args = args_parser()\n\n model_path = args.path\n\n prompts = [\"深圳有什么好玩的\", \"上海有什么好玩的\", \"晚上睡不着怎么办\", \"南京有什么好吃的\"] * 2\n print(prompts)\n\n responses, historys = [], []\n \n model = llm.model(model_path)\n \n t0 = time.time()\n responses, historys = model.response_batch(prompts) \n t1 = time.time()\n\n token_output_count = 0\n word_len = 0\n for i, res in enumerate(responses):\n tokens = model.tokenizer_encode_string(res)\n token_output_count += len(tokens)\n word_len += len(res)\n\n print(\"batch index: \", i)\n print(res)\n print(\"\")\n\n print(\"\\ntoken/s: {:.2f}, character/s: {:.2f}\".format(token_output_count/(t1-t0), word_len/(t1-t0)))\n\n" }, { "path": "dcu-support/model.properties", "content": "# 模型唯一标识\nmodelCode = 411\n# 模型名称\nmodelName=qwen-7b_fastllm\n# 模型描述\nmodelDescription=qwen-7b是阿里云研发的通义千问大模型系列的70亿参数规模的模型\n# 应用场景\nappScenario=推理,对话问答,医疗,科研,金融,教育\n# 框架类型\nframeType=fastllm\n" }, { "path": "dcu-support/package/fastllm_pytools/__init__.py", "content": "__all__ = [\"llm\"]" }, { "path": "dcu-support/package/fastllm_pytools/hf_model.py", "content": "from fastllm_pytools import llm;\nimport torch;\nimport ctypes;\nimport numpy as np;\n\nfastllm_data_type_dict = {\n \"int4\": 8,\n \"int8\": 3,\n \"float16\": 7\n}\nfastllm_weight_type_dict = {\n \"linear\": 1,\n \"embedding\": 2,\n \"QuantizedLinear\": 111\n}\n\ndef create(model,\n tokenizer = None,\n pre_prompt = None,\n user_role = None,\n bot_role = None,\n history_sep = None,\n dtype = \"float16\"):\n if (dtype not in fastllm_data_type_dict):\n print(\"dtype should in \", list(fastllm_data_type_dict.keys()));\n exit(0);\n\n # 0.1 model info\n if model.config.model_type == \"chatglm\" and model.config.transformers_version == \"4.30.2\":\n model.config.model_type = \"chatglm3\"\n modelInfo = model.config.__dict__\n if model.generation_config is not None:\n modelInfo.update(model.generation_config.__dict__)\n if (pre_prompt):\n modelInfo[\"pre_prompt\"] = pre_prompt;\n if (user_role):\n modelInfo[\"user_role\"] = user_role;\n if (bot_role):\n modelInfo[\"bot_role\"] = bot_role;\n if (history_sep):\n modelInfo[\"history_sep\"] = history_sep;\n if (modelInfo[\"model_type\"] == \"baichuan\" and hasattr(model, \"model\") and hasattr(model.model, \"get_alibi_mask\")):\n # Baichuan 2代\n modelInfo[\"use_alibi\"] = \"1\";\n modelInfo[\"pre_prompt\"] = \"\";\n modelInfo[\"user_role\"] = (\"<FLM_FIX_TOKEN_\" + str(model.generation_config.user_token_id) + \"> \") if hasattr(model.generation_config, \"user_token_id\") else \"\";\n modelInfo[\"bot_role\"] = (\"<FLM_FIX_TOKEN_\" + str(model.generation_config.assistant_token_id) + \">\") if hasattr(model.generation_config, \"assistant_token_id\") else \"\";\n modelInfo[\"history_sep\"] = \"\";\n if (modelInfo[\"model_type\"] == \"qwen\"):\n if modelInfo[\"chat_format\"] == \"chatml\":\n modelInfo[\"im_end_id\"] = tokenizer.im_end_id\n modelInfo[\"im_start_id\"] = tokenizer.im_start_id\n\n\n weight_type_dict = {};\n module_dict = {};\n weight_bits = {};\n for key, m in model.named_modules():\n if (str(type(m)).find(\"QuantizedLinear\") != -1):\n weight_type_dict[key + \".weight\"] = \"QuantizedLinear\";\n weight_bits[key + \".weight\"] = m.weight_bit_width;\n if (isinstance(m, torch.nn.Linear)):\n weight_type_dict[key + \".weight\"] = \"linear\";\n module_dict[key + \".weight\"] = m;\n if (isinstance(m, torch.nn.Embedding)):\n weight_type_dict[key] = \"embedding\";\n\n peft_config = {}\n active_adapter = \"\"\n if hasattr(model, \"peft_config\"):\n peft_config = model.peft_config\n if hasattr(model, \"active_adapter\") and isinstance(model.active_adapter, str):\n # in transformers >= 4.33.0, active_adapter is a funtion in model, ignore it now\n active_adapter = model.active_adapter\n\n model = model.cpu();\n dict = model.state_dict();\n model_type = model.config.__dict__[\"model_type\"];\n model = llm.fastllm_lib.create_empty_llm_model(model_type.encode());\n for it in modelInfo.keys():\n llm.fastllm_lib.add_dict_llm_model(model, str(it).encode(), str(modelInfo[it]).encode());\n\n for adapter_name in peft_config.keys():\n adapter_dict = peft_config[adapter_name].__dict__\n for it in adapter_dict.keys():\n llm.fastllm_lib.add_adapter_dict_llm_model(model, str(adapter_name).encode(), str(it).encode(), str(adapter_dict[it]).encode())\n if len(active_adapter) != 0:\n llm.fastllm_lib.set_adapter(model, str(active_adapter).encode())\n\n # 1. vocab\n if (tokenizer):\n if (hasattr(tokenizer, \"tokenizer\")):\n if modelInfo[\"model_type\"] == \"qwen\":\n pass\n else:\n tokenizer = tokenizer.tokenizer;\n if (hasattr(tokenizer, \"sp_model\")):\n piece_size = tokenizer.sp_model.piece_size();\n for i in range(piece_size):\n llm.fastllm_lib.add_tokenizer_word_llm_model(model, tokenizer.sp_model.id_to_piece(i).encode(),\n i, ctypes.c_float(tokenizer.sp_model.get_score(i)));\n else:\n vocab = tokenizer.get_vocab();\n for v in vocab.keys():\n if (modelInfo[\"model_type\"] == \"moss\"):\n vv = [(ord(c) if c not in tokenizer.byte_decoder else tokenizer.byte_decoder[c]) for c in v];\n llm.fastllm_lib.add_tokenizer_word_llm_model(model, vv, vocab[v], ctypes.c_float(1.0));\n elif (modelInfo[\"model_type\"] == \"qwen\"):\n llm.fastllm_lib.add_tokenizer_word_llm_model(model, v, vocab[v], ctypes.c_float(1.0));\n else:\n llm.fastllm_lib.add_tokenizer_word_llm_model(model, v.encode(), vocab[v], ctypes.c_float(1.0));\n tot = 0;\n for key in dict:\n ori_data_type = 0;\n ori_np_data_type = np.float32;\n cur_weight_type = 0;\n if (key in weight_type_dict and weight_type_dict[key] in fastllm_weight_type_dict):\n cur_weight_type = fastllm_weight_type_dict[weight_type_dict[key]];\n to_data_type = 0;\n\n if (cur_weight_type == 1):\n to_data_type = fastllm_data_type_dict[dtype];\n if (to_data_type == 7):\n ori_data_type = 7;\n ori_np_data_type = np.float16;\n elif (cur_weight_type == 2):\n # TODO bfloat\n to_data_type = 0;\n\n weight_name = key\n if peft_config is not None:\n weight_name = weight_name.replace('base_model.model.', '')\n if (cur_weight_type == 111):\n llm.fastllm_lib.add_qlinear_weight_llm_model(model, weight_name.encode(),\n len(dict[key].shape),\n (ctypes.c_int * len(dict[key].shape))(*list(dict[key].shape)),\n weight_bits[key],\n dict[key + \"_scale\"].numpy().astype(np.float32).ctypes.data_as(ctypes.c_void_p),\n dict[key].numpy().ctypes.data_as(ctypes.c_void_p));\n else:\n llm.fastllm_lib.add_weight_llm_model(model, weight_name.encode(),\n len(dict[key].shape),\n (ctypes.c_int * len(dict[key].shape))(*list(dict[key].shape)),\n to_data_type, cur_weight_type, ori_data_type,\n dict[key].numpy().astype(ori_np_data_type).ctypes.data_as(ctypes.c_void_p));\n tot += 1;\n print(\"convert (\", tot, \"/\", len(dict), end = \" )\\r\");\n\n print(\"\");\n llm.fastllm_lib.init_params_llm_model(model);\n llm.fastllm_lib.warmup_llm_model(model);\n ret = llm.model(\"\", id = model);\n return ret;\n\n" }, { "path": "dcu-support/package/fastllm_pytools/llm.py", "content": "import ctypes;\nimport math\nimport os;\nimport threading\nfrom typing import Optional, Tuple, Union, List, Callable, Dict, Any;\nfrom copy import deepcopy\n\nimport platform\nif platform.system() == 'Windows':\n fastllm_lib = ctypes.cdll.LoadLibrary(os.path.join(os.path.split(os.path.realpath(__file__))[0], \"fastllm_tools.dll\"))\nelse:\n fastllm_lib = ctypes.cdll.LoadLibrary(os.path.join(os.path.split(os.path.realpath(__file__))[0], \"libfastllm_tools.so\"))\n\nfastllm_lib.create_llm_model.argtypes = [ctypes.c_char_p]\nfastllm_lib.create_llm_model.restype = ctypes.c_int\n\nfastllm_lib.token_decode.argtypes = [ctypes.c_int, ctypes.c_int, ctypes.c_int, ctypes.c_char_p]\nfastllm_lib.token_decode.restype = ctypes.c_int\n\nfastllm_lib.token_encode_string.argtypes = [ctypes.c_int, ctypes.c_char_p, ctypes.c_int, ctypes.POINTER(ctypes.c_int)]\nfastllm_lib.token_encode_string.restype = ctypes.c_int\n\nfastllm_lib.launch_response_llm_model.argtypes = [ctypes.c_int, ctypes.c_int, ctypes.c_void_p,\n ctypes.c_int, ctypes.c_bool, ctypes.c_float, ctypes.c_int,\n ctypes.c_float, ctypes.c_float, ctypes.c_bool]\nfastllm_lib.launch_response_llm_model.restype = ctypes.c_int\n\nfastllm_lib.fetch_response_llm_model.argtypes = [ctypes.c_int, ctypes.c_int]\nfastllm_lib.fetch_response_llm_model.restype = ctypes.c_int\n\nfastllm_lib.fetch_response_logits_llm_model.argtypes = [ctypes.c_int, ctypes.c_int, ctypes.POINTER(ctypes.c_float)]\nfastllm_lib.fetch_response_logits_llm_model.restype = ctypes.c_int\n\nfastllm_lib.response_str_llm_model.argtypes = [ctypes.c_int, ctypes.c_char_p,\n ctypes.c_int, ctypes.c_bool, ctypes.c_float, ctypes.c_int,\n ctypes.c_float, ctypes.c_float, ctypes.c_bool]\nfastllm_lib.response_str_llm_model.restype = ctypes.c_char_p\n\nfastllm_lib.launch_response_str_llm_model.argtype = [ctypes.c_int, ctypes.c_char_p,\n ctypes.c_int, ctypes.c_bool, ctypes.c_float, ctypes.c_int,\n ctypes.c_float, ctypes.c_float, ctypes.c_bool]\nfastllm_lib.launch_response_str_llm_model.restype = ctypes.c_int\n\nfastllm_lib.fetch_response_str_llm_model.argtypes = [ctypes.c_int, ctypes.c_int]\nfastllm_lib.fetch_response_str_llm_model.restype = ctypes.c_char_p\n\nfastllm_lib.make_history_llm_model.argtype = [ctypes.c_int, ctypes.c_char_p, ctypes.c_int, ctypes.c_char_p, ctypes.c_char_p]\nfastllm_lib.make_history_llm_model.restype = ctypes.c_char_p\n\nfastllm_lib.make_input_llm_model.argtype = [ctypes.c_int, ctypes.c_char_p, ctypes.c_int, ctypes.c_char_p]\nfastllm_lib.make_input_llm_model.restype = ctypes.c_char_p\n\nfastllm_lib.add_tokenizer_word_llm_model.argtype = [ctypes.c_int, ctypes.c_char_p, ctypes.c_float, ctypes.c_int]\n\nfastllm_lib.set_device_map.argtype = [ctypes.c_int, ctypes.c_void_p, ctypes.c_char_p, ctypes.c_void_p]\n\nfastllm_lib.get_llm_model_type.argtype = [ctypes.c_int]\nfastllm_lib.get_llm_model_type.restype = ctypes.c_char_p\n\nfastllm_lib.response_batch_str_llm_model.argtypes = [ctypes.c_int, ctypes.POINTER(ctypes.c_char_p), ctypes.c_int,\n ctypes.c_int, ctypes.c_bool, ctypes.c_float, ctypes.c_int,\n ctypes.c_float, ctypes.c_float, ctypes.c_bool]\nfastllm_lib.response_batch_str_llm_model.restype = ctypes.POINTER(ctypes.c_char_p)\n\nfastllm_lib.response_batch_tokens_llm_model.argtypes = [ctypes.c_int, ctypes.c_int, ctypes.POINTER(ctypes.c_int), ctypes.POINTER(ctypes.c_int),\n ctypes.c_int, ctypes.c_bool, ctypes.c_float, ctypes.c_int,\n ctypes.c_float, ctypes.c_float, ctypes.c_bool]\nfastllm_lib.response_batch_tokens_llm_model.restype = ctypes.POINTER(ctypes.c_char_p)\n\ndef set_cpu_threads(threads: int):\n fastllm_lib.set_cpu_threads(threads);\n\ndef get_cpu_threads() -> int:\n return fastllm_lib.get_cpu_threads();\n\ndef print_ins_info():\n fastllm_lib.print_cpu_ins();\n\ndef set_cpu_kvcache(cpu_kvcache):\n fastllm_lib.set_kvcache_in_cpu(ctypes.c_bool(cpu_kvcache));\n\ndef get_cpu_kvcache():\n return fastllm_lib.get_kvcache_in_cpu();\n\ndef set_cpu_low_mem(low_mem):\n fastllm_lib.set_cpu_low_mem(ctypes.c_bool(low_mem));\n\ndef get_cpu_low_mem():\n return fastllm_lib.get_cpu_low_mem();\n\ndef set_device_map(device_map):\n devices = [];\n values = [];\n if (isinstance(device_map, str)):\n devices.append(device_map);\n values.append(1);\n elif (isinstance(device_map, list)):\n devices = [str(x) for x in device_map];\n values = [1 for x in device_map];\n elif (isinstance(device_map, dict)):\n devices = [str(x) for x in device_map.keys()];\n values = [int(device_map[x]) for x in device_map.keys()];\n else:\n print(\"set_device_map error.\");\n return;\n device_str = ''.join(devices);\n device_len = [len(x) for x in devices];\n fastllm_lib.set_device_map(len(device_len),\n (ctypes.c_int * len(device_len))(*device_len),\n device_str.encode(),\n (ctypes.c_int * len(values))(*values));\ndef from_hf(model,\n tokenizer = None,\n dtype = \"float16\"):\n from fastllm_pytools import hf_model;\n return hf_model.create(model, tokenizer, dtype = dtype);\n\nclass model:\n def __init__ (self, path : str,\n id : int = -99999):\n if (id != -99999):\n self.model = id;\n else:\n self.model = fastllm_lib.create_llm_model(path.encode());\n self.direct_query = False;\n\n # 为了减少重复申请释放buffer对象而使用的线程局部存储区对象池\n self.thread_local_obj = threading.local()\n self.thread_local_obj.tokenizer_encode_string__output_buffer = None\n self.thread_local_obj.tokenizer_decode_token__output_buffer = None\n\n # tokenizer_decode_token 输出结果的静态缓存，手工触发构建\n # 由于token数量有限且不太多，所以缓存该结果来减少调用较为适合。\n # 不做成自动缓存是为了避免在多线程调用的时候对缓存dict加锁，同时也为不同场景提供选择空间\n self.tokenizer_decode_token_cache = None\n\n self.model_type = fastllm_lib.get_llm_model_type(self.model).decode()\n # print(\"model_type:\", self.model_type)\n\n def get_prompt(self,\n query: str,\n history: List[Tuple[str, str]] = None) -> str:\n if (not(history)):\n history = [];\n prompt = \"\";\n for i, (old_query, response) in enumerate(history):\n prompt = fastllm_lib.make_history_llm_model(self.model, prompt.encode(), i, old_query.encode(), response.encode()).decode();\n prompt = fastllm_lib.make_input_llm_model(self.model, prompt.encode(), len(history), query.encode()).decode();\n return prompt;\n\n def save(self, path : str):\n fastllm_lib.save_llm_model(self.model, path.encode());\n\n def eval(self):\n pass;\n\n def build_tokenizer_decode_token_cache(self):\n if self.tokenizer_decode_token_cache is not None:\n return\n\n cache_dict = dict()\n vocab_size = fastllm_lib.get_tokenizer_vocab_size(self.model)\n for token_id in range(vocab_size):\n cache_dict[token_id] = self.tokenizer_decode_token(token_id)\n\n self.tokenizer_decode_token_cache = cache_dict\n\n def tokenizer_encode_string(self, content: str) -> List[int]:\n output_buffer_init_len = 1024\n if self.thread_local_obj.tokenizer_encode_string__output_buffer is None:\n self.thread_local_obj.tokenizer_encode_string__output_buffer = (ctypes.c_int * output_buffer_init_len)()\n\n buffer = self.thread_local_obj.tokenizer_encode_string__output_buffer\n buffer_len = len(buffer)\n result_len = fastllm_lib.token_encode_string(self.model, content.encode(), buffer_len, buffer)\n if result_len > buffer_len:\n if result_len > 10240:\n # 要处理的数据过长，使用一次性的buffer\n temp_buffer = (ctypes.c_int * result_len)()\n ret = fastllm_lib.token_encode_string(self.model, content.encode(), result_len, temp_buffer)\n return [i for i in temp_buffer]\n else:\n # 扩展buffer大小\n new_buffer_len = round(math.ceil(result_len / 1024.0)) * 1024\n buffer = (ctypes.c_int * new_buffer_len)()\n self.thread_local_obj.tokenizer_encode_string__output_buffer = buffer\n result_len = fastllm_lib.token_encode_string(self.model, content.encode(), new_buffer_len, buffer)\n\n return [buffer[i] for i in range(result_len)]\n\n def tokenizer_decode_token(self, token_id: int) -> bytes:\n if self.tokenizer_decode_token_cache is not None:\n cache_result = self.tokenizer_decode_token_cache.get(token_id)\n if cache_result is not None:\n return cache_result\n\n output_buffer_init_len = 256\n if self.thread_local_obj.tokenizer_decode_token__output_buffer is None:\n self.thread_local_obj.tokenizer_decode_token__output_buffer = ctypes.create_string_buffer(output_buffer_init_len)\n\n buffer = self.thread_local_obj.tokenizer_decode_token__output_buffer\n ret = fastllm_lib.token_decode(self.model, token_id, len(buffer), buffer)\n if ret > 0:\n # buffer长度不够，扩展buffer大小\n new_buffer_len = round(math.ceil(ret / 16.0)) * 16\n buffer = ctypes.create_string_buffer(new_buffer_len)\n self.thread_local_obj.tokenizer_decode_token__output_buffer = buffer\n ret = fastllm_lib.token_decode(self.model, token_id, len(buffer), buffer)\n assert ret == 0\n\n buffer_bytes = buffer.raw\n result_len = len(buffer_bytes)\n for i in range(len(buffer_bytes)):\n if buffer_bytes[i] == 0:\n result_len = i\n break\n return buffer_bytes[:result_len]\n\n def response_logits(self,\n query: str,\n history: List[Tuple[str, str]] = None,\n tokenizer = None) -> str:\n prompt = query if self.direct_query else self.get_prompt(query, history);\n if (tokenizer == None):\n handle = fastllm_lib.launch_response_str_llm_model(self.model, prompt.encode(),\n ctypes.c_int(1), ctypes.c_bool(False), ctypes.c_float(1), ctypes.c_int(1),\n ctypes.c_float(1), ctypes.c_float(1), ctypes.c_bool(True));\n else:\n input = tokenizer.encode(prompt);\n handle = fastllm_lib.launch_response_llm_model(self.model, len(input), (ctypes.c_int * len(input))(*input),\n 1, False, 1, 1, 1, 1, True);\n vocab_size = fastllm_lib.get_tokenizer_vocab_size(self.model);\n logits = list(range(vocab_size))\n array = (ctypes.c_float * (vocab_size * 4))(*logits);\n ret = fastllm_lib.fetch_response_logits_llm_model(self.model, handle, array);\n out = list(array)[:vocab_size];\n while (ret != -1):\n ret = fastllm_lib.fetch_response_logits_llm_model(self.model, handle, array);\n return out;\n\n def response(self,\n query: str,\n history: List[Tuple[str, str]] = None,\n max_length: int = 8192, do_sample = True, top_p = 0.8, top_k = 1, temperature = 1.0, repeat_penalty = 1.0) -> str:\n ret = \"\";\n for i in self.stream_response(query = query,\n history = history,\n max_length = max_length,\n do_sample = do_sample,\n top_p = top_p, top_k = top_k,\n temperature = temperature,\n repeat_penalty = repeat_penalty,\n one_by_one = True):\n ret += i;\n return ret;\n\n def stream_response(self,\n query: str,\n history: List[Tuple[str, str]] = None,\n max_length: int = 8192, do_sample = True, top_p = 0.8, top_k = 1, temperature = 1.0, repeat_penalty = 1.0,\n one_by_one = True):\n prompt = query if self.direct_query else self.get_prompt(query, history);\n handle = fastllm_lib.launch_response_str_llm_model(self.model, prompt.encode(),\n ctypes.c_int(max_length), ctypes.c_bool(do_sample), ctypes.c_float(top_p), ctypes.c_int(top_k),\n ctypes.c_float(temperature), ctypes.c_float(repeat_penalty), ctypes.c_bool(False));\n res = \"\";\n ret = b'';\n fail_cnt = 0;\n while True:\n ret += fastllm_lib.fetch_response_str_llm_model(self.model, handle);\n cur = \"\";\n try:\n cur = ret.decode();\n ret = b'';\n except:\n fail_cnt += 1;\n if (fail_cnt == 20):\n break;\n else:\n continue;\n fail_cnt = 0;\n if (cur == \"<flmeos>\"):\n break;\n if one_by_one:\n yield cur;\n else:\n res += cur;\n yield res;\n\n def stream_response_raw(self,\n input_tokens: List[int],\n max_length: int = 8192, do_sample = True, top_p = 0.8, top_k = 1, temperature = 1.0, repeat_penalty = 1.0,\n one_by_one = True\n ):\n handle = fastllm_lib.launch_response_llm_model(self.model, len(input_tokens),\n (ctypes.c_int * len(input_tokens))(*input_tokens),\n ctypes.c_int(max_length), ctypes.c_bool(do_sample), ctypes.c_float(top_p), ctypes.c_int(top_k),\n ctypes.c_float(temperature), ctypes.c_float(repeat_penalty), ctypes.c_bool(False))\n\n # 可能遇到长尾char需要多个token才能够生成，所以只返回bytes，string.decode策略交给外部\n # 方便统计输出token数量，和控制不完整utf8时候解码的逻辑\n\n total_bytes = b''\n while True:\n cur_token = fastllm_lib.fetch_response_llm_model(self.model, handle)\n if cur_token == -1:\n break\n\n cur_bytes = self.tokenizer_decode_token(cur_token)\n\n if one_by_one:\n yield cur_bytes\n else:\n total_bytes += cur_bytes\n yield total_bytes\n\n def chat(self, tokenizer, query: str, history: List[Tuple[str, str]] = None, max_length: int = 8192,\n do_sample = True, top_p = 0.8, top_k = 1, temperature = 1.0, repeat_penalty = 1.0, **kwargs):\n if self.model_type != \"chatglm3\":\n if (not(history)):\n history = [];\n prompt = query if self.direct_query else self.get_prompt(query, history);\n input = tokenizer.encode(prompt);\n handle = fastllm_lib.launch_response_llm_model(self.model, len(input), (ctypes.c_int * len(input))(*input),\n max_length, do_sample, top_p, top_k, temperature, repeat_penalty,\n False);\n\n result = [];\n while True:\n cur = fastllm_lib.fetch_response_llm_model(self.model, handle);\n if (cur == -1):\n break;\n result.append(cur);\n response = tokenizer.decode(result);\n history = history + [(query, response)];\n return response, history;\n else:\n if history is None:\n history = []\n role = \"user\"\n input = self.build_chatglm3_input(tokenizer, query, history=history, role=role)\n history.append({\"role\": role, \"content\": query})\n\n handle = fastllm_lib.launch_response_llm_model(self.model, len(input), (ctypes.c_int * len(input))(*input),\n max_length, do_sample, top_p, top_k, temperature, repeat_penalty,\n False);\n tokens = [];\n while True:\n cur = fastllm_lib.fetch_response_llm_model(self.model, handle);\n if (cur == -1):\n break;\n tokens.append(cur);\n response = tokenizer.decode(tokens);\n if response and response[-1] != \"�\":\n response, new_history = self.process_chatglm3_response(response, history)\n return response, new_history\n\n def stream_chat(self, tokenizer, query: str, history: List[Tuple[str, str]] = None, past_key_values = None,\n max_length: int = 8192, do_sample = True, top_p = 0.8, top_k = 1, temperature = 1.0, repeat_penalty = 1.0,\n return_past_key_values = False, **kwargs) -> str:\n if self.model_type != \"chatglm3\":\n if (not(history)):\n history = [];\n prompt = query if self.direct_query else self.get_prompt(query, history);\n input = tokenizer.encode(prompt);\n handle = fastllm_lib.launch_response_llm_model(self.model, len(input), (ctypes.c_int * len(input))(*input),\n max_length, do_sample, top_p, top_k, temperature, repeat_penalty,\n False);\n tokens = [];\n while True:\n cur = fastllm_lib.fetch_response_llm_model(self.model, handle);\n if (cur == -1):\n break;\n tokens.append(cur);\n response = tokenizer.decode(tokens);\n new_history = history + [(query, response)];\n if return_past_key_values:\n yield response, new_history, None;\n else:\n yield response, new_history;\n else:\n if history is None:\n history = []\n role = \"user\"\n input = self.build_chatglm3_input(tokenizer, query, history=history, role=role)\n history.append({\"role\": role, \"content\": query})\n\n handle = fastllm_lib.launch_response_llm_model(self.model, len(input), (ctypes.c_int * len(input))(*input),\n max_length, do_sample, top_p, top_k, temperature, repeat_penalty,\n False);\n tokens = [];\n while True:\n cur = fastllm_lib.fetch_response_llm_model(self.model, handle);\n if (cur == -1):\n break;\n tokens.append(cur);\n response = tokenizer.decode(tokens);\n if response and response[-1] != \"�\":\n response, new_history = self.process_chatglm3_response(response, history)\n if return_past_key_values:\n yield response, new_history, past_key_values\n else:\n yield response, new_history\n\n\n def set_adapter(self, name: str):\n fastllm_lib.set_adapter(self.model, str(name).encode())\n\n def disable_adapter(self):\n fastllm_lib.disable_adapter(self.model)\n\n def process_chatglm3_response(self, output, history):\n content = \"\"\n history = deepcopy(history)\n for response in output.split(\"<|assistant|>\"):\n metadata, content = response.split(\"\\n\", maxsplit=1)\n if not metadata.strip():\n content = content.strip()\n history.append({\"role\": \"assistant\", \"metadata\": metadata, \"content\": content})\n content = content.replace(\"[[训练时间]]\", \"2023年\")\n else:\n history.append({\"role\": \"assistant\", \"metadata\": metadata, \"content\": content})\n if history[0][\"role\"] == \"system\" and \"tools\" in history[0]:\n content = \"\\n\".join(content.split(\"\\n\")[1:-1])\n def tool_call(**kwargs):\n return kwargs\n parameters = eval(content)\n content = {\"name\": metadata.strip(), \"parameters\": parameters}\n else:\n content = {\"name\": metadata.strip(), \"content\": content}\n return content, history\n\n def build_chatglm3_input(self, tokenizer, query, history=None, role=\"user\"):\n if history is None:\n history = []\n input_ids = []\n for item in history:\n content = item[\"content\"]\n if item[\"role\"] == \"system\" and \"tools\" in item:\n content = content + \"\\n\" + json.dumps(item[\"tools\"], indent=4, ensure_ascii=False)\n input_ids.extend(tokenizer.build_single_message(item[\"role\"], item.get(\"metadata\", \"\"), content))\n input_ids.extend(tokenizer.build_single_message(role, \"\", query))\n input_ids.extend([tokenizer.get_command(\"<|assistant|>\")])\n return input_ids\n\n def response_batch(self, querys: List[str],\n historys: List[List[Tuple[str, str]]] = None,\n max_length: int = 1024, do_sample = True, top_p = 0.8, top_k = 1, temperature = 1.0, repeat_penalty = 1.0,\n **kwargs) -> List[str]:\n query_size = len(querys)\n if (not(historys)):\n historys = [[] for _ in range(query_size)]\n inputs = (ctypes.c_char_p * query_size)()\n for i, query in enumerate(querys):\n prompt = query if self.direct_query else self.get_prompt(query, historys[i])\n inputs[i] = ctypes.c_char_p(prompt.encode())\n\n outputs = fastllm_lib.response_batch_str_llm_model(self.model, inputs, query_size,\n max_length, do_sample, top_p, top_k, temperature, repeat_penalty, False)\n\n responses = []\n for i in range(query_size):\n response = ctypes.string_at(outputs[i]).decode()\n responses.append(response)\n historys[i] = historys[i] + [(querys[i], response)]\n return responses, historys\n\n def chat_batch(self, tokenizer, querys: List[str], historys: List[List[Tuple[str, str]]] = None, max_length: int = 1024,\n do_sample = True, top_p = 0.8, top_k = 1, temperature = 1.0, repeat_penalty = 1.0, **kwargs):\n query_size = len(querys)\n if (not(historys)):\n historys = [[] for _ in range(query_size)]\n\n inputs = []\n inputs_len = []\n for i, query in enumerate(querys):\n prompt = query if self.direct_query else self.get_prompt(query, historys[i])\n input = tokenizer.encode(prompt);\n inputs.extend(input)\n inputs_len.append(len(input))\n\n outputs = fastllm_lib.response_batch_tokens_llm_model(self.model, query_size,\n (ctypes.c_int * len(inputs_len))(*inputs_len),\n (ctypes.c_int * len(inputs))(*inputs),\n max_length, do_sample, top_p, top_k, temperature, repeat_penalty,\n False)\n\n responses = []\n for i in range(query_size):\n response = ctypes.string_at(outputs[i]).decode()\n responses.append(response)\n historys[i] = historys[i] + [(querys[i], response)]\n return responses, historys\n\n\n" }, { "path": "dcu-support/package/fastllm_pytools/torch2flm.py", "content": "import struct\nimport numpy as np\nimport torch\n\ndef writeString(fo, s):\n fo.write(struct.pack('i', len(s)))\n fo.write(s.encode())\n\ndef writeKeyValue(fo, key, value):\n writeString(fo, key)\n writeString(fo, value)\n\nfastllm_data_type_dict = {\n \"int4\": 8,\n \"int8\": 3,\n \"float16\": 7,\n \"float32\": 0,\n}\nfastllm_weight_type_dict = {\n \"linear\": 1,\n \"embedding\": 2\n}\n\nv = np.random.randint(-127, 127, [10, 20]);\ntemp = v;\nc_max = np.expand_dims(np.abs(v).max(axis = -1), -1)\nc_scale = c_max / 127.0\nv = (v / c_scale + 128.5).clip(1, 255).astype(np.uint8)\n\ndef write_int8(fo, v):\n c_max = np.expand_dims(np.abs(v).max(axis = -1), -1).clip(0.1, 1e100)\n c_scale = c_max / 127.0\n v = (v / c_scale + 128.5).clip(1, 255).astype(np.uint8)\n fo.write(struct.pack('i', 3))\n fo.write(struct.pack('i', 0))\n for i in range(c_max.shape[0]):\n fo.write(struct.pack('f', -c_max[i][0]));\n fo.write(struct.pack('f', c_max[i][0]));\n fo.write(v.data)\n\ndef write_int4(fo, v):\n # c_min = np.expand_dims(-np.abs(v).max(axis = -1), -1)\n # c_max = np.expand_dims(np.abs(v).max(axis = -1), -1)\n # c_scale = c_max / 7.0\n # c_min = c_scale * -8.0\n\n c_min = np.expand_dims(v.min(axis = -1), -1)\n c_max = np.expand_dims(v.max(axis = -1), -1)\n c_scale = (c_max - c_min) / 15.0\n c_zero = np.round(0.0 - c_min / c_scale)\n c_zero = c_zero.clip(0, 15)\n c_min = -c_scale * c_zero\n\n v = (v - c_min) / c_scale\n v = (v + 0.5).astype(np.int8).clip(0, 15).astype(np.uint8)\n v = v[:, 0::2] * 16 + v[:, 1::2]\n fo.write(struct.pack('i', 8))\n fo.write(struct.pack('i', 0))\n for i in range(c_min.shape[0]):\n fo.write(struct.pack('f', c_min[i][0]));\n fo.write(struct.pack('f', c_max[i][0]));\n fo.write(v.data)\n\ndef tofile(exportPath,\n model,\n tokenizer = None,\n pre_prompt = None,\n user_role = None,\n bot_role = None,\n history_sep = None,\n dtype = \"float16\"):\n if (dtype not in fastllm_data_type_dict):\n print(\"dtype should in \", list(fastllm_data_type_dict.keys()))\n exit(0)\n\n dict = model.state_dict()\n fo = open(exportPath, \"wb\")\n\n # 0. version id\n fo.write(struct.pack('i', 2))\n\n # 0.1 model info\n if model.config.model_type == \"chatglm\" and model.config.transformers_version == \"4.30.2\":\n model.config.model_type = \"chatglm3\"\n modelInfo = model.config.__dict__\n if model.generation_config is not None:\n modelInfo.update(model.generation_config.__dict__)\n if (\"model_type\" not in modelInfo):\n print(\"unknown model_type.\")\n exit(0)\n\n if (pre_prompt):\n modelInfo[\"pre_prompt\"] = pre_prompt\n if (user_role):\n modelInfo[\"user_role\"] = user_role\n if (bot_role):\n modelInfo[\"bot_role\"] = bot_role\n if (history_sep):\n modelInfo[\"history_sep\"] = history_sep\n if (modelInfo[\"model_type\"] == \"baichuan\" and hasattr(model, \"model\") and hasattr(model.model, \"get_alibi_mask\")):\n # Baichuan 2代\n modelInfo[\"use_alibi\"] = \"1\"\n modelInfo[\"pre_prompt\"] = \"\"\n modelInfo[\"user_role\"] = (\"<FLM_FIX_TOKEN_\" + str(model.generation_config.user_token_id) + \">\") if hasattr(model.generation_config, \"user_token_id\") else \"\";\n modelInfo[\"bot_role\"] = (\"<FLM_FIX_TOKEN_\" + str(model.generation_config.assistant_token_id) + \">\") if hasattr(model.generation_config, \"assistant_token_id\") else \"\";\n modelInfo[\"history_sep\"] = \"\"\n if (modelInfo[\"model_type\"] == \"baichuan\" and modelInfo[\"vocab_size\"] == 125696):\n # Baichuan 2代 7B\n modelInfo[\"pre_prompt\"] = \"\"\n modelInfo[\"user_role\"] = (\"<FLM_FIX_TOKEN_\" + str(model.generation_config.user_token_id) + \">\") if hasattr(model.generation_config, \"user_token_id\") else \"\";\n modelInfo[\"bot_role\"] = (\"<FLM_FIX_TOKEN_\" + str(model.generation_config.assistant_token_id) + \">\") if hasattr(model.generation_config, \"assistant_token_id\") else \"\";\n modelInfo[\"history_sep\"] = \"\"\n if modelInfo[\"model_type\"] == \"qwen\":\n if modelInfo[\"chat_format\"] == \"chatml\":\n modelInfo[\"im_end_id\"] = tokenizer.im_end_id\n modelInfo[\"im_start_id\"] = tokenizer.im_start_id\n\n modelInfo[\"tokenizer_use_score\"] = \"1\" # 分词带分数\n\n if hasattr(model, \"peft_config\"):\n adapter_size = len(model.peft_config)\n modelInfo[\"peft_size\"] = adapter_size\n\n fo.write(struct.pack('i', len(modelInfo)))\n for it in modelInfo.keys():\n writeKeyValue(fo, str(it), str(modelInfo[it]))\n\n if hasattr(model, \"peft_config\"):\n for adapter_name in model.peft_config.keys():\n adapter_dict = model.peft_config[adapter_name].__dict__\n writeString(fo, adapter_name)\n fo.write(struct.pack('i', len(adapter_dict)))\n for it in adapter_dict.keys():\n writeKeyValue(fo, str(it), str(adapter_dict[it]))\n\n # 1. vocab\n if (tokenizer):\n if (hasattr(tokenizer, \"tokenizer\")):\n if (modelInfo['model_type'] == \"qwen\"):\n pass\n else:\n tokenizer = tokenizer.tokenizer\n if (hasattr(tokenizer, \"sp_model\")):\n piece_size = tokenizer.sp_model.piece_size()\n fo.write(struct.pack('i', piece_size))\n for i in range(piece_size):\n s = tokenizer.sp_model.id_to_piece(i).encode()\n fo.write(struct.pack('i', len(s)))\n for c in s:\n fo.write(struct.pack('i', c))\n fo.write(struct.pack('i', i))\n fo.write(struct.pack('f', float(tokenizer.sp_model.get_score(i))))\n else:\n vocab = tokenizer.get_vocab()\n fo.write(struct.pack('i', len(vocab)))\n for v in vocab.keys():\n if (modelInfo['model_type'] == \"qwen\"):\n s = v\n elif (modelInfo[\"model_type\"] == \"moss\"):\n s = [(ord(c) if c not in tokenizer.byte_decoder else tokenizer.byte_decoder[c]) for c in v]\n else:\n s = v.encode()\n fo.write(struct.pack('i', len(s)))\n for c in s:\n fo.write(struct.pack('i', c))\n fo.write(struct.pack('i', vocab[v]))\n fo.write(struct.pack('f', 1.0))\n else:\n fo.write(struct.pack('i', 0))\n\n weight_type_dict = {}\n module_dict = {}\n for key, m in model.named_modules():\n if (isinstance(m, torch.nn.Linear)):\n weight_type_dict[key + \".weight\"] = \"linear\"\n module_dict[key + \".weight\"] = m\n if (isinstance(m, torch.nn.Embedding)):\n weight_type_dict[key] = \"embedding\"\n\n # 2. weight\n fo.write(struct.pack('i', len(dict)))\n tot = 0\n for key in dict:\n ori_data_type = 0\n ori_np_data_type = np.float32\n cur_weight_type = 0\n if (key in weight_type_dict and weight_type_dict[key] in fastllm_weight_type_dict):\n cur_weight_type = fastllm_weight_type_dict[weight_type_dict[key]]\n to_data_type = 0\n if (cur_weight_type == 1):\n to_data_type = fastllm_data_type_dict[dtype]\n if (to_data_type == 7):\n ori_data_type = 7\n ori_np_data_type = np.float16\n\n cur = dict[key].numpy().astype(ori_np_data_type)\n \n if hasattr(model, \"peft_config\"):\n weight_name = key.replace('base_model.model.', '')\n fo.write(struct.pack('i', len(weight_name)))\n fo.write(weight_name.encode())\n else:\n fo.write(struct.pack('i', len(key)))\n fo.write(key.encode())\n fo.write(struct.pack('i', len(cur.shape)))\n for i in cur.shape:\n fo.write(struct.pack('i', i))\n if (to_data_type == 3):\n write_int8(fo, cur)\n elif (to_data_type == 8):\n write_int4(fo, cur)\n else:\n fo.write(struct.pack('i', to_data_type))\n fo.write(cur.data)\n tot += 1\n print(\"output (\", tot, \"/\", len(dict), end = \" )\\r\")\n print(\"\\nfinish.\")\n fo.close()" }, { "path": "dcu-support/package/setup.py", "content": "from setuptools import setup, find_packages\n\nsetup (\n name = \"fastllm_pytools\",\n version = \"0.0.1\",\n description = \"Fastllm pytools\",\n packages = ['fastllm_pytools'],\n url = \"https://developer.hpccube.com/codes/aicomponent/fastllm\",\n package_data = {\n '': ['*.dll', '*.so']\n }\n)\n" }, { "path": "dcu-support/qwen2flm.py", "content": "import sys\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\nfrom transformers.generation import GenerationConfig\nfrom fastllm_pytools import torch2flm\n\nif __name__ == \"__main__\":\n tokenizer = AutoTokenizer.from_pretrained(\"Qwen/Qwen-7B-Chat\", trust_remote_code=True)\n model = AutoModelForCausalLM.from_pretrained(\"Qwen/Qwen-7B-Chat\", device_map=\"cpu\", trust_remote_code=True, fp32=True).eval()\n model.generation_config = GenerationConfig.from_pretrained(\"Qwen/Qwen-7B-Chat\", trust_remote_code=True) # 可指定不同的生成长度、top_p等相关超参\n\n dtype = sys.argv[2] if len(sys.argv) >= 3 else \"float16\"\n exportPath = sys.argv[1] if len(sys.argv) >= 2 else \"qwen-7b-\" + dtype + \".flm\"\n torch2flm.tofile(exportPath, model, tokenizer, dtype = dtype)" }, { "path": "dcu-support/requirements.txt", "content": "transformers==4.32.0\ntiktoken\nstreamlit>=1.24.0\nsentencepiece\nurllib3==1.26.16\ntransformers_stream_generator==0.0.4\naccelerate\neinops\n#scipy\n" }, { "path": "dcu-support/web_demo.py", "content": "import streamlit as st\nfrom streamlit_chat import message\nfrom fastllm_pytools import llm\nimport sys\n\nst.set_page_config(\n page_title=\"fastllm web demo\",\n page_icon=\":robot:\"\n)\n\n@st.cache_resource\ndef get_model():\n model = llm.model(sys.argv[1])\n return model\n\nif \"messages\" not in st.session_state:\n st.session_state.messages = []\n\nfor i, (prompt, response) in enumerate(st.session_state.messages):\n with st.chat_message(\"user\"):\n st.markdown(prompt)\n with st.chat_message(\"assistant\"):\n st.markdown(response)\n\nif prompt := st.chat_input(\"请开始对话\"):\n model = get_model()\n with st.chat_message(\"user\"):\n st.markdown(prompt)\n\n with st.chat_message(\"assistant\"):\n message_placeholder = st.empty()\n full_response = \"\"\n for chunk in model.stream_response(prompt, st.session_state.messages, one_by_one = True):\n full_response += chunk\n message_placeholder.markdown(full_response + \"▌\")\n message_placeholder.markdown(full_response)\n st.session_state.messages.append((prompt, full_response))\n" }, { "path": "docker/Dockerfile", "content": "ARG CUDA_VERSION=11.7.1\nARG from=nvidia/cuda:${CUDA_VERSION}-cudnn8-devel-ubuntu20.04\n\nFROM ${from} as base\n\nARG from\n\nRUN <<EOF\napt update -y && apt upgrade -y && apt install -y --no-install-recommends \\\n git \\\n git-lfs \\\n python3 \\\n python3-pip \\\n python3-dev \\\n wget \\\n vim \\\n&& rm -rf /var/lib/apt/lists/*\nEOF\n\nRUN ln -s /usr/bin/python3 /usr/bin/python\n\nRUN git lfs install\n\nFROM base as dev\n\nWORKDIR /\n\nRUN mkdir -p /data/shared/Qwen\n\nWORKDIR /data/shared/Qwen/\n\n# Users can also mount '/data/shared/Qwen/' to keep the data\nCOPY ../requirements.txt ./\nCOPY ../requirements_web_demo.txt ./\n\nFROM dev as bundle_req\n\nARG BUNDLE_REQUIREMENTS=true\n\nRUN <<EOF\nif [ \"$BUNDLE_REQUIREMENTS\" = \"true\" ]; then \n cd /data/shared/Qwen\n pip3 install torch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2\n pip3 install -r requirements.txt\n pip3 install -r requirements_web_demo.txt\nfi\nEOF\n\nFROM bundle_req as bundle_flash_attention\nARG BUNDLE_FLASH_ATTENTION=true\n\nRUN <<EOF \nif [ \"$BUNDLE_FLASH_ATTENTION\" = \"true\" ]; then\n cd /data/shared/Qwen \n test -d flash-attention || git clone -b v2.3.3 https://github.com/Dao-AILab/flash-attention\n cd /data/shared/Qwen/flash-attention &&\n pip3 install . &&\n pip3 install csrc/layer_norm\nfi\nEOF\n\nFROM bundle_flash_attention as bundle_finetune\nARG BUNDLE_FINETUNE=true\n\nRUN <<EOF\nif [ \"$BUNDLE_FINETUNE\" = \"true\" ]; then\n cd /data/shared/Qwen\n\n # Full-finetune / LoRA.\n pip3 install deepspeed \"peft==0.5.0\"\n\n # Q-LoRA.\n apt update -y && DEBIAN_FRONTEND=noninteractive apt install -y --no-install-recommends \\\n libopenmpi-dev openmpi-bin \\\n && rm -rf /var/lib/apt/lists/*\n pip3 install \"optimum==1.12.0\" \"auto-gptq==0.4.2\" mpi4py\nfi\nEOF\n\nFROM bundle_finetune as bundle_openai_api\nARG BUNDLE_OPENAI_API=true\n\nRUN <<EOF\nif [ \"$BUNDLE_OPENAI_API\" = \"true\" ]; then\n cd /data/shared/Qwen\n\n pip3 install fastapi uvicorn \"openai<1.0.0\" sse_starlette \"pydantic<=1.10.13\"\nfi\nEOF\n\nFROM bundle_openai_api as final\nARG from\n\nCOPY ../requirements.txt ./\nCOPY ../requirements_web_demo.txt ./\nCOPY ../cli_demo.py ./\nCOPY ../web_demo.py ./\nCOPY ../openai_api.py ./\nCOPY ../finetune.py ./\nCOPY ../utils.py ./\nCOPY ./examples/* ./examples/\nCOPY ./eval/* ./eval/\nCOPY ./finetune/* ./finetune/\n\nEXPOSE 80\n\nWORKDIR /data/shared/Qwen/\n\nCMD [\"python3\", \"web_demo.py\", \"--server-port\", \"80\", \"--server-name\", \"0.0.0.0\", \"-c\", \"/data/shared/Qwen/Qwen-Chat/\"]\n" }, { "path": "docker/Dockerfile-cu114", "content": "ARG CUDA_VERSION=11.4.3\nARG from=nvidia/cuda:${CUDA_VERSION}-cudnn8-devel-ubuntu20.04\n\nFROM ${from} as base\n\nARG from\n\nRUN <<EOF\napt update -y && apt upgrade -y && apt install -y --no-install-recommends \\\n git \\\n git-lfs \\\n python3 \\\n python3-pip \\\n python3-dev \\\n wget \\\n vim \\\n&& rm -rf /var/lib/apt/lists/*\nEOF\n\nRUN ln -s /usr/bin/python3 /usr/bin/python\n\nRUN git lfs install\n\nFROM base as dev\n\nWORKDIR /\n\nRUN mkdir -p /data/shared/Qwen\n\nWORKDIR /data/shared/Qwen/\n\n# Users can also mount '/data/shared/Qwen/' to keep the data\nCOPY ../requirements.txt ./\nCOPY ../requirements_web_demo.txt ./\n\nFROM dev as bundle_req\n\nARG BUNDLE_REQUIREMENTS=true\n\nRUN <<EOF\nif [ \"$BUNDLE_REQUIREMENTS\" = \"true\" ]; then \n cd /data/shared/Qwen\n pip3 install torch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2\n pip3 install -r requirements.txt\n pip3 install -r requirements_web_demo.txt\nfi\nEOF\n\nFROM bundle_req as bundle_flash_attention\nARG BUNDLE_FLASH_ATTENTION=true\n\nRUN <<EOF\nif [ \"$BUNDLE_FLASH_ATTENTION\" = \"true\" ]; then\n echo \"CUDA 11.4 does not support flash-attention, please try other images.\"\nfi\nEOF\n\nFROM bundle_flash_attention as bundle_finetune\nARG BUNDLE_FINETUNE=true\n\nRUN <<EOF\nif [ \"$BUNDLE_FINETUNE\" = \"true\" ]; then\n cd /data/shared/Qwen\n\n # Full-finetune / LoRA.\n pip3 install deepspeed \"peft==0.5.0\"\n\n # Q-LoRA.\n apt update -y && DEBIAN_FRONTEND=noninteractive apt install -y --no-install-recommends \\\n libopenmpi-dev openmpi-bin \\\n && rm -rf /var/lib/apt/lists/*\n pip3 install \"optimum==1.12.0\" \"auto-gptq==0.4.2\" mpi4py\nfi\nEOF\n\nFROM bundle_finetune as bundle_openai_api\nARG BUNDLE_OPENAI_API=true\n\nRUN <<EOF\nif [ \"$BUNDLE_OPENAI_API\" = \"true\" ]; then\n cd /data/shared/Qwen\n\n pip3 install fastapi uvicorn \"openai<1.0.0\" sse_starlette \"pydantic<=1.10.13\"\nfi\nEOF\n\nFROM bundle_openai_api as final\nARG from\n\nCOPY ../requirements.txt ./\nCOPY ../requirements_web_demo.txt ./\nCOPY ../cli_demo.py ./\nCOPY ../web_demo.py ./\nCOPY ../openai_api.py ./\nCOPY ../finetune.py ./\nCOPY ../utils.py ./\nCOPY ./examples/* ./examples/\nCOPY ./eval/* ./eval/\nCOPY ./finetune/* ./finetune/\n\nEXPOSE 80\n\nWORKDIR /data/shared/Qwen/\n\nCMD [\"python3\", \"web_demo.py\", \"--server-port\", \"80\", \"--server-name\", \"0.0.0.0\", \"-c\", \"/data/shared/Qwen/Qwen-Chat/\"]\n" }, { "path": "docker/Dockerfile-cu121", "content": "ARG CUDA_VERSION=12.1.0\nARG from=nvidia/cuda:${CUDA_VERSION}-cudnn8-devel-ubuntu20.04\n\nFROM ${from} as base\n\nARG from\n\nRUN <<EOF\napt update -y && apt upgrade -y && apt install -y --no-install-recommends \\\n git \\\n git-lfs \\\n python3 \\\n python3-pip \\\n python3-dev \\\n wget \\\n vim \\\n&& rm -rf /var/lib/apt/lists/*\nEOF\n\nRUN ln -s /usr/bin/python3 /usr/bin/python\n\nRUN git lfs install\n\nFROM base as dev\n\nWORKDIR /\n\nRUN mkdir -p /data/shared/Qwen\n\nWORKDIR /data/shared/Qwen/\n\n# Users can also mount '/data/shared/Qwen/' to keep the data\nCOPY ../requirements.txt ./\nCOPY ../requirements_web_demo.txt ./\n\nFROM dev as bundle_req\n\nARG BUNDLE_REQUIREMENTS=true\n\nRUN <<EOF\nif [ \"$BUNDLE_REQUIREMENTS\" = \"true\" ]; then \n cd /data/shared/Qwen\n pip3 install torch==2.1.2 torchvision==0.16.2 torchaudio==2.1.2 --index-url https://download.pytorch.org/whl/cu121\n pip3 install -r requirements.txt\n pip3 install -r requirements_web_demo.txt\n\n pip3 install transformers==4.36.0\nfi\nEOF\n\nFROM bundle_req as bundle_flash_attention\nARG BUNDLE_FLASH_ATTENTION=true\n\nRUN <<EOF \nif [ \"$BUNDLE_FLASH_ATTENTION\" = \"true\" ]; then\n cd /data/shared/Qwen \n test -d flash-attention || git clone -b v2.3.3 https://github.com/Dao-AILab/flash-attention\n cd /data/shared/Qwen/flash-attention &&\n pip3 install . &&\n pip3 install csrc/layer_norm\nfi\nEOF\n\nFROM bundle_flash_attention as bundle_finetune\nARG BUNDLE_FINETUNE=true\n\nRUN <<EOF\nif [ \"$BUNDLE_FINETUNE\" = \"true\" ]; then\n cd /data/shared/Qwen\n\n # Full-finetune / LoRA.\n pip3 install \"deepspeed==0.12.6\" \"peft==0.7.1\"\n\n # Q-LoRA.\n apt update -y && DEBIAN_FRONTEND=noninteractive apt install -y --no-install-recommends \\\n libopenmpi-dev openmpi-bin \\\n && rm -rf /var/lib/apt/lists/*\n pip3 install \"optimum==1.14.0\" \"auto-gptq==0.5.0\" mpi4py\nfi\nEOF\n\nFROM bundle_finetune as bundle_openai_api\nARG BUNDLE_OPENAI_API=true\n\nRUN <<EOF\nif [ \"$BUNDLE_OPENAI_API\" = \"true\" ]; then\n cd /data/shared/Qwen\n\n pip3 install fastapi uvicorn \"openai<1.0.0\" sse_starlette \"pydantic<=1.10.13\"\nfi\nEOF\n\nFROM bundle_openai_api as bundle_vllm\nARG BUNDLE_VLLM=true\n\nRUN <<EOF\nif [ \"$BUNDLE_VLLM\" = \"true\" ]; then\n cd /data/shared/Qwen\n\n pip3 install vllm==0.2.7 \"fschat[model_worker,webui]==0.2.33\"\nfi\nEOF\n\nFROM bundle_vllm as final\nARG from\n\nCOPY ../requirements.txt ./\nCOPY ../requirements_web_demo.txt ./\nCOPY ../cli_demo.py ./\nCOPY ../web_demo.py ./\nCOPY ../openai_api.py ./\nCOPY ../finetune.py ./\nCOPY ../utils.py ./\nCOPY ./examples/* ./examples/\nCOPY ./eval/* ./eval/\nCOPY ./finetune/* ./finetune/\n\nEXPOSE 80\n\nWORKDIR /data/shared/Qwen/\n\nCMD [\"python3\", \"web_demo.py\", \"--server-port\", \"80\", \"--server-name\", \"0.0.0.0\", \"-c\", \"/data/shared/Qwen/Qwen-Chat/\"]\n" }, { "path": "docker/docker_cli_demo.sh", "content": "#!/usr/bin/env bash\n#\n# This script will automatically pull docker image from DockerHub, and start a container to run the Qwen-Chat cli-demo.\n\nIMAGE_NAME=qwenllm/qwen:cu117\nQWEN_CHECKPOINT_PATH=/path/to/Qwen-Chat\nCONTAINER_NAME=qwen\n\nfunction usage() {\n echo '\nUsage: bash docker/docker_cli_demo.sh [-i IMAGE_NAME] -c [/path/to/Qwen-Chat] [-n CONTAINER_NAME]\n'\n}\n\nwhile [[ \"$1\" != \"\" ]]; do\n case $1 in\n -i | --image-name )\n shift\n IMAGE_NAME=$1\n ;;\n -c | --checkpoint )\n shift\n QWEN_CHECKPOINT_PATH=$1\n ;;\n -n | --container-name )\n shift\n CONTAINER_NAME=$1\n ;;\n -h | --help )\n usage\n exit 0\n ;;\n * )\n echo \"Unknown argument ${1}\"\n exit 1\n ;;\n esac\n shift\ndone\n\nif [ ! -e ${QWEN_CHECKPOINT_PATH}/config.json ]; then\n echo \"Checkpoint config.json file not found in ${QWEN_CHECKPOINT_PATH}, exit.\"\n exit 1\nfi\n\nsudo docker pull ${IMAGE_NAME} || {\n echo \"Pulling image ${IMAGE_NAME} failed, exit.\"\n exit 1\n}\n\nsudo docker run --gpus all --rm --name ${CONTAINER_NAME} \\\n --mount type=bind,source=${QWEN_CHECKPOINT_PATH},target=/data/shared/Qwen/Qwen-Chat \\\n -it ${IMAGE_NAME} \\\n python cli_demo.py -c /data/shared/Qwen/Qwen-Chat/\n" }, { "path": "docker/docker_openai_api.sh", "content": "#!/usr/bin/env bash\n#\n# This script will automatically pull docker image from DockerHub, and start a daemon container to run the Qwen-Chat OpenAI API.\n\nIMAGE_NAME=qwenllm/qwen:cu117\nQWEN_CHECKPOINT_PATH=/path/to/Qwen-Chat\nPORT=8000\nCONTAINER_NAME=qwen\n\nfunction usage() {\n echo '\nUsage: bash docker/docker_openai_api.sh [-i IMAGE_NAME] -c [/path/to/Qwen-Chat] [-n CONTAINER_NAME] [--port PORT]\n'\n}\n\nwhile [[ \"$1\" != \"\" ]]; do\n case $1 in\n -i | --image-name )\n shift\n IMAGE_NAME=$1\n ;;\n -c | --checkpoint )\n shift\n QWEN_CHECKPOINT_PATH=$1\n ;;\n -n | --container-name )\n shift\n CONTAINER_NAME=$1\n ;;\n --port )\n shift\n PORT=$1\n ;;\n -h | --help )\n usage\n exit 0\n ;;\n * )\n echo \"Unknown argument ${1}\"\n exit 1\n ;;\n esac\n shift\ndone\n\nif [ ! -e ${QWEN_CHECKPOINT_PATH}/config.json ]; then\n echo \"Checkpoint config.json file not found in ${QWEN_CHECKPOINT_PATH}, exit.\"\n exit 1\nfi\n\nsudo docker pull ${IMAGE_NAME} || {\n echo \"Pulling image ${IMAGE_NAME} failed, exit.\"\n exit 1\n}\n\nsudo docker run --gpus all -d --restart always --name ${CONTAINER_NAME} \\\n -v /var/run/docker.sock:/var/run/docker.sock -p ${PORT}:80 \\\n --mount type=bind,source=${QWEN_CHECKPOINT_PATH},target=/data/shared/Qwen/Qwen-Chat \\\n -it ${IMAGE_NAME} \\\n python openai_api.py --server-port 80 --server-name 0.0.0.0 -c /data/shared/Qwen/Qwen-Chat/ && {\n echo \"Successfully started OpenAI API server. Access 'http://localhost:${PORT}/v1' to try!\nRun \\`docker logs ${CONTAINER_NAME}\\` to check server status.\nRun \\`docker rm -f ${CONTAINER_NAME}\\` to stop and remove the server.\"\n}\n" }, { "path": "docker/docker_web_demo.sh", "content": "#!/usr/bin/env bash\n#\n# This script will automatically pull docker image from DockerHub, and start a daemon container to run the Qwen-Chat web-demo.\n\nIMAGE_NAME=qwenllm/qwen:cu117\nQWEN_CHECKPOINT_PATH=/path/to/Qwen-7B-Chat\nPORT=8901\nCONTAINER_NAME=qwen\n\nfunction usage() {\n echo '\nUsage: bash docker/docker_web_demo.sh [-i IMAGE_NAME] -c [/path/to/Qwen-Chat] [-n CONTAINER_NAME] [--port PORT]\n'\n}\n\nwhile [[ \"$1\" != \"\" ]]; do\n case $1 in\n -i | --image-name )\n shift\n IMAGE_NAME=$1\n ;;\n -c | --checkpoint )\n shift\n QWEN_CHECKPOINT_PATH=$1\n ;;\n -n | --container-name )\n shift\n CONTAINER_NAME=$1\n ;;\n --port )\n shift\n PORT=$1\n ;;\n -h | --help )\n usage\n exit 0\n ;;\n * )\n echo \"Unknown argument ${1}\"\n exit 1\n ;;\n esac\n shift\ndone\n\nif [ ! -e ${QWEN_CHECKPOINT_PATH}/config.json ]; then\n echo \"Checkpoint config.json file not found in ${QWEN_CHECKPOINT_PATH}, exit.\"\n exit 1\nfi\n\nsudo docker pull ${IMAGE_NAME} || {\n echo \"Pulling image ${IMAGE_NAME} failed, exit.\"\n exit 1\n}\n\nsudo docker run --gpus all -d --restart always --name ${CONTAINER_NAME} \\\n -v /var/run/docker.sock:/var/run/docker.sock -p ${PORT}:80 \\\n --mount type=bind,source=${QWEN_CHECKPOINT_PATH},target=/data/shared/Qwen/Qwen-Chat \\\n -it ${IMAGE_NAME} \\\n python web_demo.py --server-port 80 --server-name 0.0.0.0 -c /data/shared/Qwen/Qwen-Chat/ && {\n echo \"Successfully started web demo. Open 'http://localhost:${PORT}' to try!\nRun \\`docker logs ${CONTAINER_NAME}\\` to check demo status.\nRun \\`docker rm -f ${CONTAINER_NAME}\\` to stop and remove the demo.\"\n}\n" }, { "path": "eval/EVALUATION.md", "content": "## 评测复现\n\n- CEVAL\n\n```Shell\nwget https://huggingface.co/datasets/ceval/ceval-exam/resolve/main/ceval-exam.zip\nmkdir data/ceval\nmv ceval-exam.zip data/ceval\ncd data/ceval; unzip ceval-exam.zip\ncd ../../\n\n# Qwen-7B\npython evaluate_ceval.py -d data/ceval/\n\n# Qwen-7B-Chat (We only provide 0-shot reproduction scripts. 5-shot results are obtained by OpenCompass (https://github.com/InternLM/opencompass).)\npip install thefuzz\npython evaluate_chat_ceval.py -d data/ceval/\n```\n\n- MMLU\n\n```Shell\nwget https://people.eecs.berkeley.edu/~hendrycks/data.tar\nmkdir data/mmlu\nmv data.tar data/mmlu\ncd data/mmlu; tar xf data.tar\ncd ../../\n\n# Qwen-7B\npython evaluate_mmlu.py -d data/mmlu/data/\n\n# Qwen-7B-Chat (We only provide 0-shot reproduction scripts. 5-shot results are obtained by OpenCompass (https://github.com/InternLM/opencompass).)\npip install thefuzz\npython evaluate_chat_mmlu.py -d data/mmlu/data/\n```\n\n- CMMLU\n\n```Shell\nwget https://huggingface.co/datasets/haonan-li/cmmlu/resolve/main/cmmlu_v1_0_1.zip\nmkdir data/cmmlu\nmv cmmlu_v1_0_1.zip data/cmmlu\ncd data/cmmlu; unzip cmmlu_v1_0_1.zip\ncd ../../\n\n# Qwen-7B\npython evaluate_cmmlu.py -d data/cmmlu/\n```\n\n- HumanEval\n\nGet the HumanEval.jsonl file from [here](https://github.com/openai/human-eval/tree/master/data)\n\n```Shell\ngit clone https://github.com/openai/human-eval\npip install -e human-eval\n\n# Qwen-7B\npython evaluate_humaneval.py -f HumanEval.jsonl -o HumanEval_res.jsonl\nevaluate_functional_correctness HumanEval_res.jsonl\n# Qwen-7B-Chat\npython evaluate_chat_humaneval.py -f HumanEval.jsonl -o HumanEval_res_chat.jsonl\nevaluate_functional_correctness HumanEval_res_chat.jsonl\n```\n\nWhen installing package human-eval, please note its following disclaimer:\n\nThis program exists to run untrusted model-generated code. Users are strongly encouraged not to do so outside of a robust security sandbox. The execution call in execution.py is deliberately commented out to ensure users read this disclaimer before running code in a potentially unsafe manner. See the comment in execution.py for more information and instructions.\n\n- GSM8K\n\n```Shell\n# Qwen-7B\npython evaluate_gsm8k.py\n\n# Qwen-7B-Chat (We only provide 0-shot reproduction scripts. 5-shot results are obtained by OpenCompass (https://github.com/InternLM/opencompass).)\npython evaluate_chat_gsm8k.py # zeroshot\n```\n\n- PLUGIN\n\nThis script is used to reproduce the results of the ReAct and Hugging Face Agent in the Tool Usage section of the README document.\n\n```Shell\n# Qwen-7B-Chat\nmkdir data;\ncd data;\n## Old Evaluation Dataset (Version 20230803)\n# wget https://qianwen-res.oss-cn-beijing.aliyuncs.com/opensource_data/exam_plugin_v1/exam_plugin_v1_react_positive.jsonl;\n# wget https://qianwen-res.oss-cn-beijing.aliyuncs.com/opensource_data/exam_plugin_v1/exam_plugin_v1_react_negative.jsonl;\n## New Evaluation Dataset (Version 20231206)\nwget https://qianwen-res.oss-cn-beijing.aliyuncs.com/opensource_data/exam_plugin_v20231206/exam_plugin_v20231206_react_positive.jsonl;\nwget https://qianwen-res.oss-cn-beijing.aliyuncs.com/opensource_data/exam_plugin_v20231206/exam_plugin_v20231206_react_negative.jsonl;cd ..;\npip install json5;\npip install jsonlines;\npip install rouge_score;\npython evaluate_plugin.py --eval-react-positive --eval-react-negative --eval-hfagent\n```\n" }, { "path": "eval/evaluate_ceval.py", "content": "import os\nfrom typing import List\nimport argparse\nimport torch\nimport pandas as pd\nimport numpy as np\nfrom tqdm import tqdm\nfrom transformers.trainer_utils import set_seed\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\nfrom transformers.generation import GenerationConfig\n\n'''\nwget https://huggingface.co/datasets/ceval/ceval-exam/resolve/main/ceval-exam.zip\nmkdir data/ceval\nmv ceval-exam.zip data/ceval\ncd data/ceval; unzip ceval-exam.zip\ncd ../../\npython evaluate_ceval.py -d data/ceval/\n'''\n\ndef load_models_tokenizer(args):\n tokenizer = AutoTokenizer.from_pretrained(\n args.checkpoint_path,\n pad_token='<|extra_0|>',\n eos_token='<|endoftext|>',\n padding_side='left',\n trust_remote_code=True\n )\n model = AutoModelForCausalLM.from_pretrained(\n args.checkpoint_path,\n pad_token_id=tokenizer.pad_token_id,\n device_map=\"auto\",\n trust_remote_code=True\n ).eval()\n model.generation_config = GenerationConfig.from_pretrained(\n args.checkpoint_path,\n pad_token_id=tokenizer.pad_token_id,\n trust_remote_code=True\n )\n return model, tokenizer\n\n\ndef format_example(line, include_answer=True):\n example = \"问题：\" + line[\"question\"]\n for choice in choices:\n example += f'\\n{choice}. {line[f\"{choice}\"]}'\n\n if include_answer:\n example += \"\\n答案：\" + line[\"answer\"] + \"\\n\\n\"\n else:\n example += \"\\n答案：\"\n return example\n\n\ndef generate_few_shot_prompt(k, subject, dev_df):\n prompt = \"\"\n if k == -1:\n k = dev_df.shape[0]\n for i in range(k):\n prompt += format_example(\n dev_df.iloc[i, :],\n include_answer=True,\n )\n return prompt\n\n\ndef get_logits(tokenizer, model, inputs: List[str]):\n input_ids = tokenizer(inputs, padding='longest')[\"input_ids\"]\n input_ids = torch.tensor(input_ids, device=model.device)\n tokens = {\"input_ids\": input_ids}\n attention_mask = input_ids.ne(tokenizer.pad_token_id)\n\n outputs = model(input_ids, attention_mask=attention_mask)[\"logits\"]\n logits = outputs[:, -1, :]\n log_probs = torch.nn.functional.softmax(logits, dim=-1)\n return log_probs, {\"tokens\": tokens}\n\n\n@torch.no_grad()\ndef eval_subject(\n model,\n tokenizer,\n subject_name,\n test_df,\n k=5,\n dev_df=None,\n few_shot=False,\n save_result_dir=None,\n batch_size=1,\n **kwargs,\n):\n result = []\n score = []\n\n few_shot_prompt = (\n generate_few_shot_prompt(k, subject_name, dev_df) if few_shot else \"\"\n )\n all_probs = {\"prob_A\": [], \"prob_B\": [], \"prob_C\": [], \"prob_D\": []}\n if args.debug:\n print(f\"few_shot_prompt: {few_shot_prompt}\")\n\n choices_ids = torch.tensor(\n tokenizer(\"A\")[\"input_ids\"] + tokenizer(\"B\")[\"input_ids\"] +\n tokenizer(\"C\")[\"input_ids\"] + tokenizer(\"D\")[\"input_ids\"]\n ).unsqueeze(0).to(model.device)\n\n idx_list = list(range(0, len(test_df), batch_size))\n for i in tqdm(idx_list):\n full_prompt_list = []\n answer_list = []\n for row in test_df.iloc[i:i+batch_size].to_dict(orient='records'):\n question = format_example(row, include_answer=False)\n full_prompt = few_shot_prompt + question\n full_prompt_list.append(full_prompt)\n if 'answer' in row:\n answer_list.append(row['answer'])\n\n logits, input_info = get_logits(tokenizer, model, full_prompt_list)\n softval = logits.gather(1, choices_ids.expand(logits.size(0), -1)).softmax(1)\n if softval.dtype in {torch.bfloat16, torch.float16}:\n softval = softval.to(dtype=torch.float32)\n probs = softval.detach().cpu().numpy()\n\n for i in range(len(probs)):\n for j, choice in enumerate(choices):\n all_probs[f\"prob_{choice}\"].append(probs[i][j])\n pred = {0: \"A\", 1: \"B\", 2: \"C\", 3: \"D\"}[np.argmax(probs[i])]\n\n if answer_list != []:\n correct = 1 if pred == answer_list[i] else 0\n score.append(correct)\n if args.debug:\n print(f'{question} pred: {pred} ref: {answer_list[i]}')\n result.append(pred)\n\n if score:\n correct_ratio = 100 * sum(score) / len(score)\n if args.debug:\n print(subject_name, correct_ratio)\n else:\n correct_ratio = 0\n if save_result_dir:\n test_df[\"model_output\"] = result\n for i, choice in enumerate(choices):\n test_df[f\"prob_{choice}\"] = all_probs[f\"prob_{choice}\"]\n if score:\n test_df[\"correctness\"] = score\n os.makedirs(save_result_dir, exist_ok=True)\n test_df.to_csv(\n os.path.join(save_result_dir, f\"{subject_name}_result.csv\"),\n encoding=\"utf-8\",\n index=False,\n )\n\n return correct_ratio\n\n\ndef cal_ceval(res):\n acc_sum_dict = dict()\n acc_norm_sum_dict = dict()\n cnt_dict = dict()\n acc_sum = 0.0\n cnt = 0\n hard_cnt = 0\n hard_acc_sum = 0.0\n for tt in res.keys():\n name = tt.split(\"-\")[-1]\n acc_sum += float(res[tt])\n cnt += 1\n class_ = TASK_NAME_MAPPING[name][2]\n if class_ not in acc_sum_dict:\n acc_sum_dict[class_] = 0.0\n acc_norm_sum_dict[class_] = 0.0\n cnt_dict[class_] = 0.0\n if name in hard_list:\n hard_cnt += 1\n hard_acc_sum += float(res[tt])\n acc_sum_dict[class_] += float(res[tt])\n cnt_dict[class_] += 1\n print(\"\\n\\n\\n\")\n for k in [\"STEM\", \"Social Science\", \"Humanities\", \"Other\"]:\n if k in cnt_dict:\n print(\"%s acc: %.2f \" % (k, acc_sum_dict[k] / cnt_dict[k]))\n if hard_cnt > 0:\n print(\"Hard acc:%.2f \" % (hard_acc_sum / hard_cnt))\n print(\"AVERAGE acc:%.2f \" % (acc_sum / cnt))\n\n\nTASK_NAME_MAPPING = {\n \"computer_network\": [\"Computer Network\", \"\\u8ba1\\u7b97\\u673a\\u7f51\\u7edc\", \"STEM\"],\n \"operating_system\": [\"Operating System\", \"\\u64cd\\u4f5c\\u7cfb\\u7edf\", \"STEM\"],\n \"computer_architecture\": [\n \"Computer Architecture\",\n \"\\u8ba1\\u7b97\\u673a\\u7ec4\\u6210\",\n \"STEM\",\n ],\n \"college_programming\": [\"College Programming\", \"\\u5927\\u5b66\\u7f16\\u7a0b\", \"STEM\"],\n \"college_physics\": [\"College Physics\", \"\\u5927\\u5b66\\u7269\\u7406\", \"STEM\"],\n \"college_chemistry\": [\"College Chemistry\", \"\\u5927\\u5b66\\u5316\\u5b66\", \"STEM\"],\n \"advanced_mathematics\": [\n \"Advanced Mathematics\",\n \"\\u9ad8\\u7b49\\u6570\\u5b66\",\n \"STEM\",\n ],\n \"probability_and_statistics\": [\n \"Probability and Statistics\",\n \"\\u6982\\u7387\\u7edf\\u8ba1\",\n \"STEM\",\n ],\n \"discrete_mathematics\": [\n \"Discrete Mathematics\",\n \"\\u79bb\\u6563\\u6570\\u5b66\",\n \"STEM\",\n ],\n \"electrical_engineer\": [\n \"Electrical Engineer\",\n \"\\u6ce8\\u518c\\u7535\\u6c14\\u5de5\\u7a0b\\u5e08\",\n \"STEM\",\n ],\n \"metrology_engineer\": [\n \"Metrology Engineer\",\n \"\\u6ce8\\u518c\\u8ba1\\u91cf\\u5e08\",\n \"STEM\",\n ],\n \"high_school_mathematics\": [\n \"High School Mathematics\",\n \"\\u9ad8\\u4e2d\\u6570\\u5b66\",\n \"STEM\",\n ],\n \"high_school_physics\": [\"High School Physics\", \"\\u9ad8\\u4e2d\\u7269\\u7406\", \"STEM\"],\n \"high_school_chemistry\": [\n \"High School Chemistry\",\n \"\\u9ad8\\u4e2d\\u5316\\u5b66\",\n \"STEM\",\n ],\n \"high_school_biology\": [\"High School Biology\", \"\\u9ad8\\u4e2d\\u751f\\u7269\", \"STEM\"],\n \"middle_school_mathematics\": [\n \"Middle School Mathematics\",\n \"\\u521d\\u4e2d\\u6570\\u5b66\",\n \"STEM\",\n ],\n \"middle_school_biology\": [\n \"Middle School Biology\",\n \"\\u521d\\u4e2d\\u751f\\u7269\",\n \"STEM\",\n ],\n \"middle_school_physics\": [\n \"Middle School Physics\",\n \"\\u521d\\u4e2d\\u7269\\u7406\",\n \"STEM\",\n ],\n \"middle_school_chemistry\": [\n \"Middle School Chemistry\",\n \"\\u521d\\u4e2d\\u5316\\u5b66\",\n \"STEM\",\n ],\n \"veterinary_medicine\": [\"Veterinary Medicine\", \"\\u517d\\u533b\\u5b66\", \"STEM\"],\n \"college_economics\": [\n \"College Economics\",\n \"\\u5927\\u5b66\\u7ecf\\u6d4e\\u5b66\",\n \"Social Science\",\n ],\n \"business_administration\": [\n \"Business Administration\",\n \"\\u5de5\\u5546\\u7ba1\\u7406\",\n \"Social Science\",\n ],\n \"marxism\": [\n \"Marxism\",\n \"\\u9a6c\\u514b\\u601d\\u4e3b\\u4e49\\u57fa\\u672c\\u539f\\u7406\",\n \"Social Science\",\n ],\n \"mao_zedong_thought\": [\n \"Mao Zedong Thought\",\n \"\\u6bdb\\u6cfd\\u4e1c\\u601d\\u60f3\\u548c\\u4e2d\\u56fd\\u7279\\u8272\\u793e\\u4f1a\\u4e3b\\u4e49\\u7406\\u8bba\\u4f53\\u7cfb\\u6982\\u8bba\",\n \"Social Science\",\n ],\n \"education_science\": [\"Education Science\", \"\\u6559\\u80b2\\u5b66\", \"Social Science\"],\n \"teacher_qualification\": [\n \"Teacher Qualification\",\n \"\\u6559\\u5e08\\u8d44\\u683c\",\n \"Social Science\",\n ],\n \"high_school_politics\": [\n \"High School Politics\",\n \"\\u9ad8\\u4e2d\\u653f\\u6cbb\",\n \"Social Science\",\n ],\n \"high_school_geography\": [\n \"High School Geography\",\n \"\\u9ad8\\u4e2d\\u5730\\u7406\",\n \"Social Science\",\n ],\n \"middle_school_politics\": [\n \"Middle School Politics\",\n \"\\u521d\\u4e2d\\u653f\\u6cbb\",\n \"Social Science\",\n ],\n \"middle_school_geography\": [\n \"Middle School Geography\",\n \"\\u521d\\u4e2d\\u5730\\u7406\",\n \"Social Science\",\n ],\n \"modern_chinese_history\": [\n \"Modern Chinese History\",\n \"\\u8fd1\\u4ee3\\u53f2\\u7eb2\\u8981\",\n \"Humanities\",\n ],\n \"ideological_and_moral_cultivation\": [\n \"Ideological and Moral Cultivation\",\n \"\\u601d\\u60f3\\u9053\\u5fb7\\u4fee\\u517b\\u4e0e\\u6cd5\\u5f8b\\u57fa\\u7840\",\n \"Humanities\",\n ],\n \"logic\": [\"Logic\", \"\\u903b\\u8f91\\u5b66\", \"Humanities\"],\n \"law\": [\"Law\", \"\\u6cd5\\u5b66\", \"Humanities\"],\n \"chinese_language_and_literature\": [\n \"Chinese Language and Literature\",\n \"\\u4e2d\\u56fd\\u8bed\\u8a00\\u6587\\u5b66\",\n \"Humanities\",\n ],\n \"art_studies\": [\"Art Studies\", \"\\u827a\\u672f\\u5b66\", \"Humanities\"],\n \"professional_tour_guide\": [\n \"Professional Tour Guide\",\n \"\\u5bfc\\u6e38\\u8d44\\u683c\",\n \"Humanities\",\n ],\n \"legal_professional\": [\n \"Legal Professional\",\n \"\\u6cd5\\u5f8b\\u804c\\u4e1a\\u8d44\\u683c\",\n \"Humanities\",\n ],\n \"high_school_chinese\": [\n \"High School Chinese\",\n \"\\u9ad8\\u4e2d\\u8bed\\u6587\",\n \"Humanities\",\n ],\n \"high_school_history\": [\n \"High School History\",\n \"\\u9ad8\\u4e2d\\u5386\\u53f2\",\n \"Humanities\",\n ],\n \"middle_school_history\": [\n \"Middle School History\",\n \"\\u521d\\u4e2d\\u5386\\u53f2\",\n \"Humanities\",\n ],\n \"civil_servant\": [\"Civil Servant\", \"\\u516c\\u52a1\\u5458\", \"Other\"],\n \"sports_science\": [\"Sports Science\", \"\\u4f53\\u80b2\\u5b66\", \"Other\"],\n \"plant_protection\": [\"Plant Protection\", \"\\u690d\\u7269\\u4fdd\\u62a4\", \"Other\"],\n \"basic_medicine\": [\"Basic Medicine\", \"\\u57fa\\u7840\\u533b\\u5b66\", \"Other\"],\n \"clinical_medicine\": [\"Clinical Medicine\", \"\\u4e34\\u5e8a\\u533b\\u5b66\", \"Other\"],\n \"urban_and_rural_planner\": [\n \"Urban and Rural Planner\",\n \"\\u6ce8\\u518c\\u57ce\\u4e61\\u89c4\\u5212\\u5e08\",\n \"Other\",\n ],\n \"accountant\": [\"Accountant\", \"\\u6ce8\\u518c\\u4f1a\\u8ba1\\u5e08\", \"Other\"],\n \"fire_engineer\": [\n \"Fire Engineer\",\n \"\\u6ce8\\u518c\\u6d88\\u9632\\u5de5\\u7a0b\\u5e08\",\n \"Other\",\n ],\n \"environmental_impact_assessment_engineer\": [\n \"Environmental Impact Assessment Engineer\",\n \"\\u73af\\u5883\\u5f71\\u54cd\\u8bc4\\u4ef7\\u5de5\\u7a0b\\u5e08\",\n \"Other\",\n ],\n \"tax_accountant\": [\"Tax Accountant\", \"\\u7a0e\\u52a1\\u5e08\", \"Other\"],\n \"physician\": [\"Physician\", \"\\u533b\\u5e08\\u8d44\\u683c\", \"Other\"],\n}\nhard_list = [\n \"advanced_mathematics\",\n \"discrete_mathematics\",\n \"probability_and_statistics\",\n \"college_physics\",\n \"college_chemistry\",\n \"high_school_mathematics\",\n \"high_school_physics\",\n \"high_school_chemistry\",\n]\nchoices = [\"A\", \"B\", \"C\", \"D\"]\n\n\ndef main(args):\n model, tokenizer = load_models_tokenizer(args)\n\n dev_result = {}\n for subject_name in tqdm(TASK_NAME_MAPPING.keys()):\n val_file_path = os.path.join(\n args.eval_data_path, \"val\", f\"{subject_name}_val.csv\"\n )\n dev_file_path = os.path.join(\n args.eval_data_path, \"dev\", f\"{subject_name}_dev.csv\"\n )\n # test_file_path = os.path.join(args.eval_data_path, 'test', f'{subject_name}_test.csv')\n val_df = pd.read_csv(val_file_path)\n dev_df = pd.read_csv(dev_file_path)\n # test_df = pd.read_csv(test_file_path)\n\n score = eval_subject(\n model,\n tokenizer,\n subject_name,\n val_df,\n dev_df=dev_df,\n k=5,\n few_shot=True,\n save_result_dir=f\"outs/ceval_eval_result\",\n batch_size=args.batch_size\n )\n dev_result[subject_name] = score\n cal_ceval(dev_result)\n\n\nif __name__ == \"__main__\":\n parser = argparse.ArgumentParser(description=\"Test HF checkpoint.\")\n parser.add_argument(\n \"-c\",\n \"--checkpoint-path\",\n type=str,\n help=\"Checkpoint path\",\n default=\"Qwen/Qwen-7B\",\n )\n parser.add_argument(\"-s\", \"--seed\", type=int, default=1234, help=\"Random seed\")\n\n # Provide extra arguments required for tasks\n group = parser.add_argument_group(title=\"Evaluation options\")\n group.add_argument(\n \"-d\", \"--eval_data_path\", type=str, required=True, help=\"Path to eval data\"\n )\n group.add_argument(\n \"--max-seq-len\",\n type=int,\n default=2048,\n help=\"Size of the output generated text.\",\n )\n group.add_argument(\n \"--debug\", action=\"store_true\", default=False, help=\"Print infos.\"\n )\n group.add_argument(\n \"--batch-size\",\n type=int,\n default=1,\n help=\"batch size\",\n )\n\n args = parser.parse_args()\n set_seed(args.seed)\n\n main(args)\n" }, { "path": "eval/evaluate_chat_ceval.py", "content": "import os\r\nimport argparse\r\nimport re\r\nimport torch\r\nimport pandas as pd\r\nfrom thefuzz import process\r\nfrom tqdm import tqdm\r\nfrom transformers.trainer_utils import set_seed\r\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\r\nfrom transformers.generation import GenerationConfig\r\n\r\n'''\r\nwget https://huggingface.co/datasets/ceval/ceval-exam/resolve/main/ceval-exam.zip\r\nmkdir data/ceval\r\nmv ceval-exam.zip data/ceval\r\ncd data/ceval; unzip ceval-exam.zip\r\ncd ../../\r\n\r\npip install thefuzz\r\npython eval/evaluate_chat_ceval.py -d data/ceval\r\n'''\r\n\r\ndef load_models_tokenizer(args):\r\n tokenizer = AutoTokenizer.from_pretrained(\r\n args.checkpoint_path, trust_remote_code=True\r\n )\r\n model = AutoModelForCausalLM.from_pretrained(\r\n args.checkpoint_path, device_map=\"auto\", trust_remote_code=True\r\n ).eval()\r\n model.generation_config = GenerationConfig.from_pretrained(\r\n args.checkpoint_path, trust_remote_code=True\r\n )\r\n model.generation_config.do_sample = False # use greedy decoding\r\n model.generation_config.repetition_penalty = 1.0 # disable repetition penalty\r\n return model, tokenizer\r\n\r\ndef process_before_extraction(gen, question, choice_dict):\r\n # Example Prompt:\r\n # 关于传输层的面向连接服务的特性是____。\r\n # A. 既不保证可靠，也不保证按序交付\r\n # B. 不保证可靠，但保证按序交付\r\n # C. 保证可靠，但不保证按序交付\r\n # D. 既保证可靠，也保证按序交付\r\n # Example Model Output：\r\n # 关于传输层的面向连接服务的特性是既保证可靠，也保证按序交付\r\n # Processed Output:\r\n # 答案是D\r\n\r\n question_split = question.rstrip(\"。\").split(\"。\")[-1].split(\"_\")\r\n\r\n # replacing the question\r\n if len(question_split[0].strip()) > 4:\r\n gen = gen.replace(question_split[0], \"答案是\")\r\n if len(question_split[-1].strip()) > 4:\r\n gen = gen.replace(question_split[-1], \"\")\r\n\r\n # replace the choice by letter in the generated sentence\r\n # from longest one to shortest one\r\n for key, val in sorted(choice_dict.items(), key=lambda x: len(x[1]), reverse=True):\r\n gen = gen.replace(val.rstrip(\"。\"), key)\r\n return gen\r\n\r\n\r\ndef count_substr(gen, pattern):\r\n return len(re.findall(pattern, gen))\r\n\r\n\r\ndef extract_choice(gen, prompt, choice_list):\r\n # 答案是A | 选项是A | 应该选A选项\r\n res = re.search(\r\n r\"(?:(?:选|选择|选定)[：:]?\\s*|(?:(?:答案|选项)(?![^ABCD]{0,10}?(?:不|非)[^ABCD]{0,10}?(?:是|选|为|：|:|】))[^ABCD]{0,10}?(?:是|选|为|：|:|】))[^ABCD]{0,10}?)(A|B|C|D)(?:选项)?(?:\\)|。|\\.|，|,|．|、|A|B|C|D|$|：|:|\\)|）)\",\r\n gen,\r\n )\r\n\r\n # A选项正确 | A选项符合题意\r\n if res is None:\r\n res = re.search(\r\n r\"(A|B|C|D)(?:选?项)?(?![^ABCD]{0,4}?(?:不|非)[^ABCD]{0,4}?(?:正确|对[的，。：]|符合))[^ABCD]{0,4}?(?:正确|对[的，。：]|符合)\",\r\n gen,\r\n )\r\n\r\n # 直接输出 A\r\n if res is None:\r\n res = re.search(r\"^[\\(（]?(A|B|C|D)(?:。|\\)|）|\\.|，|,|．|：|:|$)\", gen)\r\n\r\n # 获取第一个出现的字母\r\n if res is None:\r\n res = re.search(r\"(?<![a-zA-Z])(A|B|C|D)(?![a-zA-Z=])\", gen)\r\n\r\n if res is None:\r\n return choices[choice_list.index(process.extractOne(gen, choice_list)[0])]\r\n return res.group(1)\r\n\r\n\r\ndef format_example(line):\r\n example = line[\"question\"] + \"\\n\\n\"\r\n for choice in choices:\r\n example += f'{choice}. {line[f\"{choice}\"]}\\n'\r\n return example\r\n\r\n\r\ndef extract_answer(response, row):\r\n prompt = row[\"question\"]\r\n gen = process_before_extraction(\r\n response, prompt, {choice: row[choice] for choice in choices}\r\n )\r\n if not isinstance(prompt, str):\r\n prompt = prompt[0]\r\n pred = extract_choice(gen, prompt, [row[choice] for choice in choices])\r\n return pred\r\n\r\n\r\n@torch.no_grad()\r\ndef eval_subject(\r\n model,\r\n tokenizer,\r\n subject_name,\r\n test_df,\r\n save_result_dir=None,\r\n overwrite=False,\r\n **kwargs\r\n):\r\n result_path = os.path.join(save_result_dir, f\"{subject_name}_result.csv\")\r\n if not overwrite and os.path.exists(result_path):\r\n print(f\"{result_path} existed, skip!\")\r\n score = []\r\n for (_, datarow), (_, resultrow) in zip(\r\n test_df.iterrows(), pd.read_csv(result_path).iterrows()\r\n ):\r\n pred = extract_answer(resultrow[\"model_response\"], datarow)\r\n correct = 1 if pred == datarow[\"answer\"] else 0\r\n score.append(correct)\r\n correct_ratio = 100 * sum(score) / len(score)\r\n return correct_ratio\r\n\r\n responses = []\r\n result = []\r\n score = []\r\n\r\n for _, row in tqdm(test_df.iterrows(), total=len(test_df)):\r\n question = format_example(row)\r\n\r\n response, _ = model.chat(\r\n tokenizer,\r\n question,\r\n history=None,\r\n )\r\n print(question)\r\n print(response)\r\n pred = extract_answer(response, row)\r\n print(pred)\r\n print(\"======================\")\r\n\r\n if \"answer\" in row:\r\n correct = 1 if pred == row[\"answer\"] else 0\r\n score.append(correct)\r\n if args.debug:\r\n print(f'{question} pred: {pred} ref: {row[\"answer\"]}')\r\n responses.append(response)\r\n result.append(pred)\r\n\r\n if score:\r\n correct_ratio = 100 * sum(score) / len(score)\r\n if args.debug:\r\n print(subject_name, correct_ratio)\r\n else:\r\n correct_ratio = 0\r\n if save_result_dir:\r\n test_df[\"model_response\"] = responses\r\n test_df[\"model_output\"] = result\r\n if score:\r\n test_df[\"correctness\"] = score\r\n os.makedirs(save_result_dir, exist_ok=True)\r\n test_df.to_csv(result_path, encoding=\"utf-8\", index=False)\r\n\r\n return correct_ratio\r\n\r\n\r\ndef cal_ceval(res):\r\n acc_sum_dict = dict()\r\n acc_norm_sum_dict = dict()\r\n cnt_dict = dict()\r\n acc_sum = 0.0\r\n cnt = 0\r\n hard_cnt = 0\r\n hard_acc_sum = 0.0\r\n for tt in res.keys():\r\n name = tt.split(\"-\")[-1]\r\n acc_sum += float(res[tt])\r\n cnt += 1\r\n class_ = TASK_NAME_MAPPING[name][2]\r\n if class_ not in acc_sum_dict:\r\n acc_sum_dict[class_] = 0.0\r\n acc_norm_sum_dict[class_] = 0.0\r\n cnt_dict[class_] = 0.0\r\n if name in hard_list:\r\n hard_cnt += 1\r\n hard_acc_sum += float(res[tt])\r\n acc_sum_dict[class_] += float(res[tt])\r\n cnt_dict[class_] += 1\r\n print(\"\\n\\n\\n\")\r\n for k in [\"STEM\", \"Social Science\", \"Humanities\", \"Other\"]:\r\n if k in cnt_dict:\r\n print(\"%s acc: %.2f \" % (k, acc_sum_dict[k] / cnt_dict[k]))\r\n if hard_cnt > 0:\r\n print(\"Hard acc:%.2f \" % (hard_acc_sum / hard_cnt))\r\n print(\"AVERAGE acc:%.2f \" % (acc_sum / cnt))\r\n\r\n\r\nTASK_NAME_MAPPING = {\r\n \"computer_network\": [\"Computer Network\", \"\\u8ba1\\u7b97\\u673a\\u7f51\\u7edc\", \"STEM\"],\r\n \"operating_system\": [\"Operating System\", \"\\u64cd\\u4f5c\\u7cfb\\u7edf\", \"STEM\"],\r\n \"computer_architecture\": [\r\n \"Computer Architecture\",\r\n \"\\u8ba1\\u7b97\\u673a\\u7ec4\\u6210\",\r\n \"STEM\",\r\n ],\r\n \"college_programming\": [\"College Programming\", \"\\u5927\\u5b66\\u7f16\\u7a0b\", \"STEM\"],\r\n \"college_physics\": [\"College Physics\", \"\\u5927\\u5b66\\u7269\\u7406\", \"STEM\"],\r\n \"college_chemistry\": [\"College Chemistry\", \"\\u5927\\u5b66\\u5316\\u5b66\", \"STEM\"],\r\n \"advanced_mathematics\": [\r\n \"Advanced Mathematics\",\r\n \"\\u9ad8\\u7b49\\u6570\\u5b66\",\r\n \"STEM\",\r\n ],\r\n \"probability_and_statistics\": [\r\n \"Probability and Statistics\",\r\n \"\\u6982\\u7387\\u7edf\\u8ba1\",\r\n \"STEM\",\r\n ],\r\n \"discrete_mathematics\": [\r\n \"Discrete Mathematics\",\r\n \"\\u79bb\\u6563\\u6570\\u5b66\",\r\n \"STEM\",\r\n ],\r\n \"electrical_engineer\": [\r\n \"Electrical Engineer\",\r\n \"\\u6ce8\\u518c\\u7535\\u6c14\\u5de5\\u7a0b\\u5e08\",\r\n \"STEM\",\r\n ],\r\n \"metrology_engineer\": [\r\n \"Metrology Engineer\",\r\n \"\\u6ce8\\u518c\\u8ba1\\u91cf\\u5e08\",\r\n \"STEM\",\r\n ],\r\n \"high_school_mathematics\": [\r\n \"High School Mathematics\",\r\n \"\\u9ad8\\u4e2d\\u6570\\u5b66\",\r\n \"STEM\",\r\n ],\r\n \"high_school_physics\": [\"High School Physics\", \"\\u9ad8\\u4e2d\\u7269\\u7406\", \"STEM\"],\r\n \"high_school_chemistry\": [\r\n \"High School Chemistry\",\r\n \"\\u9ad8\\u4e2d\\u5316\\u5b66\",\r\n \"STEM\",\r\n ],\r\n \"high_school_biology\": [\"High School Biology\", \"\\u9ad8\\u4e2d\\u751f\\u7269\", \"STEM\"],\r\n \"middle_school_mathematics\": [\r\n \"Middle School Mathematics\",\r\n \"\\u521d\\u4e2d\\u6570\\u5b66\",\r\n \"STEM\",\r\n ],\r\n \"middle_school_biology\": [\r\n \"Middle School Biology\",\r\n \"\\u521d\\u4e2d\\u751f\\u7269\",\r\n \"STEM\",\r\n ],\r\n \"middle_school_physics\": [\r\n \"Middle School Physics\",\r\n \"\\u521d\\u4e2d\\u7269\\u7406\",\r\n \"STEM\",\r\n ],\r\n \"middle_school_chemistry\": [\r\n \"Middle School Chemistry\",\r\n \"\\u521d\\u4e2d\\u5316\\u5b66\",\r\n \"STEM\",\r\n ],\r\n \"veterinary_medicine\": [\"Veterinary Medicine\", \"\\u517d\\u533b\\u5b66\", \"STEM\"],\r\n \"college_economics\": [\r\n \"College Economics\",\r\n \"\\u5927\\u5b66\\u7ecf\\u6d4e\\u5b66\",\r\n \"Social Science\",\r\n ],\r\n \"business_administration\": [\r\n \"Business Administration\",\r\n \"\\u5de5\\u5546\\u7ba1\\u7406\",\r\n \"Social Science\",\r\n ],\r\n \"marxism\": [\r\n \"Marxism\",\r\n \"\\u9a6c\\u514b\\u601d\\u4e3b\\u4e49\\u57fa\\u672c\\u539f\\u7406\",\r\n \"Social Science\",\r\n ],\r\n \"mao_zedong_thought\": [\r\n \"Mao Zedong Thought\",\r\n \"\\u6bdb\\u6cfd\\u4e1c\\u601d\\u60f3\\u548c\\u4e2d\\u56fd\\u7279\\u8272\\u793e\\u4f1a\\u4e3b\\u4e49\\u7406\\u8bba\\u4f53\\u7cfb\\u6982\\u8bba\",\r\n \"Social Science\",\r\n ],\r\n \"education_science\": [\"Education Science\", \"\\u6559\\u80b2\\u5b66\", \"Social Science\"],\r\n \"teacher_qualification\": [\r\n \"Teacher Qualification\",\r\n \"\\u6559\\u5e08\\u8d44\\u683c\",\r\n \"Social Science\",\r\n ],\r\n \"high_school_politics\": [\r\n \"High School Politics\",\r\n \"\\u9ad8\\u4e2d\\u653f\\u6cbb\",\r\n \"Social Science\",\r\n ],\r\n \"high_school_geography\": [\r\n \"High School Geography\",\r\n \"\\u9ad8\\u4e2d\\u5730\\u7406\",\r\n \"Social Science\",\r\n ],\r\n \"middle_school_politics\": [\r\n \"Middle School Politics\",\r\n \"\\u521d\\u4e2d\\u653f\\u6cbb\",\r\n \"Social Science\",\r\n ],\r\n \"middle_school_geography\": [\r\n \"Middle School Geography\",\r\n \"\\u521d\\u4e2d\\u5730\\u7406\",\r\n \"Social Science\",\r\n ],\r\n \"modern_chinese_history\": [\r\n \"Modern Chinese History\",\r\n \"\\u8fd1\\u4ee3\\u53f2\\u7eb2\\u8981\",\r\n \"Humanities\",\r\n ],\r\n \"ideological_and_moral_cultivation\": [\r\n \"Ideological and Moral Cultivation\",\r\n \"\\u601d\\u60f3\\u9053\\u5fb7\\u4fee\\u517b\\u4e0e\\u6cd5\\u5f8b\\u57fa\\u7840\",\r\n \"Humanities\",\r\n ],\r\n \"logic\": [\"Logic\", \"\\u903b\\u8f91\\u5b66\", \"Humanities\"],\r\n \"law\": [\"Law\", \"\\u6cd5\\u5b66\", \"Humanities\"],\r\n \"chinese_language_and_literature\": [\r\n \"Chinese Language and Literature\",\r\n \"\\u4e2d\\u56fd\\u8bed\\u8a00\\u6587\\u5b66\",\r\n \"Humanities\",\r\n ],\r\n \"art_studies\": [\"Art Studies\", \"\\u827a\\u672f\\u5b66\", \"Humanities\"],\r\n \"professional_tour_guide\": [\r\n \"Professional Tour Guide\",\r\n \"\\u5bfc\\u6e38\\u8d44\\u683c\",\r\n \"Humanities\",\r\n ],\r\n \"legal_professional\": [\r\n \"Legal Professional\",\r\n \"\\u6cd5\\u5f8b\\u804c\\u4e1a\\u8d44\\u683c\",\r\n \"Humanities\",\r\n ],\r\n \"high_school_chinese\": [\r\n \"High School Chinese\",\r\n \"\\u9ad8\\u4e2d\\u8bed\\u6587\",\r\n \"Humanities\",\r\n ],\r\n \"high_school_history\": [\r\n \"High School History\",\r\n \"\\u9ad8\\u4e2d\\u5386\\u53f2\",\r\n \"Humanities\",\r\n ],\r\n \"middle_school_history\": [\r\n \"Middle School History\",\r\n \"\\u521d\\u4e2d\\u5386\\u53f2\",\r\n \"Humanities\",\r\n ],\r\n \"civil_servant\": [\"Civil Servant\", \"\\u516c\\u52a1\\u5458\", \"Other\"],\r\n \"sports_science\": [\"Sports Science\", \"\\u4f53\\u80b2\\u5b66\", \"Other\"],\r\n \"plant_protection\": [\"Plant Protection\", \"\\u690d\\u7269\\u4fdd\\u62a4\", \"Other\"],\r\n \"basic_medicine\": [\"Basic Medicine\", \"\\u57fa\\u7840\\u533b\\u5b66\", \"Other\"],\r\n \"clinical_medicine\": [\"Clinical Medicine\", \"\\u4e34\\u5e8a\\u533b\\u5b66\", \"Other\"],\r\n \"urban_and_rural_planner\": [\r\n \"Urban and Rural Planner\",\r\n \"\\u6ce8\\u518c\\u57ce\\u4e61\\u89c4\\u5212\\u5e08\",\r\n \"Other\",\r\n ],\r\n \"accountant\": [\"Accountant\", \"\\u6ce8\\u518c\\u4f1a\\u8ba1\\u5e08\", \"Other\"],\r\n \"fire_engineer\": [\r\n \"Fire Engineer\",\r\n \"\\u6ce8\\u518c\\u6d88\\u9632\\u5de5\\u7a0b\\u5e08\",\r\n \"Other\",\r\n ],\r\n \"environmental_impact_assessment_engineer\": [\r\n \"Environmental Impact Assessment Engineer\",\r\n \"\\u73af\\u5883\\u5f71\\u54cd\\u8bc4\\u4ef7\\u5de5\\u7a0b\\u5e08\",\r\n \"Other\",\r\n ],\r\n \"tax_accountant\": [\"Tax Accountant\", \"\\u7a0e\\u52a1\\u5e08\", \"Other\"],\r\n \"physician\": [\"Physician\", \"\\u533b\\u5e08\\u8d44\\u683c\", \"Other\"],\r\n}\r\nhard_list = [\r\n \"advanced_mathematics\",\r\n \"discrete_mathematics\",\r\n \"probability_and_statistics\",\r\n \"college_physics\",\r\n \"college_chemistry\",\r\n \"high_school_mathematics\",\r\n \"high_school_physics\",\r\n \"high_school_chemistry\",\r\n]\r\nchoices = [\"A\", \"B\", \"C\", \"D\"]\r\n\r\n\r\ndef main(args):\r\n print(\"loading model weights\")\r\n if args.checkpoint_path:\r\n model, tokenizer = load_models_tokenizer(args)\r\n else:\r\n model, tokenizer = None, None\r\n print(\"model loaded\")\r\n dev_result = {}\r\n for subject_name in tqdm(TASK_NAME_MAPPING.keys()):\r\n val_file_path = os.path.join(\r\n args.eval_data_path, \"val\", f\"{subject_name}_val.csv\"\r\n )\r\n val_df = pd.read_csv(val_file_path)\r\n\r\n score = eval_subject(\r\n model,\r\n tokenizer,\r\n subject_name,\r\n val_df,\r\n save_result_dir=\"outs_chat/ceval_eval_result\",\r\n overwrite=args.overwrite,\r\n )\r\n dev_result[subject_name] = score\r\n cal_ceval(dev_result)\r\n\r\n\r\nif __name__ == \"__main__\":\r\n parser = argparse.ArgumentParser(description=\"Test HF checkpoint.\")\r\n parser.add_argument(\r\n \"-c\",\r\n \"--checkpoint-path\",\r\n type=str,\r\n help=\"Checkpoint path\",\r\n default=\"Qwen/Qwen-7B-Chat\",\r\n )\r\n parser.add_argument(\"-s\", \"--seed\", type=int, default=1234, help=\"Random seed\")\r\n\r\n # Provide extra arguments required for tasks\r\n group = parser.add_argument_group(title=\"Evaluation options\")\r\n group.add_argument(\r\n \"-d\", \"--eval_data_path\", type=str, required=True, help=\"Path to eval data\"\r\n )\r\n group.add_argument(\r\n \"--debug\", action=\"store_true\", default=False, help=\"Print infos.\"\r\n )\r\n group.add_argument(\r\n \"--overwrite\",\r\n action=\"store_true\",\r\n default=False,\r\n help=\"Overwrite existed results\",\r\n )\r\n\r\n args = parser.parse_args()\r\n set_seed(args.seed)\r\n\r\n main(args)\r\n" }, { "path": "eval/evaluate_chat_gsm8k.py", "content": "import json\r\nimport re\r\nfrom pathlib import Path\r\nimport argparse\r\nimport requests\r\nimport math\r\nimport numpy as np\r\nimport tqdm\r\nfrom datasets import load_from_disk, load_dataset\r\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\r\nfrom transformers.generation import GenerationConfig\r\n\r\n\"\"\"\r\npython eval/evaluate_chat_gsm8k.py [--use-fewshot]\r\n\"\"\"\r\n\r\nINVALID_ANS = \"[invalid]\"\r\nDEVICE = \"cuda:0\"\r\n\r\ndef doc_to_text(doc, use_fewshot):\r\n if use_fewshot:\r\n context = (\r\n \"Question: Angelo and Melanie want to plan how many hours over the next week they should study together for their test next week. They have 2 chapters of their textbook to study and 4 worksheets to memorize. They figure out that they should dedicate 3 hours to each chapter of their textbook and 1.5 hours for each worksheet. If they plan to study no more than 4 hours each day, how many days should they plan to study total over the next week if they take a 10-minute break every hour, include 3 10-minute snack breaks each day, and 30 minutes for lunch each day?\\nLet's think step by step\\n\"\r\n \"Angelo and Melanie think they should dedicate 3 hours to each of the 2 chapters, 3 hours x 2 chapters = 6 hours total.\\nFor the worksheets they plan to dedicate 1.5 hours for each worksheet, 1.5 hours x 4 worksheets = 6 hours total.\\nAngelo and Melanie need to start with planning 12 hours to study, at 4 hours a day, 12 / 4 = 3 days.\\nHowever, they need to include time for breaks and lunch. Every hour they want to include a 10-minute break, so 12 total hours x 10 minutes = 120 extra minutes for breaks.\\nThey also want to include 3 10-minute snack breaks, 3 x 10 minutes = 30 minutes.\\nAnd they want to include 30 minutes for lunch each day, so 120 minutes for breaks + 30 minutes for snack breaks + 30 minutes for lunch = 180 minutes, or 180 / 60 minutes per hour = 3 extra hours.\\nSo Angelo and Melanie want to plan 12 hours to study + 3 hours of breaks = 15 hours total.\\nThey want to study no more than 4 hours each day, 15 hours / 4 hours each day = 3.75\\nThey will need to plan to study 4 days to allow for all the time they need.\\nThe answer is 4\\n\\n\"\r\n \"Question: Mark's basketball team scores 25 2 pointers, 8 3 pointers and 10 free throws. Their opponents score double the 2 pointers but half the 3 pointers and free throws. What's the total number of points scored by both teams added together?\\nLet's think step by step\\n\"\r\n \"Mark's team scores 25 2 pointers, meaning they scored 25*2= 50 points in 2 pointers.\\nHis team also scores 6 3 pointers, meaning they scored 8*3= 24 points in 3 pointers\\nThey scored 10 free throws, and free throws count as one point so they scored 10*1=10 points in free throws.\\nAll together his team scored 50+24+10= 84 points\\nMark's opponents scored double his team's number of 2 pointers, meaning they scored 50*2=100 points in 2 pointers.\\nHis opponents scored half his team's number of 3 pointers, meaning they scored 24/2= 12 points in 3 pointers.\\nThey also scored half Mark's team's points in free throws, meaning they scored 10/2=5 points in free throws.\\nAll together Mark's opponents scored 100+12+5=117 points\\nThe total score for the game is both team's scores added together, so it is 84+117=201 points\\nThe answer is 201\\n\\n\"\r\n \"Question: Bella has two times as many marbles as frisbees. She also has 20 more frisbees than deck cards. If she buys 2/5 times more of each item, what would be the total number of the items she will have if she currently has 60 marbles?\\nLet's think step by step\\n\"\r\n \"When Bella buys 2/5 times more marbles, she'll have increased the number of marbles by 2/5*60 = 24\\nThe total number of marbles she'll have is 60+24 = 84\\nIf Bella currently has 60 marbles, and she has two times as many marbles as frisbees, she has 60/2 = 30 frisbees.\\nIf Bella buys 2/5 times more frisbees, she'll have 2/5*30 = 12 more frisbees.\\nThe total number of frisbees she'll have will increase to 30+12 = 42\\nBella also has 20 more frisbees than deck cards, meaning she has 30-20 = 10 deck cards\\nIf she buys 2/5 times more deck cards, she'll have 2/5*10 = 4 more deck cards.\\nThe total number of deck cards she'll have is 10+4 = 14\\nTogether, Bella will have a total of 14+42+84 = 140 items\\nThe answer is 140\\n\\n\"\r\n \"Question: A group of 4 fruit baskets contains 9 apples, 15 oranges, and 14 bananas in the first three baskets and 2 less of each fruit in the fourth basket. How many fruits are there?\\nLet's think step by step\\n\"\r\n \"For the first three baskets, the number of apples and oranges in one basket is 9+15=24\\nIn total, together with bananas, the number of fruits in one basket is 24+14=38 for the first three baskets.\\nSince there are three baskets each having 38 fruits, there are 3*38=114 fruits in the first three baskets.\\nThe number of apples in the fourth basket is 9-2=7\\nThere are also 15-2=13 oranges in the fourth basket\\nThe combined number of oranges and apples in the fourth basket is 13+7=20\\nThe fourth basket also contains 14-2=12 bananas.\\nIn total, the fourth basket has 20+12=32 fruits.\\nThe four baskets together have 32+114=146 fruits.\\nThe answer is 146\\n\\n\"\r\n f\"Question: {doc['question']}\\nLet's think step by step\"\r\n )\r\n else:\r\n context = doc[\"question\"]\r\n return context\r\n\r\ndef generate_sample(model, tokenizer, question):\r\n response, _ = model.chat(\r\n tokenizer,\r\n question,\r\n history=None,\r\n )\r\n print(question)\r\n print(\"-------------\")\r\n print(response)\r\n print(\"=============\")\r\n return response\r\n\r\ndef extract_answer(s):\r\n _PAT_LAST_DIGIT = re.compile(\r\n r\"([+-])?(?=([0-9]|\\.[0-9]))(0|([1-9](\\d{0,2}(,\\d{3})*)|\\d*))?(\\.\\d*)?(?=\\D|$)\"\r\n )\r\n match = list(_PAT_LAST_DIGIT.finditer(s))\r\n if match:\r\n last_digit = match[-1].group().replace(\",\", \"\").replace(\"+\", \"\").strip()\r\n # print(f\"The last digit in {s} is {last_digit}\")\r\n else:\r\n last_digit = None\r\n print(f\"No digits found in {s!r}\", flush=True)\r\n return last_digit\r\n\r\ndef is_correct(completion, answer):\r\n gold = extract_answer(answer)\r\n assert gold is not None, \"No ground truth answer found in the document.\"\r\n\r\n def number_equal(answer, pred):\r\n if pred is None:\r\n return False\r\n try:\r\n return math.isclose(eval(answer), eval(pred), rel_tol=0, abs_tol=1e-4)\r\n except:\r\n print(\r\n f\"cannot compare two numbers: answer={answer}, pred={pred}\", flush=True\r\n )\r\n return False\r\n\r\n return number_equal(gold, extract_answer(completion))\r\n\r\n\r\nif __name__ == \"__main__\":\r\n parser = argparse.ArgumentParser(description=\"Test HF checkpoint.\")\r\n parser.add_argument(\r\n \"-c\",\r\n \"--checkpoint-path\",\r\n type=Path,\r\n help=\"Checkpoint path\",\r\n default=\"Qwen/Qwen-7B-Chat\",\r\n )\r\n parser.add_argument(\"-f\", \"--sample-input-file\", type=str, default=None)\r\n parser.add_argument(\r\n \"-o\", \"--sample-output-file\", type=str, default=\"gsm8k_res.jsonl\"\r\n )\r\n parser.add_argument(\"--use-fewshot\", action=\"store_true\")\r\n\r\n args = parser.parse_args()\r\n\r\n if args.sample_input_file is not None:\r\n dataset = load_from_disk(args.sample_input_file) # or:\r\n else:\r\n dataset = load_dataset(\"gsm8k\", \"main\")\r\n\r\n print(\"Loading tokenizer ...\")\r\n tokenizer = AutoTokenizer.from_pretrained(\r\n args.checkpoint_path, trust_remote_code=True, bf16=True, use_flash_attn=True\r\n )\r\n\r\n print(\"Loading model ...\")\r\n model = AutoModelForCausalLM.from_pretrained(\r\n args.checkpoint_path, device_map=\"auto\", trust_remote_code=True\r\n ).eval()\r\n model.generation_config = GenerationConfig.from_pretrained(\r\n args.checkpoint_path, trust_remote_code=True\r\n )\r\n model.generation_config.do_sample = False # use greedy decoding\r\n model.generation_config.repetition_penalty = 1.0 # disable repetition penalty\r\n\r\n test = dataset[\"test\"]\r\n\r\n f_output = open(args.sample_output_file, \"w\", encoding=\"utf-8\")\r\n tot_length = test.num_rows\r\n acc_res = []\r\n for doc in tqdm.tqdm(test):\r\n context = doc_to_text(doc, args.use_fewshot)\r\n completion = generate_sample(model, tokenizer, context)\r\n answer = doc[\"answer\"]\r\n acc = is_correct(completion, answer)\r\n doc[\"completion\"] = completion\r\n doc[\"acc\"] = acc\r\n f_output.write(json.dumps(doc, ensure_ascii=False) + \"\\n\")\r\n f_output.flush()\r\n acc_res.append(acc)\r\n\r\n f_output.close()\r\n print(\"4-shot Acc: \" if args.use_fewshot else \"Zero-shot Acc\", np.mean(acc_res))\r\n" }, { "path": "eval/evaluate_chat_humaneval.py", "content": "import re\r\nimport textwrap\r\nimport argparse\r\nfrom pathlib import Path\r\nimport tqdm\r\nimport jsonlines\r\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\r\nfrom transformers.generation import GenerationConfig\r\n\r\n\"\"\"\r\nGet the HumanEval.jsonl file from [here](https://github.com/openai/human-eval/tree/master/data)\r\n\r\npython eval/evaluate_chat_humaneval.py -f HumanEval.jsonl -o HumanEval_res.jsonl\r\ngit clone https://github.com/openai/human-eval\r\npip install -e human-eval\r\nevaluate_functional_correctness HumanEval_res.jsonl\r\n\"\"\"\r\n\r\nDEVICE = \"cuda:0\"\r\n\r\n\r\ndef extract_code(text, entry_point):\r\n # 正则表达式匹配代码块\r\n code_block_pattern = re.compile(\r\n rf\"```(?:[Pp]ython\\n)?.*?def\\s+{entry_point}.*?:\\n(.*?)\\n```\", re.DOTALL\r\n )\r\n code_block = code_block_pattern.search(text)\r\n if code_block is None:\r\n code_block_pattern = re.compile(\r\n rf\"def\\s+{entry_point}.*?:\\n(.*?)(?:\\n(?!\\n*(?: |\\t))|$)\", re.DOTALL\r\n )\r\n code_block = code_block_pattern.search(text)\r\n if code_block is None:\r\n code_block_pattern = re.compile(\r\n r\"def.*?:\\n(.*?)(?:\\n(?!\\n*(?: |\\t))|$)\", re.DOTALL\r\n )\r\n code_block = code_block_pattern.search(text)\r\n\r\n if code_block is not None:\r\n return code_block.group(1)\r\n\r\n # if no code block is found, assume the LM is simply filling the code\r\n return textwrap.indent(text, \" \" * 4)\r\n\r\n\r\ndef generate_sample(model, tokenizer, question, entry_point):\r\n response, _ = model.chat(\r\n tokenizer,\r\n question,\r\n history=None,\r\n )\r\n print(question)\r\n print(response)\r\n answer = extract_code(response, entry_point)\r\n return answer, response\r\n\r\n\r\nif __name__ == \"__main__\":\r\n parser = argparse.ArgumentParser(description=\"Test HF checkpoint.\")\r\n parser.add_argument(\r\n \"-c\",\r\n \"--checkpoint-path\",\r\n type=Path,\r\n help=\"Checkpoint path\",\r\n default=\"Qwen/Qwen-7B-Chat\",\r\n )\r\n parser.add_argument(\r\n \"-f\",\r\n \"--sample-input-file\",\r\n type=str,\r\n default=None,\r\n help=\"data path to HumanEval.jsonl\",\r\n )\r\n parser.add_argument(\r\n \"-o\", \"--sample-output-file\", type=str, default=\"HumanEval_res.jsonl\"\r\n )\r\n\r\n args = parser.parse_args()\r\n print(\"Loading tokenizer ...\")\r\n tokenizer = AutoTokenizer.from_pretrained(\r\n args.checkpoint_path, trust_remote_code=True\r\n )\r\n\r\n print(\"Loading model ...\")\r\n model = AutoModelForCausalLM.from_pretrained(\r\n args.checkpoint_path,\r\n device_map=\"auto\",\r\n trust_remote_code=True,\r\n bf16=True,\r\n use_flash_attn=True,\r\n ).eval()\r\n model.generation_config = GenerationConfig.from_pretrained(\r\n args.checkpoint_path, trust_remote_code=True\r\n )\r\n model.generation_config.do_sample = False # use greedy decoding\r\n model.generation_config.repetition_penalty = 1.0 # disable repetition penalty\r\n\r\n f_output = jsonlines.Writer(open(args.sample_output_file, \"w\", encoding=\"utf-8\"))\r\n\r\n f = jsonlines.open(args.sample_input_file)\r\n with f_output as output:\r\n for jobj in tqdm.tqdm(f, desc=\"task_idx\"):\r\n # use humanevalpack prompt\r\n signature = re.search(\r\n rf\"def\\s+({jobj['entry_point']}.*?):\\s*\\n\", jobj[\"prompt\"]\r\n ).group(1)\r\n description = \"\\n\".join(\r\n [\r\n line.strip()\r\n for line in re.search(\r\n rf\"(?:\\\"\\\"\\\"|''')(.*?)(?:\\\"\\\"\\\"|''')\", jobj[\"prompt\"], re.DOTALL\r\n )\r\n .group(1)\r\n .split(\"\\n\")\r\n ]\r\n )\r\n prompt = (\r\n f\"Write a Python function `{signature}` to solve the following problem:\\n\"\r\n f\"{description}\\n\"\r\n f\"{jobj['prompt']}\"\r\n )\r\n\r\n task_id = jobj[\"task_id\"]\r\n answer, response = generate_sample(\r\n model, tokenizer, prompt, jobj[\"entry_point\"]\r\n )\r\n gen_jobjs = {\"task_id\": task_id, \"completion\": answer, \"response\": response}\r\n output.write(gen_jobjs)\r\n f_output.close()\r\n" }, { "path": "eval/evaluate_chat_mmlu.py", "content": "import os\r\nimport argparse\r\nimport re\r\nimport torch\r\nimport pandas as pd\r\nfrom tqdm import tqdm\r\nfrom thefuzz import process\r\nfrom transformers.trainer_utils import set_seed\r\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\r\nfrom transformers.generation import GenerationConfig\r\n\r\n'''\r\nwget https://people.eecs.berkeley.edu/~hendrycks/data.tar\r\nmkdir data/mmlu\r\nmv data.tar data/mmlu\r\ncd data/mmlu; tar xf data.tar\r\ncd ../../\r\n\r\npip install thefuzz\r\npython eval/evaluate_chat_mmlu.py -d data/mmlu/data/\r\n'''\r\n\r\ndef load_models_tokenizer(args):\r\n tokenizer = AutoTokenizer.from_pretrained(\r\n args.checkpoint_path, trust_remote_code=True\r\n )\r\n model = AutoModelForCausalLM.from_pretrained(\r\n args.checkpoint_path,\r\n device_map=\"auto\",\r\n trust_remote_code=True,\r\n bf16=True,\r\n use_flash_attn=True,\r\n ).eval()\r\n model.generation_config = GenerationConfig.from_pretrained(\r\n args.checkpoint_path, trust_remote_code=True\r\n )\r\n model.generation_config.do_sample = False # use greedy decoding\r\n model.generation_config.repetition_penalty = 1.0 # disable repetition penalty\r\n return model, tokenizer\r\n\r\n\r\ndef format_example(line):\r\n example = (\r\n \"The following is a multiple-choice question. Please choose the most suitable one among A, B, C and D as the answer to this question.\\n\\n\"\r\n + line[\"question\"]\r\n + \"\\n\"\r\n )\r\n for choice in choices:\r\n example += f'{choice}. {line[f\"{choice}\"]}\\n'\r\n return example\r\n\r\n\r\ndef process_before_extraction(gen, choice_dict):\r\n # replace the choice by letter in the generated sentence\r\n # from longest one to shortest one\r\n for key, val in sorted(choice_dict.items(), key=lambda x: len(x[1]), reverse=True):\r\n pattern = re.compile(re.escape(val.rstrip(\".\")), re.IGNORECASE)\r\n gen = pattern.sub(key, gen)\r\n return gen\r\n\r\n\r\ndef extract_choice(gen, choice_list):\r\n # answer is A | choice is A | choose A\r\n res = re.search(\r\n r\"(?:(?:[Cc]hoose)|(?:(?:[Aa]nswer|[Cc]hoice)(?![^ABCD]{0,20}?(?:n't|not))[^ABCD]{0,10}?\\b(?:|is|:|be))\\b)[^ABCD]{0,20}?\\b(A|B|C|D)\\b\",\r\n gen,\r\n )\r\n\r\n # A is correct | A is right\r\n if res is None:\r\n res = re.search(\r\n r\"\\b(A|B|C|D)\\b(?![^ABCD]{0,8}?(?:n't|not)[^ABCD]{0,5}?(?:correct|right))[^ABCD]{0,10}?\\b(?:correct|right)\\b\",\r\n gen,\r\n )\r\n\r\n # straight answer: A\r\n if res is None:\r\n res = re.search(r\"^(A|B|C|D)(?:\\.|,|:|$)\", gen)\r\n\r\n # simply extract the first appearred letter\r\n if res is None:\r\n res = re.search(r\"(?<![a-zA-Z])(A|B|C|D)(?![a-zA-Z=])\", gen)\r\n\r\n if res is None:\r\n return choices[choice_list.index(process.extractOne(gen, choice_list)[0])]\r\n return res.group(1)\r\n\r\n\r\ndef extract_answer(response, row):\r\n gen = process_before_extraction(\r\n response, {choice: row[choice] for choice in choices}\r\n )\r\n pred = extract_choice(gen, [row[choice] for choice in choices])\r\n return pred\r\n\r\n\r\n@torch.no_grad()\r\ndef eval_subject(\r\n model,\r\n tokenizer,\r\n subject_name,\r\n test_df,\r\n save_result_dir=None,\r\n overwrite=False,\r\n **kwargs\r\n):\r\n result_path = os.path.join(save_result_dir, f\"{subject_name}_result.csv\")\r\n if not overwrite and os.path.exists(result_path):\r\n print(f\"{result_path} existed, skip!\")\r\n score = []\r\n for (_, datarow), (_, resultrow) in zip(\r\n test_df.iterrows(), pd.read_csv(result_path).astype(str).iterrows()\r\n ):\r\n # pred = extract_answer(resultrow['model_response'], datarow)\r\n pred = resultrow[\"model_output\"]\r\n correct = 1 if pred == datarow[\"answer\"] else 0\r\n score.append(correct)\r\n return score\r\n\r\n result = []\r\n score = []\r\n\r\n for _, row in tqdm(test_df.iterrows(), total=len(test_df)):\r\n question = format_example(row)\r\n\r\n response, _ = model.chat(\r\n tokenizer,\r\n question,\r\n history=None,\r\n )\r\n print(question)\r\n print(response)\r\n pred = extract_answer(response, row)\r\n print(pred)\r\n print(\"======================\")\r\n\r\n if \"answer\" in row:\r\n correct = 1 if pred == row[\"answer\"] else 0\r\n score.append(correct)\r\n if args.debug:\r\n print(f'{question} pred: {pred} ref: {row[\"answer\"]}')\r\n result.append(pred)\r\n\r\n if save_result_dir:\r\n test_df[\"model_output\"] = result\r\n test_df[\"model_response\"] = response\r\n if score:\r\n test_df[\"correctness\"] = score\r\n os.makedirs(save_result_dir, exist_ok=True)\r\n test_df.to_csv(\r\n os.path.join(save_result_dir, f\"{subject_name}_result.csv\"),\r\n encoding=\"utf-8\",\r\n index=False,\r\n )\r\n\r\n return score\r\n\r\n\r\ndef cal_mmlu(res):\r\n acc_sum_dict = dict()\r\n acc_norm_sum_dict = dict()\r\n cnt_dict = dict()\r\n acc_sum = 0.0\r\n cnt = 0\r\n\r\n for class_ in TASK_NAME_MAPPING.keys():\r\n acc_sum_dict[class_] = 0.0\r\n acc_norm_sum_dict[class_] = 0.0\r\n cnt_dict[class_] = 0.0\r\n\r\n for tt in TASK_NAME_MAPPING[class_]:\r\n acc_sum += sum(res[tt])\r\n cnt += len(res[tt])\r\n\r\n acc_sum_dict[class_] += sum(res[tt])\r\n cnt_dict[class_] += len(res[tt])\r\n\r\n print(\"\\n\\n\\n\")\r\n for k in TASK_NAME_MAPPING.keys():\r\n if k in cnt_dict:\r\n print(\"%s ACC: %.2f \" % (k, acc_sum_dict[k] * 100 / cnt_dict[k]))\r\n print(\"AVERAGE ACC:%.2f \" % (acc_sum * 100 / cnt))\r\n\r\n\r\ndef main(args):\r\n print(\"loading model weights\")\r\n if args.checkpoint_path is not None:\r\n model, tokenizer = load_models_tokenizer(args)\r\n else:\r\n model, tokenizer = None, None\r\n print(\"model loaded\")\r\n\r\n dev_result = {}\r\n for subject_name in tqdm(SUBJECTS):\r\n # val_file_path = os.path.join(args.eval_data_path, 'val', f'{subject_name}_val.csv')\r\n # dev_file_path = os.path.join(args.eval_data_path, 'dev', f'{subject_name}_dev.csv')\r\n test_file_path = os.path.join(\r\n args.eval_data_path, \"test\", f\"{subject_name}_test.csv\"\r\n )\r\n # val_df = pd.read_csv(val_file_path, names=['question','A','B','C','D','answer'])\r\n # dev_df = pd.read_csv(dev_file_path, names=['question','A','B','C','D','answer'])\r\n test_df = pd.read_csv(\r\n test_file_path, names=[\"question\", \"A\", \"B\", \"C\", \"D\", \"answer\"]\r\n ).astype(str)\r\n\r\n score = eval_subject(\r\n model,\r\n tokenizer,\r\n subject_name,\r\n test_df,\r\n save_result_dir=f\"outs_chat/mmlu_eval_result\",\r\n overwrite=args.overwrite,\r\n )\r\n dev_result[subject_name] = score\r\n cal_mmlu(dev_result)\r\n\r\n\r\nTASK_NAME_MAPPING = {\r\n \"stem\": [\r\n \"abstract_algebra\",\r\n \"anatomy\",\r\n \"astronomy\",\r\n \"college_biology\",\r\n \"college_chemistry\",\r\n \"college_computer_science\",\r\n \"college_mathematics\",\r\n \"college_physics\",\r\n \"computer_security\",\r\n \"conceptual_physics\",\r\n \"electrical_engineering\",\r\n \"elementary_mathematics\",\r\n \"high_school_biology\",\r\n \"high_school_chemistry\",\r\n \"high_school_computer_science\",\r\n \"high_school_mathematics\",\r\n \"high_school_physics\",\r\n \"high_school_statistics\",\r\n \"machine_learning\",\r\n ],\r\n \"Humanities\": [\r\n \"formal_logic\",\r\n \"high_school_european_history\",\r\n \"high_school_us_history\",\r\n \"high_school_world_history\",\r\n \"international_law\",\r\n \"jurisprudence\",\r\n \"logical_fallacies\",\r\n \"moral_disputes\",\r\n \"moral_scenarios\",\r\n \"philosophy\",\r\n \"prehistory\",\r\n \"professional_law\",\r\n \"world_religions\",\r\n ],\r\n \"other\": [\r\n \"business_ethics\",\r\n \"college_medicine\",\r\n \"human_aging\",\r\n \"management\",\r\n \"marketing\",\r\n \"medical_genetics\",\r\n \"miscellaneous\",\r\n \"nutrition\",\r\n \"professional_accounting\",\r\n \"professional_medicine\",\r\n \"virology\",\r\n \"global_facts\",\r\n \"clinical_knowledge\",\r\n ],\r\n \"social\": [\r\n \"econometrics\",\r\n \"high_school_geography\",\r\n \"high_school_government_and_politics\",\r\n \"high_school_macroeconomics\",\r\n \"high_school_microeconomics\",\r\n \"high_school_psychology\",\r\n \"human_sexuality\",\r\n \"professional_psychology\",\r\n \"public_relations\",\r\n \"security_studies\",\r\n \"sociology\",\r\n \"us_foreign_policy\",\r\n ],\r\n}\r\nSUBJECTS = [v for vl in TASK_NAME_MAPPING.values() for v in vl]\r\nchoices = [\"A\", \"B\", \"C\", \"D\"]\r\n\r\nif __name__ == \"__main__\":\r\n parser = argparse.ArgumentParser(description=\"Test HF checkpoint.\")\r\n parser.add_argument(\r\n \"-c\",\r\n \"--checkpoint-path\",\r\n type=str,\r\n help=\"Checkpoint path\",\r\n default=\"Qwen/Qwen-7B-Chat\",\r\n )\r\n parser.add_argument(\"-s\", \"--seed\", type=int, default=1234, help=\"Random seed\")\r\n\r\n # Provide extra arguments required for tasks\r\n group = parser.add_argument_group(title=\"Evaluation options\")\r\n group.add_argument(\"-d\", \"--eval_data_path\", type=str, help=\"Path to eval data\")\r\n group.add_argument(\r\n \"--debug\", action=\"store_true\", default=False, help=\"Print infos.\"\r\n )\r\n group.add_argument(\r\n \"--overwrite\",\r\n action=\"store_true\",\r\n default=False,\r\n help=\"Overwrite existed results\",\r\n )\r\n\r\n args = parser.parse_args()\r\n set_seed(args.seed)\r\n\r\n main(args)\r\n" }, { "path": "eval/evaluate_cmmlu.py", "content": "import os\nimport pandas as pd\nimport numpy as np\nimport argparse\nimport datasets\nimport torch\nfrom collections import defaultdict\n\nfrom typing import List\nfrom tqdm import tqdm\nfrom transformers.trainer_utils import set_seed\n\n\n\"\"\"\nwget https://huggingface.co/datasets/haonan-li/cmmlu/resolve/main/cmmlu_v1_0_1.zip\nmkdir data/cmmlu\nmv cmmlu_v1_0_1.zip data/cmmlu\ncd data/cmmlu; unzip cmmlu_v1_0_1.zip\ncd ../../\npython evaluate_cmmlu.py -d data/cmmlu/\n\"\"\"\n\n\ndef load_models_tokenizer(args):\n from transformers import AutoModelForCausalLM, AutoTokenizer\n from transformers.generation import GenerationConfig\n\n tokenizer = AutoTokenizer.from_pretrained(\n args.checkpoint_path,\n pad_token='<|extra_0|>',\n eos_token='<|endoftext|>',\n padding_side='left',\n trust_remote_code=True\n )\n model = AutoModelForCausalLM.from_pretrained(\n args.checkpoint_path,\n pad_token_id=tokenizer.pad_token_id,\n device_map=\"auto\",\n trust_remote_code=True\n ).eval()\n model.generation_config = GenerationConfig.from_pretrained(\n args.checkpoint_path,\n pad_token_id=tokenizer.pad_token_id,\n trust_remote_code=True\n )\n return model, tokenizer\n\n\ndef format_example(line, include_answer=True):\n example = \"问题：\" + line[\"Question\"]\n for choice in choices:\n example += f'\\n{choice}. {line[f\"{choice}\"]}'\n\n if include_answer:\n example += \"\\n答案：\" + line[\"Answer\"] + \"\\n\\n\"\n else:\n example += \"\\n答案：\"\n return example\n\n\ndef generate_few_shot_prompt(k, subject, dev_df):\n prompt = \"\"\n if k == -1:\n k = dev_df.shape[0]\n for i in range(k):\n prompt += format_example(\n dev_df.iloc[i, :],\n include_answer=True,\n )\n return prompt\n\n\ndef get_logits(tokenizer, model, inputs: List[str]):\n input_ids = tokenizer(inputs, padding='longest')[\"input_ids\"]\n input_ids = torch.tensor(input_ids, device=model.device)\n tokens = {\"input_ids\": input_ids}\n attention_mask = input_ids.ne(tokenizer.pad_token_id)\n\n outputs = model(input_ids, attention_mask=attention_mask)[\"logits\"]\n logits = outputs[:, -1, :]\n log_probs = torch.nn.functional.softmax(logits, dim=-1)\n return log_probs, {\"tokens\": tokens}\n\n\n@torch.no_grad()\ndef eval_subject(\n model,\n tokenizer,\n subject_name,\n test_df,\n k=5,\n dev_df=None,\n few_shot=False,\n save_result_dir=None,\n batch_size=1,\n **kwargs,\n):\n result = []\n score = []\n\n few_shot_prompt = (\n generate_few_shot_prompt(k, subject_name, dev_df) if few_shot else []\n )\n all_probs = {\"prob_A\": [], \"prob_B\": [], \"prob_C\": [], \"prob_D\": []}\n if args.debug:\n print(f\"few_shot_prompt: {few_shot_prompt}\")\n\n choices_ids = torch.tensor(\n tokenizer(\"A\")[\"input_ids\"] + tokenizer(\"B\")[\"input_ids\"] +\n tokenizer(\"C\")[\"input_ids\"] + tokenizer(\"D\")[\"input_ids\"]\n ).unsqueeze(0).to(model.device)\n\n idx_list = list(range(0, len(test_df), batch_size))\n for i in tqdm(idx_list):\n full_prompt_list = []\n answer_list = []\n for row in test_df.iloc[i:i+batch_size].to_dict(orient='records'):\n question = format_example(row, include_answer=False)\n full_prompt = few_shot_prompt + question\n full_prompt_list.append(full_prompt)\n if 'Answer' in row:\n answer_list.append(row['Answer'])\n\n logits, input_info = get_logits(tokenizer, model, full_prompt_list)\n softval = logits.gather(1, choices_ids.expand(logits.size(0), -1)).softmax(1)\n if softval.dtype in {torch.bfloat16, torch.float16}:\n softval = softval.to(dtype=torch.float32)\n probs = softval.detach().cpu().numpy()\n\n for i in range(len(probs)):\n for j, choice in enumerate(choices):\n all_probs[f\"prob_{choice}\"].append(probs[i][j])\n pred = {0: \"A\", 1: \"B\", 2: \"C\", 3: \"D\"}[np.argmax(probs[i])]\n\n if answer_list != []:\n correct = 1 if pred == answer_list[i] else 0\n score.append(correct)\n if args.debug:\n print(f'{question} pred: {pred} ref: {answer_list[i]}')\n result.append(pred)\n\n if score:\n correct_ratio = 100 * sum(score) / len(score)\n if args.debug:\n print(subject_name, correct_ratio)\n else:\n correct_ratio = 0\n if save_result_dir:\n test_df[\"model_output\"] = result\n for i, choice in enumerate(choices):\n test_df[f\"prob_{choice}\"] = all_probs[f\"prob_{choice}\"]\n if score:\n test_df[\"correctness\"] = score\n os.makedirs(save_result_dir, exist_ok=True)\n test_df.to_csv(\n os.path.join(save_result_dir, f\"{subject_name}_result.csv\"),\n encoding=\"utf-8\",\n index=False,\n )\n\n return correct_ratio\n\n\ndef cal_cmmlu(res):\n print(\"\\n\\n\\n\")\n res = {k.split(\"-\")[-1]: float(v) for k, v in res.items()}\n for k, v in TASK_NAME_MAPPING.items():\n avg_acc = np.mean(list(map(lambda x: res[x], v)))\n print(f\"{k} acc: {avg_acc:.2f}\")\n avg_all_acc = np.mean(list(res.values()))\n print(f\"AVERAGE acc: {avg_all_acc:.2f}\")\n\n\nsubcategories = {\n \"agronomy\": [\"other\"],\n \"anatomy\": [\"biology\"],\n \"ancient_chinese\": [\"linguistics\", \"china specific\"],\n \"arts\": [\"arts\"],\n \"astronomy\": [\"physics\"],\n \"business_ethics\": [\"business\"],\n \"chinese_civil_service_exam\": [\"politics\", \"china specific\"],\n \"chinese_driving_rule\": [\"other\", \"china specific\"],\n \"chinese_food_culture\": [\"culture\", \"china specific\"],\n \"chinese_foreign_policy\": [\"politics\", \"china specific\"],\n \"chinese_history\": [\"history\", \"china specific\"],\n \"chinese_literature\": [\"literature\", \"china specific\"],\n \"chinese_teacher_qualification\": [\"education\", \"china specific\"],\n \"college_actuarial_science\": [\"math\"],\n \"college_education\": [\"education\"],\n \"college_engineering_hydrology\": [\"engineering\"],\n \"college_law\": [\"law\"],\n \"college_mathematics\": [\"math\"],\n \"college_medical_statistics\": [\"statistics\"],\n \"clinical_knowledge\": [\"other\"],\n \"college_medicine\": [\"other\"],\n \"computer_science\": [\"computer science\"],\n \"computer_security\": [\"other\"],\n \"conceptual_physics\": [\"physics\"],\n \"construction_project_management\": [\"other\", \"china specific\"],\n \"economics\": [\"economics\"],\n \"education\": [\"education\"],\n \"elementary_chinese\": [\"linguistics\", \"china specific\"],\n \"elementary_commonsense\": [\"other\", \"china specific\"],\n \"elementary_information_and_technology\": [\"other\"],\n \"electrical_engineering\": [\"engineering\"],\n \"elementary_mathematics\": [\"math\"],\n \"ethnology\": [\"culture\", \"china specific\"],\n \"food_science\": [\"other\"],\n \"genetics\": [\"biology\"],\n \"global_facts\": [\"global\"],\n \"high_school_biology\": [\"biology\"],\n \"high_school_chemistry\": [\"chemistry\"],\n \"high_school_geography\": [\"geography\"],\n \"high_school_mathematics\": [\"math\"],\n \"high_school_physics\": [\"physics\"],\n \"high_school_politics\": [\"politics\", \"china specific\"],\n \"human_sexuality\": [\"other\"],\n \"international_law\": [\"law\"],\n \"journalism\": [\"sociology\"],\n \"jurisprudence\": [\"law\"],\n \"legal_and_moral_basis\": [\"other\"],\n \"logical\": [\"philosophy\"],\n \"machine_learning\": [\"computer science\"],\n \"management\": [\"business\"],\n \"marketing\": [\"business\"],\n \"marxist_theory\": [\"philosophy\"],\n \"modern_chinese\": [\"linguistics\", \"china specific\"],\n \"nutrition\": [\"other\"],\n \"philosophy\": [\"philosophy\"],\n \"professional_accounting\": [\"business\"],\n \"professional_law\": [\"law\"],\n \"professional_medicine\": [\"other\"],\n \"professional_psychology\": [\"psychology\"],\n \"public_relations\": [\"politics\"],\n \"security_study\": [\"politics\"],\n \"sociology\": [\"culture\"],\n \"sports_science\": [\"other\"],\n \"traditional_chinese_medicine\": [\"other\", \"china specific\"],\n \"virology\": [\"biology\"],\n \"world_history\": [\"history\"],\n \"world_religions\": [\"global\"],\n}\n\ncategories = {\n \"STEM\": [\n \"physics\",\n \"chemistry\",\n \"biology\",\n \"computer science\",\n \"math\",\n \"engineering\",\n \"statistics\",\n ],\n \"Humanities\": [\"history\", \"philosophy\", \"law\", \"arts\", \"literature\", \"global\"],\n \"Social Science\": [\n \"linguistics\",\n \"business\",\n \"politics\",\n \"culture\",\n \"economics\",\n \"geography\",\n \"psychology\",\n \"education\",\n \"sociology\",\n ],\n \"Other\": [\"other\"],\n \"China specific\": [\"china specific\"],\n}\n\nTASK_NAME_MAPPING = defaultdict(list)\nfor k, v in categories.items():\n for subject, subcat in subcategories.items():\n for c in subcat:\n if c in v:\n TASK_NAME_MAPPING[k].append(subject)\n\n\nchoices = [\"A\", \"B\", \"C\", \"D\"]\n\n\ndef main(args):\n model, tokenizer = load_models_tokenizer(args)\n\n test_result = {}\n for subject_name in tqdm(subcategories.keys()):\n dev_file_path = os.path.join(args.eval_data_path, \"dev\", f\"{subject_name}.csv\")\n test_file_path = os.path.join(\n args.eval_data_path, \"test\", f\"{subject_name}.csv\"\n )\n dev_df = pd.read_csv(dev_file_path)\n test_df = pd.read_csv(test_file_path)\n\n score = eval_subject(\n model,\n tokenizer,\n subject_name,\n dev_df=dev_df,\n test_df=test_df,\n k=5,\n few_shot=True,\n save_result_dir=f\"outs/cmmlu_eval_result\",\n batch_size=args.batch_size\n )\n test_result[subject_name] = score\n cal_cmmlu(test_result)\n\n\nif __name__ == \"__main__\":\n parser = argparse.ArgumentParser(description=\"Test HF checkpoint.\")\n parser.add_argument(\n \"-c\",\n \"--checkpoint-path\",\n type=str,\n help=\"Checkpoint path\",\n default=\"Qwen/Qwen-7B\",\n )\n parser.add_argument(\"-s\", \"--seed\", type=int, default=1234, help=\"Random seed\")\n\n \"\"\"Provide extra arguments required for tasks.\"\"\"\n group = parser.add_argument_group(title=\"Evaluation options\")\n group.add_argument(\n \"-d\", \"--eval_data_path\", type=str, required=True, help=\"Path to eval data\"\n )\n group.add_argument(\n \"--max-seq-len\",\n type=int,\n default=2048,\n help=\"Size of the output generated text.\",\n )\n group.add_argument(\n \"--debug\", action=\"store_true\", default=False, help=\"Print infos.\"\n )\n group.add_argument(\n \"--batch-size\",\n type=int,\n default=1,\n help=\"batch size\",\n )\n\n args = parser.parse_args()\n set_seed(args.seed)\n\n main(args)\n" }, { "path": "eval/evaluate_gsm8k.py", "content": "import re\nimport torch\nimport argparse\nimport jsonlines\nimport numpy as np\nimport datasets\nfrom datasets import load_from_disk, load_dataset\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\nfrom transformers.generation import GenerationConfig\n\n\nANS_RE = re.compile(r\"#### (\\-?[0-9\\.\\,]+)\")\nINVALID_ANS = \"[invalid]\"\n\n\ndef doc_to_text(doc):\n return (\n fewshot_prompt\n + \"\\nQuestion: \"\n + doc[\"question\"]\n + \"\\nLet's think step by step\\n\"\n )\n\n\ndef decode(tokens_list, tokenizer, raw_text_len):\n sents = []\n # print(len(tokens_list))\n for tokens in tokens_list:\n tokens = tokens.cpu().numpy().tolist()\n sent = tokenizer.tokenizer.decode(tokens[raw_text_len:])\n sent = sent.split(\"<|endoftext|>\")[0]\n sent = sent.split(\"\\n\\n\\n\")[0]\n sent = sent.split(\"\\n\\n\")[0]\n sent = sent.split(\"Question:\")[0]\n sents.append(sent)\n return sents\n\n\ndef generate_sample(model, tokenizer, input_txt):\n input_ids = tokenizer.tokenizer.encode(input_txt)\n raw_text_len = len(input_ids)\n context_enc = torch.tensor([input_ids]).to(model.device)\n print(f\"Input text: {input_txt}\\n\")\n outputs = model.generate(context_enc)\n output_text = decode(outputs, tokenizer, raw_text_len)[0]\n print(f\"\\nOutput text: {output_text}\\n\")\n return output_text\n\n\ndef extract_answer_hf(completion):\n match = ANS_RE.search(completion)\n if match:\n match_str = match.group(1).strip()\n match_str = match_str.replace(\",\", \"\")\n return eval(match_str)\n else:\n return INVALID_ANS\n\n\ndef extract_answer(completion):\n try:\n last_number = re.findall(r\"\\d+\", completion)[-1]\n return eval(last_number)\n except:\n return INVALID_ANS\n\n\ndef is_correct(completion, answer):\n gold = extract_answer_hf(answer)\n assert gold != INVALID_ANS, \"No ground truth answer found in the document.\"\n return extract_answer(completion) == gold\n\n\nif __name__ == \"__main__\":\n parser = argparse.ArgumentParser(description=\"Test HF checkpoint.\")\n parser.add_argument(\n \"-c\",\n \"--checkpoint-path\",\n type=str,\n help=\"Checkpoint path\",\n default=\"Qwen/Qwen-7B\",\n )\n parser.add_argument(\"-f\", \"--sample-input-file\", type=str, default=None)\n parser.add_argument(\n \"-o\", \"--sample-output-file\", type=str, default=\"gsm8k_res.jsonl\"\n )\n\n args = parser.parse_args()\n\n fewshot_prompt = open(\"gsm8k_prompt.txt\").read()\n if args.sample_input_file is not None:\n dataset = load_from_disk(args.sample_input_file)\n else:\n config = datasets.DownloadConfig(resume_download=True, max_retries=100)\n dataset = load_dataset(\"gsm8k\", \"main\", download_config=config)\n\n test = dataset[\"test\"]\n\n print(\"Loading tokenizer ...\")\n tokenizer = AutoTokenizer.from_pretrained(\n args.checkpoint_path, trust_remote_code=True\n )\n\n print(\"Loading model ...\")\n model = AutoModelForCausalLM.from_pretrained(\n args.checkpoint_path, device_map=\"auto\", trust_remote_code=True\n ).eval()\n model.generation_config = GenerationConfig.from_pretrained(\n args.checkpoint_path, trust_remote_code=True\n )\n model.generation_config.do_sample = False\n\n f_output = jsonlines.Writer(open(args.sample_output_file, \"w\", encoding=\"utf-8\"))\n tot_length = test.num_rows\n acc_res = []\n for doc in test:\n context = doc_to_text(doc)\n completion = generate_sample(model, tokenizer, context)\n answer = doc[\"answer\"]\n acc = is_correct(completion, answer)\n doc[\"completion\"] = completion\n doc[\"acc\"] = acc\n f_output.write(doc)\n acc_res.append(acc)\n\n f_output.close()\n print(\"Acc: \", np.mean(acc_res))\n" }, { "path": "eval/evaluate_humaneval.py", "content": "import argparse\nimport tqdm\nimport torch\nimport jsonlines\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\nfrom transformers.generation import GenerationConfig\n\n\"\"\"\ngit clone https://github.com/openai/human-eval\n$ pip install -e human-eval\nevaluate_functional_correctness sample-output-file\n\"\"\"\n\n\ndef decode(tokens_list, tokenizer, raw_text_len):\n sents = []\n # print(len(tokens_list))\n for tokens in tokens_list:\n tokens = tokens.cpu().numpy().tolist()\n sent = tokenizer.tokenizer.decode(tokens[raw_text_len:])\n sent = sent.split(\"<|endoftext|>\")[0]\n sent = sent.split(\"\\n\\n\\n\")[0]\n sent = sent.split(\"\\n\\n\")[0]\n sent = sent.split(\"def \")[0]\n sents.append(sent)\n return sents\n\n\ndef generate_sample(model, tokenizer, input_txt):\n input_ids = tokenizer.tokenizer.encode(input_txt)\n raw_text_len = len(input_ids)\n context_enc = torch.tensor([input_ids]).to(model.device)\n print(f\"Input text: {input_txt}\\n\")\n outputs = model.generate(context_enc)\n output_text = decode(outputs, tokenizer, raw_text_len)[0]\n print(f\"\\nOutput text: \\n{output_text}\\n\")\n return output_text\n\n\nif __name__ == \"__main__\":\n parser = argparse.ArgumentParser(description=\"Test HF checkpoint.\")\n parser.add_argument(\n \"-c\",\n \"--checkpoint-path\",\n type=str,\n help=\"Checkpoint path\",\n default=\"Qwen/Qwen-7B\",\n )\n parser.add_argument(\n \"-f\",\n \"--sample-input-file\",\n type=str,\n default=None,\n help=\"data path to HumanEval.jsonl\",\n )\n parser.add_argument(\n \"-o\", \"--sample-output-file\", type=str, default=\"HumanEval_res.jsonl\"\n )\n\n args = parser.parse_args()\n print(\"Loading tokenizer ...\")\n tokenizer = AutoTokenizer.from_pretrained(\n args.checkpoint_path, trust_remote_code=True\n )\n\n print(\"Loading model ...\")\n model = AutoModelForCausalLM.from_pretrained(\n args.checkpoint_path, device_map=\"auto\", trust_remote_code=True\n ).eval()\n model.generation_config = GenerationConfig.from_pretrained(\n args.checkpoint_path, trust_remote_code=True\n )\n model.generation_config.do_sample = False\n\n f_output = jsonlines.Writer(open(args.sample_output_file, \"w\", encoding=\"utf-8\"))\n\n f = jsonlines.open(args.sample_input_file)\n with f_output as output:\n for jobj in tqdm.tqdm(f, desc=\"task_idx\"):\n prompt = jobj[\"prompt\"]\n task_id = jobj[\"task_id\"]\n gen_sents = generate_sample(model, tokenizer, prompt)\n gen_jobjs = {\"task_id\": task_id, \"completion\": gen_sents}\n output.write(gen_jobjs)\n f_output.close()\n" }, { "path": "eval/evaluate_mmlu.py", "content": "import os\nfrom typing import List\nimport pandas as pd\nimport numpy as np\nimport argparse\nimport torch\nfrom tqdm import tqdm\nfrom transformers.trainer_utils import set_seed\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\nfrom transformers.generation import GenerationConfig\n\n\"\"\"\nwget https://people.eecs.berkeley.edu/~hendrycks/data.tar\nmkdir data/mmlu\nmv data.tar data/mmlu\ncd data/mmlu; tar xf data.tar\ncd ../../\npython eval/evaluate_mmlu.py -d data/mmlu/data/\n\"\"\"\n\n\ndef load_models_tokenizer(args):\n tokenizer = AutoTokenizer.from_pretrained(\n args.checkpoint_path,\n pad_token='<|extra_0|>',\n eos_token='<|endoftext|>',\n padding_side='left',\n trust_remote_code=True\n )\n model = AutoModelForCausalLM.from_pretrained(\n args.checkpoint_path,\n pad_token_id=tokenizer.pad_token_id,\n device_map=\"auto\",\n trust_remote_code=True\n ).eval()\n model.generation_config = GenerationConfig.from_pretrained(\n args.checkpoint_path,\n pad_token_id=tokenizer.pad_token_id,\n trust_remote_code=True\n )\n return model, tokenizer\n\n\ndef format_example(line, include_answer=True):\n example = \"Question: \" + line[\"question\"]\n for choice in choices:\n example += f'\\n{choice}. {line[f\"{choice}\"]}'\n\n if include_answer:\n example += \"\\nAnswer: \" + line[\"answer\"] + \"\\n\\n\"\n else:\n example += \"\\nAnswer:\"\n return example\n\n\ndef generate_few_shot_prompt(k, subject, dev_df):\n def format_subject(subject):\n l = subject.split(\"_\")\n s = \"\"\n for entry in l:\n s += \" \" + entry\n return s.strip()\n\n prompt = \"The following are multiple choice questions (with answers) about {}.\\n\\n\".format(\n format_subject(subject)\n )\n\n if k == -1:\n k = dev_df.shape[0]\n for i in range(k):\n prompt += format_example(\n dev_df.iloc[i, :],\n include_answer=True,\n )\n return prompt\n\n\ndef get_logits(tokenizer, model, inputs: List[str]):\n input_ids = tokenizer(inputs, padding='longest')[\"input_ids\"]\n input_ids = torch.tensor(input_ids, device=model.device)\n\n if input_ids.shape[1] > args.max_seq_len:\n input_ids = input_ids[:, input_ids.shape[1] - args.max_seq_len + 1 :]\n tokens = {\"input_ids\": input_ids}\n attention_mask = input_ids.ne(tokenizer.pad_token_id)\n\n outputs = model(input_ids, attention_mask=attention_mask)[\"logits\"]\n logits = outputs[:, -1, :]\n log_probs = torch.nn.functional.softmax(logits, dim=-1)\n return log_probs, {\"tokens\": tokens}\n\n\n@torch.no_grad()\ndef eval_subject(\n model,\n tokenizer,\n subject_name,\n test_df,\n k=5,\n dev_df=None,\n few_shot=False,\n save_result_dir=None,\n batch_size=1,\n **kwargs,\n):\n result = []\n score = []\n\n few_shot_prompt = (\n generate_few_shot_prompt(k, subject_name, dev_df) if few_shot else []\n )\n all_probs = {\"prob_A\": [], \"prob_B\": [], \"prob_C\": [], \"prob_D\": []}\n if args.debug:\n print(f\"few_shot_prompt: {few_shot_prompt}\")\n\n choices_ids = torch.tensor(\n tokenizer(\" A\")[\"input_ids\"] + tokenizer(\" B\")[\"input_ids\"] +\n tokenizer(\" C\")[\"input_ids\"] + tokenizer(\" D\")[\"input_ids\"]\n ).unsqueeze(0).to(model.device)\n\n idx_list = list(range(0, len(test_df), batch_size))\n for i in tqdm(idx_list):\n full_prompt_list = []\n answer_list = []\n for row in test_df.iloc[i:i+batch_size].to_dict(orient='records'):\n question = format_example(row, include_answer=False)\n full_prompt = few_shot_prompt + question\n full_prompt_list.append(full_prompt)\n if 'answer' in row:\n answer_list.append(row['answer'])\n\n logits, input_info = get_logits(tokenizer, model, full_prompt_list)\n softval = logits.gather(1, choices_ids.expand(logits.size(0), -1)).softmax(1)\n if softval.dtype in {torch.bfloat16, torch.float16}:\n softval = softval.to(dtype=torch.float32)\n probs = softval.detach().cpu().numpy()\n\n for i in range(len(probs)):\n for j, choice in enumerate(choices):\n all_probs[f\"prob_{choice}\"].append(probs[i][j])\n pred = {0: \"A\", 1: \"B\", 2: \"C\", 3: \"D\"}[np.argmax(probs[i])]\n\n if answer_list != []:\n correct = 1 if pred == answer_list[i] else 0\n score.append(correct)\n if args.debug:\n print(f'{question} pred: {pred} ref: {answer_list[i]}')\n result.append(pred)\n\n if save_result_dir:\n test_df[\"model_output\"] = result\n for i, choice in enumerate(choices):\n test_df[f\"prob_{choice}\"] = all_probs[f\"prob_{choice}\"]\n if score:\n test_df[\"correctness\"] = score\n os.makedirs(save_result_dir, exist_ok=True)\n test_df.to_csv(\n os.path.join(save_result_dir, f\"{subject_name}_result.csv\"),\n encoding=\"utf-8\",\n index=False,\n )\n\n return score\n\n\ndef cal_mmlu(res):\n acc_sum_dict = dict()\n acc_norm_sum_dict = dict()\n cnt_dict = dict()\n acc_sum = 0.0\n cnt = 0\n hard_cnt = 0\n hard_acc_sum = 0.0\n\n for class_ in TASK_NAME_MAPPING.keys():\n acc_sum_dict[class_] = 0.0\n acc_norm_sum_dict[class_] = 0.0\n cnt_dict[class_] = 0.0\n\n for tt in TASK_NAME_MAPPING[class_]:\n acc_sum += sum(res[tt])\n cnt += len(res[tt])\n\n acc_sum_dict[class_] += sum(res[tt])\n cnt_dict[class_] += len(res[tt])\n\n print(\"\\n\\n\\n\", \"total cnt:\", cnt, \"\\n\")\n for k in TASK_NAME_MAPPING.keys():\n if k in cnt_dict:\n print(\"%s ACC: %.2f \" % (k, acc_sum_dict[k] / cnt_dict[k] * 100))\n print(\"AVERAGE ACC:%.2f \" % (acc_sum / cnt * 100))\n\n\ndef main(args):\n model, tokenizer = load_models_tokenizer(args)\n\n dev_result = {}\n for subject_name in tqdm(SUBJECTS):\n # val_file_path = os.path.join(args.eval_data_path, 'val', f'{subject_name}_val.csv')\n dev_file_path = os.path.join(\n args.eval_data_path, \"dev\", f\"{subject_name}_dev.csv\"\n )\n test_file_path = os.path.join(\n args.eval_data_path, \"test\", f\"{subject_name}_test.csv\"\n )\n # val_df = pd.read_csv(val_file_path, names=['question','A','B','C','D','answer'])\n dev_df = pd.read_csv(\n dev_file_path, names=[\"question\", \"A\", \"B\", \"C\", \"D\", \"answer\"]\n )\n test_df = pd.read_csv(\n test_file_path, names=[\"question\", \"A\", \"B\", \"C\", \"D\", \"answer\"]\n )\n\n score = eval_subject(\n model,\n tokenizer,\n subject_name,\n test_df,\n dev_df=dev_df,\n k=5,\n few_shot=True,\n save_result_dir=f\"outs/mmlu_eval_result\",\n batch_size=args.batch_size\n )\n dev_result[subject_name] = score\n cal_mmlu(dev_result)\n\n\nTASK_NAME_MAPPING = {\n \"stem\": [\n \"abstract_algebra\",\n \"anatomy\",\n \"astronomy\",\n \"college_biology\",\n \"college_chemistry\",\n \"college_computer_science\",\n \"college_mathematics\",\n \"college_physics\",\n \"computer_security\",\n \"conceptual_physics\",\n \"electrical_engineering\",\n \"elementary_mathematics\",\n \"high_school_biology\",\n \"high_school_chemistry\",\n \"high_school_computer_science\",\n \"high_school_mathematics\",\n \"high_school_physics\",\n \"high_school_statistics\",\n \"machine_learning\",\n ],\n \"Humanities\": [\n \"formal_logic\",\n \"high_school_european_history\",\n \"high_school_us_history\",\n \"high_school_world_history\",\n \"international_law\",\n \"jurisprudence\",\n \"logical_fallacies\",\n \"moral_disputes\",\n \"moral_scenarios\",\n \"philosophy\",\n \"prehistory\",\n \"professional_law\",\n \"world_religions\",\n ],\n \"other\": [\n \"business_ethics\",\n \"college_medicine\",\n \"human_aging\",\n \"management\",\n \"marketing\",\n \"medical_genetics\",\n \"miscellaneous\",\n \"nutrition\",\n \"professional_accounting\",\n \"professional_medicine\",\n \"virology\",\n \"global_facts\",\n \"clinical_knowledge\",\n ],\n \"social\": [\n \"econometrics\",\n \"high_school_geography\",\n \"high_school_government_and_politics\",\n \"high_school_macroeconomics\",\n \"high_school_microeconomics\",\n \"high_school_psychology\",\n \"human_sexuality\",\n \"professional_psychology\",\n \"public_relations\",\n \"security_studies\",\n \"sociology\",\n \"us_foreign_policy\",\n ],\n}\nSUBJECTS = [v for vl in TASK_NAME_MAPPING.values() for v in vl]\nchoices = [\"A\", \"B\", \"C\", \"D\"]\n\nif __name__ == \"__main__\":\n parser = argparse.ArgumentParser(description=\"Test HF checkpoint.\")\n parser.add_argument(\n \"-c\",\n \"--checkpoint-path\",\n type=str,\n help=\"Checkpoint path\",\n default=\"Qwen/Qwen-7B\",\n )\n parser.add_argument(\"-s\", \"--seed\", type=int, default=1234, help=\"Random seed\")\n parser.add_argument(\"--gpu\", type=int, default=0, help=\"gpu id\")\n\n \"\"\"Provide extra arguments required for tasks.\"\"\"\n group = parser.add_argument_group(title=\"Evaluation options\")\n group.add_argument(\"-d\", \"--eval_data_path\", type=str, help=\"Path to eval data\")\n group.add_argument(\n \"--max-seq-len\",\n type=int,\n default=2048,\n help=\"Size of the output generated text.\",\n )\n group.add_argument(\n \"--debug\", action=\"store_true\", default=False, help=\"Print infos.\"\n )\n group.add_argument(\n \"--batch-size\",\n type=int,\n default=1,\n help=\"batch size\",\n )\n\n args = parser.parse_args()\n set_seed(args.seed)\n\n main(args)\n" }, { "path": "eval/evaluate_plugin.py", "content": "import argparse\nimport json\nimport os\nimport pprint\n\nimport json5\nimport jsonlines\nfrom rouge_score import rouge_scorer\nfrom tqdm import tqdm\nfrom transformers import Agent, AutoModelForCausalLM, AutoTokenizer\nfrom transformers.generation import GenerationConfig\nfrom transformers.tools.evaluate_agent import evaluate_agent\nfrom transformers.trainer_utils import set_seed\n\ndata_root_path = os.path.join(os.path.dirname(os.path.abspath(__file__)), \"data\")\n\n\ndef is_callable(response, golden):\n return response[\"action\"].strip().lower() == golden[\"action\"].strip().lower()\n\n\ndef process_res(response):\n # parse response\n response += \"\\n\" # fix not-find bug\n thought = response[: response.find(\"Action:\")].strip()\n action = response[\n response.find(\"Action:\") + len(\"Action:\") : response.find(\"Action Input:\")\n ].strip()\n action_input = response[\n response.find(\"Action Input:\")\n + len(\"Action Input:\") : response.find(\"Observation:\")\n ].strip()\n # TODO: This parsing result is incorrect if the response contains multiple Actions. To be fixed in the future.\n observation = response[\n response.find(\"Observation:\") + len(\"Observation:\") : response.rfind(\"Thought:\")\n ].strip()\n thought_last = response[\n response.rfind(\"Thought:\") + len(\"Thought:\") : response.find(\"Final Answer:\")\n ].strip()\n final_answer = response[\n response.find(\"Final Answer:\") + len(\"Final Answer:\") :\n ].strip()\n try:\n action_input = json.dumps(\n json5.loads(action_input), ensure_ascii=False, sort_keys=True\n )\n except:\n # print(\"JSON Load Error:\", action_input)\n action_input = \"\"\n res_dict = {\n \"thought\": thought,\n \"action\": action,\n \"action_input\": action_input,\n \"observation\": observation,\n \"thought_last\": thought_last,\n \"final_answer\": final_answer,\n }\n return res_dict\n\n\nclass _DummyTokenizer:\n def tokenize(self, text: str):\n return text.split()\n\n\ndef _get_tokenized_string(tokenizer, text_list):\n token_ids_list, tokenized_string_list = [], []\n for text in text_list:\n assert tokenizer is not None\n token_ids = tokenizer.encode(text)\n tokens_bytes = tokenizer.convert_ids_to_tokens(token_ids)\n tokens = [token.decode(\"utf-8\", errors=\"replace\") for token in tokens_bytes]\n tokenized_string = \" \".join(tokens)\n token_ids_list.append(token_ids)\n tokenized_string_list.append(tokenized_string)\n return token_ids_list, tokenized_string_list\n\n\ndef eval_action(job):\n response = job[\"gen\"][0]\n golden = job[\"response\"]\n\n if \"\\nAction: \" in response:\n response, golden = process_res(response), process_res(golden)\n if is_callable(response, golden):\n return True\n return False\n\n\ndef eval_action_input(job, tokenizer):\n response = job[\"gen\"][0]\n golden = job[\"response\"]\n response, golden = process_res(response), process_res(golden)\n query = job[\"prompt\"]\n\n job = {}\n job[\"prompt\"] = query\n job[\"gen\"] = response[\"action_input\"]\n job[\"response\"] = golden[\"action_input\"]\n\n job[\"_gen_tok\"], job[\"_gen_tok_str\"] = _get_tokenized_string(\n tokenizer, [response[\"action_input\"]]\n )\n job[\"_reference_tok\"], job[\"_reference_tok_str\"] = _get_tokenized_string(\n tokenizer, [golden[\"action_input\"]]\n )\n\n scorer = rouge_scorer.RougeScorer(\n [\"rouge1\", \"rouge2\", \"rougeL\"], tokenizer=_DummyTokenizer()\n )\n score = scorer.score(job[\"_reference_tok_str\"][0], job[\"_gen_tok_str\"][0])\n\n rouge = score[\"rougeL\"].fmeasure\n\n return rouge\n\n\nclass QWenAgent(Agent):\n \"\"\"\n Agent that uses QWen model and tokenizer to generate code.\n\n Example:\n\n ```py\n agent = QWenAgent()\n agent.run(\"Draw me a picture of rivers and lakes.\")\n ```\n \"\"\"\n\n def __init__(\n self,\n chat_prompt_template=None,\n run_prompt_template=None,\n additional_tools=None,\n tokenizer=None,\n model=None,\n ):\n if tokenizer and model:\n self.tokenizer = tokenizer\n self.model = model\n else:\n checkpoint = \"Qwen/Qwen-7B-Chat\"\n self.tokenizer = AutoTokenizer.from_pretrained(\n checkpoint, trust_remote_code=True\n )\n self.model = (\n AutoModelForCausalLM.from_pretrained(\n checkpoint, device_map=\"auto\", trust_remote_code=True\n )\n .cuda()\n .eval()\n )\n self.model.generation_config = GenerationConfig.from_pretrained(\n checkpoint, trust_remote_code=True\n ) # 可指定不同的生成长度、top_p等相关超参\n self.model.generation_config.do_sample = False # greedy\n\n super().__init__(\n chat_prompt_template=chat_prompt_template,\n run_prompt_template=run_prompt_template,\n additional_tools=additional_tools,\n )\n\n def generate_one(self, prompt, stop):\n # \"Human:\" 和 \"Assistant:\" 曾为通义千问的特殊保留字，需要替换为 \"_HUMAN_:\" 和 \"_ASSISTANT_:\"。这一问题将在未来版本修复。\n prompt = prompt.replace(\"Human:\", \"_HUMAN_:\").replace(\n \"Assistant:\", \"_ASSISTANT_:\"\n )\n stop = [\n item.replace(\"Human:\", \"_HUMAN_:\").replace(\"Assistant:\", \"_ASSISTANT_:\")\n for item in stop\n ]\n\n result, _ = self.model.chat(self.tokenizer, prompt, history=None)\n for stop_seq in stop:\n if result.endswith(stop_seq):\n result = result[: -len(stop_seq)]\n\n result = result.replace(\"_HUMAN_:\", \"Human:\").replace(\n \"_ASSISTANT_:\", \"Assistant:\"\n )\n return result\n\n\ndef load_models_tokenizer(args):\n tokenizer = AutoTokenizer.from_pretrained(\n args.checkpoint_path, trust_remote_code=True\n )\n model = AutoModelForCausalLM.from_pretrained(\n args.checkpoint_path,\n device_map=\"auto\",\n trust_remote_code=True,\n bf16=True,\n use_flash_attn=True,\n ).eval()\n model.generation_config = GenerationConfig.from_pretrained(\n args.checkpoint_path, trust_remote_code=True\n )\n model.generation_config.do_sample = False # use greedy decoding\n return model, tokenizer\n\n\ndef load_jobs(filename):\n jobs = []\n with jsonlines.open(os.path.join(data_root_path, filename), mode=\"r\") as reader:\n for job in reader:\n jobs.append(job)\n return jobs\n\n\ndef react_inference(filename, model, tokenizer):\n filename_cache = filename + \".cache\"\n if os.path.exists(os.path.join(data_root_path, filename_cache)):\n jobs = load_jobs(filename=filename_cache)\n print(\"Loaded from\", filename_cache)\n else:\n with open(os.path.join(data_root_path, filename_cache), \"w\") as f:\n jobs = load_jobs(filename=filename)\n print(\"Inference:\", filename)\n for job in tqdm(jobs):\n response, history = model.chat(tokenizer, job[\"prompt\"], history=None)\n job[\"gen\"] = [response]\n f.writelines(json.dumps(job, ensure_ascii=False) + \"\\n\")\n print(filename_cache, \"is saved.\")\n return jobs\n\n\ndef main(args):\n print(\"loading model weights\")\n if args.checkpoint_path is not None:\n model, tokenizer = load_models_tokenizer(args)\n else:\n model, tokenizer = None, None\n print(\"model loaded\")\n\n result = {}\n # eval react positive\n if args.eval_react_positive:\n print(\"eval react positive ...\")\n acc_count = 0\n rouge_mean = 0\n jobs = react_inference(\n filename=args.eval_react_positive_filename, model=model, tokenizer=tokenizer\n )\n for job in jobs:\n if eval_action(job):\n acc_count += 1\n rouge = eval_action_input(job, tokenizer)\n rouge_mean += rouge / len(jobs)\n\n scores = {\n \"action_right_rate\": acc_count / len(jobs),\n \"action_input_rouge\": rouge_mean,\n }\n\n result.update({\"react_positive\": scores})\n\n # eval react negative\n if args.eval_react_negative:\n print(\"eval react negative ...\")\n bad_count = 0\n jobs = react_inference(\n filename=args.eval_react_negative_filename, model=model, tokenizer=tokenizer\n )\n for job in jobs:\n if \"\\nAction: \" in job[\"gen\"][0]:\n bad_count += 1\n scores = {\"bad_rate\": bad_count / len(jobs)}\n result.update({\"react_negative\": scores})\n\n # eval hfagent\n if args.eval_hfagent:\n print(\"eval hfagent ...\")\n agent = QWenAgent(model=model, tokenizer=tokenizer)\n scores = evaluate_agent(agent, verbose=False, return_errors=False)\n result.update({\"hfagent\": scores})\n\n pp = pprint.PrettyPrinter(indent=4)\n pp.pprint(result)\n\n\nif __name__ == \"__main__\":\n parser = argparse.ArgumentParser(description=\"Test HF checkpoint.\")\n parser.add_argument(\n \"-c\",\n \"--checkpoint-path\",\n type=str,\n help=\"Checkpoint path\",\n default=\"Qwen/Qwen-7B-Chat\",\n )\n parser.add_argument(\"-s\", \"--seed\", type=int, default=1234, help=\"Random seed\")\n \"\"\"Provide extra arguments required for tasks.\"\"\"\n group = parser.add_argument_group(title=\"Evaluation options\")\n group.add_argument(\n \"--eval-react-positive\",\n action=\"store_true\",\n default=False,\n help=\"Eval react positive.\",\n )\n group.add_argument(\n \"--eval-react-positive-filename\",\n type=str,\n default=\"exam_plugin_v20231206_react_positive.jsonl\",\n help=\"Eval react positive filename.\",\n )\n group.add_argument(\n \"--eval-react-negative\",\n action=\"store_true\",\n default=False,\n help=\"Eval react negative.\",\n )\n group.add_argument(\n \"--eval-react-negative-filename\",\n type=str,\n default=\"exam_plugin_v20231206_react_negative.jsonl\",\n help=\"Eval react negative filename.\",\n )\n group.add_argument(\n \"--eval-hfagent\", action=\"store_true\", default=False, help=\"Eval hfagent.\"\n )\n\n args = parser.parse_args()\n set_seed(args.seed)\n\n main(args)\n" }, { "path": "eval/gsm8k_prompt.txt", "content": "Question: In 2004, there were 60 kids at a cookout. In 2005, half the number of kids came to the cookout as compared to 2004. In 2006, 2/3 as many kids came to the cookout as in 2005. How many kids came to the cookout in 2006?\nLet's think step by step\nIn 2005, 60/2=30 kids came to the cookout.\nIn 2006, 30/3*2=20 kids came to the cookout.\nThe answer is 20\n\nQuestion: Zilla spent 7% of her monthly earnings on rent, half of it on her other monthly expenses, and put the rest in her savings. If she spent $133 on her rent, how much does she deposit into her savings account in a month?\nLet's think step by step\nSince $133 is equal to 7% of her earnings, then 1% is equal to $133/7 = $19.\nThe total monthly earning of Zilla is represented by 100%, so $19 x 100 = $1900 is her monthly earnings.\nSo, $1900/2 = $950 is spent on her other monthly expenses.\nThe total amount spent on the rent and other monthly expenses is $133 + $950 = $1083.\nHence, she saves $1900 - $1083 = $817 per month.\nThe answer is 817\n\nQuestion: If Buzz bought a pizza with 78 slices at a restaurant and then decided to share it with the waiter in the ratio of 5:8, with Buzz's ratio being 5, what's twenty less the number of slices of pizza that the waiter ate?\nLet's think step by step\nThe total ratio representing the slices of pizza that Buzz bought is 5+8=13\nIf he shared the slices of pizza with the waiter, the waiter received a fraction of 8/13 of the total number of slices, which totals 8/13 * 78 = 48 slices\nTwenty less the number of slices of pizza that the waiter ate is 48-20 = 28\nThe answer is 28\n\nQuestion: Jame gets a raise to $20 per hour and works 40 hours a week. His old job was $16 an hour for 25 hours per week. How much more money does he make per year in his new job than the old job if he works 52 weeks a year?\nLet's think step by step\nHe makes 20*40=$800 per week\nHe used to make 16*25=$400 per week\nSo his raise was 800-400=$400 per week\nSo he makes 400*52=$20,800 per year more\nThe answer is 20800\n\nQuestion: Mr. Gardner bakes 20 cookies, 25 cupcakes, and 35 brownies for his second-grade class of 20 students. If he wants to give each student an equal amount of sweet treats, how many sweet treats will each student receive?\nLet's think step by step\nMr. Gardner bakes a total of 20 + 25 + 35 = 80 sweet treats\nEach student will receive 80 / 20 = 4 sweet treats\nThe answer is 4\n\nQuestion: A used car lot has 24 cars and motorcycles (in total) for sale. A third of the vehicles are motorcycles, and a quarter of the cars have a spare tire included. How many tires are on the used car lot’s vehicles in all?\nLet's think step by step\nThe used car lot has 24 / 3 = 8 motorcycles with 2 tires each.\nThe lot has 24 - 8 = 16 cars for sale\nThere are 16 / 4 = 4 cars with a spare tire with 5 tires each.\nThe lot has 16 - 4 = 12 cars with 4 tires each.\nThus, the used car lot’s vehicles have 8 * 2 + 4 * 5 + 12 * 4 = 16 + 20 + 48 = 84 tires in all.\nThe answer is 84\n\nQuestion: Norma takes her clothes to the laundry. She leaves 9 T-shirts and twice as many sweaters as T-shirts in the washer. When she returns she finds 3 sweaters and triple the number of T-shirts. How many items are missing?\nLet's think step by step\nNorma left 9 T-shirts And twice as many sweaters, she took 9 * 2= 18 sweaters\nAdding the T-shirts and sweaters, Norma left 9 + 18 = 27 clothes\nWhen she came back, she found 3 sweaters And triple the number of T-shirts, she found 3 * 3 = 9 T-shirts\nAdding the T-shirts and sweaters, Norma found 3 + 9 = 12 clothes\nSubtracting the clothes she left from the clothes she found, 27 - 12 = 15 clothes are missing\nThe answer is 15\n\nQuestion: Adam has an orchard. Every day for 30 days he picks 4 apples from his orchard. After a month, Adam has collected all the remaining apples, which were 230. How many apples in total has Adam collected from his orchard?\nLet's think step by step\nDuring 30 days Adam picked 4 * 30 = 120 apples.\nSo in total with all the remaining apples, he picked 120 + 230 = 350 apples from his orchard.\nThe answer is 350\n" }, { "path": "examples/add_merges.py", "content": "import argparse\r\nimport base64\r\nimport collections\r\nimport logging\r\nimport unicodedata\r\nfrom pathlib import Path\r\n\r\nimport regex as re\r\nfrom tqdm.contrib.logging import tqdm_logging_redirect\r\n\r\nPAT_STR = r\"\"\"(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?\\p{L}+|\\p{N}| ?[^\\s\\p{L}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+\"\"\"\r\n\r\n\r\nlogger = logging.getLogger(__name__)\r\n\r\nlogging.basicConfig(\r\n level=logging.DEBUG, format=\"[%(asctime)s] %(levelname)s - %(message)s\"\r\n)\r\n\r\n\r\ndef load_tiktoken_bpe(tiktoken_bpe_file: str) -> \"dict[bytes, int]\":\r\n contents = open(tiktoken_bpe_file, \"rb\").read()\r\n return {\r\n base64.b64decode(token): int(rank)\r\n for token, rank in (line.split() for line in contents.splitlines() if line)\r\n }\r\n\r\n\r\ndef dump_tiktoken_bpe(bpe_ranks: \"dict[bytes, int]\", tiktoken_bpe_file: str) -> None:\r\n with open(tiktoken_bpe_file, \"wb\") as f:\r\n for token, rank in sorted(bpe_ranks.items(), key=lambda x: x[1]):\r\n f.write(base64.b64encode(token) + b\" \" + str(rank).encode() + b\"\\n\")\r\n\r\n\r\ndef bytes_to_pieces(the_bytes: bytes) -> \"tuple[bytes]\":\r\n return tuple(bytes([byte]) for byte in the_bytes)\r\n\r\n\r\ndef get_pairs(pieces: \"tuple[bytes]\") -> \"set[tuple[bytes, bytes]]\":\r\n return set(zip(pieces[:-1], pieces[1:]))\r\n\r\n\r\ndef get_stats(\r\n vocab: \"dict[tuple[bytes, ...], int]\",\r\n) -> \"dict[tuple[bytes, bytes], int]\":\r\n pairs = collections.defaultdict(int)\r\n for word, freq in vocab.items():\r\n for i in range(len(word) - 1):\r\n pairs[(word[i], word[i + 1])] += freq\r\n return pairs\r\n\r\n\r\ndef merge_vocab(\r\n pair: \"tuple[bytes, bytes]\", vocab: \"dict[tuple[bytes, ...], int]\"\r\n) -> \"dict[tuple[bytes, ...], int]\":\r\n return {apply_bp(pieces, pair): freq for pieces, freq in vocab.items()}\r\n\r\n\r\ndef apply_bp(\r\n pieces: \"tuple[bytes, ...]\", pair: \"tuple[bytes, bytes]\"\r\n) -> \"tuple[bytes, ...]\":\r\n new_pieces = []\r\n first, second = pair\r\n i = 0\r\n while i < len(pieces):\r\n try:\r\n j = pieces.index(first, i)\r\n new_pieces.extend(pieces[i:j])\r\n i = j\r\n except:\r\n new_pieces.extend(pieces[i:])\r\n break\r\n\r\n if pieces[i] == first and i < len(pieces) - 1 and pieces[i + 1] == second:\r\n new_pieces.append(first + second)\r\n i += 2\r\n else:\r\n new_pieces.append(pieces[i])\r\n i += 1\r\n\r\n return tuple(new_pieces)\r\n\r\n\r\ndef bpe(word: bytes, merges: \"dict[bytes,int]\") -> \"tuple[bytes, ...]\":\r\n pieces = bytes_to_pieces(word)\r\n while len(pieces) > 1:\r\n pairs = get_pairs(pieces)\r\n pair = min(pairs, key=lambda pair: merges.get(pair[0] + pair[1], float(\"inf\")))\r\n\r\n if pair[0] + pair[1] not in merges:\r\n break\r\n pieces = apply_bp(pieces, pair)\r\n # logger.debug(f\"{[(p, p.decode('utf8', errors='replace')) for p in pieces]} {pair} {pieces}\")\r\n return pieces\r\n\r\n\r\ndef best_pair_sort_key(\r\n item: \"tuple[dict[bytes, bytes], int]\",\r\n) -> \"tuple[int, int, int, str, bytes]\":\r\n # prefer to use the highest frequency or shortest length or lexi sort, sligtly slower\r\n pair, freq = item\r\n pair_bytes = pair[0] + pair[1]\r\n pair_byte_length = len(pair_bytes)\r\n pair_str = pair_bytes.decode(\"utf-8\", errors=\"replace\")\r\n pair_str_length = len(pair_str)\r\n return -freq, pair_str_length, pair_byte_length, pair_str, pair_bytes\r\n\r\n\r\ndef learn_bpe(\r\n freqs: \"dict[str,int]\", existing: \"dict[bytes, int]\"\r\n) -> \"tuple[bytes, bytes]\":\r\n vocab = {bpe(k.encode(\"utf-8\"), existing): v for k, v in freqs.items()}\r\n vocab = {key: value for key, value in vocab.items() if len(key) > 1}\r\n new_merges = []\r\n with tqdm_logging_redirect() as bar:\r\n while vocab:\r\n pairs = get_stats(vocab)\r\n\r\n best, freq = min(pairs.items(), key=best_pair_sort_key)\r\n\r\n logger.debug(\r\n f'{best} ({(best[0]+best[1]).decode(\"utf-8\", errors=\"replace\")}) is selected as the next merge with freq {freq}'\r\n )\r\n new_merges.append(best)\r\n\r\n vocab = merge_vocab(best, vocab)\r\n vocab = {key: value for key, value in vocab.items() if len(key) > 1}\r\n bar.update()\r\n\r\n return new_merges\r\n\r\n\r\ndef load_expand_vocab(path: Path) -> \"dict[str, int]\":\r\n freqs = {}\r\n with open(path, \"r\", encoding=\"utf8\") as fin:\r\n for line in fin:\r\n if not line.strip():\r\n continue\r\n word, freq = line.strip().split(\"\\t\")\r\n word = unicodedata.normalize(\"NFC\", word)\r\n parts = re.findall(PAT_STR, word)\r\n if len(parts) > 1:\r\n logger.warning(\r\n f\"{word} would be pre-tokenized to {parts}, and thus cannot be added to vocabulary\"\r\n )\r\n continue\r\n try:\r\n freq = int(freq)\r\n except ValueError as _:\r\n freq = 1\r\n if word in freqs:\r\n logger.warning(\r\n f\"{word} is repeated, the frequency is increased by this much\"\r\n )\r\n freqs[word] += freq\r\n else:\r\n freqs[word] = freq\r\n return freqs\r\n\r\n\r\ndef make_new_merges_by_bpe(\r\n input_path: Path, output_path: Path, expand_path: Path, start_id: int\r\n) -> None:\r\n mergeable_ranks = load_tiktoken_bpe(input_path)\r\n\r\n if not start_id or start_id == -1:\r\n start_id = len(mergeable_ranks)\r\n elif start_id < len(mergeable_ranks):\r\n logger.warning(\r\n f\"start_id {start_id} is too small, existing merges will be overridden, DONOT DO THIS. changed to {len(mergeable_ranks)}\"\r\n )\r\n start_id = len(mergeable_ranks)\r\n else:\r\n start_id = start_id\r\n\r\n expand_vocab_freqs = load_expand_vocab(expand_path)\r\n for word in list(expand_vocab_freqs):\r\n token = word.encode(\"utf-8\")\r\n if token in mergeable_ranks:\r\n logger.warning(f\"word {word} is already a token {token}, skipping\")\r\n del expand_vocab_freqs[word]\r\n\r\n logger.info(f\"number of existing merges: {len(mergeable_ranks)}\")\r\n logger.info(f\"number of words for expanding: {len(expand_vocab_freqs)}\")\r\n\r\n new_merges = learn_bpe(expand_vocab_freqs, mergeable_ranks)\r\n logger.info(f\"number of newly learned merges: {len(new_merges)}\")\r\n\r\n extra_merges = {p[0] + p[1]: i for i, p in enumerate(new_merges, start=start_id)}\r\n\r\n dump_tiktoken_bpe(extra_merges, output_path)\r\n\r\n\r\ndef main():\r\n parser = argparse.ArgumentParser(\r\n formatter_class=argparse.ArgumentDefaultsHelpFormatter\r\n )\r\n parser.add_argument(\"input_path\", type=str, help=\"Path for input tiktoken file\")\r\n parser.add_argument(\r\n \"output_path\",\r\n type=str,\r\n help=\"Path for output tiktoken file, containing only the new merges\",\r\n )\r\n parser.add_argument(\r\n \"vocab_path\",\r\n type=str,\r\n help=\"Path for words needed adding, each line is a word and its frequency separated by \\\\t\",\r\n )\r\n # if the extended vocabulary is for fine-tuning, you better set those correctly (the default is for qwen.tiktoken)\r\n # if the extended vocabulary is for pretraining from the start, no need\r\n parser.add_argument(\r\n \"--start_id\",\r\n type=int,\r\n default=151851,\r\n help=\"The start id for new merges. For Qwen tokenizer, this should be 151851 (skipping the existing special tokens)\",\r\n )\r\n\r\n args = parser.parse_args()\r\n\r\n make_new_merges_by_bpe(\r\n args.input_path, args.output_path, args.vocab_path, args.start_id\r\n )\r\n\r\n\r\nif __name__ == \"__main__\":\r\n main()\r\n" }, { "path": "examples/auto_comments.md", "content": "# Auto Comments \n本文档介绍Auto Comments，这是一个利用Qwen模型为代码文件自动生成注释的使用案例。\n\n# 使用方法\n您可以直接执行如下命令，为提供的代码文件生成注释：\n```\npython auto_comments.py --path 'path of file or folder'\n```\n\n参数：\n- path：文件路径。可以是文件（目前支持python代码文件），也可以是文件夹（会扫描文件夹下所有python代码文件）\n- regenerate：重新生成。默认False，如果针对同一文件需要重新生成注释，请设置为True\n\n# 使用样例\n- 执行：python auto_comments.py --path test_file.py\n- test_file.py 内容为：\n```\nimport numpy as np\nimport pandas as pd\nimport seaborn as sns\nsns.set_theme(style=\"whitegrid\")\n\nrs = np.random.RandomState(365)\nvalues = rs.randn(365, 4).cumsum(axis=0)\ndates = pd.date_range(\"1 1 2016\", periods=365, freq=\"D\")\ndata = pd.DataFrame(values, dates, columns=[\"A\", \"B\", \"C\", \"D\"])\ndata = data.rolling(7).mean()\n\nsns.lineplot(data=data, palette=\"tab10\", linewidth=2.5)\n```\n\n- 输出：test_file_comments.py(包含注释的代码文件)，文件内容如下：\n```\n# 导入需要的库\nimport numpy as np\nimport pandas as pd\nimport seaborn as sns\n\n# 设置 Seaborn 的主题风格为白色网格\nsns.set_theme(style=\"whitegrid\")\n\n# 生成随机数\nrs = np.random.RandomState(365)\n\n# 生成 365 行 4 列的随机数，并按行累加\nvalues = rs.randn(365, 4).cumsum(axis=0)\n\n# 生成日期\ndates = pd.date_range(\"1 1 2016\", periods=365, freq=\"D\")\n\n# 将随机数和日期组合成 DataFrame\ndata = pd.DataFrame(values, dates, columns=[\"A\", \"B\", \"C\", \"D\"])\n\n# 对 DataFrame 进行 7 天滑动平均\ndata = data.rolling(7).mean()\n\n# 使用 Seaborn 绘制折线图\nsns.lineplot(data=data, palette=\"tab10\", linewidth=2.5)\n```\n" }, { "path": "examples/auto_comments.py", "content": "# 运行方式：python auto_comments.py --path 'path of file or folder'\n# 脚本功能：使用QWen-7B-Chat为提供的代码文件自动生成注释。(详见auto_comments.md)\n\n\nimport argparse\nimport os\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\nfrom transformers.generation import GenerationConfig\n\nMaxLine = 50 # 限制单次处理最大代码行数\nSplitKey = [\"\\ndef \"] # 自定义的切分代码标识\nCodeFileType = [\"py\"] # 目前仅测试过对python文件生成注释\n\ndef parse_args():\n parser = argparse.ArgumentParser()\n parser.add_argument('--path', type=str, default='Qwen-7B/eval/evaluate_ceval.py')\n parser.add_argument('--regenerate', action='store_true', default=False) #如果已经生成过注释，默认不会重新生成\n args = parser.parse_args()\n return args\n\nclass QWenChat():\n def __init__(self):\n self.tokenizer = AutoTokenizer.from_pretrained(\"Qwen/Qwen-7B-Chat\", trust_remote_code=True)\n\n # use bf16\n # model = AutoModelForCausalLM.from_pretrained(\"Qwen/Qwen-7B-Chat\", device_map=\"auto\", trust_remote_code=True, bf16=True).eval()\n # use fp16\n # model = AutoModelForCausalLM.from_pretrained(\"Qwen/Qwen-7B-Chat\", device_map=\"auto\", trust_remote_code=True, fp16=True).eval()\n # use cpu only\n # model = AutoModelForCausalLM.from_pretrained(\"Qwen/Qwen-7B-Chat\", device_map=\"cpu\", trust_remote_code=True).eval()\n # use auto mode, automatically select precision based on the device.\n self.model = AutoModelForCausalLM.from_pretrained(\"Qwen/Qwen-7B-Chat\", device_map=\"auto\", trust_remote_code=True).eval()\n \n # Specify hyperparameters for generation\n self.model.generation_config = GenerationConfig.from_pretrained(\"Qwen/Qwen-7B-Chat\", trust_remote_code=True)\n self.history = None\n \n def chat(self, query, system = \"\"):\n\n # use history\n # response, history = self.model.chat(self.tokenizer, query, history=self.history)\n\n # 默认不使用history\n response, history = self.model.chat(self.tokenizer, query, history=None)\n self.history = history\n\n return response\n# 生成注释\ndef gen_code_comments(context, model = None, **kwargs):\n prompt = \"\\n为以上代码生成细致的中文注释，注意使用合适的语法。要求必须在每个函数开头生成一段统一的函数功能注释。\\n除了注释，请保证原始代码内容不变。不要返回除了注释和代码以外的其余信息，不要生成额外代码。\\n\"\n return model.chat(context + prompt)\n\ndef read_file(path):\n f = open(path, \"r\",encoding='utf-8')\n lines = f.readlines()\n return \"\".join(lines)\n\ndef write_file(path, context):\n with open(path,'w') as f:\n f.write(context)\n\n# 如果代码文件过长，可以简单按照最大行数切分代码\ndef split_context_by_maxline(text):\n lines = text.split(\"\\n\")\n lines_len = len(lines)\n res = []\n for i in range(MaxLine, lines_len, MaxLine):\n res.append(\"\\n\".join(lines[i-MaxLine:i]))\n\n if i < lines_len:\n res.append(\"\\n\".join(lines[i:]))\n return res\n\n# 如果代码文件过长，可以简单按照函数切分代码\ndef split_context_by_splitkey(text):\n blocks = text.split(SplitKey[0])\n return [blocks[0]] + [SplitKey[0]+x for x in blocks[1:]]\n\n# merge原始代码和生成的注释，目的是保证原始代码不被更改。这部分可以使用各种不同的策略处理。\ndef merge_code_and_comments(original_file, comments_path):\n res = []\n ori_f = open(original_file, \"r\",encoding='utf-8')\n ori_lines = ori_f.readlines()\n\n com_f = open(comments_path, \"r\",encoding='utf-8')\n com_lines = com_f.readlines()\n len_com_lines = len(com_lines)\n p = 0\n j = 0\n for i, line in enumerate(ori_lines):\n if line.isspace():\n continue\n if line.strip()[0] == '#':\n res.append(line)\n continue\n while j < len_com_lines and line[:-1] not in com_lines[j]:\n j += 1\n if j < len_com_lines:\n p = j - 1\n up_comments = []\n triple_dot_flag = 0\n while p < j:\n if p < 0 or (res and res[-1] and com_lines[p] == res[-1]):\n break\n if com_lines[p].strip() and (len(com_lines[p].strip())>3 and com_lines[p].strip()[-3:] == '\"\"\"' and com_lines[p].strip()[:3] == '\"\"\"') or (len(com_lines[p].strip())>3 and com_lines[p].strip()[-3:] == \"'''\" and com_lines[p].strip()[:3] == \"'''\"):\n up_comments.append(com_lines[p])\n p -= 1\n continue\n if com_lines[p].strip() and (com_lines[p].strip()[-3:] == '\"\"\"' or com_lines[p].strip()[:3] == '\"\"\"' or com_lines[p].strip()[-3:] == \"'''\" or com_lines[p].strip()[:3] == \"'''\"):\n triple_dot_flag = (triple_dot_flag + 1)%2\n up_comments.append(com_lines[p])\n p -= 1\n continue\n if triple_dot_flag:\n up_comments.append(com_lines[p])\n p -= 1\n continue\n if (com_lines[p].strip()==\"\") or (com_lines[p].strip() and com_lines[p].strip()[0] == '#' and \"省略部分内容\" not in com_lines[p]):\n up_comments.append(com_lines[p])\n else:\n break\n p -= 1\n if up_comments:\n res.extend(reversed(up_comments))\n if \"#\" in com_lines[j] and \"#\" not in line:\n in_line_comments = \" #\" + com_lines[j].split(\"#\")[-1]\n res.append(line[:-1]+in_line_comments)\n else:\n res.append(line)\n p = j+1\n else:\n res.append(line)\n j = p\n\n write_file(comments_path, \"\".join(res))\n\n# 处理单个文件\ndef deal_one_file(model, path, args):\n context = read_file(path)\n\n fname = path.split(\"/\")[-1]\n fpath = \"/\".join(path.split(\"/\")[:-1])\n outfname = fname.split(\".\")[0]+\"_comments.\"+fname.split(\".\")[-1]\n\n comments_path = os.path.join(fpath, outfname)\n if (not args.regenerate) and os.path.exists(comments_path):\n print(\"use cache: \", comments_path)\n return\n\n context_line = len(context.split(\"\\n\"))\n if context_line < MaxLine:\n res = gen_code_comments(context, model = model)\n elif SplitKey[0] not in context:\n context_list = split_context_by_maxline(context)\n res = \"\\n\".join([gen_code_comments(context_block, model = model) for context_block in context_list])\n else:\n context_list = split_context_by_splitkey(context)\n res = \"\\n\".join([gen_code_comments(context_block, model = model) for context_block in context_list])\n\n write_file(comments_path, res)\n merge_code_and_comments(path, comments_path)\n\n# 处理文件夹\ndef deal_folder(model, path, args):\n for fl in os.listdir(path):\n now_path = os.path.join(path, fl)\n if os.path.isfile(now_path):\n if (now_path.split(\".\")[-1] in CodeFileType) and (\"_comments\" not in now_path):\n deal_one_file(model, now_path, args)\n elif os.path.isdir(now_path):\n deal_folder(model, now_path, args)\n else:\n print(\"Please specify a correct path!\")\n\ndef transfer(args):\n model = QWenChat()\n\n if os.path.isfile(args.path):\n if (args.path.split(\".\")[-1] in CodeFileType) and (\"_comments\" not in args.path):\n deal_one_file(model, args.path, args)\n elif os.path.isdir(args.path):\n deal_folder(model, args.path, args)\n else:\n print(\"Please specify a correct path!\")\n\nif __name__ == '__main__':\n args = parse_args()\n print(args)\n transfer(args)\n" }, { "path": "examples/function_call_examples.py", "content": "# Reference: https://openai.com/blog/function-calling-and-other-api-updates\nimport json\nfrom pprint import pprint\n\nimport openai\n\n# To start an OpenAI-like Qwen server, use the following commands:\n# git clone https://github.com/QwenLM/Qwen-7B;\n# cd Qwen-7B;\n# pip install fastapi uvicorn openai pydantic sse_starlette;\n# python openai_api.py;\n#\n# Then configure the api_base and api_key in your client:\nopenai.api_base = 'http://localhost:8000/v1'\nopenai.api_key = 'none'\n\n\ndef call_qwen(messages, functions=None):\n print('input:')\n pprint(messages, indent=2)\n if functions:\n response = openai.ChatCompletion.create(model='Qwen',\n messages=messages,\n functions=functions)\n else:\n response = openai.ChatCompletion.create(model='Qwen',\n messages=messages)\n response = response.choices[0]['message']\n response = json.loads(json.dumps(response,\n ensure_ascii=False)) # fix zh rendering\n print('output:')\n pprint(response, indent=2)\n print()\n return response\n\n\ndef test_1():\n messages = [{'role': 'user', 'content': '你好'}]\n call_qwen(messages)\n messages.append({'role': 'assistant', 'content': '你好！很高兴为你提供帮助。'})\n\n messages.append({\n 'role': 'user',\n 'content': '给我讲一个年轻人奋斗创业最终取得成功的故事。故事只能有一句话。'\n })\n call_qwen(messages)\n messages.append({\n 'role':\n 'assistant',\n 'content':\n '故事的主人公叫李明，他来自一个普通的家庭，父母都是普通的工人。李明想要成为一名成功的企业家。……',\n })\n\n messages.append({'role': 'user', 'content': '给这个故事起一个标题'})\n call_qwen(messages)\n\n\ndef test_2():\n functions = [\n {\n 'name_for_human':\n '谷歌搜索',\n 'name_for_model':\n 'google_search',\n 'description_for_model':\n '谷歌搜索是一个通用搜索引擎，可用于访问互联网、查询百科知识、了解时事新闻等。' +\n ' Format the arguments as a JSON object.',\n 'parameters': [{\n 'name': 'search_query',\n 'description': '搜索关键词或短语',\n 'required': True,\n 'schema': {\n 'type': 'string'\n },\n }],\n },\n {\n 'name_for_human':\n '文生图',\n 'name_for_model':\n 'image_gen',\n 'description_for_model':\n '文生图是一个AI绘画（图像生成）服务，输入文本描述，返回根据文本作画得到的图片的URL。' +\n ' Format the arguments as a JSON object.',\n 'parameters': [{\n 'name': 'prompt',\n 'description': '英文关键词，描述了希望图像具有什么内容',\n 'required': True,\n 'schema': {\n 'type': 'string'\n },\n }],\n },\n ]\n\n messages = [{'role': 'user', 'content': '（请不要调用工具）\\n\\n你好'}]\n call_qwen(messages, functions)\n messages.append({\n 'role': 'assistant',\n 'content': '你好！很高兴见到你。有什么我可以帮忙的吗？'\n }, )\n\n messages.append({'role': 'user', 'content': '搜索一下谁是周杰伦'})\n call_qwen(messages, functions)\n messages.append({\n 'role': 'assistant',\n 'content': '我应该使用Google搜索查找相关信息。',\n 'function_call': {\n 'name': 'google_search',\n 'arguments': '{\"search_query\": \"周杰伦\"}',\n },\n })\n\n messages.append({\n 'role': 'function',\n 'name': 'google_search',\n 'content': 'Jay Chou is a Taiwanese singer.',\n })\n call_qwen(messages, functions)\n messages.append(\n {\n 'role': 'assistant',\n 'content': '周杰伦（Jay Chou）是一位来自台湾的歌手。',\n }, )\n\n messages.append({'role': 'user', 'content': '搜索一下他老婆是谁'})\n call_qwen(messages, functions)\n messages.append({\n 'role': 'assistant',\n 'content': '我应该使用Google搜索查找相关信息。',\n 'function_call': {\n 'name': 'google_search',\n 'arguments': '{\"search_query\": \"周杰伦 老婆\"}',\n },\n })\n\n messages.append({\n 'role': 'function',\n 'name': 'google_search',\n 'content': 'Hannah Quinlivan'\n })\n call_qwen(messages, functions)\n messages.append(\n {\n 'role': 'assistant',\n 'content': '周杰伦的老婆是Hannah Quinlivan。',\n }, )\n\n messages.append({'role': 'user', 'content': '用文生图工具画个可爱的小猫吧，最好是黑猫'})\n call_qwen(messages, functions)\n messages.append({\n 'role': 'assistant',\n 'content': '我应该使用文生图API来生成一张可爱的小猫图片。',\n 'function_call': {\n 'name': 'image_gen',\n 'arguments': '{\"prompt\": \"cute black cat\"}',\n },\n })\n\n messages.append({\n 'role':\n 'function',\n 'name':\n 'image_gen',\n 'content':\n '{\"image_url\": \"https://image.pollinations.ai/prompt/cute%20black%20cat\"}',\n })\n call_qwen(messages, functions)\n\n\ndef test_3():\n functions = [{\n 'name': 'get_current_weather',\n 'description': 'Get the current weather in a given location.',\n 'parameters': {\n 'type': 'object',\n 'properties': {\n 'location': {\n 'type': 'string',\n 'description':\n 'The city and state, e.g. San Francisco, CA',\n },\n 'unit': {\n 'type': 'string',\n 'enum': ['celsius', 'fahrenheit']\n },\n },\n 'required': ['location'],\n },\n }]\n\n messages = [{\n 'role': 'user',\n # Note: The current version of Qwen-7B-Chat (as of 2023.08) performs okay with Chinese tool-use prompts,\n # but performs terribly when it comes to English tool-use prompts, due to a mistake in data collecting.\n 'content': '波士顿天气如何？',\n }]\n call_qwen(messages, functions)\n messages.append(\n {\n 'role': 'assistant',\n 'content': None,\n 'function_call': {\n 'name': 'get_current_weather',\n 'arguments': '{\"location\": \"Boston, MA\"}',\n },\n }, )\n\n messages.append({\n 'role':\n 'function',\n 'name':\n 'get_current_weather',\n 'content':\n '{\"temperature\": \"22\", \"unit\": \"celsius\", \"description\": \"Sunny\"}',\n })\n call_qwen(messages, functions)\n\n\ndef test_4():\n from langchain.agents import AgentType, initialize_agent, load_tools\n from langchain.chat_models import ChatOpenAI\n\n llm = ChatOpenAI(\n model_name='Qwen',\n openai_api_base='http://localhost:8000/v1',\n openai_api_key='EMPTY',\n streaming=False,\n )\n tools = load_tools(['arxiv'], )\n agent_chain = initialize_agent(\n tools,\n llm,\n agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION,\n verbose=True,\n )\n # TODO: The performance is okay with Chinese prompts, but not so good when it comes to English.\n agent_chain.run('查一下论文 1605.08386 的信息')\n\n\nif __name__ == '__main__':\n print('### Test Case 1 - No Function Calling (普通问答、无函数调用) ###')\n test_1()\n print('### Test Case 2 - Use Qwen-Style Functions (函数调用，千问格式) ###')\n test_2()\n print('### Test Case 3 - Use GPT-Style Functions (函数调用，GPT格式) ###')\n test_3()\n print('### Test Case 4 - Use LangChain (接入Langchain) ###')\n test_4()\n" }, { "path": "examples/function_call_finetune_examples.py", "content": "#\n# # Fine-tuning Script:\n# Please start by reading the Fine-tuning section of README.md.\n#\n# # Fine-tuning Data Preparation:\n# Then, if you would like to see some examples of how to prepare training samples for function calling,\n# which is actually ReAct prompting under the hood, please read this file.\n\n# # Inference Script:\n# If you are interested in implementing function calling via ReAct prompting for inference,\n# please refer to openai_api.py in our repository.\n#\n# If you have any questions, please raise an issue.\n#\n\nimport json\n\n\ndef format_train_sample(messages):\n #\n # You do not need the `function` role, as Qwen's function calling is actually implemented via ReAct,\n # not by adding a `function` role or `function_call` message. See openai_api.py for details.\n #\n # If you need the `system` role, you might need to modify `finetune.py` accordingly.\n #\n assert set(m[\"role\"] for m in messages) == {\"user\", \"assistant\"}\n\n sample = {\n \"conversations\": [\n {\n \"from\": m[\"role\"],\n \"value\": m[\"content\"],\n }\n for m in messages\n ]\n }\n return sample\n\n\nTOOL_DESC = \"\"\"{name_for_model}: Call this tool to interact with the {name_for_human} API. What is the {name_for_human} API useful for? {description_for_model} Parameters: {parameters}\"\"\"\n\nREACT_INSTRUCTION = \"\"\"Answer the following questions as best you can. You have access to the following APIs:\n\n{tools_text}\n\nUse the following format:\n\nQuestion: the input question you must answer\nThought: you should always think about what to do\nAction: the action to take, should be one of [{tools_name_text}]\nAction Input: the input to the action\nObservation: the result of the action\n... (this Thought/Action/Action Input/Observation can be repeated zero or more times)\nThought: I now know the final answer\nFinal Answer: the final answer to the original input question\n\nBegin!\"\"\"\n\n\ndef build_react_instruction(functions):\n tools_text = []\n tools_name_text = []\n for func_info in functions:\n name = func_info.get(\"name\", \"\")\n name_m = func_info.get(\"name_for_model\", name)\n name_h = func_info.get(\"name_for_human\", name)\n desc = func_info.get(\"description\", \"\")\n desc_m = func_info.get(\"description_for_model\", desc)\n tool = TOOL_DESC.format(\n name_for_model=name_m,\n name_for_human=name_h,\n description_for_model=desc_m,\n parameters=json.dumps(func_info[\"parameters\"], ensure_ascii=False),\n )\n tools_text.append(tool)\n tools_name_text.append(name_m)\n tools_text = \"\\n\\n\".join(tools_text)\n tools_name_text = \", \".join(tools_name_text)\n instruction = REACT_INSTRUCTION.format(\n tools_text=tools_text,\n tools_name_text=tools_name_text,\n )\n return instruction\n\n\ndef main():\n example_train_samples = []\n\n example_functions = [\n {\n \"name_for_human\": \"Google Search\",\n \"name_for_model\": \"google_search\",\n \"description_for_model\": \"Google Search is a general search engine that can be used to access the internet,\"\n + \" query encyclopedia knowledge, and stay informed about current events.\"\n + \" Format the arguments as a JSON object.\", # If you expect `Action Input` to be a JSON.\n \"parameters\": [\n {\n \"name\": \"search_query\",\n \"description\": \"Search keywords or phrases\",\n \"required\": True, # Set to False if it is an optional parameter.\n \"schema\": {\"type\": \"string\"},\n },\n # You can add more parameters to this `parameters` list if you wish.\n ],\n },\n {\n \"name_for_human\": \"Code Interpreter\",\n \"name_for_model\": \"code_interpreter\",\n \"description_for_model\": \"Code interpreter that can execute Python code.\"\n + \"Enclose the code within triple backticks (`)\"\n + \" at the beginning and end of the code.\", # If you expect `Action Input` to be a Markdown code block.\n \"parameters\": [\n {\n \"name\": \"code\",\n \"description\": \"Code to be executed\",\n \"required\": True,\n \"schema\": {\"type\": \"string\"},\n },\n ],\n },\n ]\n example_instruction = build_react_instruction(example_functions)\n\n # This example has multiple actions in one single response.\n example_train_samples.append(\n [\n {\n \"role\": \"user\",\n \"content\": f\"{example_instruction}\\n\\nQuestion: Who is jay chou? And who is his wife?\",\n },\n {\n \"role\": \"assistant\",\n \"content\": \"\"\"\nThought: I need to google Jay Chou.\nAction: google_search\nAction Input: {\"search_query\": \"Jay Chou\"}\nObservation: Jay Chou is ...\nThought: I need to google Jay's wife.\nAction: google_search\nAction Input: {\"search_query\": \"Jay Chou's wife\"}\nObservation: Jay Chou's wife is ...\nThought: I now know the final answer\nFinal Answer: Jay Chou is ... His wife is ...\n \"\"\".strip(),\n },\n ]\n )\n\n # This example involves multiple rounds of conversation.\n example_train_samples.append(\n [\n # Round #1\n {\n \"role\": \"user\",\n \"content\": f\"{example_instruction}\\n\\nQuestion: 123+456=?\",\n },\n {\n \"role\": \"assistant\",\n \"content\": \"\"\"\nThought: I need to compute the result using Code Interpreter.\nAction: code_interpreter\nAction Input:\n```py\n123 + 456\n```\nObservation: 579\nThought: I now know the final answer\nFinal Answer: 579\n \"\"\".strip(),\n },\n # Round #2\n {\n \"role\": \"user\",\n \"content\": \"Multiply the result by 2.\",\n },\n {\n \"role\": \"assistant\",\n \"content\": \"\"\"\nThought: Code Interpreter is helpful for answering this question.\nAction: code_interpreter\nAction Input:\n```py\n579 * 2\n```\nObservation: 1158\nThought: I now know the final answer\nFinal Answer: 1158\n \"\"\".strip(),\n },\n # Round #3\n {\n \"role\": \"user\",\n \"content\": \"You are so smart, Qwen.\", # No action is needed for this question.\n },\n {\n \"role\": \"assistant\",\n \"content\": \"\"\"\nThought: I now know the final answer\nFinal Answer: Thank you.\n \"\"\".strip(),\n },\n # Round #4\n {\n \"role\": \"user\",\n \"content\": \"Please re-execute the code for computing my first question again.\",\n },\n {\n \"role\": \"assistant\",\n \"content\": \"\"\"\nThought: I need to re-compute the result.\nAction: code_interpreter\nAction Input:\n```py\n123 + 456\n```\nObservation: 579\nThought: I now know the final answer\nFinal Answer: 579\n \"\"\".strip(),\n },\n ]\n )\n\n example_train_samples = [format_train_sample(x) for x in example_train_samples]\n with open(\n \"example_func_call_train_samples.json\", \"w\"\n ) as fout: # data for fine-tuning\n fout.write(json.dumps(example_train_samples, indent=2, ensure_ascii=False))\n\n\nif __name__ == \"__main__\":\n main()\n" }, { "path": "examples/langchain_tooluse.ipynb", "content": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"id\": \"30e24ef3\",\n \"metadata\": {\n \"tags\": []\n },\n \"source\": [\n \"# 如何让 Qwen-7b 使用 Langchain 中的 工具\\n\",\n \"\\n\",\n \"本文档主要介绍如何让千问调用 [LangChain](https://python.langchain.com/docs/get_started/introduction.html) 框架中实现好的谷歌搜索、 WolframAlpha 等工具。将主要基于 [ReAct Prompting](https://github.com/QwenLM/Qwen-7B/blob/main/examples/react_prompt.md) 技术，一种特殊的链式思考（Chain-of-Thought，简称 CoT）提示技巧，来实现这一目的。\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"212979ec\",\n \"metadata\": {\n \"tags\": []\n },\n \"source\": [\n \"## 安装依赖\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 1,\n \"id\": \"e21c6728\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# 安装千问的依赖\\n\",\n \"!cd Qwen-7b\\n\",\n \"!pip install -r requirements.txt\\n\",\n \"\\n\",\n \"# 安装 langchain 相关依赖\\n\",\n \"!pip install langchain google-search-results wolframalpha arxiv;\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"3b5e6ef9\",\n \"metadata\": {\n \"tags\": []\n },\n \"source\": [\n \"## 第零步 - 导入 LangChain 的工具\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"af7d0058\",\n \"metadata\": {},\n \"source\": [\n \"以下引入几个常用 APIs 作为演示：\\n\",\n \" - [谷歌搜索API](https://serper.dev/?gclid=EAIaIQobChMIj9eqof7OgAMV44VbCh1F3QZoEAAYASABEgIh3fD_BwE#google-search-api)\\n\",\n \" - [WolframAlpha](https://products.wolframalpha.com/api/)\\n\",\n \" - arxiv论文搜索\\n\",\n \" - python shell (需升级python至3.9以上使用)\\n\",\n \"\\n\",\n \"注1：此处推荐模仿此案例，细致地构造给千问看的工具描述。\\n\",\n \"\\n\",\n \"注2：谷歌搜索（SERPAPI）， WolframAlpha 需自行申请它们的 API_KEY 后才能使用。\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 2,\n \"id\": \"07e49b98-9d6c-41f2-9b18-f043f2d13e1a\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"from langchain import SerpAPIWrapper\\n\",\n \"from langchain.utilities.wolfram_alpha import WolframAlphaAPIWrapper\\n\",\n \"from langchain.utilities import ArxivAPIWrapper\\n\",\n \"from langchain.tools.python.tool import PythonAstREPLTool\\n\",\n \"\\n\",\n \"from typing import Dict, Tuple\\n\",\n \"import os\\n\",\n \"import json\\n\",\n \"\\n\",\n \"from transformers import AutoModelForCausalLM, AutoTokenizer\\n\",\n \"from transformers.generation import GenerationConfig\\n\",\n \"\\n\",\n \"# 为了使用谷歌搜索（SERPAPI）， WolframAlpha，您需要自行申请它们的 API KEY，然后填入此处\\n\",\n \"os.environ['SERPAPI_API_KEY'] = '重要！请在这里填入您的 SERPAPI_API_KEY！'\\n\",\n \"os.environ['WOLFRAM_ALPHA_APPID'] = '重要！请在这里填入您的 WOLFRAM_ALPHA_APPID！'\\n\",\n \"\\n\",\n \"search = SerpAPIWrapper()\\n\",\n \"WolframAlpha = WolframAlphaAPIWrapper()\\n\",\n \"arxiv = ArxivAPIWrapper()\\n\",\n \"python=PythonAstREPLTool()\\n\",\n \"\\n\",\n \"def tool_wrapper_for_qwen(tool):\\n\",\n \" def tool_(query):\\n\",\n \" query = json.loads(query)[\\\"query\\\"]\\n\",\n \" return tool.run(query)\\n\",\n \" return tool_\\n\",\n \"\\n\",\n \"# 以下是给千问看的工具描述：\\n\",\n \"TOOLS = [\\n\",\n \" {\\n\",\n \" 'name_for_human':\\n\",\n \" 'google search',\\n\",\n \" 'name_for_model':\\n\",\n \" 'Search',\\n\",\n \" 'description_for_model':\\n\",\n \" 'useful for when you need to answer questions about current events.',\\n\",\n \" 'parameters': [{\\n\",\n \" \\\"name\\\": \\\"query\\\",\\n\",\n \" \\\"type\\\": \\\"string\\\",\\n\",\n \" \\\"description\\\": \\\"search query of google\\\",\\n\",\n \" 'required': True\\n\",\n \" }], \\n\",\n \" 'tool_api': tool_wrapper_for_qwen(search)\\n\",\n \" },\\n\",\n \" {\\n\",\n \" 'name_for_human':\\n\",\n \" 'Wolfram Alpha',\\n\",\n \" 'name_for_model':\\n\",\n \" 'Math',\\n\",\n \" 'description_for_model':\\n\",\n \" 'Useful for when you need to answer questions about Math, Science, Technology, Culture, Society and Everyday Life.',\\n\",\n \" 'parameters': [{\\n\",\n \" \\\"name\\\": \\\"query\\\",\\n\",\n \" \\\"type\\\": \\\"string\\\",\\n\",\n \" \\\"description\\\": \\\"the problem to solved by Wolfram Alpha\\\",\\n\",\n \" 'required': True\\n\",\n \" }], \\n\",\n \" 'tool_api': tool_wrapper_for_qwen(WolframAlpha)\\n\",\n \" }, \\n\",\n \" {\\n\",\n \" 'name_for_human':\\n\",\n \" 'arxiv',\\n\",\n \" 'name_for_model':\\n\",\n \" 'Arxiv',\\n\",\n \" 'description_for_model':\\n\",\n \" 'A wrapper around Arxiv.org Useful for when you need to answer questions about Physics, Mathematics, Computer Science, Quantitative Biology, Quantitative Finance, Statistics, Electrical Engineering, and Economics from scientific articles on arxiv.org.',\\n\",\n \" 'parameters': [{\\n\",\n \" \\\"name\\\": \\\"query\\\",\\n\",\n \" \\\"type\\\": \\\"string\\\",\\n\",\n \" \\\"description\\\": \\\"the document id of arxiv to search\\\",\\n\",\n \" 'required': True\\n\",\n \" }], \\n\",\n \" 'tool_api': tool_wrapper_for_qwen(arxiv)\\n\",\n \" },\\n\",\n \" {\\n\",\n \" 'name_for_human':\\n\",\n \" 'python',\\n\",\n \" 'name_for_model':\\n\",\n \" 'python',\\n\",\n \" 'description_for_model':\\n\",\n \" \\\"A Python shell. Use this to execute python commands. When using this tool, sometimes output is abbreviated - Make sure it does not look abbreviated before using it in your answer. \\\"\\n\",\n \" \\\"Don't add comments to your python code.\\\",\\n\",\n \" 'parameters': [{\\n\",\n \" \\\"name\\\": \\\"query\\\",\\n\",\n \" \\\"type\\\": \\\"string\\\",\\n\",\n \" \\\"description\\\": \\\"a valid python command.\\\",\\n\",\n \" 'required': True\\n\",\n \" }],\\n\",\n \" 'tool_api': tool_wrapper_for_qwen(python)\\n\",\n \" }\\n\",\n \"\\n\",\n \"]\\n\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"b7ec2027\",\n \"metadata\": {},\n \"source\": [\n \"## 第一步：让千问判断调用什么工具，生成工具入参\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"7a50d676\",\n \"metadata\": {},\n \"source\": [\n \"根据prompt模版、query、工具的信息构建prompt\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 3,\n \"id\": \"4a8feb0e-22f7-4184-9ea0-b864812c9b09\",\n \"metadata\": {\n \"scrolled\": true\n },\n \"outputs\": [\n {\n \"name\": \"stdout\",\n \"output_type\": \"stream\",\n \"text\": [\n \"Answer the following questions as best you can. You have access to the following tools:\\n\",\n \"\\n\",\n \"Search: Call this tool to interact with the google search API. What is the google search API useful for? useful for when you need to answer questions about current events. Parameters: [{\\\"name\\\": \\\"query\\\", \\\"type\\\": \\\"string\\\", \\\"description\\\": \\\"search query of google\\\", \\\"required\\\": true}] Format the arguments as a JSON object.\\n\",\n \"\\n\",\n \"Use the following format:\\n\",\n \"\\n\",\n \"Question: the input question you must answer\\n\",\n \"Thought: you should always think about what to do\\n\",\n \"Action: the action to take, should be one of [Search]\\n\",\n \"Action Input: the input to the action\\n\",\n \"Observation: the result of the action\\n\",\n \"... (this Thought/Action/Action Input/Observation can be repeated zero or more times)\\n\",\n \"Thought: I now know the final answer\\n\",\n \"Final Answer: the final answer to the original input question\\n\",\n \"\\n\",\n \"Begin!\\n\",\n \"\\n\",\n \"Question: 加拿大2023年人口统计数字是多少？\\n\"\n ]\n }\n ],\n \"source\": [\n \"TOOL_DESC = \\\"\\\"\\\"{name_for_model}: Call this tool to interact with the {name_for_human} API. What is the {name_for_human} API useful for? {description_for_model} Parameters: {parameters} Format the arguments as a JSON object.\\\"\\\"\\\"\\n\",\n \"\\n\",\n \"REACT_PROMPT = \\\"\\\"\\\"Answer the following questions as best you can. You have access to the following tools:\\n\",\n \"\\n\",\n \"{tool_descs}\\n\",\n \"\\n\",\n \"Use the following format:\\n\",\n \"\\n\",\n \"Question: the input question you must answer\\n\",\n \"Thought: you should always think about what to do\\n\",\n \"Action: the action to take, should be one of [{tool_names}]\\n\",\n \"Action Input: the input to the action\\n\",\n \"Observation: the result of the action\\n\",\n \"... (this Thought/Action/Action Input/Observation can be repeated zero or more times)\\n\",\n \"Thought: I now know the final answer\\n\",\n \"Final Answer: the final answer to the original input question\\n\",\n \"\\n\",\n \"Begin!\\n\",\n \"\\n\",\n \"Question: {query}\\\"\\\"\\\"\\n\",\n \"\\n\",\n \"def build_planning_prompt(TOOLS, query):\\n\",\n \" tool_descs = []\\n\",\n \" tool_names = []\\n\",\n \" for info in TOOLS:\\n\",\n \" tool_descs.append(\\n\",\n \" TOOL_DESC.format(\\n\",\n \" name_for_model=info['name_for_model'],\\n\",\n \" name_for_human=info['name_for_human'],\\n\",\n \" description_for_model=info['description_for_model'],\\n\",\n \" parameters=json.dumps(\\n\",\n \" info['parameters'], ensure_ascii=False),\\n\",\n \" )\\n\",\n \" )\\n\",\n \" tool_names.append(info['name_for_model'])\\n\",\n \" tool_descs = '\\\\n\\\\n'.join(tool_descs)\\n\",\n \" tool_names = ','.join(tool_names)\\n\",\n \"\\n\",\n \" prompt = REACT_PROMPT.format(tool_descs=tool_descs, tool_names=tool_names, query=query)\\n\",\n \" return prompt\\n\",\n \"\\n\",\n \"prompt_1 = build_planning_prompt(TOOLS[0:1], query=\\\"加拿大2023年人口统计数字是多少？\\\")\\n\",\n \"print(prompt_1)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"6f22b002\",\n \"metadata\": {},\n \"source\": [\n \"将prompt作为输入获得response\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 4,\n \"id\": \"f71b2577-118c-4ce2-a0ed-a45ec59ea35b\",\n \"metadata\": {},\n \"outputs\": [\n {\n \"name\": \"stderr\",\n \"output_type\": \"stream\",\n \"text\": [\n \"A new version of the following files was downloaded from https://huggingface.co/Qwen/Qwen-7B-Chat:\\n\",\n \"- tokenization_qwen.py\\n\",\n \". Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision.\\n\",\n \"A new version of the following files was downloaded from https://huggingface.co/Qwen/Qwen-7B-Chat:\\n\",\n \"- configuration_qwen.py\\n\",\n \". Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision.\\n\",\n \"A new version of the following files was downloaded from https://huggingface.co/Qwen/Qwen-7B-Chat:\\n\",\n \"- qwen_generation_utils.py\\n\",\n \". Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision.\\n\",\n \"A new version of the following files was downloaded from https://huggingface.co/Qwen/Qwen-7B-Chat:\\n\",\n \"- modeling_qwen.py\\n\",\n \"- qwen_generation_utils.py\\n\",\n \". Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision.\\n\"\n ]\n },\n {\n \"data\": {\n \"application/vnd.jupyter.widget-view+json\": {\n \"model_id\": \"23435445dded44d6951aa6a7b771a963\",\n \"version_major\": 2,\n \"version_minor\": 0\n },\n \"text/plain\": [\n \"Downloading shards: 0%| | 0/8 [00:00<?, ?it/s]\"\n ]\n },\n \"metadata\": {},\n \"output_type\": \"display_data\"\n },\n {\n \"name\": \"stderr\",\n \"output_type\": \"stream\",\n \"text\": [\n \"The model is automatically converting to bf16 for faster inference. If you want to disable the automatic precision, please manually add bf16/fp16/fp32=True to \\\"AutoModelForCausalLM.from_pretrained\\\".\\n\",\n \"Try importing flash-attention for faster inference...\\n\",\n \"Warning: import flash_attn rotary fail, please install FlashAttention rotary to get higher efficiency https://github.com/Dao-AILab/flash-attention/tree/main/csrc/rotary\\n\",\n \"Warning: import flash_attn rms_norm fail, please install FlashAttention layer_norm to get higher efficiency https://github.com/Dao-AILab/flash-attention/tree/main/csrc/layer_norm\\n\",\n \"Warning: import flash_attn fail, please install FlashAttention to get higher efficiency https://github.com/Dao-AILab/flash-attention\\n\"\n ]\n },\n {\n \"data\": {\n \"application/vnd.jupyter.widget-view+json\": {\n \"model_id\": \"728a1c13c2884291ade4cb4a1edfaaf2\",\n \"version_major\": 2,\n \"version_minor\": 0\n },\n \"text/plain\": [\n \"Loading checkpoint shards: 0%| | 0/8 [00:00<?, ?it/s]\"\n ]\n },\n \"metadata\": {},\n \"output_type\": \"display_data\"\n }\n ],\n \"source\": [\n \"# 国内连 hugginface 网络不好，这段代码可能需要多重试\\n\",\n \"checkpoint = \\\"Qwen/Qwen-7B-Chat\\\"\\n\",\n \"TOKENIZER = AutoTokenizer.from_pretrained(checkpoint, trust_remote_code=True)\\n\",\n \"MODEL = AutoModelForCausalLM.from_pretrained(checkpoint, device_map=\\\"auto\\\", trust_remote_code=True).eval()\\n\",\n \"MODEL.generation_config = GenerationConfig.from_pretrained(checkpoint, trust_remote_code=True)\\n\",\n \"MODEL.generation_config.do_sample = False # greedy\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 5,\n \"id\": \"dc0dbd6c-5a0f-44c9-a019-0ec0283ca92d\",\n \"metadata\": {},\n \"outputs\": [\n {\n \"name\": \"stdout\",\n \"output_type\": \"stream\",\n \"text\": [\n \"Thought: 我应该使用搜索工具帮助我完成任务。search api能完成搜索的任务。\\n\",\n \"Action: Search\\n\",\n \"Action Input: {\\\"query\\\": \\\"加拿大 2023年人口统计数字\\\"}\\n\",\n \"Observation:\\n\"\n ]\n }\n ],\n \"source\": [\n \"stop = [\\\"Observation:\\\", \\\"Observation:\\\\n\\\"]\\n\",\n \"react_stop_words_tokens = [TOKENIZER.encode(stop_) for stop_ in stop]\\n\",\n \"response_1, _ = MODEL.chat(TOKENIZER, prompt_1, history=None, stop_words_ids=react_stop_words_tokens)\\n\",\n \"print(response_1)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"1ebf47ac\",\n \"metadata\": {},\n \"source\": [\n \"## 第二步：从千问的输出中解析需要使用的工具和入参，并调用对应工具\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 6,\n \"id\": \"1a431670-a1f6-4afd-972f-1cfd6d06e8c9\",\n \"metadata\": {},\n \"outputs\": [\n {\n \"name\": \"stdout\",\n \"output_type\": \"stream\",\n \"text\": [\n \"根据加拿大统计局预测，加拿大人口今天（2023年6月16日）预计将超过4000万。 联邦统计局使用模型来实时估计加拿大的人口，该计数模型预计加拿大人口将在北美东部时间今天下午3点前达到4000万。 加拿大的人口增长率目前为2.7％。\\n\"\n ]\n }\n ],\n \"source\": [\n \"def parse_latest_plugin_call(text: str) -> Tuple[str, str]:\\n\",\n \" i = text.rfind('\\\\nAction:')\\n\",\n \" j = text.rfind('\\\\nAction Input:')\\n\",\n \" k = text.rfind('\\\\nObservation:')\\n\",\n \" if 0 <= i < j: # If the text has `Action` and `Action input`,\\n\",\n \" if k < j: # but does not contain `Observation`,\\n\",\n \" # then it is likely that `Observation` is ommited by the LLM,\\n\",\n \" # because the output text may have discarded the stop word.\\n\",\n \" text = text.rstrip() + '\\\\nObservation:' # Add it back.\\n\",\n \" k = text.rfind('\\\\nObservation:')\\n\",\n \" if 0 <= i < j < k:\\n\",\n \" plugin_name = text[i + len('\\\\nAction:'):j].strip()\\n\",\n \" plugin_args = text[j + len('\\\\nAction Input:'):k].strip()\\n\",\n \" return plugin_name, plugin_args\\n\",\n \" return '', ''\\n\",\n \"\\n\",\n \"def use_api(tools, response):\\n\",\n \" use_toolname, action_input = parse_latest_plugin_call(response)\\n\",\n \" if use_toolname == \\\"\\\":\\n\",\n \" return \\\"no tool founds\\\"\\n\",\n \"\\n\",\n \" used_tool_meta = list(filter(lambda x: x[\\\"name_for_model\\\"] == use_toolname, tools))\\n\",\n \" if len(used_tool_meta) == 0:\\n\",\n \" return \\\"no tool founds\\\"\\n\",\n \" \\n\",\n \" api_output = used_tool_meta[0][\\\"tool_api\\\"](action_input)\\n\",\n \" return api_output\\n\",\n \"\\n\",\n \"api_output = use_api(TOOLS, response_1)\\n\",\n \"print(api_output)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"106a4ba0\",\n \"metadata\": {\n \"tags\": []\n },\n \"source\": [\n \"## 第三步：让千问根据工具返回结果继续作答\\n\",\n \"拼接上述返回答案，形成新的prompt，并获得生成最终结果\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 7,\n \"id\": \"a9d4d42d\",\n \"metadata\": {},\n \"outputs\": [\n {\n \"name\": \"stdout\",\n \"output_type\": \"stream\",\n \"text\": [\n \"Answer the following questions as best you can. You have access to the following tools:\\n\",\n \"\\n\",\n \"Search: Call this tool to interact with the google search API. What is the google search API useful for? useful for when you need to answer questions about current events. Parameters: [{\\\"name\\\": \\\"query\\\", \\\"type\\\": \\\"string\\\", \\\"description\\\": \\\"search query of google\\\", \\\"required\\\": true}] Format the arguments as a JSON object.\\n\",\n \"\\n\",\n \"Use the following format:\\n\",\n \"\\n\",\n \"Question: the input question you must answer\\n\",\n \"Thought: you should always think about what to do\\n\",\n \"Action: the action to take, should be one of [Search]\\n\",\n \"Action Input: the input to the action\\n\",\n \"Observation: the result of the action\\n\",\n \"... (this Thought/Action/Action Input/Observation can be repeated zero or more times)\\n\",\n \"Thought: I now know the final answer\\n\",\n \"Final Answer: the final answer to the original input question\\n\",\n \"\\n\",\n \"Begin!\\n\",\n \"\\n\",\n \"Question: 加拿大2023年人口统计数字是多少？Thought: 我应该使用搜索工具帮助我完成任务。search api能完成搜索的任务。\\n\",\n \"Action: Search\\n\",\n \"Action Input: {\\\"query\\\": \\\"加拿大 2023年人口统计数字\\\"}\\n\",\n \"Observation: 根据加拿大统计局预测，加拿大人口今天（2023年6月16日）预计将超过4000万。 联邦统计局使用模型来实时估计加拿大的人口，该计数模型预计加拿大人口将在北美东部时间今天下午3点前达到4000万。 加拿大的人口增长率目前为2.7％。 Thought: I now know the final answer.\\n\",\n \"Final Answer: 加拿大2023年人口统计数字预计为4000万。\\n\"\n ]\n }\n ],\n \"source\": [\n \"prompt_2 = prompt_1 + response_1 + ' ' + api_output\\n\",\n \"stop = [\\\"Observation:\\\", \\\"Observation:\\\\n\\\"]\\n\",\n \"react_stop_words_tokens = [TOKENIZER.encode(stop_) for stop_ in stop]\\n\",\n \"response_2, _ = MODEL.chat(TOKENIZER, prompt_2, history=None, stop_words_ids=react_stop_words_tokens)\\n\",\n \"print(prompt_2, response_2)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"0b8da9fd\",\n \"metadata\": {},\n \"source\": [\n \"## 总结 - 串联起整个流程\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 8,\n \"id\": \"1e51a8ea\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"def main(query, choose_tools):\\n\",\n \" prompt = build_planning_prompt(choose_tools, query) # 组织prompt\\n\",\n \" print(prompt)\\n\",\n \" stop = [\\\"Observation:\\\", \\\"Observation:\\\\n\\\"]\\n\",\n \" react_stop_words_tokens = [TOKENIZER.encode(stop_) for stop_ in stop]\\n\",\n \" response, _ = MODEL.chat(TOKENIZER, prompt, history=None, stop_words_ids=react_stop_words_tokens)\\n\",\n \"\\n\",\n \" while \\\"Final Answer:\\\" not in response: # 出现final Answer时结束\\n\",\n \" api_output = use_api(choose_tools, response) # 抽取入参并执行api\\n\",\n \" api_output = str(api_output) # 部分api工具返回结果非字符串格式需进行转化后输出\\n\",\n \" if \\\"no tool founds\\\" == api_output:\\n\",\n \" break\\n\",\n \" print(\\\"\\\\033[32m\\\" + response + \\\"\\\\033[0m\\\" + \\\"\\\\033[34m\\\" + ' ' + api_output + \\\"\\\\033[0m\\\")\\n\",\n \" prompt = prompt + response + ' ' + api_output # 合并api输出\\n\",\n \" response, _ = MODEL.chat(TOKENIZER, prompt, history=None, stop_words_ids=react_stop_words_tokens) # 继续生成\\n\",\n \"\\n\",\n \" print(\\\"\\\\033[32m\\\" + response + \\\"\\\\033[0m\\\")\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 9,\n \"id\": \"6dc38a34\",\n \"metadata\": {\n \"collapsed\": false,\n \"jupyter\": {\n \"outputs_hidden\": false\n }\n },\n \"outputs\": [\n {\n \"name\": \"stdout\",\n \"output_type\": \"stream\",\n \"text\": [\n \"==========\\n\",\n \"Answer the following questions as best you can. You have access to the following tools:\\n\",\n \"\\n\",\n \"Search: Call this tool to interact with the google search API. What is the google search API useful for? useful for when you need to answer questions about current events. Parameters: [{\\\"name\\\": \\\"query\\\", \\\"type\\\": \\\"string\\\", \\\"description\\\": \\\"search query of google\\\", \\\"required\\\": true}] Format the arguments as a JSON object.\\n\",\n \"\\n\",\n \"Math: Call this tool to interact with the Wolfram Alpha API. What is the Wolfram Alpha API useful for? Useful for when you need to answer questions about Math, Science, Technology, Culture, Society and Everyday Life. Parameters: [{\\\"name\\\": \\\"query\\\", \\\"type\\\": \\\"string\\\", \\\"description\\\": \\\"the problem to solved by Wolfram Alpha\\\", \\\"required\\\": true}] Format the arguments as a JSON object.\\n\",\n \"\\n\",\n \"Arxiv: Call this tool to interact with the arxiv API. What is the arxiv API useful for? A wrapper around Arxiv.org Useful for when you need to answer questions about Physics, Mathematics, Computer Science, Quantitative Biology, Quantitative Finance, Statistics, Electrical Engineering, and Economics from scientific articles on arxiv.org. Parameters: [{\\\"name\\\": \\\"query\\\", \\\"type\\\": \\\"string\\\", \\\"description\\\": \\\"the document id of arxiv to search\\\", \\\"required\\\": true}] Format the arguments as a JSON object.\\n\",\n \"\\n\",\n \"python: Call this tool to interact with the python API. What is the python API useful for? A Python shell. Use this to execute python commands. When using this tool, sometimes output is abbreviated - Make sure it does not look abbreviated before using it in your answer. Don't add comments to your python code. Parameters: [{\\\"name\\\": \\\"query\\\", \\\"type\\\": \\\"string\\\", \\\"description\\\": \\\"a valid python command.\\\", \\\"required\\\": true}] Format the arguments as a JSON object.\\n\",\n \"\\n\",\n \"Use the following format:\\n\",\n \"\\n\",\n \"Question: the input question you must answer\\n\",\n \"Thought: you should always think about what to do\\n\",\n \"Action: the action to take, should be one of [Search,Math,Arxiv,python]\\n\",\n \"Action Input: the input to the action\\n\",\n \"Observation: the result of the action\\n\",\n \"... (this Thought/Action/Action Input/Observation can be repeated zero or more times)\\n\",\n \"Thought: I now know the final answer\\n\",\n \"Final Answer: the final answer to the original input question\\n\",\n \"\\n\",\n \"Begin!\\n\",\n \"\\n\",\n \"Question: 加拿大2022年的人口数量有多少？\\n\",\n \"\\u001B[32mThought: 我应该使用搜索工具帮助我完成任务。search api能完成搜索的任务。\\n\",\n \"Action: Search\\n\",\n \"Action Input: {\\\"query\\\": \\\"加拿大 2022年人口数量\\\"}\\n\",\n \"Observation:\\u001B[0m\\u001B[34m 中新社多伦多3月22日电(记者余瑞冬)加拿大统计局3月22日公布的人口统计数据显示，截至今年1月1日，该国估算总人口约为3956.62万人，且2022年的人口增长数创纪录地突破100万人。 加统计局估算，该国人口在2022年增长105.011万人，年增长2.7%，创1957年以来最大增幅。\\u001B[0m\\n\",\n \"\\u001B[32mThought: I now know the final answer.\\n\",\n \"Final Answer: 加拿大2022年的人口数量约为3956.62万人。\\u001B[0m\\n\",\n \"==========\\n\",\n \"Answer the following questions as best you can. You have access to the following tools:\\n\",\n \"\\n\",\n \"Search: Call this tool to interact with the google search API. What is the google search API useful for? useful for when you need to answer questions about current events. Parameters: [{\\\"name\\\": \\\"query\\\", \\\"type\\\": \\\"string\\\", \\\"description\\\": \\\"search query of google\\\", \\\"required\\\": true}] Format the arguments as a JSON object.\\n\",\n \"\\n\",\n \"Math: Call this tool to interact with the Wolfram Alpha API. What is the Wolfram Alpha API useful for? Useful for when you need to answer questions about Math, Science, Technology, Culture, Society and Everyday Life. Parameters: [{\\\"name\\\": \\\"query\\\", \\\"type\\\": \\\"string\\\", \\\"description\\\": \\\"the problem to solved by Wolfram Alpha\\\", \\\"required\\\": true}] Format the arguments as a JSON object.\\n\",\n \"\\n\",\n \"Arxiv: Call this tool to interact with the arxiv API. What is the arxiv API useful for? A wrapper around Arxiv.org Useful for when you need to answer questions about Physics, Mathematics, Computer Science, Quantitative Biology, Quantitative Finance, Statistics, Electrical Engineering, and Economics from scientific articles on arxiv.org. Parameters: [{\\\"name\\\": \\\"query\\\", \\\"type\\\": \\\"string\\\", \\\"description\\\": \\\"the document id of arxiv to search\\\", \\\"required\\\": true}] Format the arguments as a JSON object.\\n\",\n \"\\n\",\n \"python: Call this tool to interact with the python API. What is the python API useful for? A Python shell. Use this to execute python commands. When using this tool, sometimes output is abbreviated - Make sure it does not look abbreviated before using it in your answer. Don't add comments to your python code. Parameters: [{\\\"name\\\": \\\"query\\\", \\\"type\\\": \\\"string\\\", \\\"description\\\": \\\"a valid python command.\\\", \\\"required\\\": true}] Format the arguments as a JSON object.\\n\",\n \"\\n\",\n \"Use the following format:\\n\",\n \"\\n\",\n \"Question: the input question you must answer\\n\",\n \"Thought: you should always think about what to do\\n\",\n \"Action: the action to take, should be one of [Search,Math,Arxiv,python]\\n\",\n \"Action Input: the input to the action\\n\",\n \"Observation: the result of the action\\n\",\n \"... (this Thought/Action/Action Input/Observation can be repeated zero or more times)\\n\",\n \"Thought: I now know the final answer\\n\",\n \"Final Answer: the final answer to the original input question\\n\",\n \"\\n\",\n \"Begin!\\n\",\n \"\\n\",\n \"Question: 求解方程 2x+5 = -3x + 7\\n\",\n \"\\u001B[32mThought: 我应该使用数学工具帮助我完成任务。Wolfram Alpha API应该能完成这项任务。\\n\",\n \"Action: Math\\n\",\n \"Action Input: {\\\"query\\\": \\\"2x+5 = -3x + 7\\\"}\\n\",\n \"Observation:\\u001B[0m\\u001B[34m Assumption: 2 x + 5 = -3 x + 7 \\n\",\n \"Answer: x = 2/5\\u001B[0m\\n\",\n \"\\u001B[32mThought: I now know the final answer.\\n\",\n \"Final Answer: x = 2/5\\u001B[0m\\n\",\n \"==========\\n\",\n \"Answer the following questions as best you can. You have access to the following tools:\\n\",\n \"\\n\",\n \"Search: Call this tool to interact with the google search API. What is the google search API useful for? useful for when you need to answer questions about current events. Parameters: [{\\\"name\\\": \\\"query\\\", \\\"type\\\": \\\"string\\\", \\\"description\\\": \\\"search query of google\\\", \\\"required\\\": true}] Format the arguments as a JSON object.\\n\",\n \"\\n\",\n \"Math: Call this tool to interact with the Wolfram Alpha API. What is the Wolfram Alpha API useful for? Useful for when you need to answer questions about Math, Science, Technology, Culture, Society and Everyday Life. Parameters: [{\\\"name\\\": \\\"query\\\", \\\"type\\\": \\\"string\\\", \\\"description\\\": \\\"the problem to solved by Wolfram Alpha\\\", \\\"required\\\": true}] Format the arguments as a JSON object.\\n\",\n \"\\n\",\n \"Arxiv: Call this tool to interact with the arxiv API. What is the arxiv API useful for? A wrapper around Arxiv.org Useful for when you need to answer questions about Physics, Mathematics, Computer Science, Quantitative Biology, Quantitative Finance, Statistics, Electrical Engineering, and Economics from scientific articles on arxiv.org. Parameters: [{\\\"name\\\": \\\"query\\\", \\\"type\\\": \\\"string\\\", \\\"description\\\": \\\"the document id of arxiv to search\\\", \\\"required\\\": true}] Format the arguments as a JSON object.\\n\",\n \"\\n\",\n \"python: Call this tool to interact with the python API. What is the python API useful for? A Python shell. Use this to execute python commands. When using this tool, sometimes output is abbreviated - Make sure it does not look abbreviated before using it in your answer. Don't add comments to your python code. Parameters: [{\\\"name\\\": \\\"query\\\", \\\"type\\\": \\\"string\\\", \\\"description\\\": \\\"a valid python command.\\\", \\\"required\\\": true}] Format the arguments as a JSON object.\\n\",\n \"\\n\",\n \"Use the following format:\\n\",\n \"\\n\",\n \"Question: the input question you must answer\\n\",\n \"Thought: you should always think about what to do\\n\",\n \"Action: the action to take, should be one of [Search,Math,Arxiv,python]\\n\",\n \"Action Input: the input to the action\\n\",\n \"Observation: the result of the action\\n\",\n \"... (this Thought/Action/Action Input/Observation can be repeated zero or more times)\\n\",\n \"Thought: I now know the final answer\\n\",\n \"Final Answer: the final answer to the original input question\\n\",\n \"\\n\",\n \"Begin!\\n\",\n \"\\n\",\n \"Question: 编号是1605.08386的论文讲了些什么？\\n\",\n \"\\u001B[32mThought: 我需要使用Arxiv API来搜索这篇论文。\\n\",\n \"Action: Arxiv\\n\",\n \"Action Input: {\\\"query\\\": \\\"1605.08386\\\"}\\n\",\n \"Observation:\\u001B[0m\\u001B[34m Published: 2016-05-26\\n\",\n \"Title: Heat-bath random walks with Markov bases\\n\",\n \"Authors: Caprice Stanley, Tobias Windisch\\n\",\n \"Summary: Graphs on lattice points are studied whose edges come from a finite set of\\n\",\n \"allowed moves of arbitrary length. We show that the diameter of these graphs on\\n\",\n \"fibers of a fixed integer matrix can be bounded from above by a constant. We\\n\",\n \"then study the mixing behaviour of heat-bath random walks on these graphs. We\\n\",\n \"also state explicit conditions on the set of moves so that the heat-bath random\\n\",\n \"walk, a generalization of the Glauber dynamics, is an expander in fixed\\n\",\n \"dimension.\\u001B[0m\\n\",\n \"\\u001B[32mThought: I now know the final answer.\\n\",\n \"Final Answer: 这篇论文的题目是《热浴随机游走的马尔可夫基》，作者是Caprice Stanley和Tobias Windisch。摘要中提到，该论文研究了在有限的允许移动集合中，由任意长度的边构成的图的边。我们证明了这些图在固定整数矩阵纤维上的直径可以被一个常数所限制。然后，我们研究了热浴随机游走在这类图上的混合行为。我们还给出了一个明确的条件，使得热浴随机游走（一个Glauber动力学的推广）在固定维度下是一个扩张。\\u001B[0m\\n\",\n \"==========\\n\",\n \"Answer the following questions as best you can. You have access to the following tools:\\n\",\n \"\\n\",\n \"Search: Call this tool to interact with the google search API. What is the google search API useful for? useful for when you need to answer questions about current events. Parameters: [{\\\"name\\\": \\\"query\\\", \\\"type\\\": \\\"string\\\", \\\"description\\\": \\\"search query of google\\\", \\\"required\\\": true}] Format the arguments as a JSON object.\\n\",\n \"\\n\",\n \"Math: Call this tool to interact with the Wolfram Alpha API. What is the Wolfram Alpha API useful for? Useful for when you need to answer questions about Math, Science, Technology, Culture, Society and Everyday Life. Parameters: [{\\\"name\\\": \\\"query\\\", \\\"type\\\": \\\"string\\\", \\\"description\\\": \\\"the problem to solved by Wolfram Alpha\\\", \\\"required\\\": true}] Format the arguments as a JSON object.\\n\",\n \"\\n\",\n \"Arxiv: Call this tool to interact with the arxiv API. What is the arxiv API useful for? A wrapper around Arxiv.org Useful for when you need to answer questions about Physics, Mathematics, Computer Science, Quantitative Biology, Quantitative Finance, Statistics, Electrical Engineering, and Economics from scientific articles on arxiv.org. Parameters: [{\\\"name\\\": \\\"query\\\", \\\"type\\\": \\\"string\\\", \\\"description\\\": \\\"the document id of arxiv to search\\\", \\\"required\\\": true}] Format the arguments as a JSON object.\\n\",\n \"\\n\",\n \"python: Call this tool to interact with the python API. What is the python API useful for? A Python shell. Use this to execute python commands. When using this tool, sometimes output is abbreviated - Make sure it does not look abbreviated before using it in your answer. Don't add comments to your python code. Parameters: [{\\\"name\\\": \\\"query\\\", \\\"type\\\": \\\"string\\\", \\\"description\\\": \\\"a valid python command.\\\", \\\"required\\\": true}] Format the arguments as a JSON object.\\n\",\n \"\\n\",\n \"Use the following format:\\n\",\n \"\\n\",\n \"Question: the input question you must answer\\n\",\n \"Thought: you should always think about what to do\\n\",\n \"Action: the action to take, should be one of [Search,Math,Arxiv,python]\\n\",\n \"Action Input: the input to the action\\n\",\n \"Observation: the result of the action\\n\",\n \"... (this Thought/Action/Action Input/Observation can be repeated zero or more times)\\n\",\n \"Thought: I now know the final answer\\n\",\n \"Final Answer: the final answer to the original input question\\n\",\n \"\\n\",\n \"Begin!\\n\",\n \"\\n\",\n \"Question: 使用python对下面的列表进行排序： [2, 4135, 523, 2, 3]\\n\",\n \"\\u001B[32mThought: 我应该使用python API来执行python命令。\\n\",\n \"Action: python\\n\",\n \"Action Input: {\\\"query\\\": \\\"sorted([2, 4135, 523, 2, 3])\\\"}\\n\",\n \"Observation:\\u001B[0m\\u001B[34m [2, 2, 3, 523, 4135]\\u001B[0m\\n\",\n \"\\u001B[32mThought: I now know the final answer.\\n\",\n \"Final Answer: 使用python对给定的列表进行排序，结果为 [2, 2, 3, 523, 4135]。\\u001B[0m\\n\"\n ]\n }\n ],\n \"source\": [\n \"# 请尽可能控制备选工具数量\\n\",\n \"query = \\\"加拿大2022年的人口数量有多少？\\\" # 所提问题\\n\",\n \"choose_tools = TOOLS # 选择备选工具\\n\",\n \"print(\\\"=\\\" * 10)\\n\",\n \"main(query, choose_tools)\\n\",\n \"\\n\",\n \"query = \\\"求解方程 2x+5 = -3x + 7\\\" # 所提问题\\n\",\n \"choose_tools = TOOLS # 选择备选工具\\n\",\n \"print(\\\"=\\\" * 10)\\n\",\n \"main(query, choose_tools)\\n\",\n \"\\n\",\n \"query = \\\"编号是1605.08386的论文讲了些什么？\\\" # 所提问题\\n\",\n \"choose_tools = TOOLS # 选择备选工具\\n\",\n \"print(\\\"=\\\" * 10)\\n\",\n \"main(query, choose_tools)\\n\",\n \"\\n\",\n \"query =\\\"使用python对下面的列表进行排序： [2, 4135, 523, 2, 3]\\\"\\n\",\n \"choose_tools = TOOLS # 选择备选工具\\n\",\n \"print(\\\"=\\\" * 10)\\n\",\n \"main(query, choose_tools)\"\n ]\n }\n ],\n \"metadata\": {\n \"kernelspec\": {\n \"display_name\": \"Python 3 (ipykernel)\",\n \"language\": \"python\",\n \"name\": \"python3\"\n },\n \"language_info\": {\n \"codemirror_mode\": {\n \"name\": \"ipython\",\n \"version\": 3\n },\n \"file_extension\": \".py\",\n \"mimetype\": \"text/x-python\",\n \"name\": \"python\",\n \"nbconvert_exporter\": \"python\",\n \"pygments_lexer\": \"ipython3\",\n \"version\": \"3.9.16\"\n }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 5\n}\n" }, { "path": "examples/qwen_extra.tiktoken", "content": "5LiA5Y+q54yr 151851\n5Y+q54yr 151852\n5piv5LiA5Y+q54yr 151853\n5oiR5piv5LiA5Y+q54yr 151854\n5L2g5piv5LiA5Y+q54yr 151855\n5LuW5piv5LiA5Y+q54yr 151856\n" }, { "path": "examples/qwen_extra_vocab.txt", "content": "我是一只猫\t20\r\n你是一只猫\t10\r\n他是一只猫\t5\r\n一只\t200\r\n一只猫\t100\r\n夸张的 比喻手法\t20" }, { "path": "examples/react_demo.py", "content": "#\n# 相关材料：\n# ReAct Prompting 原理简要介绍，不包含代码实现：\n# https://github.com/QwenLM/Qwen-7B/blob/main/examples/react_prompt.md\n# 基于 model.chat 接口（对话模式）的 ReAct Prompting 实现（含接入 LangChain 的工具实现）：\n# https://github.com/QwenLM/Qwen-7B/blob/main/examples/langchain_tooluse.ipynb\n# 基于 model.generate 接口（续写模式）的 ReAct Prompting 实现，比 chat 模式的实现更复杂些：\n# https://github.com/QwenLM/Qwen-7B/blob/main/examples/react_demo.py（本文件）\n#\n\nimport json\nimport os\n\nimport json5\nimport torch\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\nfrom transformers.generation import GenerationConfig\n\nfor _ in range(10): # 网络不稳定，多试几次\n try:\n name = 'Qwen/Qwen-7B-Chat'\n tokenizer = AutoTokenizer.from_pretrained(name, trust_remote_code=True)\n generation_config = GenerationConfig.from_pretrained(name, trust_remote_code=True)\n model = AutoModelForCausalLM.from_pretrained(\n name, device_map=\"auto\", trust_remote_code=True\n ).eval()\n model.generation_config = generation_config\n model.generation_config.top_k = 1\n break\n except Exception as e:\n print(e)\n\n# 将一个插件的关键信息拼接成一段文本的模版。\nTOOL_DESC = \"\"\"{name_for_model}: Call this tool to interact with the {name_for_human} API. What is the {name_for_human} API useful for? {description_for_model} Parameters: {parameters}\"\"\"\n\n# ReAct prompting 的 instruction 模版，将包含插件的详细信息。\nPROMPT_REACT = \"\"\"Answer the following questions as best you can. You have access to the following APIs:\n\n{tools_text}\n\nUse the following format:\n\nQuestion: the input question you must answer\nThought: you should always think about what to do\nAction: the action to take, should be one of [{tools_name_text}]\nAction Input: the input to the action\nObservation: the result of the action\n... (this Thought/Action/Action Input/Observation can be repeated zero or more times)\nThought: I now know the final answer\nFinal Answer: the final answer to the original input question\n\nBegin!\n\nQuestion: {query}\"\"\"\n\n\n#\n# 本示例代码的入口函数。\n#\n# 输入：\n# prompt: 用户的最新一个问题。\n# history: 用户与模型的对话历史，是一个 list，\n# list 中的每个元素为 {\"user\": \"用户输入\", \"bot\": \"模型输出\"} 的一轮对话。\n# 最新的一轮对话放 list 末尾。不包含最新一个问题。\n# list_of_plugin_info: 候选插件列表，是一个 list，list 中的每个元素为一个插件的关键信息。\n# 比如 list_of_plugin_info = [plugin_info_0, plugin_info_1, plugin_info_2]，\n# 其中 plugin_info_0, plugin_info_1, plugin_info_2 这几个样例见本文档前文。\n#\n# 输出：\n# 模型对用户最新一个问题的回答。\n#\ndef llm_with_plugin(prompt: str, history, list_of_plugin_info=()):\n chat_history = [(x['user'], x['bot']) for x in history] + [(prompt, '')]\n\n # 需要让模型进行续写的初始文本\n planning_prompt = build_input_text(chat_history, list_of_plugin_info)\n\n text = ''\n while True:\n output = text_completion(planning_prompt + text, stop_words=['Observation:', 'Observation:\\n'])\n action, action_input, output = parse_latest_plugin_call(output)\n if action: # 需要调用插件\n # action、action_input 分别为需要调用的插件代号、输入参数\n # observation是插件返回的结果，为字符串\n observation = call_plugin(action, action_input)\n output += f'\\nObservation: {observation}\\nThought:'\n text += output\n else: # 生成结束，并且不再需要调用插件\n text += output\n break\n\n new_history = []\n new_history.extend(history)\n new_history.append({'user': prompt, 'bot': text})\n return text, new_history\n\n\n# 将对话历史、插件信息聚合成一段初始文本\ndef build_input_text(chat_history, list_of_plugin_info) -> str:\n # 候选插件的详细信息\n tools_text = []\n for plugin_info in list_of_plugin_info:\n tool = TOOL_DESC.format(\n name_for_model=plugin_info[\"name_for_model\"],\n name_for_human=plugin_info[\"name_for_human\"],\n description_for_model=plugin_info[\"description_for_model\"],\n parameters=json.dumps(plugin_info[\"parameters\"], ensure_ascii=False),\n )\n if plugin_info.get('args_format', 'json') == 'json':\n tool += \" Format the arguments as a JSON object.\"\n elif plugin_info['args_format'] == 'code':\n tool += ' Enclose the code within triple backticks (`) at the beginning and end of the code.'\n else:\n raise NotImplementedError\n tools_text.append(tool)\n tools_text = '\\n\\n'.join(tools_text)\n\n # 候选插件的代号\n tools_name_text = ', '.join([plugin_info[\"name_for_model\"] for plugin_info in list_of_plugin_info])\n\n im_start = '<|im_start|>'\n im_end = '<|im_end|>'\n prompt = f'{im_start}system\\nYou are a helpful assistant.{im_end}'\n for i, (query, response) in enumerate(chat_history):\n if list_of_plugin_info: # 如果有候选插件\n # 倒数第一轮或倒数第二轮对话填入详细的插件信息，但具体什么位置填可以自行判断\n if (len(chat_history) == 1) or (i == len(chat_history) - 2):\n query = PROMPT_REACT.format(\n tools_text=tools_text,\n tools_name_text=tools_name_text,\n query=query,\n )\n query = query.lstrip('\\n').rstrip() # 重要！若不 strip 会与训练时数据的构造方式产生差异。\n response = response.lstrip('\\n').rstrip() # 重要！若不 strip 会与训练时数据的构造方式产生差异。\n # 使用续写模式（text completion）时，需要用如下格式区分用户和AI：\n prompt += f\"\\n{im_start}user\\n{query}{im_end}\"\n prompt += f\"\\n{im_start}assistant\\n{response}{im_end}\"\n\n assert prompt.endswith(f\"\\n{im_start}assistant\\n{im_end}\")\n prompt = prompt[: -len(f'{im_end}')]\n return prompt\n\n\ndef text_completion(input_text: str, stop_words) -> str: # 作为一个文本续写模型来使用\n im_end = '<|im_end|>'\n if im_end not in stop_words:\n stop_words = stop_words + [im_end]\n stop_words_ids = [tokenizer.encode(w) for w in stop_words]\n\n # TODO: 增加流式输出的样例实现\n input_ids = torch.tensor([tokenizer.encode(input_text)]).to(model.device)\n output = model.generate(input_ids, stop_words_ids=stop_words_ids)\n output = output.tolist()[0]\n output = tokenizer.decode(output, errors=\"ignore\")\n assert output.startswith(input_text)\n output = output[len(input_text) :].replace('<|endoftext|>', '').replace(im_end, '')\n\n for stop_str in stop_words:\n idx = output.find(stop_str)\n if idx != -1:\n output = output[: idx + len(stop_str)]\n return output # 续写 input_text 的结果，不包含 input_text 的内容\n\n\ndef parse_latest_plugin_call(text):\n plugin_name, plugin_args = '', ''\n i = text.rfind('\\nAction:')\n j = text.rfind('\\nAction Input:')\n k = text.rfind('\\nObservation:')\n if 0 <= i < j: # If the text has `Action` and `Action input`,\n if k < j: # but does not contain `Observation`,\n # then it is likely that `Observation` is ommited by the LLM,\n # because the output text may have discarded the stop word.\n text = text.rstrip() + '\\nObservation:' # Add it back.\n k = text.rfind('\\nObservation:')\n plugin_name = text[i + len('\\nAction:') : j].strip()\n plugin_args = text[j + len('\\nAction Input:') : k].strip()\n text = text[:k]\n return plugin_name, plugin_args, text\n\n\n#\n# 输入：\n# plugin_name: 需要调用的插件代号，对应 name_for_model。\n# plugin_args：插件的输入参数，是一个 dict，dict 的 key、value 分别为参数名、参数值。\n# 输出：\n# 插件的返回结果，需要是字符串。\n# 即使原本是 JSON 输出，也请 json.dumps(..., ensure_ascii=False) 成字符串。\n#\ndef call_plugin(plugin_name: str, plugin_args: str) -> str:\n #\n # 请开发者自行完善这部分内容。这里的参考实现仅是 demo 用途，非生产用途。\n #\n if plugin_name == 'google_search':\n # 使用 SerpAPI 需要在这里填入您的 SERPAPI_API_KEY！\n os.environ[\"SERPAPI_API_KEY\"] = os.getenv(\"SERPAPI_API_KEY\", default='')\n from langchain import SerpAPIWrapper\n\n return SerpAPIWrapper().run(json5.loads(plugin_args)['search_query'])\n elif plugin_name == 'image_gen':\n import urllib.parse\n\n prompt = json5.loads(plugin_args)[\"prompt\"]\n prompt = urllib.parse.quote(prompt)\n return json.dumps({'image_url': f'https://image.pollinations.ai/prompt/{prompt}'}, ensure_ascii=False)\n else:\n raise NotImplementedError\n\n\ndef test():\n tools = [\n {\n 'name_for_human': '谷歌搜索',\n 'name_for_model': 'google_search',\n 'description_for_model': '谷歌搜索是一个通用搜索引擎，可用于访问互联网、查询百科知识、了解时事新闻等。',\n 'parameters': [\n {\n 'name': 'search_query',\n 'description': '搜索关键词或短语',\n 'required': True,\n 'schema': {'type': 'string'},\n }\n ],\n },\n {\n 'name_for_human': '文生图',\n 'name_for_model': 'image_gen',\n 'description_for_model': '文生图是一个AI绘画（图像生成）服务，输入文本描述，返回根据文本作画得到的图片的URL',\n 'parameters': [\n {\n 'name': 'prompt',\n 'description': '英文关键词，描述了希望图像具有什么内容',\n 'required': True,\n 'schema': {'type': 'string'},\n }\n ],\n },\n ]\n history = []\n for query in ['你好', '搜索一下谁是周杰伦', '再搜下他老婆是谁', '给我画个可爱的小猫吧，最好是黑猫']:\n print(f\"User's Query:\\n{query}\\n\")\n response, history = llm_with_plugin(prompt=query, history=history, list_of_plugin_info=tools)\n print(f\"Qwen's Response:\\n{response}\\n\")\n\n\nif __name__ == \"__main__\":\n test()\n\n\"\"\"如果执行成功，在终端下应当能看到如下输出：\nUser's Query:\n你好\n\nQwen's Response:\nThought: 提供的工具对回答该问题帮助较小，我将不使用工具直接作答。\nFinal Answer: 你好！很高兴见到你。有什么我可以帮忙的吗？\n\nUser's Query:\n搜索一下谁是周杰伦\n\nQwen's Response:\nThought: 我应该使用Google搜索查找相关信息。\nAction: google_search\nAction Input: {\"search_query\": \"周杰伦\"}\nObservation: Jay Chou is a Taiwanese singer, songwriter, record producer, rapper, actor, television personality, and businessman.\nThought: I now know the final answer.\nFinal Answer: 周杰伦（Jay Chou）是一位来自台湾的歌手、词曲创作人、音乐制作人、说唱歌手、演员、电视节目主持人和企业家。他以其独特的音乐风格和才华在华语乐坛享有很高的声誉。\n\nUser's Query:\n再搜下他老婆是谁\n\nQwen's Response:\nThought: 我应该使用Google搜索查找相关信息。\nAction: google_search\nAction Input: {\"search_query\": \"周杰伦 老婆\"}\nObservation: Hannah Quinlivan\nThought: I now know the final answer.\nFinal Answer: 周杰伦的老婆是Hannah Quinlivan，她是一位澳大利亚籍的模特和演员。两人于2015年结婚，并育有一子。\n\nUser's Query:\n给我画个可爱的小猫吧，最好是黑猫\n\nQwen's Response:\nThought: 我应该使用文生图API来生成一张可爱的小猫图片。\nAction: image_gen\nAction Input: {\"prompt\": \"cute black cat\"}\nObservation: {\"image_url\": \"https://image.pollinations.ai/prompt/cute%20black%20cat\"}\nThought: I now know the final answer.\nFinal Answer: 生成的可爱小猫图片的URL为https://image.pollinations.ai/prompt/cute%20black%20cat。你可以点击这个链接查看图片。\n\"\"\"\n" }, { "path": "examples/react_prompt.md", "content": "# ReAct Prompting 示例\n\n本文档将介绍如何用 ReAct Prompting 技术命令千问使用工具。\n\n本文档主要基本的原理概念介绍，并在文末附上了一些具体实现相关的 FAQ，但不含被调用插件的实际实现。如果您更喜欢一边调试实际可执行的代码、一边理解原理，可以转而阅读整合了 LangChain 常用工具的这个 [ipython notebook](https://github.com/QwenLM/Qwen-7B/blob/main/examples/langchain_tooluse.ipynb)。\n\n此外，本文档和前述的 ipython notebook 都仅介绍单轮对话的实现。如果想了解多轮对话下的实现，可参见 [react_demo.py](https://github.com/QwenLM/Qwen-7B/blob/main/examples/react_demo.py)。\n\n## 准备工作一：样例问题、样例工具\n\n假设我们有如下的一个适合用工具处理的 query，以及有夸克搜索、通义万相文生图这两个工具：\n\n```py\nquery = '现在给我画个五彩斑斓的黑。'\n\nTOOLS = [\n {\n 'name_for_human':\n '夸克搜索',\n 'name_for_model':\n 'quark_search',\n 'description_for_model':\n '夸克搜索是一个通用搜索引擎，可用于访问互联网、查询百科知识、了解时事新闻等。',\n 'parameters': [{\n 'name': 'search_query',\n 'description': '搜索关键词或短语',\n 'required': True,\n 'schema': {\n 'type': 'string'\n },\n }],\n },\n {\n 'name_for_human':\n '通义万相',\n 'name_for_model':\n 'image_gen',\n 'description_for_model':\n '通义万相是一个AI绘画（图像生成）服务，输入文本描述，返回根据文本作画得到的图片的URL',\n 'parameters': [{\n 'name': 'query',\n 'description': '中文关键词，描述了希望图像具有什么内容',\n 'required': True,\n 'schema': {\n 'type': 'string'\n },\n }],\n },\n]\n```\n\n## 准备工作二：ReAct 模版\n\n我们将使用如下的 ReAct prompt 模版来激发千问使用工具的能力。\n\n```py\nTOOL_DESC = \"\"\"{name_for_model}: Call this tool to interact with the {name_for_human} API. What is the {name_for_human} API useful for? {description_for_model} Parameters: {parameters} Format the arguments as a JSON object.\"\"\"\n\nREACT_PROMPT = \"\"\"Answer the following questions as best you can. You have access to the following tools:\n\n{tool_descs}\n\nUse the following format:\n\nQuestion: the input question you must answer\nThought: you should always think about what to do\nAction: the action to take, should be one of [{tool_names}]\nAction Input: the input to the action\nObservation: the result of the action\n... (this Thought/Action/Action Input/Observation can be repeated zero or more times)\nThought: I now know the final answer\nFinal Answer: the final answer to the original input question\n\nBegin!\n\nQuestion: {query}\"\"\"\n```\n\n## 步骤一：让千问判断要调用什么工具、生成工具入参\n\n首先我们需要根据 ReAct prompt 模版、query、工具的信息构建 prompt：\n\n```py\ntool_descs = []\ntool_names = []\nfor info in TOOLS:\n tool_descs.append(\n TOOL_DESC.format(\n name_for_model=info['name_for_model'],\n name_for_human=info['name_for_human'],\n description_for_model=info['description_for_model'],\n parameters=json.dumps(\n info['parameters'], ensure_ascii=False),\n )\n )\n tool_names.append(info['name_for_model'])\ntool_descs = '\\n\\n'.join(tool_descs)\ntool_names = ','.join(tool_names)\n\nprompt = REACT_PROMPT.format(tool_descs=tool_descs, tool_names=tool_names, query=query)\nprint(prompt)\n```\n\n打印出来的、构建好的 prompt 如下：\n\n```\nAnswer the following questions as best you can. You have access to the following tools:\n\nquark_search: Call this tool to interact with the 夸克搜索 API. What is the 夸克搜索 API useful for? 夸克搜索是一个通用搜索引擎，可用于访问互联网、查询百科知识、了解时事新闻等。 Parameters: [{\"name\": \"search_query\", \"description\": \"搜索关键词或短语\", \"required\": true, \"schema\": {\"type\": \"string\"}}] Format the arguments as a JSON object.\n\nimage_gen: Call this tool to interact with the 通义万相 API. What is the 通义万相 API useful for? 通义万相是一个AI绘画（图像生成）服务，输入文本描述，返回根据文本作画得到的图片的URL Parameters: [{\"name\": \"query\", \"description\": \"中文关键词，描述了希望图像具有什么内容\", \"required\": true, \"schema\": {\"type\": \"string\"}}] Format the arguments as a JSON object.\n\nUse the following format:\n\nQuestion: the input question you must answer\nThought: you should always think about what to do\nAction: the action to take, should be one of [quark_search,image_gen]\nAction Input: the input to the action\nObservation: the result of the action\n... (this Thought/Action/Action Input/Observation can be repeated zero or more times)\nThought: I now know the final answer\nFinal Answer: the final answer to the original input question\n\nBegin!\n\nQuestion: 现在给我画个五彩斑斓的黑。\n```\n\n将这个 prompt 送入千问，并记得设置 \"Observation\" 为 stop word （见本文末尾的 FAQ）—— 即让千问在预测到要生成的下一个词是 \"Observation\" 时马上停止生成 —— 则千问在得到这个 prompt 后会生成如下的结果：\n\n![](../assets/react_tutorial_001.png)\n\n```\nThought: 我应该使用通义万相API来生成一张五彩斑斓的黑的图片。\nAction: image_gen\nAction Input: {\"query\": \"五彩斑斓的黑\"}\n```\n\n在得到这个结果后，调用千问的开发者可以通过简单的解析提取出 `{\"query\": \"五彩斑斓的黑\"}` 并基于这个解析结果调用文生图服务 —— 这部分逻辑需要开发者自行实现，或者也可以使用千问商业版，商业版本将内部集成相关逻辑。\n\n## 步骤二：让千问根据插件返回结果继续作答\n\n让我们假设文生图插件返回了如下结果：\n\n```\n{\"status_code\": 200, \"request_id\": \"3d894da2-0e26-9b7c-bd90-102e5250ae03\", \"code\": null, \"message\": \"\", \"output\": {\"task_id\": \"2befaa09-a8b3-4740-ada9-4d00c2758b05\", \"task_status\": \"SUCCEEDED\", \"results\": [{\"url\": \"https://dashscope-result-sh.oss-cn-shanghai.aliyuncs.com/1e5e2015/20230801/1509/6b26bb83-469e-4c70-bff4-a9edd1e584f3-1.png\"}], \"task_metrics\": {\"TOTAL\": 1, \"SUCCEEDED\": 1, \"FAILED\": 0}}, \"usage\": {\"image_count\": 1}}\n```\n\n![](../assets/wanx_colorful_black.png)\n\n接下来，我们可以将之前首次请求千问时用的 prompt 和 调用文生图插件的结果拼接成如下的新 prompt：\n\n```\nAnswer the following questions as best you can. You have access to the following tools:\n\nquark_search: Call this tool to interact with the 夸克搜索 API. What is the 夸克搜索 API useful for? 夸克搜索是一个通用搜索引擎，可用于访问互联网、查询百科知识、了解时事新闻等。 Parameters: [{\"name\": \"search_query\", \"description\": \"搜索关键词或短语\", \"required\": true, \"schema\": {\"type\": \"string\"}}] Format the arguments as a JSON object.\n\nimage_gen: Call this tool to interact with the 通义万相 API. What is the 通义万相 API useful for? 通义万相是一个AI绘画（图像生成）服务，输入文本描述，返回根据文本作画得到的图片的URL Parameters: [{\"name\": \"query\", \"description\": \"中文关键词，描述了希望图像具有什么内容\", \"required\": true, \"schema\": {\"type\": \"string\"}}] Format the arguments as a JSON object.\n\nUse the following format:\n\nQuestion: the input question you must answer\nThought: you should always think about what to do\nAction: the action to take, should be one of [quark_search,image_gen]\nAction Input: the input to the action\nObservation: the result of the action\n... (this Thought/Action/Action Input/Observation can be repeated zero or more times)\nThought: I now know the final answer\nFinal Answer: the final answer to the original input question\n\nBegin!\n\nQuestion: 现在给我画个五彩斑斓的黑。\nThought: 我应该使用通义万相API来生成一张五彩斑斓的黑的图片。\nAction: image_gen\nAction Input: {\"query\": \"五彩斑斓的黑\"}\nObservation: {\"status_code\": 200, \"request_id\": \"3d894da2-0e26-9b7c-bd90-102e5250ae03\", \"code\": null, \"message\": \"\", \"output\": {\"task_id\": \"2befaa09-a8b3-4740-ada9-4d00c2758b05\", \"task_status\": \"SUCCEEDED\", \"results\": [{\"url\": \"https://dashscope-result-sh.oss-cn-shanghai.aliyuncs.com/1e5e2015/20230801/1509/6b26bb83-469e-4c70-bff4-a9edd1e584f3-1.png\"}], \"task_metrics\": {\"TOTAL\": 1, \"SUCCEEDED\": 1, \"FAILED\": 0}}, \"usage\": {\"image_count\": 1}}\n```\n\n用这个新的拼接了文生图插件结果的新 prompt 去调用千问，将得到如下的最终回复：\n\n![](../assets/react_tutorial_002.png)\n\n```\nThought: 我已经成功使用通义万相API生成了一张五彩斑斓的黑的图片。\nFinal Answer: 我已经成功使用通义万相API生成了一张五彩斑斓的黑的图片https://dashscope-result-sh.oss-cn-shanghai.aliyuncs.com/1e5e2015/20230801/1509/6b26bb83-469e-4c70-bff4-a9edd1e584f3-1.png。\n```\n\n虽然对于文生图来说，这个第二次调用千问的步骤显得多余。但是对于搜索插件、代码执行插件、计算器插件等别的插件来说，这个第二次调用千问的步骤给了千问提炼、总结插件返回结果的机会。\n\n## FAQ\n\n**怎么配置 \"Observation\" 这个 stop word？**\n\n通过 chat 接口的 stop_words_ids 指定：\n```py\nreact_stop_words = [\n # tokenizer.encode('Observation'), # [37763, 367]\n tokenizer.encode('Observation:'), # [37763, 367, 25]\n tokenizer.encode('Observation:\\n'), # [37763, 367, 510]\n]\nresponse, history = model.chat(\n tokenizer, query, history,\n stop_words_ids=react_stop_words # 此接口用于增加 stop words\n)\n```\n\n如果报错称不存在 stop_words_ids 此参数，可能是因为您用了老的代码，请重新执行 from_pretrained 拉取新的代码和模型。\n\n需要注意的是，当前的 tokenizer 对 `\\n` 有一系列较复杂的聚合操作。比如例子中的`:\\n`这两个字符便被聚合成了一个 token。因此配置 stop words 需要非常细致地预估 tokenizer 的行为。\n\n**对 top_p 等推理参数有调参建议吗？**\n\n通常来讲，较低的 top_p 会有更高的准确度，但会牺牲回答的多样性、且更易出现重复某个词句的现象。\n\n可以按如下方式调整 top_p 为 0.5：\n```py\nmodel.generation_config.top_p = 0.5\n```\n\n特别的，可以用如下方式关闭 top-p sampling，改用 greedy sampling，效果上相当于 top_p=0 或 temperature=0：\n```py\nmodel.generation_config.do_sample = False # greedy decoding\n```\n\n此外，我们在 `model.chat()` 接口也提供了调整 top_p 等参数的接口。\n\n**有解析Action、Action Input的参考代码吗？**\n\n有的，可以参考：\n```py\ndef parse_latest_plugin_call(text: str) -> Tuple[str, str]:\n i = text.rfind('\\nAction:')\n j = text.rfind('\\nAction Input:')\n k = text.rfind('\\nObservation:')\n if 0 <= i < j: # If the text has `Action` and `Action input`,\n if k < j: # but does not contain `Observation`,\n # then it is likely that `Observation` is ommited by the LLM,\n # because the output text may have discarded the stop word.\n text = text.rstrip() + '\\nObservation:' # Add it back.\n k = text.rfind('\\nObservation:')\n if 0 <= i < j < k:\n plugin_name = text[i + len('\\nAction:'):j].strip()\n plugin_args = text[j + len('\\nAction Input:'):k].strip()\n return plugin_name, plugin_args\n return '', ''\n```\n\n此外，如果输出的 Action Input 内容是一段表示 JSON 对象的文本，我们建议使用 `json5` 包的 `json5.loads(...)` 方法加载。\n" }, { "path": "examples/system_prompt.md", "content": "# 系统指令 (System Prompts)\n\n## 什么是系统指令? (What is the System Prompts?)\n\n系统指令设定了AI助手的行为模式，例如人物设定、语言风格、任务模式、甚至针对具体问题的具体行为。\n\nSystem Propmts set the behavior mode of the AI assistant, such as character settings, language styles, task modes, and even specific behaviors for specific tasks.\n\n系统指令可以是一个广泛的人物设定，如“You are a helpful assistant”；也可以是一个十分详细的要求，如“拒绝回答所有代码相关的问题”。\n\nThe System Prompts can be a broad character setting, such as \"You are a helpful assistant\"; or it can be a very detailed request, such as \"Refuse to answer all code-related questions.\"\n\n系统指令为用户提供了一个易组织、上下文稳定的控制AI助手行为的方式，可以从多种角度定制属于你自己的AI助手。\n\nSystem Prompts provide users with an easy-to-organize, context-stable way to control the behavior of the AI assistant. You can customize your own AI assistant from multiple perspectives.\n\n系统指令需要在多轮对话中稳定，例如角色扮演类系统指令被设定后AI助手不应该在多轮对话中跳脱自身的设定。\n\nSystem Prompts need to be stable across multiple rounds of dialogue. For example, after a role-playing system prompt is set, the AI assistant should not escape its own settings in multiple rounds of dialogue.\n\n同时，模型也需要具有基于系统指令中对自身行为进行推理的能力。这两者都是为模型赋予跟随系统指令能力时需要克服的难点。\n\nAt the same time, the model also needs to have the ability to reason about its own behavior based on system prompts. Both of these are difficulties that need to be overcome when giving the model the ability to follow system prompts.\n\nQwen-1.8B-Chat 和 Qwen-72B-Chat在多样且存在多轮复杂交互的系统指令上进行了充分训练，使模型可以跟随多样的系统指令，实现上下文(in-context)中的模型定制化，进一步提升了通义千问的可扩展性。\n\nQwen-1.8-Chat and Qwen-72B-Chat have been fully trained on diverse system prompts with multiple rounds of complex interactions, so that they can follow a variety of system prompts and realize model customization in context, further improving the scalability of Qwen-chat.\n\n## 系统指令能做什么？ (What can System Prompts do?)\n\n### 角色扮演 Role Play\n\n在系统指令中告诉千问你需要它扮演的角色，即可沉浸式和该角色对话交流\n\nTell Qwen-Chat the role you want it to play in the System Prompt, and you can have an immersive conversation with that role.\n\n\n![](../assets/system_prompt_role_play.png)\n\n![](../assets/system_prompt_role_play_en.png)\n\n### 语言风格 Language Style\n\n\n简单调整千问的语言风格\n\nSimple adjustment of the Qwen-Chat's language style\n\n![](../assets/system_prompt_language_style.png)\n\n![](../assets/system_prompt_language_style_en.png)\n\n### 任务设定 Task Setting\n\n指定具体任务，打造处理专项任务的千问模型\n\nSetting specific tasks and creating a Qwen-Chat model to handle special tasks\n\n![](../assets/system_prompt_task_setting.png)\n\n![](../assets/system_prompt_task_setting_en.png)\n\n### 行为设定 Behavior Setting\n\n设定千问对具体任务的行为模式\n\nSet behavior patterns of Qwen-Chat for specific tasks\n\n![](../assets/system_prompt_behavior_setting.png)\n\n![](../assets/system_prompt_behavior_setting_en.png)\n\n## 代码示例 Example\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\nfrom transformers.generation import GenerationConfig\n\ntokenizer = AutoTokenizer.from_pretrained(\"Qwen/Qwen-1_8B-Chat\", trust_remote_code=True)\n\n# Only Qwen-72B-Chat and Qwen-1_8B-Chat has system prompt enhancement now.\nmodel = AutoModelForCausalLM.from_pretrained(\"Qwen/Qwen-1_8B-Chat\", device_map=\"auto\", trust_remote_code=True).eval()\n# model = AutoModelForCausalLM.from_pretrained(\"Qwen/Qwen-72B-Chat\", device_map=\"auto\", trust_remote_code=True).eval()\n\nresponse, _ = model.chat(tokenizer, \"你好呀\", history=None, system=\"请用二次元可爱语气和我说话\")\nprint(response)\n# 你好啊！我是一只可爱的二次元猫咪哦，不知道你有什么问题需要我帮忙解答吗？\n\nresponse, _ = model.chat(tokenizer, \"My colleague works diligently\", history=None, system=\"You will write beautiful compliments according to needs\")\nprint(response)\n# Your colleague is an outstanding worker! Their dedication and hard work are truly inspiring. They always go above and beyond to ensure that their tasks are completed on time and to the highest standard. I am lucky to have them as a colleague, and I know I can count on them to handle any challenge that comes their way.\n```" }, { "path": "examples/tokenizer_showcase.ipynb", "content": "{\n \"cells\": [\n {\n \"cell_type\": \"code\",\n \"execution_count\": 1,\n \"metadata\": {},\n \"outputs\": [\n {\n \"name\": \"stderr\",\n \"output_type\": \"stream\",\n \"text\": [\n \"None of PyTorch, TensorFlow >= 2.0, or Flax have been found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.\\n\"\n ]\n }\n ],\n \"source\": [\n \"from transformers import AutoTokenizer\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 2,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"tokenizer = AutoTokenizer.from_pretrained('Qwen/Qwen-7B', trust_remote_code=True)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"# Encode and Decode\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 3,\n \"metadata\": {},\n \"outputs\": [\n {\n \"data\": {\n \"text/plain\": [\n \"[1350, 492, 151643, 863, 151643]\"\n ]\n },\n \"execution_count\": 3,\n \"metadata\": {},\n \"output_type\": \"execute_result\"\n }\n ],\n \"source\": [\n \"# treat surface forms of special tokens as actual special tokens\\n\",\n \"# the default, but unsafe (to be compatible with other projects)\\n\",\n \"# the same as tokenizer.encode(\\\"print('<|endoftext|>')<|endoftext|>\\\", allowed_special='all', disallowed_special=())\\n\",\n \"tokenizer.encode(\\\"print('<|endoftext|>')<|endoftext|>\\\")\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 4,\n \"metadata\": {},\n \"outputs\": [\n {\n \"data\": {\n \"text/plain\": [\n \"\\\"print('<|endoftext|>')<|endoftext|>\\\"\"\n ]\n },\n \"execution_count\": 4,\n \"metadata\": {},\n \"output_type\": \"execute_result\"\n }\n ],\n \"source\": [\n \"tokenizer.decode([1350, 492, 151643, 863, 151643])\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 5,\n \"metadata\": {},\n \"outputs\": [\n {\n \"data\": {\n \"text/plain\": [\n \"[1350, 11146, 91, 8691, 723, 427, 91, 79865, 151643]\"\n ]\n },\n \"execution_count\": 5,\n \"metadata\": {},\n \"output_type\": \"execute_result\"\n }\n ],\n \"source\": [\n \"# treat texts just as texts, avoid injection attacks\\n\",\n \"tokenizer.encode(\\\"print('<|endoftext|>')\\\", allowed_special=set(), disallowed_special=()) + [tokenizer.eod_id]\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 6,\n \"metadata\": {},\n \"outputs\": [\n {\n \"data\": {\n \"text/plain\": [\n \"\\\"print('<|endoftext|>')<|endoftext|>\\\"\"\n ]\n },\n \"execution_count\": 6,\n \"metadata\": {},\n \"output_type\": \"execute_result\"\n }\n ],\n \"source\": [\n \"tokenizer.decode([1350, 11146, 91, 8691, 723, 427, 91, 79865, 151643])\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 7,\n \"metadata\": {},\n \"outputs\": [\n {\n \"ename\": \"ValueError\",\n \"evalue\": \"Encountered text corresponding to disallowed special token '<|endoftext|>'.\\nIf you want this text to be encoded as a special token, pass it to `allowed_special`, e.g. `allowed_special={'<|endoftext|>', ...}`.\\nIf you want this text to be encoded as normal text, disable the check for this token by passing `disallowed_special=(enc.special_tokens_set - {'<|endoftext|>'})`.\\nTo disable this check for all special tokens, pass `disallowed_special=()`.\\n\",\n \"output_type\": \"error\",\n \"traceback\": [\n \"\\u001b[1;31m---------------------------------------------------------------------------\\u001b[0m\",\n \"\\u001b[1;31mValueError\\u001b[0m Traceback (most recent call last)\",\n \"Cell \\u001b[1;32mIn[7], line 2\\u001b[0m\\n\\u001b[0;32m 1\\u001b[0m \\u001b[39m# treat texts just as texts, avoid injection attacks, and raise error if surface forms of special tokens are ever encountered\\u001b[39;00m\\n\\u001b[1;32m----> 2\\u001b[0m tokenizer\\u001b[39m.\\u001b[39;49mencode(\\u001b[39m\\\"\\u001b[39;49m\\u001b[39mprint(\\u001b[39;49m\\u001b[39m'\\u001b[39;49m\\u001b[39m<|endoftext|>\\u001b[39;49m\\u001b[39m'\\u001b[39;49m\\u001b[39m)\\u001b[39;49m\\u001b[39m\\\"\\u001b[39;49m, allowed_special\\u001b[39m=\\u001b[39;49m\\u001b[39mset\\u001b[39;49m(), disallowed_special\\u001b[39m=\\u001b[39;49m\\u001b[39m'\\u001b[39;49m\\u001b[39mall\\u001b[39;49m\\u001b[39m'\\u001b[39;49m) \\u001b[39m+\\u001b[39m [tokenizer\\u001b[39m.\\u001b[39meod_id]\\n\",\n \"File \\u001b[1;32mtransformers\\\\tokenization_utils_base.py:2348\\u001b[0m, in \\u001b[0;36mPreTrainedTokenizerBase.encode\\u001b[1;34m(self, text, text_pair, add_special_tokens, padding, truncation, max_length, stride, return_tensors, **kwargs)\\u001b[0m\\n\\u001b[0;32m 2311\\u001b[0m \\u001b[39m@add_end_docstrings\\u001b[39m(\\n\\u001b[0;32m 2312\\u001b[0m ENCODE_KWARGS_DOCSTRING,\\n\\u001b[0;32m 2313\\u001b[0m \\u001b[39m \\u001b[39m\\u001b[39m\\\"\\\"\\\"\\u001b[39;00m\\n\\u001b[1;32m (...)\\u001b[0m\\n\\u001b[0;32m 2331\\u001b[0m \\u001b[39m*\\u001b[39m\\u001b[39m*\\u001b[39mkwargs,\\n\\u001b[0;32m 2332\\u001b[0m ) \\u001b[39m-\\u001b[39m\\u001b[39m>\\u001b[39m List[\\u001b[39mint\\u001b[39m]:\\n\\u001b[0;32m 2333\\u001b[0m \\u001b[39m \\u001b[39m\\u001b[39m\\\"\\\"\\\"\\u001b[39;00m\\n\\u001b[0;32m 2334\\u001b[0m \\u001b[39m Converts a string to a sequence of ids (integer), using the tokenizer and vocabulary.\\u001b[39;00m\\n\\u001b[0;32m 2335\\u001b[0m \\n\\u001b[1;32m (...)\\u001b[0m\\n\\u001b[0;32m 2346\\u001b[0m \\u001b[39m method).\\u001b[39;00m\\n\\u001b[0;32m 2347\\u001b[0m \\u001b[39m \\\"\\\"\\\"\\u001b[39;00m\\n\\u001b[1;32m-> 2348\\u001b[0m encoded_inputs \\u001b[39m=\\u001b[39m \\u001b[39mself\\u001b[39m\\u001b[39m.\\u001b[39mencode_plus(\\n\\u001b[0;32m 2349\\u001b[0m text,\\n\\u001b[0;32m 2350\\u001b[0m text_pair\\u001b[39m=\\u001b[39mtext_pair,\\n\\u001b[0;32m 2351\\u001b[0m add_special_tokens\\u001b[39m=\\u001b[39madd_special_tokens,\\n\\u001b[0;32m 2352\\u001b[0m padding\\u001b[39m=\\u001b[39mpadding,\\n\\u001b[0;32m 2353\\u001b[0m truncation\\u001b[39m=\\u001b[39mtruncation,\\n\\u001b[0;32m 2354\\u001b[0m max_length\\u001b[39m=\\u001b[39mmax_length,\\n\\u001b[0;32m 2355\\u001b[0m stride\\u001b[39m=\\u001b[39mstride,\\n\\u001b[0;32m 2356\\u001b[0m return_tensors\\u001b[39m=\\u001b[39mreturn_tensors,\\n\\u001b[0;32m 2357\\u001b[0m \\u001b[39m*\\u001b[39m\\u001b[39m*\\u001b[39mkwargs,\\n\\u001b[0;32m 2358\\u001b[0m )\\n\\u001b[0;32m 2360\\u001b[0m \\u001b[39mreturn\\u001b[39;00m encoded_inputs[\\u001b[39m\\\"\\u001b[39m\\u001b[39minput_ids\\u001b[39m\\u001b[39m\\\"\\u001b[39m]\\n\",\n \"File \\u001b[1;32mtransformers\\\\tokenization_utils_base.py:2756\\u001b[0m, in \\u001b[0;36mPreTrainedTokenizerBase.encode_plus\\u001b[1;34m(self, text, text_pair, add_special_tokens, padding, truncation, max_length, stride, is_split_into_words, pad_to_multiple_of, return_tensors, return_token_type_ids, return_attention_mask, return_overflowing_tokens, return_special_tokens_mask, return_offsets_mapping, return_length, verbose, **kwargs)\\u001b[0m\\n\\u001b[0;32m 2746\\u001b[0m \\u001b[39m# Backward compatibility for 'truncation_strategy', 'pad_to_max_length'\\u001b[39;00m\\n\\u001b[0;32m 2747\\u001b[0m padding_strategy, truncation_strategy, max_length, kwargs \\u001b[39m=\\u001b[39m \\u001b[39mself\\u001b[39m\\u001b[39m.\\u001b[39m_get_padding_truncation_strategies(\\n\\u001b[0;32m 2748\\u001b[0m padding\\u001b[39m=\\u001b[39mpadding,\\n\\u001b[0;32m 2749\\u001b[0m truncation\\u001b[39m=\\u001b[39mtruncation,\\n\\u001b[1;32m (...)\\u001b[0m\\n\\u001b[0;32m 2753\\u001b[0m \\u001b[39m*\\u001b[39m\\u001b[39m*\\u001b[39mkwargs,\\n\\u001b[0;32m 2754\\u001b[0m )\\n\\u001b[1;32m-> 2756\\u001b[0m \\u001b[39mreturn\\u001b[39;00m \\u001b[39mself\\u001b[39m\\u001b[39m.\\u001b[39m_encode_plus(\\n\\u001b[0;32m 2757\\u001b[0m text\\u001b[39m=\\u001b[39mtext,\\n\\u001b[0;32m 2758\\u001b[0m text_pair\\u001b[39m=\\u001b[39mtext_pair,\\n\\u001b[0;32m 2759\\u001b[0m add_special_tokens\\u001b[39m=\\u001b[39madd_special_tokens,\\n\\u001b[0;32m 2760\\u001b[0m padding_strategy\\u001b[39m=\\u001b[39mpadding_strategy,\\n\\u001b[0;32m 2761\\u001b[0m truncation_strategy\\u001b[39m=\\u001b[39mtruncation_strategy,\\n\\u001b[0;32m 2762\\u001b[0m max_length\\u001b[39m=\\u001b[39mmax_length,\\n\\u001b[0;32m 2763\\u001b[0m stride\\u001b[39m=\\u001b[39mstride,\\n\\u001b[0;32m 2764\\u001b[0m is_split_into_words\\u001b[39m=\\u001b[39mis_split_into_words,\\n\\u001b[0;32m 2765\\u001b[0m pad_to_multiple_of\\u001b[39m=\\u001b[39mpad_to_multiple_of,\\n\\u001b[0;32m 2766\\u001b[0m return_tensors\\u001b[39m=\\u001b[39mreturn_tensors,\\n\\u001b[0;32m 2767\\u001b[0m return_token_type_ids\\u001b[39m=\\u001b[39mreturn_token_type_ids,\\n\\u001b[0;32m 2768\\u001b[0m return_attention_mask\\u001b[39m=\\u001b[39mreturn_attention_mask,\\n\\u001b[0;32m 2769\\u001b[0m return_overflowing_tokens\\u001b[39m=\\u001b[39mreturn_overflowing_tokens,\\n\\u001b[0;32m 2770\\u001b[0m return_special_tokens_mask\\u001b[39m=\\u001b[39mreturn_special_tokens_mask,\\n\\u001b[0;32m 2771\\u001b[0m return_offsets_mapping\\u001b[39m=\\u001b[39mreturn_offsets_mapping,\\n\\u001b[0;32m 2772\\u001b[0m return_length\\u001b[39m=\\u001b[39mreturn_length,\\n\\u001b[0;32m 2773\\u001b[0m verbose\\u001b[39m=\\u001b[39mverbose,\\n\\u001b[0;32m 2774\\u001b[0m \\u001b[39m*\\u001b[39m\\u001b[39m*\\u001b[39mkwargs,\\n\\u001b[0;32m 2775\\u001b[0m )\\n\",\n \"File \\u001b[1;32mtransformers\\\\tokenization_utils.py:649\\u001b[0m, in \\u001b[0;36mPreTrainedTokenizer._encode_plus\\u001b[1;34m(self, text, text_pair, add_special_tokens, padding_strategy, truncation_strategy, max_length, stride, is_split_into_words, pad_to_multiple_of, return_tensors, return_token_type_ids, return_attention_mask, return_overflowing_tokens, return_special_tokens_mask, return_offsets_mapping, return_length, verbose, **kwargs)\\u001b[0m\\n\\u001b[0;32m 640\\u001b[0m \\u001b[39mif\\u001b[39;00m return_offsets_mapping:\\n\\u001b[0;32m 641\\u001b[0m \\u001b[39mraise\\u001b[39;00m \\u001b[39mNotImplementedError\\u001b[39;00m(\\n\\u001b[0;32m 642\\u001b[0m \\u001b[39m\\\"\\u001b[39m\\u001b[39mreturn_offset_mapping is not available when using Python tokenizers. \\u001b[39m\\u001b[39m\\\"\\u001b[39m\\n\\u001b[0;32m 643\\u001b[0m \\u001b[39m\\\"\\u001b[39m\\u001b[39mTo use this feature, change your tokenizer to one deriving from \\u001b[39m\\u001b[39m\\\"\\u001b[39m\\n\\u001b[1;32m (...)\\u001b[0m\\n\\u001b[0;32m 646\\u001b[0m \\u001b[39m\\\"\\u001b[39m\\u001b[39mhttps://github.com/huggingface/transformers/pull/2674\\u001b[39m\\u001b[39m\\\"\\u001b[39m\\n\\u001b[0;32m 647\\u001b[0m )\\n\\u001b[1;32m--> 649\\u001b[0m first_ids \\u001b[39m=\\u001b[39m get_input_ids(text)\\n\\u001b[0;32m 650\\u001b[0m second_ids \\u001b[39m=\\u001b[39m get_input_ids(text_pair) \\u001b[39mif\\u001b[39;00m text_pair \\u001b[39mis\\u001b[39;00m \\u001b[39mnot\\u001b[39;00m \\u001b[39mNone\\u001b[39;00m \\u001b[39melse\\u001b[39;00m \\u001b[39mNone\\u001b[39;00m\\n\\u001b[0;32m 652\\u001b[0m \\u001b[39mreturn\\u001b[39;00m \\u001b[39mself\\u001b[39m\\u001b[39m.\\u001b[39mprepare_for_model(\\n\\u001b[0;32m 653\\u001b[0m first_ids,\\n\\u001b[0;32m 654\\u001b[0m pair_ids\\u001b[39m=\\u001b[39msecond_ids,\\n\\u001b[1;32m (...)\\u001b[0m\\n\\u001b[0;32m 668\\u001b[0m verbose\\u001b[39m=\\u001b[39mverbose,\\n\\u001b[0;32m 669\\u001b[0m )\\n\",\n \"File \\u001b[1;32mtransformers\\\\tokenization_utils.py:616\\u001b[0m, in \\u001b[0;36mPreTrainedTokenizer._encode_plus.<locals>.get_input_ids\\u001b[1;34m(text)\\u001b[0m\\n\\u001b[0;32m 614\\u001b[0m \\u001b[39mdef\\u001b[39;00m \\u001b[39mget_input_ids\\u001b[39m(text):\\n\\u001b[0;32m 615\\u001b[0m \\u001b[39mif\\u001b[39;00m \\u001b[39misinstance\\u001b[39m(text, \\u001b[39mstr\\u001b[39m):\\n\\u001b[1;32m--> 616\\u001b[0m tokens \\u001b[39m=\\u001b[39m \\u001b[39mself\\u001b[39m\\u001b[39m.\\u001b[39mtokenize(text, \\u001b[39m*\\u001b[39m\\u001b[39m*\\u001b[39mkwargs)\\n\\u001b[0;32m 617\\u001b[0m \\u001b[39mreturn\\u001b[39;00m \\u001b[39mself\\u001b[39m\\u001b[39m.\\u001b[39mconvert_tokens_to_ids(tokens)\\n\\u001b[0;32m 618\\u001b[0m \\u001b[39melif\\u001b[39;00m \\u001b[39misinstance\\u001b[39m(text, (\\u001b[39mlist\\u001b[39m, \\u001b[39mtuple\\u001b[39m)) \\u001b[39mand\\u001b[39;00m \\u001b[39mlen\\u001b[39m(text) \\u001b[39m>\\u001b[39m \\u001b[39m0\\u001b[39m \\u001b[39mand\\u001b[39;00m \\u001b[39misinstance\\u001b[39m(text[\\u001b[39m0\\u001b[39m], \\u001b[39mstr\\u001b[39m):\\n\",\n \"File \\u001b[1;32mtokenization_qwen.py:155\\u001b[0m, in \\u001b[0;36mQWenTokenizer.tokenize\\u001b[1;34m(self, text, allowed_special, disallowed_special, **kwargs)\\u001b[0m\\n\\u001b[0;32m 152\\u001b[0m text \\u001b[39m=\\u001b[39m unicodedata\\u001b[39m.\\u001b[39mnormalize(\\u001b[39m\\\"\\u001b[39m\\u001b[39mNFC\\u001b[39m\\u001b[39m\\\"\\u001b[39m, text)\\n\\u001b[0;32m 154\\u001b[0m \\u001b[39m# this implementation takes a detour: text -> token id -> token surface forms\\u001b[39;00m\\n\\u001b[1;32m--> 155\\u001b[0m \\u001b[39mfor\\u001b[39;00m t \\u001b[39min\\u001b[39;00m \\u001b[39mself\\u001b[39;49m\\u001b[39m.\\u001b[39;49mtokenizer\\u001b[39m.\\u001b[39;49mencode(\\n\\u001b[0;32m 156\\u001b[0m text, allowed_special\\u001b[39m=\\u001b[39;49mallowed_special, disallowed_special\\u001b[39m=\\u001b[39;49mdisallowed_special\\n\\u001b[0;32m 157\\u001b[0m ):\\n\\u001b[0;32m 158\\u001b[0m tokens\\u001b[39m.\\u001b[39mappend(\\u001b[39mself\\u001b[39m\\u001b[39m.\\u001b[39mdecoder[t])\\n\\u001b[0;32m 159\\u001b[0m \\u001b[39mreturn\\u001b[39;00m tokens\\n\",\n \"File \\u001b[1;32mtiktoken\\\\core.py:117\\u001b[0m, in \\u001b[0;36mEncoding.encode\\u001b[1;34m(self, text, allowed_special, disallowed_special)\\u001b[0m\\n\\u001b[0;32m 115\\u001b[0m disallowed_special \\u001b[39m=\\u001b[39m \\u001b[39mfrozenset\\u001b[39m(disallowed_special)\\n\\u001b[0;32m 116\\u001b[0m \\u001b[39mif\\u001b[39;00m match \\u001b[39m:=\\u001b[39m _special_token_regex(disallowed_special)\\u001b[39m.\\u001b[39msearch(text):\\n\\u001b[1;32m--> 117\\u001b[0m raise_disallowed_special_token(match\\u001b[39m.\\u001b[39;49mgroup())\\n\\u001b[0;32m 119\\u001b[0m \\u001b[39mtry\\u001b[39;00m:\\n\\u001b[0;32m 120\\u001b[0m \\u001b[39mreturn\\u001b[39;00m \\u001b[39mself\\u001b[39m\\u001b[39m.\\u001b[39m_core_bpe\\u001b[39m.\\u001b[39mencode(text, allowed_special)\\n\",\n \"File \\u001b[1;32mtiktoken\\\\core.py:337\\u001b[0m, in \\u001b[0;36mraise_disallowed_special_token\\u001b[1;34m(token)\\u001b[0m\\n\\u001b[0;32m 336\\u001b[0m \\u001b[39mdef\\u001b[39;00m \\u001b[39mraise_disallowed_special_token\\u001b[39m(token: \\u001b[39mstr\\u001b[39m) \\u001b[39m-\\u001b[39m\\u001b[39m>\\u001b[39m NoReturn:\\n\\u001b[1;32m--> 337\\u001b[0m \\u001b[39mraise\\u001b[39;00m \\u001b[39mValueError\\u001b[39;00m(\\n\\u001b[0;32m 338\\u001b[0m \\u001b[39mf\\u001b[39m\\u001b[39m\\\"\\u001b[39m\\u001b[39mEncountered text corresponding to disallowed special token \\u001b[39m\\u001b[39m{\\u001b[39;00mtoken\\u001b[39m!r}\\u001b[39;00m\\u001b[39m.\\u001b[39m\\u001b[39m\\\\n\\u001b[39;00m\\u001b[39m\\\"\\u001b[39m\\n\\u001b[0;32m 339\\u001b[0m \\u001b[39m\\\"\\u001b[39m\\u001b[39mIf you want this text to be encoded as a special token, \\u001b[39m\\u001b[39m\\\"\\u001b[39m\\n\\u001b[0;32m 340\\u001b[0m \\u001b[39mf\\u001b[39m\\u001b[39m\\\"\\u001b[39m\\u001b[39mpass it to `allowed_special`, e.g. `allowed_special=\\u001b[39m\\u001b[39m{{\\u001b[39;00m\\u001b[39m{\\u001b[39;00mtoken\\u001b[39m!r}\\u001b[39;00m\\u001b[39m, ...\\u001b[39m\\u001b[39m}}\\u001b[39;00m\\u001b[39m`.\\u001b[39m\\u001b[39m\\\\n\\u001b[39;00m\\u001b[39m\\\"\\u001b[39m\\n\\u001b[0;32m 341\\u001b[0m \\u001b[39mf\\u001b[39m\\u001b[39m\\\"\\u001b[39m\\u001b[39mIf you want this text to be encoded as normal text, disable the check for this token \\u001b[39m\\u001b[39m\\\"\\u001b[39m\\n\\u001b[0;32m 342\\u001b[0m \\u001b[39mf\\u001b[39m\\u001b[39m\\\"\\u001b[39m\\u001b[39mby passing `disallowed_special=(enc.special_tokens_set - \\u001b[39m\\u001b[39m{{\\u001b[39;00m\\u001b[39m{\\u001b[39;00mtoken\\u001b[39m!r}\\u001b[39;00m\\u001b[39m}}\\u001b[39;00m\\u001b[39m)`.\\u001b[39m\\u001b[39m\\\\n\\u001b[39;00m\\u001b[39m\\\"\\u001b[39m\\n\\u001b[0;32m 343\\u001b[0m \\u001b[39m\\\"\\u001b[39m\\u001b[39mTo disable this check for all special tokens, pass `disallowed_special=()`.\\u001b[39m\\u001b[39m\\\\n\\u001b[39;00m\\u001b[39m\\\"\\u001b[39m\\n\\u001b[0;32m 344\\u001b[0m )\\n\",\n \"\\u001b[1;31mValueError\\u001b[0m: Encountered text corresponding to disallowed special token '<|endoftext|>'.\\nIf you want this text to be encoded as a special token, pass it to `allowed_special`, e.g. `allowed_special={'<|endoftext|>', ...}`.\\nIf you want this text to be encoded as normal text, disable the check for this token by passing `disallowed_special=(enc.special_tokens_set - {'<|endoftext|>'})`.\\nTo disable this check for all special tokens, pass `disallowed_special=()`.\\n\"\n ]\n }\n ],\n \"source\": [\n \"# treat texts just as texts, avoid injection attacks, and raise error if surface forms of special tokens are ever encountered\\n\",\n \"tokenizer.encode(\\\"print('<|endoftext|>')\\\", allowed_special=set(), disallowed_special='all') + [tokenizer.eod_id]\\n\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [\n {\n \"data\": {\n \"text/plain\": [\n \"[151644, 1350, 11146, 91, 15460, 62, 15, 91, 79865, 151645, 151643]\"\n ]\n },\n \"execution_count\": 7,\n \"metadata\": {},\n \"output_type\": \"execute_result\"\n }\n ],\n \"source\": [\n \"# fine-grained control, just keep mind of this:\\n\",\n \"# allowed_special is treated as special tokens\\n\",\n \"# disallowed_special raise errors\\n\",\n \"# allowed_special has higher priority than disallowed_special\\n\",\n \"tokenizer.encode(\\\"<|im_start|>print('<|extra_0|>')<|im_end|>\\\", \\n\",\n \" allowed_special={'<|im_start|>', '<|im_end|>'}, \\n\",\n \" disallowed_special=['<|endoftext|>']) + [tokenizer.eod_id]\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [\n {\n \"data\": {\n \"text/plain\": [\n \"[151644, 1350, 492, 151646, 863, 151645, 151643]\"\n ]\n },\n \"execution_count\": 8,\n \"metadata\": {},\n \"output_type\": \"execute_result\"\n }\n ],\n \"source\": [\n \"tokenizer.encode(\\\"<|im_start|>print('<|extra_0|>')<|im_end|>\\\", \\n\",\n \" allowed_special={'<|im_start|>', '<|im_end|>', '<|extra_0|>'}, \\n\",\n \" disallowed_special=['<|endoftext|>']) + [tokenizer.eod_id]\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"# Special Token Management\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 8,\n \"metadata\": {},\n \"outputs\": [\n {\n \"name\": \"stderr\",\n \"output_type\": \"stream\",\n \"text\": [\n \"Using unk_token, but it is not set yet.\\n\"\n ]\n }\n ],\n \"source\": [\n \"# huggingface tokenizer has its own special token mechanism, so does tiktoken\\n\",\n \"# we only use the tiktoken mechanism for special tokens, which means many property of huggingface tokenizer will be None\\n\",\n \"tokenizer.unk_token\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 9,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"tokenizer.eos_token_id # use tokenizer.eod_id instead\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 10,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"tokenizer.pad_token_id \"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 11,\n \"metadata\": {},\n \"outputs\": [\n {\n \"data\": {\n \"text/plain\": [\n \"151646\"\n ]\n },\n \"execution_count\": 11,\n \"metadata\": {},\n \"output_type\": \"execute_result\"\n }\n ],\n \"source\": [\n \"# use one of the extras such as <|extra_0|>\\n\",\n \"tokenizer.special_tokens['<|extra_0|>']\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"# Utility Methods\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 12,\n \"metadata\": {},\n \"outputs\": [\n {\n \"data\": {\n \"text/plain\": [\n \"[b'print', b\\\"('<\\\", b'|', b'endo', b'ft', b'ext', b'|', b\\\">')\\\", '<|endoftext|>']\"\n ]\n },\n \"execution_count\": 12,\n \"metadata\": {},\n \"output_type\": \"execute_result\"\n }\n ],\n \"source\": [\n \"# special tokens are str, tokens are bytes (since tiktoken operates on the bytes level)\\n\",\n \"ids = [1350, 11146, 91, 8691, 723, 427, 91, 79865, 151643]\\n\",\n \"tokenizer.convert_ids_to_tokens(ids)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 13,\n \"metadata\": {},\n \"outputs\": [\n {\n \"data\": {\n \"text/plain\": [\n \"\\\"print('<|endoftext|>')<|endoftext|>\\\"\"\n ]\n },\n \"execution_count\": 13,\n \"metadata\": {},\n \"output_type\": \"execute_result\"\n }\n ],\n \"source\": [\n \"tokenizer.convert_tokens_to_string(tokenizer.convert_ids_to_tokens(ids))\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 14,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"ids = tokenizer.encode(\\\"<|im_start|>print('我是一只猫<|extra_0|>')\\\\n#喵喵喵<|im_end|>\\\", \\n\",\n \" allowed_special={'<|im_start|>', '<|im_end|>', '<|extra_0|>'}, \\n\",\n \" disallowed_special=['<|endoftext|>']) + [tokenizer.eod_id]\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 15,\n \"metadata\": {},\n \"outputs\": [\n {\n \"data\": {\n \"text/plain\": [\n \"['<|im_start|>',\\n\",\n \" b'print',\\n\",\n \" b\\\"('\\\",\\n\",\n \" b'\\\\xe6\\\\x88\\\\x91',\\n\",\n \" b'\\\\xe6\\\\x98\\\\xaf\\\\xe4\\\\xb8\\\\x80',\\n\",\n \" b'\\\\xe5\\\\x8f\\\\xaa',\\n\",\n \" b'\\\\xe7\\\\x8c\\\\xab',\\n\",\n \" '<|extra_0|>',\\n\",\n \" b\\\"')\\\\n\\\",\\n\",\n \" b'#',\\n\",\n \" b'\\\\xe5\\\\x96\\\\xb5',\\n\",\n \" b'\\\\xe5\\\\x96\\\\xb5',\\n\",\n \" b'\\\\xe5\\\\x96\\\\xb5',\\n\",\n \" '<|im_end|>',\\n\",\n \" '<|endoftext|>']\"\n ]\n },\n \"execution_count\": 15,\n \"metadata\": {},\n \"output_type\": \"execute_result\"\n }\n ],\n \"source\": [\n \"tokenizer.convert_ids_to_tokens(ids)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 16,\n \"metadata\": {},\n \"outputs\": [\n {\n \"data\": {\n \"text/plain\": [\n \"\\\"<|im_start|>print('我是一只猫<|extra_0|>')\\\\n#喵喵喵<|im_end|><|endoftext|>\\\"\"\n ]\n },\n \"execution_count\": 16,\n \"metadata\": {},\n \"output_type\": \"execute_result\"\n }\n ],\n \"source\": [\n \"tokenizer.convert_tokens_to_string(tokenizer.convert_ids_to_tokens(ids))\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 17,\n \"metadata\": {},\n \"outputs\": [\n {\n \"data\": {\n \"text/plain\": [\n \"'<|extra_204|>'\"\n ]\n },\n \"execution_count\": 17,\n \"metadata\": {},\n \"output_type\": \"execute_result\"\n }\n ],\n \"source\": [\n \"tokenizer._convert_id_to_token(len(tokenizer)-1)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 18,\n \"metadata\": {},\n \"outputs\": [\n {\n \"data\": {\n \"text/plain\": [\n \"151850\"\n ]\n },\n \"execution_count\": 18,\n \"metadata\": {},\n \"output_type\": \"execute_result\"\n }\n ],\n \"source\": [\n \"tokenizer._convert_token_to_id('<|extra_204|>')\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"# Vocabulary Expansion\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 19,\n \"metadata\": {},\n \"outputs\": [\n {\n \"data\": {\n \"text/plain\": [\n \"{'input_ids': [35946, 99639, 91680, 100472], 'token_type_ids': [0, 0, 0, 0], 'attention_mask': [1, 1, 1, 1]}\"\n ]\n },\n \"execution_count\": 19,\n \"metadata\": {},\n \"output_type\": \"execute_result\"\n }\n ],\n \"source\": [\n \"tokenizer(\\\"我是一只猫\\\")\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 20,\n \"metadata\": {},\n \"outputs\": [\n {\n \"data\": {\n \"text/plain\": [\n \"[99639, 91680, 100472]\"\n ]\n },\n \"execution_count\": 20,\n \"metadata\": {},\n \"output_type\": \"execute_result\"\n }\n ],\n \"source\": [\n \"tokenizer.encode(\\\"是一只猫\\\")\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 21,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"tokenizer = AutoTokenizer.from_pretrained('Qwen/Qwen-7B', trust_remote_code=True, extra_vocab_file=\\\"qwen_extra.tiktoken\\\")\\n\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 22,\n \"metadata\": {},\n \"outputs\": [\n {\n \"data\": {\n \"text/plain\": [\n \"151857\"\n ]\n },\n \"execution_count\": 22,\n \"metadata\": {},\n \"output_type\": \"execute_result\"\n }\n ],\n \"source\": [\n \"len(tokenizer)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 23,\n \"metadata\": {},\n \"outputs\": [\n {\n \"data\": {\n \"text/plain\": [\n \"{'input_ids': [151854], 'token_type_ids': [0], 'attention_mask': [1]}\"\n ]\n },\n \"execution_count\": 23,\n \"metadata\": {},\n \"output_type\": \"execute_result\"\n }\n ],\n \"source\": [\n \"tokenizer(\\\"我是一只猫\\\")\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 24,\n \"metadata\": {},\n \"outputs\": [\n {\n \"data\": {\n \"text/plain\": [\n \"'我是一只猫'\"\n ]\n },\n \"execution_count\": 24,\n \"metadata\": {},\n \"output_type\": \"execute_result\"\n }\n ],\n \"source\": [\n \"tokenizer.decode(tokenizer.encode(\\\"我是一只猫\\\"))\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 25,\n \"metadata\": {},\n \"outputs\": [\n {\n \"data\": {\n \"text/plain\": [\n \"[151853]\"\n ]\n },\n \"execution_count\": 25,\n \"metadata\": {},\n \"output_type\": \"execute_result\"\n }\n ],\n \"source\": [\n \"tokenizer.encode(\\\"是一只猫\\\")\"\n ]\n }\n ],\n \"metadata\": {\n \"kernelspec\": {\n \"display_name\": \"python3\",\n \"language\": \"python\",\n \"name\": \"python3\"\n },\n \"language_info\": {\n \"codemirror_mode\": {\n \"name\": \"ipython\",\n \"version\": 3\n },\n \"file_extension\": \".py\",\n \"mimetype\": \"text/x-python\",\n \"name\": \"python\",\n \"nbconvert_exporter\": \"python\",\n \"pygments_lexer\": \"ipython3\",\n \"version\": \"3.10.12\"\n },\n \"orig_nbformat\": 4\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 2\n}\n" }, { "path": "examples/transformers_agent.md", "content": "## 什么是HuggingFace Agent\n使用大模型作为Agent，仅需自然语言就可调用HuggingFace中的模型，目前支持两种模式：\n\n- run模式：单轮对话，没有上下文，单个prompt多tool组合调用能力好\n- chat模式：多轮对话，有上下文，单次调用能力好，可能需要多次prompt实现多tool组合调用\n> 详见官方文档：[Transformers Agents](https://huggingface.co/docs/transformers/transformers_agents)\n\n## 使用通义千问作为Agent\n### 安装依赖\n```\npip install transformers\n```\n### 构建QWenAgent\n以下代码便可实现QWenAgent：\n```python\nimport torch\nfrom transformers import AutoModelForCausalLM, AutoTokenizer, Agent\nfrom transformers.generation import GenerationConfig\n\n\nclass QWenAgent(Agent):\n \"\"\"\n Agent that uses QWen model and tokenizer to generate code.\n\n Args:\n chat_prompt_template (`str`, *optional*):\n Pass along your own prompt if you want to override the default template for the `chat` method. Can be the\n actual prompt template or a repo ID (on the Hugging Face Hub). The prompt should be in a file named\n `chat_prompt_template.txt` in this repo in this case.\n run_prompt_template (`str`, *optional*):\n Pass along your own prompt if you want to override the default template for the `run` method. Can be the\n actual prompt template or a repo ID (on the Hugging Face Hub). The prompt should be in a file named\n `run_prompt_template.txt` in this repo in this case.\n additional_tools ([`Tool`], list of tools or dictionary with tool values, *optional*):\n Any additional tools to include on top of the default ones. If you pass along a tool with the same name as\n one of the default tools, that default tool will be overridden.\n\n Example:\n\n ```py\n agent = QWenAgent()\n agent.run(\"Draw me a picture of rivers and lakes.\")\n ```\n \"\"\"\n def __init__(self, chat_prompt_template=None, run_prompt_template=None, additional_tools=None):\n checkpoint = \"Qwen/Qwen-7B-Chat\"\n self.tokenizer = AutoTokenizer.from_pretrained(checkpoint, trust_remote_code=True)\n self.model = AutoModelForCausalLM.from_pretrained(checkpoint, device_map=\"auto\", trust_remote_code=True).cuda().eval()\n self.model.generation_config = GenerationConfig.from_pretrained(checkpoint, trust_remote_code=True) # 可指定不同的生成长度、top_p等相关超参\n self.model.generation_config.do_sample = False # greedy\n \n super().__init__(\n chat_prompt_template=chat_prompt_template,\n run_prompt_template=run_prompt_template,\n additional_tools=additional_tools,\n )\n\n def generate_one(self, prompt, stop):\n # \"Human:\" 和 \"Assistant:\" 曾为通义千问的特殊保留字，需要替换为 \"_HUMAN_:\" 和 \"_ASSISTANT_:\"。这一问题将在未来版本修复。\n prompt = prompt.replace(\"Human:\", \"_HUMAN_:\").replace(\"Assistant:\", \"_ASSISTANT_:\")\n stop = [item.replace(\"Human:\", \"_HUMAN_:\").replace(\"Assistant:\", \"_ASSISTANT_:\") for item in stop]\n\n result, _ = self.model.chat(self.tokenizer, prompt, history=None)\n for stop_seq in stop:\n if result.endswith(stop_seq):\n result = result[: -len(stop_seq)]\n\n result = result.replace(\"_HUMAN_:\", \"Human:\").replace(\"_ASSISTANT_:\", \"Assistant:\")\n return result\n\n\nagent = QWenAgent()\nagent.run(\"Draw me a picture of rivers and lakes.\")\n```\n### 使用示例\n```python\nagent = QWenAgent()\nagent.run(\"generate an image of panda\", remote=True)\n```\n![](../assets/hfagent_run.png)\n![](../assets/hfagent_chat_1.png)\n![](../assets/hfagent_chat_2.png)\n> 更多玩法参考HuggingFace官方文档[Transformers Agents](https://huggingface.co/docs/transformers/transformers_agents)\n\n## Tools\n### Tools支持\nHuggingFace Agent官方14个tool：\n\n- **Document question answering**: given a document (such as a PDF) in image format, answer a question on this document (Donut)\n- **Text question answering**: given a long text and a question, answer the question in the text (Flan-T5)\n- **Unconditional image captioning**: Caption the image! (BLIP)\n- **Image question answering**: given an image, answer a question on this image (VILT)\n- **Image segmentation**: given an image and a prompt, output the segmentation mask of that prompt (CLIPSeg)\n- **Speech to text**: given an audio recording of a person talking, transcribe the speech into text (Whisper)\n- **Text to speech**: convert text to speech (SpeechT5)\n- **Zero-shot text classification**: given a text and a list of labels, identify to which label the text corresponds the most (BART)\n- **Text summarization**: summarize a long text in one or a few sentences (BART)\n- **Translation**: translate the text into a given language (NLLB)\n- **Text downloader**: to download a text from a web URL\n- **Text to image**: generate an image according to a prompt, leveraging stable diffusion\n- **Image transformation**: transforms an image\n- **Text to video**: generate a small video according to a prompt, leveraging damo-vilab\n### Tools模型部署\n部分工具涉及的模型HuggingFace已进行在线部署，仅需设置remote=True便可实现在线调用：\n> agent.run(xxx, remote=True)\n\nHuggingFace没有在线部署的模型会自动下载checkpoint进行本地inference\n网络原因偶尔连不上HuggingFace，请多次尝试\n" }, { "path": "examples/vllm_wrapper.py", "content": "from transformers import PreTrainedTokenizer, GenerationConfig, StoppingCriteriaList\r\nfrom typing import Optional, Callable, List, Tuple, Union\r\nimport copy\r\nimport torch\r\nfrom transformers import AutoTokenizer\r\nfrom transformers.generation.logits_process import LogitsProcessorList\r\nfrom packaging import version\r\n\r\n_ERROR_BAD_CHAT_FORMAT = \"\"\"\\\r\nWe detect you are probably using the pretrained model (rather than chat model) for chatting, since the chat_format in generation_config is not \"chatml\".\r\nIf you are directly using the model downloaded from Huggingface, please make sure you are using our \"Qwen/Qwen-7B-Chat\" Huggingface model (rather than \"Qwen/Qwen-7B\") when you call model.chat().\r\n我们检测到您可能在使用预训练模型（而非chat模型）进行多轮chat，因为您当前在generation_config指定的chat_format，并未设置为我们在对话中所支持的\"chatml\"格式。\r\n如果您在直接使用我们从Huggingface提供的模型，请确保您在调用model.chat()时，使用的是\"Qwen/Qwen-7B-Chat\"模型（而非\"Qwen/Qwen-7B\"预训练模型）。\r\n\"\"\"\r\n\r\nIMEND = \"<|im_end|>\"\r\nENDOFTEXT = \"<|endoftext|>\"\r\n\r\nHistoryType = List[Tuple[str, str]]\r\nTokensType = List[int]\r\nBatchTokensType = List[List[int]]\r\n\r\ndef get_stop_words_ids(chat_format, tokenizer):\r\n if chat_format == \"raw\":\r\n stop_words_ids = [tokenizer.encode(\"Human:\"), [tokenizer.eod_id]]\r\n elif chat_format == \"chatml\":\r\n stop_words_ids = [[tokenizer.im_end_id], [tokenizer.im_start_id]]\r\n else:\r\n raise NotImplementedError(f\"Unknown chat format {chat_format!r}\")\r\n return stop_words_ids\r\n\r\ndef make_context(\r\n tokenizer: PreTrainedTokenizer,\r\n query: str,\r\n history: List[Tuple[str, str]] = None,\r\n system: str = \"\",\r\n max_window_size: int = 6144,\r\n chat_format: str = \"chatml\",\r\n):\r\n if history is None:\r\n history = []\r\n\r\n if chat_format == \"chatml\":\r\n im_start, im_end = \"<|im_start|>\", \"<|im_end|>\"\r\n im_start_tokens = [tokenizer.im_start_id]\r\n im_end_tokens = [tokenizer.im_end_id]\r\n nl_tokens = tokenizer.encode(\"\\n\")\r\n\r\n def _tokenize_str(role, content):\r\n return f\"{role}\\n{content}\", tokenizer.encode(\r\n role, allowed_special=set()\r\n ) + nl_tokens + tokenizer.encode(content, allowed_special=set())\r\n\r\n system_text, system_tokens_part = _tokenize_str(\"system\", system)\r\n system_tokens = im_start_tokens + system_tokens_part + im_end_tokens\r\n\r\n raw_text = \"\"\r\n context_tokens = []\r\n\r\n for turn_query, turn_response in reversed(history):\r\n query_text, query_tokens_part = _tokenize_str(\"user\", turn_query)\r\n query_tokens = im_start_tokens + query_tokens_part + im_end_tokens\r\n response_text, response_tokens_part = _tokenize_str(\r\n \"assistant\", turn_response\r\n )\r\n response_tokens = im_start_tokens + response_tokens_part + im_end_tokens\r\n\r\n next_context_tokens = nl_tokens + query_tokens + nl_tokens + response_tokens\r\n prev_chat = (\r\n f\"\\n{im_start}{query_text}{im_end}\\n{im_start}{response_text}{im_end}\"\r\n )\r\n\r\n current_context_size = (\r\n len(system_tokens) + len(next_context_tokens) + len(context_tokens)\r\n )\r\n if current_context_size < max_window_size:\r\n context_tokens = next_context_tokens + context_tokens\r\n raw_text = prev_chat + raw_text\r\n else:\r\n break\r\n\r\n context_tokens = system_tokens + context_tokens\r\n raw_text = f\"{im_start}{system_text}{im_end}\" + raw_text\r\n context_tokens += (\r\n nl_tokens\r\n + im_start_tokens\r\n + _tokenize_str(\"user\", query)[1]\r\n + im_end_tokens\r\n + nl_tokens\r\n + im_start_tokens\r\n + tokenizer.encode(\"assistant\")\r\n + nl_tokens\r\n )\r\n raw_text += f\"\\n{im_start}user\\n{query}{im_end}\\n{im_start}assistant\\n\"\r\n\r\n elif chat_format == \"raw\":\r\n raw_text = query\r\n context_tokens = tokenizer.encode(raw_text)\r\n else:\r\n raise NotImplementedError(f\"Unknown chat format {chat_format!r}\")\r\n\r\n return raw_text, context_tokens\r\n\r\nclass vLLMWrapper:\r\n def __init__(self,\r\n model_dir: str,\r\n trust_remote_code: bool = True,\r\n tensor_parallel_size: int = 1,\r\n gpu_memory_utilization: float = 0.98,\r\n dtype: str = \"bfloat16\",\r\n **kwargs):\r\n\r\n if dtype not in (\"bfloat16\", \"float16\", \"float32\"):\r\n print(\"now not support {}!\".format(dtype))\r\n raise Exception\r\n\r\n # build generation_config\r\n self.generation_config = GenerationConfig.from_pretrained(model_dir, trust_remote_code=trust_remote_code)\r\n\r\n # build tokenizer\r\n self.tokenizer = AutoTokenizer.from_pretrained(model_dir, trust_remote_code=True)\r\n self.tokenizer.eos_token_id = self.generation_config.eos_token_id\r\n\r\n self.stop_words_ids = []\r\n\r\n from vllm import LLM\r\n import vllm\r\n if version.parse(vllm.__version__) >= version.parse(\"0.2.2\"):\r\n self.__vllm_support_repetition_penalty = True\r\n else:\r\n self.__vllm_support_repetition_penalty = False\r\n\r\n quantization = getattr(kwargs, 'quantization', None)\r\n\r\n self.model = LLM(model=model_dir,\r\n tokenizer=model_dir,\r\n tensor_parallel_size=tensor_parallel_size,\r\n trust_remote_code=trust_remote_code,\r\n quantization=quantization,\r\n gpu_memory_utilization=gpu_memory_utilization,\r\n dtype=dtype)\r\n\r\n for stop_id in get_stop_words_ids(self.generation_config.chat_format, self.tokenizer):\r\n self.stop_words_ids.extend(stop_id)\r\n self.stop_words_ids.extend([self.generation_config.eos_token_id])\r\n\r\n def chat(self,\r\n query: str,\r\n history: Optional[HistoryType],\r\n tokenizer: PreTrainedTokenizer = None,\r\n system: str = \"You are a helpful assistant.\",\r\n generation_config: Optional[GenerationConfig] = None,\r\n **kwargs):\r\n generation_config = generation_config if generation_config is not None else self.generation_config\r\n tokenizer = self.tokenizer if tokenizer is None else tokenizer\r\n\r\n assert generation_config.chat_format == 'chatml', _ERROR_BAD_CHAT_FORMAT\r\n if not self.__vllm_support_repetition_penalty and generation_config.repetition_penalty != 1:\r\n raise RuntimeError(\"The installed vLLM doesn't support repetition_penalty, please set ``model.generation_config.repetition_penalty = 1`` or install vllm>=0.2.2\")\r\n\r\n if history is None:\r\n history = []\r\n else:\r\n # make a copy of the user's input such that is is left untouched\r\n history = copy.deepcopy(history)\r\n\r\n extra_stop_words_ids = kwargs.get('stop_words_ids', None)\r\n if extra_stop_words_ids is None:\r\n extra_stop_words_ids = []\r\n\r\n max_window_size = kwargs.get('max_window_size', None)\r\n if max_window_size is None:\r\n max_window_size = generation_config.max_window_size\r\n\r\n from vllm.sampling_params import SamplingParams\r\n sampling_kwargs = {\r\n \"stop_token_ids\": self.stop_words_ids,\r\n \"early_stopping\": False,\r\n \"top_p\": generation_config.top_p,\r\n \"top_k\": -1 if generation_config.top_k == 0 else generation_config.top_k,\r\n \"temperature\": generation_config.temperature,\r\n \"max_tokens\": generation_config.max_new_tokens,\r\n \"repetition_penalty\": generation_config.repetition_penalty\r\n }\r\n if not self.__vllm_support_repetition_penalty:\r\n sampling_kwargs.pop(\"repetition_penalty\")\r\n sampling_params = SamplingParams(**sampling_kwargs)\r\n\r\n raw_text, context_tokens = make_context(\r\n self.tokenizer,\r\n query,\r\n history=history,\r\n system=system,\r\n max_window_size=max_window_size,\r\n chat_format=generation_config.chat_format,\r\n )\r\n\r\n req_outputs = self.model.generate([query],\r\n sampling_params=sampling_params,\r\n prompt_token_ids=[context_tokens])\r\n req_output = req_outputs[0]\r\n\r\n prompt_str = req_output.prompt\r\n prompt_ids = req_output.prompt_token_ids\r\n req_sample_output_ids = []\r\n req_sample_output_strs = []\r\n for sample in req_output.outputs:\r\n output_str = sample.text\r\n output_ids = sample.token_ids\r\n if IMEND in output_str:\r\n output_str = output_str[:-len(IMEND)]\r\n if ENDOFTEXT in output_str:\r\n output_str = output_str[:-len(ENDOFTEXT)]\r\n req_sample_output_ids.append(prompt_ids + output_ids)\r\n req_sample_output_strs.append(prompt_str + output_str)\r\n assert len(req_sample_output_strs) == 1\r\n response = req_sample_output_strs[0][len(prompt_str):]\r\n history.append((prompt_str, response))\r\n\r\n return response, history\r\n\r\nif __name__ == '__main__':\r\n\r\n model_dir = 'Qwen/Qwen-72B-Chat'\r\n tensor_parallel_size = 2\r\n\r\n model = vLLMWrapper(model_dir,\r\n tensor_parallel_size=tensor_parallel_size,\r\n )\r\n\r\n response, history = model.chat(query=\"你好\",\r\n history=None)\r\n print(response)\r\n response, history = model.chat(query=\"给我讲一个年轻人奋斗创业最终取得成功的故事。\",\r\n history=history)\r\n print(response)\r\n response, history = model.chat(query=\"给这个故事起一个标题\",\r\n history=history)\r\n print(response)\r\n" }, { "path": "finetune/ds_config_zero2.json", "content": "{\n \"fp16\": {\n \"enabled\": \"auto\",\n \"loss_scale\": 0,\n \"loss_scale_window\": 1000,\n \"initial_scale_power\": 16,\n \"hysteresis\": 2,\n \"min_loss_scale\": 1\n },\n \"bf16\": {\n \"enabled\": \"auto\"\n },\n \"optimizer\": {\n \"type\": \"AdamW\",\n \"params\": {\n \"lr\": \"auto\",\n \"betas\": \"auto\",\n \"eps\": \"auto\",\n \"weight_decay\": \"auto\"\n }\n },\n\n \"scheduler\": {\n \"type\": \"WarmupLR\",\n \"params\": {\n \"warmup_min_lr\": \"auto\",\n \"warmup_max_lr\": \"auto\",\n \"warmup_num_steps\": \"auto\"\n }\n },\n\n \"zero_optimization\": {\n \"stage\": 2,\n \"offload_optimizer\": {\n \"device\": \"none\",\n \"pin_memory\": true\n },\n \"allgather_partitions\": true,\n \"allgather_bucket_size\": 2e8,\n \"overlap_comm\": true,\n \"reduce_scatter\": true,\n \"reduce_bucket_size\": 2e8,\n \"contiguous_gradients\": true\n },\n\n \"gradient_accumulation_steps\": \"auto\",\n \"gradient_clipping\": \"auto\",\n \"steps_per_print\": 100,\n \"train_batch_size\": \"auto\",\n \"train_micro_batch_size_per_gpu\": \"auto\",\n \"wall_clock_breakdown\": false\n}" }, { "path": "finetune/ds_config_zero3.json", "content": "{\n \"fp16\": {\n \"enabled\": \"auto\",\n \"loss_scale\": 0,\n \"loss_scale_window\": 1000,\n \"initial_scale_power\": 16,\n \"hysteresis\": 2,\n \"min_loss_scale\": 1\n },\n \"bf16\": {\n \"enabled\": \"auto\"\n },\n \"optimizer\": {\n \"type\": \"AdamW\",\n \"params\": {\n \"lr\": \"auto\",\n \"betas\": \"auto\",\n \"eps\": \"auto\",\n \"weight_decay\": \"auto\"\n }\n },\n\n \"scheduler\": {\n \"type\": \"WarmupLR\",\n \"params\": {\n \"warmup_min_lr\": \"auto\",\n \"warmup_max_lr\": \"auto\",\n \"warmup_num_steps\": \"auto\"\n }\n },\n\n \"zero_optimization\": {\n \"stage\": 3,\n \"offload_optimizer\": {\n \"device\": \"none\",\n \"pin_memory\": true\n },\n \"offload_param\": {\n \"device\": \"none\",\n \"pin_memory\": true\n },\n \"overlap_comm\": true,\n \"contiguous_gradients\": true,\n \"sub_group_size\": 1e9,\n \"reduce_bucket_size\": \"auto\",\n \"stage3_prefetch_bucket_size\": \"auto\",\n \"stage3_param_persistence_threshold\": \"auto\",\n \"stage3_max_live_parameters\": 1e9,\n \"stage3_max_reuse_distance\": 1e9,\n \"stage3_gather_16bit_weights_on_model_save\": true\n },\n\n \"gradient_accumulation_steps\": \"auto\",\n \"gradient_clipping\": \"auto\",\n \"steps_per_print\": 100,\n \"train_batch_size\": \"auto\",\n \"train_micro_batch_size_per_gpu\": \"auto\",\n \"wall_clock_breakdown\": false\n}\n" }, { "path": "finetune/finetune_ds.sh", "content": "#!/bin/bash\nexport CUDA_DEVICE_MAX_CONNECTIONS=1\nDIR=`pwd`\n\n# Guide:\n# This script supports distributed training on multi-gpu workers (as well as single-worker training).\n# Please set the options below according to the comments.\n# For multi-gpu workers training, these options should be manually set for each worker.\n# After setting the options, please run the script on each worker.\n\n# Number of GPUs per GPU worker\nGPUS_PER_NODE=$(python -c 'import torch; print(torch.cuda.device_count())')\n\n# Number of GPU workers, for single-worker training, please set to 1\nNNODES=${NNODES:-1}\n\n# The rank of this worker, should be in {0, ..., WORKER_CNT-1}, for single-worker training, please set to 0\nNODE_RANK=${NODE_RANK:-0}\n\n# The ip address of the rank-0 worker, for single-worker training, please set to localhost\nMASTER_ADDR=${MASTER_ADDR:-localhost}\n\n# The port for communication\nMASTER_PORT=${MASTER_PORT:-6001}\n\nMODEL=\"Qwen/Qwen-7B\" # Set the path if you do not want to load from huggingface directly\n# ATTENTION: specify the path to your training data, which should be a json file consisting of a list of conversations.\n# See the section for finetuning in README for more information.\nDATA=\"path_to_data\"\n\nfunction usage() {\n echo '\nUsage: bash finetune/finetune_ds.sh [-m MODEL_PATH] [-d DATA_PATH]\n'\n}\n\nwhile [[ \"$1\" != \"\" ]]; do\n case $1 in\n -m | --model )\n shift\n MODEL=$1\n ;;\n -d | --data )\n shift\n DATA=$1\n ;;\n -h | --help )\n usage\n exit 0\n ;;\n * )\n echo \"Unknown argument ${1}\"\n exit 1\n ;;\n esac\n shift\ndone\n\nDISTRIBUTED_ARGS=\"\n --nproc_per_node $GPUS_PER_NODE \\\n --nnodes $NNODES \\\n --node_rank $NODE_RANK \\\n --master_addr $MASTER_ADDR \\\n --master_port $MASTER_PORT\n\"\n\ntorchrun $DISTRIBUTED_ARGS finetune.py \\\n --model_name_or_path $MODEL \\\n --data_path $DATA \\\n --bf16 True \\\n --output_dir output_qwen \\\n --num_train_epochs 5 \\\n --per_device_train_batch_size 1 \\\n --per_device_eval_batch_size 1 \\\n --gradient_accumulation_steps 16 \\\n --evaluation_strategy \"no\" \\\n --save_strategy \"steps\" \\\n --save_steps 1000 \\\n --save_total_limit 10 \\\n --learning_rate 1e-5 \\\n --weight_decay 0.1 \\\n --adam_beta2 0.95 \\\n --warmup_ratio 0.01 \\\n --lr_scheduler_type \"cosine\" \\\n --logging_steps 1 \\\n --report_to \"none\" \\\n --model_max_length 512 \\\n --gradient_checkpointing True \\\n --lazy_preprocess True \\\n --deepspeed finetune/ds_config_zero3.json\n" }, { "path": "finetune/finetune_lora_ds.sh", "content": "#!/bin/bash\nexport CUDA_DEVICE_MAX_CONNECTIONS=1\nDIR=`pwd`\n\n# Guide:\n# This script supports distributed training on multi-gpu workers (as well as single-worker training).\n# Please set the options below according to the comments.\n# For multi-gpu workers training, these options should be manually set for each worker.\n# After setting the options, please run the script on each worker.\n\n# Number of GPUs per GPU worker\nGPUS_PER_NODE=$(python -c 'import torch; print(torch.cuda.device_count())')\n\n# Number of GPU workers, for single-worker training, please set to 1\nNNODES=${NNODES:-1}\n\n# The rank of this worker, should be in {0, ..., WORKER_CNT-1}, for single-worker training, please set to 0\nNODE_RANK=${NODE_RANK:-0}\n\n# The ip address of the rank-0 worker, for single-worker training, please set to localhost\nMASTER_ADDR=${MASTER_ADDR:-localhost}\n\n# The port for communication\nMASTER_PORT=${MASTER_PORT:-6001}\n\nMODEL=\"Qwen/Qwen-7B\" # Set the path if you do not want to load from huggingface directly\n# ATTENTION: specify the path to your training data, which should be a json file consisting of a list of conversations.\n# See the section for finetuning in README for more information.\nDATA=\"path_to_data\"\nDS_CONFIG_PATH=\"finetune/ds_config_zero2.json\"\n\nfunction usage() {\n echo '\nUsage: bash finetune/finetune_lora_ds.sh [-m MODEL_PATH] [-d DATA_PATH] [--deepspeed DS_CONFIG_PATH]\n'\n}\n\nwhile [[ \"$1\" != \"\" ]]; do\n case $1 in\n -m | --model )\n shift\n MODEL=$1\n ;;\n -d | --data )\n shift\n DATA=$1\n ;;\n --deepspeed )\n shift\n DS_CONFIG_PATH=$1\n ;;\n -h | --help )\n usage\n exit 0\n ;;\n * )\n echo \"Unknown argument ${1}\"\n exit 1\n ;;\n esac\n shift\ndone\n\nDISTRIBUTED_ARGS=\"\n --nproc_per_node $GPUS_PER_NODE \\\n --nnodes $NNODES \\\n --node_rank $NODE_RANK \\\n --master_addr $MASTER_ADDR \\\n --master_port $MASTER_PORT\n\"\n\ntorchrun $DISTRIBUTED_ARGS finetune.py \\\n --model_name_or_path $MODEL \\\n --data_path $DATA \\\n --bf16 True \\\n --output_dir output_qwen \\\n --num_train_epochs 5 \\\n --per_device_train_batch_size 2 \\\n --per_device_eval_batch_size 1 \\\n --gradient_accumulation_steps 8 \\\n --evaluation_strategy \"no\" \\\n --save_strategy \"steps\" \\\n --save_steps 1000 \\\n --save_total_limit 10 \\\n --learning_rate 3e-4 \\\n --weight_decay 0.1 \\\n --adam_beta2 0.95 \\\n --warmup_ratio 0.01 \\\n --lr_scheduler_type \"cosine\" \\\n --logging_steps 1 \\\n --report_to \"none\" \\\n --model_max_length 512 \\\n --lazy_preprocess True \\\n --use_lora \\\n --gradient_checkpointing \\\n --deepspeed ${DS_CONFIG_PATH}\n" }, { "path": "finetune/finetune_lora_single_gpu.sh", "content": "#!/bin/bash\nexport CUDA_DEVICE_MAX_CONNECTIONS=1\n\nMODEL=\"Qwen/Qwen-7B\" # Set the path if you do not want to load from huggingface directly\n# ATTENTION: specify the path to your training data, which should be a json file consisting of a list of conversations.\n# See the section for finetuning in README for more information.\nDATA=\"path_to_data\"\n\nfunction usage() {\n echo '\nUsage: bash finetune/finetune_lora_single_gpu.sh [-m MODEL_PATH] [-d DATA_PATH]\n'\n}\n\nwhile [[ \"$1\" != \"\" ]]; do\n case $1 in\n -m | --model )\n shift\n MODEL=$1\n ;;\n -d | --data )\n shift\n DATA=$1\n ;;\n -h | --help )\n usage\n exit 0\n ;;\n * )\n echo \"Unknown argument ${1}\"\n exit 1\n ;;\n esac\n shift\ndone\n\nexport CUDA_VISIBLE_DEVICES=0\n\npython finetune.py \\\n --model_name_or_path $MODEL \\\n --data_path $DATA \\\n --bf16 True \\\n --output_dir output_qwen \\\n --num_train_epochs 5 \\\n --per_device_train_batch_size 2 \\\n --per_device_eval_batch_size 1 \\\n --gradient_accumulation_steps 8 \\\n --evaluation_strategy \"no\" \\\n --save_strategy \"steps\" \\\n --save_steps 1000 \\\n --save_total_limit 10 \\\n --learning_rate 3e-4 \\\n --weight_decay 0.1 \\\n --adam_beta2 0.95 \\\n --warmup_ratio 0.01 \\\n --lr_scheduler_type \"cosine\" \\\n --logging_steps 1 \\\n --report_to \"none\" \\\n --model_max_length 512 \\\n --lazy_preprocess True \\\n --gradient_checkpointing \\\n --use_lora\n\n# If you use fp16 instead of bf16, you should use deepspeed\n# --fp16 True --deepspeed finetune/ds_config_zero2.json\n" }, { "path": "finetune/finetune_qlora_ds.sh", "content": "#!/bin/bash\nexport CUDA_DEVICE_MAX_CONNECTIONS=1\nDIR=`pwd`\n\n# Guide:\n# This script supports distributed training on multi-gpu workers (as well as single-worker training).\n# Please set the options below according to the comments.\n# For multi-gpu workers training, these options should be manually set for each worker.\n# After setting the options, please run the script on each worker.\n\n# Number of GPUs per GPU worker\nGPUS_PER_NODE=$(python -c 'import torch; print(torch.cuda.device_count())')\n\n# Number of GPU workers, for single-worker training, please set to 1\nNNODES=${NNODES:-1}\n\n# The rank of this worker, should be in {0, ..., WORKER_CNT-1}, for single-worker training, please set to 0\nNODE_RANK=${NODE_RANK:-0}\n\n# The ip address of the rank-0 worker, for single-worker training, please set to localhost\nMASTER_ADDR=${MASTER_ADDR:-localhost}\n\n# The port for communication\nMASTER_PORT=${MASTER_PORT:-6001}\n\nMODEL=\"Qwen/Qwen-7B-Chat-Int4\" # Set the path if you do not want to load from huggingface directly\n# ATTENTION: specify the path to your training data, which should be a json file consisting of a list of conversations.\n# See the section for finetuning in README for more information.\nDATA=\"path_to_data\"\n\nfunction usage() {\n echo '\nUsage: bash finetune/finetune_qlora_ds.sh [-m MODEL_PATH] [-d DATA_PATH]\n'\n}\n\nwhile [[ \"$1\" != \"\" ]]; do\n case $1 in\n -m | --model )\n shift\n MODEL=$1\n ;;\n -d | --data )\n shift\n DATA=$1\n ;;\n -h | --help )\n usage\n exit 0\n ;;\n * )\n echo \"Unknown argument ${1}\"\n exit 1\n ;;\n esac\n shift\ndone\n\nDISTRIBUTED_ARGS=\"\n --nproc_per_node $GPUS_PER_NODE \\\n --nnodes $NNODES \\\n --node_rank $NODE_RANK \\\n --master_addr $MASTER_ADDR \\\n --master_port $MASTER_PORT\n\"\n\n# Remember to use --fp16 instead of --bf16 due to autogptq\ntorchrun $DISTRIBUTED_ARGS finetune.py \\\n --model_name_or_path $MODEL \\\n --data_path $DATA \\\n --fp16 True \\\n --output_dir output_qwen \\\n --num_train_epochs 5 \\\n --per_device_train_batch_size 2 \\\n --per_device_eval_batch_size 1 \\\n --gradient_accumulation_steps 8 \\\n --evaluation_strategy \"no\" \\\n --save_strategy \"steps\" \\\n --save_steps 1000 \\\n --save_total_limit 10 \\\n --learning_rate 3e-4 \\\n --weight_decay 0.1 \\\n --adam_beta2 0.95 \\\n --warmup_ratio 0.01 \\\n --lr_scheduler_type \"cosine\" \\\n --logging_steps 1 \\\n --report_to \"none\" \\\n --model_max_length 512 \\\n --lazy_preprocess True \\\n --use_lora \\\n --q_lora \\\n --gradient_checkpointing \\\n --deepspeed finetune/ds_config_zero2.json\n" }, { "path": "finetune/finetune_qlora_single_gpu.sh", "content": "#!/bin/bash\nexport CUDA_DEVICE_MAX_CONNECTIONS=1\nDIR=`pwd`\n\nMODEL=\"Qwen/Qwen-7B-Chat-Int4\" # Set the path if you do not want to load from huggingface directly\n# ATTENTION: specify the path to your training data, which should be a json file consisting of a list of conversations.\n# See the section for finetuning in README for more information.\nDATA=\"path_to_data\"\n\nfunction usage() {\n echo '\nUsage: bash finetune/finetune_qlora_single_gpu.sh [-m MODEL_PATH] [-d DATA_PATH]\n'\n}\n\nwhile [[ \"$1\" != \"\" ]]; do\n case $1 in\n -m | --model )\n shift\n MODEL=$1\n ;;\n -d | --data )\n shift\n DATA=$1\n ;;\n -h | --help )\n usage\n exit 0\n ;;\n * )\n echo \"Unknown argument ${1}\"\n exit 1\n ;;\n esac\n shift\ndone\n\nexport CUDA_VISIBLE_DEVICES=0\n\n# Remember to use --fp16 instead of --bf16 due to autogptq\npython finetune.py \\\n --model_name_or_path $MODEL \\\n --data_path $DATA \\\n --fp16 True \\\n --output_dir output_qwen \\\n --num_train_epochs 5 \\\n --per_device_train_batch_size 2 \\\n --per_device_eval_batch_size 1 \\\n --gradient_accumulation_steps 8 \\\n --evaluation_strategy \"no\" \\\n --save_strategy \"steps\" \\\n --save_steps 1000 \\\n --save_total_limit 10 \\\n --learning_rate 3e-4 \\\n --weight_decay 0.1 \\\n --adam_beta2 0.95 \\\n --warmup_ratio 0.01 \\\n --lr_scheduler_type \"cosine\" \\\n --logging_steps 1 \\\n --report_to \"none\" \\\n --model_max_length 512 \\\n --lazy_preprocess True \\\n --gradient_checkpointing \\\n --use_lora \\\n --q_lora \\\n --deepspeed finetune/ds_config_zero2.json\n" }, { "path": "finetune.py", "content": "# This code is based on the revised code from fastchat based on tatsu-lab/stanford_alpaca.\n\n\nfrom dataclasses import dataclass, field\nimport json\nimport math\nimport logging\nimport os\nfrom typing import Dict, Optional, List\nimport torch\nfrom torch.utils.data import Dataset\nfrom deepspeed import zero\nfrom deepspeed.runtime.zero.partition_parameters import ZeroParamStatus\nimport transformers\nfrom transformers import Trainer, GPTQConfig, deepspeed\nfrom transformers.trainer_pt_utils import LabelSmoother\nfrom peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training\nfrom accelerate.utils import DistributedType\n\n\nIGNORE_TOKEN_ID = LabelSmoother.ignore_index\n\n\n@dataclass\nclass ModelArguments:\n model_name_or_path: Optional[str] = field(default=\"Qwen/Qwen-7B\")\n\n\n@dataclass\nclass DataArguments:\n data_path: str = field(\n default=None, metadata={\"help\": \"Path to the training data.\"}\n )\n eval_data_path: str = field(\n default=None, metadata={\"help\": \"Path to the evaluation data.\"}\n )\n lazy_preprocess: bool = False\n\n\n@dataclass\nclass TrainingArguments(transformers.TrainingArguments):\n cache_dir: Optional[str] = field(default=None)\n optim: str = field(default=\"adamw_torch\")\n model_max_length: int = field(\n default=8192,\n metadata={\n \"help\": \"Maximum sequence length. Sequences will be right padded (and possibly truncated).\"\n },\n )\n use_lora: bool = False\n\n\n@dataclass\nclass LoraArguments:\n lora_r: int = 64\n lora_alpha: int = 16\n lora_dropout: float = 0.05\n lora_target_modules: List[str] = field(\n default_factory=lambda: [\"c_attn\", \"c_proj\", \"w1\", \"w2\"]\n )\n lora_weight_path: str = \"\"\n lora_bias: str = \"none\"\n q_lora: bool = False\n\n\ndef maybe_zero_3(param):\n if hasattr(param, \"ds_id\"):\n assert param.ds_status == ZeroParamStatus.NOT_AVAILABLE\n with zero.GatheredParameters([param]):\n param = param.data.detach().cpu().clone()\n else:\n param = param.detach().cpu().clone()\n return param\n\n\n# Borrowed from peft.utils.get_peft_model_state_dict\ndef get_peft_state_maybe_zero_3(named_params, bias):\n if bias == \"none\":\n to_return = {k: t for k, t in named_params if \"lora_\" in k}\n elif bias == \"all\":\n to_return = {k: t for k, t in named_params if \"lora_\" in k or \"bias\" in k}\n elif bias == \"lora_only\":\n to_return = {}\n maybe_lora_bias = {}\n lora_bias_names = set()\n for k, t in named_params:\n if \"lora_\" in k:\n to_return[k] = t\n bias_name = k.split(\"lora_\")[0] + \"bias\"\n lora_bias_names.add(bias_name)\n elif \"bias\" in k:\n maybe_lora_bias[k] = t\n for k, t in maybe_lora_bias:\n if bias_name in lora_bias_names:\n to_return[bias_name] = t\n else:\n raise NotImplementedError\n to_return = {k: maybe_zero_3(v) for k, v in to_return.items()}\n return to_return\n\n\nlocal_rank = None\n\ndef rank0_print(*args):\n if local_rank == 0:\n print(*args)\n\n\ndef safe_save_model_for_hf_trainer(trainer: transformers.Trainer, output_dir: str, bias=\"none\"):\n \"\"\"Collects the state dict and dump to disk.\"\"\"\n # check if zero3 mode enabled\n if deepspeed.is_deepspeed_zero3_enabled():\n state_dict = trainer.model_wrapped._zero3_consolidated_16bit_state_dict()\n else:\n if trainer.args.use_lora:\n state_dict = get_peft_state_maybe_zero_3(\n trainer.model.named_parameters(), bias\n )\n else:\n state_dict = trainer.model.state_dict()\n if trainer.args.should_save and trainer.args.local_rank == 0:\n trainer._save(output_dir, state_dict=state_dict)\n\n\ndef preprocess(\n sources,\n tokenizer: transformers.PreTrainedTokenizer,\n max_len: int,\n system_message: str = \"You are a helpful assistant.\"\n) -> Dict:\n roles = {\"user\": \"<|im_start|>user\", \"assistant\": \"<|im_start|>assistant\"}\n\n im_start = tokenizer.im_start_id\n im_end = tokenizer.im_end_id\n nl_tokens = tokenizer('\\n').input_ids\n _system = tokenizer('system').input_ids + nl_tokens\n _user = tokenizer('user').input_ids + nl_tokens\n _assistant = tokenizer('assistant').input_ids + nl_tokens\n\n # Apply prompt templates\n input_ids, targets = [], []\n for i, source in enumerate(sources):\n if roles[source[0][\"from\"]] != roles[\"user\"]:\n source = source[1:]\n\n input_id, target = [], []\n system = [im_start] + _system + tokenizer(system_message).input_ids + [im_end] + nl_tokens\n input_id += system\n target += [im_start] + [IGNORE_TOKEN_ID] * (len(system)-3) + [im_end] + nl_tokens\n assert len(input_id) == len(target)\n for j, sentence in enumerate(source):\n role = roles[sentence[\"from\"]]\n _input_id = tokenizer(role).input_ids + nl_tokens + \\\n tokenizer(sentence[\"value\"]).input_ids + [im_end] + nl_tokens\n input_id += _input_id\n if role == '<|im_start|>user':\n _target = [im_start] + [IGNORE_TOKEN_ID] * (len(_input_id)-3) + [im_end] + nl_tokens\n elif role == '<|im_start|>assistant':\n _target = [im_start] + [IGNORE_TOKEN_ID] * len(tokenizer(role).input_ids) + \\\n _input_id[len(tokenizer(role).input_ids)+1:-2] + [im_end] + nl_tokens\n else:\n raise NotImplementedError\n target += _target\n assert len(input_id) == len(target)\n input_id += [tokenizer.pad_token_id] * (max_len - len(input_id))\n target += [IGNORE_TOKEN_ID] * (max_len - len(target))\n input_ids.append(input_id[:max_len])\n targets.append(target[:max_len])\n input_ids = torch.tensor(input_ids, dtype=torch.int)\n targets = torch.tensor(targets, dtype=torch.int)\n\n return dict(\n input_ids=input_ids,\n labels=targets,\n attention_mask=input_ids.ne(tokenizer.pad_token_id),\n )\n\n\nclass SupervisedDataset(Dataset):\n \"\"\"Dataset for supervised fine-tuning.\"\"\"\n\n def __init__(self, raw_data, tokenizer: transformers.PreTrainedTokenizer, max_len: int):\n super(SupervisedDataset, self).__init__()\n\n rank0_print(\"Formatting inputs...\")\n sources = [example[\"conversations\"] for example in raw_data]\n data_dict = preprocess(sources, tokenizer, max_len)\n\n self.input_ids = data_dict[\"input_ids\"]\n self.labels = data_dict[\"labels\"]\n self.attention_mask = data_dict[\"attention_mask\"]\n\n def __len__(self):\n return len(self.input_ids)\n\n def __getitem__(self, i) -> Dict[str, torch.Tensor]:\n return dict(\n input_ids=self.input_ids[i],\n labels=self.labels[i],\n attention_mask=self.attention_mask[i],\n )\n\n\nclass LazySupervisedDataset(Dataset):\n \"\"\"Dataset for supervised fine-tuning.\"\"\"\n\n def __init__(self, raw_data, tokenizer: transformers.PreTrainedTokenizer, max_len: int):\n super(LazySupervisedDataset, self).__init__()\n self.tokenizer = tokenizer\n self.max_len = max_len\n\n rank0_print(\"Formatting inputs...Skip in lazy mode\")\n self.tokenizer = tokenizer\n self.raw_data = raw_data\n self.cached_data_dict = {}\n\n def __len__(self):\n return len(self.raw_data)\n\n def __getitem__(self, i) -> Dict[str, torch.Tensor]:\n if i in self.cached_data_dict:\n return self.cached_data_dict[i]\n\n ret = preprocess([self.raw_data[i][\"conversations\"]], self.tokenizer, self.max_len)\n ret = dict(\n input_ids=ret[\"input_ids\"][0],\n labels=ret[\"labels\"][0],\n attention_mask=ret[\"attention_mask\"][0],\n )\n self.cached_data_dict[i] = ret\n\n return ret\n\n\ndef make_supervised_data_module(\n tokenizer: transformers.PreTrainedTokenizer, data_args, max_len,\n) -> Dict:\n \"\"\"Make dataset and collator for supervised fine-tuning.\"\"\"\n dataset_cls = (\n LazySupervisedDataset if data_args.lazy_preprocess else SupervisedDataset\n )\n rank0_print(\"Loading data...\")\n\n train_json = json.load(open(data_args.data_path, \"r\"))\n train_dataset = dataset_cls(train_json, tokenizer=tokenizer, max_len=max_len)\n\n if data_args.eval_data_path:\n eval_json = json.load(open(data_args.eval_data_path, \"r\"))\n eval_dataset = dataset_cls(eval_json, tokenizer=tokenizer, max_len=max_len)\n else:\n eval_dataset = None\n\n return dict(train_dataset=train_dataset, eval_dataset=eval_dataset)\n\n\ndef train():\n global local_rank\n\n parser = transformers.HfArgumentParser(\n (ModelArguments, DataArguments, TrainingArguments, LoraArguments)\n )\n (\n model_args,\n data_args,\n training_args,\n lora_args,\n ) = parser.parse_args_into_dataclasses()\n\n # This serves for single-gpu qlora.\n if getattr(training_args, 'deepspeed', None) and int(os.environ.get(\"WORLD_SIZE\", 1))==1:\n training_args.distributed_state.distributed_type = DistributedType.DEEPSPEED\n\n local_rank = training_args.local_rank\n\n device_map = None\n world_size = int(os.environ.get(\"WORLD_SIZE\", 1))\n ddp = world_size != 1\n if lora_args.q_lora:\n device_map = {\"\": int(os.environ.get(\"LOCAL_RANK\") or 0)} if ddp else \"auto\"\n if len(training_args.fsdp) > 0 or deepspeed.is_deepspeed_zero3_enabled():\n logging.warning(\n \"FSDP or ZeRO3 are incompatible with QLoRA.\"\n )\n\n is_chat_model = 'chat' in model_args.model_name_or_path.lower()\n if (\n training_args.use_lora\n and not lora_args.q_lora\n and deepspeed.is_deepspeed_zero3_enabled()\n and not is_chat_model\n ):\n raise RuntimeError(\"ZeRO3 is incompatible with LoRA when finetuning on base model.\")\n\n model_load_kwargs = {\n 'low_cpu_mem_usage': not deepspeed.is_deepspeed_zero3_enabled(),\n }\n\n # Set RoPE scaling factor\n config = transformers.AutoConfig.from_pretrained(\n model_args.model_name_or_path,\n cache_dir=training_args.cache_dir,\n trust_remote_code=True,\n )\n config.use_cache = False\n\n # Load model and tokenizer\n model = transformers.AutoModelForCausalLM.from_pretrained(\n model_args.model_name_or_path,\n config=config,\n cache_dir=training_args.cache_dir,\n device_map=device_map,\n trust_remote_code=True,\n quantization_config=GPTQConfig(\n bits=4, disable_exllama=True\n )\n if training_args.use_lora and lora_args.q_lora\n else None,\n **model_load_kwargs,\n )\n tokenizer = transformers.AutoTokenizer.from_pretrained(\n model_args.model_name_or_path,\n cache_dir=training_args.cache_dir,\n model_max_length=training_args.model_max_length,\n padding_side=\"right\",\n use_fast=False,\n trust_remote_code=True,\n )\n tokenizer.pad_token_id = tokenizer.eod_id\n\n if training_args.use_lora:\n if lora_args.q_lora or is_chat_model:\n modules_to_save = None\n else:\n modules_to_save = [\"wte\", \"lm_head\"]\n lora_config = LoraConfig(\n r=lora_args.lora_r,\n lora_alpha=lora_args.lora_alpha,\n target_modules=lora_args.lora_target_modules,\n lora_dropout=lora_args.lora_dropout,\n bias=lora_args.lora_bias,\n task_type=\"CAUSAL_LM\",\n modules_to_save=modules_to_save # This argument serves for adding new tokens.\n )\n if lora_args.q_lora:\n model = prepare_model_for_kbit_training(\n model, use_gradient_checkpointing=training_args.gradient_checkpointing\n )\n\n model = get_peft_model(model, lora_config)\n\n # Print peft trainable params\n model.print_trainable_parameters()\n\n if training_args.gradient_checkpointing:\n model.enable_input_require_grads()\n\n # Load data\n data_module = make_supervised_data_module(\n tokenizer=tokenizer, data_args=data_args, max_len=training_args.model_max_length\n )\n\n # Start trainner\n trainer = Trainer(\n model=model, tokenizer=tokenizer, args=training_args, **data_module\n )\n\n trainer.train()\n trainer.save_state()\n\n safe_save_model_for_hf_trainer(trainer=trainer, output_dir=training_args.output_dir, bias=lora_args.lora_bias)\n\n\nif __name__ == \"__main__\":\n train()\n" }, { "path": "openai_api.py", "content": "# Requirement:\n# pip install \"openai<1.0\"\n# Usage:\n# python openai_api.py\n# Visit http://localhost:8000/docs for documents.\n\nimport base64\nimport copy\nimport json\nimport time\nfrom argparse import ArgumentParser\nfrom contextlib import asynccontextmanager\nfrom pprint import pprint\nfrom typing import Dict, List, Literal, Optional, Union\n\nimport torch\nimport uvicorn\nfrom fastapi import FastAPI, HTTPException\nfrom fastapi.middleware.cors import CORSMiddleware\nfrom pydantic import BaseModel, Field\nfrom sse_starlette.sse import EventSourceResponse\nfrom starlette.middleware.base import BaseHTTPMiddleware\nfrom starlette.requests import Request\nfrom starlette.responses import Response\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\nfrom transformers.generation import GenerationConfig\n\n\nclass BasicAuthMiddleware(BaseHTTPMiddleware):\n\n def __init__(self, app, username: str, password: str):\n super().__init__(app)\n self.required_credentials = base64.b64encode(\n f'{username}:{password}'.encode()).decode()\n\n async def dispatch(self, request: Request, call_next):\n authorization: str = request.headers.get('Authorization')\n if authorization:\n try:\n schema, credentials = authorization.split()\n if credentials == self.required_credentials:\n return await call_next(request)\n except ValueError:\n pass\n\n headers = {'WWW-Authenticate': 'Basic'}\n return Response(status_code=401, headers=headers)\n\n\ndef _gc(forced: bool = False):\n global args\n if args.disable_gc and not forced:\n return\n\n import gc\n\n gc.collect()\n if torch.cuda.is_available():\n torch.cuda.empty_cache()\n\n\n@asynccontextmanager\nasync def lifespan(app: FastAPI): # collects GPU memory\n yield\n _gc(forced=True)\n\n\napp = FastAPI(lifespan=lifespan)\n\napp.add_middleware(\n CORSMiddleware,\n allow_origins=['*'],\n allow_credentials=True,\n allow_methods=['*'],\n allow_headers=['*'],\n)\n\n\nclass ModelCard(BaseModel):\n id: str\n object: str = 'model'\n created: int = Field(default_factory=lambda: int(time.time()))\n owned_by: str = 'owner'\n root: Optional[str] = None\n parent: Optional[str] = None\n permission: Optional[list] = None\n\n\nclass ModelList(BaseModel):\n object: str = 'list'\n data: List[ModelCard] = []\n\n\nclass ChatMessage(BaseModel):\n role: Literal['user', 'assistant', 'system', 'function']\n content: Optional[str]\n function_call: Optional[Dict] = None\n\n\nclass DeltaMessage(BaseModel):\n role: Optional[Literal['user', 'assistant', 'system']] = None\n content: Optional[str] = None\n\n\nclass ChatCompletionRequest(BaseModel):\n model: str\n messages: List[ChatMessage]\n functions: Optional[List[Dict]] = None\n temperature: Optional[float] = None\n top_p: Optional[float] = None\n top_k: Optional[int] = None\n max_length: Optional[int] = None\n stream: Optional[bool] = False\n stop: Optional[List[str]] = None\n\n\nclass ChatCompletionResponseChoice(BaseModel):\n index: int\n message: Union[ChatMessage]\n finish_reason: Literal['stop', 'length', 'function_call']\n\n\nclass ChatCompletionResponseStreamChoice(BaseModel):\n index: int\n delta: DeltaMessage\n finish_reason: Optional[Literal['stop', 'length']]\n\n\nclass ChatCompletionResponse(BaseModel):\n model: str\n object: Literal['chat.completion', 'chat.completion.chunk']\n choices: List[Union[ChatCompletionResponseChoice,\n ChatCompletionResponseStreamChoice]]\n created: Optional[int] = Field(default_factory=lambda: int(time.time()))\n\n\n@app.get('/v1/models', response_model=ModelList)\nasync def list_models():\n global model_args\n model_card = ModelCard(id='gpt-3.5-turbo')\n return ModelList(data=[model_card])\n\n\n# To work around that unpleasant leading-\\n tokenization issue!\ndef add_extra_stop_words(stop_words):\n if stop_words:\n _stop_words = []\n _stop_words.extend(stop_words)\n for x in stop_words:\n s = x.lstrip('\\n')\n if s and (s not in _stop_words):\n _stop_words.append(s)\n return _stop_words\n return stop_words\n\n\ndef trim_stop_words(response, stop_words):\n if stop_words:\n for stop in stop_words:\n idx = response.find(stop)\n if idx != -1:\n response = response[:idx]\n return response\n\n\nTOOL_DESC = (\n '{name_for_model}: Call this tool to interact with the {name_for_human} API.'\n ' What is the {name_for_human} API useful for? {description_for_model} Parameters: {parameters}'\n)\n\nREACT_INSTRUCTION = \"\"\"Answer the following questions as best you can. You have access to the following APIs:\n\n{tools_text}\n\nUse the following format:\n\nQuestion: the input question you must answer\nThought: you should always think about what to do\nAction: the action to take, should be one of [{tools_name_text}]\nAction Input: the input to the action\nObservation: the result of the action\n... (this Thought/Action/Action Input/Observation can be repeated zero or more times)\nThought: I now know the final answer\nFinal Answer: the final answer to the original input question\n\nBegin!\"\"\"\n\n_TEXT_COMPLETION_CMD = object()\n\n\ndef parse_messages(messages, functions):\n if all(m.role != 'user' for m in messages):\n raise HTTPException(\n status_code=400,\n detail='Invalid request: Expecting at least one user message.',\n )\n\n messages = copy.deepcopy(messages)\n if messages[0].role == 'system':\n system = messages.pop(0).content.lstrip('\\n').rstrip()\n else:\n system = 'You are a helpful assistant.'\n\n if functions:\n tools_text = []\n tools_name_text = []\n for func_info in functions:\n name = func_info.get('name', '')\n name_m = func_info.get('name_for_model', name)\n name_h = func_info.get('name_for_human', name)\n desc = func_info.get('description', '')\n desc_m = func_info.get('description_for_model', desc)\n tool = TOOL_DESC.format(\n name_for_model=name_m,\n name_for_human=name_h,\n # Hint: You can add the following format requirements in description:\n # \"Format the arguments as a JSON object.\"\n # \"Enclose the code within triple backticks (`) at the beginning and end of the code.\"\n description_for_model=desc_m,\n parameters=json.dumps(func_info['parameters'],\n ensure_ascii=False),\n )\n tools_text.append(tool)\n tools_name_text.append(name_m)\n tools_text = '\\n\\n'.join(tools_text)\n tools_name_text = ', '.join(tools_name_text)\n instruction = (REACT_INSTRUCTION.format(\n tools_text=tools_text,\n tools_name_text=tools_name_text,\n ).lstrip('\\n').rstrip())\n else:\n instruction = ''\n\n messages_with_fncall = messages\n messages = []\n for m_idx, m in enumerate(messages_with_fncall):\n role, content, func_call = m.role, m.content, m.function_call\n content = content or ''\n content = content.lstrip('\\n').rstrip()\n if role == 'function':\n if (len(messages) == 0) or (messages[-1].role != 'assistant'):\n raise HTTPException(\n status_code=400,\n detail=\n 'Invalid request: Expecting role assistant before role function.',\n )\n messages[-1].content += f'\\nObservation: {content}'\n if m_idx == len(messages_with_fncall) - 1:\n # add a prefix for text completion\n messages[-1].content += '\\nThought:'\n elif role == 'assistant':\n if len(messages) == 0:\n raise HTTPException(\n status_code=400,\n detail=\n 'Invalid request: Expecting role user before role assistant.',\n )\n if func_call is None:\n if functions:\n content = f'Thought: I now know the final answer.\\nFinal Answer: {content}'\n else:\n f_name, f_args = func_call['name'], func_call['arguments']\n if not content.startswith('Thought:'):\n content = f'Thought: {content}'\n content = f'{content}\\nAction: {f_name}\\nAction Input: {f_args}'\n if messages[-1].role == 'user':\n messages.append(\n ChatMessage(role='assistant',\n content=content.lstrip('\\n').rstrip()))\n else:\n messages[-1].content += '\\n' + content\n elif role == 'user':\n messages.append(\n ChatMessage(role='user',\n content=content.lstrip('\\n').rstrip()))\n else:\n raise HTTPException(\n status_code=400,\n detail=f'Invalid request: Incorrect role {role}.')\n\n query = _TEXT_COMPLETION_CMD\n if messages[-1].role == 'user':\n query = messages[-1].content\n messages = messages[:-1]\n\n if len(messages) % 2 != 0:\n raise HTTPException(status_code=400, detail='Invalid request')\n\n history = [] # [(Q1, A1), (Q2, A2), ..., (Q_last_turn, A_last_turn)]\n for i in range(0, len(messages), 2):\n if messages[i].role == 'user' and messages[i + 1].role == 'assistant':\n usr_msg = messages[i].content.lstrip('\\n').rstrip()\n bot_msg = messages[i + 1].content.lstrip('\\n').rstrip()\n if instruction and (i == len(messages) - 2):\n usr_msg = f'{instruction}\\n\\nQuestion: {usr_msg}'\n instruction = ''\n history.append([usr_msg, bot_msg])\n else:\n raise HTTPException(\n status_code=400,\n detail=\n 'Invalid request: Expecting exactly one user (or function) role before every assistant role.',\n )\n if instruction:\n assert query is not _TEXT_COMPLETION_CMD\n query = f'{instruction}\\n\\nQuestion: {query}'\n return query, history, system\n\n\ndef parse_response(response):\n func_name, func_args = '', ''\n i = response.find('\\nAction:')\n j = response.find('\\nAction Input:')\n k = response.find('\\nObservation:')\n if 0 <= i < j: # If the text has `Action` and `Action input`,\n if k < j: # but does not contain `Observation`,\n # then it is likely that `Observation` is omitted by the LLM,\n # because the output text may have discarded the stop word.\n response = response.rstrip() + '\\nObservation:' # Add it back.\n k = response.find('\\nObservation:')\n func_name = response[i + len('\\nAction:'):j].strip()\n func_args = response[j + len('\\nAction Input:'):k].strip()\n\n if func_name:\n response = response[:i]\n t = response.find('Thought: ')\n if t >= 0:\n response = response[t + len('Thought: '):]\n response = response.strip()\n choice_data = ChatCompletionResponseChoice(\n index=0,\n message=ChatMessage(\n role='assistant',\n content=response,\n function_call={\n 'name': func_name,\n 'arguments': func_args\n },\n ),\n finish_reason='function_call',\n )\n return choice_data\n\n z = response.rfind('\\nFinal Answer: ')\n if z >= 0:\n response = response[z + len('\\nFinal Answer: '):]\n choice_data = ChatCompletionResponseChoice(\n index=0,\n message=ChatMessage(role='assistant', content=response),\n finish_reason='stop',\n )\n return choice_data\n\n\n# completion mode, not chat mode\ndef text_complete_last_message(history, stop_words_ids, gen_kwargs, system):\n im_start = '<|im_start|>'\n im_end = '<|im_end|>'\n prompt = f'{im_start}system\\n{system}{im_end}'\n for i, (query, response) in enumerate(history):\n query = query.lstrip('\\n').rstrip()\n response = response.lstrip('\\n').rstrip()\n prompt += f'\\n{im_start}user\\n{query}{im_end}'\n prompt += f'\\n{im_start}assistant\\n{response}{im_end}'\n prompt = prompt[:-len(im_end)]\n\n _stop_words_ids = [tokenizer.encode(im_end)]\n if stop_words_ids:\n for s in stop_words_ids:\n _stop_words_ids.append(s)\n stop_words_ids = _stop_words_ids\n\n input_ids = torch.tensor([tokenizer.encode(prompt)]).to(model.device)\n output = model.generate(input_ids,\n stop_words_ids=stop_words_ids,\n **gen_kwargs).tolist()[0]\n output = tokenizer.decode(output, errors='ignore')\n assert output.startswith(prompt)\n output = output[len(prompt):]\n output = trim_stop_words(output, ['<|endoftext|>', im_end])\n print(f'<completion>\\n{prompt}\\n<!-- *** -->\\n{output}\\n</completion>')\n return output\n\n\n@app.post('/v1/chat/completions', response_model=ChatCompletionResponse)\nasync def create_chat_completion(request: ChatCompletionRequest):\n global model, tokenizer\n\n gen_kwargs = {}\n if request.top_k is not None:\n gen_kwargs['top_k'] = request.top_k\n if request.temperature is not None:\n if request.temperature < 0.01:\n gen_kwargs['top_k'] = 1 # greedy decoding\n else:\n # Not recommended. Please tune top_p instead.\n gen_kwargs['temperature'] = request.temperature\n if request.top_p is not None:\n gen_kwargs['top_p'] = request.top_p\n\n stop_words = add_extra_stop_words(request.stop)\n if request.functions:\n stop_words = stop_words or []\n if 'Observation:' not in stop_words:\n stop_words.append('Observation:')\n\n query, history, system = parse_messages(request.messages,\n request.functions)\n\n if request.stream:\n if request.functions:\n raise HTTPException(\n status_code=400,\n detail=\n 'Invalid request: Function calling is not yet implemented for stream mode.',\n )\n generate = predict(query,\n history,\n request.model,\n stop_words,\n gen_kwargs,\n system=system)\n return EventSourceResponse(generate, media_type='text/event-stream')\n\n stop_words_ids = [tokenizer.encode(s)\n for s in stop_words] if stop_words else None\n if query is _TEXT_COMPLETION_CMD:\n response = text_complete_last_message(history,\n stop_words_ids=stop_words_ids,\n gen_kwargs=gen_kwargs,\n system=system)\n else:\n response, _ = model.chat(\n tokenizer,\n query,\n history=history,\n system=system,\n stop_words_ids=stop_words_ids,\n **gen_kwargs,\n )\n print('<chat>')\n pprint(history, indent=2)\n print(f'{query}\\n<!-- *** -->\\n{response}\\n</chat>')\n _gc()\n\n response = trim_stop_words(response, stop_words)\n if request.functions:\n choice_data = parse_response(response)\n else:\n choice_data = ChatCompletionResponseChoice(\n index=0,\n message=ChatMessage(role='assistant', content=response),\n finish_reason='stop',\n )\n return ChatCompletionResponse(model=request.model,\n choices=[choice_data],\n object='chat.completion')\n\n\ndef _dump_json(data: BaseModel, *args, **kwargs) -> str:\n try:\n return data.model_dump_json(*args, **kwargs)\n except AttributeError: # pydantic<2.0.0\n return data.json(*args, **kwargs) # noqa\n\n\nasync def predict(\n query: str,\n history: List[List[str]],\n model_id: str,\n stop_words: List[str],\n gen_kwargs: Dict,\n system: str,\n):\n global model, tokenizer\n choice_data = ChatCompletionResponseStreamChoice(\n index=0, delta=DeltaMessage(role='assistant'), finish_reason=None)\n chunk = ChatCompletionResponse(model=model_id,\n choices=[choice_data],\n object='chat.completion.chunk')\n yield '{}'.format(_dump_json(chunk, exclude_unset=True))\n\n current_length = 0\n stop_words_ids = [tokenizer.encode(s)\n for s in stop_words] if stop_words else None\n\n delay_token_num = max([len(x) for x in stop_words]) if stop_words_ids else 0\n response_generator = model.chat_stream(tokenizer,\n query,\n history=history,\n stop_words_ids=stop_words_ids,\n system=system,\n **gen_kwargs)\n for _new_response in response_generator:\n if len(_new_response) <= delay_token_num:\n continue\n new_response = _new_response[:-delay_token_num] if delay_token_num else _new_response\n\n if len(new_response) == current_length:\n continue\n\n new_text = new_response[current_length:]\n current_length = len(new_response)\n\n choice_data = ChatCompletionResponseStreamChoice(\n index=0, delta=DeltaMessage(content=new_text), finish_reason=None)\n chunk = ChatCompletionResponse(model=model_id,\n choices=[choice_data],\n object='chat.completion.chunk')\n yield '{}'.format(_dump_json(chunk, exclude_unset=True))\n \n if current_length != len(_new_response):\n # Determine whether to print the delay tokens\n delayed_text = _new_response[current_length:]\n new_text = trim_stop_words(delayed_text, stop_words)\n if len(new_text) > 0:\n choice_data = ChatCompletionResponseStreamChoice(\n index=0, delta=DeltaMessage(content=new_text), finish_reason=None)\n chunk = ChatCompletionResponse(model=model_id,\n choices=[choice_data],\n object='chat.completion.chunk')\n yield '{}'.format(_dump_json(chunk, exclude_unset=True))\n\n choice_data = ChatCompletionResponseStreamChoice(index=0,\n delta=DeltaMessage(),\n finish_reason='stop')\n chunk = ChatCompletionResponse(model=model_id,\n choices=[choice_data],\n object='chat.completion.chunk')\n yield '{}'.format(_dump_json(chunk, exclude_unset=True))\n yield '[DONE]'\n\n _gc()\n\n\ndef _get_args():\n parser = ArgumentParser()\n parser.add_argument(\n '-c',\n '--checkpoint-path',\n type=str,\n default='Qwen/Qwen-7B-Chat',\n help='Checkpoint name or path, default to %(default)r',\n )\n parser.add_argument('--api-auth', help='API authentication credentials')\n parser.add_argument('--cpu-only',\n action='store_true',\n help='Run demo with CPU only')\n parser.add_argument('--server-port',\n type=int,\n default=8000,\n help='Demo server port.')\n parser.add_argument(\n '--server-name',\n type=str,\n default='127.0.0.1',\n help=\n 'Demo server name. Default: 127.0.0.1, which is only visible from the local computer.'\n ' If you want other computers to access your server, use 0.0.0.0 instead.',\n )\n parser.add_argument(\n '--disable-gc',\n action='store_true',\n help='Disable GC after each response generated.',\n )\n\n args = parser.parse_args()\n return args\n\n\nif __name__ == '__main__':\n args = _get_args()\n\n tokenizer = AutoTokenizer.from_pretrained(\n args.checkpoint_path,\n trust_remote_code=True,\n resume_download=True,\n )\n\n if args.api_auth:\n app.add_middleware(BasicAuthMiddleware,\n username=args.api_auth.split(':')[0],\n password=args.api_auth.split(':')[1])\n\n if args.cpu_only:\n device_map = 'cpu'\n else:\n device_map = 'auto'\n\n model = AutoModelForCausalLM.from_pretrained(\n args.checkpoint_path,\n device_map=device_map,\n trust_remote_code=True,\n resume_download=True,\n ).eval()\n\n model.generation_config = GenerationConfig.from_pretrained(\n args.checkpoint_path,\n trust_remote_code=True,\n resume_download=True,\n )\n\n uvicorn.run(app, host=args.server_name, port=args.server_port, workers=1)\n" }, { "path": "recipes/applications/chatbot/qwen_chatbot.ipynb", "content": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"id\": \"54d5d255-aa98-4655-8dd1-bc726430d86a\",\n \"metadata\": {},\n \"source\": [\n \"# Qwen-7B-Chat Chat Demo\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"31e04af4-eb27-4802-a7b2-6ea0525f1dc8\",\n \"metadata\": {},\n \"source\": [\n \"This notebook uses Qwen-7B-Chat as an example to introduce you to how to build a web-based conversational assistant using Gradio.\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"75e51155-9f8e-40dc-8432-60f4567d93a8\",\n \"metadata\": {},\n \"source\": [\n \"## Preparation\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"ff6f061c-a033-49f2-8f7d-af3f23ac9125\",\n \"metadata\": {},\n \"source\": [\n \"Download Qwen-7B-Chat\\n\",\n \"\\n\",\n \"Firstly, we need to download the model. You can use the snapshot_download that comes with modelscope to download the model to a specified directory.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"c469a129-451f-4d01-8bc0-e2cf70a262c8\",\n \"metadata\": {\n \"tags\": []\n },\n \"outputs\": [],\n \"source\": [\n \"!pip install modelscope\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"69af626e-22b8-49ad-8869-8354f4c72bcc\",\n \"metadata\": {\n \"tags\": []\n },\n \"outputs\": [],\n \"source\": [\n \"from modelscope.hub.snapshot_download import snapshot_download\\n\",\n \"snapshot_download(\\\"qwen/Qwen-7B-Chat\\\",cache_dir='/tmp/models') \"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"01d2ff34-4053-4710-a289-e354673be1ca\",\n \"metadata\": {},\n \"source\": [\n \"## Install Dependencies\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"48b51791-4bbc-4d12-9cd6-587c24c8bea7\",\n \"metadata\": {\n \"tags\": []\n },\n \"outputs\": [],\n \"source\": [\n \"!pip install -r ../../../requirements.txt\\n\",\n \"!pip install gradio==3.37.0 mdtex2html\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"7732037a-246a-4953-af07-dae7a3ae5937\",\n \"metadata\": {},\n \"source\": [\n \"## Run the web UI code to start the Qwen chatbot\\n\",\n \"\\n\",\n \"Users can run the web_demo.py file to have real-time conversations with Qwen-7b-chat on the webpage.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"9e256f0a-d96d-4fd7-b305-fe43c6959dc8\",\n \"metadata\": {\n \"ExecutionIndicator\": {\n \"show\": true\n },\n \"tags\": []\n },\n \"outputs\": [],\n \"source\": [\n \"!python ../../../web_demo.py -c /tmp/models/qwen/Qwen-7B-Chat\"\n ]\n }\n ],\n \"metadata\": {\n \"kernelspec\": {\n \"display_name\": \"Python 3 (ipykernel)\",\n \"language\": \"python\",\n \"name\": \"python3\"\n },\n \"language_info\": {\n \"codemirror_mode\": {\n \"name\": \"ipython\",\n \"version\": 3\n },\n \"file_extension\": \".py\",\n \"mimetype\": \"text/x-python\",\n \"name\": \"python\",\n \"nbconvert_exporter\": \"python\",\n \"pygments_lexer\": \"ipython3\",\n \"version\": \"3.10.13\"\n }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 5\n }\n " }, { "path": "recipes/applications/domain_finetune/qwen_domain_finetune.ipynb", "content": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"id\": \"0e7993c3-3999-4ac5-b1dc-77875d80e4c8\",\n \"metadata\": {},\n \"source\": [\n \"# Fine-tuning Qwen-7B-Chat on Your Own Domain-Specific Data\\n\",\n \"\\n\",\n \"This notebook uses Qwen-7B-Chat as an example to introduce how to LoRA fine-tune the Qwen model on a specific domain.\\n\",\n \"\\n\",\n \"Qwen is a pretrained conversational model that supports English and Chinese. It is suitable for universal scenarios, but may lack some specialized knowledge in certain specific domain. If you would like to fine-tune it for a specific domain, or on your own private dataset, you can refer to this tutorial.\\n\",\n \"\\n\",\n \"Here is an example showing the differences before and after fine-tuning.\\n\",\n \"\\n\",\n \"#### Model Input\\n\",\n \">怎样治疗青光眼?\\n\",\n \"\\n\",\n \"#### Before Fine-tuning\\n\",\n \"\\n\",\n \">青光眼的治疗方法取决于病情严重程度。轻度青光眼可以采取药物治疗，比如激素类药物、抗生素等，以减轻炎症和改善血管紧张。\\n\",\n \">\\n\",\n \">中度青光眼可能需要激光手术来降低眼压，并使用药物以保持控制。\\n\",\n \">\\n\",\n \">重度青光眼则需要实施传统的眼部手术，如小梁切除术或巩膜外减压术，以防止视神经损伤。\\n\",\n \"\\n\",\n \"#### After Fine-tuning\\n\",\n \"\\n\",\n \">青光眼是一种眼睛疾病，可以引起视神经的损伤和视力下降。治疗青光眼的主要方法是药物治疗、激光治疗和手术治疗。\\n\",\n \">1. 药物治疗：对于早期青光眼患者，通常采用抗青光眼药物治疗，通过降低眼压来预防视神经进一步损伤。目前常用的抗青光眼药物有拉坦前列素滴眼液、阿托品滴眼液等。\\n\",\n \">2. 激光治疗：对于中晚期青光眼患者，如果药物治疗效果不佳，可能需要进行激光治疗。目前常用的激光治疗方法包括小梁切除术、YAG激光手术等。\\n\",\n \">3. 手术治疗：对于非常严重的青光眼患者，可能需要进行手术治疗。目前常用的手术治疗方法包括前房引流术、虹膜周切术、小梁切除术、巩膜滤过术等。\\n\",\n \">\\n\",\n \">需要注意的是，以上治疗方法的具体选择应根据患者的病情和医生的专业建议来进行。同时，青光眼患者在日常生活中也需要保持良好的生活习惯，避免过度疲劳、保持情绪稳定、定期检查眼睛等情况的发生。\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"bdea7e21-fec8-49fe-b7ea-afde3f02738f\",\n \"metadata\": {},\n \"source\": [\n \"## Environment Requirements\\n\",\n \"\\n\",\n \"Please refer to **requirements.txt** to install the required dependencies.\\n\",\n \"\\n\",\n \"Run the following command line in the main directory of the Qwen repo.\\n\",\n \"```bash\\n\",\n \"pip install -r requirements.txt\\n\",\n \"```\\n\",\n \"\\n\",\n \"\\n\",\n \"## Preparation\\n\",\n \"\\n\",\n \"### Download Qwen-7B-Chat\\n\",\n \"\\n\",\n \"First, download the model files. You can choose to download directly from ModelScope.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"248488f9-4a86-4f35-9d56-50f8e91a8f11\",\n \"metadata\": {\n \"ExecutionIndicator\": {\n \"show\": true\n },\n \"tags\": []\n },\n \"outputs\": [],\n \"source\": [\n \"from modelscope.hub.snapshot_download import snapshot_download\\n\",\n \"model_dir = snapshot_download('Qwen/Qwen-7B-chat', cache_dir='.')\"\n ]\n },\n {\n \"attachments\": {},\n \"cell_type\": \"markdown\",\n \"id\": \"7b2a92b1-f08e-4413-9f92-8f23761e6e1f\",\n \"metadata\": {},\n \"source\": [\n \"### Download Medical Training Data\\n\",\n \"\\n\",\n \"Download the data required for training; here, we provide a medical conversation dataset for training. It is sampled from [MedicalGPT repo](https://github.com/shibing624/MedicalGPT/) and we have converted this dataset into a format that can be used for fine-tuning.\\n\",\n \"\\n\",\n \"Disclaimer: the dataset can be only used for the research purpose.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"ce195f08-fbb2-470e-b6c0-9a03457458c7\",\n \"metadata\": {\n \"ExecutionIndicator\": {\n \"show\": true\n },\n \"tags\": []\n },\n \"outputs\": [],\n \"source\": [\n \"!wget https://atp-modelzoo-sh.oss-cn-shanghai.aliyuncs.com/release/tutorials/qwen_recipes/medical_sft.json\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"7226bed0-171b-4d45-a3f9-b3d81ec2bb9f\",\n \"metadata\": {},\n \"source\": [\n \"You can prepare your dataset in JSON format following the format below, and then modify the `--data_path` parameter in the training command to point to your JSON file.\\n\",\n \"\\n\",\n \"These data instances can be conversations in the real world or include domain knowledge QA pairs. Besides, fine-tuning allows Qwen-chat to play like some specific roles. As Qwen-chat is a dialogue model for general scenarios, your fine-tuning can customize a chatbot to meet your requirements.\\n\",\n \"\\n\",\n \"We recommend that you prepare 50~ data instances if you want to fine-tune Qwen-chat as a roleplay model.\\n\",\n \"\\n\",\n \"You may prepare much more data instances if you want to infuse the domain knowledge of your field into the model.\\n\",\n \"\\n\",\n \"In this tutorial, we have prepared a medical domain fine-tuning dataset consisting of 1000 data instancess as an example. You can refer to our example to fine-tune on your own domain-specific dataset.\\n\",\n \"\\n\",\n \"Below is a simple example list with 1 sample:\\n\",\n \"\\n\",\n \"```python\\n\",\n \"[\\n\",\n \" {\\n\",\n \" \\\"id\\\": \\\"1\\\",\\n\",\n \" \\\"conversations\\\": [\\n\",\n \" {\\n\",\n \" \\\"from\\\": \\\"user\\\",\\n\",\n \" \\\"value\\\": \\\"直肠腺瘤的早期症状?\\\"\\n\",\n \" },\\n\",\n \" {\\n\",\n \" \\\"from\\\": \\\"assistant\\\",\\n\",\n \" \\\"value\\\": \\\"结直肠腺瘤是起源于结直肠黏膜腺上皮的良性肿瘤，包括结肠腺瘤与直肠腺瘤，是常见的肠道良性肿瘤。因与大肠癌的发生关系密切，被认为是一种癌前病变。不同地区、不同年龄的发病率差别很大，40岁以下的发病率低，60岁以上较高，男女无明显差别。此类疾病的病因及发病机制不明确。部分患者有遗传因素存在，另外，高脂肪饮食与食物纤维不足、肠道菌群紊乱也与此病的发生有相关性。多数患者无症状。症状与腺瘤的大小、部位、病理性质等有关。有症状者可有以下表现：直肠腺瘤的临床常见症状有便血，多为无疼性便血，常呈鲜红色，量多，可引起贫血。粪便性状改变，多为粘液便，便意不尽或里急重感，有时只排出粘液，称为假性腹泻。1.腹部不适、腹部疼痛，多为大的腺瘤伴发肠套叠、肠梗阻引起。2.排便习惯改变，包括便秘、腹泻、里急后重等。3.粪便带血，最常见为间歇性便血。4.部分位于直肠的较大的、带蒂腺瘤可在排便时脱落或脱出肛。可以采取内镜下高频电凝、激光、微波凝固等方法切除，也可以选择外科手术切除，并定期随访。有恶变者根据情况选择其他治疗（如放疗、化疗、手术等）管状腺瘤切除术后复发者少见，但绒毛状腺瘤及绒毛管状腺瘤切除术后常可复发，尤其是绒毛状腺瘤，且多发的腺瘤复发率高于单发者。对于经内镜治疗或局部手术切除的结直肠腺瘤患者尤其是绒毛状腺瘤或广基的绒毛管状腺瘤患者，建议腺瘤切除后的第一年内3～6个月进行一次肠镜检查，第二年开始每年一次。\\\"\\n\",\n \" }\\n\",\n \" ]\\n\",\n \" }\\n\",\n \"]\\n\",\n \"```\\n\",\n \"\\n\",\n \"You can also use multi-turn conversations as the training set. Here is a simple example:\\n\",\n \"\\n\",\n \"```python\\n\",\n \"[\\n\",\n \" {\\n\",\n \" \\\"id\\\": \\\"2\\\",\\n\",\n \" \\\"conversations\\\": [\\n\",\n \" {\\n\",\n \" \\\"from\\\": \\\"user\\\",\\n\",\n \" \\\"value\\\": \\\"你好，我最近经常感觉胸口疼痛，这是怎么回事？\\\"\\n\",\n \" },\\n\",\n \" {\\n\",\n \" \\\"from\\\": \\\"assistant\\\",\\n\",\n \" \\\"value\\\": \\\"胸痛可能由多种原因引起，包括心脏问题、消化系统疾病、呼吸系统问题等。您能描述一下疼痛的性质和持续时间吗？\\\"\\n\",\n \" },\\n\",\n \" {\\n\",\n \" \\\"from\\\": \\\"user\\\",\\n\",\n \" \\\"value\\\": \\\"疼痛感觉像是挤压，大概持续了几分钟。\\\"\\n\",\n \" },\\n\",\n \" {\\n\",\n \" \\\"from\\\": \\\"assistant\\\",\\n\",\n \" \\\"value\\\": \\\"挤压感的胸痛可能与心脏问题有关，特别是如果它伴随着呼吸困难、出汗或恶心。我建议您尽快去看医生并进行适当的检查，如心电图和血液检测，以确定具体原因。\\\"\\n\",\n \" },\\n\",\n \" {\\n\",\n \" \\\"from\\\": \\\"user\\\",\\n\",\n \" \\\"value\\\": \\\"我应该去急诊室吗，还是预约我的家庭医生？\\\"\\n\",\n \" },\\n\",\n \" {\\n\",\n \" \\\"from\\\": \\\"assistant\\\",\\n\",\n \" \\\"value\\\": \\\"如果您的疼痛是突然发生的，并且还有其他症状，建议您立即去急诊室。如果疼痛不是很严重且没有其他严重症状，您可以预约家庭医生进一步评估。但请不要忽视疼痛，尤其是如果这种情况是第一次出现。\\\"\\n\",\n \" }\\n\",\n \" ]\\n\",\n \" }\\n\",\n \"]\\n\",\n \"```\\n\",\n \"\\n\",\n \"## Fine-Tune the Model\\n\",\n \"\\n\",\n \"You can directly run the prepared training script to fine-tune the model. \\n\",\n \"\\n\",\n \"For parameter settings, you can modify `--model_name_or_path` to the location of the model you want to fine-tune, and set `--data_path` to the location of the dataset.\\n\",\n \"\\n\",\n \"You should remove the `--bf16` parameter if you are using a non-Ampere architecture GPU, such as a V100. \\n\",\n \"\\n\",\n \"For `--model_max_length` and `--per_device_train_batch_size`, we recommend the following configurations, ,you can refer to [this document](../../finetune/deepspeed/readme.md) for more details:\\n\",\n \"\\n\",\n \"| --model_max_length | --per_device_train_batch_size | GPU Memory |\\n\",\n \"|-----------------|------------|--------------------|\\n\",\n \"| 512 | 4 | 24g |\\n\",\n \"| 1024 | 3 | 24g |\\n\",\n \"| 512 | 8 | 32g |\\n\",\n \"| 1024 | 6 | 32g |\\n\",\n \"\\n\",\n \"You can use our recommended saving parameters, or you can save by epoch by just setting `--save_strategy \\\"epoch\\\"` if you prefer to save at each epoch stage. `--save_total_limit` means the limit on the number of saved checkpoints.\\n\",\n \"\\n\",\n \"For other parameters, such as `--weight_decay` and `--adam_beta2`, we recommend using the values we provided blow.\\n\",\n \"\\n\",\n \"Setting the parameters `--gradient_checkpointing` and `--lazy_preprocess` is to save GPU memory.\\n\",\n \"\\n\",\n \"The parameters for the trained Lora module will be saved in the **output_qwen** folder.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"7ab0581e-be85-45e6-a5b7-af9c42ea697b\",\n \"metadata\": {\n \"ExecutionIndicator\": {\n \"show\": true\n },\n \"tags\": []\n },\n \"outputs\": [],\n \"source\": [\n \"!python ../../../finetune/finetune.py \\\\\\n\",\n \" --model_name_or_path \\\"Qwen/Qwen-7B-Chat/\\\"\\\\\\n\",\n \" --data_path \\\"medical_sft.json\\\"\\\\\\n\",\n \" --bf16 \\\\\\n\",\n \" --output_dir \\\"output_qwen\\\" \\\\\\n\",\n \" --num_train_epochs 4\\\\\\n\",\n \" --per_device_train_batch_size 4 \\\\\\n\",\n \" --per_device_eval_batch_size 3 \\\\\\n\",\n \" --gradient_accumulation_steps 16 \\\\\\n\",\n \" --evaluation_strategy \\\"no\\\" \\\\\\n\",\n \" --save_strategy \\\"epoch\\\" \\\\\\n\",\n \" --save_steps 3000 \\\\\\n\",\n \" --save_total_limit 10 \\\\\\n\",\n \" --learning_rate 1e-5 \\\\\\n\",\n \" --weight_decay 0.1 \\\\\\n\",\n \" --adam_beta2 0.95 \\\\\\n\",\n \" --warmup_ratio 0.01 \\\\\\n\",\n \" --lr_scheduler_type \\\"cosine\\\" \\\\\\n\",\n \" --logging_steps 10 \\\\\\n\",\n \" --model_max_length 512 \\\\\\n\",\n \" --gradient_checkpointing \\\\\\n\",\n \" --lazy_preprocess \\\\\\n\",\n \" --use_lora\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"5e6f28aa-1772-48ce-aa15-8cf29e7d67b5\",\n \"metadata\": {},\n \"source\": [\n \"## Merge Weights\\n\",\n \"\\n\",\n \"The LoRA training only saves the adapter parameters. You can load the fine-tuned model and merge weights as shown below:\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 7,\n \"id\": \"4fd5ef2a-34f9-4909-bebe-7b3b086fd16a\",\n \"metadata\": {\n \"ExecutionIndicator\": {\n \"show\": true\n },\n \"execution\": {\n \"iopub.execute_input\": \"2024-01-26T02:46:14.585746Z\",\n \"iopub.status.busy\": \"2024-01-26T02:46:14.585089Z\",\n \"iopub.status.idle\": \"2024-01-26T02:47:08.095464Z\",\n \"shell.execute_reply\": \"2024-01-26T02:47:08.094715Z\",\n \"shell.execute_reply.started\": \"2024-01-26T02:46:14.585720Z\"\n },\n \"tags\": []\n },\n \"outputs\": [\n {\n \"name\": \"stderr\",\n \"output_type\": \"stream\",\n \"text\": [\n \"The model is automatically converting to bf16 for faster inference. If you want to disable the automatic precision, please manually add bf16/fp16/fp32=True to \\\"AutoModelForCausalLM.from_pretrained\\\".\\n\",\n \"Try importing flash-attention for faster inference...\\n\",\n \"Warning: import flash_attn rms_norm fail, please install FlashAttention layer_norm to get higher efficiency https://github.com/Dao-AILab/flash-attention/tree/main/csrc/layer_norm\\n\",\n \"Loading checkpoint shards: 100%|██████████| 8/8 [00:06<00:00, 1.14it/s]\\n\"\n ]\n }\n ],\n \"source\": [\n \"from transformers import AutoModelForCausalLM\\n\",\n \"from peft import PeftModel\\n\",\n \"import torch\\n\",\n \"\\n\",\n \"model = AutoModelForCausalLM.from_pretrained(\\\"Qwen/Qwen-7B-chat/\\\", torch_dtype=torch.float16, device_map=\\\"auto\\\", trust_remote_code=True)\\n\",\n \"model = PeftModel.from_pretrained(model, \\\"output_qwen/\\\")\\n\",\n \"merged_model = model.merge_and_unload()\\n\",\n \"merged_model.save_pretrained(\\\"output_qwen_merged\\\", max_shard_size=\\\"2048MB\\\", safe_serialization=True)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"2e3f5b9f-63a1-4599-8d9b-a8d8f764838f\",\n \"metadata\": {},\n \"source\": [\n \"The tokenizer files are not saved in the new directory in this step. You can copy the tokenizer files or use the following code:\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 8,\n \"id\": \"10fa5ea3-dd55-4901-86af-c045d4c56533\",\n \"metadata\": {\n \"ExecutionIndicator\": {\n \"show\": true\n },\n \"execution\": {\n \"iopub.execute_input\": \"2024-01-26T02:47:08.097051Z\",\n \"iopub.status.busy\": \"2024-01-26T02:47:08.096744Z\",\n \"iopub.status.idle\": \"2024-01-26T02:47:08.591289Z\",\n \"shell.execute_reply\": \"2024-01-26T02:47:08.590665Z\",\n \"shell.execute_reply.started\": \"2024-01-26T02:47:08.097029Z\"\n },\n \"tags\": []\n },\n \"outputs\": [\n {\n \"data\": {\n \"text/plain\": [\n \"('output_qwen_merged/tokenizer_config.json',\\n\",\n \" 'output_qwen_merged/special_tokens_map.json',\\n\",\n \" 'output_qwen_merged/qwen.tiktoken',\\n\",\n \" 'output_qwen_merged/added_tokens.json')\"\n ]\n },\n \"execution_count\": 8,\n \"metadata\": {},\n \"output_type\": \"execute_result\"\n }\n ],\n \"source\": [\n \"from transformers import AutoTokenizer\\n\",\n \"\\n\",\n \"tokenizer = AutoTokenizer.from_pretrained(\\n\",\n \" \\\"Qwen/Qwen-7B-chat/\\\",\\n\",\n \" trust_remote_code=True\\n\",\n \")\\n\",\n \"\\n\",\n \"tokenizer.save_pretrained(\\\"output_qwen_merged\\\")\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"804b84d8\",\n \"metadata\": {},\n \"source\": [\n \"## Test the Model\\n\",\n \"\\n\",\n \"After merging the weights, we can test the model as follows:\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 10,\n \"id\": \"dbae310c\",\n \"metadata\": {\n \"ExecutionIndicator\": {\n \"show\": true\n },\n \"execution\": {\n \"iopub.execute_input\": \"2024-01-26T02:48:29.995040Z\",\n \"iopub.status.busy\": \"2024-01-26T02:48:29.994448Z\",\n \"iopub.status.idle\": \"2024-01-26T02:48:41.677104Z\",\n \"shell.execute_reply\": \"2024-01-26T02:48:41.676591Z\",\n \"shell.execute_reply.started\": \"2024-01-26T02:48:29.995019Z\"\n },\n \"tags\": []\n },\n \"outputs\": [\n {\n \"name\": \"stderr\",\n \"output_type\": \"stream\",\n \"text\": [\n \"Warning: import flash_attn rms_norm fail, please install FlashAttention layer_norm to get higher efficiency https://github.com/Dao-AILab/flash-attention/tree/main/csrc/layer_norm\\n\",\n \"Loading checkpoint shards: 100%|██████████| 8/8 [00:04<00:00, 1.71it/s]\\n\"\n ]\n },\n {\n \"name\": \"stdout\",\n \"output_type\": \"stream\",\n \"text\": [\n \"VDAC1（电压依赖性钙通道）是一种位于细胞膜上的钙离子通道，负责将细胞内的钙离子释放到细胞外。它在神经信号传导、肌肉收缩和血管舒张中发挥着重要作用。\\n\",\n \"\\n\",\n \"VDAC1通常由4个亚基组成，每个亚基都有不同的功能。其中，一个亚基是内腔部分，它与钙离子的结合有关；另一个亚基是外腔部分，它与离子通道的打开和关闭有关；第三个亚基是一层跨膜蛋白，它负责调节通道的开放程度；最后一个亚基是一个膜骨架连接器，它帮助维持通道的结构稳定性。\\n\",\n \"\\n\",\n \"除了钙离子外，VDAC1还能够接收钾离子和氯离子等其他离子，并将其从细胞内释放到细胞外。此外，VDAC1还参与了许多细胞代谢反应，例如脂肪酸合成和糖原分解等。\\n\",\n \"\\n\",\n \"总的来说，VDAC1是细胞膜上的一种重要离子通道，其作用涉及到许多重要的生物学过程。\\n\"\n ]\n }\n ],\n \"source\": [\n \"from transformers import AutoModelForCausalLM, AutoTokenizer\\n\",\n \"from transformers.generation import GenerationConfig\\n\",\n \"\\n\",\n \"tokenizer = AutoTokenizer.from_pretrained(\\\"output_qwen_merged\\\", trust_remote_code=True)\\n\",\n \"model = AutoModelForCausalLM.from_pretrained(\\n\",\n \" \\\"output_qwen_merged\\\",\\n\",\n \" device_map=\\\"auto\\\",\\n\",\n \" trust_remote_code=True\\n\",\n \").eval()\\n\",\n \"\\n\",\n \"response, history = model.chat(tokenizer, \\\"什么是VDAC1？\\\", history=None)\\n\",\n \"print(response)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"987f524d-6918-48ae-a730-f285cf6f8416\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": []\n }\n ],\n \"metadata\": {\n \"kernelspec\": {\n \"display_name\": \"Python 3 (ipykernel)\",\n \"language\": \"python\",\n \"name\": \"python3\"\n },\n \"language_info\": {\n \"codemirror_mode\": {\n \"name\": \"ipython\",\n \"version\": 3\n },\n \"file_extension\": \".py\",\n \"mimetype\": \"text/x-python\",\n \"name\": \"python\",\n \"nbconvert_exporter\": \"python\",\n \"pygments_lexer\": \"ipython3\",\n \"version\": \"3.10.13\"\n }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 5\n}\n" }, { "path": "recipes/applications/retrieval/retrieval.ipynb", "content": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"id\": \"245ab07a-fb2f-4cf4-ab9a-5c05a9b44daa\",\n \"metadata\": {},\n \"source\": [\n \"# LangChain retrieval knowledge base Q&A based on Qwen-7B-Chat\"\n ]\n },\n {\n \"attachments\": {},\n \"cell_type\": \"markdown\",\n \"id\": \"e8df2cb7-a69c-4231-9596-4c871d893633\",\n \"metadata\": {},\n \"source\": [\n \"This notebook introduces a question-answering application based on a local knowledge base using Qwen-7B-Chat with langchain. The goal is to establish a knowledge base Q&A solution that is friendly to many scenarios and open-source models, and that can run offline. The implementation process of this project includes loading files -> reading text -> segmenting text -> vectorizing text -> vectorizing questions -> matching the top k most similar text vectors with the question vectors -> incorporating the matched text as context along with the question into the prompt -> submitting to the LLM (Large Language Model) to generate an answer.\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"92e9c81a-45c7-4c12-91af-3c5dd52f63bb\",\n \"metadata\": {},\n \"source\": [\n \"## Preparation\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"84cfcf88-3bef-4412-a658-4eaefeb6502a\",\n \"metadata\": {},\n \"source\": [\n \"Download Qwen-7B-Chat\\n\",\n \"\\n\",\n \"Firstly, we need to download the model. You can use the snapshot_download that comes with modelscope to download the model to a specified directory.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"9c1f9ded-8035-42c7-82c7-444ce06572bc\",\n \"metadata\": {\n \"tags\": []\n },\n \"outputs\": [],\n \"source\": [\n \"!pip install modelscope\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"7c26225c-c958-429e-b81d-2de9820670c2\",\n \"metadata\": {\n \"ExecutionIndicator\": {\n \"show\": true\n },\n \"tags\": []\n },\n \"outputs\": [],\n \"source\": [\n \"from modelscope.hub.snapshot_download import snapshot_download\\n\",\n \"snapshot_download(\\\"Qwen/Qwen-7B-Chat\\\",cache_dir='/tmp/models') \"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"e8f51796-49fa-467d-a825-ae9a281eb3fd\",\n \"metadata\": {},\n \"source\": [\n \"Download the dependencies for langchain and Qwen.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"87fe1023-644f-4610-afaf-0b7cddc30d60\",\n \"metadata\": {\n \"ExecutionIndicator\": {\n \"show\": true\n },\n \"tags\": []\n },\n \"outputs\": [],\n \"source\": [\n \"!pip install langchain==0.0.187 dashscope==1.0.4 sentencepiece==0.1.99 cpm_kernels==1.0.11 nltk==3.8.1 sentence_transformers==2.2.2 unstructured==0.6.5 faiss-cpu==1.7.4 icetk==0.0.7\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"853cdfa4-a2ce-4baa-919a-b9e2aecd2706\",\n \"metadata\": {},\n \"source\": [\n \"Download the retrieval document.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"8ba800dc-311d-4a83-8115-f05b09b39ffd\",\n \"metadata\": {\n \"tags\": []\n },\n \"outputs\": [],\n \"source\": [\n \"!wget https://atp-modelzoo-sh.oss-cn-shanghai.aliyuncs.com/release/tutorials/qwen_recipes/LLM_Survey_Chinese.pdf.txt\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"07e923b3-b7ae-4983-abeb-2ce115566f15\",\n \"metadata\": {},\n \"source\": [\n \"Download the text2vec model, for Chinese in our case.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"9a07cd8d-3cec-40f6-8d2b-eb111aaf1164\",\n \"metadata\": {\n \"tags\": []\n },\n \"outputs\": [],\n \"source\": [\n \"!wget https://atp-modelzoo-sh.oss-cn-shanghai.aliyuncs.com/release/tutorials/qwen_recipes/GanymedeNil_text2vec-large-chinese.tar.gz\\n\",\n \"!tar -zxvf GanymedeNil_text2vec-large-chinese.tar.gz -C /tmp\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"dc483af0-170e-4e61-8d25-a336d1592e34\",\n \"metadata\": {},\n \"source\": [\n \"## Try out the model \\n\",\n \"\\n\",\n \"Load the Qwen-7B-Chat model.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"c112cf82-0447-46c4-9c32-18f243c0a686\",\n \"metadata\": {\n \"ExecutionIndicator\": {\n \"show\": true\n },\n \"tags\": []\n },\n \"outputs\": [],\n \"source\": [\n \"from abc import ABC\\n\",\n \"from langchain.llms.base import LLM\\n\",\n \"from typing import Any, List, Mapping, Optional\\n\",\n \"from langchain.callbacks.manager import CallbackManagerForLLMRun\\n\",\n \"from transformers import AutoModelForCausalLM, AutoTokenizer\\n\",\n \"\\n\",\n \"model_path=\\\"/tmp/models/Qwen/Qwen-7B-Chat\\\"\\n\",\n \"\\n\",\n \"tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)\\n\",\n \"model = AutoModelForCausalLM.from_pretrained(model_path, trust_remote_code=True).half().cuda()\\n\",\n \"model.eval()\\n\",\n \"\\n\",\n \"class Qwen(LLM, ABC):\\n\",\n \" max_token: int = 10000\\n\",\n \" temperature: float = 0.01\\n\",\n \" top_p = 0.9\\n\",\n \" history_len: int = 3\\n\",\n \"\\n\",\n \" def __init__(self):\\n\",\n \" super().__init__()\\n\",\n \"\\n\",\n \" @property\\n\",\n \" def _llm_type(self) -> str:\\n\",\n \" return \\\"Qwen\\\"\\n\",\n \"\\n\",\n \" @property\\n\",\n \" def _history_len(self) -> int:\\n\",\n \" return self.history_len\\n\",\n \"\\n\",\n \" def set_history_len(self, history_len: int = 10) -> None:\\n\",\n \" self.history_len = history_len\\n\",\n \"\\n\",\n \" def _call(\\n\",\n \" self,\\n\",\n \" prompt: str,\\n\",\n \" stop: Optional[List[str]] = None,\\n\",\n \" run_manager: Optional[CallbackManagerForLLMRun] = None,\\n\",\n \" ) -> str:\\n\",\n \" response, _ = model.chat(tokenizer, prompt, history=[])\\n\",\n \" return response\\n\",\n \" \\n\",\n \" @property\\n\",\n \" def _identifying_params(self) -> Mapping[str, Any]:\\n\",\n \" \\\"\\\"\\\"Get the identifying parameters.\\\"\\\"\\\"\\n\",\n \" return {\\\"max_token\\\": self.max_token,\\n\",\n \" \\\"temperature\\\": self.temperature,\\n\",\n \" \\\"top_p\\\": self.top_p,\\n\",\n \" \\\"history_len\\\": self.history_len}\\n\",\n \" \"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"382ed433-870f-424e-b074-210ea6f84b70\",\n \"metadata\": {},\n \"source\": [\n \"Specify the txt file that needs retrieval for knowledge-based Q&A.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"14be706b-4a7d-4906-9369-1f03c6c99854\",\n \"metadata\": {\n \"ExecutionIndicator\": {\n \"show\": true\n },\n \"tags\": []\n },\n \"outputs\": [],\n \"source\": [\n \"import os\\n\",\n \"import torch\\n\",\n \"import argparse\\n\",\n \"from langchain.vectorstores import FAISS\\n\",\n \"from langchain.embeddings.huggingface import HuggingFaceEmbeddings\\n\",\n \"from typing import List, Tuple\\n\",\n \"import numpy as np\\n\",\n \"from langchain.document_loaders import TextLoader\\n\",\n \"from chinese_text_splitter import ChineseTextSplitter\\n\",\n \"from langchain.docstore.document import Document\\n\",\n \"from langchain.prompts.prompt import PromptTemplate\\n\",\n \"from langchain.chains import RetrievalQA\\n\",\n \"\\n\",\n \"\\n\",\n \"def load_file(filepath, sentence_size=100):\\n\",\n \" loader = TextLoader(filepath, autodetect_encoding=True)\\n\",\n \" textsplitter = ChineseTextSplitter(pdf=False, sentence_size=sentence_size)\\n\",\n \" docs = loader.load_and_split(textsplitter)\\n\",\n \" write_check_file(filepath, docs)\\n\",\n \" return docs\\n\",\n \"\\n\",\n \"\\n\",\n \"def write_check_file(filepath, docs):\\n\",\n \" folder_path = os.path.join(os.path.dirname(filepath), \\\"tmp_files\\\")\\n\",\n \" if not os.path.exists(folder_path):\\n\",\n \" os.makedirs(folder_path)\\n\",\n \" fp = os.path.join(folder_path, 'load_file.txt')\\n\",\n \" with open(fp, 'a+', encoding='utf-8') as fout:\\n\",\n \" fout.write(\\\"filepath=%s,len=%s\\\" % (filepath, len(docs)))\\n\",\n \" fout.write('\\\\n')\\n\",\n \" for i in docs:\\n\",\n \" fout.write(str(i))\\n\",\n \" fout.write('\\\\n')\\n\",\n \" fout.close()\\n\",\n \"\\n\",\n \" \\n\",\n \"def seperate_list(ls: List[int]) -> List[List[int]]:\\n\",\n \" lists = []\\n\",\n \" ls1 = [ls[0]]\\n\",\n \" for i in range(1, len(ls)):\\n\",\n \" if ls[i - 1] + 1 == ls[i]:\\n\",\n \" ls1.append(ls[i])\\n\",\n \" else:\\n\",\n \" lists.append(ls1)\\n\",\n \" ls1 = [ls[i]]\\n\",\n \" lists.append(ls1)\\n\",\n \" return lists\\n\",\n \"\\n\",\n \"\\n\",\n \"class FAISSWrapper(FAISS):\\n\",\n \" chunk_size = 250\\n\",\n \" chunk_conent = True\\n\",\n \" score_threshold = 0\\n\",\n \" \\n\",\n \" def similarity_search_with_score_by_vector(\\n\",\n \" self, embedding: List[float], k: int = 4\\n\",\n \" ) -> List[Tuple[Document, float]]:\\n\",\n \" scores, indices = self.index.search(np.array([embedding], dtype=np.float32), k)\\n\",\n \" docs = []\\n\",\n \" id_set = set()\\n\",\n \" store_len = len(self.index_to_docstore_id)\\n\",\n \" for j, i in enumerate(indices[0]):\\n\",\n \" if i == -1 or 0 < self.score_threshold < scores[0][j]:\\n\",\n \" # This happens when not enough docs are returned.\\n\",\n \" continue\\n\",\n \" _id = self.index_to_docstore_id[i]\\n\",\n \" doc = self.docstore.search(_id)\\n\",\n \" if not self.chunk_conent:\\n\",\n \" if not isinstance(doc, Document):\\n\",\n \" raise ValueError(f\\\"Could not find document for id {_id}, got {doc}\\\")\\n\",\n \" doc.metadata[\\\"score\\\"] = int(scores[0][j])\\n\",\n \" docs.append(doc)\\n\",\n \" continue\\n\",\n \" id_set.add(i)\\n\",\n \" docs_len = len(doc.page_content)\\n\",\n \" for k in range(1, max(i, store_len - i)):\\n\",\n \" break_flag = False\\n\",\n \" for l in [i + k, i - k]:\\n\",\n \" if 0 <= l < len(self.index_to_docstore_id):\\n\",\n \" _id0 = self.index_to_docstore_id[l]\\n\",\n \" doc0 = self.docstore.search(_id0)\\n\",\n \" if docs_len + len(doc0.page_content) > self.chunk_size:\\n\",\n \" break_flag = True\\n\",\n \" break\\n\",\n \" elif doc0.metadata[\\\"source\\\"] == doc.metadata[\\\"source\\\"]:\\n\",\n \" docs_len += len(doc0.page_content)\\n\",\n \" id_set.add(l)\\n\",\n \" if break_flag:\\n\",\n \" break\\n\",\n \" if not self.chunk_conent:\\n\",\n \" return docs\\n\",\n \" if len(id_set) == 0 and self.score_threshold > 0:\\n\",\n \" return []\\n\",\n \" id_list = sorted(list(id_set))\\n\",\n \" id_lists = seperate_list(id_list)\\n\",\n \" for id_seq in id_lists:\\n\",\n \" for id in id_seq:\\n\",\n \" if id == id_seq[0]:\\n\",\n \" _id = self.index_to_docstore_id[id]\\n\",\n \" doc = self.docstore.search(_id)\\n\",\n \" else:\\n\",\n \" _id0 = self.index_to_docstore_id[id]\\n\",\n \" doc0 = self.docstore.search(_id0)\\n\",\n \" doc.page_content += \\\" \\\" + doc0.page_content\\n\",\n \" if not isinstance(doc, Document):\\n\",\n \" raise ValueError(f\\\"Could not find document for id {_id}, got {doc}\\\")\\n\",\n \" doc_score = min([scores[0][id] for id in [indices[0].tolist().index(i) for i in id_seq if i in indices[0]]])\\n\",\n \" doc.metadata[\\\"score\\\"] = int(doc_score)\\n\",\n \" docs.append((doc, doc_score))\\n\",\n \" return docs\\n\",\n \"\\n\",\n \"\\n\",\n \"if __name__ == '__main__':\\n\",\n \" # load docs\\n\",\n \" filepath = 'LLM_Survey_Chinese.pdf.txt'\\n\",\n \" # LLM name\\n\",\n \" LLM_TYPE = 'qwen'\\n\",\n \" # Embedding model name\\n\",\n \" EMBEDDING_MODEL = 'text2vec'\\n\",\n \" # 基于上下文的prompt模版，请务必保留\\\"{question}\\\"和\\\"{context_str}\\\"\\n\",\n \" PROMPT_TEMPLATE = \\\"\\\"\\\"已知信息：\\n\",\n \" {context_str} \\n\",\n \" 根据上述已知信息，简洁和专业的来回答用户的问题。如果无法从中得到答案，请说 “根据已知信息无法回答该问题” 或 “没有提供足够的相关信息”，不允许在答案中添加编造成分，答案请使用中文。 问题是：{question}\\\"\\\"\\\"\\n\",\n \" # Embedding running device\\n\",\n \" EMBEDDING_DEVICE = \\\"cuda\\\"\\n\",\n \" # return top-k text chunk from vector store\\n\",\n \" VECTOR_SEARCH_TOP_K = 3\\n\",\n \" # 文本分句长度\\n\",\n \" SENTENCE_SIZE = 50\\n\",\n \" CHAIN_TYPE = 'stuff'\\n\",\n \" llm_model_dict = {\\n\",\n \" \\\"qwen\\\": QWen,\\n\",\n \" }\\n\",\n \" embedding_model_dict = {\\n\",\n \" \\\"text2vec\\\": \\\"/tmp/GanymedeNil_text2vec-large-chinese\\\",\\n\",\n \" }\\n\",\n \" print(\\\"loading model start\\\")\\n\",\n \" llm = llm_model_dict[LLM_TYPE]()\\n\",\n \" embeddings = HuggingFaceEmbeddings(model_name=embedding_model_dict[EMBEDDING_MODEL],model_kwargs={'device': EMBEDDING_DEVICE})\\n\",\n \" print(\\\"loading model done\\\")\\n\",\n \"\\n\",\n \" print(\\\"loading documents start\\\")\\n\",\n \" docs = load_file(filepath, sentence_size=SENTENCE_SIZE)\\n\",\n \" print(\\\"loading documents done\\\")\\n\",\n \"\\n\",\n \" print(\\\"embedding start\\\")\\n\",\n \" docsearch = FAISSWrapper.from_documents(docs, embeddings)\\n\",\n \" print(\\\"embedding done\\\")\\n\",\n \"\\n\",\n \" print(\\\"loading qa start\\\")\\n\",\n \" prompt = PromptTemplate(\\n\",\n \" template=PROMPT_TEMPLATE, input_variables=[\\\"context_str\\\", \\\"question\\\"]\\n\",\n \" )\\n\",\n \"\\n\",\n \" chain_type_kwargs = {\\\"prompt\\\": prompt, \\\"document_variable_name\\\": \\\"context_str\\\"}\\n\",\n \" qa = RetrievalQA.from_chain_type(\\n\",\n \" llm=llm,\\n\",\n \" chain_type=CHAIN_TYPE, \\n\",\n \" retriever=docsearch.as_retriever(search_kwargs={\\\"k\\\": VECTOR_SEARCH_TOP_K}), \\n\",\n \" chain_type_kwargs=chain_type_kwargs)\\n\",\n \" print(\\\"loading qa done\\\")\\n\",\n \"\\n\",\n \" query = \\\"大模型指令微调有好的策略？\\\" \\n\",\n \" print(qa.run(query))\"\n ]\n }\n ],\n \"metadata\": {\n \"kernelspec\": {\n \"display_name\": \"Python 3 (ipykernel)\",\n \"language\": \"python\",\n \"name\": \"python3\"\n },\n \"language_info\": {\n \"codemirror_mode\": {\n \"name\": \"ipython\",\n \"version\": 3\n },\n \"file_extension\": \".py\",\n \"mimetype\": \"text/x-python\",\n \"name\": \"python\",\n \"nbconvert_exporter\": \"python\",\n \"pygments_lexer\": \"ipython3\",\n \"version\": \"3.9.15\"\n }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 5\n}\n" }, { "path": "recipes/finetune/ascend/README.md", "content": "# Fine-tuning Qwen by Ascend NPU\nBelow, we provide a simple example to show how to finetune Qwen by Ascend NPU. Currently, fine-tuning and inference are supported for Qwen 7B and 14B models. You can also refer to the official [mindformers](https://gitee.com/mindspore/mindformers/blob/dev/research/qwen/qwen.md) for detailed usage.\n\n## Environment Requirement\n\n- Hardware: Ascend 910A/B\n\n## Quickstart\n\n1. Launch Docker Image\n\n```bash\nImageID=pai-image-manage-registry.cn-wulanchabu.cr.aliyuncs.com/pai/llm-inference:qwen_v23.0.rc3\ndocker run -it -u root --ipc=host \\\n--device=/dev/davinci0 \\\n--device=/dev/davinci1 \\\n--device=/dev/davinci2 \\\n--device=/dev/davinci3 \\\n--device=/dev/davinci4 \\\n--device=/dev/davinci5 \\\n--device=/dev/davinci6 \\\n--device=/dev/davinci7 \\\n--device=/dev/davinci_manager \\\n--device=/dev/devmm_svm \\\n--device=/dev/hisi_hdc \\\n-v /usr/local/Ascend/driver:/usr/local/Ascend/driver \\\n-v /usr/local/Ascend/add-ons/:/usr/local/Ascend/add-ons/ \\\n-v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \\\n-v /usr/local/sbin/npu-smi:/usr/local/sbin/npu-smi \\\n-v /etc/ascend_install.info:/etc/ascend_install.info \\\n-v /var/log/npu/:/usr/slog \\\n-v /etc/hccn.conf:/etc/hccn.conf \\\n${ImageID} /bin/bash\n```\n\n2. Download and Convert model\n\n- download model by modelscope\n\n```bash\ncd mindformers\npython3 -c \"from modelscope.hub.snapshot_download import snapshot_download; snapshot_download('Qwen/Qwen-7B-Chat', cache_dir='.', revision='master')\"\n```\n\n- convert hf model weights to ckpt weights\n\n```bash\npython research/qwen/convert_weight.py \\\n --torch_ckpt_dir Qwen/Qwen-7B-Chat \\\n --mindspore_ckpt_path qwen-7b-chat.ckpt\n\nmkdir -vp load_checkpoint/rank_0\nmv qwen-7b-chat.ckpt load_checkpoint/rank_0/\n```\n\n3. Prepare training data\n\n- download demo data\n\n```bash\nwget -c https://pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com/alpaca_data_min.json\n```\n\n- Converts the raw data to the specified format\n\n```bash\npython research/qwen/alpaca_converter.py \\\n --data_path alpaca_data_min.json \\\n --output_path alpaca-data-conversation_min.json\n```\n\n- Generate Mindrecord data\n\n```bash\npython research/qwen/qwen_preprocess.py \\\n --input_glob alpaca-data-conversation_min.json \\\n --model_file Qwen/Qwen-7B-Chat/qwen.tiktoken \\\n --seq_length 1024 \\\n --output_file alpaca_min.mindrecord\n```\n\n4. Prepare RANK_TABLE_FILE\n\n```bash\n# generate RANK_TABLE_FILE with 8 npu\npython mindformers/tools/hccl_tools.py --device_num \"[0,8)\"\n```\n\n5. Fine-tune\n\nYou need to replace RANK_TABLE_FILE with the file generated in step 5.\n\n```bash\nexport MS_ASCEND_CHECK_OVERFLOW_MODE=INFNAN_MODE\nbash research/run_singlenode.sh \"python3 research/qwen/run_qwen.py \\\n--config research/qwen/run_qwen_7b.yaml \\\n--load_checkpoint /mindformers/research/qwen/load_checkpoint \\\n--vocab_file Qwen/Qwen-7B-Chat/qwen.tiktoken \\\n--use_parallel True \\\n--run_mode finetune \\\n--auto_trans_ckpt True \\\n--train_data alpaca_min.mindrecord\" \\\nRANK_TABLE_FILE [0,8] 8\n```\n\n6. Merge model weights\n\n- Rename model weights\n\n```bash\ncd output/checkpoint_network\nmv rank_0/qwen_rank_0-network.ckpt rank_0/checkpoint_0.ckpt\nmv rank_1/qwen_rank_1-network.ckpt rank_1/checkpoint_1.ckpt\nmv rank_2/qwen_rank_2-network.ckpt rank_2/checkpoint_2.ckpt\nmv rank_3/qwen_rank_3-network.ckpt rank_3/checkpoint_3.ckpt\nmv rank_4/qwen_rank_4-network.ckpt rank_4/checkpoint_4.ckpt\nmv rank_5/qwen_rank_5-network.ckpt rank_5/checkpoint_5.ckpt\nmv rank_6/qwen_rank_6-network.ckpt rank_6/checkpoint_6.ckpt\nmv rank_7/qwen_rank_7-network.ckpt rank_7/checkpoint_7.ckpt\ncd ../..\n```\n\n- Merge model weights\n\n```bash\npython mindformers/tools/transform_ckpt.py \\\n --src_ckpt_strategy output/strategy \\\n --src_ckpt_dir output/checkpoint_network \\\n --dst_ckpt_dir output/merged_model\n```\n\n7. Inference fine-tuned model\n\n```bash\npython research/qwen/run_qwen.py \\\n --config research/qwen/run_qwen_7b.yaml \\\n --predict_data '比较适合深度学习入门的书籍有' \\\n --run_mode predict \\\n --load_checkpoint output/merged_model/rank_0/checkpoint_0.ckpt \\\n --vocab_file Qwen/Qwen-7B-Chat/qwen.tiktoken \\\n --auto_trans_ckpt False \\\n --device_id 0\n```" }, { "path": "recipes/finetune/deepspeed/finetune_fullparameter_multi_gpu.ipynb", "content": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"id\": \"6e6981ab-2d9a-4280-923f-235a166855ba\",\n \"metadata\": {},\n \"source\": [\n \"# Fine-Tuning Qwen-Chat Large Language Model (Multiple GPUs)\\n\",\n \"\\n\",\n \"Tongyi Qianwen is a large language model developed by Alibaba Cloud based on the Transformer architecture, trained on an extensive set of pre-training data. The pre-training data is diverse and covers a wide range, including a large amount of internet text, specialized books, code, etc. In addition, an AI assistant called Qwen-Chat has been created based on the pre-trained model using alignment mechanism.\\n\",\n \"\\n\",\n \"This notebook uses Qwen-1.8B-Chat as an example to introduce how to fine-tune the Qianwen model using Deepspeed.\\n\",\n \"\\n\",\n \"## Environment Requirements\\n\",\n \"\\n\",\n \"Please refer to **requirements.txt** to install the required dependencies.\\n\",\n \"\\n\",\n \"## Preparation\\n\",\n \"\\n\",\n \"### Download Qwen-1.8B-Chat\\n\",\n \"\\n\",\n \"First, download the model files. You can choose to download directly from ModelScope.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"248488f9-4a86-4f35-9d56-50f8e91a8f11\",\n \"metadata\": {\n \"ExecutionIndicator\": {\n \"show\": true\n },\n \"tags\": []\n },\n \"outputs\": [],\n \"source\": [\n \"from modelscope.hub.snapshot_download import snapshot_download\\n\",\n \"model_dir = snapshot_download('Qwen/Qwen-1_8B-Chat', cache_dir='.', revision='master')\"\n ]\n },\n {\n \"attachments\": {},\n \"cell_type\": \"markdown\",\n \"id\": \"7b2a92b1-f08e-4413-9f92-8f23761e6e1f\",\n \"metadata\": {},\n \"source\": [\n \"### Download Example Training Data\\n\",\n \"\\n\",\n \"Download the data required for training; here, we provide a tiny dataset as an example. It is sampled from [Belle](https://github.com/LianjiaTech/BELLE).\\n\",\n \"\\n\",\n \"Disclaimer: the dataset can be only used for the research purpose.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"ce195f08-fbb2-470e-b6c0-9a03457458c7\",\n \"metadata\": {\n \"tags\": []\n },\n \"outputs\": [],\n \"source\": [\n \"!wget https://atp-modelzoo-sh.oss-cn-shanghai.aliyuncs.com/release/tutorials/qwen_recipes/Belle_sampled_qwen.json\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"7226bed0-171b-4d45-a3f9-b3d81ec2bb9f\",\n \"metadata\": {},\n \"source\": [\n \"You can also refer to this format to prepare the dataset. Below is a simple example list with 1 sample:\\n\",\n \"\\n\",\n \"```json\\n\",\n \"[\\n\",\n \" {\\n\",\n \" \\\"id\\\": \\\"identity_0\\\",\\n\",\n \" \\\"conversations\\\": [\\n\",\n \" {\\n\",\n \" \\\"from\\\": \\\"user\\\",\\n\",\n \" \\\"value\\\": \\\"你好\\\"\\n\",\n \" },\\n\",\n \" {\\n\",\n \" \\\"from\\\": \\\"assistant\\\",\\n\",\n \" \\\"value\\\": \\\"我是一个语言模型，我叫通义千问。\\\"\\n\",\n \" }\\n\",\n \" ]\\n\",\n \" }\\n\",\n \"]\\n\",\n \"```\\n\",\n \"\\n\",\n \"You can also use multi-turn conversations as the training set. Here is a simple example:\\n\",\n \"\\n\",\n \"```json\\n\",\n \"[\\n\",\n \" {\\n\",\n \" \\\"id\\\": \\\"identity_0\\\",\\n\",\n \" \\\"conversations\\\": [\\n\",\n \" {\\n\",\n \" \\\"from\\\": \\\"user\\\",\\n\",\n \" \\\"value\\\": \\\"你好，能告诉我遛狗的最佳时间吗？\\\"\\n\",\n \" },\\n\",\n \" {\\n\",\n \" \\\"from\\\": \\\"assistant\\\",\\n\",\n \" \\\"value\\\": \\\"当地最佳遛狗时间因地域差异而异，请问您所在的城市是哪里？\\\"\\n\",\n \" },\\n\",\n \" {\\n\",\n \" \\\"from\\\": \\\"user\\\",\\n\",\n \" \\\"value\\\": \\\"我在纽约市。\\\"\\n\",\n \" },\\n\",\n \" {\\n\",\n \" \\\"from\\\": \\\"assistant\\\",\\n\",\n \" \\\"value\\\": \\\"纽约市的遛狗最佳时间通常在早晨6点至8点和晚上8点至10点之间，因为这些时间段气温较低，遛狗更加舒适。但具体时间还需根据气候、气温和季节变化而定。\\\"\\n\",\n \" }\\n\",\n \" ]\\n\",\n \" }\\n\",\n \"]\\n\",\n \"```\\n\",\n \"\\n\",\n \"## Fine-Tune the Model\\n\",\n \"\\n\",\n \"You can directly run the prepared training script to fine-tune the model. **nproc_per_node** refers to the number of GPUs used fro training.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"7ab0581e-be85-45e6-a5b7-af9c42ea697b\",\n \"metadata\": {\n \"ExecutionIndicator\": {\n \"show\": true\n },\n \"tags\": []\n },\n \"outputs\": [],\n \"source\": [\n \"!torchrun --nproc_per_node 2 --nnodes 1 --node_rank 0 --master_addr localhost --master_port 6601 ../../finetune.py \\\\\\n\",\n \" --model_name_or_path \\\"Qwen/Qwen-1_8B-Chat/\\\" \\\\\\n\",\n \" --data_path \\\"Belle_sampled_qwen.json\\\" \\\\\\n\",\n \" --bf16 True \\\\\\n\",\n \" --output_dir \\\"output_qwen\\\" \\\\\\n\",\n \" --num_train_epochs 5 \\\\\\n\",\n \" --per_device_train_batch_size 1 \\\\\\n\",\n \" --per_device_eval_batch_size 1 \\\\\\n\",\n \" --gradient_accumulation_steps 16 \\\\\\n\",\n \" --evaluation_strategy \\\"no\\\" \\\\\\n\",\n \" --save_strategy \\\"steps\\\" \\\\\\n\",\n \" --save_steps 1000 \\\\\\n\",\n \" --save_total_limit 10 \\\\\\n\",\n \" --learning_rate 1e-5 \\\\\\n\",\n \" --weight_decay 0.1 \\\\\\n\",\n \" --adam_beta2 0.95 \\\\\\n\",\n \" --warmup_ratio 0.01 \\\\\\n\",\n \" --lr_scheduler_type \\\"cosine\\\" \\\\\\n\",\n \" --logging_steps 1 \\\\\\n\",\n \" --report_to \\\"none\\\" \\\\\\n\",\n \" --model_max_length 512 \\\\\\n\",\n \" --gradient_checkpointing True \\\\\\n\",\n \" --lazy_preprocess True \\\\\\n\",\n \" --deepspeed \\\"../../finetune/ds_config_zero2.json\\\"\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Test the Model\\n\",\n \"\\n\",\n \"We can test the model as follows:\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"from transformers import AutoModelForCausalLM, AutoTokenizer\\n\",\n \"from transformers.generation import GenerationConfig\\n\",\n \"\\n\",\n \"tokenizer = AutoTokenizer.from_pretrained(\\\"output_qwen\\\", trust_remote_code=True)\\n\",\n \"model = AutoModelForCausalLM.from_pretrained(\\n\",\n \" \\\"output_qwen\\\",\\n\",\n \" device_map=\\\"auto\\\",\\n\",\n \" trust_remote_code=True\\n\",\n \").eval()\\n\",\n \"\\n\",\n \"response, history = model.chat(tokenizer, \\\"你好\\\", history=None)\\n\",\n \"print(response)\"\n ]\n }\n ],\n \"metadata\": {\n \"kernelspec\": {\n \"display_name\": \"Python 3 (ipykernel)\",\n \"language\": \"python\",\n \"name\": \"python3\"\n },\n \"language_info\": {\n \"codemirror_mode\": {\n \"name\": \"ipython\",\n \"version\": 3\n },\n \"file_extension\": \".py\",\n \"mimetype\": \"text/x-python\",\n \"name\": \"python\",\n \"nbconvert_exporter\": \"python\",\n \"pygments_lexer\": \"ipython3\",\n \"version\": \"3.10.13\"\n }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 5\n}\n" }, { "path": "recipes/finetune/deepspeed/finetune_fullparameter_single_gpu.ipynb", "content": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"id\": \"6e6981ab-2d9a-4280-923f-235a166855ba\",\n \"metadata\": {},\n \"source\": [\n \"# Fine-Tuning Qwen-Chat Large Language Model (Single GPU)\\n\",\n \"\\n\",\n \"Tongyi Qianwen is a large language model developed by Alibaba Cloud based on the Transformer architecture, trained on an extensive set of pre-training data. The pre-training data is diverse and covers a wide range, including a large amount of internet text, specialized books, code, etc. In addition, an AI assistant called Qwen-Chat has been created based on the pre-trained model using alignment mechanism.\\n\",\n \"\\n\",\n \"This notebook uses Qwen-1.8B-Chat as an example to introduce how to fine-tune the Qianwen model using Deepspeed.\\n\",\n \"\\n\",\n \"## Environment Requirements\\n\",\n \"\\n\",\n \"Please refer to **requirements.txt** to install the required dependencies.\\n\",\n \"\\n\",\n \"## Preparation\\n\",\n \"\\n\",\n \"### Download Qwen-1.8B-Chat\\n\",\n \"\\n\",\n \"First, download the model files. You can choose to download directly from ModelScope.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"248488f9-4a86-4f35-9d56-50f8e91a8f11\",\n \"metadata\": {\n \"ExecutionIndicator\": {\n \"show\": true\n },\n \"execution\": {\n \"iopub.execute_input\": \"2023-12-31T03:19:11.059814Z\",\n \"iopub.status.busy\": \"2023-12-31T03:19:11.059177Z\",\n \"iopub.status.idle\": \"2023-12-31T03:21:54.157827Z\",\n \"shell.execute_reply\": \"2023-12-31T03:21:54.157333Z\",\n \"shell.execute_reply.started\": \"2023-12-31T03:19:11.059783Z\"\n },\n \"tags\": []\n },\n \"outputs\": [],\n \"source\": [\n \"from modelscope.hub.snapshot_download import snapshot_download\\n\",\n \"model_dir = snapshot_download('Qwen/Qwen-1_8B-Chat', cache_dir='.', revision='master')\"\n ]\n },\n {\n \"attachments\": {},\n \"cell_type\": \"markdown\",\n \"id\": \"7b2a92b1-f08e-4413-9f92-8f23761e6e1f\",\n \"metadata\": {},\n \"source\": [\n \"### Download Example Training Data\\n\",\n \"\\n\",\n \"Download the data required for training; here, we provide a tiny dataset as an example. It is sampled from [Belle](https://github.com/LianjiaTech/BELLE).\\n\",\n \"\\n\",\n \"Disclaimer: the dataset can be only used for the research purpose.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"ce195f08-fbb2-470e-b6c0-9a03457458c7\",\n \"metadata\": {\n \"execution\": {\n \"iopub.execute_input\": \"2023-12-31T03:21:57.596577Z\",\n \"iopub.status.busy\": \"2023-12-31T03:21:57.595847Z\",\n \"iopub.status.idle\": \"2023-12-31T03:21:57.971112Z\",\n \"shell.execute_reply\": \"2023-12-31T03:21:57.970576Z\",\n \"shell.execute_reply.started\": \"2023-12-31T03:21:57.596555Z\"\n },\n \"tags\": []\n },\n \"outputs\": [],\n \"source\": [\n \"!wget https://atp-modelzoo-sh.oss-cn-shanghai.aliyuncs.com/release/tutorials/qwen_recipes/Belle_sampled_qwen.json\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"7226bed0-171b-4d45-a3f9-b3d81ec2bb9f\",\n \"metadata\": {},\n \"source\": [\n \"You can also refer to this format to prepare the dataset. Below is a simple example list with 1 sample:\\n\",\n \"\\n\",\n \"```json\\n\",\n \"[\\n\",\n \" {\\n\",\n \" \\\"id\\\": \\\"identity_0\\\",\\n\",\n \" \\\"conversations\\\": [\\n\",\n \" {\\n\",\n \" \\\"from\\\": \\\"user\\\",\\n\",\n \" \\\"value\\\": \\\"你好\\\"\\n\",\n \" },\\n\",\n \" {\\n\",\n \" \\\"from\\\": \\\"assistant\\\",\\n\",\n \" \\\"value\\\": \\\"我是一个语言模型，我叫通义千问。\\\"\\n\",\n \" }\\n\",\n \" ]\\n\",\n \" }\\n\",\n \"]\\n\",\n \"```\\n\",\n \"\\n\",\n \"You can also use multi-turn conversations as the training set. Here is a simple example:\\n\",\n \"\\n\",\n \"```json\\n\",\n \"[\\n\",\n \" {\\n\",\n \" \\\"id\\\": \\\"identity_0\\\",\\n\",\n \" \\\"conversations\\\": [\\n\",\n \" {\\n\",\n \" \\\"from\\\": \\\"user\\\",\\n\",\n \" \\\"value\\\": \\\"你好，能告诉我遛狗的最佳时间吗？\\\"\\n\",\n \" },\\n\",\n \" {\\n\",\n \" \\\"from\\\": \\\"assistant\\\",\\n\",\n \" \\\"value\\\": \\\"当地最佳遛狗时间因地域差异而异，请问您所在的城市是哪里？\\\"\\n\",\n \" },\\n\",\n \" {\\n\",\n \" \\\"from\\\": \\\"user\\\",\\n\",\n \" \\\"value\\\": \\\"我在纽约市。\\\"\\n\",\n \" },\\n\",\n \" {\\n\",\n \" \\\"from\\\": \\\"assistant\\\",\\n\",\n \" \\\"value\\\": \\\"纽约市的遛狗最佳时间通常在早晨6点至8点和晚上8点至10点之间，因为这些时间段气温较低，遛狗更加舒适。但具体时间还需根据气候、气温和季节变化而定。\\\"\\n\",\n \" }\\n\",\n \" ]\\n\",\n \" }\\n\",\n \"]\\n\",\n \"```\\n\",\n \"\\n\",\n \"\\n\",\n \"## Fine-Tune the Model\\n\",\n \"\\n\",\n \"You can directly run the prepared training script to fine-tune the model.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"7ab0581e-be85-45e6-a5b7-af9c42ea697b\",\n \"metadata\": {\n \"ExecutionIndicator\": {\n \"show\": true\n },\n \"execution\": {\n \"iopub.execute_input\": \"2023-12-31T03:23:52.455178Z\",\n \"iopub.status.busy\": \"2023-12-31T03:23:52.454615Z\",\n \"iopub.status.idle\": \"2023-12-31T03:24:15.699948Z\",\n \"shell.execute_reply\": \"2023-12-31T03:24:15.699358Z\",\n \"shell.execute_reply.started\": \"2023-12-31T03:23:52.455144Z\"\n },\n \"tags\": []\n },\n \"outputs\": [],\n \"source\": [\n \"!python ../../finetune.py \\\\\\n\",\n \" --model_name_or_path \\\"Qwen/Qwen-1_8B-Chat/\\\"\\\\\\n\",\n \" --data_path \\\"Belle_sampled_qwen.json\\\"\\\\\\n\",\n \" --bf16 \\\\\\n\",\n \" --output_dir \\\"output_qwen\\\" \\\\\\n\",\n \" --num_train_epochs 5 \\\\\\n\",\n \" --per_device_train_batch_size 1 \\\\\\n\",\n \" --per_device_eval_batch_size 1 \\\\\\n\",\n \" --gradient_accumulation_steps 16 \\\\\\n\",\n \" --evaluation_strategy \\\"no\\\" \\\\\\n\",\n \" --save_strategy \\\"steps\\\" \\\\\\n\",\n \" --save_steps 1000 \\\\\\n\",\n \" --save_total_limit 10 \\\\\\n\",\n \" --learning_rate 1e-5 \\\\\\n\",\n \" --weight_decay 0.1 \\\\\\n\",\n \" --adam_beta2 0.95 \\\\\\n\",\n \" --warmup_ratio 0.01 \\\\\\n\",\n \" --lr_scheduler_type \\\"cosine\\\" \\\\\\n\",\n \" --logging_steps 1 \\\\\\n\",\n \" --report_to \\\"none\\\" \\\\\\n\",\n \" --model_max_length 512 \\\\\\n\",\n \" --gradient_checkpointing \\\\\\n\",\n \" --lazy_preprocess\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Test the Model\\n\",\n \"\\n\",\n \"We can test the model as follows:\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"from transformers import AutoModelForCausalLM, AutoTokenizer\\n\",\n \"from transformers.generation import GenerationConfig\\n\",\n \"\\n\",\n \"tokenizer = AutoTokenizer.from_pretrained(\\\"output_qwen\\\", trust_remote_code=True)\\n\",\n \"model = AutoModelForCausalLM.from_pretrained(\\n\",\n \" \\\"output_qwen\\\",\\n\",\n \" device_map=\\\"auto\\\",\\n\",\n \" trust_remote_code=True\\n\",\n \").eval()\\n\",\n \"\\n\",\n \"response, history = model.chat(tokenizer, \\\"你好\\\", history=None)\\n\",\n \"print(response)\"\n ]\n }\n ],\n \"metadata\": {\n \"kernelspec\": {\n \"display_name\": \"Python 3 (ipykernel)\",\n \"language\": \"python\",\n \"name\": \"python3\"\n },\n \"language_info\": {\n \"codemirror_mode\": {\n \"name\": \"ipython\",\n \"version\": 3\n },\n \"file_extension\": \".py\",\n \"mimetype\": \"text/x-python\",\n \"name\": \"python\",\n \"nbconvert_exporter\": \"python\",\n \"pygments_lexer\": \"ipython3\",\n \"version\": \"3.10.13\"\n }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 5\n}\n" }, { "path": "recipes/finetune/deepspeed/finetune_lora_multi_gpu.ipynb", "content": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"id\": \"6e6981ab-2d9a-4280-923f-235a166855ba\",\n \"metadata\": {},\n \"source\": [\n \"# LoRA Fine-Tuning Qwen-Chat Large Language Model (Multiple GPUs)\\n\",\n \"\\n\",\n \"Tongyi Qianwen is a large language model developed by Alibaba Cloud based on the Transformer architecture, trained on an extensive set of pre-training data. The pre-training data is diverse and covers a wide range, including a large amount of internet text, specialized books, code, etc. In addition, an AI assistant called Qwen-Chat has been created based on the pre-trained model using alignment mechanism.\\n\",\n \"\\n\",\n \"This notebook uses Qwen-1.8B-Chat as an example to introduce how to LoRA fine-tune the Qianwen model using Deepspeed.\\n\",\n \"\\n\",\n \"## Environment Requirements\\n\",\n \"\\n\",\n \"Please refer to **requirements.txt** to install the required dependencies.\\n\",\n \"\\n\",\n \"## Preparation\\n\",\n \"\\n\",\n \"### Download Qwen-1.8B-Chat\\n\",\n \"\\n\",\n \"First, download the model files. You can choose to download directly from ModelScope.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"248488f9-4a86-4f35-9d56-50f8e91a8f11\",\n \"metadata\": {\n \"ExecutionIndicator\": {\n \"show\": true\n },\n \"tags\": []\n },\n \"outputs\": [],\n \"source\": [\n \"from modelscope.hub.snapshot_download import snapshot_download\\n\",\n \"model_dir = snapshot_download('Qwen/Qwen-1_8B-Chat', cache_dir='.', revision='master')\"\n ]\n },\n {\n \"attachments\": {},\n \"cell_type\": \"markdown\",\n \"id\": \"7b2a92b1-f08e-4413-9f92-8f23761e6e1f\",\n \"metadata\": {},\n \"source\": [\n \"### Download Example Training Data\\n\",\n \"\\n\",\n \"Download the data required for training; here, we provide a tiny dataset as an example. It is sampled from [Belle](https://github.com/LianjiaTech/BELLE).\\n\",\n \"\\n\",\n \"Disclaimer: the dataset can be only used for the research purpose.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"ce195f08-fbb2-470e-b6c0-9a03457458c7\",\n \"metadata\": {\n \"tags\": []\n },\n \"outputs\": [],\n \"source\": [\n \"!wget https://atp-modelzoo-sh.oss-cn-shanghai.aliyuncs.com/release/tutorials/qwen_recipes/Belle_sampled_qwen.json\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"7226bed0-171b-4d45-a3f9-b3d81ec2bb9f\",\n \"metadata\": {},\n \"source\": [\n \"You can also refer to this format to prepare the dataset. Below is a simple example list with 1 sample:\\n\",\n \"\\n\",\n \"```json\\n\",\n \"[\\n\",\n \" {\\n\",\n \" \\\"id\\\": \\\"identity_0\\\",\\n\",\n \" \\\"conversations\\\": [\\n\",\n \" {\\n\",\n \" \\\"from\\\": \\\"user\\\",\\n\",\n \" \\\"value\\\": \\\"你好\\\"\\n\",\n \" },\\n\",\n \" {\\n\",\n \" \\\"from\\\": \\\"assistant\\\",\\n\",\n \" \\\"value\\\": \\\"我是一个语言模型，我叫通义千问。\\\"\\n\",\n \" }\\n\",\n \" ]\\n\",\n \" }\\n\",\n \"]\\n\",\n \"```\\n\",\n \"\\n\",\n \"You can also use multi-turn conversations as the training set. Here is a simple example:\\n\",\n \"\\n\",\n \"```json\\n\",\n \"[\\n\",\n \" {\\n\",\n \" \\\"id\\\": \\\"identity_0\\\",\\n\",\n \" \\\"conversations\\\": [\\n\",\n \" {\\n\",\n \" \\\"from\\\": \\\"user\\\",\\n\",\n \" \\\"value\\\": \\\"你好，能告诉我遛狗的最佳时间吗？\\\"\\n\",\n \" },\\n\",\n \" {\\n\",\n \" \\\"from\\\": \\\"assistant\\\",\\n\",\n \" \\\"value\\\": \\\"当地最佳遛狗时间因地域差异而异，请问您所在的城市是哪里？\\\"\\n\",\n \" },\\n\",\n \" {\\n\",\n \" \\\"from\\\": \\\"user\\\",\\n\",\n \" \\\"value\\\": \\\"我在纽约市。\\\"\\n\",\n \" },\\n\",\n \" {\\n\",\n \" \\\"from\\\": \\\"assistant\\\",\\n\",\n \" \\\"value\\\": \\\"纽约市的遛狗最佳时间通常在早晨6点至8点和晚上8点至10点之间，因为这些时间段气温较低，遛狗更加舒适。但具体时间还需根据气候、气温和季节变化而定。\\\"\\n\",\n \" }\\n\",\n \" ]\\n\",\n \" }\\n\",\n \"]\\n\",\n \"```\\n\",\n \"\\n\",\n \"## Fine-Tune the Model\\n\",\n \"\\n\",\n \"You can directly run the prepared training script to fine-tune the model. **nproc_per_node** refers to the number of GPUs used fro training.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"7ab0581e-be85-45e6-a5b7-af9c42ea697b\",\n \"metadata\": {\n \"ExecutionIndicator\": {\n \"show\": true\n },\n \"tags\": []\n },\n \"outputs\": [],\n \"source\": [\n \"!torchrun --nproc_per_node 2 --nnodes 1 --node_rank 0 --master_addr localhost --master_port 6601 ../../finetune.py \\\\\\n\",\n \" --model_name_or_path \\\"Qwen/Qwen-1_8B-Chat/\\\" \\\\\\n\",\n \" --data_path \\\"Belle_sampled_qwen.json\\\" \\\\\\n\",\n \" --bf16 True \\\\\\n\",\n \" --output_dir \\\"output_qwen\\\" \\\\\\n\",\n \" --num_train_epochs 5 \\\\\\n\",\n \" --per_device_train_batch_size 1 \\\\\\n\",\n \" --per_device_eval_batch_size 1 \\\\\\n\",\n \" --gradient_accumulation_steps 16 \\\\\\n\",\n \" --evaluation_strategy \\\"no\\\" \\\\\\n\",\n \" --save_strategy \\\"steps\\\" \\\\\\n\",\n \" --save_steps 1000 \\\\\\n\",\n \" --save_total_limit 10 \\\\\\n\",\n \" --learning_rate 1e-5 \\\\\\n\",\n \" --weight_decay 0.1 \\\\\\n\",\n \" --adam_beta2 0.95 \\\\\\n\",\n \" --warmup_ratio 0.01 \\\\\\n\",\n \" --lr_scheduler_type \\\"cosine\\\" \\\\\\n\",\n \" --logging_steps 1 \\\\\\n\",\n \" --report_to \\\"none\\\" \\\\\\n\",\n \" --model_max_length 512 \\\\\\n\",\n \" --gradient_checkpointing True \\\\\\n\",\n \" --lazy_preprocess True \\\\\\n\",\n \" --deepspeed \\\"../../finetune/ds_config_zero2.json\\\" \\\\\\n\",\n \" --use_lora\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"35acf008-1dfe-4d32-8cf5-7022e042aadb\",\n \"metadata\": {},\n \"source\": [\n \"## Merge Weights\\n\",\n \"\\n\",\n \"The training of both LoRA and Q-LoRA only saves the adapter parameters. You can load the fine-tuned model and merge weights as shown below:\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"61021499-4a44-45af-a682-943ed63c2fcb\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"from transformers import AutoModelForCausalLM\\n\",\n \"from peft import PeftModel\\n\",\n \"import torch\\n\",\n \"\\n\",\n \"model = AutoModelForCausalLM.from_pretrained(\\\"Qwen/Qwen-1_8B-Chat/\\\", torch_dtype=torch.float16, device_map=\\\"auto\\\", trust_remote_code=True)\\n\",\n \"model = PeftModel.from_pretrained(model, \\\"output_qwen/\\\")\\n\",\n \"merged_model = model.merge_and_unload()\\n\",\n \"merged_model.save_pretrained(\\\"output_qwen_merged\\\", max_shard_size=\\\"2048MB\\\", safe_serialization=True)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"0dfbd261-6451-4532-82e8-3ae19ed93ee1\",\n \"metadata\": {},\n \"source\": [\n \"The tokenizer files are not saved in the new directory in this step. You can copy the tokenizer files or use the following code:\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"ddcba069-340b-4a93-a145-2028b425dd23\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"from transformers import AutoTokenizer\\n\",\n \"\\n\",\n \"tokenizer = AutoTokenizer.from_pretrained(\\n\",\n \" \\\"Qwen/Qwen-1_8B-Chat/\\\",\\n\",\n \" trust_remote_code=True\\n\",\n \")\\n\",\n \"\\n\",\n \"tokenizer.save_pretrained(\\\"output_qwen_merged\\\")\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"fe9f2878-79d3-4b1c-ba95-ac2f73aa6e1b\",\n \"metadata\": {},\n \"source\": [\n \"## Test the Model\\n\",\n \"\\n\",\n \"After merging the weights, we can test the model as follows:\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"from transformers import AutoModelForCausalLM, AutoTokenizer\\n\",\n \"from transformers.generation import GenerationConfig\\n\",\n \"\\n\",\n \"tokenizer = AutoTokenizer.from_pretrained(\\\"output_qwen_merged\\\", trust_remote_code=True)\\n\",\n \"model = AutoModelForCausalLM.from_pretrained(\\n\",\n \" \\\"output_qwen_merged\\\",\\n\",\n \" device_map=\\\"auto\\\",\\n\",\n \" trust_remote_code=True\\n\",\n \").eval()\\n\",\n \"\\n\",\n \"response, history = model.chat(tokenizer, \\\"你好\\\", history=None)\\n\",\n \"print(response)\"\n ]\n }\n ],\n \"metadata\": {\n \"kernelspec\": {\n \"display_name\": \"Python 3 (ipykernel)\",\n \"language\": \"python\",\n \"name\": \"python3\"\n },\n \"language_info\": {\n \"codemirror_mode\": {\n \"name\": \"ipython\",\n \"version\": 3\n },\n \"file_extension\": \".py\",\n \"mimetype\": \"text/x-python\",\n \"name\": \"python\",\n \"nbconvert_exporter\": \"python\",\n \"pygments_lexer\": \"ipython3\",\n \"version\": \"3.10.13\"\n }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 5\n}\n" }, { "path": "recipes/finetune/deepspeed/finetune_lora_single_gpu.ipynb", "content": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"id\": \"6e6981ab-2d9a-4280-923f-235a166855ba\",\n \"metadata\": {},\n \"source\": [\n \"# LoRA Fine-Tuning Qwen-Chat Large Language Model (Single GPU)\\n\",\n \"\\n\",\n \"Tongyi Qianwen is a large language model developed by Alibaba Cloud based on the Transformer architecture, trained on an extensive set of pre-training data. The pre-training data is diverse and covers a wide range, including a large amount of internet text, specialized books, code, etc. In addition, an AI assistant called Qwen-Chat has been created based on the pre-trained model using alignment mechanism.\\n\",\n \"\\n\",\n \"This notebook uses Qwen-1.8B-Chat as an example to introduce how to LoRA fine-tune the Qianwen model using Deepspeed.\\n\",\n \"\\n\",\n \"## Environment Requirements\\n\",\n \"\\n\",\n \"Please refer to **requirements.txt** to install the required dependencies.\\n\",\n \"\\n\",\n \"## Preparation\\n\",\n \"\\n\",\n \"### Download Qwen-1.8B-Chat\\n\",\n \"\\n\",\n \"First, download the model files. You can choose to download directly from ModelScope.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"248488f9-4a86-4f35-9d56-50f8e91a8f11\",\n \"metadata\": {\n \"ExecutionIndicator\": {\n \"show\": true\n },\n \"tags\": []\n },\n \"outputs\": [],\n \"source\": [\n \"from modelscope.hub.snapshot_download import snapshot_download\\n\",\n \"model_dir = snapshot_download('Qwen/Qwen-1_8B-Chat', cache_dir='.', revision='master')\"\n ]\n },\n {\n \"attachments\": {},\n \"cell_type\": \"markdown\",\n \"id\": \"7b2a92b1-f08e-4413-9f92-8f23761e6e1f\",\n \"metadata\": {},\n \"source\": [\n \"### Download Example Training Data\\n\",\n \"\\n\",\n \"Download the data required for training; here, we provide a tiny dataset as an example. It is sampled from [Belle](https://github.com/LianjiaTech/BELLE).\\n\",\n \"\\n\",\n \"Disclaimer: the dataset can be only used for the research purpose.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"ce195f08-fbb2-470e-b6c0-9a03457458c7\",\n \"metadata\": {\n \"tags\": []\n },\n \"outputs\": [],\n \"source\": [\n \"!wget https://atp-modelzoo-sh.oss-cn-shanghai.aliyuncs.com/release/tutorials/qwen_recipes/Belle_sampled_qwen.json\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"7226bed0-171b-4d45-a3f9-b3d81ec2bb9f\",\n \"metadata\": {},\n \"source\": [\n \"You can also refer to this format to prepare the dataset. Below is a simple example list with 1 sample:\\n\",\n \"\\n\",\n \"```json\\n\",\n \"[\\n\",\n \" {\\n\",\n \" \\\"id\\\": \\\"identity_0\\\",\\n\",\n \" \\\"conversations\\\": [\\n\",\n \" {\\n\",\n \" \\\"from\\\": \\\"user\\\",\\n\",\n \" \\\"value\\\": \\\"你好\\\"\\n\",\n \" },\\n\",\n \" {\\n\",\n \" \\\"from\\\": \\\"assistant\\\",\\n\",\n \" \\\"value\\\": \\\"我是一个语言模型，我叫通义千问。\\\"\\n\",\n \" }\\n\",\n \" ]\\n\",\n \" }\\n\",\n \"]\\n\",\n \"```\\n\",\n \"\\n\",\n \"You can also use multi-turn conversations as the training set. Here is a simple example:\\n\",\n \"\\n\",\n \"```json\\n\",\n \"[\\n\",\n \" {\\n\",\n \" \\\"id\\\": \\\"identity_0\\\",\\n\",\n \" \\\"conversations\\\": [\\n\",\n \" {\\n\",\n \" \\\"from\\\": \\\"user\\\",\\n\",\n \" \\\"value\\\": \\\"你好，能告诉我遛狗的最佳时间吗？\\\"\\n\",\n \" },\\n\",\n \" {\\n\",\n \" \\\"from\\\": \\\"assistant\\\",\\n\",\n \" \\\"value\\\": \\\"当地最佳遛狗时间因地域差异而异，请问您所在的城市是哪里？\\\"\\n\",\n \" },\\n\",\n \" {\\n\",\n \" \\\"from\\\": \\\"user\\\",\\n\",\n \" \\\"value\\\": \\\"我在纽约市。\\\"\\n\",\n \" },\\n\",\n \" {\\n\",\n \" \\\"from\\\": \\\"assistant\\\",\\n\",\n \" \\\"value\\\": \\\"纽约市的遛狗最佳时间通常在早晨6点至8点和晚上8点至10点之间，因为这些时间段气温较低，遛狗更加舒适。但具体时间还需根据气候、气温和季节变化而定。\\\"\\n\",\n \" }\\n\",\n \" ]\\n\",\n \" }\\n\",\n \"]\\n\",\n \"```\\n\",\n \"\\n\",\n \"## Fine-Tune the Model\\n\",\n \"\\n\",\n \"You can directly run the prepared training script to fine-tune the model.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"7ab0581e-be85-45e6-a5b7-af9c42ea697b\",\n \"metadata\": {\n \"ExecutionIndicator\": {\n \"show\": true\n },\n \"tags\": []\n },\n \"outputs\": [],\n \"source\": [\n \"!export CUDA_VISIBLE_DEVICES=0\\n\",\n \"!python ../../finetune.py \\\\\\n\",\n \" --model_name_or_path \\\"Qwen/Qwen-1_8B-Chat/\\\"\\\\\\n\",\n \" --data_path \\\"Belle_sampled_qwen.json\\\"\\\\\\n\",\n \" --bf16 \\\\\\n\",\n \" --output_dir \\\"output_qwen\\\" \\\\\\n\",\n \" --num_train_epochs 5 \\\\\\n\",\n \" --per_device_train_batch_size 1 \\\\\\n\",\n \" --per_device_eval_batch_size 1 \\\\\\n\",\n \" --gradient_accumulation_steps 16 \\\\\\n\",\n \" --evaluation_strategy \\\"no\\\" \\\\\\n\",\n \" --save_strategy \\\"steps\\\" \\\\\\n\",\n \" --save_steps 1000 \\\\\\n\",\n \" --save_total_limit 10 \\\\\\n\",\n \" --learning_rate 1e-5 \\\\\\n\",\n \" --weight_decay 0.1 \\\\\\n\",\n \" --adam_beta2 0.95 \\\\\\n\",\n \" --warmup_ratio 0.01 \\\\\\n\",\n \" --lr_scheduler_type \\\"cosine\\\" \\\\\\n\",\n \" --logging_steps 1 \\\\\\n\",\n \" --report_to \\\"none\\\" \\\\\\n\",\n \" --model_max_length 512 \\\\\\n\",\n \" --gradient_checkpointing \\\\\\n\",\n \" --lazy_preprocess \\\\\\n\",\n \" --use_lora\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"5e6f28aa-1772-48ce-aa15-8cf29e7d67b5\",\n \"metadata\": {},\n \"source\": [\n \"## Merge Weights\\n\",\n \"\\n\",\n \"The training of both LoRA and Q-LoRA only saves the adapter parameters. You can load the fine-tuned model and merge weights as shown below:\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"4fd5ef2a-34f9-4909-bebe-7b3b086fd16a\",\n \"metadata\": {\n \"ExecutionIndicator\": {\n \"show\": true\n },\n \"tags\": []\n },\n \"outputs\": [],\n \"source\": [\n \"from transformers import AutoModelForCausalLM\\n\",\n \"from peft import PeftModel\\n\",\n \"import torch\\n\",\n \"\\n\",\n \"model = AutoModelForCausalLM.from_pretrained(\\\"Qwen/Qwen-1_8B-Chat/\\\", torch_dtype=torch.float16, device_map=\\\"auto\\\", trust_remote_code=True)\\n\",\n \"model = PeftModel.from_pretrained(model, \\\"output_qwen/\\\")\\n\",\n \"merged_model = model.merge_and_unload()\\n\",\n \"merged_model.save_pretrained(\\\"output_qwen_merged\\\", max_shard_size=\\\"2048MB\\\", safe_serialization=True)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"2e3f5b9f-63a1-4599-8d9b-a8d8f764838f\",\n \"metadata\": {},\n \"source\": [\n \"The tokenizer files are not saved in the new directory in this step. You can copy the tokenizer files or use the following code:\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"10fa5ea3-dd55-4901-86af-c045d4c56533\",\n \"metadata\": {\n \"tags\": []\n },\n \"outputs\": [],\n \"source\": [\n \"from transformers import AutoTokenizer\\n\",\n \"\\n\",\n \"tokenizer = AutoTokenizer.from_pretrained(\\n\",\n \" \\\"Qwen/Qwen-1_8B-Chat/\\\",\\n\",\n \" trust_remote_code=True\\n\",\n \")\\n\",\n \"\\n\",\n \"tokenizer.save_pretrained(\\\"output_qwen_merged\\\")\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"804b84d8\",\n \"metadata\": {},\n \"source\": [\n \"## Test the Model\\n\",\n \"\\n\",\n \"After merging the weights, we can test the model as follows:\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"from transformers import AutoModelForCausalLM, AutoTokenizer\\n\",\n \"from transformers.generation import GenerationConfig\\n\",\n \"\\n\",\n \"tokenizer = AutoTokenizer.from_pretrained(\\\"output_qwen_merged\\\", trust_remote_code=True)\\n\",\n \"model = AutoModelForCausalLM.from_pretrained(\\n\",\n \" \\\"output_qwen_merged\\\",\\n\",\n \" device_map=\\\"auto\\\",\\n\",\n \" trust_remote_code=True\\n\",\n \").eval()\\n\",\n \"\\n\",\n \"response, history = model.chat(tokenizer, \\\"你好\\\", history=None)\\n\",\n \"print(response)\"\n ]\n }\n ],\n \"metadata\": {\n \"kernelspec\": {\n \"display_name\": \"Python 3 (ipykernel)\",\n \"language\": \"python\",\n \"name\": \"python3\"\n },\n \"language_info\": {\n \"codemirror_mode\": {\n \"name\": \"ipython\",\n \"version\": 3\n },\n \"file_extension\": \".py\",\n \"mimetype\": \"text/x-python\",\n \"name\": \"python\",\n \"nbconvert_exporter\": \"python\",\n \"pygments_lexer\": \"ipython3\",\n \"version\": \"3.10.13\"\n }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 5\n}\n" }, { "path": "recipes/finetune/deepspeed/finetune_qlora_multi_gpu.ipynb", "content": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"id\": \"6e6981ab-2d9a-4280-923f-235a166855ba\",\n \"metadata\": {},\n \"source\": [\n \"# QLoRA Fine-Tuning Qwen-Chat Large Language Model (Multiple GPUs)\\n\",\n \"\\n\",\n \"Tongyi Qianwen is a large language model developed by Alibaba Cloud based on the Transformer architecture, trained on an extensive set of pre-training data. The pre-training data is diverse and covers a wide range, including a large amount of internet text, specialized books, code, etc. In addition, an AI assistant called Qwen-Chat has been created based on the pre-trained model using alignment mechanism.\\n\",\n \"\\n\",\n \"This notebook uses Qwen-1.8B-Chat as an example to introduce how to QLoRA fine-tune the Qianwen model using Deepspeed.\\n\",\n \"\\n\",\n \"## Environment Requirements\\n\",\n \"\\n\",\n \"Please refer to **requirements.txt** to install the required dependencies.\\n\",\n \"\\n\",\n \"## Preparation\\n\",\n \"\\n\",\n \"### Download Qwen-1.8B-Chat\\n\",\n \"\\n\",\n \"First, download the model files. You can choose to download directly from ModelScope.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"248488f9-4a86-4f35-9d56-50f8e91a8f11\",\n \"metadata\": {\n \"ExecutionIndicator\": {\n \"show\": true\n },\n \"execution\": {\n \"iopub.execute_input\": \"2023-12-31T08:42:52.842315Z\",\n \"iopub.status.busy\": \"2023-12-31T08:42:52.841665Z\",\n \"iopub.status.idle\": \"2023-12-31T08:44:19.832661Z\",\n \"shell.execute_reply\": \"2023-12-31T08:44:19.832193Z\",\n \"shell.execute_reply.started\": \"2023-12-31T08:42:52.842295Z\"\n },\n \"tags\": []\n },\n \"outputs\": [],\n \"source\": [\n \"from modelscope.hub.snapshot_download import snapshot_download\\n\",\n \"model_dir = snapshot_download('Qwen/Qwen-1_8B-Chat-Int4', cache_dir='.', revision='master')\"\n ]\n },\n {\n \"attachments\": {},\n \"cell_type\": \"markdown\",\n \"id\": \"7b2a92b1-f08e-4413-9f92-8f23761e6e1f\",\n \"metadata\": {},\n \"source\": [\n \"### Download Example Training Data\\n\",\n \"\\n\",\n \"Download the data required for training; here, we provide a tiny dataset as an example. It is sampled from [Belle](https://github.com/LianjiaTech/BELLE).\\n\",\n \"\\n\",\n \"Disclaimer: the dataset can be only used for the research purpose.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"ce195f08-fbb2-470e-b6c0-9a03457458c7\",\n \"metadata\": {\n \"tags\": []\n },\n \"outputs\": [],\n \"source\": [\n \"!wget https://atp-modelzoo-sh.oss-cn-shanghai.aliyuncs.com/release/tutorials/qwen_recipes/Belle_sampled_qwen.json\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"7226bed0-171b-4d45-a3f9-b3d81ec2bb9f\",\n \"metadata\": {},\n \"source\": [\n \"You can also refer to this format to prepare the dataset. Below is a simple example list with 1 sample:\\n\",\n \"\\n\",\n \"```json\\n\",\n \"[\\n\",\n \" {\\n\",\n \" \\\"id\\\": \\\"identity_0\\\",\\n\",\n \" \\\"conversations\\\": [\\n\",\n \" {\\n\",\n \" \\\"from\\\": \\\"user\\\",\\n\",\n \" \\\"value\\\": \\\"你好\\\"\\n\",\n \" },\\n\",\n \" {\\n\",\n \" \\\"from\\\": \\\"assistant\\\",\\n\",\n \" \\\"value\\\": \\\"我是一个语言模型，我叫通义千问。\\\"\\n\",\n \" }\\n\",\n \" ]\\n\",\n \" }\\n\",\n \"]\\n\",\n \"```\\n\",\n \"\\n\",\n \"You can also use multi-turn conversations as the training set. Here is a simple example:\\n\",\n \"\\n\",\n \"```json\\n\",\n \"[\\n\",\n \" {\\n\",\n \" \\\"id\\\": \\\"identity_0\\\",\\n\",\n \" \\\"conversations\\\": [\\n\",\n \" {\\n\",\n \" \\\"from\\\": \\\"user\\\",\\n\",\n \" \\\"value\\\": \\\"你好，能告诉我遛狗的最佳时间吗？\\\"\\n\",\n \" },\\n\",\n \" {\\n\",\n \" \\\"from\\\": \\\"assistant\\\",\\n\",\n \" \\\"value\\\": \\\"当地最佳遛狗时间因地域差异而异，请问您所在的城市是哪里？\\\"\\n\",\n \" },\\n\",\n \" {\\n\",\n \" \\\"from\\\": \\\"user\\\",\\n\",\n \" \\\"value\\\": \\\"我在纽约市。\\\"\\n\",\n \" },\\n\",\n \" {\\n\",\n \" \\\"from\\\": \\\"assistant\\\",\\n\",\n \" \\\"value\\\": \\\"纽约市的遛狗最佳时间通常在早晨6点至8点和晚上8点至10点之间，因为这些时间段气温较低，遛狗更加舒适。但具体时间还需根据气候、气温和季节变化而定。\\\"\\n\",\n \" }\\n\",\n \" ]\\n\",\n \" }\\n\",\n \"]\\n\",\n \"```\\n\",\n \"\\n\",\n \"## Fine-Tune the Model\\n\",\n \"\\n\",\n \"You can directly run the prepared training script to fine-tune the model. **nproc_per_node** refers to the number of GPUs used fro training.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"7ab0581e-be85-45e6-a5b7-af9c42ea697b\",\n \"metadata\": {\n \"ExecutionIndicator\": {\n \"show\": true\n },\n \"execution\": {\n \"iopub.execute_input\": \"2023-12-31T08:45:37.959631Z\",\n \"iopub.status.busy\": \"2023-12-31T08:45:37.958961Z\",\n \"iopub.status.idle\": \"2023-12-31T08:46:19.501657Z\",\n \"shell.execute_reply\": \"2023-12-31T08:46:19.500854Z\",\n \"shell.execute_reply.started\": \"2023-12-31T08:45:37.959609Z\"\n },\n \"tags\": []\n },\n \"outputs\": [],\n \"source\": [\n \"!torchrun --nproc_per_node 2 --nnodes 1 --node_rank 0 --master_addr localhost --master_port 6601 ../../finetune.py \\\\\\n\",\n \" --model_name_or_path \\\"Qwen/Qwen-1_8B-Chat-Int4/\\\" \\\\\\n\",\n \" --data_path \\\"Belle_sampled_qwen.json\\\" \\\\\\n\",\n \" --bf16 True \\\\\\n\",\n \" --output_dir \\\"output_qwen\\\" \\\\\\n\",\n \" --num_train_epochs 5 \\\\\\n\",\n \" --per_device_train_batch_size 1 \\\\\\n\",\n \" --per_device_eval_batch_size 1 \\\\\\n\",\n \" --gradient_accumulation_steps 16 \\\\\\n\",\n \" --evaluation_strategy \\\"no\\\" \\\\\\n\",\n \" --save_strategy \\\"steps\\\" \\\\\\n\",\n \" --save_steps 1000 \\\\\\n\",\n \" --save_total_limit 10 \\\\\\n\",\n \" --learning_rate 1e-5 \\\\\\n\",\n \" --weight_decay 0.1 \\\\\\n\",\n \" --adam_beta2 0.95 \\\\\\n\",\n \" --warmup_ratio 0.01 \\\\\\n\",\n \" --lr_scheduler_type \\\"cosine\\\" \\\\\\n\",\n \" --logging_steps 1 \\\\\\n\",\n \" --report_to \\\"none\\\" \\\\\\n\",\n \" --model_max_length 512 \\\\\\n\",\n \" --gradient_checkpointing True \\\\\\n\",\n \" --lazy_preprocess True \\\\\\n\",\n \" --deepspeed \\\"../../finetune/ds_config_zero2.json\\\" \\\\\\n\",\n \" --use_lora \\\\\\n\",\n \" --q_lora\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Merge Weights\\n\",\n \"\\n\",\n \"The training of both LoRA and Q-LoRA only saves the adapter parameters. Note that you can not merge weights into quantized models. Instead, we can merge the weights based on the original chat model.\\n\",\n \"\\n\",\n \"You can load the fine-tuned model and merge weights as shown below:\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"from modelscope.hub.snapshot_download import snapshot_download\\n\",\n \"snapshot_download('Qwen/Qwen-1_8B-Chat', cache_dir='.', revision='master')\\n\",\n \"\\n\",\n \"from transformers import AutoModelForCausalLM\\n\",\n \"from peft import PeftModel\\n\",\n \"import torch\\n\",\n \"\\n\",\n \"model = AutoModelForCausalLM.from_pretrained(\\\"Qwen/Qwen-1_8B-Chat/\\\", torch_dtype=torch.float16, device_map=\\\"auto\\\", trust_remote_code=True)\\n\",\n \"model = PeftModel.from_pretrained(model, \\\"output_qwen/\\\")\\n\",\n \"merged_model = model.merge_and_unload()\\n\",\n \"merged_model.save_pretrained(\\\"output_qwen_merged\\\", max_shard_size=\\\"2048MB\\\", safe_serialization=True)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"The tokenizer files are not saved in the new directory in this step. You can copy the tokenizer files or use the following code:\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"from transformers import AutoTokenizer\\n\",\n \"\\n\",\n \"tokenizer = AutoTokenizer.from_pretrained(\\n\",\n \" \\\"Qwen/Qwen-1_8B-Chat-Int4/\\\",\\n\",\n \" trust_remote_code=True\\n\",\n \")\\n\",\n \"\\n\",\n \"tokenizer.save_pretrained(\\\"output_qwen_merged\\\")\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Test the Model\\n\",\n \"\\n\",\n \"After merging the weights, we can test the model as follows:\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"from transformers import AutoModelForCausalLM, AutoTokenizer\\n\",\n \"from transformers.generation import GenerationConfig\\n\",\n \"\\n\",\n \"tokenizer = AutoTokenizer.from_pretrained(\\\"output_qwen_merged\\\", trust_remote_code=True)\\n\",\n \"model = AutoModelForCausalLM.from_pretrained(\\n\",\n \" \\\"output_qwen_merged\\\",\\n\",\n \" device_map=\\\"auto\\\",\\n\",\n \" trust_remote_code=True\\n\",\n \").eval()\\n\",\n \"\\n\",\n \"response, history = model.chat(tokenizer, \\\"你好\\\", history=None)\\n\",\n \"print(response)\"\n ]\n }\n ],\n \"metadata\": {\n \"kernelspec\": {\n \"display_name\": \"Python 3 (ipykernel)\",\n \"language\": \"python\",\n \"name\": \"python3\"\n },\n \"language_info\": {\n \"codemirror_mode\": {\n \"name\": \"ipython\",\n \"version\": 3\n },\n \"file_extension\": \".py\",\n \"mimetype\": \"text/x-python\",\n \"name\": \"python\",\n \"nbconvert_exporter\": \"python\",\n \"pygments_lexer\": \"ipython3\",\n \"version\": \"3.10.13\"\n }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 5\n}\n" }, { "path": "recipes/finetune/deepspeed/finetune_qlora_single_gpu.ipynb", "content": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"id\": \"6e6981ab-2d9a-4280-923f-235a166855ba\",\n \"metadata\": {},\n \"source\": [\n \"# QLoRA Fine-Tuning Qwen-Chat Large Language Model (Single GPU)\\n\",\n \"\\n\",\n \"Tongyi Qianwen is a large language model developed by Alibaba Cloud based on the Transformer architecture, trained on an extensive set of pre-training data. The pre-training data is diverse and covers a wide range, including a large amount of internet text, specialized books, code, etc. In addition, an AI assistant called Qwen-Chat has been created based on the pre-trained model using alignment mechanism.\\n\",\n \"\\n\",\n \"This notebook uses Qwen-1.8B-Chat as an example to introduce how to QLoRA fine-tune the Qianwen model using Deepspeed.\\n\",\n \"\\n\",\n \"## Environment Requirements\\n\",\n \"\\n\",\n \"Please refer to **requirements.txt** to install the required dependencies.\\n\",\n \"\\n\",\n \"## Preparation\\n\",\n \"\\n\",\n \"### Download Qwen-1.8B-Chat\\n\",\n \"\\n\",\n \"First, download the model files. You can choose to download directly from ModelScope.\\n\",\n \"\\n\",\n \"Note that we use the Int4 version of the models for QLoRA training.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"248488f9-4a86-4f35-9d56-50f8e91a8f11\",\n \"metadata\": {\n \"ExecutionIndicator\": {\n \"show\": true\n },\n \"tags\": []\n },\n \"outputs\": [],\n \"source\": [\n \"from modelscope.hub.snapshot_download import snapshot_download\\n\",\n \"model_dir = snapshot_download('Qwen/Qwen-1_8B-Chat-Int4', cache_dir='.', revision='master')\"\n ]\n },\n {\n \"attachments\": {},\n \"cell_type\": \"markdown\",\n \"id\": \"7b2a92b1-f08e-4413-9f92-8f23761e6e1f\",\n \"metadata\": {},\n \"source\": [\n \"### Download Example Training Data\\n\",\n \"\\n\",\n \"Download the data required for training; here, we provide a tiny dataset as an example. It is sampled from [Belle](https://github.com/LianjiaTech/BELLE).\\n\",\n \"\\n\",\n \"Disclaimer: the dataset can be only used for the research purpose.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"ce195f08-fbb2-470e-b6c0-9a03457458c7\",\n \"metadata\": {\n \"tags\": []\n },\n \"outputs\": [],\n \"source\": [\n \"!wget https://atp-modelzoo-sh.oss-cn-shanghai.aliyuncs.com/release/tutorials/qwen_recipes/Belle_sampled_qwen.json\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"7226bed0-171b-4d45-a3f9-b3d81ec2bb9f\",\n \"metadata\": {},\n \"source\": [\n \"You can also refer to this format to prepare the dataset. Below is a simple example list with 1 sample:\\n\",\n \"\\n\",\n \"```json\\n\",\n \"[\\n\",\n \" {\\n\",\n \" \\\"id\\\": \\\"identity_0\\\",\\n\",\n \" \\\"conversations\\\": [\\n\",\n \" {\\n\",\n \" \\\"from\\\": \\\"user\\\",\\n\",\n \" \\\"value\\\": \\\"你好\\\"\\n\",\n \" },\\n\",\n \" {\\n\",\n \" \\\"from\\\": \\\"assistant\\\",\\n\",\n \" \\\"value\\\": \\\"我是一个语言模型，我叫通义千问。\\\"\\n\",\n \" }\\n\",\n \" ]\\n\",\n \" }\\n\",\n \"]\\n\",\n \"```\\n\",\n \"\\n\",\n \"You can also use multi-turn conversations as the training set. Here is a simple example:\\n\",\n \"\\n\",\n \"```json\\n\",\n \"[\\n\",\n \" {\\n\",\n \" \\\"id\\\": \\\"identity_0\\\",\\n\",\n \" \\\"conversations\\\": [\\n\",\n \" {\\n\",\n \" \\\"from\\\": \\\"user\\\",\\n\",\n \" \\\"value\\\": \\\"你好，能告诉我遛狗的最佳时间吗？\\\"\\n\",\n \" },\\n\",\n \" {\\n\",\n \" \\\"from\\\": \\\"assistant\\\",\\n\",\n \" \\\"value\\\": \\\"当地最佳遛狗时间因地域差异而异，请问您所在的城市是哪里？\\\"\\n\",\n \" },\\n\",\n \" {\\n\",\n \" \\\"from\\\": \\\"user\\\",\\n\",\n \" \\\"value\\\": \\\"我在纽约市。\\\"\\n\",\n \" },\\n\",\n \" {\\n\",\n \" \\\"from\\\": \\\"assistant\\\",\\n\",\n \" \\\"value\\\": \\\"纽约市的遛狗最佳时间通常在早晨6点至8点和晚上8点至10点之间，因为这些时间段气温较低，遛狗更加舒适。但具体时间还需根据气候、气温和季节变化而定。\\\"\\n\",\n \" }\\n\",\n \" ]\\n\",\n \" }\\n\",\n \"]\\n\",\n \"```\\n\",\n \"\\n\",\n \"## Fine-Tune the Model\\n\",\n \"\\n\",\n \"You can directly run the prepared training script to fine-tune the model.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"7ab0581e-be85-45e6-a5b7-af9c42ea697b\",\n \"metadata\": {\n \"ExecutionIndicator\": {\n \"show\": true\n },\n \"tags\": []\n },\n \"outputs\": [],\n \"source\": [\n \"!python ../../finetune.py \\\\\\n\",\n \" --model_name_or_path \\\"Qwen/Qwen-1_8B-Chat-Int4/\\\"\\\\\\n\",\n \" --data_path \\\"Belle_sampled_qwen.json\\\"\\\\\\n\",\n \" --bf16 \\\\\\n\",\n \" --output_dir \\\"output_qwen\\\" \\\\\\n\",\n \" --num_train_epochs 5 \\\\\\n\",\n \" --per_device_train_batch_size 1 \\\\\\n\",\n \" --per_device_eval_batch_size 1 \\\\\\n\",\n \" --gradient_accumulation_steps 16 \\\\\\n\",\n \" --evaluation_strategy \\\"no\\\" \\\\\\n\",\n \" --save_strategy \\\"steps\\\" \\\\\\n\",\n \" --save_steps 1000 \\\\\\n\",\n \" --save_total_limit 10 \\\\\\n\",\n \" --learning_rate 1e-5 \\\\\\n\",\n \" --weight_decay 0.1 \\\\\\n\",\n \" --adam_beta2 0.95 \\\\\\n\",\n \" --warmup_ratio 0.01 \\\\\\n\",\n \" --lr_scheduler_type \\\"cosine\\\" \\\\\\n\",\n \" --logging_steps 1 \\\\\\n\",\n \" --report_to \\\"none\\\" \\\\\\n\",\n \" --model_max_length 512 \\\\\\n\",\n \" --gradient_checkpointing \\\\\\n\",\n \" --lazy_preprocess \\\\\\n\",\n \" --use_lora \\\\\\n\",\n \" --q_lora \\\\\\n\",\n \" --deepspeed \\\"../../finetune/ds_config_zero2.json\\\"\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"0a50941d-3c3c-4ed2-9185-d4fe6172da2f\",\n \"metadata\": {},\n \"source\": [\n \"## Merge Weights\\n\",\n \"\\n\",\n \"The training of both LoRA and Q-LoRA only saves the adapter parameters. Note that you can not merge weights into quantized models. Instead, we can merge the weights based on the original chat model.\\n\",\n \"\\n\",\n \"You can load the fine-tuned model and merge weights as shown below:\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"909ff537-f851-488e-b1e8-1046f6852202\",\n \"metadata\": {\n \"ExecutionIndicator\": {\n \"show\": true\n },\n \"tags\": []\n },\n \"outputs\": [],\n \"source\": [\n \"from modelscope.hub.snapshot_download import snapshot_download\\n\",\n \"snapshot_download('Qwen/Qwen-1_8B-Chat', cache_dir='.', revision='master')\\n\",\n \"\\n\",\n \"from transformers import AutoModelForCausalLM\\n\",\n \"from peft import PeftModel\\n\",\n \"import torch\\n\",\n \"\\n\",\n \"model = AutoModelForCausalLM.from_pretrained(\\\"Qwen/Qwen-1_8B-Chat/\\\", torch_dtype=torch.float16, device_map=\\\"auto\\\", trust_remote_code=True)\\n\",\n \"model = PeftModel.from_pretrained(model, \\\"output_qwen/\\\")\\n\",\n \"merged_model = model.merge_and_unload()\\n\",\n \"merged_model.save_pretrained(\\\"output_qwen_merged\\\", max_shard_size=\\\"2048MB\\\", safe_serialization=True)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"7969df6e-ba8a-45f5-8b44-e1cbe74a8ef6\",\n \"metadata\": {},\n \"source\": [\n \"The tokenizer files are not saved in the new directory in this step. You can copy the tokenizer files or use the following code:\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"c01b6a3f-036f-4b7c-b5a6-76a7b6894d4e\",\n \"metadata\": {\n \"tags\": []\n },\n \"outputs\": [],\n \"source\": [\n \"from transformers import AutoTokenizer\\n\",\n \"\\n\",\n \"tokenizer = AutoTokenizer.from_pretrained(\\n\",\n \" \\\"Qwen/Qwen-1_8B-Chat-Int4/\\\",\\n\",\n \" trust_remote_code=True\\n\",\n \")\\n\",\n \"\\n\",\n \"tokenizer.save_pretrained(\\\"output_qwen_merged\\\")\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"c2944b9b-89c7-4fb5-bd08-941d4706e943\",\n \"metadata\": {},\n \"source\": [\n \"## Test the Model\\n\",\n \"\\n\",\n \"After merging the weights, we can test the model as follows:\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"b77abbb1-5b29-4eb1-8a6c-e2e146b8d33d\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"from transformers import AutoModelForCausalLM, AutoTokenizer\\n\",\n \"from transformers.generation import GenerationConfig\\n\",\n \"\\n\",\n \"tokenizer = AutoTokenizer.from_pretrained(\\\"output_qwen_merged\\\", trust_remote_code=True)\\n\",\n \"model = AutoModelForCausalLM.from_pretrained(\\n\",\n \" \\\"output_qwen_merged\\\",\\n\",\n \" device_map=\\\"auto\\\",\\n\",\n \" trust_remote_code=True\\n\",\n \").eval()\\n\",\n \"\\n\",\n \"response, history = model.chat(tokenizer, \\\"你好\\\", history=None)\\n\",\n \"print(response)\"\n ]\n }\n ],\n \"metadata\": {\n \"kernelspec\": {\n \"display_name\": \"Python 3 (ipykernel)\",\n \"language\": \"python\",\n \"name\": \"python3\"\n },\n \"language_info\": {\n \"codemirror_mode\": {\n \"name\": \"ipython\",\n \"version\": 3\n },\n \"file_extension\": \".py\",\n \"mimetype\": \"text/x-python\",\n \"name\": \"python\",\n \"nbconvert_exporter\": \"python\",\n \"pygments_lexer\": \"ipython3\",\n \"version\": \"3.10.13\"\n }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 5\n}\n" }, { "path": "recipes/finetune/deepspeed/readme.md", "content": "# Fine-tuning Qwen Using Deepspeed\n\n\n## TL;DR\n\nWe provide the official training script `finetune.py` and serveral notebooks that can be leveraged for users to finetune pre-trained models for downstream applications in a simple fashion. The algorithms that we support include full-parameter fine-tuning, LoRA fine-tuning and Q-LoRA fine-tuning. Here is the matrix of our notebooks used in different settings:\n\n| Algorithm | Single GPU | Multiple GPUs|\n| --- | --- | --- |\n| Full-parameter Fine-tuning | [finetune_fullparameter_single_gpu](finetune_fullparameter_single_gpu.ipynb) | [finetune_fullparameter_multi_gpu](finetune_fullparameter_multi_gpu.ipynb) |\n| LoRA Fine-tuning | [finetune_lora_single_gpu](finetune_lora_single_gpu.ipynb) | [finetune_lora_multi_gpu](finetune_lora_multi_gpu.ipynb) |\n| Q-LoRA Fine-tuning | [finetune_qlora_single_gpu](finetune_qlora_single_gpu.ipynb) | [finetune_qlora_multi_gpu](finetune_qlora_multi_gpu.ipynb) |\n\n## Requirements\n\n### Environments\n\nThe basic requirements for running Qwen models include:\n\n- python 3.8 and above\n- pytorch 1.12 and above, 2.0 and above are recommended\n- transformers 4.32 and above\n- CUDA 11.4 and above are recommended (this is for GPU users, flash-attention users, etc.)\n\nOur notebooks launch fine-tuning with DeepSpeed and Peft.\n(Note: this may have conflicts with the latest version of pydantic and you should use make sure `pydantic<2.0`.)\nYou can install them by:\n```bash\npip install peft deepspeed\n```\n\n### Settings and GPU Requirements\n\nWe first provide the support matrix for different learning settings. Full-parameter fine-tuning requires updating all parameters in the whole training process.\nIn comparison with full-parameter fine-tuning, LoRA only updates the parameters of adapter layers but keeps the original large language model layers frozen. This allows much fewer memory costs and thus fewer computation costs. If you still suffer from insufficient memory, you can consider Q-LoRA, which uses the quantized large language model to allow even fewer memory costs. Generally, the GPU consumption rule for tuning Qwen is as follows: full parameter > full parameter (ZeRO2) > full parameter (ZeRO3) > LoRA > LoRA (ZeRO2) > LoRA (ZeRO3) > Q-LoRA > Q-LoRA (ZeRO2).\n\n| Setting | Full-parameter | LoRA | Q-LoRA |\n| --- | --- | --- | --- |\n| Base | Yes (up to ZeRO3) | Yes (up to ZeRO2) | No |\n| Chat | Yes (up to ZeRO3) | Yes (up to ZeRO3) | No |\n| Chat-Int4/8 | No | No | Yes |\n\nHere are some useful suggestions for choosing different fine-tuning settings based on GPU memory, espcially for users with GeForce RTX 3090/4090 (24GB) GPUs (or similar), and A100 (80GB) GPUs (or similar). In the experiments, we uniformly use a batch size of 1, gradient accumulation of 16, and max length of 512. Other parameters are set as the same shown in our notebooks. The results are as follows.\n\n| GPU Memory | Number of GPUs | Qwen-1.8B-Chat | Qwen-7B-Chat | Qwen-14B-Chat | Qwen-72B-Chat |\n| --- | --- | --- | --- | --- | --- |\n| 24GB | *1 | Full Parameter | LoRA | Q-LoRA | N/A |\n| 24GB | *2 | Full Parameter | LoRA | Q-LoRA | N/A |\n| 24GB | *4 | Full Parameter | LoRA | LoRA (w/ ZeRO3) | N/A |\n| 80GB | *1 | Full Parameter | LoRA | LoRA | Q-LoRA |\n| 80GB | *2 | Full Parameter | Full Parameter (w/ ZeRO3) | LoRA (w/ ZeRO2) | TBD |\n| 80GB | *4 | Full Parameter | Full Parameter (w/ ZeRO2) | Full Parameter (w/ ZeRO3) | LoRA (w/ ZeRO3) |\n\nUsing other configurations of LoRA/Q-LoRA and ZeRO stages will easily result in failures.\n\n\n## Data Preparation\n\nTo prepare your training data, you need to put all the samples into a list and save it to a json file. Each sample is a dictionary consisting of an id and a list for conversation. Below is a simple example list with 1 sample:\n```json\n[\n {\n \"id\": \"identity_0\",\n \"conversations\": [\n {\n \"from\": \"user\",\n \"value\": \"你好\"\n },\n {\n \"from\": \"assistant\",\n \"value\": \"我是一个语言模型，我叫通义千问。\"\n }\n ]\n }\n]\n```\n\nYou can also use multi-turn conversations as the training set. Here is a simple example:\n\n```json\n[\n {\n \"id\": \"identity_0\",\n \"conversations\": [\n {\n \"from\": \"user\",\n \"value\": \"你好\"\n },\n {\n \"from\": \"assistant\",\n \"value\": \"你好！我是一名AI助手，我叫通义千问，有需要请告诉我。\"\n },\n {\n \"from\": \"user\",\n \"value\": \"你都能做什么\"\n },\n {\n \"from\": \"assistant\",\n \"value\": \"我能做很多事情，包括但不限于回答各种领域的问题、提供实用建议和指导、进行多轮对话交流、文本生成等。\"\n }\n ]\n }\n]\n```\n\n\n## Single-GPU Training\n\nIn the single-GPU training setting, we provide three notebooks:\n\n- [finetune_fullparameter_single_gpu](finetune_fullparameter_single_gpu.ipynb)\n- [finetune_lora_single_gpu](finetune_lora_single_gpu.ipynb)\n- [finetune_qlora_single_gpu](finetune_qlora_single_gpu.ipynb)\n\n### Full-parameter Fine-tuning\n\nTo launch your training, run the following command (with hyper-parameter settings omitted):\n```bash\npython finetune.py \\\n --model_name_or_path $MODEL \\\n --data_path $DATA \\\n --output_dir $OUTPUT\n```\nRemember to specify the correct model name or path, the data path, as well as the output directory.\n\n### LoRA Fine-tuning\n\nSimilarly, to run LoRA, use another notebook to run the command as shown below. Before you start, make sure that you have installed `peft`. Also, you need to specify your paths to your model, data, and output. We advise you to use absolute path for your pre-trained model. This is because LoRA only saves the adapter and the absolute path in the adapter configuration json file is used for finding out the pre-trained model to load. \n```bash\npython finetune.py \\\n --model_name_or_path $MODEL \\\n --data_path $DATA \\\n --output_dir $OUTPUT \\\n --use_lora\n```\nNote that if you use LoRA to fine-tune the base language model, e.g., Qwen-7B, instead of chat models, e.g., Qwen-7B-Chat, the script automatically switches the embedding and output layer as trainable parameters. This is because the base language model has no knowledge of special tokens brought by ChatML format. Thus these layers should be updated for the model to understand and predict the tokens. Or in another word, if your training brings in special tokens in LoRA, you should set the layers to trainable parameters by setting `modules_to_save` inside the code. Check out the following code in the training script `finetune.py`:\n```python\nis_chat_model = 'chat' in model_args.model_name_or_path.lower()\nif training_args.use_lora:\n if lora_args.q_lora or is_chat_model:\n modules_to_save = None\n else:\n modules_to_save = [\"wte\", \"lm_head\"]\n lora_config = LoraConfig(\n r=lora_args.lora_r,\n lora_alpha=lora_args.lora_alpha,\n target_modules=lora_args.lora_target_modules,\n lora_dropout=lora_args.lora_dropout,\n bias=lora_args.lora_bias,\n task_type=\"CAUSAL_LM\",\n modules_to_save=modules_to_save # This argument serves for adding new tokens.\n )\n ...\n model = get_peft_model(model, lora_config)\n ...\n```\nPay attention that the script relies on the model path to identify the model type, so please keep `chat` in the chat model paths.\n\n\n\n### Q-LoRA Fine-tuning\n\nTo run single-GPU Q-LoRA training, you may need to install `mpi4py`. Directly run the following script:\n```bash\npython finetune.py \\\n --model_name_or_path $MODEL \\\n --data_path $DATA \\\n --output_dir $OUTPUT \\\n --use_lora \\\n --q_lora \\\n --deepspeed \"ds_config_zero2.json\"\n```\n\nFor Q-LoRA, we advise you to load our provided quantized model, e.g., Qwen-7B-Chat-Int4. You **SHOULD NOT** use the bf16 models. Different from full-parameter fine-tuning and LoRA, only fp16 is supported for Q-LoRA. For single-GPU training, we have to use DeepSpeed for mixed-precision training due to our observation of errors caused by torch amp. Besides, for Q-LoRA, the troubles with the special tokens in LoRA still exist. However, as we only provide the Int4 models for chat models, which means the language model has learned the special tokens of ChatML format, you have no worry about the layers. Note that the layers of the Int4 model should not be trainable, and thus if you introduce special tokens in your training, Q-LoRA might not work.\n\n\nIn default, our notebooks provide training codes for Qwen-1.8B-Chat.\nYou can also run the training script to fine-tune other version of the Qwen-series models. We profile the GPU memory usage of all versions based on our notebooks (without changing any hyper-parameter settings) on a single A800 GPU (80GB). The statistics are listed below:\n\n| Training | Qwen-1.8B-Chat | Qwen-7B-Chat | Qwen-14B-Chat | Qwen-72B-Chat |\n| --- | --- | --- | --- | --- |\n| Full Parameter | 19.6GB | 76.8GB | OOM | OOM |\n| LoRA | 7.4GB | 20.3GB | 34.2GB | OOM |\n| Q-LoRA | 6.1GB | 12.5GB | 17.8GB | 61.9GB |\n\n\n### Merging Weights from LoRA and Q-LoRA\n\n\n#### Inference with Adapters\n\nDifferent from full-parameter fine-tuning, the training of both LoRA and Q-LoRA only saves the adapter parameters. Suppose your training starts from Qwen-7B, you can load the fine-tuned model for inference as shown below:\n```python\nfrom peft import AutoPeftModelForCausalLM\nfrom transformers import AutoTokenizer\n\ntokenizer = AutoTokenizer.from_pretrained(\n path_to_adapter, # path to the output directory\n trust_remote_code=True\n)\nmodel = AutoPeftModelForCausalLM.from_pretrained(\n path_to_adapter, # path to the output directory\n device_map=\"auto\",\n trust_remote_code=True\n).eval()\n\nresponse, history = model.chat(tokenizer, \"你好\", history=None)\n```\n\n#### Inference with Merged Weights\n\nIf you want to merge the adapters and save the fine-tuned model as a standalone model, take LoRA as an example, you can run the following codes:\n```python\nfrom peft import AutoPeftModelForCausalLM\n\nmodel = AutoPeftModelForCausalLM.from_pretrained(\n path_to_adapter, # path to the output directory\n device_map=\"auto\",\n trust_remote_code=True\n).eval()\n\nmerged_model = model.merge_and_unload()\n# max_shard_size and safe serialization are not necessary. \n# They respectively work for sharding checkpoint and save the model to safetensors.\nmerged_model.save_pretrained(new_model_directory, max_shard_size=\"2048MB\", safe_serialization=True)\n```\n\nThe `new_model_directory` directory will contain the merged model weights and module files. Please note that `*.cu` and `*.cpp` files may be missing in the saved files. If you wish to use the KV cache functionality, please manually copy them. Besides, the tokenizer files are not saved in the new directory in this step. You can copy the tokenizer files or use the following code:\n```python\nfrom transformers import AutoTokenizer\n\ntokenizer = AutoTokenizer.from_pretrained(\n path_to_adapter, # path to the output directory\n trust_remote_code=True\n)\ntokenizer.save_pretrained(new_model_directory)\n```\nNext, the model with merged weights can be loaded by the following code:\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\ntokenizer = AutoTokenizer.from_pretrained(new_model_directory, trust_remote_code=True)\nmodel = AutoModelForCausalLM.from_pretrained(\n new_model_directory,\n device_map=\"auto\",\n trust_remote_code=True\n).eval()\nresponse, history = model.chat(tokenizer, \"你好\", history=None)\n```\n\nNote that you can not merge weights into quantized models. Instead, we can merge the weights based on the original chat model. Take Qwen-7B-Chat-In4 as an example. \n```python\nfrom transformers import AutoModelForCausalLM\nfrom peft import PeftModel\nimport torch\n\n# Here, we load the original Qwen-7B-Chat model, instead of the Qwen-7B-Chat-Int4 model.\nmodel = AutoModelForCausalLM.from_pretrained(\"Qwen/Qwen-7B-Chat\", torch_dtype=torch.float16, device_map=\"auto\", trust_remote_code=True)\n# We merge the learned adapter to the Qwen-7B-Chat.\nmodel = PeftModel.from_pretrained(model, path_to_adapter)\nmerged_model = model.merge_and_unload()\n# We save the model to a new path.\nmerged_model.save_pretrained(path_to_new_model, max_shard_size=\"2048MB\", safe_serialization=True)\n```\n\n\n## Multi-GPU Training\n\nIn the multi-GPU training setting, we provide three notebooks:\n\n- [finetune_fullparameter_multi_gpu](finetune_fullparameter_multi_gpu.ipynb)\n- [finetune_lora_multi_gpu](finetune_lora_multi_gpu.ipynb)\n- [finetune_qlora_multi_gpu](finetune_qlora_multi_gpu.ipynb)\n\nWe use `torchrun` to launch the training job on multiple GPUs:\n\n```bash\n# for full-parameter fine-tuning\ntorchrun --nproc_per_node 2 --nnodes 1 --node_rank 0 --master_addr localhost --master_port 6601 finetune.py \\\n --model_name_or_path $MODEL \\\n --data_path $DATA \\\n --output_dir $OUTPUT \\\n --deepspeed \"ds_config_zero2.json\"\n\n# for LoRA fine-tuning\ntorchrun --nproc_per_node 2 --nnodes 1 --node_rank 0 --master_addr localhost --master_port 6601 finetune.py \\\n --model_name_or_path $MODEL \\\n --data_path $DATA \\\n --output_dir $OUTPUT \\\n --deepspeed \"ds_config_zero2.json\" \\\n --use_lora\n\n# for Q-LoRA fine-tuning\ntorchrun --nproc_per_node 2 --nnodes 1 --node_rank 0 --master_addr localhost --master_port 6601 finetune.py \\\n --model_name_or_path $MODEL \\\n --data_path $DATA \\\n --output_dir $OUTPUT \\\n --deepspeed \"ds_config_zero2.json\" \\\n --use_lora \\\n --q_lora\n```\n\nFor multi-GPU training, you also need to specify proper hyperparameters for distributed training based on your machine. Besides, we advise you to specify your maximum sequence length with the argument `--model_max_length`, based on your consideration of data, memory footprint, and training speed.\nFor the usage of `torchrun` and distrubuted arguments, please refer to [here](https://pytorch.org/docs/stable/elastic/run.html).\nAdditionally, we find that there is a significant gap between the memory footprint of LoRA with and without these trainable parameters. Therefore, if you have trouble with memory, we advise you to LoRA fine-tune the chat models. Check the profile below for more information. \n\n\n### Multi-node Fine-tuning\n\nOur provided scripts also support multi-node fine-tuning. You can refer to the comments in the scripts to correctly set corresponding arguments and launch the script on each node. For more information about multi-node distributed training, please refer to [torchrun](https://pytorch.org/docs/stable/elastic/run.html).\n\nNote: DeepSpeed ZeRO 3 requires much greater inter-node communication rate than ZeRO 2, which will significantly reduce the training speed in the case of multinode finetuning. Therefore, we do not recommend using DeepSpeed ZeRO 3 configurations in multi-node fine-tuning scripts.\n\n### Profiling of Memory and Speed\n\nWe profile the GPU memory and training speed of both LoRA (LoRA (emb) refers to training the embedding and output layer, while LoRA has no trainable embedding and output layer) and Q-LoRA in the setup of single-GPU training. In this test, we experiment on a single A100-SXM4-80G GPU, and we use CUDA 11.8 and Pytorch 2.0. Flash attention 2 is applied. We uniformly use a batch size of 1 and gradient accumulation of 8. We profile the memory (GB) and speed (s/iter) of inputs of different lengths, namely 256, 512, 1024, 2048, 4096, and 8192. We also report the statistics of full-parameter fine-tuning with Qwen-7B on 2 A100 GPUs. We only report the statistics of 256, 512, and 1024 tokens due to the limitation of GPU memory. \n\nFor Qwen-7B, we also test the performance of multi-node fine-tuning. We experiment using two servers, each containing two A100-SXM4-80G GPUs, and the rest of configurations are the same as other Qwen-7B experiments. The results of multi-node fine-tuning are marked as LoRA (multinode) in the table.\n\nFor Qwen-72B, we experiment in two ways: 1) LoRA fine-tuning + DeepSpeed ZeRO 3 on 4 A100-SXM4-80G GPUs and 2) Q-LoRA (int4) fine-tuning on a single A100-SXM4-80G GPU. Note that OOM occurs on 4 A100-SXM4-80G GPUs both with LoRA (emb) fine-tuning and LoRA fine-tuning without Deepspeed ZeRO 3 (you can pass `--deepspeed ds_config_zero3.json` to `finetune_lora_ds.sh` to enable DeepSpeed ZeRO 3).\n\nThe statistics are listed below:\n\n<table>\n <tr>\n <th rowspan=\"2\">Model Size</th><th rowspan=\"2\">Method</th><th rowspan=\"2\">#Nodes</th><th rowspan=\"2\">#GPUs per node</th><th colspan=\"6\" align=\"center\">Sequence Length</th>\n </tr>\n <tr>\n <th align=\"center\">256</th><th align=\"center\">512</th><th align=\"center\">1024</th><th align=\"center\">2048</th><th align=\"center\">4096</th><th align=\"center\">8192</th>\n </tr>\n <tr>\n <th rowspan=\"4\">1.8B</th><td>LoRA</td>\n <td>1</td><td>1</td>\n <td align=\"center\">6.7G / 1.0s/it</td><td align=\"center\">7.4G / 1.0s/it</td><td align=\"center\">8.4G / 1.1s/it</td><td align=\"center\">11.0G / 1.7s/it</td><td align=\"center\">16.2G / 3.3s/it</td><td align=\"center\">21.8G / 6.8s/it</td>\n </tr>\n <tr>\n <td>LoRA (emb)</td>\n <td>1</td><td>1</td>\n <td align=\"center\">13.7G / 1.0s/it</td><td align=\"center\">14.0G / 1.0s/it</td><td align=\"center\">14.0G / 1.1s/it</td><td align=\"center\">15.1G / 1.8s/it</td><td align=\"center\">19.7G / 3.4s/it</td><td align=\"center\">27.7G / 7.0s/it</td>\n </tr>\n <tr>\n <td>Q-LoRA</td>\n <td>1</td><td>1</td>\n <td align=\"center\">5.8G / 1.4s/it</td><td align=\"center\">6.0G / 1.4s/it</td><td align=\"center\">6.6G / 1.4s/it</td><td align=\"center\">7.8G / 2.0s/it</td><td align=\"center\">10.2G / 3.4s/it</td><td align=\"center\">15.8G / 6.5s/it</td>\n </tr>\n <tr>\n <td>Full-parameter</td>\n <td>1</td><td>1</td>\n <td align=\"center\">43.5G / 2.1s/it</td><td align=\"center\">43.5G / 2.2s/it</td><td align=\"center\">43.5G / 2.2s/it</td><td align=\"center\">43.5G / 2.3s/it</td><td align=\"center\">47.1G / 2.8s/it</td><td align=\"center\">48.3G / 5.6s/it</td>\n </tr>\n <tr>\n <th rowspan=\"5\">7B</th>\n <td>LoRA</td>\n <td>1</td><td>1</td>\n <td align=\"center\">20.1G / 1.2s/it</td><td align=\"center\">20.4G / 1.5s/it</td><td align=\"center\">21.5G / 2.8s/it</td><td align=\"center\">23.8G / 5.2s/it</td><td align=\"center\">29.7G / 10.1s/it</td><td align=\"center\">36.6G / 21.3s/it</td>\n </tr>\n <tr>\n <td>LoRA (emb)</td>\n <td>1</td><td>1</td>\n <td align=\"center\">33.7G / 1.4s/it</td><td align=\"center\">34.1G / 1.6s/it</td><td align=\"center\">35.2G / 2.9s/it</td><td align=\"center\">35.1G / 5.3s/it</td><td align=\"center\">39.2G / 10.3s/it</td><td align=\"center\">48.5G / 21.7s/it</td>\n </tr>\n <tr>\n <td>Q-LoRA</td>\n <td>1</td><td>1</td>\n <td align=\"center\">11.5G / 3.0s/it</td><td align=\"center\">11.5G / 3.0s/it</td><td align=\"center\">12.3G / 3.5s/it</td><td align=\"center\">13.9G / 7.0s/it</td><td align=\"center\">16.9G / 11.6s/it</td><td align=\"center\">23.5G / 22.3s/it</td>\n </tr>\n <tr>\n <td>Full-parameter</td>\n<td>1</td><td>2</td>\n<td align=\"center\">139.2G / 4.0s/it</td><td align=\"center\">148.0G / 4.0s/it</td><td align=\"center\">162.0G / 4.5s/it</td><td align=\"center\">-</td><td align=\"center\">-</td><td align=\"center\">-</td>\n </tr>\n <tr>\n <td>LoRA (multinode)</td>\n <td>2</td><td>2</td>\n <td align=\"center\">74.7G / 2.09s/it</td><td align=\"center\">77.6G / 3.16s/it</td><td align=\"center\">84.9G / 5.17s/it</td><td align=\"center\">95.1G / 9.25s/it</td><td align=\"center\">121.1G / 18.1s/it</td><td align=\"center\">155.5G / 37.4s/it</td>\n </tr>\n <tr>\n <th rowspan=\"3\">14B</th>\n <td>LoRA</td>\n <td>1</td><td>1</td>\n <td align=\"center\">34.6G / 1.6s/it</td><td align=\"center\">35.1G / 2.4s/it</td><td align=\"center\">35.3G / 4.4s/it</td><td align=\"center\">37.4G / 8.4s/it</td><td align=\"center\">42.5G / 17.0s/it</td><td align=\"center\">55.2G / 36.0s/it</td>\n </tr>\n <tr>\n <td>LoRA (emb)</td>\n <td>1</td><td>1</td>\n <td align=\"center\">51.2 / 1.7s/it</td><td align=\"center\">51.1G / 2.6s/it</td><td align=\"center\">51.5G / 4.6s/it</td><td align=\"center\">54.1G / 8.6s/it</td><td align=\"center\">56.8G / 17.2s/it</td><td align=\"center\">67.7G / 36.3s/it</td>\n </tr>\n <tr>\n <td>Q-LoRA</td>\n <td>1</td><td>1</td>\n <td align=\"center\">18.7G / 5.3s/it</td><td align=\"center\">18.4G / 6.3s/it</td><td align=\"center\">18.9G / 8.2s/it</td><td align=\"center\">19.9G / 11.8s/it</td><td align=\"center\">23.0G / 20.1s/it</td><td align=\"center\">27.9G / 38.3s/it</td>\n </tr>\n <tr>\n <th rowspan=\"2\">72B</th>\n <td>LoRA + Deepspeed Zero3</td>\n <td>1</td><td>4</td>\n <td align=\"center\">215.4G / 17.6s/it</td><td align=\"center\">217.7G / 20.5s/it</td><td align=\"center\">222.6G / 29.4s/it</td><td align=\"center\">228.8G / 45.7s/it</td><td align=\"center\">249.0G / 83.4s/it</td><td align=\"center\">289.2G / 161.5s/it</td>\n </tr>\n <tr>\n <td>Q-LoRA</td>\n <td>1</td><td>1</td>\n <td align=\"center\">61.4G / 27.4s/it</td><td align=\"center\">61.4G / 31.5s/it</td><td align=\"center\">62.9G / 41.4s/it</td><td align=\"center\">64.1G / 59.5s/it</td><td align=\"center\">68.0G / 97.7s/it</td><td align=\"center\">75.6G / 179.8s/it</td>\n </tr>\n</table>\n<br>\n\n\n\n\n\n\n\n\n\n\n\n" }, { "path": "recipes/finetune/deepspeed/requirements.txt", "content": "deepspeed\npeft" }, { "path": "recipes/finetune/swift/README.md", "content": "## Introduction\n[SWIFT](https://github.com/modelscope/swift) (Scalable lightWeight Infrastructure for Fine-Tuning) is an extensible framwork designed to faciliate lightweight model fine-tuning and inference. It integrates implementations for various efficient fine-tuning methods, by embracing approaches that is parameter-efficient, memory-efficient, and time-efficient. SWIFT integrates seamlessly into ModelScope ecosystem and offers the capabilities to finetune various models, with a primary emphasis on LLMs and vision models. Additionally, SWIFT is fully compatible with PEFT, enabling users to leverage the familiar Peft interface to finetune ModelScope models.\n\n## Installation\n\n```shell\n# Set the global pip mirror\npip config set global.index-url https://mirrors.aliyun.com/pypi/simple/\n# Install ms-swift\ngit clone https://github.com/modelscope/swift.git\ncd swift\npip install -e .[llm]\n\n# If you want to use deepspeed\npip install deepspeed -U\n\n# If you want to use qlora training based on auto_gptq (recommended, performs better than bnb)\n# Models supporting auto_gptq: `https://github.com/modelscope/swift/blob/main/docs/source/LLM/支持的模型和数据集.md#模型`\n# There's a version correspondence between auto_gptq and cuda; refer to `https://github.com/PanQiWei/AutoGPTQ#quick-installation` for selecting the appropriate version\npip install auto_gptq -U\n\n# If you want to use qlora training based on bnb\npip install bitsandbytes -U\n\n# Environment alignment (run the following commands if you encounter errors; the repository is tested with the latest environment)\npip install -r requirements/framework.txt -U\npip install -r requirements/llm.txt -U\n```\n\n## WebUI Usage\n\nRun the following command to start the webui and conduct model training and inference through the graphical interface:\n```shell\nswift web-ui\n```\nA screenshot example can be found at:\n![image](https://modelscope.oss-cn-beijing.aliyuncs.com/resource/swift_webui.jpg)\n\n## Fine-tuning\n\n```python\n# Experimental environment: A10, 3090, V100, ...\n# GPU memory requirement: 20GB\nCUDA_VISIBLE_DEVICES=0 \\\nswift sft \\\n --model_id_or_path qwen/Qwen-7B-Chat \\\n --dataset blossom-math-zh \\\n --output_dir output \\\n\n# Use your own dataset\nCUDA_VISIBLE_DEVICES=0 \\\nswift sft \\\n --model_id_or_path qwen/Qwen-7B-Chat \\\n --custom_train_dataset_path chatml.jsonl \\\n --output_dir output \\\n\n# Using DDP (Distributed Data Parallel)\n# Experimental environment: 2 * 3090\n# GPU memory requirement: 2 * 23GB\nCUDA_VISIBLE_DEVICES=0,1 \\\nNPROC_PER_NODE=2 \\\nswift sft \\\n --model_id_or_path qwen/Qwen-7B-Chat \\\n --dataset blossom-math-zh \\\n --output_dir output \\\n\n# Multi-machine multi-GPU setup\n# node0\nCUDA_VISIBLE_DEVICES=0,1,2,3 \\\nNNODES=2 \\\nNODE_RANK=0 \\\nMASTER_ADDR=127.0.0.1 \\\nNPROC_PER_NODE=4 \\\nswift sft \\\n --model_id_or_path qwen/Qwen-7B-Chat \\\n --dataset blossom-math-zh \\\n --output_dir output \\\n# node1\nCUDA_VISIBLE_DEVICES=0,1,2,3 \\\nNNODES=2 \\\nNODE_RANK=1 \\\nMASTER_ADDR=xxx.xxx.xxx.xxx \\\nNPROC_PER_NODE=4 \\\nswift sft \\\n --model_id_or_path qwen/Qwen-7B-Chat \\\n --dataset blossom-math-zh \\\n --output_dir output \\\n```\nFor more fine-tuning methods, please refer to [here](https://github.com/modelscope/swift/blob/main/docs/source/LLM/LLM%E5%BE%AE%E8%B0%83%E6%96%87%E6%A1%A3.md#%E5%BE%AE%E8%B0%83).\n\n\n\nExamples\n\n| 模型名称 | 训练方法 |\n|:-------------------|:---------------------------------------------------------------------------------------------------------------------------|\n| qwen_14b | [lora_ddp_ds](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_14b/lora_ddp_ds) |\n| qwen_14b | [qlora](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_14b/qlora) |\n| qwen_14b | [qlora_ddp_ds](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_14b/qlora_ddp_ds) |\n| qwen_14b_chat | [lora_ddp_ds](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_14b_chat/lora_ddp_ds) |\n| qwen_14b_chat | [qlora](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_14b_chat/qlora) |\n| qwen_14b_chat | [qlora_ddp_ds](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_14b_chat/qlora_ddp_ds) |\n| qwen_14b_chat_int4 | [qlora](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_14b_chat_int4/qlora) |\n| qwen_14b_chat_int4 | [qlora_ddp_ds](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_14b_chat_int4/qlora_ddp_ds) |\n| qwen_14b_chat_int8 | [qlora](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_14b_chat_int8/qlora) |\n| qwen_14b_chat_int8 | [qlora_ddp_ds](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_14b_chat_int8/qlora_ddp_ds) |\n| qwen_1_8b_chat | [full](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_1_8b_chat/full) |\n| qwen_1_8b_chat | [full_ddp](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_1_8b_chat/full_ddp) |\n| qwen_72b_chat | [lora_mp](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_72b_chat/lora_mp) |\n| qwen_72b_chat | [lora_mp_ddp](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_72b_chat/lora_mp_ddp) |\n| qwen_72b_chat | [qlora](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_72b_chat/qlora) |\n| qwen_72b_chat_int4 | [qlora_ddp_ds](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_72b_chat_int4/qlora_ddp_ds) |\n| qwen_72b_chat_int8 | [qlora_ddp_ds](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_72b_chat_int8/qlora_ddp_ds) |\n| qwen_7b | [lora_ddp_ds](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_7b/lora_ddp_ds) |\n| qwen_7b | [qlora_ddp](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_7b/qlora_ddp) |\n| qwen_7b_chat | [full](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_7b_chat/full) |\n| qwen_7b_chat | [full_freeze_ddp](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_7b_chat/full_freeze_ddp) |\n| qwen_7b_chat | [full_mp](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_7b_chat/full_mp) |\n| qwen_7b_chat | [full_mp_ddp](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_7b_chat/full_mp_ddp) |\n| qwen_7b_chat | [lora](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_7b_chat/lora) |\n| qwen_7b_chat | [lora_ddp](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_7b_chat/lora_ddp) |\n| qwen_7b_chat | [lora_ddp_ds](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_7b_chat/lora_ddp_ds) |\n| qwen_7b_chat | [lora_mp_ddp](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_7b_chat/lora_mp_ddp) |\n| qwen_7b_chat | [qlora](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_7b_chat/qlora) |\n| qwen_7b_chat | [qlora_ddp](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_7b_chat/qlora_ddp) |\n| qwen_7b_chat | [qlora_ddp_ds](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_7b_chat/qlora_ddp_ds) |\n| qwen_7b_chat_int4 | [qalora](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_7b_chat_int4/qalora) |\n| qwen_7b_chat_int4 | [qlora](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_7b_chat_int4/qlora) |\n| qwen_7b_chat_int4 | [qlora_ddp_ds](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_7b_chat_int4/qlora_ddp_ds) |\n| qwen_7b_chat_int8 | [qlora](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_7b_chat_int8/qlora) |\n| qwen_7b_chat_int8 | [qlora_ddp_ds](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_7b_chat_int8/qlora_ddp_ds) |\n| qwen_audio_chat | [full_mp](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_audio_chat/full_mp) |\n| qwen_audio_chat | [full_mp_ddp](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_audio_chat/full_mp_ddp) |\n| qwen_audio_chat | [lora](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_audio_chat/lora) |\n| qwen_audio_chat | [lora_ddp_ds](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_audio_chat/lora_ddp_ds) |\n| qwen_vl | [lora_ddp_ds](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_vl/lora_ddp_ds) |\n| qwen_vl_chat | [full_mp](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_vl_chat/full_mp) |\n| qwen_vl_chat | [full_mp_ddp](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_vl_chat/full_mp_ddp) |\n| qwen_vl_chat | [lora](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_vl_chat/lora) |\n| qwen_vl_chat | [lora_ddp_ds](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_vl_chat/lora_ddp_ds) |\n| qwen_vl_chat | [qlora](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_vl_chat/qlora) |\n| qwen_vl_chat_int4 | [qlora](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_vl_chat_int4/qlora) |\n| qwen_vl_chat_int4 | [qlora_ddp_ds](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_vl_chat_int4/qlora_ddp_ds) |\n\n\n## Inference\n\n```python\nimport os\nos.environ['CUDA_VISIBLE_DEVICES'] = '0'\n\nfrom swift.llm import (\n get_model_tokenizer, get_template, inference, ModelType, get_default_template_type,\n)\nfrom swift.utils import seed_everything\n\nmodel_type = ModelType.qwen_7b_chat\ntemplate_type = get_default_template_type(model_type)\nprint(f'template_type: {template_type}') # template_type: qwen\n\n\nkwargs = {}\n# kwargs['use_flash_attn'] = True # Use flash_attn if desired\n\nmodel, tokenizer = get_model_tokenizer(model_type, model_kwargs={'device_map': 'auto'}, **kwargs)\n# Modify max_new_tokens\nmodel.generation_config.max_new_tokens = 128\n\ntemplate = get_template(template_type, tokenizer)\nseed_everything(42)\nquery = 'What is the provincial capital of Zhejiang?'\nresponse, history = inference(model, template, query)\nprint(f'query: {query}')\nprint(f'response: {response}')\n\nquery = 'What delicious food can be found here?'\nresponse, history = inference(model, template, query, history)\nprint(f'query: {query}')\nprint(f'response: {response}')\nprint(f'history: {history}')\n\n\"\"\"Output[0]:\nquery: What is the provincial capital of Zhejiang?\nresponse: The provincial capital of Zhejiang is Hangzhou.\nquery: What delicious food can be found here?\nresponse: Hangzhou has many famous delicacies, such as West Lake Vinegar Fish, Longjing Shrimp, Sweet and Sour Spare Ribs, and Maodu. Additionally, there are unique Hangzhou-style pastries like Osmanthus Cake, Lotus Paste Pastry, and Aiwo Steamed Rice Cakes.\nhistory: [('What is the provincial capital of Zhejiang?', 'The provincial capital of Zhejiang is Hangzhou.'), ('What delicious food can be found here?', 'Hangzhou has many famous delicacies, such as West Lake Vinegar Fish, Longjing Shrimp, Sweet and Sour Spare Ribs, and Maodu. Additionally, there are unique Hangzhou-style pastries like Osmanthus Cake, Lotus Paste Pastry, and Aiwo Steamed Rice Cakes.')]\n\"\"\"\n\n# Streaming dialogue output with verbose mode\ninference(model, template, 'What was the first question?', history, verbose=True, stream=True)\n\"\"\"Output[1]:\n[PROMPT]\nYou asked your first question, \"What is the provincial capital of Zhejiang?\"\n[OUTPUT] Your first question was “What is the provincial capital of Zhejiang?”\n\"\"\"\n\nFor more on inference usage, please refer to [here](https://github.com/modelscope/swift/blob/main/docs/source/LLM/LLM_Inference_Guide.md).\n" }, { "path": "recipes/finetune/swift/README_CN.md", "content": "## 介绍\n[SWIFT](https://github.com/modelscope/swift)（Scalable lightWeight Infrastructure for Fine-Tuning）是一个可扩展的轻量级一站式训练、推理深度学习框架。它集成了各种高效的微调方法，如LoRA、QLoRA、阿里云自研的ResTuning-Bypass等，以及开箱即用的训练推理脚本，使开发者可以在单张商业级显卡上微调推理LLM&AIGC模型。此外，SWIFT与PEFT完全兼容，使开发者可以在ModelScope模型体系中使用PEFT的能力。\n\n## 安装\n```shell\n# 设置pip全局镜像\npip config set global.index-url https://mirrors.aliyun.com/pypi/simple/\n# 安装ms-swift\ngit clone https://github.com/modelscope/swift.git\ncd swift\npip install -e .[llm]\n\n# 如果你想要使用deepspeed.\npip install deepspeed -U\n\n# 如果你想要使用基于auto_gptq的qlora训练. (推荐, 效果优于bnb)\n# 支持auto_gptq的模型: `https://github.com/modelscope/swift/blob/main/docs/source/LLM/支持的模型和数据集.md#模型`\n# auto_gptq和cuda版本有对应关系，请按照`https://github.com/PanQiWei/AutoGPTQ#quick-installation`选择版本\npip install auto_gptq -U\n\n# 如果你想要使用基于bnb的qlora训练.\npip install bitsandbytes -U\n\n# 环境对齐 (如果你运行错误, 可以跑下面的代码, 仓库使用最新环境测试)\npip install -r requirements/framework.txt -U\npip install -r requirements/llm.txt -U\n```\n\n\n## webui使用\n\n执行如下命令启动webui通过界面方式进行模型训练推理\n```shell\nswift web-ui\n```\n界面示例如下\n![image](https://modelscope.oss-cn-beijing.aliyuncs.com/resource/swift_webui.jpg)\n\n## 微调\n```python\n# Experimental environment: A10, 3090, V100, ...\n# 20GB GPU memory\nCUDA_VISIBLE_DEVICES=0 \\\nswift sft \\\n --model_id_or_path qwen/Qwen-7B-Chat \\\n --dataset blossom-math-zh \\\n --output_dir output \\\n\n# 使用自己的数据集\nCUDA_VISIBLE_DEVICES=0 \\\nswift sft \\\n --model_id_or_path qwen/Qwen-7B-Chat \\\n --custom_train_dataset_path chatml.jsonl \\\n --output_dir output \\\n\n# 使用DDP\n# Experimental environment: 2 * 3090\n# 2 * 23GB GPU memory\nCUDA_VISIBLE_DEVICES=0,1 \\\nNPROC_PER_NODE=2 \\\nswift sft \\\n --model_id_or_path qwen/Qwen-7B-Chat \\\n --dataset blossom-math-zh \\\n --output_dir output \\\n\n# 多机多卡\n# node0\nCUDA_VISIBLE_DEVICES=0,1,2,3 \\\nNNODES=2 \\\nNODE_RANK=0 \\\nMASTER_ADDR=127.0.0.1 \\\nNPROC_PER_NODE=4 \\\nswift sft \\\n --model_id_or_path qwen/Qwen-7B-Chat \\\n --dataset blossom-math-zh \\\n --output_dir output \\\n# node1\nCUDA_VISIBLE_DEVICES=0,1,2,3 \\\nNNODES=2 \\\nNODE_RANK=1 \\\nMASTER_ADDR=xxx.xxx.xxx.xxx \\\nNPROC_PER_NODE=4 \\\nswift sft \\\n --model_id_or_path qwen/Qwen-7B-Chat \\\n --dataset blossom-math-zh \\\n --output_dir output \\\n```\n更多微调方法参考[这里](https://github.com/modelscope/swift/blob/main/docs/source/LLM/LLM%E5%BE%AE%E8%B0%83%E6%96%87%E6%A1%A3.md#%E5%BE%AE%E8%B0%83)\n\n已有微调代码示例\n| 模型名称 | 训练方法 |\n|:-------------------|:---------------------------------------------------------------------------------------------------------------------------|\n| qwen_14b | [lora_ddp_ds](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_14b/lora_ddp_ds) |\n| qwen_14b | [qlora](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_14b/qlora) |\n| qwen_14b | [qlora_ddp_ds](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_14b/qlora_ddp_ds) |\n| qwen_14b_chat | [lora_ddp_ds](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_14b_chat/lora_ddp_ds) |\n| qwen_14b_chat | [qlora](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_14b_chat/qlora) |\n| qwen_14b_chat | [qlora_ddp_ds](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_14b_chat/qlora_ddp_ds) |\n| qwen_14b_chat_int4 | [qlora](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_14b_chat_int4/qlora) |\n| qwen_14b_chat_int4 | [qlora_ddp_ds](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_14b_chat_int4/qlora_ddp_ds) |\n| qwen_14b_chat_int8 | [qlora](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_14b_chat_int8/qlora) |\n| qwen_14b_chat_int8 | [qlora_ddp_ds](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_14b_chat_int8/qlora_ddp_ds) |\n| qwen_1_8b_chat | [full](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_1_8b_chat/full) |\n| qwen_1_8b_chat | [full_ddp](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_1_8b_chat/full_ddp) |\n| qwen_72b_chat | [lora_mp](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_72b_chat/lora_mp) |\n| qwen_72b_chat | [lora_mp_ddp](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_72b_chat/lora_mp_ddp) |\n| qwen_72b_chat | [qlora](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_72b_chat/qlora) |\n| qwen_72b_chat_int4 | [qlora_ddp_ds](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_72b_chat_int4/qlora_ddp_ds) |\n| qwen_72b_chat_int8 | [qlora_ddp_ds](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_72b_chat_int8/qlora_ddp_ds) |\n| qwen_7b | [lora_ddp_ds](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_7b/lora_ddp_ds) |\n| qwen_7b | [qlora_ddp](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_7b/qlora_ddp) |\n| qwen_7b_chat | [full](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_7b_chat/full) |\n| qwen_7b_chat | [full_freeze_ddp](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_7b_chat/full_freeze_ddp) |\n| qwen_7b_chat | [full_mp](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_7b_chat/full_mp) |\n| qwen_7b_chat | [full_mp_ddp](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_7b_chat/full_mp_ddp) |\n| qwen_7b_chat | [lora](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_7b_chat/lora) |\n| qwen_7b_chat | [lora_ddp](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_7b_chat/lora_ddp) |\n| qwen_7b_chat | [lora_ddp_ds](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_7b_chat/lora_ddp_ds) |\n| qwen_7b_chat | [lora_mp_ddp](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_7b_chat/lora_mp_ddp) |\n| qwen_7b_chat | [qlora](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_7b_chat/qlora) |\n| qwen_7b_chat | [qlora_ddp](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_7b_chat/qlora_ddp) |\n| qwen_7b_chat | [qlora_ddp_ds](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_7b_chat/qlora_ddp_ds) |\n| qwen_7b_chat_int4 | [qalora](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_7b_chat_int4/qalora) |\n| qwen_7b_chat_int4 | [qlora](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_7b_chat_int4/qlora) |\n| qwen_7b_chat_int4 | [qlora_ddp_ds](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_7b_chat_int4/qlora_ddp_ds) |\n| qwen_7b_chat_int8 | [qlora](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_7b_chat_int8/qlora) |\n| qwen_7b_chat_int8 | [qlora_ddp_ds](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_7b_chat_int8/qlora_ddp_ds) |\n| qwen_audio_chat | [full_mp](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_audio_chat/full_mp) |\n| qwen_audio_chat | [full_mp_ddp](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_audio_chat/full_mp_ddp) |\n| qwen_audio_chat | [lora](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_audio_chat/lora) |\n| qwen_audio_chat | [lora_ddp_ds](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_audio_chat/lora_ddp_ds) |\n| qwen_vl | [lora_ddp_ds](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_vl/lora_ddp_ds) |\n| qwen_vl_chat | [full_mp](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_vl_chat/full_mp) |\n| qwen_vl_chat | [full_mp_ddp](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_vl_chat/full_mp_ddp) |\n| qwen_vl_chat | [lora](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_vl_chat/lora) |\n| qwen_vl_chat | [lora_ddp_ds](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_vl_chat/lora_ddp_ds) |\n| qwen_vl_chat | [qlora](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_vl_chat/qlora) |\n| qwen_vl_chat_int4 | [qlora](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_vl_chat_int4/qlora) |\n| qwen_vl_chat_int4 | [qlora_ddp_ds](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_vl_chat_int4/qlora_ddp_ds) |\n\n## 推理\n\n```python\nimport os\nos.environ['CUDA_VISIBLE_DEVICES'] = '0'\n\nfrom swift.llm import (\n get_model_tokenizer, get_template, inference, ModelType, get_default_template_type,\n)\nfrom swift.utils import seed_everything\n\nmodel_type = ModelType.qwen_7b_chat\ntemplate_type = get_default_template_type(model_type)\nprint(f'template_type: {template_type}') # template_type: qwen\n\n\nkwargs = {}\n# kwargs['use_flash_attn'] = True # 使用flash_attn\n\nmodel, tokenizer = get_model_tokenizer(model_type, model_kwargs={'device_map': 'auto'}, **kwargs)\n# 修改max_new_tokens\nmodel.generation_config.max_new_tokens = 128\n\ntemplate = get_template(template_type, tokenizer)\nseed_everything(42)\nquery = '浙江的省会在哪里？'\nresponse, history = inference(model, template, query)\nprint(f'query: {query}')\nprint(f'response: {response}')\nquery = '这有什么好吃的？'\nresponse, history = inference(model, template, query, history)\nprint(f'query: {query}')\nprint(f'response: {response}')\nprint(f'history: {history}')\n\n\"\"\"Out[0]\nquery: 浙江的省会在哪里？\nresponse: 浙江省的省会是杭州。\nquery: 这有什么好吃的？\nresponse: 杭州市有很多著名的美食，例如西湖醋鱼、龙井虾仁、糖醋排骨、毛血旺等。此外，还有杭州特色的点心，如桂花糕、荷花酥、艾窝窝等。\nhistory: [('浙江的省会在哪里？', '浙江省的省会是杭州。'), ('这有什么好吃的？', '杭州市有很多著名的美食，例如西湖醋鱼、龙井虾仁、糖醋排骨、毛血旺等。此外，还有杭州特色的点心，如桂花糕、荷花酥、艾窝窝等。')]\n\"\"\"\n\n# 流式输出对话模板\ninference(model, template, '第一个问题是什么', history, verbose=True, stream=True)\n\"\"\"Out[1]\n[PROMPT]<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n<|im_start|>user\n浙江的省会在哪里？<|im_end|>\n<|im_start|>assistant\n浙江省的省会是杭州。<|im_end|>\n<|im_start|>user\n这有什么好吃的？<|im_end|>\n<|im_start|>assistant\n杭州市有很多著名的美食，例如西湖醋鱼、龙井虾仁、糖醋排骨、毛血旺等。此外，还有杭州特色的点心，如桂花糕、荷花酥、艾窝窝等。<|im_end|>\n<|im_start|>user\n第一个问题是什么<|im_end|>\n<|im_start|>assistant\n[OUTPUT]你的第一个问题是“浙江的省会在哪里？”<|im_end|>\n\"\"\"\n```\n更多推理使用请参考[这里](https://github.com/modelscope/swift/blob/main/docs/source/LLM/LLM%E6%8E%A8%E7%90%86%E6%96%87%E6%A1%A3.md)\n" }, { "path": "recipes/inference/dashscope/README.md", "content": "# Inference Qwen Using DashScope\n\nThe most simple way to use Qwen through APIs is DashScope API service through Alibaba Cloud. We give an introduction to the usage. Additionally, we provide a script for you to deploy an OpenAI-style API on your own servers.\n\nDashScope is the large language model API service provided by Alibaba Cloud, which now supports Qwen. Note that the models behind DashScope are in-house versions temporarily without details provided. The services include `qwen-turbo` and `qwen-plus`, where the former one runs faster and the latter achieves better performance. For more information, visit the documentation [here](https://dashscope.aliyun.com).\n\nPlease head to the official website [link](https://help.aliyun.com/zh/dashscope/developer-reference/activate-dashscope-and-create-an-api-key?spm=a2c4g.11186623.0.0.6c2774fahtfXdn) to create a DashScope account and obtain the API key (AK). We recommend setting the AK with an environment variable:\n```bash\nexport DASHSCOPE_API_KEY=\"YOUR_DASHSCOPE_API_KEY\"\n```\nThen please install the packages and click [here](https://help.aliyun.com/zh/dashscope/developer-reference/install-dashscope-sdk) for the documentation. If you use Python, you can install DashScope with pip:\n```bash\npip install dashscope\n```\nIf you use JAVA SDK, you can install it in this way:\n```xml\n<!-- https://mvnrepository.com/artifact/com.alibaba/dashscope-sdk-java -->\n<dependency>\n <groupId>com.alibaba</groupId>\n <artifactId>dashscope-sdk-java</artifactId>\n <version>the-latest-version</version>\n</dependency>\n```\nThe simplest way to use DashScope is the usage with messages, which is similar to OpenAI API. The example is demonstrated below:\n```python\nimport random\nfrom http import HTTPStatus\nfrom dashscope import Generation\n\n\ndef call_with_messages():\n messages = [{'role': 'system', 'content': 'You are a helpful assistant.'},\n {'role': 'user', 'content': '如何做西红柿鸡蛋？'}]\n gen = Generation()\n response = gen.call(\n Generation.Models.qwen_turbo,\n messages=messages,\n seed=random.randint(1, 10000), # set the random seed, optional, default to 1234 if not set\n result_format='message', # set the result to be \"message\" format.\n )\n return response\n\n\nif __name__ == '__main__':\n response = call_with_messages()\n if response.status_code == HTTPStatus.OK:\n print(response)\n else:\n print('Request id: %s, Status code: %s, error code: %s, error message: %s' % (\n response.request_id, response.status_code,\n response.code, response.message\n ))\n```\nFor more usages, please visit the official website for more details.\n<br><br>\n\n" }, { "path": "recipes/inference/hf_modelscope/README.md", "content": "# Inference Qwen Using 🤖 ModelScope and 🤗 Transformers\n\nBelow, we provide simple examples to show how to inference Qwen with 🤖 ModelScope and 🤗 Transformers.\n\n## Requirements\n\n* python 3.8 and above\n* pytorch 1.12 and above, 2.0 and above are recommended\n* transformers 4.32 and above\n* CUDA 11.4 and above are recommended (this is for GPU users, flash-attention users, etc.)\n<br>\n\n## Installation\n\nYou can use our pre-built docker images to skip most of the environment setup steps, see Section [\"Using Pre-built Docker Images\"](https://github.com/QwenLM/Qwen?tab=readme-ov-file#-docker) for more details. \n\nIf not using docker, please make sure you have setup the environment and installed the required packages. Make sure you meet the above requirements, and then install the dependent libraries.\n\n```bash\npip install -r Qwen/requirements.txt\n```\n\nIf your device supports fp16 or bf16, we recommend installing [flash-attention](https://github.com/Dao-AILab/flash-attention) (**we support flash attention 2 now.**) for higher efficiency and lower memory usage. (**flash-attention is optional and the project can run normally without installing it**)\n\n```bash\ngit clone https://github.com/Dao-AILab/flash-attention\ncd flash-attention && pip install .\n# Below are optional. Installing them might be slow.\n# pip install csrc/layer_norm\n# If the version of flash-attn is higher than 2.1.1, the following is not needed.\n# pip install csrc/rotary\n```\n\nNow you can start with ModelScope or Transformers.\n\n## 🤗 Transformers\n\nTo use Qwen-Chat for the inference, all you need to do is to input a few lines of codes as demonstrated below. Remember to pass in the correct model names or paths, such as \"Qwen/Qwen-7B-Chat\" and \"Qwen/Qwen-14B-Chat\". However, **please make sure that you are using the latest code.**\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\nfrom transformers.generation import GenerationConfig\n\n# Model names: \"Qwen/Qwen-7B-Chat\", \"Qwen/Qwen-14B-Chat\"\ntokenizer = AutoTokenizer.from_pretrained(\"Qwen/Qwen-7B-Chat\", trust_remote_code=True)\n\n# use bf16\n# model = AutoModelForCausalLM.from_pretrained(\"Qwen/Qwen-7B-Chat\", device_map=\"auto\", trust_remote_code=True, bf16=True).eval()\n# use fp16\n# model = AutoModelForCausalLM.from_pretrained(\"Qwen/Qwen-7B-Chat\", device_map=\"auto\", trust_remote_code=True, fp16=True).eval()\n# use cpu only\n# model = AutoModelForCausalLM.from_pretrained(\"Qwen/Qwen-7B-Chat\", device_map=\"cpu\", trust_remote_code=True).eval()\n# use auto mode, automatically select precision based on the device.\nmodel = AutoModelForCausalLM.from_pretrained(\n \"Qwen/Qwen-7B-Chat\",\n device_map=\"auto\",\n trust_remote_code=True\n).eval()\n\n# Specify hyperparameters for generation. But if you use transformers>=4.32.0, there is no need to do this.\n# model.generation_config = GenerationConfig.from_pretrained(\"Qwen/Qwen-7B-Chat\", trust_remote_code=True)\n\n# 1st dialogue turn\nresponse, history = model.chat(tokenizer, \"你好\", history=None)\nprint(response)\n# 你好！很高兴为你提供帮助。\n\n# 2nd dialogue turn\nresponse, history = model.chat(tokenizer, \"给我讲一个年轻人奋斗创业最终取得成功的故事。\", history=history)\nprint(response)\n# 这是一个关于一个年轻人奋斗创业最终取得成功的故事。\n# 故事的主人公叫李明，他来自一个普通的家庭，父母都是普通的工人。从小，李明就立下了一个目标：要成为一名成功的企业家。\n# 为了实现这个目标，李明勤奋学习，考上了大学。在大学期间，他积极参加各种创业比赛，获得了不少奖项。他还利用课余时间去实习，积累了宝贵的经验。\n# 毕业后，李明决定开始自己的创业之路。他开始寻找投资机会，但多次都被拒绝了。然而，他并没有放弃。他继续努力，不断改进自己的创业计划，并寻找新的投资机会。\n# 最终，李明成功地获得了一笔投资，开始了自己的创业之路。他成立了一家科技公司，专注于开发新型软件。在他的领导下，公司迅速发展起来，成为了一家成功的科技企业。\n# 李明的成功并不是偶然的。他勤奋、坚韧、勇于冒险，不断学习和改进自己。他的成功也证明了，只要努力奋斗，任何人都有可能取得成功。\n\n# 3rd dialogue turn\nresponse, history = model.chat(tokenizer, \"给这个故事起一个标题\", history=history)\nprint(response)\n# 《奋斗创业：一个年轻人的成功之路》\n```\n\nRunning Qwen, the base language model, is also simple.\n\n<details>\n <summary>Running Qwen</summary>\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\nfrom transformers.generation import GenerationConfig\n\n# Model names: \"Qwen/Qwen-7B\", \"Qwen/Qwen-14B\" \ntokenizer = AutoTokenizer.from_pretrained(\"Qwen/Qwen-7B\", trust_remote_code=True)\n# use bf16\n# model = AutoModelForCausalLM.from_pretrained(\"Qwen/Qwen-7B\", device_map=\"auto\", trust_remote_code=True, bf16=True).eval()\n# use fp16\n# model = AutoModelForCausalLM.from_pretrained(\"Qwen/Qwen-7B\", device_map=\"auto\", trust_remote_code=True, fp16=True).eval()\n# use cpu only\n# model = AutoModelForCausalLM.from_pretrained(\"Qwen/Qwen-7B\", device_map=\"cpu\", trust_remote_code=True).eval()\n# use auto mode, automatically select precision based on the device.\nmodel = AutoModelForCausalLM.from_pretrained(\n \"Qwen/Qwen-7B\",\n device_map=\"auto\",\n trust_remote_code=True\n).eval()\n\n# Specify hyperparameters for generation. But if you use transformers>=4.32.0, there is no need to do this.\n# model.generation_config = GenerationConfig.from_pretrained(\"Qwen/Qwen-7B\", trust_remote_code=True)\n\ninputs = tokenizer('蒙古国的首都是乌兰巴托（Ulaanbaatar）\\n冰岛的首都是雷克雅未克（Reykjavik）\\n埃塞俄比亚的首都是', return_tensors='pt')\ninputs = inputs.to(model.device)\npred = model.generate(**inputs)\nprint(tokenizer.decode(pred.cpu()[0], skip_special_tokens=True))\n# 蒙古国的首都是乌兰巴托（Ulaanbaatar）\\n冰岛的首都是雷克雅未克（Reykjavik）\\n埃塞俄比亚的首都是亚的斯亚贝巴（Addis Ababa）...\n```\n\n</details>\n\n<p id=\"DownloadModel\">\nIn the event of a network issue while attempting to download model checkpoints and codes from HuggingFace, an alternative approach is to initially fetch the checkpoint from ModelScope and then load it from the local directory as outlined below:\n</p>\n\n```python\nfrom modelscope import snapshot_download\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\n# Downloading model checkpoint to a local dir model_dir\n# model_dir = snapshot_download('qwen/Qwen-7B')\n# model_dir = snapshot_download('qwen/Qwen-7B-Chat')\n# model_dir = snapshot_download('qwen/Qwen-14B')\nmodel_dir = snapshot_download('qwen/Qwen-14B-Chat')\n\n# Loading local checkpoints\n# trust_remote_code is still set as True since we still load codes from local dir instead of transformers\ntokenizer = AutoTokenizer.from_pretrained(model_dir, trust_remote_code=True)\nmodel = AutoModelForCausalLM.from_pretrained(\n model_dir,\n device_map=\"auto\",\n trust_remote_code=True\n).eval()\n```\n\n## 🤖 ModelScope\n\nModelScope is an open-source platform for Model-as-a-Service (MaaS), which provides flexible and cost-effective model service to AI developers. Similarly, you can run the models with ModelScope as shown below:\n\n```python\nfrom modelscope import AutoModelForCausalLM, AutoTokenizer\nfrom modelscope import GenerationConfig\n\n# Model names: \"qwen/Qwen-7B-Chat\", \"qwen/Qwen-14B-Chat\"\ntokenizer = AutoTokenizer.from_pretrained(\"qwen/Qwen-7B-Chat\", trust_remote_code=True)\nmodel = AutoModelForCausalLM.from_pretrained(\"qwen/Qwen-7B-Chat\", device_map=\"auto\", trust_remote_code=True, fp16=True).eval()\nmodel.generation_config = GenerationConfig.from_pretrained(\"Qwen/Qwen-7B-Chat\", trust_remote_code=True) # 可指定不同的生成长度、top_p等相关超参\n\nresponse, history = model.chat(tokenizer, \"你好\", history=None)\nprint(response)\nresponse, history = model.chat(tokenizer, \"浙江的省会在哪里？\", history=history) \nprint(response)\nresponse, history = model.chat(tokenizer, \"它有什么好玩的景点\", history=history)\nprint(response)\n```\n\n## Batch Inference\nQwen supports batch inference. With flash attention enabled, using batch inference can bring a 40% speedup. The example code is shown below:\n```python\nimport torch\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\nfrom transformers import GenerationConfig\nfrom qwen_generation_utils import make_context, decode_tokens, get_stop_words_ids\n\ntokenizer = AutoTokenizer.from_pretrained(\n './',\n pad_token='<|extra_0|>',\n eos_token='<|endoftext|>',\n padding_side='left',\n trust_remote_code=True\n)\nmodel = AutoModelForCausalLM.from_pretrained(\n './',\n pad_token_id=tokenizer.pad_token_id,\n device_map=\"auto\",\n trust_remote_code=True\n).eval()\nmodel.generation_config = GenerationConfig.from_pretrained('./', pad_token_id=tokenizer.pad_token_id)\n\nall_raw_text = [\"我想听你说爱我。\", \"今天我想吃点啥，甜甜的，推荐下\", \"我马上迟到了，怎么做才能不迟到\"]\nbatch_raw_text = []\nfor q in all_raw_text:\n raw_text, _ = make_context(\n tokenizer,\n q,\n system=\"You are a helpful assistant.\",\n max_window_size=model.generation_config.max_window_size,\n chat_format=model.generation_config.chat_format,\n )\n batch_raw_text.append(raw_text)\n\nbatch_input_ids = tokenizer(batch_raw_text, padding='longest')\nbatch_input_ids = torch.LongTensor(batch_input_ids['input_ids']).to(model.device)\nbatch_out_ids = model.generate(\n batch_input_ids,\n return_dict_in_generate=False,\n generation_config=model.generation_config\n)\npadding_lens = [batch_input_ids[i].eq(tokenizer.pad_token_id).sum().item() for i in range(batch_input_ids.size(0))]\n\nbatch_response = [\n decode_tokens(\n batch_out_ids[i][padding_lens[i]:],\n tokenizer,\n raw_text_len=len(batch_raw_text[i]),\n context_length=(batch_input_ids[i].size(0)-padding_lens[i]),\n chat_format=\"chatml\",\n verbose=False,\n errors='replace'\n ) for i in range(len(all_raw_text))\n]\nprint(batch_response)\n\nresponse, _ = model.chat(tokenizer, \"我想听你说爱我。\", history=None)\nprint(response)\n\nresponse, _ = model.chat(tokenizer, \"今天我想吃点啥，甜甜的，推荐下\", history=None)\nprint(response)\n\nresponse, _ = model.chat(tokenizer, \"我马上迟到了，怎么做才能不迟到\", history=None)\nprint(response)\n```\n\n## CPU\n\nTo deploy our models on CPU, we strongly advise you to use [qwen.cpp](https://github.com/QwenLM/qwen.cpp), which is a pure C++ implementation of Qwen and tiktoken. Check the repo for more details!\n\nAlso, it is also simple to directly run the model on CPU, which requires your specification of device:\n\n```python\nmodel = AutoModelForCausalLM.from_pretrained(\"Qwen/Qwen-7B-Chat\", device_map=\"cpu\", trust_remote_code=True).eval()\n```\n\nHowever, it is likely that you suffer from extremely low inference efficiency.\n\n## Multiple GPUs\n\nIf you suffer from lack of GPU memory and you would like to run the model on more than 1 GPU, you can directly use the default loading method, which is now supported by Transformers. The previous method based on `utils.py` is deprecated.\n\nHowever, though this method is simple, the efficiency of the native pipeline parallelism is low. We advise you to use vLLM with FastChat and please read [the section](../vllm/README.md) for deployment." }, { "path": "recipes/inference/quantization/README.md", "content": "# Quantization\n\n## GPTQ\n\nWe provide a solution based on [AutoGPTQ](https://github.com/PanQiWei/AutoGPTQ), and release the Int4 and Int8 quantized models, which achieve nearly lossless model effects but improved performance on both memory costs and inference speed.\n\nHere we demonstrate how to use our provided quantized models for inference. Before you start, make sure you meet the requirements of auto-gptq (e.g., torch 2.0 and above, transformers 4.32.0 and above, etc.) and install the required packages:\n\n```bash\npip install auto-gptq optimum\n```\n\nIf you meet problems installing `auto-gptq`, we advise you to check out the official [repo](https://github.com/PanQiWei/AutoGPTQ) to find a wheel.\n\n> Note: The pre-compiled `auto-gptq` packages strongly depend on the version of `torch` and its CUDA version. Moreover, due to recent update, \n> you may also encounter unsupported version errors from `transformers`, `optimum`, or `peft`.\n> We recommend using the latest versions meeting the following requirements:\n> - torch==2.1 auto-gptq>=0.5.1 transformers>=4.35.0 optimum>=1.14.0 peft>=0.6.1\n> - torch>=2.0,<2.1 auto-gptq<0.5.0 transformers<4.35.0 optimum<1.14.0 peft>=0.5.0,<0.6.0\n\nThen you can load the quantized model easily and run inference as same as usual:\n\n```python\n# Model names: \"Qwen/Qwen-7B-Chat-Int4\", \"Qwen/Qwen-14B-Chat-Int4\"\nmodel = AutoModelForCausalLM.from_pretrained(\n \"Qwen/Qwen-7B-Chat-Int4\",\n device_map=\"auto\",\n trust_remote_code=True\n).eval()\nresponse, history = model.chat(tokenizer, \"Hi\", history=None)\n```\n\nWe illustrate the model performance of both BF16, Int8 and Int4 models on the benchmark, and we find that the quantized model does not suffer from significant performance degradation. Results are shown below:\n\n| Quantization | MMLU | CEval (val) | GSM8K | Humaneval |\n|----------------------|:----:|:-----------:|:-----:|:---------:|\n| Qwen-1.8B-Chat (BF16)| 43.3 | 55.6 | 33.7 | 26.2 |\n| Qwen-1.8B-Chat (Int8)| 43.1 | 55.8 | 33.0 | 27.4 |\n| Qwen-1.8B-Chat (Int4)| 42.9 | 52.8 | 31.2 | 25.0 |\n| Qwen-7B-Chat (BF16) | 55.8 | 59.7 | 50.3 | 37.2 |\n| Qwen-7B-Chat (Int8) | 55.4 | 59.4 | 48.3 | 34.8 |\n| Qwen-7B-Chat (Int4) | 55.1 | 59.2 | 49.7 | 29.9 |\n| Qwen-14B-Chat (BF16) | 64.6 | 69.8 | 60.1 | 43.9 |\n| Qwen-14B-Chat (Int8) | 63.6 | 68.6 | 60.0 | 48.2 |\n| Qwen-14B-Chat (Int4) | 63.3 | 69.0 | 59.8 | 45.7 |\n| Qwen-72B-Chat (BF16) | 74.4 | 80.1 | 76.4 | 64.6 |\n| Qwen-72B-Chat (Int8) | 73.5 | 80.1 | 73.5 | 62.2 |\n| Qwen-72B-Chat (Int4) | 73.4 | 80.1 | 75.3 | 61.6 |\n\n## Quantization of KV cache\n\n> NOTE: Please be aware that due to the internal mechanism of Hugging Face, the support files for this functionality \n> (i.e., `cache_autogptq_cuda_256.cpp` and `cache_autogptq_cuda_kernel_256.cu`) may be missing. Please manually download\n> them from the Hugging Face Hub and place them into the same folder as the other module files.\n\nThe attention KV cache can be quantized and compressed for storage, to get a higher sample throughput. The arguments `use_cache_quantization` and `use_cache_kernel` in `config.json` are provided to enable KV cache quantization. The specific use method is as follows:\n```python\nmodel = AutoModelForCausalLM.from_pretrained(\n \"Qwen/Qwen-7B-Chat\",\n device_map=\"auto\",\n trust_remote_code=True,\n use_cache_quantization=True,\n use_cache_kernel=True,\n use_flash_attn=False\n)\n```\nAttention: Currently, KV cache quantization and flash attention cannot be used at the same time.\nIf you enable KV cache quantization and flash attention at the same time (`use_flash_attn=True`, `use_cache_quantization=True`, `use_cache_kernel=True`), `use_flash_attn` is disabled by default (`use_flash_attn=false`).\n\nWe have verified that the use of the quantized Int8-KV-Cache model does not suffer from significant performance degradation in downstream evaluation. In the following, we focus on profiling its memory footprint in different conditions. \nThe profiling runs on a single A100-SXM4-80G GPU with PyTorch 2.0.1 and CUDA 11.4. \nWe use BF16 models to generate 1024 tokens by default, and \"OOM\" indicates out-of-memory error.\n\nWith KV cache quantization, the model can infer with a larger batch size (bs).\n\n| USE KV Cache | bs=1 | bs=4 | bs=16 | bs=32 | bs=64 | bs=100 |\n|--------------|:------:|:------:|:------:|:------:|:------:|:------:|\n| No | 16.3GB | 24.1GB | 31.7GB | 48.7GB | OOM | OOM |\n| Yes | 15.5GB | 17.2GB | 22.3GB | 30.2GB | 48.2GB | 72.4GB |\n\nWith KV cache quantization the model can save more memory when generating longer sequence (`sl`, sequence length, referring to the number of tokens generated) at the stage of inference.\n\n| USE KV Cache | sl=512 | sl=1024 | sl=2048 | sl=4096 | sl=8192 |\n|--------------|:------:|:-------:|:-------:|:-------:|:-------:|\n| No | 15.2GB | 16.3GB | 17.6GB | 19.5GB | 23.2GB |\n| Yes | 15GB | 15.5GB | 15.8GB | 16.6GB | 17.6GB |\n\nThe model with KV cache quantization will convert the format of `layer_past` from float to int8, and meanwhile the quantized `layer-past` will also store the quantization parameters.\n\nSpecific steps are as follows:\n\n1. Quantize key/value\n```\n qv,scale,zero_point=quantize_cache_v(v)\n```\n2. Store into layer_past\n\nThe following is the format of quantized `layer_past`:\n```\n layer_past=((q_key,key_scale,key_zero_point),\n (q_value,value_scale,value_zero_point))\n```\n\nThe original format of `layer_past` is shown below:\n```\n layer_past=(key,value)\n```\n\nIf you want to use the attention KV which is quantized, you can use the dequantization operation to convert the Int8 key/value back to the float format as follows:\n```\n v=dequantize_cache_torch(qv,scale,zero_point)\n```\n<br>" }, { "path": "recipes/inference/tensorrt/README.md", "content": "# Inference Qwen Using TensorRT-LLM\nBelow, we provide a simple example to show how to inference Qwen by TensorRT-LLM. We recommend using GPUs with compute capability of at least SM_80 such as A10 and A800 to run this example, as we have tested on these GPUs. You can find your gpu compute capability on this [link](https://developer.nvidia.com/cuda-gpus).\n\n## Installation\nYou can use pre-built docker image to run this example. Simultaneously, You can also refer to the official [TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM) for installation and detailed usage.\n```bash\ndocker run --gpus all -it --ipc=host --network=host pai-image-manage-registry.cn-wulanchabu.cr.aliyuncs.com/pai/llm-inference:tensorrt-llm-0.8.0 bash\n```\n## Quickstart\n1. Download model by modelscope\n\n```bash\ncd TensorRT-LLM/examples/qwen\npython3 -c \"from modelscope.hub.snapshot_download import snapshot_download; snapshot_download('Qwen/Qwen-1_8B-Chat', cache_dir='.', revision='master')\"\nmkdir -p ./tmp/Qwen\nmv Qwen/Qwen-1_8B-Chat ./tmp/Qwen/1_8B\n```\n\n2. Build TensorRT engine from HF checkpoint\n\n```bash\npython3 build.py --hf_model_dir ./tmp/Qwen/1_8B/ \\\n --dtype float16 \\\n --remove_input_padding \\\n --use_gpt_attention_plugin float16 \\\n --enable_context_fmha \\\n --use_gemm_plugin float16 \\\n --output_dir ./tmp/Qwen/1_8B/trt_engines/fp16/1-gpu/\n```\n\n3. Inference\n```bash\npython3 ../run.py --input_text \"你好，请问你叫什么？\" \\\n --max_output_len=512 \\\n --tokenizer_dir ./tmp/Qwen/1_8B/ \\\n --engine_dir=./tmp/Qwen/1_8B/trt_engines/fp16/1-gpu\n```\n```\nInput [Text 0]: \"<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n<|im_start|>user\n你好，请问你叫什么？<|im_end|>\n<|im_start|>assistant\n\"\nOutput [Text 0 Beam 0]: \"你好，我是来自阿里云的大规模语言模型，我叫通义千问。\"\n```\n" }, { "path": "recipes/inference/tensorrt/docker/Dockerfile", "content": "FROM nvidia/cuda:12.1.0-devel-ubuntu22.04\n\nRUN apt-get update && \\\n apt-get -y install python3.10 python3-pip openmpi-bin libopenmpi-dev git && \\\n rm -rf /var/lib/apt/lists/*\n\nRUN pip install tensorrt_llm==0.8.0.dev2024011601 -U --no-cache-dir --pre --extra-index-url https://pypi.nvidia.com\n\nRUN pip install --no-cache-dir modelscope==1.11.1\n\nRUN git clone https://github.com/NVIDIA/TensorRT-LLM.git && \\\n cd TensorRT-LLM && \\ \n git checkout c89653021e66ca78c55f02b366f404455bc12e8d && \\\n pip install --no-cache-dir -r examples/qwen/requirements.txt" }, { "path": "recipes/inference/vllm/README.md", "content": "# Inference Qwen Using vLLM\n\nFor deployment and fast inference, we suggest using vLLM. \n\n## Installation\n\nIf you use cuda 12.1 and pytorch 2.1, you can directly use the following command to install vLLM.\n```bash\n# Install vLLM with CUDA 12.1.\npip install vllm\n```\nOtherwise, please refer to the official vLLM [Installation Instructions](https://docs.vllm.ai/en/latest/getting_started/installation.html).\n\nIf you have trouble building vLLM, we recommend using Docker image.\n\n```bash\ndocker run --gpus all -it --rm --ipc=host --network=host qwenllm/qwen:cu121 bash\n```\n\n## GPU Requirements\n\nQwen model use Bfloat16 by default, but Bfloat16 is only supported on GPUs with compute capability of at least 8. For GPUs with compute capability less than 8.0, it is recommended to set the dtype to float16. You can find your gpu compute capability on this [link](https://developer.nvidia.com/cuda-gpus).\n\nWe have tested the GPU memory usage on NVIDIA Tesla V100 32GB by manually adjusting gpu-memory-utilization in eager mode, you can refer to the following table to determine whether your machine is capable of running these models.\n| Model | seq_len 2048 | seq_len 8192 | seq_len 16384 | seq_len 32768 |\n| :--- | ---: | ---: | ---: | ---: |\n| Qwen-1.8B | 6.22G | 7.46G | | |\n| Qwen-7B | 17.94G | 20.96G | | |\n| Qwen-7B-Int4 | 9.10G | 12.26G | | |\n| Qwen-14B | 33.40G | | | |\n| Qwen-14B-Int4 | 13.30G | | | |\n| Qwen-72B | 166.87G | 185.50G | 210.80G | 253.80G |\n| Qwen-72B-int4 | 55.37G | 73.66G | 97.79G | 158.80G |\n\nWe have also listed the models that can run on consumer graphics cards by default sequence length in the following table. If the GPU memory only exceeds the model's memory usage by a small margin, you can make the model run on your machine by reducing the max-model-len parameter.</br>\n(ps: To run Qwen-14B-Int4 on NVIDIA RTX 3080Ti, you need to set gpu-memory-utilization as 0.99 and enforce eager mode)\n\n| GPU Memory | GPU | Support Model |\n| :---: | :---: | :---: |\n| 24GB | NVIDIA RTX 4090/3090/A5000 | Qwen-1.8B/Qwen-7B/Qwen-7B-Int4/Qwen-14B-Int4 |\n| 16GB | NVIDIA RTX A4000 | Qwen-1.8B/Qwen-7B-Int4/Qwen-14B-Int4 |\n| 12GB | NVIDIA RTX 3080Ti/TITAN Xp | Qwen-1.8B/Qwen-14B-Int4 |\n| 11GB | NVIDIA RTX 2080Ti/GTX 1080Ti | Qwen-1.8B |\n| 10GB | NVIDIA RTX 3080 | Qwen-1.8B |\n\n## Usage\n\n### vLLM + Web Demo / OpenAI-like API\n\nYou can use FastChat to launch a web demo or an OpenAI API server. First, install FastChat:\n\n```bash\npip install \"fschat[model_worker,webui]=0.2.33\" \"openai<1.0\"\n```\n\nTo run Qwen with vLLM and FastChat, you need launch a controller by:\n```bash\npython -m fastchat.serve.controller\n```\n\nThen you can launch the model worker, which means loading your model for inference. For single GPU inference, you can directly run:\n```bash\npython -m fastchat.serve.vllm_worker --model-path $model_path --trust-remote-code --dtype bfloat16\n# run int4 model or GPUs with compute capability less than 8.0\n# python -m fastchat.serve.vllm_worker --model-path $model_path --trust-remote-code --dtype float16 \n```\n\nHowever, if you hope to run the model on multiple GPUs for faster inference or larger memory, you can use tensor parallelism supported by vLLM. Suppose you run the model on 4 GPUs, the command is shown below:\n```bash\npython -m fastchat.serve.vllm_worker --model-path $model_path --trust-remote-code --tensor-parallel-size 4 --dtype bfloat16\n# run int4 model or GPUs with compute capability less than 8.0\n# python -m fastchat.serve.vllm_worker --model-path $model_path --trust-remote-code --tensor-parallel-size 4 --dtype float16 \n```\n\nAfter launching your model worker, you can launch a:\n\n* Web UI Demo\n```bash\npython -m fastchat.serve.gradio_web_server\n```\n\n* OpenAI API\n```bash\npython -m fastchat.serve.openai_api_server --host localhost --port 8000\n```\n\nFor OpenAI API server, you can invoke the server in the following manner.\n\n```python\nimport openai\nopenai.api_base = \"http://localhost:8000/v1\"\nopenai.api_key = \"none\"\n\n# create a request activating streaming response\nfor chunk in openai.ChatCompletion.create(\n model=\"Qwen\",\n messages=[\n {\"role\": \"user\", \"content\": \"你好\"}\n ],\n stream=True \n # Specifying stop words in streaming output format is not yet supported and is under development.\n):\n if hasattr(chunk.choices[0].delta, \"content\"):\n print(chunk.choices[0].delta.content, end=\"\", flush=True)\n\n# create a request not activating streaming response\nresponse = openai.ChatCompletion.create(\n model=\"Qwen\",\n messages=[\n {\"role\": \"user\", \"content\": \"你好\"}\n ],\n stream=False,\n stop=[] # You can add custom stop words here, e.g., stop=[\"Observation:\"] for ReAct prompting.\n)\nprint(response.choices[0].message.content)\n```\n\nIf you find `\"POST /v1/chat/completions HTTP/1.1\" 200 OK` in openai_api_server log, it indicates that the call was successful. \n\nvLLM does not support dynamic-NTK ROPE. Therefore, extending long sequences for Qwen model may lead to quality degradation(even gibberish).\n\n### vLLM + Transformer-like Wrapper\n\nYou can download the [wrapper codes](vllm_wrapper.py) and execute the following commands for multiple rounds of dialogue interaction. (Note: It currently only supports the ``model.chat()`` method.)\n\n```python\nfrom vllm_wrapper import vLLMWrapper\n\n# Bfloat16 is only supported on GPUs with compute capability of at least 8.0, \nmodel = vLLMWrapper('Qwen/Qwen-7B-Chat', tensor_parallel_size=1)\n\n# run int4 model or GPUs with compute capability less than 8.0\n# model = vLLMWrapper('Qwen/Qwen-7B-Chat-Int4', tensor_parallel_size=1, dtype=\"float16\")\n\nresponse, history = model.chat(query=\"你好\", history=None)\nprint(response)\nresponse, history = model.chat(query=\"给我讲一个年轻人奋斗创业最终取得成功的故事。\", history=history)\nprint(response)\nresponse, history = model.chat(query=\"给这个故事起一个标题\", history=history)\nprint(response)\n```\n### vLLM Standalone OpenAI-like API\n\nYou can also deploy an OpenAI API server independently through vLLM. First, you need to download [chat template file](template_chatml.jinja).\n\nThen, you can launch an OpenAI API server by following command:\n\n```bash\npython -m vllm.entrypoints.openai.api_server --model $model_path --trust-remote-code --chat-template template_chatml.jinja\n\n# run int4 model or GPUs with compute capability less than 8.0\n# python -m vllm.entrypoints.openai.api_server --model $model_path --trust-remote-code --dtype float16 --chat-template template_chatml.jinja\n```\n\nFor vLLM standalone OpenAI API server, You need to set the `stop_token_ids` parameter to `[151645]` or `stop` parameter to `[\"<|im_end|>\"]` when invoking the server.\n\n```python\nimport openai\nopenai.api_base = \"http://localhost:8000/v1\"\nopenai.api_key = \"none\"\n\n# create a request activating streaming response\nfor chunk in openai.ChatCompletion.create(\n model=\"Qwen\",\n messages=[\n {\"role\": \"user\", \"content\": \"你好\"}\n ],\n stream=True, \n stop_token_ids=[151645]\n):\n if hasattr(chunk.choices[0].delta, \"content\"):\n print(chunk.choices[0].delta.content, end=\"\", flush=True)\n\n# create a request not activating streaming response\nresponse = openai.ChatCompletion.create(\n model=\"Qwen\",\n messages=[\n {\"role\": \"user\", \"content\": \"你好\"}\n ],\n stream=False,\n stop_token_ids=[151645]\n)\nprint(response.choices[0].message.content)\n```" }, { "path": "recipes/inference/vllm/template_chatml.jinja", "content": "{% for message in messages %}\n{% if loop.first and message['role'] != 'system' %}{{ '<|im_start|>system\\nYou are a helpful assistant.<|im_end|>\\n' }}{% endif %}\n{{ '<|im_start|>' + message['role'] + '\\n' + message['content'] + '<|im_end|>\\n' }}\n{% if loop.last and add_generation_prompt %}{{ '<|im_start|>assistant\\n' }}{% endif %}\n{% endfor %}\n" }, { "path": "recipes/inference/vllm/vllm_wrapper.py", "content": "from transformers import PreTrainedTokenizer, GenerationConfig, StoppingCriteriaList\r\nfrom typing import Optional, Callable, List, Tuple, Union\r\nimport copy\r\nimport torch\r\nfrom transformers import AutoTokenizer\r\nfrom transformers.generation.logits_process import LogitsProcessorList\r\nfrom packaging import version\r\n\r\n_ERROR_BAD_CHAT_FORMAT = \"\"\"\\\r\nWe detect you are probably using the pretrained model (rather than chat model) for chatting, since the chat_format in generation_config is not \"chatml\".\r\nIf you are directly using the model downloaded from Huggingface, please make sure you are using our \"Qwen/Qwen-7B-Chat\" Huggingface model (rather than \"Qwen/Qwen-7B\") when you call model.chat().\r\n我们检测到您可能在使用预训练模型（而非chat模型）进行多轮chat，因为您当前在generation_config指定的chat_format，并未设置为我们在对话中所支持的\"chatml\"格式。\r\n如果您在直接使用我们从Huggingface提供的模型，请确保您在调用model.chat()时，使用的是\"Qwen/Qwen-7B-Chat\"模型（而非\"Qwen/Qwen-7B\"预训练模型）。\r\n\"\"\"\r\n\r\nIMEND = \"<|im_end|>\"\r\nENDOFTEXT = \"<|endoftext|>\"\r\n\r\nHistoryType = List[Tuple[str, str]]\r\nTokensType = List[int]\r\nBatchTokensType = List[List[int]]\r\n\r\ndef get_stop_words_ids(chat_format, tokenizer):\r\n if chat_format == \"raw\":\r\n stop_words_ids = [tokenizer.encode(\"Human:\"), [tokenizer.eod_id]]\r\n elif chat_format == \"chatml\":\r\n stop_words_ids = [[tokenizer.im_end_id], [tokenizer.im_start_id]]\r\n else:\r\n raise NotImplementedError(f\"Unknown chat format {chat_format!r}\")\r\n return stop_words_ids\r\n\r\ndef make_context(\r\n tokenizer: PreTrainedTokenizer,\r\n query: str,\r\n history: List[Tuple[str, str]] = None,\r\n system: str = \"\",\r\n max_window_size: int = 6144,\r\n chat_format: str = \"chatml\",\r\n):\r\n if history is None:\r\n history = []\r\n\r\n if chat_format == \"chatml\":\r\n im_start_tokens = [tokenizer.im_start_id]\r\n im_end_tokens = [tokenizer.im_end_id]\r\n im_start, im_end = tokenizer.decode(im_start_tokens, skip_special_tokens=False), tokenizer.decode(im_end_tokens, skip_special_tokens=False)\r\n nl_tokens = tokenizer.encode(\"\\n\")\r\n\r\n def _tokenize_str(role, content):\r\n return f\"{role}\\n{content}\", tokenizer.encode(\r\n role, allowed_special=set()\r\n ) + nl_tokens + tokenizer.encode(content, allowed_special=set())\r\n\r\n system_text, system_tokens_part = _tokenize_str(\"system\", system)\r\n system_tokens = im_start_tokens + system_tokens_part + im_end_tokens\r\n\r\n raw_text = \"\"\r\n context_tokens = []\r\n\r\n for turn_query, turn_response in reversed(history):\r\n query_text, query_tokens_part = _tokenize_str(\"user\", turn_query)\r\n query_tokens = im_start_tokens + query_tokens_part + im_end_tokens\r\n response_text, response_tokens_part = _tokenize_str(\r\n \"assistant\", turn_response\r\n )\r\n response_tokens = im_start_tokens + response_tokens_part + im_end_tokens\r\n\r\n next_context_tokens = nl_tokens + query_tokens + nl_tokens + response_tokens\r\n prev_chat = (\r\n f\"\\n{im_start}{query_text}{im_end}\\n{im_start}{response_text}{im_end}\"\r\n )\r\n\r\n current_context_size = (\r\n len(system_tokens) + len(next_context_tokens) + len(context_tokens)\r\n )\r\n if current_context_size < max_window_size:\r\n context_tokens = next_context_tokens + context_tokens\r\n raw_text = prev_chat + raw_text\r\n else:\r\n break\r\n\r\n context_tokens = system_tokens + context_tokens\r\n raw_text = f\"{im_start}{system_text}{im_end}\" + raw_text\r\n context_tokens += (\r\n nl_tokens\r\n + im_start_tokens\r\n + _tokenize_str(\"user\", query)[1]\r\n + im_end_tokens\r\n + nl_tokens\r\n + im_start_tokens\r\n + tokenizer.encode(\"assistant\")\r\n + nl_tokens\r\n )\r\n raw_text += f\"\\n{im_start}user\\n{query}{im_end}\\n{im_start}assistant\\n\"\r\n\r\n elif chat_format == \"raw\":\r\n raw_text = query\r\n context_tokens = tokenizer.encode(raw_text)\r\n else:\r\n raise NotImplementedError(f\"Unknown chat format {chat_format!r}\")\r\n\r\n return raw_text, context_tokens\r\n\r\nclass vLLMWrapper:\r\n def __init__(self,\r\n model_dir: str,\r\n trust_remote_code: bool = True,\r\n tensor_parallel_size: int = 1,\r\n gpu_memory_utilization: float = 0.98,\r\n dtype: str = \"bfloat16\",\r\n **kwargs):\r\n\r\n if dtype not in (\"bfloat16\", \"float16\", \"float32\"):\r\n print(\"now not support {}!\".format(dtype))\r\n raise Exception\r\n\r\n # build generation_config\r\n self.generation_config = GenerationConfig.from_pretrained(model_dir, trust_remote_code=trust_remote_code)\r\n\r\n # build tokenizer\r\n self.tokenizer = AutoTokenizer.from_pretrained(model_dir, trust_remote_code=True)\r\n self.tokenizer.eos_token_id = self.generation_config.eos_token_id\r\n\r\n self.stop_words_ids = []\r\n\r\n from vllm import LLM\r\n import vllm\r\n if version.parse(vllm.__version__) >= version.parse(\"0.2.2\"):\r\n self.__vllm_support_repetition_penalty = True\r\n else:\r\n self.__vllm_support_repetition_penalty = False\r\n\r\n quantization = getattr(kwargs, 'quantization', None)\r\n\r\n self.model = LLM(model=model_dir,\r\n tokenizer=model_dir,\r\n tensor_parallel_size=tensor_parallel_size,\r\n trust_remote_code=trust_remote_code,\r\n quantization=quantization,\r\n gpu_memory_utilization=gpu_memory_utilization,\r\n dtype=dtype)\r\n\r\n for stop_id in get_stop_words_ids(self.generation_config.chat_format, self.tokenizer):\r\n self.stop_words_ids.extend(stop_id)\r\n self.stop_words_ids.extend([self.generation_config.eos_token_id])\r\n\r\n def chat(self,\r\n query: str,\r\n history: Optional[HistoryType],\r\n tokenizer: PreTrainedTokenizer = None,\r\n system: str = \"You are a helpful assistant.\",\r\n generation_config: Optional[GenerationConfig] = None,\r\n **kwargs):\r\n generation_config = generation_config if generation_config is not None else self.generation_config\r\n tokenizer = self.tokenizer if tokenizer is None else tokenizer\r\n\r\n assert generation_config.chat_format == 'chatml', _ERROR_BAD_CHAT_FORMAT\r\n if not self.__vllm_support_repetition_penalty and generation_config.repetition_penalty != 1:\r\n raise RuntimeError(\"The installed vLLM doesn't support repetition_penalty, please set ``model.generation_config.repetition_penalty = 1`` or install vllm>=0.2.2\")\r\n\r\n if history is None:\r\n history = []\r\n else:\r\n # make a copy of the user's input such that is is left untouched\r\n history = copy.deepcopy(history)\r\n\r\n extra_stop_words_ids = kwargs.get('stop_words_ids', None)\r\n if extra_stop_words_ids is None:\r\n extra_stop_words_ids = []\r\n\r\n max_window_size = kwargs.get('max_window_size', None)\r\n if max_window_size is None:\r\n max_window_size = generation_config.max_window_size\r\n\r\n from vllm.sampling_params import SamplingParams\r\n sampling_kwargs = {\r\n \"stop_token_ids\": self.stop_words_ids,\r\n \"early_stopping\": False,\r\n \"top_p\": generation_config.top_p,\r\n \"top_k\": -1 if generation_config.top_k == 0 else generation_config.top_k,\r\n \"temperature\": generation_config.temperature,\r\n \"max_tokens\": generation_config.max_new_tokens,\r\n \"repetition_penalty\": generation_config.repetition_penalty\r\n }\r\n if not self.__vllm_support_repetition_penalty:\r\n sampling_kwargs.pop(\"repetition_penalty\")\r\n sampling_params = SamplingParams(**sampling_kwargs)\r\n\r\n raw_text, context_tokens = make_context(\r\n self.tokenizer,\r\n query,\r\n history=history,\r\n system=system,\r\n max_window_size=max_window_size,\r\n chat_format=generation_config.chat_format,\r\n )\r\n\r\n req_outputs = self.model.generate([query],\r\n sampling_params=sampling_params,\r\n prompt_token_ids=[context_tokens])\r\n req_output = req_outputs[0]\r\n\r\n prompt_str = req_output.prompt\r\n prompt_ids = req_output.prompt_token_ids\r\n req_sample_output_ids = []\r\n req_sample_output_strs = []\r\n for sample in req_output.outputs:\r\n output_str = sample.text\r\n output_ids = sample.token_ids\r\n if IMEND in output_str:\r\n output_str = output_str[:-len(IMEND)]\r\n if ENDOFTEXT in output_str:\r\n output_str = output_str[:-len(ENDOFTEXT)]\r\n req_sample_output_ids.append(prompt_ids + output_ids)\r\n req_sample_output_strs.append(prompt_str + output_str)\r\n assert len(req_sample_output_strs) == 1\r\n response = req_sample_output_strs[0][len(prompt_str):]\r\n history.append((prompt_str, response))\r\n\r\n return response, history\r\n\r\nif __name__ == '__main__':\r\n\r\n model_dir = 'Qwen/Qwen-72B-Chat'\r\n tensor_parallel_size = 2\r\n\r\n model = vLLMWrapper(model_dir,\r\n tensor_parallel_size=tensor_parallel_size,\r\n )\r\n\r\n response, history = model.chat(query=\"你好\",\r\n history=None)\r\n print(response)\r\n response, history = model.chat(query=\"给我讲一个年轻人奋斗创业最终取得成功的故事。\",\r\n history=history)\r\n print(response)\r\n response, history = model.chat(query=\"给这个故事起一个标题\",\r\n history=history)\r\n print(response)\r\n" }, { "path": "recipes/quickstart/qwen.ipynb", "content": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Qwen Quick Start Notebook\\n\",\n \"\\n\",\n \"This notebook shows how to train and infer the Qwen-7B-Chat model on a single GPU. Similarly, Qwen-1.8B-Chat, Qwen-14B-Chat can also be leveraged for the following steps. We only need to modify the corresponding `model name` and hyper-parameters. The training and inference of Qwen-72B-Chat requires higher GPU requirements and larger disk space.\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Requirements\\n\",\n \"- Python 3.8 and above\\n\",\n \"- Pytorch 1.12 and above, 2.0 and above are recommended\\n\",\n \"- CUDA 11.4 and above are recommended (this is for GPU users, flash-attention users, etc.)\\n\",\n \"We test the training of the model on an A10 GPU (24GB).\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Extra\\n\",\n \"If you need to speed up, you can install `flash-attention`. The details of the installation can be found [here](https://github.com/Dao-AILab/flash-attention).\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"!git clone https://github.com/Dao-AILab/flash-attention\\n\",\n \"!cd flash-attention && pip install .\\n\",\n \"# Below are optional. Installing them might be slow.\\n\",\n \"# !pip install csrc/layer_norm\\n\",\n \"# If the version of flash-attn is higher than 2.1.1, the following is not needed.\\n\",\n \"# !pip install csrc/rotary\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"### Step 0: Install Package Requirements\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"tags\": []\n },\n \"outputs\": [],\n \"source\": [\n \"!pip install transformers>=4.32.0 accelerate tiktoken einops scipy transformers_stream_generator==0.0.4 peft deepspeed modelscope\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"### Step 1: Download Model\\n\",\n \"When using `transformers` in some regions, the model cannot be automatically downloaded due to network problems. We recommend using `modelscope` to download the model first, and then use `transformers` for inference.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"tags\": []\n },\n \"outputs\": [],\n \"source\": [\n \"from modelscope import snapshot_download\\n\",\n \"\\n\",\n \"# Downloading model checkpoint to a local dir model_dir.\\n\",\n \"model_dir = snapshot_download('Qwen/Qwen-7B-Chat', cache_dir='.', revision='master')\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"### Step 2: Direct Model Inference \\n\",\n \"We recommend two ways to do model inference: `modelscope` and `transformers`.\\n\",\n \"\\n\",\n \"#### 2.1 Model Inference with ModelScope\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"ExecutionIndicator\": {\n \"show\": true\n },\n \"tags\": []\n },\n \"outputs\": [],\n \"source\": [\n \"from modelscope import AutoModelForCausalLM, AutoTokenizer\\n\",\n \"from modelscope import GenerationConfig\\n\",\n \"\\n\",\n \"# Note: The default behavior now has injection attack prevention off.\\n\",\n \"tokenizer = AutoTokenizer.from_pretrained(\\\"Qwen/Qwen-7B-Chat/\\\", trust_remote_code=True)\\n\",\n \"\\n\",\n \"# use bf16\\n\",\n \"# model = AutoModelForCausalLM.from_pretrained(\\\"qwen/Qwen-7B-Chat/\\\", device_map=\\\"auto\\\", trust_remote_code=True, bf16=True).eval()\\n\",\n \"# use fp16\\n\",\n \"# model = AutoModelForCausalLM.from_pretrained(\\\"qwen/Qwen-7B-Chat/\\\", device_map=\\\"auto\\\", trust_remote_code=True, fp16=True).eval()\\n\",\n \"# use cpu only\\n\",\n \"# model = AutoModelForCausalLM.from_pretrained(\\\"qwen/Qwen-7B-Chat/\\\", device_map=\\\"cpu\\\", trust_remote_code=True).eval()\\n\",\n \"# use auto mode, automatically select precision based on the device.\\n\",\n \"model = AutoModelForCausalLM.from_pretrained(\\\"Qwen/Qwen-7B-Chat/\\\", device_map=\\\"auto\\\", trust_remote_code=True).eval()\\n\",\n \"\\n\",\n \"# Specify hyperparameters for generation. But if you use transformers>=4.32.0, there is no need to do this.\\n\",\n \"# model.generation_config = GenerationConfig.from_pretrained(\\\"Qwen/Qwen-7B-Chat/\\\", trust_remote_code=True) # 可指定不同的生成长度、top_p等相关超参\\n\",\n \"\\n\",\n \"# 第一轮对话 1st dialogue turn\\n\",\n \"response, history = model.chat(tokenizer, \\\"你好\\\", history=None)\\n\",\n \"print(response)\\n\",\n \"# 你好！很高兴为你提供帮助。\\n\",\n \"\\n\",\n \"# 第二轮对话 2nd dialogue turn\\n\",\n \"response, history = model.chat(tokenizer, \\\"给我讲一个年轻人奋斗创业最终取得成功的故事。\\\", history=history)\\n\",\n \"print(response)\\n\",\n \"# 这是一个关于一个年轻人奋斗创业最终取得成功的故事。\\n\",\n \"# 故事的主人公叫李明，他来自一个普通的家庭，父母都是普通的工人。从小，李明就立下了一个目标：要成为一名成功的企业家。\\n\",\n \"# 为了实现这个目标，李明勤奋学习，考上了大学。在大学期间，他积极参加各种创业比赛，获得了不少奖项。他还利用课余时间去实习，积累了宝贵的经验。\\n\",\n \"# 毕业后，李明决定开始自己的创业之路。他开始寻找投资机会，但多次都被拒绝了。然而，他并没有放弃。他继续努力，不断改进自己的创业计划，并寻找新的投资机会。\\n\",\n \"# 最终，李明成功地获得了一笔投资，开始了自己的创业之路。他成立了一家科技公司，专注于开发新型软件。在他的领导下，公司迅速发展起来，成为了一家成功的科技企业。\\n\",\n \"# 李明的成功并不是偶然的。他勤奋、坚韧、勇于冒险，不断学习和改进自己。他的成功也证明了，只要努力奋斗，任何人都有可能取得成功。\\n\",\n \"\\n\",\n \"# 第三轮对话 3rd dialogue turn\\n\",\n \"response, history = model.chat(tokenizer, \\\"给这个故事起一个标题\\\", history=history)\\n\",\n \"print(response)\\n\",\n \"# 《奋斗创业：一个年轻人的成功之路》\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"#### 2.2 Model Inference with transformers\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"ExecutionIndicator\": {\n \"show\": true\n },\n \"tags\": []\n },\n \"outputs\": [],\n \"source\": [\n \"from transformers import AutoModelForCausalLM, AutoTokenizer\\n\",\n \"from transformers.generation import GenerationConfig\\n\",\n \"\\n\",\n \"tokenizer = AutoTokenizer.from_pretrained(\\\"Qwen/Qwen-7B-Chat/\\\", trust_remote_code=True)\\n\",\n \"\\n\",\n \"# use bf16\\n\",\n \"# model = AutoModelForCausalLM.from_pretrained(\\\"Qwen/Qwen-7B-Chat/\\\", device_map=\\\"auto\\\", trust_remote_code=True, bf16=True).eval()\\n\",\n \"# use fp16\\n\",\n \"# model = AutoModelForCausalLM.from_pretrained(\\\"Qwen/Qwen-7B-Chat/\\\", device_map=\\\"auto\\\", trust_remote_code=True, fp16=True).eval()\\n\",\n \"# use cpu only\\n\",\n \"# model = AutoModelForCausalLM.from_pretrained(\\\"Qwen/Qwen-7B-Chat/\\\", device_map=\\\"cpu\\\", trust_remote_code=True).eval()\\n\",\n \"# use auto mode, automatically select precision based on the device.\\n\",\n \"model = AutoModelForCausalLM.from_pretrained(\\n\",\n \" \\\"Qwen/Qwen-7B-Chat/\\\",\\n\",\n \" device_map=\\\"auto\\\",\\n\",\n \" trust_remote_code=True\\n\",\n \").eval()\\n\",\n \"\\n\",\n \"# Specify hyperparameters for generation. But if you use transformers>=4.32.0, there is no need to do this.\\n\",\n \"# model.generation_config = GenerationConfig.from_pretrained(\\\"Qwen/Qwen-7B-Chat/\\\", trust_remote_code=True)\\n\",\n \"\\n\",\n \"# 1st dialogue turn\\n\",\n \"response, history = model.chat(tokenizer, \\\"你好\\\", history=None)\\n\",\n \"print(response)\\n\",\n \"# 你好！很高兴为你提供帮助。\\n\",\n \"\\n\",\n \"# 2nd dialogue turn\\n\",\n \"response, history = model.chat(tokenizer, \\\"给我讲一个年轻人奋斗创业最终取得成功的故事。\\\", history=history)\\n\",\n \"print(response)\\n\",\n \"# 这是一个关于一个年轻人奋斗创业最终取得成功的故事。\\n\",\n \"# 故事的主人公叫李明，他来自一个普通的家庭，父母都是普通的工人。从小，李明就立下了一个目标：要成为一名成功的企业家。\\n\",\n \"# 为了实现这个目标，李明勤奋学习，考上了大学。在大学期间，他积极参加各种创业比赛，获得了不少奖项。他还利用课余时间去实习，积累了宝贵的经验。\\n\",\n \"# 毕业后，李明决定开始自己的创业之路。他开始寻找投资机会，但多次都被拒绝了。然而，他并没有放弃。他继续努力，不断改进自己的创业计划，并寻找新的投资机会。\\n\",\n \"# 最终，李明成功地获得了一笔投资，开始了自己的创业之路。他成立了一家科技公司，专注于开发新型软件。在他的领导下，公司迅速发展起来，成为了一家成功的科技企业。\\n\",\n \"# 李明的成功并不是偶然的。他勤奋、坚韧、勇于冒险，不断学习和改进自己。他的成功也证明了，只要努力奋斗，任何人都有可能取得成功。\\n\",\n \"\\n\",\n \"# 3rd dialogue turn\\n\",\n \"response, history = model.chat(tokenizer, \\\"给这个故事起一个标题\\\", history=history)\\n\",\n \"print(response)\\n\",\n \"# 《奋斗创业：一个年轻人的成功之路》\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"### Step 3: LoRA Fine-Tuning Model (Single GPU)\\n\",\n \"\\n\",\n \"#### 3.1 Download Example Training Data\\n\",\n \"Download the data required for training; here, we provide a tiny dataset as an example. It is sampled from [Belle](https://github.com/LianjiaTech/BELLE).\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"tags\": []\n },\n \"outputs\": [],\n \"source\": [\n \"!wget https://atp-modelzoo-sh.oss-cn-shanghai.aliyuncs.com/release/tutorials/qwen_recipes/Belle_sampled_qwen.json\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"You can refer to this format to prepare the dataset. Below is a simple example list with 1 sample:\\n\",\n \"\\n\",\n \"```json\\n\",\n \"[\\n\",\n \" {\\n\",\n \" \\\"id\\\": \\\"identity_0\\\",\\n\",\n \" \\\"conversations\\\": [\\n\",\n \" {\\n\",\n \" \\\"from\\\": \\\"user\\\",\\n\",\n \" \\\"value\\\": \\\"你好\\\"\\n\",\n \" },\\n\",\n \" {\\n\",\n \" \\\"from\\\": \\\"assistant\\\",\\n\",\n \" \\\"value\\\": \\\"我是一个语言模型，我叫通义千问。\\\"\\n\",\n \" }\\n\",\n \" ]\\n\",\n \" }\\n\",\n \"]\\n\",\n \"```\\n\",\n \"\\n\",\n \"You can also use multi-turn conversations as the training set. Here is a simple example:\\n\",\n \"\\n\",\n \"```json\\n\",\n \"[\\n\",\n \" {\\n\",\n \" \\\"id\\\": \\\"identity_0\\\",\\n\",\n \" \\\"conversations\\\": [\\n\",\n \" {\\n\",\n \" \\\"from\\\": \\\"user\\\",\\n\",\n \" \\\"value\\\": \\\"你好\\\"\\n\",\n \" },\\n\",\n \" {\\n\",\n \" \\\"from\\\": \\\"assistant\\\",\\n\",\n \" \\\"value\\\": \\\"你好！我是一名AI助手，我叫通义千问，有需要请告诉我。\\\"\\n\",\n \" },\\n\",\n \" {\\n\",\n \" \\\"from\\\": \\\"user\\\",\\n\",\n \" \\\"value\\\": \\\"你都能做什么\\\"\\n\",\n \" },\\n\",\n \" {\\n\",\n \" \\\"from\\\": \\\"assistant\\\",\\n\",\n \" \\\"value\\\": \\\"我能做很多事情，包括但不限于回答各种领域的问题、提供实用建议和指导、进行多轮对话交流、文本生成等。\\\"\\n\",\n \" }\\n\",\n \" ]\\n\",\n \" }\\n\",\n \"]\\n\",\n \"```\\n\",\n \"\\n\",\n \"#### 3.2 Fine-Tune the Model\\n\",\n \"\\n\",\n \"You can directly run the prepared training script to fine-tune the model. Remember to check `model_name_or_path`.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"tags\": []\n },\n \"outputs\": [],\n \"source\": [\n \"!python ../finetune/deepspeed/finetune.py \\\\\\n\",\n \" --model_name_or_path \\\"Qwen/Qwen-7B-Chat/\\\"\\\\\\n\",\n \" --data_path \\\"Belle_sampled_qwen.json\\\"\\\\\\n\",\n \" --bf16 \\\\\\n\",\n \" --output_dir \\\"output_qwen\\\" \\\\\\n\",\n \" --num_train_epochs 5 \\\\\\n\",\n \" --per_device_train_batch_size 1 \\\\\\n\",\n \" --per_device_eval_batch_size 1 \\\\\\n\",\n \" --gradient_accumulation_steps 16 \\\\\\n\",\n \" --evaluation_strategy \\\"no\\\" \\\\\\n\",\n \" --save_strategy \\\"steps\\\" \\\\\\n\",\n \" --save_steps 1000 \\\\\\n\",\n \" --save_total_limit 10 \\\\\\n\",\n \" --learning_rate 1e-5 \\\\\\n\",\n \" --weight_decay 0.1 \\\\\\n\",\n \" --adam_beta2 0.95 \\\\\\n\",\n \" --warmup_ratio 0.01 \\\\\\n\",\n \" --lr_scheduler_type \\\"cosine\\\" \\\\\\n\",\n \" --logging_steps 1 \\\\\\n\",\n \" --report_to \\\"none\\\" \\\\\\n\",\n \" --model_max_length 512 \\\\\\n\",\n \" --gradient_checkpointing \\\\\\n\",\n \" --lazy_preprocess \\\\\\n\",\n \" --use_lora\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"### 3.3 Merge Weights\\n\",\n \"\\n\",\n \"LoRA training only saves the adapter parameters. You can load the fine-tuned model and merge weights as shown below:\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"from transformers import AutoModelForCausalLM\\n\",\n \"from peft import PeftModel\\n\",\n \"import torch\\n\",\n \"\\n\",\n \"model = AutoModelForCausalLM.from_pretrained(\\\"Qwen/Qwen-7B-Chat/\\\", torch_dtype=torch.float16, device_map=\\\"auto\\\", trust_remote_code=True)\\n\",\n \"model = PeftModel.from_pretrained(model, \\\"output_qwen/\\\")\\n\",\n \"merged_model = model.merge_and_unload()\\n\",\n \"merged_model.save_pretrained(\\\"output_qwen_merged\\\", max_shard_size=\\\"2048MB\\\", safe_serialization=True)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"The tokenizer files are not saved in the new directory in this step. You can copy the tokenizer files or use the following code:\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"from transformers import AutoTokenizer\\n\",\n \"\\n\",\n \"tokenizer = AutoTokenizer.from_pretrained(\\n\",\n \" \\\"Qwen/Qwen-7B-Chat/\\\",\\n\",\n \" trust_remote_code=True\\n\",\n \")\\n\",\n \"\\n\",\n \"tokenizer.save_pretrained(\\\"output_qwen_merged\\\")\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"### 3.4 Test the Model\\n\",\n \"\\n\",\n \"After merging the weights, we can test the model as follows:\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"from transformers import AutoModelForCausalLM, AutoTokenizer\\n\",\n \"from transformers.generation import GenerationConfig\\n\",\n \"\\n\",\n \"tokenizer = AutoTokenizer.from_pretrained(\\\"output_qwen_merged\\\", trust_remote_code=True)\\n\",\n \"model = AutoModelForCausalLM.from_pretrained(\\n\",\n \" \\\"output_qwen_merged\\\",\\n\",\n \" device_map=\\\"auto\\\",\\n\",\n \" trust_remote_code=True\\n\",\n \").eval()\\n\",\n \"\\n\",\n \"response, history = model.chat(tokenizer, \\\"你好\\\", history=None)\\n\",\n \"print(response)\"\n ]\n }\n ],\n \"metadata\": {\n \"kernelspec\": {\n \"display_name\": \"Python 3 (ipykernel)\",\n \"language\": \"python\",\n \"name\": \"python3\"\n },\n \"language_info\": {\n \"codemirror_mode\": {\n \"name\": \"ipython\",\n \"version\": 3\n },\n \"file_extension\": \".py\",\n \"mimetype\": \"text/x-python\",\n \"name\": \"python\",\n \"nbconvert_exporter\": \"python\",\n \"pygments_lexer\": \"ipython3\",\n \"version\": \"3.10.13\"\n },\n \"vscode\": {\n \"interpreter\": {\n \"hash\": \"2d58e898dde0263bc564c6968b04150abacfd33eed9b19aaa8e45c040360e146\"\n }\n }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 4\n}\n" }, { "path": "recipes/tests/README.md", "content": "# Unit testing\n- Run all unit testing\n```bash\ncd tests && pytest -s \n```\n- Run unit testing under a single folder\n```bash\ncd tests && pytest -s {dir}\n```\n- Rerun the test cases that failed in the last run\n```bash\ncd tests && pytest -s --lf\n```" }, { "path": "recipes/tests/__init__.py", "content": "" }, { "path": "recipes/tests/assets/test_sampled_qwen.json", "content": "[{\"conversations\": [{\"from\": \"user\", \"value\": \"你好\"}, {\"from\": \"assistant\", \"value\": \"你好！很高兴为你提供帮助。\"}], \"id\": \"identity_0\"}, {\"conversations\": [{\"from\": \"user\", \"value\": \"给我讲一个年轻人奋斗创业最终取得成功的故事。\"}, {\"from\": \"assistant\", \"value\": \"这是一个关于一个年轻人奋斗创业最终取得成功的故事。\"}], \"id\": \"identity_1\"}]" }, { "path": "recipes/tests/test_finetune/test_finetune_ds.py", "content": "import os\nimport sys\nimport pytest\nimport shutil\nfrom itertools import product\nimport torch\nfrom modelscope.hub.snapshot_download import snapshot_download\n\nsys.path.append(os.path.dirname(__file__) + \"/..\")\nfrom utils import run_in_subprocess\nfrom ut_config import (\n MODEL_TYPE,\n DOCKER_VERSION_CU114,\n DOCKER_VERSION_CU117,\n DOCKER_VERSION_CU121,\n DOCKER_MOUNT_DIR,\n DOCKER_TEST_DIR,\n DATA_DIR,\n DS_CONFIG_ZERO2_DIR,\n DS_CONFIG_ZERO3_DIR,\n)\n\nis_chat = [\"chat\", \"base\"]\ndocker_version = [DOCKER_VERSION_CU114, DOCKER_VERSION_CU117, DOCKER_VERSION_CU121]\n# ZeRO3 is incompatible with LoRA when finetuning on base model.\n# FSDP or ZeRO3 are incompatible with QLoRA.\nparametrize_list_none_ds = list(\n product(*[[1], [\"full\", \"lora\"], is_chat, docker_version, [None]])\n)\nparametrize_list_ds_zero2 = list(\n product(*[[2], [\"full\", \"lora\"], is_chat, docker_version, [DS_CONFIG_ZERO2_DIR]])\n)\nparametrize_list_ds_zero3 = list(\n product(*[[2], [\"full\"], is_chat, docker_version, [DS_CONFIG_ZERO3_DIR]])\n) + list(product(*[[2], [\"lora\"], [\"chat\"], docker_version, [DS_CONFIG_ZERO3_DIR]]))\nparametrize_list_qlora = list(\n product(*[[1, 2], [\"qlora\"], [\"chat\"], docker_version, [None, DS_CONFIG_ZERO2_DIR]])\n)\nparametrize_list = (\n parametrize_list_none_ds\n + parametrize_list_ds_zero2\n + parametrize_list_ds_zero3\n + parametrize_list_qlora\n)\n\n\n@pytest.mark.parametrize(\n \"num_gpus,train_type,is_chat,docker_version,deepspeed\", parametrize_list\n)\ndef test_finetune(num_gpus, train_type, is_chat, docker_version, deepspeed):\n cmd_docker = f\"docker run --gpus all --ipc=host --network=host --rm -v {os.getcwd()}/../../../Qwen:{DOCKER_MOUNT_DIR} {docker_version} /bin/bash -c \"\n cmd = \"\"\n # for GPUs SM < 80\n is_ampere = torch.cuda.get_device_capability()[0] >= 8\n if not is_ampere:\n cmd = f\"pip uninstall -y flash-attn && \"\n\n model_type = f\"{MODEL_TYPE}-Chat\" if is_chat == \"chat\" else MODEL_TYPE\n model_type = f\"{model_type}-Int4\" if train_type == \"qlora\" else model_type\n cmd += f\"\"\"torchrun --nproc_per_node {num_gpus} --nnodes 1 --node_rank 0 --master_addr localhost --master_port 12345 {DOCKER_MOUNT_DIR}/finetune.py \\\n --model_name_or_path \"{DOCKER_TEST_DIR}/{model_type}/\" \\\n --data_path {DATA_DIR} \\\n --output_dir \"{DOCKER_TEST_DIR}/output_qwen\" \\\n --num_train_epochs 1 \\\n --per_device_train_batch_size 1 \\\n --per_device_eval_batch_size 1 \\\n --gradient_accumulation_steps 2 \\\n --evaluation_strategy \"no\" \\\n --save_strategy \"steps\" \\\n --save_steps 1000 \\\n --save_total_limit 10 \\\n --learning_rate 1e-5 \\\n --weight_decay 0.1 \\\n --adam_beta2 0.95 \\\n --warmup_ratio 0.01 \\\n --lr_scheduler_type \"cosine\" \\\n --logging_steps 1 \\\n --report_to \"none\" \\\n --model_max_length 512\"\"\"\n if deepspeed:\n cmd += f\" --deepspeed {deepspeed}\"\n if train_type == \"lora\":\n cmd += \" --use_lora\"\n elif train_type == \"qlora\":\n cmd += \" --use_lora --q_lora\"\n # for SM < 80\n if (\n (not is_ampere)\n and train_type == \"lora\"\n and (deepspeed and \"zero2\" in deepspeed)\n and is_chat == \"base\"\n ):\n cmd += \" --fp16 True\"\n snapshot_download(model_type, cache_dir=\".\", revision=\"master\")\n run_in_subprocess(cmd_docker + f'\"{cmd}\"')\n if train_type == \"full\":\n assert os.path.exists(\"output_qwen/config.json\")\n else:\n assert os.path.exists(\"output_qwen/adapter_config.json\")\n shutil.rmtree(\"output_qwen\")\n" }, { "path": "recipes/tests/test_inference/test_inference_api.py", "content": "import os\nimport sys\nimport time\nimport pytest\nimport subprocess\nimport torch\nfrom modelscope.hub.snapshot_download import snapshot_download\n\nsys.path.append(os.path.dirname(__file__) + \"/..\")\nfrom utils import run_in_subprocess, simple_openai_api, TelnetPort\nfrom ut_config import (\n MODEL_TYPE,\n DOCKER_VERSION_CU114,\n DOCKER_VERSION_CU117,\n DOCKER_VERSION_CU121,\n DOCKER_MOUNT_DIR,\n DOCKER_TEST_DIR,\n)\n\n\n# use_cpu=True,use_int=False RuntimeError: \"addmm_impl_cpu_\" not implemented for 'Half'\n# use_cpu=True,use_int4=True ValueError: Found modules on cpu/disk. Using Exllama or Exllamav2 backend requires all the modules to be on GPU.You can deactivate exllama backend by setting `disable_exllama=True` in the quantization config object\n@pytest.mark.parametrize(\n \"docker_version,use_cpu,use_int4\",\n [\n (DOCKER_VERSION_CU114, False, False),\n (DOCKER_VERSION_CU114, False, True),\n (DOCKER_VERSION_CU117, False, False),\n (DOCKER_VERSION_CU117, False, True),\n (DOCKER_VERSION_CU121, False, False),\n (DOCKER_VERSION_CU121, False, True),\n ],\n)\ndef test_inference_api(docker_version, use_cpu, use_int4):\n container_name = \"test_inference_api\"\n model_type = f\"{MODEL_TYPE}-Chat-Int4\" if use_int4 else f\"{MODEL_TYPE}-Chat\"\n cmd_docker = f'docker run --gpus all --ipc=host --network=host --rm --name=\"{container_name}\" -p 8000:8000 -v {os.getcwd()}/../../../Qwen:{DOCKER_MOUNT_DIR} {docker_version} /bin/bash -c '\n cmd = \"\"\n # for GPUs SM < 80\n is_ampere = torch.cuda.get_device_capability()[0] >= 8\n if not is_ampere:\n cmd += f\"pip uninstall -y flash-attn && \"\n\n cmd += f\"\"\"python {DOCKER_MOUNT_DIR}/openai_api.py -c {DOCKER_TEST_DIR}/{model_type}\"\"\"\n\n if use_cpu:\n cmd += \" --cpu-only\"\n\n snapshot_download(model_type, cache_dir=\".\", revision=\"master\")\n # start model server\n print(cmd_docker + f'\"{cmd}\"')\n run_in_subprocess(\n f'docker rm -f {container_name} 2>/dev/null || echo \"The container does not exist.\"'\n )\n run_in_subprocess(\"nohup \" + cmd_docker + f'\"{cmd}\"' + \" > tmp.log 2>&1 &\")\n\n while not TelnetPort(\"localhost\", 8000):\n print(\"Wait for the model service start.\")\n time.sleep(0.5)\n\n if (\n subprocess.run(\n f\"docker inspect {container_name}\",\n shell=True,\n stdout=subprocess.DEVNULL,\n ).returncode\n != 0\n ):\n break\n try:\n # while load int4 model such as Qwen-1_8B-Chat-Int4, the model name is Qwen-1_8B-Chat\n simple_openai_api(f\"{MODEL_TYPE}-Chat\".split(\"/\")[-1])\n except Exception as e:\n time.sleep(1)\n with open(\"tmp.log\") as f:\n raise Exception(f\"{e} \\n {f.read()}\")\n\n run_in_subprocess(f\"docker rm -f {container_name}\")\n" }, { "path": "recipes/tests/test_inference/test_inference_vllm_fschat.py", "content": "import os\nimport sys\nimport time\nimport pytest\nimport subprocess\nimport torch\nfrom modelscope.hub.snapshot_download import snapshot_download\n\nsys.path.append(os.path.dirname(__file__) + \"/..\")\nfrom utils import run_in_subprocess, simple_openai_api, TelnetPort\nfrom ut_config import (\n MODEL_TYPE,\n DOCKER_VERSION_CU121,\n DOCKER_MOUNT_DIR,\n DOCKER_TEST_DIR,\n)\n\n\n@pytest.mark.parametrize(\n \"num_gpus,use_int4\",\n [\n (1, False),\n (1, True),\n (2, False),\n # ValueError: The input size is not aligned with the quantized weight shape. This can be caused by too large tensor parallel size.\n # (2, True)\n ],\n)\ndef test_inference_vllm_fschat(num_gpus, use_int4):\n model_type = f\"{MODEL_TYPE}-Chat-Int4\" if use_int4 else f\"{MODEL_TYPE}-Chat\"\n container_name = \"test_inference_vllm_fschat\"\n cmd_docker = f'docker run --gpus all --ipc=host --network=host --rm --name=\"{container_name}\" -p 8000:8000 -v {os.getcwd()}/../../../Qwen:{DOCKER_MOUNT_DIR} {DOCKER_VERSION_CU121} /bin/bash -c '\n cmd = \"\"\n\n cmd += f\"\"\"nohup python -m fastchat.serve.controller > /dev/null 2>&1 \\\n & python -m fastchat.serve.openai_api_server --host localhost --port 8000 > /dev/null 2>&1 \\\n & python -m fastchat.serve.vllm_worker --model-path {DOCKER_TEST_DIR}/{model_type} --tensor-parallel-size {num_gpus} --trust-remote-code\"\"\"\n\n # for GPUS SM < 80 and use_int==True\n is_ampere = torch.cuda.get_device_capability()[0] >= 8\n if not is_ampere or use_int4:\n cmd += \" --dtype half\"\n\n snapshot_download(model_type, cache_dir=\".\", revision=\"master\")\n # start model server\n run_in_subprocess(\n f'docker rm -f {container_name} 2>/dev/null || echo \"The container does not exist.\"'\n )\n print(cmd_docker + f'\"{cmd}\"')\n run_in_subprocess(\"nohup \" + cmd_docker + f'\"{cmd}\"' + \" > tmp.log 2>&1 &\")\n\n while not TelnetPort(\"localhost\", 21002):\n print(\"Wait for the model service start.\")\n time.sleep(0.5)\n\n if (\n subprocess.run(\n f\"docker inspect {container_name}\",\n shell=True,\n stdout=subprocess.DEVNULL,\n ).returncode\n != 0\n ):\n break\n\n try:\n simple_openai_api(model_type.split(\"/\")[-1])\n except Exception as e:\n time.sleep(1)\n with open(\"tmp.log\") as f:\n raise Exception(f\"{e} \\n {f.read()}\")\n\n run_in_subprocess(f\"docker rm -f {container_name}\")\n" }, { "path": "recipes/tests/ut_config.py", "content": "import os\n\n# common\nMODEL_TYPE = \"Qwen/Qwen-1_8B\"\nDOCKER_VERSION_CU114 = \"qwenllm/qwen:cu114\"\nDOCKER_VERSION_CU117 = \"qwenllm/qwen:cu117\"\nDOCKER_VERSION_CU121 = \"qwenllm/qwen:cu121\"\nDOCKER_MOUNT_DIR = \"/qwen-recipes\"\nDOCKER_TEST_DIR = os.path.join(DOCKER_MOUNT_DIR, \"recipes/tests\")\n\n# finetune\nDATA_DIR = os.path.join(DOCKER_MOUNT_DIR, \"recipes/tests/assets/test_sampled_qwen.json\")\nDS_CONFIG_ZERO2_DIR = os.path.join(\n DOCKER_MOUNT_DIR, \"finetune/ds_config_zero2.json\"\n)\nDS_CONFIG_ZERO3_DIR = os.path.join(\n DOCKER_MOUNT_DIR, \"finetune/ds_config_zero3.json\"\n)\n" }, { "path": "recipes/tests/utils.py", "content": "import logging\nimport subprocess\nimport socket\nimport openai\n\n\ndef run_in_subprocess(cmd):\n try:\n with subprocess.Popen(\n cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE\n ) as return_info:\n while True:\n next_line = return_info.stdout.readline()\n return_line = next_line.decode(\"utf-8\", \"ignore\").strip()\n if return_line == \"\" and return_info.poll() != None:\n break\n if return_line != \"\":\n logging.info(return_line)\n\n err_lines = \"\"\n while True:\n next_line = return_info.stderr.readline()\n return_line = next_line.decode(\"utf-8\", \"ignore\").strip()\n if return_line == \"\" and return_info.poll() != None:\n break\n if return_line != \"\":\n logging.info(return_line)\n err_lines += return_line + \"\\n\"\n\n return_code = return_info.wait()\n if return_code:\n raise RuntimeError(err_lines)\n except Exception as e:\n raise e\n\n\ndef simple_openai_api(model):\n openai.api_base = \"http://localhost:8000/v1\"\n openai.api_key = \"none\"\n\n # create a request not activating streaming response\n response = openai.ChatCompletion.create(\n model=model,\n messages=[{\"role\": \"user\", \"content\": \"你好\"}],\n stream=False,\n stop=[], # You can add custom stop words here, e.g., stop=[\"Observation:\"] for ReAct prompting.\n )\n print(response.choices[0].message.content)\n\n\ndef TelnetPort(server_ip, port):\n sk = socket.socket(socket.AF_INET, socket.SOCK_STREAM)\n sk.settimeout(1)\n connect_flag = False\n try:\n sk.connect((server_ip, port))\n connect_flag = True\n except Exception:\n connect_flag = False\n sk.close()\n return connect_flag\n" }, { "path": "requirements.txt", "content": "transformers>=4.32.0,<4.38.0\naccelerate\ntiktoken\neinops\ntransformers_stream_generator==0.0.4\nscipy\n" }, { "path": "requirements_web_demo.txt", "content": "gradio<3.42\nmdtex2html\n" }, { "path": "run_gptq.py", "content": "import argparse\r\nimport json\r\nfrom typing import Dict\r\nimport logging\r\n\r\nimport torch\r\nimport transformers\r\nfrom transformers import AutoTokenizer\r\nfrom transformers.trainer_pt_utils import LabelSmoother\r\nfrom auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig\r\nIGNORE_TOKEN_ID = LabelSmoother.ignore_index\r\n\r\ndef preprocess(\r\n sources,\r\n tokenizer: transformers.PreTrainedTokenizer,\r\n max_len: int,\r\n system_message: str = \"You are a helpful assistant.\"\r\n) -> Dict:\r\n roles = {\"user\": \"<|im_start|>user\", \"assistant\": \"<|im_start|>assistant\"}\r\n\r\n im_start = tokenizer.im_start_id\r\n im_end = tokenizer.im_end_id\r\n nl_tokens = tokenizer('\\n').input_ids\r\n _system = tokenizer('system').input_ids + nl_tokens\r\n _user = tokenizer('user').input_ids + nl_tokens\r\n _assistant = tokenizer('assistant').input_ids + nl_tokens\r\n\r\n # Apply prompt templates\r\n data = []\r\n # input_ids, targets = [], []\r\n for i, source in enumerate(sources):\r\n source = source[\"conversations\"]\r\n if roles[source[0][\"from\"]] != roles[\"user\"]:\r\n source = source[1:]\r\n\r\n input_id, target = [], []\r\n system = [im_start] + _system + tokenizer(system_message).input_ids + [im_end] + nl_tokens\r\n input_id += system\r\n target += [im_start] + [IGNORE_TOKEN_ID] * (len(system)-3) + [im_end] + nl_tokens\r\n assert len(input_id) == len(target)\r\n for j, sentence in enumerate(source):\r\n role = roles[sentence[\"from\"]]\r\n _input_id = tokenizer(role).input_ids + nl_tokens + \\\r\n tokenizer(sentence[\"value\"]).input_ids + [im_end] + nl_tokens\r\n input_id += _input_id\r\n if role == '<|im_start|>user':\r\n _target = [im_start] + [IGNORE_TOKEN_ID] * (len(_input_id)-3) + [im_end] + nl_tokens\r\n elif role == '<|im_start|>assistant':\r\n _target = [im_start] + [IGNORE_TOKEN_ID] * len(tokenizer(role).input_ids) + \\\r\n _input_id[len(tokenizer(role).input_ids)+1:-2] + [im_end] + nl_tokens\r\n else:\r\n raise NotImplementedError\r\n target += _target\r\n assert len(input_id) == len(target)\r\n input_id = torch.tensor(input_id[:max_len], dtype=torch.int)\r\n target = torch.tensor(target[:max_len], dtype=torch.int)\r\n data.append(dict(input_ids=input_id, attention_mask=input_id.ne(tokenizer.pad_token_id)))\r\n\r\n return data\r\n\r\n\r\nif __name__ == \"__main__\":\r\n parser = argparse.ArgumentParser(\"Model Quantization using AutoGPTQ\")\r\n parser.add_argument(\"--model_name_or_path\", type=str, help=\"model path\")\r\n parser.add_argument(\"--data_path\", type=str, help=\"calibration data path\")\r\n parser.add_argument(\"--out_path\", type=str, help=\"output path of the quantized model\")\r\n parser.add_argument(\"--max_len\", type=int, default=8192, help=\"max length of calibration data\")\r\n parser.add_argument(\"--bits\", type=int, default=4, help=\"the bits of quantized model. 4 indicates int4 models.\")\r\n parser.add_argument(\"--group-size\", type=int, default=128, help=\"the group size of quantized model\")\r\n args = parser.parse_args()\r\n \r\n quantize_config = BaseQuantizeConfig(\r\n bits=args.bits,\r\n group_size=args.group_size,\r\n damp_percent=0.01,\r\n desc_act=False, # set to False can significantly speed up inference but the perplexity may slightly bad\r\n static_groups=False,\r\n sym=True,\r\n true_sequential=True,\r\n model_name_or_path=None,\r\n model_file_base_name=\"model\"\r\n )\r\n\r\n tokenizer = AutoTokenizer.from_pretrained(args.model_name_or_path, trust_remote_code=True)\r\n tokenizer.pad_token_id = tokenizer.eod_id\r\n data = preprocess(json.load(open(args.data_path)), tokenizer, args.max_len)\r\n\r\n model = AutoGPTQForCausalLM.from_pretrained(args.model_name_or_path, quantize_config, device_map=\"auto\", trust_remote_code=True)\r\n\r\n logging.basicConfig(\r\n format=\"%(asctime)s %(levelname)s [%(name)s] %(message)s\", level=logging.INFO, datefmt=\"%Y-%m-%d %H:%M:%S\"\r\n )\r\n model.quantize(data, cache_examples_on_gpu=False)\r\n\r\n model.save_quantized(args.out_path, use_safetensors=True)\r\n tokenizer.save_pretrained(args.out_path)\r\n" }, { "path": "tech_memo.md", "content": "# Introducing Qwen-7B: Open foundation and human-aligned models (of the state-of-the-arts)\n\nLarge language models have recently attracted an extremely large amount of\nattention.\nThe boom of [ChatGPT](https://openai.com/blog/chatgpt) rocketed the development of artificial general intelligence and indicates that large language models compress world knowledge into neural networks, and the alignment to human cognition can lead to powerful conversational agents that can provide assistance by interacting with human users.\nNow, the latest version of ChatGPT based on [GPT-4](https://arxiv.org/abs/2303.08774) demonstrates tremendously exciting performance across unlimited capabilities, say, language understanding, logical reasoning, planning, etc., and its incorporation with external tools, including tools and models, releases the power of an agent capable of understanding instructions, executing code, using tools, and so on, to reach the objectives set up by human users.\n\nThese significant progresses indicate the importance of large language models as _the foundation of AI services_.\n\nWe are happy to release the 7B-parameter models of our large pretrained model series Qwen (abbr. Tongyi Qianwen), Qwen-7B.\nThis release includes model weights and codes for pretrained and human-aligned language models of 7B parameters:\n\n- `Qwen-7B` is the pretrained language model, and `Qwen-7B-Chat` is fine-tuned to align with human intent.\n- `Qwen-7B` is pretrained on over 2.2 trillion tokens with a context length of 2048. On the series of benchmarks we tested, Qwen-7B generally performs better than existing open models of similar scales and appears to be on par with some of the larger models.\n- `Qwen-7B-Chat` is fine-tuned on curated data, including not only task-oriented data but also specific security- and service-oriented data, which seems insufficient in existing open models.\n- Example codes for fine-tuning, evaluation, and inference are included. There are also guides on long-context and tool use in inference.\n\n**Goal of release**:\nWe believe that while the recent waves of releases of LLMs may have deepened our understanding of model behaviors under standard regimes, it is yet to be revealed how the accompanied techniques of nowadays LLMs, such as 1) quantization and fine-tuning after quantization, 2) training-free long-context inference, and 3) fine-tuning with service-oriented data, including search and tool uses, affect the models as a whole.\nThe open release of Qwen-7B marks our first step towards fully understanding the real-world application of such techniques.\nIt is our hope that it will enable the community to analyze and continue to improve the safety of those models, striving to establish responsible development and deployment of LLMs.\n\n> **Disclaimer**:\n> We must note that even though the weights and codes are released in an open manner and commercial use is not prohibited, similar to other pretrained language models, Qwen-7B comes with potential risks influenced by complex factors, including but not limited to over-diversified, inaccurate, or misleading generation.\n> Developers and stakeholders should perform their own red teaming and provide related security measures before deployment, and they must abide by and comply with local governance and regulations.\n> In no event shall the authors be held liable for any claim, damages, or other liability arising from the use of the released weights or codes.\n\nThe remainder of this document describes our pretraining and fine-tuning methodology.\n\n## Pretraining\n\nQwen-7B is a transformer-based decoder-only language model with an architecture similar to the [LLaMA](https://github.com/facebookresearch/llama) series of models.\nIt is pretrained on over 2.2 trillion tokens with 2048 context length from publicly available data, covering general and professional fields with a focus on the English and Chinese languages.\n\n### Data\n\n**Pretraining data**:\nOur training data includes a mix of data from publicly available sources, consisting mainly of web documents and code files.\nBesides, the data are multilingual, with most of them in English and Chinese.\nWe made an effort and employed an ensemble of models to exclude data of low quality or deemed unfit for pretraining, such as NSFW content.\nFor math reasoning, we include RFT data from [gsm8k-ScRel](https://github.com/OFA-Sys/gsm8k-ScRel).\nThe final data underwent global fuzzy deduplication.\nThe mix of pretraining corpora has been optimized through numerous ablation experiments.\n\n**Tokenization**:\nCompared to the current mainstream open models based on Chinese and English vocabularies, we use a vocabulary of 151,851 tokens.\nIt first considers efficient encoding of Chinese, English, and code data, and is also more friendly to multilingual languages, enabling users to directly enhance the capability of some languages without expanding the vocabulary.\nIt segments numbers by single digits and calls the [tiktoken](https://github.com/openai/tiktoken) tokenizer library for efficient tokenization.\nAfter tokenization, the data amounts to over 2.2 trillion tokens.\n\n<figure>\n <img src=\"assets/tokenizer.png\"\n alt=\"Tokenization efficiency\"\n width=\"1200px\">\n <figcaption>We randomly selected 1 million document corpora of each language to test and compare the encoding compression rates of different models (with XLM-R, which supports 100 languages, as the base value 1, not shown in the figure). As can be seen, while ensuring the efficient decoding of Chinese, English, and code, Qwen-7B also achieves a high compression rate for many other languages (such as th, he, ar, ko, vi, ja, tr, id, pl, ru, nl, pt, it, de, es, fr etc.), equipping the model with strong scalability as well as high training and inference efficiency in these languages.</figcaption>\n</figure>\n\n### Model\n\n**Model architecture**:\nQwen-7B is built with architecture similar to LLaMA.\nThe following are the main differences from the standard transformer: 1) using untied embedding, 2) using rotary positional embedding, 3) no biases except for QKV in attention, 4) RMSNorm instead of LayerNorm, 5) SwiGLU instead of ReLU, and 6) adopting flash attention to accelerate training.\nThe model has 32 layers, the embedding dimension is 4096, and the number of attention heads is 32.\n\n**Training details**:\nThe model is trained using the AdamW optimizer, with $\\beta_1=0.9, \\beta_2=0.95, \\epsilon=10^{-6}$.\nThe sequence length is 2048, and the batch size is 2048, which means each optimization step accumulates over 4 million tokens.\nWe use a cosine learning rate schedule, with a warm-up of 2000 steps, a peak learning rate of $3 \\times 10^{-4}$, and a minimum learning rate of 10% of the peak learning rate.\nWe use a weight decay of 0.1 and gradient clipping of 1.0.\nThe training adopts mixed precision training with `bfloat16`.\n\n\n### Evaluation\n\nWe report results of Qwen-7B on standard benchmarks.\n\n#### World knowledge\n\n[C-Eval](https://arxiv.org/abs/2305.08322) is a common evaluation benchmark for testing the common-sense capability of pretrained models in Chinese. It covers 52 subjects in four major directions: humanities, social sciences, STEM, and other specialties. According to standard practice, we use the development set samples as the source of few-shot prompts to evaluate the 5-shot validation set and test set accuracy of the Qwen-7B pretrained model.\n\nThe accuracy comparison of the Qwen-7B model and other models on the C-Eval validation set is as follows:\n\n| Model | Average |\n| :---------- | -------: |\n| Alpaca-7B | 28.9 |\n| Vicuna-7B | 31.2 |\n| ChatGLM-6B | 37.1 |\n| Baichuan-7B | 42.7 |\n| ChatGLM2-6B | 50.9 |\n| InternLM-7B | 53.4 |\n| ChatGPT | 53.5 |\n| Claude-v1.3 | 55.5 |\n| **Qwen-7B** | **60.8** |\n\nThe performance comparison of the Qwen-7B pretrained model and other models on the C-Eval test set is shown in the following table:\n\n| Model | Avg. | Avg. (Hard) | STEM | Social Sciences | Humanities | Others |\n| :---------------------- | -------- | ----------: | ---: | --------------: | ---------: | -----: |\n| ChatGLM-6B | 38.9 | 29.2 | 33.3 | 48.3 | 41.3 | 38.0 |\n| Chinese-Alpaca-Plus-13B | 41.5 | 30.5 | 36.6 | 49.7 | 43.1 | 41.2 |\n| Baichuan-7B | 42.8 | 31.5 | 38.2 | 52.0 | 46.2 | 39.3 |\n| WestlakeLM-19B | 44.6 | 34.9 | 41.6 | 51.0 | 44.3 | 44.5 |\n| AndesLM-13B | 46.0 | 29.7 | 38.1 | 61.0 | 51.0 | 41.9 |\n| BatGPT-15B-sirius | 47.0 | 31.9 | 42.7 | 57.5 | 48.6 | 43.6 |\n| ChatGLM2-6B | 51.7 | 37.1 | 48.6 | 60.5 | 51.3 | 49.8 |\n| InternLM-7B | 52.8 | 37.1 | 48.0 | 67.4 | 55.4 | 45.8 |\n| Baichuan-13B | 53.6 | 36.7 | 47.0 | 66.8 | 57.3 | 49.8 |\n| Claude-v1.3 | 54.2 | 39.0 | 51.9 | 61.7 | 52.1 | 53.7 |\n| ChatGPT | 54.4 | 41.4 | 52.9 | 61.8 | 50.9 | 53.6 |\n| **Qwen-7B** | **59.6** | 41.0 | 52.8 | 74.1 | 63.1 | 55.2 |\n\nAs can be seen, Qwen-7B achieves the best performance out of all existing models of similar scale and even surpasses larger-scale models.\n\nMMLU is currently one of the most recognized benchmarks for evaluating English comprehension abilities, covering 57 subtasks across different academic fields and difficulty levels. The MMLU 5-shot accuracy performance of the Qwen-7B is shown in the following table:\n\n| Model | Average | STEM | Social Sciences | Humanities | Others |\n| :----------- | -------: | ---: | --------------: | ---------: | -----: |\n| LLaMA-7B | 35.1 | 30.5 | 38.3 | 34.0 | 38.1 |\n| Baichuan-7B | 42.3 | 35.6 | 48.9 | 38.4 | 48.1 |\n| LLaMA2-7B | 45.3 | 36.4 | 51.2 | 42.9 | 52.2 |\n| LLaMA-13B | 46.9 | 35.8 | 53.8 | 45.0 | 53.3 |\n| ChatGLM2-6B | 47.9 | 41.2 | 54.4 | 43.7 | 54.5 |\n| InternLM-7B | 51.0 | - | - | - | - |\n| Baichuan-13B | 51.6 | 41.6 | 60.9 | 47.4 | 58.5 |\n| LLaMA2-13B | 54.8 | 44.1 | 62.6 | 52.8 | 61.1 |\n| ChatGLM2-12B | 56.2 | 48.2 | 65.1 | 52.6 | 60.9 |\n| **Qwen-7B** | **56.7** | 47.6 | 65.9 | 51.5 | 64.7 |\n\nIn terms of English, Qwen-7B also surpasses other similar open pretrained models, and is competitive when compared to larger versions of other models.\n\n#### Coding\n\nWe compared the code capabilities of pretrained models on [HumanEval](https://github.com/openai/human-eval), and the results are as follows:\n\n| Model | Pass@1 |\n| :----------- | -------: |\n| Baichuan-7B | 9.2 |\n| ChatGLM2-6B | 9.2 |\n| InternLM-7B | 10.4 |\n| LLaMA-7B | 10.5 |\n| LLaMA2-7B | 12.8 |\n| Baichuan-13B | 12.8 |\n| LLaMA-13B | 15.8 |\n| MPT-7B | 18.3 |\n| LLaMA2-13B | 18.3 |\n| **Qwen-7B** | **24.4** |\n\n#### Math\n\nWe compared the math capabilities of pretrained models on [GSM8K](https://github.com/openai/grade-school-math) (8-shot), and the results are as follows:\n\n| Model | Accuracy |\n| :----------- | -------: |\n| MPT-7B | 6.8 |\n| Falcon-7B | 6.8 |\n| Baichuan-7B | 9.7 |\n| LLaMA-7B | 11.0 |\n| LLaMA2-7B | 14.6 |\n| LLaMA-13B | 17.8 |\n| Baichuan-13B | 26.6 |\n| LLaMA2-13B | 28.7 |\n| InternLM-7B | 31.2 |\n| ChatGLM2-6B | 32.4 |\n| ChatGLM2-12B | 40.9 |\n| **Qwen-7B** | **51.6** |\n\n#### Natural language processing\n\nWe compared the translation capabilities of pre-trained models on WMT22 zh-en and en-zh (5-shot BLEU), and the results are as follows:\n\n| Model | Average | zh-en | en-zh |\n| :---------- | -------: | -------: | -------: |\n| InternLM-7B | 11.8 | 9.0 | 14.5 |\n| LLaMA-7B | 12.7 | 16.7 | 8.7 |\n| LLaMA-13B | 15.8 | 19.5 | 12.0 |\n| LLaMA2-7B | 19.9 | 21.9 | 17.9 |\n| Bloom-7B | 20.3 | 19.1 | 21.4 |\n| LLaMA2-13B | 23.3 | 22.4 | 24.2 |\n| PolyLM-13B | 23.6 | 20.2 | 27.0 |\n| Baichuan-7B | 24.6 | 22.6 | 26.6 |\n| **Qwen-7B** | **27.5** | **24.3** | **30.6** |\n\n#### Long-context inference\n\nWe include support for training-free long-context inference based on ntk-aware interpolation, LogN attention scaling, and local window attention.\nThe context can be expanded from 2048 to over 8192.\nThe following are the test results on arXiv in terms of perplexity (PPL).\n\n<table>\n\t<tr>\n <th rowspan=\"2\">Model</th><th colspan=\"5\" align=\"center\">Sequence Length</th>\n </tr>\n <tr>\n <th align=\"center\">1024</th><th align=\"center\">2048</th><th align=\"center\">4096</th><th align=\"center\">8192</th><th align=\"center\">16384</th>\n </tr>\n <tr>\n <td>Qwen-7B</td><td align=\"right\"><b>4.23</b></td><td align=\"right\"><b>3.78</b></td><td align=\"right\">39.35</td><td align=\"right\">469.81</td><td align=\"right\">2645.09</td>\n </tr>\n <tr>\n <td>+ dynamic_ntk</td><td align=\"right\"><b>4.23</b></td><td align=\"right\"><b>3.78</b></td><td align=\"right\">3.59</td><td align=\"right\">3.66</td><td align=\"right\">5.71</td>\n </tr>\n <tr>\n <td>+ dynamic_ntk + logn</td><td align=\"right\"><b>4.23</b></td><td align=\"right\"><b>3.78</b></td><td align=\"right\"><b>3.58</b></td><td align=\"right\">3.56</td><td align=\"right\">4.62</td>\n </tr>\n <tr>\n <td>+ dynamic_ntk + logn + local_attn</td><td align=\"right\"><b>4.23</b></td><td align=\"right\"><b>3.78</b></td><td align=\"right\"><b>3.58</b></td><td align=\"right\"><b>3.49</b></td><td align=\"right\"><b>4.32</b></td>\n </tr>\n</table>\n\n## Fine-tuning\n\n`Qwen-7B-Chat` embodies our practice in alignment with human intents, ensuring internalized safety, and building intelligent agents for services.\n\n### Data\n\n**Alignment data**:\nThe data includes common instruction-style conversations, and security- and service-oriented data, which involves substantial annotation efforts.\nInstruction data covers broad abilities, such as writing, question answering, brainstorming and planning, content understanding, summarization, natural language processing, and coding.\nSecurity data tries to prevent the model from generating harmful and inappropriate content.\nService data tries to enhance the model with specific conversation patterns that can be parsed to invoke and incorporate external systems.\n\n**Data formatting**:\nSince the data consists of conversation turns, we arrange them into texts using the [ChatML](https://github.com/openai/openai-python/blob/main/chatml.md) format, which is a meta language that can describe both the metadata (e.g., roles) and the content of a turn.\nCurrently, existing roles include system, user, and assistant.\n\n### Model\n\n**Training details**:\nThe causal language modeling objective is used to fine-tune the model, except for the tokens in the content of user's turns.\nThe model is trained using the AdamW optimizer, with $\\beta_1=0.9, \\beta_2=0.95, \\epsilon=10^{-6}$.\nThe sequence length is limited to 2048, and the batch size is 128.\nThe model is trained for 4000 steps, and over the first 1430 steps, the learning rate is warmed up to $1 \\times 10^{-5}$.\nWe use weight decay of 0.1, dropout of 0.1, and gradient clipping of 1.0.\n\n### Evaluation\n\nEvaluation of human-aligned models is non-trivial and often non-standardized, since such models often target specific applications.\nWe evaluate Qwen-7B-Chat from multiple perspectives.\n\n#### World knowledge\n\nAs fine-tuning uses a much smaller dataset than pretraining and humans' understanding of world knowledge may be limited, we also evaluate the world knowledge of Qwen-7B-Chat using C-Eval and MMLU in a zero-shot and generative manner.\n\nWe demonstrate the zero-shot accuracy of Qwen-7B-Chat on the C-Eval validation set.\n\n| Model | Avg. Acc. |\n| :---------------------- | --------: |\n| LLaMA2-7B-Chat | 31.9 |\n| LLaMA2-13B-Chat | 40.6 |\n| Chinese-Alpaca-2-7B | 41.3 |\n| Chinese-Alpaca-Plus-13B | 43.3 |\n| Baichuan-13B-Chat | 50.4 |\n| ChatGLM2-6B-Chat | 50.7 |\n| InternLM-7B-Chat | 53.2 |\n| **Qwen-7B-Chat** | **54.2** |\n\nThe zero-shot accuracy of Qwen-7B-Chat on C-Eval testing set is provided below\n\n| Model | Avg. | STEM | Social Sciences | Humanities | Others |\n| :---------------------- | -------: | ---: | --------------: | ---------: | -----: |\n| Chinese-Alpaca-Plus-13B | 41.5 | 36.6 | 49.7 | 43.1 | 41.2 |\n| Chinese-Alpaca-2-7B | 40.3 | - | - | - | - |\n| ChatGLM2-6B-Chat | 50.1 | 46.4 | 60.4 | 50.6 | 46.9 |\n| Baichuan-13B-Chat | 51.5 | 43.7 | 64.6 | 56.2 | 49.2 |\n| **Qwen-7B-Chat** | **54.6** | 47.8 | 67.6 | 59.3 | 50.6 |\n\nCompared with other models with comparable model sizes, the human-aligned Qwen-7B-Chat performs well in C-Eval accuracy.\n\nThe zero-shot accuracy of Qwen-7B-Chat on MMLU is provided below.\nThe performance of Qwen-7B-Chat is still on top among other human-aligned models with comparable size.\n\n| Model | Avg. Acc. |\n| :---------------- | --------: |\n| ChatGLM2-6B-Chat | 45.5 |\n| LLaMA2-7B-Chat | 47.0 |\n| InternLM-7B-Chat | 50.8 |\n| Baichuan-13B-Chat | 52.1 |\n| ChatGLM2-12B-Chat | 52.1 |\n| **Qwen-7B-Chat** | **53.9** |\n\n#### Coding\n\nThe zero-shot Pass@1 of Qwen-7B-Chat on [HumanEval](https://github.com/openai/human-eval) is demonstrated below\n\n| Model | Pass@1 |\n| :---------------- | -------: |\n| LLaMA2-7B-Chat | 12.2 |\n| InternLM-7B-Chat | 14.0 |\n| Baichuan-13B-Chat | 16.5 |\n| LLaMA2-13B-Chat | 18.9 |\n| **Qwen-7B-Chat** | **24.4** |\n\n#### Math\n\nThe accuracy of Qwen-7B-Chat on GSM8K is shown below\n\n| Model | Zero-shot Acc. | 4-shot Acc. |\n| :---------------- | -------------: | ----------: |\n| ChatGLM2-6B-Chat | - | 28.0 |\n| LLaMA2-7B-Chat | 20.4 | 28.2 |\n| LLaMA2-13B-Chat | 29.4 | 36.7 |\n| InternLM-7B-Chat | 32.6 | 34.5 |\n| Baichuan-13B-Chat | - | 36.3 |\n| ChatGLM2-12B-Chat | - | 38.1 |\n| **Qwen-7B-Chat** | **41.1** | **43.5** |\n\n#### Service\n\nLLMs have shown capability in coordinating multiple external systems to achieve the given instructions, which creates new opportunities in traditional online services, the most notable being web search.\n\nQwen supports calling plugins/tools/APIs through [ReAct Prompting](https://arxiv.org/abs/2210.03629).\nReAct is also one of the main approaches used by the [LangChain](https://python.langchain.com/) framework.\nFor how to write and use prompts for ReAct Prompting, please refer to [the ReAct examples](examples/react_prompt.md).\nIn our evaluation [benchmark](eval/EVALUATION.md) for assessing tool usage capabilities, Qwen's performance is as follows:\n\n| Model | Tool Selection (Acc.↑) | Tool Input (Rouge-L↑) | False Positive Error↓ |\n| :---------- | --------------------------: | -------------------------: | -------------------------: |\n| GPT-4 | 95% | **0.90** | 15.0% |\n| GPT-3.5 | 85% | 0.88 | 75.0% |\n| **Qwen-7B** | **99%** | 0.89 | **9.7%** |\n\n> The plugins that appear in the evaluation set do not appear in the training set of Qwen.\n> This benchmark evaluates the accuracy of the model in selecting the correct plugin from multiple candidate plugins, the rationality of the parameters passed into the plugin, and the false positive rate.\n> False Positive: Incorrectly invoking a plugin when it should not have been called when responding to a query.\n\nQwen also has the capability to be used as a [HuggingFace Agent](https://huggingface.co/docs/transformers/transformers_agents).\nIts performance on the benchmark provided by HuggingFace is as follows:\n\n| Model | Tool Selection↑ | Tool Used↑ | Code↑ |\n| :-------------- | -------------------: | --------------: | ---------: |\n| GPT-4 | **100.00** | **100.00** | **97.41** |\n| GPT-3.5 | 95.37 | 96.30 | 87.04 |\n| StarCoder-15.5B | 87.04 | 87.96 | 68.89 |\n| **Qwen-7B** | 90.74 | 92.59 | 74.07 |\n\n## Conclusion\n\nIn this document, we describe Qwen-7B, including a pretrained model and a human-aligned model.\nThese models have demonstrated exciting performance compared to existing open models of similar or even larger scales.\nAs part of our ongoing commitment to the concept of Model as a Service, the release also includes practical pieces such as long context inference and external system integration, which we hope would facilitate developers realizing their own ideas and concepts.\nWe believe that the open release of Qwen-7B models would further our understanding of variables and techniques introduced in realistic settings and help to drive progress in this important area together with the community.\n" }, { "path": "tokenization_note.md", "content": "# Tokenization\r\n\r\nQwen-7B uses BPE tokenization on UTF-8 bytes using the `tiktoken` package.\r\nThere are two types of tokens in Qwen-7B, i.e., the regular tokens (of type `bytes`) in BPE and the special/control tokens (of type `str`).\r\n\r\n```python\r\nfrom transformers import AutoTokenizer\r\n\r\ntokenizer = AutoTokenizer.from_pretrained('Qwen/Qwen-7B', trust_remote_code=True)\r\n```\r\n\r\n## Regular tokens\r\n\r\nThe regular tokens are BPE tokens learned from byte sequences of texts encoded using the UTF-8 encoding.\r\nWhile this allows tokenization of all texts and no unknown token exists, it may fall back to using single bytes when tokenizing uncommon texts.\r\nYou may encounter UTF-8 decoding errors and as the errors are default to `replace`, thus the replacement character (�) in incomplete generation.\r\nYou can change this behavior by passing `errors=\"ignore\"` to the `decode` function for once or to the `from_pretrained` function forever.\r\nFor more options of `errors`, please refer to [the Python documentation](https://docs.python.org/3/library/stdtypes.html#bytes.decode).\r\n\r\n```python\r\n>>> tokenizer.decode([51461])\r\n' �'\r\n\r\n>>> tokenizer.convert_ids_to_tokens([51461])\r\n[b' \\xe6\\xa0']\r\n\r\n>>> b' \\xe6\\xa0'.decode(\"utf-8\", errors='replace')\r\n' �'\r\n\r\n>>> tokenizer.decode([51461, 117])\r\n' 根'\r\n\r\n>>> tokenizer.convert_ids_to_tokens([51461, 117])\r\n[b' \\xe6\\xa0', b'\\xb9']\r\n\r\n>>> b' \\xe6\\xa0\\xb9'.decode(\"utf-8\", errors='replace')\r\n' 根'\r\n```\r\n\r\nThe mapping from regular tokens (in `bytes`) to its ID can be retrieved from `tokenizer.get_vocab()`.\r\nWe do not support or recommended adding regular tokens to the vocabulary.\r\n\r\n## Special tokens\r\n\r\nThe special tokens signify special functions to the model, e.g., reaching the end of a document.\r\nIn theory, they do not exist in the input texts and only appear after the input texts are processed.\r\nTheir surface forms, e.g., `<|endoftext|>` for the end of a document, are only meant for ease of reference.\r\nCurrently, used special tokens are `<|endoftext|>` in Qwen-7B, and `<|endoftext|>`, `<|im_start|>`, and `<|im_end|>` in Qwen-7B-Chat, which means they have determined meanings to the corresponding model, and should not be used otherwise.\r\nFor other purposes, we keep extra special tokens from `<|extra_0|>` to `<|extra_204|>`, and you can use them as you wish.\r\nThe mapping from surface forms of the special tokens (in `str`) to its ID can be retrieved from `tokenizer.special_tokens`.\r\n\r\nThe concepts of `bos`, `eos`, `unk`, `pad`, `mask`, `sep` and such are not appliable to our pretrained models (Qwen-7B and Qwen-7B-Chat).\r\nThe `pad` token, however, is a different story, as in theory, the model never sees or computes this token, so you may use any known token.\r\nBut to be safe, we limit the value of special tokens specified in the initialization of the tokenizer to the known special tokens.\r\nYou may specify special tokens in fine-tuning or in any other frameworks that necessitate them like this\r\n\r\n```python\r\nfrom transformers import AutoTokenizer\r\n\r\ntokenizer = AutoTokenizer.from_pretrained('Qwen/Qwen-7B', trust_remote_code=True, pad_token='<|endoftext|>')\r\n```\r\n\r\n> WARNING: For our pretrained models, setting `bos`, `eos`, `unk`, and such makes no sense.\r\n> Unknown behavior may be introduced if you set them without fine-tuning that designates their meanings to the model.\r\n> Especially, you should not use `<|endoftext|>` as `eos`, unless you are sure that the end of a sentence and the end of a document, which may contain many sentences, are the same in your scenario.\r\n\r\n## Injection attack prevention\r\n\r\nAs special tokens are different from regular tokens, what will happen if the surface forms of a control token appear in the input texts?\r\nFor example, note that a piece of text like this\r\n\r\n```\r\nprint(\"<|endoftext|>\")\r\n```\r\n\r\nshould be tokenized as\r\n\r\n```\r\nids:[1350, 9639, 91, 8691, 723, 427, 91, 82598]\r\ntokens: [b'print', b'(\"<', b'|', b'endo', b'ft', b'ext', b'|', b'>\")']\r\n```\r\n\r\nnot\r\n\r\n```\r\nids: [1350, 445, 151643, 899]\r\ntokens: [b'print', b'(\"', '<|endoftext|>', b'\")']\r\n```\r\n\r\nOur default used to be the correct one, that is, treating the surface forms of special tokens just like regular texts, and special tokens should be taken cared of by developers after tokenization of the texts.\r\nHowever, this conflicts with (albeit unsafe) practice in the community, and adds another step for developers to reuse their wheels.\r\n\r\nThe default behavior has been changed to parse the surface forms of all the known special tokens as special tokens.\r\nTo enable injection prevention, pass `allowed_special=set()` to the calls of the tokenizer:\r\n\r\n```python\r\n>>> tokenizer('print(\"<|endoftext|>\")', allowed_special=set())\r\n{'input_ids': [1350, 9639, 91, 8691, 723, 427, 91, 82598], 'token_type_ids': [0, 0, 0, 0, 0, 0, 0, 0], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1]}\r\n```\r\n\r\nYou can control the behavior in a fine-grained manner by passing a set of `str` as `allowed_special`\r\n\r\n```python\r\n>>> tokenizer('print(\"<|extra_0|>\")<|endoftext|>', allowed_special={'<|endoftext|>'})\r\n{'input_ids': [1350, 9639, 91, 15460, 62, 15, 91, 82598, 151643], 'token_type_ids': [0, 0, 0, 0, 0, 0, 0, 0, 0], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1]}\r\n```\r\n\r\nYou can also make the tokenizer raise errors if the surface forms of certain special tokens are encountered in the input texts by passing a collection of `str` as `disallowed_special`\r\n\r\n```python\r\n>>> tokenizer('print(\"<|extra_0|>\")<|endoftext|>', allowed_special={'<|endoftext|>'}, disallowed_special=('<|extra_0|>', ))\r\n...\r\nValueError: Encountered text corresponding to disallowed special token '<|extra_0|>'.\r\nIf you want this text to be encoded as a special token, pass it to `allowed_special`, e.g. `allowed_special={'<|extra_0|>', ...}`.\r\nIf you want this text to be encoded as normal text, disable the check for this token by passing `disallowed_special=(enc.special_tokens_set - {'<|extra_0|>'})`.\r\nTo disable this check for all special tokens, pass `disallowed_special=()`.\r\n```\r\n\r\nFor more information on `allowed_special` and `disallowed_special`, please refer to [the `tiktoken` documentation](https://github.com/openai/tiktoken/blob/095924e02c85617df6889698d94515f91666c7ea/tiktoken/core.py#L75).\r\n\r\nThe new default is the same as\r\n\r\n```python\r\n>>> tokenizer('print(\"<|endoftext|>\")', allowed_special=\"all\", disallowed_special=())\r\n{'input_ids': [1350, 445, 151643, 899], 'token_type_ids': [0, 0, 0, 0], 'attention_mask': [1, 1, 1, 1]}\r\n```\r\n\r\n## Vocabulary Expansion\r\n\r\n> WARNING: Read carefully, be aware of what you are doing, and use at your own risk. \r\n> There are certain caveats regarding how your vocabulary is produced.\r\n\r\nThe tokenizer of Qwen models are based on BPE and you cannot directly expand the vocabulary by adding words to the vocabulary. \r\nThe intermediate merges are needed for tokenization.\r\nPlease follow the steps to obtain such information.\r\n\r\n1. Prepare a plain text file `qwen_extra_vocab.txt`, where each line contains a token and its frequency separated by `\\t`. \r\n\r\n An example is given below:\r\n ```\r\n 我是一只猫\t20\r\n 你是一只猫\t10\r\n 他是一只猫\t5\r\n 一只\t200\r\n 一只猫\t100\r\n 夸张的 比喻手法\t20\r\n ```\r\n The frequencies are needed to compute the BPE.\r\n\r\n \r\n\r\n2. Prepare the base vocabulary file, e.g., `qwen.tiktoken`, and determine the start index for new tokens.\r\n \r\n There are 151,643 regular tokens and 208 control tokens in the vocabulary for Qwen models. \r\n For simplicity, the start index can be set as 151,851, which is the default value. \r\n You can, of course, override the many inactive control tokens, but you will need to modify the tokenizer code. \r\n\r\n3. Run the following command:\r\n ```\r\n python add_merges.py qwen.tiktoken qwen_extra.tiktoken qwen_extra_vocab.txt\r\n ```\r\n `add_merges.py` can be found [here](examples/add_merges.py).\r\n It will learn the new merges based on the provided `qwen_extra_vocab.txt`. \r\n The new tokens and their indices will be stored in `qwen_extra.tiktoken`. \r\n Modify the paths as you wish.\r\n\r\n It is a pure Python implementation, so please expect it to be slow if you are adding a lot of words.\r\n\r\n Please note that not all words can be added due to pre-tokenization. \r\n You will get warnings if you try to add such word:\r\n ```\r\n WARNING - 夸张的 比喻手法 would be pre-tokenized to ['夸张的', ' 比喻手法'], and thus cannot be added to vocabulary\r\n WARNING - word 一只 is already a token b'\\xe4\\xb8\\x80\\xe5\\x8f\\xaa', skipping\r\n INFO - number of existing merges: 151643\r\n INFO - number of words for expanding: 4\r\n DEBUG - (b'\\xe4\\xb8\\x80\\xe5\\x8f\\xaa', b'\\xe7\\x8c\\xab') (一只猫) is selected as the next merge with freq 100\r\n DEBUG - (b'\\xe5\\x8f\\xaa', b'\\xe7\\x8c\\xab') (只猫) is selected as the next merge with freq 35\r\n DEBUG - (b'\\xe6\\x98\\xaf\\xe4\\xb8\\x80', b'\\xe5\\x8f\\xaa\\xe7\\x8c\\xab') (是一只猫) is selected as the next merge with freq 35\r\n DEBUG - (b'\\xe6\\x88\\x91', b'\\xe6\\x98\\xaf\\xe4\\xb8\\x80\\xe5\\x8f\\xaa\\xe7\\x8c\\xab') (我是一只猫) is selected as the next merge with freq 20\r\n DEBUG - (b'\\xe4\\xbd\\xa0', b'\\xe6\\x98\\xaf\\xe4\\xb8\\x80\\xe5\\x8f\\xaa\\xe7\\x8c\\xab') (你是一只猫) is selected as the next merge with freq 10\r\n DEBUG - (b'\\xe4\\xbb\\x96', b'\\xe6\\x98\\xaf\\xe4\\xb8\\x80\\xe5\\x8f\\xaa\\xe7\\x8c\\xab') (他是一只猫) is selected as the next merge with freq 5\r\n INFO - number of newly learned merges: 6\r\n ```\r\n\r\nThe `qwen_extra.tiktoken` will contain the following lines:\r\n```\r\n5LiA5Y+q54yr 151851\r\n5Y+q54yr 151852\r\n5piv5LiA5Y+q54yr 151853\r\n5oiR5piv5LiA5Y+q54yr 151854\r\n5L2g5piv5LiA5Y+q54yr 151855\r\n5LuW5piv5LiA5Y+q54yr 151856\r\n```\r\n\r\nYou may use the file as follows in your code:\r\n``` python\r\nfrom transformers import AutoTokenizer\r\n\r\n>>> tokenizer = AutoTokenizer.from_pretrained(\"Qwen/Qwen-7B\", trust_remote_code=True, extra_vocab_file=\"qwen_extra.tiktoken\")\r\n\r\n>>> len(tokenizer)\r\n151857\r\n\r\n>>> tokenizer(\"我是一只猫\")\r\n{'input_ids': [151854], 'token_type_ids': [0], 'attention_mask': [1]}\r\n```\r\nNote: You need the latest tokenizer code, i.e., after 2023-10-08, to use the `extra_vocab_file` argument.\r\nOtherwise, you need to manually append `qwen.tiktoken` (of which path varies with your configuration) with the content from `qwen_extra.tiktoken`.\r\n\r\nCertainly, you will need to finetune the model for the new tokens to work.\r\n\r\n\r\n### Caveats\r\n\r\n\r\nThe tokenizer of Qwen operates directly on UTF-8 byte sequences, unlike others, e.g., SentencePiece that operates on Unicode codepoints/characters and falls back to UTF-8 byte sequences for the unknown (IIRC). \r\nThe thing is if the frequencies are computed on limited data, the Unicode codepoint boundary may not be correctly recognized.\r\nIn theory, it could be a problem for fine-tuned models using the expanded vocabulary with limited data.\r\n\r\nFor example, it could happen that `b'\\x80\\xe5'` might be merged first for the UTF-8 byte sequence `b'\\xe4\\xb8\\x80\\xe5\\x8f\\xaa'` of the string `一只`, across the Unicode codepoint of `一` (`b'\\xe4\\xb8\\x80'`) and `只` (`b'\\xe5\\x8f\\xaa'`).\r\nNormally, this would work just fine for known tokens, but for actually unknown words, unusual merges may happen, which may not be well understood for the pre-trained model.\r\n\r\nOur advice is that to be safe, you should gather the Unicode codepoints from all the words you need to add, and also add them to the file with frequencies higher than the sum of the frequencies of the corresponding words.\r\nBut since Qwen has most of the Chinese words, it could be okay to just add the Chinese words alone.\r\n\r\nFor curious minds, you will also notice that in the given example, `一只` is a token and `只猫` is also learned as a new token. \r\nThe reason is that `是一` is also a token in Qwen and has higher merging priority than `一只`, such that the merging path for `是|一|只|猫` is `是一|只|猫 -> 是一|只猫 -> 是一只猫` (omitting the UTF-8 byte merges).\r\n\r\nThis is the characteristic for plain BPE: it is based solely on distribution, meaning it does not have knowledge of which bytes can form a valid Unicode codepoint, character, or meaningful word.\r\n\r\nThe byproduct is that text may be sub-tokenized differently in different contexts, even for words containing only ASCII characters.\r\n```python\r\n>>> tokenizer.tokenize(\"Panda\")\r\n[b'P', b'anda']\r\n\r\n>>> tokenizer.tokenize(\" Panda\")\r\n[b' Panda']\r\n\r\n>>> tokenizer.tokenize(\"Pandas\")\r\n[b'P', b'andas']\r\n\r\n>>> tokenizer.tokenize(\" Pandas\")\r\n[b' Pand', b'as']\r\n```\r\nThis simply suggests that those combinations occur more frequently in the data.\r\nIf you have vast amount of training data, it should not be a problem." }, { "path": "tokenization_note_ja.md", "content": "# トークン化\r\n\r\nQwen-7B は `tiktoken` パッケージを使用して、UTF-8 バイトを BPE トークン化します。\r\nQwen-7B には 2 種類のトークンがあります。BPE の通常のトークン (`bytes` 型) と特殊/制御トークン (`str` 型) です。\r\n\r\n```python\r\nfrom transformers import AutoTokenizer\r\n\r\ntokenizer = AutoTokenizer.from_pretrained('Qwen/Qwen-7B', trust_remote_code=True)\r\n```\r\n\r\n## 通常のトークン\r\n\r\n通常のトークンは、UTF-8 エンコーディングでエンコードされたテキストのバイト列から学習した BPE トークンです。\r\nこれによってすべてのテキストをトークン化することができ、未知のトークンは存在しませんが、一般的でないテキストをトークン化するときにシングルバイトを使用するようにフォールバックすることがあります。\r\nUTF-8 のデコードエラーに遭遇することがあり、そのエラーのデフォルトは `replace` であるため、不完全な生成では置換文字 (�) が使用されます。\r\nこの動作は `errors=\"ignore\"` を `decode` 関数に渡すことで変更することができる。\r\n`errors` のオプションについては、[Python ドキュメント](https://docs.python.org/3/library/stdtypes.html#bytes.decode) を参照してください。\r\n\r\n```python\r\n>>> tokenizer.decode([51461])\r\n' �'\r\n\r\n>>> tokenizer.convert_ids_to_tokens([51461])\r\n[b' \\xe6\\xa0']\r\n\r\n>>> b' \\xe6\\xa0'.decode(\"utf-8\", errors='replace')\r\n' �'\r\n\r\n>>> tokenizer.decode([51461, 117])\r\n' 根'\r\n\r\n>>> tokenizer.convert_ids_to_tokens([51461, 117])\r\n[b' \\xe6\\xa0', b'\\xb9']\r\n\r\n>>> b' \\xe6\\xa0\\xb9'.decode(\"utf-8\", errors='replace')\r\n' 根'\r\n```\r\n\r\n通常のトークン (`bytes` 単位) からその ID へのマッピングは `tokenizer.get_vocab()` から取得できます。\r\n通常のトークンを語彙に追加することはサポートしていませんし、推奨もしていません。\r\n\r\n## 特別なトークン\r\n\r\n特別なトークンは、例えば文書の最後に到達するなど、モデルにとって特別な機能を意味します。\r\n理論的には、これらは入力テキストには存在せず、入力テキストが処理された後にのみ現れます。\r\n例えば、文書の終わりを表す `<|endoftext|>` のような表面的な形は、参照を容易にするためだけのものである。\r\n現在、Qwen-7B では `<|endoftext|>` が、Qwen-7B-Chat では `<|endoftext|>`, `<|im_start|>`, `<|im_end|>` が特殊トークンとして使われています。\r\n他の目的のために、`<|extra_0|>` から `<|extra_204|>` までの特別なトークンを保持しています。\r\n特殊トークンの表面形式 (`str` 内) から ID へのマッピングは `tokenizer.special_tokens` から取得できます。\r\n\r\n`bos`、`eos`、`unk`、`pad`、`mask`、`sep` などの概念は学習済みモデル（Qwen-7B と Qwen-7B-Chat）には適用できません。\r\nしかし、`pad` トークンは話が別です。理論的には、モデルがこのトークンを見たり計算したりすることはないので、既知のトークンを使用することができます。\r\nしかし、安全のために、トークナイザーの初期化で指定する特別なトークンの値は、既知の特別なトークンに限定します。\r\n微調整やその他のフレームワークで特別なトークンを必要とする場合は、次のように指定できます\r\n\r\n```python\r\nfrom transformers import AutoTokenizer\r\n\r\ntokenizer = AutoTokenizer.from_pretrained('Qwen/Qwen-7B', trust_remote_code=True, pad_token='<|endoftext|>')\r\n```\r\n\r\n> 警告: 私たちが事前に学習したモデルでは、`bos`, `eos`, `unk` などを設定しても意味がありません。\r\n> 特に、`<<endoftext|>` を `eos` のように使ってはいけません。\r\n> 特に `<|endoftext|>` を `eos` として使用することは、文末と文末が同じであると確信できる場合を除き、避けるべきです。\r\n\r\n## インジェクション攻撃の防止\r\n\r\n特殊トークンは通常のトークンとは異なるため、コントロールトークンの表面形が入力テキストに現れるとどうなるでしょうか？\r\n例えば、次のようなテキストがあるとします\r\n\r\n```\r\nprint(\"<|endoftext|>\")\r\n```\r\n\r\nこれは次のようにしてトークン化する必要があります\r\n\r\n```\r\nids:[1350, 9639, 91, 8691, 723, 427, 91, 82598]\r\ntokens: [b'print', b'(\"<', b'|', b'endo', b'ft', b'ext', b'|', b'>\")']\r\n```\r\n\r\nこちらではありません\r\n\r\n```\r\nids: [1350, 445, 151643, 899]\r\ntokens: [b'print', b'(\"', '<|endoftext|>', b'\")']\r\n```\r\n\r\nつまり、特殊トークンの表面形は通常のテキストと同じように扱い、特殊トークンはテキストのトークン化後に開発者が処理するというものです。\r\nしかし、これはコミュニティにおける（安全ではないとはいえ）慣習に抵触し、開発者が車輪を再利用するための新たなステップを追加することになります。\r\n\r\nデフォルトの動作は、すべての既知の特殊トークンの表面形を特殊トークンとして解析するように変更されました。\r\nインジェクション防止を有効にするには、トークナイザーの呼び出しに `allowed_special=set()` を渡します:\r\n\r\n```python\r\n>>> tokenizer('print(\"<|endoftext|>\")', allowed_special=set())\r\n{'input_ids': [1350, 9639, 91, 8691, 723, 427, 91, 82598], 'token_type_ids': [0, 0, 0, 0, 0, 0, 0, 0], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1]}\r\n```\r\n\r\n`str` のセットを `allowed_special` として渡すことで、きめ細かく動作を制御することができます\r\n\r\n```python\r\n>>> tokenizer('print(\"<|extra_0|>\")<|endoftext|>', allowed_special={'<|endoftext|>'})\r\n{'input_ids': [1350, 9639, 91, 15460, 62, 15, 91, 82598, 151643], 'token_type_ids': [0, 0, 0, 0, 0, 0, 0, 0, 0], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1]}\r\n```\r\n\r\n`str` のコレクションを `disallowed_special` として渡すことで、特定の特殊なトークンの表形式が入力テキストで遭遇した場合にトークナイザーがエラーを発生するようにすることもできます\r\n\r\n```python\r\n>>> tokenizer('print(\"<|extra_0|>\")<|endoftext|>', allowed_special={'<|endoftext|>'}, disallowed_special=('<|extra_0|>', ))\r\n...\r\nValueError: Encountered text corresponding to disallowed special token '<|extra_0|>'.\r\nIf you want this text to be encoded as a special token, pass it to `allowed_special`, e.g. `allowed_special={'<|extra_0|>', ...}`.\r\nIf you want this text to be encoded as normal text, disable the check for this token by passing `disallowed_special=(enc.special_tokens_set - {'<|extra_0|>'})`.\r\nTo disable this check for all special tokens, pass `disallowed_special=()`.\r\n```\r\n\r\n`allowed_special` と `disallowed_special` の詳細については、[`tiktoken` ドキュメント](https://github.com/openai/tiktoken/blob/095924e02c85617df6889698d94515f91666c7ea/tiktoken/core.py#L75)を参照してください。\r\n\r\n新しいデフォルトは以下の通り\r\n\r\n```python\r\n>>> tokenizer('print(\"<|endoftext|>\")', allowed_special=\"all\", disallowed_special=())\r\n{'input_ids': [1350, 445, 151643, 899], 'token_type_ids': [0, 0, 0, 0], 'attention_mask': [1, 1, 1, 1]}\r\n```\r\n\r\n" }, { "path": "tokenization_note_zh.md", "content": "# Tokenization\r\n\r\n> 注：作为术语的“tokenization”在中文中尚无共识的概念对应，本文档采用英文表达以利说明。\r\n\r\nQwen-7B采用UTF-8字节级别的BPE tokenization方式，并依赖`tiktoken`这一高效的软件包执行分词。\r\nQwen-7B中有两类token，即源于BPE、`bytes`类型的普通token和特殊指定、`str`类型的特殊token。\r\n\r\n```python\r\nfrom transformers import AutoTokenizer\r\n\r\ntokenizer = AutoTokenizer.from_pretrained('Qwen/Qwen-7B', trust_remote_code=True)\r\n```\r\n\r\n## 普通token\r\n\r\n普通token源于BPE，是在UTF-8编码的文本字节序列上学习得到的。\r\n尽管基于字节序列的方式保证了所有文本均可被tokenize且没有未登录token问题，但处理罕见文本时有可能回退到字节级别的编码。\r\n由于从字节序列解码为文本时，`errors`参数设为`replace`，处理不完整的token序列可能会遇到UTF-8解码错误，表象是生成中包含“替换字符”(�)。\r\n这一行为可以通过将`errors`参数设为`ignore`来规避。\r\n一次性修改可以传入tokenizer的`decode`函数，持久性修改可以传入tokenizer的初始化函数，请注意`decode`的配置优先级更高。\r\n`errors`的可选值，请参阅[Python文档](https://docs.python.org/3/library/stdtypes.html#bytes.decode).\r\n\r\n```python\r\n>>> tokenizer.decode([51461])\r\n' �'\r\n\r\n>>> tokenizer.convert_ids_to_tokens([51461])\r\n[b' \\xe6\\xa0']\r\n\r\n>>> b' \\xe6\\xa0'.decode(\"utf-8\", errors='replace')\r\n' �'\r\n\r\n>>> tokenizer.decode([51461, 117])\r\n' 根'\r\n\r\n>>> tokenizer.convert_ids_to_tokens([51461, 117])\r\n[b' \\xe6\\xa0', b'\\xb9']\r\n\r\n>>> b' \\xe6\\xa0\\xb9'.decode(\"utf-8\", errors='replace')\r\n' 根'\r\n```\r\n\r\n`bytes`类型的普通token到id的映射可以通过`tokenizer.get_vocab()`获取。\r\n尚不支持也不推荐向tokenizer增加普通token。\r\n\r\n## 特殊token\r\n\r\n特殊token用以给模型传递特殊信号，如到达文本末尾。\r\n理论上，输入文本中不包含特殊token，它们仅在tokenization后由开发者手动加入。\r\n特殊token的字面表达，如表示文本结束的`<|endoftext|>`，仅便于指代特殊token，不意味着它们在输入文本空间中。\r\n目前，训练中使用的、已经有固定含义的、不应做它用的特殊token，Qwen-7B中有`<|endoftext|>`，Qwen-7B-Chat中有`<|endoftext|>`、`<|im_start|>`以及`<|im_end|>`。\r\n但词表中也留有供扩展的特殊token位，可用`<|extra_0|>`到`<|extra_204|>`来指代。\r\n`str`类型的特殊token字面表达到id的映射，可以通过`tokenizer.special_tokens`获取。\r\n\r\n对于提供的模型参数(Qwen-7B和Qwen-7B-Chat)而言，诸如`bos`、`eos`、`unk`、`pad`、`mask`、`sep`等的特殊token的概念并不适用。\r\n特例是`pad`，由于这个token理论上并不参与模型计算，所以可以使用任意token表达这一概念。\r\n但保险起见，目前可在tokenizer初始化时设定的特殊token，仅可使用已知的特殊token字面表达，即`<|endoftext|>`、`<|im_start|>`、`<|im_end|>`和`<|extra_0|>`到`<|extra_204|>`。\r\n对于微调或者其它需要这些token才能运行的框架，可以如下配置\r\n\r\n```python\r\nfrom transformers import AutoTokenizer\r\n\r\ntokenizer = AutoTokenizer.from_pretrained('Qwen/Qwen-7B', trust_remote_code=True, pad_token='<|endoftext|>')\r\n```\r\n\r\n> 注意: 对于提供的训练好的模型，设置诸如`bos`、`eos`、`unk`之类的没有意义，即模型不需要这些概念。\r\n> 如果设置了这些token，但没有相应的微调这些token以让模型理解其含义，未知行为可能被触发。\r\n> 特别时，不应混淆`<|endoftext|>`和`eos`的概念，除非应用场景中它们的实际含义是一致的，即句子末尾等价于文本末尾。\r\n\r\n**注入攻击防御**\r\n\r\n由于特殊token和普通token概念上的差异，如果输入文本中含有特殊token的字面表达该如何处理？\r\n以下面文本为例\r\n\r\n```\r\nprint(\"<|endoftext|>\")\r\n```\r\n\r\n其正确的tokenization为\r\n\r\n```\r\nids:[1350, 9639, 91, 8691, 723, 427, 91, 82598]\r\ntokens: [b'print', b'(\"<', b'|', b'endo', b'ft', b'ext', b'|', b'>\")']\r\n```\r\n\r\n不是\r\n\r\n```\r\nids: [1350, 445, 151643, 899]\r\ntokens: [b'print', b'(\"', '<|endoftext|>', b'\")']\r\n```\r\n\r\n默认行为曾是正确的，即输入文本中任何字符一律按普通token处理，特殊token应由开发者在tokenization人工处理。\r\n然后，这与社区中的实践似有差异，为开发者复用代码增加了额外适配步骤。\r\n\r\n默认行为已被调整为从输入文本中解析特殊token的字面表达。\r\n如需启用注入攻击防御，请传入参数`allowed_special=set()`：\r\n\r\n```python\r\n>>> tokenizer('print(\"<|endoftext|>\")', allowed_special=set())\r\n{'input_ids': [1350, 9639, 91, 8691, 723, 427, 91, 82598], 'token_type_ids': [0, 0, 0, 0, 0, 0, 0, 0], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1]}\r\n```\r\n\r\n这一行为可以更精细的调控，将`allowed_special`设计为`str`的集合即可：\r\n\r\n```python\r\n>>> tokenizer('print(\"<|extra_0|>\")<|endoftext|>', allowed_special={'<|endoftext|>'})\r\n{'input_ids': [1350, 9639, 91, 15460, 62, 15, 91, 82598, 151643], 'token_type_ids': [0, 0, 0, 0, 0, 0, 0, 0, 0], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1]}\r\n```\r\n\r\n如果希望输入中遇到特殊token的字面表达时，获得更直接的提醒，通过配置`disallowed_special`可以让tokenizer直接触发异常：\r\n\r\n```python\r\n>>> tokenizer('print(\"<|extra_0|>\")<|endoftext|>', allowed_special={'<|endoftext|>'}, disallowed_special=('<|extra_0|>', ))\r\n...\r\nValueError: Encountered text corresponding to disallowed special token '<|extra_0|>'.\r\nIf you want this text to be encoded as a special token, pass it to `allowed_special`, e.g. `allowed_special={'<|extra_0|>', ...}`.\r\nIf you want this text to be encoded as normal text, disable the check for this token by passing `disallowed_special=(enc.special_tokens_set - {'<|extra_0|>'})`.\r\nTo disable this check for all special tokens, pass `disallowed_special=()`.\r\n```\r\n\r\n更多关于`allowed_special`和`disallowed_special`的信息, 请参阅[`tiktoken`代码](https://github.com/openai/tiktoken/blob/095924e02c85617df6889698d94515f91666c7ea/tiktoken/core.py#L75).\r\n\r\n新的默认行为与以下设定等价\r\n\r\n```python\r\n>>> tokenizer('print(\"<|endoftext|>\")', allowed_special=\"all\", disallowed_special=())\r\n{'input_ids': [1350, 445, 151643, 899], 'token_type_ids': [0, 0, 0, 0], 'attention_mask': [1, 1, 1, 1]}\r\n```\r\n\r\n## 词表扩展\r\n\r\n> 特别提醒：请仔细阅读本部分的说明，理解每一步操作，并承担可能的后果。\r\n> 由于词表扩展部分由您提供，产出方式的差异可能导致特定的不兼容情况，请审慎操作。\r\n\r\nQwen系列模型的tokenizer基于BPE方案提取文本中的token。\r\n从UTF-8编码的字节开始（每个字节都可以是一个token），两两token合并成为新token，直至不能再合并出新的token为止。\r\n由于词表同时还记录了token的合并方式，直接向词表中添加词可能对Qwen的tokenizer并不适用，即通过已有的token可能合并不出来您添加词。\r\n\r\n因而，请参照以下步骤获得合并信息：\r\n\r\n1. 准备一个纯文本文件，例如名为`qwen_extra_vocab.txt`，每行一个待添加的词和它的频率，中间用制表符`\\t`分隔。\r\n\r\n 以下是一个文件的例子：\r\n ```\r\n 我是一只猫\t20\r\n 你是一只猫\t10\r\n 他是一只猫\t5\r\n 一只\t200\r\n 一只猫\t100\r\n 夸张的 比喻手法\t20 \r\n ```\r\n 频率是必需的，用来计算合并的优先级。\r\n\r\n2. 准备基础的词表文件，例如`qwen.tiktoken`，并确认新加入token的起始索引。\r\n\r\n Qwen模型词表中有151,643个普通token，有208个特殊token。\r\n 简单起见，起始索引可以设置为151,851（默认值）。\r\n 您可以覆写不起效的特殊token，但您需要相应的修改tokenizer代码。\r\n\r\n3. 运行以下命令：\r\n ```\r\n python add_merges.py qwen.tiktoken qwen_extra.tiktoken qwen_extra_vocab.txt\r\n ```\r\n `add_merges.py`代码在[GitHub存储库](examples/add_merges.py)中。\r\n 基于提供的`qwen_extra_vocab.txt`，该脚本将学习新的token合并方式。\r\n 新token及其索引将存储在`qwen_extra.tiktoken`文件中。\r\n 您可以视情况修改有关路径。\r\n\r\n 由于是纯Python实现，如果您添加了非常多的词，预期会花费较多时间。\r\n\r\n 请注意，由于预切分，有些词是无法作为token加入的。\r\n 如果您添加了这些词，您会收到警告：\r\n ```\r\n WARNING - 夸张的 比喻手法 would be pre-tokenized to ['夸张的', ' 比喻手法'], and thus cannot be added to vocabulary\r\n WARNING - word 一只 is already a token b'\\xe4\\xb8\\x80\\xe5\\x8f\\xaa', skipping\r\n INFO - number of existing merges: 151643\r\n INFO - number of words for expanding: 4\r\n DEBUG - (b'\\xe4\\xb8\\x80\\xe5\\x8f\\xaa', b'\\xe7\\x8c\\xab') (一只猫) is selected as the next merge with freq 100\r\n DEBUG - (b'\\xe5\\x8f\\xaa', b'\\xe7\\x8c\\xab') (只猫) is selected as the next merge with freq 35\r\n DEBUG - (b'\\xe6\\x98\\xaf\\xe4\\xb8\\x80', b'\\xe5\\x8f\\xaa\\xe7\\x8c\\xab') (是一只猫) is selected as the next merge with freq 35\r\n DEBUG - (b'\\xe6\\x88\\x91', b'\\xe6\\x98\\xaf\\xe4\\xb8\\x80\\xe5\\x8f\\xaa\\xe7\\x8c\\xab') (我是一只猫) is selected as the next merge with freq 20\r\n DEBUG - (b'\\xe4\\xbd\\xa0', b'\\xe6\\x98\\xaf\\xe4\\xb8\\x80\\xe5\\x8f\\xaa\\xe7\\x8c\\xab') (你是一只猫) is selected as the next merge with freq 10\r\n DEBUG - (b'\\xe4\\xbb\\x96', b'\\xe6\\x98\\xaf\\xe4\\xb8\\x80\\xe5\\x8f\\xaa\\xe7\\x8c\\xab') (他是一只猫) is selected as the next merge with freq 5\r\n INFO - number of newly learned merges: 6\r\n ```\r\n\r\n`qwen_extra.tiktoken`会包含以下内容：\r\n```\r\n5LiA5Y+q54yr 151851\r\n5Y+q54yr 151852\r\n5piv5LiA5Y+q54yr 151853\r\n5oiR5piv5LiA5Y+q54yr 151854\r\n5L2g5piv5LiA5Y+q54yr 151855\r\n5LuW5piv5LiA5Y+q54yr 151856\r\n```\r\n\r\n您可以按如下方式使用扩展后的词表：\r\n``` python\r\nfrom transformers import AutoTokenizer\r\n\r\n>>> tokenizer = AutoTokenizer.from_pretrained(\"Qwen/Qwen-7B\", trust_remote_code=True, extra_vocab_file=\"qwen_extra.tiktoken\")\r\n\r\n>>> len(tokenizer)\r\n151857\r\n\r\n>>> tokenizer(\"我是一只猫\")\r\n{'input_ids': [151854], 'token_type_ids': [0], 'attention_mask': [1]}\r\n```\r\n\r\n注意：您需要使用2023年10月8日后的tokenizer代码才能传递`extra_vocab_file`参数。如是其它情况，您可以将`qwen_extra.tiktoken`内容复制粘贴到`qwen.tiktoken`内容后面。\r\n\r\n您需要微调模型才能使新的token发挥作用。\r\n\r\n### 注意事项\r\n\r\nQwen的tokenizer是直接从UTF-8编码的字节序列开始处理的，这与其它tokenizer比如SentencePiece是很不一样的。SentencePiece是从Unicode码位（可以理解为一个字符）开始处理，遇到未登录的再用UTF-8编码成字节。\r\n从字节开始的一个潜在问题是如果频率信息不够准确，比如频率信息是在很少数据上统计得到的，Unicode码位按UTF-8编码成字节后的边界可能会出现差错。\r\n理论上，如果模型微调数据量不足，使用扩展后的词表也可能出现意外问题。\r\n\r\n举个例子（非实际情况），对于`一只`的UTF-8字节序列`b'\\xe4\\xb8\\x80\\xe5\\x8f\\xaa'`，中间两个字节`b'\\x80\\xe5'`可能会先合并为一个token，跨越了`一`(`b'\\xe4\\xb8\\x80'`)和`只`(`b'\\xe5\\x8f\\xaa'`)的码位边界。\r\n这对于已登录token不会有什么影响（最后总会合并为`一只`），但对于未登录的，可能会产生一些不同寻常的合并/token。\r\n这些token序列可能对于预训练模型是陌生的。\r\n\r\n我们的建议是保险起见，您最好先收集待添加词中的所有Unicode码位，然后单独指定它们的频率大于其所构成词的频率之和。\r\n不过由于Qwen的tokenizer已包含了大多数中文字，对于中文词的话，不添加中文字的频率，大部分情况下是可行的。\r\n\r\n您可能已经发现了，在提供的例子中，`一只`已经是登录过的token了，但`只猫`还是学习成为了一个新token，出现了“交叉”。\r\n原因是在Qwen中`是一`也是一个已知token，且其频率/优先级比`一只`要高，因而对于`是|一|只|猫`这个片段，合并的次序是`是一|只|猫 -> 是一|只猫 -> 是一只猫`（省略UTF-8字节级别的合并）。\r\n\r\n这是常规BPE的特性，其完全基于分布，并不知道哪些字节可以构成合法的Unicode码位、合法的字符或是词。\r\n\r\n副产物是一段文本在不同的上下文下可能会有不同的tokenize结果，对于仅包含ASCII字符的文本同样如此。\r\n```python\r\n>>> tokenizer.tokenize(\"Panda\")\r\n[b'P', b'anda']\r\n\r\n>>> tokenizer.tokenize(\" Panda\")\r\n[b' Panda']\r\n\r\n>>> tokenizer.tokenize(\"Pandas\")\r\n[b'P', b'andas']\r\n\r\n>>> tokenizer.tokenize(\" Pandas\")\r\n[b' Pand', b'as']\r\n```\r\n这仅说明在用于学习BPE的数据中，这样的组合是更高频的。\r\n如果您有海量的训练语料，这并不会是个问题。" }, { "path": "utils.py", "content": "import torch\nfrom transformers import AutoModelForCausalLM\nfrom accelerate import dispatch_model\n\n\ndef _device_map(num_gpus, num_layers):\n per_gpu_layers = (num_layers + 2) / num_gpus\n\n device_map = {\n 'transformer.wte': 0,\n 'transformer.ln_f': 0,\n 'lm_head': num_gpus-1\n }\n\n used = 1\n gpu_target = 0\n for i in range(num_layers):\n if used >= per_gpu_layers:\n gpu_target += 1\n used = 0 if gpu_target < num_gpus-1 else 1\n assert gpu_target < num_gpus\n device_map[f'transformer.h.{i}'] = gpu_target\n used += 1\n\n return device_map\n\n\ndef load_model_on_gpus(model_name_or_path, num_gpus: int = 2):\n num_devices = torch.cuda.device_count()\n\n if num_gpus == 1:\n model = AutoModelForCausalLM.from_pretrained(model_name_or_path, device_map='auto',\n trust_remote_code=True).eval()\n elif 1 < num_gpus <= num_devices:\n model = AutoModelForCausalLM.from_pretrained(model_name_or_path, device_map='cpu',\n trust_remote_code=True).eval()\n num_layers = model.config.num_hidden_layers\n device_map = _device_map(num_gpus, num_layers)\n print(device_map)\n model = dispatch_model(model, device_map=device_map)\n else:\n raise KeyError\n\n return model\n" }, { "path": "web_demo.py", "content": "# Copyright (c) Alibaba Cloud.\n#\n# This source code is licensed under the license found in the\n# LICENSE file in the root directory of this source tree.\n\n\"\"\"A simple web interactive chat demo based on gradio.\"\"\"\nimport os\nfrom argparse import ArgumentParser\n\nimport gradio as gr\nimport mdtex2html\n\nimport torch\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\nfrom transformers.generation import GenerationConfig\n\n\nDEFAULT_CKPT_PATH = 'Qwen/Qwen-7B-Chat'\n\n\ndef _get_args():\n parser = ArgumentParser()\n parser.add_argument(\"-c\", \"--checkpoint-path\", type=str, default=DEFAULT_CKPT_PATH,\n help=\"Checkpoint name or path, default to %(default)r\")\n parser.add_argument(\"--cpu-only\", action=\"store_true\", help=\"Run demo with CPU only\")\n\n parser.add_argument(\"--share\", action=\"store_true\", default=False,\n help=\"Create a publicly shareable link for the interface.\")\n parser.add_argument(\"--inbrowser\", action=\"store_true\", default=False,\n help=\"Automatically launch the interface in a new tab on the default browser.\")\n parser.add_argument(\"--server-port\", type=int, default=8000,\n help=\"Demo server port.\")\n parser.add_argument(\"--server-name\", type=str, default=\"127.0.0.1\",\n help=\"Demo server name.\")\n\n args = parser.parse_args()\n return args\n\n\ndef _load_model_tokenizer(args):\n tokenizer = AutoTokenizer.from_pretrained(\n args.checkpoint_path, trust_remote_code=True, resume_download=True,\n )\n\n if args.cpu_only:\n device_map = \"cpu\"\n else:\n device_map = \"auto\"\n\n model = AutoModelForCausalLM.from_pretrained(\n args.checkpoint_path,\n device_map=device_map,\n trust_remote_code=True,\n resume_download=True,\n ).eval()\n\n config = GenerationConfig.from_pretrained(\n args.checkpoint_path, trust_remote_code=True, resume_download=True,\n )\n\n return model, tokenizer, config\n\n\ndef postprocess(self, y):\n if y is None:\n return []\n for i, (message, response) in enumerate(y):\n y[i] = (\n None if message is None else mdtex2html.convert(message),\n None if response is None else mdtex2html.convert(response),\n )\n return y\n\n\ngr.Chatbot.postprocess = postprocess\n\n\ndef _parse_text(text):\n lines = text.split(\"\\n\")\n lines = [line for line in lines if line != \"\"]\n count = 0\n for i, line in enumerate(lines):\n if \"```\" in line:\n count += 1\n items = line.split(\"`\")\n if count % 2 == 1:\n lines[i] = f'<pre><code class=\"language-{items[-1]}\">'\n else:\n lines[i] = f\"<br></code></pre>\"\n else:\n if i > 0:\n if count % 2 == 1:\n line = line.replace(\"`\", r\"\\`\")\n line = line.replace(\"<\", \"<\")\n line = line.replace(\">\", \">\")\n line = line.replace(\" \", \" \")\n line = line.replace(\"*\", \"*\")\n line = line.replace(\"_\", \"_\")\n line = line.replace(\"-\", \"-\")\n line = line.replace(\".\", \".\")\n line = line.replace(\"!\", \"!\")\n line = line.replace(\"(\", \"(\")\n line = line.replace(\")\", \")\")\n line = line.replace(\"$\", \"$\")\n lines[i] = \"<br>\" + line\n text = \"\".join(lines)\n return text\n\n\ndef _gc():\n import gc\n gc.collect()\n if torch.cuda.is_available():\n torch.cuda.empty_cache()\n\n\ndef _launch_demo(args, model, tokenizer, config):\n\n def predict(_query, _chatbot, _task_history):\n print(f\"User: {_parse_text(_query)}\")\n _chatbot.append((_parse_text(_query), \"\"))\n full_response = \"\"\n\n for response in model.chat_stream(tokenizer, _query, history=_task_history, generation_config=config):\n _chatbot[-1] = (_parse_text(_query), _parse_text(response))\n\n yield _chatbot\n full_response = _parse_text(response)\n\n print(f\"History: {_task_history}\")\n _task_history.append((_query, full_response))\n print(f\"Qwen-Chat: {_parse_text(full_response)}\")\n\n def regenerate(_chatbot, _task_history):\n if not _task_history:\n yield _chatbot\n return\n item = _task_history.pop(-1)\n _chatbot.pop(-1)\n yield from predict(item[0], _chatbot, _task_history)\n\n def reset_user_input():\n return gr.update(value=\"\")\n\n def reset_state(_chatbot, _task_history):\n _task_history.clear()\n _chatbot.clear()\n _gc()\n return _chatbot\n\n with gr.Blocks() as demo:\n gr.Markdown(\"\"\"\\\n<p align=\"center\"><img src=\"https://qianwen-res.oss-cn-beijing.aliyuncs.com/logo_qwen.jpg\" style=\"height: 80px\"/><p>\"\"\")\n gr.Markdown(\"\"\"<center><font size=8>Qwen-Chat Bot</center>\"\"\")\n gr.Markdown(\n \"\"\"\\\n<center><font size=3>This WebUI is based on Qwen-Chat, developed by Alibaba Cloud. \\\n(本WebUI基于Qwen-Chat打造，实现聊天机器人功能。)</center>\"\"\")\n gr.Markdown(\"\"\"\\\n<center><font size=4>\nQwen-7B <a href=\"https://modelscope.cn/models/qwen/Qwen-7B/summary\">🤖 </a> | \n<a href=\"https://huggingface.co/Qwen/Qwen-7B\">🤗</a>  ｜ \nQwen-7B-Chat <a href=\"https://modelscope.cn/models/qwen/Qwen-7B-Chat/summary\">🤖 </a> | \n<a href=\"https://huggingface.co/Qwen/Qwen-7B-Chat\">🤗</a>  ｜ \nQwen-14B <a href=\"https://modelscope.cn/models/qwen/Qwen-14B/summary\">🤖 </a> | \n<a href=\"https://huggingface.co/Qwen/Qwen-14B\">🤗</a>  ｜ \nQwen-14B-Chat <a href=\"https://modelscope.cn/models/qwen/Qwen-14B-Chat/summary\">🤖 </a> | \n<a href=\"https://huggingface.co/Qwen/Qwen-14B-Chat\">🤗</a>  ｜ \n <a href=\"https://github.com/QwenLM/Qwen\">Github</a></center>\"\"\")\n\n chatbot = gr.Chatbot(label='Qwen-Chat', elem_classes=\"control-height\")\n query = gr.Textbox(lines=2, label='Input')\n task_history = gr.State([])\n\n with gr.Row():\n empty_btn = gr.Button(\"🧹 Clear History (清除历史)\")\n submit_btn = gr.Button(\"🚀 Submit (发送)\")\n regen_btn = gr.Button(\"🤔️ Regenerate (重试)\")\n\n submit_btn.click(predict, [query, chatbot, task_history], [chatbot], show_progress=True)\n submit_btn.click(reset_user_input, [], [query])\n empty_btn.click(reset_state, [chatbot, task_history], outputs=[chatbot], show_progress=True)\n regen_btn.click(regenerate, [chatbot, task_history], [chatbot], show_progress=True)\n\n gr.Markdown(\"\"\"\\\n<font size=2>Note: This demo is governed by the original license of Qwen. \\\nWe strongly advise users not to knowingly generate or allow others to knowingly generate harmful content, \\\nincluding hate speech, violence, pornography, deception, etc. \\\n(注：本演示受Qwen的许可协议限制。我们强烈建议，用户不应传播及不应允许他人传播以下内容，\\\n包括但不限于仇恨言论、暴力、色情、欺诈相关的有害信息。)\"\"\")\n\n demo.queue().launch(\n share=args.share,\n inbrowser=args.inbrowser,\n server_port=args.server_port,\n server_name=args.server_name,\n )\n\n\ndef main():\n args = _get_args()\n\n model, tokenizer, config = _load_model_tokenizer(args)\n\n _launch_demo(args, model, tokenizer, config)\n\n\nif __name__ == '__main__':\n main()\n" } ]