Repository: zai-org/GLM-4 Branch: main Commit: 3f79b543647b Files: 69 Total size: 328.3 KB Directory structure: gitextract_0zacjmse/ ├── .github/ │ ├── ISSUE_TEMPLATE/ │ │ ├── bug_report.yaml │ │ └── feature-request.yaml │ └── PULL_REQUEST_TEMPLATE.md ├── .gitignore ├── .pre-commit-config.yaml ├── LICENSE ├── README.md ├── README_20240605.md ├── README_zh.md ├── README_zh_240605.md ├── demo/ │ ├── composite_demo/ │ │ ├── .gitignore │ │ ├── README.md │ │ ├── README_en.md │ │ ├── browser/ │ │ │ ├── .gitignore │ │ │ ├── package.json │ │ │ ├── src/ │ │ │ │ ├── browser.ts │ │ │ │ ├── config.ts │ │ │ │ ├── server.ts │ │ │ │ ├── types.ts │ │ │ │ └── utils.ts │ │ │ └── tsconfig.json │ │ ├── requirements.txt │ │ └── src/ │ │ ├── client.py │ │ ├── clients/ │ │ │ ├── hf.py │ │ │ ├── openai.py │ │ │ └── vllm.py │ │ ├── conversation.py │ │ ├── main.py │ │ ├── tools/ │ │ │ ├── browser.py │ │ │ ├── cogview.py │ │ │ ├── config.py │ │ │ ├── interface.py │ │ │ ├── python.py │ │ │ └── tool_registry.py │ │ └── utils.py │ └── intel_device_demo/ │ ├── itrex/ │ │ ├── README.md │ │ ├── README_en.md │ │ ├── itrex_cli_demo.py │ │ └── requirements.txt │ └── openvino/ │ ├── README.md │ ├── README_en.md │ ├── convert.py │ ├── openvino_cli_demo.py │ └── requirements.txt ├── finetune/ │ ├── .gitignore │ ├── README.md │ ├── README_zh.md │ ├── configs/ │ │ ├── ds_zero_2.json │ │ ├── ds_zero_3.json │ │ ├── lora.yaml │ │ └── sft.yaml │ ├── finetune.py │ ├── finetune_vision.py │ └── requirements.txt ├── inference/ │ ├── README.md │ ├── README_zh.md │ ├── glm4v_api_request.py │ ├── glm4v_server.py │ ├── requirements.txt │ ├── trans_batch_demo.py │ ├── trans_cli_demo.py │ ├── trans_cli_vision_demo.py │ ├── trans_stress_test.py │ ├── trans_web_demo.py │ ├── trans_web_vision_demo.py │ ├── vllm_cli_demo.py │ └── vllm_cli_vision_demo.py ├── pyproject.toml └── resources/ └── WECHAT.md ================================================ FILE CONTENTS ================================================ ================================================ FILE: .github/ISSUE_TEMPLATE/bug_report.yaml ================================================ name: "\U0001F41B Bug Report" description: Submit a bug report to help us improve GLM-4-9B / 提交一个 Bug 问题报告来帮助我们改进 GLM-4-9B body: - type: textarea id: system-info attributes: label: System Info / 系統信息 description: Your operating environment / 您的运行环境信息 placeholder: Includes Cuda version, Transformers version, Python version, operating system, hardware information (if you suspect a hardware problem)... / 包括Cuda版本，Transformers版本，Python版本，操作系统，硬件信息(如果您怀疑是硬件方面的问题)... validations: required: true - type: textarea id: who-can-help attributes: label: Who can help? / 谁可以帮助到您？ description: | Your issue will be replied to more quickly if you can figure out the right person to tag with @ All issues are read by one of the maintainers, so if you don't know who to tag, just leave this blank and our maintainer will ping the right person. Please tag fewer than 3 people. 如果您能找到合适的标签 @，您的问题会更快得到回复。 所有问题都会由我们的维护者阅读，如果您不知道该标记谁，只需留空，我们的维护人员会找到合适的开发组成员来解决问题。 标记的人数应该不超过 3 个人。 If it's not a bug in these three subsections, you may not specify the helper. Our maintainer will find the right person in the development group to solve the problem. 如果不是这三个子版块的bug，您可以不指明帮助者，我们的维护人员会找到合适的开发组成员来解决问题。 placeholder: "@Username ..." - type: checkboxes id: information-scripts-examples attributes: label: Information / 问题信息 description: 'The problem arises when using: / 问题出现在' options: - label: "The official example scripts / 官方的示例脚本" - label: "My own modified scripts / 我自己修改的脚本和任务" - type: textarea id: reproduction validations: required: true attributes: label: Reproduction / 复现过程 description: | Please provide a code example that reproduces the problem you encountered, preferably with a minimal reproduction unit. If you have code snippets, error messages, stack traces, please provide them here as well. Please format your code correctly using code tags. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting Do not use screenshots, as they are difficult to read and (more importantly) do not allow others to copy and paste your code. 请提供能重现您遇到的问题的代码示例,最好是最小复现单元。 如果您有代码片段、错误信息、堆栈跟踪，也请在此提供。 请使用代码标签正确格式化您的代码。请参见 https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting 请勿使用截图，因为截图难以阅读，而且（更重要的是）不允许他人复制粘贴您的代码。 placeholder: | Steps to reproduce the behavior/复现Bug的步骤: 1. 2. 3. - type: textarea id: expected-behavior validations: required: true attributes: label: Expected behavior / 期待表现 description: "A clear and concise description of what you would expect to happen. /简单描述您期望发生的事情。" ================================================ FILE: .github/ISSUE_TEMPLATE/feature-request.yaml ================================================ name: "\U0001F680 Feature request" description: Submit a request for a new GLM-4-9B feature / 提交一个新的 GLM-4-9B 的功能建议 labels: [ "feature" ] body: - type: textarea id: feature-request validations: required: true attributes: label: Feature request / 功能建议 description: | A brief description of the functional proposal. Links to corresponding papers and code are desirable. 对功能建议的简述。最好提供对应的论文和代码链接 - type: textarea id: motivation validations: required: true attributes: label: Motivation / 动机 description: | Your motivation for making the suggestion. If that motivation is related to another GitHub issue, link to it here. 您提出建议的动机。如果该动机与另一个 GitHub 问题有关，请在此处提供对应的链接。 - type: textarea id: contribution validations: required: true attributes: label: Your contribution / 您的贡献 description: | Your PR link or any other link you can help with. 您的PR链接或者其他您能提供帮助的链接。 ================================================ FILE: .github/PULL_REQUEST_TEMPLATE.md ================================================ # Contribution Guide We welcome your contributions to this repository. To ensure elegant code style and better code quality, we have prepared the following contribution guidelines. ## What We Accept + This PR fixes a typo or improves the documentation (if this is the case, you may skip the other checks). + This PR fixes a specific issue — please reference the issue number in the PR description. Make sure your code strictly follows the coding standards below. + This PR introduces a new feature — please clearly explain the necessity and implementation of the feature. Make sure your code strictly follows the coding standards below. ## Code Style Guide Good code style is an art. We have prepared a `pyproject.toml` and a `pre-commit` hook to enforce consistent code formatting across the project. You can clean up your code following the steps below: 1. Install the required dependencies: ```shell pip install ruff pre-commit ``` 2. Then, run the following command: ```shell pre-commit run --all-files ``` If your code complies with the standards, you should not see any errors. ## Naming Conventions - Please use **English** for naming; do not use Pinyin or other languages. All comments should also be in English. - Follow **PEP8** naming conventions strictly, and use underscores to separate words. Avoid meaningless names such as `a`, `b`, `c`. ================================================ FILE: .gitignore ================================================ *venv *.DS_Store *.idea/ dataset test* ================================================ FILE: .pre-commit-config.yaml ================================================ repos: - repo: https://github.com/astral-sh/ruff-pre-commit rev: v0.4.5 hooks: - id: ruff args: [--fix, --respect-gitignore, --config=pyproject.toml] - id: ruff-format args: [--config=pyproject.toml] - repo: https://github.com/pre-commit/pre-commit-hooks rev: v4.5.0 hooks: - id: trailing-whitespace - id: end-of-file-fixer - id: check-yaml - id: check-toml - id: check-case-conflict - id: check-merge-conflict - id: debug-statements ================================================ FILE: LICENSE ================================================ Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 1. Definitions. "License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document. "Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License. "Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. "You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License. "Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files. "Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types. "Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below). "Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof. "Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution." "Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work. 2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form. 3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed. 4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions: (a) You must give any other recipients of the Work or Derivative Works a copy of this License; and (b) You must cause any modified files to carry prominent notices stating that You changed the files; and (c) You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and (d) If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License. You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License. 5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions. 6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file. 7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License. 8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages. 9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability. END OF TERMS AND CONDITIONS APPENDIX: How to apply the Apache License to your work. To apply the Apache License to your work, attach the following boilerplate notice, with the fields enclosed by brackets "[]" replaced with your own identifying information. (Don't include the brackets!) The text should be enclosed in the appropriate comment syntax for the file format. We also recommend that a file or class name and description of purpose be included on the same "printed page" as the copyright notice for easier identification within third-party archives. Copyright 2025 Zhipu AI Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. ================================================ FILE: README.md ================================================ # GLM-4-0414 Model Series

👋 Join our Discord, X and WeChat (Chinese)

📍The open-source models released this time can be experienced for free at Z.ai; for GLM commercial model services, please visit bigmodel.cn.

Read this in [中文](README_zh.md) ## Project Updates - 🔥 **News**: ```2025/07/02```: We are releasing the [GLM-4.1V-9B-Thinking](https://huggingface.co/collections/THUDM/glm-41v-thinking-6862bbfc44593a8601c2578d) series VLM, check [this github repo](https://github.com/THUDM/GLM-4.1V-Thinking) to get more information. - **News**: ```2025/04/14```: We are releasing the [GLM-4-32B-0414](https://huggingface.co/collections/THUDM/glm-4-0414-67f3cbcb34dd9d252707cb2e) series models, scaled up to 32B parameters, including models with capabilities for dialogue, reasoning, and rumination. - **News**: ``2024/06/18``: We have released our [Technical Report](https://arxiv.org/pdf/2406.12793), feel free to check it out. - **News**: ``2024/06/05``: We released the `GLM-4-9B` series of open-source models. Details can be found [here](README_20240605.md). ## Model Introduction The GLM family welcomes new members, the **GLM-4-32B-0414** series models, featuring 32 billion parameters. Its performance is comparable to OpenAI’s GPT series and DeepSeek’s V3/R1 series. It also supports very user-friendly local deployment features. GLM-4-32B-Base-0414 was pre-trained on 15T of high-quality data, including substantial reasoning-type synthetic data. This lays the foundation for subsequent reinforcement learning extensions. In the post-training stage, we employed human preference alignment for dialogue scenarios. Additionally, using techniques like rejection sampling and reinforcement learning, we enhanced the model’s performance in instruction following, engineering code, and function calling, thus strengthening the atomic capabilities required for agent tasks. GLM-4-32B-0414 achieves good results in engineering code, Artifact generation, function calling, search-based Q&A, and report generation. In particular, on several benchmarks, such as code generation or specific Q&A tasks, GLM-4-32B-Base-0414 achieves comparable performance with those larger models like GPT-4o and DeepSeek-V3-0324 (671B). **GLM-Z1-32B-0414** is a reasoning model with deep thinking capabilities. This was developed based on GLM-4-32B-0414 through cold start, extended reinforcement learning, and further training on tasks including mathematics, code, and logic. Compared to the base model, GLM-Z1-32B-0414 significantly improves mathematical abilities and the capability to solve complex tasks. During training, we also introduced general reinforcement learning based on pairwise ranking feedback, which enhances the model's general capabilities. **GLM-Z1-Rumination-32B-0414** is a deep reasoning model with rumination capabilities (against OpenAI's Deep Research). Unlike typical deep thinking models, the rumination model is capable of deeper and longer thinking to solve more open-ended and complex problems (e.g., writing a comparative analysis of AI development in two cities and their future development plans). Z1-Rumination is trained through scaling end-to-end reinforcement learning with responses graded by the ground truth answers or rubrics and can make use of search tools during its deep thinking process to handle complex tasks. The model shows significant improvements in research-style writing and complex tasks. Finally, **GLM-Z1-9B-0414** is a surprise. We employed all the aforementioned techniques to train a small model (9B). GLM-Z1-9B-0414 exhibits excellent capabilities in mathematical reasoning and general tasks. Its overall performance is top-ranked among all open-source models of the same size. Especially in resource-constrained scenarios, this model achieves an excellent balance between efficiency and effectiveness, providing a powerful option for users seeking lightweight deployment. ## Showcase ### Animation Generation
GLM-Z1-32B-0414 GLM-4-32B-0414
write a Python program that shows a ball bouncing inside a spinning hexagon. The ball should be affected by gravity and friction, and it must bounce off the rotating walls realistically
Use HTML to simulate the scenario of a small ball released from the center of a rotating hexagon. Consider the collision between the ball and the hexagon's edges, the gravity acting on the ball, and assume all collisions are perfectly elastic. (Prompt translated from Chinese)
### Web Design
GLM-4-32B-0414 GLM-4-32B-0414
Design a drawing board that supports custom function plotting, allowing adding and deleting custom functions, and assigning colors to functions. (Prompt translated from Chinese)
Design a UI for a mobile machine learning platform, which should include interfaces for training tasks, storage management, and personal statistics. The personal statistics interface should use charts to display the user's resource usage over a period. Use Tailwind CSS to style the page, and display these 3 mobile interfaces tiled on a single HTML page. (Prompt translated from Chinese)
### SVG Generation
GLM-4-32B-0414 GLM-4-32B-0414
Create a misty Jiangnan scene using SVG. (Prompt translated from Chinese)
Use SVG to illustrate the training process of an LLM. (Prompt translated from Chinese)
### Analysis and Research Report Writing
Analysis of AI Development in Chinese Cities: A Comparative Study of Beijing and Hangzhou, Alongside an Investigation of International Cases of AI in Urban Governance. (Prompt translated from Chinese)
## Model List ### GLM-4-0414 Series Models GLM-Z1-9B-0414 Open-Source Model [Try it Online](https://modelscope.cn/studios/ZhipuAI/GLM-Z1-9B-0414/summary) | Model | Type | Seq Length* | Download | |:--------------------------:|:---------:|:-----------:|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------:| | GLM-4-9B-0414 | Chat | 32K -> 128K | [🤗 Huggingface](https://huggingface.co/THUDM/GLM-4-9B-0414)
[🤖 ModelScope](https://modelscope.cn/models/ZhipuAI/GLM-4-9B-0414)
[🧩 Modelers](https://modelers.cn/models/zhipuai/GLM-4-9B-0414)
[🟣 WiseModel](https://wisemodel.cn/models/ZhipuAI/GLM-4-9B-0414) | | GLM-Z1-9B-0414 | Reasoning | 32K -> 128K | [🤗 Huggingface](https://huggingface.co/THUDM/GLM-4-Z1-9B-0414)
[🤖 ModelScope](https://modelscope.cn/models/ZhipuAI/GLM-Z1-9B-0414)
[🧩 Modelers](https://modelers.cn/models/zhipuai/GLM-Z1-9B-0414)
[🟣 WiseModel](https://wisemodel.cn/models/ZhipuAI/GLM-Z1-9B-0414) | | GLM-4-32B-Base-0414 | Base | 32K -> 128K | [🤗 Huggingface](https://huggingface.co/THUDM/GLM-4-32B-Base-0414)
[🤖 ModelScope](https://modelscope.cn/models/ZhipuAI/GLM-4-32B-Base-0414)
[🧩 Modelers](https://modelers.cn/models/zhipuai/GLM-4-32B-Base-0414)
[🟣 WiseModel](https://wisemodel.cn/models/ZhipuAI/GLM-4-32B-Base-0414) | | GLM-4-32B-0414 | Chat | 32K -> 128K | [🤗 Huggingface](https://huggingface.co/THUDM/GLM-4-32B-0414)
[🤖 ModelScope](https://modelscope.cn/models/ZhipuAI/GLM-4-32B-0414)
[🧩 Modelers](https://modelers.cn/models/zhipuai/GLM-4-32B-0414)
[🟣 WiseModel](https://wisemodel.cn/models/ZhipuAI/GLM-4-32B-Base-0414) | | GLM-Z1-32B-0414 | Reasoning | 32K -> 128K | [🤗 Huggingface](https://huggingface.co/THUDM/GLM-Z1-32B-0414)
[🤖 ModelScope](https://modelscope.cn/models/ZhipuAI/GLM-Z1-32B-0414)
[🧩 Modelers](https://modelers.cn/models/zhipuai/GLM-Z1-32B-0414)
[🟣 WiseModel](https://wisemodel.cn/models/ZhipuAI/GLM-Z1-32B-0414) | | GLM-Z1-Rumination-32B-0414 | Reasoning | 128K | [🤗 Huggingface](https://huggingface.co/THUDM/GLM-Z1-Rumination-32B-0414)
[🤖 ModelScope](https://modelscope.cn/models/ZhipuAI/GLM-Z1-Rumination-32B-0414)
[🧩 Modelers](https://modelers.cn/models/zhipuai/GLM-Z1-Rumination-32B-0414)
[🟣 WiseModel](https://wisemodel.cn/models/ZhipuAI/GLM-Z1-Rumination-32B-0414) | Due to its smaller model capacity, GLM-4-9B-0414 has not undergone the same agent capability enhancements as GLM-4-32B-0414. Instead, it has been optimized primarily for scenarios that require large-scale batch operations, such as translation tasks. \* Models are natively trained with a 32K context. For requests where the total input + output length might exceed 32K tokens, we recommend activating YaRN for better extrapolation performance. See the [Model and Prompt Implementation](#model-and-prompt-implementation) section for details. Below are the GLM-4 series models released on June 5, 2024. Details can be found [here](README_240605.md). | Model | Type | Seq Length* | Download | |:-----------------------------:|:---------:|:----------:|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------:| | GLM-4-9B | Base | 8K | [🤗 Huggingface](https://huggingface.co/THUDM/glm-4-9b)
[🤖 ModelScope](https://modelscope.cn/models/ZhipuAI/glm-4-9b)
| | GLM-4-9B-Chat | Chat | 128K | [🤗 Huggingface](https://huggingface.co/THUDM/glm-4-9b-chat)
[🤖 ModelScope](https://modelscope.cn/models/ZhipuAI/glm-4-9b-chat)
[🟣 WiseModel](https://wisemodel.cn/models/ZhipuAI/GLM-4-9B-Chat) | | GLM-4-9B-Chat-HF | Chat | 128K | [🤗 Huggingface](https://huggingface.co/THUDM/glm-4-9b-chat-hf)
[🤖 ModelScope](https://modelscope.cn/models/ZhipuAI/glm-4-9b-chat-hf) | | GLM-4-9B-Chat-1M | Chat | 1M | [🤗 Huggingface](https://huggingface.co/THUDM/glm-4-9b-chat-1m)
[🤖 ModelScope](https://modelscope.cn/models/ZhipuAI/glm-4-9b-chat-1m)
[🟣 WiseModel](https://wisemodel.cn/models/ZhipuAI/GLM-4-9B-Chat-1M) | | GLM-4-9B-Chat-1M-HF | Chat | 1M | [🤗 Huggingface](https://huggingface.co/THUDM/glm-4-9b-chat-1m-hf)
[🤖 ModelScope](https://modelscope.cn/models/ZhipuAI/glm-4-9b-chat-1m-hf) | | GLM-4V-9B | Chat | 8K | [🤗 Huggingface](https://huggingface.co/THUDM/glm-4v-9b)
[🤖 ModelScope](https://modelscope.cn/models/ZhipuAI/glm-4v-9b)
[🟣 WiseModel](https://wisemodel.cn/models/ZhipuAI/GLM-4V-9B) | ## Evaluation Results ### GLM-4-0414 Series
| Model | IFEval | BFCL-v3 (Overall) | BFCL-v3 (MultiTurn) | TAU-Bench (Retail) | TAU-Bench (Airline) | SimpleQA | HotpotQA | | ---------------- | ------ | ----------------- | ------------------- | ------------------ | ------------------- | -------- | -------- | | Qwen2.5-Max | 85.6 | 50.9 | 30.5 | 58.3 | 22.0 | 79.0 | 52.8 | | GPT-4o-1120 | 81.9 | 69.6 | 41.0 | 62.8 | 46.0 | 82.8 | 63.9 | | DeepSeek-V3-0324 | 83.4 | 66.2 | 35.8 | 60.7 | 32.4 | 82.6 | 54.6 | | DeepSeek-R1 | 84.3 | 57.5 | 12.4 | 33.0 | 37.3 | 83.9 | 63.1 | | GLM-4-32B-0414 | 87.6 | 69.6 | 41.5 | 68.7 | 51.2 | 88.1 | 63.8 | > For `SimpleQA` and `HotpotQA`, we sampled nearly 500 test cases from each test set, provided all models with basic `search` and `click` tools, ensured other settings remained consistent, and averaged the results over 3 runs. | Model | Framework | [SWE-bench Verified](https://openai.com/index/introducing-swe-bench-verified/) | [SWE-bench Verified mini](https://github.com/mariushobbhahn/SWEBench-verified-mini) | |---|---|---|---| | GLM-4-32B-0414 | Moatless[1] | 33.8 | 38.0 | | GLM-4-32B-0414 | Agentless[2] | 30.7 | 34.0 | | GLM-4-32B-0414 | OpenHands[3] | 27.2 | 28.0 | [1] [Moatless v0.0.3](https://github.com/aorwall/moatless-tools) used the following parameters: `response_format="react", thoughts_in_action=False, max_interations=30`. No retries on failed trajectories; other settings are default. [2] [Agentless v1.5.0](https://github.com/OpenAutoCoder/Agentless) used [BGE](https://github.com/FlagOpen/FlagEmbedding/blob/master/README.md) as the embedding model and [FAISS](https://github.com/facebookresearch/faiss) for similarity search. To speed up patch verification while maintaining performance, the timeout for running a single instance was changed from the default 300s to 180s. [3] [OpenHands v0.29.1](https://github.com/All-Hands-AI/OpenHands/tree/main) did not use YaRN context extension but limited runs to a maximum of 60 iterations and summarized the history to prevent exceeding the 32K context limit. Summarization was configured as `llm_config="condenser", keep_first=1, max_size=32`. No retries on failed trajectories. ### GLM-Z1-0414 Series
## Model and Prompt Implementation ### Model Implementation If you want to see our model implementation, please check the Pull Requests in the relevant repositories, which have been merged: + [vLLM Model Implementation](https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/models/glm4.py) + [transformers Model Implementation](https://github.com/huggingface/transformers/blob/main/src/transformers/models/glm4/modeling_glm4.py) + [llama.cpp Model Implementation](https://github.com/ggml-org/llama.cpp/pull/12867) ### Handling Long Context (YaRN) If the total input + output token count might exceed the model's native context length (mostly 32k for the GLM-4-0414 series), it is recommended to enable YaRN to achieve better long-context modeling capabilities. For supported frameworks, you can modify the corresponding `config.json`. Specifically, for GLM-Z1 series models, consider enabling YaRN (Rope Scaling) when the input length exceeds **8,192 tokens**. ```json "rope_scaling": { "factor": 4.0, "original_max_position_embeddings": 32768, "type": "yarn" } ``` For most user requests, if the input + output token count does not exceed the native context length, no modifications are needed. ### Model Fine-tuning You can find information about the computational resources required for model fine-tuning, as well as example fine-tuning scripts, in `finetune/README.md`. To start a simple model fine-tuning example, run the following commands: ```shell cd finetune pip install -r ../inference/requirements.txt pip install -r requirements.txt # Use single GPU for Chat Fine-tune python finetune.py data/AdvertiseGen/ THUDM/GLM-4-9B-0414 configs/lora.yaml ``` 🎉 The script also supports fine-tuning with visual tracking using **SwanLab**. You can view the training logs of the example fine-tuning script on the [SwanLab Visualization Dashboard](https://swanlab.cn/@ShaohonChen/GLM4-Finetune/overview). ### Prompt Implementation If you use the `apply_chat_template` method provided by the `transformers` library to construct prompts, here are the restrictions on `System Prompts` for different GLM-4-0414 models. + `GLM-4-32B-Base-0414`: Base model, no chat template. + `GLM-4-*-0414` / `GLM-Z1-*-0414`: If `tools` are provided, `apply_chat_template` will populate the tools into a fixed template within the `chat_template`, creating a separate `system` message with tool bindings prepended to the message list (`messages[0]`). All originally passed `messages` are automatically shifted one position back. + `GLM-Z1-Rumination-32B-0414`: + Does not support custom system prompts or custom tools. Your `tools` and `system` fields will be ignored by `apply_chat_template`. Using this model requires an external search engine or a custom retrieval API. + Supports four tools in total: ``` 1. search Description: Executes a search query and returns search results. Use this when you need to find information about a specific topic. Parameters: query (string) - The search query string. Use English words unless it's a Chinese proper noun. 2. click Description: Clicks on a link from the search results and navigates to the corresponding page. Use this when you need to view the detailed content of a specific search result. Parameters: link_id (integer) - The ID of the link to click (from the sequence number in the search results). 3. open Description: Opens a specific website. Gets the content of any website via URL. Parameters: url (string) - The target website URL or domain name. 4. finish Description: Completes the task. Use this when you have found the required information. Parameters: None ``` + The fixed template in `chat_template` uses English for the thought process. If you want to change to another language, you need to modify the following section (currently supports Chinese and English): ``` - Language Used * Search Keywords: English -> Change here to "Chinese" or another language * Thinking: English -> Change here to "Chinese" or another language ``` To see the specific chat templates for the GLM-4-0414 series models, please check the `chat_template.jinja` file in the corresponding model repository. ## Citation If you find our work helpful, please consider citing the following paper. ```bibtex @misc{glm2024chatglm, title={ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools}, author={Team GLM and Aohan Zeng and Bin Xu and Bowen Wang and Chenhui Zhang and Da Yin and Diego Rojas and Guanyu Feng and Hanlin Zhao and Hanyu Lai and Hao Yu and Hongning Wang and Jiadai Sun and Jiajie Zhang and Jiale Cheng and Jiayi Gui and Jie Tang and Jing Zhang and Juanzi Li and Lei Zhao and Lindong Wu and Lucen Zhong and Mingdao Liu and Minlie Huang and Peng Zhang and Qinkai Zheng and Rui Lu and Shuaiqi Duan and Shudan Zhang and Shulin Cao and Shuxun Yang and Weng Lam Tam and Wenyi Zhao and Xiao Liu and Xiao Xia and Xiaohan Zhang and Xiaotao Gu and Xin Lv and Xinghan Liu and Xinyi Liu and Xinyue Yang and Xixuan Song and Xunkai Zhang and Yifan An and Yifan Xu and Yilin Niu and Yuantao Yang and Yueyan Li and Yushi Bai and Yuxiao Dong and Zehan Qi and Zhaoyu Wang and Zhen Yang and Zhengxiao Du and Zhenyu Hou and Zihan Wang}, year={2024}, eprint={2406.12793}, archivePrefix={arXiv}, primaryClass={id='cs.CL' full_name='Computation and Language' is_active=True alt_name='cmp-lg' in_archive='cs' is_general=False description='Covers natural language processing. Roughly includes material in ACM Subject Class I.2.7. Note that work on artificial languages (programming languages, logics, formal systems) that does not explicitly address natural-language issues broadly construed (natural-language processing, computational linguistics, speech, text retrieval, etc.) is not appropriate for this area.'} } ``` ================================================ FILE: README_20240605.md ================================================ # GLM-4

📄 Report • 🤗 HF Repo • 🤖 ModelScope • 🟣 WiseModel • 🐦 Twitter • 👋 Join Discord and WeChat

📍Experience and use a larger-scale GLM business model on the Zhipu AI Open Platform

## Update - 🔥🔥 **News**: ```2024/11/01```: Dependencies have been updated in this repository. Please update the dependencies in `requirements.txt` to ensure the model runs correctly. The model weights for [glm-4-9b-chat-hf](https://huggingface.co/THUDM/glm-4-9b-chat-hf) are compatible with `transformers>=4.46.2` and can be implemented using the `GlmModel` class in the `transformers` library. Additionally, `tokenizer_chatglm.py` in [glm-4-9b-chat](https://huggingface.co/THUDM/glm-4-9b-chat) and [glm-4v-9b](https://huggingface.co/THUDM/glm-4v-9b) has been updated for the latest version of `transformers`. Please update the files on HuggingFace. - 🔥 **News**: ```2024/10/27```: We have open-sourced [LongReward](https://github.com/THUDM/LongReward), a model that uses AI feedback to enhance long-context large language models. - 🔥 **News**: ```2024/10/25```: We have open-sourced the end-to-end Mandarin-English voice dialogue model [GLM-4-Voice](https://github.com/THUDM/GLM-4-Voice). - 🔥 **News**: ```2024/09/05```: We have open-sourced [longcite-glm4-9b](https://huggingface.co/THUDM/LongCite-glm4-9b), a model enabling LLMs to produce fine-grained citations in long-context Q&A, along with the dataset [LongCite-45k](https://huggingface.co/datasets/THUDM/LongCite-45k). Try it out online at [Huggingface Space](https://huggingface.co/spaces/THUDM/LongCite). - 🔥 **News**: ```2024/08/15```: We have open-sourced [longwriter-glm4-9b](https://huggingface.co/THUDM/LongWriter-glm4-9b), a model capable of generating over 10,000 tokens in single-turn dialogue, along with the dataset [LongWriter-6k](https://huggingface.co/datasets/THUDM/LongWriter-6k). Experience it online at [Huggingface Space](https://huggingface.co/spaces/THUDM/LongWriter) or the [ModelScope Community Space](https://modelscope.cn/studios/ZhipuAI/LongWriter-glm4-9b-demo). - 🔥 **News**: ```2024/07/24```: We published the latest technical insights on long-text processing. Check out our technical report on training the open-source GLM-4-9B model for long texts [here](https://medium.com/@ChatGLM/glm-long-scaling-pre-trained-model-contexts-to-millions-caa3c48dea85). - 🔥 **News**: ```2024/07/09```: The GLM-4-9B-Chat model is now compatible with [Ollama](https://github.com/ollama/ollama) and [Llama.cpp](https://github.com/ggerganov/llama.cpp). See detailed information in this [PR](https://github.com/ggerganov/llama.cpp/pull/8031). - 🔥 **News**: ```2024/06/18```: We have released a [technical report](https://arxiv.org/pdf/2406.12793), available for viewing. - 🔥 **News**: ```2024/06/05```: We released the GLM-4-9B series of open-source models. ## Model Introduction GLM-4-9B is the open-source version of the latest generation of pre-trained models in the GLM-4 series launched by Zhipu AI. In the evaluation of data sets in semantics, mathematics, reasoning, code, and knowledge, **GLM-4-9B** and its human preference-aligned version **GLM-4-9B-Chat** have shown superior performance beyond Llama-3-8B. In addition to multi-round conversations, GLM-4-9B-Chat also has advanced features such as web browsing, code execution, custom tool calls (Function Call), and long text reasoning (supporting up to 128K context). This generation of models has added multi-language support, supporting 26 languages including Japanese, Korean, and German. We have also launched the **GLM-4-9B-Chat-1M** model that supports 1M context length (about 2 million Chinese characters) and the multimodal model GLM-4V-9B based on GLM-4-9B. **GLM-4V-9B** possesses dialogue capabilities in both Chinese and English at a high resolution of 1120*1120. In various multimodal evaluations, including comprehensive abilities in Chinese and English, perception & reasoning, text recognition, and chart understanding, GLM-4V-9B demonstrates superior performance compared to GPT-4-turbo-2024-04-09, Gemini 1.0 Pro, Qwen-VL-Max, and Claude 3 Opus. ## Model List | Model | Type | Seq Length | Transformers Version | Download | Online Demo | |:-------------------:|:----:|:----------:|:--------------------:|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------:|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------:| | GLM-4-9B | Base | 8K | `4.44.0 - 4.45.0` | [🤗 Huggingface](https://huggingface.co/THUDM/glm-4-9b)
[🤖 ModelScope](https://modelscope.cn/models/ZhipuAI/glm-4-9b)
[🟣 WiseModel](https://wisemodel.cn/models/ZhipuAI/glm-4-9b) | / | | GLM-4-9B-Chat | Chat | 128K | `>= 4.44.0` | [🤗 Huggingface](https://huggingface.co/THUDM/glm-4-9b-chat)
[🤖 ModelScope](https://modelscope.cn/models/ZhipuAI/glm-4-9b-chat)
[🟣 WiseModel](https://wisemodel.cn/models/ZhipuAI/GLM-4-9B-Chat) | [🤖 ModelScope CPU](https://modelscope.cn/studios/dash-infer/GLM-4-Chat-DashInfer-Demo/summary)
[🤖 ModelScope vLLM](https://modelscope.cn/studios/ZhipuAI/glm-4-9b-chat-vllm/summary) | | GLM-4-9B-Chat-HF | Chat | 128K | `>= 4.46.0` | [🤗 Huggingface](https://huggingface.co/THUDM/glm-4-9b-chat-hf)
[🤖 ModelScope](https://modelscope.cn/models/ZhipuAI/glm-4-9b-chat-hf) | [🤖 ModelScope CPU](https://modelscope.cn/studios/dash-infer/GLM-4-Chat-DashInfer-Demo/summary)
[🤖 ModelScope vLLM](https://modelscope.cn/studios/ZhipuAI/glm-4-9b-chat-vllm/summary) | | GLM-4-9B-Chat-1M | Chat | 1M | `>= 4.44.0` | [🤗 Huggingface](https://huggingface.co/THUDM/glm-4-9b-chat-1m)
[🤖 ModelScope](https://modelscope.cn/models/ZhipuAI/glm-4-9b-chat-1m)
[🟣 WiseModel](https://wisemodel.cn/models/ZhipuAI/GLM-4-9B-Chat-1M) | / | | GLM-4-9B-Chat-1M-HF | Chat | 1M | `>= 4.46.0` | [🤗 Huggingface](https://huggingface.co/THUDM/glm-4-9b-chat-1m-hf)
[🤖 ModelScope](https://modelscope.cn/models/ZhipuAI/glm-4-9b-chat-1m-hf) | / | | GLM-4V-9B | Chat | 8K | `>= 4.46.0` | [🤗 Huggingface](https://huggingface.co/THUDM/glm-4v-9b)
[🤖 ModelScope](https://modelscope.cn/models/ZhipuAI/glm-4v-9b)
[🟣 WiseModel](https://wisemodel.cn/models/ZhipuAI/GLM-4V-9B) | [🤖 ModelScope](https://modelscope.cn/studios/ZhipuAI/glm-4v-9b-Demo/summary) | ## BenchMarkß ### Typical Tasks | Model | AlignBench | MT-Bench | IFEval | MMLU | C-Eval | GSM8K | MATH | HumanEval | NaturalCodeBench | |:--------------------|:----------:|:--------:|:------:|:----:|:------:|:-----:|:----:|:---------:|:----------------:| | Llama-3-8B-Instruct | 6.40 | 8.00 | 68.58 | 68.4 | 51.3 | 79.6 | 30.0 | 62.2 | 24.7 | | ChatGLM3-6B | 5.18 | 5.50 | 28.1 | 66.4 | 69.0 | 72.3 | 25.7 | 58.5 | 11.3 | | GLM-4-9B-Chat | 7.01 | 8.35 | 69.0 | 72.4 | 75.6 | 79.6 | 50.6 | 71.8 | 32.2 | ### Base Model | Model | MMLU | C-Eval | GPQA | GSM8K | MATH | HumanEval | |:--------------------|:----:|:------:|:----:|:-----:|:----:|:---------:| | Llama-3-8B | 66.6 | 51.2 | - | 45.8 | - | 33.5 | | Llama-3-8B-Instruct | 68.4 | 51.3 | 34.2 | 79.6 | 30.0 | 62.2 | | ChatGLM3-6B-Base | 61.4 | 69.0 | 26.8 | 72.3 | 25.7 | 58.5 | | GLM-4-9B | 74.7 | 77.1 | 34.3 | 84.0 | 30.4 | 70.1 | > Since `GLM-4-9B` adds some math, reasoning, and code-related instruction data during pre-training, Llama-3-8B-Instruct > is also included in the comparison range. ### Long Context The [needle-in-the-haystack experiment](https://github.com/LargeWorldModel/LWM/blob/main/scripts/eval_needle.py) was conducted with a context length of 1M, and the results are as follows: ![needle](resources/eval_needle.jpeg) The long text capability was further evaluated on LongBench-Chat, and the results are as follows:

Description text

### Multi Language The tests for GLM-4-9B-Chat and Llama-3-8B-Instruct are conducted on six multilingual datasets. The test results and the corresponding languages selected for each dataset are shown in the table below: | Dataset | Llama-3-8B-Instruct | GLM-4-9B-Chat | Languages | |:------------|:-------------------:|:-------------:|:----------------------------------------------------------------------------------------------:| | M-MMLU | 49.6 | 56.6 | all | | FLORES | 25.0 | 28.8 | ru, es, de, fr, it, pt, pl, ja, nl, ar, tr, cs, vi, fa, hu, el, ro, sv, uk, fi, ko, da, bg, no | | MGSM | 54.0 | 65.3 | zh, en, bn, de, es, fr, ja, ru, sw, te, th | | XWinograd | 61.7 | 73.1 | zh, en, fr, jp, ru, pt | | XStoryCloze | 84.7 | 90.7 | zh, en, ar, es, eu, hi, id, my, ru, sw, te | | XCOPA | 73.3 | 80.1 | zh, et, ht, id, it, qu, sw, ta, th, tr, vi | ### Function Call Tested on [Berkeley Function Calling Leaderboard](https://github.com/ShishirPatil/gorilla/tree/main/berkeley-function-call-leaderboard). | Model | Overall Acc. | AST Summary | Exec Summary | Relevance | |:-----------------------|:------------:|:-----------:|:------------:|:---------:| | Llama-3-8B-Instruct | 58.88 | 59.25 | 70.01 | 45.83 | | gpt-4-turbo-2024-04-09 | 81.24 | 82.14 | 78.61 | 88.75 | | ChatGLM3-6B | 57.88 | 62.18 | 69.78 | 5.42 | | GLM-4-9B-Chat | 81.00 | 80.26 | 84.40 | 87.92 | ### Multi-Modal GLM-4V-9B is a multimodal language model with visual understanding capabilities. The evaluation results of its related classic tasks are as follows: | | **MMBench-EN-Test** | **MMBench-CN-Test** | **SEEDBench_IMG** | **MMStar** | **MMMU** | **MME** | **HallusionBench** | **AI2D** | **OCRBench** | |----------------------------|---------------------|---------------------|-------------------|------------|----------|---------|--------------------|----------|--------------| | **gpt-4o-2024-05-13** | 83.4 | 82.1 | 77.1 | 63.9 | 69.2 | 2310.3 | 55 | 84.6 | 736 | | **gpt-4-turbo-2024-04-09** | 81.0 | 80.2 | 73.0 | 56.0 | 61.7 | 2070.2 | 43.9 | 78.6 | 656 | | **gpt-4-1106-preview** | 77.0 | 74.4 | 72.3 | 49.7 | 53.8 | 1771.5 | 46.5 | 75.9 | 516 | | **InternVL-Chat-V1.5** | 82.3 | 80.7 | 75.2 | 57.1 | 46.8 | 2189.6 | 47.4 | 80.6 | 720 | | **LLaVA-Next-Yi-34B** | 81.1 | 79 | 75.7 | 51.6 | 48.8 | 2050.2 | 34.8 | 78.9 | 574 | | **Step-1V** | 80.7 | 79.9 | 70.3 | 50.0 | 49.9 | 2206.4 | 48.4 | 79.2 | 625 | | **MiniCPM-Llama3-V2.5** | 77.6 | 73.8 | 72.3 | 51.8 | 45.8 | 2024.6 | 42.4 | 78.4 | 725 | | **Qwen-VL-Max** | 77.6 | 75.7 | 72.7 | 49.5 | 52 | 2281.7 | 41.2 | 75.7 | 684 | | **Gemini 1.0 Pro** | 73.6 | 74.3 | 70.7 | 38.6 | 49 | 2148.9 | 45.7 | 72.9 | 680 | | **Claude 3 Opus** | 63.3 | 59.2 | 64 | 45.7 | 54.9 | 1586.8 | 37.8 | 70.6 | 694 | | **GLM-4V-9B** | 81.1 | 79.4 | 76.8 | 58.7 | 47.2 | 2163.8 | 46.6 | 81.1 | 786 | ## Quick call **For hardware configuration and system requirements, please check [here](basic_demo/README_en.md).** ### Use the following method to quickly call the GLM-4-9B-Chat language model Use the transformers backend for inference: ```python import torch from transformers import AutoModelForCausalLM, AutoTokenizer import os os.environ[ 'CUDA_VISIBLE_DEVICES'] = '0' # Set the GPU number. If inference with multiple GPUs, set multiple GPU numbers MODEL_PATH = "THUDM/glm-4-9b-chat-hf" device = "cuda" if torch.cuda.is_available() else "cpu" tokenizer = AutoTokenizer.from_pretrained(MODEL_PATH, trust_remote_code=True) query = "你好" inputs = tokenizer.apply_chat_template([{"role": "user", "content": query}], add_generation_prompt=True, tokenize=True, return_tensors="pt", return_dict=True ) inputs = inputs.to(device) model = AutoModelForCausalLM.from_pretrained( MODEL_PATH, torch_dtype=torch.bfloat16, low_cpu_mem_usage=True, trust_remote_code=True, device_map="auto" ).eval() gen_kwargs = {"max_length": 2500, "do_sample": True, "top_k": 1} with torch.no_grad(): outputs = model.generate(**inputs, **gen_kwargs) outputs = outputs[:, inputs['input_ids'].shape[1]:] print(tokenizer.decode(outputs[0], skip_special_tokens=True)) ``` Use the vLLM backend for inference: ```python from transformers import AutoTokenizer from vllm import LLM, SamplingParams # GLM-4-9B-Chat # If you encounter OOM, you can try to reduce max_model_len or increase tp_size max_model_len, tp_size = 131072, 1 model_name = "THUDM/glm-4-9b-chat-hf" prompt = [{"role": "user", "content": "你好"}] tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True) llm = LLM( model=model_name, tensor_parallel_size=tp_size, max_model_len=max_model_len, trust_remote_code=True, enforce_eager=True, # if you encounter OOM in GLM-4-9B-Chat-1M, you can try to enable the following parameters # enable_chunked_prefill=True, # max_num_batched_tokens=8192 ) stop_token_ids = [151329, 151336, 151338] sampling_params = SamplingParams(temperature=0.95, max_tokens=1024, stop_token_ids=stop_token_ids) inputs = tokenizer.apply_chat_template(prompt, tokenize=False, add_generation_prompt=True) outputs = llm.generate(prompts=inputs, sampling_params=sampling_params) print(outputs[0].outputs[0].text) ``` ### Use the following method to quickly call the GLM-4V-9B multimodal model Use the transformers backend for inference: ```python import torch from PIL import Image from transformers import AutoModelForCausalLM, AutoTokenizer import os os.environ[ 'CUDA_VISIBLE_DEVICES'] = '0' # Set the GPU number. If inference with multiple GPUs, set multiple GPU numbers MODEL_PATH = "THUDM/glm-4v-9b" device = "cuda" if torch.cuda.is_available() else "cpu" tokenizer = AutoTokenizer.from_pretrained(MODEL_PATH, trust_remote_code=True) query = '描述这张图片' image = Image.open("your image").convert('RGB') inputs = tokenizer.apply_chat_template([{"role": "user", "image": image, "content": query}], add_generation_prompt=True, tokenize=True, return_tensors="pt", return_dict=True) # chat mode inputs = inputs.to(device) model = AutoModelForCausalLM.from_pretrained( MODEL_PATH, torch_dtype=torch.bfloat16, low_cpu_mem_usage=True, trust_remote_code=True, device_map="auto" ).eval() gen_kwargs = {"max_length": 2500, "do_sample": True, "top_k": 1} with torch.no_grad(): outputs = model.generate(**inputs, **gen_kwargs) outputs = outputs[:, inputs['input_ids'].shape[1]:] print(tokenizer.decode(outputs[0])) ``` Use the vLLM backend for inference: ```python from PIL import Image from vllm import LLM, SamplingParams model_name = "THUDM/glm-4v-9b" llm = LLM(model=model_name, tensor_parallel_size=1, max_model_len=8192, trust_remote_code=True, enforce_eager=True) stop_token_ids = [151329, 151336, 151338] sampling_params = SamplingParams(temperature=0.2, max_tokens=1024, stop_token_ids=stop_token_ids) prompt = "What's the content of the image?" image = Image.open("your image").convert('RGB') inputs = { "prompt": prompt, "multi_modal_data": { "image": image }, } outputs = llm.generate(inputs, sampling_params=sampling_params) for o in outputs: generated_text = o.outputs[0].text print(generated_text) ``` ## Complete project list If you want to learn more about the GLM-4-9B series open source models, this open source repository provides developers with basic GLM-4-9B usage and development code through the following content + [basic_demo](basic_demo/README.md): Contains + Interaction code using transformers and vLLM backend + OpenAI API backend interaction code + Batch reasoning code + [composite_demo](composite_demo/README.md): Contains + Fully functional demonstration code for GLM-4-9B and GLM-4V-9B open source models, including All Tools capabilities, long document interpretation, and multimodal capabilities. + [fintune_demo](finetune_demo/README.md): Contains + PEFT (LORA, P-Tuning) fine-tuning code + SFT fine-tuning code + [intel_device_demo](intel_device_demo/): Contains + OpenVINO deployment code + Intel® Extension for Transformers deployment code ## Friendly Links + [LLaMA-Factory](https://github.com/hiyouga/LLaMA-Factory): Efficient open-source fine-tuning framework, already supports GLM-4-9B-Chat language model fine-tuning. + [SWIFT](https://github.com/modelscope/swift): LLM/VLM training framework from ModelScope, supports GLM-4-9B-Chat / GLM-4V-9b fine-tuning. + [Xorbits Inference](https://github.com/xorbitsai/inference): Performance-enhanced and comprehensive global inference framework, easily deploy your own models or import cutting-edge open source models with one click. + [LangChain-ChatChat](https://github.com/chatchat-space/Langchain-Chatchat): RAG and Agent applications based on language models such as Langchain and ChatGLM + [self-llm](https://github.com/datawhalechina/self-llm/tree/master/models/GLM-4): Datawhale's self-llm project, which includes the GLM-4-9B open source model cookbook. + [chatglm.cpp](https://github.com/li-plus/chatglm.cpp): Real-time inference on your laptop accelerated by quantization, similar to llama.cpp. + [OpenVINO](https://github.com/openvinotoolkit): glm-4-9b-chat already supports the use of OpenVINO. The toolkit accelerates inference and has a greater inference speed improvement on Intel's GPU, GPU and NPU devices. For specific usage, please refer to [OpenVINO notebooks](https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/llm-chatbot/llm-chatbot-generate-api.ipynb) ## License + The use of GLM-4 model weights must follow the [Model License](https://huggingface.co/THUDM/glm-4-9b/blob/main/LICENSE). + The code in this open source repository follows the [Apache 2.0](LICENSE) license. Please strictly follow the open source license. ## Reference If you find our work helpful, please consider citing the following paper. ``` @misc{glm2024chatglm, title={ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools}, author={Team GLM and Aohan Zeng and Bin Xu and Bowen Wang and Chenhui Zhang and Da Yin and Diego Rojas and Guanyu Feng and Hanlin Zhao and Hanyu Lai and Hao Yu and Hongning Wang and Jiadai Sun and Jiajie Zhang and Jiale Cheng and Jiayi Gui and Jie Tang and Jing Zhang and Juanzi Li and Lei Zhao and Lindong Wu and Lucen Zhong and Mingdao Liu and Minlie Huang and Peng Zhang and Qinkai Zheng and Rui Lu and Shuaiqi Duan and Shudan Zhang and Shulin Cao and Shuxun Yang and Weng Lam Tam and Wenyi Zhao and Xiao Liu and Xiao Xia and Xiaohan Zhang and Xiaotao Gu and Xin Lv and Xinghan Liu and Xinyi Liu and Xinyue Yang and Xixuan Song and Xunkai Zhang and Yifan An and Yifan Xu and Yilin Niu and Yuantao Yang and Yueyan Li and Yushi Bai and Yuxiao Dong and Zehan Qi and Zhaoyu Wang and Zhen Yang and Zhengxiao Du and Zhenyu Hou and Zihan Wang}, year={2024}, eprint={2406.12793}, archivePrefix={arXiv}, primaryClass={id='cs.CL' full_name='Computation and Language' is_active=True alt_name='cmp-lg' in_archive='cs' is_general=False description='Covers natural language processing. Roughly includes material in ACM Subject Class I.2.7. Note that work on artificial languages (programming languages, logics, formal systems) that does not explicitly address natural-language issues broadly construed (natural-language processing, computational linguistics, speech, text retrieval, etc.) is not appropriate for this area.'} } ``` ``` @misc{wang2023cogvlm, title={CogVLM: Visual Expert for Pretrained Language Models}, author={Weihan Wang and Qingsong Lv and Wenmeng Yu and Wenyi Hong and Ji Qi and Yan Wang and Junhui Ji and Zhuoyi Yang and Lei Zhao and Xixuan Song and Jiazheng Xu and Bin Xu and Juanzi Li and Yuxiao Dong and Ming Ding and Jie Tang}, year={2023}, eprint={2311.03079}, archivePrefix={arXiv}, primaryClass={cs.CV} } ``` ================================================ FILE: README_zh.md ================================================ # GLM-4-0414 系列模型

👋 加入我们的 Discord, X 和 微信

📍本次开源模型可以在 Z.ai 免费体验；使用 GLM 商业模型服务请到 bigmodel.cn。

Read this in [English](README) ## 项目更新 - 🔥 **News**：`2025/07/02`：我们正式发布 [GLM-4.1V-9B-Thinking](https://huggingface.co/collections/THUDM/glm-41v-thinking-6862bbfc44593a8601c2578d) 系列视觉理解模型，更多信息请查看 [GitHub 仓库](https://github.com/THUDM/GLM-4.1V-Thinking)。 - **News**: ```2025/04/14```: 我们发布 [GLM-4-32B-0414](https://huggingface.co/collections/THUDM/glm-4-0414-67f3cbcb34dd9d252707cb2e) 系列模型，规模提升至 32B，包含对话、推理、沉思多种能力的模型。 - **News**: ``2024/06/18``: 我们发布 [技术报告](https://arxiv.org/pdf/2406.12793), 欢迎查看。 - **News**: ``2024/06/05``: 我们发布 `GLM-4-9B` 系列开源模型，其内容可以在[这里](README_240605.md)查看。 ## 模型介绍 GLM 家族迎来新一代开源模型 **GLM-4-32B-0414** 系列，320 亿参数，效果比肩 OpenAI 的 GPT 系列和 DeepSeek 的 V3/R1 系列，且支持非常友好的本地部署特性。GLM-4-32B-Base-0414 经过 15T 高质量数据的预训练，其中包含大量推理类的合成数据，这为后续的强化学习扩展打下了基础。在后训练阶段，除了针对对话场景进行了人类偏好对齐外，我们还通过拒绝采样和强化学习等技术强化了模型在指令遵循、工程代码、函数调用方面的效果，加强了智能体任务所需的原子能力。GLM-4-32B-0414 在工程代码、Artifacts 生成、函数调用、搜索问答及报告等方面都取得了不错的效果，部分 Benchmark 甚至可以媲美更大规模的 GPT-4o、DeepSeek-V3-0324（671B）等模型。 **GLM-Z1-32B-0414** 是具有**深度思考能力**的推理模型，这是在 GLM-4-32B-0414 的基础上，通过冷启动和扩展强化学习，以及在数学、代码和逻辑等任务上对模型的进一步训练得到的。相对于基础模型，GLM-Z1-32B-0414 显著提升了数理能力和解决复杂任务的能力。在训练的过程中，我们还引入了基于对战排序反馈的通用强化学习，进一步增强了模型的通用能力。 **GLM-Z1-Rumination-32B-0414** 是具有**沉思能力**的深度推理模型（对标 Open AI 的 Deep Research）。不同于一般的深度思考模型，沉思模型通过更长时间的深度思考来解决更开放和复杂的问题（例如：撰写两个城市AI发展对比情况，以及未来的发展规划），沉思模型在深度思考过程中结合搜索工具处理复杂任务，并经过利用多种规则型奖励来指导和扩展端到端强化学习训练得到。Z1-Rumination 在研究型写作和复杂检索任务上的能力得到了显著提升。 最后，**GLM-Z1-9B-0414** 是一个惊喜。我们沿用上述一系列技术，训练了一个保持开源传统的 9B 小尺寸模型。尽管规模更小，GLM-Z1-9B-0414 在数学推理和通用任务中依然展现出极为优秀的能力，其整体表现已处于同尺寸开源模型中的领先水平。特别是在资源受限的场景下，该模型在效率与效果之间实现了出色的平衡，为追求轻量化部署的用户提供了强有力的选择。 ## 效果展示 ### 动画绘制
GLM-Z1-32B-0414 GLM-4-32B-0414
write a Python program that shows a ball bouncing inside a spinning hexagon. The ball should be affected by gravity and friction, and it must bounce off the rotating walls realistically
用 HTML 模拟一个小球在从一个旋转中的六边形中心释放后的场景。考虑小球和六边形边框的碰撞和小球受到的重力，并假设碰撞都是完全弹性碰撞
### 网页设计
GLM-4-32B-0414 GLM-4-32B-0414
设计一个支持自定义函数绘制的绘图板，可以添加和删除自定义函数，并为函数指定颜色
给我设计一个移动端机器学习平台的 UI，其中要包括训练任务，存储管理，和个人统计信息界面。个人信息统计界面要用图表展示用户过去一段时间的各类资源使用情况。使用 Tailwind CSS 来美化页面，把这 3 个手机界面平铺展示到一个 HTML 页面中
### SVG 生成
GLM-4-32B-0414 GLM-4-32B-0414
用SVG创作一幅烟雨江南
用 SVG 展示一个 LLM 的训练流程
### 分析调研撰写
中国城市 AI 发展分析：北京与杭州的对比研究。同时调研国外城市用 AI 进行城市治理的案例。
## 模型列表 ### GLM-4-0414 系列模型 GLM-Z1-9B-0414 开源模型 [在线体验](https://modelscope.cn/studios/ZhipuAI/GLM-Z1-9B-0414/summary) | Model | Type | Seq Length* | Download | |:--------------------------:|:---------:|:-----------:|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------:| | GLM-4-9B-0414 | Chat | 32K -> 128K | [🤗 Huggingface](https://huggingface.co/THUDM/GLM-4-9B-0414)
[🤖 ModelScope](https://modelscope.cn/models/ZhipuAI/GLM-4-9B-0414)
[🧩 Modelers](https://modelers.cn/models/zhipuai/GLM-4-9B-0414)
[🟣 WiseModel](https://wisemodel.cn/models/ZhipuAI/GLM-4-9B-0414) | | GLM-Z1-9B-0414 | Reasoning | 32K -> 128K | [🤗 Huggingface](https://huggingface.co/THUDM/GLM-4-Z1-9B-0414)
[🤖 ModelScope](https://modelscope.cn/models/ZhipuAI/GLM-Z1-9B-0414)
[🧩 Modelers](https://modelers.cn/models/zhipuai/GLM-Z1-9B-0414)
[🟣 WiseModel](https://wisemodel.cn/models/ZhipuAI/GLM-Z1-9B-0414) | | GLM-4-32B-Base-0414 | Base | 32K -> 128K | [🤗 Huggingface](https://huggingface.co/THUDM/GLM-4-32B-Base-0414)
[🤖 ModelScope](https://modelscope.cn/models/ZhipuAI/GLM-4-32B-Base-0414)
[🧩 Modelers](https://modelers.cn/models/zhipuai/GLM-4-32B-Base-0414)
[🟣 WiseModel](https://wisemodel.cn/models/ZhipuAI/GLM-4-32B-Base-0414) | | GLM-4-32B-0414 | Chat | 32K -> 128K | [🤗 Huggingface](https://huggingface.co/THUDM/GLM-4-32B-0414)
[🤖 ModelScope](https://modelscope.cn/models/ZhipuAI/GLM-4-32B-0414)
[🧩 Modelers](https://modelers.cn/models/zhipuai/GLM-4-32B-0414)
[🟣 WiseModel](https://wisemodel.cn/models/ZhipuAI/GLM-4-32B-Base-0414) | | GLM-Z1-32B-0414 | Reasoning | 32K -> 128K | [🤗 Huggingface](https://huggingface.co/THUDM/GLM-Z1-32B-0414)
[🤖 ModelScope](https://modelscope.cn/models/ZhipuAI/GLM-Z1-32B-0414)
[🧩 Modelers](https://modelers.cn/models/zhipuai/GLM-Z1-32B-0414)
[🟣 WiseModel](https://wisemodel.cn/models/ZhipuAI/GLM-Z1-32B-0414) | | GLM-Z1-Rumination-32B-0414 | Reasoning | 128K | [🤗 Huggingface](https://huggingface.co/THUDM/GLM-Z1-Rumination-32B-0414)
[🤖 ModelScope](https://modelscope.cn/models/ZhipuAI/GLM-Z1-Rumination-32B-0414)
[🧩 Modelers](https://modelers.cn/models/zhipuai/GLM-Z1-Rumination-32B-0414)
[🟣 WiseModel](https://wisemodel.cn/models/ZhipuAI/GLM-Z1-Rumination-32B-0414) | GLM-4-9B-0414 由于其较小的模型容量，我们未对其智能体能力进行类似 GLM-4-32B-0414 的强化，主要针对翻译等需要大批量调用的场景进行优化。 \* 模型原生采用 32K 上下文进行训练，对于输入 + 输出长度可能超过 32K 的请求，我们建议激活 YaRN 来获得较好的外推性能，详情见[部署章节](#%E6%A8%A1%E5%9E%8B%E5%92%8C%E6%8F%90%E7%A4%BA%E8%AF%8D%E5%AE%9E%E7%8E%B0)。 以下为 2024 年 6 月 5 日发布的 GLM-4 系列模型，其详细内容可以在[这里](README_zh_240605.md)查看。 | Model | Type | Seq Length* | Download | |:-----------------------------:|:---------:|:----------:|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------:| | GLM-4-9B | Base | 8K | [🤗 Huggingface](https://huggingface.co/THUDM/glm-4-9b)
[🤖 ModelScope](https://modelscope.cn/models/ZhipuAI/glm-4-9b)
| | GLM-4-9B-Chat | Chat | 128K | [🤗 Huggingface](https://huggingface.co/THUDM/glm-4-9b-chat)
[🤖 ModelScope](https://modelscope.cn/models/ZhipuAI/glm-4-9b-chat)
[🟣 WiseModel](https://wisemodel.cn/models/ZhipuAI/GLM-4-9B-Chat) | | GLM-4-9B-Chat-HF | Chat | 128K | [🤗 Huggingface](https://huggingface.co/THUDM/glm-4-9b-chat-hf)
[🤖 ModelScope](https://modelscope.cn/models/ZhipuAI/glm-4-9b-chat-hf) | | GLM-4-9B-Chat-1M | Chat | 1M | [🤗 Huggingface](https://huggingface.co/THUDM/glm-4-9b-chat-1m)
[🤖 ModelScope](https://modelscope.cn/models/ZhipuAI/glm-4-9b-chat-1m)
[🟣 WiseModel](https://wisemodel.cn/models/ZhipuAI/GLM-4-9B-Chat-1M) | | GLM-4-9B-Chat-1M-HF | Chat | 1M | [🤗 Huggingface](https://huggingface.co/THUDM/glm-4-9b-chat-1m-hf)
[🤖 ModelScope](https://modelscope.cn/models/ZhipuAI/glm-4-9b-chat-1m-hf) | | GLM-4V-9B | Chat | 8K | [🤗 Huggingface](https://huggingface.co/THUDM/glm-4v-9b)
[🤖 ModelScope](https://modelscope.cn/models/ZhipuAI/glm-4v-9b)
[🟣 WiseModel](https://wisemodel.cn/models/ZhipuAI/GLM-4V-9B) | ## 评测结果 ### GLM-4-0414 系列
| 模型 | IFEval | BFCL-v3 (Overall) | BFCL-v3 (MultiTurn) | TAU-Bench (Retail) | TAU-Bench (Airline) | SimpleQA | HotpotQA | | ---------------- | ------ | ----------------- | ------------------- | ------------------ | ------------------- | -------- | -------- | | Qwen2.5-Max | 85.6 | 50.9 | 30.5 | 58.3 | 22.0 | 79.0 | 52.8 | | GPT-4o-1120 | 81.9 | 69.6 | 41.0 | 62.8 | 46.0 | 82.8 | 63.9 | | DeepSeek-V3-0324 | 83.4 | 66.2 | 35.8 | 60.7 | 32.4 | 82.6 | 54.6 | | DeepSeek-R1 | 84.3 | 57.5 | 12.4 | 33.0 | 37.3 | 83.9 | 63.1 | | GLM-4-32B-0414 | 87.6 | 69.6 | 41.5 | 68.7 | 51.2 | 88.1 | 63.8 | > 对于 `SimpleQA` 和 `HotpotQA`，我们分别从测试集中采样了近500条测试样例，提供所有模型最基础的 `search` 和 `click` 工具，另外确保其余 Setting 保持一致后，3次评测取平均值 | 模型 | 框架 | [SWE-bench Verified](https://openai.com/index/introducing-swe-bench-verified/) | [SWE-bench Verified mini](https://github.com/mariushobbhahn/SWEBench-verified-mini) | |---|--------------------------|---|-------------------------------------------------------------------------------------| | GLM-4-32B-0414 | Moatless[1] | 33.8 | 38.0 | | GLM-4-32B-0414 | Agentless[2] | 30.7 | 34.0 | | GLM-4-32B-0414 | OpenHands[3] | 27.2 | 28.0 | [1] [Moatless v0.0.3](https://github.com/aorwall/moatless-tools) 使用如下参数 `response_format="react", thoughts_in_action=False, max_interations=30`，未对失败轨迹进行重试，其余为默认配置 [2] [Agentless v1.5.0](https://github.com/OpenAutoCoder/Agentless) 其中的 Embedding 模型使用了 [BGE](https://github.com/FlagOpen/FlagEmbedding/blob/master/README_zh.md)，基于[FAISS](https://github.com/facebookresearch/faiss)进行相似性检索，为加快patch验证的速度同时尽可能保证效果，将运行单个实例的超时时间从默认的300s修改为180s [3] [OpenHands v0.29.1](https://github.com/All-Hands-AI/OpenHands/tree/main) 未采用 YaRN 上下文扩展，而是限制了最大 60 个 iterations，并对 history 进行 summarization 以防止超出 32K 上下文限制，summarization 配置为 `llm_config="condenser", keep_first=1, max_size=32`，同样未对失败轨迹进行重试 ### GLM-Z1-0414 系列
## 模型和提示词实现 ### 模型实现 如果你想查看我们的模型实现，欢迎查看在相关仓库的模型实现 Pull Request，他们已经被合并。 + [vLLM 模型实现](https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/models/glm4.py) + [transformers 模型实现](https://github.com/huggingface/transformers/blob/main/src/transformers/models/glm4/modeling_glm4.py) + [llama.cpp 模型实现](https://github.com/ggml-org/llama.cpp/pull/12867) ### 处理长上下文（YaRN） 如果模型的输出 + 输出 token 数可能超过模型的原生上下文长度（GLM-4-0414系列多数为32k），建议开启 YaRN 来获得更好的长上下文建模能力。对于支持的框架，你可以在对应的`config.json`中修改。具体地，对于 GLM-Z1 系列模型，当输入长度超过 **8,192 tokens** 时，考虑启用 YaRN（Rope Scaling）。 ```json "rope_scaling": { "factor": 4.0, "original_max_position_embeddings": 32768, "type": "yarn" } ``` 对于多数用户请求，如果输出 + 输出 token 数不会超过原生上下文长度，则无需任何修改。 ### 模型微调 可以在`finetune/README.md`下找到模型的微调所需算力信息和案例的微调脚本。 可以通过以下命令开启一个简单的模型微调案例 ```shell cd finetune pip install -r ../inference/requirements.txt pip install -r requirements.txt # Use single GPU for Chat Fine-tune python finetune.py data/AdvertiseGen/ THUDM/GLM-4-9B-0414 configs/lora.yaml ``` 🎉 脚本也支持使用**SwanLab**进行微调可视化跟踪，可以访问[SwanLab可视化看板](https://swanlab.cn/@ShaohonChen/GLM4-Finetune/overview)获得案例微调脚本的训练日志。 ### 提示词实现 如果你使用`transformers`库提供的`apply_chat_template`方法构建提示词。以下是对不同 GLM-4-0414 模型中 `系统提示词`的限制。 + `GLM-4-32B-Base-0414`: 基座模型，无对话模板。 + `GLM-4-*-0414` / `GLM-Z1-*-0414`: 如果传入`tools`，则由 `apply_chat_template` 填充工具到`chat_template`中的固定模板，单独作为一条带有`tools`绑定的 `system`字段信息并拼接于`messages[0]`。原本传入的所有 `messages` 自动往后移动一个位置。 + `GLM-Z1-Rumination-32B-0414`: + 不支持自定义系统提示词，不支持自定义工具，你的所有 `tools` 和 `system` 字段会被 `apply_chat_template` 忽略。使用该模型需要外接搜索引擎或者自定义retrieval API。 + 一共支持四个工具，分别是 ``` 1. search 描述: 执行搜索查询并返回搜索结果。当您需要查找有关特定主题的信息时使用此功能。 参数: query (字符串) - 搜索查询字符串，除非是中文专有名词，否则使用英文单词 2. click 描述: 点击搜索结果中的链接并导航到相应页面。当您需要查看特定搜索结果的详细内容时使用此功能。 参数: link_id (整数) - 要点击的链接ID（来自搜索结果中的序号） 3. open 描述: 打开特定网站。通过URL获取任何网站的内容。 参数: url (字符串) - 目标网站URL或域名 4. finish 描述: 完成任务。当您已找到所需信息时使用此功能。 参数: 无 ``` + `chat_template`中的固定模板使用英文思过程，如果要更换其他语言，需要修改以下部分（暂时支持中文和英文） ``` <重要配置> - 采用语言 * 搜索关键词：英文 -> 在这里换成“中文”或者其他语言 * 思考：英文 -> 在这里换成“中文”或者其他语言 ``` GLM-4-0414 系列模型的提示词构造可以前往对应的模型仓库中的 `chat_template.jinja` 查看具体的模型对话模板。 ## 引用 如果你觉得我们的工作有帮助的话，请考虑引用下列论文。 ``` @misc{glm2024chatglm, title={ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools}, author={Team GLM and Aohan Zeng and Bin Xu and Bowen Wang and Chenhui Zhang and Da Yin and Diego Rojas and Guanyu Feng and Hanlin Zhao and Hanyu Lai and Hao Yu and Hongning Wang and Jiadai Sun and Jiajie Zhang and Jiale Cheng and Jiayi Gui and Jie Tang and Jing Zhang and Juanzi Li and Lei Zhao and Lindong Wu and Lucen Zhong and Mingdao Liu and Minlie Huang and Peng Zhang and Qinkai Zheng and Rui Lu and Shuaiqi Duan and Shudan Zhang and Shulin Cao and Shuxun Yang and Weng Lam Tam and Wenyi Zhao and Xiao Liu and Xiao Xia and Xiaohan Zhang and Xiaotao Gu and Xin Lv and Xinghan Liu and Xinyi Liu and Xinyue Yang and Xixuan Song and Xunkai Zhang and Yifan An and Yifan Xu and Yilin Niu and Yuantao Yang and Yueyan Li and Yushi Bai and Yuxiao Dong and Zehan Qi and Zhaoyu Wang and Zhen Yang and Zhengxiao Du and Zhenyu Hou and Zihan Wang}, year={2024}, eprint={2406.12793}, archivePrefix={arXiv}, primaryClass={id='cs.CL' full_name='Computation and Language' is_active=True alt_name='cmp-lg' in_archive='cs' is_general=False description='Covers natural language processing. Roughly includes material in ACM Subject Class I.2.7. Note that work on artificial languages (programming languages, logics, formal systems) that does not explicitly address natural-language issues broadly construed (natural-language processing, computational linguistics, speech, text retrieval, etc.) is not appropriate for this area.'} } ``` ================================================ FILE: README_zh_240605.md ================================================ # GLM-4

📄 Report • 🤗 HF Repo • 🤖 ModelScope • 🟣 WiseModel • 🐦 Twitter • 👋 加入我们的 Discord 和 微信

📍在 智谱AI开放平台 体验和使用更大规模的 GLM 商业模型。

Read this in [English](README_en.md) ## 项目更新 - 🔥🔥 **News**: ```2024/12/10```: 本仓库微调代码支持使用`Ascend NPU`进行微调。请更新微调代码并查看代码内注释。 - 🔥 **News**: ```2024/11/01```: 本仓库依赖进行升级，请更新`requirements.txt`中的依赖以保证正常运行模型。[glm-4-9b-chat-hf](https://huggingface.co/THUDM/glm-4-9b-chat-hf) 是适配 `transformers>=4.46.2` 的模型权重，使用 `transformers` 库中的 `GlmModel` 类实现。 同时，[glm-4-9b-chat](https://huggingface.co/THUDM/glm-4-9b-chat), [glm-4v-9b](https://huggingface.co/THUDM/glm-4v-9b) 中的 `tokenzier_chatglm.py` 已经更新以适配最新版本的 `transformers`库。请前往 HuggingFace 更新文件。 - 🔥 **News**: ```2024/10/27```: 我们开源了 [LongReward](https://github.com/THUDM/LongReward)，这是一个使用 AI 反馈改进长上下文大型语言模型。 - 🔥 **News**: ```2024/10/25```: 我们开源了端到端中英语音对话模型 [GLM-4-Voice](https://github.com/THUDM/GLM-4-Voice)。 - 🔥 **News**: ```2024/09/05``` 我们开源了使LLMs能够在长上下文问答中生成细粒度引用的模型 [longcite-glm4-9b](https://huggingface.co/THUDM/LongCite-glm4-9b) 以及数据集 [LongCite-45k](https://huggingface.co/datasets/THUDM/LongCite-45k), 欢迎在 [Huggingface Space](https://huggingface.co/spaces/THUDM/LongCite) 在线体验。 - 🔥**News**: ```2024/08/15```: 我们开源具备长文本输出能力(单轮对话大模型输出可超过1万token) 的模型 [longwriter-glm4-9b](https://huggingface.co/THUDM/LongWriter-glm4-9b) 以及数据集 [LongWriter-6k](https://huggingface.co/datasets/THUDM/LongWriter-6k), 欢迎在 [Huggingface Space](https://huggingface.co/spaces/THUDM/LongWriter) 或 [魔搭社区空间](https://modelscope.cn/studios/ZhipuAI/LongWriter-glm4-9b-demo) 在线体验。 - 🔥 **News**: ```2024/07/24```: 我们发布了与长文本相关的最新技术解读，关注 [这里](https://medium.com/@ChatGLM/glm-long-scaling-pre-trained-model-contexts-to-millions-caa3c48dea85) 查看我们在训练 GLM-4-9B 开源模型中关于长文本技术的技术报告。 - 🔥 **News**: ``2024/07/09``: GLM-4-9B-Chat 模型已适配 [Ollama](https://github.com/ollama/ollama), [Llama.cpp](https://github.com/ggerganov/llama.cpp)，您可以在 [PR](https://github.com/ggerganov/llama.cpp/pull/8031) 查看具体的细节。 - 🔥 **News**: ``2024/06/18``: 我们发布 [技术报告](https://arxiv.org/pdf/2406.12793), 欢迎查看。 - 🔥 **News**: ``2024/06/05``: 我们发布 GLM-4-9B 系列开源模型。 ## 模型介绍 GLM-4-9B 是智谱 AI 推出的最新一代预训练模型 GLM-4 系列中的开源版本。 在语义、数学、推理、代码和知识等多方面的数据集测评中， **GLM-4-9B** 及其人类偏好对齐的版本 **GLM-4-9B-Chat** 均表现出超越 Llama-3-8B 的卓越性能。除了能进行多轮对话，GLM-4-9B-Chat 还具备网页浏览、代码执行、自定义工具调用（Function Call）和长文本推理（支持最大 128K 上下文）等高级功能。本代模型增加了多语言支持，支持包括日语，韩语，德语在内的 26 种语言。我们还推出了支持 1M 上下文长度（约 200 万中文字符）的 **GLM-4-9B-Chat-1M** 模型和基于 GLM-4-9B 的多模态模型 GLM-4V-9B。**GLM-4V-9B** 具备 1120 * 1120 高分辨率下的中英双语多轮对话能力，在中英文综合能力、感知推理、文字识别、图表理解等多方面多模态评测中，GLM-4V-9B 表现出超越 GPT-4-turbo-2024-04-09、Gemini 1.0 Pro、Qwen-VL-Max 和 Claude 3 Opus 的卓越性能。 ## Model List | Model | Type | Seq Length | Transformers Version | Download | Online Demo | |:-------------------:|:----:|:----------:|:--------------------:|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------:|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------:| | GLM-4-9B | Base | 8K | `4.44.0 - 4.45.0` | [🤗 Huggingface](https://huggingface.co/THUDM/glm-4-9b)
[🤖 ModelScope](https://modelscope.cn/models/ZhipuAI/glm-4-9b)
[🟣 WiseModel](https://wisemodel.cn/models/ZhipuAI/glm-4-9b) | / | | GLM-4-9B-Chat | Chat | 128K | `>= 4.44.0` | [🤗 Huggingface](https://huggingface.co/THUDM/glm-4-9b-chat)
[🤖 ModelScope](https://modelscope.cn/models/ZhipuAI/glm-4-9b-chat)
[🟣 WiseModel](https://wisemodel.cn/models/ZhipuAI/GLM-4-9B-Chat) | [🤖 ModelScope CPU](https://modelscope.cn/studios/dash-infer/GLM-4-Chat-DashInfer-Demo/summary)
[🤖 ModelScope vLLM](https://modelscope.cn/studios/ZhipuAI/glm-4-9b-chat-vllm/summary) | | GLM-4-9B-Chat-HF | Chat | 128K | `>= 4.46.0` | [🤗 Huggingface](https://huggingface.co/THUDM/glm-4-9b-chat-hf)
[🤖 ModelScope](https://modelscope.cn/models/ZhipuAI/glm-4-9b-chat-hf) | [🤖 ModelScope CPU](https://modelscope.cn/studios/dash-infer/GLM-4-Chat-DashInfer-Demo/summary)
[🤖 ModelScope vLLM](https://modelscope.cn/studios/ZhipuAI/glm-4-9b-chat-vllm/summary) | | GLM-4-9B-Chat-1M | Chat | 1M | `>= 4.44.0` | [🤗 Huggingface](https://huggingface.co/THUDM/glm-4-9b-chat-1m)
[🤖 ModelScope](https://modelscope.cn/models/ZhipuAI/glm-4-9b-chat-1m)
[🟣 WiseModel](https://wisemodel.cn/models/ZhipuAI/GLM-4-9B-Chat-1M) | / | | GLM-4-9B-Chat-1M-HF | Chat | 1M | `>= 4.46.0` | [🤗 Huggingface](https://huggingface.co/THUDM/glm-4-9b-chat-1m-hf)
[🤖 ModelScope](https://modelscope.cn/models/ZhipuAI/glm-4-9b-chat-1m-hf) | / | | GLM-4V-9B | Chat | 8K | `>= 4.46.0` | [🤗 Huggingface](https://huggingface.co/THUDM/glm-4v-9b)
[🤖 ModelScope](https://modelscope.cn/models/ZhipuAI/glm-4v-9b)
[🟣 WiseModel](https://wisemodel.cn/models/ZhipuAI/GLM-4V-9B) | [🤖 ModelScope](https://modelscope.cn/studios/ZhipuAI/glm-4v-9b-Demo/summary) | ## 评测结果 ### 对话模型典型任务 | Model | AlignBench | MT-Bench | IFEval | MMLU | C-Eval | GSM8K | MATH | HumanEval | NaturalCodeBench | |:--------------------|:----------:|:--------:|:------:|:----:|:------:|:-----:|:----:|:---------:|:----------------:| | Llama-3-8B-Instruct | 6.40 | 8.00 | 68.6 | 68.4 | 51.3 | 79.6 | 30.0 | 62.2 | 24.7 | | ChatGLM3-6B | 5.18 | 5.50 | 28.1 | 61.4 | 69.0 | 72.3 | 25.7 | 58.5 | 11.3 | | GLM-4-9B-Chat | 7.01 | 8.35 | 69.0 | 72.4 | 75.6 | 79.6 | 50.6 | 71.8 | 32.2 | ### 基座模型典型任务 | Model | MMLU | C-Eval | GPQA | GSM8K | MATH | HumanEval | |:--------------------|:----:|:------:|:----:|:-----:|:----:|:---------:| | Llama-3-8B | 66.6 | 51.2 | - | 45.8 | - | 33.5 | | Llama-3-8B-Instruct | 68.4 | 51.3 | 34.2 | 79.6 | 30.0 | 62.2 | | ChatGLM3-6B-Base | 61.4 | 69.0 | 26.8 | 72.3 | 25.7 | 58.5 | | GLM-4-9B | 74.7 | 77.1 | 34.3 | 84.0 | 30.4 | 70.1 | > 由于 `GLM-4-9B` 在预训练过程中加入了部分数学、推理、代码相关的 instruction 数据，所以将 Llama-3-8B-Instruct 也列入比较范围。 ### 长文本 在 1M 的上下文长度下进行[大海捞针实验](https://github.com/LargeWorldModel/LWM/blob/main/scripts/eval_needle.py)，结果如下： ![needle](resources/eval_needle.jpeg) 在 LongBench-Chat 上对长文本能力进行了进一步评测，结果如下:

描述文字

### 多语言能力 在六个多语言数据集上对 GLM-4-9B-Chat 和 Llama-3-8B-Instruct 进行了测试，测试结果及数据集对应选取语言如下表 | Dataset | Llama-3-8B-Instruct | GLM-4-9B-Chat | Languages | |:------------|:-------------------:|:-------------:|:----------------------------------------------------------------------------------------------:| | M-MMLU | 49.6 | 56.6 | all | | FLORES | 25.0 | 28.8 | ru, es, de, fr, it, pt, pl, ja, nl, ar, tr, cs, vi, fa, hu, el, ro, sv, uk, fi, ko, da, bg, no | | MGSM | 54.0 | 65.3 | zh, en, bn, de, es, fr, ja, ru, sw, te, th | | XWinograd | 61.7 | 73.1 | zh, en, fr, jp, ru, pt | | XStoryCloze | 84.7 | 90.7 | zh, en, ar, es, eu, hi, id, my, ru, sw, te | | XCOPA | 73.3 | 80.1 | zh, et, ht, id, it, qu, sw, ta, th, tr, vi | ### 工具调用能力 我们在 [Berkeley Function Calling Leaderboard](https://github.com/ShishirPatil/gorilla/tree/main/berkeley-function-call-leaderboard) 上进行了测试并得到了以下结果： | Model | Overall Acc. | AST Summary | Exec Summary | Relevance | |:-----------------------|:------------:|:-----------:|:------------:|:---------:| | Llama-3-8B-Instruct | 58.88 | 59.25 | 70.01 | 45.83 | | gpt-4-turbo-2024-04-09 | 81.24 | 82.14 | 78.61 | 88.75 | | ChatGLM3-6B | 57.88 | 62.18 | 69.78 | 5.42 | | GLM-4-9B-Chat | 81.00 | 80.26 | 84.40 | 87.92 | ### 多模态能力 GLM-4V-9B 是一个多模态语言模型，具备视觉理解能力，其相关经典任务的评测结果如下： | | **MMBench-EN-Test** | **MMBench-CN-Test** | **SEEDBench_IMG** | **MMStar** | **MMMU** | **MME** | **HallusionBench** | **AI2D** | **OCRBench** | |----------------------------|---------------------|---------------------|-------------------|------------|----------|---------|--------------------|----------|--------------| | **gpt-4o-2024-05-13** | 83.4 | 82.1 | 77.1 | 63.9 | 69.2 | 2310.3 | 55.0 | 84.6 | 736 | | **gpt-4-turbo-2024-04-09** | 81.0 | 80.2 | 73.0 | 56.0 | 61.7 | 2070.2 | 43.9 | 78.6 | 656 | | **gpt-4-1106-preview** | 77.0 | 74.4 | 72.3 | 49.7 | 53.8 | 1771.5 | 46.5 | 75.9 | 516 | | **InternVL-Chat-V1.5** | 82.3 | 80.7 | 75.2 | 57.1 | 46.8 | 2189.6 | 47.4 | 80.6 | 720 | | **LLaVA-Next-Yi-34B** | 81.1 | 79.0 | 75.7 | 51.6 | 48.8 | 2050.2 | 34.8 | 78.9 | 574 | | **Step-1V** | 80.7 | 79.9 | 70.3 | 50.0 | 49.9 | 2206.4 | 48.4 | 79.2 | 625 | | **MiniCPM-Llama3-V2.5** | 77.6 | 73.8 | 72.3 | 51.8 | 45.8 | 2024.6 | 42.4 | 78.4 | 725 | | **Qwen-VL-Max** | 77.6 | 75.7 | 72.7 | 49.5 | 52.0 | 2281.7 | 41.2 | 75.7 | 684 | | **Gemini 1.0 Pro** | 73.6 | 74.3 | 70.7 | 38.6 | 49.0 | 2148.9 | 45.7 | 72.9 | 680 | | **Claude 3 Opus** | 63.3 | 59.2 | 64.0 | 45.7 | 54.9 | 1586.8 | 37.8 | 70.6 | 694 | | **GLM-4V-9B** | 81.1 | 79.4 | 76.8 | 58.7 | 47.2 | 2163.8 | 46.6 | 81.1 | 786 | ## 快速调用 **硬件配置和系统要求，请查看[这里](basic_demo/README.md)。** ### 使用以下方法快速调用 GLM-4-9B-Chat 语言模型 使用 transformers 后端进行推理: ```python import torch from transformers import AutoModelForCausalLM, AutoTokenizer import os os.environ['CUDA_VISIBLE_DEVICES'] = '0' # 设置 GPU 编号，如果单机单卡指定一个，单机多卡指定多个 GPU 编号 MODEL_PATH = "THUDM/glm-4-9b-chat-hf" device = "cuda" if torch.cuda.is_available() else "cpu" tokenizer = AutoTokenizer.from_pretrained(MODEL_PATH, trust_remote_code=True) query = "你好" inputs = tokenizer.apply_chat_template([{"role": "user", "content": query}], add_generation_prompt=True, tokenize=True, return_tensors="pt", return_dict=True ) inputs = inputs.to(device) model = AutoModelForCausalLM.from_pretrained( MODEL_PATH, torch_dtype=torch.bfloat16, low_cpu_mem_usage=True, trust_remote_code=True, device_map="auto" ).eval() gen_kwargs = {"max_length": 2500, "do_sample": True, "top_k": 1} with torch.no_grad(): outputs = model.generate(**inputs, **gen_kwargs) outputs = outputs[:, inputs['input_ids'].shape[1]:] print(tokenizer.decode(outputs[0], skip_special_tokens=True)) ``` 使用 vLLM 后端进行推理: ```python from transformers import AutoTokenizer from vllm import LLM, SamplingParams # GLM-4-9B-Chat-1M # max_model_len, tp_size = 1048576, 4 # 如果遇见 OOM 现象，建议减少max_model_len，或者增加tp_size max_model_len, tp_size = 131072, 1 model_name = "THUDM/glm-4-9b-chat-hf" prompt = [{"role": "user", "content": "你好"}] tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True) llm = LLM( model=model_name, tensor_parallel_size=tp_size, max_model_len=max_model_len, trust_remote_code=True, enforce_eager=True, # GLM-4-9B-Chat-1M 如果遇见 OOM 现象，建议开启下述参数 # enable_chunked_prefill=True, # max_num_batched_tokens=8192 ) stop_token_ids = [151329, 151336, 151338] sampling_params = SamplingParams(temperature=0.95, max_tokens=1024, stop_token_ids=stop_token_ids) inputs = tokenizer.apply_chat_template(prompt, tokenize=False, add_generation_prompt=True) outputs = llm.generate(prompts=inputs, sampling_params=sampling_params) print(outputs[0].outputs[0].text) ``` ### 使用以下方法快速调用 GLM-4V-9B 多模态模型 使用 transformers 后端进行推理: ```python import torch from PIL import Image from transformers import AutoModelForCausalLM, AutoTokenizer import os os.environ['CUDA_VISIBLE_DEVICES'] = '0' # 设置 GPU 编号，如果单机单卡指定一个，单机多卡指定多个 GPU 编号 MODEL_PATH = "THUDM/glm-4v-9b" device = "cuda" if torch.cuda.is_available() else "cpu" tokenizer = AutoTokenizer.from_pretrained(MODEL_PATH, trust_remote_code=True) query = '描述这张图片' image = Image.open("your image").convert('RGB') inputs = tokenizer.apply_chat_template([{"role": "user", "image": image, "content": query}], add_generation_prompt=True, tokenize=True, return_tensors="pt", return_dict=True) # chat mode inputs = inputs.to(device) model = AutoModelForCausalLM.from_pretrained( MODEL_PATH, torch_dtype=torch.bfloat16, low_cpu_mem_usage=True, trust_remote_code=True, device_map="auto" ).eval() gen_kwargs = {"max_length": 2500, "do_sample": True, "top_k": 1} with torch.no_grad(): outputs = model.generate(**inputs, **gen_kwargs) outputs = outputs[:, inputs['input_ids'].shape[1]:] print(tokenizer.decode(outputs[0])) ``` 使用 vLLM 后端进行推理: ```python from PIL import Image from vllm import LLM, SamplingParams model_name = "THUDM/glm-4v-9b" llm = LLM(model=model_name, tensor_parallel_size=1, max_model_len=8192, trust_remote_code=True, enforce_eager=True) stop_token_ids = [151329, 151336, 151338] sampling_params = SamplingParams(temperature=0.2, max_tokens=1024, stop_token_ids=stop_token_ids) prompt = "What's the content of the image?" image = Image.open("your image").convert('RGB') inputs = { "prompt": prompt, "multi_modal_data": { "image": image }, } outputs = llm.generate(inputs, sampling_params=sampling_params) for o in outputs: generated_text = o.outputs[0].text print(generated_text) ``` ## 完整项目列表 如果你想更进一步了解 GLM-4-9B 系列开源模型，本开源仓库通过以下内容为开发者提供基础的 GLM-4-9B 的使用和开发代码 + [basic_demo](basic_demo/README.md): 在这里包含了 + 使用 transformers 和 vLLM 后端的交互代码 + OpenAI API 后端交互代码 + Batch 推理代码 + [composite_demo](composite_demo/README.md): 在这里包含了 + GLM-4-9B-Chat 以及 GLM-4V-9B 开源模型的完整功能演示代码，包含了 All Tools 能力、长文档解读和多模态能力的展示。 + [fintune_demo](finetune_demo/README.md): 在这里包含了 + PEFT (LORA, P-Tuning) 微调代码 + SFT 微调代码 + [intel_device_demo](intel_device_demo/): 在这里包含了 + 使用 OpenVINO 部署模型代码 + 使用 Intel® Extension for Transformers 部署模型代码 ## 友情链接 + [LLaMA-Factory](https://github.com/hiyouga/LLaMA-Factory): 高效开源微调框架，已支持 GLM-4-9B-Chat 语言模型微调。 + [SWIFT](https://github.com/modelscope/swift): 魔搭社区的大模型/多模态大模型训练框架，已支持 GLM-4-9B-Chat / GLM-4V-9B 模型微调。 + [Xorbits Inference](https://github.com/xorbitsai/inference): 性能强大且功能全面的分布式推理框架，轻松一键部署你自己的模型或内置的前沿开源模型。 + [LangChain-ChatChat](https://github.com/chatchat-space/Langchain-Chatchat): 基于 Langchain 与 ChatGLM 等语言模型的 RAG 与 Agent 应用 + [self-llm](https://github.com/datawhalechina/self-llm/tree/master/models/GLM-4): Datawhale 团队的提供的 GLM-4-9B 系列模型使用教程。 + [chatglm.cpp](https://github.com/li-plus/chatglm.cpp): 类似 llama.cpp 的量化加速推理方案，实现笔记本上实时对话 + [OpenVINO](https://github.com/openvinotoolkit): Intel 开发的高性能 CPU,GPU及NPU 加速推理方案，可以参考此 [步骤](https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/llm-chatbot/llm-chatbot-generate-api.ipynb) 部署 glm-4-9b-chat 模型。 ## 协议 + GLM-4 模型的权重的使用则需要遵循 [模型协议](https://huggingface.co/THUDM/glm-4-9b/blob/main/LICENSE)。 + 本开源仓库的代码则遵循 [Apache 2.0](LICENSE) 协议。 请您严格遵循开源协议。 ## 引用 如果你觉得我们的工作有帮助的话，请考虑引用下列论文。 ``` @misc{glm2024chatglm, title={ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools}, author={Team GLM and Aohan Zeng and Bin Xu and Bowen Wang and Chenhui Zhang and Da Yin and Diego Rojas and Guanyu Feng and Hanlin Zhao and Hanyu Lai and Hao Yu and Hongning Wang and Jiadai Sun and Jiajie Zhang and Jiale Cheng and Jiayi Gui and Jie Tang and Jing Zhang and Juanzi Li and Lei Zhao and Lindong Wu and Lucen Zhong and Mingdao Liu and Minlie Huang and Peng Zhang and Qinkai Zheng and Rui Lu and Shuaiqi Duan and Shudan Zhang and Shulin Cao and Shuxun Yang and Weng Lam Tam and Wenyi Zhao and Xiao Liu and Xiao Xia and Xiaohan Zhang and Xiaotao Gu and Xin Lv and Xinghan Liu and Xinyi Liu and Xinyue Yang and Xixuan Song and Xunkai Zhang and Yifan An and Yifan Xu and Yilin Niu and Yuantao Yang and Yueyan Li and Yushi Bai and Yuxiao Dong and Zehan Qi and Zhaoyu Wang and Zhen Yang and Zhengxiao Du and Zhenyu Hou and Zihan Wang}, year={2024}, eprint={2406.12793}, archivePrefix={arXiv}, primaryClass={id='cs.CL' full_name='Computation and Language' is_active=True alt_name='cmp-lg' in_archive='cs' is_general=False description='Covers natural language processing. Roughly includes material in ACM Subject Class I.2.7. Note that work on artificial languages (programming languages, logics, formal systems) that does not explicitly address natural-language issues broadly construed (natural-language processing, computational linguistics, speech, text retrieval, etc.) is not appropriate for this area.'} } ``` ``` @misc{wang2023cogvlm, title={CogVLM: Visual Expert for Pretrained Language Models}, author={Weihan Wang and Qingsong Lv and Wenmeng Yu and Wenyi Hong and Ji Qi and Yan Wang and Junhui Ji and Zhuoyi Yang and Lei Zhao and Xixuan Song and Jiazheng Xu and Bin Xu and Juanzi Li and Yuxiao Dong and Ming Ding and Jie Tang}, year={2023}, eprint={2311.03079}, archivePrefix={arXiv}, primaryClass={cs.CV} } ``` ================================================ FILE: demo/composite_demo/.gitignore ================================================ *venv *.DS_Store *model *.idea/ # Created by https://www.toptal.com/developers/gitignore/api/python # Edit at https://www.toptal.com/developers/gitignore?templates=python ### Python ### # Byte-compiled / optimized / DLL files __pycache__/ *.py[cod] *$py.class # C extensions *.so # Distribution / packaging .Python build/ develop-eggs/ dist/ downloads/ eggs/ .eggs/ lib/ lib64/ parts/ sdist/ var/ wheels/ share/python-wheels/ *.egg-info/ .installed.cfg *.egg MANIFEST # PyInstaller # Usually these files are written by a python script from a template # before PyInstaller builds the exe, so as to inject date/other infos into it. *.manifest *.spec # Installer logs pip-log.txt pip-delete-this-directory.txt # Unit test / coverage reports htmlcov/ .tox/ .nox/ .coverage .coverage.* .cache nosetests.xml coverage.xml *.cover *.py,cover .hypothesis/ .pytest_cache/ cover/ # Translations *.mo *.pot # Django stuff: *.log local_settings.py db.sqlite3 db.sqlite3-journal # Flask stuff: instance/ .webassets-cache # Scrapy stuff: .scrapy # Sphinx documentation docs/_build/ # PyBuilder .pybuilder/ target/ # Jupyter Notebook .ipynb_checkpoints # IPython profile_default/ ipython_config.py # pyenv # For a library or package, you might want to ignore these files since the code is # intended to run in multiple environments; otherwise, check them in: # .python-version # pipenv # According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control. # However, in case of collaboration, if having platform-specific dependencies or dependencies # having no cross-platform support, pipenv may install dependencies that don't work, or not # install all needed dependencies. #Pipfile.lock # poetry # Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control. # This is especially recommended for binary packages to ensure reproducibility, and is more # commonly ignored for libraries. # https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control #poetry.lock # pdm # Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control. #pdm.lock # pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it # in version control. # https://pdm.fming.dev/#use-with-ide .pdm.toml # PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm __pypackages__/ # Celery stuff celerybeat-schedule celerybeat.pid # SageMath parsed files *.sage.py # Environments .env .venv env/ venv/ ENV/ env.bak/ venv.bak/ # Spyder project settings .spyderproject .spyproject # Rope project settings .ropeproject # mkdocs documentation /site # mypy .mypy_cache/ .dmypy.json dmypy.json # Pyre type checker .pyre/ # pytype static type analyzer .pytype/ # Cython debug symbols cython_debug/ # PyCharm # JetBrains specific template is maintained in a separate JetBrains.gitignore that can # be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore # and can be added to the global gitignore or merged into this file. For a more nuclear # option (not recommended) you can uncomment the following to ignore the entire idea folder. #.idea/ ### Python Patch ### # Poetry local configuration file - https://python-poetry.org/docs/configuration/#local-configuration poetry.toml # ruff .ruff_cache/ # LSP config files pyrightconfig.json # End of https://www.toptal.com/developers/gitignore/api/python ================================================ FILE: demo/composite_demo/README.md ================================================ # GLM-4-9B Web Demo Read this in [English](README_en.md) ![Demo webpage](assets/demo.png) ## 安装 我们建议通过 [Conda](https://docs.conda.io/en/latest/) 进行环境管理。 执行以下命令新建一个 conda 环境并安装所需依赖： ```bash conda create -n glm-4-demo python=3.12 conda activate glm-4-demo pip install -r requirements.txt ``` 请注意，本项目需要 Python 3.10 或更高版本。 此外，使用 Code Interpreter 还需要安装 Jupyter 内核： ```bash ipython kernel install --name glm-4-demo --user ``` 您可以修改 `~/.local/share/jupyter/kernels/glm-4-demo/kernel.json` 来改变 Jupyter 内核的配置，包括内核的启动参数等。例如，若您希望在使用 All Tools 的 Python 代码执行能力时使用 Matplotlib 画图，可以在 `argv` 数组中添加 `"--matplotlib=inline"`。 若要使用浏览器和搜索功能，还需要启动浏览器后端。首先，根据 [Node.js](https://nodejs.org/en/download/package-manager) 官网的指示安装 Node.js，然后安装包管理器 [PNPM](https://pnpm.io) 之后安装浏览器服务的依赖： ```bash cd browser npm install -g pnpm pnpm install ``` ## 运行 1. 修改 `browser/src/config.ts` 中的 `BING_SEARCH_API_KEY` 配置浏览器服务需要使用的 Bing 搜索 API Key： ```diff export default { BROWSER_TIMEOUT: 10000, BING_SEARCH_API_URL: 'https://api.bing.microsoft.com/v7.0', BING_SEARCH_API_KEY: '', HOST: 'localhost', PORT: 3000, }; ``` 如果您注册的是Bing Customer Search的API，您可以修改您的配置文件为如下，并且填写您的Custom Configuration ID: ```diff export default { LOG_LEVEL: 'debug', BROWSER_TIMEOUT: 10000, BING_SEARCH_API_URL: 'https://api.bing.microsoft.com/v7.0/custom/', BING_SEARCH_API_KEY: 'YOUR_BING_SEARCH_API_KEY', CUSTOM_CONFIG_ID : 'YOUR_CUSTOM_CONFIG_ID', //将您的Custom Configuration ID放在此处 HOST: 'localhost', PORT: 3000, }; ``` 2. 文生图功能需要调用 CogView API。修改 `src/tools/config.py` ，提供文生图功能需要使用的 [智谱 AI 开放平台](https://open.bigmodel.cn) API Key： ```diff BROWSER_SERVER_URL = 'http://localhost:3000' IPYKERNEL = 'glm-4-demo' ZHIPU_AI_KEY = '' COGVIEW_MODEL = 'cogview-3' ``` 3. 启动浏览器后端，在单独的 shell 中： ```bash cd browser pnpm start ``` 4. 运行以下命令在本地加载模型并启动 demo： ```bash streamlit run src/main.py ``` 之后即可从命令行中看到 demo 的地址，点击即可访问。初次访问需要下载并加载模型，可能需要花费一定时间。 如果已经在本地下载了模型，可以通过 `export *_MODEL_PATH=/path/to/model` 来指定从本地加载模型。可以指定的模型包括： - `CHAT_MODEL_PATH`: 用于 All Tools 模式与文档解读模式，默认为 `THUDM/glm-4-9b-chatglm-4-9b-chat`。 - `VLM_MODEL_PATH`: 用于 VLM 模式，默认为 `THUDM/glm-4v-9b`。 Chat 模型支持使用 [vLLM](https://github.com/vllm-project/vllm) 推理。若要使用，请安装 vLLM 并设置环境变量 `USE_VLLM=1`。 Chat 模型支持使用 [OpenAI API](https://platform.openai.com/docs/api-reference/introduction) 推理。若要使用，请启动basic_demo目录下的openai_api_server并设置环境变量 `USE_API=1`。该功能可以解耦推理服务器和demo服务器。 如果需要自定义 Jupyter 内核，可以通过 `export IPYKERNEL=` 来指定。 ## 使用 GLM-4 Demo 拥有三种模式： - All Tools: 具有完整工具调用能力的对话模式，原生支持网页浏览、代码执行、图片生成，并支持自定义工具。 - 文档解读: 支持上传文档进行文档解读与对话。 - 多模态: 支持上传图像进行图像理解与对话。 ### All Tools 本模式兼容 ChatGLM3-6B 的工具注册流程。 + 代码能力，绘图能力，联网能力已经自动集成，用户只需按照要求配置对应的Key。 + 本模式下不支持系统提示词，模型会自动构建提示词。 对话模式下，用户可以直接在侧边栏修改 top_p, temperature 等参数来调整模型的行为。 与模型对话时，模型将会自主决定进行工具调用。 ![Tool calling](assets/tool.png) 由于原始结果可能较长，默认情况下工具调用结果被隐藏，可以通过展开折叠框查看原始的工具调用结果。 模型拥有进行网页搜索和 Python 代码执行的能力。同时，模型也可以连续调用多个工具。例如： ![Consecutive tool calling, 1](assets/web_plot_1.png) 此时模型通过调用浏览器工具进行搜索获取到了需要的数据，之后将会调用 Python 工具执行代码，利用 Matplotlib 绘图： ![Consecutive tool calling, 2](assets/web_plot_2.png) 如果提供了智谱开放平台 API Key，模型也可以调用 CogView 进行图像生成： ![Image generation](assets/cogview.png) #### 自定义工具 可以通过在 `tool_registry.py` 中注册新的工具来增强模型的能力。只需要使用 `@register_tool` 装饰函数即可完成注册。对于工具声明，函数名称即为工具的名称，函数 docstring 即为工具的说明；对于工具的参数，使用 `Annotated[typ: type, description: str, required: bool]` 标注参数的类型、描述和是否必须。 例如，`get_weather` 工具的注册如下： ```python @register_tool def get_weather( city_name: Annotated[str, 'The name of the city to be queried', True], ) -> str: """ Get the weather for `city_name` in the following week """ ... ``` ![The model uses tool to query the weather of Bangkok.](assets/weather.png) ### 文档解读 用户可以上传文档，使用 GLM-4-9B的长文本能力，对文本进行理解。可以解析 pptx，docx，pdf等文件。 + 本模式下不支持工具调用和系统提示词。 + 如果文本很长，可能导致模型需要的显存较高，请确认你的硬件配置。 ![Doc reader demo](assets/doc_reader.png) ### 多模态 多模态模式下，用户可以利用 GLM-4V 的多模态理解能力，上传图像并与 GLM-4V 进行多轮对话： 用户可以上传图片，使用 GLM-4-9B的图像理解能力，对图片进行理解。 + 本模式必须使用 glm-4v-9b 模型。 + 本模式下不支持工具调用和系统提示词。 + 模型仅能对一张图片进行理解和联系对话，如需更换图片，需要开启一个新的对话。 + 图像支持的分辨率为 1120 x 1120 ![VLM demo](assets/vlm.png) ================================================ FILE: demo/composite_demo/README_en.md ================================================ # GLM-4-9B Web Demo ![Demo webpage](assets/demo.png) ## Installation We recommend using [Conda](https://docs.conda.io/en/latest/) for environment management. Execute the following commands to create a conda environment and install the required dependencies: ```bash conda create -n glm-4-demo python=3.12 conda activate glm-4-demo pip install -r requirements.txt ``` Please note that this project requires Python 3.10 or higher. In addition, you need to install the Jupyter kernel to use the Code Interpreter: ```bash ipython kernel install --name glm-4-demo --user ``` You can modify `~/.local/share/jupyter/kernels/glm-4-demo/kernel.json` to change the configuration of the Jupyter kernel, including the kernel startup parameters. For example, if you want to use Matplotlib to draw when using the Python code execution capability of All Tools, you can add `"--matplotlib=inline"` to the `argv` array. To use the browser and search functions, you also need to start the browser backend. First, install Node.js according to the instructions on the [Node.js](https://nodejs.org/en/download/package-manager) official website, then install the package manager [PNPM](https://pnpm.io) and then install the browser service dependencies: ```bash cd browser npm install -g pnpm pnpm install ``` ## Run 1. Modify `BING_SEARCH_API_KEY` in `browser/src/config.ts` to configure the Bing Search API Key that the browser service needs to use: ```diff export default { BROWSER_TIMEOUT: 10000, BING_SEARCH_API_URL: 'https://api.bing.microsoft.com/v7.0', BING_SEARCH_API_KEY: '', HOST: 'localhost', PORT: 3000, }; ``` 2. The Wenshengtu function needs to call the CogView API. Modify `src/tools/config.py` , provide the [Zhipu AI Open Platform](https://open.bigmodel.cn) API Key required for the Wenshengtu function: ```diff BROWSER_SERVER_URL = 'http://localhost:3000' IPYKERNEL = 'glm4-demo' ZHIPU_AI_KEY = '' COGVIEW_MODEL = 'cogview-3' ``` 3. Start the browser backend in a separate shell: ```bash cd browser pnpm start ``` 4. Run the following commands to load the model locally and start the demo: ```bash streamlit run src/main.py ``` Then you can see the demo address from the command line and click it to access it. The first access requires downloading and loading the model, which may take some time. If you have downloaded the model locally, you can specify to load the model from the local by `export *_MODEL_PATH=/path/to/model`. The models that can be specified include: - `CHAT_MODEL_PATH`: used for All Tools mode and document interpretation mode, the default is `THUDM/glm-4-9b-chat`. - `VLM_MODEL_PATH`: used for VLM mode, the default is `THUDM/glm-4v-9b`. The Chat model supports reasoning using [vLLM](https://github.com/vllm-project/vllm). To use it, please install vLLM and set the environment variable `USE_VLLM=1`. The Chat model also supports reasoning using [OpenAI API](https://platform.openai.com/docs/api-reference/introduction). To use it, please run `openai_api_server.py` in `inference` and set the environment variable `USE_API=1`. This function is used to deploy inference server and demo server in different machine. If you need to customize the Jupyter kernel, you can specify it by `export IPYKERNEL=`. ## Usage GLM4 Demo has three modes: - All Tools mode - VLM mode - Text interpretation mode ### All Tools mode You can enhance the model's capabilities by registering new tools in `tool_registry.py`. Just use `@register_tool` decorated function to complete the registration. For tool declarations, the function name is the name of the tool, and the function docstring is the description of the tool; for tool parameters, use `Annotated[typ: type, description: str, required: bool]` to annotate the parameter type, description, and whether it is required. For example, the registration of the `get_weather` tool is as follows: ```python @register_tool def get_weather( city_name: Annotated[str, 'The name of the city to be queried', True], ) -> str: """ Get the weather for `city_name` in the following week """ ... ``` This mode is compatible with the tool registration process of ChatGLM3-6B. + Code capability, drawing capability, and networking capability have been automatically integrated. Users only need to configure the corresponding Key as required. + System prompt words are not supported in this mode. The model will automatically build prompt words. ## Text interpretation mode Users can upload documents and use the long text capability of GLM-4-9B to understand the text. It can parse pptx, docx, pdf and other files. + Tool calls and system prompt words are not supported in this mode. + If the text is very long, the model may require a high amount of GPU memory. Please confirm your hardware configuration. ## Image Understanding Mode Users can upload images and use the image understanding capabilities of GLM-4-9B to understand the images. + This mode must use the glm-4v-9b model. + Tool calls and system prompts are not supported in this mode. + The model can only understand and communicate with one image. If you need to change the image, you need to open a new conversation. + The supported image resolution is 1120 x 1120 ================================================ FILE: demo/composite_demo/browser/.gitignore ================================================ # Created by https://www.toptal.com/developers/gitignore/api/node # Edit at https://www.toptal.com/developers/gitignore?templates=node ### Node ### # Logs logs *.log npm-debug.log* yarn-debug.log* yarn-error.log* lerna-debug.log* .pnpm-debug.log* # Diagnostic reports (https://nodejs.org/api/report.html) report.[0-9]*.[0-9]*.[0-9]*.[0-9]*.json # Runtime data pids *.pid *.seed *.pid.lock # Directory for instrumented libs generated by jscoverage/JSCover lib-cov # Coverage directory used by tools like istanbul coverage *.lcov # nyc test coverage .nyc_output # Grunt intermediate storage (https://gruntjs.com/creating-plugins#storing-task-files) .grunt # Bower dependency directory (https://bower.io/) bower_components # node-waf configuration .lock-wscript # Compiled binary addons (https://nodejs.org/api/addons.html) build/Release # Dependency directories node_modules/ jspm_packages/ # Snowpack dependency directory (https://snowpack.dev/) web_modules/ # TypeScript cache *.tsbuildinfo # Optional npm cache directory .npm # Optional eslint cache .eslintcache # Optional stylelint cache .stylelintcache # Microbundle cache .rpt2_cache/ .rts2_cache_cjs/ .rts2_cache_es/ .rts2_cache_umd/ # Optional REPL history .node_repl_history # Output of 'npm pack' *.tgz # Yarn Integrity file .yarn-integrity # dotenv environment variable files .env .env.development.local .env.test.local .env.production.local .env.local # parcel-bundler cache (https://parceljs.org/) .cache .parcel-cache # Next.js build output .next out # Nuxt.js build / generate output .nuxt dist # Gatsby files .cache/ # Comment in the public line in if your project uses Gatsby and not Next.js # https://nextjs.org/blog/next-9-1#public-directory-support # public # vuepress build output .vuepress/dist # vuepress v2.x temp and cache directory .temp # Docusaurus cache and generated files .docusaurus # Serverless directories .serverless/ # FuseBox cache .fusebox/ # DynamoDB Local files .dynamodb/ # TernJS port file .tern-port # Stores VSCode versions used for testing VSCode extensions .vscode-test # yarn v2 .yarn/cache .yarn/unplugged .yarn/build-state.yml .yarn/install-state.gz .pnp.* ### Node Patch ### # Serverless Webpack directories .webpack/ # Optional stylelint cache # SvelteKit build / generate output .svelte-kit # End of https://www.toptal.com/developers/gitignore/api/node ================================================ FILE: demo/composite_demo/browser/package.json ================================================ { "name": "glm4-browser", "version": "1.0.0", "description": "Browser system for GLM-4", "main": "src/server.ts", "scripts": { "dev": "npx nodemon src/server", "start": "npx ts-node src/server.ts" }, "license": "MIT", "dependencies": { "express": "^4.18.3", "jsdom": "^24.0.0", "pnpm": "^9.1.2", "turndown": "^7.1.2", "winston": "^3.11.0" }, "devDependencies": { "@types/express": "^4.17.21", "@types/jsdom": "^21.1.6", "@types/node": "^20.11.20", "@types/turndown": "^5.0.4", "nodemon": "^3.1.0", "ts-node": "^10.9.2" } } ================================================ FILE: demo/composite_demo/browser/src/browser.ts ================================================ import { JSDOM } from 'jsdom'; import TurndownService from 'turndown'; import config from './config'; import { Message, ToolObservation } from './types'; import { logger, withTimeout } from './utils'; // represent a quote from a display interface Quote { text: string; metadata: Metadata[]; } interface ActionResult { contentType: string; metadataList?: TetherQuoteMetadata[]; metadata?: any; roleMetadata: string; message: string; } // represent a piece of metadata to be marked in the final answer interface Metadata { type: string; title: string; url: string; lines: string[]; } interface TetherQuoteExtra { cited_message_idx: number; evidence_text: string; } interface TetherQuoteMetadata { type: string; title: string; url: string; text: string; pub_date?: string; extra?: TetherQuoteExtra; } interface Citation { citation_format_type: string; start_ix: number; end_ix: number; metadata?: TetherQuoteMetadata; invalid_reason?: string; } interface PageState { aCounter: number; imgCounter: number; url: URL; url_string: string; hostname: string; links: string[]; links_meta: TetherQuoteMetadata[]; lines: string[]; line_source: Record; // string representation of number interval title?: string; } interface BrowserState { pageStack: PageState[]; quoteCounter: number; quotes: Record; } function removeDenseLinks(document: Document, ratioThreshold: number = 0.5) { // Remove nav elements const navs = document.querySelectorAll('nav'); navs.forEach(nav => { if (nav.parentNode) { nav.parentNode.removeChild(nav); } }); // Query for lists, divs, spans, tables, and paragraphs const elements = document.querySelectorAll('ul, ol, div, span, nav, table, p'); elements.forEach(element => { if (element === null) return; const children = Array.from(element.childNodes); const links = element.querySelectorAll('a'); if (children.length <= 1) return; const allText = element.textContent ? element.textContent.trim().replace(/\s+/g, '') : ''; const linksText = Array.from(links) .map(link => (link.textContent ? link.textContent.trim() : '')) .join('') .replace(/\s+/g, ''); if (allText.length === 0 || linksText.length === 0) return; let ratio = linksText.length / allText.length; if (ratio > ratioThreshold && element.parentNode) { element.parentNode.removeChild(element); } }); } abstract class BaseBrowser { public static toolName = 'browser' as const; public description = 'BaseBrowser'; private turndownService = new TurndownService({ headingStyle: 'atx', }); private state: BrowserState; private transform(dom: JSDOM): string { let state = this.lastPageState(); state.aCounter = 0; state.imgCounter = 0; state.links = []; return this.turndownService.turndown(dom.window.document); } private formatPage(state: PageState): string { let formatted_lines = state.lines.join('\n'); let formatted_title = state.title ? `TITLE: ${state.title}\n\n` : ''; let formatted_range = `\nVisible: 0% - 100%`; let formatted_message = formatted_title + formatted_lines + formatted_range; return formatted_message; } private newPageState(): PageState { return { aCounter: 0, imgCounter: 0, url: new URL('about:blank'), url_string: 'about:blank', hostname: '', title: '', links: [], links_meta: [], lines: [], line_source: {}, }; } private pushPageState(): PageState { let state = this.newPageState(); this.state.pageStack.push(state); return state; } private lastPageState(): PageState { if (this.state.pageStack.length === 0) { throw new Error('No page state'); } return this.state.pageStack[this.state.pageStack.length - 1]; } private formatErrorUrl(url: string): string { let TRUNCATION_LIMIT = 80; if (url.length <= TRUNCATION_LIMIT) { return url; } return url.slice(0, TRUNCATION_LIMIT) + `... (URL truncated at ${TRUNCATION_LIMIT} chars)`; } protected functions = { search: async (query: string, recency_days: number = -1) => { logger.debug(`Searching for: ${query}`); const search = new URLSearchParams({ q: query }); recency_days > 0 && search.append('recency_days', recency_days.toString()); if (config.CUSTOM_CONFIG_ID) { search.append('customconfig', config.CUSTOM_CONFIG_ID.toString()); } const url = `${config.BING_SEARCH_API_URL}/search?${search.toString()}`; console.log('Full URL:', url); // 输出完整的 URL查看是否正确 return withTimeout( config.BROWSER_TIMEOUT, fetch(url, { headers: { 'Ocp-Apim-Subscription-Key': config.BING_SEARCH_API_KEY, } }) .then( res => res.json() as Promise<{ queryContext: { originalQuery: string; }; webPages: { webSearchUrl: string; totalEstimatedMatches: number; value: { id: string; name: string; url: string; datePublished: string; // 2018-05-18T08:00:00.0000000 datePublishedDisplayText: string; isFamilyFriendly: boolean; displayUrl: string; snippet: string; dateLastCrawled: string; cachedPageUrl: string; language: string; isNavigational: boolean; }[]; }; rankingResponse: { mainline: { items: { answerType: 'WebPages'; resultIndex: number; value: { id: string; }; }[]; }; }; }>, ), ) .then(async ({ value: res }) => { try { let state = this.pushPageState(); let metadataList: TetherQuoteMetadata[] = []; for (const [i, entry] of res.webPages.value.entries()) { const url = new URL(entry.url); const hostname = url.hostname; state.lines.push(` # 【${i}†${entry.name}†${hostname}】`); state.lines.push(entry.snippet); const quoteMetadata: Metadata = { type: 'webpage', title: entry.name, url: entry.url, lines: state.lines.slice(2 * i, 2 * i + 2), }; state.line_source[`${2 * i}-${2 * i + 1}`] = quoteMetadata; state.links[i] = entry.url; const returnMetadata: TetherQuoteMetadata = { type: quoteMetadata.type, title: quoteMetadata.title, url: quoteMetadata.url, text: state.lines[2 * i + 1], // only content, not link pub_date: entry.datePublished, }; metadataList.push(returnMetadata); } const returnContentType = 'browser_result'; return { contentType: returnContentType, roleMetadata: returnContentType, message: this.formatPage(state), metadataList, }; } catch (err) { throw new Error(`parse error: ${err}`); } }) .catch(err => { logger.error(`搜索请求失败：${query}，错误信息：${err.message}`); if (err.code === 'ECONNABORTED') { throw new Error(`Timeout while executing search for: ${query}`); } throw new Error(`网络或服务器发生错误，请检查URL: ${url}`); }); }, open_url: (url: string) => { logger.debug(`Opening ${url}`); return withTimeout( config.BROWSER_TIMEOUT, fetch(url).then(res => res.text()), ) .then(async ({ value: res, time }) => { try { const state = this.pushPageState(); state.url = new URL(url); state.url_string = url; state.hostname = state.url.hostname; const html = res; const dom = new JSDOM(html); const title = dom.window.document.title; const markdown = this.transform(dom); state.title = title; // Remove first line, because it will be served as the title const lines = markdown.split('\n'); lines.shift(); // Remove consequent empty lines let i = 0; while (i < lines.length - 1) { if (lines[i].trim() === '' && lines[i + 1].trim() === '') { lines.splice(i, 1); } else { i++; } } let page = lines.join('\n'); // The first line feed is not a typo let text_result = `\nURL: ${url}\n${page}`; state.lines = text_result.split('\n'); // all lines has only one source state.line_source = {}; state.line_source[`0-${state.lines.length - 1}`] = { type: 'webpage', title: title, url: url, lines: state.lines, }; let message = this.formatPage(state); const returnContentType = 'browser_result'; return { contentType: returnContentType, roleMetadata: returnContentType, message, metadataList: state.links_meta, }; } catch (err) { throw new Error(`parse error: ${err}`); } }) .catch(err => { logger.error(err.message); if (err.code === 'ECONNABORTED') { throw new Error(`Timeout while loading page w/ URL: ${url}`); } throw new Error(`Failed to load page w/ URL: ${url}`); }); }, mclick: (ids: number[]) => { logger.info('Entering mclick', ids); let promises: Promise[] = []; let state = this.lastPageState(); for (let id of ids) { if (isNaN(id) || id >= state.links.length) { promises.push( Promise.reject( new Error( `recorded='click(${id})' temporary=None permanent=None new_state=None final=None success=False feedback='Error parsing ID ${id}' metadata={}`, ), ), ); continue; } let url: string; try { url = new URL(state.links[id], state.url).href; } catch (err) { logger.error(`Failed in getting ${state.links[id]}, ${state.url}`); promises.push( Promise.reject( new Error( `recorded='click(${id})' temporary=None permanent='${err}' new_state=None final=None success=False feedback='Error parsing URL for ID ${id}' metadata={}`, ), ), ); continue; } const quoteIndex = this.state.quoteCounter++; // ascending in final results promises.push( withTimeout( config.BROWSER_TIMEOUT, fetch(url).then(res => res.text()), ) .then(({ value: res, time }) => { let state = this.newPageState(); state.url = new URL(url); state.hostname = state.url.hostname; try { const html = res; const dom = new JSDOM(html); const title = dom.window.document.title; state.title = title; removeDenseLinks(dom.window.document); let quoteText = this.transform(dom); // remove consecutive newline quoteText = quoteText.replace(/[\r\n]+/g, '\n'); const quoteLines = quoteText.split('\n'); state.lines = quoteLines; const metadata = { type: 'webpage', title: title, url: url, lines: quoteLines, }; const quoteMetadata = { type: 'webpage', title: title, url: url, text: quoteText, }; state.line_source = {}; state.line_source[`0-${state.lines.length - 1}`] = metadata; this.state.quotes[quoteIndex.toString()] = { text: quoteText, metadata: [metadata], }; const returnContentType = 'quote_result'; return { contentType: returnContentType, roleMetadata: `${returnContentType} [${quoteIndex}†source]`, message: quoteText, metadataList: [quoteMetadata], metadata: { url, }, }; } catch (err) { throw new Error(`parse error: ${err}`); } }) .catch(err => { logger.error(err.message); if (err.code === 'ECONNABORTED') { throw new Error(`Timeout while loading page w/ URL: ${this.formatErrorUrl(url)}`); } throw new Error(`Failed to load page w/ URL: ${this.formatErrorUrl(url)}`); }) .catch(err => { // format error message const returnContentType = 'system_error'; throw { contentType: returnContentType, roleMetadata: returnContentType, message: `recorded='click(${id})' temporary=None permanent='${ err.message }' new_state=None final=None success=False feedback='Error fetching url ${this.formatErrorUrl( url, )}' metadata={}`, metadata: { failedURL: url, }, } as ActionResult; }), ); } return Promise.allSettled(promises).then(async results => { const actionResults = results.map(r => { if (r.status === 'fulfilled') { return r.value; } else { logger.error(r.reason); return r.reason as ActionResult; } }); if (results.filter(r => r.status === 'fulfilled').length === 0) { // collect errors const err_text = (results as PromiseRejectedResult[]) .map(r => (r.reason as ActionResult).message) .join('\n'); throw new Error(err_text); } else { return actionResults; } }); }, }; constructor() { this.state = { pageStack: [], quotes: {}, quoteCounter: 7, }; this.turndownService.remove('script'); this.turndownService.remove('style'); // Add rules for turndown this.turndownService.addRule('reference', { filter: function (node, options: any): boolean { return ( options.linkStyle === 'inlined' && node.nodeName === 'A' && node.getAttribute('href') !== undefined ); }, replacement: (content, node, options): string => { let state = this.state.pageStack[this.state.pageStack.length - 1]; if (!content || !('getAttribute' in node)) return ''; let href = undefined; try { if ('getAttribute' in node) { const hostname = new URL(node.getAttribute('href')!).hostname; // Do not append hostname when in the same domain if (hostname === state.hostname || !hostname) { href = ''; } else { href = '†' + hostname; } } } catch (e) { // To prevent displaying links like '/foo/bar' href = ''; } if (href === undefined) return ''; const url = node.getAttribute('href')!; let linkId = state.links.findIndex(link => link === url); if (linkId === -1) { linkId = state.aCounter++; // logger.debug(`New link[${linkId}]: ${url}`); state.links_meta.push({ type: 'webpage', title: node.textContent!, url: href, text: node.textContent!, }); state.links.push(url); } return `【${linkId}†${node.textContent}${href}】`; }, }); this.turndownService.addRule('img', { filter: 'img', replacement: (content, node, options): string => { let state = this.state.pageStack[this.state.pageStack.length - 1]; return `[Image ${state.imgCounter++}]`; }, }); // Just to change indentation, wondering why this isn't exposed as an option this.turndownService.addRule('list', { filter: 'li', replacement: function (content, node, options) { content = content .replace(/^\n+/, '') // remove leading newlines .replace(/\n+$/, '\n') // replace trailing newlines with just a single one .replace(/\n/gm, '\n '); // indent let prefix = options.bulletListMarker + ' '; const parent = node.parentNode! as Element; if (parent.nodeName === 'OL') { const start = parent.getAttribute('start'); const index = Array.prototype.indexOf.call(parent.children, node); prefix = (start ? Number(start) + index : index + 1) + '. '; } return ' ' + prefix + content + (node.nextSibling && !/\n$/.test(content) ? '\n' : ''); }, }); // Remove bold; remove() doesn't work on this, I don't know why this.turndownService.addRule('emph', { filter: ['strong', 'b'], replacement: function (content, node, options) { if (!content.trim()) return ''; return content; }, }); } abstract actionLine(content: string): Promise; async action(content: string): Promise { const lines = content.split('\n'); let results: ActionResult[] = []; for (const line of lines) { logger.info(`Action line: ${line}`) try { const lineActionResult = await this.actionLine(line); logger.debug(`Action line result: ${JSON.stringify(lineActionResult, null, 2)}`); if (Array.isArray(lineActionResult)) { results = results.concat(lineActionResult); } else { results.push(lineActionResult); } } catch (err) { const returnContentType = 'system_error'; results.push({ contentType: returnContentType, roleMetadata: returnContentType, message: `Error when executing command ${line}\n${err}`, metadata: { failedCommand: line, }, }); } } const observations: ToolObservation[] = []; for (const result of results) { const observation: ToolObservation = { contentType: result.contentType, result: result.message, roleMetadata: result.roleMetadata, metadata: result.metadata ?? {}, }; if (result.metadataList) { observation.metadata.metadata_list = result.metadataList; } observations.push(observation); } return observations; } postProcess(message: Message, metadata: any) { const quotePattern = /【(.+?)†(.*?)】/g; const content = message.content; let match; let citations: Citation[] = []; const citation_format_type = 'tether_og'; while ((match = quotePattern.exec(content))) { logger.debug(`Citation match: ${match[0]}`); const start_ix = match.index; const end_ix = match.index + match[0].length; let invalid_reason = undefined; let metadata: TetherQuoteMetadata; try { let cited_message_idx = parseInt(match[1]); let evidence_text = match[2]; let quote = this.state.quotes[cited_message_idx.toString()]; if (quote === undefined) { invalid_reason = `'Referenced message ${cited_message_idx} in citation 【${cited_message_idx}†${evidence_text}】 is not a quote or tether browsing display.'`; logger.error(`Triggered citation error with quote undefined: ${invalid_reason}`); citations.push({ citation_format_type, start_ix, end_ix, invalid_reason, }); } else { let extra: TetherQuoteExtra = { cited_message_idx, evidence_text, }; const quote_metadata = quote.metadata[0]; metadata = { type: 'webpage', title: quote_metadata.title, url: quote_metadata.url, text: quote_metadata.lines.join('\n'), extra, }; citations.push({ citation_format_type, start_ix, end_ix, metadata, }); } } catch (err) { logger.error(`Triggered citation error: ${err}`); invalid_reason = `Citation Error: ${err}`; citations.push({ start_ix, end_ix, citation_format_type, invalid_reason, }); } } metadata.citations = citations; } getState() { return this.state; } } export class SimpleBrowser extends BaseBrowser { public description = 'SimpleBrowser'; constructor() { super(); } async actionLine(content: string): Promise { const regex = /(\w+)\(([^)]*)\)/; const matches = content.match(regex); if (matches) { const functionName = matches[1]; let args_string = matches[2]; if (functionName === 'mclick') { args_string = args_string.trim().slice(1, -1); // remove '[' and ']' } const args = args_string.split(',').map(arg => arg.trim()); let result; switch (functionName) { case 'search': logger.debug(`SimpleBrowser action search ${args[0].slice(1, -1)}`); const recency_days = /(^|\D)(\d+)($|\D)/.exec(args[1])?.[2] as undefined | `${number}`; result = await this.functions.search( args[0].slice(1, -1), // slice quote "query" recency_days && Number(recency_days), ); break; case 'open_url': logger.debug(`SimpleBrowser action open_url ${args[0].slice(1, -1)}`); result = await this.functions.open_url(args[0].slice(1, -1)); break; case 'mclick': logger.debug(`SimpleBrowser action mclick ${args}`); result = await this.functions.mclick(args.map(x => parseInt(x))); break; default: throw new Error(`Parse Error: ${content}`); } return result; } else { throw new Error('Parse Error'); } } } if (require.main === module) { (async () => { let browser = new SimpleBrowser(); let demo = async (action: string) => { logger.info(` ------ Begin of Action: ${action} ------`); let results = await browser.action(action); for (const [idx, result] of results.entries()) { logger.info(`[Result ${idx}] contentType: ${result.contentType}`); logger.info(`[Result ${idx}] roleMetadata: ${result.roleMetadata}`); logger.info(`[Result ${idx}] result: ${result.result}`); logger.info(`[Result ${idx}] metadata: ${JSON.stringify(result.metadata, null, 2)}`); } logger.info(` ------ End of Action: ${action} ------\n\n`); }; await demo("search('Apple Latest News')"); await demo('mclick([0, 1, 5, 6])'); await demo('mclick([1, 999999])'); await demo("open_url('https://chatglm.cn')"); await demo("search('zhipu latest News')"); await demo('mclick([0, 1, 5, 6])'); })(); } ================================================ FILE: demo/composite_demo/browser/src/config.ts ================================================ export default { LOG_LEVEL: 'debug', BROWSER_TIMEOUT: 10000, BING_SEARCH_API_URL: 'https://api.bing.microsoft.com/v7.0/custom/', BING_SEARCH_API_KEY: 'YOUR_BING_SEARCH_API_KEY', CUSTOM_CONFIG_ID : 'YOUR_CUSTOM_CONFIG_ID', //将您的Custom Configuration ID放在此处 HOST: 'localhost', PORT: 3000, }; ================================================ FILE: demo/composite_demo/browser/src/server.ts ================================================ import express, { Express, Request, Response } from 'express'; import { SimpleBrowser } from './browser'; import config from './config'; import { logger } from './utils'; const session_history: Record = {}; const app: Express = express(); app.use(express.json()); app.post('/', async (req: Request, res: Response) => { const { session_id, action, }: { session_id: string; action: string; } = req.body; logger.info(`session_id: ${session_id}`); logger.info(`action: ${action}`); if (!session_history[session_id]) { session_history[session_id] = new SimpleBrowser(); } const browser = session_history[session_id]; try { res.json(await browser.action(action)); } catch (err) { logger.error(err); res.status(400).json(err); } }) process.on('SIGINT', () => { process.exit(0); }); process.on('uncaughtException', e => { logger.error(e); }); const { HOST, PORT } = config; (async () => { app.listen(PORT, HOST, () => { logger.info(`⚡️[server]: Server is running at http://${HOST}:${PORT}`); try { (process).send('ready'); } catch (err) {} }); })(); ================================================ FILE: demo/composite_demo/browser/src/types.ts ================================================ export interface File { id: string; name: string; size: number; } export interface Metadata { files?: File[]; reference?: string; } export interface Message { role: 'user' | 'assistant' | 'system' | 'observation'; metadata: string; content: string; request_metadata?: Metadata; } export interface ToolObservation { contentType: string; result: string; text?: string; roleMetadata?: string; // metadata for <|observation|>${metadata} metadata: any; // metadata for response } ================================================ FILE: demo/composite_demo/browser/src/utils.ts ================================================ import winston from 'winston'; import config from './config'; export class TimeoutError extends Error {} const logLevel = config.LOG_LEVEL; export const logger = winston.createLogger({ level: logLevel, format: winston.format.combine( winston.format.colorize(), winston.format.printf(info => { return `${info.level}: ${info.message}`; }), ), transports: [new winston.transports.Console()], }); console.log('LOG_LEVEL', logLevel); export const parseHrtimeToMillisecond = (hrtime: [number, number]): number => { return (hrtime[0] + hrtime[1] / 1e9) * 1000; }; export const promiseWithTime = ( promise: Promise ): Promise<{ value: T; time: number; }> => { return new Promise((resolve, reject) => { const startTime = process.hrtime(); promise .then(value => { resolve({ value: value, time: parseHrtimeToMillisecond(process.hrtime(startTime)) }); }) .catch(err => reject(err)); }); }; export const withTimeout = ( millis: number, promise: Promise ): Promise<{ value: T; time: number; }> => { const timeout = new Promise<{ value: T; time: number }>((_, reject) => setTimeout(() => reject(new TimeoutError()), millis) ); return Promise.race([promiseWithTime(promise), timeout]); }; ================================================ FILE: demo/composite_demo/browser/tsconfig.json ================================================ { "compilerOptions": { "target": "es2022", "lib": ["es2022", "dom"], "module": "commonjs", "rootDir": "./", "outDir": "./dist", "esModuleInterop": true, "forceConsistentCasingInFileNames": true, "strict": true, }, "ts-node": { "transpileOnly": true } } ================================================ FILE: demo/composite_demo/requirements.txt ================================================ # Please install the requirments.txt in inference first! ipykernel>=6.26.0 ipython>=8.18.1 jupyter_client>=8.6.0 langchain>=0.2.12 langchain-community>=0.2.11 matplotlib>=3.9.1 pymupdf>=1.24.9 python-docx>=1.1.2 python-pptx>=0.6.23 pyyaml>=6.0.1 requests>=2.31.0 streamlit>=1.37.1 zhipuai>=2.1.4 ================================================ FILE: demo/composite_demo/src/client.py ================================================ """ This is a client part of composite_demo. We provide two clients, HFClient and VLLMClient, which are used to interact with the model. The HFClient is used to interact with the transformers backend, and the VLLMClient is used to interact with the VLLM model. """ import json from collections.abc import Generator from copy import deepcopy from enum import Enum, auto from typing import Protocol import streamlit as st from conversation import Conversation, build_system_prompt from tools.tool_registry import ALL_TOOLS class ClientType(Enum): HF = auto() VLLM = auto() API = auto() class Client(Protocol): def __init__(self, model_path: str): ... def generate_stream( self, tools: list[dict], history: list[Conversation], **parameters, ) -> Generator[tuple[str | dict, list[dict]]]: ... def process_input(history: list[dict], tools: list[dict], role_name_replace: dict = None) -> list[dict]: chat_history = [] # if len(tools) > 0: chat_history.append({"role": "system", "content": build_system_prompt(list(ALL_TOOLS), tools)}) for conversation in history: role = str(conversation.role).removeprefix("<|").removesuffix("|>") if role_name_replace: role = role_name_replace.get(role, role) item = { "role": role, "content": conversation.content, } if conversation.metadata: item["metadata"] = conversation.metadata # Only append image for user if role == "user" and conversation.image: item["image"] = conversation.image chat_history.append(item) return chat_history def process_response(output, history): content = "" history = deepcopy(history) for response in output.split("<|assistant|>"): if "\n" in response: metadata, content = response.split("\n", maxsplit=1) else: metadata, content = "", response if not metadata.strip(): content = content.strip() history.append({"role": "assistant", "metadata": metadata, "content": content}) content = content.replace("[[训练时间]]", "2023年") else: history.append({"role": "assistant", "metadata": metadata, "content": content}) if history[0]["role"] == "system" and "tools" in history[0]: parameters = json.loads(content) content = {"name": metadata.strip(), "parameters": parameters} else: content = {"name": metadata.strip(), "content": content} return content, history # glm-4v-9b is not available in vLLM backend, use HFClient instead. @st.cache_resource(max_entries=1, show_spinner="Loading model...") def get_client(model_path, typ: ClientType) -> Client: match typ: case ClientType.HF: from clients.hf import HFClient return HFClient(model_path) case ClientType.VLLM: try: from clients.vllm import VLLMClient except ImportError as e: e.msg += "; did you forget to install vLLM?" raise return VLLMClient(model_path) case ClientType.API: from clients.openai import APIClient return APIClient(model_path) raise NotImplementedError(f"Client type {typ} is not supported.") ================================================ FILE: demo/composite_demo/src/clients/hf.py ================================================ """ HuggingFace client. """ from collections.abc import Generator from threading import Thread import torch from client import Client, process_input, process_response from conversation import Conversation from transformers import AutoModelForCausalLM, AutoTokenizer, TextIteratorStreamer class HFClient(Client): def __init__(self, model_path: str): self.tokenizer = AutoTokenizer.from_pretrained( model_path, trust_remote_code=True, ) self.model = AutoModelForCausalLM.from_pretrained( model_path, torch_dtype=torch.bfloat16, device_map="cuda", ).eval() def generate_stream( self, tools: list[dict], history: list[Conversation], **parameters, ) -> Generator[tuple[str | dict, list[dict]]]: chat_history = process_input(history, tools) model_inputs = self.tokenizer.apply_chat_template( chat_history, add_generation_prompt=True, tokenize=True, return_tensors="pt", return_dict=True, ).to(self.model.device) streamer = TextIteratorStreamer( tokenizer=self.tokenizer, timeout=5, skip_prompt=True, ) generate_kwargs = { **model_inputs, "streamer": streamer, "eos_token_id": [151329, 151336, 151338], "do_sample": True, } generate_kwargs.update(parameters) t = Thread(target=self.model.generate, kwargs=generate_kwargs) t.start() total_text = "" for token_text in streamer: total_text += token_text yield process_response(total_text, chat_history) ================================================ FILE: demo/composite_demo/src/clients/openai.py ================================================ """ OpenAI API client. """ from collections.abc import Generator from client import Client, process_input, process_response from conversation import Conversation from openai import OpenAI def format_openai_tool(origin_tools): openai_tools = [] for tool in origin_tools: openai_param = {} for param in tool["params"]: openai_param[param["name"]] = {} openai_tool = { "type": "function", "function": { "name": tool["name"], "description": tool["description"], "parameters": { "type": "object", "properties": { param["name"]: {"type": param["type"], "description": param["description"]} for param in tool["params"] }, "required": [param["name"] for param in tool["params"] if param["required"]], }, }, } openai_tools.append(openai_tool) return openai_tools class APIClient(Client): def __init__(self, model_path: str): base_url = "http://127.0.0.1:8000/v1/" self.client = OpenAI(api_key="EMPTY", base_url=base_url) self.use_stream = False self.role_name_replace = {"observation": "tool"} def generate_stream( self, tools: list[dict], history: list[Conversation], **parameters, ) -> Generator[tuple[str | dict, list[dict]]]: chat_history = process_input(history, "", role_name_replace=self.role_name_replace) # messages = process_input(history, '', role_name_replace=self.role_name_replace) openai_tools = format_openai_tool(tools) response = self.client.chat.completions.create( model="glm-4", messages=chat_history, tools=openai_tools, stream=self.use_stream, max_tokens=parameters["max_new_tokens"], temperature=parameters["temperature"], presence_penalty=1.2, top_p=parameters["top_p"], tool_choice="auto", ) output = response.choices[0].message if output.tool_calls: glm4_output = output.tool_calls[0].function.name + "\n" + output.tool_calls[0].function.arguments else: glm4_output = output.content yield process_response(glm4_output, chat_history) ================================================ FILE: demo/composite_demo/src/clients/vllm.py ================================================ """ vLLM client. Please install [vLLM](https://github.com/vllm-project/vllm) according to its installation guide before running this client. """ import time from collections.abc import Generator from client import Client, process_input, process_response from conversation import Conversation from transformers import AutoTokenizer from vllm import EngineArgs, LLMEngine, SamplingParams class VLLMClient(Client): def __init__(self, model_path: str): self.tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True) self.engine_args = EngineArgs( model=model_path, tensor_parallel_size=1, dtype="bfloat16", # torch.bfloat16 is needed. gpu_memory_utilization=0.6, enforce_eager=True, worker_use_ray=False, ) self.engine = LLMEngine.from_engine_args(self.engine_args) def generate_stream( self, tools: list[dict], history: list[Conversation], **parameters ) -> Generator[tuple[str | dict, list[dict]]]: chat_history = process_input(history, tools) model_inputs = self.tokenizer.apply_chat_template(chat_history, add_generation_prompt=True, tokenize=False) parameters["max_tokens"] = parameters.pop("max_new_tokens") params_dict = { "n": 1, "best_of": 1, "top_p": 1, "top_k": -1, "length_penalty": 1, "stop_token_ids": [151329, 151336, 151338], } params_dict.update(parameters) sampling_params = SamplingParams(**params_dict) self.engine.add_request(request_id=str(time.time()), inputs=model_inputs, params=sampling_params) while self.engine.has_unfinished_requests(): request_outputs = self.engine.step() for request_output in request_outputs: yield process_response(request_output.outputs[0].text, chat_history) ================================================ FILE: demo/composite_demo/src/conversation.py ================================================ import json import re from dataclasses import dataclass from datetime import datetime from enum import Enum, auto import streamlit as st from PIL.Image import Image from streamlit.delta_generator import DeltaGenerator from tools.browser import Quote, quotes QUOTE_REGEX = re.compile(r"【(\d+)†(.+?)】") SELFCOG_PROMPT = "你是一个名为 GLM-4 的人工智能助手。你是基于智谱AI训练的语言模型 GLM-4 模型开发的，你的任务是针对用户的问题和要求提供适当的答复和支持。" DATE_PROMPT = "当前日期: %Y-%m-%d" TOOL_SYSTEM_PROMPTS = { "python": "当你向 `python` 发送包含 Python 代码的消息时，该代码将会在一个有状态的 Jupyter notebook 环境中执行。\n`python` 返回代码执行的输出，或在执行 60 秒后返回超时。\n`/mnt/data` 将会持久化存储你的文件。在此会话中，`python` 无法访问互联网。不要使用 `python` 进行任何网络请求或者在线 API 调用，这些在线内容的访问将不会成功。", "simple_browser": "你可以使用 `simple_browser` 工具。该工具支持以下函数：\n`search(query: str, recency_days: int)`：使用搜索引擎进行查询并显示结果，可以使用 `recency_days` 参数控制搜索内容的时效性。\n`mclick(ids: list[int])`：获取一系列指定 id 的页面内容。每次调用时，须选择3-10个页面。选择多个角度的页面，同时尽可能选择可信任的信息来源。考虑到部分页面是无法加载的，你也可以多打开一些可能有用的页面而不用担心内容过多。\n`open_url(url: str)`：打开指定的 URL。\n\n使用 `【{引用 id}†{引用文本}】` 来引用内容。\n\n操作步骤：1. 使用 `search` 来获得信息列表; 2. 使用 `mclick` 来获取指定 ID 页面的内容; 3. 根据获得的内容进行回复。在回复中应当引用信息来源。\n 如果用户提供了 URL，也可以用 `open_url` 直接打开页面。\n如果初次搜索结果没有找到合适的信息，也可以再次使用 `search` 进行搜索。", "cogview": "如果用户的请求中包含了对图像的描述，你可以使用 `cogview` 来生成图像并展示给用户。你需要向 `cogview` 发送图像描述，规则：\n- 发送给 `cogview` 的消息必须使用英语。用户的中文描述必须完全翻译为英语。\n- 应当尽可能详细地描述图像生成的需求，需求描述约 100 英文单词。\n- 保持用户原始描述的意图。不要虚构内容或者没见过的人物。\n- 如无特殊说明，所在地为中国，持有中国立场并遵循中国社会主义价值观。", } FILE_TEMPLATE = "[File Name]\n{file_name}\n[File Content]\n{file_content}" def build_system_prompt( enabled_tools: list[str], functions: list[dict], ): value = SELFCOG_PROMPT value += "\n\n" + datetime.now().strftime(DATE_PROMPT) if enabled_tools or functions: value += "\n\n# 可用工具" contents = [] for tool in enabled_tools: contents.append(f"\n\n## {tool}\n\n{TOOL_SYSTEM_PROMPTS[tool]}") for function in functions: content = f"\n\n## {function['name']}\n\n{json.dumps(function, ensure_ascii=False, indent=4)}" content += "\n在调用上述函数时，请使用 Json 格式表示调用的参数。" contents.append(content) value += "".join(contents) return value def response_to_str(response: str | dict[str, str]) -> str: """ Convert response to string. """ if isinstance(response, dict): return response.get("name", "") + response.get("content", "") return response class Role(Enum): SYSTEM = auto() USER = auto() ASSISTANT = auto() TOOL = auto() OBSERVATION = auto() def __str__(self): match self: case Role.SYSTEM: return "<|system|>" case Role.USER: return "<|user|>" case Role.ASSISTANT | Role.TOOL: return "<|assistant|>" case Role.OBSERVATION: return "<|observation|>" # Get the message block for the given role def get_message(self): # Compare by value here, because the enum object in the session state # is not the same as the enum cases here, due to streamlit's rerunning # behavior. match self.value: case Role.SYSTEM.value: return case Role.USER.value: return st.chat_message(name="user", avatar="user") case Role.ASSISTANT.value: return st.chat_message(name="assistant", avatar="assistant") case Role.TOOL.value: return st.chat_message(name="tool", avatar="assistant") case Role.OBSERVATION.value: return st.chat_message(name="observation", avatar="assistant") case _: st.error(f"Unexpected role: {self}") @dataclass class Conversation: role: Role content: str | dict # Processed content saved_content: str | None = None metadata: str | None = None image: str | Image | None = None def __str__(self) -> str: metadata_str = self.metadata if self.metadata else "" return f"{self.role}{metadata_str}\n{self.content}" # Human readable format def get_text(self) -> str: text = self.saved_content or self.content match self.role.value: case Role.TOOL.value: text = f"Calling tool `{self.metadata}`:\n\n```python\n{text}\n```" case Role.OBSERVATION.value: text = f"```python\n{text}\n```" return text # Display as a markdown block def show(self, placeholder: DeltaGenerator | None = None) -> str: if placeholder: message = placeholder else: message = self.role.get_message() if self.image: message.image(self.image, width=512) if self.role == Role.OBSERVATION: metadata_str = f"from {self.metadata}" if self.metadata else "" message = message.expander(f"Observation {metadata_str}") text = self.get_text() if self.role != Role.USER: show_text = text else: splitted = text.split("files uploaded.\n") if len(splitted) == 1: show_text = text else: # Show expander for document content doc = splitted[0] show_text = splitted[-1] expander = message.expander("File Content") expander.markdown(doc) message.markdown(show_text) def postprocess_text(text: str, replace_quote: bool) -> str: text = text.replace(r"\(", "$") text = text.replace(r"\)", "$") text = text.replace(r"\[", "$$") text = text.replace(r"\]", "$$") text = text.replace("<|assistant|>", "") text = text.replace("<|observation|>", "") text = text.replace("<|system|>", "") text = text.replace("<|user|>", "") text = text.replace("<|endoftext|>", "") # Replace quotes if replace_quote: for match in QUOTE_REGEX.finditer(text): quote_id = match.group(1) quote = quotes.get(quote_id, Quote("未找到引用内容", "")) text = text.replace(match.group(0), f" (来源：[{quote.title}]({quote.url})) ") return text.strip() ================================================ FILE: demo/composite_demo/src/main.py ================================================ """ This demo show the All tools and Long Context chat Capabilities of GLM-4. Please follow the Readme.md to run the demo. """ import os import traceback from enum import Enum from io import BytesIO from uuid import uuid4 import streamlit as st from client import Client, ClientType, get_client from conversation import ( FILE_TEMPLATE, Conversation, Role, postprocess_text, response_to_str, ) from PIL import Image from streamlit.delta_generator import DeltaGenerator from tools.tool_registry import dispatch_tool, get_tools from utils import extract_docx, extract_pdf, extract_pptx, extract_text CHAT_MODEL_PATH = os.environ.get("CHAT_MODEL_PATH", "THUDM/glm-4-9b-chat") VLM_MODEL_PATH = os.environ.get("VLM_MODEL_PATH", "THUDM/glm-4v-9b") USE_VLLM = os.environ.get("USE_VLLM", "0") == "1" USE_API = os.environ.get("USE_API", "0") == "1" class Mode(str, Enum): ALL_TOOLS = "🛠️ All Tools" LONG_CTX = "📝 文档解读" VLM = "🖼️ 多模态" def append_conversation( conversation: Conversation, history: list[Conversation], placeholder: DeltaGenerator | None = None, ) -> None: """ Append a conversation piece into history, meanwhile show it in a new markdown block """ history.append(conversation) conversation.show(placeholder) st.set_page_config( page_title="GLM-4 Demo", page_icon=":robot:", layout="centered", initial_sidebar_state="expanded", ) st.title("GLM-4 Demo") st.markdown( "智谱AI 公开在线技术文档: https://zhipu-ai.feishu.cn/wiki/RuMswanpkiRh3Ok4z5acOABBnjf \n\n 更多 GLM-4 开源模型的使用方法请参考文档。", unsafe_allow_html=True, ) with st.sidebar: top_p = st.slider("top_p", 0.0, 1.0, 0.8, step=0.01) top_k = st.slider("top_k", 1, 20, 10, step=1, key="top_k") temperature = st.slider("temperature", 0.0, 1.5, 0.95, step=0.01) repetition_penalty = st.slider("repetition_penalty", 0.0, 2.0, 1.0, step=0.01) max_new_tokens = st.slider("max_new_tokens", 1, 4096, 2048, step=1) cols = st.columns(2) export_btn = cols[0] clear_history = cols[1].button("Clear", use_container_width=True) retry = export_btn.button("Retry", use_container_width=True) if clear_history: page = st.session_state.page client = st.session_state.client st.session_state.clear() st.session_state.page = page st.session_state.client = client st.session_state.files_uploaded = False st.session_state.uploaded_texts = "" st.session_state.uploaded_file_nums = 0 st.session_state.history = [] if "files_uploaded" not in st.session_state: st.session_state.files_uploaded = False if "session_id" not in st.session_state: st.session_state.session_id = uuid4() if "history" not in st.session_state: st.session_state.history = [] first_round = len(st.session_state.history) == 0 def build_client(mode: Mode) -> Client: match mode: case Mode.ALL_TOOLS: st.session_state.top_k = 10 typ = ClientType.VLLM if USE_VLLM else ClientType.HF typ = ClientType.API if USE_API else typ return get_client(CHAT_MODEL_PATH, typ) case Mode.LONG_CTX: st.session_state.top_k = 10 typ = ClientType.VLLM if USE_VLLM else ClientType.HF return get_client(CHAT_MODEL_PATH, typ) case Mode.VLM: st.session_state.top_k = 1 # vLLM is not available for VLM mode return get_client(VLM_MODEL_PATH, ClientType.HF) # Callback function for page change def page_changed() -> None: global client new_page: str = st.session_state.page st.session_state.history.clear() st.session_state.client = build_client(Mode(new_page)) page = st.radio( "选择功能", [mode.value for mode in Mode], key="page", horizontal=True, index=None, label_visibility="hidden", on_change=page_changed, ) HELP = """ ### 🎉 欢迎使用 GLM-4! 请在上方选取一个功能。每次切换功能时，将会重新加载模型并清空对话历史。 文档解读模式与 VLM 模式仅支持在第一轮传入文档或图像。 """.strip() if page is None: st.markdown(HELP) exit() if page == Mode.LONG_CTX: if first_round: uploaded_files = st.file_uploader( "上传文件", type=["pdf", "txt", "py", "docx", "pptx", "json", "cpp", "md"], accept_multiple_files=True, ) if uploaded_files and not st.session_state.files_uploaded: uploaded_texts = [] for uploaded_file in uploaded_files: file_name: str = uploaded_file.name random_file_name = str(uuid4()) file_extension = os.path.splitext(file_name)[1] file_path = os.path.join("/tmp", random_file_name + file_extension) with open(file_path, "wb") as f: f.write(uploaded_file.getbuffer()) if file_name.endswith(".pdf"): content = extract_pdf(file_path) elif file_name.endswith(".docx"): content = extract_docx(file_path) elif file_name.endswith(".pptx"): content = extract_pptx(file_path) else: content = extract_text(file_path) uploaded_texts.append(FILE_TEMPLATE.format(file_name=file_name, file_content=content)) os.remove(file_path) st.session_state.uploaded_texts = "\n\n".join(uploaded_texts) st.session_state.uploaded_file_nums = len(uploaded_files) else: st.session_state.uploaded_texts = "" st.session_state.uploaded_file_nums = 0 elif page == Mode.VLM: if first_round: uploaded_image = st.file_uploader( "上传图片", type=["png", "jpg", "jpeg", "bmp", "tiff", "webp"], accept_multiple_files=False, ) if uploaded_image: data: bytes = uploaded_image.read() image = Image.open(BytesIO(data)).convert("RGB") st.session_state.uploaded_image = image else: st.session_state.uploaded_image = None prompt_text = st.chat_input("Chat with GLM-4!", key="chat_input") if prompt_text == "" and retry == False: print("\n== Clean ==\n") st.session_state.history = [] exit() history: list[Conversation] = st.session_state.history if retry: print("\n== Retry ==\n") last_user_conversation_idx = None for idx, conversation in enumerate(history): if conversation.role.value == Role.USER.value: last_user_conversation_idx = idx if last_user_conversation_idx is not None: prompt_text = history[last_user_conversation_idx].content print(f"New prompt: {prompt_text}, idx = {last_user_conversation_idx}") del history[last_user_conversation_idx:] for conversation in history: conversation.show() tools = get_tools() if page == Mode.ALL_TOOLS else [] client: Client = st.session_state.client def main(prompt_text: str): global client assert client is not None if prompt_text: prompt_text = prompt_text.strip() # Append uploaded files uploaded_texts = st.session_state.get("uploaded_texts") if page == Mode.LONG_CTX and uploaded_texts and first_round: meta_msg = "{} files uploaded.\n".format(st.session_state.uploaded_file_nums) prompt_text = uploaded_texts + "\n\n\n" + meta_msg + prompt_text # Clear after first use st.session_state.files_uploaded = True st.session_state.uploaded_texts = "" st.session_state.uploaded_file_nums = 0 image = st.session_state.get("uploaded_image") if page == Mode.VLM and image and first_round: st.session_state.uploaded_image = None role = Role.USER append_conversation(Conversation(role, prompt_text, image=image), history) placeholder = st.container() message_placeholder = placeholder.chat_message(name="assistant", avatar="assistant") markdown_placeholder = message_placeholder.empty() def add_new_block(): nonlocal message_placeholder, markdown_placeholder message_placeholder = placeholder.chat_message(name="assistant", avatar="assistant") markdown_placeholder = message_placeholder.empty() def commit_conversation( role: Role, text: str, metadata: str | None = None, image: str | None = None, new: bool = False, ): processed_text = postprocess_text(text, role.value == Role.ASSISTANT.value) conversation = Conversation(role, text, processed_text, metadata, image) # Use different placeholder for new block placeholder = message_placeholder if new else markdown_placeholder append_conversation( conversation, history, placeholder, ) response = "" for _ in range(10): last_response = None history_len = None try: for response, chat_history in client.generate_stream( tools=tools, history=history, temperature=temperature, top_p=top_p, top_k=top_k, repetition_penalty=repetition_penalty, max_new_tokens=max_new_tokens, ): if history_len is None: history_len = len(chat_history) elif history_len != len(chat_history): commit_conversation(Role.ASSISTANT, last_response) add_new_block() history_len = len(chat_history) last_response = response replace_quote = chat_history[-1]["role"] == "assistant" markdown_placeholder.markdown(postprocess_text(str(response) + "●", replace_quote=replace_quote)) else: metadata = page == Mode.ALL_TOOLS and isinstance(response, dict) and response.get("name") or None role = Role.TOOL if metadata else Role.ASSISTANT text = response.get("content") if metadata else response_to_str(response) commit_conversation(role, text, metadata) if metadata: add_new_block() try: with markdown_placeholder: with st.spinner(f"Calling tool {metadata}..."): observations = dispatch_tool(metadata, text, str(st.session_state.session_id)) except Exception as e: traceback.print_exc() st.error(f'Uncaught exception in `"{metadata}"`: {e}') break for observation in observations: observation.text = observation.text commit_conversation( Role.OBSERVATION, observation.text, observation.role_metadata, observation.image_url, new=True, ) add_new_block() continue else: break except Exception: traceback.print_exc() st.error(f"Uncaught exception: {traceback.format_exc()}") else: st.error("Too many chaining function calls!") main(prompt_text) ================================================ FILE: demo/composite_demo/src/tools/browser.py ================================================ """ Simple browser tool. # Usage Please start the backend browser server according to the instructions in the README. """ import re from dataclasses import dataclass from pprint import pprint import requests import streamlit as st from .config import BROWSER_SERVER_URL from .interface import ToolObservation QUOTE_REGEX = re.compile(r"\[(\d+)†(.+?)\]") @dataclass class Quote: title: str url: str # Quotes for displaying reference if "quotes" not in st.session_state: st.session_state.quotes = {} quotes: dict[str, Quote] = st.session_state.quotes def map_response(response: dict) -> ToolObservation: # Save quotes for reference print("===BROWSER_RESPONSE===") pprint(response) role_metadata = response.get("roleMetadata") metadata = response.get("metadata") if role_metadata.split()[0] == "quote_result" and metadata: quote_id = QUOTE_REGEX.search(role_metadata.split()[1]).group(1) quote: dict[str, str] = metadata["metadata_list"][0] quotes[quote_id] = Quote(quote["title"], quote["url"]) elif role_metadata == "browser_result" and metadata: for i, quote in enumerate(metadata["metadata_list"]): quotes[str(i)] = Quote(quote["title"], quote["url"]) return ToolObservation( content_type=response.get("contentType"), text=response.get("result"), role_metadata=role_metadata, metadata=metadata, ) def tool_call(code: str, session_id: str) -> list[ToolObservation]: request = { "session_id": session_id, "action": code, } response = requests.post(BROWSER_SERVER_URL, json=request).json() return list(map(map_response, response)) ================================================ FILE: demo/composite_demo/src/tools/cogview.py ================================================ import streamlit as st from zhipuai import ZhipuAI from zhipuai.types.image import GeneratedImage from .config import COGVIEW_MODEL, ZHIPU_AI_KEY from .interface import ToolObservation @st.cache_resource def get_zhipu_client(): return ZhipuAI(api_key=ZHIPU_AI_KEY) def map_response(img: GeneratedImage): return ToolObservation( content_type="image", text="CogView 已经生成并向用户展示了生成的图片。", image_url=img.url, role_metadata="cogview_result", ) def tool_call(prompt: str, session_id: str) -> list[ToolObservation]: client = get_zhipu_client() response = client.images.generations(model=COGVIEW_MODEL, prompt=prompt).data return list(map(map_response, response)) ================================================ FILE: demo/composite_demo/src/tools/config.py ================================================ BROWSER_SERVER_URL = "http://localhost:3000" IPYKERNEL = "glm-4-demo" ZHIPU_AI_KEY = "" COGVIEW_MODEL = "cogview-3" ================================================ FILE: demo/composite_demo/src/tools/interface.py ================================================ from dataclasses import dataclass from typing import Any @dataclass class ToolObservation: content_type: str text: str image_url: str | None = None role_metadata: str | None = None metadata: Any = None ================================================ FILE: demo/composite_demo/src/tools/python.py ================================================ import queue import re from pprint import pprint from subprocess import PIPE from typing import Literal import jupyter_client import streamlit as st from .config import IPYKERNEL from .interface import ToolObservation ANSI_ESCAPE = re.compile(r"(\x9B|\x1B\[|\u001b\[)[0-?]*[ -/]*[@-~]") CODE = re.compile(r"```([^\n]*)\n(.*?)```") class CodeKernel: def __init__( self, kernel_name="kernel", kernel_id=None, kernel_config_path="", python_path=None, ipython_path=None, init_file_path="./startup.py", verbose=1, ): self.kernel_name = kernel_name self.kernel_id = kernel_id self.kernel_config_path = kernel_config_path self.python_path = python_path self.ipython_path = ipython_path self.init_file_path = init_file_path self.verbose = verbose if python_path is None and ipython_path is None: env = None else: env = {"PATH": self.python_path + ":$PATH", "PYTHONPATH": self.python_path} # Initialize the backend kernel self.kernel_manager = jupyter_client.KernelManager( kernel_name=IPYKERNEL, connection_file=self.kernel_config_path, exec_files=[self.init_file_path], env=env ) if self.kernel_config_path: self.kernel_manager.load_connection_file() self.kernel_manager.start_kernel(stdout=PIPE, stderr=PIPE) print("Backend kernel started with the configuration: {}".format(self.kernel_config_path)) else: self.kernel_manager.start_kernel(stdout=PIPE, stderr=PIPE) print("Backend kernel started with the configuration: {}".format(self.kernel_manager.connection_file)) if verbose: pprint(self.kernel_manager.get_connection_info()) # Initialize the code kernel self.kernel = self.kernel_manager.blocking_client() # self.kernel.load_connection_file() self.kernel.start_channels() print("Code kernel started.") def execute(self, code): self.kernel.execute(code) try: shell_msg = self.kernel.get_shell_msg(timeout=30) io_msg_content = self.kernel.get_iopub_msg(timeout=30)["content"] while True: msg_out = io_msg_content ### Poll the message try: io_msg_content = self.kernel.get_iopub_msg(timeout=30)["content"] if "execution_state" in io_msg_content and io_msg_content["execution_state"] == "idle": break except queue.Empty: break return shell_msg, msg_out except Exception as e: print(e) return None def execute_interactive(self, code, verbose=False): shell_msg = self.kernel.execute_interactive(code) if shell_msg is queue.Empty: if verbose: print("Timeout waiting for shell message.") self.check_msg(shell_msg, verbose=verbose) return shell_msg def inspect(self, code, verbose=False): msg_id = self.kernel.inspect(code) shell_msg = self.kernel.get_shell_msg(timeout=30) if shell_msg is queue.Empty: if verbose: print("Timeout waiting for shell message.") self.check_msg(shell_msg, verbose=verbose) return shell_msg def get_error_msg(self, msg, verbose=False) -> str | None: if msg["content"]["status"] == "error": try: error_msg = msg["content"]["traceback"] except: try: error_msg = msg["content"]["traceback"][-1].strip() except: error_msg = "Traceback Error" if verbose: print("Error: ", error_msg) return error_msg return None def check_msg(self, msg, verbose=False): status = msg["content"]["status"] if status == "ok": if verbose: print("Execution succeeded.") elif status == "error": for line in msg["content"]["traceback"]: if verbose: print(line) def shutdown(self): # Shutdown the backend kernel self.kernel_manager.shutdown_kernel() print("Backend kernel shutdown.") # Shutdown the code kernel self.kernel.shutdown() print("Code kernel shutdown.") def restart(self): # Restart the backend kernel self.kernel_manager.restart_kernel() # print("Backend kernel restarted.") def interrupt(self): # Interrupt the backend kernel self.kernel_manager.interrupt_kernel() # print("Backend kernel interrupted.") def is_alive(self): return self.kernel.is_alive() def clean_ansi_codes(input_string): return ANSI_ESCAPE.sub("", input_string) def extract_code(text: str) -> str: matches = CODE.findall(text, re.DOTALL) return matches[-1][1] def execute(code: str, kernel: CodeKernel) -> tuple[Literal["text", "image"] | None, str]: res = "" res_type = None code = code.replace("<|observation|>", "") code = code.replace("<|assistant|>python", "") code = code.replace("<|assistant|>", "") code = code.replace("<|user|>", "") code = code.replace("<|system|>", "") msg, output = kernel.execute(code) if msg["metadata"]["status"] == "timeout": return res_type, "Timed out" elif msg["metadata"]["status"] == "error": return res_type, clean_ansi_codes("\n".join(kernel.get_error_msg(msg, verbose=True))) if "text" in output: res_type = "text" res = output["text"] elif "data" in output: for key in output["data"]: if "text/plain" in key: res_type = "text" res = output["data"][key] elif "image/png" in key: res_type = "image" res = output["data"][key] break return res_type, res @st.cache_resource def get_kernel() -> CodeKernel: return CodeKernel() def tool_call(code: str, session_id: str) -> list[ToolObservation]: kernel = get_kernel() res_type, res = execute(code, kernel) # Convert base64 to data uri text = "[Image]" if res_type == "image" else res image = f"data:image/png;base64,{res}" if res_type == "image" else None return [ToolObservation(res_type, text, image)] ================================================ FILE: demo/composite_demo/src/tools/tool_registry.py ================================================ """ This code is the tool registration part. By registering the tool, the model can call the tool. This code provides extended functionality to the model, enabling it to call and interact with a variety of utilities through defined interfaces. """ import copy import inspect import json import subprocess import traceback from collections.abc import Callable from types import GenericAlias from typing import Annotated, get_origin from .browser import tool_call as browser from .cogview import tool_call as cogview from .interface import ToolObservation from .python import tool_call as python ALL_TOOLS = { "simple_browser": browser, "python": python, "cogview": cogview, } _TOOL_HOOKS = {} _TOOL_DESCRIPTIONS = [] def register_tool(func: Callable): tool_name = func.__name__ tool_description = inspect.getdoc(func).strip() python_params = inspect.signature(func).parameters tool_params = [] for name, param in python_params.items(): annotation = param.annotation if annotation is inspect.Parameter.empty: raise TypeError(f"Parameter `{name}` missing type annotation") if get_origin(annotation) != Annotated: raise TypeError(f"Annotation type for `{name}` must be typing.Annotated") typ, (description, required) = annotation.__origin__, annotation.__metadata__ typ: str = str(typ) if isinstance(typ, GenericAlias) else typ.__name__ if not isinstance(description, str): raise TypeError(f"Description for `{name}` must be a string") if not isinstance(required, bool): raise TypeError(f"Required for `{name}` must be a bool") tool_params.append( { "name": name, "description": description, "type": typ, "required": required, } ) tool_def = { "name": tool_name, "description": tool_description, "params": tool_params, } # print("[registered tool] " + pformat(tool_def)) _TOOL_HOOKS[tool_name] = func _TOOL_DESCRIPTIONS.append(tool_def) return func def dispatch_tool(tool_name: str, code: str, session_id: str) -> list[ToolObservation]: # Dispatch predefined tools if tool_name in ALL_TOOLS: return ALL_TOOLS[tool_name](code, session_id) code = code.strip().rstrip("<|observation|>").strip() # Dispatch custom tools try: tool_params = json.loads(code) except json.JSONDecodeError as e: err = f"Error decoding JSON: {e}" return [ToolObservation("system_error", err)] if tool_name not in _TOOL_HOOKS: err = f"Tool `{tool_name}` not found. Please use a provided tool." return [ToolObservation("system_error", err)] tool_hook = _TOOL_HOOKS[tool_name] try: ret: str = tool_hook(**tool_params) return [ToolObservation(tool_name, str(ret))] except: err = traceback.format_exc() return [ToolObservation("system_error", err)] def get_tools() -> list[dict]: return copy.deepcopy(_TOOL_DESCRIPTIONS) # Tool Definitions @register_tool def random_number_generator( seed: Annotated[int, "The random seed used by the generator", True], range: Annotated[tuple[int, int], "The range of the generated numbers", True], ) -> int: """ Generates a random number x, s.t. range[0] <= x < range[1] """ if not isinstance(seed, int): raise TypeError("Seed must be an integer") if not isinstance(range, tuple): raise TypeError("Range must be a tuple") if not isinstance(range[0], int) or not isinstance(range[1], int): raise TypeError("Range must be a tuple of integers") import random return random.Random(seed).randint(*range) @register_tool def get_weather( city_name: Annotated[str, "The name of the city to be queried", True], ) -> str: """ Get the current weather for `city_name` """ if not isinstance(city_name, str): raise TypeError("City name must be a string") key_selection = { "current_condition": [ "temp_C", "FeelsLikeC", "humidity", "weatherDesc", "observation_time", ], } import requests try: resp = requests.get(f"https://wttr.in/{city_name}?format=j1") resp.raise_for_status() resp = resp.json() ret = {k: {_v: resp[k][0][_v] for _v in v} for k, v in key_selection.items()} except: import traceback ret = "Error encountered while fetching weather data!\n" + traceback.format_exc() return str(ret) @register_tool def get_shell( query: Annotated[str, "The command should run in Linux shell", True], ) -> str: """ Use shell to run command """ if not isinstance(query, str): raise TypeError("Command must be a string") try: result = subprocess.run( query, shell=True, check=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True, ) return result.stdout except subprocess.CalledProcessError as e: return e.stderr if __name__ == "__main__": # print(dispatch_tool("get_shell", {"query": "pwd"})) print(get_tools()) ================================================ FILE: demo/composite_demo/src/utils.py ================================================ import docx from langchain_community.document_loaders import PyMuPDFLoader from pptx import Presentation def extract_text(path): return open(path, "r").read() def extract_pdf(path): loader = PyMuPDFLoader(path) data = loader.load() data = [x.page_content for x in data] content = "\n\n".join(data) return content def extract_docx(path): doc = docx.Document(path) data = [] for paragraph in doc.paragraphs: data.append(paragraph.text) content = "\n\n".join(data) return content def extract_pptx(path): prs = Presentation(path) text = "" for slide in prs.slides: for shape in slide.shapes: if hasattr(shape, "text"): text += shape.text + "\n" return text ================================================ FILE: demo/intel_device_demo/itrex/README.md ================================================ # 使用 Intel® Extension for Transformers 推理 GLM-4-9B-Chat 模型 本示例介绍如何使用 Intel® Extension for Transformers 推理 GLM-4-9B-Chat 模型。 ## 设备和依赖检查 ### 相关推理测试数据 **本文档的数据均在以下硬件环境测试,实际运行环境需求和运行占用的显存略有不同，请以实际运行环境为准。** 测试硬件信息: + OS: Ubuntu 22.04 (本教程一定需要在Linux环境下执行) + Memory: 512GB + Python: 3.10.12 + CPU: Intel(R) Xeon(R) Platinum 8358 CPU / 12th Gen Intel i5-12400 ## 安装依赖 在开始推理之前，请你先安装`inference`中的依赖，同时您需要安装本目录下的依赖项： ```shell pip install -r requirements.txt ``` ## 运行模型推理 ```shell python itrex_cli_demo.py ``` 如果您是第一次推理，会有一次模型转换权重的过程，转换后的模型权重存放在`runtime_outputs`文件夹下，这大概会消耗`60G`的硬盘空间。 转换完成后，文件夹下有两个文件： + ne_chatglm2_f32.bin 52G(如果您不使用FP32进行推理，可以删掉这个文件) + ne_chatglm2_q_nf4_bestla_cfp32_sym_sfp32_g32.bin 8.1G 如果您不是第一次推理，则会跳过这个步骤，直接开始对话，推理效果如下： ```shell Welcome to the CLI chat. Type your messages below. User: 你好 AVX:1 AVX2:1 AVX512F:1 AVX512BW:1 AVX_VNNI:0 AVX512_VNNI:1 AMX_INT8:0 AMX_BF16:0 AVX512_BF16:0 AVX512_FP16:0 beam_size: 1, do_sample: 1, top_k: 40, top_p: 0.900, continuous_batching: 0, max_request_num: 1, early_stopping: 0, scratch_size_ratio: 1.000 model_file_loader: loading model from runtime_outs/ne_chatglm2_q_nf4_bestla_cfp32_sym_sfp32_g32.bin Loading the bin file with NE format... load_ne_hparams 0.hparams.n_vocab = 151552 load_ne_hparams 1.hparams.n_embd = 4096 load_ne_hparams 2.hparams.n_mult = 0 load_ne_hparams 3.hparams.n_head = 32 load_ne_hparams 4.hparams.n_head_kv = 0 load_ne_hparams 5.hparams.n_layer = 40 load_ne_hparams 6.hparams.n_rot = 0 load_ne_hparams 7.hparams.ftype = 0 load_ne_hparams 8.hparams.max_seq_len = 131072 load_ne_hparams 9.hparams.alibi_bias_max = 0.000 load_ne_hparams 10.hparams.clip_qkv = 0.000 load_ne_hparams 11.hparams.par_res = 0 load_ne_hparams 12.hparams.word_embed_proj_dim = 0 load_ne_hparams 13.hparams.do_layer_norm_before = 0 load_ne_hparams 14.hparams.multi_query_group_num = 2 load_ne_hparams 15.hparams.ffn_hidden_size = 13696 load_ne_hparams 16.hparams.inner_hidden_size = 0 load_ne_hparams 17.hparams.n_experts = 0 load_ne_hparams 18.hparams.n_experts_used = 0 load_ne_hparams 19.hparams.n_embd_head_k = 0 load_ne_hparams 20.hparams.norm_eps = 0.000000 load_ne_hparams 21.hparams.freq_base = 5000000.000 load_ne_hparams 22.hparams.freq_scale = 1.000 load_ne_hparams 23.hparams.rope_scaling_factor = 0.000 load_ne_hparams 24.hparams.original_max_position_embeddings = 0 load_ne_hparams 25.hparams.use_yarn = 0 load_ne_vocab 26.vocab.bos_token_id = 1 load_ne_vocab 27.vocab.eos_token_id = 151329 load_ne_vocab 28.vocab.pad_token_id = 151329 load_ne_vocab 29.vocab.sep_token_id = -1 init: hparams.n_vocab = 151552 init: hparams.n_embd = 4096 init: hparams.n_mult = 0 init: hparams.n_head = 32 init: hparams.n_layer = 40 init: hparams.n_rot = 0 init: hparams.ffn_hidden_size = 13696 init: n_parts = 1 load: ctx size = 16528.38 MB load: layers[0].ffn_fusion = 1 load: scratch0 = 4096.00 MB load: scratch1 = 2048.00 MB load: scratch2 = 4096.00 MB load: mem required = 26768.38 MB (+ memory per state) ............................................................................................. model_init_from_file: support_bestla_kv = 1 kv_cache_init: run_mha_reordered = 1 model_init_from_file: kv self size = 690.00 MB Assistant: 你好👋！我是人工智能助手，很高兴为你服务。有什么可以帮助你的吗？ ``` ================================================ FILE: demo/intel_device_demo/itrex/README_en.md ================================================ # Using Intel® Extension for Transformers to Inference the GLM-4-9B-Chat Model This example introduces how to use Intel® Extension for Transformers to inference the GLM-4-9B-Chat model. ## Device and Dependency Check ### Relevant Inference Test Data **The data in this document is tested on the following hardware environment. The actual running environment requirements and memory usage may vary slightly. Please refer to the actual running environment.** Test hardware information: + OS: Ubuntu 22.04 (This tutorial must be executed in a Linux environment) + Memory: 512GB + Python: 3.10.12 + CPU: Intel(R) Xeon(R) Platinum 8358 CPU / 12th Gen Intel i5-12400 ## Installing Dependencies Before starting the inference, please install the dependencies in `inference`, and you need to install the dependencies in this directory: ```shell pip install -r requirements.txt ``` ## Running Model Inference ```shell python itrex_cli_demo.py ``` If this is your first inference, there will be a process of converting model weights. The converted model weights are stored in the `runtime_outputs` folder, which will consume about `60G` of disk space. After the conversion is completed, there are two files in the folder: + ne_chatglm2_f32.bin 52G (If you do not use FP32 for inference, you can delete this file) + ne_chatglm2_q_nf4_bestla_cfp32_sym_sfp32_g32.bin 8.1G If this is not your first inference, this step will be skipped, and you will directly start the conversation. The inference result is as follows: ```shell Welcome to the CLI chat. Type your messages below. User: Hello AVX:1 AVX2:1 AVX512F:1 AVX512BW:1 AVX_VNNI:0 AVX512_VNNI:1 AMX_INT8:0 AMX_BF16:0 AVX512_BF16:0 AVX512_FP16:0 beam_size: 1, do_sample: 1, top_k: 40, top_p: 0.900, continuous_batching: 0, max_request_num: 1, early_stopping: 0, scratch_size_ratio: 1.000 model_file_loader: loading model from runtime_outs/ne_chatglm2_q_nf4_bestla_cfp32_sym_sfp32_g32.bin Loading the bin file with NE format... load_ne_hparams 0.hparams.n_vocab = 151552 load_ne_hparams 1.hparams.n_embd = 4096 load_ne_hparams 2.hparams.n_mult = 0 load_ne_hparams 3.hparams.n_head = 32 load_ne_hparams 4.hparams.n_head_kv = 0 load_ne_hparams 5.hparams.n_layer = 40 load_ne_hparams 6.hparams.n_rot = 0 load_ne_hparams 7.hparams.ftype = 0 load_ne_hparams 8.hparams.max_seq_len = 131072 load_ne_hparams 9.hparams.alibi_bias_max = 0.000 load_ne_hparams 10.hparams.clip_qkv = 0.000 load_ne_hparams 11.hparams.multi_query_group_num = 2 load_ne_hparams 12.hparams.ffn_hidden_size = 13696 load_ne_hparams 13.hparams.inner_hidden_size = 0 load_ne_hparams 14.hparams.n_experts = 0 load_ne_hparams 15.hparams.n_experts_used = 0 load_ne_hparams 16.hparams.n_embd_head_k = 0 load_ne_hparams 17.hparams.norm_eps = 0.000000 load_ne_hparams 18.hparams.freq_base = 5000000.000 load_ne_hparams 19.hparams.freq_scale = 1.000 load_ne_hparams 20.hparams.rope_scaling_factor = 0.000 load_ne_hparams 21.hparams.original_max_position_embeddings = 0 load_ne_hparams 22.hparams.use_yarn = 0 load_ne_vocab 23.vocab.bos_token_id = 1 load_ne_vocab 24.vocab.eos_token_id = 151329 load_ne_vocab 25.vocab.pad_token_id = 151329 load_ne_vocab 26.vocab.sep_token_id = -1 init: hparams.n_vocab = 151552 init: hparams.n_embd = 4096 init: hparams.n_mult = 0 init: hparams.n_head = 32 init: hparams.n_layer = 40 init: hparams.n_rot = 0 init: hparams.ffn_hidden_size = 13696 init: n_parts = 1 load: ctx size = 16528.38 MB load: layers[0].ffn_fusion = 1 load: scratch0 = 4096.00 MB load: scratch1 = 2048.00 MB load: scratch2 = 4096.00 MB load: mem required = 26768.38 MB (+ memory per state) ............................................................................................. model_init_from_file: support_bestla_kv = 1 kv_cache_init: run_mha_reordered = 1 model_init_from_file: kv self size = 690.00 MB Assistant: Hello👋! I am an AI assistant. How can I help you today? ``` ================================================ FILE: demo/intel_device_demo/itrex/itrex_cli_demo.py ================================================ """ This script creates a CLI demo with transformers backend for the glm-4-9b model with Intel® Extension for Transformers """ import os MODEL_PATH = os.environ.get("MODEL_PATH", "THUDM/GLM-4-9B-0414") from threading import Thread import torch from intel_extension_for_transformers.transformers import AutoModelForCausalLM from transformers import AutoTokenizer, StoppingCriteria, StoppingCriteriaList, TextIteratorStreamer class StopOnTokens(StoppingCriteria): def __call__(self, input_ids: torch.LongTensor, scores: torch.FloatTensor, **kwargs) -> bool: stop_ids = [151329, 151336, 151338] for stop_id in stop_ids: if input_ids[0][-1] == stop_id: return True return False def initialize_model_and_tokenizer(): tokenizer = AutoTokenizer.from_pretrained(MODEL_PATH) model = AutoModelForCausalLM.from_pretrained( MODEL_PATH, device_map="cpu", # Use Intel CPU for inference trust_remote_code=True, load_in_4bit=True, ) return tokenizer, model def get_user_input(): return input("\nUser: ") def main(): tokenizer, model = initialize_model_and_tokenizer() history = [] max_length = 100 top_p = 0.9 temperature = 0.8 stop = StopOnTokens() print("Welcome to the CLI chat. Type your messages below.") while True: user_input = get_user_input() if user_input.lower() in ["exit", "quit"]: break history.append([user_input, ""]) messages = [] for idx, (user_msg, model_msg) in enumerate(history): if idx == len(history) - 1 and not model_msg: messages.append({"role": "user", "content": user_msg}) break if user_msg: messages.append({"role": "user", "content": user_msg}) if model_msg: messages.append({"role": "assistant", "content": model_msg}) model_inputs = tokenizer.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_tensors="pt" ) streamer = TextIteratorStreamer(tokenizer=tokenizer, timeout=60, skip_prompt=True, skip_special_tokens=True) generate_kwargs = { "input_ids": model_inputs, "streamer": streamer, "max_new_tokens": max_length, "do_sample": True, "top_p": top_p, "temperature": temperature, "stopping_criteria": StoppingCriteriaList([stop]), "repetition_penalty": 1.2, "eos_token_id": model.config.eos_token_id, } t = Thread(target=model.generate, kwargs=generate_kwargs) t.start() print("Assistant:", end="", flush=True) for new_token in streamer: if new_token: print(new_token, end="", flush=True) history[-1][1] += new_token history[-1][1] = history[-1][1].strip() if __name__ == "__main__": main() ================================================ FILE: demo/intel_device_demo/itrex/requirements.txt ================================================ cmake>=3.29.5.1 huggingface-hub>=0.23.4 git+https://github.com/intel/neural-speed.git@main#egg=neural-speed intel-extension-for-transformers>=1.4.2 ================================================ FILE: demo/intel_device_demo/openvino/README.md ================================================ # 使用 OpenVINO 部署 GLM-4-9B-Chat 模型 Read this in [English](README_en.md). [OpenVINO](https://www.intel.com/content/www/us/en/developer/tools/openvino-toolkit/overview.html) 是 Intel 为深度学习推理而设计的开源工具包。它可以帮助开发者优化模型，提高推理性能，减少模型的内存占用。 本示例将展示如何使用 OpenVINO 部署 GLM-4-9B-Chat 模型。 ## 1. 环境配置 首先，你需要安装依赖 ```bash pip install -r requirements.txt ``` ## 2. 转换模型 由于需要将Huggingface模型转换为OpenVINO IR模型，因此您需要下载模型并转换。 ``` python3 convert.py --model_id THUDM/glm-4-9b-chat --output {your_path}/glm-4-9b-chat-ov ``` ### 可以选择的参数 * `--model_id` - 模型所在目录的路径（绝对路径）。 * `--output` - 转换后模型保存的地址。 * `--precision` - 转换的精度。 转换过程如下： ``` ====Exporting IR===== Framework not specified. Using pt to export the model. Loading checkpoint shards: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:04<00:00, 2.14it/s] Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained. Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained. Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained. Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained. Using framework PyTorch: 2.3.1+cu121 Mixed-Precision assignment ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 160/160 • 0:01:45 • 0:00:00 INFO:nncf:Statistics of the bitwidth distribution: ┍━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┑ │ Num bits (N) │ % all parameters (layers) │ % ratio-defining parameters (layers) │ ┝━━━━━━━━━━━━━━━━┿━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┿━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┥ │ 8 │ 31% (76 / 163) │ 20% (73 / 160) │ ├────────────────┼─────────────────────────────┼────────────────────────────────────────┤ │ 4 │ 69% (87 / 163) │ 80% (87 / 160) │ ┕━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┙ Applying Weight Compression ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 163/163 • 0:03:46 • 0:00:00 Configuration saved in glm-4-9b-ov/openvino_config.json ====Exporting tokenizer===== Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained. ``` ## 3. 运行 GLM-4-9B-Chat 模型 ``` python3 chat.py --model_path {your_path}/glm-4-9b-chat-ov --max_sequence_length 4096 --device CPU ``` ### 可以选择的参数 * `--model_path` - OpenVINO IR 模型所在目录的路径。 * `--max_sequence_length` - 输出标记的最大大小。 * `--device` - 运行推理的设备。 ### 参考代码 本代码参考 [OpenVINO 官方示例](https://github.com/OpenVINO-dev-contest/chatglm3.openvino) 进行修改。 ================================================ FILE: demo/intel_device_demo/openvino/README_en.md ================================================ # Deploy the GLM-4-9B-Chat model using OpenVINO [OpenVINO](https://www.intel.com/content/www/us/en/developer/tools/openvino-toolkit/overview.html) is an open source toolkit designed by Intel for deep learning inference. It can help developers optimize models, improve inference performance, and reduce model memory usage. This example will show how to deploy the GLM-4-9B-Chat model using OpenVINO. ## 1. Environment configuration First, you need to install the dependencies ```bash pip install -r requirements.txt ``` ## 2. Convert the model Since the Huggingface model needs to be converted to an OpenVINO IR model, you need to download the model and convert it. ``` python3 convert.py --model_id THUDM/glm-4-9b-chat --output {your_path}/glm-4-9b-chat-ov ``` The conversion process is as follows: ``` ====Exporting IR===== Framework not specified. Using pt to export the model. Loading checkpoint shards: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:04<00:00, 2.14it/s] Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained. Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained. Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained. Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained. Using framework PyTorch: 2.3.1+cu121 Mixed-Precision assignment ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 160/160 • 0:01:45 • 0:00:00 INFO:nncf:Statistics of the bitwidth distribution: ┍━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┑ │ Num bits (N) │ % all parameters (layers) │ % ratio-defining parameters (layers) │ ┝━━━━━━━━━━━━━━━━┿━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┿━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┥ │ 8 │ 31% (76 / 163) │ 20% (73 / 160) │ ├────────────────┼─────────────────────────────┼────────────────────────────────────────┤ │ 4 │ 69% (87 / 163) │ 80% (87 / 160) │ ┕━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┙ Applying Weight Compression ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 163/163 • 0:03:46 • 0:00:00 Configuration saved in glm-4-9b-ov/openvino_config.json ====Exporting tokenizer===== Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained. ``` ### Optional parameters * `--model_id` - Path to the directory where the model is located (absolute path). * `--output` - Path to where the converted model is saved. * `--precision` - Precision of the conversion. ## 3. Run the GLM-4-9B-Chat model ``` python3 chat.py --model_path {your_path}glm-4-9b-chat-ov --max_sequence_length 4096 --device CPU ``` ### Optional parameters * `--model_path` - Path to the directory where the OpenVINO IR model is located. * `--max_sequence_length` - Maximum size of the output token. * `--device` - the device to run inference on. ### Reference code This code is modified based on the [OpenVINO official example](https://github.com/OpenVINO-dev-contest/chatglm3.openvino). ================================================ FILE: demo/intel_device_demo/openvino/convert.py ================================================ """ This script is used to convert the original model to OpenVINO IR format. The Origin Code can check https://github.com/OpenVINO-dev-contest/chatglm3.openvino/blob/main/convert.py """ import argparse import os from pathlib import Path from optimum.intel import OVWeightQuantizationConfig from optimum.intel.openvino import OVModelForCausalLM from transformers import AutoConfig, AutoTokenizer if __name__ == "__main__": parser = argparse.ArgumentParser(add_help=False) parser.add_argument("-h", "--help", action="help", help="Show this help message and exit.") parser.add_argument( "-m", "--model_id", default="THUDM/GLM-4-9B-0414", required=False, type=str, help="orignal model path" ) parser.add_argument( "-p", "--precision", required=False, default="int4", type=str, choices=["fp16", "int8", "int4"], help="fp16, int8 or int4", ) parser.add_argument( "-o", "--output", default="./glm-4-9b-ov", required=False, type=str, help="Required. path to save the ir model" ) args = parser.parse_args() ir_model_path = Path(args.output) if ir_model_path.exists() == False: os.mkdir(ir_model_path) model_kwargs = { "trust_remote_code": True, "config": AutoConfig.from_pretrained(args.model_id, trust_remote_code=True), } compression_configs = { "sym": False, "group_size": 128, "ratio": 0.8, } print("====Exporting IR=====") if args.precision == "int4": ov_model = OVModelForCausalLM.from_pretrained( args.model_id, export=True, compile=False, quantization_config=OVWeightQuantizationConfig(bits=4, **compression_configs), **model_kwargs, ) elif args.precision == "int8": ov_model = OVModelForCausalLM.from_pretrained( args.model_id, export=True, compile=False, load_in_8bit=True, **model_kwargs ) else: ov_model = OVModelForCausalLM.from_pretrained( args.model_id, export=True, compile=False, load_in_8bit=False, **model_kwargs ) ov_model.save_pretrained(ir_model_path) print("====Exporting tokenizer=====") tokenizer = AutoTokenizer.from_pretrained(args.model_id, trust_remote_code=True) tokenizer.save_pretrained(ir_model_path) ================================================ FILE: demo/intel_device_demo/openvino/openvino_cli_demo.py ================================================ import argparse from threading import Thread from typing import List, Tuple import torch from optimum.intel.openvino import OVModelForCausalLM from transformers import AutoConfig, AutoTokenizer, StoppingCriteria, StoppingCriteriaList, TextIteratorStreamer class StopOnTokens(StoppingCriteria): def __init__(self, token_ids): self.token_ids = token_ids def __call__(self, input_ids: torch.LongTensor, scores: torch.FloatTensor, **kwargs) -> bool: for stop_id in self.token_ids: if input_ids[0][-1] == stop_id: return True return False if __name__ == "__main__": parser = argparse.ArgumentParser(add_help=False) parser.add_argument("-h", "--help", action="help", help="Show this help message and exit.") parser.add_argument("-m", "--model_path", required=True, type=str, help="Required. model path") parser.add_argument( "-l", "--max_sequence_length", default=256, required=False, type=int, help="Required. maximun length of output" ) parser.add_argument( "-d", "--device", default="CPU", required=False, type=str, help="Required. device for inference" ) args = parser.parse_args() model_dir = args.model_path ov_config = {"PERFORMANCE_HINT": "LATENCY", "NUM_STREAMS": "1", "CACHE_DIR": ""} tokenizer = AutoTokenizer.from_pretrained(model_dir, trust_remote_code=True) print("====Compiling model====") ov_model = OVModelForCausalLM.from_pretrained( model_dir, device=args.device, ov_config=ov_config, config=AutoConfig.from_pretrained(model_dir, trust_remote_code=True), trust_remote_code=True, ) streamer = TextIteratorStreamer(tokenizer, timeout=60.0, skip_prompt=True, skip_special_tokens=True) stop_tokens = [StopOnTokens([151329, 151336, 151338])] def convert_history_to_token(history: List[Tuple[str, str]]): messages = [] for idx, (user_msg, model_msg) in enumerate(history): if idx == len(history) - 1 and not model_msg: messages.append({"role": "user", "content": user_msg}) break if user_msg: messages.append({"role": "user", "content": user_msg}) if model_msg: messages.append({"role": "assistant", "content": model_msg}) model_inputs = tokenizer.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_tensors="pt" ) return model_inputs history = [] print("====Starting conversation====") while True: input_text = input("用户: ") if input_text.lower() == "stop": break if input_text.lower() == "clear": history = [] print("AI助手: 对话历史已清空") continue print("GLM-4-9B-OpenVINO:", end=" ") history = history + [[input_text, ""]] model_inputs = convert_history_to_token(history) generate_kwargs = dict( input_ids=model_inputs, max_new_tokens=args.max_sequence_length, temperature=0.1, do_sample=True, top_p=1.0, top_k=50, repetition_penalty=1.1, streamer=streamer, stopping_criteria=StoppingCriteriaList(stop_tokens), ) t1 = Thread(target=ov_model.generate, kwargs=generate_kwargs) t1.start() partial_text = "" for new_text in streamer: new_text = new_text print(new_text, end="", flush=True) partial_text += new_text print("\n") history[-1][1] = partial_text ================================================ FILE: demo/intel_device_demo/openvino/requirements.txt ================================================ optimum>=1.20.0 optimum-intel @ git+https://github.com/huggingface/optimum-intel.git@c1ee8ac0864e25e22ea56b5a37a35451531da0e6 ================================================ FILE: finetune/.gitignore ================================================ output/ ================================================ FILE: finetune/README.md ================================================ # GLM-4-9B Chat Fine-tuning [中文阅读](README_zh.md) ## Hardware Check All fine-tuning tests were performed in the following environment: > OS: Ubuntu 22.04 > > Memory: 512GB > > Python: 3.12.3 > > CUDA Version: 12.4 > > GPU Driver: 535.104.05 > > GPU: NVIDIA H100 80GB HBM3 (hereafter referred to as GPU) + Fine-tuning based on Llama-Factory | Fine-tuning Model | Fine-tuning solution | GPU memory usage | |-----------------------|----------------------|------------------------------| | GLM-4-9B-0414 | lora | 22G (Each GPU, Need 1 GPU) | | GLM-4-9B-0414 | SFT (Zero3 method) | 55G (Each GPU, Need 4 GPUs) | | GLM-4-9B-0414 | SFT | 80G (Each GPU, Need 8 GPUs) | | GLM-4-32B-0414 | SFT (Zero3 method) | 80G (Each GPU, Need 16 GPUs) | + Fine-tuning based on this repository | Fine-tuning Model | Fine-tuning solution | GPU memory usage | |--------------------------|------------------------------------|-------------------------------| | GLM-4V-9B | lora (PEFT), Include EVA2CLIPModel | 75G (Each GPU, Need 1 GPU) | | GLM-4-9B-Chat | lora (PEFT) | 22G (Each GPU, Need 1 GPU) | | GLM-4-9B-Chat | SFT (Zero3 method) | 80G (Each GPU, Need 8 GPUs) | ## Preparation Before starting fine-tuning, please install the dependencies in `inference`, ensure you have cloned the latest version of the model repository, and install the dependencies in this directory: ```bash pip install -r requirements.txt ``` ## Multi-round dialogue format The multi-round dialogue fine-tuning example uses the GLM-4 dialogue format convention, adding different `loss_mask` to different roles to calculate `loss` for multiple rounds of replies in one calculation. For data files, the sample uses the following format: ```json [ { "messages": [ { "role": "system", "content": "", "tools": [ { "name": "", "args": { "": "" } } // Add more tools if needed ] }, { "role": "user", "content": "" }, { "role": "assistant", "content": "" }, // If Tool Using { "role": "user", "content": "" }, { "role": "assistant", "content": "" }, { "role": "observation", "content": "" }, { "role": "assistant", "content": "" }, // Multi_turns { "role": "user", "content": "" }, { "role": "assistant", "content": "" } ] } ] ``` This is a sample without tools: ```json { "messages": [ { "role": "user", "content": "类型#裤*材质#牛仔布*风格#性感" }, { "role": "assistant", "content": "3x1的这款牛仔裤采用浅白的牛仔面料为裤身材质，其柔然的手感和细腻的质地，在穿着舒适的同时，透露着清纯甜美的个性气质。除此之外，流畅的裤身剪裁将性感的腿部曲线彰显的淋漓尽致，不失为一款随性出街的必备单品。" } ] } ``` This is a sample with tools: ```json { "messages": [ { "role": "system", "content": "", "tools": [ { "type": "function", "function": { "name": "get_recommended_books", "description": "Get recommended books based on user's interests", "parameters": { "type": "object", "properties": { "interests": { "type": "array", "items": { "type": "string" }, "description": "The interests to recommend books for" } }, "required": [ "interests" ] } } } ] }, { "role": "user", "content": "Hi, I am looking for some book recommendations. I am interested in history and science fiction." }, { "role": "assistant", "content": "{\"name\": \"get_recommended_books\", \"arguments\": {\"interests\": [\"history\", \"science fiction\"]}}" }, { "role": "observation", "content": "{\"books\": [\"Sapiens: A Brief History of Humankind by Yuval Noah Harari\", \"A Brief History of Time by Stephen Hawking\", \"Dune by Frank Herbert\", \"The Martian by Andy Weir\"]}" }, { "role": "assistant", "content": "Based on your interests in history and science fiction, I would recommend the following books: \"Sapiens: A Brief History of Humankind\" by Yuval Noah Harari, \"A Brief History of Time\" by Stephen Hawking, \"Dune\" by Frank Herbert, and \"The Martian\" by Andy Weir." } ] } ``` This is a sample with VQA Task: ```json { "messages": [ { "role": "user", "content": "图片中的动物是什么？", "image": "/root/images/0001.jpg" }, { "role": "assistant", "content": "图片中有一只猫。" }, { "role": "user", "content": "图片中的猫在做什么？" }, { "role": "assistant", "content": "这只猫坐在或站在桌子上，桌上有很多食物。" } ] } ``` - The `system` role is optional, but if it exists, it must appear before the `user` role, and the `system` role can only appear once in a complete conversation (whether it is a single round or a multi-round conversation). - The `tools` field is optional, but if it exists, it must appear after the `system` role, and the `tools` field can only appear once in a complete conversation (whether it is a single round or a multi-round conversation). When the `tools` field exists, the `system` role must exist and the `content` field is empty. - `GLM-4V-9B` does not support the `tools` field and the `system` field. And `image` must be placed in the first message. The `image` field needs to contain the `absolute path` of the image. ## Configuration file The fine-tuning configuration file is located in the `config` directory, including the following files: 1. `ds_zereo_2 / ds_zereo_3.json`: deepspeed configuration file. 2. `lora.yaml / ptuning_v2 3. .yaml / sft.yaml`: Configuration files for different modes of models, including model parameters, optimizer parameters, training parameters, etc. Some important parameters are explained as follows: + data_config section + train_file: File path of training dataset. + val_file: File path of validation dataset. + test_file: File path of test dataset. + num_proc: Number of processes to use when loading data. + max_input_length: Maximum length of input sequence. + max_output_length: Maximum length of output sequence. + training_args section + output_dir: Directory for saving model and other outputs. + max_steps: Maximum number of training steps. + per_device_train_batch_size: Training batch size per device (such as GPU). + dataloader_num_workers: Number of worker threads to use when loading data. + remove_unused_columns: Whether to remove unused columns in data. + save_strategy: Model saving strategy (for example, how many steps to save). + save_steps: How many steps to save the model. + log_level: Log level (such as info). + logging_strategy: logging strategy. + logging_steps: how many steps to log at. + per_device_eval_batch_size: per-device evaluation batch size. + evaluation_strategy: evaluation strategy (e.g. how many steps to evaluate at). + eval_steps: how many steps to evaluate at. + predict_with_generate: whether to use generation mode for prediction. + generation_config section + max_new_tokens: maximum number of new tokens to generate. + peft_config section + peft_type: type of parameter tuning to use (supports LORA and PREFIX_TUNING). + task_type: task type, here is causal language model (don't change). + Lora parameters: + r: rank of LoRA. + lora_alpha: scaling factor of LoRA. + lora_dropout: dropout probability to use in LoRA layer. + P-TuningV2 parameters: + num_virtual_tokens: the number of virtual tokens. + num_attention_heads: 2: the number of attention heads of P-TuningV2 (do not change). + token_dim: 256: the token dimension of P-TuningV2 (do not change). ## Start fine-tuning Execute **single machine multi-card/multi-machine multi-card** run through the following code, which uses `deepspeed` as the acceleration solution, and you need to install `deepspeed`. ```shell OMP_NUM_THREADS=1 torchrun --standalone --nnodes=1 --nproc_per_node=8 finetune.py data/AdvertiseGen/ THUDM/GLM-4-9b-0414 configs/lora.yaml # For Chat Fine-tune OMP_NUM_THREADS=1 torchrun --standalone --nnodes=1 --nproc_per_node=8 finetune_vision.py data/CogVLM-311K/ THUDM/glm-4v-9b configs/lora.yaml # For VQA Fine-tune ``` Execute **single machine single card** run through the following code. ```shell python finetune.py data/AdvertiseGen/ THUDM/GLM-4-9B-0414 configs/lora.yaml # For Chat Fine-tune python finetune_vision.py data/CogVLM-311K/ THUDM/glm-4v-9b configs/lora.yaml # For VQA Fine-tune ``` ## Log Visualization Support The fine-tuning code supports using SwanLab to visualize and track training metrics. You can enable tracking by installing SwanLab: ```shell pip install swanlab ``` You can visit the [SwanLab Visualization Dashboard](https://swanlab.cn/@ShaohonChen/GLM4-Finetune) to view the training logs of example fine-tuning scripts. If prompted to log in, you can obtain an API Key by visiting [https://swanlab.cn/space/~/settings](https://swanlab.cn/space/~/settings). If you only want to use the local dashboard, set `swanlab: local` in the configuration parameters and use the `swanlab watch` command to start the offline dashboard. ## Fine-tune from a saved point If you train as described above, each fine-tuning will start from the beginning. If you want to fine-tune from a half-trained model, you can add a fourth parameter, which can be passed in two ways: 1. `yes`, automatically start training from the last saved Checkpoint 2. `XX`, breakpoint number, for example `600`, start training from Checkpoint 600 For example, this is an example code to continue fine-tuning from the last saved point ```shell python finetune.py data/AdvertiseGen/ THUDM/GLM-4-9B-0414 configs/lora.yaml yes ``` ## Use the fine-tuned model ### Use the fine-tuned model in other demos in this repository or external repositories You can use our `LORA` and fully fine-tuned models in any demo. This requires you to modify the code yourself according to the following tutorial. 1. Replace the way to read the model in the demo with the way to read the model in `finetune_demo/inference.py`. > Please note that for LORA and P-TuningV2, we did not merge the trained models, but recorded the fine-tuned path > in `adapter_config.json` > If the location of your original model changes, you should modify the path of `base_model_name_or_path` > in `adapter_config.json`. ```python def load_model_and_tokenizer(model_dir: Union[str, Path]) -> tuple[ModelType, TokenizerType]: model_dir = _resolve_path(model_dir) if (model_dir / "adapter_config.json").exists(): model = AutoPeftModelForCausalLM.from_pretrained(model_dir, device_map="auto") tokenizer_dir = model.peft_config["default"].base_model_name_or_path else: model = AutoModelForCausalLM.from_pretrained(model_dir, device_map="auto") tokenizer_dir = model_dir tokenizer = AutoTokenizer.from_pretrained(tokenizer_dir) return model, tokenizer ``` 2. Read the fine-tuned model. Please note that you should use the location of the fine-tuned model. For example, if your model location is `/path/to/finetune_adapter_model` and the original model address is `path/to/base_model`, you should use `/path/to/finetune_adapter_model` as `model_dir`. 3. After completing the above operations, you can use the fine-tuned model normally. Other calling methods remain unchanged. 4. This fine-tuning script has not been tested on long texts of 128K or 1M tokens. Fine-tuning long texts requires GPU devices with larger memory and more efficient fine-tuning solutions, which developers need to handle on their own. ## Reference ``` @inproceedings{liu2022p, title={P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks}, author={Liu, Xiao and Ji, Kaixuan and Fu, Yicheng and Tam, Weng and Du, Zhengxiao and Yang, Zhilin and Tang, Jie}, booktitle={Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)}, pages={61--68}, year={2022} } @misc{tang2023toolalpaca, title={ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases}, author={Qiaoyu Tang and Ziliang Deng and Hongyu Lin and Xianpei Han and Qiao Liang and Le Sun}, year={2023}, eprint={2306.05301}, archivePrefix={arXiv}, primaryClass={cs.CL} } ``` ================================================ FILE: finetune/README_zh.md ================================================ # GLM-4-9B Chat 对话模型微调 Read this in [English](README) ## 硬件检查 所有微调测试均在以下环境和硬件下测试: > OS: Ubuntu 22.04 > > Memory: 512GB > > Python: 3.12.3 > > CUDA Version: 12.4 > > GPU Driver: 535.104.05 > > GPU: NVIDIA H100 80GB HBM3 (以下简称 GPU) + 基于 Llama-Factory 进行微调 | Fine-tuning Model | Fine-tuning solution | GPU memory usage | |-----------------------|----------------------|------------------------------| | GLM-4-9B-0414 | lora | 22G (Each GPU, Need 1 GPU) | | GLM-4-9B-0414 | SFT (Zero3 method) | 55G (Each GPU, Need 4 GPUs) | | GLM-4-9B-0414 | lora | 80G (Each GPU, Need 8 GPUs) | | GLM-4-32B-0414 | SFT (Zero3 method) | 80G (Each GPU, Need 16 GPUs) | + 基于本仓库代码微调 | Fine-tuning Model | Fine-tuning solution | GPU memory usage | |--------------------------|------------------------------------|-------------------------------| | GLM-4V-9B | lora (PEFT), Include EVA2CLIPModel | 75G (Each GPU, Need 1 GPU) | | GLM-4-9B-Chat | lora (PEFT) | 22G (Each GPU, Need 1 GPU) | | GLM-4-9B-Chat | SFT (Zero3 method) | 80G (Each GPU, Need 8 GPUs) | ## 准备工作 在开始微调之前，请你先安装 `inference` 中的依赖，并保证克隆了最新版本的模型仓库，同时您需要安装本目录下的依赖项： ```bash pip install -r requirements.txt ``` ## 多轮对话格式 多轮对话微调示例采用 GLM-4 对话格式约定，对不同角色添加不同 `loss_mask` 从而在一遍计算中为多轮回复计算 `loss`。 对于数据文件，样例采用如下格式 如果您仅希望微调模型的对话能力，而非工具能力，您应该按照以下格式整理数据。 ```json [ { "messages": [ { "role": "system", "content": "", "tools": [ { "name": "", "args": { "": "" } } // Add more tools if needed ] }, { "role": "user", "content": "" }, { "role": "assistant", "content": "" }, // If Tool Using { "role": "user", "content": "" }, { "role": "assistant", "content": "" }, { "role": "observation", "content": "" }, { "role": "assistant", "content": "" }, // Multi_turns { "role": "user", "content": "" }, { "role": "assistant", "content": "" } ] } ] ``` 这里是一个不带有工具的例子: ```json { "messages": [ { "role": "user", "content": "类型#裤*材质#牛仔布*风格#性感" }, { "role": "assistant", "content": "3x1的这款牛仔裤采用浅白的牛仔面料为裤身材质，其柔然的手感和细腻的质地，在穿着舒适的同时，透露着清纯甜美的个性气质。除此之外，流畅的裤身剪裁将性感的腿部曲线彰显的淋漓尽致，不失为一款随性出街的必备单品。" } ] } ``` 这是一个带有工具调用的例子: ```json { "messages": [ { "role": "system", "content": "", "tools": [ { "type": "function", "function": { "name": "get_recommended_books", "description": "Get recommended books based on user's interests", "parameters": { "type": "object", "properties": { "interests": { "type": "array", "items": { "type": "string" }, "description": "The interests to recommend books for" } }, "required": [ "interests" ] } } } ] }, { "role": "user", "content": "Hi, I am looking for some book recommendations. I am interested in history and science fiction." }, { "role": "assistant", "content": "{\"name\": \"get_recommended_books\", \"arguments\": {\"interests\": [\"history\", \"science fiction\"]}}" }, { "role": "observation", "content": "{\"books\": [\"Sapiens: A Brief History of Humankind by Yuval Noah Harari\", \"A Brief History of Time by Stephen Hawking\", \"Dune by Frank Herbert\", \"The Martian by Andy Weir\"]}" }, { "role": "assistant", "content": "Based on your interests in history and science fiction, I would recommend the following books: \"Sapiens: A Brief History of Humankind\" by Yuval Noah Harari, \"A Brief History of Time\" by Stephen Hawking, \"Dune\" by Frank Herbert, and \"The Martian\" by Andy Weir." } ] } ``` 这是一个视觉VQA微调的例子： ```json { "messages": [ { "role": "user", "content": "图片中的动物是什么？", "image": "/root/images/0001.jpg" }, { "role": "assistant", "content": "图片中有一只猫。" }, { "role": "user", "content": "图片中的猫在做什么？" }, { "role": "assistant", "content": "这只猫坐在或站在桌子上，桌上有很多食物。" } ] } ``` - `system` 角色为可选角色，但若存在 `system` 角色，其必须出现在 `user` 角色之前，且一个完整的对话数据（无论单轮或者多轮对话）只能出现一次 `system` 角色。 - `tools` 字段为可选字段，若存在 `tools` 字段，其必须出现在 `system` 角色之后，且一个完整的对话数据（无论单轮或者多轮对话）只能出现一次 `tools` 字段。当 `tools` 字段存在时，`system` 角色必须存在并且 `content` 字段为空。 - `GLM-4V-9B` 不支持 `tools` 字段和 `system` 字段。并且 `image` 必须放在第一条消息中。 `image` 字段需要放置置图片的 `绝对路径`。 ## 配置文件 微调配置文件位于 `config` 目录下，包括以下文件： 1. `ds_zereo_2 / ds_zereo_3.json`: deepspeed 配置文件。 2. `lora.yaml 3. .yaml / sft.yaml`: 模型不同方式的配置文件，包括模型参数、优化器参数、训练参数等。 部分重要参数解释如下： + data_config 部分 + train_file: 训练数据集的文件路径。 + val_file: 验证数据集的文件路径。 + test_file: 测试数据集的文件路径。 + num_proc: 在加载数据时使用的进程数量。 + max_input_length: 输入序列的最大长度。 + max_output_length: 输出序列的最大长度。 + training_args 部分 + output_dir: 用于保存模型和其他输出的目录。 + max_steps: 训练的最大步数。 + per_device_train_batch_size: 每个设备（如 GPU）的训练批次大小。 + dataloader_num_workers: 加载数据时使用的工作线程数量。 + remove_unused_columns: 是否移除数据中未使用的列。 + save_strategy: 模型保存策略（例如，每隔多少步保存一次）。 + save_steps: 每隔多少步保存一次模型。 + log_level: 日志级别（如 info）。 + logging_strategy: 日志记录策略。 + logging_steps: 每隔多少步记录一次日志。 + per_device_eval_batch_size: 每个设备的评估批次大小。 + evaluation_strategy: 评估策略（例如，每隔多少步进行一次评估）。 + eval_steps: 每隔多少步进行一次评估。 + predict_with_generate: 是否使用生成模式进行预测。 + generation_config 部分 + max_new_tokens: 生成的最大新 token 数量。 + peft_config 部分 + peft_type: 使用的参数有效调整类型 (支持 LORA 和 PREFIX_TUNING)。 + task_type: 任务类型，这里是因果语言模型 (不要改动)。 + Lora 参数： + r: LoRA 的秩。 + lora_alpha: LoRA 的缩放因子。 + lora_dropout: 在 LoRA 层使用的 dropout 概率。 + P-TuningV2 参数： + num_virtual_tokens: 虚拟 token 的数量。 + num_attention_heads: 2: P-TuningV2 的注意力头数(不要改动)。 + token_dim: 256: P-TuningV2 的 token 维度(不要改动)。 ## 开始微调 通过以下代码执行 **单机多卡/多机多卡** 运行，这是使用 `deepspeed` 作为加速方案的，您需要安装 `deepspeed`。接着，按照此命令运行： ```shell OMP_NUM_THREADS=1 torchrun --standalone --nnodes=1 --nproc_per_node=8 finetune.py data/AdvertiseGen/ THUDM/GLM-4-9B-0414 configs/lora.yaml # For Chat Fine-tune OMP_NUM_THREADS=1 torchrun --standalone --nnodes=1 --nproc_per_node=8 finetune_vision.py data/CogVLM-311K/ THUDM/glm-4v-9b configs/lora.yaml # For VQA Fine-tune ``` 通过以下代码执行 **单机单卡** 运行。 ```shell python finetune.py data/AdvertiseGen/ THUDM/GLM-4-9B-0414 configs/lora.yaml # For Chat Fine-tune python finetune_vision.py data/CogVLM-311K/ THUDM/glm-4v-9b configs/lora.yaml # For VQA Fine-tune ``` ## 日志可视化支持 微调代码支持使用SwanLab对训练指标进行可视化跟踪。可通过安装SwanLab开启跟踪： ```shell pip install swanlab ``` 可以访问[SwanLab可视化看板](https://swanlab.cn/@ShaohonChen/GLM4-Finetune)获得案例微调脚本的训练日志。 如果提示登录，可以通过访问[https://swanlab.cn/space/~/settings](https://swanlab.cn/space/~/settings)获取API Key。 如果仅使用本地看板，可在配置参数中设置`swanlab: local`。并使用`swanlab watch`命令开启离线看板。 ## 从保存点进行微调 如果按照上述方式进行训练，每次微调都会从头开始，如果你想从训练一半的模型开始微调，你可以加入第四个参数，这个参数有两种传入方式: 1. `yes`, 自动从最后一个保存的 Checkpoint开始训练 2. `XX`, 断点号数字 例 `600` 则从序号600 Checkpoint开始训练 例如，这就是一个从最后一个保存点继续微调的示例代码 ```shell python finetune.py data/AdvertiseGen/ THUDM/GLM-4-9B-0414 configs/lora.yaml yes ``` ## 使用微调后的模型 您可以在任何一个 demo 内使用我们的 `LORA` 和 全参微调的模型。这需要你自己按照以下教程进行修改代码。 1. 使用`finetune_demo/inference.py`中读入模型的方式替换 demo 中读入模型的方式。 > 请注意，对于 LORA 和 P-TuningV2 我们没有合并训练后的模型，而是在`adapter_config.json` > 中记录了微调型的路径，如果你的原始模型位置发生更改，则你应该修改`adapter_config.json`中`base_model_name_or_path`的路径。 ```python def load_model_and_tokenizer(model_dir: Union[str, Path]) -> tuple[ModelType, TokenizerType]: model_dir = _resolve_path(model_dir) if (model_dir / "adapter_config.json").exists(): model = AutoPeftModelForCausalLM.from_pretrained(model_dir, device_map="auto") tokenizer_dir = model.peft_config["default"].base_model_name_or_path else: model = AutoModelForCausalLM.from_pretrained(model_dir, device_map="auto") tokenizer_dir = model_dir tokenizer = AutoTokenizer.from_pretrained(tokenizer_dir) return model, tokenizer ``` 2. 读取微调的模型，请注意，你应该使用微调模型的位置，例如，若你的模型位置为`/path/to/finetune_adapter_model` ，原始模型地址为`path/to/base_model`,则你应该使用`/path/to/finetune_adapter_model`作为`model_dir`。 3. 完成上述操作后，就能正常使用微调的模型了，其他的调用方式没有变化。 4. 本微调脚本没有测试过128K 1M等长文本的微调，长文本的微调需要更大显存的GPU设备，并且需要更高效的微调方案,需要开发者自行解决。 ## 参考文献 ``` @inproceedings{liu2022p, title={P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks}, author={Liu, Xiao and Ji, Kaixuan and Fu, Yicheng and Tam, Weng and Du, Zhengxiao and Yang, Zhilin and Tang, Jie}, booktitle={Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)}, pages={61--68}, year={2022} } @misc{tang2023toolalpaca, title={ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases}, author={Qiaoyu Tang and Ziliang Deng and Hongyu Lin and Xianpei Han and Qiao Liang and Le Sun}, year={2023}, eprint={2306.05301}, archivePrefix={arXiv}, primaryClass={cs.CL} } ``` ================================================ FILE: finetune/configs/ds_zero_2.json ================================================ { "fp16": { "enabled": "auto", "loss_scale": 0, "loss_scale_window": 1000, "initial_scale_power": 16, "hysteresis": 2, "min_loss_scale": 1 }, "bf16": { "enabled": "auto" }, "zero_optimization": { "stage": 2, "allgather_partitions": true, "allgather_bucket_size": 5e8, "overlap_comm": true, "reduce_scatter": true, "reduce_bucket_size": 5e8, "contiguous_gradients": true }, "gradient_accumulation_steps": "auto", "gradient_clipping": "auto", "steps_per_print": 2000, "train_batch_size": "auto", "train_micro_batch_size_per_gpu": "auto", "wall_clock_breakdown": false } ================================================ FILE: finetune/configs/ds_zero_3.json ================================================ { "train_micro_batch_size_per_gpu": "auto", "zero_allow_untested_optimizer": true, "bf16": { "enabled": "auto" }, "optimizer": { "type": "AdamW", "params": { "lr": "auto", "betas": "auto", "eps": "auto", "weight_decay": "auto" } }, "zero_optimization": { "stage": 3, "allgather_partitions": true, "allgather_bucket_size": 5e8, "reduce_scatter": true, "contiguous_gradients": true, "overlap_comm": true, "sub_group_size": 1e9, "reduce_bucket_size": "auto", "stage3_prefetch_bucket_size": "auto", "stage3_param_persistence_threshold": "auto", "stage3_max_live_parameters": 1e9, "stage3_max_reuse_distance": 1e9, "stage3_gather_16bit_weights_on_model_save": true } } ================================================ FILE: finetune/configs/lora.yaml ================================================ data_config: train_file: train.jsonl val_file: dev.jsonl test_file: dev.jsonl num_proc: 1 combine: True freezeV: True max_input_length: 512 max_output_length: 512 # swanlab: "local" # set to local if don`t use cloud training_args: # see `transformers.Seq2SeqTrainingArguments` output_dir: ./output max_steps: 3000 # needed to be fit for the dataset learning_rate: 5e-4 # settings for data loading per_device_train_batch_size: 1 dataloader_num_workers: 16 remove_unused_columns: false # settings for saving checkpoints save_strategy: steps save_steps: 500 # settings for logging log_level: info logging_strategy: steps logging_steps: 10 run_name: "glm4-lora-finetune" # settings for evaluation per_device_eval_batch_size: 4 eval_strategy: steps eval_steps: 500 # settings for optimizer # adam_epsilon: 1e-6 # uncomment the following line to detect nan or inf values # debug: underflow_overflow predict_with_generate: true # see `transformers.GenerationConfig` generation_config: max_new_tokens: 512 # set your absolute deepspeed path here # deepspeed: configs/ds_zero_3.json deepspeed: configs/ds_zero_2.json peft_config: peft_type: LORA task_type: CAUSAL_LM r: 8 lora_alpha: 32 lora_dropout: 0.1 target_modules: ["q_proj", "k_proj", "v_proj"] ================================================ FILE: finetune/configs/sft.yaml ================================================ data_config: train_file: train.jsonl val_file: dev.jsonl test_file: dev.jsonl num_proc: 1 combine: True freezeV: True max_input_length: 512 max_output_length: 512 # swanlab: "local" # set to local if don`t use cloud training_args: # see `transformers.Seq2SeqTrainingArguments` output_dir: ./output max_steps: 3000 # needed to be fit for the dataset learning_rate: 5e-5 # settings for data loading per_device_train_batch_size: 1 dataloader_num_workers: 16 remove_unused_columns: false # settings for saving checkpoints save_strategy: steps save_steps: 500 # settings for logging log_level: info logging_strategy: steps logging_steps: 10 run_name: "glm4-sft-finetune" # settings for evaluation per_device_eval_batch_size: 16 eval_strategy: steps eval_steps: 500 # settings for optimizer # adam_epsilon: 1e-6 # uncomment the following line to detect nan or inf values # debug: underflow_overflow predict_with_generate: true generation_config: max_new_tokens: 512 # set your absolute deepspeed path here deepspeed: configs/ds_zero_3.json ================================================ FILE: finetune/finetune.py ================================================ # -*- coding: utf-8 -*- import dataclasses as dc import functools import os from collections.abc import Callable, Mapping, Sequence from pathlib import Path from typing import Annotated, Any, Optional, Union import jieba import numpy as np import ruamel.yaml as yaml import torch import typer from datasets import Dataset, DatasetDict, NamedSplit, Split, load_dataset from nltk.translate.bleu_score import SmoothingFunction, sentence_bleu from peft import PeftConfig, get_peft_config, get_peft_model from rouge_chinese import Rouge from torch import nn from transformers import ( AutoModelForCausalLM, AutoTokenizer, EvalPrediction, GenerationConfig, PreTrainedTokenizer, Seq2SeqTrainingArguments, ) from transformers import DataCollatorForSeq2Seq as _DataCollatorForSeq2Seq from transformers import Seq2SeqTrainer as _Seq2SeqTrainer # For Ascend NPU, please add this # import torch_npu # from torch_npu.contrib import transfer_to_npu app = typer.Typer(pretty_exceptions_show_locals=False) class DataCollatorForSeq2Seq(_DataCollatorForSeq2Seq): def __call__(self, features, return_tensors=None): output_ids = [feature["output_ids"] for feature in features] if "output_ids" in features[0].keys() else None if output_ids is not None: max_output_length = max(len(out) for out in output_ids) if self.pad_to_multiple_of is not None: max_output_length = ( (max_output_length + self.pad_to_multiple_of - 1) // self.pad_to_multiple_of * self.pad_to_multiple_of ) for feature in features: remainder = [self.tokenizer.pad_token_id] * (max_output_length - len(feature["output_ids"])) if isinstance(feature["output_ids"], list): feature["output_ids"] = feature["output_ids"] + remainder else: feature["output_ids"] = np.concatenate([feature["output_ids"], remainder]).astype(np.int64) return super().__call__(features, return_tensors) class Seq2SeqTrainer(_Seq2SeqTrainer): def prediction_step( self, model: nn.Module, inputs: dict[str, Any], prediction_loss_only: bool, ignore_keys=None, **gen_kwargs, ) -> tuple[Optional[float], Optional[torch.Tensor], Optional[torch.Tensor]]: with torch.no_grad(): # Ensure no gradient computation if self.args.predict_with_generate: output_ids = inputs.pop("output_ids") input_ids = inputs["input_ids"] del inputs["labels"] loss, generated_tokens, labels = super().prediction_step( model, inputs, prediction_loss_only, ignore_keys, **gen_kwargs ) generated_tokens = generated_tokens[:, input_ids.size()[1] :] labels = output_ids del inputs, input_ids, output_ids torch.cuda.empty_cache() return loss, generated_tokens, labels @dc.dataclass class DataConfig(object): train_file: Optional[str] = None val_file: Optional[str] = None test_file: Optional[str] = None num_proc: Optional[int] = None @property def data_format(self) -> str: return Path(self.train_file).suffix @property def data_files(self) -> dict[NamedSplit, str]: return { split: data_file for split, data_file in zip( [Split.TRAIN, Split.VALIDATION, Split.TEST], [self.train_file, self.val_file, self.test_file], ) if data_file is not None } @dc.dataclass class FinetuningConfig(object): data_config: DataConfig max_input_length: int max_output_length: int combine: bool freezeV: bool training_args: Seq2SeqTrainingArguments = dc.field( default_factory=lambda: Seq2SeqTrainingArguments(output_dir="./output") ) peft_config: Optional[PeftConfig] = None swanlab: Optional[str] = "cloud" def __post_init__(self): if not self.training_args.do_eval or self.data_config.val_file is None: self.training_args.do_eval = False self.training_args.evaluation_strategy = "no" self.data_config.val_file = None else: self.training_args.per_device_eval_batch_size = ( self.training_args.per_device_eval_batch_size or self.training_args.per_device_train_batch_size ) if self.swanlab != "disabled": os.environ["SWANLAB_PROJ_NAME"] = "GLM4-Finetune" if self.swanlab == "local": os.environ["SWANLAB_MODE"] = "local" @classmethod def from_dict(cls, **kwargs) -> "FinetuningConfig": training_args = kwargs.get("training_args", None) if training_args is not None and not isinstance(training_args, Seq2SeqTrainingArguments): gen_config = training_args.get("generation_config") if not isinstance(gen_config, GenerationConfig): training_args["generation_config"] = GenerationConfig(**gen_config) kwargs["training_args"] = Seq2SeqTrainingArguments(**training_args) data_config = kwargs.get("data_config") if not isinstance(data_config, DataConfig): kwargs["data_config"] = DataConfig(**data_config) peft_config = kwargs.get("peft_config", None) if peft_config is not None and not isinstance(peft_config, PeftConfig): kwargs["peft_config"] = get_peft_config(config_dict=peft_config) return cls(**kwargs) @classmethod def from_file(cls, path: Union[str, Path]) -> "FinetuningConfig": path = Path(path) parser = yaml.YAML(typ="safe", pure=True) parser.indent(mapping=2, offset=2, sequence=4) parser.default_flow_style = False kwargs = parser.load(path) return cls.from_dict(**kwargs) def _load_datasets( data_dir: str, data_format: str, data_files: dict[NamedSplit, str], num_proc: Optional[int], ) -> DatasetDict: if data_format == ".jsonl": dataset_dct = load_dataset( data_dir, data_files=data_files, split=None, num_proc=num_proc, ) else: raise NotImplementedError(f"Cannot load dataset in the '{data_format}' format.") return dataset_dct class DataManager(object): def __init__(self, data_dir: str, data_config: DataConfig): self._num_proc = data_config.num_proc self._dataset_dct = _load_datasets( data_dir, data_config.data_format, data_config.data_files, self._num_proc, ) def _get_dataset(self, split: NamedSplit) -> Optional[Dataset]: return self._dataset_dct.get(split, None) def get_dataset( self, split: NamedSplit, process_fn: Callable[[dict[str, Any]], dict[str, Any]], batched: bool = True, remove_orig_columns: bool = True, ) -> Optional[Dataset]: orig_dataset = self._get_dataset(split) if orig_dataset is None: return if remove_orig_columns: remove_columns = orig_dataset.column_names else: remove_columns = None return orig_dataset.map( process_fn, batched=batched, remove_columns=remove_columns, num_proc=self._num_proc, ) def process_message(message): if "tools" in message and message["role"] == "system": for tool in message["tools"]: parameters = tool["function"]["parameters"]["properties"] tool["function"]["parameters"]["properties"] = {k: v for k, v in parameters.items() if v is not None} elif "tools" in message: del message["tools"] return message def process_batch( batch: Mapping[str, Sequence], tokenizer: PreTrainedTokenizer, max_input_length: int, max_output_length: int, combine: bool, ) -> dict[str, list]: batched_conv = batch["messages"] batched_input_ids = [] batched_labels = [] for conv in batched_conv: input_ids = [151331, 151333] loss_masks = [False, False] if combine: new_input_ids = tokenizer.apply_chat_template(conv, tokenize=True, return_dict=False) input_ids = new_input_ids loss_masks = [False] * len(input_ids) last_assistant_index = len(input_ids) - input_ids[::-1].index(151337) - 1 for j in range(last_assistant_index + 1, len(input_ids)): loss_masks[j] = True else: for message in conv: message = process_message(message) loss_mask_val = False if message["role"] in ("system", "user", "observation") else True new_input_ids = tokenizer.apply_chat_template([message], tokenize=True, return_dict=False)[2:] input_ids += new_input_ids loss_masks += [loss_mask_val] * len(new_input_ids) input_ids.append(151336) # EOS for chat loss_masks = [False, *loss_masks] labels = [] for input_id, mask in zip(input_ids, loss_masks): if mask: labels.append(input_id) else: labels.append(-100) max_length = max_input_length + max_output_length + 1 batched_input_ids.append(input_ids[:max_length]) batched_labels.append(labels[:max_length]) del batched_conv, conv, input_ids, loss_masks, new_input_ids, labels torch.cuda.empty_cache() return {"input_ids": batched_input_ids, "labels": batched_labels} def process_batch_eval( batch: Mapping[str, Sequence], tokenizer: PreTrainedTokenizer, max_input_length: int, max_output_length: int, combine: bool, ) -> dict[str, list]: batched_conv = batch["messages"] batched_input_ids = [] batched_output_ids = [] for conv in batched_conv: if combine: new_input_ids = tokenizer.apply_chat_template(conv, tokenize=True, return_dict=False) input_ids = new_input_ids last_assistant_index = len(input_ids) - input_ids[::-1].index(151337) - 1 output_prompt, output_ids = ( input_ids[:1], input_ids[last_assistant_index:], ) output_ids.append(151336) batched_input_ids.append(input_ids[:max_input_length] + output_prompt[:1]) batched_output_ids.append(output_ids[:max_output_length]) else: input_ids = [151331, 151333] for message in conv: if len(input_ids) >= max_input_length: break else: message = process_message(message) new_input_ids = tokenizer.apply_chat_template([message], tokenize=True, return_dict=False)[2:] if message["role"] == "assistant": output_prompt, output_ids = ( new_input_ids[:1], new_input_ids[1:], ) output_ids.append(151336) batched_input_ids.append(input_ids[:max_input_length] + output_prompt[:1]) batched_output_ids.append(output_ids[:max_output_length]) input_ids += new_input_ids del batched_conv, conv, input_ids, new_input_ids, output_prompt, output_ids torch.cuda.empty_cache() return {"input_ids": batched_input_ids, "output_ids": batched_output_ids} def load_tokenizer_and_model( model_dir: str, peft_config: Optional[PeftConfig] = None, ): tokenizer = AutoTokenizer.from_pretrained(model_dir, padding_side="left", trust_remote_code=True) if peft_config is not None: model = AutoModelForCausalLM.from_pretrained( model_dir, use_cache=False, torch_dtype=torch.bfloat16, # Must use BFloat 16 ) model = get_peft_model(model, peft_config) model.print_trainable_parameters() else: model = AutoModelForCausalLM.from_pretrained( model_dir, use_cache=False, torch_dtype=torch.bfloat16, ) return tokenizer, model def compute_metrics(eval_preds: EvalPrediction, tokenizer): batched_pred_ids, batched_label_ids = eval_preds batched_pred_ids[batched_pred_ids == -100] = tokenizer.pad_token_id batched_label_ids[batched_label_ids == -100] = tokenizer.pad_token_id metrics_dct = {"rouge-1": [], "rouge-2": [], "rouge-l": [], "bleu-4": []} for pred_ids, label_ids in zip(batched_pred_ids, batched_label_ids): pred_txt = tokenizer.decode(pred_ids).strip() label_txt = tokenizer.decode(label_ids).strip() pred_tokens = list(jieba.cut(pred_txt)) label_tokens = list(jieba.cut(label_txt)) rouge = Rouge() scores = rouge.get_scores(" ".join(pred_tokens), " ".join(label_tokens)) for k, v in scores[0].items(): metrics_dct[k].append(round(v["f"] * 100, 4)) metrics_dct["bleu-4"].append( sentence_bleu( [label_tokens], pred_tokens, smoothing_function=SmoothingFunction().method3, ) ) return {k: np.mean(v) for k, v in metrics_dct.items()} @app.command() def main( data_dir: Annotated[str, typer.Argument(help="")], model_dir: Annotated[ str, typer.Argument( help="A string that specifies the model id of a pretrained model configuration hosted on huggingface.co, or a path to a directory containing a model configuration file." ), ], config_file: Annotated[str, typer.Argument(help="")], auto_resume_from_checkpoint: str = typer.Argument( default="", help="If entered as yes, automatically use the latest save checkpoint. If it is a numerical example 12 15, use the corresponding save checkpoint. If the input is no, restart training", ), ): ft_config = FinetuningConfig.from_file(config_file) tokenizer, model = load_tokenizer_and_model(model_dir, peft_config=ft_config.peft_config) data_manager = DataManager(data_dir, ft_config.data_config) train_dataset = data_manager.get_dataset( Split.TRAIN, functools.partial( process_batch, tokenizer=tokenizer, combine=ft_config.combine, max_input_length=ft_config.max_input_length, max_output_length=ft_config.max_output_length, ), batched=True, ) print("train_dataset:", train_dataset) val_dataset = data_manager.get_dataset( Split.VALIDATION, functools.partial( process_batch_eval, tokenizer=tokenizer, combine=ft_config.combine, max_input_length=ft_config.max_input_length, max_output_length=ft_config.max_output_length, ), batched=True, ) if val_dataset is not None: print("val_dataset:", val_dataset) test_dataset = data_manager.get_dataset( Split.TEST, functools.partial( process_batch_eval, tokenizer=tokenizer, combine=ft_config.combine, max_input_length=ft_config.max_input_length, max_output_length=ft_config.max_output_length, ), batched=True, ) if test_dataset is not None: print("test_dataset:", test_dataset) ft_config.training_args.generation_config.pad_token_id = 151329 ft_config.training_args.generation_config.eos_token_id = [151329, 151336, 151338] trainer = Seq2SeqTrainer( model=model, args=ft_config.training_args, data_collator=DataCollatorForSeq2Seq( tokenizer=tokenizer, padding="longest", return_tensors="pt", ), train_dataset=train_dataset, eval_dataset=val_dataset, compute_metrics=functools.partial(compute_metrics, tokenizer=tokenizer), ) if auto_resume_from_checkpoint.upper() == "" or auto_resume_from_checkpoint is None: trainer.train() else: output_dir = ft_config.training_args.output_dir dirlist = os.listdir(output_dir) checkpoint_sn = 0 for checkpoint_str in dirlist: if checkpoint_str.find("eckpoint") > 0 and checkpoint_str.find("tmp") == -1: checkpoint = int(checkpoint_str.replace("checkpoint-", "")) if checkpoint > checkpoint_sn: checkpoint_sn = checkpoint if auto_resume_from_checkpoint.upper() == "YES": if checkpoint_sn > 0: model.gradient_checkpointing_enable() model.enable_input_require_grads() checkpoint_directory = os.path.join(output_dir, "checkpoint-" + str(checkpoint_sn)) print("resume checkpoint from checkpoint-" + str(checkpoint_sn)) trainer.train(resume_from_checkpoint=checkpoint_directory) else: trainer.train() else: if auto_resume_from_checkpoint.isdigit(): if int(auto_resume_from_checkpoint) > 0: checkpoint_sn = int(auto_resume_from_checkpoint) model.gradient_checkpointing_enable() model.enable_input_require_grads() checkpoint_directory = os.path.join(output_dir, "checkpoint-" + str(checkpoint_sn)) print("resume checkpoint from checkpoint-" + str(checkpoint_sn)) trainer.train(resume_from_checkpoint=checkpoint_directory) else: print( auto_resume_from_checkpoint, "The specified checkpoint sn(" + auto_resume_from_checkpoint + ") has not been saved. Please search for the correct checkpoint in the model output directory", ) if test_dataset is not None: trainer.predict(test_dataset) if __name__ == "__main__": app() ================================================ FILE: finetune/finetune_vision.py ================================================ # -*- coding: utf-8 -*- import dataclasses as dc import functools import os from collections.abc import Callable, Mapping, Sequence from pathlib import Path from typing import Annotated, Any, Optional, Union import jieba import numpy as np import ruamel.yaml as yaml import torch import typer from datasets import Dataset, DatasetDict, NamedSplit, Split, load_dataset from nltk.translate.bleu_score import SmoothingFunction, sentence_bleu from peft import PeftConfig, get_peft_config, get_peft_model from PIL import Image from rouge_chinese import Rouge from torch import nn from transformers import ( AutoModelForCausalLM, AutoTokenizer, EvalPrediction, GenerationConfig, PreTrainedTokenizer, Seq2SeqTrainingArguments, ) from transformers import DataCollatorForSeq2Seq as _DataCollatorForSeq2Seq from transformers import Seq2SeqTrainer as _Seq2SeqTrainer # For Ascend NPU, please add this # import torch_npu # from torch_npu.contrib import transfer_to_npu app = typer.Typer(pretty_exceptions_show_locals=False) img = Image.new("L", (224, 224), 0).convert("RGB") class DataCollatorForSeq2Seq(_DataCollatorForSeq2Seq): def __call__(self, features, return_tensors=None): output_ids = [feature["output_ids"] for feature in features] if "output_ids" in features[0].keys() else None if output_ids is not None: max_output_length = max(len(out) for out in output_ids) if self.pad_to_multiple_of is not None: max_output_length = ( (max_output_length + self.pad_to_multiple_of - 1) // self.pad_to_multiple_of * self.pad_to_multiple_of ) for feature in features: remainder = [self.tokenizer.pad_token_id] * (max_output_length - len(feature["output_ids"])) if isinstance(feature["output_ids"], list): feature["output_ids"] = feature["output_ids"] + remainder else: feature["output_ids"] = np.concatenate([feature["output_ids"], remainder]).astype(np.int64) return super().__call__(features, return_tensors) class Seq2SeqTrainer(_Seq2SeqTrainer): def prediction_step( self, model: nn.Module, inputs: dict, prediction_loss_only: bool, ignore_keys=None, **gen_kwargs, ) -> tuple[Optional[float], Optional[torch.Tensor], Optional[torch.Tensor]]: with torch.no_grad(): if self.args.predict_with_generate: output_ids = inputs.pop("output_ids", None) del inputs["labels"] loss, generated_tokens, labels = super().prediction_step( model=model, inputs=inputs, prediction_loss_only=prediction_loss_only, ignore_keys=ignore_keys, **gen_kwargs, ) if generated_tokens is not None: generated_tokens = generated_tokens[:, inputs["input_ids"].size()[1] :] if self.args.predict_with_generate: labels = output_ids del inputs, output_ids torch.cuda.empty_cache() return loss, generated_tokens, labels @dc.dataclass class DataConfig(object): train_file: Optional[str] = None val_file: Optional[str] = None test_file: Optional[str] = None num_proc: Optional[int] = None @property def data_format(self) -> str: return Path(self.train_file).suffix @property def data_files(self) -> dict[NamedSplit, str]: return { split: data_file for split, data_file in zip( [Split.TRAIN, Split.VALIDATION, Split.TEST], [self.train_file, self.val_file, self.test_file], ) if data_file is not None } @dc.dataclass class FinetuningConfig(object): data_config: DataConfig max_input_length: int max_output_length: int combine: bool freezeV: bool swanlab: Optional[str] = "cloud" training_args: Seq2SeqTrainingArguments = dc.field( default_factory=lambda: Seq2SeqTrainingArguments(output_dir="./output") ) peft_config: Optional[PeftConfig] = None def __post_init__(self): if not self.training_args.do_eval or self.data_config.val_file is None: self.training_args.do_eval = False self.training_args.evaluation_strategy = "no" self.data_config.val_file = None else: self.training_args.per_device_eval_batch_size = ( self.training_args.per_device_eval_batch_size or self.training_args.per_device_train_batch_size ) if self.swanlab != "disabled": os.environ["SWANLAB_PROJ_NAME"] = "GLM4-Finetune" if self.swanlab == "local": os.environ["SWANLAB_MODE"] = "local" @classmethod def from_dict(cls, **kwargs) -> "FinetuningConfig": training_args = kwargs.get("training_args", None) if training_args is not None and not isinstance(training_args, Seq2SeqTrainingArguments): gen_config = training_args.get("generation_config") if not isinstance(gen_config, GenerationConfig): training_args["generation_config"] = GenerationConfig(**gen_config) kwargs["training_args"] = Seq2SeqTrainingArguments(**training_args) data_config = kwargs.get("data_config") if not isinstance(data_config, DataConfig): kwargs["data_config"] = DataConfig(**data_config) peft_config = kwargs.get("peft_config", None) if peft_config is not None and not isinstance(peft_config, PeftConfig): kwargs["peft_config"] = get_peft_config(config_dict=peft_config) return cls(**kwargs) @classmethod def from_file(cls, path: Union[str, Path]) -> "FinetuningConfig": path = Path(path) parser = yaml.YAML(typ="safe", pure=True) parser.indent(mapping=2, offset=2, sequence=4) parser.default_flow_style = False kwargs = parser.load(path) return cls.from_dict(**kwargs) def _load_datasets( data_dir: str, data_format: str, data_files: dict[NamedSplit, str], num_proc: Optional[int], ) -> DatasetDict: if data_format == ".jsonl": dataset_dct = load_dataset( data_dir, data_files=data_files, split=None, num_proc=num_proc, ) else: raise NotImplementedError(f"Cannot load dataset in the '{data_format}' format.") return dataset_dct class DataManager(object): def __init__(self, data_dir: str, data_config: DataConfig): self._num_proc = data_config.num_proc self._dataset_dct = _load_datasets( data_dir, data_config.data_format, data_config.data_files, self._num_proc, ) def _get_dataset(self, split: NamedSplit) -> Optional[Dataset]: return self._dataset_dct.get(split, None) def get_dataset( self, split: NamedSplit, process_fn: Callable[[dict[str, Any]], dict[str, Any]], batched: bool = True, remove_orig_columns: bool = True, ) -> Optional[Dataset]: orig_dataset = self._get_dataset(split) if orig_dataset is None: return if remove_orig_columns: remove_columns = orig_dataset.column_names else: remove_columns = None return orig_dataset.map( process_fn, batched=batched, remove_columns=remove_columns, num_proc=self._num_proc, # This is default params of orig_dataset.map, and you can change it smaller # https://github.com/THUDM/GLM-4/issues/277 writer_batch_size=1000, batch_size=1000, ) def process_batch( batch: Mapping[str, Sequence], tokenizer: PreTrainedTokenizer, max_input_length: int, max_output_length: int, combine: bool, ) -> dict[str, list]: batched_conv = batch["messages"] batched_input_ids = [] batched_attention_mask = [] batched_position_ids = [] batched_labels = [] batched_images = [] max_length = max_input_length + max_output_length for conv in batched_conv: input_ids = [151331, 151333] attention_mask = [1, 1] position_ids = list(range(len(input_ids))) loss_masks = [False, False] images = [] if conv[0].get("image"): conv[0]["image"] = Image.open(conv[0]["image"]).convert("RGB") else: conv[0]["image"] = img for message in conv: loss_mask_val = False if message["role"] in ("system", "user", "observation") else True new_input_ids_all = tokenizer.apply_chat_template([message], tokenize=True, return_dict=True, padding=True) new_input_ids = new_input_ids_all["input_ids"][0][2:] new_attention_mask = new_input_ids_all["attention_mask"][0][2:] new_position_ids = list(range(position_ids[-1] + 1, position_ids[-1] + 1 + len(new_input_ids))) if message.get("image"): # Only One Image images.append(new_input_ids_all["images"]) new_loss_masks = [loss_mask_val] * len(new_input_ids) input_ids += new_input_ids attention_mask += new_attention_mask position_ids += new_position_ids loss_masks += new_loss_masks input_ids.append(151336) # EOS attention_mask.append(1) position_ids.append(len(position_ids)) loss_masks.append(False) labels = [] for input_id, mask in zip(input_ids, loss_masks): if mask: labels.append(input_id) else: labels.append(-100) batched_input_ids.append(input_ids[:max_length]) batched_attention_mask.append(attention_mask[:max_length]) batched_position_ids.append(position_ids[:max_length]) batched_labels.append(labels[:max_length]) batched_images.append(images[0][0]) del ( batched_conv, conv, input_ids, attention_mask, position_ids, loss_masks, message, new_input_ids, new_loss_masks, labels, input_id, mask, ) torch.cuda.empty_cache() return { "input_ids": batched_input_ids, "attention_mask": batched_attention_mask, "position_ids": batched_position_ids, "labels": batched_labels, "images": batched_images, } def process_batch_eval( batch: Mapping[str, Sequence], tokenizer: PreTrainedTokenizer, max_input_length: int, max_output_length: int, combine: bool, ) -> dict[str, list]: batched_conv = batch["messages"] batched_input_ids = [] batched_attention_mask = [] batched_position_ids = [] batched_output_ids = [] batched_images = [] for conv in batched_conv: if conv[0].get("image"): image = Image.open(conv[0]["image"]).convert("RGB") else: image = img conv[0]["image"] = image new_input_ids_all = tokenizer.apply_chat_template(conv, tokenize=True, return_dict=True, padding=True) input_ids = new_input_ids_all["input_ids"][0] attention_mask = new_input_ids_all["attention_mask"][0] position_ids = list(range(len(input_ids))) dialogue_parts = [0] user_idx = [] for idx, token_id in enumerate(input_ids): if token_id == 151337: dialogue_parts.append(idx + 1) elif token_id == 151336: user_idx.append(idx) if user_idx[-1] != len(input_ids): user_idx.append(len(input_ids)) # Split the conversation into multiple dialogue segments for end_idx in range(1, len(dialogue_parts)): input_segment = input_ids[: dialogue_parts[end_idx]] attention_segment = attention_mask[: dialogue_parts[end_idx]] position_segment = position_ids[: dialogue_parts[end_idx]] output_segment = input_ids[dialogue_parts[end_idx] : user_idx[end_idx]] output_segment.append(151336) # Add EOS token batched_input_ids.append(input_segment[:max_input_length]) batched_attention_mask.append(attention_segment[:max_input_length]) batched_position_ids.append(position_segment[:max_input_length]) batched_output_ids.append(output_segment[:max_output_length]) batched_images.append(new_input_ids_all["images"][0]) del ( batched_conv, input_ids, attention_mask, position_ids, new_input_ids_all, output_segment, ) torch.cuda.empty_cache() return { "input_ids": batched_input_ids, "attention_mask": batched_attention_mask, "position_ids": batched_position_ids, "output_ids": batched_output_ids, "images": batched_images, } def load_tokenizer_and_model( model_dir: str, peft_config: Optional[PeftConfig] = None, ): tokenizer = AutoTokenizer.from_pretrained(model_dir, padding_side="left", trust_remote_code=True) if peft_config is not None: model = AutoModelForCausalLM.from_pretrained( model_dir, trust_remote_code=True, use_cache=False, torch_dtype=torch.bfloat16, # Must use BFloat 16 ) model = get_peft_model(model, peft_config) model.print_trainable_parameters() else: model = AutoModelForCausalLM.from_pretrained( model_dir, trust_remote_code=True, use_cache=False, torch_dtype=torch.bfloat16, ) return tokenizer, model def compute_metrics(eval_preds: EvalPrediction, tokenizer): batched_pred_ids, batched_label_ids = eval_preds batched_pred_ids[batched_pred_ids == -100] = tokenizer.pad_token_id batched_label_ids[batched_label_ids == -100] = tokenizer.pad_token_id metrics_dct = {"rouge-1": [], "rouge-2": [], "rouge-l": [], "bleu-4": []} for pred_ids, label_ids in zip(batched_pred_ids, batched_label_ids): pred_txt = tokenizer.decode(pred_ids).strip() label_txt = tokenizer.decode(label_ids).strip() pred_tokens = list(jieba.cut(pred_txt)) label_tokens = list(jieba.cut(label_txt)) rouge = Rouge() scores = rouge.get_scores(" ".join(pred_tokens), " ".join(label_tokens)) for k, v in scores[0].items(): metrics_dct[k].append(round(v["f"] * 100, 4)) metrics_dct["bleu-4"].append( sentence_bleu( [label_tokens], pred_tokens, smoothing_function=SmoothingFunction().method3, ) ) return {k: np.mean(v) for k, v in metrics_dct.items()} @app.command() def main( data_dir: Annotated[str, typer.Argument(help="")], model_dir: Annotated[ str, typer.Argument( help="A string that specifies the model id of a pretrained model configuration hosted on huggingface.co, or a path to a directory containing a model configuration file." ), ], config_file: Annotated[str, typer.Argument(help="")], auto_resume_from_checkpoint: str = typer.Argument( default="", help="If entered as yes, automatically use the latest save checkpoint. If it is a numerical example 12 15, use the corresponding save checkpoint. If the input is no, restart training", ), ): ft_config = FinetuningConfig.from_file(config_file) tokenizer, model = load_tokenizer_and_model(model_dir, peft_config=ft_config.peft_config) if ft_config.freezeV: for param in model.transformer.vision.parameters(): param.requires_grad = False data_manager = DataManager(data_dir, ft_config.data_config) train_dataset = data_manager.get_dataset( Split.TRAIN, functools.partial( process_batch, combine=ft_config.combine, # Not use now tokenizer=tokenizer, max_input_length=ft_config.max_input_length, max_output_length=ft_config.max_output_length, ), batched=True, ) print("train_dataset:", train_dataset) val_dataset = data_manager.get_dataset( Split.VALIDATION, functools.partial( process_batch_eval, combine=ft_config.combine, tokenizer=tokenizer, max_input_length=ft_config.max_input_length, max_output_length=ft_config.max_output_length, ), batched=True, ) if val_dataset is not None: print("val_dataset:", val_dataset) test_dataset = data_manager.get_dataset( Split.TEST, functools.partial( process_batch_eval, combine=ft_config.combine, tokenizer=tokenizer, max_input_length=ft_config.max_input_length, max_output_length=ft_config.max_output_length, ), batched=True, ) if test_dataset is not None: print("test_dataset:", test_dataset) ft_config.training_args.generation_config.pad_token_id = 151329 ft_config.training_args.generation_config.eos_token_id = [151329, 151336, 151338] trainer = Seq2SeqTrainer( model=model, args=ft_config.training_args, data_collator=DataCollatorForSeq2Seq( tokenizer=tokenizer, padding="longest", return_tensors="pt", ), train_dataset=train_dataset, eval_dataset=val_dataset, compute_metrics=functools.partial(compute_metrics, tokenizer=tokenizer), ) if auto_resume_from_checkpoint.upper() == "" or auto_resume_from_checkpoint is None: trainer.train() else: output_dir = ft_config.training_args.output_dir dirlist = os.listdir(output_dir) checkpoint_sn = 0 for checkpoint_str in dirlist: if checkpoint_str.find("eckpoint") > 0 and checkpoint_str.find("tmp") == -1: checkpoint = int(checkpoint_str.replace("checkpoint-", "")) if checkpoint > checkpoint_sn: checkpoint_sn = checkpoint if auto_resume_from_checkpoint.upper() == "YES": if checkpoint_sn > 0: model.gradient_checkpointing_enable() model.enable_input_require_grads() checkpoint_directory = os.path.join(output_dir, "checkpoint-" + str(checkpoint_sn)) print("resume checkpoint from checkpoint-" + str(checkpoint_sn)) trainer.train(resume_from_checkpoint=checkpoint_directory) else: trainer.train() else: if auto_resume_from_checkpoint.isdigit(): if int(auto_resume_from_checkpoint) > 0: checkpoint_sn = int(auto_resume_from_checkpoint) model.gradient_checkpointing_enable() model.enable_input_require_grads() checkpoint_directory = os.path.join(output_dir, "checkpoint-" + str(checkpoint_sn)) print("resume checkpoint from checkpoint-" + str(checkpoint_sn)) trainer.train(resume_from_checkpoint=checkpoint_directory) else: print( auto_resume_from_checkpoint, "The specified checkpoint sn(" + auto_resume_from_checkpoint + ") has not been saved. Please search for the correct checkpoint in the model output directory", ) if test_dataset is not None: trainer.predict(test_dataset) if __name__ == "__main__": app() ================================================ FILE: finetune/requirements.txt ================================================ jieba>=0.42.1 datasets>=2.20.0 peft>=0.15.1 deepspeed>=0.16.5 nltk==3.8.1 rouge_chinese==1.0.3 ruamel.yaml>=0.18.6 typer>=0.13.0 tqdm>=4.67.0 ================================================ FILE: inference/README.md ================================================ # Inference [中文阅读](README_zh.md) Please follow the steps in the document strictly to avoid unnecessary errors. ## Device and dependency check ### Install dependencies ```shell pip install -r requirements.txt ``` ### Related Inference Benchmark Data **All benchmark data in this document was collected under the hardware environment listed below. Actual memory usage and runtime may vary depending on your deployment setup. Please refer to your actual environment.** Test Hardware: + OS: Ubuntu 22.04 + Memory: 512GB + Python: 3.12.3 + Cmake 3.23.0 + CUDA Version: 12.4 + GPU Driver: 535.104.05 + GPU: NVIDIA H100 80GB HBM3 * 8 The following stress test results show memory usage and latency during inference. If multiple GPUs are used, "Memory Usage" refers to the maximum usage on a single GPU. #### GLM-4-32B-0414 | Precision | #GPUs | Memory Usage | First Token Latency | Token Output Speed | Input Tokens | |-------------|-------|---------------|---------------------|-------------------|--------------| | BF16 | 1 | 68 GB | 0.16s | 24.4 tokens/s | 1000 | | BF16 | 1 | 72 GB | 1.37s | 16.9 tokens/s | 8000 | | BF16 | 2 | 50 GB | 6.75s | 8.1 tokens/s | 32000 | | BF16 | 4 | 55 GB | 37.83s | 3.0 tokens/s | 100000 | #### GLM-4-9B-0414 | Precision | #GPUs | Memory Usage | First Token Latency | Token Output Speed | Input Tokens | |-----------|-------|---------------|----------------------|---------------------|---------------| | BF16 | 1 | 19 GB | 0.05s | 44.4 tokens/s | 1000 | | BF16 | 1 | 25 GB | 0.39s | 39.0 tokens/s | 8000 | | BF16 | 1 | 31 GB | 2.29s | 18.7 tokens/s | 32000 | | BF16 | 1 | 55 GB | 6.80s | 14.1 tokens/s | 100000 | #### GLM-4-9B-Chat-1M | Precision | #GPUs | Memory Usage | First Token Latency | Token Output Speed | Input Tokens | |-----------|-------|---------------|----------------------|---------------------|---------------| | BF16 | 1 | 75 GB | 98.4s | 2.3 tokens/s | 200000 | #### GLM-4V-9B | Precision | #GPUs | Memory Usage | First Token Latency | Token Output Speed | Input Tokens | |-----------|-------|---------------|----------------------|---------------------|---------------| | BF16 | 1 | 28 GB | 0.1s | 33.4 tokens/s | 1000 | | BF16 | 1 | 33 GB | 0.7s | 39.2 tokens/s | 8000 | | Precision | #GPUs | Memory Usage | First Token Latency | Token Output Speed | Input Tokens | |-----------|-------|---------------|----------------------|---------------------|---------------| | INT4 | 1 | 10 GB | 0.1s | 28.7 tokens/s | 1000 | | INT4 | 1 | 15 GB | 0.8s | 24.2 tokens/s | 8000 | ## Quick Start ### Use transformers backend code + Use the command line to communicate with the GLM-4-9B model. ```shell python trans_cli_demo.py # LLM Such as GLM-4-9B-0414 python trans_cli_vision_demo.py # GLM-4V-9B ``` + Use the Gradio web client to communicate with the GLM-4-9B model. ```shell python trans_web_demo.py # LLM Such as GLM-4-9B-0414 python trans_web_vision_demo.py # GLM-4V-9B ``` + Use Batch inference. ```shell python trans_batch_demo.py # LLM Such as GLM-4-9B-0414 ``` ### Use vLLM backend code + Use the command line to communicate with the GLM-4-9B-Chat model. ```shell python vllm_cli_demo.py # LLM Such as GLM-4-9B-0414 ``` + Launch an OpenAI-compatible API service. ```shell vllm serve THUDM/GLM-4-9B-0414 --tensor_parallel_size 2 ``` ### Use glm-4v to build an OpenAI-compatible service Start the server: ```shell python glm4v_server.py THUDM/glm-4v-9b ``` Client request: ```shell python glm4v_api_request.py ``` ## Stress test Users can use this code to test the generation speed of the model on the transformers backend on their own devices: ```shell python trans_stress_test.py ``` The stress test script supports enabling **SwanLab** to track the stress testing process and record metrics: ```shell # The API Key can be obtained by logging in to https://swanlab.cn/ python trans_stress_test.py --swanlab_api_key "Your SwanLab API Key" ``` Using the --swanlab_api_key local parameter enables SwanLab's local mode. ## Use Ascend card to run code Users can run the above code in the Ascend hardware environment. They only need to change the transformers to openmind and the cuda device in device to npu. ```shell #from transformers import AutoModelForCausalLM, AutoTokenizer from openmind import AutoModelForCausalLM, AutoTokenizer #device = 'cuda' device = 'npu' ``` ================================================ FILE: inference/README_zh.md ================================================ # Inference Read this in [English](README.md) 请严格按照文档的步骤进行操作，以避免不必要的错误。 ## 设备和依赖检查 ### 安装依赖 ```shell pip install -r requirements.txt ``` ### 相关推理测试数据 **本文档的数据均在以下硬件环境测试,实际运行环境需求和运行占用的显存略有不同，请以实际运行环境为准。** 测试硬件信息: + OS: Ubuntu 22.04 + Memory: 512GB + Python: 3.12.3 + CUDA Version: 12.4 + Cmake 3.23.0 + GPU Driver: 535.104.05 + GPU: NVIDIA H100 80GB HBM3 * 8 推理的压力测试数据如下，如有多张显卡，则显存占用代表显存占用最大一张显卡的显存消耗。 #### GLM-4-32B-0414 | 精度 | 显卡数量 | 显存占用 | 首 Token 延迟 | Token 输出速度 | 输入token数 | |------|------|-------|------------|---------------|----------| | BF16 | 1 | 68 GB | 0.16s | 24.4 tokens/s | 1000 | | BF16 | 1 | 72 GB | 1.37s | 16.9 tokens/s | 8000 | | BF16 | 2 | 50 GB | 6.75s | 8.1 tokens/s | 32000 | | BF16 | 4 | 55 GB | 37.83s | 3.0 tokens/s | 100000 | #### GLM-4-9B-0414 | 精度 | 显卡数量 | 显存占用 | 首 Token 延迟 | Token 输出速度 | 输入token数 | |------|------|-------|------------|---------------|---------| | BF16 | 1 | 19 GB | 0.05s | 44.4 tokens/s | 1000 | | BF16 | 1 | 25 GB | 0.39s | 39.0 tokens/s | 8000 | | BF16 | 1 | 31 GB | 2.29s | 18.7 tokens/s | 32000 | | BF16 | 1 | 55 GB | 6.80s | 14.1 tokens/s | 100000 | #### GLM-4-9B-Chat-1M | 精度 | 显卡数量 | 显存占用 | 首 Token 延迟 | Token 输出速度 | 输入token数 | |--------|------|------|------------|--------------|-------------| | BF16 | 1 | 75 GB | 98.4s | 2.3 tokens/s | 200000 | #### GLM-4V-9B | 精度 | 显卡数量 | 显存占用 | 首 Token 延迟 | Token 输出速度 | 输入token数 | |--------|------|------|------------|--------------|-------------| | BF16 | 1 | 28 GB | 0.1s | 33.4 tokens/s | 1000 | | BF16 | 1 | 33 GB | 0.7s | 39.2 tokens/s | 8000 | | 精度 | 显卡数量 | 显存占用 | 首 Token 延迟 | Token 输出速度 | 输入token数 | |--------|-------|--------|------------|--------------|-------------| | INT4 | 1 | 10 GB | 0.1s | 28.7 tokens/s | 1000 | | INT4 | 1 | 15 GB | 0.8s | 24.2 tokens/s | 8000 | ## 快速开始 ### 使用 transformers 后端代码 + 使用命令行与 GLM-4-9B 模型进行对话。 ```shell python trans_cli_demo.py # LLM Such as GLM-4-9B-0414 python trans_cli_vision_demo.py # GLM-4V-9B ``` + 使用 Gradio 网页端与 GLM-4-9B 模型进行对话。 ```shell python trans_web_demo.py # LLM Such as GLM-4-9B-0414 python trans_web_vision_demo.py # GLM-4V-9B ``` + 使用 Batch 推理。 ```shell python trans_batch_demo.py ``` ### 使用 vLLM 后端代码 + 使用命令行与 GLM-4-9B-Chat 模型进行对话。 ```shell python vllm_cli_demo.py # LLM Such as GLM-4-9B-0414 ``` + 构建 OpenAI 类 API 服务。 ```shell vllm serve THUDM/GLM-4-9B-0414 --tensor_parallel_size 2 ``` ### 使用 glm-4v 构建 OpenAI 服务 启动服务端 ```shell python glm4v_server.py THUDM/glm-4v-9b ``` 客户端请求： ```shell python glm4v_api_request.py ``` ## 压力测试 用户可以在自己的设备上使用本代码测试模型在 transformers后端的生成速度: ```shell python trans_stress_test.py ``` ## 压力测试 用户可以在自己的设备上使用本代码测试模型在 transformers后端的生成速度: ```shell python trans_stress_test.py ``` 压力测试脚本支持开启**SwanLab**来跟踪压力测试过程和记录指标： ```shell # API Key 可通过登录https://swanlab.cn/获取 python trans_stress_test.py --swanlab_api_key "SwanLab的API Key" ``` 使用`--swanlab_api_key local`参数可开启SwanLab本地模式 ## 使用昇腾NPU运行代码 用户可以在昇腾硬件环境下运行以上代码，只需将transformers修改为openmind，将device中的cuda设备修改为npu： ```shell #from transformers import AutoModelForCausalLM, AutoTokenizer from openmind import AutoModelForCausalLM, AutoTokenizer #device = 'cuda' device = 'npu' ``` ================================================ FILE: inference/glm4v_api_request.py ================================================ """ This script creates a OpenAI Request demo for the glm-4v-9b model, just Use OpenAI API to interact with the model. For LLM such as GLM-4-9B-0414, using with vLLM OpenAI Server. vllm serve THUDM/GLM-4-32B-0414 --tensor_parallel_size 4 """ import base64 from openai import OpenAI base_url = "http://127.0.0.1:8000/v1/" client = OpenAI(api_key="EMPTY", base_url=base_url) def create_chat_completion(messages, use_stream=False): response = client.chat.completions.create( model="glm-4v", messages=messages, stream=use_stream, max_tokens=256, temperature=0.4, presence_penalty=1.2, top_p=0.8, ) if response: if use_stream: for chunk in response: print(chunk) else: print(response) else: print("Error:", response.status_code) def encode_image(image_path): """ Encodes an image file into a base64 string. Args: image_path (str): The path to the image file. This function opens the specified image file, reads its content, and encodes it into a base64 string. The base64 encoding is used to send images over HTTP as text. """ with open(image_path, "rb") as image_file: return base64.b64encode(image_file.read()).decode("utf-8") def glm4v_simple_image_chat(use_stream=False, img_path=None): """ Facilitates a simple chat interaction involving an image. Args: use_stream (bool): Specifies whether to use streaming for chat responses. img_path (str): Path to the image file to be included in the chat. This function encodes the specified image and constructs a predefined conversation involving the image. It then calls `create_chat_completion` to generate a response from the model. The conversation includes asking about the content of the image and a follow-up question. """ img_url = f"data:image/jpeg;base64,{encode_image(img_path)}" messages = [ { "role": "user", "content": [ { "type": "text", "text": "What’s in this image?", }, { "type": "image_url", "image_url": {"url": img_url}, }, ], }, { "role": "assistant", "content": "The image displays a wooden boardwalk extending through a vibrant green grassy wetland. The sky is partly cloudy with soft, wispy clouds, indicating nice weather. Vegetation is seen on either side of the boardwalk, and trees are present in the background, suggesting that this area might be a natural reserve or park designed for ecological preservation and outdoor recreation. The boardwalk allows visitors to explore the area without disturbing the natural habitat.", }, {"role": "user", "content": "Do you think this is a spring or winter photo?"}, ] create_chat_completion(messages=messages, use_stream=use_stream) if __name__ == "__main__": glm4v_simple_image_chat(use_stream=False, img_path="demo.jpg") ================================================ FILE: inference/glm4v_server.py ================================================ import base64 import gc import sys import threading import time from contextlib import asynccontextmanager from io import BytesIO from pathlib import Path from typing import List, Literal, Optional, Tuple, Union import requests import torch import uvicorn from fastapi import FastAPI, HTTPException from fastapi.middleware.cors import CORSMiddleware from peft import PeftModelForCausalLM from PIL import Image from pydantic import BaseModel, Field from sse_starlette.sse import EventSourceResponse from transformers import AutoModel, AutoTokenizer, TextIteratorStreamer TORCH_TYPE = ( torch.bfloat16 if torch.cuda.is_available() and torch.cuda.get_device_capability()[0] >= 8 else torch.float16 ) @asynccontextmanager async def lifespan(app: FastAPI): """ An asynchronous context manager for managing the lifecycle of the FastAPI app. It ensures that GPU memory is cleared after the app's lifecycle ends, which is essential for efficient resource management in GPU environments. """ yield if torch.cuda.is_available(): torch.cuda.empty_cache() torch.cuda.ipc_collect() app = FastAPI(lifespan=lifespan) app.add_middleware( CORSMiddleware, allow_origins=["*"], allow_credentials=True, allow_methods=["*"], allow_headers=["*"], ) class ModelCard(BaseModel): """ A Pydantic model representing a model card, which provides metadata about a machine learning model. It includes fields like model ID, owner, and creation time. """ id: str object: str = "model" created: int = Field(default_factory=lambda: int(time.time())) owned_by: str = "owner" root: Optional[str] = None parent: Optional[str] = None permission: Optional[list] = None class ModelList(BaseModel): object: str = "list" data: List[ModelCard] = [] class ImageUrl(BaseModel): url: str class TextContent(BaseModel): type: Literal["text"] text: str class ImageUrlContent(BaseModel): type: Literal["image_url"] image_url: ImageUrl ContentItem = Union[TextContent, ImageUrlContent] class ChatMessageInput(BaseModel): role: Literal["user", "assistant", "system"] content: Union[str, List[ContentItem]] name: Optional[str] = None class ChatMessageResponse(BaseModel): role: Literal["assistant"] content: str = None name: Optional[str] = None class DeltaMessage(BaseModel): role: Optional[Literal["user", "assistant", "system"]] = None content: Optional[str] = None class ChatCompletionRequest(BaseModel): model: str messages: List[ChatMessageInput] temperature: Optional[float] = 0.8 top_p: Optional[float] = 0.8 max_tokens: Optional[int] = None stream: Optional[bool] = False # Additional parameters repetition_penalty: Optional[float] = 1.0 class ChatCompletionResponseChoice(BaseModel): index: int message: ChatMessageResponse class ChatCompletionResponseStreamChoice(BaseModel): index: int delta: DeltaMessage class UsageInfo(BaseModel): prompt_tokens: int = 0 total_tokens: int = 0 completion_tokens: Optional[int] = 0 class ChatCompletionResponse(BaseModel): model: str object: Literal["chat.completion", "chat.completion.chunk"] choices: List[Union[ChatCompletionResponseChoice, ChatCompletionResponseStreamChoice]] created: Optional[int] = Field(default_factory=lambda: int(time.time())) usage: Optional[UsageInfo] = None @app.get("/v1/models", response_model=ModelList) async def list_models(): """ An endpoint to list available models. It returns a list of model cards. This is useful for clients to query and understand what models are available for use. """ model_card = ModelCard(id="GLM-4v-9b") return ModelList(data=[model_card]) @app.post("/v1/chat/completions", response_model=ChatCompletionResponse) async def create_chat_completion(request: ChatCompletionRequest): global model, tokenizer if len(request.messages) < 1 or request.messages[-1].role == "assistant": raise HTTPException(status_code=400, detail="Invalid request") gen_params = dict( messages=request.messages, temperature=request.temperature, top_p=request.top_p, max_tokens=request.max_tokens or 1024, echo=False, stream=request.stream, repetition_penalty=request.repetition_penalty, ) if request.stream: generate = predict(request.model, gen_params) return EventSourceResponse(generate, media_type="text/event-stream") response = generate_glm4v(model, tokenizer, gen_params) usage = UsageInfo() message = ChatMessageResponse( role="assistant", content=response["text"], ) choice_data = ChatCompletionResponseChoice( index=0, message=message, ) task_usage = UsageInfo.model_validate(response["usage"]) for usage_key, usage_value in task_usage.model_dump().items(): setattr(usage, usage_key, getattr(usage, usage_key) + usage_value) return ChatCompletionResponse(model=request.model, choices=[choice_data], object="chat.completion", usage=usage) def predict(model_id: str, params: dict): global model, tokenizer choice_data = ChatCompletionResponseStreamChoice(index=0, delta=DeltaMessage(role="assistant"), finish_reason=None) chunk = ChatCompletionResponse(model=model_id, choices=[choice_data], object="chat.completion.chunk") yield "{}".format(chunk.model_dump_json(exclude_unset=True)) previous_text = "" for new_response in generate_stream_glm4v(model, tokenizer, params): decoded_unicode = new_response["text"] delta_text = decoded_unicode[len(previous_text) :] previous_text = decoded_unicode delta = DeltaMessage(content=delta_text, role="assistant") choice_data = ChatCompletionResponseStreamChoice(index=0, delta=delta) chunk = ChatCompletionResponse(model=model_id, choices=[choice_data], object="chat.completion.chunk") yield "{}".format(chunk.model_dump_json(exclude_unset=True)) choice_data = ChatCompletionResponseStreamChoice(index=0, delta=DeltaMessage()) chunk = ChatCompletionResponse(model=model_id, choices=[choice_data], object="chat.completion.chunk") yield "{}".format(chunk.model_dump_json(exclude_unset=True)) def generate_glm4v(model: AutoModel, tokenizer: AutoTokenizer, params: dict): """ Generates a response using the GLM-4v-9b model. It processes the chat history and image data, if any, and then invokes the model to generate a response. """ response = None for response in generate_stream_glm4v(model, tokenizer, params): pass return response def process_history_and_images( messages: List[ChatMessageInput], ) -> Tuple[Optional[str], Optional[List[Tuple[str, str]]], Optional[List[Image.Image]]]: """ Process history messages to extract text, identify the last user query, and convert base64 encoded image URLs to PIL images. Args: messages(List[ChatMessageInput]): List of ChatMessageInput objects. return: A tuple of three elements: - The last user query as a string. - Text history formatted as a list of tuples for the model. - List of PIL Image objects extracted from the messages. """ formatted_history = [] image_list = [] last_user_query = "" for i, message in enumerate(messages): role = message.role content = message.content if isinstance(content, list): # text text_content = " ".join(item.text for item in content if isinstance(item, TextContent)) else: text_content = content if isinstance(content, list): # image for item in content: if isinstance(item, ImageUrlContent): image_url = item.image_url.url if image_url.startswith("data:image/jpeg;base64,"): base64_encoded_image = image_url.split("data:image/jpeg;base64,")[1] image_data = base64.b64decode(base64_encoded_image) image = Image.open(BytesIO(image_data)).convert("RGB") else: response = requests.get(image_url, verify=False) image = Image.open(BytesIO(response.content)).convert("RGB") image_list.append(image) if role == "user": if i == len(messages) - 1: # 最后一条用户消息 last_user_query = text_content else: formatted_history.append((text_content, "")) elif role == "assistant": if formatted_history: if formatted_history[-1][1] != "": assert False, f"the last query is answered. answer again. {formatted_history[-1][0]}, {formatted_history[-1][1]}, {text_content}" formatted_history[-1] = (formatted_history[-1][0], text_content) else: assert False, "assistant reply before user" else: assert False, f"unrecognized role: {role}" return last_user_query, formatted_history, image_list @torch.inference_mode() def generate_stream_glm4v(model: AutoModel, tokenizer: AutoTokenizer, params: dict): uploaded = False messages = params["messages"] temperature = float(params.get("temperature", 1.0)) repetition_penalty = float(params.get("repetition_penalty", 1.0)) top_p = float(params.get("top_p", 1.0)) max_new_tokens = int(params.get("max_tokens", 256)) query, history, image_list = process_history_and_images(messages) inputs = [] for idx, (user_msg, model_msg) in enumerate(history): if idx == len(history) - 1 and not model_msg: inputs.append({"role": "user", "content": user_msg}) if image_list and not uploaded: inputs[-1].update({"image": image_list[0]}) uploaded = True break if user_msg: inputs.append({"role": "user", "content": user_msg}) if model_msg: inputs.append({"role": "assistant", "content": model_msg}) if len(image_list) >= 1: inputs.append({"role": "user", "content": query, "image": image_list[0]}) else: inputs.append({"role": "user", "content": query}) model_inputs = tokenizer.apply_chat_template( inputs, add_generation_prompt=True, tokenize=True, return_tensors="pt", return_dict=True ).to(next(model.parameters()).device) input_echo_len = len(model_inputs["input_ids"][0]) streamer = TextIteratorStreamer(tokenizer=tokenizer, timeout=60.0, skip_prompt=True, skip_special_tokens=True) gen_kwargs = { "repetition_penalty": repetition_penalty, "max_new_tokens": max_new_tokens, "do_sample": True if temperature > 1e-5 else False, "top_p": top_p if temperature > 1e-5 else 0, "top_k": 1, "streamer": streamer, "eos_token_id": [151329, 151336, 151338], } if temperature > 1e-5: gen_kwargs["temperature"] = temperature generated_text = "" def generate_text(): with torch.no_grad(): model.generate(**model_inputs, **gen_kwargs) generation_thread = threading.Thread(target=generate_text) generation_thread.start() total_len = input_echo_len for next_text in streamer: generated_text += next_text total_len = len(tokenizer.encode(generated_text)) yield { "text": generated_text, "usage": { "prompt_tokens": input_echo_len, "completion_tokens": total_len - input_echo_len, "total_tokens": total_len, }, } generation_thread.join() print("\033[91m--generated_text\033[0m", generated_text) yield { "text": generated_text, "usage": { "prompt_tokens": input_echo_len, "completion_tokens": total_len - input_echo_len, "total_tokens": total_len, }, } gc.collect() torch.cuda.empty_cache() if __name__ == "__main__": MODEL_PATH = sys.argv[1] model_dir = Path(MODEL_PATH).expanduser().resolve() if (model_dir / "adapter_config.json").exists(): import json with open(model_dir / "adapter_config.json", "r", encoding="utf-8") as file: config = json.load(file) model = AutoModel.from_pretrained( config.get("base_model_name_or_path"), device_map="auto", torch_dtype=TORCH_TYPE ) model = PeftModelForCausalLM.from_pretrained( model=model, model_id=model_dir, ) tokenizer = AutoTokenizer.from_pretrained(config.get("base_model_name_or_path"), encode_special_tokens=True) model.eval() else: tokenizer = AutoTokenizer.from_pretrained(MODEL_PATH, encode_special_tokens=True) model = AutoModel.from_pretrained( MODEL_PATH, torch_dtype=TORCH_TYPE, device_map="auto", ).eval() uvicorn.run(app, host="0.0.0.0", port=8000, workers=1) ================================================ FILE: inference/requirements.txt ================================================ torch>=2.6.0 torchvision>=0.21.0 transformers>=4.51.3 sentencepiece>=0.2.0 jinja2>=3.1.4 pydantic>=2.11.1 timm>=1.0.15 tiktoken>=0.9.0 numpy<2 accelerate>=1.6.0 sentence_transformers>=3.1.1 gradio>=5.23.3 openai>=1.70.0 einops>=0.8.0 pillow>=10.4.0 sse-starlette>=2.1.3 bitsandbytes>=0.44.1 # INT4 Loading, Not support for NPU peft>=0.15.0 # Using with finetune model swanlab>=0.5.5 # git+https://github.com/vllm-project/vllm.git For vLLM ================================================ FILE: inference/trans_batch_demo.py ================================================ """ Here is an example of using batch request GLM-4-0414 Models and glm-4-9b-chat-hf models with the transformers library., here you need to build the conversation format yourself and then call the batch function to make batch requests. Please note that in this demo, the memory consumption is significantly higher. """ from typing import Union from transformers import AutoModelForCausalLM, AutoTokenizer, LogitsProcessorList MODEL_PATH = "THUDM/GLM-4-9B-0414" tokenizer = AutoTokenizer.from_pretrained(MODEL_PATH) model = AutoModelForCausalLM.from_pretrained(MODEL_PATH, device_map="auto").eval() def process_model_outputs(inputs, outputs, tokenizer): responses = [] for input_ids, output_ids in zip(inputs.input_ids, outputs): response = tokenizer.decode(output_ids[len(input_ids) :], skip_special_tokens=True).strip() responses.append(response) return responses def batch( model, tokenizer, messages: Union[str, list[str]], max_input_tokens: int = 8192, max_new_tokens: int = 8192, num_beams: int = 1, do_sample: bool = True, top_p: float = 0.8, temperature: float = 0.8, logits_processor=None, ): if logits_processor is None: logits_processor = LogitsProcessorList() messages = [messages] if isinstance(messages, str) else messages batched_inputs = tokenizer( messages, return_tensors="pt", padding="max_length", truncation=True, max_length=max_input_tokens ).to(model.device) gen_kwargs = { "max_new_tokens": max_new_tokens, "num_beams": num_beams, "do_sample": do_sample, "top_p": top_p, "temperature": temperature, "logits_processor": logits_processor, "eos_token_id": model.config.eos_token_id, } batched_outputs = model.generate(**batched_inputs, **gen_kwargs) batched_response = process_model_outputs(batched_inputs, batched_outputs, tokenizer) return batched_response if __name__ == "__main__": batch_message = [ [ {"role": "user", "content": "我的爸爸和妈妈结婚为什么不能带我去"}, {"role": "assistant", "content": "因为他们结婚时你还没有出生"}, {"role": "user", "content": "我刚才的提问是"}, ], [{"role": "user", "content": "你好，你是谁"}], ] batch_inputs = [] max_input_tokens = 128 for i, messages in enumerate(batch_message): new_batch_input = tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)[12:] max_input_tokens = max(max_input_tokens, len(new_batch_input)) batch_inputs.append(new_batch_input) gen_kwargs = { "max_input_tokens": max_input_tokens, "max_new_tokens": 256, "do_sample": True, "top_p": 0.8, "temperature": 0.8, "num_beams": 1, } batch_responses = batch(model, tokenizer, batch_inputs, **gen_kwargs) for response in batch_responses: print("=" * 10) print(response) ================================================ FILE: inference/trans_cli_demo.py ================================================ """ This script creates a CLI demo with transformers backend for the glm-4-9b-chat model, allowing users to interact with the model through a command-line interface. Usage: - Run the script to start the CLI demo. - Interact with the model by typing questions and receiving responses. Note: The script includes a modification to handle markdown to plain text conversion, ensuring that the CLI interface displays formatted text correctly. If you use flash attention, you should install the flash-attn and add attn_implementation="flash_attention_2" in model loading. """ from threading import Thread import torch from transformers import ( AutoModelForCausalLM, AutoTokenizer, StoppingCriteria, StoppingCriteriaList, TextIteratorStreamer, ) MODEL_PATH = "THUDM/GLM-4-9B-0414" tokenizer = AutoTokenizer.from_pretrained(MODEL_PATH, trust_remote_code=True) model = AutoModelForCausalLM.from_pretrained( MODEL_PATH, torch_dtype=torch.bfloat16, device_map="auto", ).eval() class StopOnTokens(StoppingCriteria): def __call__(self, input_ids: torch.LongTensor, scores: torch.FloatTensor, **kwargs) -> bool: stop_ids = model.config.eos_token_id for stop_id in stop_ids: if input_ids[0][-1] == stop_id: return True return False if __name__ == "__main__": history = [] max_length = 8192 top_p = 0.8 temperature = 0.6 stop = StopOnTokens() print("Welcome to the GLM-4-9B CLI chat. Type your messages below.") while True: user_input = input("\nYou: ") if user_input.lower() in ["exit", "quit"]: break history.append([user_input, ""]) messages = [] for idx, (user_msg, model_msg) in enumerate(history): if idx == len(history) - 1 and not model_msg: messages.append({"role": "user", "content": user_msg}) break if user_msg: messages.append({"role": "user", "content": user_msg}) if model_msg: messages.append({"role": "assistant", "content": model_msg}) model_inputs = tokenizer.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt" ).to(model.device) streamer = TextIteratorStreamer(tokenizer=tokenizer, timeout=60, skip_prompt=True, skip_special_tokens=True) generate_kwargs = { "input_ids": model_inputs["input_ids"], "attention_mask": model_inputs["attention_mask"], "streamer": streamer, "max_new_tokens": max_length, "do_sample": True, "top_p": top_p, "temperature": temperature, "stopping_criteria": StoppingCriteriaList([stop]), "repetition_penalty": 1.2, "eos_token_id": model.config.eos_token_id, } t = Thread(target=model.generate, kwargs=generate_kwargs) t.start() print("GLM-4:", end="", flush=True) for new_token in streamer: if new_token: print(new_token, end="", flush=True) history[-1][1] += new_token history[-1][1] = history[-1][1].strip() ================================================ FILE: inference/trans_cli_vision_demo.py ================================================ """ This script creates a CLI demo with transformers backend for the glm-4v-9b model, allowing users to interact with the model through a command-line interface. Usage: - Run the script to start the CLI demo. - Interact with the model by typing questions and receiving responses. Note: The script includes a modification to handle markdown to plain text conversion, ensuring that the CLI interface displays formatted text correctly. """ from threading import Thread import torch from PIL import Image from transformers import ( AutoModel, AutoTokenizer, StoppingCriteria, StoppingCriteriaList, TextIteratorStreamer, ) MODEL_PATH = "THUDM/glm-4v-9b" tokenizer = AutoTokenizer.from_pretrained(MODEL_PATH, trust_remote_code=True, encode_special_tokens=True) ## For BF16 inference model = AutoModel.from_pretrained( MODEL_PATH, trust_remote_code=True, torch_dtype=torch.bfloat16, device_map="auto", ).eval() ## For INT4 inference # model = AutoModel.from_pretrained( # MODEL_PATH, # trust_remote_code=True, # quantization_config=BitsAndBytesConfig(load_in_4bit=True), # torch_dtype=torch.bfloat16, # low_cpu_mem_usage=True # ).eval() class StopOnTokens(StoppingCriteria): def __call__(self, input_ids: torch.LongTensor, scores: torch.FloatTensor, **kwargs) -> bool: stop_ids = model.config.eos_token_id for stop_id in stop_ids: if input_ids[0][-1] == stop_id: return True return False if __name__ == "__main__": history = [] max_length = 1024 top_p = 0.8 temperature = 0.6 stop = StopOnTokens() uploaded = False image = None print("Welcome to the GLM-4-9B CLI chat. Type your messages below.") image_path = input("Image Path:") try: image = Image.open(image_path).convert("RGB") except: print("Invalid image path. Continuing with text conversation.") while True: user_input = input("\nYou: ") if user_input.lower() in ["exit", "quit"]: break history.append([user_input, ""]) messages = [] for idx, (user_msg, model_msg) in enumerate(history): if idx == len(history) - 1 and not model_msg: messages.append({"role": "user", "content": user_msg}) if image and not uploaded: messages[-1].update({"image": image}) uploaded = True break if user_msg: messages.append({"role": "user", "content": user_msg}) if model_msg: messages.append({"role": "assistant", "content": model_msg}) model_inputs = tokenizer.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_tensors="pt", return_dict=True ).to(next(model.parameters()).device) streamer = TextIteratorStreamer(tokenizer=tokenizer, timeout=60, skip_prompt=True, skip_special_tokens=True) generate_kwargs = { **model_inputs, "streamer": streamer, "max_new_tokens": max_length, "do_sample": True, "top_p": top_p, "temperature": temperature, "stopping_criteria": StoppingCriteriaList([stop]), "repetition_penalty": 1.2, "eos_token_id": [151329, 151336, 151338], } t = Thread(target=model.generate, kwargs=generate_kwargs) t.start() print("GLM-4V:", end="", flush=True) for new_token in streamer: if new_token: print(new_token, end="", flush=True) history[-1][1] += new_token history[-1][1] = history[-1][1].strip() ================================================ FILE: inference/trans_stress_test.py ================================================ import argparse import datetime import time from threading import Thread import torch from transformers import AutoModelForCausalLM, AutoTokenizer, TextIteratorStreamer MODEL_PATH = "THUDM/GLM-4-9B-0414" def stress_test(run_name, input_token_len, n, output_token_len, swanlab_api_key): tokenizer = AutoTokenizer.from_pretrained(MODEL_PATH, paddsing_side="left") model = AutoModelForCausalLM.from_pretrained(MODEL_PATH, torch_dtype=torch.bfloat16, device_map="auto").eval() device = model.device # Use INT4 weight infer # model = AutoModelForCausalLM.from_pretrained( # MODEL_PATH, # trust_remote_code=True, # quantization_config=BitsAndBytesConfig(load_in_4bit=True), # low_cpu_mem_usage=True, # ).eval() # Enable SwanLab if swanlab_api_key available if swanlab_api_key: import swanlab print("Enable swanlab logging...") if not args.swanlab_api_key == "local": swanlab.login(api_key=args.swanlab_api_key) current_time = datetime.datetime.now().strftime("%Y%m%d_%H%M%S") run_name = run_name if run_name else f'{MODEL_PATH.split("/")[-1]}_{current_time}' config = { "model": model.config.to_dict(), "generation_config": model.generation_config.to_dict(), "input_token_len": input_token_len, "n": n, "output_token_len": output_token_len, "device": str(model.device), } swanlab.init( project="glm-stress-test", name=run_name, config=config, mode="local" if args.swanlab_api_key == "local" else None, ) times = [] decode_times = [] print("Warming up...") vocab_size = tokenizer.vocab_size warmup_token_len = 20 random_token_ids = torch.randint(3, vocab_size - 200, (warmup_token_len - 5,), dtype=torch.long) start_tokens = [151331, 151333, 151336, 198] end_tokens = [151337] input_ids = ( torch.tensor(start_tokens + random_token_ids.tolist() + end_tokens, dtype=torch.long).unsqueeze(0).to(device) ) attention_mask = torch.ones_like(input_ids, dtype=torch.bfloat16).to(device) position_ids = torch.arange(len(input_ids[0]), dtype=torch.bfloat16).unsqueeze(0).to(device) warmup_inputs = {"input_ids": input_ids, "attention_mask": attention_mask, "position_ids": position_ids} with torch.no_grad(): _ = model.generate( input_ids=warmup_inputs["input_ids"], attention_mask=warmup_inputs["attention_mask"], max_new_tokens=512, do_sample=False, repetition_penalty=0.1, eos_token_id=[151329, 151336, 151338], ) print("Warming up complete. Starting stress test...") for i in range(n): random_token_ids = torch.randint(3, vocab_size - 200, (input_token_len - 5,), dtype=torch.long) input_ids = ( torch.tensor(start_tokens + random_token_ids.tolist() + end_tokens, dtype=torch.long) .unsqueeze(0) .to(device) ) attention_mask = torch.ones_like(input_ids, dtype=torch.bfloat16).to(device) position_ids = torch.arange(len(input_ids[0]), dtype=torch.bfloat16).unsqueeze(0).to(device) test_inputs = {"input_ids": input_ids, "attention_mask": attention_mask, "position_ids": position_ids} streamer = TextIteratorStreamer(tokenizer=tokenizer, timeout=36000, skip_prompt=True, skip_special_tokens=True) generate_kwargs = { "input_ids": test_inputs["input_ids"], "attention_mask": test_inputs["attention_mask"], "max_new_tokens": output_token_len, "do_sample": False, "repetition_penalty": 0.1, # For generate more tokens for test. "eos_token_id": [151329, 151336, 151338], "streamer": streamer, } start_time = time.time() t = Thread(target=model.generate, kwargs=generate_kwargs) t.start() first_token_time = None all_token_times = [] for token in streamer: current_time = time.time() if first_token_time is None: first_token_time = current_time times.append(first_token_time - start_time) all_token_times.append(current_time) t.join() end_time = time.time() avg_decode_time_per_token = len(all_token_times) / (end_time - first_token_time) if all_token_times else 0 decode_times.append(avg_decode_time_per_token) print( f"Iteration {i + 1}/{n} - Prefilling Time: {times[-1]:.4f} seconds - Average Decode Time: {avg_decode_time_per_token:.4f} tokens/second" ) if swanlab_api_key: swanlab.log( { "Iteration": i + 1, "Iteration/Prefilling Time (seconds)": times[-1], "Iteration/Decode Time (tokens per second)": avg_decode_time_per_token, "Iteration/Input token Len": len(test_inputs["input_ids"][0]), "Iteration/Output token Len": len(all_token_times), "Average First Token Time (seconds)": sum(times) / (i + 1), "Average Decode Time (tokens per second)": sum(decode_times) / (i + 1), } ) torch.cuda.empty_cache() avg_first_token_time = sum(times) / n avg_decode_time = sum(decode_times) / n print(f"\nAverage First Token Time over {n} iterations: {avg_first_token_time:.4f} seconds") print(f"Average Decode Time per Token over {n} iterations: {avg_decode_time:.4f} tokens/second") return times, avg_first_token_time, decode_times, avg_decode_time if __name__ == "__main__": parser = argparse.ArgumentParser(description="Stress test for model inference") parser.add_argument("--run_name", type=str, default=None, help="Number of tokens for each test") parser.add_argument("--input_token_len", type=int, default=100000, help="Number of tokens for each test") parser.add_argument("--output_token_len", type=int, default=128, help="Number of output tokens for each test") parser.add_argument("--n", type=int, default=3, help="Number of iterations for the stress test") parser.add_argument("--swanlab_api_key", type=str, default=None, help="Enable swanlab logging if API key provided") args = parser.parse_args() stress_test(args.run_name, args.input_token_len, args.n, args.output_token_len, args.swanlab_api_key) ================================================ FILE: inference/trans_web_demo.py ================================================ from threading import Thread import gradio as gr from transformers import AutoModelForCausalLM, AutoTokenizer, TextIteratorStreamer MODEL_PATH = "THUDM/GLM-4-9B-0414" tokenizer = AutoTokenizer.from_pretrained(MODEL_PATH) model = AutoModelForCausalLM.from_pretrained(MODEL_PATH, device_map="auto") def preprocess_messages(history, system_prompt): messages = [] if system_prompt: messages.append({"role": "system", "content": system_prompt}) for idx, (user_msg, model_msg) in enumerate(history): if idx == len(history) - 1 and not model_msg: messages.append({"role": "user", "content": user_msg}) break if user_msg: messages.append({"role": "user", "content": user_msg}) if model_msg: messages.append({"role": "assistant", "content": model_msg}) return messages def predict(history, system_prompt, max_length, top_p, top_k, temperature): messages = preprocess_messages(history, system_prompt) model_inputs = tokenizer.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_tensors="pt", return_dict=True ).to(model.device) streamer = TextIteratorStreamer(tokenizer, timeout=60, skip_prompt=True, skip_special_tokens=True) generate_kwargs = { "input_ids": model_inputs["input_ids"], "attention_mask": model_inputs["attention_mask"], "streamer": streamer, "max_new_tokens": max_length, "do_sample": True, "top_p": top_p, "top_k": top_k, "temperature": temperature, "repetition_penalty": 1.2, } generate_kwargs["eos_token_id"] = tokenizer.encode("<|user|>") t = Thread(target=model.generate, kwargs=generate_kwargs) t.start() for new_token in streamer: if new_token: history[-1][1] += new_token yield history def main(): with gr.Blocks() as demo: gr.HTML("""

GLM-4-0414 Gradio Demo

""") with gr.Row(): with gr.Column(scale=3): system_prompt = gr.Textbox( show_label=True, placeholder="Enter system prompt here...", label="System Prompt", lines=2 ) with gr.Row(): with gr.Column(scale=3): chatbot = gr.Chatbot() with gr.Row(): with gr.Column(scale=2): user_input = gr.Textbox(show_label=True, placeholder="Input...", label="User Input") submitBtn = gr.Button("Submit") emptyBtn = gr.Button("Clear History") with gr.Column(scale=1): max_length = gr.Slider(0, 8192, value=4096, step=1.0, label="Maximum length", interactive=True) top_p = gr.Slider(0, 1, value=0.8, step=0.01, label="Top P", interactive=True) top_k = gr.Slider(0, 100, value=50, step=1, label="Top K", interactive=True) temperature = gr.Slider(0.01, 1, value=0.6, step=0.01, label="Temperature", interactive=True) def user(query, history): return "", history + [[query, ""]] def clear_history(): return None submitBtn.click(user, [user_input, chatbot], [user_input, chatbot], queue=False).then( predict, [chatbot, system_prompt, max_length, top_p, top_k, temperature], chatbot ) emptyBtn.click(clear_history, None, [chatbot], queue=False) demo.queue() demo.launch() if __name__ == "__main__": main() ================================================ FILE: inference/trans_web_vision_demo.py ================================================ """ This script creates a Gradio demo with a Transformers backend for the glm-4v-9b model, allowing users to interact with the model through a Gradio web UI. Usage: - Run the script to start the Gradio server. - Interact with the model via the web UI. Requirements: - Gradio package - Type `pip install gradio==4.44.1` to install Gradio. """ import os from io import BytesIO from threading import Thread import gradio as gr import requests import torch from PIL import Image from transformers import AutoModel, AutoTokenizer, StoppingCriteria, StoppingCriteriaList, TextIteratorStreamer MODEL_PATH = os.environ.get("MODEL_PATH", "THUDM/glm-4v-9b") tokenizer = AutoTokenizer.from_pretrained(MODEL_PATH, trust_remote_code=True, encode_special_tokens=True) model = AutoModel.from_pretrained( MODEL_PATH, trust_remote_code=True, device_map="auto", torch_dtype=torch.bfloat16 ).eval() class StopOnTokens(StoppingCriteria): def __call__(self, input_ids: torch.LongTensor, scores: torch.FloatTensor, **kwargs) -> bool: stop_ids = model.config.eos_token_id for stop_id in stop_ids: if input_ids[0][-1] == stop_id: return True return False def get_image(image_path=None, image_url=None): if image_path: return Image.open(image_path).convert("RGB") elif image_url: response = requests.get(image_url) return Image.open(BytesIO(response.content)).convert("RGB") return None def chatbot(image_path=None, image_url=None, assistant_prompt=""): image = get_image(image_path, image_url) messages = [{"role": "assistant", "content": assistant_prompt}, {"role": "user", "content": "", "image": image}] model_inputs = tokenizer.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_tensors="pt", return_dict=True ).to(next(model.parameters()).device) streamer = TextIteratorStreamer(tokenizer=tokenizer, timeout=60, skip_prompt=True, skip_special_tokens=True) generate_kwargs = { **model_inputs, "streamer": streamer, "max_new_tokens": 1024, "do_sample": True, "top_p": 0.8, "temperature": 0.6, "stopping_criteria": StoppingCriteriaList([StopOnTokens()]), "repetition_penalty": 1.2, "eos_token_id": [151329, 151336, 151338], } t = Thread(target=model.generate, kwargs=generate_kwargs) t.start() response = "" for new_token in streamer: if new_token: response += new_token return image, response.strip() with gr.Blocks() as demo: demo.title = "GLM-4V-9B Image Recognition Demo" demo.description = """ This demo uses the GLM-4V-9B model to got image infomation. """ with gr.Row(): with gr.Column(): image_path_input = gr.File(label="Upload Image (High-Priority)", type="filepath") image_url_input = gr.Textbox(label="Image URL (Low-Priority)") assistant_prompt_input = gr.Textbox(label="Assistant Prompt (You Can Change It)", value="这是什么？") submit_button = gr.Button("Submit") with gr.Column(): chatbot_output = gr.Textbox(label="GLM-4V-9B Model Response") image_output = gr.Image(label="Image Preview") submit_button.click( chatbot, inputs=[image_path_input, image_url_input, assistant_prompt_input], outputs=[image_output, chatbot_output], ) demo.launch(server_name="127.0.0.1", server_port=8911, inbrowser=True, share=False) ================================================ FILE: inference/vllm_cli_demo.py ================================================ """ This script creates a CLI demo with vllm backand for the glm-4-9b model, allowing users to interact with the model through a command-line interface. Usage: - Run the script to start the CLI demo. - Interact with the model by typing questions and receiving responses. Note: The script includes a modification to handle markdown to plain text conversion, ensuring that the CLI interface displays formatted text correctly. """ import asyncio import time from typing import Dict, List from transformers import AutoTokenizer from vllm import AsyncEngineArgs, AsyncLLMEngine, SamplingParams from vllm.lora.request import LoRARequest MODEL_PATH = "THUDM/GLM-4-9B-0414" LORA_PATH = "" def load_model_and_tokenizer(model_dir: str, enable_lora: bool): tokenizer = AutoTokenizer.from_pretrained(model_dir) engine_args = AsyncEngineArgs( model=model_dir, tokenizer=model_dir, enable_lora=enable_lora, tensor_parallel_size=1, dtype="bfloat16", gpu_memory_utilization=0.9, disable_log_requests=True, ) engine = AsyncLLMEngine.from_engine_args(engine_args) return engine, tokenizer enable_lora = False if LORA_PATH: enable_lora = True engine, tokenizer = load_model_and_tokenizer(MODEL_PATH, enable_lora) async def vllm_gen( lora_path: str, enable_lora: bool, messages: List[Dict[str, str]], top_p: float, temperature: float, max_dec_len: int, ): inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=False) params_dict = { "n": 1, "best_of": 1, "presence_penalty": 1.0, "frequency_penalty": 0.0, "temperature": temperature, "top_p": top_p, "max_tokens": max_dec_len, "skip_special_tokens": True, } sampling_params = SamplingParams(**params_dict) if enable_lora: async for output in engine.generate( prompt=inputs, sampling_params=sampling_params, request_id=f"{time.time()}", lora_request=LoRARequest("glm-4-lora", 1, lora_path=lora_path), ): yield output.outputs[0].text else: async for output in engine.generate( prompt=inputs, sampling_params=sampling_params, request_id=f"{time.time()}" ): yield output.outputs[0].text async def chat(): history = [] max_length = 8192 top_p = 0.8 temperature = 0.6 print("Welcome to the GLM-4-9B CLI chat. Type your messages below.") while True: user_input = input("\nYou: ") if user_input.lower() in ["exit", "quit"]: break history.append([user_input, ""]) messages = [] for idx, (user_msg, model_msg) in enumerate(history): if idx == len(history) - 1 and not model_msg: messages.append({"role": "user", "content": user_msg}) break if user_msg: messages.append({"role": "user", "content": user_msg}) if model_msg: messages.append({"role": "assistant", "content": model_msg}) print("\nGLM-4: ", end="") current_length = 0 output = "" async for output in vllm_gen(LORA_PATH, enable_lora, messages, top_p, temperature, max_length): print(output[current_length:], end="", flush=True) current_length = len(output) history[-1][1] = output if __name__ == "__main__": asyncio.run(chat()) ================================================ FILE: inference/vllm_cli_vision_demo.py ================================================ """ This script creates a CLI demo with vllm backand for the glm-4v-9b model, allowing users to interact with the model through a command-line interface. Usage: - Run the script to start the CLI demo. - Interact with the model by typing questions and receiving responses. Note: The script includes a modification to handle markdown to plain text conversion, ensuring that the CLI interface displays formatted text correctly. """ import asyncio import time from typing import Dict, List from PIL import Image from vllm import AsyncEngineArgs, AsyncLLMEngine, SamplingParams MODEL_PATH = "THUDM/glm-4v-9b" def load_model_and_tokenizer(model_dir: str): engine_args = AsyncEngineArgs( model=model_dir, tokenizer=model_dir, tensor_parallel_size=1, dtype="bfloat16", gpu_memory_utilization=0.9, enforce_eager=True, disable_log_requests=True, ) engine = AsyncLLMEngine.from_engine_args(engine_args) return engine engine = load_model_and_tokenizer(MODEL_PATH) async def vllm_gen(messages: List[Dict[str, str]], top_p: float, temperature: float, max_dec_len: int): inputs = messages[-1] params_dict = { "n": 1, "best_of": 1, "presence_penalty": 1.0, "frequency_penalty": 0.0, "temperature": temperature, "top_p": top_p, "max_tokens": max_dec_len, "skip_special_tokens": True, } sampling_params = SamplingParams(**params_dict) async for output in engine.generate(prompt=inputs, sampling_params=sampling_params, request_id=f"{time.time()}"): yield output.outputs[0].text async def chat(): history = [] max_length = 8192 top_p = 0.8 temperature = 0.6 image = None print("Welcome to the GLM-4v-9B CLI chat. Type your messages below.") image_path = input("Image Path:") try: image = Image.open(image_path).convert("RGB") except: print("Invalid image path. Continuing with text conversation.") while True: user_input = input("\nYou: ") if user_input.lower() in ["exit", "quit"]: break history.append([user_input, ""]) messages = [] for idx, (user_msg, model_msg) in enumerate(history): if idx == len(history) - 1 and not model_msg: messages.append( { "prompt": user_msg, "multi_modal_data": {"image": image}, } ) break if user_msg: messages.append({"role": "user", "prompt": user_msg}) if model_msg: messages.append({"role": "assistant", "prompt": model_msg}) print("\nGLM-4v: ", end="") current_length = 0 output = "" async for output in vllm_gen(messages, top_p, temperature, max_length): print(output[current_length:], end="", flush=True) current_length = len(output) history[-1][1] = output if __name__ == "__main__": asyncio.run(chat()) ================================================ FILE: pyproject.toml ================================================ [tool.ruff] line-length = 119 [tool.ruff.lint] # Never enforce `E501` (line length violations). ignore = ["C901", "E501", "E741", "F402", "F823"] select = ["C", "E", "F", "I", "W"] # Ignore import violations in all `__init__.py` files. [tool.ruff.lint.per-file-ignores] "__init__.py" = ["E402", "F401", "F403", "F811"] [tool.ruff.lint.isort] lines-after-imports = 2 [tool.ruff.format] # Like Black, use double quotes for strings. quote-style = "double" # Like Black, indent with spaces, rather than tabs. indent-style = "space" # Like Black, respect magic trailing commas. skip-magic-trailing-comma = false # Like Black, automatically detect the appropriate line ending. line-ending = "auto" ================================================ FILE: resources/WECHAT.md ================================================

扫码加入「GLM-4交流群」

Scan the QR code to follow to join the "ChatGLM Discussion Group"