Showing preview only (426K chars total). Download the full file or copy to clipboard to get everything.
Repository: THUDM/ChatGLM3
Branch: main
Commit: 732294f42e26
Files: 68
Total size: 376.3 KB
Directory structure:
gitextract_gh6ajpza/
├── .github/
│ ├── ISSUE_TEMPLATE/
│ │ ├── bug_report.yaml
│ │ └── feature-request.yaml
│ └── PULL_REQUEST_TEMPLATE/
│ └── pr_template.md
├── .gitignore
├── DEPLOYMENT.md
├── DEPLOYMENT_en.md
├── Intel_device_demo/
│ ├── README.md
│ ├── ipex_llm_cpu_demo/
│ │ ├── api_server.py
│ │ ├── chatglm3_infer.py
│ │ ├── chatglm3_web_demo.py
│ │ ├── generate.py
│ │ ├── openai_api_request.py
│ │ └── utils.py
│ └── openvino_demo/
│ ├── README.md
│ └── openvino_cli_demo.py
├── LICENSE
├── MODEL_LICENSE
├── PROMPT.md
├── PROMPT_en.md
├── README.md
├── README_en.md
├── basic_demo/
│ ├── cli_batch_request_demo.py
│ ├── cli_demo.py
│ ├── cli_demo_bad_word_ids.py
│ ├── web_demo_gradio.py
│ └── web_demo_streamlit.py
├── composite_demo/
│ ├── .streamlit/
│ │ └── config.toml
│ ├── README.md
│ ├── README_en.md
│ ├── client.py
│ ├── conversation.py
│ ├── demo_chat.py
│ ├── demo_ci.py
│ ├── demo_tool.py
│ ├── main.py
│ ├── requirements.txt
│ └── tool_registry.py
├── finetune_demo/
│ ├── README.md
│ ├── README_en.md
│ ├── configs/
│ │ ├── ds_zero_2.json
│ │ ├── ds_zero_3.json
│ │ ├── lora.yaml
│ │ ├── ptuning_v2.yaml
│ │ └── sft.yaml
│ ├── finetune_hf.py
│ ├── inference_hf.py
│ ├── lora_finetune.ipynb
│ └── requirements.txt
├── langchain_demo/
│ ├── ChatGLM3.py
│ ├── main.py
│ └── tools/
│ ├── Calculator.py
│ ├── DistanceConversion.py
│ └── Weather.py
├── openai_api_demo/
│ ├── api_server.py
│ ├── docker-compose.yml
│ ├── langchain_openai_api.py
│ ├── openai_api_request.py
│ └── utils.py
├── requirements.txt
├── resources/
│ └── WECHAT.md
├── tensorrt_llm_demo/
│ ├── README.md
│ └── tensorrt_llm_cli_demo.py
├── tools_using_demo/
│ ├── README.md
│ ├── README_en.md
│ ├── cli_demo_tool.py
│ ├── openai_api_demo.py
│ └── tool_register.py
└── update_requirements.sh
================================================
FILE CONTENTS
================================================
================================================
FILE: .github/ISSUE_TEMPLATE/bug_report.yaml
================================================
name: "\U0001F41B Bug Report"
description: Submit a bug report to help us improve ChatGLM3 / 提交一个 Bug 问题报告来帮助我们改进 ChatGLM3
body:
- type: textarea
id: system-info
attributes:
label: System Info / 系統信息
description: Your operating environment / 您的运行环境信息
placeholder: Includes Cuda version, Transformers version, Python version, operating system, hardware information (if you suspect a hardware problem)... / 包括Cuda版本，Transformers版本，Python版本，操作系统，硬件信息(如果您怀疑是硬件方面的问题)...
validations:
required: true
- type: textarea
id: who-can-help
attributes:
label: Who can help? / 谁可以帮助到您？
description: |
Your issue will be replied to more quickly if you can figure out the right person to tag with @
All issues are read by one of the maintainers, so if you don't know who to tag, just leave this blank and our maintainer will ping the right person.
Please tag fewer than 3 people.
如果您能找到合适的标签 @，您的问题会更快得到回复。
所有问题都会由我们的维护者阅读，如果您不知道该标记谁，只需留空，我们的维护人员会找到合适的开发组成员来解决问题。
标记的人数应该不超过 3 个人。
Related demo leader / 相关demo负责人 :
- finetune_demo: @Btlmd
- langchain_demo: @yincf
- composite_demo: @abmfy
If it's not a bug in these three subsections, you may not specify the helper. Our maintainer will find the right person in the development group to solve the problem.
如果不是这三个子版块的bug，您可以不指明帮助者，我们的维护人员会找到合适的开发组成员来解决问题。
placeholder: "@Username ..."
- type: checkboxes
id: information-scripts-examples
attributes:
label: Information / 问题信息
description: 'The problem arises when using: / 问题出现在'
options:
- label: "The official example scripts / 官方的示例脚本"
- label: "My own modified scripts / 我自己修改的脚本和任务"
- type: textarea
id: reproduction
validations:
required: true
attributes:
label: Reproduction / 复现过程
description: |
Please provide a code example that reproduces the problem you encountered, preferably with a minimal reproduction unit.
If you have code snippets, error messages, stack traces, please provide them here as well.
Please format your code correctly using code tags. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
Do not use screenshots, as they are difficult to read and (more importantly) do not allow others to copy and paste your code.
请提供能重现您遇到的问题的代码示例,最好是最小复现单元。
如果您有代码片段、错误信息、堆栈跟踪，也请在此提供。
请使用代码标签正确格式化您的代码。请参见 https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
请勿使用截图，因为截图难以阅读，而且（更重要的是）不允许他人复制粘贴您的代码。
placeholder: |
Steps to reproduce the behavior/复现Bug的步骤:
1.
2.
3.
- type: textarea
id: expected-behavior
validations:
required: true
attributes:
label: Expected behavior / 期待表现
description: "A clear and concise description of what you would expect to happen. /简单描述您期望发生的事情。"
================================================
FILE: .github/ISSUE_TEMPLATE/feature-request.yaml
================================================
name: "\U0001F680 Feature request"
description: Submit a request for a new ChatGLM3 feature / 提交一个新的 ChatGLM3 的功能建议
labels: [ "feature" ]
body:
- type: textarea
id: feature-request
validations:
required: true
attributes:
label: Feature request / 功能建议
description: |
A brief description of the functional proposal. Links to corresponding papers and code are desirable.
对功能建议的简述。最好提供对应的论文和代码链接
- type: textarea
id: motivation
validations:
required: true
attributes:
label: Motivation / 动机
description: |
Your motivation for making the suggestion. If that motivation is related to another GitHub issue, link to it here.
您提出建议的动机。如果该动机与另一个 GitHub 问题有关，请在此处提供对应的链接。
- type: textarea
id: contribution
validations:
required: true
attributes:
label: Your contribution / 您的贡献
description: |
Your PR link or any other link you can help with.
您的PR链接或者其他您能提供帮助的链接。
================================================
FILE: .github/PULL_REQUEST_TEMPLATE/pr_template.md
================================================
# Raise valuable PR / 提出有价值的PR
## Caution/ 注意事项:
Users should keep the following points in mind when submitting PRs:
1. The proposed PR should be about this project.
2. the proposed PR should be relevant, if there are multiple ideas and optimizations, they should be assigned to different PRs.
用户在提交PR时候应该注意以下几点:
1. 提出的PR应该是关于本项目的。
2. 提出的PR应该具有针对性，如果具有多个不同的想法和优化方案，应该分配到不同的PR中。
## 不应该提出的PR / PRs that should not be proposed
If a developer proposes a PR about any of the following, it may be closed or Rejected.
1. those that don't describe improvement options.
2. multiple issues of different types combined in one PR.
3. The proposed PR is highly duplicative of already existing PRs.
如果开发者提出关于以下方面的PR，则可能会被直接关闭或拒绝通过。
1. 没有说明改进方案的。
2. 多个不同类型的问题合并在一个PR中的。
3. 提出的PR与已经存在的PR高度重复的。
# 检查您的PR
- [ ] Have you read the Contributor Guidelines, Pull Request section? / 您是否阅读了贡献者指南、Pull Request 部分？
- [ ] Has this been discussed/approved via a Github issue or forum? If so, add a link. / 是否通过 Github 问题或论坛讨论/批准过？如果是，请添加链接。
- [ ] Did you make sure you updated the documentation with your changes? Here are the Documentation Guidelines, and here are the Documentation Formatting Tips. /您是否确保根据您的更改更新了文档？这里是文档指南，这里是文档格式化技巧。
- [ ] Did you write new required tests? / 您是否编写了新的必要测试？
- [ ] Are your PRs for only one issue / 您的PR是否仅针对一个问题
================================================
FILE: .gitignore
================================================
__pycache__
# finetune_demo: generated & downloaded files
finetune_demo/output
finetune_demo/data
finetune_demo/formatted_data
ToolAlpaca/
AdvertiseGen/
*.gz
*.idea
.DS_Store
================================================
FILE: DEPLOYMENT.md
================================================
## 低成本部署
### 模型量化
默认情况下，模型以 FP16 精度加载，运行上述代码需要大概 13GB 显存。如果你的 GPU 显存有限，可以尝试以量化方式加载模型，使用方法如下：
```python
model = AutoModel.from_pretrained("THUDM/chatglm3-6b",trust_remote_code=True).quantize(4).cuda()
```
模型量化会带来一定的性能损失，经过测试，ChatGLM3-6B 在 4-bit 量化下仍然能够进行自然流畅的生成。
### CPU 部署
如果你没有 GPU 硬件的话，也可以在 CPU 上进行推理，但是推理速度会更慢。使用方法如下（需要大概 32GB 内存）
```python
model = AutoModel.from_pretrained("THUDM/chatglm3-6b", trust_remote_code=True).float()
```
### Mac 部署
对于搭载了 Apple Silicon 或者 AMD GPU 的 Mac，可以使用 MPS 后端来在 GPU 上运行 ChatGLM3-6B。需要参考 Apple 的 [官方说明](https://developer.apple.com/metal/pytorch) 安装 PyTorch-Nightly（正确的版本号应该是2.x.x.dev2023xxxx，而不是 2.x.x）。
目前在 MacOS 上只支持[从本地加载模型](README.md#从本地加载模型)。将代码中的模型加载改为从本地加载，并使用 mps 后端：
```python
model = AutoModel.from_pretrained("your local path", trust_remote_code=True).to('mps')
```
加载半精度的 ChatGLM3-6B 模型需要大概 13GB 内存。内存较小的机器（比如 16GB 内存的 MacBook Pro），在空余内存不足的情况下会使用硬盘上的虚拟内存，导致推理速度严重变慢。
### 多卡部署
如果你有多张 GPU，但是每张 GPU 的显存大小都不足以容纳完整的模型，那么可以将模型切分在多张GPU上。首先安装 accelerate: `pip install accelerate`，然后通过如下方法加载模型：
```python
from utils import load_model_on_gpus
model = load_model_on_gpus("THUDM/chatglm3-6b", num_gpus=2)
```
即可将模型部署到两张 GPU 上进行推理。你可以将 `num_gpus` 改为你希望使用的 GPU 数。默认是均匀切分的，你也可以传入 `device_map` 参数来自己指定。
================================================
FILE: DEPLOYMENT_en.md
================================================
## Low-Cost Deployment
### Model Quantization
By default, the model is loaded with FP16 precision, running the above code requires about 13GB of VRAM. If your GPU's VRAM is limited, you can try loading the model quantitatively, as follows:
```python
model = AutoModel.from_pretrained("THUDM/chatglm3-6b",trust_remote_code=True).quantize(4).cuda()
```
Model quantization will bring some performance loss. Through testing, ChatGLM3-6B can still perform natural and smooth generation under 4-bit quantization.
### CPU Deployment
If you don't have GPU hardware, you can also run inference on the CPU, but the inference speed will be slower. The usage is as follows (requires about 32GB of memory):
```python
model = AutoModel.from_pretrained("THUDM/chatglm3-6b", trust_remote_code=True).float()
```
### Mac Deployment
For Macs equipped with Apple Silicon or AMD GPUs, the MPS backend can be used to run ChatGLM3-6B on the GPU. Refer to Apple's [official instructions](https://developer.apple.com/metal/pytorch) to install PyTorch-Nightly (the correct version number should be 2.x.x.dev2023xxxx, not 2.x.x).
Currently, only [loading the model locally](README_en.md#load-model-locally) is supported on MacOS. Change the model loading in the code to load locally and use the MPS backend:
```python
model = AutoModel.from_pretrained("your local path", trust_remote_code=True).to('mps')
```
Loading the half-precision ChatGLM3-6B model requires about 13GB of memory. Machines with smaller memory (such as a 16GB memory MacBook Pro) will use virtual memory on the hard disk when there is insufficient free memory, resulting in a significant slowdown in inference speed.
### Multi-GPU Deployment
If you have multiple GPUs, but each GPU's VRAM size is not enough to accommodate the complete model, then the model can be split across multiple GPUs. First, install accelerate: `pip install accelerate`, and then load the model through the following methods:
```python
from utils import load_model_on_gpus
model = load_model_on_gpus("THUDM/chatglm3-6b", num_gpus=2)
```
This allows the model to be deployed on two GPUs for inference. You can change `num_gpus` to the number of GPUs you want to use. It is evenly split by default, but you can also pass the `device_map` parameter to specify it yourself.
================================================
FILE: Intel_device_demo/README.md
================================================
# Intel Device Demo
本文件夹主要辅助开发者 在 Intel 设备上加速部署 ChatGLM3-6B 模型。
## 1. 硬件准备
本文件夹中的设备支持列表包括：
- Intel CPU 系列, 包含个人CPU 和 服务器 / 工作站 CPU
- Intel Arc 独立显卡系列，包括 Arc A770 等显卡。
- Intel CPU 核显系列
- 其他理论支持 OpenVINO 加速的Intel 工具套件。
## 2. 文件目录
- IPEX_llm_xxx_demo: IPEX-LLM 是一个为Intel XPU(Xeon/Core/Flex/Arc/PVC)打造的低精度轻量级大语言模型库，在Intel平台上具有广泛的模型支持、最低的延迟和最小的内存占用，实现加速模型部署示例。
- OpenVINO_demo: 使用 Intel OpenVINO 推理加速框架，实现加速模型部署示例。
- Pytorch_demo (暂未推出) : 使用 Intel Pytorch Extension 实现在 Pytorch 环境上开发（适用于 Intel Arc 系列 GPU）
================================================
FILE: Intel_device_demo/ipex_llm_cpu_demo/api_server.py
================================================
"""
This script implements an API for the ChatGLM3-6B model,
formatted similarly to OpenAI's API (https://platform.openai.com/docs/api-reference/chat).
It's designed to be run as a web server using FastAPI and uvicorn,
making the ChatGLM3-6B model accessible through OpenAI Client.
Key Components and Features:
- Model and Tokenizer Setup: Configures the model and tokenizer paths and loads them.
- FastAPI Configuration: Sets up a FastAPI application with CORS middleware for handling cross-origin requests.
- API Endpoints:
- "/v1/models": Lists the available models, specifically ChatGLM3-6B.
- "/v1/chat/completions": Processes chat completion requests with options for streaming and regular responses.
- "/v1/embeddings": Processes Embedding request of a list of text inputs.
- Token Limit Caution: In the OpenAI API, 'max_tokens' is equivalent to HuggingFace's 'max_new_tokens', not 'max_length'.
For instance, setting 'max_tokens' to 8192 for a 6b model would result in an error due to the model's inability to output
that many tokens after accounting for the history and prompt tokens.
- Stream Handling and Custom Functions: Manages streaming responses and custom function calls within chat responses.
- Pydantic Models: Defines structured models for requests and responses, enhancing API documentation and type safety.
- Main Execution: Initializes the model and tokenizer, and starts the FastAPI app on the designated host and port.
Note:
This script doesn't include the setup for special tokens or multi-GPU support by default.
Users need to configure their special tokens and can enable multi-GPU support as per the provided instructions.
Embedding Models only support in One GPU.
"""
import os
import time
import tiktoken
import torch
import uvicorn
from fastapi import FastAPI, HTTPException, Response
from fastapi.middleware.cors import CORSMiddleware
from contextlib import asynccontextmanager
from typing import List, Literal, Optional, Union
from loguru import logger
from pydantic import BaseModel, Field
from ipex_llm.transformers import AutoModel
from transformers import AutoTokenizer
from utils import process_response, generate_chatglm3, generate_stream_chatglm3
# from sentence_transformers import SentenceTransformer
from sse_starlette.sse import EventSourceResponse
# Set up limit request time
EventSourceResponse.DEFAULT_PING_INTERVAL = 1000
# set LLM path
MODEL_PATH = os.environ.get('MODEL_PATH', 'THUDM/chatglm3-6b')
TOKENIZER_PATH = os.environ.get("TOKENIZER_PATH", MODEL_PATH)
# set Embedding Model path
EMBEDDING_PATH = os.environ.get('EMBEDDING_PATH', 'BAAI/bge-large-zh-v1.5')
@asynccontextmanager
async def lifespan(app: FastAPI):
yield
if torch.cuda.is_available():
torch.cuda.empty_cache()
torch.cuda.ipc_collect()
app = FastAPI(lifespan=lifespan)
app.add_middleware(
CORSMiddleware,
allow_origins=["*"],
allow_credentials=True,
allow_methods=["*"],
allow_headers=["*"],
)
class ModelCard(BaseModel):
id: str
object: str = "model"
created: int = Field(default_factory=lambda: int(time.time()))
owned_by: str = "owner"
root: Optional[str] = None
parent: Optional[str] = None
permission: Optional[list] = None
class ModelList(BaseModel):
object: str = "list"
data: List[ModelCard] = []
class FunctionCallResponse(BaseModel):
name: Optional[str] = None
arguments: Optional[str] = None
class ChatMessage(BaseModel):
role: Literal["user", "assistant", "system", "function"]
content: str = None
name: Optional[str] = None
function_call: Optional[FunctionCallResponse] = None
class DeltaMessage(BaseModel):
role: Optional[Literal["user", "assistant", "system"]] = None
content: Optional[str] = None
function_call: Optional[FunctionCallResponse] = None
## for Embedding
class EmbeddingRequest(BaseModel):
input: List[str]
model: str
class CompletionUsage(BaseModel):
prompt_tokens: int
completion_tokens: int
total_tokens: int
class EmbeddingResponse(BaseModel):
data: list
model: str
object: str
usage: CompletionUsage
# for ChatCompletionRequest
class UsageInfo(BaseModel):
prompt_tokens: int = 0
total_tokens: int = 0
completion_tokens: Optional[int] = 0
class ChatCompletionRequest(BaseModel):
model: str
messages: List[ChatMessage]
temperature: Optional[float] = 0.8
top_p: Optional[float] = 0.8
max_tokens: Optional[int] = None
stream: Optional[bool] = False
tools: Optional[Union[dict, List[dict]]] = None
repetition_penalty: Optional[float] = 1.1
class ChatCompletionResponseChoice(BaseModel):
index: int
message: ChatMessage
finish_reason: Literal["stop", "length", "function_call"]
class ChatCompletionResponseStreamChoice(BaseModel):
delta: DeltaMessage
finish_reason: Optional[Literal["stop", "length", "function_call"]]
index: int
class ChatCompletionResponse(BaseModel):
model: str
id: str
object: Literal["chat.completion", "chat.completion.chunk"]
choices: List[Union[ChatCompletionResponseChoice, ChatCompletionResponseStreamChoice]]
created: Optional[int] = Field(default_factory=lambda: int(time.time()))
usage: Optional[UsageInfo] = None
@app.get("/health")
async def health() -> Response:
"""Health check."""
return Response(status_code=200)
@app.post("/v1/embeddings", response_model=EmbeddingResponse)
async def get_embeddings(request: EmbeddingRequest):
embeddings = [embedding_model.encode(text) for text in request.input]
embeddings = [embedding.tolist() for embedding in embeddings]
def num_tokens_from_string(string: str) -> int:
"""
Returns the number of tokens in a text string.
use cl100k_base tokenizer
"""
encoding = tiktoken.get_encoding('cl100k_base')
num_tokens = len(encoding.encode(string))
return num_tokens
response = {
"data": [
{
"object": "embedding",
"embedding": embedding,
"index": index
}
for index, embedding in enumerate(embeddings)
],
"model": request.model,
"object": "list",
"usage": CompletionUsage(
prompt_tokens=sum(len(text.split()) for text in request.input),
completion_tokens=0,
total_tokens=sum(num_tokens_from_string(text) for text in request.input),
)
}
return response
@app.get("/v1/models", response_model=ModelList)
async def list_models():
model_card = ModelCard(
id="chatglm3-6b"
)
return ModelList(
data=[model_card]
)
@app.post("/v1/chat/completions", response_model=ChatCompletionResponse)
async def create_chat_completion(request: ChatCompletionRequest):
global model, tokenizer
if len(request.messages) < 1 or request.messages[-1].role == "assistant":
raise HTTPException(status_code=400, detail="Invalid request")
gen_params = dict(
messages=request.messages,
temperature=request.temperature,
top_p=request.top_p,
max_tokens=request.max_tokens or 1024,
echo=False,
stream=request.stream,
repetition_penalty=request.repetition_penalty,
tools=request.tools,
)
logger.debug(f"==== request ====\n{gen_params}")
if request.stream:
# Use the stream mode to read the first few characters, if it is not a function call, direct stram output
predict_stream_generator = predict_stream(request.model, gen_params)
output = next(predict_stream_generator)
if not contains_custom_function(output):
return EventSourceResponse(predict_stream_generator, media_type="text/event-stream")
# Obtain the result directly at one time and determine whether tools needs to be called.
logger.debug(f"First result output：\n{output}")
function_call = None
if output and request.tools:
try:
function_call = process_response(output, use_tool=True)
except:
logger.warning("Failed to parse tool call")
# CallFunction
if isinstance(function_call, dict):
function_call = FunctionCallResponse(**function_call)
"""
In this demo, we did not register any tools.
You can use the tools that have been implemented in our `tools_using_demo` and implement your own streaming tool implementation here.
Similar to the following method:
function_args = json.loads(function_call.arguments)
tool_response = dispatch_tool(tool_name: str, tool_params: dict)
"""
tool_response = ""
if not gen_params.get("messages"):
gen_params["messages"] = []
gen_params["messages"].append(ChatMessage(
role="assistant",
content=output,
))
gen_params["messages"].append(ChatMessage(
role="function",
name=function_call.name,
content=tool_response,
))
# Streaming output of results after function calls
generate = predict(request.model, gen_params)
return EventSourceResponse(generate, media_type="text/event-stream")
else:
# Handled to avoid exceptions in the above parsing function process.
generate = parse_output_text(request.model, output)
return EventSourceResponse(generate, media_type="text/event-stream")
# Here is the handling of stream = False
response = generate_chatglm3(model, tokenizer, gen_params)
# Remove the first newline character
if response["text"].startswith("\n"):
response["text"] = response["text"][1:]
response["text"] = response["text"].strip()
usage = UsageInfo()
function_call, finish_reason = None, "stop"
if request.tools:
try:
function_call = process_response(response["text"], use_tool=True)
except:
logger.warning("Failed to parse tool call, maybe the response is not a tool call or have been answered.")
if isinstance(function_call, dict):
finish_reason = "function_call"
function_call = FunctionCallResponse(**function_call)
message = ChatMessage(
role="assistant",
content=response["text"],
function_call=function_call if isinstance(function_call, FunctionCallResponse) else None,
)
logger.debug(f"==== message ====\n{message}")
choice_data = ChatCompletionResponseChoice(
index=0,
message=message,
finish_reason=finish_reason,
)
task_usage = UsageInfo.model_validate(response["usage"])
for usage_key, usage_value in task_usage.model_dump().items():
setattr(usage, usage_key, getattr(usage, usage_key) + usage_value)
return ChatCompletionResponse(
model=request.model,
id="", # for open_source model, id is empty
choices=[choice_data],
object="chat.completion",
usage=usage
)
async def predict(model_id: str, params: dict):
global model, tokenizer
choice_data = ChatCompletionResponseStreamChoice(
index=0,
delta=DeltaMessage(role="assistant"),
finish_reason=None
)
chunk = ChatCompletionResponse(model=model_id, id="", choices=[choice_data], object="chat.completion.chunk")
yield "{}".format(chunk.model_dump_json(exclude_unset=True))
previous_text = ""
for new_response in generate_stream_chatglm3(model, tokenizer, params):
decoded_unicode = new_response["text"]
delta_text = decoded_unicode[len(previous_text):]
previous_text = decoded_unicode
finish_reason = new_response["finish_reason"]
if len(delta_text) == 0 and finish_reason != "function_call":
continue
function_call = None
if finish_reason == "function_call":
try:
function_call = process_response(decoded_unicode, use_tool=True)
except:
logger.warning(
"Failed to parse tool call, maybe the response is not a tool call or have been answered.")
if isinstance(function_call, dict):
function_call = FunctionCallResponse(**function_call)
delta = DeltaMessage(
content=delta_text,
role="assistant",
function_call=function_call if isinstance(function_call, FunctionCallResponse) else None,
)
choice_data = ChatCompletionResponseStreamChoice(
index=0,
delta=delta,
finish_reason=finish_reason
)
chunk = ChatCompletionResponse(
model=model_id,
id="",
choices=[choice_data],
object="chat.completion.chunk"
)
yield "{}".format(chunk.model_dump_json(exclude_unset=True))
choice_data = ChatCompletionResponseStreamChoice(
index=0,
delta=DeltaMessage(),
finish_reason="stop"
)
chunk = ChatCompletionResponse(
model=model_id,
id="",
choices=[choice_data],
object="chat.completion.chunk"
)
yield "{}".format(chunk.model_dump_json(exclude_unset=True))
yield '[DONE]'
def predict_stream(model_id, gen_params):
"""
The function call is compatible with stream mode output.
The first seven characters are determined.
If not a function call, the stream output is directly generated.
Otherwise, the complete character content of the function call is returned.
:param model_id:
:param gen_params:
:return:
"""
output = ""
is_function_call = False
has_send_first_chunk = False
for new_response in generate_stream_chatglm3(model, tokenizer, gen_params):
decoded_unicode = new_response["text"]
delta_text = decoded_unicode[len(output):]
output = decoded_unicode
# When it is not a function call and the character length is> 7,
# try to judge whether it is a function call according to the special function prefix
if not is_function_call and len(output) > 7:
# Determine whether a function is called
is_function_call = contains_custom_function(output)
if is_function_call:
continue
# Non-function call, direct stream output
finish_reason = new_response["finish_reason"]
# Send an empty string first to avoid truncation by subsequent next() operations.
if not has_send_first_chunk:
message = DeltaMessage(
content="",
role="assistant",
function_call=None,
)
choice_data = ChatCompletionResponseStreamChoice(
index=0,
delta=message,
finish_reason=finish_reason
)
chunk = ChatCompletionResponse(
model=model_id,
id="",
choices=[choice_data],
created=int(time.time()),
object="chat.completion.chunk"
)
yield "{}".format(chunk.model_dump_json(exclude_unset=True))
send_msg = delta_text if has_send_first_chunk else output
has_send_first_chunk = True
message = DeltaMessage(
content=send_msg,
role="assistant",
function_call=None,
)
choice_data = ChatCompletionResponseStreamChoice(
index=0,
delta=message,
finish_reason=finish_reason
)
chunk = ChatCompletionResponse(
model=model_id,
id="",
choices=[choice_data],
created=int(time.time()),
object="chat.completion.chunk"
)
yield "{}".format(chunk.model_dump_json(exclude_unset=True))
if is_function_call:
yield output
else:
yield '[DONE]'
async def parse_output_text(model_id: str, value: str):
"""
Directly output the text content of value
:param model_id:
:param value:
:return:
"""
choice_data = ChatCompletionResponseStreamChoice(
index=0,
delta=DeltaMessage(role="assistant", content=value),
finish_reason=None
)
chunk = ChatCompletionResponse(model=model_id, id="", choices=[choice_data], object="chat.completion.chunk")
yield "{}".format(chunk.model_dump_json(exclude_unset=True))
choice_data = ChatCompletionResponseStreamChoice(
index=0,
delta=DeltaMessage(),
finish_reason="stop"
)
chunk = ChatCompletionResponse(model=model_id, id="", choices=[choice_data], object="chat.completion.chunk")
yield "{}".format(chunk.model_dump_json(exclude_unset=True))
yield '[DONE]'
def contains_custom_function(value: str) -> bool:
"""
Determine whether 'function_call' according to a special function prefix.
For example, the functions defined in "tools_using_demo/tool_register.py" are all "get_xxx" and start with "get_"
[Note] This is not a rigorous judgment method, only for reference.
:param value:
:return:
"""
return value and 'get_' in value
if __name__ == "__main__":
# Load LLM
tokenizer = AutoTokenizer.from_pretrained(TOKENIZER_PATH, trust_remote_code=True)
model = AutoModel.from_pretrained(MODEL_PATH,
load_in_4bit=True,
trust_remote_code=True)
# load Embedding
# embedding_model = SentenceTransformer(EMBEDDING_PATH, device="cuda")
uvicorn.run(app, host='0.0.0.0', port=8000, workers=1)
================================================
FILE: Intel_device_demo/ipex_llm_cpu_demo/chatglm3_infer.py
================================================
import time
from ipex_llm.transformers import AutoModel
from transformers import AutoTokenizer
CHATGLM_V3_PROMPT_FORMAT = "\n{prompt}\n"
# Please specify the local path to the chatglm3-6b model
model_path = "D:\AI\ChatGLM3\model/chatglm3-6b/"
# Load the ChatGLM3-6B model and quantize it to INT4
model = AutoModel.from_pretrained(model_path,
load_in_4bit=True,
trust_remote_code=True)
# Load the tokenizer
tokenizer = AutoTokenizer.from_pretrained(model_path,
trust_remote_code=True)
# Prepare ChatGLM3 format prompt
prompt = CHATGLM_V3_PROMPT_FORMAT.format(prompt="Who are you?")
# Encode the prompt
input_ids = tokenizer.encode(prompt, return_tensors="pt")
st = time.time()
# Perform inference calculation and generate Tokens
output = model.generate(input_ids,max_new_tokens=32)
end = time.time()
# Decode the generated Tokens and display them
output_str = tokenizer.decode(output[0], skip_special_tokens=True)
print(f'Inference time: {end-st} s')
print('-'*20, 'Prompt', '-'*20)
print(prompt)
print('-'*20, 'Output', '-'*20)
print(output_str)
================================================
FILE: Intel_device_demo/ipex_llm_cpu_demo/chatglm3_web_demo.py
================================================
"""
This script creates an interactive web demo for the ChatGLM3-6B model using Gradio,
a Python library for building quick and easy UI components for machine learning models.
It's designed to showcase the capabilities of the ChatGLM3-6B model in a user-friendly interface,
allowing users to interact with the model through a chat-like interface.
Usage:
- Run the script to start the Gradio web server.
- Interact with the model by typing questions and receiving responses.
Requirements:
- Gradio (required for 4.13.0 and later, 3.x is not support now) should be installed.
Note: The script includes a modification to the Chatbot's postprocess method to handle markdown to HTML conversion,
ensuring that the chat interface displays formatted text correctly.
"""
import os
import streamlit as st
from ipex_llm.transformers import AutoModel
from transformers import AutoTokenizer
st.set_page_config(
page_title="ChatGLM3-6B+BigDL-LLM demo",
page_icon=":robot:",
layout="wide"
)
MODEL_PATH = os.environ.get('MODEL_PATH', 'THUDM/chatglm3-6b')
@st.cache_resource
def get_model():
model = AutoModel.from_pretrained(MODEL_PATH,
load_in_4bit=True,
trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained(MODEL_PATH,
trust_remote_code=True)
return tokenizer, model
tokenizer, model = get_model()
if "history" not in st.session_state:
st.session_state.history = []
if "past_key_values" not in st.session_state:
st.session_state.past_key_values = None
max_length = st.sidebar.slider("max_length", 0, 32768, 8192, step=1)
top_p = st.sidebar.slider("top_p", 0.0, 1.0, 0.8, step=0.01)
temperature = st.sidebar.slider("temperature", 0.0, 1.0, 0.6, step=0.01)
buttonClean = st.sidebar.button("clearing session history", key="clean")
if buttonClean:
st.session_state.history = []
st.session_state.past_key_values = None
st.rerun()
for i, message in enumerate(st.session_state.history):
if message["role"] == "user":
with st.chat_message(name="user", avatar="user"):
st.markdown(message["content"])
else:
with st.chat_message(name="assistant", avatar="assistant"):
st.markdown(message["content"])
with st.chat_message(name="user", avatar="user"):
input_placeholder = st.empty()
with st.chat_message(name="assistant", avatar="assistant"):
message_placeholder = st.empty()
prompt_text = st.chat_input("please enter your question.")
if prompt_text:
input_placeholder.markdown(prompt_text)
history = st.session_state.history
past_key_values = st.session_state.past_key_values
for response, history, past_key_values in model.stream_chat(
tokenizer,
prompt_text,
history,
past_key_values=past_key_values,
max_length=max_length,
top_p=top_p,
temperature=temperature,
return_past_key_values=True,
):
message_placeholder.markdown(response)
st.session_state.history = history
st.session_state.past_key_values = past_key_values
================================================
FILE: Intel_device_demo/ipex_llm_cpu_demo/generate.py
================================================
import torch
import time
import argparse
import numpy as np
from ipex_llm.transformers import AutoModel
from modelscope import AutoTokenizer
from transformers import AutoTokenizer
# you could tune the prompt based on your own model,
# here the prompt tuning refers to https://github.com/THUDM/ChatGLM3/blob/main/PROMPT.md
CHATGLM_V3_PROMPT_FORMAT = "<|user|>\n{prompt}\n<|assistant|>"
if __name__ == '__main__':
parser = argparse.ArgumentParser(description='Predict Tokens using `generate()` API for ModelScope ChatGLM3 model')
parser.add_argument('--repo-id-or-model-path', type=str, default="ZhipuAI/chatglm3-6b",
help='The ModelScope repo id for the ChatGLM3 model to be downloaded'
', or the path to the ModelScope checkpoint folder')
parser.add_argument('--prompt', type=str, default="AI是什么？",
help='Prompt to infer')
parser.add_argument('--n-predict', type=int, default=32,
help='Max tokens to predict')
args = parser.parse_args()
model_path = args.repo_id_or_model_path
# Load model in 4 bit,
# which convert the relevant layers in the model into INT4 format
# It is important to set `model_hub='modelscope'`, otherwise model hub is default to be huggingface
model = AutoModel.from_pretrained(model_path,
load_in_4bit=True,
trust_remote_code=True,
model_hub='modelscope')
# Load tokenizer
tokenizer = AutoTokenizer.from_pretrained(model_path,
trust_remote_code=True)
# Generate predicted tokens
with torch.inference_mode():
prompt = CHATGLM_V3_PROMPT_FORMAT.format(prompt=args.prompt)
input_ids = tokenizer.encode(prompt, return_tensors="pt")
st = time.time()
# if your selected model is capable of utilizing previous key/value attentions
# to enhance decoding speed, but has `"use_cache": false` in its model config,
# it is important to set `use_cache=True` explicitly in the `generate` function
# to obtain optimal performance with IPEX-LLM INT4 optimizations
output = model.generate(input_ids,
max_new_tokens=args.n_predict)
end = time.time()
output_str = tokenizer.decode(output[0], skip_special_tokens=True)
print(f'Inference time: {end - st} s')
print('-' * 20, 'Prompt', '-' * 20)
print(prompt)
print('-' * 20, 'Output', '-' * 20)
print(output_str)
================================================
FILE: Intel_device_demo/ipex_llm_cpu_demo/openai_api_request.py
================================================
"""
This script is an example of using the OpenAI API to create various interactions with a ChatGLM3 model.
It includes functions to:
1. Conduct a basic chat session, asking about weather conditions in multiple cities.
2. Initiate a simple chat in Chinese, asking the model to tell a short story.
3. Retrieve and print embeddings for a given text input.
Each function demonstrates a different aspect of the API's capabilities, showcasing how to make requests
and handle responses.
"""
from openai import OpenAI
import time
base_url = "http://127.0.0.1:8000/v1/"
client = OpenAI(api_key="EMPTY", base_url=base_url)
def function_chat():
messages = [{"role": "user", "content": "What's the weather like in San Francisco, Tokyo, and Paris?"}]
tools = [
{
"type": "function",
"function": {
"name": "get_current_weather",
"description": "Get the current weather in a given location",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "The city and state, e.g. San Francisco, CA",
},
"unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
},
"required": ["location"],
},
},
}
]
response = client.chat.completions.create(
model="chatglm3-6b",
messages=messages,
tools=tools,
tool_choice="auto",
)
if response:
content = response.choices[0].message.content
print(content)
else:
print("Error:", response.status_code)
def simple_chat(use_stream=True):
messages = [
{
"role": "system",
"content": "You are ChatGLM3, a large language model trained by Zhipu.AI. Follow the user's "
"instructions carefully. Respond using markdown.",
},
{
"role": "user",
"content": "你好，请你用生动的话语给我讲一个小故事吧"
}
]
response = client.chat.completions.create(
model="chatglm3-6b",
messages=messages,
stream=use_stream,
max_tokens=256,
temperature=0.8,
presence_penalty=1.1,
top_p=0.8)
if response:
if use_stream:
for chunk in response:
print(chunk.choices[0].delta.content)
else:
content = response.choices[0].message.content
print(content)
else:
print("Error:", response.status_code)
if __name__ == "__main__":
simple_chat(use_stream=False)
simple_chat(use_stream=True)
================================================
FILE: Intel_device_demo/ipex_llm_cpu_demo/utils.py
================================================
import gc
import json
import torch
from transformers import PreTrainedModel, PreTrainedTokenizer
from transformers.generation.logits_process import LogitsProcessor
from typing import Union, Tuple
class InvalidScoreLogitsProcessor(LogitsProcessor):
def __call__(
self, input_ids: torch.LongTensor, scores: torch.FloatTensor
) -> torch.FloatTensor:
if torch.isnan(scores).any() or torch.isinf(scores).any():
scores.zero_()
scores[..., 5] = 5e4
return scores
def process_response(output: str, use_tool: bool = False) -> Union[str, dict]:
content = ""
for response in output.split("<|assistant|>"):
metadata, content = response.split("\n", maxsplit=1)
if not metadata.strip():
content = content.strip()
content = content.replace("[[训练时间]]", "2023年")
else:
if use_tool:
content = "\n".join(content.split("\n")[1:-1])
def tool_call(**kwargs):
return kwargs
parameters = eval(content)
content = {
"name": metadata.strip(),
"arguments": json.dumps(parameters, ensure_ascii=False)
}
else:
content = {
"name": metadata.strip(),
"content": content
}
return content
@torch.inference_mode()
def generate_stream_chatglm3(model: PreTrainedModel, tokenizer: PreTrainedTokenizer, params: dict):
messages = params["messages"]
tools = params["tools"]
temperature = float(params.get("temperature", 1.0))
repetition_penalty = float(params.get("repetition_penalty", 1.0))
top_p = float(params.get("top_p", 1.0))
max_new_tokens = int(params.get("max_tokens", 256))
echo = params.get("echo", True)
messages = process_chatglm_messages(messages, tools=tools)
query, role = messages[-1]["content"], messages[-1]["role"]
inputs = tokenizer.build_chat_input(query, history=messages[:-1], role=role)
inputs = inputs.to(model.device)
input_echo_len = len(inputs["input_ids"][0])
if input_echo_len >= model.config.seq_length:
print(f"Input length larger than {model.config.seq_length}")
eos_token_id = [
tokenizer.eos_token_id,
tokenizer.get_command("<|user|>"),
]
gen_kwargs = {
"max_new_tokens": max_new_tokens,
"do_sample": True if temperature > 1e-5 else False,
"top_p": top_p,
"repetition_penalty": repetition_penalty,
"logits_processor": [InvalidScoreLogitsProcessor()],
}
if temperature > 1e-5:
gen_kwargs["temperature"] = temperature
total_len = 0
for total_ids in model.stream_generate(**inputs, eos_token_id=eos_token_id, **gen_kwargs):
total_ids = total_ids.tolist()[0]
total_len = len(total_ids)
if echo:
output_ids = total_ids[:-1]
else:
output_ids = total_ids[input_echo_len:-1]
response = tokenizer.decode(output_ids)
if response and response[-1] != "�":
response, stop_found = apply_stopping_strings(response, ["<|observation|>"])
yield {
"text": response,
"usage": {
"prompt_tokens": input_echo_len,
"completion_tokens": total_len - input_echo_len,
"total_tokens": total_len,
},
"finish_reason": "function_call" if stop_found else None,
}
if stop_found:
break
# Only last stream result contains finish_reason, we set finish_reason as stop
ret = {
"text": response,
"usage": {
"prompt_tokens": input_echo_len,
"completion_tokens": total_len - input_echo_len,
"total_tokens": total_len,
},
"finish_reason": "stop",
}
yield ret
gc.collect()
torch.cuda.empty_cache()
def process_chatglm_messages(messages, tools=None):
_messages = messages
messages = []
if tools:
messages.append(
{
"role": "system",
"content": "Answer the following questions as best as you can. You have access to the following tools:",
"tools": tools
}
)
for m in _messages:
role, content, func_call = m.role, m.content, m.function_call
if role == "function":
messages.append(
{
"role": "observation",
"content": content
}
)
elif role == "assistant" and func_call is not None:
for response in content.split("<|assistant|>"):
metadata, sub_content = response.split("\n", maxsplit=1)
messages.append(
{
"role": role,
"metadata": metadata,
"content": sub_content.strip()
}
)
else:
messages.append({"role": role, "content": content})
return messages
def generate_chatglm3(model: PreTrainedModel, tokenizer: PreTrainedTokenizer, params: dict):
for response in generate_stream_chatglm3(model, tokenizer, params):
pass
return response
def apply_stopping_strings(reply, stop_strings) -> Tuple[str, bool]:
stop_found = False
for string in stop_strings:
idx = reply.find(string)
if idx != -1:
reply = reply[:idx]
stop_found = True
break
if not stop_found:
# If something like "\nYo" is generated just before "\nYou: is completed, trim it
for string in stop_strings:
for j in range(len(string) - 1, 0, -1):
if reply[-j:] == string[:j]:
reply = reply[:-j]
break
else:
continue
break
return reply, stop_found
================================================
FILE: Intel_device_demo/openvino_demo/README.md
================================================
# 使用 OpenVINO 部署ChatGLM3-6B 模型
[OpenVINO](https://www.intel.com/content/www/us/en/developer/tools/openvino-toolkit/overview.html) 是 Intel 为深度学习推理而设计的开源工具包。它可以帮助开发者优化模型，提高推理性能，减少模型的内存占用。本示例将展示如何使用 OpenVINO 部署 ChatGLM3。
你需要克隆本仓库，然后按照以下步骤进行操作来将模型转换你的 OpenVINO IR 模型，随后进行推理。
## 1. 环境配置
首先，克隆OpenVINO GLM3 推理仓库并安装依赖。
```bash
git clone https://github.com/OpenVINO-dev-contest/chatglm3.openvino.git
cd chatglm3.openvino
```
接着，我们推荐您新建一个虚拟环境，然后按照以下安装依赖。
```
python3 -m venv openvino_env
source openvino_env/bin/activate
python3 -m pip install --upgrade pip
pip install wheel setuptools
pip install -r requirements.txt
```
## 2. 转换模型
由于需要将Huggingface模型转换为OpenVINO IR模型，因此您需要下载模型并转换。
```
python3 convert.py --model_id THUDM/chatglm3-6b --output {your_path}/chatglm3-6b
```
### 可以选择的参数
* `--model_id` - 模型所在目录的路径（绝对路径）。
* `--output` - 转换后模型保存的地址
## 3. 量化模型（非必须）
```
python3 quantize.py --model_path {your_path}/chatglm3-6b --precision int4 --output {your_path}/chatglm3-6b-int4
```
### 可以选择的参数
* `--model_path` - OpenVINO IR 模型所在目录的路径。
* `-- precision` - 量化精度：int8 或 int4。
* `--output` - 保存模型的路径。
## 4. 运行 ChatGLM3 模型
```
python3 chat.py --model_path {your_path}/chatglm3-6b --max_sequence_length 4096 --device CPU
```
### 可以选择的参数
* `--model_path` - OpenVINO IR 模型所在目录的路径。
* `--max_sequence_length` - 输出标记的最大大小。
* `--device` - 运行推理的设备。
## 例子
```
用户: 你好
ChatGLM3-6B-OpenVINO: 你好！有什么我可以帮助你的吗？
用户: 你是谁？
ChatGLM3-6B-OpenVINO: 我是一个名为ChatGLM3-6B的人工智能助手，是由清华大学KEG实验室和智谱AI 公司于2023 年共同训练的语言模型开发而成。我的任务是针对用户的问题和要求提供适当的答复和支持。
用户: 请给我讲一个有趣的故事
ChatGLM3-6B-OpenVINO: 从前，有一个名叫小明的小男孩，他是一个非常喜欢动物的人。有一天，他在森林里散步时，发现了一个非常漂亮的小鸟。小鸟受伤了，无法飞行。小明非常心疼，于是决定照顾这只小鸟。小明带着小鸟回家，为它搭建了一个小小的巢穴，并找来了一些软草和食物。每天，他都会给小鸟喂食，并为它换水。渐渐地，小鸟的伤势好了起来，开始在小明的家里飞来飞去，它们成了非常好的朋友。然而，一天，小明的父母告诉他，他们必须把小明养的小鸟送到森林里去。小明非常伤心，因为他已经和小鸟成为了好朋友。但是，他的父母告诉他，小鸟在森林里会更加自由自在，而且他也可以继续观看小鸟在森林中的生活。于是，小明和他的父母一起将小鸟送到了森林中。小鸟非常高兴，因为它又可以飞行了，并且还有许多其他的小动物朋友。小明也感到非常开心，因为他知道，即使不能一直拥有小鸟，他仍然可以欣赏到它们在自然中的美丽。从此以后，小明常常来到森林中，寻找小鸟。
用户: 请给这个故事起一个标题
ChatGLM3-6B-OpenVINO: 《友谊的力量：小明与小鸟的森林冒险》
```
## 常见问题
1. 为什么倒入本地模型还会报 huggingface 链接错误
- 降级 transformers 库到 4.37.2 版本
2. 需要安装 OpenVINO C++ 推理引擎吗
- 不需要
3. 一定要使用 Intel 的硬件吗？
- 我们仅在 Intel 设备上尝试，我们推荐使用x86架构的英特尔设备，包括但不限制于：
- 英特尔的CPU，包括个人电脑CPU 和服务器CPU。
- 英特尔的独立显卡。例如：ARC A770 显卡。
================================================
FILE: Intel_device_demo/openvino_demo/openvino_cli_demo.py
================================================
import argparse
from typing import List, Tuple
from threading import Thread
import torch
from optimum.intel.openvino import OVModelForCausalLM
from transformers import (AutoTokenizer, AutoConfig,
TextIteratorStreamer, StoppingCriteriaList, StoppingCriteria)
def parse_text(text):
lines = text.split("\n")
lines = [line for line in lines if line != ""]
count = 0
for i, line in enumerate(lines):
if "```" in line:
count += 1
items = line.split('`')
if count % 2 == 1:
lines[i] = f'<pre><code class="language-{items[-1]}">'
else:
lines[i] = f'<br></code></pre>'
else:
if i > 0:
if count % 2 == 1:
line = line.replace("`", "\`")
line = line.replace("<", "<")
line = line.replace(">", ">")
line = line.replace(" ", " ")
line = line.replace("*", "*")
line = line.replace("_", "_")
line = line.replace("-", "-")
line = line.replace(".", ".")
line = line.replace("!", "!")
line = line.replace("(", "(")
line = line.replace(")", ")")
line = line.replace("$", "$")
lines[i] = "<br>" + line
text = "".join(lines)
return text
class StopOnTokens(StoppingCriteria):
def __init__(self, token_ids):
self.token_ids = token_ids
def __call__(
self, input_ids: torch.LongTensor, scores: torch.FloatTensor, **kwargs
) -> bool:
for stop_id in self.token_ids:
if input_ids[0][-1] == stop_id:
return True
return False
if __name__ == "__main__":
parser = argparse.ArgumentParser(add_help=False)
parser.add_argument('-h',
'--help',
action='help',
help='Show this help message and exit.')
parser.add_argument('-m',
'--model_path',
required=True,
type=str,
help='Required. model path')
parser.add_argument('-l',
'--max_sequence_length',
default=256,
required=False,
type=int,
help='Required. maximun length of output')
parser.add_argument('-d',
'--device',
default='CPU',
required=False,
type=str,
help='Required. device for inference')
args = parser.parse_args()
model_dir = args.model_path
ov_config = {"PERFORMANCE_HINT": "LATENCY",
"NUM_STREAMS": "1", "CACHE_DIR": ""}
tokenizer = AutoTokenizer.from_pretrained(
model_dir, trust_remote_code=True)
print("====Compiling model====")
ov_model = OVModelForCausalLM.from_pretrained(
model_dir,
device=args.device,
ov_config=ov_config,
config=AutoConfig.from_pretrained(model_dir, trust_remote_code=True),
trust_remote_code=True,
)
streamer = TextIteratorStreamer(
tokenizer, timeout=60.0, skip_prompt=True, skip_special_tokens=True
)
stop_tokens = [0, 2]
stop_tokens = [StopOnTokens(stop_tokens)]
def convert_history_to_token(history: List[Tuple[str, str]]):
messages = []
for idx, (user_msg, model_msg) in enumerate(history):
if idx == len(history) - 1 and not model_msg:
messages.append({"role": "user", "content": user_msg})
break
if user_msg:
messages.append({"role": "user", "content": user_msg})
if model_msg:
messages.append({"role": "assistant", "content": model_msg})
model_inputs = tokenizer.apply_chat_template(messages,
add_generation_prompt=True,
tokenize=True,
return_tensors="pt")
return model_inputs
history = []
print("====Starting conversation====")
while True:
input_text = input("用户: ")
if input_text.lower() == 'stop':
break
if input_text.lower() == 'clear':
history = []
print("AI助手: 对话历史已清空")
continue
print("ChatGLM3-6B-OpenVINO:", end=" ")
history = history + [[parse_text(input_text), ""]]
model_inputs = convert_history_to_token(history)
generate_kwargs = dict(
input_ids=model_inputs,
max_new_tokens=args.max_sequence_length,
temperature=0.1,
do_sample=True,
top_p=1.0,
top_k=50,
repetition_penalty=1.1,
streamer=streamer,
stopping_criteria=StoppingCriteriaList(stop_tokens)
)
t1 = Thread(target=ov_model.generate, kwargs=generate_kwargs)
t1.start()
partial_text = ""
for new_text in streamer:
new_text = new_text
print(new_text, end="", flush=True)
partial_text += new_text
print("\n")
history[-1][1] = partial_text
================================================
FILE: LICENSE
================================================
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright 2024 ChatGLM team @ Zhipu AI
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
================================================
FILE: MODEL_LICENSE
================================================
The ChatGLM3-6B License
1. 定义
“许可方”是指分发其软件的 ChatGLM3-6B 模型团队。
“软件”是指根据本许可提供的 ChatGLM3-6B 模型参数。
2. 许可授予
根据本许可的条款和条件，许可方特此授予您非排他性、全球性、不可转让、不可再许可、可撤销、免版税的版权许可。
本许可允许您免费使用本仓库中的所有开源模型进行学术研究，对于希望将模型用于商业目的的用户，需在[这里](https://open.bigmodel.cn/mla/form)完成登记。
经过登记的用户可以免费使用本模型进行商业活动，但必须遵守本许可的所有条款和条件。
上述版权声明和本许可声明应包含在本软件的所有副本或重要部分中。
3.限制
您不得出于任何军事或非法目的使用、复制、修改、合并、发布、分发、复制或创建本软件的全部或部分衍生作品。
您不得利用本软件从事任何危害国家安全和国家统一，危害社会公共利益及公序良俗，侵犯他人商业秘密、知识产权、名誉权、肖像权、财产权等权益的行为。
您在使用中应遵循使用地所适用的法律法规政策、道德规范等要求。
4.免责声明
本软件“按原样”提供，不提供任何明示或暗示的保证，包括但不限于对适销性、特定用途的适用性和非侵权性的保证。 在任何情况下，作者或版权持有人均不对任何索赔、损害或其他责任负责，无论是在合同诉讼、侵权行为还是其他方面，由软件或软件的使用或其他交易引起、由软件引起或与之相关 软件。
5. 责任限制
除适用法律禁止的范围外，在任何情况下且根据任何法律理论，无论是基于侵权行为、疏忽、合同、责任或其他原因，任何许可方均不对您承担任何直接、间接、特殊、偶然、示范性、 或间接损害，或任何其他商业损失，即使许可人已被告知此类损害的可能性。
6.争议解决
本许可受中华人民共和国法律管辖并按其解释。 因本许可引起的或与本许可有关的任何争议应提交北京市海淀区人民法院。
请注意，许可证可能会更新到更全面的版本。 有关许可和版权的任何问题，请通过 license@zhipuai.cn 与我们联系。
1. Definitions
“Licensor” means the ChatGLM3-6B Model Team that distributes its Software.
“Software” means the ChatGLM3-6B model parameters made available under this license.
2. License Grant
Under the terms and conditions of this license, the Licensor hereby grants you a non-exclusive, worldwide, non-transferable, non-sublicensable, revocable, royalty-free copyright license.
This license permits you to use all open-source models in this repository for academic research free. Users who wish to use the models for commercial purposes must register [here](https://open.bigmodel.cn/mla/form).
Registered users may use the models for commercial activities free of charge, but must comply with all terms and conditions of this license.
The license notice shall be included in all copies or substantial portions of the Software.
3. Restrictions
You are not allowed to use, copy, modify, merge, publish, distribute, copy or create all or part of the derivative works of this software for any military or illegal purposes.
You are not allowed to use this software to engage in any behavior that endangers national security and unity, endangers social public interests and public order, infringes on the rights and interests of others such as trade secrets, intellectual property rights, reputation rights, portrait rights, and property rights.
You should comply with the applicable laws, regulations, policies, ethical standards, and other requirements in the place of use during use.
4. Disclaimer
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
5. Limitation of Liability
EXCEPT TO THE EXTENT PROHIBITED BY APPLICABLE LAW, IN NO EVENT AND UNDER NO LEGAL THEORY, WHETHER BASED IN TORT, NEGLIGENCE, CONTRACT, LIABILITY, OR OTHERWISE WILL ANY LICENSOR BE LIABLE TO YOU FOR ANY DIRECT, INDIRECT, SPECIAL, INCIDENTAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES, OR ANY OTHER COMMERCIAL LOSSES, EVEN IF THE LICENSOR HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.
6. Dispute Resolution
This license shall be governed and construed in accordance with the laws of People’s Republic of China. Any dispute arising from or in connection with this License shall be submitted to Haidian District People's Court in Beijing.
Note that the license is subject to update to a more comprehensive version. For any questions related to the license and copyright, please contact us at license@zhipuai.cn.
================================================
FILE: PROMPT.md
================================================
## ChatGLM3 对话格式
为了避免用户输入的注入攻击，以及统一 Code Interpreter，Tool & Agent 等任务的输入，ChatGLM3 采用了全新的对话格式。
### 规定
#### 整体结构
ChatGLM3 对话的格式由若干对话组成，其中每个对话包含对话头和内容，一个典型的多轮对话结构如下
```text
<|system|>
You are ChatGLM3, a large language model trained by Zhipu.AI. Follow the user's instructions carefully. Respond using markdown.
<|user|>
Hello
<|assistant|>
Hello, I'm ChatGLM3. What can I assist you today?
```
**实际中每轮对话内容并不一定以换行符结尾，这里只是为了美观，下同**
#### 对话头
对话头占完整的一行，格式为
```text
<|role|>{metadata}
```
其中 `<|role|>` 部分使用 special token 表示，无法从文本形式被 tokenizer 编码以防止注入。metadata 部分采用纯文本表示，为可选内容。
* `<|system|>`：系统信息，设计上可穿插于对话中，**但目前规定仅可以出现在开头**
* `<|user|>`：用户
- 不会连续出现多个来自 `<|user|>` 的信息
* `<|assistant|>`：AI 助手
- 在出现之前必须有一个来自 `<|user|>` 的信息
* `<|observation|>`：外部的返回结果
- 必须在 `<|assistant|>` 的信息之后
### 样例场景
为提升可读性，下列样例场景中表示角色的 special token 前均额外添加了一个换行符。实际使用及 tokenizer 实现中均无需额外添加这一换行。
#### 多轮对话
* 有且仅有 `<|user|>`、`<|assistant|>`、`<|system|>` 三种 role
```text
<|system|>
You are ChatGLM3, a large language model trained by Zhipu.AI. Follow the user's instructions carefully. Respond using markdown.
<|user|>
Hello
<|assistant|>
Hello, I'm ChatGLM3. What can I assist you today?
```
#### 工具调用
````
<|system|>
Answer the following questions as best as you can. You have access to the following tools:
[
{
"name": "get_current_weather",
"description": "Get the current weather in a given location",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "The city and state, e.g. San Francisco, CA",
},
"unit": {"type": "string"},
},
"required": ["location"],
},
}
]
<|user|>
今天北京的天气怎么样？
<|assistant|>
好的，让我们来查看今天的天气
<|assistant|>get_current_weather
```python
tool_call(location="beijing", unit="celsius")
```
<|observation|>
{"temperature": 22}
<|assistant|>
根据查询结果，今天北京的气温为 22 摄氏度。
````
#### 代码执行
* 有 `<|user|>`、`<|assistant|>`、`<|system|>`、`<|observation|>` 四种 role。其中 `<|assistant|>` 的 metadata 只有 interpreter。
`````text
<|system|>
你是一位智能AI助手，你叫ChatGLM3，你连接着一台电脑，但请注意不能联网。在使用Python解决任务时，你可以运行代码并得到结果，如果运行结果有错误，你需要尽可能对代码进行改进。你可以处理用户上传到电脑上的文件，文件默认存储路径是/mnt/data/。
<|user|>
#File: /mnt/data/metadata.jsonl
#Size: 35380
#File uploaded
文件中是否存在缺失值或异常值？
<|assistant|>
首先，我将读取您提供的文件，并查看其内容以确定是否存在缺失值或异常值。
我们从加载文件开始：
<|assistant|>interpreter
```python
import json
# Load the uploaded file
data = []
with open("/mnt/data/metadata.jsonl", 'r') as file:
for line in file:
data.append(json.loads(line))
# Display the first few records to get a sense of the data
data[:5]
```
<|observation|>
```result
[{'file_name': 'aceinthehole.png',
'name': 'Ace in the Hole',
'type': 'survivor',
'description': 'Lady Luck always seems to be throwing something good your way.'},
{'file_name': 'adrenaline.png',
'name': 'Adrenaline',
'type': 'survivor',
'description': 'You are fuelled by unexpected energy when on the verge of escape.'},
{'file_name': 'aftercare.png',
'name': 'Aftercare',
'type': 'survivor',
'description': "Unlocks potential in one's aura reading ability."},
{'file_name': 'agitation.png',
'name': 'Agitation',
'type': 'killer',
'description': 'You get excited in anticipation of hooking your prey.'},
{'file_name': 'alert.png',
'name': 'Alert',
'type': 'survivor',
'description': 'Your acute senses are on high alert.'}]
```
<|assistant|>
该文件看起来包含有关某些条目的元数据，每个条目有以下字段：
- `file_name`: 文件名称
- `name`: 名称
- `type`: 类型（例如 "survivor" 或 "killer"）
- `description`: 描述
接下来，我将检查这些字段是否存在缺失值。
<|assistant|>interpreter
```python
# Check for missing values in each column
missing_values = {key: sum(1 for item in data if key not in item) for key in data[0].keys()}
missing_values
```
<|observation|>
```result
{'file_name': 0, 'name': 0, 'type': 0, 'description': 0}
```
<|assistant|>
每个字段都没有缺失值。这意味着每个条目都完整。
接下来，我们可以检查是否存在任何潜在的异常值。这通常需要更深入的分析和对数据的了解，但我可以从一些基本的统计和分布开始进行检查。为了这样做，我们可以查看“type”字段的值分布，看看是否有任何意外的类别。
<|assistant|>interpreter
```python
# Check the distribution of values in the 'type' column
type_distribution = {key: sum(1 for item in data if item['type'] == key) for key in set(item['type'] for item in data)}
type_distribution
```
<|observation|>
```result
{'survivor': 116, 'killer': 103}
```
<|assistant|>
“type”字段有两个值：“survivor”和“killer”。其中，“survivor”有116个条目，而“killer”有103个条目。
从这个初步的检查中，我们没有发现任何明显的异常值。
如果您有任何特定的标准或其他字段需要检查，我可以为您提供更详细的分析。您还希望我检查什么呢？
<|user|>
帮我画一个爱心
<|assistant|>interpreter
```python
import numpy as np
import matplotlib.pyplot as plt
# 爱心形状的参数方程
def heart(t):
x = 16 * np.sin(t) ** 3
y = 13 * np.cos(t) - 5 * np.cos(2 * t) - 2 * np.cos(3 * t) - np.cos(4 * t)
return x, y
t = np.linspace(0, 2 * np.pi, 1000)
x, y = heart(t)
plt.figure(figsize=(6, 6))
plt.plot(x, y, color='red')
plt.axis('equal')
plt.axis('off')
plt.show()
```
<|observation|>
```result
[Image]
```
<|assistant|>
这是一个爱心形状。我使用了参数方程来描述这个形状，并使用matplotlib进行了绘制。如果您有任何其他需求或问题，请随时告诉我。
<|user|> # End
````
================================================
FILE: PROMPT_en.md
================================================
## ChatGLM3 Chat Format
To avoid injection attacks from user input, and to unify the input of Code Interpreter, Tool & Agent and other tasks, ChatGLM3 adopts a brand-new dialogue format.
### Regulations
#### Overall Structure
The format of the ChatGLM3 dialogue consists of several conversations, each of which contains a dialogue header and content. A typical multi-turn dialogue structure is as follows:
```text
<|system|>
You are ChatGLM3, a large language model trained by Zhipu.AI. Follow the user's instructions carefully. Respond using markdown.
<|user|>
Hello
<|assistant|>
Hello, I'm ChatGLM3. What can I assist you today?
```
#### Chat Header
The chat header occupies a complete line, formatted as:
```text
<|role|>{metadata}
```
Where `<|role|>` part is represented in a special token, which can’t be encoded by the tokenizer from the text form to prevent injection attacks. The `metadata` part is represented in plain texts and is optional content.
* `<|system|>`: System information, which can be interspersed in the dialogue in design, **but currently only appears at the beginning**
* `<|user|>`: User
- Multiple messages from `<|user|>` will not appear continuously
* `<|assistant|>`: AI assistant
- There must be a message from `<|user|>` before it appears
* `<|observation|>`: External return result
- Must be after the message from `<|assistant|>`
### Example Scenarios
For better readability, an extra `\n` is added before each role special token. This extra `\n` should not be added in actual use and tokenizer implementation.
#### Multi-turn Dialogue
* There are only three roles: `<|user|>`, `<|assistant|>`, and `<|system|>`.
```text
<|system|>
You are ChatGLM3, a large language model trained by Zhipu.AI. Follow the user's instructions carefully. Respond using markdown.
<|user|>
Hello
<|assistant|>
Hello, I'm ChatGLM3. What can I assist you today?
```
#### Tool Calling
````
<|system|>
Answer the following questions as best as you can. You have access to the following tools:
[
{
"name": "get_current_weather",
"description": "Get the current weather in a given location",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "The city and state, e.g. San Francisco, CA",
},
"unit": {"type": "string"},
},
"required": ["location"],
},
}
]
<|user|>
What's the weather in Beijing today?
<|assistant|>
Okay, let's look up the weather in Bejing today.
<|assistant|>get_current_weather
```python
tool_call(location="beijing", unit="celsius")
```
<|observation|>
{"temperature": 22}
<|assistant|>
According to the query results, the temperature in Beijing today is 22 degrees Celsius.
````
#### Code Execution
* Four available roles: `<|user|>`, `<|assistant|>`, `<|system|>`, and `<|observation|>`, where `metadata` of `<|assistant|>` can only be interpreter.
`````text
<|system|>
You are an intelligent AI assistant, named ChatGLM3. You are connected to a computer, but please note that you cannot connect to the internet. When using Python to solve tasks, you can run the code and get the results. If there are errors in the running results, you need to improve the code as much as possible. You can process files uploaded by users to your computer, and the default storage path for files is /mnt/data/.
<|user|>
#File: /mnt/data/metadata.jsonl
#Size: 35380
#File uploaded
Are there missing or abnormal values in the file?
<|assistant|>
Firstly, I will read the file you provided and review its contents to determine if there are missing or abnormal values.
We start by loading the file:
<|assistant|>interpreter
```python
import json
# Load the uploaded file
data = []
with open("/mnt/data/metadata.jsonl", 'r') as file:
for line in file:
data.append(json.loads(line))
# Display the first few records to get a sense of the data
data[:5]
```
<|observation|>
```result
[{'file_name': 'aceinthehole.png',
'name': 'Ace in the Hole',
'type': 'survivor',
'description': 'Lady Luck always seems to be throwing something good your way.'},
{'file_name': 'adrenaline.png',
'name': 'Adrenaline',
'type': 'survivor',
'description': 'You are fuelled by unexpected energy when on the verge of escape.'},
{'file_name': 'aftercare.png',
'name': 'Aftercare',
'type': 'survivor',
'description': "Unlocks potential in one's aura reading ability."},
{'file_name': 'agitation.png',
'name': 'Agitation',
'type': 'killer',
'description': 'You get excited in anticipation of hooking your prey.'},
{'file_name': 'alert.png',
'name': 'Alert',
'type': 'survivor',
'description': 'Your acute senses are on high alert.'}]
```
<|assistant|>
This file appears to contain metadata about certain items, each with the following fields:
- `file_name`
- `name`
- `type` (e.g. "survivor" or "killer")
- `description`
Next, I will check for missing values in these fields.
<|assistant|>interpreter
```python
# Check for missing values in each column
missing_values = {key: sum(1 for item in data if key not in item) for key in data[0].keys()}
missing_values
```
<|observation|>
```result
{'file_name': 0, 'name': 0, 'type': 0, 'description': 0}
```
<|assistant|>
Each field has no missing values. This means that each entry is complete.
Next, we can check for any potential outliers. This usually requires deeper analysis and understanding of the data, but I can start by examining some basic statistics and distributions. To do this, we can check the value distribution of the "type" field to see if there are any unexpected categories.
<|assistant|>interpreter
```python
# Check the distribution of values in the 'type' column
type_distribution = {key: sum(1 for item in data if item['type'] == key) for key in set(item['type'] for item in data)}
type_distribution
```
<|observation|>
```result
{'survivor': 116, 'killer': 103}
```
<|assistant|>
The 'type' field has two values: 'survivor' and 'killer'. Among them, 'survivor' has 116 entries, while 'killer' has 103 entries.
From this preliminary examination, we did not find any obvious abnormal values.
If you have any specific standards or other fields that need to be checked, I can provide you with a more detailed analysis. What else do you want me to check?
<|user|>
Help me draw a heart
<|assistant|>interpreter
```python
import numpy as np
import matplotlib.pyplot as plt
# The parametric equation for the shape of a heart
def heart(t):
x = 16 * np.sin(t) ** 3
y = 13 * np.cos(t) - 5 * np.cos(2 * t) - 2 * np.cos(3 * t) - np.cos(4 * t)
return x, y
t = np.linspace(0, 2 * np.pi, 1000)
x, y = heart(t)
plt.figure(figsize=(6, 6))
plt.plot(x, y, color='red')
plt.axis('equal')
plt.axis('off')
plt.show()
```
<|observation|>
```result
[Image]
```
<|assistant|>
This is a shape of heart. I used parametric equations to describe this shape and drew it using Matplotlib. If you have any other needs or questions, please feel free to let me know.
<|user|> # End
````
================================================
FILE: README.md
================================================
# ChatGLM3
<p align="center">
📄<a href="https://arxiv.org/pdf/2406.12793" target="_blank"> Report </a> • 🤗 <a href="https://huggingface.co/THUDM/chatglm3-6b" target="_blank">HF Repo</a> • 🤖 <a href="https://modelscope.cn/models/ZhipuAI/chatglm3-6b" target="_blank">ModelScope</a> • 🟣 <a href="https://www.wisemodel.cn/models/ZhipuAI/chatglm3-6b" target="_blank">WiseModel</a> • 📔 <a href="https://lslfd0slxc.feishu.cn/wiki/WvQbwIJ9tiPAxGk8ywDck6yfnof" target="_blank">Document</a> • 🧰 <a href="https://openxlab.org.cn/models/hot/THUDM" target="_blank">OpenXLab</a> • 🐦 <a href="https://twitter.com/thukeg" target="_blank">Twitter</a><br>
</p>
<p align="center">
👋 加入我们的 <a href="https://discord.gg/fK2dz4bg" target="_blank">Discord</a> 和 <a href="resources/WECHAT.md" target="_blank">微信</a>
</p>
<p align="center">
📍在 <a href="https://www.chatglm.cn">chatglm.cn</a> 体验更大规模的 ChatGLM 模型。
</p>
[Read this in English.](./README_en.md)
📔 关于`ChatGLM3-6B` 更为详细的使用信息，可以参考
+ [ChatGLM3 开放技术文档](https://lslfd0slxc.feishu.cn/wiki/WvQbwIJ9tiPAxGk8ywDck6yfnof?from=from_copylink)
+ [Bilibili video](https://www.bilibili.com/video/BV1uC4y1J7yA)
+ [YouTube video](https://www.youtube.com/watch?v=Pw9PB6R7ORA)
## GLM-4 开源模型和API
我们已经发布最新的 **GLM-4** 模型，该模型在多个指标上有了新的突破，您可以在以下两个渠道体验我们的最新模型。
+ [GLM-4 开源模型](https://github.com/THUDM/GLM-4) 我们已经开源了 GLM-4-9B 系列模型，在各项指标的测试上有明显提升，欢迎尝试。
+ [智谱清言](https://chatglm.cn/main/detail?fr=ecology_x) 体验最新版 GLM-4，包括 **GLMs，All tools**等功能。
+ [API平台](https://open.bigmodel.cn/?utm_campaign=open&_channel_track_key=OWTVNma9) 新一代 API 平台已经上线，您可以直接在
API
平台上体验 `GLM-4-0520`、`GLM-4-air`、`GLM-4-airx`、`GLM-4-flash`、`GLM-4`、`GLM-3-Turbo`、`CharacterGLM-3`，`CogView-3`
等新模型。
其中`GLM-4`、`GLM-3-Turbo`两个模型支持了 `System Prompt`、`Function Call`、 `Retrieval`、`Web_Search`等新功能，欢迎体验。
+ [GLM-4 API 开源教程](https://github.com/MetaGLM/glm-cookbook/) GLM-4 API教程和基础应用，欢迎尝试。
API相关问题可以在本开源教程疑问，或者使用 [GLM-4 API AI助手](https://open.bigmodel.cn/shareapp/v1/?share_code=sQwt5qyqYVaNh1O_87p8O)
来获得常见问题的帮助。
-----
## ChatGLM3 介绍
**ChatGLM3** 是智谱AI和清华大学 KEG 实验室联合发布的对话预训练模型。ChatGLM3-6B 是 ChatGLM3
系列中的开源模型，在保留了前两代模型对话流畅、部署门槛低等众多优秀特性的基础上，ChatGLM3-6B 引入了如下特性：
1. **更强大的基础模型：** ChatGLM3-6B 的基础模型 ChatGLM3-6B-Base
采用了更多样的训练数据、更充分的训练步数和更合理的训练策略。在语义、数学、推理、代码、知识等不同角度的数据集上测评显示，*
*ChatGLM3-6B-Base 具有在 10B 以下的基础模型中最强的性能**。
2. **更完整的功能支持：** ChatGLM3-6B 采用了全新设计的 [Prompt 格式](PROMPT.md)
，除正常的多轮对话外。同时原生支持[工具调用](tools_using_demo/README.md)（Function Call）、代码执行（Code Interpreter）和
Agent 任务等复杂场景。
3. **更全面的开源序列：** 除了对话模型 [ChatGLM3-6B](https://huggingface.co/THUDM/chatglm3-6b)
外，还开源了基础模型 [ChatGLM3-6B-Base](https://huggingface.co/THUDM/chatglm3-6b-base)
、长文本对话模型 [ChatGLM3-6B-32K](https://huggingface.co/THUDM/chatglm3-6b-32k)
和进一步强化了对于长文本理解能力的 [ChatGLM3-6B-128K](https://huggingface.co/THUDM/chatglm3-6b-128k)。以上所有权重对学术研究**完全开放**
，在填写 [问卷](https://open.bigmodel.cn/mla/form) 进行登记后**亦允许免费商业使用**。
-----
ChatGLM3 开源模型旨在与开源社区一起推动大模型技术发展，恳请开发者和大家遵守 [开源协议](MODEL_LICENSE)
，勿将开源模型和代码及基于开源项目产生的衍生物用于任何可能给国家和社会带来危害的用途以及用于任何未经过安全评估和备案的服务。目前，本项目团队未基于
**ChatGLM3 开源模型**开发任何应用，包括网页端、安卓、苹果 iOS 及 Windows App 等应用。
尽管模型在训练的各个阶段都尽力确保数据的合规性和准确性，但由于 ChatGLM3-6B
模型规模较小，且模型受概率随机性因素影响，无法保证输出内容的准确。同时模型的输出容易被用户的输入误导。*
*本项目不承担开源模型和代码导致的数据安全、舆情风险或发生任何模型被误导、滥用、传播、不当利用而产生的风险和责任。**
## 模型列表
| Model | Seq Length | Download |
|:----------------:|:----------:|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------:|
| ChatGLM3-6B | 8k | [HuggingFace](https://huggingface.co/THUDM/chatglm3-6b) \| [ModelScope](https://modelscope.cn/models/ZhipuAI/chatglm3-6b) \| [WiseModel](https://www.wisemodel.cn/models/ZhipuAI/chatglm3-6b) \| [OpenXLab](https://openxlab.org.cn/models/detail/THUDM/chatglm3-6b) |
| ChatGLM3-6B-Base | 8k | [HuggingFace](https://huggingface.co/THUDM/chatglm3-6b-base) \| [ModelScope](https://modelscope.cn/models/ZhipuAI/chatglm3-6b-base) \| [WiseModel](https://www.wisemodel.cn/models/ZhipuAI/chatglm3-6b-base) \| [OpenXLabl](https://openxlab.org.cn/models/detail/THUDM/chatglm3-6b-base)|
| ChatGLM3-6B-32K | 32k | [HuggingFace](https://huggingface.co/THUDM/chatglm3-6b-32k) \| [ModelScope](https://modelscope.cn/models/ZhipuAI/chatglm3-6b-32k) \| [WiseModel](https://www.wisemodel.cn/models/ZhipuAI/chatglm3-6b-32k) \| [OpenXLab](https://openxlab.org.cn/models/detail/THUDM/chatglm3-6b-32k) |
| ChatGLM3-6B-128K | 128k | [HuggingFace](https://huggingface.co/THUDM/chatglm3-6b-128k) ｜ [ModelScope](https://modelscope.cn/models/ZhipuAI/chatglm3-6b-128k)\| [OpenXLab](https://openxlab.org.cn/models/detail/THUDM/chatglm3-6b-128k) |
<br> 请注意，所有模型的最新更新都会在 Huggingface 率先发布。 ModelScope 和 WiseModel 由于没有与 Huggingface 同步，需要开发人员手动更新，可能会在
Huggingface 更新后一段时间内同步更新。
## 友情链接
以下优秀开源仓库已经对 ChatGLM3-6B 模型深度支持，欢迎大家扩展学习。
推理加速：
* [chatglm.cpp](https://github.com/li-plus/chatglm.cpp): 类似 llama.cpp 的量化加速推理方案，实现笔记本上实时对话
* [ChatGLM3-TPU](https://github.com/sophgo/ChatGLM3-TPU): 采用TPU加速推理方案，在算能端侧芯片BM1684X（16T@FP16，内存16G）上实时运行约7.5 token/s
* [TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM/tree/main):
NVIDIA开发的高性能 GPU 加速推理方案，可以参考此 [步骤](./tensorrt_llm_demo/README.md) 部署 ChatGLM3-6B 模型
* [OpenVINO](https://github.com/openvinotoolkit):
Intel 开发的高性能 CPU 和 GPU 加速推理方案，可以参考此 [步骤](./Intel_device_demo/openvino_demo/README.md) 部署 ChatGLM3-6B 模型
高效微调：
* [LLaMA-Factory](https://github.com/hiyouga/LLaMA-Factory): 优秀易上手的高效微调框架。
应用框架：
* [LangChain-Chatchat](https://github.com/chatchat-space/Langchain-Chatchat): 基于 ChatGLM 等大语言模型与 Langchain
等应用框架实现，开源、可离线部署的检索增强生成(RAG)大模型知识库项目。
* [BISHENG](https://github.com/dataelement/bisheng): 开源大模型应用开发平台,赋能和加速大模型应用开发落地，帮助用户以最佳体验进入下一代应用开发模式。
* [RAGFlow](https://github.com/infiniflow/ragflow): RAGFlow 是一款基于深度文档理解构建的开源 RAG（Retrieval-Augmented Generation）引擎。可为各种规模的企业及个人提供一套精简的 RAG 工作流程，结合大语言模型（LLM）针对用户各类不同的复杂格式数据提供可靠的问答以及有理有据的引用。
## 评测结果
### 典型任务
我们选取了 8 个中英文典型数据集，在 ChatGLM3-6B (base) 版本上进行了性能测试。
| Model | GSM8K | MATH | BBH | MMLU | C-Eval | CMMLU | MBPP | AGIEval |
|------------------|:-----:|:----:|:----:|:----:|:------:|:-----:|:----:|:-------:|
| ChatGLM2-6B-Base | 32.4 | 6.5 | 33.7 | 47.9 | 51.7 | 50.0 | - | - |
| Best Baseline | 52.1 | 13.1 | 45.0 | 60.1 | 63.5 | 62.2 | 47.5 | 45.8 |
| ChatGLM3-6B-Base | 72.3 | 25.7 | 66.1 | 61.4 | 69.0 | 67.5 | 52.4 | 53.7 |
> Best Baseline 指的是截止 2023年10月27日、模型参数在 10B 以下、在对应数据集上表现最好的预训练模型，不包括只针对某一项任务训练而未保持通用能力的模型。
> 对 ChatGLM3-6B-Base 的测试中，BBH 采用 3-shot 测试，需要推理的 GSM8K、MATH 采用 0-shot CoT 测试，MBPP 采用 0-shot
> 生成后运行测例计算 Pass@1 ，其他选择题类型数据集均采用 0-shot 测试。
我们在多个长文本应用场景下对 ChatGLM3-6B-32K 进行了人工评估测试。与二代模型相比，其效果平均提升了超过
50%。在论文阅读、文档摘要和财报分析等应用中，这种提升尤为显著。此外，我们还在 LongBench 评测集上对模型进行了测试，具体结果如下表所示
| Model | 平均 | Summary | Single-Doc QA | Multi-Doc QA | Code | Few-shot | Synthetic |
|-----------------|:----:|:-------:|:-------------:|:------------:|:----:|:--------:|:---------:|
| ChatGLM2-6B-32K | 41.5 | 24.8 | 37.6 | 34.7 | 52.8 | 51.3 | 47.7 |
| ChatGLM3-6B-32K | 50.2 | 26.6 | 45.8 | 46.1 | 56.2 | 61.2 | 65 |
## 使用方式
### 环境安装
首先需要下载本仓库：
```shell
git clone https://github.com/THUDM/ChatGLM3
cd ChatGLM3
```
然后使用 pip 安装依赖：
```
pip install -r requirements.txt
```
+ 为了保证 `torch` 的版本正确，请严格按照 [官方文档](https://pytorch.org/get-started/locally/) 的说明安装。
### 综合 Demo
我们提供了一个集成以下三种功能的综合 Demo，运行方法请参考[综合 Demo](composite_demo/README.md)
- Chat: 对话模式，在此模式下可以与模型进行对话。
- Tool: 工具模式，模型除了对话外，还可以通过工具进行其他操作。
<img src="resources/tool.png" width="400">
- Code Interpreter: 代码解释器模式，模型可以在一个 Jupyter 环境中执行代码并获取结果，以完成复杂任务。
<img src="resources/heart.png" width="400">
### 代码调用
可以通过如下代码调用 ChatGLM 模型来生成对话：
```
>> from transformers import AutoTokenizer, AutoModel
>> tokenizer = AutoTokenizer.from_pretrained("THUDM/chatglm3-6b", trust_remote_code=True)
>> model = AutoModel.from_pretrained("THUDM/chatglm3-6b", trust_remote_code=True, device='cuda')
>> model = model.eval()
>> response, history = model.chat(tokenizer, "你好", history=[])
>> print(response)
你好👋!我是人工智能助手 ChatGLM3 - 6B, 很高兴见到你, 欢迎问我任何问题。
>> response, history = model.chat(tokenizer, "晚上睡不着应该怎么办", history=history)
>> print(response)
晚上睡不着可能会让你感到焦虑或不舒服, 但以下是一些可以帮助你入睡的方法:
1.制定规律的睡眠时间表: 保持规律的睡眠时间表可以帮助你建立健康的睡眠习惯, 使你更容易入睡。尽量在每天的相同时间上床, 并在同一时间起床。
2.创造一个舒适的睡眠环境: 确保睡眠环境舒适, 安静, 黑暗且温度适宜。可以使用舒适的床上用品, 并保持房间通风。
3.放松身心: 在睡前做些放松的活动, 例如泡个热水澡, 听些轻柔的音乐, 阅读一些有趣的书籍等, 有助于缓解紧张和焦虑, 使你更容易入睡。
4.避免饮用含有咖啡因的饮料: 咖啡因是一种刺激性物质, 会影响你的睡眠质量。尽量避免在睡前饮用含有咖啡因的饮料, 例如咖啡, 茶和可乐。
5.避免在床上做与睡眠无关的事情: 在床上做些与睡眠无关的事情, 例如看电影, 玩游戏或工作等, 可能会干扰你的睡眠。
6.尝试呼吸技巧: 深呼吸是一种放松技巧, 可以帮助你缓解紧张和焦虑, 使你更容易入睡。试着慢慢吸气, 保持几秒钟, 然后缓慢呼气。
如果这些方法无法帮助你入睡, 你可以考虑咨询医生或睡眠专家, 寻求进一步的建议。
```
#### 从本地加载模型
以上代码会由 `transformers`
自动下载模型实现和参数。完整的模型实现在 [Hugging Face Hub](https://huggingface.co/THUDM/chatglm3-6b)
。如果你的网络环境较差，下载模型参数可能会花费较长时间甚至失败。此时可以先将模型下载到本地，然后从本地加载。
从 Hugging Face Hub
下载模型需要先[安装Git LFS](https://docs.github.com/zh/repositories/working-with-files/managing-large-files/installing-git-large-file-storage)
，然后运行
```Shell
git clone https://huggingface.co/THUDM/chatglm3-6b
```
如果从你从 HuggingFace 下载比较慢，也可以从 [ModelScope](https://modelscope.cn/models/ZhipuAI/chatglm3-6b)
中下载。
### 模型微调
我们提供了一个微调 ChatGLM3-6B 模型的基础套件，可以用来微调 ChatGLM3-6B 模型。微调套件的使用方法请参考
[微调套件](finetune_demo/README.md)。
### 网页版对话 Demo
可以通过以下命令启动基于 Gradio 的网页版 demo：
```shell
python web_demo_gradio.py
```
可以通过以下命令启动基于 Streamlit 的网页版 demo：
```shell
streamlit run web_demo_streamlit.py
```
网页版 demo 会运行一个 Web Server，并输出地址。在浏览器中打开输出的地址即可使用。 经测试，基于 Streamlit 的网页版 Demo 会更流畅。
### 命令行对话 Demo
运行仓库中 [cli_demo.py](basic_demo/cli_demo.py)：
```shell
python cli_demo.py
```
程序会在命令行中进行交互式的对话，在命令行中输入指示并回车即可生成回复，输入 `clear` 可以清空对话历史，输入 `stop` 终止程序。
### LangChain Demo
代码实现请参考 [LangChain Demo](langchain_demo/README.md)。
#### 工具调用
关于工具调用的方法请参考 [工具调用](tools_using_demo/README.md)。
#### OpenAI API / Zhipu API Demo
我们已经推出了 OpenAI / ZhipuAI 格式的 开源模型 API 部署代码，可以作为任意基于 ChatGPT 的应用的后端。
目前，可以通过运行仓库中的 [api_server.py](openai_api_demo/api_server.py) 进行部署
```shell
cd openai_api_demo
python api_server.py
```
同时，我们也书写了一个示例代码，用来测试API调用的性能。
+ OpenAI 测试脚本：[openai_api_request.py](openai_api_demo/openai_api_request.py)
+ ZhipuAI 测试脚本：[zhipu_api_request.py](openai_api_demo/zhipu_api_request.py)
+ 使用Curl进行测试
+ chat Curl 测试
```shell
curl -X POST "http://127.0.0.1:8000/v1/chat/completions" \
-H "Content-Type: application/json" \
-d "{\"model\": \"chatglm3-6b\", \"messages\": [{\"role\": \"system\", \"content\": \"You are ChatGLM3, a large language model trained by Zhipu.AI. Follow the user's instructions carefully. Respond using markdown.\"}, {\"role\": \"user\", \"content\": \"你好，给我讲一个故事，大概100字\"}], \"stream\": false, \"max_tokens\": 100, \"temperature\": 0.8, \"top_p\": 0.8}"
````
+ Standard openai interface agent-chat Curl 测试
```shell
curl -X POST "http://127.0.0.1:8000/v1/chat/completions" \
-H "Content-Type: application/json" \
-d "{\"model\": \"chatglm3-6b\", \"messages\": [{\"role\": \"user\", \"content\": \"37乘以8加7除2等于多少？\"}], "tools": [{"name": "track", "description": "追踪指定股票的实时价格",
"parameters": {"type": "object", "properties": {"symbol": {"description": "需要追踪的股票代码"}},
"required": []}},
{"name": "Calculator", "description": "数学计算器，计算数学问题",
"parameters": {"type": "object", "properties": {"symbol": {"description": "要计算的数学公式"}},
"required": []}}
], \"stream\": true, \"max_tokens\": 100, \"temperature\": 0.8, \"top_p\": 0.8}"
````
+ Openai style custom interface agent-chat Curl 测试（你需要实现自定义的工具描述脚本openai_api_demo/tools/schema.py的内容，并且将api_server.py中AGENT_CONTROLLER指定为'true'）：
```shell
curl -X POST "http://127.0.0.1:8000/v1/chat/completions" \
-H "Content-Type: application/json" \
-d "{\"model\": \"chatglm3-6b\", \"messages\": [{\"role\": \"user\", \"content\": \"37乘以8加7除2等于多少？\"}], \"stream\": true, \"max_tokens\": 100, \"temperature\": 0.8, \"top_p\": 0.8}"
````
该接口用于openai风格的自定义工具箱的自主调度。具有调度异常的自处理回复能力，无需另外实现调度算法，用户无需api_key。
+ 使用Python进行测试
```shell
cd openai_api_demo
python openai_api_request.py
```
如果测试成功，则模型应该返回一段故事。
## 低成本部署
### 模型量化
默认情况下，模型以 FP16 精度加载，运行上述代码需要大概 13GB 显存。如果你的 GPU 显存有限，可以尝试以量化方式加载模型，使用方法如下：
```python
model = AutoModel.from_pretrained("THUDM/chatglm3-6b", trust_remote_code=True).quantize(4).cuda()
```
模型量化会带来一定的性能损失，经过测试，ChatGLM3-6B 在 4-bit 量化下仍然能够进行自然流畅的生成。
### CPU 部署
如果你没有 GPU 硬件的话，也可以在 CPU 上进行推理，但是推理速度会更慢。使用方法如下（需要大概 32GB 内存）
```python
model = AutoModel.from_pretrained("THUDM/chatglm3-6b", trust_remote_code=True).float()
```
### Mac 部署
对于搭载了 Apple Silicon 或者 AMD GPU 的 Mac，可以使用 MPS 后端来在 GPU 上运行 ChatGLM3-6B。需要参考 Apple
的 [官方说明](https://developer.apple.com/metal/pytorch) 安装 PyTorch-Nightly（正确的版本号应该是2.x.x.dev2023xxxx，而不是
2.x.x）。
目前在 MacOS 上只支持[从本地加载模型](README.md#从本地加载模型)。将代码中的模型加载改为从本地加载，并使用 mps 后端：
```python
model = AutoModel.from_pretrained("your local path", trust_remote_code=True).to('mps')
```
加载半精度的 ChatGLM3-6B 模型需要大概 13GB 内存。内存较小的机器（比如 16GB 内存的 MacBook
Pro），在空余内存不足的情况下会使用硬盘上的虚拟内存，导致推理速度严重变慢。
### 多卡部署
如果你有多张 GPU，但是每张 GPU 的显存大小都不足以容纳完整的模型，那么可以将模型切分在多张GPU上。首先安装
accelerate: `pip install accelerate`，然后即可正常加载模型。
### OpenVINO Demo
ChatGLM3-6B 已经支持使用 OpenVINO
工具包进行加速推理，在英特尔的GPU和GPU设备上有较大推理速度提升。具体使用方法请参考 [OpenVINO Demo](Intel_device_demo/openvino_demo/README.md)。
### TensorRT-LLM Demo
ChatGLM3-6B已经支持使用 TensorRT-LLM
工具包进行加速推理，模型推理速度得到多倍的提升。具体使用方法请参考 [TensorRT-LLM Demo](tensorrt_llm_demo/tensorrt_llm_cli_demo.py)
和 官方技术文档。
## 引用
如果你觉得我们的工作有帮助的话，请考虑引用下列论文。
```
@misc{glm2024chatglm,
title={ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools},
author={Team GLM and Aohan Zeng and Bin Xu and Bowen Wang and Chenhui Zhang and Da Yin and Diego Rojas and Guanyu Feng and Hanlin Zhao and Hanyu Lai and Hao Yu and Hongning Wang and Jiadai Sun and Jiajie Zhang and Jiale Cheng and Jiayi Gui and Jie Tang and Jing Zhang and Juanzi Li and Lei Zhao and Lindong Wu and Lucen Zhong and Mingdao Liu and Minlie Huang and Peng Zhang and Qinkai Zheng and Rui Lu and Shuaiqi Duan and Shudan Zhang and Shulin Cao and Shuxun Yang and Weng Lam Tam and Wenyi Zhao and Xiao Liu and Xiao Xia and Xiaohan Zhang and Xiaotao Gu and Xin Lv and Xinghan Liu and Xinyi Liu and Xinyue Yang and Xixuan Song and Xunkai Zhang and Yifan An and Yifan Xu and Yilin Niu and Yuantao Yang and Yueyan Li and Yushi Bai and Yuxiao Dong and Zehan Qi and Zhaoyu Wang and Zhen Yang and Zhengxiao Du and Zhenyu Hou and Zihan Wang},
year={2024},
eprint={2406.12793},
archivePrefix={arXiv},
primaryClass={id='cs.CL' full_name='Computation and Language' is_active=True alt_name='cmp-lg' in_archive='cs' is_general=False description='Covers natural language processing. Roughly includes material in ACM Subject Class I.2.7. Note that work on artificial languages (programming languages, logics, formal systems) that does not explicitly address natural-language issues broadly construed (natural-language processing, computational linguistics, speech, text retrieval, etc.) is not appropriate for this area.'}
}
```
================================================
FILE: README_en.md
================================================
# ChatGLM3
<p align="center">
📄<a href="https://arxiv.org/pdf/2406.12793" target="_blank"> Report </a> • 🤗 <a href="https://huggingface.co/THUDM/chatglm3-6b" target="_blank">HF Repo</a> • 🤖 <a href="https://modelscope.cn/models/ZhipuAI/chatglm3-6b" target="_blank">ModelScope</a> • 🟣 <a href="https://www.wisemodel.cn/models/ZhipuAI/chatglm3-6b" target="_blank">WiseModel</a> • 📔 <a href="https://lslfd0slxc.feishu.cn/wiki/WvQbwIJ9tiPAxGk8ywDck6yfnof" target="_blank">Document</a> • 🧰 <a href="https://openxlab.org.cn/models/hot/THUDM" target="_blank">OpenXLab</a> • 🐦 <a href="https://twitter.com/thukeg" target="_blank">Twitter</a><br>
</p>
<p align="center">
👋 Join our <a href="https://discord.gg/fK2dz4bg" target="_blank">Discord</a> and <a href="resources/WECHAT.md" target="_blank">WeChat</a>
</p>
<p align="center">
📍Experience the larger-scale ChatGLM model at <a href="https://www.chatglm.cn">chatglm.cn</a>
</p>
📔 About `ChatGLM3-6B`For more detailed usage information, please refer to:
+ [ChatGLM3 technical documentation](https://lslfd0slxc.feishu.cn/wiki/WvQbwIJ9tiPAxGk8ywDck6yfnof?from=from_copylink)
+ [Bilibili video](https://www.bilibili.com/video/BV1uC4y1J7yA)
+ [YouTube video](https://www.youtube.com/watch?v=Pw9PB6R7ORA)
## GLM-4 Open Source Model and API
We have released the latest **GLM-4** model, which has made new breakthroughs in multiple indicators. You can directly
experience our latest model in the following two channels.
+ [GLM-4 open source model](https://github.com/THUDM/GLM-4) We have open sourced the GLM-4-9B series models, which have
significantly improved the performance of various indicators. Welcome to try.
+ [Zhipu Qingyan](https://chatglm.cn/main/detail?fr=ecology_x) Experience the latest version of GLM-4, including **GLMs,
All tools** and other functions.
+ [API platform](https://open.bigmodel.cn/?utm_campaign=open&_channel_track_key=OWTVNma9) The new generation of API
platform has been launched. You can directly experience new models such
as `GLM-4-0520`, `GLM-4-air`, `GLM-4-airx`, `GLM-4-flash`, `GLM-4`, `GLM-3-Turbo`, `CharacterGLM-3`, `CogView-3` on
the API platform.
Among them, the two models `GLM-4` and `GLM-3-Turbo` support new functions such
as `System Prompt`, `Function Call`, `Retrieval`, and `Web_Search`. You are welcome to experience them.
+ [GLM4 API open source tutorial](https://github.com/MetaGLM/glm-cookbook/) GLM-4 API tutorial and basic applications,
welcome to try.
API-related questions can be asked in this open source tutorial, or
use [GLM-4 API AI Assistant](https://open.bigmodel.cn/shareapp/v1/?share_code=sQwt5qyqYVaNh1O_87p8O)
to get help with common problems.
-----
## ChatGLM3 Introduction
**ChatGLM3** is a generation of pre-trained dialogue models jointly released by Zhipu AI and Tsinghua KEG. ChatGLM3-6B
is the open-source model in the ChatGLM3 series, maintaining many excellent features of the first two generations such
as smooth dialogue and low deployment threshold, while introducing the following features:
1. **Stronger Base Model:** The base model of ChatGLM3-6B, ChatGLM3-6B-Base, adopts a more diverse training dataset,
more sufficient training steps, and a more reasonable training strategy. Evaluations on datasets from various
perspectives such as semantics, mathematics, reasoning, code, and knowledge show that **ChatGLM3-6B-Base has the
strongest performance among base models below 10B**.
2. **More Complete Function Support:** ChatGLM3-6B adopts a newly designed [Prompt format](PROMPT_en.md), supporting
multi-turn dialogues as usual. It also natively supports [tool invocation](tools_using_demo/README_en.md) (Function
Call), code execution (Code Interpreter), and Agent tasks in complex scenarios.
3. **More Comprehensive Open-source Series:** In addition to the dialogue
model [ChatGLM3-6B](https://huggingface.co/THUDM/chatglm3-6b), the basic
model [ChatGLM3-6B-Base](https://huggingface.co/THUDM/chatglm3-6b-base), the long-text dialogue
model [ChatGLM3-6B-32K](https://huggingface.co/THUDM/chatglm3-6b-32k) and further strengthens the ability to
understand long texts [ChatGLM3-6B-128K](https://huggingface.co/THUDM/chatglm3-6b-128k) have also been open-sourced.
All these weights are **fully open** for academic research, and **free commercial use is also allowed** after
registration via a [questionnaire](https://open.bigmodel.cn/mla/form).
-----
The ChatGLM3 open-source model aims to promote the development of large-model technology together with the open-source
community. Developers and everyone are earnestly requested to comply with the [open-source protocol](MODEL_LICENSE), and
not to use the open-source models, codes, and derivatives for any purposes that might harm the nation and society, and
for any services that have not been evaluated and filed for safety. Currently, no applications, including web, Android,
Apple iOS, and Windows App, have been developed based on the **ChatGLM3 open-source model** by our project team.
Although every effort has been made to ensure the compliance and accuracy of the data at various stages of model
training, due to the smaller scale of the ChatGLM3-6B model and the influence of probabilistic randomness factors, the
accuracy of output content cannot be guaranteed. The model output is also easily misled by user input. **This project
does not assume risks and liabilities caused by data security, public opinion risks, or any misleading, abuse,
dissemination, and improper use of open-source models and codes.**
## Model List
| Model | Seq Length | Download
|:----------------:|:----------:|:-----------------------------------------------------------------------------------------------------------------------------------:
| ChatGLM3-6B | 8k | [HuggingFace](https://huggingface.co/THUDM/chatglm3-6b) \| [ModelScope](https://modelscope.cn/models/ZhipuAI/chatglm3-6b)
| ChatGLM3-6B-Base | 8k | [HuggingFace](https://huggingface.co/THUDM/chatglm3-6b-base) \| [ModelScope](https://modelscope.cn/models/ZhipuAI/chatglm3-6b-base)
| ChatGLM3-6B-32K | 32k | [HuggingFace](https://huggingface.co/THUDM/chatglm3-6b-32k) \| [ModelScope](https://modelscope.cn/models/ZhipuAI/chatglm3-6b-32k)
| ChatGLM3-6B-128K | 128k | [HuggingFace](https://huggingface.co/THUDM/chatglm3-6b-128k) ｜ [ModelScope](https://modelscope.cn/models/ZhipuAI/chatglm3-6b-128k)
## Projects
The following excellent open source repositories have in-depth support for the ChatGLM3-6B model, and everyone is
welcome to expand their learning.
Inference acceleration:
* [chatglm.cpp](https://github.com/li-plus/chatglm.cpp): Real-time inference on your laptop accelerated by quantization,
similar to llama.cpp.
* [ChatGLM3-TPU](https://github.com/sophgo/ChatGLM3-TPU): Using the TPU accelerated inference solution, it runs about
7.5 token/s in real time on the end-side chip BM1684X (16T@FP16, 16G DDR).
* [TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM/tree/main): A high-performance GPU-accelerated inference
solution developed by NVIDIA, you can refer to these [steps](./tensorrt_llm_demo/README.md) to deploy ChatGLM3.
* [OpenVINO](https://github.com/openvinotoolkit): A high-performance CPU and GPU accelerated inference solution
developed by Intel, you can refer to this [step](./Intel_device_demo/openvino_demo/README.md) to deploy the
ChatGLM3-6B model
Efficient fine-tuning:
* [LLaMA-Factory](https://github.com/hiyouga/LLaMA-Factory): An excellent, easy-to-use and efficient fine-tuning
framework.
Application framework:
* [LangChain-Chatchat](https://github.com/chatchat-space/Langchain-Chatchat): Based on large language models such as
ChatGLM and application frameworks such as Langchain, open source and offline deployable retrieval enhancement
generation (RAG) large Model knowledge base project.
* [BISHENG](https://github.com/dataelement/bisheng): open-source platform for developing LLM applications. It empowers
and accelerates the development of LLM applications and helps users to enter the next generation of application
development mode with the best experience.
* [RAGFlow](https://github.com/infiniflow/ragflow): An open-source RAG (Retrieval-Augmented Generation) engine based on
deep document understanding. It offers a streamlined RAG workflow for businesses of any scale, combining LLM (Large
Language Models) to provide truthful question-answering capabilities, backed by well-founded citations from various
complex formatted data.
## Evaluation Results
### Typical Tasks
We selected 8 typical Chinese-English datasets and conducted performance tests on the ChatGLM3-6B (base) version.
| Model | GSM8K | MATH | BBH | MMLU | C-Eval | CMMLU | MBPP | AGIEval |
|------------------|:-----:|:----:|:----:|:----:|:------:|:-----:|:----:|:-------:|
| ChatGLM2-6B-Base | 32.4 | 6.5 | 33.7 | 47.9 | 51.7 | 50.0 | - | - |
| Best Baseline | 52.1 | 13.1 | 45.0 | 60.1 | 63.5 | 62.2 | 47.5 | 45.8 |
| ChatGLM3-6B-Base | 72.3 | 25.7 | 66.1 | 61.4 | 69.0 | 67.5 | 52.4 | 53.7 |
> "Best Baseline" refers to the pre-trained models that perform best on the corresponding datasets with model parameters
> below 10B, excluding models that are trained specifically for a single task and do not maintain general capabilities.
> In the tests of ChatGLM3-6B-Base, BBH used a 3-shot test, GSM8K and MATH that require inference used a 0-shot CoT
> test, MBPP used a 0-shot generation followed by running test cases to calculate Pass@1, and other multiple-choice type
> datasets all used a 0-shot test.
We have conducted manual evaluation tests on ChatGLM3-6B-32K in multiple long-text application scenarios. Compared with
the second-generation model, its effect has improved by more than 50% on average. In applications such as paper reading,
document summarization, and financial report analysis, this improvement is particularly significant. In addition, we
also tested the model on the LongBench evaluation set, and the specific results are shown in the table below.
| Model | Average | Summary | Single-Doc QA | Multi-Doc QA | Code | Few-shot | Synthetic |
|-----------------|:-------:|:-------:|:-------------:|:------------:|:----:|:--------:|:---------:|
| ChatGLM2-6B-32K | 41.5 | 24.8 | 37.6 | 34.7 | 52.8 | 51.3 | 47.7 |
| ChatGLM3-6B-32K | 50.2 | 26.6 | 45.8 | 46.1 | 56.2 | 61.2 | 65 |
## How to Use
### Environment Installation
First, you need to download this repository:
```shell
git clone https://github.com/THUDM/ChatGLM3
cd ChatGLM3
```
Then use pip to install the dependencies:
```
pip install -r requirements.txt
```
+ In order to ensure that the version of `torch` is correct, please strictly follow the instructions
of [official documentation](https://pytorch.org/get-started/locally/) for installation.
### Integrated Demo
We provide an integrated demo that incorporates the following three functionalities. Please refer
to [Integrated Demo](composite_demo/README_en.md) for how to run it.
- Chat: Dialogue mode, where you can interact with the model.
- Tool: Tool mode, where in addition to dialogue, the model can also perform other operations using tools.
- Code Interpreter: Code interpreter mode, where the model can execute code in a Jupyter environment and obtain results
to complete complex tasks.
### Usage
The ChatGLM model can be called to start a conversation using the following code:
```
>> from transformers import AutoTokenizer, AutoModel
>> tokenizer = AutoTokenizer.from_pretrained("THUDM/chatglm3-6b", trust_remote_code=True)
>> model = AutoModel.from_pretrained("THUDM/chatglm3-6b", trust_remote_code=True, device='cuda')
>> model = model.eval()
>> response, history = model.chat(tokenizer, "你好", history=[])
>> print(response)
Hello 👋! I am the AI assistant ChatGLM3-6B, nice to meet you, feel free to ask me any questions.
>> response, history = model.chat(tokenizer, "What should I do if I can't sleep at night?", history=history)
>> print(response)
Having trouble sleeping at night may make you feel anxious or uncomfortable, but here are some ways to help you fall asleep:
1. Develop a regular sleep schedule: Keeping a regular sleep schedule can help you develop healthy sleep habits and make it easier for you to fall asleep. Try to go to bed at the same time every day and get up at the same time.
2. Create a comfortable sleeping environment: Make sure the sleeping environment is comfortable, quiet, dark and at a suitable temperature. You can use comfortable bedding and keep the room ventilated.
3. Relax your body and mind: Doing some relaxing activities before bed, such as taking a hot bath, listening to some soft music, reading some interesting books, etc., can help relieve tension and anxiety and make it easier for you to fall asleep.
4. Avoid drinking caffeinated beverages: Caffeine is a stimulant that can affect your sleep quality. Try to avoid drinking caffeinated beverages such as coffee, tea and cola before bed.
5. Avoid doing things that are not related to sleep in bed: Doing things that are not related to sleep in bed, such as watching movies, playing games or working, etc., may interfere with your sleep.
6. Try breathing techniques: Deep breathing is a relaxation technique that can help you relieve tension and anxiety, making it easier for you to fall asleep. Try breathing in slowly, holding it for a few seconds, and then exhaling slowly.
If these methods don't help you fall asleep, you may consider consulting a doctor or sleep specialist for further advice.
```
#### Load Model Locally
The above code will automatically download the model implementation and parameters by `transformers`. The complete model
implementation is available on [Hugging Face Hub](https://huggingface.co/THUDM/chatglm3-6b). If your network environment
is poor, downloading model parameters might take a long time or even fail. In this case, you can first download the
model to your local machine, and then load it from there.
To download the model from Hugging Face Hub, you need
to [install Git LFS](https://docs.github.com/en/repositories/working-with-files/managing-large-files/installing-git-large-file-storage)
first, then run
```Shell
git clone https://huggingface.co/THUDM/chatglm3-6b
```
If the download from HuggingFace is slow, you can also download it
from [ModelScope](https://modelscope.cn/models/ZhipuAI/chatglm3-6b).
# Model Fine-tuning
We provide a basic fine-tuning framework for ChatGLM3-6B. You can use it to fine-tune the model on your own dataset. For
more details, please refer to [Fine-tuning Demo](finetune_demo/README_en.md).
### Web-based Dialogue Demo
You can launch a web-based demo using Gradio with the following command:
```shell
python web_demo_gradio.py
```
You can launch a web-based demo using Streamlit with the following command:
```shell
streamlit run web_demo_streamlit.py
```
The web-based demo will run a Web Server and output an address. You can use it by opening the output address in a
browser. Based on tests, the web-based demo using Streamlit runs more smoothly.
### Command Line Dialogue Demo
Run [cli_demo.py](basic_demo/cli_demo.py) in the repository:
```shell
python cli_demo.py
```
The program will interact in the command line, enter instructions in the command line and hit enter to generate a
response. Enter `clear` to clear the dialogue history, enter `stop` to terminate the program.
### OpenAI API /Zhipu API Demo
We have launched open source model API deployment code in OpenAI / ZhipuAI format, which can be used as the backend of
any ChatGPT-based application.
Currently, you can deploy by running [api_server.py](openai_api_demo/api_server.py) in the warehouse
```shell
cd openai_api_demo
python api_server.py
```
At the same time, we also wrote a sample code to test the performance of API calls.
+ OpenAI test script: [openai_api_request.py](openai_api_demo/openai_api_request.py)
+ ZhipuAI test script: [zhipu_api_request.py](openai_api_demo/zhipu_api_request.py)
+ Test with Curl
+ chat Curl test
```shell
curl -X POST "http://127.0.0.1:8000/v1/chat/completions" \
-H "Content-Type: application/json" \
-d "{\"model\": \"chatglm3-6b\", \"messages\": [{\"role\": \"system\", \"content\": \"You are ChatGLM3, a large language model trained by Zhipu.AI. Follow the user's instructions carefully. Respond using markdown.\"}, {\"role\": \"user\", \"content\": \"你好，给我讲一个故事，大概100字\"}], \"stream\": false, \"max_tokens\": 100, \"temperature\": 0.8, \"top_p\": 0.8}"
````
+ Standard openai interface agent-chat Curl test
```shell
curl -X POST "http://127.0.0.1:8000/v1/chat/completions" \
-H "Content-Type: application/json" \
-d "{\"model\": \"chatglm3-6b\", \"messages\": [{\"role\": \"user\", \"content\": \"37乘以8加7除2等于多少？\"}], "tools": [{"name": "track", "description": "追踪指定股票的实时价格",
"parameters": {"type": "object", "properties": {"symbol": {"description": "需要追踪的股票代码"}},
"required": []}},
{"name": "Calculator", "description": "数学计算器，计算数学问题",
"parameters": {"type": "object", "properties": {"symbol": {"description": "要计算的数学公式"}},
"required": []}}
], \"stream\": true, \"max_tokens\": 100, \"temperature\": 0.8, \"top_p\": 0.8}"
````
+ Openai style custom interface agent-chat Curl test (You need to implement the contents of the custom tool description
script openai_api_demo/tools/schema.py, and specify AGENT_CONTROLLER in api_server.py as 'true')：
```shell
curl -X POST "http://127.0.0.1:8000/v1/chat/completions" \
-H "Content-Type: application/json" \
-d "{\"model\": \"chatglm3-6b\", \"messages\": [{\"role\": \"user\", \"content\": \"37乘以8加7除2等于多少？\"}], \"stream\": true, \"max_tokens\": 100, \"temperature\": 0.8, \"top_p\": 0.8}"
````
This interface is used for autonomous scheduling of OpenAI-style custom toolboxes. It has the ability to self-process
and respond to scheduling exceptions, without the need to implement additional scheduling algorithms, and users do not
need an api_key.
+ Testing with Python
```shell
cd openai_api_demo
python openai_api_request.py
```
If the test is successful, the model should return a story.
### Tool Invocation
For methods of tool invocation, please refer to [Tool Invocation](tools_using_demo/README_en.md).
## Low-Cost Deployment
### Model Quantization
By default, the model is loaded with FP16 precision, running the above code requires about 13GB of VRAM. If your GPU's
VRAM is limited, you can try loading the model quantitatively, as follows:
```python
model = AutoModel.from_pretrained("THUDM/chatglm3-6b", trust_remote_code=True).quantize(4).cuda()
```
Model quantization will bring some performance loss. Through testing, ChatGLM3-6B can still perform natural and smooth
generation under 4-bit quantization.
### CPU Deployment
If you don't have GPU hardware, you can also run inference on the CPU, but the inference speed will be slower. The usage
is as follows (requires about 32GB of memory):
```python
model = AutoModel.from_pretrained("THUDM/chatglm3-6b", trust_remote_code=True).float()
```
### Mac Deployment
For Macs equipped with Apple Silicon or AMD GPUs, the MPS backend can be used to run ChatGLM3-6B on the GPU. Refer to
Apple's [official instructions](https://developer.apple.com/metal/pytorch) to install PyTorch-Nightly (the correct
version number should be 2.x.x.dev2023xxxx, not 2.x.x).
Currently, only [loading the model locally](README_en.md#load-model-locally) is supported on MacOS. Change the model
loading in the code to load locally and use the MPS backend:
```python
model = AutoModel.from_pretrained("your local path", trust_remote_code=True).to('mps')
```
Loading the half-precision ChatGLM3-6B model requires about 13GB of memory. Machines with smaller memory (such as a 16GB
memory MacBook Pro) will use virtual memory on the hard disk when there is insufficient free memory, resulting in a
significant slowdown in inference speed.
### Multi-GPU Deployment
If you have multiple GPUs, but each GPU's VRAM size is not enough to accommodate the complete model, then the model can
be split across multiple GPUs. First, install accelerate: `pip install accelerate`, and then load the model as usual.
### OpenVINO Demo
ChatGLM3-6B already supports the use of OpenVINO
The toolkit accelerates inference and has a greater inference speed improvement on Intel's GPUs and GPU devices. For
specific usage, please refer to [OpenVINO Demo](Intel_device_demo/openvino_demo/README.md).
### TensorRT-LLM Demo
ChatGLM3-6B now supports accelerated inference using the TensorRT-LLM toolkit, significantly improving model inference
speed. For specific usage, please refer to the [TensorRT-LLM Demo](tensorrt_llm_demo/tensorrt_llm_cli_demo.py) and the
official technical documentation.
## Citation
If you find our work helpful, please consider citing the following papers.
```
@misc{glm2024chatglm,
title={ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools},
author={Team GLM and Aohan Zeng and Bin Xu and Bowen Wang and Chenhui Zhang and Da Yin and Diego Rojas and Guanyu Feng and Hanlin Zhao and Hanyu Lai and Hao Yu and Hongning Wang and Jiadai Sun and Jiajie Zhang and Jiale Cheng and Jiayi Gui and Jie Tang and Jing Zhang and Juanzi Li and Lei Zhao and Lindong Wu and Lucen Zhong and Mingdao Liu and Minlie Huang and Peng Zhang and Qinkai Zheng and Rui Lu and Shuaiqi Duan and Shudan Zhang and Shulin Cao and Shuxun Yang and Weng Lam Tam and Wenyi Zhao and Xiao Liu and Xiao Xia and Xiaohan Zhang and Xiaotao Gu and Xin Lv and Xinghan Liu and Xinyi Liu and Xinyue Yang and Xixuan Song and Xunkai Zhang and Yifan An and Yifan Xu and Yilin Niu and Yuantao Yang and Yueyan Li and Yushi Bai and Yuxiao Dong and Zehan Qi and Zhaoyu Wang and Zhen Yang and Zhengxiao Du and Zhenyu Hou and Zihan Wang},
year={2024},
eprint={2406.12793},
archivePrefix={arXiv},
primaryClass={id='cs.CL' full_name='Computation and Language' is_active=True alt_name='cmp-lg' in_archive='cs' is_general=False description='Covers natural language processing. Roughly includes material in ACM Subject Class I.2.7. Note that work on artificial languages (programming languages, logics, formal systems) that does not explicitly address natural-language issues broadly construed (natural-language processing, computational linguistics, speech, text retrieval, etc.) is not appropriate for this area.'}
}
```
================================================
FILE: basic_demo/cli_batch_request_demo.py
================================================
import os
from typing import Optional, Union
from transformers import AutoModel, AutoTokenizer, LogitsProcessorList
MODEL_PATH = os.environ.get('MODEL_PATH', 'THUDM/chatglm3-6b')
TOKENIZER_PATH = os.environ.get("TOKENIZER_PATH", MODEL_PATH)
tokenizer = AutoTokenizer.from_pretrained(TOKENIZER_PATH, trust_remote_code=True)
model = AutoModel.from_pretrained(MODEL_PATH, trust_remote_code=True, device_map="auto").eval()
def batch(
model,
tokenizer,
prompts: Union[str, list[str]],
max_length: int = 8192,
num_beams: int = 1,
do_sample: bool = True,
top_p: float = 0.8,
temperature: float = 0.8,
logits_processor: Optional[LogitsProcessorList] = LogitsProcessorList(),
):
tokenizer.encode_special_tokens = True
if isinstance(prompts, str):
prompts = [prompts]
batched_inputs = tokenizer(prompts, return_tensors="pt", padding="longest")
batched_inputs = batched_inputs.to(model.device)
eos_token_id = [
tokenizer.eos_token_id,
tokenizer.get_command("<|user|>"),
tokenizer.get_command("<|assistant|>"),
]
gen_kwargs = {
"max_length": max_length,
"num_beams": num_beams,
"do_sample": do_sample,
"top_p": top_p,
"temperature": temperature,
"logits_processor": logits_processor,
"eos_token_id": eos_token_id,
}
batched_outputs = model.generate(**batched_inputs, **gen_kwargs)
batched_response = []
for input_ids, output_ids in zip(batched_inputs.input_ids, batched_outputs):
decoded_text = tokenizer.decode(output_ids[len(input_ids):])
batched_response.append(decoded_text.strip())
return batched_response
def main(batch_queries):
gen_kwargs = {
"max_length": 2048,
"do_sample": True,
"top_p": 0.8,
"temperature": 0.8,
"num_beams": 1,
}
batch_responses = batch(model, tokenizer, batch_queries, **gen_kwargs)
return batch_responses
if __name__ == "__main__":
batch_queries = [
"<|user|>\n讲个故事\n<|assistant|>",
"<|user|>\n讲个爱情故事\n<|assistant|>",
"<|user|>\n讲个开心故事\n<|assistant|>",
"<|user|>\n讲个睡前故事\n<|assistant|>",
"<|user|>\n讲个励志的故事\n<|assistant|>",
"<|user|>\n讲个少壮不努力的故事\n<|assistant|>",
"<|user|>\n讲个青春校园恋爱故事\n<|assistant|>",
"<|user|>\n讲个工作故事\n<|assistant|>",
"<|user|>\n讲个旅游的故事\n<|assistant|>",
]
batch_responses = main(batch_queries)
for response in batch_responses:
print("=" * 10)
print(response)
================================================
FILE: basic_demo/cli_demo.py
================================================
import os
import platform
from transformers import AutoTokenizer, AutoModel
MODEL_PATH = os.environ.get('MODEL_PATH', 'THUDM/chatglm3-6b')
TOKENIZER_PATH = os.environ.get("TOKENIZER_PATH", MODEL_PATH)
tokenizer = AutoTokenizer.from_pretrained(TOKENIZER_PATH, trust_remote_code=True)
model = AutoModel.from_pretrained(MODEL_PATH, trust_remote_code=True, device_map="auto").eval()
# add .quantize(bits=4, device="cuda").cuda() before .eval() to use int4 model
# must use cuda to load int4 model
os_name = platform.system()
clear_command = 'cls' if os_name == 'Windows' else 'clear'
stop_stream = False
welcome_prompt = "欢迎使用 ChatGLM3-6B 模型，输入内容即可进行对话，clear 清空对话历史，stop 终止程序"
def build_prompt(history):
prompt = welcome_prompt
for query, response in history:
prompt += f"\n\n用户：{query}"
prompt += f"\n\nChatGLM3-6B：{response}"
return prompt
def main():
past_key_values, history = None, []
global stop_stream
print(welcome_prompt)
while True:
query = input("\n用户：")
if query.strip() == "stop":
break
if query.strip() == "clear":
past_key_values, history = None, []
os.system(clear_command)
print(welcome_prompt)
continue
print("\nChatGLM：", end="")
current_length = 0
for response, history, past_key_values in model.stream_chat(tokenizer, query, history=history, top_p=1,
temperature=0.01,
past_key_values=past_key_values,
return_past_key_values=True):
if stop_stream:
stop_stream = False
break
else:
print(response[current_length:], end="", flush=True)
current_length = len(response)
print("")
if __name__ == "__main__":
main()
================================================
FILE: basic_demo/cli_demo_bad_word_ids.py
================================================
"""
This script demonstrates how to use the `bad_words_ids` argument in the context of a conversational AI model to filter out unwanted words or phrases from the model's responses. It's designed to showcase a fundamental method of content moderation within AI-generated text, particularly useful in scenarios where maintaining the decorum of the conversation is essential.
Usage:
- Interact with the model by typing queries. The model will generate responses while avoiding the specified bad words.
- Use 'clear' to clear the conversation history and 'stop' to exit the program.
Requirements:
- The script requires the Transformers library and an appropriate model checkpoint.
Note: The `bad_words_ids` feature is an essential tool for controlling the output of language models, particularly in user-facing applications where content moderation is crucial.
"""
import os
import platform
from transformers import AutoTokenizer, AutoModel
MODEL_PATH = os.environ.get('MODEL_PATH', 'THUDM/chatglm3-6b')
TOKENIZER_PATH = os.environ.get("TOKENIZER_PATH", MODEL_PATH)
tokenizer = AutoTokenizer.from_pretrained(TOKENIZER_PATH, trust_remote_code=True)
model = AutoModel.from_pretrained(MODEL_PATH, trust_remote_code=True, device_map="auto").eval()
os_name = platform.system()
clear_command = 'cls' if os_name == 'Windows' else 'clear'
stop_stream = False
welcome_prompt = "欢迎使用 ChatGLM3-6B 模型，输入内容即可进行对话，clear 清空对话历史，stop 终止程序"
# probability tensor contains either `inf`, `nan` or element < 0
bad_words = ["你好", "ChatGLM"]
bad_word_ids = [tokenizer.encode(bad_word, add_special_tokens=False) for bad_word in bad_words]
def build_prompt(history):
prompt = welcome_prompt
for query, response in history:
prompt += f"\n\n用户：{query}"
prompt += f"\n\nChatGLM3-6B：{response}"
return prompt
def main():
past_key_values, history = None, []
global stop_stream
print(welcome_prompt)
while True:
query = input("\n用户：")
if query.strip().lower() == "stop":
break
if query.strip().lower() == "clear":
past_key_values, history = None, []
os.system(clear_command)
print(welcome_prompt)
continue
# Attempt to generate a response
try:
print("\nChatGLM：", end="")
current_length = 0
response_generated = False
for response, history, past_key_values in model.stream_chat(
tokenizer, query, history=history, top_p=1,
temperature=0.01,
past_key_values=past_key_values,
return_past_key_values=True,
bad_words_ids=bad_word_ids # assuming this is implemented correctly
):
response_generated = True
# Check if the response contains any bad words
if any(bad_word in response for bad_word in bad_words):
print("我的回答涉嫌了 bad word")
break # Break the loop if a bad word is detected
# Otherwise, print the generated response
print(response[current_length:], end="", flush=True)
current_length = len(response)
if not response_generated:
print("没有生成任何回答。")
except RuntimeError as e:
print(f"生成文本时发生错误：{e}，这可能是涉及到设定的敏感词汇")
print("")
if __name__ == "__main__":
main()
================================================
FILE: basic_demo/web_demo_gradio.py
================================================
"""
This script creates an interactive web demo for the ChatGLM3-6B model using Gradio,
a Python library for building quick and easy UI components for machine learning models.
It's designed to showcase the capabilities of the ChatGLM3-6B model in a user-friendly interface,
allowing users to interact with the model through a chat-like interface.
Usage:
- Run the script to start the Gradio web server.
- Interact with the model by typing questions and receiving responses.
Requirements:
- Gradio (required for 4.13.0 and later, 3.x is not support now) should be installed.
Note: The script includes a modification to the Chatbot's postprocess method to handle markdown to HTML conversion,
ensuring that the chat interface displays formatted text correctly.
"""
import os
import gradio as gr
import torch
from threading import Thread
from typing import Union, Annotated
from pathlib import Path
from peft import AutoPeftModelForCausalLM, PeftModelForCausalLM
from transformers import (
AutoModelForCausalLM,
AutoTokenizer,
PreTrainedModel,
PreTrainedTokenizer,
PreTrainedTokenizerFast,
StoppingCriteria,
StoppingCriteriaList,
TextIteratorStreamer
)
import socket
ModelType = Union[PreTrainedModel, PeftModelForCausalLM]
TokenizerType = Union[PreTrainedTokenizer, PreTrainedTokenizerFast]
#MODEL_PATH = os.environ.get('MODEL_PATH', 'THUDM/chatglm3-6b')
MODEL_PATH = 'chatglm3-6b'
TOKENIZER_PATH = os.environ.get("TOKENIZER_PATH", MODEL_PATH)
def _resolve_path(path: Union[str, Path]) -> Path:
return Path(path).expanduser().resolve()
def load_model_and_tokenizer(
model_dir: Union[str, Path], trust_remote_code: bool = True
) -> tuple[ModelType, TokenizerType]:
model_dir = _resolve_path(model_dir)
if (model_dir / 'adapter_config.json').exists():
model = AutoPeftModelForCausalLM.from_pretrained(
model_dir, trust_remote_code=trust_remote_code, device_map='auto'
)
tokenizer_dir = model.peft_config['default'].base_model_name_or_path
else:
model = AutoModelForCausalLM.from_pretrained(
model_dir, trust_remote_code=trust_remote_code, device_map='auto'
)
tokenizer_dir = model_dir
tokenizer = AutoTokenizer.from_pretrained(
tokenizer_dir, trust_remote_code=trust_remote_code
)
return model, tokenizer
model, tokenizer = load_model_and_tokenizer(MODEL_PATH, trust_remote_code=True)
class StopOnTokens(StoppingCriteria):
def __call__(self, input_ids: torch.LongTensor, scores: torch.FloatTensor, **kwargs) -> bool:
stop_ids = [0, 2]
for stop_id in stop_ids:
if input_ids[0][-1] == stop_id:
return True
return False
def parse_text(text):
lines = text.split("\n")
lines = [line for line in lines if line != ""]
count = 0
for i, line in enumerate(lines):
if "```" in line:
count += 1
items = line.split('`')
if count % 2 == 1:
lines[i] = f'<pre><code class="language-{items[-1]}">'
else:
lines[i] = f'<br></code></pre>'
else:
if i > 0:
if count % 2 == 1:
line = line.replace("`", "\`")
line = line.replace("<", "<")
line = line.replace(">", ">")
line = line.replace(" ", " ")
line = line.replace("*", "*")
line = line.replace("_", "_")
line = line.replace("-", "-")
line = line.replace(".", ".")
line = line.replace("!", "!")
line = line.replace("(", "(")
line = line.replace(")", ")")
line = line.replace("$", "$")
lines[i] = "<br>" + line
text = "".join(lines)
return text
def predict(history, max_length, top_p, temperature, system_prompt):
stop = StopOnTokens()
messages = []
if(system_prompt!=""):
messages.append({"role": "system", "content": system_prompt})
for idx, (user_msg, model_msg) in enumerate(history):
if idx == len(history) - 1 and not model_msg:
messages.append({"role": "user", "content": user_msg})
break
if user_msg:
messages.append({"role": "user", "content": user_msg})
if model_msg:
messages.append({"role": "assistant", "content": model_msg})
print("\n\n====conversation====\n", messages)
model_inputs = tokenizer.apply_chat_template(messages,
add_generation_prompt=True,
tokenize=True,
return_tensors="pt").to(next(model.parameters()).device)
streamer = TextIteratorStreamer(tokenizer, timeout=60, skip_prompt=True, skip_special_tokens=True)
generate_kwargs = {
"input_ids": model_inputs,
"streamer": streamer,
"max_new_tokens": max_length,
"do_sample": True,
"top_p": top_p,
"temperature": temperature,
"stopping_criteria": StoppingCriteriaList([stop]),
"repetition_penalty": 1.2,
}
t = Thread(target=model.generate, kwargs=generate_kwargs)
t.start()
for new_token in streamer:
if new_token != '':
history[-1][1] += new_token
yield history
with gr.Blocks(title="ChatGLM") as demo:
gr.Markdown("## ChatGLM3-6B")
with gr.Row():
with gr.Column(scale=4):
chatbot = gr.Chatbot(layout="panel")
with gr.Column(scale=12):
user_input = gr.Textbox(show_label=False, placeholder="Input to chat...", lines=3, container=False)
with gr.Column(min_width=32, scale=1):
submitBtn = gr.Button("Submit", variant="primary")
with gr.Column(scale=1):
emptyBtn = gr.Button("Clear History")
max_length = gr.Slider(0, 32768, value=16384, step=1.0, label="Maximum length", interactive=True)
top_p = gr.Slider(0, 1, value=0.8, step=0.01, label="Top P", interactive=True)
temperature = gr.Slider(0.01, 1, value=0.6, step=0.01, label="Temperature", interactive=True)
gr.HTML("""<span>System Prompt</span>""")
system_prompt = gr.Textbox(show_label=False, placeholder="System Prompt", lines=6, container=False)
def user(query, history):
return "", history + [[parse_text(query), ""]]
submitBtn.click(user, [user_input, chatbot], [user_input, chatbot], queue=False).then(
predict, [chatbot, max_length, top_p, temperature, system_prompt], chatbot
)
emptyBtn.click(lambda: None, None, chatbot, queue=False)
demo.queue()
demo.launch(server_name=socket.gethostbyname(socket.gethostname()), server_port=7870, inbrowser=True, share=False)
================================================
FILE: basic_demo/web_demo_streamlit.py
================================================
"""
This script is a simple web demo based on Streamlit, showcasing the use of the ChatGLM3-6B model. For a more comprehensive web demo,
it is recommended to use 'composite_demo'.
Usage:
- Run the script using Streamlit: `streamlit run web_demo_streamlit.py`
- Adjust the model parameters from the sidebar.
- Enter questions in the chat input box and interact with the ChatGLM3-6B model.
Note: Ensure 'streamlit' and 'transformers' libraries are installed and the required model checkpoints are available.
"""
import os
import streamlit as st
import torch
from transformers import AutoModel, AutoTokenizer
MODEL_PATH = os.environ.get('MODEL_PATH', 'THUDM/chatglm3-6b')
TOKENIZER_PATH = os.environ.get("TOKENIZER_PATH", MODEL_PATH)
st.set_page_config(
page_title="ChatGLM3-6B Streamlit Simple Demo",
page_icon=":robot:",
layout="wide"
)
@st.cache_resource
def get_model():
tokenizer = AutoTokenizer.from_pretrained(TOKENIZER_PATH, trust_remote_code=True)
model = AutoModel.from_pretrained(MODEL_PATH, trust_remote_code=True, device_map="auto").eval()
return tokenizer, model
# 加载Chatglm3的model和tokenizer
tokenizer, model = get_model()
if "history" not in st.session_state:
st.session_state.history = []
if "past_key_values" not in st.session_state:
st.session_state.past_key_values = None
max_length = st.sidebar.slider("max_length", 0, 32768, 8192, step=1)
top_p = st.sidebar.slider("top_p", 0.0, 1.0, 0.8, step=0.01)
temperature = st.sidebar.slider("temperature", 0.0, 1.0, 0.6, step=0.01)
buttonClean = st.sidebar.button("清理会话历史", key="clean")
if buttonClean:
st.session_state.history = []
st.session_state.past_key_values = None
if torch.cuda.is_available():
torch.cuda.empty_cache()
st.rerun()
for i, message in enumerate(st.session_state.history):
if message["role"] == "user":
with st.chat_message(name="user", avatar="user"):
st.markdown(message["content"])
else:
with st.chat_message(name="assistant", avatar="assistant"):
st.markdown(message["content"])
with st.chat_message(name="user", avatar="user"):
input_placeholder = st.empty()
with st.chat_message(name="assistant", avatar="assistant"):
message_placeholder = st.empty()
prompt_text = st.chat_input("请输入您的问题")
if prompt_text:
input_placeholder.markdown(prompt_text)
history = st.session_state.history
past_key_values = st.session_state.past_key_values
for response, history, past_key_values in model.stream_chat(
tokenizer,
prompt_text,
history,
past_key_values=past_key_values,
max_length=max_length,
top_p=top_p,
temperature=temperature,
return_past_key_values=True,
):
message_placeholder.markdown(response)
st.session_state.history = history
st.session_state.past_key_values = past_key_values
================================================
FILE: composite_demo/.streamlit/config.toml
================================================
[theme]
font = "monospace"
================================================
FILE: composite_demo/README.md
================================================
# ChatGLM3 Web Demo
## 安装
我们建议通过 [Conda](https://docs.conda.io/en/latest/) 进行环境管理。
执行以下命令新建一个 conda 环境并安装所需依赖：
```bash
conda create -n chatglm3-demo python=3.10
conda activate chatglm3-demo
pip install -r requirements.txt
```
请注意，本项目需要 Python 3.10 或更高版本。
此外，使用 Code Interpreter 还需要安装 Jupyter 内核：
```bash
ipython kernel install --name chatglm3-demo --user
```
## 运行
运行以下命令在本地加载模型并启动 demo：
```bash
streamlit run main.py
```
之后即可从命令行中看到 demo 的地址，点击即可访问。初次访问需要下载并加载模型，可能需要花费一定时间。
如果已经在本地下载了模型，可以通过 `export MODEL_PATH=/path/to/model` 来指定从本地加载模型。如果需要自定义 Jupyter 内核，可以通过 `export IPYKERNEL=<kernel_name>` 来指定。
## 使用
ChatGLM3 Demo 拥有三种模式：
- Chat: 对话模式，在此模式下可以与模型进行对话。
- Tool: 工具模式，模型除了对话外，还可以通过工具进行其他操作。
- Code Interpreter: 代码解释器模式，模型可以在一个 Jupyter 环境中执行代码并获取结果，以完成复杂任务。
### 对话模式
对话模式下，用户可以直接在侧边栏修改 top_p, temperature, System Prompt 等参数来调整模型的行为。例如
### 工具模式
可以通过在 `tool_registry.py` 中注册新的工具来增强模型的能力。只需要使用 `@register_tool` 装饰函数即可完成注册。对于工具声明，函数名称即为工具的名称，函数 docstring 即为工具的说明；对于工具的参数，使用 `Annotated[typ: type, description: str, required: bool]` 标注参数的类型、描述和是否必须。
例如，`get_weather` 工具的注册如下：
```python
@register_tool
def get_weather(
city_name: Annotated[str, 'The name of the city to be queried', True],
) -> str:
"""
Get the weather for `city_name` in the following week
"""
...
```
此外，你也可以在页面中通过 `Manual mode` 进入手动模式，在这一模式下你可以通过 YAML 来直接指定工具列表，但你需要手动将工具的输出反馈给模型。
### 代码解释器模式
由于拥有代码执行环境，此模式下的模型能够执行更为复杂的任务，例如绘制图表、执行符号运算等等。模型会根据对任务完成情况的理解自动地连续执行多个代码块，直到任务完成。因此，在这一模式下，你只需要指明希望模型执行的任务即可。
例如，我们可以让 ChatGLM3 画一个爱心：
### 额外技巧
- 在模型生成文本时，可以通过页面右上角的 `Stop` 按钮进行打断。
- 刷新页面即可清空对话记录。
# Enjoy!
================================================
FILE: composite_demo/README_en.md
================================================
# ChatGLM3 Web Demo
## Installation
We recommend managing environments through [Conda](https://docs.conda.io/en/latest/).
Execute the following commands to create a new conda environment and install the necessary dependencies:
```bash
conda create -n chatglm3-demo python=3.10
conda activate chatglm3-demo
pip install -r requirements.txt
```
Please note that this project requires Python 3.10 or higher.
Additionally, installing the Jupyter kernel is required for using the Code Interpreter:
```bash
ipython kernel install --name chatglm3-demo --user
```
## Execution
Run the following command to load the model locally and start the demo:
```bash
streamlit run main.py
```
Afterward, the address of the demo can be seen from the command line; click to access. The first visit requires the download and loading of the model, which may take some time.
If the model has already been downloaded locally, you can specify to load the model locally through `export MODEL_PATH=/path/to/model`. If you need to customize the Jupyter kernel, you can specify it through `export IPYKERNEL=<kernel_name>`.
## Usage
ChatGLM3 Demo has three modes:
- Chat: Dialogue mode, where you can interact with the model.
- Tool: Tool mode, where the model, in addition to dialogue, can perform other operations through tools.
- Code Interpreter: Code interpreter mode, where the model can execute code in a Jupyter environment and obtain results to complete complex tasks.
### Dialogue Mode
In dialogue mode, users can directly modify parameters such as top_p, temperature, System Prompt in the sidebar to adjust the behavior of the model. For example,
### Tool Mode
You can enhance the model's capabilities by registering new tools in `tool_registry.py`. Just use the `@register_tool` decorator to complete the registration. For tool declarations, the function name is the name of the tool, and the function docstring is the description of the tool; for tool parameters, use `Annotated[typ: type, description: str, required: bool]` to annotate the type, description, and whether it is necessary of the parameters.
For example, the registration of the `get_weather` tool is as follows:
```python
@register_tool
def get_weather(
city_name: Annotated[str, 'The name of the city to be queried', True],
) -> str:
"""
Get the weather for `city_name` in the following week
"""
...
```
Additionally, you can enter the manual mode through `Manual mode` on the page. In this mode, you can directly specify the tool list through YAML, but you need to manually feed back the tool's output to the model.
### Code Interpreter Mode
Due to having a code execution environment, the model in this mode can perform more complex tasks, such as drawing charts, performing symbolic operations, etc. The model will automatically execute multiple code blocks in succession based on its understanding of the task completion status until the task is completed. Therefore, in this mode, you only need to specify the task you want the model to perform.
For example, we can ask ChatGLM3 to draw a heart:
### Additional Tips
- While the model is generating text, it can be interrupted by the `Stop` button at the top right corner of the page.
- Refreshing the page will clear the dialogue history.
# Enjoy!
================================================
FILE: composite_demo/client.py
================================================
from __future__ import annotations
import os
import streamlit as st
import torch
from collections.abc import Iterable
from typing import Any, Protocol
from huggingface_hub.inference._text_generation import TextGenerationStreamResponse, Token
from transformers import AutoModel, AutoTokenizer, AutoConfig
from transformers.generation.logits_process import LogitsProcessor
from transformers.generation.utils import LogitsProcessorList
from conversation import Conversation
TOOL_PROMPT = 'Answer the following questions as best as you can. You have access to the following tools:'
MODEL_PATH = os.environ.get('MODEL_PATH', 'THUDM/chatglm3-6b')
PT_PATH = os.environ.get('PT_PATH', None)
PRE_SEQ_LEN = int(os.environ.get("PRE_SEQ_LEN", 128))
TOKENIZER_PATH = os.environ.get("TOKENIZER_PATH", MODEL_PATH)
@st.cache_resource
def get_client() -> Client:
client = HFClient(MODEL_PATH, TOKENIZER_PATH, PT_PATH)
return client
class Client(Protocol):
def generate_stream(self,
system: str | None,
tools: list[dict] | None,
history: list[Conversation],
**parameters: Any
) -> Iterable[TextGenerationStreamResponse]:
...
def stream_chat(
self, tokenizer, query: str,
history: list[tuple[str, str]] = None,
role: str = "user",
past_key_values=None,
max_new_tokens: int = 256,
do_sample=True, top_p=0.8,
temperature=0.8,
repetition_penalty=1.0,
length_penalty=1.0, num_beams=1,
logits_processor=None,
return_past_key_values=False,
**kwargs
):
class InvalidScoreLogitsProcessor(LogitsProcessor):
def __call__(self, input_ids: torch.LongTensor, scores: torch.FloatTensor) -> torch.FloatTensor:
if torch.isnan(scores).any() or torch.isinf(scores).any():
scores.zero_()
scores[..., 5] = 5e4
return scores
if history is None:
history = []
print("\n== Input ==\n", query)
print("\n==History==\n", history)
if logits_processor is None:
logits_processor = LogitsProcessorList()
logits_processor.append(InvalidScoreLogitsProcessor())
eos_token_id = [tokenizer.eos_token_id, tokenizer.get_command("<|user|>"),
tokenizer.get_command("<|observation|>")]
gen_kwargs = {"max_new_tokens": max_new_tokens,
"do_sample": do_sample,
"top_p": top_p,
"temperature": temperature,
"logits_processor": logits_processor,
"repetition_penalty": repetition_penalty,
"length_penalty": length_penalty,
"num_beams": num_beams,
**kwargs
}
if past_key_values is None:
inputs = tokenizer.build_chat_input(query, history=history, role=role)
else:
inputs = tokenizer.build_chat_input(query, role=role)
inputs = inputs.to(self.device)
if past_key_values is not None:
past_length = past_key_values[0][0].shape[0]
if self.transformer.pre_seq_len is not None:
past_length -= self.transformer.pre_seq_len
inputs.position_ids += past_length
attention_mask = inputs.attention_mask
attention_mask = torch.cat((attention_mask.new_ones(1, past_length), attention_mask), dim=1)
inputs['attention_mask'] = attention_mask
history.append({"role": role, "content": query})
input_sequence_length = inputs['input_ids'].shape[1]
if input_sequence_length + max_new_tokens >= self.config.seq_length:
yield "Current input sequence length {} plus max_new_tokens {} is too long. The maximum model sequence length is {}. You may adjust the generation parameter to enable longer chat history.".format(
input_sequence_length, max_new_tokens, self.config.seq_length
), history
return
if input_sequence_length > self.config.seq_length:
yield "Current input sequence length {} exceeds maximum model sequence length {}. Unable to generate tokens.".format(
input_sequence_length, self.config.seq_length
), history
return
for outputs in self.stream_generate(**inputs, past_key_values=past_key_values,
eos_token_id=eos_token_id, return_past_key_values=return_past_key_values,
**gen_kwargs):
if return_past_key_values:
outputs, past_key_values = outputs
outputs = outputs.tolist()[0][len(inputs["input_ids"][0]):]
response = tokenizer.decode(outputs)
if response and response[-1] != "�":
new_history = history
if return_past_key_values:
yield response, new_history, past_key_values
else:
yield response, new_history
class HFClient(Client):
def __init__(self, model_path: str, tokenizer_path: str, pt_checkpoint: str = None):
self.model_path = model_path
self.tokenizer = AutoTokenizer.from_pretrained(tokenizer_path, trust_remote_code=True)
if pt_checkpoint is not None and os.path.exists(pt_checkpoint):
config = AutoConfig.from_pretrained(
model_path,
trust_remote_code=True,
pre_seq_len=PRE_SEQ_LEN
)
self.model = AutoModel.from_pretrained(
model_path,
trust_remote_code=True,
config=config,
device_map="auto").eval()
# add .quantize(bits=4, device="cuda").cuda() before .eval() and remove device_map="auto" to use int4 model
# must use cuda to load int4 model
prefix_state_dict = torch.load(os.path.join(pt_checkpoint, "pytorch_model.bin"))
new_prefix_state_dict = {}
for k, v in prefix_state_dict.items():
if k.startswith("transformer.prefix_encoder."):
new_prefix_state_dict[k[len("transformer.prefix_encoder."):]] = v
print("Loaded from pt checkpoints", new_prefix_state_dict.keys())
self.model.transformer.prefix_encoder.load_state_dict(new_prefix_state_dict)
else:
self.model = AutoModel.from_pretrained(MODEL_PATH, trust_remote_code=True, device_map="auto").eval()
# add .quantize(bits=4, device="cuda").cuda() before .eval() and remove device_map="auto" to use int4 model
# must use cuda to load int4 model
def generate_stream(
self,
system: str | None,
tools: list[dict] | None,
history: list[Conversation],
**parameters: Any
) -> Iterable[TextGenerationStreamResponse]:
chat_history = [{
'role': 'system',
'content': system if not tools else TOOL_PROMPT,
}]
if tools:
chat_history[0]['tools'] = tools
for conversation in history[:-1]:
chat_history.append({
'role': str(conversation.role).removeprefix('<|').removesuffix('|>'),
'content': conversation.content,
})
query = history[-1].content
role = str(history[-1].role).removeprefix('<|').removesuffix('|>')
text = ''
for new_text, _ in stream_chat(
self.model,
self.tokenizer,
query,
chat_history,
role,
**parameters,
):
word = new_text.removeprefix(text)
word_stripped = word.strip()
text = new_text
yield TextGenerationStreamResponse(
generated_text=text,
token=Token(
id=0,
logprob=0,
text=word,
special=word_stripped.startswith('<|') and word_stripped.endswith('|>'),
)
)
================================================
FILE: composite_demo/conversation.py
================================================
from dataclasses import dataclass
from enum import auto, Enum
import json
from PIL.Image import Image
import streamlit as st
from streamlit.delta_generator import DeltaGenerator
TOOL_PROMPT = 'Answer the following questions as best as you can. You have access to the following tools:\n'
class Role(Enum):
SYSTEM = auto()
USER = auto()
ASSISTANT = auto()
TOOL = auto()
INTERPRETER = auto()
OBSERVATION = auto()
def __str__(self):
match self:
case Role.SYSTEM:
return "<|system|>"
case Role.USER:
return "<|user|>"
case Role.ASSISTANT | Role.TOOL | Role.INTERPRETER:
return "<|assistant|>"
case Role.OBSERVATION:
return "<|observation|>"
# Get the message block for the given role
def get_message(self):
# Compare by value here, because the enum object in the session state
# is not the same as the enum cases here, due to streamlit's rerunning
# behavior.
match self.value:
case Role.SYSTEM.value:
return
case Role.USER.value:
return st.chat_message(name="user", avatar="user")
case Role.ASSISTANT.value:
return st.chat_message(name="assistant", avatar="assistant")
case Role.TOOL.value:
return st.chat_message(name="tool", avatar="assistant")
case Role.INTERPRETER.value:
return st.chat_message(name="interpreter", avatar="assistant")
case Role.OBSERVATION.value:
return st.chat_message(name="observation", avatar="user")
case _:
st.error(f'Unexpected role: {self}')
@dataclass
class Conversation:
role: Role
content: str
tool: str | None = None
image: Image | None = None
def __str__(self) -> str:
print(self.role, self.content, self.tool)
match self.role:
case Role.SYSTEM | Role.USER | Role.ASSISTANT | Role.OBSERVATION:
return f'{self.role}\n{self.content}'
case Role.TOOL:
return f'{self.role}{self.tool}\n{self.content}'
case Role.INTERPRETER:
return f'{self.role}interpreter\n{self.content}'
# Human readable format
def get_text(self) -> str:
text = postprocess_text(self.content)
match self.role.value:
case Role.TOOL.value:
text = f'Calling tool `{self.tool}`:\n\n{text}'
case Role.INTERPRETER.value:
text = f'{text}'
case Role.OBSERVATION.value:
text = f'Observation:\n```\n{text}\n```'
return text
# Display as a markdown block
def show(self, placeholder: DeltaGenerator | None=None) -> str:
if placeholder:
message = placeholder
else:
message = self.role.get_message()
if self.image:
message.image(self.image)
else:
text = self.get_text()
message.markdown(text)
def preprocess_text(
system: str | None,
tools: list[dict] | None,
history: list[Conversation],
) -> str:
if tools:
tools = json.dumps(tools, indent=4, ensure_ascii=False)
prompt = f"{Role.SYSTEM}\n"
prompt += system if not tools else TOOL_PROMPT
if tools:
tools = json.loads(tools)
prompt += json.dumps(tools, ensure_ascii=False)
for conversation in history:
prompt += f'{conversation}'
prompt += f'{Role.ASSISTANT}\n'
return prompt
def postprocess_text(text: str) -> str:
text = text.replace("\(", "$")
text = text.replace("\)", "$")
text = text.replace("\[", "$$")
text = text.replace("\]", "$$")
text = text.replace("<|assistant|>", "")
text = text.replace("<|observation|>", "")
text = text.replace("<|system|>", "")
text = text.replace("<|user|>", "")
return text.strip()
================================================
FILE: composite_demo/demo_chat.py
================================================
import streamlit as st
from streamlit.delta_generator import DeltaGenerator
from client import get_client
from conversation import postprocess_text, preprocess_text, Conversation, Role
client = get_client()
# Append a conversation into history, while show it in a new markdown block
def append_conversation(
conversation: Conversation,
history: list[Conversation],
placeholder: DeltaGenerator | None = None,
) -> None:
history.append(conversation)
conversation.show(placeholder)
def main(
prompt_text: str,
system_prompt: str,
top_p: float = 0.8,
temperature: float = 0.95,
repetition_penalty: float = 1.0,
max_new_tokens: int = 1024,
retry: bool = False
):
placeholder = st.empty()
with placeholder.container():
if 'chat_history' not in st.session_state:
st.session_state.chat_history = []
if prompt_text == "" and retry == False:
print("\n== Clean ==\n")
st.session_state.chat_history = []
return
history: list[Conversation] = st.session_state.chat_history
for conversation in history:
conversation.show()
if retry:
print("\n== Retry ==\n")
last_user_conversation_idx = None
for idx, conversation in enumerate(history):
if conversation.role == Role.USER:
last_user_conversation_idx = idx
if last_user_conversation_idx is not None:
prompt_text = history[last_user_conversation_idx].content
del history[last_user_conversation_idx:]
if prompt_text:
prompt_text = prompt_text.strip()
append_conversation(Conversation(Role.USER, prompt_text), history)
placeholder = st.empty()
message_placeholder = placeholder.chat_message(name="assistant", avatar="assistant")
markdown_placeholder = message_placeholder.empty()
output_text = ''
for response in client.generate_stream(
system_prompt,
tools=None,
history=history,
do_sample=True,
max_new_tokens=max_new_tokens,
temperature=temperature,
top_p=top_p,
stop_sequences=[str(Role.USER)],
repetition_penalty=repetition_penalty,
):
token = response.token
if response.token.special:
print("\n==Output:==\n", output_text)
match token.text.strip():
case '<|user|>':
break
case _:
st.error(f'Unexpected special token: {token.text.strip()}')
break
output_text += response.token.text
markdown_placeholder.markdown(postprocess_text(output_text + '▌'))
append_conversation(Conversation(
Role.ASSISTANT,
postprocess_text(output_text),
), history, markdown_placeholder)
================================================
FILE: composite_demo/demo_ci.py
================================================
import base64
from io import BytesIO
import os
from pprint import pprint
import queue
import re
from subprocess import PIPE
import jupyter_client
from PIL import Image
import streamlit as st
from streamlit.delta_generator import DeltaGenerator
from client import get_client
from conversation import postprocess_text, preprocess_text, Conversation, Role
IPYKERNEL = os.environ.get('IPYKERNEL', 'chatglm3-demo')
SYSTEM_PROMPT = '你是一位智能AI助手，你叫ChatGLM，你连接着一台电脑，但请注意不能联网。在使用Python解决任务时，你可以运行代码并得到结果，如果运行结果有错误，你需要尽可能对代码进行改进。你可以处理用户上传到电脑上的文件，文件默认存储路径是/mnt/data/。'
client = get_client()
class CodeKernel(object):
def __init__(self,
kernel_name='kernel',
kernel_id=None,
kernel_config_path="",
python_path=None,
ipython_path=None,
init_file_path="./startup.py",
verbose=1):
self.kernel_name = kernel_name
self.kernel_id = kernel_id
self.kernel_config_path = kernel_config_path
self.python_path = python_path
self.ipython_path = ipython_path
self.init_file_path = init_file_path
self.verbose = verbose
if python_path is None and ipython_path is None:
env = None
else:
env = {"PATH": self.python_path + ":$PATH", "PYTHONPATH": self.python_path}
# Initialize the backend kernel
self.kernel_manager = jupyter_client.KernelManager(kernel_name=IPYKERNEL,
connection_file=self.kernel_config_path,
exec_files=[self.init_file_path],
env=env)
if self.kernel_config_path:
self.kernel_manager.load_connection_file()
self.kernel_manager.start_kernel(stdout=PIPE, stderr=PIPE)
print("Backend kernel started with the configuration: {}".format(
self.kernel_config_path))
else:
self.kernel_manager.start_kernel(stdout=PIPE, stderr=PIPE)
print("Backend kernel started with the configuration: {}".format(
self.kernel_manager.connection_file))
if verbose:
pprint(self.kernel_manager.get_connection_info())
# Initialize the code kernel
self.kernel = self.kernel_manager.blocking_client()
# self.kernel.load_connection_file()
self.kernel.start_channels()
print("Code kernel started.")
def execute(self, code):
self.kernel.execute(code)
try:
shell_msg = self.kernel.get_shell_msg(timeout=30)
io_msg_content = self.kernel.get_iopub_msg(timeout=30)['content']
while True:
msg_out = io_msg_content
### Poll the message
try:
io_msg_content = self.kernel.get_iopub_msg(timeout=30)['content']
if 'execution_state' in io_msg_content and io_msg_content['execution_state'] == 'idle':
break
except queue.Empty:
break
return shell_msg, msg_out
except Exception as e:
print(e)
return None
def execute_interactive(self, code, verbose=False):
shell_msg = self.kernel.execute_interactive(code)
if shell_msg is queue.Empty:
if verbose:
print("Timeout waiting for shell message.")
self.check_msg(shell_msg, verbose=verbose)
return shell_msg
def inspect(self, code, verbose=False):
msg_id = self.kernel.inspect(code)
shell_msg = self.kernel.get_shell_msg(timeout=30)
if shell_msg is queue.Empty:
if verbose:
print("Timeout waiting for shell message.")
self.check_msg(shell_msg, verbose=verbose)
return shell_msg
def get_error_msg(self, msg, verbose=False) -> str | None:
if msg['content']['status'] == 'error':
try:
error_msg = msg['content']['traceback']
except:
try:
error_msg = msg['content']['traceback'][-1].strip()
except:
error_msg = "Traceback Error"
if verbose:
print("Error: ", error_msg)
return error_msg
return None
def check_msg(self, msg, verbose=False):
status = msg['content']['status']
if status == 'ok':
if verbose:
print("Execution succeeded.")
elif status == 'error':
for line in msg['content']['traceback']:
if verbose:
print(line)
def shutdown(self):
# Shutdown the backend kernel
self.kernel_manager.shutdown_kernel()
print("Backend kernel shutdown.")
# Shutdown the code kernel
self.kernel.shutdown()
print("Code kernel shutdown.")
def restart(self):
# Restart the backend kernel
self.kernel_manager.restart_kernel()
# print("Backend kernel restarted.")
def interrupt(self):
# Interrupt the backend kernel
self.kernel_manager.interrupt_kernel()
# print("Backend kernel interrupted.")
def is_alive(self):
return self.kernel.is_alive()
def b64_2_img(data):
buff = BytesIO(base64.b64decode(data))
return Image.open(buff)
def clean_ansi_codes(input_string):
ansi_escape = re.compile(r'(\x9B|\x1B\[|\u001b\[)[0-?]*[ -/]*[@-~]')
return ansi_escape.sub('', input_string)
def execute(code, kernel: CodeKernel) -> tuple[str, str | Image.Image]:
res = ""
res_type = None
code = code.replace("<|observation|>", "")
code = code.replace("<|assistant|>interpreter", "")
code = code.replace("<|assistant|>", "")
code = code.replace("<|user|>", "")
code = code.replace("<|system|>", "")
msg, output = kernel.execute(code)
if msg['metadata']['status'] == "timeout":
return res_type, 'Timed out'
elif msg['metadata']['status'] == 'error':
return res_type, clean_ansi_codes('\n'.join(kernel.get_error_msg(msg, verbose=True)))
if 'text' in output:
res_type = "text"
res = output['text']
elif 'data' in output:
for key in output['data']:
if 'text/plain' in key:
res_type = "text"
res = output['data'][key]
elif 'image/png' in key:
res_type = "image"
res = output['data'][key]
break
if res_type == "image":
return res_type, b64_2_img(res)
elif res_type == "text" or res_type == "traceback":
res = res
return res_type, res
@st.cache_resource
def get_kernel():
kernel = CodeKernel()
return kernel
def extract_code(text: str) -> str:
pattern = r'```([^\n]*)\n(.*?)```'
matches = re.findall(pattern, text, re.DOTALL)
return matches[-1][1]
# Append a conversation into history, while show it in a new markdown block
def append_conversation(
conversation: Conversation,
history: list[Conversation],
placeholder: DeltaGenerator | None = None,
) -> None:
history.append(conversation)
conversation.show(placeholder)
def main(
prompt_text: str,
top_p: float = 0.2,
temperature: float = 0.1,
repetition_penalty: float = 1.1,
max_new_tokens: int = 1024,
truncate_length: int = 1024,
retry: bool = False
):
if 'ci_history' not in st.session_state:
st.session_state.ci_history = []
if prompt_text == "" and retry == False:
print("\n== Clean ==\n")
st.session_state.chat_history = []
return
history: list[Conversation] = st.session_state.chat_history
for conversation in history:
conversation.show()
if retry:
print("\n== Retry ==\n")
last_user_conversation_idx = None
for idx, conversation in enumerate(history):
if conversation.role == Role.USER:
last_user_conversation_idx = idx
if last_user_conversation_idx is not None:
prompt_text = history[last_user_conversation_idx].content
del history[last_user_conversation_idx:]
if prompt_text:
prompt_text = prompt_text.strip()
role = Role.USER
append_conversation(Conversation(role, prompt_text), history)
placeholder = st.container()
message_placeholder = placeholder.chat_message(name="assistant", avatar="assistant")
markdown_placeholder = message_placeholder.empty()
for _ in range(5):
output_text = ''
for response in client.generate_stream(
system=SYSTEM_PROMPT,
tools=None,
history=history,
do_sample=True,
max_new_token=max_new_tokens,
temperature=temperature,
top_p=top_p,
stop_sequences=[str(r) for r in (Role.USER, Role.OBSERVATION)],
repetition_penalty=repetition_penalty,
):
token = response.token
if response.token.special:
print("\n==Output:==\n", output_text)
match token.text.strip():
case '<|user|>':
append_conversation(Conversation(
Role.ASSISTANT,
postprocess_text(output_text),
), history, markdown_placeholder)
return
# Initiate tool call
case '<|assistant|>':
append_conversation(Conversation(
Role.ASSISTANT,
postprocess_text(output_text),
), history, markdown_placeholder)
message_placeholder = placeholder.chat_message(name="interpreter", avatar="assistant")
markdown_placeholder = message_placeholder.empty()
output_text = ''
continue
case '<|observation|>':
code = extract_code(output_text)
display_text = output_text.split('interpreter')[-1].strip()
append_conversation(Conversation(
Role.INTERPRETER,
postprocess_text(display_text),
), history, markdown_placeholder)
message_placeholder = placeholder.chat_message(name="observation", avatar="user")
markdown_placeholder = message_placeholder.empty()
output_text = ''
with markdown_placeholder:
with st.spinner('Executing code...'):
try:
res_type, res = execute(code, get_kernel())
except Exception as e:
st.error(f'Error when executing code: {e}')
return
print("Received:", res_type, res)
if truncate_length:
if res_type == 'text' and len(res) > truncate_length:
res = res[:truncate_length] + ' [TRUNCATED]'
append_conversation(Conversation(
Role.OBSERVATION,
'[Image]' if res_type == 'image' else postprocess_text(res),
tool=None,
image=res if res_type == 'image' else None,
), history, markdown_placeholder)
message_placeholder = placeholder.chat_message(name="assistant", avatar="assistant")
markdown_placeholder = message_placeholder.empty()
output_text = ''
break
case _:
st.error(f'Unexpected special token: {token.text.strip()}')
break
output_text += response.token.text
display_text = output_text.split('interpreter')[-1].strip()
markdown_placeholder.markdown(postprocess_text(display_text + '▌'))
else:
append_conversation(Conversation(
Role.ASSISTANT,
postprocess_text(output_text),
), history, markdown_placeholder)
return
else:
st.session_state.chat_history = []
================================================
FILE: composite_demo/demo_tool.py
================================================
import re
import yaml
from yaml import YAMLError
import streamlit as st
from streamlit.delta_generator import DeltaGenerator
from client import get_client
from conversation import postprocess_text, preprocess_text, Conversation, Role
from tool_registry import dispatch_tool, get_tools
EXAMPLE_TOOL = {
"name": "get_current_weather",
"description": "Get the current weather in a given location",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "The city and state, e.g. San Francisco, CA",
},
"unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
},
"required": ["location"],
}
}
client = get_client()
def tool_call(*args, **kwargs) -> dict:
print("=== Tool call===")
print(args)
print(kwargs)
st.session_state.calling_tool = True
return kwargs
def yaml_to_dict(tools: str) -> list[dict] | None:
try:
return yaml.safe_load(tools)
except YAMLError:
return None
def extract_code(text: str) -> str:
pattern = r'```([^\n]*)\n(.*?)```'
matches = re.findall(pattern, text, re.DOTALL)
print(matches)
return matches[-1][1]
# Append a conversation into history, while show it in a new markdown block
def append_conversation(
conversation: Conversation,
history: list[Conversation],
placeholder: DeltaGenerator | None = None,
) -> None:
history.append(conversation)
conversation.show(placeholder)
def main(
prompt_text: str,
top_p: float = 0.2,
temperature: float = 0.1,
repetition_penalty: float = 1.1,
max_new_tokens: int = 1024,
truncate_length: int = 1024,
retry: bool = False
):
manual_mode = st.toggle('Manual mode',
help='Define your tools in YAML format. You need to supply tool call results manually.'
)
if manual_mode:
with st.expander('Tools'):
tools = st.text_area(
'Define your tools in YAML format here:',
yaml.safe_dump([EXAMPLE_TOOL], sort_keys=False),
height=400,
)
tools = yaml_to_dict(tools)
if not tools:
st.error('YAML format error in tools definition')
else:
tools = get_tools()
if 'tool_history' not in st.session_state:
st.session_state.tool_history = []
if 'calling_tool' not in st.session_state:
st.session_state.calling_tool = False
if 'chat_history' not in st.session_state:
st.session_state.chat_history = []
if prompt_text == "" and retry == False:
print("\n== Clean ==\n")
st.session_state.chat_history = []
return
history: list[Conversation] = st.session_state.chat_history
for conversation in history:
conversation.show()
if retry:
print("\n== Retry ==\n")
last_user_conversation_idx = None
for idx, conversation in enumerate(history):
if conversation.role == Role.USER:
last_user_conversation_idx = idx
if last_user_conversation_idx is not None:
prompt_text = history[last_user_conversation_idx].content
del history[last_user_conversation_idx:]
if prompt_text:
prompt_text = prompt_text.strip()
role = st.session_state.calling_tool and Role.OBSERVATION or Role.USER
append_conversation(Conversation(role, prompt_text), history)
st.session_state.calling_tool = False
placeholder = st.container()
message_placeholder = placeholder.chat_message(name="assistant", avatar="assistant")
markdown_placeholder = message_placeholder.empty()
for _ in range(5):
output_text = ''
for response in client.generate_stream(
system=None,
tools=tools,
history=history,
do_sample=True,
max_new_tokens=max_new_tokens,
temperature=temperature,
top_p=top_p,
stop_sequences=[str(r) for r in (Role.USER, Role.OBSERVATION)],
repetition_penalty=repetition_penalty,
):
token = response.token
if response.token.special:
print("\n==Output:==\n", output_text)
match token.text.strip():
case '<|user|>':
append_conversation(Conversation(
Role.ASSISTANT,
postprocess_text(output_text),
), history, markdown_placeholder)
return
# Initiate tool call
case '<|assistant|>':
append_conversation(Conversation(
Role.ASSISTANT,
postprocess_text(output_text),
), history, markdown_placeholder)
output_text = ''
message_placeholder = placeholder.chat_message(name="tool", avatar="assistant")
markdown_placeholder = message_placeholder.empty()
continue
case '<|observation|>':
tool, *call_args_text = output_text.strip().split('\n')
call_args_text = '\n'.join(call_args_text)
append_conversation(Conversation(
Role.TOOL,
postprocess_text(output_text),
tool,
), history, markdown_placeholder)
message_placeholder = placeholder.chat_message(name="observation", avatar="user")
markdown_placeholder = message_placeholder.empty()
try:
code = extract_code(call_args_text)
args = eval(code, {'tool_call': tool_call}, {})
except:
st.error('Failed to parse tool call')
return
output_text = ''
if manual_mode:
st.info('Please provide tool call results below:')
return
else:
with markdown_placeholder:
with st.spinner(f'Calling tool {tool}...'):
observation = dispatch_tool(tool, args)
if len(observation) > truncate_length:
observation = observation[:truncate_length] + ' [TRUNCATED]'
append_conversation(Conversation(
Role.OBSERVATION, observation
), history, markdown_placeholder)
message_placeholder = placeholder.chat_message(name="assistant", avatar="assistant")
markdown_placeholder = message_placeholder.empty()
st.session_state.calling_tool = False
break
case _:
st.error(f'Unexpected special token: {token.text.strip()}')
return
output_text += response.token.text
markdown_placeholder.markdown(postprocess_text(output_text + '▌'))
else:
append_conversation(Conversation(
Role.ASSISTANT,
postprocess_text(output_text),
), history, markdown_placeholder)
return
================================================
FILE: composite_demo/main.py
================================================
import streamlit as st
st.set_page_config(
page_title="ChatGLM3 Demo",
page_icon=":robot:",
layout='centered',
initial_sidebar_state='expanded',
)
import demo_chat, demo_ci, demo_tool
from enum import Enum
DEFAULT_SYSTEM_PROMPT = '''
You are ChatGLM3, a large language model trained by Zhipu.AI. Follow the user's instructions carefully. Respond using markdown.
'''.strip()
# Set the title of the demo
st.title("ChatGLM3 Demo")
# Add your custom text here, with smaller font size
st.markdown(
"<sub>智谱AI 公开在线技术文档: https://lslfd0slxc.feishu.cn/wiki/WvQbwIJ9tiPAxGk8ywDck6yfnof </sub> \n\n <sub> 更多 ChatGLM3-6B 的使用方法请参考文档。</sub>",
unsafe_allow_html=True)
class Mode(str, Enum):
CHAT, TOOL, CI = '💬 Chat', '🛠️ Tool', '🧑💻 Code Interpreter'
with st.sidebar:
top_p = st.slider(
'top_p', 0.0, 1.0, 0.8, step=0.01
)
temperature = st.slider(
'temperature', 0.0, 1.5, 0.95, step=0.01
)
repetition_penalty = st.slider(
'repetition_penalty', 0.0, 2.0, 1.1, step=0.01
)
max_new_token = st.slider(
'Output length', 5, 32000, 256, step=1
)
cols = st.columns(2)
export_btn = cols[0]
clear_history = cols[1].button("Clear History", use_container_width=True)
retry = export_btn.button("Retry", use_container_width=True)
system_prompt = st.text_area(
label="System Prompt (Only for chat mode)",
height=300,
value=DEFAULT_SYSTEM_PROMPT,
)
prompt_text = st.chat_input(
'Chat with ChatGLM3!',
key='chat_input',
)
tab = st.radio(
'Mode',
[mode.value for mode in Mode],
horizontal=True,
label_visibility='hidden',
)
if clear_history or retry:
prompt_text = ""
match tab:
case Mode.CHAT:
demo_chat.main(
retry=retry,
top_p=top_p,
temperature=temperature,
prompt_text=prompt_text,
system_prompt=system_prompt,
repetition_penalty=repetition_penalty,
max_new_tokens=max_new_token
)
case Mode.TOOL:
demo_tool.main(
retry=retry,
top_p=top_p,
temperature=temperature,
prompt_text=prompt_text,
repetition_penalty=repetition_penalty,
max_new_tokens=max_new_token,
truncate_length=1024)
case Mode.CI:
demo_ci.main(
retry=retry,
top_p=top_p,
temperature=temperature,
prompt_text=prompt_text,
repetition_penalty=repetition_penalty,
max_new_tokens=max_new_token,
truncate_length=1024)
case _:
st.error(f'Unexpected tab: {tab}')
================================================
FILE: composite_demo/requirements.txt
================================================
huggingface_hub>=0.19.4
pillow>=10.1.0
pyyaml>=6.0.1
requests>=2.31.0
ipykernel>=6.26.0
ipython>=8.18.1
jupyter_client>=8.6.0
================================================
FILE: composite_demo/tool_registry.py
================================================
"""
This code is the tool registration part. By registering the tool, the model can call the tool.
This code provides extended functionality to the model, enabling it to call and interact with a variety of utilities
through defined interfaces.
"""
import copy
import inspect
from pprint import pformat
import traceback
from types import GenericAlias
from typing import get_origin, Annotated
import subprocess
_TOOL_HOOKS = {}
_TOOL_DESCRIPTIONS = {}
def register_tool(func: callable):
tool_name = func.__name__
tool_description = inspect.getdoc(func).strip()
python_params = inspect.signature(func).parameters
tool_params = []
for name, param in python_params.items():
annotation = param.annotation
if annotation is inspect.Parameter.empty:
raise TypeError(f"Parameter `{name}` missing type annotation")
if get_origin(annotation) != Annotated:
raise TypeError(f"Annotation type for `{name}` must be typing.Annotated")
typ, (description, required) = annotation.__origin__, annotation.__metadata__
typ: str = str(typ) if isinstance(typ, GenericAlias) else typ.__name__
if not isinstance(description, str):
raise TypeError(f"Description for `{name}` must be a string")
if not isinstance(required, bool):
raise TypeError(f"Required for `{name}` must be a bool")
tool_params.append({
"name": name,
"description": description,
"type": typ,
"required": required
})
tool_def = {
"name": tool_name,
"description": tool_description,
"params": tool_params
}
print("[registered tool] " + pformat(tool_def))
_TOOL_HOOKS[tool_name] = func
_TOOL_DESCRIPTIONS[tool_name] = tool_def
return func
def dispatch_tool(tool_name: str, tool_params: dict) -> str:
if tool_name not in _TOOL_HOOKS:
return f"Tool `{tool_name}` not found. Please use a provided tool."
tool_call = _TOOL_HOOKS[tool_name]
try:
ret = tool_call(**tool_params)
except:
ret = traceback.format_exc()
return str(ret)
def get_tools() -> dict:
return copy.deepcopy(_TOOL_DESCRIPTIONS)
# Tool Definitions
@register_tool
def random_number_generator(
seed: Annotated[int, 'The random seed used by the generator', True],
range: Annotated[tuple[int, int], 'The range of the generated numbers', True],
) -> int:
"""
Generates a random number x, s.t. range[0] <= x < range[1]
"""
if not isinstance(seed, int):
raise TypeError("Seed must be an integer")
if not isinstance(range, tuple):
raise TypeError("Range must be a tuple")
if not isinstance(range[0], int) or not isinstance(range[1], int):
raise TypeError("Range must be a tuple of integers")
import random
return random.Random(seed).randint(*range)
@register_tool
def get_weather(
city_name: Annotated[str, 'The name of the city to be queried', True],
) -> str:
"""
Get the current weather for `city_name`
"""
if not isinstance(city_name, str):
raise TypeError("City name must be a string")
key_selection = {
"current_condition": ["temp_C", "FeelsLikeC", "humidity", "weatherDesc", "observation_time"],
}
import requests
try:
resp = requests.get(f"https://wttr.in/{city_name}?format=j1")
resp.raise_for_status()
resp = resp.json()
ret = {k: {_v: resp[k][0][_v] for _v in v} for k, v in key_selection.items()}
except:
import traceback
ret = "Error encountered while fetching weather data!\n" + traceback.format_exc()
return str(ret)
@register_tool
def get_shell(
query: Annotated[str, 'The command should run in Linux shell', True],
) -> str:
"""
Use shell to run command
"""
if not isinstance(query, str):
raise TypeError("Command must be a string")
try:
result = subprocess.run(query, shell=True, check=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE,
text=True)
return result.stdout
except subprocess.CalledProcessError as e:
return e.stderr
if __name__ == "__main__":
# print(dispatch_tool("get_shell", {"query": "pwd"}))
print(get_tools())
================================================
FILE: finetune_demo/README.md
================================================
# ChatGLM3-6B 微调
本目录提供 ChatGLM3-6B 模型的微调示例，包括全量微调和 P-Tuning v2。格式上，提供多轮对话微调样例和输入输出格式微调样例。
如果将模型下载到了本地，本文和代码中的 `THUDM/chatglm3-6b` 字段均应替换为相应地址以从本地加载模型。
运行示例需要 `python>=3.10`，除基础的 `torch` 依赖外，示例代码运行还需要依赖。
**我们提供了 [示例notebook](lora_finetune.ipynb) 用于演示如何使用我们的微调代码。**
```bash
pip install -r requirements.txt
```
## 测试硬件标准
我们仅提供了单机多卡/多机多卡的运行示例，因此您需要至少一台具有多个 GPU 的机器。本仓库中的**默认配置文件**中，我们记录了显存的占用情况：
+ SFT 全量微调: 4张显卡平均分配，每张显卡占用 `48346MiB` 显存。
+ P-TuningV2 微调: 1张显卡，占用 `18426MiB` 显存。
+ LORA 微调: 1张显卡，占用 `14082MiB` 显存。
> 请注意，该结果仅供参考，对于不同的参数，显存占用可能会有所不同。请结合你的硬件情况进行调整。
> 请注意，我们仅仅使用英伟达 Hopper(代表显卡：H100) 和 Ampère(代表显卡:A100) 架构和系列显卡做过测试。如果您使用其他架构的显卡，可能会出现
> 1. 未知的训练问题 / 显存占用与上述有误差。
> 2. 架构过低而不支持某些特性。
> 3. 推理效果问题。
> 以上三种情况为社区曾经遇到过的问题，虽然概率较低，如果您遇到了以上问题，可以尝试在社区中解决。
## 多轮对话格式
多轮对话微调示例采用 ChatGLM3 对话格式约定，对不同角色添加不同 `loss_mask` 从而在一遍计算中为多轮回复计算 `loss`。
对于数据文件，样例采用如下格式
如果您仅希望微调模型的对话能力，而非工具能力，您应该按照以下格式整理数据。
```json
[
{
"conversations": [
{
"role": "system",
"content": "<system prompt text>"
},
{
"role": "user",
"content": "<user prompt text>"
},
{
"role": "assistant",
"content": "<assistant response text>"
},
// ... Muti Turn
{
"role": "user",
"content": "<user prompt text>"
},
{
"role": "assistant",
"content": "<assistant response text>"
}
]
}
// ...
]
```
**请注意，这种方法在微调的step较多的情况下会影响到模型的工具调用功能**
如果您希望微调模型的对话和工具能力，您应该按照以下格式整理数据。
```json
[
{
"tools": [
// available tools, format is not restricted
],
"conversations": [
{
"role": "system",
"content": "<system prompt text>"
},
{
"role": "user",
"content": "<user prompt text>"
},
{
"role": "assistant",
"content": "<assistant thought to text>"
},
{
"role": "tool",
"name": "<name of the tool to be called",
"parameters": {
"<parameter_name>": "<parameter_value>"
},
"observation": "<observation>"
// don't have to be string
},
{
"role": "assistant",
"content": "<assistant response to observation>"
},
// ... Muti Turn
{
"role": "user",
"content": "<user prompt text>"
},
{
"role": "assistant",
"content": "<assistant response text>"
}
]
}
// ...
]
```
- 关于工具描述的 system prompt 无需手动插入，预处理时会将 `tools` 字段使用 `json.dumps(..., ensure_ascii=False)`
格式化后插入为首条 system prompt。
- 每种角色可以附带一个 `bool` 类型的 `loss` 字段，表示该字段所预测的内容是否参与 `loss`
计算。若没有该字段，样例实现中默认对 `system`, `user` 不计算 `loss`，其余角色则计算 `loss`。
- `tool` 并不是 ChatGLM3 中的原生角色，这里的 `tool` 在预处理阶段将被自动转化为一个具有工具调用 `metadata` 的 `assistant`
角色（默认计算 `loss`）和一个表示工具返回值的 `observation` 角色（不计算 `loss`）。
- 目前暂未实现 `Code interpreter` 的微调任务。
- `system` 角色为可选角色，但若存在 `system` 角色，其必须出现在 `user`
角色之前，且一个完整的对话数据（无论单轮或者多轮对话）只能出现一次 `system` 角色。
## 数据集格式示例
这里以 AdvertiseGen 数据集为例,
您可以从 [Google Drive](https://drive.google.com/file/d/13_vf0xRTQsyneRKdD1bZIr93vBGOczrk/view?usp=sharing)
或者 [Tsinghua Cloud](https://cloud.tsinghua.edu.cn/f/b3f119a008264b1cabd1/?dl=1) 下载 AdvertiseGen 数据集。
将解压后的 AdvertiseGen 目录放到 `data` 目录下并自行转换为如下格式数据集。
> 请注意，现在的微调代码中加入了验证集，因此，对于一组完整的微调数据集，必须包含训练数据集和验证数据集，测试数据集可以不填写。或者直接用验证数据集代替。
```
{"conversations": [{"role": "user", "content": "类型#裙*裙长#半身裙"}, {"role": "assistant", "content": "这款百搭时尚的仙女半身裙，整体设计非常的飘逸随性，穿上之后每个女孩子都能瞬间变成小仙女啦。料子非常的轻盈，透气性也很好，穿到夏天也很舒适。"}]}
```
## 配置文件
微调配置文件位于 `config` 目录下，包括以下文件：
1. `ds_zereo_2 / ds_zereo_3.json`: deepspeed 配置文件。
2. `lora.yaml / ptuning.yaml / sft.yaml`: 模型不同方式的配置文件，包括模型参数、优化器参数、训练参数等。 部分重要参数解释如下：
+ data_config 部分
+ train_file: 训练数据集的文件路径。
+ val_file: 验证数据集的文件路径。
+ test_file: 测试数据集的文件路径。
+ num_proc: 在加载数据时使用的进程数量。
+ max_input_length: 输入序列的最大长度。
+ max_output_length: 输出序列的最大长度。
+ training_args 部分
+ output_dir: 用于保存模型和其他输出的目录。
+ max_steps: 训练的最大步数。
+ per_device_train_batch_size: 每个设备（如 GPU）的训练批次大小。
+ dataloader_num_workers: 加载数据时使用的工作线程数量。
+ remove_unused_columns: 是否移除数据中未使用的列。
+ save_strategy: 模型保存策略（例如，每隔多少步保存一次）。
+ save_steps: 每隔多少步保存一次模型。
+ log_level: 日志级别（如 info）。
+ logging_strategy: 日志记录策略。
+ logging_steps: 每隔多少步记录一次日志。
+ per_device_eval_batch_size: 每个设备的评估批次大小。
+ evaluation_strategy: 评估策略（例如，每隔多少步进行一次评估）。
+ eval_steps: 每隔多少步进行一次评估。
+ predict_with_generate: 是否使用生成模式进行预测。
+ generation_config 部分
+ max_new_tokens: 生成的最大新 token 数量。
+ peft_config 部分
+ peft_type: 使用的参数有效调整类型（如 LORA）。
+ task_type: 任务类型，这里是因果语言模型（CAUSAL_LM）。
+ Lora 参数：
+ r: LoRA 的秩。
+ lora_alpha: LoRA 的缩放因子。
+ lora_dropout: 在 LoRA 层使用的 dropout 概率
+ P-TuningV2 参数：
+ num_virtual_tokens: 虚拟 token 的数量。
## 开始微调
通过以下代码执行 **单机多卡/多机多卡** 运行，这是使用 `deepspeed` 作为加速方案的，您需要安装 `deepspeed`。
```angular2html
cd finetune_demo
OMP_NUM_THREADS=1 torchrun --standalone --nnodes=1 --nproc_per_node=8 finetune_hf.py data/AdvertiseGen/ THUDM/chatglm3-6b configs/lora.yaml
```
通过以下代码执行 **单机单卡** 运行。
```angular2html
cd finetune_demo
python finetune_hf.py data/AdvertiseGen/ THUDM/chatglm3-6b configs/lora.yaml
```
## 从保存点进行微调
如果按照上述方式进行训练，每次微调都会从头开始，如果你想从训练一半的模型开始微调，你可以加入第四个参数，这个参数有两种传入方式:
1. `yes`, 自动从最后一个保存的 Checkpoint开始训练
2. `XX`, 断点号数字 例 `600` 则从序号600 Checkpoint开始训练
例如，这就是一个从最后一个保存点继续微调的示例代码
```angular2html
cd finetune_demo
python finetune_hf.py data/AdvertiseGen/ THUDM/chatglm3-6b configs/lora.yaml yes
```
## 使用微调后的模型
### 在 inference_hf.py 中验证微调后的模型
您可以在 `finetune_demo/inference_hf.py` 中使用我们的微调后的模型，仅需要一行代码就能简单的进行测试。
```angular2html
python inference_hf.py your_finetune_path --prompt your prompt
```
这样，得到的回答就微调后的回答了。
### 在本仓库的其他 demo 或者外部仓库使用微调后的模型
您可以在任何一个 demo 内使用我们的 `lora` 和 全参微调的模型。这需要你自己按照以下教程进行修改代码。
1. 使用`finetune_demo/inference_hf.py`中读入模型的方式替换 demo 中读入模型的方式。
> 请注意，对于 LORA 和 P-TuningV2 我们没有合并训练后的模型，而是在`adapter_config.json`
> 中记录了微调型的路径，如果你的原始模型位置发生更改，则你应该修改`adapter_config.json`中`base_model_name_or_path`的路径。
```python
def load_model_and_tokenizer(
model_dir: Union[str, Path], trust_remote_code: bool = True
) -> tuple[ModelType, TokenizerType]:
model_dir = _resolve_path(model_dir)
if (model_dir / 'adapter_config.json').exists():
model = AutoPeftModelForCausalLM.from_pretrained(
model_dir, trust_remote_code=trust_remote_code, device_map='auto'
)
tokenizer_dir = model.peft_config['default'].base_model_name_or_path
else:
model = AutoModelForCausalLM.from_pretrained(
model_dir, trust_remote_code=trust_remote_code, device_map='auto'
)
tokenizer_dir = model_dir
tokenizer = AutoTokenizer.from_pretrained(
tokenizer_dir, trust_remote_code=trust_remote_code
)
return model, tokenizer
```
2. 读取微调的模型，请注意，你应该使用微调模型的位置，例如，若你的模型位置为`/path/to/finetune_adapter_model`
，原始模型地址为`path/to/base_model`,则你应该使用`/path/to/finetune_adapter_model`作为`model_dir`。
3. 完成上述操作后，就能正常使用微调的模型了，其他的调用方式没有变化。
### 提示
1. 微调代码在开始训练前，会先打印首条训练数据的预处理信息(默认已经注释，可以解除注释)，显示为
```log
Sanity
Check >> >> >> >> >> >> >
'[gMASK]': 64790 -> -100
'sop': 64792 -> -100
'<|system|>': 64794 -> -100
'': 30910 -> -100
'\n': 13 -> -100
'Answer': 20115 -> -100
'the': 267 -> -100
'following': 1762 -> -100
...
'know': 683 -> -100
'the': 267 -> -100
'response': 3010 -> -100
'details': 3296 -> -100
'.': 30930 -> -100
'<|assistant|>': 64796 -> -100
'': 30910 -> 30910
'\n': 13 -> 13
'I': 307 -> 307
'need': 720 -> 720
'to': 289 -> 289
'use': 792 -> 792
...
<< << << << << << < Sanity
Check
```
字样，每行依次表示一个 detokenized string, token_id 和 target_id。其中，`target_id`为`token_id`在模型词表中的索引，`-100`表示该
token 不参与 `loss` 计算。
2. `_prepare_model_for_training` 的作用是遍历模型的所有可训练参数，并确保它们的数据类型为`torch.float32`。
这在某些情况下是必要的，因为混合精度训练或其他操作可能会更改模型参数的数据类型。该代码默打开，可以注释，但是如果使用
`half` 格式训练出现问题，可以切换回这个代码，显存可能增加。
3. 在我们的[Huggingface模型代码](https://huggingface.co/THUDM/chatglm3-6b/blob/main/modeling_chatglm.py)中，有以下内容：
```python
if self.gradient_checkpointing and self.training:
layer_ret = torch.utils.checkpoint.checkpoint(
layer,
hidden_states,
attention_mask,
rotary_pos_emb,
kv_caches[index],
use_cache,
use_reentrant=False
)
```
这可能导致训练的时候显存增加，因此，如果您的显存不足，可以尝试将``` use_reentrant``` 修改为`True`。
4. 微调后的模型可以使用任何支持 `peft` 载入的模型加速框架，在这里，我们没有提供demo。
5. 本仓库的微调数据集格式与 API 微调数据集格式有一定区别
+ ZhipuAI API 微调数据集中的 `messages` 字段在本仓库为 `conversation` 字段。
+ ZhipuAI API 中的微调文件为 `jsonl`, 在本仓库，需要简单的将文件名改为 `json`。
## 参考文献
```
@inproceedings{liu2022p,
title={P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks},
author={Liu, Xiao and Ji, Kaixuan and Fu, Yicheng and Tam, Weng and Du, Zhengxiao and Yang, Zhilin and Tang, Jie},
booktitle={Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short
Papers)},
pages={61--68},
year={2022}
}
@misc{tang2023toolalpaca,
title={ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases},
author={Qiaoyu Tang and Ziliang Deng and Hongyu Lin and Xianpei Han and Qiao Liang and Le Sun},
year={2023},
eprint={2306.05301},
archivePrefix={arXiv},
primaryClass={cs.CL}
}
```
================================================
FILE: finetune_demo/README_en.md
================================================
# ChatGLM3-6B finetune
This directory provides fine-tuning examples of the ChatGLM3-6B model, including full fine-tuning and P-Tuning v2. In
terms of format, it provides multiple rounds of dialogue fine-tuning samples and input and output format fine-tuning
samples.
If the model is downloaded locally, the `THUDM/chatglm3-6b` field in this article and the code should be replaced with
the corresponding address to load the model locally.
Running the example requires `python>=3.10`. In addition to the basic `torch` dependency, the example code also requires
dependencies to run.
**We provide [sample notebook](lora_finetune.ipynb) to demonstrate how to use our fine-tuning code. **
```bash
pip install -r requirements.txt
```
## Test hardware standards
We only provide single-machine multi-card/multi-machine multi-card running examples, so you will need at least one
machine with multiple GPUs. In the **default configuration file** in this warehouse, we record the usage of video
memory:
+ SFT full fine-tuning: evenly distributed among 4 graphics cards, each graphics card occupies `48346MiB` of video
memory.
+ P-TuningV2 fine-tuning: 1 graphics card, occupying `18426MiB` memory.
+ LORA fine-tuning: 1 graphics card, occupying `14082MiB` memory.
> Please note that this result is for refere
gitextract_gh6ajpza/ ├── .github/ │ ├── ISSUE_TEMPLATE/ │ │ ├── bug_report.yaml │ │ └── feature-request.yaml │ └── PULL_REQUEST_TEMPLATE/ │ └── pr_template.md ├── .gitignore ├── DEPLOYMENT.md ├── DEPLOYMENT_en.md ├── Intel_device_demo/ │ ├── README.md │ ├── ipex_llm_cpu_demo/ │ │ ├── api_server.py │ │ ├── chatglm3_infer.py │ │ ├── chatglm3_web_demo.py │ │ ├── generate.py │ │ ├── openai_api_request.py │ │ └── utils.py │ └── openvino_demo/ │ ├── README.md │ └── openvino_cli_demo.py ├── LICENSE ├── MODEL_LICENSE ├── PROMPT.md ├── PROMPT_en.md ├── README.md ├── README_en.md ├── basic_demo/ │ ├── cli_batch_request_demo.py │ ├── cli_demo.py │ ├── cli_demo_bad_word_ids.py │ ├── web_demo_gradio.py │ └── web_demo_streamlit.py ├── composite_demo/ │ ├── .streamlit/ │ │ └── config.toml │ ├── README.md │ ├── README_en.md │ ├── client.py │ ├── conversation.py │ ├── demo_chat.py │ ├── demo_ci.py │ ├── demo_tool.py │ ├── main.py │ ├── requirements.txt │ └── tool_registry.py ├── finetune_demo/ │ ├── README.md │ ├── README_en.md │ ├── configs/ │ │ ├── ds_zero_2.json │ │ ├── ds_zero_3.json │ │ ├── lora.yaml │ │ ├── ptuning_v2.yaml │ │ └── sft.yaml │ ├── finetune_hf.py │ ├── inference_hf.py │ ├── lora_finetune.ipynb │ └── requirements.txt ├── langchain_demo/ │ ├── ChatGLM3.py │ ├── main.py │ └── tools/ │ ├── Calculator.py │ ├── DistanceConversion.py │ └── Weather.py ├── openai_api_demo/ │ ├── api_server.py │ ├── docker-compose.yml │ ├── langchain_openai_api.py │ ├── openai_api_request.py │ └── utils.py ├── requirements.txt ├── resources/ │ └── WECHAT.md ├── tensorrt_llm_demo/ │ ├── README.md │ └── tensorrt_llm_cli_demo.py ├── tools_using_demo/ │ ├── README.md │ ├── README_en.md │ ├── cli_demo_tool.py │ ├── openai_api_demo.py │ └── tool_register.py └── update_requirements.sh
SYMBOL INDEX (194 symbols across 31 files)
FILE: Intel_device_demo/ipex_llm_cpu_demo/api_server.py
function lifespan (line 60) | async def lifespan(app: FastAPI):
class ModelCard (line 78) | class ModelCard(BaseModel):
class ModelList (line 88) | class ModelList(BaseModel):
class FunctionCallResponse (line 93) | class FunctionCallResponse(BaseModel):
class ChatMessage (line 98) | class ChatMessage(BaseModel):
class DeltaMessage (line 105) | class DeltaMessage(BaseModel):
class EmbeddingRequest (line 112) | class EmbeddingRequest(BaseModel):
class CompletionUsage (line 117) | class CompletionUsage(BaseModel):
class EmbeddingResponse (line 123) | class EmbeddingResponse(BaseModel):
class UsageInfo (line 132) | class UsageInfo(BaseModel):
class ChatCompletionRequest (line 138) | class ChatCompletionRequest(BaseModel):
class ChatCompletionResponseChoice (line 149) | class ChatCompletionResponseChoice(BaseModel):
class ChatCompletionResponseStreamChoice (line 155) | class ChatCompletionResponseStreamChoice(BaseModel):
class ChatCompletionResponse (line 161) | class ChatCompletionResponse(BaseModel):
function health (line 171) | async def health() -> Response:
function get_embeddings (line 177) | async def get_embeddings(request: EmbeddingRequest):
function list_models (line 211) | async def list_models():
function create_chat_completion (line 221) | async def create_chat_completion(request: ChatCompletionRequest):
function predict (line 338) | async def predict(model_id: str, params: dict):
function predict_stream (line 404) | def predict_stream(model_id, gen_params):
function parse_output_text (line 484) | async def parse_output_text(model_id: str, value: str):
function contains_custom_function (line 510) | def contains_custom_function(value: str) -> bool:
FILE: Intel_device_demo/ipex_llm_cpu_demo/chatglm3_web_demo.py
function get_model (line 34) | def get_model():
FILE: Intel_device_demo/ipex_llm_cpu_demo/openai_api_request.py
function function_chat (line 20) | def function_chat():
function simple_chat (line 56) | def simple_chat(use_stream=True):
FILE: Intel_device_demo/ipex_llm_cpu_demo/utils.py
class InvalidScoreLogitsProcessor (line 9) | class InvalidScoreLogitsProcessor(LogitsProcessor):
method __call__ (line 10) | def __call__(
function process_response (line 19) | def process_response(output: str, use_tool: bool = False) -> Union[str, ...
function generate_stream_chatglm3 (line 47) | def generate_stream_chatglm3(model: PreTrainedModel, tokenizer: PreTrain...
function process_chatglm_messages (line 122) | def process_chatglm_messages(messages, tools=None):
function generate_chatglm3 (line 159) | def generate_chatglm3(model: PreTrainedModel, tokenizer: PreTrainedToken...
function apply_stopping_strings (line 165) | def apply_stopping_strings(reply, stop_strings) -> Tuple[str, bool]:
FILE: Intel_device_demo/openvino_demo/openvino_cli_demo.py
function parse_text (line 10) | def parse_text(text):
class StopOnTokens (line 42) | class StopOnTokens(StoppingCriteria):
method __init__ (line 43) | def __init__(self, token_ids):
method __call__ (line 46) | def __call__(
function convert_history_to_token (line 102) | def convert_history_to_token(history: List[Tuple[str, str]]):
FILE: basic_demo/cli_batch_request_demo.py
function batch (line 11) | def batch(
function main (line 50) | def main(batch_queries):
FILE: basic_demo/cli_demo.py
function build_prompt (line 20) | def build_prompt(history):
function main (line 28) | def main():
FILE: basic_demo/cli_demo_bad_word_ids.py
function build_prompt (line 37) | def build_prompt(history):
function main (line 44) | def main():
FILE: basic_demo/web_demo_gradio.py
function _resolve_path (line 48) | def _resolve_path(path: Union[str, Path]) -> Path:
function load_model_and_tokenizer (line 52) | def load_model_and_tokenizer(
class StopOnTokens (line 75) | class StopOnTokens(StoppingCriteria):
method __call__ (line 76) | def __call__(self, input_ids: torch.LongTensor, scores: torch.FloatTen...
function parse_text (line 84) | def parse_text(text):
function predict (line 115) | def predict(history, max_length, top_p, temperature, system_prompt):
function user (line 173) | def user(query, history):
FILE: basic_demo/web_demo_streamlit.py
function get_model (line 29) | def get_model():
FILE: composite_demo/client.py
function get_client (line 25) | def get_client() -> Client:
class Client (line 30) | class Client(Protocol):
method generate_stream (line 31) | def generate_stream(self,
function stream_chat (line 40) | def stream_chat(
class HFClient (line 125) | class HFClient(Client):
method __init__ (line 126) | def __init__(self, model_path: str, tokenizer_path: str, pt_checkpoint...
method generate_stream (line 155) | def generate_stream(
FILE: composite_demo/conversation.py
class Role (line 11) | class Role(Enum):
method __str__ (line 19) | def __str__(self):
method get_message (line 31) | def get_message(self):
class Conversation (line 52) | class Conversation:
method __str__ (line 58) | def __str__(self) -> str:
method get_text (line 69) | def get_text(self) -> str:
method show (line 81) | def show(self, placeholder: DeltaGenerator | None=None) -> str:
function preprocess_text (line 92) | def preprocess_text(
function postprocess_text (line 110) | def postprocess_text(text: str) -> str:
FILE: composite_demo/demo_chat.py
function append_conversation (line 11) | def append_conversation(
function main (line 20) | def main(
FILE: composite_demo/demo_ci.py
class CodeKernel (line 24) | class CodeKernel(object):
method __init__ (line 25) | def __init__(self,
method execute (line 71) | def execute(self, code):
method execute_interactive (line 91) | def execute_interactive(self, code, verbose=False):
method inspect (line 100) | def inspect(self, code, verbose=False):
method get_error_msg (line 110) | def get_error_msg(self, msg, verbose=False) -> str | None:
method check_msg (line 124) | def check_msg(self, msg, verbose=False):
method shutdown (line 134) | def shutdown(self):
method restart (line 142) | def restart(self):
method interrupt (line 147) | def interrupt(self):
method is_alive (line 152) | def is_alive(self):
function b64_2_img (line 156) | def b64_2_img(data):
function clean_ansi_codes (line 161) | def clean_ansi_codes(input_string):
function execute (line 166) | def execute(code, kernel: CodeKernel) -> tuple[str, str | Image.Image]:
function get_kernel (line 203) | def get_kernel():
function extract_code (line 208) | def extract_code(text: str) -> str:
function append_conversation (line 215) | def append_conversation(
function main (line 224) | def main(
FILE: composite_demo/demo_tool.py
function tool_call (line 31) | def tool_call(*args, **kwargs) -> dict:
function yaml_to_dict (line 39) | def yaml_to_dict(tools: str) -> list[dict] | None:
function extract_code (line 46) | def extract_code(text: str) -> str:
function append_conversation (line 54) | def append_conversation(
function main (line 63) | def main(
FILE: composite_demo/main.py
class Mode (line 26) | class Mode(str, Enum):
FILE: composite_demo/tool_registry.py
function register_tool (line 19) | def register_tool(func: callable):
function dispatch_tool (line 56) | def dispatch_tool(tool_name: str, tool_params: dict) -> str:
function get_tools (line 67) | def get_tools() -> dict:
function random_number_generator (line 74) | def random_number_generator(
function get_weather (line 93) | def get_weather(
function get_shell (line 120) | def get_shell(
FILE: finetune_demo/finetune_hf.py
class DataCollatorForSeq2Seq (line 42) | class DataCollatorForSeq2Seq(_DataCollatorForSeq2Seq):
method __call__ (line 43) | def __call__(self, features, return_tensors=None):
class Seq2SeqTrainer (line 70) | class Seq2SeqTrainer(_Seq2SeqTrainer):
method prediction_step (line 71) | def prediction_step(
function _resolve_path (line 106) | def _resolve_path(path: Union[str, Path]) -> Path:
function _sanity_check (line 110) | def _sanity_check(
function _get_yaml_parser (line 127) | def _get_yaml_parser() -> yaml.YAML:
class DataConfig (line 135) | class DataConfig(object):
method data_format (line 143) | def data_format(self) -> str:
method data_files (line 147) | def data_files(self) -> dict[NamedSplit, str]:
class FinetuningConfig (line 159) | class FinetuningConfig(object):
method __post_init__ (line 170) | def __post_init__(self):
method from_dict (line 183) | def from_dict(cls, **kwargs) -> 'FinetuningConfig':
method from_file (line 206) | def from_file(cls, path: Union[str, Path]) -> 'FinetuningConfig':
function _load_datasets (line 212) | def _load_datasets(
class DataManager (line 232) | class DataManager(object):
method __init__ (line 233) | def __init__(self, data_dir: str, data_config: DataConfig):
method _get_dataset (line 243) | def _get_dataset(self, split: NamedSplit) -> Optional[Dataset]:
method get_dataset (line 246) | def get_dataset(
function print_model_size (line 269) | def print_model_size(model: PreTrainedModel):
function process_batch (line 275) | def process_batch(
function process_batch_eval (line 329) | def process_batch_eval(
function _prepare_model_for_training (line 378) | def _prepare_model_for_training(model: nn.Module, use_cpu: bool):
function load_tokenizer_and_model (line 384) | def load_tokenizer_and_model(
function compute_metrics (line 419) | def compute_metrics(eval_preds: EvalPrediction, tokenizer: PreTrainedTok...
function main (line 443) | def main(
FILE: finetune_demo/inference_hf.py
function _resolve_path (line 23) | def _resolve_path(path: Union[str, Path]) -> Path:
function load_model_and_tokenizer (line 27) | def load_model_and_tokenizer(model_dir: Union[str, Path]) -> tuple[Model...
function main (line 46) | def main(
FILE: langchain_demo/ChatGLM3.py
class ChatGLM3 (line 8) | class ChatGLM3(LLM):
method __init__ (line 18) | def __init__(self):
method _llm_type (line 22) | def _llm_type(self) -> str:
method load_model (line 25) | def load_model(self, model_name_or_path=None):
method _tool_history (line 37) | def _tool_history(self, prompt: str):
method _extract_observation (line 81) | def _extract_observation(self, prompt: str):
method _extract_tool (line 89) | def _extract_tool(self):
method _call (line 127) | def _call(self, prompt: str, history: List = [], stop: Optional[List[s...
FILE: langchain_demo/tools/Calculator.py
class CalculatorInput (line 8) | class CalculatorInput(BaseModel):
class Calculator (line 12) | class Calculator(BaseTool, abc.ABC):
method __init__ (line 17) | def __init__(self):
method parameter_validation (line 20) | def parameter_validation(self, para: str):
method _run (line 34) | def _run(self, calculation: str) -> str:
FILE: langchain_demo/tools/DistanceConversion.py
class DistanceConversionInput (line 7) | class DistanceConversionInput(BaseModel):
class DistanceConverter (line 13) | class DistanceConverter(BaseTool, abc.ABC):
method __init__ (line 18) | def __init__(self):
method _run (line 21) | def _run(self, distance: float, unit: str, to_unit: str) -> str:
FILE: langchain_demo/tools/Weather.py
class WeatherInput (line 8) | class WeatherInput(BaseModel):
class Weather (line 12) | class Weather(BaseTool):
method __init__ (line 17) | def __init__(self):
method _run (line 20) | def _run(self, location: str) -> dict[str, Any]:
FILE: openai_api_demo/api_server.py
function lifespan (line 62) | async def lifespan(app: FastAPI):
class ModelCard (line 80) | class ModelCard(BaseModel):
class ModelList (line 90) | class ModelList(BaseModel):
class FunctionCallResponse (line 95) | class FunctionCallResponse(BaseModel):
class ChatMessage (line 100) | class ChatMessage(BaseModel):
class DeltaMessage (line 107) | class DeltaMessage(BaseModel):
class EmbeddingRequest (line 114) | class EmbeddingRequest(BaseModel):
class CompletionUsage (line 119) | class CompletionUsage(BaseModel):
class EmbeddingResponse (line 125) | class EmbeddingResponse(BaseModel):
class UsageInfo (line 134) | class UsageInfo(BaseModel):
class ChatCompletionRequest (line 140) | class ChatCompletionRequest(BaseModel):
class ChatCompletionResponseChoice (line 151) | class ChatCompletionResponseChoice(BaseModel):
class ChatCompletionResponseStreamChoice (line 157) | class ChatCompletionResponseStreamChoice(BaseModel):
class ChatCompletionResponse (line 163) | class ChatCompletionResponse(BaseModel):
function health (line 173) | async def health() -> Response:
function get_embeddings (line 179) | async def get_embeddings(request: EmbeddingRequest):
function list_models (line 216) | async def list_models():
function create_chat_completion (line 226) | async def create_chat_completion(request: ChatCompletionRequest):
function predict (line 343) | async def predict(model_id: str, params: dict):
function predict_stream (line 409) | def predict_stream(model_id, gen_params):
function parse_output_text (line 489) | async def parse_output_text(model_id: str, value: str):
function contains_custom_function (line 515) | def contains_custom_function(value: str) -> bool:
FILE: openai_api_demo/langchain_openai_api.py
function get_ai_response (line 16) | def get_ai_response(messages, user_input):
function continuous_conversation (line 28) | def continuous_conversation():
FILE: openai_api_demo/openai_api_request.py
function function_chat (line 19) | def function_chat():
function simple_chat (line 55) | def simple_chat(use_stream=True):
function embedding (line 86) | def embedding():
FILE: openai_api_demo/utils.py
class InvalidScoreLogitsProcessor (line 9) | class InvalidScoreLogitsProcessor(LogitsProcessor):
method __call__ (line 10) | def __call__(
function process_response (line 19) | def process_response(output: str, use_tool: bool = False) -> Union[str, ...
function generate_stream_chatglm3 (line 46) | def generate_stream_chatglm3(model: PreTrainedModel, tokenizer: PreTrain...
function process_chatglm_messages (line 122) | def process_chatglm_messages(messages, tools=None):
function generate_chatglm3 (line 164) | def generate_chatglm3(model: PreTrainedModel, tokenizer: PreTrainedToken...
function apply_stopping_strings (line 170) | def apply_stopping_strings(reply, stop_strings) -> Tuple[str, bool]:
FILE: tensorrt_llm_demo/tensorrt_llm_cli_demo.py
function find_engines (line 44) | def find_engines(dir: Path, model_name: str = "*", dtype: str = "*", tp_...
function parse_arguments (line 61) | def parse_arguments(args=None):
function main (line 85) | def main():
FILE: tools_using_demo/cli_demo_tool.py
function build_prompt (line 21) | def build_prompt(history):
function main (line 103) | def main():
FILE: tools_using_demo/openai_api_demo.py
function run_conversation (line 18) | def run_conversation(query: str, stream=False, tools=None, max_retry=5):
FILE: tools_using_demo/tool_register.py
function register_tool (line 15) | def register_tool(func: callable):
function dispatch_tool (line 53) | def dispatch_tool(tool_name: str, tool_params: dict) -> str:
function get_tools (line 64) | def get_tools() -> dict:
function random_number_generator (line 71) | def random_number_generator(
function get_weather (line 90) | def get_weather(
Condensed preview — 68 files, each showing path, character count, and a content snippet. Download the .json file or copy for the full structured content (435K chars).
[
{
"path": ".github/ISSUE_TEMPLATE/bug_report.yaml",
"chars": 3135,
"preview": "name: \"\\U0001F41B Bug Report\"\ndescription: Submit a bug report to help us improve ChatGLM3 / 提交一个 Bug 问题报告来帮助我们改进 ChatGL"
},
{
"path": ".github/ISSUE_TEMPLATE/feature-request.yaml",
"chars": 1008,
"preview": "name: \"\\U0001F680 Feature request\"\ndescription: Submit a request for a new ChatGLM3 feature / 提交一个新的 ChatGLM3 的功能建议\nlabe"
},
{
"path": ".github/PULL_REQUEST_TEMPLATE/pr_template.md",
"chars": 1333,
"preview": "# Raise valuable PR / 提出有价值的PR\n\n## Caution/ 注意事项:\nUsers should keep the following points in mind when submitting PRs:\n\n"
},
{
"path": ".gitignore",
"chars": 175,
"preview": "__pycache__\n\n# finetune_demo: generated & downloaded files\nfinetune_demo/output\nfinetune_demo/data\nfinetune_demo/formatt"
},
{
"path": "DEPLOYMENT.md",
"chars": 1246,
"preview": "## 低成本部署\n\n### 模型量化\n\n默认情况下，模型以 FP16 精度加载，运行上述代码需要大概 13GB 显存。如果你的 GPU 显存有限，可以尝试以量化方式加载模型，使用方法如下：\n\n```python\nmodel = AutoMo"
},
{
"path": "DEPLOYMENT_en.md",
"chars": 2304,
"preview": "## Low-Cost Deployment\n\n### Model Quantization\n\nBy default, the model is loaded with FP16 precision, running the above c"
},
{
"path": "Intel_device_demo/README.md",
"chars": 505,
"preview": "# Intel Device Demo\n\n本文件夹主要辅助开发者 在 Intel 设备上加速部署 ChatGLM3-6B 模型。\n\n## 1. 硬件准备\n本文件夹中的设备支持列表包括：\n- Intel CPU 系列, 包含个人CPU 和 服"
},
{
"path": "Intel_device_demo/ipex_llm_cpu_demo/api_server.py",
"chars": 17986,
"preview": "\"\"\"\nThis script implements an API for the ChatGLM3-6B model,\nformatted similarly to OpenAI's API (https://platform.opena"
},
{
"path": "Intel_device_demo/ipex_llm_cpu_demo/chatglm3_infer.py",
"chars": 1160,
"preview": "import time\nfrom ipex_llm.transformers import AutoModel\nfrom transformers import AutoTokenizer\n\nCHATGLM_V3_PROMPT_FORMAT"
},
{
"path": "Intel_device_demo/ipex_llm_cpu_demo/chatglm3_web_demo.py",
"chars": 3148,
"preview": "\"\"\"\nThis script creates an interactive web demo for the ChatGLM3-6B model using Gradio,\na Python library for building qu"
},
{
"path": "Intel_device_demo/ipex_llm_cpu_demo/generate.py",
"chars": 2639,
"preview": "import torch\nimport time\nimport argparse\nimport numpy as np\n\nfrom ipex_llm.transformers import AutoModel\nfrom modelscope"
},
{
"path": "Intel_device_demo/ipex_llm_cpu_demo/openai_api_request.py",
"chars": 2769,
"preview": "\"\"\"\nThis script is an example of using the OpenAI API to create various interactions with a ChatGLM3 model.\nIt includes "
},
{
"path": "Intel_device_demo/ipex_llm_cpu_demo/utils.py",
"chars": 6064,
"preview": "import gc\nimport json\nimport torch\nfrom transformers import PreTrainedModel, PreTrainedTokenizer\nfrom transformers.gener"
},
{
"path": "Intel_device_demo/openvino_demo/README.md",
"chars": 2276,
"preview": "\n# 使用 OpenVINO 部署ChatGLM3-6B 模型\n\n[OpenVINO](https://www.intel.com/content/www/us/en/developer/tools/openvino-toolkit/ove"
},
{
"path": "Intel_device_demo/openvino_demo/openvino_cli_demo.py",
"chars": 5492,
"preview": "import argparse\nfrom typing import List, Tuple\nfrom threading import Thread\nimport torch\nfrom optimum.intel.openvino imp"
},
{
"path": "LICENSE",
"chars": 11353,
"preview": " Apache License\n Version 2.0, January 2004\n "
},
{
"path": "MODEL_LICENSE",
"chars": 3702,
"preview": "The ChatGLM3-6B License\n\n1. 定义\n\n“许可方”是指分发其软件的 ChatGLM3-6B 模型团队。\n\n“软件”是指根据本许可提供的 ChatGLM3-6B 模型参数。\n\n2. 许可授予\n\n根据本许可的条款和条件，"
},
{
"path": "PROMPT.md",
"chars": 5061,
"preview": "## ChatGLM3 对话格式\n为了避免用户输入的注入攻击，以及统一 Code Interpreter，Tool & Agent 等任务的输入，ChatGLM3 采用了全新的对话格式。\n\n### 规定\n#### 整体结构\nChatGLM3"
},
{
"path": "PROMPT_en.md",
"chars": 7116,
"preview": "## ChatGLM3 Chat Format\nTo avoid injection attacks from user input, and to unify the input of Code Interpreter, Tool & A"
},
{
"path": "README.md",
"chars": 15694,
"preview": "# ChatGLM3\n\n<p align=\"center\">\n📄<a href=\"https://arxiv.org/pdf/2406.12793\" target=\"_blank\"> Report </a> • 🤗 <a href=\"htt"
},
{
"path": "README_en.md",
"chars": 22966,
"preview": "# ChatGLM3\n\n<p align=\"center\">\n📄<a href=\"https://arxiv.org/pdf/2406.12793\" target=\"_blank\"> Report </a> • 🤗 <a href=\"htt"
},
{
"path": "basic_demo/cli_batch_request_demo.py",
"chars": 2589,
"preview": "import os\nfrom typing import Optional, Union\nfrom transformers import AutoModel, AutoTokenizer, LogitsProcessorList\n\nMOD"
},
{
"path": "basic_demo/cli_demo.py",
"chars": 1991,
"preview": "import os\nimport platform\nfrom transformers import AutoTokenizer, AutoModel\n\nMODEL_PATH = os.environ.get('MODEL_PATH', '"
},
{
"path": "basic_demo/cli_demo_bad_word_ids.py",
"chars": 3423,
"preview": "\"\"\"\nThis script demonstrates how to use the `bad_words_ids` argument in the context of a conversational AI model to filt"
},
{
"path": "basic_demo/web_demo_gradio.py",
"chars": 6983,
"preview": "\"\"\"\nThis script creates an interactive web demo for the ChatGLM3-6B model using Gradio,\na Python library for building qu"
},
{
"path": "basic_demo/web_demo_streamlit.py",
"chars": 2923,
"preview": "\"\"\"\nThis script is a simple web demo based on Streamlit, showcasing the use of the ChatGLM3-6B model. For a more compreh"
},
{
"path": "composite_demo/.streamlit/config.toml",
"chars": 26,
"preview": "[theme]\nfont = \"monospace\""
},
{
"path": "composite_demo/README.md",
"chars": 1880,
"preview": "# ChatGLM3 Web Demo\n\n\n\n## 安装\n\n我们建议通过 [Conda](https://docs.conda.io/en/latest/) 进行环境管理。\n\n"
},
{
"path": "composite_demo/README_en.md",
"chars": 3567,
"preview": "# ChatGLM3 Web Demo\n\n\n\n## Installation\n\nWe recommend managing environments through [Cond"
},
{
"path": "composite_demo/client.py",
"chars": 8040,
"preview": "from __future__ import annotations\n\nimport os\nimport streamlit as st\nimport torch\n\nfrom collections.abc import Iterable\n"
},
{
"path": "composite_demo/conversation.py",
"chars": 4007,
"preview": "from dataclasses import dataclass\nfrom enum import auto, Enum\nimport json\n\nfrom PIL.Image import Image\nimport streamlit "
},
{
"path": "composite_demo/demo_chat.py",
"chars": 3003,
"preview": "import streamlit as st\nfrom streamlit.delta_generator import DeltaGenerator\n\nfrom client import get_client\nfrom conversa"
},
{
"path": "composite_demo/demo_ci.py",
"chars": 13035,
"preview": "import base64\nfrom io import BytesIO\nimport os\nfrom pprint import pprint\nimport queue\nimport re\nfrom subprocess import P"
},
{
"path": "composite_demo/demo_tool.py",
"chars": 8084,
"preview": "import re\nimport yaml\nfrom yaml import YAMLError\n\nimport streamlit as st\nfrom streamlit.delta_generator import DeltaGene"
},
{
"path": "composite_demo/main.py",
"chars": 2683,
"preview": "import streamlit as st\nst.set_page_config(\n page_title=\"ChatGLM3 Demo\",\n page_icon=\":robot:\",\n layout='centered"
},
{
"path": "composite_demo/requirements.txt",
"chars": 126,
"preview": "huggingface_hub>=0.19.4\npillow>=10.1.0\npyyaml>=6.0.1\nrequests>=2.31.0\nipykernel>=6.26.0\nipython>=8.18.1\njupyter_client>="
},
{
"path": "composite_demo/tool_registry.py",
"chars": 4340,
"preview": "\"\"\"\nThis code is the tool registration part. By registering the tool, the model can call the tool.\nThis code provides ex"
},
{
"path": "finetune_demo/README.md",
"chars": 9529,
"preview": "# ChatGLM3-6B 微调\n\n本目录提供 ChatGLM3-6B 模型的微调示例，包括全量微调和 P-Tuning v2。格式上，提供多轮对话微调样例和输入输出格式微调样例。\n\n如果将模型下载到了本地，本文和代码中的 `THUDM/c"
},
{
"path": "finetune_demo/README_en.md",
"chars": 15627,
"preview": "# ChatGLM3-6B finetune\n\nThis directory provides fine-tuning examples of the ChatGLM3-6B model, including full fine-tunin"
},
{
"path": "finetune_demo/configs/ds_zero_2.json",
"chars": 728,
"preview": "{\n \"fp16\": {\n \"enabled\": \"auto\",\n \"loss_scale\": 0,\n \"loss_scale_window\": 1000,\n \"initial_"
},
{
"path": "finetune_demo/configs/ds_zero_3.json",
"chars": 771,
"preview": "{\n \"train_micro_batch_size_per_gpu\": \"auto\",\n \"zero_allow_untested_optimizer\": true,\n \"bf16\": {\n \"enabled\": \"auto\""
},
{
"path": "finetune_demo/configs/lora.yaml",
"chars": 1178,
"preview": "data_config:\n train_file: train.json\n val_file: dev.json\n test_file: dev.json\n num_proc: 16\nmax_input_length: 256\nma"
},
{
"path": "finetune_demo/configs/ptuning_v2.yaml",
"chars": 1134,
"preview": "data_config:\n train_file: train.json\n val_file: dev.json\n test_file: dev.json\n num_proc: 16\nmax_input_length: 256\nma"
},
{
"path": "finetune_demo/configs/sft.yaml",
"chars": 987,
"preview": "data_config:\n train_file: train.json\n val_file: dev.json\n test_file: dev.json\n num_proc: 16\nmax_input_length: 256\nma"
},
{
"path": "finetune_demo/finetune_hf.py",
"chars": 20515,
"preview": "# -*- coding: utf-8 -*-\nimport os\nimport jieba\nimport dataclasses as dc\nimport functools\nfrom collections.abc import Cal"
},
{
"path": "finetune_demo/inference_hf.py",
"chars": 1626,
"preview": "#!/usr/bin/env python\n# -*- coding: utf-8 -*-\n\nfrom pathlib import Path\nfrom typing import Annotated, Union\n\nimport type"
},
{
"path": "finetune_demo/lora_finetune.ipynb",
"chars": 71176,
"preview": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"source\": [\n \"# 单卡GPU 进行 ChatGLM3-6B模型 LORA 高效微调\\n\",\n \"本 Cookbook"
},
{
"path": "finetune_demo/requirements.txt",
"chars": 131,
"preview": "jieba>=0.42.1\nruamel_yaml>=0.18.6\nrouge_chinese>=1.0.3\njupyter>=1.0.0\ndatasets>=2.18.0\npeft>=0.10.0\ndeepspeed==0.16.2\nmp"
},
{
"path": "langchain_demo/ChatGLM3.py",
"chars": 4898,
"preview": "import ast\nimport json\nfrom langchain.llms.base import LLM\nfrom transformers import AutoTokenizer, AutoModel, AutoConfig"
},
{
"path": "langchain_demo/main.py",
"chars": 3780,
"preview": "\"\"\"\nThis script demonstrates the use of the LangChain's StructuredChatAgent and AgentExecutor alongside various tools\n\nT"
},
{
"path": "langchain_demo/tools/Calculator.py",
"chars": 2207,
"preview": "import abc\nimport re\nfrom typing import Type\nfrom langchain.tools import BaseTool\nfrom pydantic import BaseModel, Field\n"
},
{
"path": "langchain_demo/tools/DistanceConversion.py",
"chars": 1654,
"preview": "import abc\nfrom typing import Type\nfrom langchain.tools import BaseTool\nfrom pydantic import BaseModel, Field\n\n\nclass Di"
},
{
"path": "langchain_demo/tools/Weather.py",
"chars": 1111,
"preview": "import os\nimport requests\n\nfrom typing import Type, Any\nfrom langchain.tools import BaseTool\nfrom pydantic import BaseMo"
},
{
"path": "openai_api_demo/api_server.py",
"chars": 18123,
"preview": "\"\"\"\nThis script implements an API for the ChatGLM3-6B model,\nformatted similarly to OpenAI's API (https://platform.opena"
},
{
"path": "openai_api_demo/docker-compose.yml",
"chars": 1907,
"preview": "version: \"3.6\"\n\nservices:\n glm3_api:\n image: python:3.10.13-slim\n restart: unless-s"
},
{
"path": "openai_api_demo/langchain_openai_api.py",
"chars": 1566,
"preview": "\"\"\"\nThis script is designed for interacting with a local GLM3 AI model using the `ChatGLM3` class\nfrom the `langchain_co"
},
{
"path": "openai_api_demo/openai_api_request.py",
"chars": 3013,
"preview": "\"\"\"\nThis script is an example of using the OpenAI API to create various interactions with a ChatGLM3 model.\nIt includes "
},
{
"path": "openai_api_demo/utils.py",
"chars": 6273,
"preview": "import gc\nimport json\nimport torch\nfrom transformers import PreTrainedModel, PreTrainedTokenizer\nfrom transformers.gener"
},
{
"path": "requirements.txt",
"chars": 498,
"preview": "# basic requirements\n\ntransformers==4.40.0\ncpm_kernels>=1.0.11\ntorch>=2.3.0\nvllm>=0.4.2\ngradio>=4.26.0\nsentencepiece>=0."
},
{
"path": "resources/WECHAT.md",
"chars": 193,
"preview": "<div align=\"center\">\n<img src=wechat.jpg width=\"60%\"/>\n\n<p> 扫码关注公众号，加入「ChatGLM交流群」 </p>\n<p> Scan the QR code to follow t"
},
{
"path": "tensorrt_llm_demo/README.md",
"chars": 5720,
"preview": "# 使用NVIDIA TensorRT-LLM部署ChatGLM3\n\n[TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM/tree/main)是NVIDIA开发的高性能推理框架，您可以"
},
{
"path": "tensorrt_llm_demo/tensorrt_llm_cli_demo.py",
"chars": 10885,
"preview": "\"\"\"\nThis script is a part of a larger project for generating text using large language models.\nIt includes functionaliti"
},
{
"path": "tools_using_demo/README.md",
"chars": 1976,
"preview": "# 工具调用\n本文档将介绍如何使用 ChatGLM3-6B 进行工具调用。目前只有 ChatGLM3-6B 模型支持工具调用，而 ChatGLM3-6B-Base 和 ChatGLM3-6B-32K 模型不支持。\n\n## 构建 System"
},
{
"path": "tools_using_demo/README_en.md",
"chars": 3206,
"preview": "# Tool Invocation\nThis document will introduce how to use the ChatGLM3-6B for tool invocation. Currently, only the ChatG"
},
{
"path": "tools_using_demo/cli_demo_tool.py",
"chars": 4041,
"preview": "\"\"\"\nThis demo script is designed for interacting with the ChatGLM3-6B in Function, to show Function Call capabilities.\n\""
},
{
"path": "tools_using_demo/openai_api_demo.py",
"chars": 3163,
"preview": "import json\n\nfrom openai import OpenAI\nfrom colorama import init, Fore\nfrom loguru import logger\n\nfrom tool_register imp"
},
{
"path": "tools_using_demo/tool_register.py",
"chars": 3613,
"preview": "\"\"\"\n这段代码是工具注册的部分，通过注册工具，让模型实现工具调用\n\"\"\"\nimport inspect\nimport traceback\nfrom copy import deepcopy\nfrom pprint import pform"
},
{
"path": "update_requirements.sh",
"chars": 240,
"preview": "#!/bin/bash\n\npython -m pip install --upgrade pip -i https://pypi.tuna.tsinghua.edu.cn/simple\n\nwhile read requirement; do"
}
]
About this extraction
This page contains the full source code of the THUDM/ChatGLM3 GitHub repository, extracted and formatted as plain text for AI agents and large language models (LLMs). The extraction includes 68 files (376.3 KB), approximately 110.0k tokens, and a symbol index with 194 extracted functions, classes, methods, constants, and types. Use this with OpenClaw, Claude, ChatGPT, Cursor, Windsurf, or any other AI tool that accepts text input. You can copy the full output to your clipboard or download it as a .txt file.
Extracted by GitExtract — free GitHub repo to text converter for AI. Built by Nikandr Surkov.