三步上手 LLaMA2，一起玩！相关博客教程已更新，**同样欢迎“一键三连”** 🌟🌟🌟。
> 使用 Docker 快速上手，本地部署 7B 或 13B 官方模型，或者 7B 中文模型。
### 博客教程
| 类型 | 显存需求 | 特点 | 教程地址 | 教程时间 |
| --- | --- | --- | --- | --- |
| 官方版（英文） | 8~14GB | 原汁原味 | [使用 Docker 快速上手官方版 LLaMA2 开源大模型](https://soulteary.com/2023/07/21/use-docker-to-quickly-get-started-with-the-official-version-of-llama2-open-source-large-model.html) | 2023.07.21 |
| LinkSoul 中文版（双语）| 8~14GB | 支持中文 | [使用 Docker 快速上手中文版 LLaMA2 开源大模型](https://soulteary.com/2023/07/21/use-docker-to-quickly-get-started-with-the-chinese-version-of-llama2-open-source-large-model.html) | 2023.07.21 |
| Transformers 量化（中文/官方） | 5GB | 加速推理、节约显存 | [使用 Transformers 量化 Meta AI LLaMA2 中文版大模型](https://soulteary.com/2023/07/22/quantizing-meta-ai-llama2-chinese-version-large-models-using-transformers.html) | 2023.07.22 |
| GGML (Llama.cpp) 量化 （中文/官方）| 可以不需要显存 | CPU 推理 | [构建能够使用 CPU 运行的 MetaAI LLaMA2 中文大模型](https://soulteary.com/2023/07/23/build-llama2-chinese-large-model-that-can-run-on-cpu.html) | 2023.07.23 |
你可以参考项目代码，举一反三，把模型跑起来，接入到你想玩的地方，包括并不局限于支持 LLaMA 1代的各种开源软件中。
## 预览图
## 使用方法
1. 一条命令，从项目中构建官方版（7B或13B）模型镜像，或中文版镜像（7B或INT4量化版）：
```bash
# 7B
bash scripts/make-7b.sh
# 或 13B
bash scripts/make-13b.sh
# 或 7B Chinese
bash scripts/make-7b-cn.sh
# 或 7B Chinese 4bit
bash scripts/make-7b-cn-4bit.sh
```
2. 选择适合你的命令，从 HuggingFace 下载 LLaMA2 或中文模型：
```bash
# MetaAI LLaMA2 Models (10~14GB vRAM)
git clone https://huggingface.co/meta-llama/Llama-2-7b-chat-hf
git clone https://huggingface.co/meta-llama/Llama-2-13b-chat-hf
mkdir meta-llama
mv Llama-2-7b-chat-hf meta-llama/
mv Llama-2-13b-chat-hf meta-llama/
# 或 Chinese LLaMA2 (10~14GB vRAM)
git clone https://huggingface.co/LinkSoul/Chinese-Llama-2-7b
mkdir LinkSoul
mv Chinese-Llama-2-7b LinkSoul/
# 或 Chinese LLaMA2 4BIT (5GB vRAM)
git clone https://huggingface.co/soulteary/Chinese-Llama-2-7b-4bit
mkdir soulteary
mv Chinese-Llama-2-7b-4bit soulteary/
```
将下载好的模型，保持在一个正确的目录结构中。
```bash
tree -L 2 meta-llama
soulteary
└── ...
LinkSoul
└── ...
meta-llama
├── Llama-2-13b-chat-hf
│ ├── added_tokens.json
│ ├── config.json
│ ├── generation_config.json
│ ├── LICENSE.txt
│ ├── model-00001-of-00003.safetensors
│ ├── model-00002-of-00003.safetensors
│ ├── model-00003-of-00003.safetensors
│ ├── model.safetensors.index.json
│ ├── pytorch_model-00001-of-00003.bin
│ ├── pytorch_model-00002-of-00003.bin
│ ├── pytorch_model-00003-of-00003.bin
│ ├── pytorch_model.bin.index.json
│ ├── README.md
│ ├── Responsible-Use-Guide.pdf
│ ├── special_tokens_map.json
│ ├── tokenizer_config.json
│ ├── tokenizer.model
│ └── USE_POLICY.md
└── Llama-2-7b-chat-hf
├── added_tokens.json
├── config.json
├── generation_config.json
├── LICENSE.txt
├── model-00001-of-00002.safetensors
├── model-00002-of-00002.safetensors
├── model.safetensors.index.json
├── models--meta-llama--Llama-2-7b-chat-hf
├── pytorch_model-00001-of-00003.bin
├── pytorch_model-00002-of-00003.bin
├── pytorch_model-00003-of-00003.bin
├── pytorch_model.bin.index.json
├── README.md
├── special_tokens_map.json
├── tokenizer_config.json
├── tokenizer.json
├── tokenizer.model
└── USE_POLICY.md
```
3. 选择使用下面的适合你的命令，一键运行 LLaMA2 模型应用：
```bash
# 7B
bash scripts/run-7b.sh
# 或 13B
bash scripts/run-13b.sh
# 或 Chinese 7B
bash scripts/run-7b-cn.sh
# 或 Chinese 7B 4BIT
bash scripts/run-7b-cn-4bit.sh
```
模型运行之后，在浏览器中访问 `http://localhost7860` 或者 `http://你的IP地址:7860` 就可以开始玩了。
## 相关项目
- MetaAI LLaMA2: https://ai.meta.com/llama/ ❤️
- Meta LLaMA2 7B Chat: https://huggingface.co/meta-llama/Llama-2-7b-chat
- Meta LLaMA2 13B Chat: https://huggingface.co/meta-llama/Llama-2-13b-chat
- Chinese LLaMA2 7B: https://huggingface.co/LinkSoul/Chinese-Llama-2-7b ❤️
- Chinese LLaMA2 7B GGML q4: https://huggingface.co/soulteary/Chinese-Llama-2-7b-ggml-q4
- LLaMA2 GGML Converter: https://hub.docker.com/r/soulteary/llama2
================================================
FILE: README_EN.md
================================================
# Docker LLaMA2 Chat / 羊驼二代
[](https://huggingface.co/meta-llama) [](https://huggingface.co/soulteary/Chinese-Llama-2-7b-4bit) [](https://huggingface.co/soulteary/Chinese-Llama-2-7b-ggml-q4) [](https://github.com/soulteary/docker-llama2-chat/blob/main/LICENSE)
Play! Together! **ONLY 3 STEPS!**
Get started quickly, locally using the 7B or 13B models, using Docker.
- Meta Llama2, tested by 4090, and costs 8~14GB vRAM.
- Chinese Llama2 quantified, tested by 4090, and costs 5GB vRAM.
- Use GGML(LLaMA.cpp), just use CPU play it.
## Preview
## Blogs
- [Use Docker to quickly get started with the official version of Llama2 Open-source Large Model](https://soulteary.com/2023/07/21/use-docker-to-quickly-get-started-with-the-official-version-of-llama2-open-source-large-model.html)
- [Use Docker to quickly get started with the chinese version of Llama2 Open-source Large Model](https://soulteary.com/2023/07/21/use-docker-to-quickly-get-started-with-the-chinese-version-of-llama2-open-source-large-model.html)
- [Quantizing MetaAI Llama2 chinese version large models using Transformers](https://soulteary.com/2023/07/22/quantizing-meta-ai-llama2-chinese-version-large-models-using-transformers.html)
- [Build Llama2 chinese large model that can run on CPU](https://soulteary.com/2023/07/23/build-llama2-chinese-large-model-that-can-run-on-cpu.html)
## Usage
1. Build LLaMA2 Docker image for 7B / 13B (official), 7B or 7B INT4 (chinese):
```bash
# 7B
bash scripts/make-7b.sh
# OR 13B
bash scripts/make-13b.sh
# OR 7B Chinese
bash scripts/make-7b-cn.sh
# OR 7B Chinese 4bit
bash scripts/make-7b-cn-4bit.sh
```
2. Download LLaMA2 Models from HuggingFace, or chinese models.
```bash
# MetaAI LLaMA2 Models (10~14GB vRAM)
git clone https://huggingface.co/meta-llama/Llama-2-7b-chat-hf
git clone https://huggingface.co/meta-llama/Llama-2-13b-chat-hf
mkdir meta-llama
mv Llama-2-7b-chat-hf meta-llama/
mv Llama-2-13b-chat-hf meta-llama/
# OR Chinese LLaMA2 (10~14GB vRAM)
git clone https://huggingface.co/LinkSoul/Chinese-Llama-2-7b
mkdir LinkSoul
mv Chinese-Llama-2-7b LinkSoul/
# OR Chinese LLaMA2 4BIT (5GB vRAM)
git clone https://huggingface.co/soulteary/Chinese-Llama-2-7b-4bit
mkdir soulteary
mv Chinese-Llama-2-7b-4bit soulteary/
```
keep the correct directory structure.
```bash
tree -L 2 meta-llama
soulteary
└── ...
LinkSoul
└── ...
meta-llama
├── Llama-2-13b-chat-hf
│ ├── added_tokens.json
│ ├── config.json
│ ├── generation_config.json
│ ├── LICENSE.txt
│ ├── model-00001-of-00003.safetensors
│ ├── model-00002-of-00003.safetensors
│ ├── model-00003-of-00003.safetensors
│ ├── model.safetensors.index.json
│ ├── pytorch_model-00001-of-00003.bin
│ ├── pytorch_model-00002-of-00003.bin
│ ├── pytorch_model-00003-of-00003.bin
│ ├── pytorch_model.bin.index.json
│ ├── README.md
│ ├── Responsible-Use-Guide.pdf
│ ├── special_tokens_map.json
│ ├── tokenizer_config.json
│ ├── tokenizer.model
│ └── USE_POLICY.md
└── Llama-2-7b-chat-hf
├── added_tokens.json
├── config.json
├── generation_config.json
├── LICENSE.txt
├── model-00001-of-00002.safetensors
├── model-00002-of-00002.safetensors
├── model.safetensors.index.json
├── models--meta-llama--Llama-2-7b-chat-hf
├── pytorch_model-00001-of-00003.bin
├── pytorch_model-00002-of-00003.bin
├── pytorch_model-00003-of-00003.bin
├── pytorch_model.bin.index.json
├── README.md
├── special_tokens_map.json
├── tokenizer_config.json
├── tokenizer.json
├── tokenizer.model
└── USE_POLICY.md
```
3. Run Llama2 model in docker command:
```bash
# 7B
bash scripts/run-7b.sh
# OR 13B
bash scripts/run-13b.sh
# OR Chinese 7B
bash scripts/run-7b-cn.sh
# OR Chinese 7B 4BIT
bash scripts/run-7b-cn-4bit.sh
```
enjoy, open `http://localhost7860` or `http://ip:7860` and play with the LLaMA2!
## Credit
- MetaAI LLaMA2: https://ai.meta.com/llama/ ❤️
- Meta LLaMA2 7B Chat: https://huggingface.co/meta-llama/Llama-2-7b-chat
- Meta LLaMA2 13B Chat: https://huggingface.co/meta-llama/Llama-2-13b-chat
- Chinese LLaMA2 7B: https://huggingface.co/LinkSoul/Chinese-Llama-2-7b ❤️
- Chinese LLaMA2 7B GGML q4: https://huggingface.co/soulteary/Chinese-Llama-2-7b-ggml-q4
- LLaMA2 GGML Converter: https://hub.docker.com/r/soulteary/llama2
================================================
FILE: docker/Dockerfile.13b
================================================
FROM soulteary/llama2:base
COPY llama2-13b/* ./
CMD ["python", "app.py"]
================================================
FILE: docker/Dockerfile.7b
================================================
FROM soulteary/llama2:base
COPY llama2-7b/* ./
CMD ["python", "app.py"]
================================================
FILE: docker/Dockerfile.7b-cn
================================================
FROM soulteary/llama2:base
COPY llama2-7b-cn/* ./
CMD ["python", "app.py"]
================================================
FILE: docker/Dockerfile.7b-cn-4bit
================================================
FROM soulteary/llama2:base
COPY llama2-7b-cn-4bit/* ./
CMD ["python", "app.py"]
================================================
FILE: docker/Dockerfile.base
================================================
FROM nvcr.io/nvidia/pytorch:23.06-py3
RUN pip config set global.index-url https://pypi.tuna.tsinghua.edu.cn/simple && \
pip install accelerate==0.21.0 bitsandbytes==0.40.2 gradio==3.37.0 protobuf==3.20.3 scipy==1.11.1 sentencepiece==0.1.99 transformers==4.31.0
WORKDIR /app
================================================
FILE: llama.cpp/Dockerfile.converter
================================================
FROM alpine:3.18 as code
RUN apk add --no-cache wget
WORKDIR /app
ARG CODE_BASE=eb542d3
ENV ENV_CODE_BASE=${CODE_BASE}
RUN wget https://github.com/ggerganov/llama.cpp/archive/refs/tags/master-${ENV_CODE_BASE}.tar.gz && \
tar zxvf master-${ENV_CODE_BASE}.tar.gz && \
rm -rf master-${ENV_CODE_BASE}.tar.gz
RUN mv llama.cpp-master-${ENV_CODE_BASE} llama.cpp
FROM python:3.11.4-slim-bullseye as base
COPY --from=code /app/llama.cpp /app/llama.cpp
WORKDIR /app/llama.cpp
ENV DEBIAN_FRONTEND="noninteractive"
RUN apt-get update && apt-get install -y --no-install-recommends build-essential && rm -rf /var/lib/apt/lists/*
RUN make -j$(nproc)
FROM python:3.11.4-slim-bullseye as runtime
RUN pip3 install numpy==1.24 sentencepiece==0.1.98
COPY --from=base /app/llama.cpp/ /app/llama.cpp/
WORKDIR /app/llama.cpp/
================================================
FILE: llama.cpp/Dockerfile.runtime
================================================
FROM alpine:3.18 as code
RUN apk add --no-cache wget
WORKDIR /app
ARG CODE_BASE=d2a4366
ENV ENV_CODE_BASE=${CODE_BASE}
RUN wget https://github.com/ggerganov/llama.cpp/archive/refs/tags/master-${ENV_CODE_BASE}.tar.gz && \
tar zxvf master-${ENV_CODE_BASE}.tar.gz && \
rm -rf master-${ENV_CODE_BASE}.tar.gz
RUN mv llama.cpp-master-${ENV_CODE_BASE} llama.cpp
FROM python:3.11.4-slim-bullseye as base
COPY --from=code /app/llama.cpp /app/llama.cpp
WORKDIR /app/llama.cpp
ENV DEBIAN_FRONTEND="noninteractive"
RUN apt-get update && apt-get install -y --no-install-recommends build-essential && rm -rf /var/lib/apt/lists/*
RUN make -j$(nproc)
FROM python:3.11.4-slim-bullseye as runtime
COPY --from=base /app/llama.cpp/LICENSE /app/llama.cpp/LICENSE
COPY --from=base /app/llama.cpp/main /app/llama.cpp/main
COPY --from=base /app/llama.cpp/prompts /app/llama.cpp/prompts
WORKDIR /app/llama.cpp/
================================================
FILE: llama2-13b/app.py
================================================
from typing import Iterator
import gradio as gr
import torch
from model import run
DEFAULT_SYSTEM_PROMPT = """\
You are a helpful, respectful and honest assistant. Always answer as helpfully as possible, while being safe. Your answers should not include any harmful, unethical, racist, sexist, toxic, dangerous, or illegal content. Please ensure that your responses are socially unbiased and positive in nature.\n\nIf a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. If you don't know the answer to a question, please don't share false information.\
"""
MAX_MAX_NEW_TOKENS = 2048
DEFAULT_MAX_NEW_TOKENS = 1024
DESCRIPTION = """
# Llama-2 13B Chat
This Space demonstrates model [Llama-2-13b-chat](https://huggingface.co/meta-llama/Llama-2-13b-chat) by Meta, a Llama 2 model with 13B parameters fine-tuned for chat instructions. Feel free to play with it, or duplicate to run generations without a queue! If you want to run your own service, you can also [deploy the model on Inference Endpoints](https://huggingface.co/inference-endpoints).
🔎 For more details about the Llama 2 family of models and how to use them with `transformers`, take a look [at our blog post](https://huggingface.co/blog/llama2).
🔨 Looking for an even more powerful model? Check out the large [**70B** model demo](https://huggingface.co/spaces/ysharma/Explore_llamav2_with_TGI).
🐇 For a smaller model that you can run on many GPUs, check our [7B model demo](https://huggingface.co/spaces/huggingface-projects/llama-2-7b-chat).
"""
LICENSE = """
---
As a derivate work of [Llama-2-13b-chat](https://huggingface.co/meta-llama/Llama-2-13b-chat) by Meta,
this demo is governed by the original [license](https://huggingface.co/spaces/huggingface-projects/llama-2-13b-chat/blob/main/LICENSE.txt) and [acceptable use policy](https://huggingface.co/spaces/huggingface-projects/llama-2-13b-chat/blob/main/USE_POLICY.md).
"""
if not torch.cuda.is_available():
DESCRIPTION += '\nRunning on CPU 🥶 This demo does not work on CPU.
' def clear_and_save_textbox(message: str) -> tuple[str, str]: return '', message def display_input(message: str, history: list[tuple[str, str]]) -> list[tuple[str, str]]: history.append((message, '')) return history def delete_prev_fn( history: list[tuple[str, str]]) -> tuple[list[tuple[str, str]], str]: try: message, _ = history.pop() except IndexError: message = '' return history, message or '' def generate( message: str, history_with_input: list[tuple[str, str]], system_prompt: str, max_new_tokens: int, top_p: float, temperature: float, top_k: int, ) -> Iterator[list[tuple[str, str]]]: if max_new_tokens > MAX_MAX_NEW_TOKENS: raise ValueError history = history_with_input[:-1] generator = run(message, history, system_prompt, max_new_tokens, temperature, top_p, top_k) try: first_response = next(generator) yield history + [(message, first_response)] except StopIteration: yield history + [(message, '')] for response in generator: yield history + [(message, response)] def process_example(message: str) -> tuple[str, list[tuple[str, str]]]: generator = generate(message, [], DEFAULT_SYSTEM_PROMPT, 1024, 0.95, 1, 1000) for x in generator: pass return '', x with gr.Blocks(css='style.css') as demo: gr.Markdown(DESCRIPTION) gr.DuplicateButton(value='Duplicate Space for private use', elem_id='duplicate-button') with gr.Group(): chatbot = gr.Chatbot(label='Chatbot') with gr.Row(): textbox = gr.Textbox( container=False, show_label=False, placeholder='Type a message...', scale=10, ) submit_button = gr.Button('Submit', variant='primary', scale=1, min_width=0) with gr.Row(): retry_button = gr.Button('🔄 Retry', variant='secondary') undo_button = gr.Button('↩️ Undo', variant='secondary') clear_button = gr.Button('🗑️ Clear', variant='secondary') saved_input = gr.State() with gr.Accordion(label='Advanced options', open=False): system_prompt = gr.Textbox(label='System prompt', value=DEFAULT_SYSTEM_PROMPT, lines=6) max_new_tokens = gr.Slider( label='Max new tokens', minimum=1, maximum=MAX_MAX_NEW_TOKENS, step=1, value=DEFAULT_MAX_NEW_TOKENS, ) temperature = gr.Slider( label='Temperature', minimum=0.1, maximum=4.0, step=0.1, value=1.0, ) top_p = gr.Slider( label='Top-p (nucleus sampling)', minimum=0.05, maximum=1.0, step=0.05, value=0.95, ) top_k = gr.Slider( label='Top-k', minimum=1, maximum=1000, step=1, value=50, ) gr.Examples( examples=[ 'Hello there! How are you doing?', 'Can you explain briefly to me what is the Python programming language?', 'Explain the plot of Cinderella in a sentence.', 'How many hours does it take a man to eat a Helicopter?', "Write a 100-word article on 'Benefits of Open-Source in AI research'", ], inputs=textbox, outputs=[textbox, chatbot], fn=process_example, cache_examples=True, ) gr.Markdown(LICENSE) textbox.submit( fn=clear_and_save_textbox, inputs=textbox, outputs=[textbox, saved_input], api_name=False, queue=False, ).then( fn=display_input, inputs=[saved_input, chatbot], outputs=chatbot, api_name=False, queue=False, ).then( fn=generate, inputs=[ saved_input, chatbot, system_prompt, max_new_tokens, temperature, top_p, top_k, ], outputs=chatbot, api_name=False, ) button_event_preprocess = submit_button.click( fn=clear_and_save_textbox, inputs=textbox, outputs=[textbox, saved_input], api_name=False, queue=False, ).then( fn=display_input, inputs=[saved_input, chatbot], outputs=chatbot, api_name=False, queue=False, ).then( fn=generate, inputs=[ saved_input, chatbot, system_prompt, max_new_tokens, temperature, top_p, top_k, ], outputs=chatbot, api_name=False, ) retry_button.click( fn=delete_prev_fn, inputs=chatbot, outputs=[chatbot, saved_input], api_name=False, queue=False, ).then( fn=display_input, inputs=[saved_input, chatbot], outputs=chatbot, api_name=False, queue=False, ).then( fn=generate, inputs=[ saved_input, chatbot, max_new_tokens, temperature, top_p, top_k, ], outputs=chatbot, api_name=False, ) undo_button.click( fn=delete_prev_fn, inputs=chatbot, outputs=[chatbot, saved_input], api_name=False, queue=False, ).then( fn=lambda x: x, inputs=[saved_input], outputs=textbox, api_name=False, queue=False, ) clear_button.click( fn=lambda: ([], ''), outputs=[chatbot, saved_input], queue=False, api_name=False, ) demo.queue(max_size=20).launch(server_name="0.0.0.0") ================================================ FILE: llama2-13b/model.py ================================================ from threading import Thread from typing import Iterator import torch from transformers import AutoConfig, AutoModelForCausalLM, AutoTokenizer, TextIteratorStreamer model_id = 'meta-llama/Llama-2-13b-chat-hf' if torch.cuda.is_available(): config = AutoConfig.from_pretrained(model_id) config.pretraining_tp = 1 model = AutoModelForCausalLM.from_pretrained( model_id, local_files_only=True, config=config, torch_dtype=torch.float16, load_in_4bit=True, device_map='auto' ) else: model = None tokenizer = AutoTokenizer.from_pretrained(model_id) def get_prompt(message: str, chat_history: list[tuple[str, str]], system_prompt: str) -> str: texts = [f'[INST] <Running on CPU 🥶 This demo does not work on CPU.
' def clear_and_save_textbox(message: str) -> tuple[str, str]: return '', message def display_input(message: str, history: list[tuple[str, str]]) -> list[tuple[str, str]]: history.append((message, '')) return history def delete_prev_fn( history: list[tuple[str, str]]) -> tuple[list[tuple[str, str]], str]: try: message, _ = history.pop() except IndexError: message = '' return history, message or '' def generate( message: str, history_with_input: list[tuple[str, str]], system_prompt: str, max_new_tokens: int, top_p: float, temperature: float, top_k: int, ) -> Iterator[list[tuple[str, str]]]: if max_new_tokens > MAX_MAX_NEW_TOKENS: raise ValueError history = history_with_input[:-1] generator = run(message, history, system_prompt, max_new_tokens, temperature, top_p, top_k) try: first_response = next(generator) yield history + [(message, first_response)] except StopIteration: yield history + [(message, '')] for response in generator: yield history + [(message, response)] def process_example(message: str) -> tuple[str, list[tuple[str, str]]]: generator = generate(message, [], DEFAULT_SYSTEM_PROMPT, 1024, 0.95, 1, 1000) for x in generator: pass return '', x with gr.Blocks(css='style.css') as demo: gr.Markdown(DESCRIPTION) gr.DuplicateButton(value='Duplicate Space for private use', elem_id='duplicate-button') with gr.Group(): chatbot = gr.Chatbot(label='Chatbot') with gr.Row(): textbox = gr.Textbox( container=False, show_label=False, placeholder='Type a message...', scale=10, ) submit_button = gr.Button('Submit', variant='primary', scale=1, min_width=0) with gr.Row(): retry_button = gr.Button('🔄 Retry', variant='secondary') undo_button = gr.Button('↩️ Undo', variant='secondary') clear_button = gr.Button('🗑️ Clear', variant='secondary') saved_input = gr.State() with gr.Accordion(label='Advanced options', open=False): system_prompt = gr.Textbox(label='System prompt', value=DEFAULT_SYSTEM_PROMPT, lines=6) max_new_tokens = gr.Slider( label='Max new tokens', minimum=1, maximum=MAX_MAX_NEW_TOKENS, step=1, value=DEFAULT_MAX_NEW_TOKENS, ) temperature = gr.Slider( label='Temperature', minimum=0.1, maximum=4.0, step=0.1, value=1.0, ) top_p = gr.Slider( label='Top-p (nucleus sampling)', minimum=0.05, maximum=1.0, step=0.05, value=0.95, ) top_k = gr.Slider( label='Top-k', minimum=1, maximum=1000, step=1, value=50, ) gr.Examples( examples=[ 'Hello there! How are you doing?', 'Can you explain briefly to me what is the Python programming language?', 'Explain the plot of Cinderella in a sentence.', 'How many hours does it take a man to eat a Helicopter?', "Write a 100-word article on 'Benefits of Open-Source in AI research'", ], inputs=textbox, outputs=[textbox, chatbot], fn=process_example, cache_examples=True, ) gr.Markdown(LICENSE) textbox.submit( fn=clear_and_save_textbox, inputs=textbox, outputs=[textbox, saved_input], api_name=False, queue=False, ).then( fn=display_input, inputs=[saved_input, chatbot], outputs=chatbot, api_name=False, queue=False, ).then( fn=generate, inputs=[ saved_input, chatbot, system_prompt, max_new_tokens, temperature, top_p, top_k, ], outputs=chatbot, api_name=False, ) button_event_preprocess = submit_button.click( fn=clear_and_save_textbox, inputs=textbox, outputs=[textbox, saved_input], api_name=False, queue=False, ).then( fn=display_input, inputs=[saved_input, chatbot], outputs=chatbot, api_name=False, queue=False, ).then( fn=generate, inputs=[ saved_input, chatbot, system_prompt, max_new_tokens, temperature, top_p, top_k, ], outputs=chatbot, api_name=False, ) retry_button.click( fn=delete_prev_fn, inputs=chatbot, outputs=[chatbot, saved_input], api_name=False, queue=False, ).then( fn=display_input, inputs=[saved_input, chatbot], outputs=chatbot, api_name=False, queue=False, ).then( fn=generate, inputs=[ saved_input, chatbot, max_new_tokens, temperature, top_p, top_k, ], outputs=chatbot, api_name=False, ) undo_button.click( fn=delete_prev_fn, inputs=chatbot, outputs=[chatbot, saved_input], api_name=False, queue=False, ).then( fn=lambda x: x, inputs=[saved_input], outputs=textbox, api_name=False, queue=False, ) clear_button.click( fn=lambda: ([], ''), outputs=[chatbot, saved_input], queue=False, api_name=False, ) demo.queue(max_size=20).launch(server_name="0.0.0.0") ================================================ FILE: llama2-7b/model.py ================================================ from threading import Thread from typing import Iterator import torch from transformers import AutoModelForCausalLM, AutoTokenizer, TextIteratorStreamer model_id = 'meta-llama/Llama-2-7b-chat-hf' if torch.cuda.is_available(): model = AutoModelForCausalLM.from_pretrained( model_id, local_files_only=True, torch_dtype=torch.float16, device_map='auto' ) else: model = None tokenizer = AutoTokenizer.from_pretrained(model_id) def get_prompt(message: str, chat_history: list[tuple[str, str]], system_prompt: str) -> str: texts = [f'[INST] <Running on CPU 🥶 This demo does not work on CPU.
' def clear_and_save_textbox(message: str) -> tuple[str, str]: return '', message def display_input(message: str, history: list[tuple[str, str]]) -> list[tuple[str, str]]: history.append((message, '')) return history def delete_prev_fn( history: list[tuple[str, str]]) -> tuple[list[tuple[str, str]], str]: try: message, _ = history.pop() except IndexError: message = '' return history, message or '' def generate( message: str, history_with_input: list[tuple[str, str]], system_prompt: str, max_new_tokens: int, top_p: float, temperature: float, top_k: int, ) -> Iterator[list[tuple[str, str]]]: if max_new_tokens > MAX_MAX_NEW_TOKENS: raise ValueError history = history_with_input[:-1] generator = run(message, history, system_prompt, max_new_tokens, temperature, top_p, top_k) try: first_response = next(generator) yield history + [(message, first_response)] except StopIteration: yield history + [(message, '')] for response in generator: yield history + [(message, response)] def process_example(message: str) -> tuple[str, list[tuple[str, str]]]: generator = generate(message, [], DEFAULT_SYSTEM_PROMPT, 1024, 0.95, 1, 1000) for x in generator: pass return '', x with gr.Blocks(css='style.css') as demo: gr.Markdown(DESCRIPTION) gr.DuplicateButton(value='Duplicate Space for private use', elem_id='duplicate-button') with gr.Group(): chatbot = gr.Chatbot(label='Chatbot') with gr.Row(): textbox = gr.Textbox( container=False, show_label=False, placeholder='Type a message...', scale=10, ) submit_button = gr.Button('Submit', variant='primary', scale=1, min_width=0) with gr.Row(): retry_button = gr.Button('🔄 Retry', variant='secondary') undo_button = gr.Button('↩️ Undo', variant='secondary') clear_button = gr.Button('🗑️ Clear', variant='secondary') saved_input = gr.State() with gr.Accordion(label='Advanced options', open=False): system_prompt = gr.Textbox(label='System prompt', value=DEFAULT_SYSTEM_PROMPT, lines=6) max_new_tokens = gr.Slider( label='Max new tokens', minimum=1, maximum=MAX_MAX_NEW_TOKENS, step=1, value=DEFAULT_MAX_NEW_TOKENS, ) temperature = gr.Slider( label='Temperature', minimum=0.1, maximum=4.0, step=0.1, value=1.0, ) top_p = gr.Slider( label='Top-p (nucleus sampling)', minimum=0.05, maximum=1.0, step=0.05, value=0.95, ) top_k = gr.Slider( label='Top-k', minimum=1, maximum=1000, step=1, value=50, ) gr.Examples( examples=[ 'Hello there! How are you doing?', 'Can you explain briefly to me what is the Python programming language?', 'Explain the plot of Cinderella in a sentence.', 'How many hours does it take a man to eat a Helicopter?', "Write a 100-word article on 'Benefits of Open-Source in AI research'", ], inputs=textbox, outputs=[textbox, chatbot], fn=process_example, cache_examples=True, ) gr.Markdown(LICENSE) textbox.submit( fn=clear_and_save_textbox, inputs=textbox, outputs=[textbox, saved_input], api_name=False, queue=False, ).then( fn=display_input, inputs=[saved_input, chatbot], outputs=chatbot, api_name=False, queue=False, ).then( fn=generate, inputs=[ saved_input, chatbot, system_prompt, max_new_tokens, temperature, top_p, top_k, ], outputs=chatbot, api_name=False, ) button_event_preprocess = submit_button.click( fn=clear_and_save_textbox, inputs=textbox, outputs=[textbox, saved_input], api_name=False, queue=False, ).then( fn=display_input, inputs=[saved_input, chatbot], outputs=chatbot, api_name=False, queue=False, ).then( fn=generate, inputs=[ saved_input, chatbot, system_prompt, max_new_tokens, temperature, top_p, top_k, ], outputs=chatbot, api_name=False, ) retry_button.click( fn=delete_prev_fn, inputs=chatbot, outputs=[chatbot, saved_input], api_name=False, queue=False, ).then( fn=display_input, inputs=[saved_input, chatbot], outputs=chatbot, api_name=False, queue=False, ).then( fn=generate, inputs=[ saved_input, chatbot, max_new_tokens, temperature, top_p, top_k, ], outputs=chatbot, api_name=False, ) undo_button.click( fn=delete_prev_fn, inputs=chatbot, outputs=[chatbot, saved_input], api_name=False, queue=False, ).then( fn=lambda x: x, inputs=[saved_input], outputs=textbox, api_name=False, queue=False, ) clear_button.click( fn=lambda: ([], ''), outputs=[chatbot, saved_input], queue=False, api_name=False, ) demo.queue(max_size=20).launch(server_name="0.0.0.0") ================================================ FILE: llama2-7b-cn/model.py ================================================ from threading import Thread from typing import Iterator import torch from transformers import AutoModelForCausalLM, AutoTokenizer, TextIteratorStreamer model_id = 'LinkSoul/Chinese-Llama-2-7b' if torch.cuda.is_available(): model = AutoModelForCausalLM.from_pretrained( model_id, local_files_only=True, torch_dtype=torch.float16, device_map='auto' ) else: model = None tokenizer = AutoTokenizer.from_pretrained(model_id) def get_prompt(message: str, chat_history: list[tuple[str, str]], system_prompt: str) -> str: texts = [f'[INST] <Running on CPU 🥶 This demo does not work on CPU.
' def clear_and_save_textbox(message: str) -> tuple[str, str]: return '', message def display_input(message: str, history: list[tuple[str, str]]) -> list[tuple[str, str]]: history.append((message, '')) return history def delete_prev_fn( history: list[tuple[str, str]]) -> tuple[list[tuple[str, str]], str]: try: message, _ = history.pop() except IndexError: message = '' return history, message or '' def generate( message: str, history_with_input: list[tuple[str, str]], system_prompt: str, max_new_tokens: int, top_p: float, temperature: float, top_k: int, ) -> Iterator[list[tuple[str, str]]]: if max_new_tokens > MAX_MAX_NEW_TOKENS: raise ValueError history = history_with_input[:-1] generator = run(message, history, system_prompt, max_new_tokens, temperature, top_p, top_k) try: first_response = next(generator) yield history + [(message, first_response)] except StopIteration: yield history + [(message, '')] for response in generator: yield history + [(message, response)] def process_example(message: str) -> tuple[str, list[tuple[str, str]]]: generator = generate(message, [], DEFAULT_SYSTEM_PROMPT, 1024, 0.95, 1, 1000) for x in generator: pass return '', x with gr.Blocks(css='style.css') as demo: gr.Markdown(DESCRIPTION) gr.DuplicateButton(value='Duplicate Space for private use', elem_id='duplicate-button') with gr.Group(): chatbot = gr.Chatbot(label='Chatbot') with gr.Row(): textbox = gr.Textbox( container=False, show_label=False, placeholder='Type a message...', scale=10, ) submit_button = gr.Button('Submit', variant='primary', scale=1, min_width=0) with gr.Row(): retry_button = gr.Button('🔄 Retry', variant='secondary') undo_button = gr.Button('↩️ Undo', variant='secondary') clear_button = gr.Button('🗑️ Clear', variant='secondary') saved_input = gr.State() with gr.Accordion(label='Advanced options', open=False): system_prompt = gr.Textbox(label='System prompt', value=DEFAULT_SYSTEM_PROMPT, lines=6) max_new_tokens = gr.Slider( label='Max new tokens', minimum=1, maximum=MAX_MAX_NEW_TOKENS, step=1, value=DEFAULT_MAX_NEW_TOKENS, ) temperature = gr.Slider( label='Temperature', minimum=0.1, maximum=4.0, step=0.1, value=1.0, ) top_p = gr.Slider( label='Top-p (nucleus sampling)', minimum=0.05, maximum=1.0, step=0.05, value=0.95, ) top_k = gr.Slider( label='Top-k', minimum=1, maximum=1000, step=1, value=50, ) gr.Examples( examples=[ 'Hello there! How are you doing?', 'Can you explain briefly to me what is the Python programming language?', 'Explain the plot of Cinderella in a sentence.', 'How many hours does it take a man to eat a Helicopter?', "Write a 100-word article on 'Benefits of Open-Source in AI research'", ], inputs=textbox, outputs=[textbox, chatbot], fn=process_example, cache_examples=True, ) gr.Markdown(LICENSE) textbox.submit( fn=clear_and_save_textbox, inputs=textbox, outputs=[textbox, saved_input], api_name=False, queue=False, ).then( fn=display_input, inputs=[saved_input, chatbot], outputs=chatbot, api_name=False, queue=False, ).then( fn=generate, inputs=[ saved_input, chatbot, system_prompt, max_new_tokens, temperature, top_p, top_k, ], outputs=chatbot, api_name=False, ) button_event_preprocess = submit_button.click( fn=clear_and_save_textbox, inputs=textbox, outputs=[textbox, saved_input], api_name=False, queue=False, ).then( fn=display_input, inputs=[saved_input, chatbot], outputs=chatbot, api_name=False, queue=False, ).then( fn=generate, inputs=[ saved_input, chatbot, system_prompt, max_new_tokens, temperature, top_p, top_k, ], outputs=chatbot, api_name=False, ) retry_button.click( fn=delete_prev_fn, inputs=chatbot, outputs=[chatbot, saved_input], api_name=False, queue=False, ).then( fn=display_input, inputs=[saved_input, chatbot], outputs=chatbot, api_name=False, queue=False, ).then( fn=generate, inputs=[ saved_input, chatbot, max_new_tokens, temperature, top_p, top_k, ], outputs=chatbot, api_name=False, ) undo_button.click( fn=delete_prev_fn, inputs=chatbot, outputs=[chatbot, saved_input], api_name=False, queue=False, ).then( fn=lambda x: x, inputs=[saved_input], outputs=textbox, api_name=False, queue=False, ) clear_button.click( fn=lambda: ([], ''), outputs=[chatbot, saved_input], queue=False, api_name=False, ) demo.queue(max_size=20).launch(server_name="0.0.0.0") ================================================ FILE: llama2-7b-cn-4bit/model.py ================================================ from threading import Thread from typing import Iterator import torch from transformers import AutoModelForCausalLM, AutoTokenizer, TextIteratorStreamer model_id = 'soulteary/Chinese-Llama-2-7b-4bit' if torch.cuda.is_available(): model = AutoModelForCausalLM.from_pretrained( model_id, load_in_4bit=True, local_files_only=True, torch_dtype=torch.float16, device_map='auto' ) else: model = None tokenizer = AutoTokenizer.from_pretrained(model_id) def get_prompt(message: str, chat_history: list[tuple[str, str]], system_prompt: str) -> str: texts = [f'[INST] <