[ { "path": "README.md", "content": "
\n\n# VisCPM\n**基于CPM基础模型的中英双语多模态大模型系列**\n

\n 多模态对话模型VisCPM-Chat •\n 文生图模型VisCPM-Paint •\n 使用 •\n 论文\n

\n

\n VisCPM-Chat Demo •\n VisCPM-Paint Demo •\n VisCPM-Chat🤗 •\n VisCPM-Paint🤗 \n

\n\n

\n 简体中文 | English\n

\n
\n\n**`VisCPM`** is a family of open-source large multimodal models, which support multimodal conversational capabilities (`VisCPM-Chat` model) and text-to-image generation capabilities (`VisCPM-Paint` model) in both Chinese and English, achieving state-of-the-art performance among Chinese open-source multimodal models. VisCPM is trained based on the large language model [CPM-Bee](https://github.com/OpenBMB/CPM-Bee) with 10B parameters, fusing visual encoder (Muffin) and visual decoder (Diffusion-UNet) to support visual inputs and outputs. Thanks to the good bilingual capability of CPM-Bee, `VisCPM` can be pre-trained with English multimodal data only and well generalize to achieve promising Chinese multimodal capabilities.\n\n**`VisCPM`** 是一个开源的多模态大模型系列，支持中英双语的多模态对话能力（`VisCPM-Chat`模型）和文到图生成能力（`VisCPM-Paint`模型），在中文多模态开源模型中达到最佳水平。VisCPM基于百亿参数量语言大模型[CPM-Bee](https://github.com/OpenBMB/CPM-Bee)（10B）训练，融合视觉编码器[Muffin](https://github.com/thunlp/Muffin)和视觉解码器[Diffusion-UNet](https://github.com/CompVis/stable-diffusion)以支持视觉信号的输入和输出。得益于CPM-Bee基座优秀的双语能力，`VisCPM`可以仅通过英文多模态数据预训练，泛化实现优秀的中文多模态能力。\n\n- **👐 开源使用**：VisCPM可以自由被用于个人和研究用途。我们希望通过开源VisCPM模型系列，推动多模态大模型开源社区和相关研究的发展。\n- **🌟 涵盖图文双向生成**：VisCPM模型系列较为全面地支持了图文多模态能力，涵盖多模态对话（图到文生成）能力和文到图生成能力。\n- **💫 中英双语性能优异**：得益于语言模型基座CPM-Bee优秀的双语能力，VisCPM在中英双语的多模态对话和文到图生成均取得亮眼的效果。\n\n
\n\n\n
\n\n## 📰 更新信息\n`VisCPM`在持续升级中，我们支持了低资源推理、网页版部署等功能，并提供了能力升级的更高版本的模型[OmniLMM](https://github.com/OpenBMB/OmniLMM)，欢迎大家持续关注！\n- **[2024/04/17]** 🚀 欢迎关注我们最新发布的[MiniCPM-V 2.0](https://github.com/OpenBMB/MiniCPM-V)端侧多模态大模型，具备领先的光学字符识别（OCR）和多模态理解能力，在综合性OCR能力评测基准OCRBench上达到开源模型最佳水平，甚至在场景文字理解方面实现接近Gemini Pro的性能。\n- **[2024/02/02]** 🚀 欢迎关注我们最新发布的[OmniLMM](https://github.com/OpenBMB/OmniLMM)多模态大模型！其中[OmniLMM-3B](https://huggingface.co/openbmb/MiniCPM-V)为中英双语多模态对话模型，基于中英双语大模型[MiniCPM-2.4B](https://github.com/OpenBMB/MiniCPM/)和SigLip-400M视觉编码器训练，采用与VisCPM-Chat相同的训练流程训练，可在终端设备上部署并具备先进的多模态对话能力；[OmniLMM-13B](https://huggingface.co/openbmb/OmniLMM-12B/)为英文多模态模型，基于EVA02-5B和Zephyr-7B-β初始化训练，相比同规模其他模型在多个基准测试中具有领先性能。\n- **[2024/01/16]** 🎉 [VisCPM论文](https://arxiv.org/pdf/2308.12038.pdf)被**ICLR 2024**接收，并被选为**spotlight（top 5%）**！\n- **[2023/09/06]** 🔌 VisCPM-Chat API 发布！现在您可以直接通过API轻松地使用VisCPM-Chat模型了。查看[API使用指南](#API使用指南)以了解更多详情。\n- **[2023/08/23]** 📑 VisCPM论文发布：[Large Multilingual Models Pivot Zero-Shot Multimodal Learning across Languages](https://arxiv.org/pdf/2308.12038.pdf)，论文提供了更详细的实现细节和实验结果\n- **[2023/08/18]** ⤴️ [VisCPM-Chat-v1.1](#模型下载)版本发布，带来更强的细节理解和复杂推理能力！\n- **[2023/08/18]** 🛠️ 支持[微调](#模型微调)，让VisCPM更适配你的应用场景！\n- **[2023/07/20]** 🌐 发布[VisCPM-Chat](http://120.92.209.146/)和[VisCPM-Paint](https://huggingface.co/spaces/openbmb/viscpm-paint) 的在线Demo，欢迎尝试！\n- **[2023/07/20]** 🎢 支持一键[部署本地网页版Demo](#demo部署)\n- **[2023/07/20]** ⚡️ 支持[低资源推理](#低资源推理)，最低5G显存运行多模态对话模型！\n- **[2023/07/18]** 🤗 [VisCPM-Chat](https://huggingface.co/openbmb/VisCPM-Chat)和[VisCPM-Paint](https://huggingface.co/openbmb/VisCPM-Paint) 已整合到Huggingface框架中\n\n## VisCPM-Chat\n`VisCPM-Chat`支持面向图像进行中英双语多模态对话。该模型使用`Muffin`视觉编码架构，使用CPM-Bee（10B）作为语言基座模型，并通过语言建模训练目标融合视觉和语言模型。模型训练包括预训练和指令精调两阶段：\n\n* 预训练：我们使用约100M高质量英文图文对数据对`VisCPM-Chat`进行了预训练，数据包括CC3M、CC12M、COCO、Visual Genome、Laion等。在预训练阶段，语言模型参数保持固定，仅更新视觉编码器的参数，以支持大规模视觉-语言表示的高效对齐。\n\n* 指令精调：我们采用[LLaVA-150K](https://llava-vl.github.io/)英文指令精调数据，并混合相应[翻译后的中文数据](https://huggingface.co/datasets/openbmb/llava_zh)对模型进行指令精调，以对齐模型多模态基础能力和用户使用意图。在指令精调阶段，我们更新全部模型参数，以提升指令精调数据的利用效率。有趣的是，我们发现即使仅采用英文指令数据进行指令精调，模型也可以理解中文问题，但仅能用英文回答。这表明模型的多语言多模态能力已经得到良好的泛化。在指令精调阶段进一步加入少量中文翻译数据，可以将模型回复语言和用户问题语言对齐。\n\n我们在[LLaVA标准英文测试集](https://huggingface.co/datasets/liuhaotian/LLaVA-Instruct-150K)和翻译的[中文测试集](data/translated_LLaVA_qa90)对模型进行了评测，该评测基准考察模型在开放域对话、图像细节描述、复杂推理方面的表现，并使用GPT-4进行打分。可以观察到，`VisCPM-Chat`在中文多模态能力方面取得了最佳的平均性能，在通用域对话和复杂推理表现出色，同时也表现出了不错的英文多模态能力。我们提供了两个模型版本，分别为`VisCPM-Chat-balance`和`VisCPM-Chat-zhplus`，前者在英文和中文两种语言上的能力较为平衡，后者在中文能力上更加突出。两个模型在指令精调阶段使用的数据相同，`VisCPM-Chat-zhplus`在预训练阶段额外加入了20M清洗后的原生中文图文对数据和120M翻译到中文的图文对数据。`VisCPM-Chat-v1.1`在指令精调阶段额外加入了[UniMM-Chat](https://huggingface.co/datasets/Yirany/UniMM-Chat)多模态指令精调数据集。\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
模型语言模型基座英文中文
对话精细描述复杂推理平均对话精细描述复杂推理平均
英文模型MiniGPT4Vicuna-13B65.067.376.669.7----
InstructBLIPVicuna-13B81.968.091.280.5----
LLaVAVicuna-13B89.570.496.285.6----
中英双语模型mPLUG-Owl LLaMA-7B64.647.780.164.276.361.277.872.0
VisualGLMChatGLM-6B62.463.080.668.776.687.883.682.7
Ziya-Visual Ziya-LLaMA-13B-v182.769.992.181.785.074.782.480.8
Qwen-VL Qwen-7B82.472.691.983.882.393.489.588.2
VisCPM-Chat-balanceCPMBee-10B83.368.990.581.192.776.189.286.3
VisCPM-Chat-zhplusCPMBee-10B80.165.792.579.690.381.492.188.2
VisCPM-Chat-v1.1CPMBee-10B80.167.197.181.591.390.795.492.5
\n\n![图片](figures/vlu_case4.png)\n\n## VisCPM-Paint\n`VisCPM-Paint`支持中英双语的文到图生成。该模型使用CPM-Bee（10B）作为文本编码器，使用`UNet`作为图像解码器，并通过扩散模型训练目标融合语言和视觉模型。在训练过程中，语言模型参数始终保持固定。我们使用[Stable Diffusion 2.1](https://github.com/Stability-AI/stablediffusion)的UNet参数初始化视觉解码器，并通过逐步解冻其中关键的桥接参数将其与语言模型融合。该模型在[LAION 2B](https://laion.ai/)英文图文对数据上进行了训练。\n\n与`VisCPM-Chat`类似，我们发现得益于CPM-Bee的双语能力，`VisCPM-Paint`可以仅通过英文图文对训练，泛化实现良好的中文文到图生成能力，达到中文开源模型的最佳效果。通过进一步加入20M清洗后的原生中文图文对数据，以及120M翻译到中文的图文对数据，模型的中文文到图生成能力可以获得进一步提升。我们在标准图像生成测试集MSCOCO上采样了3万张图片，计算了常用评估图像生成指标FID (Fréchet Inception Distance)评估生成图片的质量。我们同样提供了两个模型版本，分别为`VisCPM-Paint-balance`和`VisCPM-Paint-zhplus`，前者在英文和中文两种语言上的能力较为平衡，后者在中文能力上更加突出。`VisCPM-Paint-balance`只使用了英文图文对进行训练，`VisCPM-Paint-zhplus`在`VisCPM-Paint-balance`基础上增加了20M原生中文图文对数据和120M翻译到中文的图文对数据进行训练。\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
模型Zero-shot FID↓
英文中文
GLIDE12.2-
Make-A-Scene11.8-
DALL·E-210.4-
Unidiffuser9.7-
Cogview2-24.0
Stable Diffusion8.6-
AltDiffusion17.216.1
TaiyiDiffusion-15.6
VisCPM-Paint-balance9.510.9
VisCPM-Paint-zhplus9.99.6
\n\n## ⚙️ 安装\n1. 克隆仓库并进入源码目录\n```bash\ngit clone https://github.com/OpenBMB/VisCPM.git\ncd VisCPM\n```\n\n2. 创建conda环境\n```Shell\nconda create -n viscpm python=3.10 -y\nconda activate viscpm\n```\n\n3. 安装依赖\n \n```shell\npip install torch>=1.10\npip install -r requirements.txt\n```\n\n## 💡 使用\n### 模型下载\n| 模型 | 描述 | 下载链接 |\n|----------------------|-------------------|------ |\n| VisCPM-Chat-v1.1 | 新版本多模态对话模型，强化了细节理解和复杂推理能力 | [链接](https://huggingface.co/openbmb/VisCPM-Chat/blob/main/pytorch_model.v1.bin) |\n| VisCPM-Chat-balance | 中英文能力较为平衡的多模态对话模型 | [链接](https://huggingface.co/openbmb/VisCPM-Chat/resolve/main/pytorch_model.bin) |\n| VisCPM-Chat-zhplus | 中文能力突出的多模态对话模型 | [链接](https://huggingface.co/openbmb/VisCPM-Chat/resolve/main/pytorch_model.zhplus.bin) |\n| VisCPM-Paint-balance | 中英文能力较为平衡的文生图模型 | [链接](https://huggingface.co/openbmb/VisCPM-Paint/resolve/main/pytorch_model.balance.bin) |\n| VisCPM-Paint-zhplus | 中文能力突出的文生图模型 | [链接](https://huggingface.co/openbmb/VisCPM-Paint/resolve/main/pytorch_model.bin) |\n\n\n### VisCPM-Chat\n在下载模型权重后，可以使用如下代码运行VisCPM-Chat（`'/path/to/checkpoint'`改为模型存放路径）\n\n#### 单轮对话\n\n
\n\n
\n\nVisCPM-Chat可以通过几行代码实现多模态对话，我们在代码中默认开启了对输入图片的安全检查。\n```shell\n# 如果您单卡显存不足40G，可以引入如下环境变量并将安全模块开关关闭。引入后显存占用约为5G，但推理所需时间会变长。此选项依赖BMInf，需要安装BMInf依赖库。\nexport CUDA_MEMORY_CPMBEE_MAX=1g\n```\n```python\nfrom VisCPM import VisCPMChat\nfrom PIL import Image\n\nmodel_path = '/path/to/checkpoint'\nviscpm_chat = VisCPMChat(model_path, image_safety_checker=True)\n# 默认开启对输入图片的安全检查\nimage_path = 'figures/vlu_case1.png'\nimage = Image.open(image_path).convert(\"RGB\")\n\nquestion = '如果用一句中国唐代的著名诗人\"李白\"的古诗来描述这幅图像，你能想到什么？'\nanswer, _, _ = viscpm_chat.chat(image, question)\n\nprint(answer)\n```\n可得到如下结果\n```\n“黄河之水天上来，奔流到海不复回。” 李白的这句诗可以用来形容这幅图片中汹涌澎湃、波涛汹涌的景象：一条湍急的河流从山上奔腾而下，形成了一幅令人叹为观止的画面，展示出大自然的力量和雄伟壮丽。\n```\n\n#### 多轮对话\n
\n\n\n
\n\n```python\nfrom VisCPM import VisCPMChat\nfrom PIL import Image\n\nmodel_path = '/path/to/checkpoint'\nviscpm_chat = VisCPMChat(model_path, image_safety_checker=True)\n# 默认开启对输入图片的安全检查\nimage_path = 'figures/vlu_case2.jpeg'\nimage = Image.open(image_path).convert(\"RGB\")\n\nquestion = '这幅图像是在哪个节日拍摄的？'\nanswer, context, vision_hidden_states = viscpm_chat.chat(image, question)\n\n# 多轮对话传入历史 context\nquestion = '你能用什么古诗描述这幅画？'\nanswer, context, _ = viscpm_chat.chat(image, question, context, vision_hidden_states=vision_hidden_states)\n\nprint(context)\n```\n\n可得到如下结果\n```\nUser: 这幅图像是在哪个节日拍摄的？\nAI: 这幅图像是在中秋节拍摄的， 也就是中国传统节日中的月圆之夜。\nUser: 你能用什么古诗描述这幅画？\nAI: “明月几时有，把酒问青天。” 这是苏轼的《水调歌头》中的一句诗，用来形容这幅图片再贴切不过了：在中秋之夜，月亮高高地挂在天空中，一座古老的建筑沐浴着月光，营造出一种宁静祥和的气氛。\n```\n\n\n#### API使用指南\n我们提供了API接口，可以通过如下代码轻松体验VisCPM-Chat。API接口支持的输入格式和使用方式如下：\n```python\nimport requests\nimport base64\n\nurl = \"http://34.143.180.202:3389/viscpm\"\nresp = requests.post(url, json={\n # need to modify\n \"image\": base64.b64encode(open(\"path/to/image\", \"rb\").read()).decode(),\n \"question\": \"描述一下这张图片\",\n})\nresp = resp.json()\nprint(resp)\n```\n\n### VisCPM-Paint\n在下载模型权重后，可以使用如下代码运行VisCPM-Paint（`'/path/to/checkpoint'`改为模型存放路径）。\n\n![图片](figures/vlg_case1.png)\n\n![图片](figures/vlg_case2.png)\n\n生成上面图片的文本输入可参考[prompts.txt](data/prompts.txt)。\n```shell\n# 如果您单卡显存不足40G，可以引入如下环境变量并将安全模块开关关闭。引入后显存占用约为17G，但推理所需时间会变长。此选项依赖BMInf，需要安装BMInf依赖库。\nexport CUDA_MEMORY_CPMBEE_MAX=1g\n```\n```python\nfrom VisCPM import VisCPMPaint\npainter = VisCPMPaint('/path/to/checkpoint', image_safety_checker=True, prompt_safety_checker=True, add_ranker=True) \n# 默认对输入的文本和输出的图片进行安全检查，默认开启重排序\nimage = painter.generate('人闲桂花落，月静春山空') # 对应上图第一行第二张图片\nimage.save('/data/test.png') \n```\n\n我们在代码中默认开启了对输入文本和输出图片的安全检查。\n\n同时，我们默认对生成的图像使用重排序，即对同一个输入，同时生成4张图片，返回与输入相关性最高的1张图片，相关性通过[Chinese-Clip](https://github.com/OFA-Sys/Chinese-CLIP)进行打分。重排序可以提升生成图片质量的稳定性，但也会降低模型的生成速度，如希望快速得到生成结果，可以关闭重排序机制。\n\nVisCPM-Paint目前使用中文模型进行重排序打分，如果输入英文生成图片，请关闭重排序机制和输入文本检查模块。\n\n### 低资源推理\n\n为了支持更加高效的低资源推理场景，我们借助[BMInf](https://github.com/OpenBMB/BMInf)工具支持更低的显存需求。首先安装BMInf依赖`pip install bminf`，然后在命令行中指定`export CUDA_MEMORY_CPMBEE_MAX=1g`（具体数值可以根据个人需求设定），然后按照上述步骤进行推理，VisCPM-Chat最低显存占用可以降至5G，VisCPM-Paint最低显存占用可以降至17G。\n\n### Demo部署\n\n我们提供简易的基于gradio的网页版Demo，首先安装gradio:`pip install gradio`，然后执行如下命令：\n```shell\ngit clone https://github.com/OpenBMB/VisCPM.git\ncd VisCPM\npython demo_chat.py # viscpm_chat demo, or\npython demo_paint.py # viscpm_paint demo\n```\n\n### 模型微调\n为适应特定场景下的需求, 我们提供了VisCPM-Chat模型的微调代码，用户可以在私有数据上进行微调。微调代码位于`./finetune/ft_viscpm_chat`目录下，具体的微调代码使用方法如下：\n```shell\n# 获取数据集\nbash ./finetune/ft_viscpm_chat/get_llava150k_zh.sh\n# 模型微调, 注意修改其中的数据集与模型checkpoint路径\nbash ./finetune/ft_viscpm_chat/run_viscpm_chat_ft.sh\n# node: 8\n# batch_size: 8 * 1\n# 其他配置可参考'./finetune/ft_viscpm_chat/config/viscpm_chat_ft.json'与'./finetune/ft_viscpm_chat/run_viscpm_chat_ft.sh'\n```\n注: \n- 微调代码中使用了deepspeed-0.9.1配置训练环境，配置方法可以参考此[链接](https://www.deepspeed.ai/getting-started/)。\n- 目前微调代码仅在linux系统下测试，如果您在其他系统配置下进行微调，可能需要修改部分代码。\n\n## 🛡 安全\n\n### 安全声明\n作为多模态模型，`VisCPM`通过学习大量的公开图文数据来生成内容，但它无法理解、表达个人观点或价值判断，它所输出的任何内容都不代表模型开发者的观点和立场。因此用户在使用VisCPM生成的内容时，应自行负责对其进行评估和验证。\n\n### 安全模块\n为了帮助用户防止模型处理或生成不符合普遍社会价值观的内容，我们在`VisCPM`中加入了内容安全保障模块。当安全模块检测到模型处理或生成的图像文本内容不符合安全规范时，会对相应内容进行拦截。我们对`VisCPM-Chat`接受的图片输入以及`VisCPM-Paint`接受的文字输入和图片输出进行了安全检查。VisCPM的安全模块仍然不完美，可能会出现漏判和误判的情况。我们会在未来进一步提升安全模块的性能。\n\n\n## 📝 开源协议\n\nVisCPM系列模型采用协议为[\"通用模型许可协议-来源说明-宣传限制-非商业化\"](https://github.com/OpenBMB/General-Model-License/blob/main/%E9%80%9A%E7%94%A8%E6%A8%A1%E5%9E%8B%E8%AE%B8%E5%8F%AF%E5%8D%8F%E8%AE%AE-%E6%9D%A5%E6%BA%90%E8%AF%B4%E6%98%8E-%E5%AE%A3%E4%BC%A0%E9%99%90%E5%88%B6-%E9%9D%9E%E5%95%86%E4%B8%9A%E5%8C%96.md)，允许个人使用和研究用途。如需将模型用于商业用途，请联系cpm@modelbest.cn来洽谈商业授权事宜。\n\nCPM-Bee基座采用协议为[“通用模型许可协议-来源说明-宣传限制-商业授权”](https://github.com/OpenBMB/General-Model-License/blob/main/%E9%80%9A%E7%94%A8%E6%A8%A1%E5%9E%8B%E8%AE%B8%E5%8F%AF%E5%8D%8F%E8%AE%AE-%E6%9D%A5%E6%BA%90%E8%AF%B4%E6%98%8E-%E5%AE%A3%E4%BC%A0%E9%99%90%E5%88%B6-%E5%95%86%E4%B8%9A%E6%8E%88%E6%9D%83.md)，允许商用，如需将模型用于商业用途，请联系cpm@modelbest.cn来获取书面授权。\n\n\n## ✅ TODO\n\n- [ ] 支持模型量化功能，降低推理成本\n\n## 🏫 机构\n\n此项目由以下机构联合研发：\n\n- [清华自然语言处理实验室](https://nlp.csai.tsinghua.edu.cn/)\n- [面壁智能](https://modelbest.cn/)\n- [知乎](https://www.zhihu.com/)\n\n## 引用\n如果我们的工作对你有帮助的话，请考虑引用以下论文\n```bibtex\n@article{VisCPM,\n title={Large multilingual models pivot zero-shot multimodal learning across languages},\n author={Hu, Jinyi and Yao, Yuan and Wang, Chongyi and Wang, Shan and Pan, Yinxu and Chen, Qianyu and Yu, Tianyu and Wu, Hanghao and Zhao, Yue and Zhang, Haoye and others},\n journal={arXiv preprint arXiv:2308.12038},\n year={2023}\n}\n\n@article{muffin,\n title={Reformulating vision-language foundation models and datasets towards universal multimodal assistants},\n author={Yu, Tianyu and Hu, Jinyi and Yao, Yuan and Zhang, Haoye and Zhao, Yue and Wang, Chongyi and Wang, Shan and Pan, Yinxv and Xue, Jiao and Li, Dahai and others},\n journal={arXiv preprint arXiv:2310.00653},\n year={2023}\n}\n```\n" }, { "path": "README_en.md", "content": "
\n\n# VisCPM\n**Chinese-English bilingual large multi-modal model series**\n

\n Multimodal Conversation Model VisCPM-Chat •\n Text-to-image Model VisCPM-Paint •\n Inference •\n Paper\n

\n

\n VisCPM-Chat Demo •\n VisCPM-Paint Demo •\n VisCPM-Chat🤗 •\n VisCPM-Paint🤗 \n

\n\n

\n 简体中文 | English\n

\n
\n\n`VisCPM` is a family of open-source large multimodal models, which support multimodal conversational capabilities (`VisCPM-Chat` model) and text-to-image generation capabilities (`VisCPM-Paint` model) in both Chinese and English, achieving state-of-the-art performance among Chinese open-source multimodal models. VisCPM is trained based on the large language model [CPM-Bee](https://github.com/OpenBMB/CPM-Bee) with 10B parameters, fusing visual encoder [Muffin](https://github.com/thunlp/Muffin) and visual decoder [Diffusion-UNet](https://github.com/CompVis/stable-diffusion) to support visual inputs and outputs. Thanks to the good bilingual capability of CPM-Bee, `VisCPM` can be pre-trained with English multimodal data only and well generalize to achieve promising Chinese multimodal capabilities.\n\n- **👐 Open-source Usage**: VisCPM is free to be used for personal and research purposes. By open-sourcing the VisCPM model family, we hope to promote the development of the open-source community of large multimodal models and related research.\n- **🌟 Image and text generation coverage**: VisCPM models provide relatively comprehensive support for image and text multimodal capabilities, covering both multimodal conversation (image-to-text generation) capabilities and text-to-image generation capabilities.\n- **💫 Excellent bilingual performance**: Thanks to the excellent bilingual capability of the base language model CPM-Bee, VisCPM achieves outstanding results in both bilingual multimodal conversation and text-to-image generation.\n\n
\n\n
\n\n## 📰 Update News\n`VisCPM` is continuously updating. We have provided functions such as low-resource reasoning, easy-to-use web deployment. We have provide new versions with upgraded capabilities, [OmniLMM](https://github.com/OpenBMB/OmniLMM). Please continue to pay attention!\n\n- **[2024/04/22]** 🚀 Welcome to follow our latest release of the [MiniCPM-V 2.0](https://github.com/OpenBMB/MiniCPM-V) edge-side large multimodal model, which has leading Optical Character Recognition (OCR) and multimodal understanding capabilities. It has achieved the best level among open-source models in the comprehensive OCR capability benchmark OCRBench, and even approaches the performance of Gemini Pro in scene text understanding.\n- **[2024/02/02]** 🚀 Welcome to follow our latest release of the [OmniLMM](https://github.com/OpenBMB/OmniLMM) large multimodal model! Among them, [OmniLMM-3B](https://huggingface.co/openbmb/MiniCPM-V) is a bilingual multimodal dialogue model in Chinese and English, trained based on the bilingual large model [MiniCPM-2.4B](https://github.com/OpenBMB/MiniCPM/) and the SigLip-400M visual encoder, using the same training process as VisCPM-Chat. It can be deployed on terminal devices and possesses advanced multimodal dialogue capabilities; [OmniLMM-13B](https://huggingface.co/openbmb/OmniLMM-12B/) is an English multimodal model, initially trained based on EVA02-5B and Zephyr-7B-β, and compared to other models of the same scale, it demonstrates superior performance in multiple benchmark tests.\n- **[2024/01/16]** 🎉 [The paper of VisCPM](https://arxiv.org/pdf/2308.12038.pdf) is accepted by **ICLR 2024** as **spotlight (top 5%)**！\n- **[2023/09/06]** 🔌 VisCPM-Chat API Released! Now you can easily use the VisCPM-Chat model directly through the API. Check out the [API Usage Guide](#API-Usage-Guide) for more details\n- **[2023/08/23]** 📑 We release the paper of VisCPM: [Large Multilingual Models Pivot Zero-Shot Multimodal Learning across Languages](https://arxiv.org/pdf/2308.12038.pdf). More impletation details and experimental results are presented in the paper.\n- **[2023/08/18]** ⤴️ We upgrade to [VisCPM-Chat-v1.1](#model-zoo), with stronger detail understanding and complex reasoning ability!\n- **[2023/08/18]** 🛠️ We support [fine-tuning](#Fine-Tuning) to make VisCPM more suitable for your application scenarios!\n- **[2023/07/20]** 🌐 We release [VisCPM-Chat](http://120.92.209.146/) and [VisCPM-Paint](https://huggingface.co/spaces/openbmb/viscpm-paint) online demo!\n- **[2023/07/20]** 🎢 We provide one-click [deployment of local web version demo](#demo-deployment)!\n- **[2023/07/20]** ⚡️ We support [low-resource inference](#low-resource-inference), with minimum 5G GPU memory cost to run VisCPM-Chat!\n- **[2023/07/18]** 🤗 [VisCPM-Chat](https://huggingface.co/openbmb/VisCPM-Chat) and [VisCPM-Paint](https://huggingface.co/openbmb/VisCPM-Paint) have been integrated into the huggingface framework!\n\n## VisCPM-Chat\n`VisCPM-Chat` supports bilingual multimodal conversations involving images in both Chinese and English. The model utilizes `Muffin` visual encoding architecture and `CPM-Bee` (10B) as the base LLM. It combines visual and language models and is optimized with the language modeling training objective. The model training consists of two stages: Multimodal Pretraining and Instruction Tuning.\n\n* Multimodal Pretraining: `VisCPM-Chat` is pretrained using approximately 150M high-quality English text-image pairs. The data sources include CC3M, CC12M, COCO, Visual Genome, Laion, etc. In this stage, the language model parameters remain fixed, and only the parameters of visual modules are updated to enable efficient alignment of vision and language representations.\n\n* Instruction Tuning: We utilize the [LLaVA-150K](https://llava-vl.github.io/) dataset that contains English multimodal instruction-following data. We mix this data with corresponding [translated Chinese data](https://huggingface.co/datasets/openbmb/llava_zh) to fine-tune the model and align its multimodal capabilities with user intents. In this stage, we update all model parameters to improve the data efficiency of instruction tuning. Interestingly, we observe that even when using only English instruction data for fine-tuning, the model can well comprehend Chinese questions but can only respond in English. This indicates that the model has achieved good generalization in terms of its multilingual and multimodal capabilities. By incorporating a small amount of translated Chinese data during the instruction tuning stage, we can align the model's response language with the user's question language.\n\nWe evaluate the model on the standard [LLaVA English benchmark](https://huggingface.co/datasets/liuhaotian/LLaVA-Instruct-150K) and the translated [Chinese benchmark](data/translated_LLaVA_qa90) from the standard English benchmark. The evaluation benchmark examines the model's performance in conversation, detailed description, and complex reasoning, and uses GPT-4 for scoring. It can be observed that `VisCPM-Chat` achieves the best average performance in Chinese multimodal capabilities, excelling in conversation and complex reasoning, while also demonstrating good English multimodal capabilities. We provide two versions of the model, namely `VisCPM-Chat-balance` and `VisCPM-Chat-zhplus`. The former has a balanced ability in both English and Chinese, while the latter has a stronger emphasis on Chinese proficiency. Both models use the same data during the instruction tuning stage. `VisCPM-Chat-zhplus` additionally incorporates 20M cleaned native Chinese text-image pairs and 120M translated text-image pairs in Chinese during the pretraining stage. `VisCPM-Chat-v1.1` additioanlly utilizes [UniMM-Chat](https://huggingface.co/datasets/Yirany/UniMM-Chat) multimodal instruction tuning dataset.\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
ModelLLM BackboneEnglishChinese
ConversationDetailed DescriptionComplex ReasoningAvgConversationDetailed DescriptionComplex ReasoningAvg
English ModelMiniGPT4Vicuna-13B65.067.376.669.7----
InstructBLIPVicuna-13B81.968.091.280.5----
LLaVAVicuna-13B89.570.496.285.6----
En-Zh Bilingual ModelmPLUG-Owl LLaMA-7B64.647.780.164.276.361.277.872.0
VisualGLMChatGLM-6B62.463.080.668.776.687.883.682.7
Ziya-Visual Ziya-LLaMA-13B-v182.769.992.181.785.074.782.480.8
Qwen-VL Qwen-7B82.472.691.983.882.393.489.588.2
VisCPM-Chat-balanceCPMBee-10B83.368.990.581.192.776.189.286.3
VisCPM-Chat-zhplusCPMBee-10B80.165.792.579.690.381.492.188.2
VisCPM-Chat-v1.1CPMBee-10B80.167.197.181.591.390.795.492.5
\n\n## VisCPM-Paint\n`VisCPM-Paint` supports bilingual text-to-image generation. The model uses `CPM-Bee` as the text encoder, `UNet` as the image decoder, and fuses vision and language models using the objective of diffusion model. During the training process, the parameters of the language model remain fixed. The visual decoder is initialized with the parameters of [Stable Diffusion 2.1](https://github.com/Stability-AI/stablediffusion), and it is fused with the language model by gradually unfreezing key bridging parameters. The model is trained on the [LAION 2B](https://laion.ai/) English text-image pair dataset.\n\nSimilar to `VisCPM-Chat`, we found that due to the bilingual capability of `CPM-Bee`, `VisCPM-Paint` can achieve good Chinese text-to-image generation by training only on English text-image pairs, surpassing the performance of Chinese open-source models. By incorporating an additional 20M cleaned native Chinese text-image pairs and 120M translated text-image pairs in Chinese, the model's Chinese text-to-image generation ability can be further improved. We sample 30,000 images from the standard image generation test set MSCOCO and calculated commonly used evaluation metrics FID (Fréchet Inception Distance) to assess the quality of generated images. Similarly, we provide two versions of the model, namely `VisCPM-Paint-balance` and `VisCPM-Paint-zhplus`. The former has a balanced ability in both English and Chinese, while the latter emphasizes Chinese proficiency. `VisCPM-Paint-balance` is trained only using English text-image pairs, while `VisCPM-Paint-zhplus` incorporates an additional 20M native Chinese text-image pairs and 120M translated text-image pairs in Chinese based on `VisCPM-Paint-balance`.\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
ModelZero-shot FID↓
EnglishChinese
GLIDE12.2-
Make-A-Scene11.8-
DALL·E-210.4-
Unidiffuser9.7-
Cogview2-24.0
Stable Diffusion8.6-
AltDiffusion17.216.1
TaiyiDiffusion-15.6
VisCPM-Paint-balance9.510.9
VisCPM-Paint-zhplus9.99.6
\n\n## ⚙️ Install\n\n1. Clone this repository and navigate to source folder\n```bash\ngit clone https://github.com/OpenBMB/VisCPM.git\ncd VisCPM\n```\n\n2. Create conda environment\n```Shell\nconda create -n viscpm python=3.10 -y\nconda activate viscpm\n```\n\n3. Install dependencies\n\n\n```shell\npip install torch>=1.10\npip install -r requirements.txt\n```\n\n## 💡 Inference\n### Model Zoo\n| Model | Description | Download Link |\n|----------------------|-------------------|------ |\n| VisCPM-Chat-v1.1 | Latest version of multimodal conversation model with stronger detail understanding and complex reasoning ability! | [download](https://huggingface.co/openbmb/VisCPM-Chat/blob/main/pytorch_model.v1.bin) |\n| VisCPM-Chat-balance | Multimodal conversation model with balanced proficiency in both Chinese and English | [download](https://huggingface.co/openbmb/VisCPM-Chat/resolve/main/viscpm_chat_balance_checkpoint.pt) |\n| VisCPM-Chat-zhplus | Multimodal conversation model with a strong emphasis on Chinese proficiency | [download](https://huggingface.co/openbmb/VisCPM-Chat/resolve/main/viscpm_chat_zhplus_checkpoint.pt) |\n| VisCPM-Paint-balance | Text-to-image model with balanced proficiency in both Chinese and English | [download](https://huggingface.co/openbmb/VisCPM-Paint/resolve/main/viscpm_paint_balance_checkpoint.pt) |\n| VisCPM-Paint-zhplus | Text-to-image model with a strong emphasis on Chinese proficiency | [download](https://huggingface.co/openbmb/VisCPM-Paint/resolve/main/viscpm_paint_zhplus_checkpoint.pt) |\n### VisCPM-Chat\nAfter downloading the checkpoints, please refer to the following codes to run `VisCPM-Chat` (replace `'/path/to/checkpoint'` with actually path of downloaded checkpoint).\n\n#### Single-turn Conversation\n
\n\n
\n\nWe can have a multimodal conversation with VisCPM-Chat using a few lines of codes.\n```shell\n# If the memory of your GPU is less than 40G, you can introduce the following environment variables. After the introduction, the memory usage is about 17G, but the time required for inference will be longer. This feature relies on the BMInf package.\nexport CUDA_MEMORY_CPMBEE_MAX=1g\n```\n```python\nfrom VisCPM import VisCPMChat\nfrom PIL import Image\n\nmodel_path = '/path/to/checkpoint'\nviscpm_chat = VisCPMChat(model_path, image_safety_checker=True)\n# We perform security checks on the input images by default.\nimage_path = 'figures/vlu_case1.png'\nimage = Image.open(image_path).convert(\"RGB\")\n\nquestion = '如果用一句中国唐代的著名诗人\"李白\"的古诗来描述这幅图像，你能想到什么？' # If you use an ancient poem by the famous Tang Dynasty poet \"Li Bai\" to describe this image, what can you think of?\nanswer, _, _ = viscpm_chat.chat(image, question)\n\nprint(answer)\n```\nWe can obtain the following results:\n```\n“黄河之水天上来，奔流到海不复回。” 李白的这句诗可以用来形容这幅图片中汹涌澎湃、波涛汹涌的景象：一条湍急的河流从山上奔腾而下，形成了一幅令人叹为观止的画面，展示出大自然的力量和雄伟壮丽。\n\n\"The water of the Yellow River comes up from the sky, rushing to the sea and never returning.\" This line of Li Bai's poem can be used to describe the surging and turbulent scene in this picture: a rushing river rushes down from the mountain, forming a breathtaking scene. Breathtaking images that showcase the power and majesty of nature.\n```\n\n#### Multi-turn Conversation\n
\n\n\n
\n\nThe codes can be easily extended to support multi-turn multimodal conversations.\n\n```python\nfrom VisCPM import VisCPMChat\nfrom PIL import Image\n\nmodel_path = '/path/to/checkpoint'\nviscpm_chat = VisCPMChat(model_path, image_safety_checker=True)\n# We perform security checks on the input images by default.\nimage_path = 'figures/vlu_case2.jpeg'\nimage = Image.open(image_path).convert(\"RGB\")\n\nquestion = '这幅图像是在哪个节日拍摄的？' # At which festival was this image taken?\nanswer, context, vision_hidden_states = viscpm_chat.chat(image, question)\n\n# pass history context of multi-turn conversation\nquestion = '你能用什么古诗描述这幅画？' # What ancient poem can you use to describe this painting?\nanswer, context, _ = viscpm_chat.chat(image, question, context, vision_hidden_states=vision_hidden_states)\n\nprint(context)\n```\n\nWe can obtain the following results:\n```\nUser: 这幅图像是在哪个节日拍摄的？\nAI: 这幅图像是在中秋节拍摄的， 也就是中国传统节日中的月圆之夜。\nUser: 你能用什么古诗描述这幅画？\nAI: “明月几时有，把酒问青天。” 这是苏轼的《水调歌头》中的一句诗，用来形容这幅图片再贴切不过了：在中秋之夜，月亮高高地挂在天空中，一座古老的建筑沐浴着月光，营造出一种宁静祥和的气氛。\n\nUser: On which festival was this image taken?\nAI: This image was taken during the Mid-Autumn Festival, which is the night of the full moon in the traditional Chinese festival.\nUser: What ancient poem can you use to describe this painting?\nAI: \"When did the bright moon first exist? With a cup of wine in my hand, I ask the blue sky.\" This is a line from Su Shi's \"Shui Diao Ge Tou\", which can't be more appropriate to describe this picture: On the night of Mid-Autumn Festival, the moon hangs high in the sky among them, an ancient building is bathed in moonlight, creating a peaceful atmosphere.\n```\n#### API Usage Guide\nWe offer an API, and you can easily experience VisCPM-Chat with the following code. Supported input formats and usage of the API are as follows:\n```python\nimport requests\nimport base64\n\nurl = \"http://34.143.180.202:3389/viscpm\"\nresp = requests.post(url,json={\n # need to modify\n \"image\": base64.b64encode(open(\"path/to/image\", \"rb\").read()).decode(),\n \"question\": \"Describe this image\",\n})\nresp = resp.json()\nprint(resp)\n```\n\n\n### VisCPM-Paint\nAfter downloading the checkpoints, please refer to the following codes to run `VisCPM-Paint` (replace `'/path/to/checkpoint'` with actually path of downloaded checkpoint).\n\n![图片](figures/vlg_case1.png)\n\n![图片](figures/vlg_case2.png)\n\nThe input prompts of the images above can be found at [prompts.txt](data/prompts.txt).\n```shell\n# If the memory of your GPU is less than 40G, you can introduce the following environment variables. After the introduction, the memory usage is about 17G, but the time required for inference will be longer. This feature relies on the BMInf package.\nexport CUDA_MEMORY_CPMBEE_MAX=1g\n```\n```python\nfrom VisCPM import VisCPMPaint\npainter = VisCPMPaint('/path/to/checkpoint', image_safety_checker=True, prompt_safety_checker=True, add_ranker=True) \n# We perform security checks on the input text and output images by default. Additionally, the default setting includes image reranking.\nimage = painter.generate('人闲桂花落，月静春山空')\n# The sweet-scented osmanthus falls when people are idle, the moon is quiet and the mountains are empty in spring.\n# Corresponding to the second picture in the first row of the above picture.\nimage.save('/data/test.png')\n```\n\nIn our code, we have enabled the default security checks for both input text and output images.\n\nAdditionally, we have implemented a default setting of reranking for the generated images. This means that for a given input, we generate four images simultaneously and return the one with the highest relevance score to the input, which is evaluated using [Chinese-Clip](https://github.com/OFA-Sys/Chinese-CLIP). Reranking enhances the stability of the generated image quality but may also slow the model's generation speed. If you prefer to obtain the generated results quickly, you can disable the reranking mechanism.\n\nIf you are providing English text as input for generating images, it is advisable to disable the reranking mechanism and input text checker, since the scoring model used for reranking and safety checker for the input prompt are specifically trained for Chinese text.\n\n### Low-Resource Inference\n\nWe use [BMInf](https://github.com/OpenBMB/BMInf) to reduce GPU memory costs. First you need to install BMInf by `pip install bminf`, and then specify `export CUDA_MEMORY_CPMBEE_MAX=1g` the environment variable in shell, and then follow the above steps to inference. The minimum GPU memory usage of VisCPM-Chat can be reduced to 5G, and the minimum GPU memory usage of VisCPM-Paint can be reduced to 17G.\n\n### Demo Deployment\n\nWe provide a simple web version demo based on gradio. First you need to install gradio: `pip install gradio`, and then execute the following command:\n```shell\ngit clone https://github.com/OpenBMB/VisCPM.git\ncd VisCPM\npython demo_chat.py # viscpm_chat demo, or\npython demo_paint.py # viscpm_paint demo\n```\n### Fine-Tuning\nWe provide the fine-tuning code for VisCPM-Chat. Users can fine-tune it on their own private data. The fine-tuning code is located in the `finetune/ft_viscpm_chat` directory, and the specific usage of the fine-tuning code is as follows:\n```shell\n# Get the dataset\nbash ./finetune/ft_viscpm_chat/get_llava150k_zh.sh\n# Model fine-tuning, note to modify the dataset and model checkpoint paths within\nbash ./finetune/ft_viscpm_chat/run_viscpm_chat_ft.sh\n# node: 8\n# batch_size: 8 * 1\n# More details can be found in './finetune/ft_viscpm_chat/config/viscpm_chat_ft.json' and './finetune/ft_viscpm_chat/run_viscpm_chat_ft.sh'\n```\nNote:\n- `deepspeed-0.9.1` is used in the fine-tuning code, and the installation method can be found in [here](https://www.deepspeed.ai/getting-started/).\n- Currently, we have only tested the codes of fine-tuning on `Linux`. If you are fine-tuning under other system configurations, you may need to modify some of the code\n\n## 🛡 Safety \n\n### Safety Statement\nAs a multimodal model, `VisCPM` generates content by learning from a vast amount of public image and text data. However, it does not possess the ability to comprehend or express personal opinions or value judgments. Any content generated by VisCPM does not represent the viewpoints or positions of the model developers.\nTherefore, when using content generated by `VisCPM`, users should take full responsibility for evaluating and verifying it on their own\n\n### Safety Module\nTo prevent the model from being misused to process or generate content that violates widely accepted societal values, we have incorporated a content safety module in `VisCPM`. When the safety module detects image or text content that does not comply with safety regulations during model processing or generation, it intercepts the corresponding content. We performed security checks on the input images accepted by `VisCPM-Chat` and the input text and output images of `VisCPM-Paint`. While the safety module in VisCPM still has room for improvement, there may be instances of both false positives and false negatives. We will continue to enhance the performance of the safety module in future updates.\n\n\n## 📝 License\n\nVisCPM is governed by the [GML License](https://github.com/OpenBMB/General-Model-License/blob/main/%E9%80%9A%E7%94%A8%E6%A8%A1%E5%9E%8B%E8%AE%B8%E5%8F%AF%E5%8D%8F%E8%AE%AE-%E6%9D%A5%E6%BA%90%E8%AF%B4%E6%98%8E-%E5%AE%A3%E4%BC%A0%E9%99%90%E5%88%B6-%E9%9D%9E%E5%95%86%E4%B8%9A%E5%8C%96.md), and permits individual and research usages. If you intend to utilize the model for commercial purposes, please reach out to cpm@modelbest.cn to negotiate commercial licensing.\n\nThe CPM-Bee base, governed by the [General Model License (GML)](https://github.com/OpenBMB/General-Model-License/blob/main/%E9%80%9A%E7%94%A8%E6%A8%A1%E5%9E%8B%E8%AE%B8%E5%8F%AF%E5%8D%8F%E8%AE%AE-%E6%9D%A5%E6%BA%90%E8%AF%B4%E6%98%8E-%E5%AE%A3%E4%BC%A0%E9%99%90%E5%88%B6-%E5%95%86%E4%B8%9A%E6%8E%88%E6%9D%83.md), permits commercial usage. If you intend to utilize the model for commercial purposes, please reach out to cpm@modelbest.cn to obtain the certificate of authorization.\n\n## ✅ Todo\n`VisCPM` is still undergoing continuous improvement, and we will further optimize it in the following aspects:\n- [ ] Enabling model quantization\n\n## 🏫 Institutions\n\nThis project is developed by the following institutions:\n\n- [THUNLP](https://nlp.csai.tsinghua.edu.cn/)\n- [ModelBest](https://modelbest.cn/)\n- [Zhihu](https://www.zhihu.com/ )\n\n\n## Citation\nPlease consider citing the following papers if our work is helpful to you\n```bibtex\n@article{VisCPM,\n title={Large multilingual models pivot zero-shot multimodal learning across languages},\n author={Hu, Jinyi and Yao, Yuan and Wang, Chongyi and Wang, Shan and Pan, Yinxu and Chen, Qianyu and Yu, Tianyu and Wu, Hanghao and Zhao, Yue and Zhang, Haoye and others},\n journal={arXiv preprint arXiv:2308.12038},\n year={2023}\n}\n\n@article{muffin,\n title={Reformulating vision-language foundation models and datasets towards universal multimodal assistants},\n author={Yu, Tianyu and Hu, Jinyi and Yao, Yuan and Zhang, Haoye and Zhao, Yue and Wang, Chongyi and Wang, Shan and Pan, Yinxv and Xue, Jiao and Li, Dahai and others},\n journal={arXiv preprint arXiv:2310.00653},\n year={2023}\n}\n```\n" }, { "path": "VisCPM/__init__.py", "content": "from VisCPM.viscpm_chat import VisCPMChat\nfrom VisCPM.viscpm_paint import VisCPMPaint" }, { "path": "VisCPM/cpm_tokenizers/__init__.py", "content": "from .bee import CPMBeeTokenizer\n" }, { "path": "VisCPM/cpm_tokenizers/bee.py", "content": "# coding=utf-8\n# Copyright 2022 The OpenBMB team.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport pkg_resources\nimport io\nimport os\nfrom typing import IO, Dict, List, Optional, Tuple\n\nfile_path = os.path.dirname(__file__)\n\ndef load_vocab(fp: IO[bytes]) -> Dict[str, int]:\n \"\"\"Loads a vocabulary file into a dictionary.\"\"\"\n vocab: Dict[str, int] = {}\n\n for token in fp.readlines():\n if token[-1] == \"\\n\":\n token = token[:-1]\n if len(token) == 0:\n continue\n vocab[token] = len(vocab)\n return vocab\n\n\nclass Token(object):\n def __init__(self, token: str, start: int, is_unk: bool, is_special: bool):\n self.token = token\n self.start = start\n self.is_unk = is_unk\n self.is_special = is_special\n\n def __str__(self):\n return \"Token(token={}, start={}, is_unk={}, is_special={})\".format(\n self.token, self.start, self.is_unk, self.is_special\n )\n\n def __repr__(self):\n return self.__str__()\n\n\nclass CPMBeeTokenizer(object):\n def __init__(\n self, old_version=False\n ):\n self.unk_token = \"\"\n self.mask_token = \"\"\n self.bos_token = \"\"\n self.eos_token = \"\"\n self.line_token = \"\\n\"\n self.space_token = \" \"\n\n if old_version:\n self.encoder = load_vocab(open(os.path.join(file_path, \"vocabs/bee-old.txt\"), encoding=\"utf-8\"))\n else:\n self.encoder = load_vocab(open(os.path.join(file_path, \"vocabs/bee.txt\"), encoding=\"utf-8\"))\n self.encoder[self.line_token] = self.encoder[\"\"]\n self.encoder[self.space_token] = self.encoder[\"\"]\n del self.encoder[\"\"]\n del self.encoder[\"\"]\n\n self.decoder = {v: k for k, v in self.encoder.items()}\n self._special_tokens = {\n k: v for k, v in self.encoder.items() if k.startswith(\"<\") and k.endswith(\">\")\n }\n\n self._max_word_len = max([len(x) for x in self.encoder.keys()])\n\n def get_piece(self, text: str) -> str:\n text = text[: self._max_word_len]\n len_text = len(text)\n for i in range(len(text)):\n sub = text[: len_text - i]\n if (sub in self.encoder) and (sub not in self._special_tokens):\n return sub\n return text[0]\n\n @property\n def vocab_size(self):\n return len(self.encoder)\n\n @property\n def eos_id(self):\n return self.encoder[self.eos_token]\n\n @property\n def bos_id(self):\n return self.encoder[self.bos_token]\n\n @property\n def unk_id(self):\n return self.encoder[self.unk_token]\n\n @property\n def mask_id(self):\n return self.encoder[self.mask_token]\n\n def __len__(self):\n return len(self.encoder)\n\n def tokenize(self, text: str) -> List[Token]:\n output_tokens: List[Token] = []\n\n sentence_split = [\"\"]\n is_escape = False\n is_special_token = False\n for i, c in enumerate(text):\n if is_special_token:\n if c == \"<\":\n raise ValueError(\"Invalid special token at pos {}\".format(i))\n elif c == \">\":\n # end of special token\n sentence_split[-1] += c\n is_special_token = False\n sentence_split.append(\"\")\n else:\n sentence_split[-1] += c\n else:\n if c == \"<\":\n if is_escape:\n # case: <<\n sentence_split[-1] += c\n is_escape = False\n else:\n # case: x<\n is_escape = True\n else:\n if is_escape:\n # case str:\n return text.replace(\"<\", \"<<\")\n\n @staticmethod\n def unescape(text: str) -> str:\n return text.replace(\"<<\", \"<\")\n\n def encode(\n self, text: str, past_table: Dict[int, str] = {}\n ) -> Tuple[List[int], Dict[int, str]]:\n ext_table_rev: Dict[str, int] = {}\n ext_table: Dict[int, str] = {}\n for idx, val in past_table.items():\n ext_table[idx] = val\n ext_table_rev[val] = idx\n ret = []\n for x in self.tokenize(text):\n if x.is_unk or (x.is_special and (x.token not in self.encoder)):\n if x.token not in ext_table_rev:\n ext_table_rev[x.token] = len(ext_table_rev) + self.vocab_size\n ext_table[ext_table_rev[x.token]] = x.token\n ret.append(ext_table_rev[x.token])\n elif x.token in self.encoder:\n ret.append(self.encoder[x.token])\n else:\n raise ValueError(\"Unknown token `{}` at pos {}\".format(x.token, x.start))\n\n return ret, ext_table\n\n def decode(self, tokens: List[int], ext_table: Optional[Dict[int, str]] = None):\n \"\"\"Decode ids into a string.\"\"\"\n if ext_table is None:\n ext_table = {}\n ret = []\n for token in tokens:\n if token in ext_table:\n ret.append(ext_table[token])\n else:\n if token >= 0:\n w = self.decoder[token]\n if w in self._special_tokens:\n ret.append(w)\n else:\n ret.append(self.escape(w))\n return \"\".join(ret)\n" }, { "path": "VisCPM/cpm_tokenizers/vocabs/bee.txt", "content": "\n\n\n\n\n\n\n\n\n\n