Copy disabled (too large)
Download .txt
Showing preview only (17,026K chars total). Download the full file to get everything.
Repository: microsoft/JARVIS
Branch: main
Commit: 7624cf388b47
Files: 90
Total size: 47.5 MB
Directory structure:
gitextract_ov6dal7w/
├── .gitignore
├── CITATION.cff
├── CODE_OF_CONDUCT.md
├── CONTRIBUTING.md
├── LICENSE
├── README.md
├── SECURITY.md
├── SUPPORT.md
├── easytool/
│ ├── README.md
│ ├── data_funcqa/
│ │ ├── funchub/
│ │ │ └── math.py
│ │ └── tool_instruction/
│ │ ├── functions_data.json
│ │ └── tool_dic.jsonl
│ ├── data_process.py
│ ├── data_restbench/
│ │ └── tool_instruction/
│ │ └── tmdb_tool.json
│ ├── data_toolbench/
│ │ └── tool_instruction/
│ │ └── toolbench_tool_instruction.json
│ ├── easytool/
│ │ ├── __init__.py
│ │ ├── funcQA.py
│ │ ├── restbench.py
│ │ ├── toolbench.py
│ │ ├── toolbench_retrieve.py
│ │ └── util.py
│ ├── main.py
│ └── requirements.txt
├── hugginggpt/
│ ├── .dockerignore
│ ├── Dockerfile.jetson
│ ├── README.md
│ ├── server/
│ │ ├── awesome_chat.py
│ │ ├── configs/
│ │ │ ├── config.azure.yaml
│ │ │ ├── config.default.yaml
│ │ │ ├── config.gradio.yaml
│ │ │ └── config.lite.yaml
│ │ ├── data/
│ │ │ └── p0_models.jsonl
│ │ ├── demos/
│ │ │ ├── demo_choose_model.json
│ │ │ ├── demo_parse_task.json
│ │ │ └── demo_response_results.json
│ │ ├── get_token_ids.py
│ │ ├── models/
│ │ │ ├── download.ps1
│ │ │ └── download.sh
│ │ ├── models_server.py
│ │ ├── requirements.txt
│ │ └── run_gradio_demo.py
│ └── web/
│ ├── electron/
│ │ ├── .npmrc
│ │ ├── electron-builder.yml
│ │ ├── main.js
│ │ ├── package.json
│ │ └── preload.js
│ ├── env.d.ts
│ ├── index.html
│ ├── package.json
│ ├── postcss.config.js
│ ├── src/
│ │ ├── App.vue
│ │ ├── api/
│ │ │ ├── chatgpt.ts
│ │ │ └── hugginggpt.ts
│ │ ├── assets/
│ │ │ └── tailwind.css
│ │ ├── components/
│ │ │ └── Loading.vue
│ │ ├── config/
│ │ │ └── index.ts
│ │ ├── main.ts
│ │ ├── prompt/
│ │ │ └── index.ts
│ │ ├── router/
│ │ │ └── index.ts
│ │ ├── types/
│ │ │ └── index.ts
│ │ └── views/
│ │ └── home.vue
│ ├── tailwind.config.js
│ ├── tsconfig.config.json
│ ├── tsconfig.json
│ └── vite.config.ts
└── taskbench/
├── README.md
├── batch_evaluate.sh
├── data_dailylifeapis/
│ ├── alignment_ids.json
│ ├── data.json
│ ├── graph_desc.json
│ ├── tool_desc.json
│ └── user_requests.json
├── data_engine.py
├── data_huggingface/
│ ├── alignment_ids.json
│ ├── data.json
│ ├── graph_desc.json
│ ├── tool_desc.json
│ └── user_requests.json
├── data_multimedia/
│ ├── alignment_ids.json
│ ├── data.json
│ ├── graph_desc.json
│ ├── tool_desc.json
│ └── user_requests.json
├── evaluate.py
├── format_data.py
├── generate_graph.py
├── graph_sampler.py
├── inference.py
├── requirements.txt
└── visualize_graph.py
================================================
FILE CONTENTS
================================================
================================================
FILE: .gitignore
================================================
# ALL
*.dev.yaml
legacy/
# for server
server/models/*
!server/models/download.sh
!server/models/download.ps1
server/logs/
server/models_dev
server/public/*
!server/public/examples/
server/public/examples/*
!server/public/examples/a.jpg
!server/public/examples/b.jpg
!server/public/examples/c.jpg
!server/public/examples/d.jpg
!server/public/examples/e.jpg
!server/public/examples/f.jpg
!server/public/examples/g.jpg
# docker
Dockerfile
docker-compose.yml
# for gradio
# server/run_gradio.py
# for web
web/node_modules
web/package-lock.json
web/dist
web/electron-dist
web/yarn.lock
# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class
# C extensions
*.so
# Distribution / packaging
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
pip-wheel-metadata/
share/python-wheels/
*.egg-info/
.installed.cfg
*.egg
MANIFEST
# PyInstaller
# Usually these files are written by a python script from a template
# before PyInstaller builds the exe, so as to inject date/other infos into it.
*.manifest
*.spec
# Installer logs
pip-log.txt
pip-delete-this-directory.txt
# Unit test / coverage reports
htmlcov/
.tox/
.nox/
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*.cover
*.py,cover
.hypothesis/
.pytest_cache/
# Translations
*.mo
*.pot
# Django stuff:
*.log
local_settings.py
db.sqlite3
db.sqlite3-journal
# Flask stuff:
instance/
.webassets-cache
# Scrapy stuff:
.scrapy
# Sphinx documentation
docs/_build/
# PyBuilder
target/
# Jupyter Notebook
.ipynb_checkpoints
# IPython
profile_default/
ipython_config.py
# pyenv
.python-version
# pipenv
# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
# However, in case of collaboration, if having platform-specific dependencies or dependencies
# having no cross-platform support, pipenv may install dependencies that don't work, or not
# install all needed dependencies.
#Pipfile.lock
# PEP 582; used by e.g. github.com/David-OConnor/pyflow
__pypackages__/
# Celery stuff
celerybeat-schedule
celerybeat.pid
# SageMath parsed files
*.sage.py
# Environments
.env
.venv
env/
venv/
ENV/
env.bak/
venv.bak/
# Spyder project settings
.spyderproject
.spyproject
# Rope project settings
.ropeproject
# mkdocs documentation
/site
# mypy
.mypy_cache/
.dmypy.json
dmypy.json
# Pyre type checker
.pyre/
================================================
FILE: CITATION.cff
================================================
cff-version: 1.2.0
message: "If you find this work useful in your method, you can cite the paper as below."
authors:
- family-names: Shen
given-names: Yongliang
- family-names: Song
given-names: Kaitao
- family-names: Tan
given-names: Xu
- family-names: Li
given-names: Dongsheng
- family-names: Lu
given-names: Weiming
- family-names: Zhuang
given-names: Yueting
title: "HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in HuggingFace"
version: 2.0.4
license: MIT
status: preprint
date-released: 2023-03-30
url: https://github.com/microsoft/JARVIS
preferred-citation:
type: article
authors:
- family-names: Shen
given-names: Yongliang
- family-names: Song
given-names: Kaitao
- family-names: Tan
given-names: Xu
- family-names: Li
given-names: Dongsheng
- family-names: Lu
given-names: Weiming
- family-names: Zhuang
given-names: Yueting
journal: "arXiv preprint arXiv:2303.17580"
title: "HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in HuggingFace"
year: 2023
url: https://arxiv.org/abs/2303.17580
================================================
FILE: CODE_OF_CONDUCT.md
================================================
# Microsoft Open Source Code of Conduct
This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/).
Resources:
- [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/)
- [Microsoft Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/)
- Contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with questions or concerns
================================================
FILE: CONTRIBUTING.md
================================================
To contribute to this GitHub project, you can follow these steps:
1. Fork the repository you want to contribute to by clicking the "Fork" button on the project page.
2. Clone the repository to your local machine and enter the newly created repo using the following commands:
```
git clone https://github.com/YOUR-GITHUB-USERNAME/JARVIS
cd JARVIS
```
3. Create a new branch for your changes using the following command:
```
git checkout -b "branch-name"
```
4. Make your changes to the code or documentation.
5. Add the changes to the staging area using the following command:
```
git add .
```
6. Commit the changes with a meaningful commit message using the following command:
```
git commit -m "your commit message"
```
7. Push the changes to your forked repository using the following command:
```
git push origin branch-name
```
8. Go to the GitHub website and navigate to your forked repository.
9. Click the "New pull request" button.
10. Select the branch you just pushed to and the branch you want to merge into on the original repository.
11. Add a description of your changes and click the "Create pull request" button.
12. Wait for the project maintainer to review your changes and provide feedback.
13. Make any necessary changes based on feedback and repeat steps 5-12 until your changes are accepted and merged into the main project.
14. Once your changes are merged, you can update your forked repository and local copy of the repository with the following commands:
```
git fetch upstream
git checkout main
git merge upstream/main
```
Finally, delete the branch you created with the following command:
```
git branch -d branch-name
```
That's it you made it 🐣⭐⭐
================================================
FILE: LICENSE
================================================
MIT License
Copyright (c) Microsoft Corporation.
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE
================================================
FILE: README.md
================================================
# JARVIS
[](https://arxiv.org/abs/2303.17580)
[](https://huggingface.co/spaces/microsoft/HuggingGPT)
The mission of JARVIS is to explore artificial general intelligence (AGI) and deliver cutting-edge research to the whole community.
## What's New
+ [2024.01.15] We release Easytool for easier tool usage.
+ The code and datasets are available at [EasyTool](/easytool).
+ The paper is available at [EasyTool: Enhancing LLM-based Agents with Concise Tool Instruction](https://arxiv.org/abs/2401.06201).
+ [2023.11.30] We release TaskBench for evaluating task automation capability of LLMs.
+ The code and datasets are available at [TaskBench](/taskbench).
+ The paper is available at [TaskBench: Benchmarking Large Language Models for Task Automation](https://arxiv.org/abs/2311.18760).
+ [2023.07.28] We are now in the process of planning evaluation and project rebuilding. We will release a new version of Jarvis in the near future.
+ [2023.07.24] We released a light langchain version of Jarvis. See <a href="https://github.com/langchain-ai/langchain/tree/master/libs/experimental/langchain_experimental/autonomous_agents/hugginggpt">here</a>.
+ [2023.04.16] Jarvis now supports the OpenAI service on the Azure platform and the GPT-4 model.
+ [2023.04.06] We added the Gradio demo and built the web API for `/tasks` and `/results` in `server` mode.
+ The Gradio demo is now hosted on Hugging Face Space. (Build with `inference_mode=hybrid` and `local_deployment=standard`)
+ The Web API `/tasks` and `/results` access intermediate results for `Stage #1`: task planning and `Stage #1-3`: model selection with execution results. See <a href="#Server">here</a>.
+ [2023.04.03] We added the CLI mode and provided parameters for configuring the scale of local endpoints.
+ You can enjoy a lightweight experience with Jarvis without deploying the models locally. See <a href="#Configuration">here</a>.
+ Just run `python awesome_chat.py --config configs/config.lite.yaml` to experience it.
+ [2023.04.01] We updated a version of code for building.
### Overview
Language serves as an interface for LLMs to connect numerous AI models for solving complicated AI tasks!
<p align="center">
<img width="100%" alt="image" src="./hugginggpt/assets/intro.png">
</p>
See our paper: [HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in HuggingFace](http://arxiv.org/abs/2303.17580), Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu and Yueting Zhuang (the first two authors contribute equally)
We introduce a collaborative system that consists of **an LLM as the controller** and **numerous expert models as collaborative executors** (from HuggingFace Hub). The workflow of our system consists of four stages:
+ **Task Planning**: Using ChatGPT to analyze the requests of users to understand their intention, and disassemble them into possible solvable tasks.
+ **Model Selection**: To solve the planned tasks, ChatGPT selects expert models hosted on Hugging Face based on their descriptions.
+ **Task Execution**: Invokes and executes each selected model, and return the results to ChatGPT.
+ **Response Generation**: Finally, using ChatGPT to integrate the prediction of all models, and generate responses.
<p align="center"><img src="./hugginggpt/assets/overview.jpg"></p>
### System Requirements
#### Default (Recommended)
For `configs/config.default.yaml`:
+ Ubuntu 16.04 LTS
+ VRAM >= 24GB
+ RAM > 12GB (minimal), 16GB (standard), 80GB (full)
+ Disk > 284GB
+ 42GB for `damo-vilab/text-to-video-ms-1.7b`
+ 126GB for `ControlNet`
+ 66GB for `stable-diffusion-v1-5`
+ 50GB for others
#### Minimum (Lite)
For `configs/config.lite.yaml`:
+ Ubuntu 16.04 LTS
+ Nothing else
The configuration `configs/config.lite.yaml` does not require any expert models to be downloaded and deployed locally. However, it means that Jarvis is restricted to models running stably on HuggingFace Inference Endpoints.
### Quick Start
First replace `openai.key` and `huggingface.token` in `server/configs/config.default.yaml` with **your personal OpenAI Key** and **your Hugging Face Token**, or put them in the environment variables `OPENAI_API_KEY` and `HUGGINGFACE_ACCESS_TOKEN` respectively. Then run the following commands:
<span id="Server"></span>
#### For Server:
```bash
# setup env
cd server
conda create -n jarvis python=3.8
conda activate jarvis
conda install pytorch torchvision torchaudio pytorch-cuda=11.7 -c pytorch -c nvidia
pip install -r requirements.txt
# download models. Make sure that `git-lfs` is installed.
cd models
bash download.sh # required when `inference_mode` is `local` or `hybrid`.
# run server
cd ..
python models_server.py --config configs/config.default.yaml # required when `inference_mode` is `local` or `hybrid`
python awesome_chat.py --config configs/config.default.yaml --mode server # for text-davinci-003
```
Now you can access Jarvis' services by the Web API.
+ `/hugginggpt` --method `POST`, access the full service.
+ `/tasks` --method `POST`, access intermediate results for Stage #1.
+ `/results` --method `POST`, access intermediate results for Stage #1-3.
For example:
```bash
# request
curl --location 'http://localhost:8004/tasks' \
--header 'Content-Type: application/json' \
--data '{
"messages": [
{
"role": "user",
"content": "based on pose of /examples/d.jpg and content of /examples/e.jpg, please show me a new image"
}
]
}'
# response
[{"args":{"image":"/examples/d.jpg"},"dep":[-1],"id":0,"task":"openpose-control"},{"args":{"image":"/examples/e.jpg"},"dep":[-1],"id":1,"task":"image-to-text"},{"args":{"image":"<GENERATED>-0","text":"<GENERATED>-1"},"dep":[1,0],"id":2,"task":"openpose-text-to-image"}]
```
#### For Web:
We provide a user-friendly web page. After starting `awesome_chat.py` in a server mode, you can run the commands to communicate with Jarvis in your browser:
- you need to install `nodejs` and `npm` first.
- [ IMPORTANT ] if you are running the web client on another machine, you need set `http://{LAN_IP_of_the_server}:{port}/` to `HUGGINGGPT_BASE_URL` of `web/src/config/index.ts`.
- if you want to use the video generation feature, you need to compile `ffmpeg` manually with H.264.
- you can switch to ChatGPT by `double click` on the setting icon!
```bash
cd web
npm install
npm run dev
```
```bash
# Optional: Install ffmpeg
# This command need be executed without errors.
LD_LIBRARY_PATH=/usr/local/lib /usr/local/bin/ffmpeg -i input.mp4 -vcodec libx264 output.mp4
```
<span id="Gradio"></span>
#### For Gradio
The Gradio demo is now hosted on Hugging Face Space. You can also run the following commands to start the demo locally:
```bash
python models_server.py --config configs/config.gradio.yaml
python run_gradio_demo.py --config configs/config.gradio.yaml
# or run the HF Space as a Docker image (Build with `inference_mode=hibrid` and `local_deployment=standard`)
docker run -it -p 7860:7860 --platform=linux/amd64 registry.hf.space/microsoft-hugginggpt:latest python app.py
```
#### For CLI:
You can also run Jarvis more easily in CLI mode:
```bash
cd server
python awesome_chat.py --config configs/config.default.yaml --mode cli
```
Examples of CLI mode:
```
Welcome to Jarvis! A collaborative system that consists of an LLM as the controller and numerous expert models as collaborative executors. Jarvis can plan tasks, schedule Hugging Face models, generate friendly responses based on your requests, and help you with many things. Please enter your request (`exit` to exit).
[ User ]: Given a collection of image A: /examples/a.jpg, B: /examples/b.jpg, C: /examples/c.jpg, please tell me how many zebras in these picture?
[ Jarvis ]: Based on your request, I have used several models to process the images you provided. For the image A: /examples/a.jpg, I used the image-to-text model nlpconnect/vit-gpt2-image-captioning to generate the text "a cat sitting on a window sill looking out". Then I used the object-detection model facebook/detr-resnet-50 to generate the image with predicted box '/images/a3dc.jpg', which contains two objects with labels 'potted plant' and 'cat'. For the image B: /examples/b.jpg, I used the image-to-text model nlpconnect/vit-gpt2-image-captioning to generate the text "two zebras standing in a field of dry grass". Then I used the object-detection model facebook/detr-resnet-50 to generate the image with predicted box '/images/f5df.jpg', which contains three objects with labels 'zebra'. For the image C: /examples/c.jpg, I used the image-to-text model nlpconnect/vit-gpt2-image-captioning to generate the text "a zebra standing in a field of tall grass". Then I used the object-detection model facebook/detr-resnet-50 to generate the image with predicted box '/images/69c7.jpg', which contains one object with label 'zebra'. Finally, I used the canny-control model canny-control to generate the image '/images/7e18.png' based on the image /examples/f.jpg. Therefore, there are four zebras in these pictures. Is there anything else I can help you with?
[ User ]: Please answer all the named entities in the sentence: Iron Man is a superhero appearing in American comic books published by Marvel Comics. The character was co-created by writer and editor Stan Lee, developed by scripter Larry Lieber, and designed by artists Don Heck and Jack Kirby.
[ Jarvis ]: Sure, I understand your request. Based on the inference results of the models, I will answer your request step by step. Firstly, I used the token-classification model dslim/bert-base-NER to recognize the named entities in the sentence. The inference result is that the named entities are Iron Man, Stan Lee, Larry Lieber, Don Heck and Jack Kirby. Then, I used the question-answering model deepset/roberta-base-squad2 to answer your request. The inference result is that there is no output since the context cannot be empty. Therefore, I cannot make it. I hope this answer your request. Is there anything else I can help you with?
[ User ]: Please generate a canny image based on /examples/f.jpg
[ Jarvis ]: Sure. I have generated a canny image based on /examples/f.jpg. To do this, I first used the image-to-text model nlpconnect/vit-gpt2-image-captioning to generate the text description of the image, which is "a herd of giraffes and zebras grazing in a field". Then I used the canny-control model to generate the canny image of the original image, which is stored in the path "/images/1ea2.png". Finally, I used the canny-text-to-image model lllyasviel/sd-controlnet-canny to generate the canny image based on the text description and the canny image, which is stored in the path "/images/ba66.png". Do you need anything else?
```
### Configuration
The server-side configuration file is `server/configs/config.default.yaml`, and some parameters are presented as follows:
+ `model`: LLM, currently supports `text-davinci-003`. We are working on integrating more open-source LLMs.
+ `inference_mode`: mode of inference endpoints
+ `local`: only use the local inference endpoints
+ `huggingface`: only use the Hugging Face Inference Endpoints **(free of local inference endpoints)**
+ `hybrid`: both of `local` and `huggingface`
+ `local_deployment`: scale of locally deployed models, works under `local` or `hybrid` inference mode:
+ `minimal` (RAM>12GB, ControlNet only)
+ `standard` (RAM>16GB, ControlNet + Standard Pipelines)
+ `full` (RAM>42GB, All registered models)
On a personal laptop, we recommend the configuration of `inference_mode: hybrid `and `local_deployment: minimal`. But the available models under this setting may be limited due to the instability of remote Hugging Face Inference Endpoints.
### NVIDIA Jetson Embedded Device Support
A [Dockerfile](./Dockerfile.jetson) is included that provides experimental support for [NVIDIA Jetson embedded devices](https://developer.nvidia.com/embedded-computing). This image provides accelerated ffmpeg, pytorch, torchaudio, and torchvision dependencies. To build the docker image, [ensure that the default docker runtime is set to 'nvidia'](https://github.com/NVIDIA/nvidia-docker/wiki/Advanced-topics#default-runtime). A pre-built image is provided at https://hub.docker.com/r/toolboc/nv-jarvis.
```bash
#Build the docker image
docker build --pull --rm -f "Dockerfile.jetson" -t toolboc/nv-jarvis:r35.2.1
```
Due to to memory requirements, JARVIS is required to run on Jetson AGX Orin family devices (64G on-board RAM device preferred) with config options set to:
* `inference_mode: local`
* `local_deployment: standard`
Models and configs are recommended to be provided through a volume mount from the host to the container as shown in the `docker run` step below. It is possible to uncomment the `# Download local models` section of the [Dockerfile](./Dockerfile.jetson) to build a container with models included.
#### Start the model server, awesomechat, and web app on Jetson Orin AGX
```bash
# run the container which will automatically start the model server
docker run --name jarvis --net=host --gpus all -v ~/jarvis/configs:/app/server/configs -v ~/src/JARVIS/server/models:/app/server/models toolboc/nv-jarvis:r35.2.1
# (wait for model server to complete initialization)
# start awesome_chat.py
docker exec jarvis python3 awesome_chat.py --config configs/config.default.yaml --mode server
#start the web application (application will be acessible at http://localhost:9999)
docker exec jarvis npm run dev --prefix=/app/web
```
### Screenshots
<p align="center"><img src="./hugginggpt/assets/screenshot_q.jpg"><img src="./hugginggpt/assets/screenshot_a.jpg"></p>
## Citation
If you find this work useful in your method, you can cite the paper as below:
@inproceedings{shen2023hugginggpt,
author = {Shen, Yongliang and Song, Kaitao and Tan, Xu and Li, Dongsheng and Lu, Weiming and Zhuang, Yueting},
booktitle = {Advances in Neural Information Processing Systems},
title = {HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in HuggingFace},
year = {2023}
}
>
@article{shen2023taskbench,
title = {TaskBench: Benchmarking Large Language Models for Task Automation},
author = {Shen, Yongliang and Song, Kaitao and Tan, Xu and Zhang, Wenqi and Ren, Kan and Yuan, Siyu and Lu, Weiming and Li, Dongsheng and Zhuang, Yueting},
journal = {arXiv preprint arXiv:2311.18760},
year = {2023}
}
>
@article{yuan2024easytool,
title = {EASYTOOL: Enhancing LLM-based Agents with Concise Tool Instruction},
author = {Siyu Yuan and Kaitao Song and Jiangjie Chen and Xu Tan and Yongliang Shen and Ren Kan and Dongsheng Li and Deqing Yang},
journal = {arXiv preprint arXiv:2401.06201},
year = {2024}
}
================================================
FILE: SECURITY.md
================================================
<!-- BEGIN MICROSOFT SECURITY.MD V0.0.8 BLOCK -->
## Security
Microsoft takes the security of our software products and services seriously, which includes all source code repositories managed through our GitHub organizations, which include [Microsoft](https://github.com/microsoft), [Azure](https://github.com/Azure), [DotNet](https://github.com/dotnet), [AspNet](https://github.com/aspnet), [Xamarin](https://github.com/xamarin), and [our GitHub organizations](https://opensource.microsoft.com/).
If you believe you have found a security vulnerability in any Microsoft-owned repository that meets [Microsoft's definition of a security vulnerability](https://aka.ms/opensource/security/definition), please report it to us as described below.
## Reporting Security Issues
**Please do not report security vulnerabilities through public GitHub issues.**
Instead, please report them to the Microsoft Security Response Center (MSRC) at [https://msrc.microsoft.com/create-report](https://aka.ms/opensource/security/create-report).
If you prefer to submit without logging in, send email to [secure@microsoft.com](mailto:secure@microsoft.com). If possible, encrypt your message with our PGP key; please download it from the [Microsoft Security Response Center PGP Key page](https://aka.ms/opensource/security/pgpkey).
You should receive a response within 24 hours. If for some reason you do not, please follow up via email to ensure we received your original message. Additional information can be found at [microsoft.com/msrc](https://aka.ms/opensource/security/msrc).
Please include the requested information listed below (as much as you can provide) to help us better understand the nature and scope of the possible issue:
* Type of issue (e.g. buffer overflow, SQL injection, cross-site scripting, etc.)
* Full paths of source file(s) related to the manifestation of the issue
* The location of the affected source code (tag/branch/commit or direct URL)
* Any special configuration required to reproduce the issue
* Step-by-step instructions to reproduce the issue
* Proof-of-concept or exploit code (if possible)
* Impact of the issue, including how an attacker might exploit the issue
This information will help us triage your report more quickly.
If you are reporting for a bug bounty, more complete reports can contribute to a higher bounty award. Please visit our [Microsoft Bug Bounty Program](https://aka.ms/opensource/security/bounty) page for more details about our active programs.
## Preferred Languages
We prefer all communications to be in English.
## Policy
Microsoft follows the principle of [Coordinated Vulnerability Disclosure](https://aka.ms/opensource/security/cvd).
<!-- END MICROSOFT SECURITY.MD BLOCK -->
================================================
FILE: SUPPORT.md
================================================
# TODO: The maintainer of this repo has not yet edited this file
**REPO OWNER**: Do you want Customer Service & Support (CSS) support for this product/project?
- **No CSS support:** Fill out this template with information about how to file issues and get help.
- **Yes CSS support:** Fill out an intake form at [aka.ms/onboardsupport](https://aka.ms/onboardsupport). CSS will work with/help you to determine next steps.
- **Not sure?** Fill out an intake as though the answer were "Yes". CSS will help you decide.
*Then remove this first heading from this SUPPORT.MD file before publishing your repo.*
# Support
## How to file issues and get help
This project uses GitHub Issues to track bugs and feature requests. Please search the existing
issues before filing new issues to avoid duplicates. For new issues, file your bug or
feature request as a new Issue.
For help and questions about using this project, please **REPO MAINTAINER: INSERT INSTRUCTIONS HERE
FOR HOW TO ENGAGE REPO OWNERS OR COMMUNITY FOR HELP. COULD BE A STACK OVERFLOW TAG OR OTHER
CHANNEL. WHERE WILL YOU HELP PEOPLE?**.
## Microsoft Support Policy
Support for this **PROJECT or PRODUCT** is limited to the resources listed above.
================================================
FILE: easytool/README.md
================================================
<p align="center">
<img src="./assets/logo.png" width="15%"> <br>
</p>
<div align="center">
<h1>EasyTool</h1>
<h3>Enhancing LLM-based Agents with Concise Tool Instruction<h3>
</div>
## What's New
+ [2024.01.15] We release Easytool for easier tool usage.
+ The code and datasets are available at [easytool](#).
+ The paper is available at [EASYTOOL: Enhancing LLM-based Agents with Concise Tool Instruction](https://arxiv.org/abs/2401.06201).
## Overview
LLM-based agents usually employ tool documentation to grasp the selection and usage of tools from different sources, but these documentations could be inconsistent in formats, redundant with excessive length, and lacking demonstrations for instructions.
EasyTool is an easy but effective method to create clear, structured, and unified instructions from tool documentations for improving LLM-based agents in using tools.
<p align="center">
<img width="70%" alt="image" src="./assets/front.png">
</p>
## Experiment
### Prerequisites
- Prepare requirements: `pip install -r requirements.txt`
- Data Construction: `python3 data_process.py`
Before running any of the commands, ensure that you have set the necessary API keys. Replace `""` with your actual keys.
```bash
export OPENAI_API_KEY="your_openai_api_key_here"
export RAPIDAPI_KEY="your_rapidapi_key_here"
```
### ToolBench
You need first get the tool execution code (./data/toolenv/tools.) from the following link: [Google Drive](https://drive.google.com/drive/folders/1yBUQ732mPu-KclJnuQELEhtKakdXFc3J) or [Tsinghua Cloud](https://cloud.tsinghua.edu.cn/f/c9e50625743b40bfbe10/) and then save them to ./toolenv/tools
To inference with LLMs, run the following commands:
```bash
unzip data_toolbench/tool_instruction/API_description_embeddings.zip -d data_toolbench/tool_instruction/
export OPENAI_API_KEY=""
export RAPIDAPI_KEY=""
python3 main.py \
--model_name gpt-3.5-turbo \
--task toolbench \
--data_type G2 \
--tool_root_dir ./toolenv/tools
python3 main.py \
--model_name gpt-3.5-turbo \
--task toolbench \
--data_type G3 \
--tool_root_dir ./toolenv/tools
python3 main.py \
--model_name gpt-3.5-turbo \
--task toolbench_retrieve \
--data_type G2 \
--tool_root_dir ./toolenv/tools
python3 main.py \
--model_name gpt-3.5-turbo \
--task toolbench_retrieve \
--data_type G3 \
--tool_root_dir ./toolenv/tools
```
### FuncQA
To inference with LLMs, run the following commands:
```bash
export OPENAI_API_KEY=""
python3 main.py \
--model_name gpt-3.5-turbo \
--task funcqa \
--data_type funcqa_mh
python3 main.py \
--model_name gpt-3.5-turbo \
--task funcqa \
--data_type funcqa_oh
```
### RestBench
To inference with LLMs, run the following commands:
```bash
export OPENAI_API_KEY=""
python3 main.py \
--model_name gpt-3.5-turbo \
--task restbench
```
## Citation
If you find this work useful in your method, you can cite the paper as below:
@article{yuan2024easytool,
title = {EASYTOOL: Enhancing LLM-based Agents with Concise Tool Instruction},
author = {Siyu Yuan and Kaitao Song and Jiangjie Chen and Xu Tan and Yongliang Shen and Ren Kan and Dongsheng Li and Deqing Yang},
journal = {arXiv preprint arXiv:2401.06201},
year = {2024}
}
## Acknowledgement
- [ChatGPT](https://platform.openai.com/)
- [Hugging Face](https://huggingface.co/)
- [ToolBench](https://github.com/OpenBMB/ToolBench)
- [RestBench](https://github.com/Yifan-Song793/RestGPT)
- [FuncQA](https://github.com/Ber666/ToolkenGPT)
================================================
FILE: easytool/data_funcqa/funchub/math.py
================================================
import math
# this function is used to round the result to 2 decimal places
# e.g. 52.3523 -> 52.35, 52.0011 -> 52, 0.00000233 -> 0.0000023
def custom_round(x, decimal_places=2):
str_x = f"{x:.10f}"
before_decimal = str_x.split('.')[0]
after_decimal = str_x.split('.')[1]
leading_zeros = len(after_decimal) - len(after_decimal.lstrip('0'))
if leading_zeros >= 1 and before_decimal == "0":
return round(x, leading_zeros + 2)
else:
return round(x, decimal_places)
# this function converts a number in scientific notation to decimal notation
def scito_decimal(sci_str):
def split_exponent(number_str):
parts = number_str.split("e")
coefficient = parts[0]
exponent = int(parts[1]) if len(parts) == 2 else 0
return coefficient, exponent
def multiplyby_10(number_str, exponent):
if exponent == 0:
return number_str
if exponent > 0:
index = number_str.index(".") if "." in number_str else len(number_str)
number_str = number_str.replace(".", "")
new_index = index + exponent
number_str += "0" * (new_index - len(number_str))
if new_index < len(number_str):
number_str = number_str[:new_index] + "." + number_str[new_index:]
return number_str
if exponent < 0:
index = number_str.index(".") if "." in number_str else len(number_str)
number_str = number_str.replace(".", "")
new_index = index + exponent
number_str = "0" * (-new_index) + number_str
number_str = "0." + number_str
return number_str
coefficient, exponent = split_exponent(sci_str)
decimal_str = multiplyby_10(coefficient, exponent)
# remove trailing zeros
if "." in decimal_str:
decimal_str = decimal_str.rstrip("0")
return decimal_str
# normalize the result to 2 decimal places and remove trailing zeros
def normalize(res, round_to=2):
# we round the result to 2 decimal places
res = custom_round(res, round_to)
res = str(res)
if "." in res:
while res[-1] == "0":
res = res[:-1]
res = res.strip(".")
# scientific notation
if "e" in res:
res = scito_decimal(res)
return res
# 1. add
def add_(args):
return normalize(sum(args))
# 2. subtract
def subtract_(args):
res = args[0]
for arg in args[1:]:
res -= arg
return normalize(res)
# 3. multiply
def multiply_(args):
res = args[0]
for arg in args[1:]:
res *= arg
return normalize(res)
# 4. divide
def divide_(args):
res = args[0]
for arg in args[1:]:
res /= arg
return normalize(res)
# 5. power
def power_(args):
res = args[0]
for arg in args[1:]:
res **= arg
return normalize(res)
# 6. square root
def sqrt_(args):
res = args[0]
return normalize(math.sqrt(res))
# 7. 10th log
def log_(args):
# if only one argument is passed, it is 10th log
if len(args) == 1:
res = args[0]
return normalize(math.log10(res))
# if two arguments are passed, it is log with base as the second argument
elif len(args) == 2:
res = args[0]
base = args[1]
return normalize(math.log(res, base))
else:
raise Exception("Invalid number of arguments passed to log function")
# 8. natural log
def ln_(args):
res = args[0]
return normalize(math.log(res))
# 9. choose
def choose_(args):
n = args[0]
r = args[1]
return normalize(math.comb(n, r))
# 10. permutation
def permutate_(args):
n = args[0]
r = args[1]
return normalize(math.perm(n, r))
# 11. greatest common divisor
def gcd_(args):
res = args[0]
for arg in args[1:]:
res = math.gcd(res, arg)
return normalize(res)
# 12. least common multiple
def lcm_(args):
res = args[0]
for arg in args[1:]:
res = res * arg // math.gcd(res, arg)
return normalize(res)
# 13. remainder
def remainder_(args):
dividend = args[0]
divisor = args[1]
return normalize(dividend % divisor)
================================================
FILE: easytool/data_funcqa/tool_instruction/functions_data.json
================================================
{
"0":{
"ID":0,
"standardized_name":"add_",
"API_description":"'add_' returns the sum of all the arguments passed to it, normalized to 2 decimal places.",
"Usage":{
"required_parameters":[
{
"name":"input",
"type":"List"
}
],
"Example":{
"Scenario":"if you want to add 2 to 1.",
"Parameters":{
"input":[2,1]
}
}
}
},
"1": {
"ID": 1,
"standardized_name": "subtract_",
"API_description": "'subtract_' returns the difference of the arguments passed to it, starting with the first argument and subtracting all subsequent arguments, normalized to 2 decimal places.",
"Usage": {
"required_parameters": [
{
"name": "input",
"type": "List"
}
],
"Example": {
"Scenario": "if you want to subtract 2 from 1.",
"Parameters": {
"input": [1,2]
}
}
}
},
"2": {
"ID": 2,
"standardized_name": "multiply_",
"API_description": "'multiply_' returns the product of all the arguments passed to it, normalized to 2 decimal places.",
"Usage": {
"required_parameters": [
{
"name": "input",
"type": "List"
}
],
"Example": {
"Scenario": "if you want to calculate 2*1.",
"Parameters": {
"input": [2,1]
}
}
}
},
"3": {
"ID": 3,
"standardized_name": "divide_",
"API_description": "'divide_' returns the quotient of the first argument divided by all the subsequent arguments, normalized to 2 decimal places.",
"Usage": {
"required_parameters": [
{
"name": "input",
"type": "List"
}
],
"Example": {
"Scenario": "if you want to calculate 4/2.",
"Parameters": {
"input": [4,2]
}
}
}
},
"4": {
"ID": 4,
"standardized_name": "power_",
"API_description": "'power_' returns the result of raising the first argument to the power of all the subsequent arguments, normalized to 2 decimal places.",
"Usage": {
"required_parameters": [
{
"name": "input",
"type": "List"
}
],
"Example": {
"Scenario": "if you want to calculate 2^3.",
"Parameters": {
"input": [2,3]
}
}
}
},
"5": {
"ID": 5,
"standardized_name": "sqrt_",
"API_description": "'sqrt_' returns the square root of the first argument, normalized to 2 decimal places.",
"Usage": {
"required_parameters": [
{
"name": "input",
"type": "List"
}
],
"Example": {
"Scenario": "if you want to get the square root of 9.",
"Parameters": {
"input": [9]
}
}
}
},
"6": {
"ID": 6,
"standardized_name": "log_",
"API_description": "'log_' returns the base-10 logarithm of the first argument if one argument is provided, or the logarithm with base as the second argument if two arguments are provided, normalized to 2 decimal places.",
"Usage": {
"required_parameters": [
{
"name": "input",
"type": "List"
}
],
"Example": {
"Scenario": "if you want to get the logarithm of 100 with base 10.",
"Parameters": {
"input": [100,10]
}
}
}
},
"7": {
"ID": 7,
"standardized_name": "ln_",
"API_description": "'ln_' returns the natural logarithm of the first argument, normalized to 2 decimal places.",
"Usage": {
"required_parameters": [
{
"name": "input",
"type": "List"
}
],
"Example": {
"Scenario": "if you want to get the natural logarithm of 2.718.",
"Parameters": {
"input": [2.718281828459045]
}
}
}
},
"8": {
"ID": 8,
"standardized_name": "lcm_",
"API_description": "'lcm_' returns the least common multiple of all the arguments passed to it, normalized to 2 decimal places.",
"Usage": {
"required_parameters": [
{
"name": "input",
"type": "List"
}
],
"Example": {
"Scenario": "if you want to find the LCM of 12 and 18.",
"Parameters": {
"input": [12,18]
}
}
}
},
"9": {
"ID": 9,
"standardized_name": "gcd_",
"API_description": "'gcd_' returns the greatest common divisor of all the arguments passed to it, normalized to 2 decimal places.",
"Usage": {
"required_parameters": [
{
"name": "input",
"type": "List"
}
],
"Example": {
"Scenario": "if you want to find the GCD of 54 and 24.",
"Parameters": {
"input": [54,24]
}
}
}
},
"10": {
"ID": 10,
"standardized_name": "remainder_",
"API_description": "'remainder_' returns the remainder of the division of the first argument by the second argument, normalized to 2 decimal places.",
"Usage": {
"required_parameters": [
{
"name": "input",
"type": "List"
}
],
"Example": {
"Scenario": "if you want to find the remainder of 10 divided by 3.",
"Parameters": {
"input": [10,3]
}
}
}
},
"11": {
"ID": 11,
"standardized_name": "choose_",
"API_description": "'choose_' returns the number of ways to choose 'r' items from 'n' options without regard to order, normalized to 2 decimal places.",
"Usage": {
"required_parameters": [
{
"name": "input",
"type": "List"
}
],
"Example": {
"Scenario": "if you want to choose 2 items out of 5.",
"Parameters": {
"input": [5,2]
}
}
}
},
"12": {
"ID": 12,
"standardized_name": "permutate_",
"API_description": "'permutate_' returns the number of ways to arrange 'r' items out of 'n' options, normalized to 2 decimal places.",
"Usage": {
"required_parameters": [
{
"name": "input",
"type": "List"
}
],
"Example": {
"Scenario": "if you want to find the number of arrangements of 3 items out of 5.",
"Parameters": {
"input": [5,3]
}
}
}
}
}
================================================
FILE: easytool/data_funcqa/tool_instruction/tool_dic.jsonl
================================================
{"ID": 0, "description": "'add_' returns the sum of all the arguments passed to it, normalized to 2 decimal places."}
{"ID": 1, "description": "'subtract_' returns the difference of the arguments passed to it, starting with the first argument and subtracting all subsequent arguments, normalized to 2 decimal places."}
{"ID": 2, "description": "'multiply_' returns the product of all the arguments passed to it, normalized to 2 decimal places."}
{"ID": 3, "description": "'divide_' returns the quotient of the first argument divided by all the subsequent arguments, normalized to 2 decimal places."}
{"ID": 4, "description": "'power_' returns the result of raising the first argument to the power of all the subsequent arguments, normalized to 2 decimal places."}
{"ID": 5, "description": "'sqrt_' returns the square root of the first argument, normalized to 2 decimal places."}
{"ID": 6, "description": "'log_' returns the base-10 logarithm of the first argument if one argument is provided, or the logarithm with base as the second argument if two arguments are provided, normalized to 2 decimal places."}
{"ID": 7, "description": "'ln_' returns the natural logarithm of the first argument, normalized to 2 decimal places."}
{"ID": 8, "description": "'lcm_' returns the least common multiple of all the arguments passed to it, normalized to 2 decimal places."}
{"ID": 9, "description": "'gcd_' returns the greatest common divisor of all the arguments passed to it, normalized to 2 decimal places."}
{"ID": 10, "description": "'remainder_' returns the remainder of the division of the first argument by the second argument, normalized to 2 decimal places."}
{"ID": 11, "description": "'choose_' returns the number of ways to choose 'r' items from 'n' options without regard to order, normalized to 2 decimal places."}
{"ID": 12, "description": "'permutate_' returns the number of ways to arrange 'r' items out of 'n' options, normalized to 2 decimal places."}
================================================
FILE: easytool/data_process.py
================================================
import os
import gdown
import shutil
import json
from zipfile import ZipFile
urls = {
"funcqa": "https://drive.google.com/uc?id=13Sj7uIsyqWXoTh1ejWUviTzeQSES2Omd",
"restbench": "https://raw.githubusercontent.com/Yifan-Song793/RestGPT/main/datasets/tmdb.json",
"toolbench": "https://drive.google.com/uc?id=1XFjDxVZdUY7TXYF2yvzx3pJlS2fy78jk",
}
def read_jsonline(address):
not_mark = []
with open(address, 'r', encoding="utf-8") as f:
for jsonstr in f.readlines():
jsonstr = json.loads(jsonstr)
not_mark.append(jsonstr)
return not_mark
def read_json(address):
with open(address, 'r', encoding='utf-8') as json_file:
json_data = json.load(json_file)
return json_data
def toolbench_process(data_file, dataset):
ls = read_json(data_file)
all_data = read_json(f"{dataset}/tool_instruction/toolbench_tool_instruction.json")
all_dic = {}
for ID in all_data.keys():
all_dic[all_data[ID]["tool_name"]] = all_data[ID]
not_in = []
for data in ls:
Tool_dic = []
data_dic = {}
already = []
for tool in data['api_list']:
if tool['tool_name'] in all_dic:
if all_dic[tool['tool_name']]["ID"] not in already:
already.append(all_dic[tool['tool_name']]["ID"])
Tool_dic.append({"ID": all_dic[tool['tool_name']]["ID"],
"Description": all_dic[tool['tool_name']]["tool_description"], })
data["Tool_dic"] = Tool_dic
json_str = json.dumps(ls, indent=4)
with open(data_file, 'w', encoding='utf-8') as json_file:
json.dump(ls, json_file, ensure_ascii=False, indent=4)
def main():
curr_dir = os.path.dirname(__file__)
for dataset in [
"funcqa",
"restbench",
"toolbench"
]:
dataset_path = os.path.join(curr_dir, "data_{}".format(dataset), "test_data")
if not os.path.exists(dataset_path):
os.makedirs(dataset_path)
if dataset == "funcqa":
print("Processing FuncQA dataset ...\n")
# Processing FuncQA dataset ...
temp_file = os.path.join(dataset_path, "data_toolkengpt_0918.zip")
gdown.download(urls[dataset], temp_file, quiet=False)
zf = ZipFile(temp_file, 'r')
zf.extract("data/funcqa/funcqa_oh.json", ".")
zf.extract("data/funcqa/funcqa_mh.json", ".")
os.rename("data/funcqa/funcqa_oh.json", "{}/funcqa_oh.json".format(dataset_path))
os.rename("data/funcqa/funcqa_mh.json", "{}/funcqa_mh.json".format(dataset_path))
os.remove(temp_file)
shutil.rmtree("data")
print("FuncQA dataset Done!\n")
if dataset == "restbench":
print("Processing RestBench dataset ... \n")
# Processing RestBench Dataset
os.system("wget -P {} -c {}".format(dataset_path, urls[dataset]))
print("RestBench dataset Done!\n")
if dataset == "toolbench":
print("Processing ToolBench dataset ... \n")
# Processing ToolBench Dataset
temp_file = os.path.join(dataset_path, "data.zip")
gdown.download(urls[dataset], temp_file, quiet=False)
zf = ZipFile(temp_file, 'r')
zf.extract("data/test_instruction/G2_category.json", ".")
zf.extract("data/test_instruction/G3_instruction.json", ".")
os.rename("data/test_instruction/G2_category.json", "{}/G2_category.json".format(dataset_path))
os.rename("data/test_instruction/G3_instruction.json", "{}/G3_instruction.json".format(dataset_path))
toolbench_process("{}/G2_category.json".format(dataset_path), "data_{}".format(dataset))
toolbench_process("{}/G3_instruction.json".format(dataset_path), "data_{}".format(dataset))
os.remove(temp_file)
shutil.rmtree("data")
print("Toolbench dataset Done!\n")
if __name__ == '__main__':
main()
================================================
FILE: easytool/data_restbench/tool_instruction/tmdb_tool.json
================================================
[
{
"ID": 1,
"tool_name": "/movie/{movie_id}/keywords",
"tool_description": "Get the keywords that have been added to a movie. You should first know the movie_id and thus this tool should be used after /search/movie.",
"tool_usage": "GET /movie/{movie_id}/keywords",
"Example": {
"Scenario": "if you want to find out the keywords that have been added to a movie with movie_id 456.",
"Parameters": {
"input": "GET /movie/456/keywords"
}
}
},
{
"ID": 2,
"tool_name": "/tv/popular",
"tool_description": "Get a list of the current popular TV shows on TMDb.",
"tool_usage": "GET /tv/popular",
"Example": {
"Scenario": "if you want to get a list of the current popular TV shows on TMDb.",
"Parameters": {
"input": "GET /tv/popular"
}
}
},
{
"ID": 3,
"tool_name": "/person/{person_id}",
"tool_description": "Get the primary person details by id. You should first know the person_id and thus this tool should be used after /search/person.",
"tool_usage": "GET /person/{person_id}",
"Example": {
"Scenario": "if you want to know the primary details of person with person_id 456.",
"Parameters": {
"input": "GET /person/456"
}
}
},
{
"ID": 4,
"tool_name": "/movie/{movie_id}/reviews",
"tool_description": "Get the user reviews for a movie. You should first know the movie_id and thus this tool should be used after /search/movie.",
"tool_usage": "GET /movie/{movie_id}/reviews",
"Example": {
"Scenario": "If you want to read the user reviews for a movie with movie_id 456.",
"Parameters": {
"input": "GET /movie/456/reviews"
}
}
},
{
"ID": 5,
"tool_name": "/movie/{movie_id}/release_dates",
"tool_description": "Get the release date along with the certification for a movie. You should first know the movie_id and thus this tool should be used after /search/movie.",
"tool_usage": "GET /movie/{movie_id}/release_dates",
"Example": {
"Scenario": "if you want to know the release dates and certification of the movie with movie_id 123.",
"Parameters": {
"input": "GET /movie/123/release_dates"
}
}
},
{
"ID": 6,
"tool_name": "/tv/{tv_id}/season/{season_number}/episode/{episode_number}/credits",
"tool_description": "Get the credits (cast, crew and guest stars) for a TV episode. You should first know the tv_id and thus this tool should be used after /search/tv.",
"tool_usage": "GET /tv/{tv_id}/season/{season_number}/episode/{episode_number}/credits",
"Example": {
"Scenario": "If you want to get the credits of a TV episode with tv_id 456, season_number 2, and episode_number 3.",
"Parameters": {
"input": "GET /tv/456/season/2/episode/3/credits"
}
}
},
{
"ID": 7,
"tool_name": "/movie/{movie_id}/images",
"tool_description": "Get the images that belong to a movie. You should first know the movie_id and thus this tool should be used after /search/movie.",
"tool_usage": "GET /movie/{movie_id}/images",
"Example": {
"Scenario": "if you want to get the images of a movie with movie_id 456.",
"Parameters": {
"input": "GET /movie/456/images"
}
}
},
{
"ID": 8,
"tool_name": "/search/tv",
"tool_description": "Search for a TV show, which can obtain tv_id.",
"tool_usage": "GET /search/tv",
"Example": {
"Scenario": "If you want to search for the id of a TV show with the name 'Friends'",
"Parameters": {
"input": "GET /search/tv?query=Friends"
}
}
},
{
"ID": 9,
"tool_name": "/tv/{tv_id}/season/{season_number}/episode/{episode_number}",
"tool_description": "Get the TV episode details by id.",
"tool_usage": "GET /tv/{tv_id}/season/{season_number}/episode/{episode_number}",
"Example": {
"Scenario": "If you want to get details of the 3rd episode from the 2nd season of the TV show with tv_id 456.",
"Parameters": {
"input": "GET /tv/456/season/2/episode/3"
}
}
},
{
"ID": 10,
"tool_name": "/network/{network_id}/images",
"tool_description": "Get the TV network logos by id.",
"tool_usage": "GET /network/{network_id}/images",
"Example": {
"Scenario": "if you want to retrieve the logos of a TV network with network_id 50.",
"Parameters": {
"input": "GET /network/50/images"
}
}
},
{
"ID": 11,
"tool_name": "/genre/tv/list",
"tool_description": "Get the list of official genres for TV shows.",
"tool_usage": "GET /genre/tv/list",
"Example": {
"Scenario": "if you want to get the list of official genres for TV shows.",
"Parameters": {
"input": "GET /genre/tv/list"
}
}
},
{
"ID": 12,
"tool_name": "/search/movie",
"tool_description": "Search for movies, which can obtain movie_id.",
"tool_usage": "GET /search/movie",
"Example": {
"Scenario": "If you want to search for the id of a movie with the name 'Avatar'",
"Parameters": {
"input": "GET /search/movie?query=Avatar"
}
}
},
{
"ID": 13,
"tool_name": "/discover/movie",
"tool_description": "Discover movies by different types of data like average rating, number of votes, genres and certifications",
"tool_usage": "GET /discover/movie",
"Example": {
"Scenario": "If you want to discover movies by different types of data like average rating, number of votes, genres and certifications.",
"Parameters": {
"input": "GET /discover/movie"
}
}
},
{
"ID": 14,
"tool_name": "/movie/upcoming",
"tool_description": "Get a list of upcoming movies in theatres.",
"tool_usage": "GET /movie/upcoming",
"Example": {
"Scenario": "if you want to get a list of upcoming movies in theatres.",
"Parameters": {
"input": "GET /movie/upcoming"
}
}
},
{
"ID": 15,
"tool_name": "/credit/{credit_id}",
"tool_description": "Get a movie or TV credit details by id.",
"tool_usage": "GET /credit/{credit_id}",
"Example": {
"Scenario": "if you want to get the details of movie or TV credit with credit_id as 456.",
"Parameters": {
"input": "GET /credit/456"
}
}
},
{
"ID": 16,
"tool_name": "/person/{person_id}/tv_credits",
"tool_description": "Get the TV show credits for a person. You should first know the person_id and thus this tool should be used after /search/person.",
"tool_usage": "GET /person/{person_id}/tv_credits",
"Example": {
"Scenario": "If you want to get the TV show credits of a person with person_id 456.",
"Parameters": {
"input": "GET /person/456/tv_credits"
}
}
},
{
"ID": 17,
"tool_name": "/tv/latest",
"tool_description": "Get the most newly created TV show.",
"tool_usage": "GET /tv/latest",
"Example": {
"Scenario": "If you want to get the most newly created TV show.",
"Parameters": {
"input": "GET /tv/latest"
}
}
},
{
"ID": 18,
"tool_name": "/company/{company_id}",
"tool_description": "Get a companies details by id. You should first know the company_id and thus this tool should be used after /search/company.",
"tool_usage": "GET /company/{company_id}",
"Example": {
"Scenario": "if you want to know the details of a company with company_id 456.",
"Parameters": {
"input": "GET /company/456"
}
}
},
{
"ID": 19,
"tool_name": "/tv/{tv_id}/images",
"tool_description": "Get the images that belong to a TV show. You should first know the tv_id and thus this tool should be used after /search/tv.",
"tool_usage": "GET /tv/{tv_id}/images",
"Example": {
"Scenario": "if you want to get the images of a TV show with tv_id 456.",
"Parameters": {
"input": "GET /tv/456/images"
}
}
},
{
"ID": 20,
"tool_name": "/tv/{tv_id}/season/{season_number}/credits",
"tool_description": "Get the credits for TV season. You should first know the tv_id and thus this tool should be used after /search/tv.",
"tool_usage": "GET /tv/{tv_id}/season/{season_number}/credits",
"Example": {
"Scenario": "if you want to get the credits for a TV season with tv_id 456 and season_number 2.",
"Parameters": {
"input": "GET /tv/456/season/2/credits"
}
}
},
{
"ID": 21,
"tool_name": "/movie/now_playing",
"tool_description": "Get a list of movies in theatres.",
"tool_usage": "GET /movie/now_playing",
"Example": {
"Scenario": "If you want to get a list of movies currently playing in theatres.",
"Parameters": {
"input": "GET /movie/now_playing"
}
}
},
{
"ID": 22,
"tool_name": "/review/{review_id}",
"tool_description": "Get review from a reviewer",
"tool_usage": "GET /review/{review_id}",
"Example": {
"Scenario": "If you want to get a review with review_id 456.",
"Parameters": {
"input": "GET /review/456"
}
}
},
{
"ID": 23,
"tool_name": "/tv/on_the_air",
"tool_description": "Get a list of shows that are currently on the air.",
"tool_usage": "GET /tv/on_the_air",
"Example": {
"Scenario": "If you want to know the list of shows currently on air.",
"Parameters": {
"input": "GET /tv/on_the_air"
}
}
},
{
"ID": 24,
"tool_name": "/movie/{movie_id}",
"tool_description": "Get the primary information about a movie. You should first know the movie_id and thus this tool should be used after /search/movie.",
"tool_usage": "GET /movie/{movie_id}",
"Example": {
"Scenario": "if you want to get the primary information about a movie with movie_id 456.",
"Parameters": {
"input": "GET /movie/456"
}
}
},
{
"ID": 25,
"tool_name": "/tv/{tv_id}/season/{season_number}/images",
"tool_description": "Get the images that belong to a TV season. You should first know the tv_id and thus this tool should be used after /search/tv.",
"tool_usage": "GET /tv/{tv_id}/season/{season_number}/images",
"Example": {
"Scenario": "if you want to get the images of a specific season of a TV show with tv_id 456 and season_number 2.",
"Parameters": {
"input": "GET /tv/456/season/2/images"
}
}
},
{
"ID": 26,
"tool_name": "/company/{company_id}/images",
"tool_description": "Get a companies logos by id . You should first know the company_id and thus this tool should be used after /search/company.",
"tool_usage": "GET /company/{company_id}/images",
"Example": {
"Scenario": "if you want to fetch the logos of the company with company_id 789.",
"Parameters": {
"input": "GET /company/789/images"
}
}
},
{
"ID": 27,
"tool_name": "/tv/{tv_id}/reviews",
"tool_description": "Get the reviews for a TV show. You should first know the tv_id and thus this tool should be used after /search/tv.",
"tool_usage": "GET /tv/{tv_id}/reviews",
"Example": {
"Scenario": "if you want to read the reviews of a TV show with tv_id 567.",
"Parameters": {
"input": "GET /tv/567/reviews"
}
}
},
{
"ID": 28,
"tool_name": "/tv/{tv_id}/similar",
"tool_description": "Get a list of similar TV shows. You should first know the tv_id and thus this tool should be used after /search/tv.",
"tool_usage": "GET /tv/{tv_id}/similar",
"Example": {
"Scenario": "If you want to find similar TV shows to the one with tv_id 456.",
"Parameters": {
"input": "GET /tv/456/similar"
}
}
},
{
"ID": 29,
"tool_name": "/network/{network_id}",
"tool_description": "Get the details of a network.",
"tool_usage": "GET /network/{network_id}",
"Example": {
"Scenario": "If you want to know the details of a network with network_id 456.",
"Parameters": {
"input": "GET /network/456"
}
}
},
{
"ID": 30,
"tool_name": "/tv/{tv_id}/recommendations",
"tool_description": "Get the list of TV show recommendations for this item. You should first know the tv_id and thus this tool should be used after /search/tv.",
"tool_usage": "GET /tv/{tv_id}/recommendations",
"Example": {
"Scenario": "if you want to get recommendations for a TV show with tv_id 456.",
"Parameters": {
"input": "GET /tv/456/recommendations"
}
}
},
{
"ID": 31,
"tool_name": "/tv/{tv_id}/season/{season_number}/episode/{episode_number}/images",
"tool_description": "Get the images that belong to a TV episode. You should first know the tv_id and thus this tool should be used after /search/tv.",
"tool_usage": "GET /tv/{tv_id}/season/{season_number}/episode/{episode_number}/images",
"Example": {
"Scenario": "If you want to know the images of a TV episode with tv_id 456, season_number 2 and episode_number 3.",
"Parameters": {
"input": "GET /tv/456/season/2/episode/3/images"
}
}
},
{
"ID": 32,
"tool_name": "/movie/popular",
"tool_description": "Get a list of the current popular movies on TMDb.",
"tool_usage": "GET /movie/popular",
"Example": {
"Scenario": "if you want to get a list of the current popular movies on TMDb.",
"Parameters": {
"input": "GET /movie/popular"
}
}
},
{
"ID": 33,
"tool_name": "/tv/airing_today",
"tool_description": "Get a list of TV shows that are airing today.",
"tool_usage": "GET /tv/airing_today",
"Example": {
"Scenario": "If you want to get a list of TV shows that are airing today.",
"Parameters": {
"input": "GET /tv/airing_today"
}
}
},
{
"ID": 34,
"tool_name": "/tv/{tv_id}/keywords",
"tool_description": "Get the keywords that have been added to a TV show. You should first know the tv_id and thus this tool should be used after /search/tv.",
"tool_usage": "GET /tv/{tv_id}/keywords",
"Example": {
"Scenario": "if you want to know the keywords of a TV show with tv_id 456.",
"Parameters": {
"input": "GET /tv/456/keywords"
}
}
},
{
"ID": 35,
"tool_name": "/search/person",
"tool_description": "Search for people, which can obtain person_id.",
"tool_usage": "GET /search/person",
"Example": {
"Scenario": "If you want to search for the id of a person with the name 'Leonardo DiCaprio'",
"Parameters": {
"input": "GET /search/person?query=Leonardo%20DiCaprio"
}
}
},
{
"ID": 36,
"tool_name": "/search/company",
"tool_description": "Search for companies, which can obtain company_id.",
"tool_usage": "GET /search/company",
"Example": {
"Scenario": "If you want to search for the id of a company with the name 'Disney'",
"Parameters": {
"input": "GET /search/company?query=Disney"
}
}
},
{
"ID": 37,
"tool_name": "/discover/tv",
"tool_description": "Discover TV shows by different types of data like average rating, number of votes, genres, the network they aired on and air dates.",
"tool_usage": "GET /discover/tv",
"Example": {
"Scenario": "If you want to discover TV shows by their average rating, number of votes, genres, the network they aired on and air dates.",
"Parameters": {
"input": "GET /discover/tv"
}
}
},
{
"ID": 38,
"tool_name": "/movie/top_rated",
"tool_description": "Get the top rated movies on TMDb.",
"tool_usage": "GET /movie/top_rated",
"Example": {
"Scenario": "If you want to get the top rated movies on TMDb.",
"Parameters": {
"input": "GET /movie/top_rated"
}
}
},
{
"ID": 39,
"tool_name": "/movie/latest",
"tool_description": "Get the most newly created movie.",
"tool_usage": "GET /movie/latest",
"Example": {
"Scenario": "if you want to know the details of the most recently created movie.",
"Parameters": {
"input": "GET /movie/latest"
}
}
},
{
"ID": 40,
"tool_name": "/tv/top_rated",
"tool_description": "Get a list of the top rated TV shows on TMDb.",
"tool_usage": "GET /tv/top_rated",
"Example": {
"Scenario": "If you want to get a list of top rated TV shows on TMDb.",
"Parameters": {
"input": "GET /tv/top_rated"
}
}
},
{
"ID": 41,
"tool_name": "/trending/{media_type}/{time_window}",
"tool_description": "Get the daily or weekly trending items.",
"tool_usage": "GET /trending/{media_type}/{time_window}",
"Example": {
"Scenario": "if you want to know the weekly trending movies.",
"Parameters": {
"input": "GET /trending/movie/week"
}
}
},
{
"ID": 42,
"tool_name": "/genre/movie/list",
"tool_description": "Get the list of official genres for movies.",
"tool_usage": "GET /genre/movie/list",
"Example": {
"Scenario": "If you want to get the list of official genres for movies.",
"Parameters": {
"input": "GET /genre/movie/list"
}
}
},
{
"ID": 43,
"tool_name": "/tv/{tv_id}/season/{season_number}",
"tool_description": "Get the TV season details by id. You should first know the tv_id and thus this tool should be used after /search/tv.",
"tool_usage": "GET /tv/{tv_id}/season/{season_number}",
"Example": {
"Scenario": "if you want to know the details of season 2 from the TV series with tv_id 456.",
"Parameters": {
"input": "GET /tv/456/season/2"
}
}
},
{
"ID": 44,
"tool_name": "/collection/{collection_id}",
"tool_description": "Get collection details by id. You should first know the collection_id and thus this tool should be used after /search/collection.",
"tool_usage": "GET /collection/{collection_id}",
"Example": {
"Scenario": "if you want to get details of a collection with collection_id 456.",
"Parameters": {
"input": "GET /collection/456"
}
}
},
{
"ID": 45,
"tool_name": "/person/{person_id}/images",
"tool_description": "Get the images for a person. You should first know the person_id and thus this tool should be used after /search/person.",
"tool_usage": "GET /person/{person_id}/images",
"Example": {
"Scenario": "if you want to get the images of a person with person_id 456.",
"Parameters": {
"input": "GET /person/456/images"
}
}
},
{
"ID": 46,
"tool_name": "/tv/{tv_id}",
"tool_description": "Get the primary TV show details by id. You should first know the tv_id and thus this tool should be used after /search/tv.",
"tool_usage": "GET /tv/{tv_id}",
"Example": {
"Scenario": "If you want to retrieve the primary details of a TV show with tv_id 789.",
"Parameters": {
"input": "GET /tv/789"
}
}
},
{
"ID": 47,
"tool_name": "/person/popular",
"tool_description": "Get the list of popular people on TMDb.",
"tool_usage": "GET /person/popular",
"Example": {
"Scenario": "If you want to fetch the list of popular people on TMDb.",
"Parameters": {
"input": "GET /person/popular"
}
}
},
{
"ID": 48,
"tool_name": "/collection/{collection_id}/images",
"tool_description": "Get the images for a collection by id. You should first know the collection_id and thus this tool should be used after /search/collection.",
"tool_usage": "GET /collection/{collection_id}/images",
"Example": {
"Scenario": "if you want to get the images of a collection with collection_id 456.",
"Parameters": {
"input": "GET /collection/456/images"
}
}
},
{
"ID": 49,
"tool_name": "/tv/{tv_id}/credits",
"tool_description": "Get the credits (cast and crew) that have been added to a TV show. You should first know the tv_id and thus this tool should be used after /search/tv.",
"tool_usage": "GET /tv/{tv_id}/credits",
"Example": {
"Scenario": "if you want to know the credits of a TV show with tv_id 456.",
"Parameters": {
"input": "GET /tv/456/credits"
}
}
},
{
"ID": 50,
"tool_name": "/person/{person_id}/movie_credits",
"tool_description": "Get the movie credits for a person, the results contains various information such as popularity and release date. You should first know the person_id and thus this tool should be used after /search/person.",
"tool_usage": "GET /person/{person_id}/movie_credits",
"Example": {
"Scenario": "if you want to get the movie credits for a person with person_id 456.",
"Parameters": {
"input": "GET /person/456/movie_credits"
}
}
},
{
"ID": 51,
"tool_name": "/movie/{movie_id}/recommendations",
"tool_description": "Get a list of recommended movies for a movie. You should first know the movie_id and thus this tool should be used after /search/movie.",
"tool_usage": "GET /movie/{movie_id}/recommendations",
"Example": {
"Scenario": "if you want to get a list of recommended movies for the movie with movie_id 456.",
"Parameters": {
"input": "GET /movie/456/recommendations"
}
}
},
{
"ID": 52,
"tool_name": "/search/collection",
"tool_description": "Search for collections, which can obtain collection_id.",
"tool_usage": "GET /search/collection",
"Example": {
"Scenario": "if you want to search for a collection id of Star Wars.",
"Parameters": {
"input": "GET /search/collection?query=Star%20Wars"
}
}
},
{
"ID": 53,
"tool_name": "/movie/{movie_id}/credits",
"tool_description": "Get the cast and crew for a movie. You should first know the movie_id and thus this tool should be used after /search/movie.",
"tool_usage": "GET /movie/{movie_id}/credits",
"Example": {
"Scenario": "if you want to know the cast and crew of a movie with movie_id 789.",
"Parameters": {
"input": "GET /movie/789/credits"
}
}
},
{
"ID": 54,
"tool_name": "/movie/{movie_id}/similar",
"tool_description": "Get a list of similar movies. You should first know the movie_id and thus this tool should be used after /search/movie.",
"tool_usage": "GET /movie/{movie_id}/similar",
"Example": {
"Scenario": "if you want to find movies similar to the movie with movie_id 456.",
"Parameters": {
"input": "GET /movie/456/similar"
}
}
}
]
================================================
FILE: easytool/data_toolbench/tool_instruction/toolbench_tool_instruction.json
================================================
[File too large to display: 10.9 MB]
================================================
FILE: easytool/easytool/__init__.py
================================================
# __init__.py
================================================
FILE: easytool/easytool/funcQA.py
================================================
# — coding: utf-8 –
import openai
import json
import logging
import sys
import argparse
from langchain.chat_models import ChatOpenAI
from langchain.prompts import (
ChatPromptTemplate,
MessagesPlaceholder,
SystemMessagePromptTemplate,
HumanMessagePromptTemplate
)
from langchain import LLMChain
import numpy as np
import requests
import os
import subprocess
import re
import importlib.util
from sklearn.metrics.pairwise import cosine_similarity
import pickle
from util import *
from tqdm import tqdm
openai.api_key = os.environ["OPENAI_API_KEY"]
def get_last_processed_index(progress_file):
"""Retrieve the last processed index from the progress file."""
if os.path.exists(progress_file):
with open(progress_file, 'r', encoding='utf-8') as f:
last_index = f.read().strip()
return int(last_index) if last_index else 0
else:
return 0
def update_progress(progress_file, index):
"""Update the last processed index in the progress file."""
with open(progress_file, 'w', encoding='utf-8') as f:
f.write(str(index))
def choose_tool(question, Tool_dic, tool_used, model_name):
chat = ChatOpenAI(model_name=model_name)
template = "You are a helpful assistant."
system_message_prompt = SystemMessagePromptTemplate.from_template(template)
human_message_prompt = HumanMessagePromptTemplate.from_template(
"This is the user's question: {question}\n"
"These are the tools you can select to solve the question:\n"
"Tool List:\n"
"{Too_list}\n\n"
"Please note that: \n"
"1. You should only chooce one tool the Tool List to solve this question.\n"
"2. You must ONLY output the ID of the tool you chose in a parsible JSON format. Two example outputs look like:\n"
"'''\n"
"Example 1: {{\"ID\": 1}}\n"
"Example 2: {{\"ID\": 2}}\n"
"'''\n"
"Output:"
)
chat_prompt = ChatPromptTemplate.from_messages([system_message_prompt, human_message_prompt])
chain = LLMChain(llm=chat, prompt=chat_prompt)
ind = 0
Tool_list = []
for ele in Tool_dic:
for key in ele.keys():
if str(key) not in tool_used:
Tool_list.append(f'''ID: {key}\n{ele[key]}''')
while True:
try:
result = chain.run(question=question,
Too_list=Tool_dic)
clean_answer = eval(result.split("(")[0].strip())
# clean_answer = lowercase_parameter_keys(clean_answer)
# print(clean_answer)
break
except Exception as e:
print(f"choose tool fails: {e}")
print(result)
if ind > 10:
return -1
ind += 1
continue
return clean_answer
def task_decompose(question, Tool_dic, model_name):
chat = ChatOpenAI(model_name=model_name)
template = "You are a helpful assistant."
system_message_prompt = SystemMessagePromptTemplate.from_template(template)
human_message_prompt = HumanMessagePromptTemplate.from_template(
"You need to decompose a complex user's question into some simple subtasks and let the model execute it step by step.\n"
"This is the user's question: {question}\n"
"This is tool list:\n"
"{Tool_list}\n"
"Please note that: \n"
"1. You should only decompose this complex user's question into some simple subtasks which can be executed easily by using one single tool in the tool list.\n"
"2. If one subtask need the results from other subtask, you can should write clearly. For example:"
"{{\"Tasks\": [\"Convert 23 km/h to X km/min by 'divide_'\", \"Multiply X km/min by 45 min to get Y by 'multiply_'\"]}}\n"
"3. You must ONLY output in a parsible JSON format. An example output looks like:\n"
"'''\n"
"{{\"Tasks\": [\"Task 1\", \"Task 2\", ...]}}\n"
"'''\n"
"Output:"
)
chat_prompt = ChatPromptTemplate.from_messages([system_message_prompt, human_message_prompt])
chain = LLMChain(llm=chat, prompt=chat_prompt)
Tool_list = []
for ele in Tool_dic:
Tool_list.append(str(ele))
ind = 0
while True:
try:
result = chain.run(question=question, Tool_list=Tool_list)
result = eval(result.split('\n\n')[0])
a = result["Tasks"]
break
except Exception as e:
print(f"task decompose fails: {e}")
if ind > 10:
return -1
ind += 1
continue
return result
def task_topology(question, task_ls, model_name):
chat = ChatOpenAI(model_name=model_name)
template = "You are a helpful assistant."
system_message_prompt = SystemMessagePromptTemplate.from_template(template)
human_message_prompt = HumanMessagePromptTemplate.from_template(
"Given a complex user's question, I have decompose this question into some simple subtasks"
"I think there exists a logical connections and order amontg the tasks. "
"Thus you need to help me output this logical connections and order.\n"
"You must ONLY output in a parsible JSON format with the following format:\n"
"'''\n"
"[{{\"task\": task, \"id\", task_id, \"dep\": [dependency_task_id1, dependency_task_id2, ...]}}]\n"
"'''\n"
"The \"dep\" field denotes the id of the previous task which generates a new resource upon which the current task depends. If there are no dependencies, set \"dep\" to -1.\n\n"
"This is user's question: {question}\n"
"These are subtasks of this question:\n"
"{task_ls}\n"
"Output: "
)
chat_prompt = ChatPromptTemplate.from_messages([system_message_prompt, human_message_prompt])
chain = LLMChain(llm=chat, prompt=chat_prompt)
ind = 0
while True:
try:
result = chain.run(question=question, task_ls=task_ls)
result = eval(result)
for i in range(len(result)):
if isinstance(result[i]['dep'], str):
temp = []
for ele in result[i]['dep'].split(','):
temp.append(int(ele))
result[i]['dep'] = temp
elif isinstance(result[i]['dep'], int):
result[i]['dep'] = [result[i]['dep']]
elif isinstance(result[i]['dep'], list):
temp = []
for ele in result[i]['dep']:
temp.append(int(ele))
result[i]['dep'] = temp
elif result[i]['dep'] == -1:
result[i]['dep'] = [-1]
a = result[i]['dep'][0]
return result
except Exception as e:
print(f"task topology fails: {e}")
if ind > 10:
return -1
ind += 1
continue
return result
def answer_generation_direct(task, model_name):
chat = ChatOpenAI(model_name=model_name)
template = "You are a helpful assistant."
system_message_prompt = SystemMessagePromptTemplate.from_template(template)
human_message_prompt = HumanMessagePromptTemplate.from_template(
"You need to answer the user's question.\n"
"This is the user's question: {task}\n"
"Output:"
)
chat_prompt = ChatPromptTemplate.from_messages([system_message_prompt, human_message_prompt])
chain = LLMChain(llm=chat, prompt=chat_prompt)
result = chain.run(task=task)
return result
def choose_parameter(API_instruction, api, api_dic, question, model_name):
chat = ChatOpenAI(model_name=model_name)
template = "You are a helpful assistant."
system_message_prompt = SystemMessagePromptTemplate.from_template(template)
human_message_prompt = HumanMessagePromptTemplate.from_template(
"This is an API tool documentation. Given a user's question, you need to output parameters according to the API tool documentation to successfully call the API to solve the user's question.\n"
"This is API tool documentation: {api_dic}\n"
"Please note that: \n"
"1. The Example in the API tool documentation can help you better understand the use of the API.\n"
"2. Ensure the parameters you output are correct. The output must contain the required parameters, and can contain the optional parameters based on the question. If no paremters in the required parameters and optional parameters, just leave it as {{\"Parameters\":{{}}}}\n"
"3. If the user's question mentions other APIs, you should ONLY consider the API tool documentation I give and do not consider other APIs.\n"
"4. If you need to use this API multiple times, please set \"Parameters\" to a list.\n"
"5. You must ONLY output in a parsible JSON format. Two examples output looks like:\n"
"'''\n"
"Example 1: {{\"Parameters\":{{\"input\": [1,2,3]}}}}\n"
"Example 2: {{\"Parameters\":[{{\"input\": [1,2,3]}}, {{\"input\": [2,3,4]}}]}}\n"
"'''\n"
"This is user's question: {question}\n"
"Output:\n"
)
chat_prompt = ChatPromptTemplate.from_messages([system_message_prompt, human_message_prompt])
chain = LLMChain(llm=chat, prompt=chat_prompt)
ind = 0
while True:
try:
result = chain.run(api_dic=api_dic,
question=question, )
clean_answer = eval(
result.replace(": true", ": True").replace(":true", ": True").replace(":false", ": False").replace(
": false", ": False").replace("```", "").strip())
a = clean_answer["Parameters"]
return a
except Exception as e:
print(f"Choose Parameter fails: {e}")
if ind > 10:
return -1
ind += 1
continue
return a
def choose_parameter_depend(API_instruction, api, api_dic, question, model_name, previous_log):
chat = ChatOpenAI(model_name=model_name)
template = "You are a helpful assistant."
system_message_prompt = SystemMessagePromptTemplate.from_template(template)
human_message_prompt = HumanMessagePromptTemplate.from_template(
"Given a user's question and a API tool documentation, you need to output parameters according to the API tool documentation to successfully call the API to solve the user's question.\n"
"Please note that: \n"
"1. The Example in the API tool documentation can help you better understand the use of the API.\n"
"2. Ensure the parameters you output are correct. The output must contain the required parameters, and can contain the optional parameters based on the question. If no paremters in the required parameters and optional parameters, just leave it as {{\"Parameters\":{{}}}}\n"
"3. If the user's question mentions other APIs, you should ONLY consider the API tool documentation I give and do not consider other APIs.\n"
"4. The question may have dependencies on answers of other questions, so we will provide logs of previous questions and answers for your reference.\n"
"5. If you need to use this API multiple times,, please set \"Parameters\" to a list.\n"
"6. You must ONLY output in a parsible JSON format. Two examples output looks like:\n"
"'''\n"
"Example 1: {{\"Parameters\":{{\"input\": [1,2,3]}}}}\n"
"Example 2: {{\"Parameters\":[{{\"input\": [1,2,3]}}, {{\"input\": [2,3,4]}}]}}\n"
"'''\n"
"There are logs of previous questions and answers: \n {previous_log}\n"
"This is the current user's question: {question}\n"
"This is API tool documentation: {api_dic}\n"
"Output:\n"
)
chat_prompt = ChatPromptTemplate.from_messages([system_message_prompt, human_message_prompt])
chain = LLMChain(llm=chat, prompt=chat_prompt)
ind = 0
while True:
try:
result = chain.run(api_dic=api_dic,
question=question,
previous_log=previous_log)
clean_answer = eval(
result.replace(": true", ": True").replace(": false", ": False").replace("```", "").strip())
a = clean_answer["Parameters"]
return a
except Exception as e:
print(f"choose parameter depend fails: {e}")
if ind > 10:
return -1
ind += 1
continue
return a
def Call_function(B, arg, id):
app_path = 'data_funcqa/funchub/math.py'
spec = importlib.util.spec_from_file_location('math', app_path)
app_module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(app_module)
if hasattr(app_module, B):
function_B = getattr(app_module, B)
try:
call_result = function_B(arg['input'])
return call_result
except Exception as e:
try:
arg = {change_name(k.lower()): v for k, v in arg.items()}
call_result = function_B(arg['input'])
return call_result
except Exception as e:
try:
arg = {change_name(k.lower()): v for k, v in arg.items()}
arg = {change_name(k.replace("-", "_")): v for k, v in arg.items()}
call_result = function_B(arg['input'])
return call_result
except Exception as e:
print(f"fails: {e}")
with open('wrong_log.json', 'a+', encoding='utf-8') as f:
line = json.dumps({
"id": id,
"parameters": arg,
"wrong": str(e)
}, ensure_ascii=False)
f.write(line + '\n')
return -1
else:
with open('wrong_log.json', 'a+', encoding='utf-8') as f:
line = json.dumps({
"id": id,
"parameters": arg,
"wrong": f"No function named {B} in {app_path}"
}, ensure_ascii=False)
f.write(line + '\n')
return (f"No function named {B} in {app_path}")
def retrieval(question, Tool_dic, dataset, tool_used, ind, model_name, previous_log=None):
tool_id = choose_tool(question, Tool_dic, tool_used, model_name)
if tool_id == -1:
return tool_id, "", "", "", ""
tool_instruction = dataset[str(tool_id["ID"])]
API_instruction = tool_instruction["API_description"]
API_tool = tool_instruction["standardized_name"]
api_selection = [API_tool]
api_result = []
for api in api_selection:
if previous_log is None:
parameter = choose_parameter(API_instruction, api,
tool_instruction["Usage"], question, model_name)
else:
parameter = choose_parameter_depend(API_instruction, api,
tool_instruction["Usage"],
question, model_name, previous_log)
if parameter == -1:
continue
api_result.append({"api_name": api, "parameters": parameter})
if len(api_result) == 0:
call_result = ""
return tool_id, api_result, call_result, tool_instruction, API_instruction
if isinstance(api_result, set) or isinstance(api_result, list):
call_results = []
for api in api_result:
if isinstance(api["parameters"], dict):
parameters = {}
for key in api["parameters"]:
value = api["parameters"][key]
key = change_name(key)
parameters[key] = value
call_result = Call_function(API_tool, parameters, ind)
if call_result == -1:
continue
call_results.append(str(call_result))
elif isinstance(api["parameters"], list):
for para_ls in api["parameters"]:
parameters = {}
for key in para_ls:
value = para_ls[key]
key = change_name(key)
parameters[key] = value
call_result = Call_function(API_tool, parameters, ind)
if call_result == -1:
continue
call_results.append(str(call_result))
call_result = '\n\n'.join(call_results)
elif isinstance(api_result, dict):
api = api_result
if isinstance(api["parameters"], dict):
parameters = {}
for key in api["parameters"]:
value = api["parameters"][key]
key = change_name(key)
parameters[key] = value
call_result = Call_function(API_tool, parameters, ind)
elif isinstance(api["parameters"], list):
call_results = []
for para_ls in api["parameters"]:
parameters = {}
for key in para_ls:
value = para_ls[key]
key = change_name(key)
parameters[key] = value
call_result = Call_function(API_tool, parameters, ind)
if call_result == -1:
continue
call_results.append(str(call_result))
call_result = '\n\n'.join(call_results)
return tool_id, api_result, call_result, tool_instruction, API_instruction
def answer_generation(question, API_instruction, call_result, model_name):
chat = ChatOpenAI(model_name=model_name)
template = "You are a helpful assistant."
system_message_prompt = SystemMessagePromptTemplate.from_template(template)
human_message_prompt = HumanMessagePromptTemplate.from_template(
"You should answer the question based on the response output by the API tool."
"Please note that:\n"
"1. Answer the question in natural language based on the API response reasonably and effectively.\n"
"2. The user cannot directly get API response, "
"so you need to make full use of the response and give the information "
"in the response that can satisfy the user's question in as much detail as possible.\n"
"3. If the API tool does not provide useful information in the response, "
"please answer with your knowledge.\n"
"This is the user's question:\n {question}\n"
"This is the API response:\n {call_result}\n"
"Output:"
)
chat_prompt = ChatPromptTemplate.from_messages([system_message_prompt, human_message_prompt])
chain = LLMChain(llm=chat, prompt=chat_prompt)
ind = 0
while True:
try:
result = chain.run(question=question,
API_instruction=API_instruction,
call_result=call_result, )
break
except Exception as e:
print(f"answer generation fails: {e}")
if ind > 2:
return -1
ind += 1
continue
return result
def answer_generation_depend(question, API_instruction, call_result, previous_log, model_name):
chat = ChatOpenAI(model_name=model_name)
template = "You are a helpful assistant."
system_message_prompt = SystemMessagePromptTemplate.from_template(template)
human_message_prompt = HumanMessagePromptTemplate.from_template(
"You should answer the question based on the response output by the API tool."
"Please note that:\n"
"1. Try to organize the response into a natural language answer.\n"
"2. We will not show the API response to the user, "
"thus you need to make full use of the response and give the information "
"in the response that can satisfy the user's question in as much detail as possible.\n"
"3. If the API tool does not provide useful information in the response, "
"please answer with your knowledge.\n"
"4. The question may have dependencies on answers of other questions, so we will provide logs of previous questions and answers.\n"
"There are logs of previous questions and answers: \n {previous_log}\n"
"This is the user's question: {question}\n"
"This is the response output by the API tool: \n{call_result}\n"
"We will not show the API response to the user, "
"thus you need to make full use of the response and give the information "
"in the response that can satisfy the user's question in as much detail as possible.\n"
"Output:"
)
chat_prompt = ChatPromptTemplate.from_messages([system_message_prompt, human_message_prompt])
chain = LLMChain(llm=chat, prompt=chat_prompt)
ind = 0
while True:
try:
result = chain.run(question=question,
API_instruction=API_instruction,
call_result=call_result,
previous_log=previous_log)
break
except Exception as e:
print(f"answer generation depend fails: {e}")
if ind > 2:
return -1
ind += 1
continue
return result
def answer_summarize(question, answer_task, model_name):
chat = ChatOpenAI(model_name=model_name)
template = "You are a helpful assistant."
system_message_prompt = SystemMessagePromptTemplate.from_template(template)
human_message_prompt = HumanMessagePromptTemplate.from_template(
"We break down a complex user's problems into simple subtasks and provide answers to each simple subtask. "
"You need to organize these answers to each subtask and form a self-consistent final answer to the user's question\n"
"This is the user's question: {question}\n"
"These are subtasks and their answers: {answer_task}\n"
"Final answer:"
)
chat_prompt = ChatPromptTemplate.from_messages([system_message_prompt, human_message_prompt])
chain = LLMChain(llm=chat, prompt=chat_prompt)
result = chain.run(question=question, answer_task=answer_task)
return result
def answer_check(question, answer, model_name):
chat = ChatOpenAI(model_name=model_name)
template = "You are a helpful assistant."
system_message_prompt = SystemMessagePromptTemplate.from_template(template)
human_message_prompt = HumanMessagePromptTemplate.from_template(
"Please check whether the response can reasonably and accurately answer the question."
"If can, please output 'YES'; If not, please output 'NO'\n"
"You need to give reasons first and then decide whether the response can reasonably and accurately answer the question. You must only output in a parsible JSON format. Two example outputs look like:\n"
"Example 1: {{\"Reason\": \"The reason why you think the response can reasonably and accurately answer the question\", \"Choice\": \"Yes\"}}\n"
"Example 2: {{\"Reason\": \"The reason why you think the response cannot reasonably and accurately answer the question\", \"Choice\": \"No\"}}\n"
"This is the user's question: {question}\n"
"This is the response: {answer}\n"
"Output: "
)
chat_prompt = ChatPromptTemplate.from_messages([system_message_prompt, human_message_prompt])
chain = LLMChain(llm=chat, prompt=chat_prompt)
result = chain.run(question=question, answer=answer)
if 'yes'.lower() in eval(result)["Choice"].lower():
return 1
else:
return -1
def task_execution_mh(data_type, start_index, total_files,
retrieval_num, ind, model_name, dataset,
Tool_dic, test_data, progress_file):
with tqdm(total=total_files, desc="Processing files", initial=start_index) as pbar:
for i, data in enumerate(test_data[start_index:], start=start_index):
answer_ls = []
question = data["question"]
print(question)
temp = task_decompose(question, Tool_dic, model_name)['Tasks']
task_ls = []
for t in range(len(temp)):
task_ls.append({"task": temp[t], "id": t + 1})
task_ls = task_topology(question, task_ls, model_name)
task_depend = {'Original Question': question}
for task_dic in task_ls:
task_depend[task_dic['id']] = {'task': task_dic['task'], 'answer': ''}
answer_task = []
tool_instruction_ls = []
api_result_ls = []
call_result_ls = []
tool_check_reason_ls = []
for task_dic in task_ls:
task = task_dic['task']
print("Do need tool.")
tool_used = []
depend_id = [1]
for r in range(retrieval_num):
if depend_id[0] == -1:
tool_id, api_result, call_result, tool_instruction, API_instruction = retrieval(task, Tool_dic,
dataset,
tool_used, ind,
model_name)
if len(str(call_result)) > 5000:
call_result = str(call_result)[:5000]
answer = answer_generation(task, API_instruction, call_result, model_name)
else:
previous_log = task_depend
tool_id, api_result, call_result, tool_instruction, API_instruction = retrieval(task, Tool_dic,
dataset,
tool_used, ind,
model_name,
previous_log=previous_log)
if len(str(call_result)) > 5000:
call_result = str(call_result)[:5000]
answer = answer_generation_depend(task, API_instruction, call_result, previous_log, model_name)
check_index = 1
if str(call_result).strip() == '-1' or str(call_result).strip() == '':
check_index = -1
if check_index == 1:
answer_task.append({'task': task, 'answer': answer})
tool_instruction_ls.append(tool_instruction)
api_result_ls.append(api_result)
call_result_ls.append(call_result)
break
else:
answer_ls.append({'task': task, 'answer': answer})
try:
tool_used.append(str(tool_id["ID"]))
except:
continue
print('****Try Again****')
task_depend[task_dic['id']]['answer'] = answer
final_answer = answer_summarize(question, answer_task, model_name)
check_index = answer_check(question, final_answer, model_name)
ind = ind + 1
with open(f"FuncQA_{data_type}_{model_name}_easytool.jsonl", 'a+', encoding='utf-8') as f:
line = json.dumps({
"ID": ind,
"question": question,
"final_answer": final_answer,
"subtask": task_ls,
"answer_subtask": answer_task,
"answer_wrong": answer_ls,
"check_index": check_index,
"execute_log": {
"api_result_ls": api_result_ls,
"call_result_ls": call_result_ls,
"tool_check_reason_ls": tool_check_reason_ls,
"tool_instruction_ls": tool_instruction_ls,
},
"check": 0
}, ensure_ascii=False)
f.write(line + '\n')
print(final_answer)
update_progress(progress_file, i + 1)
pbar.update(1)
def task_execution_oh(data_type, start_index, total_files,
retrieval_num, ind, model_name, dataset,
Tool_dic, test_data, progress_file):
with tqdm(total=total_files, desc="Processing files", initial=start_index) as pbar:
for i, data in enumerate(test_data[start_index:], start=start_index):
answer_ls = []
question = data["question"]
print(question)
task_ls = [{"task": question}]
answer_task = []
tool_instruction_ls = []
api_result_ls = []
call_result_ls = []
tool_check_reason_ls = []
for task_dic in task_ls:
task = task_dic['task']
print("Do need tool.")
tool_used = []
depend_id = [1]
for r in range(retrieval_num):
tool_id, api_result, call_result, tool_instruction, API_instruction = retrieval(task, Tool_dic,
dataset,
tool_used, ind,
model_name)
if len(str(call_result)) > 5000:
call_result = str(call_result)[:5000]
answer = answer_generation(task, API_instruction, call_result, model_name)
check_index = 1
if str(call_result).strip() == '-1' or str(call_result).strip() == '':
check_index = -1
if check_index == 1:
answer_task.append({'task': task, 'answer': answer})
tool_instruction_ls.append(tool_instruction)
api_result_ls.append(api_result)
call_result_ls.append(call_result)
break
else:
answer_ls.append({'task': task, 'answer': answer})
try:
tool_used.append(str(tool_id["ID"]))
except:
continue
print('****Try Again****')
final_answer = answer_summarize(question, answer_task, model_name)
check_index = answer_check(question, final_answer, model_name)
ind = ind + 1
with open(f"FuncQA_{data_type}_{model_name}_easytool.jsonl", 'a+', encoding='utf-8') as f:
line = json.dumps({
"ID": ind,
"question": question,
"final_answer": final_answer,
"subtask": task_ls,
"answer_subtask": answer_task,
"answer_wrong": answer_ls,
"check_index": check_index,
"execute_log": {
"api_result_ls": api_result_ls,
"call_result_ls": call_result_ls,
"tool_check_reason_ls": tool_check_reason_ls,
"tool_instruction_ls": tool_instruction_ls,
},
"check": 0
}, ensure_ascii=False)
f.write(line + '\n')
print(final_answer)
update_progress(progress_file, i + 1)
pbar.update(1)
================================================
FILE: easytool/easytool/restbench.py
================================================
# — coding: utf-8 –
import openai
import json
import logging
import sys
import argparse
import ast
from langchain.chat_models import ChatOpenAI
from langchain.prompts import (
ChatPromptTemplate,
MessagesPlaceholder,
SystemMessagePromptTemplate,
HumanMessagePromptTemplate
)
from langchain import LLMChain
import numpy as np
import requests
import os
import subprocess
import re
import importlib.util
from sklearn.metrics.pairwise import cosine_similarity
import pickle
from util import *
from tqdm import tqdm
openai.api_key = os.environ["OPENAI_API_KEY"]
def get_last_processed_index(progress_file):
"""Retrieve the last processed index from the progress file."""
if os.path.exists(progress_file):
with open(progress_file, 'r', encoding='utf-8') as f:
last_index = f.read().strip()
return int(last_index) if last_index else 0
else:
return 0
def update_progress(progress_file, index):
"""Update the last processed index in the progress file."""
with open(progress_file, 'w', encoding='utf-8') as f:
f.write(str(index))
def task_decompose(question, Tool_dic, model_name):
chat = ChatOpenAI(model_name=model_name)
template = "You are a helpful assistant."
system_message_prompt = SystemMessagePromptTemplate.from_template(template)
human_message_prompt = HumanMessagePromptTemplate.from_template(
"We have spotify database and the following tools:\n"
"{Tool_dic}"
"You need to decompose a complex user's question into some simple subtasks and let the model execute it step by step with these tools.\n"
"Please note that: \n"
"1. you should break down tasks into appropriate subtasks to use the tools mentioned above.\n"
"2. You should not only list the subtask, but also list the ID of the tool used to solve this subtask.\n"
"3. If you think you do not need to use the tool to solve the subtask, just leave it as {{\"ID\": -1}}\n"
"4. You must consider the logical connections, order and constraints among the tools to achieve a correct tool path."
"5. You must ONLY output the ID of the tool you chose in a parsible JSON format. Two examples output look like:\n"
"'''\n"
"Question: Pause the player"
"Example 1: [{{\"Task\":\"Get information about the user’s current playback state\", \"ID\":15}}, {{\"Task\":\"Pause playback on the user's account\", \"ID\":19}}]\n"
"'''\n"
"This is the user's question: {question}\n"
"Output:"
)
chat_prompt = ChatPromptTemplate.from_messages([system_message_prompt, human_message_prompt])
chain = LLMChain(llm=chat, prompt=chat_prompt)
ind = 0
while True:
try:
result = chain.run(question=question, Tool_dic=Tool_dic)
result = ast.literal_eval(result.split('\n\n')[0])
break
except Exception as e:
print(f"task decompose fails: {e}")
if ind > 10:
return -1
ind += 1
continue
return result
def task_execution(
Tool_dic, dic_tool, test_data, progress_file,
start_index, total_files, retrieval_num, ind, model_name):
with tqdm(total=total_files, desc="Processing files", initial=start_index) as pbar:
for i, data in enumerate(test_data[start_index:], start=start_index):
question = data["query"]
print(question)
task_path = task_decompose(question, Tool_dic, model_name)
tool_choice_ls = []
for task in task_path:
if isinstance(task["ID"], list):
for ele in task["ID"]:
tool_choice_ls.append(dic_tool[ele]['tool_usage'])
elif int(task["ID"]) in dic_tool.keys():
tool_choice_ls.append(dic_tool[task["ID"]]['tool_usage'])
ind = ind + 1
with open(f"restbench_{model_name}_Easytool.jsonl", 'a+', encoding='utf-8') as f:
line = json.dumps({
"ID": ind,
"question": question,
"task_path": task_path,
"tool_choice_ls": tool_choice_ls
}, ensure_ascii=False)
f.write(line + '\n')
print(tool_choice_ls)
update_progress(progress_file, i + 1)
pbar.update(1)
================================================
FILE: easytool/easytool/toolbench.py
================================================
# — coding: utf-8 –
import openai
import json
import logging
import sys
import argparse
from langchain.chat_models import ChatOpenAI
from langchain.prompts import (
ChatPromptTemplate,
MessagesPlaceholder,
SystemMessagePromptTemplate,
HumanMessagePromptTemplate
)
from langchain import LLMChain
import numpy as np
import requests
import os
import subprocess
import re
import importlib.util
from sklearn.metrics.pairwise import cosine_similarity
import pickle
from util import *
from tqdm import tqdm
openai.api_key = os.environ["OPENAI_API_KEY"]
def get_last_processed_index(progress_file):
"""Retrieve the last processed index from the progress file."""
if os.path.exists(progress_file):
with open(progress_file, 'r', encoding='utf-8') as f:
last_index = f.read().strip()
return int(last_index) if last_index else 0
else:
return 0
def update_progress(progress_file, index):
"""Update the last processed index in the progress file."""
with open(progress_file, 'w', encoding='utf-8') as f:
f.write(str(index))
def choose_tool(question, Tool_dic, tool_used, model_name):
chat = ChatOpenAI(model_name=model_name)
template = "You are a helpful assistant."
system_message_prompt = SystemMessagePromptTemplate.from_template(template)
human_message_prompt = HumanMessagePromptTemplate.from_template(
"This is the user's question: {question}\n"
"These are the tools you can select to solve the question:\n"
"Tool List:\n"
"{Too_list}\n\n"
"Please note that: \n"
"1. You should only chooce one tool the Tool List to solve this question.\n"
"2. You must ONLY output the ID of the tool you chose in a parsible JSON format. An example output looks like:\n"
"'''\n"
"Example: {{\"ID\": XX}}\n"
"'''\n"
"Output:"
)
chat_prompt = ChatPromptTemplate.from_messages([system_message_prompt, human_message_prompt])
chain = LLMChain(llm=chat, prompt=chat_prompt)
ind = 0
Tool_list = []
for ele in Tool_dic:
for key in ele.keys():
if str(key) not in tool_used:
Tool_list.append(f'''ID: {key}\n{ele[key]}''')
while True:
try:
result = chain.run(question=question,
Too_list='\n'.join(Tool_list))
clean_answer = eval(result.split("\n\n")[-1].strip())
break
except Exception as e:
print(f"choose tool fails: {e}")
if ind > 10:
return -1
ind += 1
continue
return clean_answer
def choose_API(API_instruction, API_list, question, model_name):
input_execute_rapidapi_api_note = '''
This is an API Tool instruction. Given a question, you should choose APIs from the API list you want to use for this question in this instruction.
you must only output in a parsible Python List Format. An example output looks like:
```
["api1", "api2", ...]
```
'''.strip()
chat = ChatOpenAI(model_name=model_name)
template = "You are a helpful assistant."
system_message_prompt = SystemMessagePromptTemplate.from_template(template)
human_message_prompt = HumanMessagePromptTemplate.from_template(
"{API_instruction}\n"
"{input_execute_rapidapi_api_note}\n"
"This is the API list: {API_list}\n"
"Please note that: \n"
"1. The APIs you choose must in the API list.\n"
"2. You must ONLY output in the following parsible Python List Format.\n"
"```\n"
"Output_Example: [\"api1\", \"api2\", ...]\n"
"```\n"
"Question: {question}\n"
"Output:"
)
chat_prompt = ChatPromptTemplate.from_messages([system_message_prompt, human_message_prompt])
chain = LLMChain(llm=chat, prompt=chat_prompt)
ind = 0
while True:
try:
result = chain.run(API_instruction=API_instruction,
API_list=API_list,
question=question,
input_execute_rapidapi_api_note=input_execute_rapidapi_api_note)
clean_answer = eval(result.replace("```", "").strip().split("\n\n")[0].strip())
if isinstance(clean_answer, str):
ls = [clean_answer]
elif isinstance(clean_answer, list):
ls = clean_answer
temp = []
for ele in ls:
if ele in API_list:
temp.append(ele)
ls = temp
return ls
except Exception as e:
print(f"Choose API fails: {e}")
print(result)
if ind > 10:
return []
ind += 1
continue
return ls
def choose_parameter(API_instruction, api, api_dic, question, model_name):
chat = ChatOpenAI(model_name=model_name)
template = "You are a helpful assistant."
system_message_prompt = SystemMessagePromptTemplate.from_template(template)
human_message_prompt = HumanMessagePromptTemplate.from_template(
"This is an API tool documentation. Given a user's question, you need to output parameters according to the API tool documentation to successfully call the API to solve the user's question.\n"
"This is API tool documentation: {api_dic}\n"
"Please note that: \n"
"1. The Example in the API tool documentation can help you better understand the use of the API.\n"
"2. Ensure the parameters you output are correct. The output must contain the required parameters, and can contain the optional parameters based on the question. If no paremters in the required parameters and optional parameters, just leave it as {{\"Parameters\":{{}}}}\n"
"3. If the user's question mentions other APIs, you should ONLY consider the API tool documentation I give and do not consider other APIs.\n"
"4. If you need to use this API multiple times, please set \"Parameters\" to a list.\n"
"5. You must ONLY output in a parsible JSON format. Two examples output looks like:\n"
"'''\n"
"Example 1: {{\"Parameters\":{{\"keyword\": \"Artificial Intelligence\", \"language\": \"English\"}}}}\n"
"Example 2: {{\"Parameters\":[{{\"keyword\": \"Artificial Intelligence\", \"language\": \"English\"}}, {{\"keyword\": \"Machine Learning\", \"language\": \"English\"}}]}}\n"
"'''\n"
"This is user's question: {question}\n"
"Output:\n"
)
chat_prompt = ChatPromptTemplate.from_messages([system_message_prompt, human_message_prompt])
chain = LLMChain(llm=chat, prompt=chat_prompt)
ind = 0
while True:
try:
result = chain.run(api_dic=api_dic,
question=question, )
clean_answer = eval(
result.replace(": true", ": True").replace(":true", ": True").replace(":false", ": False").replace(
": false", ": False").replace("```", "").strip())
a = clean_answer["Parameters"]
return a
except Exception as e:
print(f"Choose Parameter fails: {e}")
if ind > 10:
return -1
ind += 1
continue
return a
def choose_parameter_depend(API_instruction, api, api_dic, question, previous_log, model_name):
chat = ChatOpenAI(model_name=model_name)
template = "You are a helpful assistant."
system_message_prompt = SystemMessagePromptTemplate.from_template(template)
human_message_prompt = HumanMessagePromptTemplate.from_template(
"Given a user's question and a API tool documentation, you need to output parameters according to the API tool documentation to successfully call the API to solve the user's question.\n"
"Please note that: \n"
"1. The Example in the API tool documentation can help you better understand the use of the API.\n"
"2. Ensure the parameters you output are correct. The output must contain the required parameters, and can contain the optional parameters based on the question. If no paremters in the required parameters and optional parameters, just leave it as {{\"Parameters\":{{}}}}\n"
"3. If the user's question mentions other APIs, you should ONLY consider the API tool documentation I give and do not consider other APIs.\n"
"4. The question may have dependencies on answers of other questions, so we will provide logs of previous questions and answers for your reference.\n"
"5. If you need to use this API multiple times,, please set \"Parameters\" to a list.\n"
"6. You must ONLY output in a parsible JSON format. Two examples output looks like:\n"
"'''\n"
"Example 1: {{\"Parameters\":{{\"keyword\": \"Artificial Intelligence\", \"language\": \"English\"}}}}\n"
"Example 2: {{\"Parameters\":[{{\"keyword\": \"Artificial Intelligence\", \"language\": \"English\"}}, {{\"keyword\": \"Machine Learning\", \"language\": \"English\"}}]}}\n"
"'''\n"
"There are logs of previous questions and answers: \n {previous_log}\n"
"This is the current user's question: {question}\n"
"This is API tool documentation: {api_dic}\n"
"Output:\n"
)
chat_prompt = ChatPromptTemplate.from_messages([system_message_prompt, human_message_prompt])
chain = LLMChain(llm=chat, prompt=chat_prompt)
ind = 0
while True:
try:
result = chain.run(api_dic=api_dic,
question=question,
previous_log=previous_log)
clean_answer = eval(
result.replace(": true", ": True").replace(": false", ": False").replace("```", "").strip())
a = clean_answer["Parameters"]
return a
except Exception as e:
print(f"choose parameter depend fails: {e}")
if ind > 10:
return -1
ind += 1
continue
return a
def answer_generation(question, API_instruction, call_result, model_name):
chat = ChatOpenAI(model_name=model_name)
template = "You are a helpful assistant."
system_message_prompt = SystemMessagePromptTemplate.from_template(template)
human_message_prompt = HumanMessagePromptTemplate.from_template(
"You should answer the question based on the response output by the API tool."
"Please note that:\n"
"1. Answer the question in natural language based on the API response reasonably and effectively.\n"
"2. The user cannot directly get API response, "
"so you need to make full use of the response and give the information "
"in the response that can satisfy the user's question in as much detail as possible.\n"
"This is the user's question:\n {question}\n"
"This is the API response:\n {call_result}\n"
"Output:"
)
chat_prompt = ChatPromptTemplate.from_messages([system_message_prompt, human_message_prompt])
chain = LLMChain(llm=chat, prompt=chat_prompt)
ind = 0
while True:
try:
result = chain.run(question=question,
call_result=call_result)
break
except Exception as e:
print(f"answer generation fails: {e}")
if ind > 2:
return -1
ind += 1
continue
return result
def answer_generation_depend(question, API_instruction, call_result, model_name, previous_log):
chat = ChatOpenAI(model_name=model_name)
template = "You are a helpful assistant."
system_message_prompt = SystemMessagePromptTemplate.from_template(template)
human_message_prompt = HumanMessagePromptTemplate.from_template(
"You should answer the question based on the response output by the API tool."
"Please note that:\n"
"1. Try to organize the response into a natural language answer.\n"
"2. We will not show the API response to the user, "
"thus you need to make full use of the response and give the information "
"in the response that can satisfy the user's question in as much detail as possible.\n"
"3. The question may have dependencies on answers of other questions, so we will provide logs of previous questions and answers.\n"
"There are logs of previous questions and answers: \n {previous_log}\n"
"This is the user's question: {question}\n"
"This is the response output by the API tool: \n{call_result}\n"
"We will not show the API response to the user, "
"thus you need to make full use of the response and give the information "
"in the response that can satisfy the user's question in as much detail as possible.\n"
"Output:"
)
chat_prompt = ChatPromptTemplate.from_messages([system_message_prompt, human_message_prompt])
chain = LLMChain(llm=chat, prompt=chat_prompt)
ind = 0
while True:
try:
result = chain.run(question=question,
call_result=call_result,
previous_log=previous_log)
break
except Exception as e:
print(f"answer generation depend fails: {e}")
if ind > 2:
return -1
ind += 1
continue
return result
def answer_check(question, answer, model_name):
chat = ChatOpenAI(model_name=model_name)
template = "You are a helpful assistant."
system_message_prompt = SystemMessagePromptTemplate.from_template(template)
human_message_prompt = HumanMessagePromptTemplate.from_template(
"Please check whether the response can reasonably and accurately answer the question."
"If can, please output 'YES'; If not, please output 'NO'\n"
"You need to give reasons first and then decide whether the response can reasonably and accurately answer the question. You must only output in a parsible JSON format. Two example outputs look like:\n"
"Example 1: {{\"Reason\": \"The reason why you think the response can reasonably and accurately answer the question\", \"Choice\": \"Yes\"}}\n"
"Example 2: {{\"Reason\": \"The reason why you think the response cannot reasonably and accurately answer the question\", \"Choice\": \"No\"}}\n"
"This is the user's question: {question}\n"
"This is the response: {answer}\n"
"Output: "
)
chat_prompt = ChatPromptTemplate.from_messages([system_message_prompt, human_message_prompt])
chain = LLMChain(llm=chat, prompt=chat_prompt)
result = chain.run(question=question, answer=answer)
if 'yes'.lower() in str(result).lower():
return 1
else:
return -1
def Call_function(A, B, arg, index, id):
if A in index:
for path in index[A]:
app_path = os.path.join(path, A, 'api.py')
if os.path.isfile(app_path):
spec = importlib.util.spec_from_file_location('api', app_path)
app_module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(app_module)
arg['toolbench_rapidapi_key'] = os.environ['RAPIDAPI_KEY']
# Check if B is a function in app
if hasattr(app_module, B):
function_B = getattr(app_module, B)
try:
call_result = function_B(**arg)
return call_result
except Exception as e:
try:
arg = {change_name(k.lower()): v for k, v in arg.items()}
call_result = function_B(**arg)
return call_result
except Exception as e:
try:
arg = {change_name(k.replace("-", "_")): v for k, v in arg.items()}
call_result = function_B(**arg)
return call_result
except Exception as e:
try:
arg = {change_name(k.replace("\\", "")): v for k, v in arg.items()}
call_result = function_B(**arg)
return call_result
except Exception as e:
print(f"Call function fails: {e}")
with open('wrong_log.json', 'a+', encoding='utf-8') as f:
line = json.dumps({
"id": id,
"parameters": arg,
"wrong": str(e)
}, ensure_ascii=False)
f.write(line + '\n')
return -1
else:
with open('wrong_log.json', 'a+', encoding='utf-8') as f:
line = json.dumps({
"id": id,
"parameters": arg,
"wrong": f"No function named {B} in {app_path}"
}, ensure_ascii=False)
f.write(line + '\n')
return (f"No function named {B} in {app_path}")
def retrieval(question, Tool_dic, dataset, tool_used, ind, model_name, index, previous_log=None):
tool_id = choose_tool(question, Tool_dic, tool_used, model_name)
if tool_id == -1:
return tool_id, "", "", "", ""
if str(tool_id["ID"]) not in dataset:
return tool_id, "", "", "", ""
tool_instruction = dataset[str(tool_id["ID"])]
API_instruction = tool_instruction["tool_description"]
API_tool = tool_instruction["standardized_name"]
API_list = []
for ele in tool_instruction["tool_guidelines"].keys():
API_list.append(ele)
api_selection = choose_API(API_instruction, API_list, question, model_name)
api_result = []
if len(api_selection) == 0:
call_result = ""
print("No Calling")
return tool_id, api_result, call_result, tool_instruction, API_instruction
for api in api_selection:
if previous_log is None:
parameter = choose_parameter(API_instruction, api,
tool_instruction["tool_guidelines"][api], question,
model_name)
else:
parameter = choose_parameter_depend(API_instruction, api,
tool_instruction["tool_guidelines"][api],
question, previous_log,
model_name)
if parameter == -1:
continue
api_result.append({"api_name": api, "parameters": parameter})
if len(api_result) == 0:
call_result = ""
return tool_id, api_result, call_result, tool_instruction, API_instruction
if isinstance(api_result, set) or isinstance(api_result, list):
call_results = []
for api in api_result:
api_name = change_name(standardize(api["api_name"]))
if isinstance(api["parameters"], dict):
parameters = {}
for key in api["parameters"]:
value = api["parameters"][key]
key = change_name(key)
parameters[key] = value
call_result = Call_function(API_tool, api_name, parameters, index, ind)
if call_result == -1:
continue
call_results.append(str(call_result))
elif isinstance(api["parameters"], list):
for para_ls in api["parameters"]:
parameters = {}
for key in para_ls:
value = para_ls[key]
key = change_name(key)
parameters[key] = value
call_result = Call_function(API_tool, api_name, parameters, index, ind)
if call_result == -1:
continue
call_results.append(str(call_result))
call_result = '\n\n'.join(call_results)
elif isinstance(api_result, dict):
api_name = change_name(standardize(api_result["api_name"]))
api = api_result
if isinstance(api["parameters"], dict):
parameters = {}
for key in api["parameters"]:
value = api["parameters"][key]
key = change_name(key)
parameters[key] = value
call_result = Call_function(API_tool, api_name, parameters, index, ind)
elif isinstance(api["parameters"], list):
call_results = []
for para_ls in api["parameters"]:
parameters = {}
for key in para_ls:
value = para_ls[key]
key = change_name(key)
parameters[key] = value
call_result = Call_function(API_tool, api_name, parameters, index, ind)
if call_result == -1:
continue
call_results.append(str(call_result))
call_result = '\n\n'.join(call_results)
return tool_id, api_result, call_result, tool_instruction, API_instruction
def task_decompose(question, model_name):
chat = ChatOpenAI(model_name=model_name)
template = "You are a helpful assistant."
system_message_prompt = SystemMessagePromptTemplate.from_template(template)
human_message_prompt = HumanMessagePromptTemplate.from_template(
"You need to decompose a complex user's question into some simple subtasks and let the model execute it step by step.\n"
"This is the user's question: {question}\n"
"Please note that: \n"
"1. You should only decompose this complex user's question into some simple subtasks which can be executed easily by using a single tool.\n"
"2. Each simple subtask should be expressed into natural language.\n"
"3. Each subtask should contain the necessary information from the original question and should be complete, explicit and self-consistent.\n"
"4. You must ONLY output the ID of the tool you chose in a parsible JSON format. An example output looks like:\n"
"'''\n"
"{{\"Tasks\": [\"Task 1\", \"Task 2\", ...]}}\n"
"'''\n"
"Output:"
)
chat_prompt = ChatPromptTemplate.from_messages([system_message_prompt, human_message_prompt])
chain = LLMChain(llm=chat, prompt=chat_prompt)
ind = 0
while True:
try:
result = chain.run(question=question)
result = eval(result.split('\n\n')[0])
a = result["Tasks"]
break
except Exception as e:
print(f"task decompose fails: {e}")
if ind > 10:
return -1
ind += 1
continue
return result
def task_topology(question, task_ls, model_name):
chat = ChatOpenAI(model_name=model_name)
template = "You are a helpful assistant."
system_message_prompt = SystemMessagePromptTemplate.from_template(template)
human_message_prompt = HumanMessagePromptTemplate.from_template(
"Given a complex user's question, I have decompose this question into some simple subtasks"
"I think there exists a logical connections and order amontg the tasks. "
"Thus you need to help me output this logical connections and order.\n"
"You must ONLY output in a parsible JSON format with the following format:\n"
"'''\n"
"[{{\"task\": task, \"id\", task_id, \"dep\": [dependency_task_id1, dependency_task_id2, ...]}}]\n"
"'''\n"
"The \"dep\" field denotes the id of the previous task which generates a new resource upon which the current task depends. If there are no dependencies, set \"dep\" to -1.\n\n"
"This is user's question: {question}\n"
"These are subtasks of this question:\n"
"{task_ls}\n"
"Output: "
)
chat_prompt = ChatPromptTemplate.from_messages([system_message_prompt, human_message_prompt])
chain = LLMChain(llm=chat, prompt=chat_prompt)
ind = 0
while True:
try:
result = chain.run(question=question, task_ls=task_ls)
result = eval(result)
for i in range(len(result)):
if isinstance(result[i]['dep'], str):
temp = []
for ele in result[i]['dep'].split(','):
temp.append(int(ele))
result[i]['dep'] = temp
elif isinstance(result[i]['dep'], int):
result[i]['dep'] = [result[i]['dep']]
elif isinstance(result[i]['dep'], list):
temp = []
for ele in result[i]['dep']:
temp.append(int(ele))
result[i]['dep'] = temp
elif result[i]['dep'] == -1:
result[i]['dep'] = [-1]
a = result[i]['dep'][0]
return result
except Exception as e:
print(f"task topology fails: {e}")
if ind > 10:
return -1
ind += 1
continue
return result
def answer_summarize(question, answer_task, model_name):
chat = ChatOpenAI(model_name=model_name)
template = "You are a helpful assistant."
system_message_prompt = SystemMessagePromptTemplate.from_template(template)
human_message_prompt = HumanMessagePromptTemplate.from_template(
"We break down a complex user's problems into simple subtasks and provide answers to each simple subtask. "
"You need to organize these answers to each subtask and form a self-consistent final answer to the user's question\n"
"This is the user's question: {question}\n"
"These are subtasks and their answers: {answer_task}\n"
"Final answer:"
)
chat_prompt = ChatPromptTemplate.from_messages([system_message_prompt, human_message_prompt])
chain = LLMChain(llm=chat, prompt=chat_prompt)
result = chain.run(question=question, answer_task=answer_task)
return result
def answer_generation_direct(task, model_name):
chat = ChatOpenAI(model_name=model_name)
template = "You are a helpful assistant."
system_message_prompt = SystemMessagePromptTemplate.from_template(template)
human_message_prompt = HumanMessagePromptTemplate.from_template(
"You need to answer the user's question.\n"
"This is the user's question: {task}\n"
"Output:"
)
chat_prompt = ChatPromptTemplate.from_messages([system_message_prompt, human_message_prompt])
chain = LLMChain(llm=chat, prompt=chat_prompt)
result = chain.run(task=task)
return result
def tool_check(task, model_name):
chat = ChatOpenAI(model_name=model_name)
template = "You are a helpful language model which can use external APIs to solve user's question."
system_message_prompt = SystemMessagePromptTemplate.from_template(template)
human_message_prompt = HumanMessagePromptTemplate.from_template(
"As a powerful language model, you're equipped to answer user's question with accumulated knowledge. "
"However, in some cases, you need to use external APIs to answer accurately."
"Thus, you need to check whether the user's question requires you to call an external API to solve it.\n"
"Here are some tips to help you check: \n"
"1. If the user's question requires real-time information, since your knowledge base isn't updated in real-time, any such question will demand an API call.\n"
"2. If you need to obtain information (e.g., ID, name, phone number, geographical location, rank, etc.), you need to call the database APIs if you are not sure.\n"
"3. If the question demand a database search or internet research to generate an answer, this is another situation where an API call is necessary.\n"
"If need, please output 'YES'; If not, please output 'NO'\n"
"You need to give reasons first and then decide whether to keep it or not. You must only output in a parsible JSON format. Two example outputs look like:\n"
"Example 1: {{\"Reason\": \"The reason why you think you do not need to call an external API to solve the user's question\", \"Choice\": \"No\"}}\n"
"Example 2: {{\"Reason\": \"The reason why you think you need to call an external API to solve the user's question\", \"Choice\": \"Yes\"}}\n"
"This is the user's question: {task}\n"
"Output:"
)
chat_prompt = ChatPromptTemplate.from_messages([system_message_prompt, human_message_prompt])
chain = LLMChain(llm=chat, prompt=chat_prompt)
ind = 0
while True:
try:
result = chain.run(task=task)
result = eval(result)
a = result["Reason"]
b = result["Choice"]
if 'yes' in b.lower():
return result, -1
else:
return result, 1
except Exception as e:
print(f"tool check fails: {e}")
if ind > 10:
return "", -1
ind += 1
continue
return result, -1
def task_execution(data_type,
base_path, index, dataset, test_data, progress_file,
start_index, total_files, retrieval_num, ind, model_name):
with tqdm(total=total_files, desc="Processing files", initial=start_index) as pbar:
for i, data in enumerate(test_data[start_index:], start=start_index):
answer_ls = []
question = data["query"]
print(question)
temp = task_decompose(question, model_name)['Tasks']
task_ls = []
for t in range(len(temp)):
task_ls.append({"task": temp[t], "id": t + 1})
task_ls = task_topology(question, task_ls, model_name)
task_depend = {}
for task_dic in task_ls:
task_depend[task_dic['id']] = {'task': task_dic['task'], 'answer': ''}
answer_task = []
api_result_ls = []
call_result_ls = []
tool_check_reason_ls = []
parameter_ls = []
for task_dic in task_ls:
task = task_dic['task']
tool_check_reason, tool_check_result = tool_check(task, model_name)
tool_check_reason_ls.append(tool_check_reason)
if tool_check_result == 1:
print("Do not need tool.")
answer = answer_generation_direct(task)
answer_task.append({'task': task, 'answer': answer})
else:
print("Do need tool.")
depend_id = task_dic['dep']
tool_used = []
for r in range(retrieval_num):
Tool_dic = data["Tool_dic"]
if depend_id[0] == -1:
tool_id, api_result, call_result, tool_instruction, API_instruction = retrieval(task,
Tool_dic,
dataset,
tool_used,
ind,
model_name,
index)
call_result = str(call_result)[:1000]
answer = answer_generation(task, API_instruction,
call_result, model_name)
else:
previous_log = []
for ids in depend_id:
previous_log.append(task_depend[ids])
tool_id, api_result, call_result, tool_instruction, API_instruction = retrieval(task,
Tool_dic,
dataset,
tool_used,
ind,
model_name,
index,
previous_log=previous_log)
call_result = str(call_result)[:1000]
answer = answer_generation_depend(task, API_instruction, call_result, model_name,
previous_log=previous_log)
check_index = answer_check(task, answer, model_name)
if check_index == 1:
answer_task.append({'task': task, 'answer': answer})
api_result_ls.append(api_result)
call_result_ls.append(call_result)
break
else:
answer_ls.append({'task': task, 'answer': answer})
try:
tool_used.append(str(tool_id["ID"]))
except:
continue
print('****Try Again****')
task_depend[task_dic['id']]['answer'] = answer
final_answer = answer_summarize(question, answer_task, model_name)
check_index = answer_check(question, final_answer, model_name)
ind = ind + 1
with open(f'''{data_type}_{model_name}_Easytool.jsonl''', 'a+', encoding='utf-8') as f:
line = json.dumps({
"ID": ind,
"question": question,
"final_answer": final_answer,
"subtask": task_ls,
"answer_subtask": answer_task,
"answer_wrong": answer_ls,
"check_index": check_index,
"execute_log": {
"api_result_ls": api_result_ls,
"parameter_ls": parameter_ls,
"call_result_ls": call_result_ls,
"tool_check_reason_ls": tool_check_reason_ls,
}
}, ensure_ascii=False)
f.write(line + '\n')
print(final_answer)
update_progress(progress_file, i + 1)
pbar.update(1)
================================================
FILE: easytool/easytool/toolbench_retrieve.py
================================================
# — coding: utf-8 –
import openai
import json
import logging
import sys
import argparse
from langchain.chat_models import ChatOpenAI
from langchain.prompts import (
ChatPromptTemplate,
MessagesPlaceholder,
SystemMessagePromptTemplate,
HumanMessagePromptTemplate
)
from langchain import LLMChain
import numpy as np
import requests
import os
import subprocess
import re
import importlib.util
from sklearn.metrics.pairwise import cosine_similarity
import pickle
from util import *
from tqdm import tqdm
openai.api_key = os.environ["OPENAI_API_KEY"]
def get_last_processed_index(progress_file):
"""Retrieve the last processed index from the progress file."""
if os.path.exists(progress_file):
with open(progress_file, 'r', encoding='utf-8') as f:
last_index = f.read().strip()
return int(last_index) if last_index else 0
else:
return 0
def update_progress(progress_file, index):
"""Update the last processed index in the progress file."""
with open(progress_file, 'w', encoding='utf-8') as f:
f.write(str(index))
def get_embedding(text):
a = openai.Embedding.create(
engine="text-embedding-ada-002",
input=text
)
return a['data'][0]["embedding"]
def retrieve_reference(embedded_texts, filenames, question, k):
input_text = question
input_embedding = get_embedding(input_text)
similarities = [cosine_similarity([input_embedding], [emb])[0][0] for emb in embedded_texts]
top_k_indices = sorted(range(len(similarities)), key=lambda i: similarities[i], reverse=True)[:k]
return [filenames[i] for i in top_k_indices]
def choose_tool(question, Tool_dic, tool_used, model_name):
chat = ChatOpenAI(model_name=model_name)
template = "You are a helpful assistant."
system_message_prompt = SystemMessagePromptTemplate.from_template(template)
human_message_prompt = HumanMessagePromptTemplate.from_template(
"This is the user's question: {question}\n"
"These are the tools you can select to solve the question:\n"
"Tool List:\n"
"{Too_list}\n\n"
"Please note that: \n"
"1. You should only chooce one tool the Tool List to solve this question.\n"
"2. You must ONLY output the ID of the tool you chose in a parsible JSON format. An example output looks like:\n"
"'''\n"
"Example: {{\"ID\": XX}}\n"
"'''\n"
"Output:"
)
chat_prompt = ChatPromptTemplate.from_messages([system_message_prompt, human_message_prompt])
chain = LLMChain(llm=chat, prompt=chat_prompt)
ind = 0
Tool_list = []
for ele in Tool_dic:
for key in ele.keys():
if str(key) not in tool_used:
Tool_list.append(f'''ID: {key}\n{ele[key]}''')
while True:
try:
result = chain.run(question=question,
Too_list='\n'.join(Tool_list))
clean_answer = eval(result.split("\n\n")[-1].strip())
break
except Exception as e:
print(f"choose tool fails:{e}")
if ind > 10:
return -1
ind += 1
continue
return clean_answer
def choose_API(API_instruction, API_list, question, model_name):
input_execute_rapidapi_api_note = '''
This is an API Tool instruction. Given a question, you should choose APIs from the API list you want to use for this question in this instruction.
you must only output in a parsible Python List Format. An example output looks like:
```
["api1", "api2", ...]
```
'''.strip()
chat = ChatOpenAI(model_name=model_name)
template = "You are a helpful assistant."
system_message_prompt = SystemMessagePromptTemplate.from_template(template)
human_message_prompt = HumanMessagePromptTemplate.from_template(
"{API_instruction}\n"
"{input_execute_rapidapi_api_note}\n"
"This is the API list: {API_list}\n"
"Please note that: \n"
"1. The APIs you choose must in the API list.\n"
"2. You must ONLY output in the following parsible Python List Format.\n"
"```\n"
"Output_Example: [\"api1\", \"api2\", ...]\n"
"```\n"
"Question: {question}\n"
"Output:"
)
chat_prompt = ChatPromptTemplate.from_messages([system_message_prompt, human_message_prompt])
chain = LLMChain(llm=chat, prompt=chat_prompt)
ind = 0
while True:
try:
result = chain.run(API_instruction=API_instruction,
API_list=API_list,
question=question,
input_execute_rapidapi_api_note=input_execute_rapidapi_api_note)
clean_answer = eval(result.replace("```", "").strip().split("\n\n")[0].strip())
if isinstance(clean_answer, str):
ls = [clean_answer]
elif isinstance(clean_answer, list):
ls = clean_answer
temp = []
for ele in ls:
if ele in API_list:
temp.append(ele)
ls = temp
return ls
except Exception as e:
print(f"Choose API fails:{e}")
print(result)
if ind > 10:
return []
ind += 1
continue
return ls
def choose_parameter(API_instruction, api, api_dic, question, model_name):
chat = ChatOpenAI(model_name=model_name)
template = "You are a helpful assistant."
system_message_prompt = SystemMessagePromptTemplate.from_template(template)
human_message_prompt = HumanMessagePromptTemplate.from_template(
"This is an API tool documentation. Given a user's question, you need to output parameters according to the API tool documentation to successfully call the API to solve the user's question.\n"
"This is API tool documentation: {api_dic}\n"
"Please note that: \n"
"1. The Example in the API tool documentation can help you better understand the use of the API.\n"
"2. Ensure the parameters you output are correct. The output must contain the required parameters, and can contain the optional parameters based on the question. If no paremters in the required parameters and optional parameters, just leave it as {{\"Parameters\":{{}}}}\n"
"3. If the user's question mentions other APIs, you should ONLY consider the API tool documentation I give and do not consider other APIs.\n"
"4. If you need to use this API multiple times, please set \"Parameters\" to a list.\n"
"5. You must ONLY output in a parsible JSON format. Two examples output looks like:\n"
"'''\n"
"Example 1: {{\"Parameters\":{{\"keyword\": \"Artificial Intelligence\", \"language\": \"English\"}}}}\n"
"Example 2: {{\"Parameters\":[{{\"keyword\": \"Artificial Intelligence\", \"language\": \"English\"}}, {{\"keyword\": \"Machine Learning\", \"language\": \"English\"}}]}}\n"
"'''\n"
"This is user's question: {question}\n"
"Output:\n"
)
chat_prompt = ChatPromptTemplate.from_messages([system_message_prompt, human_message_prompt])
chain = LLMChain(llm=chat, prompt=chat_prompt)
ind = 0
while True:
try:
result = chain.run(api_dic=api_dic,
question=question, )
clean_answer = eval(
result.replace(": true", ": True").replace(":true", ": True").replace(":false", ": False").replace(
": false", ": False").replace("```", "").strip())
a = clean_answer["Parameters"]
return a
except Exception as e:
print(f"Choose Parameter fails:{e}")
if ind > 10:
return -1
ind += 1
continue
return a
def choose_parameter_depend(API_instruction, api, api_dic, question, previous_log, model_name):
chat = ChatOpenAI(model_name=model_name)
template = "You are a helpful assistant."
system_message_prompt = SystemMessagePromptTemplate.from_template(template)
human_message_prompt = HumanMessagePromptTemplate.from_template(
"Given a user's question and a API tool documentation, you need to output parameters according to the API tool documentation to successfully call the API to solve the user's question.\n"
"Please note that: \n"
"1. The Example in the API tool documentation can help you better understand the use of the API.\n"
"2. Ensure the parameters you output are correct. The output must contain the required parameters, and can contain the optional parameters based on the question. If no paremters in the required parameters and optional parameters, just leave it as {{\"Parameters\":{{}}}}\n"
"3. If the user's question mentions other APIs, you should ONLY consider the API tool documentation I give and do not consider other APIs.\n"
"4. The question may have dependencies on answers of other questions, so we will provide logs of previous questions and answers for your reference.\n"
"5. If you need to use this API multiple times,, please set \"Parameters\" to a list.\n"
"6. You must ONLY output in a parsible JSON format. Two examples output looks like:\n"
"'''\n"
"Example 1: {{\"Parameters\":{{\"keyword\": \"Artificial Intelligence\", \"language\": \"English\"}}}}\n"
"Example 2: {{\"Parameters\":[{{\"keyword\": \"Artificial Intelligence\", \"language\": \"English\"}}, {{\"keyword\": \"Machine Learning\", \"language\": \"English\"}}]}}\n"
"'''\n"
"There are logs of previous questions and answers: \n {previous_log}\n"
"This is the current user's question: {question}\n"
"This is API tool documentation: {api_dic}\n"
"Output:\n"
)
chat_prompt = ChatPromptTemplate.from_messages([system_message_prompt, human_message_prompt])
chain = LLMChain(llm=chat, prompt=chat_prompt)
ind = 0
while True:
try:
result = chain.run(api_dic=api_dic,
question=question,
previous_log=previous_log)
clean_answer = eval(
result.replace(": true", ": True").replace(": false", ": False").replace("```", "").strip())
a = clean_answer["Parameters"]
return a
except Exception as e:
print(f"choose parameter depend fails:{e}")
if ind > 10:
return -1
ind += 1
continue
return a
def answer_generation(question, API_instruction, call_result, model_name):
chat = ChatOpenAI(model_name=model_name)
template = "You are a helpful assistant."
system_message_prompt = SystemMessagePromptTemplate.from_template(template)
human_message_prompt = HumanMessagePromptTemplate.from_template(
"You should answer the question based on the response output by the API tool."
"Please note that:\n"
"1. Answer the question in natural language based on the API response reasonably and effectively.\n"
"2. The user cannot directly get API response, "
"so you need to make full use of the response and give the information "
"in the response that can satisfy the user's question in as much detail as possible.\n"
"This is the user's question:\n {question}\n"
"This is the API response:\n {call_result}\n"
"Output:"
)
chat_prompt = ChatPromptTemplate.from_messages([system_message_prompt, human_message_prompt])
chain = LLMChain(llm=chat, prompt=chat_prompt)
ind = 0
while True:
try:
result = chain.run(question=question,
call_result=call_result)
break
except Exception as e:
print(f"answer generation fails:{e}")
if ind > 2:
return -1
ind += 1
continue
return result
def answer_generation_depend(question, API_instruction, call_result, model_name, previous_log):
chat = ChatOpenAI(model_name=model_name)
template = "You are a helpful assistant."
system_message_prompt = SystemMessagePromptTemplate.from_template(template)
human_message_prompt = HumanMessagePromptTemplate.from_template(
"You should answer the question based on the response output by the API tool."
"Please note that:\n"
"1. Try to organize the response into a natural language answer.\n"
"2. We will not show the API response to the user, "
"thus you need to make full use of the response and give the information "
"in the response that can satisfy the user's question in as much detail as possible.\n"
"3. The question may have dependencies on answers of other questions, so we will provide logs of previous questions and answers.\n"
"There are logs of previous questions and answers: \n {previous_log}\n"
"This is the user's question: {question}\n"
"This is the response output by the API tool: \n{call_result}\n"
"We will not show the API response to the user, "
"thus you need to make full use of the response and give the information "
"in the response that can satisfy the user's question in as much detail as possible.\n"
"Output:"
)
chat_prompt = ChatPromptTemplate.from_messages([system_message_prompt, human_message_prompt])
chain = LLMChain(llm=chat, prompt=chat_prompt)
ind = 0
while True:
try:
result = chain.run(question=question,
call_result=call_result,
previous_log=previous_log)
break
except Exception as e:
print(f"answer generation depend fails:{e}")
if ind > 2:
return -1
ind += 1
continue
return result
def answer_check(question, answer, model_name):
chat = ChatOpenAI(model_name=model_name)
template = "You are a helpful assistant."
system_message_prompt = SystemMessagePromptTemplate.from_template(template)
human_message_prompt = HumanMessagePromptTemplate.from_template(
"Please check whether the response can reasonably and accurately answer the question."
"If can, please output 'YES'; If not, please output 'NO'\n"
"You need to give reasons first and then decide whether the response can reasonably and accurately answer the question. You must only output in a parsible JSON format. Two example outputs look like:\n"
"Example 1: {{\"Reason\": \"The reason why you think the response can reasonably and accurately answer the question\", \"Choice\": \"Yes\"}}\n"
"Example 2: {{\"Reason\": \"The reason why you think the response cannot reasonably and accurately answer the question\", \"Choice\": \"No\"}}\n"
"This is the user's question: {question}\n"
"This is the response: {answer}\n"
"Output: "
)
chat_prompt = ChatPromptTemplate.from_messages([system_message_prompt, human_message_prompt])
chain = LLMChain(llm=chat, prompt=chat_prompt)
result = chain.run(question=question, answer=answer)
if 'yes'.lower() in str(result).lower():
return 1
else:
return -1
def Call_function(A, B, arg, index, id):
if A in index:
for path in index[A]:
app_path = os.path.join(path, A, 'api.py')
if os.path.isfile(app_path):
spec = importlib.util.spec_from_file_location('api', app_path)
app_module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(app_module)
arg['toolbench_rapidapi_key'] = os.environ['RAPIDAPI_KEY']
# Check if B is a function in app
if hasattr(app_module, B):
function_B = getattr(app_module, B)
try:
call_result = function_B(**arg)
return call_result
except Exception as e:
try:
arg = {change_name(k.lower()): v for k, v in arg.items()}
call_result = function_B(**arg)
return call_result
except Exception as e:
try:
arg = {change_name(k.replace("-", "_")): v for k, v in arg.items()}
call_result = function_B(**arg)
return call_result
except Exception as e:
try:
arg = {change_name(k.replace("\\", "")): v for k, v in arg.items()}
call_result = function_B(**arg)
return call_result
except Exception as e:
print(f"Call function fails:{e}")
with open('wrong_log.json', 'a+', encoding='utf-8') as f:
line = json.dumps({
"id": id,
"parameters": arg,
"wrong": str(e)
}, ensure_ascii=False)
f.write(line + '\n')
return -1
else:
with open('wrong_log.json', 'a+', encoding='utf-8') as f:
line = json.dumps({
"id": id,
"parameters": arg,
"wrong": f"No function named {B} in {app_path}"
}, ensure_ascii=False)
f.write(line + '\n')
return (f"No function named {B} in {app_path}")
def retrieval(question, Tool_dic, dataset, tool_used, ind, model_name, index, previous_log=None):
tool_id = choose_tool(question, Tool_dic, tool_used, model_name)
if tool_id == -1:
return tool_id, "", "", "", ""
if str(tool_id["ID"]) not in dataset:
return tool_id, "", "", "", ""
tool_instruction = dataset[str(tool_id["ID"])]
API_instruction = tool_instruction["tool_description"]
API_tool = tool_instruction["standardized_name"]
API_list = []
for ele in tool_instruction["tool_guidelines"].keys():
API_list.append(ele)
api_selection = choose_API(API_instruction, API_list, question, model_name)
api_result = []
if len(api_selection) == 0:
call_result = ""
print("No Calling")
return tool_id, api_result, call_result, tool_instruction, API_instruction
for api in api_selection:
if previous_log is None:
parameter = choose_parameter(API_instruction, api,
tool_instruction["tool_guidelines"][api], question,
model_name)
else:
parameter = choose_parameter_depend(API_instruction, api,
tool_instruction["tool_guidelines"][api],
question, previous_log,
model_name)
if parameter == -1:
continue
api_result.append({"api_name": api, "parameters": parameter})
if len(api_result) == 0:
call_result = ""
return tool_id, api_result, call_result, tool_instruction, API_instruction
if isinstance(api_result, set) or isinstance(api_result, list):
call_results = []
for api in api_result:
api_name = change_name(standardize(api["api_name"]))
if isinstance(api["parameters"], dict):
parameters = {}
for key in api["parameters"]:
value = api["parameters"][key]
key = change_name(key)
parameters[key] = value
call_result = Call_function(API_tool, api_name, parameters, index, ind)
if call_result == -1:
continue
call_results.append(str(call_result))
elif isinstance(api["parameters"], list):
for para_ls in api["parameters"]:
parameters = {}
for key in para_ls:
value = para_ls[key]
key = change_name(key)
parameters[key] = value
call_result = Call_function(API_tool, api_name, parameters, index, ind)
if call_result == -1:
continue
call_results.append(str(call_result))
call_result = '\n\n'.join(call_results)
elif isinstance(api_result, dict):
api_name = change_name(standardize(api_result["api_name"]))
api = api_result
if isinstance(api["parameters"], dict):
parameters = {}
for key in api["parameters"]:
value = api["parameters"][key]
key = change_name(key)
parameters[key] = value
call_result = Call_function(API_tool, api_name, parameters, index, ind)
elif isinstance(api["parameters"], list):
call_results = []
for para_ls in api["parameters"]:
parameters = {}
for key in para_ls:
value = para_ls[key]
key = change_name(key)
parameters[key] = value
call_result = Call_function(API_tool, api_name, parameters, index, ind)
if call_result == -1:
continue
call_results.append(str(call_result))
call_result = '\n\n'.join(call_results)
return tool_id, api_result, call_result, tool_instruction, API_instruction
def task_decompose(question, model_name):
chat = ChatOpenAI(model_name=model_name)
template = "You are a helpful assistant."
system_message_prompt = SystemMessagePromptTemplate.from_template(template)
human_message_prompt = HumanMessagePromptTemplate.from_template(
"You need to decompose a complex user's question into some simple subtasks and let the model execute it step by step.\n"
"This is the user's question: {question}\n"
"Please note that: \n"
"1. You should only decompose this complex user's question into some simple subtasks which can be executed easily by using a single tool.\n"
"2. Each simple subtask should be expressed into natural language.\n"
"3. Each subtask should contain the necessary information from the original question and should be complete, explicit and self-consistent.\n"
"4. You must ONLY output the ID of the tool you chose in a parsible JSON format. An example output looks like:\n"
"'''\n"
"{{\"Tasks\": [\"Task 1\", \"Task 2\", ...]}}\n"
"'''\n"
"Output:"
)
chat_prompt = ChatPromptTemplate.from_messages([system_message_prompt, human_message_prompt])
chain = LLMChain(llm=chat, prompt=chat_prompt)
ind = 0
while True:
try:
result = chain.run(question=question)
result = eval(result.split('\n\n')[0])
a = result["Tasks"]
break
except Exception as e:
print(f"task decompose fails:{e}")
if ind > 10:
return -1
ind += 1
continue
return result
def task_topology(question, task_ls, model_name):
chat = ChatOpenAI(model_name=model_name)
template = "You are a helpful assistant."
system_message_prompt = SystemMessagePromptTemplate.from_template(template)
human_message_prompt = HumanMessagePromptTemplate.from_template(
"Given a complex user's question, I have decompose this question into some simple subtasks"
"I think there exists a logical connections and order amontg the tasks. "
"Thus you need to help me output this logical connections and order.\n"
"You must ONLY output in a parsible JSON format with the following format:\n"
"'''\n"
"[{{\"task\": task, \"id\", task_id, \"dep\": [dependency_task_id1, dependency_task_id2, ...]}}]\n"
"'''\n"
"The \"dep\" field denotes the id of the previous task which generates a new resource upon which the current task depends. If there are no dependencies, set \"dep\" to -1.\n\n"
"This is user's question: {question}\n"
"These are subtasks of this question:\n"
"{task_ls}\n"
"Output: "
)
chat_prompt = ChatPromptTemplate.from_messages([system_message_prompt, human_message_prompt])
chain = LLMChain(llm=chat, prompt=chat_prompt)
ind = 0
while True:
try:
result = chain.run(question=question, task_ls=task_ls)
result = eval(result)
for i in range(len(result)):
if isinstance(result[i]['dep'], str):
temp = []
for ele in result[i]['dep'].split(','):
temp.append(int(ele))
result[i]['dep'] = temp
elif isinstance(result[i]['dep'], int):
result[i]['dep'] = [result[i]['dep']]
elif isinstance(result[i]['dep'], list):
temp = []
for ele in result[i]['dep']:
temp.append(int(ele))
result[i]['dep'] = temp
elif result[i]['dep'] == -1:
result[i]['dep'] = [-1]
a = result[i]['dep'][0]
return result
except Exception as e:
print(f"task topology fails:{e}")
if ind > 10:
return -1
ind += 1
continue
return result
def answer_summarize(question, answer_task, model_name):
chat = ChatOpenAI(model_name=model_name)
template = "You are a helpful assistant."
system_message_prompt = SystemMessagePromptTemplate.from_template(template)
human_message_prompt = HumanMessagePromptTemplate.from_template(
"We break down a complex user's problems into simple subtasks and provide answers to each simple subtask. "
"You need to organize these answers to each subtask and form a self-consistent final answer to the user's question\n"
"This is the user's question: {question}\n"
"These are subtasks and their answers: {answer_task}\n"
"Final answer:"
)
chat_prompt = ChatPromptTemplate.from_messages([system_message_prompt, human_message_prompt])
chain = LLMChain(llm=chat, prompt=chat_prompt)
result = chain.run(question=question, answer_task=answer_task)
return result
def answer_generation_direct(task, model_name):
chat = ChatOpenAI(model_name=model_name)
template = "You are a helpful assistant."
system_message_prompt = SystemMessagePromptTemplate.from_template(template)
human_message_prompt = HumanMessagePromptTemplate.from_template(
"You need to answer the user's question.\n"
"This is the user's question: {task}\n"
"Output:"
)
chat_prompt = ChatPromptTemplate.from_messages([system_message_prompt, human_message_prompt])
chain = LLMChain(llm=chat, prompt=chat_prompt)
result = chain.run(task=task)
return result
def tool_check(task, model_name):
chat = ChatOpenAI(model_name=model_name)
template = "You are a helpful language model which can use external APIs to solve user's question."
system_message_prompt = SystemMessagePromptTemplate.from_template(template)
human_message_prompt = HumanMessagePromptTemplate.from_template(
"As a powerful language model, you're equipped to answer user's question with accumulated knowledge. "
"However, in some cases, you need to use external APIs to answer accurately."
"Thus, you need to check whether the user's question requires you to call an external API to solve it.\n"
"Here are some tips to help you check: \n"
"1. If the user's question requires real-time information, since your knowledge base isn't updated in real-time, any such question will demand an API call.\n"
"2. If you need to obtain information (e.g., ID, name, phone number, geographical location, rank, etc.), you need to call the database APIs if you are not sure.\n"
"3. If the question demand a database search or internet research to generate an answer, this is another situation where an API call is necessary.\n"
"If need, please output 'YES'; If not, please output 'NO'\n"
"You need to give reasons first and then decide whether to keep it or not. You must only output in a parsible JSON format. Two example outputs look like:\n"
"Example 1: {{\"Reason\": \"The reason why you think you do not need to call an external API to solve the user's question\", \"Choice\": \"No\"}}\n"
"Example 2: {{\"Reason\": \"The reason why you think you need to call an external API to solve the user's question\", \"Choice\": \"Yes\"}}\n"
"This is the user's question: {task}\n"
"Output:"
)
chat_prompt = ChatPromptTemplate.from_messages([system_message_prompt, human_message_prompt])
chain = LLMChain(llm=chat, prompt=chat_prompt)
ind = 0
while True:
try:
result = chain.run(task=task)
result = eval(result)
a = result["Reason"]
b = result["Choice"]
if 'yes' in b.lower():
return result, -1
else:
return result, 1
except Exception as e:
print(f"tool check fails:{e}")
if ind > 10:
return "", -1
ind += 1
continue
return result, -1
def task_execution(data_type,
base_path, index, dataset, test_data, progress_file,
start_index, total_files, retrieval_num, ind, model_name):
with open("data_toolbench/tool_instruction/API_description_embeddings.pkl", "rb") as file:
filenames, embedded_texts = pickle.load(file)
with tqdm(total=total_files, desc="Processing files", initial=start_index) as pbar:
for i, data in enumerate(test_data[start_index:], start=start_index):
answer_ls = []
question = data["query"]
print(question)
temp = task_decompose(question, model_name)['Tasks']
task_ls = []
for t in range(len(temp)):
task_ls.append({"task": temp[t], "id": t + 1})
task_ls = task_topology(question, task_ls, model_name)
task_depend = {}
for task_dic in task_ls:
task_depend[task_dic['id']] = {'task': task_dic['task'], 'answer': ''}
answer_task = []
api_result_ls = []
call_result_ls = []
tool_check_reason_ls = []
parameter_ls = []
for task_dic in task_ls:
task = task_dic['task']
tool_check_reason, tool_check_result = tool_check(task, model_name)
tool_check_reason_ls.append(tool_check_reason)
if tool_check_result == 1:
print("Do not need tool.")
answer = answer_generation_direct(task)
answer_task.append({'task': task, 'answer': answer})
else:
print("Do need tool.")
depend_id = task_dic['dep']
tool_used = []
Tool_dic = [{tool: dataset[str(tool)]["tool_description"]} for tool in
retrieve_reference(embedded_texts, filenames, task, k=5)]
for r in range(retrieval_num):
if depend_id[0] == -1:
tool_id, api_result, call_result, tool_instruction, API_instruction = retrieval(task,
Tool_dic,
dataset,
tool_used,
ind,
model_name,
index)
call_result = str(call_result)[:1000]
answer = answer_generation(task, API_instruction,
call_result, model_name)
else:
previous_log = []
for ids in depend_id:
previous_log.append(task_depend[ids])
tool_id, api_result, call_result, tool_instruction, API_instruction = retrieval(task,
Tool_dic,
dataset,
tool_used,
ind,
model_name,
index,
previous_log=previous_log)
call_result = str(call_result)[:1000]
answer = answer_generation_depend(task, API_instruction, call_result, model_name,
previous_log=previous_log)
check_index = answer_check(task, answer, model_name)
if check_index == 1:
answer_task.append({'task': task, 'answer': answer})
api_result_ls.append(api_result)
call_result_ls.append(call_result)
break
else:
answer_ls.append({'task': task, 'answer': answer})
try:
tool_used.append(str(tool_id["ID"]))
except:
continue
print('****Try Again****')
task_depend[task_dic['id']]['answer'] = answer
final_answer = answer_summarize(question, answer_task, model_name)
check_index = answer_check(question, final_answer, model_name)
ind = ind + 1
with open(f'''{data_type}_{model_name}_retrieve_Easytool.jsonl''', 'a+', encoding='utf-8') as f:
line = json.dumps({
"ID": ind,
"question": question,
"final_answer": final_answer,
"subtask": task_ls,
"answer_subtask": answer_task,
"answer_wrong": answer_ls,
"check_index": check_index,
"execute_log": {
"api_result_ls": api_result_ls,
"parameter_ls": parameter_ls,
"call_result_ls": call_result_ls,
"tool_check_reason_ls": tool_check_reason_ls,
}
}, ensure_ascii=False)
f.write(line + '\n')
print(final_answer)
update_progress(progress_file, i + 1)
pbar.update(1)
================================================
FILE: easytool/easytool/util.py
================================================
# — coding: utf-8 –
import json
import re
import os
def read_jsonline(address):
not_mark = []
with open(address, 'r', encoding="utf-8") as f:
for jsonstr in f.readlines():
jsonstr = json.loads(jsonstr)
not_mark.append(jsonstr)
return not_mark
def save_json(ls, address):
json_str = json.dumps(ls, indent=4)
with open(address, 'w', encoding='utf-8') as json_file:
json.dump(ls, json_file, ensure_ascii=False, indent=4)
def read_json(address):
with open(address, 'r', encoding='utf-8') as json_file:
json_data = json.load(json_file)
return json_data
def remove_key(item, key_to_remove):
if isinstance(item, dict):
if key_to_remove in item:
del item[key_to_remove]
for key, value in list(item.items()): # 使用list包裹，防止字典大小改变时引发错误
item[key] = remove_key(value, key_to_remove)
elif isinstance(item, list):
for index, value in enumerate(item):
item[index] = remove_key(value, key_to_remove)
return item
def data_clean(dic, key):
dic = remove_key(dic, key)
return dic
def lowercase_parameter_keys(input_dict):
if "parameters" in input_dict and isinstance(input_dict["parameters"], dict):
# Convert all keys in the "parameters" dictionary to uppercase
input_dict["parameters"] = {change_name(k.lower()): v for k, v in input_dict["parameters"].items()}
return input_dict
def build_index(base_path):
index = {}
for root, dirs, files in os.walk(base_path):
for dir_name in dirs:
if dir_name not in index:
index[dir_name] = []
index[dir_name].append(root)
return index
def change_name(name):
change_list = ["from", "class", "return", "false", "true", "id", "and", "", "ID"]
if name in change_list:
name = "is_" + name.lower()
return name
def standardize(string):
res = re.compile("[^\\u4e00-\\u9fa5^a-z^A-Z^0-9^_]")
string = res.sub("_", string)
string = re.sub(r"(_)\1+", "_", string).lower()
while True:
if len(string) == 0:
return string
if string[0] == "_":
string = string[1:]
else:
break
while True:
if len(string) == 0:
return string
if string[-1] == "_":
string = string[:-1]
else:
break
if string[0].isdigit():
string = "get_" + string
return string
def get_last_processed_index(progress_file):
"""Retrieve the last processed index from the progress file."""
if os.path.exists(progress_file):
with open(progress_file, 'r', encoding='utf-8') as f:
last_index = f.read().strip()
return int(last_index) if last_index else 0
else:
return 0
def update_progress(progress_file, index):
"""Update the last processed index in the progress file."""
with open(progress_file, 'w', encoding='utf-8') as f:
f.write(str(index))
if __name__ == '__main__':
print("util.py")
================================================
FILE: easytool/main.py
================================================
# — coding: utf-8 –
import openai
import json
import argparse
import os
from tqdm import tqdm
from easytool import funcQA, restbench, toolbench_retrieve, toolbench
from easytool.util import *
openai.api_key = os.environ["OPENAI_API_KEY"]
if __name__ == '__main__':
parser = argparse.ArgumentParser()
parser.add_argument('--model_name', type=str, default='gpt-3.5-turbo')
parser.add_argument('--task', type=str, default='funcqa_mh', help='funcqa, toolbench_retrieve, toolbench, restbench')
parser.add_argument('--data_type', type=str, default='G3', help='G2 or G3 or funcqa_mh or funcqa_oh')
parser.add_argument('--tool_root_dir', type=str, default='.toolenv/tools/')
parser.add_argument('--retrieval_num', type=int, default=5)
args = parser.parse_args()
if args.task == 'funcqa':
dataset = read_json('data_funcqa/tool_instruction/functions_data.json')
Tool_dic = read_jsonline('data_funcqa/tool_instruction/tool_dic.jsonl')
test_data = read_json(f"data_funcqa/test_data/{args.data_type}.json")
progress_file = f"FuncQA_{args.data_type}_{args.model_name}_Easytool.txt"
elif 'toolbench' in args.task:
base_path = args.tool_root_dir
index = build_index(base_path)
dataset = read_json('data_toolbench/tool_instruction/toolbench_tool_instruction.json')
if args.data_type == 'G2':
test_data = read_json(f'''data_toolbench/test_data/{args.data_type}_category.json''')
elif args.data_type == 'G3':
test_data = read_json(f'''data_toolbench/test_data/{args.data_type}_instruction.json''')
progress_file = f'''{args.data_type}_{args.model_name}_Easytool.txt'''
elif args.task == 'restbench':
Tool_dic = read_json('data_restbench/tool_instruction/tmdb_tool.json')
dic_tool = {}
for data in Tool_dic:
dic_tool[data['ID']] = data
test_data = read_json('data_restbench/test_data/tmdb.json')
progress_file = f"restbench_{args.model_name}_Easytool.txt"
else:
print("Wrong task name")
exit()
start_index = get_last_processed_index(progress_file)
total_files = len(test_data)
retrieval_num = args.retrieval_num
ind = start_index
model_name = args.model_name
print("-------Start Execution-------")
if args.data_type == 'funcqa_mh':
funcQA.task_execution_mh(args.data_type, start_index, total_files,
retrieval_num, ind, model_name, dataset,
Tool_dic, test_data, progress_file)
elif args.data_type == 'funcqa_oh':
funcQA.task_execution_oh(args.data_type, start_index, total_files,
retrieval_num, ind, model_name, dataset,
Tool_dic, test_data, progress_file)
elif args.task == 'toolbench_retrieve':
toolbench_retrieve.task_execution(args.data_type,
base_path, index, dataset, test_data, progress_file,
start_index, total_files, retrieval_num, ind, model_name)
elif args.task == 'toolbench':
toolbench.task_execution(args.data_type,
base_path, index, dataset, test_data, progress_file,
start_index, total_files, retrieval_num, ind, model_name)
elif args.task == 'restbench':
restbench.task_execution(
Tool_dic, dic_tool, test_data, progress_file,
start_index, total_files, retrieval_num, ind, model_name)
else:
print("Wrong task name")
exit()
================================================
FILE: easytool/requirements.txt
================================================
openai==0.27.8
langchain==0.0.260
gdown==4.6.0
tqdm
argparse
numpy
requests
pickle-mixin
scikit-learn
================================================
FILE: hugginggpt/.dockerignore
================================================
.git
server/models/*
!server/models/download.sh
!server/models/download.ps1
================================================
FILE: hugginggpt/Dockerfile.jetson
================================================
# NVIDIA Jetson embedded device support with GPU accelerated local model execution for https://github.com/microsoft/JARVIS
# Base image for ffmpeg build env: https://catalog.ngc.nvidia.com/orgs/nvidia/containers/l4t-jetpack/tags
FROM nvcr.io/nvidia/l4t-jetpack:r35.2.1 AS build
RUN apt update && apt install -y --no-install-recommends \
build-essential git libass-dev libx264-dev cmake && \
rm -rf /var/lib/apt/lists/*
# Build ffmpeg dependency libraries
RUN git clone https://github.com/jocover/jetson-ffmpeg.git && \
cd jetson-ffmpeg && \
sed -i 's=Libs: -L${libdir} -lnvmpi=Libs: -L${libdir} -lnvmpi -L/usr/lib/aarch64-linux-gnu/tegra -lnvbufsurface=g' nvmpi.pc.in && \
mkdir build && \
cd build && \
cmake .. && \
make -j$(nproc) && \
sudo make install && \
sudo ldconfig && \
git clone git://source.ffmpeg.org/ffmpeg.git -b release/4.2 --depth=1 && \
cd ffmpeg && \
wget https://github.com/jocover/jetson-ffmpeg/raw/master/ffmpeg_nvmpi.patch && \
git apply ffmpeg_nvmpi.patch && \
./configure --enable-nvmpi --enable-libass --enable-libx264 --enable-gpl && \
make -j$(nproc)
# Base image: https://catalog.ngc.nvidia.com/orgs/nvidia/containers/l4t-pytorch/tags
# For running JARVIS application layer
from nvcr.io/nvidia/l4t-pytorch:r35.2.1-pth2.0-py3
ENV LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH
COPY --from=build /usr/local/lib/libnvmpi.a /usr/local/lib
COPY --from=build /usr/local/lib/libnvmpi.so.1.0.0 /usr/local/lib
COPY --from=build jetson-ffmpeg/build/ffmpeg/ffmpeg /usr/local/bin
COPY --from=build jetson-ffmpeg/build/ffmpeg/ffprobe /usr/local/bin
RUN ln /usr/local/lib/libnvmpi.so.1.0.0 /usr/local/lib/libnvmpi.so
ENV MAKEFLAGS="-j$(nproc)"
COPY ./server/requirements.txt .
# Install model server dependencies
RUN apt update && apt remove -y \
opencv-dev opencv-libs opencv-licenses opencv-main opencv-python opencv-scripts python3-numpy && \
rm -rf /var/lib/apt/lists/*
RUN python3 -m pip install importlib-metadata==4.13.0 && \
python3 -m pip install -r requirements.txt && \
rm -rf requirements.txt
# Update torch deps via reinstall
RUN python3 -m pip install torch==2.0.0a0+ec3941ad.nv23.2 torchaudio==0.13.1+b90d798 torchvision==0.14.1a0+5e8e2f1
# Downgrade opencv-python to v4.5
RUN python3 -m pip install opencv-python==4.5.5.64
# Install nvidia-opencv-dev
RUN apt update && apt install -y --no-install-recommends \
nvidia-opencv-dev && \
rm -rf /var/lib/apt/lists/*
# Fix loading of scikit dep at runtime
ENV LD_PRELOAD='/usr/local/lib/python3.8/dist-packages/scikit_learn.libs/libgomp-d22c30c5.so.1.0.0'
# Install nodejs npm from nodesource
ENV NVM_DIR /root/.nvm
ENV NODE_VERSION v18.16.0
RUN wget -q -O - https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.3/install.sh | bash && \
. "$NVM_DIR/nvm.sh" && \
nvm install $NODE_VERSION && \
nvm alias default $NODE_VERSION && \
nvm use default
ENV NODE_PATH $NVM_DIR/versions/node/$NODE_VERSION/lib/node_modules
ENV PATH $NVM_DIR/versions/node/$NODE_VERSION/bin:$PATH
WORKDIR /app
# Copy source files
COPY . .
# Install web server dependencies
RUN apt update && apt install -y --no-install-recommends \
xdg-utils && \
rm -rf /var/lib/apt/lists/* && \
cd web && \
npm install
# Download local models
# RUN apt update && apt install -y --no-install-recommends \
# git-lfs && \
# rm -rf /var/lib/apt/lists/* && \
# cd server/models && \
# bash download.sh
# Expose the model server ports
EXPOSE 8004
EXPOSE 8005
# Expose the web server port
EXPOSE 9999
WORKDIR /app/server
# Start the model and web server
CMD python3 models_server.py --config configs/config.default.yaml;
================================================
FILE: hugginggpt/README.md
================================================
<p align="center">
<img src="./assets/logo.png" width="30%"> <br>
</p>
<div align="center">
<h1>HuggingGPT</h1>
<div align="center">
<a href="https://opensource.org/licenses/Apache-2.0">
<img alt="License: Apache 2.0" src="https://img.shields.io/badge/License-Apache%202.0-4E94CE.svg">
</a>
<a href="https://arxiv.org/abs/2303.17580">
<img alt="License: Apache 2.0" src="https://img.shields.io/badge/arXiv-Paper-<COLOR>.svg">
</a>
<a href="https://huggingface.co/spaces/microsoft/HuggingGPT">
<img alt="License
gitextract_ov6dal7w/
├── .gitignore
├── CITATION.cff
├── CODE_OF_CONDUCT.md
├── CONTRIBUTING.md
├── LICENSE
├── README.md
├── SECURITY.md
├── SUPPORT.md
├── easytool/
│ ├── README.md
│ ├── data_funcqa/
│ │ ├── funchub/
│ │ │ └── math.py
│ │ └── tool_instruction/
│ │ ├── functions_data.json
│ │ └── tool_dic.jsonl
│ ├── data_process.py
│ ├── data_restbench/
│ │ └── tool_instruction/
│ │ └── tmdb_tool.json
│ ├── data_toolbench/
│ │ └── tool_instruction/
│ │ └── toolbench_tool_instruction.json
│ ├── easytool/
│ │ ├── __init__.py
│ │ ├── funcQA.py
│ │ ├── restbench.py
│ │ ├── toolbench.py
│ │ ├── toolbench_retrieve.py
│ │ └── util.py
│ ├── main.py
│ └── requirements.txt
├── hugginggpt/
│ ├── .dockerignore
│ ├── Dockerfile.jetson
│ ├── README.md
│ ├── server/
│ │ ├── awesome_chat.py
│ │ ├── configs/
│ │ │ ├── config.azure.yaml
│ │ │ ├── config.default.yaml
│ │ │ ├── config.gradio.yaml
│ │ │ └── config.lite.yaml
│ │ ├── data/
│ │ │ └── p0_models.jsonl
│ │ ├── demos/
│ │ │ ├── demo_choose_model.json
│ │ │ ├── demo_parse_task.json
│ │ │ └── demo_response_results.json
│ │ ├── get_token_ids.py
│ │ ├── models/
│ │ │ ├── download.ps1
│ │ │ └── download.sh
│ │ ├── models_server.py
│ │ ├── requirements.txt
│ │ └── run_gradio_demo.py
│ └── web/
│ ├── electron/
│ │ ├── .npmrc
│ │ ├── electron-builder.yml
│ │ ├── main.js
│ │ ├── package.json
│ │ └── preload.js
│ ├── env.d.ts
│ ├── index.html
│ ├── package.json
│ ├── postcss.config.js
│ ├── src/
│ │ ├── App.vue
│ │ ├── api/
│ │ │ ├── chatgpt.ts
│ │ │ └── hugginggpt.ts
│ │ ├── assets/
│ │ │ └── tailwind.css
│ │ ├── components/
│ │ │ └── Loading.vue
│ │ ├── config/
│ │ │ └── index.ts
│ │ ├── main.ts
│ │ ├── prompt/
│ │ │ └── index.ts
│ │ ├── router/
│ │ │ └── index.ts
│ │ ├── types/
│ │ │ └── index.ts
│ │ └── views/
│ │ └── home.vue
│ ├── tailwind.config.js
│ ├── tsconfig.config.json
│ ├── tsconfig.json
│ └── vite.config.ts
└── taskbench/
├── README.md
├── batch_evaluate.sh
├── data_dailylifeapis/
│ ├── alignment_ids.json
│ ├── data.json
│ ├── graph_desc.json
│ ├── tool_desc.json
│ └── user_requests.json
├── data_engine.py
├── data_huggingface/
│ ├── alignment_ids.json
│ ├── data.json
│ ├── graph_desc.json
│ ├── tool_desc.json
│ └── user_requests.json
├── data_multimedia/
│ ├── alignment_ids.json
│ ├── data.json
│ ├── graph_desc.json
│ ├── tool_desc.json
│ └── user_requests.json
├── evaluate.py
├── format_data.py
├── generate_graph.py
├── graph_sampler.py
├── inference.py
├── requirements.txt
└── visualize_graph.py
SYMBOL INDEX (150 symbols across 19 files)
FILE: easytool/data_funcqa/funchub/math.py
function custom_round (line 5) | def custom_round(x, decimal_places=2):
function scito_decimal (line 17) | def scito_decimal(sci_str):
function normalize (line 55) | def normalize(res, round_to=2):
function add_ (line 71) | def add_(args):
function subtract_ (line 76) | def subtract_(args):
function multiply_ (line 84) | def multiply_(args):
function divide_ (line 92) | def divide_(args):
function power_ (line 100) | def power_(args):
function sqrt_ (line 108) | def sqrt_(args):
function log_ (line 113) | def log_(args):
function ln_ (line 127) | def ln_(args):
function choose_ (line 133) | def choose_(args):
function permutate_ (line 139) | def permutate_(args):
function gcd_ (line 145) | def gcd_(args):
function lcm_ (line 152) | def lcm_(args):
function remainder_ (line 159) | def remainder_(args):
FILE: easytool/data_process.py
function read_jsonline (line 14) | def read_jsonline(address):
function read_json (line 23) | def read_json(address):
function toolbench_process (line 29) | def toolbench_process(data_file, dataset):
function main (line 54) | def main():
FILE: easytool/easytool/funcQA.py
function get_last_processed_index (line 29) | def get_last_processed_index(progress_file):
function update_progress (line 39) | def update_progress(progress_file, index):
function choose_tool (line 45) | def choose_tool(question, Tool_dic, tool_used, model_name):
function task_decompose (line 89) | def task_decompose(question, Tool_dic, model_name):
function task_topology (line 129) | def task_topology(question, task_ls, model_name):
function answer_generation_direct (line 180) | def answer_generation_direct(task, model_name):
function choose_parameter (line 195) | def choose_parameter(API_instruction, api, api_dic, question, model_name):
function choose_parameter_depend (line 237) | def choose_parameter_depend(API_instruction, api, api_dic, question, mod...
function Call_function (line 281) | def Call_function(B, arg, id):
function retrieval (line 323) | def retrieval(question, Tool_dic, dataset, tool_used, ind, model_name, p...
function answer_generation (line 398) | def answer_generation(question, API_instruction, call_result, model_name):
function answer_generation_depend (line 433) | def answer_generation_depend(question, API_instruction, call_result, pre...
function answer_summarize (line 474) | def answer_summarize(question, answer_task, model_name):
function answer_check (line 491) | def answer_check(question, answer, model_name):
function task_execution_mh (line 514) | def task_execution_mh(data_type, start_index, total_files,
function task_execution_oh (line 605) | def task_execution_oh(data_type, start_index, total_files,
FILE: easytool/easytool/restbench.py
function get_last_processed_index (line 31) | def get_last_processed_index(progress_file):
function update_progress (line 41) | def update_progress(progress_file, index):
function task_decompose (line 47) | def task_decompose(question, Tool_dic, model_name):
function task_execution (line 85) | def task_execution(
FILE: easytool/easytool/toolbench.py
function get_last_processed_index (line 29) | def get_last_processed_index(progress_file):
function update_progress (line 39) | def update_progress(progress_file, index):
function choose_tool (line 45) | def choose_tool(question, Tool_dic, tool_used, model_name):
function choose_API (line 85) | def choose_API(API_instruction, API_list, question, model_name):
function choose_parameter (line 140) | def choose_parameter(API_instruction, api, api_dic, question, model_name):
function choose_parameter_depend (line 182) | def choose_parameter_depend(API_instruction, api, api_dic, question, pre...
function answer_generation (line 226) | def answer_generation(question, API_instruction, call_result, model_name):
function answer_generation_depend (line 258) | def answer_generation_depend(question, API_instruction, call_result, mod...
function answer_check (line 296) | def answer_check(question, answer, model_name):
function Call_function (line 319) | def Call_function(A, B, arg, index, id):
function retrieval (line 370) | def retrieval(question, Tool_dic, dataset, tool_used, ind, model_name, i...
function task_decompose (line 459) | def task_decompose(question, model_name):
function task_topology (line 494) | def task_topology(question, task_ls, model_name):
function answer_summarize (line 545) | def answer_summarize(question, answer_task, model_name):
function answer_generation_direct (line 562) | def answer_generation_direct(task, model_name):
function tool_check (line 577) | def tool_check(task, model_name):
function task_execution (line 618) | def task_execution(data_type,
FILE: easytool/easytool/toolbench_retrieve.py
function get_last_processed_index (line 29) | def get_last_processed_index(progress_file):
function update_progress (line 39) | def update_progress(progress_file, index):
function get_embedding (line 45) | def get_embedding(text):
function retrieve_reference (line 53) | def retrieve_reference(embedded_texts, filenames, question, k):
function choose_tool (line 61) | def choose_tool(question, Tool_dic, tool_used, model_name):
function choose_API (line 101) | def choose_API(API_instruction, API_list, question, model_name):
function choose_parameter (line 155) | def choose_parameter(API_instruction, api, api_dic, question, model_name):
function choose_parameter_depend (line 197) | def choose_parameter_depend(API_instruction, api, api_dic, question, pre...
function answer_generation (line 241) | def answer_generation(question, API_instruction, call_result, model_name):
function answer_generation_depend (line 273) | def answer_generation_depend(question, API_instruction, call_result, mod...
function answer_check (line 311) | def answer_check(question, answer, model_name):
function Call_function (line 334) | def Call_function(A, B, arg, index, id):
function retrieval (line 385) | def retrieval(question, Tool_dic, dataset, tool_used, ind, model_name, i...
function task_decompose (line 474) | def task_decompose(question, model_name):
function task_topology (line 509) | def task_topology(question, task_ls, model_name):
function answer_summarize (line 560) | def answer_summarize(question, answer_task, model_name):
function answer_generation_direct (line 577) | def answer_generation_direct(task, model_name):
function tool_check (line 592) | def tool_check(task, model_name):
function task_execution (line 633) | def task_execution(data_type,
FILE: easytool/easytool/util.py
function read_jsonline (line 7) | def read_jsonline(address):
function save_json (line 16) | def save_json(ls, address):
function read_json (line 22) | def read_json(address):
function remove_key (line 28) | def remove_key(item, key_to_remove):
function data_clean (line 40) | def data_clean(dic, key):
function lowercase_parameter_keys (line 45) | def lowercase_parameter_keys(input_dict):
function build_index (line 52) | def build_index(base_path):
function change_name (line 62) | def change_name(name):
function standardize (line 69) | def standardize(string):
function get_last_processed_index (line 92) | def get_last_processed_index(progress_file):
function update_progress (line 102) | def update_progress(progress_file, index):
FILE: hugginggpt/server/awesome_chat.py
function convert_chat_to_completion (line 168) | def convert_chat_to_completion(data):
function send_request (line 189) | def send_request(data):
function replace_slot (line 215) | def replace_slot(text, entries):
function find_json (line 222) | def find_json(s):
function field_extract (line 230) | def field_extract(s, field):
function get_id_reason (line 239) | def get_id_reason(choose_str):
function record_case (line 245) | def record_case(success, **args):
function image_to_bytes (line 254) | def image_to_bytes(img_url):
function resource_has_dep (line 261) | def resource_has_dep(command):
function fix_dep (line 268) | def fix_dep(tasks):
function unfold (line 281) | def unfold(tasks):
function chitchat (line 307) | def chitchat(messages, api_key, api_type, api_endpoint):
function parse_task (line 317) | def parse_task(context, input, api_key, api_type, api_endpoint):
function choose_model (line 350) | def choose_model(input, task, metas, api_key, api_type, api_endpoint):
function response_results (line 377) | def response_results(input, results, api_key, api_type, api_endpoint):
function huggingface_model_inference (line 400) | def huggingface_model_inference(model_id, data, task):
function local_model_inference (line 532) | def local_model_inference(model_id, data, task):
function model_inference (line 638) | def model_inference(model_id, data, hosted_on, task):
function get_model_status (line 663) | def get_model_status(model_id, url, headers, queue = None):
function get_avaliable_models (line 678) | def get_avaliable_models(candidates, topk=5):
function collect_result (line 712) | def collect_result(command, choose, inference_result):
function run_task (line 720) | def run_task(input, command, results, api_key, api_type, api_endpoint):
function chat_huggingface (line 891) | def chat_huggingface(messages, api_key, api_type, api_endpoint, return_p...
function test (line 978) | def test():
function cli (line 1001) | def cli():
function server (line 1014) | def server():
FILE: hugginggpt/server/get_token_ids.py
function count_tokens (line 37) | def count_tokens(model_name, text):
function get_max_context_length (line 40) | def get_max_context_length(model_name):
function get_token_ids_for_task_parsing (line 43) | def get_token_ids_for_task_parsing(model_name):
function get_token_ids_for_choose_model (line 49) | def get_token_ids_for_choose_model(model_name):
FILE: hugginggpt/server/models_server.py
function load_pipes (line 81) | def load_pipes(local_deployment):
function running (line 351) | def running():
function status (line 355) | def status(model_id):
function models (line 365) | def models(model_id):
FILE: hugginggpt/server/run_gradio_demo.py
function add_message (line 11) | def add_message(content, role):
function extract_medias (line 15) | def extract_medias(message):
function set_openai_key (line 36) | def set_openai_key(openai_key):
function add_text (line 41) | def add_text(messages, message):
function bot (line 75) | def bot(messages):
FILE: hugginggpt/web/electron/main.js
function createWindow (line 5) | function createWindow () {
FILE: hugginggpt/web/src/api/chatgpt.ts
function chatgpt (line 7) | async function chatgpt(messageList: CleanChatMessage[], apiKey: string) {
FILE: hugginggpt/web/src/api/hugginggpt.ts
function hugginggpt (line 9) | async function hugginggpt(messageList: CleanChatMessage[]) {
FILE: hugginggpt/web/src/config/index.ts
constant HUGGINGGPT_BASE_URL (line 1) | const HUGGINGGPT_BASE_URL = "http://localhost:8004"
constant CHAT_GPT_URL (line 4) | const CHAT_GPT_URL = "https://api.openai.com"
constant CHAT_GPT_LLM (line 5) | const CHAT_GPT_LLM = "gpt-3.5-turbo" // gpt-3.5-turbo, gpt-4
FILE: hugginggpt/web/src/types/index.ts
type ChatMessage (line 1) | interface ChatMessage {
type CleanChatMessage (line 8) | interface CleanChatMessage {
type Collection (line 13) | interface Collection {
FILE: taskbench/evaluate.py
function sim (line 20) | def sim(name_1, name_2):
function create_cost_matrix (line 25) | def create_cost_matrix(graph_1, graph_2):
function compute_assignment_matrix (line 55) | def compute_assignment_matrix(graph_1, graph_2):
function matching (line 60) | def matching(graph_1, graph_2):
function ratio_levenshtein (line 64) | def ratio_levenshtein(x, y):
function flatten (line 73) | def flatten(gt, pred, types = None):
function print_results (line 112) | def print_results(per_type, micro, macro, types, result_dict = None):
function get_row (line 142) | def get_row(data, label):
function get_content_type (line 149) | def get_content_type(content):
function main (line 178) | def main(data_dir, prediction_dir, save_dir, splits, n_tools, mode, metr...
function evaluate (line 255) | def evaluate(data_dir, prediction_dir, llm, split, n_tool, metric, tool_...
FILE: taskbench/generate_graph.py
function generate_graph_resource (line 4) | def generate_graph_resource(tool_file):
function generate_graph_temporal (line 22) | def generate_graph_temporal(tool_file):
function generate_graph (line 49) | def generate_graph(tool_desc, data_dir, dependency_type):
FILE: taskbench/visualize_graph.py
function visialize_graph (line 8) | def visialize_graph(data_dir):
Copy disabled (too large)
Download .json
Condensed preview — 90 files, each showing path, character count, and a content snippet. Download the .json file for the full structured content (19,244K chars).
[
{
"path": ".gitignore",
"chars": 2387,
"preview": "# ALL\n*.dev.yaml\n\nlegacy/\n\n# for server\nserver/models/*\n!server/models/download.sh\n!server/models/download.ps1\nserver/lo"
},
{
"path": "CITATION.cff",
"chars": 1129,
"preview": "cff-version: 1.2.0\nmessage: \"If you find this work useful in your method, you can cite the paper as below.\"\nauthors:\n -"
},
{
"path": "CODE_OF_CONDUCT.md",
"chars": 444,
"preview": "# Microsoft Open Source Code of Conduct\n\nThis project has adopted the [Microsoft Open Source Code of Conduct](https://op"
},
{
"path": "CONTRIBUTING.md",
"chars": 1693,
"preview": "\nTo contribute to this GitHub project, you can follow these steps:\n\n1. Fork the repository you want to contribute to by "
},
{
"path": "LICENSE",
"chars": 1141,
"preview": " MIT License\n\n Copyright (c) Microsoft Corporation.\n\n Permission is hereby granted, free of charge, to any pers"
},
{
"path": "README.md",
"chars": 14987,
"preview": "# JARVIS\n\n\n[](https://arxiv.org/abs/2303.17580)\n[![Open in"
},
{
"path": "SECURITY.md",
"chars": 2757,
"preview": "<!-- BEGIN MICROSOFT SECURITY.MD V0.0.8 BLOCK -->\n\n## Security\n\nMicrosoft takes the security of our software products an"
},
{
"path": "SUPPORT.md",
"chars": 1244,
"preview": "# TODO: The maintainer of this repo has not yet edited this file\r\n\r\n**REPO OWNER**: Do you want Customer Service & Suppo"
},
{
"path": "easytool/README.md",
"chars": 3701,
"preview": "<p align=\"center\">\r\n<img src=\"./assets/logo.png\" width=\"15%\"> <br>\r\n</p>\r\n\r\n<div align=\"center\">\r\n<h1>EasyTool</h1>\r\n<h3"
},
{
"path": "easytool/data_funcqa/funchub/math.py",
"chars": 4203,
"preview": "import math\n\n# this function is used to round the result to 2 decimal places\n# e.g. 52.3523 -> 52.35, 52.0011 -> 52, 0.0"
},
{
"path": "easytool/data_funcqa/tool_instruction/functions_data.json",
"chars": 7924,
"preview": "{\n \"0\":{\n \"ID\":0,\n \"standardized_name\":\"add_\",\n \"API_description\":\"'add_' returns the sum of all the a"
},
{
"path": "easytool/data_funcqa/tool_instruction/tool_dic.jsonl",
"chars": 1974,
"preview": "{\"ID\": 0, \"description\": \"'add_' returns the sum of all the arguments passed to it, normalized to 2 decimal places.\"}\r\n{"
},
{
"path": "easytool/data_process.py",
"chars": 4162,
"preview": "import os\r\nimport gdown\r\nimport shutil\r\nimport json\r\nfrom zipfile import ZipFile\r\n\r\nurls = {\r\n \"funcqa\": \"https://dri"
},
{
"path": "easytool/data_restbench/tool_instruction/tmdb_tool.json",
"chars": 26392,
"preview": "[\r\n {\r\n \"ID\": 1,\r\n \"tool_name\": \"/movie/{movie_id}/keywords\",\r\n \"tool_description\": \"Get the key"
},
{
"path": "easytool/easytool/__init__.py",
"chars": 13,
"preview": "# __init__.py"
},
{
"path": "easytool/easytool/funcQA.py",
"chars": 32883,
"preview": "# — coding: utf-8 –\r\nimport openai\r\nimport json\r\nimport logging\r\nimport sys\r\nimport argparse\r\nfrom langchain.chat_models"
},
{
"path": "easytool/easytool/restbench.py",
"chars": 4532,
"preview": "# — coding: utf-8 –\r\nimport openai\r\nimport json\r\nimport logging\r\nimport sys\r\nimport argparse\r\nimport ast\r\nfrom langchain"
},
{
"path": "easytool/easytool/toolbench.py",
"chars": 36348,
"preview": "# — coding: utf-8 –\r\nimport openai\r\nimport json\r\nimport logging\r\nimport sys\r\nimport argparse\r\nfrom langchain.chat_models"
},
{
"path": "easytool/easytool/toolbench_retrieve.py",
"chars": 37193,
"preview": "# — coding: utf-8 –\r\nimport openai\r\nimport json\r\nimport logging\r\nimport sys\r\nimport argparse\r\nfrom langchain.chat_models"
},
{
"path": "easytool/easytool/util.py",
"chars": 3174,
"preview": "# — coding: utf-8 –\r\nimport json\r\nimport re\r\nimport os\r\n\r\n\r\ndef read_jsonline(address):\r\n not_mark = []\r\n with ope"
},
{
"path": "easytool/main.py",
"chars": 3786,
"preview": "# — coding: utf-8 –\r\nimport openai\r\nimport json\r\nimport argparse\r\nimport os\r\nfrom tqdm import tqdm\r\nfrom easytool import"
},
{
"path": "easytool/requirements.txt",
"chars": 102,
"preview": "openai==0.27.8\nlangchain==0.0.260\ngdown==4.6.0\ntqdm\nargparse\nnumpy\nrequests\npickle-mixin\nscikit-learn\n"
},
{
"path": "hugginggpt/.dockerignore",
"chars": 75,
"preview": ".git\nserver/models/*\n!server/models/download.sh\n!server/models/download.ps1"
},
{
"path": "hugginggpt/Dockerfile.jetson",
"chars": 3704,
"preview": "# NVIDIA Jetson embedded device support with GPU accelerated local model execution for https://github.com/microsoft/JARV"
},
{
"path": "hugginggpt/README.md",
"chars": 14471,
"preview": "<p align=\"center\">\n<img src=\"./assets/logo.png\" width=\"30%\"> <br>\n</p>\n\n<div align=\"center\">\n<h1>HuggingGPT</h1>\n <div "
},
{
"path": "hugginggpt/server/awesome_chat.py",
"chars": 45647,
"preview": "import base64\nimport copy\nfrom io import BytesIO\nimport io\nimport os\nimport random\nimport time\nimport traceback\nimport u"
},
{
"path": "hugginggpt/server/configs/config.azure.yaml",
"chars": 4635,
"preview": "azure:\n api_key: REPLACE_WITH_YOUR_AZURE_API_KEY_HERE\n base_url: REPLACE_WITH_YOUR_ENDPOINT_HERE\n deployment_name: RE"
},
{
"path": "hugginggpt/server/configs/config.default.yaml",
"chars": 4698,
"preview": "openai: \n api_key: REPLACE_WITH_YOUR_OPENAI_API_KEY_HERE\n# azure:\n# api_key: REPLACE_WITH_YOUR_AZURE_API_KEY_HERE\n# "
},
{
"path": "hugginggpt/server/configs/config.gradio.yaml",
"chars": 4273,
"preview": "huggingface:\n token: REPLACE_WITH_YOUR_HUGGINGFACE_TOKEN_HERE # required: huggingface token @ https://huggingface.co/se"
},
{
"path": "hugginggpt/server/configs/config.lite.yaml",
"chars": 4423,
"preview": "openai: \n api_key: REPLACE_WITH_YOUR_OPENAI_API_KEY_HERE\nhuggingface:\n token: REPLACE_WITH_YOUR_HUGGINGFACE_TOKEN_HERE"
},
{
"path": "hugginggpt/server/data/p0_models.jsonl",
"chars": 2765223,
"preview": "{\"downloads\": 1677372, \"id\": \"ProsusAI/finbert\", \"likes\": 186, \"pipeline_tag\": \"text-classification\", \"task\": \"text-clas"
},
{
"path": "hugginggpt/server/demos/demo_choose_model.json",
"chars": 142,
"preview": "[\n {\n \"role\": \"user\",\n \"content\": \"{{input}}\"\n },\n {\n \"role\": \"assistant\",\n \"conten"
},
{
"path": "hugginggpt/server/demos/demo_parse_task.json",
"chars": 4269,
"preview": "[\n {\n \"role\": \"user\",\n \"content\": \"Give you some pictures e1.jpg, e2.png, e3.jpg, help me count the num"
},
{
"path": "hugginggpt/server/demos/demo_response_results.json",
"chars": 316,
"preview": "[\n {\n \"role\": \"user\",\n \"content\": \"{{input}}\"\n },\n {\n \"role\": \"assistant\",\n \"conten"
},
{
"path": "hugginggpt/server/get_token_ids.py",
"chars": 2474,
"preview": "import tiktoken\n\nencodings = {\n \"gpt-4\": tiktoken.get_encoding(\"cl100k_base\"),\n \"gpt-4-32k\": tiktoken.get_encoding"
},
{
"path": "hugginggpt/server/models/download.ps1",
"chars": 1824,
"preview": "$models = @(\r\n \"nlpconnect/vit-gpt2-image-captioning\",\r\n \"lllyasviel/ControlNet\",\r\n \"lllyasviel/sd-controlnet-c"
},
{
"path": "hugginggpt/server/models/download.sh",
"chars": 1743,
"preview": "#!/bin/bash\n\n# Set models and datasets to download\nmodels=(\n \"nlpconnect/vit-gpt2-image-captioning\"\n \"lllyasviel/C"
},
{
"path": "hugginggpt/server/models_server.py",
"chars": 31358,
"preview": "import argparse\nimport logging\nimport random\nimport uuid\nimport numpy as np\nfrom transformers import pipeline\nfrom diffu"
},
{
"path": "hugginggpt/server/requirements.txt",
"chars": 579,
"preview": "git+https://github.com/huggingface/diffusers.git@8c530fc2f6a76a2aefb6b285dce6df1675092ac6#egg=diffusers\ngit+https://gith"
},
{
"path": "hugginggpt/server/run_gradio_demo.py",
"chars": 5117,
"preview": "import uuid\nimport gradio as gr\nimport re\nfrom diffusers.utils import load_image\nimport requests\nfrom awesome_chat impor"
},
{
"path": "hugginggpt/web/electron/.npmrc",
"chars": 159,
"preview": "registry=https://registry.npmmirror.com\nelectron_mirror=https://npmmirror.com/mirrors/electron/\nchromedriver_cdnurl=http"
},
{
"path": "hugginggpt/web/electron/electron-builder.yml",
"chars": 281,
"preview": "appId: com.chatgpt.app\nproductName: ChatGPT\ndirectories:\n output: ../electron-dist\nmac:\n category: public.app-category"
},
{
"path": "hugginggpt/web/electron/main.js",
"chars": 1398,
"preview": "// Modules to control application life and create native browser window\nconst {app, BrowserWindow} = require('electron')"
},
{
"path": "hugginggpt/web/electron/package.json",
"chars": 232,
"preview": "{\n \"name\": \"chatgpt\",\n \"version\": \"1.0.0\",\n \"main\": \"main.js\",\n \"scripts\": {\n \"dev\": \"electron .\",\n \"build\": \""
},
{
"path": "hugginggpt/web/electron/preload.js",
"chars": 542,
"preview": "/**\n * The preload script runs before. It has access to web APIs\n * as well as Electron's renderer process modules and s"
},
{
"path": "hugginggpt/web/env.d.ts",
"chars": 38,
"preview": "/// <reference types=\"vite/client\" />\n"
},
{
"path": "hugginggpt/web/index.html",
"chars": 342,
"preview": "<!DOCTYPE html>\n<html lang=\"zh-CN\">\n <head>\n <meta charset=\"UTF-8\" />\n <link rel=\"icon\" href=\"/favicon.ico\" />\n "
},
{
"path": "hugginggpt/web/package.json",
"chars": 874,
"preview": "{\n \"name\": \"vue3-ts-vite-router-tailwindcss\",\n \"version\": \"0.0.0\",\n \"main\": \"index.html\",\n \"scripts\": {\n \"dev\": \""
},
{
"path": "hugginggpt/web/postcss.config.js",
"chars": 82,
"preview": "module.exports = {\n plugins: {\n tailwindcss: {},\n autoprefixer: {},\n },\n}\n"
},
{
"path": "hugginggpt/web/src/App.vue",
"chars": 122,
"preview": "<script setup lang=\"ts\">\n</script>\n\n<template>\n <Suspense>\n <RouterView />\n </Suspense>\n</template>\n\n<style></style"
},
{
"path": "hugginggpt/web/src/api/chatgpt.ts",
"chars": 840,
"preview": "import type { CleanChatMessage } from \"@/types\";\nimport axios, { AxiosError } from \"axios\";\nimport { CHAT_GPT_URL, CHAT_"
},
{
"path": "hugginggpt/web/src/api/hugginggpt.ts",
"chars": 753,
"preview": "import type { CleanChatMessage } from \"@/types\";\nimport axios, { AxiosError } from \"axios\";\nimport { HUGGINGGPT_BASE_URL"
},
{
"path": "hugginggpt/web/src/assets/tailwind.css",
"chars": 526,
"preview": "@tailwind base;\n@tailwind components;\n@tailwind utilities;\n\n@layer components {\n .btn {\n @apply px-4 py-2 text-sm fo"
},
{
"path": "hugginggpt/web/src/components/Loading.vue",
"chars": 1704,
"preview": "<template>\n <div class=\"com__box\">\n <div class=\"loading\">\n <div></div>\n <div></div>\n <div></div>\n "
},
{
"path": "hugginggpt/web/src/config/index.ts",
"chars": 338,
"preview": "const HUGGINGGPT_BASE_URL = \"http://localhost:8004\"\n\n// use ChatGPT: double click on the setting icon\nconst CHAT_GPT_URL"
},
{
"path": "hugginggpt/web/src/main.ts",
"chars": 186,
"preview": "import { createApp } from \"vue\";\nimport App from \"./App.vue\";\nimport router from \"./router\";\nimport \"./assets/tailwind.c"
},
{
"path": "hugginggpt/web/src/prompt/index.ts",
"chars": 3516,
"preview": "import type {Collection, ChatMessage } from \"@/types\";\n\nconst ChatGPTTerminalMessage:ChatMessage[] = [\n {\n role: \"as"
},
{
"path": "hugginggpt/web/src/router/index.ts",
"chars": 301,
"preview": "import { createRouter, createWebHashHistory } from \"vue-router\";\n\nconst router = createRouter({\n history: createWebHash"
},
{
"path": "hugginggpt/web/src/types/index.ts",
"chars": 415,
"preview": "export interface ChatMessage {\n role: \"user\" | \"assistant\" | \"system\";\n type: \"text\" | \"image\" | \"audio\" | \"video\""
},
{
"path": "hugginggpt/web/src/views/home.vue",
"chars": 15066,
"preview": "<script setup lang=\"ts\">\nimport type { ChatMessage, CleanChatMessage } from \"@/types\";\nimport { ref, watch, nextTick, on"
},
{
"path": "hugginggpt/web/tailwind.config.js",
"chars": 175,
"preview": "/** @type {import('tailwindcss').Config} */\nmodule.exports = {\n content: [\"./index.html\", \"./src/**/*.{vue,js,ts,jsx,ts"
},
{
"path": "hugginggpt/web/tsconfig.config.json",
"chars": 196,
"preview": "{\n \"extends\": \"@vue/tsconfig/tsconfig.node.json\",\n \"include\": [\"vite.config.*\", \"vitest.config.*\", \"cypress.config.*\"]"
},
{
"path": "hugginggpt/web/tsconfig.json",
"chars": 306,
"preview": "{\n \"extends\": \"@vue/tsconfig/tsconfig.web.json\",\n \"include\": [\"env.d.ts\", \"src/**/*\", \"src/**/*.vue\"],\n \"compilerOpti"
},
{
"path": "hugginggpt/web/vite.config.ts",
"chars": 421,
"preview": "import { fileURLToPath, URL } from \"node:url\";\nimport path from \"path\";\nimport { defineConfig } from \"vite\";\nimport vue"
},
{
"path": "taskbench/README.md",
"chars": 19311,
"preview": "<p align=\"center\">\n<img src=\"./assets/logo2.png\" width=\"20%\"> <br>\n</p>\n\n<div align=\"center\">\n<h1>TaskBench</h1>\n <div "
},
{
"path": "taskbench/batch_evaluate.sh",
"chars": 786,
"preview": "#!/bin/bash\nset -x\nset -e\n\nexport CUDA_VISIBLE_DEVICES=0\n\ndata_dir=$1\nprediction_dir=$2\n\nif [[ $data_dir == *\"dailylifea"
},
{
"path": "taskbench/data_dailylifeapis/alignment_ids.json",
"chars": 549112,
"preview": "{\n \"node_alignment_id\": {\n \"single\": [\n \"13590101\",\n \"29497210\",\n \"43154691\",\n \"28058748\",\n "
},
{
"path": "taskbench/data_dailylifeapis/data.json",
"chars": 9463811,
"preview": "{\"id\": \"77532649\", \"seed\": 813407, \"n_tools\": 1, \"type\": \"single\", \"sampled_nodes\": \"[{\\\"task\\\": \\\"play_movie_by_title\\\""
},
{
"path": "taskbench/data_dailylifeapis/graph_desc.json",
"chars": 184749,
"preview": "{\n \"nodes\": [\n {\n \"id\": \"get_weather\",\n \"desc\": \"Get the weather for a specific city and a specific day\",\n"
},
{
"path": "taskbench/data_dailylifeapis/tool_desc.json",
"chars": 14069,
"preview": "{\n \"nodes\": [\n {\n \"id\": \"get_weather\",\n \"desc\": \"Get the weather for a specific city and a specific day\",\n"
},
{
"path": "taskbench/data_dailylifeapis/user_requests.json",
"chars": 937473,
"preview": "{\"id\": \"13590101\", \"user_request\": \"I want to watch the movie titled 'Example Movie'\"}\n{\"id\": \"29497210\", \"user_request\""
},
{
"path": "taskbench/data_huggingface/graph_desc.json",
"chars": 32444,
"preview": "{\n \"nodes\": [\n {\n \"id\": \"Token Classification\",\n \"desc\": \"Token classification is a natural language under"
},
{
"path": "taskbench/data_huggingface/tool_desc.json",
"chars": 7878,
"preview": "{\n \"nodes\": [\n {\n \"id\": \"Token Classification\",\n \"desc\": \"Token classification is a natural language under"
},
{
"path": "taskbench/data_huggingface/user_requests.json",
"chars": 1932292,
"preview": "{\"id\": \"57993067\", \"user_request\": \"I need to identify and label objects in the provided image 'example.jpg'.\"}\n{\"id\": \""
},
{
"path": "taskbench/data_multimedia/alignment_ids.json",
"chars": 659290,
"preview": "{\n \"node_alignment_id\": {\n \"single\": [\n \"30934207\",\n \"40313104\",\n \"31733796\",\n \"26103736\",\n "
},
{
"path": "taskbench/data_multimedia/graph_desc.json",
"chars": 56093,
"preview": "{\n \"nodes\": [\n {\n \"id\": \"Image Downloader\",\n \"desc\": \"Downloads an image from a given URL.\",\n \"input-"
},
{
"path": "taskbench/data_multimedia/tool_desc.json",
"chars": 9043,
"preview": "{\n \"nodes\": [\n {\n \"id\": \"Image Downloader\",\n \"desc\": \"Downloads an image from a given URL.\",\n \"input-"
},
{
"path": "taskbench/evaluate.py",
"chars": 23540,
"preview": "import traceback\nimport numpy as np\nfrom scipy.optimize import linear_sum_assignment\nimport json\nimport click\nfrom datas"
},
{
"path": "taskbench/generate_graph.py",
"chars": 2702,
"preview": "import json\nimport click\n\ndef generate_graph_resource(tool_file):\n with open(tool_file) as f:\n data = json.loa"
},
{
"path": "taskbench/requirements.txt",
"chars": 190,
"preview": "requests==2.28.1\nclick==8.0.4\nemoji==2.8.0\nnetworkx==2.8.4\naiohttp==3.8.1\nmatplotlib==3.7.1\npandas==1.2.4\nnumpy==1.23.5\n"
},
{
"path": "taskbench/visualize_graph.py",
"chars": 998,
"preview": "import json\nimport networkx as nx\nimport matplotlib.pyplot as plt\nimport click\n\n@click.command()\n@click.option('--data_d"
}
]
// ... and 9 more files (download for full content)
About this extraction
This page contains the full source code of the microsoft/JARVIS GitHub repository, extracted and formatted as plain text for AI agents and large language models (LLMs). The extraction includes 90 files (47.5 MB), approximately 4.3M tokens, and a symbol index with 150 extracted functions, classes, methods, constants, and types. Use this with OpenClaw, Claude, ChatGPT, Cursor, Windsurf, or any other AI tool that accepts text input. You can copy the full output to your clipboard or download it as a .txt file.
Extracted by GitExtract — free GitHub repo to text converter for AI. Built by Nikandr Surkov.