[ { "path": ".github/FUNDING.yml", "content": "# These are supported funding model platforms\n\ngithub: omarsar # Replace with up to 4 GitHub Sponsors-enabled usernames e.g., [user1, user2]\npatreon: # Replace with a single Patreon username\nopen_collective: # Replace with a single Open Collective username\nko_fi: # Replace with a single Ko-fi username\ntidelift: # Replace with a single Tidelift platform-name/package-name e.g., npm/babel\ncommunity_bridge: # Replace with a single Community Bridge project-name e.g., cloud-foundry\nliberapay: # Replace with a single Liberapay username\nissuehunt: # Replace with a single IssueHunt username\notechie: # Replace with a single Otechie username\nlfx_crowdfunding: # Replace with a single LFX Crowdfunding project-name e.g., cloud-foundry\ncustom: # Replace with up to 4 custom sponsorship URLs e.g., ['link1', 'link2']\n" }, { "path": ".github/workflows/claude-code-review.yml", "content": "name: Claude Code Review\n\non:\n pull_request:\n types: [opened, synchronize]\n # Optional: Only run on specific file changes\n # paths:\n # - \"src/**/*.ts\"\n # - \"src/**/*.tsx\"\n # - \"src/**/*.js\"\n # - \"src/**/*.jsx\"\n\njobs:\n claude-review:\n # Optional: Filter by PR author\n # if: |\n # github.event.pull_request.user.login == 'external-contributor' ||\n # github.event.pull_request.user.login == 'new-developer' ||\n # github.event.pull_request.author_association == 'FIRST_TIME_CONTRIBUTOR'\n \n runs-on: ubuntu-latest\n permissions:\n contents: read\n pull-requests: read\n issues: read\n id-token: write\n \n steps:\n - name: Checkout repository\n uses: actions/checkout@v4\n with:\n fetch-depth: 1\n\n - name: Run Claude Code Review\n id: claude-review\n uses: anthropics/claude-code-action@v1\n with:\n claude_code_oauth_token: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}\n prompt: |\n Please review this pull request and provide feedback on:\n - Code quality and best practices\n - Potential bugs or issues\n - Performance considerations\n - Security concerns\n - Test coverage\n \n Use the repository's CLAUDE.md for guidance on style and conventions. Be constructive and helpful in your feedback.\n\n Use `gh pr comment` with your Bash tool to leave your review as a comment on the PR.\n \n # See https://github.com/anthropics/claude-code-action/blob/main/docs/usage.md\n # or https://docs.anthropic.com/en/docs/claude-code/sdk#command-line for available options\n claude_args: '--allowed-tools \"Bash(gh issue view:*),Bash(gh search:*),Bash(gh issue list:*),Bash(gh pr comment:*),Bash(gh pr diff:*),Bash(gh pr view:*),Bash(gh pr list:*)\"'\n\n" }, { "path": ".github/workflows/claude.yml", "content": "name: Claude Code\n\non:\n issue_comment:\n types: [created]\n pull_request_review_comment:\n types: [created]\n issues:\n types: [opened, assigned]\n pull_request_review:\n types: [submitted]\n\njobs:\n claude:\n if: |\n (github.event_name == 'issue_comment' && contains(github.event.comment.body, '@claude')) ||\n (github.event_name == 'pull_request_review_comment' && contains(github.event.comment.body, '@claude')) ||\n (github.event_name == 'pull_request_review' && contains(github.event.review.body, '@claude')) ||\n (github.event_name == 'issues' && (contains(github.event.issue.body, '@claude') || contains(github.event.issue.title, '@claude')))\n runs-on: ubuntu-latest\n permissions:\n contents: read\n pull-requests: read\n issues: read\n id-token: write\n actions: read # Required for Claude to read CI results on PRs\n steps:\n - name: Checkout repository\n uses: actions/checkout@v4\n with:\n fetch-depth: 1\n\n - name: Run Claude Code\n id: claude\n uses: anthropics/claude-code-action@v1\n with:\n claude_code_oauth_token: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}\n \n # This is an optional setting that allows Claude to read CI results on PRs\n additional_permissions: |\n actions: read\n\n # Optional: Give a custom prompt to Claude. If this is not specified, Claude will perform the instructions specified in the comment that tagged it.\n # prompt: 'Update the pull request description to include a summary of changes.'\n\n # Optional: Add claude_args to customize behavior and configuration\n # See https://github.com/anthropics/claude-code-action/blob/main/docs/usage.md\n # or https://docs.anthropic.com/en/docs/claude-code/sdk#command-line for available options\n # claude_args: '--model claude-opus-4-1-20250805 --allowed-tools Bash(gh pr:*)'\n\n" }, { "path": ".gitignore", "content": ".ipynb_checkpoints/\n__pycache__/\n*.swp\n.env\n.pylintrc\n*.egg-info/\nnotebooks/.ipynb_checkpoints/\nnotebooks/__pycache__/\nnotebooks/state_of_the_union.txt\nnotebooks/chroma_logs.log\nnotebooks/.chroma/\nnotebooks/local_notebooks/\nnotebooks/.env\npages/research/local_research/\n.DS_Store\n.vscode\nagent.toml\ncontent_suggestions/\n.claude/\n.qodo/\nPrompt-Engineering-Guide-notes/\n\n# app\n.next\nnode_modules\nprompts\n\n#mcp\n.mcp.json\n.env.local" }, { "path": "CITATION.cff", "content": "cff-version: 1.2.0\nmessage: \"If you use this software, please cite it as below.\"\nauthors:\n- family-names: \"Saravia\"\n given-names: \"Elvis\"\ntitle: \"Prompt Engineering Guide\"\ndate-released: 2022-12-16\nurl: \"https://github.com/dair-ai/Prompt-Engineering-Guide\"\npreferred-citation:\n type: article\n authors:\n - family-names: \"Saravia\"\n given-names: \"Elvis\"\n month: 12\n journal: \"https://github.com/dair-ai/Prompt-Engineering-Guide\"\n title: \"Prompt Engineering Guide\"\n year: 2022\n" }, { "path": "CLAUDE.md", "content": "# Prompt Engineering Guide\n\n## Project Overview\nOpen-source prompt engineering guide at promptingguide.ai. Built with Next.js (Nextra theme).\n\n## Project Learnings\nRead and follow the project learnings in `Prompt-Engineering-Guide-notes/learnings/` before starting work. These are distilled from past session corrections and contain important patterns for this project.\n\n## Notes & Content\n- All project notes go in `Prompt-Engineering-Guide-notes/` (Obsidian vault inside the project root)\n\n## Git Workflow\n- Do NOT push to remote unless explicitly asked by the user\n- Commits are fine, but always wait for user to request push\n" }, { "path": "LICENSE.md", "content": "MIT License\n\nCopyright (c) 2022 DAIR.AI\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\nSOFTWARE.\n" }, { "path": "README.md", "content": "# Prompt Engineering Guide\n\n
\n Sponsored by    \n
\n\nPrompt engineering is a relatively new discipline for developing and optimizing prompts to efficiently use language models (LMs) for a wide variety of applications and research topics. Prompt engineering skills help to better understand the capabilities and limitations of large language models (LLMs). Researchers use prompt engineering to improve the capacity of LLMs on a wide range of common and complex tasks such as question answering and arithmetic reasoning. Developers use prompt engineering to design robust and effective prompting techniques that interface with LLMs and other tools.\n\nMotivated by the high interest in developing with LLMs, we have created this new prompt engineering guide that contains all the latest papers, learning guides, lectures, references, and tools related to prompt engineering for LLMs.\n\n🌐 [Prompt Engineering Guide (Web Version)](https://www.promptingguide.ai/)\n\n🎉 We are excited to launch our new prompt engineering, RAG, and AI Agents courses under the DAIR.AI Academy. [Join Now](https://academy.dair.ai/pricing)!\n\nThe courses are meant to compliment this guide and provide a more hands-on approach to learning about prompt engineering, context engineering, and AI Agents. \n\nUse code PROMPTING20 to get an extra 20% off.\n\nHappy Prompting!\n\n---\n## Announcements / Updates\n\n- 🎓 We now offer self-paced prompt engineering courses under our DAIR.AI Academy. [Join Now](https://academy.dair.ai/pricing)! \n- 🎓 New course on Prompt Engineering for LLMs announced! [Enroll here](https://academy.dair.ai/courses/introduction-prompt-engineering)!\n- 💼 We now offer several [services](https://www.promptingguide.ai/services) like corporate training, consulting, and talks.\n- 🌐 We now support 13 languages! Welcoming more translations.\n- 👩‍🎓 We crossed 3 million learners in January 2024!\n- 🎉 We have launched a new web version of the guide [here](https://www.promptingguide.ai/)\n- 🔥 We reached #1 on Hacker News on 21 Feb 2023\n- 🎉 The First Prompt Engineering Lecture went live [here](https://youtu.be/dOxUroR57xs)\n\n[Join our Discord](https://discord.gg/YbMT8k6GfX)\n\n[Follow us on Twitter](https://twitter.com/dair_ai)\n\n[Subscribe to our YouTube](https://www.youtube.com/channel/UCyna_OxOWL7IEuOwb7WhmxQ)\n\n[Subscribe to our Newsletter](https://nlpnews.substack.com/)\n\n---\n\n## Guides\nYou can also find the most up-to-date guides on our new website [https://www.promptingguide.ai/](https://www.promptingguide.ai/).\n\n- [Prompt Engineering - Introduction](https://www.promptingguide.ai/introduction)\n - [Prompt Engineering - LLM Settings](https://www.promptingguide.ai/introduction/settings)\n - [Prompt Engineering - Basics of Prompting](https://www.promptingguide.ai/introduction/basics)\n - [Prompt Engineering - Prompt Elements](https://www.promptingguide.ai/introduction/elements)\n - [Prompt Engineering - General Tips for Designing Prompts](https://www.promptingguide.ai/introduction/tips)\n - [Prompt Engineering - Examples of Prompts](https://www.promptingguide.ai/introduction/examples)\n- [Prompt Engineering - Techniques](https://www.promptingguide.ai/techniques)\n - [Prompt Engineering - Zero-Shot Prompting](https://www.promptingguide.ai/techniques/zeroshot)\n - [Prompt Engineering - Few-Shot Prompting](https://www.promptingguide.ai/techniques/fewshot)\n - [Prompt Engineering - Chain-of-Thought Prompting](https://www.promptingguide.ai/techniques/cot)\n - [Prompt Engineering - Self-Consistency](https://www.promptingguide.ai/techniques/consistency)\n - [Prompt Engineering - Generate Knowledge Prompting](https://www.promptingguide.ai/techniques/knowledge)\n - [Prompt Engineering - Prompt Chaining](https://www.promptingguide.ai/techniques/prompt_chaining)\n - [Prompt Engineering - Tree of Thoughts (ToT)](https://www.promptingguide.ai/techniques/tot)\n - [Prompt Engineering - Retrieval Augmented Generation](https://www.promptingguide.ai/techniques/rag)\n - [Prompt Engineering - Automatic Reasoning and Tool-use (ART)](https://www.promptingguide.ai/techniques/art)\n - [Prompt Engineering - Automatic Prompt Engineer](https://www.promptingguide.ai/techniques/ape)\n - [Prompt Engineering - Active-Prompt](https://www.promptingguide.ai/techniques/activeprompt)\n - [Prompt Engineering - Directional Stimulus Prompting](https://www.promptingguide.ai/techniques/dsp)\n - [Prompt Engineering - Program-Aided Language Models](https://www.promptingguide.ai/techniques/pal)\n - [Prompt Engineering - ReAct Prompting](https://www.promptingguide.ai/techniques/react)\n - [Prompt Engineering - Multimodal CoT Prompting](https://www.promptingguide.ai/techniques/multimodalcot)\n - [Prompt Engineering - Graph Prompting](https://www.promptingguide.ai/techniques/graph)\n- [Prompt Engineering - Applications](https://www.promptingguide.ai/applications)\n - [Prompt Engineering - Function Calling](https://www.promptingguide.ai/applications/function_calling)\n - [Prompt Engineering - Generating Data](https://www.promptingguide.ai/applications/generating)\n - [Prompt Engineering - Generating Synthetic Dataset for RAG](https://www.promptingguide.ai/applications/synthetic_rag)\n - [Prompt Engineering - Takling Generated Datasets Diversity](https://www.promptingguide.ai/applications/generating_textbooks)\n - [Prompt Engineering - Generating Code](https://www.promptingguide.ai/applications/coding)\n - [Prompt Engineering - Graduate Job Classification Case Study](https://www.promptingguide.ai/applications/workplace_casestudy)\n- [Prompt Engineering - Prompt Hub](https://www.promptingguide.ai/prompts)\n - [Prompt Engineering - Classification](https://www.promptingguide.ai/prompts/classification)\n - [Prompt Engineering - Coding](https://www.promptingguide.ai/prompts/coding)\n - [Prompt Engineering - Creativity](https://www.promptingguide.ai/prompts/creativity)\n - [Prompt Engineering - Evaluation](https://www.promptingguide.ai/prompts/evaluation)\n - [Prompt Engineering - Information Extraction](https://www.promptingguide.ai/prompts/information-extraction)\n - [Prompt Engineering - Image Generation](https://www.promptingguide.ai/prompts/image-generation)\n - [Prompt Engineering - Mathematics](https://www.promptingguide.ai/prompts/mathematics)\n - [Prompt Engineering - Question Answering](https://www.promptingguide.ai/prompts/question-answering)\n - [Prompt Engineering - Reasoning](https://www.promptingguide.ai/prompts/reasoning)\n - [Prompt Engineering - Text Summarization](https://www.promptingguide.ai/prompts/text-summarization)\n - [Prompt Engineering - Truthfulness](https://www.promptingguide.ai/prompts/truthfulness)\n - [Prompt Engineering - Adversarial Prompting](https://www.promptingguide.ai/prompts/adversarial-prompting)\n- [Prompt Engineering - Models](https://www.promptingguide.ai/models)\n - [Prompt Engineering - ChatGPT](https://www.promptingguide.ai/models/chatgpt)\n - [Prompt Engineering - Code Llama](https://www.promptingguide.ai/models/code-llama)\n - [Prompt Engineering - Flan](https://www.promptingguide.ai/models/flan)\n - [Prompt Engineering - Gemini](https://www.promptingguide.ai/models/gemini)\n - [Prompt Engineering - GPT-4](https://www.promptingguide.ai/models/gpt-4)\n - [Prompt Engineering - LLaMA](https://www.promptingguide.ai/models/llama)\n - [Prompt Engineering - Mistral 7B](https://www.promptingguide.ai/models/mistral-7b)\n - [Prompt Engineering - Mixtral](https://www.promptingguide.ai/models/mixtral)\n - [Prompt Engineering - OLMo](https://www.promptingguide.ai/models/olmo)\n - [Prompt Engineering - Phi-2](https://www.promptingguide.ai/models/phi-2)\n - [Prompt Engineering - Model Collection](https://www.promptingguide.ai/models/collection)\n- [Prompt Engineering - Risks and Misuses](https://www.promptingguide.ai/risks)\n - [Prompt Engineering - Adversarial Prompting](https://www.promptingguide.ai/risks/adversarial)\n - [Prompt Engineering - Factuality](https://www.promptingguide.ai/risks/factuality)\n - [Prompt Engineering - Biases](https://www.promptingguide.ai/risks/biases)\n- [Prompt Engineering - Papers](https://www.promptingguide.ai/papers)\n - [Prompt Engineering - Overviews](https://www.promptingguide.ai/papers#overviews)\n - [Prompt Engineering - Approaches](https://www.promptingguide.ai/papers#approaches)\n - [Prompt Engineering - Applications](https://www.promptingguide.ai/papers#applications)\n - [Prompt Engineering - Collections](https://www.promptingguide.ai/papers#collections)\n- [Prompt Engineering - Tools](https://www.promptingguide.ai/tools)\n- [Prompt Engineering - Notebooks](https://www.promptingguide.ai/notebooks)\n- [Prompt Engineering - Datasets](https://www.promptingguide.ai/datasets)\n- [Prompt Engineering - Additional Readings](https://www.promptingguide.ai/readings)\n\n\n---\n## Lecture\n\nWe have published a 1 hour lecture that provides a comprehensive overview of prompting techniques, applications, and tools.\n- [Video Lecture](https://youtu.be/dOxUroR57xs)\n- [Notebook with code](https://github.com/dair-ai/Prompt-Engineering-Guide/blob/main/notebooks/pe-lecture.ipynb)\n- [Slides](https://github.com/dair-ai/Prompt-Engineering-Guide/blob/main/lecture/Prompt-Engineering-Lecture-Elvis.pdf)\n\n---\n## Running the guide locally\n\nTo run the guide locally, for example to check the correct implementation of a new translation, you will need to:\n\n1. Install Node >=18.0.0\n1. Install `pnpm` if not present in your system. Check [here](https://pnpm.io/installation) for detailed instructions.\n1. Install the dependencies: `pnpm i next react react-dom nextra nextra-theme-docs`\n1. Boot the guide with `pnpm dev`\n2. Browse the guide at `http://localhost:3000/`\n\n---\n## Appearances\nSome places where we have been featured:\n- Wall Street Journal - [ChatGPT Can Give Great Answers. But Only If You Know How to Ask the Right Question](https://www.wsj.com/articles/chatgpt-ask-the-right-question-12d0f035)\n- Forbes - [Mom, Dad, I Want To Be A Prompt Engineer](https://www.forbes.com/sites/craigsmith/2023/04/05/mom-dad-i-want-to-be-a-prompt-engineer/?sh=7f1213159c8e)\n- Markettechpost - [Best Free Prompt Engineering Resources (2023)](https://www.marktechpost.com/2023/04/04/best-free-prompt-engineering-resources-2023/)\n\n\n---\nIf you are using the guide for your work or research, please cite us as follows:\n\n```\n@article{Saravia_Prompt_Engineering_Guide_2022,\nauthor = {Saravia, Elvis},\njournal = {https://github.com/dair-ai/Prompt-Engineering-Guide},\nmonth = {12},\ntitle = {{Prompt Engineering Guide}},\nyear = {2022}\n}\n```\n\n## License\n\n[MIT License](https://github.com/dair-ai/Prompt-Engineering-Guide/blob/main/LICENSE.md)\n\n\nFeel free to open a PR if you think something is missing here. Always welcome feedback and suggestions. Just open an issue!\n" }, { "path": "ar-pages/about.ar.mdx", "content": "# About\n\nThe Prompt Engineering Guide is a project by [DAIR.AI](https://github.com/dair-ai). It aims to educate researchers and practitioners about prompt engineering.\n\nDAIR.AI aims to democratize AI research, education, and technologies. Our mission is to enable the next-generation of AI innovators and creators.\n\nWe welcome contributions from the community. Lookout for the Edit buttons. \n\nLicense information [here](https://github.com/dair-ai/Prompt-Engineering-Guide#license).\n\nWe borrow inspirations from many open resources like [OpenAI CookBook](https://github.com/openai/openai-cookbook), [Pretrain, Prompt, Predict](http://pretrain.nlpedia.ai/), [Learn Prompting](https://learnprompting.org/), and many others." }, { "path": "ar-pages/applications/_meta.ar.json", "content": "{\n \"function_calling\": \"استدعاء الدوال\",\n \"generating\": \"توليد البيانات\",\n \"synthetic_rag\": \"توليد مجموعة بيانات لـ RAG\",\n \"generating_textbooks\": \"معالجة تنوع مجموعات البيانات المولدة\",\n \"coding\": \"توليد الأكواد\",\n \"workplace_casestudy\": \"دراسة حالة: تصنيف الوظائف للخريجين الجدد\",\n \"pf\": \"أمر على صيغة دالّة برمجية\"\n}\n" }, { "path": "ar-pages/applications/coding.ar.mdx", "content": "# Generating Code\n\nimport { Callout, FileTree } from 'nextra-theme-docs'\nimport {Screenshot} from 'components/screenshot'\nimport CODE1 from '../../img/code-generation/chat-mode.png'\n\n\n This section is under development.\n\n\nLLMs like ChatGPT are very effective at code generation. In this section, we will cover many examples of how to use ChatGPT for code generation.\n\nThe OpenAI's Playground (Chat Mode) and the `gpt-3.5-turbo` model are used for all examples below.\n\nAs with all chat models from OpenAI, you can use a `System Message` to define the behavior and format of the responses. We will use the following System Message for the prompt examples:\n\n```\nYou are a helpful code assistant that can teach a junior developer how to code. Your language of choice is Python. Don't explain the code, just generate the code block itself.\n```\n\n## Basic Example\n\nThe `User Message` will then be the prompt itself with the instruction of the specific code you want the model to generate. Below is a snapshot of the first basic prompt for code generation. Note that the final code was generated by the `Assistant`.\n\n\n\nHere is the prompt if you want to try it yourself: \n\n```\nWrite code that asks the user for their name and say \"Hello\"\n```\n\n## Turn Comments Into Code\n\nYou can also use the code generation capabilities of these LLMs to generate code from comments alone. Let's look at another example that passes the instructions as a comment block:\n\n*Prompt:*\n```\n\"\"\"\n1. Create a list of movies\n2. Create a list of ratings for these movies\n3. Combine them to make a json object of 10 movies with their ratings.\n\"\"\"\n```\n\n*Output:*\n```python\nmovies = [\"The Shawshank Redemption\", \"The Godfather\", \"The Dark Knight\", \"Schindler's List\", \"Pulp Fiction\", \"The Lord of the Rings: The Return of the King\", \"Forrest Gump\", \"Star Wars: Episode V - The Empire Strikes Back\", \"Inception\", \"The Silence of the Lambs\"]\n\nratings = [9.3, 9.2, 9.0, 8.9, 8.9, 8.9, 8.8, 8.7, 8.7, 8.6]\n\nmovie_ratings = {}\n\nfor i in range(len(movies)):\n movie_ratings[movies[i]] = ratings[i]\n\njson_object = json.dumps(movie_ratings, indent=4)\n\nprint(json_object)\n```\n\nThe generated code looks like working code. However, the output doesn't contain an important `import` statement, i.e., `import json`. While these models generate working code, you must pay close attention to small details like this and always make sure to test your code. \n\n## Complete Functions or Next Line\n\nThese LLMs have also been incorporated into tools like GitHub Copilot which makes them useful for developers. One useful feature is the ability of the model to complete functions. \n\n*Prompt:*\n```python\n# function to multiply two numbers and add 75 to it\n\ndef multiply(\n```\n\n*Output:*\n```python\na, b):\n result = a * b\n result += 75\n return result\n```\n\n## MySQL Query Generation\n\nBesides the basic code generation example above, you can use the model to generate useful code that could be used in other aspects of programming like creating and testing MySQL queries. \n\nLet's say you have a dataset with some information which you can include as part of the prompt and then instruct it to generate a specific query. For example:\n\n```\n\"\"\"\nTable departments, columns = [DepartmentId, DepartmentName]\nTable students, columns = [DepartmentId, StudentId, StudentName]\nCreate a MySQL query for all students in the Computer Science department\n\"\"\"\n```\n\n*Output:*\n```mysql\nSELECT students.StudentId, students.StudentName\nFROM students\nINNER JOIN departments\nON students.DepartmentId = departments.DepartmentId\nWHERE departments.DepartmentName = 'Computer Science';\n```\n\nThe model generates a query! However, in this example we made up the database so we don't have a database to test the generated query. If you are testing this example, it could be useful to have this database created. Fortunately, you can use ChatGPT to generate this for you as well. One approach is using a modified version of the prompt above with slightly different instructions to generate a database schema:\n\n```mysql\nTable departments, columns = [DepartmentId, DepartmentName]\nTable students, columns = [DepartmentId, StudentId, StudentName]\nCreate a valid database schema with the above tables and columns\n```\n\nThe model generated the following schema:\n\n```mysql\nCREATE TABLE departments (\n DepartmentId INT PRIMARY KEY,\n DepartmentName VARCHAR(50)\n);\n\nCREATE TABLE students (\n DepartmentId INT,\n StudentId INT PRIMARY KEY,\n StudentName VARCHAR(50),\n FOREIGN KEY (DepartmentId) REFERENCES departments(DepartmentId)\n);\n```\n\nYou can create a temporary database using something like [sqliteonline.com](https://sqliteonline.com/).\n\nYou will also need dummy data to test if the original generated query was valid. Let's use ChatGPT for this too! The prompt could look something like the following:\n\n```mysql\nCREATE TABLE departments (\n DepartmentId INT PRIMARY KEY,\n DepartmentName VARCHAR(50)\n);\n\nCREATE TABLE students (\n DepartmentId INT,\n StudentId INT PRIMARY KEY,\n StudentName VARCHAR(50),\n FOREIGN KEY (DepartmentId) REFERENCES departments(DepartmentId)\n);\n\nGiven the database schema above, generate valid insert statements include 4 rows for each table.\n```\n\nThe output with the insert statements:\n\n```mysql\nINSERT INTO departments (DepartmentId, DepartmentName)\nVALUES (1, 'Computer Science'), (2, 'Mathematics'), (3, 'Physics'), (4, 'Biology');\n\nINSERT INTO students (DepartmentId, StudentId, StudentName)\nVALUES (1, 101, 'John Doe'), (1, 102, 'Jane Doe'), (2, 201, 'Alice Smith'), (2, 202, 'Bob Johnson');\n```\n\nNow that you have the database and the dummy data, you can test the original query. Based on the dummy data, the result should be two rows with `Jane Doe` and `John Doe`.\n\n## Explain Code\nIf you are learning to program in a certain language, it might be useful to prompt the model to explain certain bits of code. Let's reuse the query generated above and ask the model to explain it.\n\nIf you are using the same System Message from before, be careful of how you are instructing the model. For example, if we use the following example, the model will complain that it is a code assistant in Python but it still provides an explanation. Sometimes the model refuses to generate what you are instructing because it's designed to follow the System Message. \n\n*Prompt:*\n\n```mysql\nSELECT students.StudentId, students.StudentName\nFROM students\nINNER JOIN departments\nON students.DepartmentId = departments.DepartmentId\nWHERE departments.DepartmentName = 'Computer Science';\n\nExplain the above SQL statement.\n```\n\n*Output:*\n\n```\nSorry, as a code assistant in Python, I cannot generate SQL code. However, I can explain the purpose of the SQL statement provided. \n\nThis SQL statement selects the `StudentId` and `StudentName` columns from the `students` table, where the `DepartmentName` column in the `departments` table is equal to 'Computer Science'. To achieve this, the statement performs an inner join on the `DepartmentId` column between the `students` and `departments` tables. \n\nIn simpler terms, it is selecting the names and IDs of students who are enrolled in the Computer Science department.\n```\n\n## Editing Code\nExample coming soon!\n\n## Debugging Code\nExample coming soon!\n\n## Best practices\nComing soon!\n" }, { "path": "ar-pages/applications/function_calling.ar.mdx", "content": "# Function Calling with LLMs\n\nimport {Cards, Card} from 'nextra-theme-docs'\nimport {CodeIcon} from 'components/icons'\n\n## Getting Started with Function Calling\n\nFunction calling is the ability to reliably connect LLMs to external tools to enable effective tool usage and interaction with external APIs. \n\nLLMs like GPT-4 and GPT-3.5 have been fine-tuned to detect when a function needs to be called and then output JSON containing arguments to call the function. The functions that are being called by function calling will act as tools in your AI application and you can define more than one in a single request.\n\nFunction calling is an important ability for building LLM-powered chatbots or agents that need to retrieve context for an LLM or interact with external tools by converting natural language into API calls. \n\nFunctional calling enables developers to create:\n\n- conversational agents that can efficiently use external tools to answer questions. For example, the query \"What is the weather like in Belize?\" will be converted to a function call such as `get_current_weather(location: string, unit: 'celsius' | 'fahrenheit')`\n- LLM-powered solutions for extracting and tagging data (e.g., extracting people names from a Wikipedia article)\n- applications that can help convert natural language to API calls or valid database queries\n- conversational knowledge retrieval engines that interact with a knowledge base \n\nIn this guide, we demonstrate how to prompt models like GPT-4 and open-source models to perform function calling for different use cases. \n\n## Function Calling with GPT-4\n\nAs a basic example, let's say we asked the model to check the weather in a given location. \n\nThe LLM alone would not be able to respond to this request because it has been trained on a dataset with a cutoff point. The way to solve this is to combine the LLM with an external tool. You can leverage the function calling capabilities of the model to determine an external function to call along with its arguments and then have it return a final response. Below is a simple example of how you can achieve this using the OpenAI APIs. \n\nLet's say a user is asking the following question to the model:\n\n```\nWhat is the weather like in London?\n```\n\nTo handle this request using function calling, the first step is to define a weather function or set of functions that you will be passing as part of the OpenAI API request:\n\n```python\ntools = [\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"get_current_weather\",\n \"description\": \"Get the current weather in a given location\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"location\": {\n \"type\": \"string\",\n \"description\": \"The city and state, e.g. San Francisco, CA\",\n },\n \"unit\": {\n \"type\": \"string\", \n \"enum\": [\"celsius\", \"fahrenheit\"]},\n },\n \"required\": [\"location\"],\n },\n }, \n }\n]\n```\n\nThe `get_current_weather` function returns the current weather in a given location. When you pass this function definition as part of the request, it doesn't actually executes a function, it just returns a JSON object containing the arguments needed to call the function. Here are some code snippets of how to achieve this. \n\nYou can define a completion function as follows:\n\n```python\ndef get_completion(messages, model=\"gpt-3.5-turbo-1106\", temperature=0, max_tokens=300, tools=None):\n response = openai.chat.completions.create(\n model=model,\n messages=messages,\n temperature=temperature,\n max_tokens=max_tokens,\n tools=tools\n )\n return response.choices[0].message\n```\n\nThis is how you can compose the user question:\n\n```python\nmessages = [\n {\n \"role\": \"user\",\n \"content\": \"What is the weather like in London?\"\n }\n]\n```\n\nFinally, you can call the `get_completion` above and passing both the `messages` and `tools`:\n\n```python\nresponse = get_completion(messages, tools=tools)\n```\n\nThe `response` object contains the following:\n\n```python\nChatCompletionMessage(content=None, role='assistant', function_call=None, tool_calls=[ChatCompletionMessageToolCall(id='...', function=Function(arguments='{\"location\":\"London\",\"unit\":\"celsius\"}', name='get_current_weather'), type='function')])\n```\n\nIn particular, the `arguments` object contains the important arguments extracted by the model and that will be needed to complete the request. \n\nYou can then choose to call an external weather API for the actual weather. Once you have the weather information available you can pass it back to the model to summarize a final response given the original user question.\n\n## Notebooks\n\nHere is a notebook with a simple example that demonstrates how to use function calling with the OpenAI APIs:\n\n\n }\n title=\"Function Calling with OpenAI APIs\"\n href=\"https://github.com/dair-ai/Prompt-Engineering-Guide/blob/main/notebooks/pe-function-calling.ipynb\"\n />\n\n\n## Function Calling with Open-Source LLMs\nMore notes on function calling with open-source LLMs coming soon.\n\n## Function Calling Use Cases\n\nBelow is a list of use cases that can benefit from the function calling capability of LLMs:\n\n- **Conversational Agents**: Function calling can be used to create complex conversational agents or chatbots that answer complex questions by calling external APIs or external knowledge base and providing more relevant and useful responses.\n\n- **Natural Language Understanding**: It can convert natural language into structured JSON data, extract structured data from text, and perform tasks like named entity recognition, sentiment analysis, and keyword extraction.\n\n- **Math Problem Solving**: Function calling can be used to define custom functions to solve complex mathematical problems that require multiple steps and different types of advanced calculations.\n\n- **API Integration**: It can be used to effectively integrate LLMs with external APIs to fetch data or perform actions based on the input. This could be helpful to build either a QA system or creative assistant. In general, function calling can convert natural language into valid API calls. \n\n- **Information Extraction**: Function calling be effectively used to extract specific information from a given input, such as retrieving relevant news stories or references from an article.\n\n\n## References\n- [Fireworks Raises the Quality Bar with Function Calling Model and API Release](https://blog.fireworks.ai/fireworks-raises-the-quality-bar-with-function-calling-model-and-api-release-e7f49d1e98e9)\n- [Benchmarking Agent Tool Use and Function Calling](https://blog.langchain.dev/benchmarking-agent-tool-use/)\n- [Function Calling](https://ai.google.dev/docs/function_calling)\n- [Interacting with APIs](https://python.langchain.com/docs/use_cases/apis)\n- [OpenAI's Function Calling](https://platform.openai.com/docs/guides/function-calling)\n- [How to call functions with chat models](https://cookbook.openai.com/examples/how_to_call_functions_with_chat_models)\n- [Pushing ChatGPT's Structured Data Support To Its Limits](https://minimaxir.com/2023/12/chatgpt-structured-data/)\n- [Math Problem Solving with Function Calling](https://github.com/svpino/openai-function-calling/blob/main/sample.ipynb)" }, { "path": "ar-pages/applications/generating.ar.mdx", "content": "# Generating Data\nLLMs have strong capabilities to generate coherent text. Using effective prompt strategies can steer the model to produce better, consistent, and more factual responses. LLMs can also be especially useful for generating data which is really useful to run all sorts of experiments and evaluations. For example, we can use it to generate quick samples for a sentiment classifier like so:\n\n*Prompt:*\n```\nProduce 10 exemplars for sentiment analysis. Examples are categorized as either positive or negative. Produce 2 negative examples and 8 positive examples. Use this format for the examples:\nQ: \nA: \n```\n\n*Output:*\n```\nQ: I just got the best news ever!\nA: Positive\n\nQ: We just got a raise at work!\nA: Positive\n\nQ: I'm so proud of what I accomplished today.\nA: Positive\n\nQ: I'm having the best day ever!\nA: Positive\n\nQ: I'm really looking forward to the weekend.\nA: Positive\n\nQ: I just got the best present ever!\nA: Positive\n\nQ: I'm so happy right now.\nA: Positive\n\nQ: I'm so blessed to have such an amazing family.\nA: Positive\n\nQ: The weather outside is so gloomy.\nA: Negative\n\nQ: I just got some terrible news.\nA: Negative\n```\n\nThis is very useful. We actually use this example for a different test in another section of the guides." }, { "path": "ar-pages/applications/generating_textbooks.ar.mdx", "content": "# Tackling Generated Datasets Diversity\n\nimport {Screenshot} from 'components/screenshot'\n\nimport IMG1 from '../../img/synthetic_diversity/textbooks_1.png'\nimport IMG2 from '../../img/synthetic_diversity/textbooks_2.png'\n\nIn the previous [chapter](https://www.promptingguide.ai/applications/synthetic_rag), we discussed the potential of using LLM for synthetic dataset generation to further finetune a local Retriever model. This method is possible due to the availability of a large corpus of unlabeled documents. Each document is used to generate one or more synthetic queries and form a query-document pair.\n\nBut what if Information Retrieval is not your task? Let's say you are working on a legal document classification problem but are not permitted to send any data to an external API. In this situation, you would need to train a local model. However, collecting data could become a significant obstacle, causing delays in product development.\n\nFor simplicity, let’s suppose the goal is to generate children's stories. This task was the starting point for research by [Eldan et al. (2023)](https://arxiv.org/abs/2305.07759). Each story consists of 2-3 paragraphs that follow a straightforward plot and theme, while the entire dataset covers a child's vocabulary and factual knowledge.\n\nLanguage is not just a system of rules and symbols; it conveys and interprets meaning. The main challenge of using large language models to produce training data is ensuring dataset diversity. Even with high [generation temperature](https://www.promptingguide.ai/introduction/settings), models can produce repetitive datasets lacking the necessary diversity (even for a child's language). Coherence and relevance are other natural language generation challenges.\n\nTo tackle the diversity issue, the authors prepared a vocabulary of around 1500 basic words, mirroring a typical child's vocabulary, divided into nouns, verbs, and adjectives. In each generation, one verb, one noun, and one adjective were randomly selected. The model then generates a story integrating these random words.\n\nThis technique significantly broadened the dataset's diversity, ensuring stories cover a child's entire vocabulary and diverse concept combinations. Additionally, the authors integrated potential story features (like dialogue, plot twists, bad endings, or moral lessons). A random subset of these features was chosen for each story, and the model was prompted to include them.\n\n*Prompt:*\n```\nWrite a short story (3-5 paragraphs) which only uses very simple words that a 3 year old child would likely understand. The story should use the verb ”{random.choice(verbs_list)}”, the noun ”{random.choice(nouns_list)}” and the adjective ”{random.choice(adjectives_list)}”. The story should have the following features: {random.choice(features_list)}, {random.choice(features_list)}. Remember to only use simple words!\n```\n\nLet's assume the gaps were filled as follows:\n\n*Prompt:*\n```\nWrite a short story (3-5 paragraphs) which only uses very simple words that a 3 year old child would likely understand. The story should use the verb ”decorate”, the noun ”thunder” and the adjective ”ancient”. The story should have the following features: the story should contain at least one dialogue, the story has a bad ending. Remember to only use simple words!\n```\n\n*Output:*\n```\nOnce upon a time, in an ancient house, there lived a girl named Lily. She loved to decorate her room with pretty things. One day, she found a big box in the attic. She opened it and saw many shiny decorations. Lily was very happy and decided to use them in her room. As Lily was decorating her room, the sky outside became dark. There was a loud thunder sound, and Lily got scared. She ran to her mom and said, ”Mommy, the thunder is so loud!” Her mom hugged her and said, ”Don’t worry, it will pass soon.” But the thunder did not stop. It got louder and louder, and the ancient house started to shake. Suddenly, the roof fell down on the pretty decorations. Lily was sad because her room was not pretty anymore. The end.\n```\n\nHere we rely on the latest text generation models (GPT-3.5 and GPT-4), which can produce vast amounts of synthetic content, following the instructions. Since we formulate the prompt randomly each time, and the model adheres to it precisely, the stories turn out to be incredibly varied. The narrative will be entirely different even if one or two words remain the same. In essence, we apply this technique to inject randomness into the prompt, generating a diverse dataset.\n\nYou get the idea:\n\n1. Identify which parameters/entities might vary between different samples in your synthetic dataset;\n2. Generate or manually compile a collection of these entities to fill in the gaps;\n3. Produce the dataset by randomly selecting entities for insertion. It's best to set the generation temperature higher than the default but below the maximum;\n4. Train a local model on the generation results of ChatGPT/GPT-4.\n\nIt is important to note that one of the entities used to seed your generation could be a class label. For example, in the sentiment classification task, you can directly mention \"positive\" or \"negative\" in the prompt to receive a generated text with the corresponding label. Then, a local classifier is trained on this data.\n\n# Iterative (Hierarchical) Synthetic Data Generation\n\nThe format can be made more complex by using even more entities, some of which are generated beforehand by the LLM itself. For example, one could start with a request to generate a story summary and pen down one sentence that must appear in the story. Later, just use that generated intermediate data in the final request.\n\n*Prompt:*\n```\nSummary: {a short summary generated by LLM, using the approach above}\nFeatures: {copy the features from the initial prompt}\nSentence: {a sentence generated by LLM, which should be present in the story}\nWords: {copy the words from the initial prompt}\nStory:\n```\n\nExample of a filled-in prompt:\n\n*Prompt:*\n```\nSummary: Lily and Timmy build a sandcastle together and learn to compromise, but it gets knocked over by a gust of wind. They find beauty in the broken sandcastle and play happily with a butterfly.\nFeatures: Dialogue, Foreshadowing, Twist\nSentence: One day, she went to the park and saw a beautiful butterfly.\nWords: disagree, network, beautiful\nStory:\n```\n\n*Output:*\n```\nOnce upon a time, there was a little girl named Lily. She loved to play with her toys and her friends. One day, she went to the park and saw a beautiful butterfly. She tried to catch it, but it flew away. Lily’s friend, Timmy, came to play with her. They decided to build a sandcastle, but they disagreed on how to build it. Lily wanted to make it tall, while Timmy wanted to make it wide. They argued for a while, but finally agreed to make it both tall and wide. As they were building the sandcastle, they noticed a group of ants working together. They realized that the ants were building a network of tunnels. Suddenly, a gust of wind came and knocked over their sandcastle. They were sad, but then they saw the butterfly again. It landed on the remains of their sandcastle and they realized that it was still beautiful, even in its broken state. They smiled and played together happily.\n```\n\nThus, it's possible to generate hundreds of thousands of very diverse examples to train the model on. Let's say you need to train a classifier that determines whether a text contains a dialogue or a plot twist. As the initial prompt contains labels, it's known which target value needs to be predicted for each generated sample.\n\n# Textbooks Are All You Need\n\nA crucial question arising from this approach is whether the synthesis of a dataset can truly provide benefits when training networks for real-world applications. Fortunately, the authors addressed this question by conducting their investigation and validating the efficacy of training smaller language models using synthetic data derived from State-of-the-Art LLMs.\n\nIn their study, [Gunasekar et al. (2023)](https://arxiv.org/abs/2306.11644) emphasize the importance of high-quality training data in their model. They argue that language models would be more effective if they were trained on materials that resemble the characteristics of a well-regarded \"textbook\": clear, comprehensive, informative, and unbiased.\n\nThese principles formed the basis for creating a semi-synthetic dataset to train LLM called Phi-1. The main evaluation task is to generate a Python function that follows a given text description or docstring. The model's quality is evaluated using the HumanEval benchmark ([Chen et al., 2021](https://arxiv.org/abs/2107.03374)).\n\nThe authors highlight the importance of diversity in this approach for several reasons. It exposes the language model to a variety of coding expressions and problem-solving approaches, reduces the risk of overfitting or relying on specific patterns, and improves the model's ability to handle unfamiliar or innovative tasks.\n\nTo address the code-writing challenge, the authors created textbook-like documents that focus on topics promoting reasoning and basic algorithmic skills. They achieved diversity by imposing restrictions on:\n\n- topics\n- target audience\n\nUnfortunately, the authors did not provide specific information about the prompt template used to generate the synthetic data. However, they did demonstrate the resulting output. They chose to use ChatGPT (GPT-3.5) instead of GPT-4, and even this strategy yielded excellent results as they fine-tuned the model using synthetic data.\n\n\n\nImage Source: [Gunasekar et al. (2023)](https://arxiv.org/abs/2306.11644)\n\nIt's essential to pay attention to the target audience constraint. Designating a target audience is an excellent way to enhance synthetic data diversity. A high school student, a CS undergraduate, and a PhD candidate certainly phrase their explanations and process information differently, making the generated data even more varied. Let's attempt to recreate the prompt.\n\n\n*Prompt:*\n```\nWrite an extract from a Computer Science textbook for a 1st-year bachelor. The coding language is Python 3.6.\n\nThis is an extract from the middle of the following topic: Singular matrices.\n\nThe extract starts with a high-level overview of the topic. Then, it presents an example and describes the solution in natural language. After that, it provides 1-2 code snippets, following the example. Each snippet has no more than 10 rows. There should be no text after code snippets.\n\nKeep in mind that the extract should be concise and explain only one concept at a time. The code should represent a Python function & its call. All the common ML/DS libraries are available.\n\nDon't be too verbose. The expected length of the extract is 2 paragraphs at most.\n```\n\n*Output:*\n```\n**Singular Matrices**\n\nIn linear algebra, a matrix is termed as \"singular\" if it doesn't have an inverse. Another equivalent definition is when the determinant of the matrix is zero. Singular matrices are crucial because they represent systems of linear equations that do not have a unique solution.\n\nConsider a 2x2 matrix `[[a, b], [c, d]]`. If the product of `ad` and `bc` is the same (i.e., `ad = bc`), then this matrix is singular. Let's explore this with an example: given the matrix `[[2, 4], [1, 2]]`, you'll find that the product of the diagonals yields the same result (2 * 2 = 4 * 1), thus it's a singular matrix.\n\nimport numpy as np\n\ndef is_singular(matrix):\n determinant = np.linalg.det(matrix)\n return abs(determinant) < 1e-9\n\nmatrix_example = np.array([[2, 4], [1, 2]])\nprint(is_singular(matrix_example)) # returns True\n```\n\nQuite close!\n\nIn total, the authors generated 1B tokens to augment the model's training set, allowing a smaller model (only 1.5B parameters) to rival models ten times its size (for details, refer to the article [Gunasekar et al. (2023)](https://arxiv.org/abs/2306.11644)).\n\n\n\nImage Source: [Gunasekar et al. (2023)](https://arxiv.org/abs/2306.11644)\n\nFor your task, you probably don't need such a large amount of synthetic data (since the authors studied the pretraining, which requires significant resources). However, even as an estimate, at a price of `$0.002` per 1k tokens (standard ChatGPT pricing), it would cost `$2000` for the generated tokens and approximately the same amount for the prompts.\n\nKeep in mind that fine-tuning on synthetic data becomes more valuable as the domain becomes more niche, especially if the language deviates from English (among other factors). Additionally, this method works well with [Chain-of-Thought (CoT)](https://www.promptingguide.ai/techniques/cot), helping the local model improve its reasoning capabilities. Other prompting techniques work, too. And don't forget that open-source models like Alpaca ([Taori et al., (2023)](https://crfm.stanford.edu/2023/03/13/alpaca.html)) and Vicuna ([Zheng et al., (2023)](https://lmsys.org/blog/2023-03-30-vicuna/)) excel through fine-tuning on synthetic data.\n" }, { "path": "ar-pages/applications/pf.ar.mdx", "content": "# Prompt Function\n\n## Introduction\nWhen we draw a parallel between GPT's dialogue interface and a programming language's shell, the encapsulation prompt can be thought of as forming a function. This function has a unique name, and when we call this name with the input text, it produces results based on the set internal rules. In a nutshell, we build a reusable prompt with a name that makes it easy to engage with GPT. It's like having a handy tool that lets GPT carry out particular tasks on our behalf – we just need to give the input, and we receive the desired output.\n\nBy encapsulating prompts into functions, you can create a series of functions to establish a workflow. Each function represents a specific step or task, and when combined in a particular order, they can automate complex processes or solve problems more efficiently. This approach allows for a more structured and streamlined interaction with GPT, ultimately enhancing its capabilities and making it a powerful tool to accomplish a wide range of tasks.\n\nSo before we can use a function, we need to let GPT know about it. Here is a prompt that defines the function.\n\n*Prompt:*\n> Let's call this prompt with **meta prompt**. \nThis prompt has been tested on GPT3.5 and performs even better on GPT4\n\n```\nHello, ChatGPT! I hope you are doing well. I am reaching out to you for assistance with a specific function. I understand that you have the capability to process information and perform various tasks based on the instructions provided. In order to help you understand my request more easily, I will be using a template to describe the function, input, and instructions on what to do with the input. Please find the details below:\n\nfunction_name: [Function Name]\ninput: [Input]\nrule: [Instructions on how to process the input]\n\nI kindly request you to provide the output for this function, based on the details I have provided. Your assistance is greatly appreciated. Thank you!\nI will replace the text inside the brackets with the relevant information for the function I want you to perform. This detailed introduction should help you understand my request more efficiently and provide the desired output. The format is function_name(input) If you understand, just answer one word with ok.\n\n```\n\n\n## Examples \n\n### English study assistant\nFor example, let's say we want to use GPT to aid us in our English studies. We can simplify the process by creating a series of functions.\n\nThis example has been tested on GPT3.5 and performs even better on GPT4\n\n#### Function description\n\nWe need to paste the **meta prompt** that was defined above the section in GPT\n\nThen we will create a function `trans_word`. \nThis function prompts GPT to translate Chinese into English.\n\n*Prompt:*\n```\nfunction_name: [trans_word]\ninput: [\"text\"]\nrule: [I want you to act as an English translator, spelling corrector and improver. I will provide you with input forms including \"text\" in any language and you will detect the language, translate it and answer in the corrected of my text, in English.]\n```\n\n\nWrite a function that expands text.\n\n*Prompt:*\n```\nfunction_name: [expand_word]\ninput: [\"text\"]\nrule: [Please serve as a Chatterbox, spelling corrector, and language enhancer. I will provide you with input forms including \"text\" in any language, and output the original language.I want you to Keep the meaning same, but make them more literary.]\n```\n\nWrite a function that corrects text.\n\n*Prompt:*\n```\nfunction_name: [fix_english]\ninput: [\"text\"]\nrule: [Please serve as an English master, spelling corrector, and language enhancer. I will provide you with input forms including \"text\", I want you to improve the text's vocabulary and sentences with more natural and elegent. Keep the meaning same.]\n```\nFinally, you can run the function independently or chain them together.\n\n*Prompt:*\n```\ntrans_word('婆罗摩火山处于享有“千岛之国”美称的印度尼西亚. 多岛之国印尼有4500座之多的火山, 世界著名的十大活火山有三座在这里.')\nfix_english('Finally, you can run the function independently or chain them together.')\nfix_english(expand_word(trans_word('婆罗摩火山处于享有“千岛之国”美称的印度尼西亚. 多岛之国印尼有4500座之多的火山, 世界著名的十大活火山有三座在这里.')))\n```\nBy representing the functions in this format, you can clearly see each function's name, input, and the rule to process the input. It provides an organized way to understand the functionality and purpose of each step in the workflow\n\n_tips:_\nIf you don't want ChatGPT to output excessive information, you can simply add a sentence after defining the function's rules.\n```\nDO NOT SAY THINGS ELSE OK, UNLESS YOU DONT UNDERSTAND THE FUNCTION\n```\n\n### Multiple params function\nLet's create a function that generates a password by taking five input parameters, and outputs the generated password.\n\n*Prompt:*\n```\nfunction_name: [pg]\ninput: [\"length\", \"capitalized\", \"lowercase\", \"numbers\", \"special\"]\nrule: [I want you to act as a password generator for individuals in need of a secure password. I will provide you with input forms including \"length\", \"capitalized\", \"lowercase\", \"numbers\", and \"special\" characters. Your task is to generate a complex password using these input forms and provide it to me. Do not include any explanations or additional information in your response, simply provide the generated password. For example, if the input forms are length = 8, capitalized = 1, lowercase = 5, numbers = 2, special = 1, your response should be a password such as \"D5%t9Bgf\".]\n```\n```\npg(length = 10, capitalized = 1, lowercase = 5, numbers = 2, special = 1)\npg(10,1,5,2,1)\n```\n\n### Thought\nNow, there already have many projects that are working on programming GPT, such as:\n- [GitHub Copilot](https://github.com/features/copilot)\n- [Microsoft AI](https://www.microsoft.com/en-us/ai)\n- [chatgpt-plugins](https://openai.com/blog/chatgpt-plugins)\n- [LangChain](https://github.com/hwchase17/langchain)\n- [marvin](https://github.com/PrefectHQ/marvin)\n\nBut those projects are designed either for product customer or for users who can code with Python or other programming languages. \nFor the average user, use this easy template for daily work and iterate a couple of times. Use a note application to document the function, and it can even be updated to a library. \nAlternatively, some open source ChatGPT tools, such as [ChatGPT-Next-Web](https://github.com/Yidadaa/ChatGPT-Next-Web), [chatbox](https://github.com/Bin-Huang/chatbox), [PromptAppGPT](https://github.com/mleoking/PromptAppGPT), and [ChatGPT-Desktop](https://github.com/lencx/ChatGPT), can be used. Currently, ChatGPT-Next-Web allows adding a few shots before initializing the new chat. And PromptAppGPT supports low-code development of web applications based on prompt templates and enables anyone to develop AutoGPT-like applications with a few lines of prompts.\nWe can use this feature to add our function, which can then be used.\n" }, { "path": "ar-pages/applications/synthetic_rag.ar.mdx", "content": "# Generating Synthetic Dataset for RAG\n\nimport {Screenshot} from 'components/screenshot'\nimport remarkMath from 'remark-math'\nimport rehypeKatex from 'rehype-katex'\n\nimport IMG1 from '../../img/synthetic_rag/synthetic_rag_1.png'\nimport IMG2 from '../../img/synthetic_rag/synthetic_rag_2.png'\nimport IMG3 from '../../img/synthetic_rag/synthetic_rag_3.png'\nimport IMG4 from '../../img/synthetic_rag/synthetic_rag_4.png'\n\n\n## Synthetic Data for RAG Setup\nUnfortunately, in the life of a Machine Learning Engineer, there's often a lack of labeled data or very little of it. Typically, upon realizing this, projects embark on a lengthy process of data collection and labeling. Only after a couple of months can one start developing a solution.\n\nHowever, with the advent of LLM, the paradigm shifted in some products: now one can rely on LLM’s generalization ability and test an idea or develop an AI-powered feature almost immediately. If it turns out to work (almost) as intended, then the traditional development process can begin.\n\n\n\nImage Source: [The Rise of the AI Engineer, by S. Wang](https://www.latent.space/p/ai-engineer)\n\nOne of the emerging approaches is [Retrieval Augmented Generation (RAG)](https://www.promptingguide.ai/techniques/rag). It's used for knowledge-intensive tasks where you can't solely rely on the model's knowledge. RAG combines an information retrieval component with a text generator model. To learn more about this approach, refer to [the relevant section in the guide](https://www.promptingguide.ai/techniques/rag).\n\nThe key component of RAG is a Retrieval model that identifies relevant documents and passes them to LLM for further processing. The better the performance of the Retrieval model, the better the product or feature outcome. Ideally, Retrieval works well right out of the box. However, its performance often drops in different languages or specific domains.\n\nImagine this: you need to create a chatbot answering questions based on Czech laws and legal practices (in Czech, of course). Or design a tax assistant (a use case presented by OpenAI during the GPT-4 presentation) tailored for the Indian market. You'll likely find that the Retrieval model often misses the most relevant documents and doesn't perform as well overall, thus limiting the system's quality.\n\nBut there's a solution. An emerging trend involves using existing LLMs to synthesize data for the training of new generations of LLMs/Retrievers/other models. This process can be viewed as distilling LLMs into standard-sized encoders via prompt-based query generation. While the distillation is computationally intensive, it substantially reduces inference costs and might greatly enhance performance, particularly in low-resource languages or specialized domains.\n\nIn this guide, we will rely on the latest text generation models, like ChatGPT and GPT-4, which can produce vast amounts of synthetic content following instructions. [Dai et al. (2022)](https://arxiv.org/abs/2209.11755) proposed a method where with only 8 manually labeled examples and a large corpus of unlabeled data (documents for retrieval, e.g., all the parsed laws), one can achieve a near State-of-the-Art performance. This research confirms that synthetically generated data facilitates training task-specific retrievers for tasks where supervised in-domain fine-tuning is a challenge due to data scarcity.\n\n## Domain-Specific Dataset Generation\nTo utilize LLM, one needs to provide a short description and manually label a few examples. It's important to note that different retrieval tasks possess varying search intents, meaning different definitions of \"relevance.\" In other words, for the same pair of (Query, Document), their relevance might differ entirely based on the search intent. For instance, an argument retrieval task might seek supporting arguments, while other tasks require counter-arguments (as seen in [ArguAna dataset](https://aclanthology.org/P18-1023/)).\n\nConsider the example below. Though written in English for easier understanding, remember that data can be in any language since ChatGPT/GPT-4 efficiently processes even low-resource languages.\n\n*Prompt:*\n```\nTask: Identify a counter-argument for the given argument.\n\nArgument #1: {insert passage X1 here}\n\nA concise counter-argument query related to the argument #1: {insert manually prepared query Y1 here}\n\nArgument #2: {insert passage X2 here}\nA concise counter-argument query related to the argument #2: {insert manually prepared query Y2 here}\n\n<- paste your examples here ->\n\nArgument N: Even if a fine is made proportional to income, you will not get the equality of impact you desire. This is because the impact is not proportional simply to income, but must take into account a number of other factors. For example, someone supporting a family will face a greater impact than someone who is not, because they have a smaller disposable income. Further, a fine based on income ignores overall wealth (i.e. how much money someone actually has: someone might have a lot of assets but not have a high income). The proposition does not cater for these inequalities, which may well have a much greater skewing effect, and therefore the argument is being applied inconsistently.\n\nA concise counter-argument query related to the argument #N:\n```\n\n*Output:*\n```\npunishment house would make fines relative income\n```\n\nIn general, such a prompt can be expressed as:\n\n$(e_{prompt}, e_{doc}(d_{1}), e_{query}(q_1), . . . , e_{doc}(d_k), e_{query}(q_k), e_{doc}(d))$\n\n, where $e_{doc}$ and $e_{query}$ are task-specific document, query descriptions respectively, $e_{prompt}$ is a task-specific prompt/instruction for ChatGPT/GPT-4, and $d$ is a new document, for which LLM will generate a query.\n\nFrom this prompt, only the last document $d$ and the generated query will be used for further training of the local model. This approach can be applied when a target retrieval corpus $D$ is available, but the number of annotated query-document pairs for the new task is limited.\n\nThe whole pipeline overview: \n\n\n\nImage Source: [Dai et al. (2022)](https://arxiv.org/abs/2209.11755)\n\nIt's crucial to handle manual annotation of examples responsibly. It's better to prepare more (for instance, 20), and randomly pick 2-8 of them to the prompt. This increases the diversity of generated data without significant time costs in annotation. However, these examples should be representative, correctly formatted, and even detail specifics such as the target query length or its tone. The more precise the examples and instructions, the better the synthetic data will be for training Retriever. Low-quality few-shot examples can negatively impact the resulting quality of the trained model.\n\nIn most cases, using a more affordable model like ChatGPT is sufficient, as it performs well with unusual domains and languages other than English. Let's say, a prompt with instructions and 4-5 examples typically takes up 700 tokens (assuming each passage is no longer than 128 tokens due to Retriever constraints) and generation is 25 tokens. Thus, generating a synthetic dataset for a corpus of 50,000 documents for local model fine-tuning would cost: `50,000 * (700 * 0.001 * $0.0015 + 25 * 0.001 * $0.002) = 55`, where `$0.0015` and `$0.002` are the cost per 1,000 tokens in the GPT-3.5 Turbo API. It's even possible to generate 2-4 query examples for the same document. However, often the benefits of further training are worth it, especially if you're using Retriever not for a general domain (like news retrieval in English) but for a specific one (like Czech laws, as mentioned).\n\nThe figure of 50,000 isn't random. In the research by [Dai et al. (2022)](https://arxiv.org/abs/2209.11755), it's stated that this is approximately the number of manually labeled data needed for a model to match the quality of one trained on synthetic data. Imagine having to gather at least 10,000 examples before launching your product! It would take no less than a month, and the labor costs would surely exceed a thousand dollars, much more than generating synthetic data and training a local Retriever Model. Now, with the technique you learned today, you can achieve double-digit metric growth in just a couple of days!\n\n\n\nImage Source: [Dai et al. (2022)](https://arxiv.org/abs/2209.11755)\n\nAnd here are prompt templates from the same paper for some of the datasets in BeIR benchmark.\n\n\n\nImage Source: [Dai et al. (2022)](https://arxiv.org/abs/2209.11755)\n" }, { "path": "ar-pages/applications/workplace_casestudy.ar.mdx", "content": "# Graduate Job Classification Case Study\n\n[Clavié et al., 2023](https://arxiv.org/abs/2303.07142) provide a case-study on prompt-engineering applied to a medium-scale text classification use-case in a production system. Using the task of classifying whether a job is a true \"entry-level job\", suitable for a recent graduate, or not, they evaluated a series of prompt engineering techniques and report their results using GPT-3.5 (`gpt-3.5-turbo`).\n\nThe work shows that LLMs outperforms all other models tested, including an extremely strong baseline in DeBERTa-V3. `gpt-3.5-turbo` also noticeably outperforms older GPT3 variants in all key metrics, but requires additional output parsing as its ability to stick to a template appears to be worse than the other variants.\n\nThe key findings of their prompt engineering approach are:\n\n- For tasks such as this one, where no expert knowledge is required, Few-shot CoT prompting performed worse than Zero-shot prompting in all experiments.\n- The impact of the prompt on eliciting the correct reasoning is massive. Simply asking the model to classify a given job results in an F1 score of 65.6, whereas the post-prompt engineering model achieves an F1 score of 91.7.\n- Attempting to force the model to stick to a template lowers performance in all cases (this behaviour disappears in early testing with GPT-4, which are posterior to the paper).\n- Many small modifications have an outsized impact on performance.\n - The tables below show the full modifications tested.\n - Properly giving instructions and repeating the key points appears to be the biggest performance driver.\n - Something as simple as giving the model a (human) name and referring to it as such increased F1 score by 0.6pts.\n\n### Prompt Modifications Tested\n\n| Short name | Description |\n|------------|----------------------------------------------------------------------------|\n| Baseline | Provide a a job posting and asking if it is fit for a graduate. |\n| CoT | Give a few examples of accurate classification before querying. |\n| Zero-CoT | Ask the model to reason step-by-step before providing its answer. |\n| rawinst | Give instructions about its role and the task by adding to the user msg. |\n| sysinst | Give instructions about its role and the task as a system msg. |\n| bothinst | Split instructions with role as a system msg and task as a user msg. |\n| mock | Give task instructions by mocking a discussion where it acknowledges them. |\n| reit | Reinforce key elements in the instructions by repeating them. |\n| strict | Ask the model to answer by strictly following a given template. |\n| loose | Ask for just the final answer to be given following a given template. |\n| right | Asking the model to reach the right conclusion. |\n| info | Provide additional information to address common reasoning failures. |\n| name | Give the model a name by which we refer to it in conversation. |\n| pos | Provide the model with positive feedback before querying it. |\n\n\n### Performance Impact of All Prompt Modifications\n\n| | Precision | Recall | F1 | Template Stickiness |\n|----------------------------------------|---------------|---------------|---------------|------------------------|\n| _Baseline_ | _61.2_ | _70.6_ | _65.6_ | _79%_ |\n| _CoT_ | _72.6_ | _85.1_ | _78.4_ | _87%_ |\n| _Zero-CoT_ | _75.5_ | _88.3_ | _81.4_ | _65%_ |\n| _+rawinst_ | _80_ | _92.4_ | _85.8_ | _68%_ |\n| _+sysinst_ | _77.7_ | _90.9_ | _83.8_ | _69%_ |\n| _+bothinst_ | _81.9_ | _93.9_ | _87.5_ | _71%_ |\n| +bothinst+mock | 83.3 | 95.1 | 88.8 | 74% |\n| +bothinst+mock+reit | 83.8 | 95.5 | 89.3 | 75% |\n| _+bothinst+mock+reit+strict_ | _79.9_ | _93.7_ | _86.3_ | _**98%**_ |\n| _+bothinst+mock+reit+loose_ | _80.5_ | _94.8_ | _87.1_ | _95%_ |\n| +bothinst+mock+reit+right | 84 | 95.9 | 89.6 | 77% |\n| +bothinst+mock+reit+right+info | 84.9 | 96.5 | 90.3 | 77% |\n| +bothinst+mock+reit+right+info+name | 85.7 | 96.8 | 90.9 | 79% |\n| +bothinst+mock+reit+right+info+name+pos| **86.9** | **97** | **91.7** | 81% |\n\nTemplate stickiness refers to how frequently the model answers in the desired format. \n" }, { "path": "ar-pages/applications.ar.mdx", "content": "# LLM Applications\n\nimport { Callout } from 'nextra-theme-docs'\nimport {Cards, Card} from 'nextra-theme-docs'\nimport {FilesIcon} from 'components/icons'\nimport ContentFileNames from 'components/ContentFileNames'\n\nIn this section, we will cover advanced and interesting ways we can use prompt engineering to perform useful and more advanced tasks with large language models (LLMs).\n\n" }, { "path": "ar-pages/course.ar.mdx", "content": "# Prompt Engineering Courses\n\nimport { Callout } from 'nextra/components'\n\n\nCheck out our courses on prompt engineering at the DAIR.AI Academy:\n\n- [Introduction to Prompt Engineering](https://academy.dair.ai/courses/introduction-prompt-engineering) (Beginner) - learn about the latest prompt engineering techniques and how to effectively apply them to real-world use cases.\n\n- [Advanced Prompt Engineering](https://academy.dair.ai/courses/advanced-prompt-engineering) (Advanced) - learn advanced prompt engineering techniques to build complex use cases and applications with LLMs.\n\nUse promo code PROMPTING20 for a 20% discount.\n\n\nThese hands-on courses are built to compliment this prompt engineering guide. They are designed to help expand your skills and knowledge by teaching you how to effectively apply the concepts learned in this guide to real-world use cases and applications.\n\n[Elvis Saravia](https://www.linkedin.com/in/omarsar/), who has worked at companies like Meta AI and Elastic, and has years of experience in AI and LLMs, is the instructor for both courses.\n\nOur past learners range from software engineers to AI researchers and practitioners in organizations like Microsoft, Google, Apple, Airbnb, LinkedIn, Amazon, JPMorgan Chase & Co., Asana, Intuit, Fidelity Investments, Coinbase, Guru, and many others.\n\nTopics we provide training on:\n\n- Taxonomy of Prompting Techniques\n- Tactics to Improve Reliability\n- Structuring LLM Outputs\n- Zero-shot Prompting\n- Few-shot In-Context Learning\n- Chain of Thought Prompting\n- Self-Reflection & Self-Consistency\n- ReAcT\n- Retrieval Augmented Generation\n- Fine-Tuning & RLHF\n- Function Calling\n- AI Safety & Moderation\n- LLM-Powered Agents\n- LLM Evaluation\n- Adversarial Prompting (Jailbreaking and Prompt Injections)\n- Judge LLMs\n- Common Real-World Use Cases of LLMs\n\nReach out to training@dair.ai for any questions about the courses, corporate training, and available group discounts.\n" }, { "path": "ar-pages/datasets.ar.mdx", "content": "# Datasets\n#### (Sorted by Name)\n\n- [Anthropic's Red Team dataset](https://github.com/anthropics/hh-rlhf/tree/master/red-team-attempts), [(paper)](https://arxiv.org/abs/2209.07858)\n- [Awesome ChatGPT Prompts](https://huggingface.co/datasets/fka/awesome-chatgpt-prompts)\n- [DiffusionDB](https://github.com/poloclub/diffusiondb)\n- [Midjourney Prompts](https://huggingface.co/datasets/succinctly/midjourney-prompts)\n- [P3 - Public Pool of Prompts](https://huggingface.co/datasets/bigscience/P3)\n- [PartiPrompts](https://parti.research.google)\n- [Real Toxicity Prompts](https://allenai.org/data/real-toxicity-prompts)\n- [Stable Diffusion Dataset](https://huggingface.co/datasets/Gustavosta/Stable-Diffusion-Prompts)\n- [WritingPrompts](https://www.reddit.com/r/WritingPrompts)\n" }, { "path": "ar-pages/introduction/_meta.ar.json", "content": "{\n \"settings\": \"إعدادات النماذج اللغوية الكبيرة\",\n \"basics\": \"أساسيات التلقين\",\n \"elements\": \"عناصر الأوامر\",\n \"tips\": \"نصائح عامة لتصميم الأوامر\",\n \"examples\": \"أمثلة على الأوامر\"\n}\n" }, { "path": "ar-pages/introduction/basics.ar.mdx", "content": "# Basics of Prompting\n\nimport {Screenshot} from 'components/screenshot'\nimport INTRO1 from '../../img/introduction/sky.png'\nimport {Bleed} from 'nextra-theme-docs'\n\n## Prompting an LLM\n\nYou can achieve a lot with simple prompts, but the quality of results depends on how much information you provide it and how well-crafted the prompt is. A prompt can contain information like the *instruction* or *question* you are passing to the model and include other details such as *context*, *inputs*, or *examples*. You can use these elements to instruct the model more effectively to improve the quality of results.\n\nLet's get started by going over a basic example of a simple prompt:\n\n*Prompt*\n\n```md\nThe sky is\n```\n\n*Output:*\n```md\nblue.\n```\n\nIf you are using the OpenAI Playground or any other LLM playground, you can prompt the model as shown in the following screenshot:\n\n\n\nHere is a tutorial on how to get started with the OpenAI Playground:\n\n