[ { "path": "Images/ReadMe.md", "content": "\n" }, { "path": "LICENSE", "content": " Apache License\n Version 2.0, January 2004\n http://www.apache.org/licenses/\n\n TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION\n\n 1. Definitions.\n\n \"License\" shall mean the terms and conditions for use, reproduction,\n and distribution as defined by Sections 1 through 9 of this document.\n\n \"Licensor\" shall mean the copyright owner or entity authorized by\n the copyright owner that is granting the License.\n\n \"Legal Entity\" shall mean the union of the acting entity and all\n other entities that control, are controlled by, or are under common\n control with that entity. For the purposes of this definition,\n \"control\" means (i) the power, direct or indirect, to cause the\n direction or management of such entity, whether by contract or\n otherwise, or (ii) ownership of fifty percent (50%) or more of the\n outstanding shares, or (iii) beneficial ownership of such entity.\n\n \"You\" (or \"Your\") shall mean an individual or Legal Entity\n exercising permissions granted by this License.\n\n \"Source\" form shall mean the preferred form for making modifications,\n including but not limited to software source code, documentation\n source, and configuration files.\n\n \"Object\" form shall mean any form resulting from mechanical\n transformation or translation of a Source form, including but\n not limited to compiled object code, generated documentation,\n and conversions to other media types.\n\n \"Work\" shall mean the work of authorship, whether in Source or\n Object form, made available under the License, as indicated by a\n copyright notice that is included in or attached to the work\n (an example is provided in the Appendix below).\n\n \"Derivative Works\" shall mean any work, whether in Source or Object\n form, that is based on (or derived from) the Work and for which the\n editorial revisions, annotations, elaborations, or other modifications\n represent, as a whole, an original work of authorship. For the purposes\n of this License, Derivative Works shall not include works that remain\n separable from, or merely link (or bind by name) to the interfaces of,\n the Work and Derivative Works thereof.\n\n \"Contribution\" shall mean any work of authorship, including\n the original version of the Work and any modifications or additions\n to that Work or Derivative Works thereof, that is intentionally\n submitted to Licensor for inclusion in the Work by the copyright owner\n or by an individual or Legal Entity authorized to submit on behalf of\n the copyright owner. For the purposes of this definition, \"submitted\"\n means any form of electronic, verbal, or written communication sent\n to the Licensor or its representatives, including but not limited to\n communication on electronic mailing lists, source code control systems,\n and issue tracking systems that are managed by, or on behalf of, the\n Licensor for the purpose of discussing and improving the Work, but\n excluding communication that is conspicuously marked or otherwise\n designated in writing by the copyright owner as \"Not a Contribution.\"\n\n \"Contributor\" shall mean Licensor and any individual or Legal Entity\n on behalf of whom a Contribution has been received by Licensor and\n subsequently incorporated within the Work.\n\n 2. Grant of Copyright License. Subject to the terms and conditions of\n this License, each Contributor hereby grants to You a perpetual,\n worldwide, non-exclusive, no-charge, royalty-free, irrevocable\n copyright license to reproduce, prepare Derivative Works of,\n publicly display, publicly perform, sublicense, and distribute the\n Work and such Derivative Works in Source or Object form.\n\n 3. Grant of Patent License. Subject to the terms and conditions of\n this License, each Contributor hereby grants to You a perpetual,\n worldwide, non-exclusive, no-charge, royalty-free, irrevocable\n (except as stated in this section) patent license to make, have made,\n use, offer to sell, sell, import, and otherwise transfer the Work,\n where such license applies only to those patent claims licensable\n by such Contributor that are necessarily infringed by their\n Contribution(s) alone or by combination of their Contribution(s)\n with the Work to which such Contribution(s) was submitted. If You\n institute patent litigation against any entity (including a\n cross-claim or counterclaim in a lawsuit) alleging that the Work\n or a Contribution incorporated within the Work constitutes direct\n or contributory patent infringement, then any patent licenses\n granted to You under this License for that Work shall terminate\n as of the date such litigation is filed.\n\n 4. Redistribution. You may reproduce and distribute copies of the\n Work or Derivative Works thereof in any medium, with or without\n modifications, and in Source or Object form, provided that You\n meet the following conditions:\n\n (a) You must give any other recipients of the Work or\n Derivative Works a copy of this License; and\n\n (b) You must cause any modified files to carry prominent notices\n stating that You changed the files; and\n\n (c) You must retain, in the Source form of any Derivative Works\n that You distribute, all copyright, patent, trademark, and\n attribution notices from the Source form of the Work,\n excluding those notices that do not pertain to any part of\n the Derivative Works; and\n\n (d) If the Work includes a \"NOTICE\" text file as part of its\n distribution, then any Derivative Works that You distribute must\n include a readable copy of the attribution notices contained\n within such NOTICE file, excluding those notices that do not\n pertain to any part of the Derivative Works, in at least one\n of the following places: within a NOTICE text file distributed\n as part of the Derivative Works; within the Source form or\n documentation, if provided along with the Derivative Works; or,\n within a display generated by the Derivative Works, if and\n wherever such third-party notices normally appear. The contents\n of the NOTICE file are for informational purposes only and\n do not modify the License. You may add Your own attribution\n notices within Derivative Works that You distribute, alongside\n or as an addendum to the NOTICE text from the Work, provided\n that such additional attribution notices cannot be construed\n as modifying the License.\n\n You may add Your own copyright statement to Your modifications and\n may provide additional or different license terms and conditions\n for use, reproduction, or distribution of Your modifications, or\n for any such Derivative Works as a whole, provided Your use,\n reproduction, and distribution of the Work otherwise complies with\n the conditions stated in this License.\n\n 5. Submission of Contributions. Unless You explicitly state otherwise,\n any Contribution intentionally submitted for inclusion in the Work\n by You to the Licensor shall be under the terms and conditions of\n this License, without any additional terms or conditions.\n Notwithstanding the above, nothing herein shall supersede or modify\n the terms of any separate license agreement you may have executed\n with Licensor regarding such Contributions.\n\n 6. Trademarks. This License does not grant permission to use the trade\n names, trademarks, service marks, or product names of the Licensor,\n except as required for reasonable and customary use in describing the\n origin of the Work and reproducing the content of the NOTICE file.\n\n 7. Disclaimer of Warranty. Unless required by applicable law or\n agreed to in writing, Licensor provides the Work (and each\n Contributor provides its Contributions) on an \"AS IS\" BASIS,\n WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or\n implied, including, without limitation, any warranties or conditions\n of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A\n PARTICULAR PURPOSE. You are solely responsible for determining the\n appropriateness of using or redistributing the Work and assume any\n risks associated with Your exercise of permissions under this License.\n\n 8. Limitation of Liability. In no event and under no legal theory,\n whether in tort (including negligence), contract, or otherwise,\n unless required by applicable law (such as deliberate and grossly\n negligent acts) or agreed to in writing, shall any Contributor be\n liable to You for damages, including any direct, indirect, special,\n incidental, or consequential damages of any character arising as a\n result of this License or out of the use or inability to use the\n Work (including but not limited to damages for loss of goodwill,\n work stoppage, computer failure or malfunction, or any and all\n other commercial damages or losses), even if such Contributor\n has been advised of the possibility of such damages.\n\n 9. Accepting Warranty or Additional Liability. While redistributing\n the Work or Derivative Works thereof, You may choose to offer,\n and charge a fee for, acceptance of support, warranty, indemnity,\n or other liability obligations and/or rights consistent with this\n License. However, in accepting such obligations, You may act only\n on Your own behalf and on Your sole responsibility, not on behalf\n of any other Contributor, and only if You agree to indemnify,\n defend, and hold each Contributor harmless for any liability\n incurred by, or claims asserted against, such Contributor by reason\n of your accepting any such warranty or additional liability.\n\n END OF TERMS AND CONDITIONS\n\n APPENDIX: How to apply the Apache License to your work.\n\n To apply the Apache License to your work, attach the following\n boilerplate notice, with the fields enclosed by brackets \"[]\"\n replaced with your own identifying information. (Don't include\n the brackets!) The text should be enclosed in the appropriate\n comment syntax for the file format. We also recommend that a\n file or class name and description of purpose be included on the\n same \"printed page\" as the copyright notice for easier\n identification within third-party archives.\n\n Copyright [yyyy] [name of copyright owner]\n\n Licensed under the Apache License, Version 2.0 (the \"License\");\n you may not use this file except in compliance with the License.\n You may obtain a copy of the License at\n\n http://www.apache.org/licenses/LICENSE-2.0\n\n Unless required by applicable law or agreed to in writing, software\n distributed under the License is distributed on an \"AS IS\" BASIS,\n WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n See the License for the specific language governing permissions and\n limitations under the License.\n" }, { "path": "README.md", "content": "# ๐Ÿ‘จ๐Ÿปโ€๐Ÿ’ป LLM Engineer Toolkit \nThis repository contains a curated list of 120+ LLM libraries category wise.\n

\n \n \"LinkedIn\"\n \n \n \"Twitter\"\n \n \n \"Twitter\"\n \n\t\n

\n\n## ๐Ÿš€ LLM Interview Questions and Answers Book \nCrack modern LLM and Generative AI interviews with this comprehensive, interview-focused guide designed specifically for ML Engineers, AI Engineers, Data Scientists and Software Engineers.\n\nThis book features **100+ carefully curated LLM interview questions**, each paired with **clear answers and in-depth explanations** so you truly understand the concepts interviewers care about. [Get the book here](https://kalyanksnlp.gumroad.com/l/llm-interview-questions-answers-book-kalyan-ks). \n\nUse the **Coupon Code: LLMQA25** for an exclusive discount (50%) on the book. (Available only for a short period of time). \n\n![LLM Interview Questions and Answers Book by Kalyan KS](https://github.com/KalyanKS-NLP/llm-engineer-toolkit/blob/main/Images/LLM_Interview_QA_Book_Image_Compress.png) \n\n## Related Repositories\n- ๐Ÿ‘จ๐Ÿปโ€๐Ÿ’ป[LLM Interview Questions and Answers Hub](https://github.com/KalyanKS-NLP/LLM-Interview-Questions-and-Answers-Hub) - 100+ LLM interview questions and answers (basic to advanced). \n- ๐Ÿš€[Prompt Engineering Techniques Hub](https://github.com/KalyanKS-NLP/Prompt-Engineering-Techniques-Hub) - 25+ prompt engineering techniques with LangChain implementations.\n- ๐Ÿฉธ[LLM, RAG and Agents Survey Papers Collection](https://github.com/KalyanKS-NLP/LLM-Survey-Papers-Collection) - Category wise collection of 200+ survey papers.\n\n\n## Stay Updated with Generative AI, LLMs, Agents and RAG.\n\nJoin ๐Ÿš€ [**AIxFunda** free newsletter](https://aixfunda.substack.com/) to get *latest updates* and *interesting tutorials* related to Generative AI, LLMs, Agents and RAG. \n- โœจ Weekly GenAI updates\n- ๐Ÿ“„ Weekly LLM, Agents and RAG paper updates\n- ๐Ÿ“ 1 fresh blog post on an interesting topic every week\n\n## Quick links\n||||\n|---|---|---|\n| [๐Ÿš€ LLM Training](#llm-training-and-fine-tuning) | [๐Ÿงฑ LLM Application Development](#llm-application-development) | [๐ŸฉธLLM RAG](#llm-rag) | \n| [๐ŸŸฉ LLM Inference](#llm-inference)| [๐Ÿšง LLM Serving](#llm-serving) | [๐Ÿ“ค LLM Data Extraction](#llm-data-extraction) |\n| [๐ŸŒ  LLM Data Generation](#llm-data-generation) | [๐Ÿ’Ž LLM Agents](#llm-agents)|[โš–๏ธ LLM Evaluation](#llm-evaluation) | \n| [๐Ÿ” LLM Monitoring](#llm-monitoring) | [๐Ÿ“… LLM Prompts](#llm-prompts) | [๐Ÿ“ LLM Structured Outputs](#llm-structured-outputs) |\n| [๐Ÿ›‘ LLM Safety and Security](#llm-safety-and-security) | [๐Ÿ’  LLM Embedding Models](#llm-embedding-models) | [โ‡๏ธ Others](#others) |\n\n\n\n## LLM Training and Fine-Tuning\n| Library | Description | Link |\n|---------------------|-------------------------------------------------------------------------------------------------|------|\n| unsloth | Fine-tune LLMs faster with less memory. | [Link](https://github.com/unslothai/unsloth) |\n| PEFT | State-of-the-art Parameter-Efficient Fine-Tuning library. | [Link](https://github.com/huggingface/peft) |\n| TRL | Train transformer language models with reinforcement learning. | [Link](https://github.com/huggingface/trl) |\n| Transformers | Transformers provides thousands of pretrained models to perform tasks on different modalities such as text, vision, and audio. | [Link](https://github.com/huggingface/transformers) |\n| Axolotl | Tool designed to streamline post-training for various AI models. | [Link](https://github.com/axolotl-ai-cloud/axolotl/) |\n| LLMBox | A comprehensive library for implementing LLMs, including a unified training pipeline and comprehensive model evaluation. | [Link](https://github.com/RUCAIBox/LLMBox) |\n| LitGPT | Train and fine-tune LLM lightning fast. | [Link](https://github.com/Lightning-AI/litgpt) |\n| Mergoo | A library for easily merging multiple LLM experts, and efficiently train the merged LLM. | [Link](https://github.com/Leeroo-AI/mergoo) |\n| Llama-Factory | Easy and efficient LLM fine-tuning. | [Link](https://github.com/hiyouga/LLaMA-Factory) |\n| Ludwig | Low-code framework for building custom LLMs, neural networks, and other AI models. | [Link](https://github.com/ludwig-ai/ludwig) |\n| Txtinstruct | A framework for training instruction-tuned models. | [Link](https://github.com/neuml/txtinstruct) |\n| Lamini | An integrated LLM inference and tuning platform. | [Link](https://github.com/lamini-ai/lamini) |\n| XTuring | xTuring provides fast, efficient and simple fine-tuning of open-source LLMs, such as Mistral, LLaMA, GPT-J, and more. | [Link](https://github.com/stochasticai/xTuring) |\n| RL4LMs | A modular RL library to fine-tune language models to human preferences. | [Link](https://github.com/allenai/RL4LMs) |\n| DeepSpeed | DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective. | [Link](https://github.com/deepspeedai/DeepSpeed) |\n| torchtune | A PyTorch-native library specifically designed for fine-tuning LLMs. | [Link](https://github.com/pytorch/torchtune) |\n| PyTorch Lightning | A library that offers a high-level interface for pretraining and fine-tuning LLMs. | [Link](https://github.com/Lightning-AI/pytorch-lightning) |\n\n\n## LLM Application Development\n

Frameworks

\n\n| Library | Description | Link |\n|--------------|------------------------------------------------------------------------------------------------------|-------|\n| LangChain | LangChain is a framework for developing applications powered by large language models (LLMs). | [Link](https://github.com/langchain-ai/langchain) |\n| Llama Index | LlamaIndex is a data framework for your LLM applications. | [Link](https://github.com/run-llama/llama_index) |\n| HayStack | Haystack is an end-to-end LLM framework that allows you to build applications powered by LLMs, Transformer models, vector search and more. | [Link](https://github.com/deepset-ai/haystack) |\n| Prompt flow | A suite of development tools designed to streamline the end-to-end development cycle of LLM-based AI applications. | [Link](https://github.com/microsoft/promptflow) |\n| Griptape | A modular Python framework for building AI-powered applications. | [Link](https://github.com/griptape-ai/griptape) |\n| Weave | Weave is a toolkit for developing Generative AI applications. | [Link](https://github.com/wandb/weave) |\n| Llama Stack | Build Llama Apps. | [Link](https://github.com/meta-llama/llama-stack) |\n\n\n

Data Preparation

\n\n| Library | Description | Link |\n|--------------|------------------------------------------------------------------------------------------------------|-------|\n| Data Prep Kit | Data Prep Kit accelerates unstructured data preparation for LLM app developers. Developers can use Data Prep Kit to cleanse, transform, and enrich use case-specific unstructured data to pre-train LLMs, fine-tune LLMs, instruct-tune LLMs, or build RAG applications. | [Link](https://github.com/data-prep-kit/data-prep-kit) | \n\n

Multi API Access

\n\n| Library | Description | Link |\n|--------------|------------------------------------------------------------------------------------------------------|-------|\n| LiteLLM | Library to call 100+ LLM APIs in OpenAI format. | [Link](https://github.com/BerriAI/litellm) |\n| AI Gateway | A Blazing Fast AI Gateway with integrated Guardrails. Route to 200+ LLMs, 50+ AI Guardrails with 1 fast & friendly API. | [Link](https://github.com/Portkey-AI/gateway) |\n\n

Routers

\n\n| Library | Description | Link |\n|--------------|------------------------------------------------------------------------------------------------------|-------|\n| RouteLLM | Framework for serving and evaluating LLM routers - save LLM costs without compromising quality. Drop-in replacement for OpenAI's client to route simpler queries to cheaper models. | [Link](https://github.com/lm-sys/RouteLLM) |\n\n\n

Memory

\n\n| Library | Description | Link |\n|--------------|------------------------------------------------------------------------------------------------------|-------|\n| mem0 | The Memory layer for your AI apps. | [Link](https://github.com/mem0ai/mem0) |\n| Memoripy | An AI memory layer with short- and long-term storage, semantic clustering, and optional memory decay for context-aware applications. | [Link](https://github.com/caspianmoon/memoripy) |\n| Letta (MemGPT) | An open-source framework for building stateful LLM applications with advanced reasoning capabilities and transparent long-term memory | [Link](https://github.com/letta-ai/letta) |\n| Memobase | A user profile-based memory system designed to bring long-term user memory to your Generative AI applications. | [Link](https://github.com/memodb-io/memobase) |\n\n

Interface

\n\n| Library | Description | Link |\n|--------------|------------------------------------------------------------------------------------------------------|-------|\n| Streamlit | A faster way to build and share data apps. Streamlit lets you transform Python scripts into interactive web apps in minutes | [Link](https://github.com/streamlit/streamlit) |\n| Gradio | Build and share delightful machine learning apps, all in Python. | [Link](https://github.com/gradio-app/gradio) |\n| AI SDK UI | Build chat and generative user interfaces. | [Link](https://sdk.vercel.ai/docs/introduction) |\n| AI-Gradio | Create AI apps powered by various AI providers. | [Link](https://github.com/AK391/ai-gradio) |\n| Simpleaichat | Python package for easily interfacing with chat apps, with robust features and minimal code complexity. | [Link](https://github.com/minimaxir/simpleaichat) |\n| Chainlit | Build production-ready Conversational AI applications in minutes. | [Link](https://github.com/Chainlit/chainlit) |\n\n\n

Low Code

\n\n| Library | Description | Link |\n|--------------|------------------------------------------------------------------------------------------------------|-------|\n| LangFlow | LangFlow is a low-code app builder for RAG and multi-agent AI applications. Itโ€™s Python-based and agnostic to any model, API, or database. | [Link](https://github.com/langflow-ai/langflow) |\n\n

Cache

\n\n| Library | Description | Link |\n|--------------|------------------------------------------------------------------------------------------------------|-------|\n| GPTCache | A Library for Creating Semantic Cache for LLM Queries. Slash Your LLM API Costs by 10x ๐Ÿ’ฐ, Boost Speed by 100x. Fully integrated with LangChain and LlamaIndex. | [Link](https://github.com/zilliztech/gptcache) |\n\n\n## LLM RAG\n\n| Library | Description | Link |\n|---------------|----------------------------------------------------------------------------------------------------------------|-------|\n| FastGraph RAG | Streamlined and promptable Fast GraphRAG framework designed for interpretable, high-precision, agent-driven retrieval workflows. | [Link](https://github.com/circlemind-ai/fast-graphrag) |\n| Chonkie | RAG chunking library that is lightweight, lightning-fast, and easy to use. | [Link](https://github.com/chonkie-ai/chonkie) |\n| RAGChecker | A Fine-grained Framework For Diagnosing RAG. | [Link](https://github.com/amazon-science/RAGChecker) |\n| RAG to Riches | Build, scale, and deploy state-of-the-art Retrieval-Augmented Generation applications. | [Link](https://github.com/SciPhi-AI/R2R) |\n| BeyondLLM | Beyond LLM offers an all-in-one toolkit for experimentation, evaluation, and deployment of Retrieval-Augmented Generation (RAG) systems. | [Link](https://github.com/aiplanethub/beyondllm) |\n| SQLite-Vec | A vector search SQLite extension that runs anywhere! | [Link](https://github.com/asg017/sqlite-vec) |\n| fastRAG | fastRAG is a research framework for efficient and optimized retrieval-augmented generative pipelines, incorporating state-of-the-art LLMs and Information Retrieval. | [Link](https://github.com/IntelLabs/fastRAG) |\n| FlashRAG | A Python Toolkit for Efficient RAG Research. | [Link](https://github.com/RUC-NLPIR/FlashRAG) |\n| Llmware | Unified framework for building enterprise RAG pipelines with small, specialized models. | [Link](https://github.com/llmware-ai/llmware) |\n| Rerankers | A lightweight unified API for various reranking models. | [Link](https://github.com/AnswerDotAI/rerankers) |\n| Vectara | Build Agentic RAG applications. | [Link](https://vectara.github.io/py-vectara-agentic/latest/) |\n\n\n## LLM Inference\n\n| Library | Description | Link |\n|---------------|------------------------------------------------------------------------------------------------------|-------|\n| llama.cpp | LLM inference in C/C++. | [Link](https://github.com/ggml-org/llama.cpp) | \n| Ollama | Local LLM inference. | [Link](https://github.com/ollama/ollama) | \n| vLLM | High-throughput and memory-efficient inference and serving engine for LLMs. | [Link](https://github.com/vllm-project/vllm) |\n| TensorRT-LLM | TensorRT-LLM is a library for optimizing Large Language Model (LLM) inference. | [Link](https://github.com/NVIDIA/TensorRT-LLM) |\n| WebLLM | High-performance In-browser LLM Inference Engine. | [Link](https://github.com/mlc-ai/web-llm) |\n| LLM Compressor | Transformers-compatible library for applying various compression algorithms to LLMs for optimized deployment. | [Link](https://github.com/vllm-project/llm-compressor) |\n| LightLLM | Python-based LLM inference and serving framework, notable for its lightweight design, easy scalability, and high-speed performance. | [Link](https://github.com/ModelTC/lightllm) |\n| torchchat | Run PyTorch LLMs locally on servers, desktop, and mobile. | [Link](https://github.com/pytorch/torchchat) |\n\n\n## LLM Serving\n\n| Library | Description | Link |\n|-----------|--------------------------------------------------------------------------|-------|\n| Langcorn | Serving LangChain LLM apps and agents automagically with FastAPI. | [Link](https://github.com/msoedov/langcorn) |\n| LitServe | Lightning-fast serving engine for any AI model of any size. It augments FastAPI with features like batching, streaming, and GPU autoscaling. | [Link](https://github.com/Lightning-AI/LitServe) |\n\n\n## LLM Data Extraction\n\n| Library | Description | Link |\n|----------------|---------------------------------------------------------------------------------------------------------------------------------------|-------|\n| Crawl4AI | Open-source LLM Friendly Web Crawler & Scraper. | [Link](https://github.com/unclecode/crawl4ai) |\n| ScrapeGraphAI | A web scraping Python library that uses LLM and direct graph logic to create scraping pipelines for websites and local documents (XML, HTML, JSON, Markdown, etc.). | [Link](https://github.com/ScrapeGraphAI/Scrapegraph-ai) |\n| Docling | Docling parses documents and exports them to the desired format with ease and speed. | [Link](https://github.com/DS4SD/docling) |\n| Llama Parse | GenAI-native document parser that can parse complex document data for any downstream LLM use case (RAG, agents). | [Link](https://github.com/run-llama/llama_cloud_services) |\n| PyMuPDF4LLM | PyMuPDF4LLM library makes it easier to extract PDF content in the format you need for LLM & RAG environments. | [Link](https://pymupdf.readthedocs.io/en/latest/pymupdf4llm/) |\n| Crawlee | A web scraping and browser automation library. | [Link](https://github.com/apify/crawlee-python) |\n| MegaParse | Parser for every type of document. | [Link](https://github.com/quivrhq/megaparse) |\n| ExtractThinker | Document Intelligence library for LLMs. | [Link](https://github.com/enoch3712/ExtractThinker) |\n\n\n## LLM Data Generation\n\n| Library | Description | Link |\n|--------------|--------------------------------------------------------------------------------------------------|-------|\n| DataDreamer | DataDreamer is a powerful open-source Python library for prompting, synthetic data generation, and training workflows. | [Link](https://github.com/datadreamer-dev/DataDreamer) |\n| fabricator | A flexible open-source framework to generate datasets with large language models. | [Link](https://github.com/flairNLP/fabricator) |\n| Promptwright | Synthetic Dataset Generation Library. | [Link](https://github.com/stacklok/promptwright) |\n| EasyInstruct | An Easy-to-use Instruction Processing Framework for Large Language Models. | [Link](https://github.com/zjunlp/EasyInstruct) |\n\n\n## LLM Agents\n\n| Library | Description | Link |\n|----------------|---------------------------------------------------------------------------------------------------------|-------|\n| CrewAI | Framework for orchestrating role-playing, autonomous AI agents. | [Link](https://github.com/crewAIInc/crewAI) |\n| LangGraph | Build resilient language agents as graphs. | [Link](https://github.com/langchain-ai/langgraph) |\n| Agno | Build AI Agents with memory, knowledge, tools, and reasoning. Chat with them using a beautiful Agent UI. | [Link](https://github.com/agno-agi/agno) |\n| Agents SDK | Build agentic apps using LLMs with context, tools, hand off to other specialized agents. | [Link](https://platform.openai.com/docs/guides/agents-sdk) |\n| AutoGen | An open-source framework for building AI agent systems. | [Link](https://github.com/microsoft/autogen) |\n| Smolagents | Library to build powerful agents in a few lines of code. | [Link](https://github.com/huggingface/smolagents) |\n| Pydantic AI | Python agent framework to build production grade applications with Generative AI. | [Link](https://ai.pydantic.dev/) |\n| CAMEL | Open-source multi-agent framework with various toolkits and use-cases available. | [Link](https://github.com/camel-ai/camel) |\n| BeeAI | Build production-ready multi-agent systems in Python. | [Link](https://github.com/i-am-bee/beeai-framework/tree/main/python) | \n| gradio-tools | A Python library for converting Gradio apps into tools that can be leveraged by an LLM-based agent to complete its task. | [Link](https://github.com/freddyaboulton/gradio-tools) |\n| Composio | Production Ready Toolset for AI Agents. | [Link](https://github.com/ComposioHQ/composio) |\n| Atomic Agents | Building AI agents, atomically. | [Link](https://github.com/BrainBlend-AI/atomic-agents) |\n| Memary | Open Source Memory Layer For Autonomous Agents. | [Link](https://github.com/kingjulio8238/Memary) |\n| Browser Use | Make websites accessible for AI agents. | [Link](https://github.com/browser-use/browser-use) |\n| OpenWebAgent | An Open Toolkit to Enable Web Agents on Large Language Models. | [Link](https://github.com/THUDM/OpenWebAgent/) |\n| Lagent | A lightweight framework for building LLM-based agents. | [Link](https://github.com/InternLM/lagent) |\n| LazyLLM | A Low-code Development Tool For Building Multi-agent LLMs Applications. | [Link](https://github.com/LazyAGI/LazyLLM) |\n| Swarms | The Enterprise-Grade Production-Ready Multi-Agent Orchestration Framework. | [Link](https://github.com/kyegomez/swarms) |\n| ChatArena | ChatArena is a library that provides multi-agent language game environments and facilitates research about autonomous LLM agents and their social interactions. | [Link](https://github.com/Farama-Foundation/chatarena) |\n| Swarm | Educational framework exploring ergonomic, lightweight multi-agent orchestration. | [Link](https://github.com/openai/swarm) |\n| AgentStack | The fastest way to build robust AI agents. | [Link](https://github.com/AgentOps-AI/AgentStack) |\n| Archgw | Intelligent gateway for Agents. | [Link](https://github.com/katanemo/archgw) |\n| Flow | A lightweight task engine for building AI agents. | [Link](https://github.com/lmnr-ai/flow) |\n| AgentOps | Python SDK for AI agent monitoring. | [Link](https://github.com/AgentOps-AI/agentops) |\n| Langroid | Multi-Agent framework. | [Link](https://github.com/langroid/langroid) |\n| Agentarium | Framework for creating and managing simulations populated with AI-powered agents. | [Link](https://github.com/Thytu/Agentarium) |\n| Upsonic | Reliable AI agent framework that supports MCP. | [Link](https://github.com/upsonic/upsonic) |\n\n\n## LLM Evaluation\n\n| Library | Description | Link |\n|------------|-----------------------------------------------------------------------------------------------------------------|-------|\n| Ragas | Ragas is your ultimate toolkit for evaluating and optimizing Large Language Model (LLM) applications. | [Link](https://github.com/explodinggradients/ragas) |\n| Giskard | Open-Source Evaluation & Testing for ML & LLM systems. | [Link](https://github.com/Giskard-AI/giskard) |\n| DeepEval | LLM Evaluation Framework | [Link](https://github.com/confident-ai/deepeval) |\n| Lighteval | All-in-one toolkit for evaluating LLMs. | [Link](https://github.com/huggingface/lighteval) |\n| Trulens | Evaluation and Tracking for LLM Experiments | [Link](https://github.com/truera/trulens) | \n| PromptBench | A unified evaluation framework for large language models. | [Link](https://github.com/microsoft/promptbench) |\n| LangTest | Deliver Safe & Effective Language Models. 60+ Test Types for Comparing LLM & NLP Models on Accuracy, Bias, Fairness, Robustness & More. | [Link](https://github.com/JohnSnowLabs/langtest) |\n| EvalPlus | A rigorous evaluation framework for LLM4Code. | [Link](https://github.com/evalplus/evalplus) |\n| FastChat | An open platform for training, serving, and evaluating large language model-based chatbots. | [Link](https://github.com/lm-sys/FastChat) |\n| judges | A small library of LLM judges. | [Link](https://github.com/quotient-ai/judges) |\n| Evals | Evals is a framework for evaluating LLMs and LLM systems, and an open-source registry of benchmarks. | [Link](https://github.com/openai/evals) |\n| AgentEvals | Evaluators and utilities for evaluating the performance of your agents. | [Link](https://github.com/langchain-ai/agentevals) |\n| LLMBox | A comprehensive library for implementing LLMs, including a unified training pipeline and comprehensive model evaluation. | [Link](https://github.com/RUCAIBox/LLMBox) |\n| Opik | An open-source end-to-end LLM Development Platform which also includes LLM evaluation. | [Link](https://github.com/comet-ml/opik) |\n| PydanticAI Evals | A powerful evaluation framework designed to help you systematically evaluate the performance of LLM applications. | [Link](https://ai.pydantic.dev/evals/) |\n| UQLM | A Python package for generation-time, zero-resource LLM hallucination using state-of-the-art uncertainty quantification techniques. | [Link](https://github.com/cvs-health/uqlm) |\n\n\n\n## LLM Monitoring\n\n| Library | Description | Link |\n|----------------------|-------------------------------------------------------------------------------------------------|-------|\n| MLflow | An open-source end-to-end MLOps/LLMOps Platform for tracking, evaluating, and monitoring LLM applications. | [Link](https://github.com/mlflow/mlflow) |\n| Opik | An open-source end-to-end LLM Development Platform which also includes LLM monitoring. | [Link](https://github.com/comet-ml/opik) |\n| LangSmith | Provides tools for logging, monitoring, and improving your LLM applications. | [Link](https://github.com/langchain-ai/langsmith-sdk) |\n| Weights & Biases (W&B) | W&B provides features for tracking LLM performance. | [Link](https://github.com/wandb) |\n| Helicone | Open source LLM-Observability Platform for Developers. One-line integration for monitoring, metrics, evals, agent tracing, prompt management, playground, etc. | [Link](https://github.com/Helicone/helicone) |\n| Evidently | An open-source ML and LLM observability framework. | [Link](https://github.com/evidentlyai/evidently) |\n| Phoenix | An open-source AI observability platform designed for experimentation, evaluation, and troubleshooting. | [Link](https://github.com/Arize-ai/phoenix) |\n| Observers | A Lightweight Library for AI Observability. | [Link](https://github.com/cfahlgren1/observers) |\n\n\n## LLM Prompts\n\n| Library | Description | Link |\n|---------------------|----------------------------------------------------------------------------------------------------------------|-------|\n| PCToolkit | A Unified Plug-and-Play Prompt Compression Toolkit of Large Language Models. | [Link](https://github.com/3DAgentWorld/Toolkit-for-Prompt-Compression) |\n| Selective Context | Selective Context compresses your prompt and context to allow LLMs (such as ChatGPT) to process 2x more content. | [Link](https://pypi.org/project/selective-context/) |\n| LLMLingua | Library for compressing prompts to accelerate LLM inference. | [Link](https://github.com/microsoft/LLMLingua) |\n| betterprompt | Test suite for LLM prompts before pushing them to production. | [Link](https://github.com/stjordanis/betterprompt) |\n| Promptify | Solve NLP Problems with LLMs & easily generate different NLP Task prompts for popular generative models like GPT, PaLM, and more with Promptify. | [Link](https://github.com/promptslab/Promptify) |\n| PromptSource | PromptSource is a toolkit for creating, sharing, and using natural language prompts. | [Link](https://pypi.org/project/promptsource/) |\n| DSPy | DSPy is the open-source framework for programmingโ€”rather than promptingโ€”language models. | [Link](https://github.com/stanfordnlp/dspy) |\n| Py-priompt | Prompt design library. | [Link](https://github.com/zenbase-ai/py-priompt) |\n| Promptimizer | Prompt optimization library. | [Link](https://github.com/hinthornw/promptimizer) |\n\n\n## LLM Structured Outputs\n| Library |\tDescription |\tLink |\n|------------|--------------------------------------------------------|------|\n|Instructor |\tPython library for working with structured outputs from large language models (LLMs). Built on top of Pydantic, it provides a simple, transparent, and user-friendly API. | [Link](https://github.com/instructor-ai/instructor) |\n| XGrammar | An open-source library for efficient, flexible, and portable structured generation. | [Link](https://github.com/mlc-ai/xgrammar) |\n| Outlines | Robust (structured) text generation | [Link](https://github.com/dottxt-ai/outlines) |\n| Guidance | Guidance is an efficient programming paradigm for steering language models. | [Link](https://github.com/guidance-ai/guidance) |\n| LMQL | A language for constraint-guided and efficient LLM programming. | [Link](https://github.com/eth-sri/lmql) |\n| Jsonformer | A Bulletproof Way to Generate Structured JSON from Language Models. | [Link](https://github.com/1rgs/jsonformer) |\n\n\n## LLM Safety and Security\n| Library | Description | Link |\n|---------------|-----------------------------------------------------------|------|\n| JailbreakEval | A collection of automated evaluators for assessing jailbreak attempts. | [Link](https://github.com/ThuCCSLab/JailbreakEval) |\n| EasyJailbreak | An easy-to-use Python framework to generate adversarial jailbreak prompts. | [Link](https://github.com/EasyJailbreak/EasyJailbreak) |\n| Guardrails | Adding guardrails to large language models. | [Link](https://github.com/guardrails-ai/guardrails) |\n| LLM Guard | The Security Toolkit for LLM Interactions. | [Link](https://github.com/protectai/llm-guard) |\n| AuditNLG | AuditNLG is an open-source library that can help reduce the risks associated with using generative AI systems for language. | [Link](https://github.com/salesforce/AuditNLG) |\n| NeMo Guardrails | NeMo Guardrails is an open-source toolkit for easily adding programmable guardrails to LLM-based conversational systems. | [Link](https://github.com/NVIDIA/NeMo-Guardrails) |\n| Garak | LLM vulnerability scanner | [Link](https://github.com/NVIDIA/garak) |\n| DeepTeam | The LLM Red Teaming Framework | [Link](https://github.com/confident-ai/deepteam)|\n\n\n## LLM Embedding Models\n| Library | Description | Link |\n|---------------------------|-----------------------------------------------------|------|\n| Sentence-Transformers | State-of-the-Art Text Embeddings | [Link](https://github.com/UKPLab/sentence-transformers) |\n| Model2Vec | Fast State-of-the-Art Static Embeddings | [Link](https://github.com/MinishLab/model2vec) |\n| Text Embedding Inference | A blazing fast inference solution for text embeddings models. TEI enables high-performance extraction for the most popular models, including FlagEmbedding, Ember, GTE and E5. | [Link](https://github.com/huggingface/text-embeddings-inference) |\n\n\n## Others\n| Library | Description | Link |\n|-------------------------|----------------------------------------------------------------------------------------------------------------------------------|------|\n| Text Machina | A modular and extensible Python framework, designed to aid in the creation of high-quality, unbiased datasets to build robust models for MGT-related tasks such as detection, attribution, and boundary detection. | [Link](https://github.com/Genaios/TextMachina) |\n| LLM Reasoners | A library for advanced large language model reasoning. | [Link](https://github.com/maitrix-org/llm-reasoners) |\n| EasyEdit | An Easy-to-use Knowledge Editing Framework for Large Language Models. | [Link](https://github.com/zjunlp/EasyEdit) |\n| CodeTF | CodeTF: One-stop Transformer Library for State-of-the-art Code LLM. | [Link](https://github.com/salesforce/CodeTF) |\n| spacy-llm | This package integrates Large Language Models (LLMs) into spaCy, featuring a modular system for fast prototyping and prompting, and turning unstructured responses into robust outputs for various NLP tasks. | [Link](https://github.com/explosion/spacy-llm) |\n| pandas-ai | Chat with your database (SQL, CSV, pandas, polars, MongoDB, NoSQL, etc.). | [Link](https://github.com/Sinaptik-AI/pandas-ai) |\n| LLM Transparency Tool | An open-source interactive toolkit for analyzing internal workings of Transformer-based language models. | [Link](https://github.com/facebookresearch/llm-transparency-tool) |\n| Vanna | Chat with your SQL database. Accurate Text-to-SQL Generation via LLMs using RAG. | [Link](https://github.com/vanna-ai/vanna) |\n| mergekit | Tools for merging pretrained large language models. | [Link](https://github.com/arcee-ai/MergeKit) |\n| MarkLLM | An Open-Source Toolkit for LLM Watermarking. | [Link](https://github.com/THU-BPM/MarkLLM) |\n| LLMSanitize | An open-source library for contamination detection in NLP datasets and Large Language Models (LLMs). | [Link](https://github.com/ntunlp/LLMSanitize) |\n| Annotateai | Automatically annotate papers using LLMs. | [Link](https://github.com/neuml/annotateai) |\n| LLM Reasoner | Make any LLM think like OpenAI o1 and DeepSeek R1. | [Link](https://github.com/harishsg993010/LLM-Reasoner) |\n\n\n## โญ๏ธ Star History\n\n[![Star History Chart](https://api.star-history.com/svg?repos=KalyanKS-NLP/llm-engineer-toolkit&type=Date)](https://star-history.com/#)\n\nPlease consider giving a star, if you find this repository useful. \n\n" } ]