[
  {
    "path": "Dockerfile",
    "content": "FROM python:3.8-slim-buster as langchain-serve-img\n\nRUN pip3 install langchain-serve\nRUN pip3 install api\n\nCMD [ \"lc-serve\", \"deploy\", \"local\", \"api\" ]\n\nFROM python:3.8-slim-buster as pdf-gpt-img\n\nWORKDIR /app\n\nCOPY requirements.txt requirements.txt\nRUN pip3 install -r requirements.txt\n\nCOPY . .\n\nCMD [ \"python3\", \"app.py\" ]\n"
  },
  {
    "path": "LICENSE.txt",
    "content": "MIT License\n\nCopyright (c) [2023] [Bhaskar Tripathi]\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\nSOFTWARE.\n"
  },
  {
    "path": "README.md",
    "content": "# pdfGPT\n## Demo\n1. **Demo URL**: https://huggingface.co/spaces/bhaskartripathi/pdfChatter\n2. **Demo Video**:\n   \n   [![IMAGE ALT TEXT HERE](https://img.youtube.com/vi/LzPgmmqpBk8/0.jpg)](https://www.youtube.com/watch?v=LzPgmmqpBk8)\n3. Despite so many fancy RAG solutions out there in Open Source and enterprise apps, pdfGPT is still the most accurate application that gives the most precise response. The first version was developed way back in 2021 as one of the world's earliest RAG open source solutions. To this day (Dec, 2024), it still remains one of the most accurate ones due to its very simple and unique architecture. It uses no third-party APIs such as langchain. It uses embeddings but no vectorDB, no indexing. But, it still doesn't compromise on the accuracy of response which is more critical than a fancy UI. The library documentation that you see below is a bit outdated as I do not get enough time to maintain it. However, if there is more demand then I am ready to put an enterprise grade RAG with more sophisticated retrieval tech available these days.\n   \n#### Version Updates (27 July, 2023):\n1. Improved error handling\n2. PDF GPT now supports Turbo models and GPT4 including 16K and 32K token model.\n3. Pre-defined questions for auto-filling the input.\n4. Implemented Chat History feature.\n![image](https://github.com/bhaskatripathi/pdfGPT/assets/35177508/11549b24-9ed4-4dcb-a877-bad9c2266bf9)\n\n\n### Note on model performance\n```If you find the response for a specific question in the PDF is not good using Turbo models, then you need to understand that Turbo models such as gpt-3.5-turbo are chat completion models and will not give a good response in some cases where the embedding similarity is low. Despite the claim by OpenAI, the turbo model is not the best model for Q&A. In those specific cases, either use the good old text-DaVinci-003 or use GPT4 and above. These models invariably give you the most relevant output.```\n\n# Upcoming Release Pipeline:\n1. Support for Falcon, Vicuna, Meta Llama\n2. OCR Support\n3. Multiple PDF file support\n4. OCR Support\n5. Node.Js based Web Application - With no trial, no API fees. 100% Open source.\n    \n### Problem Description : \n1. When you pass a large text to Open AI, it suffers from a 4K token limit. It cannot take an entire pdf file as an input\n2. Open AI sometimes becomes overtly chatty and returns irrelevant response not directly related to your query. This is because Open AI uses poor embeddings.\n3. ChatGPT cannot directly talk to external data. Some solutions use Langchain but it is token hungry if not implemented correctly.\n4. There are a number of solutions like https://www.chatpdf.com, https://www.bespacific.com/chat-with-any-pdf, https://www.filechat.io they have poor content quality and are prone to hallucination problem. One good way to avoid hallucinations and improve truthfulness is to use improved embeddings. To solve this problem, I propose to improve embeddings with Universal Sentence Encoder family of algorithms (Read more here: https://tfhub.dev/google/collections/universal-sentence-encoder/1). \n\n### Solution: What is PDF GPT ?\n1. PDF GPT allows you to chat with an uploaded PDF file using GPT functionalities.\n2. The application intelligently breaks the document into smaller chunks and employs a powerful Deep Averaging Network Encoder to generate embeddings.\n3. A semantic search is first performed on your pdf content and the most relevant embeddings are passed to the Open AI.\n4. A custom logic generates precise responses. The returned response can even cite the page number in square brackets([]) where the information is located, adding credibility to the responses and helping to locate pertinent information quickly. The Responses are much better than the naive responses by Open AI.\n5. Andrej Karpathy mentioned in this post that KNN algorithm is most appropriate for similar problems: https://twitter.com/karpathy/status/1647025230546886658\n6. Enables APIs on Production using **[langchain-serve](https://github.com/jina-ai/langchain-serve)**.\n\n### Docker\nRun `docker-compose -f docker-compose.yaml up` to use it with Docker compose.\n\n\n## UML\n```mermaid\nsequenceDiagram\n    participant User\n    participant System\n\n    User->>System: Enter API Key\n    User->>System: Upload PDF/PDF URL\n    User->>System: Ask Question\n    User->>System: Submit Call to Action\n\n    System->>System: Blank field Validations\n    System->>System: Convert PDF to Text\n    System->>System: Decompose Text to Chunks (150 word length)\n    System->>System: Check if embeddings file exists\n    System->>System: If file exists, load embeddings and set the fitted attribute to True\n    System->>System: If file doesn't exist, generate embeddings, fit the recommender, save embeddings to file and set fitted attribute to True\n    System->>System: Perform Semantic Search and return Top 5 Chunks with KNN\n    System->>System: Load Open AI prompt\n    System->>System: Embed Top 5 Chunks in Open AI Prompt\n    System->>System: Generate Answer with Davinci\n\n    System-->>User: Return Answer\n```\n\n### Flowchart\n```mermaid\nflowchart TB\nA[Input] --> B[URL]\nA -- Upload File manually --> C[Parse PDF]\nB --> D[Parse PDF] -- Preprocess --> E[Dynamic Text Chunks]\nC -- Preprocess --> E[Dynamic Text Chunks with citation history]\nE --Fit-->F[Generate text embedding with Deep Averaging Network Encoder on each chunk]\nF -- Query --> G[Get Top Results]\nG -- K-Nearest Neighbour --> K[Get Nearest Neighbour - matching citation references]\nK -- Generate Prompt --> H[Generate Answer]\nH -- Output --> I[Output]\n```\n## Star History\n\n[![Star History Chart](https://api.star-history.com/svg?repos=bhaskatripathi/pdfGPT&type=Date)](https://star-history.com/#bhaskatripathi/pdfGPT&Date)\nI am looking for more contributors from the open source community who can take up backlog items voluntarily and maintain the application jointly with me.\n\n## Also Try this project - No Code LLM Quantization\nRunning Large Language Models on consumer hardware remains a challenge due to their massive size (in gigabytes). Model Quantization is a way to reduce their size while not compromising much on their performance. No-code Model Quantization is highly desired by ML Engineers and AI researchers for quick experimentation. I developed a no-code Model Quantization tool that addresses these key hurdles:\n🔹 Size: Reduces model size, making them feasible to run on consumer-grade hardware.\n🔹 Format: Converts training-optimized formats into ones suitable for efficient inference.\n\nProject Link https://github.com/bhaskatripathi/LLM_Quantization\n\n## License\nThis project is licensed under the MIT License. See the [LICENSE.txt](LICENSE.txt) file for details.\n\n## Citation\nIf you use PDF-GPT in your research or wish to refer to the examples in this repo, please cite with:\n\n```bibtex\n@misc{pdfgpt2023,\n  author = {Bhaskar Tripathi},\n  title = {PDF-GPT},\n  year = {2023},\n  publisher = {GitHub},\n  journal = {GitHub Repository},\n  howpublished = {\\url{https://github.com/bhaskatripathi/pdfGPT}}\n}\n"
  },
  {
    "path": "api.py",
    "content": "import os\nimport re\nimport shutil\nimport urllib.request\nfrom pathlib import Path\nfrom tempfile import NamedTemporaryFile\nfrom litellm import completion\nimport fitz\nimport numpy as np\nimport openai\nimport tensorflow_hub as hub\nfrom fastapi import UploadFile\nfrom lcserve import serving\nfrom sklearn.neighbors import NearestNeighbors\n\n\nrecommender = None\n\n\ndef download_pdf(url, output_path):\n    urllib.request.urlretrieve(url, output_path)\n\n\ndef preprocess(text):\n    text = text.replace('\\n', ' ')\n    text = re.sub('\\s+', ' ', text)\n    return text\n\n\ndef pdf_to_text(path, start_page=1, end_page=None):\n    doc = fitz.open(path)\n    total_pages = doc.page_count\n\n    if end_page is None:\n        end_page = total_pages\n\n    text_list = []\n\n    for i in range(start_page - 1, end_page):\n        text = doc.load_page(i).get_text(\"text\")\n        text = preprocess(text)\n        text_list.append(text)\n\n    doc.close()\n    return text_list\n\n\ndef text_to_chunks(texts, word_length=150, start_page=1):\n    text_toks = [t.split(' ') for t in texts]\n    chunks = []\n\n    for idx, words in enumerate(text_toks):\n        for i in range(0, len(words), word_length):\n            chunk = words[i : i + word_length]\n            if (\n                (i + word_length) > len(words)\n                and (len(chunk) < word_length)\n                and (len(text_toks) != (idx + 1))\n            ):\n                text_toks[idx + 1] = chunk + text_toks[idx + 1]\n                continue\n            chunk = ' '.join(chunk).strip()\n            chunk = f'[Page no. {idx+start_page}]' + ' ' + '\"' + chunk + '\"'\n            chunks.append(chunk)\n    return chunks\n\n\nclass SemanticSearch:\n    def __init__(self):\n        self.use = hub.load('https://tfhub.dev/google/universal-sentence-encoder/4')\n        self.fitted = False\n\n    def fit(self, data, batch=1000, n_neighbors=5):\n        self.data = data\n        self.embeddings = self.get_text_embedding(data, batch=batch)\n        n_neighbors = min(n_neighbors, len(self.embeddings))\n        self.nn = NearestNeighbors(n_neighbors=n_neighbors)\n        self.nn.fit(self.embeddings)\n        self.fitted = True\n\n    def __call__(self, text, return_data=True):\n        inp_emb = self.use([text])\n        neighbors = self.nn.kneighbors(inp_emb, return_distance=False)[0]\n\n        if return_data:\n            return [self.data[i] for i in neighbors]\n        else:\n            return neighbors\n\n    def get_text_embedding(self, texts, batch=1000):\n        embeddings = []\n        for i in range(0, len(texts), batch):\n            text_batch = texts[i : (i + batch)]\n            emb_batch = self.use(text_batch)\n            embeddings.append(emb_batch)\n        embeddings = np.vstack(embeddings)\n        return embeddings\n\n\ndef load_recommender(path, start_page=1):\n    global recommender\n    if recommender is None:\n        recommender = SemanticSearch()\n\n    texts = pdf_to_text(path, start_page=start_page)\n    chunks = text_to_chunks(texts, start_page=start_page)\n    recommender.fit(chunks)\n    return 'Corpus Loaded.'\n\n\ndef generate_text(openAI_key, prompt, engine=\"text-davinci-003\"):\n    # openai.api_key = openAI_key\n    try:\n        messages=[{ \"content\": prompt,\"role\": \"user\"}]\n        completions = completion(\n            model=engine,\n            messages=messages,\n            max_tokens=512,\n            n=1,\n            stop=None,\n            temperature=0.7,\n            api_key=openAI_key\n        )\n        message = completions['choices'][0]['message']['content']\n    except Exception as e:\n        message = f'API Error: {str(e)}'\n    return message \n\n\ndef generate_answer(question, openAI_key):\n    topn_chunks = recommender(question)\n    prompt = \"\"\n    prompt += 'search results:\\n\\n'\n    for c in topn_chunks:\n        prompt += c + '\\n\\n'\n\n    prompt += (\n        \"Instructions: Compose a comprehensive reply to the query using the search results given. \"\n        \"Cite each reference using [ Page Number] notation (every result has this number at the beginning). \"\n        \"Citation should be done at the end of each sentence. If the search results mention multiple subjects \"\n        \"with the same name, create separate answers for each. Only include information found in the results and \"\n        \"don't add any additional information. Make sure the answer is correct and don't output false content. \"\n        \"If the text does not relate to the query, simply state 'Text Not Found in PDF'. Ignore outlier \"\n        \"search results which has nothing to do with the question. Only answer what is asked. The \"\n        \"answer should be short and concise. Answer step-by-step. \\n\\nQuery: {question}\\nAnswer: \"\n    )\n\n    prompt += f\"Query: {question}\\nAnswer:\"\n    answer = generate_text(openAI_key, prompt, \"text-davinci-003\")\n    return answer\n\n\ndef load_openai_key() -> str:\n    key = os.environ.get(\"OPENAI_API_KEY\")\n    if key is None:\n        raise ValueError(\n            \"[ERROR]: Please pass your OPENAI_API_KEY. Get your key here : https://platform.openai.com/account/api-keys\"\n        )\n    return key\n\n\n@serving\ndef ask_url(url: str, question: str):\n    download_pdf(url, 'corpus.pdf')\n    load_recommender('corpus.pdf')\n    openAI_key = load_openai_key()\n    return generate_answer(question, openAI_key)\n\n\n@serving\nasync def ask_file(file: UploadFile, question: str) -> str:\n    suffix = Path(file.filename).suffix\n    with NamedTemporaryFile(delete=False, suffix=suffix) as tmp:\n        shutil.copyfileobj(file.file, tmp)\n        tmp_path = Path(tmp.name)\n\n    load_recommender(str(tmp_path))\n    openAI_key = load_openai_key()\n    return generate_answer(question, openAI_key)\n"
  },
  {
    "path": "app.py",
    "content": "import json\r\nfrom tempfile import _TemporaryFileWrapper\r\n\r\nimport gradio as gr\r\nimport requests\r\n\r\n\r\ndef ask_api(\r\n    lcserve_host: str,\r\n    url: str,\r\n    file: _TemporaryFileWrapper,\r\n    question: str,\r\n    openAI_key: str,\r\n) -> str:\r\n    if not lcserve_host.startswith('http'):\r\n        return '[ERROR]: Invalid API Host'\r\n\r\n    if url.strip() == '' and file == None:\r\n        return '[ERROR]: Both URL and PDF is empty. Provide at least one.'\r\n\r\n    if url.strip() != '' and file != None:\r\n        return '[ERROR]: Both URL and PDF is provided. Please provide only one (either URL or PDF).'\r\n\r\n    if question.strip() == '':\r\n        return '[ERROR]: Question field is empty'\r\n\r\n    _data = {\r\n        'question': question,\r\n        'envs': {\r\n            'OPENAI_API_KEY': openAI_key,\r\n        },\r\n    }\r\n\r\n    if url.strip() != '':\r\n        r = requests.post(\r\n            f'{lcserve_host}/ask_url',\r\n            json={'url': url, **_data},\r\n        )\r\n\r\n    else:\r\n        with open(file.name, 'rb') as f:\r\n            r = requests.post(\r\n                f'{lcserve_host}/ask_file',\r\n                params={'input_data': json.dumps(_data)},\r\n                files={'file': f},\r\n            )\r\n\r\n    if r.status_code != 200:\r\n        raise ValueError(f'[ERROR]: {r.text}')\r\n\r\n    return r.json()['result']\r\n\r\n\r\ntitle = 'PDF GPT'\r\ndescription = \"\"\" PDF GPT allows you to chat with your PDF file using Universal Sentence Encoder and Open AI. It gives hallucination free response than other tools as the embeddings are better than OpenAI. The returned response can even cite the page number in square brackets([]) where the information is located, adding credibility to the responses and helping to locate pertinent information quickly.\"\"\"\r\n\r\nwith gr.Blocks() as demo:\r\n    gr.Markdown(f'<center><h1>{title}</h1></center>')\r\n    gr.Markdown(description)\r\n\r\n    with gr.Row():\r\n        with gr.Group():\r\n            lcserve_host = gr.Textbox(\r\n                label='Enter your API Host here',\r\n                value='http://localhost:8080',\r\n                placeholder='http://localhost:8080',\r\n            )\r\n            gr.Markdown(\r\n                '<p style=\"text-align:center\">Get your Open AI API key <a href=\"https://platform.openai.com/account/api-keys\">here</a></p>'\r\n            )\r\n            openAI_key = gr.Textbox(\r\n                label='Enter your OpenAI API key here', type='password'\r\n            )\r\n            pdf_url = gr.Textbox(label='Enter PDF URL here')\r\n            gr.Markdown(\"<center><h4>OR<h4></center>\")\r\n            file = gr.File(\r\n                label='Upload your PDF/ Research Paper / Book here', file_types=['.pdf']\r\n            )\r\n            question = gr.Textbox(label='Enter your question here')\r\n            btn = gr.Button(value='Submit')\r\n            btn.style(full_width=True)\r\n\r\n        with gr.Group():\r\n            answer = gr.Textbox(label='The answer to your question is :')\r\n\r\n        btn.click(\r\n            ask_api,\r\n            inputs=[lcserve_host, pdf_url, file, question, openAI_key],\r\n            outputs=[answer],\r\n        )\r\n\r\ndemo.app.server.timeout = 60000 # Set the maximum return time for the results of accessing the upstream server\r\n\r\ndemo.launch(server_port=7860, enable_queue=True) # `enable_queue=True` to ensure the validity of multi-user requests\r\n"
  },
  {
    "path": "docker-compose.yaml",
    "content": "version: '3'\n\nservices:\n  langchain-serve:\n    build:\n      context: . \n      target: langchain-serve-img\n    ports:\n      - '8080:8080'\n  pdf-gpt:\n    build:\n      context: . \n      target: pdf-gpt-img\n    ports:\n      - '7860:7860'"
  },
  {
    "path": "requirements.txt",
    "content": "PyMuPDF==1.22.1\r\nnumpy==1.23.5\r\nscikit-learn==1.2.2\r\ntensorflow>=2.0.0\r\ntensorflow_hub==0.13.0\r\nopenai==0.27.4\r\ngradio==4.11.0\r\nlangchain-serve>=0.0.19\r\nlitellm\r\n"
  }
]