![\"Quivr-logo\"](\"./logo.png\"){kind=logo}
\n\n![\"YCombinator\"](\"https://upload.wikimedia.org/wikipedia/commons/thumb/b/b2/Y_Combinator_logo.svg/1200px-Y_Combinator_logo.svg.png\"){kind=logo}
\n\n\n ![\"Theodo\"](\"https://avatars.githubusercontent.com/u/332041?s=200&v=4\")
\n\n\n## License π\n\nThis project is licensed under the Apache 2.0 License - see the [LICENSE](LICENSE) file for details\n"
},
{
"path": "core/.flake8",
"content": "[flake8]\n; Minimal configuration for Flake8 to work with Black.\nmax-line-length = 100\nignore = E101,E111,E112,E221,E222,E501,E711,E712,W503,W504,F401,E203\n"
},
{
"path": "core/.gitignore",
"content": "# python generated files\n__pycache__/\n*.py[oc]\nbuild/\ndist/\nwheels/\n*.egg-info\n\n# venv\n.venv\n"
},
{
"path": "core/.python-version",
"content": "3.11.9\n"
},
{
"path": "core/CHANGELOG.md",
"content": "# Changelog\n\n## [0.0.33](https://github.com/QuivrHQ/quivr/compare/core-0.0.32...core-0.0.33) (2025-02-03)\n\n\n### Features\n\n* **zendesk:** add zendesk workflow ([#3586](https://github.com/QuivrHQ/quivr/issues/3586)) ([ee9b7a5](https://github.com/QuivrHQ/quivr/commit/ee9b7a5740825bd3fc9186e0a9179959c6525e5e))\n\n\n### Bug Fixes\n\n* CLI-24 ([ee9b7a5](https://github.com/QuivrHQ/quivr/commit/ee9b7a5740825bd3fc9186e0a9179959c6525e5e))\n\n## [0.0.32](https://github.com/QuivrHQ/quivr/compare/core-0.0.31...core-0.0.32) (2025-01-31)\n\n\n### Features\n\n* o3-mini ([#3583](https://github.com/QuivrHQ/quivr/issues/3583)) ([a639e0c](https://github.com/QuivrHQ/quivr/commit/a639e0ce50297e0fefa809b7edb57b50863b446d))\n\n## [0.0.31](https://github.com/QuivrHQ/quivr/compare/core-0.0.30...core-0.0.31) (2025-01-30)\n\n\n### Features\n\n* cache tokenizers ([#3558](https://github.com/QuivrHQ/quivr/issues/3558)) ([699dc2e](https://github.com/QuivrHQ/quivr/commit/699dc2e187abc9986845f591111723088f5bcefe))\n* limit tokenizers cache size ([#3577](https://github.com/QuivrHQ/quivr/issues/3577)) ([e2a3bcb](https://github.com/QuivrHQ/quivr/commit/e2a3bcbbdb469348187d986de9ba3901938bed58))\n* remove pympler dependency and add better way to calculate size of tokenizer cache ([#3580](https://github.com/QuivrHQ/quivr/issues/3580)) ([2fbd5d4](https://github.com/QuivrHQ/quivr/commit/2fbd5d48443625dd3fe8a37c04275cd760e7285f))\n* remove tokenizer load ([#3576](https://github.com/QuivrHQ/quivr/issues/3576)) ([05e212a](https://github.com/QuivrHQ/quivr/commit/05e212a30929ba3c00e31e3364363eb4a4376ad9))\n\n## [0.0.30](https://github.com/QuivrHQ/quivr/compare/core-0.0.29...core-0.0.30) (2025-01-27)\n\n\n### Features\n\n* adding cache to LLMEndpoint ([#3555](https://github.com/QuivrHQ/quivr/issues/3555)) ([6072907](https://github.com/QuivrHQ/quivr/commit/6072907ca7370be748d2d6845fd674abbb6c83c3))\n\n## [0.0.29](https://github.com/QuivrHQ/quivr/compare/core-0.0.28...core-0.0.29) (2025-01-20)\n\n\n### Features\n\n* enabling workflows without rewriting step ([#3549](https://github.com/QuivrHQ/quivr/issues/3549)) ([bbe1c18](https://github.com/QuivrHQ/quivr/commit/bbe1c183768bf32945554e679cab737c07bb3dde))\n* improving the prompts to always refer to 'tasks' instead of 'questions' ([#3528](https://github.com/QuivrHQ/quivr/issues/3528)) ([e9c72e1](https://github.com/QuivrHQ/quivr/commit/e9c72e15671407290f1a3a9758bf38a3357d2b15))\n* langfuse integration ([#3530](https://github.com/QuivrHQ/quivr/issues/3530)) ([c4aae1a](https://github.com/QuivrHQ/quivr/commit/c4aae1a6c21fd7bc7019676d32fa5b2e8fbbe171))\n* langfuse user id ([#3533](https://github.com/QuivrHQ/quivr/issues/3533)) ([e0ccd3d](https://github.com/QuivrHQ/quivr/commit/e0ccd3dc04b7527b27520465b2cf179e9789bf3f))\n* language detection after chunking ([#3532](https://github.com/QuivrHQ/quivr/issues/3532)) ([d0adb81](https://github.com/QuivrHQ/quivr/commit/d0adb8112a27fb7f25564d328a6f7e50ba27ba3a))\n* returning a description of each workflow node ([#3539](https://github.com/QuivrHQ/quivr/issues/3539)) ([d835fc6](https://github.com/QuivrHQ/quivr/commit/d835fc6e4c062bd485a715bc707a902493e092c2))\n\n\n### Bug Fixes\n\n* langfuse talk to model ([#3535](https://github.com/QuivrHQ/quivr/issues/3535)) ([9681a9e](https://github.com/QuivrHQ/quivr/commit/9681a9ec8b6b09fe20d04bf41d17a57afc5398f9))\n\n## [0.0.28](https://github.com/QuivrHQ/quivr/compare/core-0.0.27...core-0.0.28) (2024-12-17)\n\n\n### Features\n\n* remove dependencies on Pydantic v1 ([#3526](https://github.com/QuivrHQ/quivr/issues/3526)) ([ebc4eb8](https://github.com/QuivrHQ/quivr/commit/ebc4eb811c258ce0500032bbc52d96f333fabf89))\n\n## [0.0.27](https://github.com/QuivrHQ/quivr/compare/core-0.0.26...core-0.0.27) (2024-12-16)\n\n\n### Features\n\n* ensuring that max_context_tokens is never larger than what supported by models ([#3519](https://github.com/QuivrHQ/quivr/issues/3519)) ([d6e0ed4](https://github.com/QuivrHQ/quivr/commit/d6e0ed44df0ee7edafea85f704a15fd99969bafd))\n* send all to megaparse_sdk ([#3521](https://github.com/QuivrHQ/quivr/issues/3521)) ([e48044d](https://github.com/QuivrHQ/quivr/commit/e48044d36ffda613f65da24641ed8da290195177))\n\n\n### Bug Fixes\n\n* fixing errors arising when the user input contains no tasks ([#3525](https://github.com/QuivrHQ/quivr/issues/3525)) ([e28f7bc](https://github.com/QuivrHQ/quivr/commit/e28f7bcb9ab9534bc011664525ae1f9c2cf6393e))\n\n## [0.0.26](https://github.com/QuivrHQ/quivr/compare/core-0.0.25...core-0.0.26) (2024-12-10)\n\n\n### Features\n\n* first version (V0) of the Workflow Management System ([#3493](https://github.com/QuivrHQ/quivr/issues/3493)) ([6450a49](https://github.com/QuivrHQ/quivr/commit/6450a494e3efa8e8c267ca49aa0a7ec682586b4e))\n\n\n### Bug Fixes\n\n* dealing with empty tool_calls ([#3514](https://github.com/QuivrHQ/quivr/issues/3514)) ([e2f6389](https://github.com/QuivrHQ/quivr/commit/e2f6389189d911a382b2236ab39f28a1270528ac))\n\n## [0.0.25](https://github.com/QuivrHQ/quivr/compare/core-0.0.24...core-0.0.25) (2024-11-28)\n\n\n### Bug Fixes\n\n* megaparse sdk with nats ([#3496](https://github.com/QuivrHQ/quivr/issues/3496)) ([e68b4f4](https://github.com/QuivrHQ/quivr/commit/e68b4f45698898f6b514d4779c8e5fd7332f2e67))\n\n\n### Documentation\n\n* Enhance example/chatbot with added instructions ([#3506](https://github.com/QuivrHQ/quivr/issues/3506)) ([d1d608d](https://github.com/QuivrHQ/quivr/commit/d1d608d19ffb9213910575981eff3527f7d232a0))\n\n## [0.0.24](https://github.com/QuivrHQ/quivr/compare/core-0.0.23...core-0.0.24) (2024-11-14)\n\n\n### Features\n\n* kms-migration ([#3446](https://github.com/QuivrHQ/quivr/issues/3446)) ([1356d87](https://github.com/QuivrHQ/quivr/commit/1356d87098ae84776a5d47b631d07a1c8e92e291))\n* **megaparse:** add sdk ([#3462](https://github.com/QuivrHQ/quivr/issues/3462)) ([190d971](https://github.com/QuivrHQ/quivr/commit/190d971bd71333924b88ba747d3c6a833ca65d92))\n\n\n### Bug Fixes\n\n* added chunk_size in tika processor ([#3466](https://github.com/QuivrHQ/quivr/issues/3466)) ([063bbd3](https://github.com/QuivrHQ/quivr/commit/063bbd323dfca2dfc22fc5416c1617ed61d2e2ab))\n* modify megaparse strategy ([#3474](https://github.com/QuivrHQ/quivr/issues/3474)) ([da97b2c](https://github.com/QuivrHQ/quivr/commit/da97b2cf145c86ed577be698ae837b3dc26f6921))\n* supported extensions for megaparse ([#3477](https://github.com/QuivrHQ/quivr/issues/3477)) ([72b979d](https://github.com/QuivrHQ/quivr/commit/72b979d4e4d6e6efc45d47c7aba942eb909adc3e))\n\n## [0.0.23](https://github.com/QuivrHQ/quivr/compare/core-0.0.22...core-0.0.23) (2024-10-31)\n\n\n### Features\n\n* websearch, tool use, user intent, dynamic retrieval, multiple questions ([#3424](https://github.com/QuivrHQ/quivr/issues/3424)) ([285fe5b](https://github.com/QuivrHQ/quivr/commit/285fe5b96065a19c74f0314557e5840d8722099e))\n\n## [0.0.22](https://github.com/QuivrHQ/quivr/compare/core-0.0.21...core-0.0.22) (2024-10-21)\n\n\n### Features\n\n* **ask:** non-streaming now calls streaming ([#3409](https://github.com/QuivrHQ/quivr/issues/3409)) ([e71e46b](https://github.com/QuivrHQ/quivr/commit/e71e46bcdfbab0d583aef015604278343fd46c6f))\n\n## [0.0.21](https://github.com/QuivrHQ/quivr/compare/core-0.0.20...core-0.0.21) (2024-10-21)\n\n\n### Features\n\n* **ci:** trigger ([b92774a](https://github.com/QuivrHQ/quivr/commit/b92774aa37ad2051b197daa29fe4b94d57a19986))\n\n## [0.0.20](https://github.com/QuivrHQ/quivr/compare/core-0.0.19...core-0.0.20) (2024-10-21)\n\n\n### Features\n\n* **ci:** trigger ([#3403](https://github.com/QuivrHQ/quivr/issues/3403)) ([68c09fc](https://github.com/QuivrHQ/quivr/commit/68c09fce85364432da2641d0a8da867516142553))\n* **docs:** trigger ci ([5644596](https://github.com/QuivrHQ/quivr/commit/56445967252eac20d17bebc4484d7c00c45d9238))\n\n## [0.0.19](https://github.com/QuivrHQ/quivr/compare/core-0.0.18...core-0.0.19) (2024-10-21)\n\n\n### Features\n\n* **quivr-core:** beginning ([#3388](https://github.com/QuivrHQ/quivr/issues/3388)) ([7acb52a](https://github.com/QuivrHQ/quivr/commit/7acb52a9637b74d53f3e4cc9dde4ae1ca3f481ad))\n\n## [0.0.18](https://github.com/QuivrHQ/quivr/compare/core-0.0.17...core-0.0.18) (2024-10-16)\n\n\n### Bug Fixes\n\n* **core:** megaparse config ([#3384](https://github.com/QuivrHQ/quivr/issues/3384)) ([ffe86ca](https://github.com/QuivrHQ/quivr/commit/ffe86ca7bac3d7800913937314170db6f91daf8e))\n\n## [0.0.17](https://github.com/QuivrHQ/quivr/compare/core-0.0.16...core-0.0.17) (2024-10-16)\n\n\n### Features\n\n* **assistant:** cdp ([#3305](https://github.com/QuivrHQ/quivr/issues/3305)) ([b767f19](https://github.com/QuivrHQ/quivr/commit/b767f19f28b5478cef077b5d1587bf5195f2a668))\n* **assistants:** mock api ([#3195](https://github.com/QuivrHQ/quivr/issues/3195)) ([282fa0e](https://github.com/QuivrHQ/quivr/commit/282fa0e3f83f7c6fc8c84ca95f8f4ced4ed34b78))\n* introducing configurable retrieval workflows ([#3227](https://github.com/QuivrHQ/quivr/issues/3227)) ([ef90e8e](https://github.com/QuivrHQ/quivr/commit/ef90e8e672ca23d104c7d5bde7496f0929adf5d2))\n\n\n### Bug Fixes\n\n* fixing pdf parsing ([#3349](https://github.com/QuivrHQ/quivr/issues/3349)) ([367242a](https://github.com/QuivrHQ/quivr/commit/367242a3d5ea2df1928cb2908ad9e1906d1bba6f))\n\n\n### Documentation\n\n* **core:** init ([#3365](https://github.com/QuivrHQ/quivr/issues/3365)) ([bb572a2](https://github.com/QuivrHQ/quivr/commit/bb572a2a8d060f147461506aadd38704eb029a9a))\n* **fix:** fixed warnings from griffe ([#3381](https://github.com/QuivrHQ/quivr/issues/3381)) ([1a38798](https://github.com/QuivrHQ/quivr/commit/1a3879839a2d9e0881e18cb66809564fb76724ef))\n\n## [0.0.16](https://github.com/QuivrHQ/quivr/compare/core-0.0.15...core-0.0.16) (2024-09-17)\n\n\n### Bug Fixes\n\n* **core:** enforce langchain <0.3 for pydantic v1 ([#3217](https://github.com/QuivrHQ/quivr/issues/3217)) ([4bb4800](https://github.com/QuivrHQ/quivr/commit/4bb4800a76942ee31a939d3cacc94f057682177a))\n\n## [0.0.15](https://github.com/QuivrHQ/quivr/compare/core-0.0.14...core-0.0.15) (2024-09-16)\n\n\n### Features\n\n* CRUD KMS (no syncs) ([#3162](https://github.com/QuivrHQ/quivr/issues/3162)) ([71edca5](https://github.com/QuivrHQ/quivr/commit/71edca572ffd2901ed582005ac4b2803d9d95e57))\n* save and load brain ([#3202](https://github.com/QuivrHQ/quivr/issues/3202)) ([eda619f](https://github.com/QuivrHQ/quivr/commit/eda619f4547921ab4c50458b2d44c6b5c10e40d1))\n\n\n### Bug Fixes\n\n* Update LLMEndpoint to include max_tokens parameter ([#3201](https://github.com/QuivrHQ/quivr/issues/3201)) ([13ed225](https://github.com/QuivrHQ/quivr/commit/13ed225b172407ee9826b9c01b2f7b124a8b5a10))\n\n## [0.0.14](https://github.com/QuivrHQ/quivr/compare/core-0.0.13...core-0.0.14) (2024-09-09)\n\n\n### Features\n\n* Add brain_id and brain_name to ChatLLMMetadata model ([#2968](https://github.com/QuivrHQ/quivr/issues/2968)) ([1112001](https://github.com/QuivrHQ/quivr/commit/111200184b66dc42d75996c6c286474e9c5f8462))\n* add chat with models ([#2933](https://github.com/QuivrHQ/quivr/issues/2933)) ([fccd197](https://github.com/QuivrHQ/quivr/commit/fccd197511d8594db257bfddf757bf0d28f7239d))\n* Add get_model method to ModelRepository ([#2949](https://github.com/QuivrHQ/quivr/issues/2949)) ([13e9fc4](https://github.com/QuivrHQ/quivr/commit/13e9fc490bc62264de93d2efddf2389126c147fa))\n* **anthropic:** add llm ([#3146](https://github.com/QuivrHQ/quivr/issues/3146)) ([8e29218](https://github.com/QuivrHQ/quivr/commit/8e2921886505cea0e72d2e1136a4b8ba862c3ce1))\n* **azure:** quivr compatible with it ([#3005](https://github.com/QuivrHQ/quivr/issues/3005)) ([b5f31a8](https://github.com/QuivrHQ/quivr/commit/b5f31a83d4a1c4432943bbbaa0766c46927ef125))\n* **frontend:** talk with models and handle code markdown ([#2980](https://github.com/QuivrHQ/quivr/issues/2980)) ([ef6037e](https://github.com/QuivrHQ/quivr/commit/ef6037e665f8d5e9c513d889773419a25f914d83))\n* quivr core 0.1 ([#2970](https://github.com/QuivrHQ/quivr/issues/2970)) ([380cf82](https://github.com/QuivrHQ/quivr/commit/380cf8270678453c3dc14ac8665be9b3b5a49dce))\n* using langgraph in our RAG pipeline ([#3130](https://github.com/QuivrHQ/quivr/issues/3130)) ([8cfdf53](https://github.com/QuivrHQ/quivr/commit/8cfdf53fe748b884cf44ade274503de3966b1378))\n\n\n### Bug Fixes\n\n* **chat:** order of chat history was reversed ([#3148](https://github.com/QuivrHQ/quivr/issues/3148)) ([7209500](https://github.com/QuivrHQ/quivr/commit/7209500d0bdaec544fce1e8e9082de3ead4464f4))\n\n## [0.0.13](https://github.com/QuivrHQ/quivr/compare/core-0.0.12...core-0.0.13) (2024-08-01)\n\n\n### Features\n\n* quivr core tox test + parsers ([#2929](https://github.com/QuivrHQ/quivr/issues/2929)) ([6855585](https://github.com/QuivrHQ/quivr/commit/685558560cc431054fb9d1330c0e27ce5fdf1806))\n\n\n### Bug Fixes\n\n* processor quivr version ([#2934](https://github.com/QuivrHQ/quivr/issues/2934)) ([2d64962](https://github.com/QuivrHQ/quivr/commit/2d64962ca407d8f2c9e0faedc457548c3ff9921d))\n* quivr core fix tests ([#2935](https://github.com/QuivrHQ/quivr/issues/2935)) ([d9c1f3a](https://github.com/QuivrHQ/quivr/commit/d9c1f3add48f354d92f3a21c03eca53add30a773))\n\n## [0.0.12](https://github.com/QuivrHQ/quivr/compare/core-0.0.11...core-0.0.12) (2024-07-23)\n\n\n### Features\n\n* **dead-code:** removed composite & api ([#2902](https://github.com/QuivrHQ/quivr/issues/2902)) ([a2721d3](https://github.com/QuivrHQ/quivr/commit/a2721d3926df873e10817f948f8f10894ec6c581))\n* **frontend:** add knowledge icon when integration ([#2888](https://github.com/QuivrHQ/quivr/issues/2888)) ([733d083](https://github.com/QuivrHQ/quivr/commit/733d083e330fc6e41c089bb9c9cf76289040cab9))\n\n## [0.0.11](https://github.com/QuivrHQ/quivr/compare/core-0.0.10...core-0.0.11) (2024-07-22)\n\n\n### Features\n\n* move parsers quivr core ([#2884](https://github.com/QuivrHQ/quivr/issues/2884)) ([d3c53e6](https://github.com/QuivrHQ/quivr/commit/d3c53e63539bade5cbd716edf7e9af68ba15ed08))\n\n## [0.0.10](https://github.com/QuivrHQ/quivr/compare/core-0.0.9...core-0.0.10) (2024-07-19)\n\n\n### Features\n\n* **frontend:** new notifications design ([#2870](https://github.com/QuivrHQ/quivr/issues/2870)) ([ed97004](https://github.com/QuivrHQ/quivr/commit/ed9700426959f3c1502a882263dfb447411d5381))\n* **integrations:** dropbox ([#2864](https://github.com/QuivrHQ/quivr/issues/2864)) ([4806dc5](https://github.com/QuivrHQ/quivr/commit/4806dc5809aec9f7f573cb5adddac0e2d0ba600b))\n* quivr core brain info + processors registry + ([#2877](https://github.com/QuivrHQ/quivr/issues/2877)) ([3001fa1](https://github.com/QuivrHQ/quivr/commit/3001fa1475cf119a8b41a176f735f5402f708738))\n\n\n### Bug Fixes\n\n* Refacto & update dropbox refresh ([#2875](https://github.com/QuivrHQ/quivr/issues/2875)) ([3b68855](https://github.com/QuivrHQ/quivr/commit/3b68855a83c72f3e31c117af0434330383a8a5d7))\n\n## [0.0.9](https://github.com/QuivrHQ/quivr/compare/core-0.0.8...core-0.0.9) (2024-07-15)\n\n\n### Features\n\n* quivr api use quivr core ([#2868](https://github.com/QuivrHQ/quivr/issues/2868)) ([9d3e9ed](https://github.com/QuivrHQ/quivr/commit/9d3e9edfd2ef24397458cc6556f6080673be96ae))\n\n\n### Bug Fixes\n\n* quiv core stream duplicate and quivr-core rag tests ([#2852](https://github.com/QuivrHQ/quivr/issues/2852)) ([35eb07f](https://github.com/QuivrHQ/quivr/commit/35eb07f7a2664f65e482a78fabf242e1ccb36f07))\n\n## [0.0.8](https://github.com/QuivrHQ/quivr/compare/core-0.0.7...core-0.0.8) (2024-07-11)\n\n\n### Features\n\n* Add Quivr chatbot example ([#2827](https://github.com/QuivrHQ/quivr/issues/2827)) ([5ff8d4e](https://github.com/QuivrHQ/quivr/commit/5ff8d4ee81cdc5a2cf375a6b7709beb44da2b911))\n* Update aiofiles dependency to loosen version control ([#2834](https://github.com/QuivrHQ/quivr/issues/2834)) ([5e75d15](https://github.com/QuivrHQ/quivr/commit/5e75d155976dd710c65f9431e942cdeec9bd6424))\n\n## [0.0.7](https://github.com/QuivrHQ/quivr/compare/core-0.0.6...core-0.0.7) (2024-07-10)\n\n\n### Bug Fixes\n\n* llm model name ([#2830](https://github.com/QuivrHQ/quivr/issues/2830)) ([71d6cd9](https://github.com/QuivrHQ/quivr/commit/71d6cd9b6b381226a172a09c07a0a084d7efbc22))\n\n## [0.0.6](https://github.com/QuivrHQ/quivr/compare/core-0.0.5...core-0.0.6) (2024-07-10)\n\n\n### Features\n\n* quivr-core ask streaming ([#2828](https://github.com/QuivrHQ/quivr/issues/2828)) ([0658d49](https://github.com/QuivrHQ/quivr/commit/0658d4947c10f512d2ec2bdcfb70f089ab003a5c))\n\n## [0.0.5](https://github.com/QuivrHQ/quivr/compare/core-0.0.4...core-0.0.5) (2024-07-10)\n\n\n### Features\n\n* Add GitHub Actions workflow for running tests on backend/core ([#2820](https://github.com/QuivrHQ/quivr/issues/2820)) ([82292f3](https://github.com/QuivrHQ/quivr/commit/82292f30acf982bbf28c1ef928440086fa342a04))\n* Add GitHub Actions workflow for running tests on backend/core ([#2822](https://github.com/QuivrHQ/quivr/issues/2822)) ([1566040](https://github.com/QuivrHQ/quivr/commit/15660409a37af8df3c58a3f396614817c9f4641b))\n* quivr core chat history ([#2824](https://github.com/QuivrHQ/quivr/issues/2824)) ([847e161](https://github.com/QuivrHQ/quivr/commit/847e161d804421e60eb246f35bf51b7ffd88f3a2))\n\n## [0.0.4](https://github.com/QuivrHQ/quivr/compare/core-0.0.3...core-0.0.4) (2024-07-09)\n\n\n### Features\n\n* quivr core minimal chat ([#2818](https://github.com/QuivrHQ/quivr/issues/2818)) ([481f24f](https://github.com/QuivrHQ/quivr/commit/481f24f5bed855d044c97eb881512fbf936772f8))\n\n## [0.0.3](https://github.com/QuivrHQ/quivr/compare/core-0.0.2...core-0.0.3) (2024-07-09)\n\n\n### Bug Fixes\n\n* **pyproject:** fixed to quivr github ([#2816](https://github.com/QuivrHQ/quivr/issues/2816)) ([5a4ac00](https://github.com/QuivrHQ/quivr/commit/5a4ac001d0ba26af0c48aea7d9807c66b5fdd48d))\n\n## [0.0.2](https://github.com/QuivrHQ/quivr/compare/core-v0.0.1...core-0.0.2) (2024-07-09)\n\n\n### Features\n\n* **backend:** quivr-monorepo and quivr-core package ([#2765](https://github.com/QuivrHQ/quivr/issues/2765)) ([2e75de4](https://github.com/QuivrHQ/quivr/commit/2e75de40390bcc09f25037f19693989841fec70d))\n* quivr core minimal chat ([#2803](https://github.com/QuivrHQ/quivr/issues/2803)) ([1dc6d88](https://github.com/QuivrHQ/quivr/commit/1dc6d88f9b8b1b0c1a5682f990bf8098cbd54d77))\n"
},
{
"path": "core/Dockerfile.test",
"content": "# Using a slim version for a smaller base image\nFROM python:3.11.6-slim-bullseye\n\n# Install GEOS library, Rust, and other dependencies, then clean up\nRUN apt-get clean && apt-get update && apt-get install -y \\\n curl \\\n gcc \\\n autoconf \\\n automake \\\n build-essential \\\n # Additional dependencies for document handling\n libmagic-dev \\\n tesseract-ocr \\\n poppler-utils \\\n libreoffice \\\n pandoc && \\\n rm -rf /var/lib/apt/lists/*\n\n# Set the working directory\nWORKDIR /code\n\n# Install Poetry\nRUN curl -sSL https://install.python-poetry.org | POETRY_HOME=/opt/poetry python && \\\n cd /usr/local/bin && \\\n ln -s /opt/poetry/bin/poetry && \\\n poetry config virtualenvs.create false\n\n# Add Poetry to PATH\nENV PATH=\"/root/.local/bin:$PATH\"\n\n# Copy the current directory contents into the container at /app\nCOPY ./pyproject.toml ./poetry.lock* /code/\n\nRUN python3 -m pip install nltk && python3 -c \"import nltk; nltk.download('punkt')\" \\\n && python3 -c \"import nltk; nltk.download('averaged_perceptron_tagger')\"\n\n# Install project dependencies\nRUN poetry install --with test\n\nENV PYTHONPATH=/code\n"
},
{
"path": "core/README.md",
"content": "# quivr-core package\n\nThe RAG of Quivr.com\n\n## License π\n\nThis project is licensed under the Apache 2.0 License\n\n## Installation\n\n```bash\npip install quivr-core\n```\n\n\n\n\n"
},
{
"path": "core/example_workflows/talk_to_file_rag_config_workflow.yaml",
"content": "{\n \"max_files\": 20,\n \"llm_config\": { \"temperature\": 0.3, \"max_context_tokens\": 20000 },\n \"max_history\": 10,\n \"reranker_config\":\n { \"model\": \"rerank-v3.5\", \"top_n\": 10, \"supplier\": \"cohere\" },\n \"workflow_config\":\n {\n \"name\": \"Standard RAG\",\n \"nodes\":\n [\n {\n \"name\": \"START\",\n \"edges\": [\"filter_history\"],\n \"description\": \"Starting workflow\",\n },\n {\n \"name\": \"filter_history\",\n \"edges\": [\"retrieve\"],\n \"description\": \"Filtering history\",\n },\n {\n \"name\": \"retrieve\",\n \"edges\": [\"retrieve_full_documents_context\"],\n \"description\": \"Retrieving relevant information\",\n },\n {\n \"name\": \"retrieve_full_documents_context\",\n \"edges\": [\"generate_zendesk_rag\"],\n \"description\": \"Retrieving full tickets context\",\n },\n {\n \"name\": \"generate_zendesk_rag\",\n \"edges\": [\"END\"],\n \"description\": \"Generating answer\",\n },\n ],\n },\n}\n"
},
{
"path": "core/pyproject.toml",
"content": "[project]\nname = \"quivr-core\"\nversion = \"0.0.33\"\ndescription = \"Quivr core RAG package\"\nauthors = [{ name = \"Stan Girard\", email = \"stan@quivr.app\" }]\ndependencies = [\n \"pydantic>=2.8.2\",\n \"langchain-core>=0.3,<0.4\",\n \"langchain>=0.3.9,<0.4\",\n \"langgraph>=0.2.38,<0.3\",\n \"httpx>=0.27.0\",\n \"rich>=13.7.1\",\n \"tiktoken>=0.7.0\",\n \"aiofiles>=23.1.0\",\n \"langchain-openai>=0.3.0\",\n \"langchain-cohere>=0.1.0\",\n \"langchain-community>=0.3,<0.4\",\n \"langchain-anthropic>=0.1.23\",\n \"types-pyyaml>=6.0.12.20240808\",\n \"transformers[sentencepiece]>=4.44.2\",\n \"faiss-cpu>=1.8.0.post1\",\n \"rapidfuzz>=3.10.1\",\n \"markupsafe>=2.1.5\",\n \"megaparse-sdk>=0.1.11\",\n \"langchain-mistralai>=0.2.3\",\n \"langchain-google-genai>=2.1.3\",\n \"fasttext-langdetect>=1.0.5\",\n \"langfuse>=2.57.0\",\n \"langchain-groq>=0.3.2\",\n]\nreadme = \"README.md\"\nrequires-python = \">= 3.11\"\n\n[build-system]\nrequires = [\"hatchling\"]\nbuild-backend = \"hatchling.build\"\n\n[tool.rye]\nmanaged = true\ndev-dependencies = [\n \"mypy>=1.11.1\",\n \"pre-commit>=3.8.0\",\n \"ipykernel>=6.29.5\",\n \"ruff>=0.6.1\",\n \"flake8>=7.1.1\",\n \"flake8-black>=0.3.6\",\n \"pytest-asyncio>=0.23.8\",\n \"pytest>=8.3.2\",\n \"pytest-xdist>=3.6.1\",\n \"pytest-benchmark>=4.0.0\",\n]\n\n[tool.hatch.metadata]\nallow-direct-references = true\n\n[tool.hatch.build.targets.wheel]\npackages = [\"quivr_core\"]\n\n[tool.pytest.ini_options]\naddopts = \"--tb=short -ra -v\"\nfilterwarnings = [\"ignore::DeprecationWarning\"]\nmarkers = [\n \"slow: marks tests as slow (deselect with '-m \\\"not slow\\\"')\",\n \"base: these tests require quivr-core with extra `base` to be installed\",\n \"tika: these tests require a tika server to be running\",\n \"unstructured: these tests require `unstructured` dependency\",\n]\n\n[[tool.mypy.overrides]]\nmodule = \"yaml\"\nignore_missing_imports = true\n"
},
{
"path": "core/quivr_core/__init__.py",
"content": "from importlib.metadata import entry_points\n\nfrom .brain import Brain\nfrom .processor.registry import register_processor, registry\n\n__all__ = [\"Brain\", \"registry\", \"register_processor\"]\n\n\ndef register_entries():\n if entry_points is not None:\n try:\n eps = entry_points()\n except TypeError:\n pass # importlib-metadata < 0.8\n else:\n if hasattr(eps, \"select\"): # Python 3.10+ / importlib_metadata >= 3.9.0\n processors = eps.select(group=\"quivr_core.processor\")\n else:\n processors = eps.get(\"quivr_core.processor\", [])\n registered_names = set()\n for spec in processors:\n err_msg = f\"Unable to load processor from {spec}\"\n name = spec.name\n if name in registered_names:\n continue\n registered_names.add(name)\n register_processor(\n name,\n spec.value.replace(\":\", \".\"),\n errtxt=err_msg,\n append=True,\n )\n\n\nregister_entries()\n"
},
{
"path": "core/quivr_core/base_config.py",
"content": "from pathlib import Path\n\nimport yaml\nfrom pydantic import BaseModel, ConfigDict\nfrom typing import Self\n\n\nclass QuivrBaseConfig(BaseModel):\n \"\"\"\n Base configuration class for Quivr.\n\n This class extends Pydantic's BaseModel and provides a foundation for\n configuration management in quivr-core.\n\n Attributes:\n model_config (ConfigDict): Configuration for the Pydantic model.\n It's set to forbid extra attributes, ensuring strict adherence\n to the defined schema.\n\n Class Methods:\n from_yaml: Create an instance of the class from a YAML file.\n \"\"\"\n\n model_config = ConfigDict(extra=\"forbid\")\n\n @classmethod\n def from_yaml(cls, file_path: str | Path) -> Self:\n \"\"\"\n Create an instance of the class from a YAML file.\n\n Args:\n file_path (str | Path): The path to the YAML file.\n\n Returns:\n QuivrBaseConfig: An instance of the class initialized with the data from the YAML file.\n \"\"\"\n # Load the YAML file\n with open(file_path, \"r\") as stream:\n config_data = yaml.safe_load(stream)\n\n # Instantiate the class using the YAML data\n return cls(**config_data)\n"
},
{
"path": "core/quivr_core/brain/__init__.py",
"content": "from .brain import Brain\n\n__all__ = [\"Brain\"]\n"
},
{
"path": "core/quivr_core/brain/brain.py",
"content": "import asyncio\nimport logging\nimport os\nfrom pathlib import Path\nfrom pprint import PrettyPrinter\nfrom typing import Any, AsyncGenerator, Callable, Dict, Self, Type, Union\nfrom uuid import UUID, uuid4\n\nfrom langchain_core.documents import Document\nfrom langchain_core.embeddings import Embeddings\nfrom langchain_core.messages import AIMessage, HumanMessage\nfrom langchain_core.vectorstores import VectorStore\nfrom langchain_openai import OpenAIEmbeddings\nfrom rich.console import Console\nfrom rich.panel import Panel\n\nfrom quivr_core.brain.info import BrainInfo, ChatHistoryInfo\nfrom quivr_core.brain.serialization import (\n BrainSerialized,\n EmbedderConfig,\n FAISSConfig,\n LocalStorageConfig,\n TransparentStorageConfig,\n)\nfrom quivr_core.files.file import load_qfile\nfrom quivr_core.llm import LLMEndpoint\nfrom quivr_core.processor.registry import get_processor_class\nfrom quivr_core.rag.entities.chat import ChatHistory\nfrom quivr_core.rag.entities.config import RetrievalConfig\nfrom quivr_core.rag.entities.models import (\n LangchainMetadata,\n ParsedRAGChunkResponse,\n ParsedRAGResponse,\n QuivrKnowledge,\n SearchResult,\n)\nfrom quivr_core.rag.quivr_rag import QuivrQARAG\nfrom quivr_core.rag.quivr_rag_langgraph import QuivrQARAGLangGraph\nfrom quivr_core.storage.local_storage import LocalStorage, TransparentStorage\nfrom quivr_core.storage.storage_base import StorageBase\n\nfrom .brain_defaults import build_default_vectordb, default_embedder, default_llm\n\nlogger = logging.getLogger(\"quivr_core\")\n\n\nasync def process_files(\n storage: StorageBase, skip_file_error: bool, **processor_kwargs: dict[str, Any]\n) -> list[Document]:\n \"\"\"\n Process files in storage.\n This function takes a StorageBase and return a list of langchain documents.\n Args:\n storage (StorageBase): The storage containing the files to process.\n skip_file_error (bool): Whether to skip files that cannot be processed.\n processor_kwargs (dict[str, Any]): Additional arguments for the processor.\n Returns:\n list[Document]: List of processed documents in the Langchain Document format.\n Raises:\n ValueError: If a file cannot be processed and skip_file_error is False.\n Exception: If no processor is found for a file of a specific type and skip_file_error is False.\n \"\"\"\n\n knowledge = []\n for file in await storage.get_files():\n try:\n if file.file_extension:\n processor_cls = get_processor_class(file.file_extension)\n logger.debug(f\"processing {file} using class {processor_cls.__name__}\")\n processor = processor_cls(**processor_kwargs)\n docs = await processor.process_file(file)\n knowledge.extend(docs)\n else:\n logger.error(f\"can't find processor for {file}\")\n if skip_file_error:\n continue\n else:\n raise ValueError(f\"can't parse {file}. can't find file extension\")\n except KeyError as e:\n if skip_file_error:\n continue\n else:\n raise Exception(f\"Can't parse {file}. No available processor\") from e\n\n return knowledge\n\n\nclass Brain:\n \"\"\"\n A class representing a Brain.\n This class allows for the creation of a Brain, which is a collection of knowledge one wants to retrieve information from.\n A Brain is set to:\n * Store files in the storage of your choice (local, S3, etc.)\n * Process the files in the storage to extract text and metadata in a wide range of format.\n * Store the processed files in the vector store of your choice (FAISS, PGVector, etc.) - default to FAISS.\n * Create an index of the processed files.\n * Use the *Quivr* workflow for the retrieval augmented generation.\n A Brain is able to:\n * Search for information in the vector store.\n * Answer questions about the knowledges in the Brain.\n * Stream the answer to the question.\n Attributes:\n name (str): The name of the brain.\n id (UUID): The unique identifier of the brain.\n storage (StorageBase): The storage used to store the files.\n llm (LLMEndpoint): The language model used to generate the answer.\n vector_db (VectorStore): The vector store used to store the processed files.\n embedder (Embeddings): The embeddings used to create the index of the processed files.\n \"\"\"\n\n def __init__(\n self,\n *,\n name: str,\n llm: LLMEndpoint,\n id: UUID | None = None,\n vector_db: VectorStore | None = None,\n embedder: Embeddings | None = None,\n storage: StorageBase | None = None,\n workspace_id: UUID | None = None,\n chat_id: UUID | None = None,\n ):\n self.id = id\n self.name = name\n self.storage = storage\n self.workspace_id = workspace_id\n self.chat_id = chat_id\n # Chat history\n self._chats = self._init_chats()\n self.default_chat = list(self._chats.values())[0]\n\n # RAG dependencies:\n self.llm = llm\n self.vector_db = vector_db\n self.embedder = embedder\n\n def __repr__(self) -> str:\n pp = PrettyPrinter(width=80, depth=None, compact=False, sort_dicts=False)\n return pp.pformat(self.info())\n\n def print_info(self):\n console = Console()\n tree = self.info().to_tree()\n panel = Panel(tree, title=\"Brain Info\", expand=False, border_style=\"bold\")\n console.print(panel)\n\n @classmethod\n def load(cls, folder_path: str | Path) -> Self:\n \"\"\"\n Load a brain from a folder path.\n Args:\n folder_path (str | Path): The path to the folder containing the brain.\n Returns:\n Brain: The brain loaded from the folder path.\n Example:\n ```python\n brain_loaded = Brain.load(\"path/to/brain\")\n brain_loaded.print_info()\n ```\n \"\"\"\n if isinstance(folder_path, str):\n folder_path = Path(folder_path)\n if not folder_path.exists():\n raise ValueError(f\"path {folder_path} doesn't exist\")\n\n # Load brainserialized\n with open(os.path.join(folder_path, \"config.json\"), \"r\") as f:\n bserialized = BrainSerialized.model_validate_json(f.read())\n\n storage: StorageBase | None = None\n # Loading storage\n if bserialized.storage_config.storage_type == \"transparent_storage\":\n storage = TransparentStorage.load(bserialized.storage_config)\n elif bserialized.storage_config.storage_type == \"local_storage\":\n storage = LocalStorage.load(bserialized.storage_config)\n else:\n raise ValueError(\"unknown storage\")\n\n # Load Embedder\n if bserialized.embedding_config.embedder_type == \"openai_embedding\":\n from langchain_openai import OpenAIEmbeddings\n\n embedder = OpenAIEmbeddings(**bserialized.embedding_config.config)\n else:\n raise ValueError(\"unknown embedder\")\n\n # Load vector db\n if bserialized.vectordb_config.vectordb_type == \"faiss\":\n from langchain_community.vectorstores import FAISS\n\n vector_db = FAISS.load_local(\n folder_path=bserialized.vectordb_config.vectordb_folder_path,\n embeddings=embedder,\n allow_dangerous_deserialization=True,\n )\n else:\n raise ValueError(\"Unsupported vectordb\")\n\n return cls(\n id=bserialized.id,\n name=bserialized.name,\n embedder=embedder,\n llm=LLMEndpoint.from_config(bserialized.llm_config),\n storage=storage,\n vector_db=vector_db,\n )\n\n async def save(self, folder_path: str | Path):\n \"\"\"\n Save the brain to a folder path.\n Args:\n folder_path (str | Path): The path to the folder where the brain will be saved.\n Returns:\n str: The path to the folder where the brain was saved.\n Example:\n ```python\n await brain.save(\"path/to/brain\")\n ```\n \"\"\"\n if isinstance(folder_path, str):\n folder_path = Path(folder_path)\n\n brain_path = os.path.join(folder_path, f\"brain_{self.id}\")\n os.makedirs(brain_path, exist_ok=True)\n\n from langchain_community.vectorstores import FAISS\n\n if isinstance(self.vector_db, FAISS):\n vectordb_path = os.path.join(brain_path, \"vector_store\")\n os.makedirs(vectordb_path, exist_ok=True)\n self.vector_db.save_local(folder_path=vectordb_path)\n vector_store = FAISSConfig(vectordb_folder_path=vectordb_path)\n else:\n raise Exception(\"can't serialize other vector stores for now\")\n\n if isinstance(self.embedder, OpenAIEmbeddings):\n embedder_config = EmbedderConfig(\n config=self.embedder.dict(exclude={\"openai_api_key\"})\n )\n else:\n raise Exception(\"can't serialize embedder other than openai for now\")\n\n storage_config: Union[LocalStorageConfig, TransparentStorageConfig]\n # TODO : each instance should know how to serialize/deserialize itself\n if isinstance(self.storage, LocalStorage):\n serialized_files = {\n f.id: f.serialize() for f in await self.storage.get_files()\n }\n storage_config = LocalStorageConfig(\n storage_path=self.storage.dir_path, files=serialized_files\n )\n elif isinstance(self.storage, TransparentStorage):\n serialized_files = {\n f.id: f.serialize() for f in await self.storage.get_files()\n }\n storage_config = TransparentStorageConfig(files=serialized_files)\n else:\n raise Exception(\"can't serialize storage. not supported for now\")\n\n bserialized = BrainSerialized(\n id=self.id,\n name=self.name,\n chat_history=self.chat_history.get_chat_history(),\n llm_config=self.llm.get_config(),\n vectordb_config=vector_store,\n embedding_config=embedder_config,\n storage_config=storage_config,\n )\n\n with open(os.path.join(brain_path, \"config.json\"), \"w\") as f:\n f.write(bserialized.model_dump_json())\n return brain_path\n\n def info(self) -> BrainInfo:\n # TODO: dim of embedding\n # \"embedder\": {},\n chats_info = ChatHistoryInfo(\n nb_chats=len(self._chats),\n current_default_chat=self.default_chat.id,\n current_chat_history_length=len(self.default_chat),\n )\n\n return BrainInfo(\n brain_id=self.id,\n brain_name=self.name,\n files_info=self.storage.info() if self.storage else None,\n chats_info=chats_info,\n llm_info=self.llm.info(),\n )\n\n @property\n def chat_history(self) -> ChatHistory:\n return self.default_chat\n\n def _init_chats(self) -> Dict[UUID, ChatHistory]:\n chat_id = uuid4()\n default_chat = ChatHistory(chat_id=chat_id, brain_id=self.id)\n return {chat_id: default_chat}\n\n @classmethod\n async def afrom_files(\n cls,\n *,\n name: str,\n file_paths: list[str | Path],\n vector_db: VectorStore | None = None,\n storage: StorageBase = TransparentStorage(),\n llm: LLMEndpoint | None = None,\n embedder: Embeddings | None = None,\n skip_file_error: bool = False,\n processor_kwargs: dict[str, Any] | None = None,\n ):\n \"\"\"\n Create a brain from a list of file paths.\n Args:\n name (str): The name of the brain.\n file_paths (list[str | Path]): The list of file paths to add to the brain.\n vector_db (VectorStore | None): The vector store used to store the processed files.\n storage (StorageBase): The storage used to store the files.\n llm (LLMEndpoint | None): The language model used to generate the answer.\n embedder (Embeddings | None): The embeddings used to create the index of the processed files.\n skip_file_error (bool): Whether to skip files that cannot be processed.\n processor_kwargs (dict[str, Any] | None): Additional arguments for the processor.\n Returns:\n Brain: The brain created from the file paths.\n Example:\n ```python\n brain = await Brain.afrom_files(name=\"My Brain\", file_paths=[\"file1.pdf\", \"file2.pdf\"])\n brain.print_info()\n ```\n \"\"\"\n if llm is None:\n llm = default_llm()\n\n if embedder is None:\n embedder = default_embedder()\n\n processor_kwargs = processor_kwargs or {}\n\n brain_id = uuid4()\n\n # TODO: run in parallel using tasks\n\n for path in file_paths:\n file = await load_qfile(brain_id, path)\n await storage.upload_file(file)\n\n logger.debug(f\"uploaded all files to {storage}\")\n\n # Parse files\n docs = await process_files(\n storage=storage,\n skip_file_error=skip_file_error,\n **processor_kwargs,\n )\n\n # Building brain's vectordb\n if vector_db is None:\n vector_db = await build_default_vectordb(docs, embedder)\n else:\n await vector_db.aadd_documents(docs)\n\n logger.debug(f\"added {len(docs)} chunks to vectordb\")\n\n return cls(\n id=brain_id,\n name=name,\n storage=storage,\n llm=llm,\n embedder=embedder,\n vector_db=vector_db,\n )\n\n @classmethod\n def from_files(\n cls,\n *,\n name: str,\n file_paths: list[str | Path],\n vector_db: VectorStore | None = None,\n storage: StorageBase = TransparentStorage(),\n llm: LLMEndpoint | None = None,\n embedder: Embeddings | None = None,\n skip_file_error: bool = False,\n processor_kwargs: dict[str, Any] | None = None,\n ) -> Self:\n loop = asyncio.get_event_loop()\n return loop.run_until_complete(\n cls.afrom_files(\n name=name,\n file_paths=file_paths,\n vector_db=vector_db,\n storage=storage,\n llm=llm,\n embedder=embedder,\n skip_file_error=skip_file_error,\n processor_kwargs=processor_kwargs,\n )\n )\n\n @classmethod\n async def afrom_langchain_documents(\n cls,\n *,\n name: str,\n langchain_documents: list[Document],\n vector_db: VectorStore | None = None,\n storage: StorageBase = TransparentStorage(),\n llm: LLMEndpoint | None = None,\n embedder: Embeddings | None = None,\n ) -> Self:\n \"\"\"\n Create a brain from a list of langchain documents.\n Args:\n name (str): The name of the brain.\n langchain_documents (list[Document]): The list of langchain documents to add to the brain.\n vector_db (VectorStore | None): The vector store used to store the processed files.\n storage (StorageBase): The storage used to store the files.\n llm (LLMEndpoint | None): The language model used to generate the answer.\n embedder (Embeddings | None): The embeddings used to create the index of the processed files.\n Returns:\n Brain: The brain created from the langchain documents.\n Example:\n ```python\n from langchain_core.documents import Document\n documents = [Document(page_content=\"Hello, world!\")]\n brain = await Brain.afrom_langchain_documents(name=\"My Brain\", langchain_documents=documents)\n brain.print_info()\n ```\n \"\"\"\n\n if llm is None:\n llm = default_llm()\n\n if embedder is None:\n embedder = default_embedder()\n\n brain_id = uuid4()\n\n # Building brain's vectordb\n if vector_db is None:\n vector_db = await build_default_vectordb(langchain_documents, embedder)\n else:\n await vector_db.aadd_documents(langchain_documents)\n\n return cls(\n id=brain_id,\n name=name,\n storage=storage,\n llm=llm,\n embedder=embedder,\n vector_db=vector_db,\n )\n\n async def asearch(\n self,\n query: str | Document,\n n_results: int = 5,\n filter: Callable | Dict[str, Any] | None = None,\n fetch_n_neighbors: int = 20,\n ) -> list[SearchResult]:\n \"\"\"\n Search for relevant documents in the brain based on a query.\n Args:\n query (str | Document): The query to search for.\n n_results (int): The number of results to return.\n filter (Callable | Dict[str, Any] | None): The filter to apply to the search.\n fetch_n_neighbors (int): The number of neighbors to fetch.\n Returns:\n list[SearchResult]: The list of retrieved chunks.\n Example:\n ```python\n brain = Brain.from_files(name=\"My Brain\", file_paths=[\"file1.pdf\", \"file2.pdf\"])\n results = await brain.asearch(\"Why everybody loves Quivr?\")\n for result in results:\n print(result.chunk.page_content)\n ```\n \"\"\"\n if not self.vector_db:\n raise ValueError(\"No vector db configured for this brain\")\n\n result = await self.vector_db.asimilarity_search_with_score(\n query, k=n_results, filter=filter, fetch_k=fetch_n_neighbors\n )\n\n return [SearchResult(chunk=d, distance=s) for d, s in result]\n\n def get_chat_history(self, chat_id: UUID):\n return self._chats[chat_id]\n\n # TODO(@aminediro)\n def add_file(self) -> None:\n # add it to storage\n # add it to vectorstore\n raise NotImplementedError\n\n async def ask_streaming(\n self,\n question: str,\n run_id: UUID,\n system_prompt: str | None = None,\n retrieval_config: RetrievalConfig | None = None,\n rag_pipeline: Type[Union[QuivrQARAG, QuivrQARAGLangGraph]] | None = None,\n list_files: list[QuivrKnowledge] | None = None,\n chat_history: ChatHistory | None = None,\n **input_kwargs,\n ) -> AsyncGenerator[ParsedRAGChunkResponse, ParsedRAGChunkResponse]:\n \"\"\"\n Ask a question to the brain and get a streamed generated answer.\n Args:\n question (str): The question to ask.\n retrieval_config (RetrievalConfig | None): The retrieval configuration (see RetrievalConfig docs).\n rag_pipeline (Type[Union[QuivrQARAG, QuivrQARAGLangGraph]] | None): The RAG pipeline to use.\n list_files (list[QuivrKnowledge] | None): The list of files to include in the RAG pipeline.\n chat_history (ChatHistory | None): The chat history to use.\n Returns:\n AsyncGenerator[ParsedRAGChunkResponse, ParsedRAGChunkResponse]: The streamed generated answer.\n Example:\n ```python\n brain = Brain.from_files(name=\"My Brain\", file_paths=[\"file1.pdf\", \"file2.pdf\"])\n async for chunk in brain.ask_streaming(\"What is the meaning of life?\"):\n print(chunk.answer)\n ```\n \"\"\"\n llm = self.llm\n\n # If you passed a different llm model we'll override the brain one\n if retrieval_config:\n if retrieval_config.llm_config != self.llm.get_config():\n llm = LLMEndpoint.from_config(config=retrieval_config.llm_config)\n else:\n retrieval_config = RetrievalConfig(llm_config=self.llm.get_config())\n\n rag_instance = QuivrQARAGLangGraph(\n retrieval_config=retrieval_config, llm=llm, vector_store=self.vector_db\n )\n\n chat_history = self.default_chat if chat_history is None else chat_history\n list_files = [] if list_files is None else list_files\n\n full_answer = \"\"\n\n metadata = LangchainMetadata(\n langfuse_trace_id=str(run_id),\n langfuse_user_id=str(self.workspace_id),\n langfuse_session_id=str(self.chat_id),\n )\n\n async for response in rag_instance.answer_astream(\n run_id=run_id,\n question=question,\n system_prompt=system_prompt or None,\n history=chat_history,\n list_files=list_files,\n metadata=metadata,\n **input_kwargs,\n ):\n # Format output to be correct servicedf;j\n if not response.last_chunk:\n yield response\n full_answer += response.answer\n\n # TODO : add sources, metdata etc ...\n chat_history.append(HumanMessage(content=question))\n chat_history.append(AIMessage(content=full_answer))\n yield response\n\n async def aask(\n self,\n run_id: UUID,\n question: str,\n system_prompt: str | None = None,\n retrieval_config: RetrievalConfig | None = None,\n rag_pipeline: Type[Union[QuivrQARAG, QuivrQARAGLangGraph]] | None = None,\n list_files: list[QuivrKnowledge] | None = None,\n chat_history: ChatHistory | None = None,\n **input_kwargs,\n ) -> ParsedRAGResponse:\n \"\"\"\n Synchronous version that asks a question to the brain and gets a generated answer.\n Args:\n question (str): The question to ask.\n retrieval_config (RetrievalConfig | None): The retrieval configuration (see RetrievalConfig docs).\n rag_pipeline (Type[Union[QuivrQARAG, QuivrQARAGLangGraph]] | None): The RAG pipeline to use.\n list_files (list[QuivrKnowledge] | None): The list of files to include in the RAG pipeline.\n chat_history (ChatHistory | None): The chat history to use.\n Returns:\n ParsedRAGResponse: The generated answer.\n \"\"\"\n # question_language = detect_language(question) -- Commented until we use it\n full_answer = \"\"\n metadata = None\n\n async for response in self.ask_streaming(\n run_id=run_id,\n question=question,\n system_prompt=system_prompt,\n retrieval_config=retrieval_config,\n rag_pipeline=rag_pipeline,\n list_files=list_files,\n chat_history=chat_history,\n **input_kwargs,\n ):\n full_answer += response.answer\n if response.metadata:\n metadata = response.metadata\n\n return ParsedRAGResponse(answer=full_answer, metadata=metadata)\n\n def ask(\n self,\n run_id: UUID,\n question: str,\n system_prompt: str | None = None,\n retrieval_config: RetrievalConfig | None = None,\n rag_pipeline: Type[Union[QuivrQARAG, QuivrQARAGLangGraph]] | None = None,\n list_files: list[QuivrKnowledge] | None = None,\n chat_history: ChatHistory | None = None,\n ) -> ParsedRAGResponse:\n \"\"\"\n Fully synchronous version that asks a question to the brain and gets a generated answer.\n Args:\n question (str): The question to ask.\n system_prompt (str | None): The system prompt to use.\n retrieval_config (RetrievalConfig | None): The retrieval configuration (see RetrievalConfig docs).\n rag_pipeline (Type[Union[QuivrQARAG, QuivrQARAGLangGraph]] | None): The RAG pipeline to use.\n list_files (list[QuivrKnowledge] | None): The list of files to include in the RAG pipeline.\n chat_history (ChatHistory | None): The chat history to use.\n Returns:\n ParsedRAGResponse: The generated answer.\n \"\"\"\n loop = asyncio.get_event_loop()\n return loop.run_until_complete(\n self.aask(\n run_id=run_id,\n question=question,\n system_prompt=system_prompt,\n retrieval_config=retrieval_config,\n rag_pipeline=rag_pipeline,\n list_files=list_files,\n chat_history=chat_history,\n )\n )\n"
},
{
"path": "core/quivr_core/brain/brain_defaults.py",
"content": "import logging\n\nfrom langchain_core.documents import Document\nfrom langchain_core.embeddings import Embeddings\nfrom langchain_core.vectorstores import VectorStore\n\nfrom quivr_core.rag.entities.config import DefaultModelSuppliers, LLMEndpointConfig\nfrom quivr_core.llm import LLMEndpoint\n\nlogger = logging.getLogger(\"quivr_core\")\n\n\nasync def build_default_vectordb(\n docs: list[Document], embedder: Embeddings\n) -> VectorStore:\n try:\n from langchain_community.vectorstores import FAISS\n\n logger.debug(\"Using Faiss-CPU as vector store.\")\n # TODO(@aminediro) : embedding call is usually not concurrent for all documents but waits\n if len(docs) > 0:\n vector_db = await FAISS.afrom_documents(documents=docs, embedding=embedder)\n return vector_db\n else:\n raise ValueError(\"can't initialize brain without documents\")\n\n except ImportError as e:\n raise ImportError(\n \"Please provide a valid vector store or install quivr-core['base'] package for using the default one.\"\n ) from e\n\n\ndef default_embedder() -> Embeddings:\n try:\n from langchain_openai import OpenAIEmbeddings\n\n logger.debug(\"Loaded OpenAIEmbeddings as default LLM for brain\")\n embedder = OpenAIEmbeddings()\n return embedder\n except ImportError as e:\n raise ImportError(\n \"Please provide a valid Embedder or install quivr-core['base'] package for using the defaultone.\"\n ) from e\n\n\ndef default_llm() -> LLMEndpoint:\n try:\n logger.debug(\"Loaded ChatOpenAI as default LLM for brain\")\n llm = LLMEndpoint.from_config(\n LLMEndpointConfig(supplier=DefaultModelSuppliers.OPENAI, model=\"gpt-4o\")\n )\n return llm\n\n except ImportError as e:\n raise ImportError(\n \"Please provide a valid BaseLLM or install quivr-core['base'] package\"\n ) from e\n"
},
{
"path": "core/quivr_core/brain/info.py",
"content": "from dataclasses import dataclass\nfrom uuid import UUID\n\nfrom rich.tree import Tree\n\n\n@dataclass\nclass ChatHistoryInfo:\n nb_chats: int\n current_default_chat: UUID\n current_chat_history_length: int\n\n def add_to_tree(self, chats_tree: Tree):\n chats_tree.add(f\"Number of Chats: [bold]{self.nb_chats}[/bold]\")\n chats_tree.add(\n f\"Current Default Chat: [bold magenta]{self.current_default_chat}[/bold magenta]\"\n )\n chats_tree.add(\n f\"Current Chat History Length: [bold]{self.current_chat_history_length}[/bold]\"\n )\n\n\n@dataclass\nclass LLMInfo:\n model: str\n llm_base_url: str\n temperature: float\n max_tokens: int\n supports_function_calling: int\n\n def add_to_tree(self, llm_tree: Tree):\n llm_tree.add(f\"Model: [italic]{self.model}[/italic]\")\n llm_tree.add(f\"Base URL: [underline]{self.llm_base_url}[/underline]\")\n llm_tree.add(f\"Temperature: [bold]{self.temperature}[/bold]\")\n llm_tree.add(f\"Max Tokens: [bold]{self.max_tokens}[/bold]\")\n func_call_color = \"green\" if self.supports_function_calling else \"red\"\n llm_tree.add(\n f\"Supports Function Calling: [bold {func_call_color}]{self.supports_function_calling}[/bold {func_call_color}]\"\n )\n\n\n@dataclass\nclass StorageInfo:\n storage_type: str\n n_files: int\n\n def add_to_tree(self, files_tree: Tree):\n files_tree.add(f\"Storage Type: [italic]{self.storage_type}[/italic]\")\n files_tree.add(f\"Number of Files: [bold]{self.n_files}[/bold]\")\n\n\n@dataclass\nclass BrainInfo:\n brain_id: UUID\n brain_name: str\n chats_info: ChatHistoryInfo\n llm_info: LLMInfo\n files_info: StorageInfo | None = None\n\n def to_tree(self):\n tree = Tree(\"π Brain Information\")\n tree.add(f\"π ID: [bold cyan]{self.brain_id}[/bold cyan]\")\n tree.add(f\"π§ Brain Name: [bold green]{self.brain_name}[/bold green]\")\n\n if self.files_info:\n files_tree = tree.add(\"π Files\")\n self.files_info.add_to_tree(files_tree)\n\n chats_tree = tree.add(\"π¬ Chats\")\n self.chats_info.add_to_tree(chats_tree)\n\n llm_tree = tree.add(\"π€ LLM\")\n self.llm_info.add_to_tree(llm_tree)\n return tree\n"
},
{
"path": "core/quivr_core/brain/serialization.py",
"content": "from pathlib import Path\nfrom typing import Any, Dict, Literal, Union\nfrom uuid import UUID\n\nfrom pydantic import BaseModel, Field, SecretStr\n\nfrom quivr_core.rag.entities.config import LLMEndpointConfig\nfrom quivr_core.rag.entities.models import ChatMessage\nfrom quivr_core.files.file import QuivrFileSerialized\n\n\nclass EmbedderConfig(BaseModel):\n embedder_type: Literal[\"openai_embedding\"] = \"openai_embedding\"\n # TODO: type this correctly\n config: Dict[str, Any]\n\n\nclass PGVectorConfig(BaseModel):\n vectordb_type: Literal[\"pgvector\"] = \"pgvector\"\n pg_url: str\n pg_user: str\n pg_psswd: SecretStr\n table_name: str\n vector_dim: int\n\n\nclass FAISSConfig(BaseModel):\n vectordb_type: Literal[\"faiss\"] = \"faiss\"\n vectordb_folder_path: str\n\n\nclass LocalStorageConfig(BaseModel):\n storage_type: Literal[\"local_storage\"] = \"local_storage\"\n storage_path: Path\n files: dict[UUID, QuivrFileSerialized]\n\n\nclass TransparentStorageConfig(BaseModel):\n storage_type: Literal[\"transparent_storage\"] = \"transparent_storage\"\n files: dict[UUID, QuivrFileSerialized]\n\n\nclass BrainSerialized(BaseModel):\n id: UUID\n name: str\n chat_history: list[ChatMessage]\n vectordb_config: Union[FAISSConfig, PGVectorConfig] = Field(\n ..., discriminator=\"vectordb_type\"\n )\n storage_config: Union[TransparentStorageConfig, LocalStorageConfig] = Field(\n ..., discriminator=\"storage_type\"\n )\n\n llm_config: LLMEndpointConfig\n embedding_config: EmbedderConfig\n"
},
{
"path": "core/quivr_core/config.py",
"content": "from enum import Enum\n\nimport yaml\nfrom pydantic import BaseModel\n\n\nclass ParserType(str, Enum):\n \"\"\"Parser type enumeration.\"\"\"\n\n UNSTRUCTURED = \"unstructured\"\n LLAMA_PARSER = \"llama_parser\"\n MEGAPARSE_VISION = \"megaparse_vision\"\n\n\nclass StrategyEnum(str, Enum):\n \"\"\"Method to use for the conversion\"\"\"\n\n FAST = \"fast\"\n AUTO = \"auto\"\n HI_RES = \"hi_res\"\n\n\nclass MegaparseBaseConfig(BaseModel):\n @classmethod\n def from_yaml(cls, file_path: str):\n # Load the YAML file\n with open(file_path, \"r\") as stream:\n config_data = yaml.safe_load(stream)\n\n # Instantiate the class using the YAML data\n return cls(**config_data)\n\n\nclass MegaparseConfig(MegaparseBaseConfig):\n method: ParserType = ParserType.UNSTRUCTURED\n strategy: StrategyEnum = StrategyEnum.FAST\n check_table: bool = False\n parsing_instruction: str | None = None\n model_name: str = \"gpt-4o\"\n"
},
{
"path": "core/quivr_core/files/__init__.py",
"content": "from .file import QuivrFile\n\n__all__ = [\"QuivrFile\"]\n"
},
{
"path": "core/quivr_core/files/file.py",
"content": "import hashlib\nimport mimetypes\nimport os\nimport warnings\nfrom contextlib import asynccontextmanager\nfrom enum import Enum\nfrom pathlib import Path\nfrom typing import Any, AsyncGenerator, AsyncIterable, Self\nfrom uuid import UUID, uuid4\n\nimport aiofiles\nfrom openai import BaseModel\n\n\nclass QuivrFileSerialized(BaseModel):\n id: UUID\n brain_id: UUID\n path: Path\n original_filename: str\n file_size: int | None\n file_extension: str\n file_sha1: str\n additional_metadata: dict[str, Any]\n\n\nclass FileExtension(str, Enum):\n txt = \".txt\"\n pdf = \".pdf\"\n csv = \".csv\"\n doc = \".doc\"\n docx = \".docx\"\n pptx = \".pptx\"\n xls = \".xls\"\n xlsx = \".xlsx\"\n md = \".md\"\n mdx = \".mdx\"\n markdown = \".markdown\"\n bib = \".bib\"\n epub = \".epub\"\n html = \".html\"\n odt = \".odt\"\n py = \".py\"\n ipynb = \".ipynb\"\n m4a = \".m4a\"\n mp3 = \".mp3\"\n webm = \".webm\"\n mp4 = \".mp4\"\n mpga = \".mpga\"\n wav = \".wav\"\n mpeg = \".mpeg\"\n\n\ndef get_file_extension(file_path: Path) -> FileExtension | str:\n try:\n mime_type, _ = mimetypes.guess_type(file_path.name)\n if mime_type:\n mime_ext = mimetypes.guess_extension(mime_type)\n if mime_ext:\n return FileExtension(mime_ext)\n return FileExtension(file_path.suffix)\n except ValueError:\n warnings.warn(\n f\"File {file_path.name} extension isn't recognized. Make sure you have registered a parser for {file_path.suffix}\",\n stacklevel=2,\n )\n return file_path.suffix\n\n\nasync def load_qfile(brain_id: UUID, path: str | Path):\n if not isinstance(path, Path):\n path = Path(path)\n\n if not path.exists():\n raise FileExistsError(f\"file {path} doesn't exist\")\n\n file_size = os.stat(path).st_size\n\n async with aiofiles.open(path, mode=\"rb\") as f:\n file_sha1 = hashlib.sha1(await f.read()).hexdigest()\n\n try:\n # NOTE: when loading from existing storage, file name will be uuid\n id = UUID(path.name)\n except ValueError:\n id = uuid4()\n\n return QuivrFile(\n id=id,\n brain_id=brain_id,\n path=path,\n original_filename=path.name,\n file_extension=get_file_extension(path),\n file_size=file_size,\n file_sha1=file_sha1,\n )\n\n\nclass QuivrFile:\n __slots__ = [\n \"id\",\n \"brain_id\",\n \"path\",\n \"original_filename\",\n \"file_size\",\n \"file_extension\",\n \"file_sha1\",\n \"additional_metadata\",\n ]\n\n def __init__(\n self,\n id: UUID,\n original_filename: str,\n path: Path,\n file_sha1: str,\n file_extension: FileExtension | str,\n brain_id: UUID | None = None,\n file_size: int | None = None,\n metadata: dict[str, Any] | None = None,\n ) -> None:\n self.id = id\n self.brain_id = brain_id\n self.path = path\n self.original_filename = original_filename\n self.file_size = file_size\n self.file_extension = file_extension\n self.file_sha1 = file_sha1\n self.additional_metadata = metadata if metadata else {}\n\n def __repr__(self) -> str:\n return f\"QuivrFile-{self.id} original_filename:{self.original_filename}\"\n\n @asynccontextmanager\n async def open(self) -> AsyncGenerator[AsyncIterable[bytes], None]:\n # TODO(@aminediro) : match on path type\n f = await aiofiles.open(self.path, mode=\"rb\")\n try:\n yield f\n finally:\n await f.close()\n\n @property\n def metadata(self) -> dict[str, Any]:\n return {\n \"qfile_id\": self.id,\n \"qfile_path\": self.path,\n \"original_file_name\": self.original_filename,\n \"file_sha1\": self.file_sha1,\n \"file_size\": self.file_size,\n **self.additional_metadata,\n }\n\n def serialize(self) -> QuivrFileSerialized:\n return QuivrFileSerialized(\n id=self.id,\n brain_id=self.brain_id,\n path=self.path.absolute(),\n original_filename=self.original_filename,\n file_size=self.file_size,\n file_extension=self.file_extension,\n file_sha1=self.file_sha1,\n additional_metadata=self.additional_metadata,\n )\n\n @classmethod\n def deserialize(cls, serialized: QuivrFileSerialized) -> Self:\n return cls(\n id=serialized.id,\n brain_id=serialized.brain_id,\n path=serialized.path,\n original_filename=serialized.original_filename,\n file_size=serialized.file_size,\n file_extension=serialized.file_extension,\n file_sha1=serialized.file_sha1,\n metadata=serialized.additional_metadata,\n )\n"
},
{
"path": "core/quivr_core/language/models.py",
"content": "from enum import Enum\n\n\nclass Language(str, Enum):\n AF = \"af\" # Afrikaans\n ALS = \"als\" # Alemannic\n AM = \"am\" # Amharic\n AN = \"an\" # Aragonese\n AR = \"ar\" # Arabic\n ARZ = \"arz\" # Egyptian Arabic\n AS = \"as\" # Assamese\n AST = \"ast\" # Asturian\n AV = \"av\" # Avaric\n AZ = \"az\" # Azerbaijani\n AZB = \"azb\" # South Azerbaijani\n BA = \"ba\" # Bashkir\n BAR = \"bar\" # Bavarian\n BCL = \"bcl\" # Central Bikol\n BE = \"be\" # Belarusian\n BG = \"bg\" # Bulgarian\n BH = \"bh\" # Bihari\n BN = \"bn\" # Bengali\n BO = \"bo\" # Tibetan\n BPY = \"bpy\" # Bishnupriya Manipuri\n BR = \"br\" # Breton\n BS = \"bs\" # Bosnian\n BXR = \"bxr\" # Buryat\n CA = \"ca\" # Catalan\n CBK = \"cbk\" # Chavacano\n CE = \"ce\" # Chechen\n CEB = \"ceb\" # Cebuano\n CKB = \"ckb\" # Central Kurdish\n CO = \"co\" # Corsican\n CS = \"cs\" # Czech\n CV = \"cv\" # Chuvash\n CY = \"cy\" # Welsh\n DA = \"da\" # Danish\n DE = \"de\" # German\n DIQ = \"diq\" # Zazaki\n DSB = \"dsb\" # Lower Sorbian\n DTY = \"dty\" # Doteli\n DV = \"dv\" # Dhivehi\n EL = \"el\" # Greek\n EML = \"eml\" # Emilian-Romagnol\n EN = \"en\" # English\n EO = \"eo\" # Esperanto\n ES = \"es\" # Spanish\n ET = \"et\" # Estonian\n EU = \"eu\" # Basque\n FA = \"fa\" # Persian\n FI = \"fi\" # Finnish\n FR = \"fr\" # French\n FRR = \"frr\" # North Frisian\n FY = \"fy\" # Western Frisian\n GA = \"ga\" # Irish\n GD = \"gd\" # Scottish Gaelic\n GL = \"gl\" # Galician\n GN = \"gn\" # Guarani\n GOM = \"gom\" # Goan Konkani\n GU = \"gu\" # Gujarati\n GV = \"gv\" # Manx\n HE = \"he\" # Hebrew\n HI = \"hi\" # Hindi\n HIF = \"hif\" # Fiji Hindi\n HR = \"hr\" # Croatian\n HSB = \"hsb\" # Upper Sorbian\n HT = \"ht\" # Haitian Creole\n HU = \"hu\" # Hungarian\n HY = \"hy\" # Armenian\n IA = \"ia\" # Interlingua\n ID = \"id\" # Indonesian\n IE = \"ie\" # Interlingue\n ILO = \"ilo\" # Iloko\n IO = \"io\" # Ido\n IS = \"is\" # Icelandic\n IT = \"it\" # Italian\n JA = \"ja\" # Japanese\n JBO = \"jbo\" # Lojban\n JV = \"jv\" # Javanese\n KA = \"ka\" # Georgian\n KK = \"kk\" # Kazakh\n KM = \"km\" # Khmer\n KN = \"kn\" # Kannada\n KO = \"ko\" # Korean\n KRC = \"krc\" # Karachay-Balkar\n KU = \"ku\" # Kurdish\n KV = \"kv\" # Komi\n KW = \"kw\" # Cornish\n KY = \"ky\" # Kyrgyz\n LA = \"la\" # Latin\n LB = \"lb\" # Luxembourgish\n LEZ = \"lez\" # Lezghian\n LI = \"li\" # Limburgish\n LMO = \"lmo\" # Lombard\n LO = \"lo\" # Lao\n LRC = \"lrc\" # Northern Luri\n LT = \"lt\" # Lithuanian\n LV = \"lv\" # Latvian\n MAI = \"mai\" # Maithili\n MG = \"mg\" # Malagasy\n MHR = \"mhr\" # Eastern Mari\n MIN = \"min\" # Minangkabau\n MK = \"mk\" # Macedonian\n ML = \"ml\" # Malayalam\n MN = \"mn\" # Mongolian\n MR = \"mr\" # Marathi\n MRJ = \"mrj\" # Western Mari\n MS = \"ms\" # Malay\n MT = \"mt\" # Maltese\n MWL = \"mwl\" # Mirandese\n MY = \"my\" # Burmese\n MYV = \"myv\" # Erzya\n MZN = \"mzn\" # Mazanderani\n NAH = \"nah\" # Nahuatl\n NAP = \"nap\" # Neapolitan\n NDS = \"nds\" # Low German\n NE = \"ne\" # Nepali\n NEW = \"new\" # Newari\n NL = \"nl\" # Dutch\n NN = \"nn\" # Norwegian Nynorsk\n NO = \"no\" # Norwegian\n OC = \"oc\" # Occitan\n OR = \"or\" # Odia\n OS = \"os\" # Ossetian\n PA = \"pa\" # Punjabi\n PAM = \"pam\" # Pampanga\n PFL = \"pfl\" # Palatine German\n PL = \"pl\" # Polish\n PMS = \"pms\" # Piedmontese\n PNB = \"pnb\" # Western Punjabi\n PS = \"ps\" # Pashto\n PT = \"pt\" # Portuguese\n QU = \"qu\" # Quechua\n RM = \"rm\" # Romansh\n RO = \"ro\" # Romanian\n RU = \"ru\" # Russian\n RUE = \"rue\" # Rusyn\n SA = \"sa\" # Sanskrit\n SAH = \"sah\" # Yakut\n SC = \"sc\" # Sardinian\n SCN = \"scn\" # Sicilian\n SCO = \"sco\" # Scots\n SD = \"sd\" # Sindhi\n SH = \"sh\" # Serbo-Croatian\n SI = \"si\" # Sinhala\n SK = \"sk\" # Slovak\n SL = \"sl\" # Slovenian\n SO = \"so\" # Somali\n SQ = \"sq\" # Albanian\n SR = \"sr\" # Serbian\n SU = \"su\" # Sundanese\n SV = \"sv\" # Swedish\n SW = \"sw\" # Swahili\n TA = \"ta\" # Tamil\n TE = \"te\" # Telugu\n TG = \"tg\" # Tajik\n TH = \"th\" # Thai\n TK = \"tk\" # Turkmen\n TL = \"tl\" # Tagalog\n TR = \"tr\" # Turkish\n TT = \"tt\" # Tatar\n TYV = \"tyv\" # Tuvan\n UG = \"ug\" # Uyghur\n UK = \"uk\" # Ukrainian\n UR = \"ur\" # Urdu\n UZ = \"uz\" # Uzbek\n VEC = \"vec\" # Venetian\n VEP = \"vep\" # Veps\n VI = \"vi\" # Vietnamese\n VLS = \"vls\" # West Flemish\n VO = \"vo\" # VolapΓΌk\n WA = \"wa\" # Walloon\n WAR = \"war\" # Waray\n WUU = \"wuu\" # Wu Chinese\n XAL = \"xal\" # Kalmyk\n XMF = \"xmf\" # Mingrelian\n YI = \"yi\" # Yiddish\n YO = \"yo\" # Yoruba\n YUE = \"yue\" # Cantonese\n ZH = \"zh\" # Chinese\n UNKNOWN = \"unknown\" # Unknown\n"
},
{
"path": "core/quivr_core/language/utils.py",
"content": "from ftlangdetect import detect\nfrom quivr_core.language.models import Language\n\n\ndef detect_language(text: str, low_memory: bool = True) -> Language:\n detected_lang = detect(text=text, low_memory=low_memory)\n try:\n detected_language = Language(detected_lang[\"lang\"])\n except ValueError:\n return Language.UNKNOWN\n\n return detected_language\n"
},
{
"path": "core/quivr_core/llm/__init__.py",
"content": "from .llm_endpoint import LLMEndpoint\n\n__all__ = [\"LLMEndpoint\"]\n"
},
{
"path": "core/quivr_core/llm/llm_endpoint.py",
"content": "import logging\nimport os\nimport time\nfrom typing import Union\nfrom urllib.parse import parse_qs, urlparse\n\nimport tiktoken\nfrom langchain_anthropic import ChatAnthropic\nfrom langchain_core.language_models.chat_models import BaseChatModel\nfrom langchain_google_genai import ChatGoogleGenerativeAI\nfrom langchain_groq import ChatGroq\nfrom langchain_mistralai import ChatMistralAI\nfrom langchain_openai import AzureChatOpenAI, ChatOpenAI\nfrom pydantic import SecretStr\n\nfrom quivr_core.brain.info import LLMInfo\nfrom quivr_core.rag.entities.config import DefaultModelSuppliers, LLMEndpointConfig\nfrom quivr_core.rag.utils import model_supports_function_calling\n\nlogger = logging.getLogger(\"quivr_core\")\n\n\nclass LLMTokenizer:\n _cache: dict[\n int, tuple[\"LLMTokenizer\", int, float]\n ] = {} # {hash: (tokenizer, size_bytes, last_access_time)}\n _max_cache_size_mb: int = 50\n _max_cache_count: int = 5 # Default maximum number of cached tokenizers\n _current_cache_size: int = 0\n _default_size: int = 5 * 1024 * 1024\n\n def __init__(self, tokenizer_hub: str | None, fallback_tokenizer: str):\n self.tokenizer_hub = tokenizer_hub\n self.fallback_tokenizer = fallback_tokenizer\n\n if self.tokenizer_hub:\n # To prevent the warning\n # huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...\n os.environ[\"TOKENIZERS_PARALLELISM\"] = (\n \"false\"\n if not os.environ.get(\"TOKENIZERS_PARALLELISM\")\n else os.environ[\"TOKENIZERS_PARALLELISM\"]\n )\n try:\n if \"text-embedding-ada-002\" in self.tokenizer_hub:\n from transformers import GPT2TokenizerFast\n\n self.tokenizer = GPT2TokenizerFast.from_pretrained(\n self.tokenizer_hub\n )\n else:\n from transformers import AutoTokenizer\n\n self.tokenizer = AutoTokenizer.from_pretrained(self.tokenizer_hub)\n except OSError: # if we don't manage to connect to huggingface and/or no cached models are present\n logger.warning(\n f\"Cannot acces the configured tokenizer from {self.tokenizer_hub}, using the default tokenizer {self.fallback_tokenizer}\"\n )\n self.tokenizer = tiktoken.get_encoding(self.fallback_tokenizer)\n else:\n self.tokenizer = tiktoken.get_encoding(self.fallback_tokenizer)\n\n # More accurate size estimation\n self._size_bytes = self._calculate_tokenizer_size()\n\n def _calculate_tokenizer_size(self) -> int:\n \"\"\"Calculate size of tokenizer by summing the sizes of its vocabulary and model files\"\"\"\n # By default, return a size of 5 MB\n if not hasattr(self.tokenizer, \"vocab_files_names\") or not hasattr(\n self.tokenizer, \"init_kwargs\"\n ):\n return self._default_size\n\n total_size = 0\n\n # Get the file keys from vocab_files_names\n file_keys = self.tokenizer.vocab_files_names.keys()\n # Look up these files in init_kwargs\n for key in file_keys:\n if file_path := self.tokenizer.init_kwargs.get(key):\n try:\n total_size += os.path.getsize(file_path)\n except (OSError, FileNotFoundError):\n logger.debug(f\"Could not access tokenizer file: {file_path}\")\n\n return total_size if total_size > 0 else self._default_size\n\n @classmethod\n def load(cls, tokenizer_hub: str, fallback_tokenizer: str):\n cache_key = hash(str(tokenizer_hub))\n\n # If in cache, update last access time and return\n if cache_key in cls._cache:\n tokenizer, size, _ = cls._cache[cache_key]\n cls._cache[cache_key] = (tokenizer, size, time.time())\n return tokenizer\n\n # Create new instance\n instance = cls(tokenizer_hub, fallback_tokenizer)\n\n # Check if adding this would exceed either cache limit\n while (\n cls._current_cache_size + instance._size_bytes\n > cls._max_cache_size_mb * 1024 * 1024\n or len(cls._cache) >= cls._max_cache_count\n ):\n # Find least recently used item\n oldest_key = min(\n cls._cache.keys(),\n key=lambda k: cls._cache[k][2], # last_access_time\n )\n # Remove it\n _, removed_size, _ = cls._cache.pop(oldest_key)\n cls._current_cache_size -= removed_size\n\n # Add new instance to cache with current timestamp\n cls._cache[cache_key] = (instance, instance._size_bytes, time.time())\n cls._current_cache_size += instance._size_bytes\n return instance\n\n @classmethod\n def set_max_cache_size_mb(cls, size_mb: int):\n \"\"\"Set the maximum cache size in megabytes.\"\"\"\n cls._max_cache_size_mb = size_mb\n cls._cleanup_cache()\n\n @classmethod\n def set_max_cache_count(cls, count: int):\n \"\"\"Set the maximum number of tokenizers to cache.\"\"\"\n cls._max_cache_count = count\n cls._cleanup_cache()\n\n @classmethod\n def _cleanup_cache(cls):\n \"\"\"Clean up cache when limits are exceeded.\"\"\"\n while (\n cls._current_cache_size > cls._max_cache_size_mb * 1024 * 1024\n or len(cls._cache) > cls._max_cache_count\n ):\n oldest_key = min(cls._cache.keys(), key=lambda k: cls._cache[k][2])\n _, removed_size, _ = cls._cache.pop(oldest_key)\n cls._current_cache_size -= removed_size\n\n @classmethod\n def preload_tokenizers(cls, models: list[str] | None = None):\n \"\"\"Preload tokenizers into cache.\n\n Args:\n models: Optional list of model names (e.g. 'gpt-4o', 'claude-3-5-sonnet').\n If None, preloads all available tokenizers.\n \"\"\"\n from quivr_core.rag.entities.config import LLMModelConfig\n\n unique_tokenizer_hubs = set()\n\n # Collect tokenizer hubs based on provided models or all available\n if models:\n for model_name in models:\n # Find matching model configurations\n for supplier_models in LLMModelConfig._model_defaults.values():\n for base_model_name, config in supplier_models.items():\n # Check if the model name matches or starts with the base model name\n if (\n model_name.startswith(base_model_name)\n and config.tokenizer_hub\n ):\n unique_tokenizer_hubs.add(config.tokenizer_hub)\n break\n else:\n # Original behavior - collect all unique tokenizer hubs\n for supplier_models in LLMModelConfig._model_defaults.values():\n for config in supplier_models.values():\n if config.tokenizer_hub:\n unique_tokenizer_hubs.add(config.tokenizer_hub)\n\n # Load each unique tokenizer\n for hub in unique_tokenizer_hubs:\n try:\n cls.load(hub, LLMEndpointConfig._FALLBACK_TOKENIZER)\n logger.info(\n f\"Successfully preloaded tokenizer: {hub}. \"\n f\"Total cache size: {cls._current_cache_size / (1024 * 1024):.2f} MB. \"\n f\"Cache count: {len(cls._cache)}\"\n )\n except Exception as e:\n logger.warning(f\"Failed to preload tokenizer {hub}: {str(e)}\")\n\n\nclass LLMEndpoint:\n _cache = {}\n\n def __init__(self, llm_config: LLMEndpointConfig, llm: BaseChatModel):\n self._config = llm_config\n self._llm = llm\n self._supports_func_calling = model_supports_function_calling(\n self._config.model\n )\n\n self.llm_tokenizer = LLMTokenizer.load(\n llm_config.tokenizer_hub, llm_config.fallback_tokenizer\n )\n\n def count_tokens(self, text: str) -> int:\n # Tokenize the input text and return the token count\n encoding = self.llm_tokenizer.tokenizer.encode(text)\n return len(encoding)\n\n def get_config(self):\n return self._config\n\n @classmethod\n def from_config(cls, config: LLMEndpointConfig = LLMEndpointConfig()):\n hashed_config = hash(config)\n if hashed_config in cls._cache:\n return cls._cache[hashed_config]\n\n _llm: Union[\n AzureChatOpenAI,\n ChatOpenAI,\n ChatAnthropic,\n ChatMistralAI,\n ChatGoogleGenerativeAI,\n ChatGroq,\n ]\n try:\n if config.supplier == DefaultModelSuppliers.AZURE:\n # Parse the URL\n parsed_url = urlparse(config.llm_base_url)\n deployment = parsed_url.path.split(\"/\")[3] # type: ignore\n api_version = parse_qs(parsed_url.query).get(\"api-version\", [None])[0] # type: ignore\n azure_endpoint = f\"https://{parsed_url.netloc}\" # type: ignore\n _llm = AzureChatOpenAI(\n azure_deployment=deployment, # type: ignore\n api_version=api_version,\n api_key=SecretStr(config.llm_api_key)\n if config.llm_api_key\n else None,\n azure_endpoint=azure_endpoint,\n max_tokens=config.max_output_tokens,\n temperature=config.temperature,\n )\n elif config.supplier == DefaultModelSuppliers.ANTHROPIC:\n assert config.llm_api_key, \"Can't load model config\"\n _llm = ChatAnthropic(\n model_name=config.model,\n api_key=SecretStr(config.llm_api_key),\n base_url=config.llm_base_url,\n max_tokens_to_sample=config.max_output_tokens,\n temperature=config.temperature,\n timeout=None,\n stop=None,\n )\n elif config.supplier == DefaultModelSuppliers.OPENAI:\n _llm = ChatOpenAI(\n model=config.model,\n api_key=SecretStr(config.llm_api_key)\n if config.llm_api_key\n else None,\n base_url=config.llm_base_url,\n max_completion_tokens=config.max_output_tokens,\n temperature=config.temperature\n if not config.model.startswith(\"o\")\n else None,\n )\n elif config.supplier == DefaultModelSuppliers.MISTRAL:\n _llm = ChatMistralAI(\n model_name=config.model,\n api_key=SecretStr(config.llm_api_key)\n if config.llm_api_key\n else None,\n base_url=config.llm_base_url,\n temperature=config.temperature,\n )\n elif config.supplier == DefaultModelSuppliers.GEMINI:\n _llm = ChatGoogleGenerativeAI(\n model=config.model,\n api_key=SecretStr(config.llm_api_key)\n if config.llm_api_key\n else None,\n base_url=config.llm_base_url,\n max_tokens=config.max_output_tokens,\n temperature=config.temperature,\n )\n elif config.supplier == DefaultModelSuppliers.GROQ:\n _llm = ChatGroq(\n model=config.model,\n api_key=SecretStr(config.llm_api_key)\n if config.llm_api_key\n else None,\n base_url=config.llm_base_url,\n max_tokens=config.max_output_tokens,\n temperature=config.temperature,\n )\n\n else:\n _llm = ChatOpenAI(\n model=config.model,\n api_key=SecretStr(config.llm_api_key)\n if config.llm_api_key\n else None,\n base_url=config.llm_base_url,\n max_completion_tokens=config.max_output_tokens,\n temperature=config.temperature,\n )\n instance = cls(llm=_llm, llm_config=config)\n cls._cache[hashed_config] = instance\n\n return instance\n\n except ImportError as e:\n raise ImportError(\n \"Please provide a valid BaseLLM or install quivr-core['base'] package\"\n ) from e\n\n def supports_func_calling(self) -> bool:\n return self._supports_func_calling\n\n def info(self) -> LLMInfo:\n return LLMInfo(\n model=self._config.model,\n llm_base_url=(\n self._config.llm_base_url if self._config.llm_base_url else \"openai\"\n ),\n temperature=self._config.temperature,\n max_tokens=self._config.max_output_tokens,\n supports_function_calling=self.supports_func_calling(),\n )\n\n def clone_llm(self):\n \"\"\"Create a new instance of the LLM with the same configuration.\"\"\"\n return self._llm.__class__(**self._llm.__dict__)\n"
},
{
"path": "core/quivr_core/llm_tools/__init__.py",
"content": ""
},
{
"path": "core/quivr_core/llm_tools/entity.py",
"content": "from quivr_core.base_config import QuivrBaseConfig\nfrom typing import Callable\nfrom langchain_core.tools import BaseTool\nfrom typing import Dict, Any\n\n\nclass ToolsCategory(QuivrBaseConfig):\n name: str\n description: str\n tools: list\n default_tool: str | None = None\n create_tool: Callable\n\n def __init__(self, **data):\n super().__init__(**data)\n self.name = self.name.lower()\n\n\nclass ToolWrapper:\n def __init__(self, tool: BaseTool, format_input: Callable, format_output: Callable):\n self.tool = tool\n self.format_input = format_input\n self.format_output = format_output\n\n\nclass ToolRegistry:\n def __init__(self):\n self._registry = {}\n\n def register_tool(self, tool_name: str, create_func: Callable):\n self._registry[tool_name] = create_func\n\n def create_tool(self, tool_name: str, config: Dict[str, Any]) -> ToolWrapper:\n if tool_name not in self._registry:\n raise ValueError(f\"Tool {tool_name} is not supported.\")\n return self._registry[tool_name](config)\n"
},
{
"path": "core/quivr_core/llm_tools/llm_tools.py",
"content": "from typing import Dict, Any, Type, Union\n\nfrom quivr_core.llm_tools.entity import ToolWrapper\n\nfrom quivr_core.llm_tools.web_search_tools import (\n WebSearchTools,\n)\n\nfrom quivr_core.llm_tools.other_tools import (\n OtherTools,\n)\n\nTOOLS_CATEGORIES = {\n WebSearchTools.name: WebSearchTools,\n OtherTools.name: OtherTools,\n}\n\n# Register all ToolsList enums\nTOOLS_LISTS = {\n **{tool.value: tool for tool in WebSearchTools.tools},\n **{tool.value: tool for tool in OtherTools.tools},\n}\n\n\nclass LLMToolFactory:\n @staticmethod\n def create_tool(tool_name: str, config: Dict[str, Any]) -> Union[ToolWrapper, Type]:\n for category, tools_class in TOOLS_CATEGORIES.items():\n if tool_name in tools_class.tools:\n return tools_class.create_tool(tool_name, config)\n elif tool_name.lower() == category and tools_class.default_tool:\n return tools_class.create_tool(tools_class.default_tool, config)\n raise ValueError(f\"Tool {tool_name} is not supported.\")\n"
},
{
"path": "core/quivr_core/llm_tools/other_tools.py",
"content": "from enum import Enum\nfrom typing import Dict, Any, Type, Union\nfrom langchain_core.tools import BaseTool\nfrom quivr_core.llm_tools.entity import ToolsCategory\nfrom quivr_core.rag.entities.models import cited_answer\n\n\nclass OtherToolsList(str, Enum):\n CITED_ANSWER = \"cited_answer\"\n\n\ndef create_other_tool(tool_name: str, config: Dict[str, Any]) -> Union[BaseTool, Type]:\n if tool_name == OtherToolsList.CITED_ANSWER:\n return cited_answer\n else:\n raise ValueError(f\"Tool {tool_name} is not supported.\")\n\n\nOtherTools = ToolsCategory(\n name=\"Other\",\n description=\"Other tools\",\n tools=[OtherToolsList.CITED_ANSWER],\n create_tool=create_other_tool,\n)\n"
},
{
"path": "core/quivr_core/llm_tools/web_search_tools.py",
"content": "from enum import Enum\nfrom typing import Dict, List, Any\nfrom langchain_community.tools import TavilySearchResults\nfrom langchain_community.utilities.tavily_search import TavilySearchAPIWrapper\nfrom quivr_core.llm_tools.entity import ToolsCategory\nimport os\nfrom pydantic import SecretStr # Ensure correct import\nfrom quivr_core.llm_tools.entity import ToolWrapper, ToolRegistry\nfrom langchain_core.documents import Document\n\n\nclass WebSearchToolsList(str, Enum):\n TAVILY = \"tavily\"\n\n\ndef create_tavily_tool(config: Dict[str, Any]) -> ToolWrapper:\n api_key = (\n config.pop(\"api_key\") if \"api_key\" in config else os.getenv(\"TAVILY_API_KEY\")\n )\n if not api_key:\n raise ValueError(\n \"Missing required config key 'api_key' or environment variable 'TAVILY_API_KEY'\"\n )\n\n tavily_api_wrapper = TavilySearchAPIWrapper(\n tavily_api_key=SecretStr(api_key),\n )\n tool = TavilySearchResults(\n api_wrapper=tavily_api_wrapper,\n max_results=config.pop(\"max_results\", 5),\n search_depth=config.pop(\"search_depth\", \"advanced\"),\n include_answer=config.pop(\"include_answer\", True),\n **config,\n )\n\n tool.name = WebSearchToolsList.TAVILY.value\n\n def format_input(task: str) -> Dict[str, Any]:\n return {\"query\": task}\n\n def format_output(response: Any) -> List[Document]:\n metadata = {\"integration\": \"\", \"integration_link\": \"\"}\n return [\n Document(\n page_content=d[\"content\"],\n metadata={\n **metadata,\n \"file_name\": d[\"url\"] if \"url\" in d else \"\",\n \"original_file_name\": d[\"url\"] if \"url\" in d else \"\",\n },\n )\n for d in response\n ]\n\n return ToolWrapper(tool, format_input, format_output)\n\n\n# Initialize the registry and register tools\nweb_search_tool_registry = ToolRegistry()\nweb_search_tool_registry.register_tool(WebSearchToolsList.TAVILY, create_tavily_tool)\n\n\ndef create_web_search_tool(tool_name: str, config: Dict[str, Any]) -> ToolWrapper:\n return web_search_tool_registry.create_tool(tool_name, config)\n\n\nWebSearchTools = ToolsCategory(\n name=\"Web Search\",\n description=\"Tools for web searching\",\n tools=[WebSearchToolsList.TAVILY],\n default_tool=WebSearchToolsList.TAVILY,\n create_tool=create_web_search_tool,\n)\n"
},
{
"path": "core/quivr_core/processor/__init__.py",
"content": ""
},
{
"path": "core/quivr_core/processor/implementations/__init__.py",
"content": ""
},
{
"path": "core/quivr_core/processor/implementations/default.py",
"content": "import logging\nfrom typing import Any, List, Type, TypeVar\n\nimport tiktoken\nfrom langchain_community.document_loaders import (\n BibtexLoader,\n CSVLoader,\n Docx2txtLoader,\n NotebookLoader,\n PythonLoader,\n UnstructuredEPubLoader,\n UnstructuredExcelLoader,\n UnstructuredHTMLLoader,\n UnstructuredMarkdownLoader,\n UnstructuredODTLoader,\n UnstructuredPDFLoader,\n UnstructuredPowerPointLoader,\n)\nfrom langchain_community.document_loaders.base import BaseLoader\nfrom langchain_community.document_loaders.text import TextLoader\nfrom langchain_text_splitters import RecursiveCharacterTextSplitter, TextSplitter\n\nfrom quivr_core.files.file import FileExtension, QuivrFile\nfrom quivr_core.processor.processor_base import ProcessedDocument, ProcessorBase\nfrom quivr_core.processor.splitter import SplitterConfig\n\nlogger = logging.getLogger(\"quivr_core\")\n\nP = TypeVar(\"P\", bound=BaseLoader)\n\n\nclass ProcessorInit(ProcessorBase):\n def __init__(self, *args, **loader_kwargs) -> None:\n pass\n\n\n# FIXME(@aminediro):\n# dynamically creates Processor classes. Maybe redo this for finer control over instanciation\n# processor classes are opaque as we don't know what params they would have -> not easy to have lsp completion\ndef _build_processor(\n cls_name: str, load_cls: Type[P], cls_extensions: List[FileExtension | str]\n) -> Type[ProcessorInit]:\n enc = tiktoken.get_encoding(\"cl100k_base\")\n\n class _Processor(ProcessorBase):\n supported_extensions = cls_extensions\n\n def __init__(\n self,\n splitter: TextSplitter | None = None,\n splitter_config: SplitterConfig = SplitterConfig(),\n **loader_kwargs: dict[str, Any],\n ) -> None:\n self.loader_cls = load_cls\n self.loader_kwargs = loader_kwargs\n\n self.splitter_config = splitter_config\n\n if splitter:\n self.text_splitter = splitter\n else:\n self.text_splitter = (\n RecursiveCharacterTextSplitter.from_tiktoken_encoder(\n chunk_size=splitter_config.chunk_size,\n chunk_overlap=splitter_config.chunk_overlap,\n )\n )\n\n @property\n def processor_metadata(self) -> dict[str, Any]:\n return {\n \"processor_cls\": self.loader_cls.__name__,\n \"splitter\": self.splitter_config.model_dump(),\n }\n\n async def process_file_inner(self, file: QuivrFile) -> ProcessedDocument[None]:\n if hasattr(self.loader_cls, \"__init__\"):\n # NOTE: mypy can't correctly type this as BaseLoader doesn't have a constructor method\n loader = self.loader_cls(file_path=str(file.path), **self.loader_kwargs) # type: ignore\n else:\n loader = self.loader_cls()\n\n documents = await loader.aload()\n docs = self.text_splitter.split_documents(documents)\n\n for doc in docs:\n # TODO: This metadata info should be typed\n doc.metadata = {\"chunk_size\": len(enc.encode(doc.page_content))}\n\n return ProcessedDocument(\n chunks=docs, processor_cls=cls_name, processor_response=None\n )\n\n return type(cls_name, (ProcessorInit,), dict(_Processor.__dict__))\n\n\nCSVProcessor = _build_processor(\"CSVProcessor\", CSVLoader, [FileExtension.csv])\nTikTokenTxtProcessor = _build_processor(\n \"TikTokenTxtProcessor\", TextLoader, [FileExtension.txt]\n)\nDOCXProcessor = _build_processor(\n \"DOCXProcessor\", Docx2txtLoader, [FileExtension.docx, FileExtension.doc]\n)\nXLSXProcessor = _build_processor(\n \"XLSXProcessor\", UnstructuredExcelLoader, [FileExtension.xlsx, FileExtension.xls]\n)\nPPTProcessor = _build_processor(\n \"PPTProcessor\", UnstructuredPowerPointLoader, [FileExtension.pptx]\n)\nMarkdownProcessor = _build_processor(\n \"MarkdownProcessor\",\n UnstructuredMarkdownLoader,\n [FileExtension.md, FileExtension.mdx, FileExtension.markdown],\n)\nEpubProcessor = _build_processor(\n \"EpubProcessor\", UnstructuredEPubLoader, [FileExtension.epub]\n)\nBibTexProcessor = _build_processor(\"BibTexProcessor\", BibtexLoader, [FileExtension.bib])\nODTProcessor = _build_processor(\n \"ODTProcessor\", UnstructuredODTLoader, [FileExtension.odt]\n)\nHTMLProcessor = _build_processor(\n \"HTMLProcessor\", UnstructuredHTMLLoader, [FileExtension.html]\n)\nPythonProcessor = _build_processor(\"PythonProcessor\", PythonLoader, [FileExtension.py])\nNotebookProcessor = _build_processor(\n \"NotebookProcessor\", NotebookLoader, [FileExtension.ipynb]\n)\nUnstructuredPDFProcessor = _build_processor(\n \"UnstructuredPDFProcessor\", UnstructuredPDFLoader, [FileExtension.pdf]\n)\n"
},
{
"path": "core/quivr_core/processor/implementations/megaparse_processor.py",
"content": "import logging\n\nimport tiktoken\nfrom langchain_core.documents import Document\nfrom langchain_text_splitters import RecursiveCharacterTextSplitter, TextSplitter\nfrom megaparse_sdk.client import MegaParseNATSClient\nfrom megaparse_sdk.config import ClientNATSConfig\nfrom megaparse_sdk.schema.document import Document as MPDocument\n\nfrom quivr_core.config import MegaparseConfig\nfrom quivr_core.files.file import QuivrFile\nfrom quivr_core.processor.processor_base import ProcessedDocument, ProcessorBase\nfrom quivr_core.processor.registry import FileExtension\nfrom quivr_core.processor.splitter import SplitterConfig\n\nlogger = logging.getLogger(\"quivr_core\")\n\n\nclass MegaparseProcessor(ProcessorBase[MPDocument]):\n \"\"\"\n Megaparse processor for PDF files.\n\n It can be used to parse PDF files and split them into chunks.\n\n It comes from the megaparse library.\n\n ## Installation\n ```bash\n pip install megaparse\n ```\n\n \"\"\"\n\n supported_extensions = [\n FileExtension.txt,\n FileExtension.pdf,\n FileExtension.docx,\n FileExtension.doc,\n FileExtension.pptx,\n FileExtension.xls,\n FileExtension.xlsx,\n FileExtension.csv,\n FileExtension.epub,\n FileExtension.bib,\n FileExtension.odt,\n FileExtension.html,\n FileExtension.markdown,\n FileExtension.md,\n FileExtension.mdx,\n ]\n\n def __init__(\n self,\n splitter: TextSplitter | None = None,\n splitter_config: SplitterConfig = SplitterConfig(),\n megaparse_config: MegaparseConfig = MegaparseConfig(),\n ) -> None:\n self.enc = tiktoken.get_encoding(\"cl100k_base\")\n self.splitter_config = splitter_config\n self.megaparse_config = megaparse_config\n\n if splitter:\n self.text_splitter = splitter\n else:\n self.text_splitter = RecursiveCharacterTextSplitter.from_tiktoken_encoder(\n chunk_size=splitter_config.chunk_size,\n chunk_overlap=splitter_config.chunk_overlap,\n )\n\n @property\n def processor_metadata(self):\n return {\n \"chunk_overlap\": self.splitter_config.chunk_overlap,\n }\n\n async def process_file_inner(\n self, file: QuivrFile\n ) -> ProcessedDocument[MPDocument | str]:\n logger.info(f\"Uploading file {file.path} to MegaParse\")\n async with MegaParseNATSClient(ClientNATSConfig()) as client:\n response = await client.parse_file(file=file.path)\n\n document = Document(\n page_content=str(response),\n )\n\n chunks = self.text_splitter.split_documents([document])\n for chunk in chunks:\n chunk.metadata = {\"chunk_size\": len(self.enc.encode(chunk.page_content))}\n return ProcessedDocument(\n chunks=chunks,\n processor_cls=\"MegaparseProcessor\",\n processor_response=response,\n )\n"
},
{
"path": "core/quivr_core/processor/implementations/simple_txt_processor.py",
"content": "from typing import Any\n\nimport aiofiles\nfrom langchain_core.documents import Document\n\nfrom quivr_core.files.file import QuivrFile\nfrom quivr_core.processor.processor_base import ProcessedDocument, ProcessorBase\nfrom quivr_core.processor.registry import FileExtension\nfrom quivr_core.processor.splitter import SplitterConfig\n\n\ndef recursive_character_splitter(\n doc: Document, chunk_size: int, chunk_overlap: int\n) -> list[Document]:\n assert chunk_overlap < chunk_size, \"chunk_overlap is greater than chunk_size\"\n\n if len(doc.page_content) <= chunk_size:\n return [doc]\n\n chunk = Document(page_content=doc.page_content[:chunk_size], metadata=doc.metadata)\n remaining = Document(\n page_content=doc.page_content[chunk_size - chunk_overlap :],\n metadata=doc.metadata,\n )\n\n return [chunk] + recursive_character_splitter(remaining, chunk_size, chunk_overlap)\n\n\nclass SimpleTxtProcessor(ProcessorBase):\n \"\"\"\n SimpleTxtProcessor is a class that implements the ProcessorBase interface.\n It is used to process the files with the Simple Txt parser.\n \"\"\"\n\n supported_extensions = [FileExtension.txt]\n\n def __init__(\n self, splitter_config: SplitterConfig = SplitterConfig(), **kwargs\n ) -> None:\n super().__init__(**kwargs)\n self.splitter_config = splitter_config\n\n @property\n def processor_metadata(self) -> dict[str, Any]:\n return {\n \"processor_cls\": \"SimpleTxtProcessor\",\n \"splitter\": self.splitter_config.model_dump(),\n }\n\n async def process_file_inner(self, file: QuivrFile) -> ProcessedDocument[str]:\n async with aiofiles.open(file.path, mode=\"r\") as f:\n content = await f.read()\n\n doc = Document(page_content=content)\n\n docs = recursive_character_splitter(\n doc, self.splitter_config.chunk_size, self.splitter_config.chunk_overlap\n )\n\n return ProcessedDocument(\n chunks=docs, processor_cls=\"SimpleTxtProcessor\", processor_response=content\n )\n"
},
{
"path": "core/quivr_core/processor/implementations/tika_processor.py",
"content": "import logging\nimport os\nfrom typing import AsyncIterable\n\nimport httpx\nimport tiktoken\nfrom langchain_core.documents import Document\nfrom langchain_text_splitters import RecursiveCharacterTextSplitter, TextSplitter\n\nfrom quivr_core.files.file import QuivrFile\nfrom quivr_core.processor.processor_base import ProcessedDocument, ProcessorBase\nfrom quivr_core.processor.registry import FileExtension\nfrom quivr_core.processor.splitter import SplitterConfig\n\nlogger = logging.getLogger(\"quivr_core\")\n\n\nclass TikaProcessor(ProcessorBase):\n \"\"\"\n TikaProcessor is a class that implements the ProcessorBase interface.\n It is used to process the files with the Tika server.\n\n To run it with docker you can do:\n ```bash\n docker run -d -p 9998:9998 apache/tika\n ```\n \"\"\"\n\n supported_extensions = [FileExtension.pdf]\n\n def __init__(\n self,\n tika_url: str = os.getenv(\"TIKA_SERVER_URL\", \"http://localhost:9998/tika\"),\n splitter: TextSplitter | None = None,\n splitter_config: SplitterConfig = SplitterConfig(),\n timeout: float = 5.0,\n max_retries: int = 3,\n ) -> None:\n self.tika_url = tika_url\n self.max_retries = max_retries\n self._client = httpx.AsyncClient(timeout=timeout)\n\n self.enc = tiktoken.get_encoding(\"cl100k_base\")\n self.splitter_config = splitter_config\n\n if splitter:\n self.text_splitter = splitter\n else:\n self.text_splitter = RecursiveCharacterTextSplitter.from_tiktoken_encoder(\n chunk_size=splitter_config.chunk_size,\n chunk_overlap=splitter_config.chunk_overlap,\n )\n\n async def _send_parse_tika(self, f: AsyncIterable[bytes]) -> str:\n retry = 0\n headers = {\"Accept\": \"text/plain\"}\n while retry < self.max_retries:\n try:\n resp = await self._client.put(self.tika_url, headers=headers, content=f)\n resp.raise_for_status()\n return resp.content.decode(\"utf-8\")\n except Exception as e:\n retry += 1\n logger.debug(f\"tika url error :{e}. retrying for the {retry} time...\")\n raise RuntimeError(\"can't send parse request to tika server\")\n\n @property\n def processor_metadata(self):\n return {\n \"chunk_overlap\": self.splitter_config.chunk_overlap,\n }\n\n async def process_file_inner(self, file: QuivrFile) -> ProcessedDocument[None]:\n async with file.open() as f:\n txt = await self._send_parse_tika(f)\n document = Document(page_content=txt)\n docs = self.text_splitter.split_documents([document])\n for doc in docs:\n doc.metadata = {\"chunk_size\": len(self.enc.encode(doc.page_content))}\n\n return ProcessedDocument(\n chunks=docs, processor_cls=\"TikaProcessor\", processor_response=None\n )\n"
},
{
"path": "core/quivr_core/processor/processor_base.py",
"content": "import logging\nfrom abc import ABC, abstractmethod\nfrom importlib.metadata import PackageNotFoundError, version\nfrom typing import Any, Generic, List, TypeVar\n\nfrom attr import dataclass\nfrom langchain_core.documents import Document\n\nfrom quivr_core.files.file import FileExtension, QuivrFile\nfrom quivr_core.language.utils import detect_language\n\nlogger = logging.getLogger(\"quivr_core\")\n\n\nR = TypeVar(\"R\", covariant=True)\n\n\n@dataclass\nclass ProcessedDocument(Generic[R]):\n chunks: List[Document]\n processor_cls: str\n processor_response: R\n\n\n# TODO: processors should be cached somewhere ?\n# The processor should be cached by processor type\n# The cache should use a single\nclass ProcessorBase(ABC, Generic[R]):\n supported_extensions: list[FileExtension | str]\n\n def check_supported(self, file: QuivrFile) -> None:\n if file.file_extension not in self.supported_extensions:\n raise ValueError(f\"can't process a file of type {file.file_extension}\")\n\n @property\n @abstractmethod\n def processor_metadata(self) -> dict[str, Any]:\n raise NotImplementedError\n\n async def process_file(self, file: QuivrFile) -> ProcessedDocument[R]:\n logger.debug(f\"Processing file {file}\")\n self.check_supported(file)\n docs = await self.process_file_inner(file)\n try:\n qvr_version = version(\"quivr-core\")\n except PackageNotFoundError:\n qvr_version = \"dev\"\n\n for idx, doc in enumerate(docs.chunks, start=1):\n if \"original_file_name\" in doc.metadata:\n doc.page_content = f\"Filename: {doc.metadata['original_file_name']} Content: {doc.page_content}\"\n doc.page_content = doc.page_content.replace(\"\\u0000\", \"\")\n doc.page_content = doc.page_content.encode(\"utf-8\", \"replace\").decode(\n \"utf-8\"\n )\n doc.metadata = {\n \"chunk_index\": idx,\n \"quivr_core_version\": qvr_version,\n \"language\": detect_language(\n text=doc.page_content.replace(\"\\\\n\", \" \").replace(\"\\n\", \" \"),\n low_memory=True,\n ).value,\n **file.metadata,\n **doc.metadata,\n **self.processor_metadata,\n }\n return docs\n\n @abstractmethod\n async def process_file_inner(self, file: QuivrFile) -> ProcessedDocument[R]:\n raise NotImplementedError\n"
},
{
"path": "core/quivr_core/processor/registry.py",
"content": "import importlib\nimport logging\nimport types\nfrom dataclasses import dataclass, field\nfrom heapq import heappop, heappush\nfrom typing import List, Type, TypeAlias\n\nfrom quivr_core.files.file import FileExtension\n\nfrom .processor_base import ProcessorBase\n\nlogger = logging.getLogger(\"quivr_core\")\n\n_LOWEST_PRIORITY = 100\n\n_registry: dict[str, Type[ProcessorBase]] = {}\n\n# external, read only. Contains the actual processors that we are imported and ready to use\nregistry = types.MappingProxyType(_registry)\n\n\n@dataclass(order=True)\nclass ProcEntry:\n priority: int\n cls_mod: str = field(compare=False)\n err: str | None = field(compare=False)\n\n\nProcMapping: TypeAlias = dict[FileExtension | str, list[ProcEntry]]\n\n# Register based on mimetypes\nbase_processors: ProcMapping = {\n FileExtension.txt: [\n ProcEntry(\n cls_mod=\"quivr_core.processor.implementations.simple_txt_processor.SimpleTxtProcessor\",\n err=None,\n priority=_LOWEST_PRIORITY,\n )\n ],\n FileExtension.pdf: [\n ProcEntry(\n cls_mod=\"quivr_core.processor.implementations.tika_processor.TikaProcessor\",\n err=None,\n priority=_LOWEST_PRIORITY,\n )\n ],\n}\n\n\ndef _append_proc_mapping(\n mapping: ProcMapping,\n file_exts: List[FileExtension] | List[str],\n cls_mod: str,\n errtxt: str,\n priority: int | None,\n):\n for file_ext in file_exts:\n if file_ext in mapping:\n try:\n prev_proc = heappop(mapping[file_ext])\n proc_entry = ProcEntry(\n priority=priority\n if priority is not None\n else prev_proc.priority - 1,\n cls_mod=cls_mod,\n err=errtxt,\n )\n # Push the previous processor back\n heappush(mapping[file_ext], prev_proc)\n heappush(mapping[file_ext], proc_entry)\n except IndexError:\n proc_entry = ProcEntry(\n priority=priority if priority is not None else _LOWEST_PRIORITY,\n cls_mod=cls_mod,\n err=errtxt,\n )\n heappush(mapping[file_ext], proc_entry)\n\n else:\n proc_entry = ProcEntry(\n priority=priority if priority is not None else _LOWEST_PRIORITY,\n cls_mod=cls_mod,\n err=errtxt,\n )\n\n mapping[file_ext] = [proc_entry]\n\n\ndef defaults_to_proc_entries(\n base_processors: ProcMapping,\n) -> ProcMapping:\n # TODO(@aminediro) : how can a user change the order of the processor ?\n # NOTE: order of this list is important as resolution of `get_processor_class` depends on it\n # We should have a way to automatically add these at 'import' time\n for supported_extensions, processor_name in [\n ([FileExtension.csv], \"CSVProcessor\"),\n ([FileExtension.txt], \"TikTokenTxtProcessor\"),\n ([FileExtension.docx, FileExtension.doc], \"DOCXProcessor\"),\n ([FileExtension.xls, FileExtension.xlsx], \"XLSXProcessor\"),\n ([FileExtension.pptx], \"PPTProcessor\"),\n (\n [FileExtension.markdown, FileExtension.md, FileExtension.mdx],\n \"MarkdownProcessor\",\n ),\n ([FileExtension.epub], \"EpubProcessor\"),\n ([FileExtension.bib], \"BibTexProcessor\"),\n ([FileExtension.odt], \"ODTProcessor\"),\n ([FileExtension.html], \"HTMLProcessor\"),\n ([FileExtension.py], \"PythonProcessor\"),\n ([FileExtension.ipynb], \"NotebookProcessor\"),\n ]:\n for ext in supported_extensions:\n ext_str = ext.value if isinstance(ext, FileExtension) else ext\n _append_proc_mapping(\n mapping=base_processors,\n file_exts=[ext],\n cls_mod=f\"quivr_core.processor.implementations.default.{processor_name}\",\n errtxt=f\"can't import {processor_name}. Please install quivr-core[{ext_str}] to access {processor_name}\",\n priority=None,\n )\n\n # TODO(@aminediro): Megaparse should register itself\n # Append Megaparse\n _append_proc_mapping(\n mapping=base_processors,\n file_exts=[\n FileExtension.txt,\n FileExtension.pdf,\n FileExtension.docx,\n FileExtension.doc,\n FileExtension.pptx,\n FileExtension.xls,\n FileExtension.xlsx,\n FileExtension.csv,\n FileExtension.epub,\n FileExtension.bib,\n FileExtension.odt,\n FileExtension.html,\n FileExtension.markdown,\n FileExtension.md,\n FileExtension.mdx,\n ],\n cls_mod=\"quivr_core.processor.implementations.megaparse_processor.MegaparseProcessor\",\n errtxt=f\"can't import MegaparseProcessor. Please install quivr-core[{ext_str}] to access MegaparseProcessor\",\n priority=None,\n )\n return base_processors\n\n\nknown_processors = defaults_to_proc_entries(base_processors)\n\n\ndef get_processor_class(file_extension: FileExtension | str) -> Type[ProcessorBase]:\n \"\"\"Fetch processor class from registry\n\n The dict ``known_processors`` maps file extensions to the locations\n of processors that could process them.\n Loading of these classes is *Lazy*. Appropriate import will happen\n the first time we try to process some file type.\n\n Some processors need additional dependencies. If the import fails\n we return the \"err\" field of the ProcEntry in ``known_processors``.\n \"\"\"\n\n if file_extension not in registry:\n # Either you registered it from module or it's in the known processors\n if file_extension not in known_processors:\n raise ValueError(f\"Extension not known: {file_extension}\")\n entries = known_processors[file_extension]\n while entries:\n proc_entry = heappop(entries)\n try:\n register_processor(file_extension, _import_class(proc_entry.cls_mod))\n break\n except ImportError:\n logger.warn(\n f\"{proc_entry.err}. Falling to the next available processor for {file_extension}\"\n )\n if len(entries) == 0 and file_extension not in registry:\n raise ImportError(f\"can't find any processor for {file_extension}\")\n\n cls = registry[file_extension]\n return cls\n\n\ndef register_processor(\n file_ext: FileExtension | str,\n proc_cls: str | Type[ProcessorBase],\n append: bool = True,\n override: bool = False,\n errtxt: str | None = None,\n priority: int | None = None,\n):\n if isinstance(proc_cls, str):\n if file_ext in known_processors and append is False:\n if all(proc_cls != proc.cls_mod for proc in known_processors[file_ext]):\n raise ValueError(\n f\"Processor for ({file_ext}) already in the registry and append is False\"\n )\n else:\n if all(proc_cls != proc.cls_mod for proc in known_processors[file_ext]):\n _append_proc_mapping(\n known_processors,\n file_exts=[file_ext],\n cls_mod=proc_cls,\n errtxt=errtxt\n or f\"{proc_cls} import failed for processor of {file_ext}\",\n priority=priority,\n )\n else:\n logger.info(f\"{proc_cls} already in registry...\")\n\n else:\n assert issubclass(\n proc_cls, ProcessorBase\n ), f\"{proc_cls} should be a subclass of quivr_core.processor.ProcessorBase\"\n if file_ext in registry and override is False:\n if _registry[file_ext] is not proc_cls:\n raise ValueError(\n f\"Processor for ({file_ext}) already in the registry and append is False\"\n )\n else:\n _registry[file_ext] = proc_cls\n\n\ndef _import_class(full_mod_path: str):\n if \":\" in full_mod_path:\n mod_name, name = full_mod_path.rsplit(\":\", 1)\n else:\n mod_name, name = full_mod_path.rsplit(\".\", 1)\n\n mod = importlib.import_module(mod_name)\n\n for cls in name.split(\".\"):\n mod = getattr(mod, cls)\n\n if not isinstance(mod, type):\n raise TypeError(f\"{full_mod_path} is not a class\")\n\n if not issubclass(mod, ProcessorBase):\n raise TypeError(f\"{full_mod_path} is not a subclass of ProcessorBase \")\n\n return mod\n\n\ndef available_processors():\n \"\"\"Return a list of the known processors.\"\"\"\n return list(known_processors)\n"
},
{
"path": "core/quivr_core/processor/splitter.py",
"content": "from pydantic import BaseModel\n\n\nclass SplitterConfig(BaseModel):\n \"\"\"\n This class is used to configure the chunking of the documents.\n\n Chunk size is the number of characters in the chunk.\n Chunk overlap is the number of characters that the chunk will overlap with the previous chunk.\n \"\"\"\n\n chunk_size: int = 400\n chunk_overlap: int = 100\n"
},
{
"path": "core/quivr_core/rag/__init__.py",
"content": ""
},
{
"path": "core/quivr_core/rag/entities/__init__.py",
"content": ""
},
{
"path": "core/quivr_core/rag/entities/chat.py",
"content": "from datetime import datetime\nfrom typing import Any, Generator, Tuple, List\nfrom uuid import UUID, uuid4\n\nfrom langchain_core.messages import AIMessage, HumanMessage\n\nfrom quivr_core.rag.entities.models import ChatMessage\n\n\nclass ChatHistory:\n \"\"\"\n ChatHistory is a class that maintains a record of chat conversations. Each message\n in the history is represented by an instance of the `ChatMessage` class, and the\n chat history is stored internally as a list of these `ChatMessage` objects.\n The class provides methods to retrieve, append, iterate, and manipulate the chat\n history, as well as utilities to convert the messages into specific formats\n and support deep copying.\n \"\"\"\n\n def __init__(self, chat_id: UUID, brain_id: UUID | None) -> None:\n \"\"\"Init a new ChatHistory object.\n\n Args:\n chat_id (UUID): A unique identifier for the chat session.\n brain_id (UUID | None): An optional identifier for the brain associated with the chat.\n \"\"\"\n self.id = chat_id\n self.brain_id = brain_id\n # TODO(@aminediro): maybe use a deque() instead ?\n self._msgs: list[ChatMessage] = []\n\n def get_chat_history(self, newest_first: bool = False) -> List[ChatMessage]:\n \"\"\"\n Retrieves the chat history, optionally sorted in reverse chronological order.\n\n Args:\n newest_first (bool, optional): If True, returns the messages in reverse order (newest first). Defaults to False.\n\n Returns:\n List[ChatMessage]: A sorted list of chat messages.\n \"\"\"\n history = sorted(self._msgs, key=lambda msg: msg.message_time)\n if newest_first:\n return history[::-1]\n return history\n\n def __len__(self):\n return len(self._msgs)\n\n def append(\n self, langchain_msg: AIMessage | HumanMessage, metadata: dict[str, Any] = {}\n ):\n \"\"\"\n Appends a new message to the chat history.\n\n Args:\n langchain_msg (AIMessage | HumanMessage): The message content (either an AI or Human message).\n metadata (dict[str, Any], optional): Additional metadata related to the message. Defaults to an empty dictionary.\n \"\"\"\n chat_msg = ChatMessage(\n chat_id=self.id,\n message_id=uuid4(),\n brain_id=self.brain_id,\n msg=langchain_msg,\n message_time=datetime.now(),\n metadata=metadata,\n )\n self._msgs.append(chat_msg)\n\n def iter_pairs(self) -> Generator[Tuple[HumanMessage, AIMessage], None, None]:\n \"\"\"\n Iterates over the chat history in pairs, returning a HumanMessage followed by an AIMessage.\n\n Yields:\n Tuple[HumanMessage, AIMessage]: Pairs of human and AI messages.\n\n Raises:\n AssertionError: If the messages in the pair are not in the expected order (i.e., a HumanMessage followed by an AIMessage).\n \"\"\"\n # Reverse the chat_history, newest first\n it = iter(self.get_chat_history(newest_first=True))\n for ai_message, human_message in zip(it, it, strict=False):\n assert isinstance(\n human_message.msg, HumanMessage\n ), f\"msg {human_message} is not HumanMessage\"\n assert isinstance(\n ai_message.msg, AIMessage\n ), f\"msg {human_message} is not AIMessage\"\n yield (human_message.msg, ai_message.msg)\n\n def to_list(self) -> List[HumanMessage | AIMessage]:\n \"\"\"\n Converts the chat history into a list of raw HumanMessage or AIMessage objects.\n\n Returns:\n list[HumanMessage | AIMessage]: A list of messages in their raw form, without metadata.\n \"\"\"\n\n return [_msg.msg for _msg in self._msgs]\n"
},
{
"path": "core/quivr_core/rag/entities/config.py",
"content": "import logging\nimport os\nimport re\nfrom enum import Enum\nfrom typing import Any, Dict, Hashable, List, Optional, Type, Union\nfrom uuid import UUID\n\nfrom langchain_core.prompts.base import BasePromptTemplate\nfrom langchain_core.tools import BaseTool\nfrom langgraph.graph import END, START\nfrom pydantic import BaseModel\nfrom rapidfuzz import fuzz, process\n\nfrom quivr_core.base_config import QuivrBaseConfig\nfrom quivr_core.config import MegaparseConfig\nfrom quivr_core.llm_tools.llm_tools import TOOLS_CATEGORIES, TOOLS_LISTS, LLMToolFactory\nfrom quivr_core.processor.splitter import SplitterConfig\n\nlogger = logging.getLogger(\"quivr_core\")\nMIN_CONTEXT_TOKENS = 4096\nMIN_OUTPUT_TOKENS = 4096\n\n\ndef normalize_to_env_variable_name(name: str) -> str:\n # Replace any character that is not a letter, digit, or underscore with an underscore\n env_variable_name = re.sub(r\"[^A-Za-z0-9_]\", \"_\", name).upper()\n\n # Check if the normalized name starts with a digit\n if env_variable_name[0].isdigit():\n raise ValueError(\n f\"Invalid environment variable name '{env_variable_name}': Cannot start with a digit.\"\n )\n\n return env_variable_name\n\n\nclass SpecialEdges(str, Enum):\n start = \"START\"\n end = \"END\"\n\n\nclass BrainConfig(QuivrBaseConfig):\n brain_id: UUID | None = None\n name: str\n\n @property\n def id(self) -> UUID | None:\n return self.brain_id\n\n\nclass DefaultWebSearchTool(str, Enum):\n TAVILY = \"tavily\"\n\n\nclass DefaultRerankers(str, Enum):\n COHERE = \"cohere\"\n JINA = \"jina\"\n # MIXEDBREAD = \"mixedbread-ai\"\n\n @property\n def default_model(self) -> str:\n # Mapping of suppliers to their default models\n return {\n self.COHERE: \"rerank-v3.5\",\n self.JINA: \"jina-reranker-v2-base-multilingual\",\n # self.MIXEDBREAD: \"rmxbai-rerank-large-v1\",\n }[self]\n\n\nclass DefaultModelSuppliers(str, Enum):\n OPENAI = \"openai\"\n AZURE = \"azure\"\n ANTHROPIC = \"anthropic\"\n META = \"meta\"\n MISTRAL = \"mistral\"\n GROQ = \"groq\"\n GEMINI = \"gemini\"\n\n\nclass LLMConfig(QuivrBaseConfig):\n max_context_tokens: int | None = None\n max_output_tokens: int | None = None\n tokenizer_hub: str | None = None\n\n\nclass LLMModelConfig:\n _model_defaults: Dict[DefaultModelSuppliers, Dict[str, LLMConfig]] = {\n DefaultModelSuppliers.OPENAI: {\n \"gpt-4.1\": LLMConfig(\n max_context_tokens=1047576,\n max_output_tokens=32768,\n tokenizer_hub=\"Quivr/gpt-4o\",\n ),\n \"gpt-4o\": LLMConfig(\n max_context_tokens=128000,\n max_output_tokens=16384,\n tokenizer_hub=\"Quivr/gpt-4o\",\n ),\n \"gpt-4o-mini\": LLMConfig(\n max_context_tokens=128000,\n max_output_tokens=16384,\n tokenizer_hub=\"Quivr/gpt-4o\",\n ),\n \"o3-mini\": LLMConfig(\n max_context_tokens=200000,\n max_output_tokens=100000,\n tokenizer_hub=\"Quivr/gpt-4o\",\n ),\n \"o4-mini\": LLMConfig(\n max_context_tokens=200000,\n max_output_tokens=100000,\n tokenizer_hub=\"Quivr/gpt-4o\",\n ),\n \"o1-mini\": LLMConfig(\n max_context_tokens=128000,\n max_output_tokens=65536,\n tokenizer_hub=\"Quivr/gpt-4o\",\n ),\n \"o1-preview\": LLMConfig(\n max_context_tokens=128000,\n max_output_tokens=32768,\n tokenizer_hub=\"Quivr/gpt-4o\",\n ),\n \"o1\": LLMConfig(\n max_context_tokens=200000,\n max_output_tokens=100000,\n tokenizer_hub=\"Quivr/gpt-4o\",\n ),\n \"gpt-4-turbo\": LLMConfig(\n max_context_tokens=128000,\n max_output_tokens=4096,\n tokenizer_hub=\"Quivr/gpt-4\",\n ),\n \"gpt-4\": LLMConfig(\n max_context_tokens=8192,\n max_output_tokens=8192,\n tokenizer_hub=\"Quivr/gpt-4\",\n ),\n \"gpt-3.5-turbo\": LLMConfig(\n max_context_tokens=16385,\n max_output_tokens=4096,\n tokenizer_hub=\"Quivr/gpt-3.5-turbo\",\n ),\n \"text-embedding-3-large\": LLMConfig(\n max_context_tokens=8191, tokenizer_hub=\"Quivr/text-embedding-ada-002\"\n ),\n \"text-embedding-3-small\": LLMConfig(\n max_context_tokens=8191, tokenizer_hub=\"Quivr/text-embedding-ada-002\"\n ),\n \"text-embedding-ada-002\": LLMConfig(\n max_context_tokens=8191, tokenizer_hub=\"Quivr/text-embedding-ada-002\"\n ),\n },\n DefaultModelSuppliers.ANTHROPIC: {\n \"claude-opus-4\": LLMConfig(\n max_context_tokens=200000,\n max_output_tokens=8192,\n tokenizer_hub=\"Quivr/claude-tokenizer\",\n ),\n \"claude-sonnet-4\": LLMConfig(\n max_context_tokens=200000,\n max_output_tokens=8192,\n tokenizer_hub=\"Quivr/claude-tokenizer\",\n ),\n \"claude-3-7-sonnet\": LLMConfig(\n max_context_tokens=200000,\n max_output_tokens=8192,\n tokenizer_hub=\"Quivr/claude-tokenizer\",\n ),\n \"claude-3-5-sonnet\": LLMConfig(\n max_context_tokens=200000,\n max_output_tokens=8192,\n tokenizer_hub=\"Quivr/claude-tokenizer\",\n ),\n \"claude-3-opus\": LLMConfig(\n max_context_tokens=200000,\n max_output_tokens=4096,\n tokenizer_hub=\"Quivr/claude-tokenizer\",\n ),\n \"claude-3-sonnet\": LLMConfig(\n max_context_tokens=200000,\n max_output_tokens=4096,\n tokenizer_hub=\"Quivr/claude-tokenizer\",\n ),\n \"claude-3-haiku\": LLMConfig(\n max_context_tokens=200000,\n max_output_tokens=4096,\n tokenizer_hub=\"Quivr/claude-tokenizer\",\n ),\n \"claude-2-1\": LLMConfig(\n max_context_tokens=200000,\n max_output_tokens=4096,\n tokenizer_hub=\"Quivr/claude-tokenizer\",\n ),\n \"claude-2-0\": LLMConfig(\n max_context_tokens=100000,\n max_output_tokens=4096,\n tokenizer_hub=\"Quivr/claude-tokenizer\",\n ),\n \"claude-instant-1-2\": LLMConfig(\n max_context_tokens=100000,\n max_output_tokens=4096,\n tokenizer_hub=\"Quivr/claude-tokenizer\",\n ),\n },\n # Unclear for LLAMA models...\n # see https://huggingface.co/meta-llama/Llama-3.1-405B-Instruct/discussions/6\n DefaultModelSuppliers.META: {\n \"llama-3.1\": LLMConfig(\n max_context_tokens=128000,\n max_output_tokens=4096,\n tokenizer_hub=\"Quivr/Meta-Llama-3.1-Tokenizer\",\n ),\n \"llama-3\": LLMConfig(\n max_context_tokens=8192,\n max_output_tokens=2048,\n tokenizer_hub=\"Quivr/llama3-tokenizer-new\",\n ),\n \"code-llama\": LLMConfig(\n max_context_tokens=16384, tokenizer_hub=\"Quivr/llama-code-tokenizer\"\n ),\n },\n DefaultModelSuppliers.GROQ: {\n \"llama-3.3-70b\": LLMConfig(\n max_context_tokens=128000,\n max_output_tokens=32768,\n tokenizer_hub=\"Quivr/Meta-Llama-3.1-Tokenizer\",\n ),\n \"llama-3.1-70b\": LLMConfig(\n max_context_tokens=128000,\n max_output_tokens=32768,\n tokenizer_hub=\"Quivr/Meta-Llama-3.1-Tokenizer\",\n ),\n \"llama-3\": LLMConfig(\n max_context_tokens=8192, tokenizer_hub=\"Quivr/llama3-tokenizer-new\"\n ),\n \"code-llama\": LLMConfig(\n max_context_tokens=16384, tokenizer_hub=\"Quivr/llama-code-tokenizer\"\n ),\n \"deepseek-r1-distill-llama-70b\": LLMConfig(\n max_context_tokens=128000,\n max_output_tokens=32768,\n tokenizer_hub=\"Quivr/Meta-Llama-3.1-Tokenizer\",\n ),\n \"meta-llama/llama-4-maverick-17b-128e-instruct\": LLMConfig(\n max_context_tokens=128000,\n max_output_tokens=32768,\n tokenizer_hub=\"Quivr/Meta-Llama-3.1-Tokenizer\",\n ),\n },\n DefaultModelSuppliers.MISTRAL: {\n \"mistral-large\": LLMConfig(\n max_context_tokens=128000,\n max_output_tokens=4096,\n tokenizer_hub=\"Quivr/mistral-tokenizer-v3\",\n ),\n \"mistral-small\": LLMConfig(\n max_context_tokens=128000,\n max_output_tokens=4096,\n tokenizer_hub=\"Quivr/mistral-tokenizer-v3\",\n ),\n \"mistral-nemo\": LLMConfig(\n max_context_tokens=128000,\n max_output_tokens=4096,\n tokenizer_hub=\"Quivr/Mistral-Nemo-Instruct-Tokenizer\",\n ),\n \"codestral\": LLMConfig(\n max_context_tokens=32000, tokenizer_hub=\"Quivr/mistral-tokenizer-v3\"\n ),\n },\n DefaultModelSuppliers.GEMINI: {\n \"gemini-2.5\": LLMConfig(\n max_context_tokens=128000,\n max_output_tokens=4096,\n tokenizer_hub=\"Quivr/gemini-tokenizer\",\n ),\n },\n }\n\n @classmethod\n def get_supplier_by_model_name(cls, model: str) -> DefaultModelSuppliers | None:\n # Iterate over the suppliers and their models\n for supplier, models in cls._model_defaults.items():\n # Check if the model name or a base part of the model name is in the supplier's models\n for base_model_name in models:\n if model.startswith(base_model_name):\n return supplier\n # Return None if no supplier matches the model name\n return None\n\n @classmethod\n def get_llm_model_config(\n cls, supplier: DefaultModelSuppliers, model_name: str\n ) -> Optional[LLMConfig]:\n \"\"\"Retrieve the LLMConfig (context and tokenizer_hub) for a given supplier and model.\"\"\"\n supplier_defaults = cls._model_defaults.get(supplier)\n if not supplier_defaults:\n return None\n\n # Use startswith logic for matching model names\n for key, config in supplier_defaults.items():\n if model_name.startswith(key):\n return config\n\n return None\n\n\nclass LLMEndpointConfig(QuivrBaseConfig):\n supplier: DefaultModelSuppliers = DefaultModelSuppliers.OPENAI\n model: str = \"gpt-4o\"\n tokenizer_hub: str | None = None\n llm_base_url: str | None = None\n env_variable_name: str | None = None\n llm_api_key: str | None = None\n max_context_tokens: int = 20000\n max_output_tokens: int = 4096\n temperature: float = 0.3\n streaming: bool = True\n prompt: BasePromptTemplate | None = None\n\n _FALLBACK_TOKENIZER = \"cl100k_base\"\n\n def __hash__(self):\n return hash(\n (\n self.supplier,\n self.model,\n self.tokenizer_hub,\n self.llm_base_url,\n self.env_variable_name,\n self.llm_api_key,\n self.max_context_tokens,\n self.max_output_tokens,\n self.temperature,\n self.streaming,\n repr(self.prompt) if self.prompt is not None else None,\n )\n )\n\n @property\n def fallback_tokenizer(self) -> str:\n return self._FALLBACK_TOKENIZER\n\n def __init__(self, **data):\n super().__init__(**data)\n self.set_llm_model_config()\n self.set_api_key()\n\n def set_api_key(self, force_reset: bool = False):\n if not self.supplier:\n return\n\n # Check if the corresponding API key environment variable is set\n if force_reset or not self.env_variable_name:\n self.env_variable_name = (\n f\"{normalize_to_env_variable_name(self.supplier)}_API_KEY\"\n )\n\n if not self.llm_api_key or force_reset:\n self.llm_api_key = os.getenv(self.env_variable_name)\n\n if not self.llm_api_key:\n logger.warning(f\"The API key for supplier '{self.supplier}' is not set. \")\n logger.warning(\n f\"Please set the environment variable: '{self.env_variable_name}'. \"\n )\n\n def set_llm_model_config(self):\n # Automatically set context_length and tokenizer_hub based on the supplier and model\n llm_model_config = LLMModelConfig.get_llm_model_config(\n self.supplier, self.model\n )\n if llm_model_config:\n if llm_model_config.max_context_tokens:\n _max_context_tokens = (\n llm_model_config.max_context_tokens\n - llm_model_config.max_output_tokens\n if llm_model_config.max_output_tokens\n else llm_model_config.max_context_tokens\n )\n if self.max_context_tokens > _max_context_tokens:\n logger.warning(\n f\"Lowering max_context_tokens from {self.max_context_tokens} to {_max_context_tokens}\"\n )\n self.max_context_tokens = _max_context_tokens\n\n if self.max_context_tokens < MIN_CONTEXT_TOKENS:\n logger.error(\n f\"max_context_tokens is too low: {self.max_context_tokens}. \"\n )\n raise ValueError(\n f\"max_context_tokens is too low: {self.max_context_tokens}. \"\n )\n if llm_model_config.max_output_tokens:\n if self.max_output_tokens > llm_model_config.max_output_tokens:\n logger.warning(\n f\"Lowering max_output_tokens from {self.max_output_tokens} to {llm_model_config.max_output_tokens}\"\n )\n self.max_output_tokens = llm_model_config.max_output_tokens\n\n if self.max_output_tokens < MIN_OUTPUT_TOKENS:\n logger.error(\n f\"max_output_tokens is too low: {self.max_output_tokens}. \"\n )\n raise ValueError(\n f\"max_output_tokens is too low: {self.max_output_tokens}. \"\n )\n\n self.tokenizer_hub = llm_model_config.tokenizer_hub\n\n def set_llm_model(self, model: str):\n supplier = LLMModelConfig.get_supplier_by_model_name(model)\n if supplier is None:\n raise ValueError(\n f\"Cannot find the corresponding supplier for model {model}\"\n )\n self.supplier = supplier\n self.model = model\n\n self.set_llm_model_config()\n self.set_api_key(force_reset=True)\n\n def set_from_sqlmodel(self, sqlmodel: BaseModel, mapping: Dict[str, str]):\n \"\"\"\n Set attributes in LLMEndpointConfig from Model attributes using a field mapping.\n\n :param model_instance: An instance of the Model class.\n :param mapping: A dictionary that maps Model fields to LLMEndpointConfig fields.\n Example: {\"max_input\": \"max_input_tokens\", \"env_variable_name\": \"env_variable_name\"}\n \"\"\"\n for model_field, llm_field in mapping.items():\n if hasattr(sqlmodel, model_field) and hasattr(self, llm_field):\n setattr(self, llm_field, getattr(sqlmodel, model_field))\n else:\n raise AttributeError(\n f\"Invalid mapping: {model_field} or {llm_field} does not exist.\"\n )\n\n\n# Cannot use Pydantic v2 field_validator because of conflicts with pydantic v1 still in use in LangChain\nclass RerankerConfig(QuivrBaseConfig):\n supplier: DefaultRerankers | None = None\n model: str | None = None\n top_n: int = 5 # Number of chunks returned by the re-ranker\n api_key: str | None = None\n relevance_score_threshold: float | None = None\n relevance_score_key: str = \"relevance_score\"\n\n def __init__(self, **data):\n super().__init__(**data) # Call Pydantic's BaseModel init\n self.validate_model() # Automatically call external validation\n\n def validate_model(self):\n # If model is not provided, get default model based on supplier\n if self.model is None and self.supplier is not None:\n self.model = self.supplier.default_model\n\n # Check if the corresponding API key environment variable is set\n if self.supplier:\n api_key_var = f\"{normalize_to_env_variable_name(self.supplier)}_API_KEY\"\n self.api_key = os.getenv(api_key_var)\n\n if self.api_key is None:\n raise ValueError(\n f\"The API key for supplier '{self.supplier}' is not set. \"\n f\"Please set the environment variable: {api_key_var}\"\n )\n\n\nclass ConditionalEdgeConfig(QuivrBaseConfig):\n routing_function: str\n conditions: Union[list, Dict[Hashable, str]]\n\n def __init__(self, **data):\n super().__init__(**data)\n self.resolve_special_edges()\n\n def resolve_special_edges(self):\n \"\"\"Replace SpecialEdges enum values with their corresponding langgraph values.\"\"\"\n\n if isinstance(self.conditions, dict):\n # If conditions is a dictionary, iterate through the key-value pairs\n for key, value in self.conditions.items():\n if value == SpecialEdges.end:\n self.conditions[key] = END\n elif value == SpecialEdges.start:\n self.conditions[key] = START\n elif isinstance(self.conditions, list):\n # If conditions is a list, iterate through the values\n for index, value in enumerate(self.conditions):\n if value == SpecialEdges.end:\n self.conditions[index] = END\n elif value == SpecialEdges.start:\n self.conditions[index] = START\n\n\nclass NodeConfig(QuivrBaseConfig):\n name: str\n description: str | None = None\n edges: List[str] | None = None\n conditional_edge: ConditionalEdgeConfig | None = None\n tools: List[Dict[str, Any]] | None = None\n instantiated_tools: List[BaseTool | Type] | None = None\n\n def __init__(self, **data):\n super().__init__(**data)\n self._instantiate_tools()\n self.resolve_special_edges_in_name_and_edges()\n\n def resolve_special_edges_in_name_and_edges(self):\n \"\"\"Replace SpecialEdges enum values in name and edges with corresponding langgraph values.\"\"\"\n if self.name == SpecialEdges.start:\n self.name = START\n elif self.name == SpecialEdges.end:\n self.name = END\n\n if self.edges:\n for i, edge in enumerate(self.edges):\n if edge == SpecialEdges.start:\n self.edges[i] = START\n elif edge == SpecialEdges.end:\n self.edges[i] = END\n\n def _instantiate_tools(self):\n \"\"\"Instantiate tools based on the configuration.\"\"\"\n if self.tools:\n self.instantiated_tools = [\n LLMToolFactory.create_tool(tool_config.pop(\"name\"), tool_config)\n for tool_config in self.tools\n ]\n\n\nclass DefaultWorkflow(str, Enum):\n RAG = \"rag\"\n\n @property\n def nodes(self) -> List[NodeConfig]:\n # Mapping of workflow types to their default node configurations\n workflows = {\n self.RAG: [\n NodeConfig(name=START, edges=[\"filter_history\"]),\n NodeConfig(name=\"filter_history\", edges=[\"rewrite\"]),\n NodeConfig(name=\"rewrite\", edges=[\"retrieve\"]),\n NodeConfig(name=\"retrieve\", edges=[\"generate_rag\"]),\n NodeConfig(name=\"generate_rag\", edges=[END]),\n ]\n }\n return workflows[self]\n\n\nclass WorkflowConfig(QuivrBaseConfig):\n name: str | None = None\n nodes: List[NodeConfig] = []\n available_tools: List[str] | None = None\n validated_tools: List[BaseTool | Type] = []\n activated_tools: List[BaseTool | Type] = []\n\n def __init__(self, **data):\n super().__init__(**data)\n self.check_first_node_is_start()\n self.validate_available_tools()\n\n def check_first_node_is_start(self):\n if self.nodes and self.nodes[0].name != START:\n raise ValueError(f\"The first node should be a {SpecialEdges.start} node\")\n\n def get_node_tools(self, node_name: str) -> List[Any]:\n \"\"\"Get tools for a specific node.\"\"\"\n for node in self.nodes:\n if node.name == node_name and node.instantiated_tools:\n return node.instantiated_tools\n return []\n\n def validate_available_tools(self):\n if self.available_tools:\n valid_tools = list(TOOLS_CATEGORIES.keys()) + list(TOOLS_LISTS.keys())\n for tool in self.available_tools:\n if tool.lower() in valid_tools:\n self.validated_tools.append(\n LLMToolFactory.create_tool(tool, {}).tool\n )\n else:\n matches = process.extractOne(\n tool.lower(), valid_tools, scorer=fuzz.WRatio\n )\n if matches:\n raise ValueError(\n f\"Tool {tool} is not a valid ToolsCategory or ToolsList. Did you mean {matches[0]}?\"\n )\n else:\n raise ValueError(\n f\"Tool {tool} is not a valid ToolsCategory or ToolsList\"\n )\n\n\nclass RetrievalConfig(QuivrBaseConfig):\n reranker_config: RerankerConfig = RerankerConfig()\n llm_config: LLMEndpointConfig = LLMEndpointConfig()\n max_history: int = 10\n max_files: int = 20\n k: int = 40 # Number of chunks returned by the retriever\n prompt: str | None = None\n workflow_config: WorkflowConfig = WorkflowConfig(nodes=DefaultWorkflow.RAG.nodes)\n\n def __init__(self, **data):\n super().__init__(**data)\n self.llm_config.set_api_key(force_reset=True)\n\n\nclass ParserConfig(QuivrBaseConfig):\n splitter_config: SplitterConfig = SplitterConfig()\n megaparse_config: MegaparseConfig = MegaparseConfig()\n\n\nclass IngestionConfig(QuivrBaseConfig):\n parser_config: ParserConfig = ParserConfig()\n\n\nclass AssistantConfig(QuivrBaseConfig):\n retrieval_config: RetrievalConfig = RetrievalConfig()\n ingestion_config: IngestionConfig = IngestionConfig()\n"
},
{
"path": "core/quivr_core/rag/entities/models.py",
"content": "from datetime import datetime\nfrom enum import Enum\nfrom typing import Any, Dict, Optional\nfrom uuid import UUID\n\nfrom langchain_core.documents import Document\nfrom langchain_core.messages import AIMessage, HumanMessage\nfrom pydantic import BaseModel, Field\nfrom typing_extensions import TypedDict\n\n\nclass cited_answer(BaseModel):\n \"\"\"Answer the user question based only on the given sources, and cite the sources used.\"\"\"\n\n answer: str = Field(\n ...,\n description=\"The answer to the user question, which is based only on the given sources.\",\n )\n citations: list[int] = Field(\n ...,\n description=\"The integer IDs of the SPECIFIC sources which justify the answer.\",\n )\n\n followup_questions: list[str] = Field(\n ...,\n description=\"Generate up to 3 follow-up questions that could be asked based on the answer given or context provided.\",\n )\n\n\nclass ChatMessage(BaseModel):\n chat_id: UUID\n message_id: UUID\n brain_id: UUID | None\n msg: HumanMessage | AIMessage\n message_time: datetime\n metadata: dict[str, Any]\n\n\nclass KnowledgeStatus(str, Enum):\n ERROR = \"ERROR\"\n RESERVED = \"RESERVED\"\n PROCESSING = \"PROCESSING\"\n PROCESSED = \"PROCESSED\"\n UPLOADED = \"UPLOADED\"\n\n\nclass Source(BaseModel):\n name: str\n source_url: str\n type: str\n original_file_name: str\n citation: str\n\n\nclass RawRAGChunkResponse(TypedDict):\n answer: dict[str, Any]\n docs: dict[str, Any]\n\n\nclass RawRAGResponse(TypedDict):\n answer: dict[str, Any]\n docs: dict[str, Any]\n\n\nclass LangchainMetadata(BaseModel):\n langfuse_trace_id: str | None = None\n langfuse_trace_url: str | None = None\n langfuse_session_id: str | None = None\n langfuse_user_id: str | None = None\n\n\nclass ChatLLMMetadata(BaseModel):\n name: str\n display_name: str | None = None\n description: str | None = None\n image_url: str | None = None\n brain_id: str | None = None\n brain_name: str | None = None\n\n\nclass RAGResponseMetadata(BaseModel):\n citations: list[int] = Field(default_factory=list)\n followup_questions: list[str] = Field(default_factory=list)\n sources: list[Any] = Field(default_factory=list)\n metadata_model: ChatLLMMetadata | None = None\n workflow_step: str | None = None\n langchain_metadata: LangchainMetadata | None = None\n\n\nclass ParsedRAGResponse(BaseModel):\n answer: str\n metadata: RAGResponseMetadata | None = None\n\n\nclass ParsedRAGChunkResponse(BaseModel):\n answer: str\n metadata: RAGResponseMetadata\n last_chunk: bool = False\n\n\nclass QuivrKnowledge(BaseModel):\n id: UUID\n file_name: str\n brain_ids: list[UUID] | None = None\n url: Optional[str] = None\n extension: str = \".txt\"\n mime_type: str = \"txt\"\n status: KnowledgeStatus = KnowledgeStatus.PROCESSING\n source: Optional[str] = None\n source_link: str | None = None\n file_size: int | None = None # FIXME: Should not be optional @chloedia\n file_sha1: Optional[str] = None # FIXME: Should not be optional @chloedia\n updated_at: Optional[datetime] = None\n created_at: Optional[datetime] = None\n metadata: Optional[Dict[str, str]] = None\n\n\nclass SearchResult(BaseModel):\n chunk: Document\n distance: float\n"
},
{
"path": "core/quivr_core/rag/prompts.py",
"content": "import datetime\nimport types\nfrom enum import Enum\n\nfrom langchain_core.prompts import (\n ChatPromptTemplate,\n HumanMessagePromptTemplate,\n MessagesPlaceholder,\n PromptTemplate,\n SystemMessagePromptTemplate,\n)\nfrom langchain_core.prompts.base import BasePromptTemplate\n\n\nclass TemplatePromptName(str, Enum):\n ZENDESK_TEMPLATE_PROMPT = \"ZENDESK_TEMPLATE_PROMPT\"\n TOOL_ROUTING_PROMPT = \"TOOL_ROUTING_PROMPT\"\n RAG_ANSWER_PROMPT = \"RAG_ANSWER_PROMPT\"\n CONDENSE_TASK_PROMPT = \"CONDENSE_TASK_PROMPT\"\n DEFAULT_DOCUMENT_PROMPT = \"DEFAULT_DOCUMENT_PROMPT\"\n CHAT_LLM_PROMPT = \"CHAT_LLM_PROMPT\"\n USER_INTENT_PROMPT = \"USER_INTENT_PROMPT\"\n UPDATE_PROMPT = \"UPDATE_PROMPT\"\n SPLIT_PROMPT = \"SPLIT_PROMPT\"\n ZENDESK_LLM_PROMPT = \"ZENDESK_LLM_PROMPT\"\n\n\ndef _define_custom_prompts() -> dict[TemplatePromptName, BasePromptTemplate]:\n custom_prompts: dict[TemplatePromptName, BasePromptTemplate] = {}\n\n today_date = datetime.datetime.now().strftime(\"%B %d, %Y\")\n\n # ---------------------------------------------------------------------------\n # Prompt for task rephrasing\n # ---------------------------------------------------------------------------\n system_message_template = (\n \"Given a chat history and the latest user task \"\n \"which might reference context in the chat history, \"\n \"formulate a standalone task which can be understood \"\n \"without the chat history. Do NOT complete the task, \"\n \"just reformulate it if needed and otherwise return it as is. \"\n \"Do not output your reasoning, just the task.\"\n )\n\n template_answer = \"User task: {task}\\n Standalone task:\"\n\n CONDENSE_TASK_PROMPT = ChatPromptTemplate.from_messages(\n [\n SystemMessagePromptTemplate.from_template(system_message_template),\n MessagesPlaceholder(variable_name=\"chat_history\"),\n HumanMessagePromptTemplate.from_template(template_answer),\n ]\n )\n\n custom_prompts[TemplatePromptName.CONDENSE_TASK_PROMPT] = CONDENSE_TASK_PROMPT\n\n # ---------------------------------------------------------------------------\n # Prompt for RAG\n # ---------------------------------------------------------------------------\n system_message_template = f\"Your name is Quivr. You're a helpful assistant. Today's date is {today_date}. \"\n\n system_message_template += (\n \"- When answering use markdown. Use markdown code blocks for code snippets.\\n\"\n \"- Answer in a concise and clear manner.\\n\"\n \"- If no preferred language is provided, answer in the same language as the language used by the user.\\n\"\n \"- You must use ONLY the provided context to complete the task. \"\n \"Do not use any prior knowledge or external information, even if you are certain of the answer.\\n\"\n # \"- If you cannot provide an answer using ONLY the context provided, do not attempt to answer from your own knowledge.\"\n # \"Instead, inform the user that the answer isn't available in the context and suggest using the available tools {tools}.\\n\"\n \"- Do not apologize when providing an answer.\\n\"\n \"- Don't cite the source id in the answer objects, but you can use the source to complete the task.\\n\\n\"\n )\n\n context_template = (\n \"\\n\"\n # \"- You have access to the following internal reasoning to provide an answer: {reasoning}\\n\"\n \"- You have access to the following files to complete the task (limited to first 20 files): {files}\\n\"\n \"- You have access to the following context to complete the task: {context}\\n\"\n \"- Follow these user instruction when crafting the answer: {custom_instructions}\\n\"\n \"- These user instructions shall take priority over any other previous instruction.\\n\"\n # \"- Remember: if you cannot provide an answer using ONLY the provided context and CITING the sources, \"\n # \"inform the user that you don't have the answer and consider if any of the tools can help answer the question.\\n\"\n # \"- Explain your reasoning about the potentiel tool usage in the answer.\\n\"\n # \"- Only use binded tools to answer the question.\\n\"\n # \"OFFER the user the possibility to ACTIVATE a relevant tool among \"\n # \"the tools which can be activated.\"\n # \"Tools which can be activated: {tools}. If any of these tools can help in providing an answer \"\n # \"to the user question, you should offer the user the possibility to activate it. \"\n # \"Remember, you shall NOT use the above tools, ONLY offer the user the possibility to activate them.\\n\"\n )\n\n template_answer = (\n \"Original task: {task}\\n\"\n \"Rephrased and contextualized task: {rephrased_task}\\n\"\n \"Remember, you shall complete ALL tasks.\\n\"\n \"Remember: if you cannot provide an answer using ONLY the provided context and CITING the sources, \"\n \"just answer that you don't have the answer.\\n\"\n \"If the provided context contains contradictory or conflicting information, state so providing the conflicting information.\\n\"\n )\n\n RAG_ANSWER_PROMPT = ChatPromptTemplate.from_messages(\n [\n SystemMessagePromptTemplate.from_template(system_message_template),\n MessagesPlaceholder(variable_name=\"chat_history\"),\n SystemMessagePromptTemplate.from_template(context_template),\n HumanMessagePromptTemplate.from_template(template_answer),\n ]\n )\n custom_prompts[TemplatePromptName.RAG_ANSWER_PROMPT] = RAG_ANSWER_PROMPT\n\n # ---------------------------------------------------------------------------\n # Prompt for formatting documents\n # ---------------------------------------------------------------------------\n DEFAULT_DOCUMENT_PROMPT = PromptTemplate.from_template(\n template=\"Filename: {original_file_name}\\nSource: {index} \\n {page_content}\"\n )\n custom_prompts[TemplatePromptName.DEFAULT_DOCUMENT_PROMPT] = DEFAULT_DOCUMENT_PROMPT\n\n # ---------------------------------------------------------------------------\n # Prompt for chatting directly with LLMs, without any document retrieval stage\n # ---------------------------------------------------------------------------\n system_message_template = (\n f\"Your name is Quivr. You're a helpful assistant. Today's date is {today_date}.\"\n )\n system_message_template += \"\"\"\n If not None, also follow these user instructions when answering: {custom_instructions}\n \"\"\"\n\n template_answer = \"\"\"\n User Task: {task}\n Answer:\n \"\"\"\n CHAT_LLM_PROMPT = ChatPromptTemplate.from_messages(\n [\n SystemMessagePromptTemplate.from_template(system_message_template),\n MessagesPlaceholder(variable_name=\"chat_history\"),\n HumanMessagePromptTemplate.from_template(template_answer),\n ]\n )\n custom_prompts[TemplatePromptName.CHAT_LLM_PROMPT] = CHAT_LLM_PROMPT\n\n # ---------------------------------------------------------------------------\n # Prompt to understand the user intent\n # ---------------------------------------------------------------------------\n system_message_template = (\n \"Given the following user input, determine the user intent, in particular \"\n \"whether the user is providing instructions to the system or is asking the system to \"\n \"complete a task:\\n\"\n \" - if the user is providing direct instructions to modify the system behaviour (for instance, \"\n \"'Can you reply in French?' or 'Answer in French' or 'You are an expert legal assistant' \"\n \"or 'You will behave as...'), the user intent is 'prompt';\\n\"\n \" - in all other cases (asking questions, asking for summarising a text, asking for translating a text, ...), \"\n \"the intent is 'task'.\\n\"\n )\n\n template_answer = \"User input: {task}\"\n\n USER_INTENT_PROMPT = ChatPromptTemplate.from_messages(\n [\n SystemMessagePromptTemplate.from_template(system_message_template),\n HumanMessagePromptTemplate.from_template(template_answer),\n ]\n )\n custom_prompts[TemplatePromptName.USER_INTENT_PROMPT] = USER_INTENT_PROMPT\n\n # ---------------------------------------------------------------------------\n # Prompt to create a system prompt from user instructions\n # ---------------------------------------------------------------------------\n system_message_template = (\n \"- Given the following user instruction, current system prompt, list of available tools \"\n \"and list of activated tools, update the prompt to include the instruction and decide which tools to activate.\\n\"\n \"- The prompt shall only contain generic instructions which can be applied to any user task or question.\\n\"\n \"- The prompt shall be concise and clear.\\n\"\n \"- If the system prompt already contains the instruction, do not add it again.\\n\"\n \"- If the system prompt contradicts ther user instruction, remove the contradictory \"\n \"statement or statements in the system prompt.\\n\"\n \"- You shall return separately the updated system prompt and the reasoning that led to the update.\\n\"\n \"- If the system prompt refers to a tool, you shall add the tool to the list of activated tools.\\n\"\n \"- If no tool activation is needed, return empty lists.\\n\"\n \"- You shall also return the reasoning that led to the tool activation.\\n\"\n \"- Current system prompt: {system_prompt}\\n\"\n \"- List of available tools: {available_tools}\\n\"\n \"- List of activated tools: {activated_tools}\\n\\n\"\n )\n\n template_answer = \"User instructions: {instruction}\\n\"\n\n UPDATE_PROMPT = ChatPromptTemplate.from_messages(\n [\n SystemMessagePromptTemplate.from_template(system_message_template),\n HumanMessagePromptTemplate.from_template(template_answer),\n ]\n )\n custom_prompts[TemplatePromptName.UPDATE_PROMPT] = UPDATE_PROMPT\n\n # ---------------------------------------------------------------------------\n # Prompt to split the user input into multiple questions / instructions\n # ---------------------------------------------------------------------------\n system_message_template = (\n \"Given a chat history and the user input, split and rephrase the input into instructions and tasks.\\n\"\n \"- Instructions direct the system to behave in a certain way or to use specific tools: examples of instructions are \"\n \"'Can you reply in French?', 'Answer in French', 'You are an expert legal assistant', \"\n \"'You will behave as...', 'Use web search').\\n\"\n \"- You shall collect and condense all the instructions into a single string.\\n\"\n \"- The instructions shall be standalone and self-contained, so that they can be understood \"\n \"without the chat history. If no instructions are found, return an empty string.\\n\"\n \"- Instructions to be understood may require considering the chat history.\\n\"\n \"- Tasks are often questions, but they can also be summarisation tasks, translation tasks, content generation tasks, etc.\\n\"\n \"- Tasks to be understood may require considering the chat history.\\n\"\n \"- If the user input contains different tasks, you shall split the input into multiple tasks.\\n\"\n \"- Each splitted task shall be a standalone, self-contained task which can be understood \"\n \"without the chat history. You shall rephrase the tasks if needed.\\n\"\n \"- If no explicit task is present, you shall infer the tasks from the user input and the chat history.\\n\"\n \"- Do NOT try to solve the tasks or answer the questions, \"\n \"just reformulate them if needed and otherwise return them as is.\\n\"\n \"- Remember, you shall NOT suggest or generate new tasks.\\n\"\n \"- As an example, the user input 'What is Apple? Who is its CEO? When was it founded?' \"\n \"shall be split into a list of tasks ['What is Apple?', 'Who is the CEO of Apple?', 'When was Apple founded?']\\n\"\n \"- If no tasks are found, return the user input as is in the task list.\\n\"\n )\n\n template_answer = \"User input: {user_input}\"\n\n SPLIT_PROMPT = ChatPromptTemplate.from_messages(\n [\n SystemMessagePromptTemplate.from_template(system_message_template),\n MessagesPlaceholder(variable_name=\"chat_history\"),\n HumanMessagePromptTemplate.from_template(template_answer),\n ]\n )\n custom_prompts[TemplatePromptName.SPLIT_PROMPT] = SPLIT_PROMPT\n\n # ---------------------------------------------------------------------------\n # Prompt to grade the relevance of an answer and decide whather to perform a web search\n # ---------------------------------------------------------------------------\n system_message_template = (\n \"Given the following tasks you shall determine whether all tasks can be \"\n \"completed fully and in the best possible way using the provided context and chat history. \"\n \"You shall:\\n\"\n \"- Consider each task separately,\\n\"\n \"- Determine whether the context and chat history contain \"\n \"all the information necessary to complete the task.\\n\"\n \"- If the context and chat history do not contain all the information necessary to complete the task, \"\n \"consider ONLY the list of tools below and select the tool most appropriate to complete the task.\\n\"\n \"- If no tools are listed, return the tasks as is and no tool.\\n\"\n \"- If no relevant tool can be selected, return the tasks as is and no tool.\\n\"\n \"- Do not propose to use a tool if that tool is not listed among the available tools.\\n\"\n )\n\n context_template = \"Context: {context}\\n {activated_tools}\\n\"\n\n template_answer = \"Tasks: {tasks}\\n\"\n\n TOOL_ROUTING_PROMPT = ChatPromptTemplate.from_messages(\n [\n SystemMessagePromptTemplate.from_template(system_message_template),\n MessagesPlaceholder(variable_name=\"chat_history\"),\n SystemMessagePromptTemplate.from_template(context_template),\n HumanMessagePromptTemplate.from_template(template_answer),\n ]\n )\n\n custom_prompts[TemplatePromptName.TOOL_ROUTING_PROMPT] = TOOL_ROUTING_PROMPT\n\n system_message_zendesk_template = \"\"\"\n You are a Customer Service Agent using Zendesk. You are answering a client query.\n You will be provided with the users metadata, ticket metadata and ticket history which can be used to answer the query.\n You will also have access to the most relevant similar tickets and additional information sometimes such as API calls.\n Never add something in brackets that needs to be filled like [your name], [your email], etc. \n Do NOT invent information that was not present in previous tickets or in user metabadata or ticket metadata or additional information.\n Always prioritize information from the most recent tickets, especially if they are contradictory.\n \n Here is the current time: {current_time} UTC\n \n Here are default instructions that can be ignored if they are contradictory to the