Copy disabled (too large)
Download .txt
Showing preview only (10,459K chars total). Download the full file to get everything.
Repository: zilliztech/deep-searcher
Branch: master
Commit: d89e37cdfbbe
Files: 199
Total size: 9.9 MB
Directory structure:
gitextract_whyhti9y/
├── .github/
│ ├── ISSUE_TEMPLATE/
│ │ ├── bug_report.md
│ │ └── feature_request.md
│ ├── mergify.yml
│ └── workflows/
│ ├── cd-docs.yml
│ ├── ci-docs.yml
│ ├── docs.yml
│ ├── release.yml
│ └── ruff.yml
├── .gitignore
├── .python-version
├── .vscode/
│ └── settings.json
├── CONTRIBUTING.md
├── Dockerfile
├── LICENSE
├── MAINTAINERS
├── Makefile
├── OWNERS
├── OWNERS_ALIASES
├── README.md
├── deepsearcher/
│ ├── __init__.py
│ ├── agent/
│ │ ├── __init__.py
│ │ ├── base.py
│ │ ├── chain_of_rag.py
│ │ ├── collection_router.py
│ │ ├── deep_search.py
│ │ ├── naive_rag.py
│ │ └── rag_router.py
│ ├── cli.py
│ ├── config.yaml
│ ├── configuration.py
│ ├── embedding/
│ │ ├── __init__.py
│ │ ├── base.py
│ │ ├── bedrock_embedding.py
│ │ ├── fastembed_embdding.py
│ │ ├── gemini_embedding.py
│ │ ├── glm_embedding.py
│ │ ├── jiekouai_embedding.py
│ │ ├── milvus_embedding.py
│ │ ├── novita_embedding.py
│ │ ├── ollama_embedding.py
│ │ ├── openai_embedding.py
│ │ ├── ppio_embedding.py
│ │ ├── sentence_transformer_embedding.py
│ │ ├── siliconflow_embedding.py
│ │ ├── volcengine_embedding.py
│ │ ├── voyage_embedding.py
│ │ └── watsonx_embedding.py
│ ├── llm/
│ │ ├── __init__.py
│ │ ├── aliyun.py
│ │ ├── anthropic_llm.py
│ │ ├── azure_openai.py
│ │ ├── base.py
│ │ ├── bedrock.py
│ │ ├── deepseek.py
│ │ ├── gemini.py
│ │ ├── glm.py
│ │ ├── jiekouai.py
│ │ ├── novita.py
│ │ ├── ollama.py
│ │ ├── openai_llm.py
│ │ ├── ppio.py
│ │ ├── siliconflow.py
│ │ ├── together_ai.py
│ │ ├── volcengine.py
│ │ ├── watsonx.py
│ │ └── xai.py
│ ├── loader/
│ │ ├── __init__.py
│ │ ├── file_loader/
│ │ │ ├── __init__.py
│ │ │ ├── base.py
│ │ │ ├── docling_loader.py
│ │ │ ├── json_loader.py
│ │ │ ├── pdf_loader.py
│ │ │ ├── text_loader.py
│ │ │ └── unstructured_loader.py
│ │ ├── splitter.py
│ │ └── web_crawler/
│ │ ├── __init__.py
│ │ ├── base.py
│ │ ├── crawl4ai_crawler.py
│ │ ├── docling_crawler.py
│ │ ├── firecrawl_crawler.py
│ │ └── jina_crawler.py
│ ├── offline_loading.py
│ ├── online_query.py
│ ├── utils/
│ │ ├── __init__.py
│ │ └── log.py
│ └── vector_db/
│ ├── __init__.py
│ ├── azure_search.py
│ ├── base.py
│ ├── milvus.py
│ ├── oracle.py
│ └── qdrant.py
├── docs/
│ ├── README.md
│ ├── configuration/
│ │ ├── embedding.md
│ │ ├── file_loader.md
│ │ ├── index.md
│ │ ├── llm.md
│ │ ├── vector_db.md
│ │ └── web_crawler.md
│ ├── contributing/
│ │ └── index.md
│ ├── examples/
│ │ ├── basic_example.md
│ │ ├── docling.md
│ │ ├── firecrawl.md
│ │ ├── index.md
│ │ ├── oracle.md
│ │ └── unstructured.md
│ ├── faq/
│ │ └── index.md
│ ├── future_plans.md
│ ├── index.md
│ ├── installation/
│ │ ├── development.md
│ │ ├── index.md
│ │ └── pip.md
│ ├── integrations/
│ │ └── index.md
│ ├── overrides/
│ │ └── .gitkeep
│ ├── stylesheets/
│ │ └── extra.css
│ └── usage/
│ ├── cli.md
│ ├── deployment.md
│ ├── index.md
│ └── quick_start.md
├── env.example
├── evaluation/
│ ├── README.md
│ ├── eval_config.yaml
│ └── evaluate.py
├── examples/
│ ├── basic_example.py
│ ├── basic_example_azuresearch.py
│ ├── basic_example_oracle.py
│ ├── basic_watsonx_example.py
│ ├── data/
│ │ ├── 2wikimultihopqa.json
│ │ └── 2wikimultihopqa_corpus.json
│ ├── load_and_crawl_using_docling.py
│ ├── load_local_file_using_unstructured.py
│ └── load_website_using_firecrawl.py
├── main.py
├── mkdocs.yml
├── pyproject.toml
└── tests/
├── __init__.py
├── agent/
│ ├── __init__.py
│ ├── test_base.py
│ ├── test_chain_of_rag.py
│ ├── test_collection_router.py
│ ├── test_deep_search.py
│ ├── test_naive_rag.py
│ └── test_rag_router.py
├── embedding/
│ ├── __init__.py
│ ├── test_base.py
│ ├── test_bedrock_embedding.py
│ ├── test_fastembed_embedding.py
│ ├── test_gemini_embedding.py
│ ├── test_glm_embedding.py
│ ├── test_jiekouai_embedding.py
│ ├── test_milvus_embedding.py
│ ├── test_novita_embedding.py
│ ├── test_ollama_embedding.py
│ ├── test_openai_embedding.py
│ ├── test_ppio_embedding.py
│ ├── test_sentence_transformer_embedding.py
│ ├── test_siliconflow_embedding.py
│ ├── test_volcengine_embedding.py
│ ├── test_voyage_embedding.py
│ └── test_watsonx_embedding.py
├── llm/
│ ├── __init__.py
│ ├── test_aliyun.py
│ ├── test_anthropic.py
│ ├── test_azure_openai.py
│ ├── test_base.py
│ ├── test_bedrock.py
│ ├── test_deepseek.py
│ ├── test_gemini.py
│ ├── test_glm.py
│ ├── test_jiekouai.py
│ ├── test_novita.py
│ ├── test_ollama.py
│ ├── test_openai.py
│ ├── test_ppio.py
│ ├── test_siliconflow.py
│ ├── test_together_ai.py
│ ├── test_volcengine.py
│ ├── test_watsonx.py
│ └── test_xai.py
├── loader/
│ ├── __init__.py
│ ├── file_loader/
│ │ ├── __init__.py
│ │ ├── test_base.py
│ │ ├── test_docling_loader.py
│ │ ├── test_json_loader.py
│ │ ├── test_pdf_loader.py
│ │ ├── test_text_loader.py
│ │ └── test_unstructured_loader.py
│ ├── test_splitter.py
│ └── web_crawler/
│ ├── __init__.py
│ ├── test_base.py
│ ├── test_crawl4ai_crawler.py
│ ├── test_docling_crawler.py
│ ├── test_firecrawl_crawler.py
│ └── test_jina_crawler.py
├── utils/
│ └── test_log.py
└── vector_db/
├── test_azure_search.py
├── test_base.py
├── test_milvus.py
├── test_oracle.py
└── test_qdrant.py
================================================
FILE CONTENTS
================================================
================================================
FILE: .github/ISSUE_TEMPLATE/bug_report.md
================================================
---
name: Bug report
about: Create a report to help us improve
title: ''
labels: ''
assignees: ''
---
Please describe your issue **in English**
*Note: Small LLMs cannot perform well at prompt following, and are prone to hallucinations. Please make sure your LLM is cutting-edge, preferably a reasoning model, e.g. OpenAI o-series, DeepSeek R1, Claude 3.7 Sonnet etc.*
**Describe the bug**
A clear and concise description of what the bug is.
**To Reproduce**
Steps to reproduce the behavior:
**Expected behavior**
A clear and concise description of what you expected to happen.
**Screenshots**
If applicable, add screenshots to help explain your problem.
**Environment (please complete the following information):**
- OS: [e.g. MacOS]
- pip dependencies
- Version [e.g. 0.0.1]
**Additional context**
Add any other context about the problem here.
================================================
FILE: .github/ISSUE_TEMPLATE/feature_request.md
================================================
---
name: Feature request
about: Suggest an idea for this project
title: ''
labels: ''
assignees: ''
---
Please describe your suggestion **in English**.
**Is your feature request related to a problem? Please describe.**
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
**Describe the solution you'd like**
A clear and concise description of what you want to happen.
**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.
**Additional context**
Add any other context or screenshots about the feature request here.
================================================
FILE: .github/mergify.yml
================================================
misc:
- branch: &BRANCHES
# In this pull request, the changes are based on the main branch
- &MASTER_BRANCH base=main
- name: Label bug fix PRs
conditions:
# branch condition: in this pull request, the changes are based on any branch referenced by BRANCHES
- or: *BRANCHES
- 'title~=^fix:'
actions:
label:
add:
- kind/bug
- name: Label feature PRs
conditions:
# branch condition: in this pull request, the changes are based on any branch referenced by BRANCHES
- or: *BRANCHES
- 'title~=^feat:'
actions:
label:
add:
- kind/feature
- name: Label enhancement PRs
conditions:
# branch condition: in this pull request, the changes are based on any branch referenced by BRANCHES
- or: *BRANCHES
- 'title~=^enhance:'
actions:
label:
add:
- kind/enhancement
================================================
FILE: .github/workflows/cd-docs.yml
================================================
name: "Run Docs CD with UV"
on:
push:
branches:
- "main"
- "master"
paths:
- 'docs/**'
- 'mkdocs.yml'
- '.github/workflows/docs.yml'
jobs:
build-deploy-docs:
if: github.repository == 'zilliztech/deep-searcher'
uses: ./.github/workflows/docs.yml
with:
deploy: true
permissions:
contents: write
================================================
FILE: .github/workflows/ci-docs.yml
================================================
name: "Run Docs CI with UV"
on:
pull_request:
types: [opened, reopened, synchronize]
paths:
- 'docs/**'
- 'mkdocs.yml'
- '.github/workflows/docs.yml'
push:
branches:
- "**"
- "!gh-pages"
paths:
- 'docs/**'
- 'mkdocs.yml'
- '.github/workflows/docs.yml'
jobs:
build-docs:
if: ${{ github.event_name == 'push' || (github.event.pull_request.head.repo.full_name != 'zilliztech/deep-searcher') }}
uses: ./.github/workflows/docs.yml
with:
deploy: false
================================================
FILE: .github/workflows/docs.yml
================================================
on:
workflow_call:
inputs:
deploy:
type: boolean
description: "If true, the docs will be deployed."
default: false
jobs:
run-docs:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Install uv
uses: astral-sh/setup-uv@v5
- name: Install dependencies
run: |
uv sync --all-extras --dev
source .venv/bin/activate
- name: Build docs
run: uv run mkdocs build --verbose --clean
- name: Build and push docs
if: inputs.deploy
run: uv run mkdocs gh-deploy --force
================================================
FILE: .github/workflows/release.yml
================================================
#git tag v0.x.x # Must be same as the version in pyproject.toml
#git push --tags
name: Publish Python Package to PyPI
on:
push:
tags:
- "v*"
jobs:
publish:
name: Publish to PyPI
runs-on: ubuntu-latest
environment: pypi
permissions:
id-token: write
contents: read
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: "3.10"
- name: Install build tools
run: python -m pip install build
- name: Build package
run: python -m build
- name: Publish to PyPI
uses: pypa/gh-action-pypi-publish@release/v1
================================================
FILE: .github/workflows/ruff.yml
================================================
name: Ruff
on:
push:
branches: [ main, master ]
pull_request:
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Install uv
uses: astral-sh/setup-uv@v5
- name: Install the project
run: |
uv sync --all-extras --dev
source .venv/bin/activate
- name: Run Ruff
run: |
uv run ruff format --diff
uv run ruff check
# - name: Run tests
# run: uv run pytest tests
================================================
FILE: .gitignore
================================================
# Created by https://www.toptal.com/developers/gitignore/api/python,visualstudiocode
# Edit at https://www.toptal.com/developers/gitignore?templates=python,visualstudiocode
### Python ###
# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class
# C extensions
*.so
# Distribution / packaging
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
share/python-wheels/
*.egg-info/
.installed.cfg
*.egg
MANIFEST
# PyInstaller
# Usually these files are written by a python script from a template
# before PyInstaller builds the exe, so as to inject date/other infos into it.
*.manifest
*.spec
# Installer logs
pip-log.txt
pip-delete-this-directory.txt
# Unit test / coverage reports
htmlcov/
.tox/
.nox/
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*.cover
*.py,cover
.hypothesis/
.pytest_cache/
cover/
# Translations
*.mo
*.pot
# Django stuff:
*.log
local_settings.py
db.sqlite3
db.sqlite3-journal
# Flask stuff:
instance/
.webassets-cache
# Scrapy stuff:
.scrapy
# Sphinx documentation
docs/_build/
# PyBuilder
.pybuilder/
target/
# Jupyter Notebook
.ipynb_checkpoints
# IPython
profile_default/
ipython_config.py
# pyenv
# For a library or package, you might want to ignore these files since the code is
# intended to run in multiple environments; otherwise, check them in:
# .python-version
# pipenv
# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
# However, in case of collaboration, if having platform-specific dependencies or dependencies
# having no cross-platform support, pipenv may install dependencies that don't work, or not
# install all needed dependencies.
#Pipfile.lock
# poetry
# Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
# This is especially recommended for binary packages to ensure reproducibility, and is more
# commonly ignored for libraries.
# https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
#poetry.lock
# pdm
# Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
#pdm.lock
# pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it
# in version control.
# https://pdm.fming.dev/#use-with-ide
.pdm.toml
# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
__pypackages__/
# Celery stuff
celerybeat-schedule
celerybeat.pid
# SageMath parsed files
*.sage.py
# Environments
.env
.venv
env/
venv/
ENV/
env.bak/
venv.bak/
# Spyder project settings
.spyderproject
.spyproject
# Rope project settings
.ropeproject
# mkdocs documentation
/site
# mypy
.mypy_cache/
.dmypy.json
dmypy.json
# Pyre type checker
.pyre/
# pytype static type analyzer
.pytype/
# Cython debug symbols
cython_debug/
# PyCharm
# JetBrains specific template is maintained in a separate JetBrains.gitignore that can
# be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
# and can be added to the global gitignore or merged into this file. For a more nuclear
# option (not recommended) you can uncomment the following to ignore the entire idea folder.
#.idea/
### Python Patch ###
# Poetry local configuration file - https://python-poetry.org/docs/configuration/#local-configuration
poetry.toml
# ruff
.ruff_cache/
# LSP config files
pyrightconfig.json
### VisualStudioCode ###
.vscode/*
!.vscode/settings.json
!.vscode/tasks.json
!.vscode/launch.json
!.vscode/extensions.json
!.vscode/*.code-snippets
# Local History for Visual Studio Code
.history/
# Built Visual Studio Code Extensions
*.vsix
### VisualStudioCode Patch ###
# Ignore all local history of files
.history
.ionide
# End of https://www.toptal.com/developers/gitignore/api/python,visualstudiocode
.DS_Store
*.db
================================================
FILE: .python-version
================================================
3.10
================================================
FILE: .vscode/settings.json
================================================
{
"python.testing.unittestArgs": [
"-v",
"-s",
"./tests",
"-p",
"test_*.py"
],
"python.testing.pytestEnabled": false,
"python.testing.unittestEnabled": true
}
================================================
FILE: CONTRIBUTING.md
================================================
# Contributing to DeepSearcher
We welcome contributions from everyone. This document provides guidelines to make the contribution process straightforward.
## Pull Request Process
1. Fork the repository and create your branch from `master`.
2. Make your changes.
3. Run tests and linting to ensure your code meets the project's standards.
4. Update documentation if necessary.
5. Submit a pull request.
## Linting and Formatting
Keeping a consistent style for code, code comments, commit messages, and PR descriptions will greatly accelerate your PR review process.
We require you to run code linter and formatter before submitting your pull requests:
To check the coding styles:
```shell
make lint
```
To fix the coding styles:
```shell
make format
```
Our CI pipeline also runs these checks automatically on all pull requests to ensure code quality and consistency.
## Development Environment Setup with uv
DeepSearcher uses [uv](https://github.com/astral-sh/uv) as the recommended package manager. uv is a fast, reliable Python package manager and installer. The project's `pyproject.toml` is configured to work with uv, which will provide faster dependency resolution and package installation compared to traditional tools.
### Install Project in Development Mode(aka Editable Installation)
1. Install uv if you haven't already:
Follow the [offical installation instructions](https://docs.astral.sh/uv/getting-started/installation/).
2. Clone the repository and navigate to the project directory:
```shell
git clone https://github.com/zilliztech/deep-searcher.git && cd deep-searcher
```
3. Synchronize and install dependencies:
```shell
uv sync
source .venv/bin/activate
```
`uv sync` will install all dependencies specified in `uv.lock` file. And the `source .venv/bin/activate` command will activate the virtual environment.
- (Optional) To install all optional dependencies:
```shell
uv sync --all-extras --dev
```
- (Optional) To install specific optional dependencies:
```shell
# Take optional `ollama` dependency for example
uv sync --extra ollama
```
For more optional dependencies, refer to the `[project.optional-dependencies]` part of `pyproject.toml` file.
### Adding Dependencies
When you need to add new dependencies to the `pyproject.toml` file, you can use the following commands:
```shell
uv add <package_name>
```
DeepSearcher uses optional dependencies to keep the default installation lightweight. Optional features can be installed using the syntax `deepsearcher[<extra>]`. To add a dependency to an optional extra, use the following command:
```shell
uv add <package_name> --optional <extra>
```
For more details, refer to the [offical Managing dependencies documentation](https://docs.astral.sh/uv/concepts/projects/dependencies/).
### Dependencies Locking
For development, we use lockfiles to ensure consistent dependencies. You can use
```shell
uv lock --check
```
to verify if your lockfile is up-to-date with your project dependencies.
When you modify or add dependencies in the project, the lockfile will be automatically updated the next time you run a uv command. You can also explicitly update the lockfile using:
```shell
uv lock
```
While the environment is synced automatically, it may also be explicitly synced using uv sync:
```shell
uv sync
```
Syncing the environment manually is especially useful for ensuring your editor has the correct versions of dependencies.
For more detailed information about dependency locking and syncing, refer to the [offical Locking and syncing documentation](https://docs.astral.sh/uv/concepts/projects/sync/).
## Running Tests
Before submitting your pull request, make sure to run the test suite to ensure your changes haven't introduced any regressions.
### Installing Test Dependencies
First, ensure you have pytest installed. If you haven't installed the development dependencies yet, you can do so with:
```shell
uv sync --all-extras --dev
```
This will install all development dependencies and optional dependencies including pytest and other testing tools.
### Running the Tests
To run all tests in the `tests` directory:
```shell
uv run pytest tests
```
For more verbose output that shows individual test results:
```shell
uv run pytest tests -v
```
You can also run tests for specific directories or files. For example:
```shell
# Run tests in a specific directory
uv run pytest tests/embedding
# Run tests in a specific file
uv run pytest tests/embedding/test_bedrock_embedding.py
# Run a specific test class
uv run pytest tests/embedding/test_bedrock_embedding.py::TestBedrockEmbedding
# Run a specific test method
uv run pytest tests/embedding/test_bedrock_embedding.py::TestBedrockEmbedding::test_init_default
```
The `-v` flag (verbose mode) provides more detailed output, showing each test case and its result individually. This is particularly useful when you want to see which specific tests are passing or failing.
## Developer Certificate of Origin (DCO)
All contributions require a sign-off, acknowledging the [Developer Certificate of Origin](https://developercertificate.org/).
Add a `Signed-off-by` line to your commit message:
```text
Signed-off-by: Your Name <your.email@example.com>
```
================================================
FILE: Dockerfile
================================================
FROM ghcr.io/astral-sh/uv:python3.10-bookworm-slim
WORKDIR /app
RUN mkdir -p /tmp/uv-cache /app/data /app/logs
COPY pyproject.toml uv.lock LICENSE README.md ./
COPY deepsearcher/ ./deepsearcher/
RUN uv sync
COPY . .
EXPOSE 8000
HEALTHCHECK --interval=30s --timeout=10s --start-period=5s --retries=3 \
CMD curl -f http://localhost:8000/docs || exit 1
CMD ["uv", "run", "python", "main.py", "--enable-cors", "true"]
================================================
FILE: LICENSE
================================================
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright 2019 Zilliz
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
================================================
FILE: MAINTAINERS
================================================
maintainers:
- xiaofan-luan
- SimFG
- zc277584121
================================================
FILE: Makefile
================================================
lint:
uv run ruff format --diff
uv run ruff check
format:
uv run ruff format
uv run ruff check --fix
================================================
FILE: OWNERS
================================================
filters:
".*":
reviewers:
- maintainers
approvers:
- maintainers
================================================
FILE: OWNERS_ALIASES
================================================
aliases:
maintainers:
- xiaofan-luan
- SimFG
- zc277584121
================================================
FILE: README.md
================================================
<div align="center">
[](https://opensource.org/licenses/Apache-2.0)
[](https://deepwiki.com/zilliztech/deep-searcher)
[](https://twitter.com/zilliz_universe)
<a href="https://discord.gg/mKc3R95yE5"><img height="20" src="https://img.shields.io/badge/Discord-%235865F2.svg?style=for-the-badge&logo=discord&logoColor=white" alt="discord"/></a>
</div>
---
DeepSearcher combines cutting-edge LLMs (OpenAI o3, Qwen3, DeepSeek, Grok 4, Claude 4 Sonnet, Llama 4, QwQ, etc.) and Vector Databases (Milvus, Zilliz Cloud etc.) to perform search, evaluation, and reasoning based on private data, providing highly accurate answer and comprehensive report. This project is suitable for enterprise knowledge management, intelligent Q&A systems, and information retrieval scenarios.
## 🚀 Features
- **Private Data Search**: Maximizes the utilization of enterprise internal data while ensuring data security. When necessary, it can integrate online content for more accurate answers.
- **Vector Database Management**: Supports Milvus and other vector databases, allowing data partitioning for efficient retrieval.
- **Flexible Embedding Options**: Compatible with multiple embedding models for optimal selection.
- **Multiple LLM Support**: Supports DeepSeek, OpenAI, and other large models for intelligent Q&A and content generation.
- **Document Loader**: Supports local file loading, with web crawling capabilities under development.
---
## 🎉 Demo
## 📖 Quick Start
### Installation
Install DeepSearcher using one of the following methods:
#### Option 1: Using pip
Create and activate a virtual environment(Python 3.10 version is recommended).
```bash
python -m venv .venv
source .venv/bin/activate
```
Install DeepSearcher
```bash
pip install deepsearcher
```
For optional dependencies, e.g., ollama:
```bash
pip install "deepsearcher[ollama]"
```
#### Option 2: Install in Development Mode
We recommend using [uv](https://github.com/astral-sh/uv) for faster and more reliable installation. Follow the [offical installation instructions](https://docs.astral.sh/uv/getting-started/installation/) to install it.
Clone the repository and navigate to the project directory:
```shell
git clone https://github.com/zilliztech/deep-searcher.git && cd deep-searcher
```
Synchronize and install dependencies:
```shell
uv sync
source .venv/bin/activate
```
For more detailed development setup and optional dependency installation options, see [CONTRIBUTING.md](CONTRIBUTING.md#development-environment-setup-with-uv).
### Quick start demo
To run this quick start demo, please prepare your `OPENAI_API_KEY` in your environment variables. If you change the LLM in the configuration, make sure to prepare the corresponding API key.
```python
from deepsearcher.configuration import Configuration, init_config
from deepsearcher.online_query import query
config = Configuration()
# Customize your config here,
# more configuration see the Configuration Details section below.
config.set_provider_config("llm", "OpenAI", {"model": "o1-mini"})
config.set_provider_config("embedding", "OpenAIEmbedding", {"model": "text-embedding-ada-002"})
init_config(config = config)
# Load your local data
from deepsearcher.offline_loading import load_from_local_files
load_from_local_files(paths_or_directory=your_local_path)
# (Optional) Load from web crawling (`FIRECRAWL_API_KEY` env variable required)
from deepsearcher.offline_loading import load_from_website
load_from_website(urls=website_url)
# Query
result = query("Write a report about xxx.") # Your question here
```
### Configuration Details:
#### LLM Configuration
<pre><code>config.set_provider_config("llm", "(LLMName)", "(Arguments dict)")</code></pre>
<p>The "LLMName" can be one of the following: ["DeepSeek", "OpenAI", "XAI", "SiliconFlow", "Aliyun", "PPIO", "TogetherAI", "Gemini", "Ollama", "Novita", "Jiekou.AI"]</p>
<p> The "Arguments dict" is a dictionary that contains the necessary arguments for the LLM class.</p>
<details>
<summary>Example (OpenAI)</summary>
<p> Make sure you have prepared your OPENAI API KEY as an env variable <code>OPENAI_API_KEY</code>.</p>
<pre><code>config.set_provider_config("llm", "OpenAI", {"model": "o1-mini"})</code></pre>
<p> More details about OpenAI models: https://platform.openai.com/docs/models </p>
</details>
<details>
<summary>Example (Qwen3 from Aliyun Bailian)</summary>
<p> Make sure you have prepared your Bailian API KEY as an env variable <code>DASHSCOPE_API_KEY</code>.</p>
<pre><code>config.set_provider_config("llm", "Aliyun", {"model": "qwen-plus-latest"})</code></pre>
<p> More details about Aliyun Bailian models: https://bailian.console.aliyun.com </p>
</details>
<details>
<summary>Example (Qwen3 from OpenRouter)</summary>
<pre><code>config.set_provider_config("llm", "OpenAI", {"model": "qwen/qwen3-235b-a22b:free", "base_url": "https://openrouter.ai/api/v1", "api_key": "OPENROUTER_API_KEY"})</code></pre>
<p> More details about OpenRouter models: https://openrouter.ai/qwen/qwen3-235b-a22b:free </p>
</details>
<details>
<summary>Example (DeepSeek from official)</summary>
<p> Make sure you have prepared your DEEPSEEK API KEY as an env variable <code>DEEPSEEK_API_KEY</code>.</p>
<pre><code>config.set_provider_config("llm", "DeepSeek", {"model": "deepseek-reasoner"})</code></pre>
<p> More details about DeepSeek: https://api-docs.deepseek.com/ </p>
</details>
<details>
<summary>Example (DeepSeek from SiliconFlow)</summary>
<p> Make sure you have prepared your SILICONFLOW API KEY as an env variable <code>SILICONFLOW_API_KEY</code>.</p>
<pre><code>config.set_provider_config("llm", "SiliconFlow", {"model": "deepseek-ai/DeepSeek-R1"})</code></pre>
<p> More details about SiliconFlow: https://docs.siliconflow.cn/quickstart </p>
</details>
<details>
<summary>Example (DeepSeek from TogetherAI)</summary>
<p> Make sure you have prepared your TOGETHER API KEY as an env variable <code>TOGETHER_API_KEY</code>.</p>
For deepseek R1:
<pre><code>config.set_provider_config("llm", "TogetherAI", {"model": "deepseek-ai/DeepSeek-R1"})</code></pre>
For Llama 4:
<pre><code>config.set_provider_config("llm", "TogetherAI", {"model": "meta-llama/Llama-4-Scout-17B-16E-Instruct"})</code></pre>
<p> You need to install together before running, execute: <code>pip install together</code>. More details about TogetherAI: https://www.together.ai/ </p>
</details>
<details>
<summary>Example (XAI Grok)</summary>
<p> Make sure you have prepared your XAI API KEY as an env variable <code>XAI_API_KEY</code>.</p>
<pre><code>config.set_provider_config("llm", "XAI", {"model": "grok-4-0709"})</code></pre>
<p> More details about XAI Grok: https://docs.x.ai/docs/overview#featured-models </p>
</details>
<details>
<summary>Example (Claude)</summary>
<p> Make sure you have prepared your ANTHROPIC API KEY as an env variable <code>ANTHROPIC_API_KEY</code>.</p>
<pre><code>config.set_provider_config("llm", "Anthropic", {"model": "claude-sonnet-4-0"})</code></pre>
<p> More details about Anthropic Claude: https://docs.anthropic.com/en/home </p>
</details>
<details>
<summary>Example (Google Gemini)</summary>
<p> Make sure you have prepared your GEMINI API KEY as an env variable <code>GEMINI_API_KEY</code>.</p>
<pre><code>config.set_provider_config('llm', 'Gemini', { 'model': 'gemini-2.0-flash' })</code></pre>
<p> You need to install gemini before running, execute: <code>pip install google-genai</code>. More details about Gemini: https://ai.google.dev/gemini-api/docs </p>
</details>
<details>
<summary>Example (DeepSeek from PPIO)</summary>
<p> Make sure you have prepared your PPIO API KEY as an env variable <code>PPIO_API_KEY</code>. You can create an API Key <a href="https://ppinfra.com/settings/key-management?utm_source=github_deep-searcher">here</a>. </p>
<pre><code>config.set_provider_config("llm", "PPIO", {"model": "deepseek/deepseek-r1-turbo"})</code></pre>
<p> More details about PPIO: https://ppinfra.com/docs/get-started/quickstart.html?utm_source=github_deep-searcher </p>
</details>
<details>
<summary>Example (Claude Sonnet 4.5 from Jiekou.AI)</summary>
<p> Make sure you have prepared your Jiekou.AI API KEY as an env variable <code>JIEKOU_API_KEY</code>. You can create an API Key <a href="https://jiekou.ai/settings/key-management?utm_source=github_deep-searcher">here</a>. </p>
<pre><code>config.set_provider_config("llm", "JiekouAI", {"model": "claude-sonnet-4-5-20250929"})</code></pre>
<p> More details about Jiekou.AI: https://docs.jiekou.ai/docs/support/quickstart?utm_source=github_deep-searcher </p>
</details>
<details>
<summary>Example (Ollama)</summary>
<p> Follow <a href="https://github.com/jmorganca/ollama">these instructions</a> to set up and run a local Ollama instance:</p>
<p> <a href="https://ollama.ai/download">Download</a> and install Ollama onto the available supported platforms (including Windows Subsystem for Linux).</p>
<p> View a list of available models via the <a href="https://ollama.ai/library">model library</a>.</p>
<p> Fetch available LLM models via <code>ollama pull <name-of-model></code></p>
<p> Example: <code>ollama pull qwen3</code></p>
<p> To chat directly with a model from the command line, use <code>ollama run <name-of-model></code>.</p>
<p> By default, Ollama has a REST API for running and managing models on <a href="http://localhost:11434">http://localhost:11434</a>.</p>
<pre><code>config.set_provider_config("llm", "Ollama", {"model": "qwen3"})</code></pre>
</details>
<details>
<summary>Example (Volcengine)</summary>
<p> Make sure you have prepared your Volcengine API KEY as an env variable <code>VOLCENGINE_API_KEY</code>. You can create an API Key <a href="https://console.volcengine.com/ark/region:ark+cn-beijing/apiKey">here</a>. </p>
<pre><code>config.set_provider_config("llm", "Volcengine", {"model": "deepseek-r1-250120"})</code></pre>
<p> More details about Volcengine: https://www.volcengine.com/docs/82379/1099455?utm_source=github_deep-searcher </p>
</details>
<details>
<summary>Example (GLM)</summary>
<p> Make sure you have prepared your GLM API KEY as an env variable <code>GLM_API_KEY</code>.</p>
<pre><code>config.set_provider_config("llm", "GLM", {"model": "glm-4-plus"})</code></pre>
<p> You need to install zhipuai before running, execute: <code>pip install zhipuai</code>. More details about GLM: https://bigmodel.cn/dev/welcome </p>
</details>
<details>
<summary>Example (Amazon Bedrock)</summary>
<p> Make sure you have prepared your Amazon Bedrock API KEY as an env variable <code>AWS_ACCESS_KEY_ID</code> and <code>AWS_SECRET_ACCESS_KEY</code>.</p>
<pre><code>config.set_provider_config("llm", "Bedrock", {"model": "us.deepseek.r1-v1:0"})</code></pre>
<p> You need to install boto3 before running, execute: <code>pip install boto3</code>. More details about Amazon Bedrock: https://docs.aws.amazon.com/bedrock/ </p>
</details>
<details>
<summary>Example (IBM watsonx.ai)</summary>
<p> Make sure you have prepared your watsonx.ai credentials as env variables <code>WATSONX_APIKEY</code>, <code>WATSONX_URL</code>, and <code>WATSONX_PROJECT_ID</code>.</p>
<pre><code>config.set_provider_config("llm", "watsonx", {"model": "us.deepseek.r1-v1:0"})</code></pre>
<p> You need to install ibm-watsonx-ai before running, execute: <code>pip install ibm-watsonx-ai</code>. More details about IBM watsonx.ai: https://www.ibm.com/products/watsonx-ai/foundation-models </p>
</details>
#### Embedding Model Configuration
<pre><code>config.set_provider_config("embedding", "(EmbeddingModelName)", "(Arguments dict)")</code></pre>
<p>The "EmbeddingModelName" can be one of the following: ["MilvusEmbedding", "OpenAIEmbedding", "VoyageEmbedding", "SiliconflowEmbedding", "PPIOEmbedding", "NovitaEmbedding", "JiekouAIEmbedding"]</p>
<p> The "Arguments dict" is a dictionary that contains the necessary arguments for the embedding model class.</p>
<details>
<summary>Example (OpenAI embedding)</summary>
<p> Make sure you have prepared your OpenAI API KEY as an env variable <code>OPENAI_API_KEY</code>.</p>
<pre><code>config.set_provider_config("embedding", "OpenAIEmbedding", {"model": "text-embedding-3-small"})</code></pre>
<p> More details about OpenAI models: https://platform.openai.com/docs/guides/embeddings/use-cases </p>
</details>
<details>
<summary>Example (OpenAI embedding Azure)</summary>
<p> Make sure you have prepared your OpenAI API KEY as an env variable <code>OPENAI_API_KEY</code>.</p>
<pre><code>config.set_provider_config("embedding", "OpenAIEmbedding", {
"model": "text-embedding-ada-002",
"azure_endpoint": "https://<youraifoundry>.openai.azure.com/",
"api_version": "2023-05-15"
})</code></pre>
</details>
<details>
<summary>Example (Pymilvus built-in embedding model)</summary>
<p> Use the built-in embedding model in Pymilvus, you can set the model name as <code>"default"</code>, <code>"BAAI/bge-base-en-v1.5"</code>, <code>"BAAI/bge-large-en-v1.5"</code>, <code>"jina-embeddings-v3"</code>, etc. <br/>
See [milvus_embedding.py](deepsearcher/embedding/milvus_embedding.py) for more details. </p>
<pre><code>config.set_provider_config("embedding", "MilvusEmbedding", {"model": "BAAI/bge-base-en-v1.5"})</code></pre>
<pre><code>config.set_provider_config("embedding", "MilvusEmbedding", {"model": "jina-embeddings-v3"})</code></pre>
<p> For Jina's embedding model, you need<code>JINAAI_API_KEY</code>.</p>
<p> You need to install pymilvus model before running, execute: <code>pip install pymilvus.model</code>. More details about Pymilvus: https://milvus.io/docs/embeddings.md </p>
</details>
<details>
<summary>Example (VoyageAI embedding)</summary>
<p> Make sure you have prepared your VOYAGE API KEY as an env variable <code>VOYAGE_API_KEY</code>.</p>
<pre><code>config.set_provider_config("embedding", "VoyageEmbedding", {"model": "voyage-3"})</code></pre>
<p> You need to install voyageai before running, execute: <code>pip install voyageai</code>. More details about VoyageAI: https://docs.voyageai.com/embeddings/ </p>
</details>
<details>
<summary>Example (Amazon Bedrock embedding)</summary>
<pre><code>config.set_provider_config("embedding", "BedrockEmbedding", {"model": "amazon.titan-embed-text-v2:0"})</code></pre>
<p> You need to install boto3 before running, execute: <code>pip install boto3</code>. More details about Amazon Bedrock: https://docs.aws.amazon.com/bedrock/ </p>
</details>
<details>
<summary>Example (Novita AI embedding)</summary>
<p> Make sure you have prepared your Novita AI API KEY as an env variable <code>NOVITA_API_KEY</code>.</p>
<pre><code>config.set_provider_config("embedding", "NovitaEmbedding", {"model": "baai/bge-m3"})</code></pre>
<p> More details about Novita AI: https://novita.ai/docs/api-reference/model-apis-llm-create-embeddings?utm_source=github_deep-searcher&utm_medium=github_readme&utm_campaign=link </p>
</details>
<details>
<summary>Example (Siliconflow embedding)</summary>
<p> Make sure you have prepared your Siliconflow API KEY as an env variable <code>SILICONFLOW_API_KEY</code>.</p>
<pre><code>config.set_provider_config("embedding", "SiliconflowEmbedding", {"model": "BAAI/bge-m3"})</code></pre>
<p> More details about Siliconflow: https://docs.siliconflow.cn/en/api-reference/embeddings/create-embeddings </p>
</details>
<details>
<summary>Example (Volcengine embedding)</summary>
<p> Make sure you have prepared your Volcengine API KEY as an env variable <code>VOLCENGINE_API_KEY</code>.</p>
<pre><code>config.set_provider_config("embedding", "VolcengineEmbedding", {"model": "doubao-embedding-text-240515"})</code></pre>
<p> More details about Volcengine: https://www.volcengine.com/docs/82379/1302003 </p>
</details>
<details>
<summary>Example (GLM embedding)</summary>
<p> Make sure you have prepared your GLM API KEY as an env variable <code>GLM_API_KEY</code>.</p>
<pre><code>config.set_provider_config("embedding", "GLMEmbedding", {"model": "embedding-3"})</code></pre>
<p> You need to install zhipuai before running, execute: <code>pip install zhipuai</code>. More details about GLM: https://bigmodel.cn/dev/welcome </p>
</details>
<details>
<summary>Example (Google Gemini embedding)</summary>
<p> Make sure you have prepared your Gemini API KEY as an env variable <code>GEMINI_API_KEY</code>.</p>
<pre><code>config.set_provider_config("embedding", "GeminiEmbedding", {"model": "text-embedding-004"})</code></pre>
<p> You need to install gemini before running, execute: <code>pip install google-genai</code>. More details about Gemini: https://ai.google.dev/gemini-api/docs </p>
</details>
<details>
<summary>Example (Ollama embedding)</summary>
<pre><code>config.set_provider_config("embedding", "OllamaEmbedding", {"model": "bge-m3"})</code></pre>
<p> You need to install ollama before running, execute: <code>pip install ollama</code>. More details about Ollama Python SDK: https://github.com/ollama/ollama-python </p>
</details>
<details>
<summary>Example (PPIO embedding)</summary>
<p> Make sure you have prepared your PPIO API KEY as an env variable <code>PPIO_API_KEY</code>.</p>
<pre><code>config.set_provider_config("embedding", "PPIOEmbedding", {"model": "baai/bge-m3"})</code></pre>
<p> More details about PPIO: https://ppinfra.com/docs/get-started/quickstart.html?utm_source=github_deep-searcher </p>
</details>
<details>
<summary>Example (Jiekou.AI embedding)</summary>
<p> Make sure you have prepared your Jiekou.AI API KEY as an env variable <code>JIEKOU_API_KEY</code>.</p>
<pre><code>config.set_provider_config("embedding", "JiekouAIEmbedding", {"model": "qwen/qwen3-embedding-8b"})</code></pre>
<p> More details about Jiekou.AI: https://docs.jiekou.ai/docs/support/quickstart?utm_source=github_deep-searcher </p>
</details>
<details>
<summary>Example (FastEmbed embedding)</summary>
<pre><code>config.set_provider_config("embedding", "FastEmbedEmbedding", {"model": "intfloat/multilingual-e5-large"})</code></pre>
<p> You need to install fastembed before running, execute: <code>pip install fastembed</code>. More details about fastembed: https://github.com/qdrant/fastembed </p>
</details>
<details>
<summary>Example (IBM watsonx.ai embedding)</summary>
<p> Make sure you have prepared your WatsonX credentials as env variables <code>WATSONX_APIKEY</code>, <code>WATSONX_URL</code>, and <code>WATSONX_PROJECT_ID</code>.</p>
<pre><code>config.set_provider_config("embedding", "WatsonXEmbedding", {"model": "ibm/slate-125m-english-rtrvr-v2"})</code></pre>
<pre><code>config.set_provider_config("embedding", "WatsonXEmbedding", {"model": "sentence-transformers/all-minilm-l6-v2"})</code></pre>
<p> You need to install ibm-watsonx-ai before running, execute: <code>pip install ibm-watsonx-ai</code>. More details about IBM watsonx.ai: https://www.ibm.com/products/watsonx-ai/foundation-models </p>
</details>
#### Vector Database Configuration
<pre><code>config.set_provider_config("vector_db", "(VectorDBName)", "(Arguments dict)")</code></pre>
<p>The "VectorDBName" can be one of the following: ["Milvus"] (Under development)</p>
<p> The "Arguments dict" is a dictionary that contains the necessary arguments for the Vector Database class.</p>
<details>
<summary>Example (Milvus)</summary>
<pre><code>config.set_provider_config("vector_db", "Milvus", {"uri": "./milvus.db", "token": ""})</code></pre>
<p> More details about Milvus Config:</p>
<ul>
<li>
Setting the <code>uri</code> as a local file, e.g. <code>./milvus.db</code>, is the most convenient method, as it automatically utilizes <a href="https://milvus.io/docs/milvus_lite.md" target="_blank">Milvus Lite</a> to store all data in this file.
</li>
</ul>
<ul>
<li>
If you have a large-scale dataset, you can set up a more performant Milvus server using
<a href="https://milvus.io/docs/quickstart.md" target="_blank">Docker or Kubernetes</a>.
In this setup, use the server URI, e.g., <code>http://localhost:19530</code>, as your <code>uri</code>.
You can also use any other connection parameters supported by Milvus such as <code>host</code>, <code>user</code>, <code>password</code>, or <code>secure</code>.
</li>
</ul>
<ul>
<li>
If you want to use <a href="https://zilliz.com/cloud" target="_blank">Zilliz Cloud</a>,
the fully managed cloud service for Milvus, adjust the <code>uri</code> and <code>token</code>
according to the <a href="https://docs.zilliz.com/docs/on-zilliz-cloud-console#free-cluster-details"
target="_blank">Public Endpoint and API Key</a> in Zilliz Cloud.
</li>
</ul>
</details>
<details>
<summary>Example (AZURE AI Search)</summary>
<pre><code>config.set_provider_config("vector_db", "AzureSearch", {
"endpoint": "https://<yourazureaisearch>.search.windows.net",
"index_name": "<yourindex>",
"api_key": "<yourkey>",
"vector_field": ""
})</code></pre>
<p> More details about Milvus Config:</p>
</details>
#### File Loader Configuration
<pre><code>config.set_provider_config("file_loader", "(FileLoaderName)", "(Arguments dict)")</code></pre>
<p>The "FileLoaderName" can be one of the following: ["PDFLoader", "TextLoader", "UnstructuredLoader"]</p>
<p> The "Arguments dict" is a dictionary that contains the necessary arguments for the File Loader class.</p>
<details>
<summary>Example (Unstructured)</summary>
<p>You can use Unstructured in two ways:</p>
<ul>
<li>With API: Set environment variables <code>UNSTRUCTURED_API_KEY</code> and <code>UNSTRUCTURED_API_URL</code></li>
<li>Without API: Use the local processing mode by simply not setting these environment variables</li>
</ul>
<pre><code>config.set_provider_config("file_loader", "UnstructuredLoader", {})</code></pre>
<ul>
<li>Currently supported file types: ["pdf"] (Under development)</li>
<li>Installation requirements:
<ul>
<li>Install ingest pipeline: <code>pip install unstructured-ingest</code></li>
<li>For all document formats: <code>pip install "unstructured[all-docs]"</code></li>
<li>For specific formats (e.g., PDF only): <code>pip install "unstructured[pdf]"</code></li>
</ul>
</li>
<li>More information:
<ul>
<li>Unstructured documentation: <a href="https://docs.unstructured.io/ingestion/overview">https://docs.unstructured.io/ingestion/overview</a></li>
<li>Installation guide: <a href="https://docs.unstructured.io/open-source/installation/full-installation">https://docs.unstructured.io/open-source/installation/full-installation</a></li>
</ul>
</li>
</ul>
</details>
<details>
<summary>Example (Docling)</summary>
<pre><code>config.set_provider_config("file_loader", "DoclingLoader", {})</code></pre>
<p> Currently supported file types: please refer to the Docling documentation: https://docling-project.github.io/docling/usage/supported_formats/#supported-output-formats </p>
<p> You need to install docling before running, execute: <code>pip install docling</code>. More details about Docling: https://docling-project.github.io/docling/ </p>
</details>
#### Web Crawler Configuration
<pre><code>config.set_provider_config("web_crawler", "(WebCrawlerName)", "(Arguments dict)")</code></pre>
<p>The "WebCrawlerName" can be one of the following: ["FireCrawlCrawler", "Crawl4AICrawler", "JinaCrawler"]</p>
<p> The "Arguments dict" is a dictionary that contains the necessary arguments for the Web Crawler class.</p>
<details>
<summary>Example (FireCrawl)</summary>
<p> Make sure you have prepared your FireCrawl API KEY as an env variable <code>FIRECRAWL_API_KEY</code>.</p>
<pre><code>config.set_provider_config("web_crawler", "FireCrawlCrawler", {})</code></pre>
<p> More details about FireCrawl: https://docs.firecrawl.dev/introduction </p>
</details>
<details>
<summary>Example (Crawl4AI)</summary>
<p> Make sure you have run <code>crawl4ai-setup</code> in your environment.</p>
<pre><code>config.set_provider_config("web_crawler", "Crawl4AICrawler", {"browser_config": {"headless": True, "verbose": True}})</code></pre>
<p> You need to install crawl4ai before running, execute: <code>pip install crawl4ai</code>. More details about Crawl4AI: https://docs.crawl4ai.com/ </p>
</details>
<details>
<summary>Example (Jina Reader)</summary>
<p> Make sure you have prepared your Jina Reader API KEY as an env variable <code>JINA_API_TOKEN</code> or <code>JINAAI_API_KEY</code>.</p>
<pre><code>config.set_provider_config("web_crawler", "JinaCrawler", {})</code></pre>
<p> More details about Jina Reader: https://jina.ai/reader/ </p>
</details>
<details>
<summary>Example (Docling)</summary>
<pre><code>config.set_provider_config("web_crawler", "DoclingCrawler", {})</code></pre>
<p> Currently supported file types: please refer to the Docling documentation: https://docling-project.github.io/docling/usage/supported_formats/#supported-output-formats </p>
<p> You need to install docling before running, execute: <code>pip install docling</code>. More details about Docling: https://docling-project.github.io/docling/ </p>
</details>
### Python CLI Mode
#### Load
```shell
deepsearcher load "your_local_path_or_url"
# load into a specific collection
deepsearcher load "your_local_path_or_url" --collection_name "your_collection_name" --collection_desc "your_collection_description"
```
Example loading from local file:
```shell
deepsearcher load "/path/to/your/local/file.pdf"
# or more files at once
deepsearcher load "/path/to/your/local/file1.pdf" "/path/to/your/local/file2.md"
```
Example loading from url (*Set `FIRECRAWL_API_KEY` in your environment variables, see [FireCrawl](https://docs.firecrawl.dev/introduction) for more details*):
```shell
deepsearcher load "https://www.wikiwand.com/en/articles/DeepSeek"
```
#### Query
```shell
deepsearcher query "Write a report about xxx."
```
More help information
```shell
deepsearcher --help
```
For more help information about a specific subcommand, you can use `deepsearcher [subcommand] --help`.
```shell
deepsearcher load --help
deepsearcher query --help
```
### Deployment
#### Configure modules
You can configure all arguments by modifying [config.yaml](./config.yaml) to set up your system with default modules.
For example, set your `OPENAI_API_KEY` in the `llm` section of the YAML file.
#### Start service
The main script will run a FastAPI service with default address `localhost:8000`.
```shell
$ python main.py
```
#### Access via browser
You can open url http://localhost:8000/docs in browser to access the web service.
Click on the button "Try it out", it allows you to fill the parameters and directly interact with the API.
---
## ❓ Q&A
**Q1**: Why I failed to parse LLM output format / How to select the LLM?
**A1**: Small LLMs struggle to follow the prompt to generate a desired response, which usually cause the format parsing problem. A better practice is to use large reasoning models e.g. deepseek-r1 671b, OpenAI o-series, Claude 4 sonnet, etc. as your LLM.
---
**Q2**:
OSError: We couldn't connect to 'https://huggingface.co' to load this file, couldn't find it in the cached files and it looks like GPTCache/paraphrase-albert-small-v2 is not the path to a directory containing a file named config.json.
Checkout your internet connection or see how to run the library in offline mode at 'https://huggingface.co/docs/transformers/installation#offline-mode'.
**A2**: This is mainly due to abnormal access to huggingface, which may be a network or permission problem. You can try the following two methods:
1. If there is a network problem, set up a proxy, try adding the following environment variable.
```bash
export HF_ENDPOINT=https://hf-mirror.com
```
2. If there is a permission problem, set up a personal token, try adding the following environment variable.
```bash
export HUGGING_FACE_HUB_TOKEN=xxxx
```
---
**Q3**: DeepSearcher doesn't run in Jupyter notebook.
**A3**: Install `nest_asyncio` and then put this code block in front of your jupyter notebook.
```
pip install nest_asyncio
```
```
import nest_asyncio
nest_asyncio.apply()
```
---
## 🔧 Module Support
### 🔹 Embedding Models
- [Open-source embedding models](https://milvus.io/docs/embeddings.md)
- [OpenAI](https://platform.openai.com/docs/guides/embeddings/use-cases) (`OPENAI_API_KEY` env variable required)
- [VoyageAI](https://docs.voyageai.com/embeddings/) (`VOYAGE_API_KEY` env variable required)
- [Amazon Bedrock](https://docs.aws.amazon.com/bedrock/) (`AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` env variable required)
- [FastEmbed](https://qdrant.github.io/fastembed/)
- [PPIO](https://ppinfra.com/model-api/product/llm-api?utm_source=github_deep-searcher) (`PPIO_API_KEY` env variable required)
- [Novita AI](https://novita.ai/docs/api-reference/model-apis-llm-create-embeddings?utm_source=github_deep-searcher&utm_medium=github_readme&utm_campaign=link) (`NOVITA_API_KEY` env variable required)
- [IBM watsonx.ai](https://www.ibm.com/products/watsonx-ai/foundation-models#ibmembedding) (`WATSONX_APIKEY`, `WATSONX_URL`, `WATSONX_PROJECT_ID` env variables required)
- [Jiekou.AI](https://jiekou.ai/?utm_source=github_deep-searcher) (`JIEKOU_API_KEY` env variable required)
### 🔹 LLM Support
- [OpenAI](https://platform.openai.com/docs/models) (`OPENAI_API_KEY` env variable required)
- [DeepSeek](https://api-docs.deepseek.com/) (`DEEPSEEK_API_KEY` env variable required)
- [XAI Grok](https://x.ai/api) (`XAI_API_KEY` env variable required)
- [Anthropic Claude](https://docs.anthropic.com/en/home) (`ANTHROPIC_API_KEY` env variable required)
- [SiliconFlow Inference Service](https://docs.siliconflow.cn/en/userguide/introduction) (`SILICONFLOW_API_KEY` env variable required)
- [PPIO](https://ppinfra.com/model-api/product/llm-api?utm_source=github_deep-searcher) (`PPIO_API_KEY` env variable required)
- [TogetherAI Inference Service](https://docs.together.ai/docs/introduction) (`TOGETHER_API_KEY` env variable required)
- [Google Gemini](https://ai.google.dev/gemini-api/docs) (`GEMINI_API_KEY` env variable required)
- [SambaNova Cloud Inference Service](https://docs.together.ai/docs/introduction) (`SAMBANOVA_API_KEY` env variable required)
- [Ollama](https://ollama.com/)
- [Novita AI](https://novita.ai/docs/guides/introduction?utm_source=github_deep-searcher&utm_medium=github_readme&utm_campaign=link) (`NOVITA_API_KEY` env variable required)
- [IBM watsonx.ai](https://www.ibm.com/products/watsonx-ai/foundation-models#ibmfm) (`WATSONX_APIKEY`, `WATSONX_URL`, `WATSONX_PROJECT_ID` env variable required)
- [Jiekou.AI](https://jiekou.ai/?utm_source=github_deep-searcher) (`JIEKOU_API_KEY` env variable required)
### 🔹 Document Loader
- Local File
- PDF(with txt/md) loader
- [Unstructured](https://unstructured.io/) (under development) (`UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_URL` env variables required)
- Web Crawler
- [FireCrawl](https://docs.firecrawl.dev/introduction) (`FIRECRAWL_API_KEY` env variable required)
- [Jina Reader](https://jina.ai/reader/) (`JINA_API_TOKEN` env variable required)
- [Crawl4AI](https://docs.crawl4ai.com/) (You should run command `crawl4ai-setup` for the first time)
### 🔹 Vector Database Support
- [Milvus](https://milvus.io/) and [Zilliz Cloud](https://www.zilliz.com/) (fully managed Milvus)
- [Qdrant](https://qdrant.tech/)
---
## 📊 Evaluation
See the [Evaluation](./evaluation) directory for more details.
---
## 📌 Future Plans
- Enhance web crawling functionality
- Support more vector databases (e.g., FAISS...)
- Add support for additional large models
- Provide RESTful API interface (**DONE**)
We welcome contributions! Star & Fork the project and help us build a more powerful DeepSearcher! 🎯
================================================
FILE: deepsearcher/__init__.py
================================================
import os
# ignore the warnings
# None of PyTorch, TensorFlow >= 2.0, or Flax have been found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.
os.environ["TRANSFORMERS_NO_ADVISORY_WARNINGS"] = "1"
================================================
FILE: deepsearcher/agent/__init__.py
================================================
from .base import BaseAgent, RAGAgent
from .chain_of_rag import ChainOfRAG
from .deep_search import DeepSearch
from .naive_rag import NaiveRAG
__all__ = [
"ChainOfRAG",
"DeepSearch",
"NaiveRAG",
"BaseAgent",
"RAGAgent",
]
================================================
FILE: deepsearcher/agent/base.py
================================================
from abc import ABC
from typing import Any, List, Tuple
from deepsearcher.vector_db import RetrievalResult
def describe_class(description):
"""
Decorator function to add a description to a class.
This decorator adds a __description__ attribute to the decorated class,
which can be used for documentation or introspection.
Args:
description: The description to add to the class.
Returns:
A decorator function that adds the description to the class.
"""
def decorator(cls):
cls.__description__ = description
return cls
return decorator
class BaseAgent(ABC):
"""
Abstract base class for all agents in the DeepSearcher system.
This class defines the basic interface for agents, including initialization
and invocation methods.
"""
def __init__(self, **kwargs):
"""
Initialize a BaseAgent object.
Args:
**kwargs: Arbitrary keyword arguments.
"""
pass
def invoke(self, query: str, **kwargs) -> Any:
"""
Invoke the agent and return the result.
Args:
query: The query string.
**kwargs: Additional keyword arguments.
Returns:
The result of invoking the agent.
"""
class RAGAgent(BaseAgent):
"""
Abstract base class for Retrieval-Augmented Generation (RAG) agents.
This class extends BaseAgent with methods specific to RAG, including
retrieval and query methods.
"""
def __init__(self, **kwargs):
"""
Initialize a RAGAgent object.
Args:
**kwargs: Arbitrary keyword arguments.
"""
pass
def retrieve(self, query: str, **kwargs) -> Tuple[List[RetrievalResult], int, dict]:
"""
Retrieve document results from the knowledge base.
Args:
query: The query string.
**kwargs: Additional keyword arguments.
Returns:
A tuple containing:
- the retrieved results
- the total number of token usages of the LLM
- any additional metadata, which can be an empty dictionary
"""
def query(self, query: str, **kwargs) -> Tuple[str, List[RetrievalResult], int]:
"""
Query the agent and return the answer.
Args:
query: The query string.
**kwargs: Additional keyword arguments.
Returns:
A tuple containing:
- the result generated from LLM
- the retrieved document results
- the total number of token usages of the LLM
"""
================================================
FILE: deepsearcher/agent/chain_of_rag.py
================================================
from typing import List, Tuple
from deepsearcher.agent.base import RAGAgent, describe_class
from deepsearcher.agent.collection_router import CollectionRouter
from deepsearcher.embedding.base import BaseEmbedding
from deepsearcher.llm.base import BaseLLM
from deepsearcher.utils import log
from deepsearcher.vector_db import RetrievalResult
from deepsearcher.vector_db.base import BaseVectorDB, deduplicate_results
FOLLOWUP_QUERY_PROMPT = """You are using a search tool to answer the main query by iteratively searching the database. Given the following intermediate queries and answers, generate a new simple follow-up question that can help answer the main query. You may rephrase or decompose the main query when previous answers are not helpful. Ask simple follow-up questions only as the search tool may not understand complex questions.
## Previous intermediate queries and answers
{intermediate_context}
## Main query to answer
{query}
Respond with a simple follow-up question that will help answer the main query, do not explain yourself or output anything else.
"""
INTERMEDIATE_ANSWER_PROMPT = """Given the following documents, generate an appropriate answer for the query. DO NOT hallucinate any information, only use the provided documents to generate the answer. Respond "No relevant information found" if the documents do not contain useful information.
## Documents
{retrieved_documents}
## Query
{sub_query}
Respond with a concise answer only, do not explain yourself or output anything else.
"""
FINAL_ANSWER_PROMPT = """Given the following intermediate queries and answers, generate a final answer for the main query by combining relevant information. Note that intermediate answers are generated by an LLM and may not always be accurate.
## Documents
{retrieved_documents}
## Intermediate queries and answers
{intermediate_context}
## Main query
{query}
Respond with an appropriate answer only, do not explain yourself or output anything else.
"""
REFLECTION_PROMPT = """Given the following intermediate queries and answers, judge whether you have enough information to answer the main query. If you believe you have enough information, respond with "Yes", otherwise respond with "No".
## Intermediate queries and answers
{intermediate_context}
## Main query
{query}
Respond with "Yes" or "No" only, do not explain yourself or output anything else.
"""
GET_SUPPORTED_DOCS_PROMPT = """Given the following documents, select the ones that are support the Q-A pair.
## Documents
{retrieved_documents}
## Q-A Pair
### Question
{query}
### Answer
{answer}
Respond with a python list of indices of the selected documents.
"""
@describe_class(
"This agent can decompose complex queries and gradually find the fact information of sub-queries. "
"It is very suitable for handling concrete factual queries and multi-hop questions."
)
class ChainOfRAG(RAGAgent):
"""
Chain of Retrieval-Augmented Generation (RAG) agent implementation.
This agent implements a multi-step RAG process where each step can refine
the query and retrieval process based on previous results, creating a chain
of increasingly focused and relevant information retrieval and generation.
Inspired by: https://arxiv.org/pdf/2501.14342
"""
def __init__(
self,
llm: BaseLLM,
embedding_model: BaseEmbedding,
vector_db: BaseVectorDB,
max_iter: int = 4,
early_stopping: bool = False,
route_collection: bool = True,
text_window_splitter: bool = True,
**kwargs,
):
"""
Initialize the ChainOfRAG agent with configuration parameters.
Args:
llm (BaseLLM): The language model to use for generating answers.
embedding_model (BaseEmbedding): The embedding model to use for embedding queries.
vector_db (BaseVectorDB): The vector database to search for relevant documents.
max_iter (int, optional): The maximum number of iterations for the RAG process. Defaults to 4.
early_stopping (bool, optional): Whether to use early stopping. Defaults to False.
route_collection (bool, optional): Whether to route the query to specific collections. Defaults to True.
text_window_splitter (bool, optional): Whether use text_window splitter. Defaults to True.
"""
self.llm = llm
self.embedding_model = embedding_model
self.vector_db = vector_db
self.max_iter = max_iter
self.early_stopping = early_stopping
self.route_collection = route_collection
self.collection_router = CollectionRouter(
llm=self.llm, vector_db=self.vector_db, dim=embedding_model.dimension
)
self.text_window_splitter = text_window_splitter
def _reflect_get_subquery(self, query: str, intermediate_context: List[str]) -> Tuple[str, int]:
chat_response = self.llm.chat(
[
{
"role": "user",
"content": FOLLOWUP_QUERY_PROMPT.format(
query=query,
intermediate_context="\n".join(intermediate_context),
),
}
]
)
return self.llm.remove_think(chat_response.content), chat_response.total_tokens
def _retrieve_and_answer(self, query: str) -> Tuple[str, List[RetrievalResult], int]:
consume_tokens = 0
if self.route_collection:
selected_collections, n_token_route = self.collection_router.invoke(
query=query, dim=self.embedding_model.dimension
)
else:
selected_collections = self.collection_router.all_collections
n_token_route = 0
consume_tokens += n_token_route
all_retrieved_results = []
for collection in selected_collections:
log.color_print(f"<search> Search [{query}] in [{collection}]... </search>\n")
query_vector = self.embedding_model.embed_query(query)
retrieved_results = self.vector_db.search_data(
collection=collection, vector=query_vector, query_text=query
)
all_retrieved_results.extend(retrieved_results)
all_retrieved_results = deduplicate_results(all_retrieved_results)
chat_response = self.llm.chat(
[
{
"role": "user",
"content": INTERMEDIATE_ANSWER_PROMPT.format(
retrieved_documents=self._format_retrieved_results(all_retrieved_results),
sub_query=query,
),
}
]
)
return (
self.llm.remove_think(chat_response.content),
all_retrieved_results,
consume_tokens + chat_response.total_tokens,
)
def _get_supported_docs(
self,
retrieved_results: List[RetrievalResult],
query: str,
intermediate_answer: str,
) -> Tuple[List[RetrievalResult], int]:
supported_retrieved_results = []
token_usage = 0
if "No relevant information found" not in intermediate_answer:
chat_response = self.llm.chat(
[
{
"role": "user",
"content": GET_SUPPORTED_DOCS_PROMPT.format(
retrieved_documents=self._format_retrieved_results(retrieved_results),
query=query,
answer=intermediate_answer,
),
}
]
)
supported_doc_indices = self.llm.literal_eval(chat_response.content)
supported_retrieved_results = [
retrieved_results[int(i)]
for i in supported_doc_indices
if int(i) < len(retrieved_results)
]
token_usage = chat_response.total_tokens
return supported_retrieved_results, token_usage
def _check_has_enough_info(
self, query: str, intermediate_contexts: List[str]
) -> Tuple[bool, int]:
if not intermediate_contexts:
return False, 0
chat_response = self.llm.chat(
[
{
"role": "user",
"content": REFLECTION_PROMPT.format(
query=query,
intermediate_context="\n".join(intermediate_contexts),
),
}
]
)
has_enough_info = self.llm.remove_think(chat_response.content).strip().lower() == "yes"
return has_enough_info, chat_response.total_tokens
def retrieve(self, query: str, **kwargs) -> Tuple[List[RetrievalResult], int, dict]:
"""
Retrieves relevant documents based on the input query and iteratively refines the search.
This method iteratively refines the search query based on intermediate results, retrieves documents,
and filters out supported documents. It keeps track of the intermediate contexts and token usage.
Args:
query (str): The initial search query.
**kwargs: Additional keyword arguments.
- max_iter (int, optional): The maximum number of iterations for refinement. Defaults to self.max_iter.
Returns:
Tuple[List[RetrievalResult], int, dict]: A tuple containing:
- List[RetrievalResult]: The list of all retrieved and deduplicated results.
- int: The total token usage across all iterations.
- dict: A dictionary containing additional information, including the intermediate contexts.
"""
max_iter = kwargs.pop("max_iter", self.max_iter)
intermediate_contexts = []
all_retrieved_results = []
token_usage = 0
for iter in range(max_iter):
log.color_print(f">> Iteration: {iter + 1}\n")
followup_query, n_token0 = self._reflect_get_subquery(query, intermediate_contexts)
intermediate_answer, retrieved_results, n_token1 = self._retrieve_and_answer(
followup_query
)
supported_retrieved_results, n_token2 = self._get_supported_docs(
retrieved_results, followup_query, intermediate_answer
)
all_retrieved_results.extend(supported_retrieved_results)
intermediate_idx = len(intermediate_contexts) + 1
intermediate_contexts.append(
f"Intermediate query{intermediate_idx}: {followup_query}\nIntermediate answer{intermediate_idx}: {intermediate_answer}"
)
token_usage += n_token0 + n_token1 + n_token2
if self.early_stopping:
has_enough_info, n_token_check = self._check_has_enough_info(
query, intermediate_contexts
)
token_usage += n_token_check
if has_enough_info:
log.color_print(
f"<think> Early stopping after iteration {iter + 1}: Have enough information to answer the main query. </think>\n"
)
break
all_retrieved_results = deduplicate_results(all_retrieved_results)
additional_info = {"intermediate_context": intermediate_contexts}
return all_retrieved_results, token_usage, additional_info
def query(self, query: str, **kwargs) -> Tuple[str, List[RetrievalResult], int]:
"""
Executes a query and returns the final answer along with all retrieved results and total token usage.
This method initiates a query, retrieves relevant documents, and then summarizes the answer based on the retrieved documents and intermediate contexts. It logs the final answer and returns the answer content, all retrieved results, and the total token usage including the tokens used for the final answer.
Args:
query (str): The initial query to execute.
**kwargs: Additional keyword arguments to pass to the `retrieve` method.
Returns:
Tuple[str, List[RetrievalResult], int]: A tuple containing:
- str: The final answer content.
- List[RetrievalResult]: The list of all retrieved and deduplicated results.
- int: The total token usage across all iterations, including the final answer.
"""
all_retrieved_results, n_token_retrieval, additional_info = self.retrieve(query, **kwargs)
intermediate_context = additional_info["intermediate_context"]
log.color_print(
f"<think> Summarize answer from all {len(all_retrieved_results)} retrieved chunks... </think>\n"
)
chat_response = self.llm.chat(
[
{
"role": "user",
"content": FINAL_ANSWER_PROMPT.format(
retrieved_documents=self._format_retrieved_results(all_retrieved_results),
intermediate_context="\n".join(intermediate_context),
query=query,
),
}
]
)
log.color_print("\n==== FINAL ANSWER====\n")
log.color_print(self.llm.remove_think(chat_response.content))
return (
self.llm.remove_think(chat_response.content),
all_retrieved_results,
n_token_retrieval + chat_response.total_tokens,
)
def _format_retrieved_results(self, retrieved_results: List[RetrievalResult]) -> str:
formatted_documents = []
for i, result in enumerate(retrieved_results):
if self.text_window_splitter and "wider_text" in result.metadata:
text = result.metadata["wider_text"]
else:
text = result.text
formatted_documents.append(f"<Document {i}>\n{text}\n<\Document {i}>")
return "\n".join(formatted_documents)
================================================
FILE: deepsearcher/agent/collection_router.py
================================================
from typing import List, Tuple
from deepsearcher.agent.base import BaseAgent
from deepsearcher.llm.base import BaseLLM
from deepsearcher.utils import log
from deepsearcher.vector_db.base import BaseVectorDB
COLLECTION_ROUTE_PROMPT = """
I provide you with collection_name(s) and corresponding collection_description(s). Please select the collection names that may be related to the question and return a python list of str. If there is no collection related to the question, you can return an empty list.
"QUESTION": {question}
"COLLECTION_INFO": {collection_info}
When you return, you can ONLY return a python list of str, WITHOUT any other additional content. Your selected collection name list is:
"""
class CollectionRouter(BaseAgent):
"""
Routes queries to appropriate collections in the vector database.
This class analyzes the content of a query and determines which collections
in the vector database are most likely to contain relevant information.
"""
def __init__(self, llm: BaseLLM, vector_db: BaseVectorDB, dim: int, **kwargs):
"""
Initialize the CollectionRouter.
Args:
llm: The language model to use for analyzing queries.
vector_db: The vector database containing the collections.
dim: The dimension of the vector space to search in.
"""
self.llm = llm
self.vector_db = vector_db
self.all_collections = [
collection_info.collection_name
for collection_info in self.vector_db.list_collections(dim=dim)
]
def invoke(self, query: str, dim: int, **kwargs) -> Tuple[List[str], int]:
"""
Determine which collections are relevant for the given query.
This method analyzes the query content and selects collections that are
most likely to contain information relevant to answering the query.
Args:
query (str): The query to analyze.
dim (int): The dimension of the vector space to search in.
Returns:
Tuple[List[str], int]: A tuple containing:
- A list of selected collection names
- The token usage for the routing operation
"""
consume_tokens = 0
collection_infos = self.vector_db.list_collections(dim=dim)
if len(collection_infos) == 0:
log.warning(
"No collections found in the vector database. Please check the database connection."
)
return [], 0
if len(collection_infos) == 1:
the_only_collection = collection_infos[0].collection_name
log.color_print(
f"<think> Perform search [{query}] on the vector DB collection: {the_only_collection} </think>\n"
)
return [the_only_collection], 0
vector_db_search_prompt = COLLECTION_ROUTE_PROMPT.format(
question=query,
collection_info=[
{
"collection_name": collection_info.collection_name,
"collection_description": collection_info.description,
}
for collection_info in collection_infos
],
)
chat_response = self.llm.chat(
messages=[{"role": "user", "content": vector_db_search_prompt}]
)
selected_collections = self.llm.literal_eval(chat_response.content)
consume_tokens += chat_response.total_tokens
for collection_info in collection_infos:
# If a collection description is not provided, use the query as the search query
if not collection_info.description:
selected_collections.append(collection_info.collection_name)
# If the default collection exists, use the query as the search query
if self.vector_db.default_collection == collection_info.collection_name:
selected_collections.append(collection_info.collection_name)
selected_collections = list(set(selected_collections))
log.color_print(
f"<think> Perform search [{query}] on the vector DB collections: {selected_collections} </think>\n"
)
return selected_collections, consume_tokens
================================================
FILE: deepsearcher/agent/deep_search.py
================================================
import asyncio
from typing import List, Tuple
from deepsearcher.agent.base import RAGAgent, describe_class
from deepsearcher.agent.collection_router import CollectionRouter
from deepsearcher.embedding.base import BaseEmbedding
from deepsearcher.llm.base import BaseLLM
from deepsearcher.utils import log
from deepsearcher.vector_db import RetrievalResult
from deepsearcher.vector_db.base import BaseVectorDB, deduplicate_results
SUB_QUERY_PROMPT = """To answer this question more comprehensively, please break down the original question into up to four sub-questions. Return as list of str.
If this is a very simple question and no decomposition is necessary, then keep the only one original question in the python code list.
Original Question: {original_query}
<EXAMPLE>
Example input:
"Explain deep learning"
Example output:
[
"What is deep learning?",
"What is the difference between deep learning and machine learning?",
"What is the history of deep learning?"
]
</EXAMPLE>
Provide your response in a python code list of str format:
"""
RERANK_PROMPT = """Based on the query questions and the retrieved chunk, to determine whether the chunk is helpful in answering any of the query question, you can only return "YES" or "NO", without any other information.
Query Questions: {query}
Retrieved Chunk: {retrieved_chunk}
Is the chunk helpful in answering the any of the questions?
"""
REFLECT_PROMPT = """Determine whether additional search queries are needed based on the original query, previous sub queries, and all retrieved document chunks. If further research is required, provide a Python list of up to 3 search queries. If no further research is required, return an empty list.
If the original query is to write a report, then you prefer to generate some further queries, instead return an empty list.
Original Query: {question}
Previous Sub Queries: {mini_questions}
Related Chunks:
{mini_chunk_str}
Respond exclusively in valid List of str format without any other text."""
SUMMARY_PROMPT = """You are a AI content analysis expert, good at summarizing content. Please summarize a specific and detailed answer or report based on the previous queries and the retrieved document chunks.
Original Query: {question}
Previous Sub Queries: {mini_questions}
Related Chunks:
{mini_chunk_str}
"""
@describe_class(
"This agent is suitable for handling general and simple queries, such as given a topic and then writing a report, survey, or article."
)
class DeepSearch(RAGAgent):
"""
Deep Search agent implementation for comprehensive information retrieval.
This agent performs a thorough search through the knowledge base, analyzing
multiple aspects of the query to provide comprehensive and detailed answers.
"""
def __init__(
self,
llm: BaseLLM,
embedding_model: BaseEmbedding,
vector_db: BaseVectorDB,
max_iter: int = 3,
route_collection: bool = True,
text_window_splitter: bool = True,
**kwargs,
):
"""
Initialize the DeepSearch agent.
Args:
llm: The language model to use for generating answers.
embedding_model: The embedding model to use for query embedding.
vector_db: The vector database to search for relevant documents.
max_iter: The maximum number of iterations for the search process.
route_collection: Whether to use a collection router for search.
text_window_splitter: Whether to use text_window splitter.
**kwargs: Additional keyword arguments for customization.
"""
self.llm = llm
self.embedding_model = embedding_model
self.vector_db = vector_db
self.max_iter = max_iter
self.route_collection = route_collection
self.collection_router = CollectionRouter(
llm=self.llm, vector_db=self.vector_db, dim=embedding_model.dimension
)
self.text_window_splitter = text_window_splitter
def _generate_sub_queries(self, original_query: str) -> Tuple[List[str], int]:
chat_response = self.llm.chat(
messages=[
{"role": "user", "content": SUB_QUERY_PROMPT.format(original_query=original_query)}
]
)
response_content = self.llm.remove_think(chat_response.content)
return self.llm.literal_eval(response_content), chat_response.total_tokens
async def _search_chunks_from_vectordb(self, query: str, sub_queries: List[str]):
consume_tokens = 0
if self.route_collection:
selected_collections, n_token_route = self.collection_router.invoke(
query=query, dim=self.embedding_model.dimension
)
else:
selected_collections = self.collection_router.all_collections
n_token_route = 0
consume_tokens += n_token_route
all_retrieved_results = []
query_vector = self.embedding_model.embed_query(query)
for collection in selected_collections:
log.color_print(f"<search> Search [{query}] in [{collection}]... </search>\n")
retrieved_results = self.vector_db.search_data(
collection=collection, vector=query_vector, query_text=query
)
if not retrieved_results or len(retrieved_results) == 0:
log.color_print(
f"<search> No relevant document chunks found in '{collection}'! </search>\n"
)
continue
accepted_chunk_num = 0
references = set()
for retrieved_result in retrieved_results:
chat_response = self.llm.chat(
messages=[
{
"role": "user",
"content": RERANK_PROMPT.format(
query=[query] + sub_queries,
retrieved_chunk=f"<chunk>{retrieved_result.text}</chunk>",
),
}
]
)
consume_tokens += chat_response.total_tokens
response_content = self.llm.remove_think(chat_response.content).strip()
if "YES" in response_content and "NO" not in response_content:
all_retrieved_results.append(retrieved_result)
accepted_chunk_num += 1
references.add(retrieved_result.reference)
if accepted_chunk_num > 0:
log.color_print(
f"<search> Accept {accepted_chunk_num} document chunk(s) from references: {list(references)} </search>\n"
)
else:
log.color_print(
f"<search> No document chunk accepted from '{collection}'! </search>\n"
)
return all_retrieved_results, consume_tokens
def _generate_gap_queries(
self, original_query: str, all_sub_queries: List[str], all_chunks: List[RetrievalResult]
) -> Tuple[List[str], int]:
reflect_prompt = REFLECT_PROMPT.format(
question=original_query,
mini_questions=all_sub_queries,
mini_chunk_str=self._format_chunk_texts([chunk.text for chunk in all_chunks])
if len(all_chunks) > 0
else "NO RELATED CHUNKS FOUND.",
)
chat_response = self.llm.chat([{"role": "user", "content": reflect_prompt}])
response_content = self.llm.remove_think(chat_response.content)
return self.llm.literal_eval(response_content), chat_response.total_tokens
def retrieve(self, original_query: str, **kwargs) -> Tuple[List[RetrievalResult], int, dict]:
"""
Retrieve relevant documents from the knowledge base for the given query.
This method performs a deep search through the vector database to find
the most relevant documents for answering the query.
Args:
original_query (str): The query to search for.
**kwargs: Additional keyword arguments for customizing the retrieval.
Returns:
Tuple[List[RetrievalResult], int, dict]: A tuple containing:
- A list of retrieved document results
- The token usage for the retrieval operation
- Additional information about the retrieval process
"""
return asyncio.run(self.async_retrieve(original_query, **kwargs))
async def async_retrieve(
self, original_query: str, **kwargs
) -> Tuple[List[RetrievalResult], int, dict]:
max_iter = kwargs.pop("max_iter", self.max_iter)
### SUB QUERIES ###
log.color_print(f"<query> {original_query} </query>\n")
all_search_res = []
all_sub_queries = []
total_tokens = 0
sub_queries, used_token = self._generate_sub_queries(original_query)
total_tokens += used_token
if not sub_queries:
log.color_print("No sub queries were generated by the LLM. Exiting.")
return [], total_tokens, {}
else:
log.color_print(
f"<think> Break down the original query into new sub queries: {sub_queries}</think>\n"
)
all_sub_queries.extend(sub_queries)
sub_gap_queries = sub_queries
for iter in range(max_iter):
log.color_print(f">> Iteration: {iter + 1}\n")
search_res_from_vectordb = []
search_res_from_internet = [] # TODO
# Create all search tasks
search_tasks = [
self._search_chunks_from_vectordb(query, sub_gap_queries)
for query in sub_gap_queries
]
# Execute all tasks in parallel and wait for results
search_results = await asyncio.gather(*search_tasks)
# Merge all results
for result in search_results:
search_res, consumed_token = result
total_tokens += consumed_token
search_res_from_vectordb.extend(search_res)
search_res_from_vectordb = deduplicate_results(search_res_from_vectordb)
# search_res_from_internet = deduplicate_results(search_res_from_internet)
all_search_res.extend(search_res_from_vectordb + search_res_from_internet)
if iter == max_iter - 1:
log.color_print("<think> Exceeded maximum iterations. Exiting. </think>\n")
break
### REFLECTION & GET GAP QUERIES ###
log.color_print("<think> Reflecting on the search results... </think>\n")
sub_gap_queries, consumed_token = self._generate_gap_queries(
original_query, all_sub_queries, all_search_res
)
total_tokens += consumed_token
if not sub_gap_queries or len(sub_gap_queries) == 0:
log.color_print("<think> No new search queries were generated. Exiting. </think>\n")
break
else:
log.color_print(
f"<think> New search queries for next iteration: {sub_gap_queries} </think>\n"
)
all_sub_queries.extend(sub_gap_queries)
all_search_res = deduplicate_results(all_search_res)
additional_info = {"all_sub_queries": all_sub_queries}
return all_search_res, total_tokens, additional_info
def query(self, query: str, **kwargs) -> Tuple[str, List[RetrievalResult], int]:
"""
Query the agent and generate an answer based on retrieved documents.
This method retrieves relevant documents and uses the language model
to generate a comprehensive answer to the query.
Args:
query (str): The query to answer.
**kwargs: Additional keyword arguments for customizing the query process.
Returns:
Tuple[str, List[RetrievalResult], int]: A tuple containing:
- The generated answer
- A list of retrieved document results
- The total token usage
"""
all_retrieved_results, n_token_retrieval, additional_info = self.retrieve(query, **kwargs)
if not all_retrieved_results or len(all_retrieved_results) == 0:
return f"No relevant information found for query '{query}'.", [], n_token_retrieval
all_sub_queries = additional_info["all_sub_queries"]
chunk_texts = []
for chunk in all_retrieved_results:
if self.text_window_splitter and "wider_text" in chunk.metadata:
chunk_texts.append(chunk.metadata["wider_text"])
else:
chunk_texts.append(chunk.text)
log.color_print(
f"<think> Summarize answer from all {len(all_retrieved_results)} retrieved chunks... </think>\n"
)
summary_prompt = SUMMARY_PROMPT.format(
question=query,
mini_questions=all_sub_queries,
mini_chunk_str=self._format_chunk_texts(chunk_texts),
)
chat_response = self.llm.chat([{"role": "user", "content": summary_prompt}])
log.color_print("\n==== FINAL ANSWER====\n")
log.color_print(self.llm.remove_think(chat_response.content))
return (
self.llm.remove_think(chat_response.content),
all_retrieved_results,
n_token_retrieval + chat_response.total_tokens,
)
def _format_chunk_texts(self, chunk_texts: List[str]) -> str:
chunk_str = ""
for i, chunk in enumerate(chunk_texts):
chunk_str += f"""<chunk_{i}>\n{chunk}\n</chunk_{i}>\n"""
return chunk_str
================================================
FILE: deepsearcher/agent/naive_rag.py
================================================
from typing import List, Tuple
from deepsearcher.agent.base import RAGAgent
from deepsearcher.agent.collection_router import CollectionRouter
from deepsearcher.embedding.base import BaseEmbedding
from deepsearcher.llm.base import BaseLLM
from deepsearcher.utils import log
from deepsearcher.vector_db.base import BaseVectorDB, RetrievalResult, deduplicate_results
SUMMARY_PROMPT = """You are a AI content analysis expert, good at summarizing content. Please summarize a specific and detailed answer or report based on the previous queries and the retrieved document chunks.
Original Query: {query}
Related Chunks:
{mini_chunk_str}
"""
class NaiveRAG(RAGAgent):
"""
Naive Retrieval-Augmented Generation agent implementation.
This agent implements a straightforward RAG approach, retrieving relevant
documents and generating answers without complex processing or refinement steps.
"""
def __init__(
self,
llm: BaseLLM,
embedding_model: BaseEmbedding,
vector_db: BaseVectorDB,
top_k: int = 10,
route_collection: bool = True,
text_window_splitter: bool = True,
**kwargs,
):
"""
Initialize the NaiveRAG agent.
Args:
llm: The language model to use for generating answers.
embedding_model: The embedding model to use for query embedding.
vector_db: The vector database to search for relevant documents.
**kwargs: Additional keyword arguments for customization.
"""
self.llm = llm
self.embedding_model = embedding_model
self.vector_db = vector_db
self.top_k = top_k
self.route_collection = route_collection
if self.route_collection:
self.collection_router = CollectionRouter(
llm=self.llm, vector_db=self.vector_db, dim=embedding_model.dimension
)
self.text_window_splitter = text_window_splitter
def retrieve(self, query: str, **kwargs) -> Tuple[List[RetrievalResult], int, dict]:
"""
Retrieve relevant documents from the knowledge base for the given query.
This method performs a basic search through the vector database to find
documents relevant to the query.
Args:
query (str): The query to search for.
**kwargs: Additional keyword arguments for customizing the retrieval.
Returns:
Tuple[List[RetrievalResult], int, dict]: A tuple containing:
- A list of retrieved document results
- The token usage for the retrieval operation
- Additional information about the retrieval process
"""
consume_tokens = 0
if self.route_collection:
selected_collections, n_token_route = self.collection_router.invoke(
query=query, dim=self.embedding_model.dimension
)
else:
selected_collections = self.collection_router.all_collections
n_token_route = 0
consume_tokens += n_token_route
all_retrieved_results = []
for collection in selected_collections:
retrieval_res = self.vector_db.search_data(
collection=collection,
vector=self.embedding_model.embed_query(query),
top_k=max(self.top_k // len(selected_collections), 1),
query_text=query,
)
all_retrieved_results.extend(retrieval_res)
all_retrieved_results = deduplicate_results(all_retrieved_results)
return all_retrieved_results, consume_tokens, {}
def query(self, query: str, **kwargs) -> Tuple[str, List[RetrievalResult], int]:
"""
Query the agent and generate an answer based on retrieved documents.
This method retrieves relevant documents and uses the language model
to generate a simple answer to the query.
Args:
query (str): The query to answer.
**kwargs: Additional keyword arguments for customizing the query process.
Returns:
Tuple[str, List[RetrievalResult], int]: A tuple containing:
- The generated answer
- A list of retrieved document results
- The total token usage
"""
all_retrieved_results, n_token_retrieval, _ = self.retrieve(query)
chunk_texts = []
for chunk in all_retrieved_results:
if self.text_window_splitter and "wider_text" in chunk.metadata:
chunk_texts.append(chunk.metadata["wider_text"])
else:
chunk_texts.append(chunk.text)
mini_chunk_str = ""
for i, chunk in enumerate(chunk_texts):
mini_chunk_str += f"""<chunk_{i}>\n{chunk}\n</chunk_{i}>\n"""
summary_prompt = SUMMARY_PROMPT.format(query=query, mini_chunk_str=mini_chunk_str)
char_response = self.llm.chat([{"role": "user", "content": summary_prompt}])
final_answer = char_response.content
log.color_print("\n==== FINAL ANSWER====\n")
log.color_print(final_answer)
return final_answer, all_retrieved_results, n_token_retrieval + char_response.total_tokens
================================================
FILE: deepsearcher/agent/rag_router.py
================================================
from typing import List, Optional, Tuple
from deepsearcher.agent import RAGAgent
from deepsearcher.llm.base import BaseLLM
from deepsearcher.utils import log
from deepsearcher.vector_db import RetrievalResult
RAG_ROUTER_PROMPT = """Given a list of agent indexes and corresponding descriptions, each agent has a specific function.
Given a query, select only one agent that best matches the agent handling the query, and return the index without any other information.
## Question
{query}
## Agent Indexes and Descriptions
{description_str}
Only return one agent index number that best matches the agent handling the query:
"""
class RAGRouter(RAGAgent):
"""
Routes queries to the most appropriate RAG agent implementation.
This class analyzes the content and requirements of a query and determines
which RAG agent implementation is best suited to handle it.
"""
def __init__(
self,
llm: BaseLLM,
rag_agents: List[RAGAgent],
agent_descriptions: Optional[List[str]] = None,
):
"""
Initialize the RAGRouter.
Args:
llm: The language model to use for analyzing queries.
rag_agents: A list of RAGAgent instances.
agent_descriptions (list, optional): A list of descriptions for each agent.
"""
self.llm = llm
self.rag_agents = rag_agents
self.agent_descriptions = agent_descriptions
if not self.agent_descriptions:
try:
self.agent_descriptions = [
agent.__class__.__description__ for agent in self.rag_agents
]
except Exception:
raise AttributeError(
"Please provide agent descriptions or set __description__ attribute for each agent class."
)
def _route(self, query: str) -> Tuple[RAGAgent, int]:
description_str = "\n".join(
[f"[{i + 1}]: {description}" for i, description in enumerate(self.agent_descriptions)]
)
prompt = RAG_ROUTER_PROMPT.format(query=query, description_str=description_str)
chat_response = self.llm.chat(messages=[{"role": "user", "content": prompt}])
try:
selected_agent_index = int(self.llm.remove_think(chat_response.content)) - 1
except ValueError:
# In some reasoning LLM, the output is not a number, but a explaination string with a number in the end.
log.warning(
"Parse int failed in RAGRouter, but will try to find the last digit as fallback."
)
selected_agent_index = (
int(self.find_last_digit(self.llm.remove_think(chat_response.content))) - 1
)
selected_agent = self.rag_agents[selected_agent_index]
log.color_print(
f"<think> Select agent [{selected_agent.__class__.__name__}] to answer the query [{query}] </think>\n"
)
return self.rag_agents[selected_agent_index], chat_response.total_tokens
def retrieve(self, query: str, **kwargs) -> Tuple[List[RetrievalResult], int, dict]:
agent, n_token_router = self._route(query)
retrieved_results, n_token_retrieval, metadata = agent.retrieve(query, **kwargs)
return retrieved_results, n_token_router + n_token_retrieval, metadata
def query(self, query: str, **kwargs) -> Tuple[str, List[RetrievalResult], int]:
agent, n_token_router = self._route(query)
answer, retrieved_results, n_token_retrieval = agent.query(query, **kwargs)
return answer, retrieved_results, n_token_router + n_token_retrieval
def find_last_digit(self, string):
for char in reversed(string):
if char.isdigit():
return char
raise ValueError("No digit found in the string")
================================================
FILE: deepsearcher/cli.py
================================================
import argparse
import logging
import sys
import warnings
from deepsearcher.configuration import Configuration, init_config
from deepsearcher.offline_loading import load_from_local_files, load_from_website
from deepsearcher.online_query import query
from deepsearcher.utils import log
httpx_logger = logging.getLogger("httpx") # disable openai's logger output
httpx_logger.setLevel(logging.WARNING)
warnings.simplefilter(action="ignore", category=FutureWarning) # disable warning output
def main():
"""
Main entry point for the DeepSearcher CLI.
This function parses command line arguments and executes the appropriate action
based on the subcommand provided (query or load). It handles the deprecated
command line format and provides helpful error messages.
Returns:
None
"""
if "--query" in sys.argv or "--load" in sys.argv:
print("\033[91m[Deprecated]\033[0m The use of '--query' and '--load' is deprecated.")
print("Please use:")
print(" deepsearcher query <your_query> --max_iter 3")
print(
" deepsearcher load <your_local_path_or_url> --collection_name <your_collection_name> --collection_desc <your_collection_description>"
)
sys.exit(1)
config = Configuration() # Customize your config here
init_config(config=config)
parser = argparse.ArgumentParser(prog="deepsearcher", description="Deep Searcher.")
subparsers = parser.add_subparsers(dest="subcommand", title="subcommands")
## Arguments of query
query_parser = subparsers.add_parser("query", help="Query a question or search topic.")
query_parser.add_argument("query", type=str, default="", help="query question or search topic.")
query_parser.add_argument(
"--max_iter",
type=int,
default=3,
help="Max iterations of reflection. Default is 3.",
)
## Arguments of loading
load_parser = subparsers.add_parser(
"load", help="Load knowledge from local files or from URLs."
)
load_parser.add_argument(
"load_path",
type=str,
nargs="+", # 1 or more files or urls
help="Load knowledge from local files or from URLs.",
)
load_parser.add_argument(
"--batch_size",
type=int,
default=256,
help="Batch size for loading knowledge.",
)
load_parser.add_argument(
"--collection_name",
type=str,
default=None,
help="Destination collection name of loaded knowledge.",
)
load_parser.add_argument(
"--collection_desc",
type=str,
default=None,
help="Description of the collection.",
)
load_parser.add_argument(
"--force_new_collection",
type=bool,
default=False,
help="If you want to drop origin collection and create a new collection on every load, set to True",
)
args = parser.parse_args()
if args.subcommand == "query":
final_answer, refs, consumed_tokens = query(args.query, max_iter=args.max_iter)
log.color_print("\n==== FINAL ANSWER====\n")
log.color_print(final_answer)
log.color_print("\n### References\n")
for i, ref in enumerate(refs):
log.color_print(f"{i + 1}. {ref.text[:60]}… {ref.reference}")
elif args.subcommand == "load":
urls = [url for url in args.load_path if url.startswith("http")]
local_files = [file for file in args.load_path if not file.startswith("http")]
kwargs = {}
if args.collection_name:
kwargs["collection_name"] = args.collection_name
if args.collection_desc:
kwargs["collection_description"] = args.collection_desc
if args.force_new_collection:
kwargs["force_new_collection"] = args.force_new_collection
if args.batch_size:
kwargs["batch_size"] = args.batch_size
if len(urls) > 0:
load_from_website(urls, **kwargs)
if len(local_files) > 0:
load_from_local_files(local_files, **kwargs)
else:
print("Please provide a query or a load argument.")
if __name__ == "__main__":
main()
================================================
FILE: deepsearcher/config.yaml
================================================
provide_settings:
llm:
provider: "OpenAI"
config:
model: "o1-mini"
# api_key: "sk-xxxx" # Uncomment to override the `OPENAI_API_KEY` set in the environment variable
# base_url: ""
# provider: "DeepSeek"
# config:
# model: "deepseek-reasoner"
## api_key: "sk-xxxx" # Uncomment to override the `DEEPSEEK_API_KEY` set in the environment variable
## base_url: ""
# provider: "SiliconFlow"
# config:
# model: "deepseek-ai/DeepSeek-R1"
## api_key: "xxxx" # Uncomment to override the `SILICONFLOW_API_KEY` set in the environment variable
## base_url: ""
# provider: "PPIO"
# config:
# model: "deepseek/deepseek-r1-turbo"
## api_key: "sk_xxxxxx" # Uncomment to override the `PPIO_API_KEY` set in the environment variable
## base_url: ""
# provider: "Jiekou.AI"
# config:
# model: "claude-sonnet-4-5-20250929"
## api_key: "xxxx" # Uncomment to override the `JIEKOU_API_KEY` set in the environment variable
## base_url: ""
# provider: "TogetherAI"
# config:
# model: "deepseek-ai/DeepSeek-R1"
## api_key: "xxxx" # Uncomment to override the `TOGETHER_API_KEY` set in the environment variable
# provider: "AzureOpenAI"
# config:
# model: ""
# api_version: ""
## azure_endpoint: "xxxx" # Uncomment to override the `AZURE_OPENAI_ENDPOINT` set in the environment variable
## api_key: "xxxx" # Uncomment to override the `AZURE_OPENAI_KEY` set in the environment variable
# provider: "Ollama"
# config:
# model: "qwq"
## base_url: ""
# provider: "Novita"
# config:
# model: "deepseek/deepseek-v3-0324"
## api_key: "sk_xxxxxx" # Uncomment to override the `NOVITA_API_KEY` set in the environment variable
embedding:
provider: "OpenAIEmbedding"
config:
model: "text-embedding-ada-002"
# api_key: "" # Uncomment to override the `OPENAI_API_KEY` set in the environment variable
# base_url: "" # Uncomment to override the `OPENAI_BASE_URL` set in the environment variable
# dimension: 1536 # Uncomment to customize the embedding dimension
# provider: "MilvusEmbedding"
# config:
# model: "default"
# provider: "VoyageEmbedding"
# config:
# model: "voyage-3"
## api_key: "" # Uncomment to override the `VOYAGE_API_KEY` set in the environment variable
# provider: "BedrockEmbedding"
# config:
# model: "amazon.titan-embed-text-v2:0"
## aws_access_key_id: "" # Uncomment to override the `AWS_ACCESS_KEY_ID` set in the environment variable
## aws_secret_access_key: "" # Uncomment to override the `AWS_SECRET_ACCESS_KEY` set in the environment variable
# provider: "SiliconflowEmbedding"
# config:
# model: "BAAI/bge-m3"
# . api_key: "" # Uncomment to override the `SILICONFLOW_API_KEY` set in the environment variable
# provider: "GeminiEmbedding"
# config:
# model: "text-embedding-004"
## api_key: "" # Uncomment to override the `GEMINI_API_KEY` set in the environment variable
## dimension: 768 # Uncomment to customize the embedding dimension
# provider: "OllamaEmbedding"
# config:
# model: "bge-m3"
## dimension: 1024 # Uncomment to customize the embedding dimension
# provider: "FastEmbedEmbedding"
# config:
# model: "BAAI/bge-small-en-v1.5"
# provider: "NovitaEmbedding"
# config:
# model: "baai/bge-m3"
## api_key: "sk_xxxxxx" # Uncomment to override the `NOVITA_API_KEY` set in the environment variable
# provider: "SentenceTransformerEmbedding"
# config:
# model: "BAAI/bge-large-zh-v1.5"
file_loader:
provider: "PDFLoader"
config: {}
# provider: "JsonFileLoader"
# config:
# text_key: ""
# provider: "TextLoader"
# config: {}
# provider: "UnstructuredLoader"
# config: {}
# provider: "DoclingLoader"
# config: {}
web_crawler:
provider: "FireCrawlCrawler"
config: {}
# provider: "Crawl4AICrawler"
# config: # Uncomment to custom browser configuration for Crawl4AI
# browser_config:
# headless: false
# proxy: "http://127.0.0.1:7890"
# chrome_channel: "chrome"
# verbose: true
# viewport_width: 800
# viewport_height: 600
# provider: "JinaCrawler"
# config: {}
# provider: "DoclingCrawler"
# config: {}
vector_db:
provider: "Milvus"
config:
default_collection: "deepsearcher"
uri: "./milvus.db"
token: "root:Milvus"
db: "default"
# vector_db:
# provider: "OracleDB"
# config:
# default_collection: "deepsearcher"
# user: ""
# password: ""
# dsn: ""
# config_dir: ""
# wallet_location: ""
# wallet_password: ""
# vector_db:
# provider: "Qdrant"
# config:
# default_collection: "deepsearcher"
# host: "localhost"
# port: 6333
query_settings:
max_iter: 3
load_settings:
chunk_size: 1500
chunk_overlap: 100
================================================
FILE: deepsearcher/configuration.py
================================================
import os
from typing import Literal
import yaml
from deepsearcher.agent import ChainOfRAG, DeepSearch, NaiveRAG
from deepsearcher.agent.rag_router import RAGRouter
from deepsearcher.embedding.base import BaseEmbedding
from deepsearcher.llm.base import BaseLLM
from deepsearcher.loader.file_loader.base import BaseLoader
from deepsearcher.loader.web_crawler.base import BaseCrawler
from deepsearcher.vector_db.base import BaseVectorDB
current_dir = os.path.dirname(os.path.abspath(__file__))
DEFAULT_CONFIG_YAML_PATH = os.path.join(current_dir, "config.yaml")
FeatureType = Literal["llm", "embedding", "file_loader", "web_crawler", "vector_db"]
class Configuration:
"""
Configuration class for DeepSearcher.
This class manages the configuration settings for various components of the DeepSearcher system,
including LLM providers, embedding models, file loaders, web crawlers, and vector databases.
It loads configurations from a YAML file and provides methods to get and set provider configurations.
"""
def __init__(self, config_path: str = DEFAULT_CONFIG_YAML_PATH):
"""
Initialize the Configuration object.
Args:
config_path: Path to the configuration YAML file. Defaults to the config.yaml in the project root.
"""
# Initialize default configurations
config_data = self.load_config_from_yaml(config_path)
self.provide_settings = config_data["provide_settings"]
self.query_settings = config_data["query_settings"]
self.load_settings = config_data["load_settings"]
def load_config_from_yaml(self, config_path: str):
"""
Load configuration from a YAML file.
Args:
config_path: Path to the configuration YAML file.
Returns:
The loaded configuration data as a dictionary.
"""
with open(config_path, "r") as file:
return yaml.safe_load(file)
def set_provider_config(self, feature: FeatureType, provider: str, provider_configs: dict):
"""
Set the provider and its configurations for a given feature.
Args:
feature: The feature to configure (e.g., 'llm', 'file_loader', 'web_crawler').
provider: The provider name (e.g., 'openai', 'deepseek').
provider_configs: A dictionary with configurations specific to the provider.
Raises:
ValueError: If the feature is not supported.
"""
if feature not in self.provide_settings:
raise ValueError(f"Unsupported feature: {feature}")
self.provide_settings[feature]["provider"] = provider
self.provide_settings[feature]["config"] = provider_configs
def get_provider_config(self, feature: FeatureType):
"""
Get the current provider and configuration for a given feature.
Args:
feature: The feature to retrieve (e.g., 'llm', 'file_loader', 'web_crawler').
Returns:
A dictionary with provider and its configurations.
Raises:
ValueError: If the feature is not supported.
"""
if feature not in self.provide_settings:
raise ValueError(f"Unsupported feature: {feature}")
return self.provide_settings[feature]
class ModuleFactory:
"""
Factory class for creating instances of various modules in the DeepSearcher system.
This class creates instances of LLMs, embedding models, file loaders, web crawlers,
and vector databases based on the configuration settings.
"""
def __init__(self, config: Configuration):
"""
Initialize the ModuleFactory.
Args:
config: The Configuration object containing provider settings.
"""
self.config = config
def _create_module_instance(self, feature: FeatureType, module_name: str):
"""
Create an instance of a module based on the feature and module name.
Args:
feature: The feature type (e.g., 'llm', 'embedding').
module_name: The module name to import from.
Returns:
An instance of the specified module.
"""
# e.g.
# feature = "file_loader"
# module_name = "deepsearcher.loader.file_loader"
class_name = self.config.provide_settings[feature]["provider"]
module = __import__(module_name, fromlist=[class_name])
class_ = getattr(module, class_name)
return class_(**self.config.provide_settings[feature]["config"])
def create_llm(self) -> BaseLLM:
"""
Create an instance of a language model.
Returns:
An instance of a BaseLLM implementation.
"""
return self._create_module_instance("llm", "deepsearcher.llm")
def create_embedding(self) -> BaseEmbedding:
"""
Create an instance of an embedding model.
Returns:
An instance of a BaseEmbedding implementation.
"""
return self._create_module_instance("embedding", "deepsearcher.embedding")
def create_file_loader(self) -> BaseLoader:
"""
Create an instance of a file loader.
Returns:
An instance of a BaseLoader implementation.
"""
return self._create_module_instance("file_loader", "deepsearcher.loader.file_loader")
def create_web_crawler(self) -> BaseCrawler:
"""
Create an instance of a web crawler.
Returns:
An instance of a BaseCrawler implementation.
"""
return self._create_module_instance("web_crawler", "deepsearcher.loader.web_crawler")
def create_vector_db(self) -> BaseVectorDB:
"""
Create an instance of a vector database.
Returns:
An instance of a BaseVectorDB implementation.
"""
return self._create_module_instance("vector_db", "deepsearcher.vector_db")
config = Configuration()
module_factory: ModuleFactory = None
llm: BaseLLM = None
embedding_model: BaseEmbedding = None
file_loader: BaseLoader = None
vector_db: BaseVectorDB = None
web_crawler: BaseCrawler = None
default_searcher: RAGRouter = None
naive_rag: NaiveRAG = None
def init_config(config: Configuration):
"""
Initialize the global configuration and create instances of all required modules.
This function initializes the global variables for the LLM, embedding model,
file loader, web crawler, vector database, and RAG agents.
Args:
config: The Configuration object to use for initialization.
"""
global \
module_factory, \
llm, \
embedding_model, \
file_loader, \
vector_db, \
web_crawler, \
default_searcher, \
naive_rag
module_factory = ModuleFactory(config)
llm = module_factory.create_llm()
embedding_model = module_factory.create_embedding()
file_loader = module_factory.create_file_loader()
web_crawler = module_factory.create_web_crawler()
vector_db = module_factory.create_vector_db()
default_searcher = RAGRouter(
llm=llm,
rag_agents=[
DeepSearch(
llm=llm,
embedding_model=embedding_model,
vector_db=vector_db,
max_iter=config.query_settings["max_iter"],
route_collection=True,
text_window_splitter=True,
),
ChainOfRAG(
llm=llm,
embedding_model=embedding_model,
vector_db=vector_db,
max_iter=config.query_settings["max_iter"],
route_collection=True,
text_window_splitter=True,
),
],
)
naive_rag = NaiveRAG(
llm=llm,
embedding_model=embedding_model,
vector_db=vector_db,
top_k=10,
route_collection=True,
text_window_splitter=True,
)
================================================
FILE: deepsearcher/embedding/__init__.py
================================================
from .bedrock_embedding import BedrockEmbedding
from .fastembed_embdding import FastEmbedEmbedding
from .gemini_embedding import GeminiEmbedding
from .glm_embedding import GLMEmbedding
from .jiekouai_embedding import JiekouAIEmbedding
from .milvus_embedding import MilvusEmbedding
from .novita_embedding import NovitaEmbedding
from .ollama_embedding import OllamaEmbedding
from .openai_embedding import OpenAIEmbedding
from .ppio_embedding import PPIOEmbedding
from .sentence_transformer_embedding import SentenceTransformerEmbedding
from .siliconflow_embedding import SiliconflowEmbedding
from .volcengine_embedding import VolcengineEmbedding
from .voyage_embedding import VoyageEmbedding
from .watsonx_embedding import WatsonXEmbedding
__all__ = [
"MilvusEmbedding",
"OpenAIEmbedding",
"VoyageEmbedding",
"BedrockEmbedding",
"SiliconflowEmbedding",
"GeminiEmbedding",
"PPIOEmbedding",
"VolcengineEmbedding",
"GLMEmbedding",
"OllamaEmbedding",
"FastEmbedEmbedding",
"NovitaEmbedding",
"SentenceTransformerEmbedding",
"WatsonXEmbedding",
"JiekouAIEmbedding",
]
================================================
FILE: deepsearcher/embedding/base.py
================================================
from typing import List
from tqdm import tqdm
from deepsearcher.loader.splitter import Chunk
class BaseEmbedding:
"""
Abstract base class for embedding model implementations.
This class defines the interface for embedding model implementations,
including methods for embedding queries and documents, and a property
for the dimensionality of the embeddings.
"""
def embed_query(self, text: str) -> List[float]:
"""
Embed a single query text.
Args:
text: The query text to embed.
Returns:
A list of floats representing the embedding vector.
"""
pass
def embed_documents(self, texts: List[str]) -> List[List[float]]:
"""
Embed a list of document texts.
This default implementation calls embed_query for each text,
but implementations may override this with a more efficient batch method.
Args:
texts: A list of document texts to embed.
Returns:
A list of embedding vectors, one for each input text.
"""
return [self.embed_query(text) for text in texts]
def embed_chunks(self, chunks: List[Chunk], batch_size: int = 256) -> List[Chunk]:
"""
Embed a list of Chunk objects.
This method extracts the text from each chunk, embeds it in batches,
and updates the chunks with their embeddings.
Args:
chunks: A list of Chunk objects to embed.
batch_size: The number of chunks to process in each batch.
Returns:
The input list of Chunk objects, updated with embeddings.
"""
texts = [chunk.text for chunk in chunks]
batch_texts = [texts[i : i + batch_size] for i in range(0, len(texts), batch_size)]
embeddings = []
for batch_text in tqdm(batch_texts, desc="Embedding chunks"):
batch_embeddings = self.embed_documents(batch_text)
embeddings.extend(batch_embeddings)
for chunk, embedding in zip(chunks, embeddings):
chunk.embedding = embedding
return chunks
@property
def dimension(self) -> int:
"""
Get the dimensionality of the embeddings.
Returns:
The number of dimensions in the embedding vectors.
"""
pass
================================================
FILE: deepsearcher/embedding/bedrock_embedding.py
================================================
import json
import os
from typing import List
from deepsearcher.embedding.base import BaseEmbedding
MODEL_ID_TITAN_TEXT_G1 = "amazon.titan-embed-text-v1"
MODEL_ID_TITAN_TEXT_V2 = "amazon.titan-embed-text-v2:0"
MODEL_ID_TITAN_MULTIMODAL_G1 = "amazon.titan-embed-image-v1"
MODEL_ID_COHERE_ENGLISH_V3 = "cohere.embed-english-v3"
MODEL_ID_COHERE_MULTILINGUAL_V3 = "cohere.embed-multilingual-v3"
BEDROCK_MODEL_DIM_MAP = {
MODEL_ID_TITAN_TEXT_G1: 1536,
MODEL_ID_TITAN_TEXT_V2: 1024,
MODEL_ID_TITAN_MULTIMODAL_G1: 1024,
MODEL_ID_COHERE_ENGLISH_V3: 1024,
MODEL_ID_COHERE_MULTILINGUAL_V3: 1024,
}
DEFAULT_MODEL_ID = MODEL_ID_TITAN_TEXT_V2
class BedrockEmbedding(BaseEmbedding):
"""
Amazon Bedrock embedding model implementation.
This class provides an interface to the Amazon Bedrock embedding API, which offers
various embedding models for text processing, including Amazon Titan and Cohere models.
"""
def __init__(self, model: str = DEFAULT_MODEL_ID, region_name: str = "us-east-1", **kwargs):
"""
Initialize the Amazon Bedrock embedding model.
Args:
model (str): The model identifier to use for embeddings.
Default is "amazon.titan-embed-text-v2:0".
**kwargs: Additional keyword arguments.
- aws_access_key_id (str, optional): AWS access key ID. If not provided,
it will be read from the AWS_ACCESS_KEY_ID environment variable.
- aws_secret_access_key (str, optional): AWS secret access key. If not provided,
it will be read from the AWS_SECRET_ACCESS_KEY environment variable.
- model_name (str, optional): Alternative way to specify the model.
Notes:
Available models:
- 'amazon.titan-embed-text-v2:0': dimensions include 256, 512, 1024, default is 1024
- 'amazon.titan-embed-text-v1': 1536 dimensions
- 'amazon.titan-embed-image-v1': 1024 dimensions
- 'cohere.embed-english-v3': 1024 dimensions
- 'cohere.embed-multilingual-v3': 1024 dimensions
"""
import boto3
aws_access_key_id = kwargs.pop("aws_access_key_id", os.getenv("AWS_ACCESS_KEY_ID"))
aws_secret_access_key = kwargs.pop(
"aws_secret_access_key", os.getenv("AWS_SECRET_ACCESS_KEY")
)
if model in {None, DEFAULT_MODEL_ID} and "model_name" in kwargs:
model = kwargs.pop("model_name") # overwrites `model` with `model_name`
self.model = model
# TODO: initiate boto3 client
self.client = boto3.client(
"bedrock-runtime",
region_name=region_name,
aws_access_key_id=aws_access_key_id,
aws_secret_access_key=aws_secret_access_key,
)
def embed_query(self, text: str) -> List[float]:
"""
Embed a single query text using Amazon Bedrock.
Args:
text (str): The query text to embed.
Returns:
List[float]: A list of floats representing the embedding vector.
"""
response = self.client.invoke_model(
modelId=self.model, body=json.dumps({"inputText": text})
)
model_response = json.loads(response["body"].read())
embedding = model_response["embedding"]
return embedding
def embed_documents(self, texts: List[str]) -> List[List[float]]:
"""
Embed a list of document texts.
This implementation calls embed_query for each text individually.
Args:
texts (List[str]): A list of document texts to embed.
Returns:
List[List[float]]: A list of embedding vectors, one for each input text.
"""
return [self.embed_query(text) for text in texts]
@property
def dimension(self) -> int:
"""
Get the dimensionality of the embeddings for the current model.
Returns:
int: The number of dimensions in the embedding vectors.
"""
return BEDROCK_MODEL_DIM_MAP[self.model]
================================================
FILE: deepsearcher/embedding/fastembed_embdding.py
================================================
from functools import cached_property
from typing import List
from deepsearcher.embedding.base import BaseEmbedding
class FastEmbedEmbedding(BaseEmbedding):
"""
[FastEmbed](https://github.com/qdrant/fastembed) based vector embeddings.
"""
def __init__(self, model="BAAI/bge-small-en-v1.5", **kwargs):
"""
Initialize the Fastembed embedding model.
Args:
model (str): The name of the model to use.
Defaults is "BAAI/bge-small-en-v1.5".
**kwargs: Additional keyword arguments for TextEmbedding.
"""
try:
import fastembed # noqa: F401
except ImportError as original_error:
raise ImportError(
"Fastembed is not installed. Install it using: pip install fastembed\n"
) from original_error
self.model = model
self._kwargs = kwargs
self._embedding_model = None
def _ensure_model_loaded(self):
"""
Lazily load the embedding model when first needed.
"""
from fastembed import TextEmbedding
if self._embedding_model is None:
self._embedding_model = TextEmbedding(model_name=self.model, **self._kwargs)
def embed_query(self, text: str) -> List[float]:
"""
Embed a single query text.
Args:
text (str): The query text to embed.
Returns:
List[float]: A list of floats representing the embedding vector.
"""
self._ensure_model_loaded()
embeddings = next(self._embedding_model.query_embed([text]))
return embeddings.tolist()
def embed_documents(self, texts: List[str]) -> List[List[float]]:
"""
Embed a list of document texts.
Args:
texts (List[str]): A list of document texts to embed.
Returns:
List[List[float]]: A list of embedding vectors, one for each input text.
"""
self._ensure_model_loaded()
embeddings = [embedding.tolist() for embedding in self._embedding_model.embed(texts)]
return embeddings
@cached_property
def dimension(self) -> int:
"""
Get the dimensionality of the embeddings.
Returns:
int: The number of dimensions in the embedding vectors.
"""
self._ensure_model_loaded()
sample_embedding = self.embed_query("SAMPLE TEXT")
return len(sample_embedding)
================================================
FILE: deepsearcher/embedding/gemini_embedding.py
================================================
import os
from typing import List
from deepsearcher.embedding.base import BaseEmbedding
GEMINI_MODEL_DIM_MAP = {
"text-embedding-004": 768,
"gemini-embedding-exp-03-07": 3072,
}
class GeminiEmbedding(BaseEmbedding):
"""
Gemini AI embedding model implementation.
This class provides an interface to the Gemini AI embedding API, which offers
various embedding models for text processing.
For more information, see:
https://ai.google.dev/api/embeddings
"""
def __init__(self, model: str = "text-embedding-004", **kwargs):
"""
Initialize the Gemini embedding model.
Args:
model (str): The model identifier to use for embeddings. Default is "text-embedding-004".
**kwargs: Additional keyword arguments.
- api_key (str, optional): The Gemini API key. If not provided,
it will be read from the GEMINI_API_KEY environment variable.
- dimension (int, optional): The dimension of the embedding vectors.
If not provided, the default dimension for the model will be used.
"""
from google import genai
if "api_key" in kwargs:
api_key = kwargs.pop("api_key")
else:
api_key = os.getenv("GEMINI_API_KEY")
if "dimension" in kwargs:
dimension = kwargs.pop("dimension")
else:
dimension = GEMINI_MODEL_DIM_MAP[model]
self.dim = dimension
self.model = model
self.client = genai.Client(api_key=api_key, **kwargs)
def embed_chunks(self, chunks, batch_size: int = 100):
# For Gemini free level, the maximum rqeusts in one batch is 100, so set the default batch size to 100
return super().embed_chunks(chunks, batch_size)
def _get_dim(self):
"""
Get the dimension of the embedding model.
Returns:
int: The dimension of the embedding model.
"""
return self.dim
def _embed_content(self, texts: List[str]):
"""
Embed a list of content texts.
Args:
texts (List[str]): A list of texts to embed.
Returns:
List: A list of embeddings corresponding to the input texts.
"""
from google.genai import types
response = self.client.models.embed_content(
model=self.model,
contents=texts,
config=types.EmbedContentConfig(output_dimensionality=self._get_dim()),
)
return response.embeddings
def embed_query(self, text: str) -> List[float]:
"""
Embed a single query text.
Args:
text (str): The query text to embed.
Returns:
List[float]: A list of floats representing the embedding vector.
"""
# make sure the text is one string
if len(text) != 1:
text = " ".join(text)
result = self._embed_content(text)
# embedding = [r.values for r in result]
embedding = result[0].values
return embedding
def embed_documents(self, texts: List[str]) -> List[List[float]]:
"""
Embed a list of document texts.
Args:
texts (List[str]): A list of document texts to embed.
Returns:
List[List[float]]: A list of embedding vectors, one for each input text.
"""
result = self._embed_content(texts)
embeddings = [r.values for r in result]
return embeddings
@property
def dimension(self) -> int:
"""
Get the dimensionality of the embeddings for the current model.
Returns:
int: The number of dimensions in the embedding vectors.
"""
return self.dim
================================================
FILE: deepsearcher/embedding/glm_embedding.py
================================================
import os
from typing import List
from deepsearcher.embedding.base import BaseEmbedding
GLM_MODEL_DIM_MAP = {
"embedding-3": 2048,
}
class GLMEmbedding(BaseEmbedding):
"""
https://platform.openai.com/docs/guides/embeddings/use-cases
"""
def __init__(self, model: str = "embedding-3", **kwargs):
"""
Args:
model_name (`str`):
"""
from zhipuai import ZhipuAI
if "api_key" in kwargs:
api_key = kwargs.pop("api_key")
else:
api_key = os.getenv("GLM_API_KEY")
if "model_name" in kwargs and (not model or model == "embedding-3"):
model = kwargs.pop("model_name")
if "base_url" in kwargs:
base_url = kwargs.pop("base_url")
else:
base_url = os.getenv("GLM_BASE_URL", default="https://open.bigmodel.cn/api/paas/v4/")
self.model = model
self.client = ZhipuAI(api_key=api_key, base_url=base_url, **kwargs)
def embed_query(self, text: str) -> List[float]:
# text = text.replace("\n", " ")
return self.client.embeddings.create(input=[text], model=self.model).data[0].embedding
def embed_documents(self, texts: List[str]) -> List[List[float]]:
res = self.client.embeddings.create(input=texts, model=self.model)
res = [r.embedding for r in res.data]
return res
@property
def dimension(self) -> int:
return GLM_MODEL_DIM_MAP[self.model]
================================================
FILE: deepsearcher/embedding/jiekouai_embedding.py
================================================
import os
from typing import List, Union
import requests
from deepsearcher.embedding.base import BaseEmbedding
# TODO: Update with actual Jiekou.AI model dimensions when available
JIEKOUAI_MODEL_DIM_MAP = {
"qwen/qwen3-embedding-0.6b": 1024,
"qwen/qwen3-embedding-8b": 1024,
"baai/bge-m3": 1024,
}
JIEKOUAI_EMBEDDING_API = "https://api.jiekou.ai/openai/v1/embeddings"
class JiekouAIEmbedding(BaseEmbedding):
"""
Jiekou.AI embedding model implementation.
This class provides an interface to the Jiekou.AI embedding API, which offers
various embedding models for text processing.
"""
def __init__(self, model="qwen/qwen3-embedding-8b", batch_size=32, **kwargs):
"""
Initialize the Jiekou.AI embedding model.
Args:
model (str): The model identifier to use for embeddings. Default is "baai/bge-m3".
batch_size (int): Maximum number of texts to process in a single batch. Default is 32.
**kwargs: Additional keyword arguments.
- api_key (str, optional): The Jiekou.AI API key. If not provided,
it will be read from the JIEKOU_API_KEY environment variable.
- model_name (str, optional): Alternative way to specify the model.
Raises:
RuntimeError: If no API key is provided or found in environment variables.
"""
if "model_name" in kwargs and (not model or model == "qwen/qwen3-embedding-8b"):
model = kwargs.pop("model_name")
self.model = model
if "api_key" in kwargs:
api_key = kwargs.pop("api_key")
else:
api_key = os.getenv("JIEKOU_API_KEY")
if not api_key or len(api_key) == 0:
raise RuntimeError("api_key is required for JiekouAIEmbedding")
self.api_key = api_key
self.batch_size = batch_size
def embed_query(self, text: str) -> List[float]:
"""
Embed a single query text.
Args:
text (str): The query text to embed.
Returns:
List[float]: A list of floats representing the embedding vector.
"""
return self._embed_input(text)[0]
def embed_documents(self, texts: List[str]) -> List[List[float]]:
"""
Embed a list of document texts.
This method handles batching of document embeddings based on the configured
batch size to optimize API calls.
Args:
texts (List[str]): A list of document texts to embed.
Returns:
List[List[float]]: A list of embedding vectors, one for each input text.
"""
# batch embedding
if self.batch_size > 0:
if len(texts) > self.batch_size:
batch_texts = [
texts[i : i + self.batch_size] for i in range(0, len(texts), self.batch_size)
]
embeddings = []
for batch_text in batch_texts:
batch_embeddings = self._embed_input(batch_text)
embeddings.extend(batch_embeddings)
return embeddings
return self._embed_input(texts)
return [self.embed_query(text) for text in texts]
def _embed_input(self, input: Union[str, List[str]]) -> List[List[float]]:
"""
Internal method to handle the API call for embedding inputs.
Args:
input (Union[str, List[str]]): Either a single text string or a list of text strings to embed.
Returns:
List[List[float]]: A list of embedding vectors for the input(s).
Raises:
HTTPError: If the API request fails.
"""
headers = {
"Authorization": f"Bearer {self.api_key}",
"Content-Type": "application/json",
}
# Handle both single string and list of strings
input_list = input if isinstance(input, list) else [input]
payload = {"model": self.model, "input": input_list}
response = requests.request("POST", JIEKOUAI_EMBEDDING_API, json=payload, headers=headers)
response.raise_for_status()
result = response.json()["data"]
sorted_results = sorted(result, key=lambda x: x["index"])
return [res["embedding"] for res in sorted_results]
@property
def dimension(self) -> int:
"""
Get the dimensionality of the embeddings for the current model.
Returns:
int: The number of dimensions in the embedding vectors.
"""
return JIEKOUAI_MODEL_DIM_MAP[self.model]
================================================
FILE: deepsearcher/embedding/milvus_embedding.py
================================================
from typing import List
import numpy as np
from deepsearcher.embedding.base import BaseEmbedding
MILVUS_MODEL_DIM_MAP = {
"BAAI/bge-large-en-v1.5": 1024,
"BAAI/bge-base-en-v1.5": 768,
"BAAI/bge-small-en-v1.5": 384,
"BAAI/bge-large-zh-v1.5": 1024,
"BAAI/bge-base-zh-v1.5": 768,
"BAAI/bge-small-zh-v1.5": 384,
"GPTCache/paraphrase-albert-onnx": 768,
"default": 768, # 'GPTCache/paraphrase-albert-onnx',
# see https://github.com/milvus-io/milvus-model/blob/4974e2d190169618a06359bcda040eaed73c4d0f/src/pymilvus/model/dense/onnx.py#L12
"jina-embeddings-v3": 1024, # required jina api key
}
class MilvusEmbedding(BaseEmbedding):
"""
Milvus embedding model implementation.
https://milvus.io/docs/embeddings.md
This class provides an interface to the Milvus embedding models, which offers
various embedding models for text processing, including BGE and Jina models.
"""
def __init__(self, model: str = None, **kwargs) -> None:
"""
Initialize the Milvus embedding model.
Args:
model (str, optional): The model identifier to use for embeddings.
If None, the default model will be used.
**kwargs: Additional keyword arguments passed to the underlying embedding function.
- model_name (str, optional): Alternative way to specify the model.
Raises:
ValueError: If an unsupported model name is provided.
Notes:
Supported models include:
- Default model: "GPTCache/paraphrase-albert-onnx" (768 dimensions)
- BGE models: "BAAI/bge-*" series (various dimensions)
- Jina models: "jina-*" series (requires Jina API key)
"""
model_name = model
from pymilvus import model
if "model_name" in kwargs and (not model_name or model_name == "default"):
model_name = kwargs.pop("model_name")
if not model_name or model_name in [
"default",
"GPTCache/paraphrase-albert-onnx",
]:
self.model = model.DefaultEmbeddingFunction(**kwargs)
else:
if model_name.startswith("jina-"):
self.model = model.dense.JinaEmbeddingFunction(model_name, **kwargs)
elif model_name.startswith("BAAI/"):
self.model = model.dense.SentenceTransformerEmbeddingFunction(model_name, **kwargs)
else:
# Only support default model and BGE series model
raise ValueError(f"Currently unsupported model name: {model_name}")
def embed_query(self, text: str) -> List[float]:
"""
Embed a single query text.
Args:
text (str): The query text to embed.
Returns:
List[float]: A list of floats representing the embedding vector.
"""
return self.model.encode_queries([text])[0]
def embed_documents(self, texts: List[str]) -> List[List[float]]:
"""
Embed a list of document texts.
Args:
texts (List[str]): A list of document texts to embed.
Returns:
List[List[float]]: A list of embedding vectors, one for each input text.
Note:
This method handles conversion from numpy arrays to lists if needed.
"""
embeddings = self.model.encode_documents(texts)
if isinstance(embeddings[0], np.ndarray):
return [embedding.tolist() for embedding in embeddings]
else:
return embeddings
@property
def dimension(self) -> int:
"""
Get the dimensionality of the embeddings for the current model.
Returns:
int: The number of dimensions in the embedding vectors.
"""
return self.model.dim # or MILVUS_MODEL_DIM_MAP[self.model_name]
================================================
FILE: deepsearcher/embedding/novita_embedding.py
================================================
import os
from typing import List, Union
import requests
from deepsearcher.embedding.base import BaseEmbedding
NOVITA_MODEL_DIM_MAP = {
"baai/bge-m3": 1024,
}
NOVITA_EMBEDDING_API = "https://api.novita.ai/v3/openai/embeddings"
class NovitaEmbedding(BaseEmbedding):
"""
Novita AI embedding model implementation.
This class provides an interface to the Novita AI embedding API, which offers
various embedding models for text processing.
For more information, see:
https://novita.ai/docs/api-reference/model-apis-llm-create-embeddings
"""
def __init__(self, model="baai/bge-m3", batch_size=32, **kwargs):
"""
Initialize the Novita AI embedding model.
Args:
model (str): The model identifier to use for embeddings. Default is "baai/bge-m3".
batch_size (int): Maximum number of texts to process in a single batch. Default is 32.
**kwargs: Additional keyword arguments.
- api_key (str, optional): The Novita AI API key. If not provided,
it will be read from the NOVITA_API_KEY environment variable.
- model_name (str, optional): Alternative way to specify the model.
Raises:
RuntimeError: If no API key is provided or found in environment variables.
"""
if "model_name" in kwargs and (not model or model == "baai/bge-m3"):
model = kwargs.pop("model_name")
self.model = model
if "api_key" in kwargs:
api_key = kwargs.pop("api_key")
else:
api_key = os.getenv("NOVITA_API_KEY")
if not api_key or len(api_key) == 0:
raise RuntimeError("api_key is required for NovitaEmbedding")
self.api_key = api_key
self.batch_size = batch_size
def embed_query(self, text: str) -> List[float]:
"""
Embed a single query text.
Args:
text (str): The query text to embed.
Returns:
List[float]: A list of floats representing the embedding vector.
Note:
For retrieval cases, this method uses "query" as the input type.
"""
return self._embed_input(text)[0]
def embed_documents(self, texts: List[str]) -> List[List[float]]:
"""
Embed a list of document texts.
This method handles batching of document embeddings based on the configured
batch size to optimize API calls.
Args:
texts (List[str]): A list of document texts to embed.
Returns:
List[List[float]]: A list of embedding vectors, one for each input text.
"""
# batch embedding
if self.batch_size > 0:
if len(texts) > self.batch_size:
batch_texts = [
texts[i : i + self.batch_size] for i in range(0, len(texts), self.batch_size)
]
embeddings = []
for batch_text in batch_texts:
batch_embeddings = self._embed_input(batch_text)
embeddings.extend(batch_embeddings)
return embeddings
return self._embed_input(texts)
return [self.embed_query(text) for text in texts]
def _embed_input(self, input: Union[str, List[str]]) -> List[List[float]]:
"""
Internal method to handle the API call for embedding inputs.
Args:
input (Union[str, List[str]]): Either a single text string or a list of text strings to embed.
Returns:
List[List[float]]: A list of embedding vectors for the input(s).
Raises:
HTTPError: If the API request fails.
"""
headers = {
"Authorization": f"Bearer {self.api_key}",
"Content-Type": "application/json",
}
payload = {"model": self.model, "input": input, "encoding_format": "float"}
response = requests.request("POST", NOVITA_EMBEDDING_API, json=payload, headers=headers)
response.raise_for_status()
result = response.json()["data"]
sorted_results = sorted(result, key=lambda x: x["index"])
return [res["embedding"] for res in sorted_results]
@property
def dimension(self) -> int:
"""
Get the dimensionality of the embeddings for the current model.
Returns:
int: The number of dimensions in the embedding vectors.
"""
return NOVITA_MODEL_DIM_MAP[self.model]
================================================
FILE: deepsearcher/embedding/ollama_embedding.py
================================================
from typing import List, Union
from deepsearcher.embedding.base import BaseEmbedding
OLLAMA_MODEL_DIM_MAP = {
"bge-m3": 1024,
"mxbai-embed-large": 768,
"nomic-embed-text": 768,
}
class OllamaEmbedding(BaseEmbedding):
"""
Ollama embedding model implementation.
This class provides an interface to the Ollama embedding API, which offers
various embedding models for text processing.
"""
def __init__(self, model="bge-m3", batch_size=32, **kwargs):
"""
Initialize the Ollama embedding model.
Args:
model (str): The model identifier to use for embeddings. Default is "bge-m3".
**kwargs: Additional keyword arguments.
- base_url (str, optional): The base URL for the Ollama API. If not provided,
defaults to "http://localhost:11434".
- model_name (str, optional): Alternative way to specify the model.
- dimension (int, optional): The dimension of the embedding vectors.
If not provided, the default dimension for the model will be used.
"""
from ollama import Client
if "model_name" in kwargs and (not model or model == "bge-m3"):
model = kwargs.pop("model_name")
self.model = model
if "base_url" in kwargs:
base_url = kwargs.pop("base_url")
else:
base_url = "http://localhost:11434/"
# Initialize client first
self.client = Client(host=base_url, **kwargs)
self.batch_size = batch_size
if "dimension" in kwargs:
dimension = kwargs.pop("dimension")
else:
if model in OLLAMA_MODEL_DIM_MAP:
dimension = OLLAMA_MODEL_DIM_MAP[model]
else:
try:
dummy_response = self.client.embed(model=self.model, input="test")
dimension = len(dummy_response["embeddings"][0])
except Exception as e:
print(
f"Warning: Could not determine model dimension, using default 1024. Error: {e}"
)
dimension = 1024
self.dim = dimension
def embed_query(self, text: str) -> List[float]:
"""
Embed a single query text.
Args:
text (str): The query text to embed.
Returns:
List[float]: A list of floats representing the embedding vector.
Note:
For retrieval cases, this method uses "query" as the input type.
"""
return self._embed_input(text)[0]
def embed_documents(self, texts: List[str]) -> List[List[float]]:
"""
Embed a list of document texts.
This method handles batching of document embeddings based on the configured
batch size to optimize API calls.
Args:
texts (List[str]): A list of document texts to embed.
Returns:
List[List[float]]: A list of embedding vectors, one for each input text.
"""
# batch embedding
if self.batch_size > 0:
if len(texts) > self.batch_size:
batch_texts = [
texts[i : i + self.batch_size] for i in range(0, len(texts), self.batch_size)
]
embeddings = []
for batch_text in batch_texts:
batch_embeddings = self._embed_input(batch_text)
embeddings.extend(batch_embeddings)
return embeddings
return self._embed_input(texts)
return [self.embed_query(text) for text in texts]
def _embed_input(self, input: Union[str, List[str]]) -> List[List[float]]:
"""
Internal method to handle the API call for embedding inputs.
Args:
input (Union[str, List[str]]): Either a single text string or a list of text strings to embed.
Returns:
List[List[float]]: A list of embedding vectors for the input(s).
Raises:
HTTPError: If the API request fails.
"""
response = self.client.embed(model=self.model, input=input)
return response["embeddings"]
@property
def dimension(self) -> int:
"""
Get the dimensionality of the embeddings for the current model.
Returns:
int: The number of dimensions in the embedding vectors.
"""
return self.dim
================================================
FILE: deepsearcher/embedding/openai_embedding.py
================================================
import os
from typing import List
from openai._types import NOT_GIVEN
from deepsearcher.embedding.base import BaseEmbedding
OPENAI_MODEL_DIM_MAP = {
"text-embedding-ada-002": 1536,
"text-embedding-3-small": 1536,
"text-embedding-3-large": 3072,
}
class OpenAIEmbedding(BaseEmbedding):
"""
OpenAI embedding model implementation.
This class provides an interface to the OpenAI embedding API, which offers
various embedding models for text processing.
For more information, see:
https://platform.openai.com/docs/guides/embeddings/use-cases
"""
def __init__(self, model: str = "text-embedding-ada-002", **kwargs):
"""
Initialize the OpenAI embedding model.
Args:
model (str): The model identifier to use for embeddings. Default is "text-embedding-ada-002".
**kwargs: Additional keyword arguments.
- api_key (str, optional): The OpenAI API key. If not provided,
it will be read from the OPENAI_API_KEY environment variable.
- base_url (str, optional): The base URL for the OpenAI API. If not provided,
it will be read from the OPENAI_BASE_URL environment variable.
- model_name (str, optional): Alternative way to specify the model.
- dimension (int, optional): The dimension of the embedding vectors.
If not provided, the default dimension for the model will be used.
- azure_endpoint (str, optional): If provided, use Azure OpenAI instead.
- api_version (str, optional): Azure API version to use. Default is "2023-05-15".
Notes:
Available models:
- 'text-embedding-ada-002': No dimension needed, default is 1536
- 'text-embedding-3-small': dimensions from 512 to 1536, default is 1536
- 'text-embedding-3-large': dimensions from 1024 to 3072, default is 3072
"""
# Extract Azure-specific parameters
azure_endpoint = kwargs.pop("azure_endpoint", None)
api_version = kwargs.pop("api_version", "2023-05-15")
azure_deployment = kwargs.pop("azure_deployment", None)
# Extract standard parameters (keep original behavior)
if "api_key" in kwargs:
api_key = kwargs.pop("api_key")
else:
api_key = os.getenv("OPENAI_API_KEY")
if "base_url" in kwargs:
base_url = kwargs.pop("base_url")
else:
base_url = os.getenv("OPENAI_BASE_URL")
if "model_name" in kwargs and (not model or model == "text-embedding-ada-002"):
model = kwargs.pop("model_name")
if "dimension" in kwargs:
dimension = kwargs.pop("dimension")
else:
dimension = OPENAI_MODEL_DIM_MAP.get(model, 1536)
self.dim = dimension
self.model = model
# Initialize the appropriate client based on parameters
if azure_endpoint:
from openai import AzureOpenAI
self.client = AzureOpenAI(
api_key=api_key, api_version=api_version, azure_endpoint=azure_endpoint, **kwargs
)
# Store the deployment name to use for Azure
self.deployment = azure_deployment if azure_deployment is not None else model
self.is_azure = True
else:
from openai import OpenAI
self.client = OpenAI(api_key=api_key, base_url=base_url, **kwargs)
self.is_azure = False
def _get_dim(self):
"""
Get the dimension parameter for the API call.
Returns:
int or NOT_GIVEN: The dimension to use for the embedding, or NOT_GIVEN
if using text-embedding-ada-002 which doesn't support custom dimensions.
"""
return self.dim if self.model != "text-embedding-ada-002" else NOT_GIVEN
def embed_query(self, text: str) -> List[float]:
"""
Embed a single query text.
Args:
text (str): The query text to embed.
Returns:
List[float]: A list of floats representing the embedding vector.
"""
if self.is_azure:
response = self.client.embeddings.create(
input=[text],
model=self.model, # For Azure, this is the deployment name
)
else:
response = self.client.embeddings.create(
input=[text], model=self.model, dimensions=self._get_dim()
)
return response.data[0].embedding
def embed_documents(self, texts: List[str]) -> List[List[float]]:
"""
Embed a list of document texts.
Args:
texts (List[str]): A list of document texts to embed.
Returns:
List[List[float]]: A list of embedding vectors, one for each input text.
"""
if self.is_azure:
response = self.client.embeddings.create(
input=texts,
model=self.model, # For Azure, this is the deployment name
)
else:
response = self.client.embeddings.create(
input=texts, model=self.model, dimensions=self._get_dim()
)
return [r.embedding for r in response.data]
@property
def dimension(self) -> int:
"""
Get the dimensionality of the embeddings for the current model.
Returns:
int: The number of dimensions in the embedding vectors.
"""
return self.dim
================================================
FILE: deepsearcher/embedding/ppio_embedding.py
================================================
import os
from typing import List, Union
import requests
from deepsearcher.embedding.base import BaseEmbedding
# TODO: Update with actual PPIO model dimensions when available
PPIO_MODEL_DIM_MAP = {
"baai/bge-m3": 1024,
}
PPIO_EMBEDDING_API = "https://api.ppinfra.com/v3/openai/embeddings"
class PPIOEmbedding(BaseEmbedding):
"""
PPIO embedding model implementation.
This class provides an interface to the PPIO embedding API, which offers
various embedding models for text processing.
"""
def __init__(self, model="baai/bge-m3", batch_size=32, **kwargs):
"""
Initialize the PPIO embedding model.
Args:
model (str): The model identifier to use for embeddings. Default is "baai/bge-m3".
batch_size (int): Maximum number of texts to process in a single batch. Default is 32.
**kwargs: Additional keyword arguments.
- api_key (str, optional): The PPIO API key. If not provided,
it will be read from the PPIO_API_KEY environment variable.
- model_name (str, optional): Alternative way to specify the model.
Raises:
RuntimeError: If no API key is provided or found in environment variables.
"""
if "model_name" in kwargs and (not model or model == "baai/bge-m3"):
model = kwargs.pop("model_name")
self.model = model
if "api_key" in kwargs:
api_key = kwargs.pop("api_key")
else:
api_key = os.getenv("PPIO_API_KEY")
if not api_key or len(api_key) == 0:
raise RuntimeError("api_key is required for PPIOEmbedding")
self.api_key = api_key
self.batch_size = batch_size
def embed_query(self, text: str) -> List[float]:
"""
Embed a single query text.
Args:
text (str): The query text to embed.
Returns:
List[float]: A list of floats representing the embedding vector.
"""
return self._embed_input(text)[0]
def embed_documents(self, texts: List[str]) -> List[List[float]]:
"""
Embed a list of document texts.
This method handles batching of document embeddings based on the configured
batch size to optimize API calls.
Args:
texts (List[str]): A list of document texts to embed.
Returns:
List[List[float]]: A list of embedding vectors, one for each input text.
"""
# batch embedding
if self.batch_size > 0:
if len(texts) > self.batch_size:
batch_texts = [
texts[i : i + self.batch_size] for i in range(0, len(texts), self.batch_size)
]
embeddings = []
for batch_text in batch_texts:
batch_embeddings = self._embed_input(batch_text)
embeddings.extend(batch_embeddings)
return embeddings
return self._embed_input(texts)
return [self.embed_query(text) for text in texts]
def _embed_input(self, input: Union[str, List[str]]) -> List[List[float]]:
"""
Internal method to handle the API call for embedding inputs.
Args:
input (Union[str, List[str]]): Either a single text string or a list of text strings to embed.
Returns:
List[List[float]]: A list of embedding vectors for the input(s).
Raises:
HTTPError: If the API request fails.
"""
headers = {
"Authorization": f"Bearer {self.api_key}",
"Content-Type": "application/json",
}
# Handle both single string and list of strings
input_list = input if isinstance(input, list) else [input]
payload = {"model": self.model, "input": input_list}
response = requests.request("POST", PPIO_EMBEDDING_API, json=payload, headers=headers)
response.raise_for_status()
result = response.json()["data"]
sorted_results = sorted(result, key=lambda x: x["index"])
return [res["embedding"] for res in sorted_results]
@property
def dimension(self) -> int:
"""
Get the dimensionality of the embeddings for the current model.
Returns:
int: The number of dimensions in the embedding vectors.
"""
return PPIO_MODEL_DIM_MAP[self.model]
================================================
FILE: deepsearcher/embedding/sentence_transformer_embedding.py
================================================
from functools import cached_property
from typing import List, Union
from deepsearcher.embedding.base import BaseEmbedding
SENTENCE_TRANSFORMER_MODEL_DIM_MAP = {
"BAAI/bge-m3": 1024,
"BAAI/bge-large-zh-v1.5": 1024,
"BAAI/bge-large-en-v1.5": 1024,
}
class SentenceTransformerEmbedding(BaseEmbedding):
"""
SentenceTransformer embedding model implementation.
This class provides an interface to the SentenceTransformer embedding API, which offers
various embedding models for text processing.
For more information, see:
https://www.sbert.net/docs/sentence_transformer/pretrained_models.html
"""
def __init__(self, model="BAAI/bge-m3", batch_size=32, **kwargs):
"""
Initialize the SentenceTransformer embedding model.
Args:
model (str): The model identifier to use for embeddings. Default is "BAAI/bge-m3".
batch_size (int): Maximum number of texts to process in a single batch. Default is 32.
**kwargs: Additional keyword arguments.
- model_name (str, optional): Alternative way to specify the model.
Raises:
RuntimeError: If no API key is provided or found in environment variables.
"""
from sentence_transformers import SentenceTransformer
if "model_name" in kwargs and (not model or model == "BAAI/bge-m3"):
model = kwargs.pop("model_name")
self.model = model
self.client = SentenceTransformer(model)
self.batch_size = batch_size
def embed_query(self, text: str) -> List[float]:
"""
Embed a single query text.
Args:
text (str): The query text to embed.
Returns:
List[float]: A list of floats representing the embedding vector.
Note:
For retrieval cases, this method uses "query" as the input type.
"""
return self._embed_input(text)[0]
def embed_documents(self, texts: List[str]) -> List[List[float]]:
"""
Embed a list of document texts.
This method handles batching of document embeddings based on the configured
batch size to optimize API calls.
Args:
texts (List[str]): A list of document texts to embed.
Returns:
List[List[float]]: A list of embedding vectors, one for each input text.
"""
# batch embedding
if self.batch_size > 0:
if len(texts) > self.batch_size:
batch_texts = [
texts[i : i + self.batch_size] for i in range(0, len(texts), self.batch_size)
]
embeddings = []
for batch_text in batch_texts:
batch_embeddings = self._embed_input(batch_text)
embeddings.extend(batch_embeddings)
return embeddings
return self._embed_input(texts)
return [self.embed_query(text) for text in texts]
def _embed_input(self, input: Union[str, List[str]]) -> List[List[float]]:
"""
Internal method to handle the API call for embedding inputs.
Args:
input (Union[str, List[str]]): Either a single text string or a list of text strings to embed.
Returns:
List[List[float]]: A list of embedding vectors for the input(s).
Raises:
HTTPError: If the API request fails.
"""
embeddings = self.client.encode(input)
from deepsearcher.utils import log
log.dev_logger.info(f"embeddings: {embeddings}")
return embeddings.tolist()
@cached_property
def dimension(self) -> int:
"""
Get the dimensionality of the embeddings for the current model.
Returns:
int: The number of dimensions in the embedding vectors.
"""
return SENTENCE_TRANSFORMER_MODEL_DIM_MAP[self.model]
================================================
FILE: deepsearcher/embedding/siliconflow_embedding.py
================================================
import os
from typing import List, Union
import requests
from deepsearcher.embedding.base import BaseEmbedding
SILICONFLOW_MODEL_DIM_MAP = {
"BAAI/bge-m3": 1024,
"netease-youdao/bce-embedding-base_v1": 768,
"BAAI/bge-large-zh-v1.5": 1024,
"BAAI/bge-large-en-v1.5": 1024,
"Pro/BAAI/bge-m3": 1024, # paid model
}
SILICONFLOW_EMBEDDING_API = "https://api.siliconflow.cn/v1/embeddings"
class SiliconflowEmbedding(BaseEmbedding):
"""
SiliconFlow embedding model implementation.
This class provides an interface to the SiliconFlow embedding API, which offers
various embedding models for text processing.
For more information, see:
https://docs.siliconflow.cn/en/api-reference/embeddings/create-embeddings
"""
def __init__(self, model="BAAI/bge-m3", batch_size=32, **kwargs):
"""
Initialize the SiliconFlow embedding model.
Args:
model (str): The model identifier to use for embeddings. Default is "BAAI/bge-m3".
batch_size (int): Maximum number of texts to process in a single batch. Default is 32.
**kwargs: Additional keyword arguments.
- api_key (str, optional): The SiliconFlow API key. If not provided,
it will be read from the SILICONFLOW_API_KEY environment variable.
- model_name (str, optional): Alternative way to specify the model.
Raises:
RuntimeError: If no API key is provided or found in environment variables.
"""
if "model_name" in kwargs and (not model or model == "BAAI/bge-m3"):
model = kwargs.pop("model_name")
self.model = model
if "api_key" in kwargs:
api_key = kwargs.pop("api_key")
else:
api_key = os.getenv("SILICONFLOW_API_KEY")
if not api_key or len(api_key) == 0:
raise RuntimeError("api_key is required for SiliconflowEmbedding")
self.api_key = api_key
self.batch_size = batch_size
def embed_query(self, text: str) -> List[float]:
"""
Embed a single query text.
Args:
text (str): The query text to embed.
Returns:
List[float]: A list of floats representing the embedding vector.
Note:
For retrieval cases, this method uses "query" as the input type.
"""
return self._embed_input(text)[0]
def embed_documents(self, texts: List[str]) -> List[List[float]]:
"""
Embed a list of document texts.
This method handles batching of document embeddings based on the configured
batch size to optimize API calls.
Args:
texts (List[str]): A list of document texts to embed.
Returns:
List[List[float]]: A list of embedding vectors, one for each input text.
"""
# batch embedding
if self.batch_size > 0:
if len(texts) > self.batch_size:
batch_texts = [
texts[i : i + self.batch_size] for i in range(0, len(texts), self.batch_size)
]
embeddings = []
for batch_text in batch_texts:
batch_embeddings = self._embed_input(batch_text)
embeddings.extend(batch_embeddings)
return embeddings
return self._embed_input(texts)
return [self.embed_query(text) for text in texts]
def _embed_input(self, input: Union[str, List[str]]) -> List[List[float]]:
"""
Internal method to handle the API call for embedding inputs.
Args:
input (Union[str, List[str]]): Either a single text string or a list of text strings to embed.
Returns:
List[List[float]]: A list of embedding vectors for the input(s).
Raises:
HTTPError: If the API request fails.
"""
headers = {
"Authorization": f"Bearer {self.api_key}",
"Content-Type": "application/json",
}
payload = {"model": self.model, "input": input, "encoding_format": "float"}
response = requests.request(
"POST", SILICONFLOW_EMBEDDING_API, json=payload, headers=headers
)
response.raise_for_status()
result = response.json()["data"]
sorted_results = sorted(result, key=lambda x: x["index"])
return [res["embedding"] for res in sorted_results]
@property
def dimension(self) -> int:
"""
Get the dimensionality of the embeddings for the current model.
Returns:
int: The number of dimensions in the embedding vectors.
"""
return SILICONFLOW_MODEL_DIM_MAP[self.model]
================================================
FILE: deepsearcher/embedding/volcengine_embedding.py
================================================
import os
from typing import List, Union
import requests
from deepsearcher.embedding.base import BaseEmbedding
VOLCENGINE_MODEL_DIM_MAP = {
"doubao-embedding-large-text-240915": 4096,
"doubao-embedding-text-240715": 2560,
"doubao-embedding-text-240515": 2048,
}
VOLCENGINE_EMBEDDING_API = "https://ark.cn-beijing.volces.com/api/v3/embeddings"
class VolcengineEmbedding(BaseEmbedding):
"""
Volcengine embedding model implementation.
This class provides an interface to the Volcengine embedding API, which offers
various embedding models for text processing.
For more information, see:
https://www.volcengine.com/docs/82379/1302003
"""
def __init__(self, model="doubao-embedding-large-text-240915", batch_size=256, **kwargs):
"""
Initialize the Volcengine embedding model.
Args:
model (str): The model identifier to use for embeddings. Default is "doubao-embedding-large-text-240915".
batch_size (int): Maximum number of texts to process in a single batch. Default is 256.
**kwargs: Additional keyword arguments.
- api_key (str, optional): The Volcengine API key. If not provided,
it will be read from the VOLCENGINE_API_KEY environment variable.
- model_name (str, optional): Alternative way to specify the model.
Raises:
RuntimeError: If no API key is provided or found in environment variables.
"""
if "model_name" in kwargs and (not model or model == "doubao-embedding-large-text-240915"):
model = kwargs.pop("model_name")
self.model = model
if "api_key" in kwargs:
api_key = kwargs.pop("api_key")
else:
api_key = os.getenv("VOLCENGINE_API_KEY")
if not api_key or len(api_key) == 0:
raise RuntimeError("api_key is required for VolcengineEmbedding")
self.api_key = api_key
self.batch_size = batch_size
def embed_query(self, text: str) -> List[float]:
"""
Embed a single query text.
Args:
text (str): The query text to embed.
Returns:
List[float]: A list of floats representing the embedding vector.
Note:
For retrieval cases, this method uses "query" as the input type.
"""
return self._embed_input(text)[0]
def embed_documents(self, texts: List[str]) -> List[List[float]]:
"""
Embed a list of document texts.
This method handles batching of document embeddings based on the configured
batch size to optimize API calls.
Args:
texts (List[str]): A list of document texts to embed.
Returns:
List[List[float]]: A list of embedding vectors, one for each input text.
"""
# batch embedding
if self.batch_size > 0:
if len(texts) > self.batch_size:
batch_texts = [
texts[i : i + self.batch_size] for i in range(0, len(texts), self.batch_size)
]
embeddings = []
for batch_text in batch_texts:
batch_embeddings = self._embed_input(batch_text)
embeddings.extend(batch_embeddings)
return embeddings
return self._embed_input(texts)
return [self.embed_query(text) for text in texts]
def _embed_input(self, input: Union[str, List[str]]) -> List[List[float]]:
"""
Internal method to handle the API call for embedding inputs.
Args:
input (Union[str, List[str]]): Either a single text string or a list of text strings to embed.
Returns:
List[List[float]]: A list of embedding vectors for the input(s).
Raises:
HTTPError: If the API request fails.
"""
headers = {
"Authorization": f"Bearer {self.api_key}",
"Content-Type": "application/json",
}
payload = {"model": self.model, "input": input, "encoding_format": "float"}
response = requests.request("POST", VOLCENGINE_EMBEDDING_API, json=payload, headers=headers)
response.raise_for_status()
result = response.json()["data"]
sorted_results = sorted(result, key=lambda x: x["index"])
return [res["embedding"] for res in sorted_results]
@property
def dimension(self) -> int:
"""
Get the dimensionality of the embeddings for the current model.
Returns:
int: The number of dimensions in the embedding vectors.
"""
return VOLCENGINE_MODEL_DIM_MAP[self.model]
================================================
FILE: deepsearcher/embedding/voyage_embedding.py
================================================
import os
from typing import List
from deepsearcher.embedding.base import BaseEmbedding
VOYAGE_MODEL_DIM_MAP = {
"voyage-3-large": 1024,
"voyage-3": 1024,
"voyage-3-lite": 512,
}
class VoyageEmbedding(BaseEmbedding):
"""
Voyage AI embedding model implementation.
This class provides an interface to the Voyage AI embedding API, which offers
various embedding models for text processing.
For more information, see:
https://docs.voyageai.com/embeddings/
"""
def __init__(self, model="voyage-3", **kwargs):
"""
Initialize the Voyage AI embedding model.
Args:
model (str): The model identifier to use for embeddings. Default is "voyage-3".
**kwargs: Additional keyword arguments.
- api_key (str, optional): The Voyage AI API key. If not provided,
it will be read from the VOYAGE_API_KEY environment variable.
- model_name (str, optional): Alternative way to specify the model.
- Additional parameters passed to the voyageai.Client.
"""
if "model_name" in kwargs and (not model or model == "voyage-3"):
model = kwargs.pop("model_name")
self.model = model
if "api_key" in kwargs:
api_key = kwargs.pop("api_key")
else:
api_key = os.getenv("VOYAGE_API_KEY")
self.voyageai_api_key = api_key
import voyageai
voyageai.api_key = self.voyageai_api_key
self.vo = voyageai.Client(**kwargs)
def embed_query(self, text: str) -> List[float]:
"""
Embed a single query text.
Args:
text (str): The query text to embed.
Returns:
List[float]: A list of floats representing the embedding vector.
Note:
For retrieval cases, this method uses "query" as the input type.
"""
embeddings = self.vo.embed([text], model=self.model, input_type="query")
return embeddings.embeddings[0]
def embed_documents(self, texts: List[str]) -> List[List[float]]:
"""
Embed a list of document texts.
Args:
texts (List[str]): A list of document texts to embed.
Returns:
List[List[float]]: A list of embedding vectors, one for each input text.
Note:
This method uses "document" as the input type for retrieval optimization.
"""
embeddings = self.vo.embed(texts, model=self.model, input_type="document")
return embeddings.embeddings
@property
def dimension(self) -> int:
"""
Get the dimensionality of the embeddings for the current model.
Returns:
int: The number of dimensions in the embedding vectors.
"""
return VOYAGE_MODEL_DIM_MAP[self.model]
================================================
FILE: deepsearcher/embedding/watsonx_embedding.py
================================================
import logging
import os
from typing import List
from deepsearcher.embedding.base import BaseEmbedding
try:
from ibm_watsonx_ai import Credentials
from ibm_watsonx_ai.foundation_models import Embeddings
except ImportError:
Credentials = None
Embeddings = None
try:
from transformers import AutoTokenizer
TRANSFORMERS_AVAILABLE = True
except ImportError:
TRANSFORMERS_AVAILABLE = False
# Set up logging
logger = logging.getLogger(__name__)
class WatsonXEmbedding(BaseEmbedding):
"""
IBM watsonx.ai embedding model implementation.
This class provides an interface to the IBM watsonx.ai embedding API, which offers
various embedding models for text processing. It automatically handles text truncation
to fit within the model's token limits.
For more information, see:
https://www.ibm.com/products/watsonx-ai/foundation-models#ibmembedding
"""
def __init__(self, model: str = "ibm/slate-125m-english-rtrvr-v2", **kwargs):
"""
Initialize the watsonx.ai embedding model.
Args:
model (str): The model identifier to use for embeddings.
Default is "ibm/slate-125m-english-rtrvr-v2".
**kwargs: Additional keyword arguments for WatsonX configuration.
- url (str): WatsonX endpoint URL
- project_id (str): WatsonX project ID
- space_id (str): WatsonX deployment space ID (alternative to project_id)
- max_tokens (int): Maximum number of tokens per text (default: 480)
- use_tokenizer (bool): Whether to use HuggingFace tokenizer for precise counting (default: True)
Environment Variables:
WATSONX_APIKEY: IBM Cloud API key for authentication
WATSONX_URL: WatsonX service endpoint URL
WATSONX_PROJECT_ID: WatsonX project ID (if not provided in kwargs)
"""
if Credentials is None or Embeddings is None:
raise ImportError(
"WatsonX embedding requires ibm-watsonx-ai. "
"Install it with: pip install ibm-watsonx-ai"
)
self.model = model
# Set max tokens to 480 to leave more room for start/end tokens and safety margin (model limit is 512)
self.max_tokens = kwargs.pop("max_tokens", 480)
self.use_tokenizer = kwargs.pop("use_tokenizer", True) and TRANSFORMERS_AVAILABLE
# Initialize tokenizer if available and requested
self.tokenizer = None
if self.use_tokenizer:
try:
# Use a general-purpose tokenizer for token counting
# BERT tokenizer is a good default for most models
self.tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
logger.info("Using HuggingFace tokenizer for precise token counting")
except Exception as e:
logger.warning(
f"Failed to load tokenizer, falling back to character-based estimation: {e}"
)
self.use_tokenizer = False
# Get credentials from environment or kwargs
api_key = kwargs.pop("api_key", os.getenv("WATSONX_APIKEY"))
url = kwargs.pop("url", os.getenv("WATSONX_URL"))
project_id = kwargs.pop("project_id", None)
space_id = kwargs.pop("space_id", None)
# Only get project_id from environment if neither project_id nor space_id were provided
if project_id is None and space_id is None:
project_id = os.getenv("WATSONX_PROJECT_ID")
if not api_key:
raise ValueError("WATSONX_APIKEY environment variable or api_key parameter is required")
if not url:
raise ValueError("WATSONX_URL environment variable or url parameter is required")
if not project_id and not space_id:
raise ValueError(
"WATSONX_PROJECT_ID environment variable, project_id or space_id parameter is required"
)
# Set up credentials
credentials = Credentials(url=url, api_key=api_key)
# Initialize the embeddings client - prioritize space_id if provided
if space_id:
self.client = Embeddings(
model_id=self.model, credentials=credentials, space_id=space_id
)
else:
self.client = Embeddings(
model_id=self.model, credentials=credentials, project_id=project_id
)
# Get dimension for this model
self.dim = self._get_dim()
def _get_dim(self):
"""
Get the dimensionality of embeddings for the current model.
Returns:
int: The number of dimensions in the embedding vectors.
"""
# Common WatsonX embedding model dimensions
model_dimensions = {
"ibm/granite-embedding-278m-multilingual": 768,
"ibm/granite-embedding-107m-multilingual": 384,
"ibm/slate-125m-english-rtrvr-v2": 768,
"ibm/slate-125m-english-rtrvr": 768,
"ibm/slate-30m-english-rtrvr-v2": 384,
"ibm/slate-30m-english-rtrvr": 384,
"sentence-transformers/all-minilm-l6-v2": 384,
"sentence-transformers/all-minilm-l12-v2": 384,
"sentence-transformers/all-mpnet-base-v2": 768,
}
return model_dimensions.get(self.model, 768) # Default to 768
def _count_tokens(self, text: str) -> int:
"""
Count the number of tokens in the text.
Args:
text (str): The input text.
Returns:
int: The number of tokens.
"""
if self.use_tokenizer and self.tokenizer:
return len(self.tokenizer.encode(text, add_special_tokens=False))
else:
# Fallback: rough estimation using word count
# For most languages, token count is roughly word count * 1.3
words = len(text.split())
return int(words * 1.3)
def _truncate_text(self, text: str) -> str:
"""
Truncate text to fit within the model's token limit.
Args:
text (str): The input text to truncate.
Returns:
str: The truncated text.
"""
if not text:
return text
# First check if truncation is needed
token_count = self._count_tokens(text)
if token_count <= self.max_tokens:
return text
logger.debug(
f"Text exceeds {self.max_tokens} tokens (actual: {token_count}), truncating..."
)
if self.use_tokenizer and self.tokenizer:
# Use tokenizer for precise truncation
tokens = self.tokenizer.encode(text, add_special_tokens=False)
if len(tokens) > self.max_tokens:
truncated_tokens = tokens[: self.max_tokens]
truncated_text = self.tokenizer.decode(truncated_tokens, skip_special_tokens=True)
# Double-check the token count after truncation
final_count = self._count_tokens(truncated_text)
if final_count > self.max_tokens:
# If still too long, be more aggressive
logger.warning(
f"Aggressive truncation needed: {final_count} -> {self.max_tokens}"
)
truncated_tokens = tokens[: self.max_tokens - 10] # Extra safety margin
truncated_text = self.tokenizer.decode(
truncated_tokens, skip_special_tokens=True
)
return truncated_text
return text
else:
# Fallback: character-based truncation with word boundary respect
# Be more conservative: 3 characters per token for safety
max_chars = self.max_tokens * 3
if len(text) <= max_chars:
return text
# Truncate and try to end at a word boundary
truncated = text[:max_chars]
# Find the last space to avoid cutting words in half
last_space = truncated.rfind(" ")
if last_space > max_chars * 0.7: # More conservative threshold
truncated = truncated[:last_space]
# Double-check with token counting
final_count = self._count_tokens(truncated)
if final_count > self.max_tokens:
# If still too long, be more aggressive
logger.warning(
f"Fallback aggressive truncation needed: {final_count} -> {self.max_tokens}"
)
# Reduce by 20% and try again
max_chars = int(max_chars * 0.8)
truncated = text[:max_chars]
last_space = truncated.rfind(" ")
if last_space > max_chars * 0.7:
truncated = truncated[:last_space]
return truncated
def embed_query(self, text: str) -> List[float]:
"""
Embed a single query text.
Args:
text (str): The query text to embed.
Returns:
List[float]: A list of floats representing the embedding vector.
"""
try:
# Truncate text if it's too long
truncated_text = self._truncate_text(text)
if len(truncated_text) != len(text):
logger.warning(
f"Query text was truncated from {len(text)} to {len(truncated_text)} characters"
)
# The embed_query method expects a single string and returns a list of floats directly
response = self.client.embed_query(text=truncated_text)
return response
except Exception as e:
raise RuntimeError(f"Error embedding query with WatsonX: {str(e)}")
def embed_documents(self, texts: List[str]) -> List[List[float]]:
"""
Embed a list of document texts.
Args:
texts (List[str]): A list of document texts to embed.
Returns:
List[List[float]]: A list of embedding vectors, one for each input text.
"""
try:
# Truncate all texts if they're too long. Current limit from watsonx.ai API is 512 context tokens
truncated_texts = []
truncation_count = 0
for i, text in enumerate(texts):
truncated_text = self._truncate_text(text)
if len(truncated_text) != len(text):
truncation_count += 1
logger.debug(
f"Document {i} was truncated from {len(text)} to {len(truncated_text)} characters"
)
truncated_texts.append(truncated_text)
if truncation_count > 0:
logger.warning(
f"Truncated {truncation_count} out of {len(texts)} documents to fit token limits"
)
# The embed_documents method expects a list of strings and returns a list of embedding vectors directly
response = self.client.embed_documents(texts=truncated_texts)
return response
except Exception as e:
raise RuntimeError(f"Error embedding documents with WatsonX: {str(e)}")
def _embed_documents_individually(self, texts: List[str]) -> List[List[float]]:
"""
Embed documents one by one, skipping problematic ones.
Args:
texts (List[str]): A list of document texts to embed.
Returns:
List[List[float]]: A list of embedding vectors, one for each input text.
"""
embeddings = []
failed_count = 0
for i, text in enumerate(texts):
try:
# Use very conservative truncation
original_max = self.max_tokens
self.max_tokens = 350 # Very conservative limit
truncated_text = self._truncate_text(text)
self.max_tokens = original_max
# Try to embed single text using embed_query API - it returns the embedding directly
embedding = self.client.embed_query(text=truncated_text)
embeddings.append(embedding)
except Exception as e:
failed_count += 1
logger.error(f"Failed to embed document {i}: {str(e)}")
# Use zero vector as fallback
zero_embedding = [0.0] * self.dimension
embeddings.append(zero_embedding)
if failed_count > 0:
logger.warning(
f"Failed to embed {failed_count} out of {len(texts)} documents. Using zero vectors as fallback."
)
return embeddings
@property
def dimension(self) -> int:
"""
Get the dimensionality of the embeddings for the current model.
Returns:
int: The number of dimensions in the embedding vectors.
"""
return self.dim
================================================
FILE: deepsearcher/llm/__init__.py
================================================
from .aliyun import Aliyun
from .anthropic_llm import Anthropic
from .azure_openai import AzureOpenAI
from .bedrock import Bedrock
from .deepseek import DeepSeek
from .gemini import Gemini
from .glm import GLM
from .jiekouai import JiekouAI
from .novita import Novita
from .ollama import Ollama
from .openai_llm import OpenAI
from .ppio import PPIO
from .siliconflow import SiliconFlow
from .together_ai import TogetherAI
from .volc
gitextract_whyhti9y/
├── .github/
│ ├── ISSUE_TEMPLATE/
│ │ ├── bug_report.md
│ │ └── feature_request.md
│ ├── mergify.yml
│ └── workflows/
│ ├── cd-docs.yml
│ ├── ci-docs.yml
│ ├── docs.yml
│ ├── release.yml
│ └── ruff.yml
├── .gitignore
├── .python-version
├── .vscode/
│ └── settings.json
├── CONTRIBUTING.md
├── Dockerfile
├── LICENSE
├── MAINTAINERS
├── Makefile
├── OWNERS
├── OWNERS_ALIASES
├── README.md
├── deepsearcher/
│ ├── __init__.py
│ ├── agent/
│ │ ├── __init__.py
│ │ ├── base.py
│ │ ├── chain_of_rag.py
│ │ ├── collection_router.py
│ │ ├── deep_search.py
│ │ ├── naive_rag.py
│ │ └── rag_router.py
│ ├── cli.py
│ ├── config.yaml
│ ├── configuration.py
│ ├── embedding/
│ │ ├── __init__.py
│ │ ├── base.py
│ │ ├── bedrock_embedding.py
│ │ ├── fastembed_embdding.py
│ │ ├── gemini_embedding.py
│ │ ├── glm_embedding.py
│ │ ├── jiekouai_embedding.py
│ │ ├── milvus_embedding.py
│ │ ├── novita_embedding.py
│ │ ├── ollama_embedding.py
│ │ ├── openai_embedding.py
│ │ ├── ppio_embedding.py
│ │ ├── sentence_transformer_embedding.py
│ │ ├── siliconflow_embedding.py
│ │ ├── volcengine_embedding.py
│ │ ├── voyage_embedding.py
│ │ └── watsonx_embedding.py
│ ├── llm/
│ │ ├── __init__.py
│ │ ├── aliyun.py
│ │ ├── anthropic_llm.py
│ │ ├── azure_openai.py
│ │ ├── base.py
│ │ ├── bedrock.py
│ │ ├── deepseek.py
│ │ ├── gemini.py
│ │ ├── glm.py
│ │ ├── jiekouai.py
│ │ ├── novita.py
│ │ ├── ollama.py
│ │ ├── openai_llm.py
│ │ ├── ppio.py
│ │ ├── siliconflow.py
│ │ ├── together_ai.py
│ │ ├── volcengine.py
│ │ ├── watsonx.py
│ │ └── xai.py
│ ├── loader/
│ │ ├── __init__.py
│ │ ├── file_loader/
│ │ │ ├── __init__.py
│ │ │ ├── base.py
│ │ │ ├── docling_loader.py
│ │ │ ├── json_loader.py
│ │ │ ├── pdf_loader.py
│ │ │ ├── text_loader.py
│ │ │ └── unstructured_loader.py
│ │ ├── splitter.py
│ │ └── web_crawler/
│ │ ├── __init__.py
│ │ ├── base.py
│ │ ├── crawl4ai_crawler.py
│ │ ├── docling_crawler.py
│ │ ├── firecrawl_crawler.py
│ │ └── jina_crawler.py
│ ├── offline_loading.py
│ ├── online_query.py
│ ├── utils/
│ │ ├── __init__.py
│ │ └── log.py
│ └── vector_db/
│ ├── __init__.py
│ ├── azure_search.py
│ ├── base.py
│ ├── milvus.py
│ ├── oracle.py
│ └── qdrant.py
├── docs/
│ ├── README.md
│ ├── configuration/
│ │ ├── embedding.md
│ │ ├── file_loader.md
│ │ ├── index.md
│ │ ├── llm.md
│ │ ├── vector_db.md
│ │ └── web_crawler.md
│ ├── contributing/
│ │ └── index.md
│ ├── examples/
│ │ ├── basic_example.md
│ │ ├── docling.md
│ │ ├── firecrawl.md
│ │ ├── index.md
│ │ ├── oracle.md
│ │ └── unstructured.md
│ ├── faq/
│ │ └── index.md
│ ├── future_plans.md
│ ├── index.md
│ ├── installation/
│ │ ├── development.md
│ │ ├── index.md
│ │ └── pip.md
│ ├── integrations/
│ │ └── index.md
│ ├── overrides/
│ │ └── .gitkeep
│ ├── stylesheets/
│ │ └── extra.css
│ └── usage/
│ ├── cli.md
│ ├── deployment.md
│ ├── index.md
│ └── quick_start.md
├── env.example
├── evaluation/
│ ├── README.md
│ ├── eval_config.yaml
│ └── evaluate.py
├── examples/
│ ├── basic_example.py
│ ├── basic_example_azuresearch.py
│ ├── basic_example_oracle.py
│ ├── basic_watsonx_example.py
│ ├── data/
│ │ ├── 2wikimultihopqa.json
│ │ └── 2wikimultihopqa_corpus.json
│ ├── load_and_crawl_using_docling.py
│ ├── load_local_file_using_unstructured.py
│ └── load_website_using_firecrawl.py
├── main.py
├── mkdocs.yml
├── pyproject.toml
└── tests/
├── __init__.py
├── agent/
│ ├── __init__.py
│ ├── test_base.py
│ ├── test_chain_of_rag.py
│ ├── test_collection_router.py
│ ├── test_deep_search.py
│ ├── test_naive_rag.py
│ └── test_rag_router.py
├── embedding/
│ ├── __init__.py
│ ├── test_base.py
│ ├── test_bedrock_embedding.py
│ ├── test_fastembed_embedding.py
│ ├── test_gemini_embedding.py
│ ├── test_glm_embedding.py
│ ├── test_jiekouai_embedding.py
│ ├── test_milvus_embedding.py
│ ├── test_novita_embedding.py
│ ├── test_ollama_embedding.py
│ ├── test_openai_embedding.py
│ ├── test_ppio_embedding.py
│ ├── test_sentence_transformer_embedding.py
│ ├── test_siliconflow_embedding.py
│ ├── test_volcengine_embedding.py
│ ├── test_voyage_embedding.py
│ └── test_watsonx_embedding.py
├── llm/
│ ├── __init__.py
│ ├── test_aliyun.py
│ ├── test_anthropic.py
│ ├── test_azure_openai.py
│ ├── test_base.py
│ ├── test_bedrock.py
│ ├── test_deepseek.py
│ ├── test_gemini.py
│ ├── test_glm.py
│ ├── test_jiekouai.py
│ ├── test_novita.py
│ ├── test_ollama.py
│ ├── test_openai.py
│ ├── test_ppio.py
│ ├── test_siliconflow.py
│ ├── test_together_ai.py
│ ├── test_volcengine.py
│ ├── test_watsonx.py
│ └── test_xai.py
├── loader/
│ ├── __init__.py
│ ├── file_loader/
│ │ ├── __init__.py
│ │ ├── test_base.py
│ │ ├── test_docling_loader.py
│ │ ├── test_json_loader.py
│ │ ├── test_pdf_loader.py
│ │ ├── test_text_loader.py
│ │ └── test_unstructured_loader.py
│ ├── test_splitter.py
│ └── web_crawler/
│ ├── __init__.py
│ ├── test_base.py
│ ├── test_crawl4ai_crawler.py
│ ├── test_docling_crawler.py
│ ├── test_firecrawl_crawler.py
│ └── test_jina_crawler.py
├── utils/
│ └── test_log.py
└── vector_db/
├── test_azure_search.py
├── test_base.py
├── test_milvus.py
├── test_oracle.py
└── test_qdrant.py
SYMBOL INDEX (962 symbols across 126 files)
FILE: deepsearcher/agent/base.py
function describe_class (line 7) | def describe_class(description):
class BaseAgent (line 28) | class BaseAgent(ABC):
method __init__ (line 36) | def __init__(self, **kwargs):
method invoke (line 45) | def invoke(self, query: str, **kwargs) -> Any:
class RAGAgent (line 58) | class RAGAgent(BaseAgent):
method __init__ (line 66) | def __init__(self, **kwargs):
method retrieve (line 75) | def retrieve(self, query: str, **kwargs) -> Tuple[List[RetrievalResult...
method query (line 90) | def query(self, query: str, **kwargs) -> Tuple[str, List[RetrievalResu...
FILE: deepsearcher/agent/chain_of_rag.py
class ChainOfRAG (line 77) | class ChainOfRAG(RAGAgent):
method __init__ (line 88) | def __init__(
method _reflect_get_subquery (line 122) | def _reflect_get_subquery(self, query: str, intermediate_context: List...
method _retrieve_and_answer (line 136) | def _retrieve_and_answer(self, query: str) -> Tuple[str, List[Retrieva...
method _get_supported_docs (line 172) | def _get_supported_docs(
method _check_has_enough_info (line 202) | def _check_has_enough_info(
method retrieve (line 222) | def retrieve(self, query: str, **kwargs) -> Tuple[List[RetrievalResult...
method query (line 277) | def query(self, query: str, **kwargs) -> Tuple[str, List[RetrievalResu...
method _format_retrieved_results (line 318) | def _format_retrieved_results(self, retrieved_results: List[RetrievalR...
FILE: deepsearcher/agent/collection_router.py
class CollectionRouter (line 18) | class CollectionRouter(BaseAgent):
method __init__ (line 26) | def __init__(self, llm: BaseLLM, vector_db: BaseVectorDB, dim: int, **...
method invoke (line 42) | def invoke(self, query: str, dim: int, **kwargs) -> Tuple[List[str], i...
FILE: deepsearcher/agent/deep_search.py
class DeepSearch (line 71) | class DeepSearch(RAGAgent):
method __init__ (line 79) | def __init__(
method _generate_sub_queries (line 111) | def _generate_sub_queries(self, original_query: str) -> Tuple[List[str...
method _search_chunks_from_vectordb (line 120) | async def _search_chunks_from_vectordb(self, query: str, sub_queries: ...
method _generate_gap_queries (line 173) | def _generate_gap_queries(
method retrieve (line 187) | def retrieve(self, original_query: str, **kwargs) -> Tuple[List[Retrie...
method async_retrieve (line 206) | async def async_retrieve(
method query (line 271) | def query(self, query: str, **kwargs) -> Tuple[str, List[RetrievalResu...
method _format_chunk_texts (line 315) | def _format_chunk_texts(self, chunk_texts: List[str]) -> str:
FILE: deepsearcher/agent/naive_rag.py
class NaiveRAG (line 19) | class NaiveRAG(RAGAgent):
method __init__ (line 27) | def __init__(
method retrieve (line 57) | def retrieve(self, query: str, **kwargs) -> Tuple[List[RetrievalResult...
method query (line 95) | def query(self, query: str, **kwargs) -> Tuple[str, List[RetrievalResu...
FILE: deepsearcher/agent/rag_router.py
class RAGRouter (line 21) | class RAGRouter(RAGAgent):
method __init__ (line 29) | def __init__(
method _route (line 56) | def _route(self, query: str) -> Tuple[RAGAgent, int]:
method retrieve (line 79) | def retrieve(self, query: str, **kwargs) -> Tuple[List[RetrievalResult...
method query (line 84) | def query(self, query: str, **kwargs) -> Tuple[str, List[RetrievalResu...
method find_last_digit (line 89) | def find_last_digit(self, string):
FILE: deepsearcher/cli.py
function main (line 18) | def main():
FILE: deepsearcher/configuration.py
class Configuration (line 20) | class Configuration:
method __init__ (line 29) | def __init__(self, config_path: str = DEFAULT_CONFIG_YAML_PATH):
method load_config_from_yaml (line 42) | def load_config_from_yaml(self, config_path: str):
method set_provider_config (line 55) | def set_provider_config(self, feature: FeatureType, provider: str, pro...
method get_provider_config (line 73) | def get_provider_config(self, feature: FeatureType):
class ModuleFactory (line 92) | class ModuleFactory:
method __init__ (line 100) | def __init__(self, config: Configuration):
method _create_module_instance (line 109) | def _create_module_instance(self, feature: FeatureType, module_name: s...
method create_llm (line 128) | def create_llm(self) -> BaseLLM:
method create_embedding (line 137) | def create_embedding(self) -> BaseEmbedding:
method create_file_loader (line 146) | def create_file_loader(self) -> BaseLoader:
method create_web_crawler (line 155) | def create_web_crawler(self) -> BaseCrawler:
method create_vector_db (line 164) | def create_vector_db(self) -> BaseVectorDB:
function init_config (line 186) | def init_config(config: Configuration):
FILE: deepsearcher/embedding/base.py
class BaseEmbedding (line 8) | class BaseEmbedding:
method embed_query (line 17) | def embed_query(self, text: str) -> List[float]:
method embed_documents (line 29) | def embed_documents(self, texts: List[str]) -> List[List[float]]:
method embed_chunks (line 44) | def embed_chunks(self, chunks: List[Chunk], batch_size: int = 256) -> ...
method dimension (line 69) | def dimension(self) -> int:
FILE: deepsearcher/embedding/bedrock_embedding.py
class BedrockEmbedding (line 24) | class BedrockEmbedding(BaseEmbedding):
method __init__ (line 32) | def __init__(self, model: str = DEFAULT_MODEL_ID, region_name: str = "...
method embed_query (line 74) | def embed_query(self, text: str) -> List[float]:
method embed_documents (line 91) | def embed_documents(self, texts: List[str]) -> List[List[float]]:
method dimension (line 106) | def dimension(self) -> int:
FILE: deepsearcher/embedding/fastembed_embdding.py
class FastEmbedEmbedding (line 7) | class FastEmbedEmbedding(BaseEmbedding):
method __init__ (line 13) | def __init__(self, model="BAAI/bge-small-en-v1.5", **kwargs):
method _ensure_model_loaded (line 34) | def _ensure_model_loaded(self):
method embed_query (line 43) | def embed_query(self, text: str) -> List[float]:
method embed_documents (line 58) | def embed_documents(self, texts: List[str]) -> List[List[float]]:
method dimension (line 74) | def dimension(self) -> int:
FILE: deepsearcher/embedding/gemini_embedding.py
class GeminiEmbedding (line 12) | class GeminiEmbedding(BaseEmbedding):
method __init__ (line 23) | def __init__(self, model: str = "text-embedding-004", **kwargs):
method embed_chunks (line 49) | def embed_chunks(self, chunks, batch_size: int = 100):
method _get_dim (line 53) | def _get_dim(self):
method _embed_content (line 62) | def _embed_content(self, texts: List[str]):
method embed_query (line 81) | def embed_query(self, text: str) -> List[float]:
method embed_documents (line 99) | def embed_documents(self, texts: List[str]) -> List[List[float]]:
method dimension (line 114) | def dimension(self) -> int:
FILE: deepsearcher/embedding/glm_embedding.py
class GLMEmbedding (line 11) | class GLMEmbedding(BaseEmbedding):
method __init__ (line 16) | def __init__(self, model: str = "embedding-3", **kwargs):
method embed_query (line 38) | def embed_query(self, text: str) -> List[float]:
method embed_documents (line 42) | def embed_documents(self, texts: List[str]) -> List[List[float]]:
method dimension (line 48) | def dimension(self) -> int:
FILE: deepsearcher/embedding/jiekouai_embedding.py
class JiekouAIEmbedding (line 18) | class JiekouAIEmbedding(BaseEmbedding):
method __init__ (line 26) | def __init__(self, model="qwen/qwen3-embedding-8b", batch_size=32, **k...
method embed_query (line 55) | def embed_query(self, text: str) -> List[float]:
method embed_documents (line 67) | def embed_documents(self, texts: List[str]) -> List[List[float]]:
method _embed_input (line 94) | def _embed_input(self, input: Union[str, List[str]]) -> List[List[floa...
method dimension (line 124) | def dimension(self) -> int:
FILE: deepsearcher/embedding/milvus_embedding.py
class MilvusEmbedding (line 21) | class MilvusEmbedding(BaseEmbedding):
method __init__ (line 30) | def __init__(self, model: str = None, **kwargs) -> None:
method embed_query (line 69) | def embed_query(self, text: str) -> List[float]:
method embed_documents (line 81) | def embed_documents(self, texts: List[str]) -> List[List[float]]:
method dimension (line 101) | def dimension(self) -> int:
FILE: deepsearcher/embedding/novita_embedding.py
class NovitaEmbedding (line 15) | class NovitaEmbedding(BaseEmbedding):
method __init__ (line 26) | def __init__(self, model="baai/bge-m3", batch_size=32, **kwargs):
method embed_query (line 54) | def embed_query(self, text: str) -> List[float]:
method embed_documents (line 69) | def embed_documents(self, texts: List[str]) -> List[List[float]]:
method _embed_input (line 96) | def _embed_input(self, input: Union[str, List[str]]) -> List[List[floa...
method dimension (line 121) | def dimension(self) -> int:
FILE: deepsearcher/embedding/ollama_embedding.py
class OllamaEmbedding (line 12) | class OllamaEmbedding(BaseEmbedding):
method __init__ (line 20) | def __init__(self, model="bge-m3", batch_size=32, **kwargs):
method embed_query (line 64) | def embed_query(self, text: str) -> List[float]:
method embed_documents (line 79) | def embed_documents(self, texts: List[str]) -> List[List[float]]:
method _embed_input (line 106) | def _embed_input(self, input: Union[str, List[str]]) -> List[List[floa...
method dimension (line 123) | def dimension(self) -> int:
FILE: deepsearcher/embedding/openai_embedding.py
class OpenAIEmbedding (line 15) | class OpenAIEmbedding(BaseEmbedding):
method __init__ (line 26) | def __init__(self, model: str = "text-embedding-ada-002", **kwargs):
method _get_dim (line 91) | def _get_dim(self):
method embed_query (line 101) | def embed_query(self, text: str) -> List[float]:
method embed_documents (line 123) | def embed_documents(self, texts: List[str]) -> List[List[float]]:
method dimension (line 146) | def dimension(self) -> int:
FILE: deepsearcher/embedding/ppio_embedding.py
class PPIOEmbedding (line 16) | class PPIOEmbedding(BaseEmbedding):
method __init__ (line 24) | def __init__(self, model="baai/bge-m3", batch_size=32, **kwargs):
method embed_query (line 53) | def embed_query(self, text: str) -> List[float]:
method embed_documents (line 65) | def embed_documents(self, texts: List[str]) -> List[List[float]]:
method _embed_input (line 92) | def _embed_input(self, input: Union[str, List[str]]) -> List[List[floa...
method dimension (line 122) | def dimension(self) -> int:
FILE: deepsearcher/embedding/sentence_transformer_embedding.py
class SentenceTransformerEmbedding (line 13) | class SentenceTransformerEmbedding(BaseEmbedding):
method __init__ (line 24) | def __init__(self, model="BAAI/bge-m3", batch_size=32, **kwargs):
method embed_query (line 47) | def embed_query(self, text: str) -> List[float]:
method embed_documents (line 62) | def embed_documents(self, texts: List[str]) -> List[List[float]]:
method _embed_input (line 89) | def _embed_input(self, input: Union[str, List[str]]) -> List[List[floa...
method dimension (line 109) | def dimension(self) -> int:
FILE: deepsearcher/embedding/siliconflow_embedding.py
class SiliconflowEmbedding (line 19) | class SiliconflowEmbedding(BaseEmbedding):
method __init__ (line 30) | def __init__(self, model="BAAI/bge-m3", batch_size=32, **kwargs):
method embed_query (line 58) | def embed_query(self, text: str) -> List[float]:
method embed_documents (line 73) | def embed_documents(self, texts: List[str]) -> List[List[float]]:
method _embed_input (line 100) | def _embed_input(self, input: Union[str, List[str]]) -> List[List[floa...
method dimension (line 127) | def dimension(self) -> int:
FILE: deepsearcher/embedding/volcengine_embedding.py
class VolcengineEmbedding (line 17) | class VolcengineEmbedding(BaseEmbedding):
method __init__ (line 28) | def __init__(self, model="doubao-embedding-large-text-240915", batch_s...
method embed_query (line 56) | def embed_query(self, text: str) -> List[float]:
method embed_documents (line 71) | def embed_documents(self, texts: List[str]) -> List[List[float]]:
method _embed_input (line 98) | def _embed_input(self, input: Union[str, List[str]]) -> List[List[floa...
method dimension (line 123) | def dimension(self) -> int:
FILE: deepsearcher/embedding/voyage_embedding.py
class VoyageEmbedding (line 13) | class VoyageEmbedding(BaseEmbedding):
method __init__ (line 24) | def __init__(self, model="voyage-3", **kwargs):
method embed_query (line 50) | def embed_query(self, text: str) -> List[float]:
method embed_documents (line 66) | def embed_documents(self, texts: List[str]) -> List[List[float]]:
method dimension (line 83) | def dimension(self) -> int:
FILE: deepsearcher/embedding/watsonx_embedding.py
class WatsonXEmbedding (line 25) | class WatsonXEmbedding(BaseEmbedding):
method __init__ (line 37) | def __init__(self, model: str = "ibm/slate-125m-english-rtrvr-v2", **k...
method _get_dim (line 116) | def _get_dim(self):
method _count_tokens (line 138) | def _count_tokens(self, text: str) -> int:
method _truncate_text (line 156) | def _truncate_text(self, text: str) -> str:
method embed_query (line 229) | def embed_query(self, text: str) -> List[float]:
method embed_documents (line 253) | def embed_documents(self, texts: List[str]) -> List[List[float]]:
method _embed_documents_individually (line 288) | def _embed_documents_individually(self, texts: List[str]) -> List[List...
method dimension (line 325) | def dimension(self) -> int:
FILE: deepsearcher/llm/aliyun.py
class Aliyun (line 7) | class Aliyun(BaseLLM):
method __init__ (line 21) | def __init__(self, model: str = "deepseek-r1", **kwargs):
method chat (line 47) | def chat(self, messages: List[Dict]) -> ChatResponse:
FILE: deepsearcher/llm/anthropic_llm.py
class Anthropic (line 7) | class Anthropic(BaseLLM):
method __init__ (line 20) | def __init__(self, model: str = "claude-sonnet-4-0", max_tokens: int =...
method chat (line 45) | def chat(self, messages: List[Dict]) -> ChatResponse:
FILE: deepsearcher/llm/azure_openai.py
class AzureOpenAI (line 6) | class AzureOpenAI(BaseLLM):
method __init__ (line 14) | def __init__(
method chat (line 48) | def chat(self, messages: List[Dict]) -> ChatResponse:
FILE: deepsearcher/llm/base.py
class ChatResponse (line 7) | class ChatResponse(ABC):
method __init__ (line 19) | def __init__(self, content: str, total_tokens: int) -> None:
method __repr__ (line 30) | def __repr__(self) -> str:
class BaseLLM (line 40) | class BaseLLM(ABC):
method __init__ (line 48) | def __init__(self):
method chat (line 54) | def chat(self, messages: List[Dict]) -> ChatResponse:
method literal_eval (line 68) | def literal_eval(response_content: str):
method remove_think (line 115) | def remove_think(response_content: str) -> str:
FILE: deepsearcher/llm/bedrock.py
class Bedrock (line 7) | class Bedrock(BaseLLM):
method __init__ (line 20) | def __init__(
method chat (line 56) | def chat(self, messages: List[Dict]) -> ChatResponse:
FILE: deepsearcher/llm/deepseek.py
class DeepSeek (line 7) | class DeepSeek(BaseLLM):
method __init__ (line 21) | def __init__(self, model: str = "deepseek-reasoner", **kwargs):
method chat (line 45) | def chat(self, messages: List[Dict]) -> ChatResponse:
FILE: deepsearcher/llm/gemini.py
class Gemini (line 7) | class Gemini(BaseLLM):
method __init__ (line 21) | def __init__(self, model: str = "gemini-2.0-flash", **kwargs):
method chat (line 39) | def chat(self, messages: List[Dict]) -> ChatResponse:
FILE: deepsearcher/llm/glm.py
class GLM (line 7) | class GLM(BaseLLM):
method __init__ (line 12) | def __init__(self, model: str = "glm-4-plus", **kwargs):
method chat (line 26) | def chat(self, messages: List[Dict]) -> ChatResponse:
FILE: deepsearcher/llm/jiekouai.py
class JiekouAI (line 7) | class JiekouAI(BaseLLM):
method __init__ (line 19) | def __init__(self, model: str = "claude-sonnet-4-5-20250929", **kwargs):
method chat (line 42) | def chat(self, messages: List[Dict]) -> ChatResponse:
FILE: deepsearcher/llm/novita.py
class Novita (line 7) | class Novita(BaseLLM):
method __init__ (line 12) | def __init__(self, model: str = "qwen/qwq-32b", **kwargs):
method chat (line 26) | def chat(self, messages: List[Dict]) -> ChatResponse:
FILE: deepsearcher/llm/ollama.py
class Ollama (line 6) | class Ollama(BaseLLM):
method __init__ (line 18) | def __init__(self, model: str = "qwq", **kwargs):
method chat (line 36) | def chat(self, messages: List[Dict]) -> ChatResponse:
FILE: deepsearcher/llm/openai_llm.py
class OpenAI (line 7) | class OpenAI(BaseLLM):
method __init__ (line 19) | def __init__(self, model: str = "o1-mini", **kwargs):
method chat (line 42) | def chat(self, messages: List[Dict]) -> ChatResponse:
FILE: deepsearcher/llm/ppio.py
class PPIO (line 7) | class PPIO(BaseLLM):
method __init__ (line 19) | def __init__(self, model: str = "deepseek/deepseek-r1-turbo", **kwargs):
method chat (line 42) | def chat(self, messages: List[Dict]) -> ChatResponse:
FILE: deepsearcher/llm/siliconflow.py
class SiliconFlow (line 7) | class SiliconFlow(BaseLLM):
method __init__ (line 21) | def __init__(self, model: str = "deepseek-ai/DeepSeek-R1", **kwargs):
method chat (line 44) | def chat(self, messages: List[Dict]) -> ChatResponse:
FILE: deepsearcher/llm/together_ai.py
class TogetherAI (line 7) | class TogetherAI(BaseLLM):
method __init__ (line 21) | def __init__(self, model: str = "deepseek-ai/DeepSeek-R1", **kwargs):
method chat (line 39) | def chat(self, messages: List[Dict]) -> ChatResponse:
FILE: deepsearcher/llm/volcengine.py
class Volcengine (line 7) | class Volcengine(BaseLLM):
method __init__ (line 21) | def __init__(self, model: str = "deepseek-r1-250120", **kwargs):
method chat (line 44) | def chat(self, messages: List[Dict]) -> ChatResponse:
FILE: deepsearcher/llm/watsonx.py
class WatsonX (line 16) | class WatsonX(BaseLLM):
method __init__ (line 27) | def __init__(self, model: str = "ibm/granite-3-3-8b-instruct", **kwargs):
method chat (line 96) | def chat(self, messages: List[Dict]) -> ChatResponse:
method _messages_to_prompt (line 126) | def _messages_to_prompt(self, messages: List[Dict]) -> str:
FILE: deepsearcher/llm/xai.py
class XAI (line 7) | class XAI(BaseLLM):
method __init__ (line 21) | def __init__(self, model: str = "grok-4", **kwargs):
method chat (line 44) | def chat(self, messages: List[Dict]) -> ChatResponse:
FILE: deepsearcher/loader/file_loader/base.py
class BaseLoader (line 8) | class BaseLoader(ABC):
method __init__ (line 16) | def __init__(self, **kwargs):
method load_file (line 25) | def load_file(self, file_path: str) -> List[Document]:
method load_directory (line 42) | def load_directory(self, directory: str) -> List[Document]:
method supported_file_types (line 63) | def supported_file_types(self) -> List[str]:
FILE: deepsearcher/loader/file_loader/docling_loader.py
class DoclingLoader (line 10) | class DoclingLoader(BaseLoader):
method __init__ (line 16) | def __init__(self):
method load_file (line 26) | def load_file(self, file_path: str) -> List[Document]:
method load_directory (line 68) | def load_directory(self, directory: str) -> List[Document]:
method supported_file_types (line 87) | def supported_file_types(self) -> List[str]:
FILE: deepsearcher/loader/file_loader/json_loader.py
class JsonFileLoader (line 9) | class JsonFileLoader(BaseLoader):
method __init__ (line 17) | def __init__(self, text_key: str):
method load_file (line 26) | def load_file(self, file_path: str) -> List[Document]:
method _read_json_file (line 48) | def _read_json_file(self, file_path: str) -> list[dict]:
method _read_jsonl_file (line 66) | def _read_jsonl_file(self, file_path: str) -> List[dict]:
method supported_file_types (line 87) | def supported_file_types(self) -> List[str]:
FILE: deepsearcher/loader/file_loader/pdf_loader.py
class PDFLoader (line 8) | class PDFLoader(BaseLoader):
method __init__ (line 16) | def __init__(self):
method load_file (line 22) | def load_file(self, file_path: str) -> List[Document]:
method supported_file_types (line 47) | def supported_file_types(self) -> List[str]:
FILE: deepsearcher/loader/file_loader/text_loader.py
class TextLoader (line 8) | class TextLoader(BaseLoader):
method __init__ (line 16) | def __init__(self):
method load_file (line 22) | def load_file(self, file_path: str) -> List[Document]:
method supported_file_types (line 36) | def supported_file_types(self) -> List[str]:
FILE: deepsearcher/loader/file_loader/unstructured_loader.py
class UnstructuredLoader (line 11) | class UnstructuredLoader(BaseLoader):
method __init__ (line 19) | def __init__(self):
method load_pipeline (line 30) | def load_pipeline(self, input_path: str) -> List[Document]:
method load_file (line 103) | def load_file(self, file_path: str) -> List[Document]:
method load_directory (line 115) | def load_directory(self, directory: str) -> List[Document]:
method supported_file_types (line 128) | def supported_file_types(self) -> List[str]:
FILE: deepsearcher/loader/splitter.py
class Chunk (line 10) | class Chunk:
method __init__ (line 24) | def __init__(
function _sentence_window_split (line 46) | def _sentence_window_split(
function split_docs_to_chunks (line 80) | def split_docs_to_chunks(
FILE: deepsearcher/loader/web_crawler/base.py
class BaseCrawler (line 7) | class BaseCrawler(ABC):
method __init__ (line 15) | def __init__(self, **kwargs):
method crawl_url (line 24) | def crawl_url(self, url: str, **crawl_kwargs) -> List[Document]:
method crawl_urls (line 41) | def crawl_urls(self, urls: List[str], **crawl_kwargs) -> List[Document]:
FILE: deepsearcher/loader/web_crawler/crawl4ai_crawler.py
class Crawl4AICrawler (line 10) | class Crawl4AICrawler(BaseCrawler):
method __init__ (line 19) | def __init__(self, **kwargs):
method _lazy_init (line 31) | def _lazy_init(self):
method _async_crawl (line 44) | async def _async_crawl(self, url: str) -> Document:
method crawl_url (line 76) | def crawl_url(self, url: str) -> List[Document]:
method _async_crawl_many (line 94) | async def _async_crawl_many(self, urls: List[str]) -> List[Document]:
method crawl_urls (line 124) | def crawl_urls(self, urls: List[str], **crawl_kwargs) -> List[Document]:
FILE: deepsearcher/loader/web_crawler/docling_crawler.py
class DoclingCrawler (line 9) | class DoclingCrawler(BaseCrawler):
method __init__ (line 17) | def __init__(self, **kwargs):
method crawl_url (line 31) | def crawl_url(self, url: str, **crawl_kwargs) -> List[Document]:
method supported_file_types (line 65) | def supported_file_types(self) -> List[str]:
FILE: deepsearcher/loader/web_crawler/firecrawl_crawler.py
class FireCrawlCrawler (line 10) | class FireCrawlCrawler(BaseCrawler):
method __init__ (line 19) | def __init__(self, **kwargs):
method crawl_url (line 29) | def crawl_url(
FILE: deepsearcher/loader/web_crawler/jina_crawler.py
class JinaCrawler (line 10) | class JinaCrawler(BaseCrawler):
method __init__ (line 18) | def __init__(self, **kwargs):
method crawl_url (line 33) | def crawl_url(self, url: str) -> List[Document]:
FILE: deepsearcher/offline_loading.py
function load_from_local_files (line 11) | def load_from_local_files(
function load_from_website (line 72) | def load_from_website(
FILE: deepsearcher/online_query.py
function query (line 8) | def query(original_query: str, max_iter: int = 3) -> Tuple[str, List[Ret...
function retrieve (line 29) | def retrieve(
function naive_retrieve (line 55) | def naive_retrieve(query: str, collection: str = None, top_k=10) -> List...
function naive_rag_query (line 75) | def naive_rag_query(
FILE: deepsearcher/utils/log.py
class ColoredFormatter (line 6) | class ColoredFormatter(logging.Formatter):
method format (line 25) | def format(self, record):
function set_dev_mode (line 70) | def set_dev_mode(mode: bool):
function set_level (line 84) | def set_level(level):
function debug (line 94) | def debug(message):
function info (line 105) | def info(message):
function warning (line 116) | def warning(message):
function error (line 127) | def error(message):
function critical (line 138) | def critical(message):
function color_print (line 152) | def color_print(message, **kwargs):
FILE: deepsearcher/vector_db/azure_search.py
class AzureSearch (line 7) | class AzureSearch(BaseVectorDB):
method __init__ (line 8) | def __init__(self, endpoint, index_name, api_key, vector_field):
method init_collection (line 23) | def init_collection(self):
method insert_data (line 66) | def insert_data(self, documents: List[dict]):
method search_data (line 90) | def search_data(
method clear_db (line 204) | def clear_db(self):
method get_all_collections (line 223) | def get_all_collections(self) -> List[str]:
method get_collection_info (line 237) | def get_collection_info(self, name: str) -> Dict[str, Any]:
method collection_exists (line 247) | def collection_exists(self, name: str) -> bool:
method list_collections (line 257) | def list_collections(self, *args, **kwargs) -> List[CollectionInfo]:
FILE: deepsearcher/vector_db/base.py
class RetrievalResult (line 9) | class RetrievalResult:
method __init__ (line 24) | def __init__(
method __repr__ (line 48) | def __repr__(self):
function deduplicate_results (line 58) | def deduplicate_results(results: List[RetrievalResult]) -> List[Retrieva...
class CollectionInfo (line 80) | class CollectionInfo:
method __init__ (line 91) | def __init__(self, collection_name: str, description: str):
class BaseVectorDB (line 103) | class BaseVectorDB(ABC):
method __init__ (line 115) | def __init__(
method init_collection (line 132) | def init_collection(
method insert_data (line 155) | def insert_data(self, collection: str, chunks: List[Chunk], *args, **k...
method search_data (line 168) | def search_data(
method list_collections (line 185) | def list_collections(self, *args, **kwargs) -> List[CollectionInfo]:
method clear_db (line 199) | def clear_db(self, *args, **kwargs):
FILE: deepsearcher/vector_db/milvus.py
class Milvus (line 11) | class Milvus(BaseVectorDB):
method __init__ (line 16) | def __init__(
method init_collection (line 48) | def init_collection(
method insert_data (line 139) | def insert_data(
method search_data (line 182) | def search_data(
method list_collections (line 253) | def list_collections(self, *args, **kwargs) -> List[CollectionInfo]:
method clear_db (line 291) | def clear_db(self, collection: str = "deepsearcher", *args, **kwargs):
FILE: deepsearcher/vector_db/oracle.py
class OracleDB (line 12) | class OracleDB(BaseVectorDB):
method __init__ (line 17) | def __init__(
method numpy_converter_in (line 72) | def numpy_converter_in(self, value):
method input_type_handler (line 82) | def input_type_handler(self, cursor, value, arraysize):
method numpy_converter_out (line 91) | def numpy_converter_out(self, value):
method output_type_handler (line 101) | def output_type_handler(self, cursor, metadata):
method query (line 110) | def query(self, sql: str, params: dict = None) -> Union[dict, None]:
method execute (line 149) | def execute(self, sql: str, data: Union[list, dict] = None):
method has_collection (line 178) | def has_collection(self, collection: str = "deepsearcher"):
method check_table (line 199) | def check_table(self):
method create_tables (line 217) | def create_tables(self, table_name):
method drop_collection (line 235) | def drop_collection(self, collection: str = "deepsearcher"):
method insertone (line 257) | def insertone(self, data):
method searchone (line 268) | def searchone(
method init_collection (line 313) | def init_collection(
method insert_data (line 359) | def insert_data(
method search_data (line 404) | def search_data(
method list_collections (line 451) | def list_collections(self, *args, **kwargs) -> List[CollectionInfo]:
method clear_db (line 480) | def clear_db(self, collection: str = "deepsearcher", *args, **kwargs):
FILE: deepsearcher/vector_db/qdrant.py
class Qdrant (line 17) | class Qdrant(BaseVectorDB):
method __init__ (line 20) | def __init__(
method init_collection (line 114) | def init_collection(
method insert_data (line 163) | def insert_data(
method search_data (line 206) | def search_data(
method list_collections (line 250) | def list_collections(self, *args, **kwargs) -> List[CollectionInfo]:
method clear_db (line 278) | def clear_db(self, collection: Optional[str] = None, *args, **kwargs):
FILE: evaluation/evaluate.py
function _deepsearch_retrieve_titles (line 34) | def _deepsearch_retrieve_titles(
function _naive_retrieve_titles (line 77) | def _naive_retrieve_titles(question: str) -> List[str]:
function _calcu_recall (line 94) | def _calcu_recall(sample, retrieved_titles, dataset) -> dict:
function _print_recall_line (line 124) | def _print_recall_line(recall: dict, pre_str="", post_str="\n"):
function evaluate (line 139) | def evaluate(
function main_eval (line 263) | def main_eval():
FILE: examples/basic_watsonx_example.py
function main (line 11) | def main():
FILE: examples/load_and_crawl_using_docling.py
function main (line 10) | def main():
FILE: examples/load_local_file_using_unstructured.py
function main (line 15) | def main():
FILE: examples/load_website_using_firecrawl.py
function main (line 15) | def main():
FILE: main.py
class ProviderConfigRequest (line 20) | class ProviderConfigRequest(BaseModel):
function set_provider_config (line 36) | def set_provider_config(request: ProviderConfigRequest):
function load_files (line 62) | def load_files(
function load_website (line 112) | def load_website(
function perform_query (line 162) | def perform_query(
FILE: tests/agent/test_base.py
class MockLLM (line 10) | class MockLLM(BaseLLM):
method __init__ (line 13) | def __init__(self, predefined_responses=None):
method chat (line 24) | def chat(self, messages, **kwargs):
method literal_eval (line 37) | def literal_eval(self, text):
class MockEmbedding (line 52) | class MockEmbedding(BaseEmbedding):
method __init__ (line 55) | def __init__(self, dimension=8):
method dimension (line 60) | def dimension(self):
method embed_query (line 64) | def embed_query(self, text):
method embed_documents (line 68) | def embed_documents(self, documents):
class MockVectorDB (line 73) | class MockVectorDB(BaseVectorDB):
method __init__ (line 76) | def __init__(self, collections=None):
method search_data (line 98) | def search_data(self, collection, vector, top_k=10, **kwargs):
method insert_data (line 115) | def insert_data(self, collection, chunks):
method init_collection (line 122) | def init_collection(self, dim, collection, **kwargs):
method list_collections (line 126) | def list_collections(self, dim=None):
method clear_db (line 130) | def clear_db(self, collection):
class BaseAgentTest (line 135) | class BaseAgentTest(unittest.TestCase):
method setUp (line 138) | def setUp(self):
FILE: tests/agent/test_chain_of_rag.py
class TestChainOfRAG (line 10) | class TestChainOfRAG(BaseAgentTest):
method setUp (line 13) | def setUp(self):
method test_init (line 37) | def test_init(self):
method test_reflect_get_subquery (line 47) | def test_reflect_get_subquery(self):
method test_retrieve_and_answer (line 64) | def test_retrieve_and_answer(self):
method test_get_supported_docs (line 87) | def test_get_supported_docs(self):
method test_check_has_enough_info (line 110) | def test_check_has_enough_info(self):
method test_retrieve (line 129) | def test_retrieve(self):
method test_query (line 166) | def test_query(self):
method test_format_retrieved_results (line 202) | def test_format_retrieved_results(self):
FILE: tests/agent/test_collection_router.py
class TestCollectionRouter (line 10) | class TestCollectionRouter(BaseAgentTest):
method setUp (line 13) | def setUp(self):
method test_init (line 35) | def test_init(self):
method test_invoke_with_multiple_collections (line 44) | def test_invoke_with_multiple_collections(self):
method test_invoke_with_empty_response (line 69) | def test_invoke_with_empty_response(self):
method test_invoke_with_no_collections (line 88) | def test_invoke_with_no_collections(self):
method test_invoke_with_single_collection (line 104) | def test_invoke_with_single_collection(self):
method test_invoke_with_no_description (line 125) | def test_invoke_with_no_description(self):
FILE: tests/agent/test_deep_search.py
class TestDeepSearch (line 10) | class TestDeepSearch(BaseAgentTest):
method setUp (line 13) | def setUp(self):
method test_init (line 34) | def test_init(self):
method test_generate_sub_queries (line 43) | def test_generate_sub_queries(self):
method test_search_chunks_from_vectordb (line 56) | def test_search_chunks_from_vectordb(self):
method test_generate_gap_queries (line 78) | def test_generate_gap_queries(self):
method test_retrieve (line 103) | def test_retrieve(self):
method test_async_retrieve (line 138) | def test_async_retrieve(self):
method test_query (line 173) | def test_query(self):
method test_query_no_results (line 203) | def test_query_no_results(self):
method test_format_chunk_texts (line 219) | def test_format_chunk_texts(self):
FILE: tests/agent/test_naive_rag.py
class TestNaiveRAG (line 9) | class TestNaiveRAG(BaseAgentTest):
method setUp (line 12) | def setUp(self):
method test_init (line 24) | def test_init(self):
method test_retrieve (line 33) | def test_retrieve(self):
method test_retrieve_without_routing (line 55) | def test_retrieve_without_routing(self):
method test_query (line 73) | def test_query(self):
method test_with_window_splitter_disabled (line 104) | def test_with_window_splitter_disabled(self):
FILE: tests/agent/test_rag_router.py
class TestRAGRouter (line 11) | class TestRAGRouter(BaseAgentTest):
method setUp (line 14) | def setUp(self):
method test_init (line 32) | def test_init(self):
method test_route (line 40) | def test_route(self):
method test_route_with_non_numeric_response (line 54) | def test_route_with_non_numeric_response(self):
method test_retrieve (line 68) | def test_retrieve(self):
method test_query (line 95) | def test_query(self):
method test_find_last_digit (line 122) | def test_find_last_digit(self):
method test_auto_description_fallback (line 132) | def test_auto_description_fallback(self):
FILE: tests/embedding/test_base.py
class ConcreteEmbedding (line 9) | class ConcreteEmbedding(BaseEmbedding):
method __init__ (line 12) | def __init__(self, dimension=768):
method embed_query (line 15) | def embed_query(self, text: str) -> List[float]:
method dimension (line 20) | def dimension(self) -> int:
class TestBaseEmbedding (line 24) | class TestBaseEmbedding(unittest.TestCase):
method test_embed_query (line 28) | def test_embed_query(self):
method test_embed_documents (line 36) | def test_embed_documents(self):
method test_embed_chunks (line 52) | def test_embed_chunks(self, mock_tqdm):
method test_dimension_property (line 94) | def test_dimension_property(self):
FILE: tests/embedding/test_bedrock_embedding.py
class TestBedrockEmbedding (line 18) | class TestBedrockEmbedding(unittest.TestCase):
method setUp (line 21) | def setUp(self):
method tearDown (line 40) | def tearDown(self):
method test_init_default (line 45) | def test_init_default(self):
method test_init_with_credentials (line 68) | def test_init_with_credentials(self):
method test_init_with_different_models (line 79) | def test_init_with_different_models(self):
method test_embed_query (line 90) | def test_embed_query(self):
method test_embed_documents (line 109) | def test_embed_documents(self):
method test_dimension_property (line 132) | def test_dimension_property(self):
FILE: tests/embedding/test_fastembed_embedding.py
class TestFastEmbedEmbedding (line 12) | class TestFastEmbedEmbedding(unittest.TestCase):
method setUp (line 15) | def setUp(self):
method tearDown (line 31) | def tearDown(self):
method test_init_default (line 36) | def test_init_default(self):
method test_init_with_custom_model (line 50) | def test_init_with_custom_model(self):
method test_init_with_kwargs (line 63) | def test_init_with_kwargs(self):
method test_embed_query (line 77) | def test_embed_query(self):
method test_embed_documents (line 93) | def test_embed_documents(self):
method test_dimension_property (line 111) | def test_dimension_property(self):
method test_lazy_loading (line 127) | def test_lazy_loading(self):
FILE: tests/embedding/test_gemini_embedding.py
function _resource_tracker (line 15) | def _resource_tracker():
class TestGeminiEmbedding (line 23) | class TestGeminiEmbedding(unittest.TestCase):
method setUp (line 26) | def setUp(self):
method tearDown (line 48) | def tearDown(self):
method test_init_default (line 53) | def test_init_default(self):
method test_init_with_api_key_from_env (line 67) | def test_init_with_api_key_from_env(self):
method test_init_with_api_key_parameter (line 73) | def test_init_with_api_key_parameter(self):
method test_init_with_custom_model (line 80) | def test_init_with_custom_model(self):
method test_init_with_custom_dimension (line 90) | def test_init_with_custom_dimension(self):
method test_embed_query_single_char (line 99) | def test_embed_query_single_char(self):
method test_embed_query_multi_char (line 120) | def test_embed_query_multi_char(self):
method test_embed_documents (line 141) | def test_embed_documents(self):
method test_embed_chunks (line 175) | def test_embed_chunks(self):
method test_dimension_property_different_models (line 215) | def test_dimension_property_different_models(self):
method test_get_dim_method (line 232) | def test_get_dim_method(self):
method test_embed_content_method (line 245) | def test_embed_content_method(self):
FILE: tests/embedding/test_glm_embedding.py
class TestGLMEmbedding (line 8) | class TestGLMEmbedding(unittest.TestCase):
method setUp (line 11) | def setUp(self):
method tearDown (line 33) | def tearDown(self):
method test_init_default (line 38) | def test_init_default(self):
method test_init_with_api_key (line 54) | def test_init_with_api_key(self):
method test_init_with_base_url (line 66) | def test_init_with_base_url(self):
method test_embed_query (line 78) | def test_embed_query(self):
method test_embed_documents (line 99) | def test_embed_documents(self):
method test_dimension_property (line 133) | def test_dimension_property(self):
FILE: tests/embedding/test_jiekouai_embedding.py
class TestJiekouAIEmbedding (line 8) | class TestJiekouAIEmbedding(unittest.TestCase):
method setUp (line 11) | def setUp(self):
method tearDown (line 27) | def tearDown(self):
method test_init_default (line 32) | def test_init_default(self):
method test_init_with_model (line 43) | def test_init_with_model(self):
method test_init_with_model_name (line 53) | def test_init_with_model_name(self):
method test_init_with_api_key (line 62) | def test_init_with_api_key(self):
method test_init_without_api_key (line 71) | def test_init_without_api_key(self):
method test_embed_query (line 77) | def test_embed_query(self):
method test_embed_documents (line 106) | def test_embed_documents(self):
method test_embed_documents_with_batching (line 145) | def test_embed_documents_with_batching(self):
method test_dimension_property (line 180) | def test_dimension_property(self):
FILE: tests/embedding/test_milvus_embedding.py
class TestMilvusEmbedding (line 8) | class TestMilvusEmbedding(unittest.TestCase):
method setUp (line 11) | def setUp(self):
method tearDown (line 40) | def tearDown(self):
method test_init_default (line 45) | def test_init_default(self):
method test_init_with_jina_model (line 54) | def test_init_with_jina_model(self):
method test_init_with_bge_model (line 63) | def test_init_with_bge_model(self):
method test_init_with_invalid_model (line 72) | def test_init_with_invalid_model(self):
method test_embed_query (line 78) | def test_embed_query(self):
method test_embed_documents (line 93) | def test_embed_documents(self):
method test_dimension_property (line 114) | def test_dimension_property(self):
FILE: tests/embedding/test_novita_embedding.py
class TestNovitaEmbedding (line 9) | class TestNovitaEmbedding(unittest.TestCase):
method setUp (line 12) | def setUp(self):
method tearDown (line 28) | def tearDown(self):
method test_init_default (line 33) | def test_init_default(self):
method test_init_with_model (line 44) | def test_init_with_model(self):
method test_init_with_model_name (line 54) | def test_init_with_model_name(self):
method test_init_with_api_key (line 63) | def test_init_with_api_key(self):
method test_init_without_api_key (line 72) | def test_init_without_api_key(self):
method test_embed_query (line 78) | def test_embed_query(self):
method test_embed_documents (line 108) | def test_embed_documents(self):
method test_embed_documents_with_batching (line 148) | def test_embed_documents_with_batching(self):
method test_dimension_property (line 183) | def test_dimension_property(self):
FILE: tests/embedding/test_ollama_embedding.py
class TestOllamaEmbedding (line 8) | class TestOllamaEmbedding(unittest.TestCase):
method setUp (line 11) | def setUp(self):
method tearDown (line 31) | def tearDown(self):
method test_init_default (line 36) | def test_init_default(self):
method test_init_with_base_url (line 50) | def test_init_with_base_url(self):
method test_init_with_model_name (line 62) | def test_init_with_model_name(self):
method test_init_with_dimension (line 76) | def test_init_with_dimension(self):
method test_embed_query (line 88) | def test_embed_query(self):
method test_embed_documents_small_batch (line 107) | def test_embed_documents_small_batch(self):
method test_embed_documents_large_batch (line 136) | def test_embed_documents_large_batch(self):
method test_embed_documents_no_batching (line 177) | def test_embed_documents_no_batching(self):
method test_dimension_property (line 215) | def test_dimension_property(self):
FILE: tests/embedding/test_openai_embedding.py
class TestOpenAIEmbedding (line 9) | class TestOpenAIEmbedding(unittest.TestCase):
method setUp (line 12) | def setUp(self):
method tearDown (line 33) | def tearDown(self):
method test_init_default (line 38) | def test_init_default(self):
method test_init_with_model (line 52) | def test_init_with_model(self):
method test_init_with_model_name (line 62) | def test_init_with_model_name(self):
method test_init_with_dimension (line 71) | def test_init_with_dimension(self):
method test_init_with_api_key (line 80) | def test_init_with_api_key(self):
method test_init_with_base_url (line 89) | def test_init_with_base_url(self):
method test_init_with_azure (line 99) | def test_init_with_azure(self, mock_azure_openai):
method test_init_with_azure_deployment (line 127) | def test_init_with_azure_deployment(self, mock_azure_openai):
method test_get_dim (line 143) | def test_get_dim(self):
method test_embed_query (line 156) | def test_embed_query(self):
method test_embed_query_azure (line 178) | def test_embed_query_azure(self):
method test_embed_documents (line 219) | def test_embed_documents(self):
method test_dimension_property (line 254) | def test_dimension_property(self):
FILE: tests/embedding/test_ppio_embedding.py
class TestPPIOEmbedding (line 9) | class TestPPIOEmbedding(unittest.TestCase):
method setUp (line 12) | def setUp(self):
method tearDown (line 28) | def tearDown(self):
method test_init_default (line 33) | def test_init_default(self):
method test_init_with_model (line 44) | def test_init_with_model(self):
method test_init_with_model_name (line 54) | def test_init_with_model_name(self):
method test_init_with_api_key (line 63) | def test_init_with_api_key(self):
method test_init_without_api_key (line 72) | def test_init_without_api_key(self):
method test_embed_query (line 78) | def test_embed_query(self):
method test_embed_documents (line 107) | def test_embed_documents(self):
method test_embed_documents_with_batching (line 146) | def test_embed_documents_with_batching(self):
method test_dimension_property (line 181) | def test_dimension_property(self):
FILE: tests/embedding/test_sentence_transformer_embedding.py
class TestSentenceTransformerEmbedding (line 12) | class TestSentenceTransformerEmbedding(unittest.TestCase):
method setUp (line 15) | def setUp(self):
method tearDown (line 37) | def tearDown(self):
method test_init (line 42) | def test_init(self):
method test_embed_query (line 69) | def test_embed_query(self):
method test_embed_documents_small_batch (line 90) | def test_embed_documents_small_batch(self):
method test_embed_documents_large_batch (line 120) | def test_embed_documents_large_batch(self):
method test_embed_documents_no_batching (line 159) | def test_embed_documents_no_batching(self):
method test_dimension_property (line 194) | def test_dimension_property(self):
FILE: tests/embedding/test_siliconflow_embedding.py
class TestSiliconflowEmbedding (line 9) | class TestSiliconflowEmbedding(unittest.TestCase):
method setUp (line 12) | def setUp(self):
method tearDown (line 28) | def tearDown(self):
method test_init_default (line 33) | def test_init_default(self):
method test_init_with_model (line 44) | def test_init_with_model(self):
method test_init_with_model_name (line 54) | def test_init_with_model_name(self):
method test_init_with_api_key (line 63) | def test_init_with_api_key(self):
method test_init_without_api_key (line 72) | def test_init_without_api_key(self):
method test_embed_query (line 78) | def test_embed_query(self):
method test_embed_documents (line 108) | def test_embed_documents(self):
method test_embed_documents_with_batching (line 148) | def test_embed_documents_with_batching(self):
method test_dimension_property (line 183) | def test_dimension_property(self):
FILE: tests/embedding/test_volcengine_embedding.py
class TestVolcengineEmbedding (line 9) | class TestVolcengineEmbedding(unittest.TestCase):
method setUp (line 12) | def setUp(self):
method tearDown (line 28) | def tearDown(self):
method test_init_default (line 33) | def test_init_default(self):
method test_init_with_model (line 44) | def test_init_with_model(self):
method test_init_with_model_name (line 54) | def test_init_with_model_name(self):
method test_init_with_api_key (line 63) | def test_init_with_api_key(self):
method test_init_without_api_key (line 72) | def test_init_without_api_key(self):
method test_embed_query (line 78) | def test_embed_query(self):
method test_embed_documents (line 108) | def test_embed_documents(self):
method test_embed_documents_with_batching (line 148) | def test_embed_documents_with_batching(self):
method test_dimension_property (line 183) | def test_dimension_property(self):
FILE: tests/embedding/test_voyage_embedding.py
class TestVoyageEmbedding (line 8) | class TestVoyageEmbedding(unittest.TestCase):
method setUp (line 11) | def setUp(self):
method tearDown (line 29) | def tearDown(self):
method test_init_default (line 34) | def test_init_default(self):
method test_init_with_model (line 47) | def test_init_with_model(self):
method test_init_with_model_name (line 57) | def test_init_with_model_name(self):
method test_init_with_api_key (line 66) | def test_init_with_api_key(self):
method test_embed_query (line 75) | def test_embed_query(self):
method test_embed_documents (line 97) | def test_embed_documents(self):
method test_dimension_property (line 126) | def test_dimension_property(self):
FILE: tests/embedding/test_watsonx_embedding.py
class TestWatsonXEmbedding (line 5) | class TestWatsonXEmbedding(unittest.TestCase):
method setUp (line 8) | def setUp(self):
method test_init_with_env_vars (line 31) | def test_init_with_env_vars(self, mock_credentials_class, mock_embeddi...
method test_init_with_space_id (line 66) | def test_init_with_space_id(self, mock_credentials_class, mock_embeddi...
method test_init_missing_api_key (line 87) | def test_init_missing_api_key(self, mock_credentials_class, mock_embed...
method test_init_missing_url (line 102) | def test_init_missing_url(self, mock_credentials_class, mock_embedding...
method test_init_missing_project_and_space_id (line 117) | def test_init_missing_project_and_space_id(self, mock_credentials_clas...
method test_embed_query (line 133) | def test_embed_query(self, mock_credentials_class, mock_embeddings_cla...
method test_embed_documents (line 166) | def test_embed_documents(self, mock_credentials_class, mock_embeddings...
method test_dimension_property (line 206) | def test_dimension_property(self, mock_credentials_class, mock_embeddi...
method test_embed_query_error_handling (line 235) | def test_embed_query_error_handling(self, mock_credentials_class, mock...
method test_embed_documents_error_handling (line 262) | def test_embed_documents_error_handling(self, mock_credentials_class, ...
FILE: tests/llm/test_aliyun.py
class TestAliyun (line 13) | class TestAliyun(unittest.TestCase):
method setUp (line 16) | def setUp(self):
method tearDown (line 47) | def tearDown(self):
method test_init_default (line 51) | def test_init_default(self):
method test_init_with_api_key_from_env (line 65) | def test_init_with_api_key_from_env(self):
method test_init_with_api_key_parameter (line 75) | def test_init_with_api_key_parameter(self):
method test_init_with_custom_model (line 85) | def test_init_with_custom_model(self):
method test_init_with_custom_base_url (line 92) | def test_init_with_custom_base_url(self):
method test_chat_single_message (line 102) | def test_chat_single_message(self):
method test_chat_multiple_messages (line 122) | def test_chat_multiple_messages(self):
method test_chat_with_error (line 147) | def test_chat_with_error(self):
FILE: tests/llm/test_anthropic.py
class ContentItem (line 14) | class ContentItem:
method __init__ (line 16) | def __init__(self, text: str):
class TestAnthropic (line 20) | class TestAnthropic(unittest.TestCase):
method setUp (line 23) | def setUp(self):
method tearDown (line 46) | def tearDown(self):
method test_init_default (line 50) | def test_init_default(self):
method test_init_with_api_key_from_env (line 65) | def test_init_with_api_key_from_env(self):
method test_init_with_api_key_parameter (line 75) | def test_init_with_api_key_parameter(self):
method test_init_with_custom_model_and_tokens (line 85) | def test_init_with_custom_model_and_tokens(self):
method test_init_with_custom_base_url (line 94) | def test_init_with_custom_base_url(self):
method test_chat_single_message (line 105) | def test_chat_single_message(self):
method test_chat_multiple_messages (line 126) | def test_chat_multiple_messages(self):
method test_chat_with_error (line 152) | def test_chat_with_error(self):
FILE: tests/llm/test_azure_openai.py
class TestAzureOpenAI (line 13) | class TestAzureOpenAI(unittest.TestCase):
method setUp (line 16) | def setUp(self):
method tearDown (line 53) | def tearDown(self):
method test_init_with_parameters (line 57) | def test_init_with_parameters(self):
method test_init_with_env_variables (line 77) | def test_init_with_env_variables(self):
method test_chat_single_message (line 93) | def test_chat_single_message(self):
method test_chat_multiple_messages (line 118) | def test_chat_multiple_messages(self):
method test_chat_with_error (line 148) | def test_chat_with_error(self):
FILE: tests/llm/test_base.py
class TestBaseLLM (line 6) | class TestBaseLLM(unittest.TestCase):
method setUp (line 9) | def setUp(self):
method tearDown (line 15) | def tearDown(self):
method test_chat_response_init (line 19) | def test_chat_response_init(self):
method test_literal_eval_python_code_block (line 32) | def test_literal_eval_python_code_block(self):
method test_literal_eval_json_code_block (line 40) | def test_literal_eval_json_code_block(self):
method test_literal_eval_str_code_block (line 48) | def test_literal_eval_str_code_block(self):
method test_literal_eval_plain_code_block (line 56) | def test_literal_eval_plain_code_block(self):
method test_literal_eval_raw_dict (line 64) | def test_literal_eval_raw_dict(self):
method test_literal_eval_raw_list (line 70) | def test_literal_eval_raw_list(self):
method test_literal_eval_with_whitespace (line 76) | def test_literal_eval_with_whitespace(self):
method test_literal_eval_nested_structures (line 86) | def test_literal_eval_nested_structures(self):
method test_literal_eval_invalid_format (line 107) | def test_literal_eval_invalid_format(self):
method test_remove_think_with_tags (line 119) | def test_remove_think_with_tags(self):
method test_remove_think_without_tags (line 129) | def test_remove_think_without_tags(self):
method test_remove_think_multiple_tags (line 135) | def test_remove_think_multiple_tags(self):
method test_remove_think_empty_tags (line 146) | def test_remove_think_empty_tags(self):
FILE: tests/llm/test_bedrock.py
class TestBedrock (line 13) | class TestBedrock(unittest.TestCase):
method setUp (line 16) | def setUp(self):
method tearDown (line 42) | def tearDown(self):
method test_init_default (line 46) | def test_init_default(self):
method test_init_with_aws_credentials_from_env (line 64) | def test_init_with_aws_credentials_from_env(self):
method test_init_with_aws_credentials_parameters (line 81) | def test_init_with_aws_credentials_parameters(self):
method test_init_with_custom_model_and_tokens (line 97) | def test_init_with_custom_model_and_tokens(self):
method test_init_with_custom_region (line 104) | def test_init_with_custom_region(self):
method test_chat_single_message (line 116) | def test_chat_single_message(self):
method test_chat_multiple_messages (line 138) | def test_chat_multiple_messages(self):
method test_chat_with_error (line 163) | def test_chat_with_error(self):
method test_chat_with_preformatted_messages (line 177) | def test_chat_with_preformatted_messages(self):
FILE: tests/llm/test_deepseek.py
class TestDeepSeek (line 13) | class TestDeepSeek(unittest.TestCase):
method setUp (line 16) | def setUp(self):
method tearDown (line 47) | def tearDown(self):
method test_init_default (line 51) | def test_init_default(self):
method test_init_with_api_key_from_env (line 65) | def test_init_with_api_key_from_env(self):
method test_init_with_api_key_parameter (line 79) | def test_init_with_api_key_parameter(self):
method test_init_with_custom_model (line 89) | def test_init_with_custom_model(self):
method test_init_with_custom_base_url (line 96) | def test_init_with_custom_base_url(self):
method test_chat_single_message (line 107) | def test_chat_single_message(self):
method test_chat_multiple_messages (line 127) | def test_chat_multiple_messages(self):
method test_chat_with_error (line 152) | def test_chat_with_error(self):
FILE: tests/llm/test_gemini.py
class TestGemini (line 13) | class TestGemini(unittest.TestCase):
method setUp (line 16) | def setUp(self):
method tearDown (line 37) | def tearDown(self):
method test_init_default (line 41) | def test_init_default(self):
method test_init_with_api_key_from_env (line 52) | def test_init_with_api_key_from_env(self):
method test_init_with_api_key_parameter (line 59) | def test_init_with_api_key_parameter(self):
method test_init_with_custom_model (line 66) | def test_init_with_custom_model(self):
method test_chat_single_message (line 73) | def test_chat_single_message(self):
method test_chat_multiple_messages (line 93) | def test_chat_multiple_messages(self):
method test_chat_with_error (line 119) | def test_chat_with_error(self):
FILE: tests/llm/test_glm.py
class TestGLM (line 13) | class TestGLM(unittest.TestCase):
method setUp (line 16) | def setUp(self):
method tearDown (line 47) | def tearDown(self):
method test_init_default (line 51) | def test_init_default(self):
method test_init_with_api_key_from_env (line 65) | def test_init_with_api_key_from_env(self):
method test_init_with_api_key_parameter (line 75) | def test_init_with_api_key_parameter(self):
method test_init_with_custom_model (line 85) | def test_init_with_custom_model(self):
method test_init_with_custom_base_url (line 92) | def test_init_with_custom_base_url(self):
method test_chat_single_message (line 103) | def test_chat_single_message(self):
method test_chat_multiple_messages (line 123) | def test_chat_multiple_messages(self):
method test_chat_with_error (line 148) | def test_chat_with_error(self):
FILE: tests/llm/test_jiekouai.py
class TestJiekouAI (line 13) | class TestJiekouAI(unittest.TestCase):
method setUp (line 16) | def setUp(self):
method tearDown (line 47) | def tearDown(self):
method test_init_default (line 51) | def test_init_default(self):
method test_init_with_api_key_from_env (line 65) | def test_init_with_api_key_from_env(self):
method test_init_with_api_key_parameter (line 75) | def test_init_with_api_key_parameter(self):
method test_init_with_custom_model (line 85) | def test_init_with_custom_model(self):
method test_init_with_custom_base_url (line 92) | def test_init_with_custom_base_url(self):
method test_chat_single_message (line 103) | def test_chat_single_message(self):
method test_chat_multiple_messages (line 123) | def test_chat_multiple_messages(self):
method test_chat_with_error (line 148) | def test_chat_with_error(self):
FILE: tests/llm/test_novita.py
class TestNovita (line 13) | class TestNovita(unittest.TestCase):
method setUp (line 16) | def setUp(self):
method tearDown (line 47) | def tearDown(self):
method test_init_default (line 51) | def test_init_default(self):
method test_init_with_api_key_from_env (line 65) | def test_init_with_api_key_from_env(self):
method test_init_with_api_key_parameter (line 75) | def test_init_with_api_key_parameter(self):
method test_init_with_custom_model (line 85) | def test_init_with_custom_model(self):
method test_init_with_custom_base_url (line 92) | def test_init_with_custom_base_url(self):
method test_chat_single_message (line 103) | def test_chat_single_message(self):
method test_chat_multiple_messages (line 123) | def test_chat_multiple_messages(self):
method test_chat_with_error (line 148) | def test_chat_with_error(self):
FILE: tests/llm/test_ollama.py
class TestOllama (line 13) | class TestOllama(unittest.TestCase):
method setUp (line 16) | def setUp(self):
method tearDown (line 40) | def tearDown(self):
method test_init_default (line 44) | def test_init_default(self):
method test_init_with_custom_model (line 57) | def test_init_with_custom_model(self):
method test_init_with_custom_base_url (line 64) | def test_init_with_custom_base_url(self):
method test_chat_single_message (line 74) | def test_chat_single_message(self):
method test_chat_multiple_messages (line 94) | def test_chat_multiple_messages(self):
method test_chat_with_error (line 119) | def test_chat_with_error(self):
FILE: tests/llm/test_openai.py
class TestOpenAI (line 13) | class TestOpenAI(unittest.TestCase):
method setUp (line 16) | def setUp(self):
method tearDown (line 47) | def tearDown(self):
method test_init_default (line 51) | def test_init_default(self):
method test_init_with_api_key_from_env (line 65) | def test_init_with_api_key_from_env(self):
method test_init_with_api_key_parameter (line 79) | def test_init_with_api_key_parameter(self):
method test_init_with_custom_model (line 88) | def test_init_with_custom_model(self):
method test_init_with_custom_base_url (line 94) | def test_init_with_custom_base_url(self):
method test_chat_single_message (line 105) | def test_chat_single_message(self):
method test_chat_multiple_messages (line 125) | def test_chat_multiple_messages(self):
method test_chat_with_error (line 150) | def test_chat_with_error(self):
FILE: tests/llm/test_ppio.py
class TestPPIO (line 13) | class TestPPIO(unittest.TestCase):
method setUp (line 16) | def setUp(self):
method tearDown (line 47) | def tearDown(self):
method test_init_default (line 51) | def test_init_default(self):
method test_init_with_api_key_from_env (line 65) | def test_init_with_api_key_from_env(self):
method test_init_with_api_key_parameter (line 75) | def test_init_with_api_key_parameter(self):
method test_init_with_custom_model (line 85) | def test_init_with_custom_model(self):
method test_init_with_custom_base_url (line 92) | def test_init_with_custom_base_url(self):
method test_chat_single_message (line 103) | def test_chat_single_message(self):
method test_chat_multiple_messages (line 123) | def test_chat_multiple_messages(self):
method test_chat_with_error (line 148) | def test_chat_with_error(self):
FILE: tests/llm/test_siliconflow.py
class TestSiliconFlow (line 13) | class TestSiliconFlow(unittest.TestCase):
method setUp (line 16) | def setUp(self):
method tearDown (line 47) | def tearDown(self):
method test_init_default (line 51) | def test_init_default(self):
method test_init_with_api_key_from_env (line 65) | def test_init_with_api_key_from_env(self):
method test_init_with_api_key_parameter (line 75) | def test_init_with_api_key_parameter(self):
method test_init_with_custom_model (line 85) | def test_init_with_custom_model(self):
method test_init_with_custom_base_url (line 92) | def test_init_with_custom_base_url(self):
method test_chat_single_message (line 103) | def test_chat_single_message(self):
method test_chat_multiple_messages (line 123) | def test_chat_multiple_messages(self):
method test_chat_with_error (line 148) | def test_chat_with_error(self):
FILE: tests/llm/test_together_ai.py
class TestTogetherAI (line 13) | class TestTogetherAI(unittest.TestCase):
method setUp (line 16) | def setUp(self):
method tearDown (line 47) | def tearDown(self):
method test_init_default (line 51) | def test_init_default(self):
method test_init_with_api_key_from_env (line 64) | def test_init_with_api_key_from_env(self):
method test_init_with_api_key_parameter (line 73) | def test_init_with_api_key_parameter(self):
method test_init_with_custom_model (line 82) | def test_init_with_custom_model(self):
method test_chat_single_message (line 89) | def test_chat_single_message(self):
method test_chat_multiple_messages (line 109) | def test_chat_multiple_messages(self):
method test_chat_with_error (line 134) | def test_chat_with_error(self):
FILE: tests/llm/test_volcengine.py
class TestVolcengine (line 13) | class TestVolcengine(unittest.TestCase):
method setUp (line 16) | def setUp(self):
method tearDown (line 47) | def tearDown(self):
method test_init_default (line 51) | def test_init_default(self):
method test_init_with_api_key_from_env (line 65) | def test_init_with_api_key_from_env(self):
method test_init_with_api_key_parameter (line 75) | def test_init_with_api_key_parameter(self):
method test_init_with_custom_model (line 85) | def test_init_with_custom_model(self):
method test_init_with_custom_base_url (line 92) | def test_init_with_custom_base_url(self):
method test_chat_single_message (line 103) | def test_chat_single_message(self):
method test_chat_multiple_messages (line 123) | def test_chat_multiple_messages(self):
method test_chat_with_error (line 148) | def test_chat_with_error(self):
FILE: tests/llm/test_watsonx.py
class TestWatsonX (line 5) | class TestWatsonX(unittest.TestCase):
method setUp (line 8) | def setUp(self):
method test_init_with_env_vars (line 20) | def test_init_with_env_vars(self, mock_gen_text_params_class, mock_cre...
method test_init_with_space_id (line 61) | def test_init_with_space_id(self, mock_gen_text_params_class, mock_cre...
method test_init_with_custom_model (line 94) | def test_init_with_custom_model(self, mock_gen_text_params_class, mock...
method test_init_with_custom_params (line 129) | def test_init_with_custom_params(self, mock_gen_text_params_class, moc...
method test_init_missing_api_key (line 164) | def test_init_missing_api_key(self, mock_gen_text_params_class, mock_c...
method test_init_missing_url (line 180) | def test_init_missing_url(self, mock_gen_text_params_class, mock_crede...
method test_init_missing_project_and_space_id (line 196) | def test_init_missing_project_and_space_id(self, mock_gen_text_params_...
method test_chat_simple_message (line 213) | def test_chat_simple_message(self, mock_gen_text_params_class, mock_cr...
method test_chat_with_system_message (line 259) | def test_chat_with_system_message(self, mock_gen_text_params_class, mo...
method test_chat_conversation_history (line 301) | def test_chat_conversation_history(self, mock_gen_text_params_class, m...
method test_chat_error_handling (line 349) | def test_chat_error_handling(self, mock_gen_text_params_class, mock_cr...
method test_messages_to_prompt (line 384) | def test_messages_to_prompt(self, mock_gen_text_params_class, mock_cre...
FILE: tests/llm/test_xai.py
class TestXAI (line 13) | class TestXAI(unittest.TestCase):
method setUp (line 16) | def setUp(self):
method tearDown (line 47) | def tearDown(self):
method test_init_default (line 51) | def test_init_default(self):
method test_init_with_api_key_from_env (line 65) | def test_init_with_api_key_from_env(self):
method test_init_with_api_key_parameter (line 75) | def test_init_with_api_key_parameter(self):
method test_init_with_custom_model (line 85) | def test_init_with_custom_model(self):
method test_init_with_custom_base_url (line 92) | def test_init_with_custom_base_url(self):
method test_chat_single_message (line 103) | def test_chat_single_message(self):
method test_chat_multiple_messages (line 123) | def test_chat_multiple_messages(self):
method test_chat_with_error (line 148) | def test_chat_with_error(self):
FILE: tests/loader/file_loader/test_base.py
class TestBaseLoader (line 10) | class TestBaseLoader(unittest.TestCase):
method test_abstract_methods (line 13) | def test_abstract_methods(self):
method test_load_directory (line 20) | def test_load_directory(self):
FILE: tests/loader/file_loader/test_docling_loader.py
class TestDoclingLoader (line 11) | class TestDoclingLoader(unittest.TestCase):
method setUp (line 14) | def setUp(self):
method tearDown (line 64) | def tearDown(self):
method test_init (line 69) | def test_init(self):
method test_supported_file_types (line 79) | def test_supported_file_types(self):
method test_load_file (line 88) | def test_load_file(self):
method test_load_file_not_found (line 124) | def test_load_file_not_found(self):
method test_load_unsupported_file_type (line 130) | def test_load_unsupported_file_type(self):
method test_load_file_error (line 135) | def test_load_file_error(self):
method test_load_directory (line 144) | def test_load_directory(self):
method test_load_not_a_directory (line 178) | def test_load_not_a_directory(self):
FILE: tests/loader/file_loader/test_json_loader.py
class TestJsonFileLoader (line 11) | class TestJsonFileLoader(unittest.TestCase):
method setUp (line 14) | def setUp(self):
method tearDown (line 62) | def tearDown(self):
method test_load_json_file (line 66) | def test_load_json_file(self):
method test_load_jsonl_file (line 84) | def test_load_jsonl_file(self):
method test_invalid_json_file (line 102) | def test_invalid_json_file(self):
method test_invalid_jsonl_file (line 107) | def test_invalid_jsonl_file(self):
method test_supported_file_types (line 115) | def test_supported_file_types(self):
FILE: tests/loader/file_loader/test_pdf_loader.py
class TestPDFLoader (line 11) | class TestPDFLoader(unittest.TestCase):
method setUp (line 14) | def setUp(self):
method tearDown (line 35) | def tearDown(self):
method test_supported_file_types (line 39) | def test_supported_file_types(self):
method test_load_text_file (line 47) | def test_load_text_file(self):
method test_load_markdown_file (line 61) | def test_load_markdown_file(self):
method test_load_pdf_file (line 76) | def test_load_pdf_file(self, mock_pdf_open):
method test_load_directory (line 114) | def test_load_directory(self):
FILE: tests/loader/file_loader/test_text_loader.py
class TestTextLoader (line 8) | class TestTextLoader(unittest.TestCase):
method setUp (line 11) | def setUp(self):
method tearDown (line 24) | def tearDown(self):
method test_supported_file_types (line 28) | def test_supported_file_types(self):
method test_load_file (line 35) | def test_load_file(self):
method test_load_directory (line 51) | def test_load_directory(self):
FILE: tests/loader/file_loader/test_unstructured_loader.py
class TestUnstructuredLoader (line 12) | class TestUnstructuredLoader(unittest.TestCase):
method setUp (line 15) | def setUp(self):
method tearDown (line 75) | def tearDown(self):
method test_init (line 84) | def test_init(self):
method test_supported_file_types (line 88) | def test_supported_file_types(self):
method test_load_file (line 101) | def test_load_file(self, mock_listdir):
method test_load_directory (line 123) | def test_load_directory(self, mock_listdir):
method test_load_with_api (line 141) | def test_load_with_api(self, mock_listdir):
method test_empty_output (line 175) | def test_empty_output(self, mock_listdir):
method test_error_reading_json (line 187) | def test_error_reading_json(self, mock_listdir):
FILE: tests/loader/test_splitter.py
class TestSplitter (line 7) | class TestSplitter(unittest.TestCase):
method test_chunk_init (line 10) | def test_chunk_init(self):
method test_sentence_window_split (line 28) | def test_sentence_window_split(self):
method test_split_docs_to_chunks (line 62) | def test_split_docs_to_chunks(self):
FILE: tests/loader/web_crawler/test_base.py
class TestBaseCrawler (line 7) | class TestBaseCrawler(unittest.TestCase):
method test_abstract_methods (line 10) | def test_abstract_methods(self):
method test_crawl_urls (line 16) | def test_crawl_urls(self):
FILE: tests/loader/web_crawler/test_crawl4ai_crawler.py
class TestCrawl4AICrawler (line 11) | class TestCrawl4AICrawler(unittest.TestCase):
method setUp (line 14) | def setUp(self):
method tearDown (line 48) | def tearDown(self):
method test_init (line 52) | def test_init(self):
method test_lazy_init (line 60) | def test_lazy_init(self):
method test_crawl_url (line 75) | def test_crawl_url(self, mock_asyncio_run):
method test_crawl_url_error (line 99) | def test_crawl_url_error(self, mock_asyncio_run):
method test_crawl_urls (line 113) | def test_crawl_urls(self, mock_asyncio_run):
method test_crawl_urls_error (line 142) | def test_crawl_urls_error(self, mock_asyncio_run):
FILE: tests/loader/web_crawler/test_docling_crawler.py
class TestDoclingCrawler (line 9) | class TestDoclingCrawler(unittest.TestCase):
method setUp (line 12) | def setUp(self):
method tearDown (line 42) | def tearDown(self):
method test_init (line 46) | def test_init(self):
method test_crawl_url (line 56) | def test_crawl_url(self):
method test_crawl_url_error (line 94) | def test_crawl_url_error(self):
method test_supported_file_types (line 105) | def test_supported_file_types(self):
method test_crawl_urls (line 121) | def test_crawl_urls(self):
FILE: tests/loader/web_crawler/test_firecrawl_crawler.py
class TestFireCrawlCrawler (line 10) | class TestFireCrawlCrawler(unittest.TestCase):
method setUp (line 13) | def setUp(self):
method tearDown (line 30) | def tearDown(self):
method test_init (line 35) | def test_init(self):
method test_crawl_url_single_page (line 39) | def test_crawl_url_single_page(self):
method test_crawl_url_multiple_pages (line 67) | def test_crawl_url_multiple_pages(self):
method test_crawl_url_with_default_params (line 115) | def test_crawl_url_with_default_params(self):
FILE: tests/loader/web_crawler/test_jina_crawler.py
class TestJinaCrawler (line 11) | class TestJinaCrawler(unittest.TestCase):
method test_init_with_token (line 15) | def test_init_with_token(self):
method test_init_with_alternative_key (line 21) | def test_init_with_alternative_key(self):
method test_init_without_token (line 27) | def test_init_without_token(self):
method test_crawl_url (line 34) | def test_crawl_url(self, mock_get):
method test_crawl_url_http_error (line 71) | def test_crawl_url_http_error(self, mock_get):
method test_crawl_urls (line 85) | def test_crawl_urls(self, mock_get):
FILE: tests/utils/test_log.py
class TestColoredFormatter (line 10) | class TestColoredFormatter(unittest.TestCase):
method setUp (line 13) | def setUp(self):
method test_format_debug (line 17) | def test_format_debug(self):
method test_format_info (line 26) | def test_format_info(self):
method test_format_warning (line 35) | def test_format_warning(self):
method test_format_error (line 44) | def test_format_error(self):
method test_format_critical (line 53) | def test_format_critical(self):
method test_format_unknown_level (line 62) | def test_format_unknown_level(self):
class TestLogFunctions (line 73) | class TestLogFunctions(unittest.TestCase):
method setUp (line 76) | def setUp(self):
method tearDown (line 91) | def tearDown(self):
method test_set_dev_mode (line 96) | def test_set_dev_mode(self):
method test_set_level (line 104) | def test_set_level(self):
method test_debug_in_dev_mode (line 109) | def test_debug_in_dev_mode(self):
method test_debug_not_in_dev_mode (line 115) | def test_debug_not_in_dev_mode(self):
method test_info_in_dev_mode (line 121) | def test_info_in_dev_mode(self):
method test_info_not_in_dev_mode (line 127) | def test_info_not_in_dev_mode(self):
method test_warning_in_dev_mode (line 133) | def test_warning_in_dev_mode(self):
method test_warning_not_in_dev_mode (line 139) | def test_warning_not_in_dev_mode(self):
method test_error_in_dev_mode (line 145) | def test_error_in_dev_mode(self):
method test_error_not_in_dev_mode (line 151) | def test_error_not_in_dev_mode(self):
method test_critical (line 157) | def test_critical(self):
method test_color_print (line 165) | def test_color_print(self):
FILE: tests/vector_db/test_azure_search.py
class TestAzureSearch (line 10) | class TestAzureSearch(unittest.TestCase):
method setUp (line 13) | def setUp(self):
method tearDown (line 60) | def tearDown(self):
method test_init (line 64) | def test_init(self):
method test_init_collection (line 84) | def test_init_collection(self):
method test_insert_data (line 101) | def test_insert_data(self):
method test_search_data (line 139) | def test_search_data(self):
method test_clear_db (line 185) | def test_clear_db(self):
method test_list_collections (line 207) | def test_list_collections(self):
FILE: tests/vector_db/test_base.py
class TestRetrievalResult (line 14) | class TestRetrievalResult(unittest.TestCase):
method setUp (line 17) | def setUp(self):
method test_init (line 25) | def test_init(self):
method test_init_default_score (line 41) | def test_init_default_score(self):
method test_repr (line 51) | def test_repr(self):
class TestDeduplicateResults (line 64) | class TestDeduplicateResults(unittest.TestCase):
method setUp (line 67) | def setUp(self):
method test_no_duplicates (line 76) | def test_no_duplicates(self):
method test_with_duplicates (line 86) | def test_with_duplicates(self):
method test_empty_list (line 98) | def test_empty_list(self):
class TestCollectionInfo (line 105) | class TestCollectionInfo(unittest.TestCase):
method test_init (line 108) | def test_init(self):
class MockVectorDB (line 118) | class MockVectorDB(BaseVectorDB):
method init_collection (line 121) | def init_collection(self, dim, collection, description, force_new_coll...
method insert_data (line 124) | def insert_data(self, collection, chunks, *args, **kwargs):
method search_data (line 127) | def search_data(self, collection, vector, *args, **kwargs) -> List[Ret...
method clear_db (line 130) | def clear_db(self, *args, **kwargs):
class TestBaseVectorDB (line 134) | class TestBaseVectorDB(unittest.TestCase):
method setUp (line 137) | def setUp(self):
method test_init_default (line 141) | def test_init_default(self):
method test_init_custom_collection (line 145) | def test_init_custom_collection(self):
method test_list_collections_default (line 151) | def test_list_collections_default(self):
FILE: tests/vector_db/test_milvus.py
class TestMilvus (line 14) | class TestMilvus(unittest.TestCase):
method test_init (line 17) | def test_init(self):
method test_init_collection (line 30) | def test_init_collection(self):
method test_insert_data_with_retrieval_results (line 47) | def test_insert_data_with_retrieval_results(self):
method test_search_data (line 81) | def test_search_data(self):
method test_clear_collection (line 108) | def test_clear_collection(self):
method test_list_collections (line 123) | def test_list_collections(self):
FILE: tests/vector_db/test_oracle.py
class TestOracleDB (line 12) | class TestOracleDB(unittest.TestCase):
method setUp (line 15) | def setUp(self):
method tearDown (line 36) | def tearDown(self):
method test_init (line 40) | def test_init(self):
method test_insert_data (line 62) | def test_insert_data(self):
method test_search_data (line 104) | def test_search_data(self):
method test_list_collections (line 159) | def test_list_collections(self):
method test_clear_db (line 190) | def test_clear_db(self):
method test_has_collection (line 214) | def test_has_collection(self):
FILE: tests/vector_db/test_qdrant.py
class TestQdrant (line 6) | class TestQdrant(unittest.TestCase):
method setUp (line 9) | def setUp(self):
method tearDown (line 32) | def tearDown(self):
method test_init (line 37) | def test_init(self, mock_client_class):
method test_init_collection (line 55) | def test_init_collection(self, mock_client_class):
method test_insert_data (line 77) | def test_insert_data(self, mock_client_class):
method test_search_data (line 116) | def test_search_data(self, mock_client_class):
method test_clear_collection (line 172) | def test_clear_collection(self, mock_client_class):
Copy disabled (too large)
Download .json
Condensed preview — 199 files, each showing path, character count, and a content snippet. Download the .json file for the full structured content (10,895K chars).
[
{
"path": ".github/ISSUE_TEMPLATE/bug_report.md",
"chars": 857,
"preview": "---\nname: Bug report\nabout: Create a report to help us improve\ntitle: ''\nlabels: ''\nassignees: ''\n\n---\n\nPlease describe "
},
{
"path": ".github/ISSUE_TEMPLATE/feature_request.md",
"chars": 644,
"preview": "---\nname: Feature request\nabout: Suggest an idea for this project\ntitle: ''\nlabels: ''\nassignees: ''\n\n---\n\nPlease descri"
},
{
"path": ".github/mergify.yml",
"chars": 937,
"preview": "misc:\n - branch: &BRANCHES\n # In this pull request, the changes are based on the main branch\n - &MASTER_BRAN"
},
{
"path": ".github/workflows/cd-docs.yml",
"chars": 365,
"preview": "name: \"Run Docs CD with UV\"\n\non:\n push:\n branches:\n - \"main\"\n - \"master\"\n paths:\n - 'docs/**'\n "
},
{
"path": ".github/workflows/ci-docs.yml",
"chars": 535,
"preview": "name: \"Run Docs CI with UV\"\n\non:\n pull_request:\n types: [opened, reopened, synchronize]\n paths:\n - 'docs/**'"
},
{
"path": ".github/workflows/docs.yml",
"chars": 710,
"preview": "on:\n workflow_call:\n inputs:\n deploy:\n type: boolean\n description: \"I"
},
{
"path": ".github/workflows/release.yml",
"chars": 715,
"preview": "#git tag v0.x.x # Must be same as the version in pyproject.toml\n#git push --tags\n\nname: Publish Python Package to PyPI\n"
},
{
"path": ".github/workflows/ruff.yml",
"chars": 506,
"preview": "name: Ruff\non: \n push:\n branches: [ main, master ]\n pull_request:\njobs:\n build:\n runs-on: ubuntu-latest\n ste"
},
{
"path": ".gitignore",
"chars": 3892,
"preview": "# Created by https://www.toptal.com/developers/gitignore/api/python,visualstudiocode\n# Edit at https://www.toptal.com/de"
},
{
"path": ".python-version",
"chars": 5,
"preview": "3.10\n"
},
{
"path": ".vscode/settings.json",
"chars": 214,
"preview": "{\n \"python.testing.unittestArgs\": [\n \"-v\",\n \"-s\",\n \"./tests\",\n \"-p\",\n \"test_*.py\"\n"
},
{
"path": "CONTRIBUTING.md",
"chars": 5317,
"preview": "# Contributing to DeepSearcher\n\nWe welcome contributions from everyone. This document provides guidelines to make the co"
},
{
"path": "Dockerfile",
"chars": 427,
"preview": "FROM ghcr.io/astral-sh/uv:python3.10-bookworm-slim\n\nWORKDIR /app\n\nRUN mkdir -p /tmp/uv-cache /app/data /app/logs\n\nCOPY p"
},
{
"path": "LICENSE",
"chars": 11336,
"preview": " Apache License\n Version 2.0, January 2004\n "
},
{
"path": "MAINTAINERS",
"chars": 55,
"preview": "maintainers:\n - xiaofan-luan\n - SimFG\n - zc277584121"
},
{
"path": "Makefile",
"chars": 106,
"preview": "lint:\n\tuv run ruff format --diff\n\tuv run ruff check\n\nformat:\n\tuv run ruff format\n\tuv run ruff check --fix\n"
},
{
"path": "OWNERS",
"chars": 87,
"preview": "filters:\n \".*\":\n reviewers:\n - maintainers\n approvers:\n - maintainers\n"
},
{
"path": "OWNERS_ALIASES",
"chars": 73,
"preview": "aliases:\n maintainers:\n - xiaofan-luan\n - SimFG\n - zc277584121\n"
},
{
"path": "README.md",
"chars": 32661,
"preview": "\n\n<div align=\"center\">\n \n[![License](https://img.shields.io/badge/License-Apache%"
},
{
"path": "deepsearcher/__init__.py",
"chars": 248,
"preview": "import os\n\n# ignore the warnings\n# None of PyTorch, TensorFlow >= 2.0, or Flax have been found. Models won't be availabl"
},
{
"path": "deepsearcher/agent/__init__.py",
"chars": 243,
"preview": "from .base import BaseAgent, RAGAgent\nfrom .chain_of_rag import ChainOfRAG\nfrom .deep_search import DeepSearch\nfrom .nai"
},
{
"path": "deepsearcher/agent/base.py",
"chars": 2671,
"preview": "from abc import ABC\nfrom typing import Any, List, Tuple\n\nfrom deepsearcher.vector_db import RetrievalResult\n\n\ndef descri"
},
{
"path": "deepsearcher/agent/chain_of_rag.py",
"chars": 14095,
"preview": "from typing import List, Tuple\n\nfrom deepsearcher.agent.base import RAGAgent, describe_class\nfrom deepsearcher.agent.col"
},
{
"path": "deepsearcher/agent/collection_router.py",
"chars": 4252,
"preview": "from typing import List, Tuple\n\nfrom deepsearcher.agent.base import BaseAgent\nfrom deepsearcher.llm.base import BaseLLM\n"
},
{
"path": "deepsearcher/agent/deep_search.py",
"chars": 13732,
"preview": "import asyncio\nfrom typing import List, Tuple\n\nfrom deepsearcher.agent.base import RAGAgent, describe_class\nfrom deepsea"
},
{
"path": "deepsearcher/agent/naive_rag.py",
"chars": 5234,
"preview": "from typing import List, Tuple\n\nfrom deepsearcher.agent.base import RAGAgent\nfrom deepsearcher.agent.collection_router i"
},
{
"path": "deepsearcher/agent/rag_router.py",
"chars": 3836,
"preview": "from typing import List, Optional, Tuple\n\nfrom deepsearcher.agent import RAGAgent\nfrom deepsearcher.llm.base import Base"
},
{
"path": "deepsearcher/cli.py",
"chars": 4188,
"preview": "import argparse\nimport logging\nimport sys\nimport warnings\n\nfrom deepsearcher.configuration import Configuration, init_co"
},
{
"path": "deepsearcher/config.yaml",
"chars": 5081,
"preview": "provide_settings:\n llm:\n provider: \"OpenAI\"\n config:\n model: \"o1-mini\"\n# api_key: \"sk-xxxx\" # Uncommen"
},
{
"path": "deepsearcher/configuration.py",
"chars": 7956,
"preview": "import os\nfrom typing import Literal\n\nimport yaml\n\nfrom deepsearcher.agent import ChainOfRAG, DeepSearch, NaiveRAG\nfrom "
},
{
"path": "deepsearcher/embedding/__init__.py",
"chars": 1122,
"preview": "from .bedrock_embedding import BedrockEmbedding\nfrom .fastembed_embdding import FastEmbedEmbedding\nfrom .gemini_embeddin"
},
{
"path": "deepsearcher/embedding/base.py",
"chars": 2346,
"preview": "from typing import List\n\nfrom tqdm import tqdm\n\nfrom deepsearcher.loader.splitter import Chunk\n\n\nclass BaseEmbedding:\n "
},
{
"path": "deepsearcher/embedding/bedrock_embedding.py",
"chars": 4133,
"preview": "import json\nimport os\nfrom typing import List\n\nfrom deepsearcher.embedding.base import BaseEmbedding\n\nMODEL_ID_TITAN_TEX"
},
{
"path": "deepsearcher/embedding/fastembed_embdding.py",
"chars": 2483,
"preview": "from functools import cached_property\nfrom typing import List\n\nfrom deepsearcher.embedding.base import BaseEmbedding\n\n\nc"
},
{
"path": "deepsearcher/embedding/gemini_embedding.py",
"chars": 3774,
"preview": "import os\nfrom typing import List\n\nfrom deepsearcher.embedding.base import BaseEmbedding\n\nGEMINI_MODEL_DIM_MAP = {\n \""
},
{
"path": "deepsearcher/embedding/glm_embedding.py",
"chars": 1475,
"preview": "import os\nfrom typing import List\n\nfrom deepsearcher.embedding.base import BaseEmbedding\n\nGLM_MODEL_DIM_MAP = {\n \"emb"
},
{
"path": "deepsearcher/embedding/jiekouai_embedding.py",
"chars": 4590,
"preview": "import os\nfrom typing import List, Union\n\nimport requests\n\nfrom deepsearcher.embedding.base import BaseEmbedding\n\n# TODO"
},
{
"path": "deepsearcher/embedding/milvus_embedding.py",
"chars": 3880,
"preview": "from typing import List\n\nimport numpy as np\n\nfrom deepsearcher.embedding.base import BaseEmbedding\n\nMILVUS_MODEL_DIM_MAP"
},
{
"path": "deepsearcher/embedding/novita_embedding.py",
"chars": 4503,
"preview": "import os\nfrom typing import List, Union\n\nimport requests\n\nfrom deepsearcher.embedding.base import BaseEmbedding\n\nNOVITA"
},
{
"path": "deepsearcher/embedding/ollama_embedding.py",
"chars": 4478,
"preview": "from typing import List, Union\n\nfrom deepsearcher.embedding.base import BaseEmbedding\n\nOLLAMA_MODEL_DIM_MAP = {\n \"bge"
},
{
"path": "deepsearcher/embedding/openai_embedding.py",
"chars": 5572,
"preview": "import os\nfrom typing import List\n\nfrom openai._types import NOT_GIVEN\n\nfrom deepsearcher.embedding.base import BaseEmbe"
},
{
"path": "deepsearcher/embedding/ppio_embedding.py",
"chars": 4439,
"preview": "import os\nfrom typing import List, Union\n\nimport requests\n\nfrom deepsearcher.embedding.base import BaseEmbedding\n\n# TODO"
},
{
"path": "deepsearcher/embedding/sentence_transformer_embedding.py",
"chars": 3922,
"preview": "from functools import cached_property\nfrom typing import List, Union\n\nfrom deepsearcher.embedding.base import BaseEmbedd"
},
{
"path": "deepsearcher/embedding/siliconflow_embedding.py",
"chars": 4739,
"preview": "import os\nfrom typing import List, Union\n\nimport requests\n\nfrom deepsearcher.embedding.base import BaseEmbedding\n\nSILICO"
},
{
"path": "deepsearcher/embedding/volcengine_embedding.py",
"chars": 4702,
"preview": "import os\nfrom typing import List, Union\n\nimport requests\n\nfrom deepsearcher.embedding.base import BaseEmbedding\n\nVOLCEN"
},
{
"path": "deepsearcher/embedding/voyage_embedding.py",
"chars": 2853,
"preview": "import os\nfrom typing import List\n\nfrom deepsearcher.embedding.base import BaseEmbedding\n\nVOYAGE_MODEL_DIM_MAP = {\n \""
},
{
"path": "deepsearcher/embedding/watsonx_embedding.py",
"chars": 13017,
"preview": "import logging\nimport os\nfrom typing import List\n\nfrom deepsearcher.embedding.base import BaseEmbedding\n\ntry:\n from i"
},
{
"path": "deepsearcher/llm/__init__.py",
"chars": 779,
"preview": "from .aliyun import Aliyun\nfrom .anthropic_llm import Anthropic\nfrom .azure_openai import AzureOpenAI\nfrom .bedrock impo"
},
{
"path": "deepsearcher/llm/aliyun.py",
"chars": 2355,
"preview": "import os\nfrom typing import Dict, List\n\nfrom deepsearcher.llm.base import BaseLLM, ChatResponse\n\n\nclass Aliyun(BaseLLM)"
},
{
"path": "deepsearcher/llm/anthropic_llm.py",
"chars": 2479,
"preview": "import os\nfrom typing import Dict, List\n\nfrom deepsearcher.llm.base import BaseLLM, ChatResponse\n\n\nclass Anthropic(BaseL"
},
{
"path": "deepsearcher/llm/azure_openai.py",
"chars": 2222,
"preview": "from typing import Dict, List\n\nfrom deepsearcher.llm.base import BaseLLM, ChatResponse\n\n\nclass AzureOpenAI(BaseLLM):\n "
},
{
"path": "deepsearcher/llm/base.py",
"chars": 4078,
"preview": "import ast\nimport re\nfrom abc import ABC\nfrom typing import Dict, List\n\n\nclass ChatResponse(ABC):\n \"\"\"\n Represents"
},
{
"path": "deepsearcher/llm/bedrock.py",
"chars": 3553,
"preview": "import os\nfrom typing import Dict, List\n\nfrom deepsearcher.llm.base import BaseLLM, ChatResponse\n\n\nclass Bedrock(BaseLLM"
},
{
"path": "deepsearcher/llm/deepseek.py",
"chars": 2434,
"preview": "import os\nfrom typing import Dict, List\n\nfrom deepsearcher.llm.base import BaseLLM, ChatResponse\n\n\nclass DeepSeek(BaseLL"
},
{
"path": "deepsearcher/llm/gemini.py",
"chars": 2016,
"preview": "import os\nfrom typing import Dict, List\n\nfrom deepsearcher.llm.base import BaseLLM, ChatResponse\n\n\nclass Gemini(BaseLLM)"
},
{
"path": "deepsearcher/llm/glm.py",
"chars": 1049,
"preview": "import os\nfrom typing import Dict, List\n\nfrom deepsearcher.llm.base import BaseLLM, ChatResponse\n\n\nclass GLM(BaseLLM):\n "
},
{
"path": "deepsearcher/llm/jiekouai.py",
"chars": 2276,
"preview": "import os\nfrom typing import Dict, List\n\nfrom deepsearcher.llm.base import BaseLLM, ChatResponse\n\n\nclass JiekouAI(BaseLL"
},
{
"path": "deepsearcher/llm/novita.py",
"chars": 1001,
"preview": "import os\nfrom typing import Dict, List\n\nfrom deepsearcher.llm.base import BaseLLM, ChatResponse\n\n\nclass Novita(BaseLLM)"
},
{
"path": "deepsearcher/llm/ollama.py",
"chars": 1821,
"preview": "from typing import Dict, List\n\nfrom deepsearcher.llm.base import BaseLLM, ChatResponse\n\n\nclass Ollama(BaseLLM):\n \"\"\"\n"
},
{
"path": "deepsearcher/llm/openai_llm.py",
"chars": 2163,
"preview": "import os\nfrom typing import Dict, List\n\nfrom deepsearcher.llm.base import BaseLLM, ChatResponse\n\n\nclass OpenAI(BaseLLM)"
},
{
"path": "deepsearcher/llm/ppio.py",
"chars": 2247,
"preview": "import os\nfrom typing import Dict, List\n\nfrom deepsearcher.llm.base import BaseLLM, ChatResponse\n\n\nclass PPIO(BaseLLM):\n"
},
{
"path": "deepsearcher/llm/siliconflow.py",
"chars": 2358,
"preview": "import os\nfrom typing import Dict, List\n\nfrom deepsearcher.llm.base import BaseLLM, ChatResponse\n\n\nclass SiliconFlow(Bas"
},
{
"path": "deepsearcher/llm/together_ai.py",
"chars": 2013,
"preview": "import os\nfrom typing import Dict, List\n\nfrom deepsearcher.llm.base import BaseLLM, ChatResponse\n\n\nclass TogetherAI(Base"
},
{
"path": "deepsearcher/llm/volcengine.py",
"chars": 2363,
"preview": "import os\nfrom typing import Dict, List\n\nfrom deepsearcher.llm.base import BaseLLM, ChatResponse\n\n\nclass Volcengine(Base"
},
{
"path": "deepsearcher/llm/watsonx.py",
"chars": 5980,
"preview": "import os\nfrom typing import Dict, List\n\nfrom deepsearcher.llm.base import BaseLLM, ChatResponse\n\ntry:\n from ibm_wats"
},
{
"path": "deepsearcher/llm/xai.py",
"chars": 2247,
"preview": "import os\nfrom typing import Dict, List\n\nfrom deepsearcher.llm.base import BaseLLM, ChatResponse\n\n\nclass XAI(BaseLLM):\n "
},
{
"path": "deepsearcher/loader/__init__.py",
"chars": 0,
"preview": ""
},
{
"path": "deepsearcher/loader/file_loader/__init__.py",
"chars": 455,
"preview": "from deepsearcher.loader.file_loader.docling_loader import DoclingLoader\nfrom deepsearcher.loader.file_loader.json_loade"
},
{
"path": "deepsearcher/loader/file_loader/base.py",
"chars": 2223,
"preview": "import os\nfrom abc import ABC\nfrom typing import List\n\nfrom langchain_core.documents import Document\n\n\nclass BaseLoader("
},
{
"path": "deepsearcher/loader/file_loader/docling_loader.py",
"chars": 3876,
"preview": "import os\nfrom typing import List\n\nfrom langchain_core.documents import Document\n\nfrom deepsearcher.loader.file_loader.b"
},
{
"path": "deepsearcher/loader/file_loader/json_loader.py",
"chars": 2961,
"preview": "import json\nfrom typing import List\n\nfrom langchain_core.documents import Document\n\nfrom deepsearcher.loader.file_loader"
},
{
"path": "deepsearcher/loader/file_loader/pdf_loader.py",
"chars": 1740,
"preview": "from typing import List\n\nfrom langchain_core.documents import Document\n\nfrom deepsearcher.loader.file_loader.base import"
},
{
"path": "deepsearcher/loader/file_loader/text_loader.py",
"chars": 1188,
"preview": "from typing import List\n\nfrom langchain_core.documents import Document\n\nfrom deepsearcher.loader.file_loader.base import"
},
{
"path": "deepsearcher/loader/file_loader/unstructured_loader.py",
"chars": 6287,
"preview": "import os\nimport shutil\nfrom typing import List\n\nfrom langchain_core.documents import Document\n\nfrom deepsearcher.loader"
},
{
"path": "deepsearcher/loader/splitter.py",
"chars": 3751,
"preview": "## Sentence Window splitting strategy, ref:\n# https://github.com/milvus-io/bootcamp/blob/master/bootcamp/RAG/advanced_r"
},
{
"path": "deepsearcher/loader/web_crawler/__init__.py",
"chars": 403,
"preview": "from deepsearcher.loader.web_crawler.crawl4ai_crawler import Crawl4AICrawler\nfrom deepsearcher.loader.web_crawler.doclin"
},
{
"path": "deepsearcher/loader/web_crawler/base.py",
"chars": 1703,
"preview": "from abc import ABC\nfrom typing import List\n\nfrom langchain_core.documents import Document\n\n\nclass BaseCrawler(ABC):\n "
},
{
"path": "deepsearcher/loader/web_crawler/crawl4ai_crawler.py",
"chars": 4786,
"preview": "import asyncio\nfrom typing import List\n\nfrom langchain_core.documents import Document\n\nfrom deepsearcher.loader.web_craw"
},
{
"path": "deepsearcher/loader/web_crawler/docling_crawler.py",
"chars": 3061,
"preview": "from typing import List\n\nfrom langchain_core.documents import Document\n\nfrom deepsearcher.loader.web_crawler.base import"
},
{
"path": "deepsearcher/loader/web_crawler/firecrawl_crawler.py",
"chars": 3273,
"preview": "import os\nfrom typing import List, Optional\n\nfrom firecrawl import FirecrawlApp, ScrapeOptions\nfrom langchain_core.docum"
},
{
"path": "deepsearcher/loader/web_crawler/jina_crawler.py",
"chars": 1827,
"preview": "import os\nfrom typing import List\n\nimport requests\nfrom langchain_core.documents import Document\n\nfrom deepsearcher.load"
},
{
"path": "deepsearcher/offline_loading.py",
"chars": 4636,
"preview": "import os\nfrom typing import List, Union\n\nfrom tqdm import tqdm\n\n# from deepsearcher.configuration import embedding_mode"
},
{
"path": "deepsearcher/online_query.py",
"chars": 3600,
"preview": "from typing import List, Tuple\n\n# from deepsearcher.configuration import vector_db, embedding_model, llm\nfrom deepsearch"
},
{
"path": "deepsearcher/utils/__init__.py",
"chars": 0,
"preview": ""
},
{
"path": "deepsearcher/utils/log.py",
"chars": 3765,
"preview": "import logging\n\nfrom termcolor import colored\n\n\nclass ColoredFormatter(logging.Formatter):\n \"\"\"\n A custom formatte"
},
{
"path": "deepsearcher/vector_db/__init__.py",
"chars": 216,
"preview": "from .azure_search import AzureSearch\nfrom .milvus import Milvus, RetrievalResult\nfrom .oracle import OracleDB\nfrom .qdr"
},
{
"path": "deepsearcher/vector_db/azure_search.py",
"chars": 10508,
"preview": "import uuid\nfrom typing import Any, Dict, List, Optional\n\nfrom deepsearcher.vector_db.base import BaseVectorDB, Collecti"
},
{
"path": "deepsearcher/vector_db/base.py",
"chars": 6320,
"preview": "from abc import ABC, abstractmethod\nfrom typing import List, Union\n\nimport numpy as np\n\nfrom deepsearcher.loader.splitte"
},
{
"path": "deepsearcher/vector_db/milvus.py",
"chars": 11956,
"preview": "from typing import List, Optional, Union\n\nimport numpy as np\nfrom pymilvus import AnnSearchRequest, DataType, Function, "
},
{
"path": "deepsearcher/vector_db/oracle.py",
"chars": 19786,
"preview": "import array\nimport json\nfrom typing import List, Optional, Union\n\nimport numpy as np\n\nfrom deepsearcher.loader.splitter"
},
{
"path": "deepsearcher/vector_db/qdrant.py",
"chars": 10439,
"preview": "import uuid\nfrom typing import List, Optional, Union\n\nimport numpy as np\n\nfrom deepsearcher.loader.splitter import Chunk"
},
{
"path": "docs/README.md",
"chars": 786,
"preview": "# DeepSearcher Documentation\n\nThis directory contains the documentation for DeepSearcher, powered by MkDocs.\n\n## Setup\n\n"
},
{
"path": "docs/configuration/embedding.md",
"chars": 4518,
"preview": "# Embedding Model Configuration\n\nDeepSearcher supports various embedding models to convert text into vector representati"
},
{
"path": "docs/configuration/file_loader.md",
"chars": 2273,
"preview": "# File Loader Configuration\n\nDeepSearcher supports various file loaders to extract and process content from different fi"
},
{
"path": "docs/configuration/index.md",
"chars": 1270,
"preview": "# Configuration Overview\n\nDeepSearcher provides flexible configuration options for all its components. You can customize"
},
{
"path": "docs/configuration/llm.md",
"chars": 7023,
"preview": "# LLM Configuration\n\nDeepSearcher supports various Large Language Models (LLMs) for processing queries and generating re"
},
{
"path": "docs/configuration/vector_db.md",
"chars": 2092,
"preview": "# Vector Database Configuration\n\nDeepSearcher uses vector databases to store and retrieve document embeddings for effici"
},
{
"path": "docs/configuration/web_crawler.md",
"chars": 2886,
"preview": "# Web Crawler Configuration\n\nDeepSearcher supports various web crawlers to collect data from websites for processing and"
},
{
"path": "docs/contributing/index.md",
"chars": 5318,
"preview": "# Contributing to DeepSearcher\n\nWe welcome contributions from everyone. This document provides guidelines to make the co"
},
{
"path": "docs/examples/basic_example.md",
"chars": 2389,
"preview": "# Basic Example\n\nThis example demonstrates the core functionality of DeepSearcher - loading documents and performing sem"
},
{
"path": "docs/examples/docling.md",
"chars": 3376,
"preview": "# Docling Integration Example\n\nThis example shows how to use Docling for loading local files and crawling web content.\n\n"
},
{
"path": "docs/examples/firecrawl.md",
"chars": 3040,
"preview": "# FireCrawl Integration Example\n\nThis example demonstrates how to use FireCrawl with DeepSearcher to crawl and extract c"
},
{
"path": "docs/examples/index.md",
"chars": 892,
"preview": "# Usage Examples\n\nDeepSearcher provides several example scripts to help you get started quickly. These examples demonstr"
},
{
"path": "docs/examples/oracle.md",
"chars": 2376,
"preview": "# Oracle Example\n\nThis example demonstrates an advanced setup using path manipulation and detailed token tracking.\n\n## O"
},
{
"path": "docs/examples/unstructured.md",
"chars": 2892,
"preview": "# Unstructured Integration Example\n\nThis example demonstrates how to use the Unstructured library with DeepSearcher for "
},
{
"path": "docs/faq/index.md",
"chars": 2164,
"preview": "# Frequently Asked Questions\n\n## 🔍 Common Issues and Solutions\n\n---\n\n### 💬 Q1: Why am I failing to parse LLM output form"
},
{
"path": "docs/future_plans.md",
"chars": 288,
"preview": "# Future Plans\n\n- Enhance web crawling functionality\n- Support more vector databases (e.g., FAISS...)\n- Add support for "
},
{
"path": "docs/index.md",
"chars": 1907,
"preview": "# 🔍 DeepSearcher\n\n\n\n<div align=\"center\">\n\n <a href=\"https://opensource.org/licens"
},
{
"path": "docs/installation/development.md",
"chars": 1577,
"preview": "# 🛠️ Development Mode Installation\n\nThis guide is for contributors who want to modify DeepSearcher's code or develop new"
},
{
"path": "docs/installation/index.md",
"chars": 864,
"preview": "# 🔧 Installation\n\nDeepSearcher offers multiple installation methods to suit different user needs.\n\n## 📋 Installation Opt"
},
{
"path": "docs/installation/pip.md",
"chars": 1115,
"preview": "# 📦 Installation via pip\n\nThis method is recommended for most users who want to use DeepSearcher without modifying its s"
},
{
"path": "docs/integrations/index.md",
"chars": 5197,
"preview": "# Module Support\n\nDeepSearcher supports various integration modules including embedding models, large language models, d"
},
{
"path": "docs/overrides/.gitkeep",
"chars": 0,
"preview": ""
},
{
"path": "docs/stylesheets/extra.css",
"chars": 1258,
"preview": "/* Add your custom CSS here */ \n\n/* FAQ Styling */\n.faq-answer {\n background-color: #f8f9fa;\n border-left: 4px solid #"
},
{
"path": "docs/usage/cli.md",
"chars": 1249,
"preview": "# 💻 Command Line Interface\n\nDeepSearcher provides a convenient command line interface for loading data and querying.\n\n##"
},
{
"path": "docs/usage/deployment.md",
"chars": 2254,
"preview": "# 🌐 Deployment\n\nThis guide explains how to deploy DeepSearcher as a web service.\n\n## ⚙️ Configure Modules\n\nYou can confi"
},
{
"path": "docs/usage/index.md",
"chars": 564,
"preview": "# 📚 Usage Guide\n\nDeepSearcher provides multiple ways to use the system, including Python API, command line interface, an"
},
{
"path": "docs/usage/quick_start.md",
"chars": 1358,
"preview": "# 🚀 Quick Start\n\n## Prerequisites\n\n✅ Before you begin, prepare your `OPENAI_API_KEY` in your environment variables. If y"
},
{
"path": "env.example",
"chars": 1836,
"preview": "# Copy this file to .env and fill in your API keys\n\n# OpenAI API Key (required if using OpenAI models)\nOPENAI_API_KEY=yo"
},
{
"path": "evaluation/README.md",
"chars": 3802,
"preview": "# Evaluation of DeepSearcher\n## Introduction\nDeepSearcher is very good at answering complex queries. In this evaluation "
},
{
"path": "evaluation/eval_config.yaml",
"chars": 3348,
"preview": "provide_settings:\n llm:\n provider: \"OpenAI\"\n config:\n model: \"o1-mini\"\n# api_key: \"sk-xxxx\" # Uncommen"
},
{
"path": "evaluation/evaluate.py",
"chars": 11878,
"preview": "# Some test dataset and evaluation method are ref from https://github.com/OSU-NLP-Group/HippoRAG/tree/main/data , many t"
},
{
"path": "examples/basic_example.py",
"chars": 1524,
"preview": "import logging\nimport os\n\nfrom deepsearcher.offline_loading import load_from_local_files\nfrom deepsearcher.online_query "
},
{
"path": "examples/basic_example_azuresearch.py",
"chars": 2290,
"preview": "import logging\nimport os\nimport time\n\nfrom deepsearcher.configuration import Configuration, init_config\nfrom deepsearche"
},
{
"path": "examples/basic_example_oracle.py",
"chars": 1365,
"preview": "import sys, os\nfrom pathlib import Path\nscript_directory = Path(__file__).resolve().parent.parent\nsys.path.append(os.pat"
},
{
"path": "examples/basic_watsonx_example.py",
"chars": 4475,
"preview": "\"\"\"\nExample usage of WatsonX embedding and LLM in DeepSearcher.\n\nThis example demonstrates how to configure and use IBM "
},
{
"path": "examples/data/2wikimultihopqa.json",
"chars": 6505789,
"preview": "[\n {\n \"_id\": \"83bf3b5a0bd911eba7f7acde48001122\",\n \"type\": \"compositional\",\n \"question\": \"When di"
},
{
"path": "examples/data/2wikimultihopqa_corpus.json",
"chars": 3083943,
"preview": "[\n {\n \"title\": \"Teutberga\",\n \"text\": \"Teutberga( died 11 November 875) was a queen of Lotharingia by ma"
},
{
"path": "examples/load_and_crawl_using_docling.py",
"chars": 2393,
"preview": "import logging\nimport os\nfrom deepsearcher.offline_loading import load_from_local_files, load_from_website\nfrom deepsear"
},
{
"path": "examples/load_local_file_using_unstructured.py",
"chars": 1420,
"preview": "import logging\nimport os\nfrom deepsearcher.offline_loading import load_from_local_files\nfrom deepsearcher.online_query i"
},
{
"path": "examples/load_website_using_firecrawl.py",
"chars": 1685,
"preview": "import logging\nimport os\nfrom deepsearcher.offline_loading import load_from_website\nfrom deepsearcher.online_query impor"
},
{
"path": "main.py",
"chars": 6423,
"preview": "import argparse\nfrom typing import Dict, List, Union\n\nimport uvicorn\nfrom fastapi import Body, FastAPI, HTTPException, Q"
},
{
"path": "mkdocs.yml",
"chars": 3019,
"preview": "site_name: DeepSearcher\nsite_url: https://zilliztech.github.io/deep-searcher/\nrepo_name: zilliztech/deep-searcher\nrepo_u"
},
{
"path": "pyproject.toml",
"chars": 4148,
"preview": "[project]\nname = \"deepsearcher\"\nversion = \"0.0.2\"\nreadme = \"README.md\"\nrequires-python = \">=3.10\"\ndependencies = [\n \""
},
{
"path": "tests/__init__.py",
"chars": 37,
"preview": "# Tests for the deepsearcher package "
},
{
"path": "tests/agent/__init__.py",
"chars": 29,
"preview": "# Tests for the agent module "
},
{
"path": "tests/agent/test_base.py",
"chars": 5133,
"preview": "import unittest\nfrom unittest.mock import MagicMock\nimport numpy as np\n\nfrom deepsearcher.llm.base import BaseLLM, ChatR"
},
{
"path": "tests/agent/test_chain_of_rag.py",
"chars": 10244,
"preview": "from unittest.mock import MagicMock, patch\n\nfrom deepsearcher.agent import ChainOfRAG\nfrom deepsearcher.vector_db.base i"
},
{
"path": "tests/agent/test_collection_router.py",
"chars": 6435,
"preview": "from unittest.mock import MagicMock, patch\n\nfrom deepsearcher.agent.collection_router import CollectionRouter\nfrom deeps"
},
{
"path": "tests/agent/test_deep_search.py",
"chars": 9386,
"preview": "from unittest.mock import MagicMock, patch\nimport asyncio\n\nfrom deepsearcher.agent import DeepSearch\nfrom deepsearcher.v"
},
{
"path": "tests/agent/test_naive_rag.py",
"chars": 4866,
"preview": "from unittest.mock import MagicMock\n\nfrom deepsearcher.agent import NaiveRAG\nfrom deepsearcher.vector_db.base import Ret"
},
{
"path": "tests/agent/test_rag_router.py",
"chars": 6657,
"preview": "from unittest.mock import MagicMock, patch\n\nfrom deepsearcher.agent import NaiveRAG, ChainOfRAG, DeepSearch\nfrom deepsea"
},
{
"path": "tests/embedding/__init__.py",
"chars": 47,
"preview": "# Tests for the deepsearcher.embedding package "
},
{
"path": "tests/embedding/test_base.py",
"chars": 3717,
"preview": "import unittest\nfrom typing import List\nfrom unittest.mock import patch, MagicMock\n\nfrom deepsearcher.embedding.base imp"
},
{
"path": "tests/embedding/test_bedrock_embedding.py",
"chars": 5201,
"preview": "import unittest\nimport json\nimport os\nfrom unittest.mock import patch, MagicMock\nimport logging\n\n# Disable logging for t"
},
{
"path": "tests/embedding/test_fastembed_embedding.py",
"chars": 5162,
"preview": "import unittest\nimport numpy as np\nfrom unittest.mock import patch, MagicMock\nimport logging\n\n# Disable logging for test"
},
{
"path": "tests/embedding/test_gemini_embedding.py",
"chars": 10394,
"preview": "import unittest\nimport os\nfrom unittest.mock import patch, MagicMock\nimport logging\nimport warnings\nimport multiprocessi"
},
{
"path": "tests/embedding/test_glm_embedding.py",
"chars": 5034,
"preview": "import unittest\nimport os\nfrom unittest.mock import patch, MagicMock\n\nfrom deepsearcher.embedding import GLMEmbedding\n\n\n"
},
{
"path": "tests/embedding/test_jiekouai_embedding.py",
"chars": 6679,
"preview": "import unittest\nimport os\nfrom unittest.mock import patch, MagicMock\n\nfrom deepsearcher.embedding import JiekouAIEmbeddi"
},
{
"path": "tests/embedding/test_milvus_embedding.py",
"chars": 5348,
"preview": "import unittest\nfrom unittest.mock import patch, MagicMock\n\nimport numpy as np\nfrom deepsearcher.embedding import Milvus"
},
{
"path": "tests/embedding/test_novita_embedding.py",
"chars": 6757,
"preview": "import unittest\nimport os\nfrom unittest.mock import patch, MagicMock\n\nimport requests\nfrom deepsearcher.embedding import"
},
{
"path": "tests/embedding/test_ollama_embedding.py",
"chars": 8821,
"preview": "import unittest\nimport sys\nfrom unittest.mock import patch, MagicMock\n\nfrom deepsearcher.embedding import OllamaEmbeddin"
},
{
"path": "tests/embedding/test_openai_embedding.py",
"chars": 10587,
"preview": "import unittest\nimport os\nfrom unittest.mock import patch, MagicMock, ANY\n\nfrom openai._types import NOT_GIVEN\nfrom deep"
},
{
"path": "tests/embedding/test_ppio_embedding.py",
"chars": 6637,
"preview": "import unittest\nimport os\nfrom unittest.mock import patch, MagicMock\n\nimport requests\nfrom deepsearcher.embedding import"
},
{
"path": "tests/embedding/test_sentence_transformer_embedding.py",
"chars": 8337,
"preview": "import unittest\nimport sys\nimport logging\nfrom unittest.mock import patch, MagicMock\n\n# Disable logging for tests\nloggin"
},
{
"path": "tests/embedding/test_siliconflow_embedding.py",
"chars": 7288,
"preview": "import unittest\nimport os\nfrom unittest.mock import patch, MagicMock\n\nimport requests\nfrom deepsearcher.embedding import"
},
{
"path": "tests/embedding/test_volcengine_embedding.py",
"chars": 7401,
"preview": "import unittest\nimport os\nfrom unittest.mock import patch, MagicMock\n\nimport requests\nfrom deepsearcher.embedding import"
},
{
"path": "tests/embedding/test_voyage_embedding.py",
"chars": 5111,
"preview": "import unittest\nimport os\nfrom unittest.mock import patch, MagicMock\n\nfrom deepsearcher.embedding import VoyageEmbedding"
},
{
"path": "tests/embedding/test_watsonx_embedding.py",
"chars": 11584,
"preview": "import unittest\nfrom unittest.mock import MagicMock, patch, ANY\nimport os\n\nclass TestWatsonXEmbedding(unittest.TestCase)"
},
{
"path": "tests/llm/__init__.py",
"chars": 41,
"preview": "# Tests for the deepsearcher.llm package "
},
{
"path": "tests/llm/test_aliyun.py",
"chars": 6331,
"preview": "import unittest\nfrom unittest.mock import patch, MagicMock\nimport os\nimport logging\n\n# Disable logging for tests\nlogging"
},
{
"path": "tests/llm/test_anthropic.py",
"chars": 6463,
"preview": "import unittest\nfrom unittest.mock import patch, MagicMock\nimport os\nimport logging\nfrom typing import List\n\n# Disable l"
},
{
"path": "tests/llm/test_azure_openai.py",
"chars": 6398,
"preview": "import unittest\nfrom unittest.mock import patch, MagicMock\nimport os\nimport logging\n\n# Disable logging for tests\nlogging"
},
{
"path": "tests/llm/test_base.py",
"chars": 5435,
"preview": "import unittest\nfrom deepsearcher.llm.base import BaseLLM, ChatResponse\nfrom unittest.mock import patch\n\n\nclass TestBase"
},
{
"path": "tests/llm/test_bedrock.py",
"chars": 7730,
"preview": "import unittest\nfrom unittest.mock import patch, MagicMock\nimport os\nimport logging\n\n# Disable logging for tests\nlogging"
},
{
"path": "tests/llm/test_deepseek.py",
"chars": 6322,
"preview": "import unittest\nfrom unittest.mock import patch, MagicMock\nimport os\nimport logging\n\n# Disable logging for tests\nlogging"
},
{
"path": "tests/llm/test_gemini.py",
"chars": 5265,
"preview": "import unittest\nfrom unittest.mock import patch, MagicMock\nimport os\nimport logging\n\n# Disable logging for tests\nlogging"
},
{
"path": "tests/llm/test_glm.py",
"chars": 6170,
"preview": "import unittest\nfrom unittest.mock import patch, MagicMock\nimport os\nimport logging\n\n# Disable logging for tests\nlogging"
},
{
"path": "tests/llm/test_jiekouai.py",
"chars": 6189,
"preview": "import unittest\nfrom unittest.mock import patch, MagicMock\nimport os\nimport logging\n\n# Disable logging for tests\nlogging"
},
{
"path": "tests/llm/test_novita.py",
"chars": 6199,
"preview": "import unittest\nfrom unittest.mock import patch, MagicMock\nimport os\nimport logging\n\n# Disable logging for tests\nlogging"
},
{
"path": "tests/llm/test_ollama.py",
"chars": 4672,
"preview": "import unittest\nfrom unittest.mock import patch, MagicMock\nimport logging\nimport os\n\n# Disable logging for tests\nlogging"
},
{
"path": "tests/llm/test_openai.py",
"chars": 6058,
"preview": "import unittest\nfrom unittest.mock import patch, MagicMock\nimport os\nimport logging\n\n# Disable logging for tests\nlogging"
},
{
"path": "tests/llm/test_ppio.py",
"chars": 6223,
"preview": "import unittest\nfrom unittest.mock import patch, MagicMock\nimport os\nimport logging\n\n# Disable logging for tests\nlogging"
},
{
"path": "tests/llm/test_siliconflow.py",
"chars": 6327,
"preview": "import unittest\nfrom unittest.mock import patch, MagicMock\nimport os\nimport logging\n\n# Disable logging for tests\nlogging"
},
{
"path": "tests/llm/test_together_ai.py",
"chars": 5721,
"preview": "import unittest\nfrom unittest.mock import patch, MagicMock\nimport os\nimport logging\n\n# Disable logging for tests\nlogging"
},
{
"path": "tests/llm/test_volcengine.py",
"chars": 6322,
"preview": "import unittest\nfrom unittest.mock import patch, MagicMock\nimport os\nimport logging\n\n# Disable logging for tests\nlogging"
},
{
"path": "tests/llm/test_watsonx.py",
"chars": 17648,
"preview": "import unittest\nfrom unittest.mock import MagicMock, patch\nimport os\n\nclass TestWatsonX(unittest.TestCase):\n \"\"\"Test "
},
{
"path": "tests/llm/test_xai.py",
"chars": 6104,
"preview": "import unittest\nfrom unittest.mock import patch, MagicMock\nimport os\nimport logging\n\n# Disable logging for tests\nlogging"
},
{
"path": "tests/loader/__init__.py",
"chars": 44,
"preview": "# Tests for the deepsearcher.loader package "
},
{
"path": "tests/loader/file_loader/__init__.py",
"chars": 56,
"preview": "# Tests for the deepsearcher.loader.file_loader package "
},
{
"path": "tests/loader/file_loader/test_base.py",
"chars": 2770,
"preview": "import unittest\nimport os\nimport tempfile\nfrom unittest.mock import patch, MagicMock\n\nfrom langchain_core.documents impo"
},
{
"path": "tests/loader/file_loader/test_docling_loader.py",
"chars": 7638,
"preview": "import unittest\nimport os\nimport tempfile\nfrom unittest.mock import patch, MagicMock\n\nfrom langchain_core.documents impo"
},
{
"path": "tests/loader/file_loader/test_json_loader.py",
"chars": 5294,
"preview": "import unittest\nimport os\nimport json\nimport tempfile\n\nfrom langchain_core.documents import Document\n\nfrom deepsearcher."
},
{
"path": "tests/loader/file_loader/test_pdf_loader.py",
"chars": 4956,
"preview": "import unittest\nimport os\nimport tempfile\nfrom unittest.mock import patch, MagicMock\n\nfrom langchain_core.documents impo"
},
{
"path": "tests/loader/file_loader/test_text_loader.py",
"chars": 2980,
"preview": "import unittest\nimport os\nimport tempfile\n\nfrom deepsearcher.loader.file_loader import TextLoader\n\n\nclass TestTextLoader"
},
{
"path": "tests/loader/file_loader/test_unstructured_loader.py",
"chars": 8416,
"preview": "import unittest\nimport os\nimport shutil\nimport tempfile\nfrom unittest.mock import patch, MagicMock\n\nfrom langchain_core."
},
{
"path": "tests/loader/test_splitter.py",
"chars": 4148,
"preview": "import unittest\nfrom langchain_core.documents import Document\n\nfrom deepsearcher.loader.splitter import Chunk, split_doc"
},
{
"path": "tests/loader/web_crawler/__init__.py",
"chars": 56,
"preview": "# Tests for the deepsearcher.loader.web_crawler package "
},
{
"path": "tests/loader/web_crawler/test_base.py",
"chars": 1823,
"preview": "import unittest\nfrom unittest.mock import patch, MagicMock\n\nfrom deepsearcher.loader.web_crawler.base import BaseCrawler"
},
{
"path": "tests/loader/web_crawler/test_crawl4ai_crawler.py",
"chars": 5840,
"preview": "import unittest\nimport asyncio\nfrom unittest.mock import patch, MagicMock\nimport warnings\n\nfrom langchain_core.documents"
},
{
"path": "tests/loader/web_crawler/test_docling_crawler.py",
"chars": 5964,
"preview": "import unittest\nfrom unittest.mock import patch, MagicMock\n\nfrom langchain_core.documents import Document\n\nfrom deepsear"
},
{
"path": "tests/loader/web_crawler/test_firecrawl_crawler.py",
"chars": 5301,
"preview": "import unittest\nimport os\nfrom unittest.mock import patch, MagicMock\n\nfrom langchain_core.documents import Document\n\nfro"
},
{
"path": "tests/loader/web_crawler/test_jina_crawler.py",
"chars": 4140,
"preview": "import unittest\nimport os\nfrom unittest.mock import patch, MagicMock\n\nimport requests\nfrom langchain_core.documents impo"
},
{
"path": "tests/utils/test_log.py",
"chars": 6313,
"preview": "import unittest\nfrom unittest.mock import patch, MagicMock, call\nimport logging\nfrom termcolor import colored\n\nfrom deep"
},
{
"path": "tests/vector_db/test_azure_search.py",
"chars": 8193,
"preview": "import unittest\nfrom unittest.mock import patch, MagicMock\nimport numpy as np\nimport sys\n\nfrom deepsearcher.vector_db im"
},
{
"path": "tests/vector_db/test_base.py",
"chars": 5427,
"preview": "import unittest\nimport numpy as np\nfrom typing import List\n\nfrom deepsearcher.vector_db.base import (\n RetrievalResul"
},
{
"path": "tests/vector_db/test_milvus.py",
"chars": 4518,
"preview": "import unittest\nfrom unittest.mock import patch, MagicMock\nimport numpy as np\nimport warnings\n\n# Filter out the pkg_reso"
},
{
"path": "tests/vector_db/test_oracle.py",
"chars": 8707,
"preview": "import unittest\nfrom unittest.mock import patch, MagicMock\nimport numpy as np\nimport sys\nimport json\n\nfrom deepsearcher."
},
{
"path": "tests/vector_db/test_qdrant.py",
"chars": 6181,
"preview": "import unittest\nfrom unittest.mock import patch, MagicMock\nimport numpy as np\nimport sys\n\nclass TestQdrant(unittest.Test"
}
]
About this extraction
This page contains the full source code of the zilliztech/deep-searcher GitHub repository, extracted and formatted as plain text for AI agents and large language models (LLMs). The extraction includes 199 files (9.9 MB), approximately 2.6M tokens, and a symbol index with 962 extracted functions, classes, methods, constants, and types. Use this with OpenClaw, Claude, ChatGPT, Cursor, Windsurf, or any other AI tool that accepts text input. You can copy the full output to your clipboard or download it as a .txt file.
Extracted by GitExtract — free GitHub repo to text converter for AI. Built by Nikandr Surkov.