[ { "path": ".cargo/config.toml", "content": "[build]\nrustflags = [\"--cfg\", \"tokio_unstable\"]\nrustdocflags = [\"--cfg\", \"tokio_unstable\"]\n\n[target.x86_64-unknown-linux-gnu]\n# Targeting x86-64-v2 gives a ~2% performance boost while only\n# disallowing Intel CPUs older than 2008 and AMD CPUs older than 2011.\n# None of those very old CPUs are used in GCP\n# (https://cloud.google.com/compute/docs/cpu-platforms). Unfortunately,\n# AWS does not seem to disclose the exact CPUs they use.\nrustflags = [\"-C\", \"target-cpu=x86-64-v2\", \"--cfg\", \"tokio_unstable\"]\n\n" }, { "path": ".claude/skills/bump-tantivy/SKILL.md", "content": "---\nname: bump-tantivy\ndescription: Bump tantivy to the latest commit on main branch, fix compilation issues, and open a PR\ndisable-model-invocation: true\n---\n\n# Bump Tantivy\n\nFollow these steps to bump tantivy to its latest version:\n\n## Step 1: Check that we are on the main branch\n\nRun: `git branch --show-current`\n\nIf the current branch is not `main`, abort and ask the user to switch to the main branch first.\n\n## Step 2: Ensure main is up to date\n\nRun: `git pull origin main`\n\nThis ensures we're working from the latest code.\n\n## Step 3: Get the latest tantivy SHA\n\nRun: `gh api repos/quickwit-oss/tantivy/commits/main --jq '.sha'`\n\nExtract the first 7 characters as the short SHA.\n\n## Step 4: Update Cargo.toml\n\nEdit `quickwit/Cargo.toml` and update the `rev` field in the tantivy dependency to the new short SHA.\n\nThe line looks like:\n```toml\ntantivy = { git = \"https://github.com/quickwit-oss/tantivy/\", rev = \"XXXXXXX\", ... }\n```\n\n## Step 5: Run cargo check and fix compilation errors\n\nRun `cargo check` in the `quickwit` directory to verify compilation.\n\nIf there are compilation errors:\n- If the fix is straightforward (simple API changes, renames, etc.), fix them without asking\n- If the fix is complex or unclear, ask the user before proceeding\n\nRepeat until cargo check passes.\n\n## Step 6: Format code\n\nRun `make fmt` from the `quickwit/` directory to format the code.\n\n## Step 7: Update licenses\n\nRun `make update-licenses` from the `quickwit/` directory, then move the generated file:\n```\nmv quickwit/LICENSE-3rdparty.csv ./LICENSE-3rdparty.csv\n```\n\n## Step 8: Create a new branch\n\nGet the git username: `git config user.name | tr ' ' '-' | tr '[:upper:]' '[:lower:]'`\n\nGet today's date: `date +%Y-%m-%d`\n\nCreate and checkout a new branch named: `{username}/bump-tantivy-{date}`\n\nExample: `paul/bump-tantivy-2024-03-15`\n\n## Step 9: Commit changes\n\nStage all modified files and create a commit with message:\n```\nBump tantivy to {short-sha}\n```\n\n## Step 10: Push and open a PR\n\nPush the branch and open a PR using:\n```\ngh pr create --title \"Bump tantivy to {short-sha}\" --body \"Updates tantivy dependency to the latest commit on main.\"\n```\n\nReport the PR URL to the user when complete.\n" }, { "path": ".claude/skills/fix-clippy/SKILL.md", "content": "---\nname: fix-clippy\ndescription: Fix all clippy lint warnings in the project\n---\n\n# Fix Clippy\n\nClippy issues are **warnings**, not errors. Never grep for `error` when looking for clippy issues.\n\n## Step 1: Auto-fix\n\nRun `make fix` to automatically fix clippy warnings:\n\n```\nmake fix\n```\n\n## Step 2: Fix remaining warnings manually\n\nCheck for remaining warnings that couldn't be auto-fixed:\n\n```\ncargo clippy --tests 2>&1 | grep \"^warning:\" | sort -u\n```\n\nFor each remaining warning, find the exact location and fix it manually.\n" }, { "path": ".claude/skills/fmt/SKILL.md", "content": "---\nname: fmt\ndescription: Run `make fmt` to check the code format.\n---\n\n# Format Check\n\nRun `make fmt` from the `quickwit/` subdirectory to check code formatting:\n\n```\ncd /Users/paul.masurel/git/quickwit/quickwit && make fmt\n```\n\nThis command checks:\n1. Rust code formatting\n2. License headers\n3. Log format policy (no trailing punctuation, no uppercase first character)\n\nIf there are log format issues, fix them by:\n- Making the first character lowercase\n- Removing trailing punctuation (periods, exclamation marks, etc.)\n\nFix any issues found and re-run until clean.\n" }, { "path": ".claude/skills/rationalize-deps/SKILL.md", "content": "---\nname: rationalize-deps\ndescription: Analyze Cargo.toml dependencies and attempt to remove unused features to reduce compile times and binary size\n---\n\n# Rationalize Dependencies\n\nThis skill analyzes Cargo.toml dependencies to identify and remove unused features.\n\n## Overview\n\nMany crates enable features by default that may not be needed. This skill:\n1. Identifies dependencies with default features enabled\n2. Tests if `default-features = false` works\n3. Identifies which specific features are actually needed\n4. Verifies compilation after changes\n\n## Step 1: Identify the target\n\nAsk the user which crate(s) to analyze:\n- A specific crate name (e.g., \"tokio\", \"serde\")\n- A specific workspace member (e.g., \"quickwit-search\")\n- \"all\" to scan the entire workspace\n\n## Step 2: Analyze current dependencies\n\nFor the workspace Cargo.toml (`quickwit/Cargo.toml`), list dependencies that:\n- Do NOT have `default-features = false`\n- Have default features that might be unnecessary\n\nRun: `cargo tree -p -f \"{p} {f}\" --edges features` to see what features are actually used.\n\n## Step 3: For each candidate dependency\n\n### 3a: Check the crate's default features\n\nLook up the crate on crates.io or check its Cargo.toml to understand:\n- What features are enabled by default\n- What each feature provides\n\nUse: `cargo metadata --format-version=1 | jq '.packages[] | select(.name == \"\") | .features'`\n\n### 3b: Try disabling default features\n\nModify the dependency in `quickwit/Cargo.toml`:\n\nFrom:\n```toml\nsome-crate = { version = \"1.0\" }\n```\n\nTo:\n```toml\nsome-crate = { version = \"1.0\", default-features = false }\n```\n\n### 3c: Run cargo check\n\nRun: `cargo check --workspace` (or target specific packages for faster feedback)\n\nIf compilation fails:\n1. Read the error messages to identify which features are needed\n2. Add only the required features explicitly:\n ```toml\n some-crate = { version = \"1.0\", default-features = false, features = [\"needed-feature\"] }\n ```\n3. Re-run cargo check\n\n### 3d: Binary search for minimal features\n\nIf there are many default features, use binary search:\n1. Start with no features\n2. If it fails, add half the default features\n3. Continue until you find the minimal set\n\n## Step 4: Document findings\n\nFor each dependency analyzed, report:\n- Original configuration\n- New configuration (if changed)\n- Features that were removed\n- Any features that are required\n\n## Step 5: Verify full build\n\nAfter all changes, run:\n```bash\ncargo check --workspace --all-targets\ncargo test --workspace --no-run\n```\n\n## Common Patterns\n\n### Serde\nOften only needs `derive`:\n```toml\nserde = { version = \"1.0\", default-features = false, features = [\"derive\", \"std\"] }\n```\n\n### Tokio\nIdentify which runtime features are actually used:\n```toml\ntokio = { version = \"1.0\", default-features = false, features = [\"rt-multi-thread\", \"macros\", \"sync\"] }\n```\n\n### Reqwest\nOften doesn't need all TLS backends:\n```toml\nreqwest = { version = \"0.11\", default-features = false, features = [\"rustls-tls\", \"json\"] }\n```\n\n## Rollback\n\nIf changes cause issues:\n```bash\ngit checkout quickwit/Cargo.toml\ncargo check --workspace\n```\n\n## Tips\n\n- Start with large crates that have many default features (tokio, reqwest, hyper)\n- Use `cargo bloat --crates` to identify large dependencies\n- Check `cargo tree -d` for duplicate dependencies that might indicate feature conflicts\n- Some features are needed only for tests - consider using `[dev-dependencies]` features\n" }, { "path": ".claude/skills/simple-pr/SKILL.md", "content": "---\nname: simple-pr\ndescription: Create a simple PR from staged changes with an auto-generated commit message\ndisable-model-invocation: true\n---\n\n# Simple PR\n\nFollow these steps to create a simple PR from staged changes:\n\n## Step 1: Check workspace state\n\nRun: `git status`\n\nVerify that all changes have been staged (no unstaged changes). If there are unstaged changes, abort and ask the user to stage their changes first with `git add`.\n\nAlso verify that we are on the `main` branch. If not, abort and ask the user to switch to main first.\n\n## Step 2: Ensure main is up to date\n\nRun: `git pull origin main`\n\nThis ensures we're working from the latest code.\n\n## Step 3: Review staged changes\n\nRun: `git diff --cached`\n\nReview the staged changes to understand what the PR will contain.\n\n## Step 4: Generate commit message\n\nBased on the staged changes, generate a concise commit message (1-2 sentences) that describes the \"why\" rather than the \"what\".\n\nDisplay the proposed commit message to the user and ask for confirmation before proceeding.\n\n## Step 5: Create a new branch\n\nGet the git username: `git config user.name | tr ' ' '-' | tr '[:upper:]' '[:lower:]'`\n\nCreate a short, descriptive branch name based on the changes (e.g., `fix-typo-in-readme`, `add-retry-logic`, `update-deps`).\n\nCreate and checkout the branch: `git checkout -b {username}/{short-descriptive-name}`\n\n## Step 6: Commit changes\n\nCommit with the message from step 3:\n```\ngit commit -m \"{commit-message}\"\n```\n\n## Step 7: Push and open a PR\n\nPush the branch and open a PR:\n```\ngit push -u origin {branch-name}\ngh pr create --title \"{commit-message-title}\" --body \"{longer-description-if-needed}\"\n```\n\nReport the PR URL to the user when complete.\n" }, { "path": ".devcontainer/devcontainer.json", "content": "{\n \"name\": \"Quickwit\",\n \"image\": \"mcr.microsoft.com/devcontainers/rust:bookworm\",\n \"customizations\": {\n \"codespaces\": {\n \"openFiles\": [\n \"CONTRIBUTING.md\"\n ]\n },\n \"vscode\": {\n \"extensions\": [\n \"rust-lang.rust-analyzer\"\n ]\n }\n },\n \"hostRequirements\": {\n \"cpus\": 4,\n \"memory\": \"16gb\"\n },\n \"runArgs\": [\n \"--init\"\n ],\n \"mounts\": [\n {\n \"source\": \"/var/run/docker.sock\",\n \"target\": \"/var/run/docker.sock\",\n \"type\": \"bind\"\n }\n ],\n \"features\": {\n \"docker-from-docker\": {\n \"version\": \"latest\",\n \"moby\": true\n },\n \"ghcr.io/devcontainers/features/node:1\": {\n \"version\": \"24\"\n },\n \"ghcr.io/devcontainers/features/aws-cli:1\": {},\n \"ghcr.io/devcontainers-contrib/features/protoc:1\": {}\n },\n \"postCreateCommand\": \".devcontainer/post-create.sh\"\n}\n" }, { "path": ".devcontainer/post-create.sh", "content": "#!/bin/bash\n\n# Define success and error color codes\nSUCCESS_COLOR=\"\\e[32m\"\nERROR_COLOR=\"\\e[31m\"\nRESET_COLOR=\"\\e[0m\"\n\n# Define success tracking variables\nrustupToolchainNightlyInstalled=false\ncmakeInstalled=false\n\n\n# Define installation functions\n\n#Installing manually for now until we figure out why \"ghcr.io/devcontainers-community/features/cmake\": {} is not working\ninstall_cmake() {\n echo -e \"Installing CMake...\"\n sudo apt-get update\n sudo apt-get install -y cmake > /dev/null 2>&1\n if [[ \"$(cmake --version)\" =~ \"cmake version\" ]]; then\n echo -e \"${SUCCESS_COLOR}CMake installed successfully.${RESET_COLOR}\"\n cmakeInstalled=true\n else\n echo -e \"${ERROR_COLOR}CMake installation failed. Please install it manually.${RESET_COLOR}\"\n fi\n}\n\ninstall_rustup_toolchain_nightly() {\n echo -e \"Installing Rustup nightly toolchain...\"\n rustup toolchain install nightly > /dev/null 2>&1\n rustup component add rustfmt --toolchain nightly > /dev/null 2>&1\n if [[ \"$(rustup toolchain list)\" =~ \"nightly\" && \"$(rustup component list --toolchain nightly | grep rustfmt)\" =~ \"installed\" ]]; then\n echo -e \"${SUCCESS_COLOR}Rustup nightly toolchain and rustfmt installed successfully.${RESET_COLOR}\"\n rustupToolchainNightlyInstalled=true\n else\n echo -e \"${ERROR_COLOR}Rustup nightly toolchain and/or rustfmt installation failed. Please install them manually.${RESET_COLOR}\"\n fi\n}\n\n# Install tools\ninstall_cmake\ninstall_rustup_toolchain_nightly\n\n# Copy our custom welcome message to replace the default github welcome message\nsudo cp .devcontainer/welcome.txt /usr/local/etc/vscode-dev-containers/first-run-notice.txt\n\n\n# Check the success tracking variables\nif $rustupToolchainNightlyInstalled && $cmakeInstalled; then\n echo -e \"${SUCCESS_COLOR}All tools installed successfully.${RESET_COLOR}\"\nelse\n echo -e \"${ERROR_COLOR}One or more tools failed to install. Please check the output for errors and install the failed tools manually.${RESET_COLOR}\"\nfi\n" }, { "path": ".devcontainer/welcome.txt", "content": "👋 Welcome to the project!\nAll the necessary tools have already been installed for you 🎉. \nYou can go ahead and start hacking! Happy coding💻.\n\n Here are some useful commands you can run:\n\n🔧 `make test-all` - starts necessary Docker services and runs all tests.\n🔧 `make -k test-all docker-compose-down` - the same as above, but tears down the Docker services after running all the tests.\n🔧 `make fmt` - runs formatter, this command requires the nightly toolchain to be installed by running `rustup toolchain install nightly`.\n🔧 `make fix` - runs formatter and clippy checks.\n🔧 `make typos` - runs the spellcheck tool over the codebase. (Install by running `cargo install typos`)\n🔧 `make build-docs` - builds docs.\n🔧 `make docker-compose-up` - starts Docker services.\n🔧 `make docker-compose-down` - stops Docker services.\n🔧 `make docker-compose-logs` - shows Docker logs.\n\n" }, { "path": ".dockerignore", "content": "**/*.md\n**/*.txt\n**/.*\n**/build\n**/Dockerfile\n**/node_modules\n**/qwdata\n**/target\ndocs\nexamples\n!.git/\n!quickwit-ui/build/.gitignore\n!quickwit-ui/.gitignore_for_build_directory\n" }, { "path": ".gitattributes", "content": "**/codegen/** linguist-generated\n" }, { "path": ".github/ISSUE_TEMPLATE/bug_report.md", "content": "---\nname: Bug report\nabout: Create a report to help us improve\ntitle: \"\"\nlabels: bug\nassignees: \"\"\n---\n\n**Describe the bug**\nA clear and concise description of what the bug is.\n\n**Steps to reproduce (if applicable)**\nSteps to reproduce the behavior:\n\n1.\n2.\n\n**Expected behavior**\nA clear and concise description of what you expected to happen.\n\n**Configuration:**\nPlease provide:\n\n1. Output of `quickwit --version`\n2. The index_config.yaml\n" }, { "path": ".github/ISSUE_TEMPLATE/documentation_request.md", "content": "---\nname: Documentation request\nabout: Suggest a documentation enhancement\ntitle: \"[Documentation topic]\"\nlabels: documentation\nassignees: \"\"\n---\n\n\n\n## My documentation idea\n\nUse this section to give a description of what your enhancement is about.\n\nExamples:\n\n> I would like to add how to configure MinIO storage for Quickwit:\n>\n\n**What do you all think?**\n👍 I would love to see it!\n🚀 I would love to help!\n\nThank you for your request!\n" }, { "path": ".github/ISSUE_TEMPLATE/feature_request.md", "content": "---\nname: Feature request\nabout: Suggest an idea for this project\ntitle: \"\"\nlabels: enhancement\nassignees: \"\"\n---\n\n**Is your feature request related to a problem? Please describe.**\nA clear and concise description of what the problem is. Ex. I'm always frustrated when [...]\n\n**Describe the solution you'd like**\nA clear and concise description of what you want to happen.\n\n**Describe alternatives you've considered**\nA clear and concise description of any alternative solutions or features you've considered.\n\n**Additional context**\nAdd any other context or information about the feature request here.\n" }, { "path": ".github/ISSUE_TEMPLATE/tutorial_request.md", "content": "---\nname: Tutorial request\nabout: Suggest a Quickwit tutorial\ntitle: \"[Tutorial topic]\"\nlabels: tutorial\nassignees: \"\"\n---\n\n\n\n## My tutorial idea\n\nUse this section to give a description of what your tutorial is about.\n\nExamples:\n\n> I would like to write a tutorial that shows how to use Quickwit:\n>\n> - \"for storing traces...\"\n> - \"with Grafana/Jaeger/MinIO...\"\n> - \"for ingesting terabytes per day with Kafka...\"\n\nAre there any particular tools, concepts, languages or platforms that readers\nwill learn about?\n\n**What do you all think?**\n👍 I would love to see it!\n🚀 I would love to help!\n\nThank you for your request!\n" }, { "path": ".github/PULL_REQUEST_TEMPLATE.md", "content": "### Description\n\nDescribe the proposed changes made in this PR.\n\n### How was this PR tested?\n\nDescribe how you tested this PR.\n" }, { "path": ".github/actions/cargo-build-macos-binary/action.yml", "content": "name: \"Build Quickwit binary for macOS\"\ndescription: \"Build React app and Rust binary for macOS with cargo build.\"\ninputs:\n target:\n description: \"Target\"\n required: true\n version:\n description: \"Binary version\"\n required: true\n token:\n description: \"GitHub access token\"\n required: true\nruns:\n using: \"composite\"\n steps:\n - run: echo \"ASSET_FULL_NAME=quickwit-${{ inputs.version }}-${{ inputs.target }}\" >> $GITHUB_ENV\n shell: bash\n - uses: actions/setup-node@v3\n with:\n node-version: 24\n cache: \"yarn\"\n cache-dependency-path: quickwit/quickwit-ui/yarn.lock\n - run: yarn global add node-gyp\n shell: bash\n - run: make build-ui\n shell: bash\n - name: Install protoc\n run: brew install protobuf\n shell: bash\n - name: Install rustup\n shell: bash\n run: curl https://sh.rustup.rs -sSf | sh -s -- --default-toolchain none -y\n - name: Add target ${{ inputs.target }}\n run: rustup target add ${{ inputs.target }}\n shell: bash\n working-directory: ./quickwit\n - name: Retrieve and export commit date, hash, and tags\n run: |\n echo \"QW_COMMIT_DATE=$(TZ=UTC0 git log -1 --format=%cd --date=format-local:%Y-%m-%dT%H:%M:%SZ)\" >> $GITHUB_ENV\n echo \"QW_COMMIT_HASH=$(git rev-parse HEAD)\" >> $GITHUB_ENV\n echo \"QW_COMMIT_TAGS=$(git tag --points-at HEAD | tr '\\n' ',')\" >> $GITHUB_ENV\n shell: bash\n - name: Build binary\n run: cargo build --release --features release-macos-feature-vendored-set --target ${{ matrix.target }} --bin quickwit\n shell: bash\n working-directory: ./quickwit\n env:\n QW_COMMIT_DATE: ${{ env.QW_COMMIT_DATE }}\n QW_COMMIT_HASH: ${{ env.QW_COMMIT_HASH }}\n QW_COMMIT_TAGS: ${{ env.QW_COMMIT_TAGS }}\n - name: Bundle archive\n run: |\n make archive BINARY_FILE=quickwit/target/${{ inputs.target }}/release/quickwit \\\n BINARY_VERSION=${{ inputs.version }} ARCHIVE_NAME=${{ env.ASSET_FULL_NAME }}\n shell: bash\n - name: Save binary archive for three days\n uses: actions/upload-artifact@v4.4.0\n with:\n name: ${{ env.ASSET_FULL_NAME }}.tar.gz\n path: ./${{ env.ASSET_FULL_NAME }}.tar.gz\n retention-days: 3\n - name: Deploy archive to GitHub release\n uses: quickwit-inc/upload-to-github-release@9b2c40fba23bf8dea05b7d2eece24cbc95d4a190\n env:\n GITHUB_TOKEN: ${{ inputs.token }}\n with:\n file: ${{ env.ASSET_FULL_NAME }}.tar.gz\n overwrite: true\n draft: ${{ inputs.version != 'nightly' }}\n tag_name: ${{ inputs.version }}\n" }, { "path": ".github/actions/cross-build-binary/action.yml", "content": "name: \"Build Quickwit binary with cargo cross\"\ndescription: \"Build React app and Rust binary with cargo cross.\"\ninputs:\n target:\n description: \"Target\"\n required: true\n version:\n description: \"Binary version\"\n required: true\n token:\n description: \"GitHub access token\"\n required: true\nruns:\n using: \"composite\"\n steps:\n - run: echo \"ASSET_FULL_NAME=quickwit-${{ inputs.version }}-${{ inputs.target }}\" >> $GITHUB_ENV\n shell: bash\n - uses: actions/setup-node@v3\n with:\n node-version: 24\n cache: \"yarn\"\n cache-dependency-path: quickwit/quickwit-ui/yarn.lock\n - run: yarn global add node-gyp\n shell: bash\n - run: make build-ui\n shell: bash\n - name: Install rustup\n shell: bash\n run: curl https://sh.rustup.rs -sSf | sh -s -- --default-toolchain none -y\n - name: Install cross\n run: cargo install cross\n shell: bash\n - name: Retrieve and export commit date, hash, and tags\n run: |\n echo \"QW_COMMIT_DATE=$(TZ=UTC0 git log -1 --format=%cd --date=format-local:%Y-%m-%dT%H:%M:%SZ)\" >> $GITHUB_ENV\n echo \"QW_COMMIT_HASH=$(git rev-parse HEAD)\" >> $GITHUB_ENV\n echo \"QW_COMMIT_TAGS=$(git tag --points-at HEAD | tr '\\n' ',')\" >> $GITHUB_ENV\n shell: bash\n - name: Build Quickwit\n run: cross build --release --features release-feature-vendored-set --target ${{ inputs.target }} --bin quickwit\n shell: bash\n env:\n QW_COMMIT_DATE: ${{ env.QW_COMMIT_DATE }}\n QW_COMMIT_HASH: ${{ env.QW_COMMIT_HASH }}\n QW_COMMIT_TAGS: ${{ env.QW_COMMIT_TAGS }}\n working-directory: ./quickwit\n - name: Bundle archive\n run: |\n make archive BINARY_FILE=quickwit/target/${{ inputs.target }}/release/quickwit \\\n BINARY_VERSION=${{ inputs.version }} ARCHIVE_NAME=${{ env.ASSET_FULL_NAME }}\n shell: bash\n - name: Save binary archive for three days\n uses: actions/upload-artifact@v4.4.0\n with:\n name: ${{ env.ASSET_FULL_NAME }}.tar.gz\n path: ./${{ env.ASSET_FULL_NAME }}.tar.gz\n retention-days: 3\n - name: Upload archive\n uses: quickwit-inc/upload-to-github-release@9b2c40fba23bf8dea05b7d2eece24cbc95d4a190\n env:\n GITHUB_TOKEN: ${{ inputs.token }}\n with:\n file: ${{ env.ASSET_FULL_NAME }}.tar.gz\n overwrite: true\n draft: ${{ inputs.version != 'nightly' }}\n tag_name: ${{ inputs.version }}\n" }, { "path": ".github/dependabot.yml", "content": "version: 2\nupdates:\n # Rust dependencies\n - package-ecosystem: cargo\n directory: \"/quickwit\"\n schedule:\n interval: \"monthly\"\n groups:\n rust-dependencies:\n patterns:\n - \"*\"\n open-pull-requests-limit: 10\n ignore:\n - dependency-name: \"*\"\n update-types: [\"version-update:semver-patch\"]\n\n # Docker dependencies\n - package-ecosystem: docker\n directory: \"/\"\n schedule:\n interval: \"monthly\"\n open-pull-requests-limit: 10\n\n # GitHub Actions\n - package-ecosystem: github-actions\n directory: \"/\"\n schedule:\n interval: \"monthly\"\n groups:\n github-actions:\n patterns:\n - \"*\"\n open-pull-requests-limit: 10\n\n # NPM dependencies\n - package-ecosystem: npm\n directory: \"/\"\n schedule:\n interval: \"monthly\"\n groups:\n npm-dependencies:\n patterns:\n - \"*\"\n open-pull-requests-limit: 10\n" }, { "path": ".github/workflows/ci.yml", "content": "name: CI\n\non:\n workflow_dispatch:\n pull_request:\n push:\n branches:\n - main\n - trigger-ci-workflow\n paths:\n - \"quickwit/**\"\n - \"!quickwit/quickwit-ui/**\"\n\npermissions:\n contents: read\n\nenv:\n CARGO_INCREMENTAL: 0\n QW_DISABLE_TELEMETRY: 1\n QW_TEST_DATABASE_URL: postgres://quickwit-dev:quickwit-dev@localhost:5432/quickwit-metastore-dev\n RUST_BACKTRACE: 1\n RUSTDOCFLAGS: -Dwarnings -Arustdoc::private_intra_doc_links\n RUSTFLAGS: -Dwarnings --cfg tokio_unstable\n\n# Ensures that we cancel running jobs for the same PR / same workflow.\nconcurrency:\n group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}\n cancel-in-progress: true\n\njobs:\n tests:\n name: Unit tests\n runs-on: \"ubuntu-latest\"\n timeout-minutes: 60\n permissions:\n contents: read\n actions: write\n services:\n # PostgreSQL service container\n postgres:\n image: postgres:latest\n ports:\n - 5432:5432\n env:\n POSTGRES_USER: quickwit-dev\n POSTGRES_PASSWORD: quickwit-dev\n POSTGRES_DB: quickwit-metastore-dev\n # Set health checks to wait until postgres has started\n options: >-\n --health-cmd pg_isready\n --health-interval 10s\n --health-timeout 5s\n --health-retries 5\n steps:\n - name: Cleanup Disk Space\n run: |\n df -h\n\n if [ \"$(df -BG / | awk 'NR==2 {gsub(\"G\",\"\",$4); print $4}')\" -lt 30 ]; then\n echo \"Less than 30GiB available. Running cleanup...\"\n sudo rm -rf /usr/share/dotnet\n sudo rm -rf /usr/local/lib/android\n sudo rm -rf /usr/share/swift\n sudo rm -rf /usr/local/.ghcup\n sudo rm -rf /opt/hostedtoolcache/CodeQL\n df -h\n else\n echo \"30GiB or more available. Skipping cleanup.\"\n fi\n\n - uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1\n - name: Install Ubuntu packages\n run: |\n sudo apt-get update\n sudo apt-get -y install protobuf-compiler\n - uses: actions/setup-python@83679a892e2d95755f2dac6acb0bfd1e9ac5d548 # v.6.1.0\n with:\n python-version: '3.11'\n - uses: dorny/paths-filter@de90cc6fb38fc0963ad72b210f1f284cd68cea36 # v3.0.2\n id: modified\n with:\n filters: |\n rust_src:\n - quickwit/**/*.rs\n - quickwit/**/*.toml\n - quickwit/**/*.proto\n - quickwit/rest-api-tests/**\n - .github/workflows/ci.yml\n - name: Setup stable Rust Toolchain\n if: steps.modified.outputs.rust_src == 'true'\n uses: dtolnay/rust-toolchain@f7ccc83f9ed1e5b9c81d8a67d7ad1a747e22a561 # master\n with:\n toolchain: stable\n - name: Setup cache\n uses: Swatinem/rust-cache@779680da715d629ac1d338a641029a2f4372abb5 # v2.8.2\n if: steps.modified.outputs.rust_src == 'true'\n with:\n workspaces: \"./quickwit -> target\"\n shared-key: \"quickwit-cargo\"\n - name: Install nextest\n if: always() && steps.modified.outputs.rust_src == 'true'\n uses: taiki-e/install-action@aba36d755ec7ca22d38b12111787c26115943952\n with:\n tool: cargo-nextest\n - name: cargo build\n if: always() && steps.modified.outputs.rust_src == 'true'\n run: cargo build --features=postgres --tests --bin quickwit\n working-directory: ./quickwit\n - name: cargo nextest\n if: always() && steps.modified.outputs.rust_src == 'true'\n run: cargo nextest run --features=postgres --retries 1\n working-directory: ./quickwit\n - name: Install python packages\n if: always() && steps.modified.outputs.rust_src == 'true'\n run: |\n pip install --user --require-hashes -r ${{ github.workspace }}/.github/workflows/requirements.txt\n pipenv install --deploy --ignore-pipfile\n working-directory: ./quickwit/rest-api-tests\n - name: Run REST API tests\n if: always() && steps.modified.outputs.rust_src == 'true'\n run: pipenv run python3 ./run_tests.py --binary ../target/debug/quickwit\n working-directory: ./quickwit/rest-api-tests\n\n lints:\n name: Lints\n runs-on: \"ubuntu-latest\"\n timeout-minutes: 60\n permissions:\n contents: read\n actions: write\n steps:\n - uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1\n - uses: dorny/paths-filter@de90cc6fb38fc0963ad72b210f1f284cd68cea36 # v3.0.2\n id: modified\n with:\n filters: |\n rust_src:\n - quickwit/**/*.rs\n - quickwit/**/*.toml\n - quickwit/**/*.proto\n - .github/workflows/ci.yml\n - name: Install Ubuntu packages\n if: always() && steps.modified.outputs.rust_src == 'true'\n run: |\n sudo apt-get update\n sudo apt-get -y install protobuf-compiler\n - name: Setup nightly Rust Toolchain (for rustfmt)\n if: steps.modified.outputs.rust_src == 'true'\n uses: dtolnay/rust-toolchain@f7ccc83f9ed1e5b9c81d8a67d7ad1a747e22a561 # master\n with:\n toolchain: nightly\n components: rustfmt\n - name: Setup stable Rust Toolchain\n if: steps.modified.outputs.rust_src == 'true'\n uses: dtolnay/rust-toolchain@f7ccc83f9ed1e5b9c81d8a67d7ad1a747e22a561 # master\n with:\n toolchain: stable\n - name: Setup cache\n if: steps.modified.outputs.rust_src == 'true'\n uses: Swatinem/rust-cache@779680da715d629ac1d338a641029a2f4372abb5 # v2.8.2\n with:\n workspaces: \"./quickwit -> target\"\n shared-key: \"quickwit-cargo\"\n - name: Install cargo deny\n if: always() && steps.modified.outputs.rust_src == 'true'\n uses: taiki-e/cache-cargo-install-action@34ce5120836e5f9f1508d8713d7fdea0e8facd6f # v3.0.1\n with:\n # 0.18 requires rustc 1.85\n tool: cargo-deny@0.17.0\n - name: Install cargo machete\n if: always() && steps.modified.outputs.rust_src == 'true'\n uses: taiki-e/cache-cargo-install-action@34ce5120836e5f9f1508d8713d7fdea0e8facd6f # v3.0.1\n with:\n tool: cargo-machete\n - name: cargo clippy\n if: always() && steps.modified.outputs.rust_src == 'true'\n run: cargo clippy --workspace --tests --all-features\n working-directory: ./quickwit\n - name: cargo deny\n if: always() && steps.modified.outputs.rust_src == 'true'\n run: cargo deny check licenses\n working-directory: ./quickwit\n - name: cargo machete\n if: always() && steps.modified.outputs.rust_src == 'true'\n run: cargo machete\n working-directory: ./quickwit\n - name: cargo doc\n if: always() && steps.modified.outputs.rust_src == 'true'\n run: cargo doc --no-deps\n working-directory: ./quickwit\n - name: License headers check\n if: always()\n run: bash scripts/check_license_headers.sh\n working-directory: ./quickwit\n - name: rustfmt\n if: always() && steps.modified.outputs.rust_src == 'true'\n run: cargo +nightly fmt --all -- --check\n working-directory: ./quickwit\n\n thirdparty-license:\n name: Check Datadog third-party license file\n runs-on: ubuntu-latest\n permissions:\n contents: read\n actions: write\n steps:\n - uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1\n - name: Install Rust toolchain\n uses: dtolnay/rust-toolchain@f7ccc83f9ed1e5b9c81d8a67d7ad1a747e22a561 # master\n with:\n toolchain: stable\n\n - name: Cache cargo tools\n uses: actions/cache@9255dc7a253b0ccc959486e2bca901246202afeb # v5.0.1\n with:\n path: ~/.cargo/bin\n key: ${{ runner.os }}-cargo-tools-${{ hashFiles('**/Cargo.lock') }}\n\n - name: Install dd-rust-license-tool\n run: dd-rust-license-tool --help || cargo install --git https://github.com/DataDog/rust-license-tool.git --force\n\n - name: Check Datadog third-party license file\n run: dd-rust-license-tool --config quickwit/license-tool.toml --manifest-path quickwit/Cargo.toml check\n" }, { "path": ".github/workflows/coverage.yml", "content": "name: Code coverage\n\non:\n workflow_dispatch:\n push:\n branches:\n - main\n - trigger-coverage-workflow\n paths:\n - quickwit/Cargo.toml\n - quickwit/Cargo.lock\n - quickwit/quickwit-*/**\n\npermissions:\n contents: read\n\nenv:\n AWS_REGION: us-east-1\n AWS_ACCESS_KEY_ID: \"placeholder\"\n AWS_SECRET_ACCESS_KEY: \"placeholder\"\n CARGO_INCREMENTAL: 0\n PUBSUB_EMULATOR_HOST: \"localhost:8681\"\n QW_DISABLE_TELEMETRY: 1\n QW_S3_ENDPOINT: \"http://localhost:4566\" # Services are exposed as localhost because we are not running coverage in a container.\n QW_S3_FORCE_PATH_STYLE_ACCESS: 1\n QW_TEST_DATABASE_URL: postgres://quickwit-dev:quickwit-dev@localhost:5432/quickwit-metastore-dev\n RUSTFLAGS: -Dwarnings --cfg tokio_unstable\n\njobs:\n test:\n name: Coverage\n runs-on: gh-ubuntu-arm64\n timeout-minutes: 40\n permissions:\n contents: read\n actions: write\n # Setting a containing will require to fix the QW_S3_ENDPOINT to http://localstack:4566\n services:\n localstack:\n image: localstack/localstack:latest\n ports:\n - \"4566:4566\"\n - \"4571:4571\"\n - \"8080:8080\"\n env:\n SERVICES: kinesis,s3,sqs\n options: >-\n --health-cmd \"curl -k https://localhost:4566\"\n --health-interval 10s\n --health-timeout 5s\n --health-retries 5\n\n postgres:\n image: postgres:latest\n ports:\n - \"5432:5432\"\n env:\n POSTGRES_USER: quickwit-dev\n POSTGRES_PASSWORD: quickwit-dev\n POSTGRES_DB: quickwit-metastore-dev\n options: >-\n --health-cmd pg_isready\n --health-interval 10s\n --health-timeout 5s\n --health-retries 5\n\n kafka-broker:\n image: confluentinc/confluent-local:7.4.11\n ports:\n - \"9092:9092\"\n - \"9101:9101\"\n env:\n # Mode KRaft (Single Node)\n KAFKA_NODE_ID: 1\n KAFKA_PROCESS_ROLES: 'broker,controller'\n KAFKA_CONTROLLER_QUORUM_VOTERS: '1@localhost:9093'\n KAFKA_LOG4J_LOGGERS: \"org.apache.kafka.image.loader.MetadataLoader=WARN\"\n\n # Listeners\n KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: 'CONTROLLER:PLAINTEXT,EXTERNAL:PLAINTEXT'\n KAFKA_LISTENERS: 'EXTERNAL://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093'\n KAFKA_ADVERTISED_LISTENERS: 'EXTERNAL://localhost:9092'\n KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'\n KAFKA_INTER_BROKER_LISTENER_NAME: 'EXTERNAL'\n\n # Configuration simplifiée\n KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1\n KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0\n KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1\n KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1\n\n # ID du Cluster (Nécessaire pour KRaft)\n CLUSTER_ID: 'MkU3OEVBNTcwNTJENDM2Qk'\n\n KAFKA_HEAP_OPTS: -Xms256M -Xmx256M\n\n options: >-\n --health-cmd \"ub kafka-ready -b localhost:9092 1 5\"\n --health-interval 10s\n --health-timeout 5s\n --health-retries 5\n\n gcp-pubsub-emulator:\n image: thekevjames/gcloud-pubsub-emulator:550.0.0\n ports:\n - \"8681:8681\"\n env:\n PUBSUB_PROJECT1: \"quickwit-emulator,emulator_topic:emulator_subscription\"\n\n steps:\n - uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1\n\n - name: Install lib libsasl2\n run: |\n sudo apt update\n sudo apt install libsasl2-dev\n sudo apt install libsasl2-2\n\n - uses: actions/setup-python@83679a892e2d95755f2dac6acb0bfd1e9ac5d548 # v.6.1.0\n with:\n python-version: '3.11'\n\n - uses: actions/cache@9255dc7a253b0ccc959486e2bca901246202afeb # v5.0.1\n with:\n path: |\n ~/.cargo/git\n ~/.cargo/registry\n key: ${{ runner.os }}-cargo-test-${{ hashFiles('Cargo.lock') }}\n restore-keys: |\n ${{ runner.os }}-cargo-test-${{ hashFiles('Cargo.lock') }}\n ${{ runner.os }}-cargo-test\n\n - name: Install python packages\n run: |\n pip install --user --require-hashes -r ${{ github.workspace }}/.github/workflows/requirements.txt\n pipenv install --deploy --ignore-pipfile\n working-directory: ./quickwit/quickwit-cli/tests\n\n - name: Prepare LocalStack S3\n run: pipenv run ./prepare_tests.sh\n working-directory: ./quickwit/quickwit-cli/tests\n\n # GitHub Actions does not allow services to be started with a custom command,\n # so we are running Azurite as a container manually.\n - name: Run Azurite service\n run: DOCKER_SERVICES=azurite make docker-compose-up\n\n # GitHub Actions does not allow services to be started with a custom command,\n # so we are running fake gcs server as a container manually.\n - name: Run Fake GCS Server service\n run: DOCKER_SERVICES=fake-gcs-server make docker-compose-up\n\n - name: Run Pulsar service\n run: DOCKER_SERVICES=pulsar make docker-compose-up\n\n - name: Install Rust\n run: rustup update stable\n\n - name: Install cargo-llvm-cov, cargo-nextest, and protoc\n uses: taiki-e/install-action@90558ad1e179036f31467972b00dec6cb80701fa # v2.66.3\n with:\n tool: cargo-llvm-cov,nextest,protoc\n\n # We limit the number of jobs to 4 to avoid OOM errors when linking the binary.\n - name: Generate code coverage\n run: |\n cargo llvm-cov clean --workspace\n cargo llvm-cov nextest --no-report --test failpoints --features fail/failpoints --retries 4\n # increase stack size for test_all_with_s3_localstack_cli, see quickwit#4963\n RUST_MIN_STACK=67108864 CARGO_BUILD_JOBS=4 cargo llvm-cov nextest --no-report --all-features --retries 4\n cargo llvm-cov report --lcov --output-path lcov.info\n working-directory: ./quickwit\n\n - name: Upload coverage to Codecov\n uses: codecov/codecov-action@671740ac38dd9b0130fbe1cec585b89eea48d3de # v5.5.2\n with:\n token: ${{ secrets.CODECOV_TOKEN }} # not required for public repos\n files: ./quickwit/lcov.info\n\n on-failure:\n if: ${{ github.repository_owner == 'quickwit-oss' && failure() }}\n name: On Failure\n needs: [test]\n runs-on: ubuntu-latest\n steps:\n - name: Send Message\n uses: sarisia/actions-status-discord@eb045afee445dc055c18d3d90bd0f244fd062708 # v1.16.0\n with:\n webhook: ${{ secrets.DISCORD_WEBHOOK }}\n nodetail: true\n color: \"#FF0000\"\n title: \"\"\n description: |\n ### ❌ [${{ github.event.pull_request.title }}](${{ github.event.pull_request.html_url }})\n\n @${{ github.actor }} quickwit coverage CI failed on your PR.\n\n Coverage CI contains tests that are not running in the regular CI because they are too lengthy.\n For this reason it is possible for it to break even if the tests were passing on your PR.\n This is not a catastrophy, but you are responsible for fixing it!\n\n You can run the full test suite locally with `make test-all`.\n\n Please report in this channel that you are working on it/fixed it/or if it is a flaky test/\n or if you need help.\n\n **[View logs](https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }})**\n" }, { "path": ".github/workflows/dependency.yml", "content": "name: \"Dependency Review\"\non: [pull_request]\n\npermissions:\n contents: read\n\n# Ensures that we cancel running jobs for the same PR / same workflow.\nconcurrency:\n group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}\n cancel-in-progress: true\n\njobs:\n dependency-review:\n runs-on: ubuntu-latest\n steps:\n - name: \"Checkout Repository\"\n uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1\n - name: \"Dependency Review\"\n uses: actions/dependency-review-action@98884d411b0f1c583e5ee579e7e897d4623019c2 # v4.8.1\n with:\n # This is an minor vuln on the rsa crate, used for\n # google storage.\n allow-ghsas: GHSA-c38w-74pg-36hr,GHSA-4grx-2x9w-596c\n" }, { "path": ".github/workflows/publish_cross_images.yml", "content": "name: Publish custom cross images\n\non:\n workflow_dispatch:\n push:\n branches:\n - main\n paths:\n - \"build/cross-images/**\"\n\npermissions:\n contents: read\n\njobs:\n build-cross-images:\n name: Publish cross images\n runs-on: ubuntu-latest\n environment:\n name: production\n steps:\n - name: Check out the repo\n uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1\n - name: Log in to Docker Hub\n uses: docker/login-action@5e57cd118135c172c3672efd75eb46360885c0ef # v3.6.0\n with:\n username: ${{ secrets.DOCKERHUB_USERNAME }}\n password: ${{ secrets.DOCKERHUB_ACCESS_TOKEN }}\n - name: Build and push cross images\n run: make cross-images\n" }, { "path": ".github/workflows/publish_docker_images.yml", "content": "name: Build and publish Docker images\n\non:\n workflow_dispatch:\n push:\n branches:\n - main\n - release-0.9\n paths:\n - \"quickwit/**\"\n tags:\n - airmail\n - happy-plazza\n - qw*\n - v*\n\npermissions:\n contents: read\n\nenv:\n REGISTRY_IMAGE: quickwit/quickwit\n\njobs:\n docker:\n strategy:\n matrix:\n include:\n - os: ubuntu-latest\n platform: linux/amd64\n platform_suffix: amd64\n - os: gh-ubuntu-arm64\n platform: linux/arm64\n platform_suffix: arm64\n runs-on: ${{ matrix.os }}\n permissions:\n contents: read\n actions: write\n environment:\n name: production\n steps:\n - name: Cleanup Disk Space\n run: |\n df -h\n sudo rm -rf /opt/hostedtoolcache/CodeQL\n sudo rm -rf /usr/local/.ghcup\n sudo rm -rf /usr/local/lib/android\n sudo rm -rf /usr/share/dotnet\n sudo rm -rf /usr/share/swift\n df -h\n\n - name: Checkout\n uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1\n\n - name: Login to Docker Hub\n uses: docker/login-action@5e57cd118135c172c3672efd75eb46360885c0ef # v3.6.0\n with:\n username: ${{ secrets.DOCKERHUB_USERNAME }}\n password: ${{ secrets.DOCKERHUB_ACCESS_TOKEN }}\n\n - name: Set up QEMU\n uses: docker/setup-qemu-action@c7c53464625b32c7a7e944ae62b3e17d2b600130 # v3.7.0\n\n - name: Set up Docker Buildx\n uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f # v3.12.0\n\n - name: Docker meta\n id: meta\n uses: docker/metadata-action@c299e40c65443455700f0fdfc63efafe5b349051 # v5.10.0\n with:\n images: |\n ${{ env.REGISTRY_IMAGE }}\n labels: |\n org.opencontainers.image.title=Quickwit\n maintainer=Quickwit, Inc. \n org.opencontainers.image.vendor=Quickwit, Inc.\n org.opencontainers.image.licenses=Apache-2.0\n\n - name: Retrieve commit date, hash, and tags\n run: |\n echo \"QW_COMMIT_DATE=$(TZ=UTC0 git log -1 --format=%cd --date=format-local:%Y-%m-%dT%H:%M:%SZ)\" >> $GITHUB_ENV\n echo \"QW_COMMIT_HASH=$(git rev-parse HEAD)\" >> $GITHUB_ENV\n echo \"QW_COMMIT_TAGS=$(git tag --points-at HEAD | tr '\\n' ',')\" >> $GITHUB_ENV\n if [[ \"${{ github.event_name }}\" == \"push\" && \"${{ github.ref_type }}\" == \"tag\" && \"${GITHUB_REF#refs/tags/}\" == *\"jemprof\"* ]]; then\n echo \"CARGO_FEATURES=release-jemalloc-profiled\" >> $GITHUB_ENV\n else\n echo \"CARGO_FEATURES=release-feature-set\" >> $GITHUB_ENV\n fi\n\n - name: Build and push image\n uses: docker/build-push-action@263435318d21b8e681c14492fe198d362a7d2c83 # v6.18.0\n id: build\n with:\n context: .\n platforms: ${{ matrix.platform }}\n build-args: |\n QW_COMMIT_DATE=${{ env.QW_COMMIT_DATE }}\n QW_COMMIT_HASH=${{ env.QW_COMMIT_HASH }}\n QW_COMMIT_TAGS=${{ env.QW_COMMIT_TAGS }}\n CARGO_FEATURES=${{ env.CARGO_FEATURES }}\n labels: ${{ steps.meta.outputs.labels }}\n outputs: type=image,name=${{ env.REGISTRY_IMAGE }},push-by-digest=true,name-canonical=true,push=true\n\n - name: Export digest\n run: |\n mkdir -p /tmp/digests\n digest=\"${{ steps.build.outputs.digest }}\"\n touch \"/tmp/digests/${digest#sha256:}\"\n\n - name: Upload digest\n uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6.0.0\n with:\n name: digest-${{ matrix.platform_suffix }}\n path: /tmp/digests/*\n if-no-files-found: error\n retention-days: 1\n\n merge:\n runs-on: ubuntu-latest\n needs: [docker]\n permissions:\n contents: read\n actions: read\n environment: production\n steps:\n - name: Download digests\n uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7.0.0\n with:\n pattern: digest-*\n path: /tmp/digests\n merge-multiple: true\n\n - name: Set up Docker Buildx\n uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f # v3.12.0\n\n - name: Docker meta\n id: meta\n uses: docker/metadata-action@c299e40c65443455700f0fdfc63efafe5b349051 # v5.10.0\n with:\n images: ${{ env.REGISTRY_IMAGE }}\n flavor: |\n latest=false\n tags: |\n type=edge,branch=main\n type=edge,branch=main,suffix=-slim-bookworm\n type=semver,pattern={{version}}\n type=semver,pattern={{version}},value=latest\n type=semver,pattern={{version}},suffix=-slim-bookworm\n type=ref,event=tag\n type=raw,value=v0.9.0-rc,enable=${{ github.ref == 'refs/heads/release-0.9' }}\n - name: Login to Docker Hub\n uses: docker/login-action@5e57cd118135c172c3672efd75eb46360885c0ef # v3.6.0\n with:\n username: ${{ secrets.DOCKERHUB_USERNAME }}\n password: ${{ secrets.DOCKERHUB_ACCESS_TOKEN }}\n - name: Create manifest list and push tags\n working-directory: /tmp/digests\n run: |\n docker buildx imagetools create $(jq -cr '.tags | map(\"-t \" + .) | join(\" \")' <<< \"$DOCKER_METADATA_OUTPUT_JSON\") \\\n $(printf '${{ env.REGISTRY_IMAGE }}@sha256:%s ' *)\n - name: Inspect image\n run: |\n docker buildx imagetools inspect ${{ env.REGISTRY_IMAGE }}:${{ steps.meta.outputs.version }}\n" }, { "path": ".github/workflows/publish_lambda.yaml", "content": "# This workflow creates a new release for a quickwit search aws lambda.\n# The artifact is a zip file containing a binary for ARM 64,\n# ready to be deployed by the deployer.\n#\n# See quickwit-lambda-client/README.md\nname: Release Lambda binary\n\non:\n push:\n tags:\n - 'lambda-*'\n workflow_dispatch:\n inputs:\n version:\n description: 'Version tag (e.g., v0.8.0)'\n required: false\n default: 'dev'\n\npermissions:\n contents: read\n\njobs:\n build-lambda:\n name: Build Lambda ARM64\n runs-on: ubuntu-latest\n permissions:\n contents: write\n actions: write\n steps:\n - uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1\n\n - name: Set version\n run: |\n if [ \"${{ github.ref_type }}\" = \"tag\" ]; then\n # Extract version from tag (e.g., lambda-v0.8.0 -> v0.8.0)\n echo \"ASSET_VERSION=${GITHUB_REF_NAME#lambda-}\" >> $GITHUB_ENV\n elif [ -n \"${{ github.event.inputs.version }}\" ] && [ \"${{ github.event.inputs.version }}\" != \"dev\" ]; then\n echo \"ASSET_VERSION=${{ github.event.inputs.version }}\" >> $GITHUB_ENV\n else\n echo \"ASSET_VERSION=dev-$(git rev-parse --short HEAD)\" >> $GITHUB_ENV\n fi\n\n - name: Install rustup\n run: curl https://sh.rustup.rs -sSf | sh -s -- --default-toolchain none -y\n\n - name: Install cross\n run: cargo install cross\n\n - name: Retrieve and export commit date, hash, and tags\n run: |\n echo \"QW_COMMIT_DATE=$(TZ=UTC0 git log -1 --format=%cd --date=format-local:%Y-%m-%dT%H:%M:%SZ)\" >> $GITHUB_ENV\n echo \"QW_COMMIT_HASH=$(git rev-parse HEAD)\" >> $GITHUB_ENV\n echo \"QW_COMMIT_TAGS=$(git tag --points-at HEAD | tr '\\n' ',')\" >> $GITHUB_ENV\n\n - name: Build Lambda binary\n run: cross build --release --features lambda-release --target aarch64-unknown-linux-gnu -p quickwit-lambda-server --bin quickwit-aws-lambda-leaf-search\n env:\n QW_COMMIT_DATE: ${{ env.QW_COMMIT_DATE }}\n QW_COMMIT_HASH: ${{ env.QW_COMMIT_HASH }}\n QW_COMMIT_TAGS: ${{ env.QW_COMMIT_TAGS }}\n working-directory: ./quickwit\n\n - name: Create Lambda zip\n run: |\n cd quickwit/target/aarch64-unknown-linux-gnu/release\n cp quickwit-aws-lambda-leaf-search bootstrap\n zip quickwit-aws-lambda-${{ env.ASSET_VERSION }}-aarch64.zip bootstrap\n mv quickwit-aws-lambda-${{ env.ASSET_VERSION }}-aarch64.zip ../../../../\n\n - name: Upload to GitHub release\n uses: quickwit-inc/upload-to-github-release@9b2c40fba23bf8dea05b7d2eece24cbc95d4a190\n env:\n GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}\n with:\n file: quickwit-aws-lambda-${{ env.ASSET_VERSION }}-aarch64.zip\n overwrite: true\n draft: true\n tag_name: ${{ env.ASSET_VERSION }}\n" }, { "path": ".github/workflows/publish_nightly_packages.yml", "content": "name: Build and publish nightly packages\n\non:\n workflow_dispatch:\n schedule:\n - cron: \"0 5 * * *\"\n\npermissions:\n contents: read\n\njobs:\n build-macos-binaries:\n name: Build ${{ matrix.target }}\n runs-on: macos-latest\n permissions:\n contents: write\n actions: write\n strategy:\n fail-fast: false\n matrix:\n target: [x86_64-apple-darwin, aarch64-apple-darwin]\n steps:\n - uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1\n - uses: ./.github/actions/cargo-build-macos-binary\n with:\n target: ${{ matrix.target }}\n version: nightly\n token: ${{ secrets.GITHUB_TOKEN }}\n build-linux-binaries:\n strategy:\n fail-fast: false\n matrix:\n target: [x86_64-unknown-linux-gnu, aarch64-unknown-linux-gnu]\n name: Build ${{ matrix.target }}\n runs-on: ubuntu-latest\n permissions:\n contents: write\n actions: write\n steps:\n - uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1\n - uses: ./.github/actions/cross-build-binary\n with:\n target: ${{ matrix.target }}\n version: nightly\n token: ${{ secrets.GITHUB_TOKEN }}\n" }, { "path": ".github/workflows/publish_release_packages.yml", "content": "name: Build and publish release packages\n\non:\n push:\n tags:\n - \"v*\"\n\npermissions:\n contents: read\n\njobs:\n build-macos-binaries:\n name: Build ${{ matrix.target }}\n runs-on: macos-latest\n permissions:\n contents: write\n actions: write\n strategy:\n matrix:\n target: [x86_64-apple-darwin, aarch64-apple-darwin]\n\n steps:\n - uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1\n - name: Extract asset version\n run: echo \"ASSET_VERSION=${GITHUB_REF/refs\\/tags\\//}\" >> $GITHUB_ENV\n - uses: ./.github/actions/cargo-build-macos-binary\n with:\n target: ${{ matrix.target }}\n version: ${{ env.ASSET_VERSION }}\n token: ${{ secrets.GITHUB_TOKEN }}\n\n build-linux-binaries:\n strategy:\n matrix:\n target: [x86_64-unknown-linux-gnu, aarch64-unknown-linux-gnu]\n name: Build ${{ matrix.target }}\n runs-on: ubuntu-latest\n permissions:\n contents: write\n actions: write\n steps:\n - uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1\n - name: Extract asset version\n run: echo \"ASSET_VERSION=${GITHUB_REF/refs\\/tags\\//}\" >> $GITHUB_ENV\n - uses: ./.github/actions/cross-build-binary\n with:\n target: ${{ matrix.target }}\n version: ${{ env.ASSET_VERSION }}\n token: ${{ secrets.GITHUB_TOKEN }}\n" }, { "path": ".github/workflows/requirements.txt", "content": "# contains pinned dependencies for installing pipenv to ensure repeatable builds in CI/CD workflows\ncertifi==2025.10.5 \\\n --hash=sha256:0f212c2744a9bb6de0c56639a6f68afe01ecd92d91f14ae897c4fe7bbeeef0de \\\n --hash=sha256:47c09d31ccf2acf0be3f701ea53595ee7e0b8fa08801c6624be771df09ae7b43\ndistlib==0.4.0 \\\n --hash=sha256:9659f7d87e46584a30b5780e43ac7a2143098441670ff0a49d5f9034c54a6c16 \\\n --hash=sha256:feec40075be03a04501a973d81f633735b4b69f98b05450592310c0f401a4e0d\nfilelock==3.20.3 \\\n --hash=sha256:18c57ee915c7ec61cff0ecf7f0f869936c7c30191bb0cf406f1341778d0834e1 \\\n --hash=sha256:4b0dda527ee31078689fc205ec4f1c1bf7d56cf88b6dc9426c4f230e46c2dce1\npackaging==25.0 \\\n --hash=sha256:29572ef2b1f17581046b3a2227d5c611fb25ec70ca1ba8554b24b0e69331a484 \\\n --hash=sha256:d443872c98d677bf60f6a1f2f8c1cb748e8fe762d2bf9d3148b5599295b0fc4f\npipenv==2025.0.4 \\\n --hash=sha256:36fc2a7841ccdb2f58a9f787b296c2e15dea3b5b79b84d4071812f28b7e8d7a2 \\\n --hash=sha256:e1fbe4cfd25ab179f123d1fbb1fa1cdc0b3ffcdb1f21c775dcaa12ccc356f2bb\nplatformdirs==4.5.0 \\\n --hash=sha256:70ddccdd7c99fc5942e9fc25636a8b34d04c24b335100223152c2803e4063312 \\\n --hash=sha256:e578a81bb873cbb89a41fcc904c7ef523cc18284b7e3b3ccf06aca1403b7ebd3\nvirtualenv==20.36.1 \\\n --hash=sha256:575a8d6b124ef88f6f51d56d656132389f961062a9177016a50e4f507bbcc19f \\\n --hash=sha256:8befb5c81842c641f8ee658481e42641c68b5eab3521d8e092d18320902466ba\n" }, { "path": ".github/workflows/scorecard.yml", "content": "name: OpenSSF Scorecard\non:\n schedule:\n - cron: '0 0 * * 0'\n push:\n branches:\n - main\n\npermissions:\n contents: read\n\njobs:\n analysis:\n name: Scorecards analysis\n runs-on: ubuntu-latest\n permissions:\n # Needed to upload the results to code-scanning dashboard.\n security-events: write\n # Needed to publish results\n id-token: write\n actions: read\n contents: read\n\n steps:\n - name: 'Checkout code'\n uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1\n with:\n persist-credentials: false\n\n - name: 'Run analysis'\n uses: ossf/scorecard-action@4eaacf0543bb3f2c246792bd56e8cdeffafb205a # v2.4.3\n with:\n results_file: results.sarif\n results_format: sarif\n repo_token: ${{ secrets.GITHUB_TOKEN }}\n publish_results: true\n\n # Upload the results as artifacts.\n - name: 'Upload artifact'\n uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6.0.0\n with:\n name: SARIF file\n path: results.sarif\n retention-days: 5\n\n # Upload the results to GitHub's code scanning dashboard.\n - name: 'Upload to code-scanning'\n uses: github/codeql-action/upload-sarif@cdefb33c0f6224e58673d9004f47f7cb3e328b89 # v4.31.10\n with:\n sarif_file: results.sarif\n" }, { "path": ".github/workflows/ui-ci.yml", "content": "name: UI CI\n\non:\n workflow_dispatch:\n pull_request:\n paths:\n - \"quickwit/quickwit-ui/**\"\n - \".github/workflows/ui-ci.yml\"\n push:\n branches:\n - main\n - trigger-ci-workflow\n paths:\n - \"quickwit/quickwit-ui/**\"\n - \".github/workflows/ui-ci.yml\"\n\npermissions:\n contents: read\n\njobs:\n checks:\n name: Lint, type check & unit tests\n runs-on: ubuntu-latest\n steps:\n - uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1\n - uses: actions/setup-node@395ad3262231945c25e8478fd5baf05154b1d79f # v6.1.0\n with:\n node-version: 24\n cache: \"yarn\"\n cache-dependency-path: quickwit/quickwit-ui/yarn.lock\n - name: Install JS dependencies\n run: yarn --cwd quickwit-ui install\n working-directory: ./quickwit\n - name: Lint\n run: yarn --cwd quickwit-ui lint\n working-directory: ./quickwit\n - name: Type check\n run: yarn --cwd quickwit-ui type\n working-directory: ./quickwit\n - name: Unit tests\n run: yarn --cwd quickwit-ui test\n working-directory: ./quickwit\n\n e2e:\n name: Playwright e2e\n runs-on: ubuntu-latest\n permissions:\n contents: read\n actions: write\n services:\n postgres:\n image: postgres:latest\n ports:\n - 5432:5432\n env:\n POSTGRES_USER: quickwit-dev\n POSTGRES_PASSWORD: quickwit-dev\n POSTGRES_DB: quickwit-metastore-dev\n options: >-\n --health-cmd pg_isready\n --health-interval 10s\n --health-timeout 5s\n --health-retries 5\n env:\n CARGO_INCREMENTAL: 0\n RUST_BACKTRACE: 1\n RUSTFLAGS: -Dwarnings --cfg tokio_unstable\n RUSTDOCFLAGS: -Dwarnings -Arustdoc::private_intra_doc_links\n QW_TEST_DATABASE_URL: postgres://quickwit-dev:quickwit-dev@postgres:5432/quickwit-metastore-dev\n steps:\n - uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1\n - uses: actions/setup-node@395ad3262231945c25e8478fd5baf05154b1d79f # v6.1.0\n with:\n node-version: 24\n cache: \"yarn\"\n cache-dependency-path: quickwit/quickwit-ui/yarn.lock\n - name: Setup stable Rust Toolchain\n uses: dtolnay/rust-toolchain@f7ccc83f9ed1e5b9c81d8a67d7ad1a747e22a561 # master\n with:\n toolchain: stable\n - name: Setup Rust cache\n uses: Swatinem/rust-cache@779680da715d629ac1d338a641029a2f4372abb5 # v2.8.2\n with:\n workspaces: \"./quickwit -> target\"\n shared-key: \"quickwit-cargo\"\n - name: Install JS dependencies\n run: yarn --cwd quickwit-ui install\n working-directory: ./quickwit\n - name: Install Playwright browsers\n run: npx playwright install chromium --with-deps --only-shell\n working-directory: ./quickwit/quickwit-ui\n - name: Build UI\n run: CI=false yarn --cwd quickwit-ui build\n working-directory: ./quickwit\n - name: Build Quickwit\n run: |\n sudo apt-get update && sudo apt-get -y install protobuf-compiler\n cargo build --features=postgres\n working-directory: ./quickwit\n - name: Run e2e tests\n run: |\n mkdir -p qwdata\n cargo run --features=postgres -- run --service searcher --service metastore --config ../config/quickwit.yaml &\n yarn --cwd quickwit-ui e2e-test\n working-directory: ./quickwit\n" }, { "path": ".gitignore", "content": "# Generated by Cargo\n# will have compiled files and executables\n**/target/**\n**/proptest-regressions\n**/perf.data*\n**/flamegraph.svg\nlocal/**\nquickwit/quickwit-ui/package-lock.json\n**/.DS_Store\n\nTODO.md\nQUESTIONS.txt\n\n\n# Remove Cargo.lock from gitignore if creating an executable, leave it for libraries\n# More information here https://doc.rust-lang.org/cargo/guide/cargo-toml-vs-cargo-lock.html\n#Cargo.lock\n\n# These are backup files generated by rustfmt\n**/*.rs.bk\n\n.env\n.idea\n.vscode\n.vscode-license\ndeps\nelastic-search-artifacts\nqwdata\n\n# Generated by prost/tonic build\n*_descriptor.bin\n" }, { "path": ".localstack/init.sh", "content": "#!/usr/bin/env bash\n\nset -eu\n\nawslocal s3 mb s3://quickwit-dev\nawslocal s3 mb s3://quickwit-integration-tests && awslocal s3 rm --recursive s3://quickwit-integration-tests\n\nif ! awslocal kinesis list-streams | grep -q quickwit-dev-stream ; then\n awslocal kinesis create-stream --stream-name quickwit-dev-stream --shard-count 3\nfi\n" }, { "path": "CHANGELOG.md", "content": "\n\n# [0.9.0]\n\n### Added\n- Add Ingest V2 (#5600, #5566, #5463, #5375, #5350, #5252 #5202)\n- Add SQS source (#5374, #5335, #5148)\n- Disable control plane check for searcher (#5599, #5360)\n- Partially implement `_elastic/_cluster/health` (#5595)\n- Make Jaeger span attribute-to-tag conversion exhaustive (#5574)\n- Use `content_length_limit` for ES bulk limit (#5573)\n- Limit and monitor warmup memory usage (#5568)\n- Add eviction metrics to caches (#5523)\n- Record object storage request latencies (#5521)\n- Add some kind of throttling on the janitor to prevent it from overloading (#5510)\n- Prevent single split searches from different `leaf_search` from interleaving (#5509)\n- Retry on S3 internal error (#5504)\n- Allow specifying OTEL index ID in header (#5503)\n- Add a metric to count storage errors and their error code (#5497)\n- Add support for concatenated fields (#4773, #5369, #5331)\n- Add number of splits per root/leaf search histograms (#5472)\n- Introduce a searcher config option to timeout get requests (#5467)\n- Add fingerprint to task in cluster state (#5464)\n- Enrich root/leaf search spans with number of docs and splits (#5450)\n- Add some additional search metrics (#5447)\n- Improve GC resilience and add metrics (#5420)\n- Enable force shutdown with 2nd Ctrl+C (#5414)\n- Add request_timeout_secs config to searcher config (#5402)\n- Memoize S3 client (#5377)\n- Add more env var config for Postgres (#5365)\n- Enable str fast field range queries (#5324)\n- Allow querying non-existing fields (#5308)\n- Support updating doc mapper through api (#5253)\n- Add optional special handling for hex in code tokenizer (#5200)\n- Added a circuit breaker layer (#5134)\n- Various performance optimizations in Tantivy (https://github.com/quickwit-oss/tantivy/blob/main/CHANGELOG.md)\n\n### Changed\n- Parse datetimes and timestamps with leading and/or trailing whitespace (#5544)\n- Restrict maturity period to retention (#5543)\n- Wait for merge at end of local ingest (#5542)\n- Log PostgreSQL metastore error (#5530)\n- Update azure multipart policy (#5553)\n- Stop relying on our own version of pulsar-rs (#5487)\n- Handle nested OTLP values in attributes and log bodies (#5485)\n- Improve merge pipeline finalization (#5475)\n- Allow failed splits in root search (#5440)\n- Batch delete from GC (#5404, #5380)\n- Make some S3 errors retryable (#5384)\n- Change default timestamps in OTEL logs (#5366)\n- Only return root spans for Jaeger HTTP API (#5358)\n- Share aggregation limit on node (#5357)\n\n### Fixed\n- Fix existence queries for nested fields (#5581)\n- Fix lenient option with wildcard queries (#5575)\n- Fix incompatible ES Java date format (#5462)\n- Fix bulk api response order (#5434)\n- Fix pulsar finalize (#5471)\n- Fix pulsar URI scheme (#5470)\n- Fix grafana searchers dashboard (#5455)\n- Fix jaeger http endpoint (#5378)\n- Fix file re-ingestion after EOF (#5330)\n- Fix configuration interpolation (#5403)\n- Fix jaeger duration parse error (#5518)\n- Fix unit conversion in jaeger http search endpoint (#5519)\n\n### Removed\n- Remove support for 2-digit years in java datetime parser (#5596)\n- Remove DocMapper trait (#5508)\n- Remove support for AWS Lambda (#5884)\n- Remove search stream endpoint (#5886)\n\n# [0.8.1]\n\n### Fixed\n\n- Bug in the chitchat digest message serialization (chitchat#144)\n\n## [0.8.0]\n\n### Added\n\n- Remove some noisy logs (#4447)\n- Add `/{index}/_stats` and `/_stats` ES API (#4442)\n- Use `search_after` in ES scroll API (#4280)\n- Add support for wildcard exclusion in index patterns (#4458)\n- Add `.` support in DSL indentifiers (#3989)\n- Add cat indices ES API (#4465)\n- Limit concurrent merges (#4473)\n- Add Index Template API and auto create index (#4456) (only available with ingest V2)\n- Add support for compressed ES `_bulk` requests (#4506)\n- Add support for slash `/` character in field names (#4510)\n- Handle SIGTERM shutdown signal (#4539)\n- Add `start_timestamp` and `end_timestamp` filter to ES `_field_caps` API (#4547)\n- Limit the number of merge pipelines that can be spawned concurrently (#4574)\n- Add support for `_source_excludes` and `_source_includes` query parameters in ES API (#4572)\n- Add gRPC metrics layer to clients and servers (#4591)\n- Add additional cluster metrics (#4597)\n- Add index patterns query param on GET `/indexes` endpoint (#4600)\n- Add support for GCS file backed metastore (#4604)\n- Add default search fields for OTEL traces index (#4602)\n- Add support for delete index in ES API (#4606)\n- Add a handler to dynamically change the log level (#4662)\n- Add REST endpoint to parse a query into a query AST (#4652)\n- Add postgresql index and use `IN` instead of many `OR` (#4670)\n- Add support for `_source_excludes`, `_source_includes`, `extra_filters` in `_msearch` ES API (#4696)\n- Handle `track_total_size` on request ES body (#4710)\n- Add a metric for the number number of indexes (#4711)\n- Add various performance optimizations in Quickwit and Tantivy\n\nMore details in tantivy's [changelog](https://github.com/quickwit-oss/tantivy/blob/main/CHANGELOG.md).\n\n### Fixed\n\n- Fix aggregation result on empty index (#4449)\n- Fix Gzip file source (#4457)\n- Rate limit noisy logs (#4483)\n- Prevent the exponential backoff from overflowing after 64 attempts (#4501)\n- Remove field presence in ES `_field_caps` API (#4492)\n- Remove `source` in ES parameter, remove unsupported field `fields` in response (#4590)\n- Fix aggregation `split_size` parameter, add docs and test (#4627)\n- Various fixes in chitchat (gossip): more details in [chitchat commit history](https://github.com/quickwit-oss/chitchat/commits/main/?since=2024-01-08&until=2024-03-13)\n- Various fixes in mrecordlog (WAL): more details in [mrecordlog commit history](https://github.com/quickwit-oss/mrecordlog/commits/main/?since=2024-01-08&until=2024-03-13)\n\n### Changed\n\n- (Breaking) [Add ZSTD compression to chitchat's Deltas](https://github.com/quickwit-oss/chitchat/pull/112)\n\n### Removed\n\n### Migration from 0.7.x to 0.8.0\n\nTo deploy Quickwit 0.8.0, you must either:\n- **shutdown down** your cluster **entirely** before deploying, or\n- **restart all** the nodes of your cluster after deploying.\n\nBecause we made some breaking changes in the gossip protocol (chitchat), nodes running different versions of Quickwit cannot communicate with each other and crash upon receiving messages that do not match their release version. The new protocol is now versioned, and future updates of the gossip protocol will be backward compatible.\n\n\n## [0.7.1]\n\n### Added\n\n- Add es _count API (#4410)\n- Add _elastic/_field_caps API (#4350)\n- Make gRPC message size configurable (#4388)\n- Add API endpoint to get some control-plan internal info (#4339)\n- Add Google Cloud Storage Implementation available for storage paths starting with `gs://` (#4344)\n\n### Changed\n\n- Return 404 on index not found in ES Bulk API (#4425)\n- Allow $ and @ characters in field names (#4413)\n\n### Fixed\n- Assign all sources/shards, even if this requires exceeding the indexer #4363\n- Fix traces doc mapping (service name set as fast) and update default otel logs index ID to `otel-logs-v0_7` (#4401)\n- Fix parsing multi-line queries (#4409)\n- Fix range query for optional fast field panics with Index out of bounds (#4362)\n\n### Migration from 0.7.0 to 0.7.1\n\nQuickwit 0.7.1 will create the new index `otel-logs-v0_7` which is now used by default when ingesting data with the OTEL gRPC and HTTP API.\n\nIn the traces index `otel-traces-v0_7`, the `service_name` field is now fast. No migration is done if `otel-traces-v0_7` already exists. If you want `service_name` field to be fast, you have to delete first the existing `otel-traces-v0_7` index or create your own index.\n\n## [0.7.0]\n\n### Added\n\n- Elasticsearch-compatible API\n - Added scroll and search_after APIs and support for multi-index search queries\n - Added exists, multi-match, match phrase prefix, match bool prefix, bool queries\n - Added `_field_caps` API\n- Added support for OTLP over HTTP API (Protobuf only) (#4335)\n- Added Jaeger REST endpoints for Grafana tracing support (#4197)\n- Added support for injecting custom HTTP headers and moved REST config parameters into REST config section (#4198)\n- Added support for OTLP trace data in arbitrary sources\n- Commit Kafka offsets on suggest truncate (#3638)\n- Honor `auto.offset.reset` parameter in Kafka source (#4095)\n- Added exact count optimization (#4019)\n- Added stream splits gRPC (#4109)\n- Adding a split cache in Searchers (#3857)\n- Added `coerce` and `output_format` options for numeric fields (#3704)\n- Added `PhraseMatchQuery` and `MultiMatchQuery` (#3727)\n- Added Elasticsearch's `TermsQuery` (#3747)\n- Added GCP PubSub source (#3720)\n- Parse timestamp strings (#3639)\n- Added Digital Ocean storage flavor (#3632)\n- Added new tokenizers: `source_code_default`, `source_code`, `multilang` (#3647, #3655, #3608)\n\n\n### Fixed\n\n- Fixed dates in UI (#4277)\n- Fixed duplicate splits planned on pipeline crash-respawn (#3854)\n- Fixed sorting (#3799)\n\nMore details in tantivy's [changelog](https://github.com/quickwit-oss/tantivy/blob/main/CHANGELOG.md).\n\n### Changed\n\n- Improve OTEL traces index config (#4311)\n - OTEL endpoints are now using by default indexes `otel-logs-v0_7` and `otel-traces-v0_7` instead of `otel-logs-v0_6` and `otel-traces-v0_6`\n - OTEL indexes have more fields stored as \"fast\" and have Trace and Span ID bytes field in hex format\n\n- Increased the gRPC payload limits from 10MiB to 20MiB (#4227)\n- Reject malformed Elasticsearch API requests (#4175)\n- Better logging when doc processing fails (#4323)\n- Search performance improvements\n- Indexing performance improvements\n\n### Removed\n\n### Migration from 0.6.x to 0.7\n\nThe format of the index and internal objects stored in the metastore of 0.7 is backward compatible with 0.6.\n\nIf you are using the OTEL indexes and ingesting data into indexes the `otel-logs-v0_6` and `otel-traces-v0_6`, you must stop indexing before upgrading.\nIndeed, the first time you start Quickwit 0.7, it will update the doc mapping fields of Trace ID and Span ID of those two indexes by changing their input/output formats from base64 to hex. This is automatic: you don't have to perform any manual operation.\nQuickwit 0.7 will create new indexes `otel-logs-v0_7` and `otel-traces-v0_7`, which are now used by default when ingesting data with the OTEL gRPC and HTTP API. The Jaeger gRPC and HTTP APIs will query both `otel-traces-v0_6` and `otel-traces-v0_7` by default.\nIt's possible to define the index ID you want to use for OTEL gRPC endpoints and Jaeger gRPC API by setting the request header `qw-otel-logs-index` or `qw-otel-traces-index` to the index ID you want to target.\n\n\n## [0.6.1]\n\n### Added\n- Support of phrase prefix queries in the query language.\n\n### Fixed\n- Fix timestamp field which was not allowed when defined in an object mapping.\n- Fix querying of integer on a JSON field (no document were returned).\n\n\n## [0.6.0] - 2023-06-03\n\n### Added\n- Elasticsearch/Opensearch compatible API.\n- New columnar format:\n - Fast fields can now have any cardinality (Optional, Multivalued, restricted). In fact cardinality is now only used to format the output.\n - Dynamic Fields are now fast fields.\n- String fast fields now can be normalized.\n- Various parameters of object storages can now be configured.\n- The ingest API makes it possible to force a commit, or wait for a scheduled commit to occur.\n- Ability to parse non-JSON data using VRL to extract some structure from documents.\n- Object storage can now use the `virtual-hosted–style`.\n- `date_histogram` aggregation.\n- `percentiles` aggregation.\n- Added support for Prefix Phrase query.\n- Added support for range queries.\n- The query language now supports different date formats.\n- Added support for base16 input/output configuration for bytes field. You can search for bytes fields using base16 encoded values.\n- Autotagging: fields used in the partition key are automatically added to tags.\n- Added arm64 docker image.\n- Added CORS configuration for the REST API.\n\n\n### Fixed\n- Major bug fix that required to restart quickwit when deleting and recreating an index with the same name.\n- The number of concurrent GET requests to object stores is now limited. This fixes a bug observed with when requested a lot of documents from MinIO.\n- Quickwit now searches into resource attributes when receiving a Jaeger request carrying tags\n- Object storage can be figured to:\n - avoid Bulk delete API (workaround for Google Cloud Storage).\n - Use virtual-host style addresses (workaround for Alibaba Object Storage Service).\n- Fix aggregation min doc_count empty merge bug.\n- Fix: Sort order for term aggregations.\n- Switch to ms in histogram for date type (aligning with ES).\n\n### Improvements\n\n- Search performance improvement.\n- Aggregation performance improvement.\n- Aggregation memory improvement.\n\nMore details in tantivy's [changelog](https://github.com/quickwit-oss/tantivy/blob/main/CHANGELOG.md).\n\n### Changed\n- Datetime now have up to a nanosecond precision.\n- By default, quickwit now uses the node's hostname as the default node ID.\n- By default, Quickwit is in dynamic mode and all dynamic fields are marked as fast fields.\n- JSON field uses by default the raw tokanizer and is set to fast field.\n- Various performance/compression improvements.\n- OTEL indexes Trace ID and Span ID are now bytes fields.\n- OTEL indexes stores timestamps with nanosecond precision.\n- pan status is now indexed in the OTEL trace index.\n- Default and raw tokenizers filter tokesn longer than 255 bytes instead of 40 bytes.\n\n\n## [0.5.0] - 2023-03-16\n\n### Added\n- gRPC OpenTelemetry Protocol support for traces\n- gRPC OpenTelemetry Protocol support for logs\n- Control plane (indexing tasks scheduling)\n- Ingest API rate limiter\n- Pulsar source\n- VRL transform for data sources\n- REST API enhanced to fully manage indexes, sources, and splits\n- OpenAPI specification and swagger UI for all REST available endpoints\n- Large responses from REST API can be compressed\n- Add bulk stage splits method to metastore\n- MacOS M1 binary\n- Doc mapping field names starting with `_` are now valid\n\n### Fixed\n- Fix UI index completion on search page\n- Fix CLI index describe command to show stats on published splits\n- Fix REST API to always return on error a body formatted as `{\"message\": \"error message\"}`\n- Fixed REST status code when deleting unexisting index, source and when fetching splits on unexisting index\n\n### Changed\n- Source config schema (breaking or not? use serde rename to be not breaking?)\n- RocksDB replaced by [mrecordlog](https://github.com/quickwit-oss/mrecordlog) to store ingest API queues records\n- (Breaking) Indexing partition key new DSL\n- (Breaking) Helm chart updated with the new CLI\n- (Breaking) CLI indexes, sources, and splits commands use the REST API\n- (Breaking) Index new format: you need to reindex all your data\n\n## [0.4.0] - 2022-12-03\n\n### Added\n- Boolean, datetime, and IP address fields\n- Chinese tokenizer\n- Distributed indexing (Kafka only)\n- gRPC metastore server\n- Index partitioning\n- Kubernetes\n- Node config templating\n- Prometheus metrics\n- Retention policies\n- REST API for CRUD operations on indexes/sources\n- Support for Azure Blob Storage\n- Support for BM25 document scoring\n- Support for deletions\n- Support for slop in phrase queries\n- Support for snippeting\n\n### Fixed\n- Fixed cache misses during search fetch docs phase\n- Fixed credentials leak in metastore URI\n- Fixed GC scalability issues\n- Fixed support for multi-source\n\n### Changed\n- Changed default docstore block size to 1 MiB and compression algorithm to ZSTD\n\n- Quickwit now relies on sqlx rather than Diesel for PostgreSQL interactions.\nMigrating from 0.3 should work as expected. Migrating from earlier version however is\nnot supported.\n\n### Removed\n- Removed support for i64 as timestamp field\n- Removed support for sorting index by field\n\n### Security\n- Forbid access to paths with `..` at storage level\n\n## [0.3.1] - 2022-06-22\n\n### Added\n- Add support for Google Cloud Storage\n- Sort hits by timestamp desc by default in search UI\n- Add `description` attribute to field mappings\n- Display split state in output of `quickwit split list` command\n\n### Fixed\n- Clean up local split cache after index deletion\n- Fix API URLs displayed for copy and paste in UI\n- Fix custom S3 endpoint with trailing `/`\n- Fix `quickwit index create` command with `--overwrite` option\n\n## [0.3.0] - 2022-05-31\n\n### Added\n- Embedded UI for displaying search hits and cluster state\n- Schemaless indexing with JSON field\n- Ingest API (Elasticsearch-compatible)\n- Aggregation queries\n- Support for Amazon Kinesis\n\n### Fixed\n- Switched cluster membership algorithm from S.W.I.M. to Chitchat\n\n### Removed\n- u64 as date field\n\n## [0.2.1] - 2022-02-28\n\n### Added\n- Query validation against index schema before dispatch to leaf nodes (#1109, @linxGnu)\n- Support for custom S3 endpoint (#1108)\n- Warm up terms and fastfields concurrently (#1147)\n\n### Fixed\n- Minor bug in leaf search stream (#1110)\n- Default index root URI and metastore URI correctly default to data dir (#1140, @ddelemeny)\n\n### Removed\n- QW_ENV environment variable\n\n### Security\n- Compiled binaries with Rust 1.58.1, which fixes CVE-2022-21658\n\n## [0.2.0] - 2022-01-12\n\n## [0.1.0] - 2021-07-13\n" }, { "path": "CODE_OF_CONDUCT.md", "content": "# Contributor Covenant Code of Conduct\n\n## Our Pledge\n\nWe as members, contributors, and leaders pledge to make participation in our\ncommunity a harassment-free experience for everyone, regardless of age, body\nsize, visible or invisible disability, ethnicity, sex characteristics, gender\nidentity and expression, level of experience, education, socio-economic status,\nnationality, personal appearance, race, caste, color, religion, or sexual identity\nand orientation.\n\nWe pledge to act and interact in ways that contribute to an open, welcoming,\ndiverse, inclusive, and healthy community.\n\n## Our Standards\n\nExamples of behavior that contributes to a positive environment for our\ncommunity include:\n\n* Demonstrating empathy and kindness toward other people\n* Being respectful of differing opinions, viewpoints, and experiences\n* Giving and gracefully accepting constructive feedback\n* Accepting responsibility and apologizing to those affected by our mistakes,\n and learning from the experience\n* Focusing on what is best not just for us as individuals, but for the\n overall community\n\nExamples of unacceptable behavior include:\n\n* The use of sexualized language or imagery, and sexual attention or\n advances of any kind\n* Trolling, insulting or derogatory comments, and personal or political attacks\n* Public or private harassment\n* Publishing others' private information, such as a physical or email\n address, without their explicit permission\n* Other conduct which could reasonably be considered inappropriate in a\n professional setting\n\n## Enforcement Responsibilities\n\nCommunity leaders are responsible for clarifying and enforcing our standards of\nacceptable behavior and will take appropriate and fair corrective action in\nresponse to any behavior that they deem inappropriate, threatening, offensive,\nor harmful.\n\nCommunity leaders have the right and responsibility to remove, edit, or reject\ncomments, commits, code, wiki edits, issues, and other contributions that are\nnot aligned to this Code of Conduct, and will communicate reasons for moderation\ndecisions when appropriate.\n\n## Scope\n\nThis Code of Conduct applies within all community spaces, and also applies when\nan individual is officially representing the community in public spaces.\nExamples of representing our community include using an official e-mail address,\nposting via an official social media account, or acting as an appointed\nrepresentative at an online or offline event.\n\n## Enforcement\n\nInstances of abusive, harassing, or otherwise unacceptable behavior may be\nreported to the community leaders responsible for enforcement at adrien+cc at quickwit dot io.\nAll complaints will be reviewed and investigated promptly and fairly.\n\nAll community leaders are obligated to respect the privacy and security of the\nreporter of any incident.\n\n## Enforcement Guidelines\n\nCommunity leaders will follow these Community Impact Guidelines in determining\nthe consequences for any action they deem in violation of this Code of Conduct:\n\n### 1. Correction\n\n**Community Impact**: Use of inappropriate language or other behavior deemed\nunprofessional or unwelcome in the community.\n\n**Consequence**: A private, written warning from community leaders, providing\nclarity around the nature of the violation and an explanation of why the\nbehavior was inappropriate. A public apology may be requested.\n\n### 2. Warning\n\n**Community Impact**: A violation through a single incident or series\nof actions.\n\n**Consequence**: A warning with consequences for continued behavior. No\ninteraction with the people involved, including unsolicited interaction with\nthose enforcing the Code of Conduct, for a specified period of time. This\nincludes avoiding interactions in community spaces as well as external channels\nlike social media. Violating these terms may lead to a temporary or\npermanent ban.\n\n### 3. Temporary Ban\n\n**Community Impact**: A serious violation of community standards, including\nsustained inappropriate behavior.\n\n**Consequence**: A temporary ban from any sort of interaction or public\ncommunication with the community for a specified period of time. No public or\nprivate interaction with the people involved, including unsolicited interaction\nwith those enforcing the Code of Conduct, is allowed during this period.\nViolating these terms may lead to a permanent ban.\n\n### 4. Permanent Ban\n\n**Community Impact**: Demonstrating a pattern of violation of community\nstandards, including sustained inappropriate behavior, harassment of an\nindividual, or aggression toward or disparagement of classes of individuals.\n\n**Consequence**: A permanent ban from any sort of public interaction within\nthe community.\n\n## Attribution\n\nThis Code of Conduct is adapted from the [Contributor Covenant][homepage],\nversion 2.0, available at\n[https://www.contributor-covenant.org/version/2/0/code_of_conduct.html][v2.0].\n\nCommunity Impact Guidelines were inspired by\n[Mozilla's code of conduct enforcement ladder][Mozilla CoC].\n\nFor answers to common questions about this code of conduct, see the FAQ at\n[https://www.contributor-covenant.org/faq][FAQ]. Translations are available\nat [https://www.contributor-covenant.org/translations][translations].\n\n[homepage]: https://www.contributor-covenant.org\n[v2.0]: https://www.contributor-covenant.org/version/2/0/code_of_conduct.html\n[Mozilla CoC]: https://github.com/mozilla/diversity\n[FAQ]: https://www.contributor-covenant.org/faq\n[translations]: https://www.contributor-covenant.org/translations\n" }, { "path": "CODE_STYLE.md", "content": "# Quickwit Coding Style\n\nThis document resumes a couple of points we try to embrace in our coding style. Some of these points take an opinionated side on a trade-off story.\nThe description will try to make that clear.\n\nThe driving motivation of this code style is to make your code more readable.\n\nReadable is one word that hides several dimensions:\n- the reader understands the intent very rapidly\n- the reader can proofread. It can become confident that the code is correct very easily.\n\nNoticing how the two are different should not require too much squinting.\nShoot for *proofreadability*.\n\n## Code reviews\n\nDo a pass on your own code before sending it for review to avoid wasting the review time.\nAlso, a trivial code style issues can come in the way and avoid spotting\ndeeper issues with the code.\n\nAs a reviewer, your first mission is proofreading. If you find a logical bug, feel good. You did an awesome job today.\n\nYour second goal is to make sure the code quality stays high.\n\nYou can express \"nitpicks\": suggestions about some local aspect of the code that do not matter too much. Just prepend \"nitpick:\" to your comment.\nYou can also express an opinion/advice that you know is not universal.\nMake sure you make it clear to the reviewee that it is fine to ignore the comment.\n\nDo not use rhetorical questions... If you are 95% sure of something, there is no need to express it as a question.\nPrefer `I believe this should be n+1` to `Shouldn't this be n+1?`.\n\nThe issue with rhetorical questions is that when you will have a genuine\nquestion, reviewees may over interpret it as an affirmation.\n\nAs a reviewee, if you are not used to CRs, it can feel like an adversarial process. Relax. This is normal to end up with a lot of comments on your first few CRs.\n\nYou might feel like the comments are unjustified, try as much as possible to not feel frustrated.\nIf you want to discuss it, the best place is the chat, or maybe send a PR to modify this document.\n\nBut remember to pick your battles... If you think it does not matter much but it takes 2 secs to fix, just consider doing what is suggested by the reviewer or this style guide.\n\n## Rust gives us a lot of tools... this does not mean we need to abuse them.\n\nRust is an amazing language. It offers all kinds of tools to allow for zero-cost code reuse. Within these tools, however, generics and macros tend to hurt readability (and compile-time). Let's ONLY use them where necessary.\n\nThe same goes with the chaining iterator style.\nWhen coupled with error handling, rust's chaining iterator style can\nhurt readability.\nUsing a good old procedural for-loop is fine and recommended in that case.\n\n**example needed**\n\n\n## Naming\n\nFunction and variable names are key for readability.\n\nA good function name is often sufficient for the reader to build reasonable expectations of what it does.\n\nIf this implies long names, let's have very long names.\n\nTrying to fit this rule has an interesting side effect.\nNobody likes to type long function names. It just feels ugly.\nBut these are frequently symptoms of a badly organized code, and it can\nhelp spot refactoring opportunities.\n\n**example needed**\n\n## Explanatory variables\n\nOne incredibly powerful tool and simple tool to help make your code\nmore readable is to introduce explanatory variables.\n\nExplanatory variables are intermediary variables that were not really\nnecessary, but make it possible -through their names- to convey their\nsemantics to the reader.\n\n**example needed**\n\n## Shadowing\n\nAs much as possible, do not use reuse the same variable name in a function.\nIt is never necessary, very rarely helpful and can hurt.\n\n## Types\n\nRust handles type elision. That's great.\nChances are, your editor even automatically hints the type of\nyour variables.\n\nSometimes, however, it can be helpful for the reviewer to have the type of some very strategic variables.\n\n**example needed**\n\n## Early returns\n\nWe prefer early return.\nRather than chaining `else` statement, we prefer to isolate\ncorner case in short `if` statement to prevent nesting\n\n**example needed**\n\n## Invariants\n\nA good idea to help reviewers proofread your code is to\nidentify invariants and express them as `debug_assert`.\n\nThese assert will not be part of the release binary and won't hurt the execution time.\n\n**example needed**\n\n## Errors and log messages\n\nError and log messages follow the same format. They should be concise, lowercase (except proper names), and without trailing punctuation.\n\nAs a loose rule, where it does not hurt readability, log messages should rely on `tracing` \nstructured logging instead of templating. \n\nIn other words, prefer:\n`warn!(remaining=remaining_attempts, \"trubulizor rpc plane retry failed\")`\nto \n`warn!(\"trubulizor rpc plane retry failed ({remaining_attempts} attempts remaining)\")`\n\n### Error Examples\n- \"failed to start actor runtimes\"\n- \"cannot join PostgreSQL URI {} with path {:?}\"\n- \"could not find split metadata in Metastore {}\"\n- \"unknown output format {:?}\"\n \n### Log examples\n\n\n## Comments\n\nWe use on the same code style, [rustc's doc comments](https://doc.rust-lang.org/1.0.0/style/style/comments.html).\nIn particular, the summary line should be written in third-person singular present indicative form.\n\nNo rustdoc in Quickwit or in private API is ok.\nNo rustdoc on Tantivy public API is not ok.\n\nWe usually do not expect comments to contain any implementation details.\nTo some extent, it is normal for the user to have to look at the code.\n\nWhen it is not clear, comments should convey:\n- intent\n- context (links to a Wikipedia page or a paper, link to the original issue can be helpful too)\n- hidden contracts... but really you should avoid those.\n\nInline comments in the code can be very useful to help the reader understand\nthe justification of a thorny piece of code.\n\n**example needed**\n\n## Hidden contracts\n\nWe call hidden contract, a pre-condition on the arguments that is not enforced by their types.\n\nSometimes, hidden contracts are unavoidable.\n\nFor instance, a binary search requires the array to be sorted.\n\nWhenever possible, you should avoid having hidden contracts.\n\nTo avoid hidden contracts, you should consider:\n- changing your argument types to have the type system enforce the contract\n- internalize the contract enforcement.\n\nFor instance, the following function is not good because it hides a contract on values not being empty:\n\n```\nfn min(&self, values: &[usize]) -> usize {\n\tlet mut min_val = usize::MAX;\n\tfor val in values {\n\t\tmin_val = min_val.min(val)\n\t}\n\tmin_val\n}\n```\nIt can be done by changing the prototype to a `Result` or an `Option`.\n\nIn addition, while the author might have thought that the `usize::MAX` trick was a nice touch, it can easily backfire. Panicking is often better than returning a wrong result.\n\nThe better approach here is of course an `Option` like `Iterator::min` does.\n\nAnother way to internalize the contract enforcement is to move some logic from the caller to within the function.\n\nFor instance:\n```\n// The algorithms requires splits to be sorted by `end_time`\nfn merge_candidates(splits: &mut Vec) -> Vec\n```\n\nIt is tempting to rely on the fact that splits `Vec` is always sorted on the caller side and put this as a hidden contract.\nIf it is not too much work, just redoing the sorting within merge candidates\nis a good idea. For the above function, that extra work is tiny.\n\nBy the way, did you know Rust's std sort is inspired by timsort?\nIt will perform in linear time if the array is already sorted...\n\nWhen implementing a function with a hidden contract, as long as it does not hurt the overall performance, add an assert statement to your code to check the contract. (For instance, check that the array is sorted).\n\n**example needed*\n\n## Tests\n\nTest do not need to match the same quality as the original code.\n\nWhen a bug is encountered, it is ok to introduce a test that seems weirdly\noverfitted to the specific issue. A comment should then add a link to the issue.\n\nUnit test should run fast, and if possible they should not do any IO.\nCode should be structured to make unit testing possible.\n\nSome of our unit tests would not be considered good unit tests in some companies, and that's ok.\n\nHere are the controversial bits:\n\n### Not just for spotting regression\n\nOur unit tests are not here just to spot regression.\nThey are also here to check the correctness of our code.\n\n### Not just testing public API\n\nUnit test do not only test public API.\nComplex code often calls half a dozen smaller functions.\n\nThe cardinality of the corner case of the complex code\ncan make it difficult to test all corner case.\n\nOn the other hand, the smaller functions could be tested\nexhaustively.\n\nFor this reason, testing internal private functions is actually encouraged.\n\n### Not always \"unit\" tests\n\nIdeally, unit tests should be testing one thing and one thing only, but if they don't and it helps cover more ground, this is ok.\n\n### Not necessarily deterministic.\n\nFinally, unit tests are not necessarily deterministic. We really like proptests.\nWhen proptesting, make sure to reduce as much as possible the space of exploration to get the most out of it.\n\n## async vs sync\n\nYour async code should block for at most 500 microseconds.\nIf you are unsure whether your code blocks for 500 microseconds, or if it is a non-trivial question, it should run via `tokio::spawn_block`.\n" }, { "path": "CONTRIBUTING.md", "content": "# Contributing to Quickwit\nThere are many ways to contribute to Quickwit.\nCode contributions are welcome of course, but also\nbug reports, feature requests, and evangelizing are as valuable.\n\n# Submitting a PR\nCheck if your issue is already listed on [github](https://github.com/quickwit-oss/quickwit/issues).\nIf it is not, create your own issue.\n\nPlease add the following phrase at the end of your commit `Closes #`.\nIt will automatically link your PR in the issue page. Also, once your PR is merged, it will\nclose the issue. If your PR only partially addresses the issue and you would like to\nkeep it open, just write `See #`.\n\nFeel free to send your contribution in an unfinished state to get early feedback.\nIn that case, simply mark the PR with the tag [WIP] (standing for work in progress).\n\n## PR verification checks\nWhen you submit a pull request to the project, the CI system runs several verification checks. After your PR is merged, a more exhaustive list of tests will be run.\n\nYou will be notified by email from the CI system if any issues are discovered, but if you want to run these checks locally before submitting PR or in order to verify changes you can use the following commands in the root directory:\n1. To verify that all tests are passing, run `make test-all`.\n2. To fix code style and format as well as catch common mistakes run `make fix`. Alternatively, run `make -k test-all docker-compose-down` to tear down the Docker services after running all the tests.\n3. To build docs run `make build-rustdoc`.\n\n# Development\n\n## Setup & run tests\n\n### Local Development\n\n1. Install Rust, CMake, Docker (https://docs.docker.com/engine/install/) and Docker Compose (https://docs.docker.com/compose/install/)\n2. Install node@24 and `npm install -g yarn`\n3. Install awslocal https://github.com/localstack/awscli-local\n4. Install protoc https://grpc.io/docs/protoc-installation/ (you may need to install the latest binaries rather than your distro's flavor)\n5. Install nextest https://nexte.st/docs/installation/pre-built-binaries/\n\n### GitHub Codespaces\n\n[![Open in GitHub Codespaces](https://github.com/codespaces/badge.svg)](https://codespaces.new/quickwit-oss/quickwit?devcontainer_path=.devcontainer/devcontainer.json)\n\nGitHub Codespaces provides a fully configured development environment in the cloud, making it easy to get started with Quickwit development. By clicking the badge above, you can create a codespace with all the necessary tools installed and configured.\n\n### Running tests\nRun `make test-all` to run all tests.\n\n## Useful commands\n* `make test-all` - starts necessary Docker services and runs all tests.\n* `make -k test-all docker-compose-down` - the same as above, but tears down the Docker services after running all the tests.\n* `make fmt` - runs formatter, this command requires the nightly toolchain to be installed by running `rustup toolchain install nightly`.\n* `make fix` - runs formatter and clippy checks as well as removing unused dependencies (requires `cargo install cargo-machete`).\n* `make typos` - runs the spellcheck tool over the codebase. (Install by running `cargo install typos-cli`)\n* `make doc` - builds docs.\n* `make docker-compose-up` - starts Docker services.\n* `make docker-compose-down` - stops Docker services.\n* `make docker-compose-logs` - shows Docker logs.\n\n## Start the UI\n1. Switch to the `quickwit` subdirectory of the project and create a data directory `qwdata` there if it doesn't exist\n2. Start a server `cargo r run --config ../config/quickwit.yaml`\n3. `yarn --cwd quickwit-ui install` and `yarn --cwd quickwit-ui start`\n4. Open your browser at `http://localhost:3000/ui` if it doesn't open automatically\n\n## Running UI Tests\n1. Run `yarn --cwd quickwit-ui install` and `yarn --cwd quickwit-ui test` in the `quickwit` directory\n\n## Running UI e2e tests\n1. Ensure to run a searcher `cargo r run --service searcher --config ../config/quickwit.yaml`\n2. Run `yarn --cwd quickwit-ui e2e-test`\n\n## Running services such as Amazon Kinesis or S3, Kafka, or PostgreSQL locally.\n1. Ensure Docker and Docker Compose are correctly installed on your machine (see above)\n2. Run `make docker-compose-up` to launch all the services or `make docker-compose-up DOCKER_SERVICES=kafka,postgres` to launch a subset of services.\n\n## Tracing with Jaeger\n1. Ensure Docker and Docker Compose are correctly installed on your machine (see above)\n2. Start the Jaeger services (UI, collector, agent, ...) running the command `make docker-compose-up DOCKER_SERVICES=jaeger`\n3. Start Quickwit with the following environment variables:\n\n```\nOTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317\nQW_ENABLE_OPENTELEMETRY_OTLP_EXPORTER=true\n```\n\n4. Open your browser and visit [localhost:16686](http://localhost:16686/)\n\n## Using tokio console\n1. Install tokio-console by running `cargo install tokio-console`.\n2. Install the quickwit binary in the quickwit-cli folder `RUSTFLAGS=\"--cfg tokio_unstable\" cargo install --path . --features tokio-console`\n3. Launch a long running command such as index and activate tokio with the: `QW_ENABLE_TOKIO_CONSOLE=1 quickwit index ...`\n4. Run `tokio-console`.\n\n## Building binaries\n\nCurrently, we use [cross](https://github.com/rust-embedded/cross) to build Quickwit binaries for different architectures.\nFor this to work, we've had to customize the docker images cross uses. These customizations can be found in docker files located in the `./cross-images` folder. To make cross take into account any change on those\ndocker files, you will need to build and push the images on Docker Hub by running `make cross-images`.\nWe also have nightly builds that are pushed to Docker Hub. This helps continuously check that our binaries are still built even with external dependency updates. Successful builds let you access the artifacts for the next three days. Release builds always have their artifacts attached to the release.\n\n## Docker images\n\nEach merge on the `main` branch triggers the build of a new Docker image available on DockerHub at `quickwit/quickwit:edge`. Tagging a commit also creates a new image `quickwit/quickwit:` if the tag name starts with `v*` or `qw*`. The Docker images are based on Debian.\n\n### Notes on the embedded UI\nAs the react UI is embedded in the rust binary, we need to build the react app before building the binary. Hence `make cross-image` depends on the command `build-ui`.\n\n## Testing release (alpha, beta, rc)\n\nThe following Quickwit installation command `curl -L https://install.quickwit.io | sh` always installs the latest stable version of quickwit. To make it easier in installing and testing new (alpha, beta, rc) releases, you can manually pull and execute the script as `./install.sh --allow-any-latest-version`. This will force the script to install any latest available release package.\n\n## Tracking licenses\n\nWe keep track of the licenses used by the open source crates used by this project using\n[`rust-license-tool`](https://github.com/DataDog/rust-license-tool). The listing is checked every\ntime CI is run. To update the listing, install the tool with `cargo install --git\nhttps://github.com/DataDog/rust-license-tool` and then run `dd-rust-license-tool write`. If there are\nany errors, you may need to update the listing of exceptions in `license-tool.toml`.\n\n# Documentation\n\nQuickwit documentation is located in the docs directory.\n\n## Generating the CLI docs.\n\nThe [CLI doc page](docs/reference/cli.md) is partly generated by a script.\nTo update it, first run the script:\n\n```bash\ncargo run --bin generate_markdown > ../docs/reference/cli_insert.md\n```\n\nThen manually edit the [doc page](docs/reference/cli.md) to update it and delete the generated file.\nThere are two comments to indicate where you want to insert the new docs and where it ends:\n\n```markdown\n[comment]: <> (Insert auto generated CLI docs from here.)\n\n...docs to insert...\n\n[comment]: <> (End of auto generated CLI docs.)\n```\n" }, { "path": "Dockerfile", "content": "FROM node:24@sha256:b2b2184ba9b78c022e1d6a7924ec6fba577adf28f15c9d9c457730cc4ad3807a AS ui-builder\n\nCOPY quickwit/quickwit-ui /quickwit/quickwit-ui\n\nWORKDIR /quickwit/quickwit-ui\n\nRUN touch .gitignore_for_build_directory \\\n && NODE_ENV=production make install build\n\n\nFROM rust:bookworm@sha256:b5efaabfd787a695d2e46b37d3d9c54040e11f4c10bc2e714bbadbfcc0cd6c39 AS bin-builder\n\nARG CARGO_FEATURES=release-feature-set\nARG CARGO_PROFILE=release\nARG QW_COMMIT_DATE\nARG QW_COMMIT_HASH\nARG QW_COMMIT_TAGS\n\nENV QW_COMMIT_DATE=$QW_COMMIT_DATE\nENV QW_COMMIT_HASH=$QW_COMMIT_HASH\nENV QW_COMMIT_TAGS=$QW_COMMIT_TAGS\n\nRUN apt-get -y update \\\n && apt-get -y install ca-certificates \\\n clang \\\n cmake \\\n libssl-dev \\\n llvm \\\n protobuf-compiler \\\n && rm -rf /var/lib/apt/lists/*\n\nCOPY quickwit /quickwit\nCOPY config/quickwit.yaml /quickwit/config/quickwit.yaml\nCOPY --from=ui-builder /quickwit/quickwit-ui/build /quickwit/quickwit-ui/build\n\nWORKDIR /quickwit\n\nRUN rustup toolchain install\n\nRUN echo \"Building workspace with feature(s) '$CARGO_FEATURES' and profile '$CARGO_PROFILE'\" \\\n && RUSTFLAGS=\"--cfg tokio_unstable\" \\\n cargo build \\\n -p quickwit-cli \\\n --features $CARGO_FEATURES \\\n --bin quickwit \\\n $(test \"$CARGO_PROFILE\" = \"release\" && echo \"--release\") \\\n && echo \"Copying binaries to /quickwit/bin\" \\\n && mkdir -p /quickwit/bin \\\n && find target/$CARGO_PROFILE -maxdepth 1 -perm /a+x -type f -exec mv {} /quickwit/bin \\;\n\n\nFROM debian:bookworm-slim@sha256:e899040a73d36e2b36fa33216943539d9957cba8172b858097c2cabcdb20a3e2 AS quickwit\n\nLABEL org.opencontainers.image.title=\"Quickwit\"\nLABEL maintainer=\"Quickwit, Inc. \"\nLABEL org.opencontainers.image.vendor=\"Quickwit, Inc.\"\nLABEL org.opencontainers.image.licenses=\"Apache-2.0\"\n\nRUN apt-get -y update \\\n && apt-get -y install ca-certificates \\\n libssl3 \\\n && rm -rf /var/lib/apt/lists/*\n\nWORKDIR /quickwit\nRUN mkdir config qwdata\nCOPY --from=bin-builder /quickwit/bin/quickwit /usr/local/bin/quickwit\nCOPY --from=bin-builder /quickwit/config/quickwit.yaml /quickwit/config/quickwit.yaml\n\nENV QW_CONFIG=/quickwit/config/quickwit.yaml\nENV QW_DATA_DIR=/quickwit/qwdata\nENV QW_LISTEN_ADDRESS=0.0.0.0\n\nRUN quickwit --version\n\nENTRYPOINT [\"quickwit\"]\n" }, { "path": "LICENSE", "content": " Apache License\n Version 2.0, January 2004\n http://www.apache.org/licenses/\n\n TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION\n\n 1. Definitions.\n\n \"License\" shall mean the terms and conditions for use, reproduction,\n and distribution as defined by Sections 1 through 9 of this document.\n\n \"Licensor\" shall mean the copyright owner or entity authorized by\n the copyright owner that is granting the License.\n\n \"Legal Entity\" shall mean the union of the acting entity and all\n other entities that control, are controlled by, or are under common\n control with that entity. For the purposes of this definition,\n \"control\" means (i) the power, direct or indirect, to cause the\n direction or management of such entity, whether by contract or\n otherwise, or (ii) ownership of fifty percent (50%) or more of the\n outstanding shares, or (iii) beneficial ownership of such entity.\n\n \"You\" (or \"Your\") shall mean an individual or Legal Entity\n exercising permissions granted by this License.\n\n \"Source\" form shall mean the preferred form for making modifications,\n including but not limited to software source code, documentation\n source, and configuration files.\n\n \"Object\" form shall mean any form resulting from mechanical\n transformation or translation of a Source form, including but\n not limited to compiled object code, generated documentation,\n and conversions to other media types.\n\n \"Work\" shall mean the work of authorship, whether in Source or\n Object form, made available under the License, as indicated by a\n copyright notice that is included in or attached to the work\n (an example is provided in the Appendix below).\n\n \"Derivative Works\" shall mean any work, whether in Source or Object\n form, that is based on (or derived from) the Work and for which the\n editorial revisions, annotations, elaborations, or other modifications\n represent, as a whole, an original work of authorship. For the purposes\n of this License, Derivative Works shall not include works that remain\n separable from, or merely link (or bind by name) to the interfaces of,\n the Work and Derivative Works thereof.\n\n \"Contribution\" shall mean any work of authorship, including\n the original version of the Work and any modifications or additions\n to that Work or Derivative Works thereof, that is intentionally\n submitted to Licensor for inclusion in the Work by the copyright owner\n or by an individual or Legal Entity authorized to submit on behalf of\n the copyright owner. For the purposes of this definition, \"submitted\"\n means any form of electronic, verbal, or written communication sent\n to the Licensor or its representatives, including but not limited to\n communication on electronic mailing lists, source code control systems,\n and issue tracking systems that are managed by, or on behalf of, the\n Licensor for the purpose of discussing and improving the Work, but\n excluding communication that is conspicuously marked or otherwise\n designated in writing by the copyright owner as \"Not a Contribution.\"\n\n \"Contributor\" shall mean Licensor and any individual or Legal Entity\n on behalf of whom a Contribution has been received by Licensor and\n subsequently incorporated within the Work.\n\n 2. Grant of Copyright License. Subject to the terms and conditions of\n this License, each Contributor hereby grants to You a perpetual,\n worldwide, non-exclusive, no-charge, royalty-free, irrevocable\n copyright license to reproduce, prepare Derivative Works of,\n publicly display, publicly perform, sublicense, and distribute the\n Work and such Derivative Works in Source or Object form.\n\n 3. Grant of Patent License. Subject to the terms and conditions of\n this License, each Contributor hereby grants to You a perpetual,\n worldwide, non-exclusive, no-charge, royalty-free, irrevocable\n (except as stated in this section) patent license to make, have made,\n use, offer to sell, sell, import, and otherwise transfer the Work,\n where such license applies only to those patent claims licensable\n by such Contributor that are necessarily infringed by their\n Contribution(s) alone or by combination of their Contribution(s)\n with the Work to which such Contribution(s) was submitted. If You\n institute patent litigation against any entity (including a\n cross-claim or counterclaim in a lawsuit) alleging that the Work\n or a Contribution incorporated within the Work constitutes direct\n or contributory patent infringement, then any patent licenses\n granted to You under this License for that Work shall terminate\n as of the date such litigation is filed.\n\n 4. Redistribution. You may reproduce and distribute copies of the\n Work or Derivative Works thereof in any medium, with or without\n modifications, and in Source or Object form, provided that You\n meet the following conditions:\n\n (a) You must give any other recipients of the Work or\n Derivative Works a copy of this License; and\n\n (b) You must cause any modified files to carry prominent notices\n stating that You changed the files; and\n\n (c) You must retain, in the Source form of any Derivative Works\n that You distribute, all copyright, patent, trademark, and\n attribution notices from the Source form of the Work,\n excluding those notices that do not pertain to any part of\n the Derivative Works; and\n\n (d) If the Work includes a \"NOTICE\" text file as part of its\n distribution, then any Derivative Works that You distribute must\n include a readable copy of the attribution notices contained\n within such NOTICE file, excluding those notices that do not\n pertain to any part of the Derivative Works, in at least one\n of the following places: within a NOTICE text file distributed\n as part of the Derivative Works; within the Source form or\n documentation, if provided along with the Derivative Works; or,\n within a display generated by the Derivative Works, if and\n wherever such third-party notices normally appear. The contents\n of the NOTICE file are for informational purposes only and\n do not modify the License. You may add Your own attribution\n notices within Derivative Works that You distribute, alongside\n or as an addendum to the NOTICE text from the Work, provided\n that such additional attribution notices cannot be construed\n as modifying the License.\n\n You may add Your own copyright statement to Your modifications and\n may provide additional or different license terms and conditions\n for use, reproduction, or distribution of Your modifications, or\n for any such Derivative Works as a whole, provided Your use,\n reproduction, and distribution of the Work otherwise complies with\n the conditions stated in this License.\n\n 5. Submission of Contributions. Unless You explicitly state otherwise,\n any Contribution intentionally submitted for inclusion in the Work\n by You to the Licensor shall be under the terms and conditions of\n this License, without any additional terms or conditions.\n Notwithstanding the above, nothing herein shall supersede or modify\n the terms of any separate license agreement you may have executed\n with Licensor regarding such Contributions.\n\n 6. Trademarks. This License does not grant permission to use the trade\n names, trademarks, service marks, or product names of the Licensor,\n except as required for reasonable and customary use in describing the\n origin of the Work and reproducing the content of the NOTICE file.\n\n 7. Disclaimer of Warranty. Unless required by applicable law or\n agreed to in writing, Licensor provides the Work (and each\n Contributor provides its Contributions) on an \"AS IS\" BASIS,\n WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or\n implied, including, without limitation, any warranties or conditions\n of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A\n PARTICULAR PURPOSE. You are solely responsible for determining the\n appropriateness of using or redistributing the Work and assume any\n risks associated with Your exercise of permissions under this License.\n\n 8. Limitation of Liability. In no event and under no legal theory,\n whether in tort (including negligence), contract, or otherwise,\n unless required by applicable law (such as deliberate and grossly\n negligent acts) or agreed to in writing, shall any Contributor be\n liable to You for damages, including any direct, indirect, special,\n incidental, or consequential damages of any character arising as a\n result of this License or out of the use or inability to use the\n Work (including but not limited to damages for loss of goodwill,\n work stoppage, computer failure or malfunction, or any and all\n other commercial damages or losses), even if such Contributor\n has been advised of the possibility of such damages.\n\n 9. Accepting Warranty or Additional Liability. While redistributing\n the Work or Derivative Works thereof, You may choose to offer,\n and charge a fee for, acceptance of support, warranty, indemnity,\n or other liability obligations and/or rights consistent with this\n License. However, in accepting such obligations, You may act only\n on Your own behalf and on Your sole responsibility, not on behalf\n of any other Contributor, and only if You agree to indemnify,\n defend, and hold each Contributor harmless for any liability\n incurred by, or claims asserted against, such Contributor by reason\n of your accepting any such warranty or additional liability.\n\n END OF TERMS AND CONDITIONS\n\n APPENDIX: How to apply the Apache License to your work.\n\n To apply the Apache License to your work, attach the following\n boilerplate notice, with the fields enclosed by brackets \"[]\"\n replaced with your own identifying information. (Don't include\n the brackets!) The text should be enclosed in the appropriate\n comment syntax for the file format. We also recommend that a\n file or class name and description of purpose be included on the\n same \"printed page\" as the copyright notice for easier\n identification within third-party archives.\n\n Copyright 2021-Present Datadog, Inc.\n\n Licensed under the Apache License, Version 2.0 (the \"License\");\n you may not use this file except in compliance with the License.\n You may obtain a copy of the License at\n\n http://www.apache.org/licenses/LICENSE-2.0\n\n Unless required by applicable law or agreed to in writing, software\n distributed under the License is distributed on an \"AS IS\" BASIS,\n WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n See the License for the specific language governing permissions and\n limitations under the License.\n" }, { "path": "LICENSE-3rdparty.csv", "content": "Component,Origin,License,Copyright\nadler2,https://github.com/oyvindln/adler2,0BSD OR MIT OR Apache-2.0,\"Jonas Schievink , oyvindln \"\nadvapi32-sys,https://github.com/retep998/winapi-rs,MIT,Peter Atashian \nahash,https://github.com/tkaitchuck/ahash,MIT OR Apache-2.0,Tom Kaitchuck \naho-corasick,https://github.com/BurntSushi/aho-corasick,Unlicense OR MIT,Andrew Gallant \naliasable,https://github.com/avitex/rust-aliasable,MIT,avitex \nalloca,https://github.com/playXE/alloca-rs,MIT,\"Adel Prokurov , StackOverflowExcept1on\"\nallocator-api2,https://github.com/zakarumych/allocator-api2,MIT OR Apache-2.0,Zakarum \nandroid_system_properties,https://github.com/nical/android_system_properties,MIT OR Apache-2.0,Nicolas Silva \nanes,https://github.com/zrzka/anes-rs,MIT OR Apache-2.0,Robert Vojta \nansi-str,https://github.com/zhiburt/ansi-str,MIT,Maxim Zhiburt \nansitok,https://gitlab.com/zhiburt/ansitok,MIT,Maxim Zhiburt \nanstream,https://github.com/rust-cli/anstyle,MIT OR Apache-2.0,The anstream Authors\nanstyle,https://github.com/rust-cli/anstyle,MIT OR Apache-2.0,The anstyle Authors\nanstyle-parse,https://github.com/rust-cli/anstyle,MIT OR Apache-2.0,The anstyle-parse Authors\nanstyle-query,https://github.com/rust-cli/anstyle,MIT OR Apache-2.0,The anstyle-query Authors\nanstyle-wincon,https://github.com/rust-cli/anstyle,MIT OR Apache-2.0,The anstyle-wincon Authors\nanyhow,https://github.com/dtolnay/anyhow,MIT OR Apache-2.0,David Tolnay \narc-swap,https://github.com/vorner/arc-swap,MIT OR Apache-2.0,Michal 'vorner' Vaner \narrayvec,https://github.com/bluss/arrayvec,MIT OR Apache-2.0,bluss\nassert-json-diff,https://github.com/davidpdrsn/assert-json-diff,MIT,David Pedersen \nasync-compression,https://github.com/Nullus157/async-compression,MIT OR Apache-2.0,\"Wim Looman , Allen Bui \"\nasync-speed-limit,https://github.com/tikv/async-speed-limit,MIT OR Apache-2.0,The TiKV Project Developers\nasync-stream,https://github.com/tokio-rs/async-stream,MIT,Carl Lerche \nasync-stream-impl,https://github.com/tokio-rs/async-stream,MIT,Carl Lerche \nasync-trait,https://github.com/dtolnay/async-trait,MIT OR Apache-2.0,David Tolnay \natomic-waker,https://github.com/smol-rs/atomic-waker,Apache-2.0 OR MIT,\"Stjepan Glavina , Contributors to futures-rs\"\naws-config,https://github.com/smithy-lang/smithy-rs,Apache-2.0,\"AWS Rust SDK Team , Russell Cohen \"\naws-credential-types,https://github.com/smithy-lang/smithy-rs,Apache-2.0,AWS Rust SDK Team \naws-lc-rs,https://github.com/aws/aws-lc-rs,ISC AND (Apache-2.0 OR ISC),AWS-LibCrypto\naws-lc-sys,https://github.com/aws/aws-lc-rs,ISC AND (Apache-2.0 OR ISC) AND OpenSSL,AWS-LC\naws-runtime,https://github.com/smithy-lang/smithy-rs,Apache-2.0,AWS Rust SDK Team \naws-sdk-lambda,https://github.com/awslabs/aws-sdk-rust,Apache-2.0,\"AWS Rust SDK Team , Russell Cohen \"\naws-sdk-s3,https://github.com/awslabs/aws-sdk-rust,Apache-2.0,\"AWS Rust SDK Team , Russell Cohen \"\naws-sdk-sso,https://github.com/awslabs/aws-sdk-rust,Apache-2.0,\"AWS Rust SDK Team , Russell Cohen \"\naws-sdk-ssooidc,https://github.com/awslabs/aws-sdk-rust,Apache-2.0,\"AWS Rust SDK Team , Russell Cohen \"\naws-sdk-sts,https://github.com/awslabs/aws-sdk-rust,Apache-2.0,\"AWS Rust SDK Team , Russell Cohen \"\naws-sigv4,https://github.com/smithy-lang/smithy-rs,Apache-2.0,\"AWS Rust SDK Team , David Barsky \"\naws-smithy-async,https://github.com/smithy-lang/smithy-rs,Apache-2.0,\"AWS Rust SDK Team , John DiSanti \"\naws-smithy-checksums,https://github.com/smithy-lang/smithy-rs,Apache-2.0,\"AWS Rust SDK Team , Zelda Hessler \"\naws-smithy-eventstream,https://github.com/smithy-lang/smithy-rs,Apache-2.0,\"AWS Rust SDK Team , John DiSanti \"\naws-smithy-http,https://github.com/smithy-lang/smithy-rs,Apache-2.0,\"AWS Rust SDK Team , Russell Cohen \"\naws-smithy-http-client,https://github.com/smithy-lang/smithy-rs,Apache-2.0,AWS Rust SDK Team \naws-smithy-json,https://github.com/smithy-lang/smithy-rs,Apache-2.0,\"AWS Rust SDK Team , John DiSanti \"\naws-smithy-observability,https://github.com/awslabs/smithy-rs,Apache-2.0,AWS Rust SDK Team \naws-smithy-protocol-test,https://github.com/smithy-lang/smithy-rs,Apache-2.0,\"AWS Rust SDK Team , Russell Cohen \"\naws-smithy-query,https://github.com/smithy-lang/smithy-rs,Apache-2.0,\"AWS Rust SDK Team , John DiSanti \"\naws-smithy-runtime,https://github.com/smithy-lang/smithy-rs,Apache-2.0,\"AWS Rust SDK Team , Zelda Hessler \"\naws-smithy-runtime-api,https://github.com/smithy-lang/smithy-rs,Apache-2.0,\"AWS Rust SDK Team , Zelda Hessler \"\naws-smithy-types,https://github.com/smithy-lang/smithy-rs,Apache-2.0,\"AWS Rust SDK Team , Russell Cohen \"\naws-smithy-xml,https://github.com/smithy-lang/smithy-rs,Apache-2.0,\"AWS Rust SDK Team , Russell Cohen \"\naws-types,https://github.com/smithy-lang/smithy-rs,Apache-2.0,\"AWS Rust SDK Team , Russell Cohen \"\naxum,https://github.com/tokio-rs/axum,MIT,The axum Authors\naxum-core,https://github.com/tokio-rs/axum,MIT,The axum-core Authors\nbase16ct,https://github.com/RustCrypto/formats/tree/master/base16ct,Apache-2.0 OR MIT,RustCrypto Developers\nbase64,https://github.com/marshallpierce/rust-base64,MIT OR Apache-2.0,Marshall Pierce \nbase64-simd,https://github.com/Nugine/simd,MIT,The base64-simd Authors\nbase64ct,https://github.com/RustCrypto/formats,Apache-2.0 OR MIT,RustCrypto Developers\nbit-set,https://github.com/contain-rs/bit-set,Apache-2.0 OR MIT,Alexis Beingessner \nbit-vec,https://github.com/contain-rs/bit-vec,Apache-2.0 OR MIT,Alexis Beingessner \nbitflags,https://github.com/bitflags/bitflags,MIT OR Apache-2.0,The Rust Project Developers\nbitpacking,https://github.com/quickwit-oss/bitpacking,MIT,Paul Masurel \nblock-buffer,https://github.com/RustCrypto/utils,MIT OR Apache-2.0,RustCrypto Developers\nbon,https://github.com/elastio/bon,MIT OR Apache-2.0,The bon Authors\nbon-macros,https://github.com/elastio/bon,MIT OR Apache-2.0,The bon-macros Authors\nbpu_trasher,https://github.com/pseitz/bpu_trasher,MIT,Pascal Seitz \nbs58,https://github.com/Nullus157/bs58-rs,MIT OR Apache-2.0,The bs58 Authors\nbumpalo,https://github.com/fitzgen/bumpalo,MIT OR Apache-2.0,Nick Fitzgerald \nbytecount,https://github.com/llogiq/bytecount,Apache-2.0 OR MIT,\"Andre Bogus , Joshua Landau \"\nbyteorder,https://github.com/BurntSushi/byteorder,Unlicense OR MIT,Andrew Gallant \nbytes,https://github.com/tokio-rs/bytes,MIT,\"Carl Lerche , Sean McArthur \"\nbytes-utils,https://github.com/vorner/bytes-utils,Apache-2.0 OR MIT,Michal 'vorner' Vaner \nbytesize,https://github.com/bytesize-rs/bytesize,Apache-2.0,\"Hyunsik Choi , MrCroxx , Rob Ede \"\nbytestring,https://github.com/actix/actix-net,MIT OR Apache-2.0,\"Nikolay Kim , Rob Ede \"\ncamino,https://github.com/camino-rs/camino,MIT OR Apache-2.0,\"Without Boats , Ashley Williams , Steve Klabnik , Rain \"\ncargo-platform,https://github.com/rust-lang/cargo,MIT OR Apache-2.0,The cargo-platform Authors\ncargo_metadata,https://github.com/oli-obk/cargo_metadata,MIT,Oliver Schneider \ncast,https://github.com/japaric/cast.rs,MIT OR Apache-2.0,Jorge Aparicio \ncbor-diag,https://github.com/Nullus157/cbor-diag-rs,MIT OR Apache-2.0,The cbor-diag Authors\ncc,https://github.com/rust-lang/cc-rs,MIT OR Apache-2.0,Alex Crichton \ncensus,https://github.com/quickwit-inc/census,MIT,Paul Masurel \ncfg-if,https://github.com/rust-lang/cfg-if,MIT OR Apache-2.0,Alex Crichton \nchitchat,https://github.com/quickwit-oss/chitchat,MIT,\"Quickwit, Inc. \"\nchrono,https://github.com/chronotope/chrono,MIT OR Apache-2.0,The chrono Authors\nciborium,https://github.com/enarx/ciborium,Apache-2.0,Nathaniel McCallum \nciborium-io,https://github.com/enarx/ciborium,Apache-2.0,Nathaniel McCallum \nciborium-ll,https://github.com/enarx/ciborium,Apache-2.0,Nathaniel McCallum \nclap,https://github.com/clap-rs/clap,MIT OR Apache-2.0,The clap Authors\nclap_builder,https://github.com/clap-rs/clap,MIT OR Apache-2.0,The clap_builder Authors\nclap_lex,https://github.com/clap-rs/clap,MIT OR Apache-2.0,The clap_lex Authors\ncoarsetime,https://github.com/jedisct1/rust-coarsetime,ISC,Frank Denis \ncobs,https://github.com/jamesmunns/cobs.rs,MIT OR Apache-2.0,\"Allen Welkie <>, James Munns \"\ncolorchoice,https://github.com/rust-cli/anstyle,MIT OR Apache-2.0,The colorchoice Authors\ncolored,https://github.com/mackwic/colored,MPL-2.0,Thomas Wickham \ncompression-codecs,https://github.com/Nullus157/async-compression,MIT OR Apache-2.0,\"Wim Looman , Allen Bui \"\ncompression-core,https://github.com/Nullus157/async-compression,MIT OR Apache-2.0,\"Wim Looman , Allen Bui \"\nconsole,https://github.com/console-rs/console,MIT,The console Authors\nconst-oid,https://github.com/RustCrypto/formats/tree/master/const-oid,Apache-2.0 OR MIT,RustCrypto Developers\ncore-foundation,https://github.com/servo/core-foundation-rs,MIT OR Apache-2.0,The Servo Project Developers\ncore-foundation-sys,https://github.com/servo/core-foundation-rs,MIT OR Apache-2.0,The Servo Project Developers\ncpufeatures,https://github.com/RustCrypto/utils,MIT OR Apache-2.0,RustCrypto Developers\ncrc32c,https://github.com/zowens/crc32c,Apache-2.0 OR MIT,Zack Owens\ncrc32fast,https://github.com/srijs/rust-crc32fast,MIT OR Apache-2.0,\"Sam Rijs , Alex Crichton \"\ncriterion-plot,https://github.com/criterion-rs/criterion.rs,Apache-2.0 OR MIT,\"Jorge Aparicio , Brook Heisler \"\ncron,https://github.com/zslayton/cron,MIT OR Apache-2.0,Zack Slayton \ncrossbeam-channel,https://github.com/crossbeam-rs/crossbeam,MIT OR Apache-2.0,The crossbeam-channel Authors\ncrossbeam-deque,https://github.com/crossbeam-rs/crossbeam,MIT OR Apache-2.0,The crossbeam-deque Authors\ncrossbeam-epoch,https://github.com/crossbeam-rs/crossbeam,MIT OR Apache-2.0,The crossbeam-epoch Authors\ncrossbeam-utils,https://github.com/crossbeam-rs/crossbeam,MIT OR Apache-2.0,The crossbeam-utils Authors\ncrunchy,https://github.com/eira-fransham/crunchy,MIT,Eira Fransham \ncrypto-bigint,https://github.com/RustCrypto/crypto-bigint,Apache-2.0 OR MIT,RustCrypto Developers\ncrypto-common,https://github.com/RustCrypto/traits,MIT OR Apache-2.0,RustCrypto Developers\ndarling,https://github.com/TedDriggs/darling,MIT,Ted Driggs \ndarling_core,https://github.com/TedDriggs/darling,MIT,Ted Driggs \ndarling_macro,https://github.com/TedDriggs/darling,MIT,Ted Driggs \ndashmap,https://github.com/xacrimon/dashmap,MIT,Acrimon \ndata-encoding,https://github.com/ia0/data-encoding,MIT,Julien Cretin \ndeadpool,https://github.com/bikeshedder/deadpool,MIT OR Apache-2.0,Michael P. Jung \ndeadpool-runtime,https://github.com/bikeshedder/deadpool,MIT OR Apache-2.0,Michael P. Jung \nder,https://github.com/RustCrypto/formats/tree/master/der,Apache-2.0 OR MIT,RustCrypto Developers\nderanged,https://github.com/jhpratt/deranged,MIT OR Apache-2.0,Jacob Pratt \ndialoguer,https://github.com/console-rs/dialoguer,MIT,The dialoguer Authors\ndiff,https://github.com/utkarshkukreti/diff.rs,MIT OR Apache-2.0,Utkarsh Kukreti \ndifflib,https://github.com/DimaKudosh/difflib,MIT,Dima Kudosh \ndigest,https://github.com/RustCrypto/traits,MIT OR Apache-2.0,RustCrypto Developers\ndisplaydoc,https://github.com/yaahc/displaydoc,MIT OR Apache-2.0,Jane Lusby \ndowncast,https://github.com/fkoep/downcast-rs,MIT,Felix Köpge \ndowncast-rs,https://github.com/marcianx/downcast-rs,MIT OR Apache-2.0,The downcast-rs Authors\ndtoa,https://github.com/dtolnay/dtoa,MIT OR Apache-2.0,David Tolnay \ndyn-clone,https://github.com/dtolnay/dyn-clone,MIT OR Apache-2.0,David Tolnay \necdsa,https://github.com/RustCrypto/signatures/tree/master/ecdsa,Apache-2.0 OR MIT,RustCrypto Developers\neither,https://github.com/rayon-rs/either,MIT OR Apache-2.0,bluss\nelasticsearch-dsl,https://github.com/vinted/elasticsearch-dsl-rs,MIT OR Apache-2.0,\"Evaldas Buinauskas , Search Platform \"\nelliptic-curve,https://github.com/RustCrypto/traits/tree/master/elliptic-curve,Apache-2.0 OR MIT,RustCrypto Developers\nembedded-io,https://github.com/embassy-rs/embedded-io,MIT OR Apache-2.0,The embedded-io Authors\nembedded-io,https://github.com/rust-embedded/embedded-hal,MIT OR Apache-2.0,The embedded-io Authors\nencode_unicode,https://github.com/tormol/encode_unicode,Apache-2.0 OR MIT,Torbjørn Birch Moltu \nencoding_rs,https://github.com/hsivonen/encoding_rs,(Apache-2.0 OR MIT) AND BSD-3-Clause,Henri Sivonen \nenum-iterator,https://github.com/stephaneyfx/enum-iterator,0BSD,Stephane Raux \nenum-iterator-derive,https://github.com/stephaneyfx/enum-iterator,0BSD,Stephane Raux \nenv_filter,https://github.com/rust-cli/env_logger,MIT OR Apache-2.0,The env_filter Authors\nenv_logger,https://github.com/rust-cli/env_logger,MIT OR Apache-2.0,The env_logger Authors\nequivalent,https://github.com/indexmap-rs/equivalent,Apache-2.0 OR MIT,The equivalent Authors\nerased-serde,https://github.com/dtolnay/erased-serde,MIT OR Apache-2.0,David Tolnay \nerrno,https://github.com/lambda-fairy/rust-errno,MIT OR Apache-2.0,\"Chris Wong , Dan Gohman \"\nerror-chain,https://github.com/rust-lang-nursery/error-chain,MIT OR Apache-2.0,\"Brian Anderson , Paul Colomiets , Colin Kiegel , Yamakaky , Andrew Gauger \"\nfail,https://github.com/tikv/fail-rs,Apache-2.0,The TiKV Project Developers\nfastdivide,https://github.com/fulmicoton/fastdivide,zlib-acknowledgement OR MIT,Paul Masurel \nfastrand,https://github.com/smol-rs/fastrand,Apache-2.0 OR MIT,Stjepan Glavina \nff,https://github.com/zkcrypto/ff,MIT OR Apache-2.0,\"Sean Bowe , Jack Grigg \"\nfind-msvc-tools,https://github.com/rust-lang/cc-rs,MIT OR Apache-2.0,The find-msvc-tools Authors\nfixedbitset,https://github.com/petgraph/fixedbitset,MIT OR Apache-2.0,bluss\nflate2,https://github.com/rust-lang/flate2-rs,MIT OR Apache-2.0,\"Alex Crichton , Josh Triplett \"\nfloat-cmp,https://github.com/mikedilger/float-cmp,MIT,Mike Dilger \nflume,https://github.com/zesterer/flume,Apache-2.0 OR MIT,Joshua Barretto \nfnv,https://github.com/servo/rust-fnv,Apache-2.0 OR MIT,Alex Crichton \nfoldhash,https://github.com/orlp/foldhash,Zlib,Orson Peters \nform_urlencoded,https://github.com/servo/rust-url,MIT OR Apache-2.0,The rust-url developers\nfragile,https://github.com/mitsuhiko/fragile,Apache-2.0,Armin Ronacher \nfs4,https://github.com/al8n/fs4-rs,MIT OR Apache-2.0,\"Dan Burkert , Al Liu \"\nfslock,https://github.com/brunoczim/fslock,MIT,The fslock Authors\nfutures,https://github.com/rust-lang/futures-rs,MIT OR Apache-2.0,The futures Authors\nfutures-channel,https://github.com/rust-lang/futures-rs,MIT OR Apache-2.0,The futures-channel Authors\nfutures-core,https://github.com/rust-lang/futures-rs,MIT OR Apache-2.0,The futures-core Authors\nfutures-executor,https://github.com/rust-lang/futures-rs,MIT OR Apache-2.0,The futures-executor Authors\nfutures-io,https://github.com/rust-lang/futures-rs,MIT OR Apache-2.0,The futures-io Authors\nfutures-macro,https://github.com/rust-lang/futures-rs,MIT OR Apache-2.0,The futures-macro Authors\nfutures-sink,https://github.com/rust-lang/futures-rs,MIT OR Apache-2.0,The futures-sink Authors\nfutures-task,https://github.com/rust-lang/futures-rs,MIT OR Apache-2.0,The futures-task Authors\nfutures-timer,https://github.com/async-rs/futures-timer,MIT OR Apache-2.0,Alex Crichton \nfutures-util,https://github.com/rust-lang/futures-rs,MIT OR Apache-2.0,The futures-util Authors\ngeneric-array,https://github.com/fizyk20/generic-array,MIT,\"Bartłomiej Kamiński , Aaron Trent \"\ngetrandom,https://github.com/rust-random/getrandom,MIT OR Apache-2.0,The Rand Project Developers\nglob,https://github.com/rust-lang/glob,MIT OR Apache-2.0,The Rust Project Developers\ngovernor,https://github.com/boinkor-net/governor,MIT,Andreas Fuchs \ngroup,https://github.com/zkcrypto/group,MIT OR Apache-2.0,\"Sean Bowe , Jack Grigg \"\nh2,https://github.com/hyperium/h2,MIT,\"Carl Lerche , Sean McArthur \"\nhalf,https://github.com/VoidStarKat/half-rs,MIT OR Apache-2.0,Kathryn Long \nhashbrown,https://github.com/rust-lang/hashbrown,MIT OR Apache-2.0,Amanieu d'Antras \nheaders,https://github.com/hyperium/headers,MIT,Sean McArthur \nheaders-core,https://github.com/hyperium/headers,MIT,Sean McArthur \nheck,https://github.com/withoutboats/heck,MIT OR Apache-2.0,The heck Authors\nheck,https://github.com/withoutboats/heck,MIT OR Apache-2.0,Without Boats \nhermit-abi,https://github.com/hermit-os/hermit-rs,MIT OR Apache-2.0,Stefan Lankes\nhex,https://github.com/KokaKiwi/rust-hex,MIT OR Apache-2.0,KokaKiwi \nhmac,https://github.com/RustCrypto/MACs,MIT OR Apache-2.0,RustCrypto Developers\nhome,https://github.com/rust-lang/cargo,MIT OR Apache-2.0,Brian Anderson \nhostname,https://github.com/djc/hostname,MIT,The hostname Authors\nhtmlescape,https://github.com/veddan/rust-htmlescape,Apache-2.0 OR MIT OR MPL-2.0,Viktor Dahl \nhttp,https://github.com/hyperium/http,MIT OR Apache-2.0,\"Alex Crichton , Carl Lerche , Sean McArthur \"\nhttp-body,https://github.com/hyperium/http-body,MIT,\"Carl Lerche , Lucio Franco , Sean McArthur \"\nhttp-body-util,https://github.com/hyperium/http-body,MIT,\"Carl Lerche , Lucio Franco , Sean McArthur \"\nhttp-serde,https://gitlab.com/kornelski/http-serde,Apache-2.0 OR MIT,Kornel \nhttparse,https://github.com/seanmonstar/httparse,MIT OR Apache-2.0,Sean McArthur \nhttpdate,https://github.com/pyfisch/httpdate,MIT OR Apache-2.0,Pyfisch \nhumantime,https://github.com/chronotope/humantime,MIT OR Apache-2.0,The humantime Authors\nhyper,https://github.com/hyperium/hyper,MIT,Sean McArthur \nhyper-rustls,https://github.com/rustls/hyper-rustls,Apache-2.0 OR ISC OR MIT,The hyper-rustls Authors\nhyper-timeout,https://github.com/hjr3/hyper-timeout,MIT OR Apache-2.0,Herman J. Radtke III \nhyper-util,https://github.com/hyperium/hyper-util,MIT,Sean McArthur \nhyperloglogplus,https://github.com/tabac/hyperloglog.rs,MIT,Tasos Bakogiannis \niana-time-zone,https://github.com/strawlab/iana-time-zone,MIT OR Apache-2.0,\"Andrew Straw , René Kijewski , Ryan Lopopolo \"\niana-time-zone-haiku,https://github.com/strawlab/iana-time-zone,MIT OR Apache-2.0,René Kijewski \nicu_collections,https://github.com/unicode-org/icu4x,Unicode-3.0,The ICU4X Project Developers\nicu_locale_core,https://github.com/unicode-org/icu4x,Unicode-3.0,The ICU4X Project Developers\nicu_normalizer,https://github.com/unicode-org/icu4x,Unicode-3.0,The ICU4X Project Developers\nicu_normalizer_data,https://github.com/unicode-org/icu4x,Unicode-3.0,The ICU4X Project Developers\nicu_properties,https://github.com/unicode-org/icu4x,Unicode-3.0,The ICU4X Project Developers\nicu_properties_data,https://github.com/unicode-org/icu4x,Unicode-3.0,The ICU4X Project Developers\nicu_provider,https://github.com/unicode-org/icu4x,Unicode-3.0,The ICU4X Project Developers\nident_case,https://github.com/TedDriggs/ident_case,MIT OR Apache-2.0,Ted Driggs \nidna,https://github.com/servo/rust-url,MIT OR Apache-2.0,The rust-url developers\nidna_adapter,https://github.com/hsivonen/idna_adapter,Apache-2.0 OR MIT,The rust-url developers\nindexmap,https://github.com/bluss/indexmap,Apache-2.0 OR MIT,The indexmap Authors\nindexmap,https://github.com/indexmap-rs/indexmap,Apache-2.0 OR MIT,The indexmap Authors\nindicatif,https://github.com/console-rs/indicatif,MIT,The indicatif Authors\ninventory,https://github.com/dtolnay/inventory,MIT OR Apache-2.0,David Tolnay \nipnet,https://github.com/krisprice/ipnet,MIT OR Apache-2.0,Kris Price \nipnetwork,https://github.com/achanda/ipnetwork,MIT OR Apache-2.0,\"Abhishek Chanda , Linus Färnstrand \"\niri-string,https://github.com/lo48576/iri-string,MIT OR Apache-2.0,YOSHIOKA Takuma \nis-terminal,https://github.com/sunfishcode/is-terminal,MIT,\"softprops , Dan Gohman \"\nis_terminal_polyfill,https://github.com/polyfill-rs/is_terminal_polyfill,MIT OR Apache-2.0,The is_terminal_polyfill Authors\nitertools,https://github.com/rust-itertools/itertools,MIT OR Apache-2.0,bluss\nitoa,https://github.com/dtolnay/itoa,MIT OR Apache-2.0,David Tolnay \njobserver,https://github.com/rust-lang/jobserver-rs,MIT OR Apache-2.0,Alex Crichton \njs-sys,https://github.com/wasm-bindgen/wasm-bindgen/tree/master/crates/js-sys,MIT OR Apache-2.0,The wasm-bindgen Developers\njson_comments,https://github.com/tmccombs/json-comments-rs,Apache-2.0,Thayne McCombs \nlambda_runtime,https://github.com/awslabs/aws-lambda-rust-runtime,Apache-2.0,\"David Calavera , Harold Sun \"\nlambda_runtime_api_client,https://github.com/awslabs/aws-lambda-rust-runtime,Apache-2.0,\"David Calavera , Harold Sun \"\nlazy_static,https://github.com/rust-lang-nursery/lazy-static.rs,MIT OR Apache-2.0,Marvin Löbel \nlevenshtein_automata,https://github.com/tantivy-search/levenshtein-automata,MIT,Paul Masurel \nlibc,https://github.com/rust-lang/libc,MIT OR Apache-2.0,The Rust Project Developers\nlibm,https://github.com/rust-lang/compiler-builtins,MIT,Jorge Aparicio \nlinked-hash-map,https://github.com/contain-rs/linked-hash-map,MIT OR Apache-2.0,\"Stepan Koltsov , Andrew Paseltiner \"\nlinux-raw-sys,https://github.com/sunfishcode/linux-raw-sys,Apache-2.0 WITH LLVM-exception OR Apache-2.0 OR MIT,Dan Gohman \nlitemap,https://github.com/unicode-org/icu4x,Unicode-3.0,The ICU4X Project Developers\nlock_api,https://github.com/Amanieu/parking_lot,MIT OR Apache-2.0,Amanieu d'Antras \nlog,https://github.com/rust-lang/log,MIT OR Apache-2.0,The Rust Project Developers\nlru,https://github.com/jeromefroe/lru-rs,MIT,Jerome Froelich \nlru-slab,https://github.com/Ralith/lru-slab,MIT OR Apache-2.0 OR Zlib,Benjamin Saunders \nlz4_flex,https://github.com/pseitz/lz4_flex,MIT,\"Pascal Seitz , Arthur Silva , ticki \"\nmatchers,https://github.com/hawkw/matchers,MIT,Eliza Weisman \nmatchit,https://github.com/ibraheemdev/matchit,MIT AND BSD-3-Clause,Ibraheem Ahmed \nmd-5,https://github.com/RustCrypto/hashes,MIT OR Apache-2.0,RustCrypto Developers\nmd5,https://github.com/stainless-steel/md5,Apache-2.0 OR MIT,\"Ivan Ukhov , Kamal Ahmad , Konstantin Stepanov , Lukas Kalbertodt , Nathan Musoke , Scott Mabin , Tony Arcieri , Wim de With , Yosef Dinerstein \"\nmeasure_time,https://github.com/PSeitz/rust_measure_time,MIT,Pascal Seitz \nmemchr,https://github.com/BurntSushi/memchr,Unlicense OR MIT,\"Andrew Gallant , bluss\"\nmemmap2,https://github.com/RazrFalcon/memmap2-rs,MIT OR Apache-2.0,\"Dan Burkert , Yevhenii Reizner \"\nmime,https://github.com/hyperium/mime,MIT OR Apache-2.0,Sean McArthur \nmime_guess,https://github.com/abonander/mime_guess,MIT,Austin Bonander \nmini-internal,https://github.com/dtolnay/miniserde,MIT OR Apache-2.0,David Tolnay \nmini-moka,https://github.com/moka-rs/mini-moka,MIT OR Apache-2.0,The mini-moka Authors\nminimal-lexical,https://github.com/Alexhuszagh/minimal-lexical,MIT OR Apache-2.0,Alex Huszagh \nminiserde,https://github.com/dtolnay/miniserde,MIT OR Apache-2.0,David Tolnay \nminiz_oxide,https://github.com/Frommi/miniz_oxide/tree/master/miniz_oxide,MIT OR Zlib OR Apache-2.0,\"Frommi , oyvindln , Rich Geldreich richgel99@gmail.com\"\nmio,https://github.com/tokio-rs/mio,MIT,\"Carl Lerche , Thomas de Zeeuw , Tokio Contributors \"\nmockall,https://github.com/asomers/mockall,MIT OR Apache-2.0,Alan Somers \nmockall_derive,https://github.com/asomers/mockall,MIT OR Apache-2.0,Alan Somers \nmrecordlog,https://github.com/quickwit-oss/mrecordlog,MIT,The mrecordlog Authors\nmultimap,https://github.com/havarnov/multimap,MIT OR Apache-2.0,Håvar Nøvik \nmurmurhash32,https://github.com/quickwit-inc/murmurhash32,MIT,Paul Masurel \nnew_string_template,https://github.com/hasezoey/new_string_template,MIT,hasezoey \nno-std-net,https://github.com/dunmatt/no-std-net,MIT,M@ Dunlap \nnom,https://github.com/Geal/nom,MIT,contact@geoffroycouprie.com\nnom,https://github.com/rust-bakery/nom,MIT,contact@geoffroycouprie.com\nnonzero_ext,https://github.com/antifuchs/nonzero_ext,Apache-2.0,Andreas Fuchs \nnormalize-line-endings,https://github.com/derekdreery/normalize-line-endings,Apache-2.0,Richard Dodd \nnu-ansi-term,https://github.com/nushell/nu-ansi-term,MIT,\"ogham@bsago.me, Ryan Scheel (Havvy) , Josh Triplett , The Nushell Project Developers\"\nnum-bigint,https://github.com/rust-num/num-bigint,MIT OR Apache-2.0,The Rust Project Developers\nnum-conv,https://github.com/jhpratt/num-conv,MIT OR Apache-2.0,Jacob Pratt \nnum-integer,https://github.com/rust-num/num-integer,MIT OR Apache-2.0,The Rust Project Developers\nnum-rational,https://github.com/rust-num/num-rational,MIT OR Apache-2.0,The Rust Project Developers\nnum-traits,https://github.com/rust-num/num-traits,MIT OR Apache-2.0,The Rust Project Developers\nnum_cpus,https://github.com/seanmonstar/num_cpus,MIT OR Apache-2.0,Sean McArthur \nnumfmt,https://github.com/kurtlawrence/numfmt,MIT,Kurt Lawrence \nobjc2-core-foundation,https://github.com/madsmtm/objc2,Zlib OR Apache-2.0 OR MIT,The objc2-core-foundation Authors\nobjc2-io-kit,https://github.com/madsmtm/objc2,Zlib OR Apache-2.0 OR MIT,The objc2-io-kit Authors\nonce_cell,https://github.com/matklad/once_cell,MIT OR Apache-2.0,Aleksey Kladov \nonce_cell_polyfill,https://github.com/polyfill-rs/once_cell_polyfill,MIT OR Apache-2.0,The once_cell_polyfill Authors\noneshot,https://github.com/faern/oneshot,MIT OR Apache-2.0,Linus Färnstrand \noorandom,https://hg.sr.ht/~icefox/oorandom,MIT,Simon Heath \nopenssl-probe,https://github.com/alexcrichton/openssl-probe,MIT OR Apache-2.0,Alex Crichton \nopentelemetry,https://github.com/open-telemetry/opentelemetry-rust/tree/main/opentelemetry,Apache-2.0,The opentelemetry Authors\nopentelemetry-appender-tracing,https://github.com/open-telemetry/opentelemetry-rust/tree/main/opentelemetry-appender-tracing,Apache-2.0,The opentelemetry-appender-tracing Authors\nopentelemetry-http,https://github.com/open-telemetry/opentelemetry-rust/tree/main/opentelemetry-http,Apache-2.0,The opentelemetry-http Authors\nopentelemetry-otlp,https://github.com/open-telemetry/opentelemetry-rust/tree/main/opentelemetry-otlp,Apache-2.0,The opentelemetry-otlp Authors\nopentelemetry-proto,https://github.com/open-telemetry/opentelemetry-rust/tree/main/opentelemetry-proto,Apache-2.0,The opentelemetry-proto Authors\nopentelemetry_sdk,https://github.com/open-telemetry/opentelemetry-rust/tree/main/opentelemetry-sdk,Apache-2.0,The opentelemetry_sdk Authors\nordered-float,https://github.com/reem/rust-ordered-float,MIT,\"Jonathan Reem , Matt Brubeck \"\nouroboros,https://github.com/someguynamedjosh/ouroboros,MIT OR Apache-2.0,Josh \nouroboros_macro,https://github.com/someguynamedjosh/ouroboros,MIT OR Apache-2.0,Josh \noutref,https://github.com/Nugine/outref,MIT,The outref Authors\nownedbytes,https://github.com/quickwit-oss/tantivy,MIT,\"Paul Masurel , Pascal Seitz \"\np256,https://github.com/RustCrypto/elliptic-curves/tree/master/p256,Apache-2.0 OR MIT,RustCrypto Developers\npage_size,https://github.com/Elzair/page_size_rs,MIT OR Apache-2.0,Philip Woods \npapergrid,https://github.com/zhiburt/tabled,MIT,Maxim Zhiburt \nparking_lot,https://github.com/Amanieu/parking_lot,MIT OR Apache-2.0,Amanieu d'Antras \nparking_lot_core,https://github.com/Amanieu/parking_lot,MIT OR Apache-2.0,Amanieu d'Antras \npeakmem-alloc,https://github.com/PSeitz/peakmem-alloc,MIT,Pascal Seitz \npercent-encoding,https://github.com/servo/rust-url,MIT OR Apache-2.0,The rust-url developers\nperf-event,https://github.com/jimblandy/perf-event,MIT OR Apache-2.0,Jim Blandy \nperf-event-open-sys,https://github.com/jimblandy/perf-event-open-sys,MIT OR Apache-2.0,Jim Blandy \npetgraph,https://github.com/petgraph/petgraph,MIT OR Apache-2.0,\"bluss, mitchmindtree\"\npin-project,https://github.com/taiki-e/pin-project,Apache-2.0 OR MIT,The pin-project Authors\npin-project-internal,https://github.com/taiki-e/pin-project,Apache-2.0 OR MIT,The pin-project-internal Authors\npin-project-lite,https://github.com/taiki-e/pin-project-lite,Apache-2.0 OR MIT,The pin-project-lite Authors\npin-utils,https://github.com/rust-lang-nursery/pin-utils,MIT OR Apache-2.0,Josef Brandl \npkcs8,https://github.com/RustCrypto/formats/tree/master/pkcs8,Apache-2.0 OR MIT,RustCrypto Developers\nplotters,https://github.com/plotters-rs/plotters,MIT,Hao Hou \nplotters-backend,https://github.com/plotters-rs/plotters,MIT,Hao Hou \nplotters-svg,https://github.com/plotters-rs/plotters,MIT,Hao Hou \npnet,https://github.com/libpnet/libpnet,MIT OR Apache-2.0,Robert Clipsham \npnet_base,https://github.com/libpnet/libpnet,MIT OR Apache-2.0,\"Robert Clipsham , Linus Färnstrand \"\npnet_datalink,https://github.com/libpnet/libpnet,MIT OR Apache-2.0,\"Robert Clipsham , Linus Färnstrand \"\npnet_macros,https://github.com/libpnet/libpnet,MIT OR Apache-2.0,\"Robert Clipsham , Pierre Chifflier \"\npnet_macros_support,https://github.com/libpnet/libpnet,MIT OR Apache-2.0,Robert Clipsham \npnet_packet,https://github.com/libpnet/libpnet,MIT OR Apache-2.0,Robert Clipsham \npnet_sys,https://github.com/libpnet/libpnet,MIT OR Apache-2.0,\"Robert Clipsham , Linus Färnstrand \"\npnet_transport,https://github.com/libpnet/libpnet,MIT OR Apache-2.0,Robert Clipsham \nportable-atomic,https://github.com/taiki-e/portable-atomic,Apache-2.0 OR MIT,The portable-atomic Authors\npostcard,https://github.com/jamesmunns/postcard,MIT OR Apache-2.0,James Munns \npotential_utf,https://github.com/unicode-org/icu4x,Unicode-3.0,The ICU4X Project Developers\npowerfmt,https://github.com/jhpratt/powerfmt,MIT OR Apache-2.0,Jacob Pratt \nppv-lite86,https://github.com/cryptocorrosion/cryptocorrosion,MIT OR Apache-2.0,The CryptoCorrosion Contributors\npredicates,https://github.com/assert-rs/predicates-rs,MIT OR Apache-2.0,Nick Stevens \npredicates-core,https://github.com/assert-rs/predicates-rs/tree/master/crates/core,MIT OR Apache-2.0,Nick Stevens \npredicates-tree,https://github.com/assert-rs/predicates-rs/tree/master/crates/tree,MIT OR Apache-2.0,Nick Stevens \npretty_assertions,https://github.com/rust-pretty-assertions/rust-pretty-assertions,MIT OR Apache-2.0,\"Colin Kiegel , Florent Fayolle , Tom Milligan \"\nprettyplease,https://github.com/dtolnay/prettyplease,MIT OR Apache-2.0,David Tolnay \nproc-macro-error,https://gitlab.com/CreepySkeleton/proc-macro-error,MIT OR Apache-2.0,CreepySkeleton \nproc-macro-error-attr,https://gitlab.com/CreepySkeleton/proc-macro-error,MIT OR Apache-2.0,CreepySkeleton \nproc-macro-error-attr2,https://github.com/GnomedDev/proc-macro-error-2,MIT OR Apache-2.0,\"CreepySkeleton , GnomedDev \"\nproc-macro-error2,https://github.com/GnomedDev/proc-macro-error-2,MIT OR Apache-2.0,\"CreepySkeleton , GnomedDev \"\nproc-macro2,https://github.com/dtolnay/proc-macro2,MIT OR Apache-2.0,\"David Tolnay , Alex Crichton \"\nproc-macro2-diagnostics,https://github.com/SergioBenitez/proc-macro2-diagnostics,MIT OR Apache-2.0,Sergio Benitez \nprocfs,https://github.com/eminence/procfs,MIT OR Apache-2.0,Andrew Chin \nprocfs-core,https://github.com/eminence/procfs,MIT OR Apache-2.0,Andrew Chin \nprometheus,https://github.com/tikv/rust-prometheus,Apache-2.0,\"overvenus@gmail.com, siddontang@gmail.com, vistaswx@gmail.com\"\nprost,https://github.com/tokio-rs/prost,Apache-2.0,\"Dan Burkert , Lucio Franco , Casper Meijn , Tokio Contributors \"\nprost-build,https://github.com/tokio-rs/prost,Apache-2.0,\"Dan Burkert , Lucio Franco , Casper Meijn , Tokio Contributors \"\nprost-derive,https://github.com/tokio-rs/prost,Apache-2.0,\"Dan Burkert , Lucio Franco , Casper Meijn , Tokio Contributors \"\nprost-types,https://github.com/tokio-rs/prost,Apache-2.0,\"Dan Burkert , Lucio Franco , Casper Meijn , Tokio Contributors \"\npulldown-cmark,https://github.com/raphlinus/pulldown-cmark,MIT,\"Raph Levien , Marcus Klaas de Vries \"\npulldown-cmark-to-cmark,https://github.com/Byron/pulldown-cmark-to-cmark,Apache-2.0,\"Sebastian Thiel , Dylan Owen , Alessandro Ogier , Zixian Cai <2891235+caizixian@users.noreply.github.com>, Andrew Lyjak \"\nquanta,https://github.com/metrics-rs/quanta,MIT,Toby Lawrence \nquick-error,http://github.com/tailhook/quick-error,MIT OR Apache-2.0,\"Paul Colomiets , Colin Kiegel \"\nquick_cache,https://github.com/arthurprs/quick-cache,MIT,Arthur Silva \nquinn,https://github.com/quinn-rs/quinn,MIT OR Apache-2.0,The quinn Authors\nquinn-proto,https://github.com/quinn-rs/quinn,MIT OR Apache-2.0,The quinn-proto Authors\nquinn-udp,https://github.com/quinn-rs/quinn,MIT OR Apache-2.0,The quinn-udp Authors\nquote,https://github.com/dtolnay/quote,MIT OR Apache-2.0,David Tolnay \nr-efi,https://github.com/r-efi/r-efi,MIT OR Apache-2.0 OR LGPL-2.1-or-later,The r-efi Authors\nrand,https://github.com/rust-random/rand,MIT OR Apache-2.0,\"The Rand Project Developers, The Rust Project Developers\"\nrand_chacha,https://github.com/rust-random/rand,MIT OR Apache-2.0,\"The Rand Project Developers, The Rust Project Developers, The CryptoCorrosion Contributors\"\nrand_core,https://github.com/rust-random/rand,MIT OR Apache-2.0,\"The Rand Project Developers, The Rust Project Developers\"\nrand_xorshift,https://github.com/rust-random/rngs,MIT OR Apache-2.0,\"The Rand Project Developers, The Rust Project Developers\"\nraw-cpuid,https://github.com/gz/rust-cpuid,MIT,Gerd Zellweger \nrayon,https://github.com/rayon-rs/rayon,MIT OR Apache-2.0,The rayon Authors\nrayon-core,https://github.com/rayon-rs/rayon,MIT OR Apache-2.0,The rayon-core Authors\nredox_syscall,https://gitlab.redox-os.org/redox-os/syscall,MIT,Jeremy Soller \nref-cast,https://github.com/dtolnay/ref-cast,MIT OR Apache-2.0,David Tolnay \nref-cast-impl,https://github.com/dtolnay/ref-cast,MIT OR Apache-2.0,David Tolnay \nregex,https://github.com/rust-lang/regex,MIT OR Apache-2.0,\"The Rust Project Developers, Andrew Gallant \"\nregex-automata,https://github.com/rust-lang/regex,MIT OR Apache-2.0,\"The Rust Project Developers, Andrew Gallant \"\nregex-lite,https://github.com/rust-lang/regex,MIT OR Apache-2.0,\"The Rust Project Developers, Andrew Gallant \"\nregex-syntax,https://github.com/rust-lang/regex,MIT OR Apache-2.0,\"The Rust Project Developers, Andrew Gallant \"\nreqwest,https://github.com/seanmonstar/reqwest,MIT OR Apache-2.0,Sean McArthur \nreqwest-middleware,https://github.com/TrueLayer/reqwest-middleware,MIT OR Apache-2.0,Rodrigo Gryzinski \nreqwest-retry,https://github.com/TrueLayer/reqwest-middleware,MIT OR Apache-2.0,Rodrigo Gryzinski \nretry-policies,https://github.com/TrueLayer/retry-policies,MIT OR Apache-2.0,Luca Palmieri \nrfc6979,https://github.com/RustCrypto/signatures/tree/master/rfc6979,Apache-2.0 OR MIT,RustCrypto Developers\nring,https://github.com/briansmith/ring,Apache-2.0 AND ISC,The ring Authors\nroxmltree,https://github.com/RazrFalcon/roxmltree,MIT OR Apache-2.0,Evgeniy Reizner \nrust-embed,https://pyrossh.dev/repos/rust-embed,MIT,pyrossh\nrust-embed-impl,https://pyrossh.dev/repos/rust-embed,MIT,pyrossh\nrust-embed-utils,https://pyrossh.dev/repos/rust-embed,MIT,pyrossh\nrustc-hash,https://github.com/rust-lang/rustc-hash,Apache-2.0 OR MIT,The Rust Project Developers\nrustix,https://github.com/bytecodealliance/rustix,Apache-2.0 WITH LLVM-exception OR Apache-2.0 OR MIT,\"Dan Gohman , Jakub Konka \"\nrustls,https://github.com/rustls/rustls,Apache-2.0 OR ISC OR MIT,The rustls Authors\nrustls-native-certs,https://github.com/rustls/rustls-native-certs,Apache-2.0 OR ISC OR MIT,The rustls-native-certs Authors\nrustls-pemfile,https://github.com/rustls/pemfile,Apache-2.0 OR ISC OR MIT,The rustls-pemfile Authors\nrustls-pki-types,https://github.com/rustls/pki-types,MIT OR Apache-2.0,The rustls-pki-types Authors\nrustls-webpki,https://github.com/rustls/webpki,ISC,The rustls-webpki Authors\nrustop,https://chiselapp.com/user/fifr/repository/rustop,MIT,Frank Fischer \nrustversion,https://github.com/dtolnay/rustversion,MIT OR Apache-2.0,David Tolnay \nrusty-fork,https://github.com/altsysrq/rusty-fork,MIT OR Apache-2.0,Jason Lingle\nryu,https://github.com/dtolnay/ryu,Apache-2.0 OR BSL-1.0,David Tolnay \nsame-file,https://github.com/BurntSushi/same-file,Unlicense OR MIT,Andrew Gallant \nscc,https://github.com/wvwwvwwv/scalable-concurrent-containers,Apache-2.0,wvwwvwwv \nschannel,https://github.com/steffengy/schannel-rs,MIT,\"Steven Fackler , Steffen Butzer \"\nschemars,https://github.com/GREsau/schemars,MIT,Graham Esau \nscoped-tls,https://github.com/alexcrichton/scoped-tls,MIT OR Apache-2.0,Alex Crichton \nscopeguard,https://github.com/bluss/scopeguard,MIT OR Apache-2.0,bluss\nsct,https://github.com/rustls/sct.rs,Apache-2.0 OR ISC OR MIT,Joseph Birr-Pixton \nsdd,https://github.com/wvwwvwwv/scalable-delayed-dealloc,Apache-2.0,wvwwvwwv \nsec1,https://github.com/RustCrypto/formats/tree/master/sec1,Apache-2.0 OR MIT,RustCrypto Developers\nsecurity-framework,https://github.com/kornelski/rust-security-framework,MIT OR Apache-2.0,\"Steven Fackler , Kornel \"\nsecurity-framework-sys,https://github.com/kornelski/rust-security-framework,MIT OR Apache-2.0,\"Steven Fackler , Kornel \"\nsemver,https://github.com/dtolnay/semver,MIT OR Apache-2.0,David Tolnay \nseparator,https://github.com/saghm/rust-separator,MIT,Saghm Rossi \nserde,https://github.com/serde-rs/serde,MIT OR Apache-2.0,\"Erick Tryzelaar , David Tolnay \"\nserde_core,https://github.com/serde-rs/serde,MIT OR Apache-2.0,\"Erick Tryzelaar , David Tolnay \"\nserde_derive,https://github.com/serde-rs/serde,MIT OR Apache-2.0,\"Erick Tryzelaar , David Tolnay \"\nserde_json,https://github.com/serde-rs/json,MIT OR Apache-2.0,\"Erick Tryzelaar , David Tolnay \"\nserde_json_borrow,https://github.com/PSeitz/serde_json_borrow,MIT,Pascal Seitz \nserde_path_to_error,https://github.com/dtolnay/path-to-error,MIT OR Apache-2.0,David Tolnay \nserde_qs,https://github.com/samscott89/serde_qs,MIT OR Apache-2.0,Sam Scott \nserde_spanned,https://github.com/toml-rs/toml,MIT OR Apache-2.0,The serde_spanned Authors\nserde_urlencoded,https://github.com/nox/serde_urlencoded,MIT OR Apache-2.0,Anthony Ramine \nserde_with,https://github.com/jonasbb/serde_with,MIT OR Apache-2.0,\"Jonas Bushart, Marcin Kaźmierczak\"\nserde_with_macros,https://github.com/jonasbb/serde_with,MIT OR Apache-2.0,Jonas Bushart\nserde_yaml,https://github.com/dtolnay/serde-yaml,MIT OR Apache-2.0,David Tolnay \nserial_test_derive,https://github.com/palfrey/serial_test,MIT,Tom Parker-Shemilt \nsha1,https://github.com/RustCrypto/hashes,MIT OR Apache-2.0,RustCrypto Developers\nsha2,https://github.com/RustCrypto/hashes,MIT OR Apache-2.0,RustCrypto Developers\nsharded-slab,https://github.com/hawkw/sharded-slab,MIT,Eliza Weisman \nshell-words,https://github.com/tmiasko/shell-words,MIT OR Apache-2.0,Tomasz Miąsko \nshlex,https://github.com/comex/rust-shlex,MIT OR Apache-2.0,\"comex , Fenhl , Adrian Taylor , Alex Touchet , Daniel Parks , Garrett Berg \"\nsignal-hook-registry,https://github.com/vorner/signal-hook,MIT OR Apache-2.0,\"Michal 'vorner' Vaner , Masaki Hara \"\nsignature,https://github.com/RustCrypto/traits/tree/master/signature,Apache-2.0 OR MIT,RustCrypto Developers\nsimd-adler32,https://github.com/mcountryman/simd-adler32,MIT,Marvin Countryman \nsiphasher,https://github.com/jedisct1/rust-siphash,MIT OR Apache-2.0,Frank Denis \nsketches-ddsketch,https://github.com/mheffner/rust-sketches-ddsketch,Apache-2.0,Mike Heffner \nslab,https://github.com/tokio-rs/slab,MIT,Carl Lerche \nsmallvec,https://github.com/servo/rust-smallvec,MIT OR Apache-2.0,The Servo Project Developers\nsocket2,https://github.com/rust-lang/socket2,MIT OR Apache-2.0,\"Alex Crichton , Thomas de Zeeuw \"\nspin,https://github.com/mvdnes/spin-rs,MIT,\"Mathijs van de Nes , John Ericson , Joshua Barretto \"\nspinning_top,https://github.com/rust-osdev/spinning_top,MIT OR Apache-2.0,Philipp Oppermann \nspki,https://github.com/RustCrypto/formats/tree/master/spki,Apache-2.0 OR MIT,RustCrypto Developers\nstable_deref_trait,https://github.com/storyyeller/stable_deref_trait,MIT OR Apache-2.0,Robert Grosse \nstatic_assertions,https://github.com/nvzqz/static-assertions-rs,MIT OR Apache-2.0,Nikolai Vazquez\nstrsim,https://github.com/rapidfuzz/strsim-rs,MIT,\"Danny Guo , maxbachmann \"\nsubtle,https://github.com/dalek-cryptography/subtle,BSD-3-Clause,\"Isis Lovecruft , Henry de Valence \"\nsyn,https://github.com/dtolnay/syn,MIT OR Apache-2.0,David Tolnay \nsync_wrapper,https://github.com/Actyx/sync_wrapper,Apache-2.0,Actyx AG \nsynstructure,https://github.com/mystor/synstructure,MIT,Nika Layzell \nsysinfo,https://github.com/GuillaumeGomez/sysinfo,MIT,Guillaume Gomez \ntabled,https://github.com/zhiburt/tabled,MIT,Maxim Zhiburt \ntabled_derive,https://github.com/zhiburt/tabled,MIT,Maxim Zhiburt \ntagptr,https://github.com/oliver-giersch/tagptr,MIT OR Apache-2.0,Oliver Giersch\ntantivy,https://github.com/quickwit-oss/tantivy,MIT,Paul Masurel \ntantivy-bitpacker,https://github.com/quickwit-oss/tantivy,MIT,Paul Masurel \ntantivy-columnar,https://github.com/quickwit-oss/tantivy,MIT,The tantivy-columnar Authors\ntantivy-common,https://github.com/quickwit-oss/tantivy,MIT,\"Paul Masurel , Pascal Seitz \"\ntantivy-fst,https://github.com/quickwit-inc/fst,Unlicense OR MIT,Andrew Gallant \ntantivy-query-grammar,https://github.com/quickwit-oss/tantivy,MIT,Paul Masurel \ntantivy-sstable,https://github.com/quickwit-oss/tantivy,MIT,The tantivy-sstable Authors\ntantivy-stacker,https://github.com/quickwit-oss/tantivy,MIT,The tantivy-stacker Authors\ntantivy-tokenizer-api,https://github.com/quickwit-oss/tantivy,MIT,The tantivy-tokenizer-api Authors\ntempfile,https://github.com/Stebalien/tempfile,MIT OR Apache-2.0,\"Steven Allen , The Rust Project Developers, Ashley Mannix , Jason White \"\ntermtree,https://github.com/rust-cli/termtree,MIT,The termtree Authors\ntesting_table,https://github.com/zhiburt/tabled,MIT,Maxim Zhiburt \nthiserror,https://github.com/dtolnay/thiserror,MIT OR Apache-2.0,David Tolnay \nthiserror-impl,https://github.com/dtolnay/thiserror,MIT OR Apache-2.0,David Tolnay \nthousands,https://github.com/tov/thousands-rs,MIT OR Apache-2.0,Jesse A. Tov \nthread_local,https://github.com/Amanieu/thread_local-rs,MIT OR Apache-2.0,Amanieu d'Antras \ntime,https://github.com/time-rs/time,MIT OR Apache-2.0,\"Jacob Pratt , Time contributors\"\ntime-core,https://github.com/time-rs/time,MIT OR Apache-2.0,\"Jacob Pratt , Time contributors\"\ntime-fmt,https://github.com/MiSawa/time-fmt,MIT OR Apache-2.0,mi_sawa \ntime-macros,https://github.com/time-rs/time,MIT OR Apache-2.0,\"Jacob Pratt , Time contributors\"\ntinystr,https://github.com/unicode-org/icu4x,Unicode-3.0,The ICU4X Project Developers\ntinytemplate,https://github.com/bheisler/TinyTemplate,Apache-2.0 OR MIT,Brook Heisler \ntinyvec,https://github.com/Lokathor/tinyvec,Zlib OR Apache-2.0 OR MIT,Lokathor \ntinyvec_macros,https://github.com/Soveu/tinyvec_macros,MIT OR Apache-2.0 OR Zlib,Soveu \ntokio,https://github.com/tokio-rs/tokio,MIT,Tokio Contributors \ntokio-macros,https://github.com/tokio-rs/tokio,MIT,Tokio Contributors \ntokio-metrics,https://github.com/tokio-rs/tokio-metrics,MIT,Tokio Contributors \ntokio-rustls,https://github.com/rustls/tokio-rustls,MIT OR Apache-2.0,The tokio-rustls Authors\ntokio-stream,https://github.com/tokio-rs/tokio,MIT,Tokio Contributors \ntokio-util,https://github.com/tokio-rs/tokio,MIT,Tokio Contributors \ntoml,https://github.com/toml-rs/toml,MIT OR Apache-2.0,The toml Authors\ntoml_datetime,https://github.com/toml-rs/toml,MIT OR Apache-2.0,The toml_datetime Authors\ntoml_parser,https://github.com/toml-rs/toml,MIT OR Apache-2.0,The toml_parser Authors\ntoml_writer,https://github.com/toml-rs/toml,MIT OR Apache-2.0,The toml_writer Authors\ntonic,https://github.com/hyperium/tonic,MIT,Lucio Franco \ntonic-build,https://github.com/hyperium/tonic,MIT,Lucio Franco \ntonic-health,https://github.com/hyperium/tonic,MIT,James Nugent \ntonic-prost,https://github.com/hyperium/tonic,MIT,Lucio Franco \ntonic-prost-build,https://github.com/hyperium/tonic,MIT,Lucio Franco \ntonic-reflection,https://github.com/hyperium/tonic,MIT,\"James Nugent , Samani G. Gikandi \"\ntower,https://github.com/tower-rs/tower,MIT,Tower Maintainers \ntower-http,https://github.com/tower-rs/tower-http,MIT,Tower Maintainers \ntower-layer,https://github.com/tower-rs/tower,MIT,Tower Maintainers \ntower-service,https://github.com/tower-rs/tower,MIT,Tower Maintainers \ntracing,https://github.com/tokio-rs/tracing,MIT,\"Eliza Weisman , Tokio Contributors \"\ntracing-attributes,https://github.com/tokio-rs/tracing,MIT,\"Tokio Contributors , Eliza Weisman , David Barsky \"\ntracing-core,https://github.com/tokio-rs/tracing,MIT,Tokio Contributors \ntracing-log,https://github.com/tokio-rs/tracing,MIT,Tokio Contributors \ntracing-opentelemetry,https://github.com/tokio-rs/tracing-opentelemetry,MIT,The tracing-opentelemetry Authors\ntracing-serde,https://github.com/tokio-rs/tracing,MIT,Tokio Contributors \ntracing-subscriber,https://github.com/tokio-rs/tracing,MIT,\"Eliza Weisman , David Barsky , Tokio Contributors \"\ntriomphe,https://github.com/Manishearth/triomphe,MIT OR Apache-2.0,\"Manish Goregaokar , The Servo Project Developers\"\ntry-lock,https://github.com/seanmonstar/try-lock,MIT,Sean McArthur \nttl_cache,https://github.com/stusmall/ttl_cache,MIT OR Apache-2.0,Stu Small \ntypeid,https://github.com/dtolnay/typeid,MIT OR Apache-2.0,David Tolnay \ntypenum,https://github.com/paholg/typenum,MIT OR Apache-2.0,\"Paho Lurie-Gregg , Andre Bogus \"\ntypetag,https://github.com/dtolnay/typetag,MIT OR Apache-2.0,David Tolnay \ntypetag-impl,https://github.com/dtolnay/typetag,MIT OR Apache-2.0,David Tolnay \nulid,https://github.com/dylanhart/ulid-rs,MIT,dylanhart \nunarray,https://github.com/cameron1024/unarray,MIT OR Apache-2.0,The unarray Authors\nunicase,https://github.com/seanmonstar/unicase,MIT OR Apache-2.0,Sean McArthur \nunicode-ident,https://github.com/dtolnay/unicode-ident,(MIT OR Apache-2.0) AND Unicode-3.0,David Tolnay \nunicode-width,https://github.com/unicode-rs/unicode-width,MIT OR Apache-2.0,\"kwantam , Manish Goregaokar \"\nunit-prefix,https://codeberg.org/commons-rs/unit-prefix,MIT,\"Fabio Valentini , Benjamin Sago \"\nunsafe-libyaml,https://github.com/dtolnay/unsafe-libyaml,MIT,David Tolnay \nuntrusted,https://github.com/briansmith/untrusted,ISC,Brian Smith \nureq-proto,https://github.com/algesten/ureq-proto,MIT OR Apache-2.0,Martin Algesten \nurl,https://github.com/servo/rust-url,MIT OR Apache-2.0,The rust-url developers\nurlencoding,https://github.com/kornelski/rust_urlencoding,MIT,\"Kornel , Bertram Truong \"\nusername,https://pijul.org/darcs/user,MIT OR Apache-2.0,Pierre-Étienne Meunier \nutf-8,https://github.com/SimonSapin/rust-utf8,MIT OR Apache-2.0,Simon Sapin \nutf8-ranges,https://github.com/BurntSushi/utf8-ranges,Unlicense OR MIT,Andrew Gallant \nutf8_iter,https://github.com/hsivonen/utf8_iter,Apache-2.0 OR MIT,Henri Sivonen \nutf8parse,https://github.com/alacritty/vte,Apache-2.0 OR MIT,\"Joe Wilm , Christian Duerr \"\nutoipa,https://github.com/juhaku/utoipa,MIT OR Apache-2.0,Juha Kukkonen \nutoipa-gen,https://github.com/juhaku/utoipa,MIT OR Apache-2.0,Juha Kukkonen \nuuid,https://github.com/uuid-rs/uuid,Apache-2.0 OR MIT,\"Ashley Mannix, Dylan DPC, Hunar Roop Kahlon\"\nvaluable,https://github.com/tokio-rs/valuable,MIT,The valuable Authors\nvsimd,https://github.com/Nugine/simd,MIT,The vsimd Authors\nvte,https://github.com/alacritty/vte,Apache-2.0 OR MIT,\"Joe Wilm , Christian Duerr \"\nwait-timeout,https://github.com/alexcrichton/wait-timeout,MIT OR Apache-2.0,Alex Crichton \nwalkdir,https://github.com/BurntSushi/walkdir,Unlicense OR MIT,Andrew Gallant \nwant,https://github.com/seanmonstar/want,MIT,Sean McArthur \nwarp,https://github.com/seanmonstar/warp,MIT,Sean McArthur \nwasi,https://github.com/bytecodealliance/wasi,Apache-2.0 WITH LLVM-exception OR Apache-2.0 OR MIT,The Cranelift Project Developers\nwasip2,https://github.com/bytecodealliance/wasi-rs,Apache-2.0 WITH LLVM-exception OR Apache-2.0 OR MIT,The wasip2 Authors\nwasix,https://github.com/wasix-org/wasix-abi-rust,Apache-2.0 WITH LLVM-exception OR Apache-2.0 OR MIT,\"The Cranelift Project Developers, john-sharratt\"\nwasm-bindgen,https://github.com/wasm-bindgen/wasm-bindgen,MIT OR Apache-2.0,The wasm-bindgen Developers\nwasm-bindgen-futures,https://github.com/wasm-bindgen/wasm-bindgen/tree/master/crates/futures,MIT OR Apache-2.0,The wasm-bindgen Developers\nwasm-bindgen-macro,https://github.com/wasm-bindgen/wasm-bindgen/tree/master/crates/macro,MIT OR Apache-2.0,The wasm-bindgen Developers\nwasm-bindgen-macro-support,https://github.com/wasm-bindgen/wasm-bindgen/tree/master/crates/macro-support,MIT OR Apache-2.0,The wasm-bindgen Developers\nwasm-bindgen-shared,https://github.com/wasm-bindgen/wasm-bindgen/tree/master/crates/shared,MIT OR Apache-2.0,The wasm-bindgen Developers\nwasmtimer,https://github.com/whizsid/wasmtimer-rs,MIT,\"WhizSid , Pierre Krieger \"\nweb-sys,https://github.com/wasm-bindgen/wasm-bindgen/tree/master/crates/web-sys,MIT OR Apache-2.0,The wasm-bindgen Developers\nweb-time,https://github.com/daxpedda/web-time,MIT OR Apache-2.0,The web-time Authors\nwebpki-roots,https://github.com/rustls/webpki-roots,CDLA-Permissive-2.0,The webpki-roots Authors\nwinapi,https://github.com/retep998/winapi-rs,MIT,Peter Atashian \nwinapi,https://github.com/retep998/winapi-rs,MIT OR Apache-2.0,Peter Atashian \nwinapi-i686-pc-windows-gnu,https://github.com/retep998/winapi-rs,MIT OR Apache-2.0,Peter Atashian \nwinapi-util,https://github.com/BurntSushi/winapi-util,Unlicense OR MIT,Andrew Gallant \nwinapi-x86_64-pc-windows-gnu,https://github.com/retep998/winapi-rs,MIT OR Apache-2.0,Peter Atashian \nwindows,https://github.com/microsoft/windows-rs,MIT OR Apache-2.0,Microsoft\nwindows-collections,https://github.com/microsoft/windows-rs,MIT OR Apache-2.0,The windows-collections Authors\nwindows-core,https://github.com/microsoft/windows-rs,MIT OR Apache-2.0,Microsoft\nwindows-core,https://github.com/microsoft/windows-rs,MIT OR Apache-2.0,The windows-core Authors\nwindows-future,https://github.com/microsoft/windows-rs,MIT OR Apache-2.0,The windows-future Authors\nwindows-implement,https://github.com/microsoft/windows-rs,MIT OR Apache-2.0,The windows-implement Authors\nwindows-interface,https://github.com/microsoft/windows-rs,MIT OR Apache-2.0,The windows-interface Authors\nwindows-link,https://github.com/microsoft/windows-rs,MIT OR Apache-2.0,Microsoft\nwindows-link,https://github.com/microsoft/windows-rs,MIT OR Apache-2.0,The windows-link Authors\nwindows-numerics,https://github.com/microsoft/windows-rs,MIT OR Apache-2.0,The windows-numerics Authors\nwindows-result,https://github.com/microsoft/windows-rs,MIT OR Apache-2.0,Microsoft\nwindows-result,https://github.com/microsoft/windows-rs,MIT OR Apache-2.0,The windows-result Authors\nwindows-strings,https://github.com/microsoft/windows-rs,MIT OR Apache-2.0,Microsoft\nwindows-strings,https://github.com/microsoft/windows-rs,MIT OR Apache-2.0,The windows-strings Authors\nwindows-sys,https://github.com/microsoft/windows-rs,MIT OR Apache-2.0,Microsoft\nwindows-sys,https://github.com/microsoft/windows-rs,MIT OR Apache-2.0,The windows-sys Authors\nwindows-targets,https://github.com/microsoft/windows-rs,MIT OR Apache-2.0,Microsoft\nwindows-targets,https://github.com/microsoft/windows-rs,MIT OR Apache-2.0,The windows-targets Authors\nwindows-threading,https://github.com/microsoft/windows-rs,MIT OR Apache-2.0,Microsoft\nwindows_aarch64_gnullvm,https://github.com/microsoft/windows-rs,MIT OR Apache-2.0,Microsoft\nwindows_aarch64_gnullvm,https://github.com/microsoft/windows-rs,MIT OR Apache-2.0,The windows_aarch64_gnullvm Authors\nwindows_aarch64_msvc,https://github.com/microsoft/windows-rs,MIT OR Apache-2.0,Microsoft\nwindows_aarch64_msvc,https://github.com/microsoft/windows-rs,MIT OR Apache-2.0,The windows_aarch64_msvc Authors\nwindows_i686_gnu,https://github.com/microsoft/windows-rs,MIT OR Apache-2.0,Microsoft\nwindows_i686_gnu,https://github.com/microsoft/windows-rs,MIT OR Apache-2.0,The windows_i686_gnu Authors\nwindows_i686_gnullvm,https://github.com/microsoft/windows-rs,MIT OR Apache-2.0,Microsoft\nwindows_i686_gnullvm,https://github.com/microsoft/windows-rs,MIT OR Apache-2.0,The windows_i686_gnullvm Authors\nwindows_i686_msvc,https://github.com/microsoft/windows-rs,MIT OR Apache-2.0,Microsoft\nwindows_i686_msvc,https://github.com/microsoft/windows-rs,MIT OR Apache-2.0,The windows_i686_msvc Authors\nwindows_x86_64_gnu,https://github.com/microsoft/windows-rs,MIT OR Apache-2.0,Microsoft\nwindows_x86_64_gnu,https://github.com/microsoft/windows-rs,MIT OR Apache-2.0,The windows_x86_64_gnu Authors\nwindows_x86_64_gnullvm,https://github.com/microsoft/windows-rs,MIT OR Apache-2.0,Microsoft\nwindows_x86_64_gnullvm,https://github.com/microsoft/windows-rs,MIT OR Apache-2.0,The windows_x86_64_gnullvm Authors\nwindows_x86_64_msvc,https://github.com/microsoft/windows-rs,MIT OR Apache-2.0,Microsoft\nwindows_x86_64_msvc,https://github.com/microsoft/windows-rs,MIT OR Apache-2.0,The windows_x86_64_msvc Authors\nwinnow,https://github.com/winnow-rs/winnow,MIT,The winnow Authors\nwit-bindgen,https://github.com/bytecodealliance/wit-bindgen,Apache-2.0 WITH LLVM-exception OR Apache-2.0 OR MIT,Alex Crichton \nwriteable,https://github.com/unicode-org/icu4x,Unicode-3.0,The ICU4X Project Developers\nxmlparser,https://github.com/RazrFalcon/xmlparser,MIT OR Apache-2.0,Yevhenii Reizner \nyansi,https://github.com/SergioBenitez/yansi,MIT OR Apache-2.0,Sergio Benitez \nyoke,https://github.com/unicode-org/icu4x,Unicode-3.0,Manish Goregaokar \nyoke-derive,https://github.com/unicode-org/icu4x,Unicode-3.0,Manish Goregaokar \nzerocopy,https://github.com/google/zerocopy,BSD-2-Clause OR Apache-2.0 OR MIT,\"Joshua Liebow-Feeser , Jack Wrenn \"\nzerocopy-derive,https://github.com/google/zerocopy,BSD-2-Clause OR Apache-2.0 OR MIT,\"Joshua Liebow-Feeser , Jack Wrenn \"\nzerofrom,https://github.com/unicode-org/icu4x,Unicode-3.0,Manish Goregaokar \nzerofrom-derive,https://github.com/unicode-org/icu4x,Unicode-3.0,Manish Goregaokar \nzeroize,https://github.com/RustCrypto/utils,Apache-2.0 OR MIT,The RustCrypto Project Developers\nzerotrie,https://github.com/unicode-org/icu4x,Unicode-3.0,The ICU4X Project Developers\nzerovec,https://github.com/unicode-org/icu4x,Unicode-3.0,The ICU4X Project Developers\nzerovec-derive,https://github.com/unicode-org/icu4x,Unicode-3.0,Manish Goregaokar \nzmij,https://github.com/dtolnay/zmij,MIT,David Tolnay \nzstd,https://github.com/gyscos/zstd-rs,MIT,Alexandre Bury \nzstd-safe,https://github.com/gyscos/zstd-rs,MIT OR Apache-2.0,Alexandre Bury \nzstd-sys,https://github.com/gyscos/zstd-rs,MIT OR Apache-2.0,Alexandre Bury \n" }, { "path": "Makefile", "content": "DOCKER_SERVICES ?= all\n\nQUICKWIT_SRC = quickwit\n\nhelp:\n\t@grep '^[^\\.#[:space:]].*:' Makefile\n\n\nIMAGE_TAG := $(shell git branch --show-current | tr '\\#/' '-')\n\nQW_COMMIT_DATE := $(shell TZ=UTC0 git log -1 --format=%cd --date=format-local:'%Y-%m-%dT%H:%M:%SZ')\nQW_COMMIT_HASH := $(shell git rev-parse HEAD)\nQW_COMMIT_TAGS := $(shell git tag --points-at HEAD | tr '\\n' ',')\n\ndocker-build:\n\t@docker build \\\n\t\t--build-arg QW_COMMIT_DATE=$(QW_COMMIT_DATE) \\\n\t\t--build-arg QW_COMMIT_HASH=$(QW_COMMIT_HASH) \\\n\t\t--build-arg QW_COMMIT_TAGS=$(QW_COMMIT_TAGS) \\\n\t\t-t quickwit/quickwit:$(IMAGE_TAG) .\n\n# Usage:\n# `make docker-compose-up` starts all the services.\n# `make docker-compose-up DOCKER_SERVICES='jaeger,localstack'` starts the subset of services matching the profiles.\ndocker-compose-up:\n\t@echo \"Launching ${DOCKER_SERVICES} Docker service(s)\"\n\tCOMPOSE_PROFILES=$(DOCKER_SERVICES) docker compose -f docker-compose.yml up -d --remove-orphans --wait\n\ndocker-compose-down:\n\tdocker compose -p quickwit down --remove-orphans\n\ndocker-compose-logs:\n\tdocker compose logs -f docker-compose.yml -t\n\ndocker-compose-monitoring:\n\tCOMPOSE_PROFILES=monitoring docker compose -f docker-compose.yml up -d --remove-orphans\n\ndocker-rm-postgres-volume:\n\tdocker volume rm quickwit_postgres_data\n\ndocker-rm-volumes:\n\tdocker volume rm quickwit_azurite_data quickwit_fake_gcs_server_data quickwit_grafana_conf quickwit_grafana_data quickwit_localstack_data quickwit_postgres_data\n\ndoc:\n\t@$(MAKE) -C $(QUICKWIT_SRC) doc\n\nfmt:\n\t@$(MAKE) -C $(QUICKWIT_SRC) fmt\n\nfix:\n\t@$(MAKE) -C $(QUICKWIT_SRC) fix\n\ntypos:\n\ttypos\n\n# Usage:\n# `make test-all` starts the Docker services and runs all the tests.\n# `make -k test-all docker-compose-down`, tears down the Docker services after running all the tests.\ntest-all: docker-compose-up\n\t@$(MAKE) -C $(QUICKWIT_SRC) test-all\n\ntest-failpoints:\n\t@$(MAKE) -C $(QUICKWIT_SRC) test-failpoints\n\n# This will build and push all custom cross images for cross-compilation.\n# You will need to login into Docker Hub with the `quickwit` account.\nIMAGE_TAGS = x86_64-unknown-linux-gnu aarch64-unknown-linux-gnu x86_64-unknown-linux-musl aarch64-unknown-linux-musl\n\n.PHONY: cross-images\ncross-images:\n\t@for tag in ${IMAGE_TAGS}; do \\\n\t\tdocker build --tag quickwit/cross:$$tag --file ./build/cross-images/$$tag.dockerfile ./build/cross-images; \\\n\t\tdocker push quickwit/cross:$$tag; \\\n\tdone\n\n# TODO: to be replaced by https://github.com/quickwit-oss/quickwit/issues/237\n.PHONY: build\nbuild: build-ui\n\t$(MAKE) -C $(QUICKWIT_SRC) build\n\n# Usage:\n# `BINARY_FILE=path/to/quickwit/binary BINARY_VERSION=0.1.0 ARCHIVE_NAME=quickwit make archive`\n# - BINARY_FILE: Path of the quickwit binary file.\n# - BINARY_VERSION: Version of the quickwit binary.\n# - ARCHIVE_NAME: Name of the resulting archive file (without extension).\n.PHONY: archive\narchive:\n\t@echo \"Archiving release binary & assets\"\n\t@mkdir -p \"./quickwit-${BINARY_VERSION}/config\"\n\t@mkdir -p \"./quickwit-${BINARY_VERSION}/qwdata\"\n\t@cp ./config/quickwit.yaml \"./quickwit-${BINARY_VERSION}/config\"\n\t@cp ./LICENSE \"./quickwit-${BINARY_VERSION}\"\n\t@cp \"${BINARY_FILE}\" \"./quickwit-${BINARY_VERSION}\"\n\t@tar -czf \"${ARCHIVE_NAME}.tar.gz\" \"./quickwit-${BINARY_VERSION}\"\n\t@rm -rf \"./quickwit-${BINARY_VERSION}\"\n\nworkspace-deps-tree:\n\t$(MAKE) -C $(QUICKWIT_SRC) workspace-deps-tree\n\n.PHONY: build-rustdoc\nbuild-rustdoc:\n\t$(MAKE) -C $(QUICKWIT_SRC) build-rustdoc\n\n.PHONY: build-ui\nbuild-ui:\n\t$(MAKE) -C $(QUICKWIT_SRC) build-ui\n" }, { "path": "README.md", "content": "[![CI](https://github.com/quickwit-oss/quickwit/actions/workflows/ci.yml/badge.svg)](https://github.com/quickwit-oss/quickwit/actions?query=workflow%3ACI+branch%3Amain)\n[![codecov](https://codecov.io/gh/quickwit-oss/quickwit/branch/main/graph/badge.svg?token=06SRGAV5SS)](https://codecov.io/gh/quickwit-oss/quickwit)\n[![OpenSSF Scorecard](https://api.scorecard.dev/projects/github.com/quickwit-oss/quickwit/badge)](https://scorecard.dev/viewer/?uri=github.com/quickwit-oss/quickwit)\n[![Contributor Covenant](https://img.shields.io/badge/Contributor%20Covenant-2.0-4baaaa.svg)](CODE_OF_CONDUCT.md)\n[![License: Apache 2.0](https://img.shields.io/badge/license-Apache%202.0-blue?style=flat-square)](LICENSE)\n[![Twitter Follow](https://img.shields.io/twitter/follow/Quickwit_Inc?color=%231DA1F2&logo=Twitter&style=plastic)](https://twitter.com/Quickwit_Inc)\n[![Discord](https://img.shields.io/discord/908281611840282624?logo=Discord&logoColor=%23FFFFFF&style=plastic)](https://discord.quickwit.io)\n
\n\n
\n
\n

\n \"Quickwit\n \"Quickwit\n

\n\n

\nCloud-native search engine for observability (logs, traces, and soon metrics!). An open-source alternative to Datadog, Elasticsearch, Loki, and Tempo.\n

\n\n

\n Quickstart |\n Docs |\n Tutorials |\n Chat |\n Download\n

\n
\n\nWe just released Quickwit 0.8! Read the [blog post](https://quickwit.io/blog/quickwit-0.8) to learn about the latest powerful features!\n\n### **Quickwit is the fastest search engine on cloud storage. It's the perfect fit for observability use cases**\n\n- [Log management](https://quickwit.io/docs/log-management/overview)\n- [Distributed tracing](https://quickwit.io/docs/distributed-tracing/overview)\n- Metrics support is on the roadmap\n\n### 🚀 Quickstart\n\n- [Search and analytics on Stack Overflow dataset](https://quickwit.io/docs/get-started/quickstart)\n- [Trace analytics with Grafana](https://quickwit.io/docs/get-started/tutorials/trace-analytics-with-grafana)\n- [Distributed tracing with Jaeger](https://quickwit.io/docs/get-started/tutorials/tutorial-jaeger)\n\n
\n\n\n\n
\n\n# 💡 Features\n\n- Full-text search and aggregation queries\n- Elasticsearch-compatible API, use Quickwit with any Elasticsearch or OpenSearch client\n- [Jaeger-native](https://quickwit.io/docs/distributed-tracing/plug-quickwit-to-jaeger)\n- OTEL-native for [logs](https://quickwit.io/docs/log-management/overview) and [traces](https://quickwit.io/docs/distributed-tracing/overview)\n- [Schemaless](https://quickwit.io/docs/guides/schemaless) or strict schema indexing\n- Schemaless analytics\n- Sub-second search on cloud storage (Amazon S3, Azure Blob Storage, Google Cloud Storage, …)\n- Decoupled compute and storage, stateless indexers & searchers\n- [Grafana data source](https://github.com/quickwit-oss/quickwit-datasource)\n- Kubernetes ready - See our [helm-chart](https://quickwit.io/docs/deployment/kubernetes/helm)\n- RESTful API\n\n## Enterprise ready\n\n- Multiple [data sources](https://quickwit.io/docs/ingest-data/) Kafka / Kinesis / Pulsar native\n- Multi-tenancy: indexing with many indexes and partitioning\n- Retention policies\n- Delete tasks (for GDPR use cases)\n- Distributed and highly available* engine that scales out in seconds (*HA indexing only with Kafka)\n\n# 📑 Architecture overview\n\n![Quickwit Distributed Tracing](./docs/assets/images/quickwit-overview-light.svg#gh-light-mode-only)![Quickwit Distributed Tracing](./docs/assets/images/quickwit-overview-dark.svg#gh-dark-mode-only)\n\n- [Architecture overview]([https://quickwit.io/docs/distributed-tracing/overview](https://quickwit.io/docs/overview/architecture))\n- [Log management](https://quickwit.io/docs/log-management/overview)\n- [Distributed traces](https://quickwit.io/docs/distributed-tracing/overview)\n\n\n# 📕 Documentation\n\n- [Installation](https://quickwit.io/docs/get-started/installation)\n- [Log management with Quickwit](https://quickwit.io/docs/log-management/overview)\n- [Distributed Tracing with Quickwit](https://quickwit.io/docs/distributed-tracing/overview)\n- [Ingest data](https://quickwit.io/docs/ingest-data/)\n- [REST API](https://quickwit.io/docs/reference/rest-api)\n\n# 📚 Resources\n\n- [Blog posts](https://quickwit.io/blog/)\n- [Youtube channel](https://www.youtube.com/@quickwit8103)\n- [Discord](https://discord.quickwit.io)\n\n# 🔮 Roadmap\n\n- Quickwit 0.9 (July 2024)\n - Indexing and search performance improvements\n - Index configuration updates (retention policy, indexing and search settings)\n - Concatenated field\n\n- Quickwit 0.10 (October 2024)\n - Schema (doc mapping) updates\n - Native distributed ingestion\n - Index templates\n\n# 🙋 FAQ\n\n### How can I switch from Elasticsearch or OpenSearch to Quickwit?\n\nQuickwit supports a large subset of Elasticsearch/OpenSearch API.\n\nFor instance, it has an ES-compatible ingest API to make it easier to migrate your log shippers (Vector, Fluent Bit, Syslog, ...) to Quickwit.\n\nOn the search side, the most popular Elasticsearch endpoints, query DSL, and even aggregations are supported.\n\nThe list of available endpoints and queries is available [here](https://quickwit.io/docs/reference/es_compatible_api), while the list of supported aggregations is available [here](https://quickwit.io/docs/reference/aggregation).\n\nLet us know if part of the API you are using is missing!\n\nIf the client you are using is refusing to connect to Quickwit due to missing headers, you can use the `extra_headers` option in the [node configuration](https://quickwit.io/docs/configuration/node-config#rest-configuration) to impersonate any compatible version of Elasticsearch or OpenSearch.\n\n### How is Quickwit different from traditional search engines like Elasticsearch or Solr?\n\nThe core difference and advantage of Quickwit is its architecture built from the ground to search on cloud storage. We optimized IO paths, revamped the index data structures and made search stateless and sub-second on cloud storage.\n\n### How does Quickwit compare to Elastic in terms of cost?\n\nWe estimate that Quickwit can be up to 10x cheaper on average than Elastic. To understand how, check out our [blog post](https://quickwit.io/blog/commoncrawl/) about searching the web on AWS S3.\n\n### What license does Quickwit use?\n\nQuickwit is open-source under the Apache License, Version 2.0 - Apache-2.0.\n\n### Is it possible to set up Quickwit for a High Availability (HA)?\n\nHA is available for search, for indexing it's available only with a Kafka source.\n\n# 🤝 Contribute and spread the word\n\nWe are always thrilled to receive contributions: code, documentation, issues, or feedback. Here's how you can help us build the future of log management:\n\n- Start by checking out the [GitHub issues labeled \"Good first issue\"](https://github.com/quickwit-oss/quickwit/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22). These are a great place for newcomers to contribute.\n- Read our [Contributor Covenant Code of Conduct](./CODE_OF_CONDUCT.md) to understand our community standards.\n- [Create a fork of Quickwit](https://github.com/quickwit-oss/quickwit/fork) to have your own copy of the repository where you can make changes.\n- To understand how to contribute, read our [contributing guide](./CONTRIBUTING.md).\n- Set up your development environment following our [development setup guide](./CONTRIBUTING.md#development).\n- Once you've made your changes and tested them, you can contribute by [submitting a pull request](./CONTRIBUTING.md#submitting-a-pr).\n\n✨ After your contributions are accepted, don't forget to claim your swag by emailing us at hello@quickwit.io. Thank you for contributing!\n\n# 💬 Join Our Community\n\nWe welcome everyone to our community! Whether you're contributing code or just saying hello, we'd love to hear from you. Here's how you can connect with us:\n\n- Join the conversation on [Discord](https://discord.quickwit.io).\n- Follow us on [Twitter](https://twitter.com/Quickwit_Inc).\n- Check out our [website](https://quickwit.io/) and [blog](https://quickwit.io/blog) for the latest updates.\n- Watch our [YouTube](https://www.youtube.com/channel/UCvZVuRm2FiDq1_ul0mY85wA) channel for video content.\n" }, { "path": "SECURITY.md", "content": "# Security Policy\n\n## Supported Versions\n\n| Version | Supported |\n| ------- | ------------------ |\n| 0.3.1 | :white_check_mark: |\n| < 0.3.1 | :x: |\n\n## Reporting a Vulnerability\n\nTo disclose a vulnerability in our code, please notify us by email at security@quickwit.io or private message _@fulmicoton_ or _@guilload_ on our Discord \nserver ([discord.quickwit.io](https://discord.quickwit.io)). We will open a draft security advisory on our repository and grant you access so you can\nshare with us more details about the vulnerability. After releasing a fix, we will publish the security advisory to publicly disclose the security vulnerability\nto the project's community.\n" }, { "path": "_typos.toml", "content": "[files]\nextend-exclude = [\"**/*.json\"]\n\n[default.extend-words]\n# Don't correct the surname \"Teh\"\nstrat = \"strat\"\n" }, { "path": "build/cross-images/aarch64-unknown-linux-gnu.dockerfile", "content": "FROM ghcr.io/cross-rs/aarch64-unknown-linux-gnu:0.2.4@sha256:3356619b020614effd22e83cec41236e69f17ce581ffe35e252898b0c693b4e2\n\nARG PBC_URL=\"https://github.com/protocolbuffers/protobuf/releases/download/v21.5/protoc-21.5-linux-x86_64.zip\"\n\n#TODO: \n# We can switch to static linking (remove `libsasl2-dev:arm64`) using \n# `rdkafka/gssapi-vendored` feature when there is a release including: \n# https://github.com/MaterializeInc/rust-sasl/pull/48\n\nRUN dpkg --add-architecture arm64 && \\\n apt-get update && \\\n apt-get install -y clang-3.9 \\\n libclang-3.9-dev \\\n binutils-aarch64-linux-gnu \\\n libsasl2-dev:arm64 \\\n unzip && \\\n rm -rf /var/lib/apt/lists/*\n\nRUN curl -fLO $PBC_URL && \\\n unzip protoc-21.5-linux-x86_64.zip -d ./protobuf && \\\n mv ./protobuf/bin/protoc /usr/bin/ && \\\n rm -rf ./protobuf protoc-21.5-linux-x86_64.zip\n\nENV LIBZ_SYS_STATIC=1 \\\n PKG_CONFIG_ALLOW_CROSS=true \\\n PKG_CONFIG_ALL_STATIC=true \\\n X86_64_UNKNOWN_LINUX_MUSL_OPENSSL_STATIC=1 \\\n X86_64_UNKNOWN_LINUX_MUSL_OPENSSL_DIR=/usr/local/musl/\n" }, { "path": "build/cross-images/aarch64-unknown-linux-musl.dockerfile", "content": "FROM rustembedded/cross:aarch64-unknown-linux-musl@sha256:22627e0ba533781062127b13601c37216fdca27123390b07dfabd3f31f3c84a0\n\n\n# The Rust toolchain to use when building our image. Set by `hooks/build`.\n# ARG TOOLCHAIN=stable\n\n# The OpenSSL version to use. Here is the place to check for new releases:\n#\n# - https://www.openssl.org/source/\n#\n# ALSO UPDATE hooks/build!\nARG OPENSSL_VERSION=1.1.1i\nARG ZLIB_VERSION=1.2.11\n\nRUN echo \"Building OpenSSL\" && \\\n cd /tmp && \\\n short_version=\"$(echo \"$OPENSSL_VERSION\" | sed s'/[a-z]$//' )\" && \\\n curl -fLO \"https://www.openssl.org/source/openssl-$OPENSSL_VERSION.tar.gz\" || \\\n curl -fLO \"https://www.openssl.org/source/old/$short_version/openssl-$OPENSSL_VERSION.tar.gz\" && \\\n tar xvzf \"openssl-$OPENSSL_VERSION.tar.gz\" && cd \"openssl-$OPENSSL_VERSION\" && \\\n AR=aarch64-linux-musl-ar CC=aarch64-linux-musl-gcc ./Configure no-zlib -fPIC --prefix=/usr/local/aarch64-linux-musl -DOPENSSL_NO_SECURE_MEMORY linux-aarch64 && \\\n env C_INCLUDE_PATH=/usr/local/aarch64-linux-musl/include/ make depend && \\\n env C_INCLUDE_PATH=/usr/local/aarch64-linux-musl/include/ make && \\\n make install && \\\n rm -r /tmp/*\n\nRUN echo \"Building zlib\" && \\\n cd /tmp && \\\n curl -fLO \"https://zlib.net/fossils/zlib-$ZLIB_VERSION.tar.gz\" && \\\n tar xzf \"zlib-$ZLIB_VERSION.tar.gz\" && cd \"zlib-$ZLIB_VERSION\" && \\\n AR=aarch64-linux-musl-ar CC=aarch64-linux-musl-gcc ./configure --static --prefix=/usr/local/aarch64-linux-musl && \\\n make && make install && \\\n rm -r /tmp/*\n\nENV AARCH64_UNKNOWN_LINUX_MUSL_OPENSSL_STATIC=1 \\\n CC=aarch64-linux-musl-gcc \\\n CFLAGS=-I/usr/local/aarch64-linux-musl/include \\\n LIBZ_SYS_STATIC=1 \\\n LIB_LDFLAGS=-L/usr/local/aarch64-linux-musl/lib \\\n OPENSSL_INCLUDE_DIR=/usr/local/aarch64-linux-musl/include/openssl \\\n OPENSSL_LIB_DIR=/usr/local/aarch64-linux-musl/lib \\\n PKG_CONFIG_ALLOW_CROSS=true \\\n PKG_CONFIG_ALL_STATIC=true \\\n TARGET=aarch64-unknown-linux-musl \\\n AARCH64_UNKNOWN_LINUX_MUSL_OPENSSL_DIR=/usr/local/aarch64-linux-musl \\\n OPENSSL_ROOT_DIR=/usr/local/aarch64-linux-musl\n" }, { "path": "build/cross-images/x86_64-unknown-linux-gnu.dockerfile", "content": "FROM ghcr.io/cross-rs/x86_64-unknown-linux-gnu:0.2.4@sha256:7c9067212c2283be2a1d5585af5ecebd4c4a2e18091e2a6aafd23f9b4b81d496\n\nARG PBC_URL=\"https://github.com/protocolbuffers/protobuf/releases/download/v21.5/protoc-21.5-linux-x86_64.zip\"\n\nRUN apt-get update && \\\n apt-get install -y clang-3.9 \\\n libclang-3.9-dev \\\n libsasl2-dev \\\n unzip && \\\n rm -rf /var/lib/apt/lists/*\n\nRUN curl -fLO $PBC_URL && \\\n unzip protoc-21.5-linux-x86_64.zip -d ./protobuf && \\\n mv ./protobuf/bin/protoc /usr/bin/ && \\\n rm -rf ./protobuf protoc-21.5-linux-x86_64.zip\n" }, { "path": "build/cross-images/x86_64-unknown-linux-musl.dockerfile", "content": "FROM quickwit/cross-base:x86_64-unknown-linux-musl@sha256:5bcc7843aab64f89bf85c464fa2c5a00ecc634a8b1ac88c84a864f60054450cb\n# See https://github.com/quickwit-inc/rust-musl-builder\n\nRUN echo \"Upgrading CMake\" && \\\n sudo apt-get remove cmake -y && \\\n curl -fLO https://www.cmake.org/files/v3.12/cmake-3.12.1.tar.gz && \\\n tar -xvzf cmake-3.12.1.tar.gz && \\\n cd cmake-3.12.1/ && ./configure && \\\n sudo make install\n \nENV CC=musl-gcc \\\n CFLAGS=-I/usr/local/musl/include \\\n LIB_LDFLAGS=-L/usr/lib/x86_64-linux-gnu\n" }, { "path": "config/quickwit.yaml", "content": "# ============================ Node Configuration ==============================\n#\n# Website: https://quickwit.io\n# Docs: https://quickwit.io/docs/configuration/node-config\n#\n# Configure AWS credentials: https://quickwit.io/docs/guides/aws-setup#aws-credentials\n#\n# -------------------------------- General settings --------------------------------\n#\n# Config file format version.\n#\nversion: 0.8\n#\n# Node ID. Must be unique within a cluster. If not set, a random node ID is generated on each startup.\n#\n# node_id: node-1\n#\n# Quickwit opens three sockets.\n# - for its HTTP server, hosting the UI and the REST API (TCP)\n# - for its gRPC service (TCP)\n# - for its Gossip cluster membership service (UDP)\n#\n# All three services are bound to the same host and a different port. The host can be an IP address or a hostname.\n#\n# Default HTTP server host is `127.0.0.1` and default HTTP port is 7280.\n# The default host value was chosen to avoid exposing the node to the open-world without users' explicit consent.\n# This allows for testing Quickwit in single-node mode or with multiple nodes running on the same host and listening\n# on different ports. However, in cluster mode, using this value is never appropriate because it causes the node to\n# ignore incoming traffic.\n# There are two options to set up a node in cluster mode:\n# 1. specify the node's hostname or IP\n# 2. pass `0.0.0.0` and let Quickwit do its best to discover the node's IP (see `advertise_address`)\n#\n# listen_address: 127.0.0.1\n#\n# rest:\n# listen_port: 7280\n# cors_allow_origins:\n# - \"http://localhost:3000\"\n# extra_headers:\n# x-header-1: header-value-1\n# x-header-2: header-value-2\n#\n# grpc:\n# max_message_size: 10 MiB\n#\n# IP address advertised by the node, i.e. the IP address that peer nodes should use to connect to the node for RPCs.\n# The environment variable `QW_ADVERTISE_ADDRESS` can also be used to override this value.\n# The default advertise address is `listen_address`. If `listen_address` is unspecified (`0.0.0.0`),\n# Quickwit attempts to sniff the node's IP by scanning the available network interfaces.\n# advertise_address: 192.168.0.42\n#\n# In order to join a cluster, one needs to specify a list of\n# seeds to connect to. If no port is specified, Quickwit will assume\n# the seeds are using the same port as the current node gossip port.\n# By default, the peer seed list is empty.\n#\n# peer_seeds:\n# - quickwit-searcher-0.local\n# - quickwit-searcher-1.local:10000\n#\n# Path to directory where temporary data (caches, intermediate indexing data structures)\n# is stored. Defaults to `./qwdata`.\n#\n# data_dir: /path/to/data/dir\n#\n# Metastore URI. Defaults to `data_dir/indexes#polling_interval=30s`,\n# which is a file-backed metastore and mostly convenient for testing. A cluster would\n# require a metastore backed by Amzon S3 or PostgreSQL.\n#\n# metastore_uri: s3://your-bucket/indexes\n# metastore_uri: postgres://username:password@host:port/db\n#\n# When using a file-backed metastore, the state of the metastore will be cached forever.\n# If you are indexing and searching from different processes, it is possible to periodically\n# refresh the state of the metastore on the searcher using the `polling_interval` hashtag.\n#\n# metastore_uri: s3://your-bucket/indexes#polling_interval=30s\n#\n# Default index root URI, which defines where index data (splits) is stored,\n# following the scheme `{default_index_root_uri}/{index-id}`. Defaults to `{data_dir}/indexes`.\n#\n# default_index_root_uri: s3://your-bucket/indexes\n#\n# -------------------------------- Storage settings --------------------------------\n# https://quickwit.io/docs/configuration/node-config#storage-configuration\n#\n# Hardcoding credentials into configuration files is not secure and strongly\n# discouraged. Prefer the alternative authentication methods that your storage\n# backend may provide.\n#\n# storage:\n# azure:\n# account: ${QW_AZURE_STORAGE_ACCOUNT}\n# access_key: ${QW_AZURE_STORAGE_ACCESS_KEY}\n#\n# s3:\n# access_key_id: ${AWS_ACCESS_KEY_ID}\n# secret_access_key: ${AWS_SECRET_ACCESS_KEY}\n# region: ${AWS_REGION}\n# endpoint: ${QW_S3_ENDPOINT}\n# force_path_style_access: ${QW_S3_FORCE_PATH_STYLE_ACCESS:-false}\n# disable_multi_object_delete: false\n# disable_multipart_upload: false\n#\n# -------------------------------- Metastore settings --------------------------------\n# https://quickwit.io/docs/configuration/node-config#metastore-configuration\n#\n# metastore:\n# postgres:\n# min_connections: 0\n# max_connections: 10\n# acquire_connection_timeout: 10s\n# idle_connection_timeout: 10min\n# max_connection_lifetime: 30min\n#\n# -------------------------------- Indexer settings --------------------------------\n# https://quickwit.io/docs/configuration/node-config#indexer-configuration\n\nindexer:\n enable_otlp_endpoint: ${QW_ENABLE_OTLP_ENDPOINT:-true}\n# split_store_max_num_bytes: 100G\n# split_store_max_num_splits: 1000\n# max_concurrent_split_uploads: 12\n#\n#\n# -------------------------------- Ingest API settings ------------------------------\n# https://quickwit.io/docs/configuration/node-config#ingest-api-configuration\n#\n# ingest_api:\n# max_queue_memory_usage: 2GiB\n# max_queue_disk_usage: 4GiB\n# content_length_limit: 10MiB\n#\n# -------------------------------- Searcher settings --------------------------------\n# https://quickwit.io/docs/configuration/node-config#searcher-configuration\n#\n# searcher:\n# fast_field_cache_capacity: 1G\n# split_footer_cache_capacity: 500M\n# partial_request_cache_capacity: 64M\n# max_num_concurrent_split_streams: 100\n# max_num_concurrent_split_searches: 100\n# aggregation_memory_limit: 500M\n# aggregation_bucket_limit: 65000\n# split_cache:\n# max_num_bytes: 1G\n# max_num_splits: 10000\n# num_concurrent_downloads: 1\n# -------------------------------- Jaeger settings --------------------------------\n\njaeger:\n enable_endpoint: ${QW_ENABLE_JAEGER_ENDPOINT:-true}\n" }, { "path": "config/templates/gh-archive.yaml", "content": "version: 0.8\n\ntemplate_id: gh-archive\n\nindex_id_patterns:\n - gh-archive*\n\ndescription: Index config template for the GH Archive dataset (gharchive.org)\n\npriority: 0\n\ndoc_mapping:\n field_mappings:\n - name: id\n type: text\n tokenizer: raw\n - name: type\n type: text\n fast: true\n tokenizer: raw\n - name: public\n type: bool\n fast: true\n - name: payload\n type: json\n tokenizer: default\n - name: org\n type: json\n tokenizer: default\n - name: repo\n type: json\n tokenizer: default\n - name: actor\n type: json\n tokenizer: default\n - name: other\n type: json\n tokenizer: default\n - name: created_at\n type: datetime\n fast: true\n input_formats:\n - rfc3339\n fast_precision: seconds\n timestamp_field: created_at\n\nindexing_settings:\n commit_timeout_secs: 10\n" }, { "path": "config/templates/stackoverflow.yaml", "content": "version: 0.8\n\ntemplate_id: stackoverflow\n\nindex_id_patterns:\n - stackoverflow*\n\ndescription: Index config template for the Stackoverflow tutorial (quickwit.io/docs/get-started/quickstart)\n\npriority: 0\n\ndoc_mapping:\n field_mappings:\n - name: title\n type: text\n tokenizer: default\n record: position\n stored: true\n - name: body\n type: text\n tokenizer: default\n record: position\n stored: true\n - name: creationDate\n type: datetime\n fast: true\n input_formats:\n - rfc3339\n fast_precision: seconds\n timestamp_field: creationDate\n\nsearch_settings:\n default_search_fields: [title, body]\n\nindexing_settings:\n commit_timeout_secs: 10\n" }, { "path": "config/tutorials/fluentbit-logs/index-config.yaml", "content": "version: 0.8\n\nindex_id: fluentbit-logs\n\ndoc_mapping:\n mode: dynamic\n field_mappings:\n - name: timestamp\n type: datetime\n input_formats:\n - unix_timestamp\n output_format: unix_timestamp_secs\n fast: true\n timestamp_field: timestamp\n\nindexing_settings:\n commit_timeout_secs: 10\n" }, { "path": "config/tutorials/gh-archive/index-config-for-clickhouse.yaml", "content": "#\n# Index config file for gh-archive dataset.\n#\nversion: 0.8\n\nindex_id: gh-archive\n\ndoc_mapping:\n store_source: false\n field_mappings:\n - name: id\n type: u64\n fast: true\n - name: created_at\n type: datetime\n input_formats:\n - unix_timestamp\n output_format: unix_timestamp_secs\n fast_precision: seconds\n fast: true\n - name: event_type\n type: text\n tokenizer: raw\n - name: title\n type: text\n tokenizer: default\n record: position\n - name: body\n type: text\n tokenizer: default\n record: position\n timestamp_field: created_at\n\nsearch_settings:\n default_search_fields: [title, body]\n" }, { "path": "config/tutorials/gh-archive/index-config.yaml", "content": "#\n# Index config file for gh-archive dataset.\n#\nversion: 0.8\n\nindex_id: gh-archive\n\ndoc_mapping:\n field_mappings:\n - name: id\n type: text\n tokenizer: raw\n - name: type\n type: text\n fast: true\n tokenizer: raw\n - name: public\n type: bool\n fast: true\n - name: payload\n type: json\n tokenizer: default\n - name: org\n type: json\n tokenizer: default\n - name: repo\n type: json\n tokenizer: default\n - name: actor\n type: json\n tokenizer: default\n - name: other\n type: json\n tokenizer: default\n - name: created_at\n type: datetime\n fast: true\n input_formats:\n - rfc3339\n fast_precision: seconds\n timestamp_field: created_at\n\nindexing_settings:\n commit_timeout_secs: 10\n" }, { "path": "config/tutorials/gh-archive/kafka-source.yaml", "content": "version: 0.8\nsource_id: kafka-source\nsource_type: kafka\nnum_pipelines: 2\nparams:\n topic: gh-archive\n client_params:\n bootstrap.servers: localhost:9092\n" }, { "path": "config/tutorials/gh-archive/kinesis-source.yaml", "content": "version: 0.8\nsource_id: kinesis-source\nsource_type: kinesis\nparams:\n stream_name: gh-archive\n" }, { "path": "config/tutorials/grafana/docker-compose.yml", "content": "version: \"3.9\"\n\nnetworks:\n default:\n name: quickwit-grafana\n # ipam:\n # config:\n # - subnet: 172.16.7.0/24\n # gateway: 172.16.7.1\n\nservices:\n quickwit:\n image: quickwit/quickwit:${QUICKWIT_VERSION:-0.7.1}\n grafana:\n image: grafana/grafana-oss:${GRAFANA_VERSION:-9.4.7}\n container_name: grafana\n ports:\n - \"${MAP_HOST_GRAFANA:-127.0.0.1}:3000:3000\"\n environment:\n GF_AUTH_DISABLE_LOGIN_FORM: \"true\"\n GF_AUTH_ANONYMOUS_ENABLED: \"true\"\n GF_AUTH_ANONYMOUS_ORG_ROLE: Admin\n volumes:\n - ./monitoring/grafana/dashboards:/var/lib/grafana/dashboards\n - ./monitoring/grafana/provisioning:/etc/grafana/provisioning\n\n jaeger:\n image: jaegertracing/all-in-one:${JAEGER_VERSION:-1.48.0}\n container_name: jaeger\n ports:\n - \"${MAP_HOST_JAEGER:-127.0.0.1}:16686:16686\" # Frontend\n profiles:\n - jaeger\n - monitoring\n\n otel-collector:\n image: otel/opentelemetry-collector:${OTEL_VERSION:-0.84.0}\n container_name: otel-collector\n ports:\n - \"${MAP_HOST_OTEL:-127.0.0.1}:1888:1888\" # pprof extension\n - \"${MAP_HOST_OTEL:-127.0.0.1}:8888:8888\" # Prometheus metrics exposed by the collector\n - \"${MAP_HOST_OTEL:-127.0.0.1}:8889:8889\" # Prometheus exporter metrics\n - \"${MAP_HOST_OTEL:-127.0.0.1}:13133:13133\" # health_check extension\n - \"${MAP_HOST_OTEL:-127.0.0.1}:4317:4317\" # OTLP gRPC receiver\n - \"${MAP_HOST_OTEL:-127.0.0.1}:4318:4318\" # OTLP http receiver\n - \"${MAP_HOST_OTEL:-127.0.0.1}:55679:55679\" # zpages extension\n profiles:\n - otel\n - monitoring\n volumes:\n - ./monitoring/otel-collector-config.yaml:/etc/otel-collector-config.yaml\n command: [\"--config=/etc/otel-collector-config.yaml\"]\n\n prometheus:\n image: prom/prometheus:${PROMETHEUS_VERSION:-v2.43.0}\n container_name: prometheus\n ports:\n - \"${MAP_HOST_PROMETHEUS:-127.0.0.1}:9090:9090\"\n profiles:\n - prometheus\n - monitoring\n volumes:\n - ./monitoring/prometheus.yaml:/etc/prometheus/prometheus.yml\n extra_hosts:\n - \"host.docker.internal:host-gateway\"\n\n gcp-pubsub-emulator:\n # It is not an official docker image\n # if we prefer we can build a docker from the official docker image (gcloud cli)\n # and install the pubsub emulator https://cloud.google.com/pubsub/docs/emulator\n image: thekevjames/gcloud-pubsub-emulator:${GCLOUD_EMULATOR:-455.0.0}\n container_name: gcp-pubsub-emulator\n ports:\n - \"${MAP_HOST_GCLOUD_EMULATOR:-127.0.0.1}:8681:8681\"\n environment:\n # create a fake gcp project and a topic / subscription\n - PUBSUB_PROJECT1=quickwit-emulator,emulator_topic:emulator_subscription\n profiles:\n - all\n - gcp-pubsub\n\nvolumes:\n localstack_data:\n postgres_data:\n azurite_data:\n" }, { "path": "config/tutorials/hdfs-logs/index-config-partitioned.yaml", "content": "#\n# Index config file for hdfs-logs dataset with partitioning configured.\n#\n\nversion: 0.8\n\nindex_id: hdfs-logs-partitioned\n\ndoc_mapping:\n field_mappings:\n - name: timestamp\n type: datetime\n input_formats:\n - unix_timestamp\n output_format: unix_timestamp_secs\n fast_precision: seconds\n fast: true\n - name: tenant_id\n type: u64\n - name: severity_text\n type: text\n tokenizer: raw\n - name: body\n type: text\n tokenizer: default\n record: position\n - name: resource\n type: json\n tokenizer: raw\n tag_fields: [tenant_id]\n partition_key: tenant_id\n max_num_partitions: 1000\n timestamp_field: timestamp\n\nsearch_settings:\n default_search_fields: [severity_text, body]\n\nindexing_settings:\n commit_timeout_secs: 30\n split_num_docs_target: 10000000\n merge_policy:\n type: \"limit_merge\"\n merge_factor: 10\n max_merge_ops: 3\n maturation_period: 48 hours\n" }, { "path": "config/tutorials/hdfs-logs/index-config-retention-policy.yaml", "content": "#\n# Index config file for hdfs-logs dataset with a retention policy configured.\n#\n\nversion: 0.8\n\nindex_id: hdfs-logs-retention-policy\n\ndoc_mapping:\n field_mappings:\n - name: timestamp\n type: datetime\n input_formats:\n - unix_timestamp\n output_format: unix_timestamp_secs\n fast_precision: seconds\n fast: true\n - name: tenant_id\n type: u64\n - name: severity_text\n type: text\n tokenizer: raw\n - name: body\n type: text\n tokenizer: default\n record: position\n - name: resource\n type: json\n tokenizer: raw\n tag_fields: [tenant_id]\n timestamp_field: timestamp\n\nsearch_settings:\n default_search_fields: [severity_text, body]\n\nretention:\n period: 90 days\n schedule: daily\n\nindexing_settings:\n commit_timeout_secs: 10\n split_num_docs_target: 10000000\n" }, { "path": "config/tutorials/hdfs-logs/index-config.yaml", "content": "#\n# Index config file for hdfs-logs dataset.\n#\n\nversion: 0.8\n\nindex_id: hdfs-logs\n\ndoc_mapping:\n field_mappings:\n - name: timestamp\n type: datetime\n input_formats:\n - unix_timestamp\n output_format: unix_timestamp_secs\n fast_precision: seconds\n fast: true\n - name: tenant_id\n type: u64\n - name: severity_text\n type: text\n tokenizer: raw\n - name: body\n type: text\n tokenizer: default\n record: position\n - name: resource\n type: json\n tokenizer: raw\n tag_fields: [tenant_id]\n timestamp_field: timestamp\n\nsearch_settings:\n default_search_fields: [severity_text, body]\n" }, { "path": "config/tutorials/hdfs-logs/searcher-1.yaml", "content": "version: 0.8\nnode_id: searcher-1\nlisten_address: 127.0.0.1\nrest:\n listen_port: 7280\ningest_api:\n max_queue_memory_usage: 4GiB\n max_queue_disk_usage: 8GiB\npeer_seeds:\n - 127.0.0.1:7290 # searcher-2\n - 127.0.0.1:7300 # searcher-3\n" }, { "path": "config/tutorials/hdfs-logs/searcher-2.yaml", "content": "version: 0.8\nnode_id: searcher-2\nlisten_address: 127.0.0.1\nrest:\n listen_port: 7290\npeer_seeds:\n - 127.0.0.1:7280 # searcher-1\n - 127.0.0.1:7300 # searcher-3\n" }, { "path": "config/tutorials/hdfs-logs/searcher-3.yaml", "content": "version: 0.8\nnode_id: searcher-3\nlisten_address: 127.0.0.1\nrest:\n listen_port: 7300\npeer_seeds:\n - 127.0.0.1:7280 # searcher-1\n - 127.0.0.1:7290 # searcher-2\n\n" }, { "path": "config/tutorials/otel-logs/index-config.yaml", "content": "#\n# Index config file for receiving logs in OpenTelemetry format.\n# Link: https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/logs/data-model.md\n#\n\nversion: 0.8\n\nindex_id: otel-log-v0\n\ndoc_mapping:\n field_mappings:\n - name: timestamp\n type: datetime\n input_formats:\n - unix_timestamp\n output_format: unix_timestamp_secs\n fast: true\n - name: severity\n type: text\n tokenizer: raw\n fast: true\n - name: body\n type: text\n tokenizer: default\n record: position\n - name: attributes\n type: json\n - name: resource\n type: json\n timestamp_field: timestamp\n\nsearch_settings:\n default_search_fields: [severity, body]\n\n" }, { "path": "config/tutorials/otel-logs/kafka-source.yaml", "content": "version: 0.8\nsource_id: kafka-source\nsource_type: kafka\ninput_format: otlp_logs_proto\nparams:\n topic: otlp_logs\n client_params:\n bootstrap.servers: localhost:9092\n" }, { "path": "config/tutorials/otel-logs/otel-values.yaml", "content": "mode: \"daemonset\"\npresets:\n logsCollection:\n enabled: true\n kubernetesAttributes:\n enabled: true\nconfig:\n exporters:\n otlp:\n endpoint: quickwit-indexer.qw-tutorial.svc.cluster.local:7281\n tls:\n insecure: true\n service:\n pipelines:\n logs:\n exporters:\n - otlp\n" }, { "path": "config/tutorials/otel-traces/index-config.yaml", "content": "#\n# Index config file for receiving logs in OpenTelemetry format.\n# Link: https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/logs/data-model.md\n#\n\nversion: 0.8\n\nindex_id: otel-trace-v0\n\ndoc_mapping:\n mode: lenient\n field_mappings:\n - name: trace_id\n type: bytes\n - name: trace_state\n type: text\n indexed: false\n - name: resource_attributes\n type: json\n tokenizer: raw\n - name: resource_dropped_attributes_count\n type: u64\n indexed: false\n - name: service_name\n type: text\n tokenizer: raw\n - name: span_id\n type: bytes\n - name: span_kind\n type: u64\n - name: span_name\n type: text\n tokenizer: raw\n - name: span_start_timestamp_secs\n type: datetime\n indexed: true\n fast_precision: seconds\n fast: true\n input_formats: [unix_timestamp]\n output_format: unix_timestamp_secs\n - name: span_start_timestamp_nanos\n type: i64\n indexed: false\n - name: span_end_timestamp_nanos\n type: i64\n indexed: false\n - name: span_duration_secs\n type: i64\n indexed: false\n - name: span_attributes\n type: json\n tokenizer: raw\n - name: span_dropped_attributes_count\n type: u64\n indexed: false\n - name: span_dropped_events_count\n type: u64\n indexed: false\n - name: span_dropped_links_count\n type: u64\n indexed: false\n - name: span_status\n type: json\n indexed: false\n - name: parent_span_id\n type: bytes\n - name: events\n type: array\n tokenizer: raw\n - name: links\n type: array\n tokenizer: raw\n\n timestamp_field: span_start_timestamp_secs\n\n partition_key: service_name\n max_num_partitions: 100\n\nindexing_settings:\n commit_timeout_secs: 30\n\nsearch_settings:\n default_search_fields: []\n" }, { "path": "config/tutorials/otel-traces/kafka-source.yaml", "content": "version: 0.8\nsource_id: kafka-source\nsource_type: kafka\ninput_format: otlp_traces_proto\nparams:\n topic: otlp_spans\n client_params:\n bootstrap.servers: localhost:9092\n" }, { "path": "config/tutorials/stackoverflow/index-config.yaml", "content": "#\n# Index config file for stackoverflow dataset.\n#\nversion: 0.8\n\nindex_id: stackoverflow\n\ndoc_mapping:\n field_mappings:\n - name: title\n type: text\n tokenizer: default\n record: position\n stored: true\n - name: body\n type: text\n tokenizer: default\n record: position\n stored: true\n - name: creationDate\n type: datetime\n fast: true\n input_formats:\n - rfc3339\n fast_precision: seconds\n timestamp_field: creationDate\n\nsearch_settings:\n default_search_fields: [title, body]\n\nindexing_settings:\n commit_timeout_secs: 10\n" }, { "path": "config/tutorials/stackoverflow/pulsar-source.yaml", "content": "version: 0.8\nsource_id: pulsar-source\nsource_type: pulsar\nparams:\n topics:\n - quickwit/pulsar/stackoverflow\n address: pulsar://localhost:6650\n\n" }, { "path": "config/tutorials/stackoverflow/send_messages_to_pulsar.py", "content": "import json\nimport pulsar\n\nclient = pulsar.Client('pulsar://localhost:6650')\nproducer = client.create_producer('stackoverflow')\n\nwith open('stackoverflow.posts.transformed-10000.json', encoding='utf8') as file:\n for i, line in enumerate(file):\n producer.send(line.encode('utf-8'))\n if i % 100 == 0:\n print(f\"{i}/10000 messages sent.\", i)\n\nclient.close()\n" }, { "path": "config/tutorials/vector-otel-logs/vector.toml", "content": "[sources.generate_syslog]\ntype = \"demo_logs\"\nformat = \"syslog\"\ncount = 100000\ninterval = 0.001\n\n[transforms.remap_syslog]\ninputs = [ \"generate_syslog\"]\ntype = \"remap\"\nsource = '''\n structured = parse_syslog!(.message)\n .timestamp_nanos, err = to_unix_timestamp(structured.timestamp, unit: \"nanoseconds\")\n .body = structured\n .service_name = structured.appname\n .resource_attributes.source_type = .source_type\n .resource_attributes.host.hostname = structured.hostname\n .resource_attributes.service.name = structured.appname\n .attributes.syslog.procid = structured.procid\n .attributes.syslog.facility = structured.facility\n .attributes.syslog.version = structured.version\n .severity_text = if includes([\"emerg\", \"err\", \"crit\", \"alert\"], structured.severity) {\n \"ERROR\"\n } else if structured.severity == \"warning\" {\n \"WARN\"\n } else if structured.severity == \"debug\" {\n \"DEBUG\"\n } else if includes([\"info\", \"notice\"], structured.severity) {\n \"INFO\"\n } else {\n structured.severity\n }\n .scope_name = structured.msgid\n del(.message)\n del(.timestamp)\n del(.source_type)\n'''\n\n[sinks.emit_syslog]\ninputs = [\"remap_syslog\"]\ntype = \"console\"\nencoding.codec = \"json\"\n\n[sinks.quickwit_logs]\ntype = \"http\"\nmethod = \"post\"\ninputs = [\"remap_syslog\"]\nencoding.codec = \"json\"\nframing.method = \"newline_delimited\"\nuri = \"http://127.0.0.1:7280/api/v1/otel-logs-v0_7/ingest\"\n" }, { "path": "config/tutorials/wikipedia/index-config.yaml", "content": "#\n# Index config file for wikipedia dataset.\n#\n\nversion: 0.8\n\nindex_id: wikipedia\n\ndoc_mapping:\n field_mappings:\n - name: title\n type: text\n tokenizer: default\n record: position\n stored: true\n fieldnorms: true\n - name: body\n type: text\n tokenizer: default\n record: position\n stored: true\n fieldnorms: true\n - name: url\n type: text\n tokenizer: raw\n\nsearch_settings:\n default_search_fields: [title, body]\n\nindexing_settings:\n commit_timeout_secs: 10\n" }, { "path": "config/tutorials/wikipedia/multilang-index-config.yaml", "content": "#\n# Index config file for multilang wikipedia datasets.\n#\n\nversion: 0.8\n\nindex_id: multilang-wikipedia\n\ndoc_mapping:\n tokenizers:\n - name: multilang\n type: multilang\n field_mappings:\n - name: title\n type: text\n tokenizer: multilang\n record: position\n stored: true\n fieldnorms: true\n - name: body\n type: text\n tokenizer: multilang\n record: position\n stored: true\n fieldnorms: true\n - name: url\n type: text\n tokenizer: raw\n\nsearch_settings:\n default_search_fields: [title, body]\n\nindexing_settings:\n commit_timeout_secs: 10\n" }, { "path": "distribution/docker/ubuntu/Dockerfile", "content": "FROM ubuntu:noble@sha256:66460d557b25769b102175144d538d88219c077c678a49af4afca6fbfc1b5252 AS builder\n\nRUN apt-get update && apt-get install -y curl\nRUN curl -L https://install.quickwit.io | sh\n\n\nFROM ubuntu:noble@sha256:66460d557b25769b102175144d538d88219c077c678a49af4afca6fbfc1b5252 AS quickwit\n\nLABEL org.opencontainers.image.title=\"Quickwit\"\nLABEL maintainer=\"Quickwit, Inc. \"\nLABEL org.opencontainers.image.vendor=\"Quickwit, Inc.\"\nLABEL org.opencontainers.image.licenses=\"Apache-2.0\"\n\nRUN apt-get -y update \\\n && apt-get -y install ca-certificates \\\n libssl3 \\\n && rm -rf /var/lib/apt/lists/*\n\nWORKDIR /quickwit\nRUN mkdir config qwdata\nCOPY --from=builder /quickwit-v*/quickwit /usr/local/bin/quickwit\nCOPY --from=builder /quickwit-v*/config/quickwit.yaml /quickwit/config/quickwit.yaml\n\nENV QW_CONFIG=/quickwit/config/quickwit.yaml\nENV QW_DATA_DIR=/quickwit/qwdata\nENV QW_LISTEN_ADDRESS=0.0.0.0\n\nRUN quickwit --version\n\nENTRYPOINT [\"quickwit\"]\n" }, { "path": "distribution/ecs/.gitignore", "content": ".terraform\nterraform.tfstate*\n.terraform.tfstate*\nterraform.tfvars\n" }, { "path": "distribution/ecs/README.md", "content": "# ECS deployment for quickwit\n\n## Run Quickwit in your infrastructure\n\nCreate a Quickwit module using:\n\n```terraform\nmodule \"quickwit\" {\n source = \"github.com/quickwit-oss/quickwit/distribution/ecs/quickwit\"\n\n vpc_id = # VPC in which all resources will be created\n subnet_ids = [...] # At least 2 private subnets must be specified\n quickwit_ingress_cidr_blocks = [...] # List of CIDR blocks allowed to access to the Quickwit API\n}\n```\n\nThe Quickwit cluster is running on a private subnet. For ECS to pull the image:\n- if using the default Docker Hub image `quickwit/quickwit`, the subnets\nspecified must be configured with a NAT Gateway (no public IPs are attached to\nthe tasks)\n- if using an image hosted on ECR, a VPC endpoint for ECR can be used instead of\na NAT Gateway\n\n\n## Module configurations\n\nTo get the list of available configurations, check the `./quickwit/variables.tf`\nfile.\n\n### Tips\n\nMetastore database backups are disabled as restoring one would lead to\ninconsistencies with the index store on S3. To ensure high availability, you\nshould enable `rds_config.multi_az` instead. To use your own Postgres database\ninstead of creating a new RDS instance, configure the\n`external_postgres_uri_secret_arn` variable (e.g ARN of an SSM parameter with\nthe value `postgres://user:password@domain:port/db`).\n\nUsing NAT Gateways for the image registry is quite costly (approx. $0.05/hour/AZ). If\nyou are not already using NAT Gateways in the AZs where Quickwit will be\ndeployed, you should probably push the Quickwit image to ECR and use ECR\ninterface VPC endpoints instead (approx. ~$0.01/hour/AZ).\n\nWhen using the default image, you will quickly run into the Docker Hub rate\nlimiting. We recommend pushing the Quickwit image to ECR and configure that as\n`quickwit_image`. Note that the architecture of the image that you push to ECR\nmust match the `quickwit_cpu_architecture` variable (`ARM64` by default).\n\nSidecar container and custom logging configurations can be configured using the\nvariables `sidecar_container_definitions`, `sidecar_container_dependencies`,\n`log_configuration`, `enable_cloudwatch_logging`. See [custom log\nrouting](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/using_firelens.html).\n\nYou can use sidecars to inject additional secrets as files. This can be\nuseful for configuring sources such as Kafka. See `./example/kafka.tf` for an\nexample.\n\nTo access external AWS services like the Kinesis source, use the\n`quickwit_indexer.extra_task_policy_arns` variable to attach the necessary\nIAM policies to indexers.\n\n## Running the example stack\n\nWe provide an example of self contained deployment with an ad-hoc VPC. \n\n> [!IMPORTANT]\n> This stack costs ~$200/month to run (Fargate tasks, NAT Gateways\n> and RDS)\n\n### Deploy the Quickwit module and connect through a bastion\n\nTo make it easy to access your Quickwit cluster, the example stack includes\na bastion instance. Access is secured using an SSH key pair that you need to\nprovide (e.g generated with `ssh-keygen -t ed25519`).\n\nIn the `./example` directory, create a `terraform.tfvars` file with the public\nkey of your RSA key pair:\n\n```terraform\nbastion_public_key = \"ssh-ed25519 ...\"\n```\n\n> [!NOTE]\n> You can skip the creation of the bastion by not specifying the\n> `bastion_public_key` variable, but that would make it hard to access and\n> experiment with the created Quickwit cluster.\n\nIn the same directory (`./example`) run:\n\n```bash\nterraform init\nterraform apply\n```\n\nThe successful `apply` command should output the IP of the bastion EC2 instance.\nYou can port forward Quickwit's search UI using:\n\n```bash\nssh -N -L 7280:searcher.quickwit:7280 -i {your-private-key-file} ubuntu@{bastion_ip}\n```\n\nTo ingest some example dataset, log into the bastion:\n\n```bash\nssh -i {your-private-key-file} ubuntu@{bastion_ip}\n\n# create the log index\nwget https://raw.githubusercontent.com/quickwit-oss/quickwit/main/config/tutorials/hdfs-logs/index-config.yaml\ncurl -X POST \\\n -H \"content-type: application/yaml\" \\\n --data-binary @index-config.yaml \\\n http://indexer.quickwit:7280/api/v1/indexes\n\n# import some data\nwget https://quickwit-datasets-public.s3.amazonaws.com/hdfs-logs-multitenants-10000.json\ncurl -X POST \\\n -H \"content-type: application/json\" \\\n --data-binary @hdfs-logs-multitenants-10000.json \\\n http://indexer.quickwit:7280/api/v1/hdfs-logs/ingest?commit=force\n```\n\nIf your SSH tunnel to the searcher is still running, you should be able to see\nthe ingested data in the UI.\n\n### Setup an ECR repository to avoid throttling from Docker Hub\n\nBy default, the example stack uses Docker Hub to pull the Quickwit image. This\nis convenient but it quickly runs into rate limiting. To avoid this, in the\n`terraform.tfvars` file, set the `dockerhub_pull_through_creds_secret_arn` to a\nAWS Secret with the following content:\n\n```json\n{\"username\":\"...\",\"accessToken\":\"...\"}\n```\n\nThis will:\n- provision an ECR repository and a pull through cache rule\n- configure the Quickwit module to use that repository\n" }, { "path": "distribution/ecs/example/.terraform.lock.hcl", "content": "# This file is maintained automatically by \"terraform init\".\n# Manual edits may be lost in future updates.\n\nprovider \"registry.terraform.io/hashicorp/aws\" {\n version = \"5.39.1\"\n constraints = \">= 4.66.1, >= 5.36.0, ~> 5.39.1\"\n hashes = [\n \"h1:hQLlAd6O1LdQHy1GdWtgT5fcOlc3TWW+SaaFkpe+e8E=\",\n \"zh:05c50a5d8edb3ba4ebc4eb6e0d0b5e319142f5983b27821710ed7d475d335bdc\",\n \"zh:082986a5784dd21957e632371b289e549f051a4ea21d5c78c6d744c3537f03c5\",\n \"zh:192ae622ba562eacc4921ed549a794506179233d724fdd15a4f147f3400724a0\",\n \"zh:19a1d4637a62de90b0da174c0bf01000cd900488f7e8f709d8a37f082c59756b\",\n \"zh:1d7689a8583515f1705972d7ce57ccfab96215b19905530d2c78c02dcfaff583\",\n \"zh:22c446a21209a52ab74b4ba1ede0b220531e97ce479430047e493a2c45e1d8cb\",\n \"zh:4154de82290ab4e9f81bac1ea62342de8b3b7a608f99258c190d4dd1c6663e47\",\n \"zh:6bc4859ccdc54f28af9286b2fa090a31dcb345138d68c471510b737f6a052011\",\n \"zh:73c69e000e0b321e78a4a12fef60d37285f2afec0ea7be9e06163d985101cb59\",\n \"zh:890a3422f5e445b49bae30facf448d0ec9cd647e9155d0b685b5b39e9d331a94\",\n \"zh:9b12af85486a96aedd8d7984b0ff811a4b42e3d88dad1a3fb4c0b580d04fa425\",\n \"zh:9cd88bec0f5205df9032e3126d4e57edd1c5cc8d45cda25626882dafc485a3b0\",\n \"zh:a3a8e3276d0fbf051bbafa192a2998b05745f2cf285ac8c36a9ad167a75c037f\",\n \"zh:d47e4dcf4c0ad71b9a7c720be4f3a89f6786a82e77bbe8d950794562792a1da5\",\n \"zh:f74e5b2af508c7de80a6ae5198df54a795eeba5058a0cd247828943f0c54f6e0\",\n ]\n}\n\nprovider \"registry.terraform.io/hashicorp/random\" {\n version = \"3.6.0\"\n constraints = \">= 3.1.0\"\n hashes = [\n \"h1:R5Ucn26riKIEijcsiOMBR3uOAjuOMfI1x7XvH4P6B1w=\",\n \"zh:03360ed3ecd31e8c5dac9c95fe0858be50f3e9a0d0c654b5e504109c2159287d\",\n \"zh:1c67ac51254ba2a2bb53a25e8ae7e4d076103483f55f39b426ec55e47d1fe211\",\n \"zh:24a17bba7f6d679538ff51b3a2f378cedadede97af8a1db7dad4fd8d6d50f829\",\n \"zh:30ffb297ffd1633175d6545d37c2217e2cef9545a6e03946e514c59c0859b77d\",\n \"zh:454ce4b3dbc73e6775f2f6605d45cee6e16c3872a2e66a2c97993d6e5cbd7055\",\n \"zh:78d5eefdd9e494defcb3c68d282b8f96630502cac21d1ea161f53cfe9bb483b3\",\n \"zh:91df0a9fab329aff2ff4cf26797592eb7a3a90b4a0c04d64ce186654e0cc6e17\",\n \"zh:aa57384b85622a9f7bfb5d4512ca88e61f22a9cea9f30febaa4c98c68ff0dc21\",\n \"zh:c4a3e329ba786ffb6f2b694e1fd41d413a7010f3a53c20b432325a94fa71e839\",\n \"zh:e2699bc9116447f96c53d55f2a00570f982e6f9935038c3810603572693712d0\",\n \"zh:e747c0fd5d7684e5bfad8aa0ca441903f15ae7a98a737ff6aca24ba223207e2c\",\n \"zh:f1ca75f417ce490368f047b63ec09fd003711ae48487fba90b4aba2ccf71920e\",\n ]\n}\n" }, { "path": "distribution/ecs/example/bastion.tf", "content": "variable \"bastion_public_key\" {\n description = \"The public key used to connect to the bastion host. If empty, no bastion is created.\"\n default = \"\"\n}\n\noutput \"bastion_ip\" {\n value = var.bastion_public_key != \"\" ? aws_instance.bastion[0].public_ip : null\n}\n\ndata \"aws_ami\" \"ubuntu\" {\n most_recent = true\n\n filter {\n name = \"name\"\n values = [\"ubuntu/images/hvm-ssd/ubuntu-jammy-22.04-amd64-server-*\"]\n }\n\n filter {\n name = \"virtualization-type\"\n values = [\"hvm\"]\n }\n\n owners = [\"099720109477\"] # Canonical\n}\n\nresource \"aws_security_group\" \"allow_ssh\" {\n count = var.bastion_public_key != \"\" ? 1 : 0\n name = \"qw_ecs_bastion_allow_ssh\"\n description = \"Allow SSH inbound traffic from everywhere\"\n vpc_id = module.vpc.vpc_id\n\n ingress {\n from_port = 22\n to_port = 22\n protocol = \"tcp\"\n cidr_blocks = [\"0.0.0.0/0\"]\n }\n\n egress {\n from_port = 0\n to_port = 0\n protocol = \"-1\"\n cidr_blocks = [\"0.0.0.0/0\"]\n }\n}\n\nresource \"aws_instance\" \"bastion\" {\n count = var.bastion_public_key != \"\" ? 1 : 0\n ami = data.aws_ami.ubuntu.id\n instance_type = \"t3.nano\"\n key_name = aws_key_pair.bastion_key[0].key_name\n subnet_id = module.vpc.public_subnets[0]\n associate_public_ip_address = true\n vpc_security_group_ids = [aws_security_group.allow_ssh[0].id]\n\n tags = {\n Name = \"quickwit-ecs-bastion\"\n }\n}\n\nresource \"aws_key_pair\" \"bastion_key\" {\n count = var.bastion_public_key != \"\" ? 1 : 0\n key_name = \"quickwit-ecs-bastion-key\"\n public_key = var.bastion_public_key\n}\n" }, { "path": "distribution/ecs/example/image.tf", "content": "variable \"dockerhub_pull_through_creds_secret_arn\" {\n description = \"If left empty, image is pulled directly from Docker Hub, which might be throttled.\"\n default = \"\"\n}\n\nlocals {\n ecr_repository_prefix = \"quickwit-ecs-example\"\n}\n\n# This repo is populated by the pull through cache below\nresource \"aws_ecr_repository\" \"quickwit\" {\n count = var.dockerhub_pull_through_creds_secret_arn == \"\" ? 0 : 1\n name = \"${local.ecr_repository_prefix}/quickwit/quickwit\"\n image_tag_mutability = \"MUTABLE\"\n force_delete = true\n image_scanning_configuration {\n scan_on_push = false\n }\n}\n\nresource \"aws_ecr_pull_through_cache_rule\" \"docker_hub\" {\n count = var.dockerhub_pull_through_creds_secret_arn == \"\" ? 0 : 1\n ecr_repository_prefix = local.ecr_repository_prefix\n upstream_registry_url = \"registry-1.docker.io\"\n credential_arn = var.dockerhub_pull_through_creds_secret_arn\n}\n\n\nlocals {\n ecr_domain = \"${data.aws_caller_identity.current.account_id}.dkr.ecr.${data.aws_region.current.name}.amazonaws.com\"\n image_prefix = var.dockerhub_pull_through_creds_secret_arn == \"\" ? \"\" : \"${local.ecr_domain}/${local.ecr_repository_prefix}/\"\n quickwit_image = \"${local.image_prefix}quickwit/quickwit\"\n}\n" }, { "path": "distribution/ecs/example/kafka.tf", "content": "# Example configuration for injecting SSL keys for securing a Kafka connection\n# You can then create a secured Kafka source along these lines:\n# \n# version: 0.8\n# source_id: kafka-source\n# source_type: kafka\n# num_pipelines: 2\n# params:\n# topic: your-topic\n# client_params:\n# bootstrap.servers: \"your-kafka-broker.com\"\n# security.protocol: \"SSL\"\n# ssl.ca.location: \"/quickwit/keys/ca.pem\"\n# ssl.certificate.location: \"/quickwit/keys/service.cert\"\n# ssl.key.location: \"/quickwit/keys/service.key\"\n\n\nlocals {\n ca_pem = \"echo \\\"$CA_PEM\\\" > /quickwit/cfg/ca.pem\"\n service_cert = \"echo \\\"$SERVICE_CERT\\\" > /quickwit/cfg/service.cert\"\n service_key = \"echo \\\"$SERVICE_KEY\\\" > /quickwit/cfg/service.key\"\n example_kafka_sidecar_container_definitions = {\n kafka_key_init = {\n name = \"kafka_key_init\"\n essential = false\n image = \"busybox\"\n command = [\"sh\", \"-c\", \"${local.ca_pem} && ${local.service_cert} && ${local.service_key}\"]\n enable_cloudwatch_logging = true\n mount_points = [\n {\n sourceVolume = \"quickwit-keys\"\n containerPath = \"/quickwit/keys\"\n }\n ]\n secrets = [\n {\n name = \"CA_PEM\"\n valueFrom = \"arn:aws:secretsmanager:eu-west-1:123456789:secret:your_kafka_ca_pem\"\n },\n {\n name = \"SERVICE_CERT\"\n valueFrom = \"arn:aws:secretsmanager:eu-west-1:123456789:secret:your_kafka_service_cert\"\n },\n {\n name = \"SERVICE_KEY\"\n valueFrom = \"arn:aws:secretsmanager:eu-west-1:123456789:secret:your_kafka_service_key\"\n }\n ]\n }\n }\n\n example_kafka_sidecar_container_dependencies = [\n {\n condition = \"SUCCESS\"\n containerName = \"kafka_key_init\"\n }\n ]\n}\n" }, { "path": "distribution/ecs/example/terraform.tf", "content": "terraform {\n backend \"local\" {}\n required_providers {\n aws = {\n source = \"hashicorp/aws\"\n version = \"~> 5.39.1\"\n }\n }\n}\n\nprovider \"aws\" {\n region = \"eu-west-1\"\n default_tags {\n tags = {\n provisioner = \"terraform\"\n }\n }\n}\n\ndata \"aws_region\" \"current\" {}\n\ndata \"aws_caller_identity\" \"current\" {}\n\nmodule \"quickwit\" {\n source = \"../quickwit\"\n vpc_id = module.vpc.vpc_id\n subnet_ids = module.vpc.private_subnets\n quickwit_ingress_cidr_blocks = [module.vpc.vpc_cidr_block]\n\n ## Optional configurations:\n\n # - ECR if you provide the `dockerhub_pull_through_creds_secret_arn` variable\n # - Docker Hub otherwise (subject to throttling)\n quickwit_image = \"${local.quickwit_image}:latest\"\n\n # quickwit_index_s3_prefix = \"my-bucket/my-prefix\"\n # quickwit_domain = \"quickwit\"\n # quickwit_cpu_architecture = \"ARM64\"\n\n # quickwit_indexer = {\n # desired_count = 3\n # memory = 8192\n # cpu = 4096\n # ephemeral_storage_gib = 50\n # extra_task_policy_arns = [\"arn:aws:iam::aws:policy/AmazonKinesisFullAccess\"]\n # }\n\n # quickwit_metastore = {\n # desired_count = 1\n # memory = 512\n # cpu = 256\n # }\n\n # quickwit_searcher = {\n # desired_count = 1\n # memory = 2048\n # cpu = 1024\n # }\n\n # quickwit_control_plane = {\n # memory = 512\n # cpu = 256\n # }\n\n # quickwit_janitor = {\n # memory = 512\n # cpu = 256\n # }\n\n # rds_config = {\n # instance_class = \"db.t4g.micro\"\n # multi_az = false\n # }\n\n # external_postgres_uri_secret_arn = aws_ssm_parameter.postgres_uri.arn\n\n ## Example logging configuration \n # sidecar_container_definitions = {\n # my_sidecar_container = see http://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_ContainerDefinition.html\n # }\n # sidecar_container_dependencies = [{condition = \"START\", containerName = \"my_sidecar_container\"}]\n # log_configuration = see https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/ecs_service#log_configuration\n # enable_cloudwatch_logging = false\n\n ## Example Kafka key injection (see kafka.tf)\n # sidecar_container_definitions = local.example_kafka_sidecar_container_definitions\n # sidecar_container_dependencies = local.example_kafka_sidecar_container_dependencies\n}\n\n\noutput \"indexer_service_name\" {\n value = module.quickwit.indexer_service_name\n}\n\noutput \"searcher_service_name\" {\n value = module.quickwit.searcher_service_name\n}\n" }, { "path": "distribution/ecs/example/vpc.tf", "content": "module \"vpc\" {\n source = \"terraform-aws-modules/vpc/aws\"\n version = \"5.5.3\"\n\n name = \"quickwit-ecs\"\n cidr = \"10.0.0.0/16\"\n\n azs = [\"${data.aws_region.current.name}a\", \"${data.aws_region.current.name}b\"]\n private_subnets = [\"10.0.1.0/24\", \"10.0.2.0/24\"]\n public_subnets = [\"10.0.101.0/24\", \"10.0.102.0/24\"]\n\n enable_nat_gateway = true\n}\n" }, { "path": "distribution/ecs/quickwit/cluster.tf", "content": "module \"ecs_cluster\" {\n source = \"terraform-aws-modules/ecs/aws//modules/cluster\"\n version = \"5.9.3\"\n\n cluster_name = \"quickwit-${local.module_id}\"\n}\n\nresource \"aws_service_discovery_private_dns_namespace\" \"quickwit_internal\" {\n name = var.quickwit_domain\n description = \"Internal quickwit domain\"\n vpc = var.vpc_id\n}\n\nresource \"aws_security_group\" \"quickwit_cluster_member_sg\" {\n name = \"quickwit-cluster-member-${local.module_id}\"\n description = \"Security group for members of the Quickwit cluster\"\n vpc_id = var.vpc_id\n}\n" }, { "path": "distribution/ecs/quickwit/configs.tf", "content": "locals {\n quickwit_peer_list = [\n \"${aws_service_discovery_service.metastore.name}.${aws_service_discovery_private_dns_namespace.quickwit_internal.name}\",\n \"${aws_service_discovery_service.control_plane.name}.${aws_service_discovery_private_dns_namespace.quickwit_internal.name}\",\n \"${aws_service_discovery_service.janitor.name}.${aws_service_discovery_private_dns_namespace.quickwit_internal.name}\",\n \"${aws_service_discovery_service.indexer.name}.${aws_service_discovery_private_dns_namespace.quickwit_internal.name}\",\n \"${aws_service_discovery_service.searcher.name}.${aws_service_discovery_private_dns_namespace.quickwit_internal.name}\",\n ]\n\n # id to avoid conflicts when deploying this module multiple times (random by default)\n module_id = var.module_id == \"\" ? random_id.module.hex : var.module_id\n s3_id = var.module_id == \"\" ? random_id.module.hex : \"${var.module_id}-${random_id.module.hex}\"\n\n quickwit_index_s3_prefix = var.quickwit_index_s3_prefix == \"\" ? aws_s3_bucket.index[0].id : var.quickwit_index_s3_prefix\n\n use_external_rds = var.external_postgres_uri_secret_arn != \"\"\n postgres_uri_secret_arn = var.external_postgres_uri_secret_arn != \"\" ? var.external_postgres_uri_secret_arn : aws_ssm_parameter.postgres_credential[0].arn\n}\n\nresource \"random_id\" \"module\" {\n byte_length = 3\n}\n" }, { "path": "distribution/ecs/quickwit/iam.tf", "content": "data \"aws_iam_policy_document\" \"quickwit_task_permission\" {\n # Reference: https://quickwit.io/docs/guides/aws-setup#amazon-s3\n statement {\n actions = [\n \"s3:ListBucket\",\n \"s3:GetObject\",\n \"s3:PutObject\",\n \"s3:DeleteObject\"\n ]\n\n resources = [\n \"arn:aws:s3:::${local.quickwit_index_s3_prefix}*\",\n ]\n }\n}\n\nresource \"aws_iam_policy\" \"quickwit_task_permission\" {\n name = \"quickwit-task-policy-${local.module_id}\"\n path = \"/\"\n\n policy = data.aws_iam_policy_document.quickwit_task_permission.json\n}\n\ndata \"aws_iam_policy_document\" \"quickwit_task_execution_permission\" {\n statement {\n actions = [\n \"logs:PutLogEvents\",\n \"logs:CreateLogStream\"\n ]\n\n resources = [\"*\"]\n }\n statement {\n actions = [\n \"ecr:GetDownloadUrlForLayer\",\n \"ecr:GetAuthorizationToken\",\n \"ecr:BatchGetImage\",\n \"ecr:BatchCheckLayerAvailability\",\n \"ecr:CreateRepository\",\n \"ecr:BatchImportUpstreamImage\"\n ]\n\n resources = [\"*\"]\n }\n\n statement {\n actions = [\"ssm:GetParameters\"]\n\n resources = [local.postgres_uri_secret_arn]\n }\n\n statement {\n actions = [\"secretsmanager:GetSecretValue\"]\n\n resources = [\"arn:aws:secretsmanager:*:*:secret:*\"]\n }\n\n}\n\nresource \"aws_iam_policy\" \"quickwit_task_execution_permission\" {\n name = \"quickwit-task-execution-policy-${local.module_id}\"\n path = \"/\"\n\n policy = data.aws_iam_policy_document.quickwit_task_execution_permission.json\n}\n" }, { "path": "distribution/ecs/quickwit/outputs.tf", "content": "output \"indexer_service_name\" {\n value = \"${aws_service_discovery_service.indexer.name}.${aws_service_discovery_private_dns_namespace.quickwit_internal.name}\"\n}\n\noutput \"searcher_service_name\" {\n value = \"${aws_service_discovery_service.searcher.name}.${aws_service_discovery_private_dns_namespace.quickwit_internal.name}\"\n}\n\noutput \"janitor_service_name\" {\n value = \"${aws_service_discovery_service.janitor.name}.${aws_service_discovery_private_dns_namespace.quickwit_internal.name}\"\n}\n\noutput \"control_plane_service_name\" {\n value = \"${aws_service_discovery_service.control_plane.name}.${aws_service_discovery_private_dns_namespace.quickwit_internal.name}\"\n}\n\noutput \"metastore_service_name\" {\n value = \"${aws_service_discovery_service.metastore.name}.${aws_service_discovery_private_dns_namespace.quickwit_internal.name}\"\n}\n" }, { "path": "distribution/ecs/quickwit/quickwit-control-plane.tf", "content": "module \"quickwit_control_plane\" {\n source = \"./service\"\n service_name = \"control_plane\"\n service_discovery_registry_arn = aws_service_discovery_service.control_plane.arn\n cluster_arn = module.ecs_cluster.arn\n postgres_uri_secret_arn = local.postgres_uri_secret_arn\n quickwit_peer_list = local.quickwit_peer_list\n s3_access_policy_arn = aws_iam_policy.quickwit_task_permission.arn\n task_execution_policy_arn = aws_iam_policy.quickwit_task_execution_permission.arn\n module_id = local.module_id\n quickwit_cluster_member_sg_id = aws_security_group.quickwit_cluster_member_sg.id\n\n subnet_ids = var.subnet_ids\n ingress_cidr_blocks = var.quickwit_ingress_cidr_blocks\n quickwit_image = var.quickwit_image\n quickwit_cpu_architecture = var.quickwit_cpu_architecture\n sidecar_container_definitions = var.sidecar_container_definitions\n sidecar_container_dependencies = var.sidecar_container_dependencies\n log_configuration = var.log_configuration\n enable_cloudwatch_logging = var.enable_cloudwatch_logging\n service_config = var.quickwit_control_plane\n quickwit_index_s3_prefix = local.quickwit_index_s3_prefix\n}\n\nresource \"aws_service_discovery_service\" \"control_plane\" {\n name = \"control-plane\"\n\n dns_config {\n namespace_id = aws_service_discovery_private_dns_namespace.quickwit_internal.id\n\n dns_records {\n ttl = 10\n type = \"A\"\n }\n\n routing_policy = \"MULTIVALUE\"\n }\n}\n" }, { "path": "distribution/ecs/quickwit/quickwit-indexer.tf", "content": "module \"quickwit_indexer\" {\n source = \"./service\"\n service_name = \"indexer\"\n service_discovery_registry_arn = aws_service_discovery_service.indexer.arn\n cluster_arn = module.ecs_cluster.arn\n postgres_uri_secret_arn = local.postgres_uri_secret_arn\n quickwit_peer_list = local.quickwit_peer_list\n s3_access_policy_arn = aws_iam_policy.quickwit_task_permission.arn\n task_execution_policy_arn = aws_iam_policy.quickwit_task_execution_permission.arn\n module_id = local.module_id\n quickwit_cluster_member_sg_id = aws_security_group.quickwit_cluster_member_sg.id\n\n subnet_ids = var.subnet_ids\n ingress_cidr_blocks = var.quickwit_ingress_cidr_blocks\n quickwit_image = var.quickwit_image\n quickwit_cpu_architecture = var.quickwit_cpu_architecture\n sidecar_container_definitions = var.sidecar_container_definitions\n sidecar_container_dependencies = var.sidecar_container_dependencies\n log_configuration = var.log_configuration\n enable_cloudwatch_logging = var.enable_cloudwatch_logging\n service_config = var.quickwit_indexer\n quickwit_index_s3_prefix = local.quickwit_index_s3_prefix\n # Longer termination grace period for indexers because we are waiting for the\n # data persisted in the ingesters to be indexed and committed. Should be\n # larger than the largest commit timeout.\n stop_timeout = 120\n}\n\nresource \"aws_service_discovery_service\" \"indexer\" {\n name = \"indexer\"\n\n dns_config {\n namespace_id = aws_service_discovery_private_dns_namespace.quickwit_internal.id\n\n dns_records {\n ttl = 10\n type = \"A\"\n }\n\n routing_policy = \"MULTIVALUE\"\n }\n}\n" }, { "path": "distribution/ecs/quickwit/quickwit-janitor.tf", "content": "module \"quickwit_janitor\" {\n source = \"./service\"\n service_name = \"janitor\"\n service_discovery_registry_arn = aws_service_discovery_service.janitor.arn\n cluster_arn = module.ecs_cluster.arn\n postgres_uri_secret_arn = local.postgres_uri_secret_arn\n quickwit_peer_list = local.quickwit_peer_list\n s3_access_policy_arn = aws_iam_policy.quickwit_task_permission.arn\n task_execution_policy_arn = aws_iam_policy.quickwit_task_execution_permission.arn\n module_id = local.module_id\n quickwit_cluster_member_sg_id = aws_security_group.quickwit_cluster_member_sg.id\n\n subnet_ids = var.subnet_ids\n ingress_cidr_blocks = var.quickwit_ingress_cidr_blocks\n quickwit_image = var.quickwit_image\n quickwit_cpu_architecture = var.quickwit_cpu_architecture\n sidecar_container_definitions = var.sidecar_container_definitions\n sidecar_container_dependencies = var.sidecar_container_dependencies\n log_configuration = var.log_configuration\n enable_cloudwatch_logging = var.enable_cloudwatch_logging\n service_config = var.quickwit_janitor\n quickwit_index_s3_prefix = local.quickwit_index_s3_prefix\n}\n\nresource \"aws_service_discovery_service\" \"janitor\" {\n name = \"janitor\"\n\n dns_config {\n namespace_id = aws_service_discovery_private_dns_namespace.quickwit_internal.id\n\n dns_records {\n ttl = 10\n type = \"A\"\n }\n\n routing_policy = \"MULTIVALUE\"\n }\n}\n" }, { "path": "distribution/ecs/quickwit/quickwit-metastore.tf", "content": "module \"quickwit_metastore\" {\n source = \"./service\"\n service_name = \"metastore\"\n service_discovery_registry_arn = aws_service_discovery_service.metastore.arn\n cluster_arn = module.ecs_cluster.arn\n postgres_uri_secret_arn = local.postgres_uri_secret_arn\n quickwit_peer_list = local.quickwit_peer_list\n s3_access_policy_arn = aws_iam_policy.quickwit_task_permission.arn\n task_execution_policy_arn = aws_iam_policy.quickwit_task_execution_permission.arn\n module_id = local.module_id\n quickwit_cluster_member_sg_id = aws_security_group.quickwit_cluster_member_sg.id\n\n subnet_ids = var.subnet_ids\n ingress_cidr_blocks = var.quickwit_ingress_cidr_blocks\n quickwit_image = var.quickwit_image\n quickwit_cpu_architecture = var.quickwit_cpu_architecture\n sidecar_container_definitions = var.sidecar_container_definitions\n sidecar_container_dependencies = var.sidecar_container_dependencies\n log_configuration = var.log_configuration\n enable_cloudwatch_logging = var.enable_cloudwatch_logging\n service_config = var.quickwit_metastore\n quickwit_index_s3_prefix = local.quickwit_index_s3_prefix\n}\n\nresource \"aws_service_discovery_service\" \"metastore\" {\n name = \"metastore\"\n\n dns_config {\n namespace_id = aws_service_discovery_private_dns_namespace.quickwit_internal.id\n\n dns_records {\n ttl = 10\n type = \"A\"\n }\n\n routing_policy = \"MULTIVALUE\"\n }\n}\n" }, { "path": "distribution/ecs/quickwit/quickwit-searcher.tf", "content": "module \"quickwit_searcher\" {\n source = \"./service\"\n service_name = \"searcher\"\n service_discovery_registry_arn = aws_service_discovery_service.searcher.arn\n cluster_arn = module.ecs_cluster.arn\n postgres_uri_secret_arn = local.postgres_uri_secret_arn\n quickwit_peer_list = local.quickwit_peer_list\n s3_access_policy_arn = aws_iam_policy.quickwit_task_permission.arn\n task_execution_policy_arn = aws_iam_policy.quickwit_task_execution_permission.arn\n module_id = local.module_id\n quickwit_cluster_member_sg_id = aws_security_group.quickwit_cluster_member_sg.id\n\n subnet_ids = var.subnet_ids\n ingress_cidr_blocks = var.quickwit_ingress_cidr_blocks\n quickwit_image = var.quickwit_image\n quickwit_cpu_architecture = var.quickwit_cpu_architecture\n sidecar_container_definitions = var.sidecar_container_definitions\n sidecar_container_dependencies = var.sidecar_container_dependencies\n log_configuration = var.log_configuration\n enable_cloudwatch_logging = var.enable_cloudwatch_logging\n service_config = var.quickwit_searcher\n quickwit_index_s3_prefix = local.quickwit_index_s3_prefix\n}\n\nresource \"aws_service_discovery_service\" \"searcher\" {\n name = \"searcher\"\n\n dns_config {\n namespace_id = aws_service_discovery_private_dns_namespace.quickwit_internal.id\n\n dns_records {\n ttl = 10\n type = \"A\"\n }\n\n routing_policy = \"MULTIVALUE\"\n }\n}\n" }, { "path": "distribution/ecs/quickwit/rds.tf", "content": "resource \"random_password\" \"quickwit_db\" {\n count = local.use_external_rds ? 0 : 1\n length = 64\n special = false\n}\n\nmodule \"quickwit_db\" {\n count = local.use_external_rds ? 0 : 1\n source = \"terraform-aws-modules/rds/aws\"\n version = \"6.5.2\"\n\n identifier = \"quickwit-metastore-${local.module_id}\"\n\n engine = \"postgres\"\n engine_version = \"16\"\n family = \"postgres16\" # DB parameter group\n major_engine_version = \"16\" # DB option group\n\n instance_class = var.rds_config.instance_class\n multi_az = var.rds_config.multi_az\n allocated_storage = 5\n\n db_name = \"quickwit\"\n username = \"quickwit\"\n password = random_password.quickwit_db[0].result\n\n port = \"5432\"\n publicly_accessible = false\n manage_master_user_password = false\n iam_database_authentication_enabled = true\n vpc_security_group_ids = [aws_security_group.quickwit_db[0].id]\n db_subnet_group_name = aws_db_subnet_group.quickwit[0].name\n\n maintenance_window = \"Mon:00:00-Mon:03:00\"\n\n create_monitoring_role = true\n monitoring_interval = \"30\"\n monitoring_role_name = \"RDSQuickwitMonitoringRole-${local.module_id}\"\n\n deletion_protection = false\n skip_final_snapshot = true\n}\n\nresource \"aws_security_group\" \"quickwit_db\" {\n count = local.use_external_rds ? 0 : 1\n name = \"quickwit-db-${local.module_id}\"\n description = \"Security group for the Quickwit Metastore DB\"\n vpc_id = var.vpc_id\n\n ingress {\n description = \"Connection from explicitly allowed resources\"\n from_port = 5432\n to_port = 5432\n protocol = \"tcp\"\n security_groups = [aws_security_group.quickwit_cluster_member_sg.id]\n }\n}\n\nresource \"aws_db_subnet_group\" \"quickwit\" {\n count = local.use_external_rds ? 0 : 1\n name = \"quickwit-${local.module_id}\"\n description = \"Quickwit metastore\"\n subnet_ids = var.subnet_ids\n}\n\nresource \"aws_ssm_parameter\" \"postgres_credential\" {\n count = local.use_external_rds ? 0 : 1\n name = \"/quickwit/${local.module_id}/postgres\"\n type = \"SecureString\"\n value = \"postgres://${module.quickwit_db[0].db_instance_username}:${random_password.quickwit_db[0].result}@${module.quickwit_db[0].db_instance_address}:${module.quickwit_db[0].db_instance_port}/${module.quickwit_db[0].db_instance_name}\"\n}\n" }, { "path": "distribution/ecs/quickwit/s3.tf", "content": "data \"aws_caller_identity\" \"current\" {}\n\nresource \"aws_s3_bucket\" \"index\" {\n count = var.quickwit_index_s3_prefix == \"\" ? 1 : 0\n bucket = \"quickwit-ecs-index-${data.aws_caller_identity.current.account_id}-${local.s3_id}\"\n force_destroy = true\n}\n" }, { "path": "distribution/ecs/quickwit/service/config.tf", "content": "locals {\n quickwit_data_dir = \"/quickwit/qwdata\"\n\n quickwit_common_environment = [\n {\n name = \"QW_ENABLED_SERVICES\"\n value = var.service_name\n },\n {\n name = \"QW_PEER_SEEDS\"\n value = join(\",\", var.quickwit_peer_list)\n },\n {\n name = \"NO_COLOR\"\n value = \"true\"\n },\n {\n name = \"QW_CLUSTER_ID\"\n value = \"ecs-${var.module_id}\"\n },\n {\n name = \"QW_LISTEN_ADDRESS\"\n value = \"0.0.0.0\"\n },\n {\n name = \"QW_DATA_DIR\"\n value = local.quickwit_data_dir\n },\n {\n name = \"QW_DEFAULT_INDEX_ROOT_URI\"\n value = \"s3://${var.quickwit_index_s3_prefix}\"\n },\n ]\n\n nb_extra_policies = length(var.service_config.extra_task_policy_arns)\n extra_tasks_iam_role_policies = { for i in range(local.nb_extra_policies) : \"extra_policy_${i}\" => var.service_config.extra_task_policy_arns[i] }\n tasks_iam_role_policies = merge({ s3_access = var.s3_access_policy_arn }, local.extra_tasks_iam_role_policies)\n}\n" }, { "path": "distribution/ecs/quickwit/service/ecs.tf", "content": "module \"quickwit_service\" {\n source = \"terraform-aws-modules/ecs/aws//modules/service\"\n version = \"5.9.3\"\n\n name = \"quickwit-${var.service_name}-${var.module_id}\"\n cluster_arn = var.cluster_arn\n\n cpu = var.service_config.cpu\n memory = var.service_config.memory\n ephemeral_storage = {\n size_in_gib = var.service_config.ephemeral_storage_gib\n }\n\n container_definitions = merge(var.sidecar_container_definitions, {\n quickwit = {\n cpu = var.service_config.cpu\n memory = var.service_config.memory\n\n essential = true\n image = var.quickwit_image\n enable_cloudwatch_logging = var.enable_cloudwatch_logging\n\n command = [\"run\"]\n\n environment = local.quickwit_common_environment\n\n secrets = [\n {\n name = \"QW_METASTORE_URI\"\n valueFrom = var.postgres_uri_secret_arn\n }\n ]\n\n port_mappings = [\n {\n name = \"rest\"\n containerPort = 7280\n protocol = \"tcp\"\n },\n {\n name = \"grpc\"\n containerPort = 7281\n protocol = \"tcp\"\n },\n {\n name = \"gossip\"\n containerPort = 7280\n protocol = \"udp\"\n }\n ]\n\n log_configuration = var.log_configuration\n\n mount_points = [\n {\n sourceVolume = \"quickwit-data-vol\"\n containerPath = local.quickwit_data_dir\n },\n # A volume that can be used to inject secrets as files.\n {\n sourceVolume = \"quickwit-keys\"\n containerPath = \"/quickwit/keys\"\n }\n ]\n\n stopTimeout = var.stop_timeout\n\n dependencies = var.sidecar_container_dependencies\n }\n })\n\n requires_compatibilities = [\"FARGATE\"]\n runtime_platform = {\n operating_system_family = \"LINUX\"\n cpu_architecture = var.quickwit_cpu_architecture\n }\n\n service_registries = {\n registry_arn = var.service_discovery_registry_arn\n container_name = \"quickwit\"\n }\n\n subnet_ids = var.subnet_ids\n security_group_rules = {\n ingress_internal = {\n type = \"ingress\"\n from_port = 7280\n to_port = 7281\n protocol = \"-1\"\n\n source_security_group_id = var.quickwit_cluster_member_sg_id\n }\n ingress_external = {\n type = \"ingress\"\n from_port = 7280\n to_port = 7281\n protocol = \"-1\"\n\n cidr_blocks = var.ingress_cidr_blocks\n }\n egress_all = {\n type = \"egress\"\n from_port = 0\n to_port = 0\n protocol = \"-1\"\n\n cidr_blocks = [\"0.0.0.0/0\"]\n }\n }\n security_group_ids = [var.quickwit_cluster_member_sg_id]\n\n enable_autoscaling = false\n desired_count = var.service_config.desired_count\n\n volume = [\n {\n name = \"quickwit-data-vol\"\n },\n {\n name = \"quickwit-keys\"\n }\n ]\n\n tasks_iam_role_policies = local.tasks_iam_role_policies\n\n task_exec_iam_role_policies = {\n policy = var.task_execution_policy_arn\n }\n\n}\n" }, { "path": "distribution/ecs/quickwit/service/variables.tf", "content": "variable \"service_name\" {\n description = \"One of indexer, metastore, searcher, control_plane, janitor\"\n}\n\nvariable \"service_discovery_registry_arn\" {}\n\nvariable \"sidecar_container_definitions\" {}\n\nvariable \"sidecar_container_dependencies\" {\n type = list(object({\n containerName = string\n condition = string\n }))\n default = []\n}\n\nvariable \"log_configuration\" {}\n\nvariable \"enable_cloudwatch_logging\" {\n type = bool\n}\n\nvariable \"cluster_arn\" {}\n\nvariable \"ingress_cidr_blocks\" {\n type = list(string)\n}\n\nvariable \"quickwit_cluster_member_sg_id\" {}\n\nvariable \"subnet_ids\" {\n type = list(string)\n}\n\nvariable \"postgres_uri_secret_arn\" {\n description = \"ARN of the SSM parameter or Secret Manager secret containing the URI of a Postgres instance\"\n}\n\nvariable \"quickwit_image\" {}\n\nvariable \"service_config\" {\n type = object({\n desired_count = optional(number, 1)\n memory = number\n cpu = number\n ephemeral_storage_gib = optional(number, 21)\n extra_task_policy_arns = optional(list(string), [])\n })\n}\n\nvariable \"quickwit_index_s3_prefix\" {}\n\nvariable \"quickwit_peer_list\" {\n type = list(string)\n}\n\nvariable \"s3_access_policy_arn\" {}\n\nvariable \"task_execution_policy_arn\" {}\n\nvariable \"quickwit_cpu_architecture\" {}\n\nvariable \"module_id\" {}\n\nvariable \"stop_timeout\" {\n # between 1s and 120s on Fargate, 30s is the ECS default\n default = 30\n}\n" }, { "path": "distribution/ecs/quickwit/variables.tf", "content": "## REQUIRED VARIABLES\n\nvariable \"vpc_id\" {\n description = \"VPC ID of the cluster\"\n}\n\nvariable \"subnet_ids\" {\n description = \"Subnet(s) where quickwit will be deployed\"\n type = list(string)\n}\n\n\n\n## OPTIONAL VARIABLES\n\nvariable \"module_id\" {\n description = \"Identifier for the module, e.g the stage. If not specified, a random string is generated.\"\n default = \"\"\n}\n\nvariable \"quickwit_ingress_cidr_blocks\" {\n description = \"CIDR blocks (private) that should have access to the Quickwit cluster\"\n type = list(string)\n default = []\n}\n\n\nvariable \"quickwit_index_s3_prefix\" {\n description = \"S3 bucket name and prefix for the Quickwit data, e.g. my-bucket-name/my-prefix. Quickwit will only have access to this S3 location. Leave empty to create a new bucket.\"\n default = \"\"\n}\n\nvariable \"quickwit_domain\" {\n description = \"Local domain for quickwit service discovery\"\n default = \"quickwit\"\n}\n\nvariable \"quickwit_image\" {\n description = \"Quickwit docker image\"\n default = \"quickwit/quickwit:latest\"\n}\n\nvariable \"quickwit_cpu_architecture\" {\n description = \"One of X86_64 / ARM64. Must match the arch of the provided image (var.quickwit_image).\"\n default = \"ARM64\"\n}\n\nvariable \"sidecar_container_definitions\" {\n description = \"Sidecar containers to be attached to Quickwit tasks\"\n default = {}\n}\n\nvariable \"sidecar_container_dependencies\" {\n description = \"Specify the Quickwit container's dependencies on sidecars\"\n type = list(object({\n containerName = string\n condition = string\n }))\n default = []\n}\n\nvariable \"enable_cloudwatch_logging\" {\n description = \"Cloudwatch logging for Quickwit tasks. Usually disabled when using a custom log configuration.\"\n default = true\n}\n\nvariable \"log_configuration\" {\n description = \"Custom log configuration for Quickwit tasks\"\n default = {}\n}\n\nvariable \"quickwit_indexer\" {\n description = \"Indexer service sizing configurations\"\n type = object({\n desired_count = optional(number, 1)\n memory = optional(number, 8192)\n cpu = optional(number, 2048)\n ephemeral_storage_gib = optional(number, 21)\n extra_task_policy_arns = optional(list(string), [])\n })\n default = {}\n}\n\nvariable \"quickwit_metastore\" {\n description = \"Metastore service sizing configurations\"\n type = object({\n desired_count = optional(number, 1)\n memory = optional(number, 512)\n cpu = optional(number, 256)\n })\n default = {}\n}\n\nvariable \"quickwit_searcher\" {\n description = \"Searcher service sizing configurations\"\n type = object({\n desired_count = optional(number, 1)\n memory = optional(number, 4096)\n cpu = optional(number, 1024)\n ephemeral_storage_gib = optional(number, 21)\n })\n default = {}\n}\n\nvariable \"quickwit_control_plane\" {\n description = \"Control plane service sizing configurations\"\n type = object({\n # only 1 task is necessary\n memory = optional(number, 512)\n cpu = optional(number, 256)\n })\n default = {}\n}\n\nvariable \"quickwit_janitor\" {\n description = \"Janitor service sizing configurations\"\n type = object({\n # only 1 task is necessary\n memory = optional(number, 512)\n cpu = optional(number, 256)\n })\n default = {}\n}\n\nvariable \"rds_config\" {\n description = \"Configurations of the metastore RDS database. Enable multi_az to ensure high availability.\"\n type = object({\n instance_class = optional(string, \"db.t4g.micro\")\n multi_az = optional(bool, false)\n })\n default = {}\n}\n\nvariable \"external_postgres_uri_secret_arn\" {\n description = \"ARN of the SSM parameter or Secret Manager secret containing the URI of a Postgres instance (postgres://{user}:{password}@{address}:{port}/{db_instance_name}). The Postgres instance should allow indbound connections from the subnets specified in `variable.subnet_ids`. If provided, the internal RDS will not be created and `var.rds_config` is ignored.\"\n default = \"\"\n}\n" }, { "path": "distribution/kubernetes/README.md", "content": "# Quickwit on Kubernetes\n\nTo deploy Quickwit on Kubernetes, use the official Quickwit Helm chart available at [helm.quickwit.io](https://helm.quickwit.io/) and refer to our [documentation](https://quickwit.io/docs/deployment/kubernetes/helm) for more information.\n" }, { "path": "docker-compose.yml", "content": "# By default, this docker compose script maps all services to localhost only.\n# If you need to make services available outside of your machine, add\n# appropriate service mappings to the .env file. See .env.example file for\n# configuration example.\n#\n# Notes on image versions:\n# - For the key services such as postgres and pulsar we are trying to run\n# against the oldest supported version\n# - For kafka we use the oldest version that supports KRaft.\n# - For everything else we are trying to run against the latest version.\n#\n# To run against the latest image versions update .env file. See .env.example\n# file for configuration examples. You might need to remove the old images\n# first if they are already tagged latest and volumes if their content is\n# incompatible with the latest version, as in case of postgres.\n\nname: quickwit\n\nnetworks:\n default:\n name: quickwit-network\n ipam:\n config:\n - subnet: 172.16.7.0/24\n gateway: 172.16.7.1\n\nservices:\n localstack:\n image: localstack/localstack:${LOCALSTACK_VERSION:-3.5.0}\n container_name: localstack\n ports:\n - \"${MAP_HOST_LOCALSTACK:-127.0.0.1}:4566:4566\"\n - \"${MAP_HOST_LOCALSTACK:-127.0.0.1}:4571:4571\"\n - \"${MAP_HOST_LOCALSTACK:-127.0.0.1}:8080:8080\"\n profiles:\n - all\n - localstack\n environment:\n SERVICES: kinesis,s3,sqs\n PERSISTENCE: 1\n volumes:\n - .localstack:/etc/localstack/init/ready.d\n - localstack_data:/var/lib/localstack\n healthcheck:\n test: [\"CMD\", \"curl\", \"-k\", \"-f\", \"https://localhost:4566/quickwit-integration-tests\"]\n interval: 1s\n timeout: 5s\n retries: 100\n\n postgres:\n # The oldest supported version. EOL November 14, 2024\n image: postgres:${POSTGRES_VERSION:-12.17-alpine}\n container_name: postgres\n ports:\n - \"${MAP_HOST_POSTGRES:-127.0.0.1}:5432:5432\"\n profiles:\n - all\n - postgres\n environment:\n PGDATA: /var/lib/postgresql/data/pgdata\n POSTGRES_USER: ${POSTGRES_USER:-quickwit-dev}\n POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-quickwit-dev}\n POSTGRES_DB: ${POSTGRES_DB:-quickwit-metastore-dev}\n volumes:\n - postgres_data:/var/lib/postgresql/data\n healthcheck:\n test: [\"CMD\", \"pg_isready\"]\n interval: 1s\n timeout: 5s\n retries: 100\n\n pulsar-broker:\n # The oldest version with arm64 docker images. EOL May 2 2025\n image: apachepulsar/pulsar:${PULSAR_VERSION:-3.0.0}\n container_name: pulsar-broker\n command: bin/pulsar standalone --no-functions-worker\n ports:\n - \"${MAP_HOST_PULSAR:-127.0.0.1}:6650:6650\"\n - \"${MAP_HOST_PULSAR:-127.0.0.1}:8081:8080\"\n environment:\n PULSAR_MEM: \"-Xms384M -Xmx384M\"\n # Disable functions worker to save memory/time\n PULSAR_PREFIX_functionsWorkerEnabled: \"false\"\n profiles:\n - all\n - pulsar\n\n kafka-broker:\n image: confluentinc/confluent-local:${CP_VERSION:-7.4.11}\n container_name: kafka-broker\n ports:\n - \"${MAP_HOST_KAFKA:-127.0.0.1}:9092:9092\"\n - \"${MAP_HOST_KAFKA:-127.0.0.1}:9101:9101\"\n profiles:\n - all\n - kafka\n\n environment:\n # Mode KRaft (Single Node)\n KAFKA_NODE_ID: 1\n KAFKA_PROCESS_ROLES: 'broker,controller'\n KAFKA_CONTROLLER_QUORUM_VOTERS: '1@localhost:9093'\n KAFKA_LOG4J_LOGGERS: \"org.apache.kafka.image.loader.MetadataLoader=WARN\"\n\n # Listeners\n KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: 'CONTROLLER:PLAINTEXT,EXTERNAL:PLAINTEXT'\n KAFKA_LISTENERS: 'EXTERNAL://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093'\n KAFKA_ADVERTISED_LISTENERS: 'EXTERNAL://localhost:9092'\n KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'\n KAFKA_INTER_BROKER_LISTENER_NAME: 'EXTERNAL'\n\n # Configuration simplifiée\n KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1\n KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0\n KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1\n KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1\n\n # ID du Cluster (Nécessaire pour KRaft)\n CLUSTER_ID: 'MkU3OEVBNTcwNTJENDM2Qk'\n\n KAFKA_HEAP_OPTS: -Xms256M -Xmx256M\n healthcheck:\n # test: [\"CMD-SHELL\", \"nc -z localhost 9092 || exit 1\"]\n test: [\"CMD\", \"ub\", \"kafka-ready\", \"-b\", \"localhost:9092\", \"1\", \"5\"]\n start_period: 5s\n interval: 5s\n timeout: 10s\n retries: 100\n\n azurite:\n image: mcr.microsoft.com/azure-storage/azurite:${AZURITE_VERSION:-3.24.0}\n container_name: azurite\n ports:\n - \"${MAP_HOST_AZURITE:-127.0.0.1}:10000:10000\" # Blob store port\n profiles:\n - all\n - azurite\n volumes:\n - azurite_data:/data\n command: azurite --blobHost 0.0.0.0 --loose\n\n fake-gcs-server:\n image: fsouza/fake-gcs-server:${FAKE_GCS_SERVER_VERSION:-1.47.7}\n container_name: fake-gcs-server\n ports:\n - \"${MAP_HOST_FAKE_GCS_SERVER:-127.0.0.1}:4443:4443\" # Blob store port\n profiles:\n - all\n - fake-gcs-server\n volumes:\n - fake_gcs_server_data:/data\n command: -scheme http\n\n grafana:\n image: grafana/grafana-oss:${GRAFANA_VERSION:-10.4.1}\n container_name: grafana\n ports:\n - \"${MAP_HOST_GRAFANA:-127.0.0.1}:3000:3000\"\n profiles:\n - grafana\n - monitoring\n environment:\n GF_AUTH_DISABLE_LOGIN_FORM: \"true\"\n GF_AUTH_ANONYMOUS_ENABLED: \"true\"\n GF_AUTH_ANONYMOUS_ORG_ROLE: Admin\n volumes:\n - grafana_conf:/etc/grafana\n - grafana_data:/var/lib/grafana\n - ./monitoring/grafana/dashboards:/var/lib/grafana/dashboards\n - ./monitoring/grafana/provisioning:/etc/grafana/provisioning\n\n jaeger:\n image: jaegertracing/all-in-one:${JAEGER_VERSION:-1.48.0}\n container_name: jaeger\n ports:\n - \"${MAP_HOST_JAEGER:-127.0.0.1}:16686:16686\" # Frontend\n profiles:\n - jaeger\n - monitoring\n\n otel-collector:\n image: otel/opentelemetry-collector:${OTEL_VERSION:-0.84.0}\n container_name: otel-collector\n ports:\n - \"${MAP_HOST_OTEL:-127.0.0.1}:1888:1888\" # pprof extension\n - \"${MAP_HOST_OTEL:-127.0.0.1}:8888:8888\" # Prometheus metrics exposed by the collector\n - \"${MAP_HOST_OTEL:-127.0.0.1}:8889:8889\" # Prometheus exporter metrics\n - \"${MAP_HOST_OTEL:-127.0.0.1}:13133:13133\" # health_check extension\n - \"${MAP_HOST_OTEL:-127.0.0.1}:4317:4317\" # OTLP gRPC receiver\n - \"${MAP_HOST_OTEL:-127.0.0.1}:4318:4318\" # OTLP http receiver\n - \"${MAP_HOST_OTEL:-127.0.0.1}:55679:55679\" # zpages extension\n profiles:\n - otel\n - monitoring\n volumes:\n - ./monitoring/otel-collector-config.yaml:/etc/otel-collector-config.yaml\n command: [\"--config=/etc/otel-collector-config.yaml\"]\n\n prometheus:\n image: prom/prometheus:${PROMETHEUS_VERSION:-v2.43.0}\n container_name: prometheus\n ports:\n - \"${MAP_HOST_PROMETHEUS:-127.0.0.1}:9090:9090\"\n profiles:\n - prometheus\n - monitoring\n volumes:\n - ./monitoring/prometheus.yaml:/etc/prometheus/prometheus.yml\n extra_hosts:\n - \"host.docker.internal:host-gateway\"\n\n gcp-pubsub-emulator:\n # It is not an official docker image\n # if we prefer we can build a docker from the official docker image (gcloud cli)\n # and install the pubsub emulator https://cloud.google.com/pubsub/docs/emulator\n image: thekevjames/gcloud-pubsub-emulator:${GCLOUD_EMULATOR:-550.0.0}\n container_name: gcp-pubsub-emulator\n ports:\n - \"${MAP_HOST_GCLOUD_EMULATOR:-127.0.0.1}:8681:8681\"\n environment:\n # create a fake gcp project and a topic / subscription\n - PUBSUB_PROJECT1=quickwit-emulator,emulator_topic:emulator_subscription\n profiles:\n - all\n - gcp-pubsub\n\nvolumes:\n azurite_data:\n fake_gcs_server_data:\n grafana_conf:\n grafana_data:\n localstack_data:\n postgres_data:\n" }, { "path": "docs/assets/sqs-file-source.tf", "content": "terraform {\n required_version = \"1.7.5\"\n required_providers {\n aws = {\n source = \"hashicorp/aws\"\n version = \"~> 5.39.1\"\n }\n }\n}\n\nprovider \"aws\" {\n region = \"us-east-1\"\n default_tags {\n tags = {\n provisioner = \"terraform\"\n author = \"Quickwit\"\n }\n }\n}\n\nlocals {\n sqs_notification_queue_name = \"qw-tuto-s3-event-notifications\"\n source_bucket_name = \"qw-tuto-source-bucket\"\n}\n\nresource \"aws_s3_bucket\" \"file_source\" {\n bucket_prefix = local.source_bucket_name\n force_destroy = true\n}\n\ndata \"aws_iam_policy_document\" \"sqs_notification\" {\n statement {\n effect = \"Allow\"\n\n principals {\n type = \"*\"\n identifiers = [\"*\"]\n }\n\n actions = [\"sqs:SendMessage\"]\n resources = [\"arn:aws:sqs:*:*:${local.sqs_notification_queue_name}\"]\n\n condition {\n test = \"ArnEquals\"\n variable = \"aws:SourceArn\"\n values = [aws_s3_bucket.file_source.arn]\n }\n }\n}\n\n\nresource \"aws_sqs_queue\" \"s3_events\" {\n name = local.sqs_notification_queue_name\n policy = data.aws_iam_policy_document.sqs_notification.json\n\n redrive_policy = jsonencode({\n deadLetterTargetArn = aws_sqs_queue.s3_events_deadletter.arn\n maxReceiveCount = 5\n })\n}\n\nresource \"aws_sqs_queue\" \"s3_events_deadletter\" {\n name = \"${locals.sqs_notification_queue_name}-deadletter\"\n}\n\nresource \"aws_sqs_queue_redrive_allow_policy\" \"s3_events_deadletter\" {\n queue_url = aws_sqs_queue.s3_events_deadletter.id\n\n redrive_allow_policy = jsonencode({\n redrivePermission = \"byQueue\",\n sourceQueueArns = [aws_sqs_queue.s3_events.arn]\n })\n}\n\nresource \"aws_s3_bucket_notification\" \"bucket_notification\" {\n bucket = aws_s3_bucket.file_source.id\n\n queue {\n queue_arn = aws_sqs_queue.s3_events.arn\n events = [\"s3:ObjectCreated:*\"]\n }\n}\n\ndata \"aws_iam_policy_document\" \"quickwit_node\" {\n statement {\n effect = \"Allow\"\n actions = [\n \"sqs:ReceiveMessage\",\n \"sqs:DeleteMessage\",\n \"sqs:ChangeMessageVisibility\",\n \"sqs:GetQueueAttributes\",\n ]\n resources = [aws_sqs_queue.s3_events.arn]\n }\n statement {\n effect = \"Allow\"\n actions = [\"s3:GetObject\"]\n resources = [\"${aws_s3_bucket.file_source.arn}/*\"]\n }\n}\n\nresource \"aws_iam_user\" \"quickwit_node\" {\n name = \"quickwit-filesource-tutorial\"\n path = \"/system/\"\n}\n\nresource \"aws_iam_user_policy\" \"quickwit_node\" {\n name = \"quickwit-filesource-tutorial\"\n user = aws_iam_user.quickwit_node.name\n policy = data.aws_iam_policy_document.quickwit_node.json\n}\n\nresource \"aws_iam_access_key\" \"quickwit_node\" {\n user = aws_iam_user.quickwit_node.name\n}\n\noutput \"source_bucket_name\" {\n value = aws_s3_bucket.file_source.bucket\n\n}\n\noutput \"notification_queue_url\" {\n value = aws_sqs_queue.s3_events.id\n}\n\noutput \"quickwit_node_access_key_id\" {\n value = aws_iam_access_key.quickwit_node.id\n sensitive = true\n}\n\noutput \"quickwit_node_secret_access_key\" {\n value = aws_iam_access_key.quickwit_node.secret\n sensitive = true\n}\n" }, { "path": "docs/configuration/_category_.yaml", "content": "label: 'Configuration'\nposition: 4\ncollapsed: true\n" }, { "path": "docs/configuration/index-config.md", "content": "---\ntitle: Index configuration\nsidebar_position: 3\ntoc_max_heading_level: 4\n---\n\nThis page describes how to configure an index.\n\nIn addition to the `index_id`, the index configuration lets you define five items:\n\n- The **index-uri**: it defines where the index files should be stored.\n- The **doc mapping**: it defines how a document and the fields it contains are stored and indexed for a given index.\n- The **indexing settings**: it defines the timestamp field used for sharding, and some more advanced parameters like the merge policy.\n- The **search settings**: it defines the default search fields `default_search_fields`, a list of fields that Quickwit will search into if the user query does not explicitly target a field.\n- The **retention policy**: it defines how long Quickwit should keep the indexed data. If not specified, the data is stored forever.\n\nConfiguration is set at index creation and can be changed using the [update endpoint](../reference/rest-api.md) or the [CLI](../reference/cli.md).\n\n## Config file format\n\nThe index configuration format is YAML. When a key is absent from the configuration file, the default value is used.\nHere is a complete example suited for the HDFS logs dataset:\n\n```yaml\nversion: 0.7 # File format version.\n\nindex_id: \"hdfs\"\n\nindex_uri: \"s3://my-bucket/hdfs\"\n\ndoc_mapping:\n mode: lenient\n field_mappings:\n - name: timestamp\n type: datetime\n input_formats:\n - unix_timestamp\n output_format: unix_timestamp_secs\n fast_precision: seconds\n fast: true\n - name: severity_text\n type: text\n tokenizer: raw\n fast:\n - tokenizer: lowercase\n - name: body\n type: text\n tokenizer: default\n record: position\n - name: resource\n type: object\n field_mappings:\n - name: service\n type: text\n tokenizer: raw\n tag_fields: [\"resource.service\"]\n timestamp_field: timestamp\n index_field_presence: true\n\nsearch_settings:\n default_search_fields: [severity_text, body]\n\nretention:\n period: 90 days\n schedule: daily\n```\n\n## Index ID\n\nThe index ID is a string that uniquely identifies the index within the metastore. It may only contain uppercase or lowercase ASCII letters, digits, hyphens (`-`), and underscores (`_`). Finally, it must start with a letter and contain at least 3 characters but no more than 255.\n\n## Index uri\n\nThe index-uri defines where the index files (also called splits) should be stored.\nThis parameter expects a [storage uri](storage-config#storage-uris).\n\nThe `index-uri` parameter is optional.\nBy default, the `index-uri` will be computed by concatenating the `index-id` with the\n`default_index_root_uri` defined in the [Quickwit's config](node-config).\n\n:::caution\nThe file storage will not work when running quickwit in distributed mode. Instead, AWS S3, Azure Blob Storage, Google Cloud Storage (in s3 interoperability mode) or other S3-compatible storage systems including Scaleway Object Storage and Garage should be used as storage when running several searcher nodes.\n:::\n\n## Doc mapping\n\nThe doc mapping defines how a document and the fields it contains are stored and indexed for a given index. A document is a collection of named fields, each having its own data type (text, bytes, datetime, bool, i64, u64, f64, ip, json).\n\n| Variable | Description | Default value |\n| ------------- | ------------- | ------------- |\n| `field_mappings` | Collection of field mapping, each having its own data type (text, binary, datetime, bool, i64, u64, f64, ip, json). | `[]` |\n| `mode` | Defines how quickwit should handle document fields that are not present in the `field_mappings`. In particular, the \"dynamic\" mode makes it possible to use quickwit in a schemaless manner. (See [mode](#mode)) | `dynamic`\n| `dynamic_mapping` | This parameter is only allowed when `mode` is set to `dynamic`. It then defines whether dynamically mapped fields should be indexed, stored, etc. | (See [mode](#mode))\n| `tag_fields` | Collection of fields* explicitly defined in `field_mappings` whose values will be stored as part of the `tags` metadata. Allowed types are: `text` (with raw tokenizer), `i64` and `u64`. [Learn more about tags](../overview/concepts/querying.md#tag-pruning). | `[]` |\n| `store_source` | Whether or not the original JSON document is stored or not in the index. | `false` |\n| `timestamp_field` | Timestamp field* used for sharding documents in splits. The field has to be of type `datetime`. [Learn more about time sharding](./../overview/architecture.md). | `None` |\n| `partition_key` | If set, quickwit will route documents into different splits depending on the field name declared as the `partition_key`. | `null` |\n| `max_num_partitions` | Limits the number of splits created through partitioning. (See [Partitioning](../overview/concepts/querying.md#partitioning)) | `200` |\n| `index_field_presence` | `exists` queries are enabled automatically for fast fields. To enable it for all other fields set this parameter to `true`. Enabling it can have a significant CPU-cost on indexing. | false |\n\n*: tags fields and timestamp field are expressed as a path from the root of the JSON object to the given field. If a field name contains a `.` character, it needs to be escaped with a `\\` character.\n\n### Field types\n\nEach field[^1] has a type that indicates the kind of data it contains, such as integer on 64 bits or text.\nQuickwit supports the following raw types [`text`](#text-type), [`i64`](#numeric-types-i64-u64-and-f64-type), [`u64`](#numeric-types-i64-u64-and-f64-type), [`f64`](#numeric-types-i64-u64-and-f64-type), [`datetime`](#datetime-type), [`bool`](#bool-type), [`ip`](#ip-type), [`bytes`](#bytes-type), and [`json`](#json-type), and also supports composite types such as array and object. Behind the scenes, Quickwit is using tantivy field types, don't hesitate to look at [tantivy documentation](https://github.com/tantivy-search/tantivy) if you want to go into the details.\n\n### Raw types\n\n#### Text type\n\nThis field is a text field that will be analyzed and split into tokens before indexing.\nThis kind of field is tailored for full-text search.\n\nExample of a mapping for a text field:\n\n```yaml\nname: body\ndescription: Body of the document\ntype: text\ntokenizer: default\nrecord: position\nfieldnorms: true\nfast:\n normalizer: lowercase\n```\n\n**Parameters for text field**\n\n| Variable | Description | Default value |\n| ------------- | ------------- | ------------- |\n| `description` | Optional description for the field. | `None` |\n| `stored` | Whether value is stored in the document store | `true` |\n| `indexed` | Whether value should be indexed so it can be searched | `true` |\n| `tokenizer` | Name of the `Tokenizer`. ([See tokenizers](#description-of-available-tokenizers)) for a list of available tokenizers. | `default` |\n| `record` | Describes the amount of information indexed, choices between `basic`, `freq` and `position` | `basic` |\n| `fieldnorms` | Whether to store fieldnorms for the field. Fieldnorms are required to calculate the BM25 Score of the document. | `false` |\n| `fast` | Whether value is stored in a fast field. The fast field will contain the term ids and the dictionary. The default behaviour for `true` is to store the original text unchanged. The normalizers on the fast field is separately configured. It can be configured via `normalizer: lowercase`. ([See normalizers](#description-of-available-normalizers)) for a list of available normalizers. | `false` |\n\n##### Description of available tokenizers\n\n| Tokenizer | Description |\n| ------------- | ------------- |\n| `raw` | Does not process nor tokenize text. Filters out tokens larger than 255 bytes. This is similar to the `keyword` type in Elasticsearch. |\n| `raw_lowercase` | Does not tokenize text, but lowercase it. Filters out tokens larger than 255 bytes. |\n| `default` | Chops the text on according to whitespace and punctuation, removes tokens that are too long, and converts to lowercase. Filters out tokens larger than 255 bytes. |\n| `en_stem` | Like `default`, but also applies stemming on the resulting tokens. Filters out tokens larger than 255 bytes. |\n| `whitespace` | Chops the text on according to whitespace only. Doesn't remove long tokens or converts to lowercase. |\n| `chinese_compatible` | Chop between each CJK character in addition to what `default` does. Should be used with `record: position` to be able to properly search |\n| `lowercase` | Applies a lowercase transformation on the text. It does not tokenize the text. |\n\n##### Description of available normalizers\n\n| Normalizer | Description |\n| ------------- | ------------- |\n| `raw` | Does not process nor tokenize text. Filters token larger than 255 bytes. |\n| `lowercase` | Applies a lowercase transformation on the text. Filters token larger than 255 bytes. |\n\n**Description of record options**\n\n| Record option | Description |\n| ------------- | ------------- |\n| `basic` | Records only the `DocId`s |\n| `freq` | Records the document ids as well as the term frequency |\n| `position` | Records the document id, the term frequency and the positions of occurrences. |\n\nIndexing with position is required to run phrase queries.\n\n#### Numeric types: `i64`, `u64` and `f64` type\n\nQuickwit handles three numeric types: `i64`, `u64`, and `f64`.\n\nNumeric values can be stored in a fast field (the equivalent of Lucene's `DocValues`), which is a column-oriented storage used for range queries and aggregations.\n\nWhen querying negative numbers without precising a field (using `default_search_fields`), you should single-quote the number (for instance '-5'), otherwise it will be interpreted as wanting to match anything but that number.\n\nExample of a mapping for an u64 field:\n\n```yaml\nname: rating\ndescription: Score between 0 and 5\ntype: u64\nstored: true\nindexed: true\nfast: true\n```\n\n**Parameters for i64, u64 and f64 field**\n\n| Variable | Description | Default value |\n| --------------- | ------------- | ------------- |\n| `description` | Optional description for the field. | `None` |\n| `stored` | Whether the field values are stored in the document store. | `true` |\n| `indexed` | Whether the field values are indexed. | `true` |\n| `fast` | Whether the field values are stored in a fast field. | `false` |\n| `coerce` | Whether to convert numbers passed as strings to integers or floats. | `true` |\n| `output_format` | JSON type used to return numbers in search results. Possible values are `number` or `string`. | `number` |\n\n#### `datetime` type\n\nThe `datetime` type handles dates and datetimes. Since JSON doesn’t have a date type, the `datetime` field support multiple input types and formats. The supported input types are:\n- floating-point or integer numbers representing a Unix timestamp\n- strings containing a formatted date, datetime, or Unix timestamp\n\nThe `input_formats` field parameter specifies the accepted date formats. The following input formats are natively supported:\n- `iso8601`\n- `rfc2822`\n- `rfc3339`\n- `strptime`\n- `unix_timestamp`\n\n**Input formats**\n\nWhen specifying multiple input formats, the corresponding parsers are attempted in the order they are declared. The following formats are natively supported:\n- `iso8601`, `rfc2822`, `rfc3339`: parse dates using standard ISO and RFC formats.\n- `strptime`: parse dates using the Unix [strptime](https://man7.org/linux/man-pages/man3/strptime.3.html) format with some variations:\n - `strptime` format specifiers: `%C`, `%d`, `%D`, `%e`, `%F`, `%g`, `%G`, `%h`, `%H`, `%I`, `%j`, `%k`, `%l`, `%m`, `%M`, `%n`, `%R`, `%S`, `%t`, `%T`, `%u`, `%U`, `%V`, `%w`, `%W`, `%y`, `%Y`, `%%`.\n - `%f` for milliseconds precision support.\n - `%z` timezone offsets can be specified as `(+|-)hhmm` or `(+|-)hh:mm`.\n\n:::warning\nThe timezone name format specifier (`%Z`) is not supported currently.\n:::\n\n- `unix_timestamp`: parse float and integer numbers to Unix timestamps. Floating-point values are converted to timestamps expressed in seconds. Integer values are converted to Unix timestamps whose precision, determined in `seconds`, `milliseconds`, `microseconds`, or `nanoseconds`, is inferred from the number of input digits. Internally, datetimes are converted to UTC (if the time zone is specified) and stored as *i64* integers. As a result, Quickwit only supports timestamp values ranging from `Apr 13, 1972 23:59:55` to `Mar 16, 2242 12:56:31`.\n\n:::warning\nConverting timestamps from float to integer values may occur with a loss of precision.\n:::\n\nWhen a `datetime` field is stored as a fast field, the `fast_precision` parameter indicates the precision used to truncate the values before encoding, which improves compression (truncation here means zeroing). The `fast_precision` parameter can take the following values: `seconds`, `milliseconds`, `microseconds`, or `nanoseconds`. It only affects what is stored in fast fields when a `datetime` field is marked as \"fast\". Finally, operations on `datetime` fast fields, e.g. via aggregations, need to be done at the nanosecond level.\n\n:::info\nInternally `datetime` is stored in `nanoseconds` in fast fields and in the docstore, and in `seconds` in the term dictionary.\n:::\n\nIn addition, Quickwit supports the `output_format` field parameter to specify with which precision datetimes are deserialized. This parameter supports the same value as input formats except for `unix_timestamp` which is replaced by the following formats:\n- `unix_timestamp_secs`: displays timestamps in seconds.\n- `unix_timestamp_millis`: displays timestamps in milliseconds.\n- `unix_timestamp_micros`: displays timestamps in microseconds.\n- `unix_timestamp_nanos`: displays timestamps in nanoseconds.\n\nExample of a mapping for a datetime field:\n\n```yaml\nname: timestamp\ntype: datetime\ndescription: Time at which the event was emitted\ninput_formats:\n - rfc3339\n - unix_timestamp\n - \"%Y %m %d %H:%M:%S.%f %z\"\noutput_format: unix_timestamp_secs\nstored: true\nindexed: true\nfast: true\nfast_precision: milliseconds\n```\n\n**Parameters for datetime field**\n\n| Variable | Description | Default value |\n| ------------- | ------------- | ------------- |\n| `input_formats` | Formats used to parse input dates | [`rfc3339`, `unix_timestamp`] |\n| `output_format` | Format used to display dates in search results | `rfc3339` |\n| `stored` | Whether the field values are stored in the document store | `true` |\n| `indexed` | Whether the field values are indexed | `true` |\n| `fast` | Whether the field values are stored in a fast field | `false` |\n| `fast_precision` | The precision (`seconds`, `milliseconds`, `microseconds`, or `nanoseconds`) used to store the fast values. | `seconds` |\n\n#### `bool` type\n\nThe `bool` type accepts boolean values.\n\nExample of a mapping for a boolean field:\n\n```yaml\nname: is_active\ndescription: Activation status\ntype: bool\nstored: true\nindexed: true\nfast: true\n```\n\n**Parameters for bool field**\n\n| Variable | Description | Default value |\n| ------------- | ------------- | ------------- |\n| `description` | Optional description for the field. | `None` |\n| `stored` | Whether value is stored in the document store | `true` |\n| `indexed` | Whether value is indexed | `true` |\n| `fast` | Whether value is stored in a fast field | `false` |\n\n#### `ip` type\n\nThe `ip` type accepts IP address values, both IpV4 and IpV6 are supported. Internally IpV4 are converted to IpV6.\n\nExample of a mapping for an IP field:\n\n```yaml\nname: host_ip\ndescription: Host IP address\ntype: ip\nfast: true\n```\n\n**Parameters for IP field**\n\n| Variable | Description | Default value |\n| ------------- | ------------- | ------------- |\n| `description` | Optional description for the field. | `None` |\n| `stored` | Whether value is stored in the document store | `true` |\n| `indexed` | Whether value is indexed | `true` |\n| `fast` | Whether value is stored in a fast field | `false` |\n\n\n#### `bytes` type\nThe `bytes` type accepts a binary value as a `Base64` encoded string.\n\nExample of a mapping for a bytes field:\n\n```yaml\nname: binary\ntype: bytes\nstored: true\nindexed: true\nfast: true\ninput_format: hex\noutput_format: hex\n```\n\n**Parameters for bytes field**\n\n| Variable | Description | Default value |\n| ------------- | ------------- | ------------- |\n| `description` | Optional description for the field. | `None` |\n| `stored` | Whether value is stored in the document store | `true` |\n| `indexed` | Whether value is indexed | `true` |\n| `fast` | Whether value is stored in a fast field. Only on 1:1 cardinality, not supported on `array` fields | `false` |\n| `input_format` | Encoding used to represent input bytes, either `hex` or `base64` | `base64` |\n| `output_format` | Encoding used to represent bytes in search results, either `hex` or `base64` | `base64` |\n\n#### `json` type\n\nThe `json` type accepts a JSON object.\n\nExample of a mapping for a JSON field:\n\n```yaml\nname: parameters\ntype: json\nstored: true\nindexed: true\ntokenizer: raw\nexpand_dots: false\nfast:\n normalizer: lowercase\n```\n\nStored primitive types are inferred from the JSON value types using the following rules:\n- a boolean value `true` or `false` is stored as `bool`\n- numeric values are cast to the first compatible format between `i64`, `u64` or\n `f64` (in this order)\n- for string values (surrounded with quotes), Tantivy attempts to parse a date\n in `rfc3339` format. If the parsing fails, the value is stored as `text` using\n the configured tokenization rules\n\n**Parameters for JSON field**\n\n| Variable | Description | Default value |\n| ------------- | ------------- | ------------- |\n| `description` | Optional description for the field. | `None` |\n| `stored` | Whether value is stored in the document store | `true` |\n| `indexed` | Whether value is indexed | `true` |\n| `fast` | Whether value is stored in a fast field. The default behaviour for text in the JSON is to store the text unchanged. A normalizer can be configured via `normalizer: lowercase`. ([See normalizers](#description-of-available-normalizers)) for a list of available normalizers. | `false` |\n| `tokenizer` | **Only affects strings in the json object**. Name of the `Tokenizer`, choices between `raw`, `default`, `en_stem` and `chinese_compatible` | `raw` |\n| `record` | **Only affects strings in the json object**. Describes the amount of information indexed, choices between `basic`, `freq` and `position` | `basic` |\n| `expand_dots` | If true, json keys containing a `.` should be expanded. For instance, if `expand_dots` is set to true, `{\"k8s.node.id\": \"node-2\"}` will be indexed as if it was `{\"k8s\": {\"node\": {\"id\": \"node2\"}}}`. The benefit is that escaping the `.` will not be required at query time. In other words, `k8s.node.id:node2` will match the document. This does not impact the way the document is stored. | `true` |\n\nNote that the `tokenizer` and the `record` have the same definition and the same effect as for the text field.\n\nTo search into a json object, one then needs to extend the field name with the path that will lead to the target value.\n\nFor instance, when indexing the following object:\n```json\n{\n \"product_name\": \"droopy t-shirt\",\n \"attributes\": {\n \"color\": [\"red\", \"green\", \"white\"],\n \"size:\": \"L\"\n }\n}\n```\n\nAssuming `attributes` as been defined as a field mapping as follows:\n```yaml\n- type: json\n name: attributes\n```\n\n`attributes.color:red` is then a valid query.\n\nIf, in addition, `attributes` is set as a default search field, then `color:red` is a valid query.\n\n### Composite types\n\n#### array\n\nQuickwit supports arrays for all raw types except for `object` types.\n\nTo declare an array type of `i64` in the index config, you just have to set the type to `array`.\n\n#### object\n\nQuickwit supports nested objects as long as it does not contain arrays of objects.\n\n```yaml\nname: resource\ntype: object\nfield_mappings:\n - name: service\n type: text\n```\n\n#### concatenate\n\nQuickwit supports mapping the content of multiple fields to a single one. This can be more efficient at query time than\nsearching through dozens of `default_search_fields`. It also allows querying inside a json field without knowing the path\nto the field being searched.\n\n```yaml\nname: my_default_field\ntype: concatenate\nconcatenate_fields:\n - text # things inside text, tokenized with the `default` tokenizer\n - resource.author # all fields in resource.author, assuming resource is an `object` field.\ninclude_dynamic_fields: true\ntokenizer: default\nrecord: basic\n```\n\nConcatenate fields don't support fast fields, and are never stored. They uses their own tokenizer, independently of the\ntokenizer configured on the individual fields.\nAt query time, concatenate fields don't support range queries.\nOnly the following types are supported inside a concatenate field: text, bool,\ni64, u64, f64, json. Other types are rejected at index creation, or silently\ndiscarded during indexation if they are found inside a json field. Unlike\nregular JSON fields, JSON fields in a concatenate field don't store RFC3339\ndates as Tantivy dates. This means you can still perform prefix queries,\ne.g `my_default_field:\"2025-12-12\"*` to work around the lack of support for range\nqueries.\nAdding an object field to a concatenate field doesn't automatically add its subfields (yet).\n\nIt isn't possible to add subfields from a json field to a concatenate field. For instance if `attributes` is a json field, it's not possible to add only `attributes.color` to a concatenate field.\n\nFor json fields and dynamic fields, the path is not indexed, only values are. For instance, given the following document:\n```json\n{\n \"421312\": {\n \"my-key\": \"my-value\"\n }\n}\n```\nIt is possible to search for `my-value` despite not knowing the full path, but it isn't possible to search for all documents containing a key `my-key`.\n\n\n\n### Mode\n\nThe `mode` describes how Quickwit should behave when it receives a field that is not defined in the field mapping.\n\nQuickwit offers you three different modes:\n- `dynamic` (default value): unmapped fields are gathered by Quickwit and handled as defined in the `dynamic_mapping` parameter.\n- `lenient`: unmapped fields are dismissed by Quickwit.\n- `strict`: if a document contains a field that is not mapped, quickwit will dismiss it, and count it as an error.\n\n#### Dynamic Mapping\n\n`dynamic` mode makes it possible to operate Quickwit in a schemaless manner, or with a partial schema.\nThe configuration of `dynamic` mode can be set via the `dynamic_mapping` parameter.\n`dynamic_mapping` offers the same configuration options as when configuring a `json` field. It defaults to:\n\n```yaml\nversion: 0.7\nindex_id: my-dynamic-index\ndoc_mapping:\n mode: dynamic\n dynamic_mapping:\n indexed: true\n stored: true\n tokenizer: raw\n record: basic\n expand_dots: true\n fast: true\n```\n\nWhen the `dynamic_mapping` is set as indexed (default), fields mapped through\ndynamic mode can be searched by targeting the path needed to access them from\nthe root of the JSON object.\n\nFor instance, in an entirely schemaless settings, a minimal index configuration could be:\n\n```yaml\nversion: 0.7\nindex_id: my-dynamic-index\ndoc_mapping:\n # If you have a timestamp field, it is important to tell quickwit about it.\n timestamp_field: unix_timestamp\n # mode: dynamic #< Commented out, as dynamic is the default mode.\n```\n\nWith such a simple configuration, we can index a complex document like the following:\n\n```json\n{\n \"endpoint\": \"/admin\",\n \"query_params\": {\n \"ctk\": \"e42bb897d\",\n \"page\": \"eeb\"\n },\n \"src\": {\n \"ip\": \"8.8.8.8\",\n \"port\": 53,\n },\n //...\n}\n```\n\nThe following queries are then valid, and match the document above.\n\n```bash\n// Fields can be searched simply.\nendpoint:/admin\n\n// Nested object can be queried by specifying a `.` separated\n// path from the root of the json object to the given field.\nquery_params.ctk:e42bb897d\n\n// numbers are searchable too\nsrc.port:53\n\n// and of course we can combine them with boolean operators.\nsrc.port:53 AND query_params.ctk:e42bb897d\n```\n\nThe stored primitive type inference is the [same as for JSON fields](#json-type).\n\n### Field name validation rules\n\nCurrently Quickwit only accepts field name that matches the following regular expression:\n`^[@$_\\-a-zA-Z][@$_/\\.\\-a-zA-Z0-9]{0,254}$`\n\nIn plain language:\n- it needs to have at least one character.\n- it can only contain uppercase and lowercase ASCII letters `[a-zA-Z]`, digits `[0-9]`, `.`, hyphens `-`, underscores `_`, slash `/`, at `@` and dollar `$` signs.\n- it must not start with a dot or a digit.\n- it must be different from Quickwit's reserved field mapping names `_source`, `_dynamic`, `_field_presence`.\n\n:::caution\nFor field names containing the `.` character, you will need to escape it when referencing them. Otherwise the `.` character will be interpreted as a JSON object property access. Because of this, it is recommended to avoid using field names containing the `.` character.\n:::\n\n### Behavior with null values or missing fields\n\nFields with `null` or missing fields in your JSON document will be silently ignored when indexing.\n\n## Indexing settings\n\nThis section describes indexing settings for a given index.\n\n| Variable | Description | Default value |\n| ------------- | ------------- | ------------- |\n| `commit_timeout_secs` | Maximum number of seconds before committing a split since its creation. | `60` |\n| `split_num_docs_target` | Target number of docs per split. | `10000000` |\n| `merge_policy` | Describes the strategy used to trigger split merge operations (see [Merge policies](#merge-policies) section below). |\n| `resources.heap_size` | Indexer heap size per source per index. | `2000000000` |\n| `docstore_compression_level` | Level of compression used by zstd for the docstore. Lower values may increase ingest speed, at the cost of index size | `8` |\n| `docstore_blocksize` | Size of blocks in the docstore, in bytes. Lower values may improve doc retrieval speed, at the cost of index size | `1000000` |\n\n:::note\n\nChoosing an appropriate commit timeout is critical. With a shorter commit timeout, ingested data is queryable faster. But the published splits will be smaller, increasing the overhead associated with [merges](#merge-policies). \n\nWhen decommissioning definitively an indexer node that received data through the ingest API (including the [Elastic bulk API](/docs/reference/es_compatible_api) and the OTEL [log](/docs/log-management/otel-service.md) and [trace](/docs/distributed-tracing/otel-service.md) services), we need to make sure that all the data that was persisted locally (Write Ahead Log) is indexed and committed. After receiving the termination signal, the Quickwit process waits for the indexing pipelines to finish processing this local data. This can take as long as the longest commit timeout of all indexes. Make sure that the termination grace period of the infrastructure supporting the Quickwit indexer nodes is long enough (e.g [`terminationGracePeriodSeconds`](https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/) in Kubernetes or [`stopTimeout`](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_definition_parameters.html) on AWS ECS).\n\n:::\n\n### Merge policies\n\nQuickwit makes it possible to define the strategy used to decide which splits should be merged together and when.\n\nQuickwit offers three different merge policies, each with their\nown set of parameters.\n\n#### \"Stable log\" merge policy\n\nThe stable log merge policy attempts to minimize write amplification AND keep time-pruning power as high as possible, by merging splits with a similar size, and with a close time span.\n\nQuickwit's default merge policy is the `stable_log` merge policy\nwith the following parameters:\n\n```yaml\nversion: 0.7\nindex_id: \"hdfs\"\n# ...\nindexing_settings:\n merge_policy:\n type: \"stable_log\"\n min_level_num_docs: 100000\n merge_factor: 10\n max_merge_factor: 12\n maturation_period: 48h\n```\n\n\n| Variable | Description | Default value |\n| ------------- | ------------- | ------------- |\n| `merge_factor` | *(advanced)* Number of splits to merge together in a single merge operation. | `10` |\n| `max_merge_factor` | *(advanced)* Maximum number of splits that can be merged together in a single merge operation. | `12` |\n| `min_level_num_docs` | *(advanced)* Number of docs below which all splits are considered as belonging to the same level. | `100000` |\n| `maturation_period` | Duration after which a split is considered mature, and won't be considered for merges anymore. May impact the completion time of pending delete tasks. | `48h` |\n\n#### \"Limit Merge\" merge policy\n\n*The limit merge policy is considered advanced*.\n\nThe limit merge policy simply limits write amplification by setting an upperbound\nof the number of merge operation a split should undergo.\n\n\n```yaml\nversion: 0.7\nindex_id: \"hdfs\"\n# ...\nindexing_settings:\n merge_policy:\n type: \"limit_merge\"\n max_merge_ops: 5\n merge_factor: 10\n max_merge_factor: 12\n maturation_period: 48h\n```\n\n\n| Variable | Description | Default value |\n| ------------- | ------------- | ------------- |\n| `max_merge_ops` | Maximum number of merges that a given split should undergo. | `4` |\n| `merge_factor` | *(advanced)* Number of splits to merge together in a single merge operation. | `10` |\n| `max_merge_factor` | *(advanced)* Maximum number of splits that can be merged together in a single merge operation. | `12` |\n| `maturation_period` | Duration after which a split is considered mature, and won't be considered for merges anymore. May impact the completion time of pending delete tasks. | `48h` |\n\n#### No merge\n\nThe `no_merge` merge policy entirely disables merging.\n\n:::caution\nThis setting is not recommended. Merges are necessary to reduce the number of splits, and hence improve search performances.\n:::\n\n```yaml\nversion: 0.7\nindex_id: \"hdfs\"\nindexing_settings:\n merge_policy:\n type: \"no_merge\"\n```\n\n\n\n### Indexer memory usage\n\nIndexer works with a default heap of 2 GiB of memory. This does not directly reflect the overall memory usage, but doubling this value should give a fair approximation.\n\n\n## Search settings\n\nThis section describes search settings for a given index.\n\n| Variable | Description | Default value |\n| ------------- | ------------- | ------------- |\n| `default_search_fields` | Default list of fields that will be used for search. The field names in this list may be declared explicitly in the schema, or may refer to a field captured by the dynamic mode. | `None` |\n\n## Retention policy\n\nThis section describes how Quickwit manages data retention. In Quickwit, the retention policy manager drops data on a split basis as opposed to individually dropping documents. Splits are evaluated based on their `time_range` which is derived from the index timestamp field specified in the (`doc_mapping.timestamp_field`) settings. Using this setting, the retention policy will delete a split when `now() - split.time_range.end >= retention_policy.period`\n\n```yaml\nversion: 0.7\nindex_id: hdfs\n# ...\nretention:\n period: 90 days\n schedule: daily\n```\n\n| Variable | Description | Default value |\n| ------------- | ------------- | ------------- |\n| `period` | Duration after which splits are dropped, expressed in a human-readable way (`1 day`, `2 hours`, `a week`, ...). | required |\n| `schedule` | Frequency at which the retention policy is evaluated and applied, expressed as a cron expression (`0 0 * * * *`) or human-readable form (`hourly`, `daily`, `weekly`, `monthly`, `yearly`). | `hourly` |\n\n\n`period` is specified as set of time spans. Each time span is an integer followed by a unit suffix like: `2 days 3h 24min`. The supported units are:\n - `nsec`, `ns` -- nanoseconds\n - `usec`, `us` -- microseconds\n - `msec`, `ms` -- milliseconds\n - `seconds`, `second`, `sec`, `s`\n - `minutes`, `minute`, `min`, `m`\n - `hours`, `hour`, `hr`, `h`\n - `days`, `day`, `d`\n - `weeks`, `week`, `w`\n - `months`, `month`, `M` -- a month is defined as `30.44 days`\n - `years`, `year`, `y` -- a year is defined as `365.25 days`\n" }, { "path": "docs/configuration/index.md", "content": "---\ntitle: Configuration Reference\n---\n\nimport DocCardList from '@theme/DocCardList';\n\n\n\n\n" }, { "path": "docs/configuration/lambda-config.md", "content": "---\ntitle: Lambda configuration\nsidebar_position: 6\n---\n\nQuickwit supports offloading leaf search operations to AWS Lambda for horizontal scaling. When the local search queue becomes saturated, overflow splits are automatically sent to Lambda functions for processing.\n\n:::note\nLambda offloading is currently only supported on AWS.\n:::\n\n## How it works\n\nLambda offloading is **only active when a `lambda` configuration section is present** under `searcher` in your node configuration. When configured:\n\n1. Quickwit monitors the local search queue depth\n2. When pending searches exceed the `offload_threshold`, new splits are sent to Lambda instead of being queued locally\n3. Lambda returns per-split search results that are cached and merged with local results\n\nThis allows Quickwit to handle traffic spikes without provisioning additional searcher nodes.\n\n## Startup validation\n\nWhen a `lambda` configuration is defined, Quickwit performs a **dry run invocation** at startup to verify that:\n- The Lambda function exists\n- The function version matches the embedded binary\n- The invoker has permission to call the function\n\nIf this validation fails, **Quickwit will fail to start**. This ensures that Lambda offloading works correctly before the node begins serving traffic.\n\n## Configuration\n\nAdd a `lambda` section under `searcher` in your node configuration:\n\n```yaml\nsearcher:\n lambda:\n offload_threshold: 100\n auto_deploy:\n execution_role_arn: arn:aws:iam::123456789012:role/quickwit-lambda-role\n memory_size: 5 GiB\n invocation_timeout_secs: 15\n```\n\n### Lambda configuration options\n\n| Property | Description | Default value |\n| --- | --- | --- |\n| `function_name` | Name of the AWS Lambda function to invoke. | `quickwit-lambda-search` |\n| `max_splits_per_invocation` | Maximum number of splits to send in a single Lambda invocation. Must be at least 1. | `10` |\n| `offload_threshold` | Number of pending local searches before offloading to Lambda. A value of `0` offloads everything to Lambda. | `100` |\n| `auto_deploy` | Auto-deployment configuration. If set, Quickwit automatically deploys or updates the Lambda function at startup. | (none) |\n\n### Auto-deploy configuration options\n\n| Property | Description | Default value |\n| --- | --- | --- |\n| `execution_role_arn` | **Required.** IAM role ARN for the Lambda function's execution role. | |\n| `memory_size` | Memory allocated to the Lambda function. More memory provides more CPU. | `5 GiB` |\n| `invocation_timeout_secs` | Timeout for Lambda invocations in seconds. | `15` |\n\n## Deployment options\n\n### Automatic deployment (recommended)\n\nWith `auto_deploy` configured, Quickwit automatically:\n1. Creates the Lambda function if it doesn't exist\n2. Updates the function code if the embedded binary has changed\n3. Publishes a new version with a unique identifier\n4. Garbage collects old versions (keeps current + 5 most recent)\n\nThis is the recommended approach as it ensures the Lambda function always matches the Quickwit binary version.\n\n### Manual deployment\n\nYou can deploy the Lambda function manually without `auto_deploy`:\n1. Download the Lambda zip from [GitHub releases](https://github.com/quickwit-oss/quickwit/releases)\n2. Create or update the Lambda function using AWS CLI, Terraform, or the AWS Console\n3. Publish a version with description format `quickwit_{version}_{sha256}_{timeout}_{deploy_config}\"` (e.g., `quickwit_0_8_0_fa940f44_5120_60s_6c3b2`)\n\nThe description must match the format Quickwit expects, or it won't find the function version.\n\n## IAM permissions\n\n### Permissions for the Quickwit node\n\nThe IAM role or user running Quickwit needs the following permissions to invoke Lambda:\n\n```json\n{\n \"Version\": \"2012-10-17\",\n \"Statement\": [\n {\n \"Effect\": \"Allow\",\n \"Action\": [\n \"lambda:InvokeFunction\"\n ],\n \"Resource\": \"arn:aws:lambda:*:*:function:quickwit-lambda-search:*\"\n }\n ]\n}\n```\n\nIf using `auto_deploy`, additional permissions are required for deployment:\n\n```json\n{\n \"Version\": \"2012-10-17\",\n \"Statement\": [\n {\n \"Effect\": \"Allow\",\n \"Action\": [\n \"lambda:CreateFunction\",\n \"lambda:GetFunction\",\n \"lambda:UpdateFunctionCode\",\n \"lambda:PublishVersion\",\n \"lambda:ListVersionsByFunction\",\n \"lambda:DeleteFunction\"\n ],\n \"Resource\": \"arn:aws:lambda:*:*:function:quickwit-lambda-search\"\n },\n {\n \"Effect\": \"Allow\",\n \"Action\": \"iam:PassRole\",\n \"Resource\": \"arn:aws:iam::*:role/quickwit-lambda-role\",\n \"Condition\": {\n \"StringEquals\": {\n \"iam:PassedToService\": \"lambda.amazonaws.com\"\n }\n }\n }\n ]\n}\n```\n\n### Lambda execution role\n\nThe Lambda function requires an execution role with S3 read access to your index data.\n\nExample policy:\n\n```json\n{\n \"Version\": \"2012-10-17\",\n \"Statement\": [\n {\n \"Effect\": \"Allow\",\n \"Action\": \"s3:GetObject\",\n \"Resource\": \"arn:aws:s3:::your-index-bucket/*\"\n }\n ]\n}\n```\n\nThe execution role must also have a trust policy allowing Lambda to assume it:\n\n```json\n{\n \"Version\": \"2012-10-17\",\n \"Statement\": [\n {\n \"Effect\": \"Allow\",\n \"Principal\": {\n \"Service\": \"lambda.amazonaws.com\"\n },\n \"Action\": \"sts:AssumeRole\"\n }\n ]\n}\n```\n\n## CloudWatch logging\n\nThe Lambda function emits structured logs (JSON) to stdout. To have these logs captured by CloudWatch, add the following iam permissions to the Lambda execution role:\n\n```json\n{\n \"Version\": \"2012-10-17\",\n \"Statement\": [\n {\n \"Effect\": \"Allow\",\n \"Action\": [\n \"logs:CreateLogGroup\",\n \"logs:CreateLogStream\",\n \"logs:PutLogEvents\"\n ],\n \"Resource\": \"arn:aws:logs:*:*:*\"\n }\n ]\n}\n```\n\nNo additional configuration is needed on the Quickwit side.\n\n## Versioning\n\nQuickwit uses content-based versioning for Lambda:\n- A SHA256 hash of the Lambda binary is computed at build time\n- This hash is embedded in the Lambda function description as `quickwit:{version}-{sha256_short}`\n- When Quickwit starts, it searches for a version matching this description\n- Different Quickwit builds with the same Lambda binary share the same Lambda version\n- Updating the Lambda binary automatically triggers a new deployment\n\n## Example configuration\n\n\nMinimal configuration (with auto-deployment):\n\n```yaml\nsearcher:\n lambda:\n auto_deploy:\n execution_role_arn: arn:aws:iam::123456789012:role/quickwit-lambda-role\n```\n\n\nFull configuration (auto-deployment):\n\n```yaml\nsearcher:\n lambda:\n function_name: quickwit-lambda-search\n max_splits_per_invocation: 10\n offload_threshold: 10\n auto_deploy:\n execution_role_arn: arn:aws:iam::123456789012:role/quickwit-lambda-role\n memory_size: 5 GiB\n invocation_timeout_secs: 15\n```\n\nAggressive offloading (send everything to Lambda):\n\n```yaml\nsearcher:\n lambda:\n function_name: quickwit-lambda-search\n offload_threshold: 0\n auto_deploy:\n execution_role_arn: arn:aws:iam::123456789012:role/quickwit-lambda-role\n```\n" }, { "path": "docs/configuration/metastore-config.md", "content": "---\ntitle: Metastore configuration\nsidebar_position: 4\n---\n\nQuickwit needs a place to store meta-information about its indexes.\n\nFor instance:\n\n- The index configuration.\n- Meta-information about its splits. For instance, their IDs, the number of documents they contain, their sizes, their min/max timestamp, and the set of tags present in the split.\n- The different sources checkpoints.\n- Some extra information such as the index creation time.\n\nThe metastore is entirely defined by a single URI. One can set it by editing the `metastore_uri` parameter of the [node configuration file](./node-config.md) (often named `quickwit.yaml`).\n\nCurrently, Quickwit offers two implementations:\n\n- **PostgreSQL**: recommended for distributed usage.\n- **File-backed implementation**.\n\n# PostgreSQL Metastore\n\nWe recommend the PostgreSQL metastore for any distributed usage.\n\nThe PostgreSQL metastore can be configured by setting a PostgreSQL URI in the `metastore_uri` parameter of the Quickwit configuration file. The URI takes the following format:\n\n```\npostgres://[user]:[password]@[host]:[port]/[dbname]\n```\n\nSome of those parameters can be omitted. The following PostgreSQL URIs are for instance valid:\n\n```\npostgres://localhost/mydb\npostgres://user@localhost\npostgres://user:secret@localhost\n```\n\nThe database has to be created in advance.\n\nOn its first execution, Quickwit will transparently create the necessary tables.\n\nLikewise, if you upgrade Quickwit to a version that includes some changes in the PostgreSQL schema, Quickwit will transparently operate the migration startup.\n\n# File-backed metastore\n\nFor convenience, Quickwit also makes it possible to store its metadata in files using a file-backed metastore. In that case, Quickwit will write one file per index.\n\nThe metastore is then configured by passing a [storage URI](storage-config#storage-uris) that will serve as the root of the metastore storage.\n\nThe metadata file associated with a given index will then be stored under\n\n `[storage_uri]/[index_id]/metastore.json`\n\nFor the moment, Quickwit supports two types of storage types:\n\n- a local file system URI (e.g., `file:///opt/toto`). It is also valid to pass a file path directly (without file://). `/var/quickwit`. Relative paths will be resolved with respect to the current working directory.\n- S3-compatible storage URI (e.g., `s3://my-bucket/some-path`). See the [storage config](storage-config) documentation to configure S3 or S3-compatible storage providers.\n\n### Polling configuration\n\nBy default, the File-Backed Metastore is only read once when you start a Quickwit process (searcher, indexer, ...).\n\nYou can also configure it to poll the File-Backed Metastore periodically to keep a fresh view of it. This is useful for a Searcher instance that needs to be aware of new splits published by an Indexer running in parallel.\n\nTo configure the polling interval (in seconds), add a URI fragment to the storage URI as follows: `s3://quickwit/my-indexes#polling_interval=30s`\n\n:::note\nThe polling interval can be configured in seconds only; other units, such as minutes or hours, are not supported.\n:::\n\n:::tip\nAmazon S3 charges $0.0004 per 1000 GET requests. Polling a metastore every 30 seconds costs $0.04 per month and index.\n:::\n\n### Examples\n\nThe following file-backed metastore URIs for instance are valid:\n\n```markdown\ns3://my-indexes\ns3://quickwit/my-indexes\ns3://quickwit/my-indexes#polling_interval=30s\nfile:///local/indices\nfile:///local/indices#polling_interval=30s\n/local/indices\n./quickwit-metastores\n```\n\n:::caution\nThe file-backed metastore does not support multiple instances running at the same time because it does not implement any locking mechanism to prevent concurrent writes from overwriting each other. Ensure that only one file-backed metastore instance is running at all times.\n:::\n" }, { "path": "docs/configuration/node-config.md", "content": "---\ntitle: Node configuration\nsidebar_position: 1\n---\n\nThe node configuration allows you to customize and optimize the settings for individual nodes in your cluster. It is divided into several sections:\n\n- Common configuration settings: shared top-level properties\n- Storage settings: defined in the [storage](#storage-configuration) section\n- Metastore settings: defined in the [metastore](#metastore-configuration) section\n- Ingest settings: defined in the [ingest_api](#ingest-api-configuration) section\n- Indexer settings: defined in the [indexer](#indexer-configuration) section\n- Searcher settings: defined in the [searcher](#searcher-configuration) section\n- Jaeger settings: defined in the [jaeger](#jaeger-configuration) section\n\nA commented example is available here: [quickwit.yaml](https://github.com/quickwit-oss/quickwit/blob/main/config/quickwit.yaml).\n\n## Common configuration\n\n| Property | Description | Env variable | Default value |\n| --- | --- | --- | --- |\n| `version` | Config file version. `0.7` is the only available value with a retro compatibility on `0.5` and `0.4`. | | |\n| `cluster_id` | Unique identifier of the cluster the node will be joining. Clusters sharing the same network should use distinct cluster IDs.| `QW_CLUSTER_ID` | `quickwit-default-cluster` |\n| `node_id` | Unique identifier of the node. It must be distinct from the node IDs of its cluster peers. Defaults to the instance's short hostname if not set. | `QW_NODE_ID` | short hostname |\n| `enabled_services` | Enabled services (control_plane, indexer, janitor, metastore, searcher) | `QW_ENABLED_SERVICES` | all services |\n| `listen_address` | The IP address or hostname that Quickwit service binds to for starting REST and GRPC server and connecting this node to other nodes. By default, Quickwit binds itself to 127.0.0.1 (localhost). This default is not valid when trying to form a cluster. | `QW_LISTEN_ADDRESS` | `127.0.0.1` |\n| `advertise_address` | IP address advertised by the node, i.e. the IP address that peer nodes should use to connect to the node for RPCs. | `QW_ADVERTISE_ADDRESS` | `listen_address` |\n| `gossip_listen_port` | The port which to listen for the Gossip cluster membership service (UDP). | `QW_GOSSIP_LISTEN_PORT` | `rest.listen_port` |\n| `grpc_listen_port` | The port on which gRPC services listen for traffic. | `QW_GRPC_LISTEN_PORT` | `rest.listen_port + 1` |\n| `peer_seeds` | List of IP addresses or hostnames used to bootstrap the cluster and discover the complete set of nodes. This list may contain the current node address and does not need to be exhaustive. If the list of peer seeds contains a host name, Quickwit will resolve it by querying the DNS every minute. On kubernetes for instance, it is a good practise to set it to a [headless service](https://kubernetes.io/docs/concepts/services-networking/service/#headless-services). | `QW_PEER_SEEDS` | |\n| `data_dir` | Path to directory where data (tmp data, splits kept for caching purpose) is persisted. This is mostly used in indexing. | `QW_DATA_DIR` | `./qwdata` |\n| `metastore_uri` | Metastore URI. Can be a local directory or `s3://my-bucket/indexes` or `postgres://username:password@localhost:5432/metastore`. [Learn more about the metastore configuration](metastore-config.md). | `QW_METASTORE_URI` | `{data_dir}/indexes` |\n| `default_index_root_uri` | Default index root URI that defines the location where index data (splits) is stored. The index URI is built following the scheme: `{default_index_root_uri}/{index-id}` | `QW_DEFAULT_INDEX_ROOT_URI` | `{data_dir}/indexes` |\n| environment variable only | Log level of Quickwit. Can be a direct log level, or a comma separated list of `module_name=level` | `RUST_LOG` | `info` |\n\n## REST configuration\n\nThis section contains the REST API configuration options.\n\n| Property | Description | Env variable | Default value |\n| --- | --- | --- | --- |\n| `listen_port` | The port on which the REST API listens for HTTP traffic. | `QW_REST_LISTEN_PORT` | `7280` |\n| `cors_allow_origins` | Configure the CORS origins which are allowed to access the API. [Read more](#configuring-cors-cross-origin-resource-sharing) | |\n| `extra_headers` | List of header names and values | | |\n\n### Configuring CORS (Cross-origin resource sharing)\n\nCORS (Cross-origin resource sharing) describes which address or origins can access the REST API from the browser.\nBy default, sharing resources cross-origin is not allowed.\n\nA wildcard, single origin, or multiple origins can be specified as part of the `cors_allow_origins` parameter:\n\n\nExample of a REST configuration:\n\n```yaml\nrest:\n listen_port: 1789\n extra_headers:\n x-header-1: header-value-1\n x-header-2: header-value-2\n cors_allow_origins: '*'\n\n# cors_allow_origins: https://my-hdfs-logs.domain.com # Optionally we can specify one domain\n# cors_allow_origins: # Or allow multiple origins\n# - https://my-hdfs-logs.domain.com\n# - https://my-hdfs.other-domain.com\n```\n\n## gRPC configuration\n\nThis section contains the configuration options for gRPC services and clients used for internal communication between nodes.\n\n| Property | Description | Env variable | Default value |\n| --- | --- | --- | --- |\n| `max_message_size` | The maximum size (in bytes) of messages exchanged by internal gRPC clients and services. | | `20 MiB` |\n\nExample of a gRPC configuration:\n\n```yaml\ngrpc:\n max_message_size: 30 MiB\n```\n\n:::warning\nWe advise changing the default value of 20 MiB only if you encounter the following error:\n`Error, message length too large: found 24732228 bytes, the limit is: 20971520 bytes.` In that case, increase `max_message_size` by increments of 10 MiB until the issue disappears. This is a temporary fix: the next version of Quickwit will rely exclusively on gRPC streaming endpoints and handle messages of any length.\n:::\n\n## Storage configuration\n\nPlease refer to the dedicated [storage configuration](storage-config) page to learn more about configuring Quickwit for various storage providers.\n\nHere are also some minimal examples of how to configure Quickwit with Amazon S3 or Alibaba OSS:\n\n```bash\nAWS_ACCESS_KEY_ID=\nAWS_SECRET_ACCESS_KEY=\n```\n\n*Amazon S3*\n\n```yaml\nstorage:\n s3:\n region: us-east-1\n```\n\n*Alibaba*\n\n```yaml\nstorage:\n s3:\n region: us-east-1\n endpoint: https://oss-us-east-1.aliyuncs.com\n```\n\n## Metastore configuration\n\nThis section may contain one configuration subsection per available metastore implementation. The specific configuration parameters for each implementation may vary. Currently, the available metastore implementations are:\n- File-backed\n- PostgreSQL\n\n### File-backed metastore configuration\n\nFile-backed metastore doesn't have any node level configuration. You can configure the poll interval [at the index level](./metastore-config.md#polling-configuration).\n\n### PostgreSQL metastore configuration\n\n| Property | Description | Default value |\n| --- | --- | --- |\n| `min_connections` | Minimum number of connections to maintain in the pool at all times. | `0` |\n| `max_connections` | Maximum number of connections to maintain in the pool. | `10` |\n| `acquire_connection_timeout` | Maximum amount of time to spend waiting for an available connection before aborting a query. | `10s` |\n| `idle_connection_timeout` | Maximum idle duration before closing individual connections. | `10min` |\n| `max_connection_lifetime` | Maximum lifetime of individual connections. | `30min` |\n\nExample of a metastore configuration for PostgreSQL in YAML format:\n\n```yaml\nmetastore:\n postgres:\n min_connections: 10\n max_connections: 50\n acquire_connection_timeout: 30s\n idle_connection_timeout: 1h\n max_connection_lifetime: 1d\n```\n\n## Indexer configuration\n\nThis section contains the configuration options for an indexer. The split store is documented in the [indexing document](../overview/concepts/indexing.md#split-store).\n\n| Property | Description | Default value |\n| --- | --- | --- |\n| `split_store_max_num_bytes` | Maximum size in bytes allowed in the split store. | `100G` |\n| `split_store_max_num_splits` | Maximum number of files allowed in the split store. | `1000` |\n| `max_concurrent_split_uploads` | Maximum number of concurrent split uploads allowed on the node. | `12` |\n| `merge_concurrency` | Maximum number of merge operations that can be executed on the node at one point in time. | `(2 x num threads available) / 3` |\n| `enable_otlp_endpoint` | If true, enables the OpenTelemetry exporter endpoint to ingest logs and traces via the OpenTelemetry Protocol (OTLP). | `false` |\n| `cpu_capacity` | Advisory parameter used by the control plane. The value can expressed be in threads (e.g. `2`) or in term of millicpus (`2000m`). The control plane will attempt to schedule indexing pipelines on the different nodes proportionally to the cpu capacity advertised by the indexer. It is NOT used as a limit. All pipelines will be scheduled regardless of whether the cluster has sufficient capacity or not. The control plane does not attempt to spread the work equally when the load is well below the `cpu_capacity`. Users who need a balanced load on all of their indexer nodes can set the `cpu_capacity` to an arbitrarily low value as long as they keep it proportional to the number of threads available. | `num threads available` |\n| `enable_cooperative_indexing` | Enable sharing resources more efficiently when the number of indexes actively written to is significantly higher than the number of cores but might decrease the overall indexing throughput. | `false` |\n\nExample:\n\n```yaml\nindexer:\n split_store_max_num_bytes: 100G\n split_store_max_num_splits: 1000\n max_concurrent_split_uploads: 12\n enable_otlp_endpoint: true\n```\n\n## Ingest API configuration\n\n| Property | Description | Default value |\n| --- | --- | --- |\n| `max_queue_memory_usage` | Maximum size in bytes of the in-memory Ingest queue. | `2GiB` |\n| `max_queue_disk_usage` | Maximum disk-space in bytes taken by the Ingest queue. The minimum size is at least `256M` and be at least `max_queue_memory_usage`. | `4GiB` |\n| `content_length_limit` | Maximum payload size uncompressed. Increasing this is discouraged, use a [file source](../ingest-data/sqs-files.md) instead. | `10MiB` |\n| `grpc_compression_algorithm` | Compression algorithm (`gzip` or `zstd`) to use for gRPC traffic between nodes for the ingest service | `None` |\n\nExample:\n\n```yaml\ningest_api:\n max_queue_memory_usage: 2GiB\n max_queue_disk_usage: 4GiB\n content_length_limit: 10MiB\n grpc_compression_algorithm: zstd\n```\n\n## Searcher configuration\n\nThis section contains the configuration options for a Searcher.\n\n| Property | Description | Default value |\n| --- | --- | --- |\n| `aggregation_memory_limit` | Controls the maximum amount of memory that can be used for aggregations before aborting. This limit is per searcher node. A node may run concurrent queries, which share the limit. The first query that will hit the limit will be aborted and frees its memory. It is used to prevent excessive memory usage during the aggregation phase, which can lead to performance degradation or crashes. | `500M`|\n| `aggregation_bucket_limit` | Determines the maximum number of buckets returned to the client. | `65000` |\n| `fast_field_cache_capacity` | Fast field in memory cache capacity on a Searcher. If your filter by dates, run aggregations, range queries, or even for tracing, it might worth increasing this parameter. The [metrics](../reference/metrics.md) starting by `quickwit_cache_fastfields_cache` can help you make an informed choice when setting this value. | `1G` |\n| `split_footer_cache_capacity` | Split footer in memory cache (it is essentially the hotcache) capacity on a Searcher.| `500M` |\n| `partial_request_cache_capacity` | Partial request in memory cache capacity on a Searcher. Cache intermediate state for a request, possibly making subsequent requests faster. It can be disabled by setting the size to `0`. | `64M` |\n| `max_num_concurrent_split_searches` | Maximum number of concurrent split search requests running on a Searcher. | `100` |\n| `split_cache` | Searcher split cache configuration options defined in the section below. Cache disabled if unspecified. | |\n| `request_timeout_secs` | The time before a search request is cancelled. This should match the timeout of the stack calling into quickwit if there is one set. | `30` |\n\n### Searcher split cache configuration\n\nThis section contains the configuration options for the on-disk searcher split cache. Files are stored in the data directory under `searcher-split-cache/`.\n\n| Property | Description | Default value |\n| --- | --- | --- |\n| `max_num_bytes` | Maximum disk size in bytes allowed in the split cache. Can be exceeded by the size of one split. | |\n| `max_num_splits` | Maximum number of splits allowed in the split cache. | `10000` |\n| `num_concurrent_downloads` | Maximum number of concurrent download of splits. | `1` |\n\n\nExample:\n\n```yaml\nsearcher:\n fast_field_cache_capacity: 1G\n split_footer_cache_capacity: 500M\n partial_request_cache_capacity: 64M\n split_cache:\n max_num_bytes: 1G\n max_num_splits: 10000\n num_concurrent_downloads: 1\n```\n\n## Jaeger configuration\n\n| Property | Description | Default value |\n| --- | --- | --- |\n| `enable_endpoint` | If true, enables the gRPC endpoint that allows the Jaeger Query Service to connect and retrieve traces. | `false` |\n\nExample:\n\n```yaml\njaeger:\n enable_endpoint: true\n```\n\n\n## Using environment variables in the configuration\n\nYou can use environment variable references in the config file to set values that need to be configurable during deployment. To do this, use:\n\n`${VAR_NAME}`\n\nwhere `VAR_NAME` is the name of the environment variable.\n\nEach variable reference is replaced at startup by the value of the environment variable. The replacement is case-sensitive and occurs before the configuration file is parsed. Referencing undefined variables throws an error unless you specify a default value or custom error text.\n\nTo specify a default value, use:\n\n`${VAR_NAME:-default_value}`\n\nwhere `default_value` is the value to use if the environment variable is unset.\n\n```\n: ${VAR_NAME}\nor\n: ${VAR_NAME:-default value}\n```\n\nFor example:\n\n```bash\nexport QW_LISTEN_ADDRESS=0.0.0.0\n```\n\n```yaml\n# config.yaml\nversion: 0.7\ncluster_id: quickwit-cluster\nnode_id: my-unique-node-id\nlisten_address: ${QW_LISTEN_ADDRESS}\nrest:\n listen_port: ${QW_LISTEN_PORT:-1111}\n```\n\nWill be interpreted by Quickwit as:\n\n```yaml\nversion: 0.7\ncluster_id: quickwit-cluster\nnode_id: my-unique-node-id\nlisten_address: 0.0.0.0\nrest:\n listen_port: 1111\n```\n" }, { "path": "docs/configuration/ports-config.md", "content": "---\ntitle: Ports configuration\nsidebar_position: 6\n---\n\nWhen starting a quickwit search server, one important parameter that can be configured is\nthe `rest.listen_port` (defaults to :7280).\n\nInternally, Quickwit will, in fact, use three sockets. The ports of these three sockets\ncannot be configured independently at the moment.\nThe ports used are computed relative to the `rest.listen_port` port, as follows.\n\n\n| Service | Port used | Protocol | Default |\n|-------------------------------|---------------------------|----------|-----------|\n| Http server with the rest api | `${rest.listen_port}` | TCP | 7280 |\n| Cluster membership | `${rest.listen_port}` | UDP | 7280 |\n| GRPC service | `${rest.listen_port} + 1` | TCP | 7281 |\n\nIt is not possible for the moment to configure these ports independently.\n\n\nIn order to form a cluster, you will also need to define a `peer_seeds` parameter.\nThe following addresses are valid peer seed addresses:\n\n| Type | Example without port | Example with port |\n|--------------|--------------|---------------------------|\n| IPv4 | 172.1.0.12 | 172.1.0.12:7180 |\n| IPv6 | 2001:0db8:85a3:0000:0000:8a2e:0370:7334 | [2001:0db8:85a3:0000:0000:8a2e:0370:7334:7180]:7280 |\n| hostname | node3 | node3:7180 |\n\nIf no port is specified in a peer node address, a Quickwit node will assume the peer is using the same port as itself.\n" }, { "path": "docs/configuration/source-config.md", "content": "---\ntitle: Source configuration\nsidebar_position: 5\n---\n\nQuickwit can insert data into an index from one or multiple sources.\nA source can be added after index creation using the [CLI command](../reference/cli.md#source) `quickwit source create`.\nIt can also be enabled or disabled with the `quickwit source enable/disable` subcommands.\n\nA source is declared using an object called source config, which defines the source's settings. It consists of multiple parameters:\n\n- source ID\n- source type\n- source parameters\n- input_format\n- maximum number of pipelines per indexer (optional)\n- desired number of pipelines (optional)\n- transform parameters (optional)\n\n## Source ID\n\nThe source ID is a string that uniquely identifies the source within an index. It may only contain uppercase or lowercase ASCII letters, digits, hyphens (`-`), and underscores (`_`). Finally, it must start with a letter and contain at least 3 characters but no more than 255.\n\n## Source type\n\nThe source type designates the kind of source being configured. As of version 0.5, available source types are `ingest-api`, `kafka`, `kinesis`, and `pulsar`. The `file` type is also supported but only for local ingestion from [the CLI](/docs/reference/cli.md#tool-local-ingest).\n\n## Source parameters\n\nThe source parameters indicate how to connect to a data store and are specific to the source type.\n\n### File source\n\nA file source reads data from files containing JSON objects separated by newlines (NDJSON). Gzip compression is supported provided that the file name ends with the `.gz` suffix.\n\n#### Ingest a single file (CLI only)\n\nTo ingest a specific file, run the indexing directly in an adhoc CLI process with:\n\n```bash\n./quickwit tool local-ingest --index --input-path \n```\n\nBoth local and object files are supported, provided that the environment is configured with the appropriate permissions. A tutorial is available [here](/docs/ingest-data/ingest-local-file.md).\n\n#### Notification based file ingestion (beta)\n\nQuickwit can automatically ingest all new files that are uploaded to an S3 bucket. This requires creating and configuring an [SQS notification queue](https://docs.aws.amazon.com/AmazonS3/latest/userguide/ways-to-add-notification-config-to-bucket.html). A complete example can be found [in this tutorial](/docs/ingest-data/sqs-files.md).\n\n\nThe `notifications` parameter takes an array of notification settings. Currently one notifier can be configured per source and only the SQS notification `type` is supported.\n\nRequired fields for the SQS `notifications` parameter items:\n- `type`: `sqs`\n- `queue_url`: complete URL of the SQS queue (e.g `https://sqs.us-east-1.amazonaws.com/123456789012/queue-name`)\n- `message_type`: format of the message payload, either\n - `s3_notification`: an [S3 event notification](https://docs.aws.amazon.com/AmazonS3/latest/userguide/EventNotifications.html)\n - `raw_uri`: a message containing just the file object URI (e.g. `s3://mybucket/mykey`)\n - `deduplication_window_duration_sec`: maximum duration for which ingested files checkpoints are kept (default 3600)\n - `deduplication_window_max_messages`: maximum number of ingested file checkpoints kept (default 100k)\n - `deduplication_cleanup_interval_secs`: frequency at which outdated file checkpoints are cleaned up\n\n*Adding a file source with SQS notifications to an index with the [CLI](../reference/cli.md#source)*\n\n```bash\ncat << EOF > source-config.yaml\nversion: 0.8\nsource_id: my-sqs-file-source\nsource_type: file\nnum_pipelines: 2\nparams:\n notifications:\n - type: sqs\n queue_url: https://sqs.us-east-1.amazonaws.com/123456789012/queue-name\n message_type: s3_notification\nEOF\n./quickwit source create --index my-index --source-config source-config.yaml\n```\n\n:::note\n\n- Quickwit does not automatically delete the source files after a successful ingestion. You can use [S3 object expiration](https://docs.aws.amazon.com/AmazonS3/latest/userguide/lifecycle-expire-general-considerations.html) to configure how long they should be retained in the bucket.\n- Configure the notification to only forward events of type `s3:ObjectCreated:*`. Other events are acknowledged by the source without further processing and an warning is logged.\n- We strongly recommend using a [dead letter queue](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dead-letter-queues.html) to receive all messages that couldn't be processed by the file source. A `maxReceiveCount` of 5 is a good default value. Here are some common situations where the notification message ends up in the dead letter queue:\n - the notification message could not be parsed (e.g it is not a valid S3 notification)\n - the file was not found\n - the file is corrupted (e.g unexpected compression)\n- AWS S3 notifications and AWS SQS provide \"at least once\" delivery guaranties. To avoid duplicates, the file source includes a mechanism that prevents the same file from being ingested twice. It works by storing checkpoints in the metastore that track the indexing progress for each file. You can decrease `deduplication_window_*` or increase `deduplication_cleanup_interval_secs` to reduce the load on the metastore.\n\n:::\n\n### Ingest API source\n\nAn ingest API source reads data from the [Ingest API](/docs/reference/rest-api.md#ingest-data-into-an-index). This source is automatically created at the index creation and cannot be deleted nor disabled.\n\n### Kafka source\n\nA Kafka source reads data from a Kafka stream. Each message in the stream must hold a JSON object.\n\nA tutorial is available [here](/docs/ingest-data/kafka.md).\n\n#### Kafka source parameters\n\nThe Kafka source consumes a `topic` using the client library [librdkafka](https://github.com/edenhill/librdkafka) and forwards the key-value pairs carried by the parameter `client_params` to the underlying librdkafka consumer. Common `client_params` options are bootstrap servers (`bootstrap.servers`), or security protocol (`security.protocol`). Please, refer to [Kafka](https://kafka.apache.org/documentation/#consumerconfigs) and [librdkafka](https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md) documentation pages for more advanced options.\n\n| Property | Description | Default value |\n| --- | --- | --- |\n| `topic` | Name of the topic to consume. | required |\n| `client_log_level` | librdkafka client log level. Possible values are: debug, info, warn, error. | `info` |\n| `client_params` | librdkafka client configuration parameters. | `{}` |\n| `enable_backfill_mode` | Backfill mode stops the source after reaching the end of the topic. | `false` |\n\n**Kafka client parameters**\n\n- `bootstrap.servers`\nComma-separated list of host and port pairs that are the addresses of a subset of the Kafka brokers in the Kafka cluster.\n\n- `auto.offset.reset`\nDefines the behavior of the source when consuming a partition for which there is no initial offset saved in the checkpoint. `earliest` consumes from the beginning of the partition, whereas `latest` (default) consumes from the end.\n\n- `enable.auto.commit`\nThis setting is ignored because the Kafka source manages commit offsets internally using the [checkpoint API](../overview/concepts/indexing.md#checkpoint) and forces auto-commits to be disabled.\n\n- `group.id`\nKafka-based distributed indexing relies on consumer groups. Unless overridden in the client parameters, the default group ID assigned to each consumer managed by the source is `quickwit-{index_uid}-{source_id}`.\n\n- `max.poll.interval.ms`\nShort max poll interval durations may cause a source to crash when back pressure from the indexer occurs. Therefore, Quickwit recommends using the default value of `300000` (5 minutes).\n\n*Adding a Kafka source to an index with the [CLI](../reference/cli.md#source)*\n\n```bash\ncat << EOF > source-config.yaml\nversion: 0.8\nsource_id: my-kafka-source\nsource_type: kafka\nnum_pipelines: 2\nparams:\n topic: my-topic\n client_params:\n bootstrap.servers: localhost:9092\n security.protocol: SSL\nEOF\n./quickwit source create --index my-index --source-config source-config.yaml\n```\n\n### Kinesis source\n\nA Kinesis source reads data from an [Amazon Kinesis](https://aws.amazon.com/kinesis/) stream. Each message in the stream must hold a JSON object.\n\nA tutorial is available [here](/docs/ingest-data/kinesis.md).\n\n**Kinesis source parameters**\n\nThe Kinesis source consumes a stream identified by a `stream_name` and a `region`.\n\n| Property | Description | Default value |\n| --- | --- | --- |\n| `stream_name` | Name of the stream to consume. | required |\n| `region` | The AWS region of the stream. Mutually exclusive with `endpoint`. | `us-east-1` |\n| `endpoint` | Custom endpoint for use with AWS-compatible Kinesis service. Mutually exclusive with `region`. | optional |\n\nIf no region is specified, Quickwit will attempt to find one in multiple other locations and with the following order of precedence:\n\n1. Environment variables (`AWS_REGION` then `AWS_DEFAULT_REGION`)\n\n2. Config file, typically located at `~/.aws/config` or otherwise specified by the `AWS_CONFIG_FILE` environment variable if set and not empty.\n\n3. Amazon EC2 instance metadata service determining the region of the currently running Amazon EC2 instance.\n\n4. Default value: `us-east-1`\n\n*Adding a Kinesis source to an index with the [CLI](../reference/cli.md#source)*\n\n```bash\ncat << EOF > source-config.yaml\nversion: 0.7\nsource_id: my-kinesis-source\nsource_type: kinesis\nparams:\n stream_name: my-stream\nEOF\nquickwit source create --index my-index --source-config source-config.yaml\n```\n\n### Pulsar source\n\nA Puslar source reads data from one or several Pulsar topics. Each message in topic(s) must hold a JSON object.\n\nA tutorial is available [here](/docs/ingest-data/pulsar.md).\n\n**Pulsar source parameters**\n\nThe Pulsar source consumes `topics` using the client library [pulsar-rs](https://github.com/streamnative/pulsar-rs).\n\n| Property | Description | Default value |\n| --- | --- | --- |\n| `topics` | List of topics to consume. | required |\n| `address` | Pulsar URL (pulsar:// and pulsar+ssl://). | required |\n| `consumer_name` | The consumer name to register with the pulsar source. | `quickwit` |\n\n*Adding a Pulsar source to an index with the [CLI](../reference/cli.md#source)*\n\n```bash\ncat << EOF > source-config.yaml\nversion: 0.7\nsource_id: my-pulsar-source\nsource_type: pulsar\nparams:\n topics:\n - my-topic\n address: pulsar://localhost:6650\nEOF\n./quickwit source create --index my-index --source-config source-config.yaml\n```\n\n## Number of pipelines\n\nThe `num_pipelines` parameter is only available for distributed sources like Kafka, GCP PubSub, and Pulsar.\n\nIt defines the number of pipelines to run on a cluster for the source. The actual placement of these pipelines on the different indexer\nwill be decided by the control plane.\n\n:::info\n\nNote that distributing the indexing load of partitioned sources like Kafka is done by assigning the different partitions to different pipelines. As a result, it is important to ensure that the number of partitions is a multiple of `num_pipelines`.\n\nAlso, assuming you are only indexing a single Kafka source in your Quickwit cluster, you should set the number of pipelines to a multiple of the number of indexers. Finally, if your indexing throughput is high, you should provision between 2 and 4 vCPUs per pipeline.\n\nFor instance, assume you want to index a 60-partition topic, with each partition receiving a throughput of 10 MB/s. If you measured that Quickwit can index your data at a pace of 40MB/s per pipeline, a possible setting could be:\n- 5 indexers with 8 vCPUs each\n- 15 pipelines\n\nEach indexer will then be in charge of 3 pipelines, and each pipeline will cover 4 partitions.\n:::\n\n\n## Transform parameters\n\nFor all source types but the `ingest-api`, ingested documents can be transformed before being indexed using [Vector Remap Language (VRL)](https://vector.dev/docs/reference/vrl/) scripts.\n\n| Property | Description | Default value |\n| --- | --- | --- |\n| `script` | Source code of the VRL program executed to transform documents. | required |\n| `timezone` | Timezone used in the VRL program for date and time manipulations. It must be a valid name in the [TZ database](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones) | `UTC` |\n\n```yaml\n# Your source config here\n# ...\ntransform:\n script: |\n .message = downcase(string!(.message))\n .timestamp = now()\n del(.username)\n timezone: local\n```\n\n## Input format\n\nThe `input_format` parameter specifies the expected data format of the source. The formats currently supported are:\n- `json` (default)\n- `otlp_logs_json`\n- `otlp_logs_proto`\n- `otlp_traces_json`\n- `otlp_traces_proto`\n- `plain_text`\n\n*OTLP formats*\n\nWhen ingesting OTLP data into an OTLP logs or traces index with a source other than the native OTEL endpoints, use this parameter to specify whether the exported logs or traces will be serialized in JSON or Protobuf. When possible, prefer the latter, which is a more compact encoding.\n\n*Plaint text format*\n\nUse this parameter for unstructured text data. Internally, Quickwit can only index JSON data. To allow the ingestion of plain text documents, Quickwit transform them on the fly into JSON objects of the following form: `{\"plain_text\": \"\"}`. Then, they can be optionally transformed into more complex documents using a VRL script. (see [transform feature](#transform-parameters)).\n\nThe following is an example of how one could parse and transform a CSV dataset containing a list of users described by 3 attributes: first name, last name, and age.\n\n```yaml\n# Your source config here\n# ...\ninput_format: plain_text\ntransform:\n script: |\n user = parse_csv!(.plain_text)\n .first_name = user[0]\n .last_name = user[1]\n .age = to_int!(user[2])\n del(.plain_text)\n```\n\n## Enabling/disabling a source from an index\n\nA source can be enabled or disabled from an index using the [CLI command](../reference/cli.md) `quickwit source enable` or `quickwit source disable`:\n\n```bash\nquickwit source disable --index my-index --source my-source\n```\n\nA source is enabled by default. When disabling a source, the related indexing pipelines will be shut down on each relevant indexer and indexing for this source will be paused.\n\n## Deleting a source from an index\n\nA source can be removed from an index using the [CLI command](../reference/cli.md) `quickwit source delete`:\n\n```bash\nquickwit source delete --index my-index --source my-source\n```\n\nWhen deleting a source, the checkpoint associated with the source is also removed.\n" }, { "path": "docs/configuration/storage-config.md", "content": "---\ntitle: Storage configuration\nsidebar_position: 2\n---\n\n## Supported Storage Providers\n\nQuickwit currently supports four types of storage providers:\n- Amazon S3 and S3-compatible (Garage, MinIO, ...)\n- Azure Blob Storage\n- Local file storage*\n- Google Cloud Storage (native API)\n\n## Storage URIs\n\nStorage URIs refer to different storage providers identified by a URI \"protocol\" or \"scheme\". Quickwit supports the following storage URI protocols:\n- `s3://` for Amazon S3 and S3-compatible\n- `azure://` for Azure Blob Storage\n- `file://` for local file systems\n- `gs://` for Google Cloud Storage\n\nIn general, you can use a storage URI or a file path anywhere you would intuitively expect a file path. For instance:\n- when setting the `index_uri` of an index to specify the storage provider and location;\n- when setting the `metastore_uri` in a node config to set up a file-backed metastore;\n- when passing a file path as a command line argument.\n\n### Local file storage URIs\n\nQuickwit interprets regular file paths as local file system URIs. Relative file paths are allowed and are resolved relatively to the current working directory (CWD). `~` can be used as a shortcut to refer to the user’s home directory. The following are valid local file system URIs:\n\n```markdown\n- /var/quickwit\n- file:///var/quickwit\n- /home/quickwit/data\n- ~/data\n- ./quickwit\n```\n\n:::caution\nWhen using the `file://` protocol, a third `/` is necessary to express an absolute path. For instance, the following URI `file://home/quickwit/` is interpreted as `./home/quickwit`\n:::\n\n## Storage configuration\n\nThis section contains one configuration subsection per storage provider. If a storage configuration parameter is not explicitly set, Quickwit relies on the default values provided by the storage provider SDKs ([Azure SDK for Rust](https://github.com/Azure/azure-sdk-for-rust), [AWS SDK for Rust](https://github.com/awslabs/aws-sdk-rust)).\n\n### S3 storage configuration\n\n| Property | Description | Default value |\n| --- | --- | --- |\n| `flavor` | The optional storage flavor to use. Available flavors are `digital_ocean`, `garage`, `gcs`, and `minio`. | |\n| `access_key_id` | The AWS access key ID. | |\n| `secret_access_key` | The AWS secret access key. | |\n| `region` | The AWS region to send requests to. | `us-east-1` (SDK default) |\n| `endpoint` | Custom endpoint for use with S3-compatible providers. | SDK default |\n| `force_path_style_access` | Disables [virtual-hosted–style](https://docs.aws.amazon.com/AmazonS3/latest/userguide/VirtualHosting.html) requests. Required by some S3-compatible providers (Ceph, MinIO). | `false` |\n| `disable_multi_object_delete` | Disables [Multi-Object Delete](https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObjects.html) requests. Required by some S3-compatible providers (GCS). | `false` |\n| `disable_multipart_upload` | Disables [multipart upload](https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html) of objects. Required by some S3-compatible providers (GCS). | `false` |\n\n:::warning\nHardcoding credentials into configuration files is not secure and strongly discouraged. Prefer the alternative authentication methods that your storage backend may provide.\n:::\n\n#### Environment variables\n\n| Env variable | Description |\n| --- | --- |\n| `QW_S3_ENDPOINT` | Custom S3 endpoint. |\n| `QW_S3_MAX_CONCURRENCY` | Limit the number of concurrent requests to S3 |\n\n#### Storage flavors\n\nStorage flavors ensure that Quickwit works correctly with storage providers that deviate from the S3 API by automatically configuring the appropriate settings. The available flavors are:\n- `digital_ocean`\n- `garage`\n- `gcs`\n- `minio`\n\n*Digital Ocean*\n\nThe Digital Ocean flavor (`digital_ocean`) forces path-style access and turns off multi-object delete requests.\n\n*Garage flavor*\n\nThe Garage flavor (`garage`) overrides the `region` parameter to `garage` and forces path-style access.\n\n*Google Cloud Storage*\n\nThe Google Cloud Storage flavor (`gcs`) turns off multi-object delete requests and multipart uploads.\n\n*MinIO flavor*\n\nThe MinIO flavor (`minio`) overrides the `region` parameter to `minio` and forces path-style access.\n\nExample of a storage configuration for Google Cloud Storage in YAML format:\n\n```yaml\nstorage:\n s3:\n flavor: gcs\n region: us-east1\n endpoint: https://storage.googleapis.com\n```\n\n### Azure storage configuration\n\n| Property | Description | Default value |\n| --- | --- | --- |\n| `account` | The Azure storage account name. | |\n| `access_key` | The Azure storage account access key. | |\n\n#### Environment variables\n\n| Env variable | Description |\n| --- | --- |\n| `QW_AZURE_STORAGE_ACCOUNT` | Azure Blob Storage account name. |\n| `QW_AZURE_STORAGE_ACCESS_KEY` | Azure Blob Storage account access key. |\n\nExample of a storage configuration for Azure in YAML format:\n\n```yaml\nstorage:\n azure:\n account: your-azure-account-name\n access_key: your-azure-access-key\n```\n\n## Storage configuration examples for various object storage providers\n\n### Garage\n\n[Garage](https://garagehq.deuxfleurs.fr/) is an open-source distributed object storage service tailored for self-hosting.\n\n```yaml\nstorage:\n s3:\n flavor: garage\n endpoint: http://127.0.0.1:3900\n```\n\n### MinIO\n\n[MinIO](https://min.io/) is a high-performance object storage.\n\n```yaml\nstorage:\n s3:\n flavor: minio\n endpoint: http://127.0.0.1:9000\n```\n\nNote: `default_index_root_uri` or index URIs do not include the endpoint, you should set it as a typical S3 path such as `s3://indexes`.\n" }, { "path": "docs/configuration/template-config.md", "content": "---\ntitle: Index template configuration\nsidebar_position: 7\ntoc_max_heading_level: 4\n---\n\nThis page describes how to configure an index template.\n\nIndex templates let you dynamically create indexes according to predefined rules. Templates are used automatically when documents are received on the ingest API for an index that doesn't exist.\n\nThe index template configuration lets you define the following parameters:\n- `template_id` (required)\n- `description`\n- `index_id_patterns` (required)\n- `index_root_uri`\n- `priority`\n\nBesides, the following parameters can also be configured and are the same as those found in the [index configuration](../configuration/index-config.md):\n- doc mapping (required)\n- indexing settings\n- search settings\n- retention policy\n\nYou can manage templates using the [index template API](../reference/rest-api.md#index-template-api).\n\n## Config file format\n\nThe index configuration format is YAML or JSON. When a key is absent from the configuration file, the default value is used.\nHere is a complete example:\n\n```yaml\nversion: 0.9 # File format version.\n\ntemplate_id: \"hdfs-dev\"\n\nindex_root_uri: \"s3://my-bucket/hdfs-dev/\"\n\ndescription: \"HDFS log management dev\"\n\nindex_id_patterns:\n - hdfs-dev-*\n - hdfs-staging-*\n\npriority: 100\n\ndoc_mapping:\n mode: lenient\n field_mappings:\n - name: timestamp\n type: datetime\n input_formats:\n - unix_timestamp\n output_format: unix_timestamp_secs\n fast_precision: seconds\n fast: true\n - name: severity_text\n type: text\n tokenizer: raw\n fast:\n - tokenizer: lowercase\n - name: body\n type: text\n tokenizer: default\n record: position\n - name: resource\n type: object\n field_mappings:\n - name: service\n type: text\n tokenizer: raw\n tag_fields: [\"resource.service\"]\n timestamp_field: timestamp\n index_field_presence: true\n\nsearch_settings:\n default_search_fields: [severity_text, body]\n\nretention:\n period: 90 days\n schedule: daily\n```\n\n## Template ID\n\nThe `template_id` is a string that uniquely identifies the index template within the metastore. It may only contain uppercase or lowercase ASCII letters, digits, hyphens (`-`), and underscores (`_`). It must start with a letter and contain at least 3 characters but no more than 255.\n\n## Description\n\nAn optional string that describes what the index template is used for.\n\n## Index root uri\n\nThe `index_root_uri` defines where the index files (also called splits) should be stored.\nThis parameter expects a [storage uri](storage-config#storage-uris).\n\nThe actual URI of the index is the path concatenation of the `index_root_uri` with the index id. \n\nIf `index_root_uri` is not defined, the `default_index_root_uri` from [Quickwit's node config](node-config) will be used.\n\n## Index ID patterns\n\n`index_id_patterns` is a list of strings that define which indices should be created according to this template. Use [glob-like](https://en.wikipedia.org/wiki/Glob_(programming)) wildcard ( \\* ) expressions to target indices that match a pattern: test\\* or \\*test or te\\*t or \\*test\\*. You can also use negative patterns by prepending the hyphen `-` character.\n\nPatterns must obey the following rules:\n- It must follow the regex `^-?[a-zA-Z\\*][a-zA-Z0-9-_\\.\\*]{0,254}$`.\n- It cannot contain consecutive asterisks (`*`).\n- If it does not contain an asterisk (`*`), the length must be greater than or equal to 3 characters.\n\n## Priority\n\nWhen multiple templates match a new index ID, the template with the highest `priority` is used to configure the index.\n" }, { "path": "docs/deployment/_category_.yaml", "content": "label: 'Deployment'\nposition: 7\ncollapsed: true\n" }, { "path": "docs/deployment/cluster-sizing.md", "content": "---\ntitle: Cluster sizing\nsidebar_position: 3\n---\n\nIn this guide, we discuss how to size your Quickwit cluster and nodes. As shown\nin the [architecture section](../overview/architecture.md), a Quickwit cluster\nhas 5 main components: the Indexers, Searchers, Control Plane,\nMetastore and, Janitor. Each component has different resource requirements\nand can be scaled independently. We will also discuss how to size the metastore\nPostgreSQL database.\n\n:::note\n\nThis guide provides general guidelines. The actual resource requirements depend\nstrongly on your workload. We recommend monitoring the resource usage and\nadjusting the cluster size accordingly.\n\n:::\n\n## Quickwit services\n\n### Indexers\n\nHere are some high-level guidelines to size your Indexer nodes:\n- Quickwit can index at around **7.5MB per second per core**\n- For the general use case, configure 4GB of RAM per core\n - Workloads with a large number of indexes or data sources consume more RAM\n \n - Don't use instances with less than 8GB of RAM\n \n- Mount the data directory to a volume of at least 120GB to store:\n - the [split cache](../configuration/node-config.md#Indexer-configuration) (default 100GB)\n - the [ingest queue](../configuration/node-config.md#ingest-api-configuration) (default 4GiB)\n - a little extra for the indexes that are being built (first generation and merges)\n- Local SSDs are preferred for deploying Indexers since they generally provide the best performance per dollar and save some network bandwidth. However, remote disks can also if they provide roughly 20 MB/s of write throughput per core when using the ingest API or 10 MB/s when relying on other sources. For Amazon EBS volumes, this is equivalent to 320 or 160 IOPS per core (assuming 64 KB IOPS).\n\n:::note\n\nTo utilize all CPUs on Indexer nodes that have more than 4 cores, your indexing\nworkload needs to be broken down into multiple indexing pipelines. This can be\nachieved by creating multiple indexes or by using a [partitioned data\nsource](../configuration/source-config.md#number-of-pipelines) such as\n[Kafka](../configuration/source-config.md#kafka-source) or the [ingest API\n(v2)](../ingest-data/ingest-api.md#ingest-api-versions).\n\n:::\n\n\n### Searchers\n\nSearch performance is highly dependent on the workload. For example, term queries\nare usually cheaper than aggregations. A good starting point for dimensioning\nSearcher nodes:\n- Configure 8GB of RAM per core when using a high latency / low bandwidth object\n store like AWS S3\n- Decrease the RAM / CPU ratio (e.g 4GB/core) when using a faster object store\n- Provision more RAM if you expect many concurrent aggregation requests. By\n default, each request can use up to 500MB of RAM on each node.\n- Avoid instances with less than 4GB of RAM\n\n- Searcher nodes don't use disk unless the [split\n cache](../configuration/node-config.md#Searcher-split-cache-configuration) is\n explicitly enabled\n\nOne strength of Quickwit is that its Searchers are stateless, which makes it\neasy to scale them up and down based on the workload. Scale the number of\nSearcher nodes based on:\n- the number of concurrent requests expected\n- aggregations that run on large amounts of data (without\n [time](../overview/concepts/querying.md#time-sharding) or\n [tag](../overview/concepts/querying.md#tag-pruning) pruning)\n\n### Other services\n\nThe Control Plane, Metastore and, Janitor are lightweight components.\n\n- **Control Plane**: A cluster must have only one Control Plane. It needs a\n single core and 2GB of RAM. It doesn't require any disk.\n\n- **Metastore**: A cluster must have exactly one Metastore when using the\n [file-backed metastore](../configuration/metastore-config.md#file-backed-metastore).\n When using the [PostgreSQL metastore](#postgres-metastore-backend), you can\n run one or several Metastore pods for high availability (HA). The Metastore\n requires a single core and 2GB of RAM. For clusters handling hundreds of\n indexes, you may increase the size to 2 cores and 4GB of RAM. It doesn't\n write to disk (when using PostgreSQL, the database handles persistence).\n\n- **Janitor**: A cluster must have only one Janitor. In general, it requires 1\n core and 2GB of RAM and doesn't use the disk. If you use the [delete\n API](https://quickwit.io/docs/overview/concepts/deletes), the Janitor should\n be dimensioned like an indexer.\n\n### Single node deployments\n\nFor experimentations and small scale POCs, it is possible to deploy all the\nservices on a single node (see\n[tutorial](../get-started/tutorials/tutorial-hdfs-logs.md)). We recommend at\nleast 2 cores and 8GB of RAM.\n\n## Postgres Metastore backend\n\nFor most use cases, a PostgreSQL instance with 4GB of RAM and 1 core is\nsufficient:\n- with the AWS RDS managed service, use the t4g.medium instance type. Enable\n multi-AZ with one standby for high availability.\n" }, { "path": "docs/deployment/deployment-modes.md", "content": "---\ntitle: Deployment modes\nsidebar_position: 1\n---\n\nAs an application, Quickwit is built out of multiple services and is designed to run as a horizontally-scalable distributed system. Currently, Quickwit supports four core services (indexer, searcher, metastore, control plane) and one maintenance service (janitor):\n\n- Indexers ingest documents from data sources and build indexes.\n- Searchers execute search queries submitted via the REST API.\n- The Metastore stores index metadata in a PostgreSQL-compatible database or cloud-hosted file.\n- The Control Plane distributes and coordinates indexing workloads on indexers.\n- The Janitor performs periodic maintenance tasks.\n\nQuickwit is distributed as a single binary or Docker image. The behavior of that executable file or image is controlled with the `--service` option of the `quickwit run` command and defines which services run on a node. You may start one service, multiple, or all of them. Nodes always serve the REST API and the search and admin UI. In addition, they will redirect requests that they cannot satisfy to the appropriate nodes in the cluster. Finally, each service can run on one or several nodes depending on the expected load on the system.\n\n## Standalone mode (single node)\n\nThis deployment mode is the simplest way to get started with Quickwit. Launch all the services with the `quickwit run` [command](../reference/cli.md), and you are now ready to ingest data and search your indexes.\n\n## Cluster mode (multi-node)\n\nYou can deploy Quickwit on multiple nodes. We provide a [Helm chart](./kubernetes/helm.md) to help you deploy Quickwit on Kubernetes. In cluster mode, you must store your index data on a shared storage backend such as Amazon S3 or MinIO.\n\n## One indexer, multiple searchers\n\nOne indexer running on a small instance (4 vCPUs) can ingest documents at a throughput of 20-40MB/s (1-3+ TB/day). A deployment with one indexer is thus an excellent place to start. However, you may need several searchers to handle large datasets or serve many resource-intensive requests such as aggregation queries.\n\n## Multiple indexers, multiple searchers\n\nIndexing a single [data source](../configuration/source-config.md) on several indexers is only possible with a [Kafka source](../configuration/source-config.md#kafka-source).\nSupport for native distributed indexing was added with Quickwit 0.9.\n\n## File-backed metastore limitations\n\nThe file-backed metastore is a good fit for standalone and small deployments. However, it does not support multiple instances running at the same time. As long as you can guarantee that no more than one metastore is running at any given time, the file-backed metastore is safe to use. For heavy workloads, we recommend using a PostgreSQL metastore.\n" }, { "path": "docs/deployment/kubernetes/_category_.yaml", "content": "label: 'Kubernetes'\nposition: 2\ncollapsed: true\n" }, { "path": "docs/deployment/kubernetes/gke.md", "content": "---\ntitle: Install Quickwit on Google GKE\nsidebar_label: Google GKE\nsidebar_position: 2\n---\n\nThis guide will help you set up a Quickwit cluster with the correct GCS permissions.\n\n\n## Set up\n\nBefore installing Quickwit with Helm, let's create a namespace for our playground.\n\n```\nexport NS=quickwit-tutorial\nkubectl create ns ${NS}\n```\n\nQuickwit stores its index on an object storage. We will use GCS, which is natively supported since the 0.7 version (for versions < 0.7, you should use an S3 interoperability key).\n\nThe following steps create a GCP and a GKE service account and bind them together.\nWe are going to create them, set the right permissions and bind them.\n\n```bash\nexport PROJECT_ID={your-project-id}\nexport GCP_SERVICE_ACCOUNT=quickwit-tutorial\nexport GKE_SERVICE_ACCOUNT=quickwit-sa\nexport BUCKET=your-bucket\n\nkubectl create serviceaccount ${GKE_SERVICE_ACCOUNT} -n ${NS}\n\ngcloud iam service-accounts create ${GCP_SERVICE_ACCOUNT} --project=${PROJECT_ID}\n\ngcloud storage buckets add-iam-policy-binding gs://${BUCKET} \\\n--member \"serviceAccount:${GCP_SERVICE_ACCOUNT}@${PROJECT_ID}.iam.gserviceaccount.com\" \\\n--role \"roles/storage.objectAdmin\"\n\n# Notice that the member is related to a namespace.\ngcloud iam service-accounts add-iam-policy-binding ${GCP_SERVICE_ACCOUNT}@${PROJECT_ID}.iam.gserviceaccount.com \\\n--role roles/iam.workloadIdentityUser \\\n--member \"serviceAccount:${PROJECT_ID}.svc.id.goog[${NS}/${GKE_SERVICE_ACCOUNT}]\"\n\n# Now we can annotate our service account!\nkubectl annotate serviceaccount ${GKE_SERVICE_ACCOUNT} \\\niam.gke.io/gcp-service-account=${GCP_SERVICE_ACCOUNT}@${PROJECT_ID}.iam.gserviceaccount.com \\\n-n ${NS}\n```\n\n## Install Quickwit using Helm\n\nWe are now ready to install Quickwit on GKE. If you'd like to know more about Helm, consult our [comprehensive guide](./helm.md) for installing Quickwit on Kubernetes.\n\n```bash\nhelm repo add quickwit https://helm.quickwit.io\nhelm repo update quickwit\n```\n\nLet's set Quickwit `values.yaml`:\n\n```yaml\n# We use the edge version here as we recently fixed\n# a bug which prevents the metastore from running on GCS.\nimage:\n repository: quickwit/quickwit\n pullPolicy: Always\n tag: edge\n\nserviceAccount:\n create: false\n name: quickwit-sa\n\nconfig:\n default_index_root_uri: gs://{BUCKET}/qw-indexes\n metastore_uri: gs://{BUCKET}/qw-indexes\n\n```\n\nWe're ready to deploy:\n\n```bash\nhelm install quickwit/quickwit -f values.yaml\n```\n\n## Check that Quickwit is running\n\nIt should take a few seconds for the cluster to start. During the startup process, individual pods might restart themselves several times.\n\nTo access the UI, you can run the following command and then open your browser at [http://localhost:7280](http://localhost:7280):\n\n```\nkubectl port-forward svc/release-name-quickwit-searcher 7280:7280\n```\n\n\n## Uninstall the deployment\n\nRun the following Helm command to uninstall the deployment\n\n```bash\nhelm uninstall \n```\n\nAnd don't forget to clean your bucket, Quickwit should have stored 3 files in `gs://{BUCKET}/qw-indexes`.\n" }, { "path": "docs/deployment/kubernetes/glasskube.md", "content": "---\ntitle: Install Quickwit with Glasskube\nsidebar_label: Glasskube\nsidebar_position: 3\n---\n\n[Glasskube](https://glasskube.dev) is a package manager for Kubernetes that empowers you to effortlessly install, upgrade, configure, and manage your Kubernetes cluster packages, all while streamlining repetitive and cumbersome maintenance tasks.\n\n## Requirements\n\nTo deploy Quickwit on Kubernetes, you will need:\n\n- kubectl, compatible with your cluster (+/- 1 minor release from your cluster) (`kubectl version`)\n- A Kubernetes cluster\n\n1. Install `kubectl` and `glasskube` cli.\n\nTo install `kubectl` locally, you can refer to [this documentation](https://kubernetes.io/docs/tasks/tools/#install-kubectl).\n\nTo install `glasskube` cli locally, you can refer to [this documentation](https://glasskube.dev/docs/getting-started/install) and choose the right installation options according to your operating system.\n\nFor example, let's assume that you're on MacOS using homebrew and kind, this is what you'll have to do:\n\n```shell\nbrew install glasskube/tap/glasskube # install the glasskube cli\nkind create cluster # create a kind Kubernetes cluster\n```\n\n2. Install glasskube in your Kubernetes cluster:\n\n```shell\nglasskube bootstrap\n```\n\n3. Start and access to the Glasskube's GUI:\n\n```shell\nglasskube serve\n```\n\nYou'll be able to access to the GUI of Glasskube here: http://localhost:8580\n\n## Install Quickwit using Glasskube\n\n`glasskube` will install Quickwit in the `quickwit` namespace. You can perform the Quickwit installation directly with the GUI:\n\n![screenshot-glasskube-ui.png](../../assets/images/screenshot-glasskube-ui.png)\n\nOr use the CLI instead:\n\n```shell\nglasskube install quickwit\n```\n\nIn both, you'll have to set the value of those parameters:\n\n* `defaultIndexRootUri`: the default index URI is a S3 compliant bucket which usually looks like this: `s3:///`\n* `metastoreUri`: if you're not using PostgreSQL and object storage, you can pick the same bucket and value you used for the `defaultIndexRootUri` parameter\n* `s3Endpoint`: the http(s) URL of your object storage service which should looks like `https://s3.{region}.{your object storage domain}`\n* `s3Flavor`: which can be one of the following: `do`, `garage`, `gcp`, `minio`. You can leave it empty if your object storage is compliant with AWS S3\n* `s3Region`\n* `s3AccessKeyId`\n* `s3SecretAccessKey`\n\n## Uninstall quickwit\n\n```shell\nglasskube uninstall quickwit\n```\n" }, { "path": "docs/deployment/kubernetes/helm.md", "content": "---\ntitle: Install Quickwit with Helm\nsidebar_label: Helm\nsidebar_position: 1\n---\n\n[Helm](https://helm.sh) is a package manager for Kubernetes that allows you to configure, install, and upgrade containerized applications in a Kubernetes cluster in a version-controlled and reproducible way.\n\nYou can install Quickwit on Kubernetes with the official Quickwit Helm chart. If you encounter any problem with the chart, please, open an issue in our [GitHub repository](https://github.com/quickwit-oss/helm-charts).\n\n## Requirements\n\nTo deploy Quickwit on Kubernetes, you will need:\n\n- kubectl, compatible with your cluster (+/- 1 minor release from your cluster) (`kubectl version`)\n- Helm v3 (`helm version`)\n- A Kubernetes cluster\n\n1. Install `kubectl` and `helm`\n\nTo install `kubectl` and `helm` locally, follow the [Kubernetes](https://kubernetes.io/docs/tasks/tools/#install-kubectl) and [Helm](https://helm.sh/docs/intro/install/) documentation pages.\n\n2. Add the Quickwit Helm chart repository to Helm\n\n```bash\nhelm repo add quickwit https://helm.quickwit.io\n```\n\n3. Update the repository\n\n```bash\nhelm repo update quickwit\n```\n\n4. Create and customize your configuration file `values.yaml`\n\nYou can inspect the default configuration values of the chart using the following command:\n\n```bash\nhelm show values quickwit/quickwit\n```\n\nHere is an example of a minimal configuration with a file-backed metastore:\n\n```yaml\nenvironment:\n QW_METASTORE_URI: s3:///quickwit-indexes\n\nconfig:\n default_index_root_uri: s3:///quickwit-indexes\n storage:\n s3:\n region: eu-east-1\n # We recommend using IAM roles and permissions to access Amazon S3 resources,\n # but you can specify a pair of access and secret keys if necessary.\n access_key_id: \n secret_access_key: \n```\n\n5. Deploy Quickwit\n\n```bash\nhelm install quickwit/quickwit -f values.yaml\n```\n\n6. Check that Quickwit is running\n\nIt might take some time for the cluster to start. During the startup process individual pods might restart themselves several times. The command on the previous step will print the instructions on how to connect to the cluster. This endpoint can be used to access the quickwit search UI, as well execute standard API commands against.\n\n## Using PostgreSQL as a metadata store\n\nThe file-backed metastore is mainly useful for testing purposes. Though a file-backed metastore might be easier to setup, we strongly encourage you to use a PostgreSQL metastore in production. For the quickwit installation to work with PostgreSQL metadata you need to provide connection PostgreSQL information instead of metastore URI:\n\n```yaml\nconfig:\n default_index_root_uri: s3:///quickwit-indexes\n\n postgres:\n host: \n port: 5432\n database: quickwit-metastore\n username: quickwit\n password: # This password will be stored as a Kubernetes Secret\n\n storage: {}\n s3:\n region: eu-east-1\n # We recommend using IAM roles and permissions to access Amazon S3 resources,\n # but you can specify a pair of access and secret keys if necessary.\n access_key_id: \n secret_access_key: \n```\n\n## Uninstall the deployment\n\nRun the following Helm command to uninstall the deployment\n\n```bash\nhelm uninstall \n```\n" }, { "path": "docs/distributed-tracing/_category_.yaml", "content": "label: 'Distributed tracing'\nposition: 6\ncollapsed: true\n" }, { "path": "docs/distributed-tracing/otel-service.md", "content": "---\ntitle: OTEL service\nsidebar_position: 5\n---\n\nQuickwit natively supports the [OpenTelemetry Protocol (OTLP)](https://opentelemetry.io/docs/reference/specification/protocol/otlp/) and provides a gRPC endpoint to receive spans from an OpenTelemetry collector, or from your application directly, via an exporter. This endpoint is enabled by default.\n\nWhen enabled, Quickwit will start the gRPC service ready to receive spans from an OpenTelemetry collector. The spans are indexed in the `otel-trace-v0_7` index by default, and this index will be automatically created if not present. The index doc mapping is described in the next [section](#trace-and-span-data-model).\n\nIf for any reason, you want to disable this endpoint, you can:\n- Set the `QW_ENABLE_OTLP_ENDPOINT` environment variable to `false` when starting Quickwit.\n- Or [configure the node config](/docs/configuration/node-config.md) by setting the indexer setting `enable_otlp_endpoint` to `false`.\n\n```yaml title=node-config.yaml\n# ... Indexer configuration ...\nindexer:\n enable_otlp_endpoint: false\n```\n\n## Sending spans in your own index\n\nYou can send spans in the index of your choice by setting the header `qw-otel-traces-index` of your gRPC request to the targeted index ID.\n\n\n## Trace and span data model\n\nA trace is a collection of spans that represents a single request. A span represents a single operation within a trace. OpenTelemetry collectors send spans, Quickwit then indexes them in the `otel-trace-v0_7` index by default that maps OpenTelemetry span model to an indexed document in Quickwit.\n\nThe span model is derived from the [OpenTelemetry specification](https://opentelemetry.io/docs/reference/specification/trace/api/).\n\nBelow is the doc mapping of the `otel-trace-v0_7` index:\n\n```yaml\n\nversion: 0.7\n\nindex_id: otel-trace-v0_7\n\ndoc_mapping:\n mode: strict\n field_mappings:\n - name: trace_id\n type: bytes\n input_format: hex\n output_format: hex\n fast: true\n - name: trace_state\n type: text\n indexed: false\n - name: service_name\n type: text\n tokenizer: raw\n fast: true\n - name: resource_attributes\n type: json\n tokenizer: raw\n - name: resource_dropped_attributes_count\n type: u64\n indexed: false\n - name: scope_name\n type: text\n indexed: false\n - name: scope_version\n type: text\n indexed: false\n - name: scope_attributes\n type: json\n indexed: false\n - name: scope_dropped_attributes_count\n type: u64\n indexed: false\n - name: span_id\n type: bytes\n input_format: hex\n output_format: hex\n - name: span_kind\n type: u64\n - name: span_name\n type: text\n tokenizer: raw\n fast: true\n - name: span_fingerprint\n type: text\n tokenizer: raw\n - name: span_start_timestamp_nanos\n type: datetime\n input_formats: [unix_timestamp]\n output_format: unix_timestamp_nanos\n indexed: false\n fast: true\n fast_precision: milliseconds\n - name: span_end_timestamp_nanos\n type: datetime\n input_formats: [unix_timestamp]\n output_format: unix_timestamp_nanos\n indexed: false\n fast: false\n - name: span_duration_millis\n type: u64\n indexed: false\n fast: true\n - name: span_attributes\n type: json\n tokenizer: raw\n fast: true\n - name: span_dropped_attributes_count\n type: u64\n indexed: false\n - name: span_dropped_events_count\n type: u64\n indexed: false\n - name: span_dropped_links_count\n type: u64\n indexed: false\n - name: span_status\n type: json\n indexed: true\n - name: parent_span_id\n type: bytes\n input_format: hex\n output_format: hex\n indexed: false\n - name: events\n type: array\n tokenizer: raw\n fast: true\n - name: event_names\n type: array\n tokenizer: default\n record: position\n stored: false\n - name: links\n type: array\n tokenizer: raw\n\n timestamp_field: span_start_timestamp_nanos\n\nindexing_settings:\n commit_timeout_secs: 10\n\nsearch_settings:\n default_search_fields: []\n```\n\n## Known limitations\n\nThere are a few limitations on the current distributed tracing setup in Quickwit 0.9:\n- The OTLP gRPC service does not provide High-Durability. This will be fixed in 0.10.\n- OTLP HTTP is only available with the Binary Protobuf Encoding. OTLP HTTP with JSON encoding is not planned yet, but this can be easily fixed in the next version. Please open an issue if you need this feature.\n\nIf you are interested in new features or discovered other limitations, please open an issue on [GitHub](https://github.com/quickwit-oss/quickwit).\n" }, { "path": "docs/distributed-tracing/overview.md", "content": "---\ntitle: Distributed Tracing with Quickwit\nsidebar_label: Overview\nsidebar_position: 1\n---\n\nDistributed Tracing is a process that tracks your application requests flowing through your different services: frontend, backend, databases and more. It's a powerful tool to understand how your application works and to debug performance issues.\n\nQuickwit is a cloud-native engine to index and search unstructured data which makes it a perfect fit for a traces backend.\n\nMoreover, Quickwit supports natively the [OpenTelemetry gRPC and HTTP (protobuf only) protocol](https://opentelemetry.io/docs/reference/specification/protocol/otlp/) and the [Jaeger gRPC API (SpanReader only)](https://www.jaegertracing.io/). **This means that you can use Quickwit to store your traces and to query them with Jaeger UI**.\n\n![Quickwit Distributed Tracing](../assets/images/distributed-tracing-overview-light.png#gh-light-mode-only)![Quickwit Distributed Tracing](../assets/images/distributed-tracing-overview-dark.png#gh-dark-mode-only)\n\n## Plug Quickwit to Jaeger\n\nQuickwit implements a gRPC service compatible with Jaeger UI. All you need is to configure Jaeger with a (span) storage type `grpc`[^1] and you will be able to visualize your traces in Jaeger that are stored in any Quickwit's indexes matching the pattern `otel-traces-v0_*`.\n\nWe made a tutorial on [how to plug Quickwit to Jaeger UI](plug-quickwit-to-jaeger.md) that will guide you through the process.\n\n[^1]: It was `grpc-plugin` until the version 1.58 of Jaeger.\n\n## Send traces to Quickwit\n\n- [Using OTEL collector](send-traces/using-otel-collector.md)\n- [Using python OTEL SDK](send-traces/using-otel-sdk-python.md)\n\n" }, { "path": "docs/distributed-tracing/plug-quickwit-to-jaeger.md", "content": "---\ntitle: Plug Quickwit to Jaeger\ndescription: A simple tutorial to use Jaeger with Quickwit backend.\nicon_url: /img/tutorials/quickwit-logo.png\ntags: [traces, ingestion]\nsidebar_position: 2\n---\n\nIn this tutorial, we will show you how Quickwit can eat its own dog food: we will send Quickwit traces into Jaeger and analyze them, which will generate new traces to analyze :)\n\n## Start Quickwit\n\nFirst, start a [Quickwit instance](../get-started/installation.md) with the OTLP service enabled:\n\n```bash\nQW_ENABLE_OPENTELEMETRY_OTLP_EXPORTER=true \\\nOTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:7281 \\\n./quickwit run\n```\n\nWe also set `QW_ENABLE_OPENTELEMETRY_OTLP_EXPORTER` and `OTEL_EXPORTER_OTLP_ENDPOINT` environment variables so that Quickwit will send its own traces to itself.\n\n## Start Jaeger UI\n\nLet's start a Jaeger UI instance with docker. Here we need to inform jaeger that it should use quickwit as its backend.\n\nDue to some idiosyncrasy associated with networking with containers, we will have to use a different approach on MacOS & Windows on one side, and Linux on the other side.\n\n### MacOS & Windows\n\nWe can rely on `host.docker.internal` to get the docker bridge ip address, pointing to our quickwit server.\n\n```bash\ndocker run --rm --name jaeger-qw \\\n -e SPAN_STORAGE_TYPE=grpc \\\n -e GRPC_STORAGE_SERVER=host.docker.internal:7281 \\\n -p 16686:16686 \\\n jaegertracing/jaeger-query:1.60\n```\n\n### Linux\n\nBy default, Quickwit is listening to `127.0.0.1`, and will not respond to request directed\nto the docker bridge (`172.17.0.1`). There are different ways to solve this problem.\nThe easiest is probably to use host network mode.\n\n```bash\ndocker run --rm --name jaeger-qw --network=host \\\n -e SPAN_STORAGE_TYPE=grpc \\\n -e GRPC_STORAGE_SERVER=127.0.0.1:7281 \\\n -p 16686:16686 \\\n jaegertracing/jaeger-query:1.60\n\n```\n\n## Search traces in Jaeger UI\n\nAs Quickwit is indexing its own traces, you should be able to see them in Jaeger UI after 5 seconds (the time it takes for Quickwit to do its first commit).\n\nOpen the Jaeger UI at [http://localhost:16686](http://localhost:16686) and search for traces! By executing search queries, you will then see Quickwit's own traces:\n\n- `find_traces` is the endpoint called when you search for traces in Jaeger UI, it then calls `find_trace_ids`.\n- `find_traces_ids` is doing an aggregation query on spans to get unique trace IDs.\n- `root_search` is Quickwit search entry point. It calls search on each split (piece of index) in parallel, in a distributed manner, or just locally if there is only one node.\n- `leaf_search` is the search entry point on each node. It calls `leaf_search_single_split` on each split.\n- `leaf_search_single_split` is the search entry point on a split. It will call consecutively `warmup` and `tantivy_search`.\n- `warmup` is the warmup phase of the search. It prefetches data needed to execute the search query.\n- `tantivy_search` is the search phase of the search. It is executing the search query at horse speeds with the [Tantivy](https://github.com/quickwit-oss/tantivy).\n\n![Quickwit trace in Jaeger UI](../assets/images/jaeger-ui-quickwit-trace-analysis.png)\n\n## Next steps\n\nYou are now ready for the next step: instrumenting your application and sending its traces to Quickwit. You can do it:\n- In [python](send-traces/using-otel-sdk-python.md).\n- And in any other language that OpenTelemetry supports.\n" }, { "path": "docs/distributed-tracing/send-traces/_category_.yaml", "content": "label: 'Sending traces'\nposition: 3\ncollapsed: false\n" }, { "path": "docs/distributed-tracing/send-traces/using-otel-collector.md", "content": "---\ntitle: Using OTEL Collector\ndescription: Using OTEL Collector\ntags: [otel, collector, traces]\nsidebar_position: 1\n---\n\nimport Tabs from '@theme/Tabs';\nimport TabItem from '@theme/TabItem';\n\nIf you already have your own OpenTelemetry Collector and want to export your traces to Quickwit, you need a new OLTP gRPC exporter in your config.yaml:\n\n\n\n\n\n```yaml title=\"otel-collector-config.yaml\"\nreceivers:\n otlp:\n protocols:\n grpc:\n http:\n\nprocessors:\n batch:\n\nexporters:\n otlp/quickwit:\n endpoint: host.docker.internal:7281\n tls:\n insecure: true\n # By default, traces are sent to the otel-traces-v0_7.\n # You can customize the index ID By setting this header.\n # headers:\n # qw-otel-traces-index: otel-traces-v0_7\n\nservice:\n pipelines:\n traces:\n receivers: [otlp]\n processors: [batch]\n exporters: [otlp/quickwit]\n```\n\n\n\n\n\n```yaml title=\"otel-collector-config.yaml\"\nreceivers:\n otlp:\n protocols:\n grpc:\n http:\n\nprocessors:\n batch:\n\nexporters:\n otlp/quickwit:\n endpoint: 127.0.0.1:7281\n tls:\n insecure: true\n\nservice:\n pipelines:\n traces:\n receivers: [otlp]\n processors: [batch]\n exporters: [otlp/quickwit]\n```\n\n\n\n\n\n\n## Test your OTEL configuration\n\n1. [Install](../../get-started/installation.md) and start a Quickwit server:\n \n```bash\n./quickwit run\n```\n\n2. Start a collector with the previous config:\n\n\n\n\n\n```bash\ndocker run -v ${PWD}/otel-collector-config.yaml:/etc/otelcol/config.yaml -p 4317:4317 -p 4318:4318 -p 7281:7281 otel/opentelemetry-collector\n```\n\n\n\n\n\n```bash\ndocker run -v ${PWD}/otel-collector-config.yaml:/etc/otelcol/config.yaml --network=host -p 4317:4317 -p 4318:4318 -p 7281:7281 otel/opentelemetry-collector\n```\n\n\n\n\n\n3. Send a trace to your collector with cURL:\n\n```bash\ncurl -XPOST \"http://localhost:4318/v1/traces\" -H \"Content-Type: application/json\" \\\n--data-binary @- << EOF\n{\n \"resource_spans\": [\n {\n \"resource\": {\n \"attributes\": [\n {\n \"key\": \"service.name\",\n \"value\": {\n \"string_value\": \"test-with-curl\"\n }\n }\n ]\n },\n \"scope_spans\": [\n {\n \"scope\": {\n \"name\": \"manual-test\"\n },\n \"spans\": [\n {\n \"time_unix_nano\": \"1678974011000000000\",\n \"observed_time_unix_nano\": \"1678974011000000000\",\n \"start_time_unix_nano\": \"1678974011000000000\",\n \"end_time_unix_nano\": \"1678974021000000000\",\n \"trace_id\": \"3c191d03fa8be0653c191d03fa8be065\",\n \"span_id\": \"3c191d03fa8be065\",\n \"kind\": 2,\n \"events\": [],\n \"status\": {\n \"code\": 1\n }\n }\n ]\n }\n ]\n }\n ]\n}\nEOF\n```\n\nYou should see a log on the Quickwit server similar to the following:\n\n```bash\n2023-03-16T13:44:09.369Z INFO quickwit_indexing::actors::indexer: new-split split_id=\"01GVNAKT5TQW0T2QGA245XCMTJ\" partition_id=6444214793425557444\n```\n\nThis means that Quickwit has received the trace and created a new split. Wait for the split to be published before searching for traces.\n\n## Next step\n\nFollow our tutorial on [how to send traces from your python app](using-otel-sdk-python.md).\n" }, { "path": "docs/distributed-tracing/send-traces/using-otel-sdk-python.md", "content": "---\ntitle: Using OTEL SDK - Python\ndescription: A simple tutorial to send traces to Quickwit from a Python Flask app.\nicon_url: /img/tutorials/python-logo.png\ntags: [python, traces, ingestion]\nsidebar_position: 2\n---\n\nimport Tabs from '@theme/Tabs';\nimport TabItem from '@theme/TabItem';\n\nIn this tutorial, we will show you how to instrument a Python [Flask](https://flask.palletsprojects.com/en/2.2.x/) app with OpenTelemetry and send traces to Quickwit. This tutorial was inspired by the [Python OpenTelemetry](https://opentelemetry.io/docs/instrumentation/python/getting-started/) documentation, huge thanks to the OpenTelemetry team!\n\n## Prerequisites\n\n- Python3 installed\n- Docker installed\n\n## Start a Quickwit instance\n\n[Install Quickwit](/docs/get-started/installation.md) and start a Quickwit instance:\n\n```bash\n./quickwit run\n```\n\n## Start Jaeger UI\n\nLet's start a Jaeger UI instance with docker. Here we need to inform jaeger that it should use quickwit as its backend.\n\nDue to some idiosyncrasy associated with networking with containers, we will have to use a different approach on MacOS & Windows on one side, and Linux on the other side.\n\n### MacOS & Windows\n\nWe can rely on `host.docker.internal` to get the docker bridge ip address, pointing to our quickwit server.\n\n```bash\ndocker run --rm --name jaeger-qw \\\n -e SPAN_STORAGE_TYPE=grpc \\\n -e GRPC_STORAGE_SERVER=host.docker.internal:7281 \\\n -p 16686:16686 \\\n jaegertracing/jaeger-query:1.60\n```\n\n### Linux\n\nBy default, quickwit is listening to `127.0.0.1`, and will not respond to request directed\nto the docker bridge (`172.17.0.1`). There are different ways to solve this problem.\nThe easiest is probably to use host network mode.\n\n```bash\ndocker run --rm --name jaeger-qw --network=host \\\n -e SPAN_STORAGE_TYPE=grpc \\\n -e GRPC_STORAGE_SERVER=127.0.0.1:7281 \\\n -p 16686:16686 \\\n jaegertracing/jaeger-query:1.60\n```\n\n## Run a simple Flask app\n\nWe will start a flask application that is doing three things on each HTTP call `http://localhost:5000/process-ip`:\n\n- Fetching an IP address from [https://httpbin.org/ip](https://httpbin.org/ip).\n- Parsing it and fake processing it with a random sleep.\n- Displaying it with a random sleep.\n\n\nLet's first install the dependencies:\n\n```bash\npip install flask\npip install opentelemetry-distro\npip install opentelemetry-exporter-otlp\n```\n\nThe opentelemetry-distro package installs the API, SDK, and the opentelemetry-bootstrap and opentelemetry-instrument tools that you’ll use.\n\nHere is the code of our app:\n\n```python title=my_app.py\nimport random\nimport time\nimport requests\n\nfrom flask import Flask\n\napp = Flask(__name__)\n\n@app.route(\"/process-ip\")\ndef process_ip():\n body = fetch()\n ip = parse(body)\n display(ip)\n return ip\n\ndef fetch():\n resp = requests.get('https://httpbin.org/ip')\n body = resp.json()\n return body\n\ndef parse(body):\n # Sleep for a random amount of time to make the span more visible.\n secs = random.randint(1, 100) / 1000\n time.sleep(secs)\n\n return body[\"origin\"]\n\ndef display(ip):\n # Sleep for a random amount of time to make the span more visible.\n secs = random.randint(1, 100) / 1000\n time.sleep(secs)\n\n message = f\"Your IP address is `{ip}`.\"\n print(message)\n\nif __name__ == \"__main__\":\n app.run(port=5000)\n```\n\n## Auto-instrumentation\n\nOpenTelemetry provides a tool called `opentelemetry-bootstrap` that automatically instruments your Python application.\n\n```bash\nopentelemetry-bootstrap -a install\n```\n\nAnd that's it, we are now ready to run the app:\n\n```bash\n# We don't need metrics.\nOTEL_METRICS_EXPORTER=none \\\nOTEL_TRACES_EXPORTER=console \\\nOTEL_SERVICE_NAME=my_app \\\npython my_app.py\n```\n\nBy hitting [http://localhost:5000/process-ip](http://localhost:5000/process-ip) you should see the corresponding trace in the console.\n\nThis is nice but it would be even better if we could have the time passed in each steps, get the status code of the HTTP request, and the content type of the response. Let's do that by manually instrumentating our app!\n\n## Manual instrumentation\n\n```python title=my_instrumented_app.py\nimport random\nimport time\nimport requests\n\nfrom flask import Flask\n\nfrom opentelemetry import trace\n\n# Creates a tracer from the global tracer provider\ntracer = trace.get_tracer(__name__)\n\napp = Flask(__name__)\n\n@app.route(\"/process-ip\")\n@tracer.start_as_current_span(\"process_ip\")\ndef process_ip():\n body = fetch()\n ip = parse(body)\n display(ip)\n return ip\n\n@tracer.start_as_current_span(\"fetch\")\ndef fetch():\n resp = requests.get('https://httpbin.org/ip')\n body = resp.json()\n\n headers = resp.headers\n current_span = trace.get_current_span()\n current_span.set_attribute(\"status_code\", resp.status_code)\n current_span.set_attribute(\"content_type\", headers[\"Content-Type\"])\n current_span.set_attribute(\"content_length\", headers[\"Content-Length\"])\n\n return body\n\n@tracer.start_as_current_span(\"parse\")\ndef parse(body):\n # Sleep for a random amount of time to make the span more visible.\n secs = random.randint(1, 100) / 1000\n time.sleep(secs)\n\n return body[\"origin\"]\n\n@tracer.start_as_current_span(\"display\")\ndef display(ip):\n # Sleep for a random amount of time to make the span more visible.\n secs = random.randint(1, 100) / 1000\n time.sleep(secs)\n\n message = f\"Your IP address is `{ip}`.\"\n print(message)\n\n current_span = trace.get_current_span()\n current_span.add_event(message)\n\nif __name__ == \"__main__\":\n app.run(port=5000)\n\n```\n\nWe can now start the new instrumented app:\n\n```bash\nOTEL_METRICS_EXPORTER=none \\\nOTEL_TRACES_EXPORTER=console \\\nOTEL_SERVICE_NAME=my_app \\\nopentelemetry-instrument python my_instrumented_app.py\n```\n\nIf you hit again [http://localhost:5000/process-ip](http://localhost:5000/process-ip), you should see new spans with name `fetch`, `parse`, and `display` and with the corresponding custom attributes!\n\n\n## Sending traces to Quickwit\n\nTo send traces to Quickwit, we need to use the OTLP exporter. This is a simple as this:\n\n```bash\nOTEL_METRICS_EXPORTER=none \\ # We don't need metrics\nOTEL_SERVICE_NAME=my_app \\\nOTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://localhost:7281 \\\nopentelemetry-instrument python my_instrumented_app.py\n```\n\nNow, if you hit [http://localhost:5000/process-ip](http://localhost:5000/process-ip), traces will be send to Quickwit, you just need to wait around 30 seconds before they are indexed. It's time for a coffee break!\n\n30 seconds has passed, let's query the traces from our service:\n\n```bash\ncurl -XPOST http://localhost:7280/api/v1/otel-trace-v0/search -H 'Content-Type: application/json' -d '{\n \"query\": \"resource_attributes.service.name:my_app\"\n}'\n```\n\nAnd then open the Jaeger UI [localhost:16686](http://localhost:16686/) and play with it, you have now a Jaeger UI powered by a Quickwit storage backend!\n\n![Flask trace analysis in Jaeger UI](../../assets/images/jaeger-ui-python-app-trace-analysis.png)\n\n![Flask traces in Jaeger UI](../../assets/images/jaeger-ui-python-app-traces.png)\n\n## Sending traces to your OpenTelemetry collector\n\nStart a collector as described in the [OpenTelemetry collector tutorial](using-otel-collector.md) and execute the following command:\n\n```bash\nOTEL_METRICS_EXPORTER=none \\ # We don't need metrics\nOTEL_SERVICE_NAME=my_app \\\nopentelemetry-instrument python instrumented_app.py\n```\n\nTraces will be sent to your collector, and then to Quickwit.\n\n\n## Wrap up\n\nIn this tutorial, we have seen how to instrument a Python application with OpenTelemetry and send traces to Quickwit. We have also seen how to use the Jaeger UI to analyze traces.\n\nAll the code snippets in our [tutorial repository](https://github.com/quickwit-oss/tutorials).\n\nPlease let us know what you think about this tutorial, and if you have any questions, feel free to reach out to us on [Discord](https://discord.gg/7eNYX4d) or [Twitter](https://twitter.com/quickwit_inc).\n" }, { "path": "docs/get-started/_category_.yaml", "content": "label: 'Get started'\nposition: 2\ncollapsed: false\n" }, { "path": "docs/get-started/installation.md", "content": "---\ntitle: Installation\nsidebar_position: 2\n---\n\nimport Tabs from '@theme/Tabs';\nimport TabItem from '@theme/TabItem';\nimport { useDocsVersion } from '@docusaurus/theme-common/internal';\n\nexport const RenderIf = ({children, condition}) => (\n <>\n {condition && children}\n \n);\n\nQuickwit compiles to a single binary and we provide different methods to install it:\n\n- Linux/MacOS binaries that you can [download manually](#download) or with the [install script](#install-script)\n- [Docker image](#use-the-docker-image)\n- [Helm chart](../deployment/kubernetes/helm.md)\n- [Glasskube](../deployment/kubernetes/glasskube.md)\n\n## Prerequisites\n\nQuickwit is officially only supported for Linux. Freebsd and MacOS are not officially supported, but should work as well.\n\nQuickwit supplies binaries for x86-64 and aarch64. No special instruction set is required, but on x86-64 SSE3 is recommended.\nSupport of aarch64 is currently experimental.\n\n## Download\n\n\n\nVersion: nightly - \nLicense: [Apache 2.0](https://github.com/quickwit-oss/quickwit/blob/main/LICENSE) -\nDownloads `.tar.gz`:\n- [Linux ARM64](https://github.com/quickwit-oss/quickwit/releases/download/nightly/quickwit-nightly-aarch64-unknown-linux-gnu.tar.gz)\n- [Linux x86_64](https://github.com/quickwit-oss/quickwit/releases/download/nightly/quickwit-nightly-x86_64-unknown-linux-gnu.tar.gz)\n- [macOS aarch64](https://github.com/quickwit-oss/quickwit/releases/download/nightly/quickwit-nightly-aarch64-apple-darwin.tar.gz)\n- [macOS x86_64](https://github.com/quickwit-oss/quickwit/releases/download/nightly/quickwit-nightly-x86_64-apple-darwin.tar.gz)\n\n\n\n\n\n\nversion: 0.8.1 - [Release notes](https://github.com/quickwit-oss/quickwit/releases/tag/v0.8.1) - [Changelog](https://github.com/quickwit-oss/quickwit/blob/main/CHANGELOG.md)\nLicense: [Apache 2.0](https://github.com/quickwit-oss/quickwit/blob/main/LICENSE)\nDownloads `.tar.gz`:\n- [Linux ARM64](https://github.com/quickwit-oss/quickwit/releases/download/v0.8.1/quickwit-v0.8.1-aarch64-unknown-linux-gnu.tar.gz)\n- [Linux x86_64](https://github.com/quickwit-oss/quickwit/releases/download/v0.8.1/quickwit-v0.8.1-x86_64-unknown-linux-gnu.tar.gz)\n- [macOS aarch64](https://github.com/quickwit-oss/quickwit/releases/download/v0.8.1/quickwit-v0.8.1-aarch64-apple-darwin.tar.gz)\n- [macOS x86_64](https://github.com/quickwit-oss/quickwit/releases/download/v0.8.1/quickwit-v0.8.1-x86_64-apple-darwin.tar.gz)\n\n\n\nCheck out the available builds in greater detail on [GitHub](https://github.com/quickwit-oss/quickwit/releases)\n\n### Note on external dependencies\n\nQuickwit depends on the following external libraries to work correctly:\n- `libssl`: the industry defacto cryptography library.\nThese libraries can be installed on your system using the native package manager.\nYou can install these dependencies using the following command:\n\n\n\n\n\n```bash\napt-get -y update && apt-get -y install libssl\n```\n\n\n\n\n\n```bash\nyum -y update && yum -y install openssl\n```\n\n\n\n\n\n```bash\npacman -S openssl\n```\n\n\n\n\n\nAdditionally it requires a few more dependencies to compile it. These dependencies are not required on production system:\n- `clang`: used to compile some dependencies.\n- `protobuf-compiler`: used to compile protobuf definitions.\n- `libssl-dev`: headers for libssl.\n- `pkg-config`: used to locate libssl.\n- `cmake`: used to build librdkafka, for kafka support.\nThese dependencies can be installed on your system using the native package manager.\nYou can install these dependencies using the following command:\n\n\n\n\n\n```bash\napt install -y clang protobuf-compiler libssl-dev pkg-config cmake\n```\n\n\n\n\n\n```bash\nyum -y update && yum -y install clang openssl-devel pkgconfig cmake3\n# amazonlinux only has protobuf-compiler 2.5, we need something much more up to date.\nwget https://github.com/protocolbuffers/protobuf/releases/download/v21.9/protoc-21.9-linux-x86_64.zip\nsudo unzip protoc-21.9-linux-x86_64.zip -d /usr/local\n# amazonlinux use cmake2 as cmake, we need cmake3\nln -s /usr/bin/cmake3 /usr/bin/cmake\n```\n\n\n\n\n\n```bash\npacman -S clang protobuf openssl pkg-config cmake make\n```\n\n\n\n\n\n## Install script\n\nTo easily install Quickwit on your machine, just run the command below from your preferred shell.\nThe script detects the architecture and then downloads the correct binary archive for the machine.\n\n```bash\ncurl -L https://install.quickwit.io | sh\n```\n\nAll this script does is download the correct binary archive for your machine and extracts it in the current working directory. This means you can download any desired archive from [github](https://github.com/quickwit-oss/quickwit/releases) that matches your OS architecture and manually extract it anywhere.\n\nOnce installed or extracted, all of Quickwit's installation files can be found in a directory named `quickwit-{version}` where `version` is the corresponding version of Quickwit. This directory has the following layout:\n\n```bash\nquickwit-{version}\n ├── config\n │ └── quickwit.yaml\n ├── LICENSE\n ├── quickwit\n └── qwdata\n```\n\n- `config/quickwit.yaml`: is the default configuration file.\n- `LICENSE`: the license file.\n- `quickwit`: the quickwit executable binary.\n- `qwdata/`: the default data directory.\n\n\n## Use the Docker image\n\nIf you use Docker, this might be one of the quickest way to get going.\nThe following command will pull the image from [Docker Hub](https://hub.docker.com/r/quickwit/quickwit)\nand start a container ready to execute Quickwit commands.\n\n```bash\ndocker run --rm quickwit/quickwit --version\n\n# If you are using Apple silicon based macOS system you might need to specify the platform.\n# You can also safely ignore jemalloc warnings.\ndocker run --rm --platform linux/amd64 quickwit/quickwit --version\n```\n\nTo get the full gist of this, follow the [Quickstart guide](./quickstart.md).\n" }, { "path": "docs/get-started/query-language-intro.md", "content": "---\ntitle: Introduction to Quickwit's query language\nsidebar_position: 3\n---\n\nQuickwit allows you to search on your indexed documents using a simple query language. Here's a quick overview.\n\n## Clauses\n\nThe main concept of this language is a clause, which represents a simple condition that can be tested against documents. \n\n### Querying fields\n\nA clause operates on fields of your document. It has the following syntax :\n```\nfield:condition\n```\n\nFor example, when searching documents where the field `app_name` contains the token `tantivy`, you would write the following clause:\n```\napp_name:tantivy\n```\n\nIn many cases the field name can be omitted, quickwit will then use the `default_search_fields` configured for the index.\n\n### Clauses Cheat Sheet\n\nQuickwit support various types of clauses to express different kinds of conditions. Here's a quick overview of them:\n\n| type | syntax | examples | description| `default_search_field`|\n|-------------|--------|----------|------------|-----------------------|\n| term | `field:token` | `app_name:tantivy`
`process_id:1234`
`word` | A term clause tests the existence of avalue in the field's tokens | yes |\n| term prefix | `field:prefix*` | `app_name:tant*`
`quick*` | A term clause tests the existence of a token starting with the provided value | yes |\n| term set | `field:IN [token token ..]` |`severity:IN [error warn]` | A term set clause tests the existence of any of the provided value in the field's tokens| yes |\n| phrase | `field:\"sequence of tokens\"` | `full_name:\"john doe\"` | A phrase clause tests the existence of the provided sequence of tokens | yes |\n| phrase prefix | `field:\"sequence of tokens\"*` | `title:\"how to m\"*` | A phrase prefix clause tests the existence of a sequence of tokens, the last one used like in a prefix clause | yes |\n| all | `*` | `*` | A match-all clause will match every document | no |\n| exist | `field:*` | `error:*` | An exist clause tests the existence of any value for the field, it will match only if the field exists | no |\n| range | `field:bounds` |`duration:[0 TO 1000}`
`last_name:[banner TO miller]` | A term clause tests the existence of a token between the provided bounds | no |\n\n## Queries\n\n### Combining queries\n\nClauses can be combined using boolean operators `AND` and `OR` to create more complex search expressions\nAn `AND` query will match only if conditions on both sides of the operator are met\n```\ntype:rose AND color:red\n```\n\nAn `OR` query will match if either or both conditions on each side of the operator are met\n```\nweekday:6 OR weekday:7\n```\n\nIf no operator is provided, `AND` is implicitly assumed.\n\n```\ntype:violet color:blue\n```\n\n### Grouping queries\nYou can build complex expressions by grouping clauses using parentheses.\n```\n(type:rose AND color:red) OR (type:violet AND color:blue)\n```\n\nWhen no parentheses are used, `AND` takes precedence over `OR`, meaning that the following query is equivalent to the one above.\n\n```\ntype:rose AND color:red OR type:violet AND color:blue\n```\n\n### Negating queries\n\nAn expression can be negated either with the operator `NOT` or by prefixing the query with a dash `-`.\n\n`NOT` and `-` take precedence over everything, such that `-a AND b` means `(-a) AND b`, not `-(a AND B)`.\n\n```\nNOT severity:debug\n```\n\nor\n\n```\ntype:proposal -(status:rejected OR status:pending)\n```\n\n\n## Dive deeper\n\nIf you want to know more about the query language, head to the [Query Language Reference](../reference/query-language.md)\n" }, { "path": "docs/get-started/quickstart.md", "content": "---\ntitle: Quickstart\nsidebar_position: 1\n---\n\nimport Tabs from '@theme/Tabs';\nimport TabItem from '@theme/TabItem';\n\nIn this quick start guide, we will install Quickwit, create an index, add documents and finally execute search queries. All the Quickwit commands used in this guide are documented [in the CLI reference documentation](/docs/reference/cli.md).\n\n## Install Quickwit using Quickwit installer\n\nThe Quickwit installer automatically picks the correct binary archive for your environment and then downloads and unpacks it in your working directory.\nThis method works only for [some OS/architectures](installation.md#download), and you will also need to install some [external dependencies](installation.md#note-on-external-dependencies).\n\n```bash\ncurl -L https://install.quickwit.io | sh\n```\n\n```bash\ncd ./quickwit-v*/\n./quickwit --version\n```\n\nYou can now move this executable directory wherever sensible for your environment and possibly add it to your `PATH` environment.\n\n## Use Quickwit's Docker image\n\nYou can also pull and run the Quickwit binary in an isolated Docker container.\n\n```bash\n# Create first the data directory.\nmkdir qwdata\ndocker run --rm quickwit/quickwit --version\n```\n\nIf you are using Apple silicon based macOS system you might need to specify the platform. You can also safely ignore jemalloc warnings.\n\n```bash\ndocker run --rm --platform linux/amd64 quickwit/quickwit --version\n```\n\n## Start Quickwit server\n\n\n\n\n\n```bash\n./quickwit run\n```\n\n\n\n\n\n```bash\ndocker run --rm -v $(pwd)/qwdata:/quickwit/qwdata -p 127.0.0.1:7280:7280 quickwit/quickwit run\n```\n\n\n\n\n\nTips: you can use the environment variable `RUST_LOG` to control quickwit verbosity.\n\nCheck it's working by browsing the [UI at http://localhost:7280](http://localhost:7280) or do a simple GET with cURL:\n\n```bash\ncurl http://localhost:7280/api/v1/version\n```\n\n## Create your first index\n\nBefore adding documents to Quickwit, you need to create an index configured with a YAML config file. This config file notably lets you define how to map your input documents to your index fields and whether these fields should be stored and indexed. See the [index config documentation](/docs/configuration/index-config.md).\n\nLet's create an index configured to receive Stackoverflow posts (questions and answers).\n\n```bash\n# First, download the stackoverflow dataset config from Quickwit repository.\ncurl -o stackoverflow-index-config.yaml https://raw.githubusercontent.com/quickwit-oss/quickwit/main/config/tutorials/stackoverflow/index-config.yaml\n```\n\nThe index config defines three fields: `title`, `body` and `creationDate`. `title` and `body` are [indexed and tokenized](../configuration/index-config.md#text-type), and they are also used as default search fields, which means they will be used for search if you do not target a specific field in your query. `creationDate` serves as the timestamp for each record. There are no more explicit field definitions as we can use the default dynamic [mode](/docs/configuration/index-config.md#mode): the undeclared fields will still be indexed, by default fast fields are enabled to enable aggregation queries. and the `raw` tokenizer is used for text. \n\nAnd here is the complete config:\n\n```yaml title=\"stackoverflow-index-config.yaml\"\n#\n# Index config file for stackoverflow dataset.\n#\nversion: 0.7\n\nindex_id: stackoverflow\n\ndoc_mapping:\n field_mappings:\n - name: title\n type: text\n tokenizer: default\n record: position\n stored: true\n - name: body\n type: text\n tokenizer: default\n record: position\n stored: true\n - name: creationDate\n type: datetime\n fast: true\n input_formats:\n - rfc3339\n fast_precision: seconds\n timestamp_field: creationDate\n\nsearch_settings:\n default_search_fields: [title, body]\n\nindexing_settings:\n commit_timeout_secs: 30\n```\n\nNow we can create the index with the command:\n\n\n\n\n\n```bash\n./quickwit index create --index-config ./stackoverflow-index-config.yaml\n```\n\n\n\n\n\n```bash\ncurl -XPOST http://127.0.0.1:7280/api/v1/indexes --header \"content-type: application/yaml\" --data-binary @./stackoverflow-index-config.yaml\n```\n\n\n\n\n\nCheck that a directory `./qwdata/indexes/stackoverflow` has been created, Quickwit will write index files here and a `metastore.json` which contains the [index metadata](../overview/architecture.md#index).\nYou're now ready to fill the index.\n\n\n## Let's add some documents\n\nQuickwit can index data from many [sources](/docs/configuration/source-config.md). We will use a new line delimited json [ndjson](http://ndjson.org/) datasets as our data source.\nLet's download [a bunch of stackoverflow posts (10 000)](https://quickwit-datasets-public.s3.amazonaws.com/stackoverflow.posts.transformed-10000.json) in [ndjson](http://ndjson.org/) format and index it.\n\n```bash\n# Download the first 10_000 Stackoverflow posts articles.\ncurl -O https://quickwit-datasets-public.s3.amazonaws.com/stackoverflow.posts.transformed-10000.json\n```\n\n\n\n\n\n```bash\n# Index our 10k documents.\n./quickwit index ingest --index stackoverflow --input-path stackoverflow.posts.transformed-10000.json --force\n```\n\n\n\n\n\n```bash\n# Index our 10k documents.\ncurl -XPOST \"http://127.0.0.1:7280/api/v1/stackoverflow/ingest?commit=force\" --data-binary @stackoverflow.posts.transformed-10000.json\n```\n\n\n\n\n\nAs soon as the ingest command finishes you can start querying data by using the following `search` command:\n\n\n\n\n\n```bash\n./quickwit index search --index stackoverflow --query \"search AND engine\"\n```\n\n\n\n\n\n```bash\ncurl \"http://127.0.0.1:7280/api/v1/stackoverflow/search?query=search+AND+engine\"\n```\n\n\n\n\n\nIt should return 10 hits. Now you're ready to play with the search API.\n\n\n## Execute search queries\n\n\nLet's start with a query on the field `title`: `title:search AND engine`:\n```bash\ncurl \"http://127.0.0.1:7280/api/v1/stackoverflow/search?query=title:search+AND+engine\"\n```\n\nThe same request can be expressed as a JSON query:\n```bash\ncurl -XPOST \"http://localhost:7280/api/v1/stackoverflow/search\" -H 'Content-Type: application/json' -d '{\n \"query\": \"title:search AND engine\"\n}'\n```\n\nThis format is more verbose but it allows you to use more advanced features such as aggregations. The following query finds most popular tags used on the questions in this dataset:\n```bash\ncurl -XPOST \"http://localhost:7280/api/v1/stackoverflow/search\" -H 'Content-Type: application/json' -d '{\n \"query\": \"type:question\",\n \"max_hits\": 0,\n \"aggs\": {\n \"foo\": {\n \"terms\":{\n \"field\":\"tags\",\n \"size\": 10\n }\n }\n }\n}'\n```\n\nAs you are experimenting with different queries check out the server logs to see what's happening.\n\n:::note\n\nDon't forget to encode correctly the query params to avoid bad request (status 400).\n\n:::\n\n\n\n## Clean\n\nLet's do some cleanup by deleting the index:\n\n\n\n\n\n```bash\n./quickwit index delete --index stackoverflow\n```\n\n\n\n\n\n```bash\ncurl -XDELETE http://127.0.0.1:7280/api/v1/indexes/stackoverflow\n```\n\n\n\n\n\nCongrats! You can level up with the following tutorials to discover all Quickwit features.\n\n\n## TLDR\n\nRun the following command from within Quickwit's installation directory.\n\n```bash\ncurl -o stackoverflow-index-config.yaml https://raw.githubusercontent.com/quickwit-oss/quickwit/main/config/tutorials/stackoverflow/index-config.yaml\n./quickwit index create --index-config ./stackoverflow-index-config.yaml\ncurl -O https://quickwit-datasets-public.s3.amazonaws.com/stackoverflow.posts.transformed-10000.json\n./quickwit index ingest --index stackoverflow --input-path ./stackoverflow.posts.transformed-10000.json --force\n./quickwit index search --index stackoverflow --query \"search AND engine\"\n./quickwit index delete --index stackoverflow\n```\n\n\n## Next tutorials\n\n- [Search on logs with timestamp pruning](/docs/get-started/tutorials/tutorial-hdfs-logs)\n- [Setup a distributed search on AWS S3](/docs/get-started/tutorials/tutorial-hdfs-logs-distributed-search-aws-s3)\n" }, { "path": "docs/get-started/tutorials/_category_.yaml", "content": "label: 'Tutorials'\nposition: 2\ncollapsed: false\n" }, { "path": "docs/get-started/tutorials/prometheus-metrics.md", "content": "---\ntitle: Metrics with Grafana and Prometheus\ndescription: A simple tutorial to display Quickwit metrics with Grafana.\nicon_url: /img/tutorials/quickwit-logo.png\ntags: [grafana, prometheus, integration]\nsidebar_position: 2\n---\n\nIn this tutorial, you will learn how to set up Grafana to display Quickwit metrics using Prometheus. Grafana will visualize the metrics collected from Quickwit, allowing you to monitor its performance effectively.\n\n## Step 1: Create a Docker Compose File\n\nFirst, create a `docker-compose.yml` file in your project directory. This file will configure and run Quickwit, Prometheus, and Grafana as Docker services.\n\nHere’s the complete Docker Compose configuration:\n\n```yaml\nservices:\n quickwit:\n image: quickwit/quickwit\n environment:\n QW_ENABLE_OPENTELEMETRY_OTLP_EXPORTER: \"true\"\n OTEL_EXPORTER_OTLP_ENDPOINT: \"http://localhost:7281\"\n ports:\n - 7280:7280\n command: [\"run\"]\n\n grafana:\n image: grafana/grafana-oss\n container_name: grafana\n ports:\n - \"${MAP_HOST_GRAFANA:-127.0.0.1}:3000:3000\"\n environment:\n GF_INSTALL_PLUGINS: https://github.com/quickwit-oss/quickwit-datasource/releases/download/v0.4.6/quickwit-quickwit-datasource-0.4.6.zip;quickwit-quickwit-datasource\n GF_AUTH_DISABLE_LOGIN_FORM: \"true\"\n GF_AUTH_ANONYMOUS_ENABLED: \"true\"\n GF_AUTH_ANONYMOUS_ORG_ROLE: Admin\n\n prometheus:\n image: prom/prometheus:latest\n container_name: prometheus\n volumes:\n - ./prometheus.yml:/etc/prometheus/prometheus.yml # Ensure prometheus.yml exists in the same directory\n ports:\n - 9090:9090\n```\n\n### Explanation of Services\n\n- **Quickwit**: Runs the Quickwit service on port `7280`.\n- **Grafana**: Queries and displays data from Prometheus.\n- **Prometheus**: Collects metrics from Quickwit using the `/metrics` endpoint.\n\n## Step 2: Configure Prometheus\n\nPrometheus needs a configuration file to define how it scrapes metrics from Quickwit. Create a file named `prometheus.yml` in the same directory as your Docker Compose file with the following content:\n\n```yaml\nglobal:\n scrape_interval: 1s\n scrape_timeout: 1s\n\nscrape_configs:\n - job_name: quickwit\n metrics_path: /metrics\n static_configs:\n - targets:\n - quickwit:7280\n```\n\n## Step 3: Start the Services\n\nRun the following command in your terminal to start all services defined in the Docker Compose file:\n\n```bash\ndocker compose up\n```\n\nThis will launch Quickwit, Prometheus, and Grafana services.\n\n## Step 4: Configure Grafana to Use Prometheus\n\n1. Open Grafana in your browser at `http://localhost:3000`.\n2. Navigate to **Configuration** > **Data Sources**.\n3. Click **Add Data Source**, select **Prometheus**, and set the URL to `http://prometheus:9090`.\n4. Click **Save & Test** to verify the connection.\n\n## Step 5: Create or Use Pre-Configured Dashboards\n\nNow that Grafana is set up with Prometheus as a data source, you can create custom dashboards or use Quickwit's pre-configured dashboards:\n\n1. Go to the **Dashboards** section in Grafana.\n2. Import or create a new dashboard to visualize metrics.\n3. Alternatively, use one of Quickwit’s [pre-configured dashboards](../../operating/monitoring).\n\n" }, { "path": "docs/get-started/tutorials/trace-analytics-with-grafana.md", "content": "---\ntitle: Logs and Traces with Grafana\ndescription: A simple tutorial to use Grafana with Quickwit's datasource plugin.\nicon_url: /img/tutorials/quickwit-logo.png\ntags: [grafana, integration]\nsidebar_position: 2\n---\n\nIn this tutorial, we will set up a Grafana Dashboard showing Quickwit traces using Docker Compose.\n\nYou only need a few minutes to get Grafana working with Quickwit and build meaningful dashboards.\n\n## Create a Docker Compose recipe\n\nFirst, create a `docker-compose.yml` file. This file will define the services needed to run Quickwit with OpenTelemetry and Grafana with the Quickwit Datasource plugin.\n\nBelow is the complete Docker Compose configuration:\n\n```yaml\nversion: '3.0'\nservices:\n quickwit:\n image: quickwit/quickwit\n environment:\n QW_ENABLE_OPENTELEMETRY_OTLP_EXPORTER: \"true\"\n OTEL_EXPORTER_OTLP_ENDPOINT: \"http://localhost:7281\"\n ports:\n - 7280:7280\n command: [\"run\"]\n\n grafana:\n image: grafana/grafana-oss\n container_name: grafana\n ports:\n - \"${MAP_HOST_GRAFANA:-127.0.0.1}:3000:3000\"\n environment:\n GF_INSTALL_PLUGINS: https://github.com/quickwit-oss/quickwit-datasource/releases/download/v0.4.6/quickwit-quickwit-datasource-0.4.6.zip;quickwit-quickwit-datasource\n GF_AUTH_DISABLE_LOGIN_FORM: \"true\"\n GF_AUTH_ANONYMOUS_ENABLED: \"true\"\n GF_AUTH_ANONYMOUS_ORG_ROLE: Admin\n```\n\nThe default Grafana port is 3000. If this port is already taken, you can modify the port mapping, for example, changing 3000:3000 to 3100:3000 or any other available port.\n\nSave and run the recipe:\n\n```bash\n$ docker compose up\n```\n\nYou should be able to access Quickwit's UI on `http://localhost:7280/` and Grafana's UI on `http://localhost:3000/`.\n\n## Setting up the datasource\n\nIn Grafana, head to [Data Sources](http://localhost:3000/connections/datasources). If the plugin is installed correctly you should be able to find Quickwit in the list.\n\nWe're going to set up a new Quickwit data source looking at Quickwit's own OpenTelemetry traces, let's configure the datasource with the following parameters:\n\n- URL : `http://quickwit:7280/api/v1` _This uses the docker service name as the host_\n- Index ID : `otel-traces-v0_7`\n\nSave and test, you should obtain a confirmation that the datasource is correctly set up.\n\n\n![Quickwit Plugin configuration success](../../assets/images/grafana-ui-quickwit-datasource-plugin-success.png)\n\n\nYou can also set up a new Quickwit data source looking at Quickwit's own OpenTelemetry logs (or your own logs index), let's configure the datasource with the following parameters:\n\n- URL : `http://quickwit:7280/api/v1` _This uses the docker service name as the host_\n- Index ID : `otel-logs-v0_7`\n\n\n## Creating a dashboard\n\nYou can then [create a new dashboard](http://localhost:3000/dashboard/new) and add a visualization : you should be able to choose the traces quickwit datasource here.\n\nQuickwit sends itself its own traces, so you should already have data to display. Let's configure some panels !\n\n- a Table counting span_names \n - **Panel type** : Table\n - **Query**: _empty_\n - **Metric** : Count\n - **Group by** : Terms : `span_name` : order by Count\n- a Bar Chart showing the amount of tantivy searches per hour :\n - **Panel type**: Time Series\n - **Query** : \"span_name:tantivy_search\"\n - **Metric**: Count\n - **Group by** : Date Histogram : `span_start_timestamp_nanos` : Interval 1h\n- a Bar Chart showing the amount of ERROR logs per hour for the last 6 hours :\n - **Panel type**: Bar Chart\n - **Query**: \"service_name:quickwit AND events.event_attributes.level:ERROR\"\n - **Metric**: Count\n - **Group by** : Terms : `span_start_timestamp_nanos` : Interval 1h\n- another query on the same Bar Chart for WARN logs\n\n## The result\n\nHere's what your first dashboard can look like :\n\n![Quickwit Panel in Grafana Dashboard](../../assets/images/screenshot-grafana-tutorial-dashboard.png)\n\n\n" }, { "path": "docs/get-started/tutorials/tutorial-hdfs-logs-distributed-search-aws-s3.md", "content": "---\ntitle: Distributed search on AWS S3\ndescription: Index log entries on AWS S3 using an EC2 instance and launch a distributed cluster.\ntags: [aws, integration]\nicon_url: /img/tutorials/aws-logo.png\nsidebar_position: 6\n---\n\nIn this guide, we will index about 40 million log entries (13 GB decompressed) on AWS S3 using an EC2 instance and launch a three-node distributed search cluster.\n\nExample of a log entry:\n```json\n{\n \"timestamp\": 1460530013,\n \"severity_text\": \"INFO\",\n \"body\": \"PacketResponder: BP-108841162-10.10.34.11-1440074360971:blk_1074072698_331874, type=HAS_DOWNSTREAM_IN_PIPELINE terminating\",\n \"resource\": {\n \"service\": \"datanode/01\"\n },\n \"attributes\": {\n \"class\": \"org.apache.hadoop.hdfs.server.datanode.DataNode\"\n }\n}\n```\n\n:::caution\n\nBefore using Quickwit with an object storage, check out our [advice](../../operating/aws-costs) for deploying on AWS S3 to avoid some bad surprises at the end of the month.\n\n:::\n\nFirst of all, let's create an EC2 instance, install a Quickwit binary, and [configure it](../../guides/aws-setup) to let Quickwit access your S3 buckets. This instance will be used for indexing our dataset (note that you can also index your dataset from your local machine if it has the rights to read/write on AWS S3).\n\n## Install\n\n```bash\ncurl -L https://install.quickwit.io | sh\ncd quickwit-v*/\n```\n\n## Configure Quickwit with S3\n\nLet's define the S3 path where we want to store our indexes.\n\n```bash\nexport S3_PATH=s3://{path/to/bucket}/indexes\n```\n\n:::note\nYou'll want to include the necessary authorization for the given bucket, this can be done by setting the `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`\nenvironment variables, or via the AWS credentials file. Usually located at `~/.aws/credentials`.\n\nFor more info check out [our AWS setup guide](https://quickwit.io/docs/guides/aws-setup)\n:::\n\nNow we can create a Quickwit config file.\n\n```bash\n# Create Quickwit config file.\necho \"version: 0.7\nnode_id: searcher-1\nlisten_address: 0.0.0.0\nmetastore_uri: ${S3_PATH}\ndefault_index_root_uri: ${S3_PATH}\n\" > config.yaml\n```\n\n> You can also pass environment variables directly:\n> ```yaml\n> # config.yaml\n> node_id: searcher-1\n> listen_address: 0.0.0.0\n> version: 0.7\n> metastore_uri: ${S3_PATH}\n> default_index_root_uri: ${S3_PATH}\n>```\n\nWe are now ready to start Quickwit.\n\n```bash\n./quickwit run --config config.yaml\n```\n\n## Create your index\n\n```bash\n# First, download the hdfs logs config from Quickwit repository.\ncurl -o hdfs_logs_index_config.yaml https://raw.githubusercontent.com/quickwit-oss/quickwit/main/config/tutorials/hdfs-logs/index-config.yaml\n```\n\nThe index config defines five fields: `timestamp`, `tenant_id`, `severity_text`, `body`, and one JSON field\nfor the nested values `resource.service`, we could use an object field here and maintain a fixed schema, but for convenience we're going to use a JSON field.\nIt also sets the `default_search_fields`, the `tag_fields`, and the `timestamp_field`. The `timestamp_field` and `tag_fields` are\nused by Quickwit for [splits pruning](../../overview/architecture) at query time to boost search speed. \nCheck out the [index config docs](../../configuration/index-config) for more details.\n\n```yaml title=\"hdfs_logs_index_config.yaml\"\nversion: 0.7\n\nindex_id: hdfs-logs\n\ndoc_mapping:\n field_mappings:\n - name: timestamp\n type: datetime\n input_formats:\n - unix_timestamp\n output_format: unix_timestamp_secs\n fast_precision: seconds\n fast: true\n - name: tenant_id\n type: u64\n - name: severity_text\n type: text\n tokenizer: raw\n - name: body\n type: text\n tokenizer: default\n record: position\n - name: resource\n type: json\n tokenizer: raw\n tag_fields: [tenant_id]\n timestamp_field: timestamp\n\nsearch_settings:\n default_search_fields: [severity_text, body]\n```\n\nWe can now create the index with the `create` subcommand.\n\n```bash\n./quickwit index create --index-config hdfs_logs_index_config.yaml\n```\n\n:::note\n\nThis step can also be executed on your local machine. The `create` command creates the index locally and then uploads a \njson file `metastore.json` to your bucket at `s3://path-to-your-bucket/hdfs-logs/metastore.json`.\n\n:::\n\n## Index logs\nThe dataset is a compressed [NDJSON file](https://quickwit-datasets-public.s3.amazonaws.com/hdfs-logs-multitenants.json.gz). \nInstead of downloading and indexing the data in separate steps, we will use pipes to send a decompressed stream to Quickwit directly.\n\n```bash\nwget https://quickwit-datasets-public.s3.amazonaws.com/hdfs-logs-multitenants.json.gz\ngunzip -c hdfs-logs-multitenants.json.gz | ./quickwit index ingest --index hdfs-logs\n```\n\n:::note\n\n8GB of RAM is enough to index this dataset; an instance like `t4g.large` with 8GB and 2 vCPU indexed this dataset in less than 10 minutes \n(provided that you have some CPU credits).\n\nThis step can also be done on your local machine. \nThe `ingest` subcommand generates locally [splits](../../overview/architecture) of 10 million documents and will upload \nthem on your bucket. Concretely, each split is a bundle of index files and metadata files.\n\n:::\n\n\nYou can check it's working by using `search` subcommand and look for `ERROR` in `severity_text` field:\n```bash\n./quickwit index search --index hdfs-logs --query \"severity_text:ERROR\"\n```\n\nwhich returns the json\n\n```json\n{\n \"num_hits\": 345,\n \"hits\": [\n {\n \"attributes\": {\n \"class\": \"org.apache.hadoop.hdfs.server.datanode.DataNode\"\n },\n \"body\": \"RECEIVED SIGNAL 15: SIGTERM\",\n \"resource\": {\n \"service\": \"datanode/16\"\n },\n \"severity_text\": \"ERROR\",\n \"tenant_id\": 51,\n \"timestamp\": 1469687755\n },\n ...\n ],\n \"elapsed_time_micros\": 522542\n}\n```\n\nYou can see that this query has 345 hits. In this case for the first run, the server responded in 523 milliseconds.\nSubsequent runs use the cached metastore and can be resolved in under 100 milliseconds.\n\nNow that we have indexed the logs and can search from one instance, it's time to configure and start two other instances to form a cluster.\n\n## Start two more instances\n\nQuickwit needs a port `rest.listen_port` for serving the HTTP rest API via TCP as well as maintaining the cluster formation via UDP. \nAlso, it needs `{rest.listen_port} + 1` for gRPC communication between instances.\n\nIn AWS, you can create a security group to group these inbound rules. Check out the [network section](../../guides/aws-setup) of our AWS setup guide.\n\nTo make things easier, let's create a security group that opens the TCP/UDP port range [7200-7300]. \nNext, create three EC2 instances using the previously created security group. Take note of each instance's public IP address.\n\nNow ssh into the first EC2 instance, install Quickwit, and [configure the environment](../../guides/aws-setup) to let Quickwit access the index S3 buckets.\n\nLet's install Quickwit on the second and third EC2 instances.\n\n```bash\ncurl -L https://install.quickwit.io | sh\ncd quickwit-v*/\n```\n\nAnd configure the environment so instances can form a cluster:\n\n```bash\nexport S3_PATH=s3://{path/to/bucket}/indexes\nexport IP_NODE_1={first-ec2-instance-public-ip}\n```\n\n```bash\n# configuration for our second node\necho \"version: 0.7\nnode_id: searcher-2\nmetastore_uri: ${S3_PATH}\ndefault_index_root_uri: ${S3_PATH}\nlisten_address: 0.0.0.0\npeer_seeds:\n - ${IP_NODE_1} # searcher-1\n\" > config.yaml\n\n# Start a Quickwit searcher.\n./quickwit run --service searcher --config config.yaml\n```\n\n```bash\n# configuration for our third node\necho \"version: 0.7\nnode_id: searcher-3\nlisten_address: 0.0.0.0\npeer_seeds:\n - ${IP_NODE_1} # searcher-1\nmetastore_uri: ${S3_PATH}\ndefault_index_root_uri: ${S3_PATH}\n\" > config.yaml\n\n# Start a Quickwit searcher.\n./quickwit run --service searcher --config config.yaml\n```\n\n\nYou will see in the terminal the confirmation that the instance has joined the existing cluster. Example of such a log:\n\n```\n2023-03-19T16:44:56.918Z INFO quickwit_cluster::cluster: Joining cluster. cluster_id=quickwit-default-cluster node_id=searcher-2 enabled_services={Searcher} gossip_listen_addr=0.0.0.0:7280 gossip_advertise_addr=172.31.30.168:7280 grpc_advertise_addr=172.31.30.168:7281 peer_seed_addrs=172.31.91.203:7280\n```\n\nNow we can query one of our instance directly by issuing http requests to one of the nodes rest API endpoint.\n\n```\ncurl -v \"http://0.0.0.0:7280/api/v1/hdfs-logs/search?query=severity_text:ERROR\"\n```\n\nCheck out the logs of all instances and you will see that all nodes are working.\n\n## Load balancing incoming requests\n\nNow that you have a search cluster, ideally, you will want to load balance external requests. \nThis can quickly be done by adding an AWS load balancer to listen to incoming HTTP or HTTPS traffic and forward it to a target group.\nYou can now play with your cluster, kill processes randomly, add/remove new instances, and keep calm.\n\n## Clean\n\nLet's do some cleanup by deleting the index:\n\n```bash\n./quickwit index delete --index hdfs-logs\n```\n\nAlso remember to remove the security group to protect your EC2 instances. You can just remove the instances if you don't need them.\n\nCongratz! You finished this tutorial!\n\nTo continue your Quickwit journey, check out the [search REST API reference](/docs/reference/rest-api) or the [query language reference](/docs/reference/query-language).\n" }, { "path": "docs/get-started/tutorials/tutorial-hdfs-logs.md", "content": "---\ntitle: Index a logging dataset locally\ndescription: Index log entries on a local machine.\ntags: [self-hosted, setup]\nicon_url: /img/quickwit-icon.svg\nsidebar_position: 3\n---\n\nimport Tabs from '@theme/Tabs';\nimport TabItem from '@theme/TabItem';\n\nIn this guide, we will index about 20 million log entries (7 GB decompressed) on a local machine. If you want to start a server with indexes on AWS S3 with several search nodes, check out the [tutorial for distributed search](tutorial-hdfs-logs-distributed-search-aws-s3.md).\n\nHere is an example of a log entry:\n```json\n{\n \"timestamp\": 1460530013,\n \"severity_text\": \"INFO\",\n \"body\": \"PacketResponder: BP-108841162-10.10.34.11-1440074360971:blk_1074072698_331874, type=HAS_DOWNSTREAM_IN_PIPELINE terminating\",\n \"resource\": {\n \"service\": \"datanode/01\"\n },\n \"attributes\": {\n \"class\": \"org.apache.hadoop.hdfs.server.datanode.DataNode\"\n },\n \"tenant_id\": 58\n}\n```\n\n\n## Install\n\nLet's download and install Quickwit.\n\n```bash\ncurl -L https://install.quickwit.io | sh\ncd quickwit-v*/\n```\n\nOr pull and run the Quickwit binary in an isolated Docker container.\n\n```bash\ndocker run quickwit/quickwit --version\n```\n\n## Start a Quickwit server\n\n\n\n\n\n```bash\n./quickwit run\n```\n\n\n\n\n\n```bash\ndocker run --rm -v $(pwd)/qwdata:/quickwit/qwdata -p 127.0.0.1:7280:7280 quickwit/quickwit run\n```\n\nYou may need to specify the platform if you are using Apple silicon based macOS system with the `--platform linux/amd64` flag. You can also safely ignore jemalloc warnings.\n\n\n\n\n\n\n## Create your index\n\nLet's create an index configured to receive these logs.\n\n```bash\n# First, download the hdfs logs config from Quickwit repository.\ncurl -o hdfs_logs_index_config.yaml https://raw.githubusercontent.com/quickwit-oss/quickwit/main/config/tutorials/hdfs-logs/index-config.yaml\n```\n\nThe index config defines five fields: `timestamp`, `tenant_id`, `severity_text`, `body`, and one JSON field\nfor the nested values `resource.service`, we could use an object field here and maintain a fixed schema, but for convenience we're going to use a JSON field.\nIt also sets the `default_search_fields`, the `tag_fields`, and the `timestamp_field`.\nThe `timestamp_field` and `tag_fields` are used by Quickwit for [splits pruning](../../overview/concepts/querying.md#time-sharding) at query time to boost search speed. \nCheck out the [index config docs](../../configuration/index-config) for more details.\n\n```yaml title=\"hdfs-logs-index.yaml\"\nversion: 0.7\n\nindex_id: hdfs-logs\n\ndoc_mapping:\n field_mappings:\n - name: timestamp\n type: datetime\n input_formats:\n - unix_timestamp\n output_format: unix_timestamp_secs\n fast_precision: seconds\n fast: true\n - name: tenant_id\n type: u64\n - name: severity_text\n type: text\n tokenizer: raw\n - name: body\n type: text\n tokenizer: default\n record: position\n - name: resource\n type: json\n tokenizer: raw\n tag_fields: [tenant_id]\n timestamp_field: timestamp\n\nsearch_settings:\n default_search_fields: [severity_text, body]\n```\n\nNow let's create the index with the `create` subcommand (assuming you are inside Quickwit install directory):\n\n\n\n\n\n```bash\n./quickwit index create --index-config hdfs_logs_index_config.yaml\n```\n\n\n\n\n\n```bash\ncurl -XPOST http://localhost:7280/api/v1/indexes -H \"content-type: application/yaml\" --data-binary @hdfs_logs_index_config.yaml\n```\n\n\n\n\n\n\nYou're now ready to fill the index.\n\n## Index logs\nThe dataset is a compressed [NDJSON file](https://quickwit-datasets-public.s3.amazonaws.com/hdfs-logs-multitenants.json.gz).\nInstead of downloading it and then indexing the data, we will use pipes to directly send a decompressed stream to Quickwit.\nThis can take up to 10 minutes on a modern machine, the perfect time for a coffee break.\n\n\n\n\n\n```bash\ncurl https://quickwit-datasets-public.s3.amazonaws.com/hdfs-logs-multitenants.json.gz | gunzip | ./quickwit index ingest --index hdfs-logs\n```\n\n\n\n\n\n```bash\ncurl https://quickwit-datasets-public.s3.amazonaws.com/hdfs-logs-multitenants.json.gz | gunzip | docker run -v $(pwd)/qwdata:/quickwit/qwdata -i quickwit/quickwit index ingest --index hdfs-logs\n```\n\n\n\n\n\n\n\nIf you are in a hurry, use the sample dataset that contains 10 000 documents, we will use this dataset for the example queries:\n\n\n\n\n\n```bash\ncurl https://quickwit-datasets-public.s3.amazonaws.com/hdfs-logs-multitenants-10000.json | ./quickwit index ingest --index hdfs-logs\n```\n\n\n\n\n\nOn macOS or Windows:\n\n```bash\ncurl https://quickwit-datasets-public.s3.amazonaws.com/hdfs-logs-multitenants-10000.json | docker run -v $(pwd)/qwdata:/quickwit/qwdata -i quickwit/quickwit index ingest --index hdfs-logs --endpoint http://host.docker.internal:7280\n```\n\nOn linux:\n\n```bash\ncurl https://quickwit-datasets-public.s3.amazonaws.com/hdfs-logs-multitenants-10000.json | docker run --network=host -v $(pwd)/qwdata:/quickwit/qwdata -i quickwit/quickwit index ingest --index hdfs-logs --endpoint http://127.0.0.1:7280\n```\n\n\n\n\n\n```bash\nwget https://quickwit-datasets-public.s3.amazonaws.com/hdfs-logs-multitenants-10000.json\ncurl -XPOST http://localhost:7280/api/v1/hdfs-logs/ingest -H \"content-type: application/json\" --data-binary @hdfs-logs-multitenants-10000.json\n```\n\n\n\n\n\nYou can check it's working by searching for `INFO` in `severity_text` field:\n\n\n\n\n\n```bash\n./quickwit index search --index hdfs-logs --query \"severity_text:INFO\"\n```\n\n\n\n\n\nOn macOS or Windows:\n\n```bash\ndocker run -v $(pwd)/qwdata:/quickwit/qwdata quickwit/quickwit index search --index hdfs-logs --query \"severity_text:INFO\" --endpoint http://host.docker.internal:7280\n```\n\nOn linux:\n\n```bash\ndocker run --network=host -v $(pwd)/qwdata:/quickwit/qwdata quickwit/quickwit index search --index hdfs-logs --query \"severity_text:INFO\" --endpoint http://127.0.0.1:7280\n```\n\n\n\n\n\n:::note\n\nThe `ingest` subcommand generates [splits](../../overview/architecture) of 5 million documents. Each split is a small piece of index represented by a file in which index files and metadata files are saved.\n\n:::\n\n\nThe query which returns the json:\n\n```json\n{\n \"num_hits\": 10000,\n \"hits\": [\n {\n \"body\": \"Receiving BP-108841162-10.10.34.11-1440074360971:blk_1073836032_95208 src: /10.10.34.20:60300 dest: /10.10.34.13:50010\",\n \"resource\": {\n \"service\": \"datanode/03\"\n },\n \"severity_text\": \"INFO\",\n \"tenant_id\": 58,\n \"timestamp\": 1440670490\n }\n ...\n ],\n \"elapsed_time_micros\": 2490\n}\n```\n\nThe index config shows that we can use the timestamp field parameters `start_timestamp` and `end_timestamp` and benefit from time pruning. \nBehind the scenes, Quickwit will only query [splits](../../overview/architecture) that have logs in this time range.\n\nLet's use these parameters with the following query:\n\n```bash\ncurl 'http://127.0.0.1:7280/api/v1/hdfs-logs/search?query=severity_text:INFO&start_timestamp=1440670490&end_timestamp=1450670490'\n```\n\n## Clean\n\nLet's do some cleanup by deleting the index:\n\n\n\n\n\n```bash\n./quickwit index delete --index hdfs-logs\n```\n\n\n\n\n\n```bash\ncurl -XDELETE http://127.0.0.1:7280/api/v1/indexes/hdfs-logs\n```\n\n\n\n\n\nCongratz! You finished this tutorial!\n\n\nTo continue your Quickwit journey, check out the [tutorial for distributed search](tutorial-hdfs-logs-distributed-search-aws-s3.md) or dig into the [search REST API](/docs/reference/rest-api) or [query language](/docs/reference/query-language).\n" }, { "path": "docs/get-started/tutorials/tutorial-jaeger.md", "content": "---\ntitle: Traces with Jaeger\nsidebar_position: 2\n---\n\nIn this quick start guide, we will set up a Quickwit instance and analyze its own traces with Jaeger using Docker Compose.\n\nYou only need a minute to get Jaeger working with Quickwit storage backend.\n\n## Start Quickwit and Jaeger\n\nLet's use `docker compose` with the following configuration:\n\n```yaml title=\"docker-compose.yaml\"\nversion: \"3\"\n\nservices:\n quickwit:\n image: quickwit/quickwit:${QW_VERSION:-0.8.1}\n volumes:\n - ./qwdata:/quickwit/qwdata\n ports:\n - 7280:7280\n environment:\n - QW_ENABLE_OPENTELEMETRY_OTLP_EXPORTER=true\n - OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:7281\n command: [\"run\"]\n\n jaeger-query:\n image: jaegertracing/jaeger-query:1.60\n ports:\n - 16686:16686\n environment:\n - SPAN_STORAGE_TYPE=grpc\n - GRPC_STORAGE_SERVER=quickwit:7281\n - GRPC_STORAGE_TLS=false\n```\n\nAs you can see in the docker compose file, Quickwit is configured to send its own traces `OTEL_EXPORTER_OTLP_ENDPOINT` to itself `http://localhost:7281`.\nOn the other side, Jaeger is configured to use a gRPC storage server `quickwit:7281`.\n\nSave and run the recipe:\n\n```bash\n$ docker compose up\n```\n\nYou should be able to access Quickwit's UI on `http://localhost:7280/` and Jager's UI on `http://localhost:16686/`.\n\n\n## Searching and view traces in Jaeger\n\nQuickwit generates many traces, let's take a look at some of them:\n- `find_traces`: generated by the \"Find traces\" Jaeger button.\n- `get_operations`: generated by Jaeger when it is fetching the list of operations.\n- `get_services`: generated by Jaeger when it is fetching the list of services.\n- `ingest-spans`: generated when Quickwit receives spans on the gRPC OTLP API.\n- ...\n\nHere are the screenshots of the search and trace view:\n\n![Jaeger search view](../../assets/images/jaeger-ui-quickwit-search-traces.png)\n![Jaeger trace view](../../assets/images/jaeger-ui-quickwit-trace-view.png)\n\n## Searching traces with Quickwit UI\n\nYou can also use the Quickwit UI at [http://localhost:7280](http://localhost:7280) to search traces.\n\nHere are a couple of query examples:\n- `service_name:quickwit AND events.event_attributes.level:INFO`\n- `span_duration_millis:>100`\n- `resource_attributes.service.version:v0.8.1`\n- `service_name:quickwit`\n\nThat's it! You can level up with the following tutorials to discover all Quickwit features.\n\n## Next tutorials\n\n- [Send traces using an OTEL collector](/docs/distributed-tracing/send-traces/using-otel-collector.md)\n- [Send traces from a python web server](/docs/distributed-tracing/send-traces/using-otel-sdk-python.md)\n" }, { "path": "docs/guides/_category_.yaml", "content": "label: 'Guides'\nposition: 8\ncollapsed: true\n" }, { "path": "docs/guides/aws-setup.md", "content": "---\ntitle: AWS cluster setup\nsidebar_position: 3\n---\n\nSetting up a Quickwit cluster on AWS requires the configuration of three elements:\n- AWS credentials\n- AWS region\n- Network configuration\n\n## AWS credentials\n\nWhen starting a node, Quickwit attempts to find AWS credentials using the credential provider chain implemented by [rusoto_core::ChainProvider](https://docs.rs/rusoto_credential/latest/rusoto_credential/struct.ChainProvider.html) and looks for credentials in this order:\n\n1. Environment variables `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, or `AWS_SESSION_TOKEN` (optional).\n\n2. Credential profiles file, typically located at `~/.aws/credentials` or otherwise specified by the `AWS_SHARED_CREDENTIALS_FILE` and `AWS_PROFILE` environment variables if set and not empty.\n\n3. Amazon ECS container credentials, loaded from the Amazon ECS container if the environment variable `AWS_CONTAINER_CREDENTIALS_RELATIVE_URI` is set.\n\n4. Instance profile credentials, used on Amazon EC2 instances, and delivered through the Amazon EC2 metadata service.\n\nAn error is returned if no credentials are found in the chain.\n\n## AWS region\n\nQuickwit attempts to find an AWS region in multiple locations and with the following order of precedence:\n\n1. Environment variables (`AWS_REGION` then `AWS_DEFAULT_REGION`)\n\n2. Config file, typically located at `~/.aws/config` or otherwise specified by the `AWS_CONFIG_FILE` environment variable if set and not empty.\n\n3. Amazon EC2 instance metadata service indicating the region of the currently running Amazon EC2 instance.\n\n4. Default value: `us-east-1`\n\n:::note\n\nAWS credentials or region resolution may take a few seconds, especially if the Amazon EC2 instance metadata service is slow or unavailable.\n\n:::\n\n## IAM permissions\n\n### Amazon S3\n\nRequired authorized actions:\n- `ListBucket` (on the bucket directly)\n- `GetObject`\n- `PutObject`\n- `DeleteObject`\n- `ListMultipartUploadParts`\n- `AbortMultipartUpload`\n\nHere is an example of a bucket policy:\n```json\n{\n \"Version\": \"2012-10-17\",\n \"Statement\": [\n {\n \"Effect\": \"Allow\",\n \"Action\": [\n \"s3:ListBucket\"\n ],\n \"Resource\": [\n \"arn:aws:s3:::my-bucket\"\n ]\n },\n {\n \"Effect\": \"Allow\",\n \"Action\": [\n \"s3:GetObject\",\n \"s3:PutObject\",\n \"s3:DeleteObject\",\n \"s3:ListMultipartUploadParts\",\n \"s3:AbortMultipartUpload\"\n ],\n \"Resource\": [\n \"arn:aws:s3:::my-bucket/*\"\n ]\n }\n ]\n}\n```\n\nYou can run the following commands to verify that AWS credentials, region, and IAM permissions are properly configured for Amazon S3:\n\n```bash\nMY_BUCKET=\naws s3 ls $MY_BUCKET\necho \"Hello, World!\" | aws s3 cp - $MY_BUCKET/hello\naws s3 ls $MY_BUCKET/hello\naws s3 cp $MY_BUCKET/hello -\naws s3 rm $MY_BUCKET/hello\n```\n\n### Amazon Kinesis\n\n- `GetRecords`\n- `GetShardIterator`\n- `ListShards`\n\nYou can run the following commands to verify that AWS credentials, region, and IAM permissions are properly configured for Amazon Kinesis:\n\n```bash\nMY_STREAM=\n\n# List the shards in the stream and select the first one.\nSHARD_ID=$(\n aws kinesis list-shards --stream-name $MY_STREAM \\\n | jq -r .Shards[0].ShardId\n)\n\n# Get a shard iterator for the selected shard.\nSHARD_ITERATOR=$(\n aws kinesis get-shard-iterator --stream-name $MY_STREAM \\\n --shard-id $SHARD_ID \\\n --shard-iterator-type TRIM_HORIZON \\\n | jq -r .ShardIterator\n)\n\n# Fetch some records from the shard and display the first one.\naws kinesis get-records --shard-iterator $SHARD_ITERATOR | jq -r .Records[0]\n```\n\n## Network configuration\n\n### Security groups\n\nTo communicate with each other, nodes must reside in security groups that allow inbound and outbound traffic on one UDP port and two TCP ports. Please, refer to the [ports configuration](/configuration/ports-config.md) page for more details.\n\n## Common errors\n\nIf you set the wrong credentials, you will see this error message with `Unauthorized` in your terminal:\n\n```bash\nCommand failed: Another error occurred. `Metastore error`. Cause: `StorageError(kind=Unauthorized, source=failed to fetch object: s3://quickwit-dev/my-hdfs/metastore.json)`\n```\n\nIf you put the wrong region, you will see this one:\n\n```bash\nCommand failed: Another error occurred. `Metastore error`. Cause: `StorageError(kind=Internal, source=failed to fetch object: s3://your-bucket/your-index/metastore.json)`.\n```\n" }, { "path": "docs/guides/schemaless.md", "content": "---\ntitle: Schemaless\nsidebar_position: 1\n---\n\n# Strict schema or schemaless?\n\nQuickwit lets you place the cursor on how strict you would like your schema to be. In other words, it is possible to operate Quickwit with a very strict mapping, in an entirely schemaless manner, and anywhere in between. Let's see how this works!\n\n:::note\n\nTo execute the CLI commands throughout this guide, [install](/docs/get-started/installation.md) Quickwit and start a server in a terminal with the following command:\n\n```bash\n./quickwit run\n```\n\n:::\n\n## A strict mapping\n\nThat's the most straightforward approach.\nAs a user, you need to precisely define the list of fields to be ingested by Quickwit.\n\nFor instance, a reasonable mapping for an application log could be:\n\n```yaml title=my_strict_index.yaml\nversion: 0.7\n\nindex_id: my_strict_index\n\ndoc_mapping:\n mode: strict # <--- The mode attribute\n field_mappings:\n - name: timestamp\n type: datetime\n input_formats:\n - unix_timestamp\n output_format: unix_timestamp_secs\n fast_precision: seconds\n fast: true\n - name: server\n type: text\n tokenizer: raw\n - name: message\n type: text\n record: position\n - name: severity\n tokenizer: raw\n timestamp_field: timestamp\n\nsearch_settings:\n default_search_fields: [severity, message]\n\nindexing_settings:\n commit_timeout_secs: 30\n```\n\nThe `mode` attribute controls what should be done if an ingested document\ncontains a field that is not defined in the document mapping. By default, your index is in the `dynamic` mode. In `dynamic` mode, the fields that do not appear in the document mapping will be indexed in a schemaless fashion.\nSee details in the [dynamic mode section](#dynamic-mode).\n\n\nIf `mode` is set to `strict` on the other hand, documents containing fields\nthat are not defined in the mapping will be entirely discarded.\n\nFinally the last possible value for `mode` is `lenient`. In lenient mode, fields that are not present in the field mapping will simply be ignored.\n\n## The dynamic mode: schemaless with a partial schema {#dynamic-mode}\n\n`mode` can take the value: `dynamic`.\nWhen set to dynamic, all extra fields will actually be mapped using a catch-all configuration.\n\nBy default, this catch-all configuration indexes and stores all of these fields, but this can be configured by setting the [`dynamic_mapping` attribute](../configuration/index-config#mode).\nA minimalist, yet perfectly valid and useful index configuration is then:\n\n```yaml title=my_dynamic_index.yaml\nversion: 0.7\nindex_id: my_dynamic_index\ndoc_mapping:\n mode: dynamic\n```\n\nThis configuration makes it possible to ingest any JSON object and search them.\n\nHowever, the dynamic mode can also be used in conjunction with field mappings.\nThis combination is especially powerful for event logs which cannot be mapped to a single schema.\n\nFor instance, let's consider the following user event log:\n\n```json file title=my_logs.json\n{\n \"timestamp\": 1653021741,\n \"user_id\": \"8705a7fak\",\n \"event_type\": \"login\",\n \"ab_groups\": [\"phoenix-red-ux\"]\n}\n{\n \"timestamp\": 1653021746,\n \"user_id\": \"7618fe06\",\n \"event_type\": \"order\",\n \"ab_groups\": [\"phoenix-red-ux\", \"new-ranker\"],\n \"cart\": [\n {\n \"product_id\": 120391,\n \"product_description\": \"Cherry Pi: A single-board computer that is compatible...\"\n }\n ]\n}\n{\n \"timestamp\": 1653021748,\n \"user_id\": \"8705a7fak\",\n \"event_type\": \"login\",\n \"ab_groups\": [\"phoenix-red-ux\"]\n}\n```\n\nEach event type comes with its own set of attributes. Declaring our mapping as the union of all of these event-specific mappings would be a tedious exercise.\n\nInstead, we can cherry-pick the fields that are common to all of the logs, and rely on dynamic mode to handle the rest.\n\n```yaml title=my_dynamic_index.yaml\nversion: 0.7\nindex_id: my_dynamic_index\ndoc_mapping:\n mode: dynamic\n field_mappings:\n - name: timestamp\n type: datetime\n input_formats:\n - unix_timestamp\n output_format: unix_timestamp_secs\n fast_precision: seconds\n fast: true\n - name: user_id\n type: text\n tokenizer: raw\n - name: event_type\n type: text\n tokenizer: raw\n timestamp_field: timestamp\n\nindexing_settings:\n commit_timeout_secs: 30 # <--- Your document will be searchable ~30 seconds after you ingest them.\n```\n\nOur index is now ready to handle queries like this:\n\n```\nevent_type:order AND cart.product_id:120391\n```\n\nExecute the following commands to create the index, ingest a few documents and search through them:\n\n```bash\ncat << EOF > my_dynamic_index.yaml\nversion: 0.7\nindex_id: my_dynamic_index\ndoc_mapping:\n mode: dynamic\n field_mappings:\n - name: timestamp\n type: datetime\n input_formats:\n - unix_timestamp\n output_format: unix_timestamp_secs\n fast_precision: seconds\n fast: true\n - name: user_id\n type: text\n tokenizer: raw\n - name: event_type\n type: text\n tokenizer: raw\n timestamp_field: timestamp\n\nindexing_settings:\n commit_timeout_secs: 30\nEOF\n\n# Create index.\n./quickwit index create --index-config ./my_dynamic_index.yaml --overwrite --yes\n\ncat << EOF > my_logs.json\n{\"timestamp\":1653021741,\"user_id\":\"8705a7fak\",\"event_type\":\"login\",\"ab_groups\":[\"phoenix-red-ux\"]}\n{\"timestamp\":1653021746,\"user_id\":\"7618fe06\",\"event_type\":\"order\",\"ab_groups\":[\"phoenix-red-ux\",\"new-ranker\"],\"cart\":[{\"product_id\":120391,\"product_description\":\"Cherry Pi: A single-board computer that is compatible...\"}]}\n{\"timestamp\":1653021748,\"user_id\":\"8705a7fak\",\"event_type\":\"login\",\"ab_groups\":[\"phoenix-red-ux\"]}\nEOF\n\n# Ingest documents.\n./quickwit index ingest --index my_dynamic_index --input-path my_logs.json --force\n\n# Execute search query.\n./quickwit index search --index my_dynamic_index --query \"event_type:order AND cart.product_id:120391\n\n```\n\n## A schema with schemaless pockets\n\nSome logs are isolating these event-specific attributes in a\nsub-field. For instance, let's have a look at an OpenTelemetry JSON log.\n\n```json title=otel_logs.json\n{\n \"Timestamp\": 1653028151,\n \"Attributes\": {\n \"split_id\": \"28f897f2-0419-4d88-8abc-ada72b4b5256\"\n },\n \"Resource\": {\n \"service\": \"donut_shop\",\n \"k8s_pod_uid\": \"27413708-876b-4652-8ca4-50e8b4a5caa2\"\n },\n \"TraceId\": \"f4dbb3edd765f620\",\n \"SpanId\": \"43222c2d51a7abe3\",\n \"SeverityText\": \"INFO\",\n \"SeverityNumber\": 9,\n \"Body\": \"merge ended\"\n}\n```\n\nIn this log, the `Attributes` and the `Resource` fields contain arbitrary key-values.\n\nQuickwit 0.3 introduced a JSON field type to handle this use case.\nA good index configuration here could be:\n\n```yaml title=otel_logs.yaml\nversion: 0.7\nindex_id: otel_logs\ndoc_mapping:\n mode: dynamic\n field_mappings:\n - name: Timestamp\n type: datetime\n fast: true\n input_formats:\n - unix_timestamp\n output_format: unix_timestamp_secs\n fast_precision: seconds\n fast: true\n - name: Attributes\n type: json\n tokenizer: raw\n - name: Resource\n type: json\n tokenizer: raw\n - name: TraceId\n type: text\n tokenizer: raw\n - name: SpanId\n type: text\n tokenizer: raw\n - name: SeverityText\n type: text\n tokenizer: raw\n fast: true\n - name: Body\n type: text\n timestamp_field: Timestamp\n \nsearch_settings:\n default_search_fields: [SeverityText, Body, Attributes, Resource]\n\nindexing_settings:\n commit_timeout_secs: 10\n```\n\nWe can now naturally search our logs with the following query:\n\n```\nmerge AND service:donuts_shop\n```\n\nLet's execute the following commands to create the index, ingest a document and execute a search query:\n\n```bash\n# Create index.\n./quickwit index create --index-config ./otel_logs.yaml --overwrite --yes\n\ncat << EOF > otel_logs.json\n{\"Timestamp\":1653028151,\"Attributes\":{\"split_id\":\"28f897f2-0419-4d88-8abc-ada72b4b5256\"},\"Resource\":{\"service\":\"donut_shop\",\"k8s_pod_uid\":\"27413708-876b-4652-8ca4-50e8b4a5caa2\"},\"TraceId\":\"f4dbb3edd765f620\",\"SpanId\":\"43222c2d51a7abe3\",\"SeverityText\":\"INFO\",\"SeverityNumber\":9,\"Body\":\"merge ended\"}\nEOF\n\n# Ingest documents.\n./quickwit index ingest --index otel_logs --input-path otel_logs.json --force\n\n# Execute search query.\n./quickwit index search --index otel_logs --query \"merge AND service:donut_shop\"\n\n```\n" }, { "path": "docs/guides/storage-setup/_category_.yaml", "content": "label: 'Storage Setup'\nposition: 2\ncollapsed: true\n" }, { "path": "docs/guides/storage-setup/aws-s3.md", "content": "---\ntitle: AWS S3\nsidebar_position: 1\n---\n\nIn this guide, you will learn how to configure a Quickwit [storage](../../configuration/storage-config) for Amazon S3.\n\n## Set your AWS credentials\n\nA simple way to do it is to declare the environment variables `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`. For more details, read our guide on [AWS setup](../aws-setup).\n\n## Set the Metastore URI and default index URI\n\nHere is an example of how to set up your [node config file](../../configuration/node-config) with S3:\n\n```yaml\nmetastore_uri: s3://{my-bucket}/indexes\ndefault_index_uri: s3://{my-bucket}/indexes\n```\n\n## Set the Index URI\n\nHere is an example of how to set up your index URI in the [index config file](../../configuration/index-config):\n```yaml\nindex_uri: s3://{my-bucket}/indexes/{my-index-id}\n```\n" }, { "path": "docs/ingest-data/_category_.yaml", "content": "label: 'Ingest data'\nposition: 4\ncollapsed: true\n" }, { "path": "docs/ingest-data/index.md", "content": "---\ntitle: Ingest data from multiple sources\n---\n\nimport DocCardList from '@theme/DocCardList';\n\n\n\nIt is possible to ingest data with log shippers like [OpenTelemetry](../log-management/overview.md#opentelemetry-agent), [Fluentbit](../log-management/send-logs/using-fluentbit.md), or [Vector](../log-management/send-logs/using-vector.md). It's also possible to send traces from your apps to the [OpenTelemetry Collector](../log-management/send-logs/using-otel-collector-with-helm.md) and then to Quickwit.\n\n\n" }, { "path": "docs/ingest-data/ingest-api.md", "content": "---\ntitle: Ingest API\ndescription: A short tutorial describing how to send data in Quickwit using the ingest API\ntags: [ingest-api, integration]\nicon_url: /img/tutorials/quickwit-logo.svg\nsidebar_position: 1\n---\n\nimport Tabs from '@theme/Tabs';\nimport TabItem from '@theme/TabItem';\n\nIn this tutorial, we will describe how to send data to Quickwit using the ingest API.\n\nYou will need a [local Quickwit instance](../get-started/installation) up and running to follow this tutorial.\n\nTo start it, run `./quickwit run` in a terminal.\n\n## Create an index\n\nFirst, let's create a schemaless index.\n\n```bash\n# Create the index config file.\ncat << EOF > stackoverflow-schemaless-config.yaml\nversion: 0.7\nindex_id: stackoverflow-schemaless\ndoc_mapping:\n mode: dynamic\n dynamic_mapping:\n tokenizer: default\nindexing_settings:\n commit_timeout_secs: 30\nEOF\n# Use the CLI to create the index...\n./quickwit index create --index-config stackoverflow-schemaless-config.yaml\n# Or with cURL.\ncurl -XPOST -H 'Content-Type: application/yaml' 'http://localhost:7280/api/v1/indexes' --data-binary @stackoverflow-schemaless-config.yaml\n```\n\nNote that for this example, we configure the dynamic mapping to use the [default tokenizer](../configuration/index-config.md#description-of-available-tokenizers). This is necessary to enable full-text search on all text fields.\n\n## Ingest data\n\nLet's first download a sample of the [StackOverflow dataset](https://www.kaggle.com/stackoverflow/stacksample).\n\n```bash\n# Download the first 10_000 Stackoverflow posts articles.\ncurl -O https://quickwit-datasets-public.s3.amazonaws.com/stackoverflow.posts.transformed-10000.json\n```\n\nYou can ingest data either with the CLI or with cURL. The CLI is more convenient for ingesting several GB as Quickwit may return `429` responses if the ingest queue is full. Quickwit CLI will automatically retry ingestion in this case.\n\n```bash\n# Ingest the first 10_000 Stackoverflow posts articles with the CLI...\n./quickwit index ingest --index stackoverflow-schemaless --input-path stackoverflow.posts.transformed-10000.json --force\n\n# OR with cURL.\ncurl -XPOST -H 'Content-Type: application/json' 'http://localhost:7280/api/v1/stackoverflow-schemaless/ingest?commit=force' --data-binary @stackoverflow.posts.transformed-10000.json\n```\n\n## Execute search queries\n\nYou can now search the index.\n\n```bash\ncurl 'http://localhost:7280/api/v1/stackoverflow-schemaless/search?query=body:python'\n```\n\n## Tear down resources (optional)\n\n```bash\ncurl -XDELETE 'http://localhost:7280/api/v1/indexes/stackoverflow-schemaless'\n```\n\nThis concludes the tutorial. You can now move on to the [next tutorial](/docs/ingest-data/kafka.md) to learn how to ingest data from Kafka.\n\n## Ingest API versions\n\nIn 0.9, Quickwit introduced a new version of the ingest API that enables distributing the indexing in the cluster regardless of the node that received the ingest request. This new ingestion service is often referred to as \"Ingest V2\" compared to the legacy ingestion (V1). In upcoming versions the new ingest API will also be capable of replicating the write ahead log in order to achieve higher durability.\n\nBy default, both ingestion services are enabled and ingest V2 is used. You can toggle this behavior with the following environment variables:\n\n| Variable | Description | Default value |\n| --------------------- | --------------|-------------- |\n| `QW_ENABLE_INGEST_V2` | Start the V2 ingest service and use it by default. | true | \n| `QW_DISABLE_INGEST_V1`| V1 ingest will be used by the APIs only if V2 is disabled. Running V1 along V2 is necessary to migrate to V2 without loosing existing unindexed V1 logs. | false |\n\n:::note\n\nThese configurations drive the ingest service used both by the `api/v1//ingest` endpoint and the [bulk API](../reference/es_compatible_api.md#_bulk--batch-ingestion-endpoint).\n\n:::\n" }, { "path": "docs/ingest-data/ingest-local-file.md", "content": "---\ntitle: Local file\ndescription: A short tutorial describing how to index a local file with the Quickiwt CLI \ntags: [local-ingest, integration]\nicon_url: /img/tutorials/file-ndjson.svg\nsidebar_position: 2\n---\n\nimport Tabs from '@theme/Tabs';\nimport TabItem from '@theme/TabItem';\n\nIn this tutorial, we will describe how to index a local file with the Quickwit CLI.\n\nYou will need the [Quickwit binary](/docs/get-started/installation.md) to follow this tutorial.\n\n## Create an index\n\nFirst, let's create a schemaless index. We need to start a Quickwit server only for the creation so we will start it and shut it down afterwards.\n\nStart the Quickwit server.\n\n```bash\n./quickwit run\n```\n\nAnd create the index in a separate terminal.\n\n```bash\n# Create the index config file.\ncat << EOF > stackoverflow-schemaless-config.yaml\nversion: 0.7\nindex_id: stackoverflow-schemaless\ndoc_mapping:\n mode: dynamic\nindexing_settings:\n commit_timeout_secs: 30\nEOF\n\n./quickwit index create --index-config stackoverflow-schemaless-config.yaml\n```\n\nYou can now shutdown the server by pressing `Ctrl+C` in the first terminal.\n\n## Ingest the file\n\nTo ingest a file, you just need to execute the following command:\n\n```bash\n./quickwit tool local-ingest --index stackoverflow-schemaless --input-path stackoverflow.posts.transformed-10000.json\n```\n\nAfter a few seconds you should see the following output:\n\n```bash\n❯ Ingesting documents locally...\n\n---------------------------------------------------\n Connectivity checklist\n ✔ metastore\n ✔ storage\n ✔ _ingest-cli-source\n\n Num docs 10000 Parse errs 0 PublSplits 1 Input size 6MB Thrghput 3.34MB/s Time 00:00:02\n Num docs 10000 Parse errs 0 PublSplits 1 Input size 6MB Thrghput 2.23MB/s Time 00:00:03\n Num docs 10000 Parse errs 0 PublSplits 1 Input size 6MB Thrghput 1.67MB/s Time 00:00:04\n\nIndexed 10,000 documents in 4s.\nNow, you can query the index with the following command:\nquickwit index search --index stackoverflow-schemaless --config ./config/quickwit.yaml --query \"my query\"\nClearing local cache directory...\n✔ Local cache directory cleared.\n✔ Documents successfully indexed.\n```\n\n:::tip\n\nObject store URIs like `s3://mybucket/mykey.json` are also supported as `--input-path`, provided that your environment is configured with the appropriate permissions.\n\n:::\n\n## Tear down resources (optional)\n\nThat's it! You can now tear down the resources you created. You can do so by running the following command:\n\n```bash\n./quickwit run\n```\n\nAnd in a separate terminal:\n\n```bash\n./quickwit index delete --index-id stackoverflow-schemaless\n```\n\nThis concludes the tutorial. You can now move on to the next tutorial.\n" }, { "path": "docs/ingest-data/kafka.md", "content": "---\ntitle: Kafka\ndescription: A short tutorial describing how to set up Quickwit to ingest data from Kafka in a few minutes\ntags: [kafka, integration]\nicon_url: /img/tutorials/kafka.svg\nsidebar_position: 2\n---\n\nIn this tutorial, we will describe how to set up Quickwit to ingest data from Kafka in a few minutes. First, we will create an index and configure a Kafka source. Then, we will create a Kafka topic and load some events from the [GH Archive](https://www.gharchive.org/) into it. Finally, we will execute some search and aggregation queries to explore the freshly ingested data.\n\n## Prerequisites\n\nYou will need the following to complete this tutorial:\n- A running Kafka cluster (see Kafka [quickstart](https://kafka.apache.org/quickstart))\n- A local Quickwit [installation](/docs/get-started/installation.md)\n\n## Create index\n\nFirst, let's create a new index. Here is the index config and doc mapping corresponding to the schema of the GH Archive events:\n\n```yaml title=\"index-config.yaml\"\n#\n# Index config file for gh-archive dataset.\n#\nversion: 0.7\n\nindex_id: gh-archive\n\ndoc_mapping:\n field_mappings:\n - name: id\n type: text\n tokenizer: raw\n - name: type\n type: text\n fast: true\n tokenizer: raw\n - name: public\n type: bool\n fast: true\n - name: payload\n type: json\n tokenizer: default\n - name: org\n type: json\n tokenizer: default\n - name: repo\n type: json\n tokenizer: default\n - name: actor\n type: json\n tokenizer: default\n - name: other\n type: json\n tokenizer: default\n - name: created_at\n type: datetime\n fast: true\n input_formats:\n - rfc3339\n fast_precision: seconds\n timestamp_field: created_at\n\nindexing_settings:\n commit_timeout_secs: 10\n```\n\nExecute these Bash commands to download the index config and create the `gh-archive` index:\n\n```bash\n# Download GH Archive index config.\nwget -O gh-archive.yaml https://raw.githubusercontent.com/quickwit-oss/quickwit/main/config/tutorials/gh-archive/index-config.yaml\n\n# Create index.\n./quickwit index create --index-config gh-archive.yaml\n```\n\n## Create and populate Kafka topic\n\nNow, let's create a Kafka topic and load some events into it.\n\n```bash\n# Create a topic named `gh-archive` with 3 partitions.\nbin/kafka-topics.sh --create --topic gh-archive --partitions 3 --bootstrap-server localhost:9092\n\n# Download a few GH Archive files.\nwget https://data.gharchive.org/2022-05-12-{10..15}.json.gz\n\n# Load the events into Kafka topic.\ngunzip -c 2022-05-12*.json.gz | \\\nbin/kafka-console-producer.sh --topic gh-archive --bootstrap-server localhost:9092\n```\n\n## Create Kafka source\n\n:::note\nThis tutorial assumes that the Kafka cluster is available locally on the default port (9092). If it's not the case, please, update the `bootstrap.servers` parameter accordingly.\n:::\n\n```yaml title=\"kafka-source.yaml\"\n#\n# Kafka source config file.\n#\nversion: 0.8\nsource_id: kafka-source\nsource_type: kafka\nnum_pipelines: 2\nparams:\n topic: gh-archive\n client_params:\n bootstrap.servers: localhost:9092\n```\n\nRun these commands to download the source config file and create the source.\n\n```bash\n# Download Kafka source config.\nwget https://raw.githubusercontent.com/quickwit-oss/quickwit/main/config/tutorials/gh-archive/kafka-source.yaml\n\n# Create source.\n./quickwit source create --index gh-archive --source-config kafka-source.yaml\n```\n:::note\n\nIf you get the following error:\n\n``` Command failed: Topic `gh-archive` has no partitions.```\n\nIt means the Kafka topic `gh-archive` was not properly created in the previous step.\n\n:::\n\n\n\n## Launch indexing and search services\n\nFinally, execute this command to start Quickwit in server mode.\n\n```bash\n# Launch Quickwit services.\n./quickwit run\n```\n\nUnder the hood, this command spawns an indexer and a searcher. On startup, the indexer will connect to the Kafka topic specified by the source and start streaming and indexing events from the partitions composing the topic. With the default commit timeout value (see [indexing settings](../configuration/index-config#indexing-settings)), the indexer should publish the first split after approximately 60 seconds.\n\nYou can run this command (in another shell) to inspect the properties of the index and check the current number of published splits:\n\n```bash\n# Display some general information about the index.\n./quickwit index describe --index gh-archive\n```\n\nOnce the first split is published, you can start running search queries. For instance, we can find all the events for the Kubernetes [repository](https://github.com/kubernetes/kubernetes):\n\n```bash\ncurl 'http://localhost:7280/api/v1/gh-archive/search?query=org.login:kubernetes%20AND%20repo.name:kubernetes'\n```\n\nIt is also possible to access these results through the [Quickwit UI](http://localhost:7280/ui/search?query=org.login%3Akubernetes+AND+repo.name%3Akubernetes&index_id=gh-archive&max_hits=10).\n\n\nWe can also group these events by type and count them:\n\n```\ncurl -XPOST -H 'Content-Type: application/json' 'http://localhost:7280/api/v1/gh-archive/search' -d '\n{\n \"query\":\"org.login:kubernetes AND repo.name:kubernetes\",\n \"max_hits\":0,\n \"aggs\":{\n \"count_by_event_type\":{\n \"terms\":{\n \"field\":\"type\"\n }\n }\n }\n}'\n```\n\n\n## Secured Kafka connection (optional)\n\nThe Quickwit Kafka source supports SSL and SASL authentication. This is\nparticularly useful when consuming data from an external Kafka service.\n\n:::tip\n\nThe certificate and key files must be present on all Quickwit nodes for the\nKafka source to be created and for the indexing pipelines to run successfully.\n\n:::\n\n### SSL configuration\n\n```yaml\nversion: 0.8\nsource_id: kafka-source-ssl\nsource_type: kafka\nnum_pipelines: 2\nparams:\n topic: gh-archive\n client_params:\n bootstrap.servers: your-kafka-broker.com\n security.protocol: SSL\n ssl.ca.location: /path/to/ca.pem\n ssl.certificate.location: /path/to/service.cert\n ssl.key.location: /path/to/service.key\n```\n\n### SASL configuration\n\n```yaml\nversion: 0.8\nsource_id: kafka-source-sasl\nsource_type: kafka\nnum_pipelines: 2\nparams:\n topic: gh-archive\n client_params:\n bootstrap.servers: your-kafka-broker.com\n ssl.ca.location: /path/to/ca.pem\n security.protocol: SASL_SSL\n sasl.mechanisms: SCRAM-SHA-256\n sasl.username: your_sasl_username\n sasl.password: your_sasl_password\n```\n\n:::note\n\nIf you get the following error:\n\n```Client creation error: ssl.ca.location failed: error:05880002:x509 certificate routines::system lib```\n\nIt usually means the path to the CA certificate is incorrect. Update the\n`ssl.ca.location` parameter accordingly.\n\n:::\n\n## Tear down resources (optional)\n\nLet's delete the files and resources created for the purpose of this tutorial.\n\n```bash\n# Delete Kafka topic.\nbin/kafka-topics.sh --delete --topic gh-archive --bootstrap-server localhost:9092\n\n# Delete index.\n./quickwit index delete --index gh-archive\n\n# Delete source config.\nrm kafka-source.yaml\n```\n\nThis concludes the tutorial. If you have any questions regarding Quickwit or encounter any issues, don't hesitate to ask a [question](https://github.com/quickwit-oss/quickwit/discussions) or open an [issue](https://github.com/quickwit-oss/quickwit/issues) on [GitHub](https://github.com/quickwit-oss/quickwit) or contact us directly on [Discord](https://discord.com/invite/MT27AG5EVE).\n" }, { "path": "docs/ingest-data/kinesis.md", "content": "---\ntitle: Kinesis\ndescription: A short tutorial describing how to set up Quickwit to ingest data from Kinesis in a few minutes\ntags: [aws, integration]\nicon_url: /img/tutorials/aws-kinesis.svg\nsidebar_position: 4\n---\n\nIn this tutorial, we will describe how to set up Quickwit to ingest data from Kinesis in a few minutes. First, we will create an index and configure a Kinesis source. Then, we will create a Kinesis stream and load some events from the [GH Archive](https://www.gharchive.org/) into it. Finally, we will execute some search and aggregation queries to explore the freshly ingested data.\n\n:::caution\nYou will incur some charges for using the Amazon Kinesis service during this tutorial.\n:::\n\n## Prerequisites\n\nYou will need the following to complete this tutorial:\n- The AWS CLI version 2 (see [Getting started with the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-prereqs.html) for prerequisites and installation)\n- A local Quickwit [installation](/docs/get-started/installation.md)\n- [jq](https://stedolan.github.io/jq/download/)\n- [GNU parallel](https://www.gnu.org/software/parallel/)\n\n:::note\n`jq` is required to reshape the events into records ingestable by the Amazon Kinesis API.\n:::\n\n### Create index\n\nFirst, let's create a new index. Here is the index config and doc mapping corresponding to the schema of the GH Archive events:\n\n```yaml title=\"index-config.yaml\"\n#\n# Index config file for gh-archive dataset.\n#\nversion: 0.7\n\nindex_id: gh-archive\n\ndoc_mapping:\n field_mappings:\n - name: id\n type: text\n tokenizer: raw\n - name: type\n type: text\n fast: true\n tokenizer: raw\n - name: public\n type: bool\n fast: true\n - name: payload\n type: json\n tokenizer: default\n - name: org\n type: json\n tokenizer: default\n - name: repo\n type: json\n tokenizer: default\n - name: actor\n type: json\n tokenizer: default\n - name: other\n type: json\n tokenizer: default\n - name: created_at\n type: datetime\n fast: true\n input_formats:\n - rfc3339\n fast_precision: seconds\n timestamp_field: created_at\n\nindexing_settings:\n commit_timeout_secs: 10\n\n```\n\nExecute these Bash commands to download the index config and create the `gh-archive` index.\n\n```bash\n# Download GH Archive index config.\nwget -O gh-archive.yaml https://raw.githubusercontent.com/quickwit-oss/quickwit/main/config/tutorials/gh-archive/index-config.yaml\n\n# Create index.\n./quickwit index create --index-config gh-archive.yaml\n```\n\n\n## Create and populate Kinesis stream\n\nNow, let's create a Kinesis stream and load some events into it.\n\n:::tip\nThis step may be fairly slow depending on how much bandwidth is available. The current command limits the volume of data to ingest by taking the first 10 000 lines of every single file downloaded from the GH Archive. If you have enough bandwidth, you can remove it to ingest the whole set of files. You can also speed things up by increasing the number of shards and/or the number of jobs launched by `parallel` (`-j` option).\n:::\n\n```bash\n# Create a stream named `gh-archive` with 3 shards.\naws kinesis create-stream --stream-name gh-archive --shard-count 8\n\n# Download a few GH Archive files.\nwget https://data.gharchive.org/2022-05-12-{10..12}.json.gz\n\n# Load the events into Kinesis stream\ngunzip -c 2022-05-12*.json.gz | \\\nhead -n 10000 | \\\nparallel --gnu -j8 -N 500 --pipe \\\n'jq --slurp -c \"{\\\"Records\\\": [.[] | {\\\"Data\\\": (. | tostring), \\\"PartitionKey\\\": .id }], \\\"StreamName\\\": \\\"gh-archive\\\"}\" > records-{%}.json && \\\naws kinesis put-records --cli-input-json file://records-{%}.json --cli-binary-format raw-in-base64-out >> out.log'\n```\n\n## Create Kinesis source\n\n```yaml title=\"kinesis-source.yaml\"\n#\n# Kinesis source config file.\n#\nversion: 0.7\nsource_id: kinesis-source\nsource_type: kinesis\nparams:\n stream_name: gh-archive\n```\n\nRun these commands to download the source config file and create the source.\n\n```bash\n# Download Kinesis source config.\nwget https://raw.githubusercontent.com/quickwit-oss/quickwit/main/config/tutorials/gh-archive/kinesis-source.yaml\n\n# Create source.\n./quickwit source create --index gh-archive --source-config kinesis-source.yaml\n```\n\n:::note\n\nIf this command fails with the following error message:\n```\nCommand failed: Stream gh-archive under account XXXXXXXXX not found.\n\nCaused by:\n 0: Stream gh-archive under account XXXXXXXX not found.\n 1: Stream gh-archive under account XXXXXXXX not found.\n```\n\nit means the Kinesis stream was not properly created in the previous step.\n:::\n\n## Launch indexing and search services\n\nFinally, execute this command to start Quickwit in server mode.\n\n```bash\n# Launch Quickwit services.\n./quickwit run\n```\n\nUnder the hood, this command spawns an indexer and a searcher. On startup, the indexer will connect to the Kinesis stream specified by the source and start streaming and indexing events from the shards composing the stream. With the default commit timeout value (see [indexing settings](../configuration/index-config#indexing-settings)), the indexer should publish the first split after approximately 60 seconds.\n\nYou can run this command (in another shell) to inspect the properties of the index and check the current number of published splits:\n\n```bash\n# Display some general information about the index.\n./quickwit index describe --index gh-archive\n```\n\nIt is also possible to get index information through the [Quickwit UI](http://localhost:7280/ui/indexes/gh-archive).\n\nOnce the first split is published, you can start running search queries. For instance, we can find all the events for the Kubernetes [repository](https://github.com/kubernetes/kubernetes):\n\n```bash\ncurl 'http://localhost:7280/api/v1/gh-archive/search?query=org.login:kubernetes%20AND%20repo.name:kubernetes'\n```\n\nIt is also possible to access these results through the [UI](http://localhost:7280/ui/search?query=org.login%3Akubernetes+AND+repo.name%3Akubernetes&index_id=gh-archive&max_hits=10).\n\nWe can also group these events by type and count them:\n\n```\ncurl -XPOST -H 'Content-Type: application/json' 'http://localhost:7280/api/v1/gh-archive/search' -d '\n{\n \"query\":\"org.login:kubernetes AND repo.name:kubernetes\",\n \"max_hits\":0,\n \"aggs\":{\n \"count_by_event_type\":{\n \"terms\":{\n \"field\":\"type\"\n }\n }\n }\n}'\n```\n\n## Tear down resources (optional)\n\nLet's delete the files and resources created for the purpose of this tutorial.\n\n```bash\n# Delete Kinesis stream.\naws kinesis delete-stream --stream-name gh-archive\n\n# Delete index.\n./quickwit index delete --index gh-archive\n\n# Delete source config.\nrm kinesis-source.yaml\n```\n\nThis concludes the tutorial. If you have any questions regarding Quickwit or encounter any issues, don't hesitate to ask a [question](https://github.com/quickwit-oss/quickwit/discussions) or open an [issue](https://github.com/quickwit-oss/quickwit/issues) on [GitHub](https://github.com/quickwit-oss/quickwit) or contact us directly on [Discord](https://discord.com/invite/MT27AG5EVE).\n" }, { "path": "docs/ingest-data/pulsar.md", "content": "---\ntitle: Pulsar\ndescription: A short tutorial describing how to set up Quickwit to ingest data from Pulsar in a few minutes\ntags: [pulsar, integration]\nicon_url: /img/tutorials/pulsar.svg\nsidebar_position: 3\n---\n\nimport Tabs from '@theme/Tabs';\nimport TabItem from '@theme/TabItem';\n\nIn this tutorial, we will describe how to set up Quickwit to ingest data from Pulsar in a few minutes. First, we will create an index and configure a Pulsar source. Then, we will create a Pulsar topic and load some events from the [Stack Overflow dataset](https://www.kaggle.com/stackoverflow/stacksample) into it. Finally, we will execute some searches.\n\n## Prerequisites\n\nYou will need the following to complete this tutorial:\n- A local running [Quickwit instance](/docs/get-started/installation.md)\n- A local running [Pulsar instance](https://pulsar.apache.org/docs/next/getting-started-standalone/)\n\n### Quickwit setup\n\n[Download](/docs/get-started/installation.md) Quickwit and start a server. Then open a new terminal to execute CLI commands with the same binary. \n\n```bash\n./quickwit run\n```\n\nTest that the cluster is running:\n\n```bash\n./quickwit index list\n```\n\n### Pulsar setup\n\n\n\n\n\n```bash\nwget https://archive.apache.org/dist/pulsar/pulsar-2.11.0/apache-pulsar-2.11.0-bin.tar.gz\ntar xvfz apache-pulsar-2.11.0-bin.tar.gz\ncd apache-pulsar-2.11.0\nbin/pulsar standalone\n```\n\n\n\n\n\n```bash\ndocker run -it -p 6650:6650 -p 8080:8080 apachepulsar/pulsar:2.11.0 bin/pulsar standalone\n```\n\nSee the details on the [official documentation](https://pulsar.apache.org/docs/next/getting-started-docker/).\n\n\n\n\n\n## Prepare Quickwit\n\nFirst, let's create a new index. Here is the index config and doc mapping corresponding to the schema of Stack Overflow posts:\n\n```yaml title=\"index-config.yaml\"\n#\n# Index config file for Stack Overflow dataset.\n#\nversion: 0.7\n\nindex_id: stackoverflow\n\ndoc_mapping:\n field_mappings:\n - name: user\n type: text\n fast: true\n tokenizer: raw\n - name: tags\n type: array\n fast: true\n tokenizer: raw\n - name: type\n type: text\n fast: true\n tokenizer: raw\n - name: title\n type: text\n tokenizer: default\n record: position\n stored: true\n - name: body\n type: text\n tokenizer: default\n record: position\n stored: true\n - name: questionId\n type: u64\n - name: answerId\n type: u64\n - name: acceptedAnswerId\n type: u64\n - name: creationDate\n type: datetime\n fast: true\n input_formats:\n - rfc3339\n fast_precision: seconds\n timestamp_field: creationDate\n\nsearch_settings:\n default_search_fields: [title, body]\n\nindexing_settings:\n commit_timeout_secs: 10\n```\n\nExecute these Bash commands to download the index config and create the `stackoverflow` index.\n\n```bash\n# Download stackoverflow index config.\nwget -O stackoverflow.yaml https://raw.githubusercontent.com/quickwit-oss/quickwit/main/config/tutorials/stackoverflow/index-config.yaml\n\n# Create index.\n./quickwit index create --index-config stackoverflow.yaml\n```\n\n## Create the Pulsar source\n\nA Pulsar source just needs to define the list of topics and the instance address.\n\n```yaml title=\"pulsar-source.yaml\"\n#\n# Pulsar source config file.\n#\nversion: 0.7\nsource_id: pulsar-source\nsource_type: pulsar\nparams:\n topics:\n - stackoverflow\n address: pulsar://localhost:6650\n```\n\nRun these commands to download the source config file and create the source.\n\n```bash\n# Download Pulsar source config.\nwget -O stackoverflow-pulsar-source.yaml https://raw.githubusercontent.com/quickwit-oss/quickwit/main/config/tutorials/stackoverflow/pulsar-source.yaml\n\n# Create source.\n./quickwit source create --index stackoverflow --source-config stackoverflow-pulsar-source.yaml\n```\n\nAs soon as the Pulsar source is created, Quickwit control plane will ask an indexer to start a new indexing pipeline. You will see logs like below by looking on the indexer:\n\n```bash\nINFO spawn_pipeline{index=stackoverflow gen=0}:pulsar-consumer{subscription_name=\"quickwit-stackoverflow-pulsar-source\" params=PulsarSourceParams { topics: [\"stackoverflow\"], address: \"pulsar://localhost:6650\", consumer_name: \"quickwit\", authentication: None } current_positions={}}: quickwit_indexing::source::pulsar_source: Seeking to last checkpoint positions. positions={}\n```\n\n## Create and populate a Pulsar topic\n\nWe will use the Pulsar's default tenant/namespace `public/default`. To populate the topic, we will use a python script:\n\n```python title=send_messages_to_pulsar.py\nimport json\nimport pulsar\n\nclient = pulsar.Client('pulsar://localhost:6650')\nproducer = client.create_producer('public/default/stackoverflow')\n\nwith open('stackoverflow.posts.transformed-10000.json', encoding='utf8') as file:\n for i, line in enumerate(file):\n producer.send(line.encode('utf-8'))\n if i % 100 == 0:\n print(f\"{i}/10000 messages sent.\", i)\n\nclient.close()\n```\n\nInstall locally the python client, more details on [documentation page](https://pulsar.apache.org/docs/2.11.x/client-libraries-python/):\n\n```bash\n# Download the first 10_000 Stackoverflow posts articles.\ncurl -O https://quickwit-datasets-public.s3.amazonaws.com/stackoverflow.posts.transformed-10000.json\n\n# Install pulsar python client.\n# Requires a python version < 3.11\npip3 install 'pulsar-client==2.10.1'\nwget https://raw.githubusercontent.com/quickwit-oss/quickwit/main/config/tutorials/stackoverflow/send_messages_to_pulsar.py\npython3 send_messages_to_pulsar.py\n```\n\n## Time to search!\n\nYou can run this command to inspect the properties of the index and check the current number of published splits and documents:\n\n```bash\n# Display some general information about the index.\n./quickwit index describe --index stackoverflow\n```\n\nYou will notably see the number of published documents.\n\nYou are now ready to execute some queries.\n\n```bash\ncurl 'http://localhost:7280/api/v1/stackoverflow/search?query=search+AND+engine'\n```\n\nIf your Quickwit server is local, you can access to the results through Quickwit UI on [localhost:7280](http://localhost:7280/ui/search?query=&index_id=stackoverflow&max_hits=10).\n\n\n## Tear down resources (optional)\n\nLet's delete the files and resources created for the purpose of this tutorial.\n\n```bash\n# Delete quickwit index.\n./quickwit index delete --index stackoverflow --yes\n# Delete Pulsar topic.\nbin/pulsar-admin topics delete stackoverflow\n```\n\nThis concludes the tutorial. If you have any questions regarding Quickwit or encounter any issues, don't hesitate to ask a [question](https://github.com/quickwit-oss/quickwit/discussions) or open an [issue](https://github.com/quickwit-oss/quickwit/issues) on [GitHub](https://github.com/quickwit-oss/quickwit) or contact us directly on [Discord](https://discord.com/invite/MT27AG5EVE).\n" }, { "path": "docs/ingest-data/sqs-files.md", "content": "---\ntitle: S3 with SQS notifications\ndescription: A short tutorial describing how to set up Quickwit to ingest data from S3 files using an SQS notifier\ntags: [s3, sqs, integration]\nicon_url: /img/tutorials/file-ndjson.svg\nsidebar_position: 5\n---\n\nIn this tutorial, we describe how to set up Quickwit to ingest data from S3\nwith bucket notification events flowing through SQS. We will first create the\nAWS resources (S3 bucket, SQS queue, notifications) using terraform. We will\nthen configure the Quickwit index and file source. Finally we will send some\ndata to the source bucket and verify that it gets indexed.\n\n## AWS resources\n\nThe complete terraform script can be downloaded [here](../assets/sqs-file-source.tf).\n\nFirst, create the bucket that will receive the source data files (NDJSON format):\n\n```\nresource \"aws_s3_bucket\" \"file_source\" {\n bucket_prefix = \"qw-tuto-source-bucket\"\n}\n```\n\nThen setup the SQS queue that will carry the notifications when files are added\nto the bucket. The queue is configured with a policy that allows the source\nbucket to write the S3 notification messages to it. Also create a dead letter\nqueue (DLQ) to receive the messages that couldn't be processed by the file\nsource (e.g corrupted files). Messages are moved to the DLQ after 5 indexing\nattempts. \n\n```\nlocals {\n sqs_notification_queue_name = \"qw-tuto-s3-event-notifications\"\n}\n\ndata \"aws_iam_policy_document\" \"sqs_notification\" {\n statement {\n effect = \"Allow\"\n\n principals {\n type = \"*\"\n identifiers = [\"*\"]\n }\n\n actions = [\"sqs:SendMessage\"]\n resources = [\"arn:aws:sqs:*:*:${local.sqs_notification_queue_name}\"]\n\n condition {\n test = \"ArnEquals\"\n variable = \"aws:SourceArn\"\n values = [aws_s3_bucket.file_source.arn]\n }\n }\n}\n\nresource \"aws_sqs_queue\" \"s3_events_deadletter\" {\n name = \"${locals.sqs_notification_queue_name}-deadletter\"\n}\n\nresource \"aws_sqs_queue\" \"s3_events\" {\n name = local.sqs_notification_queue_name\n policy = data.aws_iam_policy_document.sqs_notification.json\n\n redrive_policy = jsonencode({\n deadLetterTargetArn = aws_sqs_queue.s3_events_deadletter.arn\n maxReceiveCount = 5\n })\n}\n\nresource \"aws_sqs_queue_redrive_allow_policy\" \"s3_events_deadletter\" {\n queue_url = aws_sqs_queue.s3_events_deadletter.id\n\n redrive_allow_policy = jsonencode({\n redrivePermission = \"byQueue\",\n sourceQueueArns = [aws_sqs_queue.s3_events.arn]\n })\n}\n```\n\nConfigure the bucket notification that writes messages to SQS each time a new\nfile is created in the source bucket:\n\n```\nresource \"aws_s3_bucket_notification\" \"bucket_notification\" {\n bucket = aws_s3_bucket.file_source.id\n\n queue {\n queue_arn = aws_sqs_queue.s3_events.arn\n events = [\"s3:ObjectCreated:*\"]\n }\n}\n```\n\n:::note\n\nOnly events of type `s3:ObjectCreated:*` are supported. Other types (e.g.\n`ObjectRemoved`) are acknowledged and a warning is logged.\n\n:::\n\nThe source needs to have access to both the notification queue and the source\nbucket. The following policy document contains the minimum permissions required\nby the source:\n\n```\ndata \"aws_iam_policy_document\" \"quickwit_node\" {\n statement {\n effect = \"Allow\"\n actions = [\n \"sqs:ReceiveMessage\",\n \"sqs:DeleteMessage\",\n \"sqs:ChangeMessageVisibility\",\n \"sqs:GetQueueAttributes\",\n ]\n resources = [aws_sqs_queue.s3_events.arn]\n }\n statement {\n effect = \"Allow\"\n actions = [\"s3:GetObject\"]\n resources = [\"${aws_s3_bucket.file_source.arn}/*\"]\n }\n}\n```\n\nCreate the IAM user and credentials that will be used to\nassociate this policy to your local Quickwit instance:\n\n```\nresource \"aws_iam_user\" \"quickwit_node\" {\n name = \"quickwit-filesource-tutorial\"\n path = \"/system/\"\n}\n\nresource \"aws_iam_user_policy\" \"quickwit_node\" {\n name = \"quickwit-filesource-tutorial\"\n user = aws_iam_user.quickwit_node.name\n policy = data.aws_iam_policy_document.quickwit_node.json\n}\n\nresource \"aws_iam_access_key\" \"quickwit_node\" {\n user = aws_iam_user.quickwit_node.name\n}\n```\n\n\n:::warning\n\nWe don't recommend using IAM user credentials for running Quickwit nodes in\nproduction. This is just a simplified setup for the sake of the tutorial. When\nrunning on EC2/ECS, attach the policy document to an IAM roles instead.\n\n:::\n\nDownload the [complete terraform script](../assets/sqs-file-source.tf) and\ndeploy it using `terraform init` and `terraform apply`. After a successful\nexecution, the outputs required to configure Quickwit will be listed. You can\ndisplay the values of the sensitive outputs (key id and secret key) with:\n\n\n```bash\nterraform output quickwit_node_access_key_id\nterraform output quickwit_node_secret_access_key\n```\n\n## Run Quickwit\n\n[Install Quickwit locally](/docs/get-started/installation), then in your install\ndirectory, run Quickwit with the necessary access rights by replacing the\n`` and `` with the\nmatching Terraform output values:\n\n```bash\nAWS_ACCESS_KEY_ID= \\\nAWS_SECRET_ACCESS_KEY= \\\nAWS_REGION=us-east-1 \\\n./quickwit run\n```\n\n## Configure the index and the source\n\nIn another terminal, in the Quickwit install directory, create an index:\n\n```bash\ncat << EOF > tutorial-sqs-file-index.yaml\nversion: 0.7\nindex_id: tutorial-sqs-file\ndoc_mapping:\n mode: dynamic\nindexing_settings:\n commit_timeout_secs: 30\nEOF\n\n./quickwit index create --index-config tutorial-sqs-file-index.yaml\n```\n\nReplacing `` with the corresponding Terraform output\nvalue, create a file source for that index:\n\n```bash\ncat << EOF > tutorial-sqs-file-source.yaml\nversion: 0.8\nsource_id: sqs-filesource\nsource_type: file\nnum_pipelines: 2\nparams:\n notifications:\n - type: sqs\n queue_url: \n message_type: s3_notification\nEOF\n\n./quickwit source create --index tutorial-sqs-file --source-config tutorial-sqs-file-source.yaml\n```\n\n:::tip\n\nThe `num_pipeline` configuration controls how many consumers will poll from the queue in parallel. Choose the number according to the indexer compute resources you want to dedicate to this source. As a rule of thumb, configure 1 pipeline for every 2 cores.\n\n:::\n\n## Ingest data\n\nWe can now ingest data into Quickwit by uploading files to S3. If you have the\nAWS CLI installed, run the following command, replacing ``\nwith the associated Terraform output:\n\n```bash\ncurl https://quickwit-datasets-public.s3.amazonaws.com/hdfs-logs-multitenants-10000.json | \\\n aws s3 cp - s3:///hdfs-logs-multitenants-10000.json\n```\n\nIf you prefer not to use the AWS CLI, you can also download the file and upload\nit manually to the source bucket using the AWS console.\n\nWait approximately 1 minute and the data should appear in the index:\n\n```bash\n./quickwit index describe --index tutorial-sqs-file\n```\n\n## Tear down the resources\n\nThe AWS resources instantiated in this tutorial don't incur any fixed costs, but\nwe still recommend deleting them when you are done. In the directory with the\nTerraform script, run `terraform destroy`.\n" }, { "path": "docs/internals/backward-compatibility.md", "content": "# Backward compatibility in Quickwit.\n\nIf you are reading this, chances are you want to make a change to one of the resource\nof Quickwit's meta/config.\n\nThere are basically 3 types of configuration:\n\nEdited by the user and read back from file on startup:\n- QuickwitConfig\n\nEdited by the user then stored in the metastore:\n- IndexConfig\n- SourceConfig\n- VersionedIndexTemplate\n\nAssembled by Quickwit then stored in the metastore:\n- IndexMetadata\n- SplitMetadata\n- FileBackedIndex (file backed metastore only)\n- Manifest (file backed metastore only)\n\nQuickwit currently manages the backward compatibility of all of these resources except the `QuickwitConfig`.\n\nThis document describes how to handle a change, and how to make test such a change, and spot eventual regression.\n\n## How do I update `{IndexMetadata, SplitMetadata, FileBackedIndex, SourceConfig, IndexConfig, Manifest}`?\n\nThere are two types of upgrades:\n- naturally backward compatible change\n- change requiring a new version\n\n### Naturally backward compatible change\n\nSerde offers some attributes to make backward compatible changes to our model.\nFor instance, it is possible to add a new field to a struct and slap\na `serde(default)` attribute to it in order to handle older serialized version of the\nstruct.\n\nIf you want to avoid to generate any diff on the non-regression json files,\nyou can also avoid use `#[serde(skip_serializing_if)]`, although by default,\nit is recommended to not use it.\n\nIt is also possible to rename a field in a backward compatible manner\nby using the `#[serde(alias)]`.\n\nFor this type of change it is not required to update the serialization version.\n\nNevertheless, the regression tests will spot these changes. When that happens:\n- modify your model with the help of the attributes above.\n- modify the example for the model by editing its `TestableForRegression` trait implementation.\n- run the backward compatibility tests (see below)\n- check the diff between the `xxx.modified.json` files created and the matching `xxx.json` files. \nIf the changes are acceptable, replace the content of the `xxx.json` files and commit them.\n\nBe particularly careful to changes on files corresponding to the most recent version. If the \nchanges are not compatible, create a new configuration version.\n\n### Change requiring a new version\n\nFor changes requiring a new version, you will have to increment the configuration\nversion. You need to make sure that all of these resources share the same version number.\n\n- update the resource struct you want to change.\n- create a new item in the `VersionedXXXX` struct. It is usually located in a serialize.rs file\n- `Serialize` is not needed for the previous serialized version. We just need `Deserialize`. We can \nremove the `Serialize` impl from the derive statement, and mark it a `skip_serializing` as follows.\n\ne.g.\n```\n#[serde(tag = \"version\")]\npub(crate) enum VersionedXXXXXX {\n #[serde(rename = \"0\")]\n V0(#[serde(skip_serializing)] XXXX_V0),\n #[serde(rename = \"1\")]\n V1(XXXX_V1),\n}\n```\n- complete the conversion `From for XXXX` and `From for VersionedXXXX`\n- run the backward compatibility tests (see below)\n- for older versions, check the diff between the `xxx.expected.modified.json` files created and the matching `xxx.expected.json` files. \nIf the changes are acceptable, replace the content of the `xxx.expected.json` files and commit them.\n- check the `yyyy.json` that was created for the new version and commit it along with the `yyyy.expected.json` file (identical).\n- possibly update the generation of the default XXXX instance used for regression. It is in the function `TestableForRegression::sample_for_regression`.\n\n\n## Backward compatibility tests\n\nThese tests are used to ensure the backward compatibility of Quickwit.\nRight now, `SplitMetadata`, `IndexMetadata`, `Manifest` and `FileBackedIndex` are tested.\n\nWe want to be able to read all past versions of these files, but only write the most recent format.\n\nThe tests consist of pairs of JSON files, `XXXX.json` and `XXXX.expected.json`:\n- `XXXX.json` is the first serialized value of a new version.\n- `XXXX.expected.json` is the result of `serialize_new_version(deserialize(XXXX.json))`.\n\nFormat changes are automatically detected. There are two possible situations when a format changes.\n\n#### Updating expected.json\n\nWe need to keep `*.expected.json` files up-to-date with the format changes.\n\nThis is done in a semi-automatic fashion.\n\nChecks are performed in two steps:\n- first pass, `deserialize(original_json) == deserialize(expectation_json)`\n- second pass, `expectation_json = serialize(deserialize(expectation_json))`\n\nWhen changing the json format, it is expected to see this test fail.\nThe unit test then updates automatically the `expected.json`. The developer just has to\ncheck the diff of the result (in particular no information should be lost) and commit the \nupdated expected.json files.\n\nAdding this update operation within the unit test is a tad unexpected, but it has the merit of\nintegrating well with CI. If a developer forgets to update the expected.json file,\nthe CI will catch it.\n\n#### Adding a new test case.\n\nIf the serialization format changes, a new version should be created and the unit test will\nautomatically add a new unit test generated from the sample tested objects.\nConcretely, it will just write two files `XXXX.json` and `XXXX.expected.json` for each model.\n\nThe two files will be identical. This is expected as this is a unit test for the most recent \nversion. The unit test will start making sense in future updates thanks to the update phase\ndescribed in the previous section.\n" }, { "path": "docs/internals/date-time.md", "content": "# Datetime format\n\nQuickwit's DateTime is a wrapper around Tantivy's provided DateTime type which is internally represented as an `i64` microseconds value. For optimization reasons, Tantivy stores the value differently at the following locations:\n- DocStore: Dates are stored as they are received from the input document.\n- TermDict: Dates are stored with `seconds` precision.\n- FastField: Dates are stored using the DateTime type configured precision that can take of the following values: `seconds`, `milliseconds`, `microseconds`.\n" }, { "path": "docs/internals/ingest-v2.md", "content": "# Ingest V2\n\nIngest V2 is the latest ingestion API that is designed to be more efficient and scalable for thousands of indexes than the previous version. It is the default since 0.9.\n\n## Architecture\n\nJust like ingest V1, the new ingest uses [`mrecordlog`](https://github.com/quickwit-oss/mrecordlog) to persist ingested documents that are waiting to be indexed. But unlike V1, which always persists the documents locally on the node that receives them, ingest V2 can dynamically distribute them into WAL units called _shards_. The assigned shard can be local or on another indexer. The control plane is in charge of distributing the shards to balance the indexing work as well as possible across all indexer nodes. The progress within each shard is not tracked as an index metadata checkpoint anymore but in a dedicated metastore `shards` table.\n\nIn the future, the shard based ingest will also be capable of writing a replica for each shard, thus ensuring a high durability of the documents that are waiting to be indexed (durability of the indexed documents is guarantied by the object store).\n\n## Toggling between ingest V1 and V2\n\nVariables driving the ingest configuration are documented [here](../ingest-data/ingest-api.md#ingest-api-versions).\n\nWith ingest V2, you can also activate the `enable_cooperative_indexing` option in the indexer configuration. This setting is useful for deployments with very large numbers (dozens) of actively written indexers, to limit the indexing workbench memory consumption. The indexer configuration is in the node configuration:\n\n```yaml\nversion: 0.8\n# [...]\nindexer:\n enable_cooperative_indexing: true\n```\n\nSee [full configuration example](https://github.com/quickwit-oss/quickwit/blob/main/config/quickwit.yaml).\n\n## Differences between ingest V1 and V2\n\n- V1 uses the `queues/` directory whereas V2 uses the `wal/` directory\n- both V1 and V2 are configured with:\n - `ingest_api.max_queue_memory_usage` \n - `ingest_api.max_queue_disk_usage` \n- but ingest V2 can also be configured with:\n - `ingest_api.replication_factor`, not working yet\n- ingest V1 always writes to the WAL of the node receiving the request, V2 potentially forwards it to another node, dynamically assigned by the control plane to distribute the indexing work more evenly.\n- ingest V2 parses and validates input documents synchronously. Schema and JSON formatting errors are returned in the ingest response (for ingest V1 those errors were available in the server logs only).\n" }, { "path": "docs/internals/scroll.md", "content": "# Scroll API\n\nThe scroll API has been implemented to offer compatibility with ElasticSearch.\nThe API and the implementation are quirky and are detailed in this document.\n\n## API description\n\nYou can find information about the scroll API here.\nhttps://www.elastic.co/guide/en/elasticsearch/reference/current/paginate-search-results.html#scroll-search-results\nhttps://www.elastic.co/guide/en/elasticsearch/reference/current/scroll-api.html\n\nThe user runs a regular search request with a `scroll` param.\nThe search result then contains the normal response, but a `_scroll` property is added to the search body.\n\nThat id is then meant to be sent to a scroll rest API.\nThis API successive calls will then return pages incrementally.\n\n## Quirk and difficulty.\n\nThe scrolled results should be consistent with the state of the original index.\nFor this reason we need to capture the state of the index at the point of the original request.\n\nIf a network error happens between the client and the server at page N, there is no way for the client to ask the reemission of page N.\nPage N+1 will be returned on the next call.\n\n## Implementation\n\nServer side, we store a replicated scroll context.\n\nIt contains:\n- the detail about the original query (we need to be able to reemit paginated queries)\n- the \"point-in-time\" list of split metadatas used for the query\n- a cached list of partial docs (= not the doc content, just its address and its score) to avoid\nperforming search over and over.\n- the total number of results, in order to append that information to our response.\nsearching at every single scroll requests.\n\nWe use a simple leaderless KV store to keep the state required to run the scroll API.\nWe generate a scroll ULID and use it to get a list of the servers with the best affinity according\nto rendez vous hashing. We then go through them in order and attempt to put that key on up to 2 servers. Failures for these PUTs are silent.\n\nFor each call to scroll, one of two things can happen:\n- the partial docs for the page requested is in the partial doc cache. We just run the fetch_docs phase, and update the context with the `start_offset`.\n- the partial docs for the page request are not in the partial doc cache. We then run a new search query.\n\nWe attempt to fetch `SCROLL_BATCH_LEN` in order to fill the partial doc address cache for subsequent calls.\n\n# A strange `scroll_id`.\n\nThe elasticsearch API is needlessly broken as it returns the same scroll_id most of the time.\nThe \"page-change\" mutation is something that happens on the server side.\n\nIn quickwit on the other hand, the scroll id is the concatenation of the\n- ULID: used as the address for the search context.\n- the start_offset.\n- the number of hits per page\n- a search_after key\n\nWe only mutate the state server side to update the cache whenever needed.\n\nThe idea here is that if that if the put request failed, we can still return the right results even if we have an obsolete version of the `ScrollContext`.\n\n# Quickwit implementation (improvement, quirks and shortcuts)\n\nWe do not do explicitly protect the split from our store Point-In-Time information\nfrom deletion. Instead we simply rely on the existing grace period mechanism (a split\nonly is effectively garbage collected 32mn after it is marked as deleted).\n\nFor this reason we limit the scroll period to 30mn and subsequent scroll calls do not\nextend the scroll period.\n\nAlso thanks to this period, we do not add any extra replication repair mechanism.\nSome scroll calls will end up being broken if we were to remove 2 servers within 30mn.\n\nQuickwit caches partial hits in batches of 1000 results.\nQuerying page N leverages `search_after`, so that accessing further pages isn't more\ncostly than accessing the first ones.\n" }, { "path": "docs/internals/searcher-split-cache.md", "content": "\n# Searcher split cache\n\nQuickwit includes a split cache. It can be useful for specific workloads:\n- to improve performance\n- to reduce the cost associated with GET requests.\n\nThe split cache stores entire split files on disk.\nIt works under the following configurable constraints:\n- number of concurrent downloads\n- amount of disk space\n- number of on-disk files.\n\nSearcher get tipped by indexers about the existence of splits (for which they have the best affinity).\nThey also might learn about split existence, upon read requests.\n\nThe searcher is then in charge of maintaining an in-memory data structure with a bounded list of splits it knows about and their score.\nThe current strategy for admission/evicton is a simple LRU logic.\n\nIf the most recently accessed split not already in cache has been accessed, we consider downloading it.\nIf the limits have been reached, we only proceed to eviction if one of the split currently\nin cache has been less recently accessed.\n\n\n" }, { "path": "docs/internals/sorting.md", "content": "# Sorting\n\nQuickwit can sort results based on fastfield values or score. This document discuss where and how\n it happens.\nIt also tries to describe optimizations that may be enabled (but are not necessarily implemented)\nby this behavior.\n\n## Behavior\n\nSorting is controlled by the `sort_by` query parameter. It accepts a comma separated list of fields\nto use for sorting. Sorting is Descending by default. The sorting order can be reversed by prefixing\na field name with a hyphen `-`.\nThe special value `_score` means sorting by score, it is also Descending by default.\n\nIn case of equality between two documents, the GlobalDocId, composed of (SplitId, SegmentId, DocId)\nis used as a tie breaker. It is used to sort in the same order as the first field being sorted by.\nThis means it is in Descending order by default.\n\nIf a document doesn't have a value for a sorting field, that document is considered to go after any\ndocument which has a value, independently of sort order. That is, when sorting the value 1,2 and\nNone, ascending sort would give `[1, 2, None]`, and descending sort would give `[2, 1, None]`.\n\nIf a client does not request sorting, documents are sorted using (SplitId, SegmentId, DocId), on\nDescending order. In other words, everything happens as if documents were sorted by a constant\nvalue.\n\n\n\n# Code\n\nA new structure TopK is introduced which is used both for in-split sorting and for merging of\nresults. It reduces the risks of inconsistencies between in-split and between-split behavior.\n`SortOrder` gets new `compare` and `compare_opt` method which can be used to compare two values with\n respect to the particular sort order required, and with proper handling of the `None` special case.\n\n# Optimization permitted\n\nBoth orders allow an optimization when sorting by date (either direction), by leveraging splits\nmeta-data to know in advance if a split can, or not, contain better results. Changing the sorting\norder for \"not sorted\" queries allows to leverage SplitId as a way to know whether a split can\ncontain or not better results (if its SplitId is more/less than the current worst best-hit, the\nsplit does not need to be searched).\n\n\n\nThese optimization have limited to no impact if we give an exact count of matching documents.\nAn option to request only a lower bound would be required for these optimizations to make sense.\n" }, { "path": "docs/internals/split-format.md", "content": "# Split format\n\nQuickwit's index are divided into small independent immutable piece of index called split.\n\nFor convenience, a split consists in a single file, with the extension `.split`.\n\nIn reality, this file hides an internal mini static filesystem,\nwith:\n- the Tantivy index files (`.idx`, `.pos`, `.term`...)\n- a Quickwit specific file with the list of fields, including those indexed as part of a JSON type. \nIt contains the field name, type and capabilities.\n\nThe split file data layout looks like this:\n- concatenation all of the files in the split\n- a footer\n\nThe footer follows the following format.\n\n- a json object called `BundleStorageFileOffsets` containing the `[start, end)` byte-offsets\nof all files.\n- the length of this json (8 bytes little endian)\n- a hotcache, a small static cache that contains some important file sections.\n- the length of this hotcache (8 bytes little endian)\n\nThis footer plays a key role a very important role in quickwit.\nIt packs in one read all of the information required to open a split.\n\nWhen opening a file from a distant storage, Quickwit's metastore stores the byte offsets of this footer to make this read possible.\n\nIf this footer offset information is not available, for instance if the split is just a file on the filesystem, it is still possible to open it by reading the last 8 bytes of the split (encoding the length of the hotcache), deducing the position of the meta information and unpacking this in turn.\n" }, { "path": "docs/internals/template-index.md", "content": "# Index template API\n\nIndex templates are a way to create indexes automatically with some given configuration when Quickwit receives documents for an index that doesn't exist yet.\n\nExample of templates: [https://github.com/quickwit-oss/quickwit/tree/main/config/templates](https://github.com/quickwit-oss/quickwit/tree/main/config/templates).\n\n# Curl to run to use the REST API to create Stackoverflow template\n\n```bash\ncurl -XPOST -H 'Content-Type: application/yaml' 'http://localhost:7280/api/v1/templates' --data-binary @config/templates/stackoverflow.yaml\n\n# Lists templates.\ncurl 'http://localhost:7280/api/v1/templates'\n\n# Update Stackoverflow template.\ncurl -XPUT -H 'Content-Type: application/yaml' 'http://localhost:7280/api/v1/templates/stackoverflow' --data-binary @config/templates/stackoverflow.yaml\n\n# Download dataset.\ncurl -O https://quickwit-datasets-public.s3.amazonaws.com/stackoverflow.posts.transformed-10000.json\n\n# Ingest 10k docs into `stackoverflow-foo` index.\ncurl -XPOST \"http://127.0.0.1:7280/api/v1/stackoverflow-foo/ingest\" --data-binary @stackoverflow.posts.transformed-10000.json\n\n# Ingest 10k docs into `stackoverflow-bar` index.\ncurl -XPOST \"http://127.0.0.1:7280/api/v1/stackoverflow-bar/ingest\" --data-binary @stackoverflow.posts.transformed-10000.json\n\n# Delete Stackoverflow template.\ncurl -XDELETE 'http://localhost:7280/api/v1/templates/stackoverflow'\n\n```bash\n" }, { "path": "docs/log-management/_category_.yaml", "content": "label: 'Log management'\nposition: 5\ncollapsed: true\n" }, { "path": "docs/log-management/otel-service.md", "content": "---\ntitle: OTEL service\nsidebar_position: 4\n---\n\nQuickwit natively supports the [OpenTelemetry Protocol (OTLP)](https://opentelemetry.io/docs/reference/specification/protocol/otlp/) and provides a gRPC endpoint to receive spans from an OpenTelemetry collector. This endpoint is enabled by default.\n\nWhen enabled, Quickwit will start the gRPC service ready to receive logs from an OpenTelemetry collector. The logs are indexed in the `otel-logs-v0_7` index by default, and this index will be automatically created if not present. The index doc mapping is described in the next [section](#trace-and-span-data-model).\n\nIf for any reason, you want to disable this endpoint, you can:\n- Set the `QW_ENABLE_OTLP_ENDPOINT` environment variable to `false` when starting Quickwit.\n- Or [configure the node config](/docs/configuration/node-config.md) by setting the indexer setting `enable_otlp_endpoint` to `false`.\n\n```yaml title=node-config.yaml\n# ... Indexer configuration ...\nindexer:\n enable_otlp_endpoint: false\n```\n\n## Sending logs in your own index\n\nYou can send logs in the index of your choice by setting the header `qw-otel-logs-index` of your gRPC request to the targeted index ID.\n\n\n## OpenTelemetry logs data model\n\nQuickwit sends OpenTelemetry logs into the `otel-logs-v0_7` index by default which is automatically created if you enable the OpenTelemetry service.\nThe doc mapping of this index described below is derived from the [OpenTelemetry logs data model](https://opentelemetry.io/docs/reference/specification/logs/data-model/).\n\n```yaml\n\nversion: 0.7\n\nindex_id: otel-logs-v0_7\n\ndoc_mapping:\n mode: strict\n field_mappings:\n - name: timestamp_nanos\n type: datetime\n input_formats: [unix_timestamp]\n output_format: unix_timestamp_nanos\n indexed: false\n fast: true\n fast_precision: milliseconds\n - name: observed_timestamp_nanos\n type: datetime\n input_formats: [unix_timestamp]\n output_format: unix_timestamp_nanos\n - name: service_name\n type: text\n tokenizer: raw\n fast: true\n - name: severity_text\n type: text\n tokenizer: raw\n fast: true\n - name: severity_number\n type: u64\n fast: true\n - name: body\n type: json\n tokenizer: default\n - name: attributes\n type: json\n tokenizer: raw\n fast: true\n - name: dropped_attributes_count\n type: u64\n indexed: false\n - name: trace_id\n type: bytes\n input_format: hex\n output_format: hex\n - name: span_id\n type: bytes\n input_format: hex\n output_format: hex\n - name: trace_flags\n type: u64\n indexed: false\n - name: resource_attributes\n type: json\n tokenizer: raw\n fast: true\n - name: resource_dropped_attributes_count\n type: u64\n indexed: false\n - name: scope_name\n type: text\n indexed: false\n - name: scope_version\n type: text\n indexed: false\n - name: scope_attributes\n type: json\n indexed: false\n - name: scope_dropped_attributes_count\n type: u64\n indexed: false\n\n timestamp_field: timestamp_nanos\n\nindexing_settings:\n commit_timeout_secs: 10\n\nsearch_settings:\n default_search_fields: [body.message]\n```\n\n## UI Integration\n\nCurrently, Quickwit provides a simplistic UI to get basic information from the cluster, indexes and search documents.\nIf a simple UI is not sufficient for you and you need additional features, Grafana and Elasticsearch query API support are planned for Q2 2023, stay tuned!\n\nYou can also send traces to Quickwit that you can visualize in Jaeger UI, as explained in the following [tutorial](../distributed-tracing/send-traces/using-otel-sdk-python.md).\n\n\n## Known limitations\n\nThere are a few limitations on the log management setup in Quickwit 0.9:\n- The ingest API does not provide High-Durability. This will be fixed in 0.10.\n- OTLP HTTP is only available with the Binary Protobuf Encoding. OTLP HTTP with JSON encoding is not planned yet, but this can be easily fixed in the next version. Please open an issue if you need this feature.\n\nIf you are interested in new features or discover other limitations, please open an issue on [GitHub](https://github.com/quickwit-oss/quickwit).\n" }, { "path": "docs/log-management/overview.md", "content": "---\ntitle: Log management with Quickwit\nsidebar_label: Overview\nsidebar_position: 1\n---\n\nQuickwit is built from the ground up to [efficiently index unstructured data](../guides/schemaless.md), and search it effortlessly on cloud storage.\nMoreover, Quickwit supports OpenTelemetry gRPC and HTTP (protobuf only) protocols out of the box and provides a REST API ready to ingest any JSON formatted logs.\n**This makes Quickwit a perfect fit for logs!**.\n\n![Quickwit Log Management](../assets/images/log-management-overview-light.svg#gh-light-mode-only)![Quickwit Log Management](../assets/images/log-management-overview-dark.svg#gh-dark-mode-only)\n\n## Sending logs to Quickwit\n\n- [Using OTEL collector](send-logs/using-otel-collector.md)\n- [Using OTEL collector with Helm](send-logs/using-otel-collector-with-helm.md)\n- [Using Fluentbit](send-logs/using-fluentbit.md)\n- [Using Vector](send-logs/using-vector.md)\n\n" }, { "path": "docs/log-management/send-logs/_category_.yaml", "content": "label: 'Sending logs'\nposition: 2\ncollapsed: false\n" }, { "path": "docs/log-management/send-logs/send-docker-logs.md", "content": "---\ntitle: Send docker logs into Quickwit\nsidebar_label: Docker logs into Quickwit\ndescription: Send docker logs into Quickwit\ntags: [otel, docker, collector, log]\nsidebar_position: 5\n---\n\nTo send docker container logs into Quickwit, you just need to setup an OpenTelemetry Collector with the file logs receiver. In this tutorial, we will use `docker compose` to start the collector and Quickwit.\n\nYou only need a minute to get your Quickwit log UI!\n\n![Quickwit UI Logs](../../assets/images/screenshot-quickwit-ui-docker-compose-logs.png)\n\n## OTEL collector configuration\n\nThe following collector configuration will collect docker logs in `/var/lib/docker/containers/*/*-json.log` (depending on your system, log files can be at a different location), add a few attributes and send them to Quickwit through gRPC at `http://quickwit:7281`.\n\n\n```yaml title=\"otel-collector-config.yaml\"\nreceivers:\n filelog:\n include:\n - /var/lib/docker/containers/*/*-json.log\n operators:\n - id: parser-docker\n timestamp:\n layout: '%Y-%m-%dT%H:%M:%S.%LZ'\n parse_from: attributes.time\n type: json_parser\n - field: attributes.time\n type: remove\n - id: extract_metadata_from_docker_tag\n parse_from: attributes.attrs.tag\n regex: ^(?P[^\\|]+)\\|(?P[^\\|]+)\\|(?P[^$]+)$\n type: regex_parser\n if: 'attributes?.attrs?.tag != nil'\n - from: attributes.name\n to: resource[\"docker.container.name\"]\n type: move\n if: 'attributes?.name != nil'\n - from: attributes.image_name\n to: resource[\"docker.image.name\"]\n type: move\n if: 'attributes?.image_name != nil'\n - from: attributes.id\n to: resource[\"docker.container.id\"]\n type: move\n if: 'attributes?.id != nil'\n - from: attributes.log\n to: body\n type: move\n\nprocessors:\n batch:\n timeout: 5s\n\nexporters:\n otlp/qw:\n endpoint: quickwit:7281\n tls:\n insecure: true\n\nservice:\n pipelines:\n logs:\n receivers: [filelog]\n processors: [batch]\n exporters: [otlp/qw]\n```\n\n## Start the OTEL collector and a Quickwit instance\n\nLet's use `docker compose` with the following configuration:\n\n```yaml title=\"docker-compose.yaml\"\nversion: \"3\"\n\nx-default-logging: &logging\n driver: \"json-file\"\n options:\n max-size: \"5m\"\n max-file: \"2\"\n tag: \"{{.Name}}|{{.ImageName}}|{{.ID}}\"\n\nservices:\n quickwit:\n image: quickwit/quickwit:${QW_VERSION:-0.8.1}\n volumes:\n - ./qwdata:/quickwit/qwdata\n ports:\n - 7280:7280\n environment:\n - NO_COLOR=true\n command: [\"run\"]\n logging: *logging\n\n otel-collector:\n user: \"0\" # Needed to access the directory /var/lib/docker/containers/\n image: otel/opentelemetry-collector-contrib:${OTEL_VERSION:-0.87.0}\n volumes:\n - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml\n - /var/lib/docker/containers:/var/lib/docker/containers:ro\n command: [\"--config=/etc/otel-collector-config.yaml\"] \n logging: *logging\n```\n\n\nYou will notice the custom `logging`, the OTEL collector will use that additional information to enrich the logs.\n\n## Run it and search\n\nDownload the configuration files and start the containers:\n \n```bash\n\nmkdir qwdata\ndocker compose up\n```\n\nAfter a few seconds, you will see the logs in the Quickwit UI [http://localhost:7280](http://localhost:7280).\n\n\nHere is what it should look like:\n\n```json\n{\n \"attributes\": {\n \"log.file.name\": \"34ad1a84c71de1d29ad75f99b56d01205e2976440f2398734037151ba2bcde1a-json.log\",\n \"stream\": \"stdout\"\n },\n \"body\": {\n \"message\": \"2023-10-23T16:39:57.892 INFO --- [ asgi_gw_1] localstack.request.aws : AWS s3.ListObjects => 200\\n\"\n },\n \"observed_timestamp_nanos\": 1698079197979435000,\n \"service_name\": \"unknown_service\",\n \"severity_number\": 0,\n \"timestamp_nanos\": 1698079197892726000,\n \"trace_flags\": 0\n}\n```\n\n\n## Troubleshooting\n\nIt's possible that you get no logs in the UI. In this case, check the `docker compose` logs. The problem can typically come from a wrong configuration of the OTEL collector.\n" }, { "path": "docs/log-management/send-logs/using-fluentbit.md", "content": "---\ntitle: Send logs using Fluentbit\nsidebar_label: Using Fluentbit\ndescription: A simple tutorial to send logs from Fluentbit to Quickwit in a few minutes.\nicon_url: /img/tutorials/fluentbit-logo.png\ntags: [logs, ingestion]\nsidebar_position: 4\n---\n\nimport Tabs from '@theme/Tabs';\nimport TabItem from '@theme/TabItem';\n\n[Fluent Bit](https://fluentbit.io/) is an open-source logging and metrics processor and forwarder to multiple destinations.\n\nIn this guide, we will show you how to connect it to Quickwit.\n\n## Prerequisites\n\n- [Install Quickwit](/docs/get-started/installation.md)\n- Start a Quickwit instance with `./quickwit run`\n- [Install Fluentbit](https://docs.fluentbit.io/manual/installation/getting-started-with-fluent-bit)\n\n\n## Create a simple index for Fluentbit logs\n\nLet's create a schemaless index with only one field `timestamp`. The mode `dynamic` indicates that Quickwit will index all fields even if they are not defined in the doc mapping.\n\n```yaml title=\"index-config.yaml\"\nversion: 0.7\n\nindex_id: fluentbit-logs\n\ndoc_mapping:\n mode: dynamic\n field_mappings:\n - name: timestamp\n type: datetime\n input_formats:\n - unix_timestamp\n output_format: unix_timestamp_secs\n fast: true\n timestamp_field: timestamp\n\nindexing_settings:\n commit_timeout_secs: 10\n```\n\n```bash\ncurl -o fluentbit-logs.yaml https://raw.githubusercontent.com/quickwit-oss/quickwit/main/config/tutorials/fluentbit-logs/index-config.yaml\n```\n\nAnd then create the index with `cURL` or the `CLI`:\n\n\n\n\n\n```bash\ncurl -XPOST http://localhost:7280/api/v1/indexes -H \"content-type: application/yaml\" --data-binary @fluentbit-logs.yaml\n```\n\n\n\n\n\n```bash\n./quickwit index create --index-config fluentbit-logs.yaml\n```\n\n\n\n\n\n\n## Setup Fluentbit\n\nFluentbit configuration file is made of inputs and outputs. For this tutorial, we will use a dummy configuration:\n\n``` title=fluent-bit.conf\n[INPUT]\n Name dummy\n Tag dummy.log\n\n[OUTPUT]\n Name http\n Match *\n URI /api/v1/fluentbit-logs/ingest\n Host localhost\n Port 7280\n tls Off\n Format json_lines\n Json_date_key timestamp\n Json_date_format epoch\n```\n\nFluentbit will send `dummy` logs to Quickwit endpoint `/api/v1/fluentbit-logs/ingest`.\n\nLet's start Fluentbit.\n\n```bash\nfluent-bit -c fluent-bit.conf\n```\n\n## Search logs\n\nQuickwit is now ingesting logs coming from Fluentbit and you can search them either with `cURL` or by using the UI:\n- `curl \"http://127.0.0.1:7280/api/v1/fluentbit-logs/search?query=severity:DEBUG\"`\n- Open your browser at `http://127.0.0.1:7280/ui/search?query=severity:DEBUG&index_id=fluentbit-logs&max_hits=10`.\n\n\n## Further improvements\n\nYou will soon be able to do aggregations on dynamic fields (planned for 0.7).\n" }, { "path": "docs/log-management/send-logs/using-otel-collector-with-helm.md", "content": "---\ntitle: Send K8s logs using OTEL collector\nsidebar_label: Using OTEL with Helm\ndescription: Send K8s logs with OTEL collectors and Helm to Quickwit in a few minutes.\ntags: [k8s, helm]\nicon_url: /img/tutorials/helm-otel-k8s-tutorial-illustation.jpg\nsidebar_position: 2\n---\n\nThis guide will help you to unlock log search on your k8s cluster logs. We will first deploy Quickwit and OTEL collectors with [Helm](https://helm.sh/) and then see how to index and search them.\n\n## Prerequisites\n\nYou will need the following to complete this tutorial:\n- A Kubernetes cluster.\n- The command line tool [kubectl](https://kubernetes.io/docs/reference/kubectl/).\n- The command line tool [Helm](https://helm.sh/).\n- An access to an object storage like AWS S3, GCS, Azure blob storage, or Scaleway to store index data.\n\n\n## Install with Helm\n\nLet's first create a namespace to isolate our experiment and set it as the default namespace.\n\n```bash\nkubectl create namespace qw-tutorial\nkubectl config set-context --current --namespace=qw-tutorial\n```\n\n\nThen let's add [Quickwit](https://github.com/quickwit-oss/helm-charts) and [Otel](https://github.com/open-telemetry/opentelemetry-helm-charts) helm repositories:\n\n```bash\nhelm repo add quickwit https://helm.quickwit.io\nhelm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts\n```\n\nYou should now see the two repos in helm:\n\n```bash\nhelm repo list\nNAME \tURL\nquickwit \thttps://helm.quickwit.io\nopen-telemetry \thttps://open-telemetry.github.io/opentelemetry-helm-charts\n```\n\n\n### Deploy Quickwit\n\nLet's create a basic chart configuration:\n\n```bash\nexport AWS_REGION=us-east-1\nexport AWS_ACCESS_KEY_ID=XXXX\nexport AWS_SECRET_ACCESS_KEY=XXXX\nexport DEFAULT_INDEX_ROOT_URI=s3://your-bucket/indexes\n```\n\n```bash\n# Create Quickwit config file.\necho \"\nsearcher:\n replicaCount: 1\nindexer:\n replicaCount: 1\nmetastore:\n replicaCount: 1\njanitor:\n enabled: true\ncontrol_plane:\n enabled: true\n\nenvironment:\n # Remove ANSI colors.\n NO_COLOR: 1\n\n# Quickwit configuration\nconfig:\n storage:\n s3:\n region: ${AWS_REGION}\n access_key_id: ${AWS_ACCESS_KEY_ID}\n secret_access_key: ${AWS_SECRET_ACCESS_KEY}\n # If you are not on AWS S3, you can define a flavor (gcs, minio, garage...)\n # and additional variables for your object storage.\n # flavor: gcs\n # endpoint: https://storage.googleapis.com\n\n # Metastore on S3.\n metastore_uri: ${DEFAULT_INDEX_ROOT_URI}\n\n default_index_root_uri: ${DEFAULT_INDEX_ROOT_URI}\n\n # Indexer settings\n indexer:\n # By activating the OTEL service, Quickwit will be able\n # to receive gRPC requests from OTEL collectors.\n enable_otlp_endpoint: true\n\" > qw-tutorial-values.yaml\n```\n\nBefore installing Quickwit chart, make sure you have access to S3 and that you did not make a typo in the `default_index_root_uri`. This can be easily done with `aws` CLI with a simple `ls`:\n\n```bash\naws s3 ls ${DEFAULT_INDEX_ROOT_URI}\n```\n\nIf the CLI did not return an error, you are ready to install the chart:\n\n```bash\nhelm install quickwit quickwit/quickwit -f qw-tutorial-values.yaml\n```\n\nIn a few moments, you will see the pods running Quickwit services:\n\n```bash\nkubectl get pods\nNAME READY STATUS RESTARTS AGE\nquickwit-control-plane-7fc495f4c4-slqv4 1/1 Running 2 (84s ago) 87s\nquickwit-indexer-0 1/1 Running 2 (84s ago) 87s\nquickwit-janitor-7f75f4bc8-jrfv6 1/1 Running 2 (84s ago) 87s\nquickwit-metastore-6989978fc-9s82j 1/1 Running 2 (85s ago) 87s\nquickwit-searcher-0 1/1 Running 2 (84s ago) 87s\n```\n\nLet's check Quickwit is working:\n\n```bash\nkubectl port-forward svc/quickwit-searcher 7280\n```\n\nThen open your browser `http://localhost:7280/ui/indexes`. You should see the list of indexes. If everything is fine, keep the kubectl command running and open a new terminal.\n\n### Deploy OTEL collectors\n\nWe need to configure a bit the collectors in order to:\n- collect logs from k8s\n- enrich the logs with k8s attributes\n- export the logs to Quickwit indexer.\n\n```bash\necho \"\nmode: daemonset\npresets:\n logsCollection:\n enabled: true\n kubernetesAttributes:\n enabled: true\nconfig:\n exporters:\n otlp:\n endpoint: quickwit-indexer.qw-tutorial.svc.cluster.local:7281\n tls:\n insecure: true\n # By default, logs are sent to the otel-logs-v0_7.\n # You can customize the index ID By setting this header.\n # headers:\n # qw-otel-logs-index: otel-logs-v0_7\n service:\n pipelines:\n logs:\n exporters:\n - otlp\n\" > otel-values.yaml\n```\n\n```\nhelm install otel-collector open-telemetry/opentelemetry-collector -f otel-values.yaml\n```\n\nAfter a few seconds, you should see logs on your indexer that show indexing has started. It looks like this:\n```\n2022-11-30T18:27:37.628Z INFO spawn_merge_pipeline{index=otel-log-v0 gen=0}: quickwit_indexing::actors::merge_pipeline: Spawning merge pipeline. index_id=otel-log-v0 source_id=_ingest-api-source pipeline_ord=0 root_dir=/quickwit/qwdata/indexing/otel-log-v0/_ingest-api-source merge_policy=StableLogMergePolicy { config: StableLogMergePolicyConfig { min_level_num_docs: 100000, merge_factor: 10, max_merge_factor: 12, maturation_period: 172800s }, split_num_docs_target: 10000000 }\n2022-11-30T18:27:37.628Z INFO quickwit_serve::grpc: Starting gRPC server. enabled_grpc_services={\"otlp-log\", \"otlp-trace\"} grpc_listen_addr=0.0.0.0:7281\n2022-11-30T18:27:37.628Z INFO quickwit_serve::rest: Starting REST server. rest_listen_addr=0.0.0.0:7280\n2022-11-30T18:27:37.628Z INFO quickwit_serve::rest: Searcher ready to accept requests at http://0.0.0.0:7280/\n2022-11-30T18:27:42.654Z INFO quickwit_indexing::actors::indexer: new-split split_id=\"01GK4WPTXK8GH3AGTRNBN9A8YG\" partition_id=0\n2022-11-30T18:27:52.643Z INFO quickwit_indexing::actors::indexer: send-to-index-serializer commit_trigger=Timeout split_ids=01GK4WPTXK8GH3AGTRNBN9A8YG num_docs=22\n2022-11-30T18:27:52.652Z INFO index_batch{index_id=otel-log-v0 source_id=_ingest-api-source pipeline_ord=0}:packager: quickwit_indexing::actors::packager: start-packaging-splits split_ids=[\"01GK4WPTXK8GH3AGTRNBN9A8YG\"]\n2022-11-30T18:27:52.652Z INFO index_batch{index_id=otel-log-v0 source_id=_ingest-api-source pipeline_ord=0}:packager: quickwit_indexing::actors::packager: create-packaged-split split_id=\"01GK4WPTXK8GH3AGTRNBN9A8YG\"\n2022-11-30T18:27:52.653Z INFO index_batch{index_id=otel-log-v0 source_id=_ingest-api-source pipeline_ord=0}:uploader: quickwit_indexing::actors::uploader: start-stage-and-store-splits split_ids=[\"01GK4WPTXK8GH3AGTRNBN9A8YG\"]\n2022-11-30T18:27:52.733Z INFO index_batch{index_id=otel-log-v0 source_id=_ingest-api-source pipeline_ord=0}:uploader:stage_and_upload{split=01GK4WPTXK8GH3AGTRNBN9A8YG}:store_split: quickwit_indexing::split_store::indexing_split_store: store-split-remote-success split_size_in_megabytes=0.018351 num_docs=22 elapsed_secs=0.07654519 throughput_mb_s=0.23974074 is_mature=false\n```\n\nIf you see some errors there, it's probably coming from a misconfiguration of your object storage. If you need some help, please open an issue on [GitHub](https://github.com/quickwit-oss/quickwit) or come on our [discord server](https://discord.gg/MT27AG5EVE).\n\n\n### Ready to search logs\n\nYou are now ready to search, wait 30 seconds and you will see the first indexed logs: just [open the UI](http://localhost:7280/ui/search?query=*&index_id=otel-logs-v0&max_hits=10&sort_by=-timestamp_secs) and play with it. Funny thing you will see quickwit logs in it :).\n\nExample of queries:\n\n- [body.message:quickwit](http://localhost:7280/ui/search?query=body.message:quickwit&index_id=otel-logs-v0&max_hits=10&sort_by=-timestamp_secs)\n- [resource_attributes.k8s.container.name:quickwit](http://localhost:7280/ui/search?query=resource_attributes.k8s.container.name%3Aquickwit&index_id=otel-logs-v0&max_hits=10&sort_by=-timestamp_secs)\n- [resource_attributes.k8s.container.restart_count:1](http://localhost:7280/ui/search?query=resource_attributes.k8s.container.restart_count%3A1&index_id=otel-logs-v0&max_hits=10&sort_by=-timestamp_secs)\n\n\n![UI screenshot](../../assets/screenshot-ui-otel-logs.png)\n\nThat's all, folks!\n\n### Clean up\n\nLet's first delete the index and then uninstall the charts.\n\n```bash\n# Delete the index. The command will return the list of delete split files.\ncurl -XDELETE http://127.0.0.1:7280/api/v1/indexes/otel-logs-v0\n\n# Uninstall charts\nhelm uninstall otel-collector\nhelm uninstall quickwit\n\n# Delete namespace\nkubectl delete namespace qw-tutorial\n```\n\nFinally, you need to delete three JSON files created by Quickwit on your object storage:\n\n```bash\n# if your version <= 0.7.1\naws s3 rm ${DEFAULT_INDEX_ROOT_URI}/indexes_states.json\n# if your version > 0.7.1\naws s3 rm ${DEFAULT_INDEX_ROOT_URI}/manifest.json\n# the metastore file of the logs index\naws s3 rm ${DEFAULT_INDEX_ROOT_URI}/otel-logs-v0_7/metastore.json\n# the metastore file of the traces index\naws s3 rm ${DEFAULT_INDEX_ROOT_URI}/otel-traces-v0_7/metastore.json\n```\n\n## Next step\n\nFollow our [tutorial](../../get-started/tutorials/trace-analytics-with-grafana.md) to install Quickwit Grafana plugin to explore your logs, create dashboards and alerts.\n" }, { "path": "docs/log-management/send-logs/using-otel-collector.md", "content": "---\ntitle: Send logs from OTEL Collector\nsidebar_label: Using OTEL collector\ndescription: Using OTEL Collector\ntags: [otel, collector, log]\nsidebar_position: 1\n---\n\nimport Tabs from '@theme/Tabs';\nimport TabItem from '@theme/TabItem';\n\nIf you already have your own OpenTelemetry Collector and want to export your logs to Quickwit, you need a new OLTP gRPC exporter in your config.yaml:\n\n\n\n\n\n```yaml title=\"otel-collector-config.yaml\"\nreceivers:\n otlp:\n protocols:\n grpc:\n http:\n\nprocessors:\n batch:\n\nexporters:\n otlp/quickwit:\n endpoint: host.docker.internal:7281\n tls:\n insecure: true \n # By default, logs are sent to the otel-logs-v0_7.\n # You can customize the index ID By setting this header.\n # headers:\n # qw-otel-logs-index: otel-logs-v0_7\nservice:\n pipelines:\n logs:\n receivers: [otlp]\n processors: [batch]\n exporters: [otlp/quickwit]\n```\n\n\n\n\n\n```yaml title=\"otel-collector-config.yaml\"\nreceivers:\n otlp:\n protocols:\n grpc:\n http:\n\nprocessors:\n batch:\n\nexporters:\n otlp/quickwit:\n endpoint: 127.0.0.1:7281\n tls:\n insecure: true\n # By default, logs are sent to the otel-logs-v0_7.\n # You can customize the index ID By setting this header.\n # headers:\n # qw-otel-logs-index: otel-logs-v0_7\n\nservice:\n pipelines:\n logs:\n receivers: [otlp]\n processors: [batch]\n exporters: [otlp/quickwit]\n```\n\n\n\n\n\n\n## Test your OTEL configuration\n\n1. [Install](../../get-started/installation.md) and start a Quickwit server:\n \n```bash\n./quickwit run\n```\n\n2. Start a collector with the previous config:\n\n\n\n\n\n```bash\ndocker run -v ${PWD}/otel-collector-config.yaml:/etc/otelcol/config.yaml -p 4317:4317 -p 4318:4318 -p 7281:7281 otel/opentelemetry-collector\n```\n\n\n\n\n\n```bash\ndocker run -v ${PWD}/otel-collector-config.yaml:/etc/otelcol/config.yaml --network=host -p 4317:4317 -p 4318:4318 -p 7281:7281 otel/opentelemetry-collector\n```\n\n\n\n\n\n3. Send a log to your collector with cURL:\n\n```bash\ncurl -XPOST \"http://localhost:4318/v1/logs\" -H \"Content-Type: application/json\" \\\n--data-binary @- << EOF\n{\n \"resource_logs\": [\n {\n \"resource\": {\n \"attributes\": [\n {\n \"key\": \"service.name\",\n \"value\": {\n \"stringValue\": \"test-with-curl\"\n }\n }\n ]\n },\n \"scope_logs\": [\n {\n \"scope\": {\n \"name\": \"manual-test\"\n },\n \"log_records\": [\n {\n \"time_unix_nano\": \"1678974011000000000\",\n \"observed_time_unix_nano\": \"1678974011000000000\",\n \"name\": \"test\",\n \"severity_text\": \"INFO\"\n }\n ]\n }\n ]\n }\n ]\n}\nEOF\n```\n\nYou should see a log on the Quickwit server similar to the following:\n\n```bash\n2023-03-16T13:44:09.369Z INFO quickwit_indexing::actors::indexer: new-split split_id=\"01GVNAKT5TQW0T2QGA245XCMTJ\" partition_id=6444214793425557444\n```\n\nThis means that Quickwit has received the log and created a new split. Wait for the split to be published before searching for logs.\n" }, { "path": "docs/log-management/send-logs/using-vector.md", "content": "---\ntitle: Send logs from Vector\nsidebar_label: Using Vector\ndescription: A simple tutorial to send logs from Vector to Quickwit in a few minutes.\nicon_url: /img/tutorials/vector-logo.png\ntags: [logs, ingestion]\nsidebar_position: 3\n---\n\nimport Tabs from '@theme/Tabs';\nimport TabItem from '@theme/TabItem';\n\n[Vector](https://vector.dev/) is an amazing piece of software (in Rust obviously) and brings a new fresh wind in the observability space,\nit is well-known for collecting logs from every part of your infrastructure, transforming and aggregating them, and finally forwarding them to a sink.\n\nIn this guide, we will show you how to connect it to Quickwit.\n\n## Start Quickwit server\n\n\n\n\n\n```bash\n# Create Quickwit data dir.\nmkdir qwdata\n./quickwit run\n```\n\n\n\n\n\n```bash\n# Create Quickwit data dir.\nmkdir qwdata\ndocker run --rm -v $(pwd)/qwdata:/quickwit/qwdata -p 7280:7280 quickwit/quickwit run\n```\n\n\n\n\n\n## Taking advantage of Quickwit's native support for logs\n\nLet's embrace the OpenTelemetry standard and take advantage of Quickwit features. With the native support for OpenTelemetry standards, Quickwit already comes with an index called `otel-logs_v0_7` that is compatible with the OpenTelemetry [logs data model](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/logs/data-model.md). This means we can start pushing log data without any prior usual index setup.\n\nThe OpenTelemetry index configuration can be found in the [quickwit-opentelemetry/src/otlp/logs.rs](https://github.com/quickwit-oss/quickwit/blob/main/quickwit/quickwit-opentelemetry/src/otlp/logs.rs) source file.\n\n## Setup Vector\n\nOur sink here will be Quickwit ingest API `http://127.0.0.1:7280/api/v1/otel-logs-v0_7/ingest`.\nTo keep it simple in this tutorial, we will use a log source called `demo_logs` that generates logs in a given format. Let's choose the common `syslog` format\n(Vector does not generate logs in the OpenTelemetry format directly!) and use the transform feature to map the `syslog` format into the OpenTelemetry format.\n\n\n```toml title=vector.toml\n[sources.generate_syslog]\ntype = \"demo_logs\"\nformat = \"syslog\"\ncount = 100000\ninterval = 0.001\n\n[transforms.remap_syslog]\ninputs = [ \"generate_syslog\"]\ntype = \"remap\"\nsource = '''\n structured = parse_syslog!(.message)\n .timestamp_nanos = to_unix_timestamp!(structured.timestamp, unit: \"nanoseconds\")\n .body = structured\n .service_name = structured.appname\n .resource_attributes.source_type = .source_type\n .resource_attributes.host.hostname = structured.hostname\n .resource_attributes.service.name = structured.appname\n .attributes.syslog.procid = structured.procid\n .attributes.syslog.facility = structured.facility\n .attributes.syslog.version = structured.version\n .severity_text = if includes([\"emerg\", \"err\", \"crit\", \"alert\"], structured.severity) {\n \"ERROR\"\n } else if structured.severity == \"warning\" {\n \"WARN\"\n } else if structured.severity == \"debug\" {\n \"DEBUG\"\n } else if includes([\"info\", \"notice\"], structured.severity) {\n \"INFO\"\n } else {\n structured.severity\n }\n .scope_name = structured.msgid\n del(.message)\n del(.timestamp)\n del(.service)\n del(.source_type)\n'''\n\n# useful to see the logs in the terminal\n# [sinks.emit_syslog]\n# inputs = [\"remap_syslog\"]\n# type = \"console\"\n# encoding.codec = \"json\"\n\n[sinks.quickwit_logs]\ntype = \"http\"\nmethod = \"post\"\ninputs = [\"remap_syslog\"]\nencoding.codec = \"json\"\nframing.method = \"newline_delimited\"\nuri = \"http://127.0.0.1:7280/api/v1/otel-logs-v0_7/ingest\"\n```\nDownload the above Vector config file.\n\n```bash\ncurl -o vector.toml https://raw.githubusercontent.com/quickwit-oss/quickwit/main/config/tutorials/vector-otel-logs/vector.toml\n```\n\nNow let's start Vector so that we can start sending logs to Quickwit.\n\n```bash\ndocker run -v $(pwd)/vector.toml:/etc/vector/vector.toml:ro -p 8383:8383 --net=host timberio/vector:0.25.0-distroless-libc\n```\n\n## Search logs\n\nQuickwit is now ingesting logs coming from Vector and you can search them either with `curl` or by using the UI:\n- `curl -XGET http://127.0.0.1:7280/api/v1/otel-logs-v0_7/search?query=severity_text:ERROR`\n- Open your browser at `http://127.0.0.1:7280/ui/search?query=severity_text:ERROR&index_id=otel-logs-v0_7&max_hits=10` and play with it!\n\n## Compute aggregation on severity_text\n\nFor aggregations, we can't use yet Quickwit UI but we can use cURL.\n\nLet's craft a nice aggregation query to count how many `INFO`, `DEBUG`, `WARN`, and `ERROR` per minute (all datetime are stored in microseconds thus the interval of 60_000_000 microseconds) we have:\n\n```json title=aggregation-query.json\n{\n \"query\": \"*\",\n \"max_hits\": 0,\n \"aggs\": {\n \"count_per_minute\": {\n \"histogram\": {\n \"field\": \"timestamp_nanos\",\n \"interval\": 60000000\n },\n \"aggs\": {\n \"severity_text_count\": {\n \"terms\": {\n \"field\": \"severity_text\"\n }\n }\n }\n }\n }\n}\n```\n\n```bash\ncurl -XPOST -H \"Content-Type: application/json\" http://127.0.0.1:7280/api/v1/otel-logs-v0_7/search --data @aggregation-query.json\n```\n\n## Going further\n\nNow you can also deploy Grafana and connect to Quickwit as data source for query, dashboard, alerts and more!\n" }, { "path": "docs/log-management/supported-agents.md", "content": "---\ntitle: Supported agents\nsidebar_position: 3\n---\n\nQuickwit is compatible with the following agents:\n\n## OpenTelemetry agent\n\nBefore using an [OpenTelemetry collector](https://opentelemetry.io/docs/collector/), check that [Quickwit OpenTelemetry service](otel-service.md) is enabled.\nOnce started, Quickwit is then ready to receive and ingest OpenTelemetry gRPC requests.\n\nHere is a configuration example of an OpenTelemetry agent that sends logs into Quickwit:\n\n```yaml\nmode: daemonset\npresets:\n logsCollection:\n enabled: true\n kubernetesAttributes:\n enabled: true\nconfig:\n exporters:\n otlp:\n # Replace quickwit-host with the hostname of your Quickwit node/service.\n # On k8s, it should be of the form `{quickwit-indexer-service-name}.{namespace}.svc.cluster.local:7281\n endpoint: quickwit-host:7281\n tls:\n insecure: true\n service:\n pipelines:\n logs:\n exporters:\n - otlp\n```\n\nFind more configuration details on the [OpenTelemetry documentation](https://opentelemetry.io/docs/collector/configuration/). You can also check out our [tutorial to send logs with OTEL collector](send-logs/using-otel-collector.md) to Quickwit.\n\n## HTTP-based agents\n\nIt's also possible to use other agents that send HTTP requests to Quickwit Ingest API. Quickwit also partially supports Elasticseardch `_bulk` API. Thus, there is a good chance that your agent is already compatible with Quickwit.\nCurrently, we have tested the following HTTP-based agents:\n\n- [Vector](send-logs/using-vector.md)\n- [Fluentbit](send-logs/using-fluentbit.md)\n- FluentD (tutorial coming soon)\n- Logstash: Quickwit does not support the Elasticsearch output. However, it's possible to send logs with the HTTP output but with `json` [format](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-http.html) only.\n\nQuickwit natively supports the [OpenTelemetry Protocol (OTLP)](https://opentelemetry.io/docs/reference/specification/protocol/otlp/) and provides a gRPC endpoint to receive logs from an OpenTelemetry collector by default.\n\nThe logs received by this endpoint are indexed on the `otel-logs-v0` index. This index will be automatically created if not present. The index doc mapping is described in this [section](#opentelemetry-logs-data-model).\n\nYou can also send your logs directly to this index by using the [ingest API](/docs/reference/rest-api.md#ingest-data-into-an-index).\n\n## OpenTelemetry service\n\nQuickwit natively supports the [OpenTelemetry Protocol (OTLP)](https://opentelemetry.io/docs/reference/specification/protocol/otlp/) and provides a gRPC endpoint to receive spans from an OpenTelemetry collector. This endpoint is enabled by default.\n\nWhen enabled, Quickwit will start the gRPC service ready to receive spans from an OpenTelemetry collector. The spans are indexed in the `otel-trace-v0_7` index by default, and this index will be automatically created if not present. The index doc mapping is described in the next [section](#trace-and-span-data-model).\n\nIf for any reason, you want to disable this endpoint, you can:\n- Set the `QW_ENABLE_OTLP_ENDPOINT` environment variable to `false` when starting Quickwit.\n- Or [configure the node config](/docs/configuration/node-config.md) by setting the indexer setting `enable_otlp_endpoint` to `false`.\n\n```yaml title=node-config.yaml\n# ... Indexer configuration ...\nindexer:\n enable_otlp_endpoint: false\n```\n" }, { "path": "docs/operating/_category_.yaml", "content": "label: 'Operating Quickwit'\nposition: 7\ncollapsed: true\n" }, { "path": "docs/operating/aws-costs.md", "content": "---\ntitle: AWS Cost Optimization\nsidebar_position: 3\n---\n\nQuickwit has been tested on Amazon S3. This page sums up what we have learned from that experience.\n\n## Real World Example\nIn this [blog post](https://quickwit.io/blog/benchmarking-quickwit-engine-on-an-adversarial-dataset#indexing-costs), we indexed 23 TB of data and evaluated performance and costs.\nYou may be able to deduce the costs of indexing and querying on your dataset.\n\n## Data transfers costs and latency\n\nCloud providers charge for data transfers in and out of their networks. In addition, querying an index from a remote machine adds some extra latency.\nFor those reasons, we recommend that you test and use the Quickwit from an instance located within your cloud provider's network.\n\n## Optimizing bandwidth with wisely chosen instances\n\nWe recommend picking instances with high network performance to allow faster downloads from Amazon S3. In our experience, `c5n.2xlarge` instances offer the best bang for your buck.\n\n## Requests cost\n\nA final note on object storage requests costs. These are [quite low](https://aws.amazon.com/s3/pricing/) actually, $0,0004 / 1000 requests for GET and $0.005 / 1000 requests for PUT on AWS S3.\n\n### PUT requests\n\nDuring indexing, Quickwit uploads new splits on Amazon S3 and progressively merges them until they reach 10 million documents that we call “mature splits”. Such splits have a typical size between 1GB and 10GB and will usually require 2 PUT requests to be uploaded (1 PUT request / 5GB).\n\nWith default indexing parameters `commit_timeout_secs` of 60 seconds and `merge_policy.merge_factor` of 10 and assuming you want to ingest 1 million documents every minute, this will cost you less than $1 / month.\n\n### GET requests\n\nWhen querying, Quickwit needs to make multiple GET requests:\n\n```jsx\n#num requests = #num splits * ((#num search fields * #num terms * 3) + (#num search fields with fieldnorms enabled) + 1 (timestamp fast field if present)) + #num docs returned\n```\n\nThe above formula assumes that the hotcache is cached, which will be loaded after the first query for every split.\n`#num splits` can be reduced with [pruning](../overview/concepts/querying.md).\n\nWhen positions are not enabled, only 2 GET requests will be executed per term.\n\nThese requests costs could add up quickly if you have a high number of splits or QPS > 10.\nDon't hesitate to [contact us](mailto:hello@quickwit.io) if this is the case :).\n" }, { "path": "docs/operating/data-directory.md", "content": "---\ntitle: Data directory\nsidebar_position: 1\n---\n\nQuickwit operates on a local directory to store temporary files. By default, this working directory is named `qwdata` and placed right next to the Quickwit binary.\n\nLet's have a look at how Quickwit organizes the data directory.\n\n## Data directory layout\n\nWhen operating Quickwit, you will end up with the following tree:\n\n```bash\nqwdata\n├── cache\n│ └── splits\n| ├── 03BSWV41QNGY5MZV9JYAE1DCGA.split\n│ └── 01GSWV41QNGY5MZV9JYAE1DCB7.split\n├── delete_task_service\n│ └── wikipedia%01H13SVKDS03P%TpCfrA\n├── indexing\n│ ├── wikipedia%01H13SVKDS03P%_ingest-api-source%RbaOAI\n│ └── wikipedia%01H13SVKDS03P%kafka-source%cNqQtI\n├── wal\n│ ├── wal-00000000000000000056\n│ └── wal-00000000000000000057\n└── queues\n ├── partition_id\n ├── wal-00000000000000000028\n └── wal-00000000000000000029\n```\n\n### `/queues` and `/wal` directories\n \nThese directories are created only if the ingest API service is running on your node. They contain write ahead log files of the ingest API to guard against data loss. The `/queues` directory is used by the legacy version of the ingest (sometimes referred to as ingest V1). It is meant to be phased out in upcoming versions of Quickwit. Learn more about ingest API versions [here](../ingest-data/ingest-api.md#ingest-api-versions).\n\nThe log file is truncated when Quickwit commits a split (piece of index), which means that the split is stored on the storage and its metadata are in the metastore.\n\nYou can configure `max_queue_memory_usage` and `max_queue_disk_usage` in the [node config file](../configuration/node-config.md#ingest-api-configuration) to limit the max disk usage.\n\n### `/indexing` directory\n\nThis directory holds the local indexing directory of each indexing source of each index managed by Quickwit. In the above tree, you can see two directories corresponding to the `wikipedia` index, which means that index is currently handling two sources.\n\n\n### `/delete_task_service` directory\n\nThis directory is used by the Janitor service to apply deletes on indexes. During this process, splits are downloaded, a new split is created while applying deletes and uploaded to the target storage. This directory gets created only on nodes running the Janitor service.\n\n### `/cache` directory\n\nThis directory is used for caching splits that will undergo a merge operation to save disk IOPS. Splits are evicted if they are older than two days. If cache limits are reached, oldest splits are evicted.\n\nYou can [configure](../configuration/node-config#indexer-configuration) the number of splits the cache can hold with `split_store_max_num_splits` and limit the overall size in bytes of splits with `split_store_max_num_bytes`.\n\n### `/searcher-split-cache` directory\n\nThis directory is used by searcher nodes to cache entire splits and reduce calls to the object store. It won't be created unless you set the `split_cache` fields in the [searcher configuration](../configuration/node-config.md#searcher-configuration).\n\n\n## Setting the right splits cache limits\n\nCaching splits saves disk IOPS when Quickwit needs to merge splits.\n\nSetting the right limits will depend on your [merge policy](../configuration/index-config.md#merge-policies) and the number of partitions you are using. The default splits cache limits should fit most use cases.\n\n### Splits cache with the default configuration\n\nFor a given index, Quickwit commits one split every minute and uses the \"Stable log\" [merge policy](../configuration/index-config.md#merge-policies). This merge policy by default merges splits by group of 10, 11, or 12 until splits have more than 10 millions of documents. A split will typically undergo 3 or 4 merges and after will be considered as mature and evicted from the cache.\n\nThe following table shows how many splits will be created after a given amount of time assuming a 20MB/s ingestion rate with a compression ratio of 0.5:\n\n| Time (minutes) | Number of splits | Splits size (GB) |\n| -------------- | -------------------------------------- | ----------- |\n| 1 | 1 | 0.6 GB |\n| 2 | 2 | 1.2 GB |\n| 10 | 10 | 6 GB |\n| 11 | 1 + 1 (merged once) | 6.6 GB |\n| 21 | 1 + 2 (merged once) | 12.6 GB |\n| 91 | 1 + 9 (merged once) | 54.6 GB |\n| 101 | 1 + 1 (merged twice) | 60.6 GB |\n| 111 | 2 + 1 (merged once) + 1 (merged twice) | 66.6 GB |\n| 201 | 1 + 0 (merged once) + 2 (merged twice) | 120.6 GB |\n| .. | ... | |\n\nIn this case, the default cache limits of 1000 splits and 100GB are good enough to avoid downloading splits from the storage for the first two merges. This is perfectly fine for a production use case. You may want to increase the splits cache size to avoid any split download.\n\nYou can monitor the download rate with our [indexers dashboard](monitoring.md).\n\n### Splits cache with partitioning\n\nWhen using [partitions](../overview/concepts/querying.md#partitioning), Quickwit will create one split per partition and the number of splits can add up very quickly.\n\nLet's take a concrete example with the following assumptions:\n- A [commit timeout](../configuration/index-config.md#indexing-settings) of 1 minute.\n- A partitioning that has 100 partitions. Quickwit will create 100 splits per minute assuming a document of each partition is ingested in one minute.\n- A merge policy that merges splits of same partition as soon as there is 10 splits.\n\nThe following table shows how many splits will be created after a given amount of time:\n\n| Time (minutes) | Number of splits |\n| ------------ | ---------------- |\n| 1 | 100 |\n| 2 | 200 |\n| 10 | 1000 |\n| 11 | 100 + 100 (merged once) |\n| 21 | 100 + 200 (merged once) |\n| 91 | 100 + 900 (merged once) |\n| 100 | 1000 + 900 (merged once) |\n| 101 | 100 + 0 (merge once) + 100 (merged twice) |\n| 200 | 1000 + 900 (merged once) + 100 (merged twice) |\n| 201 | 100 + 0 (merged once) + 200 (merged twice) |\n\nWith these assumptions, you have to set `split_store_max_num_splits` to at least 1000 to avoid downloading splits from the storage for the first merge operation. And as merging can take a bit of time, you should set `split_store_max_num_splits` to a value that can hold all the splits that are not yet merged plus the incoming splits, a value of 1100 splits should be enough. If you want to store split until the second merge, a limit of 2500 splits should be good enough.\n\n## Troubleshooting with a huge number of local splits\n\nWhen starting, Quickwit is scanning all the splits in the cache directory to know which split is present locally, this can take a few minutes if you have tens of thousands splits. On Kubernetes, as your pod can be restarted if it takes too long to start, you may want to clean up the data directory or increase the liveliness probe timeout.\nAlso please report such a behavior on [GitHub](https://github.com/quickwit-oss/quickwit) as we can certainly optimize this start phase.\n" }, { "path": "docs/operating/monitoring.md", "content": "---\ntitle: Monitoring with Grafana\nsidebar_position: 2\n---\n\nYou can monitor your Quickwit cluster with Grafana.\nFollow the tutorial at [Quickwit Monitoring with Grafana](../get-started/tutorials/prometheus-metrics) on how to set it up.\n\nWe provide three Grafana dashboards to help you monitor:\n- [indexers performance](https://github.com/quickwit-oss/quickwit/blob/main/monitoring/grafana/dashboards/indexers.json)\n- [searchers performance](https://github.com/quickwit-oss/quickwit/blob/main/monitoring/grafana/dashboards/searchers.json)\n- [metastore queries](https://github.com/quickwit-oss/quickwit/blob/main/monitoring/grafana/dashboards/metastore.json)\n\nDashboards rely on a prometheus datasource fed with [Quickwit metrics](../reference/metrics.md).\n\n## Screenshots\n\n![Indexers Grafana Dashboard](../assets/images/screenshot-indexers-grafana-dashboard.png)\n\n![Searchers Grafana Dashboard](../assets/images/screenshot-searchers-grafana-dashboard.png)\n\n![Metastore Grafana Dashboard](../assets/images/screenshot-metastore-grafana-dashboard.png)\n" }, { "path": "docs/operating/upgrades.md", "content": "---\ntitle: Version upgrade\nsidebar_position: 4\n---\n\n## Migration from 0.6.x to 0.7.0\n\nThe format of the index and internal objects stored in the metastore of 0.7 is backward compatible with 0.6.\n\nIf you are using the OTEL indexes and ingesting data into indexes the `otel-logs-v0_6` and `otel-traces-v0_6`, you must stop indexing before upgrading. Indeed, the first time you start Quickwit 0.7, it will update the doc mapping fields of Trace ID and Span ID of those two indexes by changing their input/output formats from `base64` to `hex`. This is automatic: you don't have to perform any manual operation.\n\nQuickwit 0.7 will also create the new index `otel-traces-v0_7`, which is now used by default when ingesting data with the OTEL gRPC and HTTP API. The Jaeger gRPC and HTTP APIs will query both `otel-traces-v0_6` and `otel-traces-v0_7` by default. It's possible to define the index ID you want to use for OTEL gRPC endpoints and Jaeger gRPC API by setting the request header `qw-otel-logs-index` or `qw-otel-traces-index` to the index ID you want to target.\n\n\n## Migration from 0.7.0 to 0.7.1\n\nQuickwit 0.7.1 will create the new index `otel-logs-v0_7` which is now used by default when ingesting data with the OTEL gRPC and HTTP API.\n\nIn the traces index `otel-traces-v0_7`, the `service_name` field is now `fast`. \nNo migration is done if `otel-traces-v0_7` already exists. If you want `service_name` field to be `fast`, you have to delete first the existing `otel-traces-v0_7` index or you need to create your own index.\n\n## Migration from 0.8 to 0.9\n\nQuickwit 0.9 introduces a new ingestion service to to power the ingest and bulk APIs (v2). The new ingest is enabled and used by default, even though the legacy one (v1) remains enabled to finish indexing residual data in the legacy write ahead logs. Note that `ingest_api.max_queue_disk_usage` is enforced on both ingest versions separately, which means that the cumulated disk usage might be up to twice this limit.\n\nWhen upgrading to 0.9, we recommend to perform a full cluster restart.\n\n\n\nShutdown order:\n1) indexers, searchers and janitor\n2) control plane\n3) metastores\n\nStart up order:\n1) metastores\n2) control plane\n3) indexers, searchers and janitor\n" }, { "path": "docs/overview/_category_.yaml", "content": "label: 'Introduction'\nposition: 1\ncollapsed: true\n" }, { "path": "docs/overview/architecture.md", "content": "---\ntitle: Architecture\nsidebar_position: 2\n---\n\nQuickwit distributed search engine relies on 4 major services and one maintenance service:\n\n- The Searchers for executing search queries from the REST API.\n- The Indexers that index data from data sources.\n- The Metastore that stores the index metadata in a PostgreSQL-like database or in a cloud storage file.\n- The Control plane that schedules indexing tasks to the indexers.\n- The Janitor that executes periodic maintenance tasks.\n\nMoreover, Quickwit leverages existing infrastructure by relying on battled-tested technologies for index storage, metadata storage, and ingestion:\n\n- Cloud storage like AWS S3, Google Cloud Storage, Azure Blob Storage or other S3 compatible storage for index storage.\n- Postgresql for metadata storage.\n- Distributed queues like Kafka and Pulsar for ingestion.\n\n## Architecture diagram\n\nThe following diagram shows a Quickwit cluster with its four major components and the janitor whose role is to execute periodic maintenance tasks, see the [Janitor section](#janitor) for more details.\n\n![Quickwit Architecture](../assets/images/quickwit-architecture-light.svg#gh-light-mode-only)![Quickwit Log Management](../assets/images/quickwit-architecture-dark.svg#gh-dark-mode-only)\n\n## Index & splits\n\nA Quickwit index stores documents and makes it possible to query them efficiently. The index organizes documents into a collection of smaller independent indexes called **splits**.\n\nA document is a collection of fields. Fields can be stored in different data structures:\n\n- an inverted index, which enables fast full-text search.\n- a columnar storage called `fast field`. It is the equivalent of doc values in [Lucene](https://lucene.apache.org/). Fast fields are required to compute aggregates over the documents matching a query. They can also allow some advanced types of filtering.\n- a row-storage called the doc store. It makes it possible to get the content of the matching documents.\n\nYou can configure your index to control how to map your JSON object to a Quickwit document and, for each field, define whether it should be stored, indexed, or be a fast field. [Learn how to configure your index](../configuration/index-config.md)\n\n### Splits\n\nA split is a small piece of an index identified by a UUID. For each split, Quickwit adds up a `hotcache` file along with index files. This **hotcache** is what makes it possible for Searchers to open a split in less than 60ms, even on high latency storage.\n\nThe Quickwit index is aware of its splits by keeping splits metadata, notably:\n\n- the split state which indicates if the split is ready for search\n- the min/max time range computed on the timestamp field if present.\n\nThis timestamp metadata can be handy at query time. If the user specifies a time range filter to their query, Quickwit will use it to **prune irrelevant splits**.\n\nIndex metadata needs to be accessible by every instance of the cluster. This is made possible thanks to the `metastore`.\n\n### Index storage\n\nQuickwit stores the indexes data (splits files) on cloud storage (AWS S3, Google Cloud Storage, Azure Blob Storage or other S3 compatible storage) and also on local disk for single-server deployment.\n\n## Metastore\n\nQuickwit gathers index metadata into a metastore to make them available across the cluster. \n\nOn the write path, indexers push index data on the index storage and publish metadata to the metastore.\n\nOn the read path, for a given query on a given index, a search node will ask the metastore for the index metadata and then use it to do the query planning and finally execute the plan.\n\nIn a clustered deployment, the metastore is typically a traditional RDBMS like PostgreSQL which we only support today. In a single-server deployment, it’s also possible to rely on a local file or on Amazon S3.\n\n## Quickwit cluster and services\n\n### Cluster formation\n\nQuickwit uses [chitchat](https://github.com/quickwit-oss/chitchat), a cluster membership protocol with failure detection implemented by Quickwit. The protocol is inspired by Scuttlebutt reconciliation and phi-accrual detection, ideas borrowed from Cassandra and DynamoDB.\n\n[Learn more on chitchat](https://github.com/quickwit-oss/chitchat).\n\n### Indexers\n\nSee [dedicated indexing doc page](./concepts/indexing.md).\n\n### Searchers\n\nQuickwit's search cluster has the following characteristics:\n\n- It is composed of stateless nodes: any node can answer any query about any splits.\n- A node can distribute search workload to other nodes.\n- Load-balancing is made with rendezvous hashing to allow for efficient caching.\n\nThis design provides high availability while keeping the architecture simple.\n\n**Workload distribution: root and leaf nodes**\n\nAny search node can handle any search request. A node that receives a query will act as the root node for the span of the request. It will then process it in 3 steps:\n\n- Get the index metadata from the metastore and identify the splits relevant to the query.\n- Distributes the split workload among the nodes of the cluster. These nodes are assuming the role of leaf nodes.\n- Waits for results from leaf nodes, merges them, and returns the aggregated results.\n\n**Stateless nodes**\n\nQuickwit cluster distributes search workloads while keeping nodes stateless.\n\nThanks to the hotcache, opening a split on Amazon S3 only takes 60ms. It makes it possible to remain totally stateless: a node does not need to know anything about the indexes. Adding or removing nodes takes seconds and does not require moving data around.\n\n**Rendezvous hashing**\n\nThe root node uses [Rendezvous hashing](https://en.wikipedia.org/wiki/Rendezvous_hashing) to distribute the workload among leaf nodes. Rendez-vous hashing makes it possible to define a node/split affinity function with excellent stability properties when a node joins or leaves the cluster. This trick unlocks efficient caching.\n\nLearn more about query internals on the [querying doc page](./concepts/querying.md).\n\n\n### Control plane\n\nThe control plane service schedules indexing tasks to indexers. The scheduling is executed when the scheduler receives external or internal events and on certains conditions:\n\n- The scheduler listens to metastore events: source create, delete, toggle, or index delete. On each of these events, it will schedule a new plan, named the `desired plan` and send indexing tasks to the indexers.\n- On every `HEARTBEAT` (3 seconds), the scheduler controls if the `desired plan` and the indexing tasks running on indexers are in sync. If not, it will reapply the desired plan to indexers.\n- Every minute, the scheduler rebuilds a plan with the latest metastore state, and if it differs from the last applied plan, it will apply the new one. This is necessary as the scheduler may have not received all metastore events due to network issues.\n\n### Janitor\n\nThe Janitor service runs maintenance tasks on indexes: garbage collection, delete query tasks, and retention policy tasks.\n\n## Data sources\n\nQuickwit supports [multiple sources](../ingest-data/) to ingest data from.\n\nA file is ideal for a one-time ingestion like an initial load, the ingest API or a message queue are ideal to continuously feed data into the system. \n\nQuickwit indexers connect directly to external message queues like Kafka, Pulsar or Kinesis and guarantee the exactly-once semantics. If you need support for other distributed queues, please vote for yours [here](https://github.com/quickwit-oss/quickwit/issues/1000).\n" }, { "path": "docs/overview/concepts/_category_.yaml", "content": "label: 'Advanced concepts'\nposition: 3\ncollapsed: true\n" }, { "path": "docs/overview/concepts/deletes.md", "content": "---\ntitle: Deletes\nsidebar_position: 3\n---\n\nQuickwit supports deletes thanks to the [delete API](../../reference/rest-api.md#delete-api). It's important to note that this feature is mainly intended to comply with GDPR (General Data Protection Regulation) and should be used parsimoniously as deletes are expensive: typically a few queries per hour or day is recommended.\n\n## Delete tasks\n\nA delete task on a given index is executed on all splits created before the delete task creation. This can be a long-running task that could last several hours if the delete query is matching documents present in many splits.\n\nTo track the progress of the execution, each delete task is given a unique and incremental identifier called \"operation stamp\" or `opstamp`. All existing splits will undergo a delete operation and, after its success, each split metadata will be updated with the corresponding operation stamp.\n\nAll splits created after the creation of a delete tasks will have a `opstamp` greater or equal to the `opstamp` of the delete task (greater if other delete tasks have been created at the same moment).\n\nQuickwit batches delete operations on a given split: for example, if a split has it delete `opstamp = n` and the last created delete task has a `opstamp = n + 10`, ten (10) delete queries will be executed at once on the split.\n\n## Delete API\n\nDelete tasks are created through the [Delete REST API](../../reference/rest-api.md#delete-api).\n\n## Pitfalls\n\n### Immature splits\n\nDelete operations are applied only to “mature” splits, that is splits that will no longer undergo merges. Whether a split is mature depends on the [merge policy](../../configuration/index-config.md#merge-policies). It is possible to define `maturation_period` after which a split will be mature. Thus, a delete request created at `t0` will first apply deletes to mature splits and, in the worst case, will wait the `t0 + maturation_period` for immature splits to become mature.\n\n\n### Monitoring and dev XP\n\nIt's currently not possible to monitor delete operations. An [issue](https://github.com/quickwit-oss/quickwit/issues/2494) is opened to improve the dev experience, don't hesitate to add your comments it and follow its progress.\n" }, { "path": "docs/overview/concepts/indexing.md", "content": "---\ntitle: Indexing\nsidebar_position: 1\n---\n\n## Supported data formats\n\nQuickwit ingests JSON records and refers to them as \"documents\" or \"docs\". Each document must be a JSON object. When ingesting files, documents must be separated by a newline.\n\nQuickwit does not yet support file formats such as `Avro` or `CSV`. Compression formats such as `bzip2` or `gzip` are also not supported yet.\n\n## Data model\n\nQuickwit supports both schemaless indexes and fixed schemas. The \"document mapping\" of an index, also commonly called \"doc mapping\", is a list of field names and types that declares the schema of an index. For a schemaless or mixed fixed schema and schemaless indexing, follow our [guide on schemaless indexing](../../guides/schemaless.md). Additionally, a doc mapping specifies how documents are indexed (tokenizers) and stored (column-oriented vs. row-oriented).\n\n\n## Merge process and merge policy\n\nAn index is broken into immutable splits. The size of a split is defined by the number of documents it carries. A split is considered \"mature\" when its size reaches a threshold defined in the index config as `split_num_docs_target`.\n\nAn indexer buffers incoming documents and produces a new split when the size of the buffer reaches `split_num_docs_target` or `commit_timeout_secs` seconds have passed since the first document has been enqueued, depending on which event occurs first. In the latter case, the indexer generates immature splits. The merge process designates the iterative procedure that groups and merges immature splits together to produce mature splits.\n\nThe merge policy controls the merge algorithm, which is mainly driven by the two parameters `split_num_docs_target` and `merge_factor`. Each time a new split is published, the merge policy examines the list of immature splits and attempts to merge `merge_factor` splits together in order to produce larger splits. The merge policy may also decide to merge fewer or more splits together if deemed necessary. Finally, the merge algorithm never merges more than `max_merge_factor` splits together.\n\n### Split store\n\nThe split store is a cache that keeps recently published and immature splits on disk to speed up the merge process. After a successful merge phase, the split store evicts dangling splits.\n\nThe disk space allocated to the split store is controlled by the config parameters `split_store_max_num_splits` and `split_store_max_num_bytes`.\n\n## Data sources\n\nA data source designates the location and set of parameters that allow to connect to and ingest data from an external data store, which can be a file, a stream, or a database. Often, Quickwit simply refers to data sources as \"sources\". The indexing engine supports local adhoc file ingests using [the CLI](/docs/reference/cli#tool-local-ingest) and streaming sources (e.g. the Kafka source). Quickwit can insert data into an index from one or multiple sources. More details can be found [in the source configuration page](https://quickwit.io/docs/configuration/source-config).\n\n## Checkpoint\n\nQuickwit achieves exactly-once processing using checkpoints. For each source, a \"source checkpoint\" records up to which point documents have been processed in the target file or stream. Checkpoints are stored in the metastore and updated atomically each time a new split is published. When an indexing error occurs, the indexing process is resumed right after the last successfully published checkpoint. Internally, a source checkpoint is represented as an object mapping from absolute paths or partition IDs to offsets or sequence numbers.\n" }, { "path": "docs/overview/concepts/querying.md", "content": "---\ntitle: Querying\nsidebar_position: 2\n---\n\nA search query received by a searcher will be executed using a map-reduce approach following these steps:\n\n1. The Searcher identifies relevant splits based on the request’s [timestamp interval](#time-sharding) and [tags](#tag-pruning).\n2. It distributes the splits workload among other searchers available in the cluster using *[rendez-vous hashing](https://en.wikipedia.org/wiki/Rendezvous_hashing)* to optimize caching and load.\n3. It finally waits for all results, merges them, and returns them to the client.\n\nA search stream query follows the same execution path as for a search query except for the last step: instead of waiting for each Searcher's result, the searcher streams the results as soon as it starts receiving some from a searcher.\n\n### **Time sharding**\n\nOn datasets with a time component, Quickwit will shard data into timestamp-aware splits. With this feature, Quickwit is capable of filtering out most splits before they can make it to the query processing stage, thus reducing drastically the amount of data needed to process the query.\n\nThe following query parameters are available to apply timestamped pruning to your query:\n\n- `startTimestamp`: restricts search to documents with a `timestamp >= start_timestamp`\n- `endTimestamp`: restricts search to documents with a `timestamp < end_timestamp`\n\n### Tag pruning\n\nQuickwit also provides pruning on a second dimension called `tags`. By [setting a field as tagged](../../configuration/index-config.md) Quickwit will generate split metadata at indexing in order to filter splits that match requested tags at query time. Note that this metadata is only generated when the cardinality of the field is less than 1,000.\n\nTag pruning is notably useful on multi-tenant datasets.\n\n### Partitioning\n\nQuickwit makes it possible to route documents into different splits based on a partitioning key.\n\nThis feature is especially useful in a context where documents with different\ntags are all mixed together in the same source (usually a Kafka topic).\n\nIn that case, simply marking the field as tag will have no positive effect on search, as all produced splits will contain almost all tags.\n\nThe `partition_key` attributes (defined in the doc mapping) lets you configure the logic used by Quickwit to route documents into isolated splits.\nQuickwit will also enforce this isolation during merges. This functionality is, in a sense, similar to sharding.\n\nQuickwit supports a simple DSL for partitioning described in the next section.\n\nPartition & tags are often used to:\n\n- separate `tenants` in a multi-tenant application\n- separate `team` or `application` in an observation logging case.\n\nEmitting many splits can heavily stress an `indexer`. For this reason,\nanother parameter of the doc mapping called `max_num_partitions` acts as a safety valve. If the number of partitions is\nabout to exceed `max_num_partitions`, a single extra partition is created\nand all extra partitions will be grouped together into this special partition.\n\nIf you are expecting 20 partitions, we strongly recommend you to not set\n`max_num_partitions` to 20, but instead use a larger value (200 for instance).\nQuickwit should handle that number of partitions smoothly, and it will avoid documents belonging to different partitions from being grouped together due to\na few faulty documents.\n\n### Partition key DSL\n\nQuickwit allows you to configure how document are routed with a simple DSL. Here are some sample expression with a short description of their result:\n\n- `tenant_id`: create one partition per tenant\\_id\n- `tenant_id,app_id`: create one partition per unique combination of tenant\\_id and app\\_id\n- `tenant_id,hash_mod(app_id, 8)`: for each tenant, create up to 8 partitions containing each data related to some applications\n- `hash_mod((tenant_id,app_id), 50)`: create 50 partition in total, containing some combination of tenant and apps.\n\n\nThe partition key DSL is generated by this grammar:\n```\nRoutingExpr := RoutingSubExpr [ , RoutingExpr ]\nRougingSubExpr := Identifier [ \\( Arguments \\) ]\nIdentifier := FieldChar [ Identifier ]\nFieldChar := { a..z | A..Z | 0..9 | _ }\nArguments := Argument [ , Arguments ]\nArgument := { \\( RoutingExpr \\) | RoutingSubExpr | DirectValue }\n# We may want other DirectValue in the future\nDirectValue := Number\nNumber := { 0..9 } [ Number ]\n```\nSupported functions are currently:\n- `hash_mod(RoutingExpr, Number)`: hash `RoutingExpr` and divide the result by `Number`, keeping only the reminder.\n\nWhen using `hash_mod` with a tuple of key like in `hash_mod((tenant_id,app_id), 50)`, beware it might route together documents which would make tags less effective.\nFor instance, if tenant\\_1,app\\_1 and tenant\\_2,app\\_2 are both sent to partition one, but tenant\\_1,app\\_2 is sent to partition two, a query for tenant\\_1,app\\_2 will\nstill search inside the 1st partition as it will be tagged with tenant\\_1,tenant\\_2,app\\_1 and app\\_2. You should therefore prefer a partition key such as\n`hash_mod(tenant_id, 10),hash_mod(app_id, 5)` which will generate as many splits, but with better tags.\n\n### Caching\n\nQuickwit does caching in many places to deliver a highly performing query engine.\n\nIn memory:\n\n- Hotcache caching: A static cache that holds information about a split file internal representation. It helps speed up the opening of a split file. Its size can be defined via the `split_footer_cache_capacity` configuration parameter.\n- Fast field caching: Fast fields tend to be accessed very frequently by users especially for stream requests. They are cached in a RAM whose size can be limited by the `fast_field_cache_capacity` configuration value.\n- Partial request caching: In some cases, like when using dashboards, some very similar requests might be issued, with only timestamp bounds changing. Some partial results can be cached to make these requests faster and issue less requests to the storage. They are cached in a RAM whose size can be limited by the `partial_request_cache_capacity` configuration value.\n\nOn disk:\n\n- The split cache stores entire splits on disk. It can be enabled by setting the `split_cache` configuration fields. This cache can help reduce object store costs and load. Searchers populate this cache when splits are created or queried and evict them with a simple LRU strategy.\n\nLearn more about cache parameters in the [searcher configuration docs](../../configuration/node-config.md#searcher-configuration).\n\n### Scoring\n\nQuickwit supports sorting docs by their BM25 scores. In order to query by score, [fieldnorms](../../configuration/index-config.md#text-type) must be enabled for the field. By default, BM25 scoring is disabled to improve query latencies but it can be opt-in by setting the `sort_by` option to `_score` in queries.\n\n### Document ID\n\nEach document in Quickwit is assigned a unique document ID, which is a combination of the split ID and the Tantivy DocId within the split. This implies that you cannot assign a custom ID and that the ID changes when splits undergo merges. This ID is used for every search query as sort order (after the explicitly specified sort values) to make the results deterministic.\n" }, { "path": "docs/overview/index.md", "content": "---\ntitle: Quickwit documentation\nslug: /\nsidebar_position: 1\n---\n\nimport CallToAction from '@theme/CallToAction';\n\nQuickwit is the first engine to execute complex search and analytics queries directly on cloud storage with sub-second latency. Powered by Rust and its decoupled compute and storage architecture, it is designed to be resource-efficient, easy to operate, and scale to petabytes of data.\n\nQuickwit is a great fit for log management, distributed tracing, and generally immutable data such as conversational data (emails, texts, messaging platforms) and event-based analytics.\n\n\n\n\n## Use cases\n\n- [Log management](../log-management/overview.md)\n- [Distributed Tracing](../distributed-tracing/overview.md)\n\n## Key concepts\n\n- [Architecture](architecture.md)\n- [Indexing](concepts/indexing.md)\n- [Querying](concepts/querying.md)\n\n## Reference\n\n- [Configuration](../configuration/index.md)\n- [REST API](../reference/rest-api.md)\n- [CLI](../reference/cli.md)\n" }, { "path": "docs/overview/introduction.md", "content": "---\ntitle: What is Quickwit?\nsidebar_position: 1\n---\n\nQuickwit is the first engine to execute complex search and analytics queries directly on cloud storage with sub-second latency. Powered by Rust and its decoupled compute and storage architecture, it is designed to be resource-efficient, easy to operate, and scale to petabytes of data.\n\nQuickwit is a great fit for log management, distributed tracing, and generally immutable data such as conversational data (emails, texts, messaging platforms) and event-based analytics.\n\n\n## Why Quickwit is different from other search engines?\n\nQuickwit is designed for sub-second search straight from object storage allowing true decoupled compute and storage. And it means a lot for your infrastructure:\n\n- You store once for all your data on cheap, safe and unlimited storage.\n- You scale out your cluster in seconds, no need to move data around.\n- Indexing and search workloads are decoupled, you can scale them independently.\n- Your tenants are easily isolated and you can charge them for their usage.\n\nQuickwit is also designed to index and search semi-structured data. Its schemaless indexing allows you to index JSON document with an arbitrary amount of field without heavily impacting your performance. Aggregation are not yet supported but we are working on it, stay tuned!\n\n## When to use Quickwit\n\nQuickwit is a great fit for log management, distributed tracing, and generally immutable data such as conversational data (emails, texts, messaging platforms), event-based analytics, audit logs, security logs, and more.\n\nCheck out our guides to see how you can use Quickwit:\n\n- [Log management](../log-management/overview.md)\n- [Distributed Tracing](../distributed-tracing/overview.md)\n\n\n## Key features\n\n- Full-text search and aggregation queries\n- Elasticsearch query language support\n- Sub-second search on cloud storage (Amazon S3, Azure Blob Storage, …)\n- Decoupled compute and storage, stateless indexers & searchers\n- [Schemaless](https://quickwit.io/docs/guides/schemaless) or strict schema indexing\n- Schemaless analytics\n- [Grafana data source](https://github.com/quickwit-oss/quickwit-datasource)\n- [Jaeger-native](https://quickwit.io/docs/distributed-tracing/plug-quickwit-to-jaeger)\n- OTEL-native for [logs](https://quickwit.io/docs/log-management/overview) and [traces](https://quickwit.io/docs/distributed-tracing/overview)\n- Kubernetes ready - See our [helm-chart](https://quickwit.io/docs/deployment/kubernetes)\n- RESTful API\n\n### Enterprise-grade features\n\n- Multiple [data sources](../ingest-data/index.md) Kafka / Kinesis / Pulsar native\n- Multi-tenancy: indexing with many indexes and partitioning\n- Retention policies\n- Delete tasks (for GRPR use cases)\n- Distributed and highly available* engine that scales out in seconds (HA indexing only with Kafka)\n\n## When not to use Quickwit\n\nUse cases where you would likely *not* want to use Quickwit include:\n\n- You need a low-latency search for e-commerce websites.\n- Your data is mutable.\n\n## Time to discover Quickwit\n\n- [Quickstart](../get-started/quickstart.md)\n- [Concepts](architecture.md)\n- [Last release blog post](https://quickwit.io/blog/quickwit-0.7)\n" }, { "path": "docs/reference/_category_.yaml", "content": "label: 'Reference'\nposition: 11\ncollapsed: true\n" }, { "path": "docs/reference/aggregation.md", "content": "---\ntitle: Aggregations API\nsidebar_position: 30\n---\n\nAn aggregation summarizes your data as statistics on buckets or metrics.\n\nAggregations can provide answers to questions like:\n\n- What is the average price of all sold articles?\n- How many errors with status code 500 do we have per day?\n- What is the average listing price of cars grouped by color?\n\nThere are two categories: [Metrics](#metric-aggregations) and [Buckets](#bucket-aggregations).\n\n#### Prerequisite\n\nTo be able to use aggregations on a field, the field needs to have a fast field index created. A fast field index is a columnar storage,\nwhere documents values are extracted and stored.\n\nExample to create a fast field on text for term aggregations.\n```yaml\nname: category\ntype: text\ntokenizer: raw\nrecord: basic\nfast: true\n```\n\nSee the [index configuration](../configuration/index-config.md) page for more details and examples.\n\n#### API Endpoint\n\nThe endpoints for aggregations are the search endpoints:\n- Quickwit API: `api/v1//search`\n- Elasticsearch API: `api/v1/_elastic//_search`.\n\n#### Format\n\nThe aggregation request and result de/serialize into elasticsearch compatible JSON.\nIf not documented otherwise you should be able to drop in your elasticsearch aggregation queries.\n\nIn some examples below is not the full request shown, but only the payload for `aggregations`.\n\n#### Example\n\nRequest\n```json skip\n{\n \"query\": \"*\",\n \"max_hits\": 0,\n \"aggs\": {\n \"sites_and_aqi\": {\n \"terms\": {\n \"field\": \"County\",\n \"size\": 2,\n \"order\": { \"average_aqi\": \"asc\" }\n },\n \"aggs\": {\n \"average_aqi\": {\n \"avg\": { \"field\": \"AQI\" }\n }\n }\n }\n }\n}\n```\n\n\nResponse\n```json\n...\n\"aggregations\": {\n \"sites_and_aqi\": {\n \"buckets\": [\n {\n \"average_aqi\": {\n \"value\": 32.62267569707098\n },\n \"doc_count\": 56845,\n \"key\": \"臺東縣\"\n },\n {\n \"average_aqi\": {\n \"value\": 35.97893635571055\n },\n \"doc_count\": 28675,\n \"key\": \"花蓮縣\"\n }\n ],\n \"sum_other_doc_count\": 1872055\n }\n}\n```\n\n### Supported Aggregations\n\n - Bucket\n - [Histogram](#histogram)\n - [DateHistogram](#date-histogram)\n - [Range](#range)\n - [Terms](#terms)\n- Metric\n - [Average](#average)\n - [Count](#count)\n - [Max](#max)\n - [Min](#min)\n - [Stats](#stats)\n - [Sum](#sum)\n - [Percentiles](#percentiles)\n - [Cardinality](#cardinality)\n\n\n## Bucket Aggregations\n\nBucketAggregations create buckets of documents. Each bucket is associated with a rule which determines whether or not a document falls into it.\nIn other words, the buckets effectively define document sets. Buckets are not necessarily disjunct, therefore a document can fall into multiple buckets.\nIn addition to the buckets themselves, the bucket aggregations also compute and return the number of documents for each bucket.\nBucket aggregations, as opposed to metric aggregations, can hold sub-aggregations.\nThese sub-aggregations will be aggregated for the buckets created by their “parent” bucket aggregation.\nThere are different bucket aggregators, each with a different “bucketing” strategy.\nSome define a single bucket, some define a fixed number of multiple buckets, and others dynamically create the buckets during the aggregation process.\n\n\n### Histogram\n\nA histogram is a type of bucket aggregation where documents are grouped into buckets based on a fixed interval. Each document's value is \"rounded down\" to the nearest bucket boundary.\n\nE.g. if we have a price 18 and an interval of 5, the document will fall into the bucket with the key 15. The formula used for this is: `((val - offset) / interval).floor() * interval + offset`.\n\n#### Histogram on datetime fields\n\nSee [`DateHistogram`](#date-histogram) for more convenient API for `datetime` fields.\n\nFields of type `datetime` are handled the same way as any numeric field. However, all values in the requests such as intervals, offsets, bounds, and range boundaries need to be expressed in milliseconds.\n\nHistogram with one bucket per day on a `datetime` field. `interval` needs to be provided in milliseconds.\nIn the following example, we grouped documents per day (`1 day = 86400000 milliseconds`).\nThe returned format is currently fixed at `RFC3339`.\n\n##### Request\n```json skip\n{\n \"query\": \"*\",\n \"max_hits\": 0,\n \"aggs\": {\n \"count_per_day\":{\n \"histogram\":{\n \"field\": \"datetime\",\n \"interval\": 86400000\n }\n }\n }\n}\n```\n##### Response\n\n```json skip\n{\n ...\n \"aggregations\": {\n \"count_per_day\": {\n \"buckets\": [\n {\n \"doc_count\": 1,\n \"key\": 1546300800000000.0,\n \"key_as_string\": \"2019-01-01T00:00:00Z\"\n },\n {\n \"doc_count\": 2,\n \"key\": 1546560000000000.0,\n \"key_as_string\": \"2019-01-04T00:00:00Z\"\n }\n ]\n }\n }\n}\n```\n\n\n#### Returned Buckets\n\nBy default buckets are returned between the min and max value of the documents, including empty buckets. Setting `min_doc_count > 0` will filter empty buckets.\n\nThe value range of the buckets can be extended via [`extended_bounds`](#extended_bounds) or limit the range via [`hard_bounds`](#hard_bounds).\n\n#### Example\n\n```json\n{\n \"query\": \"*\",\n \"max_hits\": 0,\n \"aggs\": {\n \"prices\": {\n \"histogram\": {\n \"field\": \"price\",\n \"interval\": 10\n }\n }\n }\n}\n```\n\n#### Parameters\n\n###### **field**\n\nThe field to aggregate on.\n\nCurrently this aggregation only works on fast fields of type `u64`, `f64`, `i64`, and `datetime`.\n\n###### **keyed**\n\nChange response format from an array to a hashmap, `key` in the bucket will be the `key` in the hashmap.\n\n###### **interval**\n\nThe interval to chunk your data range. Each bucket spans a value range of [0..interval). Must be larger than 0.\n\n###### **offset**\n\nIntervals implicitly defines an absolute grid of buckets `[interval * k, interval * (k + 1))`.\nOffset makes it possible to shift this grid into `[offset + interval * k, offset + interval (k + 1))`. Offset has to be in the range [0, interval).\n\nAs an example, if there are two documents with value 8 and 12 and interval 10.0, they would fall into the buckets with the key 0 and 10. With offset 5 and interval 10, they would both fall into the bucket with the key 5 and the range [5..15)\n\n```json\n{\n \"query\": \"*\",\n \"max_hits\": 0,\n \"aggs\": {\n \"prices\": {\n \"histogram\": {\n \"field\": \"price\",\n \"interval\": 10,\n \"offset\": 2.5\n }\n }\n }\n}\n```\n\n\n###### **min_doc_count**\n\nThe minimum number of documents in a bucket to be returned. Defaults to 0.\n\n###### **hard_bounds**\n\nLimits the data range to [min, max] closed interval.\nThis can be used to filter values if they are not in the data range.\nhard_bounds only limits the buckets, to force a range set both `extended_bounds` and `hard_bounds` to the same range.\n\n```json\n{\n \"query\": \"*\",\n \"max_hits\": 0,\n \"aggs\": {\n \"prices\": {\n \"histogram\": {\n \"field\": \"price\",\n \"interval\": 10,\n \"hard_bounds\": {\n \"min\": 0,\n \"max\": 100\n }\n }\n }\n }\n}\n```\n\n###### **extended_bounds**\n\nCan be set to extend your bounds. The range of the buckets is by default defined by the data range of the values of the documents. As the name suggests, this can only be used to extend the value range. If the bounds for min or max are not extending the range, the value has no effect on the returned buckets.\nCannot be set in conjunction with `min_doc_count` > 0, since the empty buckets from extended bounds would not be returned.\n\n```json\n{\n \"query\": \"*\",\n \"max_hits\": 0,\n \"aggs\": {\n \"prices\": {\n \"histogram\": {\n \"field\": \"price\",\n \"interval\": 10,\n \"extended_bounds\": {\n \"min\": 0,\n \"max\": 100\n }\n }\n }\n }\n}\n```\n\n### Date Histogram\n\n`DateHistogram` is similar to `Histogram`, but it can only be used with [datetime type](../configuration/index-config#datetime-type) and provides a more convenient API to define intervals.\n\nLike the histogram, values are rounded down to the closest bucket.\n\nThe returned format is currently fixed at `Rfc3339`.\n\n##### Limitations\nOnly fixed time intervals via the `fixed_interval` parameter are supported.\nThe parameters `interval` and `calendar_interval` are unsupported.\n\n##### Request\n```json skip\n{\n \"query\": \"*\",\n \"max_hits\": 0,\n \"aggs\": {\n \"sales_over_time\": {\n \"date_histogram\": {\n \"field\": \"sold_at\",\n \"fixed_interval\": \"30d\"\n \"offset\": \"-4d\"\n }\n }\n }\n}\n```\n##### Response\n\n```json skip\n{\n ...\n \"aggregations\": {\n \"sales_over_time\" : {\n \"buckets\" : [{\n \"key_as_string\" : \"2015-01-01T00:00:00Z\",\n \"key\" : 1420070400000,\n \"doc_count\" : 4\n }]\n }\n }\n}\n```\n\n\n#### Parameters\n\n###### **field**\n\nThe field to aggregate on.\n\nCurrently this aggregation only works on fast fields of type `datetime`.\n\n###### **keyed**\n\nChange response format from an array to a hashmap, `key` in the bucket will be the `key` in the hashmap.\n\n###### **interval**\n\nThe interval to chunk your data range. Each bucket spans a value range of [0..interval). Must be larger than 0.\n\nFixed intervals are configured with the `fixed_interval` parameter.\nFixed intervals are a fixed number of SI units and\nnever deviate, regardless of where they fall on the calendar. One second is always\ncomposed of 1000ms. This allows fixed intervals to be specified in any multiple of the\nsupported units. However, it means fixed intervals cannot express other units such as\nmonths, since the duration of a month is not a fixed quantity. Attempting to specify a\ncalendar interval like month or quarter will return an Error.\n\nThe accepted units for fixed intervals are:\n* `ms`: milliseconds\n* `s`: seconds. Defined as 1000 milliseconds each.\n* `m`: minutes. Defined as 60 seconds each (60_000 milliseconds).\n* `h`: hours. Defined as 60 minutes each (3_600_000 milliseconds).\n* `d`: days. Defined as 24 hours (86_400_000 milliseconds).\n\nFractional time values are not supported, but this can be addressed by shifting to another\ntime unit (e.g., `1.5h` could instead be specified as `90m`).\n\n###### **offset**\n\nIntervals implicitly define an absolute grid of buckets `[interval * k, interval * (k + 1))`.\nOffset makes it possible to shift this grid into `[offset + interval * k, offset + interval (k + 1))`. Offset has to be in the range [0, interval).\n\nThis is especially useful when using `fixed_interval`, to shift the first bucket e.g. at the start of the year.\n\nThe `offset` parameter has the same syntax as the `fixed_interval` parameter, but also allows for negative values.\n\n###### **min_doc_count**\n\nThe minimum number of documents in a bucket to be returned. Defaults to 0.\n\n###### **hard_bounds**\nSame as in [`Histogram`](#hard_bounds) but `min` and `max` parameters need to be set as timestamp with milliseconds precision.\n\n###### **extended_bounds**\nSame as in [`Histogram`](#extended_bounds) but `min` and `max` parameters need to be set as timestamp with milliseconds precision.\n\n### Range\n\nProvide user-defined buckets to aggregate on. Two special buckets will automatically be created to cover the whole range of values.\nThe provided buckets have to be continuous. During the aggregation, the values extracted from the fast_field field will be checked against each bucket range.\nNote that this aggregation includes the from value and excludes the to value for each range.\n\n#### Limitations/Compatibility\n\nOverlapping ranges are not yet supported.\n\n##### Request\n```json skip\n{\n \"query\": \"*\",\n \"max_hits\": 0,\n \"aggs\": {\n \"my_scores\": {\n \"range\": {\n \"field\": \"score\",\n \"ranges\": [\n { \"to\": 3.0, \"key\": \"low\" },\n { \"from\": 3.0, \"to\": 7.0, \"key\": \"medium-low\" },\n { \"from\": 7.0, \"to\": 20.0, \"key\": \"medium-high\" },\n { \"from\": 20.0, \"key\": \"high\" }\n ]\n }\n }\n }\n}\n```\n\n##### Response\n\n```json skip\n{\n ...\n \"aggregations\": {\n \"my_scores\" : {\n \"buckets\": [\n {\"key\": \"low\", \"doc_count\": 0, \"to\": 3.0},\n {\"key\": \"medium-low\", \"doc_count\": 10, \"from\": 3.0, \"to\": 7.0},\n {\"key\": \"medium-high\", \"doc_count\": 10, \"from\": 7.0, \"to\": 20.0},\n {\"key\": \"high\", \"doc_count\": 80, \"from\": 20.0}\n ]\n }\n }\n}\n```\n\n#### Parameters\n\n###### **keyed**\n\nChange response format from an array to a hashmap, the serialized range will be the `key` in the hashmap.\nIf a custom `key` is provided, it will be used instead.\n\n###### **field**\n\nThe field to aggregate on.\n\nCurrently this aggregation only works on fast fields of type `u64`, `f64`, `i64`, and `datetime`.\n\n###### **ranges**\n\nThe list of buckets, with `from` and `to` values.\nThe `from` value is inclusive in the range.\nThe `to` value is not inclusive in the range.\n`key` is optional, and will be used as the bucket key in the response.\n\nThe first bucket can omit the `from` value, and the last bucket the `to` value.\nNote that this aggregation includes the `from` value and excludes the `to` value for each range. Extra buckets will be created until the first `to`, and last `from`, if necessary.\n\n### Terms\n\nCreates a bucket for every unique term and counts the number of occurrences.\n\nRequest\n```json skip\n{\n \"query\": \"*\",\n \"max_hits\": 0,\n \"aggs\": {\n \"genres\": {\n \"terms\": { \"field\": \"genre\" }\n }\n }\n}\n```\n\nResponse\n```json\n...\n\"aggregations\": {\n \"genres\": {\n \"doc_count_error_upper_bound\": 0,\n \"sum_other_doc_count\": 0,\n \"buckets\": [\n { \"key\": \"drumnbass\", \"doc_count\": 6 },\n { \"key\": \"raggae\", \"doc_count\": 4 },\n { \"key\": \"jazz\", \"doc_count\": 2 }\n ]\n }\n}\n```\n\n\n#### Document count error\nIn Quickwit, we have one segment per split. Therefore the results returned from a split, is equivalent to results returned from a segment.\nTo improve performance, results from one split are cut off at `shard_size`.\nWhen combining results of multiple splits, terms that\ndon't make it in the top n of a result from a split increase the theoretical upper bound error by lowest\nterm-count.\n\nEven with a larger `shard_size` value, doc_count values for a terms aggregation may be\napproximate. As a result, any sub-aggregations on the terms aggregation may also be approximate.\n`sum_other_doc_count` is the number of documents that didn’t make it into the top size\nterms. If this is greater than 0, the terms agg had to throw away some\nbuckets, either because they didn’t fit into `size` on the root node or they didn’t fit into\n`shard_size` on the leaf node.\n\n#### Per bucket document count error\nIf you set the `show_term_doc_count_error` parameter to true, the terms aggregation will include\ndoc_count_error_upper_bound, which is an upper bound to the error on the doc_count returned by\neach split. It’s the sum of the size of the largest bucket on each split that didn’t fit\ninto `shard_size`.\n\n#### Parameters\n\n###### **field**\n\nThe field to aggregate on.\n\nCurrently term aggregation only works on fast fields of type `text`, `f64`, `i64` and `u64`.\n\n###### **size**\n\nBy default, the top 10 terms with the most documents are returned. Larger values for size are more expensive.\n\n###### **shard_size**\n\nTo obtain more accurate results, we fetch more than the `size` from each segment/split.\n\nIncreasing this value will enhance accuracy but will also increase CPU/memory usage. \nRefer to the [`document count error`](#document-count-error) section for more information on how `shard_size` impacts accuracy.\n\n`shard_size` represents the number of terms that are returned from one split. \nFor example, if there are 100 splits and `shard_size` is set to 1000, the root node may receive up to 100_000 terms to merge. \nAssuming an average cost of 50 bytes per term, this would require up to 5MB of memory. \nThe actual number of terms sent to the root depends on the number of splits handled by one node and how the intermediate results can be merged (e.g., the cardinality of the terms).\n\nNote on differences between Quickwit and Elasticsearch:\n* Unlike Elasticsearch, Quickwit does not use global ordinals, so serialized terms need to be sent to the root node.\n* The concept of shards in Elasticsearch differs from splits in Quickwit. In Elasticsearch, a shard contains up to 200M documents and is a collection of segments. In contrast, a Quickwit split comprises a single segment, typically with 5M documents. Therefore, `shard_size` in Elasticsearch applies to a group of segments, whereas in Quickwit, it applies to a single segment.\n\nDefaults to `size * 10`.\n\n###### **show_term_doc_count_error**\n\nIf you set the show_term_doc_count_error parameter to true, the terms aggregation will include doc_count_error_upper_bound, which is an upper bound to the error on the doc_count returned by each split.\nIt’s the sum of the size of the largest bucket on each split that didn’t fit into `shard_size`.\n\nDefaults to true when ordering by count desc.\n\n\n###### **min_doc_count**\n\nFilter all terms that are lower than `min_doc_count`. Defaults to 1.\n\n_Expensive_ : When set to 0, this will return all terms in the field.\n\n###### **missing**\n\nThe `missing` parameter defines how documents that are missing a value should be treated.\nBy default they will be ignored but it is also possible to treat them as if they had a value.\n```json skip\n{ \"field\": \"genre\", \"missing\": \"NO_DATA\" }\n```\n\n###### **order**\n\nSet the order. String is here a target, which is either “_count”, “_key”, or the name of a metric sub_aggregation.\nSingle value metrics like average can be addressed by its name. Multi value metrics like stats are required to address their field by name e.g. “stats.avg”.\n_Limitation_ : Ordering is only supported by one property currently. Passing an array for `order` is _not_ supported `\"order\": [{ \"average_price\": \"asc\" }, { \"_key\": \"asc\" }]`.\n\nOrder alphabetically\n```json skip\n{\n \"query\": \"*\",\n \"max_hits\": 0,\n \"aggs\": {\n \"genres\": {\n \"terms\": {\n \"field\": \"genre\",\n \"order\": { \"_key\": \"asc\" }\n }\n }\n }\n}\n```\n\n\nOrder by sub_aggregation\n\n```json skip\n{\n \"query\": \"*\",\n \"max_hits\": 0,\n \"aggs\": {\n \"articles_by_price\": {\n \"terms\": {\n \"field\": \"article_name\",\n \"order\": { \"average_price\": \"asc\" }\n },\n \"aggs\": {\n \"average_price\": {\n \"avg\": { \"field\": \"price\" }\n }\n }\n }\n }\n}\n```\n\n\n\n## Metric Aggregations\n\nThe aggregations in this family compute metrics based on values extracted from the documents that are being aggregated.\nValues are extracted from the fast field of the document. Some aggregations output a single numeric metric (e.g. Average)\nand are called single-value numeric metrics aggregation, others generate multiple metrics (e.g. Stats) and are called multi-value numeric metrics aggregation.\n\nIn contrast to bucket aggregations, metrics don't allow sub-aggregations, since there is no document set to aggregate on.\n\n### Average\n\nA single-value metric aggregation that computes the average of numeric values that are extracted from the aggregated documents.\nSupported field types are `u64`, `f64`, `i64`, and `datetime`.\n\n**Request**\n```json skip\n{\n \"query\": \"*\",\n \"max_hits\": 0,\n \"aggs\": {\n \"average_price\": {\n \"avg\": { \"field\": \"price\" }\n }\n }\n}\n```\n\n**Response**\n```json\n{\n \"num_hits\": 9582098,\n \"hits\": [],\n \"elapsed_time_micros\": 101942,\n \"errors\": [],\n \"aggregations\": {\n \"average_price\": {\n \"value\": 133.7\n }\n }\n}\n```\n\n#### Parameters\n\n###### **missing**\nThe `missing` parameter defines how documents that are missing a value should be treated.\nBy default they will be ignored but it is also possible to treat them as if they had a value.\n```json skip\n{ \"field\": \"price\", \"missing\": \"10.0\" }\n```\n\n### Count\n\nA single-value metric aggregation that counts the number of values that are extracted from the aggregated documents.\nSupported field types are `u64`, `f64`, `i64`, and `datetime`.\n\n**Request**\n```json skip\n{\n \"query\": \"*\",\n \"max_hits\": 0,\n \"aggs\": {\n \"price_count\": {\n \"value_count\": { \"field\": \"price\" }\n }\n }\n}\n```\n\n**Response**\n```json\n{\n \"num_hits\": 9582098,\n \"hits\": [],\n \"elapsed_time_micros\": 102956,\n \"errors\": [],\n \"aggregations\": {\n \"price_count\": {\n \"value\": 9582098\n }\n }\n}\n```\n#### Parameters\n\n###### **missing**\nThe `missing` parameter defines how documents that are missing a value should be treated.\nBy default they will be ignored but it is also possible to treat them as if they had a value.\n```json skip\n{ \"field\": \"price\", \"missing\": \"10.0\" }\n```\n\n### Max\n\nA single-value metric aggregation that computes the maximum of numeric values that are that are extracted from the aggregated documents.\nSupported field types are `u64`, `f64`, `i64`, and `datetime`.\n\n**Request**\n```json skip\n{\n \"query\": \"*\",\n \"max_hits\": 0,\n \"aggs\": {\n \"max_price\": {\n \"max\": { \"field\": \"price\" }\n }\n }\n}\n```\n\n**Response**\n```json\n{\n \"num_hits\": 9582098,\n \"hits\": [],\n \"elapsed_time_micros\": 101543,\n \"errors\": [],\n \"aggregations\": {\n \"max_price\": {\n \"value\": 1353.23\n }\n }\n}\n```\n#### Parameters\n\n###### **missing**\nThe `missing` parameter defines how documents that are missing a value should be treated.\nBy default they will be ignored but it is also possible to treat them as if they had a value.\n```json skip\n{ \"field\": \"price\", \"missing\": \"10.0\" }\n```\n\n### Min\n\nA single-value metric aggregation that computes the minimum of numeric values that are that are extracted from the aggregated documents.\nSupported field types are `u64`, `f64`, `i64`, and `datetime`.\n\n**Request**\n```json skip\n{\n \"query\": \"*\",\n \"max_hits\": 0,\n \"aggs\": {\n \"min_price\": {\n \"min\": { \"field\": \"price\" }\n }\n }\n}\n```\n\n**Response**\n```json\n{\n \"num_hits\": 9582098,\n \"hits\": [],\n \"elapsed_time_micros\": 102342,\n \"errors\": [],\n \"aggregations\": {\n \"min_price\": {\n \"value\": 0.01\n }\n }\n}\n```\n#### Parameters\n\n###### **missing**\nThe `missing` parameter defines how documents that are missing a value should be treated.\nBy default they will be ignored but it is also possible to treat them as if they had a value.\n```json skip\n{ \"field\": \"price\", \"missing\": \"10.0\" }\n```\n\n### Stats\n\nA multi-value metric aggregation that computes stats (average, count, min, max, standard deviation, and sum) of numeric values that are extracted from the aggregated documents.\nSupported field types are `u64`, `f64`, `i64`, and `datetime`.\n\n**Request**\n```json skip\n{\n \"query\": \"*\",\n \"max_hits\": 0,\n \"aggs\": {\n \"timestamp_stats\": {\n \"stats\": { \"field\": \"timestamp\" }\n }\n }\n}\n```\n\n\n\n**Response**\n```json\n{\n \"num_hits\": 10000783,\n \"hits\": [],\n \"elapsed_time_micros\": 65297,\n \"errors\": [],\n \"aggregations\": {\n \"timestamp_stats\": {\n \"avg\": 1462320207.9803998,\n \"count\": 10000783,\n \"max\": 1475669670.0,\n \"min\": 1440670432.0,\n \"standard_deviation\": 11867304.28681695,\n \"sum\": 1.4624347076526848e16\n }\n }\n}\n```\n#### Parameters\n\n###### **missing**\nThe `missing` parameter defines how documents that are missing a value should be treated.\nBy default they will be ignored but it is also possible to treat them as if they had a value.\n```json skip\n{ \"field\": \"price\", \"missing\": \"10.0\" }\n```\n\n### Extended Stats\n\nExtended stats is the same as `stats`, but with following additional metrics: `sum_of_squares`, `variance`, `std_deviation`, and `std_deviation_bounds`.\nSupported field types are `u64`, `f64`, `i64`, and `datetime`.\n\n**Request**\n```json\n{\n \"query\": \"*\",\n \"max_hits\": 0,\n \"aggs\": {\n \"response_extended_stats\": {\n \"extended_stats\": { \"field\": \"response\" }\n }\n }\n}\n```\n\n**Response**\n```json\n{\n ..\n \"aggregations\": {\n \"response_extended_stats\": {\n \"avg\": 65.55555555555556,\n \"count\": 9,\n \"max\": 130.0,\n \"min\": 20.0,\n \"std_deviation\": 42.97573245736381,\n \"std_deviation_bounds\": {\n \"lower\": -20.395909359172062,\n \"lower_population\": -20.395909359172062,\n \"lower_sampling\": -25.60973998562673,\n \"upper\": 151.50702047028318,\n \"upper_population\": 151.50702047028318,\n \"upper_sampling\": 156.72085109673785\n },\n \"std_deviation_population\": 42.97573245736381,\n \"std_deviation_sampling\": 45.582647770591144,\n \"sum\": 590.0,\n \"sum_of_squares\": 55300.0,\n \"variance\": 1846.9135802469136,\n \"variance_population\": 1846.9135802469136,\n \"variance_sampling\": 2077.777777777778\n }\n }\n}\n```\n\n#### Parameters\n\n###### **missing**\nThe `missing` parameter defines how documents that are missing a value should be treated.\nBy default they will be ignored but it is also possible to treat them as if they had a value.\n```json skip\n{ \"field\": \"price\", \"missing\": \"10.0\" }\n```\n\n###### **sigma**\n\nThe sigma parameter controls how many standard deviations +/- from the mean should be displayed.\nThe default value is 2.\n```json skip\n{ \"field\": \"price\", \"sigma\": \"3.0\" }\n```\n\n### Sum\n\nA single-value metric aggregation that sums up numeric values that are that are extracted from the aggregated documents.\nSupported field types are `u64`, `f64`, `i64`, and `datetime`.\n\n**Request**\n```json skip\n{\n \"query\": \"*\",\n \"max_hits\": 0,\n \"aggs\": {\n \"total_price\": {\n \"sum\": { \"field\": \"price\" }\n }\n }\n}\n```\n\n**Response**\n```json\n{\n \"num_hits\": 9582098,\n \"hits\": [],\n \"elapsed_time_micros\": 101142,\n \"errors\": [],\n \"aggregations\": {\n \"total_price\": {\n \"value\": 12966782476.54\n }\n }\n}\n```\n\n#### Parameters\n\n###### **missing**\nThe `missing` parameter defines how documents that are missing a value should be treated.\nBy default they will be ignored but it is also possible to treat them as if they had a value.\n```json skip\n{ \"field\": \"price\", \"missing\": \"10.0\" }\n```\n\n\n\n### Percentiles\nThe percentiles aggregation is a useful tool for understanding the distribution of a data set.\nIt calculates the values below which a given percentage of the data falls.\nFor instance, the 95th percentile indicates the value below which 95% of the data points can be found.\n\nThis aggregation can be particularly interesting for analyzing website or service response times.\nFor example, if the 95th percentile website load time is significantly higher than the median, this indicates\nthat a small percentage of users are experiencing much slower load times than the majority.\n\nTo use the percentiles aggregation, you'll need to provide a field to aggregate on.\nIn the case of website load times, this would typically be a field containing the duration of time it takes for the site to load.\n\n**Request**\n```json skip\n{\n \"query\": \"*\",\n \"max_hits\": 0,\n \"aggs\": {\n \"loading_times\": {\n \"percentiles\": {\n \"field\": \"load_time\"\n \"percents\": [90, 95, 99]\n }\n }\n }\n}\n```\n\n**Response**\n```JSON\n{\n \"num_hits\": 9582098,\n \"hits\": [],\n \"elapsed_time_micros\": 101142,\n \"errors\": [],\n \"aggregations\": {\n \"loading_times\": {\n \"values\": {\n \"90.0\": 33.4,\n \"95.0\": 83.4,\n \"99.0\": 230.3\n }\n }\n }\n}\n```\n\n`percents` may be omitted, it will default to `[1, 5, 25, 50 (median), 75, 95, and 99]`.\n\n#### Estimating Percentiles\n\nWhile percentiles provide valuable insights into the distribution of data, it's important to understand that they are often estimates.\nThis is because calculating exact percentiles for large data sets can be computationally expensive and time-consuming.\n\n#### Parameters\n\n###### **missing**\nThe `missing` parameter defines how documents that are missing a value should be treated.\nBy default they will be ignored but it is also possible to treat them as if they had a value.\n```json skip\n{ \"field\": \"price\", \"missing\": \"10.0\" }\n```\n\n\n### Cardinality\nThe cardinality aggregation is used to approximate the count of distinct values in a field. \nCardinality aggregations are essential when working with large datasets where computing the exact count of distinct values would be computationally expensive. \n\nThe cardinality aggregation can be useful to e.g. to count the number of unique users visiting a website or to determine the number of unique IP addresses that have logged into a server over a certain period.\n\nThe algorithm behind the cardinality aggregation is based on HyperLogLog++, which provides an approximate count over the hashed values.\n\nTo use the cardinality aggregation, you need to specify the field on which to perform the aggregation.\n\n**Request**\n```json\n{\n \"query\": \"*\",\n \"max_hits\": 0,\n \"aggs\": {\n \"unique_users\": {\n \"cardinality\": {\n \"field\": \"user_id\"\n }\n }\n }\n}\n```\n\n**Response**\n```json\n{\n \"num_hits\": 9582098,\n \"hits\": [],\n \"elapsed_time_micros\": 101142,\n \"errors\": [],\n \"aggregations\": {\n \"unique_users\": {\n \"value\": 345672\n }\n }\n}\n```\n\n\n#### Parameters\n\n###### **missing**\nThe `missing` parameter defines how documents that are missing a value should be treated.\nBy default they will be ignored but it is also possible to treat them as if they had a value.\n```json skip\n{ \"field\": \"price\", \"missing\": \"10.0\" }\n```\n\n#### Performance\n\nThe cardinality aggregation on text fields is computationally expensive for datasets with a large amount of unique values. \nThis is because the aggregation computes the hash for each unique term in the field. \nIn order to do this, Quickwit will for each split first collect the term ids and then fetch the compressed terms for those term ids from the dictionary.\nDecompressing the terms is comparatively expensive and keeping the term ids increases the memory usage.\n\nFor numeric fields, the cardinality aggregation is much more efficient as it directly computes the hash of the numeric values and adds them to HLL++.\n\n##### Limitations\nThe parameter `precision_threshold` is ignored currently. Normally it allows to set the threshold until the aggregation is exact.\n\n\n" }, { "path": "docs/reference/cli.md", "content": "---\ntitle: Command-line options\nsidebar_position: 50\n---\n\nQuickwit command line tool lets you start a Quickwit server and manage indexes (create, delete, ingest), splits and sources (create, delete, toggle). To start a server, `quickwit` needs a [node config file path](../configuration/node-config.md) that you can specify with `QW_CONFIG` environment variable: `export QW_CONFIG=./config/quickwit.yaml`.\n\nThis page documents all the available commands, related options, and environment variables.\n\n### Common Options\n\nTo manage indexes, splits and sources on a remote cluster you might need to specify the connection to a Quickwit node. The following options are supported:\n\n| Option | Description | Default |\n|---------------------|-----------------------------|------------------------:|\n| `--endpoint` | The url of a Quickwit node. | `http://127.0.0.1:7280` |\n| `--timeout` | Command timeout. | *See below* |\n| `--connect-timeout` | Connect timeout. | `5s` |\n\nThe default timeouts are command specific:\n- **search** - 1 minute\n- **ingest** (without force or wait) - 1 minute\n- **ingest** (with force or wait) - 30 minute\n- all other operations - 10 seconds\n\nThe timeout can be expressed as in seconds, minutes, hours or days. For example:\n\n- `10s` - 10 seconds timeout\n- `1m` - 1 minute timeout\n- `2h` - 2 hours timeout\n- `1d` - 1 day timeout\n- `none` - no timeout is applied.\n\n:::caution\n\nBefore using Quickwit with object storage, consult our [guidelines](../operating/aws-costs.md) for deploying on AWS S3 to avoid surprises on your next bill.\n\n:::\n\n\n## Commands\n\n[Command-line synopsis syntax](https://developers.google.com/style/code-syntax)\n\n### Help\n\n`quickwit` or `quickwit --help` displays the list of available commands.\n\n`quickwit --help` displays the documentation for the command and a usage example.\n\n### Version\n\n`quickwit --version` displays the version. It is helpful for reporting bugs.\n\n\n### Syntax\n\nThe CLI is structured into high-level commands with subcommands.\n`quickwit [command] [subcommand] [args]`.\n\n* `command`: `run`, `index`, `split`, `source` and `tool`.\n\n\n\n## run\nStarts a Quickwit node with all services enabled by default: `indexer`, `searcher`, `metastore`, `control-plane`, and `janitor`.\n\n\n### Indexer service\n\nThe indexer service runs indexing pipelines assigned by the control plane.\n\n### Searcher service \nStarts a web server at `rest_listing_address:rest_list_port` that exposes the [Quickwit REST API](rest-api.md)\nwhere `rest_listing_address` and `rest_list_port` are defined in Quickwit config file (quickwit.yaml).\nThe node can optionally join a cluster using the `peer_seeds` parameter.\nThis list of node addresses is used to discover the remaining peer nodes in the cluster through a gossip protocol, see [chitchat](https://github.com/quickwit-oss/chitchat).\n\n### Metastore service\n\nThe metastore service exposes Quickwit metastore over the network. This is a core internal service that is needed to operate Quickwit. As such, at least one running instance of this service is required for other services to work.\n\n### Control plane service\n\nThe control plane service schedules indexing tasks to indexers. It listens to metastore events such as\nan source create, delete, toggle, or index delete and reacts accordingly to update the indexing plan.\n\n### Janitor service\n\nThe Janitor service runs maintenance tasks on indexes: garbage collection, documents delete, and retention policy tasks.\n\n:::note\nQuickwit needs to open the following port for cluster formation and workload distribution:\n\n TCP port (default is 7280) for REST API\n TCP and UDP port (default is 7280) for cluster membership protocol\n TCP port + 1 (default is 7281) for gRPC address for the distributed search\n\nIf ports are already taken, the serve command will fail.\n:::\n \n`quickwit run [args]`\n\n*Synopsis*\n\n```bash\nquickwit run\n [--config ]\n [--service ]\n```\n\n*Options*\n\n| Option | Description | Default |\n|-----------------|-------------|--------:|\n| `--config` | Config file location | `config/quickwit.yaml` |\n| `--service` | Services (`indexer`, `searcher`, `metastore`, `control-plane`, or `janitor`) to run. If unspecified, all the supported services are started. | |\n\n*Examples*\n\n*Starts an indexer and a metastore services*\n```bash\nquickwit run --service indexer --service metastore --endpoint=http://127.0.0.1:7280\n```\n\n*Start a control plane, metastore and janitor services*\n```bash\nquickwit run --service control_plane --service metastore --service janitor --config=./config/quickwit.yaml\n```\n\n*Make a search request on a wikipedia index*\n```bash\n# To create wikipedia index and ingest data, go to our tutorials https://quickwit.io/docs/get-started/.\n# Start a searcher.\nquickwit run --service searcher --service metastore --config=./config/quickwit.yaml\n# Make a request.\ncurl \"http://127.0.0.1:7280/api/v1/wikipedia/search?query=barack+obama\"\n\n```\n\n## index\nManages indexes: creates, updates, deletes, ingests, searches, describes...\n\n### index create\n\nCreates an index of ID `index` at `index-uri` configured by a [YAML config file](../configuration/index-config.md) located at `index-config`.\nThe index config lets you define the mapping of your document on the index and how each field is stored and indexed.\nIf `index-uri` is omitted, `index-uri` will be set to `{default_index_root_uri}/{index}`, more info on [Quickwit config docs](../configuration/node-config.md).\nThe command fails if an index already exists unless `overwrite` is passed.\nWhen `overwrite` is enabled, the command deletes all the files stored at `index-uri` before creating a new index.\n \n`quickwit index create [args]`\n\n*Synopsis*\n\n```bash\nquickwit index create\n --index-config \n [--overwrite]\n```\n\n*Options*\n\n| Option | Description |\n|-----------------|-------------|\n| `--index-config` | Location of the index config file. |\n| `--overwrite` | Overwrites pre-existing index. This will delete all existing data stored at `index-uri` before creating a new index. |\n\n*Examples*\n\n*Create a new index.*\n```bash\n# Start a Quickwit server.\nquickwit run --config=./config/quickwit.yaml\n# Open a new terminal and run:\ncurl -o wikipedia_index_config.yaml https://raw.githubusercontent.com/quickwit-oss/quickwit/main/config/tutorials/wikipedia/index-config.yaml\nquickwit index create --endpoint=http://127.0.0.1:7280 --index-config wikipedia_index_config.yaml\n\n```\n\n### index update\n\nUpdates an index using an index config file. \n`quickwit index update [args]`\n\n*Synopsis*\n\n```bash\nquickwit index update\n --index \n --index-config \n```\n\n*Options*\n\n| Option | Description |\n|-----------------|-------------|\n| `--index` | ID of the target index |\n| `--index-config` | Location of the index config file. |\n| `--create` | Create the index if it doesn't exist. |\n### index clear\n\nClears an index: deletes all splits and resets checkpoint. \n`quickwit index clear [args]`\n`quickwit index clr [args]`\n\n*Synopsis*\n\n```bash\nquickwit index clear\n --index \n```\n\n*Options*\n\n| Option | Description |\n|-----------------|-------------|\n| `--index` | Index ID |\n### index delete\n\nDeletes an index. \n`quickwit index delete [args]`\n`quickwit index del [args]`\n\n*Synopsis*\n\n```bash\nquickwit index delete\n --index \n [--dry-run]\n```\n\n*Options*\n\n| Option | Description |\n|-----------------|-------------|\n| `--index` | ID of the target index |\n| `--dry-run` | Executes the command in dry run mode and only displays the list of splits candidates for deletion. |\n\n*Examples*\n\n*Delete your index*\n```bash\n# Start a Quickwit server.\nquickwit run --service metastore --config=./config/quickwit.yaml\n# Open a new terminal and run:\nquickwit index delete --index wikipedia --endpoint=http://127.0.0.1:7280\n\n```\n\n### index describe\n\nDisplays descriptive statistics of an index. \n`quickwit index describe [args]`\n\n*Synopsis*\n\n```bash\nquickwit index describe\n --index \n```\n\n*Options*\n\n| Option | Description |\n|-----------------|-------------|\n| `--index` | ID of the target index |\n\n*Examples*\n\n*Displays descriptive statistics of your index*\n```bash\n# Start a Quickwit server.\nquickwit run --service metastore --config=./config/quickwit.yaml\n# Open a new terminal and run:\nquickwit index describe --endpoint=http://127.0.0.1:7280 --index wikipedia\n\n1. General infos\n===============================================================================\nIndex id: wikipedia\nIndex uri: file:///home/quickwit-indices/qwdata/indexes/wikipedia\nNumber of published splits: 1\nNumber of published documents: 300000\nSize of published splits: 448 MB\n\n2. Statistics on splits\n===============================================================================\nDocument count stats:\nMean ± σ in [min … max]: 300000 ± 0 in [300000 … 300000]\nQuantiles [1%, 25%, 50%, 75%, 99%]: [300000, 300000, 300000, 300000, 300000]\n\nSize in MB stats:\nMean ± σ in [min … max]: 448 ± 0 in [448 … 448]\nQuantiles [1%, 25%, 50%, 75%, 99%]: [448, 448, 448, 448, 448]\n\n```\n\n### index list\n\nList indexes. \n`quickwit index list [args]`\n`quickwit index ls [args]`\n\n*Examples*\n\n*List indexes*\n```bash\n# Start a Quickwit server.\nquickwit run --config=./config/quickwit.yaml\n# Open a new terminal and run:\nquickwit index list --endpoint=http://127.0.0.1:7280\n# Or with alias.\nquickwit index ls --endpoint=http://127.0.0.1:7280\n\n Indexes \n+-----------+--------------------------------------------------------+\n| Index ID | Index URI |\n+-----------+--------------------------------------------------------+\n| hdfs-logs | file:///home/quickwit-indices/qwdata/indexes/hdfs-logs |\n+-----------+--------------------------------------------------------+\n| wikipedia | file:///home/quickwit-indices/qwdata/indexes/wikipedia |\n+-----------+--------------------------------------------------------+\n\n\n```\n\n### index ingest\n\nIndexes a dataset consisting of newline-delimited JSON objects located at `input-path` or read from *stdin*.\nThe data is appended to the target index of ID `index` unless `overwrite` is passed. `input-path` can be a file or another command output piped into stdin.\nCurrently, only local datasets are supported.\nBy default, Quickwit's indexer will work with a heap of 2 GiB of memory. Learn how to change `heap-size` in the [index config doc page](../configuration/index-config.md).\n \n`quickwit index ingest [args]`\n\n*Synopsis*\n\n```bash\nquickwit index ingest\n --index \n [--input-path ]\n [--batch-size-limit ]\n [--wait]\n [--detailed-response]\n [--force]\n [--commit-timeout ]\n```\n\n*Options*\n\n| Option | Description |\n|-----------------|-------------|\n| `--index` | ID of the target index |\n| `--input-path` | Location of the input file. |\n| `--batch-size-limit` | Size limit of each submitted document batch. |\n| `--wait` | Wait for all documents to be committed and available for search before exiting. Applies only to the last batch, see [#5417](https://github.com/quickwit-oss/quickwit/issues/5417). |\n| `--detailed-response` | Print detailed errors. Enabling might impact performance negatively. |\n| `--force` | Force a commit after the last document is sent, and wait for all documents to be committed and available for search before exiting. Applies only to the last batch, see [#5417](https://github.com/quickwit-oss/quickwit/issues/5417). |\n| `--commit-timeout` | Timeout for ingest operations that require waiting for the final commit (`--wait` or `--force`). This is different from the `commit_timeout_secs` indexing setting, which sets the maximum time before committing splits after their creation. |\n\n*Examples*\n\n*Indexing a dataset from a file*\n```bash\n# Start a Quickwit server.\nquickwit run --config=./config/quickwit.yaml\n# Open a new terminal and run:\ncurl -o wiki-articles-10000.json https://quickwit-datasets-public.s3.amazonaws.com/wiki-articles-10000.json\nquickwit index ingest --endpoint=http://127.0.0.1:7280 --index wikipedia --input-path wiki-articles-10000.json\n\n```\n\n*Indexing a dataset from stdin*\n```bash\n# Start a Quickwit server.\nquickwit run --config=./config/quickwit.yaml\n# Open a new terminal and run:\ncat wiki-articles-10000.json | quickwit index ingest --endpoint=http://127.0.0.1:7280 --index wikipedia\n\n```\n\n### index search\n\nSearches an index with ID `--index` and returns the documents matching the query specified with `--query`.\nMore details on the [query language page](query-language.md).\nThe offset of the first hit returned and the number of hits returned can be set with the `start-offset` and `max-hits` options.\nIt's possible to override the default search fields `search-fields` option to define the list of fields that Quickwit will search into if \nthe user query does not explicitly target a field in the query. Quickwit will return snippets of the matching content when requested via the `snippet-fields` options.\nSearch can also be limited to a time range using the `start-timestamp` and `end-timestamp` options.\nThese timestamp options are useful for boosting query performance when using a time series dataset.\n\n:::warning\nThe `start_timestamp` and `end_timestamp` should be specified in seconds regardless of the timestamp field precision. The timestamp field precision only affects the way it's stored as fast-fields, whereas the document filtering is always performed in seconds.\n:::\n \n`quickwit index search [args]`\n\n*Synopsis*\n\n```bash\nquickwit index search\n --index \n --query \n [--aggregation ]\n [--max-hits ]\n [--start-offset ]\n [--search-fields ]\n [--snippet-fields ]\n [--start-timestamp ]\n [--end-timestamp ]\n [--sort-by-score]\n```\n\n*Options*\n\n| Option | Description | Default |\n|-----------------|-------------|--------:|\n| `--index` | ID of the target index | |\n| `--query` | Query expressed in natural query language ((barack AND obama) OR \"president of united states\"). Learn more on https://quickwit.io/docs/reference/search-language. | |\n| `--aggregation` | JSON serialized aggregation request in tantivy/elasticsearch format. | |\n| `--max-hits` | Maximum number of hits returned. | `20` |\n| `--start-offset` | Offset in the global result set of the first hit returned. | `0` |\n| `--search-fields` | List of fields that Quickwit will search into if the user query does not explicitly target a field in the query. It overrides the default search fields defined in the index config. Space-separated list, e.g. \"field1 field2\". | |\n| `--snippet-fields` | List of fields that Quickwit will return snippet highlight on. Space-separated list, e.g. \"field1 field2\". | |\n| `--start-timestamp` | Filters out documents before that timestamp (time-series indexes only). | |\n| `--end-timestamp` | Filters out documents after that timestamp (time-series indexes only). | |\n| `--sort-by-score` | Sorts documents by their BM25 score. | |\n\n*Examples*\n\n*Searching a index*\n```bash\n# Start a Quickwit server.\nquickwit run --config=./config/quickwit.yaml\n# Open a new terminal and run:\nquickwit index search --endpoint=http://127.0.0.1:7280 --index wikipedia --query \"Barack Obama\"\n# If you have jq installed.\nquickwit index search --endpoint=http://127.0.0.1:7280 --index wikipedia --query \"Barack Obama\" | jq '.hits[].title'\n\n```\n\n*Sorting documents by their BM25 score*\n```bash\n# Start a Quickwit server.\nquickwit run --config=./config/quickwit.yaml\n# Open a new terminal and run:\nquickwit index search --endpoint=http://127.0.0.1:7280 --index wikipedia --query \"obama\" --sort-by-score\n\n```\n\n*Limiting the result set to 50 hits*\n```bash\n# Start a Quickwit server.\nquickwit run --config=./config/quickwit.yaml\n# Open a new terminal and run:\nquickwit index search --endpoint=http://127.0.0.1:7280 --index wikipedia --query \"Barack Obama\" --max-hits 50\n# If you have jq installed.\nquickwit index search --endpoint=http://127.0.0.1:7280 --index wikipedia --query \"Barack Obama\" --max-hits 50 | jq '.num_hits'\n\n```\n\n*Looking for matches in the title only*\n```bash\n# Start a Quickwit server.\nquickwit run --config=./config/quickwit.yaml\n# Open a new terminal and run:\nquickwit index search --endpoint=http://127.0.0.1:7280 --index wikipedia --query \"obama\" --search-fields body\n# If you have jq installed.\nquickwit index search --endpoint=http://127.0.0.1:7280 --index wikipedia --query \"obama\" --search-fields body | jq '.hits[].title'\n\n```\n\n## source\nManages sources: creates, updates, deletes sources...\n\n### source create\n\nAdds a new source to an index. \n`quickwit source create [args]`\n\n*Synopsis*\n\n```bash\nquickwit source create\n --index \n --source-config \n```\n\n*Options*\n\n| Option | Description |\n|-----------------|-------------|\n| `--index` | ID of the target index |\n| `--source-config` | Path to source config file. Please, refer to the documentation for more details. |\n### source update\n\nUpdate an existing source. \n`quickwit source update [args]`\n\n*Synopsis*\n\n```bash\nquickwit source update\n --index \n --source \n --source-config \n```\n\n*Options*\n\n| Option | Description |\n|-----------------|-------------|\n| `--index` | ID of the target index |\n| `--source` | ID of the source |\n| `--source-config` | Path to source config file. Please, refer to the documentation for more details. |\n| `--create` | Create the source if it doesn't exist. |\n### source enable\n\nEnables a source for an index. \n`quickwit source enable [args]`\n\n*Synopsis*\n\n```bash\nquickwit source enable\n --index \n --source \n```\n\n*Options*\n\n| Option | Description |\n|-----------------|-------------|\n| `--index` | ID of the target index |\n| `--source` | ID of the source. |\n### source disable\n\nDisables a source for an index. \n`quickwit source disable [args]`\n\n*Synopsis*\n\n```bash\nquickwit source disable\n --index \n --source \n```\n\n*Options*\n\n| Option | Description |\n|-----------------|-------------|\n| `--index` | ID of the target index |\n| `--source` | ID of the source. |\n### source ingest-api\n\nEnables/disables the ingest API of an index. \n`quickwit source ingest-api [args]`\n\n*Synopsis*\n\n```bash\nquickwit source ingest-api\n --index \n [--enable]\n [--disable]\n```\n\n*Options*\n\n| Option | Description |\n|-----------------|-------------|\n| `--index` | ID of the target index |\n| `--enable` | Enables the ingest API. |\n| `--disable` | Disables the ingest API. |\n### source delete\n\nDeletes a source from an index. \n`quickwit source delete [args]`\n`quickwit source del [args]`\n\n*Synopsis*\n\n```bash\nquickwit source delete\n --index \n --source \n```\n\n*Options*\n\n| Option | Description |\n|-----------------|-------------|\n| `--index` | ID of the target index |\n| `--source` | ID of the source. |\n\n*Examples*\n\n*Delete a `wikipedia-source` source*\n```bash\n# Start a Quickwit server.\nquickwit run --service metastore --config=./config/quickwit.yaml\n# Open a new terminal and run:\nquickwit source delete --endpoint=http://127.0.0.1:7280 --index wikipedia --source wikipedia-source\n\n```\n\n### source describe\n\nDescribes a source. \n`quickwit source describe [args]`\n`quickwit source desc [args]`\n\n*Synopsis*\n\n```bash\nquickwit source describe\n --index \n --source \n```\n\n*Options*\n\n| Option | Description |\n|-----------------|-------------|\n| `--index` | ID of the target index |\n| `--source` | ID of the source. |\n### source list\n\nLists the sources of an index. \n`quickwit source list [args]`\n`quickwit source ls [args]`\n\n*Synopsis*\n\n```bash\nquickwit source list\n --index \n```\n\n*Options*\n\n| Option | Description |\n|-----------------|-------------|\n| `--index` | ID of the target index |\n\n*Examples*\n\n*List `wikipedia` index sources*\n```bash\n# Start a Quickwit server.\nquickwit run --service metastore --config=./config/quickwit.yaml\n# Open a new terminal and run:\nquickwit source list --endpoint=http://127.0.0.1:7280 --index wikipedia\n\n```\n\n### source reset-checkpoint\n\nResets a source checkpoint. \n`quickwit source reset-checkpoint [args]`\n`quickwit source reset [args]`\n\n*Synopsis*\n\n```bash\nquickwit source reset-checkpoint\n --index \n --source \n```\n\n*Options*\n\n| Option | Description |\n|-----------------|-------------|\n| `--index` | Index ID |\n| `--source` | Source ID |\n## split\nManages splits: lists, describes, marks for deletion...\n\n### split list\n\nLists the splits of an index. \n`quickwit split list [args]`\n`quickwit split ls [args]`\n\n*Synopsis*\n\n```bash\nquickwit split list\n --index \n [--offset ]\n [--limit ]\n [--states ]\n [--create-date ]\n [--start-date ]\n [--end-date ]\n [--output-format ]\n```\n\n*Options*\n\n| Option | Description |\n|-----------------|-------------|\n| `--index` | Target index ID |\n| `--offset` | Number of splits to skip. |\n| `--limit` | Maximum number of splits to retrieve. |\n| `--states` | Selects the splits whose states are included in this comma-separated list of states. Possible values are `staged`, `published`, and `marked`. |\n| `--create-date` | Selects the splits whose creation dates are before this date. |\n| `--start-date` | Selects the splits that contain documents after this date (time-series indexes only). |\n| `--end-date` | Selects the splits that contain documents before this date (time-series indexes only). |\n| `--output-format` | Output format. Possible values are `table`, `json`, and `pretty-json`. |\n### split describe\n\nDisplays metadata about a split. \n`quickwit split describe [args]`\n`quickwit split desc [args]`\n\n*Synopsis*\n\n```bash\nquickwit split describe\n --index \n --split \n [--verbose]\n```\n\n*Options*\n\n| Option | Description |\n|-----------------|-------------|\n| `--index` | ID of the target index |\n| `--split` | ID of the target split |\n| `--verbose` | Displays additional metadata about the hotcache. |\n### split mark-for-deletion\n\nMarks one or multiple splits of an index for deletion. \n`quickwit split mark-for-deletion [args]`\n`quickwit split mark [args]`\n\n*Synopsis*\n\n```bash\nquickwit split mark-for-deletion\n --index \n --splits \n [--yes]\n```\n\n*Options*\n\n| Option | Description |\n|-----------------|-------------|\n| `--index` | Target index ID |\n| `--splits` | Comma-separated list of split IDs |\n| `--yes` | Assume \"yes\" as an answer to all prompts and run non-interactively. |\n## tool\nPerforms utility operations. Requires a node config.\n\n### tool local-ingest\n\nIndexes NDJSON documents locally. \n`quickwit tool local-ingest [args]`\n\n*Synopsis*\n\n```bash\nquickwit tool local-ingest\n --index \n [--input-path ]\n [--input-format ]\n [--overwrite]\n [--transform-script ]\n [--keep-cache]\n```\n\n*Options*\n\n| Option | Description | Default |\n|-----------------|-------------|--------:|\n| `--index` | ID of the target index | |\n| `--input-path` | Location of the input file. | |\n| `--input-format` | Format of the input data. | `json` |\n| `--overwrite` | Overwrites pre-existing index. | |\n| `--transform-script` | VRL program to transform docs before ingesting. | |\n| `--keep-cache` | Does not clear local cache directory upon completion. | |\n### tool extract-split\n\nDownloads and extracts a split to a directory. \n`quickwit tool extract-split [args]`\n\n*Synopsis*\n\n```bash\nquickwit tool extract-split\n --index \n --split \n [--target-dir ]\n```\n\n*Options*\n\n| Option | Description |\n|-----------------|-------------|\n| `--index` | ID of the target index |\n| `--split` | ID of the target split |\n| `--target-dir` | Directory to extract the split to. |\n### tool gc\n\nGarbage collects stale staged splits and splits marked for deletion. \n:::note\nIntermediate files are created while executing Quickwit commands.\nThese intermediate files are always cleaned at the end of each successfully executed command.\nHowever, failed or interrupted commands can leave behind intermediate files that need to be removed.\nAlso, note that using a very short grace period (like seconds) can cause the removal of intermediate files being operated on, especially when using Quickwit concurrently on the same index.\nIn practice, you can settle with the default value (1 hour) and only specify a lower value if you really know what you are doing.\n\n:::\n`quickwit tool gc [args]`\n\n*Synopsis*\n\n```bash\nquickwit tool gc\n --index \n [--grace-period ]\n [--dry-run]\n```\n\n*Options*\n\n| Option | Description | Default |\n|-----------------|-------------|--------:|\n| `--index` | ID of the target index | |\n| `--grace-period` | Threshold period after which stale staged splits are garbage collected. | `1h` |\n| `--dry-run` | Executes the command in dry run mode and only displays the list of splits candidates for garbage collection. | |\n\n\n\n## Environment Variables\n\n### QW_CLUSTER_ENDPOINT\n\nSpecifies the address of the cluster to connect to. Management commands `index`, `split` and `source` require the `cluster_endpoint`, which you can set once and for all with the `QW_CLUSTER_ENDPOINT` environment variable.\n\n### QW_CONFIG\n\nSpecifies the path to the [quickwit config](../configuration/node-config.md). Commands `run` and `tools` require the `config`, which you can set once and for all with the `QW_CONFIG` environment variable.\n\n*Example*\n\n`export QW_CONFIG=config/quickwit.yaml`\n\n### QW_DISABLE_TELEMETRY\n\nDisables [telemetry](../telemetry.md) when set to any non-empty value.\n\n*Example*\n\n`QW_DISABLE_TELEMETRY=1 quickwit help`\n\n### QW_POSTGRES_SKIP_MIGRATIONS\n\nDon't run database migrations (but verify that migrations were run successfully before, and no that unknown migration was run).\n\n### QW_POSTGRES_SKIP_MIGRATION_LOCKING\n\nDon't lock the database during migration. This may increase compatibility with alternative databases using the PostgreSQL wire protocol. However, it\nis dangerous to use this if you can't guarantee that only one node will run the migrations.\n\n### RUST_LOG\n\nConfigure quickwit log level.\n\n*Examples*\n\n```\n# run with higher verbosity\nRUST_LOG=debug quickwit run\n# run with log level info, except for indexing related logs\nRUST_LOG=info,quickwit_indexing=debug quickwit run\n```\n" }, { "path": "docs/reference/es_compatible_api.md", "content": "---\ntitle: Elasticsearch compatible API\nsidebar_position: 20\n---\n\n\nIn order to facilitate migrations and integrations with existing tools,\nQuickwit offers an Elasticsearch/Opensearch compatible API.\nThis API is incomplete. This page lists the available features and endpoints.\n\n## Supported endpoints\n\nAll the API endpoints start with the `api/v1/_elastic/` prefix.\n\n### `_bulk`   Batch ingestion endpoint\n\n```\nPOST api/v1/_elastic/_bulk\n```\n```\nPOST api/v1/_elastic//_bulk\n```\n\nThe _bulk ingestion API makes it possible to index a batch of documents, possibly targeting several indices in the same request.\n\n#### Request Body example\n\n```json\n{ \"create\" : { \"_index\" : \"wikipedia\", \"_id\" : \"1\" } }\n{\"url\":\"https://en.wikipedia.org/wiki?id=1\",\"title\":\"foo\",\"body\":\"foo\"}\n{ \"create\" : { \"_index\" : \"wikipedia\", \"_id\" : \"2\" } }\n{\"url\":\"https://en.wikipedia.org/wiki?id=2\",\"title\":\"bar\",\"body\":\"bar\"}\n{ \"create\" : { \"_index\" : \"wikipedia\", \"_id\" : \"3\" } }\n{\"url\":\"https://en.wikipedia.org/wiki?id=3\",\"title\":\"baz\",\"body\":\"baz\"}'\n```\n\nIngest a batch of documents to make them searchable using the [Elasticsearch](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html) bulk API. This endpoint provides compatibility with tools or systems that already send data to Elasticsearch for indexing. Currently, only the `create` action of the bulk API is supported, all other actions such as `delete` or `update` are ignored.\n\nIf an index is specified via the url path, it will act as a default value\nfor the `_index` properties.\n\nThe [`refresh`](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-refresh.html) parameter is supported.\n\n:::caution\nThe quickwit API will not report errors, you need to check the server logs.\n\nIn Elasticsearch, the `create` action has a specific behavior when the ingested documents contain an identifier (the `_id` field). It only inserts such a document if it was not inserted before. This is extremely handy to achieve At-Most-Once indexing.\nQuickwit does not have any notion of document id and does not support this feature.\n:::\n\n:::info\nThe payload size is limited to 10MB as this endpoint is intended to receive documents in batch.\n:::\n\n#### Query parameter\n\n| Variable | Type | Description | Default value |\n| --------- | -------- | ---------------------------------------------------------------- | ------------- |\n| `refresh` | `String` | The commit behavior: blank string, `true`, `wait_for` or `false` | `false` |\n\n#### Response\n\nThe response is a JSON object, and the content type is `application/json; charset=UTF-8.`\n\n| Field | Description | Type |\n| ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | :------: |\n| `num_docs_for_processing` | Total number of documents ingested for processing. The documents may not have been processed. The API will not return indexing errors, check the server logs for errors. | `number` |\n\n\n\n\n### `_search`   Index search endpoint\n\n```\nPOST api/v1/_elastic//_search\n```\n```\nGET api/v1/_elastic//_search\n```\n\n#### Request Body example\n\n```json\n{\n \"size\": 10,\n \"query\": {\n \"bool\": {\n \"must\": [\n {\n \"query_string\": {\n \"query\": \"bitpacking\"\n }\n },\n {\n \"term\": {\n \"actor.login\": {\n \"value\": \"fulmicoton\"\n }\n }\n }\n ]\n }\n },\n \"sort\": [\n {\n \"actor.id\": {\n \"order\": null\n }\n }\n ],\n \"aggs\": {\n \"event_types\": {\n \"terms\": {\n \"field\": \"type\",\n \"size\": 5\n }\n }\n }\n}\n```\n\nSearch into a specific index using the [Elasticsearch search API](https://www.elastic.co/guide/en/elasticsearch/reference/8.8/search-search.html).\n\nSome of the parameter can be passed as query string parameter, and some via JSON payload.\nIf a parameter appears both as a query string parameter and in the JSON payload, the query string parameter value will take priority.\n\n#### Supported Query string parameters\n\n\n| Variable | Type | Description | Default value |\n| ------------------ | ------------- | -------------------------------------------------------------------------------- | ------------- |\n| `default_operator` | `AND` or `OR` | The default operator used to combine search terms. It should be `AND` or `OR`. | `OR` |\n| `from` | `Integer` | The rank of the first hit to return. This is useful for pagination. | 0 |\n| `q` | `String` | The search query. | (Optional) |\n| `size` | `Integer` | Number of hits to return. | 10 |\n| `sort` | `String` | Describes how documents should be ranked. See [Sort order](#sort-order) | (Optional) |\n| `scroll` | `Duration` | Creates a scroll context for \"time to live\". See [Scroll](#_searchscroll--scroll-api). | (Optional) |\n| `allow_partial_search_results` | `Boolean` | Returns a partial response if some (but not all) of the split searches were unsuccessful. | `true` |\n\n#### Supported Request Body parameters\n\n| Variable | Type | Description | Default value |\n| ------------------ | ----------------- | ------------------------------------------------------------------------------ | ------------- |\n| `default_operator` | `\"AND\"` or `\"OR\"` | The default operator used to combine search terms. It should be `AND` or `OR`. | `OR` |\n| `from` | `Integer` | The rank of the first hit to return. This is useful for pagination. | 0 |\n| `query` | `Json object` | Describe the search query. See [Query DSL](#query-dsl) | (Optional) |\n| `size` | `Integer` | Number of hits to return. | 10 |\n| `sort` | `JsonObject[]` | Describes how documents should be ranked. See [Sort order](#sort-order) | `[]` |\n| `search_after` | `Any[]` | Ignore documents with a SortingValue preceding or equal to the parameter | (Optional) |\n| `aggs` | `Json object` | Aggregation definition. See [Aggregations](aggregation.md). | `{}` |\n\n\n#### Sort order\n\nYou can define up to two criteria on which to apply sort.\nThe second criterion will only be used in presence of a tie for the first criterion.\n\nA given criterion can either be\n- the name of a fast field (explicitly defined in the schema or captured by the dynamic mode)\n- `_score` to sort by BM25.\n\nBy default, the sort order is `ascending` for fast fields and descending for `_score`.\n\nWhen sorting by a fast field and this field contains several values in a single document, only the first value is used for sorting.\n\nThe sort order can be set as descending/ascending using the\nfollowing syntax.\n\n```json\n{\n // ...\n \"sort\" : [\n { \"timestamp\" : {\"order\" : \"asc\"}},\n { \"serial_number\" : \"desc\" }\n ]\n // ...\n}\n\n```\n\nIt is also possible to not supply an order and rely on the default order using the following syntax.\n\n```json\n{ //...\n \"sort\" : [\"_score\", \"timestamp\"]\n // ...\n}\n```\n\nIf no format is provided for timestamps, timestamps are returned with milliseconds precision.\n\nIf you need nanosecond precision, you can use the `epoch_nanos_int` format. Beware this means the resulting\nJSON may contain high numbers for which there is loss of precision when using languages where all numbers are\nfloats, such as JavaScript.\n\n```json\n{\n // ...\n \"sort\" : [\n { \"timestamp\" : {\"format\": \"epoch_nanos_int\",\"order\" : \"asc\"}},\n { \"serial_number\" : \"desc\" }\n ]\n // ...\n}\n\n#### Search after\n\nWhen sorting results, the answer looks like the following\n\n```json\n{\n // ...\n \"hits\": {\n // ...\n \"hits\": [\n // ...\n {\n // ...\n \"sort\": [\n 1701962929199\n ]\n }\n ]\n }\n}\n```\n\nYou can pass the `sort` value of the last hit in a subsequent request where other fields are kept unchanged:\n```json\n{\n // keep all fields from the original request\n \"search_after\": [\n 1701962929199\n ]\n}\n```\n\nThis allows you to paginate your results.\n\n### `_msearch`   Multi search API\n\n```\nPOST api/v1/_elastic/_msearch\n```\n\n#### Request Body example\n\n```json\n{\"index\": \"gharchive\" }\n{\"query\" : {\"match\" : { \"author.login\": \"fulmicoton\"}}}\n{\"index\": \"gharchive\"}\n{\"query\" : {\"match_all\" : {}}}\n```\n\n[Multi search endpoint ES API reference](https://www.elastic.co/guide/en/elasticsearch/reference/8.8/search-multi-search.html)\n\nRuns several search requests at once.\n\nThe payload is expected to alternate:\n- a `header` json object, containing the targeted index id.\n- a `search request body` as defined in the [`_search` endpoint section].\n\n\n### `_search/scroll`   Scroll API\n\n```\nGET api/v1/_elastic/_search/scroll\n```\n\n#### Supported Request Body parameters\n\n| Variable | Type | Description | Default value |\n| ----------- | ------------------------------------------- | ----------- | ------------- |\n| `scroll_id` | Scroll id (obtained from a search response) | Required | |\n\n\nThe `_search/scroll` endpoint, in combination with the `_search` API makes it possible to request successive pages of search results.\nFirst, the client needs to call the `search api` with a `scroll` query parameter, and then pass the `scroll_id` returned in the response payload to `_search/scroll` endpoint.\n\nEach subsequent call to the `_search/scroll` endpoint will return a new `scroll_id` pointing to the next page.\n\n:::tip\n\nUsing `_search` and then `_search/scroll` is somewhat similar to using `_search` with the `search_after` parameter, except that it creates a lightweight snapshot view of the dataset during the initial call to `_search`. Further calls to `_search/scroll` only return results from that view, thus ensuring more consistent results.\n\n:::\n\n### `_cat`   Cat API\n\n```\nGET api/v1/_elastic/_cat/indices/\n```\n```\nGET api/v1/_elastic/_cat/indices\n```\n\n#### Supported Query string parameters\n\n| Variable | Type | Description | Default value |\n|----------|------------|--------------------------------------------------------------------------------------------------------|---------------|\n| `format` | `String` | Format for response. Only JSON supported for now. | |\n| `h` | `String[]` | Comma-separated list of column names to display. | (Optional) |\n| `health` | `String` | Filter for health: `green`, `yellow`, or `red`. | (Optional) |\n| `bytes` | `String` | Unit used to display byte values. Unsupported for now. | (Optional) |\n| `s` | `String` | Comma-separated list of column names or column aliases used to sort the response. Unsupported for now. | (Optional) |\n| `v` | `Boolean` | If true, the response includes column headings. Unsupported for now. | (Optional) |\n\nUse the [cat indices API](https://www.elastic.co/guide/en/elasticsearch/reference/current/cat-indices.html) to get the following information for each index in a cluster:\n* Shard count\n* Document count\n* Deleted document count\n* Primary store size\n* Total store size\n\n#### Response\n\nThe response is a JSON object, and the content type is `application/json; charset=UTF-8.`\n\n| Field | Description | Type |\n|------------------|--------------------------------------------------|:--------:|\n| `uuid` | Index uuid | `String` |\n| `index` | Index name | `String` |\n| `health` | Health of the index `green`, `yellow`, or `red`. | `String` |\n| `status` | Status of the index `open`. | `String` |\n| `rep` | Replication factor. | `Number` |\n| `pri` | Number of primary shards | `Number` |\n| `pri.store.size` | Stored size of primary shard. | `String` |\n| `store.size` | Stored size of index. | `String` |\n| `dataset.size` | Indexed data size. | `String` |\n| `docs.count` | Number of records in index. | `Number` |\n| `docs.deleted` | Number of deleted records in index. | `Number` |\n\nExample response:\n\n```json\n[\n {\n \"dataset.size\": \"0b\",\n \"docs.count\": \"0\",\n \"docs.deleted\": \"0\",\n \"health\": \"green\",\n \"index\": \"otel-traces-v0_7\",\n \"pri\": \"1\",\n \"pri.store.size\": \"0b\",\n \"rep\": \"1\",\n \"status\": \"open\",\n \"store.size\": \"0b\",\n \"uuid\": \"otel-traces-v0_7:01HTJC6TQDGM07KBDQZ2KDHW53\"\n },\n {\n \"dataset.size\": \"387.5gb\",\n \"docs.count\": \"224453081\",\n \"docs.deleted\": \"0\",\n \"health\": \"green\",\n \"index\": \"otel-logs-v0_7\",\n \"pri\": \"1\",\n \"pri.store.size\": \"37.5gb\",\n \"rep\": \"1\",\n \"status\": \"open\",\n \"store.size\": \"37.5gb\",\n \"uuid\": \"otel-logs-v0_7:01HTJC6TME1JGXBFERHZ0FJ860\"\n }\n]\n```\n\n[HTTP accept header]: https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html\n\n\n### `_field_caps`   Field capabilities API\n\n```\nGET api/v1/_elastic//_field_caps\n```\n```\nPOST api/v1/_elastic//_field_caps\n```\n```\nGET api/v1/_elastic/_field_caps\n```\n```\nPOST api/v1/_elastic/_field_caps\n```\n\nThe [field capabilities API](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-field-caps.html) returns information about the capabilities of fields among multiple indices.\n\n#### Supported Query string parameters\n\n| Variable | Type | Description | Default value |\n| --------------------- | ---------- | ------------------------------------------------------------------------------ | ------------- |\n| `fields` | `String` | Comma-separated list of fields to retrieve capabilities for. Supports wildcards (`*`). | (Optional) |\n| `allow_no_indices` | `Boolean` | If `true`, missing or closed indices are not an error. | (Optional) |\n| `expand_wildcards` | `String` | Controls what kind of indices that wildcard patterns can match. | (Optional) |\n| `ignore_unavailable` | `Boolean` | If `true`, unavailable indices are ignored. | (Optional) |\n| `start_timestamp` | `Integer` | *(Quickwit-specific)* If set, restricts splits to documents with a timestamp range start >= `start_timestamp` (seconds since epoch). | (Optional) |\n| `end_timestamp` | `Integer` | *(Quickwit-specific)* If set, restricts splits to documents with a timestamp range end < `end_timestamp` (seconds since epoch). | (Optional) |\n\n#### Supported Request Body parameters\n\n| Variable | Type | Description | Default value |\n| ------------------ | ------------- | --------------------------------------------------------------------------- | ------------- |\n| `index_filter` | `Json object` | A query to filter indices. If provided, only fields from indices that can potentially match the filter are returned. See [index_filter](#index_filter). | (Optional) |\n| `runtime_mappings` | `Json object` | Accepted but not supported. | (Optional) |\n\n#### `index_filter`\n\nThe `index_filter` parameter allows you to filter which indices contribute to the field capabilities response. When provided, Quickwit uses the filter query to prune indices (splits) that cannot match the filter, and only returns field capabilities for the remaining ones.\n\nLike Elasticsearch, this is a **best-effort** approach: Quickwit may return field capabilities from indices that do not actually contain any matching documents. In Quickwit, the filtering is limited to the existing split-pruning based on metadata:\n\n- **Time pruning**: Range queries on the timestamp field can eliminate splits whose time range does not overlap with the filter.\n- **Tag pruning**: Term queries on [tag fields](../configuration/index-config.md#tag-fields) can eliminate splits that do not contain the requested tag value.\n\nOther filter types (e.g. full-text queries or term queries on non-tag fields) are accepted but will not prune any splits — all indices will be returned as if no filter was specified. In particular, Quickwit does not check whether terms are present in the term dictionary.\n\n#### Request Body example\n\n```json\n{\n \"index_filter\": {\n \"range\": {\n \"timestamp\": {\n \"gte\": \"2024-01-01T00:00:00Z\",\n \"lt\": \"2024-02-01T00:00:00Z\"\n }\n }\n }\n}\n```\n\n```json\n{\n \"index_filter\": {\n \"term\": {\n \"status\": \"active\"\n }\n }\n}\n```\n\n\n## Query DSL\n\n[Elasticsearch Query DSL reference](https://www.elastic.co/guide/en/elasticsearch/reference/8.8/query-dsl.html).\n\nThe following query types are supported.\n\n### `query_string`\n\n[Elasticsearch reference documentation](https://www.elastic.co/guide/en/elasticsearch/reference/8.8/query-dsl-query-string-query.html)\n\n\n#### Example\n\n```json\n{\n \"query\": {\n \"query_string\": {\n \"query\": \"bitpacking AND author.login:fulmicoton\",\n \"fields\": [\n \"payload.description\"\n ]\n }\n }\n}\n```\n\n#### Supported parameters\n\n| Variable | Type | Description | Default value |\n| ------------------ | --------------------- | --------------------------------------------------------------------------------------------------------------------------- | ------------- |\n| `query` | `String` | Query meant to be parsed. | - |\n| `fields` | `String[]` (Optional) | Default search target fields. | - |\n| `default_operator` | `\"AND\"` or `\"OR\"` | In the absence of boolean operator defines whether terms should be combined as a conjunction (`AND`) or disjunction (`OR`). | `OR` |\n| `boost` | `Number` | Multiplier boost for score computation. | 1.0 |\n| `lenient` | `Boolean` | [See note](#about-the-lenient-argument). | false |\n\n\n### `bool`\n\n[Elasticsearch reference documentation](https://www.elastic.co/guide/en/elasticsearch/reference/8.8/query-dsl-term-query.html)\n\n#### Example\n\n```json\n{\n \"query\": {\n \"bool\": {\n \"must\": [\n {\n \"query_string\": {\n \"query\": \"bitpacking\"\n }\n }\n ],\n \"must_not\": {\n \"term\": {\n \"type\": {\n \"value\": \"CommitEvent\"\n }\n }\n }\n }\n }\n}\n```\n\n#### Supported parameters\n\n| Variable | Type | Description | Default value |\n| ---------- | ------------------------- | ----------------------------------------------------------------- | ------------- |\n| `must` | `JsonObject[]` (Optional) | Sub-queries required to match the document. | [] |\n| `must_not` | `JsonObject[]` (Optional) | Sub-queries required to not match the document. | [] |\n| `should` | `JsonObject[]` (Optional) | Sub-queries that should match the documents. | [] |\n| `filter` | `JsonObject[]` | Like must queries, but the match does not influence the `_score`. | [] |\n| `boost` | `Number` | Multiplier boost for score computation. | 1.0 |\n| `minimum_should_match` | `Number` or `Str` | If present, quickwit will only match documents for which at least `minimum_should_match` should clauses are matching. `2`, `-1`, `\"10%\"` and `\"-10%\"` are supported. | |\n\n### `range`\n\n[Elasticsearch reference documentation](https://www.elastic.co/guide/en/elasticsearch/reference/8.8/query-dsl-range-query.html)\n\n#### Example\n\n```json\n{\n \"query\": {\n \"range\": {\n \"my_date_field\": {\n \"lt\": \"2015-02-01T00:00:13Z\",\n \"gte\": \"2015-02-01T00:00:10Z\"\n }\n }\n }\n}\n\n```\n\n#### Supported parameters\n\n| Variable | Type | Description | Default value |\n| -------- | ------------------------------- | -------------------------------------- | ------------- |\n| `gt` | bool, string, Number (Optional) | Greater than | None |\n| `gte` | bool, string, Number (Optional) | Greater than or equal | None |\n| `lt` | bool, string, Number (Optional) | Less than | None |\n| `lte` | bool, string, Number (Optional) | Less than or equal | None |\n| `boost` | `Number` | Multiplier boost for score computation | 1.0 |\n\n\n### `match`\n\n[Elasticsearch reference documentation](https://www.elastic.co/guide/en/elasticsearch/reference/8.8/query-dsl-match-query.html)\n\n#### Example\n\n```json\n{\n \"query\": {\n \"match\": {\n \"type\": {\n \"query\": \"CommitEvent\",\n \"zero_terms_query\": \"all\"\n }\n }\n }\n}\n```\n\n#### Supported Parameters\n\n| Variable | Type | Description | Default |\n| ------------------ | ----------------- | ------------------------------------------------------------------------------------------------------------------------------ | ------- |\n| `query` | String | Full-text search query. | - |\n| `operator` | `\"AND\"` or `\"OR\"` | Defines whether all terms should be present (`AND`) or if at least one term is sufficient to match (`OR`). | OR |\n| `zero_terms_query` | `all` or `none` | Defines if all (`all`) or no documents (`none`) should be returned if the query does not contain any terms after tokenization. | `none` |\n| `boost` | `Number` | Multiplier boost for score computation | 1.0 |\n| `lenient` | `Boolean` | [See note](#about-the-lenient-argument). | false |\n\n\n\n### `match_phrase`\n\n[Elasticsearch reference documentation](https://www.elastic.co/guide/en/elasticsearch/reference/8.8/query-dsl-match-query-phrase.html)\n\n#### Example\n\n```json\n{\n \"query\": {\n \"match_phrase\": {\n \"title\": \"search keywords\",\n \"analyzer\": \"default\"\n }\n }\n}\n```\n\n\n\n### `match_phrase_prefix`\n\n[Elasticsearch reference documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-match-query-phrase-prefix.html)\n\n#### Example\n\n```json\n{\n \"query\": {\n \"match_phrase_prefix\": {\n \"payload.commits.message\": {\n \"query\": \"automated comm\" // This will match \"automated commit\" for instance.\n }\n }\n }\n}\n```\n\n#### Supported Parameters\n\n| Variable | Type | Description | Default |\n| ------------------ | --------------- | ------------------------------------------------------------------------------------------------------------------------------ | --------------------------- |\n| `query` | String | Full-text search query. The last token will be prefix-matched | - |\n| `zero_terms_query` | `all` or `none` | Defines if all (`all`) or no documents (`none`) should be returned if the query does not contain any terms after tokenization. | `none` |\n| `max_expansions` | `Integer` | Number of terms to be match by the prefix matching. | 50 |\n| `slop` | `Integer` | Allows extra tokens between the query tokens. | 0 |\n| `analyzer` | String | Analyzer meant to cut the query into terms. It is recommended to NOT use this parameter. | The actual field tokenizer. |\n\n\n### `match_bool_prefix`\n\n[Elasticsearch reference documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-match-query-phrase-prefix.html)\n\n#### Example\n\n```json\n{\n \"query\": {\n \"match_bool_prefix\": {\n \"payload.commits.message\": {\n \"query\": \"automated comm\" // This will match \"automated commit\" for instance.\n }\n }\n }\n}\n```\n\nContrary to ES/Opensearch, in Quickwit, at most 50 terms will be considered when searching the last term of the query as a prefix `match_bool_prefix`.\n\n#### Supported Parameters\n\n| Variable | Type | Description | Default |\n| ------------------ | ----------------- | ------------------------------------------------------------------------------------------------------------------------------ | ------- |\n| `query` | String | Full-text search query. The last token will be prefix-matched | - |\n| `operator` | `\"AND\"` or `\"OR\"` | Defines whether all terms should be present (`AND`) or if at least one term is sufficient to match (`OR`). | OR |\n| `zero_terms_query` | `all` or `none` | Defines if all (`all`) or no documents (`none`) should be returned if the query does not contain any terms after tokenization. | `none` |\n\n\n\n### `Multi-match`\n\n[Elasticsearch reference documentation](https://www.elastic.co/guide/en/elasticsearch/reference/8.8/query-dsl-multi-match-query.html)\n\n#### Example\n```json\n{\n \"query\": {\n \"multi_match\": {\n \"query\": \"search keywords\",\n \"fields\": [\n \"title\",\n \"body\"\n ]\n }\n }\n}\n```\n\n```json\n{\n \"query\": {\n \"multi_match\": {\n \"query\": \"search keywords\",\n \"type\": \"most_fields\",\n \"fields\": [\n \"title\",\n \"body\"\n ]\n }\n }\n}\n```\n\n```json\n{\n \"query\": {\n \"multi_match\": {\n \"query\": \"search keywords\",\n \"type\": \"phrase\",\n \"fields\": [\n \"title\",\n \"body\"\n ]\n }\n }\n}\n```\n\n```json\n{\n \"query\": {\n \"multi_match\" : {\n \"query\": \"search key\",\n \"type\": \"phrase_prefix\",\n \"fields\": [ \"title\", \"body\" ]\n }\n }\n}\n```\n\n#### Supported parameters\n\n| Variable | Type | Description | Default value |\n| ------------------ | --------------------- | ---------------------------------------------| ------------- |\n| `type` | `String` | See supported types below | `most_fields` |\n| `fields` | `String[]` (Optional) | Default search target fields. | - |\n| `lenient` | `Boolean` | [See note](#about-the-lenient-argument). | false |\n\nSupported types:\n\n| `type` value | Description |\n| --------------- | ------------------------------------------------------------------------------------------- |\n| `most_fields` | Finds documents matching any field and combines the `_score` from each field (default). |\n| `phrase` | Runs a `match_phrase` query on each field. |\n| `phrase_prefix` | Runs a `match_phrase_prefix` query on each field. |\n| `bool_prefix` | Runs a `match_bool_prefix` query on each field. |\n\n:::warning\n\nIn `phrase`, `phrase_prefix` and `bool_prefix` modes, Quickwit sums the score of the different fields instead of returning their max.\n\nMoreover, while Quickwit does not support `best_fields` or `cross_fields`, it will not return an error when presented a `best_fields` or `cross_fields` type. For compatibilility reasons, Quickwit silently accepts these parameters and interprets them as a `most_fields` type.\n\n:::\n\n### `term`\n\n[Elasticsearch reference documentation](https://www.elastic.co/guide/en/elasticsearch/reference/8.8/query-dsl-term-query.html)\n\n:::note\n\nWhen working on text, it is recommended to only use `term` queries on fields configured with `tokenizer: raw`. This is the Quickwit equivalent of the Elasticsearch `keyword` type.\n\n:::\n\n#### Example\n\n```json\n{\n \"query\": {\n \"term\": {\n \"payload.commits.message\": {\n \"value\": \"automated\",\n \"boost\": 2.0\n }\n }\n }\n}\n```\n\n#### Supported Parameters\n\n| Variable | Type | Description | Default |\n| ------------------ | ------- | ---------------------------------------------------------------------------- | ------- |\n| `value` | String | Term value. This is the string representation of a token after tokenization. | - |\n| `boost` | Number | Multiplier boost for score computation | 1.0 |\n| `case_insensitive` | Boolean | Allows ASCII case insensitive matching of the value. | false |\n\n\n\n\n### `match_all` / `match_none`\n\n[Elasticsearch reference documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-match-all-query.html)\n\n#### Example\n\n```json\n{\"match_all\": {}}\n```\n```json\n{\"match_none\": {}}\n```\n\n\n### `exists`\n\n[Elasticsearch reference documentation](https://www.elastic.co/guide/en/elasticsearch/reference/8.8/query-dsl-exists-query.html)\n\nQuery matching only documents containing a non-null value for a given field.\n\n#### Example\n\n```json\n{\n \"query\": {\n \"exists\": {\n \"field\": \"author.login\"\n }\n }\n}\n```\n\n#### Supported Parameters\n\n| Variable | Type | Description | Default |\n| -------- | ------ | ------------------------------------------------------- | ------- |\n| `field` | String | Only documents with a value for field will be returned. | - |\n\n### `prefix`\n\n[Elasticsearch reference documentation](https://www.elastic.co/guide/en/elasticsearch/reference/8.8/query-dsl-prefix-query.html)\n\nReturns documents that contain a specific prefix in a provided field.\n\n#### Example\n\n```json\n{\n \"query\": {\n \"prefix\": {\n \"author.login\" {\n \"value\": \"adm\",\n }\n }\n }\n}\n```\n\n#### Supported Parameters\n\n| Variable | Type | Description | Default |\n| ------------------ | ------- | ---------------------------------------------------- | ------- |\n| `value` | String | Beginning characters of terms you wish to find. | - |\n| `case_insensitive` | Boolean | Allows ASCII case insensitive matching of the value. | false |\n\n### `wildcard`\n\n[Elasticsearch reference documentation](https://www.elastic.co/guide/en/elasticsearch/reference/8.8/query-dsl-wildcard-query.html)\n\nReturns documents that contain terms matching a wildcard pattern:\n* `?` replaces one and only one term character\n* `*` replaces any number of term characters or an empty string\n\n#### Example\n\n```json\n{\n \"query\": {\n \"wildcard\": {\n \"author.login\" {\n \"value\": \"adm?n*\",\n }\n }\n }\n}\n```\n\n#### Supported Parameters\n\n| Variable | Type | Description | Default |\n| ------------------ | ------- | ---------------------------------------------------- | ------- |\n| `value` | String | Wildcard pattern for terms you wish to find. | - |\n| `boost` | Number | Multiplier boost for score computation. | 1.0 |\n| `case_insensitive` | Boolean | Allows ASCII case insensitive matching of the value. | false |\n\n\n### `regexp`\n\n[Elasticsearch reference documentation](https://www.elastic.co/guide/en/elasticsearch/reference/8.8/query-dsl-regexp-query.html)\n\nReturns documents that contain terms matching a regular expression.\n\n#### Example\n\n```json\n{\n \"query\": {\n \"regexp\": {\n \"author.login\" {\n \"value\": \"adm.*n\",\n }\n }\n }\n}\n```\n\n#### Supported Parameters\n\n| Variable | Type | Description | Default |\n| ------------------ | ------- | ---------------------------------------------------- | ------- |\n| `value` | String | Wildcard pattern for terms you wish to find. | - |\n| `case_insensitive` | Boolean | Allows ASCII case insensitive matching of the value. | false |\n\n\n### About the `lenient` argument\n\nQuickwit and Elasticsearch have different interpretations of the `lenient` setting:\n- In Quickwit, lenient mode allows ignoring parts of the query that reference non-existing columns. This is a behavior that Elasticsearch supports by default.\n- In Elasticsearch, lenient mode primarily addresses type errors (such as searching for text in an integer field). Quickwit always supports this behavior, regardless of the `lenient` setting.\n\n## Search multiple indices\n\nSearch APIs that accept requests path parameter also support multi-target syntax.\n\n### Multi-target syntax\n\nIn multi-target syntax, you can use a comma or its URL encoded version '%2C' separated list to run a request on multiple indices: test1,test2,test3. You can also sue [glob-like](https://en.wikipedia.org/wiki/Glob_(programming)) wildcard ( \\* ) expressions to target indices that match a pattern: test\\* or \\*test or te\\*t or \\*test\\*.\n\nThe multi-target expression has the following constraints:\n\n - It must follow the regex `^[a-zA-Z\\*][a-zA-Z0-9-_\\.\\*]{0,254}$`.\n - It cannot contain consecutive asterisks (`*`).\n - If it does not contain an asterisk (`*`), the length must be greater than or equal to 3 characters.\n\n### Examples\n```\nGET api/v1/_elastic/stackoverflow-000001,stackoverflow-000002/_search\n{\n \"query\": {\n \"query_string\": {\n \"query\": \"search AND engine\",\n \"fields\": [\n \"title\",\n \"body\"\n ]\n }\n }\n}\n```\n\n```\nGET api/v1/_elastic/stackoverflow*/_search\n{\n \"query\": {\n \"query_string\": {\n \"query\": \"search AND engine\",\n \"fields\": [\n \"title\",\n \"body\"\n ]\n }\n }\n}\n```\n" }, { "path": "docs/reference/metrics.md", "content": "---\ntitle: Metrics\nsidebar_position: 70\n---\n\nQuickwit exposes key metrics in the [Prometheus](https://prometheus.io/) format on the `/metrics` endpoint. You can use any front-end that supports Prometheus to examine the behavior of Quickwit visually.\n\n## Cache Metrics\n\nCurrently Quickwit exposes metrics for three caches: `fastfields`, `shortlived`, `splitfooter`. These metrics share the same structure.\n\n| Namespace | Metric Name | Description | Type |\n| --------- | ----------- | ----------- | ---- |\n| `quickwit_cache_{cache_name}` | `in_cache_count` | Count of {cache_name} in cache | `gauge` |\n| `quickwit_cache_{cache_name}` | `in_cache_num_bytes` | Number of {cache_name} bytes in cache | `gauge` |\n| `quickwit_cache_{cache_name}` | `cache_hit_total` | Number of {cache_name} cache hits | `counter` |\n| `quickwit_cache_{cache_name}` | `cache_hits_bytes` | Number of {cache_name} cache hits in bytes | `counter` |\n| `quickwit_cache_{cache_name}` | `cache_miss_total` | Number of {cache_name} cache hits | `counter` |\n\n## CLI Metrics\n\n| Namespace | Metric Name | Description | Type |\n| --------- | ----------- | ----------- | ---- |\n| `quickwit` | `allocated_num_bytes` | Number of bytes allocated memory, as reported by jemalloc. | `gauge` |\n\n## Common Metrics\n\n| Namespace | Metric Name | Description | Labels | Type |\n| --------- | ----------- | ----------- | ------ | ---- |\n| `quickwit` | `write_bytes`| Number of bytes written by a given component in [`indexer`, `merger`, `deleter`, `split_downloader_{merge,delete}`] | [`index`, `component`] | `counter` |\n\n## Indexing Metrics\n\n| Namespace | Metric Name | Description | Labels | Type |\n| --------- | ----------- | ----------- | ------ | ---- |\n| `quickwit_indexing` | `processed_docs_total`| Number of processed docs by index, source and processed status in [`valid`, `schema_error`, `parse_error`, `transform_error`] | [`index`, `source`, `docs_processed_status`] | `counter` |\n| `quickwit_indexing` | `processed_bytes`| Number of processed bytes by index, source and processed status in [`valid`, `schema_error`, `parse_error`, `transform_error`] | [`index`, `source`, `docs_processed_status`] | `counter` |\n| `quickwit_indexing` | `available_concurrent_upload_permits`| Number of available concurrent upload permits by component in [`merger`, `indexer`] | [`component`] | `gauge` |\n| `quickwit_indexing` | `ongoing_merge_operations`| Number of available concurrent upload permits by component in [`merger`, `indexer`]. | [`index`, `source`] | `gauge` |\n\n## Ingest Metrics\n\n| Namespace | Metric Name | Description | Type |\n| --------- | ----------- | ----------- | ---- |\n| `quickwit_ingest` | `docs_bytes_total` | Total size of the docs ingested, measured in ingester's leader, after validation and before persistence/replication | `counter` |\n| `quickwit_ingest` | `docs_total` | Total number of the docs ingested, measured in ingester's leader, after validation and before persistence/replication | `counter` |\n| `quickwit_ingest` | `queue_count` | Number of queues currently active | `counter` |\n\n## Metastore Metrics\n\nAll metastore methods are monitored by the 3 metrics:\n\n| Namespace | Metric Name | Description | Labels | Type |\n| --------- | ----------- | ----------- | ------ | ---- |\n| `quickwit_metastore` | `requests_total` | Number of requests | [`operation`, `index`] | `counter` |\n| `quickwit_metastore` | `request_errors_total` | Number of failed requests | [`operation`, `index`] | `counter` |\n| `quickwit_metastore` | `request_duration_seconds` | Duration of requests | [`operation`, `index`, `error`] | `histogram` |\n\nExamples of operation names: `create_index`, `index_metadata`, `delete_index`, `stage_splits`, `publish_splits`, `list_splits`, `add_source`, ...\n\n## Rest API Metrics\n\n| Namespace | Metric Name | Description | Type |\n| --------- | ----------- | ----------- | ---- |\n| `quickwit` | `http_requests_total` | Total number of HTTP requests received | `counter` |\n\n## Search Metrics\n\n| Namespace | Metric Name | Description | Type |\n| --------- | ----------- | ----------- | ---- |\n| `quickwit_search` | `leaf_searches_splits_total` | Number of leaf searches (count of splits) started | `counter` |\n| `quickwit_search` | `leaf_search_split_duration_secs` | Number of seconds required to run a leaf search over a single split. The timer starts after the semaphore is obtained | `histogram` |\n| `quickwit_search` | `active_search_threads_count` | Number of threads in use in the CPU thread pool | `gauge` |\n\n## Storage Metrics\n\n| Namespace | Metric Name | Description | Type |\n| --------- | ----------- | ----------- | ---- |\n| `quickwit_storage` | `object_storage_gets_total` | Number of objects fetched | `counter` |\n| `quickwit_storage` | `object_storage_puts_total` | Number of objects uploaded. May differ from object_storage_requests_parts due to multipart upload | `counter` |\n| `quickwit_storage` | `object_storage_puts_parts` | Number of object parts uploaded | `counter` |\n| `quickwit_storage` | `object_storage_download_num_bytes` | Amount of data downloaded from an object storage | `counter` |\n" }, { "path": "docs/reference/query-language.md", "content": "---\ntitle: Query Language Reference\nsidebar_position: 40\n---\n\n## Pseudo-grammar\n\n```\nquery = '(' query ')'\n | query operator query\n | unary_operator query\n | query query\n | clause\n\noperator = 'AND' | 'OR'\n\nunary_operator = 'NOT' | '-'\n\nclause = field_name ':' field_clause\n | defaultable_clause\n | '*'\n\nfield_clause = term | term_prefix | term_set | phrase | phrase_prefix | range | '*'\ndefaultable_clause = term | term_prefix | term_set | phrase | phrase_prefix\n```\n---\n## Writing Queries\n### Escaping Special Characters\n\nSome characters need to be escaped in non quoted terms because they are syntactically significant otherwise: special reserved characters are: `+` , `^`, `` ` ``, `:`, `{`, `}`, `\"`, `[`, `]`, `(`, `)`, `~`, `!`, `\\\\`, `*`, `SPACE`. If such such characters appear in query terms, they need to be escaped by prefixing them with an anti-slash `\\`.\n\nIn quoted terms, the quote character in use `'` or `\"` needs to be escaped.\n\n###### Allowed characters in field names\n\nSee the [Field name validation rules](https://quickwit.io/docs/configuration/index-config#field-name-validation-rules) in the index config documentation.\n\n### Addressing nested structures\n\nData stored deep inside nested data structures like `object` or `json` fields can be addressed using dots as separators in the field name.\nFor instance, the document `{\"product\": {\"attributes\": {color\": \"red\"}}}` is matched by\n```\nproduct.attributes.color:red\n```\n\nIf the keys of your object contain dots, the above syntax has some ambiguity : by default `{\"k8s.component.name\": \"quickwit\"}` will be matched by \n```k8s.component.name:quickwit```\n\nIt is possible to remove the ambiguity by setting expand_dots in the json field configuration. \nIn that case, it will be necessary to escape the `.` in the query to match this document like this :\n```\nk8s\\.component\\.name:quickwit\n```\n\n---\n\n## Structured data\n### Datetime\nDatetime values must be provided in rfc3339 format, such as `1970-01-01T00:00:00Z`\n\n### IP addresses\nIP addresses can be provided as IPv4 or IPv6. It is recommended to search with the format used when indexing documents.\nThere is no support for searching for a range of IP using CIDR notation, but you can use normal range queries.\n\n---\n\n## Types of clauses\n\n### Term `field:term`\n```\nterm = term_char+\n```\n\nMatches documents if the targeted field contains a token equal to the provided term. \n\n`field:value` will match any document where the field 'field' has a token 'value'.\n\n### Wildcard `field:wil?car*d`\n```\nwildcard = [term_char\\*\\?]+\n```\n\nMatches documents if the targeted field contains a token that matches the wildcard:\n- `?` replaces one and only one term character\n- `*` replaces any number of term characters or an empty string\n\nExamples:\n- `field:quick*` will match any document where the field 'field' has a token like `quickwit` or `quickstart`, but not `qui` or `abcd`.\n- `field:h?llo` will match any document where the field 'field' has a token like `hello` or `hallo`, but not `heillo` or `hllo`.\n\nQueries with prefixes (`field:qui*`) are much more efficient than queries starting with a wildcard (`field:*wit`)\n\n\n### Term set `field:IN [a b c]`\n```\nterm_set = 'IN' '[' term_list ']'\nterm_list = term_list term\n | term\n```\nMatches if the document contains any of the tokens provided. \n\n###### Examples\n`field:IN [ab cd]` will match 'ab' or 'cd', but nothing else.\n\n###### Performance Note\nThis is a lot like writing `field:ab OR field:cd`. When there are only a handful of terms to search for, using ORs is usually faster.\nWhen there are many values to match, a term set query can become more efficient.\n\n\n\n### Phrase `field:\"sequence of words\"`\n```\nphrase = phrase_string\n | phrase_string slop\nphrase_string = '\"' phrase_char '\"'\nslop = '~' [01-9]+\n\n```\n\nMatches if the field contains the sequence of token provided:\n- `field:\"looks good to me\"` will match any document containing that sequence of tokens.\n- `field:\"look* good to me\"` with the default tokenizer is equivalent to `field:\"look good to me\"`, i.e. the '*' character is pruned by the tokenizer and not interpreted as a wildcard.\n\n:::info\n\nThe field must have been configured with `record: position` when indexing.\n\n:::\n\n###### Slop operator\nIs is also possible to add a slop, which allow matching a sequence with some distance. For instance `\"looks to me\"~1` will match \"looks good to me\", but not \"looks very good to me\".\nTransposition costs 2, e.g. `\"A B\"~1` will not match `\"B A\"` but it would with `\"A B\"~2`.\nTransposition is not a special case, in the example above A is moved 1 position and B is moved 1 position, so the slop is 2.\n\n### Phrase Prefix `field:\"finish this phr\"*`\n```\nphrase_prefix = phrase '*'\n```\n\nMatches if the field contains the sequence of token provided, where the last token in the query may be only a prefix of the token in the document.\n\nThe field must have been configured with `record: position` when indexing.\n\nThere is no slop for phrase prefix queries.\n\n###### Examples\n `field:\"thanks for your contrib\"*` will match 'thanks for your contribution'.\n\n###### Limitation\n\nQuickwit may trim some results matched by this clause in some cases. If you search for `\"thanks for your co\"*`, it will enumerate the first 50 tokens which start with \"co\" (in their storage order), and search for any documents where \"thanks for your\" is followed by any of these tokens.\n\nIf there are many tokens starting with \"co\", \"contribution\" might not be one of the 50 selected tokens, and the query won't match a document containing \"thanks for your contribution\". Normal prefix queries don't suffer from this issue.\n\n### Range `field:[low_bound TO high_bound}`\n```\nrange = explicit_range | comparison_half_range\n\nexplicit_range = left_bound_char bounds right_bound_char\nleft_bound_char = '[' | '{' \nright_bound_char = '}' | ']'\nbounds = term TO term\n | term TO '*'\n | '*' TO term\n\ncomparison_range = comparison_operator term\ncomparison_operator = '<' | '>' | '<=' | '>='\n```\n\nMatches if the document contains a token between the provided bounds for that field.\nFor range queries, you must provide a field. Quickwit won't use `default_search_fields` automatically.\n\n###### Order\nFor text fields, the ranges are defined by lexicographic order on uft-8 encoded byte arrays. It means for a text field, 100 is between 1 and 2.\n\n\nWhen using ranges on integers, it behaves naturally.\n\n###### Inclusive and exclusive bounds\nInclusive bounds are represented by square brackets `[]`. They will match tokens equal to the bound term.\nExclusive bounds are represented by curly brackets `{}`. They will not match tokens equal to the bound term.\n\n###### Half-Open bounds\nYou can make an half open range by using `*` as one of the bounds. `field:[b TO *]` will match 'bb' and 'zz', but not 'ab'.\nYou can also use a comparison based syntax:`field:b`, `field:<=b` or `field:>=b`.\n\n\n\n###### Examples\n- Inclusive Range: `ip:[127.0.0.1 TO 127.0.0.50]`\n- Exclusive Range: `ip:{127.0.0.1 TO 127.0.0.50}`\n- Unbounded Inclusive Range: `ip:[127.0.0.1 TO *] or ip:>=127.0.0.1`\n- Unbounded Exclusive Range: `ip:{127.0.0.1 TO *] or ip:>127.0.0.1`\n\n\n### Exists `field:*`\n\nMatches documents where the field is set. You have to specify a field for this query, Quickwit won't use `default_search_fields` automatically.\n\n### Match All `*`\n\nMatches every document. You can't put a field in front. It is simply written as `*`.\n\n---\n\n## Building Queries\nMost queries are composed of more than one clause. When doing so, you may add operators between clauses.\n\nImplicitly if no operator is provided, 'AND' is assumed.\n\n### Conjunction `AND`\nAn `AND` query will match only if both sides match.\n\n\n\n### Disjunction `OR`\nAn `OR` query will match if either (or both) sides match.\n\n\n\n### Negation `NOT` or `-`\nA `NOT` query will match if the clause it is applied to does not match.\nThe `-` prefix is equivalent to the `NOT` operator.\n\n### Grouping `()`\nParentheses are used to force the order of evaluation of operators.\nFor instance, if a query should match if 'field1' is 'one' or 'two', and 'field2' is 'three', you can use `(field1:one OR field1:two) AND field2:three`.\n\n### Operator Precedence\nWithout parentheses, `AND` takes precedence over `OR`. That is, `a AND b OR c` is interpreted as `(a AND b) or c`.\n\n`NOT` and `-` takes precedence over everything, such that `-a AND b` means `(-a) AND b`, not `-(a AND B)`.\n\n\n---\n\n## Other considerations \n\n### Default Search Fields\nIn many case it is possible to omit the field you search if it was configured in the `default_search_fields` array of the index configuration. If more than one field is configured as default, the resulting implicit clauses are combined using a conjunction ('OR').\n\n### Tokenization\nNote that the result of a query can depend on the tokenizer used for the field getting searched. Hence this document always speaks of tokens, which may be the exact value the document contain (in case of the raw tokenizer), or a subset of it (for instance any tokenizer cutting on spaces).\n\n\n" }, { "path": "docs/reference/rest-api.md", "content": "---\ntitle: REST API\nsidebar_position: 10\n---\n\n## API version\n\nAll the API endpoints start with the `api/v1/` prefix. `v1` indicates that we are currently using version 1 of the API.\n\n\n## OpenAPI specification\n\nThe OpenAPI specification of the REST API is available at `/openapi.json` and a Swagger UI version is available at `/ui/api-playground`.\n\n## Parameters\n\nParameters passed in the URL must be properly URL-encoded, using the UTF-8 encoding for non-ASCII characters.\n\n```\nGET [..]/search?query=barack%20obama\n```\n\n## Error handling\n\nSuccessful requests return a 2xx HTTP status code.\n\nFailed requests return a 4xx HTTP status code. The response body of failed requests holds a JSON object containing a `message` field that describes the error.\n\n```json\n{\n \"message\": \"Failed to parse query\"\n}\n```\n\n## Search API\n\n### Search in an index\n\nSearch for documents matching a query in the given index `api/v1//search`. This endpoint is available as long as you have at least one node running a searcher service in the cluster.\nThe search endpoint accepts `GET` and `POST` requests. The [parameters](#get-parameters) are URL parameters for `GET` requests or JSON key-value pairs for `POST` requests.\n\n```\nGET api/v1//search?query=searchterm\n```\n\n```\nPOST api/v1//search\n{\n \"query\": searchterm\n}\n```\n\n#### Path variable\n\n| Variable | Description |\n| ------------- | ------------- |\n| `index id` | The index id |\n\n#### Parameters\n\n| Variable | Type | Description | Default value |\n|---------------------|------------|-----------------|-----------------|\n| `query` | `String` | Query text. See the [query language doc](query-language.md) | _required_ |\n| `start_timestamp` | `i64` | If set, restrict search to documents with a `timestamp >= start_timestamp`, taking advantage of potential time pruning opportunities. The value must be in seconds. | |\n| `end_timestamp` | `i64` | If set, restrict search to documents with a `timestamp < end_timestamp`, taking advantage of potential time pruning opportunities. The value must be in seconds. | |\n| `start_offset` | `Integer` | Number of documents to skip | `0` |\n| `max_hits` | `Integer` | Maximum number of hits to return (by default 20) | `20` |\n| `search_field` | `[String]` | Fields to search on if no field name is specified in the query. Comma-separated list, e.g. \"field1,field2\" | index_config.search_settings.default_search_fields |\n| `snippet_fields` | `[String]` | Fields to extract snippet on. Comma-separated list, e.g. \"field1,field2\" | |\n| `sort_by` | `[String]` | Fields to sort the query results on. You can sort by one or two fast fields or by BM25 `_score` (requires fieldnorms). By default, hits are sorted in reverse order of their [document ID](/docs/overview/concepts/querying.md#document-id) (to show recent events first). | |\n| `format` | `Enum` | The output format. Allowed values are \"json\" or \"pretty_json\" | `pretty_json` |\n| `aggs` | `JSON` | The aggregations request. See the [aggregations doc](aggregation.md) for supported aggregations. | |\n\n:::info\nThe `start_timestamp` and `end_timestamp` should be specified in seconds regardless of the timestamp field precision.\n:::\n\n#### Response\n\nThe response is a JSON object, and the content type is `application/json; charset=UTF-8.`\n\n| Field | Description | Type |\n| -------------------- | ------------------------------ | :--------: |\n| `hits` | Results of the query | `[hit]` |\n| `num_hits` | Total number of matches | `number` |\n| `elapsed_time_micros` | Processing time of the query | `number` |\n\n### Search multiple indices\nSearch APIs that accept `index id` requests path parameter also support multi-target syntax.\n\n#### Multi-target syntax\n\nIn multi-target syntax, you can use a comma or its URL encoded version '%2C' separated list to run a request on multiple indices: test1,test2,test3. You can also use [glob-like](https://en.wikipedia.org/wiki/Glob_(programming)) wildcard ( \\* ) expressions to target indices that match a pattern: test\\* or \\*test or te\\*t or \\*test\\*.\n\nThe following are some constrains about the multi-target expression.\n\n - It must follow the regex `^[a-zA-Z\\*][a-zA-Z0-9-_\\.\\*]{0,254}$`.\n - It cannot contain consecutive asterisks (`*`).\n - If it does not contain an asterisk (`*`), the length must be greater than or equal to 3 characters.\n\n#### Examples\n```\nGET api/v1/stackoverflow-000001,stackoverflow-000002/search\n{\n \"query\": \"search AND engine\",\n}\n```\n\n```\nGET api/v1/stackoverflow*/search\n{\n \"query\": \"search AND engine\",\n}\n```\n\n## Ingest API\n\n### Ingest data into an index\n\n```\nPOST api/v1//ingest -d \\\n'{\"url\":\"https://en.wikipedia.org/wiki?id=1\",\"title\":\"foo\",\"body\":\"foo\"}\n{\"url\":\"https://en.wikipedia.org/wiki?id=2\",\"title\":\"bar\",\"body\":\"bar\"}\n{\"url\":\"https://en.wikipedia.org/wiki?id=3\",\"title\":\"baz\",\"body\":\"baz\"}'\n```\n\nIngest a batch of documents to make them searchable in a given ``. Currently, NDJSON is the only accepted payload format. This endpoint is only available on a node that is running an indexer service.\n\n#### Controlling when the indexed documents will be available for search\n\nNewly added documents will not appear in the search results until they are added to a split and that split is committed. This process is automatic and is controlled by `split_num_docs_target` and `commit_timeout_secs` parameters. By default, the ingest command exits as soon as the records are added to the indexing queue, which means that the new documents will not appear in the search results at this moment. This behavior can be changed by adding `commit=wait_for` or `commit=force` parameters to the query. The `wait_for` parameter will cause the command to wait for the documents to be committed according to the standard time or number of documents rules. The `force` parameter will trigger a commit after all documents in the request are processed. It will also wait for this commit to finish before returning. Please note that the `force` option may have a significant performance cost especially if it is used on small batches.\n\n```\nPOST api/v1//ingest?commit=wait_for -d \\\n'{\"url\":\"https://en.wikipedia.org/wiki?id=1\",\"title\":\"foo\",\"body\":\"foo\"}\n{\"url\":\"https://en.wikipedia.org/wiki?id=2\",\"title\":\"bar\",\"body\":\"bar\"}\n{\"url\":\"https://en.wikipedia.org/wiki?id=3\",\"title\":\"baz\",\"body\":\"baz\"}'\n```\n\n:::info\n\nThe payload size is limited to 10MB [by default](../configuration/node-config.md#ingest-api-configuration) since this endpoint is intended to receive documents in batches.\n\n:::\n\n#### Path variable\n\n| Variable | Description |\n| ------------- | ------------- |\n| `index id` | The index id |\n\n#### Query parameters\n\n| Variable | Type | Description | Default value |\n|---------------------|------------|----------------------------------------------------|---------------|\n| `commit` | `String` | The commit behavior: `auto`, `wait_for` or `force` | `auto` |\n| `detailed_response` | `bool` | Enable `parse_failures` in the response. Setting to `true` might impact performances negatively. | `false` |\n\n#### Response\n\nThe response is a JSON object, and the content type is `application/json; charset=UTF-8.`\n\n| Field | Description | Type |\n|-----------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------:|\n| `num_docs_for_processing` | Total number of documents submitted for processing. The documents may not have been processed. | `number` |\n| `num_ingested_docs` | Number of documents successfully persisted in the write ahead log | `number` |\n| `num_rejected_docs` | Number of documents that couldn't be parsed (invalid json, bad schema...) | `number` |\n| `parse_failures` | List detailing parsing failures. Only available if `detailed_response` is set to `true`. | `list(object)` |\n\nThe parse failure objects contain the following fields:\n- `message`: a detailed message explaining the error\n- `reason`: one of `invalid_json`, `invalid_schema` or `unspecified`\n- `document`: the utf-8 decoded string of the document byte chunk that generated the error\n\n\n## Index API\n\n### Create an index\n\n```\nPOST api/v1/indexes\n```\n\nCreate an index by posting an `IndexConfig` payload. The API accepts JSON with `content-type: application/json` and YAML with `content-type: application/yaml`.\n\n#### POST payload\n\n| Variable | Type | Description | Default value |\n|---------------------|--------------------|-----------------------------------------------------------------------------------------------------------------------|---------------------------------------|\n| `version` | `String` | Config format version, use the same as your Quickwit version. | _required_ |\n| `index_id` | `String` | Index ID, see its [validation rules](../configuration/index-config.md#index-id) on identifiers. | _required_ |\n| `index_uri` | `String` | Defines where the index files are stored. This parameter expects a [storage URI](../configuration/storage-config.md#storage-uris). | `{default_index_root_uri}/{index_id}` |\n| `doc_mapping` | `DocMapping` | Doc mapping object as specified in the [index config docs](../configuration/index-config.md#doc-mapping). | _required_ |\n| `indexing_settings` | `IndexingSettings` | Indexing settings object as specified in the [index config docs](../configuration/index-config.md#indexing-settings). | |\n| `search_settings` | `SearchSettings` | Search settings object as specified in the [index config docs](../configuration/index-config.md#search-settings). | |\n| `retention` | `Retention` | Retention policy object as specified in the [index config docs](../configuration/index-config.md#retention-policy). | |\n\n\n**Payload Example**\n\ncurl -XPOST http://localhost:7280/api/v1/indexes --data @index_config.json -H \"Content-Type: application/json\"\n\n```json title=\"index_config.json\n{\n \"version\": \"0.8\",\n \"index_id\": \"hdfs-logs\",\n \"doc_mapping\": {\n \"field_mappings\": [\n {\n \"name\": \"tenant_id\",\n \"type\": \"u64\",\n \"fast\": true\n },\n {\n \"name\": \"app_id\",\n \"type\": \"u64\",\n \"fast\": true\n },\n {\n \"name\": \"timestamp\",\n \"type\": \"datetime\",\n \"input_formats\": [\"unix_timestamp\"],\n \"fast_precision\": \"seconds\",\n \"fast\": true\n },\n {\n \"name\": \"body\",\n \"type\": \"text\",\n \"record\": \"position\"\n }\n ],\n \"partition_key\": \"tenant_id\",\n \"max_num_partitions\": 200,\n \"tag_fields\": [\"tenant_id\"],\n \"timestamp_field\": \"timestamp\"\n },\n \"search_settings\": {\n \"default_search_fields\": [\"body\"]\n },\n \"indexing_settings\": {\n \"merge_policy\": {\n \"type\": \"limit_merge\",\n \"max_merge_ops\": 3,\n \"merge_factor\": 10,\n \"max_merge_factor\": 12\n }\n },\n \"retention\": {\n \"period\": \"7 days\",\n \"schedule\": \"@daily\"\n }\n}\n```\n\n#### Response\n\nThe response is the index metadata of the created index, and the content type is `application/json; charset=UTF-8.`\n\n| Field | Description | Type |\n|----------------------|-----------------------------------------------|:---------------------:|\n| `version` | The current index configuration format version. | `string` |\n| `index_uid` | The server-generated index UID. | `string` |\n| `index_config` | The posted index config. | `IndexConfig` |\n| `checkpoint` | Map of checkpoints by source. | `IndexCheckpoint` |\n| `create_timestamp` | Index creation timestamp | `number` |\n| `sources` | List of the index sources configurations. | `Array` |\n\n\n### Update an index\n\n```\nPUT api/v1/indexes/\n```\n\n#### Path variable\n\n| Variable | Description |\n| ------------- | ------------- |\n| `index id` | The index id |\n\n#### Query parameters\n\n| Variable | Type | Description | Default value |\n|-----------|--------|-----------------------------------------------|---------------|\n| `create` | `bool` | Create the index if it doesn't already exists | `false` |\n\nUpdate the configurations of an index. This endpoint follows PUT semantics, which means that all the fields of the current configuration are replaced by the values specified in this request or the associated defaults. In particular, if the field is optional (e.g. `retention_policy`), omitting it will delete the associated configuration. If the new configuration file contains updates that cannot be applied, the request fails, and none of the updates are applied. The API accepts JSON with `content-type: application/json` and YAML with `content-type: application/yaml`.\n\n- The retention policy update is automatically picked up by the janitor service on its next state refresh.\n- The search settings update is automatically picked up by searcher nodes when the next query is executed.\n- The indexing settings update is automatically picked up by the indexer nodes once the control plane emits a new indexing plan.\n- The doc mapping update is automatically picked up by the indexer nodes once the control plane emit a new indexing plan.\n\n:::warning\n\nIf you use the ingest or ES bulk API (V2), the old doc mapping will still be used to validate new documents that end up being persisted on existing shards (see [#5738](https://github.com/quickwit-oss/quickwit/issues/5738)).\n\n:::\n\nUpdating the doc mapping doesn't reindex existing data. Queries and results are mapped on a best-effort basis when querying older splits. For more details, check [the reference](updating-mapper.md) out.\n\n#### PUT payload\n\n| Variable | Type | Description | Default value |\n|---------------------|--------------------|-----------------------------------------------------------------------------------------------------------------------|---------------------------------------|\n| `version` | `String` | Config format version, use the same as your Quickwit version. | _required_ |\n| `index_id` | `String` | Index ID, must be the same index as in the request URI. | _required_ |\n| `index_uri` | `String` | Defines where the index files are stored. Cannot be updated. | `{default_index_root_uri}/{index_id}` |\n| `doc_mapping` | `DocMapping` | Doc mapping object as specified in the [index config docs](../configuration/index-config.md#doc-mapping). | _required_ |\n| `indexing_settings` | `IndexingSettings` | Indexing settings object as specified in the [index config docs](../configuration/index-config.md#indexing-settings). | |\n| `search_settings` | `SearchSettings` | Search settings object as specified in the [index config docs](../configuration/index-config.md#search-settings). | |\n| `retention` | `Retention` | Retention policy object as specified in the [index config docs](../configuration/index-config.md#retention-policy). | |\n\n\n**Payload Example**\n\ncurl -XPUT http://localhost:7280/api/v1/indexes/hdfs-logs --data @updated_index_update.json -H \"Content-Type: application/json\"\n\n```json title=\"updated_index_update.json\n{\n \"version\": \"0.8\",\n \"index_id\": \"hdfs-logs\",\n \"doc_mapping\": {\n \"field_mappings\": [\n {\n \"name\": \"tenant_id\",\n \"type\": \"u64\",\n \"fast\": true\n },\n {\n \"name\": \"app_id\",\n \"type\": \"u64\",\n \"fast\": true\n },\n {\n \"name\": \"timestamp\",\n \"type\": \"datetime\",\n \"input_formats\": [\"unix_timestamp\"],\n \"fast_precision\": \"seconds\",\n \"fast\": true\n },\n {\n \"name\": \"body\",\n \"type\": \"text\",\n \"record\": \"position\"\n }\n ],\n \"partition_key\": \"tenant_id\",\n \"max_num_partitions\": 200,\n \"tag_fields\": [\"tenant_id\"],\n \"timestamp_field\": \"timestamp\"\n },\n \"search_settings\": {\n \"default_search_fields\": [\"body\"]\n },\n \"indexing_settings\": {\n \"merge_policy\": {\n \"type\": \"limit_merge\",\n \"max_merge_ops\": 3,\n \"merge_factor\": 10,\n \"max_merge_factor\": 12\n }\n },\n \"retention\": {\n \"period\": \"30 days\",\n \"schedule\": \"@daily\"\n }\n}\n```\n\n#### Response\n\nThe response is the index metadata of the updated index, and the content type is `application/json; charset=UTF-8.`\n\n| Field | Description | Type |\n|----------------------|-----------------------------------------|:---------------------:|\n| `version` | The current server configuration version. | `string` |\n| `index_uid` | The server-generated index UID. | `string` |\n| `index_config` | The posted index config. | `IndexConfig` |\n| `checkpoint` | Map of checkpoints by source. | `IndexCheckpoint` |\n| `create_timestamp` | Index creation timestamp | `number` |\n| `sources` | List of the index sources configurations. | `Array` |\n\n\n### Get an index metadata\n\n```\nGET api/v1/indexes/\n```\n\nGet the index metadata of ID `index id`.\n\n#### Response\n\nThe response is the index metadata of the requested index, and the content type is `application/json; charset=UTF-8.`\n\n| Field | Description | Type |\n|----------------------|-------------------------------------------|:---------------------:|\n| `version` | The current server configuration version. | `string` |\n| `index_uid` | The server-generated index UID. | `string` |\n| `index_config` | The posted index config. | `IndexConfig` |\n| `checkpoint` | Map of checkpoints by source. | `IndexCheckpoint` |\n| `create_timestamp` | Index creation timestamp. | `number` |\n| `sources` | List of the index sources configurations. | `Array` |\n\n\n### Describe an index\n\n```\nGET api/v1/indexes//describe\n```\nDescribes an index of ID `index id`.\n\n#### Response\n\nThe response is the stats about the requested index, and the content type is `application/json; charset=UTF-8.`\n\n| Field | Description | Type |\n|-------------------------------------|----------------------------------------------------------|:---------------------:|\n| `index_id` | Index ID of index. | `String` |\n| `index_uri` | Uri of index | `String` |\n| `num_published_splits` | Number of published splits. | `number` |\n| `size_published_splits` | Size of published splits. | `number` |\n| `num_published_docs` | Number of published documents. | `number` |\n| `size_published_docs_uncompressed` | Size of the published documents in bytes (uncompressed). | `number` |\n| `timestamp_field_name` | Name of timestamp field. | `String` |\n| `min_timestamp` | Starting time of timestamp. | `number` |\n| `max_timestamp` | Ending time of timestamp. | `number` |\n\n\n### Get splits\n\n```\nGET api/v1/indexes//splits\n```\nGet splits belongs to an index of ID `index id`.\n\n#### Path variable\n\n| Variable | Description |\n| ------------- | ------------- |\n| `index id` | The index id |\n\n#### Get parameters\n\n| Variable | Type | Description |\n|---------------------|------------|------------------------------------------------------------------------------------------------------------------|\n| `offset` | `number` | If set, restrict the number of splits to skip|\n| `limit ` | `number` | If set, restrict maximum number of splits to retrieve|\n| `split_states` | `usize` | If set, specific split state(s) to filter by|\n| `start_timestamp` | `number` | If set, restrict splits to documents with a `timestamp >= start_timestamp|\n| `end_timestamp` | `number` | If set, restrict splits to documents with a `timestamp < end_timestamp|\n| `end_create_timestamp` | `number` | If set, restrict splits whose creation dates are before this date|\n\n\n#### Response\n\nThe response is the stats about the requested index, and the content type is `application/json; charset=UTF-8.`\n\n| Field | Description | Type |\n|-------------------------------------|----------------------------------------------------------|:---------------------:|\n| `offset` | Index ID of index. | `String` |\n| `size` | Uri of index | `String` |\n| `splits` | Number of published splits. | `List` |\n\n#### Examples\n```\nGET /api/v1/indexes/stackoverflow/splits?offset=0&limit=10\n```\n```json\n{\n \"offset\": 0,\n \"size\": 1,\n \"splits\": [\n {\n \"split_state\": \"Published\",\n \"update_timestamp\": 1695642901,\n \"publish_timestamp\": 1695642901,\n \"version\": \"0.7\",\n \"split_id\": \"01HB632HD8W6WHNM7CZFH3KG1X\",\n \"index_uid\": \"stackoverflow:01HB6321TDT3SP58D4EZP14KSX\",\n \"partition_id\": 0,\n \"source_id\": \"_ingest-api-source\",\n \"node_id\": \"jerry\",\n \"num_docs\": 10000,\n \"uncompressed_docs_size_in_bytes\": 6674940,\n \"time_range\": {\n \"start\": 1217540572,\n \"end\": 1219335682\n },\n \"create_timestamp\": 1695642900,\n \"maturity\": {\n \"type\": \"immature\",\n \"maturation_period_millis\": 172800000\n },\n \"tags\": [],\n \"footer_offsets\": {\n \"start\": 4714989,\n \"end\": 4719999\n },\n \"delete_opstamp\": 0,\n \"num_merge_ops\": 0\n }\n ]\n}\n```\n\n\n### Clears an index\n\n```\nPUT api/v1/indexes//clear\n```\n\nClears index of ID `index id`: all splits will be deleted (metastore + storage) and all source checkpoints will be reset.\n\nIt returns an empty body.\n\n\n### Delete an index\n\n```\nDELETE api/v1/indexes/\n```\n\nDelete index of ID `index id`.\n\n#### Response\n\nThe response is the list of deleted split files; the content type is `application/json; charset=UTF-8.`\n\n```json\n[\n {\n \"split_id\": \"01GK1XNAECH7P14850S9VV6P94\",\n \"num_docs\": 1337,\n \"uncompressed_docs_size_bytes\": 23933408,\n \"file_name\": \"01GK1XNAECH7P14850S9VV6P94.split\",\n \"file_size_bytes\": 2991676\n }\n]\n```\n\n### Get all indexes metadata\n\n```\nGET api/v1/indexes\n```\n\nRetrieve the metadata of all indexes present in the metastore.\n\n#### Response\n\nThe response is an array of `IndexMetadata`, and the content type is `application/json; charset=UTF-8.`\n\n\n### Create a source\n\n```\nPOST api/v1/indexes//sources\n```\n\nCreate source by posting a source config JSON payload.\n\n#### POST payload\n\n| Variable | Type | Description | Default value |\n|-------------------|----------|----------------------------------------------------------------------------------------|---------------|\n| `version** | `String` | Config format version, put your current Quickwit version. | _required_ |\n| `source_id` | `String` | Source ID. See ID [validation rules](../configuration/source-config.md). | _required_ |\n| `source_type` | `String` | Source type: `kafka`, `kinesis` or `pulsar`. | _required_ |\n| `num_pipelines` | `usize` | Number of running indexing pipelines per node for this source. | `1` |\n| `transform` | `object` | A [VRL](https://vector.dev/docs/reference/vrl/) transformation applied to incoming documents, as defined in [source config docs](../configuration/source-config.md#transform-parameters). | `null` |\n| `params` | `object` | Source parameters as defined in [source config docs](../configuration/source-config.md). | _required_ |\n\n\n**Payload Example**\n\ncurl -XPOST http://localhost:7280/api/v1/indexes/my-index/sources --data @source_config.json -H \"Content-Type: application/json\"\n\n```json title=\"source_config.json\n{\n \"version\": \"0.8\",\n \"source_id\": \"kafka-source\",\n \"source_type\": \"kafka\",\n \"params\": {\n \"topic\": \"quickwit-fts-staging\",\n \"client_params\": {\n \"bootstrap.servers\": \"kafka-quickwit-server:9092\"\n }\n }\n}\n```\n\n#### Response\n\nThe response is the created source config, and the content type is `application/json; charset=UTF-8.`\n\n### Update a source\n\n```\nPUT api/v1/indexes//sources/\n```\n\n#### Path variable\n\n| Variable | Description |\n| ------------- | ------------- |\n| `index id` | The index id |\n| `source id` | The source id |\n\n#### Query parameters\n\n| Variable | Type | Description | Default value |\n|-----------|--------|-----------------------------------------------|---------------|\n| `create` | `bool` | Create the index if it doesn't already exists | `false` |\n\nUpdate a source by posting a source config JSON payload.\n\n#### PUT payload\n\n| Variable | Type | Description | Default value |\n|-------------------|----------|----------------------------------------------------------------------------------------|---------------|\n| `version** | `String` | Config format version, put your current Quickwit version. | _required_ |\n| `source_id` | `String` | Source ID, must be the same source as in the request URL. | _required_ |\n| `source_type` | `String` | Source type: `kafka`, `kinesis` or `pulsar`. Cannot be updated. | _required_ |\n| `num_pipelines` | `usize` | Number of running indexing pipelines per node for this source. | `1` |\n| `transform` | `object` | A [VRL](https://vector.dev/docs/reference/vrl/) transformation applied to incoming documents, as defined in [source config docs](../configuration/source-config.md#transform-parameters). | `null` |\n| `params` | `object` | Source parameters as defined in [source config docs](../configuration/source-config.md). | _required_ |\n\n:::warning\n\nWhile updating `num_pipelines` and `transform` is generally safe and reversible, updating `params` has consequences specific to the source type and might have side effects such as loosing the source's checkpoints. Perform such updates with great care. \n\n:::\n\n**Payload Example**\n\ncurl -XPOST http://localhost:7280/api/v1/indexes/my-index/sources --data @source_config.json -H \"Content-Type: application/json\"\n\n```json title=\"source_config.json\n{\n \"version\": \"0.8\",\n \"source_id\": \"kafka-source\",\n \"source_type\": \"kafka\",\n \"params\": {\n \"topic\": \"quickwit-fts-staging\",\n \"client_params\": {\n \"bootstrap.servers\": \"kafka-quickwit-server:9092\"\n }\n }\n}\n```\n\n#### Response\n\nThe response is the created source config, and the content type is `application/json; charset=UTF-8.`\n\n### Toggle source\n\n```\nPUT api/v1/indexes//sources//toggle\n```\n\nToggle (enable/disable) source `source id` of index ID `index id`.\n\nIt returns an empty body.\n\n#### PUT payload\n\n| Variable | Type | Description |\n|-------------------|----------|------------------------------------------------------------------------------------------------------|\n| `enable` | `bool` | If `true` enable the source, else disable it. |\n\n### Reset source checkpoint\n\n```\nPUT api/v1/indexes//sources//reset-checkpoint\n```\n\nResets checkpoints of source `source id` of index ID `index id`.\n\nIt returns an empty body.\n\n### Delete a source\n\n```\nDELETE api/v1/indexes//sources/\n```\n\nDelete source of ID ``.\n\n\n## Cluster API\n\nThis endpoint lets you check the state of the cluster from the point of view of the node handling the request.\n\n```\nGET api/v1/cluster?format=pretty_json\n```\n\n#### Parameters\n\nName | Type | Description | Default value\n--- | --- | --- | ---\n`format` | `String` | The output format requested for the response: `json` or `pretty_json` | `pretty_json`\n\n\n## Delete API\n\nThe delete API enables to delete documents matching a query.\n\n### Create a delete task\n\n```\nPOST api/v1//delete-tasks\n```\n\nCreate a delete task that will delete all documents matching the provided query in the given index ``.\nThe endpoint simply appends your delete task to the delete task queue in the metastore. The deletion will eventually be executed.\n\n#### Path variable\n\n| Variable | Description |\n| ------------- | ------------- |\n| `index id` | The index id |\n\n\n#### POST payload `DeleteQuery`\n\n\n| Variable | Type | Description | Default value |\n|---------------------|------------|---------------------------------------------------------------------------------------------------------|----------------------------------------------------|\n| `query` | `String` | Query text. See the [query language doc](query-language.md) | _required_ |\n| `search_field` | `[String]` | Fields to search on. Comma-separated list, e.g. \"field1,field2\" | index_config.search_settings.default_search_fields |\n| `start_timestamp` | `i64` | If set, restrict search to documents with a `timestamp >= start_timestamp`. The value must be in seconds. | |\n| `end_timestamp` | `i64` | If set, restrict search to documents with a `timestamp < end_timestamp`. The value must be in seconds. | |\n\n\n**Example**\n\n```json\n{\n \"query\": \"body:trash\",\n \"start_timestamp\": \"1669738645\",\n \"end_timestamp\": \"1669825046\",\n}\n```\n\n#### Response\n\nThe response is the created delete task represented in JSON, `DeleteTask`, the content type is `application/json; charset=UTF-8.`\n\n| Field | Description | Type |\n|----------------------|--------------------------------------------------------|:-------------:|\n| `create_timestamp` | Create timestamp of the delete query in seconds | `i64` |\n| `opstamp` | Unique operation stamp associated with the delete task | `u64` |\n| `delete_query` | The posted delete query | `DeleteQuery` |\n\n\n### List delete queries\n\n```\nGET api/v1//delete-tasks\n```\n\nGet the list of delete tasks for a given `index_id`.\n\n\n#### Response\n\nThe response is an array of `DeleteTask`.\n\n\n## Index template API\n\nThis API manages index template resources. Templates are higher level configuration objects used to automatically create indexes according to predefined rules. See [index template configuration](../configuration/template-config.md).\n\n### Create a template\n\n```\nPOST api/v1/templates\n```\n\n#### POST payload\n\nCreate an index template by posting a [template configuration](../configuration/template-config.md) payload. The API accepts JSON with the header `content-type: application/json` and YAML with `content-type: application/yaml`.\n\n**Example**\n\n```yaml\nversion: 0.9 # File format version.\n\ntemplate_id: \"all-logs\"\n\nindex_root_uri: \"s3://my-bucket/logs/\"\n\ndescription: \"All my logs\"\n\nindex_id_patterns:\n - logs-*\n\npriority: 100\n\ndoc_mapping:\n mode: dynamic\n field_mappings:\n - name: timestamp\n type: datetime\n input_formats:\n - unix_timestamp\n output_format: unix_timestamp_secs\n fast: true\n timestamp_field: timestamp\n```\n\n#### Response\n\nThe created index template configuration as JSON.\n\n\n### Update a template\n\n```\nPUT api/v1/templates/