[
  {
    "path": ".bleep",
    "content": "5a1cf681f7e2691687623b60387a88076493015f"
  },
  {
    "path": ".cargo/audit.toml",
    "content": "[advisories]\nignore = [\n    # This came from the prometheus crate's protobuf encoder.\n    # We don't use the protobuf encoder, only the text one.\n    # https://rustsec.org/advisories/RUSTSEC-2024-0437\n    \"RUSTSEC-2024-0437\",\n]\n"
  },
  {
    "path": ".cargo/config.toml",
    "content": "[resolver]\nincompatible-rust-versions = \"fallback\""
  },
  {
    "path": ".github/CONTRIBUTING.md",
    "content": "# Contributing\n\nWelcome to Pingora! Before you make a contribution, be it a bug report, documentation improvement,\npull request (PR), etc., please read and follow these guidelines.\n\n## Start with filing an issue\n\nMore often than not, **start by filing an issue on GitHub**. If you have a bug report or feature\nrequest, open a GitHub issue. Non-trivial PRs will also require a GitHub issue. The issue provides\nus with a space to discuss proposed changes with you and the community.\n\nHaving a discussion via GitHub issue upfront is the best way to ensure your contribution lands in\nPingora. We don't want you to spend your time making a PR, only to find that we won't accept it on\na design basis. For example, we may find that your proposed feature works better as a third-party\nmodule built on top of or for use with Pingora and encourage you to pursue that direction instead.\n\n**You do not need to file an issue for small fixes.** What counts as a \"small\" or trivial fix is a\njudgment call, so here's a few examples to clarify:\n- fixing a typo\n- refactoring a bit of code\n- most documentation or comment edits\n\nStill, _sometimes_ we may review your PR and ask you to file an issue if we expect there are larger\ndesign decisions to be made.\n\n## Making a PR\n\nAfter you've filed an issue, you can make your PR referencing that issue number. Once you open your\nPR, it will be labelled _Needs Review_. A maintainer will review your PR as soon as they can. The\nreviewer may ask for changes - they will mark the PR as _Changes Requested_ and will give you\ndetails about the requested changes. Feel free to ask lots of questions! The maintainers are there\nto help you.\n\nOnce we (the maintainers) decide to accept your change, we will label your PR as _Accepted_.\nLater (usually within a week or two), we will rebase your commits onto the `main` branch in a\nseparate PR, batched alongside other _Accepted_ commits and any internal changes. (This process\nallows us to sync the state of our internal repository with the public repository.) Once your\nchange lands in `main`, we will close your PR.\n\n### Caveats\n\nCurrently, internal contributions will take priority. Today Pingora is being maintained by\nCloudflare's Content Delivery team, and internal Cloudflare proxy services are a primary user of\nPingora. We value the community's work on Pingora, but the reality is that our team has a limited\namount of resources and time. We can't promise we will review or address all PRs or issues in a\ntimely manner.\n\n## Conduct\n\nPingora and Cloudflare OpenSource generally follows the [Contributor Covenant Code of Conduct].\nViolating the CoC could result in a warning or a ban to Pingora or any and all repositories in the Cloudflare organization.\n\n[Contributor Covenant Code of Conduct]: https://github.com/cloudflare/.github/blob/26b37ca2ba7ab3d91050ead9f2c0e30674d3b91e/CODE_OF_CONDUCT.md\n\n## Contact\n\nIf you have any questions, please reach out to [opensource@cloudflare.com](mailto:opensource@cloudflare.com).\n"
  },
  {
    "path": ".github/ISSUE_TEMPLATE/bug_report.md",
    "content": "---\nname: Bug Report\nabout: Report an issue to help us improve\ntitle: ''\nlabels: ''\nassignees: ''\n---\n\n## Describe the bug\n\nA clear and concise description of what the bug is.\n\n## Pingora info\n\nPlease include the following information about your environment:\n\n**Pingora version**: release number of commit hash\n**Rust version**: i.e. `cargo --version`\n**Operating system version**: e.g. Ubuntu 22.04, Debian 12.4\n\n## Steps to reproduce\n\nPlease provide step-by-step instructions to reproduce the issue. Include any relevant code\nsnippets.\n\n## Expected results\n\nWhat were you expecting to happen?\n\n## Observed results\n\nWhat actually happened?\n\n## Additional context\n\nWhat other information would you like to provide? e.g. screenshots, how you're working around the\nissue, or other clues you think could be helpful to identify the root cause.\n"
  },
  {
    "path": ".github/ISSUE_TEMPLATE/feature_request.md",
    "content": "---\nname: Feature request\nabout: Propose a new feature\ntitle: ''\nlabels: ''\nassignees: ''\n---\n\n## What is the problem your feature solves, or the need it fulfills?\n\nA clear and concise description of why this feature should be added. What is the problem? Who is\nthis for?\n\n## Describe the solution you'd like\n\nWhat do you propose to resolve the problem or fulfill the need above? How would you like it to\nwork?\n\n## Describe alternatives you've considered\n\nWhat other solutions, features, or workarounds have you considered that might also solve the issue?\nWhat are the tradeoffs for these alternatives compared to what you're proposing?\n\n## Additional context\n\nThis could include references to documentation or papers, prior art, screenshots, or benchmark\nresults.\n"
  },
  {
    "path": ".github/workflows/audit.yml",
    "content": "name: Security Audit\n\non:\n  push:\n    branches:\n      - master\n    paths:\n      - \"**/Cargo.toml\"\n  schedule:\n    - cron: \"0 2 * * *\" # run at 2 AM UTC\n\npermissions:\n  contents: read\n\njobs:\n  security-audit:\n    permissions:\n      checks: write # for rustsec/audit-check to create check\n      contents: read # for actions/checkout to fetch code\n      issues: write # for rustsec/audit-check to create issues\n    runs-on: ubuntu-latest\n    steps:\n      - uses: actions/checkout@v4\n\n      - name: Generate Cargo.lock\n        # https://github.com/rustsec/audit-check/issues/27\n        run: cargo generate-lockfile --ignore-rust-version\n\n      - name: Audit Check\n        # https://github.com/rustsec/audit-check/issues/2\n        uses: rustsec/audit-check@master\n        with:\n          token: ${{ secrets.GITHUB_TOKEN }}\n"
  },
  {
    "path": ".github/workflows/build.yml",
    "content": "on: [push, pull_request]\n\nname: build\n\njobs:\n  pingora:\n    strategy:\n      fail-fast: false\n      matrix:\n        # nightly, msrv, and latest stable\n        toolchain: [nightly, 1.84.0, 1.91.1]\n    runs-on: ubuntu-latest\n    # Only run on \"pull_request\" event for external PRs. This is to avoid\n    # duplicate builds for PRs created from internal branches.\n    if: github.event_name == 'push' || github.event.pull_request.head.repo.full_name != github.repository\n    steps:\n      - name: Checkout sources\n        uses: actions/checkout@v4\n        with:\n          submodules: \"recursive\"\n\n      - name: Install build dependencies\n        run: |\n          sudo apt update\n          sudo apt install -y cmake libclang-dev wget gnupg ca-certificates lsb-release --no-install-recommends\n          # openresty is used for convenience in tests as a server.\n          wget -O - https://openresty.org/package/pubkey.gpg | sudo gpg --dearmor -o /usr/share/keyrings/openresty.gpg\n          echo \"deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/openresty.gpg] http://openresty.org/package/ubuntu $(lsb_release -sc) main\" | sudo tee /etc/apt/sources.list.d/openresty.list > /dev/null\n          sudo apt update\n          sudo apt install -y openresty --no-install-recommends\n\n      - name: Install toolchain\n        uses: dtolnay/rust-toolchain@master\n        with:\n          toolchain: ${{ matrix.toolchain }}\n          components: rustfmt, clippy\n\n      - name: Run cargo fmt\n        run: cargo fmt --all -- --check\n\n      - name: Run cargo test\n        run: cargo test --verbose --lib --bins --tests --no-fail-fast\n\n      # Need to run doc tests separately.\n      # (https://github.com/rust-lang/cargo/issues/6669)\n      - name: Run cargo doc test\n        run: cargo test --verbose --doc\n\n      - name: Run cargo clippy\n        run: |\n          [[ ${{ matrix.toolchain }} != 1.91.1 ]] || cargo clippy --all-targets --all -- --allow=unknown-lints --deny=warnings\n\n      - name: Run cargo audit\n        run: |\n          [[ ${{ matrix.toolchain }} != 1.91.1 ]] || (cargo install --locked cargo-audit && cargo generate-lockfile --ignore-rust-version && cargo audit)\n\n      - name: Run cargo machete\n        run: |\n          [[ ${{ matrix.toolchain }} != 1.91.1 ]] || (cargo install cargo-machete --version 0.7.0 && cargo machete)\n"
  },
  {
    "path": ".github/workflows/docs.yml",
    "content": "on:\n  push:\n    branches:\n      - master\n\nname: Docs\n\njobs:\n  docs:\n    runs-on: ubuntu-latest\n    steps:\n      - name: Checkout sources\n        uses: actions/checkout@v4\n        with:\n          submodules: \"recursive\"\n\n      - name: Install build dependencies\n        run: |\n          sudo apt update\n          sudo apt install -y cmake libclang-dev\n\n      - name: Install stable toolchain\n        uses: dtolnay/rust-toolchain@stable\n\n      - name: Run cargo doc\n        run: cargo doc --no-deps --all-features\n"
  },
  {
    "path": ".github/workflows/mark-stale.yaml",
    "content": "name: 'Close stale questions'\non:\n  schedule:\n    - cron: '30 1 * * *'\n  workflow_dispatch:\n\npermissions:\n  issues: write\n  pull-requests: write\n\njobs:\n  stale:\n    runs-on: ubuntu-latest\n    steps:\n      - uses: actions/stale@v9\n        with:\n          stale-issue-message: 'This question has been stale for a week. It will be closed in an additional day if not updated.'\n          close-issue-message: 'This issue has been closed because it has been stalled with no activity.'\n          days-before-stale: -1\n          days-before-issue-stale: 7\n          days-before-issue-close: 1\n          stale-issue-label: 'stale'\n          only-issue-labels: 'question'\n"
  },
  {
    "path": ".github/workflows/semgrep.yml",
    "content": "on:\n  pull_request: {}\n  workflow_dispatch: {}\n  push: \n    branches:\n      - main\n      - master\n  schedule:\n    - cron: '0 0 * * *'\nname: Semgrep config\njobs:\n  semgrep:\n    name: semgrep/ci\n    runs-on: ubuntu-latest\n    env:\n      SEMGREP_APP_TOKEN: ${{ secrets.SEMGREP_APP_TOKEN }}\n      SEMGREP_URL: https://cloudflare.semgrep.dev\n      SEMGREP_APP_URL: https://cloudflare.semgrep.dev\n      SEMGREP_VERSION_CHECK_URL: https://cloudflare.semgrep.dev/api/check-version\n    container:\n      image: returntocorp/semgrep\n    steps:\n      - uses: actions/checkout@v4\n      - run: semgrep ci\n"
  },
  {
    "path": ".gitignore",
    "content": "Cargo.lock\n/target\n**/*.rs.bk\ndhat-heap.json\n.vscode\n.idea\n.cover\nbleeper.user.toml"
  },
  {
    "path": ".rustfmt.toml",
    "content": "edition = \"2021\"\n"
  },
  {
    "path": "CHANGELOG.md",
    "content": "# Changelog\n\nAll notable changes to this project will be documented in this file.\n\n## [0.8.0](https://github.com/cloudflare/pingora/compare/0.7.0...0.8.0) - 2026-03-02\n\n\n**🚀 Features**\n\n* Add support for client certificate verification in mTLS configuration.\n* Add upstream\\_write\\_pending\\_time to Session for upload diagnostics.\n* Pipe subrequests utility: creates a state machine to treat subrequests as a \"pipe,\" enabling direct sending of request body and writing of response tasks, with a handler for error propagation and support for reusing a preset or captured input body for chained subrequests.\n* Add the ability to limit the number of times a downstream connection can be reused\n* Add a system for specifying and using service-level dependencies\n* Add a builder for pingora proxy service, e.g. to specify ServerOptions.\n\n**🐛 Bug Fixes**\n\n* Fix various Windows compiler issues.\n* Handle custom ALPNs in s2n impl of ALPN::to\\_wire\\_protocols() to fix s2n compile issues.\n* Fix: don't use “all” permissions for socket.\n* Fix a bug with the ketama load balancing where configurations were not persisted after updates.\n* Ensure http1 downstream session is not reused on more body bytes than expected.\n* Send RST\\_STREAM CANCEL on application read timeouts for h2 client.\n* Start close-delimited body mode after 101 is received for WebSocket upgrades. `UpgradedBody` is now an explicit HttpTask.\n* Avoid close delimit mode on http/1.0 req.\n* Reject invalid content-length http/1 requests to eliminate ambiguous request framing.\n* Validate invalid content-length on http/1 resp by default, and removes content-length from the response if transfer-encoding is present, per RFC.\n* Correct the custom protocol code for shutdown: changed the numeric code passed on shutdown to 0 to indicate an explicit shutdown rather than a transport error.\n\n**⚙️ Miscellaneous Tasks**\n\n* Remove `CacheKey::default` impl, users of caching should implement `cache_key_callback` themselves\n* Allow server bootstrapping to take place in the context of services with dependents and dependencies\n* Don't consider \"bytes=\" a valid range header: added an early check for an empty/whitespace-only range-set after the `bytes=` prefix, returning 416 Range Not Satisfiable, consistent with RFC 9110 14.1.2.\n* Strip {content, transfer}-encoding from 416s to mirror the behavior for 304 Not Modified responses.\n* Disable CONNECT method proxying by default, with an option to enable via server options; unsupported requests will now be automatically rejected.\n\n## [0.7.0](https://github.com/cloudflare/pingora/compare/0.6.0...0.7.0) - 2026-01-30\n\n### Highlights\n\n- Extensible SslDigest to save user-defined TLS context\n- Add ConnectionFilter trait for early TCP connection filtering\n\n### 🚀 Features\n\n- Add ConnectionFilter trait for early TCP connection filtering\n- Introduce a virtual L4 stream abstraction\n- Add support for verify_cert and verify_hostname using rustls\n- Exposes the HttpProxy struct to allow external crates to customize the proxy logic.\n- Exposes a new_mtls method for creating a HttpProxy with a client_cert_key to enable mtls peers.\n- Add SSLKEYLOGFILE support to rustls connector\n- Allow spawning background subrequests from main session\n- Allow Extensions in cache LockCore and user tracing\n- Add body-bytes tracking across H1/H2 and proxy metrics\n- Allow setting max_weight on MissFinishType::Appended\n- Allow adding SslDigestExtensions on downstream and upstream\n- Add Custom session support for encapsulated HTTP\n\n### 🐛 Bug Fixes\n\n- Use write timeout consistently for h2 body writes\n- Prevent downstream error prior to header from canceling cache fill\n- Fix debug log and new tests\n- Fix size calculation for buffer capacity\n- Fix cache admission on header only misses\n- Fix duplicate zero-size chunk on cache hit\n- Fix chunked trailer end parsing\n- Lock age timeouts cause lock reacquisition\n- Fix transfer fd compile error for non linux os\n\n### Sec\n\n- Removed atty\n- Upgrade lru to >= 0.16.3 crate version because of RUSTSEC-2026-0002\n\n### Everything Else\n\n- Add tracing to log reason for not caching an asset on cache put\n- Evict when asset count exceeds optional watermark\n- Remove trailing comma from Display for HttpPeer\n- Make ProxyHTTP::upstream_response_body_filter return an optional duration for rate limiting\n- Restore daemonize STDOUT/STDERR when error log file is not specified\n- Log task info when upstream header failed to send\n- Check cache enablement to determine cache fill\n- Update meta when revalidating before lock release\n- Add ForceFresh status to cache hit filter\n- Pass stale status to cache lock\n- Bump max multipart ranges to 200\n- Downgrade Expires header warn to debug log\n- CI and effective msrv bump to 1.83\n- Add default noop custom param to client Session\n- Use static str in ErrorSource or ErrorType as_str\n- Use bstr for formatting byte strings\n- Tweak the implementation of and documentation of `connection_filter` feature\n- Set h1.1 when proxying cacheable responses\n- Add or remove accept-ranges on range header filter\n- Update msrv in github ci, fixup .bleep\n- Override request keepalive on process shutdown\n- Add shutdown flag to proxy session\n- Add ResponseHeader in pingora_http crate's prelude\n- Add a configurable upgrade for pingora-ketama that reduces runtime cpu and memory\n- Add to cache api spans\n- Increase visibility of multirange items\n- Use seek_multipart on body readers\n- Log read error when reading trailers end\n- Re-add the warning about cache-api volatility\n- Default to close on downstream response before body finish\n- Ensure idle_timeout is polled even if idle_timeout is unset so notify events are registered for h2 idle pool, filter out closed connections when retrieving from h2 in use pool.\n- Add simple read test for invalid extra char in header end\n- Allow customizing lock status on Custom NoCacheReasons\n- Close h1 conn by default if req header unfinished\n- Add configurable retries for upgrade sock connect/accept\n- Deflake test by increasing write size\n- Make the version restrictions on rmp and rmp-serde more strict to prevent forcing consumers to use 2024 edition\n- Rewind preread bytes when parsing next H1 response\n- Add epoch and epoch_override to CacheMeta\n\n## [0.6.0](https://github.com/cloudflare/pingora/compare/0.5.0...0.6.0) - 2025-08-15\n\n### Highlights\n- This release bumps the minimum h2 crate dependency to guard against the [MadeYouReset]((https://blog.cloudflare.com/madeyoureset-an-http-2-vulnerability-thwarted-by-rapid-reset-mitigations/)) H2 attack\n\n\n### 🚀 Features\n\n- Log runtime names during Server shutdown\n- Enabling tracking the execution phase of a server\n- Allow using in-memory compression dicts\n- Make H2Options configurable at HttpServer, HttpProxy\n  Also adds HttpServerOptions to the HttpServer implementation, and\n  updates the HttpEchoApp to use HttpServer for easier adhoc testing.\n\n### 🐛 Bug Fixes\n\n- Fix: read body without discard\n\n### Everything Else\n\n- Try loading each LRU shard individually and warn on errors\n- Update LRU save to disk to be atomic\n- Allow cache to spawn_async_purge\n- Pass hit handler in hit filter\n- Cache hit filter can mutate cache, allow resetting cache lock\n- Persist keepalive_timeout between requests on same stream\n- Properly check for H2 io ReadError retry types\n- Add cache lock wait timeout for readers\n- Fix CacheLock status timeout conditions\n- Handle close on partial chunk head\n- Allow optional to reset session timeouts\n- Clippy fixes for 1.87, add 1.87 to GitHub CI\n- Run `range_{header,body}_filter` after disabling cache\n- Convert `InterpretCacheControl` members to `Duration`\n- Disable downstream ranging on max file size\n- Allow explicit infinite keepalive timeout to be respected\n  Note that a necessary follow up is to refactor the infinite keepalive\n  timeout to only apply to first read between requests on reused conns.\n- Add method to disable keepalive if downstream is unfinished\n- Discard extra upstream body and disable keepalive\n- Explicitly disable keepalive on upstream connection when excess body\n  (content-length) is detected.\n- Add brief sleep to shutdown signal tests to avoid flake\n- Allow override of cache lock timeouts\n- Allow arbitrary bytes in CacheKey instead of just Strings\n- Corrects out-of-order data return after multiple peek calls with different buffer sizes.\n- Mark previously too large chunked assets as cacheable\n- Boring/OpenSSL load cert chain from connector options\n- Add initial support for multipart range requests\n- Adds a callback to HttpHealthCheck for collecting detailed backend summary information\n- Multipart range filter state fixes\n\n\n### Docs\n\n- Explanation of request_body_filter phase\n\n\n\n## [0.5.0](https://github.com/cloudflare/pingora/compare/0.4.0...0.5.0) - 2025-05-09\n\n### 🚀 Features\n\n- [Add tweak_new_upstream_tcp_connection hook to invoke logic on new upstream TCP sockets prior to connection](https://github.com/cloudflare/pingora/commit/be4a023d18c2b061f64ad5efd0868f9498199c91)\n- [Add ability to configure max retries for upstream proxy failures](https://github.com/cloudflare/pingora/commit/6c5d6021a6e67c971e835bef269655d0db94c2d1)\n- [Allow tcp user timeout to be configurable](https://github.com/cloudflare/pingora/commit/e77ca63da58892281f36dcb97c51a8b1e882e2f6)\n- [Add peer address to downstream handshake error logs](https://github.com/cloudflare/pingora/commit/3f9e0a2fae8feaea12a1a9687e6b4bf4616f66c5)\n- [Allow proxy to set stream level downstream read timeout](https://github.com/cloudflare/pingora/commit/87ae8ce2e7883c0a924a776b193c8a4f858b9349)\n- [Improve support for sending custom response headers and bodies for error messages](https://github.com/cloudflare/pingora/commit/a8a6e77eef2c0f4d2a45f00c5b0e316dd373f2f2)\n- [Allow configuring multiple listener tasks per endpoint](https://github.com/cloudflare/pingora/commit/69254671148938f6bc467f6decc2fc89ee7f531e)\n- [Add get_stale and get_stale_while_update for memory-cache](https://github.com/cloudflare/pingora/commit/bb28044cbe9ac9251940b8a313d970c7d15aaff6)\n\n### 🐛 Bug Fixes\n\n- [Fix deadloop if proxy_handle_upstream exits earlier than proxy_handle_downstream](https://github.com/cloudflare/pingora/commit/bb111aaa92b3753e650957df3a68f56b0cffc65d)\n- [Check on h2 stream end if error occurred for forwarding HTTP tasks](https://github.com/cloudflare/pingora/commit/e18f41bb6ddb1d6354e824df3b91d77f3255bea2)\n- [Check for content-length underflow on end of stream h2 header](https://github.com/cloudflare/pingora/commit/575d1aafd7c679a50a443701a4c55dcfdbc443b2)\n- [Correctly send empty h2 data frames prior to capacity polling](https://github.com/cloudflare/pingora/commit/c54190432a2efea30c5a0187bb7d078d33570a43)\n- [Signal that the response is done when body write finishes to avoid h1 downstream/h2 upstream errors](https://github.com/cloudflare/pingora/commit/5750e4279e75b1e764dcfc5530aa7a7cebe3abef)\n- [Ignore h2 pipe error when finishing an H2 upstream](https://github.com/cloudflare/pingora/commit/8ad15031291eb5779e0e93e714eb969c4132f632)\n- [Add finish_request_body() for HTTP healthchecks so that H2 healthchecks succeed](https://github.com/cloudflare/pingora/commit/67bc7cc170e754d335cc1d6d526f203c4345eceb)\n- [Fix Windows compile errors by updating `impl<T> UniqueID` to use correct return type](https://github.com/cloudflare/pingora/commit/1756948df77d257bddf7ab798cc3fddf348a91c8)\n- [Fixed compilation errors on Windows](https://github.com/cloudflare/pingora/commit/906cb90864bf6e441727083c9cbd4f6fb289d6f5)\n- [Poll for H2 capacity before sending H2 body to propagate backpressure](https://github.com/cloudflare/pingora/commit/b6f24ff3725d9d8b6a740d87cad959d94befbe54)\n- [Fix for write_error_response for http2 downstreams to set EOS](https://github.com/cloudflare/pingora/commit/c0fa5065812d87e6e404c5624b26cd99c5194079)\n- [Always drain v1 request body before session reuse](https://github.com/cloudflare/pingora/commit/fda3317ec822678564d641e7cf1c9b77ee3759ff)\n- [Fixes HTTP1 client reads to properly timeout on initial read](https://github.com/cloudflare/pingora/commit/3c7db34acb0d930ae7043290a88bc56c1cd77e45)\n- [Fixes issue where if TLS client never sends any bytes, hangs forever](https://github.com/cloudflare/pingora/commit/d1bf0bcac98f943fd716278d674e7d10dce2223e)\n\n### Everything Else\n\n- [Add builder api for pingora listeners](https://github.com/cloudflare/pingora/commit/3f564af3ae56e898478e13e71d67d095d7f5dbbd)\n- [Better handling for h1 requests that contain both transfer-encoding and content-length](https://github.com/cloudflare/pingora/commit/9287b82645be4a52b0b63530ba38aa0c7ddc4b77)\n- [Allow setting raw path in request to support non-UTF8 use cases](https://github.com/cloudflare/pingora/commit/e6b823c5d89860bb97713fdf14f197f799aed6af)\n- [Allow reusing session on errors prior to proxy upstream](https://github.com/cloudflare/pingora/commit/f8d01278a586c60392b1e3b92e5ed97a415d8fe7)\n- [Avoid allocating large buffer in the accept() loop](https://github.com/cloudflare/pingora/commit/ef234f5baa45650be064c7dd34c2f17986361480)\n- [Ensure HTTP/1.1 when forcing chunked encoding](https://github.com/cloudflare/pingora/commit/9281cab8eab1b545f15f0e387d2ba4cd2ca27364)\n- [Reject if the HTTP header contains duplicated Content-Length values](https://github.com/cloudflare/pingora/commit/eef35768d11305d1293468a6c3ce91a3858dc0fc)\n- [proxy_upstream_filter tries to reuse downstream by default](https://github.com/cloudflare/pingora/commit/86293e65b5c7d8a96f3a333a1f191766dc95bee5)\n- [Allow building server that avoids std::process::exit during shutdown](https://github.com/cloudflare/pingora/commit/2d977d4eb808d8bcbc0ce87cabac4cf4854dfb80)\n- [Update Sentry crate to 0.36](https://github.com/cloudflare/pingora/commit/01a1f9a65c51a4351c29d6961ea3164a6a811958)\n- [Update the bounds on `MemoryCache` methods to accept broader key types](https://github.com/cloudflare/pingora/commit/d66923a9a41d00b326cef5dfb57d8c020d6a4abb)\n- [Flush already received data if upstream write errors](https://github.com/cloudflare/pingora/commit/aa7c2f1a89a652137a987e5f5dbdab228c2f4d06)\n- [Allow modules to receive HttpTask::Done, flush response compression on receiving Done task](https://github.com/cloudflare/pingora/commit/c82fb6ba57b95c256b58095881a33a9bc08f170a)\n- API signature changes as part of experimental proxy cache support\n- Note MSRV was effectively bumped to 1.82 from 1.72 due to a dependency update, though older compilers may still be able to build by pinning dependencies, e.g. `cargo update -p backtrace --precise 0.3.74`.\n\n## [0.4.0](https://github.com/cloudflare/pingora/compare/0.3.0...0.4.0) - 2024-11-01\n\n### 🚀 Features\n- [Add preliminary rustls support](https://github.com/cloudflare/pingora/commit/354a6ee1e99b82e23fc0f27a37d8bf41e62b2dc5)\n- [Add experimental support for windows](https://github.com/cloudflare/pingora/commit/4aadba12727afe6178f3b9fc2a3cad2223ac7b2e)\n- [Add the option to use no TLS implementation](https://github.com/cloudflare/pingora/commit/d8f3ffae77ddc1edd285ab1d517a1b6748ce3d58)\n- [Add support for gRPC-web module to bridge gRPC-web client requests to gRPC server requests](https://github.com/cloudflare/pingora/commit/9917177c646a0ab58197f15ec57a3bcbe1e0a201)\n- [Add the support for h2c and http1 to coexist](https://github.com/cloudflare/pingora/commit/792d5fd3c14c1cd588b155ddf09c09a4c125a26b)\n- [Add the support for custom L4 connector](https://github.com/cloudflare/pingora/commit/7c122e7f36de5c946ac960a1691c5dd41f26e6e6)\n- [Support opaque extension field in Backend](https://github.com/cloudflare/pingora/commit/999e379064d2c1266a267abdf9f4f41b14bffcf5)\n- [Add the ability to ignore informational responses when proxying downstream](https://github.com/cloudflare/pingora/commit/be97e35031cf4f5a01191f1848bdf491bd9f0d62)\n- [Add un-gzip support and allow decompress by algorithm](https://github.com/cloudflare/pingora/commit/e1c6e57db3e613991eda3160d15f81e0669ea066)\n- [Add the ability to observe backend health status](https://github.com/cloudflare/pingora/commit/8a0c73f174a27a87c54426a748c4818b10de9425)\n- [Add the support for passing sentry release](https://github.com/cloudflare/pingora/commit/07a970e413009ee62fc4c15e0820ae1aa036af22)\n- [Add the support for binding to local port ranges](https://github.com/cloudflare/pingora/commit/d1d7a87b761eeb4f71fcaa3f7c4ae8e32f1d93c8)\n- [Support retrieving rx timestamp for TcpStream](https://github.com/cloudflare/pingora/commit/d811795938cee5a6eb7cd46399cef17210a0d0c5)\n\n### 🐛 Bug Fixes\n- [Handle bare IPv6 address in raw connect Host](https://github.com/cloudflare/pingora/commit/9f50e6ccb09db2940eec6fc170a1e9e9b14a95d0)\n- [Set proper response headers when compression is enabled](https://github.com/cloudflare/pingora/commit/55049c4e7983055551b34feee397c736ffc912bb)\n- [Check the current advertised h2 max streams](https://github.com/cloudflare/pingora/commit/7419b1967e7686b00aefb7bcd2a4dfe59b31e639)\n- Other bug fixes and improvements\n\n\n### ⚙️ Changes and Miscellaneous Tasks\n- [Make sentry an optional feature](https://github.com/cloudflare/pingora/commit/ab1b717bf587723c1c537d6549a8f8096f0900d4)\n- [Make timeouts Sync](https://github.com/cloudflare/pingora/commit/18db42cd2cb892432fd7896f0da7e9d19221214b)\n- [Retry all h2 connection when encountering graceful shutdown](https://github.com/cloudflare/pingora/commit/11b5882a422774cffbd14d9a9ea7dfc9dc98b02c)\n- [Make l4 module pub to expose Connect](https://github.com/cloudflare/pingora/commit/91702bb0c0c5e1f2d5e2f40a19a3f340bb5a6d82)\n- [Auto snake case set-cookie header when downgrade to from h2 to http1.1](https://github.com/cloudflare/pingora/commit/2c6190c634f2a5dd2f00e8597902f2b735a9d84f)\n- [shutdown h2 connection gracefully with GOAWAYs](https://github.com/cloudflare/pingora/commit/04d7cfeef6205d2cf33ad5704a363ee107250771)\n- Other API signature updates\n\n## [0.3.0](https://github.com/cloudflare/pingora/compare/0.2.0...0.3.0) - 2024-07-12\n\n### 🚀 Features\n- Add support for HTTP modules. This feature allows users to import modules written by 3rd parties.\n- Add `request_body_filter`. Now request body can be inspected and modified.\n- Add H2c support.\n- Add TCP fast open support.\n- Add support for server side TCP keep-alive.\n- Add support to get TCP_INFO.\n- Add support to set DSCP.\n- Add `or_err()`/`or_err_with` API to convert `Options` to `pingora::Error`.\n- Add `or_fail()` API to convert `impl std::error::Error` to `pingora::Error`.\n- Add the API to track socket read and write pending time.\n- Compression: allow setting level per algorithm.\n\n### 🐛 Bug Fixes\n- Fixed a panic when using multiple H2 streams in the same H2 connection to upstreams.\n- Pingora now respects the `Connection` header it sends to upstream.\n- Accept-Ranges header is now removed when response is compressed.\n- Fix ipv6_only socket flag.\n- A new H2 connection is opened now if the existing connection returns GOAWAY with graceful shutdown error.\n- Fix a FD mismatch error when 0.0.0.0 is used as the upstream IP\n\n### ⚙️ Changes and Miscellaneous Tasks\n- Dependency: replace `structopt` with `clap`\n- Rework the API of HTTP modules\n- Optimize remove_header() API call\n- UDS parsing now requires the path to have `unix:` prefix. The support for the path without prefix is deprecated and will be removed on the next release.\n- Other minor API changes\n\n## [0.2.0](https://github.com/cloudflare/pingora/compare/0.1.1...0.2.0) - 2024-05-10\n\n### 🚀 Features\n- Add support for downstream h2 trailers and add an upstream h2 response trailer filter\n- Add the ability to set TCP recv buf size\n- Add a convenience function to retrieve Session digest\n- Add `body_bytes_read()` method to Session\n- Add `cache_not_modified_filter`\n- Add `SSLKEYLOG` support for tls upstream\n- Add `Service<HttpProxy<T>>` constructor for providing name\n- Add `purge_response` callback\n- Make `pop_closed` pub, to simplify DIY drains\n\n### 🐛 Bug Fixes\n- Fixed gRPC trailer proxying\n- Fixed `response_body_filter` `end_of_stream` always being false\n- Fixed compile error in Rust <= 1.73\n- Fixed non linux build\n- Fixed the counting problem of used_weight data field in `LruUnit<T>`\n- Fixed `cargo run --example server` missing cert\n- Fixed error log string interpolation outside of proper context\n- Fixed tinylfu test flake\n\n### ⚙️ Changes and Miscellaneous Tasks\n- API change: `Server::run_forever` now takes ownership and ensures exit semantics\n- API change: `cleanup()` method of `ServerApp` trait is now async\n- Behavior change: Always return `HttpTask::Body` on body done instead of `HttpTask::done`\n- Behavior change: HTTP/1 reason phrase is now parsed and proxied\n- Updated `h2` dependency for RUSTSEC-2024-0332\n- Updated zstd dependencies\n- Code optimization and refactor in a few crates\n- More examples and docs\n\n## [0.1.1](https://github.com/cloudflare/pingora/compare/0.1.0...0.1.1) - 2024-04-05\n\n### 🚀 Features\n- `Server::new` now accepts `Into<Option<T>>`\n- Implemented client `HttpSession::get_keepalive_values` for Keep-Alive parsing\n- Expose `ListenFds` and `Fds` to fix a voldemort types issue\n- Expose config options in `ServerConf`, provide new `Server` constructor\n- `upstream_response_filter` now runs on upstream 304 responses during cache revalidation\n- Added `server_addr` and `client_addr` APIs to `Session`\n- Allow body modification in `response_body_filter`\n- Allow configuring grace period and graceful shutdown timeout\n- Added TinyUFO sharded skip list storage option\n\n### 🐛 Bug Fixes\n- Fixed build failures with the `boringssl` feature\n- Fixed compile warnings with nightly Rust\n- Fixed an issue where Upgrade request bodies might not be handled correctly\n- Fix compilation to only include openssl or boringssl rather than both\n- Fix OS read errors so they are reported as `ReadError` rather than `ReadTimeout` when reading http/1.1 response headers\n\n### ⚙️ Miscellaneous Tasks\n- Performance improvements in `pingora-ketama`\n- Added more TinyUFO benchmarks\n- Added tests for `pingora-cache` purge\n- Limit buffer size for `InvalidHTTPHeader` error logs\n- Example code: improvements in pingora client, new LB cluster example\n- Typo fixes and clarifications across comments and docs\n\n## [0.1.0] - 2024-02-28\n### Highlights\n- First Public Release of Pingora 🎉\n"
  },
  {
    "path": "Cargo.toml",
    "content": "\n\n\n\n\n[workspace]\nresolver = \"2\"\nmembers = [\n    \"pingora\",\n    \"pingora-core\",\n    \"pingora-pool\",\n    \"pingora-error\",\n    \"pingora-limits\",\n    \"pingora-timeout\",\n    \"pingora-header-serde\",\n    \"pingora-proxy\",\n    \"pingora-cache\",\n    \"pingora-http\",\n    \"pingora-lru\",\n    \"pingora-openssl\",\n    \"pingora-boringssl\",\n    \"pingora-runtime\",\n    \"pingora-rustls\",\n    \"pingora-s2n\",\n    \"pingora-ketama\",\n    \"pingora-load-balancing\",\n    \"pingora-memory-cache\",\n    \"tinyufo\",\n]\n\n[workspace.dependencies]\nbstr = \"1.12.0\"\ntokio = \"1\"\ntokio-stream = { version = \"0.1\" }\nasync-trait = \"0.1.42\"\nhttparse = \"1\"\nbytes = \"1.0\"\nderivative = \"2.2.0\"\nhttp = \"1\"\nlog = \"0.4\"\nh2 = \">=0.4.11\"\nonce_cell = \"1\"\nlru = \"0.16.3\"\nahash = \">=0.8.9\"\n\n[profile.bench]\ndebug = true\n"
  },
  {
    "path": "Dockerfile",
    "content": "FROM debian:latest as builder\n\nARG BUILDARCH\nRUN apt-get -qq update \\\n    && apt-get -qq install -y --no-install-recommends \\\n       gcc g++ libfindbin-libs-perl \\\n       make cmake libclang-dev git \\\n       wget curl gnupg ca-certificates lsb-release \\\n    && wget --no-check-certificate -O - https://openresty.org/package/pubkey.gpg | gpg --dearmor -o /usr/share/keyrings/openresty.gpg \\\n    && if [ \"${BUILDARCH}\" = \"arm64\" ]; then URL=\"http://openresty.org/package/arm64/debian\"; else URL=\"http://openresty.org/package/debian\"; fi \\\n    && echo \"deb [arch=$BUILDARCH signed-by=/usr/share/keyrings/openresty.gpg] ${URL} $(lsb_release -sc) openresty\" | tee /etc/apt/sources.list.d/openresty.list > /dev/null \\\n    && apt-get -qq update \\\n    && apt-get -qq install -y openresty --no-install-recommends\n\nRUN curl https://sh.rustup.rs -sSf | sh -s -- -y\nENV PATH=\"/root/.cargo/bin:${PATH}\"\n\nWORKDIR /var/opt/pingora\nCOPY . .\nRUN cargo build\n"
  },
  {
    "path": "LICENSE",
    "content": "\n                                 Apache License\n                           Version 2.0, January 2004\n                        http://www.apache.org/licenses/\n\n   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION\n\n   1. Definitions.\n\n      \"License\" shall mean the terms and conditions for use, reproduction,\n      and distribution as defined by Sections 1 through 9 of this document.\n\n      \"Licensor\" shall mean the copyright owner or entity authorized by\n      the copyright owner that is granting the License.\n\n      \"Legal Entity\" shall mean the union of the acting entity and all\n      other entities that control, are controlled by, or are under common\n      control with that entity. For the purposes of this definition,\n      \"control\" means (i) the power, direct or indirect, to cause the\n      direction or management of such entity, whether by contract or\n      otherwise, or (ii) ownership of fifty percent (50%) or more of the\n      outstanding shares, or (iii) beneficial ownership of such entity.\n\n      \"You\" (or \"Your\") shall mean an individual or Legal Entity\n      exercising permissions granted by this License.\n\n      \"Source\" form shall mean the preferred form for making modifications,\n      including but not limited to software source code, documentation\n      source, and configuration files.\n\n      \"Object\" form shall mean any form resulting from mechanical\n      transformation or translation of a Source form, including but\n      not limited to compiled object code, generated documentation,\n      and conversions to other media types.\n\n      \"Work\" shall mean the work of authorship, whether in Source or\n      Object form, made available under the License, as indicated by a\n      copyright notice that is included in or attached to the work\n      (an example is provided in the Appendix below).\n\n      \"Derivative Works\" shall mean any work, whether in Source or Object\n      form, that is based on (or derived from) the Work and for which the\n      editorial revisions, annotations, elaborations, or other modifications\n      represent, as a whole, an original work of authorship. For the purposes\n      of this License, Derivative Works shall not include works that remain\n      separable from, or merely link (or bind by name) to the interfaces of,\n      the Work and Derivative Works thereof.\n\n      \"Contribution\" shall mean any work of authorship, including\n      the original version of the Work and any modifications or additions\n      to that Work or Derivative Works thereof, that is intentionally\n      submitted to Licensor for inclusion in the Work by the copyright owner\n      or by an individual or Legal Entity authorized to submit on behalf of\n      the copyright owner. For the purposes of this definition, \"submitted\"\n      means any form of electronic, verbal, or written communication sent\n      to the Licensor or its representatives, including but not limited to\n      communication on electronic mailing lists, source code control systems,\n      and issue tracking systems that are managed by, or on behalf of, the\n      Licensor for the purpose of discussing and improving the Work, but\n      excluding communication that is conspicuously marked or otherwise\n      designated in writing by the copyright owner as \"Not a Contribution.\"\n\n      \"Contributor\" shall mean Licensor and any individual or Legal Entity\n      on behalf of whom a Contribution has been received by Licensor and\n      subsequently incorporated within the Work.\n\n   2. Grant of Copyright License. Subject to the terms and conditions of\n      this License, each Contributor hereby grants to You a perpetual,\n      worldwide, non-exclusive, no-charge, royalty-free, irrevocable\n      copyright license to reproduce, prepare Derivative Works of,\n      publicly display, publicly perform, sublicense, and distribute the\n      Work and such Derivative Works in Source or Object form.\n\n   3. Grant of Patent License. Subject to the terms and conditions of\n      this License, each Contributor hereby grants to You a perpetual,\n      worldwide, non-exclusive, no-charge, royalty-free, irrevocable\n      (except as stated in this section) patent license to make, have made,\n      use, offer to sell, sell, import, and otherwise transfer the Work,\n      where such license applies only to those patent claims licensable\n      by such Contributor that are necessarily infringed by their\n      Contribution(s) alone or by combination of their Contribution(s)\n      with the Work to which such Contribution(s) was submitted. If You\n      institute patent litigation against any entity (including a\n      cross-claim or counterclaim in a lawsuit) alleging that the Work\n      or a Contribution incorporated within the Work constitutes direct\n      or contributory patent infringement, then any patent licenses\n      granted to You under this License for that Work shall terminate\n      as of the date such litigation is filed.\n\n   4. Redistribution. You may reproduce and distribute copies of the\n      Work or Derivative Works thereof in any medium, with or without\n      modifications, and in Source or Object form, provided that You\n      meet the following conditions:\n\n      (a) You must give any other recipients of the Work or\n          Derivative Works a copy of this License; and\n\n      (b) You must cause any modified files to carry prominent notices\n          stating that You changed the files; and\n\n      (c) You must retain, in the Source form of any Derivative Works\n          that You distribute, all copyright, patent, trademark, and\n          attribution notices from the Source form of the Work,\n          excluding those notices that do not pertain to any part of\n          the Derivative Works; and\n\n      (d) If the Work includes a \"NOTICE\" text file as part of its\n          distribution, then any Derivative Works that You distribute must\n          include a readable copy of the attribution notices contained\n          within such NOTICE file, excluding those notices that do not\n          pertain to any part of the Derivative Works, in at least one\n          of the following places: within a NOTICE text file distributed\n          as part of the Derivative Works; within the Source form or\n          documentation, if provided along with the Derivative Works; or,\n          within a display generated by the Derivative Works, if and\n          wherever such third-party notices normally appear. The contents\n          of the NOTICE file are for informational purposes only and\n          do not modify the License. You may add Your own attribution\n          notices within Derivative Works that You distribute, alongside\n          or as an addendum to the NOTICE text from the Work, provided\n          that such additional attribution notices cannot be construed\n          as modifying the License.\n\n      You may add Your own copyright statement to Your modifications and\n      may provide additional or different license terms and conditions\n      for use, reproduction, or distribution of Your modifications, or\n      for any such Derivative Works as a whole, provided Your use,\n      reproduction, and distribution of the Work otherwise complies with\n      the conditions stated in this License.\n\n   5. Submission of Contributions. Unless You explicitly state otherwise,\n      any Contribution intentionally submitted for inclusion in the Work\n      by You to the Licensor shall be under the terms and conditions of\n      this License, without any additional terms or conditions.\n      Notwithstanding the above, nothing herein shall supersede or modify\n      the terms of any separate license agreement you may have executed\n      with Licensor regarding such Contributions.\n\n   6. Trademarks. This License does not grant permission to use the trade\n      names, trademarks, service marks, or product names of the Licensor,\n      except as required for reasonable and customary use in describing the\n      origin of the Work and reproducing the content of the NOTICE file.\n\n   7. Disclaimer of Warranty. Unless required by applicable law or\n      agreed to in writing, Licensor provides the Work (and each\n      Contributor provides its Contributions) on an \"AS IS\" BASIS,\n      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or\n      implied, including, without limitation, any warranties or conditions\n      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A\n      PARTICULAR PURPOSE. You are solely responsible for determining the\n      appropriateness of using or redistributing the Work and assume any\n      risks associated with Your exercise of permissions under this License.\n\n   8. Limitation of Liability. In no event and under no legal theory,\n      whether in tort (including negligence), contract, or otherwise,\n      unless required by applicable law (such as deliberate and grossly\n      negligent acts) or agreed to in writing, shall any Contributor be\n      liable to You for damages, including any direct, indirect, special,\n      incidental, or consequential damages of any character arising as a\n      result of this License or out of the use or inability to use the\n      Work (including but not limited to damages for loss of goodwill,\n      work stoppage, computer failure or malfunction, or any and all\n      other commercial damages or losses), even if such Contributor\n      has been advised of the possibility of such damages.\n\n   9. Accepting Warranty or Additional Liability. While redistributing\n      the Work or Derivative Works thereof, You may choose to offer,\n      and charge a fee for, acceptance of support, warranty, indemnity,\n      or other liability obligations and/or rights consistent with this\n      License. However, in accepting such obligations, You may act only\n      on Your own behalf and on Your sole responsibility, not on behalf\n      of any other Contributor, and only if You agree to indemnify,\n      defend, and hold each Contributor harmless for any liability\n      incurred by, or claims asserted against, such Contributor by reason\n      of your accepting any such warranty or additional liability.\n\n   END OF TERMS AND CONDITIONS\n\n   APPENDIX: How to apply the Apache License to your work.\n\n      To apply the Apache License to your work, attach the following\n      boilerplate notice, with the fields enclosed by brackets \"[]\"\n      replaced with your own identifying information. (Don't include\n      the brackets!)  The text should be enclosed in the appropriate\n      comment syntax for the file format. We also recommend that a\n      file or class name and description of purpose be included on the\n      same \"printed page\" as the copyright notice for easier\n      identification within third-party archives.\n\n   Copyright [yyyy] [name of copyright owner]\n\n   Licensed under the Apache License, Version 2.0 (the \"License\");\n   you may not use this file except in compliance with the License.\n   You may obtain a copy of the License at\n\n       http://www.apache.org/licenses/LICENSE-2.0\n\n   Unless required by applicable law or agreed to in writing, software\n   distributed under the License is distributed on an \"AS IS\" BASIS,\n   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n   See the License for the specific language governing permissions and\n   limitations under the License.\n"
  },
  {
    "path": "README.md",
    "content": "# Pingora\n\n![Pingora banner image](./docs/assets/pingora_banner.png)\n\n## What is Pingora\nPingora is a Rust framework to [build fast, reliable and programmable networked systems](https://blog.cloudflare.com/pingora-open-source).\n\nPingora is battle tested as it has been serving more than 40 million Internet requests per second for [more than a few years](https://blog.cloudflare.com/how-we-built-pingora-the-proxy-that-connects-cloudflare-to-the-internet).\n\n## Feature highlights\n* Async Rust: fast and reliable\n* HTTP 1/2 end to end proxy\n* TLS over OpenSSL, BoringSSL, s2n-tls, or rustls(experimental).\n* gRPC and websocket proxying\n* Graceful reload\n* Customizable load balancing and failover strategies\n* Support for a variety of observability tools\n\n## Reasons to use Pingora\n* **Security** is your top priority: Pingora is a more memory safe alternative for services that are written in C/C++\n* Your service is **performance-sensitive**: Pingora is fast and efficient\n* Your service requires extensive **customization**: The APIs Pingora proxy framework provides are highly programmable\n\n# Getting started\n\nSee our [quick starting guide](./docs/quick_start.md) to see how easy it is to build a load balancer.\n\nOur [user guide](./docs/user_guide/index.md) covers more topics such as how to configure and run Pingora servers, as well as how to build custom HTTP servers and proxy logic on top of Pingora's framework.\n\nAPI docs are also available for all the crates.\n\n# Notable crates in this workspace\n* Pingora: the \"public facing\" crate to build networked systems and proxies\n* Pingora-core: this crate defines the protocols, functionalities and basic traits\n* Pingora-proxy: the logic and APIs to build HTTP proxies\n* Pingora-error: the common error type used across Pingora crates\n* Pingora-http: the HTTP header definitions and APIs\n* Pingora-openssl & pingora-boringssl: SSL related extensions and APIs\n* Pingora-ketama: the [Ketama](https://github.com/RJ/ketama) consistent algorithm\n* Pingora-limits: efficient counting algorithms\n* Pingora-load-balancing: load balancing algorithm extensions for pingora-proxy\n* Pingora-memory-cache: Async in-memory caching with cache lock to prevent cache stampede\n* Pingora-s2n: SSL extensions and APIs related to s2n-tls\n* Pingora-timeout: A more efficient async timer system\n* TinyUfo: The caching algorithm behind pingora-memory-cache\n\nNote that Pingora proxy integration with caching should be considered experimental, and as such APIs related to caching are currently highly volatile.\n\n# System requirements\n\n## Systems\nLinux is our tier 1 environment and main focus.\n\nWe will try our best for most code to compile for Unix environments. This is for developers and users to have an easier time developing with Pingora in Unix-like environments like macOS (though some features might be missing)\n\nWindows support is preliminary by community's best effort only.\n\nBoth x86_64 and aarch64 architectures will be supported.\n\n## Rust version\n\nPingora keeps a rolling MSRV (minimum supported Rust version) policy of 6 months. This means we will accept PRs that upgrade the MSRV as long as the new Rust version used is at least 6 months old. However, we generally will not bump the highest MSRV across the workspace without a sufficiently compelling reason.\n\nOur current MSRV is 1.84.\n\nCurrently not all crates enforce `rust-version` as it is possible to use some crates on lower versions.\n\n## Build Requirements\n\nSome of the crates in this repository have dependencies on additional tools and\nlibraries that must be satisfied in order to build them:\n\n* Make sure that [Clang] is installed on your system (for boringssl)\n* Make sure that [Perl 5] is installed on your system (for openssl)\n\n[Clang]:https://clang.llvm.org/\n[Perl 5]:https://www.perl.org/\n\n# Contributing\nPlease see our [contribution guidelines](./.github/CONTRIBUTING.md).\n\n# License\nThis project is Licensed under [Apache License, Version 2.0](./LICENSE).\n"
  },
  {
    "path": "cliff.toml",
    "content": "# git-cliff ~ default configuration file\n# https://git-cliff.org/docs/configuration\n#\n# Lines starting with \"#\" are comments.\n# Configuration options are organized into tables and keys.\n# See documentation for more information on available options.\n\n[changelog]\n# changelog header\nheader = \"\"\"\n# Changelog\\n\nAll notable changes to this project will be documented in this file.\\n\n\"\"\"\n# template for the changelog body\n# https://keats.github.io/tera/docs/#introduction\nbody = \"\"\"\n{% if version %}\\\n  {% if previous.version %}\\\n    ## [{{ version | trim_start_matches(pat=\"v\") }}](https://github.com/cloudflare/pingora/compare/{{ previous.version }}...{{ version }}) - {{ timestamp | date(format=\"%Y-%m-%d\") }}\n  {% else %}\\\n    ## [{{ version | trim_start_matches(pat=\"v\") }}] - {{ timestamp | date(format=\"%Y-%m-%d\") }}\n  {% endif %}\\\n{% else %}\\\n    ## [unreleased]\n{% endif %}\\\n\n### Highlights\n  - Human-written change summaries go here\n\n{% for group, commits in commits | group_by(attribute=\"group\") %}\n    ### {{ group | striptags | trim | upper_first }}\n    {% for commit in commits %}\n        - {% if commit.scope %}*({{ commit.scope }})* {% endif %}\\\n            {% if commit.breaking %}[**breaking**] {% endif %}\\\n            {{ commit.message | upper_first }}\\\n    {% endfor %}\n{% endfor %}\\n\n\"\"\"\n# template for the changelog footer\nfooter = \"\"\"\n\"\"\"\n# remove the leading and trailing whitespace\ntrim = true\n\n[git]\n# parse the commits based on https://www.conventionalcommits.org\nconventional_commits = true\n\n# filter out the commits that are not conventional\nfilter_unconventional = false\n\n# process each line of a commit as an individual commit\nsplit_commits = false\n\n# regex for preprocessing the commit messages\ncommit_preprocessors = [\n  { pattern = '\\n\\w+(?:\\-\\w+)*:\\s+[^\\n]+', replace = \"\\n\" },\n  { pattern = '\\n+', replace = \"\\n  \" },\n  { pattern = '\\s+$', replace = \"\" }\n]\n\n# regex for parsing and grouping commits\ncommit_parsers = [\n  { message = \"^feat\", group = \"<!-- 0 -->🚀 Features\" },\n  { message = \"^fix\", group = \"<!-- 1 -->🐛 Bug Fixes\" },\n  { message = \"^doc\", group = \"<!-- 3 -->📚 Documentation\", skip = true  },\n  { message = \"^perf\", group = \"<!-- 4 -->⚡ Performance\" },\n  { message = \"^refactor\", group = \"<!-- 2 -->🚜 Refactor\", skip = true  },\n  { message = \"^style\", group = \"<!-- 5 -->🎨 Styling\", skip = true  },\n  { message = \"^test\", group = \"<!-- 6 -->🧪 Testing\", skip = true  },\n  { message = \"^chore\\\\(release\\\\): prepare for\", skip = true },\n  { message = \"^chore\\\\(deps.*\\\\)\", skip = true },\n  { message = \"^chore\\\\(pr\\\\)\", skip = true },\n  { message = \"^chore\\\\(pull\\\\)\", skip = true },\n  { message = \"^chore|^ci\", group = \"<!-- 7 -->⚙️ Miscellaneous Tasks\" },\n  { body = \".*security\", group = \"<!-- 8 -->🛡️ Security\" },\n  { message = \"^revert\", group = \"<!-- 9 -->◀️ Revert\" },\n  { message = '\\S+(?:\\s+\\S+){6,}', group = \"<!--10--> Everything Else\" }\n]\n\n# protect breaking changes from being skipped due to matching a skipping commit_parser\nprotect_breaking_commits = false\n\n# filter out the commits that are not matched by commit parsers\nfilter_commits = false\ntag_pattern = \"[0-9].[0-9].[0-9]\"\ntopo_order = false"
  },
  {
    "path": "clippy.toml",
    "content": "msrv = \"1.84\"\n"
  },
  {
    "path": "docs/README.md",
    "content": "# Pingora User Manual\n\n## Quick Start\nIn this section we show you how to build a bare-bones load balancer.\n\n[Read the quick start here.](quick_start.md)\n\n## User Guide\nCovers how to configure and run Pingora servers, as well as how to build custom HTTP server and proxy logic on top of Pingora's framework.\n\n[Read the user guide here.](user_guide/index.md)\n\n## API Reference\nTBD\n"
  },
  {
    "path": "docs/quick_start.md",
    "content": "# Quick Start: load balancer\n\n## Introduction\n\nThis quick start shows how to build a bare-bones load balancer using pingora and pingora-proxy.\n\nThe goal of the load balancer is for every incoming HTTP request, select one of the two backends: https://1.1.1.1 and https://1.0.0.1 in a round-robin fashion.\n\n## Build a basic load balancer\n\nCreate a new cargo project for our load balancer. Let's call it `load_balancer`\n\n```\ncargo new load_balancer\n```\n\n### Include the Pingora Crate and Basic Dependencies\n\nIn your project's `cargo.toml` file add the following to your dependencies\n```\nasync-trait=\"0.1\"\npingora = { version = \"0.3\", features = [ \"lb\" ] }\n```\n\n### Create a pingora server\nFirst, let's create a pingora server. A pingora `Server` is a process which can host one or many\nservices. The pingora `Server` takes care of configuration and CLI argument parsing, daemonization,\nsignal handling, and graceful restart or shutdown.\n\nThe preferred usage is to initialize the `Server` in the `main()` function and\nuse `run_forever()` to spawn all the runtime threads and block the main thread until the server is\nready to exit.\n\n\n```rust\nuse async_trait::async_trait;\nuse pingora::prelude::*;\nuse std::sync::Arc;\n\nfn main() {\n    let mut my_server = Server::new(None).unwrap();\n    my_server.bootstrap();\n    my_server.run_forever();\n}\n```\n\nThis will compile and run, but it doesn't do anything interesting.\n\n### Create a load balancer proxy\nNext let's create a load balancer. Our load balancer holds a static list of upstream IPs. The `pingora-load-balancing` crate already provides the `LoadBalancer` struct with common selection algorithms such as round robin and hashing. So let’s just use it. If the use case requires more sophisticated or customized server selection logic, users can simply implement it themselves in this function.\n\n\n```rust\npub struct LB(Arc<LoadBalancer<RoundRobin>>);\n```\n\nIn order to make the server a proxy, we need to implement the `ProxyHttp` trait for it.\n\nAny object that implements the `ProxyHttp` trait essentially defines how a request is handled in\nthe proxy. The only required method in the `ProxyHttp` trait is `upstream_peer()` which returns\nthe address where the request should be proxied to.\n\nIn the body of the `upstream_peer()`, let's use the `select()` method for the `LoadBalancer` to round-robin across the upstream IPs. In this example we use HTTPS to connect to the backends, so we also need to specify to `use_tls` and set the SNI when constructing our [`Peer`](user_guide/peer.md)) object.\n\n```rust\n#[async_trait]\nimpl ProxyHttp for LB {\n\n    /// For this small example, we don't need context storage\n    type CTX = ();\n    fn new_ctx(&self) -> () {\n        ()\n    }\n\n    async fn upstream_peer(&self, _session: &mut Session, _ctx: &mut ()) -> Result<Box<HttpPeer>> {\n        let upstream = self.0\n            .select(b\"\", 256) // hash doesn't matter for round robin\n            .unwrap();\n\n        println!(\"upstream peer is: {upstream:?}\");\n\n        // Set SNI to one.one.one.one\n        let peer = Box::new(HttpPeer::new(upstream, true, \"one.one.one.one\".to_string()));\n        Ok(peer)\n    }\n}\n```\n\nIn order for the 1.1.1.1 backends to accept our requests, a host header must be present. Adding this header\ncan be done by the `upstream_request_filter()` callback which modifies the request header after\nthe connection to the backends are established and before the request header is sent.\n\n```rust\nimpl ProxyHttp for LB {\n    // ...\n    async fn upstream_request_filter(\n        &self,\n        _session: &mut Session,\n        upstream_request: &mut RequestHeader,\n        _ctx: &mut Self::CTX,\n    ) -> Result<()> {\n        upstream_request.insert_header(\"Host\", \"one.one.one.one\").unwrap();\n        Ok(())\n    }\n}\n```\n\n\n### Create a pingora-proxy service\nNext, let's create a proxy service that follows the instructions of the load balancer above.\n\nA pingora `Service` listens to one or multiple (TCP or Unix domain socket) endpoints. When a new connection is established\nthe `Service` hands the connection over to its \"application.\" `pingora-proxy` is such an application\nwhich proxies the HTTP request to the given backend as configured above.\n\nIn the example below, we create a `LB` instance with two backends `1.1.1.1:443` and `1.0.0.1:443`.\nWe put that `LB` instance to a proxy `Service` via the  `http_proxy_service()` call and then tell our\n`Server` to host that proxy `Service`.\n\n```rust\nfn main() {\n    let mut my_server = Server::new(None).unwrap();\n    my_server.bootstrap();\n\n    let upstreams =\n        LoadBalancer::try_from_iter([\"1.1.1.1:443\", \"1.0.0.1:443\"]).unwrap();\n\n    let mut lb = http_proxy_service(&my_server.configuration, LB(Arc::new(upstreams)));\n        lb.add_tcp(\"0.0.0.0:6188\");\n\n    my_server.add_service(lb);\n\n    my_server.run_forever();\n}\n```\n\n### Run it\n\nNow that we have added the load balancer to the service, we can run our new \nproject with \n\n```cargo run```\n\nTo test it, simply send the server a few requests with the command:\n```\ncurl 127.0.0.1:6188 -svo /dev/null\n```\n\nYou can also navigate your browser to [http://localhost:6188](http://localhost:6188)\n\nThe following output shows that the load balancer is doing its job to balance across the two backends:\n```\nupstream peer is: Backend { addr: Inet(1.0.0.1:443), weight: 1 }\nupstream peer is: Backend { addr: Inet(1.1.1.1:443), weight: 1 }\nupstream peer is: Backend { addr: Inet(1.0.0.1:443), weight: 1 }\nupstream peer is: Backend { addr: Inet(1.1.1.1:443), weight: 1 }\nupstream peer is: Backend { addr: Inet(1.0.0.1:443), weight: 1 }\n...\n```\n\nWell done! At this point you have a functional load balancer. It is a _very_ \nbasic load balancer though, so the next section will walk you through how to\nmake it more robust with some built-in pingora tooling.\n\n## Add functionality\n\nPingora provides several helpful features that can be enabled and configured \nwith just a few lines of code. These range from simple peer health checks to \nthe ability to seamlessly update running binary with zero service interruptions.\n\n### Peer health checks\n\nTo make our load balancer more reliable, we would like to add some health checks \nto our upstream peers. That way if there is a peer that has gone down, we can \nquickly stop routing our traffic to that peer.\n\nFirst let's see how our simple load balancer behaves when one of the peers is\ndown. To do this, we'll update the list of peers to include a peer that is \nguaranteed to be broken.\n\n```rust\nfn main() {\n    // ...\n    let upstreams =\n        LoadBalancer::try_from_iter([\"1.1.1.1:443\", \"1.0.0.1:443\", \"127.0.0.1:343\"]).unwrap();\n    // ...\n}\n```\n\nNow if we run our load balancer again with `cargo run`, and test it with \n\n```\ncurl 127.0.0.1:6188 -svo /dev/null\n```\n\nWe can see that one in every 3 request fails with `502: Bad Gateway`. This is \nbecause our peer selection is strictly following the `RoundRobin` selection \npattern we gave it with no consideration to whether that peer is healthy. We can\nfix this by adding a basic health check service. \n\n```rust\nfn main() {\n    let mut my_server = Server::new(None).unwrap();\n    my_server.bootstrap();\n\n    // Note that upstreams needs to be declared as `mut` now\n    let mut upstreams =\n        LoadBalancer::try_from_iter([\"1.1.1.1:443\", \"1.0.0.1:443\", \"127.0.0.1:343\"]).unwrap();\n\n    let hc = TcpHealthCheck::new();\n    upstreams.set_health_check(hc);\n    upstreams.health_check_frequency = Some(std::time::Duration::from_secs(1));\n\n    let background = background_service(\"health check\", upstreams);\n    let upstreams = background.task();\n\n    // `upstreams` no longer need to be wrapped in an arc\n    let mut lb = http_proxy_service(&my_server.configuration, LB(upstreams));\n    lb.add_tcp(\"0.0.0.0:6188\");\n\n    my_server.add_service(background);\n\n    my_server.add_service(lb);\n    my_server.run_forever();\n}\n```\n\nNow if we again run and test our load balancer, we see that all requests \nsucceed and the broken peer is never used. Based on the configuration we used, \nif that peer were to become healthy again, it would be re-included in the round\nrobin again in within 1 second.\n\n### Command line options\n\nThe pingora `Server` type provides a lot of built-in functionality that we can\ntake advantage of with single-line change. \n\n```rust\nfn main() {\n    let mut my_server = Server::new(Some(Opt::parse_args())).unwrap();\n    ...\n}\n```\n\nWith this change, the command-line arguments passed to our load balancer will be \nconsumed by Pingora. We can test this by running:\n\n```\ncargo run -- -h\n```\n\nWe should see a help menu with the list of arguments now available to us. We \nwill take advantage of those in the next sections to do more with our load \nbalancer for free\n\n### Running in the background\n\nPassing the parameter `-d` or `--daemon` will tell the program to run in the background.\n\n```\ncargo run -- -d\n```\n\nTo stop this service, you can send `SIGTERM` signal to it for a graceful shutdown, in which the service will stop accepting new request but try to finish all ongoing requests before exiting.\n```\npkill -SIGTERM load_balancer\n```\n (`SIGTERM` is the default signal for `pkill`.)\n\n### Configurations\nPingora configuration files help define how to run the service. Here is an \nexample config file that defines how many threads the service can have, the \nlocation of the pid file, the error log file, and the upgrade coordination \nsocket (which we will explain later). Copy the contents below and put them into\na file called `conf.yaml` in your `load_balancer` project directory.\n\n```yaml\n---\nversion: 1\nthreads: 2\npid_file: /tmp/load_balancer.pid\nerror_log: /tmp/load_balancer_err.log\nupgrade_sock: /tmp/load_balancer.sock\n```\n\nTo use this conf file:\n```\nRUST_LOG=INFO cargo run -- -c conf.yaml -d\n```\n`RUST_LOG=INFO` is here so that the service actually populate the error log.\n\nNow you can find the pid of the service.\n```\n cat /tmp/load_balancer.pid\n```\n\n### Gracefully upgrade the service\n(Linux only)\n\nLet's say we changed the code of the load balancer and recompiled the binary. Now we want to upgrade the service running in the background to this newer version.\n\nIf we simply stop the old service, then start the new one, some request arriving in between could be lost. Fortunately, Pingora provides a graceful way to upgrade the service.\n\nThis is done by, first, send `SIGQUIT` signal to the running server, and then start the new server with the parameter `-u` \\ `--upgrade`.\n\n```\npkill -SIGQUIT load_balancer &&\\\nRUST_LOG=INFO cargo run -- -c conf.yaml -d -u\n```\n\nIn this process, The old running server will wait and hand over its listening sockets to the new server. Then the old server runs until all its ongoing requests finish.\n\nFrom a client's perspective, the service is always running because the listening socket is never closed.\n\n## Full examples\n\nThe full code for this example is available in this repository under\n\n[pingora-proxy/examples/load_balancer.rs](../pingora-proxy/examples/load_balancer.rs)\n\nOther examples that you may find helpful are also available here\n\n[pingora-proxy/examples/](../pingora-proxy/examples/)\n[pingora/examples](../pingora/examples/)"
  },
  {
    "path": "docs/user_guide/conf.md",
    "content": "# Configuration\n\nA Pingora configuration file is a list of Pingora settings in yaml format.\n\nExample\n```yaml\n---\nversion: 1\nthreads: 2\npid_file: /run/pingora.pid\nupgrade_sock: /tmp/pingora_upgrade.sock\nuser: nobody\ngroup: webusers\n```\n## Settings\n| Key      | meaning        | value type |\n| ------------- |-------------| ----|\n| version | the version of the conf, currently it is a constant `1` | number |\n| pid_file | The path to the pid file | string |\n| daemon | whether to run the server in the background | bool |\n| error_log | the path to error log output file. STDERR is used if not set | string |\n| upgrade_sock | the path to the upgrade socket. | string |\n| threads | number of threads per service | number |\n| user | the user the pingora server should be run under after daemonization | string |\n| group | the group the pingora server should be run under after daemonization | string |\n| client_bind_to_ipv4 | source IPv4 addresses to bind to when connecting to server | list of string |\n| client_bind_to_ipv6 | source IPv6 addresses to bind to when connecting to server| list of string |\n| ca_file | The path to the root CA file | string |\n| s2n_config_cache_size | The maximum number of unique s2n configs to cache. A value of 0 disables the cache. Default: 10 (s2n-tls only) | number |\n| work_stealing | Enable work stealing runtime (default true). See Pingora runtime (WIP) section for more info | bool |\n| upstream_keepalive_pool_size | The number of total connections to keep in the connection pool | number |\n\n## Extension\nAny unknown settings will be ignored. This allows extending the conf file to add and pass user defined settings. See User defined configuration section.\n"
  },
  {
    "path": "docs/user_guide/ctx.md",
    "content": "# Sharing state across phases with `CTX`\n\n## Using `CTX`\nThe custom filters users implement in different phases of the request don't interact with each other directly. In order to share information and state across the filters, users can define a `CTX` struct. Each request owns a single `CTX` object. All the filters are able to read and update members of the `CTX` object. The CTX object will be dropped at the end of the request.\n\n### Example\n\nIn the following example, the proxy parses the request header in the `request_filter` phase, it stores the boolean flag so that later in the `upstream_peer` phase the flag is used to decide which server to route traffic to. (Technically, the header can be parsed in `upstream_peer` phase, but we just do it in an earlier phase just for the demonstration.)\n\n```Rust\npub struct MyProxy();\n\npub struct MyCtx {\n    beta_user: bool,\n}\n\nfn check_beta_user(req: &pingora_http::RequestHeader) -> bool {\n    // some simple logic to check if user is beta\n    req.headers.get(\"beta-flag\").is_some()\n}\n\n#[async_trait]\nimpl ProxyHttp for MyProxy {\n    type CTX = MyCtx;\n    fn new_ctx(&self) -> Self::CTX {\n        MyCtx { beta_user: false }\n    }\n\n    async fn request_filter(&self, session: &mut Session, ctx: &mut Self::CTX) -> Result<bool> {\n        ctx.beta_user = check_beta_user(session.req_header());\n        Ok(false)\n    }\n\n    async fn upstream_peer(\n        &self,\n        _session: &mut Session,\n        ctx: &mut Self::CTX,\n    ) -> Result<Box<HttpPeer>> {\n        let addr = if ctx.beta_user {\n            info!(\"I'm a beta user\");\n            (\"1.0.0.1\", 443)\n        } else {\n            (\"1.1.1.1\", 443)\n        };\n\n        let peer = Box::new(HttpPeer::new(addr, true, \"one.one.one.one\".to_string()));\n        Ok(peer)\n    }\n}\n```\n\n## Sharing state across requests\nSharing state such as a counter, cache and other info across requests is common. There is nothing special needed for sharing resources and data across requests in Pingora. `Arc`, `static` or any other mechanism can be used.\n\n\n### Example\nLet's modify the example above to track the number of beta visitors as well as the number of total visitors. The counters can either be defined in the `MyProxy` struct itself or defined as a global variable. Because the counters can be concurrently accessed, Mutex is used here.\n\n```Rust\n// global counter\nstatic REQ_COUNTER: Mutex<usize> = Mutex::new(0);\n\npub struct MyProxy {\n    // counter for the service\n    beta_counter: Mutex<usize>, // AtomicUsize works too\n}\n\npub struct MyCtx {\n    beta_user: bool,\n}\n\nfn check_beta_user(req: &pingora_http::RequestHeader) -> bool {\n    // some simple logic to check if user is beta\n    req.headers.get(\"beta-flag\").is_some()\n}\n\n#[async_trait]\nimpl ProxyHttp for MyProxy {\n    type CTX = MyCtx;\n    fn new_ctx(&self) -> Self::CTX {\n        MyCtx { beta_user: false }\n    }\n\n    async fn request_filter(&self, session: &mut Session, ctx: &mut Self::CTX) -> Result<bool> {\n        ctx.beta_user = check_beta_user(session.req_header());\n        Ok(false)\n    }\n\n    async fn upstream_peer(\n        &self,\n        _session: &mut Session,\n        ctx: &mut Self::CTX,\n    ) -> Result<Box<HttpPeer>> {\n        let mut req_counter = REQ_COUNTER.lock().unwrap();\n        *req_counter += 1;\n\n        let addr = if ctx.beta_user {\n            let mut beta_count = self.beta_counter.lock().unwrap();\n            *beta_count += 1;\n            info!(\"I'm a beta user #{beta_count}\");\n            (\"1.0.0.1\", 443)\n        } else {\n            info!(\"I'm an user #{req_counter}\");\n            (\"1.1.1.1\", 443)\n        };\n\n        let peer = Box::new(HttpPeer::new(addr, true, \"one.one.one.one\".to_string()));\n        Ok(peer)\n    }\n}\n```\n\nThe complete example can be found under [`pingora-proxy/examples/ctx.rs`](../../pingora-proxy/examples/ctx.rs). You can run it using `cargo`:\n```\nRUST_LOG=INFO cargo run --example ctx\n```"
  },
  {
    "path": "docs/user_guide/daemon.md",
    "content": "# Daemonization\n\nWhen a Pingora server is configured to run as a daemon, after its bootstrapping, it will move itself to the background and optionally change to run under the configured user and group. The `pid_file` option comes handy in this case for the user to track the PID of the daemon in the background.\n\nDaemonization also allows the server to perform privileged actions like loading secrets and then switch to an unprivileged user before accepting any requests from the network.\n\nThis process happens in the `run_forever()` call. Because daemonization involves `fork()`, certain things like threads created before this call are likely lost.\n"
  },
  {
    "path": "docs/user_guide/error_log.md",
    "content": "# Error logging\n\nPingora libraries are built to expect issues like disconnects, timeouts and invalid inputs from the network. A common way to record these issues are to output them in error log (STDERR or log files).\n\n## Log level guidelines\nPingora adopts the idea behind [log](https://docs.rs/log/latest/log/). There are five log levels:\n* `error`: This level should be used when the error stops the request from being handled correctly. For example when the server we try to connect to is offline.\n* `warning`: This level should be used when an error occurs but the system recovers from it. For example when the primary DNS timed out but the system is able to query the secondary DNS.\n* `info`: Pingora logs when the server is starting up or shutting down.\n* `debug`: Internal details. This log level is not compiled in `release` builds.\n* `trace`: Fine-grained internal details. This log level is not compiled in `release` builds.\n\nThe pingora-proxy crate has a well-defined interface to log errors, so that users don't have to manually log common proxy errors. See its guide for more details.\n"
  },
  {
    "path": "docs/user_guide/errors.md",
    "content": "# How to return errors\n\nFor easy error handling, the `pingora-error` crate exports a custom `Result` type used throughout other Pingora crates.\n\nThe `Error` struct used in this `Result`'s error variant is a wrapper around arbitrary error types. It allows the user to tag the source of the underlying error and attach other custom context info.\n\nUsers will often need to return errors by propagating an existing error or creating a wholly new one. `pingora-error` makes this easy with its error building functions.\n\n## Examples\n\nFor example, one could return an error when an expected header is not present:\n\n```rust\nfn validate_req_header(req: &RequestHeader) -> Result<()> {\n    // validate that the `host` header exists\n    req.headers()\n        .get(http::header::HOST)\n        .ok_or_else(|| Error::explain(InvalidHTTPHeader, \"No host header detected\"))\n}\n\nimpl MyServer {\n    pub async fn handle_request_filter(\n        &self,\n        http_session: &mut Session,\n        ctx: &mut CTX,\n    ) -> Result<bool> {\n        validate_req_header(session.req_header()?).or_err(HTTPStatus(400), \"Missing required headers\")?;\n        Ok(true)\n    }\n}\n```\n\n`validate_req_header` returns an `Error` if the `host` header is not found, using `Error::explain` to create a new `Error` along with an associated type (`InvalidHTTPHeader`) and helpful context that may be logged in an error log.\n\nThis error will eventually propagate to the request filter, where it is returned as a new `HTTPStatus` error using `or_err`. (As part of the default pingora-proxy `fail_to_proxy()` phase, not only will this error be logged, but it will result in sending a `400 Bad Request` response downstream.)\n\nNote that the original causing error will be visible in the error logs as well. `or_err` wraps the original causing error in a new one with additional context, but `Error`'s `Display` implementation also prints the chain of causing errors.\n\n## Guidelines\n\nAn error has a _type_ (e.g. `ConnectionClosed`), a _source_ (e.g. `Upstream`, `Downstream`, `Internal`), and optionally, a _cause_ (another wrapped error) and a _context_ (arbitrary user-provided string details).\n\nA minimal error can be created using functions like `new_in` / `new_up` / `new_down`, each of which specifies a source and asks the user to provide a type.\n\nGenerally speaking:\n* To create a new error, without a direct cause but with more context, use `Error::explain`. You can also use `explain_err` on a `Result` to replace the potential error inside it with a new one.\n* To wrap a causing error in a new one with more context, use `Error::because`. You can also use `or_err` on a `Result` to replace the potential error inside it by wrapping the original one.\n\n## Retry\n\nErrors can be \"retry-able.\" If the error is retry-able, pingora-proxy will be allowed to retry the upstream request. Some errors are only retry-able on [reused connections](pooling.md), e.g. to handle situations where the remote end has dropped a connection we attempted to reuse.\n\nBy default a newly created `Error` either takes on its direct causing error's retry status, or, if left unspecified, is considered not retry-able.\n"
  },
  {
    "path": "docs/user_guide/failover.md",
    "content": "# Handling failures and failover\n\nPingora-proxy allows users to define how to handle failures throughout the life of a proxied request.\n\nWhen a failure happens before the response header is sent downstream, users have a few options:\n1. Send an error page downstream and then give up.\n2. Retry the same upstream again.\n3. Try another upstream if applicable.\n\nOtherwise, once the response header is already sent downstream, there is nothing the proxy can do other than logging an error and then giving up on the request.\n\n\n## Retry / Failover\nIn order to implement retry or failover, `fail_to_connect()` / `error_while_proxy()` needs to mark the error as \"retry-able.\" For failover, `fail_to_connect() / error_while_proxy()` also needs to update the `CTX` to tell `upstream_peer()` not to use the same `Peer` again.\n\n### Safety\nIn general, idempotent HTTP requests, e.g., `GET`, are safe to retry. Other requests, e.g., `POST`, are not safe to retry if the requests have already been sent. When `fail_to_connect()` is called, pingora-proxy guarantees that nothing was sent upstream. Users are not recommended to retry a non-idempotent request after `error_while_proxy()` unless they know the upstream server enough to know whether it is safe.\n\n### Example\nIn the following example we set a `tries` variable on the `CTX` to track how many connection attempts we've made. When setting our peer in `upstream_peer` we check if `tries` is less than one and connect to 192.0.2.1. On connect failure we increment `tries` in `fail_to_connect` and set `e.set_retry(true)` which tells Pingora this is a retryable error. On retry, we enter `upstream_peer` again and this time connect to 1.1.1.1. If we're unable to connect to 1.1.1.1 we return a 502 since we only set `e.set_retry(true)` in `fail_to_connect` when `tries` is zero.\n\n```Rust\npub struct MyProxy();\n\npub struct MyCtx {\n    tries: usize,\n}\n\n#[async_trait]\nimpl ProxyHttp for MyProxy {\n    type CTX = MyCtx;\n    fn new_ctx(&self) -> Self::CTX {\n        MyCtx { tries: 0 }\n    }\n\n    fn fail_to_connect(\n        &self,\n        _session: &mut Session,\n        _peer: &HttpPeer,\n        ctx: &mut Self::CTX,\n        mut e: Box<Error>,\n    ) -> Box<Error> {\n        if ctx.tries > 0 {\n            return e;\n        }\n        ctx.tries += 1;\n        e.set_retry(true);\n        e\n    }\n\n    async fn upstream_peer(\n        &self,\n        _session: &mut Session,\n        ctx: &mut Self::CTX,\n    ) -> Result<Box<HttpPeer>> {\n        let addr = if ctx.tries < 1 {\n            (\"192.0.2.1\", 443)\n        } else {\n            (\"1.1.1.1\", 443)\n        };\n\n        let mut peer = Box::new(HttpPeer::new(addr, true, \"one.one.one.one\".to_string()));\n        peer.options.connection_timeout = Some(Duration::from_millis(100));\n        Ok(peer)\n    }\n}\n```\n"
  },
  {
    "path": "docs/user_guide/graceful.md",
    "content": "# Graceful restart and shutdown\n\nGraceful restart, upgrade, and shutdown mechanisms are very commonly used to avoid errors or downtime when releasing new versions of Pingora servers.\n\nPingora graceful upgrade mechanism guarantees the following:\n* A request is guaranteed to be handled either by the old server instance or the new one. No request will see connection refused when trying to connect to the server endpoints.\n* A request that can finish within the grace period is guaranteed not to be terminated.\n\n## How to graceful upgrade\n### Step 0\nConfigure the upgrade socket. The old and new server need to agree on the same path to this socket. See configuration manual for details.\n\n### Step 1\nStart the new instance with the `--upgrade` CLI option. The new instance will not try to listen to the service endpoint right away. It will try to acquire the listening socket from the old instance instead.\n\n### Step 2\nSend SIGQUIT signal to the old instance. The old instance will start to transfer the listening socket to the new instance.\n\nOnce step 2 is successful, the new instance will start to handle new incoming connections right away. Meanwhile, the old instance will enter its graceful shutdown mode. It waits a short period of time (to give the new instance time to initialize and prepare to handle traffic), after which it will not accept any new connections.\n"
  },
  {
    "path": "docs/user_guide/index.md",
    "content": "# User Guide\n\nIn this guide, we will cover the most used features, operations and settings of Pingora.\n\n## Running Pingora servers\n* [Start and stop](start_stop.md)\n* [Graceful restart and graceful shutdown](graceful.md)\n* [Configuration](conf.md)\n* [Daemonization](daemon.md)\n* [Systemd integration](systemd.md)\n* [Handling panics](panic.md)\n* [Error logging](error_log.md)\n* [Prometheus](prom.md)\n\n## Building HTTP proxies\n* [Life of a request: `pingora-proxy` phases and filters](phase.md)\n* [`Peer`: how to connect to upstream](peer.md)\n* [Sharing state across phases with `CTX`](ctx.md)\n* [How to return errors](errors.md)\n* [Examples: take control of the request](modify_filter.md)\n* [Connection pooling and reuse](pooling.md)\n* [Handling failures and failover](failover.md)\n* [RateLimiter quickstart](rate_limiter.md)\n\n## Advanced topics (WIP)\n* [Pingora internals](internals.md)\n* Using BoringSSL\n* User defined configuration\n* Pingora async runtime and threading model\n* Background Service\n* Blocking code in async context\n* Tracing\n"
  },
  {
    "path": "docs/user_guide/internals.md",
    "content": "# Pingora Internals\n\n(Special thanks to [James Munns](https://github.com/jamesmunns) for writing this section)\n\n\n## Starting the `Server`\n\nThe pingora system starts by spawning a *server*. The server is responsible for starting *services*, and listening for termination events.\n\n```\n                               ┌───────────┐\n                    ┌─────────>│  Service  │\n                    │          └───────────┘\n┌────────┐          │          ┌───────────┐\n│ Server │──Spawns──┼─────────>│  Service  │\n└────────┘          │          └───────────┘\n                    │          ┌───────────┐\n                    └─────────>│  Service  │\n                               └───────────┘\n```\n\nAfter spawning the *services*, the server continues to listen to a termination event, which it will propagate to the created services.\n\n## Services\n\n*Services* are entities that handle listening to given sockets, and perform the core functionality. A *service* is tied to a particular protocol and set of options.\n\n> NOTE: there are also \"background\" services, which just do *stuff*, and aren't necessarily listening to a socket. For now we're just talking about listener services.\n\nEach service has its own threadpool/tokio runtime, with a number of threads based on the configured value. Worker threads are not shared cross-service. Service runtime threadpools may be work-stealing (tokio-default), or non-work-stealing (N isolated single threaded runtimes).\n\n```\n┌─────────────────────────┐\n│ ┌─────────────────────┐ │\n│ │┌─────────┬─────────┐│ │\n│ ││  Conn   │  Conn   ││ │\n│ │├─────────┼─────────┤│ │\n│ ││Endpoint │Endpoint ││ │\n│ │├─────────┴─────────┤│ │\n│ ││     Listeners     ││ │\n│ │├─────────┬─────────┤│ │\n│ ││ Worker  │ Worker  ││ │\n│ ││ Thread  │ Thread  ││ │\n│ │├─────────┴─────────┤│ │\n│ ││  Tokio Executor   ││ │\n│ │└───────────────────┘│ │\n│ └─────────────────────┘ │\n│ ┌───────┐               │\n└─┤Service├───────────────┘\n  └───────┘\n```\n\n## Service Listeners\n\nAt startup, each Service is assigned a set of downstream endpoints that they listen to. A single service may listen to more than one endpoint. The Server also passes along any relevant configuration, including TLS settings if relevant.\n\nThese endpoints are converted into listening sockets, called `TransportStack`s. Each `TransportStack` is assigned to an async task within that service's executor.\n\n```\n                                 ┌───────────────────┐\n                                 │┌─────────────────┐│    ┌ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┐  ┌ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─\n ┌─────────┐                     ││ TransportStack  ││                                ┌────────────────────┐│\n┌┤Listeners├────────┐            ││                 ││    │                       │  ││                    │\n│└─────────┘        │            ││ (Listener, TLS  │├──────spawn(run_endpoint())────>│ Service<ServerApp> ││\n│┌─────────────────┐│            ││    Acceptor,    ││    │                       │  ││                    │\n││    Endpoint     ││            ││   UpgradeFDs)   ││                                └────────────────────┘│\n││   addr/ports    ││            │├─────────────────┤│    │                       │  │\n││ + TLS Settings  ││            ││ TransportStack  ││                                ┌────────────────────┐│\n│├─────────────────┤│            ││                 ││    │                       │  ││                    │\n││    Endpoint     ││──build()─> ││ (Listener, TLS  │├──────spawn(run_endpoint())────>│ Service<ServerApp> ││\n││   addr/ports    ││            ││    Acceptor,    ││    │                       │  ││                    │\n││ + TLS Settings  ││            ││   UpgradeFDs)   ││                                └────────────────────┘│\n│├─────────────────┤│            │├─────────────────┤│    │                       │  │\n││    Endpoint     ││            ││ TransportStack  ││                                ┌────────────────────┐│\n││   addr/ports    ││            ││                 ││    │                       │  ││                    │\n││ + TLS Settings  ││            ││ (Listener, TLS  │├──────spawn(run_endpoint())────>│ Service<ServerApp> ││\n│└─────────────────┘│            ││    Acceptor,    ││    │                       │  ││                    │\n└───────────────────┘            ││   UpgradeFDs)   ││                                └────────────────────┘│\n                                 │└─────────────────┘│    │ ┌───────────────┐     │  │ ┌──────────────┐\n                                 └───────────────────┘     ─│start_service()│─ ─ ─    ─│ Worker Tasks ├ ─ ─ ┘\n                                                            └───────────────┘          └──────────────┘\n```\n\n## Downstream connection lifecycle\n\nEach service processes incoming connections by spawning a task-per-connection. These connections are held open\nas long as there are new events to be handled.\n\n```\n                                  ┌ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┐\n\n                                  │  ┌───────────────┐   ┌────────────────┐   ┌─────────────────┐    ┌─────────────┐  │\n┌────────────────────┐               │ UninitStream  │   │    Service     │   │       App       │    │  Task Ends  │\n│                    │            │  │ ::handshake() │──>│::handle_event()│──>│ ::process_new() │──┬>│             │  │\n│ Service<ServerApp> │──spawn()──>   └───────────────┘   └────────────────┘   └─────────────────┘  │ └─────────────┘\n│                    │            │                                                    ▲           │                  │\n└────────────────────┘                                                                 │         while\n                                  │                                                    └─────────reuse                │\n                                     ┌───────────────────────────┐\n                                  └ ─│  Task on Service Runtime  │─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┘\n                                     └───────────────────────────┘\n```\n\n## What is a proxy then?\n\nInterestingly, the `pingora` `Server` itself has no particular notion of a Proxy.\n\nInstead, it only thinks in terms of `Service`s, which are expected to contain a particular implementor of the `ServiceApp` trait.\n\nFor example, this is how an `HttpProxy` struct, from the `pingora-proxy` crate, \"becomes\" a `Service` spawned by the `Server`:\n\n```\n┌─────────────┐\n│  HttpProxy  │\n│  (struct)   │\n└─────────────┘\n       │\n   implements   ┌─────────────┐\n       │        │HttpServerApp│\n       └───────>│   (trait)   │\n                └─────────────┘\n                       │\n                   implements   ┌─────────────┐\n                       │        │  ServerApp  │\n                       └───────>│   (trait)   │\n                                └─────────────┘\n                                       │\n                                   contained    ┌─────────────────────┐\n                                     within     │                     │\n                                       └───────>│ Service<ServiceApp> │\n                                                │                     │\n                                                └─────────────────────┘\n```\n\nDifferent functionalities and helpers are provided at different layers in this representation.\n\n```\n┌─────────────┐        ┌──────────────────────────────────────┐\n│  HttpProxy  │        │Handles high level Proxying workflow, │\n│  (struct)   │─ ─ ─ ─ │   customizable via ProxyHttp trait   │\n└──────┬──────┘        └──────────────────────────────────────┘\n       │\n┌──────▼──────┐        ┌──────────────────────────────────────┐\n│HttpServerApp│        │ Handles selection of H1 vs H2 stream │\n│   (trait)   │─ ─ ─ ─ │     handling, incl H2 handshake      │\n└──────┬──────┘        └──────────────────────────────────────┘\n       │\n┌──────▼──────┐        ┌──────────────────────────────────────┐\n│  ServerApp  │        │ Handles dispatching of App instances │\n│   (trait)   │─ ─ ─ ─ │   as individual tasks, per Session   │\n└──────┬──────┘        └──────────────────────────────────────┘\n       │\n┌──────▼──────┐        ┌──────────────────────────────────────┐\n│ Service<A>  │        │ Handles dispatching of App instances │\n│  (struct)   │─ ─ ─ ─ │  as individual tasks, per Listener   │\n└─────────────┘        └──────────────────────────────────────┘\n```\n\nThe `HttpProxy` struct handles the high level workflow of proxying an HTTP connection\n\nIt uses the `ProxyHttp` (note the flipped wording order!) **trait** to allow customization\nat each of the following steps (note: taken from [the phase chart](./phase_chart.md) doc):\n\n```mermaid\n graph TD;\n    start(\"new request\")-->request_filter;\n    request_filter-->upstream_peer;\n\n    upstream_peer-->Connect{{IO: connect to upstream}};\n\n    Connect--connection success-->connected_to_upstream;\n    Connect--connection failure-->fail_to_connect;\n\n    connected_to_upstream-->upstream_request_filter;\n    upstream_request_filter --> SendReq{{IO: send request to upstream}};\n    SendReq-->RecvResp{{IO: read response from upstream}};\n    RecvResp-->upstream_response_filter-->response_filter-->upstream_response_body_filter-->response_body_filter-->logging-->endreq(\"request done\");\n\n    fail_to_connect --can retry-->upstream_peer;\n    fail_to_connect --can't retry-->fail_to_proxy--send error response-->logging;\n\n    RecvResp--failure-->IOFailure;\n    SendReq--failure-->IOFailure;\n    error_while_proxy--can retry-->upstream_peer;\n    error_while_proxy--can't retry-->fail_to_proxy;\n\n    request_filter --send response-->logging\n\n\n    Error>any response filter error]-->error_while_proxy\n    IOFailure>IO error]-->error_while_proxy\n\n```\n\n## Zooming out\n\nBefore we zoom in, it's probably good to zoom out and remind ourselves how\na proxy generally works:\n\n```\n┌────────────┐          ┌─────────────┐         ┌────────────┐\n│ Downstream │          │    Proxy    │         │  Upstream  │\n│   Client   │─────────>│             │────────>│   Server   │\n└────────────┘          └─────────────┘         └────────────┘\n```\n\nThe proxy will be taking connections from the **Downstream** client, and (if\neverything goes right), establishing a connection with the appropriate\n**Upstream** server. This selected upstream server is referred to as\nthe **Peer**.\n\nOnce the connection is established, the Downstream and Upstream can communicate\nbidirectionally.\n\nSo far, the discussion of Server, Services, and Listeners have focused on the LEFT\nhalf of this diagram, handling incoming Downstream connections, and getting it TO\nthe proxy component.\n\nNext, we'll look at the RIGHT half of this diagram, connecting to Upstreams.\n\n## Managing the Upstream\n\nConnections to Upstream Peers are made through `Connector`s. This is not a specific type or trait, but more\nof a \"style\".\n\nConnectors are responsible for a few things:\n\n* Establishing a connection with a Peer\n* Maintaining a connection pool with the Peer, allowing for connection reuse across:\n    * Multiple requests from a single downstream client\n    * Multiple requests from different downstream clients\n* Measuring health of connections, for connections like H2, which perform regular pings\n* Handling protocols with multiple poolable layers, like H2\n* Caching, if relevant to the protocol and enabled\n* Compression, if relevant to the protocol and enabled\n\nNow in context, we can see how each end of the Proxy is handled:\n\n```\n┌────────────┐          ┌─────────────┐         ┌────────────┐\n│ Downstream │       ┌ ─│─   Proxy  ┌ ┼ ─       │  Upstream  │\n│   Client   │─────────>│ │           │──┼─────>│   Server   │\n└────────────┘       │  └───────────┼─┘         └────────────┘\n                      ─ ─ ┘          ─ ─ ┘\n                        ▲              ▲\n                     ┌──┘              └──┐\n                     │                    │\n                ┌ ─ ─ ─ ─ ┐         ┌ ─ ─ ─ ─ ─\n                 Listeners           Connectors│\n                └ ─ ─ ─ ─ ┘         └ ─ ─ ─ ─ ─\n```\n\n## What about multiple peers?\n\n`Connectors` only handle the connection to a single peer, so selecting one of potentially multiple Peers\nis actually handled one level up, in the `upstream_peer()` method of the `ProxyHttp` trait.\n"
  },
  {
    "path": "docs/user_guide/modify_filter.md",
    "content": "# Examples: taking control of the request\n\nIn this section we will go through how to route, modify or reject requests.\n\n## Routing\nAny information from the request can be used to make routing decision. Pingora doesn't impose any constraints on how users could implement their own routing logic.\n\nIn the following example, the proxy sends traffic to 1.0.0.1 only when the request path start with `/family/`. All the other requests are routed to 1.1.1.1.\n\n```Rust\npub struct MyGateway;\n\n#[async_trait]\nimpl ProxyHttp for MyGateway {\n    type CTX = ();\n    fn new_ctx(&self) -> Self::CTX {}\n\n    async fn upstream_peer(\n        &self,\n        session: &mut Session,\n        _ctx: &mut Self::CTX,\n    ) -> Result<Box<HttpPeer>> {\n        let addr = if session.req_header().uri.path().starts_with(\"/family/\") {\n            (\"1.0.0.1\", 443)\n        } else {\n            (\"1.1.1.1\", 443)\n        };\n\n        info!(\"connecting to {addr:?}\");\n\n        let peer = Box::new(HttpPeer::new(addr, true, \"one.one.one.one\".to_string()));\n        Ok(peer)\n    }\n}\n```\n\n\n## Modifying headers\n\nBoth request and response headers can be added, removed or modified in their corresponding phases. In the following example, we add logic to the `response_filter` phase to update the `Server` header and remove the `alt-svc` header.\n\n```Rust\n#[async_trait]\nimpl ProxyHttp for MyGateway {\n    ...\n    async fn response_filter(\n        &self,\n        _session: &mut Session,\n        upstream_response: &mut ResponseHeader,\n        _ctx: &mut Self::CTX,\n    ) -> Result<()>\n    where\n        Self::CTX: Send + Sync,\n    {\n        // replace existing header if any\n        upstream_response\n            .insert_header(\"Server\", \"MyGateway\")\n            .unwrap();\n        // because we don't support h3\n        upstream_response.remove_header(\"alt-svc\");\n\n        Ok(())\n    }\n}\n```\n\n## Return Error pages\n\nSometimes instead of proxying the traffic, under certain conditions, such as authentication failures, you might want the proxy to just return an error page.\n\n```Rust\nfn check_login(req: &pingora_http::RequestHeader) -> bool {\n    // implement you logic check logic here\n    req.headers.get(\"Authorization\").map(|v| v.as_bytes()) == Some(b\"password\")\n}\n\n#[async_trait]\nimpl ProxyHttp for MyGateway {\n    ...\n    async fn request_filter(&self, session: &mut Session, _ctx: &mut Self::CTX) -> Result<bool> {\n        if session.req_header().uri.path().starts_with(\"/login\")\n            && !check_login(session.req_header())\n        {\n            let _ = session.respond_error(403).await;\n            // true: tell the proxy that the response is already written\n            return Ok(true);\n        }\n        Ok(false)\n    }\n```\n## Logging\n\nLogging logic can be added to the `logging` phase of Pingora. The logging phase runs on every request right before Pingora proxy finish processing it. This phase runs for both successful and failed requests.\n\nIn the example below, we add Prometheus metric and access logging to the proxy. In order for the metrics to be scraped, we also start a Prometheus metric server on a different port.\n\n\n``` Rust\npub struct MyGateway {\n    req_metric: prometheus::IntCounter,\n}\n\n#[async_trait]\nimpl ProxyHttp for MyGateway {\n    ...\n    async fn logging(\n        &self,\n        session: &mut Session,\n        _e: Option<&pingora::Error>,\n        ctx: &mut Self::CTX,\n    ) {\n        let response_code = session\n            .response_written()\n            .map_or(0, |resp| resp.status.as_u16());\n        // access log\n        info!(\n            \"{} response code: {response_code}\",\n            self.request_summary(session, ctx)\n        );\n\n        self.req_metric.inc();\n    }\n\nfn main() {\n   ...\n    let mut prometheus_service_http =\n        pingora::services::listening::Service::prometheus_http_service();\n    prometheus_service_http.add_tcp(\"127.0.0.1:6192\");\n    my_server.add_service(prometheus_service_http);\n\n    my_server.run_forever();\n}\n```"
  },
  {
    "path": "docs/user_guide/panic.md",
    "content": "# Handling panics\n\nAny panic that happens to particular requests does not affect other ongoing requests or the server's ability to handle other requests. Sockets acquired by the panicking requests are dropped (closed). The panics will be captured by the tokio runtime and then ignored.\n\nIn order to monitor the panics, Pingora server has built-in Sentry integration.\n```rust\nmy_server.sentry = Some(\n    sentry::ClientOptions{\n        dsn: \"SENTRY_DSN\".into_dsn().unwrap(),\n        ..Default::default()\n    }\n);\n```\n\nEven though a panic is not fatal in Pingora, it is still not the preferred way to handle failures like network timeouts. Panics should be reserved for unexpected logic errors.\n"
  },
  {
    "path": "docs/user_guide/peer.md",
    "content": "# `Peer`: how to connect to upstream\n\nIn the `upstream_peer()` phase the user should return a `Peer` object which defines how to connect to a certain upstream.\n\n## `Peer`\nA `HttpPeer` defines which upstream to connect to.\n| attribute      | meaning        |\n| ------------- |-------------|\n|address: `SocketAddr`| The IP:Port to connect to |\n|scheme: `Scheme`| Http or Https |\n|sni: `String`| The SNI to use, Https only |\n|proxy: `Option<Proxy>`| The setting to proxy the request through a [CONNECT proxy](https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods/CONNECT) |\n|client_cert_key: `Option<Arc<CertKey>>`| The client certificate to use in mTLS connections to upstream |\n|options: `PeerOptions`| See below |\n\n\n## `PeerOptions`\nA `PeerOptions` defines how to connect to the upstream.\n| attribute      | meaning        |\n| ------------- |-------------|\n|bind_to: `Option<InetSocketAddr>`| Which local address to bind to as the client IP |\n|connection_timeout: `Option<Duration>`| How long to wait before giving up *establishing* a TCP connection |\n|total_connection_timeout: `Option<Duration>`| How long to wait before giving up *establishing* a connection including TLS handshake time |\n|read_timeout: `Option<Duration>`| How long to wait before each individual `read()` from upstream. The timer is reset after each `read()` |\n|idle_timeout: `Option<Duration>`| How long to wait before closing a idle connection waiting for connection reuse |\n|write_timeout: `Option<Duration>`| How long to wait before a `write()` to upstream finishes |\n|verify_cert: `bool`| Whether to check if upstream' server cert is valid and validated |\n|verify_hostname: `bool`| Whether to check if upstream server cert's CN matches the SNI |\n|use_system_certs: `bool`| Whether the system trust store should be loaded and used when verifying certificates. Impacts performance (s2n-tls only) |\n|alternative_cn: `Option<String>`| Accept the cert if the CN matches this name |\n|alpn: `ALPN`| Which HTTP protocol to advertise during ALPN, http1.1 and/or http2 |\n|ca: `Option<Arc<Box<[X509]>>>`| Which Root CA to use to validate the server's cert |\n|psk: `Option<Arc<PskConfig>>` | The PSK configuration to use in [PSK-TLS](https://datatracker.ietf.org/doc/html/rfc4279) handshakes (s2n-tls only) |\n|s2n_security_policy: `Option<S2NPolicy>` | S2N [Security Policy](https://aws.github.io/s2n-tls/usage-guide/ch06-security-policies.html) to use. Defaults to `default_tls13` if undefined. (s2n-tls only) |\n|max_blinding_delay: `Option<u32>` | S2N-TLS will delay a response up to the [max blinding delay](https://aws.github.io/s2n-tls/usage-guide/ch03-error-handling.html#blinding) (default 30) seconds whenever an error triggered by a peer occurs to mitigate against timing side channels. (s2n-tls only) |\n|tcp_keepalive: `Option<TcpKeepalive>`| TCP keepalive settings to upstream |\n\n## Examples\nTBD\n"
  },
  {
    "path": "docs/user_guide/phase.md",
    "content": "# Life of a request: pingora-proxy phases and filters\n\n## Intro\nThe pingora-proxy HTTP proxy framework supports highly programmable proxy behaviors. This is done by allowing users to inject custom logic into different phases (stages) in the life of a request.\n\n## Life of a proxied HTTP request\n1. The life of a proxied HTTP request starts when the proxy reads the request header from the **downstream** (i.e., the client).\n2. Then, the proxy connects to the **upstream** (i.e., the remote server). This step is skipped if there is a previously established [connection to reuse](pooling.md).\n3. The proxy then sends the request header to the upstream.\n4. Once the request header is sent, the proxy enters a duplex mode, which simultaneously proxies:\n    a. upstream response (both header and body) to the downstream, and\n    b. downstream request body to upstream (if any).\n5. Once the entire request/response finishes, the life of the request is ended. All resources are released. The downstream connections and the upstream connections are recycled to be reused if applicable.\n\n## Pingora-proxy phases and filters\nPingora-proxy allows users to insert arbitrary logic into the life of a request.\n```mermaid\n graph TD;\n    start(\"new request\")-->early_request_filter;\n    early_request_filter-->request_filter;\n    request_filter-->upstream_peer;\n\n    upstream_peer-->Connect{{IO: connect to upstream}};\n\n    Connect--connection success-->connected_to_upstream;\n    Connect--connection failure-->fail_to_connect;\n\n    connected_to_upstream-->upstream_request_filter;\n    upstream_request_filter --> request_body_filter;\n    request_body_filter --> SendReq{{IO: send request to upstream}};\n    SendReq-->RecvResp{{IO: read response from upstream}};\n    RecvResp-->upstream_response_filter-->response_filter-->upstream_response_body_filter-->response_body_filter-->logging-->endreq(\"request done\");\n\n    fail_to_connect --can retry-->upstream_peer;\n    fail_to_connect --can't retry-->fail_to_proxy--send error response-->logging;\n\n    RecvResp--failure-->IOFailure;\n    SendReq--failure-->IOFailure;\n    error_while_proxy--can retry-->upstream_peer;\n    error_while_proxy--can't retry-->fail_to_proxy;\n\n    request_filter --send response-->logging\n\n\n    Error>any response filter error]-->error_while_proxy\n    IOFailure>IO error]-->error_while_proxy\n```\n\n### General filter usage guidelines\n* Most filters return a [`pingora_error::Result<_>`](errors.md). When the returned value is `Result::Err`, `fail_to_proxy()` will be called and the request will be terminated.\n* Most filters are async functions, which allows other async operations such as IO to be performed within the filters.\n* A per-request `CTX` object can be defined to share states across the filters of the same request. All filters have mutable access to this object.\n* Most filters are optional.\n* The reason both `upstream_response_*_filter()` and `response_*_filter()` exist is for HTTP caching integration reasons (still WIP).\n\n\n### `early_request_filter()`\nThis is the first phase of every request.\n\nThis function is similar to `request_filter()` but executes before any other logic, including downstream module logic. The main purpose of this function is to provide finer-grained control of the behavior of the modules.\n\n### `request_filter()`\nThis phase is usually for validating request inputs, rate limiting, and initializing context.\n\n### `request_body_filter()`\nThis phase is triggered after a request body is ready to send to upstream. It will be called every time a piece of request body is received.\n\n### `proxy_upstream_filter()`\nThis phase determines if we should continue to the upstream to serve a response. If we short-circuit, a 502 is returned by default, but a different response can be implemented.\n\nThis phase returns a boolean determining if we should continue to the upstream or error.\n\n### `upstream_peer()`\nThis phase decides which upstream to connect to (e.g. with DNS lookup and hashing/round-robin), and how to connect to it.\n\nThis phase returns a `Peer` that defines the upstream to connect to. Implementing this phase is **required**.\n\n### `connected_to_upstream()`\nThis phase is executed when upstream is successfully connected.\n\nUsually this phase is for logging purposes. Connection info such as RTT and upstream TLS ciphers are reported in this phase.\n\n### `fail_to_connect()`\nThe counterpart of `connected_to_upstream()`. This phase is called if an error is encountered when connecting to upstream.\n\nIn this phase users can report the error in Sentry/Prometheus/error log. Users can also decide if the error is retry-able.\n\nIf the error is retry-able, `upstream_peer()` will be called again, in which case the user can decide whether to retry the same upstream or failover to a secondary one.\n\nIf the error is not retry-able, the request will end.\n\n### `upstream_request_filter()`\nThis phase is to modify requests before sending to upstream.\n\n### `upstream_response_filter()/upstream_response_body_filter()/upstream_response_trailer_filter()`\nThis phase is triggered after an upstream response header/body/trailer is received.\n\nThis phase is to modify or process response headers, body, or trailers before sending to downstream. Note that this phase is called _prior_ to HTTP caching and therefore any changes made here will affect the response stored in the HTTP cache.\n\n### `response_filter()/response_body_filter()/response_trailer_filter()`\nThis phase is triggered after a response header/body/trailer is ready to send to downstream.\n\nThis phase is to modify them before sending to downstream.\n\n### `error_while_proxy()`\nThis phase is triggered during proxy errors to upstream, this is after the connection is established.\n\nThis phase may decide to retry a request if the connection was re-used and the HTTP method is idempotent.\n\n### `fail_to_proxy()`\nThis phase is called whenever an error is encounter during any of the phases above.\n\nThis phase is usually for error logging and error reporting to downstream.\n\n### `logging()`\nThis is the last phase that runs after the request is finished (or errors) and before any of its resources are released. Every request will end up in this final phase.\n\nThis phase is usually for logging and post request cleanup.\n\n### `request_summary()`\nThis is not a phase, but a commonly used callback.\n\nEvery error that reaches `fail_to_proxy()` will be automatically logged in the error log. `request_summary()` will be called to dump the info regarding the request when logging the error.\n\nThis callback returns a string which allows users to customize what info to dump in the error log to help track and debug the failures.\n\n### `suppress_error_log()`\nThis is also not a phase, but another callback.\n\n`fail_to_proxy()` errors are automatically logged in the error log, but users may not be interested in every error. For example, downstream errors are logged if the client disconnects early, but these errors can become noisy if users are mainly interested in observing upstream issues. This callback can inspect the error and returns true or false. If true, the error will not be written to the log.\n\n### Cache filters\n\nTo be documented\n"
  },
  {
    "path": "docs/user_guide/phase_chart.md",
    "content": "Pingora proxy phases without caching\n```mermaid\n graph TD;\n    start(\"new request\")-->early_request_filter;\n    early_request_filter-->request_filter;\n    request_filter-->upstream_peer;\n\n    upstream_peer-->Connect{{IO: connect to upstream}};\n\n    Connect--connection success-->connected_to_upstream;\n    Connect--connection failure-->fail_to_connect;\n\n    connected_to_upstream-->upstream_request_filter;\n    upstream_request_filter --> request_body_filter;\n    request_body_filter --> SendReq{{IO: send request to upstream}};\n    SendReq-->RecvResp{{IO: read response from upstream}};\n    RecvResp-->upstream_response_filter-->response_filter-->upstream_response_body_filter-->response_body_filter-->logging-->endreq(\"request done\");\n\n    fail_to_connect --can retry-->upstream_peer;\n    fail_to_connect --can't retry-->fail_to_proxy--send error response-->logging;\n\n    RecvResp--failure-->IOFailure;\n    SendReq--failure-->IOFailure;\n    error_while_proxy--can retry-->upstream_peer;\n    error_while_proxy--can't retry-->fail_to_proxy;\n\n    request_filter --send response-->logging\n\n\n    Error>any response filter error]-->error_while_proxy\n    IOFailure>IO error]-->error_while_proxy\n```"
  },
  {
    "path": "docs/user_guide/pooling.md",
    "content": "# Connection pooling and reuse\n\nWhen the request to a `Peer` (upstream server) is finished, the connection to that peer is kept alive and added to a connection pool to be _reused_ by subsequent requests. This happens automatically without any special configuration.\n\nRequests that reuse previously established connections avoid the latency and compute cost of setting up a new connection, improving the Pingora server's overall performance and scalability.\n\n## Same `Peer`\nOnly the connections to the exact same `Peer` can be reused by a request. For correctness and security reasons, two `Peer`s are the same if and only if all the following attributes are the same\n* IP:port\n* scheme\n* SNI\n* client cert\n* verify cert\n* verify hostname\n* alternative_cn\n* proxy settings\n\n## Disable pooling\nTo disable connection pooling and reuse to a certain `Peer`, just set the `idle_timeout` to 0 seconds to all requests using that `Peer`.\n\n## Failure\nA connection is considered not reusable if errors happen during the request.\n"
  },
  {
    "path": "docs/user_guide/prom.md",
    "content": "# Prometheus\n\nPingora has a built-in prometheus HTTP metric server for scraping.\n\n```rust\n    ...\n    let mut prometheus_service_http = Service::prometheus_http_service();\n    prometheus_service_http.add_tcp(\"0.0.0.0:1234\");\n    my_server.add_service(prometheus_service_http);\n    my_server.run_forever();\n```\n\nThe simplest way to use it is to have [static metrics](https://docs.rs/prometheus/latest/prometheus/#static-metrics).\n\n```rust\nstatic MY_COUNTER: Lazy<IntGauge> = Lazy::new(|| {\n    register_int_gauge!(\"my_counter\", \"my counter\").unwrap()\n});\n\n```\n\nThis static metric will automatically appear in the Prometheus metric endpoint.\n"
  },
  {
    "path": "docs/user_guide/rate_limiter.md",
    "content": "# **RateLimiter quickstart**\nPingora provides a crate `pingora-limits` which provides a simple and easy to use rate limiter for your application. Below is an example of how you can use [`Rate`](https://docs.rs/pingora-limits/latest/pingora_limits/rate/struct.Rate.html) to create an application that uses multiple limiters to restrict the rate at which requests can be made on a per-app basis (determined by a request header).\n\n## Steps\n1. Add the following dependencies to your `Cargo.toml`:\n   ```toml\n   async-trait=\"0.1\"\n   pingora = { version = \"0.3\", features = [ \"lb\" ] }\n   pingora-limits = \"0.3.0\"\n   once_cell = \"1.19.0\"\n   ```\n2. Declare a global rate limiter map to store the rate limiter for each client. In this example, we use `appid`.\n3. Override the `request_filter` method in the `ProxyHttp` trait to implement rate limiting.\n   1. Retrieve the client appid from header.\n   2. Retrieve the current window requests from the rate limiter map. If there is no rate limiter for the client, create a new one and insert it into the map.\n   3. If the current window requests exceed the limit, return 429 and set RateLimiter associated headers.\n   4. If the request is not rate limited, return `Ok(false)` to continue the request.\n\n## Example\n```rust\nuse async_trait::async_trait;\nuse once_cell::sync::Lazy;\nuse pingora::prelude::*;\nuse pingora_limits::rate::Rate;\nuse std::sync::Arc;\nuse std::time::Duration;\n\nfn main() {\n    let mut server = Server::new(Some(Opt::default())).unwrap();\n    server.bootstrap();\n    let mut upstreams = LoadBalancer::try_from_iter([\"1.1.1.1:443\", \"1.0.0.1:443\"]).unwrap();\n    // Set health check\n    let hc = TcpHealthCheck::new();\n    upstreams.set_health_check(hc);\n    upstreams.health_check_frequency = Some(Duration::from_secs(1));\n    // Set background service\n    let background = background_service(\"health check\", upstreams);\n    let upstreams = background.task();\n    // Set load balancer\n    let mut lb = http_proxy_service(&server.configuration, LB(upstreams));\n    lb.add_tcp(\"0.0.0.0:6188\");\n\n    // let rate = Rate\n    server.add_service(background);\n    server.add_service(lb);\n    server.run_forever();\n}\n\npub struct LB(Arc<LoadBalancer<RoundRobin>>);\n\nimpl LB {\n    pub fn get_request_appid(&self, session: &mut Session) -> Option<String> {\n        match session\n            .req_header()\n            .headers\n            .get(\"appid\")\n            .map(|v| v.to_str())\n        {\n            None => None,\n            Some(v) => match v {\n                Ok(v) => Some(v.to_string()),\n                Err(_) => None,\n            },\n        }\n    }\n}\n\n// Rate limiter\nstatic RATE_LIMITER: Lazy<Rate> = Lazy::new(|| Rate::new(Duration::from_secs(1)));\n\n// max request per second per client\nstatic MAX_REQ_PER_SEC: isize = 1;\n\n#[async_trait]\nimpl ProxyHttp for LB {\n    type CTX = ();\n\n    fn new_ctx(&self) {}\n\n    async fn upstream_peer(\n        &self,\n        _session: &mut Session,\n        _ctx: &mut Self::CTX,\n    ) -> Result<Box<HttpPeer>> {\n        let upstream = self.0.select(b\"\", 256).unwrap();\n        // Set SNI\n        let peer = Box::new(HttpPeer::new(upstream, true, \"one.one.one.one\".to_string()));\n        Ok(peer)\n    }\n\n    async fn upstream_request_filter(\n        &self,\n        _session: &mut Session,\n        upstream_request: &mut RequestHeader,\n        _ctx: &mut Self::CTX,\n    ) -> Result<()>\n    where\n        Self::CTX: Send + Sync,\n    {\n        upstream_request\n            .insert_header(\"Host\", \"one.one.one.one\")\n            .unwrap();\n        Ok(())\n    }\n\n    async fn request_filter(&self, session: &mut Session, _ctx: &mut Self::CTX) -> Result<bool>\n    where\n        Self::CTX: Send + Sync,\n    {\n        let appid = match self.get_request_appid(session) {\n            None => return Ok(false), // no client appid found, skip rate limiting\n            Some(addr) => addr,\n        };\n\n        // retrieve the current window requests\n        let curr_window_requests = RATE_LIMITER.observe(&appid, 1);\n        if curr_window_requests > MAX_REQ_PER_SEC {\n            // rate limited, return 429\n            let mut header = ResponseHeader::build(429, None).unwrap();\n            header\n                .insert_header(\"X-Rate-Limit-Limit\", MAX_REQ_PER_SEC.to_string())\n                .unwrap();\n            header.insert_header(\"X-Rate-Limit-Remaining\", \"0\").unwrap();\n            header.insert_header(\"X-Rate-Limit-Reset\", \"1\").unwrap();\n            session.set_keepalive(None);\n            session\n                .write_response_header(Box::new(header), true)\n                .await?;\n            return Ok(true);\n        }\n        Ok(false)\n    }\n}\n```\n\n## Testing\nTo use the example above,\n\n1. Run your program with `cargo run`.\n2. Verify the program is working with a few executions of ` curl localhost:6188 -H \"appid:1\" -v`\n   - The first request should work and any later requests that arrive within 1s of a previous request should fail with:\n     ```\n     *   Trying 127.0.0.1:6188...\n     * Connected to localhost (127.0.0.1) port 6188 (#0)\n     > GET / HTTP/1.1\n     > Host: localhost:6188\n     > User-Agent: curl/7.88.1\n     > Accept: */*\n     > appid:1\n     >\n     < HTTP/1.1 429 Too Many Requests\n     < X-Rate-Limit-Limit: 1\n     < X-Rate-Limit-Remaining: 0\n     < X-Rate-Limit-Reset: 1\n     < Date: Sun, 14 Jul 2024 20:29:02 GMT\n     < Connection: close\n     <\n     * Closing connection 0\n     ```\n\n## Complete Example\nYou can run the pre-made example code in the [`pingora-proxy` examples folder](https://github.com/cloudflare/pingora/tree/main/pingora-proxy/examples/rate_limiter.rs) with\n\n```\ncargo run --example rate_limiter\n```\n"
  },
  {
    "path": "docs/user_guide/start_stop.md",
    "content": "# Starting and stopping Pingora server\n\nA pingora server is a regular unprivileged multithreaded process.\n\n## Start\nBy default, the server will run in the foreground.\n\nA Pingora server by default takes the following command-line arguments:\n\n| Argument      | Effect        | default|\n| ------------- |-------------| ----|\n| -d, --daemon | Daemonize the server | false |\n| -t, --test | Test the server conf and then exit (WIP) | false |\n| -c, --conf | The path to the configuration file | empty string |\n| -u, --upgrade | This server should gracefully upgrade a running server | false |\n\n## Stop\nA Pingora server will listen to the following signals.\n\n### SIGINT: fast shutdown\nUpon receiving SIGINT (ctrl + c), the server will exit immediately with no delay. All unfinished requests will be interrupted. This behavior is usually less preferred because it could break requests.\n\n### SIGTERM: graceful shutdown\nUpon receiving SIGTERM, the server will notify all its services to shutdown, wait for some preconfigured time and then exit. This behavior gives requests a grace period to finish.\n\n### SIGQUIT: graceful upgrade\nSimilar to SIGTERM, but the server will also transfer all its listening sockets to a new Pingora server so that there is no downtime during the upgrade. See the [graceful upgrade](graceful.md) section for more details.\n"
  },
  {
    "path": "docs/user_guide/systemd.md",
    "content": "# Systemd integration\n\nA Pingora server doesn't depend on systemd but it can easily be made into a systemd service.\n\n```ini\n[Service]\nType=forking\nPIDFile=/run/pingora.pid\nExecStart=/bin/pingora -d -c /etc/pingora.conf\nExecReload=kill -QUIT $MAINPID\nExecReload=/bin/pingora -u -d -c /etc/pingora.conf\n```\n\nThe example systemd setup integrates Pingora's graceful upgrade into systemd. To upgrade the pingora service, simply install a version of the binary and then call `systemctl reload pingora.service`.\n"
  },
  {
    "path": "pingora/Cargo.toml",
    "content": "[package]\nname = \"pingora\"\nversion = \"0.8.0\"\nauthors = [\"Yuchen Wu <yuchen@cloudflare.com>\"]\nlicense = \"Apache-2.0\"\nedition = \"2021\"\nrepository = \"https://github.com/cloudflare/pingora\"\ndescription = \"\"\"\nA framework to build fast, reliable and programmable networked systems at Internet scale.\n\"\"\"\ncategories = [\"asynchronous\", \"network-programming\"]\nkeywords = [\"async\", \"proxy\", \"http\", \"pingora\"]\n\n# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html\n\n[lib]\nname = \"pingora\"\npath = \"src/lib.rs\"\n\n[package.metadata.docs.rs]\nfeatures = [\"document-features\"]\nrustdoc-args = [\"--cfg\", \"docsrs\"]\n\n[dependencies]\npingora-core = { version = \"0.8.0\", path = \"../pingora-core\", default-features = false }\npingora-http = { version = \"0.8.0\", path = \"../pingora-http\" }\npingora-timeout = { version = \"0.8.0\", path = \"../pingora-timeout\" }\npingora-load-balancing = { version = \"0.8.0\", path = \"../pingora-load-balancing\", optional = true, default-features = false }\npingora-proxy = { version = \"0.8.0\", path = \"../pingora-proxy\", optional = true, default-features = false }\npingora-cache = { version = \"0.8.0\", path = \"../pingora-cache\", optional = true, default-features = false }\n\n# Only used for documenting features, but doesn't work in any other dependency \n# group :(\ndocument-features = { version = \"0.2.10\", optional = true }\n\n[dev-dependencies]\nclap = { version = \"4.5\", features = [\"derive\"] }\ntokio = { workspace = true, features = [\"rt-multi-thread\", \"signal\"] }\nenv_logger = \"0.11\"\nreqwest = { version = \"0.11\", features = [\"rustls\"], default-features = false }\nhyper = \"0.14\"\nasync-trait = { workspace = true }\nhttp = { workspace = true }\nlog = { workspace = true }\nprometheus = \"0.13\"\nonce_cell = { workspace = true }\nbytes = { workspace = true }\nregex = \"1\"\n\n[target.'cfg(unix)'.dev-dependencies]\nhyperlocal = \"0.8\"\njemallocator = \"0.5\"\n\n[features]\ndefault = []\n\n#! ### Tls\n#! Tls is provided by adding one of these features. If no tls-providing feature\n#! is added, only unencrypted http. Only one tls-providing feature can be\n#! selected at a time\n\n## Use [OpenSSL](https://crates.io/crates/openssl) for tls\n##\n## Requires native openssl libraries and build tooling\nopenssl = [\n    \"pingora-core/openssl\",\n    \"pingora-proxy?/openssl\",\n    \"pingora-cache?/openssl\",\n    \"pingora-load-balancing?/openssl\",\n    \"openssl_derived\",\n]\n\n## Use [BoringSSL](https://crates.io/crates/boring) for tls \n##\n## Requires native boring libraries and build tooling\nboringssl = [\n    \"pingora-core/boringssl\",\n    \"pingora-proxy?/boringssl\",\n    \"pingora-cache?/boringssl\",\n    \"pingora-load-balancing?/boringssl\",\n    \"openssl_derived\",\n]\n\n## Use  [s2n-tls](https://crates.io/crates/s2n-tls) for tls\n##\n## Requires native s2n-tls libraries and build tooling\ns2n = [\n    \"pingora-core/s2n\",\n    \"pingora-proxy?/s2n\",\n    \"pingora-cache?/s2n\",\n    \"pingora-load-balancing?/s2n\",\n    \"any_tls\",\n]\n\n## Use  [rustls](https://crates.io/crates/rustls) for tls \n##\n## ⚠️ _Highly Experimental_! ⚠️ Try it, but don't rely on it (yet)\nrustls = [\n    \"pingora-core/rustls\",\n    \"pingora-proxy?/rustls\",\n    \"pingora-cache?/rustls\",\n    \"pingora-load-balancing?/rustls\",\n    \"any_tls\",\n]\n\n#! ### Pingora extensions\n\n## Include the [proxy](crate::proxy) module\n##\n## This feature will include and export `pingora_proxy::prelude::*`\nproxy = [\"pingora-proxy\"]\n\n## Include the [lb](crate::lb) (load-balancing) module\n##\n## This feature will include and export `pingora_load_balancing::prelude::*`\nlb = [\"pingora-load-balancing\", \"proxy\"]\n\n## Include the [cache](crate::cache) module\n##\n## This feature will include and export `pingora_cache::prelude::*`\ncache = [\"pingora-cache\"]\n\n## Enable time/scheduling functionality\ntime = []\n\n## Enable sentry for error notifications\nsentry = [\"pingora-core/sentry\"]\n\n## Enable pre-TLS connection filtering\nconnection_filter = [\n    \"pingora-core/connection_filter\",\n    \"pingora-proxy?/connection_filter\",\n]\n\n\n# These features are intentionally not documented\nopenssl_derived = [\"any_tls\"]\nany_tls = []\npatched_http1 = [\"pingora-core/patched_http1\"]\ndocument-features = [\n    \"dep:document-features\",\n    \"proxy\",\n    \"lb\",\n    \"cache\",\n    \"time\",\n    \"sentry\",\n    \"connection_filter\"\n]\n"
  },
  {
    "path": "pingora/LICENSE",
    "content": "\n                                 Apache License\n                           Version 2.0, January 2004\n                        http://www.apache.org/licenses/\n\n   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION\n\n   1. Definitions.\n\n      \"License\" shall mean the terms and conditions for use, reproduction,\n      and distribution as defined by Sections 1 through 9 of this document.\n\n      \"Licensor\" shall mean the copyright owner or entity authorized by\n      the copyright owner that is granting the License.\n\n      \"Legal Entity\" shall mean the union of the acting entity and all\n      other entities that control, are controlled by, or are under common\n      control with that entity. For the purposes of this definition,\n      \"control\" means (i) the power, direct or indirect, to cause the\n      direction or management of such entity, whether by contract or\n      otherwise, or (ii) ownership of fifty percent (50%) or more of the\n      outstanding shares, or (iii) beneficial ownership of such entity.\n\n      \"You\" (or \"Your\") shall mean an individual or Legal Entity\n      exercising permissions granted by this License.\n\n      \"Source\" form shall mean the preferred form for making modifications,\n      including but not limited to software source code, documentation\n      source, and configuration files.\n\n      \"Object\" form shall mean any form resulting from mechanical\n      transformation or translation of a Source form, including but\n      not limited to compiled object code, generated documentation,\n      and conversions to other media types.\n\n      \"Work\" shall mean the work of authorship, whether in Source or\n      Object form, made available under the License, as indicated by a\n      copyright notice that is included in or attached to the work\n      (an example is provided in the Appendix below).\n\n      \"Derivative Works\" shall mean any work, whether in Source or Object\n      form, that is based on (or derived from) the Work and for which the\n      editorial revisions, annotations, elaborations, or other modifications\n      represent, as a whole, an original work of authorship. For the purposes\n      of this License, Derivative Works shall not include works that remain\n      separable from, or merely link (or bind by name) to the interfaces of,\n      the Work and Derivative Works thereof.\n\n      \"Contribution\" shall mean any work of authorship, including\n      the original version of the Work and any modifications or additions\n      to that Work or Derivative Works thereof, that is intentionally\n      submitted to Licensor for inclusion in the Work by the copyright owner\n      or by an individual or Legal Entity authorized to submit on behalf of\n      the copyright owner. For the purposes of this definition, \"submitted\"\n      means any form of electronic, verbal, or written communication sent\n      to the Licensor or its representatives, including but not limited to\n      communication on electronic mailing lists, source code control systems,\n      and issue tracking systems that are managed by, or on behalf of, the\n      Licensor for the purpose of discussing and improving the Work, but\n      excluding communication that is conspicuously marked or otherwise\n      designated in writing by the copyright owner as \"Not a Contribution.\"\n\n      \"Contributor\" shall mean Licensor and any individual or Legal Entity\n      on behalf of whom a Contribution has been received by Licensor and\n      subsequently incorporated within the Work.\n\n   2. Grant of Copyright License. Subject to the terms and conditions of\n      this License, each Contributor hereby grants to You a perpetual,\n      worldwide, non-exclusive, no-charge, royalty-free, irrevocable\n      copyright license to reproduce, prepare Derivative Works of,\n      publicly display, publicly perform, sublicense, and distribute the\n      Work and such Derivative Works in Source or Object form.\n\n   3. Grant of Patent License. Subject to the terms and conditions of\n      this License, each Contributor hereby grants to You a perpetual,\n      worldwide, non-exclusive, no-charge, royalty-free, irrevocable\n      (except as stated in this section) patent license to make, have made,\n      use, offer to sell, sell, import, and otherwise transfer the Work,\n      where such license applies only to those patent claims licensable\n      by such Contributor that are necessarily infringed by their\n      Contribution(s) alone or by combination of their Contribution(s)\n      with the Work to which such Contribution(s) was submitted. If You\n      institute patent litigation against any entity (including a\n      cross-claim or counterclaim in a lawsuit) alleging that the Work\n      or a Contribution incorporated within the Work constitutes direct\n      or contributory patent infringement, then any patent licenses\n      granted to You under this License for that Work shall terminate\n      as of the date such litigation is filed.\n\n   4. Redistribution. You may reproduce and distribute copies of the\n      Work or Derivative Works thereof in any medium, with or without\n      modifications, and in Source or Object form, provided that You\n      meet the following conditions:\n\n      (a) You must give any other recipients of the Work or\n          Derivative Works a copy of this License; and\n\n      (b) You must cause any modified files to carry prominent notices\n          stating that You changed the files; and\n\n      (c) You must retain, in the Source form of any Derivative Works\n          that You distribute, all copyright, patent, trademark, and\n          attribution notices from the Source form of the Work,\n          excluding those notices that do not pertain to any part of\n          the Derivative Works; and\n\n      (d) If the Work includes a \"NOTICE\" text file as part of its\n          distribution, then any Derivative Works that You distribute must\n          include a readable copy of the attribution notices contained\n          within such NOTICE file, excluding those notices that do not\n          pertain to any part of the Derivative Works, in at least one\n          of the following places: within a NOTICE text file distributed\n          as part of the Derivative Works; within the Source form or\n          documentation, if provided along with the Derivative Works; or,\n          within a display generated by the Derivative Works, if and\n          wherever such third-party notices normally appear. The contents\n          of the NOTICE file are for informational purposes only and\n          do not modify the License. You may add Your own attribution\n          notices within Derivative Works that You distribute, alongside\n          or as an addendum to the NOTICE text from the Work, provided\n          that such additional attribution notices cannot be construed\n          as modifying the License.\n\n      You may add Your own copyright statement to Your modifications and\n      may provide additional or different license terms and conditions\n      for use, reproduction, or distribution of Your modifications, or\n      for any such Derivative Works as a whole, provided Your use,\n      reproduction, and distribution of the Work otherwise complies with\n      the conditions stated in this License.\n\n   5. Submission of Contributions. Unless You explicitly state otherwise,\n      any Contribution intentionally submitted for inclusion in the Work\n      by You to the Licensor shall be under the terms and conditions of\n      this License, without any additional terms or conditions.\n      Notwithstanding the above, nothing herein shall supersede or modify\n      the terms of any separate license agreement you may have executed\n      with Licensor regarding such Contributions.\n\n   6. Trademarks. This License does not grant permission to use the trade\n      names, trademarks, service marks, or product names of the Licensor,\n      except as required for reasonable and customary use in describing the\n      origin of the Work and reproducing the content of the NOTICE file.\n\n   7. Disclaimer of Warranty. Unless required by applicable law or\n      agreed to in writing, Licensor provides the Work (and each\n      Contributor provides its Contributions) on an \"AS IS\" BASIS,\n      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or\n      implied, including, without limitation, any warranties or conditions\n      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A\n      PARTICULAR PURPOSE. You are solely responsible for determining the\n      appropriateness of using or redistributing the Work and assume any\n      risks associated with Your exercise of permissions under this License.\n\n   8. Limitation of Liability. In no event and under no legal theory,\n      whether in tort (including negligence), contract, or otherwise,\n      unless required by applicable law (such as deliberate and grossly\n      negligent acts) or agreed to in writing, shall any Contributor be\n      liable to You for damages, including any direct, indirect, special,\n      incidental, or consequential damages of any character arising as a\n      result of this License or out of the use or inability to use the\n      Work (including but not limited to damages for loss of goodwill,\n      work stoppage, computer failure or malfunction, or any and all\n      other commercial damages or losses), even if such Contributor\n      has been advised of the possibility of such damages.\n\n   9. Accepting Warranty or Additional Liability. While redistributing\n      the Work or Derivative Works thereof, You may choose to offer,\n      and charge a fee for, acceptance of support, warranty, indemnity,\n      or other liability obligations and/or rights consistent with this\n      License. However, in accepting such obligations, You may act only\n      on Your own behalf and on Your sole responsibility, not on behalf\n      of any other Contributor, and only if You agree to indemnify,\n      defend, and hold each Contributor harmless for any liability\n      incurred by, or claims asserted against, such Contributor by reason\n      of your accepting any such warranty or additional liability.\n\n   END OF TERMS AND CONDITIONS\n\n   APPENDIX: How to apply the Apache License to your work.\n\n      To apply the Apache License to your work, attach the following\n      boilerplate notice, with the fields enclosed by brackets \"[]\"\n      replaced with your own identifying information. (Don't include\n      the brackets!)  The text should be enclosed in the appropriate\n      comment syntax for the file format. We also recommend that a\n      file or class name and description of purpose be included on the\n      same \"printed page\" as the copyright notice for easier\n      identification within third-party archives.\n\n   Copyright [yyyy] [name of copyright owner]\n\n   Licensed under the Apache License, Version 2.0 (the \"License\");\n   you may not use this file except in compliance with the License.\n   You may obtain a copy of the License at\n\n       http://www.apache.org/licenses/LICENSE-2.0\n\n   Unless required by applicable law or agreed to in writing, software\n   distributed under the License is distributed on an \"AS IS\" BASIS,\n   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n   See the License for the specific language governing permissions and\n   limitations under the License.\n"
  },
  {
    "path": "pingora/examples/app/echo.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\nuse async_trait::async_trait;\nuse bytes::Bytes;\nuse http::{Response, StatusCode};\nuse log::debug;\nuse once_cell::sync::Lazy;\nuse pingora_timeout::timeout;\nuse prometheus::{register_int_counter, IntCounter};\nuse std::sync::Arc;\nuse std::time::Duration;\nuse tokio::io::{AsyncReadExt, AsyncWriteExt};\n\nuse pingora::apps::http_app::ServeHttp;\nuse pingora::apps::ServerApp;\nuse pingora::protocols::http::ServerSession;\nuse pingora::protocols::Stream;\nuse pingora::server::ShutdownWatch;\n\nstatic REQ_COUNTER: Lazy<IntCounter> =\n    Lazy::new(|| register_int_counter!(\"reg_counter\", \"Number of requests\").unwrap());\n\n#[derive(Clone)]\npub struct EchoApp;\n\n#[async_trait]\nimpl ServerApp for EchoApp {\n    async fn process_new(\n        self: &Arc<Self>,\n        mut io: Stream,\n        _shutdown: &ShutdownWatch,\n    ) -> Option<Stream> {\n        let mut buf = [0; 1024];\n        loop {\n            let n = io.read(&mut buf).await.unwrap();\n            if n == 0 {\n                debug!(\"session closing\");\n                return None;\n            }\n            io.write_all(&buf[0..n]).await.unwrap();\n            io.flush().await.unwrap();\n        }\n    }\n}\n\npub struct HttpEchoApp;\n\n#[async_trait]\nimpl ServeHttp for HttpEchoApp {\n    async fn response(&self, http_stream: &mut ServerSession) -> Response<Vec<u8>> {\n        REQ_COUNTER.inc();\n        // read timeout of 2s\n        let read_timeout = 2000;\n        let body = match timeout(\n            Duration::from_millis(read_timeout),\n            http_stream.read_request_body(),\n        )\n        .await\n        {\n            Ok(res) => match res.unwrap() {\n                Some(bytes) => bytes,\n                None => Bytes::from(\"no body!\"),\n            },\n            Err(_) => {\n                panic!(\"Timed out after {:?}ms\", read_timeout);\n            }\n        };\n\n        Response::builder()\n            .status(StatusCode::OK)\n            .header(http::header::CONTENT_TYPE, \"text/html\")\n            .header(http::header::CONTENT_LENGTH, body.len())\n            .body(body.to_vec())\n            .unwrap()\n    }\n}\n"
  },
  {
    "path": "pingora/examples/app/mod.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\npub mod echo;\npub mod proxy;\n"
  },
  {
    "path": "pingora/examples/app/proxy.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\nuse async_trait::async_trait;\nuse log::debug;\n\nuse std::sync::Arc;\nuse tokio::io::{AsyncReadExt, AsyncWriteExt};\nuse tokio::select;\n\nuse pingora::apps::ServerApp;\nuse pingora::connectors::TransportConnector;\nuse pingora::protocols::Stream;\nuse pingora::server::ShutdownWatch;\nuse pingora::upstreams::peer::BasicPeer;\n\npub struct ProxyApp {\n    client_connector: TransportConnector,\n    proxy_to: BasicPeer,\n}\n\nenum DuplexEvent {\n    DownstreamRead(usize),\n    UpstreamRead(usize),\n}\n\nimpl ProxyApp {\n    pub fn new(proxy_to: BasicPeer) -> Self {\n        ProxyApp {\n            client_connector: TransportConnector::new(None),\n            proxy_to,\n        }\n    }\n\n    async fn duplex(&self, mut server_session: Stream, mut client_session: Stream) {\n        let mut upstream_buf = [0; 1024];\n        let mut downstream_buf = [0; 1024];\n        loop {\n            let downstream_read = server_session.read(&mut upstream_buf);\n            let upstream_read = client_session.read(&mut downstream_buf);\n            let event: DuplexEvent;\n            select! {\n                n = downstream_read => event\n                    = DuplexEvent::DownstreamRead(n.unwrap()),\n                n = upstream_read => event\n                    = DuplexEvent::UpstreamRead(n.unwrap()),\n            }\n            match event {\n                DuplexEvent::DownstreamRead(0) => {\n                    debug!(\"downstream session closing\");\n                    return;\n                }\n                DuplexEvent::UpstreamRead(0) => {\n                    debug!(\"upstream session closing\");\n                    return;\n                }\n                DuplexEvent::DownstreamRead(n) => {\n                    client_session.write_all(&upstream_buf[0..n]).await.unwrap();\n                    client_session.flush().await.unwrap();\n                }\n                DuplexEvent::UpstreamRead(n) => {\n                    server_session\n                        .write_all(&downstream_buf[0..n])\n                        .await\n                        .unwrap();\n                    server_session.flush().await.unwrap();\n                }\n            }\n        }\n    }\n}\n\n#[async_trait]\nimpl ServerApp for ProxyApp {\n    async fn process_new(\n        self: &Arc<Self>,\n        io: Stream,\n        _shutdown: &ShutdownWatch,\n    ) -> Option<Stream> {\n        let client_session = self.client_connector.new_stream(&self.proxy_to).await;\n\n        match client_session {\n            Ok(client_session) => {\n                self.duplex(io, client_session).await;\n                None\n            }\n            Err(e) => {\n                debug!(\"Failed to create client session: {}\", e);\n                None\n            }\n        }\n    }\n}\n"
  },
  {
    "path": "pingora/examples/client.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\nuse pingora::{connectors::http::Connector, prelude::*};\nuse regex::Regex;\n\n#[tokio::main]\nasync fn main() -> Result<()> {\n    let connector = Connector::new(None);\n\n    // create the HTTP session\n    let peer_addr = \"1.1.1.1:443\";\n    let mut peer = HttpPeer::new(peer_addr, true, \"one.one.one.one\".into());\n    peer.options.set_http_version(2, 1);\n    let (mut http, _reused) = connector.get_http_session(&peer).await?;\n\n    // perform a GET request\n    let mut new_request = RequestHeader::build(\"GET\", b\"/\", None)?;\n    new_request.insert_header(\"Host\", \"one.one.one.one\")?;\n    http.write_request_header(Box::new(new_request)).await?;\n\n    // Servers usually don't respond until the full request body is read.\n    http.finish_request_body().await?;\n    http.read_response_header().await?;\n\n    // display the headers from the response\n    if let Some(header) = http.response_header() {\n        println!(\"{header:#?}\");\n    } else {\n        return Error::e_explain(ErrorType::InvalidHTTPHeader, \"No response header\");\n    };\n\n    // collect the response body\n    let mut response_body = String::new();\n    while let Some(chunk) = http.read_response_body().await? {\n        response_body.push_str(&String::from_utf8_lossy(&chunk));\n    }\n\n    // verify that the response body is valid HTML by displaying the page <title>\n    let re = Regex::new(r\"<title>(.*?)</title>\")\n        .or_err(ErrorType::InternalError, \"Failed to compile regex\")?;\n    if let Some(title) = re\n        .captures(&response_body)\n        .and_then(|caps| caps.get(1).map(|match_| match_.as_str()))\n    {\n        println!(\"Page Title: {title}\");\n    } else {\n        return Error::e_explain(\n            ErrorType::new(\"InvalidHTML\"),\n            \"No <title> found in response body\",\n        );\n    }\n\n    // gracefully release the connection\n    connector\n        .release_http_session(http, &peer, Some(std::time::Duration::from_secs(5)))\n        .await;\n\n    Ok(())\n}\n"
  },
  {
    "path": "pingora/examples/server.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n#[global_allocator]\nstatic GLOBAL: jemallocator::Jemalloc = jemallocator::Jemalloc;\n\nuse pingora::listeners::tls::TlsSettings;\nuse pingora::protocols::TcpKeepalive;\nuse pingora::server::configuration::Opt;\nuse pingora::server::{Server, ShutdownWatch};\nuse pingora::services::background::{background_service, BackgroundService};\nuse pingora::services::listening::Service as ListeningService;\nuse pingora::services::ServiceWithDependents;\n\nuse async_trait::async_trait;\nuse clap::Parser;\nuse tokio::time::interval;\n\nuse std::time::Duration;\n\nmod app;\nmod service;\n\npub struct ExampleBackgroundService;\n#[async_trait]\nimpl BackgroundService for ExampleBackgroundService {\n    async fn start(&self, mut shutdown: ShutdownWatch) {\n        let mut period = interval(Duration::from_secs(1));\n        loop {\n            tokio::select! {\n                _ = shutdown.changed() => {\n                    // shutdown\n                    break;\n                }\n                _ = period.tick() => {\n                    // do some work\n                    // ...\n                }\n            }\n        }\n    }\n}\n#[cfg(feature = \"openssl_derived\")]\nmod boringssl_openssl {\n    use super::*;\n    use pingora::tls::pkey::{PKey, Private};\n    use pingora::tls::x509::X509;\n\n    pub(super) struct DynamicCert {\n        cert: X509,\n        key: PKey<Private>,\n    }\n\n    impl DynamicCert {\n        pub(super) fn new(cert: &str, key: &str) -> Box<Self> {\n            let cert_bytes = std::fs::read(cert).unwrap();\n            let cert = X509::from_pem(&cert_bytes).unwrap();\n\n            let key_bytes = std::fs::read(key).unwrap();\n            let key = PKey::private_key_from_pem(&key_bytes).unwrap();\n            Box::new(DynamicCert { cert, key })\n        }\n    }\n\n    #[async_trait]\n    impl pingora::listeners::TlsAccept for DynamicCert {\n        async fn certificate_callback(&self, ssl: &mut pingora::tls::ssl::SslRef) {\n            use pingora::tls::ext;\n            ext::ssl_use_certificate(ssl, &self.cert).unwrap();\n            ext::ssl_use_private_key(ssl, &self.key).unwrap();\n        }\n    }\n}\n\nconst USAGE: &str = r#\"\nUsage\nport 6142: TCP echo server\nnc 127.0.0.1 6142\n\nport 6143: TLS echo server\nopenssl s_client -connect 127.0.0.1:6143\n\nport 6145: Http echo server\ncurl http://127.0.0.1:6145 -v -d 'hello'\n\nport 6148: Https echo server\ncurl https://127.0.0.1:6148 -vk -d 'hello'\n\nport 6141: TCP proxy\ncurl http://127.0.0.1:6141 -v -H 'host: 1.1.1.1'\n\nport 6144: TLS proxy\ncurl https://127.0.0.1:6144 -vk -H 'host: one.one.one.one' -o /dev/null\n\nport 6150: metrics endpoint\ncurl http://127.0.0.1:6150\n\"#;\n\npub fn main() {\n    env_logger::init();\n\n    print!(\"{USAGE}\");\n\n    let opt = Some(Opt::parse());\n    let mut my_server = Server::new(opt).unwrap();\n    my_server.bootstrap();\n\n    let cert_path = format!(\"{}/tests/keys/server.crt\", env!(\"CARGO_MANIFEST_DIR\"));\n    let key_path = format!(\"{}/tests/keys/key.pem\", env!(\"CARGO_MANIFEST_DIR\"));\n\n    let mut echo_service = service::echo::echo_service();\n    echo_service.add_tcp(\"127.0.0.1:6142\");\n    echo_service\n        .add_tls(\"0.0.0.0:6143\", &cert_path, &key_path)\n        .unwrap();\n\n    let mut echo_service_http = service::echo::echo_service_http();\n\n    let mut options = pingora::listeners::TcpSocketOptions::default();\n    options.tcp_fastopen = Some(10);\n    options.tcp_keepalive = Some(TcpKeepalive {\n        idle: Duration::from_secs(60),\n        interval: Duration::from_secs(5),\n        count: 5,\n        #[cfg(target_os = \"linux\")]\n        user_timeout: Duration::from_secs(85),\n    });\n\n    echo_service_http.add_tcp_with_settings(\"0.0.0.0:6145\", options);\n    echo_service_http.add_uds(\"/tmp/echo.sock\", None);\n\n    let mut tls_settings;\n\n    // NOTE: dynamic certificate callback is only supported with BoringSSL/OpenSSL\n    #[cfg(feature = \"openssl_derived\")]\n    {\n        use std::ops::DerefMut;\n\n        let dynamic_cert = boringssl_openssl::DynamicCert::new(&cert_path, &key_path);\n        tls_settings = TlsSettings::with_callbacks(dynamic_cert).unwrap();\n        // by default intermediate supports both TLS 1.2 and 1.3. We force to tls 1.2 just for the demo\n\n        tls_settings\n            .deref_mut()\n            .deref_mut()\n            .set_max_proto_version(Some(pingora::tls::ssl::SslVersion::TLS1_2))\n            .unwrap();\n    }\n    #[cfg(feature = \"rustls\")]\n    {\n        tls_settings = TlsSettings::intermediate(&cert_path, &key_path).unwrap();\n    }\n    #[cfg(feature = \"s2n\")]\n    {\n        tls_settings = TlsSettings::intermediate(&cert_path, &key_path).unwrap();\n    }\n    #[cfg(not(feature = \"any_tls\"))]\n    {\n        tls_settings = TlsSettings;\n    }\n\n    tls_settings.enable_h2();\n    echo_service_http.add_tls_with_settings(\"0.0.0.0:6148\", None, tls_settings);\n\n    let proxy_service = service::proxy::proxy_service(\n        \"0.0.0.0:6141\", // listen\n        \"1.1.1.1:80\",   // proxy to\n    );\n\n    let proxy_service_ssl = service::proxy::proxy_service_tls(\n        \"0.0.0.0:6144\",    // listen\n        \"1.1.1.1:443\",     // proxy to\n        \"one.one.one.one\", // SNI\n        &cert_path,\n        &key_path,\n    );\n\n    let mut prometheus_service_http = ListeningService::prometheus_http_service();\n    prometheus_service_http.add_tcp(\"127.0.0.1:6150\");\n\n    let background_service = background_service(\"example\", ExampleBackgroundService {});\n\n    let services: Vec<Box<dyn ServiceWithDependents>> = vec![\n        Box::new(echo_service),\n        Box::new(echo_service_http),\n        Box::new(proxy_service),\n        Box::new(proxy_service_ssl),\n        Box::new(prometheus_service_http),\n        Box::new(background_service),\n    ];\n    my_server.add_services(services);\n    my_server.run_forever();\n}\n"
  },
  {
    "path": "pingora/examples/service/echo.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\nuse crate::app::echo::{EchoApp, HttpEchoApp};\nuse pingora::apps::http_app::HttpServer;\nuse pingora::services::listening::Service;\n\npub fn echo_service() -> Service<EchoApp> {\n    Service::new(\"Echo Service\".to_string(), EchoApp)\n}\n\npub fn echo_service_http() -> Service<HttpServer<HttpEchoApp>> {\n    let server = HttpServer::new_app(HttpEchoApp);\n    Service::new(\"Echo Service HTTP\".to_string(), server)\n}\n"
  },
  {
    "path": "pingora/examples/service/mod.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\npub mod echo;\npub mod proxy;\n"
  },
  {
    "path": "pingora/examples/service/proxy.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\nuse crate::app::proxy::ProxyApp;\nuse pingora_core::listeners::Listeners;\nuse pingora_core::services::listening::Service;\nuse pingora_core::upstreams::peer::BasicPeer;\n\npub fn proxy_service(addr: &str, proxy_addr: &str) -> Service<ProxyApp> {\n    let proxy_to = BasicPeer::new(proxy_addr);\n\n    Service::with_listeners(\n        \"Proxy Service\".to_string(),\n        Listeners::tcp(addr),\n        ProxyApp::new(proxy_to),\n    )\n}\n\npub fn proxy_service_tls(\n    addr: &str,\n    proxy_addr: &str,\n    proxy_sni: &str,\n    cert_path: &str,\n    key_path: &str,\n) -> Service<ProxyApp> {\n    let mut proxy_to = BasicPeer::new(proxy_addr);\n    // set SNI to enable TLS\n    proxy_to.sni = proxy_sni.into();\n    Service::with_listeners(\n        \"Proxy Service TLS\".to_string(),\n        Listeners::tls(addr, cert_path, key_path).unwrap(),\n        ProxyApp::new(proxy_to),\n    )\n}\n"
  },
  {
    "path": "pingora/src/lib.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n#![warn(clippy::all)]\n#![allow(clippy::new_without_default)]\n#![allow(clippy::type_complexity)]\n#![allow(clippy::match_wild_err_arm)]\n#![allow(clippy::missing_safety_doc)]\n#![allow(clippy::upper_case_acronyms)]\n// This enables the feature that labels modules that are only available with\n// certain pingora features\n#![cfg_attr(docsrs, feature(doc_cfg))]\n\n//! # Pingora\n//!\n//! Pingora is a framework to build fast, reliable and programmable networked systems at Internet scale.\n//!\n//! # Features\n//! - Http 1.x and Http 2\n//! - Modern TLS with OpenSSL or BoringSSL (FIPS compatible)\n//! - Zero downtime upgrade\n//!\n//! # Usage\n//! This crate provides low level service and protocol implementation and abstraction.\n//!\n//! If looking to build a (reverse) proxy, see [`pingora-proxy`](https://docs.rs/pingora-proxy) crate.\n//!\n//! # Feature flags\n#![cfg_attr(\n    feature = \"document-features\",\n    cfg_attr(doc, doc = ::document_features::document_features!())\n)]\n\npub use pingora_core::*;\n\n/// HTTP header objects that preserve http header cases\npub mod http {\n    pub use pingora_http::*;\n}\n\n#[cfg(feature = \"cache\")]\n#[cfg_attr(docsrs, doc(cfg(feature = \"cache\")))]\n/// Caching services and tooling\npub mod cache {\n    pub use pingora_cache::*;\n}\n\n#[cfg(feature = \"lb\")]\n#[cfg_attr(docsrs, doc(cfg(feature = \"lb\")))]\n/// Load balancing recipes\npub mod lb {\n    pub use pingora_load_balancing::*;\n}\n\n#[cfg(feature = \"proxy\")]\n#[cfg_attr(docsrs, doc(cfg(feature = \"proxy\")))]\n/// Proxying recipes\npub mod proxy {\n    pub use pingora_proxy::*;\n}\n\n#[cfg(feature = \"time\")]\n#[cfg_attr(docsrs, doc(cfg(feature = \"time\")))]\n/// Timeouts and other useful time utilities\npub mod time {\n    pub use pingora_timeout::*;\n}\n\n/// A useful set of types for getting started\npub mod prelude {\n    pub use pingora_core::prelude::*;\n    pub use pingora_http::prelude::*;\n    pub use pingora_timeout::*;\n\n    #[cfg(feature = \"cache\")]\n    #[cfg_attr(docsrs, doc(cfg(feature = \"cache\")))]\n    pub use pingora_cache::prelude::*;\n\n    #[cfg(feature = \"lb\")]\n    #[cfg_attr(docsrs, doc(cfg(feature = \"lb\")))]\n    pub use pingora_load_balancing::prelude::*;\n\n    #[cfg(feature = \"proxy\")]\n    #[cfg_attr(docsrs, doc(cfg(feature = \"proxy\")))]\n    pub use pingora_proxy::prelude::*;\n\n    #[cfg(feature = \"time\")]\n    #[cfg_attr(docsrs, doc(cfg(feature = \"time\")))]\n    pub use pingora_timeout::*;\n}\n"
  },
  {
    "path": "pingora/tests/pingora_conf.yaml",
    "content": "---\nversion: 1\nclient_bind_to_ipv4:\n    - 127.0.0.2\nca_file: tests/keys/server.crt"
  },
  {
    "path": "pingora-boringssl/Cargo.toml",
    "content": "[package]\nname = \"pingora-boringssl\"\nversion = \"0.8.0\"\nauthors = [\"Yuchen Wu <yuchen@cloudflare.com>\"]\nlicense = \"Apache-2.0\"\nedition = \"2021\"\nrepository = \"https://github.com/cloudflare/pingora\"\ncategories = [\"asynchronous\", \"network-programming\"]\nkeywords = [\"async\", \"tls\", \"ssl\", \"pingora\"]\ndescription = \"\"\"\nBoringSSL async APIs for Pingora.\n\"\"\"\n\n# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html\n[lib]\nname = \"pingora_boringssl\"\npath = \"src/lib.rs\"\n\n[dependencies]\nboring = { version = \"4.5\", features = [\"pq-experimental\"] }\nboring-sys = \"4.5\"\nfutures-util = { version = \"0.3\", default-features = false }\ntokio = { workspace = true, features = [\"io-util\", \"net\", \"macros\", \"rt-multi-thread\"] }\nlibc = \"0.2.70\"\nforeign-types-shared = { version = \"0.3\" }\n\n\n[dev-dependencies]\ntokio-test = \"0.4\"\ntokio = { workspace = true, features = [\"full\"] }\n\n[features]\ndefault = []\npq_use_second_keyshare = []\n# waiting for boring-rs release\nread_uninit = []\n"
  },
  {
    "path": "pingora-boringssl/LICENSE",
    "content": "\n                                 Apache License\n                           Version 2.0, January 2004\n                        http://www.apache.org/licenses/\n\n   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION\n\n   1. Definitions.\n\n      \"License\" shall mean the terms and conditions for use, reproduction,\n      and distribution as defined by Sections 1 through 9 of this document.\n\n      \"Licensor\" shall mean the copyright owner or entity authorized by\n      the copyright owner that is granting the License.\n\n      \"Legal Entity\" shall mean the union of the acting entity and all\n      other entities that control, are controlled by, or are under common\n      control with that entity. For the purposes of this definition,\n      \"control\" means (i) the power, direct or indirect, to cause the\n      direction or management of such entity, whether by contract or\n      otherwise, or (ii) ownership of fifty percent (50%) or more of the\n      outstanding shares, or (iii) beneficial ownership of such entity.\n\n      \"You\" (or \"Your\") shall mean an individual or Legal Entity\n      exercising permissions granted by this License.\n\n      \"Source\" form shall mean the preferred form for making modifications,\n      including but not limited to software source code, documentation\n      source, and configuration files.\n\n      \"Object\" form shall mean any form resulting from mechanical\n      transformation or translation of a Source form, including but\n      not limited to compiled object code, generated documentation,\n      and conversions to other media types.\n\n      \"Work\" shall mean the work of authorship, whether in Source or\n      Object form, made available under the License, as indicated by a\n      copyright notice that is included in or attached to the work\n      (an example is provided in the Appendix below).\n\n      \"Derivative Works\" shall mean any work, whether in Source or Object\n      form, that is based on (or derived from) the Work and for which the\n      editorial revisions, annotations, elaborations, or other modifications\n      represent, as a whole, an original work of authorship. For the purposes\n      of this License, Derivative Works shall not include works that remain\n      separable from, or merely link (or bind by name) to the interfaces of,\n      the Work and Derivative Works thereof.\n\n      \"Contribution\" shall mean any work of authorship, including\n      the original version of the Work and any modifications or additions\n      to that Work or Derivative Works thereof, that is intentionally\n      submitted to Licensor for inclusion in the Work by the copyright owner\n      or by an individual or Legal Entity authorized to submit on behalf of\n      the copyright owner. For the purposes of this definition, \"submitted\"\n      means any form of electronic, verbal, or written communication sent\n      to the Licensor or its representatives, including but not limited to\n      communication on electronic mailing lists, source code control systems,\n      and issue tracking systems that are managed by, or on behalf of, the\n      Licensor for the purpose of discussing and improving the Work, but\n      excluding communication that is conspicuously marked or otherwise\n      designated in writing by the copyright owner as \"Not a Contribution.\"\n\n      \"Contributor\" shall mean Licensor and any individual or Legal Entity\n      on behalf of whom a Contribution has been received by Licensor and\n      subsequently incorporated within the Work.\n\n   2. Grant of Copyright License. Subject to the terms and conditions of\n      this License, each Contributor hereby grants to You a perpetual,\n      worldwide, non-exclusive, no-charge, royalty-free, irrevocable\n      copyright license to reproduce, prepare Derivative Works of,\n      publicly display, publicly perform, sublicense, and distribute the\n      Work and such Derivative Works in Source or Object form.\n\n   3. Grant of Patent License. Subject to the terms and conditions of\n      this License, each Contributor hereby grants to You a perpetual,\n      worldwide, non-exclusive, no-charge, royalty-free, irrevocable\n      (except as stated in this section) patent license to make, have made,\n      use, offer to sell, sell, import, and otherwise transfer the Work,\n      where such license applies only to those patent claims licensable\n      by such Contributor that are necessarily infringed by their\n      Contribution(s) alone or by combination of their Contribution(s)\n      with the Work to which such Contribution(s) was submitted. If You\n      institute patent litigation against any entity (including a\n      cross-claim or counterclaim in a lawsuit) alleging that the Work\n      or a Contribution incorporated within the Work constitutes direct\n      or contributory patent infringement, then any patent licenses\n      granted to You under this License for that Work shall terminate\n      as of the date such litigation is filed.\n\n   4. Redistribution. You may reproduce and distribute copies of the\n      Work or Derivative Works thereof in any medium, with or without\n      modifications, and in Source or Object form, provided that You\n      meet the following conditions:\n\n      (a) You must give any other recipients of the Work or\n          Derivative Works a copy of this License; and\n\n      (b) You must cause any modified files to carry prominent notices\n          stating that You changed the files; and\n\n      (c) You must retain, in the Source form of any Derivative Works\n          that You distribute, all copyright, patent, trademark, and\n          attribution notices from the Source form of the Work,\n          excluding those notices that do not pertain to any part of\n          the Derivative Works; and\n\n      (d) If the Work includes a \"NOTICE\" text file as part of its\n          distribution, then any Derivative Works that You distribute must\n          include a readable copy of the attribution notices contained\n          within such NOTICE file, excluding those notices that do not\n          pertain to any part of the Derivative Works, in at least one\n          of the following places: within a NOTICE text file distributed\n          as part of the Derivative Works; within the Source form or\n          documentation, if provided along with the Derivative Works; or,\n          within a display generated by the Derivative Works, if and\n          wherever such third-party notices normally appear. The contents\n          of the NOTICE file are for informational purposes only and\n          do not modify the License. You may add Your own attribution\n          notices within Derivative Works that You distribute, alongside\n          or as an addendum to the NOTICE text from the Work, provided\n          that such additional attribution notices cannot be construed\n          as modifying the License.\n\n      You may add Your own copyright statement to Your modifications and\n      may provide additional or different license terms and conditions\n      for use, reproduction, or distribution of Your modifications, or\n      for any such Derivative Works as a whole, provided Your use,\n      reproduction, and distribution of the Work otherwise complies with\n      the conditions stated in this License.\n\n   5. Submission of Contributions. Unless You explicitly state otherwise,\n      any Contribution intentionally submitted for inclusion in the Work\n      by You to the Licensor shall be under the terms and conditions of\n      this License, without any additional terms or conditions.\n      Notwithstanding the above, nothing herein shall supersede or modify\n      the terms of any separate license agreement you may have executed\n      with Licensor regarding such Contributions.\n\n   6. Trademarks. This License does not grant permission to use the trade\n      names, trademarks, service marks, or product names of the Licensor,\n      except as required for reasonable and customary use in describing the\n      origin of the Work and reproducing the content of the NOTICE file.\n\n   7. Disclaimer of Warranty. Unless required by applicable law or\n      agreed to in writing, Licensor provides the Work (and each\n      Contributor provides its Contributions) on an \"AS IS\" BASIS,\n      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or\n      implied, including, without limitation, any warranties or conditions\n      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A\n      PARTICULAR PURPOSE. You are solely responsible for determining the\n      appropriateness of using or redistributing the Work and assume any\n      risks associated with Your exercise of permissions under this License.\n\n   8. Limitation of Liability. In no event and under no legal theory,\n      whether in tort (including negligence), contract, or otherwise,\n      unless required by applicable law (such as deliberate and grossly\n      negligent acts) or agreed to in writing, shall any Contributor be\n      liable to You for damages, including any direct, indirect, special,\n      incidental, or consequential damages of any character arising as a\n      result of this License or out of the use or inability to use the\n      Work (including but not limited to damages for loss of goodwill,\n      work stoppage, computer failure or malfunction, or any and all\n      other commercial damages or losses), even if such Contributor\n      has been advised of the possibility of such damages.\n\n   9. Accepting Warranty or Additional Liability. While redistributing\n      the Work or Derivative Works thereof, You may choose to offer,\n      and charge a fee for, acceptance of support, warranty, indemnity,\n      or other liability obligations and/or rights consistent with this\n      License. However, in accepting such obligations, You may act only\n      on Your own behalf and on Your sole responsibility, not on behalf\n      of any other Contributor, and only if You agree to indemnify,\n      defend, and hold each Contributor harmless for any liability\n      incurred by, or claims asserted against, such Contributor by reason\n      of your accepting any such warranty or additional liability.\n\n   END OF TERMS AND CONDITIONS\n\n   APPENDIX: How to apply the Apache License to your work.\n\n      To apply the Apache License to your work, attach the following\n      boilerplate notice, with the fields enclosed by brackets \"[]\"\n      replaced with your own identifying information. (Don't include\n      the brackets!)  The text should be enclosed in the appropriate\n      comment syntax for the file format. We also recommend that a\n      file or class name and description of purpose be included on the\n      same \"printed page\" as the copyright notice for easier\n      identification within third-party archives.\n\n   Copyright [yyyy] [name of copyright owner]\n\n   Licensed under the Apache License, Version 2.0 (the \"License\");\n   you may not use this file except in compliance with the License.\n   You may obtain a copy of the License at\n\n       http://www.apache.org/licenses/LICENSE-2.0\n\n   Unless required by applicable law or agreed to in writing, software\n   distributed under the License is distributed on an \"AS IS\" BASIS,\n   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n   See the License for the specific language governing permissions and\n   limitations under the License.\n"
  },
  {
    "path": "pingora-boringssl/src/boring_tokio.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! This file reimplements tokio-boring with the [overhauled](https://github.com/sfackler/tokio-openssl/commit/56f6618ab619f3e431fa8feec2d20913bf1473aa)\n//! tokio-openssl interface while the tokio APIs from official [boring] crate is not yet caught up to it.\n\nuse boring::error::ErrorStack;\nuse boring::ssl::{self, ErrorCode, ShutdownResult, Ssl, SslRef, SslStream as SslStreamCore};\nuse futures_util::future;\nuse std::fmt;\nuse std::io::{self, Read, Write};\nuse std::pin::Pin;\nuse std::task::{Context, Poll};\nuse tokio::io::{AsyncRead, AsyncWrite, ReadBuf};\n\nstruct StreamWrapper<S> {\n    stream: S,\n    context: usize,\n}\n\nimpl<S> fmt::Debug for StreamWrapper<S>\nwhere\n    S: fmt::Debug,\n{\n    fn fmt(&self, fmt: &mut fmt::Formatter<'_>) -> fmt::Result {\n        fmt::Debug::fmt(&self.stream, fmt)\n    }\n}\n\nimpl<S> StreamWrapper<S> {\n    /// # Safety\n    ///\n    /// Must be called with `context` set to a valid pointer to a live `Context` object, and the\n    /// wrapper must be pinned in memory.\n    unsafe fn parts(&mut self) -> (Pin<&mut S>, &mut Context<'_>) {\n        debug_assert_ne!(self.context, 0);\n        let stream = Pin::new_unchecked(&mut self.stream);\n        let context = &mut *(self.context as *mut _);\n        (stream, context)\n    }\n}\n\nimpl<S> Read for StreamWrapper<S>\nwhere\n    S: AsyncRead,\n{\n    fn read(&mut self, buf: &mut [u8]) -> io::Result<usize> {\n        let (stream, cx) = unsafe { self.parts() };\n        let mut buf = ReadBuf::new(buf);\n        match stream.poll_read(cx, &mut buf)? {\n            Poll::Ready(()) => Ok(buf.filled().len()),\n            Poll::Pending => Err(io::Error::from(io::ErrorKind::WouldBlock)),\n        }\n    }\n}\n\nimpl<S> Write for StreamWrapper<S>\nwhere\n    S: AsyncWrite,\n{\n    fn write(&mut self, buf: &[u8]) -> io::Result<usize> {\n        let (stream, cx) = unsafe { self.parts() };\n        match stream.poll_write(cx, buf) {\n            Poll::Ready(r) => r,\n            Poll::Pending => Err(io::Error::from(io::ErrorKind::WouldBlock)),\n        }\n    }\n\n    fn flush(&mut self) -> io::Result<()> {\n        let (stream, cx) = unsafe { self.parts() };\n        match stream.poll_flush(cx) {\n            Poll::Ready(r) => r,\n            Poll::Pending => Err(io::Error::from(io::ErrorKind::WouldBlock)),\n        }\n    }\n}\n\nfn cvt<T>(r: io::Result<T>) -> Poll<io::Result<T>> {\n    match r {\n        Ok(v) => Poll::Ready(Ok(v)),\n        Err(ref e) if e.kind() == io::ErrorKind::WouldBlock => Poll::Pending,\n        Err(e) => Poll::Ready(Err(e)),\n    }\n}\n\nfn cvt_ossl<T>(r: Result<T, ssl::Error>) -> Poll<Result<T, ssl::Error>> {\n    match r {\n        Ok(v) => Poll::Ready(Ok(v)),\n        Err(e) => match e.code() {\n            ErrorCode::WANT_READ | ErrorCode::WANT_WRITE => Poll::Pending,\n            _ => Poll::Ready(Err(e)),\n        },\n    }\n}\n\n/// An asynchronous version of [`boring::ssl::SslStream`].\n#[derive(Debug)]\npub struct SslStream<S>(SslStreamCore<StreamWrapper<S>>);\n\nimpl<S: AsyncRead + AsyncWrite> SslStream<S> {\n    /// Like [`SslStream::new`](ssl::SslStream::new).\n    pub fn new(ssl: Ssl, stream: S) -> Result<Self, ErrorStack> {\n        SslStreamCore::new(ssl, StreamWrapper { stream, context: 0 }).map(SslStream)\n    }\n\n    /// Like [`SslStream::connect`](ssl::SslStream::connect).\n    pub fn poll_connect(\n        self: Pin<&mut Self>,\n        cx: &mut Context<'_>,\n    ) -> Poll<Result<(), ssl::Error>> {\n        self.with_context(cx, |s| cvt_ossl(s.connect()))\n    }\n\n    /// A convenience method wrapping [`poll_connect`](Self::poll_connect).\n    pub async fn connect(mut self: Pin<&mut Self>) -> Result<(), ssl::Error> {\n        future::poll_fn(|cx| self.as_mut().poll_connect(cx)).await\n    }\n\n    /// Like [`SslStream::accept`](ssl::SslStream::accept).\n    pub fn poll_accept(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Result<(), ssl::Error>> {\n        self.with_context(cx, |s| cvt_ossl(s.accept()))\n    }\n\n    /// A convenience method wrapping [`poll_accept`](Self::poll_accept).\n    pub async fn accept(mut self: Pin<&mut Self>) -> Result<(), ssl::Error> {\n        future::poll_fn(|cx| self.as_mut().poll_accept(cx)).await\n    }\n\n    /// Like [`SslStream::do_handshake`](ssl::SslStream::do_handshake).\n    pub fn poll_do_handshake(\n        self: Pin<&mut Self>,\n        cx: &mut Context<'_>,\n    ) -> Poll<Result<(), ssl::Error>> {\n        self.with_context(cx, |s| cvt_ossl(s.do_handshake()))\n    }\n\n    /// A convenience method wrapping [`poll_do_handshake`](Self::poll_do_handshake).\n    pub async fn do_handshake(mut self: Pin<&mut Self>) -> Result<(), ssl::Error> {\n        future::poll_fn(|cx| self.as_mut().poll_do_handshake(cx)).await\n    }\n\n    // TODO: early data\n}\n\nimpl<S> SslStream<S> {\n    /// Returns a shared reference to the `Ssl` object associated with this stream.\n    pub fn ssl(&self) -> &SslRef {\n        self.0.ssl()\n    }\n\n    /// Returns a shared reference to the underlying stream.\n    pub fn get_ref(&self) -> &S {\n        &self.0.get_ref().stream\n    }\n\n    /// Returns a mutable reference to the underlying stream.\n    pub fn get_mut(&mut self) -> &mut S {\n        &mut self.0.get_mut().stream\n    }\n\n    /// Returns a pinned mutable reference to the underlying stream.\n    pub fn get_pin_mut(self: Pin<&mut Self>) -> Pin<&mut S> {\n        unsafe { Pin::new_unchecked(&mut self.get_unchecked_mut().0.get_mut().stream) }\n    }\n\n    fn with_context<F, R>(self: Pin<&mut Self>, ctx: &mut Context<'_>, f: F) -> R\n    where\n        F: FnOnce(&mut SslStreamCore<StreamWrapper<S>>) -> R,\n    {\n        let this = unsafe { self.get_unchecked_mut() };\n        this.0.get_mut().context = ctx as *mut _ as usize;\n        let r = f(&mut this.0);\n        this.0.get_mut().context = 0;\n        r\n    }\n}\n\n#[cfg(feature = \"read_uninit\")]\nimpl<S> AsyncRead for SslStream<S>\nwhere\n    S: AsyncRead + AsyncWrite,\n{\n    fn poll_read(\n        self: Pin<&mut Self>,\n        ctx: &mut Context<'_>,\n        buf: &mut ReadBuf<'_>,\n    ) -> Poll<io::Result<()>> {\n        self.with_context(ctx, |s| {\n            // SAFETY: read_uninit does not de-initialize the buffer.\n            match cvt(s.read_uninit(unsafe { buf.unfilled_mut() }))? {\n                Poll::Ready(nread) => {\n                    unsafe {\n                        buf.assume_init(nread);\n                    }\n                    buf.advance(nread);\n                    Poll::Ready(Ok(()))\n                }\n                Poll::Pending => Poll::Pending,\n            }\n        })\n    }\n}\n\n#[cfg(not(feature = \"read_uninit\"))]\nimpl<S> AsyncRead for SslStream<S>\nwhere\n    S: AsyncRead + AsyncWrite,\n{\n    fn poll_read(\n        self: Pin<&mut Self>,\n        ctx: &mut Context<'_>,\n        buf: &mut ReadBuf<'_>,\n    ) -> Poll<io::Result<()>> {\n        self.with_context(ctx, |s| {\n            // This isn't really \"proper\", but rust-openssl doesn't currently expose a suitable interface even though\n            // OpenSSL itself doesn't require the buffer to be initialized. So this is good enough for now.\n            let slice = unsafe {\n                let buf = buf.unfilled_mut();\n                std::slice::from_raw_parts_mut(buf.as_mut_ptr().cast::<u8>(), buf.len())\n            };\n            match cvt(s.read(slice))? {\n                Poll::Ready(nread) => {\n                    unsafe {\n                        buf.assume_init(nread);\n                    }\n                    buf.advance(nread);\n                    Poll::Ready(Ok(()))\n                }\n                Poll::Pending => Poll::Pending,\n            }\n        })\n    }\n}\n\nimpl<S> AsyncWrite for SslStream<S>\nwhere\n    S: AsyncRead + AsyncWrite,\n{\n    fn poll_write(self: Pin<&mut Self>, ctx: &mut Context, buf: &[u8]) -> Poll<io::Result<usize>> {\n        self.with_context(ctx, |s| cvt(s.write(buf)))\n    }\n\n    fn poll_flush(self: Pin<&mut Self>, ctx: &mut Context) -> Poll<io::Result<()>> {\n        self.with_context(ctx, |s| cvt(s.flush()))\n    }\n\n    fn poll_shutdown(mut self: Pin<&mut Self>, ctx: &mut Context) -> Poll<io::Result<()>> {\n        match self.as_mut().with_context(ctx, |s| s.shutdown()) {\n            Ok(ShutdownResult::Sent) | Ok(ShutdownResult::Received) => {}\n            Err(ref e) if e.code() == ErrorCode::ZERO_RETURN => {}\n            Err(ref e) if e.code() == ErrorCode::WANT_READ || e.code() == ErrorCode::WANT_WRITE => {\n                return Poll::Pending;\n            }\n            Err(e) => {\n                return Poll::Ready(Err(e.into_io_error().unwrap_or_else(io::Error::other)));\n            }\n        }\n\n        self.get_pin_mut().poll_shutdown(ctx)\n    }\n}\n\n#[tokio::test]\nasync fn test_google() {\n    use boring::ssl;\n    use std::net::ToSocketAddrs;\n    use std::pin::Pin;\n    use tokio::io::{AsyncReadExt, AsyncWriteExt};\n    use tokio::net::TcpStream;\n\n    let addr = \"8.8.8.8:443\".to_socket_addrs().unwrap().next().unwrap();\n    let stream = TcpStream::connect(&addr).await.unwrap();\n\n    let ssl_context = ssl::SslContext::builder(ssl::SslMethod::tls())\n        .unwrap()\n        .build();\n    let ssl = ssl::Ssl::new(&ssl_context).unwrap();\n    let mut stream = crate::tokio_ssl::SslStream::new(ssl, stream).unwrap();\n\n    Pin::new(&mut stream).connect().await.unwrap();\n\n    stream.write_all(b\"GET / HTTP/1.0\\r\\n\\r\\n\").await.unwrap();\n\n    let mut buf = vec![];\n    stream.read_to_end(&mut buf).await.unwrap();\n    let response = String::from_utf8_lossy(&buf);\n    let response = response.trim_end();\n\n    // any response code is fine\n    assert!(response.starts_with(\"HTTP/1.0 \"));\n    assert!(response.ends_with(\"</html>\") || response.ends_with(\"</HTML>\"));\n}\n"
  },
  {
    "path": "pingora-boringssl/src/ext.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! the extended functionalities that are yet exposed via the [`boring`] APIs\n\nuse boring::error::ErrorStack;\nuse boring::pkey::{HasPrivate, PKeyRef};\nuse boring::ssl::{Ssl, SslAcceptor, SslRef};\nuse boring::x509::store::X509StoreRef;\nuse boring::x509::verify::X509VerifyParamRef;\nuse boring::x509::X509Ref;\nuse foreign_types_shared::ForeignTypeRef;\nuse libc::*;\nuse std::ffi::CString;\n\nfn cvt(r: c_int) -> Result<c_int, ErrorStack> {\n    if r != 1 {\n        Err(ErrorStack::get())\n    } else {\n        Ok(r)\n    }\n}\n\n/// Add name as an additional reference identifier that can match the peer's certificate\n///\n/// See [X509_VERIFY_PARAM_set1_host](https://www.openssl.org/docs/man3.1/man3/X509_VERIFY_PARAM_set1_host.html).\npub fn add_host(verify_param: &mut X509VerifyParamRef, host: &str) -> Result<(), ErrorStack> {\n    if host.is_empty() {\n        return Ok(());\n    }\n    unsafe {\n        cvt(boring_sys::X509_VERIFY_PARAM_add1_host(\n            verify_param.as_ptr(),\n            host.as_ptr() as *const _,\n            host.len(),\n        ))\n        .map(|_| ())\n    }\n}\n\n/// Set the verify cert store of `ssl`\n///\n/// See [SSL_set1_verify_cert_store](https://www.openssl.org/docs/man1.1.1/man3/SSL_set1_verify_cert_store.html).\npub fn ssl_set_verify_cert_store(\n    ssl: &mut SslRef,\n    cert_store: &X509StoreRef,\n) -> Result<(), ErrorStack> {\n    unsafe {\n        cvt(boring_sys::SSL_set1_verify_cert_store(\n            ssl.as_ptr(),\n            cert_store.as_ptr(),\n        ))?;\n    }\n    Ok(())\n}\n\n/// Load the certificate into `ssl`\n///\n/// See [SSL_use_certificate](https://www.openssl.org/docs/man1.1.1/man3/SSL_use_certificate.html).\npub fn ssl_use_certificate(ssl: &mut SslRef, cert: &X509Ref) -> Result<(), ErrorStack> {\n    unsafe {\n        cvt(boring_sys::SSL_use_certificate(ssl.as_ptr(), cert.as_ptr()))?;\n    }\n    Ok(())\n}\n\n/// Load the private key into `ssl`\n///\n/// See [SSL_use_certificate](https://www.openssl.org/docs/man1.1.1/man3/SSL_use_PrivateKey.html).\npub fn ssl_use_private_key<T>(ssl: &mut SslRef, key: &PKeyRef<T>) -> Result<(), ErrorStack>\nwhere\n    T: HasPrivate,\n{\n    unsafe {\n        cvt(boring_sys::SSL_use_PrivateKey(ssl.as_ptr(), key.as_ptr()))?;\n    }\n    Ok(())\n}\n\n/// Add the certificate into the cert chain of `ssl`\n///\n/// See [SSL_add1_chain_cert](https://www.openssl.org/docs/man1.1.1/man3/SSL_add1_chain_cert.html)\npub fn ssl_add_chain_cert(ssl: &mut SslRef, cert: &X509Ref) -> Result<(), ErrorStack> {\n    unsafe {\n        cvt(boring_sys::SSL_add1_chain_cert(ssl.as_ptr(), cert.as_ptr()))?;\n    }\n    Ok(())\n}\n\n/// Set renegotiation\n///\n/// This function is specific to BoringSSL\n/// See <https://commondatastorage.googleapis.com/chromium-boringssl-docs/ssl.h.html#SSL_set_renegotiate_mode>\npub fn ssl_set_renegotiate_mode_freely(ssl: &mut SslRef) {\n    unsafe {\n        boring_sys::SSL_set_renegotiate_mode(\n            ssl.as_ptr(),\n            boring_sys::ssl_renegotiate_mode_t::ssl_renegotiate_freely,\n        );\n    }\n}\n\n/// Set the curves/groups of `ssl`\n///\n/// See [set_groups_list](https://www.openssl.org/docs/manmaster/man3/SSL_CTX_set1_curves.html).\npub fn ssl_set_groups_list(ssl: &mut SslRef, groups: &str) -> Result<(), ErrorStack> {\n    let groups = CString::new(groups).unwrap();\n    unsafe {\n        // somehow SSL_set1_groups_list doesn't exist but SSL_set1_curves_list means the same anyways\n        cvt(boring_sys::SSL_set1_curves_list(\n            ssl.as_ptr(),\n            groups.as_ptr(),\n        ))?;\n    }\n    Ok(())\n}\n\n/// Set's whether a second keyshare to be sent in client hello when PQ is used.\n///\n/// Default is true. When `true`, the first PQ (if any) and none-PQ keyshares are sent.\n/// When `false`, only the first configured keyshares are sent.\n#[cfg(feature = \"pq_use_second_keyshare\")]\npub fn ssl_use_second_key_share(ssl: &mut SslRef, enabled: bool) {\n    unsafe { boring_sys::SSL_use_second_keyshare(ssl.as_ptr(), enabled as _) }\n}\n#[cfg(not(feature = \"pq_use_second_keyshare\"))]\npub fn ssl_use_second_key_share(_ssl: &mut SslRef, _enabled: bool) {}\n\n/// Clear the error stack\n///\n/// SSL calls should check and clear the BoringSSL error stack. But some calls fail to do so.\n/// This causes the next unrelated SSL call to fail due to the leftover errors. This function allows\n/// the caller to clear the error stack before performing SSL calls to avoid this issue.\npub fn clear_error_stack() {\n    let _ = ErrorStack::get();\n}\n\n/// Create a new [Ssl] from &[SslAcceptor]\n///\n/// This function is needed because [Ssl::new()] doesn't take `&SslContextRef` like openssl-rs\npub fn ssl_from_acceptor(acceptor: &SslAcceptor) -> Result<Ssl, ErrorStack> {\n    Ssl::new_from_ref(acceptor.context())\n}\n\n/// Suspend the TLS handshake when a certificate is needed.\n///\n/// This function will cause tls handshake to pause and return the error: SSL_ERROR_WANT_X509_LOOKUP.\n/// The caller should set the certificate and then call [unblock_ssl_cert()] before continue the\n/// handshake on the tls connection.\npub fn suspend_when_need_ssl_cert(ssl: &mut SslRef) {\n    unsafe {\n        boring_sys::SSL_set_cert_cb(ssl.as_ptr(), Some(raw_cert_block), std::ptr::null_mut());\n    }\n}\n\n/// Unblock a TLS handshake after the certificate is set.\n///\n/// The user should continue to call tls handshake after this function is called.\npub fn unblock_ssl_cert(ssl: &mut SslRef) {\n    unsafe {\n        boring_sys::SSL_set_cert_cb(ssl.as_ptr(), None, std::ptr::null_mut());\n    }\n}\n\n// Just block the handshake\nextern \"C\" fn raw_cert_block(_ssl: *mut boring_sys::SSL, _arg: *mut c_void) -> c_int {\n    -1\n}\n\n/// Whether the TLS error is SSL_ERROR_WANT_X509_LOOKUP\npub fn is_suspended_for_cert(error: &boring::ssl::Error) -> bool {\n    error.code().as_raw() == boring_sys::SSL_ERROR_WANT_X509_LOOKUP\n}\n\n#[allow(clippy::mut_from_ref)]\n/// Get a mutable SslRef ouf of SslRef. which is a missing functionality for certain SslStream\n/// # Safety\n/// the caller needs to make sure that they hold a &mut SslRef\npub unsafe fn ssl_mut(ssl: &SslRef) -> &mut SslRef {\n    unsafe { SslRef::from_ptr_mut(ssl.as_ptr()) }\n}\n"
  },
  {
    "path": "pingora-boringssl/src/lib.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! The BoringSSL API compatibility layer.\n//!\n//! This crate aims at making [boring] APIs exchangeable with [openssl-rs](https://docs.rs/openssl/latest/openssl/).\n//! In other words, this crate and [`pingora-openssl`](https://docs.rs/pingora-openssl) expose identical rust APIs.\n\n#![warn(clippy::all)]\n\nuse boring as ssl_lib;\npub use boring_sys as ssl_sys;\npub mod boring_tokio;\npub use boring_tokio as tokio_ssl;\npub mod ext;\n\n// export commonly used libs\npub use ssl_lib::error;\npub use ssl_lib::hash;\npub use ssl_lib::nid;\npub use ssl_lib::pkey;\npub use ssl_lib::ssl;\npub use ssl_lib::x509;\n"
  },
  {
    "path": "pingora-cache/Cargo.toml",
    "content": "[package]\nname = \"pingora-cache\"\nversion = \"0.8.0\"\nauthors = [\"Yuchen Wu <yuchen@cloudflare.com>\"]\nlicense = \"Apache-2.0\"\nedition = \"2021\"\nrust-version = \"1.84\"\nrepository = \"https://github.com/cloudflare/pingora\"\ncategories = [\"asynchronous\", \"network-programming\"]\nkeywords = [\"async\", \"http\", \"cache\"]\ndescription = \"\"\"\nHTTP caching APIs for Pingora proxy.\n\"\"\"\n\n# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html\n[lib]\nname = \"pingora_cache\"\npath = \"src/lib.rs\"\n\n[dependencies]\npingora-core = { version = \"0.8.0\", path = \"../pingora-core\", default-features = false }\npingora-error = { version = \"0.8.0\", path = \"../pingora-error\" }\npingora-header-serde = { version = \"0.8.0\", path = \"../pingora-header-serde\" }\npingora-http = { version = \"0.8.0\", path = \"../pingora-http\" }\npingora-lru = { version = \"0.8.0\", path = \"../pingora-lru\" }\npingora-timeout = { version = \"0.8.0\", path = \"../pingora-timeout\" }\nbstr = { workspace = true }\nhttp = { workspace = true }\nindexmap = \"1\"\nonce_cell = { workspace = true }\nregex = \"1\"\nblake2 = \"0.10\"\nserde = { version = \"1.0\", features = [\"derive\"] }\nrmp-serde = \"1.3.0\"\nbytes = { workspace = true }\nhttpdate = \"1.0.2\"\nlog = { workspace = true }\nasync-trait = { workspace = true }\nparking_lot = \"0.12\"\ncf-rustracing = \"1.0\"\ncf-rustracing-jaeger = \"1.0\"\nrmp = \"0.8.14\"\ntokio = { workspace = true }\nlru = { workspace = true }\nahash = { workspace = true }\nhex = \"0.4\"\nhttparse = { workspace = true }\nstrum = { version = \"0.26\", features = [\"derive\"] }\nrand = \"0.8\"\n\n[dev-dependencies]\ntokio-test = \"0.4\"\ntokio = { workspace = true, features = [\"fs\"] }\nenv_logger = \"0.11\"\ndhat = \"0\"\nfutures = \"0.3\"\n\n[[bench]]\nname = \"simple_lru_memory\"\nharness = false\n\n[[bench]]\nname = \"lru_memory\"\nharness = false\n\n[[bench]]\nname = \"lru_serde\"\nharness = false\n\n[features]\ndefault = []\nopenssl = [\"pingora-core/openssl\"]\nboringssl = [\"pingora-core/boringssl\"]\nrustls = [\"pingora-core/rustls\"]\ns2n = [\"pingora-core/s2n\"]\n"
  },
  {
    "path": "pingora-cache/LICENSE",
    "content": "\n                                 Apache License\n                           Version 2.0, January 2004\n                        http://www.apache.org/licenses/\n\n   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION\n\n   1. Definitions.\n\n      \"License\" shall mean the terms and conditions for use, reproduction,\n      and distribution as defined by Sections 1 through 9 of this document.\n\n      \"Licensor\" shall mean the copyright owner or entity authorized by\n      the copyright owner that is granting the License.\n\n      \"Legal Entity\" shall mean the union of the acting entity and all\n      other entities that control, are controlled by, or are under common\n      control with that entity. For the purposes of this definition,\n      \"control\" means (i) the power, direct or indirect, to cause the\n      direction or management of such entity, whether by contract or\n      otherwise, or (ii) ownership of fifty percent (50%) or more of the\n      outstanding shares, or (iii) beneficial ownership of such entity.\n\n      \"You\" (or \"Your\") shall mean an individual or Legal Entity\n      exercising permissions granted by this License.\n\n      \"Source\" form shall mean the preferred form for making modifications,\n      including but not limited to software source code, documentation\n      source, and configuration files.\n\n      \"Object\" form shall mean any form resulting from mechanical\n      transformation or translation of a Source form, including but\n      not limited to compiled object code, generated documentation,\n      and conversions to other media types.\n\n      \"Work\" shall mean the work of authorship, whether in Source or\n      Object form, made available under the License, as indicated by a\n      copyright notice that is included in or attached to the work\n      (an example is provided in the Appendix below).\n\n      \"Derivative Works\" shall mean any work, whether in Source or Object\n      form, that is based on (or derived from) the Work and for which the\n      editorial revisions, annotations, elaborations, or other modifications\n      represent, as a whole, an original work of authorship. For the purposes\n      of this License, Derivative Works shall not include works that remain\n      separable from, or merely link (or bind by name) to the interfaces of,\n      the Work and Derivative Works thereof.\n\n      \"Contribution\" shall mean any work of authorship, including\n      the original version of the Work and any modifications or additions\n      to that Work or Derivative Works thereof, that is intentionally\n      submitted to Licensor for inclusion in the Work by the copyright owner\n      or by an individual or Legal Entity authorized to submit on behalf of\n      the copyright owner. For the purposes of this definition, \"submitted\"\n      means any form of electronic, verbal, or written communication sent\n      to the Licensor or its representatives, including but not limited to\n      communication on electronic mailing lists, source code control systems,\n      and issue tracking systems that are managed by, or on behalf of, the\n      Licensor for the purpose of discussing and improving the Work, but\n      excluding communication that is conspicuously marked or otherwise\n      designated in writing by the copyright owner as \"Not a Contribution.\"\n\n      \"Contributor\" shall mean Licensor and any individual or Legal Entity\n      on behalf of whom a Contribution has been received by Licensor and\n      subsequently incorporated within the Work.\n\n   2. Grant of Copyright License. Subject to the terms and conditions of\n      this License, each Contributor hereby grants to You a perpetual,\n      worldwide, non-exclusive, no-charge, royalty-free, irrevocable\n      copyright license to reproduce, prepare Derivative Works of,\n      publicly display, publicly perform, sublicense, and distribute the\n      Work and such Derivative Works in Source or Object form.\n\n   3. Grant of Patent License. Subject to the terms and conditions of\n      this License, each Contributor hereby grants to You a perpetual,\n      worldwide, non-exclusive, no-charge, royalty-free, irrevocable\n      (except as stated in this section) patent license to make, have made,\n      use, offer to sell, sell, import, and otherwise transfer the Work,\n      where such license applies only to those patent claims licensable\n      by such Contributor that are necessarily infringed by their\n      Contribution(s) alone or by combination of their Contribution(s)\n      with the Work to which such Contribution(s) was submitted. If You\n      institute patent litigation against any entity (including a\n      cross-claim or counterclaim in a lawsuit) alleging that the Work\n      or a Contribution incorporated within the Work constitutes direct\n      or contributory patent infringement, then any patent licenses\n      granted to You under this License for that Work shall terminate\n      as of the date such litigation is filed.\n\n   4. Redistribution. You may reproduce and distribute copies of the\n      Work or Derivative Works thereof in any medium, with or without\n      modifications, and in Source or Object form, provided that You\n      meet the following conditions:\n\n      (a) You must give any other recipients of the Work or\n          Derivative Works a copy of this License; and\n\n      (b) You must cause any modified files to carry prominent notices\n          stating that You changed the files; and\n\n      (c) You must retain, in the Source form of any Derivative Works\n          that You distribute, all copyright, patent, trademark, and\n          attribution notices from the Source form of the Work,\n          excluding those notices that do not pertain to any part of\n          the Derivative Works; and\n\n      (d) If the Work includes a \"NOTICE\" text file as part of its\n          distribution, then any Derivative Works that You distribute must\n          include a readable copy of the attribution notices contained\n          within such NOTICE file, excluding those notices that do not\n          pertain to any part of the Derivative Works, in at least one\n          of the following places: within a NOTICE text file distributed\n          as part of the Derivative Works; within the Source form or\n          documentation, if provided along with the Derivative Works; or,\n          within a display generated by the Derivative Works, if and\n          wherever such third-party notices normally appear. The contents\n          of the NOTICE file are for informational purposes only and\n          do not modify the License. You may add Your own attribution\n          notices within Derivative Works that You distribute, alongside\n          or as an addendum to the NOTICE text from the Work, provided\n          that such additional attribution notices cannot be construed\n          as modifying the License.\n\n      You may add Your own copyright statement to Your modifications and\n      may provide additional or different license terms and conditions\n      for use, reproduction, or distribution of Your modifications, or\n      for any such Derivative Works as a whole, provided Your use,\n      reproduction, and distribution of the Work otherwise complies with\n      the conditions stated in this License.\n\n   5. Submission of Contributions. Unless You explicitly state otherwise,\n      any Contribution intentionally submitted for inclusion in the Work\n      by You to the Licensor shall be under the terms and conditions of\n      this License, without any additional terms or conditions.\n      Notwithstanding the above, nothing herein shall supersede or modify\n      the terms of any separate license agreement you may have executed\n      with Licensor regarding such Contributions.\n\n   6. Trademarks. This License does not grant permission to use the trade\n      names, trademarks, service marks, or product names of the Licensor,\n      except as required for reasonable and customary use in describing the\n      origin of the Work and reproducing the content of the NOTICE file.\n\n   7. Disclaimer of Warranty. Unless required by applicable law or\n      agreed to in writing, Licensor provides the Work (and each\n      Contributor provides its Contributions) on an \"AS IS\" BASIS,\n      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or\n      implied, including, without limitation, any warranties or conditions\n      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A\n      PARTICULAR PURPOSE. You are solely responsible for determining the\n      appropriateness of using or redistributing the Work and assume any\n      risks associated with Your exercise of permissions under this License.\n\n   8. Limitation of Liability. In no event and under no legal theory,\n      whether in tort (including negligence), contract, or otherwise,\n      unless required by applicable law (such as deliberate and grossly\n      negligent acts) or agreed to in writing, shall any Contributor be\n      liable to You for damages, including any direct, indirect, special,\n      incidental, or consequential damages of any character arising as a\n      result of this License or out of the use or inability to use the\n      Work (including but not limited to damages for loss of goodwill,\n      work stoppage, computer failure or malfunction, or any and all\n      other commercial damages or losses), even if such Contributor\n      has been advised of the possibility of such damages.\n\n   9. Accepting Warranty or Additional Liability. While redistributing\n      the Work or Derivative Works thereof, You may choose to offer,\n      and charge a fee for, acceptance of support, warranty, indemnity,\n      or other liability obligations and/or rights consistent with this\n      License. However, in accepting such obligations, You may act only\n      on Your own behalf and on Your sole responsibility, not on behalf\n      of any other Contributor, and only if You agree to indemnify,\n      defend, and hold each Contributor harmless for any liability\n      incurred by, or claims asserted against, such Contributor by reason\n      of your accepting any such warranty or additional liability.\n\n   END OF TERMS AND CONDITIONS\n\n   APPENDIX: How to apply the Apache License to your work.\n\n      To apply the Apache License to your work, attach the following\n      boilerplate notice, with the fields enclosed by brackets \"[]\"\n      replaced with your own identifying information. (Don't include\n      the brackets!)  The text should be enclosed in the appropriate\n      comment syntax for the file format. We also recommend that a\n      file or class name and description of purpose be included on the\n      same \"printed page\" as the copyright notice for easier\n      identification within third-party archives.\n\n   Copyright [yyyy] [name of copyright owner]\n\n   Licensed under the Apache License, Version 2.0 (the \"License\");\n   you may not use this file except in compliance with the License.\n   You may obtain a copy of the License at\n\n       http://www.apache.org/licenses/LICENSE-2.0\n\n   Unless required by applicable law or agreed to in writing, software\n   distributed under the License is distributed on an \"AS IS\" BASIS,\n   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n   See the License for the specific language governing permissions and\n   limitations under the License.\n"
  },
  {
    "path": "pingora-cache/benches/lru_memory.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n#[global_allocator]\nstatic ALLOC: dhat::Alloc = dhat::Alloc;\n\nuse pingora_cache::{\n    eviction::{lru::Manager, EvictionManager},\n    CacheKey,\n};\n\nconst ITEMS: usize = 5 * usize::pow(2, 20);\n\n/*\n    Total:     681,836,456 bytes (100%, 28,192,797.16/s) in 10,485,845 blocks (100%, 433,572.15/s), avg size 65.02 bytes, avg lifetime 5,935,075.17 µs (24.54% of program duration)\n    At t-gmax: 569,114,536 bytes (100%) in 5,242,947 blocks (100%), avg size 108.55 bytes\n    At t-end:  88 bytes (100%) in 3 blocks (100%), avg size 29.33 bytes\n    Allocated at {\n      #0: [root]\n    }\n  ├── PP 1.1/5 {\n  │     Total:     293,601,280 bytes (43.06%, 12,139,921.91/s) in 5,242,880 blocks (50%, 216,784.32/s), avg size 56 bytes, avg lifetime 11,870,032.65 µs (49.08% of program duration)\n  │     Max:       293,601,280 bytes in 5,242,880 blocks, avg size 56 bytes\n  │     At t-gmax: 293,601,280 bytes (51.59%) in 5,242,880 blocks (100%), avg size 56 bytes\n  │     At t-end:  0 bytes (0%) in 0 blocks (0%), avg size 0 bytes\n  │     Allocated at {\n  │       #1: 0x5555703cf69c: alloc::alloc::exchange_malloc (alloc/src/alloc.rs:326:11)\n  │       #2: 0x5555703cf69c: alloc::boxed::Box<T>::new (alloc/src/boxed.rs:215:9)\n  │       #3: 0x5555703cf69c: pingora_lru::LruUnit<T>::admit (pingora-lru/src/lib.rs:201:20)\n  │       #4: 0x5555703cf69c: pingora_lru::Lru<T,_>::admit (pingora-lru/src/lib.rs:48:26)\n  │       #5: 0x5555703cf69c: <pingora_cache::eviction::lru::Manager<_> as pingora_cache::eviction::EvictionManager>::admit (src/eviction/lru.rs:114:9)\n  │       #6: 0x5555703cf69c: lru_memory::main (pingora-cache/benches/lru_memory.rs:78:9)\n  │     }\n  │   }\n  ├── PP 1.2/5 {\n  │     Total:     203,685,456 bytes (29.87%, 8,422,052.97/s) in 50 blocks (0%, 2.07/s), avg size 4,073,709.12 bytes, avg lifetime 6,842,528.74 µs (28.29% of program duration)\n  │     Max:       132,906,576 bytes in 32 blocks, avg size 4,153,330.5 bytes\n  │     At t-gmax: 132,906,576 bytes (23.35%) in 32 blocks (0%), avg size 4,153,330.5 bytes\n  │     At t-end:  0 bytes (0%) in 0 blocks (0%), avg size 0 bytes\n  │     Allocated at {\n  │       #1: 0x5555703cec54: <alloc::alloc::Global as core::alloc::Allocator>::allocate (alloc/src/alloc.rs:237:9)\n  │       #2: 0x5555703cec54: alloc::raw_vec::RawVec<T,A>::allocate_in (alloc/src/raw_vec.rs:185:45)\n  │       #3: 0x5555703cec54: alloc::raw_vec::RawVec<T,A>::with_capacity_in (alloc/src/raw_vec.rs:131:9)\n  │       #4: 0x5555703cec54: alloc::vec::Vec<T,A>::with_capacity_in (src/vec/mod.rs:641:20)\n  │       #5: 0x5555703cec54: alloc::vec::Vec<T>::with_capacity (src/vec/mod.rs:483:9)\n  │       #6: 0x5555703cec54: pingora_lru::linked_list::Nodes::with_capacity (pingora-lru/src/linked_list.rs:50:25)\n  │       #7: 0x5555703cec54: pingora_lru::linked_list::LinkedList::with_capacity (pingora-lru/src/linked_list.rs:121:20)\n  │       #8: 0x5555703cec54: pingora_lru::LruUnit<T>::with_capacity (pingora-lru/src/lib.rs:176:20)\n  │       #9: 0x5555703cec54: pingora_lru::Lru<T,_>::with_capacity (pingora-lru/src/lib.rs:28:36)\n  │       #10: 0x5555703cec54: pingora_cache::eviction::lru::Manager<_>::with_capacity (src/eviction/lru.rs:22:17)\n  │       #11: 0x5555703cec54: lru_memory::main (pingora-cache/benches/lru_memory.rs:74:19)\n  │     }\n  │   }\n  ├── PP 1.3/5 {\n  │     Total:     142,606,592 bytes (20.92%, 5,896,544.09/s) in 32 blocks (0%, 1.32/s), avg size 4,456,456 bytes, avg lifetime 22,056,252.88 µs (91.2% of program duration)\n  │     Max:       142,606,592 bytes in 32 blocks, avg size 4,456,456 bytes\n  │     At t-gmax: 142,606,592 bytes (25.06%) in 32 blocks (0%), avg size 4,456,456 bytes\n  │     At t-end:  0 bytes (0%) in 0 blocks (0%), avg size 0 bytes\n  │     Allocated at {\n  │       #1: 0x5555703ceb64: alloc::alloc::alloc (alloc/src/alloc.rs:95:14)\n  │       #2: 0x5555703ceb64: <hashbrown::raw::alloc::inner::Global as hashbrown::raw::alloc::inner::Allocator>::allocate (src/raw/alloc.rs:47:35)\n  │       #3: 0x5555703ceb64: hashbrown::raw::alloc::inner::do_alloc (src/raw/alloc.rs:62:9)\n  │       #4: 0x5555703ceb64: hashbrown::raw::RawTableInner<A>::new_uninitialized (src/raw/mod.rs:1080:38)\n  │       #5: 0x5555703ceb64: hashbrown::raw::RawTableInner<A>::fallible_with_capacity (src/raw/mod.rs:1109:30)\n  │       #6: 0x5555703ceb64: hashbrown::raw::RawTable<T,A>::fallible_with_capacity (src/raw/mod.rs:460:20)\n  │       #7: 0x5555703ceb64: hashbrown::raw::RawTable<T,A>::with_capacity_in (src/raw/mod.rs:481:15)\n  │       #8: 0x5555703ceb64: hashbrown::raw::RawTable<T>::with_capacity (src/raw/mod.rs:411:9)\n  │       #9: 0x5555703ceb64: hashbrown::map::HashMap<K,V,S>::with_capacity_and_hasher (hashbrown-0.12.3/src/map.rs:422:20)\n  │       #10: 0x5555703ceb64: hashbrown::map::HashMap<K,V>::with_capacity (hashbrown-0.12.3/src/map.rs:326:9)\n  │       #11: 0x5555703ceb64: pingora_lru::LruUnit<T>::with_capacity (pingora-lru/src/lib.rs:175:27)\n  │       #12: 0x5555703ceb64: pingora_lru::Lru<T,_>::with_capacity (pingora-lru/src/lib.rs:28:36)\n  │       #13: 0x5555703ceb64: pingora_cache::eviction::lru::Manager<_>::with_capacity (src/eviction/lru.rs:22:17)\n  │       #14: 0x5555703ceb64: lru_memory::main (pingora-cache/benches/lru_memory.rs:74:19)\n  │     }\n  │   }\n*/\nfn main() {\n    let _profiler = dhat::Profiler::new_heap();\n    let manager = Manager::<32>::with_capacity(ITEMS, ITEMS / 32);\n    let unused_ttl = std::time::SystemTime::now();\n    for i in 0..ITEMS {\n        let item = CacheKey::new(\"\", i.to_string(), \"\").to_compact();\n        manager.admit(item, 1, unused_ttl);\n    }\n}\n"
  },
  {
    "path": "pingora-cache/benches/lru_serde.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\nuse std::time::Instant;\n\nuse pingora_cache::{\n    eviction::{lru::Manager, EvictionManager},\n    CacheKey,\n};\n\nconst ITEMS: usize = 5 * usize::pow(2, 20);\n\nfn main() {\n    let manager = Manager::<32>::with_capacity(ITEMS, ITEMS / 32);\n    let manager2 = Manager::<32>::with_capacity(ITEMS, ITEMS / 32);\n    let unused_ttl = std::time::SystemTime::now();\n    for i in 0..ITEMS {\n        let item = CacheKey::new(\"\", i.to_string(), \"\").to_compact();\n        manager.admit(item, 1, unused_ttl);\n    }\n\n    /* lru serialize shard 19 22.573338ms, 5241623 bytes\n     * lru deserialize shard 19 39.260669ms, 5241623 bytes */\n    for i in 0..32 {\n        let before = Instant::now();\n        let ser = manager.serialize_shard(i).unwrap();\n        let elapsed = before.elapsed();\n        println!(\"lru serialize shard {i} {elapsed:?}, {} bytes\", ser.len());\n\n        let before = Instant::now();\n        manager2.deserialize_shard(&ser).unwrap();\n        let elapsed = before.elapsed();\n        println!(\"lru deserialize shard {i} {elapsed:?}, {} bytes\", ser.len());\n    }\n}\n"
  },
  {
    "path": "pingora-cache/benches/simple_lru_memory.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n#[global_allocator]\nstatic ALLOC: dhat::Alloc = dhat::Alloc;\n\nuse pingora_cache::{\n    eviction::{simple_lru::Manager, EvictionManager},\n    CacheKey,\n};\n\nconst ITEMS: usize = 5 * usize::pow(2, 20);\n\n/*\n   Total:     704,643,412 bytes (100%, 29,014,058.85/s) in 10,485,787 blocks (100%, 431,757.73/s), avg size 67.2 bytes, avg lifetime 6,163,799.09 µs (25.38% of program duration)\n   At t-gmax: 520,093,936 bytes (100%) in 5,242,886 blocks (100%), avg size 99.2 bytes\n  ├── PP 1.1/4 {\n  │     Total:     377,487,360 bytes (53.57%, 15,543,238.31/s) in 5,242,880 blocks (50%, 215,878.31/s), avg size 72 bytes, avg lifetime 12,327,602.83 µs (50.76% of program duration)\n  │     Max:       377,487,360 bytes in 5,242,880 blocks, avg size 72 bytes\n  │     At t-gmax: 377,487,360 bytes (72.58%) in 5,242,880 blocks (100%), avg size 72 bytes\n  │     At t-end:  0 bytes (0%) in 0 blocks (0%), avg size 0 bytes\n  │     Allocated at {\n  │       #1: 0x5555791dd7e0: alloc::alloc::exchange_malloc (alloc/src/alloc.rs:326:11)\n  │       #2: 0x5555791dd7e0: alloc::boxed::Box<T>::new (alloc/src/boxed.rs:215:9)\n  │       #3: 0x5555791dd7e0: lru::LruCache<K,V,S>::replace_or_create_node (lru-0.8.1/src/lib.rs:391:20)\n  │       #4: 0x5555791dd7e0: lru::LruCache<K,V,S>::capturing_put (lru-0.8.1/src/lib.rs:355:44)\n  │       #5: 0x5555791dd7e0: lru::LruCache<K,V,S>::push (lru-0.8.1/src/lib.rs:334:9)\n  │       #6: 0x5555791dd7e0: pingora_cache::eviction::simple_lru::Manager::insert (src/eviction/simple_lru.rs:49:23)\n  │       #7: 0x5555791dd7e0: <pingora_cache::eviction::simple_lru::Manager as pingora_cache::eviction::EvictionManager>::admit (src/eviction/simple_lru.rs:166:9)\n  │       #8: 0x5555791dd7e0: simple_lru_memory::main (pingora-cache/benches/simple_lru_memory.rs:21:9)\n  │     }\n  │   }\n  ├── PP 1.2/4 {\n  │     Total:     285,212,780 bytes (40.48%, 11,743,784.5/s) in 22 blocks (0%, 0.91/s), avg size 12,964,217.27 bytes, avg lifetime 1,116,774.23 µs (4.6% of program duration)\n  │     Max:       213,909,520 bytes in 2 blocks, avg size 106,954,760 bytes\n  │     At t-gmax: 142,606,344 bytes (27.42%) in 1 blocks (0%), avg size 142,606,344 bytes\n  │     At t-end:  0 bytes (0%) in 0 blocks (0%), avg size 0 bytes\n  │     Allocated at {\n  │       #1: 0x5555791dae20: alloc::alloc::alloc (alloc/src/alloc.rs:95:14)\n  │       #2: 0x5555791dae20: <hashbrown::raw::alloc::inner::Global as hashbrown::raw::alloc::inner::Allocator>::allocate (src/raw/alloc.rs:47:35)\n  │       #3: 0x5555791dae20: hashbrown::raw::alloc::inner::do_alloc (src/raw/alloc.rs:62:9)\n  │       #4: 0x5555791dae20: hashbrown::raw::RawTableInner<A>::new_uninitialized (src/raw/mod.rs:1080:38)\n  │       #5: 0x5555791dae20: hashbrown::raw::RawTableInner<A>::fallible_with_capacity (src/raw/mod.rs:1109:30)\n  │       #6: 0x5555791dae20: hashbrown::raw::RawTableInner<A>::prepare_resize (src/raw/mod.rs:1353:29)\n  │       #7: 0x5555791dae20: hashbrown::raw::RawTableInner<A>::resize_inner (src/raw/mod.rs:1426:29)\n  │       #8: 0x5555791dae20: hashbrown::raw::RawTableInner<A>::reserve_rehash_inner (src/raw/mod.rs:1403:13)\n  │       #9: 0x5555791dae20: hashbrown::raw::RawTable<T,A>::reserve_rehash (src/raw/mod.rs:680:13)\n  │       #10: 0x5555791dde50: hashbrown::raw::RawTable<T,A>::reserve (src/raw/mod.rs:646:16)\n  │       #11: 0x5555791dde50: hashbrown::raw::RawTable<T,A>::insert (src/raw/mod.rs:725:17)\n  │       #12: 0x5555791dde50: hashbrown::map::HashMap<K,V,S,A>::insert (hashbrown-0.12.3/src/map.rs:1679:13)\n  │       #13: 0x5555791dde50: lru::LruCache<K,V,S>::capturing_put (lru-0.8.1/src/lib.rs:361:17)\n  │       #14: 0x5555791dde50: lru::LruCache<K,V,S>::push (lru-0.8.1/src/lib.rs:334:9)\n  │       #15: 0x5555791dde50: pingora_cache::eviction::simple_lru::Manager::insert (src/eviction/simple_lru.rs:49:23)\n  │       #16: 0x5555791dde50: <pingora_cache::eviction::simple_lru::Manager as pingora_cache::eviction::EvictionManager>::admit (src/eviction/simple_lru.rs:166:9)\n  │       #17: 0x5555791dde50: simple_lru_memory::main (pingora-cache/benches/simple_lru_memory.rs:21:9)\n  │     }\n  │   }\n*/\nfn main() {\n    let _profiler = dhat::Profiler::new_heap();\n    let manager = Manager::new(ITEMS);\n    let unused_ttl = std::time::SystemTime::now();\n    for i in 0..ITEMS {\n        let item = CacheKey::new(\"\", i.to_string(), \"\").to_compact();\n        manager.admit(item, 1, unused_ttl);\n    }\n}\n"
  },
  {
    "path": "pingora-cache/src/cache_control.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! Functions and utilities to help parse Cache-Control headers\n\nuse super::*;\n\nuse http::header::HeaderName;\nuse http::HeaderValue;\nuse indexmap::IndexMap;\nuse once_cell::sync::Lazy;\nuse pingora_error::{Error, ErrorType};\nuse regex::bytes::Regex;\nuse std::num::IntErrorKind;\nuse std::slice;\nuse std::str;\n\n/// The max delta-second per [RFC 9111](https://datatracker.ietf.org/doc/html/rfc9111#section-1.2.2)\n// \"If a cache receives a delta-seconds value\n// greater than the greatest integer it can represent, or if any of its\n// subsequent calculations overflows, the cache MUST consider the value\n// to be 2147483648 (2^31) or the greatest positive integer it can\n// conveniently represent.\n//\n//    |  *Note:* The value 2147483648 is here for historical reasons,\n//    |  represents infinity (over 68 years), and does not need to be\n//    |  stored in binary form; an implementation could produce it as a\n//    |  string if any overflow occurs, even if the calculations are\n//    |  performed with an arithmetic type incapable of directly\n//    |  representing that number.  What matters here is that an\n//    |  overflow be detected and not treated as a negative value in\n//    |  later calculations.\"\n//\n// We choose to use i32::MAX for our overflow value to stick to the letter of the RFC.\npub const DELTA_SECONDS_OVERFLOW_VALUE: u32 = i32::MAX as u32;\npub const DELTA_SECONDS_OVERFLOW_DURATION: Duration =\n    Duration::from_secs(DELTA_SECONDS_OVERFLOW_VALUE as u64);\n\n/// Cache control directive key type\npub type DirectiveKey = String;\n\n/// Cache control directive value type\n#[derive(Debug)]\npub struct DirectiveValue(pub Vec<u8>);\n\nimpl AsRef<[u8]> for DirectiveValue {\n    fn as_ref(&self) -> &[u8] {\n        &self.0\n    }\n}\n\nimpl DirectiveValue {\n    /// A [DirectiveValue] without quotes (`\"`).\n    pub fn parse_as_bytes(&self) -> &[u8] {\n        self.0\n            .strip_prefix(b\"\\\"\")\n            .and_then(|bytes| bytes.strip_suffix(b\"\\\"\"))\n            .unwrap_or(&self.0[..])\n    }\n\n    /// A [DirectiveValue] without quotes (`\"`) as `str`.\n    pub fn parse_as_str(&self) -> Result<&str> {\n        str::from_utf8(self.parse_as_bytes()).or_else(|e| {\n            Error::e_because(ErrorType::InternalError, \"could not parse value as utf8\", e)\n        })\n    }\n\n    /// Parse the [DirectiveValue] as delta seconds\n    ///\n    /// `\"`s are ignored. The value is capped to [DELTA_SECONDS_OVERFLOW_VALUE].\n    pub fn parse_as_delta_seconds(&self) -> Result<u32> {\n        match self.parse_as_str()?.parse::<u32>() {\n            Ok(value) => Ok(value),\n            Err(e) => {\n                // delta-seconds expect to handle positive overflow gracefully\n                if e.kind() == &IntErrorKind::PosOverflow {\n                    Ok(DELTA_SECONDS_OVERFLOW_VALUE)\n                } else {\n                    Error::e_because(ErrorType::InternalError, \"could not parse value as u32\", e)\n                }\n            }\n        }\n    }\n}\n\n/// An ordered map to store cache control key value pairs.\npub type DirectiveMap = IndexMap<DirectiveKey, Option<DirectiveValue>>;\n\n/// Parsed Cache-Control directives\n#[derive(Debug)]\npub struct CacheControl {\n    /// The parsed directives\n    pub directives: DirectiveMap,\n}\n\n/// Cacheability calculated from cache control.\n#[derive(Debug, PartialEq, Eq)]\npub enum Cacheable {\n    /// Cacheable\n    Yes,\n    /// Not cacheable\n    No,\n    /// No directive found for explicit cacheability\n    Default,\n}\n\n/// An iter over all the cache control directives\npub struct ListValueIter<'a>(slice::Split<'a, u8, fn(&u8) -> bool>);\n\nimpl<'a> ListValueIter<'a> {\n    pub fn from(value: &'a DirectiveValue) -> Self {\n        ListValueIter(value.parse_as_bytes().split(|byte| byte == &b','))\n    }\n}\n\n// https://datatracker.ietf.org/doc/html/rfc9110#name-whitespace\n// optional whitespace OWS = *(SP / HTAB); SP = 0x20, HTAB = 0x09\nfn trim_ows(bytes: &[u8]) -> &[u8] {\n    fn not_ows(b: &u8) -> bool {\n        b != &b'\\x20' && b != &b'\\x09'\n    }\n    // find first non-OWS char from front (head) and from end (tail)\n    let head = bytes.iter().position(not_ows).unwrap_or(0);\n    let tail = bytes\n        .iter()\n        .rposition(not_ows)\n        .map(|rpos| rpos + 1)\n        .unwrap_or(head);\n    &bytes[head..tail]\n}\n\nimpl<'a> Iterator for ListValueIter<'a> {\n    type Item = &'a [u8];\n\n    fn next(&mut self) -> Option<Self::Item> {\n        Some(trim_ows(self.0.next()?))\n    }\n}\n\n// Originally from https://github.com/hapijs/wreck which has the following comments:\n// Cache-Control   = 1#cache-directive\n// cache-directive = token [ \"=\" ( token / quoted-string ) ]\n// token           = [^\\x00-\\x20\\(\\)<>@\\,;\\:\\\\\"\\/\\[\\]\\?\\=\\{\\}\\x7F]+\n// quoted-string   = \"(?:[^\"\\\\]|\\\\.)*\"\n//\n// note the `token` implementation excludes disallowed ASCII ranges\n// and disallowed delimiters: https://datatracker.ietf.org/doc/html/rfc9110#section-5.6.2\n// though it does not forbid `obs-text`: %x80-FF\nstatic RE_CACHE_DIRECTIVE: Lazy<Regex> =\n    // to break our version down further:\n    // `(?-u)`: unicode support disabled, which puts the regex into \"ASCII compatible mode\" for specifying literal bytes like \\x7F: https://docs.rs/regex/1.10.4/regex/bytes/index.html#syntax\n    // `(?:^|(?:\\s*[,;]\\s*)`: allow either , or ; as a delimiter\n    // `([^\\x00-\\x20\\(\\)<>@,;:\\\\\"/\\[\\]\\?=\\{\\}\\x7F]+)`: token (directive name capture group)\n    // `(?:=((?:[^\\x00-\\x20\\(\\)<>@,;:\\\\\"/\\[\\]\\?=\\{\\}\\x7F]+|(?:\"(?:[^\"\\\\]|\\\\.)*\"))))`: token OR quoted-string (directive value capture-group)\n    Lazy::new(|| {\n        Regex::new(r#\"(?-u)(?:^|(?:\\s*[,;]\\s*))([^\\x00-\\x20\\(\\)<>@,;:\\\\\"/\\[\\]\\?=\\{\\}\\x7F]+)(?:=((?:[^\\x00-\\x20\\(\\)<>@,;:\\\\\"/\\[\\]\\?=\\{\\}\\x7F]+|(?:\"(?:[^\"\\\\]|\\\\.)*\"))))?\"#).unwrap()\n    });\n\nimpl CacheControl {\n    // Our parsing strategy is more permissive than the RFC in a few ways:\n    // - Allows semicolons as delimiters (in addition to commas). See the regex above.\n    // - Allows octets outside of visible ASCII in `token`s, and in later RFCs, octets outside of\n    //   the `quoted-string` range: https://datatracker.ietf.org/doc/html/rfc9110#section-5.6.2\n    //   See the regex above.\n    // - Doesn't require no-value for \"boolean directives,\" such as must-revalidate\n    // - Allows quoted-string format for numeric values.\n    fn from_headers(headers: http::header::GetAll<HeaderValue>) -> Option<Self> {\n        let mut directives = IndexMap::new();\n        // should iterate in header line insertion order\n        for line in headers {\n            for captures in RE_CACHE_DIRECTIVE.captures_iter(line.as_bytes()) {\n                // directive key\n                // header values don't have to be utf-8, but we store keys as strings for case-insensitive hashing\n                let key = captures.get(1).and_then(|cap| {\n                    str::from_utf8(cap.as_bytes())\n                        .ok()\n                        .map(|token| token.to_lowercase())\n                });\n                if key.is_none() {\n                    continue;\n                }\n                // directive value\n                // match token or quoted-string\n                let value = captures\n                    .get(2)\n                    .map(|cap| DirectiveValue(cap.as_bytes().to_vec()));\n                directives.insert(key.unwrap(), value);\n            }\n        }\n        Some(CacheControl { directives })\n    }\n\n    /// Parse from the given header name in `headers`\n    pub fn from_headers_named(header_name: &str, headers: &http::HeaderMap) -> Option<Self> {\n        if !headers.contains_key(header_name) {\n            return None;\n        }\n\n        Self::from_headers(headers.get_all(header_name))\n    }\n\n    /// Parse from the given header name in the [ReqHeader]\n    pub fn from_req_headers_named(header_name: &str, req_header: &ReqHeader) -> Option<Self> {\n        Self::from_headers_named(header_name, &req_header.headers)\n    }\n\n    /// Parse `Cache-Control` header name from the [ReqHeader]\n    pub fn from_req_headers(req_header: &ReqHeader) -> Option<Self> {\n        Self::from_req_headers_named(\"cache-control\", req_header)\n    }\n\n    /// Parse from the given header name in the [RespHeader]\n    pub fn from_resp_headers_named(header_name: &str, resp_header: &RespHeader) -> Option<Self> {\n        Self::from_headers_named(header_name, &resp_header.headers)\n    }\n\n    /// Parse `Cache-Control` header name from the [RespHeader]\n    pub fn from_resp_headers(resp_header: &RespHeader) -> Option<Self> {\n        Self::from_resp_headers_named(\"cache-control\", resp_header)\n    }\n\n    /// Whether the given directive is in the cache control.\n    pub fn has_key(&self, key: &str) -> bool {\n        self.directives.contains_key(key)\n    }\n\n    /// Whether the `public` directive is in the cache control.\n    pub fn public(&self) -> bool {\n        self.has_key(\"public\")\n    }\n\n    /// Whether the given directive exists, and it has no value.\n    fn has_key_without_value(&self, key: &str) -> bool {\n        matches!(self.directives.get(key), Some(None))\n    }\n\n    /// Whether the standalone `private` exists in the cache control\n    // RFC 7234: using the #field-name versions of `private`\n    // means a shared cache \"MUST NOT store the specified field-name(s),\n    // whereas it MAY store the remainder of the response.\"\n    // It must be a boolean form (no value) to apply to the whole response.\n    // https://datatracker.ietf.org/doc/html/rfc7234#section-5.2.2.6\n    pub fn private(&self) -> bool {\n        self.has_key_without_value(\"private\")\n    }\n\n    fn get_field_names(&self, key: &str) -> Option<ListValueIter<'_>> {\n        let value = self.directives.get(key)?.as_ref()?;\n        Some(ListValueIter::from(value))\n    }\n\n    /// Get the values of `private=`\n    pub fn private_field_names(&self) -> Option<ListValueIter<'_>> {\n        self.get_field_names(\"private\")\n    }\n\n    /// Whether the standalone `no-cache` exists in the cache control\n    pub fn no_cache(&self) -> bool {\n        self.has_key_without_value(\"no-cache\")\n    }\n\n    /// Get the values of `no-cache=`\n    pub fn no_cache_field_names(&self) -> Option<ListValueIter<'_>> {\n        self.get_field_names(\"no-cache\")\n    }\n\n    /// Whether `no-store` exists.\n    pub fn no_store(&self) -> bool {\n        self.has_key(\"no-store\")\n    }\n\n    fn parse_delta_seconds(&self, key: &str) -> Result<Option<u32>> {\n        if let Some(Some(dir_value)) = self.directives.get(key) {\n            Ok(Some(dir_value.parse_as_delta_seconds()?))\n        } else {\n            Ok(None)\n        }\n    }\n\n    /// Return the `max-age` seconds\n    pub fn max_age(&self) -> Result<Option<u32>> {\n        self.parse_delta_seconds(\"max-age\")\n    }\n\n    /// Return the `s-maxage` seconds\n    pub fn s_maxage(&self) -> Result<Option<u32>> {\n        self.parse_delta_seconds(\"s-maxage\")\n    }\n\n    /// Return the `stale-while-revalidate` seconds\n    pub fn stale_while_revalidate(&self) -> Result<Option<u32>> {\n        self.parse_delta_seconds(\"stale-while-revalidate\")\n    }\n\n    /// Return the `stale-if-error` seconds\n    pub fn stale_if_error(&self) -> Result<Option<u32>> {\n        self.parse_delta_seconds(\"stale-if-error\")\n    }\n\n    /// Whether `must-revalidate` exists.\n    pub fn must_revalidate(&self) -> bool {\n        self.has_key(\"must-revalidate\")\n    }\n\n    /// Whether `proxy-revalidate` exists.\n    pub fn proxy_revalidate(&self) -> bool {\n        self.has_key(\"proxy-revalidate\")\n    }\n\n    /// Whether `only-if-cached` exists.\n    pub fn only_if_cached(&self) -> bool {\n        self.has_key(\"only-if-cached\")\n    }\n}\n\nimpl InterpretCacheControl for CacheControl {\n    fn is_cacheable(&self) -> Cacheable {\n        if self.no_store() || self.private() {\n            return Cacheable::No;\n        }\n        if self.has_key(\"s-maxage\") || self.has_key(\"max-age\") || self.public() {\n            return Cacheable::Yes;\n        }\n        Cacheable::Default\n    }\n\n    fn allow_caching_authorized_req(&self) -> bool {\n        // RFC 7234 https://datatracker.ietf.org/doc/html/rfc7234#section-3\n        // \"MUST NOT\" store requests with Authorization header\n        // unless response contains one of these directives\n        self.must_revalidate() || self.public() || self.has_key(\"s-maxage\")\n    }\n\n    fn fresh_duration(&self) -> Option<Duration> {\n        if self.no_cache() {\n            // always treated as stale\n            return Some(Duration::ZERO);\n        }\n        let seconds = self\n            .s_maxage()\n            .ok()?\n            // s-maxage not present\n            .or_else(|| self.max_age().unwrap_or(None))\n            .map(|duration| Duration::from_secs(duration as u64))?;\n        Some(seconds)\n    }\n\n    fn serve_stale_while_revalidate_duration(&self) -> Option<Duration> {\n        // RFC 7234: these directives forbid serving stale.\n        // https://datatracker.ietf.org/doc/html/rfc7234#section-4.2.4\n        if self.must_revalidate() || self.proxy_revalidate() || self.has_key(\"s-maxage\") {\n            return Some(Duration::ZERO);\n        }\n        self.stale_while_revalidate()\n            .unwrap_or(None)\n            .map(|secs| Duration::from_secs(secs as u64))\n    }\n\n    fn serve_stale_if_error_duration(&self) -> Option<Duration> {\n        if self.must_revalidate() || self.proxy_revalidate() || self.has_key(\"s-maxage\") {\n            return Some(Duration::ZERO);\n        }\n        self.stale_if_error()\n            .unwrap_or(None)\n            .map(|secs| Duration::from_secs(secs as u64))\n    }\n\n    // Strip header names listed in `private` or `no-cache` directives from a response.\n    fn strip_private_headers(&self, resp_header: &mut ResponseHeader) {\n        fn strip_listed_headers(resp: &mut ResponseHeader, field_names: ListValueIter) {\n            for name in field_names {\n                if let Ok(header) = HeaderName::from_bytes(name) {\n                    resp.remove_header(&header);\n                }\n            }\n        }\n\n        if let Some(headers) = self.private_field_names() {\n            strip_listed_headers(resp_header, headers);\n        }\n        // We interpret `no-cache` the same way as `private`,\n        // though technically it has a less restrictive requirement\n        // (\"MUST NOT be sent in the response to a subsequent request\n        // without successful revalidation with the origin server\").\n        // https://datatracker.ietf.org/doc/html/rfc7234#section-5.2.2.2\n        if let Some(headers) = self.no_cache_field_names() {\n            strip_listed_headers(resp_header, headers);\n        }\n    }\n}\n\n/// `InterpretCacheControl` provides a meaningful interface to the parsed `CacheControl`.\n/// These functions actually interpret the parsed cache-control directives to return\n/// the freshness or other cache meta values that cache-control is signaling.\n///\n/// By default `CacheControl` implements an RFC-7234 compliant reading that assumes it is being\n/// used with a shared (proxy) cache.\npub trait InterpretCacheControl {\n    /// Does cache-control specify this response is cacheable?\n    ///\n    /// Note that an RFC-7234 compliant cacheability check must also\n    /// check if the request contained the Authorization header and\n    /// `allow_caching_authorized_req`.\n    fn is_cacheable(&self) -> Cacheable;\n\n    /// Does this cache-control allow caching a response to\n    /// a request with the Authorization header?\n    fn allow_caching_authorized_req(&self) -> bool;\n\n    /// Returns freshness ttl specified in cache-control\n    ///\n    /// - `Some(_)` indicates cache-control specifies a valid ttl. Some(Duration::ZERO) = always stale.\n    /// - `None` means cache-control did not specify a valid ttl.\n    fn fresh_duration(&self) -> Option<Duration>;\n\n    /// Returns stale-while-revalidate ttl,\n    ///\n    /// The result should consider all the relevant cache directives, not just SWR header itself.\n    ///\n    /// Some(0) means serving such stale is disallowed by directive like `must-revalidate`\n    /// or `stale-while-revalidater=0`.\n    ///\n    /// `None` indicates no SWR ttl was specified.\n    fn serve_stale_while_revalidate_duration(&self) -> Option<Duration>;\n\n    /// Returns stale-if-error ttl,\n    ///\n    /// The result should consider all the relevant cache directives, not just SIE header itself.\n    ///\n    /// Some(0) means serving such stale is disallowed by directive like `must-revalidate`\n    /// or `stale-if-error=0`.\n    ///\n    /// `None` indicates no SIE ttl was specified.\n    fn serve_stale_if_error_duration(&self) -> Option<Duration>;\n\n    /// Strip header names listed in `private` or `no-cache` directives from a response,\n    /// usually prior to storing that response in cache.\n    fn strip_private_headers(&self, resp_header: &mut ResponseHeader);\n}\n\n#[cfg(test)]\nmod tests {\n    use super::*;\n    use http::header::CACHE_CONTROL;\n    use http::{request, response};\n\n    fn build_response(cc_key: HeaderName, cc_value: &str) -> response::Parts {\n        let (parts, _) = response::Builder::new()\n            .header(cc_key, cc_value)\n            .body(())\n            .unwrap()\n            .into_parts();\n        parts\n    }\n\n    #[test]\n    fn test_simple_cache_control() {\n        let resp = build_response(CACHE_CONTROL, \"public, max-age=10000\");\n        let cc = CacheControl::from_resp_headers(&resp).unwrap();\n        assert!(cc.public());\n        assert_eq!(cc.max_age().unwrap().unwrap(), 10000);\n    }\n\n    #[test]\n    fn test_private_cache_control() {\n        let resp = build_response(CACHE_CONTROL, \"private\");\n        let cc = CacheControl::from_resp_headers(&resp).unwrap();\n\n        assert!(cc.private());\n        assert!(cc.max_age().unwrap().is_none());\n    }\n\n    #[test]\n    fn test_directives_across_header_lines() {\n        let (parts, _) = response::Builder::new()\n            .header(CACHE_CONTROL, \"public,\")\n            .header(\"cache-Control\", \"max-age=10000\")\n            .body(())\n            .unwrap()\n            .into_parts();\n        let cc = CacheControl::from_resp_headers(&parts).unwrap();\n\n        assert!(cc.public());\n        assert_eq!(cc.max_age().unwrap().unwrap(), 10000);\n    }\n\n    #[test]\n    fn test_recognizes_semicolons_as_delimiters() {\n        let resp = build_response(CACHE_CONTROL, \"public; max-age=0\");\n        let cc = CacheControl::from_resp_headers(&resp).unwrap();\n\n        assert!(cc.public());\n        assert_eq!(cc.max_age().unwrap().unwrap(), 0);\n    }\n\n    #[test]\n    fn test_unknown_directives() {\n        let resp = build_response(CACHE_CONTROL, \"public,random1=random2, rand3=\\\"\\\"\");\n        let cc = CacheControl::from_resp_headers(&resp).unwrap();\n        let mut directive_iter = cc.directives.iter();\n\n        let first = directive_iter.next().unwrap();\n        assert_eq!(first.0, &\"public\");\n        assert!(first.1.is_none());\n\n        let second = directive_iter.next().unwrap();\n        assert_eq!(second.0, &\"random1\");\n        assert_eq!(second.1.as_ref().unwrap().0, \"random2\".as_bytes());\n\n        let third = directive_iter.next().unwrap();\n        assert_eq!(third.0, &\"rand3\");\n        assert_eq!(third.1.as_ref().unwrap().0, \"\\\"\\\"\".as_bytes());\n\n        assert!(directive_iter.next().is_none());\n    }\n\n    #[test]\n    fn test_case_insensitive_directive_keys() {\n        let resp = build_response(\n            CACHE_CONTROL,\n            \"Public=\\\"something\\\", mAx-AGe=\\\"10000\\\", foo=cRaZyCaSe, bAr=\\\"inQuotes\\\"\",\n        );\n        let cc = CacheControl::from_resp_headers(&resp).unwrap();\n\n        assert!(cc.public());\n        assert_eq!(cc.max_age().unwrap().unwrap(), 10000);\n\n        let mut directive_iter = cc.directives.iter();\n        let first = directive_iter.next().unwrap();\n        assert_eq!(first.0, &\"public\");\n        assert_eq!(first.1.as_ref().unwrap().0, \"\\\"something\\\"\".as_bytes());\n\n        let second = directive_iter.next().unwrap();\n        assert_eq!(second.0, &\"max-age\");\n        assert_eq!(second.1.as_ref().unwrap().0, \"\\\"10000\\\"\".as_bytes());\n\n        // values are still stored with casing\n        let third = directive_iter.next().unwrap();\n        assert_eq!(third.0, &\"foo\");\n        assert_eq!(third.1.as_ref().unwrap().0, \"cRaZyCaSe\".as_bytes());\n\n        let fourth = directive_iter.next().unwrap();\n        assert_eq!(fourth.0, &\"bar\");\n        assert_eq!(fourth.1.as_ref().unwrap().0, \"\\\"inQuotes\\\"\".as_bytes());\n\n        assert!(directive_iter.next().is_none());\n    }\n\n    #[test]\n    fn test_non_ascii() {\n        let resp = build_response(CACHE_CONTROL, \"püblic=💖, max-age=\\\"💯\\\"\");\n        let cc = CacheControl::from_resp_headers(&resp).unwrap();\n\n        // Not considered valid registered directive keys / values\n        assert!(!cc.public());\n        assert_eq!(\n            cc.max_age().unwrap_err().context.unwrap().to_string(),\n            \"could not parse value as u32\"\n        );\n\n        let mut directive_iter = cc.directives.iter();\n        let first = directive_iter.next().unwrap();\n        assert_eq!(first.0, &\"püblic\");\n        assert_eq!(first.1.as_ref().unwrap().0, \"💖\".as_bytes());\n\n        let second = directive_iter.next().unwrap();\n        assert_eq!(second.0, &\"max-age\");\n        assert_eq!(second.1.as_ref().unwrap().0, \"\\\"💯\\\"\".as_bytes());\n\n        assert!(directive_iter.next().is_none());\n    }\n\n    #[test]\n    fn test_non_utf8_key() {\n        let mut resp = response::Builder::new().body(()).unwrap();\n        resp.headers_mut().insert(\n            CACHE_CONTROL,\n            HeaderValue::from_bytes(b\"bar\\xFF=\\\"baz\\\", a=b\").unwrap(),\n        );\n        let (parts, _) = resp.into_parts();\n        let cc = CacheControl::from_resp_headers(&parts).unwrap();\n\n        // invalid bytes for key\n        let mut directive_iter = cc.directives.iter();\n        let first = directive_iter.next().unwrap();\n        assert_eq!(first.0, &\"a\");\n        assert_eq!(first.1.as_ref().unwrap().0, \"b\".as_bytes());\n\n        assert!(directive_iter.next().is_none());\n    }\n\n    #[test]\n    fn test_non_utf8_value() {\n        // RFC 7230: 0xFF is part of obs-text and is officially considered a valid octet in quoted-strings\n        let mut resp = response::Builder::new().body(()).unwrap();\n        resp.headers_mut().insert(\n            CACHE_CONTROL,\n            HeaderValue::from_bytes(b\"max-age=ba\\xFFr, bar=\\\"baz\\xFF\\\", a=b\").unwrap(),\n        );\n        let (parts, _) = resp.into_parts();\n        let cc = CacheControl::from_resp_headers(&parts).unwrap();\n\n        assert_eq!(\n            cc.max_age().unwrap_err().context.unwrap().to_string(),\n            \"could not parse value as utf8\"\n        );\n\n        let mut directive_iter = cc.directives.iter();\n\n        let first = directive_iter.next().unwrap();\n        assert_eq!(first.0, &\"max-age\");\n        assert_eq!(first.1.as_ref().unwrap().0, b\"ba\\xFFr\");\n\n        let second = directive_iter.next().unwrap();\n        assert_eq!(second.0, &\"bar\");\n        assert_eq!(second.1.as_ref().unwrap().0, b\"\\\"baz\\xFF\\\"\");\n\n        let third = directive_iter.next().unwrap();\n        assert_eq!(third.0, &\"a\");\n        assert_eq!(third.1.as_ref().unwrap().0, \"b\".as_bytes());\n\n        assert!(directive_iter.next().is_none());\n    }\n\n    #[test]\n    fn test_age_overflow() {\n        let resp = build_response(\n            CACHE_CONTROL,\n            \"max-age=-99999999999999999999999999, s-maxage=99999999999999999999999999\",\n        );\n        let cc = CacheControl::from_resp_headers(&resp).unwrap();\n\n        assert_eq!(\n            cc.s_maxage().unwrap().unwrap(),\n            DELTA_SECONDS_OVERFLOW_VALUE\n        );\n        // negative ages still result in errors even with overflow handling\n        assert_eq!(\n            cc.max_age().unwrap_err().context.unwrap().to_string(),\n            \"could not parse value as u32\"\n        );\n    }\n\n    #[test]\n    fn test_fresh_sec() {\n        let resp = build_response(CACHE_CONTROL, \"\");\n        let cc = CacheControl::from_resp_headers(&resp).unwrap();\n        assert!(cc.fresh_duration().is_none());\n\n        let resp = build_response(CACHE_CONTROL, \"max-age=12345\");\n        let cc = CacheControl::from_resp_headers(&resp).unwrap();\n        assert_eq!(cc.fresh_duration().unwrap(), Duration::from_secs(12345));\n\n        let resp = build_response(CACHE_CONTROL, \"max-age=99999,s-maxage=123\");\n        let cc = CacheControl::from_resp_headers(&resp).unwrap();\n        // prefer s-maxage over max-age\n        assert_eq!(cc.fresh_duration().unwrap(), Duration::from_secs(123));\n    }\n\n    #[test]\n    fn test_cacheability() {\n        let resp = build_response(CACHE_CONTROL, \"\");\n        let cc = CacheControl::from_resp_headers(&resp).unwrap();\n        assert_eq!(cc.is_cacheable(), Cacheable::Default);\n\n        // uncacheable\n        let resp = build_response(CACHE_CONTROL, \"private, max-age=12345\");\n        let cc = CacheControl::from_resp_headers(&resp).unwrap();\n        assert_eq!(cc.is_cacheable(), Cacheable::No);\n\n        let resp = build_response(CACHE_CONTROL, \"no-store, max-age=12345\");\n        let cc = CacheControl::from_resp_headers(&resp).unwrap();\n        assert_eq!(cc.is_cacheable(), Cacheable::No);\n\n        // cacheable\n        let resp = build_response(CACHE_CONTROL, \"public\");\n        let cc = CacheControl::from_resp_headers(&resp).unwrap();\n        assert_eq!(cc.is_cacheable(), Cacheable::Yes);\n\n        let resp = build_response(CACHE_CONTROL, \"max-age=0\");\n        let cc = CacheControl::from_resp_headers(&resp).unwrap();\n        assert_eq!(cc.is_cacheable(), Cacheable::Yes);\n    }\n\n    #[test]\n    fn test_no_cache() {\n        let resp = build_response(CACHE_CONTROL, \"no-cache, max-age=12345\");\n        let cc = CacheControl::from_resp_headers(&resp).unwrap();\n        assert_eq!(cc.is_cacheable(), Cacheable::Yes);\n        assert_eq!(cc.fresh_duration().unwrap(), Duration::ZERO);\n    }\n\n    #[test]\n    fn test_no_cache_field_names() {\n        let resp = build_response(CACHE_CONTROL, \"no-cache=\\\"set-cookie\\\", max-age=12345\");\n        let cc = CacheControl::from_resp_headers(&resp).unwrap();\n        assert!(!cc.private());\n        assert_eq!(cc.is_cacheable(), Cacheable::Yes);\n        assert_eq!(cc.fresh_duration().unwrap(), Duration::from_secs(12345));\n        let mut field_names = cc.no_cache_field_names().unwrap();\n        assert_eq!(\n            str::from_utf8(field_names.next().unwrap()).unwrap(),\n            \"set-cookie\"\n        );\n        assert!(field_names.next().is_none());\n\n        let mut resp = response::Builder::new().body(()).unwrap();\n        resp.headers_mut().insert(\n            CACHE_CONTROL,\n            HeaderValue::from_bytes(\n                b\"private=\\\"\\\", no-cache=\\\"a\\xFF, set-cookie, Baz\\x09 , c,d  ,, \\\"\",\n            )\n            .unwrap(),\n        );\n        let (parts, _) = resp.into_parts();\n        let cc = CacheControl::from_resp_headers(&parts).unwrap();\n        let mut field_names = cc.private_field_names().unwrap();\n        assert_eq!(str::from_utf8(field_names.next().unwrap()).unwrap(), \"\");\n        assert!(field_names.next().is_none());\n        let mut field_names = cc.no_cache_field_names().unwrap();\n        assert!(str::from_utf8(field_names.next().unwrap()).is_err());\n        assert_eq!(\n            str::from_utf8(field_names.next().unwrap()).unwrap(),\n            \"set-cookie\"\n        );\n        assert_eq!(str::from_utf8(field_names.next().unwrap()).unwrap(), \"Baz\");\n        assert_eq!(str::from_utf8(field_names.next().unwrap()).unwrap(), \"c\");\n        assert_eq!(str::from_utf8(field_names.next().unwrap()).unwrap(), \"d\");\n        assert_eq!(str::from_utf8(field_names.next().unwrap()).unwrap(), \"\");\n        assert_eq!(str::from_utf8(field_names.next().unwrap()).unwrap(), \"\");\n        assert!(field_names.next().is_none());\n    }\n\n    #[test]\n    fn test_strip_private_headers() {\n        let mut resp = ResponseHeader::build(200, None).unwrap();\n        resp.append_header(\n            CACHE_CONTROL,\n            \"no-cache=\\\"x-private-header\\\", max-age=12345\",\n        )\n        .unwrap();\n        resp.append_header(\"X-Private-Header\", \"dropped\").unwrap();\n\n        let cc = CacheControl::from_resp_headers(&resp).unwrap();\n        cc.strip_private_headers(&mut resp);\n        assert!(!resp.headers.contains_key(\"X-Private-Header\"));\n    }\n\n    #[test]\n    fn test_stale_while_revalidate() {\n        let resp = build_response(CACHE_CONTROL, \"max-age=12345, stale-while-revalidate=5\");\n        let cc = CacheControl::from_resp_headers(&resp).unwrap();\n        assert_eq!(cc.stale_while_revalidate().unwrap().unwrap(), 5);\n        assert_eq!(\n            cc.serve_stale_while_revalidate_duration().unwrap(),\n            Duration::from_secs(5)\n        );\n        assert!(cc.serve_stale_if_error_duration().is_none());\n    }\n\n    #[test]\n    fn test_stale_if_error() {\n        let resp = build_response(CACHE_CONTROL, \"max-age=12345, stale-if-error=3600\");\n        let cc = CacheControl::from_resp_headers(&resp).unwrap();\n        assert_eq!(cc.stale_if_error().unwrap().unwrap(), 3600);\n        assert_eq!(\n            cc.serve_stale_if_error_duration().unwrap(),\n            Duration::from_secs(3600)\n        );\n        assert!(cc.serve_stale_while_revalidate_duration().is_none());\n    }\n\n    #[test]\n    fn test_must_revalidate() {\n        let resp = build_response(\n            CACHE_CONTROL,\n            \"max-age=12345, stale-while-revalidate=60, stale-if-error=30, must-revalidate\",\n        );\n        let cc = CacheControl::from_resp_headers(&resp).unwrap();\n        assert!(cc.must_revalidate());\n        assert_eq!(cc.stale_while_revalidate().unwrap().unwrap(), 60);\n        assert_eq!(cc.stale_if_error().unwrap().unwrap(), 30);\n        assert_eq!(\n            cc.serve_stale_while_revalidate_duration().unwrap(),\n            Duration::ZERO\n        );\n        assert_eq!(cc.serve_stale_if_error_duration().unwrap(), Duration::ZERO);\n    }\n\n    #[test]\n    fn test_proxy_revalidate() {\n        let resp = build_response(\n            CACHE_CONTROL,\n            \"max-age=12345, stale-while-revalidate=60, stale-if-error=30, proxy-revalidate\",\n        );\n        let cc = CacheControl::from_resp_headers(&resp).unwrap();\n        assert!(cc.proxy_revalidate());\n        assert_eq!(cc.stale_while_revalidate().unwrap().unwrap(), 60);\n        assert_eq!(cc.stale_if_error().unwrap().unwrap(), 30);\n        assert_eq!(\n            cc.serve_stale_while_revalidate_duration().unwrap(),\n            Duration::ZERO\n        );\n        assert_eq!(cc.serve_stale_if_error_duration().unwrap(), Duration::ZERO);\n    }\n\n    #[test]\n    fn test_s_maxage_stale() {\n        let resp = build_response(\n            CACHE_CONTROL,\n            \"s-maxage=0, stale-while-revalidate=60, stale-if-error=30\",\n        );\n        let cc = CacheControl::from_resp_headers(&resp).unwrap();\n        assert_eq!(cc.stale_while_revalidate().unwrap().unwrap(), 60);\n        assert_eq!(cc.stale_if_error().unwrap().unwrap(), 30);\n        assert_eq!(\n            cc.serve_stale_while_revalidate_duration().unwrap(),\n            Duration::ZERO\n        );\n        assert_eq!(cc.serve_stale_if_error_duration().unwrap(), Duration::ZERO);\n    }\n\n    #[test]\n    fn test_authorized_request() {\n        let resp = build_response(CACHE_CONTROL, \"max-age=10\");\n        let cc = CacheControl::from_resp_headers(&resp).unwrap();\n        assert!(!cc.allow_caching_authorized_req());\n\n        let resp = build_response(CACHE_CONTROL, \"s-maxage=10\");\n        let cc = CacheControl::from_resp_headers(&resp).unwrap();\n        assert!(cc.allow_caching_authorized_req());\n\n        let resp = build_response(CACHE_CONTROL, \"public\");\n        let cc = CacheControl::from_resp_headers(&resp).unwrap();\n        assert!(cc.allow_caching_authorized_req());\n\n        let resp = build_response(CACHE_CONTROL, \"must-revalidate, max-age=0\");\n        let cc = CacheControl::from_resp_headers(&resp).unwrap();\n        assert!(cc.allow_caching_authorized_req());\n\n        let resp = build_response(CACHE_CONTROL, \"\");\n        let cc = CacheControl::from_resp_headers(&resp).unwrap();\n        assert!(!cc.allow_caching_authorized_req());\n    }\n\n    fn build_request(cc_key: HeaderName, cc_value: &str) -> request::Parts {\n        let (parts, _) = request::Builder::new()\n            .header(cc_key, cc_value)\n            .body(())\n            .unwrap()\n            .into_parts();\n        parts\n    }\n\n    #[test]\n    fn test_request_only_if_cached() {\n        let req = build_request(CACHE_CONTROL, \"only-if-cached=1\");\n        let cc = CacheControl::from_req_headers(&req).unwrap();\n        assert!(cc.only_if_cached())\n    }\n}\n"
  },
  {
    "path": "pingora-cache/src/eviction/lru.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! A shared LRU cache manager\n\nuse super::EvictionManager;\nuse crate::key::CompactCacheKey;\n\nuse async_trait::async_trait;\nuse log::{info, warn};\nuse pingora_error::{BError, ErrorType::*, OrErr, Result};\nuse pingora_lru::Lru;\nuse rand::Rng;\nuse serde::de::SeqAccess;\nuse serde::{Deserialize, Serialize};\nuse std::fs::{rename, File};\nuse std::hash::{Hash, Hasher};\nuse std::io::prelude::*;\nuse std::path::Path;\nuse std::time::SystemTime;\n\n/// A shared LRU cache manager designed to manage a large volume of assets.\n///\n/// - Space optimized in-memory LRU (see [pingora_lru]).\n/// - Instead of a single giant LRU, this struct shards the assets into `N` independent LRUs.\n///\n/// This allows [EvictionManager::save()] not to lock the entire cache manager while performing\n/// serialization.\npub struct Manager<const N: usize>(Lru<CompactCacheKey, N>);\n\n#[derive(Debug, Serialize, Deserialize)]\nstruct SerdeHelperNode(CompactCacheKey, usize);\n\nimpl<const N: usize> Manager<N> {\n    /// Create a [Manager] with the given size limit and estimated per shard capacity.\n    ///\n    /// The `capacity` is for preallocating to avoid reallocation cost when the LRU grows.\n    pub fn with_capacity(limit: usize, capacity: usize) -> Self {\n        Manager(Lru::with_capacity(limit, capacity))\n    }\n\n    /// Create a [Manager] with an optional watermark in addition to weight limit.\n    ///\n    /// When `watermark` is set, the underlying LRU will also evict to keep total item count\n    /// under or equal to that watermark.\n    pub fn with_capacity_and_watermark(\n        limit: usize,\n        capacity: usize,\n        watermark: Option<usize>,\n    ) -> Self {\n        Manager(Lru::with_capacity_and_watermark(limit, capacity, watermark))\n    }\n\n    /// Get the number of shards\n    pub fn shards(&self) -> usize {\n        self.0.shards()\n    }\n\n    /// Get the weight (total size) of a specific shard\n    pub fn shard_weight(&self, shard: usize) -> usize {\n        self.0.shard_weight(shard)\n    }\n\n    /// Get the number of items in a specific shard\n    pub fn shard_len(&self, shard: usize) -> usize {\n        self.0.shard_len(shard)\n    }\n\n    /// Get the shard index for a given cache key\n    ///\n    /// This allows callers to know which shard was affected by an operation\n    /// without acquiring any locks.\n    pub fn get_shard_for_key(&self, key: &CompactCacheKey) -> usize {\n        (u64key(key) % N as u64) as usize\n    }\n\n    /// Serialize the given shard\n    pub fn serialize_shard(&self, shard: usize) -> Result<Vec<u8>> {\n        use rmp_serde::encode::Serializer;\n        use serde::ser::SerializeSeq;\n        use serde::ser::Serializer as _;\n\n        assert!(shard < N);\n\n        // NOTE: This could use a lot of memory to buffer the serialized data in memory\n        // NOTE: This for loop could lock the LRU for too long\n        let mut nodes = Vec::with_capacity(self.0.shard_len(shard));\n        self.0.iter_for_each(shard, |(node, size)| {\n            nodes.push(SerdeHelperNode(node.clone(), size));\n        });\n        let mut ser = Serializer::new(vec![]);\n        let mut seq = ser\n            .serialize_seq(Some(self.0.shard_len(shard)))\n            .or_err(InternalError, \"fail to serialize node\")?;\n        for node in nodes {\n            seq.serialize_element(&node).unwrap(); // write to vec, safe\n        }\n\n        seq.end().or_err(InternalError, \"when serializing LRU\")?;\n        Ok(ser.into_inner())\n    }\n\n    /// Deserialize a shard\n    ///\n    /// Shard number is not needed because the key itself will hash to the correct shard.\n    pub fn deserialize_shard(&self, buf: &[u8]) -> Result<()> {\n        use rmp_serde::decode::Deserializer;\n        use serde::de::Deserializer as _;\n\n        let mut de = Deserializer::new(buf);\n        let visitor = InsertToManager { lru: self };\n        de.deserialize_seq(visitor)\n            .or_err(InternalError, \"when deserializing LRU\")?;\n        Ok(())\n    }\n\n    /// Peek the weight associated with a cache key without changing its LRU order.\n    pub fn peek_weight(&self, item: &CompactCacheKey) -> Option<usize> {\n        let key = u64key(item);\n        self.0.peek_weight(key)\n    }\n}\n\nstruct InsertToManager<'a, const N: usize> {\n    lru: &'a Manager<N>,\n}\n\nimpl<'de, const N: usize> serde::de::Visitor<'de> for InsertToManager<'_, N> {\n    type Value = ();\n\n    fn expecting(&self, formatter: &mut std::fmt::Formatter) -> std::fmt::Result {\n        formatter.write_str(\"array of lru nodes\")\n    }\n\n    fn visit_seq<A>(self, mut seq: A) -> Result<Self::Value, A::Error>\n    where\n        A: SeqAccess<'de>,\n    {\n        while let Some(node) = seq.next_element::<SerdeHelperNode>()? {\n            let key = u64key(&node.0);\n            self.lru.0.insert_tail(key, node.0, node.1); // insert in the back\n        }\n        Ok(())\n    }\n}\n\n#[inline]\nfn u64key(key: &CompactCacheKey) -> u64 {\n    // note that std hash is not uniform, I'm not sure if ahash is also the case\n    let mut hasher = ahash::AHasher::default();\n    key.hash(&mut hasher);\n    hasher.finish()\n}\n\nconst FILE_NAME: &str = \"lru.data\";\n\n#[inline]\nfn err_str_path(s: &str, path: &Path) -> String {\n    format!(\"{s} {}\", path.display())\n}\n\n#[async_trait]\nimpl<const N: usize> EvictionManager for Manager<N> {\n    fn total_size(&self) -> usize {\n        self.0.weight()\n    }\n    fn total_items(&self) -> usize {\n        self.0.len()\n    }\n    fn evicted_size(&self) -> usize {\n        self.0.evicted_weight()\n    }\n    fn evicted_items(&self) -> usize {\n        self.0.evicted_len()\n    }\n\n    fn admit(\n        &self,\n        item: CompactCacheKey,\n        size: usize,\n        _fresh_until: SystemTime,\n    ) -> Vec<CompactCacheKey> {\n        let key = u64key(&item);\n        self.0.admit(key, item, size);\n        self.0\n            .evict_to_limit()\n            .into_iter()\n            .map(|(key, _weight)| key)\n            .collect()\n    }\n\n    fn increment_weight(\n        &self,\n        item: &CompactCacheKey,\n        delta: usize,\n        max_weight: Option<usize>,\n    ) -> Vec<CompactCacheKey> {\n        let key = u64key(item);\n        self.0.increment_weight(key, delta, max_weight);\n        self.0\n            .evict_to_limit()\n            .into_iter()\n            .map(|(key, _weight)| key)\n            .collect()\n    }\n\n    fn remove(&self, item: &CompactCacheKey) {\n        let key = u64key(item);\n        self.0.remove(key);\n    }\n\n    fn access(&self, item: &CompactCacheKey, size: usize, _fresh_until: SystemTime) -> bool {\n        let key = u64key(item);\n        if !self.0.promote(key) {\n            self.0.admit(key, item.clone(), size);\n            false\n        } else {\n            true\n        }\n    }\n\n    fn peek(&self, item: &CompactCacheKey) -> bool {\n        let key = u64key(item);\n        self.0.peek(key)\n    }\n\n    async fn save(&self, dir_path: &str) -> Result<()> {\n        let dir_path_str = dir_path.to_owned();\n\n        tokio::task::spawn_blocking(move || {\n            let dir_path = Path::new(&dir_path_str);\n            std::fs::create_dir_all(dir_path)\n                .or_err_with(InternalError, || err_str_path(\"fail to create\", dir_path))\n        })\n        .await\n        .or_err(InternalError, \"async blocking IO failure\")??;\n\n        for i in 0..N {\n            let data = self.serialize_shard(i)?;\n            let dir_path = dir_path.to_owned();\n            tokio::task::spawn_blocking(move || {\n                let dir_path = Path::new(&dir_path);\n                let final_path = dir_path.join(format!(\"{}.{i}\", FILE_NAME));\n                // create a temporary filename using a randomized u32 hash to minimize the chance of multiple writers writing to the same tmp file\n                let random_suffix: u32 = rand::thread_rng().gen();\n                let temp_path =\n                    dir_path.join(format!(\"{}.{i}.{:08x}.tmp\", FILE_NAME, random_suffix));\n                let mut file = File::create(&temp_path)\n                    .or_err_with(InternalError, || err_str_path(\"fail to create\", &temp_path))?;\n                file.write_all(&data).or_err_with(InternalError, || {\n                    err_str_path(\"fail to write to\", &temp_path)\n                })?;\n                file.flush().or_err_with(InternalError, || {\n                    err_str_path(\"fail to flush temp file\", &temp_path)\n                })?;\n                rename(&temp_path, &final_path).or_err_with(InternalError, || {\n                    format!(\n                        \"Failed to rename file from {} to {}\",\n                        temp_path.display(),\n                        final_path.display(),\n                    )\n                })\n            })\n            .await\n            .or_err(InternalError, \"async blocking IO failure\")??;\n        }\n        Ok(())\n    }\n\n    async fn load(&self, dir_path: &str) -> Result<()> {\n        // TODO: check the saved shards so that we load all the save files\n        let mut loaded_shards = 0;\n        for i in 0..N {\n            let dir_path = dir_path.to_owned();\n\n            let data = tokio::task::spawn_blocking(move || {\n                let file_path = Path::new(&dir_path).join(format!(\"{}.{i}\", FILE_NAME));\n                let mut file = File::open(&file_path)\n                    .or_err_with(InternalError, || err_str_path(\"fail to open\", &file_path))?;\n                let mut buffer = Vec::with_capacity(8192);\n                file.read_to_end(&mut buffer)\n                    .or_err_with(InternalError, || {\n                        err_str_path(\"fail to read from\", &file_path)\n                    })?;\n                Ok::<Vec<u8>, BError>(buffer)\n            })\n            .await\n            .or_err(InternalError, \"async blocking IO failure\")??;\n\n            if let Err(e) = self.deserialize_shard(&data) {\n                warn!(\"Failed to deserialize shard {}: {}. Skipping shard.\", i, e);\n                continue; // Skip shard and move onto the next one\n            }\n            loaded_shards += 1;\n        }\n\n        // Log how many shards were successfully loaded\n        if loaded_shards < N {\n            warn!(\n                \"Only loaded {}/{} shards. Cache may be incomplete.\",\n                loaded_shards, N\n            )\n        } else {\n            info!(\"Successfully loaded {}/{} shards.\", loaded_shards, N)\n        }\n\n        cleanup_temp_files(dir_path);\n\n        Ok(())\n    }\n}\n\nfn cleanup_temp_files(dir_path: &str) {\n    let dir_path = Path::new(dir_path).to_owned();\n\n    tokio::task::spawn_blocking({\n        move || {\n            if !dir_path.exists() {\n                return;\n            }\n\n            let entries = match std::fs::read_dir(&dir_path) {\n                Ok(entries) => entries,\n                Err(e) => {\n                    warn!(\"Failed to read directory {}: {e}\", dir_path.display());\n                    return;\n                }\n            };\n\n            let mut cleaned_count = 0;\n            let mut error_count = 0;\n\n            for entry in entries {\n                let entry = match entry {\n                    Ok(entry) => entry,\n                    Err(e) => {\n                        warn!(\n                            \"Failed to read directory entry in {}: {e}\",\n                            dir_path.display()\n                        );\n                        error_count += 1;\n                        continue;\n                    }\n                };\n\n                let file_name = entry.file_name();\n                let file_name_str = file_name.to_string_lossy();\n\n                if file_name_str.starts_with(FILE_NAME) && file_name_str.ends_with(\".tmp\") {\n                    match std::fs::remove_file(entry.path()) {\n                        Ok(()) => {\n                            info!(\"Cleaned up orphaned temp file: {}\", entry.path().display());\n                            cleaned_count += 1;\n                        }\n                        Err(e) => {\n                            warn!(\"Failed to remove temp file {}: {e}\", entry.path().display());\n                            error_count += 1;\n                        }\n                    }\n                }\n            }\n\n            if cleaned_count > 0 || error_count > 0 {\n                info!(\n                    \"Temp file cleanup completed. Removed: {cleaned_count}, Errors: {error_count}\"\n                );\n            }\n        }\n    });\n}\n\n#[cfg(test)]\nmod test {\n    use super::*;\n    use crate::CacheKey;\n\n    // we use shard (N) = 1 for eviction consistency in all tests\n\n    #[test]\n    fn test_admission() {\n        let lru = Manager::<1>::with_capacity(4, 10);\n        let key1 = CacheKey::new(\"\", \"a\", \"1\").to_compact();\n        let until = SystemTime::now(); // unused value as a placeholder\n        let v = lru.admit(key1.clone(), 1, until);\n        assert_eq!(v.len(), 0);\n        let key2 = CacheKey::new(\"\", \"b\", \"1\").to_compact();\n        let v = lru.admit(key2.clone(), 2, until);\n        assert_eq!(v.len(), 0);\n        let key3 = CacheKey::new(\"\", \"c\", \"1\").to_compact();\n        let v = lru.admit(key3, 1, until);\n        assert_eq!(v.len(), 0);\n\n        // lru si full (4) now\n\n        let key4 = CacheKey::new(\"\", \"d\", \"1\").to_compact();\n        let v = lru.admit(key4, 2, until);\n        // need to reduce used by at least 2, both key1 and key2 are evicted to make room for 3\n        assert_eq!(v.len(), 2);\n        assert_eq!(v[0], key1);\n        assert_eq!(v[1], key2);\n    }\n\n    #[test]\n    fn test_access() {\n        let lru = Manager::<1>::with_capacity(4, 10);\n        let key1 = CacheKey::new(\"\", \"a\", \"1\").to_compact();\n        let until = SystemTime::now(); // unused value as a placeholder\n        let v = lru.admit(key1.clone(), 1, until);\n        assert_eq!(v.len(), 0);\n        let key2 = CacheKey::new(\"\", \"b\", \"1\").to_compact();\n        let v = lru.admit(key2.clone(), 2, until);\n        assert_eq!(v.len(), 0);\n        let key3 = CacheKey::new(\"\", \"c\", \"1\").to_compact();\n        let v = lru.admit(key3, 1, until);\n        assert_eq!(v.len(), 0);\n\n        // lru is full (4) now\n        // make key1 most recently used\n        lru.access(&key1, 1, until);\n        assert_eq!(v.len(), 0);\n\n        let key4 = CacheKey::new(\"\", \"d\", \"1\").to_compact();\n        let v = lru.admit(key4, 2, until);\n        assert_eq!(v.len(), 1);\n        assert_eq!(v[0], key2);\n    }\n\n    #[test]\n    fn test_remove() {\n        let lru = Manager::<1>::with_capacity(4, 10);\n        let key1 = CacheKey::new(\"\", \"a\", \"1\").to_compact();\n        let until = SystemTime::now(); // unused value as a placeholder\n        let v = lru.admit(key1.clone(), 1, until);\n        assert_eq!(v.len(), 0);\n        let key2 = CacheKey::new(\"\", \"b\", \"1\").to_compact();\n        let v = lru.admit(key2.clone(), 2, until);\n        assert_eq!(v.len(), 0);\n        let key3 = CacheKey::new(\"\", \"c\", \"1\").to_compact();\n        let v = lru.admit(key3, 1, until);\n        assert_eq!(v.len(), 0);\n\n        // lru is full (4) now\n        // remove key1\n        lru.remove(&key1);\n\n        // key2 is the least recently used one now\n        let key4 = CacheKey::new(\"\", \"d\", \"1\").to_compact();\n        let v = lru.admit(key4, 2, until);\n        assert_eq!(v.len(), 1);\n        assert_eq!(v[0], key2);\n    }\n\n    #[test]\n    fn test_access_add() {\n        let lru = Manager::<1>::with_capacity(4, 10);\n        let until = SystemTime::now(); // unused value as a placeholder\n\n        let key1 = CacheKey::new(\"\", \"a\", \"1\").to_compact();\n        lru.access(&key1, 1, until);\n        let key2 = CacheKey::new(\"\", \"b\", \"1\").to_compact();\n        lru.access(&key2, 2, until);\n        let key3 = CacheKey::new(\"\", \"c\", \"1\").to_compact();\n        lru.access(&key3, 2, until);\n\n        let key4 = CacheKey::new(\"\", \"d\", \"1\").to_compact();\n        let v = lru.admit(key4, 2, until);\n        // need to reduce used by at least 2, both key1 and key2 are evicted to make room for 3\n        assert_eq!(v.len(), 2);\n        assert_eq!(v[0], key1);\n        assert_eq!(v[1], key2);\n    }\n\n    #[test]\n    fn test_admit_update() {\n        let lru = Manager::<1>::with_capacity(4, 10);\n        let key1 = CacheKey::new(\"\", \"a\", \"1\").to_compact();\n        let until = SystemTime::now(); // unused value as a placeholder\n        let v = lru.admit(key1.clone(), 1, until);\n        assert_eq!(v.len(), 0);\n        let key2 = CacheKey::new(\"\", \"b\", \"1\").to_compact();\n        let v = lru.admit(key2.clone(), 2, until);\n        assert_eq!(v.len(), 0);\n        let key3 = CacheKey::new(\"\", \"c\", \"1\").to_compact();\n        let v = lru.admit(key3, 1, until);\n        assert_eq!(v.len(), 0);\n\n        // lru is full (4) now\n        // update key2 to reduce its size by 1\n        let v = lru.admit(key2, 1, until);\n        assert_eq!(v.len(), 0);\n\n        // lru is not full anymore\n        let key4 = CacheKey::new(\"\", \"d\", \"1\").to_compact();\n        let v = lru.admit(key4.clone(), 1, until);\n        assert_eq!(v.len(), 0);\n\n        // make key4 larger\n        let v = lru.admit(key4, 2, until);\n        // need to evict now\n        assert_eq!(v.len(), 1);\n        assert_eq!(v[0], key1);\n    }\n\n    #[test]\n    fn test_peek() {\n        let lru = Manager::<1>::with_capacity(4, 10);\n        let until = SystemTime::now(); // unused value as a placeholder\n\n        let key1 = CacheKey::new(\"\", \"a\", \"1\").to_compact();\n        lru.access(&key1, 1, until);\n        let key2 = CacheKey::new(\"\", \"b\", \"1\").to_compact();\n        lru.access(&key2, 2, until);\n        assert!(lru.peek(&key1));\n        assert!(lru.peek(&key2));\n    }\n\n    #[test]\n    fn test_serde() {\n        let lru = Manager::<1>::with_capacity(4, 10);\n        let key1 = CacheKey::new(\"\", \"a\", \"1\").to_compact();\n        let until = SystemTime::now(); // unused value as a placeholder\n        let v = lru.admit(key1.clone(), 1, until);\n        assert_eq!(v.len(), 0);\n        let key2 = CacheKey::new(\"\", \"b\", \"1\").to_compact();\n        let v = lru.admit(key2.clone(), 2, until);\n        assert_eq!(v.len(), 0);\n        let key3 = CacheKey::new(\"\", \"c\", \"1\").to_compact();\n        let v = lru.admit(key3, 1, until);\n        assert_eq!(v.len(), 0);\n\n        // lru is full (4) now\n        // make key1 most recently used\n        lru.access(&key1, 1, until);\n        assert_eq!(v.len(), 0);\n\n        // load lru2 with lru's data\n        let ser = lru.serialize_shard(0).unwrap();\n        let lru2 = Manager::<1>::with_capacity(4, 10);\n        lru2.deserialize_shard(&ser).unwrap();\n\n        let key4 = CacheKey::new(\"\", \"d\", \"1\").to_compact();\n        let v = lru2.admit(key4, 2, until);\n        assert_eq!(v.len(), 1);\n        assert_eq!(v[0], key2);\n    }\n\n    #[tokio::test]\n    async fn test_save_to_disk() {\n        let until = SystemTime::now(); // unused value as a placeholder\n        let lru = Manager::<2>::with_capacity(10, 10);\n\n        lru.admit(CacheKey::new(\"\", \"a\", \"1\").to_compact(), 1, until);\n        lru.admit(CacheKey::new(\"\", \"b\", \"1\").to_compact(), 2, until);\n        lru.admit(CacheKey::new(\"\", \"c\", \"1\").to_compact(), 1, until);\n        lru.admit(CacheKey::new(\"\", \"d\", \"1\").to_compact(), 1, until);\n        lru.admit(CacheKey::new(\"\", \"e\", \"1\").to_compact(), 2, until);\n        lru.admit(CacheKey::new(\"\", \"f\", \"1\").to_compact(), 1, until);\n\n        // load lru2 with lru's data\n        lru.save(\"/tmp/test_lru_save\").await.unwrap();\n        let lru2 = Manager::<2>::with_capacity(4, 10);\n        lru2.load(\"/tmp/test_lru_save\").await.unwrap();\n\n        let ser0 = lru.serialize_shard(0).unwrap();\n        let ser1 = lru.serialize_shard(1).unwrap();\n\n        assert_eq!(ser0, lru2.serialize_shard(0).unwrap());\n        assert_eq!(ser1, lru2.serialize_shard(1).unwrap());\n    }\n\n    #[tokio::test]\n    async fn test_temp_file_cleanup() {\n        let test_dir = \"/tmp/test_lru_cleanup\";\n        let dir_path = Path::new(test_dir);\n\n        // Create test directory\n        std::fs::create_dir_all(dir_path).unwrap();\n\n        // Create some fake temp files\n        let temp_files = [\n            \"lru.data.0.12345678.tmp\",\n            \"lru.data.1.abcdef00.tmp\",\n            \"other_file.tmp\", // Should not be removed\n            \"lru.data.2\",     // Should not be removed\n        ];\n\n        for file in temp_files {\n            let file_path = dir_path.join(file);\n            std::fs::write(&file_path, b\"test\").unwrap();\n        }\n\n        // Run cleanup\n        cleanup_temp_files(test_dir);\n\n        tokio::time::sleep(core::time::Duration::from_secs(1)).await;\n\n        // Check results\n        assert!(!dir_path.join(\"lru.data.0.12345678.tmp\").exists());\n        assert!(!dir_path.join(\"lru.data.1.abcdef00.tmp\").exists());\n        assert!(dir_path.join(\"other_file.tmp\").exists()); // Should remain\n        assert!(dir_path.join(\"lru.data.2\").exists()); // Should remain\n\n        // Cleanup test directory\n        std::fs::remove_dir_all(dir_path).unwrap();\n    }\n}\n"
  },
  {
    "path": "pingora-cache/src/eviction/mod.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! Cache eviction module\n\nuse crate::key::CompactCacheKey;\n\nuse async_trait::async_trait;\nuse pingora_error::Result;\nuse std::time::SystemTime;\n\npub mod lru;\npub mod simple_lru;\n\n/// The trait that a cache eviction algorithm needs to implement\n///\n/// NOTE: these trait methods require &self not &mut self, which means concurrency should\n/// be handled the implementations internally.\n#[async_trait]\npub trait EvictionManager: Send + Sync {\n    /// Total size of the cache in bytes tracked by this eviction manager\n    fn total_size(&self) -> usize;\n    /// Number of assets tracked by this eviction manager\n    fn total_items(&self) -> usize;\n    /// Number of bytes that are already evicted\n    ///\n    /// The accumulated number is returned to play well with Prometheus counter metric type.\n    fn evicted_size(&self) -> usize;\n    /// Number of assets that are already evicted\n    ///\n    /// The accumulated number is returned to play well with Prometheus counter metric type.\n    fn evicted_items(&self) -> usize;\n\n    /// Admit an item\n    ///\n    /// Return one or more items to evict. The sizes of these items are deducted\n    /// from the total size already. The caller needs to make sure that these assets are actually\n    /// removed from the storage.\n    ///\n    /// If the item is already admitted, A. update its freshness; B. if the new size is larger than the\n    /// existing one, Some(_) might be returned for the caller to evict.\n    fn admit(\n        &self,\n        item: CompactCacheKey,\n        size: usize,\n        fresh_until: SystemTime,\n    ) -> Vec<CompactCacheKey>;\n\n    /// Adjust an item's weight upwards by a delta. If the item is not already admitted,\n    /// nothing will happen.\n    ///\n    /// An optional `max_weight` hint indicates the known max weight of the current key in case the\n    /// weight should not be incremented above this amount.\n    ///\n    /// Return one or more items to evict. The sizes of these items are deducted\n    /// from the total size already. The caller needs to make sure that these assets are actually\n    /// removed from the storage.\n    fn increment_weight(\n        &self,\n        item: &CompactCacheKey,\n        delta: usize,\n        max_weight: Option<usize>,\n    ) -> Vec<CompactCacheKey>;\n\n    /// Remove an item from the eviction manager.\n    ///\n    /// The size of the item will be deducted.\n    fn remove(&self, item: &CompactCacheKey);\n\n    /// Access an item that should already be in cache.\n    ///\n    /// If the item is not tracked by this [EvictionManager], track it but no eviction will happen.\n    ///\n    /// The call used for asking the eviction manager to track the assets that are already admitted\n    /// in the cache storage system.\n    fn access(&self, item: &CompactCacheKey, size: usize, fresh_until: SystemTime) -> bool;\n\n    /// Peek into the manager to see if the item is already tracked by the system\n    ///\n    /// This function should have no side-effect on the asset itself. For example, for LRU, this\n    /// method shouldn't change the popularity of the asset being peeked.\n    fn peek(&self, item: &CompactCacheKey) -> bool;\n\n    /// Serialize to save the state of this eviction manager to disk\n    ///\n    /// This function is for preserving the eviction manager's state across server restarts.\n    ///\n    /// `dir_path` define the directory on disk that the data should use.\n    // dir_path is &str no AsRef<Path> so that trait objects can be used\n    async fn save(&self, dir_path: &str) -> Result<()>;\n\n    /// The counterpart of [Self::save()].\n    async fn load(&self, dir_path: &str) -> Result<()>;\n}\n"
  },
  {
    "path": "pingora-cache/src/eviction/simple_lru.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! A simple LRU cache manager built on top of the `lru` crate\n\nuse super::EvictionManager;\nuse crate::key::CompactCacheKey;\n\nuse async_trait::async_trait;\nuse lru::LruCache;\nuse parking_lot::RwLock;\nuse pingora_error::{BError, ErrorType::*, OrErr, Result};\nuse rand::Rng;\nuse serde::de::SeqAccess;\nuse serde::{Deserialize, Serialize};\nuse std::collections::hash_map::DefaultHasher;\nuse std::fs::File;\nuse std::hash::{Hash, Hasher};\nuse std::io::prelude::*;\nuse std::path::Path;\nuse std::sync::atomic::{AtomicUsize, Ordering};\nuse std::time::SystemTime;\n\n#[derive(Debug, Deserialize, Serialize)]\nstruct Node {\n    key: CompactCacheKey,\n    size: usize,\n}\n\n/// A simple LRU eviction manager\n///\n/// The implementation is not optimized. All operations require global locks.\npub struct Manager {\n    lru: RwLock<LruCache<u64, Node>>,\n    limit: usize,\n    items_watermark: Option<usize>,\n    used: AtomicUsize,\n    items: AtomicUsize,\n    evicted_size: AtomicUsize,\n    evicted_items: AtomicUsize,\n}\n\nimpl Manager {\n    /// Create a new [Manager] with the given total size limit `limit`.\n    pub fn new(limit: usize) -> Self {\n        Manager {\n            lru: RwLock::new(LruCache::unbounded()),\n            limit,\n            items_watermark: None,\n            used: AtomicUsize::new(0),\n            items: AtomicUsize::new(0),\n            evicted_size: AtomicUsize::new(0),\n            evicted_items: AtomicUsize::new(0),\n        }\n    }\n\n    /// Create a new [Manager] with optional watermark in addition to size limit `limit`.\n    pub fn new_with_watermark(limit: usize, items_watermark: Option<usize>) -> Self {\n        Manager {\n            lru: RwLock::new(LruCache::unbounded()),\n            limit,\n            items_watermark,\n            used: AtomicUsize::new(0),\n            items: AtomicUsize::new(0),\n            evicted_size: AtomicUsize::new(0),\n            evicted_items: AtomicUsize::new(0),\n        }\n    }\n\n    fn insert(&self, hash_key: u64, node: CompactCacheKey, size: usize, reverse: bool) {\n        use std::cmp::Ordering::*;\n        let node = Node { key: node, size };\n        let old = {\n            let mut lru = self.lru.write();\n            let old = lru.push(hash_key, node);\n            if reverse && old.is_none() {\n                lru.demote(&hash_key);\n            }\n            old\n        };\n        if let Some(old) = old {\n            // replacing a node, just need to update used size\n            match size.cmp(&old.1.size) {\n                Greater => self.used.fetch_add(size - old.1.size, Ordering::Relaxed),\n                Less => self.used.fetch_sub(old.1.size - size, Ordering::Relaxed),\n                Equal => 0, // same size, update nothing, use 0 to match other arms' type\n            };\n        } else {\n            self.used.fetch_add(size, Ordering::Relaxed);\n            self.items.fetch_add(1, Ordering::Relaxed);\n        }\n    }\n\n    fn increase_weight(&self, key: u64, delta: usize) {\n        let mut lru = self.lru.write();\n        let Some(node) = lru.get_key_value_mut(&key) else {\n            return;\n        };\n        node.1.size += delta;\n        self.used.fetch_add(delta, Ordering::Relaxed);\n    }\n\n    #[inline]\n    fn over_limits(&self) -> bool {\n        self.used.load(Ordering::Relaxed) > self.limit\n            || self\n                .items_watermark\n                .is_some_and(|w| self.items.load(Ordering::Relaxed) > w)\n    }\n\n    // evict items until the used capacity is below the size limit and watermark count\n    fn evict(&self) -> Vec<CompactCacheKey> {\n        if self.used.load(Ordering::Relaxed) <= self.limit\n            && self\n                .items_watermark\n                .is_none_or(|w| self.items.load(Ordering::Relaxed) <= w)\n        {\n            return vec![];\n        }\n\n        let mut to_evict = Vec::with_capacity(1); // we will at least pop 1 item\n\n        while self.over_limits() {\n            if let Some((_, node)) = self.lru.write().pop_lru() {\n                self.used.fetch_sub(node.size, Ordering::Relaxed);\n                self.items.fetch_sub(1, Ordering::Relaxed);\n                self.evicted_size.fetch_add(node.size, Ordering::Relaxed);\n                self.evicted_items.fetch_add(1, Ordering::Relaxed);\n                to_evict.push(node.key);\n            } else {\n                // lru empty\n                return to_evict;\n            }\n        }\n        to_evict\n    }\n\n    // This could use a lot of memory to buffer the serialized data in memory and could lock the LRU\n    // for too long\n    fn serialize(&self) -> Result<Vec<u8>> {\n        use rmp_serde::encode::Serializer;\n        use serde::ser::SerializeSeq;\n        use serde::ser::Serializer as _;\n        // NOTE: This could use a lot of memory to buffer the serialized data in memory\n        let mut ser = Serializer::new(vec![]);\n        // NOTE: This long for loop could lock the LRU for too long\n        let lru = self.lru.read();\n        let mut seq = ser\n            .serialize_seq(Some(lru.len()))\n            .or_err(InternalError, \"fail to serialize node\")?;\n        for item in lru.iter() {\n            seq.serialize_element(item.1).unwrap(); // write to vec, safe\n        }\n        seq.end().or_err(InternalError, \"when serializing LRU\")?;\n        Ok(ser.into_inner())\n    }\n\n    fn deserialize(&self, buf: &[u8]) -> Result<()> {\n        use rmp_serde::decode::Deserializer;\n        use serde::de::Deserializer as _;\n        let mut de = Deserializer::new(buf);\n        let visitor = InsertToManager { lru: self };\n        de.deserialize_seq(visitor)\n            .or_err(InternalError, \"when deserializing LRU\")?;\n        Ok(())\n    }\n}\n\nstruct InsertToManager<'a> {\n    lru: &'a Manager,\n}\n\nimpl<'de> serde::de::Visitor<'de> for InsertToManager<'_> {\n    type Value = ();\n\n    fn expecting(&self, formatter: &mut std::fmt::Formatter) -> std::fmt::Result {\n        formatter.write_str(\"array of lru nodes\")\n    }\n\n    fn visit_seq<A>(self, mut seq: A) -> Result<Self::Value, A::Error>\n    where\n        A: SeqAccess<'de>,\n    {\n        while let Some(node) = seq.next_element::<Node>()? {\n            let key = u64key(&node.key);\n            self.lru.insert(key, node.key, node.size, true); // insert in the back\n        }\n        Ok(())\n    }\n}\n\n#[inline]\nfn u64key(key: &CompactCacheKey) -> u64 {\n    let mut hasher = DefaultHasher::new();\n    key.hash(&mut hasher);\n    hasher.finish()\n}\n\nconst FILE_NAME: &str = \"simple_lru.data\";\n\n#[async_trait]\nimpl EvictionManager for Manager {\n    fn total_size(&self) -> usize {\n        self.used.load(Ordering::Relaxed)\n    }\n    fn total_items(&self) -> usize {\n        self.items.load(Ordering::Relaxed)\n    }\n    fn evicted_size(&self) -> usize {\n        self.evicted_size.load(Ordering::Relaxed)\n    }\n    fn evicted_items(&self) -> usize {\n        self.evicted_items.load(Ordering::Relaxed)\n    }\n\n    fn admit(\n        &self,\n        item: CompactCacheKey,\n        size: usize,\n        _fresh_until: SystemTime,\n    ) -> Vec<CompactCacheKey> {\n        let key = u64key(&item);\n        self.insert(key, item, size, false);\n        self.evict()\n    }\n\n    fn increment_weight(\n        &self,\n        item: &CompactCacheKey,\n        delta: usize,\n        _max_weight: Option<usize>,\n    ) -> Vec<CompactCacheKey> {\n        let key = u64key(item);\n        self.increase_weight(key, delta);\n        self.evict()\n    }\n\n    fn remove(&self, item: &CompactCacheKey) {\n        let key = u64key(item);\n        let node = self.lru.write().pop(&key);\n        if let Some(n) = node {\n            self.used.fetch_sub(n.size, Ordering::Relaxed);\n            self.items.fetch_sub(1, Ordering::Relaxed);\n        }\n    }\n\n    fn access(&self, item: &CompactCacheKey, size: usize, _fresh_until: SystemTime) -> bool {\n        let key = u64key(item);\n        if self.lru.write().get(&key).is_none() {\n            self.insert(key, item.clone(), size, false);\n            false\n        } else {\n            true\n        }\n    }\n\n    fn peek(&self, item: &CompactCacheKey) -> bool {\n        let key = u64key(item);\n        self.lru.read().peek(&key).is_some()\n    }\n\n    async fn save(&self, dir_path: &str) -> Result<()> {\n        let data = self.serialize()?;\n        let dir_str = dir_path.to_owned();\n        tokio::task::spawn_blocking(move || {\n            let dir_path = Path::new(&dir_str);\n            std::fs::create_dir_all(dir_path)\n                .or_err_with(InternalError, || format!(\"fail to create {dir_str}\"))?;\n\n            let final_file_path = dir_path.join(FILE_NAME);\n            // create a temporary filename using a randomized u32 hash to minimize the chance of multiple writers writing to the same tmp file\n            let random_suffix: u32 = rand::thread_rng().gen();\n            let temp_file_path = dir_path.join(format!(\"{}.{:08x}.tmp\", FILE_NAME, random_suffix));\n            let mut file = File::create(&temp_file_path).or_err_with(InternalError, || {\n                format!(\"fail to create temporary file {}\", temp_file_path.display())\n            })?;\n            file.write_all(&data).or_err_with(InternalError, || {\n                format!(\"fail to write to {}\", temp_file_path.display())\n            })?;\n            file.flush().or_err_with(InternalError, || {\n                format!(\"fail to flush temp file {}\", temp_file_path.display())\n            })?;\n            std::fs::rename(&temp_file_path, &final_file_path).or_err_with(InternalError, || {\n                format!(\n                    \"fail to rename temporary file {} to {}\",\n                    temp_file_path.display(),\n                    final_file_path.display()\n                )\n            })\n        })\n        .await\n        .or_err(InternalError, \"async blocking IO failure\")?\n    }\n\n    async fn load(&self, dir_path: &str) -> Result<()> {\n        let dir_path = dir_path.to_owned();\n        let data = tokio::task::spawn_blocking(move || {\n            let file_path = Path::new(&dir_path).join(FILE_NAME);\n            let mut file = File::open(file_path.clone()).or_err_with(InternalError, || {\n                format!(\"fail to open {}\", file_path.display())\n            })?;\n            let mut buffer = Vec::with_capacity(8192);\n            file.read_to_end(&mut buffer)\n                .or_err(InternalError, \"fail to read from {file_path}\")?;\n            Ok::<Vec<u8>, BError>(buffer)\n        })\n        .await\n        .or_err(InternalError, \"async blocking IO failure\")??;\n        self.deserialize(&data)\n    }\n}\n\n#[cfg(test)]\nmod test {\n    use super::*;\n    use crate::CacheKey;\n\n    #[test]\n    fn test_admission() {\n        let lru = Manager::new(4);\n        let key1 = CacheKey::new(\"\", \"a\", \"1\").to_compact();\n        let until = SystemTime::now(); // unused value as a placeholder\n        let v = lru.admit(key1.clone(), 1, until);\n        assert_eq!(v.len(), 0);\n        let key2 = CacheKey::new(\"\", \"b\", \"1\").to_compact();\n        let v = lru.admit(key2.clone(), 2, until);\n        assert_eq!(v.len(), 0);\n        let key3 = CacheKey::new(\"\", \"c\", \"1\").to_compact();\n        let v = lru.admit(key3, 1, until);\n        assert_eq!(v.len(), 0);\n\n        // lru si full (4) now\n\n        let key4 = CacheKey::new(\"\", \"d\", \"1\").to_compact();\n        let v = lru.admit(key4, 2, until);\n        // need to reduce used by at least 2, both key1 and key2 are evicted to make room for 3\n        assert_eq!(v.len(), 2);\n        assert_eq!(v[0], key1);\n        assert_eq!(v[1], key2);\n    }\n\n    #[test]\n    fn test_access() {\n        let lru = Manager::new(4);\n        let key1 = CacheKey::new(\"\", \"a\", \"1\").to_compact();\n        let until = SystemTime::now(); // unused value as a placeholder\n        let v = lru.admit(key1.clone(), 1, until);\n        assert_eq!(v.len(), 0);\n        let key2 = CacheKey::new(\"\", \"b\", \"1\").to_compact();\n        let v = lru.admit(key2.clone(), 2, until);\n        assert_eq!(v.len(), 0);\n        let key3 = CacheKey::new(\"\", \"c\", \"1\").to_compact();\n        let v = lru.admit(key3, 1, until);\n        assert_eq!(v.len(), 0);\n\n        // lru is full (4) now\n        // make key1 most recently used\n        lru.access(&key1, 1, until);\n        assert_eq!(v.len(), 0);\n\n        let key4 = CacheKey::new(\"\", \"d\", \"1\").to_compact();\n        let v = lru.admit(key4, 2, until);\n        assert_eq!(v.len(), 1);\n        assert_eq!(v[0], key2);\n    }\n\n    #[test]\n    fn test_remove() {\n        let lru = Manager::new(4);\n        let key1 = CacheKey::new(\"\", \"a\", \"1\").to_compact();\n        let until = SystemTime::now(); // unused value as a placeholder\n        let v = lru.admit(key1.clone(), 1, until);\n        assert_eq!(v.len(), 0);\n        let key2 = CacheKey::new(\"\", \"b\", \"1\").to_compact();\n        let v = lru.admit(key2.clone(), 2, until);\n        assert_eq!(v.len(), 0);\n        let key3 = CacheKey::new(\"\", \"c\", \"1\").to_compact();\n        let v = lru.admit(key3, 1, until);\n        assert_eq!(v.len(), 0);\n\n        // lru is full (4) now\n        // remove key1\n        lru.remove(&key1);\n\n        // key2 is the least recently used one now\n        let key4 = CacheKey::new(\"\", \"d\", \"1\").to_compact();\n        let v = lru.admit(key4, 2, until);\n        assert_eq!(v.len(), 1);\n        assert_eq!(v[0], key2);\n    }\n\n    #[test]\n    fn test_access_add() {\n        let lru = Manager::new(4);\n        let until = SystemTime::now(); // unused value as a placeholder\n\n        let key1 = CacheKey::new(\"\", \"a\", \"1\").to_compact();\n        lru.access(&key1, 1, until);\n        let key2 = CacheKey::new(\"\", \"b\", \"1\").to_compact();\n        lru.access(&key2, 2, until);\n        let key3 = CacheKey::new(\"\", \"c\", \"1\").to_compact();\n        lru.access(&key3, 2, until);\n\n        let key4 = CacheKey::new(\"\", \"d\", \"1\").to_compact();\n        let v = lru.admit(key4, 2, until);\n        // need to reduce used by at least 2, both key1 and key2 are evicted to make room for 3\n        assert_eq!(v.len(), 2);\n        assert_eq!(v[0], key1);\n        assert_eq!(v[1], key2);\n    }\n\n    #[test]\n    fn test_admit_update() {\n        let lru = Manager::new(4);\n        let key1 = CacheKey::new(\"\", \"a\", \"1\").to_compact();\n        let until = SystemTime::now(); // unused value as a placeholder\n        let v = lru.admit(key1.clone(), 1, until);\n        assert_eq!(v.len(), 0);\n        let key2 = CacheKey::new(\"\", \"b\", \"1\").to_compact();\n        let v = lru.admit(key2.clone(), 2, until);\n        assert_eq!(v.len(), 0);\n        let key3 = CacheKey::new(\"\", \"c\", \"1\").to_compact();\n        let v = lru.admit(key3, 1, until);\n        assert_eq!(v.len(), 0);\n\n        // lru is full (4) now\n        // update key2 to reduce its size by 1\n        let v = lru.admit(key2, 1, until);\n        assert_eq!(v.len(), 0);\n\n        // lru is not full anymore\n        let key4 = CacheKey::new(\"\", \"d\", \"1\").to_compact();\n        let v = lru.admit(key4.clone(), 1, until);\n        assert_eq!(v.len(), 0);\n\n        // make key4 larger\n        let v = lru.admit(key4, 2, until);\n        // need to evict now\n        assert_eq!(v.len(), 1);\n        assert_eq!(v[0], key1);\n    }\n\n    #[test]\n    fn test_serde() {\n        let lru = Manager::new(4);\n        let key1 = CacheKey::new(\"\", \"a\", \"1\").to_compact();\n        let until = SystemTime::now(); // unused value as a placeholder\n        let v = lru.admit(key1.clone(), 1, until);\n        assert_eq!(v.len(), 0);\n        let key2 = CacheKey::new(\"\", \"b\", \"1\").to_compact();\n        let v = lru.admit(key2.clone(), 2, until);\n        assert_eq!(v.len(), 0);\n        let key3 = CacheKey::new(\"\", \"c\", \"1\").to_compact();\n        let v = lru.admit(key3, 1, until);\n        assert_eq!(v.len(), 0);\n\n        // lru is full (4) now\n        // make key1 most recently used\n        lru.access(&key1, 1, until);\n        assert_eq!(v.len(), 0);\n\n        // load lru2 with lru's data\n        let ser = lru.serialize().unwrap();\n        let lru2 = Manager::new(4);\n        lru2.deserialize(&ser).unwrap();\n\n        let key4 = CacheKey::new(\"\", \"d\", \"1\").to_compact();\n        let v = lru2.admit(key4, 2, until);\n        assert_eq!(v.len(), 1);\n        assert_eq!(v[0], key2);\n    }\n\n    #[tokio::test]\n    async fn test_save_to_disk() {\n        let lru = Manager::new(4);\n        let key1 = CacheKey::new(\"\", \"a\", \"1\").to_compact();\n        let until = SystemTime::now(); // unused value as a placeholder\n        let v = lru.admit(key1.clone(), 1, until);\n        assert_eq!(v.len(), 0);\n        let key2 = CacheKey::new(\"\", \"b\", \"1\").to_compact();\n        let v = lru.admit(key2.clone(), 2, until);\n        assert_eq!(v.len(), 0);\n        let key3 = CacheKey::new(\"\", \"c\", \"1\").to_compact();\n        let v = lru.admit(key3, 1, until);\n        assert_eq!(v.len(), 0);\n\n        // lru is full (4) now\n        // make key1 most recently used\n        lru.access(&key1, 1, until);\n        assert_eq!(v.len(), 0);\n\n        // load lru2 with lru's data\n        lru.save(\"/tmp/test_simple_lru_save\").await.unwrap();\n        let lru2 = Manager::new(4);\n        lru2.load(\"/tmp/test_simple_lru_save\").await.unwrap();\n\n        let key4 = CacheKey::new(\"\", \"d\", \"1\").to_compact();\n        let v = lru2.admit(key4, 2, until);\n        assert_eq!(v.len(), 1);\n        assert_eq!(v[0], key2);\n    }\n\n    #[test]\n    fn test_watermark_eviction() {\n        const SIZE_LIMIT: usize = usize::MAX / 2;\n        let lru = Manager::new_with_watermark(SIZE_LIMIT, Some(4));\n        let until = SystemTime::now();\n\n        // admit 6 items of size 1\n        for name in [\"a\", \"b\", \"c\", \"d\", \"e\", \"f\"] {\n            let key = CacheKey::new(\"\", name, \"1\").to_compact();\n            let _ = lru.admit(key, 1, until);\n        }\n\n        // test items were evicted due to watermark\n        assert_eq!(lru.total_items(), 4);\n        assert_eq!(lru.evicted_items(), 2);\n        assert_eq!(lru.evicted_size(), 2);\n        assert!(lru.total_size() <= SIZE_LIMIT);\n    }\n}\n"
  },
  {
    "path": "pingora-cache/src/filters.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! Utility functions to help process HTTP headers for caching\n\nuse super::*;\nuse crate::cache_control::{CacheControl, Cacheable, InterpretCacheControl};\nuse crate::RespCacheable::*;\n\nuse cache_control::DELTA_SECONDS_OVERFLOW_VALUE;\nuse http::{header, HeaderValue};\nuse httpdate::HttpDate;\nuse log::debug;\nuse pingora_http::RequestHeader;\n\n/// Decide if the request can be cacheable\npub fn request_cacheable(req_header: &ReqHeader) -> bool {\n    // TODO: the check is incomplete\n    matches!(req_header.method, Method::GET | Method::HEAD)\n}\n\n/// Decide if the response is cacheable.\n///\n/// `cache_control` is the parsed [CacheControl] from the response header. It is a standalone\n/// argument so that caller has the flexibility to choose to use, change or ignore it.\npub fn resp_cacheable(\n    cache_control: Option<&CacheControl>,\n    mut resp_header: ResponseHeader,\n    authorization_present: bool,\n    defaults: &CacheMetaDefaults,\n) -> RespCacheable {\n    let now = SystemTime::now();\n    let expire_time = calculate_fresh_until(\n        now,\n        cache_control,\n        &resp_header,\n        authorization_present,\n        defaults,\n    );\n    if let Some(fresh_until) = expire_time {\n        let (stale_while_revalidate_duration, stale_if_error_duration) =\n            calculate_serve_stale_durations(cache_control, defaults);\n\n        if let Some(cc) = cache_control {\n            cc.strip_private_headers(&mut resp_header);\n        }\n        return Cacheable(CacheMeta::new(\n            fresh_until,\n            now,\n            stale_while_revalidate_duration,\n            stale_if_error_duration,\n            resp_header,\n        ));\n    }\n    Uncacheable(NoCacheReason::OriginNotCache)\n}\n\n/// Calculate the [SystemTime] at which the asset expires\n///\n/// Return None when not cacheable.\npub fn calculate_fresh_until(\n    now: SystemTime,\n    cache_control: Option<&CacheControl>,\n    resp_header: &RespHeader,\n    authorization_present: bool,\n    defaults: &CacheMetaDefaults,\n) -> Option<SystemTime> {\n    fn freshness_ttl_to_time(now: SystemTime, fresh: Duration) -> Option<SystemTime> {\n        if fresh.is_zero() {\n            // ensure that the response is treated as stale\n            now.checked_sub(Duration::from_secs(1))\n        } else {\n            now.checked_add(fresh)\n        }\n    }\n\n    // A request with Authorization is normally not cacheable, unless Cache-Control allows it\n    if authorization_present {\n        let uncacheable = cache_control\n            .as_ref()\n            .is_none_or(|cc| !cc.allow_caching_authorized_req());\n        if uncacheable {\n            return None;\n        }\n    }\n\n    let uncacheable = cache_control\n        .as_ref()\n        .is_some_and(|cc| cc.is_cacheable() == Cacheable::No);\n    if uncacheable {\n        return None;\n    }\n\n    // For TTL check cache-control first, then expires header, then defaults\n    cache_control\n        .and_then(|cc| {\n            cc.fresh_duration()\n                .and_then(|ttl| freshness_ttl_to_time(now, ttl))\n        })\n        .or_else(|| calculate_expires_header_time(resp_header))\n        .or_else(|| {\n            defaults\n                .fresh_sec(resp_header.status)\n                .and_then(|ttl| freshness_ttl_to_time(now, ttl))\n        })\n}\n\n/// Calculate the expire time from the `Expires` header only\npub fn calculate_expires_header_time(resp_header: &RespHeader) -> Option<SystemTime> {\n    // according to RFC 7234:\n    // https://datatracker.ietf.org/doc/html/rfc7234#section-4.2.1\n    // - treat multiple expires headers as invalid\n    // https://datatracker.ietf.org/doc/html/rfc7234#section-5.3\n    // - \"MUST interpret invalid date formats... as representing a time in the past\"\n    fn parse_expires_value(expires_value: &HeaderValue) -> Option<SystemTime> {\n        let expires = expires_value.to_str().ok()?;\n        Some(SystemTime::from(\n            expires\n                .parse::<HttpDate>()\n                .map_err(|e| debug!(\"Invalid HttpDate in Expires: {}, error: {}\", expires, e))\n                .ok()?,\n        ))\n    }\n\n    let mut expires_iter = resp_header.headers.get_all(\"expires\").iter();\n    let expires_header = expires_iter.next();\n    if expires_header.is_none() || expires_iter.next().is_some() {\n        return None;\n    }\n    parse_expires_value(expires_header.unwrap()).or(Some(SystemTime::UNIX_EPOCH))\n}\n\n/// Calculates stale-while-revalidate and stale-if-error seconds from Cache-Control or the [CacheMetaDefaults].\npub fn calculate_serve_stale_durations(\n    cache_control: Option<&impl InterpretCacheControl>,\n    defaults: &CacheMetaDefaults,\n) -> (u32, u32) {\n    let serve_stale_while_revalidate = cache_control\n        .and_then(|cc| cc.serve_stale_while_revalidate_duration())\n        .unwrap_or_else(|| Duration::from_secs(defaults.serve_stale_while_revalidate_sec() as u64));\n    let serve_stale_if_error = cache_control\n        .and_then(|cc| cc.serve_stale_if_error_duration())\n        .unwrap_or_else(|| Duration::from_secs(defaults.serve_stale_if_error_sec() as u64));\n    (\n        serve_stale_while_revalidate\n            .as_secs()\n            .try_into()\n            .unwrap_or(DELTA_SECONDS_OVERFLOW_VALUE),\n        serve_stale_if_error\n            .as_secs()\n            .try_into()\n            .unwrap_or(DELTA_SECONDS_OVERFLOW_VALUE),\n    )\n}\n\n/// Filters to run when sending requests to upstream\npub mod upstream {\n    use super::*;\n\n    /// Adjust the request header for cacheable requests\n    ///\n    /// This filter does the following in order to fetch the entire response to cache\n    /// - Convert HEAD to GET\n    /// - `If-*` headers are removed\n    /// - `Range` header is removed\n    ///\n    /// When `meta` is set, this function will inject `If-modified-since` according to the `Last-Modified` header\n    /// and inject `If-none-match` according to `Etag` header\n    pub fn request_filter(req: &mut RequestHeader, meta: Option<&CacheMeta>) {\n        // change HEAD to GET, HEAD itself is not semantically cacheable\n        if req.method == Method::HEAD {\n            req.set_method(Method::GET);\n        }\n\n        // remove downstream precondition headers https://datatracker.ietf.org/doc/html/rfc7232#section-3\n        // we'd like to cache the 200 not the 304\n        req.remove_header(&header::IF_MATCH);\n        req.remove_header(&header::IF_NONE_MATCH);\n        req.remove_header(&header::IF_MODIFIED_SINCE);\n        req.remove_header(&header::IF_UNMODIFIED_SINCE);\n        // see below range header\n        req.remove_header(&header::IF_RANGE);\n\n        // remove downstream range header as we'd like to cache the entire response (this might change in the future)\n        req.remove_header(&header::RANGE);\n\n        // we have a presumably staled response already, add precondition headers for revalidation\n        if let Some(m) = meta {\n            // rfc7232: \"SHOULD send both validators in cache validation\" but\n            // there have been weird cases that an origin has matching etag but not Last-Modified\n            if let Some(since) = m.headers().get(&header::LAST_MODIFIED) {\n                req.insert_header(header::IF_MODIFIED_SINCE, since).unwrap();\n            }\n            if let Some(etag) = m.headers().get(&header::ETAG) {\n                req.insert_header(header::IF_NONE_MATCH, etag).unwrap();\n            }\n        }\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use super::*;\n    use crate::RespCacheable::Cacheable;\n    use http::header::{HeaderName, CACHE_CONTROL, EXPIRES, SET_COOKIE};\n    use http::StatusCode;\n    use httpdate::fmt_http_date;\n\n    fn init_log() {\n        let _ = env_logger::builder().is_test(true).try_init();\n    }\n\n    const DEFAULTS: CacheMetaDefaults = CacheMetaDefaults::new(\n        |status| {\n            match status {\n                StatusCode::OK => Some(10),\n                StatusCode::NOT_FOUND => Some(5),\n                StatusCode::PARTIAL_CONTENT => None,\n                _ => Some(1),\n            }\n            .map(Duration::from_secs)\n        },\n        0,\n        DELTA_SECONDS_OVERFLOW_VALUE, /* \"infinite\" stale-if-error */\n    );\n\n    // Cache nothing, by default\n    const BYPASS_CACHE_DEFAULTS: CacheMetaDefaults = CacheMetaDefaults::new(|_| None, 0, 0);\n\n    fn build_response(status: u16, headers: &[(HeaderName, &str)]) -> ResponseHeader {\n        let mut header = ResponseHeader::build(status, Some(headers.len())).unwrap();\n        for (k, v) in headers {\n            header.append_header(k.to_string(), *v).unwrap();\n        }\n        header\n    }\n\n    fn resp_cacheable_wrapper(\n        resp: ResponseHeader,\n        defaults: &CacheMetaDefaults,\n        authorization_present: bool,\n    ) -> Option<CacheMeta> {\n        if let Cacheable(meta) = resp_cacheable(\n            CacheControl::from_resp_headers(&resp).as_ref(),\n            resp,\n            authorization_present,\n            defaults,\n        ) {\n            Some(meta)\n        } else {\n            None\n        }\n    }\n\n    #[test]\n    fn test_resp_cacheable() {\n        let meta = resp_cacheable_wrapper(\n            build_response(200, &[(CACHE_CONTROL, \"max-age=12345\")]),\n            &DEFAULTS,\n            false,\n        );\n\n        let meta = meta.unwrap();\n        assert!(meta.is_fresh(SystemTime::now()));\n        assert!(meta.is_fresh(\n            SystemTime::now()\n                .checked_add(Duration::from_secs(12))\n                .unwrap()\n        ),);\n        assert!(!meta.is_fresh(\n            SystemTime::now()\n                .checked_add(Duration::from_secs(12346))\n                .unwrap()\n        ));\n    }\n\n    #[test]\n    fn test_resp_uncacheable_directives() {\n        let meta = resp_cacheable_wrapper(\n            build_response(200, &[(CACHE_CONTROL, \"private, max-age=12345\")]),\n            &DEFAULTS,\n            false,\n        );\n        assert!(meta.is_none());\n\n        let meta = resp_cacheable_wrapper(\n            build_response(200, &[(CACHE_CONTROL, \"no-store, max-age=12345\")]),\n            &DEFAULTS,\n            false,\n        );\n        assert!(meta.is_none());\n    }\n\n    #[test]\n    fn test_resp_cache_authorization() {\n        let meta = resp_cacheable_wrapper(build_response(200, &[]), &DEFAULTS, true);\n        assert!(meta.is_none());\n\n        let meta = resp_cacheable_wrapper(\n            build_response(200, &[(CACHE_CONTROL, \"max-age=10\")]),\n            &DEFAULTS,\n            true,\n        );\n        assert!(meta.is_none());\n\n        let meta = resp_cacheable_wrapper(\n            build_response(200, &[(CACHE_CONTROL, \"s-maxage=10\")]),\n            &DEFAULTS,\n            true,\n        );\n        assert!(meta.unwrap().is_fresh(SystemTime::now()));\n\n        let meta = resp_cacheable_wrapper(\n            build_response(200, &[(CACHE_CONTROL, \"public, max-age=10\")]),\n            &DEFAULTS,\n            true,\n        );\n        assert!(meta.unwrap().is_fresh(SystemTime::now()));\n\n        let meta = resp_cacheable_wrapper(\n            build_response(200, &[(CACHE_CONTROL, \"must-revalidate\")]),\n            &DEFAULTS,\n            true,\n        );\n        assert!(meta.unwrap().is_fresh(SystemTime::now()));\n    }\n\n    #[test]\n    fn test_resp_zero_max_age() {\n        let meta = resp_cacheable_wrapper(\n            build_response(200, &[(CACHE_CONTROL, \"max-age=0, public\")]),\n            &DEFAULTS,\n            false,\n        );\n\n        // cacheable, but needs revalidation\n        assert!(!meta.unwrap().is_fresh(SystemTime::now()));\n    }\n\n    #[test]\n    fn test_resp_expires() {\n        let five_sec_time = SystemTime::now()\n            .checked_add(Duration::from_secs(5))\n            .unwrap();\n\n        // future expires is cacheable\n        let meta = resp_cacheable_wrapper(\n            build_response(200, &[(EXPIRES, &fmt_http_date(five_sec_time))]),\n            &DEFAULTS,\n            false,\n        );\n\n        let meta = meta.unwrap();\n        assert!(meta.is_fresh(SystemTime::now()));\n        assert!(!meta.is_fresh(\n            SystemTime::now()\n                .checked_add(Duration::from_secs(6))\n                .unwrap()\n        ));\n\n        // even on default uncacheable statuses\n        let meta = resp_cacheable_wrapper(\n            build_response(206, &[(EXPIRES, &fmt_http_date(five_sec_time))]),\n            &DEFAULTS,\n            false,\n        );\n        assert!(meta.is_some());\n    }\n\n    #[test]\n    fn test_resp_past_expires() {\n        // cacheable, but expired\n        let meta = resp_cacheable_wrapper(\n            build_response(200, &[(EXPIRES, \"Fri, 15 May 2015 15:34:21 GMT\")]),\n            &BYPASS_CACHE_DEFAULTS,\n            false,\n        );\n        assert!(!meta.unwrap().is_fresh(SystemTime::now()));\n    }\n\n    #[test]\n    fn test_resp_nonstandard_expires() {\n        // init log to allow inspecting warnings\n        init_log();\n\n        // invalid cases, according to parser\n        // (but should be stale according to RFC)\n        let meta = resp_cacheable_wrapper(\n            build_response(200, &[(EXPIRES, \"Mon, 13 Feb 0002 12:00:00 GMT\")]),\n            &BYPASS_CACHE_DEFAULTS,\n            false,\n        );\n        assert!(!meta.unwrap().is_fresh(SystemTime::now()));\n\n        let meta = resp_cacheable_wrapper(\n            build_response(200, &[(EXPIRES, \"Fri, 01 Dec 99999 16:00:00 GMT\")]),\n            &BYPASS_CACHE_DEFAULTS,\n            false,\n        );\n        assert!(!meta.unwrap().is_fresh(SystemTime::now()));\n\n        let meta = resp_cacheable_wrapper(\n            build_response(200, &[(EXPIRES, \"0\")]),\n            &BYPASS_CACHE_DEFAULTS,\n            false,\n        );\n        assert!(!meta.unwrap().is_fresh(SystemTime::now()));\n    }\n\n    #[test]\n    fn test_resp_multiple_expires() {\n        let five_sec_time = SystemTime::now()\n            .checked_add(Duration::from_secs(5))\n            .unwrap();\n        let ten_sec_time = SystemTime::now()\n            .checked_add(Duration::from_secs(10))\n            .unwrap();\n\n        // multiple expires = uncacheable\n        let meta = resp_cacheable_wrapper(\n            build_response(\n                200,\n                &[\n                    (EXPIRES, &fmt_http_date(five_sec_time)),\n                    (EXPIRES, &fmt_http_date(ten_sec_time)),\n                ],\n            ),\n            &BYPASS_CACHE_DEFAULTS,\n            false,\n        );\n        assert!(meta.is_none());\n\n        // unless the default is cacheable\n        let meta = resp_cacheable_wrapper(\n            build_response(\n                200,\n                &[\n                    (EXPIRES, &fmt_http_date(five_sec_time)),\n                    (EXPIRES, &fmt_http_date(ten_sec_time)),\n                ],\n            ),\n            &DEFAULTS,\n            false,\n        );\n        assert!(meta.is_some());\n    }\n\n    #[test]\n    fn test_resp_cache_control_with_expires() {\n        let five_sec_time = SystemTime::now()\n            .checked_add(Duration::from_secs(5))\n            .unwrap();\n        // cache-control takes precedence over expires\n        let meta = resp_cacheable_wrapper(\n            build_response(\n                200,\n                &[\n                    (EXPIRES, &fmt_http_date(five_sec_time)),\n                    (CACHE_CONTROL, \"max-age=0\"),\n                ],\n            ),\n            &DEFAULTS,\n            false,\n        );\n        assert!(!meta.unwrap().is_fresh(SystemTime::now()));\n    }\n\n    #[test]\n    fn test_resp_stale_while_revalidate() {\n        // respect defaults\n        let meta = resp_cacheable_wrapper(\n            build_response(200, &[(CACHE_CONTROL, \"max-age=10\")]),\n            &DEFAULTS,\n            false,\n        );\n\n        let meta = meta.unwrap();\n        let eleven_sec_time = SystemTime::now()\n            .checked_add(Duration::from_secs(11))\n            .unwrap();\n        assert!(!meta.is_fresh(eleven_sec_time));\n        assert!(!meta.serve_stale_while_revalidate(SystemTime::now()));\n        assert!(!meta.serve_stale_while_revalidate(eleven_sec_time));\n\n        // override with stale-while-revalidate\n        let meta = resp_cacheable_wrapper(\n            build_response(\n                200,\n                &[(CACHE_CONTROL, \"max-age=10, stale-while-revalidate=5\")],\n            ),\n            &DEFAULTS,\n            false,\n        );\n\n        let meta = meta.unwrap();\n        let eleven_sec_time = SystemTime::now()\n            .checked_add(Duration::from_secs(11))\n            .unwrap();\n        let sixteen_sec_time = SystemTime::now()\n            .checked_add(Duration::from_secs(16))\n            .unwrap();\n        assert!(!meta.is_fresh(eleven_sec_time));\n        assert!(meta.serve_stale_while_revalidate(eleven_sec_time));\n        assert!(!meta.serve_stale_while_revalidate(sixteen_sec_time));\n    }\n\n    #[test]\n    fn test_resp_stale_if_error() {\n        // respect defaults\n        let meta = resp_cacheable_wrapper(\n            build_response(200, &[(CACHE_CONTROL, \"max-age=10\")]),\n            &DEFAULTS,\n            false,\n        );\n\n        let meta = meta.unwrap();\n        let fifty_years_time = SystemTime::now()\n            .checked_add(Duration::from_secs(86400 * 365 * 50))\n            .unwrap();\n        assert!(!meta.is_fresh(fifty_years_time));\n        assert!(meta.serve_stale_if_error(fifty_years_time));\n\n        // override with stale-if-error\n        let meta = resp_cacheable_wrapper(\n            build_response(\n                200,\n                &[(\n                    CACHE_CONTROL,\n                    \"max-age=10, stale-while-revalidate=5, stale-if-error=60\",\n                )],\n            ),\n            &DEFAULTS,\n            false,\n        );\n\n        let meta = meta.unwrap();\n        let eleven_sec_time = SystemTime::now()\n            .checked_add(Duration::from_secs(11))\n            .unwrap();\n        let seventy_sec_time = SystemTime::now()\n            .checked_add(Duration::from_secs(70))\n            .unwrap();\n        assert!(!meta.is_fresh(eleven_sec_time));\n        assert!(meta.serve_stale_if_error(SystemTime::now()));\n        assert!(meta.serve_stale_if_error(eleven_sec_time));\n        assert!(!meta.serve_stale_if_error(seventy_sec_time));\n\n        // never serve stale\n        let meta = resp_cacheable_wrapper(\n            build_response(200, &[(CACHE_CONTROL, \"max-age=10, stale-if-error=0\")]),\n            &DEFAULTS,\n            false,\n        );\n\n        let meta = meta.unwrap();\n        let eleven_sec_time = SystemTime::now()\n            .checked_add(Duration::from_secs(11))\n            .unwrap();\n        assert!(!meta.is_fresh(eleven_sec_time));\n        assert!(!meta.serve_stale_if_error(eleven_sec_time));\n    }\n\n    #[test]\n    fn test_resp_status_cache_defaults() {\n        // 200 response\n        let meta = resp_cacheable_wrapper(build_response(200, &[]), &DEFAULTS, false);\n        assert!(meta.is_some());\n\n        let meta = meta.unwrap();\n        assert!(meta.is_fresh(\n            SystemTime::now()\n                .checked_add(Duration::from_secs(9))\n                .unwrap()\n        ));\n        assert!(!meta.is_fresh(\n            SystemTime::now()\n                .checked_add(Duration::from_secs(11))\n                .unwrap()\n        ));\n\n        // 404 response, different ttl\n        let meta = resp_cacheable_wrapper(build_response(404, &[]), &DEFAULTS, false);\n        assert!(meta.is_some());\n\n        let meta = meta.unwrap();\n        assert!(meta.is_fresh(\n            SystemTime::now()\n                .checked_add(Duration::from_secs(4))\n                .unwrap()\n        ));\n        assert!(!meta.is_fresh(\n            SystemTime::now()\n                .checked_add(Duration::from_secs(6))\n                .unwrap()\n        ));\n\n        // 206 marked uncacheable (no cache TTL)\n        let meta = resp_cacheable_wrapper(build_response(206, &[]), &DEFAULTS, false);\n        assert!(meta.is_none());\n\n        // default uncacheable status with explicit Cache-Control is cacheable\n        let meta = resp_cacheable_wrapper(\n            build_response(206, &[(CACHE_CONTROL, \"public, max-age=10\")]),\n            &DEFAULTS,\n            false,\n        );\n        assert!(meta.is_some());\n\n        let meta = meta.unwrap();\n        assert!(meta.is_fresh(\n            SystemTime::now()\n                .checked_add(Duration::from_secs(9))\n                .unwrap()\n        ));\n        assert!(!meta.is_fresh(\n            SystemTime::now()\n                .checked_add(Duration::from_secs(11))\n                .unwrap()\n        ));\n\n        // 416 matches any status\n        let meta = resp_cacheable_wrapper(build_response(416, &[]), &DEFAULTS, false);\n        assert!(meta.is_some());\n\n        let meta = meta.unwrap();\n        assert!(meta.is_fresh(SystemTime::now()));\n        assert!(!meta.is_fresh(\n            SystemTime::now()\n                .checked_add(Duration::from_secs(2))\n                .unwrap()\n        ));\n    }\n\n    #[test]\n    fn test_resp_cache_no_cache_fields() {\n        // check #field-names are stripped from the cache header\n        let meta = resp_cacheable_wrapper(\n            build_response(\n                200,\n                &[\n                    (SET_COOKIE, \"my-cookie\"),\n                    (CACHE_CONTROL, \"private=\\\"something\\\", max-age=10\"),\n                    (HeaderName::from_bytes(b\"Something\").unwrap(), \"foo\"),\n                ],\n            ),\n            &DEFAULTS,\n            false,\n        );\n        let meta = meta.unwrap();\n        assert!(meta.headers().contains_key(SET_COOKIE));\n        assert!(!meta.headers().contains_key(\"Something\"));\n\n        let meta = resp_cacheable_wrapper(\n            build_response(\n                200,\n                &[\n                    (SET_COOKIE, \"my-cookie\"),\n                    (\n                        CACHE_CONTROL,\n                        \"max-age=0, no-cache=\\\"meta1, SeT-Cookie ,meta2\\\"\",\n                    ),\n                    (HeaderName::from_bytes(b\"meta1\").unwrap(), \"foo\"),\n                ],\n            ),\n            &DEFAULTS,\n            false,\n        );\n        let meta = meta.unwrap();\n        assert!(!meta.headers().contains_key(SET_COOKIE));\n        assert!(!meta.headers().contains_key(\"meta1\"));\n    }\n}\n"
  },
  {
    "path": "pingora-cache/src/hashtable.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! Concurrent hash tables and LRUs\n\nuse lru::LruCache;\nuse parking_lot::{RwLock, RwLockReadGuard, RwLockWriteGuard};\nuse std::collections::HashMap;\n\n// There are probably off-the-shelf crates of this, DashMap?\n/// A hash table that shards to a constant number of tables to reduce lock contention\n#[derive(Debug)]\npub struct ConcurrentHashTable<V, const N: usize> {\n    tables: [RwLock<HashMap<u128, V>>; N],\n}\n\n#[inline]\nfn get_shard(key: u128, n_shards: usize) -> usize {\n    (key % n_shards as u128) as usize\n}\n\nimpl<V, const N: usize> ConcurrentHashTable<V, N>\nwhere\n    [RwLock<HashMap<u128, V>>; N]: Default,\n{\n    pub fn new() -> Self {\n        ConcurrentHashTable {\n            tables: Default::default(),\n        }\n    }\n    pub fn get(&self, key: u128) -> &RwLock<HashMap<u128, V>> {\n        &self.tables[get_shard(key, N)]\n    }\n\n    #[allow(dead_code)]\n    pub fn get_shard_at_idx(&self, idx: usize) -> Option<&RwLock<HashMap<u128, V>>> {\n        self.tables.get(idx)\n    }\n\n    #[allow(dead_code)]\n    pub fn read(&self, key: u128) -> RwLockReadGuard<'_, HashMap<u128, V>> {\n        self.get(key).read()\n    }\n\n    pub fn write(&self, key: u128) -> RwLockWriteGuard<'_, HashMap<u128, V>> {\n        self.get(key).write()\n    }\n\n    #[allow(dead_code)]\n    pub fn for_each<F>(&self, mut f: F)\n    where\n        F: FnMut(&u128, &V),\n    {\n        for shard in &self.tables {\n            let guard = shard.read();\n            for (key, value) in guard.iter() {\n                f(key, value);\n            }\n        }\n    }\n\n    // TODO: work out the lifetimes to provide get/set directly\n}\n\nimpl<V, const N: usize> Default for ConcurrentHashTable<V, N>\nwhere\n    [RwLock<HashMap<u128, V>>; N]: Default,\n{\n    fn default() -> Self {\n        Self::new()\n    }\n}\n\n#[doc(hidden)] // not need in public API\npub struct LruShard<V>(RwLock<LruCache<u128, V>>);\nimpl<V> Default for LruShard<V> {\n    fn default() -> Self {\n        // help satisfy default construction of arrays\n        LruShard(RwLock::new(LruCache::unbounded()))\n    }\n}\n\n/// Sharded concurrent data structure for LruCache\npub struct ConcurrentLruCache<V, const N: usize> {\n    lrus: [LruShard<V>; N],\n}\n\nimpl<V, const N: usize> ConcurrentLruCache<V, N>\nwhere\n    [LruShard<V>; N]: Default,\n{\n    pub fn new(shard_capacity: usize) -> Self {\n        use std::num::NonZeroUsize;\n        // safe, 1 != 0\n        const ONE: NonZeroUsize = NonZeroUsize::new(1).unwrap();\n        let mut cache = ConcurrentLruCache {\n            lrus: Default::default(),\n        };\n        for lru in &mut cache.lrus {\n            lru.0\n                .write()\n                .resize(shard_capacity.try_into().unwrap_or(ONE));\n        }\n        cache\n    }\n    pub fn get(&self, key: u128) -> &RwLock<LruCache<u128, V>> {\n        &self.lrus[get_shard(key, N)].0\n    }\n\n    #[allow(dead_code)]\n    pub fn read(&self, key: u128) -> RwLockReadGuard<'_, LruCache<u128, V>> {\n        self.get(key).read()\n    }\n\n    pub fn write(&self, key: u128) -> RwLockWriteGuard<'_, LruCache<u128, V>> {\n        self.get(key).write()\n    }\n\n    // TODO: work out the lifetimes to provide get/set directly\n}\n"
  },
  {
    "path": "pingora-cache/src/key.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! Cache key\n\nuse blake2::{Blake2b, Digest};\nuse http::Extensions;\nuse serde::{Deserialize, Serialize};\nuse std::fmt::{Display, Formatter, Result as FmtResult};\n\n// 16-byte / 128-bit key: large enough to avoid collision\nconst KEY_SIZE: usize = 16;\n\n/// An 128 bit hash binary\npub type HashBinary = [u8; KEY_SIZE];\n\nfn hex2str(hex: &[u8]) -> String {\n    use std::fmt::Write;\n    let mut s = String::with_capacity(KEY_SIZE * 2);\n    for c in hex {\n        write!(s, \"{:02x}\", c).unwrap(); // safe, just dump hex to string\n    }\n    s\n}\n\n/// Decode the hex str into [HashBinary].\n///\n/// Return `None` when the decode fails or the input is not exact 32 (to decode to 16 bytes).\npub fn str2hex(s: &str) -> Option<HashBinary> {\n    if s.len() != KEY_SIZE * 2 {\n        return None;\n    }\n    let mut output = [0; KEY_SIZE];\n    // no need to bubble the error, it should be obvious why the decode fails\n    hex::decode_to_slice(s.as_bytes(), &mut output).ok()?;\n    Some(output)\n}\n\n/// The trait for cache key\npub trait CacheHashKey {\n    /// Return the hash of the cache key\n    fn primary_bin(&self) -> HashBinary;\n\n    /// Return the variance hash of the cache key.\n    ///\n    /// `None` if no variance.\n    fn variance_bin(&self) -> Option<HashBinary>;\n\n    /// Return the hash including both primary and variance keys\n    fn combined_bin(&self) -> HashBinary {\n        let key = self.primary_bin();\n        if let Some(v) = self.variance_bin() {\n            let mut hasher = Blake2b128::new();\n            hasher.update(key);\n            hasher.update(v);\n            hasher.finalize().into()\n        } else {\n            // if there is no variance, combined_bin should return the same as primary_bin\n            key\n        }\n    }\n\n    /// An extra tag for identifying users\n    ///\n    /// For example, if the storage backend implements per user quota, this tag can be used.\n    fn user_tag(&self) -> &str;\n\n    /// The hex string of [Self::primary_bin()]\n    fn primary(&self) -> String {\n        hex2str(&self.primary_bin())\n    }\n\n    /// The hex string of [Self::variance_bin()]\n    fn variance(&self) -> Option<String> {\n        self.variance_bin().as_ref().map(|b| hex2str(&b[..]))\n    }\n\n    /// The hex string of [Self::combined_bin()]\n    fn combined(&self) -> String {\n        hex2str(&self.combined_bin())\n    }\n}\n\n/// General purpose cache key\n#[derive(Debug, Clone)]\npub struct CacheKey {\n    // Namespace and primary fields are essentially strings,\n    // except they allow invalid UTF-8 sequences.\n    // These fields should be able to be hashed.\n    namespace: Vec<u8>,\n    primary: Vec<u8>,\n    primary_bin_override: Option<HashBinary>,\n    variance: Option<HashBinary>,\n    /// An extra tag for identifying users\n    ///\n    /// For example, if the storage backend implements per user quota, this tag can be used.\n    pub user_tag: String,\n\n    /// Grab-bag for user-defined extensions. These will not be persisted to disk.\n    pub extensions: Extensions,\n}\n\nimpl CacheKey {\n    /// Set the value of the variance hash\n    pub fn set_variance_key(&mut self, key: HashBinary) {\n        self.variance = Some(key)\n    }\n\n    /// Get the value of the variance hash\n    pub fn get_variance_key(&self) -> Option<&HashBinary> {\n        self.variance.as_ref()\n    }\n\n    /// Removes the variance from this cache key\n    pub fn remove_variance_key(&mut self) {\n        self.variance = None\n    }\n\n    /// Override the primary key hash\n    pub fn set_primary_bin_override(&mut self, key: HashBinary) {\n        self.primary_bin_override = Some(key)\n    }\n\n    /// Try to get primary key as UTF-8 str, if valid\n    pub fn primary_key_str(&self) -> Option<&str> {\n        std::str::from_utf8(&self.primary).ok()\n    }\n\n    /// Try to get namespace key as UTF-8 str, if valid\n    pub fn namespace_str(&self) -> Option<&str> {\n        std::str::from_utf8(&self.namespace).ok()\n    }\n}\n\n/// Storage optimized cache key to keep in memory or in storage\n// 16 bytes + 8 bytes (+16 * u8) + user_tag.len() + 16 Bytes (Box<str>)\n#[derive(Debug, Deserialize, Serialize, Clone, Hash, PartialEq, Eq, PartialOrd, Ord)]\npub struct CompactCacheKey {\n    pub primary: HashBinary,\n    // save 8 bytes for non-variance but waste 8 bytes for variance vs, store flat 16 bytes\n    pub variance: Option<Box<HashBinary>>,\n    pub user_tag: Box<str>, // the len should be small to keep memory usage bounded\n}\n\nimpl Display for CompactCacheKey {\n    fn fmt(&self, f: &mut Formatter) -> FmtResult {\n        write!(f, \"{}\", hex2str(&self.primary))?;\n        if let Some(var) = &self.variance {\n            write!(f, \", variance: {}\", hex2str(var.as_ref()))?;\n        }\n        write!(f, \", user_tag: {}\", self.user_tag)\n    }\n}\n\nimpl CacheHashKey for CompactCacheKey {\n    fn primary_bin(&self) -> HashBinary {\n        self.primary\n    }\n\n    fn variance_bin(&self) -> Option<HashBinary> {\n        self.variance.as_ref().map(|s| *s.as_ref())\n    }\n\n    fn user_tag(&self) -> &str {\n        &self.user_tag\n    }\n}\n\n/*\n * We use blake2 hashing, which is faster and more secure, to replace md5.\n * We have not given too much thought on whether non-crypto hash can be safely\n * use because hashing performance is not critical.\n * Note: we should avoid hashes like ahash which does not have consistent output\n * across machines because it is designed purely for in memory hashtable\n*/\n\n// hash output: we use 128 bits (16 bytes) hash which will map to 32 bytes hex string\npub(crate) type Blake2b128 = Blake2b<blake2::digest::consts::U16>;\n\n/// helper function: hash str to u8\npub fn hash_u8(key: &str) -> u8 {\n    let mut hasher = Blake2b128::new();\n    hasher.update(key);\n    let raw = hasher.finalize();\n    raw[0]\n}\n\n/// helper function: hash key (String or Bytes) to [HashBinary]\npub fn hash_key<K: AsRef<[u8]>>(key: K) -> HashBinary {\n    let mut hasher = Blake2b128::new();\n    hasher.update(key.as_ref());\n    let raw = hasher.finalize();\n    raw.into()\n}\n\nimpl CacheKey {\n    fn primary_hasher(&self) -> Blake2b128 {\n        let mut hasher = Blake2b128::new();\n        hasher.update(&self.namespace);\n        hasher.update(&self.primary);\n        hasher\n    }\n\n    /// Create a new [CacheKey] from the given namespace, primary, and user_tag input.\n    ///\n    /// Both `namespace` and `primary` will be used for the primary hash\n    pub fn new<B1, B2, S>(namespace: B1, primary: B2, user_tag: S) -> Self\n    where\n        B1: Into<Vec<u8>>,\n        B2: Into<Vec<u8>>,\n        S: Into<String>,\n    {\n        CacheKey {\n            namespace: namespace.into(),\n            primary: primary.into(),\n            primary_bin_override: None,\n            variance: None,\n            user_tag: user_tag.into(),\n            extensions: Extensions::new(),\n        }\n    }\n\n    /// Return the namespace of this key\n    pub fn namespace(&self) -> &[u8] {\n        &self.namespace[..]\n    }\n\n    /// Return the primary key of this key\n    pub fn primary_key(&self) -> &[u8] {\n        &self.primary[..]\n    }\n\n    /// Convert this key to [CompactCacheKey].\n    pub fn to_compact(&self) -> CompactCacheKey {\n        let primary = self.primary_bin();\n        CompactCacheKey {\n            primary,\n            variance: self.variance_bin().map(Box::new),\n            user_tag: self.user_tag.clone().into_boxed_str(),\n        }\n    }\n}\n\nimpl CacheHashKey for CacheKey {\n    fn primary_bin(&self) -> HashBinary {\n        if let Some(primary_bin_override) = self.primary_bin_override {\n            primary_bin_override\n        } else {\n            self.primary_hasher().finalize().into()\n        }\n    }\n\n    fn variance_bin(&self) -> Option<HashBinary> {\n        self.variance\n    }\n\n    fn user_tag(&self) -> &str {\n        &self.user_tag\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use super::*;\n\n    #[test]\n    fn test_cache_key_hash() {\n        let key = CacheKey {\n            namespace: Vec::new(),\n            primary: b\"aa\".to_vec(),\n            primary_bin_override: None,\n            variance: None,\n            user_tag: \"1\".into(),\n            extensions: Extensions::new(),\n        };\n        let hash = key.primary();\n        assert_eq!(hash, \"ac10f2aef117729f8dad056b3059eb7e\");\n        assert!(key.variance().is_none());\n        assert_eq!(key.combined(), hash);\n        let compact = key.to_compact();\n        assert_eq!(compact.primary(), hash);\n        assert!(compact.variance().is_none());\n        assert_eq!(compact.combined(), hash);\n    }\n\n    #[test]\n    fn test_cache_key_hash_override() {\n        let mut key = CacheKey {\n            namespace: Vec::new(),\n            primary: b\"aa\".to_vec(),\n            primary_bin_override: str2hex(\"27c35e6e9373877f29e562464e46497e\"),\n            variance: None,\n            user_tag: \"1\".into(),\n            extensions: Extensions::new(),\n        };\n        let hash = key.primary();\n        assert_eq!(hash, \"27c35e6e9373877f29e562464e46497e\");\n        assert!(key.variance().is_none());\n        assert_eq!(key.combined(), hash);\n        let compact = key.to_compact();\n        assert_eq!(compact.primary(), hash);\n        assert!(compact.variance().is_none());\n        assert_eq!(compact.combined(), hash);\n\n        // make sure set_primary_bin_override overrides the primary key hash correctly\n        key.set_primary_bin_override(str2hex(\"004174d3e75a811a5b44c46b3856f3ee\").unwrap());\n        let hash = key.primary();\n        assert_eq!(hash, \"004174d3e75a811a5b44c46b3856f3ee\");\n        assert!(key.variance().is_none());\n        assert_eq!(key.combined(), hash);\n        let compact = key.to_compact();\n        assert_eq!(compact.primary(), hash);\n        assert!(compact.variance().is_none());\n        assert_eq!(compact.combined(), hash);\n    }\n\n    #[test]\n    fn test_cache_key_vary_hash() {\n        let key = CacheKey {\n            namespace: Vec::new(),\n            primary: b\"aa\".to_vec(),\n            primary_bin_override: None,\n            variance: Some([0u8; 16]),\n            user_tag: \"1\".into(),\n            extensions: Extensions::new(),\n        };\n        let hash = key.primary();\n        assert_eq!(hash, \"ac10f2aef117729f8dad056b3059eb7e\");\n        assert_eq!(key.variance().unwrap(), \"00000000000000000000000000000000\");\n        assert_eq!(key.combined(), \"004174d3e75a811a5b44c46b3856f3ee\");\n        let compact = key.to_compact();\n        assert_eq!(compact.primary(), \"ac10f2aef117729f8dad056b3059eb7e\");\n        assert_eq!(\n            compact.variance().unwrap(),\n            \"00000000000000000000000000000000\"\n        );\n        assert_eq!(compact.combined(), \"004174d3e75a811a5b44c46b3856f3ee\");\n    }\n\n    #[test]\n    fn test_cache_key_vary_hash_override() {\n        let key = CacheKey {\n            namespace: Vec::new(),\n            primary: b\"saaaad\".to_vec(),\n            primary_bin_override: str2hex(\"ac10f2aef117729f8dad056b3059eb7e\"),\n            variance: Some([0u8; 16]),\n            user_tag: \"1\".into(),\n            extensions: Extensions::new(),\n        };\n        let hash = key.primary();\n        assert_eq!(hash, \"ac10f2aef117729f8dad056b3059eb7e\");\n        assert_eq!(key.variance().unwrap(), \"00000000000000000000000000000000\");\n        assert_eq!(key.combined(), \"004174d3e75a811a5b44c46b3856f3ee\");\n        let compact = key.to_compact();\n        assert_eq!(compact.primary(), \"ac10f2aef117729f8dad056b3059eb7e\");\n        assert_eq!(\n            compact.variance().unwrap(),\n            \"00000000000000000000000000000000\"\n        );\n        assert_eq!(compact.combined(), \"004174d3e75a811a5b44c46b3856f3ee\");\n    }\n\n    #[test]\n    fn test_hex_str() {\n        let mut key = [0; KEY_SIZE];\n        for (i, v) in key.iter_mut().enumerate() {\n            // key: [0, 1, 2, .., 15]\n            *v = i as u8;\n        }\n        let hex_str = hex2str(&key);\n        let key2 = str2hex(&hex_str).unwrap();\n        for i in 0..KEY_SIZE {\n            assert_eq!(key[i], key2[i]);\n        }\n    }\n    #[test]\n    fn test_primary_key_str_valid_utf8() {\n        let valid_utf8_key = CacheKey {\n            namespace: Vec::new(),\n            primary: b\"/valid/path?query=1\".to_vec(),\n            primary_bin_override: None,\n            variance: None,\n            user_tag: \"1\".into(),\n            extensions: Extensions::new(),\n        };\n\n        assert_eq!(\n            valid_utf8_key.primary_key_str(),\n            Some(\"/valid/path?query=1\")\n        )\n    }\n\n    #[test]\n    fn test_primary_key_str_invalid_utf8() {\n        let invalid_utf8_key = CacheKey {\n            namespace: Vec::new(),\n            primary: vec![0x66, 0x6f, 0x6f, 0xff],\n            primary_bin_override: None,\n            variance: None,\n            user_tag: \"1\".into(),\n            extensions: Extensions::new(),\n        };\n\n        assert!(invalid_utf8_key.primary_key_str().is_none())\n    }\n}\n"
  },
  {
    "path": "pingora-cache/src/lib.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! The HTTP caching layer for proxies.\n\n#![allow(clippy::new_without_default)]\n\nuse cf_rustracing::tag::Tag;\nuse http::{method::Method, request::Parts as ReqHeader, response::Parts as RespHeader};\nuse key::{CacheHashKey, CompactCacheKey, HashBinary};\nuse lock::WritePermit;\nuse log::warn;\nuse pingora_error::Result;\nuse pingora_http::ResponseHeader;\nuse pingora_timeout::timeout;\nuse std::time::{Duration, Instant, SystemTime};\nuse storage::MissFinishType;\nuse strum::IntoStaticStr;\nuse trace::{CacheTraceCTX, Span};\n\npub mod cache_control;\npub mod eviction;\npub mod filters;\npub mod hashtable;\npub mod key;\npub mod lock;\npub mod max_file_size;\nmod memory;\npub mod meta;\npub mod predictor;\npub mod put;\npub mod storage;\npub mod trace;\nmod variance;\n\nuse crate::max_file_size::MaxFileSizeTracker;\npub use key::CacheKey;\nuse lock::{CacheKeyLockImpl, LockStatus, Locked};\npub use memory::MemCache;\npub use meta::{set_compression_dict_content, set_compression_dict_path};\npub use meta::{CacheMeta, CacheMetaDefaults};\npub use storage::{HitHandler, MissHandler, PurgeType, Storage};\npub use variance::VarianceBuilder;\n\npub mod prelude {}\n\n/// The state machine for http caching\n///\n/// This object is used to handle the state and transitions for HTTP caching through the life of a\n/// request.\npub struct HttpCache {\n    phase: CachePhase,\n    // Box the rest so that a disabled HttpCache struct is small\n    inner: Option<Box<HttpCacheInner>>,\n    digest: HttpCacheDigest,\n}\n\n/// This reflects the phase of HttpCache during the lifetime of a request\n#[derive(Clone, Copy, Debug, PartialEq, Eq)]\npub enum CachePhase {\n    /// Cache disabled, with reason (NeverEnabled if never explicitly used)\n    Disabled(NoCacheReason),\n    /// Cache enabled but nothing is set yet\n    Uninit,\n    /// Cache was enabled, the request decided not to use it\n    // HttpCache.inner_enabled is kept\n    Bypass,\n    /// Awaiting the cache key to be generated\n    CacheKey,\n    /// Cache hit\n    Hit,\n    /// No cached asset is found\n    Miss,\n    /// A staled (expired) asset is found\n    Stale,\n    /// A staled (expired) asset was found, but another request is revalidating it\n    StaleUpdating,\n    /// A staled (expired) asset was found, so a fresh one was fetched\n    Expired,\n    /// A staled (expired) asset was found, and it was revalidated to be fresh\n    Revalidated,\n    /// Revalidated, but deemed uncacheable, so we do not freshen it\n    RevalidatedNoCache(NoCacheReason),\n}\n\nimpl CachePhase {\n    /// Convert [CachePhase] as `str`, for logging and debugging.\n    pub fn as_str(&self) -> &'static str {\n        match self {\n            CachePhase::Disabled(_) => \"disabled\",\n            CachePhase::Uninit => \"uninitialized\",\n            CachePhase::Bypass => \"bypass\",\n            CachePhase::CacheKey => \"key\",\n            CachePhase::Hit => \"hit\",\n            CachePhase::Miss => \"miss\",\n            CachePhase::Stale => \"stale\",\n            CachePhase::StaleUpdating => \"stale-updating\",\n            CachePhase::Expired => \"expired\",\n            CachePhase::Revalidated => \"revalidated\",\n            CachePhase::RevalidatedNoCache(_) => \"revalidated-nocache\",\n        }\n    }\n}\n\n/// The possible reasons for not caching\n#[derive(Copy, Clone, Debug, PartialEq, Eq)]\npub enum NoCacheReason {\n    /// Caching is not enabled to begin with\n    NeverEnabled,\n    /// Origin directives indicated this was not cacheable\n    OriginNotCache,\n    /// Response size was larger than the cache's configured maximum asset size\n    ResponseTooLarge,\n    /// Disabling caching due to unknown body size and previously exceeding maximum asset size;\n    /// the asset is otherwise cacheable, but cache needs to confirm the final size of the asset\n    /// before it can mark it as cacheable again.\n    PredictedResponseTooLarge,\n    /// Due to internal caching storage error\n    StorageError,\n    /// Due to other types of internal issues\n    InternalError,\n    /// will be cacheable but skip cache admission now\n    ///\n    /// This happens when the cache predictor predicted that this request is not cacheable, but\n    /// the response turns out to be OK to cache. However, it might be too large to re-enable caching\n    /// for this request\n    Deferred,\n    /// Due to the proxy upstream filter declining the current request from going upstream\n    DeclinedToUpstream,\n    /// Due to the upstream being unreachable or otherwise erroring during proxying\n    UpstreamError,\n    /// The writer of the cache lock sees that the request is not cacheable (Could be OriginNotCache)\n    CacheLockGiveUp,\n    /// This request waited too long for the writer of the cache lock to finish, so this request will\n    /// fetch from the origin without caching\n    CacheLockTimeout,\n    /// Other custom defined reasons\n    Custom(&'static str),\n}\n\nimpl NoCacheReason {\n    /// Convert [NoCacheReason] as `str`, for logging and debugging.\n    pub fn as_str(&self) -> &'static str {\n        use NoCacheReason::*;\n        match self {\n            NeverEnabled => \"NeverEnabled\",\n            OriginNotCache => \"OriginNotCache\",\n            ResponseTooLarge => \"ResponseTooLarge\",\n            PredictedResponseTooLarge => \"PredictedResponseTooLarge\",\n            StorageError => \"StorageError\",\n            InternalError => \"InternalError\",\n            Deferred => \"Deferred\",\n            DeclinedToUpstream => \"DeclinedToUpstream\",\n            UpstreamError => \"UpstreamError\",\n            CacheLockGiveUp => \"CacheLockGiveUp\",\n            CacheLockTimeout => \"CacheLockTimeout\",\n            Custom(s) => s,\n        }\n    }\n}\n\n/// Information collected about the caching operation that will not be cleared\n#[derive(Debug, Default)]\npub struct HttpCacheDigest {\n    pub lock_duration: Option<Duration>,\n    // time spent in cache lookup and reading the header\n    pub lookup_duration: Option<Duration>,\n}\n\n/// Convenience function to add a duration to an optional duration\nfn add_duration_to_opt(target_opt: &mut Option<Duration>, to_add: Duration) {\n    *target_opt = Some(target_opt.map_or(to_add, |existing| existing + to_add));\n}\n\nimpl HttpCacheDigest {\n    fn add_lookup_duration(&mut self, extra_lookup_duration: Duration) {\n        add_duration_to_opt(&mut self.lookup_duration, extra_lookup_duration)\n    }\n\n    fn add_lock_duration(&mut self, extra_lock_duration: Duration) {\n        add_duration_to_opt(&mut self.lock_duration, extra_lock_duration)\n    }\n}\n\n/// Response cacheable decision\n///\n///\n#[derive(Debug)]\npub enum RespCacheable {\n    Cacheable(CacheMeta),\n    Uncacheable(NoCacheReason),\n}\n\nimpl RespCacheable {\n    /// Whether it is cacheable\n    #[inline]\n    pub fn is_cacheable(&self) -> bool {\n        matches!(*self, Self::Cacheable(_))\n    }\n\n    /// Unwrap [RespCacheable] to get the [CacheMeta] stored\n    /// # Panic\n    /// Panic when this object is not cacheable. Check [Self::is_cacheable()] first.\n    pub fn unwrap_meta(self) -> CacheMeta {\n        match self {\n            Self::Cacheable(meta) => meta,\n            Self::Uncacheable(_) => panic!(\"expected Cacheable value\"),\n        }\n    }\n}\n\n/// Indicators of which level of cache freshness logic to force apply to an asset.\n///\n/// For example, should an existing fresh asset be revalidated or re-retrieved altogether.\n#[derive(Debug, Clone, Copy, PartialEq, Eq)]\npub enum ForcedFreshness {\n    /// Indicates the asset should be considered stale and revalidated\n    ForceExpired,\n\n    /// Indicates the asset should be considered absent and treated like a miss\n    /// instead of a hit\n    ForceMiss,\n\n    /// Indicates the asset should be considered fresh despite possibly being stale\n    ForceFresh,\n}\n\n/// Freshness state of cache hit asset\n///\n///\n#[derive(Debug, Copy, Clone, IntoStaticStr, PartialEq, Eq)]\n#[strum(serialize_all = \"snake_case\")]\npub enum HitStatus {\n    /// The asset's freshness directives indicate it has expired\n    Expired,\n\n    /// The asset was marked as expired, and should be treated as stale\n    ForceExpired,\n\n    /// The asset was marked as absent, and should be treated as a miss\n    ForceMiss,\n\n    /// An error occurred while processing the asset, so it should be treated as\n    /// a miss\n    FailedHitFilter,\n\n    /// The asset is not expired\n    Fresh,\n\n    /// Asset exists but is expired, forced to be a hit\n    ForceFresh,\n}\n\nimpl HitStatus {\n    /// For displaying cache hit status\n    pub fn as_str(&self) -> &'static str {\n        self.into()\n    }\n\n    /// Whether cached asset can be served as fresh\n    pub fn is_fresh(&self) -> bool {\n        *self == HitStatus::Fresh || *self == HitStatus::ForceFresh\n    }\n\n    /// Check whether the hit status should be treated as a miss. A forced miss\n    /// is obviously treated as a miss. A hit-filter failure is treated as a\n    /// miss because we can't use the asset as an actual hit. If we treat it as\n    /// expired, we still might not be able to use it even if revalidation\n    /// succeeds.\n    pub fn is_treated_as_miss(self) -> bool {\n        matches!(self, HitStatus::ForceMiss | HitStatus::FailedHitFilter)\n    }\n}\n\npub struct LockCtx {\n    pub lock: Option<Locked>,\n    pub cache_lock: &'static CacheKeyLockImpl,\n    pub wait_timeout: Option<Duration>,\n}\n\n// Fields like storage handlers that are needed only when cache is enabled (or bypassing).\nstruct HttpCacheInnerEnabled {\n    pub meta: Option<CacheMeta>,\n    // when set, even if an asset exists, it would only be considered valid after this timestamp\n    pub valid_after: Option<SystemTime>,\n    pub miss_handler: Option<MissHandler>,\n    pub body_reader: Option<HitHandler>,\n    pub storage: &'static (dyn storage::Storage + Sync), // static for now\n    pub eviction: Option<&'static (dyn eviction::EvictionManager + Sync)>,\n    pub lock_ctx: Option<LockCtx>,\n    pub traces: trace::CacheTraceCTX,\n}\n\nstruct HttpCacheInner {\n    // Prefer adding fields to InnerEnabled if possible, these fields are released\n    // when cache is disabled.\n    // If fields are needed after cache disablement, add directly to Inner.\n    pub enabled_ctx: Option<Box<HttpCacheInnerEnabled>>,\n    pub key: Option<CacheKey>,\n    // when set, an asset will be rejected from the cache if it exceeds configured size in bytes\n    pub max_file_size_tracker: Option<MaxFileSizeTracker>,\n    pub predictor: Option<&'static (dyn predictor::CacheablePredictor + Sync)>,\n}\n\n#[derive(Debug, Default)]\n#[non_exhaustive]\npub struct CacheOptionOverrides {\n    pub wait_timeout: Option<Duration>,\n}\n\nimpl HttpCache {\n    /// Create a new [HttpCache].\n    ///\n    /// Caching is not enabled by default.\n    pub fn new() -> Self {\n        HttpCache {\n            phase: CachePhase::Disabled(NoCacheReason::NeverEnabled),\n            inner: None,\n            digest: HttpCacheDigest::default(),\n        }\n    }\n\n    /// Whether the cache is enabled\n    pub fn enabled(&self) -> bool {\n        !matches!(self.phase, CachePhase::Disabled(_) | CachePhase::Bypass)\n    }\n\n    /// Whether the cache is being bypassed\n    pub fn bypassing(&self) -> bool {\n        matches!(self.phase, CachePhase::Bypass)\n    }\n\n    /// Return the [CachePhase]\n    pub fn phase(&self) -> CachePhase {\n        self.phase\n    }\n\n    /// Whether anything was fetched from the upstream\n    ///\n    /// This essentially checks all possible [CachePhase] who need to contact the upstream server\n    pub fn upstream_used(&self) -> bool {\n        use CachePhase::*;\n        match self.phase {\n            Disabled(_) | Bypass | Miss | Expired | Revalidated | RevalidatedNoCache(_) => true,\n            Hit | Stale | StaleUpdating => false,\n            Uninit | CacheKey => false, // invalid states for this call, treat them as false to keep it simple\n        }\n    }\n\n    /// Check whether the backend storage is the type `T`.\n    pub fn storage_type_is<T: 'static>(&self) -> bool {\n        self.inner\n            .as_ref()\n            .and_then(|inner| {\n                inner\n                    .enabled_ctx\n                    .as_ref()\n                    .and_then(|ie| ie.storage.as_any().downcast_ref::<T>())\n            })\n            .is_some()\n    }\n\n    /// Release the cache lock if the current request is a cache writer.\n    ///\n    /// Generally callers should prefer using `disable` when a cache lock should be released\n    /// due to an error to clear all cache context. This function is for releasing the cache lock\n    /// while still keeping the cache around for reading, e.g. when serving stale.\n    pub fn release_write_lock(&mut self, reason: NoCacheReason) {\n        use NoCacheReason::*;\n        if let Some(inner) = self.inner.as_mut() {\n            if let Some(lock_ctx) = inner\n                .enabled_ctx\n                .as_mut()\n                .and_then(|ie| ie.lock_ctx.as_mut())\n            {\n                let lock = lock_ctx.lock.take();\n                if let Some(Locked::Write(permit)) = lock {\n                    let lock_status = match reason {\n                        // let the next request try to fetch it\n                        InternalError | StorageError | Deferred | UpstreamError => {\n                            LockStatus::TransientError\n                        }\n                        // depends on why the proxy upstream filter declined the request,\n                        // for now still allow next request try to acquire to avoid thundering herd\n                        DeclinedToUpstream => LockStatus::TransientError,\n                        // no need for the lock anymore\n                        OriginNotCache | ResponseTooLarge | PredictedResponseTooLarge => {\n                            LockStatus::GiveUp\n                        }\n                        Custom(reason) => lock_ctx.cache_lock.custom_lock_status(reason),\n                        // should never happen, NeverEnabled shouldn't hold a lock\n                        NeverEnabled => panic!(\"NeverEnabled holds a write lock\"),\n                        CacheLockGiveUp | CacheLockTimeout => {\n                            panic!(\"CacheLock* are for cache lock readers only\")\n                        }\n                    };\n                    lock_ctx\n                        .cache_lock\n                        .release(inner.key.as_ref().unwrap(), permit, lock_status);\n                }\n            }\n        }\n    }\n\n    /// Disable caching\n    pub fn disable(&mut self, reason: NoCacheReason) {\n        // XXX: compile type enforce?\n        assert!(\n            reason != NoCacheReason::NeverEnabled,\n            \"NeverEnabled not allowed as a disable reason\"\n        );\n        match self.phase {\n            CachePhase::Disabled(old_reason) => {\n                // replace reason\n                if old_reason == NoCacheReason::NeverEnabled {\n                    // safeguard, don't allow replacing NeverEnabled as a reason\n                    // TODO: can be promoted to assertion once confirmed nothing is attempting this\n                    warn!(\"Tried to replace cache NeverEnabled with reason: {reason:?}\");\n                    return;\n                }\n                self.phase = CachePhase::Disabled(reason);\n            }\n            _ => {\n                self.phase = CachePhase::Disabled(reason);\n                self.release_write_lock(reason);\n                // enabled_ctx will be cleared out\n                let mut inner_enabled = self\n                    .inner_mut()\n                    .enabled_ctx\n                    .take()\n                    .expect(\"could remove enabled_ctx on disable\");\n                // log initial disable reason\n                inner_enabled\n                    .traces\n                    .cache_span\n                    .set_tag(|| trace::Tag::new(\"disable_reason\", reason.as_str()));\n            }\n        }\n    }\n\n    /* The following methods panic when they are used in the wrong phase.\n     * This is better than returning errors as such panics are only caused by coding error, which\n     * should be fixed right away. Tokio runtime only crashes the current task instead of the whole\n     * program when these panics happen. */\n\n    /// Set the cache to bypass\n    ///\n    /// # Panic\n    /// This call is only allowed in [CachePhase::CacheKey] phase (before any cache lookup is performed).\n    /// Use it in any other phase will lead to panic.\n    pub fn bypass(&mut self) {\n        match self.phase {\n            CachePhase::CacheKey => {\n                // before cache lookup / found / miss\n                self.phase = CachePhase::Bypass;\n                self.inner_enabled_mut()\n                    .traces\n                    .cache_span\n                    .set_tag(|| trace::Tag::new(\"bypassed\", true));\n            }\n            _ => panic!(\"wrong phase to bypass HttpCache {:?}\", self.phase),\n        }\n    }\n\n    /// Enable the cache\n    ///\n    /// - `storage`: the cache storage backend that implements [storage::Storage]\n    /// - `eviction`: optionally the eviction manager, without it, nothing will be evicted from the storage\n    /// - `predictor`: optionally a cache predictor. The cache predictor predicts whether something is likely\n    ///   to be cacheable or not. This is useful because the proxy can apply different types of optimization to\n    ///   cacheable and uncacheable requests.\n    /// - `cache_lock`: optionally a cache lock which handles concurrent lookups to the same asset. Without it\n    ///   such lookups will all be allowed to fetch the asset independently.\n    pub fn enable(\n        &mut self,\n        storage: &'static (dyn storage::Storage + Sync),\n        eviction: Option<&'static (dyn eviction::EvictionManager + Sync)>,\n        predictor: Option<&'static (dyn predictor::CacheablePredictor + Sync)>,\n        cache_lock: Option<&'static CacheKeyLockImpl>,\n        option_overrides: Option<CacheOptionOverrides>,\n    ) {\n        match self.phase {\n            CachePhase::Disabled(_) => {\n                self.phase = CachePhase::Uninit;\n\n                let lock_ctx = cache_lock.map(|cache_lock| LockCtx {\n                    cache_lock,\n                    lock: None,\n                    wait_timeout: option_overrides\n                        .as_ref()\n                        .and_then(|overrides| overrides.wait_timeout),\n                });\n\n                self.inner = Some(Box::new(HttpCacheInner {\n                    enabled_ctx: Some(Box::new(HttpCacheInnerEnabled {\n                        meta: None,\n                        valid_after: None,\n                        miss_handler: None,\n                        body_reader: None,\n                        storage,\n                        eviction,\n                        lock_ctx,\n                        traces: CacheTraceCTX::new(),\n                    })),\n                    key: None,\n                    max_file_size_tracker: None,\n                    predictor,\n                }));\n            }\n            _ => panic!(\"Cannot enable already enabled HttpCache {:?}\", self.phase),\n        }\n    }\n\n    /// Set the cache lock implementation.\n    /// # Panic\n    /// Must be called before a cache lock is attempted to be acquired,\n    /// i.e. in the `cache_key_callback` or `cache_hit_filter` phases.\n    pub fn set_cache_lock(\n        &mut self,\n        cache_lock: Option<&'static CacheKeyLockImpl>,\n        option_overrides: Option<CacheOptionOverrides>,\n    ) {\n        match self.phase {\n            CachePhase::Disabled(_)\n            | CachePhase::CacheKey\n            | CachePhase::Stale\n            | CachePhase::Hit => {\n                let inner_enabled = self.inner_enabled_mut();\n                if inner_enabled\n                    .lock_ctx\n                    .as_ref()\n                    .is_some_and(|ctx| ctx.lock.is_some())\n                {\n                    panic!(\"lock already set when resetting cache lock\")\n                } else {\n                    let lock_ctx = cache_lock.map(|cache_lock| LockCtx {\n                        cache_lock,\n                        lock: None,\n                        wait_timeout: option_overrides.and_then(|overrides| overrides.wait_timeout),\n                    });\n                    inner_enabled.lock_ctx = lock_ctx;\n                }\n            }\n            _ => panic!(\"wrong phase: {:?}\", self.phase),\n        }\n    }\n\n    // Enable distributed tracing\n    pub fn enable_tracing(&mut self, parent_span: trace::Span) {\n        if let Some(inner_enabled) = self.inner.as_mut().and_then(|i| i.enabled_ctx.as_mut()) {\n            inner_enabled.traces.enable(parent_span);\n        }\n    }\n\n    // Get the cache parent tracing span\n    pub fn get_cache_span(&self) -> Option<trace::SpanHandle> {\n        self.inner\n            .as_ref()\n            .and_then(|i| i.enabled_ctx.as_ref().map(|ie| ie.traces.get_cache_span()))\n    }\n\n    // Get the cache `miss` tracing span\n    pub fn get_miss_span(&self) -> Option<trace::SpanHandle> {\n        self.inner\n            .as_ref()\n            .and_then(|i| i.enabled_ctx.as_ref().map(|ie| ie.traces.get_miss_span()))\n    }\n\n    // Get the cache `hit` tracing span\n    pub fn get_hit_span(&self) -> Option<trace::SpanHandle> {\n        self.inner\n            .as_ref()\n            .and_then(|i| i.enabled_ctx.as_ref().map(|ie| ie.traces.get_hit_span()))\n    }\n\n    // shortcut to access inner fields, panic if phase is disabled\n    #[inline]\n    fn inner_enabled_mut(&mut self) -> &mut HttpCacheInnerEnabled {\n        self.inner.as_mut().unwrap().enabled_ctx.as_mut().unwrap()\n    }\n\n    #[inline]\n    fn inner_enabled(&self) -> &HttpCacheInnerEnabled {\n        self.inner.as_ref().unwrap().enabled_ctx.as_ref().unwrap()\n    }\n\n    // shortcut to access inner fields, panic if cache was never enabled\n    #[inline]\n    fn inner_mut(&mut self) -> &mut HttpCacheInner {\n        self.inner.as_mut().unwrap()\n    }\n\n    #[inline]\n    fn inner(&self) -> &HttpCacheInner {\n        self.inner.as_ref().unwrap()\n    }\n\n    /// Set the cache key\n    /// # Panic\n    /// Cache key is only allowed to be set in its own phase. Set it in other phases will cause panic.\n    pub fn set_cache_key(&mut self, key: CacheKey) {\n        match self.phase {\n            CachePhase::Uninit | CachePhase::CacheKey => {\n                self.phase = CachePhase::CacheKey;\n                self.inner_mut().key = Some(key);\n            }\n            _ => panic!(\"wrong phase {:?}\", self.phase),\n        }\n    }\n\n    /// Return the cache key used for asset lookup\n    /// # Panic\n    /// Can only be called after the cache key is set and the cache is not disabled. Panic otherwise.\n    pub fn cache_key(&self) -> &CacheKey {\n        match self.phase {\n            CachePhase::Disabled(NoCacheReason::NeverEnabled) | CachePhase::Uninit => {\n                panic!(\"wrong phase {:?}\", self.phase)\n            }\n            _ => self\n                .inner()\n                .key\n                .as_ref()\n                .expect(\"cache key should be set (set_cache_key not called?)\"),\n        }\n    }\n\n    /// Return the max size allowed to be cached.\n    pub fn max_file_size_bytes(&self) -> Option<usize> {\n        assert!(\n            !matches!(\n                self.phase,\n                CachePhase::Disabled(NoCacheReason::NeverEnabled)\n            ),\n            \"tried to access max file size bytes when cache never enabled\"\n        );\n        self.inner()\n            .max_file_size_tracker\n            .as_ref()\n            .map(|t| t.max_file_size_bytes())\n    }\n\n    /// Set the maximum response _body_ size in bytes that will be admitted to the cache.\n    ///\n    /// Response header size should not contribute to the max file size.\n    ///\n    /// To track body bytes, call `track_bytes_for_max_file_size`.\n    pub fn set_max_file_size_bytes(&mut self, max_file_size_bytes: usize) {\n        match self.phase {\n            CachePhase::Disabled(_) => panic!(\"wrong phase {:?}\", self.phase),\n            _ => {\n                self.inner_mut().max_file_size_tracker =\n                    Some(MaxFileSizeTracker::new(max_file_size_bytes));\n            }\n        }\n    }\n\n    /// Record body bytes for the max file size tracker.\n    ///\n    /// The `bytes_len` input contributes to a cumulative body byte tracker.\n    ///\n    /// Once the cumulative body bytes exceeds the maximum allowable cache file size (as configured\n    /// by `set_max_file_size_bytes`), then the return value will be false.\n    ///\n    /// Else the return value is true as long as the max file size is not exceeded.\n    /// If max file size was not configured, the return value is always true.\n    pub fn track_body_bytes_for_max_file_size(&mut self, bytes_len: usize) -> bool {\n        // This is intended to be callable when cache has already been disabled,\n        // so that we can re-mark an asset as cacheable if the body size is under limits.\n        assert!(\n            !matches!(\n                self.phase,\n                CachePhase::Disabled(NoCacheReason::NeverEnabled)\n            ),\n            \"tried to access max file size bytes when cache never enabled\"\n        );\n        self.inner_mut()\n            .max_file_size_tracker\n            .as_mut()\n            .is_none_or(|t| t.add_body_bytes(bytes_len))\n    }\n\n    /// Check if the max file size has been exceeded according to max file size tracker.\n    ///\n    /// Return true if max file size was exceeded.\n    pub fn exceeded_max_file_size(&self) -> bool {\n        assert!(\n            !matches!(\n                self.phase,\n                CachePhase::Disabled(NoCacheReason::NeverEnabled)\n            ),\n            \"tried to access max file size bytes when cache never enabled\"\n        );\n        self.inner()\n            .max_file_size_tracker\n            .as_ref()\n            .is_some_and(|t| !t.allow_caching())\n    }\n\n    /// Set that cache is found in cache storage.\n    ///\n    /// This function is called after [Self::cache_lookup()] which returns the [CacheMeta] and\n    /// [HitHandler].\n    ///\n    /// The `hit_status` enum allows the caller to force expire assets.\n    pub fn cache_found(&mut self, meta: CacheMeta, hit_handler: HitHandler, hit_status: HitStatus) {\n        // Stale allowed because of cache lock and then retry\n        if !matches!(self.phase, CachePhase::CacheKey | CachePhase::Stale) {\n            panic!(\"wrong phase {:?}\", self.phase)\n        }\n\n        self.phase = match hit_status {\n            HitStatus::Fresh | HitStatus::ForceFresh => CachePhase::Hit,\n            HitStatus::Expired | HitStatus::ForceExpired => CachePhase::Stale,\n            HitStatus::FailedHitFilter | HitStatus::ForceMiss => self.phase,\n        };\n\n        let phase = self.phase;\n        let inner = self.inner_mut();\n\n        let key = inner.key.as_ref().expect(\"key must be set on hit\");\n        let inner_enabled = inner\n            .enabled_ctx\n            .as_mut()\n            .expect(\"cache_found must be called while cache enabled\");\n\n        // The cache lock might not be set for stale hit or hits treated as\n        // misses, so we need to initialize it here\n        let stale = phase == CachePhase::Stale;\n        if stale || hit_status.is_treated_as_miss() {\n            if let Some(lock_ctx) = inner_enabled.lock_ctx.as_mut() {\n                lock_ctx.lock = Some(lock_ctx.cache_lock.lock(key, stale));\n            }\n        }\n\n        if hit_status.is_treated_as_miss() {\n            // Clear the body and meta for hits that are treated as misses\n            inner_enabled.body_reader = None;\n            inner_enabled.meta = None;\n        } else {\n            // Set the metadata appropriately for legit hits\n            inner_enabled.traces.start_hit_span(phase, hit_status);\n            inner_enabled.traces.log_meta_in_hit_span(&meta);\n            if let Some(eviction) = inner_enabled.eviction {\n                // TODO: make access() accept CacheKey\n                let cache_key = key.to_compact();\n                if hit_handler.should_count_access() {\n                    let size = hit_handler.get_eviction_weight();\n                    eviction.access(&cache_key, size, meta.0.internal.fresh_until);\n                }\n            }\n            inner_enabled.meta = Some(meta);\n            inner_enabled.body_reader = Some(hit_handler);\n        }\n    }\n\n    /// Mark `self` to be cache miss.\n    ///\n    /// This function is called after [Self::cache_lookup()] finds nothing or the caller decides\n    /// not to use the assets found.\n    /// # Panic\n    /// Panic in other phases.\n    pub fn cache_miss(&mut self) {\n        match self.phase {\n            // from CacheKey: set state to miss during cache lookup\n            // from Bypass: response became cacheable, set state to miss to cache\n            // from Stale: waited for cache lock, then retried and found asset was gone\n            CachePhase::CacheKey | CachePhase::Bypass | CachePhase::Stale => {\n                self.phase = CachePhase::Miss;\n                // It's possible that we've set the meta on lookup and have come back around\n                // here after not being able to acquire the cache lock, and our item has since\n                // purged or expired. We should be sure that the meta is not set in this case\n                // as there shouldn't be a meta set for cache misses.\n                self.inner_enabled_mut().meta = None;\n                self.inner_enabled_mut().traces.start_miss_span();\n            }\n            _ => panic!(\"wrong phase {:?}\", self.phase),\n        }\n    }\n\n    /// Return the [HitHandler]\n    /// # Panic\n    /// Call this after [Self::cache_found()], panic in other phases.\n    pub fn hit_handler(&mut self) -> &mut HitHandler {\n        match self.phase {\n            CachePhase::Hit\n            | CachePhase::Stale\n            | CachePhase::StaleUpdating\n            | CachePhase::Revalidated\n            | CachePhase::RevalidatedNoCache(_) => {\n                self.inner_enabled_mut().body_reader.as_mut().unwrap()\n            }\n            _ => panic!(\"wrong phase {:?}\", self.phase),\n        }\n    }\n\n    /// Return the body reader during a cache admission (miss/expired) which decouples the downstream\n    /// read and upstream cache write\n    pub fn miss_body_reader(&mut self) -> Option<&mut HitHandler> {\n        match self.phase {\n            CachePhase::Miss | CachePhase::Expired => {\n                let inner_enabled = self.inner_enabled_mut();\n                if inner_enabled.storage.support_streaming_partial_write() {\n                    inner_enabled.body_reader.as_mut()\n                } else {\n                    // body_reader could be set even when the storage doesn't support streaming\n                    // Expired cache would have the reader set.\n                    None\n                }\n            }\n            _ => None,\n        }\n    }\n\n    /// Return whether the underlying storage backend supports streaming partial write.\n    ///\n    /// Returns None if cache is not enabled.\n    pub fn support_streaming_partial_write(&self) -> Option<bool> {\n        self.inner.as_ref().and_then(|inner| {\n            inner\n                .enabled_ctx\n                .as_ref()\n                .map(|c| c.storage.support_streaming_partial_write())\n        })\n    }\n\n    /// Call this when cache hit is fully read.\n    ///\n    /// This call will release resource if any and log the timing in tracing if set.\n    /// # Panic\n    /// Panic in phases where there is no cache hit.\n    pub async fn finish_hit_handler(&mut self) -> Result<()> {\n        match self.phase {\n            CachePhase::Hit\n            | CachePhase::Miss\n            | CachePhase::Expired\n            | CachePhase::Stale\n            | CachePhase::StaleUpdating\n            | CachePhase::Revalidated\n            | CachePhase::RevalidatedNoCache(_) => {\n                let inner = self.inner_mut();\n                let inner_enabled = inner.enabled_ctx.as_mut().expect(\"cache enabled\");\n                if inner_enabled.body_reader.is_none() {\n                    // already finished, we allow calling this function more than once\n                    return Ok(());\n                }\n                let body_reader = inner_enabled.body_reader.take().unwrap();\n                let key = inner.key.as_ref().unwrap();\n                let result = body_reader\n                    .finish(\n                        inner_enabled.storage,\n                        key,\n                        &inner_enabled.traces.hit_span.handle(),\n                    )\n                    .await;\n                inner_enabled.traces.finish_hit_span();\n                result\n            }\n            _ => panic!(\"wrong phase {:?}\", self.phase),\n        }\n    }\n\n    /// Set the [MissHandler] according to cache_key and meta, can only call once\n    pub async fn set_miss_handler(&mut self) -> Result<()> {\n        match self.phase {\n            // set_miss_handler() needs to be called after set_cache_meta() (which change Stale to Expire).\n            // This is an artificial rule to enforce the state transitions\n            CachePhase::Miss | CachePhase::Expired => {\n                let inner = self.inner_mut();\n                let inner_enabled = inner\n                    .enabled_ctx\n                    .as_mut()\n                    .expect(\"cache enabled on miss and expired\");\n                if inner_enabled.miss_handler.is_some() {\n                    panic!(\"write handler is already set\")\n                }\n                let meta = inner_enabled.meta.as_ref().unwrap();\n                let key = inner.key.as_ref().unwrap();\n                let miss_handler = inner_enabled\n                    .storage\n                    .get_miss_handler(key, meta, &inner_enabled.traces.get_miss_span())\n                    .await?;\n\n                inner_enabled.miss_handler = Some(miss_handler);\n\n                if inner_enabled.storage.support_streaming_partial_write() {\n                    // If a reader can access partial write, the cache lock can be released here\n                    // to let readers start reading the body.\n                    if let Some(lock_ctx) = inner_enabled.lock_ctx.as_mut() {\n                        let lock = lock_ctx.lock.take();\n                        if let Some(Locked::Write(permit)) = lock {\n                            lock_ctx.cache_lock.release(key, permit, LockStatus::Done);\n                        }\n                    }\n                    // Downstream read and upstream write can be decoupled\n                    let body_reader = inner_enabled\n                        .storage\n                        .lookup_streaming_write(\n                            key,\n                            inner_enabled\n                                .miss_handler\n                                .as_ref()\n                                .expect(\"miss handler already set\")\n                                .streaming_write_tag(),\n                            &inner_enabled.traces.get_miss_span(),\n                        )\n                        .await?;\n\n                    if let Some((_meta, body_reader)) = body_reader {\n                        inner_enabled.body_reader = Some(body_reader);\n                    } else {\n                        // body_reader should exist now because streaming_partial_write is to support it\n                        panic!(\"unable to get body_reader for {:?}\", meta);\n                    }\n                }\n                Ok(())\n            }\n            _ => panic!(\"wrong phase {:?}\", self.phase),\n        }\n    }\n\n    /// Return the [MissHandler] to write the response body to cache.\n    ///\n    /// `None`: the handler has not been set or already finished\n    pub fn miss_handler(&mut self) -> Option<&mut MissHandler> {\n        match self.phase {\n            CachePhase::Miss | CachePhase::Expired => {\n                self.inner_enabled_mut().miss_handler.as_mut()\n            }\n            _ => panic!(\"wrong phase {:?}\", self.phase),\n        }\n    }\n\n    /// Finish cache admission\n    ///\n    /// If [self] is dropped without calling this, the cache admission is considered incomplete and\n    /// should be cleaned up.\n    ///\n    /// This call will also trigger eviction if set.\n    pub async fn finish_miss_handler(&mut self) -> Result<()> {\n        match self.phase {\n            CachePhase::Miss | CachePhase::Expired => {\n                let inner = self.inner_mut();\n                let inner_enabled = inner\n                    .enabled_ctx\n                    .as_mut()\n                    .expect(\"cache enabled on miss and expired\");\n                if inner_enabled.miss_handler.is_none() {\n                    // already finished, we allow calling this function more than once\n                    return Ok(());\n                }\n                let miss_handler = inner_enabled.miss_handler.take().unwrap();\n                let size = miss_handler.finish().await?;\n                let key = inner\n                    .key\n                    .as_ref()\n                    .expect(\"key set by miss or expired phase\");\n                if let Some(lock_ctx) = inner_enabled.lock_ctx.as_mut() {\n                    let lock = lock_ctx.lock.take();\n                    if let Some(Locked::Write(permit)) = lock {\n                        // no need to call r.unlock() because release() will call it\n                        // r is a guard to make sure the lock is unlocked when this request is dropped\n                        lock_ctx.cache_lock.release(key, permit, LockStatus::Done);\n                    }\n                }\n                if let Some(eviction) = inner_enabled.eviction {\n                    let cache_key = key.to_compact();\n                    let meta = inner_enabled.meta.as_ref().unwrap();\n                    let evicted = match size {\n                        MissFinishType::Created(size) => {\n                            eviction.admit(cache_key, size, meta.0.internal.fresh_until)\n                        }\n                        MissFinishType::Appended(size, max_size) => {\n                            eviction.increment_weight(&cache_key, size, max_size)\n                        }\n                    };\n                    // actual eviction can be done async\n                    let span = inner_enabled.traces.child(\"eviction\");\n                    let handle = span.handle();\n                    let storage = inner_enabled.storage;\n                    tokio::task::spawn(async move {\n                        for item in evicted {\n                            if let Err(e) = storage.purge(&item, PurgeType::Eviction, &handle).await\n                            {\n                                warn!(\"Failed to purge {item} during eviction for finish miss handler: {e}\");\n                            }\n                        }\n                    });\n                }\n                inner_enabled.traces.finish_miss_span();\n                Ok(())\n            }\n            _ => panic!(\"wrong phase {:?}\", self.phase),\n        }\n    }\n\n    /// Set the [CacheMeta] of the cache\n    pub fn set_cache_meta(&mut self, meta: CacheMeta) {\n        match self.phase {\n            // TODO: store the staled meta somewhere else for future use?\n            CachePhase::Stale | CachePhase::Miss => {\n                let inner_enabled = self.inner_enabled_mut();\n                // TODO: have a separate expired span?\n                inner_enabled.traces.log_meta_in_miss_span(&meta);\n                inner_enabled.meta = Some(meta);\n            }\n            _ => panic!(\"wrong phase {:?}\", self.phase),\n        }\n        if self.phase == CachePhase::Stale {\n            self.phase = CachePhase::Expired;\n        }\n    }\n\n    /// Set the [CacheMeta] of the cache after revalidation.\n    ///\n    /// Certain info such as the original cache admission time will be preserved. Others will\n    /// be replaced by the input `meta`.\n    pub async fn revalidate_cache_meta(&mut self, mut meta: CacheMeta) -> Result<bool> {\n        let result = match self.phase {\n            CachePhase::Stale => {\n                let inner = self.inner_mut();\n                let inner_enabled = inner\n                    .enabled_ctx\n                    .as_mut()\n                    .expect(\"stale phase has cache enabled\");\n                // TODO: we should keep old meta in place, just use new one to update it\n                // that requires cacheable_filter to take a mut header and just return InternalMeta\n\n                // update new meta with old meta's created time\n                let old_meta = inner_enabled.meta.take().unwrap();\n                let created = old_meta.0.internal.created;\n                meta.0.internal.created = created;\n                // meta.internal.updated was already set to new meta's `created`,\n                // no need to set `updated` here\n                // Merge old extensions with new ones. New exts take precedence if they conflict.\n                let mut extensions = old_meta.0.extensions;\n                extensions.extend(meta.0.extensions);\n                meta.0.extensions = extensions;\n\n                inner_enabled.meta.replace(meta);\n\n                let mut span = inner_enabled.traces.child(\"update_meta\");\n                let result = inner_enabled\n                    .storage\n                    .update_meta(\n                        inner.key.as_ref().unwrap(),\n                        inner_enabled.meta.as_ref().unwrap(),\n                        &span.handle(),\n                    )\n                    .await;\n                span.set_tag(|| trace::Tag::new(\"updated\", result.is_ok()));\n\n                // regardless of result, release the cache lock\n                if let Some(lock_ctx) = inner_enabled.lock_ctx.as_mut() {\n                    let lock = lock_ctx.lock.take();\n                    if let Some(Locked::Write(permit)) = lock {\n                        lock_ctx.cache_lock.release(\n                            inner.key.as_ref().expect(\"key set by stale phase\"),\n                            permit,\n                            LockStatus::Done,\n                        );\n                    }\n                }\n\n                result\n            }\n            _ => panic!(\"wrong phase {:?}\", self.phase),\n        };\n        self.phase = CachePhase::Revalidated;\n        result\n    }\n\n    /// After a successful revalidation, update certain headers for the cached asset\n    /// such as `Etag` with the fresh response header `resp`.\n    pub fn revalidate_merge_header(&mut self, resp: &RespHeader) -> ResponseHeader {\n        match self.phase {\n            CachePhase::Stale => {\n                /*\n                 * https://datatracker.ietf.org/doc/html/rfc9110#section-15.4.5\n                 * 304 response MUST generate ... would have been sent in a 200 ...\n                 * - Content-Location, Date, ETag, and Vary\n                 * - Cache-Control and Expires...\n                 */\n                let mut old_header = self.inner_enabled().meta.as_ref().unwrap().0.header.clone();\n                let mut clone_header = |header_name: &'static str| {\n                    for (i, value) in resp.headers.get_all(header_name).iter().enumerate() {\n                        if i == 0 {\n                            old_header\n                                .insert_header(header_name, value)\n                                .expect(\"can add valid header\");\n                        } else {\n                            old_header\n                                .append_header(header_name, value)\n                                .expect(\"can add valid header\");\n                        }\n                    }\n                };\n                clone_header(\"cache-control\");\n                clone_header(\"expires\");\n                clone_header(\"cache-tag\");\n                clone_header(\"cdn-cache-control\");\n                clone_header(\"etag\");\n                // https://datatracker.ietf.org/doc/html/rfc9111#section-4.3.4\n                // \"...cache MUST update its header fields with the header fields provided in the 304...\"\n                // But if the Vary header changes, the cached response may no longer match the\n                // incoming request.\n                //\n                // For simplicity, ignore changing Vary in revalidation for now.\n                // TODO: if we support vary during revalidation, there are a few edge cases to\n                // consider (what if Vary header appears/disappears/changes)?\n                //\n                // clone_header(\"vary\");\n                old_header\n            }\n            _ => panic!(\"wrong phase {:?}\", self.phase),\n        }\n    }\n\n    /// Mark this asset uncacheable after revalidation\n    pub fn revalidate_uncacheable(&mut self, header: ResponseHeader, reason: NoCacheReason) {\n        match self.phase {\n            CachePhase::Stale => {\n                // replace cache meta header\n                self.inner_enabled_mut().meta.as_mut().unwrap().0.header = header;\n                // upstream request done, release write lock\n                self.release_write_lock(reason);\n            }\n            _ => panic!(\"wrong phase {:?}\", self.phase),\n        }\n        self.phase = CachePhase::RevalidatedNoCache(reason);\n        // TODO: remove this asset from cache once finished?\n    }\n\n    /// Mark this asset as stale, but being updated separately from this request.\n    pub fn set_stale_updating(&mut self) {\n        match self.phase {\n            CachePhase::Stale => self.phase = CachePhase::StaleUpdating,\n            _ => panic!(\"wrong phase {:?}\", self.phase),\n        }\n    }\n\n    /// Update the variance of the [CacheMeta].\n    ///\n    /// Note that this process may change the lookup `key`, and eventually (when the asset is\n    /// written to storage) invalidate other cached variants under the same primary key as the\n    /// current asset.\n    pub fn update_variance(&mut self, variance: Option<HashBinary>) {\n        // If this is a cache miss, we will simply update the variance in the meta.\n        //\n        // If this is an expired response, we will have to consider a few cases:\n        //\n        // **Case 1**: Variance was absent, but caller sets it now.\n        // We will just insert it into the meta. The current asset becomes the primary variant.\n        // Because the current location of the asset is already the primary variant, nothing else\n        // needs to be done.\n        //\n        // **Case 2**: Variance was present, but it changed or was removed.\n        // We want the current asset to take over the primary slot, in order to invalidate all\n        // other variants derived under the old Vary.\n        //\n        // **Case 3**: Variance did not change.\n        // Nothing needs to happen.\n        let inner = match self.phase {\n            CachePhase::Miss | CachePhase::Expired => self.inner_mut(),\n            _ => panic!(\"wrong phase {:?}\", self.phase),\n        };\n        let inner_enabled = inner\n            .enabled_ctx\n            .as_mut()\n            .expect(\"cache enabled on miss and expired\");\n\n        // Update the variance in the meta\n        if let Some(variance_hash) = variance.as_ref() {\n            inner_enabled\n                .meta\n                .as_mut()\n                .unwrap()\n                .set_variance_key(*variance_hash);\n        } else {\n            inner_enabled.meta.as_mut().unwrap().remove_variance();\n        }\n\n        // Change the lookup `key` if necessary, in order to admit asset into the primary slot\n        // instead of the secondary slot.\n        let key = inner.key.as_ref().unwrap();\n        if let Some(old_variance) = key.get_variance_key().as_ref() {\n            // This is a secondary variant slot.\n            if Some(*old_variance) != variance.as_ref() {\n                // This new variance does not match the variance in the cache key we used to look\n                // up this asset.\n                // Drop the cache lock to avoid leaving a dangling lock\n                // (because we locked with the old cache key for the secondary slot)\n                // TODO: maybe we should try to signal waiting readers to compete for the primary key\n                // lock instead? we will not be modifying this secondary slot so it's not actually\n                // ready for readers\n                if let Some(lock_ctx) = inner_enabled.lock_ctx.as_mut() {\n                    if let Some(Locked::Write(permit)) = lock_ctx.lock.take() {\n                        lock_ctx.cache_lock.release(key, permit, LockStatus::Done);\n                    }\n                }\n                // Remove the `variance` from the `key`, so that we admit this asset into the\n                // primary slot. (`key` is used to tell storage where to write the data.)\n                inner.key.as_mut().unwrap().remove_variance_key();\n            }\n        }\n    }\n\n    /// Return the [CacheMeta] of this asset\n    ///\n    /// # Panic\n    /// Panic in phases which has no cache meta.\n    pub fn cache_meta(&self) -> &CacheMeta {\n        match self.phase {\n            // TODO: allow in Bypass phase?\n            CachePhase::Stale\n            | CachePhase::StaleUpdating\n            | CachePhase::Expired\n            | CachePhase::Hit\n            | CachePhase::Revalidated\n            | CachePhase::RevalidatedNoCache(_) => self.inner_enabled().meta.as_ref().unwrap(),\n            CachePhase::Miss => {\n                // this is the async body read case, safe because body_reader is only set\n                // after meta is retrieved\n                if self.inner_enabled().body_reader.is_some() {\n                    self.inner_enabled().meta.as_ref().unwrap()\n                } else {\n                    panic!(\"wrong phase {:?}\", self.phase);\n                }\n            }\n\n            _ => panic!(\"wrong phase {:?}\", self.phase),\n        }\n    }\n\n    /// Return the [CacheMeta] of this asset if any\n    ///\n    /// Different from [Self::cache_meta()], this function is allowed to be called in\n    /// [CachePhase::Miss] phase where the cache meta maybe set.\n    /// # Panic\n    /// Panic in phases that shouldn't have cache meta.\n    pub fn maybe_cache_meta(&self) -> Option<&CacheMeta> {\n        match self.phase {\n            CachePhase::Miss\n            | CachePhase::Stale\n            | CachePhase::StaleUpdating\n            | CachePhase::Expired\n            | CachePhase::Hit\n            | CachePhase::Revalidated\n            | CachePhase::RevalidatedNoCache(_) => self.inner_enabled().meta.as_ref(),\n            _ => panic!(\"wrong phase {:?}\", self.phase),\n        }\n    }\n\n    /// Return the [`CacheKey`] of this asset if any.\n    ///\n    /// This is allowed to be called in any phase. If the cache key callback was not called,\n    /// this will return None.\n    pub fn maybe_cache_key(&self) -> Option<&CacheKey> {\n        (!matches!(\n            self.phase(),\n            CachePhase::Disabled(NoCacheReason::NeverEnabled) | CachePhase::Uninit\n        ))\n        .then(|| self.cache_key())\n    }\n\n    /// Perform the cache lookup from the given cache storage with the given cache key\n    ///\n    /// A cache hit will return [CacheMeta] which contains the header and meta info about\n    /// the cache as well as a [HitHandler] to read the cache hit body.\n    /// # Panic\n    /// Panic in other phases.\n    pub async fn cache_lookup(&mut self) -> Result<Option<(CacheMeta, HitHandler)>> {\n        match self.phase {\n            // Stale is allowed here because stale-> cache_lock -> lookup again\n            CachePhase::CacheKey | CachePhase::Stale => {\n                let inner = self\n                    .inner\n                    .as_mut()\n                    .expect(\"Cache phase is checked and should have inner\");\n                let inner_enabled = inner\n                    .enabled_ctx\n                    .as_mut()\n                    .expect(\"Cache enabled on cache_lookup\");\n                let mut span = inner_enabled.traces.child(\"lookup\");\n                let key = inner.key.as_ref().unwrap(); // safe, this phase should have cache key\n                let now = Instant::now();\n                let result = inner_enabled.storage.lookup(key, &span.handle()).await?;\n                // one request may have multiple lookups\n                self.digest.add_lookup_duration(now.elapsed());\n                let result = result.and_then(|(meta, header)| {\n                    if let Some(ts) = inner_enabled.valid_after {\n                        if meta.created() < ts {\n                            span.set_tag(|| trace::Tag::new(\"not valid\", true));\n                            return None;\n                        }\n                    }\n                    Some((meta, header))\n                });\n                if result.is_none() {\n                    if let Some(lock_ctx) = inner_enabled.lock_ctx.as_mut() {\n                        lock_ctx.lock = Some(lock_ctx.cache_lock.lock(key, false));\n                    }\n                }\n                span.set_tag(|| trace::Tag::new(\"found\", result.is_some()));\n                Ok(result)\n            }\n            _ => panic!(\"wrong phase {:?}\", self.phase),\n        }\n    }\n\n    /// Update variance and see if the meta matches the current variance\n    ///\n    /// `cache_lookup() -> compute vary hash -> cache_vary_lookup()`\n    /// This function allows callers to compute vary based on the initial cache hit.\n    /// `meta` should be the ones returned from the initial cache_lookup()\n    /// - return true if the meta is the variance.\n    /// - return false if the current meta doesn't match the variance, need to cache_lookup() again\n    pub fn cache_vary_lookup(&mut self, variance: HashBinary, meta: &CacheMeta) -> bool {\n        match self.phase {\n            // Stale is allowed here because stale-> cache_lock -> lookup again\n            CachePhase::CacheKey | CachePhase::Stale => {\n                let inner = self.inner_mut();\n                // make sure that all variances found are fresher than this asset\n                // this is because when purging all the variance, only the primary slot is deleted\n                // the created TS of the primary is the tombstone of all the variances\n                inner\n                    .enabled_ctx\n                    .as_mut()\n                    .expect(\"cache enabled\")\n                    .valid_after = Some(meta.created());\n\n                // update vary\n                let key = inner.key.as_mut().unwrap();\n                // if no variance was previously set, then this is the first cache hit\n                let is_initial_cache_hit = key.get_variance_key().is_none();\n                key.set_variance_key(variance);\n                let variance_binary = key.variance_bin();\n                let matches_variance = meta.variance() == variance_binary;\n\n                // We should remove the variance in the lookup `key` if this is the primary variant\n                // slot. We know this is the primary variant slot if this is the initial cache hit,\n                // AND the variance in the `key` already matches the `meta`'s.\n                //\n                // For the primary variant slot, the storage backend needs to use the primary key\n                // for both cache lookup and updating the meta. Otherwise it will look for the\n                // asset in the wrong location during revalidation.\n                //\n                // We can recreate the \"full\" cache key by using the meta's variance, if needed.\n                if matches_variance && is_initial_cache_hit {\n                    inner.key.as_mut().unwrap().remove_variance_key();\n                }\n\n                matches_variance\n            }\n            _ => panic!(\"wrong phase {:?}\", self.phase),\n        }\n    }\n\n    /// Whether this request is behind a cache lock in order to wait for another request to read the\n    /// asset.\n    pub fn is_cache_locked(&self) -> bool {\n        matches!(\n            self.inner_enabled()\n                .lock_ctx\n                .as_ref()\n                .and_then(|l| l.lock.as_ref()),\n            Some(Locked::Read(_))\n        )\n    }\n\n    /// Whether this request is the leader request to fetch the assets for itself and other requests\n    /// behind the cache lock.\n    pub fn is_cache_lock_writer(&self) -> bool {\n        matches!(\n            self.inner_enabled()\n                .lock_ctx\n                .as_ref()\n                .and_then(|l| l.lock.as_ref()),\n            Some(Locked::Write(_))\n        )\n    }\n\n    /// Take the write lock from this request to transfer it to another one.\n    /// # Panic\n    ///  Call is_cache_lock_writer() to check first, will panic otherwise.\n    pub fn take_write_lock(&mut self) -> (WritePermit, &'static CacheKeyLockImpl) {\n        let lock_ctx = self\n            .inner_enabled_mut()\n            .lock_ctx\n            .as_mut()\n            .expect(\"take_write_lock() called without cache lock\");\n        let lock = lock_ctx\n            .lock\n            .take()\n            .expect(\"take_write_lock() called without lock\");\n        match lock {\n            Locked::Write(w) => (w, lock_ctx.cache_lock),\n            Locked::Read(_) => panic!(\"take_write_lock() called on read lock\"),\n        }\n    }\n\n    /// Set the write lock, which is usually transferred from [Self::take_write_lock()]\n    ///\n    /// # Panic\n    /// Panics if cache lock was not originally configured for this request.\n    // TODO: it may make sense to allow configuring the CacheKeyLock here too that the write permit\n    // is associated with\n    // (The WritePermit comes from the CacheKeyLock and should be used when releasing from the CacheKeyLock,\n    // shouldn't be possible to give a WritePermit to a request using a different CacheKeyLock)\n    pub fn set_write_lock(&mut self, write_lock: WritePermit) {\n        if let Some(lock_ctx) = self.inner_enabled_mut().lock_ctx.as_mut() {\n            lock_ctx.lock.replace(Locked::Write(write_lock));\n        }\n    }\n\n    /// Whether this request's cache hit is staled\n    fn has_staled_asset(&self) -> bool {\n        matches!(self.phase, CachePhase::Stale | CachePhase::StaleUpdating)\n    }\n\n    /// Whether this asset is staled and stale if error is allowed\n    pub fn can_serve_stale_error(&self) -> bool {\n        self.has_staled_asset() && self.cache_meta().serve_stale_if_error(SystemTime::now())\n    }\n\n    /// Whether this asset is staled and stale while revalidate is allowed.\n    pub fn can_serve_stale_updating(&self) -> bool {\n        self.has_staled_asset()\n            && self\n                .cache_meta()\n                .serve_stale_while_revalidate(SystemTime::now())\n    }\n\n    /// Wait for the cache read lock to be unlocked\n    /// # Panic\n    /// Check [Self::is_cache_locked()], panic if this request doesn't have a read lock.\n    pub async fn cache_lock_wait(&mut self) -> LockStatus {\n        let inner_enabled = self.inner_enabled_mut();\n        let mut span = inner_enabled.traces.child(\"cache_lock\");\n        // should always call is_cache_locked() before this function, which should guarantee that\n        // the inner cache has a read lock and lock ctx\n        let (read_lock, status) = if let Some(lock_ctx) = inner_enabled.lock_ctx.as_mut() {\n            let lock = lock_ctx.lock.take(); // remove the lock from self\n            if let Some(Locked::Read(r)) = lock {\n                let now = Instant::now();\n                // it's possible for a request to be locked more than once,\n                // so wait the remainder of our configured timeout\n                let status = if let Some(wait_timeout) = lock_ctx.wait_timeout {\n                    let wait_timeout =\n                        wait_timeout.saturating_sub(self.lock_duration().unwrap_or(Duration::ZERO));\n                    match timeout(wait_timeout, r.wait()).await {\n                        Ok(()) => r.lock_status(),\n                        Err(_) => LockStatus::WaitTimeout,\n                    }\n                } else {\n                    r.wait().await;\n                    r.lock_status()\n                };\n                self.digest.add_lock_duration(now.elapsed());\n                (r, status)\n            } else {\n                panic!(\"cache_lock_wait on wrong type of lock\")\n            }\n        } else {\n            panic!(\"cache_lock_wait without cache lock\")\n        };\n        if let Some(lock_ctx) = self.inner_enabled().lock_ctx.as_ref() {\n            lock_ctx\n                .cache_lock\n                .trace_lock_wait(&mut span, &read_lock, status);\n        }\n        status\n    }\n\n    /// How long did this request wait behind the read lock\n    pub fn lock_duration(&self) -> Option<Duration> {\n        self.digest.lock_duration\n    }\n\n    /// How long did this request spent on cache lookup and reading the header\n    pub fn lookup_duration(&self) -> Option<Duration> {\n        self.digest.lookup_duration\n    }\n\n    /// Delete the asset from the cache storage\n    /// # Panic\n    /// Need to be called after the cache key is set. Panic otherwise.\n    pub async fn purge(&self) -> Result<bool> {\n        match self.phase {\n            CachePhase::CacheKey => {\n                let inner = self.inner();\n                let inner_enabled = self.inner_enabled();\n                let span = inner_enabled.traces.child(\"purge\");\n                let key = inner.key.as_ref().unwrap().to_compact();\n                Self::purge_impl(inner_enabled.storage, inner_enabled.eviction, &key, span).await\n            }\n            _ => panic!(\"wrong phase {:?}\", self.phase),\n        }\n    }\n\n    /// Delete the asset from the cache storage via a spawned task.\n    /// Returns corresponding `JoinHandle` of that task.\n    /// # Panic\n    /// Need to be called after the cache key is set. Panic otherwise.\n    pub fn spawn_async_purge(\n        &self,\n        context: &'static str,\n    ) -> tokio::task::JoinHandle<Result<bool>> {\n        if matches!(self.phase, CachePhase::Disabled(_) | CachePhase::Uninit) {\n            panic!(\"wrong phase {:?}\", self.phase);\n        }\n\n        let inner_enabled = self.inner_enabled();\n        let span = inner_enabled.traces.child(\"purge\");\n        let key = self.inner().key.as_ref().unwrap().to_compact();\n        let storage = inner_enabled.storage;\n        let eviction = inner_enabled.eviction;\n        tokio::task::spawn(async move {\n            Self::purge_impl(storage, eviction, &key, span)\n                .await\n                .map_err(|e| {\n                    warn!(\"Failed to purge {key} (context: {context}): {e}\");\n                    e\n                })\n        })\n    }\n\n    async fn purge_impl(\n        storage: &'static (dyn storage::Storage + Sync),\n        eviction: Option<&'static (dyn eviction::EvictionManager + Sync)>,\n        key: &CompactCacheKey,\n        mut span: Span,\n    ) -> Result<bool> {\n        let result = storage\n            .purge(key, PurgeType::Invalidation, &span.handle())\n            .await;\n        let purged = matches!(result, Ok(true));\n        // need to inform eviction manager if asset was removed\n        if let Some(eviction) = eviction.as_ref() {\n            if purged {\n                eviction.remove(key);\n            }\n        }\n        span.set_tag(|| trace::Tag::new(\"purged\", purged));\n        result\n    }\n\n    /// Check the cacheable prediction\n    ///\n    /// Return true if the predictor is not set\n    pub fn cacheable_prediction(&self) -> bool {\n        if let Some(predictor) = self.inner().predictor {\n            predictor.cacheable_prediction(self.cache_key())\n        } else {\n            true\n        }\n    }\n\n    /// Tell the predictor that this response, which is previously predicted to be uncacheable,\n    /// is cacheable now.\n    pub fn response_became_cacheable(&self) {\n        if let Some(predictor) = self.inner().predictor {\n            predictor.mark_cacheable(self.cache_key());\n        }\n    }\n\n    /// Tell the predictor that this response is uncacheable so that it will know next time\n    /// this request arrives.\n    pub fn response_became_uncacheable(&self, reason: NoCacheReason) {\n        if let Some(predictor) = self.inner().predictor {\n            predictor.mark_uncacheable(self.cache_key(), reason);\n        }\n    }\n\n    /// Tag all spans as being part of a subrequest.\n    pub fn tag_as_subrequest(&mut self) {\n        self.inner_enabled_mut()\n            .traces\n            .cache_span\n            .set_tag(|| Tag::new(\"is_subrequest\", true))\n    }\n}\n"
  },
  {
    "path": "pingora-cache/src/lock.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! Cache lock\n\nuse crate::{hashtable::ConcurrentHashTable, key::CacheHashKey, CacheKey};\nuse crate::{Span, Tag};\n\nuse http::Extensions;\nuse pingora_timeout::timeout;\nuse std::sync::Arc;\nuse std::time::Duration;\n\npub type CacheKeyLockImpl = dyn CacheKeyLock + Send + Sync;\n\npub trait CacheKeyLock {\n    /// Try to lock a cache fetch\n    ///\n    /// If `stale_writer` is true, this fetch is to revalidate an asset already in cache.\n    /// Else this fetch was a cache miss (i.e. not found via lookup, or force missed).\n    ///\n    /// Users should call after a cache miss before fetching the asset.\n    /// The returned [Locked] will tell the caller either to fetch or wait.\n    fn lock(&self, key: &CacheKey, stale_writer: bool) -> Locked;\n\n    /// Release a lock for the given key\n    ///\n    /// When the write lock is dropped without being released, the read lock holders will consider\n    /// it to be failed so that they will compete for the write lock again.\n    fn release(&self, key: &CacheKey, permit: WritePermit, reason: LockStatus);\n\n    /// Set tags on a trace span for the cache lock wait.\n    fn trace_lock_wait(&self, span: &mut Span, _read_lock: &ReadLock, lock_status: LockStatus) {\n        let tag_value: &'static str = lock_status.into();\n        span.set_tag(|| Tag::new(\"status\", tag_value));\n    }\n\n    /// Set a lock status for a custom `NoCacheReason`.\n    fn custom_lock_status(&self, _custom_no_cache: &'static str) -> LockStatus {\n        // treat custom no cache reasons as GiveUp by default\n        // (like OriginNotCache)\n        LockStatus::GiveUp\n    }\n}\n\nconst N_SHARDS: usize = 16;\n\n/// The global cache locking manager\n#[derive(Debug)]\npub struct CacheLock {\n    lock_table: ConcurrentHashTable<LockStub, N_SHARDS>,\n    // fixed lock timeout values for now\n    age_timeout_default: Duration,\n}\n\n/// A struct representing locked cache access\n#[derive(Debug)]\npub enum Locked {\n    /// The writer is allowed to fetch the asset\n    Write(WritePermit),\n    /// The reader waits for the writer to fetch the asset\n    Read(ReadLock),\n}\n\nimpl Locked {\n    /// Is this a write lock\n    pub fn is_write(&self) -> bool {\n        matches!(self, Self::Write(_))\n    }\n}\n\nimpl CacheLock {\n    /// Create a new [CacheLock] with the given lock timeout\n    ///\n    /// Age timeout refers to how long a writer has been holding onto a particular lock, and wait\n    /// timeout refers to how long a reader may hold onto any number of locks before giving up.\n    /// When either timeout is reached, the read locks are automatically unlocked.\n    pub fn new_boxed(age_timeout: Duration) -> Box<Self> {\n        Box::new(CacheLock {\n            lock_table: ConcurrentHashTable::new(),\n            age_timeout_default: age_timeout,\n        })\n    }\n\n    /// Create a new [CacheLock] with the given lock timeout\n    ///\n    /// Age timeout refers to how long a writer has been holding onto a particular lock, and wait\n    /// timeout refers to how long a reader may hold onto any number of locks before giving up.\n    /// When either timeout is reached, the read locks are automatically unlocked.\n    pub fn new(age_timeout_default: Duration) -> Self {\n        CacheLock {\n            lock_table: ConcurrentHashTable::new(),\n            age_timeout_default,\n        }\n    }\n}\n\nimpl CacheKeyLock for CacheLock {\n    fn lock(&self, key: &CacheKey, stale_writer: bool) -> Locked {\n        let hash = key.combined_bin();\n        let key = u128::from_be_bytes(hash); // endianness doesn't matter\n        let table = self.lock_table.get(key);\n        if let Some(lock) = table.read().get(&key) {\n            // already has an ongoing request\n            // If the lock status is dangling or timeout, the lock will _remain_ in the table\n            // and readers should attempt to replace it.\n            // In the case of writer timeout, any remaining readers that were waiting on THIS\n            // LockCore should have (or are about to) timed out on their own.\n            // Finding a Timeout status means that THIS writer's lock already expired, so future\n            // requests ought to recreate the lock.\n            if !matches!(\n                lock.0.lock_status(),\n                LockStatus::Dangling | LockStatus::AgeTimeout\n            ) {\n                return Locked::Read(lock.read_lock());\n            }\n            // Dangling: the previous writer quit without unlocking the lock. Requests should\n            // compete for the write lock again.\n        }\n\n        let mut table = table.write();\n        // check again in case another request already added it\n        if let Some(lock) = table.get(&key) {\n            if !matches!(\n                lock.0.lock_status(),\n                LockStatus::Dangling | LockStatus::AgeTimeout\n            ) {\n                return Locked::Read(lock.read_lock());\n            }\n        }\n        let (permit, stub) =\n            WritePermit::new(self.age_timeout_default, stale_writer, Extensions::new());\n        table.insert(key, stub);\n        Locked::Write(permit)\n    }\n\n    fn release(&self, key: &CacheKey, mut permit: WritePermit, reason: LockStatus) {\n        let hash = key.combined_bin();\n        let key = u128::from_be_bytes(hash); // endianness doesn't matter\n        if permit.lock.lock_status() == LockStatus::AgeTimeout {\n            // if lock age timed out, then readers are capable of\n            // replacing the lock associated with this permit from the lock table\n            // (see lock() implementation)\n            // keep the lock status as Timeout accordingly when unlocking\n            // (because we aren't removing it from the lock_table)\n            permit.unlock(LockStatus::AgeTimeout);\n        } else if let Some(_lock) = self.lock_table.write(key).remove(&key) {\n            permit.unlock(reason);\n        }\n        // these situations above should capture all possible options,\n        // else dangling cache lock may start\n    }\n}\n\nuse log::warn;\nuse std::sync::atomic::{AtomicU8, Ordering};\nuse std::time::Instant;\nuse strum::{FromRepr, IntoStaticStr};\nuse tokio::sync::Semaphore;\n\n/// Status which the read locks could possibly see.\n#[derive(Debug, Copy, Clone, PartialEq, Eq, IntoStaticStr, FromRepr)]\n#[repr(u8)]\npub enum LockStatus {\n    /// Waiting for the writer to populate the asset\n    Waiting = 0,\n    /// The writer finishes, readers can start\n    Done = 1,\n    /// The writer encountered error, such as network issue. A new writer will be elected.\n    TransientError = 2,\n    /// The writer observed that no cache lock is needed (e.g., uncacheable), readers should start\n    /// to fetch independently without a new writer\n    GiveUp = 3,\n    /// The write lock is dropped without being unlocked\n    Dangling = 4,\n    /// Reader has held onto cache locks for too long, give up\n    WaitTimeout = 5,\n    /// The lock is held for too long by the writer\n    AgeTimeout = 6,\n}\n\nimpl From<LockStatus> for u8 {\n    fn from(l: LockStatus) -> u8 {\n        match l {\n            LockStatus::Waiting => 0,\n            LockStatus::Done => 1,\n            LockStatus::TransientError => 2,\n            LockStatus::GiveUp => 3,\n            LockStatus::Dangling => 4,\n            LockStatus::WaitTimeout => 5,\n            LockStatus::AgeTimeout => 6,\n        }\n    }\n}\n\nimpl From<u8> for LockStatus {\n    fn from(v: u8) -> Self {\n        Self::from_repr(v).unwrap_or(Self::GiveUp)\n    }\n}\n\n#[derive(Debug)]\npub struct LockCore {\n    pub lock_start: Instant,\n    pub age_timeout: Duration,\n    pub(super) lock: Semaphore,\n    // use u8 for Atomic enum\n    lock_status: AtomicU8,\n    stale_writer: bool,\n    extensions: Extensions,\n}\n\nimpl LockCore {\n    pub fn new_arc(timeout: Duration, stale_writer: bool, extensions: Extensions) -> Arc<Self> {\n        Arc::new(LockCore {\n            lock: Semaphore::new(0),\n            age_timeout: timeout,\n            lock_start: Instant::now(),\n            lock_status: AtomicU8::new(LockStatus::Waiting.into()),\n            stale_writer,\n            extensions,\n        })\n    }\n\n    pub fn locked(&self) -> bool {\n        self.lock.available_permits() == 0\n    }\n\n    pub fn unlock(&self, reason: LockStatus) {\n        assert!(\n            reason != LockStatus::WaitTimeout,\n            \"WaitTimeout is not stored in LockCore\"\n        );\n        self.lock_status.store(reason.into(), Ordering::SeqCst);\n        // Any small positive number will do, 10 is used for RwLock as well.\n        // No need to wake up all at once.\n        self.lock.add_permits(10);\n    }\n\n    pub fn lock_status(&self) -> LockStatus {\n        self.lock_status.load(Ordering::SeqCst).into()\n    }\n\n    /// Was this lock for a stale cache fetch writer?\n    pub fn stale_writer(&self) -> bool {\n        self.stale_writer\n    }\n\n    pub fn extensions(&self) -> &Extensions {\n        &self.extensions\n    }\n}\n\n// all 3 structs below are just Arc<LockCore> with different interfaces\n\n/// ReadLock: the requests who get it need to wait until it is released\n#[derive(Debug)]\npub struct ReadLock(Arc<LockCore>);\n\nimpl ReadLock {\n    /// Wait for the writer to release the lock\n    pub async fn wait(&self) {\n        if !self.locked() {\n            return;\n        }\n\n        // FIXME: for now it is the awkward responsibility of the ReadLock to set the\n        // timeout status on the lock itself because the write permit cannot lock age\n        // timeout on its own\n        // TODO: need to be careful not to wake everyone up at the same time\n        // (maybe not an issue because regular cache lock release behaves that way)\n        if let Some(duration) = self.0.age_timeout.checked_sub(self.0.lock_start.elapsed()) {\n            match timeout(duration, self.0.lock.acquire()).await {\n                Ok(Ok(_)) => { // permit is returned to Semaphore right away\n                }\n                Ok(Err(e)) => {\n                    warn!(\"error acquiring semaphore {e:?}\")\n                }\n                Err(_) => {\n                    self.0\n                        .lock_status\n                        .store(LockStatus::AgeTimeout.into(), Ordering::SeqCst);\n                }\n            }\n        } else {\n            // expiration has already occurred, store timeout status\n            self.0\n                .lock_status\n                .store(LockStatus::AgeTimeout.into(), Ordering::SeqCst);\n        }\n    }\n\n    /// Test if it is still locked\n    pub fn locked(&self) -> bool {\n        self.0.locked()\n    }\n\n    /// Whether the lock is expired, e.g., the writer has been holding the lock for too long\n    pub fn expired(&self) -> bool {\n        // NOTE: this is whether the lock is currently expired\n        // not whether it was timed out during wait()\n        self.0.lock_start.elapsed() >= self.0.age_timeout\n    }\n\n    /// The current status of the lock\n    pub fn lock_status(&self) -> LockStatus {\n        let status = self.0.lock_status();\n        if matches!(status, LockStatus::Waiting) && self.expired() {\n            LockStatus::AgeTimeout\n        } else {\n            status\n        }\n    }\n\n    pub fn extensions(&self) -> &Extensions {\n        self.0.extensions()\n    }\n}\n\n/// WritePermit: requires who get it need to populate the cache and then release it\n#[derive(Debug)]\npub struct WritePermit {\n    lock: Arc<LockCore>,\n    finished: bool,\n}\n\nimpl WritePermit {\n    /// Create a new lock, with a permit to be given to the associated writer.\n    pub fn new(\n        timeout: Duration,\n        stale_writer: bool,\n        extensions: Extensions,\n    ) -> (WritePermit, LockStub) {\n        let lock = LockCore::new_arc(timeout, stale_writer, extensions);\n        let stub = LockStub(lock.clone());\n        (\n            WritePermit {\n                lock,\n                finished: false,\n            },\n            stub,\n        )\n    }\n\n    /// Was this lock for a stale cache fetch writer?\n    pub fn stale_writer(&self) -> bool {\n        self.lock.stale_writer()\n    }\n\n    pub fn unlock(&mut self, reason: LockStatus) {\n        self.finished = true;\n        self.lock.unlock(reason);\n    }\n\n    pub fn lock_status(&self) -> LockStatus {\n        self.lock.lock_status()\n    }\n\n    pub fn extensions(&self) -> &Extensions {\n        self.lock.extensions()\n    }\n}\n\nimpl Drop for WritePermit {\n    fn drop(&mut self) {\n        // Writer exited without properly unlocking. We let others to compete for the write lock again\n        if !self.finished {\n            debug_assert!(false, \"Dangling cache lock started!\");\n            self.unlock(LockStatus::Dangling);\n        }\n    }\n}\n\n#[derive(Debug)]\npub struct LockStub(pub Arc<LockCore>);\nimpl LockStub {\n    pub fn read_lock(&self) -> ReadLock {\n        ReadLock(self.0.clone())\n    }\n\n    pub fn extensions(&self) -> &Extensions {\n        &self.0.extensions\n    }\n}\n\n#[cfg(test)]\nmod test {\n    use super::*;\n    use crate::CacheKey;\n\n    #[test]\n    fn test_get_release() {\n        let cache_lock = CacheLock::new_boxed(Duration::from_secs(1000));\n        let key1 = CacheKey::new(\"\", \"a\", \"1\");\n        let locked1 = cache_lock.lock(&key1, false);\n        assert!(locked1.is_write()); // write permit\n        let locked2 = cache_lock.lock(&key1, false);\n        assert!(!locked2.is_write()); // read lock\n        if let Locked::Write(permit) = locked1 {\n            cache_lock.release(&key1, permit, LockStatus::Done);\n        }\n        let locked3 = cache_lock.lock(&key1, false);\n        assert!(locked3.is_write()); // write permit again\n        if let Locked::Write(permit) = locked3 {\n            cache_lock.release(&key1, permit, LockStatus::Done);\n        }\n    }\n\n    #[tokio::test]\n    async fn test_lock() {\n        let cache_lock = CacheLock::new_boxed(Duration::from_secs(1000));\n        let key1 = CacheKey::new(\"\", \"a\", \"1\");\n        let mut permit = match cache_lock.lock(&key1, false) {\n            Locked::Write(w) => w,\n            _ => panic!(),\n        };\n        let lock = match cache_lock.lock(&key1, false) {\n            Locked::Read(r) => r,\n            _ => panic!(),\n        };\n        assert!(lock.locked());\n        let handle = tokio::spawn(async move {\n            lock.wait().await;\n            assert_eq!(lock.lock_status(), LockStatus::Done);\n        });\n        permit.unlock(LockStatus::Done);\n        handle.await.unwrap(); // check lock is unlocked and the task is returned\n    }\n\n    #[tokio::test]\n    async fn test_lock_timeout() {\n        let cache_lock = CacheLock::new_boxed(Duration::from_secs(1));\n        let key1 = CacheKey::new(\"\", \"a\", \"1\");\n        let mut permit = match cache_lock.lock(&key1, false) {\n            Locked::Write(w) => w,\n            _ => panic!(),\n        };\n        let lock = match cache_lock.lock(&key1, false) {\n            Locked::Read(r) => r,\n            _ => panic!(),\n        };\n        assert!(lock.locked());\n\n        let handle = tokio::spawn(async move {\n            // timed out\n            lock.wait().await;\n            assert_eq!(lock.lock_status(), LockStatus::AgeTimeout);\n        });\n\n        tokio::time::sleep(Duration::from_millis(2100)).await;\n\n        handle.await.unwrap(); // check lock is timed out\n\n        // expired lock - we will be able to install a new lock instead\n        let mut permit2 = match cache_lock.lock(&key1, false) {\n            Locked::Write(w) => w,\n            _ => panic!(),\n        };\n        let lock2 = match cache_lock.lock(&key1, false) {\n            Locked::Read(r) => r,\n            _ => panic!(),\n        };\n        assert!(lock2.locked());\n        let handle = tokio::spawn(async move {\n            // timed out\n            lock2.wait().await;\n            assert_eq!(lock2.lock_status(), LockStatus::Done);\n        });\n\n        permit.unlock(LockStatus::Done);\n        permit2.unlock(LockStatus::Done);\n        handle.await.unwrap();\n    }\n\n    #[tokio::test]\n    async fn test_lock_expired_release() {\n        let cache_lock = CacheLock::new_boxed(Duration::from_secs(1));\n        let key1 = CacheKey::new(\"\", \"a\", \"1\");\n        let permit = match cache_lock.lock(&key1, false) {\n            Locked::Write(w) => w,\n            _ => panic!(),\n        };\n\n        let lock = match cache_lock.lock(&key1, false) {\n            Locked::Read(r) => r,\n            _ => panic!(),\n        };\n        assert!(lock.locked());\n        let handle = tokio::spawn(async move {\n            // timed out\n            lock.wait().await;\n            assert_eq!(lock.lock_status(), LockStatus::AgeTimeout);\n        });\n\n        tokio::time::sleep(Duration::from_millis(1100)).await; // let lock age time out\n        handle.await.unwrap(); // check lock is timed out\n\n        // writer finally finishes\n        cache_lock.release(&key1, permit, LockStatus::Done);\n\n        // can reacquire after release\n        let mut permit = match cache_lock.lock(&key1, false) {\n            Locked::Write(w) => w,\n            _ => panic!(),\n        };\n        assert_eq!(permit.lock.lock_status(), LockStatus::Waiting);\n\n        let lock2 = match cache_lock.lock(&key1, false) {\n            Locked::Read(r) => r,\n            _ => panic!(),\n        };\n        assert!(lock2.locked());\n        let handle = tokio::spawn(async move {\n            // timed out\n            lock2.wait().await;\n            assert_eq!(lock2.lock_status(), LockStatus::Done);\n        });\n\n        permit.unlock(LockStatus::Done);\n        handle.await.unwrap();\n    }\n\n    #[tokio::test]\n    async fn test_lock_expired_no_reader() {\n        let cache_lock = CacheLock::new_boxed(Duration::from_secs(1));\n        let key1 = CacheKey::new(\"\", \"a\", \"1\");\n        let mut permit = match cache_lock.lock(&key1, false) {\n            Locked::Write(w) => w,\n            _ => panic!(),\n        };\n        tokio::time::sleep(Duration::from_millis(1100)).await; // let lock age time out\n\n        // lock expired without reader, but status is not yet set\n        assert_eq!(permit.lock.lock_status(), LockStatus::Waiting);\n\n        let lock = match cache_lock.lock(&key1, false) {\n            Locked::Read(r) => r,\n            _ => panic!(),\n        };\n        // reader expires write permit\n        lock.wait().await;\n        assert_eq!(lock.lock_status(), LockStatus::AgeTimeout);\n        assert_eq!(permit.lock.lock_status(), LockStatus::AgeTimeout);\n        permit.unlock(LockStatus::AgeTimeout);\n    }\n\n    #[tokio::test]\n    async fn test_lock_concurrent() {\n        let _ = env_logger::builder().is_test(true).try_init();\n        // Test that concurrent attempts to compete for a lock run without issues\n        let cache_lock = Arc::new(CacheLock::new_boxed(Duration::from_secs(1)));\n        let key1 = CacheKey::new(\"\", \"a\", \"1\");\n\n        let mut handles = vec![];\n\n        const READERS: usize = 30;\n        for _ in 0..READERS {\n            let key1 = key1.clone();\n            let cache_lock = cache_lock.clone();\n            // simulate a cache lookup / lock attempt loop\n            handles.push(tokio::spawn(async move {\n                // timed out\n                loop {\n                    match cache_lock.lock(&key1, false) {\n                        Locked::Write(permit) => {\n                            let _ = tokio::time::sleep(Duration::from_millis(5)).await;\n                            cache_lock.release(&key1, permit, LockStatus::Done);\n                            break;\n                        }\n                        Locked::Read(r) => {\n                            r.wait().await;\n                        }\n                    }\n                }\n            }));\n        }\n\n        for handle in handles {\n            handle.await.unwrap();\n        }\n    }\n}\n"
  },
  {
    "path": "pingora-cache/src/max_file_size.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! Set limit on the largest size to cache\n\nuse pingora_error::ErrorType;\n\n/// Error type returned when the limit is reached.\npub const ERR_RESPONSE_TOO_LARGE: ErrorType = ErrorType::Custom(\"response too large\");\n\n// Body bytes tracker to adjust (predicted) cacheability,\n// even if cache has been disabled.\n#[derive(Debug)]\npub(crate) struct MaxFileSizeTracker {\n    body_bytes: usize,\n    max_size: usize,\n}\n\nimpl MaxFileSizeTracker {\n    // Create a new Tracker object.\n    pub fn new(max_size: usize) -> MaxFileSizeTracker {\n        MaxFileSizeTracker {\n            body_bytes: 0,\n            max_size,\n        }\n    }\n\n    // Add bytes to the tracker.\n    // If return value is true, the tracker bytes are under the max size allowed.\n    pub fn add_body_bytes(&mut self, bytes: usize) -> bool {\n        self.body_bytes += bytes;\n        self.allow_caching()\n    }\n\n    pub fn max_file_size_bytes(&self) -> usize {\n        self.max_size\n    }\n\n    pub fn allow_caching(&self) -> bool {\n        self.body_bytes <= self.max_size\n    }\n}\n"
  },
  {
    "path": "pingora-cache/src/memory.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! Hash map based in memory cache\n//!\n//! For testing only, not for production use\n\n//TODO: Mark this module #[test] only\n\nuse super::*;\nuse crate::key::CompactCacheKey;\nuse crate::storage::{streaming_write::U64WriteId, HandleHit, HandleMiss};\nuse crate::trace::SpanHandle;\n\nuse async_trait::async_trait;\nuse bytes::Bytes;\nuse parking_lot::RwLock;\nuse pingora_error::*;\nuse std::any::Any;\nuse std::collections::HashMap;\nuse std::sync::atomic::{AtomicU64, Ordering};\nuse std::sync::Arc;\nuse tokio::sync::watch;\n\ntype BinaryMeta = (Vec<u8>, Vec<u8>);\n\npub(crate) struct CacheObject {\n    pub meta: BinaryMeta,\n    pub body: Arc<Vec<u8>>,\n}\n\npub(crate) struct TempObject {\n    pub meta: BinaryMeta,\n    // these are Arc because they need to continue to exist after this TempObject is removed\n    pub body: Arc<RwLock<Vec<u8>>>,\n    bytes_written: Arc<watch::Sender<PartialState>>, // this should match body.len()\n}\n\nimpl TempObject {\n    fn new(meta: BinaryMeta) -> Self {\n        let (tx, _rx) = watch::channel(PartialState::Partial(0));\n        TempObject {\n            meta,\n            body: Arc::new(RwLock::new(Vec::new())),\n            bytes_written: Arc::new(tx),\n        }\n    }\n    // this is not at all optimized\n    fn make_cache_object(&self) -> CacheObject {\n        let meta = self.meta.clone();\n        let body = Arc::new(self.body.read().clone());\n        CacheObject { meta, body }\n    }\n}\n\n/// Hash map based in memory cache\n///\n/// For testing only, not for production use.\npub struct MemCache {\n    pub(crate) cached: Arc<RwLock<HashMap<String, CacheObject>>>,\n    pub(crate) temp: Arc<RwLock<HashMap<String, HashMap<u64, TempObject>>>>,\n    pub(crate) last_temp_id: AtomicU64,\n}\n\nimpl MemCache {\n    /// Create a new [MemCache]\n    pub fn new() -> Self {\n        MemCache {\n            cached: Arc::new(RwLock::new(HashMap::new())),\n            temp: Arc::new(RwLock::new(HashMap::new())),\n            last_temp_id: AtomicU64::new(0),\n        }\n    }\n}\n\npub enum MemHitHandler {\n    Complete(CompleteHit),\n    Partial(PartialHit),\n}\n\n#[derive(Copy, Clone)]\nenum PartialState {\n    Partial(usize),\n    Complete(usize),\n}\n\npub struct CompleteHit {\n    body: Arc<Vec<u8>>,\n    done: bool,\n    range_start: usize,\n    range_end: usize,\n}\n\nimpl CompleteHit {\n    fn get(&mut self) -> Option<Bytes> {\n        if self.done {\n            None\n        } else {\n            self.done = true;\n            Some(Bytes::copy_from_slice(\n                &self.body.as_slice()[self.range_start..self.range_end],\n            ))\n        }\n    }\n\n    fn seek(&mut self, start: usize, end: Option<usize>) -> Result<()> {\n        if start >= self.body.len() {\n            return Error::e_explain(\n                ErrorType::InternalError,\n                format!(\"seek start out of range {start} >= {}\", self.body.len()),\n            );\n        }\n        self.range_start = start;\n        if let Some(end) = end {\n            // end over the actual last byte is allowed, we just need to return the actual bytes\n            self.range_end = std::cmp::min(self.body.len(), end);\n        }\n        // seek resets read so that one handler can be used for multiple ranges\n        self.done = false;\n        Ok(())\n    }\n}\n\npub struct PartialHit {\n    body: Arc<RwLock<Vec<u8>>>,\n    bytes_written: watch::Receiver<PartialState>,\n    bytes_read: usize,\n}\n\nimpl PartialHit {\n    async fn read(&mut self) -> Option<Bytes> {\n        loop {\n            let bytes_written = *self.bytes_written.borrow_and_update();\n            let bytes_end = match bytes_written {\n                PartialState::Partial(s) => s,\n                PartialState::Complete(c) => {\n                    // no more data will arrive\n                    if c == self.bytes_read {\n                        return None;\n                    }\n                    c\n                }\n            };\n            assert!(bytes_end >= self.bytes_read);\n\n            // more data available to read\n            if bytes_end > self.bytes_read {\n                let new_bytes =\n                    Bytes::copy_from_slice(&self.body.read()[self.bytes_read..bytes_end]);\n                self.bytes_read = bytes_end;\n                return Some(new_bytes);\n            }\n\n            // wait for more data\n            if self.bytes_written.changed().await.is_err() {\n                // err: sender dropped, body is finished\n                // FIXME: sender could drop because of an error\n                return None;\n            }\n        }\n    }\n}\n\n#[async_trait]\nimpl HandleHit for MemHitHandler {\n    async fn read_body(&mut self) -> Result<Option<Bytes>> {\n        match self {\n            Self::Complete(c) => Ok(c.get()),\n            Self::Partial(p) => Ok(p.read().await),\n        }\n    }\n    async fn finish(\n        self: Box<Self>, // because self is always used as a trait object\n        _storage: &'static (dyn storage::Storage + Sync),\n        _key: &CacheKey,\n        _trace: &SpanHandle,\n    ) -> Result<()> {\n        Ok(())\n    }\n\n    fn can_seek(&self) -> bool {\n        match self {\n            Self::Complete(_) => true,\n            Self::Partial(_) => false, // TODO: support seeking in partial reads\n        }\n    }\n\n    fn seek(&mut self, start: usize, end: Option<usize>) -> Result<()> {\n        match self {\n            Self::Complete(c) => c.seek(start, end),\n            Self::Partial(_) => Error::e_explain(\n                ErrorType::InternalError,\n                \"seek not supported for partial cache\",\n            ),\n        }\n    }\n\n    fn should_count_access(&self) -> bool {\n        match self {\n            // avoid counting accesses for partial reads to keep things simple\n            Self::Complete(_) => true,\n            Self::Partial(_) => false,\n        }\n    }\n\n    fn get_eviction_weight(&self) -> usize {\n        match self {\n            // FIXME: just body size, also track meta size\n            Self::Complete(c) => c.body.len(),\n            // partial read cannot be estimated since body size is unknown\n            Self::Partial(_) => 0,\n        }\n    }\n\n    fn as_any(&self) -> &(dyn Any + Send + Sync) {\n        self\n    }\n\n    fn as_any_mut(&mut self) -> &mut (dyn Any + Send + Sync) {\n        self\n    }\n}\n\npub struct MemMissHandler {\n    body: Arc<RwLock<Vec<u8>>>,\n    bytes_written: Arc<watch::Sender<PartialState>>,\n    // these are used only in finish() to data from temp to cache\n    key: String,\n    temp_id: U64WriteId,\n    // key -> cache object\n    cache: Arc<RwLock<HashMap<String, CacheObject>>>,\n    // key -> (temp writer id -> temp object) to support concurrent writers\n    temp: Arc<RwLock<HashMap<String, HashMap<u64, TempObject>>>>,\n}\n\n#[async_trait]\nimpl HandleMiss for MemMissHandler {\n    async fn write_body(&mut self, data: bytes::Bytes, eof: bool) -> Result<()> {\n        let current_bytes = match *self.bytes_written.borrow() {\n            PartialState::Partial(p) => p,\n            PartialState::Complete(_) => panic!(\"already EOF\"),\n        };\n        self.body.write().extend_from_slice(&data);\n        let written = current_bytes + data.len();\n        let new_state = if eof {\n            PartialState::Complete(written)\n        } else {\n            PartialState::Partial(written)\n        };\n        self.bytes_written.send_replace(new_state);\n        Ok(())\n    }\n\n    async fn finish(self: Box<Self>) -> Result<MissFinishType> {\n        // safe, the temp object is inserted when the miss handler is created\n        let cache_object = self\n            .temp\n            .read()\n            .get(&self.key)\n            .unwrap()\n            .get(&self.temp_id.into())\n            .unwrap()\n            .make_cache_object();\n        let size = cache_object.body.len(); // FIXME: this just body size, also track meta size\n        self.cache.write().insert(self.key.clone(), cache_object);\n        self.temp\n            .write()\n            .get_mut(&self.key)\n            .and_then(|map| map.remove(&self.temp_id.into()));\n        Ok(MissFinishType::Created(size))\n    }\n\n    fn streaming_write_tag(&self) -> Option<&[u8]> {\n        Some(self.temp_id.as_bytes())\n    }\n}\n\nimpl Drop for MemMissHandler {\n    fn drop(&mut self) {\n        self.temp\n            .write()\n            .get_mut(&self.key)\n            .and_then(|map| map.remove(&self.temp_id.into()));\n    }\n}\n\nfn hit_from_temp_obj(temp_obj: &TempObject) -> Result<Option<(CacheMeta, HitHandler)>> {\n    let meta = CacheMeta::deserialize(&temp_obj.meta.0, &temp_obj.meta.1)?;\n    let partial = PartialHit {\n        body: temp_obj.body.clone(),\n        bytes_written: temp_obj.bytes_written.subscribe(),\n        bytes_read: 0,\n    };\n    let hit_handler = MemHitHandler::Partial(partial);\n    Ok(Some((meta, Box::new(hit_handler))))\n}\n\n#[async_trait]\nimpl Storage for MemCache {\n    async fn lookup(\n        &'static self,\n        key: &CacheKey,\n        _trace: &SpanHandle,\n    ) -> Result<Option<(CacheMeta, HitHandler)>> {\n        let hash = key.combined();\n        // always prefer partial read otherwise fresh asset will not be visible on expired asset\n        // until it is fully updated\n        // no preference on which partial read we get (if there are multiple writers)\n        if let Some((_, temp_obj)) = self\n            .temp\n            .read()\n            .get(&hash)\n            .and_then(|map| map.iter().next())\n        {\n            hit_from_temp_obj(temp_obj)\n        } else if let Some(obj) = self.cached.read().get(&hash) {\n            let meta = CacheMeta::deserialize(&obj.meta.0, &obj.meta.1)?;\n            let hit_handler = CompleteHit {\n                body: obj.body.clone(),\n                done: false,\n                range_start: 0,\n                range_end: obj.body.len(),\n            };\n            let hit_handler = MemHitHandler::Complete(hit_handler);\n            Ok(Some((meta, Box::new(hit_handler))))\n        } else {\n            Ok(None)\n        }\n    }\n\n    async fn lookup_streaming_write(\n        &'static self,\n        key: &CacheKey,\n        streaming_write_tag: Option<&[u8]>,\n        _trace: &SpanHandle,\n    ) -> Result<Option<(CacheMeta, HitHandler)>> {\n        let hash = key.combined();\n        let write_tag: U64WriteId = streaming_write_tag\n            .expect(\"tag must be set during streaming write\")\n            .try_into()\n            .expect(\"tag must be correct length\");\n        hit_from_temp_obj(\n            self.temp\n                .read()\n                .get(&hash)\n                .and_then(|map| map.get(&write_tag.into()))\n                .expect(\"must have partial write in progress\"),\n        )\n    }\n\n    async fn get_miss_handler(\n        &'static self,\n        key: &CacheKey,\n        meta: &CacheMeta,\n        _trace: &SpanHandle,\n    ) -> Result<MissHandler> {\n        let hash = key.combined();\n        let meta = meta.serialize()?;\n        let temp_obj = TempObject::new(meta);\n        let temp_id = self.last_temp_id.fetch_add(1, Ordering::Relaxed);\n        let miss_handler = MemMissHandler {\n            body: temp_obj.body.clone(),\n            bytes_written: temp_obj.bytes_written.clone(),\n            key: hash.clone(),\n            cache: self.cached.clone(),\n            temp: self.temp.clone(),\n            temp_id: temp_id.into(),\n        };\n        self.temp\n            .write()\n            .entry(hash)\n            .or_default()\n            .insert(miss_handler.temp_id.into(), temp_obj);\n        Ok(Box::new(miss_handler))\n    }\n\n    async fn purge(\n        &'static self,\n        key: &CompactCacheKey,\n        _type: PurgeType,\n        _trace: &SpanHandle,\n    ) -> Result<bool> {\n        // This usually purges the primary key because, without a lookup, the variance key is usually\n        // empty\n        let hash = key.combined();\n        let temp_removed = self.temp.write().remove(&hash).is_some();\n        let cache_removed = self.cached.write().remove(&hash).is_some();\n        Ok(temp_removed || cache_removed)\n    }\n\n    async fn update_meta(\n        &'static self,\n        key: &CacheKey,\n        meta: &CacheMeta,\n        _trace: &SpanHandle,\n    ) -> Result<bool> {\n        let hash = key.combined();\n        if let Some(obj) = self.cached.write().get_mut(&hash) {\n            obj.meta = meta.serialize()?;\n            Ok(true)\n        } else {\n            panic!(\"no meta found\")\n        }\n    }\n\n    fn support_streaming_partial_write(&self) -> bool {\n        true\n    }\n\n    fn as_any(&self) -> &(dyn Any + Send + Sync) {\n        self\n    }\n}\n\n#[cfg(test)]\nmod test {\n    use super::*;\n    use cf_rustracing::span::Span;\n    use once_cell::sync::Lazy;\n\n    fn gen_meta() -> CacheMeta {\n        let mut header = ResponseHeader::build(200, None).unwrap();\n        header.append_header(\"foo1\", \"bar1\").unwrap();\n        header.append_header(\"foo2\", \"bar2\").unwrap();\n        header.append_header(\"foo3\", \"bar3\").unwrap();\n        header.append_header(\"Server\", \"Pingora\").unwrap();\n        let internal = crate::meta::InternalMeta::default();\n        CacheMeta(Box::new(crate::meta::CacheMetaInner {\n            internal,\n            header,\n            extensions: http::Extensions::new(),\n        }))\n    }\n\n    #[tokio::test]\n    async fn test_write_then_read() {\n        static MEM_CACHE: Lazy<MemCache> = Lazy::new(MemCache::new);\n        let span = &Span::inactive().handle();\n\n        let key1 = CacheKey::new(\"\", \"a\", \"1\");\n        let res = MEM_CACHE.lookup(&key1, span).await.unwrap();\n        assert!(res.is_none());\n\n        let cache_meta = gen_meta();\n\n        let mut miss_handler = MEM_CACHE\n            .get_miss_handler(&key1, &cache_meta, span)\n            .await\n            .unwrap();\n        miss_handler\n            .write_body(b\"test1\"[..].into(), false)\n            .await\n            .unwrap();\n        miss_handler\n            .write_body(b\"test2\"[..].into(), false)\n            .await\n            .unwrap();\n        miss_handler.finish().await.unwrap();\n\n        let (cache_meta2, mut hit_handler) = MEM_CACHE.lookup(&key1, span).await.unwrap().unwrap();\n        assert_eq!(\n            cache_meta.0.internal.fresh_until,\n            cache_meta2.0.internal.fresh_until\n        );\n\n        let data = hit_handler.read_body().await.unwrap().unwrap();\n        assert_eq!(\"test1test2\", data);\n        let data = hit_handler.read_body().await.unwrap();\n        assert!(data.is_none());\n    }\n\n    #[tokio::test]\n    async fn test_read_range() {\n        static MEM_CACHE: Lazy<MemCache> = Lazy::new(MemCache::new);\n        let span = &Span::inactive().handle();\n\n        let key1 = CacheKey::new(\"\", \"a\", \"1\");\n        let res = MEM_CACHE.lookup(&key1, span).await.unwrap();\n        assert!(res.is_none());\n\n        let cache_meta = gen_meta();\n\n        let mut miss_handler = MEM_CACHE\n            .get_miss_handler(&key1, &cache_meta, span)\n            .await\n            .unwrap();\n        miss_handler\n            .write_body(b\"test1test2\"[..].into(), false)\n            .await\n            .unwrap();\n        miss_handler.finish().await.unwrap();\n\n        let (cache_meta2, mut hit_handler) = MEM_CACHE.lookup(&key1, span).await.unwrap().unwrap();\n        assert_eq!(\n            cache_meta.0.internal.fresh_until,\n            cache_meta2.0.internal.fresh_until\n        );\n\n        // out of range\n        assert!(hit_handler.seek(10000, None).is_err());\n\n        assert!(hit_handler.seek(5, None).is_ok());\n        let data = hit_handler.read_body().await.unwrap().unwrap();\n        assert_eq!(\"test2\", data);\n        let data = hit_handler.read_body().await.unwrap();\n        assert!(data.is_none());\n\n        assert!(hit_handler.seek(4, Some(5)).is_ok());\n        let data = hit_handler.read_body().await.unwrap().unwrap();\n        assert_eq!(\"1\", data);\n        let data = hit_handler.read_body().await.unwrap();\n        assert!(data.is_none());\n    }\n\n    #[tokio::test]\n    async fn test_write_while_read() {\n        use futures::FutureExt;\n\n        static MEM_CACHE: Lazy<MemCache> = Lazy::new(MemCache::new);\n        let span = &Span::inactive().handle();\n\n        let key1 = CacheKey::new(\"\", \"a\", \"1\");\n        let res = MEM_CACHE.lookup(&key1, span).await.unwrap();\n        assert!(res.is_none());\n\n        let cache_meta = gen_meta();\n\n        let mut miss_handler = MEM_CACHE\n            .get_miss_handler(&key1, &cache_meta, span)\n            .await\n            .unwrap();\n\n        // first reader\n        let (cache_meta1, mut hit_handler1) = MEM_CACHE.lookup(&key1, span).await.unwrap().unwrap();\n        assert_eq!(\n            cache_meta.0.internal.fresh_until,\n            cache_meta1.0.internal.fresh_until\n        );\n\n        // No body to read\n        let res = hit_handler1.read_body().now_or_never();\n        assert!(res.is_none());\n\n        miss_handler\n            .write_body(b\"test1\"[..].into(), false)\n            .await\n            .unwrap();\n\n        let data = hit_handler1.read_body().await.unwrap().unwrap();\n        assert_eq!(\"test1\", data);\n        let res = hit_handler1.read_body().now_or_never();\n        assert!(res.is_none());\n\n        miss_handler\n            .write_body(b\"test2\"[..].into(), false)\n            .await\n            .unwrap();\n        let data = hit_handler1.read_body().await.unwrap().unwrap();\n        assert_eq!(\"test2\", data);\n\n        // second reader\n        let (cache_meta2, mut hit_handler2) = MEM_CACHE.lookup(&key1, span).await.unwrap().unwrap();\n        assert_eq!(\n            cache_meta.0.internal.fresh_until,\n            cache_meta2.0.internal.fresh_until\n        );\n\n        let data = hit_handler2.read_body().await.unwrap().unwrap();\n        assert_eq!(\"test1test2\", data);\n        let res = hit_handler2.read_body().now_or_never();\n        assert!(res.is_none());\n\n        let res = hit_handler1.read_body().now_or_never();\n        assert!(res.is_none());\n\n        miss_handler.finish().await.unwrap();\n\n        let data = hit_handler1.read_body().await.unwrap();\n        assert!(data.is_none());\n        let data = hit_handler2.read_body().await.unwrap();\n        assert!(data.is_none());\n    }\n\n    #[tokio::test]\n    async fn test_purge_partial() {\n        static MEM_CACHE: Lazy<MemCache> = Lazy::new(MemCache::new);\n        let cache = &MEM_CACHE;\n\n        let key = CacheKey::new(\"\", \"a\", \"1\").to_compact();\n        let hash = key.combined();\n        let meta = (\n            \"meta_key\".as_bytes().to_vec(),\n            \"meta_value\".as_bytes().to_vec(),\n        );\n\n        let temp_obj = TempObject::new(meta);\n        let mut map = HashMap::new();\n        map.insert(0, temp_obj);\n        cache.temp.write().insert(hash.clone(), map);\n\n        assert!(cache.temp.read().contains_key(&hash));\n\n        let result = cache\n            .purge(&key, PurgeType::Invalidation, &Span::inactive().handle())\n            .await;\n        assert!(result.is_ok());\n\n        assert!(!cache.temp.read().contains_key(&hash));\n    }\n\n    #[tokio::test]\n    async fn test_purge_complete() {\n        static MEM_CACHE: Lazy<MemCache> = Lazy::new(MemCache::new);\n        let cache = &MEM_CACHE;\n\n        let key = CacheKey::new(\"\", \"a\", \"1\").to_compact();\n        let hash = key.combined();\n        let meta = (\n            \"meta_key\".as_bytes().to_vec(),\n            \"meta_value\".as_bytes().to_vec(),\n        );\n        let body = vec![1, 2, 3, 4, 5, 6, 7, 8, 9, 0];\n        let cache_obj = CacheObject {\n            meta,\n            body: Arc::new(body),\n        };\n        cache.cached.write().insert(hash.clone(), cache_obj);\n\n        assert!(cache.cached.read().contains_key(&hash));\n\n        let result = cache\n            .purge(&key, PurgeType::Invalidation, &Span::inactive().handle())\n            .await;\n        assert!(result.is_ok());\n\n        assert!(!cache.cached.read().contains_key(&hash));\n    }\n}\n"
  },
  {
    "path": "pingora-cache/src/meta.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! Metadata for caching\n\npub use http::Extensions;\nuse log::warn;\nuse once_cell::sync::{Lazy, OnceCell};\nuse pingora_error::{Error, ErrorType::*, OrErr, Result};\nuse pingora_header_serde::HeaderSerde;\nuse pingora_http::{HMap, ResponseHeader};\nuse serde::{Deserialize, Serialize};\nuse std::borrow::Cow;\nuse std::time::{Duration, SystemTime};\n\nuse crate::key::HashBinary;\n\npub(crate) type InternalMeta = internal_meta::InternalMetaLatest;\nmod internal_meta {\n    use super::*;\n\n    pub(crate) type InternalMetaLatest = InternalMetaV2;\n\n    #[derive(Debug, Deserialize, Serialize, Clone)]\n    pub(crate) struct InternalMetaV0 {\n        pub(crate) fresh_until: SystemTime,\n        pub(crate) created: SystemTime,\n        pub(crate) stale_while_revalidate_sec: u32,\n        pub(crate) stale_if_error_sec: u32,\n        // Do not add more field\n    }\n\n    impl InternalMetaV0 {\n        #[allow(dead_code)]\n        fn serialize(&self) -> Result<Vec<u8>> {\n            rmp_serde::encode::to_vec(self).or_err(InternalError, \"failed to encode cache meta\")\n        }\n\n        fn deserialize(buf: &[u8]) -> Result<Self> {\n            rmp_serde::decode::from_slice(buf)\n                .or_err(InternalError, \"failed to decode cache meta v0\")\n        }\n    }\n\n    #[derive(Debug, Deserialize, Serialize, Clone)]\n    pub(crate) struct InternalMetaV1 {\n        pub(crate) version: u8,\n        pub(crate) fresh_until: SystemTime,\n        pub(crate) created: SystemTime,\n        pub(crate) stale_while_revalidate_sec: u32,\n        pub(crate) stale_if_error_sec: u32,\n        // Do not add more field\n    }\n\n    impl InternalMetaV1 {\n        #[allow(dead_code)]\n        pub const VERSION: u8 = 1;\n\n        #[allow(dead_code)]\n        pub fn serialize(&self) -> Result<Vec<u8>> {\n            assert_eq!(self.version, 1);\n            rmp_serde::encode::to_vec(self).or_err(InternalError, \"failed to encode cache meta\")\n        }\n\n        fn deserialize(buf: &[u8]) -> Result<Self> {\n            rmp_serde::decode::from_slice(buf)\n                .or_err(InternalError, \"failed to decode cache meta v1\")\n        }\n    }\n\n    #[derive(Debug, Deserialize, Serialize, Clone)]\n    pub(crate) struct InternalMetaV2 {\n        pub(crate) version: u8,\n        pub(crate) fresh_until: SystemTime,\n        pub(crate) created: SystemTime,\n        pub(crate) updated: SystemTime,\n        pub(crate) stale_while_revalidate_sec: u32,\n        pub(crate) stale_if_error_sec: u32,\n        // Only the extended field to be added below. One field at a time.\n        // 1. serde default in order to accept an older version schema without the field existing\n        // 2. serde skip_serializing_if in order for software with only an older version of this\n        //    schema to decode it\n        // After full releases, remove `skip_serializing_if` so that we can add the next extended field.\n        #[serde(default)]\n        pub(crate) variance: Option<HashBinary>,\n        #[serde(default)]\n        #[serde(skip_serializing_if = \"Option::is_none\")]\n        pub(crate) epoch_override: Option<SystemTime>,\n    }\n\n    impl Default for InternalMetaV2 {\n        fn default() -> Self {\n            let epoch = SystemTime::UNIX_EPOCH;\n            InternalMetaV2 {\n                version: InternalMetaV2::VERSION,\n                fresh_until: epoch,\n                created: epoch,\n                updated: epoch,\n                stale_while_revalidate_sec: 0,\n                stale_if_error_sec: 0,\n                variance: None,\n                epoch_override: None,\n            }\n        }\n    }\n\n    impl InternalMetaV2 {\n        pub const VERSION: u8 = 2;\n\n        pub fn serialize(&self) -> Result<Vec<u8>> {\n            assert_eq!(self.version, Self::VERSION);\n            rmp_serde::encode::to_vec(self).or_err(InternalError, \"failed to encode cache meta\")\n        }\n\n        fn deserialize(buf: &[u8]) -> Result<Self> {\n            rmp_serde::decode::from_slice(buf)\n                .or_err(InternalError, \"failed to decode cache meta v2\")\n        }\n    }\n\n    impl From<InternalMetaV0> for InternalMetaV2 {\n        fn from(v0: InternalMetaV0) -> Self {\n            InternalMetaV2 {\n                version: InternalMetaV2::VERSION,\n                fresh_until: v0.fresh_until,\n                created: v0.created,\n                updated: v0.created,\n                stale_while_revalidate_sec: v0.stale_while_revalidate_sec,\n                stale_if_error_sec: v0.stale_if_error_sec,\n                ..Default::default()\n            }\n        }\n    }\n\n    impl From<InternalMetaV1> for InternalMetaV2 {\n        fn from(v1: InternalMetaV1) -> Self {\n            InternalMetaV2 {\n                version: InternalMetaV2::VERSION,\n                fresh_until: v1.fresh_until,\n                created: v1.created,\n                updated: v1.created,\n                stale_while_revalidate_sec: v1.stale_while_revalidate_sec,\n                stale_if_error_sec: v1.stale_if_error_sec,\n                ..Default::default()\n            }\n        }\n    }\n\n    // cross version decode\n    pub(crate) fn deserialize(buf: &[u8]) -> Result<InternalMetaLatest> {\n        const MIN_SIZE: usize = 10; // a small number to read the first few bytes\n        if buf.len() < MIN_SIZE {\n            return Error::e_explain(\n                InternalError,\n                format!(\"Buf too short ({}) to be InternalMeta\", buf.len()),\n            );\n        }\n        let preread_buf = &mut &buf[..MIN_SIZE];\n        // the struct is always packed as a fixed size array\n        match rmp::decode::read_array_len(preread_buf)\n            .or_err(InternalError, \"failed to decode cache meta array size\")?\n        {\n            // v0 has 4 items and no version number\n            4 => Ok(InternalMetaV0::deserialize(buf)?.into()),\n            // other V should have version number encoded\n            _ => {\n                // rmp will encode `version` < 128 into a fixint (one byte),\n                // so we use read_pfix\n                let version = rmp::decode::read_pfix(preread_buf)\n                    .or_err(InternalError, \"failed to decode meta version\")?;\n                match version {\n                    1 => Ok(InternalMetaV1::deserialize(buf)?.into()),\n                    2 => InternalMetaV2::deserialize(buf),\n                    _ => Error::e_explain(\n                        InternalError,\n                        format!(\"Unknown InternalMeta version {version}\"),\n                    ),\n                }\n            }\n        }\n    }\n\n    #[cfg(test)]\n    mod tests {\n        use super::*;\n\n        #[test]\n        fn test_internal_meta_serde_v0() {\n            let meta = InternalMetaV0 {\n                fresh_until: SystemTime::now(),\n                created: SystemTime::now(),\n                stale_while_revalidate_sec: 0,\n                stale_if_error_sec: 0,\n            };\n            let binary = meta.serialize().unwrap();\n            let meta2 = InternalMetaV0::deserialize(&binary).unwrap();\n            assert_eq!(meta.fresh_until, meta2.fresh_until);\n        }\n\n        #[test]\n        fn test_internal_meta_serde_v1() {\n            let meta = InternalMetaV1 {\n                version: InternalMetaV1::VERSION,\n                fresh_until: SystemTime::now(),\n                created: SystemTime::now(),\n                stale_while_revalidate_sec: 0,\n                stale_if_error_sec: 0,\n            };\n            let binary = meta.serialize().unwrap();\n            let meta2 = InternalMetaV1::deserialize(&binary).unwrap();\n            assert_eq!(meta.fresh_until, meta2.fresh_until);\n        }\n\n        #[test]\n        fn test_internal_meta_serde_v2() {\n            let meta = InternalMetaV2::default();\n            let binary = meta.serialize().unwrap();\n            let meta2 = InternalMetaV2::deserialize(&binary).unwrap();\n            assert_eq!(meta2.version, 2);\n            assert_eq!(meta.fresh_until, meta2.fresh_until);\n            assert_eq!(meta.created, meta2.created);\n            assert_eq!(meta.updated, meta2.updated);\n        }\n\n        #[test]\n        fn test_internal_meta_serde_across_versions() {\n            let meta = InternalMetaV0 {\n                fresh_until: SystemTime::now(),\n                created: SystemTime::now(),\n                stale_while_revalidate_sec: 0,\n                stale_if_error_sec: 0,\n            };\n            let binary = meta.serialize().unwrap();\n            let meta2 = deserialize(&binary).unwrap();\n            assert_eq!(meta2.version, 2);\n            assert_eq!(meta.fresh_until, meta2.fresh_until);\n\n            let meta = InternalMetaV1 {\n                version: 1,\n                fresh_until: SystemTime::now(),\n                created: SystemTime::now(),\n                stale_while_revalidate_sec: 0,\n                stale_if_error_sec: 0,\n            };\n            let binary = meta.serialize().unwrap();\n            let meta2 = deserialize(&binary).unwrap();\n            assert_eq!(meta2.version, 2);\n            assert_eq!(meta.fresh_until, meta2.fresh_until);\n            // `updated` == `created` when upgrading to v2\n            assert_eq!(meta2.created, meta2.updated);\n        }\n\n        // make sure that v2 format is backward compatible\n        // this is the base version of v2 without any extended fields\n        #[derive(Deserialize, Serialize)]\n        struct InternalMetaV2Base {\n            version: u8,\n            fresh_until: SystemTime,\n            created: SystemTime,\n            updated: SystemTime,\n            stale_while_revalidate_sec: u32,\n            stale_if_error_sec: u32,\n        }\n\n        impl InternalMetaV2Base {\n            pub const VERSION: u8 = 2;\n            pub fn serialize(&self) -> Result<Vec<u8>> {\n                assert!(self.version >= Self::VERSION);\n                rmp_serde::encode::to_vec(self).or_err(InternalError, \"failed to encode cache meta\")\n            }\n            fn deserialize(buf: &[u8]) -> Result<Self> {\n                rmp_serde::decode::from_slice(buf)\n                    .or_err(InternalError, \"failed to decode cache meta v2\")\n            }\n        }\n\n        // this is the base version of v2 with variance but without epoch_override\n        #[derive(Deserialize, Serialize)]\n        struct InternalMetaV2BaseWithVariance {\n            version: u8,\n            fresh_until: SystemTime,\n            created: SystemTime,\n            updated: SystemTime,\n            stale_while_revalidate_sec: u32,\n            stale_if_error_sec: u32,\n            #[serde(default)]\n            #[serde(skip_serializing_if = \"Option::is_none\")]\n            variance: Option<HashBinary>,\n        }\n\n        impl Default for InternalMetaV2BaseWithVariance {\n            fn default() -> Self {\n                let epoch = SystemTime::UNIX_EPOCH;\n                InternalMetaV2BaseWithVariance {\n                    version: InternalMetaV2::VERSION,\n                    fresh_until: epoch,\n                    created: epoch,\n                    updated: epoch,\n                    stale_while_revalidate_sec: 0,\n                    stale_if_error_sec: 0,\n                    variance: None,\n                }\n            }\n        }\n\n        impl InternalMetaV2BaseWithVariance {\n            pub const VERSION: u8 = 2;\n            pub fn serialize(&self) -> Result<Vec<u8>> {\n                assert!(self.version >= Self::VERSION);\n                rmp_serde::encode::to_vec(self).or_err(InternalError, \"failed to encode cache meta\")\n            }\n            fn deserialize(buf: &[u8]) -> Result<Self> {\n                rmp_serde::decode::from_slice(buf)\n                    .or_err(InternalError, \"failed to decode cache meta v2\")\n            }\n        }\n\n        #[test]\n        fn test_internal_meta_serde_v2_extend_fields_variance() {\n            // ext V2 to base v2\n            let meta = InternalMetaV2BaseWithVariance::default();\n            let binary = meta.serialize().unwrap();\n            let meta2 = InternalMetaV2Base::deserialize(&binary).unwrap();\n            assert_eq!(meta2.version, 2);\n            assert_eq!(meta.fresh_until, meta2.fresh_until);\n            assert_eq!(meta.created, meta2.created);\n            assert_eq!(meta.updated, meta2.updated);\n\n            // base V2 to ext v2\n            let now = SystemTime::now();\n            let meta = InternalMetaV2Base {\n                version: InternalMetaV2::VERSION,\n                fresh_until: now,\n                created: now,\n                updated: now,\n                stale_while_revalidate_sec: 0,\n                stale_if_error_sec: 0,\n            };\n            let binary = meta.serialize().unwrap();\n            let meta2 = InternalMetaV2BaseWithVariance::deserialize(&binary).unwrap();\n            assert_eq!(meta2.version, 2);\n            assert_eq!(meta.fresh_until, meta2.fresh_until);\n            assert_eq!(meta.created, meta2.created);\n            assert_eq!(meta.updated, meta2.updated);\n        }\n\n        #[test]\n        fn test_internal_meta_serde_v2_extend_fields_epoch_override() {\n            let now = SystemTime::now();\n\n            // ext V2 (with epoch_override = None) to V2 with variance (without epoch_override field)\n            let meta = InternalMetaV2 {\n                fresh_until: now,\n                created: now,\n                updated: now,\n                epoch_override: None, // None means it will be skipped during serialization\n                ..Default::default()\n            };\n            let binary = meta.serialize().unwrap();\n            let meta2 = InternalMetaV2BaseWithVariance::deserialize(&binary).unwrap();\n            assert_eq!(meta2.version, 2);\n            assert_eq!(meta.fresh_until, meta2.fresh_until);\n            assert_eq!(meta.created, meta2.created);\n            assert_eq!(meta.updated, meta2.updated);\n            assert!(meta2.variance.is_none());\n\n            // V2 base with variance (without epoch_override) to ext V2 (with epoch_override)\n            let mut meta = InternalMetaV2BaseWithVariance {\n                version: InternalMetaV2::VERSION,\n                fresh_until: now,\n                created: now,\n                updated: now,\n                stale_while_revalidate_sec: 0,\n                stale_if_error_sec: 0,\n                variance: None,\n            };\n            let binary = meta.serialize().unwrap();\n            let meta2 = InternalMetaV2::deserialize(&binary).unwrap();\n            assert_eq!(meta2.version, 2);\n            assert_eq!(meta.fresh_until, meta2.fresh_until);\n            assert_eq!(meta.created, meta2.created);\n            assert_eq!(meta.updated, meta2.updated);\n            assert!(meta2.variance.is_none());\n            assert!(meta2.epoch_override.is_none());\n\n            // try with variance set\n            meta.variance = Some(*b\"variance_testing\");\n            let binary = meta.serialize().unwrap();\n            let meta2 = InternalMetaV2::deserialize(&binary).unwrap();\n            assert_eq!(meta2.version, 2);\n            assert_eq!(meta.fresh_until, meta2.fresh_until);\n            assert_eq!(meta.created, meta2.created);\n            assert_eq!(meta.updated, meta2.updated);\n            assert_eq!(meta.variance, meta2.variance);\n            assert!(meta2.epoch_override.is_none());\n        }\n    }\n}\n\n#[derive(Debug)]\npub(crate) struct CacheMetaInner {\n    // http header and Internal meta have different ways of serialization, so keep them separated\n    pub(crate) internal: InternalMeta,\n    pub(crate) header: ResponseHeader,\n    /// An opaque type map to hold extra information for communication between cache backends\n    /// and users. This field is **not** guaranteed be persistently stored in the cache backend.\n    pub extensions: Extensions,\n}\n\n/// The cacheable response header and cache metadata\n#[derive(Debug)]\npub struct CacheMeta(pub(crate) Box<CacheMetaInner>);\n\nimpl CacheMeta {\n    /// Create a [CacheMeta] from the given metadata and the response header\n    pub fn new(\n        fresh_until: SystemTime,\n        created: SystemTime,\n        stale_while_revalidate_sec: u32,\n        stale_if_error_sec: u32,\n        header: ResponseHeader,\n    ) -> CacheMeta {\n        CacheMeta(Box::new(CacheMetaInner {\n            internal: InternalMeta {\n                version: InternalMeta::VERSION,\n                fresh_until,\n                created,\n                updated: created, // created == updated for new meta\n                stale_while_revalidate_sec,\n                stale_if_error_sec,\n                ..Default::default()\n            },\n            header,\n            extensions: Extensions::new(),\n        }))\n    }\n\n    /// When the asset was created/admitted to cache\n    pub fn created(&self) -> SystemTime {\n        self.0.internal.created\n    }\n\n    /// The last time the asset was revalidated\n    ///\n    /// This value will be the same as [Self::created()] if no revalidation ever happens\n    pub fn updated(&self) -> SystemTime {\n        self.0.internal.updated\n    }\n\n    /// The reference point for cache age. This represents the \"starting point\" for `fresh_until`.\n    ///\n    /// This defaults to the `updated` timestamp but is overridden by the `epoch_override` field\n    /// if set.\n    pub fn epoch(&self) -> SystemTime {\n        self.0.internal.epoch_override.unwrap_or(self.updated())\n    }\n\n    /// Get the epoch override for this asset\n    pub fn epoch_override(&self) -> Option<SystemTime> {\n        self.0.internal.epoch_override\n    }\n\n    /// Set the epoch override for this asset\n    ///\n    /// When set, this will be used as the reference point for calculating age and freshness\n    /// instead of the updated time.\n    pub fn set_epoch_override(&mut self, epoch: SystemTime) {\n        self.0.internal.epoch_override = Some(epoch);\n    }\n\n    /// Remove the epoch override for this asset\n    pub fn remove_epoch_override(&mut self) {\n        self.0.internal.epoch_override = None;\n    }\n\n    /// Is the asset still valid\n    pub fn is_fresh(&self, time: SystemTime) -> bool {\n        // NOTE: HTTP cache time resolution is second\n        self.0.internal.fresh_until >= time\n    }\n\n    /// How long (in seconds) the asset should be fresh since its admission/revalidation\n    ///\n    /// This is essentially the max-age value (or its equivalence).\n    /// If an epoch override is set, it will be used as the reference point instead of the updated time.\n    pub fn fresh_sec(&self) -> u64 {\n        // swallow `duration_since` error, assets that are always stale have earlier `fresh_until` than `created`\n        // practically speaking we can always treat these as 0 ttl\n        // XXX: return Error if `fresh_until` is much earlier than expected?\n        let reference = self.epoch();\n        self.0\n            .internal\n            .fresh_until\n            .duration_since(reference)\n            .map_or(0, |duration| duration.as_secs())\n    }\n\n    /// Until when the asset is considered fresh\n    pub fn fresh_until(&self) -> SystemTime {\n        self.0.internal.fresh_until\n    }\n\n    /// How old the asset is since its admission/revalidation\n    ///\n    /// If an epoch override is set, it will be used as the reference point instead of the updated time.\n    pub fn age(&self) -> Duration {\n        let reference = self.epoch();\n        SystemTime::now()\n            .duration_since(reference)\n            .unwrap_or_default()\n    }\n\n    /// The stale-while-revalidate limit in seconds\n    pub fn stale_while_revalidate_sec(&self) -> u32 {\n        self.0.internal.stale_while_revalidate_sec\n    }\n\n    /// The stale-if-error limit in seconds\n    pub fn stale_if_error_sec(&self) -> u32 {\n        self.0.internal.stale_if_error_sec\n    }\n\n    /// Can the asset be used to serve stale during revalidation at the given time.\n    ///\n    /// NOTE: the serve stale functions do not check !is_fresh(time),\n    /// i.e. the object is already assumed to be stale.\n    pub fn serve_stale_while_revalidate(&self, time: SystemTime) -> bool {\n        self.can_serve_stale(self.0.internal.stale_while_revalidate_sec, time)\n    }\n\n    /// Can the asset be used to serve stale after error at the given time.\n    ///\n    /// NOTE: the serve stale functions do not check !is_fresh(time),\n    /// i.e. the object is already assumed to be stale.\n    pub fn serve_stale_if_error(&self, time: SystemTime) -> bool {\n        self.can_serve_stale(self.0.internal.stale_if_error_sec, time)\n    }\n\n    /// Disable serve stale for this asset\n    pub fn disable_serve_stale(&mut self) {\n        self.0.internal.stale_if_error_sec = 0;\n        self.0.internal.stale_while_revalidate_sec = 0;\n    }\n\n    /// Get the variance hash of this asset\n    pub fn variance(&self) -> Option<HashBinary> {\n        self.0.internal.variance\n    }\n\n    /// Set the variance key of this asset\n    pub fn set_variance_key(&mut self, variance_key: HashBinary) {\n        self.0.internal.variance = Some(variance_key);\n    }\n\n    /// Set the variance (hash) of this asset\n    pub fn set_variance(&mut self, variance: HashBinary) {\n        self.0.internal.variance = Some(variance)\n    }\n\n    /// Removes the variance (hash) of this asset\n    pub fn remove_variance(&mut self) {\n        self.0.internal.variance = None\n    }\n\n    /// Get the response header in this asset\n    pub fn response_header(&self) -> &ResponseHeader {\n        &self.0.header\n    }\n\n    /// Modify the header in this asset\n    pub fn response_header_mut(&mut self) -> &mut ResponseHeader {\n        &mut self.0.header\n    }\n\n    /// Expose the extensions to read\n    pub fn extensions(&self) -> &Extensions {\n        &self.0.extensions\n    }\n\n    /// Expose the extensions to modify\n    pub fn extensions_mut(&mut self) -> &mut Extensions {\n        &mut self.0.extensions\n    }\n\n    /// Get a copy of the response header\n    pub fn response_header_copy(&self) -> ResponseHeader {\n        self.0.header.clone()\n    }\n\n    /// get all the headers of this asset\n    pub fn headers(&self) -> &HMap {\n        &self.0.header.headers\n    }\n\n    fn can_serve_stale(&self, serve_stale_sec: u32, time: SystemTime) -> bool {\n        if serve_stale_sec == 0 {\n            return false;\n        }\n        if let Some(stale_until) = self\n            .0\n            .internal\n            .fresh_until\n            .checked_add(Duration::from_secs(serve_stale_sec.into()))\n        {\n            stale_until >= time\n        } else {\n            // overflowed: treat as infinite ttl\n            true\n        }\n    }\n\n    /// Serialize this object\n    pub fn serialize(&self) -> Result<(Vec<u8>, Vec<u8>)> {\n        let internal = self.0.internal.serialize()?;\n        let header = header_serialize(&self.0.header)?;\n        log::debug!(\"header to serialize: {:?}\", &self.0.header);\n        Ok((internal, header))\n    }\n\n    /// Deserialize from the binary format\n    pub fn deserialize(internal: &[u8], header: &[u8]) -> Result<Self> {\n        let internal = internal_meta::deserialize(internal)?;\n        let header = header_deserialize(header)?;\n        Ok(CacheMeta(Box::new(CacheMetaInner {\n            internal,\n            header,\n            extensions: Extensions::new(),\n        })))\n    }\n}\n\nuse http::StatusCode;\n\n/// The function to generate TTL from the given [StatusCode].\npub type FreshDurationByStatusFn = fn(StatusCode) -> Option<Duration>;\n\n/// The default settings to generate [CacheMeta]\npub struct CacheMetaDefaults {\n    // if a status code is not included in fresh_sec, it's not considered cacheable by default.\n    fresh_sec_fn: FreshDurationByStatusFn,\n    stale_while_revalidate_sec: u32,\n    // TODO: allow \"error\" condition to be configurable?\n    stale_if_error_sec: u32,\n}\n\nimpl CacheMetaDefaults {\n    /// Create a new [CacheMetaDefaults]\n    pub const fn new(\n        fresh_sec_fn: FreshDurationByStatusFn,\n        stale_while_revalidate_sec: u32,\n        stale_if_error_sec: u32,\n    ) -> Self {\n        CacheMetaDefaults {\n            fresh_sec_fn,\n            stale_while_revalidate_sec,\n            stale_if_error_sec,\n        }\n    }\n\n    /// Return the default TTL for the given [StatusCode]\n    ///\n    /// `None`: do no cache this code.\n    pub fn fresh_sec(&self, resp_status: StatusCode) -> Option<Duration> {\n        // safe guard to make sure 304 response to share the same default ttl of 200\n        if resp_status == StatusCode::NOT_MODIFIED {\n            (self.fresh_sec_fn)(StatusCode::OK)\n        } else {\n            (self.fresh_sec_fn)(resp_status)\n        }\n    }\n\n    /// The default SWR seconds\n    pub fn serve_stale_while_revalidate_sec(&self) -> u32 {\n        self.stale_while_revalidate_sec\n    }\n\n    /// The default SIE seconds\n    pub fn serve_stale_if_error_sec(&self) -> u32 {\n        self.stale_if_error_sec\n    }\n}\n\n/// The dictionary content for header compression.\n///\n/// Used during initialization of [`HEADER_SERDE`].\nstatic COMPRESSION_DICT_CONTENT: OnceCell<Cow<'static, [u8]>> = OnceCell::new();\n\nstatic HEADER_SERDE: Lazy<HeaderSerde> = Lazy::new(|| {\n    let dict_opt = if let Some(dict_content) = COMPRESSION_DICT_CONTENT.get() {\n        Some(dict_content.to_vec())\n    } else {\n        warn!(\"no header compression dictionary loaded - use set_compression_dict_content() or set_compression_dict_path() to set one\");\n        None\n    };\n\n    HeaderSerde::new(dict_opt)\n});\n\npub(crate) fn header_serialize(header: &ResponseHeader) -> Result<Vec<u8>> {\n    HEADER_SERDE.serialize(header)\n}\n\npub(crate) fn header_deserialize<T: AsRef<[u8]>>(buf: T) -> Result<ResponseHeader> {\n    HEADER_SERDE.deserialize(buf.as_ref())\n}\n\n/// Load the header compression dictionary from a file, which helps serialize http header.\n///\n/// Returns false if it is already set or if the file could not be read.\n///\n/// Use [`set_compression_dict_content`] to set the dictionary from memory instead.\npub fn set_compression_dict_path(path: &str) -> bool {\n    match std::fs::read(path) {\n        Ok(dict) => COMPRESSION_DICT_CONTENT.set(dict.into()).is_ok(),\n        Err(e) => {\n            warn!(\n                \"failed to read header compress dictionary file at {}, {:?}\",\n                path, e\n            );\n            false\n        }\n    }\n}\n\n/// Set the header compression dictionary content, which helps serialize http header.\n///\n/// Returns false if it is already set.\n///\n/// This is an alernative to [`set_compression_dict_path`], allowing use of\n/// a dictionary without an external file.\npub fn set_compression_dict_content(content: Cow<'static, [u8]>) -> bool {\n    COMPRESSION_DICT_CONTENT.set(content).is_ok()\n}\n\n#[cfg(test)]\nmod tests {\n    use super::*;\n    use std::time::Duration;\n\n    #[test]\n    fn test_cache_meta_age_without_override() {\n        let now = SystemTime::now();\n        let header = ResponseHeader::build_no_case(200, None).unwrap();\n        let meta = CacheMeta::new(now + Duration::from_secs(300), now, 0, 0, header);\n\n        // Without epoch_override, age() should use updated() as reference\n        std::thread::sleep(Duration::from_millis(100));\n        let age = meta.age();\n        assert!(age.as_secs() < 1, \"age should be close to 0\");\n\n        // epoch() should return updated() when no override is set\n        assert_eq!(meta.epoch(), meta.updated());\n    }\n\n    #[test]\n    fn test_cache_meta_age_with_epoch_override_past() {\n        let now = SystemTime::now();\n        let header = ResponseHeader::build(200, None).unwrap();\n        let mut meta = CacheMeta::new(now + Duration::from_secs(300), now, 0, 0, header);\n\n        // Set epoch_override to 10 seconds in the past\n        let epoch_override = now - Duration::from_secs(10);\n        meta.set_epoch_override(epoch_override);\n\n        // age() should now use epoch_override as the reference\n        let age = meta.age();\n        assert!(age.as_secs() >= 10);\n        assert!(age.as_secs() < 12);\n\n        // epoch() should return the override\n        assert_eq!(meta.epoch(), epoch_override);\n        assert_eq!(meta.epoch_override(), Some(epoch_override));\n    }\n\n    #[test]\n    fn test_cache_meta_age_with_epoch_override_future() {\n        let now = SystemTime::now();\n        let header = ResponseHeader::build(200, None).unwrap();\n        let mut meta = CacheMeta::new(now + Duration::from_secs(100), now, 0, 0, header);\n\n        // Set epoch_override to a future time\n        let future_epoch = now + Duration::from_secs(10);\n        meta.set_epoch_override(future_epoch);\n\n        let age_with_epoch = meta.age();\n        // age should be 0 since epoch_override is in the future\n        assert_eq!(age_with_epoch, Duration::ZERO);\n    }\n\n    #[test]\n    fn test_cache_meta_fresh_sec() {\n        let header = ResponseHeader::build(StatusCode::OK, None).unwrap();\n        let mut meta = CacheMeta::new(\n            SystemTime::now() + Duration::from_secs(100),\n            SystemTime::now() - Duration::from_secs(100),\n            0,\n            0,\n            header,\n        );\n\n        meta.0.internal.updated = SystemTime::UNIX_EPOCH + Duration::from_secs(1000);\n        meta.0.internal.fresh_until = SystemTime::UNIX_EPOCH + Duration::from_secs(1100);\n\n        // Without epoch_override, fresh_sec should use updated as reference\n        let fresh_sec_without_override = meta.fresh_sec();\n        assert_eq!(fresh_sec_without_override, 100); // 1100 - 1000 = 100 seconds\n\n        // With epoch_override set to a later time (1050), fresh_sec should be calculated from that reference\n        let epoch_override = SystemTime::UNIX_EPOCH + Duration::from_secs(1050);\n        meta.set_epoch_override(epoch_override);\n        assert_eq!(meta.epoch_override(), Some(epoch_override));\n        assert_eq!(meta.epoch(), epoch_override);\n\n        let fresh_sec_with_override = meta.fresh_sec();\n        // fresh_until - epoch_override = 1100 - 1050 = 50 seconds\n        assert_eq!(fresh_sec_with_override, 50);\n\n        meta.remove_epoch_override();\n        assert_eq!(meta.epoch_override(), None);\n        assert_eq!(meta.epoch(), meta.updated());\n        assert_eq!(meta.fresh_sec(), 100); // back to normal calculation\n    }\n}\n"
  },
  {
    "path": "pingora-cache/src/predictor.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! Cacheability Predictor\n\nuse crate::hashtable::{ConcurrentLruCache, LruShard};\n\npub type CustomReasonPredicate = fn(&'static str) -> bool;\n\n/// Cacheability Predictor\n///\n/// Remembers previously uncacheable assets.\n/// Allows bypassing cache / cache lock early based on historical precedent.\n///\n/// NOTE: to simply avoid caching requests with certain characteristics,\n/// add checks in request_cache_filter to avoid enabling cache in the first place.\n/// The predictor's bypass mechanism handles cases where the request _looks_ cacheable\n/// but its previous responses suggest otherwise. The request _could_ be cacheable in the future.\npub struct Predictor<const N_SHARDS: usize> {\n    uncacheable_keys: ConcurrentLruCache<(), N_SHARDS>,\n    skip_custom_reasons_fn: Option<CustomReasonPredicate>,\n}\n\nuse crate::{key::CacheHashKey, CacheKey, NoCacheReason};\nuse log::debug;\n\n/// The cache predictor trait.\n///\n/// This trait allows user defined predictor to replace [Predictor].\npub trait CacheablePredictor {\n    /// Return true if likely cacheable, false if likely not.\n    fn cacheable_prediction(&self, key: &CacheKey) -> bool;\n\n    /// Mark cacheable to allow next request to cache.\n    /// Returns false if the key was already marked cacheable.\n    fn mark_cacheable(&self, key: &CacheKey) -> bool;\n\n    /// Mark uncacheable to actively bypass cache on the next request.\n    /// May skip marking on certain NoCacheReasons.\n    /// Returns None if we skipped marking uncacheable.\n    /// Returns Some(false) if the key was already marked uncacheable.\n    fn mark_uncacheable(&self, key: &CacheKey, reason: NoCacheReason) -> Option<bool>;\n}\n\n// This particular bit of `where [LruShard...; N]: Default` nonsense arises from\n// ConcurrentLruCache needing this trait bound, which in turns arises from the Rust\n// compiler not being able to guarantee that all array sizes N implement `Default`.\n// See https://github.com/rust-lang/rust/issues/61415\nimpl<const N_SHARDS: usize> Predictor<N_SHARDS>\nwhere\n    [LruShard<()>; N_SHARDS]: Default,\n{\n    /// Create a new Predictor with `N_SHARDS * shard_capacity` total capacity for\n    /// uncacheable cache keys.\n    ///\n    /// - `shard_capacity`: defines number of keys remembered as uncacheable per LRU shard.\n    /// - `skip_custom_reasons_fn`: an optional predicate used in `mark_uncacheable`\n    ///   that can customize which `Custom` `NoCacheReason`s ought to be remembered as uncacheable.\n    ///   If the predicate returns true, then the predictor will skip remembering the current\n    ///   cache key as uncacheable (and avoid bypassing cache on the next request).\n    pub fn new(\n        shard_capacity: usize,\n        skip_custom_reasons_fn: Option<CustomReasonPredicate>,\n    ) -> Predictor<N_SHARDS> {\n        Predictor {\n            uncacheable_keys: ConcurrentLruCache::<(), N_SHARDS>::new(shard_capacity),\n            skip_custom_reasons_fn,\n        }\n    }\n}\n\nimpl<const N_SHARDS: usize> CacheablePredictor for Predictor<N_SHARDS>\nwhere\n    [LruShard<()>; N_SHARDS]: Default,\n{\n    fn cacheable_prediction(&self, key: &CacheKey) -> bool {\n        // variance key is ignored because this check happens before cache lookup\n        let hash = key.primary_bin();\n        let key = u128::from_be_bytes(hash); // Endianness doesn't matter\n\n        // Note: LRU updated in mark_* functions only,\n        // as we assume the caller always updates the cacheability of the response later\n        !self.uncacheable_keys.read(key).contains(&key)\n    }\n\n    fn mark_cacheable(&self, key: &CacheKey) -> bool {\n        // variance key is ignored because cacheable_prediction() is called before cache lookup\n        // where the variance key is unknown\n        let hash = key.primary_bin();\n        let key = u128::from_be_bytes(hash);\n\n        let cache = self.uncacheable_keys.get(key);\n        if !cache.read().contains(&key) {\n            // not in uncacheable list, nothing to do\n            return true;\n        }\n\n        let mut cache = cache.write();\n        cache.pop(&key);\n        debug!(\"bypassed request became cacheable\");\n        false\n    }\n\n    fn mark_uncacheable(&self, key: &CacheKey, reason: NoCacheReason) -> Option<bool> {\n        // only mark as uncacheable for the future on certain reasons,\n        // (e.g. InternalErrors)\n        use NoCacheReason::*;\n        match reason {\n            // CacheLockGiveUp: the writer will set OriginNotCache (if applicable)\n            // readers don't need to do it\n            NeverEnabled\n            | StorageError\n            | InternalError\n            | Deferred\n            | CacheLockGiveUp\n            | CacheLockTimeout\n            | DeclinedToUpstream\n            | UpstreamError\n            | PredictedResponseTooLarge => {\n                return None;\n            }\n            // Skip certain NoCacheReason::Custom according to user\n            Custom(reason) if self.skip_custom_reasons_fn.is_some_and(|f| f(reason)) => {\n                return None;\n            }\n            Custom(_) | OriginNotCache | ResponseTooLarge => { /* mark uncacheable for these only */\n            }\n        }\n\n        // variance key is ignored because cacheable_prediction() is called before cache lookup\n        // where the variance key is unknown\n        let hash = key.primary_bin();\n        let key = u128::from_be_bytes(hash);\n\n        let mut cache = self.uncacheable_keys.get(key).write();\n        // put() returns Some(old_value) if the key existed, else None\n        let new_key = cache.put(key, ()).is_none();\n        if new_key {\n            debug!(\"request marked uncacheable\");\n        }\n        Some(new_key)\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use super::*;\n    #[test]\n    fn test_mark_cacheability() {\n        let predictor = Predictor::<1>::new(10, None);\n        let key = CacheKey::new(\"a\", \"b\", \"c\");\n        // cacheable if no history\n        assert!(predictor.cacheable_prediction(&key));\n\n        // don't remember internal / storage errors\n        predictor.mark_uncacheable(&key, NoCacheReason::InternalError);\n        assert!(predictor.cacheable_prediction(&key));\n        predictor.mark_uncacheable(&key, NoCacheReason::StorageError);\n        assert!(predictor.cacheable_prediction(&key));\n\n        // origin explicitly said uncacheable\n        predictor.mark_uncacheable(&key, NoCacheReason::OriginNotCache);\n        assert!(!predictor.cacheable_prediction(&key));\n\n        // mark cacheable again\n        predictor.mark_cacheable(&key);\n        assert!(predictor.cacheable_prediction(&key));\n    }\n\n    #[test]\n    fn test_custom_skip_predicate() {\n        let predictor = Predictor::<1>::new(\n            10,\n            Some(|custom_reason| matches!(custom_reason, \"Skipping\")),\n        );\n        let key = CacheKey::new(\"a\", \"b\", \"c\");\n        // cacheable if no history\n        assert!(predictor.cacheable_prediction(&key));\n\n        // custom predicate still uses default skip reasons\n        predictor.mark_uncacheable(&key, NoCacheReason::InternalError);\n        assert!(predictor.cacheable_prediction(&key));\n\n        // other custom reasons can still be marked uncacheable\n        predictor.mark_uncacheable(&key, NoCacheReason::Custom(\"DontCacheMe\"));\n        assert!(!predictor.cacheable_prediction(&key));\n\n        let key = CacheKey::new(\"a\", \"c\", \"d\");\n        assert!(predictor.cacheable_prediction(&key));\n        // specific custom reason is skipped\n        predictor.mark_uncacheable(&key, NoCacheReason::Custom(\"Skipping\"));\n        assert!(predictor.cacheable_prediction(&key));\n    }\n\n    #[test]\n    fn test_mark_uncacheable_lru() {\n        let predictor = Predictor::<1>::new(3, None);\n        let key1 = CacheKey::new(\"a\", \"b\", \"c\");\n        predictor.mark_uncacheable(&key1, NoCacheReason::OriginNotCache);\n        assert!(!predictor.cacheable_prediction(&key1));\n\n        let key2 = CacheKey::new(\"a\", \"bc\", \"c\");\n        predictor.mark_uncacheable(&key2, NoCacheReason::OriginNotCache);\n        assert!(!predictor.cacheable_prediction(&key2));\n\n        let key3 = CacheKey::new(\"a\", \"cd\", \"c\");\n        predictor.mark_uncacheable(&key3, NoCacheReason::OriginNotCache);\n        assert!(!predictor.cacheable_prediction(&key3));\n\n        // promote / reinsert key1\n        predictor.mark_uncacheable(&key1, NoCacheReason::OriginNotCache);\n\n        let key4 = CacheKey::new(\"a\", \"de\", \"c\");\n        predictor.mark_uncacheable(&key4, NoCacheReason::OriginNotCache);\n        assert!(!predictor.cacheable_prediction(&key4));\n\n        // key 1 was recently used\n        assert!(!predictor.cacheable_prediction(&key1));\n        // key 2 was evicted\n        assert!(predictor.cacheable_prediction(&key2));\n        assert!(!predictor.cacheable_prediction(&key3));\n        assert!(!predictor.cacheable_prediction(&key4));\n    }\n}\n"
  },
  {
    "path": "pingora-cache/src/put.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! Cache Put module\n\nuse crate::max_file_size::ERR_RESPONSE_TOO_LARGE;\nuse crate::*;\nuse bytes::Bytes;\nuse http::header;\nuse log::warn;\nuse pingora_core::protocols::http::{\n    v1::common::header_value_content_length, HttpTask, ServerSession,\n};\nuse pingora_error::Error;\n\n/// The interface to define cache put behavior\npub trait CachePut {\n    /// Return whether to cache the asset according to the given response header.\n    fn cacheable(&self, response: ResponseHeader) -> RespCacheable {\n        let cc = cache_control::CacheControl::from_resp_headers(&response);\n        filters::resp_cacheable(cc.as_ref(), response, false, Self::cache_defaults())\n    }\n\n    /// Return the [CacheMetaDefaults]\n    fn cache_defaults() -> &'static CacheMetaDefaults;\n\n    /// Put interesting things in the span given the parsed response header.\n    fn trace_header(&mut self, _response: &ResponseHeader) {}\n}\n\nuse parse_response::ResponseParse;\n\n/// The cache put context\npub struct CachePutCtx<C: CachePut> {\n    cache_put: C, // the user defined cache put behavior\n    key: CacheKey,\n    storage: &'static (dyn storage::Storage + Sync), // static for now\n    eviction: Option<&'static (dyn eviction::EvictionManager + Sync)>,\n    miss_handler: Option<MissHandler>,\n    max_file_size_tracker: Option<MaxFileSizeTracker>,\n    meta: Option<CacheMeta>,\n    parser: ResponseParse,\n    // FIXME: cache put doesn't have cache lock but some storage cannot handle concurrent put\n    // to the same asset.\n    trace: trace::Span,\n}\n\nimpl<C: CachePut> CachePutCtx<C> {\n    /// Create a new [CachePutCtx]\n    pub fn new(\n        cache_put: C,\n        key: CacheKey,\n        storage: &'static (dyn storage::Storage + Sync),\n        eviction: Option<&'static (dyn eviction::EvictionManager + Sync)>,\n        trace: trace::Span,\n    ) -> Self {\n        CachePutCtx {\n            cache_put,\n            key,\n            storage,\n            eviction,\n            miss_handler: None,\n            max_file_size_tracker: None,\n            meta: None,\n            parser: ResponseParse::new(),\n            trace,\n        }\n    }\n\n    /// Set the max cacheable size limit\n    pub fn set_max_file_size_bytes(&mut self, max_file_size_bytes: usize) {\n        self.max_file_size_tracker = Some(MaxFileSizeTracker::new(max_file_size_bytes));\n    }\n\n    async fn put_header(&mut self, meta: CacheMeta) -> Result<()> {\n        let mut trace = self.trace.child(\"cache put header\", |o| o.start());\n        let miss_handler = self\n            .storage\n            .get_miss_handler(&self.key, &meta, &trace.handle())\n            .await?;\n        trace::tag_span_with_meta(&mut trace, &meta);\n        self.miss_handler = Some(miss_handler);\n        self.meta = Some(meta);\n        Ok(())\n    }\n\n    async fn put_body(&mut self, data: Bytes, eof: bool) -> Result<()> {\n        // fail if writing the body would exceed the max_file_size_bytes\n        if let Some(size_tracker) = self.max_file_size_tracker.as_mut() {\n            let body_size_allowed = size_tracker.add_body_bytes(data.len());\n            if !body_size_allowed {\n                return Error::e_explain(\n                    ERR_RESPONSE_TOO_LARGE,\n                    format!(\n                        \"writing data of size {} bytes would exceed max file size of {} bytes\",\n                        data.len(),\n                        size_tracker.max_file_size_bytes(),\n                    ),\n                );\n            }\n        }\n\n        let miss_handler = self.miss_handler.as_mut().unwrap();\n        miss_handler.write_body(data, eof).await\n    }\n\n    async fn finish(&mut self) -> Result<()> {\n        let Some(miss_handler) = self.miss_handler.take() else {\n            // no miss_handler, uncacheable\n            return Ok(());\n        };\n        let finish = miss_handler.finish().await?;\n        if let Some(eviction) = self.eviction.as_ref() {\n            let cache_key = self.key.to_compact();\n            let meta = self.meta.as_ref().unwrap();\n            let evicted = match finish {\n                MissFinishType::Appended(delta, max_size) => {\n                    eviction.increment_weight(&cache_key, delta, max_size)\n                }\n                MissFinishType::Created(size) => {\n                    eviction.admit(cache_key, size, meta.0.internal.fresh_until)\n                }\n            };\n            // actual eviction can be done async\n            let trace = self\n                .trace\n                .child(\"cache put eviction\", |o| o.start())\n                .handle();\n            let storage = self.storage;\n            tokio::task::spawn(async move {\n                for item in evicted {\n                    if let Err(e) = storage.purge(&item, PurgeType::Eviction, &trace).await {\n                        warn!(\"Failed to purge {item} during eviction for cache put: {e}\");\n                    }\n                }\n            });\n        }\n\n        Ok(())\n    }\n\n    fn trace_header(&mut self, header: &ResponseHeader) {\n        self.trace.set_tag(|| {\n            Tag::new(\n                \"cache-control\",\n                header\n                    .headers\n                    .get_all(http::header::CACHE_CONTROL)\n                    .into_iter()\n                    .map(|v| String::from_utf8_lossy(v.as_bytes()).to_string())\n                    .collect::<Vec<_>>()\n                    .join(\",\"),\n            )\n        });\n    }\n\n    async fn do_cache_put(&mut self, data: &[u8]) -> Result<Option<NoCacheReason>> {\n        let tasks = self.parser.inject_data(data)?;\n        for task in tasks {\n            match task {\n                HttpTask::Header(header, _eos) => {\n                    self.trace_header(&header);\n                    match self.cache_put.cacheable(*header) {\n                        RespCacheable::Cacheable(meta) => {\n                            if let Some(max_file_size_tracker) = &self.max_file_size_tracker {\n                                let content_length_hdr = meta.headers().get(header::CONTENT_LENGTH);\n                                if let Some(content_length) =\n                                    header_value_content_length(content_length_hdr)\n                                {\n                                    if content_length > max_file_size_tracker.max_file_size_bytes()\n                                    {\n                                        return Ok(Some(NoCacheReason::ResponseTooLarge));\n                                    }\n                                }\n                            }\n\n                            self.put_header(meta).await?;\n                        }\n                        RespCacheable::Uncacheable(reason) => {\n                            return Ok(Some(reason));\n                        }\n                    }\n                }\n                HttpTask::Body(data, eos) => {\n                    if let Some(data) = data {\n                        self.put_body(data, eos).await?;\n                    }\n                }\n                _ => {\n                    panic!(\"unexpected HttpTask during cache put {task:?}\");\n                }\n            }\n        }\n        Ok(None)\n    }\n\n    /// Start the cache put logic for the given request\n    ///\n    /// This function will start to read the request body to put into cache.\n    /// Return:\n    /// - `Ok(None)` when the payload will be cache.\n    /// - `Ok(Some(reason))` when the payload is not cacheable\n    pub async fn cache_put(\n        &mut self,\n        session: &mut ServerSession,\n    ) -> Result<Option<NoCacheReason>> {\n        let mut no_cache_reason = None;\n        while let Some(data) = session.read_request_body().await? {\n            if no_cache_reason.is_some() {\n                // even uncacheable, the entire body needs to be drains for 1. downstream\n                // not throwing errors 2. connection reuse\n                continue;\n            }\n            no_cache_reason = self.do_cache_put(&data).await?\n        }\n        self.parser.finish()?;\n        self.finish().await?;\n\n        if let Some(reason) = no_cache_reason {\n            self.trace\n                .set_tag(|| Tag::new(\"uncacheable_reason\", reason.as_str()));\n        }\n\n        Ok(no_cache_reason)\n    }\n}\n\n#[cfg(test)]\nmod test {\n    use super::*;\n    use cf_rustracing::span::Span;\n    use once_cell::sync::Lazy;\n\n    struct TestCachePut();\n    impl CachePut for TestCachePut {\n        fn cache_defaults() -> &'static CacheMetaDefaults {\n            const DEFAULT: CacheMetaDefaults =\n                CacheMetaDefaults::new(|_| Some(Duration::from_secs(1)), 1, 1);\n            &DEFAULT\n        }\n    }\n\n    type TestCachePutCtx = CachePutCtx<TestCachePut>;\n    static CACHE_BACKEND: Lazy<MemCache> = Lazy::new(MemCache::new);\n\n    #[tokio::test]\n    async fn test_cache_put() {\n        let key = CacheKey::new(\"\", \"a\", \"1\");\n        let span = Span::inactive();\n        let put = TestCachePut();\n        let mut ctx = TestCachePutCtx::new(put, key.clone(), &*CACHE_BACKEND, None, span);\n        let payload = b\"HTTP/1.1 200 OK\\r\\n\\\n        Date: Thu, 26 Apr 2018 05:42:05 GMT\\r\\n\\\n        Content-Type: text/html; charset=utf-8\\r\\n\\\n        Connection: keep-alive\\r\\n\\\n        X-Frame-Options: SAMEORIGIN\\r\\n\\\n        Cache-Control: public, max-age=1\\r\\n\\\n        Server: origin-server\\r\\n\\\n        Content-Length: 4\\r\\n\\r\\nrust\";\n        // here we skip mocking a real http session for simplicity\n        let res = ctx.do_cache_put(payload).await.unwrap();\n        assert!(res.is_none()); // cacheable\n        ctx.parser.finish().unwrap();\n        ctx.finish().await.unwrap();\n\n        let span = Span::inactive();\n        let (meta, mut hit) = CACHE_BACKEND\n            .lookup(&key, &span.handle())\n            .await\n            .unwrap()\n            .unwrap();\n        assert_eq!(\n            meta.headers().get(\"date\").unwrap(),\n            \"Thu, 26 Apr 2018 05:42:05 GMT\"\n        );\n        let data = hit.read_body().await.unwrap().unwrap();\n        assert_eq!(data, \"rust\");\n    }\n\n    #[tokio::test]\n    async fn test_cache_put_uncacheable() {\n        let key = CacheKey::new(\"\", \"a\", \"1\");\n        let span = Span::inactive();\n        let put = TestCachePut();\n        let mut ctx = TestCachePutCtx::new(put, key.clone(), &*CACHE_BACKEND, None, span);\n        let payload = b\"HTTP/1.1 200 OK\\r\\n\\\n        Date: Thu, 26 Apr 2018 05:42:05 GMT\\r\\n\\\n        Content-Type: text/html; charset=utf-8\\r\\n\\\n        Connection: keep-alive\\r\\n\\\n        X-Frame-Options: SAMEORIGIN\\r\\n\\\n        Cache-Control: no-store\\r\\n\\\n        Server: origin-server\\r\\n\\\n        Content-Length: 4\\r\\n\\r\\nrust\";\n        // here we skip mocking a real http session for simplicity\n        let no_cache = ctx.do_cache_put(payload).await.unwrap().unwrap();\n        assert_eq!(no_cache, NoCacheReason::OriginNotCache);\n        ctx.parser.finish().unwrap();\n        ctx.finish().await.unwrap();\n    }\n\n    #[tokio::test]\n    async fn test_cache_put_204_invalid_body() {\n        let key = CacheKey::new(\"\", \"b\", \"1\");\n        let span = Span::inactive();\n        let put = TestCachePut();\n        let mut ctx = TestCachePutCtx::new(put, key.clone(), &*CACHE_BACKEND, None, span);\n        let payload = b\"HTTP/1.1 204 OK\\r\\n\\\n        Date: Thu, 26 Apr 2018 05:42:05 GMT\\r\\n\\\n        Content-Type: text/html; charset=utf-8\\r\\n\\\n        Connection: keep-alive\\r\\n\\\n        X-Frame-Options: SAMEORIGIN\\r\\n\\\n        Cache-Control: public, max-age=1\\r\\n\\\n        Server: origin-server\\r\\n\\\n        Content-Length: 4\\r\\n\\r\\n\";\n        // here we skip mocking a real http session for simplicity\n        let res = ctx.do_cache_put(payload).await.unwrap();\n        assert!(res.is_none()); // cacheable\n                                // 204 should not have body, invalid client input may try to pass one\n        let res = ctx.do_cache_put(b\"rust\").await.unwrap();\n        assert!(res.is_none()); // still cacheable\n        ctx.parser.finish().unwrap();\n        ctx.finish().await.unwrap();\n\n        let span = Span::inactive();\n        let (meta, mut hit) = CACHE_BACKEND\n            .lookup(&key, &span.handle())\n            .await\n            .unwrap()\n            .unwrap();\n        assert_eq!(\n            meta.headers().get(\"date\").unwrap(),\n            \"Thu, 26 Apr 2018 05:42:05 GMT\"\n        );\n        // just treated as empty body\n        // (TODO: should we reset content-length/transfer-encoding\n        // headers on 204/304?)\n        let data = hit.read_body().await.unwrap().unwrap();\n        assert!(data.is_empty());\n    }\n\n    #[tokio::test]\n    async fn test_cache_put_extra_body() {\n        let key = CacheKey::new(\"\", \"c\", \"1\");\n        let span = Span::inactive();\n        let put = TestCachePut();\n        let mut ctx = TestCachePutCtx::new(put, key.clone(), &*CACHE_BACKEND, None, span);\n        let payload = b\"HTTP/1.1 200 OK\\r\\n\\\n        Date: Thu, 26 Apr 2018 05:42:05 GMT\\r\\n\\\n        Content-Type: text/html; charset=utf-8\\r\\n\\\n        Connection: keep-alive\\r\\n\\\n        X-Frame-Options: SAMEORIGIN\\r\\n\\\n        Cache-Control: public, max-age=1\\r\\n\\\n        Server: origin-server\\r\\n\\\n        Content-Length: 4\\r\\n\\r\\n\";\n        // here we skip mocking a real http session for simplicity\n        let res = ctx.do_cache_put(payload).await.unwrap();\n        assert!(res.is_none()); // cacheable\n                                // pass in more extra request body that needs to be drained\n        let res = ctx.do_cache_put(b\"rustab\").await.unwrap();\n        assert!(res.is_none()); // still cacheable\n        let res = ctx.do_cache_put(b\"cdef\").await.unwrap();\n        assert!(res.is_none()); // still cacheable\n        ctx.parser.finish().unwrap();\n        ctx.finish().await.unwrap();\n\n        let span = Span::inactive();\n        let (meta, mut hit) = CACHE_BACKEND\n            .lookup(&key, &span.handle())\n            .await\n            .unwrap()\n            .unwrap();\n        assert_eq!(\n            meta.headers().get(\"date\").unwrap(),\n            \"Thu, 26 Apr 2018 05:42:05 GMT\"\n        );\n        let data = hit.read_body().await.unwrap().unwrap();\n        // body only contains specified content-length bounds\n        assert_eq!(data, \"rust\");\n    }\n}\n\n// maybe this can simplify some logic in pingora::h1\n\nmod parse_response {\n    use super::*;\n    use bstr::ByteSlice;\n    use bytes::BytesMut;\n    use httparse::Status;\n    use pingora_error::{\n        Error,\n        ErrorType::{self, *},\n    };\n\n    pub const INCOMPLETE_BODY: ErrorType = ErrorType::new(\"IncompleteHttpBody\");\n\n    const MAX_HEADERS: usize = 256;\n    const INIT_HEADER_BUF_SIZE: usize = 4096;\n\n    #[derive(Debug, Clone, Copy, PartialEq)]\n    enum ParseState {\n        Init,\n        PartialHeader,\n        PartialBodyContentLength(usize, usize),\n        PartialBody(usize),\n        Done(usize),\n        Invalid(httparse::Error),\n    }\n\n    impl ParseState {\n        fn is_done(&self) -> bool {\n            matches!(self, Self::Done(_))\n        }\n        fn read_header(&self) -> bool {\n            matches!(self, Self::Init | Self::PartialHeader)\n        }\n        fn read_body(&self) -> bool {\n            matches!(\n                self,\n                Self::PartialBodyContentLength(..) | Self::PartialBody(_)\n            )\n        }\n    }\n\n    pub(super) struct ResponseParse {\n        state: ParseState,\n        buf: BytesMut,\n        header_bytes: Bytes,\n    }\n\n    impl ResponseParse {\n        pub fn new() -> Self {\n            ResponseParse {\n                state: ParseState::Init,\n                buf: BytesMut::with_capacity(INIT_HEADER_BUF_SIZE),\n                header_bytes: Bytes::new(),\n            }\n        }\n\n        pub fn inject_data(&mut self, data: &[u8]) -> Result<Vec<HttpTask>> {\n            if self.state.is_done() {\n                // just ignore extra response body after parser is done\n                // could be invalid body appended to a no-content status\n                // or invalid body after content-length\n                // TODO: consider propagating an error to the client\n                return Ok(vec![]);\n            }\n\n            self.put_data(data);\n\n            let mut tasks = vec![];\n            while !self.state.is_done() {\n                if self.state.read_header() {\n                    let header = self.parse_header()?;\n                    let Some(header) = header else {\n                        break;\n                    };\n                    tasks.push(HttpTask::Header(Box::new(header), self.state.is_done()));\n                } else if self.state.read_body() {\n                    let body = self.parse_body()?;\n                    let Some(body) = body else {\n                        break;\n                    };\n                    tasks.push(HttpTask::Body(Some(body), self.state.is_done()));\n                } else {\n                    break;\n                }\n            }\n            Ok(tasks)\n        }\n\n        fn put_data(&mut self, data: &[u8]) {\n            use ParseState::*;\n            if matches!(self.state, Done(_) | Invalid(_)) {\n                panic!(\"Wrong phase {:?}\", self.state);\n            }\n            self.buf.extend_from_slice(data);\n        }\n\n        fn parse_header(&mut self) -> Result<Option<ResponseHeader>> {\n            let mut headers = [httparse::EMPTY_HEADER; MAX_HEADERS];\n            let mut resp = httparse::Response::new(&mut headers);\n            let mut parser = httparse::ParserConfig::default();\n            parser.allow_spaces_after_header_name_in_responses(true);\n            parser.allow_obsolete_multiline_headers_in_responses(true);\n\n            let res = parser.parse_response(&mut resp, &self.buf);\n            let res = match res {\n                Ok(res) => res,\n                Err(e) => {\n                    self.state = ParseState::Invalid(e);\n                    return Error::e_because(\n                        InvalidHTTPHeader,\n                        format!(\"buf: {:?}\", self.buf.as_bstr()),\n                        e,\n                    );\n                }\n            };\n\n            let split_to = match res {\n                Status::Complete(s) => s,\n                Status::Partial => {\n                    self.state = ParseState::PartialHeader;\n                    return Ok(None);\n                }\n            };\n            // safe to unwrap, valid response always has code set.\n            let mut response =\n                ResponseHeader::build(resp.code.unwrap(), Some(resp.headers.len())).unwrap();\n            for header in resp.headers {\n                // TODO: consider hold a Bytes and all header values can be Bytes referencing the\n                // original buffer without reallocation\n                response.append_header(header.name.to_owned(), header.value.to_owned())?;\n            }\n            // TODO: see above, we can make header value `Bytes` referencing header_bytes\n            let header_bytes = self.buf.split_to(split_to).freeze();\n            self.header_bytes = header_bytes;\n            self.state = body_type(&response);\n\n            Ok(Some(response))\n        }\n\n        fn parse_body(&mut self) -> Result<Option<Bytes>> {\n            use ParseState::*;\n            if self.buf.is_empty() {\n                return Ok(None);\n            }\n            match self.state {\n                Init | PartialHeader | Invalid(_) => {\n                    panic!(\"Wrong phase {:?}\", self.state);\n                }\n                Done(_) => Ok(None),\n                PartialBodyContentLength(total, mut seen) => {\n                    let end = if total < self.buf.len() + seen {\n                        // TODO: warn! more data than expected\n                        total - seen\n                    } else {\n                        self.buf.len()\n                    };\n                    seen += end;\n                    if seen >= total {\n                        self.state = Done(seen);\n                    } else {\n                        self.state = PartialBodyContentLength(total, seen);\n                    }\n                    Ok(Some(self.buf.split_to(end).freeze()))\n                }\n                PartialBody(seen) => {\n                    self.state = PartialBody(seen + self.buf.len());\n                    Ok(Some(self.buf.split().freeze()))\n                }\n            }\n        }\n\n        pub fn finish(&mut self) -> Result<()> {\n            if let ParseState::PartialBody(seen) = self.state {\n                self.state = ParseState::Done(seen);\n            }\n            if !self.state.is_done() {\n                Error::e_explain(INCOMPLETE_BODY, format!(\"{:?}\", self.state))\n            } else {\n                Ok(())\n            }\n        }\n    }\n\n    fn body_type(resp: &ResponseHeader) -> ParseState {\n        use http::StatusCode;\n\n        if matches!(\n            resp.status,\n            StatusCode::NO_CONTENT | StatusCode::NOT_MODIFIED\n        ) {\n            // these status codes cannot have body by definition\n            return ParseState::Done(0);\n        }\n        if let Some(cl) = resp.headers.get(http::header::CONTENT_LENGTH) {\n            // ignore invalid header value\n            if let Some(cl) = std::str::from_utf8(cl.as_bytes())\n                .ok()\n                .and_then(|cl| cl.parse::<usize>().ok())\n            {\n                return if cl == 0 {\n                    ParseState::Done(0)\n                } else {\n                    ParseState::PartialBodyContentLength(cl, 0)\n                };\n            }\n        }\n        // HTTP/1.0 and chunked encoding are both treated as PartialBody\n        // The response body payload should _not_ be chunked encoded\n        // even if the Transfer-Encoding: chunked header is added\n        ParseState::PartialBody(0)\n    }\n\n    #[cfg(test)]\n    mod test {\n        use super::*;\n\n        #[test]\n        fn test_basic_response() {\n            let input = b\"HTTP/1.1 200 OK\\r\\n\\r\\n\";\n            let mut parser = ResponseParse::new();\n            let output = parser.inject_data(input).unwrap();\n            assert_eq!(output.len(), 1);\n            let HttpTask::Header(header, eos) = &output[0] else {\n                panic!(\"{:?}\", output);\n            };\n            assert_eq!(header.status, 200);\n            assert!(!eos);\n\n            let body = b\"abc\";\n            let output = parser.inject_data(body).unwrap();\n            assert_eq!(output.len(), 1);\n            let HttpTask::Body(data, _eos) = &output[0] else {\n                panic!(\"{:?}\", output);\n            };\n            assert_eq!(data.as_ref().unwrap(), &body[..]);\n            parser.finish().unwrap();\n        }\n\n        #[test]\n        fn test_partial_response_headers() {\n            let input = b\"HTTP/1.1 200 OK\\r\\n\";\n            let mut parser = ResponseParse::new();\n            let output = parser.inject_data(input).unwrap();\n            // header is not complete\n            assert_eq!(output.len(), 0);\n\n            let output = parser\n                .inject_data(\"Server: pingora\\r\\n\\r\\n\".as_bytes())\n                .unwrap();\n            assert_eq!(output.len(), 1);\n            let HttpTask::Header(header, eos) = &output[0] else {\n                panic!(\"{:?}\", output);\n            };\n            assert_eq!(header.status, 200);\n            assert_eq!(header.headers.get(\"Server\").unwrap(), \"pingora\");\n            assert!(!eos);\n        }\n\n        #[test]\n        fn test_invalid_headers() {\n            let input = b\"HTP/1.1 200 OK\\r\\nServer: pingora\\r\\n\\r\\n\";\n            let mut parser = ResponseParse::new();\n            let output = parser.inject_data(input);\n            // header is not complete\n            assert!(output.is_err());\n            match parser.state {\n                ParseState::Invalid(httparse::Error::Version) => {}\n                _ => panic!(\"should have failed to parse\"),\n            }\n        }\n\n        #[test]\n        fn test_body_content_length() {\n            let input = b\"HTTP/1.1 200 OK\\r\\nContent-Length: 6\\r\\n\\r\\nabc\";\n            let mut parser = ResponseParse::new();\n            let output = parser.inject_data(input).unwrap();\n\n            assert_eq!(output.len(), 2);\n            let HttpTask::Header(header, _eos) = &output[0] else {\n                panic!(\"{:?}\", output);\n            };\n            assert_eq!(header.status, 200);\n\n            let HttpTask::Body(data, eos) = &output[1] else {\n                panic!(\"{:?}\", output);\n            };\n            assert_eq!(data.as_ref().unwrap(), \"abc\");\n            assert!(!eos);\n\n            let output = parser.inject_data(b\"def\").unwrap();\n            assert_eq!(output.len(), 1);\n            let HttpTask::Body(data, eos) = &output[0] else {\n                panic!(\"{:?}\", output);\n            };\n            assert_eq!(data.as_ref().unwrap(), \"def\");\n            assert!(eos);\n\n            parser.finish().unwrap();\n        }\n\n        #[test]\n        fn test_body_chunked() {\n            let input = b\"HTTP/1.1 200 OK\\r\\nTransfer-Encoding: chunked\\r\\n\\r\\nrust\";\n            let mut parser = ResponseParse::new();\n            let output = parser.inject_data(input).unwrap();\n\n            assert_eq!(output.len(), 2);\n            let HttpTask::Header(header, _eos) = &output[0] else {\n                panic!(\"{:?}\", output);\n            };\n            assert_eq!(header.status, 200);\n\n            let HttpTask::Body(data, eos) = &output[1] else {\n                panic!(\"{:?}\", output);\n            };\n            assert_eq!(data.as_ref().unwrap(), \"rust\");\n            assert!(!eos);\n\n            parser.finish().unwrap();\n        }\n\n        #[test]\n        fn test_body_content_length_early() {\n            let input = b\"HTTP/1.1 200 OK\\r\\nContent-Length: 6\\r\\n\\r\\nabc\";\n            let mut parser = ResponseParse::new();\n            let output = parser.inject_data(input).unwrap();\n\n            assert_eq!(output.len(), 2);\n            let HttpTask::Header(header, _eos) = &output[0] else {\n                panic!(\"{:?}\", output);\n            };\n            assert_eq!(header.status, 200);\n\n            let HttpTask::Body(data, eos) = &output[1] else {\n                panic!(\"{:?}\", output);\n            };\n            assert_eq!(data.as_ref().unwrap(), \"abc\");\n            assert!(!eos);\n\n            parser.finish().unwrap_err();\n        }\n\n        #[test]\n        fn test_body_content_length_more_data() {\n            let input = b\"HTTP/1.1 200 OK\\r\\nContent-Length: 2\\r\\n\\r\\nabc\";\n            let mut parser = ResponseParse::new();\n            let output = parser.inject_data(input).unwrap();\n\n            assert_eq!(output.len(), 2);\n            let HttpTask::Header(header, _eos) = &output[0] else {\n                panic!(\"{:?}\", output);\n            };\n            assert_eq!(header.status, 200);\n\n            let HttpTask::Body(data, eos) = &output[1] else {\n                panic!(\"{:?}\", output);\n            };\n            assert_eq!(data.as_ref().unwrap(), \"ab\");\n            assert!(eos);\n\n            // extra data is dropped without error\n            parser.finish().unwrap();\n        }\n\n        #[test]\n        fn test_body_chunked_partial_chunk() {\n            let input = b\"HTTP/1.1 200 OK\\r\\nTransfer-Encoding: chunked\\r\\n\\r\\nru\";\n            let mut parser = ResponseParse::new();\n            let output = parser.inject_data(input).unwrap();\n\n            assert_eq!(output.len(), 2);\n            let HttpTask::Header(header, _eos) = &output[0] else {\n                panic!(\"{:?}\", output);\n            };\n            assert_eq!(header.status, 200);\n\n            let HttpTask::Body(data, eos) = &output[1] else {\n                panic!(\"{:?}\", output);\n            };\n            assert_eq!(data.as_ref().unwrap(), \"ru\");\n            assert!(!eos);\n\n            let output = parser.inject_data(b\"st\\r\\n\").unwrap();\n            assert_eq!(output.len(), 1);\n            let HttpTask::Body(data, eos) = &output[0] else {\n                panic!(\"{:?}\", output);\n            };\n            assert_eq!(data.as_ref().unwrap(), \"st\\r\\n\");\n            assert!(!eos);\n        }\n\n        #[test]\n        fn test_no_body_content_length() {\n            let input = b\"HTTP/1.1 200 OK\\r\\nContent-Length: 0\\r\\n\\r\\n\";\n            let mut parser = ResponseParse::new();\n            let output = parser.inject_data(input).unwrap();\n\n            assert_eq!(output.len(), 1);\n            let HttpTask::Header(header, eos) = &output[0] else {\n                panic!(\"{:?}\", output);\n            };\n            assert_eq!(header.status, 200);\n            assert!(eos);\n\n            parser.finish().unwrap();\n        }\n\n        #[test]\n        fn test_no_body_304_no_content_length() {\n            let input = b\"HTTP/1.1 304 Not Modified\\r\\nCache-Control: public, max-age=10\\r\\n\\r\\n\";\n            let mut parser = ResponseParse::new();\n            let output = parser.inject_data(input).unwrap();\n\n            assert_eq!(output.len(), 1);\n            let HttpTask::Header(header, eos) = &output[0] else {\n                panic!(\"{:?}\", output);\n            };\n            assert_eq!(header.status, 304);\n            assert!(eos);\n\n            parser.finish().unwrap();\n        }\n\n        #[test]\n        fn test_204_with_chunked_body() {\n            let input = b\"HTTP/1.1 204 No Content\\r\\nCache-Control: public, max-age=10\\r\\nTransfer-Encoding: chunked\\r\\n\\r\\n\";\n            let mut parser = ResponseParse::new();\n            let output = parser.inject_data(input).unwrap();\n\n            assert_eq!(output.len(), 1);\n            let HttpTask::Header(header, eos) = &output[0] else {\n                panic!(\"{:?}\", output);\n            };\n            assert_eq!(header.status, 204);\n            assert!(eos);\n\n            // 204 should not have a body, parser ignores bad input\n            let output = parser.inject_data(b\"4\\r\\nrust\\r\\n0\\r\\n\\r\\n\").unwrap();\n            assert!(output.is_empty());\n            parser.finish().unwrap();\n        }\n\n        #[test]\n        fn test_204_with_content_length() {\n            let input = b\"HTTP/1.1 204 No Content\\r\\nCache-Control: public, max-age=10\\r\\nContent-Length: 4\\r\\n\\r\\n\";\n            let mut parser = ResponseParse::new();\n            let output = parser.inject_data(input).unwrap();\n\n            assert_eq!(output.len(), 1);\n            let HttpTask::Header(header, eos) = &output[0] else {\n                panic!(\"{:?}\", output);\n            };\n            assert_eq!(header.status, 204);\n            assert!(eos);\n\n            // 204 should not have a body, parser ignores bad input\n            let output = parser.inject_data(b\"rust\").unwrap();\n            assert!(output.is_empty());\n            parser.finish().unwrap();\n        }\n\n        #[test]\n        fn test_200_with_zero_content_length_more_data() {\n            let input = b\"HTTP/1.1 200 OK\\r\\nCache-Control: public, max-age=10\\r\\nContent-Length: 0\\r\\n\\r\\n\";\n            let mut parser = ResponseParse::new();\n            let output = parser.inject_data(input).unwrap();\n\n            assert_eq!(output.len(), 1);\n            let HttpTask::Header(header, eos) = &output[0] else {\n                panic!(\"{:?}\", output);\n            };\n            assert_eq!(header.status, 200);\n            assert!(eos);\n\n            let output = parser.inject_data(b\"rust\").unwrap();\n            assert!(output.is_empty());\n            parser.finish().unwrap();\n        }\n    }\n}\n"
  },
  {
    "path": "pingora-cache/src/storage.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! Cache backend storage abstraction\n\nuse super::{CacheKey, CacheMeta};\nuse crate::key::CompactCacheKey;\nuse crate::trace::SpanHandle;\n\nuse async_trait::async_trait;\nuse pingora_error::Result;\nuse std::any::Any;\n\n/// The reason a purge() is called\n#[derive(Debug, Clone, Copy)]\npub enum PurgeType {\n    // For eviction because the cache storage is full\n    Eviction,\n    // For cache invalidation\n    Invalidation,\n}\n\n/// Cache storage interface\n#[async_trait]\npub trait Storage {\n    // TODO: shouldn't have to be static\n\n    /// Lookup the storage for the given [CacheKey].\n    async fn lookup(\n        &'static self,\n        key: &CacheKey,\n        trace: &SpanHandle,\n    ) -> Result<Option<(CacheMeta, HitHandler)>>;\n\n    /// Lookup the storage for the given [CacheKey] using a streaming write tag.\n    ///\n    /// When streaming partial writes is supported, the request that initiates the write will also\n    /// pass an optional `streaming_write_tag` so that the storage may try to find the associated\n    /// [HitHandler], for the same ongoing write.\n    ///\n    /// Therefore, when the write tag is set, the storage implementation should either return a\n    /// [HitHandler] that can be matched to that tag, or none at all. Otherwise when the storage\n    /// supports concurrent streaming writes for the same key, the calling request may receive a\n    /// different body from the one it expected.\n    ///\n    /// By default this defers to the standard `Storage::lookup` implementation.\n    async fn lookup_streaming_write(\n        &'static self,\n        key: &CacheKey,\n        _streaming_write_tag: Option<&[u8]>,\n        trace: &SpanHandle,\n    ) -> Result<Option<(CacheMeta, HitHandler)>> {\n        self.lookup(key, trace).await\n    }\n\n    /// Write the given [CacheMeta] to the storage. Return [MissHandler] to write the body later.\n    async fn get_miss_handler(\n        &'static self,\n        key: &CacheKey,\n        meta: &CacheMeta,\n        trace: &SpanHandle,\n    ) -> Result<MissHandler>;\n\n    /// Delete the cached asset for the given key\n    ///\n    /// [CompactCacheKey] is used here because it is how eviction managers store the keys\n    async fn purge(\n        &'static self,\n        key: &CompactCacheKey,\n        purge_type: PurgeType,\n        trace: &SpanHandle,\n    ) -> Result<bool>;\n\n    /// Update cache header and metadata for the already stored asset.\n    async fn update_meta(\n        &'static self,\n        key: &CacheKey,\n        meta: &CacheMeta,\n        trace: &SpanHandle,\n    ) -> Result<bool>;\n\n    /// Whether this storage backend supports reading partially written data\n    ///\n    /// This is to indicate when cache should unlock readers\n    fn support_streaming_partial_write(&self) -> bool {\n        false\n    }\n\n    /// Helper function to cast the trait object to concrete types\n    fn as_any(&self) -> &(dyn Any + Send + Sync + 'static);\n}\n\n/// Cache hit handling trait\n#[async_trait]\npub trait HandleHit {\n    /// Read cached body\n    ///\n    /// Return `None` when no more body to read.\n    async fn read_body(&mut self) -> Result<Option<bytes::Bytes>>;\n\n    /// Finish the current cache hit\n    async fn finish(\n        self: Box<Self>, // because self is always used as a trait object\n        storage: &'static (dyn Storage + Sync),\n        key: &CacheKey,\n        trace: &SpanHandle,\n    ) -> Result<()>;\n\n    /// Whether this storage allows seeking to a certain range of body for single ranges.\n    fn can_seek(&self) -> bool {\n        false\n    }\n\n    /// Whether this storage allows seeking to a certain range of body for multipart ranges.\n    ///\n    /// By default uses the `can_seek` implementation.\n    fn can_seek_multipart(&self) -> bool {\n        self.can_seek()\n    }\n\n    /// Try to seek to a certain range of the body for single ranges.\n    ///\n    /// `end: None` means to read to the end of the body.\n    fn seek(&mut self, _start: usize, _end: Option<usize>) -> Result<()> {\n        // to prevent impl can_seek() without impl seek\n        todo!(\"seek() needs to be implemented\")\n    }\n\n    /// Try to seek to a certain range of the body for multipart ranges.\n    ///\n    /// Works in an identical manner to `seek()`.\n    ///\n    /// `end: None` means to read to the end of the body.\n    ///\n    /// By default uses the `seek` implementation, but hit handlers may customize the\n    /// implementation specifically to anticipate multipart requests.\n    fn seek_multipart(&mut self, start: usize, end: Option<usize>) -> Result<()> {\n        // to prevent impl can_seek() without impl seek\n        self.seek(start, end)\n    }\n\n    // TODO: fn is_stream_hit()\n\n    /// Should we count this hit handler instance as an access in the eviction manager.\n    ///\n    /// Defaults to returning true to track all cache hits as accesses. Customize this if certain\n    /// hits should not affect the eviction system's view of the asset.\n    fn should_count_access(&self) -> bool {\n        true\n    }\n\n    /// Returns the weight of the current cache hit asset to report to the eviction manager.\n    ///\n    /// This allows the eviction system to initialize a weight for the asset, in case it is not\n    /// already tracking it (e.g. storage is out of sync with the eviction manager).\n    ///\n    /// Defaults to 0.\n    fn get_eviction_weight(&self) -> usize {\n        0\n    }\n\n    /// Helper function to cast the trait object to concrete types\n    fn as_any(&self) -> &(dyn Any + Send + Sync);\n\n    /// Helper function to cast the trait object to concrete types\n    fn as_any_mut(&mut self) -> &mut (dyn Any + Send + Sync);\n}\n\n/// Hit Handler\npub type HitHandler = Box<dyn HandleHit + Sync + Send>;\n\n/// MissFinishType\npub enum MissFinishType {\n    /// A new asset was created with the given size.\n    Created(usize),\n    /// Appended size to existing asset, with an optional max size param.\n    Appended(usize, Option<usize>),\n}\n\n/// Cache miss handling trait\n#[async_trait]\npub trait HandleMiss {\n    /// Write the given body to the storage\n    async fn write_body(&mut self, data: bytes::Bytes, eof: bool) -> Result<()>;\n\n    /// Finish the cache admission\n    ///\n    /// When `self` is dropped without calling this function, the storage should consider this write\n    /// failed.\n    async fn finish(\n        self: Box<Self>, // because self is always used as a trait object\n    ) -> Result<MissFinishType>;\n\n    /// Return a streaming write tag recognized by the underlying [`Storage`].\n    ///\n    /// This is an arbitrary data identifier that is used to associate this miss handler's current\n    /// write with a hit handler for the same write. This identifier will be compared by the\n    /// storage during `lookup_streaming_write`.\n    // This write tag is essentially an borrowed data blob of bytes retrieved from the miss handler\n    // and passed to storage, which means it can support strings or small data types, e.g. bytes\n    // represented by a u64.\n    // The downside with the current API is that such a data blob must be owned by the miss handler\n    // and stored in a way that permits retrieval as a byte slice (not computed on the fly).\n    // But most use cases likely only require a simple integer and may not like the overhead of a\n    // Vec/String allocation or even a Cow, though such data types can also be used here.\n    fn streaming_write_tag(&self) -> Option<&[u8]> {\n        None\n    }\n}\n\n/// Miss Handler\npub type MissHandler = Box<dyn HandleMiss + Sync + Send>;\n\npub mod streaming_write {\n    /// Portable u64 (sized) write id convenience type for use with streaming writes.\n    ///\n    /// Often an integer value is sufficient for a streaming write tag. This convenience type enables\n    /// storing such a value and functions for consistent conversion between byte sequence data types.\n    #[derive(Debug, Clone, Copy)]\n    pub struct U64WriteId([u8; 8]);\n\n    impl U64WriteId {\n        pub fn as_bytes(&self) -> &[u8] {\n            &self.0[..]\n        }\n    }\n\n    impl From<u64> for U64WriteId {\n        fn from(value: u64) -> U64WriteId {\n            U64WriteId(value.to_be_bytes())\n        }\n    }\n    impl From<U64WriteId> for u64 {\n        fn from(value: U64WriteId) -> u64 {\n            u64::from_be_bytes(value.0)\n        }\n    }\n    impl TryFrom<&[u8]> for U64WriteId {\n        type Error = std::array::TryFromSliceError;\n\n        fn try_from(value: &[u8]) -> std::result::Result<Self, Self::Error> {\n            Ok(U64WriteId(value.try_into()?))\n        }\n    }\n\n    /// Portable u32 (sized) write id convenience type for use with streaming writes.\n    ///\n    /// Often an integer value is sufficient for a streaming write tag. This convenience type enables\n    /// storing such a value and functions for consistent conversion between byte sequence data types.\n    #[derive(Debug, Clone, Copy)]\n    pub struct U32WriteId([u8; 4]);\n\n    impl U32WriteId {\n        pub fn as_bytes(&self) -> &[u8] {\n            &self.0[..]\n        }\n    }\n\n    impl From<u32> for U32WriteId {\n        fn from(value: u32) -> U32WriteId {\n            U32WriteId(value.to_be_bytes())\n        }\n    }\n    impl From<U32WriteId> for u32 {\n        fn from(value: U32WriteId) -> u32 {\n            u32::from_be_bytes(value.0)\n        }\n    }\n    impl TryFrom<&[u8]> for U32WriteId {\n        type Error = std::array::TryFromSliceError;\n\n        fn try_from(value: &[u8]) -> std::result::Result<Self, Self::Error> {\n            Ok(U32WriteId(value.try_into()?))\n        }\n    }\n}\n"
  },
  {
    "path": "pingora-cache/src/trace.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! Distributed tracing helpers\n\nuse cf_rustracing_jaeger::span::SpanContextState;\nuse std::time::SystemTime;\n\nuse crate::{CacheMeta, CachePhase, HitStatus};\n\npub use cf_rustracing::tag::Tag;\n\npub type Span = cf_rustracing::span::Span<SpanContextState>;\npub type SpanHandle = cf_rustracing::span::SpanHandle<SpanContextState>;\n\n#[derive(Debug)]\npub(crate) struct CacheTraceCTX {\n    // parent span\n    pub cache_span: Span,\n    // only spans across multiple calls need to store here\n    pub miss_span: Span,\n    pub hit_span: Span,\n}\n\npub fn tag_span_with_meta(span: &mut Span, meta: &CacheMeta) {\n    fn ts2epoch(ts: SystemTime) -> f64 {\n        ts.duration_since(SystemTime::UNIX_EPOCH)\n            .unwrap_or_default() // should never overflow but be safe here\n            .as_secs_f64()\n    }\n    let internal = &meta.0.internal;\n    span.set_tags(|| {\n        [\n            Tag::new(\"created\", ts2epoch(internal.created)),\n            Tag::new(\"fresh_until\", ts2epoch(internal.fresh_until)),\n            Tag::new(\"updated\", ts2epoch(internal.updated)),\n            Tag::new(\"stale_if_error_sec\", internal.stale_if_error_sec as i64),\n            Tag::new(\n                \"stale_while_revalidate_sec\",\n                internal.stale_while_revalidate_sec as i64,\n            ),\n            Tag::new(\"variance\", internal.variance.is_some()),\n        ]\n    });\n}\n\nimpl CacheTraceCTX {\n    pub fn new() -> Self {\n        CacheTraceCTX {\n            cache_span: Span::inactive(),\n            miss_span: Span::inactive(),\n            hit_span: Span::inactive(),\n        }\n    }\n\n    pub fn enable(&mut self, cache_span: Span) {\n        self.cache_span = cache_span;\n    }\n\n    pub fn get_cache_span(&self) -> SpanHandle {\n        self.cache_span.handle()\n    }\n\n    #[inline]\n    pub fn child(&self, name: &'static str) -> Span {\n        self.cache_span.child(name, |o| o.start())\n    }\n\n    pub fn start_miss_span(&mut self) {\n        self.miss_span = self.child(\"miss\");\n    }\n\n    pub fn get_miss_span(&self) -> SpanHandle {\n        self.miss_span.handle()\n    }\n\n    pub fn finish_miss_span(&mut self) {\n        self.miss_span.set_finish_time(SystemTime::now);\n    }\n\n    pub fn start_hit_span(&mut self, phase: CachePhase, hit_status: HitStatus) {\n        self.hit_span = self.child(\"hit\");\n        self.hit_span.set_tag(|| Tag::new(\"phase\", phase.as_str()));\n        self.hit_span\n            .set_tag(|| Tag::new(\"status\", hit_status.as_str()));\n    }\n\n    pub fn get_hit_span(&self) -> SpanHandle {\n        self.hit_span.handle()\n    }\n\n    pub fn finish_hit_span(&mut self) {\n        self.hit_span.set_finish_time(SystemTime::now);\n    }\n\n    pub fn log_meta_in_hit_span(&mut self, meta: &CacheMeta) {\n        tag_span_with_meta(&mut self.hit_span, meta);\n    }\n\n    pub fn log_meta_in_miss_span(&mut self, meta: &CacheMeta) {\n        tag_span_with_meta(&mut self.miss_span, meta);\n    }\n}\n"
  },
  {
    "path": "pingora-cache/src/variance.rs",
    "content": "use std::{borrow::Cow, collections::BTreeMap};\n\nuse blake2::Digest;\n\nuse crate::key::{Blake2b128, HashBinary};\n\n/// A builder for variance keys, used for distinguishing multiple cached assets\n/// at the same URL. This is intended to be easily passed to helper functions,\n/// which can each populate a portion of the variance.\npub struct VarianceBuilder<'a> {\n    values: BTreeMap<Cow<'a, str>, Cow<'a, [u8]>>,\n}\n\nimpl<'a> VarianceBuilder<'a> {\n    /// Create an empty variance key. Has no variance by default - add some variance using\n    /// [`Self::add_value`].\n    pub fn new() -> Self {\n        VarianceBuilder {\n            values: BTreeMap::new(),\n        }\n    }\n\n    /// Add a byte string to the variance key. Not sensitive to insertion order.\n    /// `value` is intended to take either `&str` or `&[u8]`.\n    pub fn add_value(&mut self, name: &'a str, value: &'a (impl AsRef<[u8]> + ?Sized)) {\n        self.values\n            .insert(name.into(), Cow::Borrowed(value.as_ref()));\n    }\n\n    /// Move a byte string to the variance key. Not sensitive to insertion order. Useful when\n    /// writing helper functions which generate a value then add said value to the VarianceBuilder.\n    /// Without this, the helper function would have to move the value to the calling function\n    /// to extend its lifetime to at least match the VarianceBuilder.\n    pub fn add_owned_value(&mut self, name: &'a str, value: Vec<u8>) {\n        self.values.insert(name.into(), Cow::Owned(value));\n    }\n\n    /// Check whether this variance key actually has variance, or just refers to the root asset\n    pub fn has_variance(&self) -> bool {\n        !self.values.is_empty()\n    }\n\n    /// Hash this variance key. Returns [`None`] if [`Self::has_variance`] is false.\n    pub fn finalize(self) -> Option<HashBinary> {\n        const SALT: &[u8; 1] = &[0u8; 1];\n        if self.has_variance() {\n            let mut hash = Blake2b128::new();\n            for (name, value) in self.values.iter() {\n                hash.update(name.as_bytes());\n                hash.update(SALT);\n                hash.update(value);\n                hash.update(SALT);\n            }\n            Some(hash.finalize().into())\n        } else {\n            None\n        }\n    }\n}\n\n#[cfg(test)]\nmod test {\n    use super::*;\n\n    #[test]\n    fn test_basic() {\n        let key_empty = VarianceBuilder::new().finalize();\n        assert_eq!(None, key_empty);\n\n        let mut key_value = VarianceBuilder::new();\n        key_value.add_value(\"a\", \"a\");\n        let key_value = key_value.finalize();\n\n        let mut key_owned_value = VarianceBuilder::new();\n        key_owned_value.add_owned_value(\"a\", \"a\".as_bytes().to_vec());\n        let key_owned_value = key_owned_value.finalize();\n\n        assert_ne!(key_empty, key_value);\n        assert_ne!(key_empty, key_owned_value);\n        assert_eq!(key_value, key_owned_value);\n    }\n\n    #[test]\n    fn test_value_ordering() {\n        let mut key_abc = VarianceBuilder::new();\n        key_abc.add_value(\"a\", \"a\");\n        key_abc.add_value(\"b\", \"b\");\n        key_abc.add_value(\"c\", \"c\");\n        let key_abc = key_abc.finalize().unwrap();\n\n        let mut key_bac = VarianceBuilder::new();\n        key_bac.add_value(\"b\", \"b\");\n        key_bac.add_value(\"a\", \"a\");\n        key_bac.add_value(\"c\", \"c\");\n        let key_bac = key_bac.finalize().unwrap();\n\n        let mut key_cba = VarianceBuilder::new();\n        key_cba.add_value(\"c\", \"c\");\n        key_cba.add_value(\"b\", \"b\");\n        key_cba.add_value(\"a\", \"a\");\n        let key_cba = key_cba.finalize().unwrap();\n\n        assert_eq!(key_abc, key_bac);\n        assert_eq!(key_abc, key_cba);\n    }\n\n    #[test]\n    fn test_value_overriding() {\n        let mut key_a = VarianceBuilder::new();\n        key_a.add_value(\"a\", \"a\");\n        let key_a = key_a.finalize().unwrap();\n\n        let mut key_b = VarianceBuilder::new();\n        key_b.add_value(\"a\", \"b\");\n        key_b.add_value(\"a\", \"a\");\n        let key_b = key_b.finalize().unwrap();\n\n        assert_eq!(key_a, key_b);\n    }\n}\n"
  },
  {
    "path": "pingora-core/Cargo.toml",
    "content": "[package]\nname = \"pingora-core\"\nversion = \"0.8.0\"\nauthors = [\"Yuchen Wu <yuchen@cloudflare.com>\"]\nlicense = \"Apache-2.0\"\nedition = \"2021\"\nrepository = \"https://github.com/cloudflare/pingora\"\ncategories = [\"asynchronous\", \"network-programming\"]\nkeywords = [\"async\", \"http\", \"network\", \"pingora\"]\nexclude = [\"tests/*\"]\ndescription = \"\"\"\nPingora's APIs and traits for the core network protocols.\n\"\"\"\n\n# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html\n\n[lib]\nname = \"pingora_core\"\npath = \"src/lib.rs\"\n\n[dependencies]\npingora-runtime = { version = \"0.8.0\", path = \"../pingora-runtime\" }\npingora-openssl = { version = \"0.8.0\", path = \"../pingora-openssl\", optional = true }\npingora-boringssl = { version = \"0.8.0\", path = \"../pingora-boringssl\", optional = true }\npingora-pool = { version = \"0.8.0\", path = \"../pingora-pool\" }\npingora-error = { version = \"0.8.0\", path = \"../pingora-error\" }\npingora-timeout = { version = \"0.8.0\", path = \"../pingora-timeout\" }\npingora-http = { version = \"0.8.0\", path = \"../pingora-http\" }\npingora-rustls = { version = \"0.8.0\", path = \"../pingora-rustls\", optional = true }\npingora-s2n = { version = \"0.8.0\", path = \"../pingora-s2n\", optional = true }\nbstr = { workspace = true }\ntokio = { workspace = true, features = [\"net\", \"rt-multi-thread\", \"signal\"] }\ntokio-stream = { workspace = true }\nfutures = \"0.3\"\nasync-trait = { workspace = true }\nhttparse = { workspace = true }\nbytes = { workspace = true }\nhttp = { workspace = true }\nlog = { workspace = true }\nh2 = { workspace = true }\nderivative.workspace = true\nclap = { version = \"4.5\", features = [\"derive\"] }\nonce_cell = { workspace = true }\nserde = { version = \"1.0\", features = [\"derive\"] }\nserde_yaml = \"0.9\"\nstrum = \"0.26.2\"\nstrum_macros = \"0.26.2\"\nlibc = \"0.2.70\"\nchrono = { version = \"~0.4.31\", features = [\"alloc\"], default-features = false }\nprometheus = \"0.13\"\nsentry = { version = \"0.36\", features = [\n    \"backtrace\",\n    \"contexts\",\n    \"panic\",\n    \"reqwest\",\n    \"rustls\",\n], default-features = false, optional = true }\nregex = \"1\"\npercent-encoding = \"2.1\"\nparking_lot = { version = \"0.12\", features = [\"arc_lock\"] }\nsocket2 = { version = \">=0.4, <1.0.0\", features = [\"all\"] }\nflate2 = { version = \"1\", features = [\"zlib-ng\"], default-features = false }\nsfv = \"0.10.4\"\nrand = \"0.8\"\nahash = { workspace = true }\nunicase = \"2\"\nbrotli = \"3\"\nopenssl-probe = \"0.1.6\"\ntokio-test = \"0.4\"\nzstd = \"0\"\nhttpdate = \"1\"\nx509-parser = { version = \"0.16.0\", optional = true }\nouroboros = { version = \"0.18.4\", optional = true }\nlru = { workspace = true, optional = true }\ndaggy = \"0.8\"\n\n[target.'cfg(unix)'.dependencies]\ndaemonize = \"0.5.0\"\nnix = \"~0.24.3\"\n\n[target.'cfg(windows)'.dependencies]\nwindows-sys = { version = \"0.59.0\", features = [\"Win32_Networking_WinSock\"] }\n\n[dev-dependencies]\nh2 = { workspace = true, features = [\"unstable\"] }\ntokio-stream = { version = \"0.1\", features = [\"full\"] }\nenv_logger = \"0.11\"\nreqwest = { version = \"0.11\", features = [\n    \"rustls-tls\",\n], default-features = false }\nhyper = \"0.14\"\nrstest = \"0.23.0\"\nrustls = \"0.23\"\n\n[target.'cfg(unix)'.dev-dependencies]\nhyperlocal = \"0.8\"\njemallocator = \"0.5\"\n\n[features]\ndefault = []\nopenssl = [\"pingora-openssl\", \"openssl_derived\"]\nboringssl = [\"pingora-boringssl\", \"openssl_derived\"]\nrustls = [\"pingora-rustls\", \"any_tls\", \"dep:x509-parser\", \"ouroboros\"]\ns2n = [\"pingora-s2n\", \"any_tls\", \"dep:x509-parser\", \"ouroboros\", \"lru\"]\npatched_http1 = [\"pingora-http/patched_http1\"]\nopenssl_derived = [\"any_tls\"]\nany_tls = []\nsentry = [\"dep:sentry\"]\nconnection_filter = []\n"
  },
  {
    "path": "pingora-core/LICENSE",
    "content": "\n                                 Apache License\n                           Version 2.0, January 2004\n                        http://www.apache.org/licenses/\n\n   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION\n\n   1. Definitions.\n\n      \"License\" shall mean the terms and conditions for use, reproduction,\n      and distribution as defined by Sections 1 through 9 of this document.\n\n      \"Licensor\" shall mean the copyright owner or entity authorized by\n      the copyright owner that is granting the License.\n\n      \"Legal Entity\" shall mean the union of the acting entity and all\n      other entities that control, are controlled by, or are under common\n      control with that entity. For the purposes of this definition,\n      \"control\" means (i) the power, direct or indirect, to cause the\n      direction or management of such entity, whether by contract or\n      otherwise, or (ii) ownership of fifty percent (50%) or more of the\n      outstanding shares, or (iii) beneficial ownership of such entity.\n\n      \"You\" (or \"Your\") shall mean an individual or Legal Entity\n      exercising permissions granted by this License.\n\n      \"Source\" form shall mean the preferred form for making modifications,\n      including but not limited to software source code, documentation\n      source, and configuration files.\n\n      \"Object\" form shall mean any form resulting from mechanical\n      transformation or translation of a Source form, including but\n      not limited to compiled object code, generated documentation,\n      and conversions to other media types.\n\n      \"Work\" shall mean the work of authorship, whether in Source or\n      Object form, made available under the License, as indicated by a\n      copyright notice that is included in or attached to the work\n      (an example is provided in the Appendix below).\n\n      \"Derivative Works\" shall mean any work, whether in Source or Object\n      form, that is based on (or derived from) the Work and for which the\n      editorial revisions, annotations, elaborations, or other modifications\n      represent, as a whole, an original work of authorship. For the purposes\n      of this License, Derivative Works shall not include works that remain\n      separable from, or merely link (or bind by name) to the interfaces of,\n      the Work and Derivative Works thereof.\n\n      \"Contribution\" shall mean any work of authorship, including\n      the original version of the Work and any modifications or additions\n      to that Work or Derivative Works thereof, that is intentionally\n      submitted to Licensor for inclusion in the Work by the copyright owner\n      or by an individual or Legal Entity authorized to submit on behalf of\n      the copyright owner. For the purposes of this definition, \"submitted\"\n      means any form of electronic, verbal, or written communication sent\n      to the Licensor or its representatives, including but not limited to\n      communication on electronic mailing lists, source code control systems,\n      and issue tracking systems that are managed by, or on behalf of, the\n      Licensor for the purpose of discussing and improving the Work, but\n      excluding communication that is conspicuously marked or otherwise\n      designated in writing by the copyright owner as \"Not a Contribution.\"\n\n      \"Contributor\" shall mean Licensor and any individual or Legal Entity\n      on behalf of whom a Contribution has been received by Licensor and\n      subsequently incorporated within the Work.\n\n   2. Grant of Copyright License. Subject to the terms and conditions of\n      this License, each Contributor hereby grants to You a perpetual,\n      worldwide, non-exclusive, no-charge, royalty-free, irrevocable\n      copyright license to reproduce, prepare Derivative Works of,\n      publicly display, publicly perform, sublicense, and distribute the\n      Work and such Derivative Works in Source or Object form.\n\n   3. Grant of Patent License. Subject to the terms and conditions of\n      this License, each Contributor hereby grants to You a perpetual,\n      worldwide, non-exclusive, no-charge, royalty-free, irrevocable\n      (except as stated in this section) patent license to make, have made,\n      use, offer to sell, sell, import, and otherwise transfer the Work,\n      where such license applies only to those patent claims licensable\n      by such Contributor that are necessarily infringed by their\n      Contribution(s) alone or by combination of their Contribution(s)\n      with the Work to which such Contribution(s) was submitted. If You\n      institute patent litigation against any entity (including a\n      cross-claim or counterclaim in a lawsuit) alleging that the Work\n      or a Contribution incorporated within the Work constitutes direct\n      or contributory patent infringement, then any patent licenses\n      granted to You under this License for that Work shall terminate\n      as of the date such litigation is filed.\n\n   4. Redistribution. You may reproduce and distribute copies of the\n      Work or Derivative Works thereof in any medium, with or without\n      modifications, and in Source or Object form, provided that You\n      meet the following conditions:\n\n      (a) You must give any other recipients of the Work or\n          Derivative Works a copy of this License; and\n\n      (b) You must cause any modified files to carry prominent notices\n          stating that You changed the files; and\n\n      (c) You must retain, in the Source form of any Derivative Works\n          that You distribute, all copyright, patent, trademark, and\n          attribution notices from the Source form of the Work,\n          excluding those notices that do not pertain to any part of\n          the Derivative Works; and\n\n      (d) If the Work includes a \"NOTICE\" text file as part of its\n          distribution, then any Derivative Works that You distribute must\n          include a readable copy of the attribution notices contained\n          within such NOTICE file, excluding those notices that do not\n          pertain to any part of the Derivative Works, in at least one\n          of the following places: within a NOTICE text file distributed\n          as part of the Derivative Works; within the Source form or\n          documentation, if provided along with the Derivative Works; or,\n          within a display generated by the Derivative Works, if and\n          wherever such third-party notices normally appear. The contents\n          of the NOTICE file are for informational purposes only and\n          do not modify the License. You may add Your own attribution\n          notices within Derivative Works that You distribute, alongside\n          or as an addendum to the NOTICE text from the Work, provided\n          that such additional attribution notices cannot be construed\n          as modifying the License.\n\n      You may add Your own copyright statement to Your modifications and\n      may provide additional or different license terms and conditions\n      for use, reproduction, or distribution of Your modifications, or\n      for any such Derivative Works as a whole, provided Your use,\n      reproduction, and distribution of the Work otherwise complies with\n      the conditions stated in this License.\n\n   5. Submission of Contributions. Unless You explicitly state otherwise,\n      any Contribution intentionally submitted for inclusion in the Work\n      by You to the Licensor shall be under the terms and conditions of\n      this License, without any additional terms or conditions.\n      Notwithstanding the above, nothing herein shall supersede or modify\n      the terms of any separate license agreement you may have executed\n      with Licensor regarding such Contributions.\n\n   6. Trademarks. This License does not grant permission to use the trade\n      names, trademarks, service marks, or product names of the Licensor,\n      except as required for reasonable and customary use in describing the\n      origin of the Work and reproducing the content of the NOTICE file.\n\n   7. Disclaimer of Warranty. Unless required by applicable law or\n      agreed to in writing, Licensor provides the Work (and each\n      Contributor provides its Contributions) on an \"AS IS\" BASIS,\n      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or\n      implied, including, without limitation, any warranties or conditions\n      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A\n      PARTICULAR PURPOSE. You are solely responsible for determining the\n      appropriateness of using or redistributing the Work and assume any\n      risks associated with Your exercise of permissions under this License.\n\n   8. Limitation of Liability. In no event and under no legal theory,\n      whether in tort (including negligence), contract, or otherwise,\n      unless required by applicable law (such as deliberate and grossly\n      negligent acts) or agreed to in writing, shall any Contributor be\n      liable to You for damages, including any direct, indirect, special,\n      incidental, or consequential damages of any character arising as a\n      result of this License or out of the use or inability to use the\n      Work (including but not limited to damages for loss of goodwill,\n      work stoppage, computer failure or malfunction, or any and all\n      other commercial damages or losses), even if such Contributor\n      has been advised of the possibility of such damages.\n\n   9. Accepting Warranty or Additional Liability. While redistributing\n      the Work or Derivative Works thereof, You may choose to offer,\n      and charge a fee for, acceptance of support, warranty, indemnity,\n      or other liability obligations and/or rights consistent with this\n      License. However, in accepting such obligations, You may act only\n      on Your own behalf and on Your sole responsibility, not on behalf\n      of any other Contributor, and only if You agree to indemnify,\n      defend, and hold each Contributor harmless for any liability\n      incurred by, or claims asserted against, such Contributor by reason\n      of your accepting any such warranty or additional liability.\n\n   END OF TERMS AND CONDITIONS\n\n   APPENDIX: How to apply the Apache License to your work.\n\n      To apply the Apache License to your work, attach the following\n      boilerplate notice, with the fields enclosed by brackets \"[]\"\n      replaced with your own identifying information. (Don't include\n      the brackets!)  The text should be enclosed in the appropriate\n      comment syntax for the file format. We also recommend that a\n      file or class name and description of purpose be included on the\n      same \"printed page\" as the copyright notice for easier\n      identification within third-party archives.\n\n   Copyright [yyyy] [name of copyright owner]\n\n   Licensed under the Apache License, Version 2.0 (the \"License\");\n   you may not use this file except in compliance with the License.\n   You may obtain a copy of the License at\n\n       http://www.apache.org/licenses/LICENSE-2.0\n\n   Unless required by applicable law or agreed to in writing, software\n   distributed under the License is distributed on an \"AS IS\" BASIS,\n   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n   See the License for the specific language governing permissions and\n   limitations under the License.\n"
  },
  {
    "path": "pingora-core/examples/bootstrap_as_a_service.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! Example demonstrating how to start a server using [`Server::bootstrap_as_a_service`]\n//! instead of calling [`Server::bootstrap`] directly.\n//!\n//! # Why `bootstrap_as_a_service`?\n//!\n//! [`Server::bootstrap`] runs the bootstrap phase synchronously before any services start.\n//! This means the calling thread blocks during socket FD acquisition and Sentry initialization.\n//!\n//! [`Server::bootstrap_as_a_service`] instead schedules bootstrap as a dependency-aware init\n//! service. This allows other services to declare a dependency on the bootstrap handle and\n//! ensures they only start after bootstrap completes — while keeping setup fully asynchronous\n//! and composable with the rest of the service graph.\n//!\n//! Use `bootstrap_as_a_service` when:\n//! - You want to integrate bootstrap into the service dependency graph\n//! - You want services to wait for bootstrap without blocking the main thread\n//! - You are building more complex startup sequences (e.g. multiple ordered init steps)\n//!\n//! # Running the example\n//!\n//! ```bash\n//! cargo run --example bootstrap_as_a_service --package pingora-core\n//! ```\n//!\n//! # Expected behaviour\n//!\n//! Bootstrap runs as a service before `MyService` starts. `MyService` declares a dependency\n//! on the bootstrap handle, so it will not be started until bootstrap has completed.\n\nuse async_trait::async_trait;\nuse log::info;\nuse pingora_core::server::configuration::Opt;\n#[cfg(unix)]\nuse pingora_core::server::ListenFds;\nuse pingora_core::server::{Server, ShutdownWatch};\nuse pingora_core::services::Service;\n\n/// A simple application service that requires bootstrap to be complete before it starts.\npub struct MyService;\n\n#[async_trait]\nimpl Service for MyService {\n    async fn start_service(\n        &mut self,\n        #[cfg(unix)] _fds: Option<ListenFds>,\n        mut shutdown: ShutdownWatch,\n        _listeners_per_fd: usize,\n    ) {\n        info!(\"MyService: bootstrap is complete, starting up\");\n\n        // Keep running until a shutdown signal is received.\n        shutdown.changed().await.ok();\n\n        info!(\"MyService: shutting down\");\n    }\n\n    fn name(&self) -> &str {\n        \"my_service\"\n    }\n\n    fn threads(&self) -> Option<usize> {\n        Some(1)\n    }\n}\n\nfn main() {\n    env_logger::Builder::from_default_env()\n        .filter_level(log::LevelFilter::Info)\n        .init();\n\n    let opt = Opt::parse_args();\n    let mut server = Server::new(Some(opt)).unwrap();\n\n    // Schedule bootstrap as a service instead of calling server.bootstrap() directly.\n    // The returned handle can be used to declare dependencies so that other services\n    // only start after bootstrap has finished.\n    let bootstrap_handle = server.bootstrap_as_a_service();\n\n    // Register our application service and get its handle.\n    let service_handle = server.add_service(MyService);\n\n    // MyService will not start until the bootstrap service has signaled that it is ready.\n    service_handle.add_dependency(&bootstrap_handle);\n\n    info!(\"Starting server — bootstrap will run as a service before MyService starts\");\n\n    server.run_forever();\n}\n"
  },
  {
    "path": "pingora-core/examples/client_cert.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n#![cfg_attr(not(feature = \"openssl\"), allow(unused))]\n\nuse std::any::Any;\nuse std::net::{IpAddr, Ipv4Addr, Ipv6Addr};\nuse std::sync::Arc;\n\nuse async_trait::async_trait;\nuse clap::Parser;\nuse http::header::{CONTENT_LENGTH, CONTENT_TYPE};\nuse http::{Response, StatusCode};\nuse pingora_core::apps::http_app::ServeHttp;\nuse pingora_core::listeners::tls::TlsSettings;\nuse pingora_core::listeners::TlsAccept;\nuse pingora_core::protocols::http::ServerSession;\nuse pingora_core::protocols::tls::TlsRef;\nuse pingora_core::server::configuration::Opt;\nuse pingora_core::server::Server;\nuse pingora_core::services::listening::Service;\nuse pingora_core::Result;\n#[cfg(feature = \"openssl\")]\nuse pingora_openssl::{\n    nid::Nid,\n    ssl::{NameType, SslFiletype, SslVerifyMode},\n    x509::{GeneralName, X509Name},\n};\n\n// Custom structure to hold TLS information\nstruct MyTlsInfo {\n    // SNI (Server Name Indication) from the TLS handshake\n    sni: Option<String>,\n    // SANs (Subject Alternative Names) from client certificate\n    sans: Vec<String>,\n    // Common Name (CN) from client certificate\n    common_name: Option<String>,\n}\n\nstruct MyApp;\n\n#[async_trait]\nimpl ServeHttp for MyApp {\n    async fn response(&self, session: &mut ServerSession) -> http::Response<Vec<u8>> {\n        static EMPTY_VEC: Vec<String> = vec![];\n\n        // Extract TLS info from the session's digest extensions\n        let my_tls_info = session\n            .digest()\n            .and_then(|digest| digest.ssl_digest.as_ref())\n            .and_then(|ssl_digest| ssl_digest.extension.get::<MyTlsInfo>());\n        let sni = my_tls_info\n            .and_then(|my_tls_info| my_tls_info.sni.as_deref())\n            .unwrap_or(\"<none>\");\n        let sans = my_tls_info\n            .map(|my_tls_info| &my_tls_info.sans)\n            .unwrap_or(&EMPTY_VEC);\n        let common_name = my_tls_info\n            .and_then(|my_tls_info| my_tls_info.common_name.as_deref())\n            .unwrap_or(\"<none>\");\n\n        // Create response message\n        let mut message = String::new();\n        message += &format!(\"Your SNI was: {sni}\\n\");\n        message += &format!(\"Your SANs were: {sans:?}\\n\");\n        message += &format!(\"Client Common Name (CN): {}\\n\", common_name);\n        let message = message.into_bytes();\n\n        Response::builder()\n            .status(StatusCode::OK)\n            .header(CONTENT_TYPE, \"text/plain\")\n            .header(CONTENT_LENGTH, message.len())\n            .body(message)\n            .unwrap()\n    }\n}\n\nstruct MyTlsCallbacks;\n\n#[async_trait]\nimpl TlsAccept for MyTlsCallbacks {\n    #[cfg(feature = \"openssl\")]\n    async fn handshake_complete_callback(\n        &self,\n        tls_ref: &TlsRef,\n    ) -> Option<Arc<dyn Any + Send + Sync>> {\n        // Here you can inspect the TLS connection and return an extension if needed.\n\n        // Extract SNI (Server Name Indication)\n        let sni = tls_ref\n            .servername(NameType::HOST_NAME)\n            .map(ToOwned::to_owned);\n\n        // Extract SAN (Subject Alternative Names) from the client certificate\n        let sans = tls_ref\n            .peer_certificate()\n            .and_then(|cert| cert.subject_alt_names())\n            .map_or(vec![], |sans| {\n                sans.into_iter()\n                    .filter_map(|san| san_to_string(&san))\n                    .collect::<Vec<_>>()\n            });\n\n        // Extract Common Name (CN) from the client certificate\n        let common_name = tls_ref.peer_certificate().and_then(|cert| {\n            let cn = cert.subject_name().entries_by_nid(Nid::COMMONNAME).next()?;\n            Some(cn.data().as_utf8().ok()?.to_string())\n        });\n\n        let tls_info = MyTlsInfo {\n            sni,\n            sans,\n            common_name,\n        };\n        Some(Arc::new(tls_info))\n    }\n}\n\n// Convert GeneralName of SAN to String representation\n#[cfg(feature = \"openssl\")]\nfn san_to_string(san: &GeneralName) -> Option<String> {\n    if let Some(dnsname) = san.dnsname() {\n        return Some(dnsname.to_owned());\n    }\n    if let Some(uri) = san.uri() {\n        return Some(uri.to_owned());\n    }\n    if let Some(email) = san.email() {\n        return Some(email.to_owned());\n    }\n    if let Some(ip) = san.ipaddress() {\n        return bytes_to_ip_addr(ip).map(|addr| addr.to_string());\n    }\n    None\n}\n\n// Convert byte slice to IpAddr\nfn bytes_to_ip_addr(bytes: &[u8]) -> Option<IpAddr> {\n    match bytes.len() {\n        4 => {\n            let addr = Ipv4Addr::new(bytes[0], bytes[1], bytes[2], bytes[3]);\n            Some(IpAddr::V4(addr))\n        }\n        16 => {\n            let mut octets = [0u8; 16];\n            octets.copy_from_slice(bytes);\n            let addr = Ipv6Addr::from(octets);\n            Some(IpAddr::V6(addr))\n        }\n        _ => None,\n    }\n}\n\n// This example demonstrates an HTTP server that requires client certificates.\n// The server extracts the SNI (Server Name Indication) from the TLS handshake and\n// SANs (Subject Alternative Names) from the client certificate, then returns them\n// as part of the HTTP response.\n//\n// ## How to run\n//\n//   cargo run -F openssl --example client_cert\n//\n//   # In another terminal, run the following command to test the server:\n//   cd pingora-core\n//   curl -k -i \\\n//     --cert examples/keys/clients/cert-1.pem --key examples/keys/clients/key-1.pem \\\n//     --resolve myapp.example.com:6196:127.0.0.1 \\\n//     https://myapp.example.com:6196/\n//   curl -k -i \\\n//     --cert examples/keys/clients/cert-2.pem --key examples/keys/clients/key-2.pem \\\n//     --resolve myapp.example.com:6196:127.0.0.1 \\\n//     https://myapp.example.com:6196/\n//   curl -k -i \\\n//     --cert examples/keys/clients/invalid-cert.pem --key examples/keys/clients/invalid-key.pem \\\n//     --resolve myapp.example.com:6196:127.0.0.1 \\\n//     https://myapp.example.com:6196/\n#[cfg(feature = \"openssl\")]\nfn main() -> Result<(), Box<dyn std::error::Error>> {\n    env_logger::init();\n\n    // read command line arguments\n    let opt = Opt::parse();\n    let mut my_server = Server::new(Some(opt))?;\n    my_server.bootstrap();\n\n    let mut my_app = Service::new(\"my app\".to_owned(), MyApp);\n\n    // Paths to server certificate, private key, and client CA certificate\n    let manifest_dir = env!(\"CARGO_MANIFEST_DIR\");\n    let server_cert_path = format!(\"{manifest_dir}/examples/keys/server/cert.pem\");\n    let server_key_path = format!(\"{manifest_dir}/examples/keys/server/key.pem\");\n    let client_ca_path = format!(\"{manifest_dir}/examples/keys/client-ca/cert.pem\");\n\n    // Create TLS settings with callbacks\n    let callbacks = Box::new(MyTlsCallbacks);\n    let mut tls_settings = TlsSettings::with_callbacks(callbacks)?;\n    // Set server certificate and private key\n    tls_settings.set_certificate_chain_file(&server_cert_path)?;\n    tls_settings.set_private_key_file(server_key_path, SslFiletype::PEM)?;\n    // Require client certificate\n    tls_settings.set_verify(SslVerifyMode::PEER | SslVerifyMode::FAIL_IF_NO_PEER_CERT);\n    // Set CA for client certificate verification\n    tls_settings.set_ca_file(&client_ca_path)?;\n    // Optionally, set the list of acceptable client CAs sent to the client\n    tls_settings.set_client_ca_list(X509Name::load_client_ca_file(&client_ca_path)?);\n\n    my_app.add_tls_with_settings(\"0.0.0.0:6196\", None, tls_settings);\n    my_server.add_service(my_app);\n\n    my_server.run_forever();\n}\n\n#[cfg(not(feature = \"openssl\"))]\nfn main() {\n    eprintln!(\"This example requires the 'openssl' feature to be enabled.\");\n}\n"
  },
  {
    "path": "pingora-core/examples/keys/client-ca/cert.pem",
    "content": "-----BEGIN CERTIFICATE-----\nMIICTjCCAfWgAwIBAgIULuUoq/di4EKmLyN0YwAkd6MQjv4wCgYIKoZIzj0EAwIw\ndTELMAkGA1UEBhMCVVMxEzARBgNVBAgMCkNhbGlmb3JuaWExFjAUBgNVBAcMDVNh\nbiBGcmFuY2lzY28xGDAWBgNVBAoMD0Nsb3VkZmxhcmUsIEluYzEfMB0GA1UEAwwW\nRXhhbXBsZSBDbGllbnQgUm9vdCBDQTAeFw0yNTExMTkwNDU5MjRaFw0zNTExMTcw\nNDU5MjRaMHUxCzAJBgNVBAYTAlVTMRMwEQYDVQQIDApDYWxpZm9ybmlhMRYwFAYD\nVQQHDA1TYW4gRnJhbmNpc2NvMRgwFgYDVQQKDA9DbG91ZGZsYXJlLCBJbmMxHzAd\nBgNVBAMMFkV4YW1wbGUgQ2xpZW50IFJvb3QgQ0EwWTATBgcqhkjOPQIBBggqhkjO\nPQMBBwNCAARxcxOAR4zUDPilKpMLiBzNs+HxdW6ZBlHVA7/0VyJtSPw03IdlbtFs\nFhgcIa8uQ9nrppHlrzploTA7cg7YWUoso2MwYTAPBgNVHRMBAf8EBTADAQH/MA4G\nA1UdDwEB/wQEAwIBBjAdBgNVHQ4EFgQUL6S83l9AGZmmwHh+64YlUtMQzZcwHwYD\nVR0jBBgwFoAUL6S83l9AGZmmwHh+64YlUtMQzZcwCgYIKoZIzj0EAwIDRwAwRAIg\ncohFQxG22J2YKw+DGAidU5u3mxtB/BALxIusqd+OfFUCIGmT2GHVxz1FwK2pJrM1\nFTWEcEbAw3r86iIVJBYP4qX6\n-----END CERTIFICATE-----\n"
  },
  {
    "path": "pingora-core/examples/keys/client-ca/key.pem",
    "content": "-----BEGIN EC PRIVATE KEY-----\nMHcCAQEEIJOxEQowpYL5VLNf+qaCEBhic8e26UyR0ku65Sk6gjMIoAoGCCqGSM49\nAwEHoUQDQgAEcXMTgEeM1Az4pSqTC4gczbPh8XVumQZR1QO/9FcibUj8NNyHZW7R\nbBYYHCGvLkPZ66aR5a86ZaEwO3IO2FlKLA==\n-----END EC PRIVATE KEY-----\n"
  },
  {
    "path": "pingora-core/examples/keys/clients/cert-1.pem",
    "content": "-----BEGIN CERTIFICATE-----\nMIICjjCCAjWgAwIBAgIUYUSqEzxm/oebfxxQmZEesZL2WFAwCgYIKoZIzj0EAwIw\ndTELMAkGA1UEBhMCVVMxEzARBgNVBAgMCkNhbGlmb3JuaWExFjAUBgNVBAcMDVNh\nbiBGcmFuY2lzY28xGDAWBgNVBAoMD0Nsb3VkZmxhcmUsIEluYzEfMB0GA1UEAwwW\nRXhhbXBsZSBDbGllbnQgUm9vdCBDQTAeFw0yNTExMTkwNTEyMThaFw0zNTExMTcw\nNTEyMThaMG8xCzAJBgNVBAYTAlVTMRMwEQYDVQQIDApDYWxpZm9ybmlhMRYwFAYD\nVQQHDA1TYW4gRnJhbmNpc2NvMRgwFgYDVQQKDA9DbG91ZGZsYXJlLCBJbmMxGTAX\nBgNVBAMMEGV4YW1wbGUtY2xpZW50LTEwWTATBgcqhkjOPQIBBggqhkjOPQMBBwNC\nAATDe6hBwpmE4Jt//sIWGWuBDYXHezVoFeoHsDzcWo6RwyHDfm7lvnACmqWAdRUV\n1GA7yfkzc1CaTqnvU8GjFdfXo4GoMIGlMAwGA1UdEwEB/wQCMAAwDgYDVR0PAQH/\nBAQDAgWgMBMGA1UdJQQMMAoGCCsGAQUFBwMCMDAGA1UdEQQpMCeGJXNwaWZmZTov\nL2V4YW1wbGUuY29tL2V4YW1wbGUtY2xpZW50LTEwHQYDVR0OBBYEFAjfTzgX+AVh\nM+BIaU0qTgINZWOdMB8GA1UdIwQYMBaAFC+kvN5fQBmZpsB4fuuGJVLTEM2XMAoG\nCCqGSM49BAMCA0cAMEQCIHyJDCvYKgxVthHcLjlEGW4Pj0Y7XnQUCJARa3jAUTd9\nAiB8tSXbo6J6Jhy6nasaxT1HAZwjgMVQwdo8O8UYOXXZpQ==\n-----END CERTIFICATE-----\n"
  },
  {
    "path": "pingora-core/examples/keys/clients/cert-2.pem",
    "content": "-----BEGIN CERTIFICATE-----\nMIIC0zCCAnmgAwIBAgIUVQlGCD9Zryvkh9G8GZXFBa2L9kQwCgYIKoZIzj0EAwIw\ndTELMAkGA1UEBhMCVVMxEzARBgNVBAgMCkNhbGlmb3JuaWExFjAUBgNVBAcMDVNh\nbiBGcmFuY2lzY28xGDAWBgNVBAoMD0Nsb3VkZmxhcmUsIEluYzEfMB0GA1UEAwwW\nRXhhbXBsZSBDbGllbnQgUm9vdCBDQTAeFw0yNTExMTkwODA5MDlaFw0zNTExMTcw\nODA5MDlaMG8xCzAJBgNVBAYTAlVTMRMwEQYDVQQIDApDYWxpZm9ybmlhMRYwFAYD\nVQQHDA1TYW4gRnJhbmNpc2NvMRgwFgYDVQQKDA9DbG91ZGZsYXJlLCBJbmMxGTAX\nBgNVBAMMEGV4YW1wbGUtY2xpZW50LTIwWTATBgcqhkjOPQIBBggqhkjOPQMBBwNC\nAAS2J10rq5Rt4TjhqEjHED0UPdceuzHUcw8doLC4StBIxJIrFk9Ag0g5ti9vN4fG\nkK6J11GXk/pBmu3O3s48Gsfgo4HsMIHpMAwGA1UdEwEB/wQCMAAwDgYDVR0PAQH/\nBAQDAgWgMBMGA1UdJQQMMAoGCCsGAQUFBwMCMHQGA1UdEQRtMGuGJXNwaWZmZTov\nL2V4YW1wbGUuY29tL2V4YW1wbGUtY2xpZW50LTKCFGNsaWVudC0yLmV4YW1wbGUu\nY29thwR/AAABhxAAAAAAAAAAAAAAAAAAAAABgRRjbGllbnQtMkBleGFtcGxlLmNv\nbTAdBgNVHQ4EFgQUGHwnr7Ube1hqsodgcxJkfYuCKE8wHwYDVR0jBBgwFoAUL6S8\n3l9AGZmmwHh+64YlUtMQzZcwCgYIKoZIzj0EAwIDSAAwRQIgK4JL1OO2nB7MqvGW\ny2nbH4yYMu2jUkYhw9HFLUG2B6MCIQC4iDWKXp7R977LvuaaQaNcMmbGysrmfo8V\nwOmp1JGOtA==\n-----END CERTIFICATE-----\n"
  },
  {
    "path": "pingora-core/examples/keys/clients/invalid-cert.pem",
    "content": "-----BEGIN CERTIFICATE-----\nMIICjzCCAjWgAwIBAgIUHYIVFYFooGVi2bNlk5R6GsbDKqUwCgYIKoZIzj0EAwIw\ndTELMAkGA1UEBhMCVVMxEzARBgNVBAgMCkNhbGlmb3JuaWExFjAUBgNVBAcMDVNh\nbiBGcmFuY2lzY28xGDAWBgNVBAoMD0Nsb3VkZmxhcmUsIEluYzEfMB0GA1UEAwwW\nRXhhbXBsZSBDbGllbnQgUm9vdCBDQTAeFw0yNTExMTkwODEzNDJaFw0zNTExMTcw\nODEzNDJaMG8xCzAJBgNVBAYTAlVTMRMwEQYDVQQIDApDYWxpZm9ybmlhMRYwFAYD\nVQQHDA1TYW4gRnJhbmNpc2NvMRgwFgYDVQQKDA9DbG91ZGZsYXJlLCBJbmMxGTAX\nBgNVBAMMEGV4YW1wbGUtY2xpZW50LTMwWTATBgcqhkjOPQIBBggqhkjOPQMBBwNC\nAATGKppMkUDsNvpzPPPiKmz53bbyIJPemIq5OdgJli8XZUFozxroJuFKhUuJOuFF\nJns2pzLHewIDzFXgErPqPxA/o4GoMIGlMAwGA1UdEwEB/wQCMAAwDgYDVR0PAQH/\nBAQDAgWgMBMGA1UdJQQMMAoGCCsGAQUFBwMCMDAGA1UdEQQpMCeGJXNwaWZmZTov\nL2V4YW1wbGUuY29tL2V4YW1wbGUtY2xpZW50LTMwHQYDVR0OBBYEFDV/v0zsiC/t\naomzxKa0jJ4SlmSzMB8GA1UdIwQYMBaAFK04aCtyumAb4PEMnh9OXLW7EIJSMAoG\nCCqGSM49BAMCA0gAMEUCIH/wxvS0ae8DF1QteE+2FDOd/G2WeBMjsS8A6VyebAru\nAiEAl2vjq0KePvM2X0jTZ/+RMJO33HOpYr0+PZw6FAa+aaw=\n-----END CERTIFICATE-----\n"
  },
  {
    "path": "pingora-core/examples/keys/clients/invalid-key.pem",
    "content": "-----BEGIN EC PRIVATE KEY-----\nMHcCAQEEIFyLneOGHgjTBS8I2GB8kF0LHgDS/eTJBSDNS4PAkJ0JoAoGCCqGSM49\nAwEHoUQDQgAExiqaTJFA7Db6czzz4ips+d228iCT3piKuTnYCZYvF2VBaM8a6Cbh\nSoVLiTrhRSZ7Nqcyx3sCA8xV4BKz6j8QPw==\n-----END EC PRIVATE KEY-----\n"
  },
  {
    "path": "pingora-core/examples/keys/clients/key-1.pem",
    "content": "-----BEGIN EC PRIVATE KEY-----\nMHcCAQEEIFNioASifzPy0Fcp+qmMoMUhFOJGLki20ygISqZb+HY1oAoGCCqGSM49\nAwEHoUQDQgAEw3uoQcKZhOCbf/7CFhlrgQ2Fx3s1aBXqB7A83FqOkcMhw35u5b5w\nApqlgHUVFdRgO8n5M3NQmk6p71PBoxXX1w==\n-----END EC PRIVATE KEY-----\n"
  },
  {
    "path": "pingora-core/examples/keys/clients/key-2.pem",
    "content": "-----BEGIN EC PRIVATE KEY-----\nMHcCAQEEICd8DwjvpvE6nIKKKH2smrnLBM5zQyIkAKwBCiiRZGGsoAoGCCqGSM49\nAwEHoUQDQgAEtiddK6uUbeE44ahIxxA9FD3XHrsx1HMPHaCwuErQSMSSKxZPQINI\nObYvbzeHxpCuiddRl5P6QZrtzt7OPBrH4A==\n-----END EC PRIVATE KEY-----\n"
  },
  {
    "path": "pingora-core/examples/keys/server/cert.pem",
    "content": "-----BEGIN CERTIFICATE-----\nMIICVzCCAf6gAwIBAgIUYGbx/r4kY40a+zNq7IW/1lsvzk0wCgYIKoZIzj0EAwIw\nbDELMAkGA1UEBhMCVVMxEzARBgNVBAgMCkNhbGlmb3JuaWExFjAUBgNVBAcMDVNh\nbiBGcmFuY2lzY28xGDAWBgNVBAoMD0Nsb3VkZmxhcmUsIEluYzEWMBQGA1UEAwwN\nb3BlbnJ1c3R5Lm9yZzAeFw0yNTExMTkwNDUxMzdaFw0zNTExMTcwNDUxMzdaMGwx\nCzAJBgNVBAYTAlVTMRMwEQYDVQQIDApDYWxpZm9ybmlhMRYwFAYDVQQHDA1TYW4g\nRnJhbmNpc2NvMRgwFgYDVQQKDA9DbG91ZGZsYXJlLCBJbmMxFjAUBgNVBAMMDW9w\nZW5ydXN0eS5vcmcwWTATBgcqhkjOPQIBBggqhkjOPQMBBwNCAAT9EuNEw3e3syHW\nSNnyJw7QVtOzDlILlt6F+jXT8UMBoMn4OnwC7AFlV8XzR9UpYSf1yq7Raps7c8TU\nW9YF6ee4o34wfDAdBgNVHQ4EFgQU6B2YXLmWaboIZsf9YOCePRQXrO4wHwYDVR0j\nBBgwFoAU6B2YXLmWaboIZsf9YOCePRQXrO4wDwYDVR0TAQH/BAUwAwEB/zApBgNV\nHREEIjAggg8qLm9wZW5ydXN0eS5vcmeCDW9wZW5ydXN0eS5vcmcwCgYIKoZIzj0E\nAwIDRwAwRAIgcSThJ5CWjuyWKfHbR+RuJ/9DtH1ag/47OolMQAvOczsCIDKVgPO/\nA69bTOk4sq0y92YBBbe3hF82KrsgTR3nlkKF\n-----END CERTIFICATE-----\n"
  },
  {
    "path": "pingora-core/examples/keys/server/key.pem",
    "content": "-----BEGIN PRIVATE KEY-----\nMIGHAgEAMBMGByqGSM49AgEGCCqGSM49AwEHBG0wawIBAQQgTAnVhDuKvV5epzX4\nuuC8kEZL2vUPI49gUmS5kM+j5VWhRANCAAT9EuNEw3e3syHWSNnyJw7QVtOzDlIL\nlt6F+jXT8UMBoMn4OnwC7AFlV8XzR9UpYSf1yq7Raps7c8TUW9YF6ee4\n-----END PRIVATE KEY-----\n"
  },
  {
    "path": "pingora-core/examples/service_dependencies.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! Example demonstrating service dependency management.\n//!\n//! This example shows how services can declare dependencies on other services using\n//! a fluent API with [`ServiceHandle`] references, ensuring they start in the correct\n//! order and wait for dependencies to be ready.\n//!\n//! # Running the example\n//!\n//! ```bash\n//! cargo run --example service_dependencies --package pingora-core\n//! ```\n//!\n//! Expected output:\n//! - DatabaseService starts and initializes (takes 2 seconds)\n//! - CacheService starts and initializes (takes 1 second)\n//! - ApiService waits for both dependencies, then starts\n//!\n//! # Key Features Demonstrated\n//!\n//! - Fluent API for declaring dependencies via [`ServiceHandle::add_dependency()`]\n//! - Type-safe dependency declaration (no strings)\n//! - Multiple ways to implement services based on readiness needs:\n//!   - **DatabaseService**: Custom readiness timing (uses `ServiceWithDependencies`)\n//!   - **CacheService**: Ready immediately (uses `Service`)\n//!   - **ApiService**: Ready immediately (uses `Service`)\n//! - Automatic dependency ordering and validation\n//! - Prevention of typos in service names (compile-time safety)\n\nuse async_trait::async_trait;\nuse log::info;\nuse pingora_core::server::configuration::Opt;\n#[cfg(unix)]\nuse pingora_core::server::ListenFds;\nuse pingora_core::server::{Server, ShutdownWatch};\nuse pingora_core::services::{Service, ServiceWithDependents};\n// DatabaseService needs to control readiness timing\nuse pingora_core::services::ServiceReadyNotifier;\nuse std::sync::Arc;\nuse tokio::sync::Mutex;\nuse tokio::time::{sleep, Duration};\n\n/// A custom service that delays signaling ready until initialization is complete\npub struct DatabaseService {\n    connection_string: Arc<Mutex<Option<String>>>,\n}\n\nimpl DatabaseService {\n    fn new() -> Self {\n        Self {\n            connection_string: Arc::new(Mutex::new(None)),\n        }\n    }\n\n    fn get_connection_string(&self) -> Arc<Mutex<Option<String>>> {\n        self.connection_string.clone()\n    }\n}\n\n#[async_trait]\nimpl ServiceWithDependents for DatabaseService {\n    async fn start_service(\n        &mut self,\n        #[cfg(unix)] _fds: Option<ListenFds>,\n        mut shutdown: ShutdownWatch,\n        _listeners_per_fd: usize,\n        ready_notifier: ServiceReadyNotifier,\n    ) {\n        info!(\"DatabaseService: Starting initialization...\");\n\n        // Simulate database connection setup\n        sleep(Duration::from_secs(2)).await;\n\n        // Store the connection string\n        {\n            let mut conn = self.connection_string.lock().await;\n            *conn = Some(\"postgresql://localhost:5432/mydb\".to_string());\n        }\n\n        info!(\"DatabaseService: Initialization complete, signaling ready\");\n\n        // Signal that the service is ready\n        ready_notifier.notify_ready();\n\n        // Keep running until shutdown\n        shutdown.changed().await.ok();\n        info!(\"DatabaseService: Shutting down\");\n    }\n\n    fn name(&self) -> &str {\n        \"database\"\n    }\n\n    fn threads(&self) -> Option<usize> {\n        Some(1)\n    }\n}\n\n/// A cache service that uses the simplified API\n/// Signals ready immediately (using default implementation)\npub struct CacheService;\n\n#[async_trait]\nimpl Service for CacheService {\n    // Uses default start_service implementation which signals ready immediately\n\n    async fn start_service(\n        &mut self,\n        #[cfg(unix)] _fds: Option<ListenFds>,\n        mut shutdown: ShutdownWatch,\n        _listeners_per_fd: usize,\n    ) {\n        info!(\"CacheService: Starting (ready immediately)...\");\n\n        // Simulate cache warmup\n        sleep(Duration::from_secs(1)).await;\n        info!(\"CacheService: Warmup complete\");\n\n        // Keep running until shutdown\n        shutdown.changed().await.ok();\n        info!(\"CacheService: Shutting down\");\n    }\n\n    fn name(&self) -> &str {\n        \"cache\"\n    }\n\n    fn threads(&self) -> Option<usize> {\n        Some(1)\n    }\n}\n\n/// An API service that depends on both database and cache\n/// Uses the simplest API - signals ready immediately and just implements [Service]\npub struct ApiService {\n    db_connection: Arc<Mutex<Option<String>>>,\n}\n\nimpl ApiService {\n    fn new(db_connection: Arc<Mutex<Option<String>>>) -> Self {\n        Self { db_connection }\n    }\n}\n\n#[async_trait]\nimpl Service for ApiService {\n    // Uses default start_service - signals ready immediately\n\n    async fn start_service(\n        &mut self,\n        #[cfg(unix)] _fds: Option<ListenFds>,\n        mut shutdown: ShutdownWatch,\n        _listeners_per_fd: usize,\n    ) {\n        info!(\"ApiService: Starting (dependencies should be ready)...\");\n\n        // Verify database connection is available\n        {\n            let conn = self.db_connection.lock().await;\n            if let Some(conn_str) = &*conn {\n                info!(\"ApiService: Using database connection: {}\", conn_str);\n            } else {\n                panic!(\"ApiService: Database connection not available!\");\n            }\n        }\n\n        info!(\"ApiService: Ready to serve requests\");\n\n        // Keep running until shutdown\n        shutdown.changed().await.ok();\n        info!(\"ApiService: Shutting down\");\n    }\n\n    fn name(&self) -> &str {\n        \"api\"\n    }\n\n    fn threads(&self) -> Option<usize> {\n        Some(1)\n    }\n}\n\nfn main() {\n    env_logger::Builder::from_default_env()\n        .filter_level(log::LevelFilter::Info)\n        .init();\n\n    info!(\"Starting server with service dependencies...\");\n\n    let opt = Opt::parse_args();\n    let mut server = Server::new(Some(opt)).unwrap();\n    server.bootstrap();\n\n    // Create the database service\n    let db_service = DatabaseService::new();\n    let db_connection = db_service.get_connection_string();\n\n    // Create services\n    let cache_service = CacheService;\n    let api_service = ApiService::new(db_connection);\n\n    // Add services and get their handles\n    let db_handle = server.add_service(db_service);\n    let cache_handle = server.add_service(cache_service);\n    let api_handle = server.add_service(api_service);\n\n    // Declare dependencies using the fluent API\n    // The API service will not start until both dependencies signal ready\n    api_handle.add_dependency(db_handle);\n    api_handle.add_dependency(&cache_handle);\n\n    info!(\"Services configured. Starting server...\");\n    info!(\"Expected startup order:\");\n    info!(\"  1. database (will initialize for 2 seconds)\");\n    info!(\"  2. cache (will initialize for 1 second)\");\n    info!(\"  3. api (will wait for both, then start)\");\n    info!(\"\");\n    info!(\"Press Ctrl+C to shut down\");\n\n    server.run_forever();\n}\n"
  },
  {
    "path": "pingora-core/src/apps/http_app.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! A simple HTTP application trait that maps a request to a response\n\nuse async_trait::async_trait;\nuse http::Response;\nuse log::{debug, error, trace};\nuse pingora_http::ResponseHeader;\nuse std::sync::Arc;\n\nuse crate::apps::{HttpPersistentSettings, HttpServerApp, HttpServerOptions, ReusedHttpStream};\nuse crate::modules::http::{HttpModules, ModuleBuilder};\nuse crate::protocols::http::v2::server::H2Options;\nuse crate::protocols::http::HttpTask;\nuse crate::protocols::http::ServerSession;\nuse crate::server::ShutdownWatch;\n\n/// This trait defines how to map a request to a response\n#[async_trait]\npub trait ServeHttp {\n    /// Define the mapping from a request to a response.\n    /// Note that the request header is already read, but the implementation needs to read the\n    /// request body if any.\n    ///\n    /// # Limitation\n    /// In this API, the entire response has to be generated before the end of this call.\n    /// So it is not suitable for streaming response or interactive communications.\n    /// Users need to implement their own [`super::HttpServerApp`] for those use cases.\n    async fn response(&self, http_session: &mut ServerSession) -> Response<Vec<u8>>;\n}\n\n// TODO: remove this in favor of HttpServer?\n#[async_trait]\nimpl<SV> HttpServerApp for SV\nwhere\n    SV: ServeHttp + Send + Sync,\n{\n    async fn process_new_http(\n        self: &Arc<Self>,\n        mut http: ServerSession,\n        shutdown: &ShutdownWatch,\n    ) -> Option<ReusedHttpStream> {\n        match http.read_request().await {\n            Ok(res) => match res {\n                false => {\n                    debug!(\"Failed to read request header\");\n                    return None;\n                }\n                true => {\n                    debug!(\"Successfully get a new request\");\n                }\n            },\n            Err(e) => {\n                error!(\"HTTP server fails to read from downstream: {e}\");\n                return None;\n            }\n        }\n        trace!(\"{:?}\", http.req_header());\n        if *shutdown.borrow() {\n            http.set_keepalive(None);\n        } else {\n            http.set_keepalive(Some(60));\n        }\n        let new_response = self.response(&mut http).await;\n        let (parts, body) = new_response.into_parts();\n        let resp_header: ResponseHeader = parts.into();\n        match http.write_response_header(Box::new(resp_header)).await {\n            Ok(()) => {\n                debug!(\"HTTP response header done.\");\n            }\n            Err(e) => {\n                error!(\n                    \"HTTP server fails to write to downstream: {e}, {}\",\n                    http.request_summary()\n                );\n            }\n        }\n        if !body.is_empty() {\n            // TODO: check if chunked encoding is needed\n            match http.write_response_body(body.into(), true).await {\n                Ok(_) => debug!(\"HTTP response written.\"),\n                Err(e) => error!(\n                    \"HTTP server fails to write to downstream: {e}, {}\",\n                    http.request_summary()\n                ),\n            }\n        }\n        let persistent_settings = HttpPersistentSettings::for_session(&http);\n        match http.finish().await {\n            Ok(c) => c.map(|s| ReusedHttpStream::new(s, Some(persistent_settings))),\n            Err(e) => {\n                error!(\"HTTP server fails to finish the request: {e}\");\n                None\n            }\n        }\n    }\n}\n\n/// A helper struct for HTTP server with http modules embedded\npub struct HttpServer<SV> {\n    app: SV,\n    modules: HttpModules,\n    pub server_options: Option<HttpServerOptions>,\n    pub h2_options: Option<H2Options>,\n}\n\nimpl<SV> HttpServer<SV> {\n    /// Create a new [HttpServer] with the given app which implements [ServeHttp]\n    pub fn new_app(app: SV) -> Self {\n        HttpServer {\n            app,\n            modules: HttpModules::new(),\n            server_options: None,\n            h2_options: None,\n        }\n    }\n\n    /// Add [ModuleBuilder] to this [HttpServer]\n    pub fn add_module(&mut self, module: ModuleBuilder) {\n        self.modules.add_module(module)\n    }\n}\n\n#[async_trait]\nimpl<SV> HttpServerApp for HttpServer<SV>\nwhere\n    SV: ServeHttp + Send + Sync,\n{\n    async fn process_new_http(\n        self: &Arc<Self>,\n        mut http: ServerSession,\n        shutdown: &ShutdownWatch,\n    ) -> Option<ReusedHttpStream> {\n        match http.read_request().await {\n            Ok(res) => match res {\n                false => {\n                    debug!(\"Failed to read request header\");\n                    return None;\n                }\n                true => {\n                    debug!(\"Successfully get a new request\");\n                }\n            },\n            Err(e) => {\n                error!(\"HTTP server fails to read from downstream: {e}\");\n                return None;\n            }\n        }\n        trace!(\"{:?}\", http.req_header());\n        if *shutdown.borrow() {\n            http.set_keepalive(None);\n        } else {\n            http.set_keepalive(Some(60));\n        }\n        let mut module_ctx = self.modules.build_ctx();\n        let req = http.req_header_mut();\n        module_ctx.request_header_filter(req).await.ok()?;\n        let new_response = self.app.response(&mut http).await;\n        let (parts, body) = new_response.into_parts();\n        let mut resp_header: ResponseHeader = parts.into();\n        module_ctx\n            .response_header_filter(&mut resp_header, body.is_empty())\n            .await\n            .ok()?;\n\n        let task = HttpTask::Header(Box::new(resp_header), body.is_empty());\n        trace!(\"{task:?}\");\n\n        match http.response_duplex_vec(vec![task]).await {\n            Ok(_) => {\n                debug!(\"HTTP response header done.\");\n            }\n            Err(e) => {\n                error!(\n                    \"HTTP server fails to write to downstream: {e}, {}\",\n                    http.request_summary()\n                );\n            }\n        }\n\n        let mut body = Some(body.into());\n        module_ctx.response_body_filter(&mut body, true).ok()?;\n\n        let task = HttpTask::Body(body, true);\n\n        trace!(\"{task:?}\");\n\n        // TODO: check if chunked encoding is needed\n        match http.response_duplex_vec(vec![task]).await {\n            Ok(_) => debug!(\"HTTP response written.\"),\n            Err(e) => error!(\n                \"HTTP server fails to write to downstream: {e}, {}\",\n                http.request_summary()\n            ),\n        }\n        let persistent_settings = HttpPersistentSettings::for_session(&http);\n        match http.finish().await {\n            Ok(c) => c.map(|s| ReusedHttpStream::new(s, Some(persistent_settings))),\n            Err(e) => {\n                error!(\"HTTP server fails to finish the request: {e}\");\n                None\n            }\n        }\n    }\n\n    fn h2_options(&self) -> Option<H2Options> {\n        self.h2_options.clone()\n    }\n\n    fn server_options(&self) -> Option<&HttpServerOptions> {\n        self.server_options.as_ref()\n    }\n}\n"
  },
  {
    "path": "pingora-core/src/apps/mod.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! The abstraction and implementation interface for service application logic\n\npub mod http_app;\npub mod prometheus_http_app;\n\nuse crate::server::ShutdownWatch;\nuse async_trait::async_trait;\nuse log::{debug, error};\nuse std::future::poll_fn;\nuse std::sync::Arc;\n\nuse crate::protocols::http::v2::server;\nuse crate::protocols::http::ServerSession;\nuse crate::protocols::Digest;\nuse crate::protocols::Stream;\nuse crate::protocols::ALPN;\n\n// https://datatracker.ietf.org/doc/html/rfc9113#section-3.4\nconst H2_PREFACE: &[u8] = b\"PRI * HTTP/2.0\\r\\n\\r\\nSM\\r\\n\\r\\n\";\n\n#[async_trait]\n/// This trait defines the interface of a transport layer (TCP or TLS) application.\npub trait ServerApp {\n    /// Whenever a new connection is established, this function will be called with the established\n    /// [`Stream`] object provided.\n    ///\n    /// The application can do whatever it wants with the `session`.\n    ///\n    /// After processing the `session`, if the `session`'s connection is reusable, This function\n    /// can return it to the service by returning `Some(session)`. The returned `session` will be\n    /// fed to another [`Self::process_new()`] for another round of processing.\n    /// If not reusable, `None` should be returned.\n    ///\n    /// The `shutdown` argument will change from `false` to `true` when the server receives a\n    /// signal to shutdown. This argument allows the application to react accordingly.\n    async fn process_new(\n        self: &Arc<Self>,\n        mut session: Stream,\n        // TODO: make this ShutdownWatch so that all task can await on this event\n        shutdown: &ShutdownWatch,\n    ) -> Option<Stream>;\n\n    /// This callback will be called once after the service stops listening to its endpoints.\n    async fn cleanup(&self) {}\n}\n#[non_exhaustive]\n#[derive(Default)]\n/// HTTP Server options that control how the server handles some transport types.\npub struct HttpServerOptions {\n    /// Allow HTTP/2 for plaintext.\n    pub h2c: bool,\n\n    /// Allow proxying CONNECT requests when handling HTTP traffic.\n    ///\n    /// When disabled, CONNECT requests are rejected with 405 by proxy services.\n    pub allow_connect_method_proxying: bool,\n\n    #[doc(hidden)]\n    pub force_custom: bool,\n\n    /// Maximum number of requests that this connection will handle. This is\n    /// equivalent to [Nginx's keepalive requests](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#keepalive_requests)\n    /// which says:\n    ///\n    /// > Closing connections periodically is necessary to free per-connection\n    /// > memory allocations. Therefore, using too high maximum number of\n    /// > requests could result in excessive memory usage and not recommended.\n    ///\n    /// Unlike nginx, the default behavior here is _no limit_.\n    pub keepalive_request_limit: Option<u32>,\n}\n\n#[derive(Debug, Clone)]\npub struct HttpPersistentSettings {\n    keepalive_timeout: Option<u64>,\n    keepalive_reuses_remaining: Option<u32>,\n}\n\nimpl HttpPersistentSettings {\n    pub fn for_session(session: &ServerSession) -> Self {\n        HttpPersistentSettings {\n            keepalive_timeout: session.get_keepalive(),\n            keepalive_reuses_remaining: session.get_keepalive_reuses_remaining(),\n        }\n    }\n\n    pub fn apply_to_session(self, session: &mut ServerSession) {\n        let Self {\n            keepalive_timeout,\n            mut keepalive_reuses_remaining,\n        } = self;\n\n        // Reduce the number of times the connection for this session can be\n        // reused by one. A session with reuse count of zero won't be reused\n        if let Some(reuses) = keepalive_reuses_remaining.as_mut() {\n            *reuses = reuses.saturating_sub(1);\n        }\n\n        session.set_keepalive(keepalive_timeout);\n        session.set_keepalive_reuses_remaining(keepalive_reuses_remaining);\n    }\n}\n\n#[derive(Debug)]\npub struct ReusedHttpStream {\n    stream: Stream,\n    persistent_settings: Option<HttpPersistentSettings>,\n}\n\nimpl ReusedHttpStream {\n    pub fn new(stream: Stream, persistent_settings: Option<HttpPersistentSettings>) -> Self {\n        ReusedHttpStream {\n            stream,\n            persistent_settings,\n        }\n    }\n\n    pub fn consume(self) -> (Stream, Option<HttpPersistentSettings>) {\n        (self.stream, self.persistent_settings)\n    }\n}\n\n/// This trait defines the interface of an HTTP application.\n#[async_trait]\npub trait HttpServerApp {\n    /// Similar to the [`ServerApp`], this function is called whenever a new HTTP session is established.\n    ///\n    /// After successful processing, [`ServerSession::finish()`] can be called to return an optionally reusable\n    /// connection back to the service. The caller needs to make sure that the connection is in a reusable state\n    /// i.e., no error or incomplete read or write headers or bodies. Otherwise a `None` should be returned.\n    async fn process_new_http(\n        self: &Arc<Self>,\n        mut session: ServerSession,\n        // TODO: make this ShutdownWatch so that all task can await on this event\n        shutdown: &ShutdownWatch,\n    ) -> Option<ReusedHttpStream>;\n\n    /// Provide options on how HTTP/2 connection should be established. This function will be called\n    /// every time a new HTTP/2 **connection** needs to be established.\n    ///\n    /// A `None` means to use the built-in default options. See [`server::H2Options`] for more details.\n    fn h2_options(&self) -> Option<server::H2Options> {\n        None\n    }\n\n    /// Provide HTTP server options used to override default behavior. This function will be called\n    /// every time a new connection is processed.\n    ///\n    /// A `None` means no server options will be applied.\n    fn server_options(&self) -> Option<&HttpServerOptions> {\n        None\n    }\n\n    async fn http_cleanup(&self) {}\n\n    #[doc(hidden)]\n    async fn process_custom_session(\n        self: Arc<Self>,\n        _stream: Stream,\n        _shutdown: &ShutdownWatch,\n    ) -> Option<Stream> {\n        None\n    }\n}\n\n#[async_trait]\nimpl<T> ServerApp for T\nwhere\n    T: HttpServerApp + Send + Sync + 'static,\n{\n    async fn process_new(\n        self: &Arc<Self>,\n        mut stream: Stream,\n        shutdown: &ShutdownWatch,\n    ) -> Option<Stream> {\n        let mut h2c = self.server_options().as_ref().map_or(false, |o| o.h2c);\n        let custom = self\n            .server_options()\n            .as_ref()\n            .map_or(false, |o| o.force_custom);\n\n        // try to read h2 preface\n        if h2c && !custom {\n            let mut buf = [0u8; H2_PREFACE.len()];\n            let peeked = stream\n                .try_peek(&mut buf)\n                .await\n                .map_err(|e| {\n                    // this error is normal when h1 reuse and close the connection\n                    debug!(\"Read error while peeking h2c preface {e}\");\n                    e\n                })\n                .ok()?;\n            // not all streams support peeking\n            if peeked {\n                // turn off h2c (use h1) if h2 preface doesn't exist\n                h2c = buf == H2_PREFACE;\n            }\n        }\n        if h2c || matches!(stream.selected_alpn_proto(), Some(ALPN::H2)) {\n            // create a shared connection digest\n            let digest = Arc::new(Digest {\n                ssl_digest: stream.get_ssl_digest(),\n                // TODO: log h2 handshake time\n                timing_digest: stream.get_timing_digest(),\n                proxy_digest: stream.get_proxy_digest(),\n                socket_digest: stream.get_socket_digest(),\n            });\n\n            let h2_options = self.h2_options();\n            let h2_conn = server::handshake(stream, h2_options).await;\n            let mut h2_conn = match h2_conn {\n                Err(e) => {\n                    error!(\"H2 handshake error {e}\");\n                    return None;\n                }\n                Ok(c) => c,\n            };\n\n            let mut shutdown = shutdown.clone();\n            loop {\n                // this loop ends when the client decides to close the h2 conn\n                // TODO: add a timeout?\n                let h2_stream = tokio::select! {\n                    _ = shutdown.changed() => {\n                        h2_conn.graceful_shutdown();\n                        let _ = poll_fn(|cx| h2_conn.poll_closed(cx))\n                            .await.map_err(|e| error!(\"H2 error waiting for shutdown {e}\"));\n                        return None;\n                    }\n                    h2_stream = server::HttpSession::from_h2_conn(&mut h2_conn, digest.clone()) => h2_stream\n                };\n                let h2_stream = match h2_stream {\n                    Err(e) => {\n                        // It is common for the client to just disconnect TCP without properly\n                        // closing H2. So we don't log the errors here\n                        debug!(\"H2 error when accepting new stream {e}\");\n                        return None;\n                    }\n                    Ok(s) => s?, // None means the connection is ready to be closed\n                };\n                let app = self.clone();\n                let shutdown = shutdown.clone();\n                pingora_runtime::current_handle().spawn(async move {\n                    // Note, `PersistentSettings` not currently relevant for h2\n                    app.process_new_http(ServerSession::new_http2(h2_stream), &shutdown)\n                        .await;\n                });\n            }\n        } else if custom || matches!(stream.selected_alpn_proto(), Some(ALPN::Custom(_))) {\n            return self.clone().process_custom_session(stream, shutdown).await;\n        } else {\n            // No ALPN or ALPN::H1 and h2c was not configured, fallback to HTTP/1.1\n            let mut session = ServerSession::new_http1(stream);\n            if *shutdown.borrow() {\n                // stop downstream from reusing if this service is shutting down soon\n                session.set_keepalive(None);\n            } else {\n                // default 60s\n                session.set_keepalive(Some(60));\n            }\n            session.set_keepalive_reuses_remaining(\n                self.server_options()\n                    .and_then(|opts| opts.keepalive_request_limit),\n            );\n\n            let mut result = self.process_new_http(session, shutdown).await;\n            while let Some((stream, persistent_settings)) = result.map(|r| r.consume()) {\n                let mut session = ServerSession::new_http1(stream);\n                if let Some(persistent_settings) = persistent_settings {\n                    persistent_settings.apply_to_session(&mut session);\n                }\n\n                result = self.process_new_http(session, shutdown).await;\n            }\n        }\n        None\n    }\n\n    async fn cleanup(&self) {\n        self.http_cleanup().await;\n    }\n}\n"
  },
  {
    "path": "pingora-core/src/apps/prometheus_http_app.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! An HTTP application that reports Prometheus metrics.\n\nuse async_trait::async_trait;\nuse http::Response;\nuse prometheus::{Encoder, TextEncoder};\n\nuse super::http_app::HttpServer;\nuse crate::apps::http_app::ServeHttp;\nuse crate::modules::http::compression::ResponseCompressionBuilder;\nuse crate::protocols::http::ServerSession;\n\n/// An HTTP application that reports Prometheus metrics.\n///\n/// This application will report all the [static metrics](https://docs.rs/prometheus/latest/prometheus/index.html#static-metrics)\n/// collected via the [Prometheus](https://docs.rs/prometheus/) crate;\npub struct PrometheusHttpApp;\n\n#[async_trait]\nimpl ServeHttp for PrometheusHttpApp {\n    async fn response(&self, _http_session: &mut ServerSession) -> Response<Vec<u8>> {\n        let encoder = TextEncoder::new();\n        let metric_families = prometheus::gather();\n        let mut buffer = vec![];\n        encoder.encode(&metric_families, &mut buffer).unwrap();\n        Response::builder()\n            .status(200)\n            .header(http::header::CONTENT_TYPE, encoder.format_type())\n            .header(http::header::CONTENT_LENGTH, buffer.len())\n            .body(buffer)\n            .unwrap()\n    }\n}\n\n/// The [HttpServer] for [PrometheusHttpApp]\n///\n/// This type provides the functionality of [PrometheusHttpApp] with compression enabled\npub type PrometheusServer = HttpServer<PrometheusHttpApp>;\n\nimpl PrometheusServer {\n    pub fn new() -> Self {\n        let mut server = Self::new_app(PrometheusHttpApp);\n        // enable gzip level 7 compression\n        server.add_module(ResponseCompressionBuilder::enable(7));\n        server\n    }\n}\n"
  },
  {
    "path": "pingora-core/src/connectors/http/custom/mod.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\nuse async_trait::async_trait;\nuse std::time::Duration;\n\nuse pingora_error::Result;\n\nuse crate::{\n    protocols::{http::custom::client::Session, Stream},\n    upstreams::peer::Peer,\n};\n\n// Either returns a Custom Session or the Stream for creating a new H1 session as a fallback.\npub enum Connection<S: Session> {\n    Session(S),\n    Stream(Stream),\n}\n#[doc(hidden)]\n#[async_trait]\npub trait Connector: Send + Sync + Unpin + 'static {\n    type Session: Session;\n\n    async fn get_http_session<P: Peer + Send + Sync + 'static>(\n        &self,\n        peer: &P,\n    ) -> Result<(Connection<Self::Session>, bool)>;\n\n    async fn reused_http_session<P: Peer + Send + Sync + 'static>(\n        &self,\n        peer: &P,\n    ) -> Option<Self::Session>;\n\n    async fn release_http_session<P: Peer + Send + Sync + 'static>(\n        &self,\n        mut session: Self::Session,\n        peer: &P,\n        idle_timeout: Option<Duration>,\n    );\n}\n\n#[doc(hidden)]\n#[async_trait]\nimpl Connector for () {\n    type Session = ();\n\n    async fn get_http_session<P: Peer + Send + Sync + 'static>(\n        &self,\n        _peer: &P,\n    ) -> Result<(Connection<Self::Session>, bool)> {\n        unreachable!(\"connector: get_http_session\")\n    }\n\n    async fn reused_http_session<P: Peer + Send + Sync + 'static>(\n        &self,\n        _peer: &P,\n    ) -> Option<Self::Session> {\n        unreachable!(\"connector: reused_http_session\")\n    }\n\n    async fn release_http_session<P: Peer + Send + Sync + 'static>(\n        &self,\n        _session: Self::Session,\n        _peer: &P,\n        _idle_timeout: Option<Duration>,\n    ) {\n        unreachable!(\"connector: release_http_session\")\n    }\n}\n"
  },
  {
    "path": "pingora-core/src/connectors/http/mod.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! Connecting to HTTP servers\n\nuse crate::connectors::http::custom::Connection;\nuse crate::connectors::ConnectorOptions;\nuse crate::listeners::ALPN;\nuse crate::protocols::http::client::HttpSession;\nuse crate::protocols::http::v1::client::HttpSession as Http1Session;\nuse crate::upstreams::peer::Peer;\nuse pingora_error::Result;\nuse std::time::Duration;\n\npub mod custom;\npub mod v1;\npub mod v2;\n\npub struct Connector<C = ()>\nwhere\n    C: custom::Connector,\n{\n    h1: v1::Connector,\n    h2: v2::Connector,\n    custom: C,\n}\n\nimpl Connector<()> {\n    pub fn new(options: Option<ConnectorOptions>) -> Self {\n        Connector {\n            h1: v1::Connector::new(options.clone()),\n            h2: v2::Connector::new(options.clone()),\n            custom: Default::default(),\n        }\n    }\n}\n\nimpl<C> Connector<C>\nwhere\n    C: custom::Connector,\n{\n    pub fn new_custom(options: Option<ConnectorOptions>, custom: C) -> Self {\n        Connector {\n            h1: v1::Connector::new(options.clone()),\n            h2: v2::Connector::new(options.clone()),\n            custom,\n        }\n    }\n\n    /// Get an [HttpSession] to the given server.\n    ///\n    /// The second return value indicates whether the session is connected via a reused stream.\n    pub async fn get_http_session<P: Peer + Send + Sync + 'static>(\n        &self,\n        peer: &P,\n    ) -> Result<(HttpSession<C::Session>, bool)> {\n        let peer_opts = peer.get_peer_options();\n\n        // Switch to custom protocol as early as possible\n        if peer_opts.is_some_and(|o| matches!(o.alpn, ALPN::Custom(_))) {\n            // We create the Connector before TLS, so we need to make sure that the server also supports the same custom protocol.\n            // We will first check for sessions that we can reuse, if not we will create a new one based on the negotiated protocol\n\n            // Step 1: Look for reused Custom Session\n            if let Some(session) = self.custom.reused_http_session(peer).await {\n                return Ok((HttpSession::Custom(session), true));\n            }\n            // Step 2: Check reuse pool for reused H1 session\n            if let Some(h1) = self.h1.reused_http_session(peer).await {\n                return Ok((HttpSession::H1(h1), true));\n            }\n            // Step 3: Try and create a new Custom session\n            let (connection, reused) = self.custom.get_http_session(peer).await?;\n            // We create the Connector before TLS, so we need to make sure that the server also supports the same custom protocol\n            match connection {\n                Connection::Session(s) => {\n                    return Ok((HttpSession::Custom(s), reused));\n                }\n                // Negotiated ALPN is not custom, create a new H1 session\n                Connection::Stream(s) => {\n                    return Ok((\n                        HttpSession::H1(Http1Session::new_with_options(s, peer)),\n                        false,\n                    ));\n                }\n            }\n        }\n\n        // NOTE: maybe TODO: we do not yet enforce that only TLS traffic can use h2, which is the\n        // de facto requirement for h2, because non TLS traffic lack the negotiation mechanism.\n\n        // We assume no peer option == no ALPN == h1 only\n        let h1_only = peer\n            .get_peer_options()\n            .is_none_or(|o| o.alpn.get_max_http_version() == 1);\n        if h1_only {\n            let (h1, reused) = self.h1.get_http_session(peer).await?;\n            Ok((HttpSession::H1(h1), reused))\n        } else {\n            // the peer allows h2, we first check the h2 reuse pool\n            let reused_h2 = self.h2.reused_http_session(peer).await?;\n            if let Some(h2) = reused_h2 {\n                return Ok((HttpSession::H2(h2), true));\n            }\n            let h2_only = peer\n                .get_peer_options()\n                .is_some_and(|o| o.alpn.get_min_http_version() == 2)\n                && !self.h2.h1_is_preferred(peer);\n            if !h2_only {\n                // We next check the reuse pool for h1 before creating a new h2 connection.\n                // This is because the server may not support h2 at all, connections to\n                // the server could all be h1.\n                if let Some(h1) = self.h1.reused_http_session(peer).await {\n                    return Ok((HttpSession::H1(h1), true));\n                }\n            }\n            let session = self.h2.new_http_session(peer).await?;\n            Ok((session, false))\n        }\n    }\n\n    pub async fn release_http_session<P: Peer + Send + Sync + 'static>(\n        &self,\n        session: HttpSession<C::Session>,\n        peer: &P,\n        idle_timeout: Option<Duration>,\n    ) {\n        match session {\n            HttpSession::H1(h1) => self.h1.release_http_session(h1, peer, idle_timeout).await,\n            HttpSession::H2(h2) => self.h2.release_http_session(h2, peer, idle_timeout),\n            HttpSession::Custom(c) => {\n                self.custom\n                    .release_http_session(c, peer, idle_timeout)\n                    .await;\n            }\n        }\n    }\n\n    /// Tell the connector to always send h1 for ALPN for the given peer in the future.\n    pub fn prefer_h1(&self, peer: &impl Peer) {\n        self.h2.prefer_h1(peer);\n    }\n}\n\n#[cfg(test)]\n#[cfg(feature = \"any_tls\")]\nmod tests {\n    use super::*;\n    use crate::connectors::TransportConnector;\n    use crate::listeners::tls::TlsSettings;\n    use crate::listeners::{Listeners, TransportStack, ALPN};\n    use crate::protocols::http::v1::client::HttpSession as Http1Session;\n    use crate::protocols::tls::CustomALPN;\n    use crate::upstreams::peer::HttpPeer;\n    use crate::upstreams::peer::PeerOptions;\n    use async_trait::async_trait;\n    use pingora_http::RequestHeader;\n    use std::sync::Arc;\n    use std::sync::Mutex;\n    use tokio::io::AsyncWriteExt;\n    use tokio::net::TcpListener;\n    use tokio::task::JoinHandle;\n    use tokio::time::sleep;\n\n    async fn get_http(http: &mut Http1Session, expected_status: u16) {\n        let mut req = Box::new(RequestHeader::build(\"GET\", b\"/\", None).unwrap());\n        req.append_header(\"Host\", \"one.one.one.one\").unwrap();\n        http.write_request_header(req).await.unwrap();\n        http.read_response().await.unwrap();\n        http.respect_keepalive();\n\n        assert_eq!(http.get_status().unwrap(), expected_status);\n        while http.read_body_bytes().await.unwrap().is_some() {}\n    }\n\n    #[tokio::test]\n    async fn test_connect_h2() {\n        let connector = Connector::new(None);\n        let mut peer = HttpPeer::new((\"1.1.1.1\", 443), true, \"one.one.one.one\".into());\n        peer.options.set_http_version(2, 2);\n        let (h2, reused) = connector.get_http_session(&peer).await.unwrap();\n        assert!(!reused);\n        match &h2 {\n            HttpSession::H1(_) => panic!(\"expect h2\"),\n            HttpSession::H2(h2_stream) => assert!(!h2_stream.ping_timedout()),\n            HttpSession::Custom(_) => panic!(\"expect h2\"),\n        }\n\n        connector.release_http_session(h2, &peer, None).await;\n\n        let (h2, reused) = connector.get_http_session(&peer).await.unwrap();\n        // reused this time\n        assert!(reused);\n        match &h2 {\n            HttpSession::H1(_) => panic!(\"expect h2\"),\n            HttpSession::H2(h2_stream) => assert!(!h2_stream.ping_timedout()),\n            HttpSession::Custom(_) => panic!(\"expect h2\"),\n        }\n    }\n\n    #[tokio::test]\n    async fn test_connect_h1() {\n        let connector = Connector::new(None);\n        let mut peer = HttpPeer::new((\"1.1.1.1\", 443), true, \"one.one.one.one\".into());\n        peer.options.set_http_version(1, 1);\n        let (mut h1, reused) = connector.get_http_session(&peer).await.unwrap();\n        assert!(!reused);\n        match &mut h1 {\n            HttpSession::H1(http) => {\n                get_http(http, 200).await;\n            }\n            HttpSession::H2(_) => panic!(\"expect h1\"),\n            HttpSession::Custom(_) => panic!(\"expect h1\"),\n        }\n        connector.release_http_session(h1, &peer, None).await;\n\n        let (mut h1, reused) = connector.get_http_session(&peer).await.unwrap();\n        // reused this time\n        assert!(reused);\n        match &mut h1 {\n            HttpSession::H1(_) => {}\n            HttpSession::H2(_) => panic!(\"expect h1\"),\n            HttpSession::Custom(_) => panic!(\"expect h1\"),\n        }\n    }\n\n    #[tokio::test]\n    async fn test_connect_h2_fallback_h1_reuse() {\n        // this test verify that if the server doesn't support h2, the Connector will reuse the\n        // h1 session instead.\n\n        let connector = Connector::new(None);\n        let mut peer = HttpPeer::new((\"1.1.1.1\", 443), true, \"one.one.one.one\".into());\n        // As it is hard to find a server that support only h1, we use the following hack to trick\n        // the connector to think the server supports only h1. We force ALPN to use h1 and then\n        // return the connection to the Connector. And then we use a Peer that allows h2\n        peer.options.set_http_version(1, 1);\n        let (mut h1, reused) = connector.get_http_session(&peer).await.unwrap();\n        assert!(!reused);\n        match &mut h1 {\n            HttpSession::H1(http) => {\n                get_http(http, 200).await;\n            }\n            HttpSession::H2(_) => panic!(\"expect h1\"),\n            HttpSession::Custom(_) => panic!(\"expect h1\"),\n        }\n        connector.release_http_session(h1, &peer, None).await;\n\n        let mut peer = HttpPeer::new((\"1.1.1.1\", 443), true, \"one.one.one.one\".into());\n        peer.options.set_http_version(2, 1);\n\n        let (mut h1, reused) = connector.get_http_session(&peer).await.unwrap();\n        // reused this time\n        assert!(reused);\n        match &mut h1 {\n            HttpSession::H1(_) => {}\n            HttpSession::H2(_) => panic!(\"expect h1\"),\n            HttpSession::Custom(_) => panic!(\"expect h1\"),\n        }\n    }\n\n    #[tokio::test]\n    async fn test_connect_prefer_h1() {\n        let connector = Connector::new(None);\n        let mut peer = HttpPeer::new((\"1.1.1.1\", 443), true, \"one.one.one.one\".into());\n        peer.options.set_http_version(2, 1);\n        connector.prefer_h1(&peer);\n\n        let (mut h1, reused) = connector.get_http_session(&peer).await.unwrap();\n        assert!(!reused);\n        match &mut h1 {\n            HttpSession::H1(http) => {\n                get_http(http, 200).await;\n            }\n            HttpSession::H2(_) => panic!(\"expect h1\"),\n            HttpSession::Custom(_) => panic!(\"expect h1\"),\n        }\n        connector.release_http_session(h1, &peer, None).await;\n\n        peer.options.set_http_version(2, 2);\n        let (mut h1, reused) = connector.get_http_session(&peer).await.unwrap();\n        // reused this time\n        assert!(reused);\n        match &mut h1 {\n            HttpSession::H1(_) => {}\n            HttpSession::H2(_) => panic!(\"expect h1\"),\n            HttpSession::Custom(_) => panic!(\"expect h1\"),\n        }\n    }\n    // Track the flow of calls when using a custom protocol. For this we need to create a Mock Connector\n    struct MockConnector {\n        transport: TransportConnector,\n        reusable: Arc<Mutex<bool>>, // Mock for tracking reusable sessions\n    }\n\n    #[async_trait]\n    impl custom::Connector for MockConnector {\n        type Session = ();\n\n        async fn get_http_session<P: Peer + Send + Sync + 'static>(\n            &self,\n            peer: &P,\n        ) -> Result<(Connection<Self::Session>, bool)> {\n            let (stream, _) = self.transport.get_stream(peer).await?;\n\n            match stream.selected_alpn_proto() {\n                Some(ALPN::Custom(_)) => Ok((custom::Connection::Session(()), false)),\n                _ => Ok(((custom::Connection::Stream(stream)), false)),\n            }\n        }\n\n        async fn reused_http_session<P: Peer + Send + Sync + 'static>(\n            &self,\n            _peer: &P,\n        ) -> Option<Self::Session> {\n            let mut flag = self.reusable.lock().unwrap();\n            if *flag {\n                *flag = false;\n                Some(())\n            } else {\n                None\n            }\n        }\n\n        async fn release_http_session<P: Peer + Send + Sync + 'static>(\n            &self,\n            _session: Self::Session,\n            _peer: &P,\n            _idle_timeout: Option<Duration>,\n        ) {\n            let mut flag = self.reusable.lock().unwrap();\n            *flag = true;\n        }\n    }\n\n    // Finds an available TCP port on localhost for test server setup.\n    async fn get_available_port() -> u16 {\n        TcpListener::bind(\"127.0.0.1:0\")\n            .await\n            .unwrap()\n            .local_addr()\n            .unwrap()\n            .port()\n    }\n    // Creates a test connector for integration/unit tests.\n    // For rustls, only ConnectorOptions are used here; the actual dangerous verifier is patched in the TLS connector.\n    fn create_test_connector() -> Connector<MockConnector> {\n        #[cfg(feature = \"rustls\")]\n        let custom_transport = {\n            let options = ConnectorOptions::new(1);\n            TransportConnector::new(Some(options))\n        };\n        #[cfg(not(feature = \"rustls\"))]\n        let custom_transport = TransportConnector::new(None);\n        Connector {\n            h1: v1::Connector::new(None),\n            h2: v2::Connector::new(None),\n            custom: MockConnector {\n                transport: custom_transport,\n                reusable: Arc::new(Mutex::new(false)),\n            },\n        }\n    }\n\n    // Creates a test peer that uses a custom ALPN protocol and disables cert/hostname verification for tests.\n    fn create_peer_with_custom_proto(port: u16, proto: &[u8]) -> HttpPeer {\n        let mut peer = HttpPeer::new((\"127.0.0.1\", port), true, \"localhost\".into());\n        let mut options = PeerOptions::new();\n        options.alpn = ALPN::Custom(CustomALPN::new(proto.to_vec()));\n        // Disable cert verification for this test (self-signed or invalid certs are OK)\n        options.verify_cert = false;\n        options.verify_hostname = false;\n        peer.options = options;\n        peer\n    }\n    async fn build_custom_tls_listener(port: u16, custom_alpn: CustomALPN) -> TransportStack {\n        let cert_path = format!(\"{}/tests/keys/server.crt\", env!(\"CARGO_MANIFEST_DIR\"));\n        let key_path = format!(\"{}/tests/keys/key.pem\", env!(\"CARGO_MANIFEST_DIR\"));\n        let addr = format!(\"127.0.0.1:{}\", port);\n        let mut listeners = Listeners::new();\n        let mut tls_settings = TlsSettings::intermediate(&cert_path, &key_path).unwrap();\n\n        tls_settings.set_alpn(ALPN::Custom(custom_alpn));\n        listeners.add_tls_with_settings(&addr, None, tls_settings);\n        listeners\n            .build(\n                #[cfg(unix)]\n                None,\n            )\n            .await\n            .unwrap()\n            .pop()\n            .unwrap()\n    }\n\n    // Spawn a simple TLS Server\n    fn spawn_test_tls_server(listener: TransportStack) -> JoinHandle<()> {\n        tokio::spawn(async move {\n            loop {\n                let stream = match listener.accept().await {\n                    Ok(stream) => stream,\n                    Err(_) => break, // Exit if listener is closed\n                };\n                let mut stream = stream.handshake().await.unwrap();\n\n                let _ = stream.write_all(b\"CUSTOM\").await; // Ignore write errors\n            }\n        })\n    }\n\n    // Both server and client are using the same custom protocol\n    #[tokio::test]\n    async fn test_custom_client_custom_upstream() {\n        let port = get_available_port().await;\n        let custom_protocol = b\"custom\".to_vec();\n\n        let listener =\n            build_custom_tls_listener(port, CustomALPN::new(custom_protocol.clone())).await;\n        let server_handle = spawn_test_tls_server(listener);\n        // Wait for server to start up\n        sleep(Duration::from_millis(100)).await;\n\n        let connector = create_test_connector();\n        let peer = create_peer_with_custom_proto(port, &custom_protocol);\n\n        // Check that the agreed ALPN is custom and matches the expected value\n        if let Ok((stream, reused)) = connector.custom.transport.get_stream(&peer).await {\n            assert!(!reused);\n            match stream.selected_alpn_proto() {\n                Some(ALPN::Custom(protocol)) => {\n                    assert_eq!(\n                        protocol.protocol(),\n                        custom_protocol.as_slice(),\n                        \"Negotiated custom ALPN does not match expected value\"\n                    );\n                }\n                other => panic!(\"Expected custom ALPN, got {:?}\", other),\n            }\n        } else {\n            panic!(\"Should be able to create a stream\");\n        }\n\n        let (custom, reused) = connector.get_http_session(&peer).await.unwrap();\n        assert!(!reused);\n        match custom {\n            HttpSession::H1(_) => panic!(\"expect custom\"),\n            HttpSession::H2(_) => panic!(\"expect custom\"),\n            HttpSession::Custom(_) => {}\n        }\n        connector.release_http_session(custom, &peer, None).await;\n\n        // Assert it returns a reused custom session this time\n        let (custom, reused) = connector.get_http_session(&peer).await.unwrap();\n        assert!(reused);\n        match custom {\n            HttpSession::H1(_) => panic!(\"expect custom\"),\n            HttpSession::H2(_) => panic!(\"expect custom\"),\n            HttpSession::Custom(_) => {}\n        }\n\n        // Kill the server task\n        server_handle.abort();\n        sleep(Duration::from_millis(100)).await;\n    }\n\n    // Both client and server are using custom protocols, but different ones - we should create H1 sessions as fallback.\n    // For RusTLS if there is no agreed protocol, the handshake directly fails, so this won't work\n    // TODO: If no ALPN is matched, rustls should return None instead of failing the handshake.\n    #[cfg(not(feature = \"rustls\"))]\n    #[tokio::test]\n    async fn test_incompatible_custom_client_custom_upstream() {\n        let port = get_available_port().await;\n        let custom_protocol = b\"custom\".to_vec();\n\n        let listener =\n            build_custom_tls_listener(port, CustomALPN::new(b\"different_custom\".to_vec())).await;\n        let server_handle = spawn_test_tls_server(listener);\n        // Wait for server to start up\n        sleep(Duration::from_millis(100)).await;\n\n        let connector = create_test_connector();\n        let peer = create_peer_with_custom_proto(port, &custom_protocol);\n\n        // Verify that there is no agreed ALPN\n        if let Ok((stream, reused)) = connector.custom.transport.get_stream(&peer).await {\n            assert!(!reused);\n            assert!(stream.selected_alpn_proto().is_none());\n        } else {\n            panic!(\"Should be able to create a stream\");\n        }\n\n        let (h1, reused) = connector.get_http_session(&peer).await.unwrap();\n        assert!(!reused);\n        match h1 {\n            HttpSession::H1(_) => {}\n            HttpSession::H2(_) => panic!(\"expect h1\"),\n            HttpSession::Custom(_) => panic!(\"expect h1\"),\n        }\n        // Not testing session reuse logic here as we haven't implemented it. Next test will test this.\n\n        // Kill the server task\n        server_handle.abort();\n        sleep(Duration::from_millis(100)).await;\n    }\n\n    // Client thinks server is custom but server is not Custom. Should fallback to H1\n    #[tokio::test]\n    async fn test_custom_client_non_custom_upstream() {\n        let custom_proto = b\"custom\".to_vec();\n        let connector = create_test_connector();\n        // Upstream supports H1 and H2\n        let mut peer = HttpPeer::new((\"1.1.1.1\", 443), true, \"one.one.one.one\".into());\n        // Client sets upstream ALPN as custom protocol\n        peer.options.alpn = ALPN::Custom(CustomALPN::new(custom_proto));\n\n        // Verify that there is no agreed ALPN\n        if let Ok((stream, reused)) = connector.custom.transport.get_stream(&peer).await {\n            assert!(!reused);\n            assert!(stream.selected_alpn_proto().is_none());\n        } else {\n            panic!(\"Should be able to create a stream\");\n        }\n\n        let (mut h1, reused) = connector.get_http_session(&peer).await.unwrap();\n        // Assert it returns a new H1 session\n        assert!(!reused);\n        match &mut h1 {\n            HttpSession::H1(http) => {\n                get_http(http, 200).await;\n            }\n            HttpSession::H2(_) => panic!(\"expect h1\"),\n            HttpSession::Custom(_) => panic!(\"expect h1\"),\n        }\n        connector.release_http_session(h1, &peer, None).await;\n\n        // Assert it returns a reused h1 session this time\n        let (mut h1, reused) = connector.get_http_session(&peer).await.unwrap();\n        assert!(reused);\n        match &mut h1 {\n            HttpSession::H1(_) => {}\n            HttpSession::H2(_) => panic!(\"expect h1\"),\n            HttpSession::Custom(_) => panic!(\"expect h1\"),\n        }\n    }\n}\n\n// Used for disabling certificate/hostname verification in rustls for tests and custom ALPN/self-signed scenarios.\n#[cfg(all(test, feature = \"rustls\"))]\npub mod rustls_no_verify {\n    use rustls::client::danger::{ServerCertVerified, ServerCertVerifier};\n    use rustls::pki_types::{CertificateDer, ServerName};\n    use rustls::Error as TLSError;\n    use std::sync::Arc;\n    #[derive(Debug)]\n    pub struct NoCertificateVerification;\n\n    impl ServerCertVerifier for NoCertificateVerification {\n        fn verify_server_cert(\n            &self,\n            _end_entity: &CertificateDer,\n            _intermediates: &[CertificateDer],\n            _server_name: &ServerName,\n            _scts: &[u8],\n            _now: rustls::pki_types::UnixTime,\n        ) -> Result<ServerCertVerified, TLSError> {\n            Ok(ServerCertVerified::assertion())\n        }\n\n        fn verify_tls12_signature(\n            &self,\n            _message: &[u8],\n            _cert: &CertificateDer,\n            _dss: &rustls::DigitallySignedStruct,\n        ) -> Result<rustls::client::danger::HandshakeSignatureValid, TLSError> {\n            Ok(rustls::client::danger::HandshakeSignatureValid::assertion())\n        }\n\n        fn verify_tls13_signature(\n            &self,\n            _message: &[u8],\n            _cert: &CertificateDer,\n            _dss: &rustls::DigitallySignedStruct,\n        ) -> Result<rustls::client::danger::HandshakeSignatureValid, TLSError> {\n            Ok(rustls::client::danger::HandshakeSignatureValid::assertion())\n        }\n\n        fn supported_verify_schemes(&self) -> Vec<rustls::SignatureScheme> {\n            vec![rustls::SignatureScheme::ECDSA_NISTP256_SHA256]\n        }\n    }\n\n    pub fn apply_no_verify(config: &mut rustls::ClientConfig) {\n        config\n            .dangerous()\n            .set_certificate_verifier(Arc::new(NoCertificateVerification));\n    }\n}\n"
  },
  {
    "path": "pingora-core/src/connectors/http/v1.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\nuse crate::connectors::{ConnectorOptions, TransportConnector};\nuse crate::protocols::http::v1::client::HttpSession;\nuse crate::upstreams::peer::Peer;\n\nuse pingora_error::Result;\nuse std::time::Duration;\n\npub struct Connector {\n    transport: TransportConnector,\n}\n\nimpl Connector {\n    pub fn new(options: Option<ConnectorOptions>) -> Self {\n        Connector {\n            transport: TransportConnector::new(options),\n        }\n    }\n\n    pub async fn get_http_session<P: Peer + Send + Sync + 'static>(\n        &self,\n        peer: &P,\n    ) -> Result<(HttpSession, bool)> {\n        let (stream, reused) = self.transport.get_stream(peer).await?;\n        let http = HttpSession::new_with_options(stream, peer);\n        Ok((http, reused))\n    }\n\n    pub async fn reused_http_session<P: Peer + Send + Sync + 'static>(\n        &self,\n        peer: &P,\n    ) -> Option<HttpSession> {\n        let stream = self.transport.reused_stream(peer).await?;\n        let http = HttpSession::new_with_options(stream, peer);\n        Some(http)\n    }\n\n    pub async fn release_http_session<P: Peer + Send + Sync + 'static>(\n        &self,\n        mut session: HttpSession,\n        peer: &P,\n        idle_timeout: Option<Duration>,\n    ) {\n        session.respect_keepalive();\n        if let Some(stream) = session.reuse().await {\n            self.transport\n                .release_stream(stream, peer.reuse_hash(), idle_timeout);\n        }\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use super::*;\n    use crate::protocols::l4::socket::SocketAddr;\n    use crate::upstreams::peer::HttpPeer;\n    use crate::upstreams::peer::Peer;\n    use pingora_http::RequestHeader;\n    use std::fmt::{Display, Formatter, Result as FmtResult};\n\n    async fn get_http(http: &mut HttpSession, expected_status: u16) {\n        let mut req = Box::new(RequestHeader::build(\"GET\", b\"/\", None).unwrap());\n        req.append_header(\"Host\", \"one.one.one.one\").unwrap();\n        http.write_request_header(req).await.unwrap();\n        http.read_response().await.unwrap();\n        http.respect_keepalive();\n\n        assert_eq!(http.get_status().unwrap(), expected_status);\n        while http.read_body_bytes().await.unwrap().is_some() {}\n    }\n\n    #[tokio::test]\n    async fn test_connect() {\n        let connector = Connector::new(None);\n        let peer = HttpPeer::new((\"1.1.1.1\", 80), false, \"\".into());\n        // make a new connection to 1.1.1.1\n        let (http, reused) = connector.get_http_session(&peer).await.unwrap();\n        let server_addr = http.server_addr().unwrap();\n        assert_eq!(*server_addr, \"1.1.1.1:80\".parse::<SocketAddr>().unwrap());\n        assert!(!reused);\n\n        // this http is not even used, so not be able to reuse\n        connector.release_http_session(http, &peer, None).await;\n        let (mut http, reused) = connector.get_http_session(&peer).await.unwrap();\n        assert!(!reused);\n\n        get_http(&mut http, 301).await;\n        connector.release_http_session(http, &peer, None).await;\n        let (_, reused) = connector.get_http_session(&peer).await.unwrap();\n        assert!(reused);\n    }\n\n    #[cfg(unix)]\n    #[tokio::test]\n    async fn test_reuse_rejects_fd_mismatch() {\n        use std::os::unix::prelude::AsRawFd;\n\n        #[derive(Clone)]\n        struct MismatchPeer {\n            reuse_hash: u64,\n            address: SocketAddr,\n        }\n\n        impl Display for MismatchPeer {\n            fn fmt(&self, f: &mut Formatter<'_>) -> FmtResult {\n                write!(f, \"{:?}\", self.address)\n            }\n        }\n\n        impl Peer for MismatchPeer {\n            fn address(&self) -> &SocketAddr {\n                &self.address\n            }\n\n            fn tls(&self) -> bool {\n                false\n            }\n\n            fn sni(&self) -> &str {\n                \"\"\n            }\n\n            fn reuse_hash(&self) -> u64 {\n                self.reuse_hash\n            }\n\n            fn matches_fd<V: AsRawFd>(&self, _fd: V) -> bool {\n                false\n            }\n        }\n\n        let connector = Connector::new(None);\n        let peer = HttpPeer::new((\"1.1.1.1\", 80), false, \"\".into());\n        let (mut http, reused) = connector.get_http_session(&peer).await.unwrap();\n        assert!(!reused);\n        get_http(&mut http, 301).await;\n        connector.release_http_session(http, &peer, None).await;\n\n        let mismatch_peer = MismatchPeer {\n            reuse_hash: peer.reuse_hash(),\n            address: peer.address().clone(),\n        };\n\n        assert!(connector\n            .reused_http_session(&mismatch_peer)\n            .await\n            .is_none());\n    }\n\n    #[tokio::test]\n    #[cfg(feature = \"any_tls\")]\n    async fn test_connect_tls() {\n        let connector = Connector::new(None);\n        let peer = HttpPeer::new((\"1.1.1.1\", 443), true, \"one.one.one.one\".into());\n        // make a new connection to https://1.1.1.1\n        let (http, reused) = connector.get_http_session(&peer).await.unwrap();\n        let server_addr = http.server_addr().unwrap();\n        assert_eq!(*server_addr, \"1.1.1.1:443\".parse::<SocketAddr>().unwrap());\n        assert!(!reused);\n\n        // this http is not even used, so not be able to reuse\n        connector.release_http_session(http, &peer, None).await;\n        let (mut http, reused) = connector.get_http_session(&peer).await.unwrap();\n        assert!(!reused);\n\n        get_http(&mut http, 200).await;\n        connector.release_http_session(http, &peer, None).await;\n        let (_, reused) = connector.get_http_session(&peer).await.unwrap();\n        assert!(reused);\n    }\n}\n"
  },
  {
    "path": "pingora-core/src/connectors/http/v2.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\nuse super::HttpSession;\nuse crate::connectors::{ConnectorOptions, TransportConnector};\nuse crate::protocols::http::custom::client::Session;\nuse crate::protocols::http::v1::client::HttpSession as Http1Session;\nuse crate::protocols::http::v2::client::{drive_connection, Http2Session};\nuse crate::protocols::{Digest, Stream, UniqueIDType};\nuse crate::upstreams::peer::{Peer, ALPN};\n\nuse bytes::Bytes;\nuse h2::client::SendRequest;\nuse log::debug;\nuse parking_lot::{Mutex, RwLock};\nuse pingora_error::{Error, ErrorType::*, OrErr, Result};\nuse pingora_pool::{ConnectionMeta, ConnectionPool, PoolNode};\nuse std::collections::HashMap;\nuse std::sync::atomic::{AtomicBool, AtomicUsize, Ordering};\nuse std::sync::Arc;\nuse std::time::Duration;\nuse tokio::sync::watch;\n\nstruct Stub(SendRequest<Bytes>);\n\nimpl Stub {\n    async fn new_stream(&self) -> Result<SendRequest<Bytes>> {\n        let send_req = self.0.clone();\n        send_req\n            .ready()\n            .await\n            .or_err(H2Error, \"while creating new stream\")\n    }\n}\n\npub(crate) struct ConnectionRefInner {\n    connection_stub: Stub,\n    closed: watch::Receiver<bool>,\n    ping_timeout_occurred: Arc<AtomicBool>,\n    id: UniqueIDType,\n    // max concurrent streams this connection is allowed to create\n    max_streams: usize,\n    // how many concurrent streams already active\n    current_streams: AtomicUsize,\n    // The connection is gracefully shutting down, no more stream is allowed\n    shutting_down: AtomicBool,\n    // because `SendRequest` doesn't actually have access to the underlying Stream,\n    // we log info about timing and tcp info here.\n    pub(crate) digest: Digest,\n    // To serialize certain operations when trying to release the connect back to the pool,\n    pub(crate) release_lock: Arc<Mutex<()>>,\n}\n\n#[derive(Clone)]\npub struct ConnectionRef(Arc<ConnectionRefInner>);\n\nimpl ConnectionRef {\n    pub fn new(\n        send_req: SendRequest<Bytes>,\n        closed: watch::Receiver<bool>,\n        ping_timeout_occurred: Arc<AtomicBool>,\n        id: UniqueIDType,\n        max_streams: usize,\n        digest: Digest,\n    ) -> Self {\n        ConnectionRef(Arc::new(ConnectionRefInner {\n            connection_stub: Stub(send_req),\n            closed,\n            ping_timeout_occurred,\n            id,\n            max_streams,\n            current_streams: AtomicUsize::new(0),\n            shutting_down: false.into(),\n            digest,\n            release_lock: Arc::new(Mutex::new(())),\n        }))\n    }\n\n    pub fn more_streams_allowed(&self) -> bool {\n        let current = self.0.current_streams.load(Ordering::Relaxed);\n        !self.is_shutting_down()\n            && self.0.max_streams > current\n            && self.0.connection_stub.0.current_max_send_streams() > current\n    }\n\n    pub fn is_idle(&self) -> bool {\n        self.0.current_streams.load(Ordering::Relaxed) == 0\n    }\n\n    pub fn release_stream(&self) {\n        self.0.current_streams.fetch_sub(1, Ordering::SeqCst);\n    }\n\n    pub fn id(&self) -> UniqueIDType {\n        self.0.id\n    }\n\n    pub fn digest(&self) -> &Digest {\n        &self.0.digest\n    }\n\n    pub fn digest_mut(&mut self) -> Option<&mut Digest> {\n        Arc::get_mut(&mut self.0).map(|inner| &mut inner.digest)\n    }\n\n    pub fn ping_timedout(&self) -> bool {\n        self.0.ping_timeout_occurred.load(Ordering::Relaxed)\n    }\n\n    pub fn is_closed(&self) -> bool {\n        *self.0.closed.borrow()\n    }\n\n    // different from is_closed, existing streams can still be processed but can no longer create\n    // new stream.\n    pub fn is_shutting_down(&self) -> bool {\n        self.0.shutting_down.load(Ordering::Relaxed)\n    }\n\n    // spawn a stream if more stream is allowed, otherwise return Ok(None)\n    pub async fn spawn_stream(&self) -> Result<Option<Http2Session>> {\n        // Atomically check if the current_stream is over the limit\n        // load(), compare and then fetch_add() cannot guarantee the same\n        let current_streams = self.0.current_streams.fetch_add(1, Ordering::SeqCst);\n        if current_streams >= self.0.max_streams {\n            // already over the limit, reset the counter to the previous value\n            self.0.current_streams.fetch_sub(1, Ordering::SeqCst);\n            return Ok(None);\n        }\n\n        match self.0.connection_stub.new_stream().await {\n            Ok(send_req) => Ok(Some(Http2Session::new(send_req, self.clone()))),\n            Err(e) => {\n                // fail to create the stream, reset the counter\n                self.0.current_streams.fetch_sub(1, Ordering::SeqCst);\n                // Remote sends GOAWAY(NO_ERROR): graceful shutdown: this connection no longer\n                // accepts new streams. We can still try to create new connection.\n                if e.root_cause()\n                    .downcast_ref::<h2::Error>()\n                    .map(|e| {\n                        e.is_go_away() && e.is_remote() && e.reason() == Some(h2::Reason::NO_ERROR)\n                    })\n                    .unwrap_or(false)\n                {\n                    self.0.shutting_down.store(true, Ordering::Relaxed);\n                    Ok(None)\n                } else {\n                    Err(e)\n                }\n            }\n        }\n    }\n}\n\npub struct InUsePool {\n    // TODO: use pingora hashmap to shard the lock contention\n    pools: RwLock<HashMap<u64, PoolNode<ConnectionRef>>>,\n}\n\nimpl InUsePool {\n    fn new() -> Self {\n        InUsePool {\n            pools: RwLock::new(HashMap::new()),\n        }\n    }\n\n    /// Attempt to remove an empty [`PoolNode`] entry from the pools `HashMap`.\n    ///\n    /// Same rationale as [`ConnectionPool::try_remove_empty_node`]: prevents\n    /// unbounded growth when many unique reuse hashes are seen over time.\n    /// The write lock + re-check ensures we never remove a node that was\n    /// concurrently repopulated.\n    fn try_remove_empty_node(&self, reuse_hash: u64) {\n        let mut pools = self.pools.write();\n        if let Some(pool) = pools.get(&reuse_hash) {\n            if pool.is_empty() {\n                pools.remove(&reuse_hash);\n            }\n        }\n    }\n\n    pub fn insert(&self, reuse_hash: u64, conn: ConnectionRef) {\n        {\n            let pools = self.pools.read();\n            if let Some(pool) = pools.get(&reuse_hash) {\n                pool.insert(conn.id(), conn);\n                return;\n            }\n        } // drop read lock\n\n        let mut pools = self.pools.write();\n        // Double-check: another thread may have inserted a node between\n        // dropping the read lock and acquiring this write lock.\n        if let Some(pool) = pools.get(&reuse_hash) {\n            pool.insert(conn.id(), conn);\n            return;\n        }\n        let pool = PoolNode::new();\n        pool.insert(conn.id(), conn);\n        pools.insert(reuse_hash, pool);\n    }\n\n    // retrieve a h2 conn ref to create a new stream\n    // the caller should return the conn ref to this pool if there are still\n    // capacity left for more streams\n    pub fn get(&self, reuse_hash: u64) -> Option<ConnectionRef> {\n        let (result, maybe_empty) = {\n            let pools = self.pools.read();\n            match pools.get(&reuse_hash) {\n                Some(pool) => match pool.get_any() {\n                    Some((_, conn)) => (Some(conn), pool.is_empty()),\n                    None => (None, true),\n                },\n                None => (None, false),\n            }\n        }; // read lock released here\n\n        if maybe_empty {\n            self.try_remove_empty_node(reuse_hash);\n        }\n\n        result\n    }\n\n    // release a h2_stream, this functional will cause an ConnectionRef to be returned (if exist)\n    // the caller should update the ref and then decide where to put it (in use pool or idle)\n    pub fn release(&self, reuse_hash: u64, id: UniqueIDType) -> Option<ConnectionRef> {\n        let (result, maybe_empty) = {\n            let pools = self.pools.read();\n            if let Some(pool) = pools.get(&reuse_hash) {\n                let removed = pool.remove(id);\n                (removed, pool.is_empty())\n            } else {\n                (None, false)\n            }\n        }; // read lock released here\n\n        if maybe_empty {\n            self.try_remove_empty_node(reuse_hash);\n        }\n\n        result\n    }\n}\n\nconst DEFAULT_POOL_SIZE: usize = 128;\n\n/// Http2 connector\npub struct Connector {\n    // just for creating connections, the Stream of h2 should be reused\n    transport: TransportConnector,\n    // the h2 connection idle pool\n    idle_pool: Arc<ConnectionPool<ConnectionRef>>,\n    // the pool of h2 connections that have ongoing streams\n    in_use_pool: InUsePool,\n}\n\nimpl Connector {\n    /// Create a new [Connector] from the given [ConnectorOptions]\n    pub fn new(options: Option<ConnectorOptions>) -> Self {\n        let pool_size = options\n            .as_ref()\n            .map_or(DEFAULT_POOL_SIZE, |o| o.keepalive_pool_size);\n        // connection offload is handled by the [TransportConnector]\n        Connector {\n            transport: TransportConnector::new(options),\n            idle_pool: Arc::new(ConnectionPool::new(pool_size)),\n            in_use_pool: InUsePool::new(),\n        }\n    }\n\n    pub fn transport(&self) -> &TransportConnector {\n        &self.transport\n    }\n\n    pub fn idle_pool(&self) -> &Arc<ConnectionPool<ConnectionRef>> {\n        &self.idle_pool\n    }\n\n    pub fn in_use_pool(&self) -> &InUsePool {\n        &self.in_use_pool\n    }\n\n    /// Create a new Http2 connection to the given server\n    ///\n    /// Either an Http2 or Http1 session can be returned depending on the server's preference.\n    pub async fn new_http_session<P: Peer + Send + Sync + 'static, C: Session>(\n        &self,\n        peer: &P,\n    ) -> Result<HttpSession<C>> {\n        let stream = self.transport.new_stream(peer).await?;\n\n        // check alpn\n        match stream.selected_alpn_proto() {\n            Some(ALPN::H2) => { /* continue */ }\n            Some(_) => {\n                // H2 not supported\n                return Ok(HttpSession::H1(Http1Session::new_with_options(\n                    stream, peer,\n                )));\n            }\n            None => {\n                // if tls but no ALPN, default to h1\n                // else if plaintext and min http version is 1, this is most likely h1\n                if peer.tls()\n                    || peer\n                        .get_peer_options()\n                        .is_none_or(|o| o.alpn.get_min_http_version() == 1)\n                {\n                    return Ok(HttpSession::H1(Http1Session::new_with_options(\n                        stream, peer,\n                    )));\n                }\n                // else: min http version=H2 over plaintext, there is no ALPN anyways, we trust\n                // the caller that the server speaks h2c\n            }\n        }\n        let max_h2_stream = peer.get_peer_options().map_or(1, |o| o.max_h2_streams);\n        let conn = handshake(stream, max_h2_stream, peer.h2_ping_interval()).await?;\n        let h2_stream = conn\n            .spawn_stream()\n            .await?\n            .expect(\"newly created connections should have at least one free stream\");\n        if conn.more_streams_allowed() {\n            self.in_use_pool.insert(peer.reuse_hash(), conn);\n        }\n        Ok(HttpSession::H2(h2_stream))\n    }\n\n    /// Try to create a new http2 stream from any existing H2 connection.\n    ///\n    /// None means there is no \"free\" connection left.\n    pub async fn reused_http_session<P: Peer + Send + Sync + 'static>(\n        &self,\n        peer: &P,\n    ) -> Result<Option<Http2Session>> {\n        // check in use pool first so that we use fewer total connections\n        // then idle pool\n        let reuse_hash = peer.reuse_hash();\n\n        // NOTE: We grab a conn from the pools, create a new stream and put the conn back if the\n        // conn has more free streams. During this process another caller could arrive but is not\n        // able to find the conn even the conn has free stream to use.\n        // We accept this false negative to keep the implementation simple. This false negative\n        // makes an actual impact when there are only a few connection.\n        // Alternative design 1. given each free stream a conn object: a lot of Arc<>\n        // Alternative design 2. mutex the pool, which creates lock contention when concurrency is high\n        // Alternative design 3. do not pop conn from the pool so that multiple callers can grab it\n        // which will cause issue where spawn_stream() could return None because others call it\n        // first. Thus a caller might have to retry or give up. This issue is more likely to happen\n        // when concurrency is high.\n        let maybe_conn = self\n            .in_use_pool\n            .get(reuse_hash)\n            // filter out closed, InUsePool does not have notify closed eviction like the idle pool\n            // and it's possible we get an in use connection that is closed and not yet released\n            .filter(|c| !c.is_closed())\n            .or_else(|| self.idle_pool.get(&reuse_hash));\n        if let Some(conn) = maybe_conn {\n            #[cfg(unix)]\n            if !peer.matches_fd(conn.id()) {\n                return Ok(None);\n            }\n            #[cfg(windows)]\n            {\n                use std::os::windows::io::{AsRawSocket, RawSocket};\n                struct WrappedRawSocket(RawSocket);\n                impl AsRawSocket for WrappedRawSocket {\n                    fn as_raw_socket(&self) -> RawSocket {\n                        self.0\n                    }\n                }\n                if !peer.matches_sock(WrappedRawSocket(conn.id() as RawSocket)) {\n                    return Ok(None);\n                }\n            }\n            let h2_stream = conn.spawn_stream().await?;\n            if conn.more_streams_allowed() {\n                self.in_use_pool.insert(reuse_hash, conn);\n            }\n            Ok(h2_stream)\n        } else {\n            Ok(None)\n        }\n    }\n\n    /// Release a finished h2 stream.\n    ///\n    /// This function will terminate the [Http2Session]. The corresponding h2 connection will now\n    /// have one more free stream to use.\n    ///\n    /// The h2 connection will be closed after `idle_timeout` if it has no active streams.\n    pub fn release_http_session<P: Peer + Send + Sync + 'static>(\n        &self,\n        session: Http2Session,\n        peer: &P,\n        idle_timeout: Option<Duration>,\n    ) {\n        let id = session.conn.id();\n        let reuse_hash = peer.reuse_hash();\n        // get a ref to the connection, which we might need below, before dropping the h2\n        let conn = session.conn();\n\n        // The lock here is to make sure that in_use_pool.insert() below cannot be called after\n        // in_use_pool.release(), which would have put the conn entry in both pools.\n        // It also makes sure that only one conn will trigger the conn.is_idle() condition, which\n        // avoids putting the same conn into the idle_pool more than once.\n        let locked = conn.0.release_lock.lock_arc();\n        // this drop() will both drop the actual stream and call the conn.release_stream()\n        drop(session);\n        // find and remove the conn stored in in_use_pool so that it could be put in the idle pool\n        // if necessary\n        let conn = self.in_use_pool.release(reuse_hash, id).unwrap_or(conn);\n        if conn.is_closed() || conn.is_shutting_down() {\n            // should never be put back to the pool\n            return;\n        }\n        if conn.is_idle() {\n            drop(locked);\n            let meta = ConnectionMeta {\n                key: reuse_hash,\n                id,\n            };\n            let closed = conn.0.closed.clone();\n            let (notify_evicted, watch_use) = self.idle_pool.put(&meta, conn);\n            let pool = self.idle_pool.clone(); //clone the arc\n            let rt = pingora_runtime::current_handle();\n            rt.spawn(async move {\n                pool.idle_timeout(&meta, idle_timeout, notify_evicted, closed, watch_use)\n                    .await;\n            });\n        } else {\n            self.in_use_pool.insert(reuse_hash, conn);\n            drop(locked);\n        }\n    }\n\n    /// Tell the connector to always send h1 for ALPN for the given peer in the future.\n    pub fn prefer_h1(&self, peer: &impl Peer) {\n        self.transport.prefer_h1(peer);\n    }\n\n    pub(crate) fn h1_is_preferred(&self, peer: &impl Peer) -> bool {\n        self.transport\n            .preferred_http_version\n            .get(peer)\n            .is_some_and(|v| matches!(v, ALPN::H1))\n    }\n}\n\n// The h2 library we use has unbounded internal buffering, which will cause excessive memory\n// consumption when the downstream is slower than upstream. This window size caps the buffering by\n// limiting how much data can be inflight. However, setting this value will also cap the max\n// download speed by limiting the bandwidth-delay product of a link.\n// Long term, we should advertising large window but shrink it when a small buffer is full.\n// 8 Mbytes = 80 Mbytes X 100ms, which should be enough for most links.\nconst H2_WINDOW_SIZE: u32 = 1 << 23;\n\npub async fn handshake(\n    stream: Stream,\n    max_streams: usize,\n    h2_ping_interval: Option<Duration>,\n) -> Result<ConnectionRef> {\n    use h2::client::Builder;\n    use pingora_runtime::current_handle;\n\n    // Safe guard: new_http_session() assumes there should be at least one free stream\n    if max_streams == 0 {\n        return Error::e_explain(H2Error, \"zero max_stream configured\");\n    }\n\n    let id = stream.id();\n    let digest = Digest {\n        // NOTE: this field is always false because the digest is shared across all streams\n        // The streams should log their own reuse info\n        ssl_digest: stream.get_ssl_digest(),\n        // TODO: log h2 handshake time\n        timing_digest: stream.get_timing_digest(),\n        proxy_digest: stream.get_proxy_digest(),\n        socket_digest: stream.get_socket_digest(),\n    };\n    // TODO: make these configurable\n    let (send_req, connection) = Builder::new()\n        .enable_push(false)\n        .initial_max_send_streams(max_streams)\n        // The limit for the server. Server push is not allowed, so this value doesn't matter\n        .max_concurrent_streams(1)\n        .max_frame_size(64 * 1024) // advise server to send larger frames\n        .initial_window_size(H2_WINDOW_SIZE)\n        // should this be max_streams * H2_WINDOW_SIZE?\n        .initial_connection_window_size(H2_WINDOW_SIZE)\n        .handshake(stream)\n        .await\n        .or_err(HandshakeError, \"during H2 handshake\")?;\n    debug!(\"H2 handshake to server done.\");\n    let ping_timeout_occurred = Arc::new(AtomicBool::new(false));\n    let ping_timeout_clone = ping_timeout_occurred.clone();\n    let max_allowed_streams = std::cmp::min(max_streams, connection.max_concurrent_send_streams());\n\n    // Safe guard: new_http_session() assumes there should be at least one free stream\n    // The server won't commonly advertise 0 max stream.\n    if max_allowed_streams == 0 {\n        return Error::e_explain(H2Error, \"zero max_concurrent_send_streams received\");\n    }\n\n    let (closed_tx, closed_rx) = watch::channel(false);\n\n    current_handle().spawn(async move {\n        drive_connection(\n            connection,\n            id,\n            closed_tx,\n            h2_ping_interval,\n            ping_timeout_clone,\n        )\n        .await;\n    });\n    Ok(ConnectionRef::new(\n        send_req,\n        closed_rx,\n        ping_timeout_occurred,\n        id,\n        max_allowed_streams,\n        digest,\n    ))\n}\n\n// TODO(slava): add custom unit tests\n#[cfg(test)]\nmod tests {\n    use super::*;\n    use crate::upstreams::peer::HttpPeer;\n\n    #[tokio::test]\n    #[cfg(feature = \"any_tls\")]\n    async fn test_connect_h2() {\n        let connector = Connector::new(None);\n        let mut peer = HttpPeer::new((\"1.1.1.1\", 443), true, \"one.one.one.one\".into());\n        peer.options.set_http_version(2, 2);\n        let h2 = connector\n            .new_http_session::<HttpPeer, ()>(&peer)\n            .await\n            .unwrap();\n        match h2 {\n            HttpSession::H1(_) => panic!(\"expect h2\"),\n            HttpSession::H2(h2_stream) => assert!(!h2_stream.ping_timedout()),\n            HttpSession::Custom(_) => panic!(\"expect h2\"),\n        }\n    }\n\n    #[tokio::test]\n    #[cfg(feature = \"any_tls\")]\n    async fn test_connect_h1() {\n        let connector = Connector::new(None);\n        let mut peer = HttpPeer::new((\"1.1.1.1\", 443), true, \"one.one.one.one\".into());\n        // a hack to force h1, new_http_session() in the future might validate this setting\n        peer.options.set_http_version(1, 1);\n        let h2 = connector\n            .new_http_session::<HttpPeer, ()>(&peer)\n            .await\n            .unwrap();\n        match h2 {\n            HttpSession::H1(_) => {}\n            HttpSession::H2(_) => panic!(\"expect h1\"),\n            HttpSession::Custom(_) => panic!(\"expect h1\"),\n        }\n    }\n\n    #[tokio::test]\n    async fn test_connect_h1_plaintext() {\n        let connector = Connector::new(None);\n        let mut peer = HttpPeer::new((\"1.1.1.1\", 80), false, \"\".into());\n        peer.options.set_http_version(2, 1);\n        let h2 = connector\n            .new_http_session::<HttpPeer, ()>(&peer)\n            .await\n            .unwrap();\n        match h2 {\n            HttpSession::H1(_) => {}\n            HttpSession::H2(_) => panic!(\"expect h1\"),\n            HttpSession::Custom(_) => panic!(\"expect h1\"),\n        }\n    }\n\n    #[tokio::test]\n    #[cfg(feature = \"any_tls\")]\n    async fn test_h2_single_stream() {\n        let connector = Connector::new(None);\n        let mut peer = HttpPeer::new((\"1.1.1.1\", 443), true, \"one.one.one.one\".into());\n        peer.options.set_http_version(2, 2);\n        peer.options.max_h2_streams = 1;\n        let h2 = connector\n            .new_http_session::<HttpPeer, ()>(&peer)\n            .await\n            .unwrap();\n        let h2_1 = match h2 {\n            HttpSession::H1(_) => panic!(\"expect h2\"),\n            HttpSession::H2(h2_stream) => h2_stream,\n            HttpSession::Custom(_) => panic!(\"expect h2\"),\n        };\n\n        let id = h2_1.conn.id();\n\n        assert!(connector\n            .reused_http_session(&peer)\n            .await\n            .unwrap()\n            .is_none());\n\n        connector.release_http_session(h2_1, &peer, None);\n\n        let h2_2 = connector.reused_http_session(&peer).await.unwrap().unwrap();\n        assert_eq!(id, h2_2.conn.id());\n\n        connector.release_http_session(h2_2, &peer, None);\n\n        let h2_3 = connector.reused_http_session(&peer).await.unwrap().unwrap();\n        assert_eq!(id, h2_3.conn.id());\n    }\n\n    #[tokio::test]\n    #[cfg(feature = \"any_tls\")]\n    async fn test_h2_multiple_stream() {\n        let connector = Connector::new(None);\n        let mut peer = HttpPeer::new((\"1.1.1.1\", 443), true, \"one.one.one.one\".into());\n        peer.options.set_http_version(2, 2);\n        peer.options.max_h2_streams = 3;\n        let h2 = connector\n            .new_http_session::<HttpPeer, ()>(&peer)\n            .await\n            .unwrap();\n        let h2_1 = match h2 {\n            HttpSession::H1(_) => panic!(\"expect h2\"),\n            HttpSession::H2(h2_stream) => h2_stream,\n            HttpSession::Custom(_) => panic!(\"expect h2\"),\n        };\n\n        let id = h2_1.conn.id();\n\n        let h2_2 = connector.reused_http_session(&peer).await.unwrap().unwrap();\n        assert_eq!(id, h2_2.conn.id());\n        let h2_3 = connector.reused_http_session(&peer).await.unwrap().unwrap();\n        assert_eq!(id, h2_3.conn.id());\n\n        // max stream is 3 for now\n        assert!(connector\n            .reused_http_session(&peer)\n            .await\n            .unwrap()\n            .is_none());\n\n        connector.release_http_session(h2_1, &peer, None);\n\n        let h2_4 = connector.reused_http_session(&peer).await.unwrap().unwrap();\n        assert_eq!(id, h2_4.conn.id());\n\n        connector.release_http_session(h2_2, &peer, None);\n        connector.release_http_session(h2_3, &peer, None);\n        connector.release_http_session(h2_4, &peer, None);\n\n        // all streams are released, now the connection is idle\n        let h2_5 = connector.reused_http_session(&peer).await.unwrap().unwrap();\n        assert_eq!(id, h2_5.conn.id());\n    }\n\n    #[cfg(all(feature = \"any_tls\", unix))]\n    #[tokio::test]\n    async fn test_h2_reuse_rejects_fd_mismatch() {\n        use crate::protocols::l4::socket::SocketAddr;\n        use crate::upstreams::peer::Peer;\n        use std::fmt::{Display, Formatter, Result as FmtResult};\n        use std::os::unix::prelude::AsRawFd;\n\n        #[derive(Clone)]\n        struct MismatchPeer {\n            reuse_hash: u64,\n            address: SocketAddr,\n        }\n\n        impl Display for MismatchPeer {\n            fn fmt(&self, f: &mut Formatter<'_>) -> FmtResult {\n                write!(f, \"{:?}\", self.address)\n            }\n        }\n\n        impl Peer for MismatchPeer {\n            fn address(&self) -> &SocketAddr {\n                &self.address\n            }\n\n            fn tls(&self) -> bool {\n                true\n            }\n\n            fn sni(&self) -> &str {\n                \"\"\n            }\n\n            fn reuse_hash(&self) -> u64 {\n                self.reuse_hash\n            }\n\n            fn matches_fd<V: AsRawFd>(&self, _fd: V) -> bool {\n                false\n            }\n        }\n\n        let connector = Connector::new(None);\n        let mut peer = HttpPeer::new((\"1.1.1.1\", 443), true, \"one.one.one.one\".into());\n        peer.options.set_http_version(2, 2);\n        peer.options.max_h2_streams = 1;\n\n        let h2 = connector\n            .new_http_session::<HttpPeer, ()>(&peer)\n            .await\n            .unwrap();\n        let h2_stream = match h2 {\n            HttpSession::H1(_) => panic!(\"expect h2\"),\n            HttpSession::H2(h2_stream) => h2_stream,\n            HttpSession::Custom(_) => panic!(\"expect h2\"),\n        };\n\n        connector.release_http_session(h2_stream, &peer, None);\n\n        let mismatch_peer = MismatchPeer {\n            reuse_hash: peer.reuse_hash(),\n            address: peer.address().clone(),\n        };\n\n        assert!(connector\n            .reused_http_session(&mismatch_peer)\n            .await\n            .unwrap()\n            .is_none());\n    }\n}\n"
  },
  {
    "path": "pingora-core/src/connectors/l4.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n#[cfg(unix)]\nuse crate::protocols::l4::ext::connect_uds;\nuse crate::protocols::l4::ext::{\n    connect_with as tcp_connect, set_dscp, set_recv_buf, set_tcp_fastopen_connect,\n};\nuse crate::protocols::l4::socket::SocketAddr;\nuse crate::protocols::l4::stream::Stream;\nuse crate::protocols::{GetSocketDigest, SocketDigest};\nuse crate::upstreams::peer::Peer;\nuse async_trait::async_trait;\nuse log::debug;\nuse pingora_error::{Context, Error, ErrorType::*, OrErr, Result};\nuse rand::seq::SliceRandom;\nuse std::net::SocketAddr as InetSocketAddr;\n#[cfg(unix)]\nuse std::os::unix::io::AsRawFd;\n#[cfg(windows)]\nuse std::os::windows::io::AsRawSocket;\n\n/// The interface to establish a L4 connection\n#[async_trait]\npub trait Connect: std::fmt::Debug {\n    async fn connect(&self, addr: &SocketAddr) -> Result<Stream>;\n}\n\n/// Settings for binding on connect\n#[derive(Clone, Debug, Default)]\npub struct BindTo {\n    // local ip address\n    pub addr: Option<InetSocketAddr>,\n    // port range\n    port_range: Option<(u16, u16)>,\n    // whether we fallback and try again on bind errors when a port range is set\n    fallback: bool,\n}\n\nimpl BindTo {\n    /// Sets the port range we will bind to where the first item in the tuple is the lower bound\n    /// and the second item is the upper bound.\n    ///\n    /// Note this bind option is only supported on Linux since 6.3, this is a no-op on other systems.\n    /// To reset the range, pass a `None` or `Some((0,0))`, more information can be found [here](https://man7.org/linux/man-pages/man7/ip.7.html)\n    pub fn set_port_range(&mut self, range: Option<(u16, u16)>) -> Result<()> {\n        if range.is_none() && self.port_range.is_none() {\n            // nothing to do\n            return Ok(());\n        }\n\n        match range {\n            // 0,0 is valid for resets\n            None | Some((0, 0)) => self.port_range = Some((0, 0)),\n            // set the port range if valid\n            Some((low, high)) if low > 0 && low < high => {\n                self.port_range = Some((low, high));\n            }\n            _ => return Error::e_explain(SocketError, \"invalid port range: {range}\"),\n        }\n        Ok(())\n    }\n\n    /// Set whether we fallback on no address available if a port range is set\n    pub fn set_fallback(&mut self, fallback: bool) {\n        self.fallback = fallback\n    }\n\n    /// Configured bind port range\n    pub fn port_range(&self) -> Option<(u16, u16)> {\n        self.port_range\n    }\n\n    /// Whether we attempt to fallback on no address available\n    pub fn will_fallback(&self) -> bool {\n        self.fallback && self.port_range.is_some()\n    }\n}\n\n/// Establish a connection (l4) to the given peer using its settings and an optional bind address.\npub(crate) async fn connect<P>(peer: &P, bind_to: Option<BindTo>) -> Result<Stream>\nwhere\n    P: Peer + Send + Sync,\n{\n    if peer.get_proxy().is_some() {\n        return proxy_connect(peer)\n            .await\n            .err_context(|| format!(\"Fail to establish CONNECT proxy: {}\", peer));\n    }\n    let peer_addr = peer.address();\n    let mut stream: Stream =\n        if let Some(custom_l4) = peer.get_peer_options().and_then(|o| o.custom_l4.as_ref()) {\n            custom_l4.connect(peer_addr).await?\n        } else {\n            match peer_addr {\n                SocketAddr::Inet(addr) => {\n                    let connect_future = tcp_connect(addr, bind_to.as_ref(), |socket| {\n                        #[cfg(unix)]\n                        let raw = socket.as_raw_fd();\n                        #[cfg(windows)]\n                        let raw = socket.as_raw_socket();\n\n                        if peer.tcp_fast_open() {\n                            set_tcp_fastopen_connect(raw)?;\n                        }\n                        if let Some(recv_buf) = peer.tcp_recv_buf() {\n                            debug!(\"Setting recv buf size\");\n                            set_recv_buf(raw, recv_buf)?;\n                        }\n                        if let Some(dscp) = peer.dscp() {\n                            debug!(\"Setting dscp\");\n                            set_dscp(raw, dscp)?;\n                        }\n\n                        if let Some(tweak_hook) = peer\n                            .get_peer_options()\n                            .and_then(|o| o.upstream_tcp_sock_tweak_hook.clone())\n                        {\n                            tweak_hook(socket)?;\n                        }\n\n                        Ok(())\n                    });\n                    let conn_res = match peer.connection_timeout() {\n                        Some(t) => pingora_timeout::timeout(t, connect_future)\n                            .await\n                            .explain_err(ConnectTimedout, |_| {\n                                format!(\"timeout {t:?} connecting to server {peer}\")\n                            })?,\n                        None => connect_future.await,\n                    };\n                    match conn_res {\n                        Ok(socket) => {\n                            debug!(\"connected to new server: {}\", peer.address());\n                            Ok(socket.into())\n                        }\n                        Err(e) => {\n                            let c = format!(\"Fail to connect to {peer}\");\n                            match e.etype() {\n                                SocketError | BindError => Error::e_because(InternalError, c, e),\n                                _ => Err(e.more_context(c)),\n                            }\n                        }\n                    }\n                }\n                #[cfg(unix)]\n                SocketAddr::Unix(addr) => {\n                    let connect_future = connect_uds(\n                        addr.as_pathname()\n                            .expect(\"non-pathname unix sockets not supported as peer\"),\n                    );\n                    let conn_res = match peer.connection_timeout() {\n                        Some(t) => pingora_timeout::timeout(t, connect_future)\n                            .await\n                            .explain_err(ConnectTimedout, |_| {\n                                format!(\"timeout {t:?} connecting to server {peer}\")\n                            })?,\n                        None => connect_future.await,\n                    };\n                    match conn_res {\n                        Ok(socket) => {\n                            debug!(\"connected to new server: {}\", peer.address());\n                            Ok(socket.into())\n                        }\n                        Err(e) => {\n                            let c = format!(\"Fail to connect to {peer}\");\n                            match e.etype() {\n                                SocketError | BindError => Error::e_because(InternalError, c, e),\n                                _ => Err(e.more_context(c)),\n                            }\n                        }\n                    }\n                }\n            }?\n        };\n\n    let tracer = peer.get_tracer();\n    if let Some(t) = tracer {\n        t.0.on_connected();\n        stream.tracer = Some(t);\n    }\n\n    // settings applied based on stream type\n    if let Some(ka) = peer.tcp_keepalive() {\n        stream.set_keepalive(ka)?;\n    }\n    stream.set_nodelay()?;\n\n    #[cfg(unix)]\n    let digest = SocketDigest::from_raw_fd(stream.as_raw_fd());\n    #[cfg(windows)]\n    let digest = SocketDigest::from_raw_socket(stream.as_raw_socket());\n    digest\n        .peer_addr\n        .set(Some(peer_addr.clone()))\n        .expect(\"newly created OnceCell must be empty\");\n    stream.set_socket_digest(digest);\n\n    Ok(stream)\n}\n\npub(crate) fn bind_to_random<P: Peer>(\n    peer: &P,\n    v4_list: &[InetSocketAddr],\n    v6_list: &[InetSocketAddr],\n) -> Option<BindTo> {\n    // helper function for randomly picking address\n    fn bind_to_ips(ips: &[InetSocketAddr]) -> Option<InetSocketAddr> {\n        match ips.len() {\n            0 => None,\n            1 => Some(ips[0]),\n            _ => {\n                // pick a random bind ip\n                ips.choose(&mut rand::thread_rng()).copied()\n            }\n        }\n    }\n\n    let mut bind_to = peer.get_peer_options().and_then(|o| o.bind_to.clone());\n    if bind_to.as_ref().map(|b| b.addr).is_some() {\n        // already have a bind address selected\n        return bind_to;\n    }\n\n    let addr = match peer.address() {\n        SocketAddr::Inet(sockaddr) => match sockaddr {\n            InetSocketAddr::V4(_) => bind_to_ips(v4_list),\n            InetSocketAddr::V6(_) => bind_to_ips(v6_list),\n        },\n        #[cfg(unix)]\n        SocketAddr::Unix(_) => None,\n    };\n\n    if addr.is_some() {\n        if let Some(bind_to) = bind_to.as_mut() {\n            bind_to.addr = addr;\n        } else {\n            bind_to = Some(BindTo {\n                addr,\n                ..Default::default()\n            });\n        }\n    }\n    bind_to\n}\n\nuse crate::protocols::raw_connect;\n\n#[cfg(unix)]\nasync fn proxy_connect<P: Peer>(peer: &P) -> Result<Stream> {\n    // safe to unwrap\n    let proxy = peer.get_proxy().unwrap();\n    let options = peer.get_peer_options().unwrap();\n\n    // combine required and optional headers\n    let mut headers = proxy\n        .headers\n        .iter()\n        .chain(options.extra_proxy_headers.iter());\n\n    // not likely to timeout during connect() to UDS\n    let stream: Box<Stream> = Box::new(\n        connect_uds(&proxy.next_hop)\n            .await\n            .or_err_with(ConnectError, || {\n                format!(\"CONNECT proxy connect() error to {:?}\", &proxy.next_hop)\n            })?\n            .into(),\n    );\n\n    let req_header = raw_connect::generate_connect_header(&proxy.host, proxy.port, &mut headers)?;\n    let fut = raw_connect::connect(stream, &req_header, peer);\n    let (mut stream, digest) = match peer.connection_timeout() {\n        Some(t) => pingora_timeout::timeout(t, fut)\n            .await\n            .explain_err(ConnectTimedout, |_| \"establishing CONNECT proxy\")?,\n        None => fut.await,\n    }\n    .map_err(|mut e| {\n        // http protocol may ask to retry if reused client\n        e.retry.decide_reuse(false);\n        e\n    })?;\n    debug!(\"CONNECT proxy established: {:?}\", proxy);\n    stream.set_proxy_digest(digest);\n    let stream = stream.into_any().downcast::<Stream>().unwrap(); // safe, it is Stream from above\n    Ok(*stream)\n}\n\n#[cfg(windows)]\nasync fn proxy_connect<P: Peer>(peer: &P) -> Result<Stream> {\n    panic!(\"peer proxy not supported on windows\")\n}\n\n#[cfg(test)]\nmod tests {\n    use super::*;\n    use crate::upstreams::peer::{BasicPeer, HttpPeer, Proxy};\n    use pingora_error::ErrorType;\n    use std::collections::BTreeMap;\n    use std::path::PathBuf;\n    use std::sync::atomic::{AtomicBool, Ordering};\n    use std::sync::Arc;\n    use std::time::{Duration, Instant};\n    use tokio::io::AsyncWriteExt;\n    use tokio::time::sleep;\n\n    /// Some of the tests below are flaky when making new connections to mock\n    /// servers. The servers are simple tokio listeners, so failures there are\n    /// not indicative of real errors. This function will retry the peer/server\n    /// in increasing intervals until it either succeeds in connecting or a long\n    /// timeout expires (max 10sec)\n    #[cfg(unix)]\n    async fn wait_for_peer<P>(peer: &P)\n    where\n        P: Peer + Send + Sync,\n    {\n        use ErrorType as E;\n        let start = Instant::now();\n        let mut res = connect(peer, None).await;\n        let mut delay = Duration::from_millis(5);\n        let max_delay = Duration::from_secs(10);\n\n        while start.elapsed() < max_delay {\n            match &res {\n                Err(e) if e.etype == E::ConnectRefused => {}\n                _ => break,\n            }\n            sleep(delay).await;\n            delay *= 2;\n            res = connect(peer, None).await;\n        }\n    }\n\n    #[tokio::test]\n    async fn test_conn_error_refused() {\n        let peer = BasicPeer::new(\"127.0.0.1:79\"); // hopefully port 79 is not used\n        let new_session = connect(&peer, None).await;\n        assert_eq!(new_session.unwrap_err().etype(), &ConnectRefused)\n    }\n\n    // TODO broken on arm64\n    #[ignore]\n    #[tokio::test]\n    async fn test_conn_error_no_route() {\n        let peer = BasicPeer::new(\"[::3]:79\"); // no route\n        let new_session = connect(&peer, None).await;\n        assert_eq!(new_session.unwrap_err().etype(), &ConnectNoRoute)\n    }\n\n    #[tokio::test]\n    async fn test_conn_error_addr_not_avail() {\n        let peer = HttpPeer::new(\"127.0.0.1:121\".to_string(), false, \"\".to_string());\n        let addr = \"192.0.2.2:0\".parse().ok();\n        let bind_to = BindTo {\n            addr,\n            ..Default::default()\n        };\n        let new_session = connect(&peer, Some(bind_to)).await;\n        assert_eq!(new_session.unwrap_err().etype(), &InternalError)\n    }\n\n    #[tokio::test]\n    async fn test_conn_error_other() {\n        let peer = HttpPeer::new(\"240.0.0.1:80\".to_string(), false, \"\".to_string()); // non localhost\n        let addr = \"127.0.0.1:0\".parse().ok();\n        // create an error: cannot send from src addr: localhost to dst addr: a public IP\n        let bind_to = BindTo {\n            addr,\n            ..Default::default()\n        };\n        let new_session = connect(&peer, Some(bind_to)).await;\n        let error = new_session.unwrap_err();\n        // XXX: some system will allow the socket to bind and connect without error, only to timeout\n        assert!(\n            error.etype() == &ConnectError\n                || error.etype() == &ConnectTimedout\n                // The error seen on mac: https://github.com/cloudflare/pingora/pull/679\n                || (error.etype() == &InternalError),\n            \"{error:?}\"\n        )\n    }\n\n    #[tokio::test]\n    async fn test_conn_timeout() {\n        // 192.0.2.1 is effectively a blackhole\n        let mut peer = BasicPeer::new(\"192.0.2.1:79\");\n        peer.options.connection_timeout = Some(std::time::Duration::from_millis(1)); //1ms\n        let new_session = connect(&peer, None).await;\n        assert_eq!(new_session.unwrap_err().etype(), &ConnectTimedout)\n    }\n\n    #[tokio::test]\n    async fn test_tweak_hook() {\n        const INIT_FLAG: bool = false;\n\n        let flag = Arc::new(AtomicBool::new(INIT_FLAG));\n\n        let mut peer = BasicPeer::new(\"1.1.1.1:80\");\n\n        let move_flag = Arc::clone(&flag);\n\n        peer.options.upstream_tcp_sock_tweak_hook = Some(Arc::new(move |_| {\n            move_flag.fetch_xor(true, Ordering::SeqCst);\n            Ok(())\n        }));\n\n        connect(&peer, None).await.unwrap();\n\n        assert_eq!(!INIT_FLAG, flag.load(Ordering::SeqCst));\n    }\n\n    #[tokio::test]\n    async fn test_custom_connect() {\n        #[derive(Debug)]\n        struct MyL4;\n        #[async_trait]\n        impl Connect for MyL4 {\n            async fn connect(&self, _addr: &SocketAddr) -> Result<Stream> {\n                tokio::net::TcpStream::connect(\"1.1.1.1:80\")\n                    .await\n                    .map(|s| s.into())\n                    .or_fail()\n            }\n        }\n        // :79 shouldn't be able to be connected to\n        let mut peer = BasicPeer::new(\"1.1.1.1:79\");\n        peer.options.custom_l4 = Some(std::sync::Arc::new(MyL4 {}));\n\n        let new_session = connect(&peer, None).await;\n\n        // but MyL4 connects to :80 instead\n        assert!(new_session.is_ok());\n    }\n\n    #[cfg(unix)]\n    #[tokio::test]\n    async fn test_connect_proxy_fail() {\n        let mut peer = HttpPeer::new(\"1.1.1.1:80\".to_string(), false, \"\".to_string());\n        let mut path = PathBuf::new();\n        path.push(\"/tmp/123\");\n        peer.proxy = Some(Proxy {\n            next_hop: path.into(),\n            host: \"1.1.1.1\".into(),\n            port: 80,\n            headers: BTreeMap::new(),\n        });\n        let new_session = connect(&peer, None).await;\n        let e = new_session.unwrap_err();\n        assert_eq!(e.etype(), &ConnectError);\n        assert!(!e.retry());\n    }\n\n    #[cfg(unix)]\n    #[tokio::test(flavor = \"multi_thread\")]\n    async fn test_connect_proxy_work() {\n        use crate::connectors::test_utils;\n\n        let socket_path = test_utils::unique_uds_path(\"connect_proxy_work\");\n        let (ready_rx, shutdown_tx, server_handle) =\n            test_utils::spawn_mock_uds_server(socket_path.clone(), b\"HTTP/1.1 200 OK\\r\\n\\r\\n\");\n\n        // Wait for the server to be ready\n        ready_rx.await.unwrap();\n\n        let mut peer = HttpPeer::new(\"1.1.1.1:80\".to_string(), false, \"\".to_string());\n        let mut path = PathBuf::new();\n        path.push(&socket_path);\n        peer.proxy = Some(Proxy {\n            next_hop: path.into(),\n            host: \"1.1.1.1\".into(),\n            port: 80,\n            headers: BTreeMap::new(),\n        });\n        let new_session = connect(&peer, None).await;\n        assert!(new_session.is_ok());\n\n        // Clean up\n        let _ = shutdown_tx.send(());\n        server_handle.await.unwrap();\n    }\n\n    #[cfg(unix)]\n    #[tokio::test(flavor = \"multi_thread\")]\n    async fn test_connect_proxy_conn_closed() {\n        use crate::connectors::test_utils;\n\n        let socket_path = test_utils::unique_uds_path(\"connect_proxy_conn_closed\");\n        let (ready_rx, shutdown_tx, server_handle) =\n            test_utils::spawn_mock_uds_server_close_immediate(socket_path.clone());\n\n        // Wait for the server to be ready\n        ready_rx.await.unwrap();\n\n        let mut peer = HttpPeer::new(\"1.1.1.1:80\".to_string(), false, \"\".to_string());\n        let mut path = PathBuf::new();\n        path.push(&socket_path);\n        peer.proxy = Some(Proxy {\n            next_hop: path.into(),\n            host: \"1.1.1.1\".into(),\n            port: 80,\n            headers: BTreeMap::new(),\n        });\n        let new_session = connect(&peer, None).await;\n        let err = new_session.unwrap_err();\n        assert_eq!(err.etype(), &ConnectionClosed);\n        assert!(!err.retry());\n\n        // Clean up\n        let _ = shutdown_tx.send(());\n        server_handle.await.unwrap();\n    }\n\n    #[cfg(target_os = \"linux\")]\n    #[tokio::test(flavor = \"multi_thread\")]\n    async fn test_bind_to_port_range_on_connect() {\n        fn get_ip_local_port_range() -> (u16, u16) {\n            let path = \"/proc/sys/net/ipv4/ip_local_port_range\";\n            let file = std::fs::read_to_string(path).unwrap();\n            let mut parts = file.split_whitespace();\n            (\n                parts.next().unwrap().parse().unwrap(),\n                parts.next().unwrap().parse().unwrap(),\n            )\n        }\n\n        // one-off mock server\n        async fn mock_inet_connect_server() -> u16 {\n            use tokio::net::TcpListener;\n            let listener = TcpListener::bind(\"127.0.0.1:0\").await.unwrap();\n\n            let port = listener.local_addr().unwrap().port();\n\n            tokio::spawn(async move {\n                if let Ok((mut stream, _addr)) = listener.accept().await {\n                    stream.write_all(b\"HTTP/1.1 200 OK\\r\\n\\r\\n\").await.unwrap();\n                    // wait a bit so that the client can read\n                    tokio::time::sleep(std::time::Duration::from_millis(100)).await;\n                }\n            });\n\n            port\n        }\n\n        fn in_port_range(session: Stream, lower: u16, upper: u16) -> bool {\n            let digest = session.get_socket_digest();\n            let local_addr = digest\n                .as_ref()\n                .and_then(|s| s.local_addr())\n                .unwrap()\n                .as_inet()\n                .unwrap();\n\n            // assert range\n            local_addr.port() >= lower && local_addr.port() <= upper\n        }\n\n        let port = mock_inet_connect_server().await;\n\n        // need to read /proc/sys/net/ipv4/ip_local_port_range for this test to work\n        // IP_LOCAL_PORT_RANGE clamp only works on ports in /proc/sys/net/ipv4/ip_local_port_range\n        let (low, _) = get_ip_local_port_range();\n        let high = low + 1;\n\n        let peer = HttpPeer::new(format!(\"127.0.0.1:{port}\"), false, \"\".to_string());\n        let mut bind_to = BindTo {\n            addr: \"127.0.0.1:0\".parse().ok(),\n            ..Default::default()\n        };\n\n        // wait for the server to start\n        wait_for_peer(&peer).await;\n\n        bind_to.set_port_range(Some((low, high))).unwrap();\n\n        let mut success_count = 0;\n        let mut address_unavailable_count = 0;\n\n        // Issue a bunch of requests at once and ensure that all successful\n        // requests have ports in the right range and that there is at least\n        // one address-unavailable error because we are restricting the number\n        // of ports so heavily\n        for _ in 0..10 {\n            match connect(&peer, Some(bind_to.clone())).await {\n                Ok(session) => {\n                    assert!(in_port_range(session, low, high));\n                    success_count += 1;\n                }\n                Err(e) if format!(\"{e:?}\").contains(\"AddrNotAvailable\") => {\n                    address_unavailable_count += 1;\n                }\n                Err(e) => {\n                    panic!(\"Unexpected error {e:?}\")\n                }\n            }\n        }\n\n        assert!(address_unavailable_count > 0);\n        assert!(success_count >= (high - low));\n\n        // enable fallback, assert not in port range but successful\n        bind_to.set_fallback(true);\n        let session4 = connect(&peer, Some(bind_to.clone())).await.unwrap();\n        assert!(!in_port_range(session4, low, high));\n\n        // works without bind IP, shift up to use new ports\n        let low = low + 2;\n        let high = low + 1;\n        let mut bind_to = BindTo::default();\n        bind_to.set_port_range(Some((low, high))).unwrap();\n        let session5 = connect(&peer, Some(bind_to.clone())).await.unwrap();\n        assert!(in_port_range(session5, low, high));\n    }\n\n    #[test]\n    fn test_bind_to_port_ranges() {\n        let addr = \"127.0.0.1:0\".parse().ok();\n        let mut bind_to = BindTo {\n            addr,\n            ..Default::default()\n        };\n\n        // None because the previous value was None\n        bind_to.set_port_range(None).unwrap();\n        assert!(bind_to.port_range.is_none());\n\n        // zeroes are handled\n        bind_to.set_port_range(Some((0, 0))).unwrap();\n        assert_eq!(bind_to.port_range, Some((0, 0)));\n\n        // zeroes because the previous value was Some\n        bind_to.set_port_range(None).unwrap();\n        assert_eq!(bind_to.port_range, Some((0, 0)));\n\n        // low > high is error\n        assert!(bind_to.set_port_range(Some((2000, 1000))).is_err());\n\n        // low < high success\n        bind_to.set_port_range(Some((1000, 2000))).unwrap();\n        assert_eq!(bind_to.port_range, Some((1000, 2000)));\n    }\n}\n"
  },
  {
    "path": "pingora-core/src/connectors/mod.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! Connecting to servers\n\npub mod http;\npub mod l4;\nmod offload;\n\n#[cfg(feature = \"any_tls\")]\nmod tls;\n\n#[cfg(not(feature = \"any_tls\"))]\nuse crate::tls::connectors as tls;\n\nuse crate::protocols::Stream;\nuse crate::server::configuration::ServerConf;\nuse crate::upstreams::peer::{Peer, ALPN};\n\npub use l4::Connect as L4Connect;\nuse l4::{connect as l4_connect, BindTo};\nuse log::{debug, error, warn};\nuse offload::OffloadRuntime;\nuse parking_lot::RwLock;\nuse pingora_error::{Error, ErrorType::*, OrErr, Result};\nuse pingora_pool::{ConnectionMeta, ConnectionPool};\nuse std::collections::HashMap;\nuse std::net::SocketAddr;\nuse std::sync::Arc;\nuse tls::TlsConnector;\nuse tokio::sync::Mutex;\n\n/// The options to configure a [TransportConnector]\n#[derive(Clone)]\npub struct ConnectorOptions {\n    /// Path to the CA file used to validate server certs.\n    ///\n    /// If `None`, the CA in the [default](https://www.openssl.org/docs/manmaster/man3/SSL_CTX_set_default_verify_paths.html)\n    /// locations will be loaded\n    pub ca_file: Option<String>,\n    /// The maximum number of unique s2n configs to cache. Creating a new s2n config is an\n    /// expensive operation, so we cache and re-use config objects with identical configurations.\n    /// Defaults to a cache size of 10. A value of 0 disables the cache.\n    ///\n    /// WARNING: Disabling the s2n config cache can result in poor performance\n    #[cfg(feature = \"s2n\")]\n    pub s2n_config_cache_size: Option<usize>,\n    /// The default client cert and key to use for mTLS\n    ///\n    /// Each individual connection can use their own cert key to override this.\n    pub cert_key_file: Option<(String, String)>,\n    /// When enabled allows TLS keys to be written to a file specified by the SSLKEYLOG\n    /// env variable. This can be used by tools like Wireshark to decrypt traffic\n    /// for debugging purposes.\n    pub debug_ssl_keylog: bool,\n    /// How many connections to keepalive\n    pub keepalive_pool_size: usize,\n    /// Optionally offload the connection establishment to dedicated thread pools\n    ///\n    /// TCP and TLS connection establishment can be CPU intensive. Sometimes such tasks can slow\n    /// down the entire service, which causes timeouts which leads to more connections which\n    /// snowballs the issue. Use this option to isolate these CPU intensive tasks from impacting\n    /// other traffic.\n    ///\n    /// Syntax: (#pools, #thread in each pool)\n    pub offload_threadpool: Option<(usize, usize)>,\n    /// Bind to any of the given source IPv6 addresses\n    pub bind_to_v4: Vec<SocketAddr>,\n    /// Bind to any of the given source IPv4 addresses\n    pub bind_to_v6: Vec<SocketAddr>,\n}\n\nimpl ConnectorOptions {\n    /// Derive the [ConnectorOptions] from a [ServerConf]\n    pub fn from_server_conf(server_conf: &ServerConf) -> Self {\n        // if both pools and threads are Some(>0)\n        let offload_threadpool = server_conf\n            .upstream_connect_offload_threadpools\n            .zip(server_conf.upstream_connect_offload_thread_per_pool)\n            .filter(|(pools, threads)| *pools > 0 && *threads > 0);\n\n        // create SocketAddrs with port 0 for src addr bind\n\n        let bind_to_v4 = server_conf\n            .client_bind_to_ipv4\n            .iter()\n            .map(|v4| {\n                let ip = v4.parse().unwrap();\n                SocketAddr::new(ip, 0)\n            })\n            .collect();\n\n        let bind_to_v6 = server_conf\n            .client_bind_to_ipv6\n            .iter()\n            .map(|v6| {\n                let ip = v6.parse().unwrap();\n                SocketAddr::new(ip, 0)\n            })\n            .collect();\n        ConnectorOptions {\n            ca_file: server_conf.ca_file.clone(),\n            cert_key_file: None, // TODO: use it\n            #[cfg(feature = \"s2n\")]\n            s2n_config_cache_size: server_conf.s2n_config_cache_size,\n            debug_ssl_keylog: server_conf.upstream_debug_ssl_keylog,\n            keepalive_pool_size: server_conf.upstream_keepalive_pool_size,\n            offload_threadpool,\n            bind_to_v4,\n            bind_to_v6,\n        }\n    }\n\n    /// Create a new [ConnectorOptions] with the given keepalive pool size\n    pub fn new(keepalive_pool_size: usize) -> Self {\n        ConnectorOptions {\n            ca_file: None,\n            #[cfg(feature = \"s2n\")]\n            s2n_config_cache_size: None,\n            cert_key_file: None,\n            debug_ssl_keylog: false,\n            keepalive_pool_size,\n            offload_threadpool: None,\n            bind_to_v4: vec![],\n            bind_to_v6: vec![],\n        }\n    }\n}\n\n/// [TransportConnector] provides APIs to connect to servers via TCP or TLS with connection reuse\npub struct TransportConnector {\n    tls_ctx: tls::Connector,\n    connection_pool: Arc<ConnectionPool<Arc<Mutex<Stream>>>>,\n    offload: Option<OffloadRuntime>,\n    bind_to_v4: Vec<SocketAddr>,\n    bind_to_v6: Vec<SocketAddr>,\n    preferred_http_version: PreferredHttpVersion,\n}\n\nconst DEFAULT_POOL_SIZE: usize = 128;\n\nimpl TransportConnector {\n    /// Create a new [TransportConnector] with the given [ConnectorOptions]\n    pub fn new(mut options: Option<ConnectorOptions>) -> Self {\n        let pool_size = options\n            .as_ref()\n            .map_or(DEFAULT_POOL_SIZE, |c| c.keepalive_pool_size);\n        // Take the offloading setting there because this layer has implement offloading,\n        // so no need for stacks at lower layer to offload again.\n        let offload = options.as_mut().and_then(|o| o.offload_threadpool.take());\n        let bind_to_v4 = options\n            .as_ref()\n            .map_or_else(Vec::new, |o| o.bind_to_v4.clone());\n        let bind_to_v6 = options\n            .as_ref()\n            .map_or_else(Vec::new, |o| o.bind_to_v6.clone());\n        TransportConnector {\n            tls_ctx: tls::Connector::new(options),\n            connection_pool: Arc::new(ConnectionPool::new(pool_size)),\n            offload: offload.map(|v| OffloadRuntime::new(v.0, v.1)),\n            bind_to_v4,\n            bind_to_v6,\n            preferred_http_version: PreferredHttpVersion::new(),\n        }\n    }\n\n    /// Connect to the given server [Peer]\n    ///\n    /// No connection is reused.\n    pub async fn new_stream<P: Peer + Send + Sync + 'static>(&self, peer: &P) -> Result<Stream> {\n        let rt = self\n            .offload\n            .as_ref()\n            .map(|o| o.get_runtime(peer.reuse_hash()));\n        let bind_to = l4::bind_to_random(peer, &self.bind_to_v4, &self.bind_to_v6);\n        let alpn_override = self.preferred_http_version.get(peer);\n        let stream = if let Some(rt) = rt {\n            let peer = peer.clone();\n            let tls_ctx = self.tls_ctx.clone();\n            rt.spawn(async move { do_connect(&peer, bind_to, alpn_override, &tls_ctx.ctx).await })\n                .await\n                .or_err(InternalError, \"offload runtime failure\")??\n        } else {\n            do_connect(peer, bind_to, alpn_override, &self.tls_ctx.ctx).await?\n        };\n\n        Ok(stream)\n    }\n\n    /// Try to find a reusable connection to the given server [Peer]\n    pub async fn reused_stream<P: Peer + Send + Sync>(&self, peer: &P) -> Option<Stream> {\n        match self.connection_pool.get(&peer.reuse_hash()) {\n            Some(s) => {\n                debug!(\"find reusable stream, trying to acquire it\");\n                {\n                    let _ = s.lock().await;\n                } // wait for the idle poll to release it\n                match Arc::try_unwrap(s) {\n                    Ok(l) => {\n                        let mut stream = l.into_inner();\n                        // test_reusable_stream: we assume server would never actively send data\n                        // first on an idle stream.\n                        #[cfg(unix)]\n                        if peer.matches_fd(stream.id()) && test_reusable_stream(&mut stream) {\n                            Some(stream)\n                        } else {\n                            None\n                        }\n                        #[cfg(windows)]\n                        {\n                            use std::os::windows::io::{AsRawSocket, RawSocket};\n                            struct WrappedRawSocket(RawSocket);\n                            impl AsRawSocket for WrappedRawSocket {\n                                fn as_raw_socket(&self) -> RawSocket {\n                                    self.0\n                                }\n                            }\n                            if peer.matches_sock(WrappedRawSocket(stream.id() as RawSocket))\n                                && test_reusable_stream(&mut stream)\n                            {\n                                Some(stream)\n                            } else {\n                                None\n                            }\n                        }\n                    }\n                    Err(_) => {\n                        error!(\"failed to acquire reusable stream\");\n                        None\n                    }\n                }\n            }\n            None => {\n                debug!(\"No reusable connection found for {peer}\");\n                None\n            }\n        }\n    }\n\n    /// Return the [Stream] to the [TransportConnector] for connection reuse.\n    ///\n    /// Not all TCP/TLS connections can be reused. It is the caller's responsibility to make sure\n    /// that protocol over the [Stream] supports connection reuse and the [Stream] itself is ready\n    /// to be reused.\n    ///\n    /// If a [Stream] is dropped instead of being returned via this function. it will be closed.\n    pub fn release_stream(\n        &self,\n        mut stream: Stream,\n        key: u64, // usually peer.reuse_hash()\n        idle_timeout: Option<std::time::Duration>,\n    ) {\n        if !test_reusable_stream(&mut stream) {\n            return;\n        }\n        let id = stream.id();\n        let meta = ConnectionMeta::new(key, id);\n        debug!(\"Try to keepalive client session\");\n        let stream = Arc::new(Mutex::new(stream));\n        let locked_stream = stream.clone().try_lock_owned().unwrap(); // safe as we just created it\n        let (notify_close, watch_use) = self.connection_pool.put(&meta, stream);\n        let pool = self.connection_pool.clone(); //clone the arc\n        let rt = pingora_runtime::current_handle();\n        rt.spawn(async move {\n            pool.idle_poll(locked_stream, &meta, idle_timeout, notify_close, watch_use)\n                .await;\n        });\n    }\n\n    /// Get a stream to the given server [Peer]\n    ///\n    /// This function will try to find a reusable [Stream] first. If there is none, a new connection\n    /// will be made to the server.\n    ///\n    /// The returned boolean will indicate whether the stream is reused.\n    pub async fn get_stream<P: Peer + Send + Sync + 'static>(\n        &self,\n        peer: &P,\n    ) -> Result<(Stream, bool)> {\n        let reused_stream = self.reused_stream(peer).await;\n        if let Some(s) = reused_stream {\n            Ok((s, true))\n        } else {\n            let s = self.new_stream(peer).await?;\n            Ok((s, false))\n        }\n    }\n\n    /// Tell the connector to always send h1 for ALPN for the given peer in the future.\n    pub fn prefer_h1(&self, peer: &impl Peer) {\n        self.preferred_http_version.add(peer, 1);\n    }\n}\n\n// Perform the actual L4 and tls connection steps while respecting the peer's\n// connection timeout if there is one\nasync fn do_connect<P: Peer + Send + Sync>(\n    peer: &P,\n    bind_to: Option<BindTo>,\n    alpn_override: Option<ALPN>,\n    tls_ctx: &TlsConnector,\n) -> Result<Stream> {\n    // Create the future that does the connections, but don't evaluate it until\n    // we decide if we need a timeout or not\n    let connect_future = do_connect_inner(peer, bind_to, alpn_override, tls_ctx);\n\n    match peer.total_connection_timeout() {\n        Some(t) => match pingora_timeout::timeout(t, connect_future).await {\n            Ok(res) => res,\n            Err(_) => Error::e_explain(\n                ConnectTimedout,\n                format!(\"connecting to server {peer}, total-connection timeout {t:?}\"),\n            ),\n        },\n        None => connect_future.await,\n    }\n}\n\n// Perform the actual L4 and tls connection steps with no timeout\nasync fn do_connect_inner<P: Peer + Send + Sync>(\n    peer: &P,\n    bind_to: Option<BindTo>,\n    alpn_override: Option<ALPN>,\n    tls_ctx: &TlsConnector,\n) -> Result<Stream> {\n    let stream = l4_connect(peer, bind_to).await?;\n    if peer.tls() {\n        let tls_stream = tls::connect(stream, peer, alpn_override, tls_ctx).await?;\n        Ok(Box::new(tls_stream))\n    } else {\n        Ok(Box::new(stream))\n    }\n}\n\nstruct PreferredHttpVersion {\n    // TODO: shard to avoid the global lock\n    versions: RwLock<HashMap<u64, u8>>, // <hash of peer, version>\n}\n\n// TODO: limit the size of this\n\nimpl PreferredHttpVersion {\n    pub fn new() -> Self {\n        PreferredHttpVersion {\n            versions: RwLock::default(),\n        }\n    }\n\n    pub fn add(&self, peer: &impl Peer, version: u8) {\n        let key = peer.reuse_hash();\n        let mut v = self.versions.write();\n        v.insert(key, version);\n    }\n\n    pub fn get(&self, peer: &impl Peer) -> Option<ALPN> {\n        let key = peer.reuse_hash();\n        let v = self.versions.read();\n        v.get(&key)\n            .copied()\n            .map(|v| if v == 1 { ALPN::H1 } else { ALPN::H2H1 })\n    }\n}\n\nuse futures::future::FutureExt;\nuse tokio::io::AsyncReadExt;\n\n/// Test whether a stream is already closed or not reusable (server sent unexpected data)\nfn test_reusable_stream(stream: &mut Stream) -> bool {\n    let mut buf = [0; 1];\n    // tokio::task::unconstrained because now_or_never may yield None when the future is ready\n    let result = tokio::task::unconstrained(stream.read(&mut buf[..])).now_or_never();\n    if let Some(data_result) = result {\n        match data_result {\n            Ok(n) => {\n                if n == 0 {\n                    debug!(\"Idle connection is closed\");\n                } else {\n                    warn!(\"Unexpected data read in idle connection\");\n                }\n            }\n            Err(e) => {\n                debug!(\"Idle connection is broken: {e:?}\");\n            }\n        }\n        false\n    } else {\n        true\n    }\n}\n\n/// Test utilities for creating mock acceptors.\n#[cfg(all(test, unix))]\npub(crate) mod test_utils {\n    use tokio::io::AsyncWriteExt;\n    use tokio::net::UnixListener;\n\n    /// Generates a unique socket path for testing to avoid conflicts when running in parallel\n    pub fn unique_uds_path(test_name: &str) -> String {\n        format!(\n            \"/tmp/test_{test_name}_{:?}_{}.sock\",\n            std::thread::current().id(),\n            std::process::id()\n        )\n    }\n\n    /// A mock UDS server that accepts one connection, sends data, and waits for shutdown signal\n    ///\n    /// Returns: (ready_rx, shutdown_tx, server_handle)\n    /// - ready_rx: Wait on this to know when server is ready to accept connections\n    /// - shutdown_tx: Send on this to tell server to shut down\n    /// - server_handle: Join handle for the server task\n    pub fn spawn_mock_uds_server(\n        socket_path: String,\n        response: &'static [u8],\n    ) -> (\n        tokio::sync::oneshot::Receiver<()>,\n        tokio::sync::oneshot::Sender<()>,\n        tokio::task::JoinHandle<()>,\n    ) {\n        let (ready_tx, ready_rx) = tokio::sync::oneshot::channel();\n        let (shutdown_tx, shutdown_rx) = tokio::sync::oneshot::channel();\n\n        let server_handle = tokio::spawn(async move {\n            let _ = std::fs::remove_file(&socket_path);\n            let listener = UnixListener::bind(&socket_path).unwrap();\n            // Signal that the server is ready to accept connections\n            let _ = ready_tx.send(());\n\n            if let Ok((mut stream, _addr)) = listener.accept().await {\n                let _ = stream.write_all(response).await;\n                // Keep the connection open until the test tells us to shutdown\n                let _ = shutdown_rx.await;\n            }\n            let _ = std::fs::remove_file(&socket_path);\n        });\n\n        (ready_rx, shutdown_tx, server_handle)\n    }\n\n    /// A mock UDS server that immediately closes connections (for testing error handling)\n    ///\n    /// Returns: (ready_rx, shutdown_tx, server_handle)\n    pub fn spawn_mock_uds_server_close_immediate(\n        socket_path: String,\n    ) -> (\n        tokio::sync::oneshot::Receiver<()>,\n        tokio::sync::oneshot::Sender<()>,\n        tokio::task::JoinHandle<()>,\n    ) {\n        let (ready_tx, ready_rx) = tokio::sync::oneshot::channel();\n        let (shutdown_tx, shutdown_rx) = tokio::sync::oneshot::channel();\n\n        let server_handle = tokio::spawn(async move {\n            let _ = std::fs::remove_file(&socket_path);\n            let listener = UnixListener::bind(&socket_path).unwrap();\n            // Signal that the server is ready to accept connections\n            let _ = ready_tx.send(());\n\n            if let Ok((mut stream, _addr)) = listener.accept().await {\n                let _ = stream.shutdown().await;\n                // Wait for shutdown signal before cleaning up\n                let _ = shutdown_rx.await;\n            }\n            let _ = std::fs::remove_file(&socket_path);\n        });\n\n        (ready_rx, shutdown_tx, server_handle)\n    }\n}\n\n#[cfg(test)]\n#[cfg(feature = \"any_tls\")]\nmod tests {\n    use pingora_error::ErrorType;\n    use tls::Connector;\n\n    use super::*;\n    use crate::upstreams::peer::BasicPeer;\n\n    // 192.0.2.1 is effectively a black hole\n    const BLACK_HOLE: &str = \"192.0.2.1:79\";\n\n    #[tokio::test]\n    async fn test_connect() {\n        let connector = TransportConnector::new(None);\n        let peer = BasicPeer::new(\"1.1.1.1:80\");\n        // make a new connection to 1.1.1.1\n        let stream = connector.new_stream(&peer).await.unwrap();\n        connector.release_stream(stream, peer.reuse_hash(), None);\n\n        let (_, reused) = connector.get_stream(&peer).await.unwrap();\n        assert!(reused);\n    }\n\n    #[tokio::test]\n    async fn test_connect_tls() {\n        let connector = TransportConnector::new(None);\n        let mut peer = BasicPeer::new(\"1.1.1.1:443\");\n        // BasicPeer will use tls when SNI is set\n        peer.sni = \"one.one.one.one\".to_string();\n        // make a new connection to https://1.1.1.1\n        let stream = connector.new_stream(&peer).await.unwrap();\n        connector.release_stream(stream, peer.reuse_hash(), None);\n\n        let (_, reused) = connector.get_stream(&peer).await.unwrap();\n        assert!(reused);\n    }\n\n    #[tokio::test(flavor = \"multi_thread\")]\n    #[cfg(unix)]\n    async fn test_connect_uds() {\n        let socket_path = test_utils::unique_uds_path(\"transport_connector\");\n        let (ready_rx, shutdown_tx, server_handle) =\n            test_utils::spawn_mock_uds_server(socket_path.clone(), b\"it works!\");\n\n        // Wait for the server to be ready before connecting\n        ready_rx.await.unwrap();\n\n        // create a new service at /tmp\n        let connector = TransportConnector::new(None);\n        let peer = BasicPeer::new_uds(&socket_path).unwrap();\n        // make a new connection to mock uds\n        let mut stream = connector.new_stream(&peer).await.unwrap();\n        let mut buf = [0; 9];\n        let _ = stream.read(&mut buf).await.unwrap();\n        assert_eq!(&buf, b\"it works!\");\n\n        // Test connection reuse by releasing and getting the stream back\n        connector.release_stream(stream, peer.reuse_hash(), None);\n        let (stream, reused) = connector.get_stream(&peer).await.unwrap();\n        assert!(reused);\n\n        // Clean up: drop the stream, tell server to shutdown, and wait for it\n        drop(stream);\n        let _ = shutdown_tx.send(());\n        server_handle.await.unwrap();\n    }\n\n    async fn do_test_conn_timeout(conf: Option<ConnectorOptions>) {\n        let connector = TransportConnector::new(conf);\n        let mut peer = BasicPeer::new(BLACK_HOLE);\n        peer.options.connection_timeout = Some(std::time::Duration::from_millis(1));\n        let stream = connector.new_stream(&peer).await;\n        match stream {\n            Ok(_) => panic!(\"should throw an error\"),\n            Err(e) => assert_eq!(e.etype(), &ConnectTimedout),\n        }\n    }\n\n    #[tokio::test]\n    async fn test_conn_timeout() {\n        do_test_conn_timeout(None).await;\n    }\n\n    #[tokio::test]\n    async fn test_conn_timeout_with_offload() {\n        let mut conf = ConnectorOptions::new(8);\n        conf.offload_threadpool = Some((2, 2));\n        do_test_conn_timeout(Some(conf)).await;\n    }\n\n    #[tokio::test]\n    async fn test_connector_bind_to() {\n        // connect to remote while bind to localhost will fail\n        let peer = BasicPeer::new(\"240.0.0.1:80\");\n        let mut conf = ConnectorOptions::new(1);\n        conf.bind_to_v4.push(\"127.0.0.1:0\".parse().unwrap());\n        let connector = TransportConnector::new(Some(conf));\n\n        let stream = connector.new_stream(&peer).await;\n        let error = stream.unwrap_err();\n        // XXX: some systems will allow the socket to bind and connect without error, only to timeout\n        assert!(error.etype() == &ConnectError || error.etype() == &ConnectTimedout)\n    }\n\n    /// Helper function for testing error handling in the `do_connect` function.\n    /// This assumes that the connection will fail to on the peer and returns\n    /// the decomposed error type and message\n    async fn get_do_connect_failure_with_peer(peer: &BasicPeer) -> (ErrorType, String) {\n        let tls_connector = Connector::new(None);\n        let stream = do_connect(peer, None, None, &tls_connector.ctx).await;\n        match stream {\n            Ok(_) => panic!(\"should throw an error\"),\n            Err(e) => (\n                e.etype().clone(),\n                e.context\n                    .as_ref()\n                    .map(|ctx| ctx.as_str().to_owned())\n                    .unwrap_or_default(),\n            ),\n        }\n    }\n\n    #[tokio::test]\n    async fn test_do_connect_with_total_timeout() {\n        let mut peer = BasicPeer::new(BLACK_HOLE);\n        peer.options.total_connection_timeout = Some(std::time::Duration::from_millis(1));\n        let (etype, context) = get_do_connect_failure_with_peer(&peer).await;\n        assert_eq!(etype, ConnectTimedout);\n        assert!(context.contains(\"total-connection timeout\"));\n    }\n\n    #[tokio::test]\n    async fn test_tls_connect_timeout_supersedes_total() {\n        let mut peer = BasicPeer::new(BLACK_HOLE);\n        peer.options.total_connection_timeout = Some(std::time::Duration::from_millis(10));\n        peer.options.connection_timeout = Some(std::time::Duration::from_millis(1));\n        let (etype, context) = get_do_connect_failure_with_peer(&peer).await;\n        assert_eq!(etype, ConnectTimedout);\n        assert!(!context.contains(\"total-connection timeout\"));\n    }\n\n    #[tokio::test]\n    async fn test_do_connect_without_total_timeout() {\n        let peer = BasicPeer::new(BLACK_HOLE);\n        let (etype, context) = get_do_connect_failure_with_peer(&peer).await;\n        assert!(etype != ConnectTimedout || !context.contains(\"total-connection timeout\"));\n    }\n}\n"
  },
  {
    "path": "pingora-core/src/connectors/offload.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\nuse log::debug;\nuse once_cell::sync::OnceCell;\nuse rand::Rng;\nuse tokio::runtime::{Builder, Handle};\nuse tokio::sync::oneshot::{channel, Sender};\n\n// TODO: use pingora_runtime\n// a shared runtime (thread pools)\npub(crate) struct OffloadRuntime {\n    shards: usize,\n    thread_per_shard: usize,\n    // Lazily init the runtimes so that they are created after pingora\n    // daemonize itself. Otherwise the runtime threads are lost.\n    pools: OnceCell<Box<[(Handle, Sender<()>)]>>,\n}\n\nimpl OffloadRuntime {\n    pub fn new(shards: usize, thread_per_shard: usize) -> Self {\n        assert!(shards != 0);\n        assert!(thread_per_shard != 0);\n        OffloadRuntime {\n            shards,\n            thread_per_shard,\n            pools: OnceCell::new(),\n        }\n    }\n\n    fn init_pools(&self) -> Box<[(Handle, Sender<()>)]> {\n        let threads = self.shards * self.thread_per_shard;\n        let mut pools = Vec::with_capacity(threads);\n        for _ in 0..threads {\n            // We use single thread runtimes to reduce the scheduling overhead of multithread\n            // tokio runtime, which can be 50% of the on CPU time of the runtimes\n            let rt = Builder::new_current_thread().enable_all().build().unwrap();\n            let handler = rt.handle().clone();\n            let (tx, rx) = channel::<()>();\n            std::thread::Builder::new()\n                .name(\"Offload thread\".to_string())\n                .spawn(move || {\n                    debug!(\"Offload thread started\");\n                    // the thread that calls block_on() will drive the runtime\n                    // rx will return when tx is dropped so this runtime and thread will exit\n                    rt.block_on(rx)\n                })\n                .unwrap();\n            pools.push((handler, tx));\n        }\n\n        pools.into_boxed_slice()\n    }\n\n    pub fn get_runtime(&self, hash: u64) -> &Handle {\n        let mut rng = rand::thread_rng();\n\n        // choose a shard based on hash and a random thread with in that shard\n        // e.g. say thread_per_shard=2, shard 1 thread 1 is 1 * 2 + 1 = 3\n        // [[th0, th1], [th2, th3], ...]\n        let shard = hash as usize % self.shards;\n        let thread_in_shard = rng.gen_range(0..self.thread_per_shard);\n        let pools = self.pools.get_or_init(|| self.init_pools());\n        &pools[shard * self.thread_per_shard + thread_in_shard].0\n    }\n}\n"
  },
  {
    "path": "pingora-core/src/connectors/tls/boringssl_openssl/mod.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\nuse log::debug;\nuse pingora_error::{Error, ErrorType::*, OrErr, Result};\nuse std::sync::{Arc, Once};\n\nuse crate::connectors::tls::replace_leftmost_underscore;\nuse crate::connectors::ConnectorOptions;\nuse crate::protocols::tls::client::handshake;\nuse crate::protocols::tls::SslStream;\nuse crate::protocols::IO;\nuse crate::tls::ext::{\n    add_host, clear_error_stack, ssl_add_chain_cert, ssl_set_groups_list,\n    ssl_set_renegotiate_mode_freely, ssl_set_verify_cert_store, ssl_use_certificate,\n    ssl_use_private_key, ssl_use_second_key_share,\n};\n#[cfg(feature = \"boringssl\")]\nuse crate::tls::ssl::SslCurve;\nuse crate::tls::ssl::{SslConnector, SslFiletype, SslMethod, SslVerifyMode, SslVersion};\nuse crate::tls::x509::store::X509StoreBuilder;\nuse crate::upstreams::peer::{Peer, ALPN};\n\npub type TlsConnector = SslConnector;\n\nconst CIPHER_LIST: &str = \"AES-128-GCM-SHA256\\\n    :AES-256-GCM-SHA384\\\n    :CHACHA20-POLY1305-SHA256\\\n    :ECDHE-ECDSA-AES128-GCM-SHA256\\\n    :ECDHE-ECDSA-AES256-GCM-SHA384\\\n    :ECDHE-RSA-AES128-GCM-SHA256\\\n    :ECDHE-RSA-AES256-GCM-SHA384\\\n    :ECDHE-RSA-AES128-SHA\\\n    :ECDHE-RSA-AES256-SHA384\\\n    :AES128-GCM-SHA256\\\n    :AES256-GCM-SHA384\\\n    :AES128-SHA\\\n    :AES256-SHA\\\n    :DES-CBC3-SHA\";\n\n/**\n * Enabled signature algorithms for signing/verification (ECDSA).\n * As of 4/10/2023, the only addition to boringssl's defaults is ECDSA_SECP521R1_SHA512.\n */\nconst SIGALG_LIST: &str = \"ECDSA_SECP256R1_SHA256\\\n    :RSA_PSS_RSAE_SHA256\\\n    :RSA_PKCS1_SHA256\\\n    :ECDSA_SECP384R1_SHA384\\\n    :RSA_PSS_RSAE_SHA384\\\n    :RSA_PKCS1_SHA384\\\n    :RSA_PSS_RSAE_SHA512\\\n    :RSA_PKCS1_SHA512\\\n    :RSA_PKCS1_SHA1\\\n    :ECDSA_SECP521R1_SHA512\";\n/**\n * Enabled curves for ECDHE (signature key exchange).\n * As of 4/10/2023, the only addition to boringssl's defaults is SECP521R1.\n *\n * N.B. The ordering of these curves is important. The boringssl library will select the first one\n * as a guess when negotiating a handshake with a server using TLSv1.3. We should opt for curves\n * that are both computationally cheaper and more supported.\n */\n#[cfg(feature = \"boringssl\")]\nconst BORINGSSL_CURVE_LIST: &[SslCurve] = &[\n    SslCurve::X25519,\n    SslCurve::SECP256R1,\n    SslCurve::SECP384R1,\n    SslCurve::SECP521R1,\n];\n\nstatic INIT_CA_ENV: Once = Once::new();\nfn init_ssl_cert_env_vars() {\n    // this sets env vars to pick up the root certs\n    // it is universal across openssl and boringssl\n    // safety: although impossible to prove safe we assume it's safe since the call is\n    // wrapped in a call_once and it's unlikely other threads are reading these vars\n    INIT_CA_ENV.call_once(|| unsafe { openssl_probe::init_openssl_env_vars() });\n}\n\n#[derive(Clone)]\npub struct Connector {\n    pub(crate) ctx: Arc<SslConnector>, // Arc to support clone\n}\n\nimpl Connector {\n    pub fn new(options: Option<ConnectorOptions>) -> Self {\n        let mut builder = SslConnector::builder(SslMethod::tls()).unwrap();\n        // TODO: make these conf\n        // Set supported ciphers.\n        builder.set_cipher_list(CIPHER_LIST).unwrap();\n        // Set supported signature algorithms and ECDH (key exchange) curves.\n        builder\n            .set_sigalgs_list(&SIGALG_LIST.to_lowercase())\n            .unwrap();\n        #[cfg(feature = \"boringssl\")]\n        builder.set_curves(BORINGSSL_CURVE_LIST).unwrap();\n        builder\n            .set_max_proto_version(Some(SslVersion::TLS1_3))\n            .unwrap();\n        builder\n            .set_min_proto_version(Some(SslVersion::TLS1))\n            .unwrap();\n        if let Some(conf) = options.as_ref() {\n            if let Some(ca_file_path) = conf.ca_file.as_ref() {\n                builder.set_ca_file(ca_file_path).unwrap();\n            } else {\n                init_ssl_cert_env_vars();\n                // load from default system wide trust location. (the name is misleading)\n                builder.set_default_verify_paths().unwrap();\n            }\n            if let Some((cert, key)) = conf.cert_key_file.as_ref() {\n                builder.set_certificate_chain_file(cert).unwrap();\n\n                builder.set_private_key_file(key, SslFiletype::PEM).unwrap();\n            }\n            if conf.debug_ssl_keylog {\n                // write TLS keys to file specified by SSLKEYLOGFILE if it exists\n                if let Some(keylog) = std::env::var_os(\"SSLKEYLOGFILE\").and_then(|path| {\n                    std::fs::OpenOptions::new()\n                        .append(true)\n                        .create(true)\n                        .open(path)\n                        .ok()\n                }) {\n                    use std::io::Write;\n                    builder.set_keylog_callback(move |_, line| {\n                        let _ = writeln!(&keylog, \"{}\", line);\n                    });\n                }\n            }\n        } else {\n            init_ssl_cert_env_vars();\n            builder.set_default_verify_paths().unwrap();\n        }\n\n        Connector {\n            ctx: Arc::new(builder.build()),\n        }\n    }\n}\n\npub(crate) async fn connect<T, P>(\n    stream: T,\n    peer: &P,\n    alpn_override: Option<ALPN>,\n    tls_ctx: &SslConnector,\n) -> Result<SslStream<T>>\nwhere\n    T: IO,\n    P: Peer + Send + Sync,\n{\n    let mut ssl_conf = tls_ctx.configure().unwrap();\n\n    ssl_set_renegotiate_mode_freely(&mut ssl_conf);\n\n    // Set up CA/verify cert store\n    // TODO: store X509Store in the peer directly\n    if let Some(ca_list) = peer.get_ca() {\n        let mut store_builder = X509StoreBuilder::new().unwrap();\n        for ca in &***ca_list {\n            store_builder.add_cert(ca.clone()).unwrap();\n        }\n        ssl_set_verify_cert_store(&mut ssl_conf, &store_builder.build())\n            .or_err(InternalError, \"failed to load cert store\")?;\n    }\n\n    // Set up client cert/key\n    if let Some(key_pair) = peer.get_client_cert_key() {\n        debug!(\"setting client cert and key\");\n        ssl_use_certificate(&mut ssl_conf, key_pair.leaf())\n            .or_err(InternalError, \"invalid client cert\")?;\n        ssl_use_private_key(&mut ssl_conf, key_pair.key())\n            .or_err(InternalError, \"invalid client key\")?;\n\n        let intermediates = key_pair.intermediates();\n        if !intermediates.is_empty() {\n            debug!(\"adding intermediate certificates for mTLS chain\");\n            for int in intermediates {\n                ssl_add_chain_cert(&mut ssl_conf, int)\n                    .or_err(InternalError, \"invalid intermediate client cert\")?;\n            }\n        }\n    }\n\n    if let Some(curve) = peer.get_peer_options().and_then(|o| o.curves) {\n        ssl_set_groups_list(&mut ssl_conf, curve).or_err(InternalError, \"invalid curves\")?;\n    }\n\n    // second_keyshare is default true\n    if !peer.get_peer_options().is_none_or(|o| o.second_keyshare) {\n        ssl_use_second_key_share(&mut ssl_conf, false);\n    }\n\n    // disable verification if sni does not exist\n    // XXX: verify on empty string cause null string seg fault\n    if peer.sni().is_empty() {\n        ssl_conf.set_use_server_name_indication(false);\n        /* NOTE: technically we can still verify who signs the cert but turn it off to be\n        consistent with nginx's behavior */\n        ssl_conf.set_verify(SslVerifyMode::NONE);\n    } else if peer.verify_cert() {\n        if peer.verify_hostname() {\n            let verify_param = ssl_conf.param_mut();\n            add_host(verify_param, peer.sni()).or_err(InternalError, \"failed to add host\")?;\n            // if sni had underscores in leftmost label replace and add\n            if let Some(sni_s) = replace_leftmost_underscore(peer.sni()) {\n                add_host(verify_param, sni_s.as_ref()).unwrap();\n            }\n            if let Some(alt_cn) = peer.alternative_cn() {\n                if !alt_cn.is_empty() {\n                    add_host(verify_param, alt_cn).unwrap();\n                    // if alt_cn had underscores in leftmost label replace and add\n                    if let Some(alt_cn_s) = replace_leftmost_underscore(alt_cn) {\n                        add_host(verify_param, alt_cn_s.as_ref()).unwrap();\n                    }\n                }\n            }\n        }\n        ssl_conf.set_verify(SslVerifyMode::PEER);\n    } else {\n        ssl_conf.set_verify(SslVerifyMode::NONE);\n    }\n\n    /*\n       We always set set_verify_hostname(false) here because:\n        - verify case.)  otherwise ssl.connect calls X509_VERIFY_PARAM_set1_host\n                         which overrides the names added by add_host. Verify is\n                         essentially on as long as the names are added.\n        - off case.)    the non verify hostname case should have it disabled\n    */\n    ssl_conf.set_verify_hostname(false);\n\n    if let Some(alpn) = alpn_override.as_ref().or(peer.get_alpn()) {\n        ssl_conf.set_alpn_protos(alpn.to_wire_preference()).unwrap();\n    }\n\n    clear_error_stack();\n\n    let complete_hook = peer\n        .get_peer_options()\n        .and_then(|o| o.upstream_tls_handshake_complete_hook.clone());\n    let connect_future = handshake(ssl_conf, peer.sni(), stream, complete_hook);\n\n    match peer.connection_timeout() {\n        Some(t) => match pingora_timeout::timeout(t, connect_future).await {\n            Ok(res) => res,\n            Err(_) => Error::e_explain(\n                ConnectTimedout,\n                format!(\"connecting to server {}, timeout {:?}\", peer, t),\n            ),\n        },\n        None => connect_future.await,\n    }\n}\n"
  },
  {
    "path": "pingora-core/src/connectors/tls/mod.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n#[cfg(feature = \"openssl_derived\")]\nmod boringssl_openssl;\n\n#[cfg(feature = \"openssl_derived\")]\npub use boringssl_openssl::*;\n\n#[cfg(feature = \"s2n\")]\nmod s2n;\n\n#[cfg(feature = \"s2n\")]\npub use s2n::*;\n\n#[cfg(feature = \"rustls\")]\nmod rustls;\n\n#[cfg(feature = \"rustls\")]\npub use rustls::*;\n\n///    OpenSSL considers underscores in hostnames non-compliant.\n///    We replace the underscore in the leftmost label as we must support these\n///    hostnames for wildcard matches and we have not patched OpenSSL.\n///\n///    https://github.com/openssl/openssl/issues/12566\n///\n///    > The labels must follow the rules for ARPANET host names. They must\n///    > start with a letter, end with a letter or digit, and have as interior\n///    > characters only letters, digits, and hyphen.  There are also some\n///    > restrictions on the length.  Labels must be 63 characters or less.\n///    - https://datatracker.ietf.org/doc/html/rfc1034#section-3.5\n#[cfg(feature = \"any_tls\")]\npub fn replace_leftmost_underscore(sni: &str) -> Option<String> {\n    // wildcard is only leftmost label\n    if let Some((leftmost, rest)) = sni.split_once('.') {\n        // if not a subdomain or leftmost does not contain underscore return\n        if !rest.contains('.') || !leftmost.contains('_') {\n            return None;\n        }\n        // we have a subdomain, replace underscores\n        let leftmost = leftmost.replace('_', \"-\");\n        return Some(format!(\"{leftmost}.{rest}\"));\n    }\n    None\n}\n\n#[cfg(feature = \"any_tls\")]\n#[cfg(test)]\nmod tests {\n    use super::*;\n\n    #[test]\n    fn test_replace_leftmost_underscore() {\n        let none_cases = [\n            \"\",\n            \"some\",\n            \"some.com\",\n            \"1.1.1.1:5050\",\n            \"dog.dot.com\",\n            \"dog.d_t.com\",\n            \"dog.dot.c_m\",\n            \"d_g.com\",\n            \"_\",\n            \"dog.c_m\",\n        ];\n\n        for case in none_cases {\n            assert!(replace_leftmost_underscore(case).is_none(), \"{}\", case);\n        }\n\n        assert_eq!(\n            Some(\"bb-b.some.com\".to_string()),\n            replace_leftmost_underscore(\"bb_b.some.com\")\n        );\n        assert_eq!(\n            Some(\"a-a-a.some.com\".to_string()),\n            replace_leftmost_underscore(\"a_a_a.some.com\")\n        );\n        assert_eq!(\n            Some(\"-.some.com\".to_string()),\n            replace_leftmost_underscore(\"_.some.com\")\n        );\n    }\n}\n"
  },
  {
    "path": "pingora-core/src/connectors/tls/rustls/mod.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\nuse std::sync::Arc;\n\nuse log::debug;\nuse pingora_error::{\n    Error,\n    ErrorType::{ConnectTimedout, InvalidCert},\n    OrErr, Result,\n};\nuse pingora_rustls::{\n    load_ca_file_into_store, load_certs_and_key_files, load_platform_certs_incl_env_into_store,\n    version, CertificateDer, CertificateError, ClientConfig as RusTlsClientConfig,\n    DigitallySignedStruct, KeyLogFile, PrivateKeyDer, RootCertStore, RusTlsError, ServerName,\n    SignatureScheme, TlsConnector as RusTlsConnector, UnixTime, WebPkiServerVerifier,\n};\n\n// Uses custom certificate verification from rustls's 'danger' module.\nuse pingora_rustls::{\n    HandshakeSignatureValid, ServerCertVerified, ServerCertVerifier as RusTlsServerCertVerifier,\n};\n\nuse crate::protocols::tls::{client::handshake, TlsStream};\nuse crate::{connectors::ConnectorOptions, listeners::ALPN, protocols::IO, upstreams::peer::Peer};\n\nuse super::replace_leftmost_underscore;\n\n#[derive(Clone)]\npub struct Connector {\n    pub ctx: Arc<TlsConnector>,\n}\n\nimpl Connector {\n    /// Create a new connector based on the optional configurations. If no\n    /// configurations are provided, no customized certificates or keys will be\n    /// used\n    pub fn new(config_opt: Option<ConnectorOptions>) -> Self {\n        TlsConnector::build_connector(config_opt).unwrap()\n    }\n}\n\npub struct TlsConnector {\n    config: Arc<RusTlsClientConfig>,\n    ca_certs: Arc<RootCertStore>,\n}\n\nimpl TlsConnector {\n    pub(crate) fn build_connector(options: Option<ConnectorOptions>) -> Result<Connector>\n    where\n        Self: Sized,\n    {\n        // NOTE: Rustls only supports TLS 1.2 & 1.3\n\n        // TODO: currently using Rustls defaults\n        // - support SSLKEYLOGFILE\n        // - set supported ciphers/algorithms/curves\n        // - add options for CRL/OCSP validation\n\n        let (ca_certs, certs_key) = {\n            let mut ca_certs = RootCertStore::empty();\n            let mut certs_key = None;\n\n            if let Some(conf) = options.as_ref() {\n                if let Some(ca_file_path) = conf.ca_file.as_ref() {\n                    load_ca_file_into_store(ca_file_path, &mut ca_certs)?;\n                } else {\n                    load_platform_certs_incl_env_into_store(&mut ca_certs)?;\n                }\n                if let Some((cert, key)) = conf.cert_key_file.as_ref() {\n                    certs_key = load_certs_and_key_files(cert, key)?;\n                }\n            } else {\n                load_platform_certs_incl_env_into_store(&mut ca_certs)?;\n            }\n\n            (ca_certs, certs_key)\n        };\n\n        // TODO: WebPkiServerVerifier for CRL/OCSP validation\n        let builder =\n            RusTlsClientConfig::builder_with_protocol_versions(&[&version::TLS12, &version::TLS13])\n                .with_root_certificates(ca_certs.clone());\n\n        let mut config = match certs_key {\n            Some((certs, key)) => {\n                match builder.with_client_auth_cert(certs.clone(), key.clone_key()) {\n                    Ok(config) => config,\n                    Err(err) => {\n                        // TODO: is there a viable alternative to the panic?\n                        // falling back to no client auth... does not seem to be reasonable.\n                        panic!(\"Failed to configure client auth cert/key. Error: {}\", err);\n                    }\n                }\n            }\n            None => builder.with_no_client_auth(),\n        };\n\n        // Enable SSLKEYLOGFILE support for debugging TLS traffic\n        if let Some(options) = options.as_ref() {\n            if options.debug_ssl_keylog {\n                config.key_log = Arc::new(KeyLogFile::new());\n            }\n        }\n\n        Ok(Connector {\n            ctx: Arc::new(TlsConnector {\n                config: Arc::new(config),\n                ca_certs: Arc::new(ca_certs),\n            }),\n        })\n    }\n}\n\npub async fn connect<T, P>(\n    stream: T,\n    peer: &P,\n    alpn_override: Option<ALPN>,\n    tls_ctx: &TlsConnector,\n) -> Result<TlsStream<T>>\nwhere\n    T: IO,\n    P: Peer + Send + Sync,\n{\n    let config = &tls_ctx.config;\n\n    // TODO: setup CA/verify cert store from peer\n    // peer.get_ca() returns None by default. It must be replaced by the\n    // implementation of `peer`\n    let key_pair = peer.get_client_cert_key();\n    let mut updated_config_opt: Option<RusTlsClientConfig> = match key_pair {\n        None => None,\n        Some(key_arc) => {\n            debug!(\"setting client cert and key\");\n\n            let mut cert_chain = vec![];\n            debug!(\"adding leaf certificate to mTLS cert chain\");\n            cert_chain.push(key_arc.leaf());\n\n            debug!(\"adding intermediate certificates to mTLS cert chain\");\n            key_arc\n                .intermediates()\n                .to_owned()\n                .iter()\n                .copied()\n                .for_each(|i| cert_chain.push(i));\n\n            let certs: Vec<CertificateDer> = cert_chain.into_iter().map(|c| c.into()).collect();\n            let private_key: PrivateKeyDer =\n                key_arc.key().as_slice().to_owned().try_into().unwrap();\n\n            let builder = RusTlsClientConfig::builder_with_protocol_versions(&[\n                &version::TLS12,\n                &version::TLS13,\n            ])\n            .with_root_certificates(Arc::clone(&tls_ctx.ca_certs));\n            debug!(\"added root ca certificates\");\n\n            let mut updated_config = builder.with_client_auth_cert(certs, private_key).or_err(\n                InvalidCert,\n                \"Failed to use peer cert/key to update Rustls config\",\n            )?;\n            // Preserve keylog setting from original config\n            updated_config.key_log = Arc::clone(&config.key_log);\n            Some(updated_config)\n        }\n    };\n\n    if let Some(alpn) = alpn_override.as_ref().or(peer.get_alpn()) {\n        let alpn_protocols = alpn.to_wire_protocols();\n        if let Some(updated_config) = updated_config_opt.as_mut() {\n            updated_config.alpn_protocols = alpn_protocols;\n        } else {\n            let mut updated_config = RusTlsClientConfig::clone(config);\n            updated_config.alpn_protocols = alpn_protocols;\n            updated_config_opt = Some(updated_config);\n        }\n    }\n\n    let mut domain = peer.sni().to_string();\n\n    if let Some(updated_config) = updated_config_opt.as_mut() {\n        let verification_mode = if peer.sni().is_empty() {\n            updated_config.enable_sni = false;\n            /* NOTE: technically we can still verify who signs the cert but turn it off to be\n            consistent with nginx's behavior */\n            Some(VerificationMode::SkipAll) // disable verification if sni does not exist\n        } else if !peer.verify_cert() {\n            Some(VerificationMode::SkipAll)\n        } else if !peer.verify_hostname() {\n            Some(VerificationMode::SkipHostname)\n        } else {\n            // if sni had underscores in leftmost label replace and add\n            if let Some(sni_s) = replace_leftmost_underscore(peer.sni()) {\n                domain = sni_s;\n            }\n            None\n            // to use the custom verifier for the full verify:\n            // Some(VerificationMode::Full)\n        };\n\n        // Builds the custom_verifier when verification_mode is set.\n        if let Some(mode) = verification_mode {\n            let delegate = WebPkiServerVerifier::builder(Arc::clone(&tls_ctx.ca_certs))\n                .build()\n                .or_err(InvalidCert, \"Failed to build WebPkiServerVerifier\")?;\n\n            let custom_verifier = Arc::new(CustomServerCertVerifier::new(delegate, mode));\n\n            updated_config\n                .dangerous()\n                .set_certificate_verifier(custom_verifier);\n        }\n    }\n\n    // TODO: curve setup from peer\n    // - second key share from peer, currently only used in boringssl with PQ features\n\n    // Patch config for dangerous verifier if needed, but only in test builds.\n    #[cfg(test)]\n    if !peer.verify_cert() || !peer.verify_hostname() {\n        use crate::connectors::http::rustls_no_verify::apply_no_verify;\n        if let Some(cfg) = updated_config_opt.as_mut() {\n            apply_no_verify(cfg);\n        } else {\n            let mut tmp = RusTlsClientConfig::clone(config);\n            apply_no_verify(&mut tmp);\n            updated_config_opt = Some(tmp);\n        }\n    }\n\n    let tls_conn = if let Some(cfg) = updated_config_opt {\n        RusTlsConnector::from(Arc::new(cfg))\n    } else {\n        RusTlsConnector::from(Arc::clone(config))\n    };\n\n    let connect_future = handshake(&tls_conn, &domain, stream);\n\n    match peer.connection_timeout() {\n        Some(t) => match pingora_timeout::timeout(t, connect_future).await {\n            Ok(res) => res,\n            Err(_) => Error::e_explain(\n                ConnectTimedout,\n                format!(\"connecting to server {}, timeout {:?}\", peer, t),\n            ),\n        },\n        None => connect_future.await,\n    }\n}\n\n#[allow(dead_code)]\n#[derive(Debug)]\npub enum VerificationMode {\n    SkipHostname,\n    SkipAll,\n    Full,\n    // Note: \"Full\" Included for completeness, making this verifier self-contained\n    // and explicit about all possible verification modes, not just exceptions.\n}\n\n#[derive(Debug)]\npub struct CustomServerCertVerifier {\n    delegate: Arc<WebPkiServerVerifier>,\n    verification_mode: VerificationMode,\n}\n\nimpl CustomServerCertVerifier {\n    pub fn new(delegate: Arc<WebPkiServerVerifier>, verification_mode: VerificationMode) -> Self {\n        Self {\n            delegate,\n            verification_mode,\n        }\n    }\n}\n\n// CustomServerCertVerifier delegates TLS signature verification and allows 3 VerificationMode:\n// Full: delegates all verification to the original WebPkiServerVerifier\n// SkipHostname: same as \"Full\" but ignores \"NotValidForName\" certificate errors\n// SkipAll: all certificate verification checks are skipped.\nimpl RusTlsServerCertVerifier for CustomServerCertVerifier {\n    fn verify_server_cert(\n        &self,\n        _end_entity: &CertificateDer<'_>,\n        _intermediates: &[CertificateDer<'_>],\n        _server_name: &ServerName<'_>,\n        _ocsp: &[u8],\n        _now: UnixTime,\n    ) -> Result<ServerCertVerified, RusTlsError> {\n        match self.verification_mode {\n            VerificationMode::Full => self.delegate.verify_server_cert(\n                _end_entity,\n                _intermediates,\n                _server_name,\n                _ocsp,\n                _now,\n            ),\n            VerificationMode::SkipHostname => {\n                match self.delegate.verify_server_cert(\n                    _end_entity,\n                    _intermediates,\n                    _server_name,\n                    _ocsp,\n                    _now,\n                ) {\n                    Ok(scv) => Ok(scv),\n                    Err(RusTlsError::InvalidCertificate(cert_error)) => {\n                        if let CertificateError::NotValidForNameContext { .. } = cert_error {\n                            Ok(ServerCertVerified::assertion())\n                        } else {\n                            Err(RusTlsError::InvalidCertificate(cert_error))\n                        }\n                    }\n                    Err(e) => Err(e),\n                }\n            }\n            VerificationMode::SkipAll => Ok(ServerCertVerified::assertion()),\n        }\n    }\n\n    fn verify_tls12_signature(\n        &self,\n        message: &[u8],\n        cert: &CertificateDer<'_>,\n        dss: &DigitallySignedStruct,\n    ) -> Result<HandshakeSignatureValid, RusTlsError> {\n        self.delegate.verify_tls12_signature(message, cert, dss)\n    }\n\n    fn verify_tls13_signature(\n        &self,\n        message: &[u8],\n        cert: &CertificateDer<'_>,\n        dss: &DigitallySignedStruct,\n    ) -> Result<HandshakeSignatureValid, RusTlsError> {\n        self.delegate.verify_tls13_signature(message, cert, dss)\n    }\n\n    fn supported_verify_schemes(&self) -> Vec<SignatureScheme> {\n        self.delegate.supported_verify_schemes()\n    }\n}\n"
  },
  {
    "path": "pingora-core/src/connectors/tls/s2n/mod.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\nuse std::hash::{Hash, Hasher};\nuse std::num::NonZero;\nuse std::sync::{Arc, Mutex};\n\nuse ahash::AHasher;\nuse lru::LruCache;\nuse pingora_error::{Error, Result};\nuse pingora_error::{ErrorType::*, OrErr};\n\nuse pingora_s2n::{\n    load_pem_file, ClientAuthType, Config, IgnoreVerifyHostnameCallback,\n    TlsConnector as S2NTlsConnector, DEFAULT_TLS13,\n};\n\nuse crate::utils::tls::{CertKey, X509Pem};\nuse crate::{\n    connectors::ConnectorOptions,\n    listeners::ALPN,\n    protocols::{\n        tls::{client::handshake, S2NConnectionBuilder, TlsStream},\n        IO,\n    },\n    upstreams::peer::Peer,\n};\n\nconst DEFAULT_CONFIG_CACHE_SIZE: NonZero<usize> = NonZero::new(10).unwrap();\n\n#[derive(Clone)]\npub struct Connector {\n    pub ctx: TlsConnector,\n}\n\nimpl Connector {\n    /// Create a new connector based on the optional configurations. If no\n    /// configurations are provided, no customized certificates or keys will be\n    /// used\n    pub fn new(options: Option<ConnectorOptions>) -> Self {\n        Connector {\n            ctx: TlsConnector::new(options),\n        }\n    }\n}\n\n/// Holds default options for configuring a TLS connection and an LRU cache for `s2n_config`.\n///\n/// In `s2n-tls`, each connection requires an associated `s2n_config`, which is expensive to create.\n/// Although `s2n_config` objects can be cheaply cloned, they are immutable once built.\n///\n/// To avoid the overhead of constructing a new config for every connection, we maintain a cache\n/// that stores previously built configs. Configs are retrieved from the cache based on the\n/// configuration options used to create them.\n#[derive(Clone)]\npub struct TlsConnector {\n    config_cache: Option<Arc<Mutex<LruCache<u64, Config>>>>,\n    options: Option<ConnectorOptions>,\n}\n\nimpl TlsConnector {\n    pub fn new(options: Option<ConnectorOptions>) -> Self {\n        TlsConnector {\n            config_cache: Self::create_config_cache(&options),\n            options,\n        }\n    }\n\n    /// Provided with a set of config options, either creates a new s2n config or\n    /// fetches one from the LRU Cache.\n    fn load_config(&self, config_options: S2NConfigOptions) -> Result<Config> {\n        if self.config_cache.is_some() {\n            let config_hash = config_options.config_hash();\n            if let Some(config) = self.load_config_from_cache(config_hash) {\n                return Ok(config);\n            } else {\n                let config = create_s2n_config(&self.options, config_options)?;\n                self.put_config_in_cache(config_hash, config.clone());\n                return Ok(config);\n            }\n        } else {\n            create_s2n_config(&self.options, config_options)\n        }\n    }\n\n    fn load_config_from_cache(&self, config_hash: u64) -> Option<Config> {\n        if let Some(config_cache) = &self.config_cache {\n            let mut cache = config_cache.lock().unwrap();\n            cache.get(&config_hash).cloned()\n        } else {\n            None\n        }\n    }\n\n    fn put_config_in_cache(&self, config_hash: u64, config: Config) {\n        if let Some(config_cache) = &self.config_cache {\n            let mut cache = config_cache.lock().unwrap();\n            cache.put(config_hash, config);\n        }\n    }\n\n    fn create_config_cache(\n        options: &Option<ConnectorOptions>,\n    ) -> Option<Arc<Mutex<LruCache<u64, Config>>>> {\n        let mut cache_size = DEFAULT_CONFIG_CACHE_SIZE;\n        if let Some(opts) = options {\n            if let Some(cache_size_config) = opts.s2n_config_cache_size {\n                if cache_size_config <= 0 {\n                    return None;\n                } else {\n                    cache_size = NonZero::new(cache_size_config).unwrap();\n                }\n            }\n        }\n        return Some(Arc::new(Mutex::new(LruCache::new(cache_size))));\n    }\n}\n\npub(crate) async fn connect<T, P>(\n    stream: T,\n    peer: &P,\n    alpn_override: Option<ALPN>,\n    tls_ctx: &TlsConnector,\n) -> Result<TlsStream<T>>\nwhere\n    T: IO,\n    P: Peer + Send + Sync,\n{\n    // Default security policy with TLS 1.3 support\n    // https://aws.github.io/s2n-tls/usage-guide/ch06-security-policies.html\n    let security_policy = peer.get_s2n_security_policy().unwrap_or(&DEFAULT_TLS13);\n\n    let config_options = S2NConfigOptions::from_peer(peer, alpn_override);\n    let config = tls_ctx.load_config(config_options)?;\n\n    let connection_builder = S2NConnectionBuilder {\n        config: config,\n        psk_config: peer.get_psk().cloned(),\n        security_policy: Some(security_policy.clone()),\n    };\n\n    let domain = peer\n        .alternative_cn()\n        .map(|s| s.as_str())\n        .unwrap_or(peer.sni());\n    let connector = S2NTlsConnector::new(connection_builder);\n\n    let connect_future = handshake(&connector, domain, stream);\n\n    match peer.connection_timeout() {\n        Some(t) => match pingora_timeout::timeout(t, connect_future).await {\n            Ok(res) => res,\n            Err(_) => Error::e_explain(\n                ConnectTimedout,\n                format!(\"connecting to server {}, timeout {:?}\", peer, t),\n            ),\n        },\n        None => connect_future.await,\n    }\n}\n\nfn create_s2n_config(\n    connector_options: &Option<ConnectorOptions>,\n    config_options: S2NConfigOptions,\n) -> Result<Config> {\n    let mut builder = Config::builder();\n\n    if let Some(conf) = connector_options.as_ref() {\n        if let Some(ca_file_path) = conf.ca_file.as_ref() {\n            let ca_pem = load_pem_file(&ca_file_path)?;\n            builder\n                .trust_pem(&ca_pem)\n                .or_err(InternalError, \"failed to load ca cert\")?;\n        }\n\n        if let Some((cert_file, key_file)) = conf.cert_key_file.as_ref() {\n            let cert = load_pem_file(cert_file)?;\n            let key = load_pem_file(key_file)?;\n            builder\n                .load_pem(&cert, &key)\n                .or_err(InternalError, \"failed to load client cert\")?;\n            builder\n                .set_client_auth_type(ClientAuthType::Required)\n                .or_err(InternalError, \"failed to load client key\")?;\n        }\n    }\n\n    if let Some(max_blinding_delay) = config_options.max_blinding_delay {\n        builder\n            .set_max_blinding_delay(max_blinding_delay)\n            .or_err(InternalError, \"failed to set max blinding delay\")?;\n    }\n\n    if let Some(ca) = config_options.ca {\n        builder\n            .trust_pem(&ca.raw_pem)\n            .or_err(InternalError, \"invalid peer ca cert\")?;\n    }\n\n    if let Some(client_cert_key) = config_options.client_cert_key {\n        builder\n            .load_pem(&client_cert_key.raw_pem(), &client_cert_key.key())\n            .or_err(InternalError, \"invalid peer client cert or key\")?;\n    }\n\n    if let Some(alpn) = config_options.alpn {\n        builder\n            .set_application_protocol_preference(alpn.to_wire_protocols())\n            .or_err(InternalError, \"failed to set peer alpn\")?;\n    }\n\n    if !config_options.verify_cert {\n        // Disabling x509 verification is considered unsafe\n        unsafe {\n            builder\n                .disable_x509_verification()\n                .or_err(InternalError, \"failed to disable certificate verification\")?;\n        }\n    }\n\n    if !config_options.verify_hostname {\n        // Set verify hostname callback that always returns success\n        builder\n            .set_verify_host_callback(IgnoreVerifyHostnameCallback::new())\n            .or_err(InternalError, \"failed to disable hostname verification\")?;\n    }\n\n    if !config_options.use_system_certs {\n        builder.with_system_certs(false).or_err(\n            InternalError,\n            \"failed to disable system certificate loading\",\n        )?;\n    }\n\n    Ok(builder\n        .build()\n        .or_err(InternalError, \"failed to build s2n config\")?)\n}\n\n#[derive(Clone)]\nstruct S2NConfigOptions {\n    max_blinding_delay: Option<u32>,\n    alpn: Option<ALPN>,\n    verify_cert: bool,\n    verify_hostname: bool,\n    use_system_certs: bool,\n    ca: Option<Arc<X509Pem>>,\n    client_cert_key: Option<Arc<CertKey>>,\n}\n\nimpl S2NConfigOptions {\n    fn from_peer<P>(peer: &P, alpn_override: Option<ALPN>) -> Self\n    where\n        P: Peer + Send + Sync,\n    {\n        S2NConfigOptions {\n            max_blinding_delay: peer.get_max_blinding_delay(),\n            alpn: alpn_override.or(peer.get_alpn().cloned()),\n            verify_cert: peer.verify_cert(),\n            verify_hostname: peer.verify_hostname(),\n            use_system_certs: peer.use_system_certs(),\n            ca: peer.get_ca().cloned(),\n            client_cert_key: peer.get_client_cert_key().cloned(),\n        }\n    }\n\n    fn config_hash(&self) -> u64 {\n        let mut hasher = AHasher::default();\n        self.hash(&mut hasher);\n        hasher.finish()\n    }\n}\n\nimpl Hash for S2NConfigOptions {\n    fn hash<H: Hasher>(&self, state: &mut H) {\n        self.max_blinding_delay.hash(state);\n        self.alpn.hash(state);\n        self.verify_cert.hash(state);\n        self.verify_hostname.hash(state);\n        self.use_system_certs.hash(state);\n        self.ca.hash(state);\n        self.client_cert_key.hash(state);\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use std::{fs, sync::Arc};\n\n    use crate::{\n        connectors::tls::{s2n::S2NConfigOptions, TlsConnector},\n        listeners::ALPN,\n        utils::tls::{CertKey, X509Pem},\n    };\n\n    const CA_CERT_FILE: &str = \"tests/certs/ca.crt\";\n    const ALT_CA_CERT_FILE: &str = \"tests/certs/alt-ca.crt\";\n\n    const CERT_FILE: &str = \"tests/certs/server.crt\";\n    const ALT_CERT_FILE: &str = \"tests/certs/alt-server.crt\";\n\n    const KEY_FILE: &str = \"tests/certs/server.key\";\n\n    fn read_file(file: &str) -> Vec<u8> {\n        fs::read(file).unwrap()\n    }\n\n    fn load_pem_from_file(file: &str) -> X509Pem {\n        X509Pem::new(read_file(file))\n    }\n\n    fn create_config_options() -> S2NConfigOptions {\n        S2NConfigOptions {\n            max_blinding_delay: Some(10),\n            alpn: Some(ALPN::H1),\n            verify_cert: true,\n            verify_hostname: true,\n            use_system_certs: true,\n            ca: Some(Arc::new(load_pem_from_file(CA_CERT_FILE))),\n            client_cert_key: Some(Arc::new(CertKey::new(\n                read_file(CERT_FILE),\n                read_file(KEY_FILE),\n            ))),\n        }\n    }\n\n    #[test]\n    fn config_cache_hit_identical() {\n        let connector = TlsConnector::new(None);\n        let config_options = create_config_options();\n\n        let config = connector.load_config(config_options.clone()).unwrap();\n        let cached_config = connector.load_config_from_cache(config_options.config_hash());\n\n        assert!(cached_config.is_some());\n        assert_eq!(config, cached_config.unwrap());\n    }\n\n    #[test]\n    fn config_cache_miss_max_blinding_delay_changed() {\n        let connector = TlsConnector::new(None);\n        let mut config_options = create_config_options();\n\n        let _config = connector.load_config(config_options.clone()).unwrap();\n        config_options.max_blinding_delay = Some(20);\n        let cached_config = connector.load_config_from_cache(config_options.config_hash());\n\n        assert!(cached_config.is_none());\n    }\n\n    #[test]\n    fn config_cache_miss_alpn_changed() {\n        let connector = TlsConnector::new(None);\n        let mut config_options = create_config_options();\n\n        let _config = connector.load_config(config_options.clone()).unwrap();\n        config_options.alpn = Some(ALPN::H2H1);\n        let cached_config = connector.load_config_from_cache(config_options.config_hash());\n\n        assert!(cached_config.is_none());\n    }\n\n    #[test]\n    fn config_cache_miss_verify_cert_changed() {\n        let connector = TlsConnector::new(None);\n        let mut config_options = create_config_options();\n\n        let _config = connector.load_config(config_options.clone()).unwrap();\n        config_options.verify_cert = false;\n        let cached_config = connector.load_config_from_cache(config_options.config_hash());\n\n        assert!(cached_config.is_none());\n    }\n\n    #[test]\n    fn config_cache_miss_verify_hostname_changed() {\n        let connector = TlsConnector::new(None);\n        let mut config_options = create_config_options();\n\n        let _config = connector.load_config(config_options.clone()).unwrap();\n        config_options.verify_hostname = false;\n        let cached_config = connector.load_config_from_cache(config_options.config_hash());\n\n        assert!(cached_config.is_none());\n    }\n\n    #[test]\n    fn config_cache_miss_use_system_certs_changed() {\n        let connector = TlsConnector::new(None);\n        let mut config_options = create_config_options();\n\n        let _config = connector.load_config(config_options.clone()).unwrap();\n        config_options.use_system_certs = false;\n        let cached_config = connector.load_config_from_cache(config_options.config_hash());\n\n        assert!(cached_config.is_none());\n    }\n\n    #[test]\n    fn config_cache_miss_ca_changed() {\n        let connector = TlsConnector::new(None);\n        let mut config_options = create_config_options();\n\n        let _config = connector.load_config(config_options.clone()).unwrap();\n        config_options.ca = Some(Arc::new(load_pem_from_file(ALT_CA_CERT_FILE)));\n        let cached_config = connector.load_config_from_cache(config_options.config_hash());\n\n        assert!(cached_config.is_none());\n    }\n\n    #[test]\n    fn config_cache_miss_client_cert_key_changed() {\n        let connector = TlsConnector::new(None);\n        let mut config_options = create_config_options();\n\n        let _config = connector.load_config(config_options.clone()).unwrap();\n        config_options.client_cert_key = Some(Arc::new(CertKey::new(\n            read_file(ALT_CERT_FILE),\n            read_file(KEY_FILE),\n        )));\n        let cached_config = connector.load_config_from_cache(config_options.config_hash());\n\n        assert!(cached_config.is_none());\n    }\n}\n"
  },
  {
    "path": "pingora-core/src/lib.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n#![warn(clippy::all)]\n#![allow(clippy::new_without_default)]\n#![allow(clippy::type_complexity)]\n#![allow(clippy::match_wild_err_arm)]\n#![allow(clippy::missing_safety_doc)]\n#![allow(clippy::upper_case_acronyms)]\n\n//! # Pingora\n//!\n//! Pingora is a collection of service frameworks and network libraries battle-tested by the Internet.\n//! It is to build robust, scalable and secure network infrastructures and services at Internet scale.\n//!\n//! # Features\n//! - Http 1.x and Http 2\n//! - Modern TLS with OpenSSL or BoringSSL (FIPS compatible)\n//! - Zero downtime upgrade\n//!\n//! # Usage\n//! This crate provides low level service and protocol implementation and abstraction.\n//!\n//! If looking to build a (reverse) proxy, see [`pingora-proxy`](https://docs.rs/pingora-proxy) crate.\n//!\n//! # Optional features\n//!\n//! ## TLS backends (mutually exclusive)\n//! - `openssl`: Use OpenSSL as the TLS library (default if no TLS feature is specified)\n//! - `boringssl`: Use BoringSSL as the TLS library (FIPS compatible)\n//! - `rustls`: Use Rustls as the TLS library\n//!\n//! ## Additional features\n//! - `connection_filter`: Enable early TCP connection filtering before TLS handshake.\n//!   This allows implementing custom logic to accept/reject connections based on peer address\n//!   with zero overhead when disabled.\n//! - `sentry`: Enable Sentry error reporting integration\n//! - `patched_http1`: Enable patched HTTP/1 parser\n//!\n//! # Connection Filtering\n//!\n//! With the `connection_filter` feature enabled, you can implement early connection filtering\n//! at the TCP level, before any TLS handshake or HTTP processing occurs. This is useful for:\n//! - IP-based access control\n//! - Rate limiting at the connection level\n//! - Geographic restrictions\n//! - DDoS mitigation\n//!\n//! ## Example\n//!\n//! ```rust,ignore\n//! # #[cfg(feature = \"connection_filter\")]\n//! # {\n//! use async_trait::async_trait;\n//! use pingora_core::listeners::ConnectionFilter;\n//! use std::net::SocketAddr;\n//! use std::sync::Arc;\n//!\n//! #[derive(Debug)]\n//! struct MyFilter;\n//!\n//! #[async_trait]\n//! impl ConnectionFilter for MyFilter {\n//!     async fn should_accept(&self, addr: &SocketAddr) -> bool {\n//!         // Custom logic to filter connections\n//!         !is_blocked_ip(addr.ip())\n//!     }\n//! }\n//!\n//! // Apply the filter to a service\n//! let mut service = my_service();\n//! service.set_connection_filter(Arc::new(MyFilter));\n//! # }\n//! ```\n//!\n//! When the `connection_filter` feature is disabled, the filter API remains available\n//! but becomes a no-op, ensuring zero overhead for users who don't need this functionality.\n\n// This enables the feature that labels modules that are only available with\n// certain pingora features\n#![cfg_attr(docsrs, feature(doc_cfg))]\n\npub mod apps;\npub mod connectors;\npub mod listeners;\npub mod modules;\npub mod protocols;\npub mod server;\npub mod services;\npub mod upstreams;\npub mod utils;\n\npub use pingora_error::{ErrorType::*, *};\n\n// If both openssl and boringssl are enabled, prefer boringssl.\n// This is to make sure that boringssl can override the default openssl feature\n// when this crate is used indirectly by other crates.\n#[cfg(feature = \"boringssl\")]\npub use pingora_boringssl as tls;\n\n#[cfg(feature = \"openssl\")]\npub use pingora_openssl as tls;\n\n#[cfg(feature = \"rustls\")]\npub use pingora_rustls as tls;\n\n#[cfg(feature = \"s2n\")]\npub use pingora_s2n as tls;\n\n#[cfg(not(feature = \"any_tls\"))]\npub use protocols::tls::noop_tls as tls;\n\npub mod prelude {\n    pub use crate::server::configuration::Opt;\n    pub use crate::server::Server;\n    pub use crate::services::background::background_service;\n    pub use crate::upstreams::peer::HttpPeer;\n    pub use pingora_error::{ErrorType::*, *};\n}\n"
  },
  {
    "path": "pingora-core/src/listeners/connection_filter.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! Connection filtering trait for early connection filtering\n//!\n//! This module provides the [`ConnectionFilter`] trait which allows filtering\n//! incoming connections at the TCP level, before the TLS handshake occurs.\n//!\n//! # Feature Flag\n//!\n//! This functionality requires the `connection_filter` feature to be enabled:\n//! ```toml\n//! [dependencies]\n//! pingora-core = { version = \"0.5\", features = [\"connection_filter\"] }\n//! ```\n//!\n//! When the feature is disabled, a no-op implementation is provided for API compatibility.\n\nuse async_trait::async_trait;\nuse std::fmt::Debug;\nuse std::net::SocketAddr;\n\n/// A trait for filtering incoming connections at the TCP level.\n///\n/// Implementations of this trait can inspect the peer address of incoming\n/// connections and decide whether to accept or reject them before any\n/// further processing (including TLS handshake) occurs.\n///\n/// # Example\n///\n/// ```rust,no_run\n/// use async_trait::async_trait;\n/// use pingora_core::listeners::ConnectionFilter;\n/// use std::net::{IpAddr, Ipv4Addr, SocketAddr};\n///\n/// #[derive(Debug)]\n/// struct BlocklistFilter {\n///     blocked_ips: Vec<IpAddr>,\n/// }\n///\n/// #[async_trait]\n/// impl ConnectionFilter for BlocklistFilter {\n///     async fn should_accept(&self, addr: &SocketAddr) -> bool {\n///         !self.blocked_ips.contains(&addr.ip())\n///     }\n/// }\n/// ```\n///\n/// # Performance Considerations\n///\n/// This filter is called for every incoming connection, so implementations\n/// should be efficient. Consider caching or pre-computing data structures\n/// for IP filtering rather than doing expensive operations per connection.\n#[async_trait]\npub trait ConnectionFilter: Debug + Send + Sync {\n    /// Determines whether an incoming connection should be accepted.\n    ///\n    /// This method is called after a TCP connection is accepted but before\n    /// any further processing (including TLS handshake).\n    ///\n    /// # Arguments\n    ///\n    /// * `addr` - The socket address of the incoming connection\n    ///\n    /// # Returns\n    ///\n    /// * `true` - Accept the connection and continue processing\n    /// * `false` - Drop the connection immediately\n    ///\n    /// # Example\n    ///\n    /// ```rust,no_run\n    /// async fn should_accept(&self, addr: &SocketAddr) -> bool {\n    ///     // Accept only connections from private IP ranges\n    ///     match addr.ip() {\n    ///         IpAddr::V4(ip) => ip.is_private(),\n    ///         IpAddr::V6(_) => true,\n    ///     }\n    /// }\n    ///\n    async fn should_accept(&self, _addr: Option<&SocketAddr>) -> bool {\n        true\n    }\n}\n\n/// Default implementation that accepts all connections.\n///\n/// This filter accepts all incoming connections without any filtering.\n/// It's used as the default when no custom filter is specified.\n#[derive(Debug, Clone)]\npub struct AcceptAllFilter;\n\n#[async_trait]\nimpl ConnectionFilter for AcceptAllFilter {\n    // Uses default implementation\n}\n\n#[cfg(test)]\nmod tests {\n    use super::*;\n    use std::net::{IpAddr, Ipv4Addr};\n\n    #[derive(Debug, Clone)]\n    struct BlockListFilter {\n        blocked_ips: Vec<IpAddr>,\n    }\n\n    #[async_trait]\n    impl ConnectionFilter for BlockListFilter {\n        async fn should_accept(&self, addr_opt: Option<&SocketAddr>) -> bool {\n            addr_opt\n                .map(|addr| !self.blocked_ips.contains(&addr.ip()))\n                .unwrap_or(true)\n        }\n    }\n\n    #[tokio::test]\n    async fn test_accept_all_filter() {\n        let filter = AcceptAllFilter;\n        let addr = SocketAddr::new(IpAddr::V4(Ipv4Addr::new(127, 0, 0, 1)), 8080);\n        assert!(filter.should_accept(Some(&addr)).await);\n    }\n\n    #[tokio::test]\n    async fn test_blocklist_filter() {\n        let filter = BlockListFilter {\n            blocked_ips: vec![IpAddr::V4(Ipv4Addr::new(192, 168, 1, 1))],\n        };\n\n        let blocked_addr = SocketAddr::new(IpAddr::V4(Ipv4Addr::new(192, 168, 1, 1)), 8080);\n        let allowed_addr = SocketAddr::new(IpAddr::V4(Ipv4Addr::new(192, 168, 1, 2)), 8080);\n\n        assert!(!filter.should_accept(Some(&blocked_addr)).await);\n        assert!(filter.should_accept(Some(&allowed_addr)).await);\n    }\n}\n"
  },
  {
    "path": "pingora-core/src/listeners/l4.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n#[cfg(feature = \"connection_filter\")]\nuse log::debug;\nuse log::warn;\nuse pingora_error::{\n    ErrorType::{AcceptError, BindError},\n    OrErr, Result,\n};\nuse std::io::ErrorKind;\nuse std::net::{SocketAddr, ToSocketAddrs};\n#[cfg(unix)]\nuse std::os::unix::io::{AsRawFd, FromRawFd};\n#[cfg(unix)]\nuse std::os::unix::net::UnixListener as StdUnixListener;\n#[cfg(windows)]\nuse std::os::windows::io::{AsRawSocket, FromRawSocket};\nuse std::time::Duration;\nuse std::{fs::Permissions, sync::Arc};\nuse tokio::net::TcpSocket;\n\n#[cfg(feature = \"connection_filter\")]\nuse super::connection_filter::ConnectionFilter;\n#[cfg(feature = \"connection_filter\")]\nuse crate::listeners::AcceptAllFilter;\n\nuse crate::protocols::l4::ext::{set_dscp, set_recv_buf, set_snd_buf, set_tcp_fastopen_backlog};\nuse crate::protocols::l4::listener::Listener;\npub use crate::protocols::l4::stream::Stream;\n#[cfg(feature = \"connection_filter\")]\nuse crate::protocols::GetSocketDigest;\nuse crate::protocols::TcpKeepalive;\n#[cfg(unix)]\nuse crate::server::ListenFds;\n\nconst TCP_LISTENER_MAX_TRY: usize = 30;\nconst TCP_LISTENER_TRY_STEP: Duration = Duration::from_secs(1);\n// TODO: configurable backlog\nconst LISTENER_BACKLOG: u32 = 65535;\n\n/// Address for listening server, either TCP/UDS socket.\n#[derive(Clone, Debug)]\npub enum ServerAddress {\n    Tcp(String, Option<TcpSocketOptions>),\n    #[cfg(unix)]\n    Uds(String, Option<Permissions>),\n}\n\nimpl AsRef<str> for ServerAddress {\n    fn as_ref(&self) -> &str {\n        match &self {\n            Self::Tcp(l, _) => l,\n            #[cfg(unix)]\n            Self::Uds(l, _) => l,\n        }\n    }\n}\n\nimpl ServerAddress {\n    fn tcp_sock_opts(&self) -> Option<&TcpSocketOptions> {\n        match &self {\n            Self::Tcp(_, op) => op.into(),\n            _ => None,\n        }\n    }\n}\n\n/// TCP socket configuration options, this is used for setting options on\n/// listening sockets and accepted connections.\n#[non_exhaustive]\n#[derive(Clone, Debug, Default)]\npub struct TcpSocketOptions {\n    /// IPV6_V6ONLY flag (if true, limit socket to IPv6 communication only).\n    /// This is mostly useful when binding to `[::]`, which on most Unix distributions\n    /// will bind to both IPv4 and IPv6 addresses by default.\n    pub ipv6_only: Option<bool>,\n    /// Enable TCP fast open and set the backlog size of it.\n    /// See the [man page](https://man7.org/linux/man-pages/man7/tcp.7.html) for more information.\n    pub tcp_fastopen: Option<usize>,\n    /// Enable TCP keepalive on accepted connections.\n    /// See the [man page](https://man7.org/linux/man-pages/man7/tcp.7.html) for more information.\n    pub tcp_keepalive: Option<TcpKeepalive>,\n    /// Specifies the server should set the following DSCP value on outgoing connections.\n    /// See the [RFC](https://datatracker.ietf.org/doc/html/rfc2474) for more details.\n    pub dscp: Option<u8>,\n    /// Enable SO_REUSEPORT to allow multiple sockets to bind to the same address and port.\n    /// This is useful for load balancing across multiple worker processes.\n    /// See the [man page](https://man7.org/linux/man-pages/man7/socket.7.html) for more information.\n    pub so_reuseport: Option<bool>,\n    /// Set the send buffer size for accepted connections. See\n    /// [SO_SNDBUF](https://man7.org/linux/man-pages/man7/socket.7.html).\n    pub tcp_snd_buf: Option<usize>,\n    /// Set the receive buffer size for accepted connections. See\n    /// [SO_RCVBUF](https://man7.org/linux/man-pages/man7/socket.7.html).\n    pub tcp_recv_buf: Option<usize>,\n    // TODO: allow configuring reuseaddr, backlog, etc. from here?\n}\n\n#[cfg(unix)]\nmod uds {\n    use super::{OrErr, Result};\n    use crate::protocols::l4::listener::Listener;\n    use log::{debug, error};\n    use pingora_error::ErrorType::BindError;\n    use std::fs::{self, Permissions};\n    use std::io::ErrorKind;\n    use std::os::unix::fs::PermissionsExt;\n    use std::os::unix::net::UnixListener as StdUnixListener;\n    use tokio::net::UnixListener;\n\n    use super::LISTENER_BACKLOG;\n\n    pub(super) fn set_perms(path: &str, perms: Option<Permissions>) -> Result<()> {\n        // set read/write permissions for all users on the socket by default\n        let perms = perms.unwrap_or(Permissions::from_mode(0o666));\n        fs::set_permissions(path, perms).or_err_with(BindError, || {\n            format!(\"Fail to bind to {path}, could not set permissions\")\n        })\n    }\n\n    pub(super) fn set_backlog(l: StdUnixListener, backlog: u32) -> Result<UnixListener> {\n        let socket: socket2::Socket = l.into();\n        // Note that we call listen on an already listening socket\n        // POSIX undefined but on Linux it will update the backlog size\n        socket\n            .listen(backlog as i32)\n            .or_err_with(BindError, || format!(\"listen() failed on {socket:?}\"))?;\n        UnixListener::from_std(socket.into()).or_err(BindError, \"Failed to convert to tokio socket\")\n    }\n\n    pub(super) fn bind(addr: &str, perms: Option<Permissions>) -> Result<Listener> {\n        /*\n          We remove the filename/address in case there is a dangling reference.\n\n          \"Binding to a socket with a filename creates a socket in the\n          filesystem that must be deleted by the caller when it is no\n          longer needed (using unlink(2))\"\n        */\n        match std::fs::remove_file(addr) {\n            Ok(()) => {\n                debug!(\"unlink {addr} done\");\n            }\n            Err(e) => match e.kind() {\n                ErrorKind::NotFound => debug!(\"unlink {addr} not found: {e}\"),\n                _ => error!(\"unlink {addr} failed: {e}\"),\n            },\n        }\n        let listener_socket = UnixListener::bind(addr)\n            .or_err_with(BindError, || format!(\"Bind() failed on {addr}\"))?;\n        set_perms(addr, perms)?;\n        let std_listener = listener_socket.into_std().unwrap();\n        Ok(set_backlog(std_listener, LISTENER_BACKLOG)?.into())\n    }\n}\n\n// currently, these options can only apply on sockets prior to calling bind()\nfn apply_tcp_socket_options(sock: &TcpSocket, opt: Option<&TcpSocketOptions>) -> Result<()> {\n    let Some(opt) = opt else {\n        return Ok(());\n    };\n\n    let socket_ref = socket2::SockRef::from(sock);\n\n    if let Some(ipv6_only) = opt.ipv6_only {\n        socket_ref\n            .set_only_v6(ipv6_only)\n            .or_err(BindError, \"failed to set IPV6_V6ONLY\")?;\n    }\n\n    #[cfg(unix)]\n    if let Some(reuseport) = opt.so_reuseport {\n        socket_ref\n            .set_reuse_port(reuseport)\n            .or_err(BindError, \"failed to set SO_REUSEPORT\")?;\n    }\n\n    #[cfg(unix)]\n    let raw = sock.as_raw_fd();\n    #[cfg(windows)]\n    let raw = sock.as_raw_socket();\n\n    if let Some(backlog) = opt.tcp_fastopen {\n        set_tcp_fastopen_backlog(raw, backlog)?;\n    }\n\n    if let Some(dscp) = opt.dscp {\n        set_dscp(raw, dscp)?;\n    }\n    Ok(())\n}\n\nfn from_raw_fd(address: &ServerAddress, fd: i32) -> Result<Listener> {\n    match address {\n        #[cfg(unix)]\n        ServerAddress::Uds(addr, perm) => {\n            let std_listener = unsafe { StdUnixListener::from_raw_fd(fd) };\n            // set permissions just in case\n            uds::set_perms(addr, perm.clone())?;\n            Ok(uds::set_backlog(std_listener, LISTENER_BACKLOG)?.into())\n        }\n        ServerAddress::Tcp(_, _) => {\n            #[cfg(unix)]\n            let std_listener_socket = unsafe { std::net::TcpStream::from_raw_fd(fd) };\n            #[cfg(windows)]\n            let std_listener_socket = unsafe { std::net::TcpStream::from_raw_socket(fd as u64) };\n            let listener_socket = TcpSocket::from_std_stream(std_listener_socket);\n            // Note that we call listen on an already listening socket\n            // POSIX undefined but on Linux it will update the backlog size\n            Ok(listener_socket\n                .listen(LISTENER_BACKLOG)\n                .or_err_with(BindError, || format!(\"Listen() failed on {address:?}\"))?\n                .into())\n        }\n    }\n}\n\nasync fn bind_tcp(addr: &str, opt: Option<TcpSocketOptions>) -> Result<Listener> {\n    let mut try_count = 0;\n    loop {\n        let sock_addr = addr\n            .to_socket_addrs() // NOTE: this could invoke a blocking network lookup\n            .or_err_with(BindError, || format!(\"Invalid listen address {addr}\"))?\n            .next() // take the first one for now\n            .unwrap(); // assume there is always at least one\n\n        let listener_socket = match sock_addr {\n            SocketAddr::V4(_) => TcpSocket::new_v4(),\n            SocketAddr::V6(_) => TcpSocket::new_v6(),\n        }\n        .or_err_with(BindError, || format!(\"fail to create address {sock_addr}\"))?;\n\n        // NOTE: this is to preserve the current TcpListener::bind() behavior.\n        // We have a few tests relying on this behavior to allow multiple identical\n        // test servers to coexist.\n        listener_socket\n            .set_reuseaddr(true)\n            .or_err(BindError, \"fail to set_reuseaddr(true)\")?;\n\n        apply_tcp_socket_options(&listener_socket, opt.as_ref())?;\n\n        match listener_socket.bind(sock_addr) {\n            Ok(()) => {\n                break Ok(listener_socket\n                    .listen(LISTENER_BACKLOG)\n                    .or_err(BindError, \"bind() failed\")?\n                    .into())\n            }\n            Err(e) => {\n                if e.kind() != ErrorKind::AddrInUse {\n                    break Err(e).or_err_with(BindError, || format!(\"bind() failed on {addr}\"));\n                }\n                try_count += 1;\n                if try_count >= TCP_LISTENER_MAX_TRY {\n                    break Err(e).or_err_with(BindError, || {\n                        format!(\"bind() failed, after retries, {addr} still in use\")\n                    });\n                }\n                warn!(\"{addr} is in use, will try again\");\n                tokio::time::sleep(TCP_LISTENER_TRY_STEP).await;\n            }\n        }\n    }\n}\n\nasync fn bind(addr: &ServerAddress) -> Result<Listener> {\n    match addr {\n        #[cfg(unix)]\n        ServerAddress::Uds(l, perm) => uds::bind(l, perm.clone()),\n        ServerAddress::Tcp(l, opt) => bind_tcp(l, opt.clone()).await,\n    }\n}\n\n#[derive(Clone, Debug)]\npub struct ListenerEndpoint {\n    listen_addr: ServerAddress,\n    listener: Arc<Listener>,\n    #[cfg(feature = \"connection_filter\")]\n    connection_filter: Arc<dyn ConnectionFilter>,\n}\n\n#[derive(Default)]\npub struct ListenerEndpointBuilder {\n    listen_addr: Option<ServerAddress>,\n    #[cfg(feature = \"connection_filter\")]\n    connection_filter: Option<Arc<dyn ConnectionFilter>>,\n}\n\nimpl ListenerEndpointBuilder {\n    pub fn new() -> ListenerEndpointBuilder {\n        Self {\n            listen_addr: None,\n            #[cfg(feature = \"connection_filter\")]\n            connection_filter: None,\n        }\n    }\n\n    pub fn listen_addr(&mut self, addr: ServerAddress) -> &mut Self {\n        self.listen_addr = Some(addr);\n        self\n    }\n\n    #[cfg(feature = \"connection_filter\")]\n    pub fn connection_filter(&mut self, filter: Arc<dyn ConnectionFilter>) -> &mut Self {\n        self.connection_filter = Some(filter);\n        self\n    }\n\n    #[cfg(unix)]\n    pub async fn listen(self, fds: Option<ListenFds>) -> Result<ListenerEndpoint> {\n        let listen_addr = self\n            .listen_addr\n            .expect(\"Tried to listen with no addr specified\");\n\n        let listener = if let Some(fds_table) = fds {\n            let addr_str = listen_addr.as_ref();\n\n            // consider make this mutex std::sync::Mutex or OnceCell\n            let mut table = fds_table.lock().await;\n\n            if let Some(fd) = table.get(addr_str) {\n                from_raw_fd(&listen_addr, *fd)?\n            } else {\n                // not found\n                let listener = bind(&listen_addr).await?;\n                table.add(addr_str.to_string(), listener.as_raw_fd());\n                listener\n            }\n        } else {\n            // not found, no fd table\n            bind(&listen_addr).await?\n        };\n\n        #[cfg(feature = \"connection_filter\")]\n        let connection_filter = self\n            .connection_filter\n            .unwrap_or_else(|| Arc::new(AcceptAllFilter));\n\n        Ok(ListenerEndpoint {\n            listen_addr,\n            listener: Arc::new(listener),\n            #[cfg(feature = \"connection_filter\")]\n            connection_filter,\n        })\n    }\n\n    #[cfg(windows)]\n    pub async fn listen(self) -> Result<ListenerEndpoint> {\n        let listen_addr = self\n            .listen_addr\n            .expect(\"Tried to listen with no addr specified\");\n\n        let listener = bind(&listen_addr).await?;\n\n        #[cfg(feature = \"connection_filter\")]\n        let connection_filter = self\n            .connection_filter\n            .unwrap_or_else(|| Arc::new(AcceptAllFilter));\n\n        Ok(ListenerEndpoint {\n            listen_addr,\n            listener: Arc::new(listener),\n            #[cfg(feature = \"connection_filter\")]\n            connection_filter,\n        })\n    }\n}\n\nimpl ListenerEndpoint {\n    pub fn builder() -> ListenerEndpointBuilder {\n        ListenerEndpointBuilder::new()\n    }\n\n    pub fn as_str(&self) -> &str {\n        self.listen_addr.as_ref()\n    }\n\n    fn apply_stream_settings(&self, stream: &mut Stream) -> Result<()> {\n        // settings are applied based on whether the underlying stream supports it\n        stream.set_nodelay()?;\n        let Some(op) = self.listen_addr.tcp_sock_opts() else {\n            return Ok(());\n        };\n        if let Some(ka) = op.tcp_keepalive.as_ref() {\n            stream.set_keepalive(ka)?;\n        }\n        if let Some(dscp) = op.dscp {\n            #[cfg(unix)]\n            set_dscp(stream.as_raw_fd(), dscp)?;\n            #[cfg(windows)]\n            set_dscp(stream.as_raw_socket(), dscp)?;\n        }\n        if let Some(snd_buf) = op.tcp_snd_buf {\n            #[cfg(unix)]\n            set_snd_buf(stream.as_raw_fd(), snd_buf)?;\n            #[cfg(windows)]\n            set_snd_buf(stream.as_raw_socket(), snd_buf)?;\n        }\n        if let Some(recv_buf) = op.tcp_recv_buf {\n            #[cfg(unix)]\n            set_recv_buf(stream.as_raw_fd(), recv_buf)?;\n            #[cfg(windows)]\n            set_recv_buf(stream.as_raw_socket(), recv_buf)?;\n        }\n        Ok(())\n    }\n\n    pub async fn accept(&self) -> Result<Stream> {\n        #[cfg(feature = \"connection_filter\")]\n        {\n            loop {\n                let mut stream = self\n                    .listener\n                    .accept()\n                    .await\n                    .or_err(AcceptError, \"Fail to accept()\")?;\n\n                // Performance: nested if-let avoids cloning/allocations on each connection accept\n                let should_accept = if let Some(digest) = stream.get_socket_digest() {\n                    if let Some(peer_addr) = digest.peer_addr() {\n                        self.connection_filter\n                            .should_accept(peer_addr.as_inet())\n                            .await\n                    } else {\n                        // No peer address available - accept by default\n                        true\n                    }\n                } else {\n                    // No socket digest available - accept by default\n                    true\n                };\n\n                if !should_accept {\n                    debug!(\"Connection rejected by filter\");\n                    drop(stream);\n                    continue;\n                }\n\n                self.apply_stream_settings(&mut stream)?;\n                return Ok(stream);\n            }\n        }\n        #[cfg(not(feature = \"connection_filter\"))]\n        {\n            let mut stream = self\n                .listener\n                .accept()\n                .await\n                .or_err(AcceptError, \"Fail to accept()\")?;\n            self.apply_stream_settings(&mut stream)?;\n            Ok(stream)\n        }\n    }\n}\n\n#[cfg(test)]\nmod test {\n    use super::*;\n\n    #[tokio::test]\n    async fn test_listen_tcp() {\n        let addr = \"127.0.0.1:7100\";\n\n        let mut builder = ListenerEndpoint::builder();\n\n        builder.listen_addr(ServerAddress::Tcp(addr.into(), None));\n\n        #[cfg(unix)]\n        let listener = builder.listen(None).await.unwrap();\n\n        #[cfg(windows)]\n        let listener = builder.listen().await.unwrap();\n\n        tokio::spawn(async move {\n            // just try to accept once\n            listener.accept().await.unwrap();\n        });\n        tokio::net::TcpStream::connect(addr)\n            .await\n            .expect(\"can connect to TCP listener\");\n    }\n\n    #[tokio::test]\n    async fn test_listen_tcp_ipv6_only() {\n        let sock_opt = Some(TcpSocketOptions {\n            ipv6_only: Some(true),\n            ..Default::default()\n        });\n\n        let mut builder = ListenerEndpoint::builder();\n\n        builder.listen_addr(ServerAddress::Tcp(\"[::]:7101\".into(), sock_opt));\n\n        #[cfg(unix)]\n        let listener = builder.listen(None).await.unwrap();\n\n        #[cfg(windows)]\n        let listener = builder.listen().await.unwrap();\n\n        tokio::spawn(async move {\n            // just try to accept twice\n            listener.accept().await.unwrap();\n            listener.accept().await.unwrap();\n        });\n        tokio::net::TcpStream::connect(\"127.0.0.1:7101\")\n            .await\n            .expect_err(\"cannot connect to v4 addr\");\n        tokio::net::TcpStream::connect(\"[::1]:7101\")\n            .await\n            .expect(\"can connect to v6 addr\");\n    }\n\n    #[cfg(unix)]\n    #[tokio::test]\n    async fn test_listen_uds() {\n        let addr = \"/tmp/test_listen_uds\";\n\n        let mut builder = ListenerEndpoint::builder();\n\n        builder.listen_addr(ServerAddress::Uds(addr.into(), None));\n\n        let listener = builder.listen(None).await.unwrap();\n\n        tokio::spawn(async move {\n            // just try to accept once\n            listener.accept().await.unwrap();\n        });\n        tokio::net::UnixStream::connect(addr)\n            .await\n            .expect(\"can connect to UDS listener\");\n    }\n\n    #[cfg(unix)]\n    #[tokio::test]\n    async fn test_tcp_so_reuseport() {\n        let addr = \"127.0.0.1:7201\";\n        let sock_opt = TcpSocketOptions {\n            so_reuseport: Some(true),\n            ..Default::default()\n        };\n\n        // Create first listener with SO_REUSEPORT\n        let mut builder1 = ListenerEndpoint::builder();\n        builder1.listen_addr(ServerAddress::Tcp(addr.into(), Some(sock_opt.clone())));\n        let listener1 = builder1.listen(None).await.unwrap();\n\n        // Create second listener with the same address and SO_REUSEPORT\n        // This should succeed because SO_REUSEPORT is enabled\n        let mut builder2 = ListenerEndpoint::builder();\n        builder2.listen_addr(ServerAddress::Tcp(addr.into(), Some(sock_opt)));\n        let listener2 = builder2.listen(None).await.unwrap();\n\n        // Both listeners should be able to bind to the same address\n        assert_eq!(listener1.as_str(), addr);\n        assert_eq!(listener2.as_str(), addr);\n    }\n\n    #[tokio::test]\n    async fn test_tcp_so_reuseport_false() {\n        let addr = \"127.0.0.1:7202\";\n        let sock_opt_no_reuseport = TcpSocketOptions {\n            so_reuseport: Some(false), // Explicitly disable SO_REUSEPORT\n            ..Default::default()\n        };\n\n        // Create first listener without SO_REUSEPORT\n        let mut builder1 = ListenerEndpoint::builder();\n        builder1.listen_addr(ServerAddress::Tcp(\n            addr.into(),\n            Some(sock_opt_no_reuseport.clone()),\n        ));\n        let listener1 = builder1.listen(None).await.unwrap();\n\n        // Try to create second listener with the same address and no SO_REUSEPORT\n        // This should fail with \"address already in use\"\n        let mut builder2 = ListenerEndpoint::builder();\n        builder2.listen_addr(ServerAddress::Tcp(addr.into(), Some(sock_opt_no_reuseport)));\n        let result = builder2.listen(None).await;\n\n        // The second bind should fail\n        assert!(result.is_err());\n        let error_msg = format!(\"{:?}\", result.unwrap_err());\n        assert!(\n            error_msg.contains(\"address\")\n                || error_msg.contains(\"in use\")\n                || error_msg.contains(\"bind\")\n        );\n\n        // Verify the first listener still works\n        assert_eq!(listener1.as_str(), addr);\n    }\n\n    #[cfg(feature = \"connection_filter\")]\n    #[tokio::test]\n    async fn test_connection_filter_accept() {\n        use crate::listeners::ConnectionFilter;\n        use async_trait::async_trait;\n        use std::sync::atomic::{AtomicUsize, Ordering};\n\n        #[derive(Debug)]\n        struct CountingFilter {\n            accept_count: Arc<AtomicUsize>,\n            reject_count: Arc<AtomicUsize>,\n        }\n\n        #[async_trait]\n        impl ConnectionFilter for CountingFilter {\n            async fn should_accept(&self, _addr: Option<&SocketAddr>) -> bool {\n                let count = self.accept_count.fetch_add(1, Ordering::SeqCst);\n                if count % 2 == 0 {\n                    true\n                } else {\n                    self.reject_count.fetch_add(1, Ordering::SeqCst);\n                    false\n                }\n            }\n        }\n\n        let addr = \"127.0.0.1:7300\";\n        let accept_count = Arc::new(AtomicUsize::new(0));\n        let reject_count = Arc::new(AtomicUsize::new(0));\n\n        let filter = Arc::new(CountingFilter {\n            accept_count: accept_count.clone(),\n            reject_count: reject_count.clone(),\n        });\n\n        let mut builder = ListenerEndpoint::builder();\n        builder\n            .listen_addr(ServerAddress::Tcp(addr.into(), None))\n            .connection_filter(filter);\n\n        #[cfg(unix)]\n        let listener = builder.listen(None).await.unwrap();\n        #[cfg(windows)]\n        let listener = builder.listen().await.unwrap();\n\n        let listener_clone = listener.clone();\n        tokio::spawn(async move {\n            let _stream1 = listener_clone.accept().await.unwrap();\n            let _stream2 = listener_clone.accept().await.unwrap();\n        });\n\n        tokio::time::sleep(Duration::from_millis(10)).await;\n\n        let _conn1 = tokio::net::TcpStream::connect(addr).await.unwrap();\n        let _conn2 = tokio::net::TcpStream::connect(addr).await.unwrap();\n        let _conn3 = tokio::net::TcpStream::connect(addr).await.unwrap();\n\n        tokio::time::sleep(Duration::from_millis(50)).await;\n\n        assert_eq!(accept_count.load(Ordering::SeqCst), 3);\n        assert_eq!(reject_count.load(Ordering::SeqCst), 1);\n    }\n\n    #[cfg(feature = \"connection_filter\")]\n    #[tokio::test]\n    async fn test_connection_filter_blocks_all() {\n        use crate::listeners::ConnectionFilter;\n        use async_trait::async_trait;\n        use std::sync::atomic::{AtomicUsize, Ordering};\n\n        #[derive(Debug)]\n        struct RejectAllFilter {\n            reject_count: Arc<AtomicUsize>,\n        }\n\n        #[async_trait]\n        impl ConnectionFilter for RejectAllFilter {\n            async fn should_accept(&self, _addr: Option<&SocketAddr>) -> bool {\n                self.reject_count.fetch_add(1, Ordering::SeqCst);\n                false\n            }\n        }\n\n        let addr = \"127.0.0.1:7301\";\n        let reject_count = Arc::new(AtomicUsize::new(0));\n\n        let mut builder = ListenerEndpoint::builder();\n        builder\n            .listen_addr(ServerAddress::Tcp(addr.into(), None))\n            .connection_filter(Arc::new(RejectAllFilter {\n                reject_count: reject_count.clone(),\n            }));\n\n        #[cfg(unix)]\n        let listener = builder.listen(None).await.unwrap();\n        #[cfg(windows)]\n        let listener = builder.listen().await.unwrap();\n\n        let listener_clone = listener.clone();\n        let _accept_handle = tokio::spawn(async move {\n            // This will never return since all connections are rejected\n            let _ = listener_clone.accept().await;\n        });\n\n        tokio::time::sleep(Duration::from_millis(50)).await;\n\n        let mut handles = vec![];\n        for _ in 0..3 {\n            let handle = tokio::spawn(async move {\n                if let Ok(stream) = tokio::net::TcpStream::connect(addr).await {\n                    drop(stream);\n                }\n            });\n            handles.push(handle);\n        }\n\n        for handle in handles {\n            let _ = handle.await;\n        }\n\n        // Wait for rejections to be counted with timeout\n        let start = tokio::time::Instant::now();\n        let timeout = Duration::from_secs(2);\n\n        loop {\n            let rejected = reject_count.load(Ordering::SeqCst);\n            if rejected >= 3 {\n                assert_eq!(rejected, 3, \"Should reject exactly 3 connections\");\n                break;\n            }\n\n            if start.elapsed() > timeout {\n                panic!(\n                    \"Timeout waiting for rejections, got {} expected 3\",\n                    rejected\n                );\n            }\n\n            tokio::time::sleep(Duration::from_millis(10)).await;\n        }\n    }\n}\n"
  },
  {
    "path": "pingora-core/src/listeners/mod.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! The listening endpoints (TCP and TLS) and their configurations.\n//!\n//! This module provides the infrastructure for setting up network listeners\n//! that accept incoming connections. It supports TCP, Unix domain sockets,\n//! and TLS endpoints.\n//!\n//! # Connection Filtering\n//!\n//! With the `connection_filter` feature enabled, this module also provides\n//! early connection filtering capabilities through the [`ConnectionFilter`] trait.\n//! This allows dropping unwanted connections at the TCP level before any\n//! expensive operations like TLS handshakes.\n//!\n//! ## Example with Connection Filtering\n//!\n//! ```rust,no_run\n//! # #[cfg(feature = \"connection_filter\")]\n//! # {\n//! use pingora_core::listeners::{Listeners, ConnectionFilter};\n//! use std::sync::Arc;\n//!\n//! // Create a custom filter\n//! let filter = Arc::new(MyCustomFilter::new());\n//!\n//! // Apply to listeners\n//! let mut listeners = Listeners::new();\n//! listeners.set_connection_filter(filter);\n//! listeners.add_tcp(\"0.0.0.0:8080\");\n//! # }\n//! ```\n\nmod l4;\n\n#[cfg(feature = \"connection_filter\")]\npub mod connection_filter;\n\n#[cfg(feature = \"connection_filter\")]\npub use connection_filter::{AcceptAllFilter, ConnectionFilter};\n\n#[cfg(not(feature = \"connection_filter\"))]\n#[derive(Debug, Clone)]\npub struct AcceptAllFilter;\n\n#[cfg(not(feature = \"connection_filter\"))]\npub trait ConnectionFilter: std::fmt::Debug + Send + Sync {\n    fn should_accept(&self, _addr: &std::net::SocketAddr) -> bool {\n        true\n    }\n}\n\n#[cfg(not(feature = \"connection_filter\"))]\nimpl ConnectionFilter for AcceptAllFilter {\n    fn should_accept(&self, _addr: &std::net::SocketAddr) -> bool {\n        true\n    }\n}\n#[cfg(feature = \"any_tls\")]\npub mod tls;\n\n#[cfg(not(feature = \"any_tls\"))]\npub use crate::tls::listeners as tls;\n\nuse crate::protocols::{l4::socket::SocketAddr, tls::TlsRef, Stream};\n\n#[cfg(unix)]\nuse crate::server::ListenFds;\n\nuse async_trait::async_trait;\nuse pingora_error::Result;\nuse std::{any::Any, fs::Permissions, sync::Arc};\n\nuse l4::{ListenerEndpoint, Stream as L4Stream};\nuse tls::{Acceptor, TlsSettings};\n\npub use crate::protocols::tls::ALPN;\nuse crate::protocols::GetSocketDigest;\npub use l4::{ServerAddress, TcpSocketOptions};\n\n/// The APIs to customize things like certificate during TLS server side handshake\n#[async_trait]\npub trait TlsAccept {\n    // TODO: return error?\n    /// This function is called in the middle of a TLS handshake. Structs who\n    /// implement this function should provide tls certificate and key to the\n    /// [TlsRef] via `ssl_use_certificate` and `ssl_use_private_key`.\n    /// Note. This is only supported for openssl and boringssl\n    async fn certificate_callback(&self, _ssl: &mut TlsRef) -> () {\n        // does nothing by default\n    }\n\n    /// This function is called after the TLS handshake is complete.\n    ///\n    /// Any value returned from this function (other than `None`) will be stored in the\n    /// `extension` field of `SslDigest`. This allows you to attach custom application-specific\n    /// data to the TLS connection, which will be accessible from the HTTP layer via the\n    /// `SslDigest` attached to the session digest.\n    async fn handshake_complete_callback(\n        &self,\n        _ssl: &TlsRef,\n    ) -> Option<Arc<dyn Any + Send + Sync>> {\n        None\n    }\n}\n\npub type TlsAcceptCallbacks = Box<dyn TlsAccept + Send + Sync>;\n\nstruct TransportStackBuilder {\n    l4: ServerAddress,\n    tls: Option<TlsSettings>,\n    #[cfg(feature = \"connection_filter\")]\n    connection_filter: Option<Arc<dyn ConnectionFilter>>,\n}\n\nimpl TransportStackBuilder {\n    pub async fn build(\n        &mut self,\n        #[cfg(unix)] upgrade_listeners: Option<ListenFds>,\n    ) -> Result<TransportStack> {\n        let mut builder = ListenerEndpoint::builder();\n\n        builder.listen_addr(self.l4.clone());\n\n        #[cfg(feature = \"connection_filter\")]\n        if let Some(filter) = &self.connection_filter {\n            builder.connection_filter(filter.clone());\n        }\n\n        #[cfg(unix)]\n        let l4 = builder.listen(upgrade_listeners).await?;\n\n        #[cfg(windows)]\n        let l4 = builder.listen().await?;\n\n        Ok(TransportStack {\n            l4,\n            tls: self.tls.take().map(|tls| Arc::new(tls.build())),\n        })\n    }\n}\n\n#[derive(Clone)]\npub(crate) struct TransportStack {\n    l4: ListenerEndpoint,\n    tls: Option<Arc<Acceptor>>,\n}\n\nimpl TransportStack {\n    pub fn as_str(&self) -> &str {\n        self.l4.as_str()\n    }\n\n    pub async fn accept(&self) -> Result<UninitializedStream> {\n        let stream = self.l4.accept().await?;\n        Ok(UninitializedStream {\n            l4: stream,\n            tls: self.tls.clone(),\n        })\n    }\n\n    pub fn cleanup(&mut self) {\n        // placeholder\n    }\n}\n\npub(crate) struct UninitializedStream {\n    l4: L4Stream,\n    tls: Option<Arc<Acceptor>>,\n}\n\nimpl UninitializedStream {\n    pub async fn handshake(mut self) -> Result<Stream> {\n        self.l4.set_buffer();\n        if let Some(tls) = self.tls {\n            let tls_stream = tls.tls_handshake(self.l4).await?;\n            Ok(Box::new(tls_stream))\n        } else {\n            Ok(Box::new(self.l4))\n        }\n    }\n\n    /// Get the peer address of the connection if available\n    pub fn peer_addr(&self) -> Option<SocketAddr> {\n        self.l4\n            .get_socket_digest()\n            .and_then(|d| d.peer_addr().cloned())\n    }\n}\n\n/// The struct to hold one more multiple listening endpoints\npub struct Listeners {\n    stacks: Vec<TransportStackBuilder>,\n    #[cfg(feature = \"connection_filter\")]\n    connection_filter: Option<Arc<dyn ConnectionFilter>>,\n}\n\nimpl Listeners {\n    /// Create a new [`Listeners`] with no listening endpoints.\n    pub fn new() -> Self {\n        Listeners {\n            stacks: vec![],\n            #[cfg(feature = \"connection_filter\")]\n            connection_filter: None,\n        }\n    }\n    /// Create a new [`Listeners`] with a TCP server endpoint from the given string.\n    pub fn tcp(addr: &str) -> Self {\n        let mut listeners = Self::new();\n        listeners.add_tcp(addr);\n        listeners\n    }\n\n    /// Create a new [`Listeners`] with a Unix domain socket endpoint from the given string.\n    #[cfg(unix)]\n    pub fn uds(addr: &str, perm: Option<Permissions>) -> Self {\n        let mut listeners = Self::new();\n        listeners.add_uds(addr, perm);\n        listeners\n    }\n\n    /// Create a new [`Listeners`] with a TLS (TCP) endpoint with the given address string,\n    /// and path to the certificate/private key pairs.\n    /// This endpoint will adopt the [Mozilla Intermediate](https://wiki.mozilla.org/Security/Server_Side_TLS#Intermediate_compatibility_.28recommended.29)\n    /// server side TLS settings.\n    pub fn tls(addr: &str, cert_path: &str, key_path: &str) -> Result<Self> {\n        let mut listeners = Self::new();\n        listeners.add_tls(addr, cert_path, key_path)?;\n        Ok(listeners)\n    }\n\n    /// Add a TCP endpoint to `self`.\n    pub fn add_tcp(&mut self, addr: &str) {\n        self.add_address(ServerAddress::Tcp(addr.into(), None));\n    }\n\n    /// Add a TCP endpoint to `self`, with the given [`TcpSocketOptions`].\n    pub fn add_tcp_with_settings(&mut self, addr: &str, sock_opt: TcpSocketOptions) {\n        self.add_address(ServerAddress::Tcp(addr.into(), Some(sock_opt)));\n    }\n\n    /// Add a Unix domain socket endpoint to `self`.\n    #[cfg(unix)]\n    pub fn add_uds(&mut self, addr: &str, perm: Option<Permissions>) {\n        self.add_address(ServerAddress::Uds(addr.into(), perm));\n    }\n\n    /// Add a TLS endpoint to `self` with the [Mozilla Intermediate](https://wiki.mozilla.org/Security/Server_Side_TLS#Intermediate_compatibility_.28recommended.29)\n    /// server side TLS settings.\n    pub fn add_tls(&mut self, addr: &str, cert_path: &str, key_path: &str) -> Result<()> {\n        self.add_tls_with_settings(addr, None, TlsSettings::intermediate(cert_path, key_path)?);\n        Ok(())\n    }\n\n    /// Add a TLS endpoint to `self` with the given socket and server side TLS settings.\n    /// See [`TlsSettings`] and [`TcpSocketOptions`] for more details.\n    pub fn add_tls_with_settings(\n        &mut self,\n        addr: &str,\n        sock_opt: Option<TcpSocketOptions>,\n        settings: TlsSettings,\n    ) {\n        self.add_endpoint(ServerAddress::Tcp(addr.into(), sock_opt), Some(settings));\n    }\n\n    /// Add the given [`ServerAddress`] to `self`.\n    pub fn add_address(&mut self, addr: ServerAddress) {\n        self.add_endpoint(addr, None);\n    }\n\n    /// Set a connection filter for all endpoints in this listener collection\n    #[cfg(feature = \"connection_filter\")]\n    pub fn set_connection_filter(&mut self, filter: Arc<dyn ConnectionFilter>) {\n        log::debug!(\"Setting connection filter on Listeners\");\n\n        // Store the filter for future endpoints\n        self.connection_filter = Some(filter.clone());\n\n        // Apply to existing stacks\n        for stack in &mut self.stacks {\n            stack.connection_filter = Some(filter.clone());\n        }\n    }\n\n    /// Add the given [`ServerAddress`] to `self` with the given [`TlsSettings`] if provided\n    pub fn add_endpoint(&mut self, l4: ServerAddress, tls: Option<TlsSettings>) {\n        self.stacks.push(TransportStackBuilder {\n            l4,\n            tls,\n            #[cfg(feature = \"connection_filter\")]\n            connection_filter: self.connection_filter.clone(),\n        })\n    }\n\n    pub(crate) async fn build(\n        &mut self,\n        #[cfg(unix)] upgrade_listeners: Option<ListenFds>,\n    ) -> Result<Vec<TransportStack>> {\n        let mut stacks = Vec::with_capacity(self.stacks.len());\n\n        for b in self.stacks.iter_mut() {\n            let new_stack = b\n                .build(\n                    #[cfg(unix)]\n                    upgrade_listeners.clone(),\n                )\n                .await?;\n\n            stacks.push(new_stack);\n        }\n\n        Ok(stacks)\n    }\n\n    pub(crate) fn cleanup(&self) {\n        // placeholder\n    }\n}\n\n#[cfg(test)]\nmod test {\n    use super::*;\n    #[cfg(feature = \"connection_filter\")]\n    use std::sync::atomic::{AtomicUsize, Ordering};\n    #[cfg(feature = \"any_tls\")]\n    use tokio::io::AsyncWriteExt;\n    use tokio::net::TcpStream;\n    use tokio::time::{sleep, Duration};\n\n    #[tokio::test]\n    async fn test_listen_tcp() {\n        let addr1 = \"127.0.0.1:7101\";\n        let addr2 = \"127.0.0.1:7102\";\n        let mut listeners = Listeners::tcp(addr1);\n        listeners.add_tcp(addr2);\n\n        let listeners = listeners\n            .build(\n                #[cfg(unix)]\n                None,\n            )\n            .await\n            .unwrap();\n\n        assert_eq!(listeners.len(), 2);\n        for listener in listeners {\n            tokio::spawn(async move {\n                // just try to accept once\n                let stream = listener.accept().await.unwrap();\n                stream.handshake().await.unwrap();\n            });\n        }\n\n        // make sure the above starts before the lines below\n        sleep(Duration::from_millis(10)).await;\n\n        TcpStream::connect(addr1).await.unwrap();\n        TcpStream::connect(addr2).await.unwrap();\n    }\n\n    #[tokio::test]\n    #[cfg(feature = \"any_tls\")]\n    async fn test_listen_tls() {\n        use tokio::io::AsyncReadExt;\n\n        let addr = \"127.0.0.1:7103\";\n        let cert_path = format!(\"{}/tests/keys/server.crt\", env!(\"CARGO_MANIFEST_DIR\"));\n        let key_path = format!(\"{}/tests/keys/key.pem\", env!(\"CARGO_MANIFEST_DIR\"));\n        let mut listeners = Listeners::tls(addr, &cert_path, &key_path).unwrap();\n        let listener = listeners\n            .build(\n                #[cfg(unix)]\n                None,\n            )\n            .await\n            .unwrap()\n            .pop()\n            .unwrap();\n\n        tokio::spawn(async move {\n            // just try to accept once\n            let stream = listener.accept().await.unwrap();\n            let mut stream = stream.handshake().await.unwrap();\n            let mut buf = [0; 1024];\n            let _ = stream.read(&mut buf).await.unwrap();\n            stream\n                .write_all(b\"HTTP/1.1 200 OK\\r\\nContent-Length: 1\\r\\n\\r\\na\")\n                .await\n                .unwrap();\n        });\n        // make sure the above starts before the lines below\n        sleep(Duration::from_millis(10)).await;\n\n        let client = reqwest::Client::builder()\n            .danger_accept_invalid_certs(true)\n            .build()\n            .unwrap();\n\n        let res = client.get(format!(\"https://{addr}\")).send().await.unwrap();\n        assert_eq!(res.status(), reqwest::StatusCode::OK);\n    }\n\n    #[cfg(feature = \"connection_filter\")]\n    #[test]\n    fn test_connection_filter_inheritance() {\n        #[derive(Debug, Clone)]\n        struct TestFilter {\n            counter: Arc<AtomicUsize>,\n        }\n\n        #[async_trait]\n        impl ConnectionFilter for TestFilter {\n            async fn should_accept(&self, _addr: Option<&std::net::SocketAddr>) -> bool {\n                self.counter.fetch_add(1, Ordering::SeqCst);\n                true\n            }\n        }\n\n        let mut listeners = Listeners::new();\n\n        // Add an endpoint before setting filter\n        listeners.add_tcp(\"127.0.0.1:7104\");\n\n        // Set the connection filter\n        let filter = Arc::new(TestFilter {\n            counter: Arc::new(AtomicUsize::new(0)),\n        });\n        listeners.set_connection_filter(filter.clone());\n\n        // Add endpoints after setting filter\n        listeners.add_tcp(\"127.0.0.1:7105\");\n        #[cfg(feature = \"any_tls\")]\n        {\n            // Only test TLS if the feature is enabled\n            if let Ok(tls_settings) = TlsSettings::intermediate(\n                &format!(\"{}/tests/keys/server.crt\", env!(\"CARGO_MANIFEST_DIR\")),\n                &format!(\"{}/tests/keys/key.pem\", env!(\"CARGO_MANIFEST_DIR\")),\n            ) {\n                listeners.add_tls_with_settings(\"127.0.0.1:7106\", None, tls_settings);\n            }\n        }\n\n        // Verify all stacks have the filter (only when feature is enabled)\n        for stack in &listeners.stacks {\n            assert!(\n                stack.connection_filter.is_some(),\n                \"All stacks should have the connection filter set\"\n            );\n        }\n    }\n}\n"
  },
  {
    "path": "pingora-core/src/listeners/tls/boringssl_openssl/mod.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\nuse log::debug;\nuse pingora_error::{ErrorType, OrErr, Result};\nuse std::ops::{Deref, DerefMut};\n\nuse crate::listeners::tls::boringssl_openssl::alpn::valid_alpn;\npub use crate::protocols::tls::ALPN;\nuse crate::protocols::IO;\nuse crate::tls::ssl::AlpnError;\nuse crate::tls::ssl::{SslAcceptor, SslAcceptorBuilder, SslFiletype, SslMethod};\nuse crate::{\n    listeners::TlsAcceptCallbacks,\n    protocols::tls::{\n        server::{handshake, handshake_with_callback},\n        SslStream,\n    },\n};\npub const TLS_CONF_ERR: ErrorType = ErrorType::Custom(\"TLSConfigError\");\n\npub(crate) struct Acceptor {\n    ssl_acceptor: SslAcceptor,\n    callbacks: Option<TlsAcceptCallbacks>,\n}\n\n/// The TLS settings of a listening endpoint\npub struct TlsSettings {\n    accept_builder: SslAcceptorBuilder,\n    callbacks: Option<TlsAcceptCallbacks>,\n}\n\nimpl From<SslAcceptorBuilder> for TlsSettings {\n    fn from(settings: SslAcceptorBuilder) -> Self {\n        TlsSettings {\n            accept_builder: settings,\n            callbacks: None,\n        }\n    }\n}\n\nimpl Deref for TlsSettings {\n    type Target = SslAcceptorBuilder;\n\n    fn deref(&self) -> &Self::Target {\n        &self.accept_builder\n    }\n}\n\nimpl DerefMut for TlsSettings {\n    fn deref_mut(&mut self) -> &mut Self::Target {\n        &mut self.accept_builder\n    }\n}\n\nimpl TlsSettings {\n    /// Create a new [`TlsSettings`] with the [Mozilla Intermediate](https://wiki.mozilla.org/Security/Server_Side_TLS#Intermediate_compatibility_.28recommended.29)\n    /// server side TLS settings. Users can adjust the TLS settings after this object is created.\n    /// Return error if the provided certificate and private key are invalid or not found.\n    pub fn intermediate(cert_path: &str, key_path: &str) -> Result<Self> {\n        let mut accept_builder = SslAcceptor::mozilla_intermediate_v5(SslMethod::tls()).or_err(\n            TLS_CONF_ERR,\n            \"fail to create mozilla_intermediate_v5 Acceptor\",\n        )?;\n        accept_builder\n            .set_private_key_file(key_path, SslFiletype::PEM)\n            .or_err_with(TLS_CONF_ERR, || format!(\"fail to read key file {key_path}\"))?;\n        accept_builder\n            .set_certificate_chain_file(cert_path)\n            .or_err_with(TLS_CONF_ERR, || {\n                format!(\"fail to read cert file {cert_path}\")\n            })?;\n        Ok(TlsSettings {\n            accept_builder,\n            callbacks: None,\n        })\n    }\n\n    /// Create a new [`TlsSettings`] similar to [TlsSettings::intermediate()]. A struct that implements [TlsAcceptCallbacks]\n    /// is needed to provide the certificate during the TLS handshake.\n    pub fn with_callbacks(callbacks: TlsAcceptCallbacks) -> Result<Self> {\n        let accept_builder = SslAcceptor::mozilla_intermediate_v5(SslMethod::tls()).or_err(\n            TLS_CONF_ERR,\n            \"fail to create mozilla_intermediate_v5 Acceptor\",\n        )?;\n        Ok(TlsSettings {\n            accept_builder,\n            callbacks: Some(callbacks),\n        })\n    }\n\n    /// Enable HTTP/2 support for this endpoint, which is default off.\n    /// This effectively sets the ALPN to prefer HTTP/2 with HTTP/1.1 allowed\n    pub fn enable_h2(&mut self) {\n        self.set_alpn(ALPN::H2H1);\n    }\n\n    /// Set the ALPN preference of this endpoint. See [`ALPN`] for more details\n    pub fn set_alpn(&mut self, alpn: ALPN) {\n        match alpn {\n            ALPN::H2H1 => self\n                .accept_builder\n                .set_alpn_select_callback(alpn::prefer_h2),\n            ALPN::H1 => self.accept_builder.set_alpn_select_callback(alpn::h1_only),\n            ALPN::H2 => self.accept_builder.set_alpn_select_callback(alpn::h2_only),\n            ALPN::Custom(custom) => {\n                self.accept_builder\n                    .set_alpn_select_callback(move |_, alpn_in| {\n                        if !valid_alpn(alpn_in) {\n                            return Err(AlpnError::NOACK);\n                        }\n                        match alpn::select_protocol(alpn_in, custom.protocol()) {\n                            Some(p) => Ok(p),\n                            None => Err(AlpnError::NOACK),\n                        }\n                    });\n            }\n        }\n    }\n\n    pub(crate) fn build(self) -> Acceptor {\n        Acceptor {\n            ssl_acceptor: self.accept_builder.build(),\n            callbacks: self.callbacks,\n        }\n    }\n}\n\nimpl Acceptor {\n    pub async fn tls_handshake<S: IO>(&self, stream: S) -> Result<SslStream<S>> {\n        debug!(\"new ssl session\");\n        // TODO: be able to offload this handshake in a thread pool\n        if let Some(cb) = self.callbacks.as_ref() {\n            handshake_with_callback(&self.ssl_acceptor, stream, cb).await\n        } else {\n            handshake(&self.ssl_acceptor, stream).await\n        }\n    }\n}\n\nmod alpn {\n    use super::*;\n    use crate::tls::ssl::{select_next_proto, AlpnError, SslRef};\n\n    pub(super) fn valid_alpn(alpn_in: &[u8]) -> bool {\n        if alpn_in.is_empty() {\n            return false;\n        }\n        // TODO: can add more thorough validation here.\n        true\n    }\n\n    /// Finds the first protocol in the client-offered ALPN list that matches the given protocol.\n    ///\n    /// This is a helper for ALPN negotiation. It iterates over the client's protocol list\n    /// (in wire format) and returns the first protocol that matches proto\n    /// The returned reference always points into `client_protocols`, so lifetimes are correct.\n    pub(super) fn select_protocol<'a>(\n        client_protocols: &'a [u8],\n        proto: &[u8],\n    ) -> Option<&'a [u8]> {\n        let mut bytes = client_protocols;\n        while !bytes.is_empty() {\n            let len = bytes[0] as usize;\n            bytes = &bytes[1..];\n            if len == proto.len() && &bytes[..len] == proto {\n                return Some(&bytes[..len]);\n            }\n            bytes = &bytes[len..];\n        }\n        None\n    }\n\n    // A standard implementation provided by the SSL lib is used below\n\n    pub fn prefer_h2<'a>(_ssl: &mut SslRef, alpn_in: &'a [u8]) -> Result<&'a [u8], AlpnError> {\n        if !valid_alpn(alpn_in) {\n            return Err(AlpnError::NOACK);\n        }\n        match select_next_proto(ALPN::H2H1.to_wire_preference(), alpn_in) {\n            Some(p) => Ok(p),\n            _ => Err(AlpnError::NOACK), // unknown ALPN, just ignore it. Most clients will fallback to h1\n        }\n    }\n\n    pub fn h1_only<'a>(_ssl: &mut SslRef, alpn_in: &'a [u8]) -> Result<&'a [u8], AlpnError> {\n        if !valid_alpn(alpn_in) {\n            return Err(AlpnError::NOACK);\n        }\n        match select_next_proto(ALPN::H1.to_wire_preference(), alpn_in) {\n            Some(p) => Ok(p),\n            _ => Err(AlpnError::NOACK), // unknown ALPN, just ignore it. Most clients will fallback to h1\n        }\n    }\n\n    pub fn h2_only<'a>(_ssl: &mut SslRef, alpn_in: &'a [u8]) -> Result<&'a [u8], AlpnError> {\n        if !valid_alpn(alpn_in) {\n            return Err(AlpnError::ALERT_FATAL);\n        }\n        match select_next_proto(ALPN::H2.to_wire_preference(), alpn_in) {\n            Some(p) => Ok(p),\n            _ => Err(AlpnError::ALERT_FATAL), // cannot agree\n        }\n    }\n}\n"
  },
  {
    "path": "pingora-core/src/listeners/tls/mod.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n#[cfg(feature = \"openssl_derived\")]\nmod boringssl_openssl;\n\n#[cfg(feature = \"openssl_derived\")]\npub use boringssl_openssl::*;\n\n#[cfg(feature = \"rustls\")]\nmod rustls;\n\n#[cfg(feature = \"rustls\")]\npub use rustls::*;\n\n#[cfg(feature = \"s2n\")]\nmod s2n;\n\n#[cfg(feature = \"s2n\")]\npub use s2n::*;\n"
  },
  {
    "path": "pingora-core/src/listeners/tls/rustls/mod.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\nuse std::sync::Arc;\n\nuse crate::listeners::TlsAcceptCallbacks;\nuse crate::protocols::tls::{server::handshake, server::handshake_with_callback, TlsStream};\nuse log::debug;\nuse pingora_error::ErrorType::InternalError;\nuse pingora_error::{Error, OrErr, Result};\nuse pingora_rustls::load_certs_and_key_files;\nuse pingora_rustls::ClientCertVerifier;\nuse pingora_rustls::ServerConfig;\nuse pingora_rustls::{version, TlsAcceptor as RusTlsAcceptor};\n\nuse crate::protocols::{ALPN, IO};\n\n/// The TLS settings of a listening endpoint\npub struct TlsSettings {\n    alpn_protocols: Option<Vec<Vec<u8>>>,\n    cert_path: String,\n    key_path: String,\n    client_cert_verifier: Option<Arc<dyn ClientCertVerifier>>,\n}\n\npub struct Acceptor {\n    pub acceptor: RusTlsAcceptor,\n    callbacks: Option<TlsAcceptCallbacks>,\n}\n\nimpl TlsSettings {\n    /// Create a Rustls acceptor based on the current setting for certificates,\n    /// keys, and protocols.\n    ///\n    /// _NOTE_ This function will panic if there is an error in loading\n    /// certificate files or constructing the builder\n    ///\n    /// Todo: Return a result instead of panicking XD\n    pub fn build(self) -> Acceptor {\n        let Ok(Some((certs, key))) = load_certs_and_key_files(&self.cert_path, &self.key_path)\n        else {\n            panic!(\n                \"Failed to load provided certificates \\\"{}\\\" or key \\\"{}\\\".\",\n                self.cert_path, self.key_path\n            )\n        };\n\n        let builder =\n            ServerConfig::builder_with_protocol_versions(&[&version::TLS12, &version::TLS13]);\n        let builder = if let Some(verifier) = self.client_cert_verifier {\n            builder.with_client_cert_verifier(verifier)\n        } else {\n            builder.with_no_client_auth()\n        };\n        let mut config = builder\n            .with_single_cert(certs, key)\n            .explain_err(InternalError, |e| {\n                format!(\"Failed to create server listener config: {e}\")\n            })\n            .unwrap();\n\n        if let Some(alpn_protocols) = self.alpn_protocols {\n            config.alpn_protocols = alpn_protocols;\n        }\n\n        Acceptor {\n            acceptor: RusTlsAcceptor::from(Arc::new(config)),\n            callbacks: None,\n        }\n    }\n\n    /// Enable HTTP/2 support for this endpoint, which is default off.\n    /// This effectively sets the ALPN to prefer HTTP/2 with HTTP/1.1 allowed\n    pub fn enable_h2(&mut self) {\n        self.set_alpn(ALPN::H2H1);\n    }\n\n    pub fn set_alpn(&mut self, alpn: ALPN) {\n        self.alpn_protocols = Some(alpn.to_wire_protocols());\n    }\n\n    /// Configure mTLS by providing a rustls client certificate verifier.\n    pub fn set_client_cert_verifier(&mut self, verifier: Arc<dyn ClientCertVerifier>) {\n        self.client_cert_verifier = Some(verifier);\n    }\n\n    pub fn intermediate(cert_path: &str, key_path: &str) -> Result<Self>\n    where\n        Self: Sized,\n    {\n        Ok(TlsSettings {\n            alpn_protocols: None,\n            cert_path: cert_path.to_string(),\n            key_path: key_path.to_string(),\n            client_cert_verifier: None,\n        })\n    }\n\n    pub fn with_callbacks() -> Result<Self>\n    where\n        Self: Sized,\n    {\n        // TODO: verify if/how callback in handshake can be done using Rustls\n        Error::e_explain(\n            InternalError,\n            \"Certificate callbacks are not supported with feature \\\"rustls\\\".\",\n        )\n    }\n}\n\nimpl Acceptor {\n    pub async fn tls_handshake<S: IO>(&self, stream: S) -> Result<TlsStream<S>> {\n        debug!(\"new tls session\");\n        // TODO: be able to offload this handshake in a thread pool\n        if let Some(cb) = self.callbacks.as_ref() {\n            handshake_with_callback(self, stream, cb).await\n        } else {\n            handshake(self, stream).await\n        }\n    }\n}\n"
  },
  {
    "path": "pingora-core/src/listeners/tls/s2n/mod.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\nuse std::sync::Arc;\n\nuse log::debug;\nuse pingora_error::Result;\nuse pingora_s2n::{\n    load_certs_and_key_files, ClientAuthType, Config, IgnoreVerifyHostnameCallback, S2NPolicy,\n    TlsAcceptor, DEFAULT_TLS13,\n};\n\nuse crate::protocols::tls::server::handshake;\nuse crate::protocols::tls::{CaType, PskConfig, PskType, S2NConnectionBuilder, TlsStream};\nuse crate::protocols::{ALPN, IO};\n\n/// The TLS settings of a listening endpoint\npub struct TlsSettings {\n    cert_path: Option<String>,\n    key_path: Option<String>,\n    ca: Option<CaType>,\n    alpn: Option<ALPN>,\n    psk_config: Option<Arc<PskType>>,\n    security_policy: Option<S2NPolicy>,\n    client_auth_required: bool,\n    verify_client_hostname: bool,\n    max_blinding_delay: Option<u32>,\n}\n\npub struct Acceptor {\n    pub acceptor: TlsAcceptor<S2NConnectionBuilder>,\n}\n\nimpl TlsSettings {\n    pub fn build(self) -> Acceptor {\n        let mut builder = Config::builder();\n\n        // Default security policy with TLS 1.3 support\n        // https://aws.github.io/s2n-tls/usage-guide/ch06-security-policies.html\n        let policy = self.security_policy.unwrap_or(DEFAULT_TLS13);\n\n        if let Some(max_blinding_delay) = self.max_blinding_delay {\n            builder.set_max_blinding_delay(max_blinding_delay).unwrap();\n        }\n\n        if self.client_auth_required {\n            builder\n                .set_client_auth_type(ClientAuthType::Required)\n                .unwrap();\n        }\n\n        if let Some(alpn) = self.alpn {\n            builder\n                .set_application_protocol_preference(alpn.to_wire_protocols())\n                .unwrap();\n        }\n\n        if let (Some(cert_path), Some(key_path)) = (self.cert_path, self.key_path) {\n            let Ok((cert, key)) = load_certs_and_key_files(&cert_path, &key_path) else {\n                panic!(\n                    \"Failed to load provided certificates \\\"{}\\\" or key \\\"{}\\\".\",\n                    cert_path, key_path\n                )\n            };\n\n            builder.load_pem(&cert, &key).unwrap();\n        }\n\n        if let Some(ca) = self.ca {\n            builder.trust_pem(&ca.raw_pem).expect(\"invalid ca pem\");\n        }\n\n        if !self.verify_client_hostname {\n            builder\n                .set_verify_host_callback(IgnoreVerifyHostnameCallback::new())\n                .unwrap();\n        }\n\n        let config = builder.build().unwrap();\n        let connection_builder = S2NConnectionBuilder {\n            config: config,\n            psk_config: self.psk_config.clone(),\n            security_policy: Some(policy.clone()),\n        };\n\n        Acceptor {\n            acceptor: TlsAcceptor::new(connection_builder),\n        }\n    }\n\n    /// Enable HTTP/2 support for this endpoint, which is default off.\n    /// This effectively sets the ALPN to prefer HTTP/2 with HTTP/1.1 allowed\n    pub fn enable_h2(&mut self) {\n        self.set_alpn(ALPN::H2H1);\n    }\n\n    fn set_alpn(&mut self, alpn: ALPN) {\n        self.alpn = Some(alpn);\n    }\n\n    /// Configure CA to use for mTLS\n    pub fn set_ca(&mut self, ca: CaType) {\n        self.ca = Some(ca);\n    }\n\n    /// Configure pre-shared keys to use for TLS-PSK handshake\n    /// https://datatracker.ietf.org/doc/html/rfc4279\n    pub fn set_psk_config(&mut self, psk_config: PskConfig) {\n        self.psk_config = Some(Arc::new(psk_config));\n    }\n\n    /// S2N-TLS security policy to use. If not set, the default policy\n    /// \"default_tls13\" will be used.\n    /// https://aws.github.io/s2n-tls/usage-guide/ch06-security-policies.html\n    pub fn set_policy(&mut self, policy: S2NPolicy) {\n        self.security_policy = Some(policy);\n    }\n\n    /// The certificate and private key to use for TLS connections\n    pub fn set_cert(&mut self, cert_path: &str, key_path: &str) {\n        self.cert_path = Some(cert_path.to_string());\n        self.key_path = Some(key_path.to_string());\n    }\n\n    /// Require client certificate authentication (mTLS)\n    pub fn set_client_auth_required(&mut self, required: bool) {\n        self.client_auth_required = required;\n    }\n\n    /// If validating client certificate, also verify client hostname (mTLS)\n    pub fn set_verify_client_hostname(&mut self, verify: bool) {\n        self.verify_client_hostname = verify;\n    }\n\n    /// S2N-TLS will delay a response up to the max blinding delay (default 30)\n    /// seconds whenever an error triggered by a peer occurs to mitigate against\n    /// timing side channels.\n    pub fn set_max_blinding_delay(&mut self, delay: u32) {\n        self.max_blinding_delay = Some(delay);\n    }\n\n    pub fn intermediate(cert_path: &str, key_path: &str) -> Result<Self>\n    where\n        Self: Sized,\n    {\n        Ok(TlsSettings {\n            cert_path: Some(cert_path.to_string()),\n            key_path: Some(key_path.to_string()),\n            ca: None,\n            security_policy: None,\n            alpn: None,\n            psk_config: None,\n            client_auth_required: false,\n            verify_client_hostname: false,\n            max_blinding_delay: None,\n        })\n    }\n\n    pub fn new() -> Self {\n        TlsSettings {\n            cert_path: None,\n            key_path: None,\n            ca: None,\n            security_policy: None,\n            alpn: None,\n            psk_config: None,\n            client_auth_required: false,\n            verify_client_hostname: false,\n            max_blinding_delay: None,\n        }\n    }\n}\n\nimpl Acceptor {\n    pub async fn tls_handshake<S: IO>(&self, stream: S) -> Result<TlsStream<S>> {\n        debug!(\"new tls session\");\n        handshake(self, stream).await\n    }\n}\n"
  },
  {
    "path": "pingora-core/src/modules/http/compression.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! HTTP compression filter\n\nuse super::*;\nuse crate::protocols::http::compression::ResponseCompressionCtx;\nuse std::ops::{Deref, DerefMut};\n\n/// HTTP response compression module\npub struct ResponseCompression(ResponseCompressionCtx);\n\nimpl Deref for ResponseCompression {\n    type Target = ResponseCompressionCtx;\n\n    fn deref(&self) -> &Self::Target {\n        &self.0\n    }\n}\n\nimpl DerefMut for ResponseCompression {\n    fn deref_mut(&mut self) -> &mut Self::Target {\n        &mut self.0\n    }\n}\n\n#[async_trait]\nimpl HttpModule for ResponseCompression {\n    fn as_any(&self) -> &dyn std::any::Any {\n        self\n    }\n    fn as_any_mut(&mut self) -> &mut dyn std::any::Any {\n        self\n    }\n\n    async fn request_header_filter(&mut self, req: &mut RequestHeader) -> Result<()> {\n        self.0.request_filter(req);\n        Ok(())\n    }\n\n    async fn response_header_filter(\n        &mut self,\n        resp: &mut ResponseHeader,\n        end_of_stream: bool,\n    ) -> Result<()> {\n        self.0.response_header_filter(resp, end_of_stream);\n        Ok(())\n    }\n\n    fn response_body_filter(\n        &mut self,\n        body: &mut Option<Bytes>,\n        end_of_stream: bool,\n    ) -> Result<()> {\n        if !self.0.is_enabled() {\n            return Ok(());\n        }\n        let compressed = self.0.response_body_filter(body.as_ref(), end_of_stream);\n        if compressed.is_some() {\n            *body = compressed;\n        }\n        Ok(())\n    }\n\n    fn response_done_filter(&mut self) -> Result<Option<Bytes>> {\n        if !self.0.is_enabled() {\n            return Ok(None);\n        }\n        // Flush or finish any remaining encoded bytes upon HTTP response completion\n        // (if it was not already ended in the body filter).\n        Ok(self.0.response_body_filter(None, true))\n    }\n}\n\n/// The builder for HTTP response compression module\npub struct ResponseCompressionBuilder {\n    level: u32,\n}\n\nimpl ResponseCompressionBuilder {\n    /// Return a [ModuleBuilder] for [ResponseCompression] with the given compression level\n    pub fn enable(level: u32) -> ModuleBuilder {\n        Box::new(ResponseCompressionBuilder { level })\n    }\n}\n\nimpl HttpModuleBuilder for ResponseCompressionBuilder {\n    fn init(&self) -> Module {\n        Box::new(ResponseCompression(ResponseCompressionCtx::new(\n            self.level, false, false,\n        )))\n    }\n\n    fn order(&self) -> i16 {\n        // run the response filter later than most others filters\n        i16::MIN / 2\n    }\n}\n"
  },
  {
    "path": "pingora-core/src/modules/http/grpc_web.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\nuse super::*;\nuse crate::protocols::http::bridge::grpc_web::GrpcWebCtx;\nuse std::ops::{Deref, DerefMut};\n\n/// gRPC-web bridge module, this will convert\n/// HTTP/1.1 gRPC-web requests to H2 gRPC requests\n#[derive(Default)]\npub struct GrpcWebBridge(GrpcWebCtx);\n\nimpl Deref for GrpcWebBridge {\n    type Target = GrpcWebCtx;\n\n    fn deref(&self) -> &Self::Target {\n        &self.0\n    }\n}\n\nimpl DerefMut for GrpcWebBridge {\n    fn deref_mut(&mut self) -> &mut Self::Target {\n        &mut self.0\n    }\n}\n\n#[async_trait]\nimpl HttpModule for GrpcWebBridge {\n    fn as_any(&self) -> &dyn std::any::Any {\n        self\n    }\n\n    fn as_any_mut(&mut self) -> &mut dyn std::any::Any {\n        self\n    }\n\n    async fn request_header_filter(&mut self, req: &mut RequestHeader) -> Result<()> {\n        self.0.request_header_filter(req);\n        Ok(())\n    }\n\n    async fn response_header_filter(\n        &mut self,\n        resp: &mut ResponseHeader,\n        _end_of_stream: bool,\n    ) -> Result<()> {\n        self.0.response_header_filter(resp);\n        Ok(())\n    }\n\n    fn response_trailer_filter(\n        &mut self,\n        trailers: &mut Option<Box<HeaderMap>>,\n    ) -> Result<Option<Bytes>> {\n        if let Some(trailers) = trailers {\n            return self.0.response_trailer_filter(trailers);\n        }\n        Ok(None)\n    }\n}\n\n/// The builder for gRPC-web bridge module\npub struct GrpcWeb;\n\nimpl HttpModuleBuilder for GrpcWeb {\n    fn init(&self) -> Module {\n        Box::new(GrpcWebBridge::default())\n    }\n}\n"
  },
  {
    "path": "pingora-core/src/modules/http/mod.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! Modules for HTTP traffic.\n//!\n//! [HttpModule]s define request and response filters to use while running an\n//! [HttpServer](crate::apps::http_app::HttpServer)\n//! application.\n//! See the [ResponseCompression](crate::modules::http::compression::ResponseCompression)\n//! module for an example of how to implement a basic module.\n\npub mod compression;\npub mod grpc_web;\n\nuse async_trait::async_trait;\nuse bytes::Bytes;\nuse http::HeaderMap;\nuse once_cell::sync::OnceCell;\nuse pingora_error::Result;\nuse pingora_http::{RequestHeader, ResponseHeader};\nuse std::any::Any;\nuse std::any::TypeId;\nuse std::collections::HashMap;\nuse std::sync::Arc;\n\n/// The trait an HTTP traffic module needs to implement\n#[async_trait]\npub trait HttpModule {\n    async fn request_header_filter(&mut self, _req: &mut RequestHeader) -> Result<()> {\n        Ok(())\n    }\n\n    async fn request_body_filter(\n        &mut self,\n        _body: &mut Option<Bytes>,\n        _end_of_stream: bool,\n    ) -> Result<()> {\n        Ok(())\n    }\n\n    async fn response_header_filter(\n        &mut self,\n        _resp: &mut ResponseHeader,\n        _end_of_stream: bool,\n    ) -> Result<()> {\n        Ok(())\n    }\n\n    fn response_body_filter(\n        &mut self,\n        _body: &mut Option<Bytes>,\n        _end_of_stream: bool,\n    ) -> Result<()> {\n        Ok(())\n    }\n\n    fn response_trailer_filter(\n        &mut self,\n        _trailers: &mut Option<Box<HeaderMap>>,\n    ) -> Result<Option<Bytes>> {\n        Ok(None)\n    }\n\n    fn response_done_filter(&mut self) -> Result<Option<Bytes>> {\n        Ok(None)\n    }\n\n    fn as_any(&self) -> &dyn Any;\n    fn as_any_mut(&mut self) -> &mut dyn Any;\n}\n\npub type Module = Box<dyn HttpModule + 'static + Send + Sync>;\n\n/// Trait to init the http module ctx for each request\npub trait HttpModuleBuilder {\n    /// The order the module will run\n    ///\n    /// The lower the value, the later it runs relative to other filters.\n    /// If the order of the filter is not important, leave it to the default 0.\n    fn order(&self) -> i16 {\n        0\n    }\n\n    /// Initialize and return the per request module context\n    fn init(&self) -> Module;\n}\n\npub type ModuleBuilder = Box<dyn HttpModuleBuilder + 'static + Send + Sync>;\n\n/// The object to hold multiple http modules\npub struct HttpModules {\n    modules: Vec<ModuleBuilder>,\n    module_index: OnceCell<Arc<HashMap<TypeId, usize>>>,\n}\n\nimpl HttpModules {\n    /// Create a new [HttpModules]\n    pub fn new() -> Self {\n        HttpModules {\n            modules: vec![],\n            module_index: OnceCell::new(),\n        }\n    }\n\n    /// Add a new [ModuleBuilder] to [HttpModules]\n    ///\n    /// Each type of [HttpModule] can be only added once.\n    /// # Panic\n    /// Panic if any [HttpModule] is added more than once.\n    pub fn add_module(&mut self, builder: ModuleBuilder) {\n        if self.module_index.get().is_some() {\n            // We use a shared module_index the index would be out of sync if we\n            // add more modules.\n            panic!(\"cannot add module after ctx is already built\")\n        }\n        self.modules.push(builder);\n        // not the most efficient way but should be fine\n        // largest order first\n        self.modules.sort_by_key(|m| -m.order());\n    }\n\n    /// Build the contexts of all the modules added to this [HttpModules]\n    pub fn build_ctx(&self) -> HttpModuleCtx {\n        let module_ctx: Vec<_> = self.modules.iter().map(|b| b.init()).collect();\n        let module_index = self\n            .module_index\n            .get_or_init(|| {\n                let mut module_index = HashMap::with_capacity(self.modules.len());\n                for (i, c) in module_ctx.iter().enumerate() {\n                    let exist = module_index.insert(c.as_any().type_id(), i);\n                    if exist.is_some() {\n                        panic!(\"duplicated filters found\")\n                    }\n                }\n                Arc::new(module_index)\n            })\n            .clone();\n\n        HttpModuleCtx {\n            module_ctx,\n            module_index,\n        }\n    }\n}\n\n/// The Contexts of multiple modules\n///\n/// This is the object that will apply all the included modules to a certain HTTP request.\n/// The modules are ordered according to their `order()`.\npub struct HttpModuleCtx {\n    // the modules in the order of execution\n    module_ctx: Vec<Module>,\n    // find the module in the vec with its type ID\n    module_index: Arc<HashMap<TypeId, usize>>,\n}\n\nimpl HttpModuleCtx {\n    /// Create a placeholder empty [HttpModuleCtx].\n    ///\n    /// [HttpModules] should be used to create nonempty [HttpModuleCtx].\n    pub fn empty() -> Self {\n        HttpModuleCtx {\n            module_ctx: vec![],\n            module_index: Arc::new(HashMap::new()),\n        }\n    }\n\n    /// Get a ref to [HttpModule] if any.\n    pub fn get<T: 'static>(&self) -> Option<&T> {\n        let idx = self.module_index.get(&TypeId::of::<T>())?;\n        let ctx = &self.module_ctx[*idx];\n        Some(\n            ctx.as_any()\n                .downcast_ref::<T>()\n                .expect(\"type should always match\"),\n        )\n    }\n\n    /// Get a mut ref to [HttpModule] if any.\n    pub fn get_mut<T: 'static>(&mut self) -> Option<&mut T> {\n        let idx = self.module_index.get(&TypeId::of::<T>())?;\n        let ctx = &mut self.module_ctx[*idx];\n        Some(\n            ctx.as_any_mut()\n                .downcast_mut::<T>()\n                .expect(\"type should always match\"),\n        )\n    }\n\n    /// Run the `request_header_filter` for all the modules according to their orders.\n    pub async fn request_header_filter(&mut self, req: &mut RequestHeader) -> Result<()> {\n        for filter in self.module_ctx.iter_mut() {\n            filter.request_header_filter(req).await?;\n        }\n        Ok(())\n    }\n\n    /// Run the `request_body_filter` for all the modules according to their orders.\n    pub async fn request_body_filter(\n        &mut self,\n        body: &mut Option<Bytes>,\n        end_of_stream: bool,\n    ) -> Result<()> {\n        for filter in self.module_ctx.iter_mut() {\n            filter.request_body_filter(body, end_of_stream).await?;\n        }\n        Ok(())\n    }\n\n    /// Run the `response_header_filter` for all the modules according to their orders.\n    pub async fn response_header_filter(\n        &mut self,\n        req: &mut ResponseHeader,\n        end_of_stream: bool,\n    ) -> Result<()> {\n        for filter in self.module_ctx.iter_mut() {\n            filter.response_header_filter(req, end_of_stream).await?;\n        }\n        Ok(())\n    }\n\n    /// Run the `response_body_filter` for all the modules according to their orders.\n    pub fn response_body_filter(\n        &mut self,\n        body: &mut Option<Bytes>,\n        end_of_stream: bool,\n    ) -> Result<()> {\n        for filter in self.module_ctx.iter_mut() {\n            filter.response_body_filter(body, end_of_stream)?;\n        }\n        Ok(())\n    }\n\n    /// Run the `response_trailer_filter` for all the modules according to their orders.\n    ///\n    /// Returns an `Option<Bytes>` which can be used to write response trailers into\n    /// the response body. Note, if multiple modules attempt to write trailers into\n    /// the body the last one will be used.\n    ///\n    /// Implementors that intend to write trailers into the body need to ensure their filter\n    /// is using an encoding that supports this.\n    pub fn response_trailer_filter(\n        &mut self,\n        trailers: &mut Option<Box<HeaderMap>>,\n    ) -> Result<Option<Bytes>> {\n        let mut encoded = None;\n        for filter in self.module_ctx.iter_mut() {\n            if let Some(buf) = filter.response_trailer_filter(trailers)? {\n                encoded = Some(buf);\n            }\n        }\n        Ok(encoded)\n    }\n\n    /// Run the `response_done_filter` for all the modules according to their orders.\n    ///\n    /// This filter may be invoked in certain response paths to signal end of response\n    /// if not already done so via trailers or body (with end flag set).\n    ///\n    /// Returns an `Option<Bytes>` which can be used to write additional response body\n    /// bytes. Note, if multiple modules attempt to write body bytes, only the last one\n    /// will be used.\n    pub fn response_done_filter(&mut self) -> Result<Option<Bytes>> {\n        let mut encoded = None;\n        for filter in self.module_ctx.iter_mut() {\n            if let Some(buf) = filter.response_done_filter()? {\n                encoded = Some(buf);\n            }\n        }\n        Ok(encoded)\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use super::*;\n\n    struct MyModule;\n    #[async_trait]\n    impl HttpModule for MyModule {\n        fn as_any(&self) -> &dyn Any {\n            self\n        }\n        fn as_any_mut(&mut self) -> &mut dyn Any {\n            self\n        }\n        async fn request_header_filter(&mut self, req: &mut RequestHeader) -> Result<()> {\n            req.insert_header(\"my-filter\", \"1\")\n        }\n    }\n    struct MyModuleBuilder;\n    impl HttpModuleBuilder for MyModuleBuilder {\n        fn order(&self) -> i16 {\n            1\n        }\n\n        fn init(&self) -> Module {\n            Box::new(MyModule)\n        }\n    }\n\n    struct MyOtherModule;\n    #[async_trait]\n    impl HttpModule for MyOtherModule {\n        fn as_any(&self) -> &dyn Any {\n            self\n        }\n        fn as_any_mut(&mut self) -> &mut dyn Any {\n            self\n        }\n        async fn request_header_filter(&mut self, req: &mut RequestHeader) -> Result<()> {\n            if req.headers.get(\"my-filter\").is_some() {\n                // if this MyOtherModule runs after MyModule\n                req.insert_header(\"my-filter\", \"2\")\n            } else {\n                // if this MyOtherModule runs before MyModule\n                req.insert_header(\"my-other-filter\", \"1\")\n            }\n        }\n    }\n    struct MyOtherModuleBuilder;\n    impl HttpModuleBuilder for MyOtherModuleBuilder {\n        fn order(&self) -> i16 {\n            -1\n        }\n\n        fn init(&self) -> Module {\n            Box::new(MyOtherModule)\n        }\n    }\n\n    #[test]\n    fn test_module_get() {\n        let mut http_module = HttpModules::new();\n        http_module.add_module(Box::new(MyModuleBuilder));\n        http_module.add_module(Box::new(MyOtherModuleBuilder));\n        let mut ctx = http_module.build_ctx();\n        assert!(ctx.get::<MyModule>().is_some());\n        assert!(ctx.get::<MyOtherModule>().is_some());\n        assert!(ctx.get::<usize>().is_none());\n        assert!(ctx.get_mut::<MyModule>().is_some());\n        assert!(ctx.get_mut::<MyOtherModule>().is_some());\n        assert!(ctx.get_mut::<usize>().is_none());\n    }\n\n    #[tokio::test]\n    async fn test_module_filter() {\n        let mut http_module = HttpModules::new();\n        http_module.add_module(Box::new(MyOtherModuleBuilder));\n        http_module.add_module(Box::new(MyModuleBuilder));\n        let mut ctx = http_module.build_ctx();\n        let mut req = RequestHeader::build(\"Get\", b\"/\", None).unwrap();\n        ctx.request_header_filter(&mut req).await.unwrap();\n        // MyModule runs before MyOtherModule\n        assert_eq!(req.headers.get(\"my-filter\").unwrap(), \"2\");\n        assert!(req.headers.get(\"my-other-filter\").is_none());\n    }\n}\n"
  },
  {
    "path": "pingora-core/src/modules/mod.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! Modules to extend the functionalities of pingora services.\npub mod http;\n"
  },
  {
    "path": "pingora-core/src/protocols/digest.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! Extra information about the connection\n\nuse std::sync::Arc;\nuse std::time::{Duration, SystemTime};\n\nuse once_cell::sync::OnceCell;\n\nuse super::l4::ext::{get_original_dest, get_recv_buf, get_snd_buf, get_tcp_info, TCP_INFO};\nuse super::l4::socket::SocketAddr;\nuse super::raw_connect::ProxyDigest;\nuse super::tls::digest::SslDigest;\n\n/// The information can be extracted from a connection\n#[derive(Clone, Debug, Default)]\npub struct Digest {\n    /// Information regarding the TLS of this connection if any\n    pub ssl_digest: Option<Arc<SslDigest>>,\n    /// Timing information\n    pub timing_digest: Vec<Option<TimingDigest>>,\n    /// information regarding the CONNECT proxy this connection uses.\n    pub proxy_digest: Option<Arc<ProxyDigest>>,\n    /// Information about underlying socket/fd of this connection\n    pub socket_digest: Option<Arc<SocketDigest>>,\n}\n\n/// The interface to return protocol related information\npub trait ProtoDigest {\n    fn get_digest(&self) -> Option<&Digest> {\n        None\n    }\n}\n\n/// The timing information of the connection\n#[derive(Clone, Debug)]\npub struct TimingDigest {\n    /// When this connection was established\n    pub established_ts: SystemTime,\n}\n\nimpl Default for TimingDigest {\n    fn default() -> Self {\n        TimingDigest {\n            established_ts: SystemTime::UNIX_EPOCH,\n        }\n    }\n}\n\n#[derive(Debug)]\n/// The interface to return socket-related information\npub struct SocketDigest {\n    #[cfg(unix)]\n    raw_fd: std::os::unix::io::RawFd,\n    #[cfg(windows)]\n    raw_sock: std::os::windows::io::RawSocket,\n    /// Remote socket address\n    pub peer_addr: OnceCell<Option<SocketAddr>>,\n    /// Local socket address\n    pub local_addr: OnceCell<Option<SocketAddr>>,\n    /// Original destination address\n    pub original_dst: OnceCell<Option<SocketAddr>>,\n}\n\nimpl SocketDigest {\n    #[cfg(unix)]\n    pub fn from_raw_fd(raw_fd: std::os::unix::io::RawFd) -> SocketDigest {\n        SocketDigest {\n            raw_fd,\n            peer_addr: OnceCell::new(),\n            local_addr: OnceCell::new(),\n            original_dst: OnceCell::new(),\n        }\n    }\n\n    #[cfg(windows)]\n    pub fn from_raw_socket(raw_sock: std::os::windows::io::RawSocket) -> SocketDigest {\n        SocketDigest {\n            raw_sock,\n            peer_addr: OnceCell::new(),\n            local_addr: OnceCell::new(),\n            original_dst: OnceCell::new(),\n        }\n    }\n\n    #[cfg(unix)]\n    pub fn peer_addr(&self) -> Option<&SocketAddr> {\n        self.peer_addr\n            .get_or_init(|| SocketAddr::from_raw_fd(self.raw_fd, true))\n            .as_ref()\n    }\n\n    #[cfg(windows)]\n    pub fn peer_addr(&self) -> Option<&SocketAddr> {\n        self.peer_addr\n            .get_or_init(|| SocketAddr::from_raw_socket(self.raw_sock, true))\n            .as_ref()\n    }\n\n    #[cfg(unix)]\n    pub fn local_addr(&self) -> Option<&SocketAddr> {\n        self.local_addr\n            .get_or_init(|| SocketAddr::from_raw_fd(self.raw_fd, false))\n            .as_ref()\n    }\n\n    #[cfg(windows)]\n    pub fn local_addr(&self) -> Option<&SocketAddr> {\n        self.local_addr\n            .get_or_init(|| SocketAddr::from_raw_socket(self.raw_sock, false))\n            .as_ref()\n    }\n\n    fn is_inet(&self) -> bool {\n        self.local_addr().and_then(|p| p.as_inet()).is_some()\n    }\n\n    #[cfg(unix)]\n    pub fn tcp_info(&self) -> Option<TCP_INFO> {\n        if self.is_inet() {\n            get_tcp_info(self.raw_fd).ok()\n        } else {\n            None\n        }\n    }\n\n    #[cfg(windows)]\n    pub fn tcp_info(&self) -> Option<TCP_INFO> {\n        if self.is_inet() {\n            get_tcp_info(self.raw_sock).ok()\n        } else {\n            None\n        }\n    }\n\n    #[cfg(unix)]\n    pub fn get_recv_buf(&self) -> Option<usize> {\n        if self.is_inet() {\n            get_recv_buf(self.raw_fd).ok()\n        } else {\n            None\n        }\n    }\n\n    #[cfg(windows)]\n    pub fn get_recv_buf(&self) -> Option<usize> {\n        if self.is_inet() {\n            get_recv_buf(self.raw_sock).ok()\n        } else {\n            None\n        }\n    }\n\n    #[cfg(unix)]\n    pub fn get_snd_buf(&self) -> Option<usize> {\n        if self.is_inet() {\n            get_snd_buf(self.raw_fd).ok()\n        } else {\n            None\n        }\n    }\n\n    #[cfg(windows)]\n    pub fn get_snd_buf(&self) -> Option<usize> {\n        if self.is_inet() {\n            get_snd_buf(self.raw_sock).ok()\n        } else {\n            None\n        }\n    }\n\n    #[cfg(unix)]\n    pub fn original_dst(&self) -> Option<&SocketAddr> {\n        self.original_dst\n            .get_or_init(|| {\n                get_original_dest(self.raw_fd)\n                    .ok()\n                    .flatten()\n                    .map(SocketAddr::Inet)\n            })\n            .as_ref()\n    }\n\n    #[cfg(windows)]\n    pub fn original_dst(&self) -> Option<&SocketAddr> {\n        self.original_dst\n            .get_or_init(|| {\n                get_original_dest(self.raw_sock)\n                    .ok()\n                    .flatten()\n                    .map(SocketAddr::Inet)\n            })\n            .as_ref()\n    }\n}\n\n/// The interface to return timing information\npub trait GetTimingDigest {\n    /// Return the timing for each layer from the lowest layer to upper\n    fn get_timing_digest(&self) -> Vec<Option<TimingDigest>>;\n    fn get_read_pending_time(&self) -> Duration {\n        Duration::ZERO\n    }\n    fn get_write_pending_time(&self) -> Duration {\n        Duration::ZERO\n    }\n}\n\n/// The interface to set or return proxy information\npub trait GetProxyDigest {\n    fn get_proxy_digest(&self) -> Option<Arc<ProxyDigest>>;\n    fn set_proxy_digest(&mut self, _digest: ProxyDigest) {}\n}\n\n/// The interface to set or return socket information\npub trait GetSocketDigest {\n    fn get_socket_digest(&self) -> Option<Arc<SocketDigest>>;\n    fn set_socket_digest(&mut self, _socket_digest: SocketDigest) {}\n}\n"
  },
  {
    "path": "pingora-core/src/protocols/http/body_buffer.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\nuse bytes::{Bytes, BytesMut};\n\n/// A buffer with size limit. When the total amount of data written to the buffer is below the limit\n/// all the data will be held in the buffer. Otherwise, the buffer will report to be truncated.\npub struct FixedBuffer {\n    buffer: BytesMut,\n    capacity: usize,\n    truncated: bool,\n}\n\nimpl FixedBuffer {\n    pub fn new(capacity: usize) -> Self {\n        FixedBuffer {\n            buffer: BytesMut::new(),\n            capacity,\n            truncated: false,\n        }\n    }\n\n    // TODO: maybe store a Vec of Bytes for zero-copy\n    pub fn write_to_buffer(&mut self, data: &Bytes) {\n        if !self.truncated && (self.buffer.len() + data.len() <= self.capacity) {\n            self.buffer.extend_from_slice(data);\n        } else {\n            // TODO: clear data because the data held here is useless anyway?\n            self.truncated = true;\n        }\n    }\n    pub fn clear(&mut self) {\n        self.truncated = false;\n        self.buffer.clear();\n    }\n    pub fn is_empty(&self) -> bool {\n        self.buffer.len() == 0\n    }\n    pub fn is_truncated(&self) -> bool {\n        self.truncated\n    }\n    pub fn get_buffer(&self) -> Option<Bytes> {\n        // TODO: return None if truncated?\n        if !self.is_empty() {\n            Some(self.buffer.clone().freeze())\n        } else {\n            None\n        }\n    }\n}\n"
  },
  {
    "path": "pingora-core/src/protocols/http/bridge/grpc_web.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\nuse bytes::{BufMut, Bytes, BytesMut};\nuse http::{\n    header::{CONTENT_LENGTH, CONTENT_TYPE, TRANSFER_ENCODING},\n    HeaderMap,\n};\nuse pingora_error::{ErrorType::ReadError, OrErr, Result};\nuse pingora_http::{RequestHeader, ResponseHeader};\n\n/// Used for bridging gRPC to gRPC-web and vice-versa.\n/// See gRPC-web [spec](https://github.com/grpc/grpc/blob/master/doc/PROTOCOL-WEB.md) and\n/// gRPC h2 [spec](https://github.com/grpc/grpc/blob/master/doc/PROTOCOL-HTTP2.md) for more details.\n#[derive(Default, PartialEq, Debug)]\npub enum GrpcWebCtx {\n    #[default]\n    Disabled,\n    Init,\n    Upgrade,\n    Trailers,\n    Done,\n}\n\nconst GRPC: &str = \"application/grpc\";\nconst GRPC_WEB: &str = \"application/grpc-web\";\n\nimpl GrpcWebCtx {\n    pub fn init(&mut self) {\n        *self = Self::Init;\n    }\n\n    /// gRPC-web request is fed into this filter, if the module is initialized\n    /// we attempt to convert it to a gRPC request\n    pub fn request_header_filter(&mut self, req: &mut RequestHeader) {\n        if *self != Self::Init {\n            // not enabled\n            return;\n        }\n\n        let content_type = req\n            .headers\n            .get(CONTENT_TYPE)\n            .and_then(|v| v.to_str().ok())\n            .unwrap_or_default();\n\n        // check we have a valid grpc-web prefix\n        if !(content_type.len() >= GRPC_WEB.len()\n            && content_type[..GRPC_WEB.len()].eq_ignore_ascii_case(GRPC_WEB))\n        {\n            // not gRPC-web\n            return;\n        }\n\n        // change content type to grpc\n        let ct = content_type.to_lowercase().replace(GRPC_WEB, GRPC);\n        req.insert_header(CONTENT_TYPE, ct).expect(\"insert header\");\n\n        // The 'te' request header is used to detect incompatible proxies\n        // which are supposed to remove 'te' if it is unsupported.\n        // This header is required by gRPC over h2 protocol.\n        // https://github.com/grpc/grpc/blob/master/doc/PROTOCOL-HTTP2.md\n        req.insert_header(\"te\", \"trailers\").expect(\"insert header\");\n\n        // For gRPC requests, EOS (end-of-stream) is indicated by the presence of the\n        // END_STREAM flag on the last received DATA frame.\n        // In scenarios where the Request stream needs to be closed\n        // but no data remains to be sent implementations\n        // MUST send an empty DATA frame with this flag set.\n        req.set_send_end_stream(false);\n\n        *self = Self::Upgrade\n    }\n\n    /// gRPC response is fed into this filter, if the module is in the bridge state\n    /// attempt to convert the response it to a gRPC-web response\n    pub fn response_header_filter(&mut self, resp: &mut ResponseHeader) {\n        if *self != Self::Upgrade {\n            // not an upgrade\n            return;\n        }\n\n        if resp.status.is_informational() {\n            // proxy informational statuses through\n            return;\n        }\n\n        let content_type = resp\n            .headers\n            .get(CONTENT_TYPE)\n            .and_then(|v| v.to_str().ok())\n            .unwrap_or_default();\n\n        // upstream h2, no reason to normalize case\n        if !content_type.starts_with(GRPC) {\n            // not gRPC\n            *self = Self::Disabled;\n            return;\n        }\n\n        // change content type to gRPC-web\n        let ct = content_type.replace(GRPC, GRPC_WEB);\n        resp.insert_header(CONTENT_TYPE, ct).expect(\"insert header\");\n\n        // always use chunked for gRPC-web\n        resp.remove_header(&CONTENT_LENGTH);\n        resp.insert_header(TRANSFER_ENCODING, \"chunked\")\n            .expect(\"insert header\");\n\n        *self = Self::Trailers\n    }\n\n    /// Used to convert gRPC trailers into gRPC-web trailers, note\n    /// gRPC-web trailers are encoded into the response body so we return\n    /// the encoded bytes here.\n    pub fn response_trailer_filter(\n        &mut self,\n        resp_trailers: &mut HeaderMap,\n    ) -> Result<Option<Bytes>> {\n        /* Trailer header frame and trailer headers\n            0 - - 1 - - 2 - - 3 - - 4 - - 5 - - 6 - - 7 - - 8\n            | Ind |        Length         |     Headers     | <- trailer header indicator, length of headers\n            |                    Headers                    | <- rest is headers\n            |                    Headers                    |\n        */\n        // TODO compressed trailer?\n        // grpc-web trailers frame head\n        const GRPC_WEB_TRAILER: u8 = 0x80;\n\n        // number of bytes in trailer header\n        const GRPC_TRAILER_HEADER_LEN: usize = 5;\n\n        // just some estimate\n        const DEFAULT_TRAILER_BUFFER_SIZE: usize = 256;\n\n        if *self != Self::Trailers {\n            // not an upgrade\n            *self = Self::Disabled;\n            return Ok(None);\n        }\n\n        // trailers are expected to arrive all at once encoded into a single trailers frame\n        // trailers in frame are separated by CRLFs\n        let mut buf = BytesMut::with_capacity(DEFAULT_TRAILER_BUFFER_SIZE);\n        let mut trailers = buf.split_off(GRPC_TRAILER_HEADER_LEN);\n\n        // iterate the key/value pairs and encode them into the tmp buffer\n        for (key, value) in resp_trailers.iter() {\n            // encode header\n            trailers.put_slice(key.as_ref());\n            trailers.put_slice(b\":\");\n\n            // encode value\n            trailers.put_slice(value.as_ref());\n\n            // encode header separator\n            trailers.put_slice(b\"\\r\\n\");\n        }\n\n        // ensure trailer length within u32\n        let len = trailers.len().try_into().or_err_with(ReadError, || {\n            format!(\"invalid gRPC trailer length: {}\", trailers.len())\n        })?;\n        buf.put_u8(GRPC_WEB_TRAILER);\n        buf.put_u32(len);\n        buf.unsplit(trailers);\n\n        *self = Self::Done;\n        Ok(Some(buf.freeze()))\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use super::*;\n    use http::{request::Request, response::Response, Version};\n\n    #[test]\n    fn non_grpc_web_request_ignored() {\n        let request = Request::get(\"https://pingora.dev/\")\n            .header(CONTENT_TYPE, \"application/grpc-we\")\n            .version(Version::HTTP_2) // only set this to verify send_end_stream is configured\n            .body(())\n            .unwrap();\n        let mut request = request.into_parts().0.into();\n\n        let mut filter = GrpcWebCtx::default();\n        filter.init();\n        filter.request_header_filter(&mut request);\n        assert_eq!(filter, GrpcWebCtx::Init);\n\n        let headers = &request.headers;\n        assert_eq!(headers.get(\"te\"), None);\n        assert_eq!(headers.get(\"application/grpc\"), None);\n        assert_eq!(request.send_end_stream(), Some(true));\n    }\n\n    #[test]\n    fn grpc_web_request_module_disabled_ignored() {\n        let request = Request::get(\"https://pingora.dev/\")\n            .header(CONTENT_TYPE, \"application/grpc-web\")\n            .version(Version::HTTP_2) // only set this to verify send_end_stream is configured\n            .body(())\n            .unwrap();\n        let mut request = request.into_parts().0.into();\n\n        // do not init\n        let mut filter = GrpcWebCtx::default();\n        filter.request_header_filter(&mut request);\n        assert_eq!(filter, GrpcWebCtx::Disabled);\n\n        let headers = &request.headers;\n        assert_eq!(headers.get(\"te\"), None);\n        assert_eq!(headers.get(CONTENT_TYPE).unwrap(), \"application/grpc-web\");\n        assert_eq!(request.send_end_stream(), Some(true));\n    }\n\n    #[test]\n    fn grpc_web_request_upgrade() {\n        let request = Request::get(\"https://pingora.org/\")\n            .header(CONTENT_TYPE, \"application/gRPC-web+thrift\")\n            .version(Version::HTTP_2) // only set this to verify send_end_stream is configured\n            .body(())\n            .unwrap();\n        let mut request = request.into_parts().0.into();\n\n        let mut filter = GrpcWebCtx::default();\n        filter.init();\n        filter.request_header_filter(&mut request);\n        assert_eq!(filter, GrpcWebCtx::Upgrade);\n\n        let headers = &request.headers;\n        assert_eq!(headers.get(\"te\").unwrap(), \"trailers\");\n        assert_eq!(\n            headers.get(CONTENT_TYPE).unwrap(),\n            \"application/grpc+thrift\"\n        );\n        assert_eq!(request.send_end_stream(), Some(false));\n    }\n\n    #[test]\n    fn non_grpc_response_ignored() {\n        let response = Response::builder()\n            .header(CONTENT_TYPE, \"text/html\")\n            .header(CONTENT_LENGTH, \"10\")\n            .body(())\n            .unwrap();\n        let mut response = response.into_parts().0.into();\n\n        let mut filter = GrpcWebCtx::Upgrade;\n        filter.response_header_filter(&mut response);\n        assert_eq!(filter, GrpcWebCtx::Disabled);\n\n        let headers = &response.headers;\n        assert_eq!(headers.get(CONTENT_TYPE).unwrap(), \"text/html\");\n        assert_eq!(headers.get(CONTENT_LENGTH).unwrap(), \"10\");\n    }\n\n    #[test]\n    fn grpc_response_module_disabled_ignored() {\n        let response = Response::builder()\n            .header(CONTENT_TYPE, \"application/grpc\")\n            .body(())\n            .unwrap();\n        let mut response = response.into_parts().0.into();\n\n        let mut filter = GrpcWebCtx::default();\n        filter.response_header_filter(&mut response);\n        assert_eq!(filter, GrpcWebCtx::Disabled);\n\n        let headers = &response.headers;\n        assert_eq!(headers.get(CONTENT_TYPE).unwrap(), \"application/grpc\");\n    }\n\n    #[test]\n    fn grpc_response_upgrade() {\n        let response = Response::builder()\n            .header(CONTENT_TYPE, \"application/grpc+proto\")\n            .header(CONTENT_LENGTH, \"0\")\n            .body(())\n            .unwrap();\n        let mut response = response.into_parts().0.into();\n\n        let mut filter = GrpcWebCtx::Upgrade;\n        filter.response_header_filter(&mut response);\n        assert_eq!(filter, GrpcWebCtx::Trailers);\n\n        let headers = &response.headers;\n        assert_eq!(\n            headers.get(CONTENT_TYPE).unwrap(),\n            \"application/grpc-web+proto\"\n        );\n        assert_eq!(headers.get(TRANSFER_ENCODING).unwrap(), \"chunked\");\n        assert!(headers.get(CONTENT_LENGTH).is_none());\n    }\n\n    #[test]\n    fn grpc_response_informational_proxied() {\n        let response = Response::builder().status(100).body(()).unwrap();\n        let mut response = response.into_parts().0.into();\n\n        let mut filter = GrpcWebCtx::Upgrade;\n        filter.response_header_filter(&mut response);\n        assert_eq!(filter, GrpcWebCtx::Upgrade); // still upgrade\n    }\n\n    #[test]\n    fn grpc_response_trailer_headers_convert_to_byte_buf() {\n        let mut response = Response::builder()\n            .header(\"grpc-status\", \"0\")\n            .header(\"grpc-message\", \"OK\")\n            .body(())\n            .unwrap();\n        let response = response.headers_mut();\n\n        let mut filter = GrpcWebCtx::Trailers;\n        let buf = filter.response_trailer_filter(response).unwrap().unwrap();\n        assert_eq!(filter, GrpcWebCtx::Done);\n\n        let expected = b\"grpc-status:0\\r\\ngrpc-message:OK\\r\\n\";\n        let expected_len: u32 = expected.len() as u32; // 32 bytes\n\n        // assert the length prefix message frame\n        // [1 byte (header)| 4 byte (length) | 15 byte (grpc-status:0\\r\\n) | 17 bytes (grpc-message:OK\\r\\n)]\n        assert_eq!(0x80, buf[0]); // frame should start with trailer header\n        assert_eq!(expected_len.to_be_bytes(), buf[1..5]); // next 4 bytes length of trailer\n        assert_eq!(expected[..15], buf[5..20]); // grpc-status:0\\r\\n (15 bytes)\n        assert_eq!(expected[15..], buf[20..]); // grpc-message:OK\\r\\n (17 bytes)\n    }\n}\n"
  },
  {
    "path": "pingora-core/src/protocols/http/bridge/mod.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\npub mod grpc_web;\n"
  },
  {
    "path": "pingora-core/src/protocols/http/client.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\nuse bytes::Bytes;\nuse pingora_error::Result;\nuse pingora_http::{RequestHeader, ResponseHeader};\nuse std::time::Duration;\n\nuse super::v2::client::Http2Session;\nuse super::{custom::client::Session, v1::client::HttpSession as Http1Session};\nuse crate::protocols::{Digest, SocketAddr, Stream};\n\n/// A type for Http client session. It can be either an Http1 connection or an Http2 stream.\npub enum HttpSession<S = ()> {\n    H1(Http1Session),\n    H2(Http2Session),\n    Custom(S),\n}\n\nimpl<S: Session> HttpSession<S> {\n    pub fn as_http1(&self) -> Option<&Http1Session> {\n        match self {\n            Self::H1(s) => Some(s),\n            Self::H2(_) => None,\n            Self::Custom(_) => None,\n        }\n    }\n\n    pub fn as_http2(&self) -> Option<&Http2Session> {\n        match self {\n            Self::H1(_) => None,\n            Self::H2(s) => Some(s),\n            Self::Custom(_) => None,\n        }\n    }\n\n    pub fn as_custom(&self) -> Option<&S> {\n        match self {\n            Self::H1(_) => None,\n            Self::H2(_) => None,\n            Self::Custom(c) => Some(c),\n        }\n    }\n\n    pub fn as_custom_mut(&mut self) -> Option<&mut S> {\n        match self {\n            Self::H1(_) => None,\n            Self::H2(_) => None,\n            Self::Custom(c) => Some(c),\n        }\n    }\n\n    /// Write the request header to the server\n    /// After the request header is sent. The caller can either start reading the response or\n    /// sending request body if any.\n    pub async fn write_request_header(&mut self, req: Box<RequestHeader>) -> Result<()> {\n        match self {\n            HttpSession::H1(h1) => {\n                h1.write_request_header(req).await?;\n                Ok(())\n            }\n            HttpSession::H2(h2) => h2.write_request_header(req, false),\n            HttpSession::Custom(c) => c.write_request_header(req, false).await,\n        }\n    }\n\n    /// Write a chunk of the request body.\n    pub async fn write_request_body(&mut self, data: Bytes, end: bool) -> Result<()> {\n        match self {\n            HttpSession::H1(h1) => {\n                // TODO: maybe h1 should also have the concept of `end`\n                h1.write_body(&data).await?;\n                Ok(())\n            }\n            HttpSession::H2(h2) => h2.write_request_body(data, end).await,\n            HttpSession::Custom(c) => c.write_request_body(data, end).await,\n        }\n    }\n\n    /// Signal that the request body has ended\n    pub async fn finish_request_body(&mut self) -> Result<()> {\n        match self {\n            HttpSession::H1(h1) => {\n                h1.finish_body().await?;\n                Ok(())\n            }\n            HttpSession::H2(h2) => h2.finish_request_body(),\n            HttpSession::Custom(c) => c.finish_request_body().await,\n        }\n    }\n\n    /// Set the read timeout for reading header and body.\n    ///\n    /// The timeout is per read operation, not on the overall time reading the entire response\n    pub fn set_read_timeout(&mut self, timeout: Option<Duration>) {\n        match self {\n            HttpSession::H1(h1) => h1.read_timeout = timeout,\n            HttpSession::H2(h2) => h2.read_timeout = timeout,\n            HttpSession::Custom(c) => c.set_read_timeout(timeout),\n        }\n    }\n\n    /// Set the write timeout for writing header and body.\n    ///\n    /// The timeout is per write operation, not on the overall time writing the entire request.\n    pub fn set_write_timeout(&mut self, timeout: Option<Duration>) {\n        match self {\n            HttpSession::H1(h1) => h1.write_timeout = timeout,\n            HttpSession::H2(h2) => h2.write_timeout = timeout,\n            HttpSession::Custom(c) => c.set_write_timeout(timeout),\n        }\n    }\n\n    /// Read the response header from the server\n    /// For http1, this function can be called multiple times, if the headers received are just\n    /// informational headers.\n    pub async fn read_response_header(&mut self) -> Result<()> {\n        match self {\n            HttpSession::H1(h1) => {\n                h1.read_response().await?;\n                Ok(())\n            }\n            HttpSession::H2(h2) => h2.read_response_header().await,\n            HttpSession::Custom(c) => c.read_response_header().await,\n        }\n    }\n\n    /// Read response body\n    ///\n    /// `None` when no more body to read.\n    pub async fn read_response_body(&mut self) -> Result<Option<Bytes>> {\n        match self {\n            HttpSession::H1(h1) => h1.read_body_bytes().await,\n            HttpSession::H2(h2) => h2.read_response_body().await,\n            HttpSession::Custom(c) => c.read_response_body().await,\n        }\n    }\n\n    /// No (more) body to read\n    pub fn response_done(&mut self) -> bool {\n        match self {\n            HttpSession::H1(h1) => h1.is_body_done(),\n            HttpSession::H2(h2) => h2.response_finished(),\n            HttpSession::Custom(c) => c.response_finished(),\n        }\n    }\n\n    /// Give up the http session abruptly.\n    /// For H1 this will close the underlying connection\n    /// For H2 this will send RST_STREAM frame to end this stream if the stream has not ended at all\n    pub async fn shutdown(&mut self) {\n        match self {\n            Self::H1(s) => s.shutdown().await,\n            Self::H2(s) => s.shutdown(),\n            Self::Custom(c) => c.shutdown(0, \"shutdown\").await,\n        }\n    }\n\n    /// Get the response header of the server\n    ///\n    /// `None` if the response header is not read yet.\n    pub fn response_header(&self) -> Option<&ResponseHeader> {\n        match self {\n            Self::H1(s) => s.resp_header(),\n            Self::H2(s) => s.response_header(),\n            Self::Custom(c) => c.response_header(),\n        }\n    }\n\n    /// Return the [Digest] of the connection\n    ///\n    /// For reused connection, the timing in the digest will reflect its initial handshakes\n    /// The caller should check if the connection is reused to avoid misuse of the timing field.\n    pub fn digest(&self) -> Option<&Digest> {\n        match self {\n            Self::H1(s) => Some(s.digest()),\n            Self::H2(s) => s.digest(),\n            Self::Custom(c) => c.digest(),\n        }\n    }\n\n    /// Return a mutable [Digest] reference for the connection.\n    ///\n    /// Will return `None` if this is an H2 session and multiple streams are open.\n    pub fn digest_mut(&mut self) -> Option<&mut Digest> {\n        match self {\n            Self::H1(s) => Some(s.digest_mut()),\n            Self::H2(s) => s.digest_mut(),\n            Self::Custom(s) => s.digest_mut(),\n        }\n    }\n\n    /// Return the server (peer) address of the connection.\n    pub fn server_addr(&self) -> Option<&SocketAddr> {\n        match self {\n            Self::H1(s) => s.server_addr(),\n            Self::H2(s) => s.server_addr(),\n            Self::Custom(s) => s.server_addr(),\n        }\n    }\n\n    /// Return the client (local) address of the connection.\n    pub fn client_addr(&self) -> Option<&SocketAddr> {\n        match self {\n            Self::H1(s) => s.client_addr(),\n            Self::H2(s) => s.client_addr(),\n            Self::Custom(s) => s.client_addr(),\n        }\n    }\n\n    /// Get the reference of the [Stream] that this HTTP/1 session is operating upon.\n    /// None if the HTTP session is over H2\n    pub fn stream(&self) -> Option<&Stream> {\n        match self {\n            Self::H1(s) => Some(s.stream()),\n            Self::H2(_) => None,\n            Self::Custom(_) => None,\n        }\n    }\n}\n"
  },
  {
    "path": "pingora-core/src/protocols/http/compression/brotli.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\nuse super::Encode;\nuse super::COMPRESSION_ERROR;\n\nuse brotli::{CompressorWriter, DecompressorWriter};\nuse bytes::Bytes;\nuse pingora_error::{OrErr, Result};\nuse std::io::Write;\nuse std::time::{Duration, Instant};\n\npub struct Decompressor {\n    decompress: DecompressorWriter<Vec<u8>>,\n    total_in: usize,\n    total_out: usize,\n    duration: Duration,\n}\n\nimpl Decompressor {\n    pub fn new() -> Self {\n        Decompressor {\n            // default buf is 4096 if 0 is used, TODO: figure out the significance of this value\n            decompress: DecompressorWriter::new(vec![], 0),\n            total_in: 0,\n            total_out: 0,\n            duration: Duration::new(0, 0),\n        }\n    }\n}\n\nimpl Encode for Decompressor {\n    fn encode(&mut self, input: &[u8], end: bool) -> Result<Bytes> {\n        const MAX_INIT_COMPRESSED_SIZE_CAP: usize = 4 * 1024;\n        // Brotli compress ratio can be 3.5 to 4.5\n        const ESTIMATED_COMPRESSION_RATIO: usize = 4;\n        let start = Instant::now();\n        self.total_in += input.len();\n        // cap the buf size amplification, there is a DoS risk of always allocate\n        // 4x the memory of the input buffer\n        let reserve_size = if input.len() < MAX_INIT_COMPRESSED_SIZE_CAP {\n            input.len() * ESTIMATED_COMPRESSION_RATIO\n        } else {\n            input.len()\n        };\n        self.decompress.get_mut().reserve(reserve_size);\n        self.decompress\n            .write_all(input)\n            .or_err(COMPRESSION_ERROR, \"while decompress Brotli\")?;\n        // write to vec will never fail. The only possible error is that the input data\n        // is invalid (not brotli compressed)\n        if end {\n            self.decompress\n                .flush()\n                .or_err(COMPRESSION_ERROR, \"while decompress Brotli\")?;\n        }\n        self.total_out += self.decompress.get_ref().len();\n        self.duration += start.elapsed();\n        Ok(std::mem::take(self.decompress.get_mut()).into()) // into() Bytes will drop excess capacity\n    }\n\n    fn stat(&self) -> (&'static str, usize, usize, Duration) {\n        (\"de-brotli\", self.total_in, self.total_out, self.duration)\n    }\n}\n\npub struct Compressor {\n    compress: CompressorWriter<Vec<u8>>,\n    total_in: usize,\n    total_out: usize,\n    duration: Duration,\n}\n\nimpl Compressor {\n    pub fn new(level: u32) -> Self {\n        Compressor {\n            // buf_size:4096 , lgwin:19 TODO: fine tune these\n            compress: CompressorWriter::new(vec![], 4096, level, 19),\n            total_in: 0,\n            total_out: 0,\n            duration: Duration::new(0, 0),\n        }\n    }\n}\n\nimpl Encode for Compressor {\n    fn encode(&mut self, input: &[u8], end: bool) -> Result<Bytes> {\n        // reserve at most 16k\n        const MAX_INIT_COMPRESSED_BUF_SIZE: usize = 16 * 1024;\n        let start = Instant::now();\n        self.total_in += input.len();\n\n        // reserve at most input size, cap at 16k, compressed output should be smaller\n        self.compress\n            .get_mut()\n            .reserve(std::cmp::min(MAX_INIT_COMPRESSED_BUF_SIZE, input.len()));\n        self.compress\n            .write_all(input)\n            .or_err(COMPRESSION_ERROR, \"while compress Brotli\")?;\n        // write to vec will never fail.\n        if end {\n            self.compress\n                .flush()\n                .or_err(COMPRESSION_ERROR, \"while compress Brotli\")?;\n        }\n        self.total_out += self.compress.get_ref().len();\n        self.duration += start.elapsed();\n        Ok(std::mem::take(self.compress.get_mut()).into()) // into() Bytes will drop excess capacity\n    }\n\n    fn stat(&self) -> (&'static str, usize, usize, Duration) {\n        (\"brotli\", self.total_in, self.total_out, self.duration)\n    }\n}\n\n#[cfg(test)]\nmod tests_stream {\n    use super::*;\n\n    #[test]\n    fn decompress_brotli_data() {\n        let mut compressor = Decompressor::new();\n        let decompressed = compressor\n            .encode(\n                &[\n                    0x1f, 0x0f, 0x00, 0xf8, 0x45, 0x07, 0x87, 0x3e, 0x10, 0xfb, 0x55, 0x92, 0xec,\n                    0x12, 0x09, 0xcc, 0x38, 0xdd, 0x51, 0x1e,\n                ],\n                true,\n            )\n            .unwrap();\n\n        assert_eq!(&decompressed[..], &b\"adcdefgabcdefgh\\n\"[..]);\n    }\n\n    #[test]\n    fn compress_brotli_data() {\n        let mut compressor = Compressor::new(11);\n        let compressed = compressor.encode(&b\"adcdefgabcdefgh\\n\"[..], true).unwrap();\n\n        assert_eq!(\n            &compressed[..],\n            &[\n                0x85, 0x07, 0x00, 0xf8, 0x45, 0x07, 0x87, 0x3e, 0x10, 0xfb, 0x55, 0x92, 0xec, 0x12,\n                0x09, 0xcc, 0x38, 0xdd, 0x51, 0x1e,\n            ],\n        );\n    }\n}\n"
  },
  {
    "path": "pingora-core/src/protocols/http/compression/gzip.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\nuse super::{Encode, COMPRESSION_ERROR};\n\nuse bytes::Bytes;\nuse flate2::write::{GzDecoder, GzEncoder};\nuse pingora_error::{OrErr, Result};\nuse std::io::Write;\nuse std::time::{Duration, Instant};\n\npub struct Decompressor {\n    decompress: GzDecoder<Vec<u8>>,\n    total_in: usize,\n    total_out: usize,\n    duration: Duration,\n}\n\nimpl Decompressor {\n    pub fn new() -> Self {\n        Decompressor {\n            decompress: GzDecoder::new(vec![]),\n            total_in: 0,\n            total_out: 0,\n            duration: Duration::new(0, 0),\n        }\n    }\n}\n\nimpl Encode for Decompressor {\n    fn encode(&mut self, input: &[u8], end: bool) -> Result<Bytes> {\n        const MAX_INIT_COMPRESSED_SIZE_CAP: usize = 4 * 1024;\n        const ESTIMATED_COMPRESSION_RATIO: usize = 3; // estimated 2.5-3x compression\n        let start = Instant::now();\n        self.total_in += input.len();\n        // cap the buf size amplification, there is a DoS risk of always allocate\n        // 3x the memory of the input buffer\n        let reserve_size = if input.len() < MAX_INIT_COMPRESSED_SIZE_CAP {\n            input.len() * ESTIMATED_COMPRESSION_RATIO\n        } else {\n            input.len()\n        };\n        self.decompress.get_mut().reserve(reserve_size);\n        self.decompress\n            .write_all(input)\n            .or_err(COMPRESSION_ERROR, \"while decompress Gzip\")?;\n        // write to vec will never fail, only possible error is that the input data\n        // was not actually gzip compressed\n        if end {\n            self.decompress\n                .try_finish()\n                .or_err(COMPRESSION_ERROR, \"while decompress Gzip\")?;\n        }\n        self.total_out += self.decompress.get_ref().len();\n        self.duration += start.elapsed();\n        Ok(std::mem::take(self.decompress.get_mut()).into()) // into() Bytes will drop excess capacity\n    }\n\n    fn stat(&self) -> (&'static str, usize, usize, Duration) {\n        (\"de-gzip\", self.total_in, self.total_out, self.duration)\n    }\n}\n\npub struct Compressor {\n    // TODO: enum for other compression algorithms\n    compress: GzEncoder<Vec<u8>>,\n    total_in: usize,\n    total_out: usize,\n    duration: Duration,\n}\n\nimpl Compressor {\n    pub fn new(level: u32) -> Compressor {\n        Compressor {\n            compress: GzEncoder::new(vec![], flate2::Compression::new(level)),\n            total_in: 0,\n            total_out: 0,\n            duration: Duration::new(0, 0),\n        }\n    }\n}\n\nimpl Encode for Compressor {\n    // infallible because compression can take any data\n    fn encode(&mut self, input: &[u8], end: bool) -> Result<Bytes> {\n        // reserve at most 16k\n        const MAX_INIT_COMPRESSED_BUF_SIZE: usize = 16 * 1024;\n        let start = Instant::now();\n        self.total_in += input.len();\n        self.compress\n            .get_mut()\n            .reserve(std::cmp::min(MAX_INIT_COMPRESSED_BUF_SIZE, input.len()));\n        self.write_all(input).unwrap(); // write to vec, should never fail\n        if end {\n            self.try_finish().unwrap(); // write to vec, should never fail\n        }\n        self.total_out += self.compress.get_ref().len();\n        self.duration += start.elapsed();\n        Ok(std::mem::take(self.compress.get_mut()).into()) // into() Bytes will drop excess capacity\n    }\n\n    fn stat(&self) -> (&'static str, usize, usize, Duration) {\n        (\"gzip\", self.total_in, self.total_out, self.duration)\n    }\n}\n\nuse std::ops::{Deref, DerefMut};\nimpl Deref for Decompressor {\n    type Target = GzDecoder<Vec<u8>>;\n\n    fn deref(&self) -> &Self::Target {\n        &self.decompress\n    }\n}\n\nimpl DerefMut for Decompressor {\n    fn deref_mut(&mut self) -> &mut Self::Target {\n        &mut self.decompress\n    }\n}\n\nimpl Deref for Compressor {\n    type Target = GzEncoder<Vec<u8>>;\n\n    fn deref(&self) -> &Self::Target {\n        &self.compress\n    }\n}\n\nimpl DerefMut for Compressor {\n    fn deref_mut(&mut self) -> &mut Self::Target {\n        &mut self.compress\n    }\n}\n\n#[cfg(test)]\nmod tests_stream {\n    use super::*;\n\n    #[test]\n    fn gzip_data() {\n        let mut compressor = Compressor::new(6);\n        let compressed = compressor.encode(b\"abcdefg\", true).unwrap();\n        // gzip magic headers\n        assert_eq!(&compressed[..3], &[0x1f, 0x8b, 0x08]);\n        // check the crc32 footer\n        assert_eq!(\n            &compressed[compressed.len() - 9..],\n            &[0, 166, 106, 42, 49, 7, 0, 0, 0]\n        );\n        assert_eq!(compressor.total_in, 7);\n        assert_eq!(compressor.total_out, compressed.len());\n\n        assert!(compressor.get_ref().is_empty());\n    }\n\n    #[test]\n    fn gunzip_data() {\n        let mut decompressor = Decompressor::new();\n\n        let compressed_bytes = &[\n            0x1f, 0x8b, 0x08, 0, 0, 0, 0, 0, 0, 255, 75, 76, 74, 78, 73, 77, 75, 7, 0, 166, 106,\n            42, 49, 7, 0, 0, 0,\n        ];\n        let decompressed = decompressor.encode(compressed_bytes, true).unwrap();\n\n        assert_eq!(&decompressed[..], b\"abcdefg\");\n        assert_eq!(decompressor.total_in, compressed_bytes.len());\n        assert_eq!(decompressor.total_out, decompressed.len());\n\n        assert!(decompressor.get_ref().is_empty());\n    }\n}\n"
  },
  {
    "path": "pingora-core/src/protocols/http/compression/mod.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! HTTP response (de)compression libraries\n//!\n//! Gzip, Brotli, Zstd, and dictionary-compressed Zstd (dcz, [RFC 9842](https://datatracker.ietf.org/doc/html/rfc9842)) are supported.\n\nuse super::HttpTask;\n\nuse bytes::Bytes;\nuse log::{debug, warn};\nuse pingora_error::{ErrorType, Result};\nuse pingora_http::{RequestHeader, ResponseHeader};\nuse std::time::Duration;\n\nuse strum::EnumCount;\nuse strum_macros::EnumCount as EnumCountMacro;\n\nmod brotli;\nmod gzip;\nmod zstd;\n\n/// Re-export [RFC 9842](https://datatracker.ietf.org/doc/html/rfc9842) constants for external use.\npub use zstd::{DCZ_HEADER_SIZE, DCZ_MAGIC};\n\n/// The type of error to return when (de)compression fails\npub const COMPRESSION_ERROR: ErrorType = ErrorType::new(\"CompressionError\");\n\n/// The trait for both compress and decompress because the interface and syntax are the same:\n/// encode some bytes to other bytes\npub trait Encode {\n    /// Encode the input bytes. The `end` flag signals the end of the entire input. The `end` flag\n    /// helps the encoder to flush out the remaining buffered encoded data because certain compression\n    /// algorithms prefer to collect large enough data to compress all together.\n    fn encode(&mut self, input: &[u8], end: bool) -> Result<Bytes>;\n    /// Return the Encoder's name, the total input bytes, the total output bytes and the total\n    /// duration spent on encoding the data.\n    fn stat(&self) -> (&'static str, usize, usize, Duration);\n}\n\n/// The response compression object. Currently support gzip compression and brotli decompression.\n///\n/// To use it, the caller should create a [`ResponseCompressionCtx`] per HTTP session.\n/// The caller should call the corresponding filters for the request header, response header and\n/// response body. If the algorithms are supported, the output response body will be encoded.\n/// The response header will be adjusted accordingly as well. If the algorithm is not supported\n/// or no encoding is needed, the response is untouched.\n///\n/// If configured and if the request's `accept-encoding` header contains the algorithm supported and the\n/// incoming response doesn't have that encoding, the filter will compress the response.\n/// If configured and supported, and if the incoming response's `content-encoding` isn't one of the\n/// request's `accept-encoding` supported algorithm, the ctx will decompress the response.\n///\n/// # Currently supported algorithms and actions\n/// - Brotli decompression: if the response is br compressed, this ctx can decompress it\n/// - Gzip compression: if the response is uncompressed, this ctx can compress it with gzip\npub struct ResponseCompressionCtx(CtxInner);\n\n/// Dictionary data for [RFC 9842](https://datatracker.ietf.org/doc/html/rfc9842) shared dictionary compression.\n#[derive(Clone, Debug)]\npub struct DictionaryData {\n    pub bytes: Bytes,\n    pub hash: [u8; 32],\n}\n\nenum CtxInner {\n    HeaderPhase {\n        // Store the preferred list to compare with content-encoding\n        accept_encoding: Vec<Algorithm>,\n        encoding_levels: [u32; Algorithm::COUNT],\n        decompress_enable: [bool; Algorithm::COUNT],\n        preserve_etag: [bool; Algorithm::COUNT],\n        // Optional dictionary for dcz compression (RFC 9842).\n        dictionary: Option<DictionaryData>,\n    },\n    BodyPhase(Option<Box<dyn Encode + Send + Sync>>),\n}\n\nimpl ResponseCompressionCtx {\n    /// Create a new [`ResponseCompressionCtx`] with the expected compression level. `0` will disable\n    /// the compression. The compression level is applied across all algorithms.\n    /// The `decompress_enable` flag will tell the ctx to decompress if needed.\n    /// The `preserve_etag` flag indicates whether the ctx should avoid modifying the etag,\n    /// which will otherwise be weakened if the flag is false and (de)compression is applied.\n    pub fn new(compression_level: u32, decompress_enable: bool, preserve_etag: bool) -> Self {\n        Self(CtxInner::HeaderPhase {\n            accept_encoding: Vec::new(),\n            encoding_levels: [compression_level; Algorithm::COUNT],\n            decompress_enable: [decompress_enable; Algorithm::COUNT],\n            preserve_etag: [preserve_etag; Algorithm::COUNT],\n            dictionary: None,\n        })\n    }\n\n    /// Whether the encoder is enabled.\n    /// The enablement will change according to the request and response filter by this ctx.\n    pub fn is_enabled(&self) -> bool {\n        match &self.0 {\n            CtxInner::HeaderPhase {\n                decompress_enable,\n                encoding_levels: levels,\n                ..\n            } => levels.iter().any(|l| *l != 0) || decompress_enable.iter().any(|d| *d),\n            CtxInner::BodyPhase(c) => c.is_some(),\n        }\n    }\n\n    /// Return the stat of this ctx:\n    /// algorithm name, in bytes, out bytes, time took for the compression\n    pub fn get_info(&self) -> Option<(&'static str, usize, usize, Duration)> {\n        match &self.0 {\n            CtxInner::HeaderPhase { .. } => None,\n            CtxInner::BodyPhase(c) => c.as_ref().map(|c| c.stat()),\n        }\n    }\n\n    /// Adjust the compression level for all compression algorithms.\n    /// # Panic\n    /// This function will panic if it has already started encoding the response body.\n    pub fn adjust_level(&mut self, new_level: u32) {\n        match &mut self.0 {\n            CtxInner::HeaderPhase {\n                encoding_levels: levels,\n                ..\n            } => {\n                *levels = [new_level; Algorithm::COUNT];\n            }\n            CtxInner::BodyPhase(_) => panic!(\"Wrong phase: BodyPhase\"),\n        }\n    }\n\n    /// Adjust the compression level for a specific algorithm.\n    /// # Panic\n    /// This function will panic if it has already started encoding the response body.\n    pub fn adjust_algorithm_level(&mut self, algorithm: Algorithm, new_level: u32) {\n        match &mut self.0 {\n            CtxInner::HeaderPhase {\n                encoding_levels: levels,\n                ..\n            } => {\n                levels[algorithm.index()] = new_level;\n            }\n            CtxInner::BodyPhase(_) => panic!(\"Wrong phase: BodyPhase\"),\n        }\n    }\n\n    /// Adjust the decompression flag for all compression algorithms.\n    /// # Panic\n    /// This function will panic if it has already started encoding the response body.\n    pub fn adjust_decompression(&mut self, enabled: bool) {\n        match &mut self.0 {\n            CtxInner::HeaderPhase {\n                decompress_enable, ..\n            } => {\n                *decompress_enable = [enabled; Algorithm::COUNT];\n            }\n            CtxInner::BodyPhase(_) => panic!(\"Wrong phase: BodyPhase\"),\n        }\n    }\n\n    /// Adjust the decompression flag for a specific algorithm.\n    /// # Panic\n    /// This function will panic if it has already started encoding the response body.\n    pub fn adjust_algorithm_decompression(&mut self, algorithm: Algorithm, enabled: bool) {\n        match &mut self.0 {\n            CtxInner::HeaderPhase {\n                decompress_enable, ..\n            } => {\n                decompress_enable[algorithm.index()] = enabled;\n            }\n            CtxInner::BodyPhase(_) => panic!(\"Wrong phase: BodyPhase\"),\n        }\n    }\n\n    /// Adjust preserve etag setting.\n    /// # Panic\n    /// This function will panic if it has already started encoding the response body.\n    pub fn adjust_preserve_etag(&mut self, enabled: bool) {\n        match &mut self.0 {\n            CtxInner::HeaderPhase { preserve_etag, .. } => {\n                *preserve_etag = [enabled; Algorithm::COUNT];\n            }\n            CtxInner::BodyPhase(_) => panic!(\"Wrong phase: BodyPhase\"),\n        }\n    }\n\n    /// Adjust preserve etag setting for a specific algorithm.\n    /// # Panic\n    /// This function will panic if it has already started encoding the response body.\n    pub fn adjust_algorithm_preserve_etag(&mut self, algorithm: Algorithm, enabled: bool) {\n        match &mut self.0 {\n            CtxInner::HeaderPhase { preserve_etag, .. } => {\n                preserve_etag[algorithm.index()] = enabled;\n            }\n            CtxInner::BodyPhase(_) => panic!(\"Wrong phase: BodyPhase\"),\n        }\n    }\n\n    /// Set the dictionary for [RFC 9842](https://datatracker.ietf.org/doc/html/rfc9842) dictionary compression.\n    /// # Panic\n    /// This function will panic if it has already started encoding the response body.\n    pub fn set_dictionary(&mut self, dictionary_bytes: Bytes, dictionary_hash: [u8; 32]) {\n        match &mut self.0 {\n            CtxInner::HeaderPhase { dictionary, .. } => {\n                *dictionary = Some(DictionaryData {\n                    bytes: dictionary_bytes,\n                    hash: dictionary_hash,\n                });\n            }\n            CtxInner::BodyPhase(_) => panic!(\"Wrong phase: BodyPhase\"),\n        }\n    }\n\n    /// Check if a dictionary has been set.\n    pub fn has_dictionary(&self) -> bool {\n        match &self.0 {\n            CtxInner::HeaderPhase { dictionary, .. } => dictionary.is_some(),\n            CtxInner::BodyPhase(_) => false,\n        }\n    }\n\n    /// Clear any previously set dictionary.\n    /// # Panic\n    /// This function will panic if it has already started encoding the response body.\n    pub fn clear_dictionary(&mut self) {\n        match &mut self.0 {\n            CtxInner::HeaderPhase { dictionary, .. } => {\n                *dictionary = None;\n            }\n            CtxInner::BodyPhase(_) => panic!(\"Wrong phase: BodyPhase\"),\n        }\n    }\n\n    /// Feed the request header into this ctx.\n    pub fn request_filter(&mut self, req: &RequestHeader) {\n        if !self.is_enabled() {\n            return;\n        }\n        match &mut self.0 {\n            CtxInner::HeaderPhase {\n                accept_encoding, ..\n            } => parse_accept_encoding(\n                req.headers.get(http::header::ACCEPT_ENCODING),\n                accept_encoding,\n            ),\n            CtxInner::BodyPhase(_) => panic!(\"Wrong phase: BodyPhase\"),\n        }\n    }\n\n    /// Feed the response header into this ctx\n    pub fn response_header_filter(&mut self, resp: &mut ResponseHeader, end: bool) {\n        if !self.is_enabled() {\n            return;\n        }\n        match &self.0 {\n            CtxInner::HeaderPhase {\n                decompress_enable,\n                preserve_etag,\n                accept_encoding,\n                encoding_levels: levels,\n                dictionary,\n            } => {\n                if resp.status.is_informational() {\n                    if resp.status == http::status::StatusCode::SWITCHING_PROTOCOLS {\n                        // no transformation for websocket (TODO: cite RFC)\n                        self.0 = CtxInner::BodyPhase(None);\n                    }\n                    // else, wait for the final response header for decision\n                    return;\n                }\n                // do nothing if no body\n                if end {\n                    self.0 = CtxInner::BodyPhase(None);\n                    return;\n                }\n\n                if depends_on_accept_encoding(\n                    resp,\n                    levels.iter().any(|level| *level != 0),\n                    decompress_enable,\n                ) {\n                    // The response depends on the Accept-Encoding header, make sure to indicate it\n                    // in the Vary response header.\n                    // https://www.rfc-editor.org/rfc/rfc9110#name-vary\n                    add_vary_header(resp, &http::header::ACCEPT_ENCODING);\n                }\n\n                let action = decide_action(resp, accept_encoding);\n                debug!(\"compression action: {action:?}\");\n                let (encoder, preserve_etag) = match action {\n                    Action::Noop => (None, false),\n                    Action::Compress(algorithm) => {\n                        let idx = algorithm.index();\n                        let compressor = match algorithm {\n                            Algorithm::Dcz => dictionary.as_ref().and_then(|d| {\n                                algorithm.maybe_compressor_with_dictionary(levels[idx], d)\n                            }),\n                            _ => algorithm.compressor(levels[idx]),\n                        };\n                        (compressor, preserve_etag[idx])\n                    }\n                    Action::Decompress(algorithm) => {\n                        let idx = algorithm.index();\n                        (\n                            algorithm.decompressor(decompress_enable[idx]),\n                            preserve_etag[idx],\n                        )\n                    }\n                };\n                if encoder.is_some() {\n                    adjust_response_header(resp, &action, preserve_etag);\n                }\n                self.0 = CtxInner::BodyPhase(encoder);\n            }\n            CtxInner::BodyPhase(_) => panic!(\"Wrong phase: BodyPhase\"),\n        }\n    }\n\n    /// Stream the response body chunks into this ctx. The return value will be the compressed\n    /// data.\n    ///\n    /// Return None if compression is not enabled.\n    pub fn response_body_filter(&mut self, data: Option<&Bytes>, end: bool) -> Option<Bytes> {\n        match &mut self.0 {\n            CtxInner::HeaderPhase { .. } => panic!(\"Wrong phase: HeaderPhase\"),\n            CtxInner::BodyPhase(compressor) => {\n                let result = compressor\n                    .as_mut()\n                    .map(|c| {\n                        // Feed even empty slice to compressor because it might yield data\n                        // when `end` is true\n                        let data = if let Some(b) = data { b.as_ref() } else { &[] };\n                        c.encode(data, end)\n                    })\n                    .transpose();\n                result.unwrap_or_else(|e| {\n                    warn!(\"Failed to compress, compression disabled, {}\", e);\n                    // no point to transcode further data because bad data is already seen\n                    self.0 = CtxInner::BodyPhase(None);\n                    None\n                })\n            }\n        }\n    }\n\n    // TODO: retire this function, replace it with the two functions above\n    /// Feed the response into this ctx.\n    /// This filter will mutate the response accordingly if encoding is needed.\n    pub fn response_filter(&mut self, t: &mut HttpTask) {\n        if !self.is_enabled() {\n            return;\n        }\n        match t {\n            HttpTask::Header(resp, end) => self.response_header_filter(resp, *end),\n            HttpTask::Body(data, end) => {\n                let compressed = self.response_body_filter(data.as_ref(), *end);\n                if compressed.is_some() {\n                    *t = HttpTask::Body(compressed, *end);\n                }\n            }\n            HttpTask::Done => {\n                // try to finish/flush compression\n                let compressed = self.response_body_filter(None, true);\n                if compressed.is_some() {\n                    // compressor has more data to flush\n                    *t = HttpTask::Body(compressed, true);\n                }\n            }\n            _ => { /* Trailer, Failed: do nothing? */ }\n        }\n    }\n}\n\n#[derive(Debug, PartialEq, Eq, Clone, Copy, EnumCountMacro)]\npub enum Algorithm {\n    Any, // the \"*\"\n    Gzip,\n    Brotli,\n    Zstd,\n    Dcb,\n    Dcz,\n    // TODO: Identity,\n    // TODO: Deflate\n    Other, // anything unknown\n}\n\nimpl Algorithm {\n    pub fn as_str(&self) -> &'static str {\n        match self {\n            Algorithm::Gzip => \"gzip\",\n            Algorithm::Brotli => \"br\",\n            Algorithm::Zstd => \"zstd\",\n            Algorithm::Dcb => \"dcb\",\n            Algorithm::Dcz => \"dcz\",\n            Algorithm::Any => \"*\",\n            Algorithm::Other => \"other\",\n        }\n    }\n\n    pub fn compressor(&self, level: u32) -> Option<Box<dyn Encode + Send + Sync>> {\n        if level == 0 {\n            None\n        } else {\n            match self {\n                Self::Gzip => Some(Box::new(gzip::Compressor::new(level))),\n                Self::Brotli => Some(Box::new(brotli::Compressor::new(level))),\n                Self::Zstd => Some(Box::new(zstd::Compressor::new(level))),\n                _ => None, // not implemented\n            }\n        }\n    }\n\n    pub fn maybe_compressor_with_dictionary(\n        &self,\n        level: u32,\n        dictionary: &DictionaryData,\n    ) -> Option<Box<dyn Encode + Send + Sync>> {\n        if level == 0 {\n            None\n        } else {\n            match self {\n                Self::Dcz => {\n                    match zstd::DictionaryCompressor::new(level, &dictionary.bytes, dictionary.hash)\n                    {\n                        Ok(c) => Some(Box::new(c)),\n                        Err(e) => {\n                            warn!(\"Failed to create DCZ compressor: {e}\");\n                            None\n                        }\n                    }\n                }\n                _ => None,\n            }\n        }\n    }\n\n    pub fn decompressor(&self, enabled: bool) -> Option<Box<dyn Encode + Send + Sync>> {\n        if !enabled {\n            None\n        } else {\n            match self {\n                Self::Gzip => Some(Box::new(gzip::Decompressor::new())),\n                Self::Brotli => Some(Box::new(brotli::Decompressor::new())),\n                _ => None, // not implemented\n            }\n        }\n    }\n\n    pub fn index(&self) -> usize {\n        *self as usize\n    }\n}\n\nimpl From<&str> for Algorithm {\n    fn from(s: &str) -> Self {\n        use unicase::UniCase;\n\n        let coding = UniCase::new(s);\n        if coding == UniCase::ascii(\"gzip\") {\n            Algorithm::Gzip\n        } else if coding == UniCase::ascii(\"br\") {\n            Algorithm::Brotli\n        } else if coding == UniCase::ascii(\"zstd\") {\n            Algorithm::Zstd\n        } else if coding == UniCase::ascii(\"dcb\") {\n            Algorithm::Dcb\n        } else if coding == UniCase::ascii(\"dcz\") {\n            Algorithm::Dcz\n        } else if s.is_empty() {\n            Algorithm::Any\n        } else {\n            Algorithm::Other\n        }\n    }\n}\n\n#[derive(Debug, PartialEq, Eq, Clone, Copy)]\nenum Action {\n    Noop, // do nothing, e.g. when the input is already gzip\n    Compress(Algorithm),\n    Decompress(Algorithm),\n}\n\n// parse Accept-Encoding header and put it to the list\nfn parse_accept_encoding(accept_encoding: Option<&http::HeaderValue>, list: &mut Vec<Algorithm>) {\n    // https://www.rfc-editor.org/rfc/rfc9110#name-accept-encoding\n    if let Some(ac) = accept_encoding {\n        // fast path\n        if ac.as_bytes() == b\"gzip\" {\n            list.push(Algorithm::Gzip);\n            return;\n        }\n        // properly parse AC header\n        match sfv::Parser::parse_list(ac.as_bytes()) {\n            Ok(parsed) => {\n                for item in parsed {\n                    if let sfv::ListEntry::Item(i) = item {\n                        if let Some(s) = i.bare_item.as_token() {\n                            // TODO: support q value\n                            let algorithm = Algorithm::from(s);\n                            // ignore algorithms that we don't understand ignore\n                            if algorithm != Algorithm::Other {\n                                list.push(Algorithm::from(s));\n                            }\n                        }\n                    }\n                }\n            }\n            Err(e) => {\n                warn!(\"Failed to parse accept-encoding {ac:?}, {e}\")\n            }\n        }\n    } else {\n        // \"If no Accept-Encoding header, any content coding is acceptable\"\n        // keep the list empty\n    }\n}\n\n#[test]\nfn test_accept_encoding_req_header() {\n    let mut header = RequestHeader::build(\"GET\", b\"/\", None).unwrap();\n    let mut ac_list = Vec::new();\n    parse_accept_encoding(\n        header.headers.get(http::header::ACCEPT_ENCODING),\n        &mut ac_list,\n    );\n    assert!(ac_list.is_empty());\n\n    let mut ac_list = Vec::new();\n    header.insert_header(\"accept-encoding\", \"gzip\").unwrap();\n    parse_accept_encoding(\n        header.headers.get(http::header::ACCEPT_ENCODING),\n        &mut ac_list,\n    );\n    assert_eq!(ac_list[0], Algorithm::Gzip);\n\n    let mut ac_list = Vec::new();\n    header\n        .insert_header(\"accept-encoding\", \"what, br, gzip\")\n        .unwrap();\n    parse_accept_encoding(\n        header.headers.get(http::header::ACCEPT_ENCODING),\n        &mut ac_list,\n    );\n    assert_eq!(ac_list[0], Algorithm::Brotli);\n    assert_eq!(ac_list[1], Algorithm::Gzip);\n}\n\n// test whether the response depends on Accept-Encoding header\nfn depends_on_accept_encoding(\n    resp: &ResponseHeader,\n    compress_enabled: bool,\n    decompress_enabled: &[bool],\n) -> bool {\n    use http::header::CONTENT_ENCODING;\n\n    (decompress_enabled.iter().any(|enabled| *enabled)\n        && resp.headers.get(CONTENT_ENCODING).is_some())\n        || (compress_enabled && compressible(resp))\n}\n\n#[test]\nfn test_decide_on_accept_encoding() {\n    let mut resp = ResponseHeader::build(200, None).unwrap();\n    resp.insert_header(\"content-length\", \"50\").unwrap();\n    resp.insert_header(\"content-type\", \"text/html\").unwrap();\n    resp.insert_header(\"content-encoding\", \"gzip\").unwrap();\n\n    // enabled\n    assert!(depends_on_accept_encoding(&resp, false, &[true]));\n\n    // decompress disabled => disabled\n    assert!(!depends_on_accept_encoding(&resp, false, &[false]));\n\n    // no content-encoding => disabled\n    resp.remove_header(\"content-encoding\");\n    assert!(!depends_on_accept_encoding(&resp, false, &[true]));\n\n    // compress enabled and compressible response => enabled\n    assert!(depends_on_accept_encoding(&resp, true, &[false]));\n\n    // compress disabled and compressible response => disabled\n    assert!(!depends_on_accept_encoding(&resp, false, &[false]));\n\n    // compress enabled and not compressible response => disabled\n    resp.insert_header(\"content-type\", \"text/html+zip\").unwrap();\n    assert!(!depends_on_accept_encoding(&resp, true, &[false]));\n}\n\n// filter response header to see if (de)compression is needed\nfn decide_action(resp: &ResponseHeader, accept_encoding: &[Algorithm]) -> Action {\n    use http::header::CONTENT_ENCODING;\n\n    let content_encoding = if let Some(ce) = resp.headers.get(CONTENT_ENCODING) {\n        // https://www.rfc-editor.org/rfc/rfc9110#name-content-encoding\n        if let Ok(ce_str) = std::str::from_utf8(ce.as_bytes()) {\n            Some(Algorithm::from(ce_str))\n        } else {\n            // not utf-8, treat it as unknown encoding to leave it untouched\n            Some(Algorithm::Other)\n        }\n    } else {\n        // no Accept-encoding\n        None\n    };\n\n    if let Some(ce) = content_encoding {\n        if accept_encoding.contains(&ce) {\n            // downstream can accept this encoding, nothing to do\n            Action::Noop\n        } else {\n            // always decompress because uncompressed is always acceptable\n            // https://www.rfc-editor.org/rfc/rfc9110#field.accept-encoding\n            // \"If the representation has no content coding, then it is acceptable by default\n            // unless specifically excluded...\" TODO: check the exclude case\n            // TODO: we could also transcode it to a preferred encoding, e.g. br->gzip\n            Action::Decompress(ce)\n        }\n    } else if accept_encoding.is_empty() // both CE and AE are empty\n        || !compressible(resp) // the type is not compressible\n        || accept_encoding[0] == Algorithm::Any\n    {\n        Action::Noop\n    } else {\n        // try to compress with the first AC\n        // TODO: support to configure preferred encoding\n        Action::Compress(accept_encoding[0])\n    }\n}\n\n#[test]\nfn test_decide_action() {\n    use Action::*;\n    use Algorithm::*;\n\n    let header = ResponseHeader::build(200, None).unwrap();\n    // no compression asked, no compression needed\n    assert_eq!(decide_action(&header, &[]), Noop);\n\n    // already gzip, no compression needed\n    let mut header = ResponseHeader::build(200, None).unwrap();\n    header.insert_header(\"content-type\", \"text/html\").unwrap();\n    header.insert_header(\"content-encoding\", \"gzip\").unwrap();\n    assert_eq!(decide_action(&header, &[Gzip]), Noop);\n\n    // already gzip, no compression needed, upper case\n    let mut header = ResponseHeader::build(200, None).unwrap();\n    header.insert_header(\"content-encoding\", \"GzIp\").unwrap();\n    header.insert_header(\"content-type\", \"text/html\").unwrap();\n    assert_eq!(decide_action(&header, &[Gzip]), Noop);\n\n    // no encoding, compression needed, accepted content-type, large enough\n    // Will compress\n    let mut header = ResponseHeader::build(200, None).unwrap();\n    header.insert_header(\"content-length\", \"20\").unwrap();\n    header.insert_header(\"content-type\", \"text/html\").unwrap();\n    assert_eq!(decide_action(&header, &[Gzip]), Compress(Gzip));\n\n    // too small\n    let mut header = ResponseHeader::build(200, None).unwrap();\n    header.insert_header(\"content-length\", \"19\").unwrap();\n    header.insert_header(\"content-type\", \"text/html\").unwrap();\n    assert_eq!(decide_action(&header, &[Gzip]), Noop);\n\n    // already compressed MIME\n    let mut header = ResponseHeader::build(200, None).unwrap();\n    header.insert_header(\"content-length\", \"20\").unwrap();\n    header\n        .insert_header(\"content-type\", \"text/html+zip\")\n        .unwrap();\n    assert_eq!(decide_action(&header, &[Gzip]), Noop);\n\n    // unsupported MIME\n    let mut header = ResponseHeader::build(200, None).unwrap();\n    header.insert_header(\"content-length\", \"20\").unwrap();\n    header.insert_header(\"content-type\", \"image/jpg\").unwrap();\n    assert_eq!(decide_action(&header, &[Gzip]), Noop);\n\n    // compressed, need decompress\n    let mut header = ResponseHeader::build(200, None).unwrap();\n    header.insert_header(\"content-encoding\", \"gzip\").unwrap();\n    assert_eq!(decide_action(&header, &[]), Decompress(Gzip));\n\n    // accept-encoding different, need decompress\n    let mut header = ResponseHeader::build(200, None).unwrap();\n    header.insert_header(\"content-encoding\", \"gzip\").unwrap();\n    assert_eq!(decide_action(&header, &[Brotli]), Decompress(Gzip));\n\n    // less preferred but no need to decompress\n    let mut header = ResponseHeader::build(200, None).unwrap();\n    header.insert_header(\"content-encoding\", \"gzip\").unwrap();\n    assert_eq!(decide_action(&header, &[Brotli, Gzip]), Noop);\n\n    // dcb passthrough: client accepts dcb, response has dcb\n    let mut header = ResponseHeader::build(200, None).unwrap();\n    header.insert_header(\"content-encoding\", \"dcb\").unwrap();\n    assert_eq!(decide_action(&header, &[Dcb, Brotli]), Noop);\n\n    // dcz passthrough: client accepts dcz, response has dcz\n    let mut header = ResponseHeader::build(200, None).unwrap();\n    header.insert_header(\"content-encoding\", \"dcz\").unwrap();\n    assert_eq!(decide_action(&header, &[Dcz, Zstd]), Noop);\n\n    // Client wants dcz but response has brotli, decompress brotli\n    let mut header = ResponseHeader::build(200, None).unwrap();\n    header.insert_header(\"content-encoding\", \"br\").unwrap();\n    assert_eq!(decide_action(&header, &[Dcz]), Decompress(Brotli));\n\n    // Client wants dcz but response has zstd, decompress zstd\n    let mut header = ResponseHeader::build(200, None).unwrap();\n    header.insert_header(\"content-encoding\", \"zstd\").unwrap();\n    assert_eq!(decide_action(&header, &[Dcz]), Decompress(Zstd));\n\n    // Client wants dcb but response has gzip, decompress gzip\n    let mut header = ResponseHeader::build(200, None).unwrap();\n    header.insert_header(\"content-encoding\", \"gzip\").unwrap();\n    assert_eq!(decide_action(&header, &[Dcb]), Decompress(Gzip));\n\n    // Client wants dcb but response has brotli, decompress brotli\n    let mut header = ResponseHeader::build(200, None).unwrap();\n    header.insert_header(\"content-encoding\", \"br\").unwrap();\n    assert_eq!(decide_action(&header, &[Dcb]), Decompress(Brotli));\n}\n\nuse once_cell::sync::Lazy;\nuse regex::Regex;\n\n// Allow text, application, font, a few image/ MIME types and binary/octet-stream\n// TODO: fine tune this list\nstatic MIME_CHECK: Lazy<Regex> = Lazy::new(|| {\n    Regex::new(r\"^(?:text/|application/|font/|image/(?:x-icon|svg\\+xml|nd\\.microsoft\\.icon)|binary/octet-stream)\")\n        .unwrap()\n});\n\n// check if the response mime type is compressible\nfn compressible(resp: &ResponseHeader) -> bool {\n    // arbitrary size limit, things to consider\n    // 1. too short body may have little redundancy to compress\n    // 2. gzip header and footer overhead\n    // 3. latency is the same as long as data fits in a TCP congestion window regardless of size\n    const MIN_COMPRESS_LEN: usize = 20;\n\n    // check if response is too small to compress\n    if let Some(cl) = resp.headers.get(http::header::CONTENT_LENGTH) {\n        if let Some(cl_num) = std::str::from_utf8(cl.as_bytes())\n            .ok()\n            .and_then(|v| v.parse::<usize>().ok())\n        {\n            if cl_num < MIN_COMPRESS_LEN {\n                return false;\n            }\n        }\n    }\n    // no Content-Length or large enough, check content-type next\n    if let Some(ct) = resp.headers.get(http::header::CONTENT_TYPE) {\n        if let Ok(ct_str) = std::str::from_utf8(ct.as_bytes()) {\n            if ct_str.contains(\"zip\") {\n                // heuristic: don't compress mime type that has zip in it\n                false\n            } else {\n                // check if mime type in allow list\n                MIME_CHECK.find(ct_str).is_some()\n            }\n        } else {\n            false // invalid CT header, don't compress\n        }\n    } else {\n        false // don't compress empty content-type\n    }\n}\n\n// add Vary header with the specified value or extend an existing Vary header value\nfn add_vary_header(resp: &mut ResponseHeader, value: &http::header::HeaderName) {\n    use http::header::{HeaderValue, VARY};\n\n    let already_present = resp.headers.get_all(VARY).iter().any(|existing| {\n        existing\n            .as_bytes()\n            .split(|b| *b == b',')\n            .map(|mut v| {\n                // This is equivalent to slice.trim_ascii() which is unstable\n                while let [first, rest @ ..] = v {\n                    if first.is_ascii_whitespace() {\n                        v = rest;\n                    } else {\n                        break;\n                    }\n                }\n                while let [rest @ .., last] = v {\n                    if last.is_ascii_whitespace() {\n                        v = rest;\n                    } else {\n                        break;\n                    }\n                }\n                v\n            })\n            .any(|v| v == b\"*\" || v.eq_ignore_ascii_case(value.as_ref()))\n    });\n\n    if !already_present {\n        resp.append_header(&VARY, HeaderValue::from_name(value.clone()))\n            .unwrap();\n    }\n}\n\n#[test]\nfn test_add_vary_header() {\n    let mut header = ResponseHeader::build(200, None).unwrap();\n    add_vary_header(&mut header, &http::header::ACCEPT_ENCODING);\n    assert_eq!(\n        header\n            .headers\n            .get_all(\"Vary\")\n            .into_iter()\n            .collect::<Vec<_>>(),\n        vec![\"accept-encoding\"]\n    );\n\n    let mut header = ResponseHeader::build(200, None).unwrap();\n    header.insert_header(\"Vary\", \"Accept-Language\").unwrap();\n    add_vary_header(&mut header, &http::header::ACCEPT_ENCODING);\n    assert_eq!(\n        header\n            .headers\n            .get_all(\"Vary\")\n            .into_iter()\n            .collect::<Vec<_>>(),\n        vec![\"Accept-Language\", \"accept-encoding\"]\n    );\n\n    let mut header = ResponseHeader::build(200, None).unwrap();\n    header\n        .insert_header(\"Vary\", \"Accept-Language, Accept-Encoding\")\n        .unwrap();\n    add_vary_header(&mut header, &http::header::ACCEPT_ENCODING);\n    assert_eq!(\n        header\n            .headers\n            .get_all(\"Vary\")\n            .into_iter()\n            .collect::<Vec<_>>(),\n        vec![\"Accept-Language, Accept-Encoding\"]\n    );\n\n    let mut header = ResponseHeader::build(200, None).unwrap();\n    header.insert_header(\"Vary\", \"*\").unwrap();\n    add_vary_header(&mut header, &http::header::ACCEPT_ENCODING);\n    assert_eq!(\n        header\n            .headers\n            .get_all(\"Vary\")\n            .into_iter()\n            .collect::<Vec<_>>(),\n        vec![\"*\"]\n    );\n}\n\nfn adjust_response_header(resp: &mut ResponseHeader, action: &Action, preserve_etag: bool) {\n    use http::header::{\n        HeaderValue, ACCEPT_RANGES, CONTENT_ENCODING, CONTENT_LENGTH, ETAG, TRANSFER_ENCODING,\n    };\n\n    fn set_stream_headers(resp: &mut ResponseHeader) {\n        // because the transcoding is streamed, content length is not known ahead\n        resp.remove_header(&CONTENT_LENGTH);\n        // remove Accept-Ranges header because range requests will no longer work\n        resp.remove_header(&ACCEPT_RANGES);\n\n        // we stream body now TODO: chunked is for h1 only\n        resp.insert_header(&TRANSFER_ENCODING, HeaderValue::from_static(\"chunked\"))\n            .unwrap();\n    }\n\n    fn weaken_or_clear_etag(resp: &mut ResponseHeader) {\n        // RFC9110: https://datatracker.ietf.org/doc/html/rfc9110#section-8.8.1\n        // \"a validator is weak if it is shared by two or more representations\n        // of a given resource at the same time, unless those representations\n        // have identical representation data\"\n        // Follow nginx gzip filter's example when changing content encoding:\n        // - if the ETag is not a valid strong ETag, clear it (i.e. does not start with `\"`)\n        // - else, weaken it\n        if let Some(etag) = resp.headers.get(&ETAG) {\n            let etag_bytes = etag.as_bytes();\n            if etag_bytes.starts_with(b\"W/\") {\n                // this is already a weak ETag, noop\n            } else if etag_bytes.starts_with(b\"\\\"\") {\n                // strong ETag, weaken since we are changing the byte representation\n                let weakened_etag = HeaderValue::from_bytes(&[b\"W/\", etag_bytes].concat())\n                    .expect(\"valid header value prefixed with \\\"W/\\\" should remain valid\");\n                resp.insert_header(&ETAG, weakened_etag)\n                    .expect(\"can insert weakened etag when etag was already valid\");\n            } else {\n                // invalid strong ETag, just clear it\n                // https://datatracker.ietf.org/doc/html/rfc9110#section-8.8.3\n                // says the opaque-tag section needs to be a quoted string\n                resp.remove_header(&ETAG);\n            }\n        }\n    }\n\n    match action {\n        Action::Noop => { /* do nothing */ }\n        Action::Decompress(_) => {\n            resp.remove_header(&CONTENT_ENCODING);\n            set_stream_headers(resp);\n            if !preserve_etag {\n                weaken_or_clear_etag(resp);\n            }\n        }\n        Action::Compress(a) => {\n            resp.insert_header(&CONTENT_ENCODING, HeaderValue::from_static(a.as_str()))\n                .unwrap();\n            set_stream_headers(resp);\n            if !preserve_etag {\n                weaken_or_clear_etag(resp);\n            }\n        }\n    }\n}\n\n#[test]\nfn test_adjust_response_header() {\n    use Action::*;\n    use Algorithm::*;\n\n    // noop\n    let mut header = ResponseHeader::build(200, None).unwrap();\n    header.insert_header(\"content-length\", \"20\").unwrap();\n    header.insert_header(\"content-encoding\", \"gzip\").unwrap();\n    header.insert_header(\"accept-ranges\", \"bytes\").unwrap();\n    header.insert_header(\"etag\", \"\\\"abc123\\\"\").unwrap();\n    adjust_response_header(&mut header, &Noop, false);\n    assert_eq!(\n        header.headers.get(\"content-encoding\").unwrap().as_bytes(),\n        b\"gzip\"\n    );\n    assert_eq!(\n        header.headers.get(\"content-length\").unwrap().as_bytes(),\n        b\"20\"\n    );\n    assert_eq!(\n        header.headers.get(\"etag\").unwrap().as_bytes(),\n        b\"\\\"abc123\\\"\"\n    );\n    assert!(header.headers.get(\"transfer-encoding\").is_none());\n\n    // decompress gzip\n    let mut header = ResponseHeader::build(200, None).unwrap();\n    header.insert_header(\"content-length\", \"20\").unwrap();\n    header.insert_header(\"content-encoding\", \"gzip\").unwrap();\n    header.insert_header(\"accept-ranges\", \"bytes\").unwrap();\n    header.insert_header(\"etag\", \"\\\"abc123\\\"\").unwrap();\n    adjust_response_header(&mut header, &Decompress(Gzip), false);\n    assert!(header.headers.get(\"content-encoding\").is_none());\n    assert!(header.headers.get(\"content-length\").is_none());\n    assert_eq!(\n        header.headers.get(\"transfer-encoding\").unwrap().as_bytes(),\n        b\"chunked\"\n    );\n    assert!(header.headers.get(\"accept-ranges\").is_none());\n    assert_eq!(\n        header.headers.get(\"etag\").unwrap().as_bytes(),\n        b\"W/\\\"abc123\\\"\"\n    );\n    // when preserve_etag on, strong etag is kept\n    header.insert_header(\"etag\", \"\\\"abc123\\\"\").unwrap();\n    adjust_response_header(&mut header, &Decompress(Gzip), true);\n    assert_eq!(\n        header.headers.get(\"etag\").unwrap().as_bytes(),\n        b\"\\\"abc123\\\"\"\n    );\n\n    // compress\n    let mut header = ResponseHeader::build(200, None).unwrap();\n    header.insert_header(\"content-length\", \"20\").unwrap();\n    header.insert_header(\"accept-ranges\", \"bytes\").unwrap();\n    // try invalid etag, should be cleared\n    header.insert_header(\"etag\", \"abc123\").unwrap();\n    adjust_response_header(&mut header, &Compress(Gzip), false);\n    assert_eq!(\n        header.headers.get(\"content-encoding\").unwrap().as_bytes(),\n        b\"gzip\"\n    );\n    assert!(header.headers.get(\"content-length\").is_none());\n    assert!(header.headers.get(\"accept-ranges\").is_none());\n    assert_eq!(\n        header.headers.get(\"transfer-encoding\").unwrap().as_bytes(),\n        b\"chunked\"\n    );\n    assert!(header.headers.get(\"etag\").is_none());\n    // when preserve_etag on, etag is kept\n    header.insert_header(\"etag\", \"abc123\").unwrap();\n    adjust_response_header(&mut header, &Compress(Gzip), true);\n    assert_eq!(header.headers.get(\"etag\").unwrap().as_bytes(), b\"abc123\");\n}\n\n#[cfg(test)]\nmod tests_dictionary_compression {\n    use super::*;\n\n    const TEST_DICTIONARY: &[u8] = b\"The quick brown fox jumps over the lazy dog. \\\n        Common HTTP headers: Content-Type, Accept-Encoding, Cache-Control. \\\n        JSON patterns: {\\\"key\\\": \\\"value\\\"}, [\\\"array\\\", \\\"items\\\"].\";\n\n    fn test_dictionary_hash() -> [u8; 32] {\n        let mut hash = [0u8; 32];\n        for (i, byte) in TEST_DICTIONARY.iter().take(32).enumerate() {\n            hash[i] = *byte;\n        }\n        hash\n    }\n\n    #[test]\n    fn set_and_clear_dictionary() {\n        let mut ctx = ResponseCompressionCtx::new(3, false, false);\n        assert!(!ctx.has_dictionary());\n\n        ctx.set_dictionary(Bytes::from_static(TEST_DICTIONARY), test_dictionary_hash());\n        assert!(ctx.has_dictionary());\n\n        ctx.clear_dictionary();\n        assert!(!ctx.has_dictionary());\n    }\n\n    #[test]\n    fn dcz_compression_with_dictionary() {\n        let mut ctx = ResponseCompressionCtx::new(3, false, false);\n        let hash = test_dictionary_hash();\n        ctx.set_dictionary(Bytes::from_static(TEST_DICTIONARY), hash);\n\n        let mut req = RequestHeader::build(\"GET\", b\"/test.js\", None).unwrap();\n        req.insert_header(\"accept-encoding\", \"dcz, br, gzip\")\n            .unwrap();\n        ctx.request_filter(&req);\n\n        let mut resp = ResponseHeader::build(200, None).unwrap();\n        resp.insert_header(\"content-type\", \"application/javascript\")\n            .unwrap();\n        resp.insert_header(\"content-length\", \"1000\").unwrap();\n        ctx.response_header_filter(&mut resp, false);\n\n        assert_eq!(\n            resp.headers.get(\"content-encoding\").unwrap().as_bytes(),\n            b\"dcz\"\n        );\n\n        let input = Bytes::from_static(b\"The quick brown fox jumps over the lazy dog again.\");\n        let compressed = ctx.response_body_filter(Some(&input), true).unwrap();\n\n        assert!(compressed.len() >= 40);\n        assert_eq!(&compressed[..8], &zstd::DCZ_MAGIC);\n        assert_eq!(&compressed[8..40], &hash);\n    }\n\n    #[test]\n    fn dcz_without_dictionary_no_compression() {\n        let mut ctx = ResponseCompressionCtx::new(3, false, false);\n\n        let mut req = RequestHeader::build(\"GET\", b\"/test.js\", None).unwrap();\n        req.insert_header(\"accept-encoding\", \"dcz\").unwrap();\n        ctx.request_filter(&req);\n\n        let mut resp = ResponseHeader::build(200, None).unwrap();\n        resp.insert_header(\"content-type\", \"application/javascript\")\n            .unwrap();\n        resp.insert_header(\"content-length\", \"1000\").unwrap();\n        ctx.response_header_filter(&mut resp, false);\n\n        // no dictionary set, no compression applied\n        assert!(resp.headers.get(\"content-encoding\").is_none());\n    }\n\n    #[test]\n    fn dcz_no_fallback_without_dictionary() {\n        let mut ctx = ResponseCompressionCtx::new(3, false, false);\n\n        let mut req = RequestHeader::build(\"GET\", b\"/test.js\", None).unwrap();\n        req.insert_header(\"accept-encoding\", \"dcz, br, gzip\")\n            .unwrap();\n        ctx.request_filter(&req);\n\n        let mut resp = ResponseHeader::build(200, None).unwrap();\n        resp.insert_header(\"content-type\", \"application/javascript\")\n            .unwrap();\n        resp.insert_header(\"content-length\", \"1000\").unwrap();\n        ctx.response_header_filter(&mut resp, false);\n\n        // dcz first but no dictionary, no automatic fallback\n        assert!(resp.headers.get(\"content-encoding\").is_none());\n    }\n\n    #[test]\n    fn maybe_compressor_with_dictionary_dcz_only() {\n        let dict_data = DictionaryData {\n            bytes: Bytes::from_static(TEST_DICTIONARY),\n            hash: test_dictionary_hash(),\n        };\n\n        // only Dcz returns a compressor\n        assert!(Algorithm::Dcz\n            .maybe_compressor_with_dictionary(3, &dict_data)\n            .is_some());\n        assert!(Algorithm::Gzip\n            .maybe_compressor_with_dictionary(3, &dict_data)\n            .is_none());\n        assert!(Algorithm::Brotli\n            .maybe_compressor_with_dictionary(3, &dict_data)\n            .is_none());\n        assert!(Algorithm::Zstd\n            .maybe_compressor_with_dictionary(3, &dict_data)\n            .is_none());\n        // level 0 disables\n        assert!(Algorithm::Dcz\n            .maybe_compressor_with_dictionary(0, &dict_data)\n            .is_none());\n    }\n\n    #[test]\n    fn dcz_full_flow() {\n        let mut ctx = ResponseCompressionCtx::new(3, false, false);\n        let hash = test_dictionary_hash();\n        ctx.set_dictionary(Bytes::from_static(TEST_DICTIONARY), hash);\n\n        let mut req = RequestHeader::build(\"GET\", b\"/app.js\", None).unwrap();\n        req.insert_header(\"accept-encoding\", \"dcz\").unwrap();\n        ctx.request_filter(&req);\n\n        let mut resp = ResponseHeader::build(200, None).unwrap();\n        resp.insert_header(\"content-type\", \"application/javascript\")\n            .unwrap();\n        resp.insert_header(\"content-length\", \"500\").unwrap();\n        ctx.response_header_filter(&mut resp, false);\n\n        assert_eq!(\n            resp.headers.get(\"content-encoding\").unwrap().as_bytes(),\n            b\"dcz\"\n        );\n        assert!(resp.headers.get(\"content-length\").is_none());\n        assert_eq!(\n            resp.headers.get(\"transfer-encoding\").unwrap().as_bytes(),\n            b\"chunked\"\n        );\n\n        let chunk1 = Bytes::from_static(b\"First chunk. \");\n        let output1 = ctx.response_body_filter(Some(&chunk1), false);\n        assert!(output1.is_some());\n\n        let chunk2 = Bytes::from_static(b\"Second chunk.\");\n        let output2 = ctx.response_body_filter(Some(&chunk2), true);\n        assert!(output2.is_some());\n\n        let (name, total_in, total_out, _) = ctx.get_info().unwrap();\n        assert_eq!(name, \"dcz\");\n        assert_eq!(total_in, chunk1.len() + chunk2.len());\n        assert!(total_out > 0);\n    }\n}\n"
  },
  {
    "path": "pingora-core/src/protocols/http/compression/zstd.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\nuse super::{Encode, COMPRESSION_ERROR};\nuse bytes::Bytes;\nuse parking_lot::Mutex;\nuse pingora_error::{OrErr, Result};\nuse std::io::Write;\nuse std::time::{Duration, Instant};\nuse zstd::stream::write::Encoder;\n\n/// [RFC 9842](https://datatracker.ietf.org/doc/html/rfc9842) magic number for dcz.\npub const DCZ_MAGIC: [u8; 8] = [0x5e, 0x2a, 0x4d, 0x18, 0x20, 0x00, 0x00, 0x00];\n\n/// [RFC 9842](https://datatracker.ietf.org/doc/html/rfc9842) header size: 8-byte magic + 32-byte SHA-256 hash.\npub const DCZ_HEADER_SIZE: usize = 40;\n\npub struct Compressor {\n    compress: Mutex<Encoder<'static, Vec<u8>>>,\n    total_in: usize,\n    total_out: usize,\n    duration: Duration,\n}\n\nimpl Compressor {\n    pub fn new(level: u32) -> Self {\n        Compressor {\n            // Mutex because Encoder is not Sync\n            // https://github.com/gyscos/zstd-rs/issues/186\n            compress: Mutex::new(Encoder::new(vec![], level as i32).unwrap()),\n            total_in: 0,\n            total_out: 0,\n            duration: Duration::new(0, 0),\n        }\n    }\n}\n\nimpl Encode for Compressor {\n    fn encode(&mut self, input: &[u8], end: bool) -> Result<Bytes> {\n        // reserve at most 16k\n        const MAX_INIT_COMPRESSED_BUF_SIZE: usize = 16 * 1024;\n        let start = Instant::now();\n        self.total_in += input.len();\n        let mut compress = self.compress.lock();\n        // reserve at most input size, cap at 16k, compressed output should be smaller\n        compress\n            .get_mut()\n            .reserve(std::cmp::min(MAX_INIT_COMPRESSED_BUF_SIZE, input.len()));\n        compress\n            .write_all(input)\n            .or_err(COMPRESSION_ERROR, \"while compress zstd\")?;\n        // write to vec will never fail.\n        if end {\n            compress\n                .do_finish()\n                .or_err(COMPRESSION_ERROR, \"while compress zstd\")?;\n        }\n        self.total_out += compress.get_ref().len();\n        self.duration += start.elapsed();\n        Ok(std::mem::take(compress.get_mut()).into()) // into() Bytes will drop excess capacity\n    }\n\n    fn stat(&self) -> (&'static str, usize, usize, Duration) {\n        (\"zstd\", self.total_in, self.total_out, self.duration)\n    }\n}\n\n/// Dictionary compressor for [RFC 9842](https://datatracker.ietf.org/doc/html/rfc9842) (dcz).\n/// Prepends [`DCZ_HEADER_SIZE`]-byte header to output.\npub struct DictionaryCompressor {\n    compress: Mutex<Encoder<'static, Vec<u8>>>,\n    dictionary_hash: [u8; 32],\n    header_written: bool,\n    total_in: usize,\n    total_out: usize,\n    duration: Duration,\n}\n\nimpl DictionaryCompressor {\n    pub fn new(level: u32, dictionary: &[u8], dictionary_hash: [u8; 32]) -> Result<Self> {\n        let encoder = Encoder::with_dictionary(vec![], level as i32, dictionary).or_err(\n            COMPRESSION_ERROR,\n            \"failed to create zstd encoder with dictionary\",\n        )?;\n\n        Ok(DictionaryCompressor {\n            compress: Mutex::new(encoder),\n            dictionary_hash,\n            header_written: false,\n            total_in: 0,\n            total_out: 0,\n            duration: Duration::new(0, 0),\n        })\n    }\n\n    fn build_header(&self) -> [u8; DCZ_HEADER_SIZE] {\n        let mut header = [0u8; DCZ_HEADER_SIZE];\n        header[..8].copy_from_slice(&DCZ_MAGIC);\n        header[8..].copy_from_slice(&self.dictionary_hash);\n        header\n    }\n}\n\nimpl Encode for DictionaryCompressor {\n    fn encode(&mut self, input: &[u8], end: bool) -> Result<Bytes> {\n        const MAX_INIT_COMPRESSED_BUF_SIZE: usize = 16 * 1024;\n        let start = Instant::now();\n        self.total_in += input.len();\n        let mut compress = self.compress.lock();\n\n        let reserve_size = if !self.header_written {\n            DCZ_HEADER_SIZE + std::cmp::min(MAX_INIT_COMPRESSED_BUF_SIZE, input.len())\n        } else {\n            std::cmp::min(MAX_INIT_COMPRESSED_BUF_SIZE, input.len())\n        };\n        compress.get_mut().reserve(reserve_size);\n\n        if !self.header_written {\n            compress.get_mut().extend_from_slice(&self.build_header());\n            self.header_written = true;\n        }\n\n        compress\n            .write_all(input)\n            .or_err(COMPRESSION_ERROR, \"while compress dcz\")?;\n        if end {\n            compress\n                .do_finish()\n                .or_err(COMPRESSION_ERROR, \"while compress dcz\")?;\n        }\n        self.total_out += compress.get_ref().len();\n        self.duration += start.elapsed();\n        Ok(std::mem::take(compress.get_mut()).into())\n    }\n\n    fn stat(&self) -> (&'static str, usize, usize, Duration) {\n        (\"dcz\", self.total_in, self.total_out, self.duration)\n    }\n}\n\n#[cfg(test)]\nmod tests_stream {\n    use super::*;\n\n    #[test]\n    fn compress_zstd_data() {\n        let mut compressor = Compressor::new(11);\n        let input = b\"adcdefgabcdefghadcdefgabcdefghadcdefgabcdefghadcdefgabcdefgh\\n\";\n        let compressed = compressor.encode(&input[..], false).unwrap();\n        // waiting for more data\n        assert!(compressed.is_empty());\n\n        let compressed = compressor.encode(&input[..], true).unwrap();\n\n        // the zstd Magic_Number\n        assert_eq!(&compressed[..4], &[0x28, 0xB5, 0x2F, 0xFD]);\n        assert!(compressed.len() < input.len());\n    }\n}\n\n#[cfg(test)]\nmod tests_dictionary {\n    use super::*;\n\n    const TEST_DICTIONARY: &[u8] = b\"The quick brown fox jumps over the lazy dog. \\\n        This is a test dictionary with common words and patterns that might appear \\\n        in the content being compressed. HTTP headers, JSON structures, HTML tags.\";\n\n    // This is not a real SHA-256 hash as specified in\n    // [RFC 9842](https://datatracker.ietf.org/doc/html/rfc9842).\n    // The compression module treats the dictionary hash as opaque bytes, so any\n    // 32-byte value is sufficient to test that the hash is correctly written\n    // into the DCZ header.\n    fn test_dictionary_hash() -> [u8; 32] {\n        use std::collections::hash_map::DefaultHasher;\n        use std::hash::{Hash, Hasher};\n\n        let mut hasher = DefaultHasher::new();\n        TEST_DICTIONARY.hash(&mut hasher);\n        let hash = hasher.finish();\n\n        let mut result = [0u8; 32];\n        result[..8].copy_from_slice(&hash.to_le_bytes());\n        result[8..16].copy_from_slice(&hash.to_be_bytes());\n        for (i, byte) in result[16..32].iter_mut().enumerate() {\n            *byte = ((i + 16) as u8).wrapping_mul(hash as u8);\n        }\n        result\n    }\n\n    #[test]\n    fn compress_dcz_prepends_header() {\n        let hash = test_dictionary_hash();\n        let mut compressor = DictionaryCompressor::new(3, TEST_DICTIONARY, hash).unwrap();\n\n        let input = b\"The quick brown fox jumps over the lazy dog again.\";\n        let compressed = compressor.encode(input, true).unwrap();\n\n        assert!(compressed.len() >= DCZ_HEADER_SIZE);\n        // RFC 9842 magic\n        assert_eq!(&compressed[..8], &DCZ_MAGIC);\n        // dictionary hash\n        assert_eq!(&compressed[8..40], &hash);\n        // zstd magic follows\n        assert_eq!(&compressed[40..44], &[0x28, 0xB5, 0x2F, 0xFD]);\n    }\n\n    #[test]\n    fn compress_dcz_header_written_once() {\n        let hash = test_dictionary_hash();\n        let mut compressor = DictionaryCompressor::new(3, TEST_DICTIONARY, hash).unwrap();\n\n        let chunk1 = compressor.encode(b\"First chunk of data. \", false).unwrap();\n        assert!(chunk1.len() >= DCZ_HEADER_SIZE);\n        assert_eq!(&chunk1[..8], &DCZ_MAGIC);\n\n        let chunk2 = compressor.encode(b\"Second chunk of data.\", true).unwrap();\n        if chunk2.len() >= 8 {\n            assert_ne!(&chunk2[..8], &DCZ_MAGIC);\n        }\n    }\n\n    #[test]\n    fn compress_dcz_stats() {\n        let hash = test_dictionary_hash();\n        let mut compressor = DictionaryCompressor::new(3, TEST_DICTIONARY, hash).unwrap();\n\n        let input = b\"Some test data to compress with the dictionary.\";\n        let _ = compressor.encode(input, true).unwrap();\n\n        let (name, total_in, total_out, duration) = compressor.stat();\n        assert_eq!(name, \"dcz\");\n        assert_eq!(total_in, input.len());\n        assert!(total_out >= DCZ_HEADER_SIZE);\n        assert!(duration.as_nanos() > 0);\n    }\n\n    #[test]\n    fn compress_dcz_empty_input() {\n        let hash = test_dictionary_hash();\n        let mut compressor = DictionaryCompressor::new(3, TEST_DICTIONARY, hash).unwrap();\n\n        let compressed = compressor.encode(b\"\", true).unwrap();\n        assert!(compressed.len() >= DCZ_HEADER_SIZE);\n        assert_eq!(&compressed[..8], &DCZ_MAGIC);\n    }\n\n    #[test]\n    fn compress_dcz_streaming() {\n        let hash = test_dictionary_hash();\n        let mut compressor = DictionaryCompressor::new(3, TEST_DICTIONARY, hash).unwrap();\n\n        let chunks: &[&[u8]] = &[b\"First part. \", b\"Second part. \", b\"Final part.\"];\n\n        let mut all_output = Vec::new();\n        for (i, chunk) in chunks.iter().enumerate() {\n            let output = compressor.encode(chunk, i == chunks.len() - 1).unwrap();\n            all_output.extend_from_slice(&output);\n        }\n\n        assert!(all_output.len() >= DCZ_HEADER_SIZE);\n        assert_eq!(&all_output[..8], &DCZ_MAGIC);\n\n        let (_, total_in, _, _) = compressor.stat();\n        let expected_in: usize = chunks.iter().map(|c| c.len()).sum();\n        assert_eq!(total_in, expected_in);\n    }\n\n    #[test]\n    fn compress_dcz_achieves_compression() {\n        let hash = test_dictionary_hash();\n        let mut compressor = DictionaryCompressor::new(3, TEST_DICTIONARY, hash).unwrap();\n\n        let input = b\"The quick brown fox jumps over the lazy dog. \\\n            The quick brown fox jumps over the lazy dog. \\\n            The quick brown fox jumps over the lazy dog.\";\n\n        let compressed = compressor.encode(input, true).unwrap();\n        let compressed_data_size = compressed.len() - DCZ_HEADER_SIZE;\n        assert!(compressed_data_size < input.len());\n    }\n\n    #[test]\n    fn compress_dcz_roundtrip() {\n        let hash = test_dictionary_hash();\n        let mut compressor = DictionaryCompressor::new(3, TEST_DICTIONARY, hash).unwrap();\n\n        let input = b\"The quick brown fox jumps over the lazy dog. \\\n            HTTP headers, JSON structures, HTML tags. \\\n            Common patterns that appear in web content.\";\n        let compressed = compressor.encode(input, true).unwrap();\n\n        // Verify DCZ header is present then strip it\n        assert!(compressed.len() >= DCZ_HEADER_SIZE);\n        assert_eq!(&compressed[..8], &DCZ_MAGIC);\n        let zstd_data = &compressed[DCZ_HEADER_SIZE..];\n\n        // Decompress with the same dictionary and verify roundtrip\n        let mut decoder =\n            zstd::stream::read::Decoder::with_dictionary(zstd_data, TEST_DICTIONARY).unwrap();\n        let mut decompressed = Vec::new();\n        std::io::Read::read_to_end(&mut decoder, &mut decompressed).unwrap();\n\n        assert_eq!(decompressed, input);\n    }\n}\n"
  },
  {
    "path": "pingora-core/src/protocols/http/conditional_filter.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! Conditional filter (not modified) utilities\n\nuse http::{header::*, StatusCode};\nuse httpdate::{parse_http_date, HttpDate};\nuse pingora_error::{ErrorType::InvalidHTTPHeader, OrErr, Result};\nuse pingora_http::{RequestHeader, ResponseHeader};\n\n/// Evaluates conditional headers according to the [RFC](https://datatracker.ietf.org/doc/html/rfc9111#name-handling-a-received-validat).\n///\n/// Returns true if the request should receive 304 Not Modified.\npub fn not_modified_filter(req: &RequestHeader, resp: &ResponseHeader) -> bool {\n    // https://datatracker.ietf.org/doc/html/rfc9110#name-304-not-modified\n    // 304 can only validate 200\n    if resp.status != StatusCode::OK {\n        return false;\n    }\n\n    // Evulation of conditional headers, based on RFC:\n    // https://datatracker.ietf.org/doc/html/rfc9111#name-handling-a-received-validat\n\n    // TODO: If-Match and If-Unmodified-Since, and returning 412 Precondition Failed\n    // Note that this function is currently used only for proxy cache,\n    // and the current RFCs have some conflicting opinions as to whether\n    // If-Match and If-Unmodified-Since can be used. https://github.com/httpwg/http-core/issues/1111\n\n    // Conditional request precedence:\n    // https://datatracker.ietf.org/doc/html/rfc9110#name-precedence-of-preconditions\n    // If-None-Match should be handled before If-Modified-Since.\n    // XXX: In nginx, IMS is actually checked first, which may cause compatibility issues\n    // for certain origins/clients.\n\n    if req.headers.contains_key(IF_NONE_MATCH) {\n        if let Some(etag) = resp.headers.get(ETAG) {\n            for inm in req.headers.get_all(IF_NONE_MATCH) {\n                if weak_validate_etag(inm.as_bytes(), etag.as_bytes()) {\n                    return true;\n                }\n            }\n        }\n        // https://datatracker.ietf.org/doc/html/rfc9110#field.if-modified-since\n        // \"MUST ignore If-Modified-Since if the request contains an If-None-Match header\"\n        return false;\n    }\n\n    // GET/HEAD only https://datatracker.ietf.org/doc/html/rfc9110#field.if-modified-since\n    if matches!(req.method, http::Method::GET | http::Method::HEAD) {\n        if let Ok(Some(if_modified_since)) = req_header_as_http_date(req, &IF_MODIFIED_SINCE) {\n            if let Ok(Some(last_modified)) = resp_header_as_http_date(resp, &LAST_MODIFIED) {\n                if if_modified_since >= last_modified {\n                    return true;\n                }\n            }\n        }\n    }\n    false\n}\n\n// Trim ASCII whitespace bytes from the start of the slice.\n// This is pretty much copied from the nightly API.\n// TODO: use `trim_ascii_start` when it stabilizes https://doc.rust-lang.org/std/primitive.slice.html#method.trim_ascii_start\nfn trim_ascii_start(mut bytes: &[u8]) -> &[u8] {\n    while let [first, rest @ ..] = bytes {\n        if first.is_ascii_whitespace() {\n            bytes = rest;\n        } else {\n            break;\n        }\n    }\n    bytes\n}\n\n/// Search for an ETag matching `target_etag` from the input header, using\n/// [weak comparison](https://datatracker.ietf.org/doc/html/rfc9110#section-8.8.3.2).\n/// Multiple ETags can exist in the header as a comma-separated list.\n///\n/// Returns true if a matching ETag exists.\npub fn weak_validate_etag(input_etag_header: &[u8], target_etag: &[u8]) -> bool {\n    // ETag comparison: https://datatracker.ietf.org/doc/html/rfc9110#section-8.8.3.2\n    fn strip_weak_prefix(etag: &[u8]) -> &[u8] {\n        // Weak ETags are prefaced with `W/`\n        etag.strip_prefix(b\"W/\").unwrap_or(etag)\n    }\n    // https://datatracker.ietf.org/doc/html/rfc9110#section-13.1.2 unsafe method only\n    if input_etag_header == b\"*\" {\n        return true;\n    }\n\n    // The RFC defines ETags here: https://datatracker.ietf.org/doc/html/rfc9110#section-8.8.3\n    // The RFC requires ETags to be wrapped in double quotes, though some legacy origins or clients\n    // don't adhere to this.\n    // Unfortunately by allowing non-quoted etags, parsing becomes a little more complicated.\n    //\n    // This implementation uses nginx's algorithm for parsing ETags, which can handle both quoted\n    // and non-quoted ETags. It essentially does a substring comparison at each comma divider,\n    // searching for an exact match of the ETag (optional double quotes included) followed by\n    // either EOF or another comma.\n    //\n    // Clients and upstreams should still ideally adhere to quoted ETags to disambiguate\n    // situations where commas are contained within the ETag (allowed by the RFC).\n    // XXX: This nginx algorithm will handle matching against ETags with commas correctly, but only\n    // if the target ETag is a quoted RFC-compliant ETag.\n    //\n    // For example, consider an if-none-match header: `\"xyzzy,xyz,x,y\", \"xyzzy\"`.\n    // If the target ETag is double quoted as mandated by the RFC like `\"xyz,x\"`, this algorithm\n    // will correctly report no matching ETag.\n    // But if the target ETag is not double quoted like `xyz,x`, it will \"incorrectly\" match\n    // against the substring after the first comma inside the first quoted ETag.\n\n    // Search for the target at each comma delimiter\n    let target_etag = strip_weak_prefix(target_etag);\n    let mut remaining = strip_weak_prefix(input_etag_header);\n    while let Some(search_slice) = remaining.get(0..target_etag.len()) {\n        if search_slice == target_etag {\n            remaining = &remaining[target_etag.len()..];\n            // check if there's any content after the matched substring\n            // skip any whitespace\n            remaining = trim_ascii_start(remaining);\n            if matches!(remaining.first(), None | Some(b',')) {\n                // we are either at the end of the header, or at a comma delimiter\n                // which means this is a match\n                return true;\n            }\n        }\n        // find the next delimiter (ignore any remaining part of the non-matching etag)\n        let Some(next_delimiter_pos) = remaining.iter().position(|&b| b == b',') else {\n            break;\n        };\n        remaining = &remaining[next_delimiter_pos..];\n        // find the next etag slice to compare\n        // ignore extraneous delimiters and whitespace\n        let Some(next_etag_pos) = remaining\n            .iter()\n            .position(|&b| !b.is_ascii_whitespace() && b != b',')\n        else {\n            break;\n        };\n        remaining = &remaining[next_etag_pos..];\n\n        remaining = strip_weak_prefix(remaining);\n    }\n    // remaining length < target etag length\n    false\n}\n\n/// Utility function to parse an HTTP request header as an [HTTP-date](https://datatracker.ietf.org/doc/html/rfc9110#name-date-time-formats).\npub fn req_header_as_http_date<H>(req: &RequestHeader, header_name: H) -> Result<Option<HttpDate>>\nwhere\n    H: AsHeaderName,\n{\n    let Some(header_value) = req.headers.get(header_name) else {\n        return Ok(None);\n    };\n    Ok(Some(parse_bytes_as_http_date(header_value.as_bytes())?))\n}\n\n/// Utility function to parse an HTTP response header as an [HTTP-date](https://datatracker.ietf.org/doc/html/rfc9110#name-date-time-formats).\npub fn resp_header_as_http_date<H>(\n    resp: &ResponseHeader,\n    header_name: H,\n) -> Result<Option<HttpDate>>\nwhere\n    H: AsHeaderName,\n{\n    let Some(header_value) = resp.headers.get(header_name) else {\n        return Ok(None);\n    };\n    Ok(Some(parse_bytes_as_http_date(header_value.as_bytes())?))\n}\n\nfn parse_bytes_as_http_date(bytes: &[u8]) -> Result<HttpDate> {\n    let input_time = std::str::from_utf8(bytes).explain_err(InvalidHTTPHeader, |_| {\n        \"HTTP date has unsupported characters (bytes outside of UTF-8)\"\n    })?;\n    Ok(parse_http_date(input_time)\n        .or_err(InvalidHTTPHeader, \"Invalid HTTP date\")?\n        .into())\n}\n\n/// Utility function to convert the input response header to a 304 Not Modified response.\npub fn to_304(resp: &mut ResponseHeader) {\n    // https://datatracker.ietf.org/doc/html/rfc9110#name-304-not-modified\n    // XXX: https://datatracker.ietf.org/doc/html/rfc9110#name-content-length\n    // \"A server may send content-length in 304\", but no common web server does it\n    // So we drop both content-length and content-type for consistency/less surprise\n    resp.set_status(StatusCode::NOT_MODIFIED).unwrap();\n    resp.remove_header(&CONTENT_LENGTH);\n    resp.remove_header(&CONTENT_TYPE);\n    // https://datatracker.ietf.org/doc/html/rfc9110#section-15.4.5-4\n    // \"SHOULD NOT generate representation metadata other than the above listed fields\n    // unless said metadata exists for the purpose of guiding cache updates\"\n    // Remove some more representation metadata headers\n    resp.remove_header(&TRANSFER_ENCODING);\n    // note that the following are also stripped by nginx\n    resp.remove_header(&CONTENT_ENCODING);\n    resp.remove_header(&ACCEPT_RANGES);\n}\n\n#[cfg(test)]\nmod tests {\n    use super::*;\n\n    #[test]\n    fn test_if_modified_since() {\n        fn build_req(if_modified_since: &[u8]) -> RequestHeader {\n            let mut req = RequestHeader::build(\"GET\", b\"/\", None).unwrap();\n            req.insert_header(\"If-Modified-Since\", if_modified_since)\n                .unwrap();\n            req\n        }\n\n        fn build_resp(last_modified: &[u8]) -> ResponseHeader {\n            let mut resp = ResponseHeader::build(200, None).unwrap();\n            resp.insert_header(\"Last-Modified\", last_modified).unwrap();\n            resp\n        }\n\n        // same date\n        let last_modified = b\"Fri, 26 Mar 2010 00:05:00 GMT\";\n        let req = build_req(b\"Fri, 26 Mar 2010 00:05:00 GMT\");\n        let resp = build_resp(last_modified);\n        assert!(not_modified_filter(&req, &resp));\n\n        // before\n        let req = build_req(b\"Fri, 26 Mar 2010 00:03:00 GMT\");\n        let resp = build_resp(last_modified);\n        assert!(!not_modified_filter(&req, &resp));\n\n        // after\n        let req = build_req(b\"Sun, 28 Mar 2010 01:07:00 GMT\");\n        let resp = build_resp(last_modified);\n        assert!(not_modified_filter(&req, &resp));\n    }\n\n    #[test]\n    fn test_weak_validate_etag() {\n        let target_weak_etag = br#\"W/\"xyzzy\"\"#;\n        let target_etag = br#\"\"xyzzy\"\"#;\n        assert!(weak_validate_etag(b\"*\", target_weak_etag));\n        assert!(weak_validate_etag(b\"*\", target_etag));\n\n        assert!(weak_validate_etag(target_etag, target_etag));\n        assert!(weak_validate_etag(target_etag, target_weak_etag));\n        assert!(weak_validate_etag(target_weak_etag, target_etag));\n        assert!(weak_validate_etag(target_weak_etag, target_weak_etag));\n\n        let mismatch_weak_etag = br#\"W/\"abc\"\"#;\n        let mismatch_etag = br#\"\"abc\"\"#;\n        assert!(!weak_validate_etag(mismatch_etag, target_etag));\n        assert!(!weak_validate_etag(mismatch_etag, target_weak_etag));\n        assert!(!weak_validate_etag(mismatch_weak_etag, target_etag));\n        assert!(!weak_validate_etag(mismatch_weak_etag, target_weak_etag));\n\n        let multiple_etags = br#\"a, \"xyzzy\",\"r2d2xxxx\", \"c3piozzzz\",zzzfoo\"#;\n        assert!(weak_validate_etag(multiple_etags, target_etag));\n        assert!(weak_validate_etag(multiple_etags, target_weak_etag));\n\n        let multiple_mismatch_etags = br#\"foobar\", \"r2d2xxxx\", \"c3piozzzz\",zzzfoo\"#;\n        assert!(!weak_validate_etag(multiple_mismatch_etags, target_etag));\n        assert!(!weak_validate_etag(\n            multiple_mismatch_etags,\n            target_weak_etag\n        ));\n\n        let multiple_mismatch_etags =\n            br#\"foobar\", \"r2d2xxxxyzzy\", \"c3piozzzz\",zzzfoo, \"xyzzy,xyzzy\"\"#;\n        assert!(!weak_validate_etag(multiple_mismatch_etags, target_etag));\n        assert!(!weak_validate_etag(\n            multiple_mismatch_etags,\n            target_weak_etag\n        ));\n\n        let target_comma_etag = br#\"\",,,\"\"#;\n        let multiple_mismatch_etags = br#\",\", \",,,,\", ,,,,,,,,\",,\",\",,,,,,\"\"#;\n        assert!(!weak_validate_etag(\n            multiple_mismatch_etags,\n            target_comma_etag\n        ));\n        let multiple_etags = br#\",\", \",,,,\", ,,,,,,,,\",,,\",\",,,,,,\"\"#;\n        assert!(weak_validate_etag(multiple_etags, target_comma_etag));\n    }\n\n    #[test]\n    fn test_weak_validate_etag_unquoted() {\n        // legacy unquoted etag\n        let target_unquoted = b\"xyzzy\";\n        assert!(weak_validate_etag(b\"*\", target_unquoted));\n\n        let strong_etag = br#\"\"xyzzy\"\"#;\n        assert!(!weak_validate_etag(strong_etag, target_unquoted));\n        assert!(!weak_validate_etag(target_unquoted, strong_etag));\n\n        let multiple_etags = br#\"a, \"r2d2xxxx\", \"c3piozzzz\",   xyzzy\"#;\n        assert!(weak_validate_etag(multiple_etags, target_unquoted));\n\n        let multiple_mismatch_etags =\n            br#\"foobar\", \"r2d2xxxxyzzy\", \"c3piozzzz\",zzzfoo, \"xyzzy,xyzzy\"\"#;\n        assert!(!weak_validate_etag(\n            multiple_mismatch_etags,\n            target_unquoted\n        ));\n\n        // in certain edge cases where commas are used alongside quoted ETags,\n        // the test can fail if target is unquoted (the last ETag is intended to be one ETag)\n        let multiple_mismatch_etags =\n            br#\"foobar\", \"r2d2xxxxyzzy\", \"c3piozzzz\",zzzfoo, \"xyzzy,xyzzy,xy\"\"#;\n        assert!(weak_validate_etag(multiple_mismatch_etags, target_unquoted));\n    }\n}\n"
  },
  {
    "path": "pingora-core/src/protocols/http/custom/client.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\nuse std::time::Duration;\n\nuse async_trait::async_trait;\nuse bytes::Bytes;\nuse futures::Stream;\nuse http::HeaderMap;\nuse pingora_error::Result;\nuse pingora_http::{RequestHeader, ResponseHeader};\n\nuse crate::protocols::{l4::socket::SocketAddr, Digest, UniqueIDType};\n\nuse super::{BodyWrite, CustomMessageWrite};\n\n#[doc(hidden)]\n#[async_trait]\npub trait Session: Send + Sync + Unpin + 'static {\n    async fn write_request_header(&mut self, req: Box<RequestHeader>, end: bool) -> Result<()>;\n\n    async fn write_request_body(&mut self, data: Bytes, end: bool) -> Result<()>;\n\n    async fn finish_request_body(&mut self) -> Result<()>;\n\n    fn set_read_timeout(&mut self, timeout: Option<Duration>);\n\n    fn set_write_timeout(&mut self, timeout: Option<Duration>);\n\n    async fn read_response_header(&mut self) -> Result<()>;\n\n    async fn read_response_body(&mut self) -> Result<Option<Bytes>>;\n\n    fn response_finished(&self) -> bool;\n\n    async fn shutdown(&mut self, code: u32, ctx: &str);\n\n    fn response_header(&self) -> Option<&ResponseHeader>;\n\n    fn was_upgraded(&self) -> bool;\n\n    fn digest(&self) -> Option<&Digest>;\n\n    fn digest_mut(&mut self) -> Option<&mut Digest>;\n\n    fn server_addr(&self) -> Option<&SocketAddr>;\n\n    fn client_addr(&self) -> Option<&SocketAddr>;\n\n    async fn read_trailers(&mut self) -> Result<Option<HeaderMap>>;\n\n    fn fd(&self) -> UniqueIDType;\n\n    async fn check_response_end_or_error(&mut self, headers: bool) -> Result<bool>;\n\n    fn take_request_body_writer(&mut self) -> Option<Box<dyn BodyWrite>>;\n\n    async fn finish_custom(&mut self) -> Result<()>;\n\n    fn take_custom_message_reader(\n        &mut self,\n    ) -> Option<Box<dyn Stream<Item = Result<Bytes>> + Unpin + Send + Sync + 'static>>;\n\n    async fn drain_custom_messages(&mut self) -> Result<()>;\n\n    fn take_custom_message_writer(&mut self) -> Option<Box<dyn CustomMessageWrite>>;\n}\n\n#[doc(hidden)]\n#[async_trait]\nimpl Session for () {\n    async fn write_request_header(&mut self, _req: Box<RequestHeader>, _end: bool) -> Result<()> {\n        unreachable!(\"client session: write_request_header\")\n    }\n\n    async fn write_request_body(&mut self, _data: Bytes, _end: bool) -> Result<()> {\n        unreachable!(\"client session: write_request_body\")\n    }\n\n    async fn finish_request_body(&mut self) -> Result<()> {\n        unreachable!(\"client session: finish_request_body\")\n    }\n\n    fn set_read_timeout(&mut self, _timeout: Option<Duration>) {\n        unreachable!(\"client session: set_read_timeout\")\n    }\n\n    fn set_write_timeout(&mut self, _timeout: Option<Duration>) {\n        unreachable!(\"client session: set_write_timeout\")\n    }\n\n    async fn read_response_header(&mut self) -> Result<()> {\n        unreachable!(\"client session: read_response_header\")\n    }\n\n    async fn read_response_body(&mut self) -> Result<Option<Bytes>> {\n        unreachable!(\"client session: read_response_body\")\n    }\n\n    fn response_finished(&self) -> bool {\n        unreachable!(\"client session: response_finished\")\n    }\n\n    async fn shutdown(&mut self, _code: u32, _ctx: &str) {\n        unreachable!(\"client session: shutdown\")\n    }\n\n    fn response_header(&self) -> Option<&ResponseHeader> {\n        unreachable!(\"client session: response_header\")\n    }\n\n    fn was_upgraded(&self) -> bool {\n        unreachable!(\"client session: was upgraded\")\n    }\n\n    fn digest(&self) -> Option<&Digest> {\n        unreachable!(\"client session: digest\")\n    }\n\n    fn digest_mut(&mut self) -> Option<&mut Digest> {\n        unreachable!(\"client session: digest_mut\")\n    }\n\n    fn server_addr(&self) -> Option<&SocketAddr> {\n        unreachable!(\"client session: server_addr\")\n    }\n\n    fn client_addr(&self) -> Option<&SocketAddr> {\n        unreachable!(\"client session: client_addr\")\n    }\n\n    async fn finish_custom(&mut self) -> Result<()> {\n        unreachable!(\"client session: finish_custom\")\n    }\n\n    async fn read_trailers(&mut self) -> Result<Option<HeaderMap>> {\n        unreachable!(\"client session: read_trailers\")\n    }\n\n    fn fd(&self) -> UniqueIDType {\n        unreachable!(\"client session: fd\")\n    }\n\n    async fn check_response_end_or_error(&mut self, _headers: bool) -> Result<bool> {\n        unreachable!(\"client session: check_response_end_or_error\")\n    }\n\n    fn take_custom_message_reader(\n        &mut self,\n    ) -> Option<Box<dyn Stream<Item = Result<Bytes>> + Unpin + Send + Sync + 'static>> {\n        unreachable!(\"client session: get_custom_message_reader\")\n    }\n\n    async fn drain_custom_messages(&mut self) -> Result<()> {\n        unreachable!(\"client session: drain_custom_messages\")\n    }\n\n    fn take_custom_message_writer(&mut self) -> Option<Box<dyn CustomMessageWrite>> {\n        unreachable!(\"client session: get_custom_message_writer\")\n    }\n\n    fn take_request_body_writer(&mut self) -> Option<Box<dyn BodyWrite>> {\n        unreachable!(\"client session: take_request_body_writer\")\n    }\n}\n"
  },
  {
    "path": "pingora-core/src/protocols/http/custom/mod.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\nuse std::time::Duration;\n\nuse async_trait::async_trait;\nuse bytes::Bytes;\nuse futures::Stream;\nuse log::debug;\nuse pingora_error::Result;\nuse tokio_stream::StreamExt;\n\npub mod client;\npub mod server;\n\npub const CUSTOM_MESSAGE_QUEUE_SIZE: usize = 128;\n\npub fn is_informational_except_101<T: PartialOrd<u32>>(code: T) -> bool {\n    // excluding `101 Switching Protocols`, because it's not followed by any other\n    // response and it's a final\n    // The WebSocket Protocol https://datatracker.ietf.org/doc/html/rfc6455\n    code > 99 && code < 200 && code != 101\n}\n\n#[async_trait]\npub trait CustomMessageWrite: Send + Sync + Unpin + 'static {\n    fn set_write_timeout(&mut self, timeout: Option<Duration>);\n    async fn write_custom_message(&mut self, msg: Bytes) -> Result<()>;\n    async fn finish_custom(&mut self) -> Result<()>;\n}\n\n#[doc(hidden)]\n#[async_trait]\nimpl CustomMessageWrite for () {\n    fn set_write_timeout(&mut self, _timeout: Option<Duration>) {}\n\n    async fn write_custom_message(&mut self, msg: Bytes) -> Result<()> {\n        debug!(\"write_custom_message: {:?}\", msg);\n        Ok(())\n    }\n\n    async fn finish_custom(&mut self) -> Result<()> {\n        debug!(\"finish_custom\");\n        Ok(())\n    }\n}\n\n#[async_trait]\npub trait BodyWrite: Send + Sync + Unpin + 'static {\n    async fn write_all_buf(&mut self, data: &mut Bytes) -> Result<()>;\n    async fn finish(&mut self) -> Result<()>;\n    async fn cleanup(&mut self) -> Result<()>;\n    fn upgrade_body_writer(&mut self);\n}\n\npub async fn drain_custom_messages(\n    reader: Option<Box<dyn Stream<Item = Result<Bytes>> + Unpin + Send + Sync + 'static>>,\n) -> Result<()> {\n    let Some(mut reader) = reader else {\n        return Ok(());\n    };\n\n    while let Some(res) = reader.next().await {\n        let msg = res?;\n        debug!(\"consume_custom_messages: {msg:?}\");\n    }\n\n    Ok(())\n}\n\n#[macro_export]\nmacro_rules! custom_session {\n    ($base_obj:ident . $($method_tokens:tt)+) => {\n        if let Some(custom_session) = $base_obj.as_custom_mut() {\n            #[allow(clippy::semicolon_if_nothing_returned)]\n            custom_session.$($method_tokens)+;\n        }\n    };\n}\n"
  },
  {
    "path": "pingora-core/src/protocols/http/custom/server.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\nuse std::time::Duration;\n\nuse async_trait::async_trait;\nuse bytes::Bytes;\nuse futures::Stream;\nuse http::HeaderMap;\nuse pingora_error::Result;\nuse pingora_http::{RequestHeader, ResponseHeader};\n\nuse crate::protocols::{http::HttpTask, l4::socket::SocketAddr, Digest};\n\nuse super::CustomMessageWrite;\n\n#[doc(hidden)]\n#[async_trait]\npub trait Session: Send + Sync + Unpin + 'static {\n    fn req_header(&self) -> &RequestHeader;\n\n    fn req_header_mut(&mut self) -> &mut RequestHeader;\n\n    async fn read_body_bytes(&mut self) -> Result<Option<Bytes>>;\n\n    async fn drain_request_body(&mut self) -> Result<()>;\n\n    async fn write_response_header(&mut self, resp: Box<ResponseHeader>, end: bool) -> Result<()>;\n\n    async fn write_response_header_ref(&mut self, resp: &ResponseHeader, end: bool) -> Result<()>;\n\n    async fn write_body(&mut self, data: Bytes, end: bool) -> Result<()>;\n\n    async fn write_trailers(&mut self, trailers: HeaderMap) -> Result<()>;\n\n    async fn response_duplex_vec(&mut self, tasks: Vec<HttpTask>) -> Result<bool>;\n\n    fn set_read_timeout(&mut self, timeout: Option<Duration>);\n\n    fn get_read_timeout(&self) -> Option<Duration>;\n\n    fn set_write_timeout(&mut self, timeout: Option<Duration>);\n\n    fn get_write_timeout(&self) -> Option<Duration>;\n\n    fn set_total_drain_timeout(&mut self, timeout: Option<Duration>);\n\n    fn get_total_drain_timeout(&self) -> Option<Duration>;\n\n    fn request_summary(&self) -> String;\n\n    fn response_written(&self) -> Option<&ResponseHeader>;\n\n    async fn shutdown(&mut self, code: u32, ctx: &str);\n\n    fn is_body_done(&mut self) -> bool;\n\n    async fn finish(&mut self) -> Result<()>;\n\n    fn is_body_empty(&mut self) -> bool;\n\n    async fn read_body_or_idle(&mut self, no_body_expected: bool) -> Result<Option<Bytes>>;\n\n    fn body_bytes_sent(&self) -> usize;\n\n    fn body_bytes_read(&self) -> usize;\n\n    fn digest(&self) -> Option<&Digest>;\n\n    fn digest_mut(&mut self) -> Option<&mut Digest>;\n\n    fn client_addr(&self) -> Option<&SocketAddr>;\n\n    fn server_addr(&self) -> Option<&SocketAddr>;\n\n    fn pseudo_raw_h1_request_header(&self) -> Bytes;\n\n    fn enable_retry_buffering(&mut self);\n\n    fn retry_buffer_truncated(&self) -> bool;\n\n    fn get_retry_buffer(&self) -> Option<Bytes>;\n\n    async fn finish_custom(&mut self) -> Result<()>;\n\n    fn take_custom_message_reader(\n        &mut self,\n    ) -> Option<Box<dyn Stream<Item = Result<Bytes>> + Unpin + Send + Sync + 'static>>;\n\n    fn restore_custom_message_reader(\n        &mut self,\n        reader: Box<dyn Stream<Item = Result<Bytes>> + Unpin + Send + Sync + 'static>,\n    ) -> Result<()>;\n\n    fn take_custom_message_writer(&mut self) -> Option<Box<dyn CustomMessageWrite>>;\n\n    fn restore_custom_message_writer(&mut self, writer: Box<dyn CustomMessageWrite>) -> Result<()>;\n\n    /// Whether this request is for upgrade (e.g., websocket).\n    ///\n    /// Returns `true` if the request has HTTP/1.1 version and contains an Upgrade header.\n    fn is_upgrade_req(&self) -> bool {\n        false\n    }\n\n    /// Whether this session was fully upgraded (completed Upgrade handshake).\n    ///\n    /// Returns `true` if the request was an upgrade request and a 101 response was sent.\n    fn was_upgraded(&self) -> bool {\n        false\n    }\n}\n\n#[doc(hidden)]\n#[async_trait]\nimpl Session for () {\n    fn req_header(&self) -> &RequestHeader {\n        unreachable!(\"server session: req_header\")\n    }\n\n    fn req_header_mut(&mut self) -> &mut RequestHeader {\n        unreachable!(\"server session: req_header_mut\")\n    }\n\n    async fn read_body_bytes(&mut self) -> Result<Option<Bytes>> {\n        unreachable!(\"server session: read_body_bytes\")\n    }\n\n    async fn drain_request_body(&mut self) -> Result<()> {\n        unreachable!(\"server session: drain_request_body\")\n    }\n\n    async fn write_response_header(\n        &mut self,\n        _resp: Box<ResponseHeader>,\n        _end: bool,\n    ) -> Result<()> {\n        unreachable!(\"server session: write_response_header\")\n    }\n\n    async fn write_response_header_ref(\n        &mut self,\n        _resp: &ResponseHeader,\n        _end: bool,\n    ) -> Result<()> {\n        unreachable!(\"server session: write_response_header_ref\")\n    }\n\n    async fn write_body(&mut self, _data: Bytes, _end: bool) -> Result<()> {\n        unreachable!(\"server session: write_body\")\n    }\n\n    async fn write_trailers(&mut self, _trailers: HeaderMap) -> Result<()> {\n        unreachable!(\"server session: write_trailers\")\n    }\n\n    async fn response_duplex_vec(&mut self, _tasks: Vec<HttpTask>) -> Result<bool> {\n        unreachable!(\"server session: response_duplex_vec\")\n    }\n\n    fn set_read_timeout(&mut self, _timeout: Option<Duration>) {\n        unreachable!(\"server session: set_read_timeout\")\n    }\n\n    fn get_read_timeout(&self) -> Option<Duration> {\n        unreachable!(\"server_session: get_read_timeout\")\n    }\n\n    fn set_write_timeout(&mut self, _timeout: Option<Duration>) {\n        unreachable!(\"server session: set_write_timeout\")\n    }\n\n    fn get_write_timeout(&self) -> Option<Duration> {\n        unreachable!(\"server_session: get_write_timeout\")\n    }\n\n    fn set_total_drain_timeout(&mut self, _timeout: Option<Duration>) {\n        unreachable!(\"server session: set_total_drain_timeout\")\n    }\n\n    fn get_total_drain_timeout(&self) -> Option<Duration> {\n        unreachable!(\"server_session: get_total_drain_timeout\")\n    }\n\n    fn request_summary(&self) -> String {\n        unreachable!(\"server session: request_summary\")\n    }\n\n    fn response_written(&self) -> Option<&ResponseHeader> {\n        unreachable!(\"server session: response_written\")\n    }\n\n    async fn shutdown(&mut self, _code: u32, _ctx: &str) {\n        unreachable!(\"server session: shutdown\")\n    }\n\n    fn is_body_done(&mut self) -> bool {\n        unreachable!(\"server session: is_body_done\")\n    }\n\n    async fn finish(&mut self) -> Result<()> {\n        unreachable!(\"server session: finish\")\n    }\n\n    fn is_body_empty(&mut self) -> bool {\n        unreachable!(\"server session: is_body_empty\")\n    }\n\n    async fn read_body_or_idle(&mut self, _no_body_expected: bool) -> Result<Option<Bytes>> {\n        unreachable!(\"server session: read_body_or_idle\")\n    }\n\n    fn body_bytes_sent(&self) -> usize {\n        unreachable!(\"server session: body_bytes_sent\")\n    }\n\n    fn body_bytes_read(&self) -> usize {\n        unreachable!(\"server session: body_bytes_read\")\n    }\n\n    fn digest(&self) -> Option<&Digest> {\n        unreachable!(\"server session: digest\")\n    }\n\n    fn digest_mut(&mut self) -> Option<&mut Digest> {\n        unreachable!(\"server session: digest_mut\")\n    }\n\n    fn client_addr(&self) -> Option<&SocketAddr> {\n        unreachable!(\"server session: client_addr\")\n    }\n\n    fn server_addr(&self) -> Option<&SocketAddr> {\n        unreachable!(\"server session: server_addr\")\n    }\n\n    fn pseudo_raw_h1_request_header(&self) -> Bytes {\n        unreachable!(\"server session: pseudo_raw_h1_request_header\")\n    }\n\n    fn enable_retry_buffering(&mut self) {\n        unreachable!(\"server session: enable_retry_bufferings\")\n    }\n\n    fn retry_buffer_truncated(&self) -> bool {\n        unreachable!(\"server session: retry_buffer_truncated\")\n    }\n\n    fn get_retry_buffer(&self) -> Option<Bytes> {\n        unreachable!(\"server session: get_retry_buffer\")\n    }\n\n    async fn finish_custom(&mut self) -> Result<()> {\n        unreachable!(\"server session: finish_custom\")\n    }\n\n    fn take_custom_message_reader(\n        &mut self,\n    ) -> Option<Box<dyn Stream<Item = Result<Bytes>> + Unpin + Send + Sync + 'static>> {\n        unreachable!(\"server session: get_custom_message_reader\")\n    }\n\n    fn restore_custom_message_reader(\n        &mut self,\n        _reader: Box<dyn Stream<Item = Result<Bytes>> + Unpin + Send + Sync + 'static>,\n    ) -> Result<()> {\n        unreachable!(\"server session: get_custom_message_reader\")\n    }\n\n    fn take_custom_message_writer(&mut self) -> Option<Box<dyn CustomMessageWrite>> {\n        unreachable!(\"server session: get_custom_message_writer\")\n    }\n\n    fn restore_custom_message_writer(\n        &mut self,\n        _writer: Box<dyn CustomMessageWrite>,\n    ) -> Result<()> {\n        unreachable!(\"server session: restore_custom_message_writer\")\n    }\n\n    fn is_upgrade_req(&self) -> bool {\n        unreachable!(\"server session: is_upgrade_req\")\n    }\n\n    fn was_upgraded(&self) -> bool {\n        unreachable!(\"server session: was_upgraded\")\n    }\n}\n"
  },
  {
    "path": "pingora-core/src/protocols/http/date.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\nuse chrono::DateTime;\nuse http::header::HeaderValue;\nuse std::cell::RefCell;\nuse std::time::{Duration, SystemTime};\n\nfn to_date_string(epoch_sec: i64) -> String {\n    let dt = DateTime::from_timestamp(epoch_sec, 0).unwrap();\n    dt.format(\"%a, %d %b %Y %H:%M:%S GMT\").to_string()\n}\n\nstruct CacheableDate {\n    h1_date: HeaderValue,\n    epoch: Duration,\n}\n\nimpl CacheableDate {\n    pub fn new() -> Self {\n        let d = SystemTime::now()\n            .duration_since(SystemTime::UNIX_EPOCH)\n            .unwrap();\n        CacheableDate {\n            h1_date: HeaderValue::from_str(&to_date_string(d.as_secs() as i64)).unwrap(),\n            epoch: d,\n        }\n    }\n\n    pub fn update(&mut self, d_now: Duration) {\n        if d_now.as_secs() != self.epoch.as_secs() {\n            self.epoch = d_now;\n            self.h1_date = HeaderValue::from_str(&to_date_string(d_now.as_secs() as i64)).unwrap();\n        }\n    }\n\n    pub fn get_date(&mut self) -> HeaderValue {\n        let d = SystemTime::now()\n            .duration_since(SystemTime::UNIX_EPOCH)\n            .unwrap();\n        self.update(d);\n        self.h1_date.clone()\n    }\n}\n\nthread_local! {\n    static CACHED_DATE: RefCell<CacheableDate>\n        = RefCell::new(CacheableDate::new());\n}\n\npub fn get_cached_date() -> HeaderValue {\n    CACHED_DATE.with(|cache_date| (*cache_date.borrow_mut()).get_date())\n}\n\n#[cfg(test)]\nmod test {\n    use super::*;\n\n    fn now_date_header() -> HeaderValue {\n        HeaderValue::from_str(&to_date_string(\n            SystemTime::now()\n                .duration_since(SystemTime::UNIX_EPOCH)\n                .unwrap()\n                .as_secs() as i64,\n        ))\n        .unwrap()\n    }\n\n    #[test]\n    fn test_date_string() {\n        let date_str = to_date_string(1);\n        assert_eq!(\"Thu, 01 Jan 1970 00:00:01 GMT\", date_str);\n    }\n\n    #[test]\n    fn test_date_cached() {\n        assert_eq!(get_cached_date(), now_date_header());\n    }\n}\n"
  },
  {
    "path": "pingora-core/src/protocols/http/error_resp.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! Error response generating utilities.\n\nuse http::header;\nuse once_cell::sync::Lazy;\nuse pingora_http::ResponseHeader;\n\nuse super::SERVER_NAME;\n\n/// Generate an error response with the given status code.\n///\n/// This error response has a zero `Content-Length` and `Cache-Control: private, no-store`.\npub fn gen_error_response(code: u16) -> ResponseHeader {\n    let mut resp = ResponseHeader::build(code, Some(4)).unwrap();\n    resp.insert_header(header::SERVER, &SERVER_NAME[..])\n        .unwrap();\n    resp.insert_header(header::DATE, \"Sun, 06 Nov 1994 08:49:37 GMT\")\n        .unwrap(); // placeholder\n    resp.insert_header(header::CONTENT_LENGTH, \"0\").unwrap();\n    resp.insert_header(header::CACHE_CONTROL, \"private, no-store\")\n        .unwrap();\n    resp\n}\n\n/// Pre-generated 502 response\npub static HTTP_502_RESPONSE: Lazy<ResponseHeader> = Lazy::new(|| gen_error_response(502));\n/// Pre-generated 400 response\npub static HTTP_400_RESPONSE: Lazy<ResponseHeader> = Lazy::new(|| gen_error_response(400));\n"
  },
  {
    "path": "pingora-core/src/protocols/http/mod.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! HTTP/1.x and HTTP/2 implementation APIs\n\npub mod body_buffer;\npub mod bridge;\npub mod client;\npub mod compression;\npub mod conditional_filter;\npub mod custom;\npub mod date;\npub mod error_resp;\npub mod server;\npub mod subrequest;\npub mod v1;\npub mod v2;\n\npub use server::Session as ServerSession;\n\n/// The Pingora server name string\npub const SERVER_NAME: &[u8; 7] = b\"Pingora\";\n\n/// An enum to hold all possible HTTP response events.\n#[derive(Debug)]\npub enum HttpTask {\n    /// the response header and the boolean end of response flag\n    Header(Box<pingora_http::ResponseHeader>, bool),\n    /// A piece of request or response body and the end of request/response boolean flag.\n    Body(Option<bytes::Bytes>, bool),\n    /// Request or response body bytes that have been upgraded on H1.1, and EOF bool flag.\n    UpgradedBody(Option<bytes::Bytes>, bool),\n    /// HTTP response trailer\n    Trailer(Option<Box<http::HeaderMap>>),\n    /// Signal that the response is already finished\n    Done,\n    /// Signal that the reading of the response encountered errors.\n    Failed(pingora_error::BError),\n}\n\nimpl HttpTask {\n    /// Whether this [`HttpTask`] means the end of the response.\n    pub fn is_end(&self) -> bool {\n        match self {\n            HttpTask::Header(_, end) => *end,\n            HttpTask::Body(_, end) => *end,\n            HttpTask::UpgradedBody(_, end) => *end,\n            HttpTask::Trailer(_) => true,\n            HttpTask::Done => true,\n            HttpTask::Failed(_) => true,\n        }\n    }\n\n    /// The [`HttpTask`] type as string.\n    pub fn type_str(&self) -> &'static str {\n        match self {\n            HttpTask::Header(..) => \"Header\",\n            HttpTask::Body(..) => \"Body\",\n            HttpTask::UpgradedBody(..) => \"UpgradedBody\",\n            HttpTask::Trailer(_) => \"Trailer\",\n            HttpTask::Done => \"Done\",\n            HttpTask::Failed(_) => \"Failed\",\n        }\n    }\n}\n"
  },
  {
    "path": "pingora-core/src/protocols/http/server.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! HTTP server session APIs\n\nuse super::custom::server::Session as SessionCustom;\nuse super::error_resp;\nuse super::subrequest::server::HttpSession as SessionSubrequest;\nuse super::v1::server::HttpSession as SessionV1;\nuse super::v2::server::HttpSession as SessionV2;\nuse super::HttpTask;\nuse crate::custom_session;\nuse crate::protocols::{Digest, SocketAddr, Stream};\nuse bytes::Bytes;\nuse http::HeaderValue;\nuse http::{header::AsHeaderName, HeaderMap};\nuse pingora_error::{Error, Result};\nuse pingora_http::{RequestHeader, ResponseHeader};\nuse std::time::Duration;\n\n/// HTTP server session object for both HTTP/1.x and HTTP/2\npub enum Session {\n    H1(SessionV1),\n    H2(SessionV2),\n    Subrequest(SessionSubrequest),\n    Custom(Box<dyn SessionCustom>),\n}\n\nimpl Session {\n    /// Create a new [`Session`] from an established connection for HTTP/1.x\n    pub fn new_http1(stream: Stream) -> Self {\n        Self::H1(SessionV1::new(stream))\n    }\n\n    /// Create a new [`Session`] from an established HTTP/2 stream\n    pub fn new_http2(session: SessionV2) -> Self {\n        Self::H2(session)\n    }\n\n    /// Create a new [`Session`] from a subrequest session\n    pub fn new_subrequest(session: SessionSubrequest) -> Self {\n        Self::Subrequest(session)\n    }\n\n    /// Create a new [`Session`] from a custom session\n    pub fn new_custom(session: Box<dyn SessionCustom>) -> Self {\n        Self::Custom(session)\n    }\n\n    /// Whether the session is HTTP/2. If not it is HTTP/1.x\n    pub fn is_http2(&self) -> bool {\n        matches!(self, Self::H2(_))\n    }\n\n    /// Whether the session is for a subrequest.\n    pub fn is_subrequest(&self) -> bool {\n        matches!(self, Self::Subrequest(_))\n    }\n\n    /// Whether the session is Custom\n    pub fn is_custom(&self) -> bool {\n        matches!(self, Self::Custom(_))\n    }\n\n    /// Read the request header. This method is required to be called first before doing anything\n    /// else with the session.\n    /// - `Ok(true)`: successful\n    /// - `Ok(false)`: client exit without sending any bytes. This is normal on reused connection.\n    ///   In this case the user should give up this session.\n    pub async fn read_request(&mut self) -> Result<bool> {\n        match self {\n            Self::H1(s) => {\n                let read = s.read_request().await?;\n                Ok(read.is_some())\n            }\n            // This call will always return `Ok(true)` for Http2 because the request is already read\n            Self::H2(_) => Ok(true),\n            Self::Subrequest(s) => {\n                let read = s.read_request().await?;\n                Ok(read.is_some())\n            }\n            Self::Custom(_) => Ok(true),\n        }\n    }\n\n    /// Return the request header it just read.\n    /// # Panic\n    /// This function will panic if [`Self::read_request()`] is not called.\n    pub fn req_header(&self) -> &RequestHeader {\n        match self {\n            Self::H1(s) => s.req_header(),\n            Self::H2(s) => s.req_header(),\n            Self::Subrequest(s) => s.req_header(),\n            Self::Custom(s) => s.req_header(),\n        }\n    }\n\n    /// Return a mutable reference to request header it just read.\n    /// # Panic\n    /// This function will panic if [`Self::read_request()`] is not called.\n    pub fn req_header_mut(&mut self) -> &mut RequestHeader {\n        match self {\n            Self::H1(s) => s.req_header_mut(),\n            Self::H2(s) => s.req_header_mut(),\n            Self::Subrequest(s) => s.req_header_mut(),\n            Self::Custom(s) => s.req_header_mut(),\n        }\n    }\n\n    /// Return the header by name. None if the header doesn't exist.\n    ///\n    /// In case there are multiple headers under the same name, the first one will be returned. To\n    /// get all the headers: use `self.req_header().headers.get_all()`.\n    pub fn get_header<K: AsHeaderName>(&self, key: K) -> Option<&HeaderValue> {\n        self.req_header().headers.get(key)\n    }\n\n    /// Get the header value in its raw format.\n    /// If the header doesn't exist, return an empty slice.\n    pub fn get_header_bytes<K: AsHeaderName>(&self, key: K) -> &[u8] {\n        self.get_header(key).map_or(b\"\", |v| v.as_bytes())\n    }\n\n    /// Read the request body. Ok(None) if no (more) body to read\n    pub async fn read_request_body(&mut self) -> Result<Option<Bytes>> {\n        match self {\n            Self::H1(s) => s.read_body_bytes().await,\n            Self::H2(s) => s.read_body_bytes().await,\n            Self::Subrequest(s) => s.read_body_bytes().await,\n            Self::Custom(s) => s.read_body_bytes().await,\n        }\n    }\n\n    /// Discard the request body by reading it until completion.\n    ///\n    /// This is useful for making streams reusable (in particular for HTTP/1.1) after returning an\n    /// error before the whole body has been read.\n    pub async fn drain_request_body(&mut self) -> Result<()> {\n        match self {\n            Self::H1(s) => s.drain_request_body().await,\n            Self::H2(s) => s.drain_request_body().await,\n            Self::Subrequest(s) => s.drain_request_body().await,\n            Self::Custom(s) => s.drain_request_body().await,\n        }\n    }\n\n    /// Write the response header to client\n    /// Informational headers (status code 100-199, excluding 101) can be written multiple times the final\n    /// response header (status code 200+ or 101) is written.\n    pub async fn write_response_header(&mut self, resp: Box<ResponseHeader>) -> Result<()> {\n        match self {\n            Self::H1(s) => {\n                s.write_response_header(resp).await?;\n                Ok(())\n            }\n            Self::H2(s) => s.write_response_header(resp, false),\n            Self::Subrequest(s) => {\n                s.write_response_header(resp).await?;\n                Ok(())\n            }\n            Self::Custom(s) => s.write_response_header(resp, false).await,\n        }\n    }\n\n    /// Similar to `write_response_header()`, this fn will clone the `resp` internally\n    pub async fn write_response_header_ref(&mut self, resp: &ResponseHeader) -> Result<()> {\n        match self {\n            Self::H1(s) => {\n                s.write_response_header_ref(resp).await?;\n                Ok(())\n            }\n            Self::H2(s) => s.write_response_header_ref(resp, false),\n            Self::Subrequest(s) => {\n                s.write_response_header_ref(resp).await?;\n                Ok(())\n            }\n            Self::Custom(s) => s.write_response_header_ref(resp, false).await,\n        }\n    }\n\n    /// Write the response body to client\n    pub async fn write_response_body(&mut self, data: Bytes, end: bool) -> Result<()> {\n        if data.is_empty() && !end {\n            // writing 0 byte to a chunked encoding h1 would finish the stream\n            // writing 0 bytes to h2 is noop\n            // we don't want to actually write in either cases\n            return Ok(());\n        }\n        match self {\n            Self::H1(s) => {\n                if !data.is_empty() {\n                    s.write_body(&data).await?;\n                }\n                if end {\n                    s.finish_body().await?;\n                }\n                Ok(())\n            }\n            Self::H2(s) => s.write_body(data, end).await,\n            Self::Subrequest(s) => {\n                s.write_body(data).await?;\n                Ok(())\n            }\n            Self::Custom(s) => s.write_body(data, end).await,\n        }\n    }\n\n    /// Write the response trailers to client\n    pub async fn write_response_trailers(&mut self, trailers: HeaderMap) -> Result<()> {\n        match self {\n            Self::H1(_) => Ok(()), // TODO: support trailers for h1\n            Self::H2(s) => s.write_trailers(trailers),\n            Self::Subrequest(s) => s.write_trailers(Some(Box::new(trailers))).await,\n            Self::Custom(s) => s.write_trailers(trailers).await,\n        }\n    }\n\n    /// Finish the life of this request.\n    /// For H1, if connection reuse is supported, a Some(Stream) will be returned, otherwise None.\n    /// For H2, always return None because H2 stream is not reusable.\n    /// For subrequests, there is no true underlying stream to return.\n    pub async fn finish(self) -> Result<Option<Stream>> {\n        match self {\n            Self::H1(mut s) => {\n                // need to flush body due to buffering\n                s.finish_body().await?;\n                s.reuse().await\n            }\n            Self::H2(mut s) => {\n                s.finish()?;\n                Ok(None)\n            }\n            Self::Subrequest(mut s) => {\n                s.finish().await?;\n                Ok(None)\n            }\n            Self::Custom(mut s) => {\n                s.finish().await?;\n                Ok(None)\n            }\n        }\n    }\n\n    /// Callback for cleanup logic on downstream specifically when we fail to proxy the session\n    /// other than cleanup via finish().\n    ///\n    /// If caching the downstream failure may be independent of (and precede) an upstream error in\n    /// which case this function may be called more than once.\n    pub fn on_proxy_failure(&mut self, e: Box<Error>) {\n        match self {\n            Self::H1(_) | Self::H2(_) | Self::Custom(_) => {\n                // all cleanup logic handled in finish(),\n                // stream and resources dropped when session dropped\n            }\n            Self::Subrequest(ref mut s) => s.on_proxy_failure(e),\n        }\n    }\n\n    pub async fn response_duplex_vec(&mut self, tasks: Vec<HttpTask>) -> Result<bool> {\n        match self {\n            Self::H1(s) => s.response_duplex_vec(tasks).await,\n            Self::H2(s) => s.response_duplex_vec(tasks).await,\n            Self::Subrequest(s) => s.response_duplex_vec(tasks).await,\n            Self::Custom(s) => s.response_duplex_vec(tasks).await,\n        }\n    }\n\n    /// Set connection reuse. `duration` defines how long the connection is kept open for the next\n    /// request to reuse. Noop for h2 and subrequest\n    pub fn set_keepalive(&mut self, duration: Option<u64>) {\n        match self {\n            Self::H1(s) => s.set_server_keepalive(duration),\n            Self::H2(_) => {}\n            Self::Subrequest(_) => {}\n            Self::Custom(_) => {}\n        }\n    }\n\n    /// Get the keepalive timeout. None if keepalive is disabled. Not applicable for h2 or\n    /// subrequest\n    pub fn get_keepalive(&self) -> Option<u64> {\n        match self {\n            Self::H1(s) => s.get_keepalive_timeout(),\n            Self::H2(_) => None,\n            Self::Subrequest(_) => None,\n            Self::Custom(_) => None,\n        }\n    }\n\n    /// Set the number of times the upstream connection connection for this\n    /// session can be reused via keepalive. Noop for h2 and subrequest\n    pub fn set_keepalive_reuses_remaining(&mut self, reuses: Option<u32>) {\n        if let Self::H1(s) = self {\n            s.set_keepalive_reuses_remaining(reuses);\n        }\n    }\n\n    /// Get the number of times the upstream connection connection for this\n    /// session can be reused via keepalive. Not applicable for h2 or\n    /// subrequest\n    pub fn get_keepalive_reuses_remaining(&self) -> Option<u32> {\n        if let Self::H1(s) = self {\n            s.get_keepalive_reuses_remaining()\n        } else {\n            None\n        }\n    }\n\n    /// Sets the downstream read timeout. This will trigger if we're unable\n    /// to read from the stream after `timeout`.\n    ///\n    /// This is a noop for h2.\n    pub fn set_read_timeout(&mut self, timeout: Option<Duration>) {\n        match self {\n            Self::H1(s) => s.set_read_timeout(timeout),\n            Self::H2(_) => {}\n            Self::Subrequest(s) => s.set_read_timeout(timeout),\n            Self::Custom(c) => c.set_read_timeout(timeout),\n        }\n    }\n\n    /// Gets the downstream read timeout if set.\n    pub fn get_read_timeout(&self) -> Option<Duration> {\n        match self {\n            Self::H1(s) => s.get_read_timeout(),\n            Self::H2(_) => None,\n            Self::Subrequest(s) => s.get_read_timeout(),\n            Self::Custom(s) => s.get_read_timeout(),\n        }\n    }\n\n    /// Sets the downstream write timeout. This will trigger if we're unable\n    /// to write to the stream after `timeout`. If a `min_send_rate` is\n    /// configured then the `min_send_rate` calculated timeout has higher priority.\n    pub fn set_write_timeout(&mut self, timeout: Option<Duration>) {\n        match self {\n            Self::H1(s) => s.set_write_timeout(timeout),\n            Self::H2(s) => s.set_write_timeout(timeout),\n            Self::Subrequest(s) => s.set_write_timeout(timeout),\n            Self::Custom(c) => c.set_write_timeout(timeout),\n        }\n    }\n\n    /// Gets the downstream write timeout if set.\n    pub fn get_write_timeout(&self) -> Option<Duration> {\n        match self {\n            Self::H1(s) => s.get_write_timeout(),\n            Self::H2(s) => s.get_write_timeout(),\n            Self::Subrequest(s) => s.get_write_timeout(),\n            Self::Custom(s) => s.get_write_timeout(),\n        }\n    }\n\n    /// Sets the total drain timeout, which will be applied while discarding the\n    /// request body using `drain_request_body`.\n    ///\n    /// For HTTP/1.1, reusing a session requires ensuring that the request body\n    /// is consumed. If the timeout is exceeded, the caller should give up on\n    /// trying to reuse the session.\n    pub fn set_total_drain_timeout(&mut self, timeout: Option<Duration>) {\n        match self {\n            Self::H1(s) => s.set_total_drain_timeout(timeout),\n            Self::H2(s) => s.set_total_drain_timeout(timeout),\n            Self::Subrequest(s) => s.set_total_drain_timeout(timeout),\n            Self::Custom(c) => c.set_total_drain_timeout(timeout),\n        }\n    }\n\n    /// Gets the total drain timeout if set.\n    pub fn get_total_drain_timeout(&self) -> Option<Duration> {\n        match self {\n            Self::H1(s) => s.get_total_drain_timeout(),\n            Self::H2(s) => s.get_total_drain_timeout(),\n            Self::Subrequest(s) => s.get_total_drain_timeout(),\n            Self::Custom(s) => s.get_total_drain_timeout(),\n        }\n    }\n\n    /// Sets the minimum downstream send rate in bytes per second. This\n    /// is used to calculate a write timeout in seconds based on the size\n    /// of the buffer being written. If a `min_send_rate` is configured it\n    /// has higher priority over a set `write_timeout`. The minimum send\n    /// rate must be greater than zero.\n    ///\n    /// Calculated write timeout is guaranteed to be at least 1s if `min_send_rate`\n    /// is greater than zero, a send rate of zero is equivalent to disabling.\n    ///\n    /// This is a noop for h2.\n    pub fn set_min_send_rate(&mut self, rate: Option<usize>) {\n        match self {\n            Self::H1(s) => s.set_min_send_rate(rate),\n            Self::H2(_) => {}\n            Self::Subrequest(_) => {}\n            Self::Custom(_) => {}\n        }\n    }\n\n    /// Sets whether we ignore writing informational responses downstream.\n    ///\n    /// For HTTP/1.1 this is a noop if the response is Upgrade or Continue and\n    /// Expect: 100-continue was set on the request.\n    ///\n    /// This is a noop for h2 because informational responses are always ignored.\n    /// Subrequests will always proxy the info response and let the true downstream\n    /// decide to ignore or not.\n    pub fn set_ignore_info_resp(&mut self, ignore: bool) {\n        match self {\n            Self::H1(s) => s.set_ignore_info_resp(ignore),\n            Self::H2(_) => {} // always ignored\n            Self::Subrequest(_) => {}\n            Self::Custom(_) => {} // always ignored\n        }\n    }\n\n    /// Sets whether keepalive should be disabled if response is written prior to\n    /// downstream body finishing.\n    ///\n    /// This is a noop for h2.\n    pub fn set_close_on_response_before_downstream_finish(&mut self, close: bool) {\n        match self {\n            Self::H1(s) => s.set_close_on_response_before_downstream_finish(close),\n            Self::H2(_) => {}         // always ignored\n            Self::Subrequest(_) => {} // always ignored\n            Self::Custom(_) => {}     // always ignored\n        }\n    }\n\n    /// Return a digest of the request including the method, path and Host header\n    // TODO: make this use a `Formatter`\n    pub fn request_summary(&self) -> String {\n        match self {\n            Self::H1(s) => s.request_summary(),\n            Self::H2(s) => s.request_summary(),\n            Self::Subrequest(s) => s.request_summary(),\n            Self::Custom(s) => s.request_summary(),\n        }\n    }\n\n    /// Return the written response header. `None` if it is not written yet.\n    /// Only the final (status code >= 200 or 101) response header will be returned\n    pub fn response_written(&self) -> Option<&ResponseHeader> {\n        match self {\n            Self::H1(s) => s.response_written(),\n            Self::H2(s) => s.response_written(),\n            Self::Subrequest(s) => s.response_written(),\n            Self::Custom(s) => s.response_written(),\n        }\n    }\n\n    /// Give up the http session abruptly.\n    /// For H1 this will close the underlying connection\n    /// For H2 this will send RESET frame to end this stream without impacting the connection\n    /// For subrequests, this will drop task senders and receivers.\n    pub async fn shutdown(&mut self) {\n        match self {\n            Self::H1(s) => s.shutdown().await,\n            Self::H2(s) => s.shutdown(),\n            Self::Subrequest(s) => s.shutdown(),\n            Self::Custom(s) => s.shutdown(0, \"shutdown\").await,\n        }\n    }\n\n    pub fn to_h1_raw(&self) -> Bytes {\n        match self {\n            Self::H1(s) => s.get_headers_raw_bytes(),\n            Self::H2(s) => s.pseudo_raw_h1_request_header(),\n            Self::Subrequest(s) => s.get_headers_raw_bytes(),\n            Self::Custom(c) => c.pseudo_raw_h1_request_header(),\n        }\n    }\n\n    /// Whether the whole request body is sent\n    pub fn is_body_done(&mut self) -> bool {\n        match self {\n            Self::H1(s) => s.is_body_done(),\n            Self::H2(s) => s.is_body_done(),\n            Self::Subrequest(s) => s.is_body_done(),\n            Self::Custom(s) => s.is_body_done(),\n        }\n    }\n\n    /// Notify the client that the entire body is sent\n    /// for H1 chunked encoding, this will end the last empty chunk\n    /// for H1 content-length, this has no effect.\n    /// for H2, this will send an empty DATA frame with END_STREAM flag\n    /// for subrequest, this will send a Done http task\n    pub async fn finish_body(&mut self) -> Result<()> {\n        match self {\n            Self::H1(s) => s.finish_body().await.map(|_| ()),\n            Self::H2(s) => s.finish(),\n            Self::Subrequest(s) => s.finish().await.map(|_| ()),\n            Self::Custom(s) => s.finish().await,\n        }\n    }\n\n    pub fn generate_error(error: u16) -> ResponseHeader {\n        match error {\n            /* common error responses are pre-generated */\n            502 => error_resp::HTTP_502_RESPONSE.clone(),\n            400 => error_resp::HTTP_400_RESPONSE.clone(),\n            _ => error_resp::gen_error_response(error),\n        }\n    }\n\n    /// Send error response to client using a pre-generated error message.\n    pub async fn respond_error(&mut self, error: u16) -> Result<()> {\n        self.respond_error_with_body(error, Bytes::default()).await\n    }\n\n    /// Send error response to client using a pre-generated error message and custom body.\n    pub async fn respond_error_with_body(&mut self, error: u16, body: Bytes) -> Result<()> {\n        let mut resp = Self::generate_error(error);\n        if !body.is_empty() {\n            // error responses have a default content-length of zero\n            resp.set_content_length(body.len())?\n        }\n        self.write_error_response(resp, body).await\n    }\n\n    /// Send an error response to a client with a response header and body.\n    pub async fn write_error_response(&mut self, resp: ResponseHeader, body: Bytes) -> Result<()> {\n        // TODO: we shouldn't be closing downstream connections on internally generated errors\n        // and possibly other upstream connect() errors (connection refused, timeout, etc)\n        //\n        // This change is only here because we DO NOT re-use downstream connections\n        // today on these errors and we should signal to the client that pingora is dropping it\n        // rather than a misleading the client with 'keep-alive'\n        self.set_keepalive(None);\n\n        // If a response was already written and it's not informational 1xx, return.\n        // The only exception is an informational 101 Switching Protocols, which is treated\n        // as final response https://www.rfc-editor.org/rfc/rfc9110#section-15.2.2.\n        if let Some(resp_written) = self.response_written().as_ref() {\n            if !resp_written.status.is_informational() || resp_written.status == 101 {\n                return Ok(());\n            }\n        }\n\n        self.write_response_header(Box::new(resp)).await?;\n\n        if !body.is_empty() {\n            self.write_response_body(body, true).await?;\n        } else {\n            self.finish_body().await?;\n        }\n\n        custom_session!(self.finish_custom().await?);\n\n        Ok(())\n    }\n\n    /// Whether there is no request body\n    pub fn is_body_empty(&mut self) -> bool {\n        match self {\n            Self::H1(s) => s.is_body_empty(),\n            Self::H2(s) => s.is_body_empty(),\n            Self::Subrequest(s) => s.is_body_empty(),\n            Self::Custom(s) => s.is_body_empty(),\n        }\n    }\n\n    pub fn retry_buffer_truncated(&self) -> bool {\n        match self {\n            Self::H1(s) => s.retry_buffer_truncated(),\n            Self::H2(s) => s.retry_buffer_truncated(),\n            Self::Subrequest(s) => s.retry_buffer_truncated(),\n            Self::Custom(s) => s.retry_buffer_truncated(),\n        }\n    }\n\n    pub fn enable_retry_buffering(&mut self) {\n        match self {\n            Self::H1(s) => s.enable_retry_buffering(),\n            Self::H2(s) => s.enable_retry_buffering(),\n            Self::Subrequest(s) => s.enable_retry_buffering(),\n            Self::Custom(s) => s.enable_retry_buffering(),\n        }\n    }\n\n    pub fn get_retry_buffer(&self) -> Option<Bytes> {\n        match self {\n            Self::H1(s) => s.get_retry_buffer(),\n            Self::H2(s) => s.get_retry_buffer(),\n            Self::Subrequest(s) => s.get_retry_buffer(),\n            Self::Custom(s) => s.get_retry_buffer(),\n        }\n    }\n\n    /// Read body (same as `read_request_body()`) or pending forever until downstream\n    /// terminates the session.\n    pub async fn read_body_or_idle(&mut self, no_body_expected: bool) -> Result<Option<Bytes>> {\n        match self {\n            Self::H1(s) => s.read_body_or_idle(no_body_expected).await,\n            Self::H2(s) => s.read_body_or_idle(no_body_expected).await,\n            Self::Subrequest(s) => s.read_body_or_idle(no_body_expected).await,\n            Self::Custom(s) => s.read_body_or_idle(no_body_expected).await,\n        }\n    }\n\n    pub fn as_http1(&self) -> Option<&SessionV1> {\n        match self {\n            Self::H1(s) => Some(s),\n            Self::H2(_) => None,\n            Self::Subrequest(_) => None,\n            Self::Custom(_) => None,\n        }\n    }\n\n    pub fn as_http2(&self) -> Option<&SessionV2> {\n        match self {\n            Self::H1(_) => None,\n            Self::H2(s) => Some(s),\n            Self::Subrequest(_) => None,\n            Self::Custom(_) => None,\n        }\n    }\n\n    pub fn as_subrequest(&self) -> Option<&SessionSubrequest> {\n        match self {\n            Self::H1(_) => None,\n            Self::H2(_) => None,\n            Self::Subrequest(s) => Some(s),\n            Self::Custom(_) => None,\n        }\n    }\n\n    pub fn as_subrequest_mut(&mut self) -> Option<&mut SessionSubrequest> {\n        match self {\n            Self::H1(_) => None,\n            Self::H2(_) => None,\n            Self::Subrequest(s) => Some(s),\n            Self::Custom(_) => None,\n        }\n    }\n\n    pub fn as_custom(&self) -> Option<&dyn SessionCustom> {\n        match self {\n            Self::H1(_) => None,\n            Self::H2(_) => None,\n            Self::Subrequest(_) => None,\n            Self::Custom(c) => Some(c.as_ref()),\n        }\n    }\n\n    pub fn as_custom_mut(&mut self) -> Option<&mut Box<dyn SessionCustom>> {\n        match self {\n            Self::H1(_) => None,\n            Self::H2(_) => None,\n            Self::Subrequest(_) => None,\n            Self::Custom(c) => Some(c),\n        }\n    }\n\n    /// Write a 100 Continue response to the client.\n    pub async fn write_continue_response(&mut self) -> Result<()> {\n        match self {\n            Self::H1(s) => s.write_continue_response().await,\n            Self::H2(s) => s.write_response_header(\n                Box::new(ResponseHeader::build(100, Some(0)).unwrap()),\n                false,\n            ),\n            Self::Subrequest(s) => s.write_continue_response().await,\n            // TODO(slava): is there any write_continue_response calls?\n            Self::Custom(s) => {\n                s.write_response_header(\n                    Box::new(ResponseHeader::build(100, Some(0)).unwrap()),\n                    false,\n                )\n                .await\n            }\n        }\n    }\n\n    /// Whether this request is for upgrade (e.g., websocket).\n    pub fn is_upgrade_req(&self) -> bool {\n        match self {\n            Self::H1(s) => s.is_upgrade_req(),\n            Self::H2(_) => false,\n            Self::Subrequest(s) => s.is_upgrade_req(),\n            Self::Custom(s) => s.is_upgrade_req(),\n        }\n    }\n\n    /// Whether this session was fully upgraded (completed Upgrade handshake).\n    pub fn was_upgraded(&self) -> bool {\n        match self {\n            Self::H1(s) => s.was_upgraded(),\n            Self::H2(_) => false,\n            Self::Subrequest(s) => s.was_upgraded(),\n            Self::Custom(s) => s.was_upgraded(),\n        }\n    }\n\n    /// Return how many response body bytes (application, not wire) already sent downstream\n    pub fn body_bytes_sent(&self) -> usize {\n        match self {\n            Self::H1(s) => s.body_bytes_sent(),\n            Self::H2(s) => s.body_bytes_sent(),\n            Self::Subrequest(s) => s.body_bytes_sent(),\n            Self::Custom(s) => s.body_bytes_sent(),\n        }\n    }\n\n    /// Return how many request body bytes (application, not wire) already read from downstream\n    pub fn body_bytes_read(&self) -> usize {\n        match self {\n            Self::H1(s) => s.body_bytes_read(),\n            Self::H2(s) => s.body_bytes_read(),\n            Self::Subrequest(s) => s.body_bytes_read(),\n            Self::Custom(s) => s.body_bytes_read(),\n        }\n    }\n\n    /// Return the [Digest] for the connection.\n    pub fn digest(&self) -> Option<&Digest> {\n        match self {\n            Self::H1(s) => Some(s.digest()),\n            Self::H2(s) => s.digest(),\n            Self::Subrequest(s) => s.digest(),\n            Self::Custom(s) => s.digest(),\n        }\n    }\n\n    /// Return a mutable [Digest] reference for the connection.\n    ///\n    /// Will return `None` if multiple H2 streams are open.\n    pub fn digest_mut(&mut self) -> Option<&mut Digest> {\n        match self {\n            Self::H1(s) => Some(s.digest_mut()),\n            Self::H2(s) => s.digest_mut(),\n            Self::Subrequest(s) => s.digest_mut(),\n            Self::Custom(s) => s.digest_mut(),\n        }\n    }\n\n    /// Return the client (peer) address of the connection.\n    pub fn client_addr(&self) -> Option<&SocketAddr> {\n        match self {\n            Self::H1(s) => s.client_addr(),\n            Self::H2(s) => s.client_addr(),\n            Self::Subrequest(s) => s.client_addr(),\n            Self::Custom(s) => s.client_addr(),\n        }\n    }\n\n    /// Return the server (local) address of the connection.\n    pub fn server_addr(&self) -> Option<&SocketAddr> {\n        match self {\n            Self::H1(s) => s.server_addr(),\n            Self::H2(s) => s.server_addr(),\n            Self::Subrequest(s) => s.server_addr(),\n            Self::Custom(s) => s.server_addr(),\n        }\n    }\n\n    /// Get the reference of the [Stream] that this HTTP/1 session is operating upon.\n    /// None if the HTTP session is over H2, or a subrequest\n    pub fn stream(&self) -> Option<&Stream> {\n        match self {\n            Self::H1(s) => Some(s.stream()),\n            Self::H2(_) => None,\n            Self::Subrequest(_) => None,\n            Self::Custom(_) => None,\n        }\n    }\n}\n"
  },
  {
    "path": "pingora-core/src/protocols/http/subrequest/body.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! Subrequest body reader and writer.\n//!\n//! This implementation is very similar to v1 if not identical in many cases.\n//! However it is generally much simpler because it does not have to handle\n//! wire format bytes, simply basic checks such as content-length and when the\n//! underlying channel (sender or receiver) is closed.\n\nuse bytes::Bytes;\nuse log::{debug, trace, warn};\nuse pingora_error::{\n    Error,\n    ErrorType::{self, *},\n    OrErr, Result,\n};\nuse std::fmt::Debug;\nuse tokio::sync::{mpsc, oneshot};\n\nuse crate::protocols::http::HttpTask;\nuse http::HeaderMap;\n\npub const PREMATURE_BODY_END: ErrorType = ErrorType::new(\"PrematureBodyEnd\");\n\n#[derive(Clone, Debug, PartialEq, Eq)]\npub enum ParseState {\n    ToStart,\n    Complete(usize),       // total size\n    Partial(usize, usize), // size read, remaining size\n    Done(usize),           // done but there is error, size read\n    UntilClose(usize),     // read until connection closed, size read\n}\n\ntype PS = ParseState;\n\npub struct BodyReader {\n    pub body_state: ParseState,\n    notify_wants_body: Option<oneshot::Sender<()>>,\n}\n\nimpl BodyReader {\n    pub fn new(notify_wants_body: Option<oneshot::Sender<()>>) -> Self {\n        BodyReader {\n            body_state: PS::ToStart,\n            notify_wants_body,\n        }\n        // TODO: if wants body signal is None, init empty\n    }\n\n    pub fn need_init(&self) -> bool {\n        matches!(self.body_state, PS::ToStart)\n    }\n\n    pub fn init_content_length(&mut self, cl: usize) {\n        match cl {\n            0 => self.body_state = PS::Complete(0),\n            _ => {\n                self.body_state = PS::Partial(0, cl);\n            }\n        }\n    }\n\n    pub fn init_close_delimited(&mut self) {\n        self.body_state = PS::UntilClose(0);\n    }\n\n    /// Convert how we interpret the remainder of the body to read until close.\n    /// This is used for responses without explicit framing.\n    pub fn convert_to_close_delimited(&mut self) {\n        if matches!(self.body_state, PS::UntilClose(_)) {\n            // nothing to do, already in close-delimited mode\n            return;\n        }\n\n        // reset body counter\n        self.body_state = PS::UntilClose(0);\n    }\n\n    pub fn body_done(&self) -> bool {\n        matches!(self.body_state, PS::Complete(_) | PS::Done(_))\n    }\n\n    pub fn body_empty(&self) -> bool {\n        self.body_state == PS::Complete(0)\n    }\n\n    pub async fn read_body(&mut self, rx: &mut mpsc::Receiver<HttpTask>) -> Result<Option<Bytes>> {\n        match self.body_state {\n            PS::Complete(_) => Ok(None),\n            PS::Done(_) => Ok(None),\n            PS::Partial(_, _) => self.do_read_body(rx).await,\n            PS::UntilClose(_) => self.do_read_body_until_closed(rx).await,\n            PS::ToStart => panic!(\"need to init BodyReader first\"),\n        }\n    }\n\n    pub async fn do_read_body(\n        &mut self,\n        rx: &mut mpsc::Receiver<HttpTask>,\n    ) -> Result<Option<Bytes>> {\n        if let Some(notify) = self.notify_wants_body.take() {\n            // fine if downstream isn't actively being read\n            let _ = notify.send(());\n        }\n        let (bytes, end) = match rx.recv().await {\n            Some(HttpTask::Body(bytes, end)) => (bytes, end),\n            Some(task) => {\n                // TODO: return an error into_down for Failed?\n                return Error::e_explain(\n                    InternalError,\n                    format!(\"Unexpected HttpTask {task:?} while reading body (subrequest)\"),\n                );\n            }\n            None => (None, true), // downstream ended\n        };\n\n        match self.body_state {\n            PS::Partial(read, to_read) => {\n                let n = bytes.as_ref().map_or(0, |b| b.len());\n                debug!(\n                    \"BodyReader body_state: {:?}, read data from IO: {n} (subrequest)\",\n                    self.body_state,\n                );\n                if bytes.is_none() {\n                    self.body_state = PS::Done(read);\n                    return Error::e_explain(ConnectionClosed, format!(\n                        \"Peer prematurely closed connection with {to_read} bytes of body remaining to read (subrequest)\",\n                    ));\n                }\n                if end && n < to_read {\n                    // TODO: this doesn't flush the bytes we did receive to upstream\n                    self.body_state = PS::Done(read + n);\n                    return Error::e_explain(PREMATURE_BODY_END, format!(\n                        \"Peer prematurely ended body with {} bytes of body remaining to read (subrequest)\",\n                        to_read - n\n                    ));\n                }\n                if n >= to_read {\n                    if n > to_read {\n                        warn!(\n                            \"Peer sent more data then expected: extra {}\\\n                               bytes, discarding them (subrequest)\",\n                            n - to_read\n                        );\n                    }\n                    self.body_state = PS::Complete(read + to_read);\n                    Ok(bytes.map(|b| b.slice(0..to_read)))\n                } else {\n                    self.body_state = PS::Partial(read + n, to_read - n);\n                    Ok(bytes)\n                }\n            }\n            _ => panic!(\"wrong body state: {:?} (subrequest)\", self.body_state),\n        }\n    }\n\n    pub async fn do_read_body_until_closed(\n        &mut self,\n        rx: &mut mpsc::Receiver<HttpTask>,\n    ) -> Result<Option<Bytes>> {\n        if let Some(notify) = self.notify_wants_body.take() {\n            // fine if downstream isn't active, receiver will indicate this\n            let _ = notify.send(());\n        }\n\n        let (bytes, end) = match rx.recv().await {\n            Some(HttpTask::Body(bytes, end)) => (bytes, end),\n            Some(task) => {\n                return Error::e_explain(\n                    InternalError,\n                    format!(\"Unexpected HttpTask {task:?} while reading body (subrequest)\"),\n                );\n            }\n            None => (None, true), // downstream ended\n        };\n        let n = bytes.as_ref().map_or(0, |b| b.len());\n        match self.body_state {\n            PS::UntilClose(read) => {\n                if bytes.is_none() {\n                    self.body_state = PS::Complete(read);\n                    Ok(None)\n                } else if end {\n                    // explicit end also signifies completion\n                    self.body_state = PS::Complete(read + n);\n                    Ok(bytes)\n                } else {\n                    self.body_state = PS::UntilClose(read + n);\n                    Ok(bytes)\n                }\n            }\n            _ => panic!(\"wrong body state: {:?} (subrequest)\", self.body_state),\n        }\n    }\n}\n\n#[derive(Clone, Debug, PartialEq, Eq)]\npub enum BodyMode {\n    ToSelect,\n    ContentLength(usize, usize), // total length to write, bytes already written\n    UntilClose(usize),           //bytes written\n    Complete(usize),             //bytes written\n}\n\ntype BM = BodyMode;\n\npub struct BodyWriter {\n    pub body_mode: BodyMode,\n}\n\nimpl BodyWriter {\n    pub fn new() -> Self {\n        BodyWriter {\n            body_mode: BM::ToSelect,\n        }\n    }\n\n    pub fn init_close_delimited(&mut self) {\n        self.body_mode = BM::UntilClose(0);\n    }\n\n    pub fn init_content_length(&mut self, cl: usize) {\n        self.body_mode = BM::ContentLength(cl, 0);\n    }\n\n    pub async fn write_body(\n        &mut self,\n        sender: &mut mpsc::Sender<HttpTask>,\n        bytes: Bytes,\n    ) -> Result<Option<usize>> {\n        trace!(\"Writing Body, size: {} (subrequest)\", bytes.len());\n        match self.body_mode {\n            BM::Complete(_) => Ok(None),\n            BM::ContentLength(_, _) => self.do_write_body(sender, bytes).await,\n            BM::UntilClose(_) => self.do_write_until_close_body(sender, bytes).await,\n            BM::ToSelect => panic!(\"wrong body phase: ToSelect (subrequest)\"),\n        }\n    }\n\n    pub fn finished(&self) -> bool {\n        match self.body_mode {\n            BM::Complete(_) => true,\n            BM::ContentLength(total, written) => written >= total,\n            _ => false,\n        }\n    }\n\n    async fn do_write_body(\n        &mut self,\n        tx: &mut mpsc::Sender<HttpTask>,\n        bytes: Bytes,\n    ) -> Result<Option<usize>> {\n        match self.body_mode {\n            BM::ContentLength(total, written) => {\n                if written >= total {\n                    // already written full length\n                    return Ok(None);\n                }\n                let mut to_write = total - written;\n                if to_write < bytes.len() {\n                    warn!(\"Trying to write data over content-length (subrequest): {total}\");\n                } else {\n                    to_write = bytes.len();\n                }\n                let res = tx.send(HttpTask::Body(Some(bytes), false)).await;\n                match res {\n                    Ok(()) => {\n                        self.body_mode = BM::ContentLength(total, written + to_write);\n                        Ok(Some(to_write))\n                    }\n                    Err(e) => Error::e_because(WriteError, \"while writing body (subrequest)\", e),\n                }\n            }\n            _ => panic!(\"wrong body mode: {:?} (subrequest)\", self.body_mode),\n        }\n    }\n\n    async fn do_write_until_close_body(\n        &mut self,\n        tx: &mut mpsc::Sender<HttpTask>,\n        bytes: Bytes,\n    ) -> Result<Option<usize>> {\n        match self.body_mode {\n            BM::UntilClose(written) => {\n                let res = tx.send(HttpTask::Body(Some(bytes.clone()), false)).await;\n                match res {\n                    Ok(()) => {\n                        self.body_mode = BM::UntilClose(written + bytes.len());\n                        Ok(Some(bytes.len()))\n                    }\n                    Err(e) => Error::e_because(WriteError, \"while writing body (subrequest)\", e),\n                }\n            }\n            _ => panic!(\"wrong body mode: {:?} (subrequest)\", self.body_mode),\n        }\n    }\n\n    pub async fn finish(&mut self, sender: &mut mpsc::Sender<HttpTask>) -> Result<Option<usize>> {\n        match self.body_mode {\n            BM::Complete(_) => Ok(None),\n            BM::ContentLength(_, _) => self.do_finish_body(sender).await,\n            BM::UntilClose(_) => self.do_finish_until_close_body(sender).await,\n            BM::ToSelect => Ok(None),\n        }\n    }\n\n    async fn do_finish_body(&mut self, tx: &mut mpsc::Sender<HttpTask>) -> Result<Option<usize>> {\n        match self.body_mode {\n            BM::ContentLength(total, written) => {\n                self.body_mode = BM::Complete(written);\n                if written < total {\n                    return Error::e_explain(\n                        PREMATURE_BODY_END,\n                        format!(\"Content-length: {total} bytes written: {written} (subrequest)\"),\n                    );\n                }\n                tx.send(HttpTask::Done).await.or_err(\n                    WriteError,\n                    \"while sending done task to downstream (subrequest)\",\n                )?;\n                Ok(Some(written))\n            }\n            _ => panic!(\"wrong body mode: {:?} (subrequest)\", self.body_mode),\n        }\n    }\n\n    async fn do_finish_until_close_body(\n        &mut self,\n        tx: &mut mpsc::Sender<HttpTask>,\n    ) -> Result<Option<usize>> {\n        match self.body_mode {\n            BM::UntilClose(written) => {\n                self.body_mode = BM::Complete(written);\n                tx.send(HttpTask::Done).await.or_err(\n                    WriteError,\n                    \"while sending done task to downstream (subrequest)\",\n                )?;\n                Ok(Some(written))\n            }\n            _ => panic!(\"wrong body mode: {:?} (subrequest)\", self.body_mode),\n        }\n    }\n\n    pub async fn write_trailers(\n        &mut self,\n        tx: &mut mpsc::Sender<HttpTask>,\n        trailers: Option<Box<HeaderMap>>,\n    ) -> Result<()> {\n        // TODO more safeguards e.g. trailers after end of stream\n        tx.send(HttpTask::Trailer(trailers)).await.or_err(\n            WriteError,\n            \"while writing response trailers to downstream (subrequest)\",\n        )?;\n        Ok(())\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use super::*;\n\n    fn init_log() {\n        let _ = env_logger::builder().is_test(true).try_init();\n    }\n\n    const TASK_BUFFER_SIZE: usize = 4;\n\n    #[tokio::test]\n    async fn read_with_body_content_length() {\n        init_log();\n        let input = b\"abc\";\n        let (tx, mut rx) = mpsc::channel::<HttpTask>(TASK_BUFFER_SIZE);\n\n        let mut body_reader = BodyReader::new(None);\n        body_reader.init_content_length(3);\n\n        tx.send(HttpTask::Body(Some(Bytes::from(&input[..])), false))\n            .await\n            .unwrap();\n        let res = body_reader.read_body(&mut rx).await.unwrap().unwrap();\n        assert_eq!(res, &input[..]);\n        assert_eq!(body_reader.body_state, ParseState::Complete(3));\n    }\n\n    #[tokio::test]\n    async fn read_with_body_content_length_2() {\n        init_log();\n        let input1 = b\"a\";\n        let input2 = b\"bc\";\n        let (tx, mut rx) = mpsc::channel::<HttpTask>(TASK_BUFFER_SIZE);\n\n        let mut body_reader = BodyReader::new(None);\n        body_reader.init_content_length(3);\n\n        tx.send(HttpTask::Body(Some(Bytes::from(&input1[..])), false))\n            .await\n            .unwrap();\n        let res = body_reader.read_body(&mut rx).await.unwrap().unwrap();\n        assert_eq!(res, &input1[..]);\n        assert_eq!(body_reader.body_state, ParseState::Partial(1, 2));\n\n        tx.send(HttpTask::Body(Some(Bytes::from(&input2[..])), true))\n            .await\n            .unwrap();\n        let res = body_reader.read_body(&mut rx).await.unwrap().unwrap();\n        assert_eq!(res, &input2[..]);\n        assert_eq!(body_reader.body_state, ParseState::Complete(3));\n    }\n\n    #[tokio::test]\n    async fn read_with_body_content_length_empty_task() {\n        init_log();\n        let input1 = b\"a\";\n        let input2 = b\"\"; // zero length body task\n        let (tx, mut rx) = mpsc::channel::<HttpTask>(TASK_BUFFER_SIZE);\n\n        let mut body_reader = BodyReader::new(None);\n        body_reader.init_content_length(3);\n\n        tx.send(HttpTask::Body(Some(Bytes::from(&input1[..])), false))\n            .await\n            .unwrap();\n        let res = body_reader.read_body(&mut rx).await.unwrap().unwrap();\n        assert_eq!(res, &input1[..]);\n        assert_eq!(body_reader.body_state, ParseState::Partial(1, 2));\n\n        // subrequest can allow empty body tasks\n        tx.send(HttpTask::Body(Some(Bytes::from(&input2[..])), false))\n            .await\n            .unwrap();\n        let res = body_reader.read_body(&mut rx).await.unwrap().unwrap();\n        assert_eq!(res, &input2[..]);\n        assert_eq!(body_reader.body_state, ParseState::Partial(1, 2));\n\n        // premature end of stream still errors\n        tx.send(HttpTask::Body(Some(Bytes::from(&input2[..])), true))\n            .await\n            .unwrap();\n        let res = body_reader.read_body(&mut rx).await.unwrap_err();\n        assert_eq!(&PREMATURE_BODY_END, res.etype());\n        assert_eq!(body_reader.body_state, ParseState::Done(1));\n    }\n\n    #[tokio::test]\n    async fn read_with_body_content_length_less() {\n        init_log();\n        let input1 = b\"a\";\n        let (tx, mut rx) = mpsc::channel::<HttpTask>(TASK_BUFFER_SIZE);\n\n        let mut body_reader = BodyReader::new(None);\n        body_reader.init_content_length(3);\n\n        tx.send(HttpTask::Body(Some(Bytes::from(&input1[..])), false))\n            .await\n            .unwrap();\n        let res = body_reader.read_body(&mut rx).await.unwrap().unwrap();\n        assert_eq!(res, &input1[..]);\n        assert_eq!(body_reader.body_state, ParseState::Partial(1, 2));\n\n        drop(tx);\n        let res = body_reader.read_body(&mut rx).await.unwrap_err();\n        assert_eq!(&ConnectionClosed, res.etype());\n        assert_eq!(body_reader.body_state, ParseState::Done(1));\n    }\n\n    #[tokio::test]\n    async fn read_with_body_content_length_more() {\n        init_log();\n        let input1 = b\"a\";\n        let input2 = b\"bcd\";\n        let (tx, mut rx) = mpsc::channel::<HttpTask>(TASK_BUFFER_SIZE);\n\n        let mut body_reader = BodyReader::new(None);\n        body_reader.init_content_length(3);\n\n        tx.send(HttpTask::Body(Some(Bytes::from(&input1[..])), false))\n            .await\n            .unwrap();\n        let res = body_reader.read_body(&mut rx).await.unwrap().unwrap();\n        assert_eq!(res, &input1[..]);\n        assert_eq!(body_reader.body_state, ParseState::Partial(1, 2));\n\n        tx.send(HttpTask::Body(Some(Bytes::from(&input2[..])), false))\n            .await\n            .unwrap();\n        let res = body_reader.read_body(&mut rx).await.unwrap().unwrap();\n        assert_eq!(res, &input2[0..2]);\n        assert_eq!(body_reader.body_state, ParseState::Complete(3));\n    }\n\n    #[tokio::test]\n    async fn read_with_body_until_close() {\n        init_log();\n        let input1 = b\"a\";\n        let input2 = b\"\"; // zero length body but not actually close\n        let (tx, mut rx) = mpsc::channel::<HttpTask>(TASK_BUFFER_SIZE);\n        let mut body_reader = BodyReader::new(None);\n        body_reader.init_close_delimited();\n\n        tx.send(HttpTask::Body(Some(Bytes::from(&input1[..])), false))\n            .await\n            .unwrap();\n        let res = body_reader.read_body(&mut rx).await.unwrap().unwrap();\n        assert_eq!(res, &input1[..]);\n        assert_eq!(body_reader.body_state, ParseState::UntilClose(1));\n\n        tx.send(HttpTask::Body(Some(Bytes::from(&input2[..])), false))\n            .await\n            .unwrap();\n        let res = body_reader.read_body(&mut rx).await.unwrap().unwrap();\n        assert_eq!(res, &input2[..]);\n        assert_eq!(body_reader.body_state, ParseState::UntilClose(1));\n\n        // sending end closed\n        drop(tx);\n        let res = body_reader.read_body(&mut rx).await.unwrap();\n        assert_eq!(res, None);\n        assert_eq!(body_reader.body_state, ParseState::Complete(1));\n    }\n\n    #[tokio::test]\n    async fn write_body_cl() {\n        init_log();\n        let output = b\"a\";\n        let (mut tx, mut rx) = mpsc::channel::<HttpTask>(TASK_BUFFER_SIZE);\n        let mut body_writer = BodyWriter::new();\n        body_writer.init_content_length(1);\n        assert_eq!(body_writer.body_mode, BodyMode::ContentLength(1, 0));\n        let res = body_writer\n            .write_body(&mut tx, Bytes::from(&output[..]))\n            .await\n            .unwrap()\n            .unwrap();\n        assert_eq!(res, 1);\n        assert_eq!(body_writer.body_mode, BodyMode::ContentLength(1, 1));\n        // write again, over the limit\n        let res = body_writer\n            .write_body(&mut tx, Bytes::from(&output[..]))\n            .await\n            .unwrap();\n        assert_eq!(res, None);\n        assert_eq!(body_writer.body_mode, BodyMode::ContentLength(1, 1));\n        let res = body_writer.finish(&mut tx).await.unwrap().unwrap();\n        assert_eq!(res, 1);\n        assert_eq!(body_writer.body_mode, BodyMode::Complete(1));\n\n        // only one body task written\n        match rx.try_recv().unwrap() {\n            HttpTask::Body(b, end) => {\n                assert_eq!(b.unwrap(), &output[..]);\n                assert!(!end);\n            }\n            task => panic!(\"unexpected task {task:?}\"),\n        }\n        assert!(matches!(rx.try_recv().unwrap(), HttpTask::Done));\n        drop(tx);\n\n        assert_eq!(\n            rx.try_recv().unwrap_err(),\n            mpsc::error::TryRecvError::Disconnected\n        );\n    }\n\n    #[tokio::test]\n    async fn write_body_until_close() {\n        init_log();\n        let data = b\"a\";\n        let (mut tx, mut rx) = mpsc::channel::<HttpTask>(TASK_BUFFER_SIZE);\n        let mut body_writer = BodyWriter::new();\n        body_writer.init_close_delimited();\n        assert_eq!(body_writer.body_mode, BodyMode::UntilClose(0));\n        let res = body_writer\n            .write_body(&mut tx, Bytes::from(&data[..]))\n            .await\n            .unwrap()\n            .unwrap();\n        assert_eq!(res, 1);\n        assert_eq!(body_writer.body_mode, BodyMode::UntilClose(1));\n        match rx.try_recv().unwrap() {\n            HttpTask::Body(b, end) => {\n                assert_eq!(b.unwrap().as_ref(), data);\n                assert!(!end);\n            }\n            task => panic!(\"unexpected task {task:?}\"),\n        }\n\n        let res = body_writer\n            .write_body(&mut tx, Bytes::from(&data[..]))\n            .await\n            .unwrap()\n            .unwrap();\n        assert_eq!(res, 1);\n        assert_eq!(body_writer.body_mode, BodyMode::UntilClose(2));\n        let res = body_writer.finish(&mut tx).await.unwrap().unwrap();\n        assert_eq!(res, 2);\n        assert_eq!(body_writer.body_mode, BodyMode::Complete(2));\n        match rx.try_recv().unwrap() {\n            HttpTask::Body(b, end) => {\n                assert_eq!(b.unwrap().as_ref(), data);\n                assert!(!end);\n            }\n            task => panic!(\"unexpected task {task:?}\"),\n        }\n        assert!(matches!(rx.try_recv().unwrap(), HttpTask::Done));\n\n        assert_eq!(rx.try_recv().unwrap_err(), mpsc::error::TryRecvError::Empty);\n    }\n}\n"
  },
  {
    "path": "pingora-core/src/protocols/http/subrequest/dummy.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\nuse crate::protocols::raw_connect::ProxyDigest;\nuse crate::protocols::{\n    GetProxyDigest, GetSocketDigest, GetTimingDigest, Peek, SocketDigest, Ssl, TimingDigest,\n    UniqueID, UniqueIDType,\n};\nuse async_trait::async_trait;\nuse core::pin::Pin;\nuse core::task::{Context, Poll};\nuse std::io::Cursor;\nuse std::sync::Arc;\nuse tokio::io::{AsyncRead, AsyncWrite, Error, ReadBuf};\n\n// An async IO stream that returns the request when being read from and dumps the data to the void\n// when being write to\n#[derive(Debug)]\npub(crate) struct DummyIO(Cursor<Vec<u8>>);\n\nimpl DummyIO {\n    pub fn new(read_bytes: &[u8]) -> Self {\n        DummyIO(Cursor::new(Vec::from(read_bytes)))\n    }\n}\n\nimpl AsyncRead for DummyIO {\n    fn poll_read(\n        mut self: Pin<&mut Self>,\n        cx: &mut Context<'_>,\n        buf: &mut ReadBuf<'_>,\n    ) -> Poll<Result<(), Error>> {\n        if self.0.position() < self.0.get_ref().len() as u64 {\n            Pin::new(&mut self.0).poll_read(cx, buf)\n        } else {\n            // all data is read, pending forever otherwise the stream is considered closed\n            Poll::Pending\n        }\n    }\n}\n\nimpl AsyncWrite for DummyIO {\n    fn poll_write(\n        self: Pin<&mut Self>,\n        _cx: &mut Context<'_>,\n        buf: &[u8],\n    ) -> Poll<Result<usize, Error>> {\n        Poll::Ready(Ok(buf.len()))\n    }\n\n    fn poll_flush(self: Pin<&mut Self>, _cx: &mut Context<'_>) -> Poll<Result<(), Error>> {\n        Poll::Ready(Ok(()))\n    }\n    fn poll_shutdown(self: Pin<&mut Self>, _cx: &mut Context<'_>) -> Poll<Result<(), Error>> {\n        Poll::Ready(Ok(()))\n    }\n}\n\nimpl UniqueID for DummyIO {\n    fn id(&self) -> UniqueIDType {\n        0 // placeholder\n    }\n}\n\nimpl Ssl for DummyIO {}\n\nimpl GetTimingDigest for DummyIO {\n    fn get_timing_digest(&self) -> Vec<Option<TimingDigest>> {\n        vec![]\n    }\n}\n\nimpl GetProxyDigest for DummyIO {\n    fn get_proxy_digest(&self) -> Option<Arc<ProxyDigest>> {\n        None\n    }\n}\n\nimpl GetSocketDigest for DummyIO {\n    fn get_socket_digest(&self) -> Option<Arc<SocketDigest>> {\n        None\n    }\n}\n\nimpl Peek for DummyIO {}\n\n#[async_trait]\nimpl crate::protocols::Shutdown for DummyIO {\n    async fn shutdown(&mut self) -> () {}\n}\n\n#[tokio::test]\nasync fn test_dummy_io() {\n    use futures::FutureExt;\n    use tokio::io::{AsyncReadExt, AsyncWriteExt};\n\n    let mut dummy = DummyIO::new(&[1, 2]);\n    let res = dummy.read_u8().await;\n    assert_eq!(res.unwrap(), 1);\n    let res = dummy.read_u8().await;\n    assert_eq!(res.unwrap(), 2);\n    let res = dummy.read_u8().now_or_never();\n    assert!(res.is_none()); // pending forever\n    let res = dummy.write_u8(0).await;\n    assert!(res.is_ok());\n}\n"
  },
  {
    "path": "pingora-core/src/protocols/http/subrequest/mod.rs",
    "content": "pub(crate) mod body;\npub(crate) mod dummy;\npub mod server;\n"
  },
  {
    "path": "pingora-core/src/protocols/http/subrequest/server.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! # HTTP server session for subrequests\n//!\n//! This server session is _very_ similar to the implementation for v1, if not\n//! identical in many cases. Though in theory subrequests are HTTP version\n//! agnostic in reality this means that they must interpret any version-specific\n//! idiosyncracies such as Connection: upgrade headers in H1 because they\n//! \"stand-in\" for the actual main Session when running proxy logic. As much as\n//! possible they should defer downstream-specific logic to the actual downstream\n//! session and act more or less as a pipe.\n//!\n//! The session also instantiates a [`SubrequestHandle`] that contains necessary\n//! communication channels with the subrequest, to make it possible to send\n//! and receive data.\n//!\n//! Its write calls will send `HttpTask`s to the handle channels, instead of\n//! flushing to an actual underlying stream.\n//!\n//! Connection reuse and keep-alive are not supported because there is no\n//! actual underlying stream, only transient channels per request.\n\nuse bytes::Bytes;\nuse http::HeaderValue;\nuse http::{header, header::AsHeaderName, HeaderMap, Method};\nuse log::{debug, trace, warn};\nuse pingora_error::{Error, ErrorType::*, OkOrErr, Result};\nuse pingora_http::{RequestHeader, ResponseHeader};\nuse pingora_timeout::timeout;\nuse std::time::Duration;\nuse tokio::sync::{mpsc, oneshot};\n\nuse super::body::{BodyReader, BodyWriter};\nuse crate::protocols::http::{\n    body_buffer::FixedBuffer,\n    server::Session as GenericHttpSession,\n    subrequest::dummy::DummyIO,\n    v1::common::{header_value_content_length, is_chunked_encoding_from_headers, BODY_BUF_LIMIT},\n    v1::server::HttpSession as SessionV1,\n    HttpTask,\n};\nuse crate::protocols::{Digest, SocketAddr};\n\n/// The HTTP server session\npub struct HttpSession {\n    // these are only options because we allow dropping them separately on shutdown\n    tx: Option<mpsc::Sender<HttpTask>>,\n    rx: Option<mpsc::Receiver<HttpTask>>,\n    // Currently subrequest session is initialized via a dummy SessionV1 only\n    // TODO: need to be able to indicate H2 / other HTTP versions here\n    v1_inner: Box<SessionV1>,\n    proxy_error: Option<oneshot::Sender<Box<Error>>>, // option to consume the sender\n    read_req_header: bool,\n    response_written: Option<ResponseHeader>,\n    read_timeout: Option<Duration>,\n    write_timeout: Option<Duration>,\n    total_drain_timeout: Option<Duration>,\n    body_bytes_sent: usize,\n    body_bytes_read: usize,\n    retry_buffer: Option<FixedBuffer>,\n    body_reader: BodyReader,\n    body_writer: BodyWriter,\n    upgraded: bool,\n    // TODO: likely doesn't need to be a separate bool when/if moving away from dummy SessionV1\n    clear_request_body_headers: bool,\n    digest: Option<Box<Digest>>,\n}\n\n/// A handle to the subrequest session itself to interact or read from it.\npub struct SubrequestHandle {\n    /// Channel sender (for subrequest input)\n    pub tx: mpsc::Sender<HttpTask>,\n    /// Channel receiver (for subrequest output)\n    pub rx: mpsc::Receiver<HttpTask>,\n    /// Indicates when subrequest wants to start reading body input\n    pub subreq_wants_body: oneshot::Receiver<()>,\n    /// Any final or downstream error that was encountered while proxying\n    pub subreq_proxy_error: oneshot::Receiver<Box<Error>>,\n}\n\nimpl SubrequestHandle {\n    /// Spawn a task to drain received HttpTasks.\n    pub fn drain_tasks(mut self) -> tokio::task::JoinHandle<()> {\n        tokio::spawn(async move {\n            let _tx = self.tx; // keep handle to sender alive\n            while self.rx.recv().await.is_some() {}\n            trace!(\"subrequest dropped\");\n        })\n    }\n}\n\nimpl HttpSession {\n    /// Create a new http server session for a subrequest.\n    /// The created session needs to call [`Self::read_request()`] first before performing\n    /// any other operations.\n    pub fn new_from_session(session: &GenericHttpSession) -> (Self, SubrequestHandle) {\n        let v1_inner = SessionV1::new(Box::new(DummyIO::new(&session.to_h1_raw())));\n        let digest = session.digest().cloned();\n        // allow buffering a small number of tasks, otherwise exert backpressure\n        const CHANNEL_BUFFER_SIZE: usize = 4;\n        let (downstream_tx, downstream_rx) = mpsc::channel(CHANNEL_BUFFER_SIZE);\n        let (upstream_tx, upstream_rx) = mpsc::channel(CHANNEL_BUFFER_SIZE);\n        let (wants_body_tx, wants_body_rx) = oneshot::channel();\n        let (proxy_error_tx, proxy_error_rx) = oneshot::channel();\n        (\n            HttpSession {\n                v1_inner: Box::new(v1_inner),\n                tx: Some(upstream_tx),\n                rx: Some(downstream_rx),\n                proxy_error: Some(proxy_error_tx),\n                body_reader: BodyReader::new(Some(wants_body_tx)),\n                body_writer: BodyWriter::new(),\n                read_req_header: false,\n                response_written: None,\n                read_timeout: None,\n                write_timeout: None,\n                total_drain_timeout: None,\n                body_bytes_sent: 0,\n                body_bytes_read: 0,\n                retry_buffer: None,\n                upgraded: false,\n                clear_request_body_headers: false,\n                digest: digest.map(Box::new),\n            },\n            SubrequestHandle {\n                tx: downstream_tx,\n                rx: upstream_rx,\n                subreq_wants_body: wants_body_rx,\n                subreq_proxy_error: proxy_error_rx,\n            },\n        )\n    }\n\n    /// Read the request header. Return `Ok(Some(n))` where the read and parsing are successful.\n    pub async fn read_request(&mut self) -> Result<Option<usize>> {\n        let res = self.v1_inner.read_request().await?;\n        if res.is_none() {\n            // this is when h1 client closes the connection without sending data,\n            // which shouldn't be the case for a subrequest session just created\n            return Error::e_explain(InternalError, \"no session request header provided\");\n        }\n        self.read_req_header = true;\n        if self.clear_request_body_headers {\n            // indicated that we wanted to clear these headers in the past, do so now\n            self.clear_request_body_headers();\n        }\n        Ok(res)\n    }\n\n    /// Validate the request header read. This function must be called after the request header\n    /// read.\n    /// # Panics\n    /// this function and most other functions will panic if called before [`Self::read_request()`]\n    pub fn validate_request(&self) -> Result<()> {\n        self.v1_inner.validate_request()\n    }\n\n    /// Return a reference of the `RequestHeader` this session read\n    /// # Panics\n    /// this function and most other functions will panic if called before [`Self::read_request()`]\n    pub fn req_header(&self) -> &RequestHeader {\n        self.v1_inner.req_header()\n    }\n\n    /// Return a mutable reference of the `RequestHeader` this session read\n    /// # Panics\n    /// this function and most other functions will panic if called before [`Self::read_request()`]\n    pub fn req_header_mut(&mut self) -> &mut RequestHeader {\n        self.v1_inner.req_header_mut()\n    }\n\n    /// Get the header value for the given header name\n    /// If there are multiple headers under the same name, the first one will be returned\n    /// Use `self.req_header().header.get_all(name)` to get all the headers under the same name\n    pub fn get_header(&self, name: impl AsHeaderName) -> Option<&HeaderValue> {\n        self.v1_inner.get_header(name)\n    }\n\n    /// Return the method of this request. None if the request is not read yet.\n    pub(super) fn get_method(&self) -> Option<&http::Method> {\n        self.v1_inner.get_method()\n    }\n\n    /// Return the path of the request (i.e., the `/hello?1` of `GET /hello?1 HTTP1.1`)\n    /// An empty slice will be used if there is no path or the request is not read yet\n    pub(super) fn get_path(&self) -> &[u8] {\n        self.v1_inner.get_path()\n    }\n\n    /// Return the host header of the request. An empty slice will be used if there is no host header\n    pub(super) fn get_host(&self) -> &[u8] {\n        self.v1_inner.get_host()\n    }\n\n    /// Return a string `$METHOD $PATH, Host: $HOST`. Mostly for logging and debug purpose\n    pub fn request_summary(&self) -> String {\n        format!(\n            \"{} {}, Host: {} (subrequest)\",\n            self.get_method().map_or(\"-\", |r| r.as_str()),\n            String::from_utf8_lossy(self.get_path()),\n            String::from_utf8_lossy(self.get_host())\n        )\n    }\n\n    /// Is the request a upgrade request\n    pub fn is_upgrade_req(&self) -> bool {\n        self.v1_inner.is_upgrade_req()\n    }\n\n    /// Get the request header as raw bytes, `b\"\"` when the header doesn't exist\n    pub fn get_header_bytes(&self, name: impl AsHeaderName) -> &[u8] {\n        self.v1_inner.get_header_bytes(name)\n    }\n\n    /// Read the request body. `Ok(None)` when there is no (more) body to read.\n    pub async fn read_body_bytes(&mut self) -> Result<Option<Bytes>> {\n        let read = self.read_body().await?;\n        Ok(read.inspect(|b| {\n            self.body_bytes_read += b.len();\n            if let Some(buffer) = self.retry_buffer.as_mut() {\n                buffer.write_to_buffer(b);\n            }\n        }))\n    }\n\n    async fn do_read_body(&mut self) -> Result<Option<Bytes>> {\n        self.init_body_reader();\n        self.body_reader\n            .read_body(self.rx.as_mut().expect(\"rx valid before shutdown\"))\n            .await\n    }\n\n    /// Read the body bytes with timeout.\n    async fn read_body(&mut self) -> Result<Option<Bytes>> {\n        match self.read_timeout {\n            Some(t) => match timeout(t, self.do_read_body()).await {\n                Ok(res) => res,\n                Err(_) => Error::e_explain(\n                    ReadTimedout,\n                    format!(\"reading body, timeout: {t:?} (subrequest)\"),\n                ),\n            },\n            None => self.do_read_body().await,\n        }\n    }\n\n    async fn do_drain_request_body(&mut self) -> Result<()> {\n        loop {\n            match self.read_body_bytes().await {\n                Ok(Some(_)) => { /* continue to drain */ }\n                Ok(None) => return Ok(()), // done\n                Err(e) => return Err(e),\n            }\n        }\n    }\n\n    /// Drain the request body. `Ok(())` when there is no (more) body to read.\n    pub async fn drain_request_body(&mut self) -> Result<()> {\n        if self.is_body_done() {\n            return Ok(());\n        }\n        match self.total_drain_timeout {\n            Some(t) => match timeout(t, self.do_drain_request_body()).await {\n                Ok(res) => res,\n                Err(_) => Error::e_explain(\n                    ReadTimedout,\n                    format!(\"draining body, timeout: {t:?} (subrequest)\"),\n                ),\n            },\n            None => self.do_drain_request_body().await,\n        }\n    }\n\n    /// Whether there is no (more) body to be read.\n    pub fn is_body_done(&mut self) -> bool {\n        self.init_body_reader();\n        self.body_reader.body_done()\n    }\n\n    /// Whether the request has an empty body\n    /// Because HTTP 1.1 clients have to send either `Content-Length` or `Transfer-Encoding` in order\n    /// to signal the server that it will send the body, this function returns accurate results even\n    /// only when the request header is just read.\n    pub fn is_body_empty(&mut self) -> bool {\n        self.init_body_reader();\n        self.body_reader.body_empty()\n    }\n\n    /// Write the response header to the client.\n    /// This function can be called more than once to send 1xx informational headers excluding 101.\n    pub async fn write_response_header(&mut self, header: Box<ResponseHeader>) -> Result<()> {\n        if let Some(resp) = self.response_written.as_ref() {\n            if !resp.status.is_informational() || self.upgraded {\n                warn!(\"Respond header is already sent, cannot send again (subrequest)\");\n                return Ok(());\n            }\n        }\n\n        // XXX: don't add additional downstream headers, unlike h1, subreq is mostly treated as a pipe\n\n        // Allow informational header (excluding 101) to pass through without affecting the state\n        // of the request\n        if header.status == 101 || !header.status.is_informational() {\n            // reset request body to done for incomplete upgrade handshakes\n            if let Some(upgrade_ok) = self.is_upgrade(&header) {\n                if upgrade_ok {\n                    debug!(\"ok upgrade handshake\");\n                    // For ws we use HTTP1_0 do_read_body_until_closed\n                    //\n                    // On ws close the initiator sends a close frame and\n                    // then waits for a response from the peer, once it receives\n                    // a response it closes the conn. After receiving a\n                    // control frame indicating the connection should be closed,\n                    // a peer discards any further data received.\n                    // https://www.rfc-editor.org/rfc/rfc6455#section-1.4\n                    self.upgraded = true;\n                    // Now that the upgrade was successful, we need to change\n                    // how we interpret the rest of the body as pass-through.\n                    if self.body_reader.need_init() {\n                        self.init_body_reader();\n                    } else {\n                        // already initialized\n                        // immediately start reading the rest of the body as upgraded\n                        // (in theory most upgraded requests shouldn't have any body)\n                        //\n                        // TODO: https://datatracker.ietf.org/doc/html/rfc9110#name-upgrade\n                        // the most spec-compliant behavior is to switch interpretation\n                        // after sending the former body. For now we immediately\n                        // switch interpretation to match nginx behavior.\n                        // TODO: this has no effect resetting the body counter of TE chunked\n                        self.body_reader.convert_to_close_delimited();\n                    }\n                } else {\n                    debug!(\"bad upgrade handshake!\");\n                    // continue to read body as-is, this is now just a regular request\n                }\n            }\n            self.init_body_writer(&header);\n        }\n\n        // TODO propagate h2 end\n        debug!(\"send response header (subrequest)\");\n        match self\n            .tx\n            .as_mut()\n            .expect(\"tx valid before shutdown\")\n            .send(HttpTask::Header(header.clone(), false))\n            .await\n        {\n            Ok(()) => {\n                self.response_written = Some(*header);\n                Ok(())\n            }\n            Err(e) => Error::e_because(WriteError, \"writing response header\", e),\n        }\n    }\n\n    /// Return the response header if it is already sent.\n    pub fn response_written(&self) -> Option<&ResponseHeader> {\n        self.response_written.as_ref()\n    }\n\n    /// `Some(true)` if the this is a successful upgrade\n    /// `Some(false)` if the request is an upgrade but the response refuses it\n    /// `None` if the request is not an upgrade.\n    pub fn is_upgrade(&self, header: &ResponseHeader) -> Option<bool> {\n        self.v1_inner.is_upgrade(header)\n    }\n\n    /// Was this request successfully turned into an upgraded connection?\n    ///\n    /// Both the request had to have been an `Upgrade` request\n    /// and the response had to have been a `101 Switching Protocols`.\n    // XXX: this should only be valid if subrequest is standing in for\n    // a v1 session.\n    pub fn was_upgraded(&self) -> bool {\n        self.upgraded\n    }\n\n    fn init_body_writer(&mut self, header: &ResponseHeader) {\n        use http::StatusCode;\n        /* the following responses don't have body 204, 304, and HEAD */\n        if matches!(\n            header.status,\n            StatusCode::NO_CONTENT | StatusCode::NOT_MODIFIED\n        ) || self.get_method() == Some(&Method::HEAD)\n        {\n            self.body_writer.init_content_length(0);\n            return;\n        }\n\n        if header.status.is_informational() && header.status != StatusCode::SWITCHING_PROTOCOLS {\n            // 1xx response, not enough to init body\n            return;\n        }\n\n        if self.is_upgrade(header) == Some(true) {\n            self.body_writer.init_close_delimited();\n        } else if is_chunked_encoding_from_headers(&header.headers) {\n            // transfer-encoding takes priority over content-length\n            self.body_writer.init_close_delimited();\n        } else {\n            let content_length =\n                header_value_content_length(header.headers.get(http::header::CONTENT_LENGTH));\n            match content_length {\n                Some(length) => {\n                    self.body_writer.init_content_length(length);\n                }\n                None => {\n                    /* TODO: 1. connection: keepalive cannot be used,\n                    2. mark connection must be closed */\n                    self.body_writer.init_close_delimited();\n                }\n            }\n        }\n    }\n\n    /// Same as [`Self::write_response_header()`] but takes a reference.\n    pub async fn write_response_header_ref(&mut self, resp: &ResponseHeader) -> Result<()> {\n        self.write_response_header(Box::new(resp.clone())).await\n    }\n\n    async fn do_write_body(&mut self, buf: Bytes) -> Result<Option<usize>> {\n        let written = self\n            .body_writer\n            .write_body(self.tx.as_mut().expect(\"tx valid before shutdown\"), buf)\n            .await;\n\n        if let Ok(Some(num_bytes)) = written {\n            self.body_bytes_sent += num_bytes;\n        }\n\n        written\n    }\n\n    /// Write response body to the client. Return `Ok(None)` when there shouldn't be more body\n    /// to be written, e.g., writing more bytes than what the `Content-Length` header suggests\n    pub async fn write_body(&mut self, buf: Bytes) -> Result<Option<usize>> {\n        // TODO: check if the response header is written\n        match self.write_timeout {\n            Some(t) => match timeout(t, self.do_write_body(buf)).await {\n                Ok(res) => res,\n                Err(_) => Error::e_explain(WriteTimedout, format!(\"writing body, timeout: {t:?}\")),\n            },\n            None => self.do_write_body(buf).await,\n        }\n    }\n\n    fn maybe_force_close_body_reader(&mut self) {\n        if self.upgraded && !self.body_reader.body_done() {\n            // response is done, reset the request body to close\n            self.body_reader.init_content_length(0);\n        }\n    }\n\n    /// Signal that there is no more body to write.\n    /// This call will try to flush the buffer if there is any un-flushed data.\n    /// For chunked encoding response, this call will also send the last chunk.\n    /// For upgraded sessions, this call will also close the reading of the client body.\n    pub async fn finish(&mut self) -> Result<Option<usize>> {\n        let res = self\n            .body_writer\n            .finish(self.tx.as_mut().expect(\"tx valid before shutdown\"))\n            .await?;\n\n        self.maybe_force_close_body_reader();\n        Ok(res)\n    }\n\n    /// Signal to error listener held by SubrequestHandle that a proxy error was encountered,\n    /// and pass along what that error was.\n    ///\n    /// This is helpful to signal what errors were encountered outside of the proxy state machine,\n    /// e.g. during subrequest request filters.\n    ///\n    /// Note: in the case of multiple proxy failures e.g. when caching, only the first error will\n    /// be propagated (i.e. downstream error first if it goes away before upstream).\n    pub fn on_proxy_failure(&mut self, e: Box<Error>) {\n        // fine if handle is gone\n        if let Some(sender) = self.proxy_error.take() {\n            let _ = sender.send(e);\n        }\n    }\n\n    /// Return how many response body bytes (application, not wire) already sent downstream\n    pub fn body_bytes_sent(&self) -> usize {\n        self.body_bytes_sent\n    }\n\n    /// Return how many request body bytes (application, not wire) already read from downstream\n    pub fn body_bytes_read(&self) -> usize {\n        self.body_bytes_read\n    }\n\n    fn is_chunked_encoding(&self) -> bool {\n        is_chunked_encoding_from_headers(&self.req_header().headers)\n    }\n\n    /// Clear body-related subrequest headers.\n    ///\n    /// This is ok to call before the request is read; the headers will then be cleared after\n    /// reading the request header.\n    pub fn clear_request_body_headers(&mut self) {\n        self.clear_request_body_headers = true;\n        if self.read_req_header {\n            let req = self.v1_inner.req_header_mut();\n            req.remove_header(&header::CONTENT_LENGTH);\n            req.remove_header(&header::TRANSFER_ENCODING);\n            req.remove_header(&header::CONTENT_TYPE);\n            req.remove_header(&header::CONTENT_ENCODING);\n        }\n    }\n\n    fn init_body_reader(&mut self) {\n        if self.body_reader.need_init() {\n            // reset retry buffer\n            if let Some(buffer) = self.retry_buffer.as_mut() {\n                buffer.clear();\n            }\n\n            if self.was_upgraded() {\n                // if upgraded _post_ 101 (and body was not init yet)\n                // treat as upgraded body (pass through until closed)\n                self.body_reader.init_close_delimited();\n            } else if self.is_chunked_encoding() {\n                // if chunked encoding, content-length should be ignored\n                // TE is not visible at subrequest HttpTask level\n                // so this means read until request closure\n                self.body_reader.init_close_delimited();\n            } else {\n                let cl = header_value_content_length(self.get_header(header::CONTENT_LENGTH));\n                match cl {\n                    Some(i) => {\n                        self.body_reader.init_content_length(i);\n                    }\n                    None => {\n                        // Per RFC 9112: \"Request messages are never close-delimited because they are\n                        // always explicitly framed by length or transfer coding, with the absence of\n                        // both implying the request ends immediately after the header section.\"\n                        // All HTTP/1.x requests without Content-Length or Transfer-Encoding have 0 body\n                        self.body_reader.init_content_length(0);\n                    }\n                }\n            }\n        }\n    }\n\n    pub fn retry_buffer_truncated(&self) -> bool {\n        self.retry_buffer\n            .as_ref()\n            .map_or_else(|| false, |r| r.is_truncated())\n    }\n\n    pub fn enable_retry_buffering(&mut self) {\n        if self.retry_buffer.is_none() {\n            self.retry_buffer = Some(FixedBuffer::new(BODY_BUF_LIMIT))\n        }\n    }\n\n    pub fn get_retry_buffer(&self) -> Option<Bytes> {\n        self.retry_buffer.as_ref().and_then(|b| {\n            if b.is_truncated() {\n                None\n            } else {\n                b.get_buffer()\n            }\n        })\n    }\n\n    /// This function will (async) block forever until the client closes the connection.\n    pub async fn idle(&mut self) -> Result<HttpTask> {\n        let rx = self.rx.as_mut().expect(\"rx valid before shutdown\");\n        let mut task = rx\n            .recv()\n            .await\n            .or_err(ReadError, \"during HTTP idle state\")?;\n        // just consume empty body or done messages, the downstream channel is not a real\n        // connection and only used for this one request\n        while matches!(&task, HttpTask::Done)\n            || matches!(&task, HttpTask::Body(b, _) if b.as_ref().is_none_or(|b| b.is_empty()))\n        {\n            task = rx\n                .recv()\n                .await\n                .or_err(ReadError, \"during HTTP idle state\")?;\n        }\n        Ok(task)\n    }\n\n    /// This function will return body bytes (same as [`Self::read_body_bytes()`]), but after\n    /// the client body finishes (`Ok(None)` is returned), calling this function again will block\n    /// forever, same as [`Self::idle()`].\n    pub async fn read_body_or_idle(&mut self, no_body_expected: bool) -> Result<Option<Bytes>> {\n        if no_body_expected || self.is_body_done() {\n            let read_task = self.idle().await?;\n            Error::e_explain(\n                ConnectError,\n                format!(\"Sent unexpected task {read_task:?} after end of body (subrequest)\"),\n            )\n        } else {\n            self.read_body_bytes().await\n        }\n    }\n\n    /// Return the raw bytes of the request header.\n    pub fn get_headers_raw_bytes(&self) -> Bytes {\n        self.v1_inner.get_headers_raw_bytes()\n    }\n\n    /// Close the subrequest channels, indicating that no more data will be sent\n    /// or received. This is expected to be called before dropping the `Session` itself.\n    pub fn shutdown(&mut self) {\n        drop(self.tx.take());\n        drop(self.rx.take());\n    }\n\n    /// Sets the downstream read timeout. This will trigger if we're unable\n    /// to read from the subrequest channels after `timeout`.\n    pub fn set_read_timeout(&mut self, timeout: Option<Duration>) {\n        self.read_timeout = timeout;\n    }\n\n    /// Get the downstream read timeout.\n    pub fn get_read_timeout(&self) -> Option<Duration> {\n        self.read_timeout\n    }\n\n    /// Sets the downstream write timeout. This will trigger if we're unable\n    /// to write to the subrequest channel after `timeout`.\n    pub fn set_write_timeout(&mut self, timeout: Option<Duration>) {\n        self.write_timeout = timeout;\n    }\n\n    /// Get the downstream write timeout.\n    pub fn get_write_timeout(&self) -> Option<Duration> {\n        self.write_timeout\n    }\n\n    /// Sets the total drain timeout.\n    /// Note that the downstream read timeout still applies between body byte reads.\n    pub fn set_total_drain_timeout(&mut self, timeout: Option<Duration>) {\n        self.total_drain_timeout = timeout;\n    }\n\n    /// Get the downstream total drain timeout.\n    pub fn get_total_drain_timeout(&self) -> Option<Duration> {\n        self.total_drain_timeout\n    }\n\n    /// Return the [Digest], this is originally from the main request.\n    pub fn digest(&self) -> Option<&Digest> {\n        self.digest.as_deref()\n    }\n\n    /// Return a mutable [Digest] reference.\n    pub fn digest_mut(&mut self) -> Option<&mut Digest> {\n        self.digest.as_deref_mut()\n    }\n\n    /// Return the client (peer) address of the main request.\n    pub fn client_addr(&self) -> Option<&SocketAddr> {\n        self.digest()\n            .and_then(|d| d.socket_digest.as_ref())\n            .map(|d| d.peer_addr())?\n    }\n\n    /// Return the server (local) address of the main request.\n    pub fn server_addr(&self) -> Option<&SocketAddr> {\n        self.digest()\n            .and_then(|d| d.socket_digest.as_ref())\n            .map(|d| d.local_addr())?\n    }\n\n    /// Write a `100 Continue` response to the client.\n    pub async fn write_continue_response(&mut self) -> Result<()> {\n        // only send if we haven't already\n        if self.response_written.is_none() {\n            // size hint Some(0) because default is 8\n            return self\n                .write_response_header(Box::new(ResponseHeader::build(100, Some(0)).unwrap()))\n                .await;\n        }\n        Ok(())\n    }\n\n    async fn write_non_empty_body(&mut self, data: Option<Bytes>, upgraded: bool) -> Result<()> {\n        if upgraded != self.upgraded {\n            if upgraded {\n                panic!(\"Unexpected UpgradedBody task received on un-upgraded downstream session (subrequest)\");\n            } else {\n                panic!(\"Unexpected Body task received on upgraded downstream session (subrequest)\");\n            }\n        }\n        let Some(d) = data else {\n            return Ok(());\n        };\n        if d.is_empty() {\n            return Ok(());\n        }\n        self.write_body(d).await.map_err(|e| e.into_down())?;\n        Ok(())\n    }\n\n    async fn response_duplex(&mut self, task: HttpTask) -> Result<bool> {\n        let end_stream = match task {\n            HttpTask::Header(header, end_stream) => {\n                self.write_response_header(header)\n                    .await\n                    .map_err(|e| e.into_down())?;\n                end_stream\n            }\n            HttpTask::Body(data, end_stream) => {\n                self.write_non_empty_body(data, false).await?;\n                end_stream\n            }\n            HttpTask::UpgradedBody(data, end_stream) => {\n                self.write_non_empty_body(data, true).await?;\n                end_stream\n            }\n            HttpTask::Trailer(trailers) => {\n                self.write_trailers(trailers).await?;\n                true\n            }\n            HttpTask::Done => true,\n            HttpTask::Failed(e) => return Err(e),\n        };\n        if end_stream {\n            // no-op if body wasn't initialized or is finished already\n            self.finish().await.map_err(|e| e.into_down())?;\n        }\n        Ok(end_stream || self.body_writer.finished())\n    }\n\n    // TODO: use vectored write to avoid copying\n    pub async fn response_duplex_vec(&mut self, mut tasks: Vec<HttpTask>) -> Result<bool> {\n        // TODO: send httptask failed on each error?\n        let n_tasks = tasks.len();\n        if n_tasks == 1 {\n            // fallback to single operation to avoid copy\n            return self.response_duplex(tasks.pop().unwrap()).await;\n        }\n        let mut end_stream = false;\n        for task in tasks.into_iter() {\n            end_stream = match task {\n                HttpTask::Header(header, end_stream) => {\n                    self.write_response_header(header)\n                        .await\n                        .map_err(|e| e.into_down())?;\n                    end_stream\n                }\n                HttpTask::Body(data, end_stream) => {\n                    self.write_non_empty_body(data, false).await?;\n                    end_stream\n                }\n                HttpTask::UpgradedBody(data, end_stream) => {\n                    self.write_non_empty_body(data, true).await?;\n                    end_stream\n                }\n                HttpTask::Done => {\n                    // write done\n                    // we'll send HttpTask::Done at the end of this loop in finish\n                    true\n                }\n                HttpTask::Trailer(trailers) => {\n                    self.write_trailers(trailers).await?;\n                    true\n                }\n                HttpTask::Failed(e) => {\n                    // write failed\n                    // error should also be returned when sender drops\n                    return Err(e);\n                }\n            } || end_stream; // safe guard in case `end` in tasks flips from true to false\n        }\n        if end_stream {\n            // no-op if body wasn't initialized or is finished already\n            self.finish().await.map_err(|e| e.into_down())?;\n        }\n        Ok(end_stream || self.body_writer.finished())\n    }\n\n    /// Write response trailers to the client, this also closes the stream.\n    pub async fn write_trailers(&mut self, trailers: Option<Box<HeaderMap>>) -> Result<()> {\n        self.body_writer\n            .write_trailers(\n                self.tx.as_mut().expect(\"tx valid before shutdown\"),\n                trailers,\n            )\n            .await\n    }\n}\n\n#[cfg(test)]\nmod tests_stream {\n    use super::*;\n    use crate::protocols::http::subrequest::body::{BodyMode, ParseState};\n    use bytes::BufMut;\n    use http::StatusCode;\n    use rstest::rstest;\n\n    use std::str;\n    use tokio_test::io::Builder;\n\n    fn init_log() {\n        let _ = env_logger::builder().is_test(true).try_init();\n    }\n\n    async fn session_from_input(input: &[u8]) -> (HttpSession, SubrequestHandle) {\n        let mock_io = Builder::new().read(input).build();\n        let mut http_stream = GenericHttpSession::new_http1(Box::new(mock_io));\n        http_stream.read_request().await.unwrap();\n        let (mut http_stream, handle) = HttpSession::new_from_session(&http_stream);\n        http_stream.read_request().await.unwrap();\n        (http_stream, handle)\n    }\n\n    async fn build_upgrade_req(upgrade: &str, conn: &str) -> (HttpSession, SubrequestHandle) {\n        let input = format!(\"GET / HTTP/1.1\\r\\nHost: pingora.org\\r\\nUpgrade: {upgrade}\\r\\nConnection: {conn}\\r\\n\\r\\n\");\n        session_from_input(input.as_bytes()).await\n    }\n\n    async fn build_req() -> (HttpSession, SubrequestHandle) {\n        let input = \"GET / HTTP/1.1\\r\\nHost: pingora.org\\r\\n\\r\\n\".to_string();\n        session_from_input(input.as_bytes()).await\n    }\n\n    #[tokio::test]\n    async fn read_basic() {\n        init_log();\n        let input = b\"GET / HTTP/1.1\\r\\n\\r\\n\";\n        let (http_stream, _handle) = session_from_input(input).await;\n        assert_eq!(0, http_stream.req_header().headers.len());\n        assert_eq!(Method::GET, http_stream.req_header().method);\n        assert_eq!(b\"/\", http_stream.req_header().uri.path().as_bytes());\n    }\n\n    #[tokio::test]\n    async fn read_upgrade_req() {\n        // http 1.0\n        let input = b\"GET / HTTP/1.0\\r\\nHost: pingora.org\\r\\nUpgrade: websocket\\r\\nConnection: upgrade\\r\\n\\r\\n\";\n        let (http_stream, _handle) = session_from_input(input).await;\n        assert!(!http_stream.is_upgrade_req());\n\n        // different method\n        let input = b\"POST / HTTP/1.1\\r\\nHost: pingora.org\\r\\nUpgrade: websocket\\r\\nConnection: upgrade\\r\\n\\r\\n\";\n        let (http_stream, _handle) = session_from_input(input).await;\n        assert!(http_stream.is_upgrade_req());\n\n        // missing upgrade header\n        let input = b\"GET / HTTP/1.1\\r\\nHost: pingora.org\\r\\nConnection: upgrade\\r\\n\\r\\n\";\n        let (http_stream, _handle) = session_from_input(input).await;\n        assert!(!http_stream.is_upgrade_req());\n\n        // no connection header\n        let input = b\"GET / HTTP/1.1\\r\\nHost: pingora.org\\r\\nUpgrade: WebSocket\\r\\n\\r\\n\";\n        let (http_stream, _handle) = session_from_input(input).await;\n        assert!(http_stream.is_upgrade_req());\n\n        let (http_stream, _handle) = build_upgrade_req(\"websocket\", \"Upgrade\").await;\n        assert!(http_stream.is_upgrade_req());\n\n        // mixed case\n        let (http_stream, _handle) = build_upgrade_req(\"WebSocket\", \"Upgrade\").await;\n        assert!(http_stream.is_upgrade_req());\n    }\n\n    #[tokio::test]\n    async fn read_upgrade_req_with_1xx_response() {\n        let (mut http_stream, _handle) = build_upgrade_req(\"websocket\", \"upgrade\").await;\n        assert!(http_stream.is_upgrade_req());\n        let mut response = ResponseHeader::build(StatusCode::CONTINUE, None).unwrap();\n        response.set_version(http::Version::HTTP_11);\n        http_stream\n            .write_response_header(Box::new(response))\n            .await\n            .unwrap();\n        // 100 won't affect body state\n        assert!(http_stream.is_body_done());\n    }\n\n    #[tokio::test]\n    async fn write() {\n        let (mut http_stream, mut handle) = build_req().await;\n        let mut new_response = ResponseHeader::build(StatusCode::OK, None).unwrap();\n        new_response.append_header(\"Foo\", \"Bar\").unwrap();\n        http_stream\n            .write_response_header_ref(&new_response)\n            .await\n            .unwrap();\n        match handle.rx.try_recv().unwrap() {\n            HttpTask::Header(header, end) => {\n                assert_eq!(header.status, StatusCode::OK);\n                assert_eq!(header.headers[\"foo\"], \"Bar\");\n                assert!(!end);\n            }\n            t => panic!(\"unexpected task {t:?}\"),\n        }\n    }\n\n    #[tokio::test]\n    async fn write_informational() {\n        let (mut http_stream, mut handle) = build_req().await;\n        let response_100 = ResponseHeader::build(StatusCode::CONTINUE, None).unwrap();\n        http_stream\n            .write_response_header_ref(&response_100)\n            .await\n            .unwrap();\n        match handle.rx.try_recv().unwrap() {\n            HttpTask::Header(header, end) => {\n                assert_eq!(header.status, StatusCode::CONTINUE);\n                assert!(!end);\n            }\n            t => panic!(\"unexpected task {t:?}\"),\n        }\n\n        let response_200 = ResponseHeader::build(StatusCode::OK, None).unwrap();\n        http_stream\n            .write_response_header_ref(&response_200)\n            .await\n            .unwrap();\n        match handle.rx.try_recv().unwrap() {\n            HttpTask::Header(header, end) => {\n                assert_eq!(header.status, StatusCode::OK);\n                assert!(!end);\n            }\n            t => panic!(\"unexpected task {t:?}\"),\n        }\n    }\n\n    #[tokio::test]\n    async fn write_101_switching_protocol() {\n        let (mut http_stream, mut handle) = build_upgrade_req(\"WebSocket\", \"Upgrade\").await;\n        let mut response_101 =\n            ResponseHeader::build(StatusCode::SWITCHING_PROTOCOLS, None).unwrap();\n        response_101.append_header(\"Foo\", \"Bar\").unwrap();\n        http_stream\n            .write_response_header_ref(&response_101)\n            .await\n            .unwrap();\n\n        match handle.rx.try_recv().unwrap() {\n            HttpTask::Header(header, end) => {\n                assert_eq!(header.status, StatusCode::SWITCHING_PROTOCOLS);\n                assert!(!end);\n            }\n            t => panic!(\"unexpected task {t:?}\"),\n        }\n        assert!(http_stream.upgraded);\n\n        let wire_body = Bytes::from(&b\"PAYLOAD\"[..]);\n        let n = http_stream\n            .write_body(wire_body.clone())\n            .await\n            .unwrap()\n            .unwrap();\n        assert_eq!(wire_body.len(), n);\n        // this write should be ignored\n        let response_502 = ResponseHeader::build(StatusCode::BAD_GATEWAY, None).unwrap();\n        http_stream\n            .write_response_header_ref(&response_502)\n            .await\n            .unwrap();\n\n        match handle.rx.try_recv().unwrap() {\n            HttpTask::Body(body, _end) => {\n                assert_eq!(body.unwrap().len(), n);\n            }\n            t => panic!(\"unexpected task {t:?}\"),\n        }\n        assert_eq!(\n            handle.rx.try_recv().unwrap_err(),\n            mpsc::error::TryRecvError::Empty\n        );\n    }\n\n    #[tokio::test]\n    async fn write_body_cl() {\n        let (mut http_stream, _handle) = build_req().await;\n        let wire_body = Bytes::from(&b\"a\"[..]);\n        let mut new_response = ResponseHeader::build(StatusCode::OK, None).unwrap();\n        new_response.append_header(\"Content-Length\", \"1\").unwrap();\n        http_stream\n            .write_response_header_ref(&new_response)\n            .await\n            .unwrap();\n        assert_eq!(\n            http_stream.body_writer.body_mode,\n            BodyMode::ContentLength(1, 0)\n        );\n        let n = http_stream\n            .write_body(wire_body.clone())\n            .await\n            .unwrap()\n            .unwrap();\n        assert_eq!(wire_body.len(), n);\n        let n = http_stream.finish().await.unwrap().unwrap();\n        assert_eq!(wire_body.len(), n);\n    }\n\n    #[tokio::test]\n    async fn write_body_until_close() {\n        let (mut http_stream, _handle) = build_req().await;\n        let new_response = ResponseHeader::build(StatusCode::OK, None).unwrap();\n        http_stream\n            .write_response_header_ref(&new_response)\n            .await\n            .unwrap();\n        assert_eq!(http_stream.body_writer.body_mode, BodyMode::UntilClose(0));\n        let wire_body = Bytes::from(&b\"PAYLOAD\"[..]);\n        let n = http_stream\n            .write_body(wire_body.clone())\n            .await\n            .unwrap()\n            .unwrap();\n        assert_eq!(wire_body.len(), n);\n        let n = http_stream.finish().await.unwrap().unwrap();\n        assert_eq!(wire_body.len(), n);\n    }\n\n    #[tokio::test]\n    async fn read_with_illegal() {\n        init_log();\n        let input1 = b\"GET /a?q=b c HTTP/1.1\\r\\n\";\n        let input2 = b\"Host: pingora.org\\r\\nContent-Length: 3\\r\\n\\r\\n\";\n        let input3 = b\"abc\";\n        let mock_io = Builder::new().read(&input1[..]).read(&input2[..]).build();\n        let mut http_stream = GenericHttpSession::new_http1(Box::new(mock_io));\n        http_stream.read_request().await.unwrap();\n        let (mut http_stream, handle) = HttpSession::new_from_session(&http_stream);\n        http_stream.read_request().await.unwrap();\n        handle\n            .tx\n            .send(HttpTask::Body(Some(Bytes::from(&input3[..])), false))\n            .await\n            .unwrap();\n\n        assert_eq!(http_stream.get_path(), &b\"/a?q=b%20c\"[..]);\n        let res = http_stream.read_body().await.unwrap().unwrap();\n        assert_eq!(res, &input3[..]);\n        assert_eq!(http_stream.body_reader.body_state, ParseState::Complete(3));\n    }\n\n    #[tokio::test]\n    async fn test_write_body_write_timeout() {\n        let (mut http_stream, _handle) = build_req().await;\n        http_stream.write_timeout = Some(Duration::from_millis(100));\n        let mut new_response = ResponseHeader::build(StatusCode::OK, None).unwrap();\n        new_response.append_header(\"Content-Length\", \"10\").unwrap();\n        http_stream\n            .write_response_header_ref(&new_response)\n            .await\n            .unwrap();\n        let body_write_buf = Bytes::from(&b\"abc\"[..]);\n        http_stream\n            .write_body(body_write_buf.clone())\n            .await\n            .unwrap();\n        http_stream\n            .write_body(body_write_buf.clone())\n            .await\n            .unwrap();\n        http_stream.write_body(body_write_buf).await.unwrap();\n        // channel full\n        let last_body = Bytes::from(&b\"a\"[..]);\n        let res = http_stream.write_body(last_body).await;\n        assert_eq!(res.unwrap_err().etype(), &WriteTimedout);\n    }\n\n    #[tokio::test]\n    async fn test_write_continue_resp() {\n        let (mut http_stream, mut handle) = build_req().await;\n        http_stream.write_continue_response().await.unwrap();\n        match handle.rx.try_recv().unwrap() {\n            HttpTask::Header(header, end) => {\n                assert_eq!(header.status, StatusCode::CONTINUE);\n                assert!(!end);\n            }\n            t => panic!(\"unexpected task {t:?}\"),\n        }\n    }\n\n    async fn session_from_input_no_validate(input: &[u8]) -> (HttpSession, SubrequestHandle) {\n        let mock_io = Builder::new().read(input).build();\n        let mut http_stream = GenericHttpSession::new_http1(Box::new(mock_io));\n        // Read the request in v1 inner session to set up headers properly\n        http_stream.read_request().await.unwrap();\n        let (http_stream, handle) = HttpSession::new_from_session(&http_stream);\n        (http_stream, handle)\n    }\n\n    #[rstest]\n    #[case::negative(\"-1\")]\n    #[case::not_a_number(\"abc\")]\n    #[case::float(\"1.5\")]\n    #[case::empty(\"\")]\n    #[case::spaces(\"  \")]\n    #[case::mixed(\"123abc\")]\n    #[tokio::test]\n    async fn validate_request_rejects_invalid_content_length(#[case] invalid_value: &str) {\n        init_log();\n        let input = format!(\n            \"POST / HTTP/1.1\\r\\nHost: pingora.org\\r\\nContent-Length: {}\\r\\n\\r\\n\",\n            invalid_value\n        );\n        let mock_io = Builder::new().read(input.as_bytes()).build();\n        let mut http_stream = GenericHttpSession::new_http1(Box::new(mock_io));\n        // read_request calls validate_request internally on the v1 inner stream, so it should fail here\n        let res = http_stream.read_request().await;\n        assert!(res.is_err());\n        assert_eq!(\n            res.unwrap_err().etype(),\n            &pingora_error::ErrorType::InvalidHTTPHeader\n        );\n    }\n\n    #[rstest]\n    #[case::valid_zero(\"0\")]\n    #[case::valid_small(\"123\")]\n    #[case::valid_large(\"999999\")]\n    #[tokio::test]\n    async fn validate_request_accepts_valid_content_length(#[case] valid_value: &str) {\n        init_log();\n        let input = format!(\n            \"POST / HTTP/1.1\\r\\nHost: pingora.org\\r\\nContent-Length: {}\\r\\n\\r\\n\",\n            valid_value\n        );\n        let (mut http_stream, _handle) = session_from_input_no_validate(input.as_bytes()).await;\n        let res = http_stream.read_request().await;\n        assert!(res.is_ok());\n    }\n\n    #[tokio::test]\n    async fn validate_request_accepts_no_content_length() {\n        init_log();\n        let input = b\"GET / HTTP/1.1\\r\\nHost: pingora.org\\r\\n\\r\\n\";\n        let (mut http_stream, _handle) = session_from_input_no_validate(input).await;\n        let res = http_stream.read_request().await;\n        assert!(res.is_ok());\n    }\n\n    const POST_CL_UPGRADE_REQ: &[u8] = b\"POST / HTTP/1.1\\r\\nHost: pingora.org\\r\\nUpgrade: websocket\\r\\nConnection: upgrade\\r\\nContent-Length: 10\\r\\n\\r\\n\";\n    const POST_CHUNKED_UPGRADE_REQ: &[u8] = b\"POST / HTTP/1.1\\r\\nHost: pingora.org\\r\\nUpgrade: websocket\\r\\nConnection: upgrade\\r\\nTransfer-Encoding: chunked\\r\\n\\r\\n\";\n    const POST_BODY_DATA: &[u8] = b\"abcdefghij\";\n\n    async fn build_upgrade_req_with_body(header: &[u8]) -> (HttpSession, SubrequestHandle) {\n        let mock_io = Builder::new().read(header).build();\n        let mut http_stream = GenericHttpSession::new_http1(Box::new(mock_io));\n        http_stream.read_request().await.unwrap();\n        let (mut http_stream, handle) = HttpSession::new_from_session(&http_stream);\n        http_stream.read_request().await.unwrap();\n        (http_stream, handle)\n    }\n\n    #[rstest]\n    #[case::content_length(POST_CL_UPGRADE_REQ)]\n    #[case::chunked(POST_CHUNKED_UPGRADE_REQ)]\n    #[tokio::test]\n    async fn read_upgrade_req_with_body(#[case] header: &[u8]) {\n        init_log();\n        let (mut http_stream, handle) = build_upgrade_req_with_body(header).await;\n        assert!(http_stream.is_upgrade_req());\n        // request has body\n        assert!(!http_stream.is_body_done());\n\n        // Send body via the handle\n        handle\n            .tx\n            .send(HttpTask::Body(Some(Bytes::from(POST_BODY_DATA)), true))\n            .await\n            .unwrap();\n\n        let mut buf = vec![];\n        while let Some(b) = http_stream.read_body_bytes().await.unwrap() {\n            buf.put_slice(&b);\n        }\n        assert_eq!(buf, POST_BODY_DATA);\n        assert_eq!(http_stream.body_reader.body_state, ParseState::Complete(10));\n        assert_eq!(http_stream.body_bytes_read(), 10);\n\n        assert!(http_stream.is_body_done());\n\n        let mut response = ResponseHeader::build(StatusCode::SWITCHING_PROTOCOLS, None).unwrap();\n        response.set_version(http::Version::HTTP_11);\n        http_stream\n            .write_response_header(Box::new(response))\n            .await\n            .unwrap();\n        // body reader type switches\n        assert!(!http_stream.is_body_done());\n\n        // now send ws data\n        let ws_data = b\"data\";\n        handle\n            .tx\n            .send(HttpTask::Body(Some(Bytes::from(&ws_data[..])), false))\n            .await\n            .unwrap();\n\n        let buf = http_stream.read_body_bytes().await.unwrap().unwrap();\n        assert_eq!(buf, ws_data.as_slice());\n        assert!(!http_stream.is_body_done());\n\n        // EOF ends body\n        drop(handle.tx);\n        assert!(http_stream.read_body_bytes().await.unwrap().is_none());\n        assert!(http_stream.is_body_done());\n    }\n}\n"
  },
  {
    "path": "pingora-core/src/protocols/http/v1/body.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\nuse bytes::{Buf, BufMut, Bytes, BytesMut};\nuse log::{debug, trace, warn};\nuse pingora_error::{\n    Error,\n    ErrorType::{self, *},\n    OrErr, Result,\n};\nuse std::fmt::Debug;\nuse tokio::io::{AsyncRead, AsyncReadExt, AsyncWrite, AsyncWriteExt};\n\nuse crate::protocols::l4::stream::AsyncWriteVec;\nuse crate::utils::BufRef;\n\n// TODO: make this dynamically adjusted\nconst BODY_BUFFER_SIZE: usize = 1024 * 64;\n// limit how much incomplete chunk-size and chunk-ext to buffer\nconst PARTIAL_CHUNK_HEAD_LIMIT: usize = 1024 * 8;\n// Trailers: https://datatracker.ietf.org/doc/html/rfc9112#section-7.1.2\n// TODO: proper trailer handling and parsing\n// generally trailers are an uncommonly used HTTP/1.1 feature, this is a somewhat\n// arbitrary cap on trailer size after the 0 chunk size (like header buf)\nconst TRAILER_SIZE_LIMIT: usize = 1024 * 64;\n\nconst LAST_CHUNK: &[u8; 5] = &[b'0', CR, LF, CR, LF];\nconst CR: u8 = b'\\r';\nconst LF: u8 = b'\\n';\nconst CRLF: &[u8; 2] = &[CR, LF];\n// This is really the CRLF end of the last trailer (or 0 chunk), + the last CRLF.\nconst TRAILERS_END: &[u8; 4] = &[CR, LF, CR, LF];\n\npub const INVALID_CHUNK: ErrorType = ErrorType::new(\"InvalidChunk\");\npub const INVALID_TRAILER_END: ErrorType = ErrorType::new(\"InvalidTrailerEnd\");\npub const PREMATURE_BODY_END: ErrorType = ErrorType::new(\"PrematureBodyEnd\");\n\n#[derive(Clone, Debug, PartialEq, Eq)]\npub enum ParseState {\n    ToStart,\n    // Complete: total size (contetn-length)\n    Complete(usize),\n    // Partial: size read, remaining size (content-length)\n    Partial(usize, usize),\n    // Chunked: Chunked encoding, prior to the final 0\\r\\n chunk.\n    // size read, next to read in current buf start, read in current buf start, remaining chunked size to read from IO\n    Chunked(usize, usize, usize, usize),\n    // ChunkedFinal: Final section once the 0\\r\\n chunk is read.\n    // size read, trailer sizes parsed so far, use existing buf end, trailers end read\n    ChunkedFinal(usize, usize, usize, u8),\n    // Done: done but there is error, size read\n    Done(usize),\n    // UntilClose: read until connection closed, size read\n    UntilClose(usize),\n}\n\ntype PS = ParseState;\n\nimpl ParseState {\n    pub fn finish(&self, additional_bytes: usize) -> Self {\n        match self {\n            PS::Partial(read, to_read) => PS::Complete(read + to_read),\n            PS::Chunked(read, _, _, _) => PS::Complete(read + additional_bytes),\n            PS::ChunkedFinal(read, _, _, _) => PS::Complete(read + additional_bytes),\n            PS::UntilClose(read) => PS::Complete(read + additional_bytes),\n            _ => self.clone(), /* invalid transaction */\n        }\n    }\n\n    pub fn done(&self, additional_bytes: usize) -> Self {\n        match self {\n            PS::Partial(read, _) => PS::Done(read + additional_bytes),\n            PS::Chunked(read, _, _, _) => PS::Done(read + additional_bytes),\n            PS::ChunkedFinal(read, _, _, _) => PS::Done(read + additional_bytes),\n            PS::UntilClose(read) => PS::Done(read + additional_bytes),\n            _ => self.clone(), /* invalid transaction */\n        }\n    }\n\n    pub fn read_final_chunk(&self, remaining_buf_size: usize) -> Self {\n        match self {\n            PS::Chunked(read, _, _, _) => {\n                // The BodyReader is currently expected to copy the remaining buf\n                // into self.body_buf.\n                //\n                // the 2 == the CRLF from the last chunk-size, 0 + CRLF\n                // because ChunkedFinal is looking for CRLF + CRLF to end\n                // the whole message.\n                // This extra 2 bytes technically ends up cutting into the max trailers size,\n                // which we consider fine for now until full trailers support.\n                PS::ChunkedFinal(*read, 0, remaining_buf_size, 2)\n            }\n            PS::ChunkedFinal(..) => panic!(\"already read final chunk\"),\n            _ => self.clone(), /* invalid transaction */\n        }\n    }\n\n    pub fn partial_chunk(&self, bytes_read: usize, bytes_to_read: usize) -> Self {\n        match self {\n            PS::Chunked(read, _, _, _) => PS::Chunked(read + bytes_read, 0, 0, bytes_to_read),\n            PS::ChunkedFinal(..) => panic!(\"chunked transactions not applicable after final chunk\"),\n            _ => self.clone(), /* invalid transaction */\n        }\n    }\n\n    pub fn multi_chunk(&self, bytes_read: usize, buf_start_index: usize) -> Self {\n        match self {\n            PS::Chunked(read, _, buf_end, _) => {\n                PS::Chunked(read + bytes_read, buf_start_index, *buf_end, 0)\n            }\n            PS::ChunkedFinal(..) => panic!(\"chunked transactions not applicable after final chunk\"),\n            _ => self.clone(), /* invalid transaction */\n        }\n    }\n\n    pub fn partial_chunk_head(&self, head_end: usize, head_size: usize) -> Self {\n        match self {\n            /* inform reader to read more to form a legal chunk */\n            PS::Chunked(read, _, _, _) => PS::Chunked(*read, 0, head_end, head_size),\n            PS::ChunkedFinal(..) => panic!(\"chunked transactions not applicable after final chunk\"),\n            _ => self.clone(), /* invalid transaction */\n        }\n    }\n\n    pub fn new_buf(&self, buf_end: usize) -> Self {\n        match self {\n            PS::Chunked(read, _, _, _) => PS::Chunked(*read, 0, buf_end, 0),\n            PS::ChunkedFinal(..) => panic!(\"chunked transactions not applicable after final chunk\"),\n            _ => self.clone(), /* invalid transaction */\n        }\n    }\n}\n\npub struct BodyReader {\n    pub body_state: ParseState,\n    pub body_buf: Option<BytesMut>,\n    pub body_buf_size: usize,\n    rewind_buf_len: usize,\n    upstream: bool,\n    body_buf_overread: Option<BytesMut>,\n}\n\nimpl BodyReader {\n    pub fn new(upstream: bool) -> Self {\n        BodyReader {\n            body_state: PS::ToStart,\n            body_buf: None,\n            body_buf_size: BODY_BUFFER_SIZE,\n            rewind_buf_len: 0,\n            upstream,\n            body_buf_overread: None,\n        }\n    }\n\n    pub fn need_init(&self) -> bool {\n        matches!(self.body_state, PS::ToStart)\n    }\n\n    pub fn reinit(&mut self) {\n        self.body_state = PS::ToStart;\n    }\n\n    fn prepare_buf(&mut self, buf_to_rewind: &[u8]) {\n        let mut body_buf = BytesMut::with_capacity(self.body_buf_size);\n        if !buf_to_rewind.is_empty() {\n            self.rewind_buf_len = buf_to_rewind.len();\n            // TODO: this is still 1 copy. Make it zero\n            body_buf.put_slice(buf_to_rewind);\n        }\n        if self.body_buf_size > buf_to_rewind.len() {\n            //body_buf.resize(self.body_buf_size, 0);\n            unsafe {\n                body_buf.set_len(self.body_buf_size);\n            }\n        }\n        self.body_buf = Some(body_buf);\n    }\n\n    pub fn init_chunked(&mut self, buf_to_rewind: &[u8]) {\n        self.body_state = PS::Chunked(0, 0, 0, 0);\n        self.prepare_buf(buf_to_rewind);\n    }\n\n    pub fn init_content_length(&mut self, cl: usize, buf_to_rewind: &[u8]) {\n        match cl {\n            0 => {\n                self.body_state = PS::Complete(0);\n                // Store any extra bytes that were read as overread\n                if !buf_to_rewind.is_empty() {\n                    let mut overread = BytesMut::with_capacity(buf_to_rewind.len());\n                    overread.put_slice(buf_to_rewind);\n                    self.body_buf_overread = Some(overread);\n                }\n            }\n            _ => {\n                self.prepare_buf(buf_to_rewind);\n                self.body_state = PS::Partial(0, cl);\n            }\n        }\n    }\n\n    pub fn init_close_delimited(&mut self, buf_to_rewind: &[u8]) {\n        self.prepare_buf(buf_to_rewind);\n        self.body_state = PS::UntilClose(0);\n    }\n\n    /// Convert how we interpret the remainder of the body to read until close.\n    /// This is used for responses without explicit framing (e.g., HTTP/1.0 responses).\n    ///\n    /// Does nothing if already in close-delimited mode.\n    pub fn convert_to_close_delimited(&mut self) {\n        if matches!(self.body_state, PS::UntilClose(_)) {\n            // nothing to do, already in close-delimited mode\n            return;\n        }\n\n        if self.rewind_buf_len == 0 {\n            // take any extra bytes and send them as-is,\n            // reset body counter\n            let extra = self.body_buf_overread.take();\n            let buf = extra.as_deref().unwrap_or_default();\n            self.prepare_buf(buf);\n        } // if rewind_buf_len is not 0, body read has not yet been polled\n        self.body_state = PS::UntilClose(0);\n    }\n\n    pub fn get_body(&self, buf_ref: &BufRef) -> &[u8] {\n        // TODO: these get_*() could panic. handle them better\n        buf_ref.get(self.body_buf.as_ref().unwrap())\n    }\n\n    #[allow(dead_code)]\n    pub fn get_body_overread(&self) -> Option<&[u8]> {\n        self.body_buf_overread.as_deref()\n    }\n\n    pub fn has_bytes_overread(&self) -> bool {\n        self.get_body_overread().is_some_and(|b| !b.is_empty())\n    }\n\n    pub fn body_done(&self) -> bool {\n        matches!(self.body_state, PS::Complete(_) | PS::Done(_))\n    }\n\n    pub fn body_empty(&self) -> bool {\n        self.body_state == PS::Complete(0)\n    }\n\n    fn finish_body_buf(&mut self, end_of_body: usize, total_read: usize) {\n        let body_buf_mut = self.body_buf.as_mut().expect(\"must have read body buf\");\n        // remove unused buffer\n        body_buf_mut.truncate(total_read);\n        let overread_bytes = body_buf_mut.split_off(end_of_body);\n        self.body_buf_overread = (!overread_bytes.is_empty()).then_some(overread_bytes);\n    }\n\n    pub async fn read_body<S>(&mut self, stream: &mut S) -> Result<Option<BufRef>>\n    where\n        S: AsyncRead + Unpin + Send,\n    {\n        match self.body_state {\n            PS::Complete(_) => Ok(None),\n            PS::Done(_) => Ok(None),\n            PS::Partial(_, _) => self.do_read_body(stream).await,\n            PS::Chunked(..) => self.do_read_chunked_body(stream).await,\n            PS::ChunkedFinal(..) => self.do_read_chunked_body_final(stream).await,\n            PS::UntilClose(_) => self.do_read_body_until_closed(stream).await,\n            PS::ToStart => panic!(\"need to init BodyReader first\"),\n        }\n    }\n\n    pub async fn do_read_body<S>(&mut self, stream: &mut S) -> Result<Option<BufRef>>\n    where\n        S: AsyncRead + Unpin + Send,\n    {\n        let mut body_buf = self.body_buf.as_deref_mut().unwrap();\n        let mut n = self.rewind_buf_len;\n        self.rewind_buf_len = 0; // we only need to read rewind data once\n        if n == 0 {\n            // downstream should not discard remaining data if peer sent more.\n            if !self.upstream {\n                if let PS::Partial(_, to_read) = self.body_state {\n                    if to_read < body_buf.len() {\n                        body_buf = &mut body_buf[..to_read];\n                    }\n                }\n            }\n            /* Need to actually read */\n            n = stream\n                .read(body_buf)\n                .await\n                .or_err(ReadError, \"when reading body\")?;\n        }\n        match self.body_state {\n            PS::Partial(read, to_read) => {\n                debug!(\n                    \"BodyReader body_state: {:?}, read data from IO: {n}\",\n                    self.body_state\n                );\n                if n == 0 {\n                    self.body_state = PS::Done(read);\n                    Error::e_explain(ConnectionClosed, format!(\n                        \"Peer prematurely closed connection with {} bytes of body remaining to read\",\n                        to_read\n                    ))\n                } else if n >= to_read {\n                    if n > to_read {\n                        warn!(\n                            \"Peer sent more data then expected: extra {}\\\n                               bytes, discarding them\",\n                            n - to_read\n                        )\n                    }\n                    self.body_state = PS::Complete(read + to_read);\n                    self.finish_body_buf(to_read, n);\n                    Ok(Some(BufRef::new(0, to_read)))\n                } else {\n                    self.body_state = PS::Partial(read + n, to_read - n);\n                    Ok(Some(BufRef::new(0, n)))\n                }\n            }\n            _ => panic!(\"wrong body state: {:?}\", self.body_state),\n        }\n    }\n\n    pub async fn do_read_body_until_closed<S>(&mut self, stream: &mut S) -> Result<Option<BufRef>>\n    where\n        S: AsyncRead + Unpin + Send,\n    {\n        let body_buf = self.body_buf.as_deref_mut().unwrap();\n        let mut n = self.rewind_buf_len;\n        self.rewind_buf_len = 0; // we only need to read rewind data once\n        if n == 0 {\n            /* Need to actually read */\n            n = stream\n                .read(body_buf)\n                .await\n                .or_err(ReadError, \"when reading body\")?;\n        }\n        match self.body_state {\n            PS::UntilClose(read) => {\n                if n == 0 {\n                    self.body_state = PS::Complete(read);\n                    Ok(None)\n                } else {\n                    self.body_state = PS::UntilClose(read + n);\n                    Ok(Some(BufRef::new(0, n)))\n                }\n            }\n            _ => panic!(\"wrong body state: {:?}\", self.body_state),\n        }\n    }\n\n    pub async fn do_read_chunked_body<S>(&mut self, stream: &mut S) -> Result<Option<BufRef>>\n    where\n        S: AsyncRead + Unpin + Send,\n    {\n        match self.body_state {\n            PS::Chunked(\n                total_read,\n                existing_buf_start,\n                mut existing_buf_end,\n                mut expecting_from_io,\n            ) => {\n                if existing_buf_start == 0 {\n                    // read a new buf from IO\n                    let body_buf = self.body_buf.as_deref_mut().unwrap();\n                    if existing_buf_end == 0 {\n                        existing_buf_end = self.rewind_buf_len;\n                        self.rewind_buf_len = 0; // we only need to read rewind data once\n                        if existing_buf_end == 0 {\n                            existing_buf_end = stream\n                                .read(body_buf)\n                                .await\n                                .or_err(ReadError, \"when reading body\")?;\n                        }\n                    } else {\n                        /* existing_buf_end != 0 this is partial chunk head */\n                        /* copy the #expecting_from_io bytes until index existing_buf_end\n                         * to the front and read more to form a valid chunk head.\n                         * existing_buf_end is the end of the partial head and\n                         * expecting_from_io is the len of it */\n                        body_buf\n                            .copy_within(existing_buf_end - expecting_from_io..existing_buf_end, 0);\n                        let new_bytes = stream\n                            .read(&mut body_buf[expecting_from_io..])\n                            .await\n                            .or_err(ReadError, \"when reading body\")?;\n                        if new_bytes == 0 {\n                            self.body_state = self.body_state.done(0);\n                            return Error::e_explain(\n                                ConnectionClosed,\n                                format!(\n                                    \"Connection prematurely closed without the termination chunk \\\n                                    (partial chunk head), read {total_read} bytes\"\n                                ),\n                            );\n                        }\n\n                        /* more data is read, extend the buffer */\n                        existing_buf_end = expecting_from_io + new_bytes;\n                        expecting_from_io = 0;\n                    }\n                    self.body_state = self.body_state.new_buf(existing_buf_end);\n                }\n                if existing_buf_end == 0 {\n                    self.body_state = self.body_state.done(0);\n                    Error::e_explain(\n                        ConnectionClosed,\n                        format!(\n                            \"Connection prematurely closed without the termination chunk, \\\n                            read {total_read} bytes\"\n                        ),\n                    )\n                } else {\n                    if expecting_from_io > 0 {\n                        let body_buf = self.body_buf.as_ref().unwrap();\n                        trace!(\n                            \"partial chunk payload, expecting_from_io: {}, \\\n                                existing_buf_end {}, buf: {:?}\",\n                            expecting_from_io,\n                            existing_buf_end,\n                            self.body_buf.as_ref().unwrap()[..existing_buf_end].escape_ascii()\n                        );\n\n                        // partial chunk payload, will read more\n                        if expecting_from_io >= existing_buf_end + 2 {\n                            // not enough (doesn't contain CRLF end)\n                            self.body_state = self.body_state.partial_chunk(\n                                existing_buf_end,\n                                expecting_from_io - existing_buf_end,\n                            );\n                            return Ok(Some(BufRef::new(0, existing_buf_end)));\n                        }\n                        /* could be expecting DATA + CRLF or just CRLF */\n                        let payload_size = expecting_from_io.saturating_sub(2);\n                        /* expecting_from_io < existing_buf_end + 2 */\n                        let need_lf_only = expecting_from_io == 1; // otherwise we need the whole CRLF\n                        if expecting_from_io > existing_buf_end {\n                            // potentially:\n                            // | CR | LF |\n                            //      |    |\n                            // (existing_buf_end)\n                            //           |\n                            //           (expecting_from_io)\n                            if payload_size < existing_buf_end {\n                                Self::validate_crlf(\n                                    &mut self.body_state,\n                                    &body_buf[payload_size..existing_buf_end],\n                                    need_lf_only,\n                                    false,\n                                )?;\n                            }\n                        } else {\n                            // expecting_from_io <= existing_buf_end\n                            // chunk CRLF end should end here\n                            assert!(Self::validate_crlf(\n                                &mut self.body_state,\n                                &body_buf[payload_size..expecting_from_io],\n                                need_lf_only,\n                                false,\n                            )?);\n                        }\n                        if expecting_from_io >= existing_buf_end {\n                            self.body_state = self\n                                .body_state\n                                .partial_chunk(payload_size, expecting_from_io - existing_buf_end);\n\n                            return Ok(Some(BufRef::new(0, payload_size)));\n                        }\n\n                        /* expecting_from_io < existing_buf_end */\n                        self.body_state =\n                            self.body_state.multi_chunk(payload_size, expecting_from_io);\n\n                        return Ok(Some(BufRef::new(0, payload_size)));\n                    }\n                    let (buf_res, last_chunk_size_end) =\n                        self.parse_chunked_buf(existing_buf_start, existing_buf_end)?;\n                    if buf_res.is_some() {\n                        if let Some(idx) = last_chunk_size_end {\n                            // just read the last 0 + CRLF, but not final end CRLF\n                            // copy the rest of the buffer to the start of the body_buf\n                            // so we can parse the remaining bytes as trailers / end\n                            let body_buf = self.body_buf.as_deref_mut().unwrap();\n                            trace!(\n                                \"last chunk size end buf {:?}\",\n                                &body_buf[..existing_buf_end].escape_ascii(),\n                            );\n                            body_buf.copy_within(idx..existing_buf_end, 0);\n                        }\n                    }\n                    Ok(buf_res)\n                }\n            }\n            _ => panic!(\"wrong body state: {:?}\", self.body_state),\n        }\n    }\n\n    // Returns: BufRef of next body chunk,\n    // terminating chunk-size index end if read completely (0 + CRLF).\n    // Note input indices are absolute (to body_buf).\n    fn parse_chunked_buf(\n        &mut self,\n        buf_index_start: usize,\n        buf_index_end: usize,\n    ) -> Result<(Option<BufRef>, Option<usize>)> {\n        let buf = &self.body_buf.as_ref().unwrap()[buf_index_start..buf_index_end];\n        let chunk_status = httparse::parse_chunk_size(buf);\n        match chunk_status {\n            Ok(status) => {\n                match status {\n                    httparse::Status::Complete((payload_index, chunk_size)) => {\n                        // TODO: Check chunk_size overflow\n                        trace!(\n                            \"Got size {chunk_size}, payload_index: {payload_index}, chunk: {:?}\",\n                            String::from_utf8_lossy(buf).escape_default(),\n                        );\n                        let chunk_size = chunk_size as usize;\n                        // https://github.com/seanmonstar/httparse/issues/149\n                        // httparse does not treat zero-size chunk differently, it does not check\n                        // that terminating chunk is 0 + double CRLF\n                        if chunk_size == 0 {\n                            /* terminating chunk, also need to handle trailer. */\n                            let chunk_end_index = payload_index + 2;\n                            return if chunk_end_index <= buf.len()\n                                && buf[payload_index..chunk_end_index] == CRLF[..]\n                            {\n                                // full terminating CRLF MAY exist in current buf\n                                // Skip ChunkedFinal state and go directly to Complete\n                                // as optimization.\n                                self.body_state = self.body_state.finish(0);\n                                self.finish_body_buf(\n                                    buf_index_start + chunk_end_index,\n                                    buf_index_end,\n                                );\n                                Ok((None, Some(buf_index_start + payload_index)))\n                            } else {\n                                // Indicate start of parsing final chunked trailers,\n                                // with remaining buf to read\n                                self.body_state = self.body_state.read_final_chunk(\n                                    buf_index_end - (buf_index_start + payload_index),\n                                );\n\n                                Ok((\n                                    Some(BufRef::new(0, 0)),\n                                    Some(buf_index_start + payload_index),\n                                ))\n                            };\n                        }\n                        // chunk-size CRLF [payload_index] byte*[chunk_size] CRLF\n                        let data_end_index = payload_index + chunk_size;\n                        let chunk_end_index = data_end_index + 2;\n                        if chunk_end_index >= buf.len() {\n                            // no multi chunk in this buf\n                            let actual_size = if data_end_index > buf.len() {\n                                buf.len() - payload_index\n                            } else {\n                                chunk_size\n                            };\n\n                            let crlf_start = chunk_end_index.saturating_sub(2);\n                            if crlf_start < buf.len() {\n                                Self::validate_crlf(\n                                    &mut self.body_state,\n                                    &buf[crlf_start..],\n                                    false,\n                                    false,\n                                )?;\n                            }\n                            // else need to read more to get to CRLF\n\n                            self.body_state = self\n                                .body_state\n                                .partial_chunk(actual_size, chunk_end_index - buf.len());\n                            return Ok((\n                                Some(BufRef::new(buf_index_start + payload_index, actual_size)),\n                                None,\n                            ));\n                        }\n                        /* got multiple chunks, return the first */\n                        assert!(Self::validate_crlf(\n                            &mut self.body_state,\n                            &buf[data_end_index..chunk_end_index],\n                            false,\n                            false,\n                        )?);\n                        self.body_state = self\n                            .body_state\n                            .multi_chunk(chunk_size, buf_index_start + chunk_end_index);\n                        Ok((\n                            Some(BufRef::new(buf_index_start + payload_index, chunk_size)),\n                            None,\n                        ))\n                    }\n                    httparse::Status::Partial => {\n                        if buf.len() > PARTIAL_CHUNK_HEAD_LIMIT {\n                            // https://datatracker.ietf.org/doc/html/rfc9112#name-chunk-extensions\n                            // \"A server ought to limit the total length of chunk extensions received\"\n                            // The buf.len() here is the total length of chunk-size + chunk-ext seen\n                            // so far. This check applies to both server and client\n                            self.body_state = self.body_state.done(0);\n                            Error::e_explain(INVALID_CHUNK, \"Chunk ext over limit\")\n                        } else {\n                            self.body_state =\n                                self.body_state.partial_chunk_head(buf_index_end, buf.len());\n                            Ok((Some(BufRef::new(0, 0)), None))\n                        }\n                    }\n                }\n            }\n            Err(e) => {\n                let context = format!(\"Invalid chunked encoding: {e:?}\");\n                debug!(\n                    \"{context}, {:?}\",\n                    String::from_utf8_lossy(buf).escape_default()\n                );\n                self.body_state = self.body_state.done(0);\n                Error::e_explain(INVALID_CHUNK, context)\n            }\n        }\n    }\n\n    pub async fn do_read_chunked_body_final<S>(&mut self, stream: &mut S) -> Result<Option<BufRef>>\n    where\n        S: AsyncRead + Unpin + Send,\n    {\n        // parse section after last-chunk: https://datatracker.ietf.org/doc/html/rfc9112#section-7.1\n        // This is the section after the final chunk we're trying to read, which can include\n        // HTTP1 trailers (currently we just discard them).\n        // Really we are just waiting for a consecutive CRLF + CRLF to end the body.\n        match self.body_state {\n            PS::ChunkedFinal(read, trailers_read, existing_buf_end, end_read) => {\n                let body_buf = self.body_buf.as_deref_mut().unwrap();\n                let (buf, n) = if existing_buf_end != 0 {\n                    // finish rest of buf that was read with Chunked state\n                    // existing_buf_end is non-zero only once\n                    self.body_state = PS::ChunkedFinal(read, trailers_read, 0, end_read);\n                    (&body_buf[..existing_buf_end], existing_buf_end)\n                } else {\n                    let n = stream\n                        .read(body_buf)\n                        .await\n                        .or_err(ReadError, \"when reading trailers end\")?;\n\n                    (&body_buf[..n], n)\n                };\n\n                if n == 0 {\n                    self.body_state = PS::Done(read);\n                    return Error::e_explain(\n                        ConnectionClosed,\n                        format!(\n                            \"Connection prematurely closed without the termination chunk, \\\n                            read {read} bytes, {trailers_read} trailer bytes\"\n                        ),\n                    );\n                }\n\n                let mut start = 0;\n                // try to find end within the current IO buffer\n                while start < n {\n                    // Adjusts body state through each iteration to add trailers read\n                    // Each iteration finds the next CR or LF to advance the buf\n                    let (trailers_read, end_read) = match self.body_state {\n                        PS::ChunkedFinal(_, new_trailers_read, _, new_end_read) => {\n                            (new_trailers_read, new_end_read)\n                        }\n                        _ => unreachable!(),\n                    };\n\n                    let mut buf = &buf[start..n];\n                    trace!(\n                        \"Parsing chunk end for buf {:?}\",\n                        String::from_utf8_lossy(buf).escape_default(),\n                    );\n\n                    if end_read == 0 {\n                        // find the next CRLF sequence / potential end\n                        let (trailers_read, no_crlf) =\n                            if let Some(p) = buf.iter().position(|b| *b == CR || *b == LF) {\n                                buf = &buf[p..];\n                                start += p;\n                                (trailers_read + p, false)\n                            } else {\n                                // consider this all trailer bytes\n                                (trailers_read + (n - start), true)\n                            };\n\n                        if trailers_read > TRAILER_SIZE_LIMIT {\n                            self.body_state = self.body_state.done(0);\n                            return Error::e_explain(\n                                INVALID_TRAILER_END,\n                                \"Trailer size over limit\",\n                            );\n                        }\n\n                        self.body_state = PS::ChunkedFinal(read, trailers_read, 0, 0);\n\n                        if no_crlf {\n                            // break and allow polling read body again\n                            break;\n                        }\n                    }\n                    match Self::parse_trailers_end(&mut self.body_state, buf)? {\n                        TrailersEndParseState::NotEnd(next_parse_index) => {\n                            trace!(\n                                \"Parsing chunk end for buf {:?}, resume at {next_parse_index}\",\n                                String::from_utf8_lossy(buf).escape_default(),\n                            );\n\n                            start += next_parse_index;\n                        }\n                        TrailersEndParseState::Complete(end_idx) => {\n                            trace!(\n                                \"Parsing chunk end for buf {:?}, finished at {end_idx}\",\n                                String::from_utf8_lossy(buf).escape_default(),\n                            );\n\n                            self.finish_body_buf(start + end_idx, n);\n                            return Ok(None);\n                        }\n                    }\n                }\n            }\n            _ => panic!(\"wrong body state: {:?}\", self.body_state),\n        }\n        // indicate final section is not done\n        Ok(Some(BufRef(0, 0)))\n    }\n\n    // Parses up to one CRLF at a time to determine if, given the body state,\n    // we've parsed a full trailer end.\n    // Panics if empty buffer is given.\n    fn parse_trailers_end(\n        body_state: &mut ParseState,\n        buf: &[u8],\n    ) -> Result<TrailersEndParseState> {\n        assert!(!buf.is_empty(), \"parse_trailers_end given empty buffer\");\n\n        match body_state.clone() {\n            PS::ChunkedFinal(read, trailers_read, _, end_read) => {\n                // Look at the body buf we just read and see if it matches\n                // the ending CRLF + CRLF sequence.\n                let end_read = end_read as usize;\n                assert!(end_read < TRAILERS_END.len());\n                let to_read = std::cmp::min(buf.len(), TRAILERS_END.len() - end_read);\n                let buf = &buf[..to_read];\n\n                // If the start of the buf is not CRLF and we are not in the middle of reading a\n                // valid CRLF sequence, return to let caller seek for next CRLF\n                if end_read % 2 == 0 && buf[0] != CR && buf[0] != LF {\n                    trace!(\n                        \"parse trailers end {:?}, not CRLF sequence\",\n                        String::from_utf8_lossy(buf).escape_default(),\n                    );\n                    *body_state = PS::ChunkedFinal(read, trailers_read + end_read, 0, 0);\n                    return Ok(TrailersEndParseState::NotEnd(0));\n                }\n                // Check for malformed CRLF in trailers (or final end of trailers section)\n                let next_parse_index = match end_read {\n                    0 | 2 => {\n                        // expect start with CR\n                        if Self::validate_crlf(body_state, buf, false, true)? {\n                            // found CR + LF\n                            2\n                        } else {\n                            // read CR at least\n                            1\n                        }\n                    }\n                    1 | 3 => {\n                        // assert: only way this can return false is with an empty buffer\n                        assert!(Self::validate_crlf(body_state, buf, true, true)?);\n                        1\n                    }\n                    _ => unreachable!(),\n                };\n                let next_end_read = end_read + next_parse_index;\n                let finished = next_end_read == TRAILERS_END.len();\n                if finished {\n                    trace!(\n                        \"parse trailers end {:?}, complete {next_end_read}\",\n                        String::from_utf8_lossy(buf).escape_default(),\n                    );\n                    *body_state = PS::Complete(read);\n                    Ok(TrailersEndParseState::Complete(next_parse_index))\n                } else {\n                    // either we read the end of one trailer and another one follows,\n                    // or trailer end CRLF sequence so far is valid but we need more bytes\n                    // to determine if more CRLF actually follows\n                    trace!(\n                        \"parse trailers end {:?}, resume at {next_parse_index}\",\n                        String::from_utf8_lossy(buf).escape_default(),\n                    );\n                    // unwrap safety for try_into() u8: next_end_read always <\n                    // TRAILERS_END.len()\n                    *body_state =\n                        PS::ChunkedFinal(read, trailers_read, 0, next_end_read.try_into().unwrap());\n                    Ok(TrailersEndParseState::NotEnd(next_parse_index))\n                }\n            }\n            _ => panic!(\"wrong body state: {:?}\", body_state),\n        }\n    }\n\n    // Validates that the starting bytes of `buf` are the expected CRLF bytes.\n    // Expects: buf that starts at the indices where CRLF should be for chunked bodies.\n    // If need_lf_only, we will only check for LF, else we will check starting with CR.\n    //\n    // Returns Ok() if buf begins with expected bytes (CR, LF, or CRLF).\n    // The inner bool returned is whether the whole CRLF sequence was completed.\n    fn validate_crlf(\n        body_state: &mut ParseState,\n        buf: &[u8],\n        need_lf_only: bool,\n        for_trailer_end: bool,\n    ) -> Result<bool> {\n        let etype = if for_trailer_end {\n            INVALID_TRAILER_END\n        } else {\n            INVALID_CHUNK\n        };\n        if need_lf_only {\n            if buf.is_empty() {\n                Ok(false)\n            } else {\n                let b = &buf[..1];\n                if b == b\"\\n\" {\n                    // only LF left\n                    Ok(true)\n                } else {\n                    *body_state = body_state.done(0);\n                    Error::e_explain(\n                        etype,\n                        format!(\n                            \"Invalid chunked encoding: {} was not LF\",\n                            String::from_utf8_lossy(b).escape_default(),\n                        ),\n                    )\n                }\n            }\n        } else {\n            match buf.len() {\n                0 => Ok(false),\n                1 => {\n                    let b = &buf[..1];\n                    if b == b\"\\r\" {\n                        Ok(false)\n                    } else {\n                        *body_state = body_state.done(0);\n                        Error::e_explain(\n                            etype,\n                            format!(\n                                \"Invalid chunked encoding: {} was not CR\",\n                                String::from_utf8_lossy(b).escape_default(),\n                            ),\n                        )\n                    }\n                }\n                _ => {\n                    let b = &buf[..2];\n                    if b == b\"\\r\\n\" {\n                        Ok(true)\n                    } else {\n                        *body_state = body_state.done(0);\n                        Error::e_explain(\n                            etype,\n                            format!(\n                                \"Invalid chunked encoding: {} was not CRLF\",\n                                String::from_utf8_lossy(b).escape_default(),\n                            ),\n                        )\n                    }\n                }\n            }\n        }\n    }\n}\n\npub enum TrailersEndParseState {\n    NotEnd(usize),   // start of bytes after CR or LF bytes\n    Complete(usize), // index of message completion\n}\n\n#[derive(Clone, Debug, PartialEq, Eq)]\npub enum BodyMode {\n    ToSelect,\n    ContentLength(usize, usize), // total length to write, bytes already written\n    ChunkedEncoding(usize),      //bytes written\n    UntilClose(usize),           //bytes written\n    Complete(usize),             //bytes written\n}\n\ntype BM = BodyMode;\n\npub struct BodyWriter {\n    pub body_mode: BodyMode,\n}\n\nimpl BodyWriter {\n    pub fn new() -> Self {\n        BodyWriter {\n            body_mode: BM::ToSelect,\n        }\n    }\n\n    pub fn init_chunked(&mut self) {\n        self.body_mode = BM::ChunkedEncoding(0);\n    }\n\n    pub fn init_close_delimited(&mut self) {\n        self.body_mode = BM::UntilClose(0);\n    }\n\n    pub fn init_content_length(&mut self, cl: usize) {\n        self.body_mode = BM::ContentLength(cl, 0);\n    }\n\n    pub fn convert_to_close_delimited(&mut self) {\n        if matches!(self.body_mode, BodyMode::UntilClose(_)) {\n            // nothing to do, already in close-delimited mode\n            return;\n        }\n\n        // NOTE: any stream buffered data will be flushed in next\n        // close-delimited write\n        // reset body state to close-delimited (UntilClose)\n        self.body_mode = BM::UntilClose(0);\n    }\n\n    // NOTE on buffering/flush stream when writing the body\n    // Buffering writes can reduce the syscalls hence improves efficiency of the system\n    // But it hurts real time communication\n    // So we only allow buffering when the body size is known ahead, which is less likely\n    // to be real time interaction\n\n    pub async fn write_body<S>(&mut self, stream: &mut S, buf: &[u8]) -> Result<Option<usize>>\n    where\n        S: AsyncWrite + Unpin + Send,\n    {\n        trace!(\"Writing Body, size: {}\", buf.len());\n        match self.body_mode {\n            BM::Complete(_) => Ok(None),\n            BM::ContentLength(_, _) => self.do_write_body(stream, buf).await,\n            BM::ChunkedEncoding(_) => self.do_write_chunked_body(stream, buf).await,\n            BM::UntilClose(_) => self.do_write_until_close_body(stream, buf).await,\n            BM::ToSelect => Ok(None), // Error here?\n        }\n    }\n\n    pub fn finished(&self) -> bool {\n        match self.body_mode {\n            BM::Complete(_) => true,\n            BM::ContentLength(total, written) => written >= total,\n            _ => false,\n        }\n    }\n\n    pub fn is_close_delimited(&self) -> bool {\n        matches!(self.body_mode, BM::UntilClose(_))\n    }\n\n    async fn do_write_body<S>(&mut self, stream: &mut S, buf: &[u8]) -> Result<Option<usize>>\n    where\n        S: AsyncWrite + Unpin + Send,\n    {\n        match self.body_mode {\n            BM::ContentLength(total, written) => {\n                if written >= total {\n                    // already written full length\n                    return Ok(None);\n                }\n                let mut to_write = total - written;\n                if to_write < buf.len() {\n                    warn!(\"Trying to write data over content-length: {total}\");\n                } else {\n                    to_write = buf.len();\n                }\n                let res = stream.write_all(&buf[..to_write]).await;\n                match res {\n                    Ok(()) => {\n                        self.body_mode = BM::ContentLength(total, written + to_write);\n                        if self.finished() {\n                            stream.flush().await.or_err(WriteError, \"flushing body\")?;\n                        }\n                        Ok(Some(to_write))\n                    }\n                    Err(e) => Error::e_because(WriteError, \"while writing body\", e),\n                }\n            }\n            _ => panic!(\"wrong body mode: {:?}\", self.body_mode),\n        }\n    }\n\n    async fn do_write_chunked_body<S>(\n        &mut self,\n        stream: &mut S,\n        buf: &[u8],\n    ) -> Result<Option<usize>>\n    where\n        S: AsyncWrite + Unpin + Send,\n    {\n        match self.body_mode {\n            BM::ChunkedEncoding(written) => {\n                let chunk_size = buf.len();\n\n                let chuck_size_buf = format!(\"{:X}\\r\\n\", chunk_size);\n                let mut output_buf = Bytes::from(chuck_size_buf).chain(buf).chain(&b\"\\r\\n\"[..]);\n                stream\n                    .write_vec_all(&mut output_buf)\n                    .await\n                    .or_err(WriteError, \"while writing body\")?;\n                stream.flush().await.or_err(WriteError, \"flushing body\")?;\n                self.body_mode = BM::ChunkedEncoding(written + chunk_size);\n                Ok(Some(chunk_size))\n            }\n            _ => panic!(\"wrong body mode: {:?}\", self.body_mode),\n        }\n    }\n\n    async fn do_write_until_close_body<S>(\n        &mut self,\n        stream: &mut S,\n        buf: &[u8],\n    ) -> Result<Option<usize>>\n    where\n        S: AsyncWrite + Unpin + Send,\n    {\n        match self.body_mode {\n            BM::UntilClose(written) => {\n                let res = stream.write_all(buf).await;\n                match res {\n                    Ok(()) => {\n                        self.body_mode = BM::UntilClose(written + buf.len());\n                        stream.flush().await.or_err(WriteError, \"flushing body\")?;\n                        Ok(Some(buf.len()))\n                    }\n                    Err(e) => Error::e_because(WriteError, \"while writing body\", e),\n                }\n            }\n            _ => panic!(\"wrong body mode: {:?}\", self.body_mode),\n        }\n    }\n\n    pub async fn finish<S>(&mut self, stream: &mut S) -> Result<Option<usize>>\n    where\n        S: AsyncWrite + Unpin + Send,\n    {\n        match self.body_mode {\n            BM::Complete(_) => Ok(None),\n            BM::ContentLength(_, _) => self.do_finish_body(stream),\n            BM::ChunkedEncoding(_) => self.do_finish_chunked_body(stream).await,\n            BM::UntilClose(_) => self.do_finish_until_close_body(stream),\n            BM::ToSelect => Ok(None),\n        }\n    }\n\n    fn do_finish_body<S>(&mut self, _stream: S) -> Result<Option<usize>> {\n        match self.body_mode {\n            BM::ContentLength(total, written) => {\n                self.body_mode = BM::Complete(written);\n                if written < total {\n                    return Error::e_explain(\n                        PREMATURE_BODY_END,\n                        format!(\"Content-length: {total} bytes written: {written}\"),\n                    );\n                }\n                Ok(Some(written))\n            }\n            _ => panic!(\"wrong body mode: {:?}\", self.body_mode),\n        }\n    }\n\n    async fn do_finish_chunked_body<S>(&mut self, stream: &mut S) -> Result<Option<usize>>\n    where\n        S: AsyncWrite + Unpin + Send,\n    {\n        match self.body_mode {\n            BM::ChunkedEncoding(written) => {\n                let res = stream.write_all(&LAST_CHUNK[..]).await;\n                self.body_mode = BM::Complete(written);\n                match res {\n                    Ok(()) => Ok(Some(written)),\n                    Err(e) => Error::e_because(WriteError, \"while writing body\", e),\n                }\n            }\n            _ => panic!(\"wrong body mode: {:?}\", self.body_mode),\n        }\n    }\n\n    fn do_finish_until_close_body<S>(&mut self, _stream: &mut S) -> Result<Option<usize>> {\n        match self.body_mode {\n            BM::UntilClose(written) => {\n                self.body_mode = BM::Complete(written);\n                Ok(Some(written))\n            }\n            _ => panic!(\"wrong body mode: {:?}\", self.body_mode),\n        }\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use super::*;\n    use tokio_test::io::Builder;\n\n    fn init_log() {\n        let _ = env_logger::builder().is_test(true).try_init();\n    }\n\n    #[tokio::test]\n    async fn read_with_body_content_length() {\n        init_log();\n        let input = b\"abc\";\n        let mut mock_io = Builder::new().read(&input[..]).build();\n        let mut body_reader = BodyReader::new(false);\n        body_reader.init_content_length(3, b\"\");\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(0, 3));\n        assert_eq!(body_reader.body_state, ParseState::Complete(3));\n        assert_eq!(input, body_reader.get_body(&res));\n        assert_eq!(body_reader.get_body_overread(), None);\n    }\n\n    #[tokio::test]\n    async fn read_with_body_content_length_2() {\n        init_log();\n        let input1 = b\"a\";\n        let input2 = b\"bc\";\n        let mut mock_io = Builder::new().read(&input1[..]).read(&input2[..]).build();\n        let mut body_reader = BodyReader::new(false);\n        body_reader.init_content_length(3, b\"\");\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(0, 1));\n        assert_eq!(body_reader.body_state, ParseState::Partial(1, 2));\n        assert_eq!(input1, body_reader.get_body(&res));\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(0, 2));\n        assert_eq!(body_reader.body_state, ParseState::Complete(3));\n        assert_eq!(input2, body_reader.get_body(&res));\n        assert_eq!(body_reader.get_body_overread(), None);\n    }\n\n    #[tokio::test]\n    async fn read_with_body_content_length_less() {\n        init_log();\n        let input1 = b\"a\";\n        let input2 = b\"\"; // simulating close\n        let mut mock_io = Builder::new().read(&input1[..]).read(&input2[..]).build();\n        let mut body_reader = BodyReader::new(false);\n        body_reader.init_content_length(3, b\"\");\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(0, 1));\n        assert_eq!(body_reader.body_state, ParseState::Partial(1, 2));\n        assert_eq!(input1, body_reader.get_body(&res));\n        let res = body_reader.read_body(&mut mock_io).await.unwrap_err();\n        assert_eq!(&ConnectionClosed, res.etype());\n        assert_eq!(body_reader.body_state, ParseState::Done(1));\n        assert_eq!(body_reader.get_body_overread(), None);\n    }\n\n    #[tokio::test]\n    async fn read_with_body_content_length_more() {\n        init_log();\n        let input1 = b\"a\";\n        let input2 = b\"bcd\";\n        let mut mock_io = Builder::new().read(&input1[..]).read(&input2[..]).build();\n        let mut body_reader = BodyReader::new(false);\n        body_reader.init_content_length(3, b\"\");\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(0, 1));\n        assert_eq!(body_reader.body_state, ParseState::Partial(1, 2));\n        assert_eq!(input1, body_reader.get_body(&res));\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(0, 2));\n        assert_eq!(body_reader.body_state, ParseState::Complete(3));\n        assert_eq!(&input2[0..2], body_reader.get_body(&res));\n        // read remaining data\n        body_reader.init_content_length(1, b\"\");\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(body_reader.body_state, ParseState::Complete(1));\n        assert_eq!(&input2[2..], body_reader.get_body(&res));\n    }\n\n    #[tokio::test]\n    async fn read_with_body_content_length_overread() {\n        init_log();\n        let input1 = b\"a\";\n        let input2 = b\"bcd\";\n        let mut mock_io = Builder::new().read(&input1[..]).read(&input2[..]).build();\n        let mut body_reader = BodyReader::new(true);\n        body_reader.init_content_length(3, b\"\");\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(0, 1));\n        assert_eq!(body_reader.body_state, ParseState::Partial(1, 2));\n        assert_eq!(input1, body_reader.get_body(&res));\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(0, 2));\n        assert_eq!(body_reader.body_state, ParseState::Complete(3));\n        assert_eq!(&input2[0..2], body_reader.get_body(&res));\n        assert_eq!(body_reader.get_body_overread(), Some(&b\"d\"[..]));\n    }\n\n    #[tokio::test]\n    async fn read_with_body_content_length_rewind() {\n        init_log();\n        let rewind = b\"ab\";\n        let input = b\"c\";\n        let mut mock_io = Builder::new().read(&input[..]).build();\n        let mut body_reader = BodyReader::new(false);\n        body_reader.init_content_length(3, rewind);\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(0, 2));\n        assert_eq!(body_reader.body_state, ParseState::Partial(2, 1));\n        assert_eq!(rewind, body_reader.get_body(&res));\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(0, 1));\n        assert_eq!(body_reader.body_state, ParseState::Complete(3));\n        assert_eq!(input, body_reader.get_body(&res));\n        assert_eq!(body_reader.get_body_overread(), None);\n    }\n\n    #[tokio::test]\n    async fn read_with_body_http10() {\n        init_log();\n        let input1 = b\"a\";\n        let input2 = b\"\"; // simulating close\n        let mut mock_io = Builder::new().read(&input1[..]).read(&input2[..]).build();\n        let mut body_reader = BodyReader::new(false);\n        body_reader.init_close_delimited(b\"\");\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(0, 1));\n        assert_eq!(body_reader.body_state, ParseState::UntilClose(1));\n        assert_eq!(input1, body_reader.get_body(&res));\n        let res = body_reader.read_body(&mut mock_io).await.unwrap();\n        assert_eq!(res, None);\n        assert_eq!(body_reader.body_state, ParseState::Complete(1));\n        assert_eq!(body_reader.get_body_overread(), None);\n    }\n\n    #[tokio::test]\n    async fn read_with_body_http10_rewind() {\n        init_log();\n        let rewind = b\"ab\";\n        let input1 = b\"c\";\n        let input2 = b\"\"; // simulating close\n        let mut mock_io = Builder::new().read(&input1[..]).read(&input2[..]).build();\n        let mut body_reader = BodyReader::new(false);\n        body_reader.init_close_delimited(rewind);\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(0, 2));\n        assert_eq!(body_reader.body_state, ParseState::UntilClose(2));\n        assert_eq!(rewind, body_reader.get_body(&res));\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(0, 1));\n        assert_eq!(body_reader.body_state, ParseState::UntilClose(3));\n        assert_eq!(input1, body_reader.get_body(&res));\n        let res = body_reader.read_body(&mut mock_io).await.unwrap();\n        assert_eq!(res, None);\n        assert_eq!(body_reader.body_state, ParseState::Complete(3));\n        assert_eq!(body_reader.get_body_overread(), None);\n    }\n\n    #[tokio::test]\n    async fn read_with_body_zero_chunk() {\n        init_log();\n        let input = b\"0\\r\\n\\r\\n\";\n        let mut mock_io = Builder::new().read(&input[..]).build();\n        let mut body_reader = BodyReader::new(false);\n        body_reader.init_chunked(b\"\");\n        let res = body_reader.read_body(&mut mock_io).await.unwrap();\n        assert_eq!(res, None);\n        assert_eq!(body_reader.body_state, ParseState::Complete(0));\n        assert_eq!(body_reader.get_body_overread(), None);\n    }\n\n    #[tokio::test]\n    async fn read_with_body_zero_chunk_malformed() {\n        init_log();\n        let input = b\"0\\r\\nr\\n\";\n        let mut mock_io = Builder::new().read(&input[..]).build();\n        let mut body_reader = BodyReader::new(false);\n        body_reader.init_chunked(b\"\");\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(0, 0));\n        assert_eq!(body_reader.body_state, ParseState::ChunkedFinal(0, 0, 2, 2));\n\n        // \\n without leading \\r\n        let e = body_reader.read_body(&mut mock_io).await.unwrap_err();\n        assert_eq!(*e.etype(), INVALID_TRAILER_END);\n        assert_eq!(body_reader.body_state, ParseState::Done(0));\n        assert_eq!(body_reader.get_body_overread(), None);\n    }\n\n    #[tokio::test]\n    async fn read_with_body_zero_chunk_split() {\n        init_log();\n        let input1 = b\"0\\r\\n\";\n        let input2 = b\"\\r\\n\";\n        let mut mock_io = Builder::new().read(&input1[..]).read(&input2[..]).build();\n        let mut body_reader = BodyReader::new(false);\n        body_reader.init_chunked(b\"\");\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(0, 0));\n        assert_eq!(body_reader.body_state, ParseState::ChunkedFinal(0, 0, 0, 2));\n        let res = body_reader.read_body(&mut mock_io).await.unwrap();\n        assert_eq!(res, None);\n        assert_eq!(body_reader.body_state, ParseState::Complete(0));\n        assert_eq!(body_reader.get_body_overread(), None);\n    }\n\n    #[tokio::test]\n    async fn read_with_body_zero_chunk_split_head() {\n        init_log();\n        let input1 = b\"0\\r\";\n        let input2 = b\"\\n\";\n        let input3 = b\"\\r\\n\";\n        let mut mock_io = Builder::new()\n            .read(&input1[..])\n            .read(&input2[..])\n            .read(&input3[..])\n            .build();\n        let mut body_reader = BodyReader::new(false);\n        body_reader.init_chunked(b\"\");\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(0, 0));\n        assert_eq!(body_reader.body_state, ParseState::Chunked(0, 0, 2, 2));\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(0, 0));\n        assert_eq!(body_reader.body_state, ParseState::ChunkedFinal(0, 0, 0, 2));\n        let res = body_reader.read_body(&mut mock_io).await.unwrap();\n        assert_eq!(res, None);\n        assert_eq!(body_reader.body_state, ParseState::Complete(0));\n        assert_eq!(body_reader.get_body_overread(), None);\n    }\n\n    #[tokio::test]\n    async fn read_with_body_zero_chunk_split_head_2() {\n        init_log();\n        let input1 = b\"0\";\n        let input2 = b\"\\r\\n\";\n        let input3 = b\"\\r\\n\";\n        let mut mock_io = Builder::new()\n            .read(&input1[..])\n            .read(&input2[..])\n            .read(&input3[..])\n            .build();\n        let mut body_reader = BodyReader::new(false);\n        body_reader.init_chunked(b\"\");\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(0, 0));\n        assert_eq!(body_reader.body_state, ParseState::Chunked(0, 0, 1, 1));\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(0, 0));\n        assert_eq!(body_reader.body_state, ParseState::ChunkedFinal(0, 0, 0, 2));\n        let res = body_reader.read_body(&mut mock_io).await.unwrap();\n        assert_eq!(res, None);\n        assert_eq!(body_reader.body_state, ParseState::Complete(0));\n        assert_eq!(body_reader.get_body_overread(), None);\n    }\n\n    #[tokio::test]\n    async fn read_with_body_zero_chunk_split_head_3() {\n        init_log();\n        let input1 = b\"0\\r\";\n        let input2 = b\"\\n\";\n        let input3 = b\"\\r\";\n        let input4 = b\"\\n\";\n        let mut mock_io = Builder::new()\n            .read(&input1[..])\n            .read(&input2[..])\n            .read(&input3[..])\n            .read(&input4[..])\n            .build();\n        let mut body_reader = BodyReader::new(false);\n        body_reader.init_chunked(b\"\");\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(0, 0));\n        assert_eq!(body_reader.body_state, ParseState::Chunked(0, 0, 2, 2));\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(0, 0));\n        assert_eq!(body_reader.body_state, ParseState::ChunkedFinal(0, 0, 0, 2));\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(0, 0));\n        assert_eq!(body_reader.body_state, ParseState::ChunkedFinal(0, 0, 0, 3));\n\n        let res = body_reader.read_body(&mut mock_io).await.unwrap();\n        assert_eq!(res, None);\n        assert_eq!(body_reader.body_state, ParseState::Complete(0));\n        assert_eq!(body_reader.get_body_overread(), None);\n    }\n\n    #[tokio::test]\n    async fn read_with_body_chunk_ext() {\n        init_log();\n        let input = b\"0;aaaa\\r\\n\\r\\n\";\n        let mut mock_io = Builder::new().read(&input[..]).build();\n        let mut body_reader = BodyReader::new(false);\n        body_reader.init_chunked(b\"\");\n        let res = body_reader.read_body(&mut mock_io).await.unwrap();\n        assert_eq!(res, None);\n        assert_eq!(body_reader.body_state, ParseState::Complete(0));\n        assert_eq!(body_reader.get_body_overread(), None);\n    }\n\n    #[tokio::test]\n    async fn read_with_body_chunk_ext_oversize() {\n        init_log();\n        let chunk_size = b\"0;\";\n        let ext1 = [b'a'; 1024 * 5];\n        let ext2 = [b'a'; 1024 * 3];\n        let mut mock_io = Builder::new()\n            .read(&chunk_size[..])\n            .read(&ext1[..])\n            .read(&ext2[..])\n            .build();\n        let mut body_reader = BodyReader::new(false);\n        body_reader.init_chunked(b\"\");\n        // read chunk-size, chunk incomplete\n        let res = body_reader.read_body(&mut mock_io).await.unwrap();\n        assert_eq!(res, Some(BufRef::new(0, 0)));\n        // read ext1, chunk incomplete\n        let res = body_reader.read_body(&mut mock_io).await.unwrap();\n        assert_eq!(res, Some(BufRef::new(0, 0)));\n        // read ext2, now oversized\n        let res = body_reader.read_body(&mut mock_io).await;\n        assert!(res.is_err());\n        assert_eq!(body_reader.body_state, ParseState::Done(0));\n        assert_eq!(body_reader.get_body_overread(), None);\n    }\n\n    #[tokio::test]\n    async fn read_with_body_1_chunk() {\n        init_log();\n        let input1 = b\"1\\r\\na\\r\\n\";\n        let input2 = b\"0\\r\\n\\r\\n\";\n        let mut mock_io = Builder::new().read(&input1[..]).read(&input2[..]).build();\n        let mut body_reader = BodyReader::new(false);\n        body_reader.init_chunked(b\"\");\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(3, 1));\n        assert_eq!(&input1[3..4], body_reader.get_body(&res));\n        assert_eq!(body_reader.body_state, ParseState::Chunked(1, 0, 0, 0));\n        let res = body_reader.read_body(&mut mock_io).await.unwrap();\n        assert_eq!(res, None);\n        assert_eq!(body_reader.body_state, ParseState::Complete(1));\n        assert_eq!(body_reader.get_body_overread(), None);\n    }\n\n    #[tokio::test]\n    async fn read_with_body_1_chunk_malformed() {\n        init_log();\n        let input1 = b\"1\\r\\na\\rn\";\n        let mut mock_io = Builder::new().read(&input1[..]).build();\n        let mut body_reader = BodyReader::new(false);\n        body_reader.init_chunked(b\"\");\n\n        let e = body_reader.read_body(&mut mock_io).await.unwrap_err();\n        assert_eq!(*e.etype(), INVALID_CHUNK);\n        assert_eq!(body_reader.body_state, ParseState::Done(0));\n\n        assert_eq!(body_reader.get_body_overread(), None);\n    }\n\n    #[tokio::test]\n    async fn read_with_body_1_chunk_partial_end() {\n        init_log();\n        let input1 = b\"1\\r\\na\\r\";\n        let input2 = b\"\\n0\\r\\n\\r\\n\";\n        let mut mock_io = Builder::new().read(&input1[..]).read(&input2[..]).build();\n        let mut body_reader = BodyReader::new(false);\n        body_reader.init_chunked(b\"\");\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(3, 1));\n        assert_eq!(&input1[3..4], body_reader.get_body(&res));\n        assert_eq!(body_reader.body_state, ParseState::Chunked(1, 0, 0, 1));\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(0, 0));\n        assert_eq!(body_reader.body_state, ParseState::Chunked(1, 1, 6, 0));\n        let res = body_reader.read_body(&mut mock_io).await.unwrap();\n        assert_eq!(res, None);\n        assert_eq!(body_reader.body_state, ParseState::Complete(1));\n        assert_eq!(body_reader.get_body_overread(), None);\n    }\n\n    #[tokio::test]\n    async fn read_with_body_1_chunk_partial_end_1() {\n        init_log();\n        let input1 = b\"3\\r\\n\";\n        let input2 = b\"abc\\r\";\n        let input3 = b\"\\n0\\r\\n\\r\\n\";\n        let mut mock_io = Builder::new()\n            .read(&input1[..])\n            .read(&input2[..])\n            .read(&input3[..])\n            .build();\n        let mut body_reader = BodyReader::new(false);\n        body_reader.init_chunked(b\"\");\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(3, 0));\n        assert_eq!(b\"\", body_reader.get_body(&res));\n        assert_eq!(body_reader.body_state, ParseState::Chunked(0, 0, 0, 5));\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(0, 3));\n        assert_eq!(&input2[0..3], body_reader.get_body(&res));\n        assert_eq!(body_reader.body_state, ParseState::Chunked(3, 0, 0, 1));\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(0, 0));\n        let res = body_reader.read_body(&mut mock_io).await.unwrap();\n        assert_eq!(res, None);\n        assert_eq!(body_reader.body_state, ParseState::Complete(3));\n        assert_eq!(body_reader.get_body_overread(), None);\n    }\n\n    #[tokio::test]\n    async fn read_with_body_1_chunk_partial_end_2() {\n        init_log();\n        let input1 = b\"3\\r\\n\";\n        let input2 = b\"abc\";\n        let input3 = b\"\\r\\n0\\r\\n\\r\\n\";\n        let mut mock_io = Builder::new()\n            .read(&input1[..])\n            .read(&input2[..])\n            .read(&input3[..])\n            .build();\n        let mut body_reader = BodyReader::new(false);\n        body_reader.init_chunked(b\"\");\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(3, 0));\n        assert_eq!(b\"\", body_reader.get_body(&res));\n        assert_eq!(body_reader.body_state, ParseState::Chunked(0, 0, 0, 5));\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(0, 3));\n        assert_eq!(&input2[0..3], body_reader.get_body(&res));\n        assert_eq!(body_reader.body_state, ParseState::Chunked(3, 0, 0, 2));\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(0, 0));\n        let res = body_reader.read_body(&mut mock_io).await.unwrap();\n        assert_eq!(res, None);\n        assert_eq!(body_reader.body_state, ParseState::Complete(3));\n        assert_eq!(body_reader.get_body_overread(), None);\n    }\n\n    #[tokio::test]\n    async fn read_with_body_1_chunk_incomplete() {\n        init_log();\n        let input1 = b\"1\\r\\na\\r\\n\";\n        let mut mock_io = Builder::new().read(&input1[..]).build();\n        let mut body_reader = BodyReader::new(false);\n        body_reader.init_chunked(b\"\");\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(3, 1));\n        assert_eq!(&input1[3..4], body_reader.get_body(&res));\n        assert_eq!(body_reader.body_state, ParseState::Chunked(1, 0, 0, 0));\n        let res = body_reader.read_body(&mut mock_io).await;\n        assert!(res.is_err());\n        assert_eq!(body_reader.body_state, ParseState::Done(1));\n    }\n\n    #[tokio::test]\n    async fn read_with_body_1_chunk_partial_end_malformed() {\n        init_log();\n        let input1 = b\"1\\r\\na\\r\";\n        let input2 = b\"n0\\r\\n\\r\\n\";\n        let mut mock_io = Builder::new().read(&input1[..]).read(&input2[..]).build();\n        let mut body_reader = BodyReader::new(false);\n        body_reader.init_chunked(b\"\");\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(3, 1));\n        assert_eq!(&input1[3..4], body_reader.get_body(&res));\n        assert_eq!(body_reader.body_state, ParseState::Chunked(1, 0, 0, 1));\n        let e = body_reader.read_body(&mut mock_io).await.unwrap_err();\n        assert_eq!(*e.etype(), INVALID_CHUNK);\n        assert_eq!(body_reader.body_state, ParseState::Done(1));\n        assert_eq!(body_reader.get_body_overread(), None);\n    }\n\n    #[tokio::test]\n    async fn read_with_body_1_chunk_rewind() {\n        init_log();\n        let rewind = b\"1\\r\\nx\\r\\n\";\n        let input1 = b\"1\\r\\na\\r\\n\";\n        let input2 = b\"0\\r\\n\\r\\n\";\n        let mut mock_io = Builder::new().read(&input1[..]).read(&input2[..]).build();\n        let mut body_reader = BodyReader::new(false);\n        body_reader.init_chunked(rewind);\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(3, 1));\n        assert_eq!(&rewind[3..4], body_reader.get_body(&res));\n        assert_eq!(body_reader.body_state, ParseState::Chunked(1, 0, 0, 0));\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(3, 1));\n        assert_eq!(&input1[3..4], body_reader.get_body(&res));\n        assert_eq!(body_reader.body_state, ParseState::Chunked(2, 0, 0, 0));\n        let res = body_reader.read_body(&mut mock_io).await.unwrap();\n        assert_eq!(res, None);\n        assert_eq!(body_reader.body_state, ParseState::Complete(2));\n        assert_eq!(body_reader.get_body_overread(), None);\n    }\n\n    #[tokio::test]\n    async fn read_with_body_multi_chunk() {\n        init_log();\n        let input1 = b\"1\\r\\na\\r\\n2\\r\\nbc\\r\\n\";\n        let input2 = b\"0\\r\\n\\r\\n\";\n        let mut mock_io = Builder::new().read(&input1[..]).read(&input2[..]).build();\n        let mut body_reader = BodyReader::new(false);\n        body_reader.init_chunked(b\"\");\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(3, 1));\n        assert_eq!(&input1[3..4], body_reader.get_body(&res));\n        assert_eq!(body_reader.body_state, ParseState::Chunked(1, 6, 13, 0));\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(9, 2));\n        assert_eq!(&input1[9..11], body_reader.get_body(&res));\n        assert_eq!(body_reader.body_state, ParseState::Chunked(3, 0, 0, 0));\n        let res = body_reader.read_body(&mut mock_io).await.unwrap();\n        assert_eq!(res, None);\n        assert_eq!(body_reader.body_state, ParseState::Complete(3));\n        assert_eq!(body_reader.get_body_overread(), None);\n    }\n\n    #[tokio::test]\n    async fn read_with_body_multi_chunk_malformed() {\n        init_log();\n        let input1 = b\"1\\r\\na\\r\\n2\\r\\nbcr\\n\";\n        let mut mock_io = Builder::new().read(&input1[..]).build();\n        let mut body_reader = BodyReader::new(false);\n        body_reader.init_chunked(b\"\");\n\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(3, 1));\n        assert_eq!(&input1[3..4], body_reader.get_body(&res));\n        assert_eq!(body_reader.body_state, ParseState::Chunked(1, 6, 13, 0));\n\n        let e = body_reader.read_body(&mut mock_io).await.unwrap_err();\n        assert_eq!(*e.etype(), INVALID_CHUNK);\n        assert_eq!(body_reader.body_state, ParseState::Done(1));\n        assert_eq!(body_reader.get_body_overread(), None);\n\n        let input1 = b\"1\\r\\nar\\n2\\r\\nbc\\rn\";\n        let mut mock_io = Builder::new().read(&input1[..]).build();\n        let mut body_reader = BodyReader::new(false);\n        body_reader.init_chunked(b\"\");\n\n        let e = body_reader.read_body(&mut mock_io).await.unwrap_err();\n        assert_eq!(*e.etype(), INVALID_CHUNK);\n        assert_eq!(body_reader.body_state, ParseState::Done(0));\n        assert_eq!(body_reader.get_body_overread(), None);\n    }\n\n    #[tokio::test]\n    async fn read_with_body_partial_chunk() {\n        init_log();\n        let input1 = b\"3\\r\\na\";\n        let input2 = b\"bc\\r\\n0\\r\\n\\r\\n\";\n        let mut mock_io = Builder::new().read(&input1[..]).read(&input2[..]).build();\n        let mut body_reader = BodyReader::new(false);\n        body_reader.init_chunked(b\"\");\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(3, 1));\n        assert_eq!(&input1[3..4], body_reader.get_body(&res));\n        assert_eq!(body_reader.body_state, ParseState::Chunked(1, 0, 0, 4));\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(0, 2));\n        assert_eq!(&input2[0..2], body_reader.get_body(&res));\n        assert_eq!(body_reader.body_state, ParseState::Chunked(3, 4, 9, 0));\n        let res = body_reader.read_body(&mut mock_io).await.unwrap();\n        assert_eq!(res, None);\n        assert_eq!(body_reader.body_state, ParseState::Complete(3));\n        assert_eq!(body_reader.get_body_overread(), None);\n    }\n\n    #[tokio::test]\n    async fn read_with_body_partial_chunk_end() {\n        init_log();\n        let input1 = b\"3\\r\\nabc\";\n        let input2 = b\"\\r\\n0\\r\\n\\r\\n\";\n        let mut mock_io = Builder::new().read(&input1[..]).read(&input2[..]).build();\n        let mut body_reader = BodyReader::new(false);\n        body_reader.init_chunked(b\"\");\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(3, 3));\n        assert_eq!(&input1[3..6], body_reader.get_body(&res));\n        // \\r\\n (2 bytes) left to read from IO\n        assert_eq!(body_reader.body_state, ParseState::Chunked(3, 0, 0, 2));\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(0, 0));\n        assert_eq!(&input2[0..0], body_reader.get_body(&res));\n        assert_eq!(body_reader.body_state, ParseState::Chunked(3, 2, 7, 0));\n        let res = body_reader.read_body(&mut mock_io).await.unwrap();\n        assert_eq!(res, None);\n        assert_eq!(body_reader.body_state, ParseState::Complete(3));\n        assert_eq!(body_reader.get_body_overread(), None);\n    }\n\n    #[tokio::test]\n    async fn read_with_body_partial_head_chunk() {\n        init_log();\n        let input1 = b\"1\\r\";\n        let input2 = b\"\\na\\r\\n0\\r\\n\\r\\n\";\n        let mut mock_io = Builder::new().read(&input1[..]).read(&input2[..]).build();\n        let mut body_reader = BodyReader::new(false);\n        body_reader.init_chunked(b\"\");\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(0, 0));\n        assert_eq!(body_reader.body_state, ParseState::Chunked(0, 0, 2, 2));\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(3, 1)); // input1 concat input2\n        assert_eq!(&input2[1..2], body_reader.get_body(&res));\n        assert_eq!(body_reader.body_state, ParseState::Chunked(1, 6, 11, 0));\n        let res = body_reader.read_body(&mut mock_io).await.unwrap();\n        assert_eq!(res, None);\n        assert_eq!(body_reader.body_state, ParseState::Complete(1));\n        assert_eq!(body_reader.get_body_overread(), None);\n    }\n\n    #[tokio::test]\n    async fn read_with_body_partial_head_terminal_crlf() {\n        init_log();\n        let input1 = b\"1\\r\";\n        let input2 = b\"\\na\\r\\n0\\r\\n\\r\";\n        let input3 = b\"\\n\";\n        let mut mock_io = Builder::new()\n            .read(&input1[..])\n            .read(&input2[..])\n            .read(&input3[..])\n            .build();\n        let mut body_reader = BodyReader::new(false);\n        body_reader.init_chunked(b\"\");\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(0, 0));\n        assert_eq!(body_reader.body_state, ParseState::Chunked(0, 0, 2, 2));\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(3, 1)); // input1 concat input2\n        assert_eq!(&input2[1..2], body_reader.get_body(&res));\n        assert_eq!(body_reader.body_state, ParseState::Chunked(1, 6, 10, 0));\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(0, 0)); // only part of terminal crlf, one more byte to read\n        assert_eq!(body_reader.body_state, ParseState::ChunkedFinal(1, 0, 1, 2));\n        // TODO: can optimize this to avoid the second read_body call\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(0, 0));\n        assert_eq!(body_reader.body_state, ParseState::ChunkedFinal(1, 0, 0, 3));\n\n        let res = body_reader.read_body(&mut mock_io).await.unwrap();\n        assert_eq!(res, None);\n        assert_eq!(body_reader.body_state, ParseState::Complete(1));\n        assert_eq!(body_reader.get_body_overread(), None);\n    }\n\n    #[tokio::test]\n    async fn read_with_body_partial_head_terminal_crlf_2() {\n        init_log();\n        let input1 = b\"1\\r\";\n        let input2 = b\"\\na\\r\\n0\\r\";\n        let input3 = b\"\\n\\r\\n\";\n        let mut mock_io = Builder::new()\n            .read(&input1[..])\n            .read(&input2[..])\n            .read(&input3[..])\n            .build();\n        let mut body_reader = BodyReader::new(false);\n        body_reader.init_chunked(b\"\");\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(0, 0));\n        assert_eq!(body_reader.body_state, ParseState::Chunked(0, 0, 2, 2));\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(3, 1)); // input1 concat input2\n        assert_eq!(&input2[1..2], body_reader.get_body(&res));\n        assert_eq!(body_reader.body_state, ParseState::Chunked(1, 6, 8, 0));\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(0, 0)); // only part of terminal crlf, one more byte to read\n        assert_eq!(body_reader.body_state, ParseState::Chunked(1, 0, 8, 2));\n        // optimized to go right to complete state\n        let res = body_reader.read_body(&mut mock_io).await.unwrap();\n        assert_eq!(res, None);\n        assert_eq!(body_reader.body_state, ParseState::Complete(1));\n        assert_eq!(body_reader.get_body_overread(), None);\n    }\n\n    #[tokio::test]\n    async fn read_with_body_partial_head_terminal_crlf_3() {\n        init_log();\n        let input1 = b\"1\\r\\na\\r\\n0\";\n        let input2 = b\"\\r\";\n        let input3 = b\"\\n\";\n        let input4 = b\"\\r\";\n        let input5 = b\"\\n\";\n        let mut mock_io = Builder::new()\n            .read(&input1[..])\n            .read(&input2[..])\n            .read(&input3[..])\n            .read(&input4[..])\n            .read(&input5[..])\n            .build();\n\n        let mut body_reader = BodyReader::new(false);\n        body_reader.init_chunked(b\"\");\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(3, 1));\n        assert_eq!(&input1[3..4], body_reader.get_body(&res));\n        assert_eq!(body_reader.body_state, ParseState::Chunked(1, 6, 7, 0));\n        // to 0\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(0, 0));\n        assert_eq!(body_reader.body_state, ParseState::Chunked(1, 0, 7, 1));\n        // \\r\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(0, 0));\n        assert_eq!(body_reader.body_state, ParseState::Chunked(1, 0, 2, 2));\n        // \\n\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(0, 0));\n        assert_eq!(body_reader.body_state, ParseState::ChunkedFinal(1, 0, 0, 2));\n        // \\r\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(0, 0));\n        assert_eq!(body_reader.body_state, ParseState::ChunkedFinal(1, 0, 0, 3));\n        // \\n\n        let res = body_reader.read_body(&mut mock_io).await.unwrap();\n        assert_eq!(res, None);\n        assert_eq!(body_reader.body_state, ParseState::Complete(1));\n        assert_eq!(body_reader.get_body_overread(), None);\n    }\n\n    #[tokio::test]\n    async fn read_with_body_partial_head_terminal_crlf_malformed() {\n        init_log();\n        let input1 = b\"1\\r\";\n        let input2 = b\"\\na\\r\\n0\\r\\nr\";\n        let mut mock_io = Builder::new().read(&input1[..]).read(&input2[..]).build();\n        let mut body_reader = BodyReader::new(false);\n        body_reader.init_chunked(b\"\");\n\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(0, 0));\n        assert_eq!(body_reader.body_state, ParseState::Chunked(0, 0, 2, 2));\n\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(3, 1)); // input1 concat input2\n        assert_eq!(&input2[1..2], body_reader.get_body(&res));\n        assert_eq!(body_reader.body_state, ParseState::Chunked(1, 6, 10, 0));\n\n        // TODO: may be able to optimize this extra read_body out\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(0, 0));\n        assert_eq!(body_reader.body_state, ParseState::ChunkedFinal(1, 0, 1, 2));\n        // \"r\" is interpreted as a hanging trailer\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(0, 0));\n        assert_eq!(body_reader.body_state, ParseState::ChunkedFinal(1, 3, 0, 0));\n\n        let res = body_reader.read_body(&mut mock_io).await.unwrap_err();\n        assert_eq!(&ConnectionClosed, res.etype());\n        assert_eq!(body_reader.body_state, ParseState::Done(1));\n        assert_eq!(body_reader.get_body_overread(), None);\n    }\n\n    #[tokio::test]\n    async fn read_with_body_partial_head_terminal_crlf_overread() {\n        init_log();\n        let input1 = b\"1\\r\";\n        let input2 = b\"\\na\\r\\n0\\r\\n\\r\";\n        let input3 = b\"\\nabcd\";\n        let mut mock_io = Builder::new()\n            .read(&input1[..])\n            .read(&input2[..])\n            .read(&input3[..])\n            .build();\n        let mut body_reader = BodyReader::new(false);\n        body_reader.init_chunked(b\"\");\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(0, 0));\n        assert_eq!(body_reader.body_state, ParseState::Chunked(0, 0, 2, 2));\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(3, 1)); // input1 concat input2\n        assert_eq!(&input2[1..2], body_reader.get_body(&res));\n        assert_eq!(body_reader.body_state, ParseState::Chunked(1, 6, 10, 0));\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(0, 0)); // read only part of terminal crlf\n        assert_eq!(body_reader.body_state, ParseState::ChunkedFinal(1, 0, 1, 2));\n        // TODO: can optimize this to avoid the second read_body call\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(0, 0));\n        assert_eq!(body_reader.body_state, ParseState::ChunkedFinal(1, 0, 0, 3));\n\n        let res = body_reader.read_body(&mut mock_io).await.unwrap();\n        assert_eq!(res, None);\n        assert_eq!(body_reader.body_state, ParseState::Complete(1));\n        assert_eq!(body_reader.get_body_overread(), Some(&b\"abcd\"[..]));\n    }\n\n    #[tokio::test]\n    async fn read_with_body_multi_chunk_overread() {\n        init_log();\n        let input1 = b\"1\\r\\na\\r\\n2\\r\\nbc\\r\\n\";\n        let input2 = b\"0\\r\\n\\r\\nabc\";\n        let mut mock_io = Builder::new().read(&input1[..]).read(&input2[..]).build();\n        let mut body_reader = BodyReader::new(false);\n        body_reader.init_chunked(b\"\");\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(3, 1));\n        assert_eq!(&input1[3..4], body_reader.get_body(&res));\n        assert_eq!(body_reader.body_state, ParseState::Chunked(1, 6, 13, 0));\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(9, 2));\n        assert_eq!(&input1[9..11], body_reader.get_body(&res));\n        assert_eq!(body_reader.body_state, ParseState::Chunked(3, 0, 0, 0));\n        let res = body_reader.read_body(&mut mock_io).await.unwrap();\n        assert_eq!(res, None);\n        assert_eq!(body_reader.body_state, ParseState::Complete(3));\n        assert_eq!(body_reader.get_body_overread(), Some(&b\"abc\"[..]));\n    }\n\n    #[tokio::test]\n    async fn read_with_body_partial_head_chunk_incomplete() {\n        init_log();\n        let input1 = b\"1\\r\";\n        let mut mock_io = Builder::new().read(&input1[..]).build();\n        let mut body_reader = BodyReader::new(false);\n        body_reader.init_chunked(b\"\");\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(0, 0));\n        assert_eq!(body_reader.body_state, ParseState::Chunked(0, 0, 2, 2));\n        let res = body_reader.read_body(&mut mock_io).await;\n        assert!(res.is_err());\n        assert_eq!(body_reader.body_state, ParseState::Done(0));\n    }\n\n    #[tokio::test]\n    async fn read_with_body_trailers() {\n        init_log();\n        let input1 = b\"1\\r\\na\\r\\n2\\r\\nbc\\r\\n\";\n        let input2 = b\"0\\r\\nabc: hi\";\n        let input3 = b\"\\r\\ndef: bye\\r\";\n        let input4 = b\"\\nghi: more\\r\\n\";\n        let input5 = b\"\\r\\n\";\n        let mut mock_io = Builder::new()\n            .read(&input1[..])\n            .read(&input2[..])\n            .read(&input3[..])\n            .read(&input4[..])\n            .read(&input5[..])\n            .build();\n        let mut body_reader = BodyReader::new(false);\n        body_reader.init_chunked(b\"\");\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(3, 1));\n        assert_eq!(&input1[3..4], body_reader.get_body(&res));\n        assert_eq!(body_reader.body_state, ParseState::Chunked(1, 6, 13, 0));\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(9, 2));\n        assert_eq!(&input1[9..11], body_reader.get_body(&res));\n        assert_eq!(body_reader.body_state, ParseState::Chunked(3, 0, 0, 0));\n        // abc: hi\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(0, 0));\n        assert_eq!(body_reader.body_state, ParseState::ChunkedFinal(3, 0, 7, 2));\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(0, 0));\n        assert_eq!(\n            body_reader.body_state,\n            // NOTE: 0 chunk-size CRLF counted in trailer size too\n            ParseState::ChunkedFinal(3, 9, 0, 0)\n        );\n        // def: bye\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(0, 0));\n        assert_eq!(\n            body_reader.body_state,\n            ParseState::ChunkedFinal(3, 19, 0, 1)\n        );\n        // ghi: more\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(0, 0));\n        assert_eq!(\n            body_reader.body_state,\n            ParseState::ChunkedFinal(3, 30, 0, 2)\n        );\n\n        let res = body_reader.read_body(&mut mock_io).await.unwrap();\n        assert_eq!(res, None);\n        assert_eq!(body_reader.body_state, ParseState::Complete(3));\n        assert_eq!(body_reader.get_body_overread(), None);\n    }\n\n    #[tokio::test]\n    async fn read_with_body_trailers_2() {\n        init_log();\n        let input1 = b\"1\\r\\na\\r\\n0\\r\";\n        let input2 = b\"\\nabc: hi\\r\\n\\r\\n\";\n        let mut mock_io = Builder::new().read(&input1[..]).read(&input2[..]).build();\n        let mut body_reader = BodyReader::new(false);\n        body_reader.init_chunked(b\"\");\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(3, 1));\n        assert_eq!(&input1[3..4], body_reader.get_body(&res));\n        assert_eq!(body_reader.body_state, ParseState::Chunked(1, 6, 8, 0));\n        // 0 \\r\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(0, 0));\n        assert_eq!(body_reader.body_state, ParseState::Chunked(1, 0, 8, 2));\n        // \\n TODO: optimize this call out\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(0, 0));\n        assert_eq!(\n            body_reader.body_state,\n            ParseState::ChunkedFinal(1, 0, 11, 2)\n        );\n        // abc: hi with end in same read\n        let res = body_reader.read_body(&mut mock_io).await.unwrap();\n        assert_eq!(res, None);\n        assert_eq!(body_reader.body_state, ParseState::Complete(1));\n        assert_eq!(body_reader.get_body_overread(), None);\n    }\n\n    #[tokio::test]\n    async fn read_with_body_trailers_3() {\n        init_log();\n        let input1 = b\"1\\r\\na\\r\\n0\\r\";\n        let input2 = b\"\\nabc: hi\";\n        let input3 = b\"\\r\\n\\r\\n\";\n        let mut mock_io = Builder::new()\n            .read(&input1[..])\n            .read(&input2[..])\n            .read(&input3[..])\n            .build();\n        let mut body_reader = BodyReader::new(false);\n        body_reader.init_chunked(b\"\");\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(3, 1));\n        assert_eq!(&input1[3..4], body_reader.get_body(&res));\n        assert_eq!(body_reader.body_state, ParseState::Chunked(1, 6, 8, 0));\n        // 0 \\r\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(0, 0));\n        assert_eq!(body_reader.body_state, ParseState::Chunked(1, 0, 8, 2));\n        // \\n TODO: optimize this call out\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(0, 0));\n        assert_eq!(body_reader.body_state, ParseState::ChunkedFinal(1, 0, 7, 2));\n        // abc: hi\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(0, 0));\n        assert_eq!(\n            body_reader.body_state,\n            // NOTE: 0 chunk-size CRLF counted in trailer size too\n            ParseState::ChunkedFinal(1, 9, 0, 0)\n        );\n        let res = body_reader.read_body(&mut mock_io).await.unwrap();\n        assert_eq!(res, None);\n        assert_eq!(body_reader.body_state, ParseState::Complete(1));\n        assert_eq!(body_reader.get_body_overread(), None);\n    }\n\n    #[tokio::test]\n    async fn read_with_body_trailers_4() {\n        init_log();\n        let input1 = b\"1\\r\\na\\r\\n0\\r\";\n        let input2 = b\"\\nabc: hi\\r\\n\\r\";\n        let input3 = b\"\\n\";\n        let mut mock_io = Builder::new()\n            .read(&input1[..])\n            .read(&input2[..])\n            .read(&input3[..])\n            .build();\n        let mut body_reader = BodyReader::new(false);\n        body_reader.init_chunked(b\"\");\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(3, 1));\n        assert_eq!(&input1[3..4], body_reader.get_body(&res));\n        assert_eq!(body_reader.body_state, ParseState::Chunked(1, 6, 8, 0));\n        // 0 \\r\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(0, 0));\n        assert_eq!(body_reader.body_state, ParseState::Chunked(1, 0, 8, 2));\n        // \\n TODO: optimize this call out\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(0, 0));\n        assert_eq!(\n            body_reader.body_state,\n            ParseState::ChunkedFinal(1, 0, 10, 2)\n        );\n        // abc: hi\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(0, 0));\n        assert_eq!(\n            body_reader.body_state,\n            // NOTE: 0 chunk-size CRLF counted in trailer size too\n            ParseState::ChunkedFinal(1, 9, 0, 3)\n        );\n        let res = body_reader.read_body(&mut mock_io).await.unwrap();\n        assert_eq!(res, None);\n        assert_eq!(body_reader.body_state, ParseState::Complete(1));\n        assert_eq!(body_reader.get_body_overread(), None);\n    }\n\n    #[tokio::test]\n    async fn read_with_body_trailers_malformed() {\n        init_log();\n        let input1 = b\"1\\r\\na\\r\\n0\\r\";\n        let input2 = b\"\\nabc: hi\\rn\";\n        let mut mock_io = Builder::new().read(&input1[..]).read(&input2[..]).build();\n        let mut body_reader = BodyReader::new(false);\n        body_reader.init_chunked(b\"\");\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(3, 1));\n        assert_eq!(&input1[3..4], body_reader.get_body(&res));\n        assert_eq!(body_reader.body_state, ParseState::Chunked(1, 6, 8, 0));\n        // 0 \\r\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(0, 0));\n        assert_eq!(body_reader.body_state, ParseState::Chunked(1, 0, 8, 2));\n        // abc: hi to \\rn\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(0, 0));\n        assert_eq!(body_reader.body_state, ParseState::ChunkedFinal(1, 0, 9, 2));\n        // \\rn not valid\n        let e = body_reader.read_body(&mut mock_io).await.unwrap_err();\n        assert_eq!(*e.etype(), INVALID_TRAILER_END);\n        assert_eq!(body_reader.body_state, ParseState::Done(1));\n        assert_eq!(body_reader.get_body_overread(), None);\n    }\n\n    #[tokio::test]\n    async fn read_with_body_trailers_malformed_2() {\n        init_log();\n        let input1 = b\"1\\r\\na\\r\\n0\\r\";\n        let input2 = b\"\\nabc: hi\\r\\n\";\n        // no end\n        let mut mock_io = Builder::new().read(&input1[..]).read(&input2[..]).build();\n        let mut body_reader = BodyReader::new(false);\n        body_reader.init_chunked(b\"\");\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(3, 1));\n        assert_eq!(&input1[3..4], body_reader.get_body(&res));\n        assert_eq!(body_reader.body_state, ParseState::Chunked(1, 6, 8, 0));\n        // 0 \\r\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(0, 0));\n        assert_eq!(body_reader.body_state, ParseState::Chunked(1, 0, 8, 2));\n        // abc: hi to \\r\\n\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(0, 0));\n        assert_eq!(body_reader.body_state, ParseState::ChunkedFinal(1, 0, 9, 2));\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(0, 0));\n        assert_eq!(body_reader.body_state, ParseState::ChunkedFinal(1, 9, 0, 2));\n        // EOF\n        let e = body_reader.read_body(&mut mock_io).await.unwrap_err();\n        assert_eq!(*e.etype(), ConnectionClosed);\n        assert_eq!(body_reader.body_state, ParseState::Done(1));\n        assert_eq!(body_reader.get_body_overread(), None);\n    }\n\n    #[tokio::test]\n    async fn read_with_body_trailers_malformed_3() {\n        init_log();\n        let input1 = b\"1\\r\\na\\r\\n0\\r\\n\";\n        let input2 = b\"abc: hi\\r\\n\";\n        let input3 = b\"r\\n\";\n        let mut mock_io = Builder::new()\n            .read(&input1[..])\n            .read(&input2[..])\n            .read(&input3[..])\n            .build();\n        let mut body_reader = BodyReader::new(false);\n        body_reader.init_chunked(b\"\");\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(3, 1));\n        assert_eq!(&input1[3..4], body_reader.get_body(&res));\n        assert_eq!(body_reader.body_state, ParseState::Chunked(1, 6, 9, 0));\n        // 0 \\r\\n\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(0, 0));\n        assert_eq!(body_reader.body_state, ParseState::ChunkedFinal(1, 0, 0, 2));\n        // abc: hi\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(0, 0));\n        assert_eq!(body_reader.body_state, ParseState::ChunkedFinal(1, 9, 0, 2));\n        // r\\n not valid\n        let e = body_reader.read_body(&mut mock_io).await.unwrap_err();\n        assert_eq!(*e.etype(), INVALID_TRAILER_END);\n        assert_eq!(body_reader.body_state, ParseState::Done(1));\n        assert_eq!(body_reader.get_body_overread(), None);\n    }\n\n    #[tokio::test]\n    async fn read_with_body_trailers_overflow() {\n        init_log();\n        let input1 = b\"1\\r\\na\\r\\n0\\r\\n\";\n        let input2 = b\"abc: \";\n        let trailer1 = [b'a'; 1024 * 60];\n        let trailer2 = [b'a'; 1024 * 5];\n        let input3 = b\"defghi: \";\n        let mut mock_io = Builder::new()\n            .read(&input1[..])\n            .read(&input2[..])\n            .read(&trailer1[..])\n            .read(&CRLF[..])\n            .read(&input3[..])\n            .read(&trailer2[..])\n            .build();\n        let mut body_reader = BodyReader::new(false);\n        body_reader.init_chunked(b\"\");\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(3, 1));\n        assert_eq!(&input1[3..4], body_reader.get_body(&res));\n        assert_eq!(body_reader.body_state, ParseState::Chunked(1, 6, 9, 0));\n        // 0 \\r\\n\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(0, 0));\n        assert_eq!(body_reader.body_state, ParseState::ChunkedFinal(1, 0, 0, 2));\n        // abc:\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(0, 0));\n        assert_eq!(body_reader.body_state, ParseState::ChunkedFinal(1, 7, 0, 0));\n        // aaa...\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(0, 0));\n        assert_eq!(\n            body_reader.body_state,\n            ParseState::ChunkedFinal(1, 1024 * 60 + 7, 0, 0)\n        );\n        // CRLF\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(0, 0));\n        assert_eq!(\n            body_reader.body_state,\n            ParseState::ChunkedFinal(1, 1024 * 60 + 7, 0, 2)\n        );\n        // defghi:\n        let res = body_reader.read_body(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(0, 0));\n        assert_eq!(\n            body_reader.body_state,\n            ParseState::ChunkedFinal(1, 1024 * 60 + 17, 0, 0)\n        );\n        // overflow\n        let e = body_reader.read_body(&mut mock_io).await.unwrap_err();\n        assert_eq!(*e.etype(), INVALID_TRAILER_END);\n        assert_eq!(body_reader.body_state, ParseState::Done(1));\n        assert_eq!(body_reader.get_body_overread(), None);\n    }\n\n    #[tokio::test]\n    async fn write_body_cl() {\n        init_log();\n        let output = b\"a\";\n        let mut mock_io = Builder::new().write(&output[..]).build();\n        let mut body_writer = BodyWriter::new();\n        body_writer.init_content_length(1);\n        assert_eq!(body_writer.body_mode, BodyMode::ContentLength(1, 0));\n        let res = body_writer\n            .write_body(&mut mock_io, &output[..])\n            .await\n            .unwrap()\n            .unwrap();\n        assert_eq!(res, 1);\n        assert_eq!(body_writer.body_mode, BodyMode::ContentLength(1, 1));\n        // write again, over the limit\n        let res = body_writer\n            .write_body(&mut mock_io, &output[..])\n            .await\n            .unwrap();\n        assert_eq!(res, None);\n        assert_eq!(body_writer.body_mode, BodyMode::ContentLength(1, 1));\n        let res = body_writer.finish(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, 1);\n        assert_eq!(body_writer.body_mode, BodyMode::Complete(1));\n    }\n\n    #[tokio::test]\n    async fn write_body_chunked() {\n        init_log();\n        let data = b\"abcdefghij\";\n        let output = b\"A\\r\\nabcdefghij\\r\\n\";\n        let mut mock_io = Builder::new()\n            .write(&output[..])\n            .write(&output[..])\n            .write(&LAST_CHUNK[..])\n            .build();\n        let mut body_writer = BodyWriter::new();\n        body_writer.init_chunked();\n        assert_eq!(body_writer.body_mode, BodyMode::ChunkedEncoding(0));\n        let res = body_writer\n            .write_body(&mut mock_io, &data[..])\n            .await\n            .unwrap()\n            .unwrap();\n        assert_eq!(res, data.len());\n        assert_eq!(body_writer.body_mode, BodyMode::ChunkedEncoding(data.len()));\n        let res = body_writer\n            .write_body(&mut mock_io, &data[..])\n            .await\n            .unwrap()\n            .unwrap();\n        assert_eq!(res, data.len());\n        assert_eq!(\n            body_writer.body_mode,\n            BodyMode::ChunkedEncoding(data.len() * 2)\n        );\n        let res = body_writer.finish(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, data.len() * 2);\n        assert_eq!(body_writer.body_mode, BodyMode::Complete(data.len() * 2));\n    }\n\n    #[tokio::test]\n    async fn write_body_http10() {\n        init_log();\n        let data = b\"a\";\n        let mut mock_io = Builder::new().write(&data[..]).write(&data[..]).build();\n        let mut body_writer = BodyWriter::new();\n        body_writer.init_close_delimited();\n        assert_eq!(body_writer.body_mode, BodyMode::UntilClose(0));\n        let res = body_writer\n            .write_body(&mut mock_io, &data[..])\n            .await\n            .unwrap()\n            .unwrap();\n        assert_eq!(res, 1);\n        assert_eq!(body_writer.body_mode, BodyMode::UntilClose(1));\n        let res = body_writer\n            .write_body(&mut mock_io, &data[..])\n            .await\n            .unwrap()\n            .unwrap();\n        assert_eq!(res, 1);\n        assert_eq!(body_writer.body_mode, BodyMode::UntilClose(2));\n        let res = body_writer.finish(&mut mock_io).await.unwrap().unwrap();\n        assert_eq!(res, 2);\n        assert_eq!(body_writer.body_mode, BodyMode::Complete(2));\n    }\n}\n"
  },
  {
    "path": "pingora-core/src/protocols/http/v1/client.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! HTTP/1.x client session\n\nuse bytes::{BufMut, Bytes, BytesMut};\nuse http::{header, header::AsHeaderName, HeaderValue, StatusCode, Version};\nuse log::{debug, trace};\nuse pingora_error::{Error, ErrorType::*, OrErr, Result, RetryType};\nuse pingora_http::{HMap, IntoCaseHeaderName, RequestHeader, ResponseHeader};\nuse pingora_timeout::timeout;\nuse std::io::ErrorKind;\nuse std::str;\nuse std::time::Duration;\nuse tokio::io::{AsyncReadExt, AsyncWriteExt};\n\nuse super::body::{BodyReader, BodyWriter};\nuse super::common::*;\nuse crate::protocols::http::HttpTask;\nuse crate::protocols::{Digest, SocketAddr, Stream, UniqueID, UniqueIDType};\nuse crate::utils::{BufRef, KVRef};\n\n/// The HTTP 1.x client session\npub struct HttpSession {\n    buf: Bytes,\n    pub(crate) underlying_stream: Stream,\n    raw_header: Option<BufRef>,\n    preread_body: Option<BufRef>,\n    body_reader: BodyReader,\n    body_writer: BodyWriter,\n    // timeouts:\n    /// The read timeout, which will be applied to both reading the header and the body.\n    /// The timeout is reset on every read. This is not a timeout on the overall duration of the\n    /// response.\n    pub read_timeout: Option<Duration>,\n    /// The write timeout which will be applied to both writing request header and body.\n    /// The timeout is reset on every write. This is not a timeout on the overall duration of the\n    /// request.\n    pub write_timeout: Option<Duration>,\n    keepalive_timeout: KeepaliveStatus,\n    pub(crate) digest: Box<Digest>,\n    response_header: Option<Box<ResponseHeader>>,\n    request_written: Option<Box<RequestHeader>>,\n    bytes_sent: usize,\n    /// Total response body payload bytes received from upstream\n    body_recv: usize,\n    // Tracks whether upgrade handshake was successfully completed\n    upgraded: bool,\n    // Tracks whether downstream request body started sending upgraded bytes\n    received_upgrade_req_body: bool,\n    // Tracks whether the response read was ever close-delimited\n    // (even after body complete)\n    close_delimited_resp: bool,\n    // If allowed, does not fail with error on invalid content-length\n    // (treats as close-delimited response).\n    allow_h1_response_invalid_content_length: bool,\n}\n\n/// HTTP 1.x client session\nimpl HttpSession {\n    /// Create a new http client session from an established (TCP or TLS) [`Stream`].\n    pub fn new(stream: Stream) -> Self {\n        // TODO: maybe we should put digest in the connection itself\n        let digest = Box::new(Digest {\n            ssl_digest: stream.get_ssl_digest(),\n            timing_digest: stream.get_timing_digest(),\n            proxy_digest: stream.get_proxy_digest(),\n            socket_digest: stream.get_socket_digest(),\n        });\n        HttpSession {\n            underlying_stream: stream,\n            buf: Bytes::new(), // zero size, will be replaced by parsed header later\n            raw_header: None,\n            preread_body: None,\n            body_reader: BodyReader::new(true),\n            body_writer: BodyWriter::new(),\n            keepalive_timeout: KeepaliveStatus::Off,\n            response_header: None,\n            request_written: None,\n            read_timeout: None,\n            write_timeout: None,\n            digest,\n            bytes_sent: 0,\n            body_recv: 0,\n            upgraded: false,\n            received_upgrade_req_body: false,\n            close_delimited_resp: false,\n            allow_h1_response_invalid_content_length: false,\n        }\n    }\n\n    /// Create a new http client session and apply peer options\n    pub fn new_with_options<P: crate::upstreams::peer::Peer>(stream: Stream, peer: &P) -> Self {\n        let mut session = Self::new(stream);\n        if let Some(options) = peer.get_peer_options() {\n            session.set_allow_h1_response_invalid_content_length(\n                options.allow_h1_response_invalid_content_length,\n            );\n        }\n        session\n    }\n\n    /// Write the request header to the server\n    /// After the request header is sent. The caller can either start reading the response or\n    /// sending request body if any.\n    pub async fn write_request_header(&mut self, req: Box<RequestHeader>) -> Result<usize> {\n        // TODO: make sure this can only be called once\n        // init body writer\n        self.init_req_body_writer(&req);\n\n        let to_wire = http_req_header_to_wire(&req).unwrap();\n        trace!(\"Writing request header: {to_wire:?}\");\n\n        let write_fut = self.underlying_stream.write_all(to_wire.as_ref());\n        match self.write_timeout {\n            Some(t) => match timeout(t, write_fut).await {\n                Ok(res) => res,\n                Err(_) => Err(std::io::Error::from(ErrorKind::TimedOut)),\n            },\n            None => write_fut.await,\n        }\n        .map_err(|e| match e.kind() {\n            ErrorKind::TimedOut => {\n                Error::because(WriteTimedout, \"while writing request headers (timeout)\", e)\n            }\n            _ => Error::because(WriteError, \"while writing request headers\", e),\n        })?;\n\n        self.underlying_stream\n            .flush()\n            .await\n            .or_err(WriteError, \"flushing request header\")?;\n\n        // write was successful\n        self.request_written = Some(req);\n        Ok(to_wire.len())\n    }\n\n    async fn do_write_body(&mut self, buf: &[u8]) -> Result<Option<usize>> {\n        let written = self\n            .body_writer\n            .write_body(&mut self.underlying_stream, buf)\n            .await;\n\n        if let Ok(Some(num_bytes)) = written {\n            self.bytes_sent += num_bytes;\n        }\n\n        written\n    }\n\n    /// Write request body. Return Ok(None) if no more body should be written, either due to\n    /// Content-Length or the last chunk is already sent\n    pub async fn write_body(&mut self, buf: &[u8]) -> Result<Option<usize>> {\n        // TODO: verify that request header is sent already\n        match self.write_timeout {\n            Some(t) => match timeout(t, self.do_write_body(buf)).await {\n                Ok(res) => res,\n                Err(_) => Error::e_explain(WriteTimedout, format!(\"writing body, timeout: {t:?}\")),\n            },\n            None => self.do_write_body(buf).await,\n        }\n    }\n\n    fn maybe_force_close_body_reader(&mut self) {\n        if self.upgraded && self.received_upgrade_req_body && !self.body_reader.body_done() {\n            // request is done, reset the response body to close\n            self.body_reader.init_content_length(0, b\"\");\n        }\n    }\n\n    /// Flush local buffer and notify the server by sending the last chunk if chunked encoding is\n    /// used.\n    pub async fn finish_body(&mut self) -> Result<Option<usize>> {\n        let res = self.body_writer.finish(&mut self.underlying_stream).await?;\n        self.underlying_stream\n            .flush()\n            .await\n            .or_err(WriteError, \"flushing body\")?;\n\n        self.maybe_force_close_body_reader();\n        Ok(res)\n    }\n\n    // Validate the response header read. This function must be called after the response header\n    // read.\n    fn validate_response(&self) -> Result<()> {\n        let resp_header = self\n            .response_header\n            .as_ref()\n            .expect(\"response header must be read\");\n\n        // ad-hoc checks\n        super::common::check_dup_content_length(&resp_header.headers)?;\n\n        // Validate content-length value if present\n        // Note: Content-Length is already removed if Transfer-Encoding is present\n        if !self.allow_h1_response_invalid_content_length {\n            self.get_content_length()?;\n        }\n\n        Ok(())\n    }\n\n    /// Read the response header from the server\n    /// This function can be called multiple times, if the headers received are just informational\n    /// headers.\n    pub async fn read_response(&mut self) -> Result<usize> {\n        if self.preread_body.as_ref().is_none_or(|b| b.is_empty()) {\n            // preread_body is set after a completed valid response header is read\n            // if called multiple times (i.e. after informational responses),\n            // we want to parse the already read buffer bytes as more headers.\n            // (https://datatracker.ietf.org/doc/html/rfc9110#section-15.2\n            // \"A 1xx response is terminated by the end of the header section;\n            // it cannot contain content or trailers.\")\n            // If this next read_response call completes successfully,\n            // self.buf will be reset to the last response + any body.\n            self.buf.clear();\n        }\n        let mut buf = BytesMut::with_capacity(INIT_HEADER_BUF_SIZE);\n        let mut already_read: usize = 0;\n        loop {\n            if already_read > MAX_HEADER_SIZE {\n                /* NOTE: this check only blocks second read. The first large read is allowed\n                since the buf is already allocated. The goal is to avoid slowly bloating\n                this buffer */\n                return Error::e_explain(\n                    InvalidHTTPHeader,\n                    format!(\"Response header larger than {MAX_HEADER_SIZE}\"),\n                );\n            }\n\n            let preread = self.preread_body.take();\n            let read_result = if let Some(preread) = preread.filter(|b| !b.is_empty()) {\n                buf.put_slice(preread.get(&self.buf));\n                Ok(preread.len())\n            } else {\n                let read_fut = self.underlying_stream.read_buf(&mut buf);\n                match self.read_timeout {\n                    Some(t) => timeout(t, read_fut).await.map_err(|_| {\n                        Error::explain(ReadTimedout, \"while reading response headers\")\n                    })?,\n                    None => read_fut.await,\n                }\n            };\n            let n = match read_result {\n                Ok(n) => match n {\n                    0 => {\n                        let mut e = Error::explain(\n                            ConnectionClosed,\n                            format!(\n                                \"while reading response headers, bytes already read: {already_read}\",\n                            ),\n                        );\n                        e.retry = RetryType::ReusedOnly;\n                        return Err(e);\n                    }\n                    _ => {\n                        n /* read n bytes, continue */\n                    }\n                },\n                Err(e) => {\n                    let true_io_error = e.raw_os_error().is_some();\n                    let mut e = Error::because(\n                        ReadError,\n                        format!(\n                            \"while reading response headers, bytes already read: {already_read}\",\n                        ),\n                        e,\n                    );\n                    // Likely OSError, typical if a previously reused connection drops it\n                    if true_io_error {\n                        e.retry = RetryType::ReusedOnly;\n                    } // else: not safe to retry TLS error\n                    return Err(e);\n                }\n            };\n            already_read += n;\n            let mut headers = [httparse::EMPTY_HEADER; MAX_HEADERS];\n            let mut resp = httparse::Response::new(&mut headers);\n            let parsed = parse_resp_buffer(&mut resp, &buf);\n            match parsed {\n                HeaderParseState::Complete(s) => {\n                    self.raw_header = Some(BufRef(0, s));\n                    self.preread_body = Some(BufRef(s, already_read));\n                    let base = buf.as_ptr() as usize;\n                    let mut header_refs = Vec::<KVRef>::with_capacity(resp.headers.len());\n\n                    // Note: resp.headers has the correct number of headers\n                    // while header_refs doesn't as it is still empty\n                    let _num_headers = populate_headers(base, &mut header_refs, resp.headers);\n\n                    let mut response_header = Box::new(ResponseHeader::build(\n                        resp.code.unwrap(),\n                        Some(resp.headers.len()),\n                    )?);\n\n                    // TODO: enforce https://datatracker.ietf.org/doc/html/rfc9110#section-15.2\n                    // \"Since HTTP/1.0 did not define any 1xx status codes,\n                    // a server MUST NOT send a 1xx response to an HTTP/1.0 client.\"\n                    response_header.set_version(match resp.version {\n                        Some(1) => Version::HTTP_11,\n                        Some(0) => Version::HTTP_10,\n                        _ => Version::HTTP_09,\n                    });\n\n                    response_header.set_reason_phrase(resp.reason)?;\n\n                    let buf = buf.freeze();\n\n                    for header in header_refs {\n                        let header_name = header.get_name_bytes(&buf);\n                        let header_name = header_name.into_case_header_name();\n                        let value_bytes = header.get_value_bytes(&buf);\n                        let header_value = if cfg!(debug_assertions) {\n                            // from_maybe_shared_unchecked() in debug mode still checks whether\n                            // the header value is valid, which breaks the _obsolete_multiline\n                            // support. To work around this, in debug mode, we replace CRLF with\n                            // whitespace\n                            if let Some(p) = value_bytes.windows(CRLF.len()).position(|w| w == CRLF)\n                            {\n                                let mut new_header = Vec::from_iter(value_bytes);\n                                new_header[p] = b' ';\n                                new_header[p + 1] = b' ';\n                                unsafe {\n                                    http::HeaderValue::from_maybe_shared_unchecked(new_header)\n                                }\n                            } else {\n                                unsafe {\n                                    http::HeaderValue::from_maybe_shared_unchecked(value_bytes)\n                                }\n                            }\n                        } else {\n                            // safe because this is from what we parsed\n                            unsafe { http::HeaderValue::from_maybe_shared_unchecked(value_bytes) }\n                        };\n                        response_header\n                            .append_header(header_name, header_value)\n                            .or_err(InvalidHTTPHeader, \"while parsing request header\")?;\n                    }\n\n                    let contains_transfer_encoding = response_header\n                        .headers\n                        .contains_key(header::TRANSFER_ENCODING);\n                    let contains_content_length =\n                        response_header.headers.contains_key(header::CONTENT_LENGTH);\n\n                    // Transfer encoding overrides content length, so when\n                    // both are present, we MUST remove content length. This is\n                    // https://datatracker.ietf.org/doc/html/rfc9112#section-6.3-2.3\n                    if contains_content_length && contains_transfer_encoding {\n                        response_header.remove_header(&header::CONTENT_LENGTH);\n                    }\n\n                    self.buf = buf;\n                    self.response_header = Some(response_header);\n                    self.validate_response()?;\n                    // convert to upgrade body type\n                    // https://datatracker.ietf.org/doc/html/rfc9110#status.101\n                    // as an \"informational\" header, this cannot have a body\n                    self.upgraded = self\n                        .is_upgrade(self.response_header.as_deref().expect(\"init above\"))\n                        .unwrap_or(false);\n                    // init body reader if upgrade status has changed body mode\n                    // (read_response_task will immediately try to init body afterwards anyways)\n                    // informational headers will automatically avoid initializing body reader\n                    self.init_body_reader();\n                    // note that the (request) body writer is converted to close delimit\n                    // when the upgraded body tasks are received\n                    return Ok(s);\n                }\n                HeaderParseState::Partial => { /* continue the loop */ }\n                HeaderParseState::Invalid(e) => {\n                    return Error::e_because(\n                        InvalidHTTPHeader,\n                        format!(\"buf: {}\", buf.escape_ascii()),\n                        e,\n                    );\n                }\n            }\n        }\n    }\n\n    /// Similar to [`Self::read_response()`], read the response header and then return a copy of it.\n    pub async fn read_resp_header_parts(&mut self) -> Result<Box<ResponseHeader>> {\n        self.read_response().await?;\n        // safe to unwrap because it is just read\n        Ok(Box::new(self.resp_header().unwrap().clone()))\n    }\n\n    /// Return a reference of the [`ResponseHeader`] if the response is read\n    pub fn resp_header(&self) -> Option<&ResponseHeader> {\n        self.response_header.as_deref()\n    }\n\n    /// Get the header value for the given header name from the response header\n    /// If there are multiple headers under the same name, the first one will be returned\n    /// Use `self.resp_header().header.get_all(name)` to get all the headers under the same name\n    /// Always return `None` if the response is not read yet.\n    pub fn get_header(&self, name: impl AsHeaderName) -> Option<&HeaderValue> {\n        self.response_header\n            .as_ref()\n            .and_then(|h| h.headers.get(name))\n    }\n\n    /// Get the request header as raw bytes, `b\"\"` when the header doesn't exist or response not read\n    pub fn get_header_bytes(&self, name: impl AsHeaderName) -> &[u8] {\n        self.get_header(name).map_or(b\"\", |v| v.as_bytes())\n    }\n\n    /// Return the status code of the response if read\n    pub fn get_status(&self) -> Option<StatusCode> {\n        self.response_header.as_ref().map(|h| h.status)\n    }\n\n    async fn do_read_body(&mut self) -> Result<Option<BufRef>> {\n        self.init_body_reader();\n        self.body_reader\n            .read_body(&mut self.underlying_stream)\n            .await\n    }\n\n    /// Read the response body into the internal buffer.\n    /// Return `Ok(Some(ref)) after a successful read.\n    /// Return `Ok(None)` if there is no more body to read.\n    pub async fn read_body_ref(&mut self) -> Result<Option<&[u8]>> {\n        let result = match self.read_timeout {\n            Some(t) => match timeout(t, self.do_read_body()).await {\n                Ok(res) => res,\n                Err(_) => Error::e_explain(ReadTimedout, format!(\"reading body, timeout: {t:?}\")),\n            },\n            None => self.do_read_body().await,\n        };\n\n        result.map(|maybe_body| {\n            maybe_body.map(|body_ref| {\n                let slice = self.body_reader.get_body(&body_ref);\n                self.body_recv = self.body_recv.saturating_add(slice.len());\n                slice\n            })\n        })\n    }\n\n    /// Similar to [`Self::read_body_ref`] but return `Bytes` instead of a slice reference.\n    pub async fn read_body_bytes(&mut self) -> Result<Option<Bytes>> {\n        let read = self.read_body_ref().await?;\n        Ok(read.map(Bytes::copy_from_slice))\n    }\n\n    /// Upstream response body bytes received (payload only; excludes headers/framing).\n    pub fn body_bytes_received(&self) -> usize {\n        self.body_recv\n    }\n\n    /// Whether there is no more body to read.\n    pub fn is_body_done(&mut self) -> bool {\n        self.init_body_reader();\n        self.body_reader.body_done()\n    }\n\n    pub fn set_allow_h1_response_invalid_content_length(&mut self, allow: bool) {\n        self.allow_h1_response_invalid_content_length = allow;\n    }\n\n    pub(super) fn get_headers_raw(&self) -> &[u8] {\n        // TODO: these get_*() could panic. handle them better\n        self.raw_header.as_ref().unwrap().get(&self.buf[..])\n    }\n\n    /// Get the raw response header bytes\n    pub fn get_headers_raw_bytes(&self) -> Bytes {\n        self.raw_header.as_ref().unwrap().get_bytes(&self.buf)\n    }\n\n    fn set_keepalive(&mut self, seconds: Option<u64>) {\n        match seconds {\n            Some(sec) => {\n                if sec > 0 {\n                    self.keepalive_timeout = KeepaliveStatus::Timeout(Duration::from_secs(sec));\n                } else {\n                    self.keepalive_timeout = KeepaliveStatus::Infinite;\n                }\n            }\n            None => {\n                self.keepalive_timeout = KeepaliveStatus::Off;\n            }\n        }\n    }\n\n    /// Apply keepalive settings according to the server's response\n    /// For HTTP 1.1, assume keepalive as long as there is no `Connection: Close` request header.\n    /// For HTTP 1.0, only keepalive if there is an explicit header `Connection: keep-alive`.\n    pub fn respect_keepalive(&mut self) {\n        if self.upgraded || self.get_status() == Some(StatusCode::SWITCHING_PROTOCOLS) {\n            // make sure the connection is closed at the end when 101/upgrade is used\n            self.set_keepalive(None);\n            return;\n        }\n        if self.body_reader.need_init() || self.close_delimited_resp {\n            // Defense-in-depth: response body close-delimited (or no body interpretation\n            // upon reuse check)\n            // explicitly disable reuse\n            self.set_keepalive(None);\n            return;\n        }\n        if self.body_reader.has_bytes_overread() {\n            // if more bytes sent than expected, there are likely more bytes coming\n            // so don't reuse this connection\n            self.set_keepalive(None);\n            return;\n        }\n\n        // Per [RFC 9112 Section 6.1-16](https://datatracker.ietf.org/doc/html/rfc9112#section-6.1-16),\n        // if Transfer-Encoding is received in HTTP/1.0 response, connection MUST be closed after processing.\n        if self.resp_header().map(|h| h.version) == Some(Version::HTTP_10)\n            && self\n                .resp_header()\n                .and_then(|h| h.headers.get(header::TRANSFER_ENCODING))\n                .is_some()\n        {\n            self.set_keepalive(None);\n            return;\n        }\n        if let Some(keepalive) = self.is_connection_keepalive() {\n            if keepalive {\n                let (timeout, _max_use) = self.get_keepalive_values();\n                // TODO: respect max_use\n                match timeout {\n                    Some(d) => self.set_keepalive(Some(d)),\n                    None => self.set_keepalive(Some(0)), // infinite\n                }\n            } else {\n                self.set_keepalive(None);\n            }\n        } else if self.resp_header().map(|h| h.version) == Some(Version::HTTP_11) {\n            self.set_keepalive(Some(0)); // on by default for http 1.1\n        } else {\n            self.set_keepalive(None); // off by default for http 1.0\n        }\n    }\n\n    // Whether this session will be kept alive\n    pub fn will_keepalive(&self) -> bool {\n        !matches!(self.keepalive_timeout, KeepaliveStatus::Off)\n    }\n\n    fn is_connection_keepalive(&self) -> Option<bool> {\n        let request_keepalive = self\n            .request_written\n            .as_ref()\n            .and_then(|req| is_buf_keepalive(req.headers.get(header::CONNECTION)));\n\n        match request_keepalive {\n            // ignore what the server sends if request disables keepalive explicitly\n            Some(false) => Some(false),\n            _ => is_buf_keepalive(self.get_header(header::CONNECTION)),\n        }\n    }\n\n    /// `Keep-Alive: timeout=5, max=1000` => 5, 1000\n    /// This is defined in the below spec, this not part of any RFC, so\n    /// it's behavior is different on different platforms.\n    /// https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Keep-Alive\n    fn get_keepalive_values(&self) -> (Option<u64>, Option<usize>) {\n        let Some(keep_alive_header) = self.get_header(\"Keep-Alive\") else {\n            return (None, None);\n        };\n\n        let Ok(header_value) = str::from_utf8(keep_alive_header.as_bytes()) else {\n            return (None, None);\n        };\n\n        let mut timeout = None;\n        let mut max = None;\n\n        for param in header_value.split(',') {\n            let parts = param.split_once('=').map(|(k, v)| (k.trim(), v));\n            match parts {\n                Some((\"timeout\", timeout_value)) => timeout = timeout_value.trim().parse().ok(),\n                Some((\"max\", max_value)) => max = max_value.trim().parse().ok(),\n                _ => {}\n            }\n        }\n\n        (timeout, max)\n    }\n\n    /// Close the connection abruptly. This allows to signal the server that the connection is closed\n    /// before dropping [`HttpSession`]\n    pub async fn shutdown(&mut self) {\n        let _ = self.underlying_stream.shutdown().await;\n    }\n\n    /// Consume `self`, if the connection can be reused, the underlying stream will be returned.\n    /// The returned connection can be kept in a connection pool so that next time the same\n    /// server is being contacted. A new client session can be created via [`Self::new()`].\n    /// If the connection cannot be reused, the underlying stream will be closed and `None` will be\n    /// returned.\n    pub async fn reuse(mut self) -> Option<Stream> {\n        // TODO: this function is unnecessarily slow for keepalive case\n        // because that case does not need async\n        match self.keepalive_timeout {\n            KeepaliveStatus::Off => {\n                debug!(\"HTTP shutdown connection\");\n                self.shutdown().await;\n                None\n            }\n            _ => Some(self.underlying_stream),\n        }\n    }\n\n    fn init_body_reader(&mut self) {\n        if self.body_reader.need_init() {\n            // follow https://datatracker.ietf.org/doc/html/rfc9112#section-6.3\n            let preread_body = self.preread_body.as_ref().unwrap().get(&self.buf[..]);\n\n            if let Some(req) = self.request_written.as_ref() {\n                if req.method == http::method::Method::HEAD {\n                    self.body_reader.init_content_length(0, preread_body);\n                    return;\n                }\n            }\n\n            let upgraded = if let Some(code) = self.get_status() {\n                match code.as_u16() {\n                    101 => self.is_upgrade_req(),\n                    100..=199 => {\n                        // informational headers, not enough to init body reader\n                        return;\n                    }\n                    204 | 304 => {\n                        // no body by definition\n                        self.body_reader.init_content_length(0, preread_body);\n                        return;\n                    }\n                    _ => false,\n                }\n            } else {\n                false\n            };\n\n            if upgraded {\n                self.body_reader.init_close_delimited(preread_body);\n                self.close_delimited_resp = true;\n            } else if self.is_chunked_encoding() {\n                // if chunked encoding, content-length should be ignored\n                self.body_reader.init_chunked(preread_body);\n            } else if let Some(cl) = self.get_content_length().unwrap_or(None) {\n                self.body_reader.init_content_length(cl, preread_body);\n            } else {\n                self.body_reader.init_close_delimited(preread_body);\n                self.close_delimited_resp = true;\n            }\n        }\n    }\n\n    /// Whether this request is for upgrade\n    pub fn is_upgrade_req(&self) -> bool {\n        match self.request_written.as_deref() {\n            Some(req) => is_upgrade_req(req),\n            None => false,\n        }\n    }\n\n    /// `Some(true)` if the this is a successful upgrade\n    /// `Some(false)` if the request is an upgrade but the response refuses it\n    /// `None` if the request is not an upgrade.\n    fn is_upgrade(&self, header: &ResponseHeader) -> Option<bool> {\n        if self.is_upgrade_req() {\n            Some(is_upgrade_resp(header))\n        } else {\n            None\n        }\n    }\n\n    /// Was this request successfully turned into an upgraded connection?\n    ///\n    /// Both the request had to have been an `Upgrade` request\n    /// and the response had to have been a `101 Switching Protocols`.\n    pub fn was_upgraded(&self) -> bool {\n        self.upgraded\n    }\n\n    /// If upgraded but not yet converted, then body writer will be\n    /// converted to http1.0 mode (pass through bytes as-is).\n    pub fn maybe_upgrade_body_writer(&mut self) {\n        if self.was_upgraded() {\n            self.received_upgrade_req_body = true;\n            self.body_writer.convert_to_close_delimited();\n        }\n    }\n\n    fn get_content_length(&self) -> Result<Option<usize>> {\n        buf_to_content_length(\n            self.get_header(header::CONTENT_LENGTH)\n                .map(|v| v.as_bytes()),\n        )\n    }\n\n    fn is_chunked_encoding(&self) -> bool {\n        self.resp_header()\n            .map(|h| is_chunked_encoding_from_headers(&h.headers))\n            .unwrap_or(false)\n    }\n\n    fn init_req_body_writer(&mut self, header: &RequestHeader) {\n        self.init_body_writer_comm(&header.headers)\n    }\n\n    fn init_body_writer_comm(&mut self, headers: &HMap) {\n        if is_chunked_encoding_from_headers(headers) {\n            // transfer-encoding takes priority over content-length\n            self.body_writer.init_chunked();\n        } else {\n            let content_length =\n                header_value_content_length(headers.get(http::header::CONTENT_LENGTH));\n            match content_length {\n                Some(length) => {\n                    self.body_writer.init_content_length(length);\n                }\n                None => {\n                    // Per RFC 9112: \"Request messages are never close-delimited because they are\n                    // always explicitly framed by length or transfer coding, with the absence of\n                    // both implying the request ends immediately after the header section.\"\n                    // Requests without Content-Length or Transfer-Encoding have 0 body\n                    self.body_writer.init_content_length(0);\n                }\n            }\n        }\n    }\n\n    // should (continue to) try to read response header or start reading response body\n    fn should_read_resp_header(&self) -> bool {\n        match self.get_status().map(|s| s.as_u16()) {\n            Some(101) => false,      // switching protocol successful, no more header to read\n            Some(100..=199) => true, // only informational header read\n            Some(_) => false,\n            None => true, // no response code, no header read yet\n        }\n    }\n\n    pub async fn read_response_task(&mut self) -> Result<HttpTask> {\n        if self.should_read_resp_header() {\n            let resp_header = self.read_resp_header_parts().await?;\n            let end_of_body = self.is_body_done();\n            debug!(\"Response header: {resp_header:?}\");\n            trace!(\n                \"Raw Response header: {:?}\",\n                str::from_utf8(self.get_headers_raw()).unwrap()\n            );\n            Ok(HttpTask::Header(resp_header, end_of_body))\n        } else if self.is_body_done() {\n            // no body\n            debug!(\"Response is done\");\n            Ok(HttpTask::Done)\n        } else {\n            /* need to read body */\n            let body = self.read_body_bytes().await?;\n            let end_of_body = self.is_body_done();\n            debug!(\n                \"Response body: {} bytes, end: {end_of_body}\",\n                body.as_ref().map_or(0, |b| b.len())\n            );\n            trace!(\"Response body: {body:?}, upgraded: {}\", self.upgraded);\n            if self.upgraded {\n                Ok(HttpTask::UpgradedBody(body, end_of_body))\n            } else {\n                Ok(HttpTask::Body(body, end_of_body))\n            }\n        }\n        // TODO: support h1 trailer\n    }\n\n    /// Return the [Digest] of the connection\n    ///\n    /// For reused connection, the timing in the digest will reflect its initial handshakes\n    /// The caller should check if the connection is reused to avoid misuse the timing field.\n    pub fn digest(&self) -> &Digest {\n        &self.digest\n    }\n\n    /// Return a mutable [Digest] reference for the connection.\n    pub fn digest_mut(&mut self) -> &mut Digest {\n        &mut self.digest\n    }\n\n    /// Return the server (peer) address recorded in the connection digest.\n    pub fn server_addr(&self) -> Option<&SocketAddr> {\n        self.digest()\n            .socket_digest\n            .as_ref()\n            .map(|d| d.peer_addr())?\n    }\n\n    /// Return the client (local) address recorded in the connection digest.\n    pub fn client_addr(&self) -> Option<&SocketAddr> {\n        self.digest()\n            .socket_digest\n            .as_ref()\n            .map(|d| d.local_addr())?\n    }\n\n    /// Get the reference of the [Stream] that this HTTP session is operating upon.\n    pub fn stream(&self) -> &Stream {\n        &self.underlying_stream\n    }\n\n    /// Consume `self`, the underlying [Stream] will be returned and can be used\n    /// directly, for example, in the case of HTTP upgrade. It is not flushed\n    /// prior to being returned.\n    pub fn into_inner(self) -> Stream {\n        self.underlying_stream\n    }\n}\n\n#[inline]\nfn parse_resp_buffer<'buf>(\n    resp: &mut httparse::Response<'_, 'buf>,\n    buf: &'buf [u8],\n) -> HeaderParseState {\n    let mut parser = httparse::ParserConfig::default();\n    parser.allow_spaces_after_header_name_in_responses(true);\n    parser.allow_obsolete_multiline_headers_in_responses(true);\n    let res = match parser.parse_response(resp, buf) {\n        Ok(s) => s,\n        Err(e) => {\n            return HeaderParseState::Invalid(e);\n        }\n    };\n    match res {\n        httparse::Status::Complete(s) => HeaderParseState::Complete(s),\n        _ => HeaderParseState::Partial,\n    }\n}\n\n// TODO: change it to to_buf\n#[inline]\npub fn http_req_header_to_wire(req: &RequestHeader) -> Option<BytesMut> {\n    let mut buf = BytesMut::with_capacity(512);\n\n    // Request-Line\n    let method = req.method.as_str().as_bytes();\n    buf.put_slice(method);\n    buf.put_u8(b' ');\n    buf.put_slice(req.raw_path());\n    buf.put_u8(b' ');\n\n    let version = match req.version {\n        Version::HTTP_09 => \"HTTP/0.9\",\n        Version::HTTP_10 => \"HTTP/1.0\",\n        Version::HTTP_11 => \"HTTP/1.1\",\n        Version::HTTP_2 => \"HTTP/2\",\n        _ => {\n            return None; /*TODO: unsupported version */\n        }\n    };\n    buf.put_slice(version.as_bytes());\n    buf.put_slice(CRLF);\n\n    // headers\n    req.header_to_h1_wire(&mut buf);\n    buf.put_slice(CRLF);\n    Some(buf)\n}\n\nimpl UniqueID for HttpSession {\n    fn id(&self) -> UniqueIDType {\n        self.underlying_stream.id()\n    }\n}\n\n#[cfg(test)]\nmod tests_stream {\n    use super::*;\n    use crate::protocols::http::v1::body::{BodyMode, ParseState};\n    use crate::upstreams::peer::PeerOptions;\n    use crate::ErrorType;\n    use rstest::rstest;\n    use tokio_test::io::Builder;\n\n    fn init_log() {\n        let _ = env_logger::builder().is_test(true).try_init();\n    }\n\n    #[tokio::test]\n    async fn read_basic_response() {\n        init_log();\n        let input = b\"HTTP/1.1 200 OK\\r\\n\\r\\n\";\n        let mock_io = Builder::new().read(&input[..]).build();\n        let mut http_stream = HttpSession::new(Box::new(mock_io));\n        let res = http_stream.read_response().await;\n        assert_eq!(input.len(), res.unwrap());\n        assert_eq!(0, http_stream.resp_header().unwrap().headers.len());\n    }\n\n    #[tokio::test]\n    async fn read_response_custom_reason() {\n        init_log();\n        let input = b\"HTTP/1.1 200 Just Fine\\r\\n\\r\\n\";\n        let mock_io = Builder::new().read(&input[..]).build();\n        let mut http_stream = HttpSession::new(Box::new(mock_io));\n        let res = http_stream.read_response().await;\n        assert_eq!(input.len(), res.unwrap());\n        assert_eq!(\n            http_stream.resp_header().unwrap().get_reason_phrase(),\n            Some(\"Just Fine\")\n        );\n    }\n\n    #[tokio::test]\n    async fn read_response_default() {\n        init_log();\n        let input_header = b\"HTTP/1.1 200 OK\\r\\n\\r\\n\";\n        let input_body = b\"abc\";\n        let input_close = b\"\"; // simulating close\n        let mock_io = Builder::new()\n            .read(&input_header[..])\n            .read(&input_body[..])\n            .read(&input_close[..])\n            .build();\n        let mut http_stream = HttpSession::new(Box::new(mock_io));\n        let res = http_stream.read_response().await;\n        assert_eq!(input_header.len(), res.unwrap());\n        let res = http_stream.read_body_ref().await.unwrap();\n        assert_eq!(res.unwrap(), input_body);\n        assert_eq!(\n            http_stream.body_reader.body_state,\n            ParseState::UntilClose(3)\n        );\n        let res = http_stream.read_body_ref().await.unwrap();\n        assert_eq!(res, None);\n        assert_eq!(http_stream.body_reader.body_state, ParseState::Complete(3));\n    }\n\n    #[tokio::test]\n    async fn body_bytes_received_content_length() {\n        init_log();\n        let input_header = b\"HTTP/1.1 200 OK\\r\\nContent-Length: 3\\r\\n\\r\\n\";\n        let input_body = b\"abc\";\n        let input_close = b\"\"; // simulating close\n        let mock_io = Builder::new()\n            .read(&input_header[..])\n            .read(&input_body[..])\n            .read(&input_close[..])\n            .build();\n        let mut http = HttpSession::new(Box::new(mock_io));\n        http.read_response().await.unwrap();\n        let _ = http.read_body_ref().await.unwrap();\n        let _ = http.read_body_ref().await.unwrap();\n        assert_eq!(http.body_bytes_received(), 3);\n    }\n\n    #[tokio::test]\n    async fn body_bytes_received_chunked() {\n        init_log();\n        let input_header = b\"HTTP/1.1 200 OK\\r\\nTransfer-Encoding: chunked\\r\\n\\r\\n\";\n        let input_body = b\"3\\r\\nabc\\r\\n0\\r\\n\\r\\n\";\n        let mock_io = Builder::new()\n            .read(&input_header[..])\n            .read(&input_body[..])\n            .build();\n        let mut http = HttpSession::new(Box::new(mock_io));\n        http.read_response().await.unwrap();\n        // first read returns the payload chunk\n        let first = http.read_body_ref().await.unwrap();\n        assert_eq!(first.unwrap(), b\"abc\");\n        // next read consumes terminating chunk\n        let _ = http.read_body_ref().await.unwrap();\n        assert_eq!(http.body_bytes_received(), 3);\n    }\n\n    #[tokio::test]\n    async fn h1_body_bytes_received_http10_until_close() {\n        init_log();\n        let header = b\"HTTP/1.1 200 OK\\r\\n\\r\\n\";\n        let body = b\"abc\";\n        let close = b\"\";\n        let mock = Builder::new()\n            .read(&header[..])\n            .read(&body[..])\n            .read(&close[..])\n            .build();\n        let mut http = HttpSession::new(Box::new(mock));\n        http.read_response().await.unwrap();\n        let _ = http.read_body_ref().await.unwrap();\n        let _ = http.read_body_ref().await.unwrap();\n        assert_eq!(http.body_bytes_received(), 3);\n    }\n\n    #[tokio::test]\n    async fn h1_body_bytes_received_chunked_multi() {\n        init_log();\n        let header = b\"HTTP/1.1 200 OK\\r\\nTransfer-Encoding: chunked\\r\\n\\r\\n\";\n        let body = b\"1\\r\\na\\r\\n2\\r\\nbc\\r\\n0\\r\\n\\r\\n\"; // payload abc\n        let mock = Builder::new().read(&header[..]).read(&body[..]).build();\n        let mut http = HttpSession::new(Box::new(mock));\n        http.read_response().await.unwrap();\n        // first chunk\n        let s1 = http.read_body_ref().await.unwrap().unwrap();\n        assert_eq!(s1, b\"a\");\n        // second chunk\n        let s2 = http.read_body_ref().await.unwrap().unwrap();\n        assert_eq!(s2, b\"bc\");\n        // end\n        let _ = http.read_body_ref().await.unwrap();\n        assert_eq!(http.body_bytes_received(), 3);\n    }\n\n    #[tokio::test]\n    async fn h1_body_bytes_received_preread_in_header_buf() {\n        init_log();\n        // header and a small body arrive together\n        let combined = b\"HTTP/1.1 200 OK\\r\\n\\r\\nabc\";\n        let close = b\"\";\n        let mock = Builder::new().read(&combined[..]).read(&close[..]).build();\n        let mut http = HttpSession::new(Box::new(mock));\n        http.read_response().await.unwrap();\n        // first body read should return the preread bytes\n        let s = http.read_body_ref().await.unwrap().unwrap();\n        assert_eq!(s, b\"abc\");\n        // then EOF\n        let _ = http.read_body_ref().await.unwrap();\n        assert_eq!(http.body_bytes_received(), 3);\n    }\n\n    #[tokio::test]\n    async fn h1_body_bytes_received_overread_content_length() {\n        init_log();\n        let header1 = b\"HTTP/1.1 200 OK\\r\\n\";\n        let header2 = b\"Content-Length: 2\\r\\n\\r\\n\";\n        let body = b\"abc\"; // one extra byte beyond CL\n        let mock = Builder::new()\n            .read(&header1[..])\n            .read(&header2[..])\n            .read(&body[..])\n            .build();\n        let mut http = HttpSession::new(Box::new(mock));\n        http.read_response().await.unwrap();\n        let s = http.read_body_ref().await.unwrap().unwrap();\n        assert_eq!(s, b\"ab\");\n        // then end\n        let _ = http.read_body_ref().await.unwrap();\n        assert_eq!(http.body_bytes_received(), 2);\n    }\n\n    #[tokio::test]\n    async fn h1_body_bytes_received_after_100_continue() {\n        init_log();\n        let info = b\"HTTP/1.1 100 Continue\\r\\n\\r\\n\";\n        let header = b\"HTTP/1.1 200 OK\\r\\nContent-Length: 1\\r\\n\\r\\n\";\n        let body = b\"x\";\n        let mock = Builder::new()\n            .read(&info[..])\n            .read(&header[..])\n            .read(&body[..])\n            .build();\n        let mut http = HttpSession::new(Box::new(mock));\n        // read informational\n        match http.read_response_task().await.unwrap() {\n            HttpTask::Header(h, eob) => {\n                assert_eq!(h.status, 100);\n                assert!(!eob);\n            }\n            _ => panic!(\"expected informational header\"),\n        }\n        // read final header\n        match http.read_response_task().await.unwrap() {\n            HttpTask::Header(h, eob) => {\n                assert_eq!(h.status, 200);\n                assert!(!eob);\n            }\n            _ => panic!(\"expected final header\"),\n        }\n        // read body\n        let s = http.read_body_ref().await.unwrap().unwrap();\n        assert_eq!(s, b\"x\");\n        let _ = http.read_body_ref().await.unwrap();\n        assert_eq!(http.body_bytes_received(), 1);\n    }\n\n    #[tokio::test]\n    async fn read_response_overread() {\n        init_log();\n        let input_header = b\"HTTP/1.1 200 OK\\r\\n\";\n        let input_header2 = b\"Content-Length: 2\\r\\n\\r\\n\";\n        let input_body = b\"abc\";\n        let mock_io = Builder::new()\n            .read(&input_header[..])\n            .read(&input_header2[..])\n            .read(&input_body[..])\n            .build();\n        let mut http_stream = HttpSession::new(Box::new(mock_io));\n        let res = http_stream.read_response().await;\n        assert_eq!(input_header.len() + input_header2.len(), res.unwrap());\n        let res = http_stream.read_body_ref().await.unwrap();\n        assert_eq!(res.unwrap(), &input_body[..2]);\n        assert_eq!(http_stream.body_reader.body_state, ParseState::Complete(2));\n        let res = http_stream.read_body_ref().await.unwrap();\n        assert_eq!(res, None);\n        assert_eq!(http_stream.body_reader.body_state, ParseState::Complete(2));\n        http_stream.respect_keepalive();\n        assert!(!http_stream.will_keepalive());\n    }\n\n    #[tokio::test]\n    async fn read_resp_header_with_space() {\n        init_log();\n        let input = b\"HTTP/1.1 200 OK\\r\\nServer : pingora\\r\\n\\r\\n\";\n        let mock_io = Builder::new().read(&input[..]).build();\n        let mut http_stream = HttpSession::new(Box::new(mock_io));\n        let res = http_stream.read_response().await;\n        assert_eq!(input.len(), res.unwrap());\n        assert_eq!(1, http_stream.resp_header().unwrap().headers.len());\n        assert_eq!(http_stream.get_header(\"Server\").unwrap(), \"pingora\");\n    }\n\n    #[cfg(feature = \"patched_http1\")]\n    #[tokio::test]\n    async fn read_resp_header_with_utf8() {\n        init_log();\n        let input = \"HTTP/1.1 200 OK\\r\\nServer👍: pingora\\r\\n\\r\\n\".as_bytes();\n        let mock_io = Builder::new().read(input).build();\n        let mut http_stream = HttpSession::new(Box::new(mock_io));\n        let resp = http_stream.read_resp_header_parts().await.unwrap();\n        assert_eq!(1, http_stream.resp_header().unwrap().headers.len());\n        assert_eq!(http_stream.get_header(\"Server👍\").unwrap(), \"pingora\");\n        assert_eq!(resp.headers.get(\"Server👍\").unwrap(), \"pingora\");\n    }\n\n    #[tokio::test]\n    #[should_panic(expected = \"There is still data left to read.\")]\n    async fn read_timeout() {\n        init_log();\n        let input = b\"HTTP/1.1 200 OK\\r\\n\\r\\n\";\n        let mock_io = Builder::new()\n            .wait(Duration::from_secs(2))\n            .read(&input[..])\n            .build();\n        let mut http_stream = HttpSession::new(Box::new(mock_io));\n        http_stream.read_timeout = Some(Duration::from_secs(1));\n        let res = http_stream.read_response().await;\n        assert_eq!(res.unwrap_err().etype(), &ErrorType::ReadTimedout);\n    }\n\n    #[tokio::test]\n    async fn read_2_buf() {\n        init_log();\n        let input1 = b\"HTTP/1.1 200 OK\\r\\n\";\n        let input2 = b\"Server: pingora\\r\\n\\r\\n\";\n        let mock_io = Builder::new().read(&input1[..]).read(&input2[..]).build();\n        let mut http_stream = HttpSession::new(Box::new(mock_io));\n        let res = http_stream.read_response().await;\n        assert_eq!(input1.len() + input2.len(), res.unwrap());\n        assert_eq!(\n            input1.len() + input2.len(),\n            http_stream.get_headers_raw().len()\n        );\n        assert_eq!(1, http_stream.resp_header().unwrap().headers.len());\n        assert_eq!(http_stream.get_header(\"Server\").unwrap(), \"pingora\");\n\n        assert_eq!(Some(StatusCode::OK), http_stream.get_status());\n        assert_eq!(Version::HTTP_11, http_stream.resp_header().unwrap().version);\n    }\n\n    #[tokio::test]\n    #[should_panic(expected = \"There is still data left to read.\")]\n    async fn read_invalid() {\n        let input1 = b\"HTP/1.1 200 OK\\r\\n\";\n        let input2 = b\"Server: pingora\\r\\n\\r\\n\";\n        let mock_io = Builder::new().read(&input1[..]).read(&input2[..]).build();\n        let mut http_stream = HttpSession::new(Box::new(mock_io));\n        let res = http_stream.read_response().await;\n        assert_eq!(&ErrorType::InvalidHTTPHeader, res.unwrap_err().etype());\n    }\n\n    #[tokio::test]\n    async fn write() {\n        let wire = b\"GET /test HTTP/1.1\\r\\nFoo: Bar\\r\\n\\r\\n\";\n        let mock_io = Builder::new().write(wire).build();\n        let mut http_stream = HttpSession::new(Box::new(mock_io));\n        let mut new_request = RequestHeader::build(\"GET\", b\"/test\", None).unwrap();\n        new_request.insert_header(\"Foo\", \"Bar\").unwrap();\n        let n = http_stream\n            .write_request_header(Box::new(new_request))\n            .await\n            .unwrap();\n        assert_eq!(wire.len(), n);\n    }\n\n    #[rstest]\n    #[case::negative(\"-1\")]\n    #[case::not_a_number(\"abc\")]\n    #[case::float(\"1.5\")]\n    #[case::empty(\"\")]\n    #[case::spaces(\"  \")]\n    #[case::mixed(\"123abc\")]\n    #[tokio::test]\n    async fn validate_response_rejects_invalid_content_length(#[case] invalid_value: &str) {\n        init_log();\n        let input = format!(\n            \"HTTP/1.1 200 OK\\r\\nServer: test\\r\\nContent-Length: {}\\r\\n\\r\\n\",\n            invalid_value\n        );\n        let mock_io = Builder::new().read(input.as_bytes()).build();\n        let mut http_stream = HttpSession::new(Box::new(mock_io));\n        // read_response calls validate_response internally, so it should fail here\n        let res = http_stream.read_response().await;\n        assert!(res.is_err());\n        assert_eq!(res.unwrap_err().etype(), &ErrorType::InvalidHTTPHeader);\n    }\n\n    #[tokio::test]\n    async fn allow_invalid_content_length_close_delimited_when_configured() {\n        init_log();\n        let input_header = b\"HTTP/1.1 200 OK\\r\\nServer: test\\r\\nContent-Length: abc\\r\\n\\r\\n\";\n        let input_body = b\"abc\";\n        let input_close = b\"\";\n        let mock_io = Builder::new()\n            .read(&input_header[..])\n            .read(&input_body[..])\n            .read(&input_close[..])\n            .build();\n        let mut http_stream = HttpSession::new(Box::new(mock_io));\n        let mut peer_options = PeerOptions::new();\n        peer_options.allow_h1_response_invalid_content_length = true;\n        http_stream.set_allow_h1_response_invalid_content_length(\n            peer_options.allow_h1_response_invalid_content_length,\n        );\n\n        let res = http_stream.read_response().await;\n        assert!(res.is_ok());\n        let body = http_stream.read_body_ref().await.unwrap().unwrap();\n        assert_eq!(body, input_body);\n        assert_eq!(\n            http_stream.body_reader.body_state,\n            ParseState::UntilClose(3)\n        );\n        let body = http_stream.read_body_ref().await.unwrap();\n        assert!(body.is_none());\n        assert_eq!(http_stream.body_reader.body_state, ParseState::Complete(3));\n    }\n\n    #[rstest]\n    #[case::valid_zero(\"0\")]\n    #[case::valid_small(\"123\")]\n    #[case::valid_large(\"999999\")]\n    #[tokio::test]\n    async fn validate_response_accepts_valid_content_length(#[case] valid_value: &str) {\n        init_log();\n        let input = format!(\n            \"HTTP/1.1 200 OK\\r\\nServer: test\\r\\nContent-Length: {}\\r\\n\\r\\n\",\n            valid_value\n        );\n        let mock_io = Builder::new().read(input.as_bytes()).build();\n        let mut http_stream = HttpSession::new(Box::new(mock_io));\n        let res = http_stream.read_response().await;\n        assert!(res.is_ok());\n    }\n\n    #[tokio::test]\n    async fn validate_response_accepts_no_content_length() {\n        init_log();\n        let input = b\"HTTP/1.1 200 OK\\r\\nServer: test\\r\\n\\r\\n\";\n        let mock_io = Builder::new().read(&input[..]).build();\n        let mut http_stream = HttpSession::new(Box::new(mock_io));\n        let res = http_stream.read_response().await;\n        assert!(res.is_ok());\n    }\n\n    #[rstest]\n    #[case(None, None, None)]\n    #[case(Some(\"transfer-encoding\"), None, None)]\n    #[case(Some(\"transfer-encoding\"), Some(\"CONTENT-LENGTH\"), Some(\"4\"))]\n    #[case(Some(\"TRANSFER-ENCODING\"), Some(\"CONTENT-LENGTH\"), Some(\"4\"))]\n    #[case(Some(\"TRANSFER-ENCODING\"), None, None)]\n    #[case(None, Some(\"CONTENT-LENGTH\"), Some(\"4\"))]\n    #[case(Some(\"TRANSFER-ENCODING\"), Some(\"content-length\"), Some(\"4\"))]\n    #[case(None, Some(\"content-length\"), Some(\"4\"))]\n    #[case(Some(\"TRANSFER-ENCODING\"), Some(\"CONTENT-LENGTH\"), Some(\"abc\"))]\n    #[tokio::test]\n    async fn response_transfer_encoding_and_content_length_handling(\n        #[case] transfer_encoding_header: Option<&str>,\n        #[case] content_length_header: Option<&str>,\n        #[case] content_length_value: Option<&str>,\n    ) {\n        init_log();\n        let input1 = b\"HTTP/1.1 200 OK\\r\\n\";\n        let mut input2 = \"Server: test\\r\\n\".to_owned();\n\n        if let Some(transfer_encoding) = transfer_encoding_header {\n            input2 += &format!(\"{transfer_encoding}: chunked\\r\\n\");\n        }\n        if let Some(content_length) = content_length_header {\n            let value = content_length_value.unwrap_or(\"4\");\n            input2 += &format!(\"{content_length}: {value}\\r\\n\")\n        }\n\n        input2 += \"\\r\\n\";\n        let mock_io = Builder::new()\n            .read(&input1[..])\n            .read(input2.as_bytes())\n            .build();\n        let mut http_stream = HttpSession::new(Box::new(mock_io));\n        let _ = http_stream.read_response().await.unwrap();\n\n        match (content_length_header, transfer_encoding_header) {\n            (Some(_) | None, Some(_)) => {\n                assert!(http_stream.get_header(header::TRANSFER_ENCODING).is_some());\n                assert!(http_stream.get_header(header::CONTENT_LENGTH).is_none());\n            }\n            (Some(_), None) => {\n                assert!(http_stream.get_header(header::TRANSFER_ENCODING).is_none());\n                assert!(http_stream.get_header(header::CONTENT_LENGTH).is_some());\n            }\n            _ => {\n                assert!(http_stream.get_header(header::CONTENT_LENGTH).is_none());\n                assert!(http_stream.get_header(header::TRANSFER_ENCODING).is_none());\n            }\n        }\n    }\n\n    #[tokio::test]\n    #[should_panic(expected = \"There is still data left to write.\")]\n    async fn write_timeout() {\n        let wire = b\"GET /test HTTP/1.1\\r\\nFoo: Bar\\r\\n\\r\\n\";\n        let mock_io = Builder::new()\n            .wait(Duration::from_secs(2))\n            .write(wire)\n            .build();\n        let mut http_stream = HttpSession::new(Box::new(mock_io));\n        http_stream.write_timeout = Some(Duration::from_secs(1));\n        let mut new_request = RequestHeader::build(\"GET\", b\"/test\", None).unwrap();\n        new_request.insert_header(\"Foo\", \"Bar\").unwrap();\n        let res = http_stream\n            .write_request_header(Box::new(new_request))\n            .await;\n        assert_eq!(res.unwrap_err().etype(), &ErrorType::WriteTimedout);\n    }\n\n    #[tokio::test]\n    #[should_panic(expected = \"There is still data left to write.\")]\n    async fn write_body_timeout() {\n        // Test needs Content-Length header to actually attempt to write body\n        let header = b\"POST /test HTTP/1.1\\r\\nContent-Length: 3\\r\\n\\r\\n\";\n        let body = b\"abc\";\n        let mock_io = Builder::new()\n            .write(&header[..])\n            .wait(Duration::from_secs(2))\n            .write(&body[..])\n            .build();\n        let mut http_stream = HttpSession::new(Box::new(mock_io));\n        http_stream.write_timeout = Some(Duration::from_secs(1));\n\n        let mut new_request = RequestHeader::build(\"POST\", b\"/test\", None).unwrap();\n        new_request.insert_header(\"Content-Length\", \"3\").unwrap();\n        http_stream\n            .write_request_header(Box::new(new_request))\n            .await\n            .unwrap();\n        let res = http_stream.write_body(body).await;\n        assert_eq!(res.unwrap_err().etype(), &WriteTimedout);\n    }\n\n    #[cfg(feature = \"patched_http1\")]\n    #[tokio::test]\n    async fn write_invalid_path() {\n        let wire = b\"GET /\\x01\\xF0\\x90\\x80 HTTP/1.1\\r\\nFoo: Bar\\r\\n\\r\\n\";\n        let mock_io = Builder::new().write(wire).build();\n        let mut http_stream = HttpSession::new(Box::new(mock_io));\n        let mut new_request = RequestHeader::build(\"GET\", b\"/\\x01\\xF0\\x90\\x80\", None).unwrap();\n        new_request.insert_header(\"Foo\", \"Bar\").unwrap();\n        let n = http_stream\n            .write_request_header(Box::new(new_request))\n            .await\n            .unwrap();\n        assert_eq!(wire.len(), n);\n    }\n\n    #[tokio::test]\n    async fn read_informational() {\n        init_log();\n        let input1 = b\"HTTP/1.1 100 Continue\\r\\n\\r\\n\";\n        let input2 = b\"HTTP/1.1 204 OK\\r\\nServer: pingora\\r\\n\\r\\n\";\n        let mock_io = Builder::new().read(&input1[..]).read(&input2[..]).build();\n        let mut http_stream = HttpSession::new(Box::new(mock_io));\n\n        // read 100 header first\n        let task = http_stream.read_response_task().await.unwrap();\n        match task {\n            HttpTask::Header(h, eob) => {\n                assert_eq!(h.status, 100);\n                assert!(!eob);\n            }\n            _ => {\n                panic!(\"task should be header\")\n            }\n        }\n        // read 200 header next\n        let task = http_stream.read_response_task().await.unwrap();\n        match task {\n            HttpTask::Header(h, eob) => {\n                assert_eq!(h.status, 204);\n                assert!(eob);\n            }\n            _ => {\n                panic!(\"task should be header\")\n            }\n        }\n    }\n\n    #[tokio::test]\n    async fn read_informational_combined_with_final() {\n        init_log();\n        let input = b\"HTTP/1.1 100 Continue\\r\\n\\r\\nHTTP/1.1 200 OK\\r\\nServer: pingora\\r\\nContent-Length: 3\\r\\n\\r\\n\";\n        let body = b\"abc\";\n        let mock_io = Builder::new().read(&input[..]).read(&body[..]).build();\n        let mut http_stream = HttpSession::new(Box::new(mock_io));\n\n        // read 100 header first\n        let task = http_stream.read_response_task().await.unwrap();\n        match task {\n            HttpTask::Header(h, eob) => {\n                assert_eq!(h.status, 100);\n                assert!(!eob);\n            }\n            _ => {\n                panic!(\"task should be header\")\n            }\n        }\n        // read 200 header next\n        let task = http_stream.read_response_task().await.unwrap();\n        match task {\n            HttpTask::Header(h, eob) => {\n                assert_eq!(h.status, 200);\n                assert!(!eob);\n            }\n            _ => {\n                panic!(\"task should be header\")\n            }\n        }\n        // read body next\n        let task = http_stream.read_response_task().await.unwrap();\n        match task {\n            HttpTask::Body(b, eob) => {\n                assert_eq!(b.unwrap(), &body[..]);\n                assert!(eob);\n            }\n            _ => {\n                panic!(\"task {task:?} should be body\")\n            }\n        }\n    }\n\n    #[tokio::test]\n    async fn read_informational_multiple_combined_with_final() {\n        init_log();\n        let input = b\"HTTP/1.1 100 Continue\\r\\n\\r\\nHTTP/1.1 103 Early Hints\\r\\n\\r\\nHTTP/1.1 204 No Content\\r\\nServer: pingora\\r\\n\\r\\n\";\n        let mock_io = Builder::new().read(&input[..]).build();\n        let mut http_stream = HttpSession::new(Box::new(mock_io));\n\n        // read 100 header first\n        let task = http_stream.read_response_task().await.unwrap();\n        match task {\n            HttpTask::Header(h, eob) => {\n                assert_eq!(h.status, 100);\n                assert!(!eob);\n            }\n            _ => {\n                panic!(\"task should be header\")\n            }\n        }\n\n        // then read 103 header\n        let task = http_stream.read_response_task().await.unwrap();\n        match task {\n            HttpTask::Header(h, eob) => {\n                assert_eq!(h.status, 103);\n                assert!(!eob);\n            }\n            _ => {\n                panic!(\"task should be header\")\n            }\n        }\n\n        // finally read 200 header\n        let task = http_stream.read_response_task().await.unwrap();\n        match task {\n            HttpTask::Header(h, eob) => {\n                assert_eq!(h.status, 204);\n                assert!(eob);\n            }\n            _ => {\n                panic!(\"task should be header\")\n            }\n        }\n    }\n\n    #[tokio::test]\n    async fn read_informational_then_keepalive_response() {\n        init_log();\n        // Test that after reading an informational response (100 Continue),\n        // keepalive still works properly\n        let wire = b\"GET / HTTP/1.1\\r\\n\\r\\n\";\n        let input1 = b\"HTTP/1.1 100 Continue\\r\\n\\r\\n\";\n        let input2 = b\"HTTP/1.1 200 OK\\r\\nContent-Length: 13\\r\\n\\r\\n\"; // Proper Content-Length\n        let body = b\"response body\";\n\n        let mock_io = Builder::new()\n            .write(&wire[..])\n            .read(&input1[..])\n            .read(&input2[..])\n            .read(&body[..])\n            .build();\n        let mut http_stream = HttpSession::new(Box::new(mock_io));\n\n        // Write request\n        let new_request = RequestHeader::build(\"GET\", b\"/\", None).unwrap();\n        http_stream\n            .write_request_header(Box::new(new_request))\n            .await\n            .unwrap();\n\n        // Read 100 Continue\n        let task = http_stream.read_response_task().await.unwrap();\n        match task {\n            HttpTask::Header(h, eob) => {\n                assert_eq!(h.status, 100);\n                assert!(!eob);\n            }\n            _ => {\n                panic!(\"task should be informational header\")\n            }\n        }\n\n        // Read final 200 OK header\n        let task = http_stream.read_response_task().await.unwrap();\n        match task {\n            HttpTask::Header(h, eob) => {\n                assert_eq!(h.status, 200);\n                assert!(!eob); // Should not be end of body yet\n            }\n            _ => {\n                panic!(\"task should be final header\")\n            }\n        }\n\n        // Read body\n        let task = http_stream.read_response_task().await.unwrap();\n        match task {\n            HttpTask::Body(b, eob) => {\n                assert_eq!(b.unwrap(), &body[..]);\n                assert!(eob); // EOF - body is complete\n            }\n            _ => {\n                panic!(\"task {task:?} should be body\")\n            }\n        }\n\n        assert_eq!(http_stream.body_reader.body_state, ParseState::Complete(13));\n\n        // Keepalive should be enabled for properly-framed HTTP/1.1\n        http_stream.respect_keepalive();\n        assert!(http_stream.will_keepalive());\n    }\n\n    #[tokio::test]\n    async fn init_body_for_upgraded_req() {\n        let wire =\n            b\"GET / HTTP/1.1\\r\\nConnection: Upgrade\\r\\nUpgrade: WS\\r\\nContent-Length: 0\\r\\n\\r\\n\";\n        let input1 = b\"HTTP/1.1 101 Switching Protocols\\r\\n\\r\\n\";\n        let input2 = b\"PAYLOAD\";\n        let ws_data = b\"data\";\n\n        let mock_io = Builder::new()\n            .write(wire)\n            .read(&input1[..])\n            .write(&ws_data[..])\n            .read(&input2[..])\n            .build();\n        let mut http_stream = HttpSession::new(Box::new(mock_io));\n        let mut new_request = RequestHeader::build(\"GET\", b\"/\", None).unwrap();\n        new_request.insert_header(\"Connection\", \"Upgrade\").unwrap();\n        new_request.insert_header(\"Upgrade\", \"WS\").unwrap();\n        new_request.insert_header(\"Content-Length\", \"0\").unwrap();\n        let _ = http_stream\n            .write_request_header(Box::new(new_request))\n            .await\n            .unwrap();\n        assert_eq!(\n            http_stream.body_writer.body_mode,\n            BodyMode::ContentLength(0, 0)\n        );\n        assert!(http_stream.body_writer.finished());\n\n        let task = http_stream.read_response_task().await.unwrap();\n        match task {\n            HttpTask::Header(h, eob) => {\n                assert_eq!(h.status, 101);\n                assert!(!eob);\n            }\n            _ => {\n                panic!(\"task should be header\")\n            }\n        }\n        // changed body mode\n        assert_eq!(\n            http_stream.body_reader.body_state,\n            ParseState::UntilClose(0)\n        );\n        // request writer will be explicitly initialized in a separate call\n        assert!(http_stream.body_writer.finished());\n        http_stream.maybe_upgrade_body_writer();\n\n        assert!(!http_stream.body_writer.finished());\n        assert_eq!(http_stream.body_writer.body_mode, BodyMode::UntilClose(0));\n\n        http_stream.write_body(&ws_data[..]).await.unwrap();\n        // read WS\n        let task = http_stream.read_response_task().await.unwrap();\n        match task {\n            HttpTask::UpgradedBody(b, eob) => {\n                assert_eq!(b.unwrap(), &input2[..]);\n                assert!(!eob);\n            }\n            _ => {\n                panic!(\"task should be upgraded body\")\n            }\n        }\n    }\n\n    #[tokio::test]\n    async fn init_preread_body_for_upgraded_req() {\n        let wire =\n            b\"GET / HTTP/1.1\\r\\nConnection: Upgrade\\r\\nUpgrade: WS\\r\\nContent-Length: 0\\r\\n\\r\\n\";\n        let input = b\"HTTP/1.1 101 Switching Protocols\\r\\n\\r\\nPAYLOAD\";\n        let ws_data = b\"data\";\n\n        let mock_io = Builder::new()\n            .write(wire)\n            .read(&input[..])\n            .write(&ws_data[..])\n            .build();\n        let mut http_stream = HttpSession::new(Box::new(mock_io));\n        let mut new_request = RequestHeader::build(\"GET\", b\"/\", None).unwrap();\n        new_request.insert_header(\"Connection\", \"Upgrade\").unwrap();\n        new_request.insert_header(\"Upgrade\", \"WS\").unwrap();\n        new_request.insert_header(\"Content-Length\", \"0\").unwrap();\n        let _ = http_stream\n            .write_request_header(Box::new(new_request))\n            .await\n            .unwrap();\n        assert_eq!(\n            http_stream.body_writer.body_mode,\n            BodyMode::ContentLength(0, 0)\n        );\n        assert!(http_stream.body_writer.finished());\n\n        let task = http_stream.read_response_task().await.unwrap();\n        match task {\n            HttpTask::Header(h, eob) => {\n                assert_eq!(h.status, 101);\n                assert!(!eob);\n            }\n            _ => {\n                panic!(\"task should be header\")\n            }\n        }\n        // changed body mode\n        assert_eq!(\n            http_stream.body_reader.body_state,\n            ParseState::UntilClose(0)\n        );\n        // request writer will be explicitly initialized in a separate call\n        assert!(http_stream.body_writer.finished());\n        http_stream.maybe_upgrade_body_writer();\n\n        assert!(!http_stream.body_writer.finished());\n        assert_eq!(http_stream.body_writer.body_mode, BodyMode::UntilClose(0));\n\n        http_stream.write_body(&ws_data[..]).await.unwrap();\n        // read WS\n        let task = http_stream.read_response_task().await.unwrap();\n        match task {\n            HttpTask::UpgradedBody(b, eob) => {\n                assert_eq!(b.unwrap(), &b\"PAYLOAD\"[..]);\n                assert!(!eob);\n            }\n            _ => {\n                panic!(\"task should be upgraded body\")\n            }\n        }\n    }\n\n    #[tokio::test]\n    async fn read_body_eos_after_upgrade() {\n        let wire =\n            b\"GET / HTTP/1.1\\r\\nConnection: Upgrade\\r\\nUpgrade: WS\\r\\nContent-Length: 10\\r\\n\\r\\n\";\n        let input1 = b\"HTTP/1.1 101 Switching Protocols\\r\\n\\r\\n\";\n        let input2 = b\"PAYLOAD\";\n        let body_data = b\"0123456789\";\n        let ws_data = b\"data\";\n\n        let mock_io = Builder::new()\n            .write(wire)\n            .read(&input1[..])\n            .write(&body_data[..])\n            .read(&input2[..])\n            .write(&ws_data[..])\n            .build();\n\n        let mut http_stream = HttpSession::new(Box::new(mock_io));\n        let mut new_request = RequestHeader::build(\"GET\", b\"/\", None).unwrap();\n        new_request.insert_header(\"Connection\", \"Upgrade\").unwrap();\n        new_request.insert_header(\"Upgrade\", \"WS\").unwrap();\n        new_request.insert_header(\"Content-Length\", \"10\").unwrap();\n        let _ = http_stream\n            .write_request_header(Box::new(new_request))\n            .await\n            .unwrap();\n        assert_eq!(\n            http_stream.body_writer.body_mode,\n            BodyMode::ContentLength(10, 0)\n        );\n        assert!(!http_stream.body_writer.finished());\n\n        let task = http_stream.read_response_task().await.unwrap();\n        match task {\n            HttpTask::Header(h, eob) => {\n                assert_eq!(h.status, 101);\n                assert!(!eob);\n            }\n            _ => {\n                panic!(\"task should be header\")\n            }\n        }\n        // changed body mode\n        assert_eq!(\n            http_stream.body_reader.body_state,\n            ParseState::UntilClose(0)\n        );\n\n        // write regular request payload\n        http_stream.write_body(&body_data[..]).await.unwrap();\n        http_stream.finish_body().await.unwrap();\n\n        // we should still be able to read more response body\n        // read WS\n        let task = http_stream.read_response_task().await.unwrap();\n        match task {\n            HttpTask::UpgradedBody(b, eob) => {\n                assert_eq!(b.unwrap(), &input2[..]);\n                assert!(!eob);\n            }\n            t => {\n                panic!(\"task {t:?} should be upgraded body\")\n            }\n        }\n\n        // body IS finished, prior to upgrade on the downstream side\n        assert!(http_stream.body_writer.finished());\n        http_stream.maybe_upgrade_body_writer();\n\n        assert!(!http_stream.body_writer.finished());\n        assert_eq!(http_stream.body_writer.body_mode, BodyMode::UntilClose(0));\n\n        http_stream.write_body(&ws_data[..]).await.unwrap();\n        assert_eq!(http_stream.body_writer.body_mode, BodyMode::UntilClose(4));\n        http_stream.finish_body().await.unwrap();\n    }\n\n    #[tokio::test]\n    async fn read_switching_protocol() {\n        init_log();\n\n        let wire =\n            b\"GET / HTTP/1.1\\r\\nConnection: Upgrade\\r\\nUpgrade: WS\\r\\nContent-Length: 0\\r\\n\\r\\n\";\n        let input1 = b\"HTTP/1.1 101 Continue\\r\\n\\r\\n\";\n        let input2 = b\"PAYLOAD\";\n\n        let mock_io = Builder::new()\n            .write(&wire[..])\n            .read(&input1[..])\n            .read(&input2[..])\n            .build();\n\n        let mut http_stream = HttpSession::new(Box::new(mock_io));\n        let mut new_request = RequestHeader::build(\"GET\", b\"/\", None).unwrap();\n        new_request.insert_header(\"Connection\", \"Upgrade\").unwrap();\n        new_request.insert_header(\"Upgrade\", \"WS\").unwrap();\n        new_request.insert_header(\"Content-Length\", \"0\").unwrap();\n        let _ = http_stream\n            .write_request_header(Box::new(new_request))\n            .await\n            .unwrap();\n        assert_eq!(\n            http_stream.body_writer.body_mode,\n            BodyMode::ContentLength(0, 0)\n        );\n        assert!(http_stream.body_writer.finished());\n\n        // read 100 header first\n        let task = http_stream.read_response_task().await.unwrap();\n        match task {\n            HttpTask::Header(h, eob) => {\n                assert_eq!(h.status, 101);\n                assert!(!eob);\n            }\n            _ => {\n                panic!(\"task should be header\")\n            }\n        }\n        // read body\n        let task = http_stream.read_response_task().await.unwrap();\n        match task {\n            HttpTask::UpgradedBody(b, eob) => {\n                assert_eq!(b.unwrap(), &input2[..]);\n                assert!(!eob);\n            }\n            _ => {\n                panic!(\"task should be upgraded body\")\n            }\n        }\n        // read body\n        let task = http_stream.read_response_task().await.unwrap();\n        match task {\n            HttpTask::UpgradedBody(b, eob) => {\n                assert!(b.is_none());\n                assert!(eob);\n            }\n            _ => {\n                panic!(\"task should be body with end of stream\")\n            }\n        }\n    }\n\n    // Note: in debug mode, due to from_maybe_shared_unchecked() still tries to validate headers\n    // values, so the code has to replace CRLF with whitespaces. In release mode, the CRLF is\n    // reserved\n    #[tokio::test]\n    async fn read_obsolete_multiline_headers() {\n        init_log();\n        let input = b\"HTTP/1.1 200 OK\\r\\nServer : pingora\\r\\n Foo: Bar\\r\\n\\r\\n\";\n        let mock_io = Builder::new().read(&input[..]).build();\n        let mut http_stream = HttpSession::new(Box::new(mock_io));\n        let res = http_stream.read_response().await;\n        assert_eq!(input.len(), res.unwrap());\n\n        assert_eq!(1, http_stream.resp_header().unwrap().headers.len());\n        assert_eq!(\n            http_stream.get_header(\"Server\").unwrap(),\n            \"pingora   Foo: Bar\"\n        );\n\n        let input = b\"HTTP/1.1 200 OK\\r\\nServer : pingora\\r\\n\\t  Fizz: Buzz\\r\\n\\r\\n\";\n        let mock_io = Builder::new().read(&input[..]).build();\n        let mut http_stream = HttpSession::new(Box::new(mock_io));\n        let res = http_stream.read_response().await;\n        assert_eq!(input.len(), res.unwrap());\n        assert_eq!(1, http_stream.resp_header().unwrap().headers.len());\n        assert_eq!(\n            http_stream.get_header(\"Server\").unwrap(),\n            \"pingora  \\t  Fizz: Buzz\"\n        );\n    }\n\n    #[cfg(feature = \"patched_http1\")]\n    #[tokio::test]\n    async fn read_headers_skip_invalid_line() {\n        init_log();\n        let input = b\"HTTP/1.1 200 OK\\r\\n;\\r\\nFoo: Bar\\r\\n\\r\\n\";\n        let mock_io = Builder::new().read(&input[..]).build();\n        let mut http_stream = HttpSession::new(Box::new(mock_io));\n        let res = http_stream.read_response().await;\n        assert_eq!(input.len(), res.unwrap());\n        assert_eq!(1, http_stream.resp_header().unwrap().headers.len());\n        assert_eq!(http_stream.get_header(\"Foo\").unwrap(), \"Bar\");\n    }\n\n    #[tokio::test]\n    async fn read_keepalive_headers() {\n        init_log();\n\n        async fn build_resp_with_keepalive(conn: &str) -> HttpSession {\n            // Include Content-Length to avoid triggering defense-in-depth close-delimited check\n            let input =\n                format!(\"HTTP/1.1 200 OK\\r\\nConnection: {conn}\\r\\nContent-Length: 0\\r\\n\\r\\n\");\n            let mock_io = Builder::new().read(input.as_bytes()).build();\n            let mut http_stream = HttpSession::new(Box::new(mock_io));\n            let res = http_stream.read_response().await;\n            assert_eq!(input.len(), res.unwrap());\n            http_stream.respect_keepalive();\n            http_stream\n        }\n\n        assert_eq!(\n            build_resp_with_keepalive(\"close\").await.keepalive_timeout,\n            KeepaliveStatus::Off\n        );\n\n        assert_eq!(\n            build_resp_with_keepalive(\"keep-alive\")\n                .await\n                .keepalive_timeout,\n            KeepaliveStatus::Infinite\n        );\n\n        assert_eq!(\n            build_resp_with_keepalive(\"foo\").await.keepalive_timeout,\n            KeepaliveStatus::Infinite\n        );\n\n        assert_eq!(\n            build_resp_with_keepalive(\"upgrade,close\")\n                .await\n                .keepalive_timeout,\n            KeepaliveStatus::Off\n        );\n\n        assert_eq!(\n            build_resp_with_keepalive(\"upgrade, close\")\n                .await\n                .keepalive_timeout,\n            KeepaliveStatus::Off\n        );\n\n        assert_eq!(\n            build_resp_with_keepalive(\"Upgrade, close\")\n                .await\n                .keepalive_timeout,\n            KeepaliveStatus::Off\n        );\n\n        assert_eq!(\n            build_resp_with_keepalive(\"Upgrade,close\")\n                .await\n                .keepalive_timeout,\n            KeepaliveStatus::Off\n        );\n\n        assert_eq!(\n            build_resp_with_keepalive(\"close,upgrade\")\n                .await\n                .keepalive_timeout,\n            KeepaliveStatus::Off\n        );\n\n        assert_eq!(\n            build_resp_with_keepalive(\"close, upgrade\")\n                .await\n                .keepalive_timeout,\n            KeepaliveStatus::Off\n        );\n\n        assert_eq!(\n            build_resp_with_keepalive(\"close,Upgrade\")\n                .await\n                .keepalive_timeout,\n            KeepaliveStatus::Off\n        );\n\n        assert_eq!(\n            build_resp_with_keepalive(\"close, Upgrade\")\n                .await\n                .keepalive_timeout,\n            KeepaliveStatus::Off\n        );\n\n        async fn build_resp_with_keepalive_values(keep_alive: &str) -> HttpSession {\n            let input = format!(\"HTTP/1.1 200 OK\\r\\nKeep-Alive: {keep_alive}\\r\\n\\r\\n\");\n            let mock_io = Builder::new().read(input.as_bytes()).build();\n            let mut http_stream = HttpSession::new(Box::new(mock_io));\n            let res = http_stream.read_response().await;\n            assert_eq!(input.len(), res.unwrap());\n            http_stream.respect_keepalive();\n            http_stream\n        }\n\n        assert_eq!(\n            build_resp_with_keepalive_values(\"timeout=5, max=1000\")\n                .await\n                .get_keepalive_values(),\n            (Some(5), Some(1000))\n        );\n\n        assert_eq!(\n            build_resp_with_keepalive_values(\"max=1000, timeout=5\")\n                .await\n                .get_keepalive_values(),\n            (Some(5), Some(1000))\n        );\n\n        assert_eq!(\n            build_resp_with_keepalive_values(\" timeout = 5, max = 1000 \")\n                .await\n                .get_keepalive_values(),\n            (Some(5), Some(1000))\n        );\n\n        assert_eq!(\n            build_resp_with_keepalive_values(\"timeout=5\")\n                .await\n                .get_keepalive_values(),\n            (Some(5), None)\n        );\n\n        assert_eq!(\n            build_resp_with_keepalive_values(\"max=1000\")\n                .await\n                .get_keepalive_values(),\n            (None, Some(1000))\n        );\n\n        assert_eq!(\n            build_resp_with_keepalive_values(\"a=b\")\n                .await\n                .get_keepalive_values(),\n            (None, None)\n        );\n\n        assert_eq!(\n            build_resp_with_keepalive_values(\"\")\n                .await\n                .get_keepalive_values(),\n            (None, None)\n        );\n    }\n\n    /* Note: body tests are covered in server.rs */\n\n    #[tokio::test]\n    async fn test_http10_response_with_transfer_encoding_disables_keepalive() {\n        // Transfer-Encoding in HTTP/1.0 response requires connection close\n        let input = b\"HTTP/1.0 200 OK\\r\\n\\\nTransfer-Encoding: chunked\\r\\n\\\nConnection: keep-alive\\r\\n\\\n\\r\\n\\\n5\\r\\n\\\nhello\\r\\n\\\n0\\r\\n\\\n\\r\\n\";\n        let mock_io = Builder::new().read(&input[..]).build();\n        let mut http_stream = HttpSession::new(Box::new(mock_io));\n        http_stream.read_response().await.unwrap();\n        http_stream.respect_keepalive();\n\n        // Keepalive must be disabled even if Connection: keep-alive header present\n        assert!(!http_stream.will_keepalive());\n        assert_eq!(http_stream.keepalive_timeout, KeepaliveStatus::Off);\n    }\n\n    #[tokio::test]\n    async fn test_http11_response_with_transfer_encoding_allows_keepalive() {\n        // HTTP/1.1 with Transfer-Encoding should allow keepalive (contrast with HTTP/1.0)\n        let input = b\"HTTP/1.1 200 OK\\r\\n\\\nTransfer-Encoding: chunked\\r\\n\\\n\\r\\n\\\n5\\r\\n\\\nhello\\r\\n\\\n0\\r\\n\\\n\\r\\n\";\n        let mock_io = Builder::new().read(&input[..]).build();\n        let mut http_stream = HttpSession::new(Box::new(mock_io));\n        http_stream.read_response().await.unwrap();\n        http_stream.respect_keepalive();\n\n        // HTTP/1.1 should allow keepalive by default\n        assert!(http_stream.will_keepalive());\n    }\n\n    #[tokio::test]\n    async fn test_response_multiple_transfer_encoding_headers() {\n        init_log();\n        // Multiple TE headers should be treated as comma-separated\n        let input = b\"HTTP/1.1 200 OK\\r\\n\\\nTransfer-Encoding: gzip\\r\\n\\\nTransfer-Encoding: chunked\\r\\n\\\n\\r\\n\\\n5\\r\\n\\\nhello\\r\\n\\\n0\\r\\n\\\n\\r\\n\";\n\n        let mock_io = Builder::new().read(&input[..]).build();\n        let mut http_stream = HttpSession::new(Box::new(mock_io));\n        http_stream.read_response().await.unwrap();\n\n        // Should correctly identify chunked encoding from last header\n        assert!(http_stream.is_chunked_encoding());\n\n        // Verify body can be read correctly\n        let body = http_stream.read_body_bytes().await.unwrap();\n        assert_eq!(body.as_ref().unwrap().as_ref(), b\"hello\");\n        http_stream.finish_body().await.unwrap();\n    }\n\n    #[tokio::test]\n    async fn test_response_multiple_te_headers_chunked_not_last() {\n        init_log();\n        // Chunked in first header but not last - should NOT be chunked\n        let input = b\"HTTP/1.1 200 OK\\r\\n\\\nTransfer-Encoding: chunked\\r\\n\\\nTransfer-Encoding: identity\\r\\n\\\nContent-Length: 5\\r\\n\\\n\\r\\n\\\nhello\";\n\n        let mock_io = Builder::new().read(&input[..]).build();\n        let mut http_stream = HttpSession::new(Box::new(mock_io));\n        http_stream.read_response().await.unwrap();\n\n        // Should NOT be chunked - identity is final encoding\n        assert!(!http_stream.is_chunked_encoding());\n    }\n\n    #[test]\n    fn test_is_chunked_encoding_before_response() {\n        // Test that is_chunked_encoding returns false when no response received yet\n        let mock_io = Builder::new().build();\n        let http_stream = HttpSession::new(Box::new(mock_io));\n\n        // Should return false when no response header exists yet\n        assert!(!http_stream.is_chunked_encoding());\n    }\n\n    #[tokio::test]\n    async fn write_request_body_implicit_zero_content_length() {\n        init_log();\n        let header = b\"POST /test HTTP/1.1\\r\\n\\r\\n\";\n        let mock_io = Builder::new().write(&header[..]).build();\n        let mut http_stream = HttpSession::new(Box::new(mock_io));\n\n        let new_request = RequestHeader::build(\"POST\", b\"/test\", None).unwrap();\n        http_stream\n            .write_request_header(Box::new(new_request))\n            .await\n            .unwrap();\n\n        assert_eq!(\n            http_stream.body_writer.body_mode,\n            BodyMode::ContentLength(0, 0)\n        );\n    }\n\n    #[tokio::test]\n    async fn write_request_body_with_content_length() {\n        init_log();\n        let header = b\"POST /test HTTP/1.1\\r\\nContent-Length: 3\\r\\n\\r\\n\";\n        let body = b\"abc\";\n        let mock_io = Builder::new().write(&header[..]).write(&body[..]).build();\n        let mut http_stream = HttpSession::new(Box::new(mock_io));\n\n        let mut new_request = RequestHeader::build(\"POST\", b\"/test\", None).unwrap();\n        new_request.insert_header(\"Content-Length\", \"3\").unwrap();\n        http_stream\n            .write_request_header(Box::new(new_request))\n            .await\n            .unwrap();\n\n        assert_eq!(\n            http_stream.body_writer.body_mode,\n            BodyMode::ContentLength(3, 0)\n        );\n\n        http_stream.write_body(body).await.unwrap();\n        assert_eq!(\n            http_stream.body_writer.body_mode,\n            BodyMode::ContentLength(3, 3)\n        );\n    }\n\n    #[tokio::test]\n    async fn close_delimited_response_explicitly_disables_keepalive() {\n        init_log();\n        // Defense-in-depth: if we read a close-delimited response body,\n        // keepalive should be disabled\n        let wire = b\"GET / HTTP/1.1\\r\\n\\r\\n\";\n        let input_header = b\"HTTP/1.1 200 OK\\r\\n\\r\\n\";\n        let input_body = b\"abc\";\n        let input_close = b\"\"; // simulating close\n        let mock_io = Builder::new()\n            .write(&wire[..])\n            .read(&input_header[..])\n            .read(&input_body[..])\n            .read(&input_close[..])\n            .build();\n        let mut http_stream = HttpSession::new(Box::new(mock_io));\n\n        // Write request first\n        let new_request = RequestHeader::build(\"GET\", b\"/\", None).unwrap();\n        http_stream\n            .write_request_header(Box::new(new_request))\n            .await\n            .unwrap();\n\n        // Read response\n        http_stream.read_response().await.unwrap();\n\n        // Read the body (this will initialize the body reader)\n        http_stream.read_body_ref().await.unwrap();\n\n        // Body reader should be in UntilClose mode (close-delimited response)\n        assert_eq!(\n            http_stream.body_reader.body_state,\n            ParseState::UntilClose(3)\n        );\n\n        let res2 = http_stream.read_body_ref().await.unwrap();\n        assert!(res2.is_none()); // EOF\n\n        // Body should now be Complete\n        assert_eq!(http_stream.body_reader.body_state, ParseState::Complete(3));\n\n        http_stream.respect_keepalive();\n        assert!(!http_stream.will_keepalive());\n    }\n}\n\n#[cfg(test)]\nmod test_sync {\n    use super::*;\n    use log::error;\n\n    #[test]\n    fn test_request_to_wire() {\n        let mut new_request = RequestHeader::build(\"GET\", b\"/\", None).unwrap();\n        new_request.insert_header(\"Foo\", \"Bar\").unwrap();\n        let wire = http_req_header_to_wire(&new_request).unwrap();\n        let mut headers = [httparse::EMPTY_HEADER; 128];\n        let mut req = httparse::Request::new(&mut headers);\n        let result = req.parse(wire.as_ref());\n        match result {\n            Ok(_) => {}\n            Err(e) => error!(\"{:?}\", e),\n        }\n        assert!(result.unwrap().is_complete());\n        // FIXME: the order is not guaranteed\n        assert_eq!(\"/\", req.path.unwrap());\n        assert_eq!(b\"Foo\", headers[0].name.as_bytes());\n        assert_eq!(b\"Bar\", headers[0].value);\n    }\n}\n"
  },
  {
    "path": "pingora-core/src/protocols/http/v1/common.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! Common functions and constants\n\nuse http::{header, HeaderValue};\nuse log::warn;\nuse pingora_error::{Error, ErrorType::*, Result};\nuse pingora_http::{HMap, RequestHeader, ResponseHeader};\nuse std::str;\nuse std::time::Duration;\n\nuse super::body::BodyWriter;\nuse crate::utils::KVRef;\n\npub(super) const MAX_HEADERS: usize = 256;\n\npub(super) const INIT_HEADER_BUF_SIZE: usize = 4096;\npub(super) const MAX_HEADER_SIZE: usize = 1048575;\n\npub(crate) const BODY_BUF_LIMIT: usize = 1024 * 64;\n\npub const CRLF: &[u8; 2] = b\"\\r\\n\";\npub const HEADER_KV_DELIMITER: &[u8; 2] = b\": \";\n\npub(super) enum HeaderParseState {\n    Complete(usize),\n    Partial,\n    Invalid(httparse::Error),\n}\n\n#[derive(Clone, Debug, PartialEq, Eq)]\npub(super) enum KeepaliveStatus {\n    Timeout(Duration),\n    Infinite,\n    Off,\n}\n\nstruct ConnectionValue {\n    keep_alive: bool,\n    upgrade: bool,\n    close: bool,\n}\n\nimpl ConnectionValue {\n    fn new() -> Self {\n        ConnectionValue {\n            keep_alive: false,\n            upgrade: false,\n            close: false,\n        }\n    }\n\n    fn close(mut self) -> Self {\n        self.close = true;\n        self\n    }\n    fn upgrade(mut self) -> Self {\n        self.upgrade = true;\n        self\n    }\n    fn keep_alive(mut self) -> Self {\n        self.keep_alive = true;\n        self\n    }\n}\n\nfn parse_connection_header(value: &[u8]) -> ConnectionValue {\n    // only parse keep-alive, close, and upgrade tokens\n    // https://www.rfc-editor.org/rfc/rfc9110.html#section-7.6.1\n\n    const KEEP_ALIVE: &str = \"keep-alive\";\n    const CLOSE: &str = \"close\";\n    const UPGRADE: &str = \"upgrade\";\n\n    // fast path\n    if value.eq_ignore_ascii_case(CLOSE.as_bytes()) {\n        ConnectionValue::new().close()\n    } else if value.eq_ignore_ascii_case(KEEP_ALIVE.as_bytes()) {\n        ConnectionValue::new().keep_alive()\n    } else if value.eq_ignore_ascii_case(UPGRADE.as_bytes()) {\n        ConnectionValue::new().upgrade()\n    } else {\n        // slow path, parse the connection value\n        let mut close = false;\n        let mut upgrade = false;\n        let value = str::from_utf8(value).unwrap_or(\"\");\n        for token in value\n            .split(',')\n            .map(|s| s.trim())\n            .filter(|&x| !x.is_empty())\n        {\n            if token.eq_ignore_ascii_case(CLOSE) {\n                close = true;\n            } else if token.eq_ignore_ascii_case(UPGRADE) {\n                upgrade = true;\n            }\n            if upgrade && close {\n                return ConnectionValue::new().upgrade().close();\n            }\n        }\n        if close {\n            ConnectionValue::new().close()\n        } else if upgrade {\n            ConnectionValue::new().upgrade()\n        } else {\n            ConnectionValue::new()\n        }\n    }\n}\n\npub(crate) fn init_body_writer_comm(body_writer: &mut BodyWriter, headers: &HMap) {\n    if is_chunked_encoding_from_headers(headers) {\n        // transfer-encoding takes priority over content-length\n        body_writer.init_chunked();\n    } else {\n        let content_length = header_value_content_length(headers.get(http::header::CONTENT_LENGTH));\n        match content_length {\n            Some(length) => {\n                body_writer.init_content_length(length);\n            }\n            None => {\n                /* TODO: 1. connection: keepalive cannot be used,\n                2. mark connection must be closed */\n                body_writer.init_close_delimited();\n            }\n        }\n    }\n}\n\n/// Find the last comma-separated token in a Transfer-Encoding header value.\n/// Takes the literal last token after the last comma, even if empty.\n#[inline]\nfn find_last_te_token(bytes: &[u8]) -> &[u8] {\n    let last_token = bytes\n        .iter()\n        .rposition(|&b| b == b',')\n        .map(|pos| &bytes[pos + 1..])\n        .unwrap_or(bytes);\n\n    last_token.trim_ascii()\n}\n\n/// Check if chunked encoding is the final encoding across all transfer-encoding headers\npub(crate) fn is_chunked_encoding_from_headers(headers: &HMap) -> bool {\n    // Get the last Transfer-Encoding header value\n    let last_te = headers\n        .get_all(http::header::TRANSFER_ENCODING)\n        .into_iter()\n        .next_back();\n\n    let Some(last_header_value) = last_te else {\n        return false;\n    };\n\n    let bytes = last_header_value.as_bytes();\n\n    // Fast path: exact match for \"chunked\"\n    if bytes.eq_ignore_ascii_case(b\"chunked\") {\n        return true;\n    }\n\n    // Slow path: parse comma-separated values\n    find_last_te_token(bytes).eq_ignore_ascii_case(b\"chunked\")\n}\n\npub fn is_upgrade_req(req: &RequestHeader) -> bool {\n    req.version == http::Version::HTTP_11 && req.headers.get(header::UPGRADE).is_some()\n}\n\npub fn is_expect_continue_req(req: &RequestHeader) -> bool {\n    req.version == http::Version::HTTP_11\n        // https://www.rfc-editor.org/rfc/rfc9110#section-10.1.1\n        && req.headers.get(header::EXPECT).is_some_and(|v| {\n            v.as_bytes().eq_ignore_ascii_case(b\"100-continue\")\n        })\n}\n\n// Unlike the upgrade check on request, this function doesn't check the Upgrade or Connection header\n// because when seeing 101, we assume the server accepts to switch protocol.\n// In reality it is not common that some servers don't send all the required headers to establish\n// websocket connections.\npub fn is_upgrade_resp(header: &ResponseHeader) -> bool {\n    header.status == 101 && header.version == http::Version::HTTP_11\n}\n\n#[inline]\npub fn header_value_content_length(\n    header_value: Option<&http::header::HeaderValue>,\n) -> Option<usize> {\n    match header_value {\n        Some(value) => buf_to_content_length(Some(value.as_bytes())).ok().flatten(),\n        None => None,\n    }\n}\n\n#[inline]\npub(super) fn buf_to_content_length(header_value: Option<&[u8]>) -> Result<Option<usize>> {\n    match header_value {\n        Some(buf) => {\n            match str::from_utf8(buf) {\n                // check valid string\n                Ok(str_cl_value) => match str_cl_value.parse::<i64>() {\n                    Ok(cl_length) => {\n                        if cl_length >= 0 {\n                            Ok(Some(cl_length as usize))\n                        } else {\n                            warn!(\"negative content-length header value {cl_length}\");\n                            Error::e_explain(\n                                InvalidHTTPHeader,\n                                format!(\"negative Content-Length header value: {cl_length}\"),\n                            )\n                        }\n                    }\n                    Err(_) => {\n                        warn!(\"invalid content-length header value {str_cl_value}\");\n                        Error::e_explain(\n                            InvalidHTTPHeader,\n                            format!(\"invalid Content-Length header value: {str_cl_value}\"),\n                        )\n                    }\n                },\n                Err(_) => {\n                    warn!(\"invalid content-length header encoding\");\n                    Error::e_explain(InvalidHTTPHeader, \"invalid Content-Length header encoding\")\n                }\n            }\n        }\n        None => Ok(None),\n    }\n}\n\n#[inline]\npub(super) fn is_buf_keepalive(header_value: Option<&HeaderValue>) -> Option<bool> {\n    header_value.and_then(|value| {\n        let value = parse_connection_header(value.as_bytes());\n        if value.keep_alive {\n            Some(true)\n        } else if value.close {\n            Some(false)\n        } else {\n            None\n        }\n    })\n}\n\n#[inline]\npub(super) fn populate_headers(\n    base: usize,\n    header_ref: &mut Vec<KVRef>,\n    headers: &[httparse::Header],\n) -> usize {\n    let mut used_header_index = 0;\n    for header in headers.iter() {\n        if !header.name.is_empty() {\n            header_ref.push(KVRef::new(\n                header.name.as_ptr() as usize - base,\n                header.name.len(),\n                header.value.as_ptr() as usize - base,\n                header.value.len(),\n            ));\n            used_header_index += 1;\n        }\n    }\n    used_header_index\n}\n\n// RFC 7230:\n// If a message is received without Transfer-Encoding and with\n// either multiple Content-Length header fields having differing\n// field-values or a single Content-Length header field having an\n// invalid value, then the message framing is invalid and the\n// recipient MUST treat it as an unrecoverable error.\npub(super) fn check_dup_content_length(headers: &HMap) -> Result<()> {\n    if headers.get(header::TRANSFER_ENCODING).is_some() {\n        // If TE header, ignore CL\n        return Ok(());\n    }\n    let mut cls = headers.get_all(header::CONTENT_LENGTH).into_iter();\n    if cls.next().is_none() {\n        // no CL header is fine.\n        return Ok(());\n    }\n    if cls.next().is_some() {\n        // duplicated CL is bad\n        return crate::Error::e_explain(\n            crate::ErrorType::InvalidHTTPHeader,\n            \"duplicated Content-Length header\",\n        );\n    }\n    Ok(())\n}\n\n#[cfg(test)]\nmod test {\n    use super::*;\n    use http::{\n        header::{CONTENT_LENGTH, TRANSFER_ENCODING},\n        StatusCode, Version,\n    };\n    use rstest::rstest;\n\n    #[test]\n    fn test_check_dup_content_length() {\n        let mut headers = HMap::new();\n\n        assert!(check_dup_content_length(&headers).is_ok());\n\n        headers.append(CONTENT_LENGTH, \"1\".try_into().unwrap());\n        assert!(check_dup_content_length(&headers).is_ok());\n\n        headers.append(CONTENT_LENGTH, \"2\".try_into().unwrap());\n        assert!(check_dup_content_length(&headers).is_err());\n\n        headers.append(TRANSFER_ENCODING, \"chunkeds\".try_into().unwrap());\n        assert!(check_dup_content_length(&headers).is_ok());\n    }\n\n    #[test]\n    fn test_is_upgrade_resp() {\n        let mut response = ResponseHeader::build(StatusCode::SWITCHING_PROTOCOLS, None).unwrap();\n        response.set_version(Version::HTTP_11);\n        response.insert_header(\"Upgrade\", \"websocket\").unwrap();\n        response.insert_header(\"Connection\", \"upgrade\").unwrap();\n        assert!(is_upgrade_resp(&response));\n\n        // wrong http version\n        response.set_version(Version::HTTP_10);\n        response.insert_header(\"Upgrade\", \"websocket\").unwrap();\n        response.insert_header(\"Connection\", \"upgrade\").unwrap();\n        assert!(!is_upgrade_resp(&response));\n\n        // not 101\n        response.set_status(StatusCode::OK).unwrap();\n        response.set_version(Version::HTTP_11);\n        assert!(!is_upgrade_resp(&response));\n    }\n\n    #[test]\n    fn test_is_chunked_encoding_from_headers_empty() {\n        let empty_headers = HMap::new();\n        assert!(!is_chunked_encoding_from_headers(&empty_headers));\n    }\n\n    #[rstest]\n    #[case::single_chunked(\"chunked\", true)]\n    #[case::comma_separated_final(\"identity, chunked\", true)]\n    #[case::whitespace_around(\"  chunked  \", true)]\n    #[case::empty_elements_before(\", , , chunked\", true)]\n    #[case::only_identity(\"identity\", false)]\n    #[case::trailing_comma(\"chunked, \", false)]\n    #[case::multiple_trailing_commas(\"chunked, , \", false)]\n    #[case::empty_value(\"\", false)]\n    #[case::whitespace_only(\"   \", false)]\n    fn test_is_chunked_encoding_single_header(#[case] value: &str, #[case] expected: bool) {\n        let mut headers = HMap::new();\n        headers.insert(TRANSFER_ENCODING, value.try_into().unwrap());\n        assert_eq!(is_chunked_encoding_from_headers(&headers), expected);\n    }\n\n    #[rstest]\n    #[case::two_headers_chunked_last(&[\"identity\", \"chunked\"], true)]\n    #[case::three_headers_chunked_last(&[\"gzip\", \"identity\", \"chunked\"], true)]\n    #[case::last_has_comma_separated(&[\"gzip\", \"identity, chunked\"], true)]\n    #[case::whitespace_in_last(&[\"gzip\", \"  chunked  \"], true)]\n    #[case::two_headers_no_chunked(&[\"identity\", \"gzip\"], false)]\n    #[case::chunked_not_last(&[\"chunked\", \"identity\"], false)]\n    #[case::last_has_chunked_not_final(&[\"gzip\", \"chunked, identity\"], false)]\n    #[case::chunked_overridden(&[\"chunked\", \"identity, gzip\"], false)]\n    #[case::trailing_comma_in_last(&[\"gzip\", \"chunked, \"], false)]\n    fn test_is_chunked_encoding_multiple_headers(#[case] values: &[&str], #[case] expected: bool) {\n        let mut headers = HMap::new();\n        for value in values {\n            headers.append(TRANSFER_ENCODING, (*value).try_into().unwrap());\n        }\n        assert_eq!(is_chunked_encoding_from_headers(&headers), expected);\n    }\n}\n"
  },
  {
    "path": "pingora-core/src/protocols/http/v1/mod.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! HTTP/1.x implementation\n\npub(crate) mod body;\npub mod client;\npub mod common;\npub mod server;\n"
  },
  {
    "path": "pingora-core/src/protocols/http/v1/server.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! HTTP/1.x server session\n\nuse bstr::ByteSlice;\nuse bytes::Bytes;\nuse bytes::{BufMut, BytesMut};\nuse http::header::{CONTENT_LENGTH, TRANSFER_ENCODING};\nuse http::HeaderValue;\nuse http::{header, header::AsHeaderName, Method, Version};\nuse log::{debug, trace, warn};\nuse once_cell::sync::Lazy;\nuse percent_encoding::{percent_encode, AsciiSet, CONTROLS};\nuse pingora_error::{Error, ErrorType::*, OrErr, Result};\nuse pingora_http::{IntoCaseHeaderName, RequestHeader, ResponseHeader};\nuse pingora_timeout::timeout;\nuse regex::bytes::Regex;\nuse std::time::Duration;\nuse tokio::io::{AsyncReadExt, AsyncWriteExt};\n\nuse super::body::{BodyReader, BodyWriter};\nuse super::common::*;\nuse crate::protocols::http::{body_buffer::FixedBuffer, date, HttpTask};\nuse crate::protocols::{Digest, SocketAddr, Stream};\nuse crate::utils::{BufRef, KVRef};\n\n/// The HTTP 1.x server session\npub struct HttpSession {\n    underlying_stream: Stream,\n    /// The buf that holds the raw request header + possibly a portion of request body\n    /// Request body can appear here because they could arrive with the same read() that\n    /// sends the request header.\n    buf: Bytes,\n    /// A slice reference to `buf` which points to the exact range of request header\n    raw_header: Option<BufRef>,\n    /// A slice reference to `buf` which points to the range of a portion of request body if any\n    preread_body: Option<BufRef>,\n    /// A state machine to track how to read the request body\n    body_reader: BodyReader,\n    /// A state machine to track how to write the response body\n    body_writer: BodyWriter,\n    /// An internal buffer to buf multiple body writes to reduce the underlying syscalls\n    body_write_buf: BytesMut,\n    /// Track how many application (not on the wire) body bytes already sent\n    body_bytes_sent: usize,\n    /// Track how many application (not on the wire) body bytes already read\n    body_bytes_read: usize,\n    /// Whether to update headers like connection, Date\n    update_resp_headers: bool,\n    /// timeouts:\n    keepalive_timeout: KeepaliveStatus,\n    read_timeout: Option<Duration>,\n    write_timeout: Option<Duration>,\n    /// How long to wait to make downstream session reusable, if body needs to be drained.\n    total_drain_timeout: Option<Duration>,\n    /// A copy of the response that is already written to the client\n    response_written: Option<Box<ResponseHeader>>,\n    /// The parsed request header\n    request_header: Option<Box<RequestHeader>>,\n    /// An internal buffer that holds a copy of the request body up to a certain size\n    retry_buffer: Option<FixedBuffer>,\n    /// Whether this session is an upgraded session. This flag is calculated when sending the\n    /// response header to the client.\n    upgraded: bool,\n    /// Digest to track underlying connection metrics\n    digest: Box<Digest>,\n    /// Minimum send rate to the client\n    min_send_rate: Option<usize>,\n    /// When this is enabled informational response headers will not be proxied downstream\n    ignore_info_resp: bool,\n    /// Disable keepalive if response is sent before downstream body is finished\n    close_on_response_before_downstream_finish: bool,\n\n    /// Number of times the upstream connection associated with this session can be reused\n    /// after this session ends\n    keepalive_reuses_remaining: Option<u32>,\n}\n\nimpl HttpSession {\n    /// Create a new http server session from an established (TCP or TLS) [`Stream`].\n    /// The created session needs to call [`Self::read_request()`] first before performing\n    /// any other operations.\n    pub fn new(underlying_stream: Stream) -> Self {\n        // TODO: maybe we should put digest in the connection itself\n        let digest = Box::new(Digest {\n            ssl_digest: underlying_stream.get_ssl_digest(),\n            timing_digest: underlying_stream.get_timing_digest(),\n            proxy_digest: underlying_stream.get_proxy_digest(),\n            socket_digest: underlying_stream.get_socket_digest(),\n        });\n\n        HttpSession {\n            underlying_stream,\n            buf: Bytes::new(), // zero size, with be replaced by parsed header later\n            raw_header: None,\n            preread_body: None,\n            body_reader: BodyReader::new(false),\n            body_writer: BodyWriter::new(),\n            body_write_buf: BytesMut::new(),\n            keepalive_timeout: KeepaliveStatus::Off,\n            update_resp_headers: true,\n            response_written: None,\n            request_header: None,\n            read_timeout: Some(Duration::from_secs(60)),\n            write_timeout: None,\n            total_drain_timeout: None,\n            body_bytes_sent: 0,\n            body_bytes_read: 0,\n            retry_buffer: None,\n            upgraded: false,\n            digest,\n            min_send_rate: None,\n            ignore_info_resp: false,\n            // default on to avoid rejecting requests after body as pipelined\n            close_on_response_before_downstream_finish: true,\n            keepalive_reuses_remaining: None,\n        }\n    }\n\n    /// Read the request header. Return `Ok(Some(n))` where the read and parsing are successful.\n    /// Return `Ok(None)` when the client closed the connection without sending any data, which\n    /// is common on a reused connection.\n    pub async fn read_request(&mut self) -> Result<Option<usize>> {\n        const MAX_ERR_BUF_LEN: usize = 2048;\n\n        self.buf.clear();\n        let mut buf = BytesMut::with_capacity(INIT_HEADER_BUF_SIZE);\n        let mut already_read: usize = 0;\n        loop {\n            if already_read > MAX_HEADER_SIZE {\n                /* NOTE: this check only blocks second read. The first large read is allowed\n                since the buf is already allocated. The goal is to avoid slowly bloating\n                this buffer */\n                return Error::e_explain(\n                    InvalidHTTPHeader,\n                    format!(\"Request header larger than {MAX_HEADER_SIZE}\"),\n                );\n            }\n\n            let read_result = {\n                let read_event = self.underlying_stream.read_buf(&mut buf);\n                match self.keepalive_timeout {\n                    KeepaliveStatus::Timeout(d) => match timeout(d, read_event).await {\n                        Ok(res) => res,\n                        Err(e) => {\n                            debug!(\"keepalive timeout {d:?} reached, {e}\");\n                            return Ok(None);\n                        }\n                    },\n                    KeepaliveStatus::Infinite => {\n                        // FIXME: this should only apply to reads between requests\n                        read_event.await\n                    }\n                    KeepaliveStatus::Off => match self.read_timeout {\n                        Some(t) => match timeout(t, read_event).await {\n                            Ok(res) => res,\n                            Err(e) => {\n                                debug!(\"read timeout {t:?} reached, {e}\");\n                                return Error::e_explain(ReadTimedout, format!(\"timeout: {t:?}\"));\n                            }\n                        },\n                        None => read_event.await,\n                    },\n                }\n            };\n            let n = match read_result {\n                Ok(n_read) => {\n                    if n_read == 0 {\n                        if already_read > 0 {\n                            return Error::e_explain(\n                                ConnectionClosed,\n                                format!(\n                                    \"while reading request headers, bytes already read: {}\",\n                                    already_read\n                                ),\n                            );\n                        } else {\n                            /* common when client decides to close a keepalived session */\n                            debug!(\"Client prematurely closed connection with 0 byte sent\");\n                            return Ok(None);\n                        }\n                    }\n                    n_read\n                }\n\n                Err(e) => {\n                    if already_read > 0 {\n                        return Error::e_because(ReadError, \"while reading request headers\", e);\n                    }\n                    /* nothing harmful since we have not ready any thing yet */\n                    return Ok(None);\n                }\n            };\n            already_read += n;\n\n            // Use loop as GOTO to retry escaped request buffer, not a real loop\n            loop {\n                let mut headers = [httparse::EMPTY_HEADER; MAX_HEADERS];\n                let mut req = httparse::Request::new(&mut headers);\n                let parsed = parse_req_buffer(&mut req, &buf);\n                match parsed {\n                    HeaderParseState::Complete(s) => {\n                        self.raw_header = Some(BufRef(0, s));\n                        self.preread_body = Some(BufRef(s, already_read));\n\n                        // We have the header name and values we parsed to be just 0 copy Bytes\n                        // referencing the original buf. That requires we convert the buf from\n                        // BytesMut to Bytes. But `req` holds a reference to `buf`. So we use the\n                        // `KVRef`s to record the offset of each piece of data, drop `req`, convert\n                        // buf, the do the 0 copy update\n                        let base = buf.as_ptr() as usize;\n                        let mut header_refs = Vec::<KVRef>::with_capacity(req.headers.len());\n                        // Note: req.headers has the correct number of headers\n                        // while header_refs doesn't as it is still empty\n                        let _num_headers = populate_headers(base, &mut header_refs, req.headers);\n\n                        let mut request_header = Box::new(RequestHeader::build(\n                            req.method.unwrap_or(\"\"),\n                            // we path httparse to allow unsafe bytes in the str\n                            req.path.unwrap_or(\"\").as_bytes(),\n                            Some(req.headers.len()),\n                        )?);\n\n                        request_header.set_version(match req.version {\n                            Some(1) => Version::HTTP_11,\n                            Some(0) => Version::HTTP_10,\n                            _ => Version::HTTP_09,\n                        });\n\n                        let buf = buf.freeze();\n\n                        for header in header_refs {\n                            let header_name = header.get_name_bytes(&buf);\n                            let header_name = header_name.into_case_header_name();\n                            let value_bytes = header.get_value_bytes(&buf);\n                            // safe because this is from what we parsed\n                            let header_value = unsafe {\n                                http::HeaderValue::from_maybe_shared_unchecked(value_bytes)\n                            };\n\n                            request_header\n                                .append_header(header_name, header_value)\n                                .or_err(InvalidHTTPHeader, \"while parsing request header\")?;\n                        }\n\n                        let contains_transfer_encoding =\n                            request_header.headers.contains_key(TRANSFER_ENCODING);\n                        let contains_content_length =\n                            request_header.headers.contains_key(CONTENT_LENGTH);\n\n                        // Transfer encoding overrides content length, so when\n                        // both are present, we can remove content length. This\n                        // is per https://datatracker.ietf.org/doc/html/rfc9112#section-6.3\n                        //\n                        // RFC 9112 Section 6.1 (https://datatracker.ietf.org/doc/html/rfc9112#section-6.1-15)\n                        // also requires us to disable keepalive when both headers are present.\n                        let has_both_te_and_cl =\n                            contains_content_length && contains_transfer_encoding;\n                        if has_both_te_and_cl {\n                            request_header.remove_header(&CONTENT_LENGTH);\n                        }\n\n                        self.buf = buf;\n                        self.request_header = Some(request_header);\n\n                        self.body_reader.reinit();\n                        self.response_written = None;\n                        self.respect_keepalive();\n\n                        // Disable keepalive if both Transfer-Encoding and Content-Length were present\n                        if has_both_te_and_cl {\n                            self.set_keepalive(None);\n                        }\n                        self.validate_request()?;\n\n                        return Ok(Some(s));\n                    }\n                    HeaderParseState::Partial => {\n                        break; /* continue the read loop */\n                    }\n                    HeaderParseState::Invalid(e) => match e {\n                        httparse::Error::Token | httparse::Error::Version => {\n                            // try to escape URI\n                            if let Some(new_buf) = escape_illegal_request_line(&buf) {\n                                buf = new_buf;\n                                already_read = buf.len();\n                            } else {\n                                debug!(\"Invalid request header from {:?}\", self.underlying_stream);\n                                buf.truncate(MAX_ERR_BUF_LEN);\n                                return Error::e_because(\n                                    InvalidHTTPHeader,\n                                    format!(\"buf: {}\", buf.escape_ascii()),\n                                    e,\n                                );\n                            }\n                        }\n                        _ => {\n                            debug!(\"Invalid request header from {:?}\", self.underlying_stream);\n                            buf.truncate(MAX_ERR_BUF_LEN);\n                            return Error::e_because(\n                                InvalidHTTPHeader,\n                                format!(\"buf: {:?}\", buf.as_bstr()),\n                                e,\n                            );\n                        }\n                    },\n                }\n            }\n        }\n    }\n\n    /// Validate the request header read. This function must be called after the request header\n    /// read.\n    /// # Panics\n    /// this function and most other functions will panic if called before [`Self::read_request()`]\n    pub fn validate_request(&self) -> Result<()> {\n        let req_header = self.req_header();\n\n        // ad-hoc checks\n        super::common::check_dup_content_length(&req_header.headers)?;\n\n        if req_header.headers.contains_key(TRANSFER_ENCODING) {\n            // Per [RFC 9112 Section 6.1-16](https://datatracker.ietf.org/doc/html/rfc9112#section-6.1-16),\n            // HTTP/1.0 requests with Transfer-Encoding MUST be treated as having faulty framing.\n            // We reject with 400 Bad Request and close the connection.\n            if req_header.version == http::Version::HTTP_10 {\n                return Error::e_explain(\n                    InvalidHTTPHeader,\n                    \"HTTP/1.0 requests cannot include Transfer-Encoding header\",\n                );\n            }\n            // If chunked is not the final Transfer-Encoding, reject request\n            // See https://datatracker.ietf.org/doc/html/rfc9112#section-6.3-2.4.3\n            if !self.is_chunked_encoding() {\n                return Error::e_explain(InvalidHTTPHeader, \"non-chunked final Transfer-Encoding\");\n            }\n        }\n        // validate content-length value if present to avoid ambiguous framing\n        self.get_content_length()?;\n\n        Ok(())\n    }\n\n    /// Return a reference of the `RequestHeader` this session read\n    /// # Panics\n    /// this function and most other functions will panic if called before [`Self::read_request()`]\n    pub fn req_header(&self) -> &RequestHeader {\n        self.request_header\n            .as_ref()\n            .expect(\"Request header is not read yet\")\n    }\n\n    /// Return a mutable reference of the `RequestHeader` this session read\n    /// # Panics\n    /// this function and most other functions will panic if called before [`Self::read_request()`]\n    pub fn req_header_mut(&mut self) -> &mut RequestHeader {\n        self.request_header\n            .as_mut()\n            .expect(\"Request header is not read yet\")\n    }\n\n    /// Get the header value for the given header name\n    /// If there are multiple headers under the same name, the first one will be returned\n    /// Use `self.req_header().header.get_all(name)` to get all the headers under the same name\n    pub fn get_header(&self, name: impl AsHeaderName) -> Option<&HeaderValue> {\n        self.request_header\n            .as_ref()\n            .and_then(|h| h.headers.get(name))\n    }\n\n    /// Return the method of this request. None if the request is not read yet.\n    pub(crate) fn get_method(&self) -> Option<&http::Method> {\n        self.request_header.as_ref().map(|r| &r.method)\n    }\n\n    /// Return the path of the request (i.e., the `/hello?1` of `GET /hello?1 HTTP1.1`)\n    /// An empty slice will be used if there is no path or the request is not read yet\n    pub(crate) fn get_path(&self) -> &[u8] {\n        self.request_header.as_ref().map_or(b\"\", |r| r.raw_path())\n    }\n\n    /// Return the host header of the request. An empty slice will be used if there is no host header\n    pub(crate) fn get_host(&self) -> &[u8] {\n        self.request_header\n            .as_ref()\n            .and_then(|h| h.headers.get(header::HOST))\n            .map_or(b\"\", |h| h.as_bytes())\n    }\n\n    /// Return a string `$METHOD $PATH, Host: $HOST`. Mostly for logging and debug purpose\n    pub fn request_summary(&self) -> String {\n        format!(\n            \"{} {}, Host: {}\",\n            self.get_method().map_or(\"-\", |r| r.as_str()),\n            String::from_utf8_lossy(self.get_path()),\n            String::from_utf8_lossy(self.get_host())\n        )\n    }\n\n    /// Is the request a upgrade request\n    pub fn is_upgrade_req(&self) -> bool {\n        match self.request_header.as_deref() {\n            Some(req) => is_upgrade_req(req),\n            None => false,\n        }\n    }\n\n    /// Get the request header as raw bytes, `b\"\"` when the header doesn't exist\n    pub fn get_header_bytes(&self, name: impl AsHeaderName) -> &[u8] {\n        self.get_header(name).map_or(b\"\", |v| v.as_bytes())\n    }\n\n    /// Read the request body. `Ok(None)` when there is no (more) body to read.\n    pub async fn read_body_bytes(&mut self) -> Result<Option<Bytes>> {\n        let read = self.read_body().await?;\n        Ok(read.map(|b| {\n            let bytes = Bytes::copy_from_slice(self.get_body(&b));\n            self.body_bytes_read += bytes.len();\n            if let Some(buffer) = self.retry_buffer.as_mut() {\n                buffer.write_to_buffer(&bytes);\n            }\n            bytes\n        }))\n    }\n\n    async fn do_read_body(&mut self) -> Result<Option<BufRef>> {\n        self.init_body_reader();\n        self.body_reader\n            .read_body(&mut self.underlying_stream)\n            .await\n    }\n\n    /// Read the body into the internal buffer\n    async fn read_body(&mut self) -> Result<Option<BufRef>> {\n        match self.read_timeout {\n            Some(t) => match timeout(t, self.do_read_body()).await {\n                Ok(res) => res,\n                Err(_) => Error::e_explain(ReadTimedout, format!(\"reading body, timeout: {t:?}\")),\n            },\n            None => self.do_read_body().await,\n        }\n    }\n\n    async fn do_drain_request_body(&mut self) -> Result<()> {\n        loop {\n            match self.read_body_bytes().await {\n                Ok(Some(_)) => { /* continue to drain */ }\n                Ok(None) => return Ok(()), // done\n                Err(e) => return Err(e),\n            }\n        }\n    }\n\n    /// Drain the request body. `Ok(())` when there is no (more) body to read.\n    pub async fn drain_request_body(&mut self) -> Result<()> {\n        if self.is_body_done() {\n            return Ok(());\n        }\n        match self.total_drain_timeout {\n            Some(t) => match timeout(t, self.do_drain_request_body()).await {\n                Ok(res) => res,\n                Err(_) => Error::e_explain(ReadTimedout, format!(\"draining body, timeout: {t:?}\")),\n            },\n            None => self.do_drain_request_body().await,\n        }\n    }\n\n    /// Whether there is no (more) body to be read.\n    pub fn is_body_done(&mut self) -> bool {\n        self.init_body_reader();\n        self.body_reader.body_done()\n    }\n\n    /// Whether the request has an empty body\n    /// Because HTTP 1.1 clients have to send either `Content-Length` or `Transfer-Encoding` in order\n    /// to signal the server that it will send the body, this function returns accurate results even\n    /// only when the request header is just read.\n    pub fn is_body_empty(&mut self) -> bool {\n        self.init_body_reader();\n        self.body_reader.body_empty()\n    }\n\n    /// Write the response header to the client.\n    /// This function can be called more than once to send 1xx informational headers excluding 101.\n    pub async fn write_response_header(&mut self, mut header: Box<ResponseHeader>) -> Result<()> {\n        if header.status.is_informational() && self.ignore_info_resp(header.status.into()) {\n            debug!(\"ignoring informational headers\");\n            return Ok(());\n        }\n\n        if let Some(resp) = self.response_written.as_ref() {\n            if !resp.status.is_informational() || self.upgraded {\n                warn!(\"Respond header is already sent, cannot send again\");\n                return Ok(());\n            }\n        }\n\n        // if body unfinished, or request header was not finished reading\n        if self.close_on_response_before_downstream_finish\n            && (self.request_header.is_none() || !self.is_body_done())\n        {\n            debug!(\"set connection close before downstream finish\");\n            self.set_keepalive(None);\n        }\n\n        // no need to add these headers to 1xx responses\n        if !header.status.is_informational() && self.update_resp_headers {\n            /* update headers */\n            header.insert_header(header::DATE, date::get_cached_date())?;\n\n            // TODO: make these lazy static\n            let connection_value = if self.will_keepalive() {\n                \"keep-alive\"\n            } else {\n                \"close\"\n            };\n            header.insert_header(header::CONNECTION, connection_value)?;\n        }\n\n        if header.status == 101 {\n            // make sure the connection is closed at the end when 101/upgrade is used\n            self.set_keepalive(None);\n        }\n\n        // Allow informational header (excluding 101) to pass through without affecting the state\n        // of the request\n        if header.status == 101 || !header.status.is_informational() {\n            // reset request body to done for incomplete upgrade handshakes\n            if let Some(upgrade_ok) = self.is_upgrade(&header) {\n                if upgrade_ok {\n                    debug!(\"ok upgrade handshake\");\n                    // For ws we use HTTP1_0 do_read_body_until_closed\n                    //\n                    // On ws close the initiator sends a close frame and\n                    // then waits for a response from the peer, once it receives\n                    // a response it closes the conn. After receiving a\n                    // control frame indicating the connection should be closed,\n                    // a peer discards any further data received.\n                    // https://www.rfc-editor.org/rfc/rfc6455#section-1.4\n                    self.upgraded = true;\n                    // Now that the upgrade was successful, we need to change\n                    // how we interpret the rest of the body as pass-through.\n                    if self.body_reader.need_init() {\n                        self.init_body_reader();\n                    } else {\n                        // already initialized\n                        // immediately start reading the rest of the body as upgraded\n                        // (in practice most upgraded requests shouldn't have any body)\n                        //\n                        // TODO: https://datatracker.ietf.org/doc/html/rfc9110#name-upgrade\n                        // the most spec-compliant behavior is to switch interpretation\n                        // after sending the former body,\n                        // we immediately switch interpretation to match nginx\n                        self.body_reader.convert_to_close_delimited();\n                    }\n                } else {\n                    // this was a request that requested Upgrade,\n                    // but upstream did not comply\n                    debug!(\"bad upgrade handshake!\");\n                    // continue to read body as-is, this is now just a regular request\n                }\n            }\n            self.init_body_writer(&header);\n        }\n\n        // Defense-in-depth: if response body is close-delimited, mark session\n        // as un-reusable\n        if self.body_writer.is_close_delimited() {\n            self.set_keepalive(None);\n        }\n\n        // Don't have to flush response with content length because it is less\n        // likely to be real time communication. So do flush when\n        // 1.1xx response: client needs to see it before the rest of response\n        // 2.No content length: the response could be generated in real time\n        let flush = header.status.is_informational()\n            || header.headers.get(header::CONTENT_LENGTH).is_none();\n\n        let mut write_buf = BytesMut::with_capacity(INIT_HEADER_BUF_SIZE);\n        http_resp_header_to_buf(&header, &mut write_buf).unwrap();\n        match self.underlying_stream.write_all(&write_buf).await {\n            Ok(()) => {\n                // flush the stream if 1xx header or there is no response body\n                if flush || self.body_writer.finished() {\n                    self.underlying_stream\n                        .flush()\n                        .await\n                        .or_err(WriteError, \"flushing response header\")?;\n                }\n                self.response_written = Some(header);\n                self.body_bytes_sent += write_buf.len();\n                Ok(())\n            }\n            Err(e) => Error::e_because(WriteError, \"writing response header\", e),\n        }\n    }\n\n    /// Return the response header if it is already sent.\n    pub fn response_written(&self) -> Option<&ResponseHeader> {\n        self.response_written.as_deref()\n    }\n\n    /// `Some(true)` if the this is a successful upgrade\n    /// `Some(false)` if the request is an upgrade but the response refuses it\n    /// `None` if the request is not an upgrade.\n    pub fn is_upgrade(&self, header: &ResponseHeader) -> Option<bool> {\n        if self.is_upgrade_req() {\n            Some(is_upgrade_resp(header))\n        } else {\n            None\n        }\n    }\n\n    /// Was this request successfully turned into an upgraded connection?\n    ///\n    /// Both the request had to have been an `Upgrade` request\n    /// and the response had to have been a `101 Switching Protocols`.\n    pub fn was_upgraded(&self) -> bool {\n        self.upgraded\n    }\n\n    fn set_keepalive(&mut self, seconds: Option<u64>) {\n        match seconds {\n            Some(sec) => {\n                if sec > 0 {\n                    self.keepalive_timeout = KeepaliveStatus::Timeout(Duration::from_secs(sec));\n                } else {\n                    self.keepalive_timeout = KeepaliveStatus::Infinite;\n                }\n            }\n            None => {\n                self.keepalive_timeout = KeepaliveStatus::Off;\n            }\n        }\n    }\n\n    pub fn get_keepalive_timeout(&self) -> Option<u64> {\n        match self.keepalive_timeout {\n            KeepaliveStatus::Timeout(d) => Some(d.as_secs()),\n            KeepaliveStatus::Infinite => Some(0),\n            KeepaliveStatus::Off => None,\n        }\n    }\n\n    pub fn set_keepalive_reuses_remaining(&mut self, remaining: Option<u32>) {\n        self.keepalive_reuses_remaining = remaining;\n    }\n\n    pub fn get_keepalive_reuses_remaining(&self) -> Option<u32> {\n        self.keepalive_reuses_remaining\n    }\n\n    /// Return whether the session will be keepalived for connection reuse.\n    pub fn will_keepalive(&self) -> bool {\n        !matches!(\n            (&self.keepalive_timeout, self.keepalive_reuses_remaining),\n            (KeepaliveStatus::Off, _) | (_, Some(0))\n        )\n    }\n\n    // `Keep-Alive: timeout=5, max=1000` => 5, 1000\n    fn get_keepalive_values(&self) -> (Option<u64>, Option<usize>) {\n        // TODO: implement this parsing\n        (None, None)\n    }\n\n    fn ignore_info_resp(&self, status: u16) -> bool {\n        // ignore informational response if ignore flag is set and it's not an Upgrade and Expect: 100-continue isn't set\n        self.ignore_info_resp && status != 101 && !(status == 100 && self.is_expect_continue_req())\n    }\n\n    fn is_expect_continue_req(&self) -> bool {\n        match self.request_header.as_deref() {\n            Some(req) => is_expect_continue_req(req),\n            None => false,\n        }\n    }\n\n    fn is_connection_keepalive(&self) -> Option<bool> {\n        is_buf_keepalive(self.get_header(header::CONNECTION))\n    }\n\n    // calculate write timeout from min_send_rate if set, otherwise return write_timeout\n    fn write_timeout(&self, buf_len: usize) -> Option<Duration> {\n        let Some(min_send_rate) = self.min_send_rate.filter(|r| *r > 0) else {\n            return self.write_timeout;\n        };\n\n        // min timeout is 1s\n        let ms = (buf_len.max(min_send_rate) as f64 / min_send_rate as f64) * 1000.0;\n        // truncates unrealistically large values (we'll be out of memory before this happens)\n        Some(Duration::from_millis(ms as u64))\n    }\n\n    /// Apply keepalive settings according to the client\n    /// For HTTP 1.1, assume keepalive as long as there is no `Connection: Close` request header.\n    /// For HTTP 1.0, only keepalive if there is an explicit header `Connection: keep-alive`.\n    pub fn respect_keepalive(&mut self) {\n        if let Some(keepalive) = self.is_connection_keepalive() {\n            if keepalive {\n                let (timeout, _max_use) = self.get_keepalive_values();\n                // TODO: respect max_use\n                match timeout {\n                    Some(d) => self.set_keepalive(Some(d)),\n                    None => self.set_keepalive(Some(0)), // infinite\n                }\n            } else {\n                self.set_keepalive(None);\n            }\n        } else if self.req_header().version == Version::HTTP_11 {\n            self.set_keepalive(Some(0)); // on by default for http 1.1\n        } else {\n            self.set_keepalive(None); // off by default for http 1.0\n        }\n    }\n\n    fn init_body_writer(&mut self, header: &ResponseHeader) {\n        use http::StatusCode;\n        /* the following responses don't have body 204, 304, and HEAD */\n        if matches!(\n            header.status,\n            StatusCode::NO_CONTENT | StatusCode::NOT_MODIFIED\n        ) || self.get_method() == Some(&Method::HEAD)\n        {\n            self.body_writer.init_content_length(0);\n            return;\n        }\n\n        if header.status.is_informational() && header.status != StatusCode::SWITCHING_PROTOCOLS {\n            // 1xx response, not enough to init body\n            return;\n        }\n\n        if self.is_upgrade(header) == Some(true) {\n            self.body_writer.init_close_delimited();\n        } else {\n            init_body_writer_comm(&mut self.body_writer, &header.headers);\n        }\n    }\n\n    /// Same as [`Self::write_response_header()`] but takes a reference.\n    pub async fn write_response_header_ref(&mut self, resp: &ResponseHeader) -> Result<()> {\n        self.write_response_header(Box::new(resp.clone())).await\n    }\n\n    async fn do_write_body(&mut self, buf: &[u8]) -> Result<Option<usize>> {\n        let written = self\n            .body_writer\n            .write_body(&mut self.underlying_stream, buf)\n            .await;\n\n        if let Ok(Some(num_bytes)) = written {\n            self.body_bytes_sent += num_bytes;\n        }\n\n        written\n    }\n\n    /// Write response body to the client. Return `Ok(None)` when there shouldn't be more body\n    /// to be written, e.g., writing more bytes than what the `Content-Length` header suggests\n    pub async fn write_body(&mut self, buf: &[u8]) -> Result<Option<usize>> {\n        // TODO: check if the response header is written\n        match self.write_timeout(buf.len()) {\n            Some(t) => match timeout(t, self.do_write_body(buf)).await {\n                Ok(res) => res,\n                Err(_) => Error::e_explain(WriteTimedout, format!(\"writing body, timeout: {t:?}\")),\n            },\n            None => self.do_write_body(buf).await,\n        }\n    }\n\n    async fn do_write_body_buf(&mut self) -> Result<Option<usize>> {\n        // Don't flush empty chunks, they are considered end of body for chunks\n        if self.body_write_buf.is_empty() {\n            return Ok(None);\n        }\n\n        let written = self\n            .body_writer\n            .write_body(&mut self.underlying_stream, &self.body_write_buf)\n            .await;\n\n        if let Ok(Some(num_bytes)) = written {\n            self.body_bytes_sent += num_bytes;\n        }\n\n        // make sure this buf is safe to reuse\n        self.body_write_buf.clear();\n\n        written\n    }\n\n    async fn write_body_buf(&mut self) -> Result<Option<usize>> {\n        match self.write_timeout(self.body_write_buf.len()) {\n            Some(t) => match timeout(t, self.do_write_body_buf()).await {\n                Ok(res) => res,\n                Err(_) => Error::e_explain(WriteTimedout, format!(\"writing body, timeout: {t:?}\")),\n            },\n            None => self.do_write_body_buf().await,\n        }\n    }\n\n    fn maybe_force_close_body_reader(&mut self) {\n        if self.upgraded && !self.body_reader.body_done() {\n            // response is done, reset the request body to close\n            self.body_reader.init_content_length(0, b\"\");\n        }\n    }\n\n    /// Signal that there is no more body to write.\n    /// This call will try to flush the buffer if there is any un-flushed data.\n    /// For chunked encoding response, this call will also send the last chunk.\n    /// For upgraded sessions, this call will also close the reading of the client body.\n    pub async fn finish_body(&mut self) -> Result<Option<usize>> {\n        let res = self.body_writer.finish(&mut self.underlying_stream).await?;\n        self.underlying_stream\n            .flush()\n            .await\n            .or_err(WriteError, \"flushing body\")?;\n\n        trace!(\n            \"finish body (response body writer), upgraded: {}\",\n            self.upgraded\n        );\n        self.maybe_force_close_body_reader();\n        Ok(res)\n    }\n\n    /// Return how many response body bytes (application, not wire) already sent downstream\n    pub fn body_bytes_sent(&self) -> usize {\n        self.body_bytes_sent\n    }\n\n    /// Return how many request body bytes (application, not wire) already read from downstream\n    pub fn body_bytes_read(&self) -> usize {\n        self.body_bytes_read\n    }\n\n    fn is_chunked_encoding(&self) -> bool {\n        is_chunked_encoding_from_headers(&self.req_header().headers)\n    }\n\n    fn get_content_length(&self) -> Result<Option<usize>> {\n        buf_to_content_length(\n            self.get_header(header::CONTENT_LENGTH)\n                .map(|v| v.as_bytes()),\n        )\n    }\n\n    fn init_body_reader(&mut self) {\n        if self.body_reader.need_init() {\n            // reset retry buffer\n            if let Some(buffer) = self.retry_buffer.as_mut() {\n                buffer.clear();\n            }\n\n            // follow https://datatracker.ietf.org/doc/html/rfc9112#section-6.3\n            let preread_body = self.preread_body.as_ref().unwrap().get(&self.buf[..]);\n\n            if self.was_upgraded() {\n                // if upgraded _post_ 101 (and body was not init yet)\n                // treat as upgraded body (pass through until closed)\n                self.body_reader.init_close_delimited(preread_body);\n            } else if self.is_chunked_encoding() {\n                // if chunked encoding, content-length should be ignored\n                self.body_reader.init_chunked(preread_body);\n            } else {\n                // At this point, validate_request() should have already been called,\n                // so get_content_length() should not return an error for invalid values\n                let cl = self.get_content_length().unwrap_or(None);\n                match cl {\n                    Some(i) => {\n                        self.body_reader.init_content_length(i, preread_body);\n                    }\n                    None => {\n                        // https://datatracker.ietf.org/doc/html/rfc9112#section-6.3\n                        // \"Request messages are never close-delimited because they are\n                        // always explicitly framed by length or transfer coding, with the absence of\n                        // both implying the request ends immediately after the header section.\"\n                        self.body_reader.init_content_length(0, preread_body);\n                    }\n                }\n            }\n        }\n    }\n\n    pub fn retry_buffer_truncated(&self) -> bool {\n        self.retry_buffer\n            .as_ref()\n            .map_or_else(|| false, |r| r.is_truncated())\n    }\n\n    pub fn enable_retry_buffering(&mut self) {\n        if self.retry_buffer.is_none() {\n            self.retry_buffer = Some(FixedBuffer::new(BODY_BUF_LIMIT))\n        }\n    }\n\n    pub fn get_retry_buffer(&self) -> Option<Bytes> {\n        self.retry_buffer.as_ref().and_then(|b| {\n            if b.is_truncated() {\n                None\n            } else {\n                b.get_buffer()\n            }\n        })\n    }\n\n    fn get_body(&self, buf_ref: &BufRef) -> &[u8] {\n        // TODO: these get_*() could panic. handle them better\n        self.body_reader.get_body(buf_ref)\n    }\n\n    /// This function will (async) block forever until the client closes the connection.\n    pub async fn idle(&mut self) -> Result<usize> {\n        // NOTE: this implementation breaks http pipelining, ideally we need poll_error\n        // NOTE: buf cannot be empty, openssl-rs read() requires none empty buf.\n        let mut buf: [u8; 1] = [0; 1];\n        self.underlying_stream\n            .read(&mut buf)\n            .await\n            .or_err(ReadError, \"during HTTP idle state\")\n    }\n\n    /// This function will return body bytes (same as [`Self::read_body_bytes()`]), but after\n    /// the client body finishes (`Ok(None)` is returned), calling this function again will block\n    /// forever, same as [`Self::idle()`].\n    pub async fn read_body_or_idle(&mut self, no_body_expected: bool) -> Result<Option<Bytes>> {\n        if no_body_expected || self.is_body_done() {\n            // XXX: account for upgraded body reader change, if the read half split from the write half\n            let read = self.idle().await?;\n            if read == 0 {\n                Error::e_explain(\n                    ConnectionClosed,\n                    if self.response_written.is_none() {\n                        \"Prematurely before response header is sent\"\n                    } else {\n                        \"Prematurely before response body is complete\"\n                    },\n                )\n            } else {\n                Error::e_explain(ConnectError, \"Sent data after end of body\")\n            }\n        } else {\n            self.read_body_bytes().await\n        }\n    }\n\n    /// Return the raw bytes of the request header.\n    pub fn get_headers_raw_bytes(&self) -> Bytes {\n        self.raw_header.as_ref().unwrap().get_bytes(&self.buf)\n    }\n\n    /// Close the connection abruptly. This allows to signal the client that the connection is closed\n    /// before dropping [`HttpSession`]\n    pub async fn shutdown(&mut self) {\n        let _ = self.underlying_stream.shutdown().await;\n    }\n\n    /// Set the server keepalive timeout.\n    /// `None`: disable keepalive, this session cannot be reused.\n    /// `Some(0)`: reusing this session is allowed and there is no timeout.\n    /// `Some(>0)`: reusing this session is allowed within the given timeout in seconds.\n    /// If the client disallows connection reuse, then `keepalive` will be ignored.\n    pub fn set_server_keepalive(&mut self, keepalive: Option<u64>) {\n        if let Some(false) = self.is_connection_keepalive() {\n            // connection: close is set\n            self.set_keepalive(None);\n        } else {\n            self.set_keepalive(keepalive);\n        }\n    }\n\n    /// Sets the downstream read timeout. This will trigger if we're unable\n    /// to read from the stream after `timeout`.\n    pub fn set_read_timeout(&mut self, timeout: Option<Duration>) {\n        self.read_timeout = timeout;\n    }\n\n    /// Gets the downstream read timeout.\n    pub fn get_read_timeout(&self) -> Option<Duration> {\n        self.read_timeout\n    }\n\n    /// Sets the downstream write timeout. This will trigger if we're unable\n    /// to write to the stream after `timeout`. If a `min_send_rate` is\n    /// configured then the `min_send_rate` calculated timeout has higher priority.\n    pub fn set_write_timeout(&mut self, timeout: Option<Duration>) {\n        self.write_timeout = timeout;\n    }\n\n    /// Gets the downstream write timeout.\n    pub fn get_write_timeout(&self) -> Option<Duration> {\n        self.write_timeout\n    }\n\n    /// Sets the total drain timeout. For HTTP/1.1, reusing a session requires\n    /// ensuring that the request body is consumed. This `timeout` will be used\n    /// to determine how long to wait for the entirety of the downstream request\n    /// body to finish after the upstream response is completed to return the\n    /// session to the reuse pool. If the timeout is exceeded, we will give up\n    /// on trying to reuse the session.\n    ///\n    /// Note that the downstream read timeout still applies between body byte reads.\n    pub fn set_total_drain_timeout(&mut self, timeout: Option<Duration>) {\n        self.total_drain_timeout = timeout;\n    }\n\n    /// Get the total drain timeout.\n    pub fn get_total_drain_timeout(&self) -> Option<Duration> {\n        self.total_drain_timeout\n    }\n\n    /// Sets the minimum downstream send rate in bytes per second. This\n    /// is used to calculate a write timeout in seconds based on the size\n    /// of the buffer being written. If a `min_send_rate` is configured it\n    /// has higher priority over a set `write_timeout`. The minimum send\n    /// rate must be greater than zero.\n    ///\n    /// Calculated write timeout is guaranteed to be at least 1s if `min_send_rate`\n    /// is greater than zero, a send rate of zero is equivalent to disabling.\n    pub fn set_min_send_rate(&mut self, min_send_rate: Option<usize>) {\n        if let Some(rate) = min_send_rate.filter(|r| *r > 0) {\n            self.min_send_rate = Some(rate);\n        } else {\n            self.min_send_rate = None;\n        }\n    }\n\n    /// Sets whether we ignore writing informational responses downstream.\n    ///\n    /// This is a noop if the response is Upgrade or Continue and\n    /// Expect: 100-continue was set on the request.\n    pub fn set_ignore_info_resp(&mut self, ignore: bool) {\n        self.ignore_info_resp = ignore;\n    }\n\n    /// Sets whether keepalive should be disabled if response is written prior to\n    /// downstream body finishing.\n    ///\n    /// This may be set to avoid draining downstream if the body is no longer necessary.\n    pub fn set_close_on_response_before_downstream_finish(&mut self, close: bool) {\n        self.close_on_response_before_downstream_finish = close;\n    }\n\n    /// Return the [Digest] of the connection.\n    pub fn digest(&self) -> &Digest {\n        &self.digest\n    }\n\n    /// Return a mutable [Digest] reference for the connection.\n    pub fn digest_mut(&mut self) -> &mut Digest {\n        &mut self.digest\n    }\n\n    /// Return the client (peer) address of the underlying connection.\n    pub fn client_addr(&self) -> Option<&SocketAddr> {\n        self.digest()\n            .socket_digest\n            .as_ref()\n            .map(|d| d.peer_addr())?\n    }\n\n    /// Return the server (local) address of the underlying connection.\n    pub fn server_addr(&self) -> Option<&SocketAddr> {\n        self.digest()\n            .socket_digest\n            .as_ref()\n            .map(|d| d.local_addr())?\n    }\n\n    /// Consume `self`, if the connection can be reused, the underlying stream will be returned\n    /// to be fed to the next [`Self::new()`]. This drains any remaining request body if it hasn't\n    /// yet been read and the stream is reusable.\n    ///\n    /// The next session can just call [`Self::read_request()`].\n    ///\n    /// If the connection cannot be reused, the underlying stream will be closed and `None` will be\n    /// returned. If there was an error while draining any remaining request body that error will\n    /// be returned.\n    pub async fn reuse(mut self) -> Result<Option<Stream>> {\n        if !self.will_keepalive() {\n            debug!(\"HTTP shutdown connection\");\n            self.shutdown().await;\n            Ok(None)\n        } else {\n            self.drain_request_body().await?;\n            // XXX: currently pipelined requests are not properly read without\n            // pipelining support, and pingora 400s if pipelined requests are sent\n            // in the middle of another request.\n            // We will mark the connection as un-reusable so it may be closed,\n            // the pipelined request left unread, and the client can attempt to resend\n            if self.body_reader.has_bytes_overread() {\n                debug!(\"bytes overread on request, disallowing reuse\");\n                Ok(None)\n            } else {\n                Ok(Some(self.underlying_stream))\n            }\n        }\n    }\n\n    /// Write a `100 Continue` response to the client.\n    pub async fn write_continue_response(&mut self) -> Result<()> {\n        // only send if we haven't already\n        if self.response_written.is_none() {\n            // size hint Some(0) because default is 8\n            return self\n                .write_response_header(Box::new(ResponseHeader::build(100, Some(0)).unwrap()))\n                .await;\n        }\n        Ok(())\n    }\n\n    async fn write_non_empty_body(&mut self, data: Option<Bytes>, upgraded: bool) -> Result<()> {\n        // Both upstream and downstream should agree on upgrade status.\n        // Upgrade can only occur if both downstream and upstream sessions are H1.1\n        // and see a 101 response, which logically MUST have been received\n        // prior to this task.\n        if upgraded != self.upgraded {\n            if upgraded {\n                panic!(\"Unexpected UpgradedBody task received on un-upgraded downstream session\");\n            } else {\n                panic!(\"Unexpected Body task received on upgraded downstream session\");\n            }\n        }\n        let Some(d) = data else {\n            return Ok(());\n        };\n        if d.is_empty() {\n            return Ok(());\n        }\n        self.write_body(&d).await.map_err(|e| e.into_down())?;\n        Ok(())\n    }\n\n    async fn response_duplex(&mut self, task: HttpTask) -> Result<bool> {\n        let end_stream = match task {\n            HttpTask::Header(header, end_stream) => {\n                self.write_response_header(header)\n                    .await\n                    .map_err(|e| e.into_down())?;\n                end_stream\n            }\n            HttpTask::Body(data, end_stream) => {\n                self.write_non_empty_body(data, false).await?;\n                end_stream\n            }\n            HttpTask::UpgradedBody(data, end_stream) => {\n                self.write_non_empty_body(data, true).await?;\n                end_stream\n            }\n            HttpTask::Trailer(_) => true, // h1 trailer is not supported yet\n            HttpTask::Done => true,\n            HttpTask::Failed(e) => return Err(e),\n        };\n        if end_stream {\n            // no-op if body wasn't initialized or is finished already\n            self.finish_body().await.map_err(|e| e.into_down())?;\n        }\n        Ok(end_stream || self.body_writer.finished())\n    }\n\n    fn buffer_body_data(&mut self, data: Option<Bytes>, upgraded: bool) {\n        if upgraded != self.upgraded {\n            if upgraded {\n                panic!(\"Unexpected Body task received on upgraded downstream session\");\n            } else {\n                panic!(\"Unexpected UpgradedBody task received on un-upgraded downstream session\");\n            }\n        }\n\n        let Some(d) = data else {\n            return;\n        };\n        if !d.is_empty() && !self.body_writer.finished() {\n            self.body_write_buf.put_slice(&d);\n        }\n    }\n\n    // TODO: use vectored write to avoid copying\n    pub async fn response_duplex_vec(&mut self, mut tasks: Vec<HttpTask>) -> Result<bool> {\n        let n_tasks = tasks.len();\n        if n_tasks == 1 {\n            // fallback to single operation to avoid copy\n            return self.response_duplex(tasks.pop().unwrap()).await;\n        }\n\n        let mut end_stream = false;\n        for task in tasks.into_iter() {\n            end_stream = match task {\n                HttpTask::Header(header, end_stream) => {\n                    self.write_response_header(header)\n                        .await\n                        .map_err(|e| e.into_down())?;\n                    end_stream\n                }\n                HttpTask::Body(data, end_stream) => {\n                    self.buffer_body_data(data, false);\n                    end_stream\n                }\n                HttpTask::UpgradedBody(data, end_stream) => {\n                    self.buffer_body_data(data, true);\n                    end_stream\n                }\n                HttpTask::Trailer(_) => true, // h1 trailer is not supported yet\n                HttpTask::Done => true,\n                HttpTask::Failed(e) => {\n                    // flush the data we have and quit\n                    self.write_body_buf().await.map_err(|e| e.into_down())?;\n                    self.underlying_stream\n                        .flush()\n                        .await\n                        .or_err(WriteError, \"flushing response\")?;\n                    return Err(e);\n                }\n            }\n        }\n        self.write_body_buf().await.map_err(|e| e.into_down())?;\n        if end_stream {\n            // no-op if body wasn't initialized or is finished already\n            self.finish_body().await.map_err(|e| e.into_down())?;\n        }\n        Ok(end_stream || self.body_writer.finished())\n    }\n\n    /// Get the reference of the [Stream] that this HTTP session is operating upon.\n    pub fn stream(&self) -> &Stream {\n        &self.underlying_stream\n    }\n\n    /// Consume `self`, the underlying stream will be returned and can be used\n    /// directly, for example, in the case of HTTP upgrade. The stream is not\n    /// flushed prior to being returned.\n    pub fn into_inner(self) -> Stream {\n        self.underlying_stream\n    }\n}\n\n// Regex to parse request line that has illegal chars in it\nstatic REQUEST_LINE_REGEX: Lazy<Regex> =\n    Lazy::new(|| Regex::new(r\"^\\w+ (?P<uri>.+) HTTP/\\d(?:\\.\\d)?\").unwrap());\n\n// the chars httparse considers illegal in URL\n// Almost https://url.spec.whatwg.org/#query-percent-encode-set + {}\nconst URI_ESC_CHARSET: &AsciiSet = &CONTROLS.add(b' ').add(b'<').add(b'>').add(b'\"');\n\nfn escape_illegal_request_line(buf: &BytesMut) -> Option<BytesMut> {\n    if let Some(captures) = REQUEST_LINE_REGEX.captures(buf) {\n        // return if nothing matches: not a request line at all\n        let uri = captures.name(\"uri\")?;\n\n        let escaped_uri = percent_encode(uri.as_bytes(), URI_ESC_CHARSET);\n\n        // rebuild the entire request buf in a new buffer\n        // TODO: this might be able to be done in place\n\n        // need to be slightly bigger than the current buf;\n        let mut new_buf = BytesMut::with_capacity(buf.len() + 32);\n        new_buf.extend_from_slice(&buf[..uri.start()]);\n\n        for s in escaped_uri {\n            new_buf.extend_from_slice(s.as_bytes());\n        }\n\n        if new_buf.len() == uri.end() {\n            // buf unchanged, nothing is escaped, return None to avoid loop\n            return None;\n        }\n\n        new_buf.extend_from_slice(&buf[uri.end()..]);\n\n        Some(new_buf)\n    } else {\n        None\n    }\n}\n\n#[inline]\nfn parse_req_buffer<'buf>(\n    req: &mut httparse::Request<'_, 'buf>,\n    buf: &'buf [u8],\n) -> HeaderParseState {\n    use httparse::Result;\n\n    #[cfg(feature = \"patched_http1\")]\n    fn parse<'buf>(req: &mut httparse::Request<'_, 'buf>, buf: &'buf [u8]) -> Result<usize> {\n        req.parse_unchecked(buf)\n    }\n\n    #[cfg(not(feature = \"patched_http1\"))]\n    fn parse<'buf>(req: &mut httparse::Request<'_, 'buf>, buf: &'buf [u8]) -> Result<usize> {\n        req.parse(buf)\n    }\n\n    let res = match parse(req, buf) {\n        Ok(s) => s,\n        Err(e) => {\n            return HeaderParseState::Invalid(e);\n        }\n    };\n    match res {\n        httparse::Status::Complete(s) => HeaderParseState::Complete(s),\n        _ => HeaderParseState::Partial,\n    }\n}\n\n#[inline]\nfn http_resp_header_to_buf(\n    resp: &ResponseHeader,\n    buf: &mut BytesMut,\n) -> std::result::Result<(), ()> {\n    // Status-Line\n    let version = match resp.version {\n        Version::HTTP_09 => \"HTTP/0.9 \",\n        Version::HTTP_10 => \"HTTP/1.0 \",\n        Version::HTTP_11 => \"HTTP/1.1 \",\n        _ => {\n            return Err(()); /*TODO: unsupported version */\n        }\n    };\n    buf.put_slice(version.as_bytes());\n    let status = resp.status;\n    buf.put_slice(status.as_str().as_bytes());\n    buf.put_u8(b' ');\n    let reason = resp.get_reason_phrase();\n    if let Some(reason_buf) = reason {\n        buf.put_slice(reason_buf.as_bytes());\n    }\n    buf.put_slice(CRLF);\n\n    // headers\n    // TODO: style: make sure Server and Date headers are the first two\n    resp.header_to_h1_wire(buf);\n\n    buf.put_slice(CRLF);\n    Ok(())\n}\n\n#[cfg(test)]\nmod tests_stream {\n    use super::*;\n    use crate::protocols::http::v1::body::{BodyMode, ParseState};\n    use http::StatusCode;\n    use pingora_error::ErrorType;\n    use rstest::rstest;\n    use std::str;\n    use tokio_test::io::Builder;\n\n    fn init_log() {\n        let _ = env_logger::builder().is_test(true).try_init();\n    }\n\n    #[tokio::test]\n    async fn read_basic() {\n        init_log();\n        let input = b\"GET / HTTP/1.1\\r\\n\\r\\n\";\n        let mock_io = Builder::new().read(&input[..]).build();\n        let mut http_stream = HttpSession::new(Box::new(mock_io));\n        let res = http_stream.read_request().await;\n        assert_eq!(input.len(), res.unwrap().unwrap());\n        assert_eq!(0, http_stream.req_header().headers.len());\n    }\n\n    #[cfg(feature = \"patched_http1\")]\n    #[tokio::test]\n    async fn read_invalid_path() {\n        init_log();\n        let input = b\"GET /\\x01\\xF0\\x90\\x80 HTTP/1.1\\r\\n\\r\\n\";\n        let mock_io = Builder::new().read(&input[..]).build();\n        let mut http_stream = HttpSession::new(Box::new(mock_io));\n        let res = http_stream.read_request().await;\n        assert_eq!(input.len(), res.unwrap().unwrap());\n        assert_eq!(0, http_stream.req_header().headers.len());\n        assert_eq!(b\"/\\x01\\xF0\\x90\\x80\", http_stream.get_path());\n    }\n\n    #[tokio::test]\n    async fn read_2_buf() {\n        init_log();\n        let input1 = b\"GET / HTTP/1.1\\r\\n\";\n        let input2 = b\"Host: pingora.org\\r\\n\\r\\n\";\n        let mock_io = Builder::new().read(&input1[..]).read(&input2[..]).build();\n        let mut http_stream = HttpSession::new(Box::new(mock_io));\n        let res = http_stream.read_request().await;\n        assert_eq!(input1.len() + input2.len(), res.unwrap().unwrap());\n        assert_eq!(\n            input1.len() + input2.len(),\n            http_stream.raw_header.as_ref().unwrap().len()\n        );\n        assert_eq!(1, http_stream.req_header().headers.len());\n        assert_eq!(Some(&Method::GET), http_stream.get_method());\n        assert_eq!(b\"/\", http_stream.get_path());\n        assert_eq!(Version::HTTP_11, http_stream.req_header().version);\n\n        assert_eq!(b\"pingora.org\", http_stream.get_header_bytes(\"Host\"));\n    }\n\n    #[tokio::test]\n    async fn read_with_body_content_length() {\n        init_log();\n        let input1 = b\"GET / HTTP/1.1\\r\\n\";\n        let input2 = b\"Host: pingora.org\\r\\nContent-Length: 3\\r\\n\\r\\n\";\n        let input3 = b\"abc\";\n        let mock_io = Builder::new()\n            .read(&input1[..])\n            .read(&input2[..])\n            .read(&input3[..])\n            .build();\n        let mut http_stream = HttpSession::new(Box::new(mock_io));\n        http_stream.read_request().await.unwrap();\n        let res = http_stream.read_body_bytes().await.unwrap().unwrap();\n        assert_eq!(res, input3.as_slice());\n        assert_eq!(http_stream.body_reader.body_state, ParseState::Complete(3));\n        assert_eq!(http_stream.body_bytes_read(), 3);\n    }\n\n    #[tokio::test]\n    #[should_panic(expected = \"There is still data left to read.\")]\n    async fn read_with_body_timeout() {\n        init_log();\n        let input1 = b\"GET / HTTP/1.1\\r\\n\";\n        let input2 = b\"Host: pingora.org\\r\\nContent-Length: 3\\r\\n\\r\\n\";\n        let input3 = b\"abc\";\n        let mock_io = Builder::new()\n            .read(&input1[..])\n            .read(&input2[..])\n            .wait(Duration::from_secs(2))\n            .read(&input3[..])\n            .build();\n        let mut http_stream = HttpSession::new(Box::new(mock_io));\n        http_stream.read_timeout = Some(Duration::from_secs(1));\n        http_stream.read_request().await.unwrap();\n        let res = http_stream.read_body_bytes().await;\n        assert_eq!(http_stream.body_bytes_read(), 0);\n        assert_eq!(res.unwrap_err().etype(), &ReadTimedout);\n    }\n\n    #[tokio::test]\n    async fn read_with_body_content_length_single_read() {\n        init_log();\n        let input1 = b\"GET / HTTP/1.1\\r\\n\";\n        let input2 = b\"Host: pingora.org\\r\\nContent-Length: 3\\r\\n\\r\\nabc\";\n        let mock_io = Builder::new().read(&input1[..]).read(&input2[..]).build();\n        let mut http_stream = HttpSession::new(Box::new(mock_io));\n        http_stream.read_request().await.unwrap();\n        let res = http_stream.read_body_bytes().await.unwrap().unwrap();\n        assert_eq!(res, b\"abc\".as_slice());\n        assert_eq!(http_stream.body_reader.body_state, ParseState::Complete(3));\n        assert_eq!(http_stream.body_bytes_read(), 3);\n    }\n\n    #[tokio::test]\n    #[should_panic(expected = \"There is still data left to read.\")]\n    async fn read_with_body_http10() {\n        init_log();\n        let input1 = b\"GET / HTTP/1.0\\r\\n\";\n        let input2 = b\"Host: pingora.org\\r\\n\\r\\n\";\n        let input3 = b\"a\"; // This should NOT be read as body\n        let input4 = b\"\"; // simulating close - should also NOT be reached\n        let mock_io = Builder::new()\n            .read(&input1[..])\n            .read(&input2[..])\n            .read(&input3[..])\n            .read(&input4[..])\n            .build();\n        let mut http_stream = HttpSession::new(Box::new(mock_io));\n        http_stream.read_request().await.unwrap();\n        let res = http_stream.read_body_bytes().await.unwrap();\n        assert!(res.is_none());\n        assert_eq!(http_stream.body_bytes_read(), 0);\n        assert_eq!(http_stream.body_reader.body_state, ParseState::Complete(0));\n    }\n\n    #[tokio::test]\n    async fn read_with_body_http10_single_read() {\n        init_log();\n        // should have 0 body, even when data follows the headers\n        let input1 = b\"GET / HTTP/1.0\\r\\n\";\n        let input2 = b\"Host: pingora.org\\r\\n\\r\\na\";\n        let mock_io = Builder::new().read(&input1[..]).read(&input2[..]).build();\n        let mut http_stream = HttpSession::new(Box::new(mock_io));\n        http_stream.read_request().await.unwrap();\n        let res = http_stream.read_body_bytes().await.unwrap();\n        assert!(res.is_none());\n        assert_eq!(http_stream.body_bytes_read(), 0);\n        assert_eq!(http_stream.body_reader.body_state, ParseState::Complete(0));\n        assert_eq!(http_stream.body_reader.get_body_overread().unwrap(), b\"a\");\n    }\n\n    #[tokio::test]\n    async fn read_http11_default_no_body() {\n        init_log();\n        let input1 = b\"GET / HTTP/1.1\\r\\n\";\n        let input2 = b\"Host: pingora.org\\r\\n\\r\\n\";\n        let mock_io = Builder::new().read(&input1[..]).read(&input2[..]).build();\n        let mut http_stream = HttpSession::new(Box::new(mock_io));\n        http_stream.read_request().await.unwrap();\n        let res = http_stream.read_body_bytes().await.unwrap();\n        assert!(res.is_none());\n        assert_eq!(http_stream.body_bytes_read(), 0);\n        assert_eq!(http_stream.body_reader.body_state, ParseState::Complete(0));\n    }\n\n    #[tokio::test]\n    async fn read_http10_with_content_length() {\n        init_log();\n        let input1 = b\"POST / HTTP/1.0\\r\\n\";\n        let input2 = b\"Host: pingora.org\\r\\nContent-Length: 3\\r\\n\\r\\n\";\n        let input3 = b\"abc\";\n        let mock_io = Builder::new()\n            .read(&input1[..])\n            .read(&input2[..])\n            .read(&input3[..])\n            .build();\n        let mut http_stream = HttpSession::new(Box::new(mock_io));\n        http_stream.read_request().await.unwrap();\n        let res = http_stream.read_body_bytes().await.unwrap().unwrap();\n        assert_eq!(res, input3.as_slice());\n        assert_eq!(http_stream.body_reader.body_state, ParseState::Complete(3));\n        assert_eq!(http_stream.body_bytes_read(), 3);\n    }\n\n    #[tokio::test]\n    async fn read_with_body_chunked_0_incomplete() {\n        init_log();\n        let input1 = b\"GET / HTTP/1.1\\r\\n\";\n        let input2 = b\"Host: pingora.org\\r\\nTransfer-Encoding: chunked\\r\\n\\r\\n\";\n        let input3 = b\"0\\r\\n\";\n        let mock_io = Builder::new()\n            .read(&input1[..])\n            .read(&input2[..])\n            .read(&input3[..])\n            .build();\n        let mut http_stream = HttpSession::new(Box::new(mock_io));\n        http_stream.read_request().await.unwrap();\n        assert!(http_stream.is_chunked_encoding());\n        let res = http_stream.read_body_bytes().await.unwrap().unwrap();\n        assert_eq!(res, b\"\".as_slice());\n        let e = http_stream.read_body_bytes().await.unwrap_err();\n        assert_eq!(*e.etype(), ErrorType::ConnectionClosed);\n        assert_eq!(http_stream.body_reader.body_state, ParseState::Done(0));\n    }\n\n    #[tokio::test]\n    async fn read_with_body_chunked_0_extra() {\n        init_log();\n        let input1 = b\"GET / HTTP/1.1\\r\\n\";\n        let input2 = b\"Host: pingora.org\\r\\nTransfer-Encoding: chunked\\r\\n\\r\\n\";\n        let input3 = b\"0\\r\\n\";\n        let input4 = b\"abc\";\n        let mock_io = Builder::new()\n            .read(&input1[..])\n            .read(&input2[..])\n            .read(&input3[..])\n            .read(&input4[..])\n            .build();\n        let mut http_stream = HttpSession::new(Box::new(mock_io));\n        http_stream.read_request().await.unwrap();\n        assert!(http_stream.is_chunked_encoding());\n        let res = http_stream.read_body_bytes().await.unwrap().unwrap();\n        assert_eq!(res, b\"\".as_slice());\n        let res = http_stream.read_body_bytes().await.unwrap().unwrap();\n        assert_eq!(res, b\"\".as_slice());\n        let e = http_stream.read_body_bytes().await.unwrap_err();\n        assert_eq!(*e.etype(), ErrorType::ConnectionClosed);\n        assert_eq!(http_stream.body_reader.body_state, ParseState::Done(0));\n    }\n\n    #[tokio::test]\n    async fn read_with_body_chunked_single_read() {\n        init_log();\n        let input1 = b\"GET / HTTP/1.1\\r\\n\";\n        let input2 = b\"Host: pingora.org\\r\\nTransfer-Encoding: chunked\\r\\n\\r\\n1\\r\\na\\r\\n\";\n        let input3 = b\"0\\r\\n\\r\\n\";\n        let mock_io = Builder::new()\n            .read(&input1[..])\n            .read(&input2[..])\n            .read(&input3[..])\n            .build();\n        let mut http_stream = HttpSession::new(Box::new(mock_io));\n        http_stream.read_request().await.unwrap();\n        assert!(http_stream.is_chunked_encoding());\n        let res = http_stream.read_body_bytes().await.unwrap().unwrap();\n        assert_eq!(res, b\"a\".as_slice());\n        assert_eq!(\n            http_stream.body_reader.body_state,\n            ParseState::Chunked(1, 0, 0, 0)\n        );\n        let res = http_stream.read_body_bytes().await.unwrap();\n        assert!(res.is_none());\n        assert_eq!(http_stream.body_bytes_read(), 1);\n        assert_eq!(http_stream.body_reader.body_state, ParseState::Complete(1));\n    }\n\n    #[tokio::test]\n    async fn read_with_body_chunked_single_read_extra() {\n        init_log();\n        let input1 = b\"GET / HTTP/1.1\\r\\n\";\n        let input2 = b\"Host: pingora.org\\r\\nTransfer-Encoding: chunked\\r\\n\\r\\n1\\r\\na\\r\\n\";\n        let input3 = b\"0\\r\\n\\r\\nabc\";\n        let mock_io = Builder::new()\n            .read(&input1[..])\n            .read(&input2[..])\n            .read(&input3[..])\n            .build();\n        let mut http_stream = HttpSession::new(Box::new(mock_io));\n        http_stream.read_request().await.unwrap();\n        assert!(http_stream.is_chunked_encoding());\n        let res = http_stream.read_body_bytes().await.unwrap().unwrap();\n        assert_eq!(res, b\"a\".as_slice());\n        assert_eq!(\n            http_stream.body_reader.body_state,\n            ParseState::Chunked(1, 0, 0, 0)\n        );\n        let res = http_stream.read_body_bytes().await.unwrap();\n        assert!(res.is_none());\n        assert_eq!(http_stream.body_bytes_read(), 1);\n        assert_eq!(http_stream.body_reader.body_state, ParseState::Complete(1));\n        assert_eq!(http_stream.body_reader.get_body_overread().unwrap(), b\"abc\");\n    }\n\n    #[rstest]\n    #[case(None, None)]\n    #[case(Some(\"transfer-encoding\"), None)]\n    #[case(Some(\"transfer-encoding\"), Some(\"CONTENT-LENGTH\"))]\n    #[case(Some(\"TRANSFER-ENCODING\"), Some(\"CONTENT-LENGTH\"))]\n    #[case(Some(\"TRANSFER-ENCODING\"), None)]\n    #[case(None, Some(\"CONTENT-LENGTH\"))]\n    #[case(Some(\"TRANSFER-ENCODING\"), Some(\"content-length\"))]\n    #[case(None, Some(\"content-length\"))]\n    #[tokio::test]\n    async fn transfer_encoding_and_content_length_disallowed(\n        #[case] transfer_encoding_header: Option<&str>,\n        #[case] content_length_header: Option<&str>,\n    ) {\n        init_log();\n        let input1 = b\"GET / HTTP/1.1\\r\\n\";\n        let mut input2 = \"Host: pingora.org\\r\\n\".to_owned();\n\n        if let Some(transfer_encoding) = transfer_encoding_header {\n            input2 += &format!(\"{transfer_encoding}: chunked\\r\\n\");\n        }\n        if let Some(content_length) = content_length_header {\n            input2 += &format!(\"{content_length}: 4\\r\\n\")\n        }\n\n        input2 += \"\\r\\n3e\\r\\na\\r\\n\";\n        let mock_io = Builder::new()\n            .read(&input1[..])\n            .read(input2.as_bytes())\n            .build();\n        let mut http_stream = HttpSession::new(Box::new(mock_io));\n        let _ = http_stream.read_request().await.unwrap();\n\n        match (content_length_header, transfer_encoding_header) {\n            (Some(_) | None, Some(_)) => {\n                assert!(http_stream.get_header(TRANSFER_ENCODING).is_some());\n                assert!(http_stream.get_header(CONTENT_LENGTH).is_none());\n            }\n            (Some(_), None) => {\n                assert!(http_stream.get_header(TRANSFER_ENCODING).is_none());\n                assert!(http_stream.get_header(CONTENT_LENGTH).is_some());\n            }\n            _ => {\n                assert!(http_stream.get_header(CONTENT_LENGTH).is_none());\n                assert!(http_stream.get_header(TRANSFER_ENCODING).is_none());\n            }\n        }\n    }\n\n    #[rstest]\n    #[case::negative(\"-1\")]\n    #[case::not_a_number(\"abc\")]\n    #[case::float(\"1.5\")]\n    #[case::empty(\"\")]\n    #[case::spaces(\"  \")]\n    #[case::mixed(\"123abc\")]\n    #[tokio::test]\n    async fn validate_request_rejects_invalid_content_length(#[case] invalid_value: &str) {\n        init_log();\n        let input = format!(\n            \"POST / HTTP/1.1\\r\\nHost: pingora.org\\r\\nContent-Length: {}\\r\\n\\r\\n\",\n            invalid_value\n        );\n        let mock_io = Builder::new().read(input.as_bytes()).build();\n        let mut http_stream = HttpSession::new(Box::new(mock_io));\n        // read_request calls validate_request internally, so it should fail here\n        let res = http_stream.read_request().await;\n        assert!(res.is_err());\n        assert_eq!(res.unwrap_err().etype(), &InvalidHTTPHeader);\n    }\n\n    #[rstest]\n    #[case::valid_zero(\"0\")]\n    #[case::valid_small(\"123\")]\n    #[case::valid_large(\"999999\")]\n    #[tokio::test]\n    async fn validate_request_accepts_valid_content_length(#[case] valid_value: &str) {\n        init_log();\n        let input = format!(\n            \"POST / HTTP/1.1\\r\\nHost: pingora.org\\r\\nContent-Length: {}\\r\\n\\r\\n\",\n            valid_value\n        );\n        let mock_io = Builder::new().read(input.as_bytes()).build();\n        let mut http_stream = HttpSession::new(Box::new(mock_io));\n        let res = http_stream.read_request().await;\n        assert!(res.is_ok());\n    }\n\n    #[tokio::test]\n    async fn validate_request_accepts_no_content_length() {\n        init_log();\n        let input = b\"GET / HTTP/1.1\\r\\nHost: pingora.org\\r\\n\\r\\n\";\n        let mock_io = Builder::new().read(&input[..]).build();\n        let mut http_stream = HttpSession::new(Box::new(mock_io));\n        let res = http_stream.read_request().await;\n        assert!(res.is_ok());\n    }\n\n    #[tokio::test]\n    #[should_panic(expected = \"There is still data left to read.\")]\n    async fn read_invalid() {\n        let input1 = b\"GET / HTP/1.1\\r\\n\";\n        let input2 = b\"Host: pingora.org\\r\\n\\r\\n\";\n        let mock_io = Builder::new().read(&input1[..]).read(&input2[..]).build();\n        let mut http_stream = HttpSession::new(Box::new(mock_io));\n        let res = http_stream.read_request().await;\n        assert_eq!(&InvalidHTTPHeader, res.unwrap_err().etype());\n    }\n\n    #[tokio::test]\n    async fn read_invalid_header_end() {\n        let input = b\"POST / HTTP/1.1\\r\\nHost: pingora.org\\r\\nContent-Length: 3\\r\\r\\nConnection: keep-alive\\r\\n\\r\\nabc\";\n        let mock_io = Builder::new().read(&input[..]).build();\n        let mut http_stream = HttpSession::new(Box::new(mock_io));\n        let res = http_stream.read_request().await;\n        assert_eq!(&InvalidHTTPHeader, res.unwrap_err().etype());\n    }\n\n    async fn build_upgrade_req(upgrade: &str, conn: &str) -> HttpSession {\n        let input = format!(\"GET / HTTP/1.1\\r\\nHost: pingora.org\\r\\nUpgrade: {upgrade}\\r\\nConnection: {conn}\\r\\n\\r\\n\");\n        let mock_io = Builder::new().read(input.as_bytes()).build();\n        let mut http_stream = HttpSession::new(Box::new(mock_io));\n        http_stream.read_request().await.unwrap();\n        http_stream\n    }\n\n    #[tokio::test]\n    async fn read_upgrade_req() {\n        // http 1.0\n        let input = b\"GET / HTTP/1.0\\r\\nHost: pingora.org\\r\\nUpgrade: websocket\\r\\nConnection: upgrade\\r\\n\\r\\n\";\n        let mock_io = Builder::new().read(&input[..]).build();\n        let mut http_stream = HttpSession::new(Box::new(mock_io));\n        http_stream.read_request().await.unwrap();\n        assert!(!http_stream.is_upgrade_req());\n\n        // different method\n        let input = b\"POST / HTTP/1.1\\r\\nHost: pingora.org\\r\\nUpgrade: websocket\\r\\nConnection: upgrade\\r\\n\\r\\n\";\n        let mock_io = Builder::new().read(&input[..]).build();\n        let mut http_stream = HttpSession::new(Box::new(mock_io));\n        http_stream.read_request().await.unwrap();\n        assert!(http_stream.is_upgrade_req());\n\n        // missing upgrade header\n        let input = b\"GET / HTTP/1.1\\r\\nHost: pingora.org\\r\\nConnection: upgrade\\r\\n\\r\\n\";\n        let mock_io = Builder::new().read(&input[..]).build();\n        let mut http_stream = HttpSession::new(Box::new(mock_io));\n        http_stream.read_request().await.unwrap();\n        assert!(!http_stream.is_upgrade_req());\n\n        // no connection header\n        let input = b\"GET / HTTP/1.1\\r\\nHost: pingora.org\\r\\nUpgrade: WebSocket\\r\\n\\r\\n\";\n        let mock_io = Builder::new().read(&input[..]).build();\n        let mut http_stream = HttpSession::new(Box::new(mock_io));\n        http_stream.read_request().await.unwrap();\n        assert!(http_stream.is_upgrade_req());\n\n        assert!(build_upgrade_req(\"websocket\", \"Upgrade\")\n            .await\n            .is_upgrade_req());\n\n        // mixed case\n        assert!(build_upgrade_req(\"WebSocket\", \"Upgrade\")\n            .await\n            .is_upgrade_req());\n    }\n\n    const POST_CL_UPGRADE_REQ: &[u8] = b\"POST / HTTP/1.1\\r\\nHost: pingora.org\\r\\nUpgrade: websocket\\r\\nConnection: upgrade\\r\\nContent-Length: 10\\r\\n\\r\\n\";\n    const POST_BODY_DATA: &[u8] = b\"abcdefghij\";\n    const POST_CHUNKED_UPGRADE_REQ: &[u8] = b\"POST / HTTP/1.1\\r\\nHost: pingora.org\\r\\nUpgrade: websocket\\r\\nConnection: upgrade\\r\\nTransfer-Encoding: chunked\\r\\n\\r\\n\";\n    const POST_BODY_DATA_CHUNKED: &[u8] = b\"3\\r\\nabc\\r\\n7\\r\\ndefghij\\r\\n0\\r\\n\\r\\n\";\n\n    #[rstest]\n    #[case::content_length(POST_CL_UPGRADE_REQ, POST_BODY_DATA, POST_BODY_DATA)]\n    #[case::chunked(POST_CHUNKED_UPGRADE_REQ, POST_BODY_DATA, POST_BODY_DATA_CHUNKED)]\n    #[tokio::test]\n    async fn read_upgrade_req_with_body(\n        #[case] header: &[u8],\n        #[case] body: &[u8],\n        #[case] body_wire: &[u8],\n    ) {\n        let ws_data = b\"data\";\n        let mock_io = Builder::new()\n            .read(header)\n            .read(body_wire)\n            .write(b\"HTTP/1.1 101 Switching Protocols\\r\\n\\r\\n\")\n            .read(&ws_data[..])\n            .build();\n        let mut http_stream = HttpSession::new(Box::new(mock_io));\n        http_stream.read_request().await.unwrap();\n        assert!(http_stream.is_upgrade_req());\n        // request has body\n        assert!(!http_stream.is_body_done());\n\n        let mut buf = vec![];\n        while let Some(b) = http_stream.read_body_bytes().await.unwrap() {\n            buf.put_slice(&b);\n        }\n        assert_eq!(buf, body);\n        assert_eq!(http_stream.body_reader.body_state, ParseState::Complete(10));\n        assert_eq!(http_stream.body_bytes_read(), 10);\n\n        assert!(http_stream.is_body_done());\n\n        let mut response = ResponseHeader::build(StatusCode::SWITCHING_PROTOCOLS, None).unwrap();\n        response.set_version(http::Version::HTTP_11);\n        http_stream\n            .write_response_header(Box::new(response))\n            .await\n            .unwrap();\n        // body reader type switches\n        assert!(!http_stream.is_body_done());\n\n        // now the ws data\n        let buf = http_stream.read_body_bytes().await.unwrap().unwrap();\n        assert_eq!(buf, ws_data.as_slice());\n        assert!(!http_stream.is_body_done());\n\n        // EOF ends body\n        assert!(http_stream.read_body_bytes().await.unwrap().is_none());\n        assert!(http_stream.is_body_done());\n    }\n\n    #[rstest]\n    #[case::content_length(POST_CL_UPGRADE_REQ, POST_BODY_DATA, POST_BODY_DATA)]\n    #[case::chunked(POST_CHUNKED_UPGRADE_REQ, POST_BODY_DATA, POST_BODY_DATA_CHUNKED)]\n    #[tokio::test]\n    async fn read_upgrade_req_with_body_extra(\n        #[case] header: &[u8],\n        #[case] body: &[u8],\n        #[case] body_wire: &[u8],\n    ) {\n        let ws_data = b\"data\";\n        let data_wire = [body_wire, ws_data.as_slice()].concat();\n        let mock_io = Builder::new()\n            .read(header)\n            .read(&data_wire[..])\n            .write(b\"HTTP/1.1 101 Switching Protocols\\r\\n\\r\\n\")\n            .build();\n        let mut http_stream = HttpSession::new(Box::new(mock_io));\n        http_stream.read_request().await.unwrap();\n        assert!(http_stream.is_upgrade_req());\n        // request has body\n        assert!(!http_stream.is_body_done());\n\n        let mut buf = vec![];\n        while let Some(b) = http_stream.read_body_bytes().await.unwrap() {\n            buf.put_slice(&b);\n        }\n        assert_eq!(buf, body);\n        assert_eq!(http_stream.body_reader.body_state, ParseState::Complete(10));\n        assert_eq!(http_stream.body_bytes_read(), 10);\n\n        assert!(http_stream.is_body_done());\n\n        let mut response = ResponseHeader::build(StatusCode::SWITCHING_PROTOCOLS, None).unwrap();\n        response.set_version(http::Version::HTTP_11);\n        http_stream\n            .write_response_header(Box::new(response))\n            .await\n            .unwrap();\n        // body reader type switches\n        assert!(!http_stream.is_body_done());\n\n        // now the ws data\n        let buf = http_stream.read_body_bytes().await.unwrap().unwrap();\n        assert_eq!(buf, ws_data.as_slice());\n        assert!(!http_stream.is_body_done());\n\n        // EOF ends body\n        assert!(http_stream.read_body_bytes().await.unwrap().is_none());\n        assert!(http_stream.is_body_done());\n    }\n\n    #[rstest]\n    #[case::content_length(POST_CL_UPGRADE_REQ, POST_BODY_DATA, POST_BODY_DATA)]\n    #[case::chunked(POST_CHUNKED_UPGRADE_REQ, POST_BODY_DATA, POST_BODY_DATA_CHUNKED)]\n    #[tokio::test]\n    async fn read_upgrade_req_with_preread_body(\n        #[case] header: &[u8],\n        #[case] body: &[u8],\n        #[case] body_wire: &[u8],\n    ) {\n        let ws_data = b\"data\";\n        let data_wire = [header, body_wire, ws_data.as_slice()].concat();\n        let mock_io = Builder::new()\n            .read(&data_wire[..])\n            .write(b\"HTTP/1.1 101 Switching Protocols\\r\\n\\r\\n\")\n            .build();\n        let mut http_stream = HttpSession::new(Box::new(mock_io));\n        http_stream.read_request().await.unwrap();\n        assert!(http_stream.is_upgrade_req());\n        // request has body\n        assert!(!http_stream.is_body_done());\n\n        let mut buf = vec![];\n        while let Some(b) = http_stream.read_body_bytes().await.unwrap() {\n            buf.put_slice(&b);\n        }\n        assert_eq!(buf, body);\n        assert_eq!(http_stream.body_reader.body_state, ParseState::Complete(10));\n        assert_eq!(http_stream.body_bytes_read(), 10);\n\n        assert!(http_stream.is_body_done());\n\n        let mut response = ResponseHeader::build(StatusCode::SWITCHING_PROTOCOLS, None).unwrap();\n        response.set_version(http::Version::HTTP_11);\n        http_stream\n            .write_response_header(Box::new(response))\n            .await\n            .unwrap();\n        // body reader type switches\n        assert!(!http_stream.is_body_done());\n\n        // now the ws data\n        let buf = http_stream.read_body_bytes().await.unwrap().unwrap();\n        assert_eq!(buf, ws_data.as_slice());\n        assert!(!http_stream.is_body_done());\n\n        // EOF ends body\n        assert!(http_stream.read_body_bytes().await.unwrap().is_none());\n        assert!(http_stream.is_body_done());\n    }\n\n    #[rstest]\n    #[case::content_length(POST_CL_UPGRADE_REQ, POST_BODY_DATA)]\n    #[case::chunked(POST_CHUNKED_UPGRADE_REQ, POST_BODY_DATA_CHUNKED)]\n    #[tokio::test]\n    async fn read_upgrade_req_with_preread_body_after_101(\n        #[case] header: &[u8],\n        #[case] body_wire: &[u8],\n    ) {\n        let ws_data = b\"data\";\n        let data_wire = [header, body_wire, ws_data.as_slice()].concat();\n        let mock_io = Builder::new()\n            .read(&data_wire[..])\n            .write(b\"HTTP/1.1 101 Switching Protocols\\r\\n\\r\\n\")\n            .build();\n        let mut http_stream = HttpSession::new(Box::new(mock_io));\n        http_stream.read_request().await.unwrap();\n        assert!(http_stream.is_upgrade_req());\n        // request has body\n        assert!(!http_stream.is_body_done());\n\n        let mut response = ResponseHeader::build(StatusCode::SWITCHING_PROTOCOLS, None).unwrap();\n        response.set_version(http::Version::HTTP_11);\n        http_stream\n            .write_response_header(Box::new(response))\n            .await\n            .unwrap();\n        // body reader type switches to http10\n        assert!(!http_stream.is_body_done());\n\n        let mut buf = vec![];\n        while let Some(b) = http_stream.read_body_bytes().await.unwrap() {\n            buf.put_slice(&b);\n        }\n        let expected_body = [body_wire, ws_data.as_slice()].concat();\n        assert_eq!(buf, expected_body.as_bytes());\n        assert_eq!(http_stream.body_bytes_read(), expected_body.len());\n        assert!(http_stream.is_body_done());\n    }\n\n    #[tokio::test]\n    async fn read_upgrade_req_with_1xx_response() {\n        let input = b\"GET / HTTP/1.1\\r\\nHost: pingora.org\\r\\nUpgrade: websocket\\r\\nConnection: upgrade\\r\\n\\r\\n\";\n        let mock_io = Builder::new()\n            .read(&input[..])\n            .write(b\"HTTP/1.1 100 Continue\\r\\n\\r\\n\")\n            .write(b\"HTTP/1.1 101 Switching Protocols\\r\\n\\r\\n\")\n            .build();\n        let mut http_stream = HttpSession::new(Box::new(mock_io));\n        http_stream.read_request().await.unwrap();\n        assert!(http_stream.is_upgrade_req());\n        let mut response = ResponseHeader::build(StatusCode::CONTINUE, None).unwrap();\n        response.set_version(http::Version::HTTP_11);\n        http_stream\n            .write_response_header(Box::new(response))\n            .await\n            .unwrap();\n        // 100 won't affect body state\n        // current GET request is done\n        assert!(http_stream.is_body_done());\n\n        let mut response = ResponseHeader::build(StatusCode::SWITCHING_PROTOCOLS, None).unwrap();\n        response.set_version(http::Version::HTTP_11);\n        http_stream\n            .write_response_header(Box::new(response))\n            .await\n            .unwrap();\n        // body reader type switches\n        assert!(!http_stream.is_body_done());\n        // EOF ends body\n        assert!(http_stream.read_body_bytes().await.unwrap().is_none());\n        assert!(http_stream.is_body_done());\n    }\n\n    #[tokio::test]\n    async fn test_upgrade_without_content_length_with_ws_data() {\n        let request = b\"GET / HTTP/1.1\\r\\nHost: pingora.org\\r\\nUpgrade: websocket\\r\\nConnection: upgrade\\r\\n\\r\\n\";\n        let ws_data = b\"websocket data\";\n\n        let mock_io = Builder::new()\n            .read(request)\n            .write(b\"HTTP/1.1 101 Switching Protocols\\r\\n\\r\\n\")\n            .read(ws_data) // websocket data sent after 101\n            .build();\n\n        let mut http_stream = HttpSession::new(Box::new(mock_io));\n        http_stream.read_request().await.unwrap();\n        assert!(http_stream.is_upgrade_req());\n\n        // When enabled (default), is_body_done() is called before the upgrade\n        http_stream.set_close_on_response_before_downstream_finish(false);\n\n        // Send 101 response - this is where the bug occurs\n        let mut response = ResponseHeader::build(StatusCode::SWITCHING_PROTOCOLS, None).unwrap();\n        response.set_version(http::Version::HTTP_11);\n        http_stream\n            .write_response_header(Box::new(response))\n            .await\n            .unwrap();\n\n        assert_eq!(\n            http_stream.body_reader.body_state,\n            ParseState::UntilClose(0),\n            \"Body reader should be in UntilClose mode after 101 for upgraded connections\"\n        );\n\n        // Try to read websocket data\n        let mut buf = vec![];\n        while let Some(b) = http_stream.read_body_bytes().await.unwrap() {\n            buf.put_slice(&b);\n        }\n        assert_eq!(buf, ws_data, \"Expected to read websocket data after 101\");\n    }\n\n    #[tokio::test]\n    async fn set_server_keepalive() {\n        // close\n        let input = b\"GET / HTTP/1.1\\r\\nHost: pingora.org\\r\\nConnection: close\\r\\n\\r\\n\";\n        let mock_io = Builder::new().read(&input[..]).build();\n        let mut http_stream = HttpSession::new(Box::new(mock_io));\n        http_stream.read_request().await.unwrap();\n        // verify close\n        assert_eq!(http_stream.keepalive_timeout, KeepaliveStatus::Off);\n        http_stream.set_server_keepalive(Some(60));\n        // verify no change on override\n        assert_eq!(http_stream.keepalive_timeout, KeepaliveStatus::Off);\n\n        // explicit keep-alive\n        let input = b\"GET / HTTP/1.1\\r\\nHost: pingora.org\\r\\nConnection: keep-alive\\r\\n\\r\\n\";\n        let mock_io = Builder::new().read(&input[..]).build();\n        let mut http_stream = HttpSession::new(Box::new(mock_io));\n        // default is infinite for 1.1\n        http_stream.read_request().await.unwrap();\n        assert_eq!(http_stream.keepalive_timeout, KeepaliveStatus::Infinite);\n        http_stream.set_server_keepalive(Some(60));\n        // override respected\n        assert_eq!(\n            http_stream.keepalive_timeout,\n            KeepaliveStatus::Timeout(Duration::from_secs(60))\n        );\n\n        // not specified\n        let input = b\"GET / HTTP/1.1\\r\\nHost: pingora.org\\r\\n\\r\\n\";\n        let mock_io = Builder::new().read(&input[..]).build();\n        let mut http_stream = HttpSession::new(Box::new(mock_io));\n        http_stream.read_request().await.unwrap();\n        // default is infinite for 1.1\n        assert_eq!(http_stream.keepalive_timeout, KeepaliveStatus::Infinite);\n        http_stream.set_server_keepalive(Some(60));\n        // override respected\n        assert_eq!(\n            http_stream.keepalive_timeout,\n            KeepaliveStatus::Timeout(Duration::from_secs(60))\n        );\n    }\n\n    #[tokio::test]\n    async fn write() {\n        let read_wire = b\"GET / HTTP/1.1\\r\\n\\r\\n\";\n        let write_expected = b\"HTTP/1.1 200 OK\\r\\nFoo: Bar\\r\\n\\r\\n\";\n        let mock_io = Builder::new().read(read_wire).write(write_expected).build();\n        let mut http_stream = HttpSession::new(Box::new(mock_io));\n        http_stream.read_request().await.unwrap();\n        let mut new_response = ResponseHeader::build(StatusCode::OK, None).unwrap();\n        new_response.append_header(\"Foo\", \"Bar\").unwrap();\n        http_stream.update_resp_headers = false;\n        http_stream\n            .write_response_header_ref(&new_response)\n            .await\n            .unwrap();\n    }\n\n    #[tokio::test]\n    async fn write_custom_reason() {\n        let read_wire = b\"GET / HTTP/1.1\\r\\n\\r\\n\";\n        let write_expected = b\"HTTP/1.1 200 Just Fine\\r\\nFoo: Bar\\r\\n\\r\\n\";\n        let mock_io = Builder::new().read(read_wire).write(write_expected).build();\n        let mut http_stream = HttpSession::new(Box::new(mock_io));\n        http_stream.read_request().await.unwrap();\n        let mut new_response = ResponseHeader::build(StatusCode::OK, None).unwrap();\n        new_response.set_reason_phrase(Some(\"Just Fine\")).unwrap();\n        new_response.append_header(\"Foo\", \"Bar\").unwrap();\n        http_stream.update_resp_headers = false;\n        http_stream\n            .write_response_header_ref(&new_response)\n            .await\n            .unwrap();\n    }\n\n    #[tokio::test]\n    async fn write_informational() {\n        let read_wire = b\"GET / HTTP/1.1\\r\\n\\r\\n\";\n        let write_expected = b\"HTTP/1.1 100 Continue\\r\\n\\r\\nHTTP/1.1 200 OK\\r\\nFoo: Bar\\r\\n\\r\\n\";\n        let mock_io = Builder::new().read(read_wire).write(write_expected).build();\n        let mut http_stream = HttpSession::new(Box::new(mock_io));\n        http_stream.read_request().await.unwrap();\n        let response_100 = ResponseHeader::build(StatusCode::CONTINUE, None).unwrap();\n        http_stream\n            .write_response_header_ref(&response_100)\n            .await\n            .unwrap();\n        let mut response_200 = ResponseHeader::build(StatusCode::OK, None).unwrap();\n        response_200.append_header(\"Foo\", \"Bar\").unwrap();\n        http_stream.update_resp_headers = false;\n        http_stream\n            .write_response_header_ref(&response_200)\n            .await\n            .unwrap();\n    }\n\n    #[tokio::test]\n    async fn write_informational_ignored() {\n        let read_wire = b\"GET / HTTP/1.1\\r\\n\\r\\n\";\n        let write_expected = b\"HTTP/1.1 200 OK\\r\\nFoo: Bar\\r\\n\\r\\n\";\n        let mock_io = Builder::new().read(read_wire).write(write_expected).build();\n        let mut http_stream = HttpSession::new(Box::new(mock_io));\n        // ignore the 100 Continue\n        http_stream.ignore_info_resp = true;\n        http_stream.read_request().await.unwrap();\n        let response_100 = ResponseHeader::build(StatusCode::CONTINUE, None).unwrap();\n        http_stream\n            .write_response_header_ref(&response_100)\n            .await\n            .unwrap();\n        let mut response_200 = ResponseHeader::build(StatusCode::OK, None).unwrap();\n        response_200.append_header(\"Foo\", \"Bar\").unwrap();\n        http_stream.update_resp_headers = false;\n        http_stream\n            .write_response_header_ref(&response_200)\n            .await\n            .unwrap();\n    }\n\n    #[tokio::test]\n    async fn write_informational_100_not_ignored_if_expect_continue() {\n        let input = b\"GET / HTTP/1.1\\r\\nExpect: 100-continue\\r\\n\\r\\n\";\n        let output = b\"HTTP/1.1 100 Continue\\r\\n\\r\\nHTTP/1.1 200 OK\\r\\nFoo: Bar\\r\\n\\r\\n\";\n\n        let mock_io = Builder::new().read(&input[..]).write(output).build();\n        let mut http_stream = HttpSession::new(Box::new(mock_io));\n        http_stream.read_request().await.unwrap();\n        http_stream.ignore_info_resp = true;\n        // 100 Continue is not ignored due to Expect: 100-continue on request\n        let response_100 = ResponseHeader::build(StatusCode::CONTINUE, None).unwrap();\n        http_stream\n            .write_response_header_ref(&response_100)\n            .await\n            .unwrap();\n        let mut response_200 = ResponseHeader::build(StatusCode::OK, None).unwrap();\n        response_200.append_header(\"Foo\", \"Bar\").unwrap();\n        http_stream.update_resp_headers = false;\n        http_stream\n            .write_response_header_ref(&response_200)\n            .await\n            .unwrap();\n    }\n\n    #[tokio::test]\n    async fn write_informational_1xx_ignored_if_expect_continue() {\n        let input = b\"GET / HTTP/1.1\\r\\nExpect: 100-continue\\r\\n\\r\\n\";\n        let output = b\"HTTP/1.1 200 OK\\r\\nFoo: Bar\\r\\n\\r\\n\";\n\n        let mock_io = Builder::new().read(&input[..]).write(output).build();\n        let mut http_stream = HttpSession::new(Box::new(mock_io));\n        http_stream.read_request().await.unwrap();\n        http_stream.ignore_info_resp = true;\n        // 102 Processing is ignored\n        let response_102 = ResponseHeader::build(StatusCode::PROCESSING, None).unwrap();\n        http_stream\n            .write_response_header_ref(&response_102)\n            .await\n            .unwrap();\n        let mut response_200 = ResponseHeader::build(StatusCode::OK, None).unwrap();\n        response_200.append_header(\"Foo\", \"Bar\").unwrap();\n        http_stream.update_resp_headers = false;\n        http_stream\n            .write_response_header_ref(&response_200)\n            .await\n            .unwrap();\n    }\n\n    #[tokio::test]\n    async fn write_101_switching_protocol() {\n        let read_wire = b\"GET / HTTP/1.1\\r\\nUpgrade: websocket\\r\\n\\r\\n\";\n        let wire = b\"HTTP/1.1 101 Switching Protocols\\r\\nFoo: Bar\\r\\n\\r\\n\";\n        let wire_body = b\"nPAYLOAD\";\n        let mock_io = Builder::new()\n            .read(read_wire)\n            .write(wire)\n            .write(wire_body)\n            .build();\n        let mut http_stream = HttpSession::new(Box::new(mock_io));\n        http_stream.read_request().await.unwrap();\n        let mut response_101 =\n            ResponseHeader::build(StatusCode::SWITCHING_PROTOCOLS, None).unwrap();\n        response_101.append_header(\"Foo\", \"Bar\").unwrap();\n        http_stream\n            .write_response_header_ref(&response_101)\n            .await\n            .unwrap();\n        assert_eq!(http_stream.body_writer.body_mode, BodyMode::UntilClose(0));\n\n        let n = http_stream.write_body(wire_body).await.unwrap().unwrap();\n        assert_eq!(wire_body.len(), n);\n        assert_eq!(http_stream.body_writer.body_mode, BodyMode::UntilClose(n));\n\n        // this write should be ignored\n        let response_502 = ResponseHeader::build(StatusCode::BAD_GATEWAY, None).unwrap();\n        http_stream\n            .write_response_header_ref(&response_502)\n            .await\n            .unwrap();\n    }\n\n    #[tokio::test]\n    async fn write_body_cl() {\n        let read_wire = b\"GET / HTTP/1.1\\r\\n\\r\\n\";\n        let wire_header = b\"HTTP/1.1 200 OK\\r\\nContent-Length: 1\\r\\n\\r\\n\";\n        let wire_body = b\"a\";\n        let mock_io = Builder::new()\n            .read(read_wire)\n            .write(wire_header)\n            .write(wire_body)\n            .build();\n        let mut http_stream = HttpSession::new(Box::new(mock_io));\n        http_stream.read_request().await.unwrap();\n        let mut new_response = ResponseHeader::build(StatusCode::OK, None).unwrap();\n        new_response.append_header(\"Content-Length\", \"1\").unwrap();\n        http_stream.update_resp_headers = false;\n        http_stream\n            .write_response_header_ref(&new_response)\n            .await\n            .unwrap();\n        assert_eq!(\n            http_stream.body_writer.body_mode,\n            BodyMode::ContentLength(1, 0)\n        );\n        let n = http_stream.write_body(wire_body).await.unwrap().unwrap();\n        assert_eq!(wire_body.len(), n);\n        let n = http_stream.finish_body().await.unwrap().unwrap();\n        assert_eq!(wire_body.len(), n);\n    }\n\n    #[tokio::test]\n    async fn write_body_http10() {\n        let read_wire = b\"GET / HTTP/1.1\\r\\n\\r\\n\";\n        let wire_header = b\"HTTP/1.1 200 OK\\r\\n\\r\\n\";\n        let wire_body = b\"a\";\n        let mock_io = Builder::new()\n            .read(read_wire)\n            .write(wire_header)\n            .write(wire_body)\n            .build();\n        let mut http_stream = HttpSession::new(Box::new(mock_io));\n        http_stream.read_request().await.unwrap();\n        let new_response = ResponseHeader::build(StatusCode::OK, None).unwrap();\n        http_stream.update_resp_headers = false;\n        http_stream\n            .write_response_header_ref(&new_response)\n            .await\n            .unwrap();\n        assert_eq!(http_stream.body_writer.body_mode, BodyMode::UntilClose(0));\n        let n = http_stream.write_body(wire_body).await.unwrap().unwrap();\n        assert_eq!(wire_body.len(), n);\n        let n = http_stream.finish_body().await.unwrap().unwrap();\n        assert_eq!(wire_body.len(), n);\n    }\n\n    #[tokio::test]\n    async fn write_body_chunk() {\n        let read_wire = b\"GET / HTTP/1.1\\r\\n\\r\\n\";\n        let wire_header = b\"HTTP/1.1 200 OK\\r\\nTransfer-Encoding: chunked\\r\\n\\r\\n\";\n        let wire_body = b\"1\\r\\na\\r\\n\";\n        let wire_end = b\"0\\r\\n\\r\\n\";\n        let mock_io = Builder::new()\n            .read(read_wire)\n            .write(wire_header)\n            .write(wire_body)\n            .write(wire_end)\n            .build();\n        let mut http_stream = HttpSession::new(Box::new(mock_io));\n        http_stream.read_request().await.unwrap();\n        let mut new_response = ResponseHeader::build(StatusCode::OK, None).unwrap();\n        new_response\n            .append_header(\"Transfer-Encoding\", \"chunked\")\n            .unwrap();\n        http_stream.update_resp_headers = false;\n        http_stream\n            .write_response_header_ref(&new_response)\n            .await\n            .unwrap();\n        assert_eq!(\n            http_stream.body_writer.body_mode,\n            BodyMode::ChunkedEncoding(0)\n        );\n        let n = http_stream.write_body(b\"a\").await.unwrap().unwrap();\n        assert_eq!(b\"a\".len(), n);\n        let n = http_stream.finish_body().await.unwrap().unwrap();\n        assert_eq!(b\"a\".len(), n);\n    }\n\n    #[tokio::test]\n    async fn read_with_illegal() {\n        init_log();\n        let input1 = b\"GET /a?q=b c HTTP/1.1\\r\\n\";\n        let input2 = b\"Host: pingora.org\\r\\nContent-Length: 3\\r\\n\\r\\n\";\n        let input3 = b\"abc\";\n        let mock_io = Builder::new()\n            .read(&input1[..])\n            .read(&input2[..])\n            .read(&input3[..])\n            .build();\n        let mut http_stream = HttpSession::new(Box::new(mock_io));\n        http_stream.read_request().await.unwrap();\n        assert_eq!(http_stream.get_path(), &b\"/a?q=b%20c\"[..]);\n        let res = http_stream.read_body().await.unwrap().unwrap();\n        assert_eq!(res, BufRef::new(0, 3));\n        assert_eq!(http_stream.body_reader.body_state, ParseState::Complete(3));\n        assert_eq!(input3, http_stream.get_body(&res));\n    }\n\n    #[test]\n    fn escape_illegal() {\n        init_log();\n        // in query string\n        let input = BytesMut::from(\n            &b\"GET /a?q=<\\\"b c\\\"> HTTP/1.1\\r\\nHost: pingora.org\\r\\nContent-Length: 3\\r\\n\\r\\n\"[..],\n        );\n        let output = escape_illegal_request_line(&input).unwrap();\n        assert_eq!(\n            &output,\n            &b\"GET /a?q=%3C%22b%20c%22%3E HTTP/1.1\\r\\nHost: pingora.org\\r\\nContent-Length: 3\\r\\n\\r\\n\"[..]\n        );\n\n        // in path\n        let input = BytesMut::from(\n            &b\"GET /a:\\\"bc\\\" HTTP/1.1\\r\\nHost: pingora.org\\r\\nContent-Length: 3\\r\\n\\r\\n\"[..],\n        );\n        let output = escape_illegal_request_line(&input).unwrap();\n        assert_eq!(\n            &output,\n            &b\"GET /a:%22bc%22 HTTP/1.1\\r\\nHost: pingora.org\\r\\nContent-Length: 3\\r\\n\\r\\n\"[..]\n        );\n\n        // empty uri, unable to parse\n        let input =\n            BytesMut::from(&b\"GET  HTTP/1.1\\r\\nHost: pingora.org\\r\\nContent-Length: 3\\r\\n\\r\\n\"[..]);\n        assert!(escape_illegal_request_line(&input).is_none());\n    }\n\n    #[tokio::test]\n    async fn test_write_body_buf() {\n        let read_wire = b\"GET / HTTP/1.1\\r\\n\\r\\n\";\n        let write_expected = b\"HTTP/1.1 200 OK\\r\\nFoo: Bar\\r\\n\\r\\n\";\n        let mock_io = Builder::new().read(read_wire).write(write_expected).build();\n        let mut http_stream = HttpSession::new(Box::new(mock_io));\n        http_stream.read_request().await.unwrap();\n        let mut new_response = ResponseHeader::build(StatusCode::OK, None).unwrap();\n        new_response.append_header(\"Foo\", \"Bar\").unwrap();\n        http_stream.update_resp_headers = false;\n        http_stream\n            .write_response_header_ref(&new_response)\n            .await\n            .unwrap();\n        let written = http_stream.write_body_buf().await.unwrap();\n        assert!(written.is_none());\n    }\n\n    #[tokio::test]\n    #[should_panic(expected = \"There is still data left to write.\")]\n    async fn test_write_body_buf_write_timeout() {\n        let read_wire = b\"GET / HTTP/1.1\\r\\n\\r\\n\";\n        let wire1 = b\"HTTP/1.1 200 OK\\r\\nContent-Length: 3\\r\\n\\r\\n\";\n        let wire2 = b\"abc\";\n        let mock_io = Builder::new()\n            .read(read_wire)\n            .write(wire1)\n            .wait(Duration::from_millis(500))\n            .write(wire2)\n            .build();\n        let mut http_stream = HttpSession::new(Box::new(mock_io));\n        http_stream.read_request().await.unwrap();\n        http_stream.write_timeout = Some(Duration::from_millis(100));\n        let mut new_response = ResponseHeader::build(StatusCode::OK, None).unwrap();\n        new_response.append_header(\"Content-Length\", \"3\").unwrap();\n        http_stream.update_resp_headers = false;\n        http_stream\n            .write_response_header_ref(&new_response)\n            .await\n            .unwrap();\n        http_stream.body_write_buf = BytesMut::from(&b\"abc\"[..]);\n        let res = http_stream.write_body_buf().await;\n        assert_eq!(res.unwrap_err().etype(), &WriteTimedout);\n    }\n\n    #[tokio::test]\n    async fn test_write_continue_resp() {\n        let read_wire = b\"GET / HTTP/1.1\\r\\n\\r\\n\";\n        let write_expected = b\"HTTP/1.1 100 Continue\\r\\n\\r\\n\";\n        let mock_io = Builder::new().read(read_wire).write(write_expected).build();\n        let mut http_stream = HttpSession::new(Box::new(mock_io));\n        http_stream.read_request().await.unwrap();\n        http_stream.write_continue_response().await.unwrap();\n    }\n\n    #[test]\n    fn test_get_write_timeout() {\n        let mut http_stream = HttpSession::new(Box::new(Builder::new().build()));\n        let expected = Duration::from_secs(5);\n\n        http_stream.set_write_timeout(Some(expected));\n        assert_eq!(Some(expected), http_stream.write_timeout(50));\n    }\n\n    #[test]\n    fn test_get_write_timeout_none() {\n        let http_stream = HttpSession::new(Box::new(Builder::new().build()));\n        assert!(http_stream.write_timeout(50).is_none());\n    }\n\n    #[test]\n    fn test_get_write_timeout_min_send_rate_zero() {\n        let mut http_stream = HttpSession::new(Box::new(Builder::new().build()));\n        http_stream.set_min_send_rate(Some(0));\n        assert!(http_stream.write_timeout(50).is_none());\n\n        let mut http_stream = HttpSession::new(Box::new(Builder::new().build()));\n        http_stream.set_min_send_rate(None);\n        assert!(http_stream.write_timeout(50).is_none());\n    }\n\n    #[test]\n    fn test_get_write_timeout_min_send_rate_overrides_write_timeout() {\n        let mut http_stream = HttpSession::new(Box::new(Builder::new().build()));\n        let expected = Duration::from_millis(29800);\n\n        http_stream.set_write_timeout(Some(Duration::from_secs(60)));\n        http_stream.set_min_send_rate(Some(5000));\n\n        assert_eq!(Some(expected), http_stream.write_timeout(149000));\n    }\n\n    #[test]\n    fn test_get_write_timeout_min_send_rate_max_zero_buf() {\n        let mut http_stream = HttpSession::new(Box::new(Builder::new().build()));\n        let expected = Duration::from_secs(1);\n\n        http_stream.set_min_send_rate(Some(1));\n        assert_eq!(Some(expected), http_stream.write_timeout(0));\n    }\n\n    #[tokio::test]\n    async fn test_te_and_cl_disables_keepalive() {\n        // When both Transfer-Encoding and Content-Length are present,\n        // we must disable keepalive per RFC 9112 Section 6.1\n        // https://datatracker.ietf.org/doc/html/rfc9112#section-6.1-15\n        let input = b\"POST / HTTP/1.1\\r\\n\\\nHost: pingora.org\\r\\n\\\nTransfer-Encoding: chunked\\r\\n\\\nContent-Length: 10\\r\\n\\\n\\r\\n\\\n5\\r\\n\\\nhello\\r\\n\\\n0\\r\\n\\\n\\r\\n\";\n        let mock_io = Builder::new().read(&input[..]).build();\n        let mut http_stream = HttpSession::new(Box::new(mock_io));\n        http_stream.read_request().await.unwrap();\n\n        // Keepalive should be disabled\n        assert_eq!(http_stream.keepalive_timeout, KeepaliveStatus::Off);\n\n        // Content-Length header should have been removed\n        assert!(!http_stream\n            .req_header()\n            .headers\n            .contains_key(CONTENT_LENGTH));\n\n        // Transfer-Encoding should still be present\n        assert!(http_stream\n            .req_header()\n            .headers\n            .contains_key(TRANSFER_ENCODING));\n    }\n\n    #[tokio::test]\n    async fn test_http10_request_with_transfer_encoding_rejected() {\n        // HTTP/1.0 requests MUST NOT contain Transfer-Encoding\n        let input = b\"POST / HTTP/1.0\\r\\n\\\nHost: pingora.org\\r\\n\\\nTransfer-Encoding: chunked\\r\\n\\\n\\r\\n\\\n5\\r\\n\\\nhello\\r\\n\\\n0\\r\\n\\\n\\r\\n\";\n        let mock_io = Builder::new().read(&input[..]).build();\n        let mut http_stream = HttpSession::new(Box::new(mock_io));\n        let result = http_stream.read_request().await;\n\n        // Should be rejected with InvalidHTTPHeader error\n        assert!(result.is_err());\n        let err = result.unwrap_err();\n        assert_eq!(err.etype(), &InvalidHTTPHeader);\n        assert!(err.to_string().contains(\"Transfer-Encoding\"));\n    }\n\n    #[tokio::test]\n    async fn test_http10_request_without_transfer_encoding_accepted() {\n        // HTTP/1.0 requests without Transfer-Encoding should be accepted\n        let input = b\"POST / HTTP/1.0\\r\\n\\\nHost: pingora.org\\r\\n\\\nContent-Length: 5\\r\\n\\\n\\r\\n\\\nhello\";\n        let mock_io = Builder::new().read(&input[..]).build();\n        let mut http_stream = HttpSession::new(Box::new(mock_io));\n        let result = http_stream.read_request().await;\n\n        // Should succeed\n        assert!(result.is_ok());\n        assert_eq!(http_stream.req_header().version, http::Version::HTTP_10);\n    }\n\n    #[tokio::test]\n    async fn test_http11_request_with_transfer_encoding_accepted() {\n        // HTTP/1.1 with Transfer-Encoding should be accepted (contrast with HTTP/1.0)\n        let input = b\"POST / HTTP/1.1\\r\\n\\\nHost: pingora.org\\r\\n\\\nTransfer-Encoding: chunked\\r\\n\\\n\\r\\n\\\n5\\r\\n\\\nhello\\r\\n\\\n0\\r\\n\\\n\\r\\n\";\n        let mock_io = Builder::new().read(&input[..]).build();\n        let mut http_stream = HttpSession::new(Box::new(mock_io));\n        let result = http_stream.read_request().await;\n\n        // Should succeed\n        assert!(result.is_ok());\n        assert_eq!(http_stream.req_header().version, http::Version::HTTP_11);\n    }\n\n    #[tokio::test]\n    async fn test_request_multiple_transfer_encoding_headers() {\n        init_log();\n        // Multiple TE headers should be treated as comma-separated\n        let input = b\"POST / HTTP/1.1\\r\\n\\\nHost: pingora.org\\r\\n\\\nTransfer-Encoding: gzip\\r\\n\\\nTransfer-Encoding: chunked\\r\\n\\\n\\r\\n\\\n5\\r\\n\\\nhello\\r\\n\\\n0\\r\\n\\\n\\r\\n\";\n\n        let mock_io = Builder::new().read(&input[..]).build();\n        let mut http_stream = HttpSession::new(Box::new(mock_io));\n        http_stream.read_request().await.unwrap();\n\n        // Should correctly identify chunked encoding from last header\n        assert!(http_stream.is_chunked_encoding());\n\n        // Verify body can be read correctly\n        let body = http_stream.read_body_bytes().await.unwrap();\n        assert_eq!(body.unwrap().as_ref(), b\"hello\");\n    }\n\n    #[tokio::test]\n    async fn test_request_multiple_te_headers_chunked_not_last() {\n        init_log();\n        // Chunked in first header but not last - should NOT be chunked\n        // Only the final Transfer-Encoding determines if body is chunked\n        let input = b\"POST / HTTP/1.1\\r\\n\\\nHost: pingora.org\\r\\n\\\nTransfer-Encoding: chunked\\r\\n\\\nTransfer-Encoding: identity\\r\\n\\\nContent-Length: 5\\r\\n\\\n\\r\\n\";\n\n        let mock_io = Builder::new().read(&input[..]).build();\n        let mut http_stream = HttpSession::new(Box::new(mock_io));\n        // should fail validation\n        http_stream.read_request().await.unwrap_err();\n    }\n\n    #[tokio::test]\n    async fn test_no_more_reuses_explicitly_disables_reuse() {\n        init_log();\n        let wire_req = b\"GET /test HTTP/1.1\\r\\n\\r\\n\";\n        let wire_header = b\"HTTP/1.1 200 OK\\r\\n\\r\\n\";\n        let mock_io = Builder::new()\n            .read(&wire_req[..])\n            .write(wire_header)\n            .build();\n        let mut http_session = HttpSession::new(Box::new(mock_io));\n\n        // Setting the number of keepalive reuses here overrides the keepalive\n        // setting below\n        http_session.set_keepalive_reuses_remaining(Some(0));\n\n        http_session.read_request().await.unwrap();\n\n        let new_response = ResponseHeader::build(StatusCode::OK, None).unwrap();\n        http_session.update_resp_headers = false;\n        http_session\n            .write_response_header(Box::new(new_response))\n            .await\n            .unwrap();\n\n        assert_eq!(http_session.body_writer.body_mode, BodyMode::UntilClose(0));\n\n        http_session.finish_body().await.unwrap().unwrap();\n\n        http_session.set_keepalive(Some(100));\n        let reused = http_session.reuse().await.unwrap();\n        assert!(reused.is_none());\n    }\n\n    #[tokio::test]\n    async fn test_close_delimited_response_explicitly_disables_reuse() {\n        init_log();\n        let wire_req = b\"GET /test HTTP/1.1\\r\\n\\r\\n\";\n        let wire_header = b\"HTTP/1.1 200 OK\\r\\n\\r\\n\";\n        let mock_io = Builder::new()\n            .read(&wire_req[..])\n            .write(wire_header)\n            .build();\n        let mut http_stream = HttpSession::new(Box::new(mock_io));\n        http_stream.read_request().await.unwrap();\n\n        let new_response = ResponseHeader::build(StatusCode::OK, None).unwrap();\n        http_stream.update_resp_headers = false;\n        http_stream\n            .write_response_header(Box::new(new_response))\n            .await\n            .unwrap();\n\n        assert_eq!(http_stream.body_writer.body_mode, BodyMode::UntilClose(0));\n\n        http_stream.finish_body().await.unwrap().unwrap();\n\n        let reused = http_stream.reuse().await.unwrap();\n        assert!(reused.is_none());\n    }\n}\n\n#[cfg(test)]\nmod test_sync {\n    use super::*;\n    use http::StatusCode;\n    use log::{debug, error};\n    use std::str;\n\n    fn init_log() {\n        let _ = env_logger::builder().is_test(true).try_init();\n    }\n\n    #[test]\n    fn test_response_to_wire() {\n        init_log();\n        let mut new_response = ResponseHeader::build(StatusCode::OK, None).unwrap();\n        new_response.append_header(\"Foo\", \"Bar\").unwrap();\n        let mut wire = BytesMut::with_capacity(INIT_HEADER_BUF_SIZE);\n        http_resp_header_to_buf(&new_response, &mut wire).unwrap();\n        debug!(\"{}\", str::from_utf8(wire.as_ref()).unwrap());\n        let mut headers = [httparse::EMPTY_HEADER; 128];\n        let mut resp = httparse::Response::new(&mut headers);\n        let result = resp.parse(wire.as_ref());\n        match result {\n            Ok(_) => {}\n            Err(e) => error!(\"{:?}\", e),\n        }\n        assert!(result.unwrap().is_complete());\n        // FIXME: the order is not guaranteed\n        assert_eq!(b\"Foo\", headers[0].name.as_bytes());\n        assert_eq!(b\"Bar\", headers[0].value);\n    }\n}\n\n#[cfg(test)]\nmod test_timeouts {\n    use super::*;\n    use std::future::IntoFuture;\n    use tokio_test::io::{Builder, Mock};\n\n    /// An upper limit for any read within any test to prevent tests from hanging forever if\n    /// an internal read call never returns, etc.\n    const TEST_MAX_WAIT_FOR_READ: Duration = Duration::from_secs(3);\n\n    /// The duration of 600 seconds is chosen to be \"effectively forever\" for the purpose of testing\n    const TEST_FOREVER_DURATION: Duration = Duration::from_secs(600);\n\n    /// The read_timeout to use, when we want to test that a read operation times out\n    const TEST_READ_TIMEOUT: Duration = Duration::from_secs(1);\n\n    #[derive(Debug)]\n    struct ReadBlockedForeverError;\n\n    /// Returns a client stream that will \"never\" send any bytes / return from a read operation\n    fn mocked_blocking_headers_forever_stream() -> Box<Mock> {\n        Box::new(Builder::new().wait(TEST_FOREVER_DURATION).build())\n    }\n\n    fn mocked_blocking_body_forever_stream() -> Box<Mock> {\n        let http1 = b\"GET / HTTP/1.1\\r\\n\";\n        let http2 = b\"Host: pingora.example\\r\\nContent-Length: 3\\r\\n\\r\\n\";\n        Box::new(\n            Builder::new()\n                .read(&http1[..])\n                .read(&http2[..])\n                .wait(TEST_FOREVER_DURATION)\n                .build(),\n        )\n    }\n\n    /// Helper function to test a read operation with a tokio timeout\n    /// to prevent tests from hanging forever in case of a bug\n    async fn test_read_with_tokio_timeout<F, T>(\n        read_future: F,\n    ) -> Result<Result<T, Box<Error>>, ReadBlockedForeverError>\n    where\n        F: IntoFuture<Output = Result<T, Box<Error>>>,\n    {\n        let read_result = tokio::time::timeout(TEST_MAX_WAIT_FOR_READ, read_future).await;\n        read_result.map_err(|_| ReadBlockedForeverError)\n    }\n\n    #[tokio::test]\n    async fn test_read_http_request_headers_timeout_for_read_request() {\n        // confirm that a `read_timeout` of `None` would've waited \"indefinitely\"\n        let mut http_stream = HttpSession::new(mocked_blocking_headers_forever_stream());\n        http_stream.read_timeout = None;\n        let res = test_read_with_tokio_timeout(http_stream.read_request()).await;\n        assert!(res.is_err()); // test timeout occurred, and not any internal Pingora timeout\n\n        // confirm that the `read_timeout` is respected\n        let mut http_stream = HttpSession::new(mocked_blocking_headers_forever_stream());\n        http_stream.read_timeout = Some(TEST_READ_TIMEOUT);\n        let res = test_read_with_tokio_timeout(http_stream.read_request()).await;\n        assert!(res.is_ok());\n        assert_eq!(res.unwrap().unwrap_err().etype(), &ReadTimedout);\n    }\n\n    #[tokio::test]\n    async fn test_read_http_body_timeout_for_read_body_bytes() {\n        // confirm that a `read_timeout` of `None` would've waited \"indefinitely\"\n        let mut http_stream = HttpSession::new(mocked_blocking_body_forever_stream());\n        http_stream.read_timeout = None;\n        http_stream.read_request().await.unwrap();\n        let res = test_read_with_tokio_timeout(http_stream.read_body_bytes()).await;\n        assert!(res.is_err()); // test timeout occurred, and not any internal Pingora timeout\n\n        // confirm that the `read_timeout` is respected\n        let mut http_stream = HttpSession::new(mocked_blocking_body_forever_stream());\n        http_stream.read_timeout = Some(TEST_READ_TIMEOUT);\n        http_stream.read_request().await.unwrap();\n        let res = test_read_with_tokio_timeout(http_stream.read_body_bytes()).await;\n        assert!(res.is_ok());\n        assert_eq!(res.unwrap().unwrap_err().etype(), &ReadTimedout);\n    }\n}\n\n#[cfg(test)]\nmod test_overread {\n    use super::*;\n    use rstest::rstest;\n    use tokio_test::io::Builder;\n\n    fn init_log() {\n        let _ = env_logger::builder().is_test(true).try_init();\n    }\n\n    /// Test session reuse with preread body (all data in single read).\n    /// When extra bytes are read beyond the request body, the session should NOT be reused.\n    /// Test matrix includes whether reading body bytes is polled.\n    #[rstest]\n    #[case(0, None, true, true)] // CL:0, no extra, read body -> should reuse\n    #[case(0, None, false, true)] // CL:0, no extra, no read -> should reuse\n    #[case(0, Some(&b\"extra_data_here\"[..]), true, false)] // CL:0, extra, read body -> should NOT reuse\n    #[case(0, Some(&b\"extra_data_here\"[..]), false, false)] // CL:0, extra, no read -> should NOT reuse\n    #[case(5, None, true, true)] // CL:5, no extra, read body -> should reuse\n    #[case(5, None, false, true)] // CL:5, no extra, no read -> should reuse\n    #[case(5, Some(&b\"extra\"[..]), true, false)] // CL:5, extra, read body -> should NOT reuse\n    #[case(5, Some(&b\"extra\"[..]), false, false)] // CL:5, extra, no read -> should NOT reuse\n    #[tokio::test]\n    async fn test_reuse_with_preread_body_overread(\n        #[case] content_length: usize,\n        #[case] extra_bytes: Option<&[u8]>,\n        #[case] read_body: bool,\n        #[case] expect_reuse: bool,\n    ) {\n        init_log();\n\n        let body = b\"hello\";\n\n        // Build the complete HTTP request in a single buffer\n        // (all body is preread with header)\n        let mut request_data = Vec::new();\n        request_data.extend_from_slice(b\"GET / HTTP/1.1\\r\\n\");\n        request_data.extend_from_slice(\n            format!(\"Host: pingora.org\\r\\nContent-Length: {content_length}\\r\\n\\r\\n\",).as_bytes(),\n        );\n\n        if content_length > 0 {\n            request_data.extend_from_slice(&body[..content_length]);\n        }\n\n        if let Some(extra) = extra_bytes {\n            request_data.extend_from_slice(extra);\n        }\n\n        let mock_io = Builder::new().read(&request_data).build();\n        let mut http_stream = HttpSession::new(Box::new(mock_io));\n        http_stream.read_request().await.unwrap();\n\n        // Conditionally read the body\n        if read_body {\n            let result = http_stream.read_body_bytes().await.unwrap();\n\n            if content_length == 0 {\n                assert!(\n                    result.is_none(),\n                    \"Body should be empty for Content-Length: 0\"\n                );\n            } else {\n                let body_result = result.unwrap();\n                assert_eq!(body_result.as_ref(), &body[..content_length]);\n            }\n            assert_eq!(http_stream.body_bytes_read(), content_length);\n        }\n\n        let reused = http_stream.reuse().await.unwrap();\n        assert_eq!(reused.is_some(), expect_reuse);\n    }\n\n    /// Test session reuse with chunked encoding and separate reads.\n    /// When extra bytes are read beyond the request body, the session should NOT be reused.\n    /// Test matrix includes whether reading body bytes is polled.\n    #[rstest]\n    #[case(true)]\n    #[case(false)]\n    #[tokio::test]\n    async fn test_reuse_with_chunked_body_overread(#[case] read_body: bool) {\n        init_log();\n\n        let headers = b\"GET / HTTP/1.1\\r\\nHost: pingora.org\\r\\nTransfer-Encoding: chunked\\r\\n\\r\\n\";\n        let body_and_extra = b\"5\\r\\nhello\\r\\n0\\r\\n\\r\\nextra\";\n\n        let mock_io = Builder::new().read(headers).read(body_and_extra).build();\n\n        let mut http_stream = HttpSession::new(Box::new(mock_io));\n        http_stream.read_request().await.unwrap();\n        assert!(http_stream.is_chunked_encoding());\n\n        if read_body {\n            let result = http_stream.read_body_bytes().await.unwrap();\n            assert_eq!(result.unwrap().as_ref(), b\"hello\");\n\n            // Read terminating chunk (returns None)\n            let result = http_stream.read_body_bytes().await.unwrap();\n            assert!(result.is_none());\n\n            assert_eq!(http_stream.body_bytes_read(), 5);\n        }\n\n        let reused = http_stream.reuse().await.unwrap();\n        assert!(reused.is_none());\n    }\n}\n"
  },
  {
    "path": "pingora-core/src/protocols/http/v2/client.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! HTTP/2 client session and connection\n// TODO: this module needs a refactor\n\nuse bytes::Bytes;\nuse futures::FutureExt;\nuse h2::client::{self, ResponseFuture, SendRequest};\nuse h2::{Reason, RecvStream, SendStream};\nuse http::HeaderMap;\nuse log::{debug, error, warn};\nuse pingora_error::{Error, ErrorType, ErrorType::*, OrErr, Result, RetryType};\nuse pingora_http::{RequestHeader, ResponseHeader};\nuse pingora_timeout::timeout;\nuse std::io::ErrorKind;\nuse std::sync::atomic::{AtomicBool, Ordering};\nuse std::sync::Arc;\nuse std::task::{ready, Context, Poll};\nuse std::time::Duration;\nuse tokio::io::{AsyncRead, AsyncWrite};\nuse tokio::sync::watch;\n\nuse crate::connectors::http::v2::ConnectionRef;\nuse crate::protocols::{Digest, SocketAddr, UniqueIDType};\n\npub const PING_TIMEDOUT: ErrorType = ErrorType::new(\"PingTimedout\");\n\npub struct Http2Session {\n    send_req: SendRequest<Bytes>,\n    send_body: Option<SendStream<Bytes>>,\n    resp_fut: Option<ResponseFuture>,\n    req_sent: Option<Box<RequestHeader>>,\n    response_header: Option<ResponseHeader>,\n    response_body_reader: Option<RecvStream>,\n    /// The read timeout, which will be applied to both reading the header and the body.\n    /// The timeout is reset on every read. This is not a timeout on the overall duration of the\n    /// response.\n    pub read_timeout: Option<Duration>,\n    /// The write timeout which will be applied to writing request body.\n    /// The timeout is reset on every write. This is not a timeout on the overall duration of the\n    /// request.\n    pub write_timeout: Option<Duration>,\n    pub conn: ConnectionRef,\n    // Indicate that whether a END_STREAM is already sent\n    ended: bool,\n    // Total DATA payload bytes received from upstream response\n    body_recv: usize,\n}\n\nimpl Drop for Http2Session {\n    fn drop(&mut self) {\n        self.conn.release_stream();\n    }\n}\n\nimpl Http2Session {\n    pub(crate) fn new(send_req: SendRequest<Bytes>, conn: ConnectionRef) -> Self {\n        Http2Session {\n            send_req,\n            send_body: None,\n            resp_fut: None,\n            req_sent: None,\n            response_header: None,\n            response_body_reader: None,\n            read_timeout: None,\n            write_timeout: None,\n            conn,\n            ended: false,\n            body_recv: 0,\n        }\n    }\n\n    fn sanitize_request_header(req: &mut RequestHeader) -> Result<()> {\n        req.set_version(http::Version::HTTP_2);\n        if req.uri.authority().is_some() {\n            return Ok(());\n        }\n        // use host header to populate :authority field\n        let Some(authority) = req.headers.get(http::header::HOST).map(|v| v.as_bytes()) else {\n            return Error::e_explain(InvalidHTTPHeader, \"no authority header for h2\");\n        };\n        let uri = http::uri::Builder::new()\n            .scheme(\"https\") // fixed for now\n            .authority(authority)\n            .path_and_query(req.uri.path_and_query().as_ref().unwrap().as_str())\n            .build();\n        match uri {\n            Ok(uri) => {\n                req.set_uri(uri);\n                Ok(())\n            }\n            Err(_) => Error::e_explain(\n                InvalidHTTPHeader,\n                format!(\"invalid authority from host {authority:?}\"),\n            ),\n        }\n    }\n\n    /// Write the request header to the server\n    pub fn write_request_header(&mut self, mut req: Box<RequestHeader>, end: bool) -> Result<()> {\n        if self.req_sent.is_some() {\n            // cannot send again, TODO: warn\n            return Ok(());\n        }\n        Self::sanitize_request_header(&mut req)?;\n        let parts = req.as_owned_parts();\n        let request = http::Request::from_parts(parts, ());\n        // There is no write timeout for h2 because the actual write happens async from this fn\n        let (resp_fut, send_body) = self\n            .send_req\n            .send_request(request, end)\n            .or_err(H2Error, \"while sending request\")\n            .map_err(|e| self.handle_err(e))?;\n        self.req_sent = Some(req);\n        self.send_body = Some(send_body);\n        self.resp_fut = Some(resp_fut);\n        self.ended = self.ended || end;\n\n        Ok(())\n    }\n\n    /// Write a request body chunk\n    pub async fn write_request_body(&mut self, data: Bytes, end: bool) -> Result<()> {\n        if self.ended {\n            warn!(\"Try to write request body after end of stream, dropping the extra data\");\n            return Ok(());\n        }\n\n        let body_writer = self\n            .send_body\n            .as_mut()\n            .expect(\"Try to write request body before sending request header\");\n\n        super::write_body(body_writer, data, end, self.write_timeout)\n            .await\n            .map_err(|e| self.handle_err(e))?;\n        self.ended = self.ended || end;\n        Ok(())\n    }\n\n    /// Signal that the request body has ended\n    pub fn finish_request_body(&mut self) -> Result<()> {\n        if self.ended {\n            return Ok(());\n        }\n\n        let body_writer = self\n            .send_body\n            .as_mut()\n            .expect(\"Try to finish request stream before sending request header\");\n\n        // Just send an empty data frame with end of stream set\n        body_writer\n            .send_data(\"\".into(), true)\n            .or_err(WriteError, \"while writing empty h2 request body\")\n            .map_err(|e| self.handle_err(e))?;\n        self.ended = true;\n        Ok(())\n    }\n\n    /// Read the response header\n    pub async fn read_response_header(&mut self) -> Result<()> {\n        // TODO: how to read 1xx headers?\n        // https://github.com/hyperium/h2/issues/167\n\n        if self.response_header.is_some() {\n            panic!(\"H2 response header is already read\")\n        }\n\n        let Some(resp_fut) = self.resp_fut.take() else {\n            panic!(\"Try to take response header, but it is already taken\")\n        };\n\n        let res = match self.read_timeout {\n            Some(t) => timeout(t, resp_fut)\n                .await\n                .map_err(|_| Error::explain(ReadTimedout, \"while reading h2 response header\"))\n                .map_err(|e| self.handle_err(e))?,\n            None => resp_fut.await,\n        };\n        let (resp, body_reader) = res.map_err(handle_read_header_error)?.into_parts();\n        self.response_header = Some(resp.into());\n        self.response_body_reader = Some(body_reader);\n\n        Ok(())\n    }\n\n    #[doc(hidden)]\n    pub fn poll_read_response_header(\n        &mut self,\n        cx: &mut Context<'_>,\n    ) -> Poll<Result<(), h2::Error>> {\n        if self.response_header.is_some() {\n            panic!(\"H2 response header is already read\")\n        }\n\n        let Some(mut resp_fut) = self.resp_fut.take() else {\n            panic!(\"Try to take response header, but it is already taken\")\n        };\n\n        let res = match resp_fut.poll_unpin(cx) {\n            Poll::Ready(Ok(res)) => res,\n            Poll::Ready(Err(err)) => return Poll::Ready(Err(err)),\n            Poll::Pending => {\n                self.resp_fut = Some(resp_fut);\n                return Poll::Pending;\n            }\n        };\n\n        let (resp, body_reader) = res.into_parts();\n        self.response_header = Some(resp.into());\n        self.response_body_reader = Some(body_reader);\n\n        Poll::Ready(Ok(()))\n    }\n\n    /// Read the response body\n    ///\n    /// `None` means, no more body to read\n    pub async fn read_response_body(&mut self) -> Result<Option<Bytes>> {\n        let Some(body_reader) = self.response_body_reader.as_mut() else {\n            // req is not sent or response is already read\n            // TODO: warn\n            return Ok(None);\n        };\n\n        let fut = body_reader.data();\n        let res = match self.read_timeout {\n            Some(t) => timeout(t, fut)\n                .await\n                .map_err(|_| Error::explain(ReadTimedout, \"while reading h2 response body\"))?,\n            None => fut.await,\n        };\n        let body = res\n            .transpose()\n            .or_err(ReadError, \"while read h2 response body\")\n            .map_err(|mut e| {\n                // cannot use handle_err() because of borrow checker\n                if self.conn.ping_timedout() {\n                    e.etype = PING_TIMEDOUT;\n                }\n                e\n            })?;\n\n        if let Some(data) = body.as_ref() {\n            body_reader\n                .flow_control()\n                .release_capacity(data.len())\n                .or_err(ReadError, \"while releasing h2 response body capacity\")?;\n            self.body_recv = self.body_recv.saturating_add(data.len());\n        }\n\n        Ok(body)\n    }\n\n    #[doc(hidden)]\n    pub fn poll_read_response_body(\n        &mut self,\n        cx: &mut Context<'_>,\n    ) -> Poll<Option<Result<Bytes, h2::Error>>> {\n        let Some(body_reader) = self.response_body_reader.as_mut() else {\n            // req is not sent or response is already read\n            // TODO: warn\n            return Poll::Ready(None);\n        };\n\n        let data = match ready!(body_reader.poll_data(cx)).transpose() {\n            Ok(data) => data,\n            Err(err) => return Poll::Ready(Some(Err(err))),\n        };\n\n        if let Some(data) = data {\n            body_reader.flow_control().release_capacity(data.len())?;\n            return Poll::Ready(Some(Ok(data)));\n        }\n\n        Poll::Ready(None)\n    }\n\n    /// Whether the response has ended\n    pub fn response_finished(&self) -> bool {\n        // if response_body_reader doesn't exist, the response is not even read yet\n        self.response_body_reader\n            .as_ref()\n            .is_some_and(|reader| reader.is_end_stream())\n    }\n\n    /// Check whether stream finished with error.\n    /// Like `response_finished`, but also attempts to poll the h2 stream for errors that may have\n    /// caused the stream to terminate, and returns them as `H2Error`s.\n    pub fn check_response_end_or_error(&mut self) -> Result<bool> {\n        let Some(reader) = self.response_body_reader.as_mut() else {\n            // response is not even read\n            return Ok(false);\n        };\n\n        if !reader.is_end_stream() {\n            return Ok(false);\n        }\n\n        // https://github.com/hyperium/h2/issues/806\n        // The fundamental issue is that h2::RecvStream may return `is_end_stream` true\n        // when the stream was naturally closed via END_STREAM /OR/ if there was an error\n        // while reading data frames that forced the closure.\n        // The h2 API as-is makes it difficult to determine which situation is occurring.\n        //\n        // `poll_data` should be returning None after `is_end_stream`, if the stream\n        // is truly expecting no more data to be sent.\n        // https://docs.rs/h2/latest/h2/struct.RecvStream.html#method.is_end_stream\n        // So poll the data once to check this condition. If an error is returned, that indicates\n        // that the stream closed due to an error e.g. h2 protocol error.\n        //\n        // tokio::task::unconstrained because now_or_never may yield None when the future is ready\n        match tokio::task::unconstrained(reader.data()).now_or_never() {\n            Some(None) => Ok(true),\n            Some(Some(Ok(_))) => Error::e_explain(H2Error, \"unexpected data after end stream\"),\n            Some(Some(Err(e))) => Error::e_because(H2Error, \"while checking end stream\", e),\n            None => {\n                // RecvStream data() should be ready to poll after the stream ends,\n                // this indicates an unexpected change in the h2 crate\n                panic!(\"data() not ready after end stream\")\n            }\n        }\n    }\n\n    /// Read the optional trailer headers\n    pub async fn read_trailers(&mut self) -> Result<Option<HeaderMap>> {\n        let Some(reader) = self.response_body_reader.as_mut() else {\n            // response is not even read\n            // TODO: warn\n            return Ok(None);\n        };\n        let fut = reader.trailers();\n\n        let res = match self.read_timeout {\n            Some(t) => timeout(t, fut)\n                .await\n                .map_err(|_| Error::explain(ReadTimedout, \"while reading h2 trailer\"))\n                .map_err(|e| self.handle_err(e))?,\n            None => fut.await,\n        };\n        match res {\n            Ok(t) => Ok(t),\n            Err(e) => {\n                // GOAWAY with no error: this is graceful shutdown, continue as if no trailer\n                // RESET_STREAM with no error: https://datatracker.ietf.org/doc/html/rfc9113#section-8.1:\n                // this is to signal client to stop uploading request without breaking the response.\n                // TODO: should actually stop uploading\n                // TODO: should we try reading again?\n                // TODO: handle this when reading headers and body as well\n                // https://github.com/hyperium/h2/issues/741\n\n                if (e.is_go_away() || e.is_reset())\n                    && e.is_remote()\n                    && e.reason() == Some(Reason::NO_ERROR)\n                {\n                    Ok(None)\n                } else {\n                    Err(e)\n                }\n            }\n        }\n        .or_err(ReadError, \"while reading h2 trailers\")\n    }\n\n    /// The request header if it is already sent\n    pub fn request_header(&self) -> Option<&RequestHeader> {\n        self.req_sent.as_deref()\n    }\n\n    /// The response header if it is already read\n    pub fn response_header(&self) -> Option<&ResponseHeader> {\n        self.response_header.as_ref()\n    }\n\n    /// Give up the http session abruptly.\n    pub fn shutdown(&mut self) {\n        if !self.ended || !self.response_finished() {\n            if let Some(send_body) = self.send_body.as_mut() {\n                send_body.send_reset(h2::Reason::INTERNAL_ERROR)\n            }\n        }\n    }\n\n    /// Drop everything in this h2 stream. Return the connection ref.\n    /// After this function the underlying h2 connection should already notify the closure of this\n    /// stream so that another stream can be created if needed.\n    pub(crate) fn conn(&self) -> ConnectionRef {\n        self.conn.clone()\n    }\n\n    /// Whether ping timeout occurred. After a ping timeout, the h2 connection will be terminated.\n    /// Ongoing h2 streams will receive an stream/connection error. The streams should check this\n    /// flag to tell whether the error is triggered by the timeout.\n    pub(crate) fn ping_timedout(&self) -> bool {\n        self.conn.ping_timedout()\n    }\n\n    /// Return the [Digest] of the connection\n    ///\n    /// For reused connection, the timing in the digest will reflect its initial handshakes\n    /// The caller should check if the connection is reused to avoid misuse the timing field.\n    pub fn digest(&self) -> Option<&Digest> {\n        Some(self.conn.digest())\n    }\n\n    /// Return a mutable [Digest] reference for the connection\n    ///\n    /// Will return `None` if multiple H2 streams are open.\n    pub fn digest_mut(&mut self) -> Option<&mut Digest> {\n        self.conn.digest_mut()\n    }\n\n    /// Return the server (peer) address recorded in the connection digest.\n    pub fn server_addr(&self) -> Option<&SocketAddr> {\n        self.conn\n            .digest()\n            .socket_digest\n            .as_ref()\n            .map(|d| d.peer_addr())?\n    }\n\n    /// Return the client (local) address recorded in the connection digest.\n    pub fn client_addr(&self) -> Option<&SocketAddr> {\n        self.conn\n            .digest()\n            .socket_digest\n            .as_ref()\n            .map(|d| d.local_addr())?\n    }\n\n    /// the FD of the underlying connection\n    pub fn fd(&self) -> UniqueIDType {\n        self.conn.id()\n    }\n\n    /// Upstream response body bytes received (HTTP/2 DATA payload; excludes headers/framing).\n    pub fn body_bytes_received(&self) -> usize {\n        self.body_recv\n    }\n\n    /// take the body sender to another task to perform duplex read and write\n    pub fn take_request_body_writer(&mut self) -> Option<SendStream<Bytes>> {\n        self.send_body.take()\n    }\n\n    fn handle_err(&self, mut e: Box<Error>) -> Box<Error> {\n        if self.ping_timedout() {\n            e.etype = PING_TIMEDOUT;\n        }\n\n        // is_go_away: retry via another connection, this connection is being teardown\n        // should retry\n        if self.response_header.is_none() {\n            if let Some(err) = e.root_cause().downcast_ref::<h2::Error>() {\n                if err.is_go_away()\n                    && err.is_remote()\n                    && (err.reason() == Some(h2::Reason::NO_ERROR))\n                {\n                    e.retry = true.into();\n                }\n            }\n        }\n        e\n    }\n}\n\n/* helper functions */\n\n/* Types of errors during h2 header read\n 1. peer requests to downgrade to h1, mostly IIS server for NTLM: we will downgrade and retry\n 2. peer sends invalid h2 frames, usually sending h1 only header: we will downgrade and retry\n 3. peer sends GO_AWAY(NO_ERROR) connection is being shut down: we will retry\n 4. peer IO error on reused conn, usually firewall kills old conn: we will retry\n 5. peer sends REFUSED_STREAM on RST_STREAM, this is safe to retry\n 6. All other errors will terminate the request\n*/\nfn handle_read_header_error(e: h2::Error) -> Box<Error> {\n    if e.is_remote() && (e.reason() == Some(h2::Reason::HTTP_1_1_REQUIRED)) {\n        let mut err = Error::because(H2Downgrade, \"while reading h2 header\", e);\n        err.retry = true.into();\n        err\n    } else if e.is_go_away() && e.is_library() && (e.reason() == Some(h2::Reason::PROTOCOL_ERROR)) {\n        // remote send invalid H2 responses\n        let mut err = Error::because(InvalidH2, \"while reading h2 header\", e);\n        err.retry = true.into();\n        err\n    } else if e.is_go_away() && e.is_remote() && (e.reason() == Some(h2::Reason::NO_ERROR)) {\n        // is_go_away: retry via another connection, this connection is being teardown\n        let mut err = Error::because(H2Error, \"while reading h2 header\", e);\n        err.retry = true.into();\n        err\n    } else if e.is_reset() && e.is_remote() && (e.reason() == Some(h2::Reason::REFUSED_STREAM)) {\n        // The REFUSED_STREAM error code can be included in a RST_STREAM frame to indicate\n        // that the stream is being closed prior to any processing having occurred.\n        // Any request that was sent on the reset stream can be safely retried.\n        // https://datatracker.ietf.org/doc/html/rfc9113#section-8.7\n        let mut err = Error::because(H2Error, \"while reading h2 header\", e);\n        err.retry = true.into();\n        err\n    } else if e.is_io() {\n        // is_io: typical if a previously reused connection silently drops it\n        // only retry if the connection is reused\n        // safety: e.get_io() will always succeed if e.is_io() is true\n        let io_err = e.get_io().expect(\"checked is io\");\n\n        // for h2 hyperium raw_os_error() will be None unless this is a new connection\n        // where we handshake() and from_io() is called, check ErrorKind explicitly with true_io_error\n        let true_io_error = io_err.raw_os_error().is_some()\n            || matches!(\n                io_err.kind(),\n                ErrorKind::ConnectionReset | ErrorKind::TimedOut | ErrorKind::BrokenPipe\n            );\n        let mut err = Error::because(ReadError, \"while reading h2 header\", e);\n        if true_io_error {\n            err.retry = RetryType::ReusedOnly;\n        } // else could be TLS error, which is unsafe to retry\n        err\n    } else {\n        Error::because(H2Error, \"while reading h2 header\", e)\n    }\n}\n\nuse tokio::sync::oneshot;\n\npub async fn drive_connection<S>(\n    mut c: client::Connection<S>,\n    id: UniqueIDType,\n    closed: watch::Sender<bool>,\n    ping_interval: Option<Duration>,\n    ping_timeout_occurred: Arc<AtomicBool>,\n) where\n    S: AsyncRead + AsyncWrite + Send + Unpin,\n{\n    let interval = ping_interval.unwrap_or(Duration::ZERO);\n    if !interval.is_zero() {\n        // for ping to inform this fn to drop the connection\n        let (tx, rx) = oneshot::channel::<()>();\n        // for this fn to inform ping to give up when it is already dropped\n        let dropped = Arc::new(AtomicBool::new(false));\n        let dropped2 = dropped.clone();\n\n        if let Some(ping_pong) = c.ping_pong() {\n            pingora_runtime::current_handle().spawn(async move {\n                do_ping_pong(ping_pong, interval, tx, dropped2, id).await;\n            });\n        } else {\n            warn!(\"Cannot get ping-pong handler from h2 connection\");\n        }\n\n        tokio::select! {\n            r = c => match r {\n                Ok(_) => debug!(\"H2 connection finished fd: {id}\"),\n                Err(e) => debug!(\"H2 connection fd: {id} errored: {e:?}\"),\n            },\n            r = rx => match r {\n                Ok(_) => {\n                    ping_timeout_occurred.store(true, Ordering::Relaxed);\n                    warn!(\"H2 connection Ping timeout/Error fd: {id}, closing conn\");\n                },\n                Err(e) => warn!(\"H2 connection Ping Rx error {e:?}\"),\n            },\n        };\n\n        dropped.store(true, Ordering::Relaxed);\n    } else {\n        match c.await {\n            Ok(_) => debug!(\"H2 connection finished fd: {id}\"),\n            Err(e) => debug!(\"H2 connection fd: {id} errored: {e:?}\"),\n        }\n    }\n    let _ = closed.send(true);\n}\n\nconst PING_TIMEOUT: Duration = Duration::from_secs(5);\n\nasync fn do_ping_pong(\n    mut ping_pong: h2::PingPong,\n    interval: Duration,\n    tx: oneshot::Sender<()>,\n    dropped: Arc<AtomicBool>,\n    id: UniqueIDType,\n) {\n    // delay before sending the first ping, no need to race with the first request\n    tokio::time::sleep(interval).await;\n    loop {\n        if dropped.load(Ordering::Relaxed) {\n            break;\n        }\n        let ping_fut = ping_pong.ping(h2::Ping::opaque());\n        debug!(\"H2 fd: {id} ping sent\");\n        match tokio::time::timeout(PING_TIMEOUT, ping_fut).await {\n            Err(_) => {\n                error!(\"H2 fd: {id} ping timeout\");\n                let _ = tx.send(());\n                break;\n            }\n            Ok(r) => match r {\n                Ok(_) => {\n                    debug!(\"H2 fd: {} pong received\", id);\n                    tokio::time::sleep(interval).await;\n                }\n                Err(e) => {\n                    if dropped.load(Ordering::Relaxed) {\n                        // drive_connection() exits first, no need to error again\n                        break;\n                    }\n                    error!(\"H2 fd: {id} ping error: {e}\");\n                    let _ = tx.send(());\n                    break;\n                }\n            },\n        }\n    }\n}\n\n#[cfg(test)]\nmod tests_h2 {\n    use super::*;\n    use bytes::Bytes;\n    use http::{Response, StatusCode};\n    use tokio::io::duplex;\n\n    #[tokio::test]\n    async fn h2_body_bytes_received_multi_frames() {\n        let (client_io, server_io) = duplex(65536);\n\n        // Server: respond with two DATA frames \"a\" and \"bc\"\n        tokio::spawn(async move {\n            let mut conn = h2::server::handshake(server_io).await.unwrap();\n            if let Some(result) = conn.accept().await {\n                let (req, mut send_resp) = result.unwrap();\n                assert_eq!(req.method(), http::Method::GET);\n                let resp = Response::builder().status(StatusCode::OK).body(()).unwrap();\n                let mut send_stream = send_resp.send_response(resp, false).unwrap();\n                send_stream.send_data(Bytes::from(\"a\"), false).unwrap();\n                send_stream.send_data(Bytes::from(\"bc\"), true).unwrap();\n                // Signal graceful shutdown so the accept loop can exit after the client finishes\n                conn.graceful_shutdown();\n            }\n            // Drive the server connection until the client closes\n            while let Some(_res) = conn.accept().await {}\n        });\n\n        // Client: build Http2Session and read response\n        let (send_req, connection) = h2::client::handshake(client_io).await.unwrap();\n        let (closed_tx, closed_rx) = tokio::sync::watch::channel(false);\n        let ping_timeout = Arc::new(AtomicBool::new(false));\n        tokio::spawn(async move {\n            let _ = connection.await;\n            let _ = closed_tx.send(true);\n        });\n\n        let digest = Digest::default();\n        let conn_ref = crate::connectors::http::v2::ConnectionRef::new(\n            send_req.clone(),\n            closed_rx,\n            ping_timeout,\n            0,\n            1,\n            digest,\n        );\n        let mut h2s = Http2Session::new(send_req, conn_ref);\n\n        // minimal request\n        let mut req = RequestHeader::build(\"GET\", b\"/\", None).unwrap();\n        req.insert_header(http::header::HOST, \"example.com\")\n            .unwrap();\n        h2s.write_request_header(Box::new(req), true).unwrap();\n        h2s.read_response_header().await.unwrap();\n\n        let mut total = 0;\n        while let Some(chunk) = h2s.read_response_body().await.unwrap() {\n            total += chunk.len();\n        }\n        assert_eq!(total, 3);\n        assert_eq!(h2s.body_bytes_received(), 3);\n    }\n}\n"
  },
  {
    "path": "pingora-core/src/protocols/http/v2/mod.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! HTTP/2 implementation\n\nuse std::time::Duration;\n\nuse crate::{Error, ErrorType::*, OrErr, Result};\nuse pingora_timeout::timeout;\n\nuse bytes::Bytes;\nuse h2::SendStream;\n\npub mod client;\npub mod server;\n\nasync fn reserve_and_send(\n    writer: &mut SendStream<Bytes>,\n    remaining: &mut Bytes,\n    end: bool,\n) -> Result<()> {\n    // reserve remaining bytes then wait\n    writer.reserve_capacity(remaining.len());\n    let res = std::future::poll_fn(|cx| writer.poll_capacity(cx)).await;\n\n    match res {\n        None => Error::e_explain(H2Error, \"cannot reserve capacity\"),\n        Some(ready) => {\n            let n = ready.or_err(H2Error, \"while waiting for capacity\")?;\n            let remaining_size = remaining.len();\n            let data_to_send = remaining.split_to(std::cmp::min(remaining_size, n));\n            writer\n                .send_data(data_to_send, remaining.is_empty() && end)\n                .or_err(WriteError, \"while writing h2 request body\")?;\n            Ok(())\n        }\n    }\n}\n\n/// A helper function to write the body of h2 streams.\npub async fn write_body(\n    writer: &mut SendStream<Bytes>,\n    data: Bytes,\n    end: bool,\n    write_timeout: Option<Duration>,\n) -> Result<()> {\n    let mut remaining = data;\n\n    // Cannot poll 0 capacity, so send it directly.\n    if remaining.is_empty() {\n        writer\n            .send_data(remaining, end)\n            .or_err(WriteError, \"while writing h2 request body\")?;\n        return Ok(());\n    }\n\n    loop {\n        match write_timeout {\n            Some(t) => match timeout(t, reserve_and_send(writer, &mut remaining, end)).await {\n                Ok(res) => res?,\n                Err(_) => Error::e_explain(\n                    WriteTimedout,\n                    format!(\"while writing h2 request body, timeout: {t:?}\"),\n                )?,\n            },\n            None => {\n                reserve_and_send(writer, &mut remaining, end).await?;\n            }\n        }\n        if remaining.is_empty() {\n            return Ok(());\n        }\n    }\n}\n\n#[cfg(test)]\nmod test {\n    use std::{sync::Arc, time::Duration};\n\n    use bytes::Bytes;\n    use futures::SinkExt;\n    use h2::frame::*;\n    use http::{HeaderMap, Method, Uri};\n    use tokio::io::{duplex, AsyncReadExt, AsyncWriteExt, DuplexStream};\n    use tokio_stream::StreamExt;\n\n    use pingora_http::{RequestHeader, ResponseHeader};\n    use pingora_timeout::sleep;\n\n    use crate::protocols::{\n        http::v2::server::{handshake, HttpSession},\n        Digest,\n    };\n\n    #[tokio::test]\n    async fn test_client_write_timeout() {\n        let mut handles = vec![];\n\n        let (client, mut server) = duplex(65536);\n\n        // Client\n        handles.push(tokio::spawn(async move {\n            let conn = crate::connectors::http::v2::handshake(Box::new(client), 500, None)\n                .await\n                .unwrap();\n\n            let mut h2_stream = conn.spawn_stream().await.unwrap().unwrap();\n            h2_stream.write_timeout = Some(Duration::from_millis(100));\n\n            let mut request = RequestHeader::build(\"GET\", b\"/\", None).unwrap();\n            request.insert_header(\"Host\", \"one.one.one.one\").unwrap();\n\n            h2_stream\n                .write_request_header(Box::new(request), false)\n                .unwrap();\n\n            h2_stream.read_response_header().await.unwrap();\n            assert_eq!(h2_stream.response_header().unwrap().status.as_u16(), 200);\n\n            let err = h2_stream\n                .write_request_body(Bytes::from_static(b\"client body\"), true)\n                .await\n                .err()\n                .unwrap();\n            assert_eq!(err.etype(), &pingora_error::ErrorType::WriteTimedout);\n        }));\n\n        // Server\n        handles.push(tokio::spawn(async move {\n            // 0. Prepare outbound frames\n            let mut outbound: Vec<h2::frame::Frame<Bytes>> = Vec::new();\n\n            let mut settings = Settings::default();\n\n            settings.set_initial_window_size(Some(1));\n            settings.set_max_concurrent_streams(Some(1));\n\n            outbound.push(settings.into());\n            outbound.push(Settings::ack().into());\n\n            let headers = HeaderMap::new();\n\n            outbound.push(\n                Headers::new(1.into(), Pseudo::response(http::StatusCode::OK), headers).into(),\n            );\n\n            outbound.push(WindowUpdate::new(1.into(), 10000).into());\n\n            // 1. Read preface from the client\n            server.read_exact(&mut [0u8; 24]).await.unwrap();\n\n            let mut server: h2::Codec<DuplexStream, Bytes> = h2::Codec::new(server);\n\n            // 2. Drain client's frames\n            for _ in 0..3 {\n                _ = server.next().await.unwrap();\n            }\n\n            // 3. Send frames\n            for (i, frame) in outbound.into_iter().enumerate() {\n                if i == 3 {\n                    // Delay WindowUpdate to trigger client side write timeout on capacity await\n                    sleep(Duration::from_millis(200)).await;\n                }\n                _ = server.send(frame).await;\n            }\n        }));\n\n        for handle in handles {\n            // ensure no panics\n            assert!(handle.await.is_ok());\n        }\n    }\n\n    #[tokio::test]\n    async fn test_server_write_timeout() {\n        let mut handles = vec![];\n\n        let (mut client, server) = duplex(65536);\n\n        // Client\n        handles.push(tokio::spawn(async move {\n            // 0. Prepare outbound frames\n            let mut outbound: Vec<h2::frame::Frame<Bytes>> = Vec::new();\n\n            let mut settings = Settings::default();\n\n            settings.set_initial_window_size(Some(1));\n            settings.set_max_concurrent_streams(Some(1));\n            outbound.push(settings.into());\n\n            outbound.push(Settings::ack().into());\n\n            let mut headers = Headers::new(\n                1.into(),\n                Pseudo::request(\n                    Method::GET,\n                    Uri::from_static(\"https://one.one.one.one\"),\n                    None,\n                ),\n                HeaderMap::new(),\n            );\n            headers.set_end_headers();\n            outbound.push(headers.into());\n\n            outbound.push(WindowUpdate::new(1.into(), 10000).into());\n\n            // 1. Write h2 preface\n            client\n                .write_all(b\"PRI * HTTP/2.0\\r\\n\\r\\nSM\\r\\n\\r\\n\")\n                .await\n                .unwrap();\n\n            // 2. Send frames\n            let mut client: h2::Codec<DuplexStream, Bytes> = h2::Codec::new(client);\n\n            for (i, frame) in outbound.into_iter().enumerate() {\n                if i == 3 {\n                    // Delay WindowUpdate to trigger server side write timeout on capacity await\n                    sleep(Duration::from_millis(200)).await;\n                }\n                _ = client.send(frame).await;\n            }\n\n            // 3. Drain server's frames\n            for _ in 0..3 {\n                _ = client.next().await.unwrap();\n            }\n        }));\n\n        // Server\n        let mut connection = handshake(Box::new(server), None).await.unwrap();\n        let digest = Arc::new(Digest::default());\n\n        while let Some(mut h2_stream) = HttpSession::from_h2_conn(&mut connection, digest.clone())\n            .await\n            .unwrap()\n        {\n            handles.push(tokio::spawn(async move {\n                h2_stream.set_write_timeout(Some(Duration::from_millis(100)));\n                let req = h2_stream.req_header();\n                assert_eq!(req.method, Method::GET);\n\n                let response_header = Box::new(ResponseHeader::build(200, None).unwrap());\n                assert!(h2_stream\n                    .write_response_header(response_header.clone(), false)\n                    .is_ok());\n\n                let err = h2_stream\n                    .write_body(Bytes::from_static(b\"server body\"), true)\n                    .await\n                    .err()\n                    .unwrap();\n                assert_eq!(err.etype(), &pingora_error::ErrorType::WriteTimedout);\n            }));\n        }\n\n        for handle in handles {\n            // ensure no panics\n            assert!(handle.await.is_ok());\n        }\n    }\n}\n"
  },
  {
    "path": "pingora-core/src/protocols/http/v2/server.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! HTTP/2 server session\n\nuse bytes::Bytes;\nuse futures::Future;\nuse h2::server;\nuse h2::server::SendResponse;\nuse h2::{RecvStream, SendStream};\nuse http::header::HeaderName;\nuse http::uri::PathAndQuery;\nuse http::{header, HeaderMap, Response};\nuse log::{debug, warn};\nuse pingora_http::{RequestHeader, ResponseHeader};\nuse pingora_timeout::timeout;\nuse std::sync::Arc;\nuse std::task::ready;\nuse std::time::Duration;\n\nuse crate::protocols::http::body_buffer::FixedBuffer;\nuse crate::protocols::http::date::get_cached_date;\nuse crate::protocols::http::v1::client::http_req_header_to_wire;\nuse crate::protocols::http::HttpTask;\nuse crate::protocols::{Digest, SocketAddr, Stream};\nuse crate::{Error, ErrorType, OrErr, Result};\n\nconst BODY_BUF_LIMIT: usize = 1024 * 64;\n\ntype H2Connection<S> = server::Connection<S, Bytes>;\n\npub use h2::server::Builder as H2Options;\n\n/// Perform HTTP/2 connection handshake with an established (TLS) connection.\n///\n/// The optional `options` allow to adjust certain HTTP/2 parameters and settings.\n/// See [`H2Options`] for more details.\npub async fn handshake(io: Stream, options: Option<H2Options>) -> Result<H2Connection<Stream>> {\n    let options = options.unwrap_or_default();\n    let res = options.handshake(io).await;\n\n    match res {\n        Ok(connection) => {\n            debug!(\"H2 handshake done.\");\n            Ok(connection)\n        }\n        Err(e) => Error::e_because(\n            ErrorType::HandshakeError,\n            \"while h2 handshaking with client\",\n            e,\n        ),\n    }\n}\n\nuse futures::task::Context;\nuse futures::task::Poll;\nuse std::pin::Pin;\n/// The future to poll for an idle session.\n///\n/// Calling `.await` in this object will not return until the client decides to close this stream.\npub struct Idle<'a>(&'a mut HttpSession);\n\nimpl Future for Idle<'_> {\n    type Output = Result<h2::Reason>;\n\n    fn poll(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Self::Output> {\n        if let Some(body_writer) = self.0.send_response_body.as_mut() {\n            body_writer.poll_reset(cx)\n        } else {\n            self.0.send_response.poll_reset(cx)\n        }\n        .map_err(|e| Error::because(ErrorType::H2Error, \"downstream error while idling\", e))\n    }\n}\n\n/// HTTP/2 server session\npub struct HttpSession {\n    request_header: RequestHeader,\n    request_body_reader: RecvStream,\n    send_response: SendResponse<Bytes>,\n    send_response_body: Option<SendStream<Bytes>>,\n    // Remember what has been written\n    response_written: Option<Box<ResponseHeader>>,\n    // Indicate that whether a END_STREAM is already sent\n    // in order to tell whether needs to send one extra FRAME when this response finishes\n    ended: bool,\n    // How many (application, not wire) request body bytes have been read so far.\n    body_read: usize,\n    // How many (application, not wire) response body bytes have been sent so far.\n    body_sent: usize,\n    // buffered request body for retry logic\n    retry_buffer: Option<FixedBuffer>,\n    // digest to record underlying connection info\n    digest: Arc<Digest>,\n    /// The write timeout which will be applied to writing response body.\n    /// The timeout is reset on every write. This is not a timeout on the overall duration of the\n    /// response.\n    pub write_timeout: Option<Duration>,\n    // How long to wait when draining (discarding) request body\n    total_drain_timeout: Option<Duration>,\n}\n\nimpl HttpSession {\n    /// Create a new [`HttpSession`] from the HTTP/2 connection.\n    /// This function returns a new HTTP/2 session when the provided HTTP/2 connection, `conn`,\n    /// establishes a new HTTP/2 stream to this server.\n    ///\n    /// A [`Digest`] from the IO stream is also stored in the resulting session, since the\n    /// session doesn't have access to the underlying stream (and the stream itself isn't\n    /// accessible from the `h2::server::Connection`).\n    ///\n    /// Note: in order to handle all **existing** and new HTTP/2 sessions, the server must call\n    /// this function in a loop until the client decides to close the connection.\n    ///\n    /// `None` will be returned when the connection is closing so that the loop can exit.\n    ///\n    pub async fn from_h2_conn(\n        conn: &mut H2Connection<Stream>,\n        digest: Arc<Digest>,\n    ) -> Result<Option<Self>> {\n        // NOTE: conn.accept().await is what drives the entire connection.\n        let res = conn.accept().await.transpose().or_err(\n            ErrorType::H2Error,\n            \"while accepting new downstream requests\",\n        )?;\n\n        Ok(res.map(|(req, send_response)| {\n            let (request_header, request_body_reader) = req.into_parts();\n            HttpSession {\n                request_header: request_header.into(),\n                request_body_reader,\n                send_response,\n                send_response_body: None,\n                response_written: None,\n                ended: false,\n                body_read: 0,\n                body_sent: 0,\n                retry_buffer: None,\n                digest,\n                write_timeout: None,\n                total_drain_timeout: None,\n            }\n        }))\n    }\n\n    /// The request sent from the client\n    ///\n    /// Different from its HTTP/1.X counterpart, this function never panics as the request is already\n    /// read when established a new HTTP/2 stream.\n    pub fn req_header(&self) -> &RequestHeader {\n        &self.request_header\n    }\n\n    /// A mutable reference to request sent from the client\n    ///\n    /// Different from its HTTP/1.X counterpart, this function never panics as the request is already\n    /// read when established a new HTTP/2 stream.\n    pub fn req_header_mut(&mut self) -> &mut RequestHeader {\n        &mut self.request_header\n    }\n\n    /// Read request body bytes. `None` when there is no more body to read.\n    pub async fn read_body_bytes(&mut self) -> Result<Option<Bytes>> {\n        // TODO: timeout\n        let data = self.request_body_reader.data().await.transpose().or_err(\n            ErrorType::ReadError,\n            \"while reading downstream request body\",\n        )?;\n        if let Some(data) = data.as_ref() {\n            self.body_read += data.len();\n            if let Some(buffer) = self.retry_buffer.as_mut() {\n                buffer.write_to_buffer(data);\n            }\n            let _ = self\n                .request_body_reader\n                .flow_control()\n                .release_capacity(data.len());\n        }\n        Ok(data)\n    }\n\n    #[doc(hidden)]\n    pub fn poll_read_body_bytes(\n        &mut self,\n        cx: &mut Context<'_>,\n    ) -> Poll<Option<Result<Bytes, h2::Error>>> {\n        let data = match ready!(self.request_body_reader.poll_data(cx)).transpose() {\n            Ok(data) => data,\n            Err(err) => return Poll::Ready(Some(Err(err))),\n        };\n\n        if let Some(data) = data {\n            self.body_read += data.len();\n            self.request_body_reader\n                .flow_control()\n                .release_capacity(data.len())?;\n            return Poll::Ready(Some(Ok(data)));\n        }\n\n        Poll::Ready(None)\n    }\n\n    async fn do_drain_request_body(&mut self) -> Result<()> {\n        loop {\n            match self.read_body_bytes().await {\n                Ok(Some(_)) => { /* continue to drain */ }\n                Ok(None) => return Ok(()), // done\n                Err(e) => return Err(e),\n            }\n        }\n    }\n\n    /// Drain the request body. `Ok(())` when there is no (more) body to read.\n    // NOTE for h2 it may be worth allowing cancellation of the stream via reset.\n    pub async fn drain_request_body(&mut self) -> Result<()> {\n        if self.is_body_done() {\n            return Ok(());\n        }\n        match self.total_drain_timeout {\n            Some(t) => match timeout(t, self.do_drain_request_body()).await {\n                Ok(res) => res,\n                Err(_) => Error::e_explain(\n                    ErrorType::ReadTimedout,\n                    format!(\"draining body, timeout: {t:?}\"),\n                ),\n            },\n            None => self.do_drain_request_body().await,\n        }\n    }\n\n    /// Sets the downstream write timeout. This will trigger if we're unable\n    /// to write to the stream after `timeout`.\n    pub fn set_write_timeout(&mut self, timeout: Option<Duration>) {\n        self.write_timeout = timeout;\n    }\n\n    /// Get the write timeout.\n    pub fn get_write_timeout(&self) -> Option<Duration> {\n        self.write_timeout\n    }\n\n    /// Sets the total drain timeout. This `timeout` will be used while draining\n    /// the request body.\n    pub fn set_total_drain_timeout(&mut self, timeout: Option<Duration>) {\n        self.total_drain_timeout = timeout;\n    }\n\n    /// Get the total drain timeout.\n    pub fn get_total_drain_timeout(&self) -> Option<Duration> {\n        self.total_drain_timeout\n    }\n\n    // the write_* don't have timeouts because the actual writing happens on the connection\n    // not here.\n\n    /// Write the response header to the client.\n    /// # the `end` flag\n    /// `end` marks the end of this session.\n    /// If the `end` flag is set, no more header or body can be sent to the client.\n    pub fn write_response_header(\n        &mut self,\n        mut header: Box<ResponseHeader>,\n        end: bool,\n    ) -> Result<()> {\n        if self.ended {\n            // TODO: error or warn?\n            return Ok(());\n        }\n\n        if header.status.is_informational() {\n            // ignore informational response 1xx header because send_response() can only be called once\n            // https://github.com/hyperium/h2/issues/167\n            debug!(\"ignoring informational headers\");\n            return Ok(());\n        }\n\n        if self.response_written.as_ref().is_some() {\n            warn!(\"Response header is already sent, cannot send again\");\n            return Ok(());\n        }\n\n        /* update headers */\n        header.insert_header(header::DATE, get_cached_date())?;\n\n        // remove other h1 hop headers that cannot be present in H2\n        // https://httpwg.org/specs/rfc7540.html#n-connection-specific-header-fields\n        header.remove_header(&header::TRANSFER_ENCODING);\n        header.remove_header(&header::CONNECTION);\n        header.remove_header(&header::UPGRADE);\n        header.remove_header(&HeaderName::from_static(\"keep-alive\"));\n        header.remove_header(&HeaderName::from_static(\"proxy-connection\"));\n\n        let resp = Response::from_parts(header.as_owned_parts(), ());\n\n        let body_writer = self.send_response.send_response(resp, end).or_err(\n            ErrorType::WriteError,\n            \"while writing h2 response to downstream\",\n        )?;\n\n        self.response_written = Some(header);\n        self.send_response_body = Some(body_writer);\n        self.ended = self.ended || end;\n        Ok(())\n    }\n\n    /// Write response body to the client. See [Self::write_response_header] for how to use `end`.\n    pub async fn write_body(&mut self, data: Bytes, end: bool) -> Result<()> {\n        match self.write_timeout {\n            Some(t) => match timeout(t, self.do_write_body(data, end)).await {\n                Ok(res) => res,\n                Err(_) => Error::e_explain(\n                    ErrorType::WriteTimedout,\n                    format!(\"writing body, timeout: {t:?}\"),\n                ),\n            },\n            None => self.do_write_body(data, end).await,\n        }\n    }\n\n    async fn do_write_body(&mut self, data: Bytes, end: bool) -> Result<()> {\n        if self.ended {\n            // NOTE: in h1, we also track to see if content-length matches the data\n            // We have not tracked that in h2\n            warn!(\"Try to write body after end of stream, dropping the extra data\");\n            return Ok(());\n        }\n        let Some(writer) = self.send_response_body.as_mut() else {\n            return Err(Error::explain(\n                ErrorType::H2Error,\n                \"try to send body before header is sent\",\n            ));\n        };\n        let data_len = data.len();\n        super::write_body(writer, data, end, self.write_timeout)\n            .await\n            .map_err(|e| e.into_down())?;\n        self.body_sent += data_len;\n        self.ended = self.ended || end;\n        Ok(())\n    }\n\n    /// Write response trailers to the client, this also closes the stream.\n    pub fn write_trailers(&mut self, trailers: HeaderMap) -> Result<()> {\n        if self.ended {\n            warn!(\"Tried to write trailers after end of stream, dropping them\");\n            return Ok(());\n        }\n        let Some(writer) = self.send_response_body.as_mut() else {\n            return Err(Error::explain(\n                ErrorType::H2Error,\n                \"try to send trailers before header is sent\",\n            ));\n        };\n        writer.send_trailers(trailers).or_err(\n            ErrorType::WriteError,\n            \"while writing h2 response trailers to downstream\",\n        )?;\n        // sending trailers closes the stream\n        self.ended = true;\n        Ok(())\n    }\n\n    /// Similar to [Self::write_response_header], this function takes a reference instead\n    pub fn write_response_header_ref(&mut self, header: &ResponseHeader, end: bool) -> Result<()> {\n        self.write_response_header(Box::new(header.clone()), end)\n    }\n\n    // TODO: trailer\n\n    /// Mark the session end. If no `end` flag is already set before this call, this call will\n    /// signal the client. Otherwise this call does nothing.\n    ///\n    /// Dropping this object without sending `end` will cause an error to the client, which will cause\n    /// the client to treat this session as bad or incomplete.\n    pub fn finish(&mut self) -> Result<()> {\n        if self.ended {\n            // already ended the stream\n            return Ok(());\n        }\n        if let Some(writer) = self.send_response_body.as_mut() {\n            // use an empty data frame to signal the end\n            writer.send_data(\"\".into(), true).or_err(\n                ErrorType::WriteError,\n                \"while writing h2 response body to downstream\",\n            )?;\n            self.ended = true;\n        };\n        // else: the response header is not sent, do nothing now.\n        // When send_response_body is dropped, an RST_STREAM will be sent\n\n        Ok(())\n    }\n\n    pub async fn response_duplex_vec(&mut self, tasks: Vec<HttpTask>) -> Result<bool> {\n        let mut end_stream = false;\n        for task in tasks.into_iter() {\n            end_stream = match task {\n                HttpTask::Header(header, end) => {\n                    self.write_response_header(header, end)\n                        .map_err(|e| e.into_down())?;\n                    end\n                }\n                HttpTask::Body(data, end) => match data {\n                    Some(d) => {\n                        if !d.is_empty() {\n                            self.write_body(d, end).await.map_err(|e| e.into_down())?;\n                        }\n                        end\n                    }\n                    None => end,\n                },\n                HttpTask::UpgradedBody(..) => {\n                    // Seeing an Upgraded body means that the upstream session\n                    // was H1.1 that upgraded.\n                    //\n                    // While the downstream H2 session may encapsulate the opaque body bytes,\n                    // this represents an undefined discrepancy and change between how\n                    // the upstream and downstream sessions began intepreting the response body.\n                    return Error::e_explain(\n                        ErrorType::InternalError,\n                        \"upgraded body on h2 server session\",\n                    );\n                }\n                HttpTask::Trailer(Some(trailers)) => {\n                    self.write_trailers(*trailers)?;\n                    true\n                }\n                HttpTask::Trailer(None) => true,\n                HttpTask::Done => true,\n                HttpTask::Failed(e) => {\n                    return Err(e);\n                }\n            } || end_stream // safe guard in case `end` in tasks flips from true to false\n        }\n        if end_stream {\n            // no-op if finished already\n            self.finish().map_err(|e| e.into_down())?;\n        }\n        Ok(end_stream)\n    }\n\n    /// Return a string `$METHOD $PATH, Host: $HOST`. Mostly for logging and debug purpose\n    pub fn request_summary(&self) -> String {\n        format!(\n            \"{} {}, Host: {}:{}\",\n            self.request_header.method,\n            self.request_header\n                .uri\n                .path_and_query()\n                .map(PathAndQuery::as_str)\n                .unwrap_or_default(),\n            self.request_header.uri.host().unwrap_or_default(),\n            self.req_header()\n                .uri\n                .port()\n                .as_ref()\n                .map(|port| port.as_str())\n                .unwrap_or_default()\n        )\n    }\n\n    /// Return the written response header. `None` if it is not written yet.\n    pub fn response_written(&self) -> Option<&ResponseHeader> {\n        self.response_written.as_deref()\n    }\n\n    /// Give up the stream abruptly.\n    ///\n    /// This will send a `INTERNAL_ERROR` stream error to the client\n    pub fn shutdown(&mut self) {\n        if !self.ended {\n            self.send_response.send_reset(h2::Reason::INTERNAL_ERROR);\n        }\n    }\n\n    #[doc(hidden)]\n    pub fn take_response_body_writer(&mut self) -> Option<SendStream<Bytes>> {\n        self.send_response_body.take()\n    }\n\n    // This is a hack for pingora-proxy to create subrequests from h2 server session\n    // TODO: be able to convert from h2 to h1 subrequest\n    pub fn pseudo_raw_h1_request_header(&self) -> Bytes {\n        let buf = http_req_header_to_wire(&self.request_header).unwrap(); // safe, None only when version unknown\n        buf.freeze()\n    }\n\n    /// Whether there is no more body to read\n    pub fn is_body_done(&self) -> bool {\n        // Check no body in request\n        // Also check we hit end of stream\n        self.is_body_empty() || self.request_body_reader.is_end_stream()\n    }\n\n    /// Whether there is any body to read. true means there no body in request.\n    pub fn is_body_empty(&self) -> bool {\n        self.body_read == 0\n            && (self.request_body_reader.is_end_stream()\n                || self\n                    .request_header\n                    .headers\n                    .get(header::CONTENT_LENGTH)\n                    .is_some_and(|cl| cl.as_bytes() == b\"0\"))\n    }\n\n    pub fn retry_buffer_truncated(&self) -> bool {\n        self.retry_buffer\n            .as_ref()\n            .map_or_else(|| false, |r| r.is_truncated())\n    }\n\n    pub fn enable_retry_buffering(&mut self) {\n        if self.retry_buffer.is_none() {\n            self.retry_buffer = Some(FixedBuffer::new(BODY_BUF_LIMIT))\n        }\n    }\n\n    pub fn get_retry_buffer(&self) -> Option<Bytes> {\n        self.retry_buffer.as_ref().and_then(|b| {\n            if b.is_truncated() {\n                None\n            } else {\n                b.get_buffer()\n            }\n        })\n    }\n\n    /// `async fn idle() -> Result<Reason, Error>;`\n    /// This async fn will be pending forever until the client closes the stream/connection\n    /// This function is used for watching client status so that the server is able to cancel\n    /// its internal tasks as the client waiting for the tasks goes away\n    pub fn idle(&mut self) -> Idle<'_> {\n        Idle(self)\n    }\n\n    /// Similar to `read_body_bytes()` but will be pending after Ok(None) is returned,\n    /// until the client closes the connection\n    pub async fn read_body_or_idle(&mut self, no_body_expected: bool) -> Result<Option<Bytes>> {\n        if no_body_expected || self.is_body_done() {\n            let reason = self.idle().await?;\n            Error::e_explain(\n                ErrorType::H2Error,\n                format!(\"Client closed H2, reason: {reason}\"),\n            )\n        } else {\n            self.read_body_bytes().await\n        }\n    }\n\n    /// Return how many response body bytes (application, not wire) already sent downstream\n    pub fn body_bytes_sent(&self) -> usize {\n        self.body_sent\n    }\n\n    /// Return how many request body bytes (application, not wire) already read from downstream\n    pub fn body_bytes_read(&self) -> usize {\n        self.body_read\n    }\n\n    /// Return the [Digest] of the connection.\n    pub fn digest(&self) -> Option<&Digest> {\n        Some(&self.digest)\n    }\n\n    /// Return a mutable [Digest] reference for the connection.\n    pub fn digest_mut(&mut self) -> Option<&mut Digest> {\n        Arc::get_mut(&mut self.digest)\n    }\n\n    /// Return the server (local) address recorded in the connection digest.\n    pub fn server_addr(&self) -> Option<&SocketAddr> {\n        self.digest.socket_digest.as_ref().map(|d| d.local_addr())?\n    }\n\n    /// Return the client (peer) address recorded in the connection digest.\n    pub fn client_addr(&self) -> Option<&SocketAddr> {\n        self.digest.socket_digest.as_ref().map(|d| d.peer_addr())?\n    }\n}\n\n#[cfg(test)]\nmod test {\n    use super::*;\n    use http::{HeaderValue, Method, Request};\n    use tokio::io::duplex;\n\n    #[tokio::test]\n    async fn test_server_handshake_accept_request() {\n        let (client, server) = duplex(65536);\n        let client_body = \"test client body\";\n        let server_body = \"test server body\";\n\n        let mut expected_trailers = HeaderMap::new();\n        expected_trailers.insert(\"test\", HeaderValue::from_static(\"trailers\"));\n        let trailers = expected_trailers.clone();\n\n        let mut handles = vec![];\n        handles.push(tokio::spawn(async move {\n            let (h2, connection) = h2::client::handshake(client).await.unwrap();\n            tokio::spawn(async move {\n                connection.await.unwrap();\n            });\n\n            let mut h2 = h2.ready().await.unwrap();\n\n            let request = Request::builder()\n                .method(Method::GET)\n                .uri(\"https://www.example.com/\")\n                .body(())\n                .unwrap();\n\n            let (response, mut req_body) = h2.send_request(request, false).unwrap();\n            req_body.reserve_capacity(client_body.len());\n            req_body.send_data(client_body.into(), true).unwrap();\n\n            let (head, mut body) = response.await.unwrap().into_parts();\n            assert_eq!(head.status, 200);\n            let data = body.data().await.unwrap().unwrap();\n            assert_eq!(data, server_body);\n            let resp_trailers = body.trailers().await.unwrap().unwrap();\n            assert_eq!(resp_trailers, expected_trailers);\n        }));\n\n        let mut connection = handshake(Box::new(server), None).await.unwrap();\n        let digest = Arc::new(Digest::default());\n\n        while let Some(mut http) = HttpSession::from_h2_conn(&mut connection, digest.clone())\n            .await\n            .unwrap()\n        {\n            let trailers = trailers.clone();\n            handles.push(tokio::spawn(async move {\n                let req = http.req_header();\n                assert_eq!(req.method, Method::GET);\n                assert_eq!(req.uri, \"https://www.example.com/\");\n\n                http.enable_retry_buffering();\n\n                assert!(!http.is_body_empty());\n                assert!(!http.is_body_done());\n\n                let body = http.read_body_or_idle(false).await.unwrap().unwrap();\n                assert_eq!(body, client_body);\n                assert!(http.is_body_done());\n                assert_eq!(http.body_bytes_read(), 16);\n\n                let retry_body = http.get_retry_buffer().unwrap();\n                assert_eq!(retry_body, client_body);\n\n                // test idling before response header is sent\n                tokio::select! {\n                    _ = http.idle() => {panic!(\"downstream should be idling\")},\n                    _= tokio::time::sleep(tokio::time::Duration::from_secs(1)) => {}\n                }\n\n                let response_header = Box::new(ResponseHeader::build(200, None).unwrap());\n                assert!(http\n                    .write_response_header(response_header.clone(), false)\n                    .is_ok());\n                // this write should be ignored otherwise we will error\n                assert!(http.write_response_header(response_header, false).is_ok());\n\n                // test idling after response header is sent\n                tokio::select! {\n                    _ = http.read_body_or_idle(false) => {panic!(\"downstream should be idling\")},\n                    _= tokio::time::sleep(tokio::time::Duration::from_secs(1)) => {}\n                }\n\n                // end: false here to verify finish() closes the stream nicely\n                http.write_body(server_body.into(), false).await.unwrap();\n                assert_eq!(http.body_bytes_sent(), 16);\n\n                http.write_trailers(trailers).unwrap();\n                http.finish().unwrap();\n            }));\n        }\n        for handle in handles {\n            // ensure no panics\n            assert!(handle.await.is_ok());\n        }\n    }\n\n    #[tokio::test]\n    async fn test_req_content_length_eq_0_and_no_header_eos() {\n        let (client, server) = duplex(65536);\n\n        let server_body = \"test server body\";\n\n        let mut handles = vec![];\n\n        handles.push(tokio::spawn(async move {\n            let (h2, connection) = h2::client::handshake(client).await.unwrap();\n            tokio::spawn(async move {\n                connection.await.unwrap();\n            });\n\n            let mut h2 = h2.ready().await.unwrap();\n\n            let request = Request::builder()\n                .method(Method::POST)\n                .uri(\"https://www.example.com/\")\n                .header(\"content-length\", \"0\") // explicitly set\n                .body(())\n                .unwrap();\n\n            let (response, mut req_body) = h2.send_request(request, false).unwrap(); // no EOS\n\n            let (head, mut body) = response.await.unwrap().into_parts();\n\n            assert_eq!(head.status, 200);\n            let data = body.data().await.unwrap().unwrap();\n            assert_eq!(data, server_body);\n\n            req_body.send_data(\"\".into(), true).unwrap(); // set EOS after read the resp body\n        }));\n\n        let mut connection = handshake(Box::new(server), None).await.unwrap();\n        let digest = Arc::new(Digest::default());\n\n        while let Some(mut http) = HttpSession::from_h2_conn(&mut connection, digest.clone())\n            .await\n            .unwrap()\n        {\n            handles.push(tokio::spawn(async move {\n                let req = http.req_header();\n                assert_eq!(req.method, Method::POST);\n                assert_eq!(req.uri, \"https://www.example.com/\");\n\n                // 1. Check body related methods\n                http.enable_retry_buffering();\n                assert!(http.is_body_empty());\n                assert!(http.is_body_done());\n                let retry_body = http.get_retry_buffer();\n                assert!(retry_body.is_none());\n\n                // 2. Send response\n                let response_header = Box::new(ResponseHeader::build(200, None).unwrap());\n                assert!(http\n                    .write_response_header(response_header.clone(), false)\n                    .is_ok());\n\n                http.write_body(server_body.into(), false).await.unwrap();\n                assert_eq!(http.body_bytes_sent(), 16);\n\n                // 3. Waiting for the reset from the client\n                assert!(http.read_body_or_idle(http.is_body_done()).await.is_err());\n            }));\n        }\n\n        for handle in handles {\n            // ensure no panics\n            assert!(handle.await.is_ok());\n        }\n    }\n\n    #[tokio::test]\n    async fn test_req_header_no_eos_empty_data_with_eos() {\n        let (client, server) = duplex(65536);\n\n        let server_body = \"test server body\";\n\n        let mut handles = vec![];\n\n        handles.push(tokio::spawn(async move {\n            let (h2, connection) = h2::client::handshake(client).await.unwrap();\n            tokio::spawn(async move {\n                connection.await.unwrap();\n            });\n\n            let mut h2 = h2.ready().await.unwrap();\n\n            let request = Request::builder()\n                .method(Method::POST)\n                .uri(\"https://www.example.com/\")\n                .body(())\n                .unwrap();\n\n            let (response, mut req_body) = h2.send_request(request, false).unwrap(); // no EOS\n\n            let (head, mut body) = response.await.unwrap().into_parts();\n\n            assert_eq!(head.status, 200);\n            let data = body.data().await.unwrap().unwrap();\n            assert_eq!(data, server_body);\n\n            req_body.send_data(\"\".into(), true).unwrap(); // set EOS after read the resp body\n        }));\n\n        let mut connection = handshake(Box::new(server), None).await.unwrap();\n        let digest = Arc::new(Digest::default());\n\n        while let Some(mut http) = HttpSession::from_h2_conn(&mut connection, digest.clone())\n            .await\n            .unwrap()\n        {\n            handles.push(tokio::spawn(async move {\n                let req = http.req_header();\n                assert_eq!(req.method, Method::POST);\n                assert_eq!(req.uri, \"https://www.example.com/\");\n\n                // 1. Check body related methods\n                http.enable_retry_buffering();\n                assert!(!http.is_body_empty());\n                assert!(!http.is_body_done());\n                let retry_body = http.get_retry_buffer();\n                assert!(retry_body.is_none());\n\n                // 2. Send response\n                let response_header = Box::new(ResponseHeader::build(200, None).unwrap());\n                assert!(http\n                    .write_response_header(response_header.clone(), false)\n                    .is_ok());\n\n                http.write_body(server_body.into(), false).await.unwrap();\n                assert_eq!(http.body_bytes_sent(), 16);\n\n                // 3. Waiting for the client to close stream.\n                http.read_body_or_idle(http.is_body_done()).await.unwrap();\n            }));\n        }\n\n        for handle in handles {\n            // ensure no panics\n            assert!(handle.await.is_ok());\n        }\n    }\n}\n"
  },
  {
    "path": "pingora-core/src/protocols/l4/ext.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! Extensions to the regular TCP APIs\n\n#![allow(non_camel_case_types)]\n\n#[cfg(unix)]\nuse libc::socklen_t;\n#[cfg(target_os = \"linux\")]\nuse libc::{c_int, c_ulonglong, c_void};\nuse pingora_error::{Error, ErrorType::*, OrErr, Result};\nuse std::io::{self, ErrorKind};\nuse std::mem;\nuse std::net::SocketAddr;\n#[cfg(unix)]\nuse std::os::unix::io::{AsRawFd, RawFd};\n#[cfg(windows)]\nuse std::os::windows::io::{AsRawSocket, RawSocket};\nuse std::time::Duration;\n#[cfg(unix)]\nuse tokio::net::UnixStream;\nuse tokio::net::{TcpSocket, TcpStream};\n\nuse crate::connectors::l4::BindTo;\n\n/// The (copy of) the kernel struct tcp_info returns\n#[repr(C)]\n#[derive(Copy, Clone, Debug)]\npub struct TCP_INFO {\n    pub tcpi_state: u8,\n    pub tcpi_ca_state: u8,\n    pub tcpi_retransmits: u8,\n    pub tcpi_probes: u8,\n    pub tcpi_backoff: u8,\n    pub tcpi_options: u8,\n    pub tcpi_snd_wscale_4_rcv_wscale_4: u8,\n    pub tcpi_delivery_rate_app_limited: u8,\n    pub tcpi_rto: u32,\n    pub tcpi_ato: u32,\n    pub tcpi_snd_mss: u32,\n    pub tcpi_rcv_mss: u32,\n    pub tcpi_unacked: u32,\n    pub tcpi_sacked: u32,\n    pub tcpi_lost: u32,\n    pub tcpi_retrans: u32,\n    pub tcpi_fackets: u32,\n    pub tcpi_last_data_sent: u32,\n    pub tcpi_last_ack_sent: u32,\n    pub tcpi_last_data_recv: u32,\n    pub tcpi_last_ack_recv: u32,\n    pub tcpi_pmtu: u32,\n    pub tcpi_rcv_ssthresh: u32,\n    pub tcpi_rtt: u32,\n    pub tcpi_rttvar: u32,\n    pub tcpi_snd_ssthresh: u32,\n    pub tcpi_snd_cwnd: u32,\n    pub tcpi_advmss: u32,\n    pub tcpi_reordering: u32,\n    pub tcpi_rcv_rtt: u32,\n    pub tcpi_rcv_space: u32,\n    pub tcpi_total_retrans: u32,\n    pub tcpi_pacing_rate: u64,\n    pub tcpi_max_pacing_rate: u64,\n    pub tcpi_bytes_acked: u64,\n    pub tcpi_bytes_received: u64,\n    pub tcpi_segs_out: u32,\n    pub tcpi_segs_in: u32,\n    pub tcpi_notsent_bytes: u32,\n    pub tcpi_min_rtt: u32,\n    pub tcpi_data_segs_in: u32,\n    pub tcpi_data_segs_out: u32,\n    pub tcpi_delivery_rate: u64,\n    pub tcpi_busy_time: u64,\n    pub tcpi_rwnd_limited: u64,\n    pub tcpi_sndbuf_limited: u64,\n    pub tcpi_delivered: u32,\n    pub tcpi_delivered_ce: u32,\n    pub tcpi_bytes_sent: u64,\n    pub tcpi_bytes_retrans: u64,\n    pub tcpi_dsack_dups: u32,\n    pub tcpi_reord_seen: u32,\n    pub tcpi_rcv_ooopack: u32,\n    pub tcpi_snd_wnd: u32,\n    pub tcpi_rcv_wnd: u32,\n    // and more, see include/linux/tcp.h\n}\n\nimpl TCP_INFO {\n    /// Create a new zeroed out [`TCP_INFO`]\n    pub unsafe fn new() -> Self {\n        mem::zeroed()\n    }\n\n    /// Return the size of [`TCP_INFO`]\n    #[cfg(unix)]\n    pub fn len() -> socklen_t {\n        mem::size_of::<Self>() as socklen_t\n    }\n\n    /// Return the size of [`TCP_INFO`]\n    #[cfg(windows)]\n    pub fn len() -> usize {\n        mem::size_of::<Self>()\n    }\n}\n\n#[cfg(target_os = \"linux\")]\nfn set_opt<T: Copy>(sock: c_int, opt: c_int, val: c_int, payload: T) -> io::Result<()> {\n    unsafe {\n        let payload = &payload as *const T as *const c_void;\n        cvt_linux_error(libc::setsockopt(\n            sock,\n            opt,\n            val,\n            payload as *const _,\n            mem::size_of::<T>() as socklen_t,\n        ))?;\n        Ok(())\n    }\n}\n\n#[cfg(target_os = \"linux\")]\nfn get_opt<T>(\n    sock: c_int,\n    opt: c_int,\n    val: c_int,\n    payload: &mut T,\n    size: &mut socklen_t,\n) -> io::Result<()> {\n    unsafe {\n        let payload = payload as *mut T as *mut c_void;\n        cvt_linux_error(libc::getsockopt(sock, opt, val, payload as *mut _, size))?;\n        Ok(())\n    }\n}\n\n#[cfg(target_os = \"linux\")]\nfn get_opt_sized<T>(sock: c_int, opt: c_int, val: c_int) -> io::Result<T> {\n    let mut payload = mem::MaybeUninit::zeroed();\n    let expected_size = mem::size_of::<T>() as socklen_t;\n    let mut size = expected_size;\n    get_opt(sock, opt, val, &mut payload, &mut size)?;\n\n    if size != expected_size {\n        return Err(std::io::Error::other(\"get_opt size mismatch\"));\n    }\n    // Assume getsockopt() will set the value properly\n    let payload = unsafe { payload.assume_init() };\n    Ok(payload)\n}\n\n#[cfg(target_os = \"linux\")]\nfn cvt_linux_error(t: i32) -> io::Result<i32> {\n    if t == -1 {\n        Err(io::Error::last_os_error())\n    } else {\n        Ok(t)\n    }\n}\n\n#[cfg(target_os = \"linux\")]\nfn ip_bind_addr_no_port(fd: RawFd, val: bool) -> io::Result<()> {\n    set_opt(\n        fd,\n        libc::IPPROTO_IP,\n        libc::IP_BIND_ADDRESS_NO_PORT,\n        val as c_int,\n    )\n}\n\n#[cfg(all(unix, not(target_os = \"linux\")))]\nfn ip_bind_addr_no_port(_fd: RawFd, _val: bool) -> io::Result<()> {\n    Ok(())\n}\n\n/// IP_LOCAL_PORT_RANGE is only supported on Linux 6.3 and higher,\n/// ip_local_port_range() is a no-op on unsupported versions.\n/// See the [man page](https://man7.org/linux/man-pages/man7/ip.7.html) for more details.\n#[cfg(target_os = \"linux\")]\nfn ip_local_port_range(fd: RawFd, low: u16, high: u16) -> io::Result<()> {\n    const IP_LOCAL_PORT_RANGE: i32 = 51;\n    let range: u32 = (low as u32) | ((high as u32) << 16);\n\n    let result = set_opt(fd, libc::IPPROTO_IP, IP_LOCAL_PORT_RANGE, range as c_int);\n    match result {\n        Err(e) if e.raw_os_error() != Some(libc::ENOPROTOOPT) => Err(e),\n        _ => Ok(()), // no error or ENOPROTOOPT\n    }\n}\n\n#[cfg(all(unix, not(target_os = \"linux\")))]\nfn ip_local_port_range(_fd: RawFd, _low: u16, _high: u16) -> io::Result<()> {\n    Ok(())\n}\n\n#[cfg(windows)]\nfn ip_local_port_range(_fd: RawSocket, _low: u16, _high: u16) -> io::Result<()> {\n    Ok(())\n}\n\n#[cfg(target_os = \"linux\")]\nfn set_so_keepalive(fd: RawFd, val: bool) -> io::Result<()> {\n    set_opt(fd, libc::SOL_SOCKET, libc::SO_KEEPALIVE, val as c_int)\n}\n\n#[cfg(target_os = \"linux\")]\nfn set_so_keepalive_idle(fd: RawFd, val: Duration) -> io::Result<()> {\n    set_opt(\n        fd,\n        libc::IPPROTO_TCP,\n        libc::TCP_KEEPIDLE,\n        val.as_secs() as c_int, // only the seconds part of val is used\n    )\n}\n\n#[cfg(target_os = \"linux\")]\nfn set_so_keepalive_user_timeout(fd: RawFd, val: Duration) -> io::Result<()> {\n    set_opt(\n        fd,\n        libc::IPPROTO_TCP,\n        libc::TCP_USER_TIMEOUT,\n        val.as_millis() as c_int, // only the ms part of val is used\n    )\n}\n\n#[cfg(target_os = \"linux\")]\nfn set_so_keepalive_interval(fd: RawFd, val: Duration) -> io::Result<()> {\n    set_opt(\n        fd,\n        libc::IPPROTO_TCP,\n        libc::TCP_KEEPINTVL,\n        val.as_secs() as c_int, // only the seconds part of val is used\n    )\n}\n\n#[cfg(target_os = \"linux\")]\nfn set_so_keepalive_count(fd: RawFd, val: usize) -> io::Result<()> {\n    set_opt(fd, libc::IPPROTO_TCP, libc::TCP_KEEPCNT, val as c_int)\n}\n\n#[cfg(target_os = \"linux\")]\nfn set_keepalive(fd: RawFd, ka: &TcpKeepalive) -> io::Result<()> {\n    set_so_keepalive(fd, true)?;\n    set_so_keepalive_idle(fd, ka.idle)?;\n    set_so_keepalive_interval(fd, ka.interval)?;\n    set_so_keepalive_count(fd, ka.count)?;\n    set_so_keepalive_user_timeout(fd, ka.user_timeout)\n}\n\n#[cfg(all(unix, not(target_os = \"linux\")))]\nfn set_keepalive(_fd: RawFd, _ka: &TcpKeepalive) -> io::Result<()> {\n    Ok(())\n}\n\n#[cfg(windows)]\nfn set_keepalive(_sock: RawSocket, _ka: &TcpKeepalive) -> io::Result<()> {\n    Ok(())\n}\n\n/// Get the kernel TCP_INFO for the given FD.\n#[cfg(target_os = \"linux\")]\npub fn get_tcp_info(fd: RawFd) -> io::Result<TCP_INFO> {\n    get_opt_sized(fd, libc::IPPROTO_TCP, libc::TCP_INFO)\n}\n\n#[cfg(all(unix, not(target_os = \"linux\")))]\npub fn get_tcp_info(_fd: RawFd) -> io::Result<TCP_INFO> {\n    Ok(unsafe { TCP_INFO::new() })\n}\n\n#[cfg(windows)]\npub fn get_tcp_info(_fd: RawSocket) -> io::Result<TCP_INFO> {\n    Ok(unsafe { TCP_INFO::new() })\n}\n\n/// Set the TCP receive buffer size. See SO_RCVBUF.\n#[cfg(target_os = \"linux\")]\npub fn set_recv_buf(fd: RawFd, val: usize) -> Result<()> {\n    set_opt(fd, libc::SOL_SOCKET, libc::SO_RCVBUF, val as c_int)\n        .or_err(ConnectError, \"failed to set SO_RCVBUF\")\n}\n\n#[cfg(all(unix, not(target_os = \"linux\")))]\npub fn set_recv_buf(_fd: RawFd, _: usize) -> Result<()> {\n    Ok(())\n}\n\n#[cfg(windows)]\npub fn set_recv_buf(_sock: RawSocket, _: usize) -> Result<()> {\n    Ok(())\n}\n\n/// Set the TCP send buffer size. See SO_SNDBUF.\n#[cfg(target_os = \"linux\")]\npub fn set_snd_buf(fd: RawFd, val: usize) -> Result<()> {\n    set_opt(fd, libc::SOL_SOCKET, libc::SO_SNDBUF, val as c_int)\n        .or_err(ConnectError, \"failed to set SO_SNDBUF\")\n}\n\n#[cfg(all(unix, not(target_os = \"linux\")))]\npub fn set_snd_buf(_fd: RawFd, _: usize) -> Result<()> {\n    Ok(())\n}\n\n#[cfg(windows)]\npub fn set_snd_buf(_sock: RawSocket, _: usize) -> Result<()> {\n    Ok(())\n}\n\n#[cfg(target_os = \"linux\")]\npub fn get_recv_buf(fd: RawFd) -> io::Result<usize> {\n    get_opt_sized::<c_int>(fd, libc::SOL_SOCKET, libc::SO_RCVBUF).map(|v| v as usize)\n}\n\n#[cfg(all(unix, not(target_os = \"linux\")))]\npub fn get_recv_buf(_fd: RawFd) -> io::Result<usize> {\n    Ok(0)\n}\n\n#[cfg(windows)]\npub fn get_recv_buf(_sock: RawSocket) -> io::Result<usize> {\n    Ok(0)\n}\n\n#[cfg(target_os = \"linux\")]\npub fn get_snd_buf(fd: RawFd) -> io::Result<usize> {\n    get_opt_sized::<c_int>(fd, libc::SOL_SOCKET, libc::SO_SNDBUF).map(|v| v as usize)\n}\n\n#[cfg(all(unix, not(target_os = \"linux\")))]\npub fn get_snd_buf(_fd: RawFd) -> io::Result<usize> {\n    Ok(0)\n}\n\n#[cfg(windows)]\npub fn get_snd_buf(_sock: RawSocket) -> io::Result<usize> {\n    Ok(0)\n}\n\n/// Enable client side TCP fast open.\n#[cfg(target_os = \"linux\")]\npub fn set_tcp_fastopen_connect(fd: RawFd) -> Result<()> {\n    set_opt(\n        fd,\n        libc::IPPROTO_TCP,\n        libc::TCP_FASTOPEN_CONNECT,\n        1 as c_int,\n    )\n    .or_err(ConnectError, \"failed to set TCP_FASTOPEN_CONNECT\")\n}\n\n#[cfg(all(unix, not(target_os = \"linux\")))]\npub fn set_tcp_fastopen_connect(_fd: RawFd) -> Result<()> {\n    Ok(())\n}\n\n#[cfg(windows)]\npub fn set_tcp_fastopen_connect(_sock: RawSocket) -> Result<()> {\n    Ok(())\n}\n\n/// Enable server side TCP fast open.\n#[cfg(target_os = \"linux\")]\npub fn set_tcp_fastopen_backlog(fd: RawFd, backlog: usize) -> Result<()> {\n    set_opt(fd, libc::IPPROTO_TCP, libc::TCP_FASTOPEN, backlog as c_int)\n        .or_err(ConnectError, \"failed to set TCP_FASTOPEN\")\n}\n\n#[cfg(all(unix, not(target_os = \"linux\")))]\npub fn set_tcp_fastopen_backlog(_fd: RawFd, _backlog: usize) -> Result<()> {\n    Ok(())\n}\n\n#[cfg(windows)]\npub fn set_tcp_fastopen_backlog(_sock: RawSocket, _backlog: usize) -> Result<()> {\n    Ok(())\n}\n\n#[cfg(target_os = \"linux\")]\npub fn set_dscp(fd: RawFd, value: u8) -> Result<()> {\n    use super::socket::SocketAddr;\n    use pingora_error::OkOrErr;\n\n    let sock = SocketAddr::from_raw_fd(fd, false);\n    let addr = sock\n        .as_ref()\n        .and_then(|s| s.as_inet())\n        .or_err(SocketError, \"failed to set dscp, invalid IP socket\")?;\n\n    if addr.is_ipv6() {\n        set_opt(fd, libc::IPPROTO_IPV6, libc::IPV6_TCLASS, value as c_int)\n            .or_err(SocketError, \"failed to set dscp (IPV6_TCLASS)\")\n    } else {\n        set_opt(fd, libc::IPPROTO_IP, libc::IP_TOS, value as c_int)\n            .or_err(SocketError, \"failed to set dscp (IP_TOS)\")\n    }\n}\n\n#[cfg(all(unix, not(target_os = \"linux\")))]\npub fn set_dscp(_fd: RawFd, _value: u8) -> Result<()> {\n    Ok(())\n}\n\n#[cfg(windows)]\npub fn set_dscp(_sock: RawSocket, _value: u8) -> Result<()> {\n    Ok(())\n}\n\n#[cfg(target_os = \"linux\")]\npub fn get_socket_cookie(fd: RawFd) -> io::Result<u64> {\n    get_opt_sized::<c_ulonglong>(fd, libc::SOL_SOCKET, libc::SO_COOKIE)\n}\n\n#[cfg(all(unix, not(target_os = \"linux\")))]\npub fn get_socket_cookie(_fd: RawFd) -> io::Result<u64> {\n    Ok(0) // SO_COOKIE is a Linux concept\n}\n\n#[cfg(target_os = \"linux\")]\npub fn get_original_dest(fd: RawFd) -> Result<Option<SocketAddr>> {\n    use super::socket;\n    use pingora_error::OkOrErr;\n    use std::net::{SocketAddrV4, SocketAddrV6};\n\n    let sock = socket::SocketAddr::from_raw_fd(fd, false);\n    let addr = sock\n        .as_ref()\n        .and_then(|s| s.as_inet())\n        .or_err(SocketError, \"failed get original dest, invalid IP socket\")?;\n\n    let dest = if addr.is_ipv4() {\n        get_opt_sized::<libc::sockaddr_in>(fd, libc::SOL_IP, libc::SO_ORIGINAL_DST).map(|addr| {\n            SocketAddr::V4(SocketAddrV4::new(\n                u32::from_be(addr.sin_addr.s_addr).into(),\n                u16::from_be(addr.sin_port),\n            ))\n        })\n    } else {\n        get_opt_sized::<libc::sockaddr_in6>(fd, libc::SOL_IPV6, libc::IP6T_SO_ORIGINAL_DST).map(\n            |addr| {\n                SocketAddr::V6(SocketAddrV6::new(\n                    addr.sin6_addr.s6_addr.into(),\n                    u16::from_be(addr.sin6_port),\n                    addr.sin6_flowinfo,\n                    addr.sin6_scope_id,\n                ))\n            },\n        )\n    };\n    dest.or_err(SocketError, \"failed to get original dest\")\n        .map(Some)\n}\n\n#[cfg(all(unix, not(target_os = \"linux\")))]\npub fn get_original_dest(_fd: RawFd) -> Result<Option<SocketAddr>> {\n    Ok(None)\n}\n\n#[cfg(windows)]\npub fn get_original_dest(_sock: RawSocket) -> Result<Option<SocketAddr>> {\n    Ok(None)\n}\n\n/// connect() to the given address while optionally binding to the specific source address and port range.\n///\n/// The `set_socket` callback can be used to tune the socket before `connect()` is called.\n///\n/// If a [`BindTo`] is set with a port range and fallback setting enabled this function will retry\n/// on EADDRNOTAVAIL ignoring the port range.\n///\n/// `IP_BIND_ADDRESS_NO_PORT` is used.\n/// `IP_LOCAL_PORT_RANGE` is used if a port range is set on [`BindTo`].\npub(crate) async fn connect_with<F: FnOnce(&TcpSocket) -> Result<()> + Clone>(\n    addr: &SocketAddr,\n    bind_to: Option<&BindTo>,\n    set_socket: F,\n) -> Result<TcpStream> {\n    if bind_to.as_ref().is_some_and(|b| b.will_fallback()) {\n        // if we see an EADDRNOTAVAIL error clear the port range and try again\n        let connect_result = inner_connect_with(addr, bind_to, set_socket.clone()).await;\n        if let Err(e) = connect_result.as_ref() {\n            if matches!(e.etype(), BindError) {\n                let mut new_bind_to = BindTo::default();\n                new_bind_to.addr = bind_to.as_ref().and_then(|b| b.addr);\n                // reset the port range\n                new_bind_to.set_port_range(None).unwrap();\n                return inner_connect_with(addr, Some(&new_bind_to), set_socket).await;\n            }\n        }\n        connect_result\n    } else {\n        // not retryable\n        inner_connect_with(addr, bind_to, set_socket).await\n    }\n}\n\nasync fn inner_connect_with<F: FnOnce(&TcpSocket) -> Result<()>>(\n    addr: &SocketAddr,\n    bind_to: Option<&BindTo>,\n    set_socket: F,\n) -> Result<TcpStream> {\n    let socket = if addr.is_ipv4() {\n        TcpSocket::new_v4()\n    } else {\n        TcpSocket::new_v6()\n    }\n    .or_err(SocketError, \"failed to create socket\")?;\n\n    #[cfg(unix)]\n    {\n        ip_bind_addr_no_port(socket.as_raw_fd(), true).or_err(\n            SocketError,\n            \"failed to set socket opts IP_BIND_ADDRESS_NO_PORT\",\n        )?;\n\n        if let Some(bind_to) = bind_to {\n            if let Some((low, high)) = bind_to.port_range() {\n                ip_local_port_range(socket.as_raw_fd(), low, high)\n                    .or_err(SocketError, \"failed to set socket opts IP_LOCAL_PORT_RANGE\")?;\n            }\n\n            if let Some(baddr) = bind_to.addr {\n                socket\n                    .bind(baddr)\n                    .or_err_with(BindError, || format!(\"failed to bind to socket {}\", baddr))?;\n            }\n        }\n    }\n\n    #[cfg(windows)]\n    if let Some(bind_to) = bind_to {\n        if let Some(baddr) = bind_to.addr {\n            socket\n                .bind(baddr)\n                .or_err_with(BindError, || format!(\"failed to bind to socket {}\", baddr))?;\n        };\n    };\n    // TODO: add support for bind on other platforms\n\n    set_socket(&socket)?;\n\n    socket\n        .connect(*addr)\n        .await\n        .map_err(|e| wrap_os_connect_error(e, format!(\"Fail to connect to {}\", *addr)))\n}\n\n/// connect() to the given address while optionally binding to the specific source address.\n///\n/// `IP_BIND_ADDRESS_NO_PORT` is used\n/// `IP_LOCAL_PORT_RANGE` is used if a port range is set on [`BindTo`].\npub async fn connect(addr: &SocketAddr, bind_to: Option<&BindTo>) -> Result<TcpStream> {\n    connect_with(addr, bind_to, |_| Ok(())).await\n}\n\n/// connect() to the given Unix domain socket\n#[cfg(unix)]\npub async fn connect_uds(path: &std::path::Path) -> Result<UnixStream> {\n    UnixStream::connect(path)\n        .await\n        .map_err(|e| wrap_os_connect_error(e, format!(\"Fail to connect to {}\", path.display())))\n}\n\nfn wrap_os_connect_error(e: std::io::Error, context: String) -> Box<Error> {\n    match e.kind() {\n        ErrorKind::ConnectionRefused => Error::because(ConnectRefused, context, e),\n        ErrorKind::TimedOut => Error::because(ConnectTimedout, context, e),\n        ErrorKind::AddrNotAvailable => Error::because(BindError, context, e),\n        ErrorKind::PermissionDenied | ErrorKind::AddrInUse => {\n            Error::because(InternalError, context, e)\n        }\n        _ => match e.raw_os_error() {\n            Some(libc::ENETUNREACH | libc::EHOSTUNREACH) => {\n                Error::because(ConnectNoRoute, context, e)\n            }\n            _ => Error::because(ConnectError, context, e),\n        },\n    }\n}\n\n/// The configuration for TCP keepalive\n#[derive(Clone, Debug)]\npub struct TcpKeepalive {\n    /// The time a connection needs to be idle before TCP begins sending out keep-alive probes.\n    pub idle: Duration,\n    /// The number of seconds between TCP keep-alive probes.\n    pub interval: Duration,\n    /// The maximum number of TCP keep-alive probes to send before giving up and killing the connection\n    pub count: usize,\n    /// the maximum amount of time in milliseconds that transmitted data may\n    /// remain unacknowledged, or buffered data may remain untransmitted (due to\n    /// zero window size) before TCP will forcibly close the corresponding\n    /// connection and return ETIMEDOUT. If the value is specified as 0 (the\n    /// default), TCP will use the system default.\n    #[cfg(target_os = \"linux\")]\n    pub user_timeout: Duration,\n}\n\nimpl std::fmt::Display for TcpKeepalive {\n    #[cfg(target_os = \"linux\")]\n    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {\n        write!(\n            f,\n            \"{:?}/{:?}/{}/{:?}\",\n            self.idle, self.interval, self.count, self.user_timeout\n        )\n    }\n    #[cfg(not(target_os = \"linux\"))]\n    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {\n        write!(f, \"{:?}/{:?}/{}\", self.idle, self.interval, self.count)\n    }\n}\n\n/// Apply the given TCP keepalive settings to the given connection\npub fn set_tcp_keepalive(stream: &TcpStream, ka: &TcpKeepalive) -> Result<()> {\n    #[cfg(unix)]\n    let raw = stream.as_raw_fd();\n    #[cfg(windows)]\n    let raw = stream.as_raw_socket();\n    // TODO: check localhost or if keepalive is already set\n    set_keepalive(raw, ka).or_err(ConnectError, \"failed to set keepalive\")\n}\n\n#[cfg(test)]\nmod test {\n    use super::*;\n\n    #[test]\n    fn test_set_recv_buf() {\n        use tokio::net::TcpSocket;\n        let socket = TcpSocket::new_v4().unwrap();\n        #[cfg(unix)]\n        set_recv_buf(socket.as_raw_fd(), 102400).unwrap();\n        #[cfg(windows)]\n        set_recv_buf(socket.as_raw_socket(), 102400).unwrap();\n\n        #[cfg(target_os = \"linux\")]\n        {\n            // kernel doubles whatever is set\n            assert_eq!(get_recv_buf(socket.as_raw_fd()).unwrap(), 102400 * 2);\n        }\n    }\n\n    #[cfg(target_os = \"linux\")]\n    #[ignore] // this test requires the Linux system to have net.ipv4.tcp_fastopen set\n    #[tokio::test]\n    async fn test_set_fast_open() {\n        use std::time::Instant;\n\n        // connect once to make sure their is a SYN cookie to use for TFO\n        connect_with(&\"1.1.1.1:80\".parse().unwrap(), None, |socket| {\n            set_tcp_fastopen_connect(socket.as_raw_fd())\n        })\n        .await\n        .unwrap();\n\n        let start = Instant::now();\n        connect_with(&\"1.1.1.1:80\".parse().unwrap(), None, |socket| {\n            set_tcp_fastopen_connect(socket.as_raw_fd())\n        })\n        .await\n        .unwrap();\n        let connection_time = start.elapsed();\n\n        // connect() return right away as the SYN goes out only when the first write() is called.\n        assert!(connection_time.as_millis() < 4);\n    }\n}\n"
  },
  {
    "path": "pingora-core/src/protocols/l4/listener.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! Listeners\n\nuse std::io;\n#[cfg(unix)]\nuse std::os::unix::io::AsRawFd;\n#[cfg(windows)]\nuse std::os::windows::io::AsRawSocket;\nuse tokio::net::TcpListener;\n#[cfg(unix)]\nuse tokio::net::UnixListener;\n\nuse crate::protocols::digest::{GetSocketDigest, SocketDigest};\nuse crate::protocols::l4::stream::Stream;\n\n/// The type for generic listener for both TCP and Unix domain socket\n#[derive(Debug)]\npub enum Listener {\n    Tcp(TcpListener),\n    #[cfg(unix)]\n    Unix(UnixListener),\n}\n\nimpl From<TcpListener> for Listener {\n    fn from(s: TcpListener) -> Self {\n        Self::Tcp(s)\n    }\n}\n\n#[cfg(unix)]\nimpl From<UnixListener> for Listener {\n    fn from(s: UnixListener) -> Self {\n        Self::Unix(s)\n    }\n}\n\n#[cfg(unix)]\nimpl AsRawFd for Listener {\n    fn as_raw_fd(&self) -> std::os::unix::io::RawFd {\n        match &self {\n            Self::Tcp(l) => l.as_raw_fd(),\n            Self::Unix(l) => l.as_raw_fd(),\n        }\n    }\n}\n\n#[cfg(windows)]\nimpl AsRawSocket for Listener {\n    fn as_raw_socket(&self) -> std::os::windows::io::RawSocket {\n        match &self {\n            Self::Tcp(l) => l.as_raw_socket(),\n        }\n    }\n}\n\nimpl Listener {\n    /// Accept a connection from the listening endpoint\n    pub async fn accept(&self) -> io::Result<Stream> {\n        match &self {\n            Self::Tcp(l) => l.accept().await.map(|(stream, peer_addr)| {\n                let mut s: Stream = stream.into();\n                #[cfg(unix)]\n                let digest = SocketDigest::from_raw_fd(s.as_raw_fd());\n                #[cfg(windows)]\n                let digest = SocketDigest::from_raw_socket(s.as_raw_socket());\n                digest\n                    .peer_addr\n                    .set(Some(peer_addr.into()))\n                    .expect(\"newly created OnceCell must be empty\");\n                s.set_socket_digest(digest);\n                // TODO: if listening on a specific bind address, we could save\n                // an extra syscall looking up the local_addr later if we can pass\n                // and init it in the socket digest here\n                s\n            }),\n            #[cfg(unix)]\n            Self::Unix(l) => l.accept().await.map(|(stream, peer_addr)| {\n                let mut s: Stream = stream.into();\n                let digest = SocketDigest::from_raw_fd(s.as_raw_fd());\n                // note: if unnamed/abstract UDS, it will be `None`\n                // (see TryFrom<tokio::net::unix::SocketAddr>)\n                let addr = peer_addr.try_into().ok();\n                digest\n                    .peer_addr\n                    .set(addr)\n                    .expect(\"newly created OnceCell must be empty\");\n                s.set_socket_digest(digest);\n                s\n            }),\n        }\n    }\n}\n"
  },
  {
    "path": "pingora-core/src/protocols/l4/mod.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! Transport layer protocol implementation\n\npub mod ext;\npub mod listener;\npub mod socket;\npub mod stream;\npub mod virt;\n"
  },
  {
    "path": "pingora-core/src/protocols/l4/socket.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! Generic socket type\n\nuse crate::{Error, OrErr};\nuse log::warn;\n#[cfg(unix)]\nuse nix::sys::socket::{getpeername, getsockname, SockaddrStorage};\nuse std::cmp::Ordering;\nuse std::hash::{Hash, Hasher};\nuse std::net::SocketAddr as StdSockAddr;\n#[cfg(unix)]\nuse std::os::unix::net::SocketAddr as StdUnixSockAddr;\n#[cfg(unix)]\nuse tokio::net::unix::SocketAddr as TokioUnixSockAddr;\n\n/// [`SocketAddr`] is a storage type that contains either an Internet (IP address)\n/// socket address or a Unix domain socket address.\n#[derive(Debug, Clone)]\npub enum SocketAddr {\n    Inet(StdSockAddr),\n    #[cfg(unix)]\n    Unix(StdUnixSockAddr),\n}\n\nimpl SocketAddr {\n    /// Get a reference to the IP socket if it is one\n    pub fn as_inet(&self) -> Option<&StdSockAddr> {\n        if let SocketAddr::Inet(addr) = self {\n            Some(addr)\n        } else {\n            None\n        }\n    }\n\n    /// Get a reference to the Unix domain socket if it is one\n    #[cfg(unix)]\n    pub fn as_unix(&self) -> Option<&StdUnixSockAddr> {\n        if let SocketAddr::Unix(addr) = self {\n            Some(addr)\n        } else {\n            None\n        }\n    }\n\n    /// Set the port if the address is an IP socket.\n    pub fn set_port(&mut self, port: u16) {\n        if let SocketAddr::Inet(addr) = self {\n            addr.set_port(port)\n        }\n    }\n\n    #[cfg(unix)]\n    fn from_sockaddr_storage(sock: &SockaddrStorage) -> Option<SocketAddr> {\n        if let Some(v4) = sock.as_sockaddr_in() {\n            return Some(SocketAddr::Inet(StdSockAddr::V4(\n                std::net::SocketAddrV4::new(v4.ip().into(), v4.port()),\n            )));\n        } else if let Some(v6) = sock.as_sockaddr_in6() {\n            return Some(SocketAddr::Inet(StdSockAddr::V6(\n                std::net::SocketAddrV6::new(v6.ip(), v6.port(), v6.flowinfo(), v6.scope_id()),\n            )));\n        }\n\n        // TODO: don't set abstract / unnamed for now,\n        // for parity with how we treat these types in TryFrom<TokioUnixSockAddr>\n        Some(SocketAddr::Unix(\n            sock.as_unix_addr()\n                .map(|addr| addr.path().map(StdUnixSockAddr::from_pathname))??\n                .ok()?,\n        ))\n    }\n\n    #[cfg(unix)]\n    pub fn from_raw_fd(fd: std::os::unix::io::RawFd, peer_addr: bool) -> Option<SocketAddr> {\n        let sockaddr_storage = if peer_addr {\n            getpeername(fd)\n        } else {\n            getsockname(fd)\n        };\n        match sockaddr_storage {\n            Ok(sockaddr) => Self::from_sockaddr_storage(&sockaddr),\n            // could be errors such as EBADF, i.e. fd is no longer a valid socket\n            // fail open in this case\n            Err(_e) => None,\n        }\n    }\n\n    #[cfg(windows)]\n    pub fn from_raw_socket(\n        sock: std::os::windows::io::RawSocket,\n        is_peer_addr: bool,\n    ) -> Option<SocketAddr> {\n        use crate::protocols::windows::{local_addr, peer_addr};\n        if is_peer_addr {\n            peer_addr(sock)\n        } else {\n            local_addr(sock)\n        }\n        .map(|s| s.into())\n        .ok()\n    }\n}\n\nimpl std::fmt::Display for SocketAddr {\n    fn fmt(&self, f: &mut std::fmt::Formatter) -> std::fmt::Result {\n        match self {\n            SocketAddr::Inet(addr) => write!(f, \"{addr}\"),\n            #[cfg(unix)]\n            SocketAddr::Unix(addr) => {\n                if let Some(path) = addr.as_pathname() {\n                    write!(f, \"{}\", path.display())\n                } else {\n                    write!(f, \"{addr:?}\")\n                }\n            }\n        }\n    }\n}\n\nimpl Hash for SocketAddr {\n    fn hash<H: Hasher>(&self, state: &mut H) {\n        match self {\n            Self::Inet(sockaddr) => sockaddr.hash(state),\n            #[cfg(unix)]\n            Self::Unix(sockaddr) => {\n                if let Some(path) = sockaddr.as_pathname() {\n                    // use the underlying path as the hash\n                    path.hash(state);\n                } else {\n                    // unnamed or abstract UDS\n                    // abstract UDS name not yet exposed by std API\n                    // panic for now, we can decide on the right way to hash them later\n                    panic!(\"Unnamed and abstract UDS types not yet supported for hashing\")\n                }\n            }\n        }\n    }\n}\n\nimpl PartialEq for SocketAddr {\n    fn eq(&self, other: &Self) -> bool {\n        match self {\n            Self::Inet(addr) => Some(addr) == other.as_inet(),\n            #[cfg(unix)]\n            Self::Unix(addr) => {\n                let path = addr.as_pathname();\n                // can only compare UDS with path, assume false on all unnamed UDS\n                path.is_some() && path == other.as_unix().and_then(|addr| addr.as_pathname())\n            }\n        }\n    }\n}\n\nimpl PartialOrd for SocketAddr {\n    fn partial_cmp(&self, other: &Self) -> Option<Ordering> {\n        Some(self.cmp(other))\n    }\n}\n\nimpl Ord for SocketAddr {\n    fn cmp(&self, other: &Self) -> Ordering {\n        match self {\n            Self::Inet(addr) => {\n                if let Some(o) = other.as_inet() {\n                    addr.cmp(o)\n                } else {\n                    // always make Inet < Unix \"smallest for variants at the top\"\n                    Ordering::Less\n                }\n            }\n            #[cfg(unix)]\n            Self::Unix(addr) => {\n                if let Some(o) = other.as_unix() {\n                    // NOTE: unnamed UDS are consider the same\n                    addr.as_pathname().cmp(&o.as_pathname())\n                } else {\n                    // always make Inet < Unix \"smallest for variants at the top\"\n                    Ordering::Greater\n                }\n            }\n        }\n    }\n}\n\nimpl Eq for SocketAddr {}\n\nimpl std::str::FromStr for SocketAddr {\n    type Err = Box<Error>;\n\n    // This is very basic parsing logic, it might treat invalid IP:PORT str as UDS path\n    #[cfg(unix)]\n    fn from_str(s: &str) -> Result<Self, Self::Err> {\n        if s.starts_with(\"unix:\") {\n            // format unix:/tmp/server.socket\n            let path = s.trim_start_matches(\"unix:\");\n            let uds_socket = StdUnixSockAddr::from_pathname(path)\n                .or_err(crate::BindError, \"invalid UDS path\")?;\n            Ok(SocketAddr::Unix(uds_socket))\n        } else {\n            match StdSockAddr::from_str(s) {\n                Ok(addr) => Ok(SocketAddr::Inet(addr)),\n                Err(_) => {\n                    // Try to parse as UDS for backward compatibility\n                    let uds_socket = StdUnixSockAddr::from_pathname(s)\n                        .or_err(crate::BindError, \"invalid UDS path\")?;\n                    warn!(\"Raw Unix domain socket path support will be deprecated, add 'unix:' prefix instead\");\n                    Ok(SocketAddr::Unix(uds_socket))\n                }\n            }\n        }\n    }\n\n    #[cfg(windows)]\n    fn from_str(s: &str) -> Result<Self, Self::Err> {\n        let addr = StdSockAddr::from_str(s).or_err(crate::BindError, \"invalid socket addr\")?;\n        Ok(SocketAddr::Inet(addr))\n    }\n}\n\nimpl std::net::ToSocketAddrs for SocketAddr {\n    type Iter = std::iter::Once<StdSockAddr>;\n\n    // Error if UDS addr\n    fn to_socket_addrs(&self) -> std::io::Result<Self::Iter> {\n        if let Some(inet) = self.as_inet() {\n            Ok(std::iter::once(*inet))\n        } else {\n            Err(std::io::Error::other(\n                \"UDS socket cannot be used as inet socket\",\n            ))\n        }\n    }\n}\n\nimpl From<StdSockAddr> for SocketAddr {\n    fn from(sockaddr: StdSockAddr) -> Self {\n        SocketAddr::Inet(sockaddr)\n    }\n}\n\n#[cfg(unix)]\nimpl From<StdUnixSockAddr> for SocketAddr {\n    fn from(sockaddr: StdUnixSockAddr) -> Self {\n        SocketAddr::Unix(sockaddr)\n    }\n}\n\n// TODO: ideally mio/tokio will start using the std version of the unix `SocketAddr`\n// so we can avoid a fallible conversion\n// https://github.com/tokio-rs/mio/issues/1527\n#[cfg(unix)]\nimpl TryFrom<TokioUnixSockAddr> for SocketAddr {\n    type Error = String;\n\n    fn try_from(value: TokioUnixSockAddr) -> Result<Self, Self::Error> {\n        if let Some(Ok(addr)) = value.as_pathname().map(StdUnixSockAddr::from_pathname) {\n            Ok(addr.into())\n        } else {\n            // may be unnamed/abstract UDS\n            Err(format!(\"could not convert {value:?} to SocketAddr\"))\n        }\n    }\n}\n\n#[cfg(test)]\nmod test {\n    use super::*;\n\n    #[test]\n    fn parse_ip() {\n        let ip: SocketAddr = \"127.0.0.1:80\".parse().unwrap();\n        assert!(ip.as_inet().is_some());\n    }\n\n    #[cfg(unix)]\n    #[test]\n    fn parse_uds() {\n        let uds: SocketAddr = \"/tmp/my.sock\".parse().unwrap();\n        assert!(uds.as_unix().is_some());\n    }\n\n    #[cfg(unix)]\n    #[test]\n    fn parse_uds_with_prefix() {\n        let uds: SocketAddr = \"unix:/tmp/my.sock\".parse().unwrap();\n        assert!(uds.as_unix().is_some());\n    }\n}\n"
  },
  {
    "path": "pingora-core/src/protocols/l4/stream.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! Transport layer connection\n\nuse async_trait::async_trait;\nuse futures::FutureExt;\nuse log::{debug, error};\n\nuse pingora_error::{ErrorType::*, OrErr, Result};\n#[cfg(target_os = \"linux\")]\nuse std::io::IoSliceMut;\n#[cfg(unix)]\nuse std::os::unix::io::AsRawFd;\n#[cfg(windows)]\nuse std::os::windows::io::AsRawSocket;\nuse std::pin::Pin;\nuse std::sync::Arc;\nuse std::task::{Context, Poll};\nuse std::time::{Duration, Instant, SystemTime};\n#[cfg(target_os = \"linux\")]\nuse tokio::io::Interest;\nuse tokio::io::{self, AsyncRead, AsyncWrite, AsyncWriteExt, BufStream, ReadBuf};\nuse tokio::net::TcpStream;\n#[cfg(unix)]\nuse tokio::net::UnixStream;\n\nuse crate::protocols::l4::ext::{set_tcp_keepalive, TcpKeepalive};\nuse crate::protocols::l4::virt;\nuse crate::protocols::raw_connect::ProxyDigest;\nuse crate::protocols::{\n    GetProxyDigest, GetSocketDigest, GetTimingDigest, Peek, Shutdown, SocketDigest, Ssl,\n    TimingDigest, UniqueID, UniqueIDType,\n};\nuse crate::upstreams::peer::Tracer;\n\n#[derive(Debug)]\nenum RawStream {\n    Tcp(TcpStream),\n    #[cfg(unix)]\n    Unix(UnixStream),\n    Virtual(virt::VirtualSocketStream),\n}\n\nimpl AsyncRead for RawStream {\n    fn poll_read(\n        self: Pin<&mut Self>,\n        cx: &mut Context<'_>,\n        buf: &mut ReadBuf<'_>,\n    ) -> Poll<io::Result<()>> {\n        // Safety: Basic enum pin projection\n        unsafe {\n            match &mut Pin::get_unchecked_mut(self) {\n                RawStream::Tcp(s) => Pin::new_unchecked(s).poll_read(cx, buf),\n                #[cfg(unix)]\n                RawStream::Unix(s) => Pin::new_unchecked(s).poll_read(cx, buf),\n                RawStream::Virtual(s) => Pin::new_unchecked(s).poll_read(cx, buf),\n            }\n        }\n    }\n}\n\nimpl AsyncWrite for RawStream {\n    fn poll_write(self: Pin<&mut Self>, cx: &mut Context, buf: &[u8]) -> Poll<io::Result<usize>> {\n        // Safety: Basic enum pin projection\n        unsafe {\n            match &mut Pin::get_unchecked_mut(self) {\n                RawStream::Tcp(s) => Pin::new_unchecked(s).poll_write(cx, buf),\n                #[cfg(unix)]\n                RawStream::Unix(s) => Pin::new_unchecked(s).poll_write(cx, buf),\n                RawStream::Virtual(s) => Pin::new_unchecked(s).poll_write(cx, buf),\n            }\n        }\n    }\n\n    fn poll_flush(self: Pin<&mut Self>, cx: &mut Context) -> Poll<io::Result<()>> {\n        // Safety: Basic enum pin projection\n        unsafe {\n            match &mut Pin::get_unchecked_mut(self) {\n                RawStream::Tcp(s) => Pin::new_unchecked(s).poll_flush(cx),\n                #[cfg(unix)]\n                RawStream::Unix(s) => Pin::new_unchecked(s).poll_flush(cx),\n                RawStream::Virtual(s) => Pin::new_unchecked(s).poll_flush(cx),\n            }\n        }\n    }\n\n    fn poll_shutdown(self: Pin<&mut Self>, cx: &mut Context) -> Poll<io::Result<()>> {\n        // Safety: Basic enum pin projection\n        unsafe {\n            match &mut Pin::get_unchecked_mut(self) {\n                RawStream::Tcp(s) => Pin::new_unchecked(s).poll_shutdown(cx),\n                #[cfg(unix)]\n                RawStream::Unix(s) => Pin::new_unchecked(s).poll_shutdown(cx),\n                RawStream::Virtual(s) => Pin::new_unchecked(s).poll_shutdown(cx),\n            }\n        }\n    }\n\n    fn poll_write_vectored(\n        self: Pin<&mut Self>,\n        cx: &mut Context<'_>,\n        bufs: &[std::io::IoSlice<'_>],\n    ) -> Poll<io::Result<usize>> {\n        // Safety: Basic enum pin projection\n        unsafe {\n            match &mut Pin::get_unchecked_mut(self) {\n                RawStream::Tcp(s) => Pin::new_unchecked(s).poll_write_vectored(cx, bufs),\n                #[cfg(unix)]\n                RawStream::Unix(s) => Pin::new_unchecked(s).poll_write_vectored(cx, bufs),\n                RawStream::Virtual(s) => Pin::new_unchecked(s).poll_write_vectored(cx, bufs),\n            }\n        }\n    }\n\n    fn is_write_vectored(&self) -> bool {\n        match self {\n            RawStream::Tcp(s) => s.is_write_vectored(),\n            #[cfg(unix)]\n            RawStream::Unix(s) => s.is_write_vectored(),\n            RawStream::Virtual(s) => s.is_write_vectored(),\n        }\n    }\n}\n\n#[cfg(unix)]\nimpl AsRawFd for RawStream {\n    fn as_raw_fd(&self) -> std::os::unix::io::RawFd {\n        match self {\n            RawStream::Tcp(s) => s.as_raw_fd(),\n            RawStream::Unix(s) => s.as_raw_fd(),\n            RawStream::Virtual(_) => -1, // Virtual stream does not have a real fd\n        }\n    }\n}\n\n#[cfg(windows)]\nimpl AsRawSocket for RawStream {\n    fn as_raw_socket(&self) -> std::os::windows::io::RawSocket {\n        match self {\n            RawStream::Tcp(s) => s.as_raw_socket(),\n            // Virtual stream does not have a real socket, return INVALID_SOCKET (!0)\n            RawStream::Virtual(_) => !0,\n        }\n    }\n}\n\n#[derive(Debug)]\nstruct RawStreamWrapper {\n    pub(crate) stream: RawStream,\n    /// store the last rx timestamp of the stream.\n    pub(crate) rx_ts: Option<SystemTime>,\n    /// enable reading rx timestamp\n    #[cfg(target_os = \"linux\")]\n    pub(crate) enable_rx_ts: bool,\n    #[cfg(target_os = \"linux\")]\n    /// This can be reused across multiple recvmsg calls. The cmsg buffer may\n    /// come from old sockets created by older version of pingora and so,\n    /// this vector can only grow.\n    reusable_cmsg_space: Vec<u8>,\n}\n\nimpl RawStreamWrapper {\n    pub fn new(stream: RawStream) -> Self {\n        RawStreamWrapper {\n            stream,\n            rx_ts: None,\n            #[cfg(target_os = \"linux\")]\n            enable_rx_ts: false,\n            #[cfg(target_os = \"linux\")]\n            reusable_cmsg_space: nix::cmsg_space!(nix::sys::time::TimeSpec),\n        }\n    }\n\n    #[cfg(target_os = \"linux\")]\n    pub fn enable_rx_ts(&mut self, enable_rx_ts: bool) {\n        self.enable_rx_ts = enable_rx_ts;\n    }\n}\n\nimpl AsyncRead for RawStreamWrapper {\n    #[cfg(not(target_os = \"linux\"))]\n    fn poll_read(\n        self: Pin<&mut Self>,\n        cx: &mut Context<'_>,\n        buf: &mut ReadBuf<'_>,\n    ) -> Poll<io::Result<()>> {\n        // Safety: Basic enum pin projection\n        unsafe {\n            let rs_wrapper = Pin::get_unchecked_mut(self);\n            match &mut rs_wrapper.stream {\n                RawStream::Tcp(s) => Pin::new_unchecked(s).poll_read(cx, buf),\n                #[cfg(unix)]\n                RawStream::Unix(s) => Pin::new_unchecked(s).poll_read(cx, buf),\n                RawStream::Virtual(s) => Pin::new_unchecked(s).poll_read(cx, buf),\n            }\n        }\n    }\n\n    #[cfg(target_os = \"linux\")]\n    fn poll_read(\n        self: Pin<&mut Self>,\n        cx: &mut Context<'_>,\n        buf: &mut ReadBuf<'_>,\n    ) -> Poll<io::Result<()>> {\n        use futures::ready;\n        use nix::sys::socket::{recvmsg, ControlMessageOwned, MsgFlags, SockaddrStorage};\n\n        // if we do not need rx timestamp, then use the standard path\n        if !self.enable_rx_ts {\n            // Safety: Basic enum pin projection\n            unsafe {\n                let rs_wrapper = Pin::get_unchecked_mut(self);\n                match &mut rs_wrapper.stream {\n                    RawStream::Tcp(s) => return Pin::new_unchecked(s).poll_read(cx, buf),\n                    RawStream::Unix(s) => return Pin::new_unchecked(s).poll_read(cx, buf),\n                    RawStream::Virtual(s) => return Pin::new_unchecked(s).poll_read(cx, buf),\n                }\n            }\n        }\n\n        // Safety: Basic pin projection to get mutable stream\n        let rs_wrapper = unsafe { Pin::get_unchecked_mut(self) };\n        match &mut rs_wrapper.stream {\n            RawStream::Tcp(s) => {\n                loop {\n                    ready!(s.poll_read_ready(cx))?;\n                    // Safety: maybe uninitialized bytes will only be passed to recvmsg\n                    let b = unsafe {\n                        &mut *(buf.unfilled_mut() as *mut [std::mem::MaybeUninit<u8>]\n                            as *mut [u8])\n                    };\n                    let mut iov = [IoSliceMut::new(b)];\n                    rs_wrapper.reusable_cmsg_space.clear();\n\n                    match s.try_io(Interest::READABLE, || {\n                        recvmsg::<SockaddrStorage>(\n                            s.as_raw_fd(),\n                            &mut iov,\n                            Some(&mut rs_wrapper.reusable_cmsg_space),\n                            MsgFlags::empty(),\n                        )\n                        .map_err(|errno| errno.into())\n                    }) {\n                        Ok(r) => {\n                            if let Some(ControlMessageOwned::ScmTimestampsns(rtime)) = r\n                                .cmsgs()\n                                .find(|i| matches!(i, ControlMessageOwned::ScmTimestampsns(_)))\n                            {\n                                // The returned timestamp is a real (i.e. not monotonic) timestamp\n                                // https://docs.kernel.org/networking/timestamping.html\n                                rs_wrapper.rx_ts =\n                                    SystemTime::UNIX_EPOCH.checked_add(rtime.system.into());\n                            }\n                            // Safety: We trust `recvmsg` to have filled up `r.bytes` bytes in the buffer.\n                            unsafe {\n                                buf.assume_init(r.bytes);\n                            }\n                            buf.advance(r.bytes);\n                            return Poll::Ready(Ok(()));\n                        }\n                        Err(ref e) if e.kind() == io::ErrorKind::WouldBlock => continue,\n                        Err(e) => return Poll::Ready(Err(e)),\n                    }\n                }\n            }\n            // Unix RX timestamp only works with datagram for now, so we do not care about it\n            RawStream::Unix(s) => unsafe { Pin::new_unchecked(s).poll_read(cx, buf) },\n            RawStream::Virtual(s) => unsafe { Pin::new_unchecked(s).poll_read(cx, buf) },\n        }\n    }\n}\n\nimpl AsyncWrite for RawStreamWrapper {\n    fn poll_write(self: Pin<&mut Self>, cx: &mut Context, buf: &[u8]) -> Poll<io::Result<usize>> {\n        // Safety: Basic enum pin projection\n        unsafe {\n            match &mut Pin::get_unchecked_mut(self).stream {\n                RawStream::Tcp(s) => Pin::new_unchecked(s).poll_write(cx, buf),\n                #[cfg(unix)]\n                RawStream::Unix(s) => Pin::new_unchecked(s).poll_write(cx, buf),\n                RawStream::Virtual(s) => Pin::new_unchecked(s).poll_write(cx, buf),\n            }\n        }\n    }\n\n    fn poll_flush(self: Pin<&mut Self>, cx: &mut Context) -> Poll<io::Result<()>> {\n        // Safety: Basic enum pin projection\n        unsafe {\n            match &mut Pin::get_unchecked_mut(self).stream {\n                RawStream::Tcp(s) => Pin::new_unchecked(s).poll_flush(cx),\n                #[cfg(unix)]\n                RawStream::Unix(s) => Pin::new_unchecked(s).poll_flush(cx),\n                RawStream::Virtual(s) => Pin::new_unchecked(s).poll_flush(cx),\n            }\n        }\n    }\n\n    fn poll_shutdown(self: Pin<&mut Self>, cx: &mut Context) -> Poll<io::Result<()>> {\n        // Safety: Basic enum pin projection\n        unsafe {\n            match &mut Pin::get_unchecked_mut(self).stream {\n                RawStream::Tcp(s) => Pin::new_unchecked(s).poll_shutdown(cx),\n                #[cfg(unix)]\n                RawStream::Unix(s) => Pin::new_unchecked(s).poll_shutdown(cx),\n                RawStream::Virtual(s) => Pin::new_unchecked(s).poll_shutdown(cx),\n            }\n        }\n    }\n\n    fn poll_write_vectored(\n        self: Pin<&mut Self>,\n        cx: &mut Context<'_>,\n        bufs: &[std::io::IoSlice<'_>],\n    ) -> Poll<io::Result<usize>> {\n        // Safety: Basic enum pin projection\n        unsafe {\n            match &mut Pin::get_unchecked_mut(self).stream {\n                RawStream::Tcp(s) => Pin::new_unchecked(s).poll_write_vectored(cx, bufs),\n                #[cfg(unix)]\n                RawStream::Unix(s) => Pin::new_unchecked(s).poll_write_vectored(cx, bufs),\n                RawStream::Virtual(s) => Pin::new_unchecked(s).poll_write_vectored(cx, bufs),\n            }\n        }\n    }\n\n    fn is_write_vectored(&self) -> bool {\n        self.stream.is_write_vectored()\n    }\n}\n\n#[cfg(unix)]\nimpl AsRawFd for RawStreamWrapper {\n    fn as_raw_fd(&self) -> std::os::unix::io::RawFd {\n        self.stream.as_raw_fd()\n    }\n}\n\n#[cfg(windows)]\nimpl AsRawSocket for RawStreamWrapper {\n    fn as_raw_socket(&self) -> std::os::windows::io::RawSocket {\n        self.stream.as_raw_socket()\n    }\n}\n\n// Large read buffering helps reducing syscalls with little trade-off\n// Ssl layer always does \"small\" reads in 16k (TLS record size) so L4 read buffer helps a lot.\nconst BUF_READ_SIZE: usize = 64 * 1024;\n// Small write buf to match MSS. Too large write buf delays real time communication.\n// This buffering effectively implements something similar to Nagle's algorithm.\n// The benefit is that user space can control when to flush, where Nagle's can't be controlled.\n// And userspace buffering reduce both syscalls and small packets.\nconst BUF_WRITE_SIZE: usize = 1460;\n\n// NOTE: with writer buffering, users need to call flush() to make sure the data is actually\n// sent. Otherwise data could be stuck in the buffer forever or get lost when stream is closed.\n\n/// A concrete type for transport layer connection + extra fields for logging\n#[derive(Debug)]\npub struct Stream {\n    // Use `Option` to be able to swap to adjust the buffer size. Always safe to unwrap\n    stream: Option<BufStream<RawStreamWrapper>>,\n    // the data put back at the front of the read buffer, in order to replay the read\n    rewind_read_buf: Vec<Vec<u8>>,\n    buffer_write: bool,\n    proxy_digest: Option<Arc<ProxyDigest>>,\n    socket_digest: Option<Arc<SocketDigest>>,\n    /// When this connection is established\n    pub established_ts: SystemTime,\n    /// The distributed tracing object for this stream\n    pub tracer: Option<Tracer>,\n    read_pending_time: AccumulatedDuration,\n    write_pending_time: AccumulatedDuration,\n    /// Last rx timestamp associated with the last recvmsg call.\n    pub rx_ts: Option<SystemTime>,\n}\n\nimpl Stream {\n    fn stream(&self) -> &BufStream<RawStreamWrapper> {\n        self.stream.as_ref().expect(\"stream should always be set\")\n    }\n\n    fn stream_mut(&mut self) -> &mut BufStream<RawStreamWrapper> {\n        self.stream.as_mut().expect(\"stream should always be set\")\n    }\n\n    /// set TCP nodelay for this connection if `self` is TCP\n    pub fn set_nodelay(&mut self) -> Result<()> {\n        match &self.stream_mut().get_mut().stream {\n            RawStream::Tcp(s) => {\n                s.set_nodelay(true)\n                    .or_err(ConnectError, \"failed to set_nodelay\")?;\n            }\n            RawStream::Virtual(s) => {\n                s.set_socket_option(virt::VirtualSockOpt::NoDelay)\n                    .or_err(ConnectError, \"failed to set_nodelay on virtual socket\")?;\n            }\n            _ => (),\n        }\n        Ok(())\n    }\n\n    /// set TCP keepalive settings for this connection if `self` is TCP\n    pub fn set_keepalive(&mut self, ka: &TcpKeepalive) -> Result<()> {\n        match &self.stream_mut().get_mut().stream {\n            RawStream::Tcp(s) => {\n                debug!(\"Setting tcp keepalive\");\n                set_tcp_keepalive(s, ka)?;\n            }\n            RawStream::Virtual(s) => {\n                s.set_socket_option(virt::VirtualSockOpt::KeepAlive(ka.clone()))\n                    .or_err(ConnectError, \"failed to set_keepalive on virtual socket\")?;\n            }\n            _ => (),\n        }\n        Ok(())\n    }\n\n    #[cfg(target_os = \"linux\")]\n    pub fn set_rx_timestamp(&mut self) -> Result<()> {\n        use nix::sys::socket::{setsockopt, sockopt, TimestampingFlag};\n\n        if let RawStream::Tcp(s) = &self.stream_mut().get_mut().stream {\n            let timestamp_options = TimestampingFlag::SOF_TIMESTAMPING_RX_SOFTWARE\n                | TimestampingFlag::SOF_TIMESTAMPING_SOFTWARE;\n            setsockopt(s.as_raw_fd(), sockopt::Timestamping, &timestamp_options)\n                .or_err(InternalError, \"failed to set SOF_TIMESTAMPING_RX_SOFTWARE\")?;\n            self.stream_mut().get_mut().enable_rx_ts(true);\n        }\n\n        Ok(())\n    }\n\n    #[cfg(not(target_os = \"linux\"))]\n    pub fn set_rx_timestamp(&mut self) -> io::Result<()> {\n        Ok(())\n    }\n\n    /// Put Some data back to the head of the stream to be read again\n    pub(crate) fn rewind(&mut self, data: &[u8]) {\n        if !data.is_empty() {\n            self.rewind_read_buf.push(data.to_vec());\n        }\n    }\n\n    /// Set the buffer of BufStream\n    /// It is only set later because of the malloc overhead in critical accept() path\n    pub(crate) fn set_buffer(&mut self) {\n        use std::mem;\n        // Since BufStream doesn't provide an API to adjust the buf directly,\n        // we take the raw stream out of it and put it in a new BufStream with the size we want\n        let stream = mem::take(&mut self.stream);\n        let stream =\n            stream.map(|s| BufStream::with_capacity(BUF_READ_SIZE, BUF_WRITE_SIZE, s.into_inner()));\n        let _ = mem::replace(&mut self.stream, stream);\n    }\n}\n\nimpl From<TcpStream> for Stream {\n    fn from(s: TcpStream) -> Self {\n        Stream {\n            stream: Some(BufStream::with_capacity(\n                0,\n                0,\n                RawStreamWrapper::new(RawStream::Tcp(s)),\n            )),\n            rewind_read_buf: Vec::new(),\n            buffer_write: true,\n            established_ts: SystemTime::now(),\n            proxy_digest: None,\n            socket_digest: None,\n            tracer: None,\n            read_pending_time: AccumulatedDuration::new(),\n            write_pending_time: AccumulatedDuration::new(),\n            rx_ts: None,\n        }\n    }\n}\n\nimpl From<virt::VirtualSocketStream> for Stream {\n    fn from(s: virt::VirtualSocketStream) -> Self {\n        Stream {\n            stream: Some(BufStream::with_capacity(\n                0,\n                0,\n                RawStreamWrapper::new(RawStream::Virtual(s)),\n            )),\n            rewind_read_buf: Vec::new(),\n            buffer_write: true,\n            established_ts: SystemTime::now(),\n            proxy_digest: None,\n            socket_digest: None,\n            tracer: None,\n            read_pending_time: AccumulatedDuration::new(),\n            write_pending_time: AccumulatedDuration::new(),\n            rx_ts: None,\n        }\n    }\n}\n\n#[cfg(unix)]\nimpl From<UnixStream> for Stream {\n    fn from(s: UnixStream) -> Self {\n        Stream {\n            stream: Some(BufStream::with_capacity(\n                0,\n                0,\n                RawStreamWrapper::new(RawStream::Unix(s)),\n            )),\n            rewind_read_buf: Vec::new(),\n            buffer_write: true,\n            established_ts: SystemTime::now(),\n            proxy_digest: None,\n            socket_digest: None,\n            tracer: None,\n            read_pending_time: AccumulatedDuration::new(),\n            write_pending_time: AccumulatedDuration::new(),\n            rx_ts: None,\n        }\n    }\n}\n\n#[cfg(unix)]\nimpl AsRawFd for Stream {\n    fn as_raw_fd(&self) -> std::os::unix::io::RawFd {\n        self.stream().get_ref().as_raw_fd()\n    }\n}\n\n#[cfg(windows)]\nimpl AsRawSocket for Stream {\n    fn as_raw_socket(&self) -> std::os::windows::io::RawSocket {\n        self.stream().get_ref().as_raw_socket()\n    }\n}\n\n#[cfg(unix)]\nimpl UniqueID for Stream {\n    fn id(&self) -> UniqueIDType {\n        self.as_raw_fd()\n    }\n}\n\n#[cfg(windows)]\nimpl UniqueID for Stream {\n    fn id(&self) -> usize {\n        self.as_raw_socket() as usize\n    }\n}\n\nimpl Ssl for Stream {}\n\n#[async_trait]\nimpl Peek for Stream {\n    async fn try_peek(&mut self, buf: &mut [u8]) -> std::io::Result<bool> {\n        use tokio::io::AsyncReadExt;\n        self.read_exact(buf).await?;\n        // rewind regardless of what is read\n        self.rewind(buf);\n        Ok(true)\n    }\n}\n\n#[async_trait]\nimpl Shutdown for Stream {\n    async fn shutdown(&mut self) {\n        AsyncWriteExt::shutdown(self).await.unwrap_or_else(|e| {\n            debug!(\"Failed to shutdown connection: {:?}\", e);\n        });\n    }\n}\n\nimpl GetTimingDigest for Stream {\n    fn get_timing_digest(&self) -> Vec<Option<TimingDigest>> {\n        let mut digest = Vec::with_capacity(2); // expect to have both L4 stream and TLS layer\n        digest.push(Some(TimingDigest {\n            established_ts: self.established_ts,\n        }));\n        digest\n    }\n\n    fn get_read_pending_time(&self) -> Duration {\n        self.read_pending_time.total\n    }\n\n    fn get_write_pending_time(&self) -> Duration {\n        self.write_pending_time.total\n    }\n}\n\nimpl GetProxyDigest for Stream {\n    fn get_proxy_digest(&self) -> Option<Arc<ProxyDigest>> {\n        self.proxy_digest.clone()\n    }\n\n    fn set_proxy_digest(&mut self, digest: ProxyDigest) {\n        self.proxy_digest = Some(Arc::new(digest));\n    }\n}\n\nimpl GetSocketDigest for Stream {\n    fn get_socket_digest(&self) -> Option<Arc<SocketDigest>> {\n        self.socket_digest.clone()\n    }\n\n    fn set_socket_digest(&mut self, socket_digest: SocketDigest) {\n        self.socket_digest = Some(Arc::new(socket_digest))\n    }\n}\n\nimpl Drop for Stream {\n    fn drop(&mut self) {\n        if let Some(t) = self.tracer.as_ref() {\n            t.0.on_disconnected();\n        }\n        /* use nodelay/local_addr function to detect socket status */\n        let ret = match &self.stream().get_ref().stream {\n            RawStream::Tcp(s) => s.nodelay().err(),\n            #[cfg(unix)]\n            RawStream::Unix(s) => s.local_addr().err(),\n            RawStream::Virtual(_) => {\n                // TODO: should this do something?\n                None\n            }\n        };\n        if let Some(e) = ret {\n            match e.kind() {\n                tokio::io::ErrorKind::Other => {\n                    if let Some(ecode) = e.raw_os_error() {\n                        if ecode == 9 {\n                            // Or we could panic here\n                            error!(\"Crit: socket {:?} is being double closed\", self.stream);\n                        }\n                    }\n                }\n                _ => {\n                    debug!(\"Socket is already broken {:?}\", e);\n                }\n            }\n        } else {\n            // try flush the write buffer. We use now_or_never() because\n            // 1. Drop cannot be async\n            // 2. write should usually be ready, unless the buf is full.\n            let _ = self.flush().now_or_never();\n        }\n        debug!(\"Dropping socket {:?}\", self.stream);\n    }\n}\n\nimpl AsyncRead for Stream {\n    fn poll_read(\n        mut self: Pin<&mut Self>,\n        cx: &mut Context<'_>,\n        buf: &mut ReadBuf<'_>,\n    ) -> Poll<io::Result<()>> {\n        let result = if !self.rewind_read_buf.is_empty() {\n            let data_to_read = self.rewind_read_buf.pop().unwrap(); // safe\n            let mut data_to_read = data_to_read.as_slice();\n            let result = Pin::new(&mut data_to_read).poll_read(cx, buf);\n            // return the remaining data back to the head of rewind_read_buf\n            if !data_to_read.is_empty() {\n                let remaining_buf = Vec::from(data_to_read);\n                self.rewind_read_buf.push(remaining_buf);\n            }\n            result\n        } else {\n            Pin::new(&mut self.stream_mut()).poll_read(cx, buf)\n        };\n        self.read_pending_time.poll_time(&result);\n        self.rx_ts = self.stream().get_ref().rx_ts;\n        result\n    }\n}\n\nimpl AsyncWrite for Stream {\n    fn poll_write(\n        mut self: Pin<&mut Self>,\n        cx: &mut Context,\n        buf: &[u8],\n    ) -> Poll<io::Result<usize>> {\n        let result = if self.buffer_write {\n            Pin::new(&mut self.stream_mut()).poll_write(cx, buf)\n        } else {\n            Pin::new(&mut self.stream_mut().get_mut()).poll_write(cx, buf)\n        };\n        self.write_pending_time.poll_write_time(&result, buf.len());\n        result\n    }\n\n    fn poll_flush(mut self: Pin<&mut Self>, cx: &mut Context) -> Poll<io::Result<()>> {\n        let result = Pin::new(&mut self.stream_mut()).poll_flush(cx);\n        self.write_pending_time.poll_time(&result);\n        result\n    }\n\n    fn poll_shutdown(mut self: Pin<&mut Self>, cx: &mut Context) -> Poll<io::Result<()>> {\n        Pin::new(&mut self.stream_mut()).poll_shutdown(cx)\n    }\n\n    fn poll_write_vectored(\n        mut self: Pin<&mut Self>,\n        cx: &mut Context<'_>,\n        bufs: &[std::io::IoSlice<'_>],\n    ) -> Poll<io::Result<usize>> {\n        let total_size = bufs.iter().fold(0, |acc, s| acc + s.len());\n\n        let result = if self.buffer_write {\n            Pin::new(&mut self.stream_mut()).poll_write_vectored(cx, bufs)\n        } else {\n            Pin::new(&mut self.stream_mut().get_mut()).poll_write_vectored(cx, bufs)\n        };\n\n        self.write_pending_time.poll_write_time(&result, total_size);\n        result\n    }\n\n    fn is_write_vectored(&self) -> bool {\n        if self.buffer_write {\n            self.stream().is_write_vectored() // it is true\n        } else {\n            self.stream().get_ref().is_write_vectored()\n        }\n    }\n}\n\npub mod async_write_vec {\n    use bytes::Buf;\n    use futures::ready;\n    use std::future::Future;\n    use std::io::IoSlice;\n    use std::pin::Pin;\n    use std::task::{Context, Poll};\n    use tokio::io;\n    use tokio::io::AsyncWrite;\n\n    /*\n        the missing write_buf https://github.com/tokio-rs/tokio/pull/3156#issuecomment-738207409\n        https://github.com/tokio-rs/tokio/issues/2610\n        In general vectored write is lost when accessing the trait object: Box<S: AsyncWrite>\n    */\n\n    #[must_use = \"futures do nothing unless you `.await` or poll them\"]\n    pub struct WriteVec<'a, W, B> {\n        writer: &'a mut W,\n        buf: &'a mut B,\n    }\n\n    #[must_use = \"futures do nothing unless you `.await` or poll them\"]\n    pub struct WriteVecAll<'a, W, B> {\n        writer: &'a mut W,\n        buf: &'a mut B,\n    }\n\n    pub trait AsyncWriteVec {\n        fn poll_write_vec<B: Buf>(\n            self: Pin<&mut Self>,\n            _cx: &mut Context<'_>,\n            _buf: &mut B,\n        ) -> Poll<io::Result<usize>>;\n\n        fn write_vec<'a, B>(&'a mut self, src: &'a mut B) -> WriteVec<'a, Self, B>\n        where\n            Self: Sized,\n            B: Buf,\n        {\n            WriteVec {\n                writer: self,\n                buf: src,\n            }\n        }\n\n        fn write_vec_all<'a, B>(&'a mut self, src: &'a mut B) -> WriteVecAll<'a, Self, B>\n        where\n            Self: Sized,\n            B: Buf,\n        {\n            WriteVecAll {\n                writer: self,\n                buf: src,\n            }\n        }\n    }\n\n    impl<W, B> Future for WriteVec<'_, W, B>\n    where\n        W: AsyncWriteVec + Unpin,\n        B: Buf,\n    {\n        type Output = io::Result<usize>;\n\n        fn poll(mut self: Pin<&mut Self>, ctx: &mut Context<'_>) -> Poll<io::Result<usize>> {\n            let me = &mut *self;\n            Pin::new(&mut *me.writer).poll_write_vec(ctx, me.buf)\n        }\n    }\n\n    impl<W, B> Future for WriteVecAll<'_, W, B>\n    where\n        W: AsyncWriteVec + Unpin,\n        B: Buf,\n    {\n        type Output = io::Result<()>;\n\n        fn poll(mut self: Pin<&mut Self>, ctx: &mut Context<'_>) -> Poll<io::Result<()>> {\n            let me = &mut *self;\n            while me.buf.has_remaining() {\n                let n = ready!(Pin::new(&mut *me.writer).poll_write_vec(ctx, me.buf))?;\n                if n == 0 {\n                    return Poll::Ready(Err(io::ErrorKind::WriteZero.into()));\n                }\n            }\n            Poll::Ready(Ok(()))\n        }\n    }\n\n    /* from https://github.com/tokio-rs/tokio/blob/master/tokio-util/src/lib.rs#L177 */\n    impl<T> AsyncWriteVec for T\n    where\n        T: AsyncWrite,\n    {\n        fn poll_write_vec<B: Buf>(\n            self: Pin<&mut Self>,\n            ctx: &mut Context,\n            buf: &mut B,\n        ) -> Poll<io::Result<usize>> {\n            const MAX_BUFS: usize = 64;\n\n            if !buf.has_remaining() {\n                return Poll::Ready(Ok(0));\n            }\n\n            let n = if self.is_write_vectored() {\n                let mut slices = [IoSlice::new(&[]); MAX_BUFS];\n                let cnt = buf.chunks_vectored(&mut slices);\n                ready!(self.poll_write_vectored(ctx, &slices[..cnt]))?\n            } else {\n                ready!(self.poll_write(ctx, buf.chunk()))?\n            };\n\n            buf.advance(n);\n\n            Poll::Ready(Ok(n))\n        }\n    }\n}\n\npub use async_write_vec::AsyncWriteVec;\n\n#[derive(Debug)]\nstruct AccumulatedDuration {\n    total: Duration,\n    last_start: Option<Instant>,\n}\n\nimpl AccumulatedDuration {\n    fn new() -> Self {\n        AccumulatedDuration {\n            total: Duration::ZERO,\n            last_start: None,\n        }\n    }\n\n    fn start(&mut self) {\n        if self.last_start.is_none() {\n            self.last_start = Some(Instant::now());\n        }\n    }\n\n    fn stop(&mut self) {\n        if let Some(start) = self.last_start.take() {\n            self.total += start.elapsed();\n        }\n    }\n\n    fn poll_write_time(&mut self, result: &Poll<io::Result<usize>>, buf_size: usize) {\n        match result {\n            Poll::Ready(Ok(n)) => {\n                if *n == buf_size {\n                    self.stop();\n                } else {\n                    // partial write\n                    self.start();\n                }\n            }\n            Poll::Ready(Err(_)) => {\n                self.stop();\n            }\n            _ => self.start(),\n        }\n    }\n\n    fn poll_time(&mut self, result: &Poll<io::Result<()>>) {\n        match result {\n            Poll::Ready(_) => {\n                self.stop();\n            }\n            _ => self.start(),\n        }\n    }\n}\n\n#[cfg(test)]\n#[cfg(target_os = \"linux\")]\nmod tests {\n    use super::*;\n    use std::sync::Arc;\n    use tokio::io::AsyncReadExt;\n    use tokio::io::AsyncWriteExt;\n    use tokio::net::TcpListener;\n    use tokio::sync::Notify;\n\n    #[cfg(target_os = \"linux\")]\n    #[tokio::test]\n    async fn test_rx_timestamp() {\n        let message = \"hello world\".as_bytes();\n        let listener = TcpListener::bind(\"127.0.0.1:0\").await.unwrap();\n        let addr = listener.local_addr().unwrap();\n        let notify = Arc::new(Notify::new());\n        let notify2 = notify.clone();\n\n        tokio::spawn(async move {\n            let (mut stream, _) = listener.accept().await.unwrap();\n            notify2.notified().await;\n            stream.write_all(message).await.unwrap();\n        });\n\n        let mut stream: Stream = TcpStream::connect(addr).await.unwrap().into();\n        stream.set_rx_timestamp().unwrap();\n        // Receive the message\n        // setsockopt for SO_TIMESTAMPING is asynchronous so sleep a little bit\n        // to let kernel do the work\n        std::thread::sleep(Duration::from_micros(100));\n        notify.notify_one();\n\n        let mut buffer = vec![0u8; message.len()];\n        let n = stream.read(buffer.as_mut_slice()).await.unwrap();\n        assert_eq!(n, message.len());\n        assert!(stream.rx_ts.is_some());\n    }\n\n    #[cfg(target_os = \"linux\")]\n    #[tokio::test]\n    async fn test_rx_timestamp_standard_path() {\n        let message = \"hello world\".as_bytes();\n        let listener = TcpListener::bind(\"127.0.0.1:0\").await.unwrap();\n        let addr = listener.local_addr().unwrap();\n        let notify = Arc::new(Notify::new());\n        let notify2 = notify.clone();\n\n        tokio::spawn(async move {\n            let (mut stream, _) = listener.accept().await.unwrap();\n            notify2.notified().await;\n            stream.write_all(message).await.unwrap();\n        });\n\n        let mut stream: Stream = TcpStream::connect(addr).await.unwrap().into();\n        std::thread::sleep(Duration::from_micros(100));\n        notify.notify_one();\n\n        let mut buffer = vec![0u8; message.len()];\n        let n = stream.read(buffer.as_mut_slice()).await.unwrap();\n        assert_eq!(n, message.len());\n        assert!(stream.rx_ts.is_none());\n    }\n\n    #[tokio::test]\n    async fn test_stream_rewind() {\n        let message = b\"hello world\";\n        let listener = TcpListener::bind(\"127.0.0.1:0\").await.unwrap();\n        let addr = listener.local_addr().unwrap();\n        let notify = Arc::new(Notify::new());\n        let notify2 = notify.clone();\n\n        tokio::spawn(async move {\n            let (mut stream, _) = listener.accept().await.unwrap();\n            notify2.notified().await;\n            stream.write_all(message).await.unwrap();\n        });\n\n        let mut stream: Stream = TcpStream::connect(addr).await.unwrap().into();\n\n        let rewind_test = b\"this is Sparta!\";\n        stream.rewind(rewind_test);\n\n        // partially read rewind_test because of the buffer size limit\n        let mut buffer = vec![0u8; message.len()];\n        let n = stream.read(buffer.as_mut_slice()).await.unwrap();\n        assert_eq!(n, message.len());\n        assert_eq!(buffer, rewind_test[..message.len()]);\n\n        // read the rest of rewind_test\n        let n = stream.read(buffer.as_mut_slice()).await.unwrap();\n        assert_eq!(n, rewind_test.len() - message.len());\n        assert_eq!(buffer[..n], rewind_test[message.len()..]);\n\n        // read the actual data\n        notify.notify_one();\n        let n = stream.read(buffer.as_mut_slice()).await.unwrap();\n        assert_eq!(n, message.len());\n        assert_eq!(buffer, message);\n    }\n\n    #[tokio::test]\n    async fn test_stream_peek() {\n        let message = b\"hello world\";\n        dbg!(\"try peek\");\n        let listener = TcpListener::bind(\"127.0.0.1:0\").await.unwrap();\n        let addr = listener.local_addr().unwrap();\n        let notify = Arc::new(Notify::new());\n        let notify2 = notify.clone();\n\n        tokio::spawn(async move {\n            let (mut stream, _) = listener.accept().await.unwrap();\n            notify2.notified().await;\n            stream.write_all(message).await.unwrap();\n            drop(stream);\n        });\n\n        notify.notify_one();\n\n        let mut stream: Stream = TcpStream::connect(addr).await.unwrap().into();\n        let mut buffer = vec![0u8; 5];\n        assert!(stream.try_peek(&mut buffer).await.unwrap());\n        assert_eq!(buffer, message[0..5]);\n        let mut buffer = vec![];\n        stream.read_to_end(&mut buffer).await.unwrap();\n        assert_eq!(buffer, message);\n    }\n\n    #[tokio::test]\n    async fn test_stream_two_subsequent_peek_calls_before_read() {\n        let message = b\"abcdefghijklmn\";\n\n        let listener = TcpListener::bind(\"127.0.0.1:0\").await.unwrap();\n        let addr = listener.local_addr().unwrap();\n        let notify = Arc::new(Notify::new());\n        let notify2 = notify.clone();\n\n        tokio::spawn(async move {\n            let (mut stream, _) = listener.accept().await.unwrap();\n            notify2.notified().await;\n            stream.write_all(message).await.unwrap();\n            drop(stream);\n        });\n\n        notify.notify_one();\n\n        let mut stream: Stream = TcpStream::connect(addr).await.unwrap().into();\n\n        // Peek 4 bytes\n        let mut buffer = vec![0u8; 4];\n        assert!(stream.try_peek(&mut buffer).await.unwrap());\n        assert_eq!(buffer, message[0..4]);\n\n        // Peek 2 bytes\n        let mut buffer = vec![0u8; 2];\n        assert!(stream.try_peek(&mut buffer).await.unwrap());\n        assert_eq!(buffer, message[0..2]);\n\n        // Read 1 byte: ['a']\n        let mut buffer = vec![0u8; 1];\n        stream.read_exact(&mut buffer).await.unwrap();\n        assert_eq!(buffer, message[0..1]);\n\n        // Read as many bytes as possible, return 1 byte ['b']\n        //  from the first retry buffer chunk\n        let mut buffer = vec![0u8; 100];\n        let n = stream.read(&mut buffer).await.unwrap();\n        assert_eq!(n, 1);\n        assert_eq!(buffer[..n], message[1..2]);\n\n        // Read the rest ['cdefghijklmn']\n        let mut buffer = vec![];\n        stream.read_to_end(&mut buffer).await.unwrap();\n        assert_eq!(buffer, message[2..]);\n    }\n}\n"
  },
  {
    "path": "pingora-core/src/protocols/l4/virt.rs",
    "content": "//! Provides [`VirtualSocketStream`].\n\nuse std::{\n    pin::Pin,\n    task::{Context, Poll},\n};\n\nuse tokio::io::{AsyncRead, AsyncWrite};\n\nuse super::ext::TcpKeepalive;\n\n/// A limited set of socket options that can be set on a [`VirtualSocket`].\n#[non_exhaustive]\n#[derive(Debug, Clone)]\npub enum VirtualSockOpt {\n    NoDelay,\n    KeepAlive(TcpKeepalive),\n}\n\n/// A \"virtual\" socket that supports async read and write operations.\npub trait VirtualSocket: AsyncRead + AsyncWrite + Unpin + Send + Sync + std::fmt::Debug {\n    /// Set a socket option.\n    fn set_socket_option(&self, opt: VirtualSockOpt) -> std::io::Result<()>;\n}\n\n/// Wrapper around any type implementing  [`VirtualSocket`].\n#[derive(Debug)]\npub struct VirtualSocketStream {\n    pub(crate) socket: Box<dyn VirtualSocket>,\n}\n\nimpl VirtualSocketStream {\n    pub fn new(socket: Box<dyn VirtualSocket>) -> Self {\n        Self { socket }\n    }\n\n    #[inline]\n    pub fn set_socket_option(&self, opt: VirtualSockOpt) -> std::io::Result<()> {\n        self.socket.set_socket_option(opt)\n    }\n}\n\nimpl AsyncRead for VirtualSocketStream {\n    #[inline]\n    fn poll_read(\n        self: Pin<&mut Self>,\n        cx: &mut Context<'_>,\n        buf: &mut tokio::io::ReadBuf<'_>,\n    ) -> Poll<std::io::Result<()>> {\n        Pin::new(&mut *self.get_mut().socket).poll_read(cx, buf)\n    }\n}\n\nimpl AsyncWrite for VirtualSocketStream {\n    #[inline]\n    fn poll_write(\n        self: Pin<&mut Self>,\n        cx: &mut Context<'_>,\n        buf: &[u8],\n    ) -> Poll<std::io::Result<usize>> {\n        Pin::new(&mut *self.get_mut().socket).poll_write(cx, buf)\n    }\n\n    #[inline]\n    fn poll_flush(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<std::io::Result<()>> {\n        Pin::new(&mut *self.get_mut().socket).poll_flush(cx)\n    }\n\n    #[inline]\n    fn poll_shutdown(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<std::io::Result<()>> {\n        Pin::new(&mut *self.get_mut().socket).poll_shutdown(cx)\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use std::sync::{Arc, Mutex};\n\n    use tokio::io::{AsyncReadExt, AsyncWriteExt as _};\n\n    use crate::protocols::l4::stream::Stream;\n\n    use super::*;\n\n    #[derive(Debug)]\n    struct StaticVirtualSocket {\n        content: Vec<u8>,\n        read_pos: usize,\n        write_buf: Arc<Mutex<Vec<u8>>>,\n    }\n\n    impl AsyncRead for StaticVirtualSocket {\n        fn poll_read(\n            mut self: Pin<&mut Self>,\n            _cx: &mut Context<'_>,\n            buf: &mut tokio::io::ReadBuf<'_>,\n        ) -> Poll<std::io::Result<()>> {\n            debug_assert!(self.read_pos <= self.content.len());\n\n            let remaining = self.content.len() - self.read_pos;\n            if remaining == 0 {\n                return Poll::Ready(Ok(()));\n            }\n\n            let to_read = std::cmp::min(remaining, buf.remaining());\n            buf.put_slice(&self.content[self.read_pos..self.read_pos + to_read]);\n            self.read_pos += to_read;\n\n            Poll::Ready(Ok(()))\n        }\n    }\n\n    impl AsyncWrite for StaticVirtualSocket {\n        fn poll_write(\n            self: Pin<&mut Self>,\n            _cx: &mut Context<'_>,\n            buf: &[u8],\n        ) -> Poll<std::io::Result<usize>> {\n            // write to internal buffer\n            let this = self.get_mut();\n            this.write_buf.lock().unwrap().extend_from_slice(buf);\n            Poll::Ready(Ok(buf.len()))\n        }\n\n        fn poll_flush(self: Pin<&mut Self>, _cx: &mut Context<'_>) -> Poll<std::io::Result<()>> {\n            Poll::Ready(Ok(()))\n        }\n\n        fn poll_shutdown(self: Pin<&mut Self>, _cx: &mut Context<'_>) -> Poll<std::io::Result<()>> {\n            Poll::Ready(Ok(()))\n        }\n    }\n\n    impl VirtualSocket for StaticVirtualSocket {\n        fn set_socket_option(&self, _opt: VirtualSockOpt) -> std::io::Result<()> {\n            Ok(())\n        }\n    }\n\n    /// Basic test that ensures reading and writing works with a virtual socket.\n    //\n    /// Mostly just ensures that construction works and the plumbing is correct.\n    #[tokio::test]\n    async fn test_stream_virtual() {\n        let content = b\"hello virtual world\";\n        let write_buf = Arc::new(Mutex::new(Vec::new()));\n        let mut stream = Stream::from(VirtualSocketStream::new(Box::new(StaticVirtualSocket {\n            content: content.to_vec(),\n            read_pos: 0,\n            write_buf: write_buf.clone(),\n        })));\n\n        let mut buf = Vec::new();\n        let out = stream.read_to_end(&mut buf).await.unwrap();\n        assert_eq!(out, content.len());\n        assert_eq!(buf, content);\n\n        stream.write_all(content).await.unwrap();\n        assert_eq!(write_buf.lock().unwrap().as_slice(), content);\n    }\n}\n"
  },
  {
    "path": "pingora-core/src/protocols/mod.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! Abstractions and implementations for protocols including TCP, TLS and HTTP\n\nmod digest;\npub mod http;\npub mod l4;\npub mod raw_connect;\npub mod tls;\n#[cfg(windows)]\nmod windows;\n\npub use digest::{\n    Digest, GetProxyDigest, GetSocketDigest, GetTimingDigest, ProtoDigest, SocketDigest,\n    TimingDigest,\n};\npub use l4::ext::TcpKeepalive;\npub use tls::ALPN;\n\nuse async_trait::async_trait;\nuse std::fmt::Debug;\nuse std::net::{IpAddr, Ipv4Addr};\nuse std::sync::Arc;\n\n#[cfg(unix)]\npub type UniqueIDType = i32;\n#[cfg(windows)]\npub type UniqueIDType = usize;\n\n/// Define how a protocol should shutdown its connection.\n#[async_trait]\npub trait Shutdown {\n    async fn shutdown(&mut self) -> ();\n}\n\n/// Define how a given session/connection identifies itself.\npub trait UniqueID {\n    /// The ID returned should be unique among all existing connections of the same type.\n    /// But ID can be recycled after a connection is shutdown.\n    fn id(&self) -> UniqueIDType;\n}\n\n/// Interface to get TLS info\npub trait Ssl {\n    /// Return the TLS info if the connection is over TLS\n    fn get_ssl(&self) -> Option<&TlsRef> {\n        None\n    }\n\n    /// Return the [`tls::SslDigest`] for logging\n    fn get_ssl_digest(&self) -> Option<Arc<tls::SslDigest>> {\n        None\n    }\n\n    /// Return selected ALPN if any\n    fn selected_alpn_proto(&self) -> Option<ALPN> {\n        None\n    }\n}\n\n/// The ability peek data before consuming it\n#[async_trait]\npub trait Peek {\n    /// Peek data but not consuming it. This call should block until some data\n    /// is sent.\n    /// Return `false` if peeking is not supported/allowed.\n    async fn try_peek(&mut self, _buf: &mut [u8]) -> std::io::Result<bool> {\n        Ok(false)\n    }\n}\n\nuse std::any::Any;\nuse tokio::io::{AsyncRead, AsyncWrite};\n\n/// The abstraction of transport layer IO\npub trait IO:\n    AsyncRead\n    + AsyncWrite\n    + Shutdown\n    + UniqueID\n    + Ssl\n    + GetTimingDigest\n    + GetProxyDigest\n    + GetSocketDigest\n    + Peek\n    + Unpin\n    + Debug\n    + Send\n    + Sync\n{\n    /// helper to cast as the reference of the concrete type\n    fn as_any(&self) -> &dyn Any;\n    /// helper to cast back of the concrete type\n    fn into_any(self: Box<Self>) -> Box<dyn Any>;\n}\n\nimpl<\n        T: AsyncRead\n            + AsyncWrite\n            + Shutdown\n            + UniqueID\n            + Ssl\n            + GetTimingDigest\n            + GetProxyDigest\n            + GetSocketDigest\n            + Peek\n            + Unpin\n            + Debug\n            + Send\n            + Sync,\n    > IO for T\nwhere\n    T: 'static,\n{\n    fn as_any(&self) -> &dyn Any {\n        self\n    }\n    fn into_any(self: Box<Self>) -> Box<dyn Any> {\n        self\n    }\n}\n\n/// The type of any established transport layer connection\npub type Stream = Box<dyn IO>;\n\n// Implement IO trait for 3rd party types, mostly for testing\nmod ext_io_impl {\n    use super::*;\n    use tokio_test::io::Mock;\n\n    #[async_trait]\n    impl Shutdown for Mock {\n        async fn shutdown(&mut self) -> () {}\n    }\n    impl UniqueID for Mock {\n        fn id(&self) -> UniqueIDType {\n            0\n        }\n    }\n    impl Ssl for Mock {}\n    impl GetTimingDigest for Mock {\n        fn get_timing_digest(&self) -> Vec<Option<TimingDigest>> {\n            vec![]\n        }\n    }\n    impl GetProxyDigest for Mock {\n        fn get_proxy_digest(&self) -> Option<Arc<raw_connect::ProxyDigest>> {\n            None\n        }\n    }\n    impl GetSocketDigest for Mock {\n        fn get_socket_digest(&self) -> Option<Arc<SocketDigest>> {\n            None\n        }\n    }\n\n    impl Peek for Mock {}\n\n    use std::io::Cursor;\n\n    #[async_trait]\n    impl<T: Send> Shutdown for Cursor<T> {\n        async fn shutdown(&mut self) -> () {}\n    }\n    impl<T> UniqueID for Cursor<T> {\n        fn id(&self) -> UniqueIDType {\n            0\n        }\n    }\n    impl<T> Ssl for Cursor<T> {}\n    impl<T> GetTimingDigest for Cursor<T> {\n        fn get_timing_digest(&self) -> Vec<Option<TimingDigest>> {\n            vec![]\n        }\n    }\n    impl<T> GetProxyDigest for Cursor<T> {\n        fn get_proxy_digest(&self) -> Option<Arc<raw_connect::ProxyDigest>> {\n            None\n        }\n    }\n    impl<T> GetSocketDigest for Cursor<T> {\n        fn get_socket_digest(&self) -> Option<Arc<SocketDigest>> {\n            None\n        }\n    }\n    impl<T> Peek for Cursor<T> {}\n\n    use tokio::io::DuplexStream;\n\n    #[async_trait]\n    impl Shutdown for DuplexStream {\n        async fn shutdown(&mut self) -> () {}\n    }\n    impl UniqueID for DuplexStream {\n        fn id(&self) -> UniqueIDType {\n            0\n        }\n    }\n    impl Ssl for DuplexStream {}\n    impl GetTimingDigest for DuplexStream {\n        fn get_timing_digest(&self) -> Vec<Option<TimingDigest>> {\n            vec![]\n        }\n    }\n    impl GetProxyDigest for DuplexStream {\n        fn get_proxy_digest(&self) -> Option<Arc<raw_connect::ProxyDigest>> {\n            None\n        }\n    }\n    impl GetSocketDigest for DuplexStream {\n        fn get_socket_digest(&self) -> Option<Arc<SocketDigest>> {\n            None\n        }\n    }\n\n    impl Peek for DuplexStream {}\n}\n\n#[cfg(unix)]\npub mod ext_test {\n    use std::sync::Arc;\n\n    use async_trait::async_trait;\n\n    use super::{\n        raw_connect, GetProxyDigest, GetSocketDigest, GetTimingDigest, Peek, Shutdown,\n        SocketDigest, Ssl, TimingDigest, UniqueID, UniqueIDType,\n    };\n\n    #[async_trait]\n    impl Shutdown for tokio::net::UnixStream {\n        async fn shutdown(&mut self) -> () {}\n    }\n    impl UniqueID for tokio::net::UnixStream {\n        fn id(&self) -> UniqueIDType {\n            0\n        }\n    }\n    impl Ssl for tokio::net::UnixStream {}\n    impl GetTimingDigest for tokio::net::UnixStream {\n        fn get_timing_digest(&self) -> Vec<Option<TimingDigest>> {\n            vec![]\n        }\n    }\n    impl GetProxyDigest for tokio::net::UnixStream {\n        fn get_proxy_digest(&self) -> Option<Arc<raw_connect::ProxyDigest>> {\n            None\n        }\n    }\n    impl GetSocketDigest for tokio::net::UnixStream {\n        fn get_socket_digest(&self) -> Option<Arc<SocketDigest>> {\n            None\n        }\n    }\n\n    impl Peek for tokio::net::UnixStream {}\n}\n\n#[cfg(unix)]\npub(crate) trait ConnFdReusable {\n    fn check_fd_match<V: AsRawFd>(&self, fd: V) -> bool;\n}\n\n#[cfg(windows)]\npub(crate) trait ConnSockReusable {\n    fn check_sock_match<V: AsRawSocket>(&self, sock: V) -> bool;\n}\n\nuse l4::socket::SocketAddr;\nuse log::{debug, error};\n#[cfg(unix)]\nuse nix::sys::socket::{getpeername, SockaddrStorage, UnixAddr};\n#[cfg(unix)]\nuse std::os::unix::prelude::AsRawFd;\n#[cfg(windows)]\nuse std::os::windows::io::AsRawSocket;\nuse std::{net::SocketAddr as InetSocketAddr, path::Path};\n\nuse crate::protocols::tls::TlsRef;\n\n#[cfg(unix)]\nimpl ConnFdReusable for SocketAddr {\n    fn check_fd_match<V: AsRawFd>(&self, fd: V) -> bool {\n        match self {\n            SocketAddr::Inet(addr) => addr.check_fd_match(fd),\n            SocketAddr::Unix(addr) => addr\n                .as_pathname()\n                .expect(\"non-pathname unix sockets not supported as peer\")\n                .check_fd_match(fd),\n        }\n    }\n}\n\n#[cfg(windows)]\nimpl ConnSockReusable for SocketAddr {\n    fn check_sock_match<V: AsRawSocket>(&self, sock: V) -> bool {\n        match self {\n            SocketAddr::Inet(addr) => addr.check_sock_match(sock),\n        }\n    }\n}\n\n#[cfg(unix)]\nimpl ConnFdReusable for Path {\n    fn check_fd_match<V: AsRawFd>(&self, fd: V) -> bool {\n        let fd = fd.as_raw_fd();\n        match getpeername::<UnixAddr>(fd) {\n            Ok(peer) => match UnixAddr::new(self) {\n                Ok(addr) => {\n                    if addr == peer {\n                        debug!(\"Unix FD to: {peer} is reusable\");\n                        true\n                    } else {\n                        error!(\"Crit: unix FD mismatch: fd: {fd:?}, peer: {peer}, addr: {addr}\",);\n                        false\n                    }\n                }\n                Err(e) => {\n                    error!(\"Bad addr: {self:?}, error: {e:?}\");\n                    false\n                }\n            },\n            Err(e) => {\n                error!(\"Idle unix connection is broken: {e:?}\");\n                false\n            }\n        }\n    }\n}\n\n#[cfg(unix)]\nimpl ConnFdReusable for InetSocketAddr {\n    fn check_fd_match<V: AsRawFd>(&self, fd: V) -> bool {\n        let fd = fd.as_raw_fd();\n        match getpeername::<SockaddrStorage>(fd) {\n            Ok(peer) => {\n                const ZERO: IpAddr = IpAddr::V4(Ipv4Addr::new(0, 0, 0, 0));\n                if self.ip() == ZERO {\n                    // https://www.rfc-editor.org/rfc/rfc1122.html#section-3.2.1.3\n                    // 0.0.0.0 should only be used as source IP not destination\n                    // However in some systems this destination IP is mapped to 127.0.0.1.\n                    // We just skip this check here to avoid false positive mismatch.\n                    return true;\n                }\n                let addr = SockaddrStorage::from(*self);\n                if addr == peer {\n                    debug!(\"Inet FD to: {addr} is reusable\");\n                    true\n                } else {\n                    error!(\"Crit: FD mismatch: fd: {fd:?}, addr: {addr}, peer: {peer}\",);\n                    false\n                }\n            }\n            Err(e) => {\n                debug!(\"Idle connection is broken: {e:?}\");\n                false\n            }\n        }\n    }\n}\n\n#[cfg(windows)]\nimpl ConnSockReusable for InetSocketAddr {\n    fn check_sock_match<V: AsRawSocket>(&self, sock: V) -> bool {\n        let sock = sock.as_raw_socket();\n        match windows::peer_addr(sock) {\n            Ok(peer) => {\n                const ZERO: IpAddr = IpAddr::V4(Ipv4Addr::new(0, 0, 0, 0));\n                if self.ip() == ZERO {\n                    // https://www.rfc-editor.org/rfc/rfc1122.html#section-3.2.1.3\n                    // 0.0.0.0 should only be used as source IP not destination\n                    // However in some systems this destination IP is mapped to 127.0.0.1.\n                    // We just skip this check here to avoid false positive mismatch.\n                    return true;\n                }\n                if self == &peer {\n                    debug!(\"Inet FD to: {self} is reusable\");\n                    true\n                } else {\n                    error!(\"Crit: FD mismatch: fd: {sock:?}, addr: {self}, peer: {peer}\",);\n                    false\n                }\n            }\n            Err(e) => {\n                debug!(\"Idle connection is broken: {e:?}\");\n                false\n            }\n        }\n    }\n}\n"
  },
  {
    "path": "pingora-core/src/protocols/raw_connect.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! CONNECT protocol over http 1.1 via raw Unix domain socket\n//!\n//! This mod implements the most rudimentary CONNECT client over raw stream.\n//! The idea is to yield raw stream once the CONNECT handshake is complete\n//! so that the protocol encapsulated can use the stream directly.\n//! This idea only works for CONNECT over HTTP 1.1 and localhost (or where the server is close by).\n\nuse std::any::Any;\n\nuse super::http::v1::client::HttpSession;\nuse super::http::v1::common::*;\nuse super::Stream;\n\nuse bytes::{BufMut, BytesMut};\nuse http::request::Parts as ReqHeader;\nuse http::Version;\nuse pingora_error::{Error, ErrorType::*, OrErr, Result};\nuse pingora_http::ResponseHeader;\nuse tokio::io::AsyncWriteExt;\n\n/// Try to establish a CONNECT proxy via the given `stream`.\n///\n/// `request_header` should include the necessary request headers for the CONNECT protocol.\n///\n/// When successful, a [`Stream`] will be returned which is the established CONNECT proxy connection.\npub async fn connect<P>(\n    stream: Stream,\n    request_header: &ReqHeader,\n    peer: &P,\n) -> Result<(Stream, ProxyDigest)>\nwhere\n    P: crate::upstreams::peer::Peer,\n{\n    let mut http = HttpSession::new(stream);\n\n    // We write to stream directly because HttpSession doesn't write req header in auth form\n    let to_wire = http_req_header_to_wire_auth_form(request_header);\n    http.underlying_stream\n        .write_all(to_wire.as_ref())\n        .await\n        .or_err(WriteError, \"while writing request headers\")?;\n    http.underlying_stream\n        .flush()\n        .await\n        .or_err(WriteError, \"while flushing request headers\")?;\n\n    // TODO: set http.read_timeout\n    let resp_header = http.read_resp_header_parts().await?;\n    Ok((\n        http.underlying_stream,\n        validate_connect_response(resp_header, peer, request_header)?,\n    ))\n}\n\n/// Generate the CONNECT header for the given destination\npub fn generate_connect_header<'a, H, S>(\n    host: &str,\n    port: u16,\n    headers: H,\n) -> Result<Box<ReqHeader>>\nwhere\n    S: AsRef<[u8]>,\n    H: Iterator<Item = (S, &'a Vec<u8>)>,\n{\n    // TODO: valid that host doesn't have port\n\n    let authority = if host.parse::<std::net::Ipv6Addr>().is_ok() {\n        format!(\"[{host}]:{port}\")\n    } else {\n        format!(\"{host}:{port}\")\n    };\n\n    let req = http::request::Builder::new()\n        .version(http::Version::HTTP_11)\n        .method(http::method::Method::CONNECT)\n        .uri(format!(\"https://{authority}/\")) // scheme doesn't matter\n        .header(http::header::HOST, &authority);\n\n    let (mut req, _) = match req.body(()) {\n        Ok(r) => r.into_parts(),\n        Err(e) => {\n            return Err(e).or_err(InvalidHTTPHeader, \"Invalid CONNECT request\");\n        }\n    };\n\n    for (k, v) in headers {\n        let header_name = http::header::HeaderName::from_bytes(k.as_ref())\n            .or_err(InvalidHTTPHeader, \"Invalid CONNECT request\")?;\n        let header_value = http::header::HeaderValue::from_bytes(v.as_slice())\n            .or_err(InvalidHTTPHeader, \"Invalid CONNECT request\")?;\n        req.headers.insert(header_name, header_value);\n    }\n\n    Ok(Box::new(req))\n}\n\n/// The information about the CONNECT proxy.\n#[derive(Debug)]\npub struct ProxyDigest {\n    /// The response header the proxy returns\n    pub response: Box<ResponseHeader>,\n    /// Optional arbitrary data.\n    pub user_data: Option<Box<dyn Any + Send + Sync>>,\n}\n\nimpl ProxyDigest {\n    pub fn new(\n        response: Box<ResponseHeader>,\n        user_data: Option<Box<dyn Any + Send + Sync>>,\n    ) -> Self {\n        ProxyDigest {\n            response,\n            user_data,\n        }\n    }\n}\n\n/// The error returned when the CONNECT proxy fails to establish.\n#[derive(Debug)]\npub struct ConnectProxyError {\n    /// The response header the proxy returns\n    pub response: Box<ResponseHeader>,\n}\n\nimpl ConnectProxyError {\n    pub fn boxed_new(response: Box<ResponseHeader>) -> Box<Self> {\n        Box::new(ConnectProxyError { response })\n    }\n}\n\nimpl std::fmt::Display for ConnectProxyError {\n    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {\n        const PROXY_STATUS: &str = \"proxy-status\";\n\n        let reason = self\n            .response\n            .headers\n            .get(PROXY_STATUS)\n            .and_then(|s| s.to_str().ok())\n            .unwrap_or(\"missing proxy-status header value\");\n        write!(\n            f,\n            \"Failed CONNECT Response: status {}, proxy-status {reason}\",\n            &self.response.status\n        )\n    }\n}\n\nimpl std::error::Error for ConnectProxyError {}\n\n#[inline]\nfn http_req_header_to_wire_auth_form(req: &ReqHeader) -> BytesMut {\n    let mut buf = BytesMut::with_capacity(512);\n\n    // Request-Line\n    let method = req.method.as_str().as_bytes();\n    buf.put_slice(method);\n    buf.put_u8(b' ');\n    // NOTE: CONNECT doesn't need URI path so we just skip that\n    if let Some(path) = req.uri.authority() {\n        buf.put_slice(path.as_str().as_bytes());\n    }\n    buf.put_u8(b' ');\n\n    let version = match req.version {\n        Version::HTTP_09 => \"HTTP/0.9\",\n        Version::HTTP_10 => \"HTTP/1.0\",\n        Version::HTTP_11 => \"HTTP/1.1\",\n        _ => \"HTTP/0.9\",\n    };\n    buf.put_slice(version.as_bytes());\n    buf.put_slice(CRLF);\n\n    // headers\n    let headers = &req.headers;\n    for (key, value) in headers.iter() {\n        buf.put_slice(key.as_ref());\n        buf.put_slice(HEADER_KV_DELIMITER);\n        buf.put_slice(value.as_ref());\n        buf.put_slice(CRLF);\n    }\n\n    buf.put_slice(CRLF);\n    buf\n}\n\n#[inline]\nfn validate_connect_response<P>(\n    resp: Box<ResponseHeader>,\n    peer: &P,\n    req: &ReqHeader,\n) -> Result<ProxyDigest>\nwhere\n    P: crate::upstreams::peer::Peer,\n{\n    if !resp.status.is_success() {\n        return Error::e_because(\n            ConnectProxyFailure,\n            \"None 2xx code\",\n            ConnectProxyError::boxed_new(resp),\n        );\n    }\n\n    // Checking Content-Length and Transfer-Encoding is optional because we already ignore them.\n    // We choose to do so because we want to be strict for internal use of CONNECT.\n    // Ignore Content-Length header because our internal CONNECT server is coded to send it.\n    if resp.headers.get(http::header::TRANSFER_ENCODING).is_some() {\n        return Error::e_because(\n            ConnectProxyFailure,\n            \"Invalid Transfer-Encoding presents\",\n            ConnectProxyError::boxed_new(resp),\n        );\n    }\n\n    let user_data = peer\n        .proxy_digest_user_data_hook()\n        .and_then(|hook| hook(req, &resp));\n    Ok(ProxyDigest::new(resp, user_data))\n}\n\n#[cfg(test)]\nmod test_sync {\n    use super::*;\n    use std::collections::BTreeMap;\n    use tokio_test::io::Builder;\n\n    #[test]\n    fn test_generate_connect_header() {\n        let mut headers = BTreeMap::new();\n        headers.insert(String::from(\"foo\"), b\"bar\".to_vec());\n        let req = generate_connect_header(\"pingora.org\", 123, headers.iter()).unwrap();\n\n        assert_eq!(req.method, http::method::Method::CONNECT);\n        assert_eq!(req.uri.authority().unwrap(), \"pingora.org:123\");\n        assert_eq!(req.headers.get(\"Host\").unwrap(), \"pingora.org:123\");\n        assert_eq!(req.headers.get(\"foo\").unwrap(), \"bar\");\n    }\n\n    #[test]\n    fn test_generate_connect_header_ipv6() {\n        let mut headers = BTreeMap::new();\n        headers.insert(String::from(\"foo\"), b\"bar\".to_vec());\n        let req = generate_connect_header(\"::1\", 123, headers.iter()).unwrap();\n\n        assert_eq!(req.method, http::method::Method::CONNECT);\n        assert_eq!(req.uri.authority().unwrap(), \"[::1]:123\");\n        assert_eq!(req.headers.get(\"Host\").unwrap(), \"[::1]:123\");\n        assert_eq!(req.headers.get(\"foo\").unwrap(), \"bar\");\n    }\n\n    #[test]\n    fn test_request_to_wire_auth_form() {\n        let new_request = http::Request::builder()\n            .method(\"CONNECT\")\n            .uri(\"https://pingora.org:123/\")\n            .header(\"Foo\", \"Bar\")\n            .body(())\n            .unwrap();\n        let (new_request, _) = new_request.into_parts();\n        let wire = http_req_header_to_wire_auth_form(&new_request);\n        assert_eq!(\n            &b\"CONNECT pingora.org:123 HTTP/1.1\\r\\nfoo: Bar\\r\\n\\r\\n\"[..],\n            &wire\n        );\n    }\n\n    #[test]\n    fn test_validate_connect_response() {\n        use crate::upstreams::peer::BasicPeer;\n\n        struct DummyUserData {\n            some_num: i32,\n            some_string: String,\n        }\n\n        let peer_no_data = BasicPeer::new(\"127.0.0.1:80\");\n        let mut peer_with_data = peer_no_data.clone();\n        peer_with_data.options.proxy_digest_user_data_hook = Some(std::sync::Arc::new(\n            |_req: &http::request::Parts, _resp: &pingora_http::ResponseHeader| {\n                Some(Box::new(DummyUserData {\n                    some_num: 42,\n                    some_string: \"test\".to_string(),\n                }) as Box<dyn std::any::Any + Send + Sync>)\n            },\n        ));\n\n        let request = http::Request::builder()\n            .method(\"CONNECT\")\n            .uri(\"https://example.com:443/\")\n            .body(())\n            .unwrap();\n        let (req_header, _) = request.into_parts();\n\n        let resp = ResponseHeader::build(200, None).unwrap();\n        let proxy_digest =\n            validate_connect_response(Box::new(resp), &peer_with_data, &req_header).unwrap();\n        assert!(proxy_digest.user_data.is_some());\n        let user_data = proxy_digest\n            .user_data\n            .as_ref()\n            .unwrap()\n            .downcast_ref::<DummyUserData>()\n            .unwrap();\n        assert_eq!(user_data.some_num, 42);\n        assert_eq!(user_data.some_string, \"test\");\n\n        let resp = ResponseHeader::build(200, None).unwrap();\n        let proxy_digest =\n            validate_connect_response(Box::new(resp), &peer_no_data, &req_header).unwrap();\n        assert!(proxy_digest.user_data.is_none());\n\n        let resp = ResponseHeader::build(404, None).unwrap();\n        assert!(validate_connect_response(Box::new(resp), &peer_with_data, &req_header).is_err());\n\n        let mut resp = ResponseHeader::build(200, None).unwrap();\n        resp.append_header(\"content-length\", 0).unwrap();\n        assert!(validate_connect_response(Box::new(resp), &peer_no_data, &req_header).is_ok());\n\n        let mut resp = ResponseHeader::build(200, None).unwrap();\n        resp.append_header(\"transfer-encoding\", 0).unwrap();\n        assert!(validate_connect_response(Box::new(resp), &peer_no_data, &req_header).is_err());\n    }\n\n    #[tokio::test]\n    async fn test_connect_write_request() {\n        use crate::upstreams::peer::BasicPeer;\n\n        let wire = b\"CONNECT pingora.org:123 HTTP/1.1\\r\\nhost: pingora.org:123\\r\\n\\r\\n\";\n        let mock_io = Box::new(Builder::new().write(wire).build());\n\n        let headers: BTreeMap<String, Vec<u8>> = BTreeMap::new();\n        let req = generate_connect_header(\"pingora.org\", 123, headers.iter()).unwrap();\n        let peer = BasicPeer::new(\"127.0.0.1:123\");\n        // ConnectionClosed\n        assert!(connect(mock_io, &req, &peer).await.is_err());\n\n        let to_wire = b\"CONNECT pingora.org:123 HTTP/1.1\\r\\nhost: pingora.org:123\\r\\n\\r\\n\";\n        let from_wire = b\"HTTP/1.1 200 OK\\r\\n\\r\\n\";\n        let mock_io = Box::new(Builder::new().write(to_wire).read(from_wire).build());\n\n        let req = generate_connect_header(\"pingora.org\", 123, headers.iter()).unwrap();\n        let result = connect(mock_io, &req, &peer).await;\n        assert!(result.is_ok());\n    }\n}\n"
  },
  {
    "path": "pingora-core/src/protocols/tls/boringssl_openssl/client.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! TLS client specific implementation\n\nuse crate::protocols::raw_connect::ProxyDigest;\nuse crate::protocols::tls::SslStream;\nuse crate::protocols::{\n    GetProxyDigest, GetSocketDigest, GetTimingDigest, SocketDigest, TimingDigest, IO,\n};\nuse crate::tls::{ssl, ssl::ConnectConfiguration, ssl::SslRef, ssl_sys::X509_V_ERR_INVALID_CALL};\n\nuse pingora_error::{Error, ErrorType::*, OrErr, Result};\nuse std::any::Any;\nuse std::sync::Arc;\nuse std::time::Duration;\n\n/// Perform the TLS handshake for the given connection with the given configuration\npub async fn handshake<S: IO>(\n    conn_config: ConnectConfiguration,\n    domain: &str,\n    io: S,\n    complete_hook: Option<Arc<dyn Fn(&SslRef) -> Option<Arc<dyn Any + Send + Sync>> + Send + Sync>>,\n) -> Result<SslStream<S>> {\n    let ssl = conn_config\n        .into_ssl(domain)\n        .explain_err(TLSHandshakeFailure, |e| format!(\"ssl config error: {e}\"))?;\n    let mut stream = SslStream::new(ssl, io)\n        .explain_err(TLSHandshakeFailure, |e| format!(\"ssl stream error: {e}\"))?;\n    let handshake_result = stream.connect().await;\n    match handshake_result {\n        Ok(()) => {\n            if let Some(hook) = complete_hook {\n                if let Some(extension) = hook(stream.ssl()) {\n                    if let Some(digest_mut) = stream.ssl_digest_mut() {\n                        digest_mut.extension.set(extension);\n                    }\n                }\n            }\n            Ok(stream)\n        }\n        Err(e) => {\n            let context = format!(\"TLS connect() failed: {e}, SNI: {domain}\");\n            match e.code() {\n                ssl::ErrorCode::SSL => {\n                    // Unify the return type of `verify_result` for openssl\n                    #[cfg(not(feature = \"boringssl\"))]\n                    fn verify_result<S>(stream: SslStream<S>) -> Result<(), i32> {\n                        match stream.ssl().verify_result().as_raw() {\n                            crate::tls::ssl_sys::X509_V_OK => Ok(()),\n                            e => Err(e),\n                        }\n                    }\n\n                    // Unify the return type of `verify_result` for boringssl\n                    #[cfg(feature = \"boringssl\")]\n                    fn verify_result<S>(stream: SslStream<S>) -> Result<(), i32> {\n                        stream.ssl().verify_result().map_err(|e| e.as_raw())\n                    }\n\n                    match verify_result(stream) {\n                        Ok(()) => Error::e_explain(TLSHandshakeFailure, context),\n                        // X509_V_ERR_INVALID_CALL in case verify result was never set\n                        Err(X509_V_ERR_INVALID_CALL) => {\n                            Error::e_explain(TLSHandshakeFailure, context)\n                        }\n                        _ => Error::e_explain(InvalidCert, context),\n                    }\n                }\n                /* likely network error, but still mark as TLS error */\n                _ => Error::e_explain(TLSHandshakeFailure, context),\n            }\n        }\n    }\n}\n\nimpl<S> GetTimingDigest for SslStream<S>\nwhere\n    S: GetTimingDigest,\n{\n    fn get_timing_digest(&self) -> Vec<Option<TimingDigest>> {\n        let mut ts_vec = self.get_ref().get_timing_digest();\n        ts_vec.push(Some(self.timing.clone()));\n        ts_vec\n    }\n    fn get_read_pending_time(&self) -> Duration {\n        self.get_ref().get_read_pending_time()\n    }\n\n    fn get_write_pending_time(&self) -> Duration {\n        self.get_ref().get_write_pending_time()\n    }\n}\n\nimpl<S> GetProxyDigest for SslStream<S>\nwhere\n    S: GetProxyDigest,\n{\n    fn get_proxy_digest(&self) -> Option<Arc<ProxyDigest>> {\n        self.get_ref().get_proxy_digest()\n    }\n}\n\nimpl<S> GetSocketDigest for SslStream<S>\nwhere\n    S: GetSocketDigest,\n{\n    fn get_socket_digest(&self) -> Option<Arc<SocketDigest>> {\n        self.get_ref().get_socket_digest()\n    }\n    fn set_socket_digest(&mut self, socket_digest: SocketDigest) {\n        self.get_mut().set_socket_digest(socket_digest)\n    }\n}\n"
  },
  {
    "path": "pingora-core/src/protocols/tls/boringssl_openssl/mod.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\npub mod client;\npub mod server;\nmod stream;\n\n#[cfg(feature = \"boringssl\")]\nuse pingora_boringssl as ssl_lib;\n\n#[cfg(feature = \"openssl\")]\nuse pingora_openssl as ssl_lib;\n\nuse ssl_lib::{ssl::SslRef, x509::X509};\npub use stream::*;\n\npub type TlsRef = SslRef;\npub type CaType = Box<[X509]>;\n"
  },
  {
    "path": "pingora-core/src/protocols/tls/boringssl_openssl/server.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! TLS server specific implementation\n\nuse crate::listeners::TlsAcceptCallbacks;\nuse crate::protocols::tls::SslStream;\nuse crate::protocols::{Shutdown, IO};\nuse crate::tls::ext;\nuse crate::tls::ext::ssl_from_acceptor;\nuse crate::tls::ssl;\nuse crate::tls::ssl::SslAcceptor;\n\nuse async_trait::async_trait;\nuse log::warn;\nuse pingora_error::{ErrorType::*, OrErr, Result};\nuse std::pin::Pin;\nuse tokio::io::{AsyncRead, AsyncWrite, AsyncWriteExt};\n\n/// Prepare a TLS stream for handshake\npub fn prepare_tls_stream<S: IO>(ssl_acceptor: &SslAcceptor, io: S) -> Result<SslStream<S>> {\n    let ssl = ssl_from_acceptor(ssl_acceptor)\n        .explain_err(TLSHandshakeFailure, |e| format!(\"ssl_acceptor error: {e}\"))?;\n    SslStream::new(ssl, io).explain_err(TLSHandshakeFailure, |e| format!(\"ssl stream error: {e}\"))\n}\n\n/// Perform TLS handshake for the given connection with the given configuration\npub async fn handshake<S: IO>(ssl_acceptor: &SslAcceptor, io: S) -> Result<SslStream<S>> {\n    let mut stream = prepare_tls_stream(ssl_acceptor, io)?;\n    stream\n        .accept()\n        .await\n        .explain_err(TLSHandshakeFailure, |e| format!(\"TLS accept() failed: {e}\"))?;\n    Ok(stream)\n}\n\n/// Perform TLS handshake for the given connection with the given configuration and callbacks\npub async fn handshake_with_callback<S: IO>(\n    ssl_acceptor: &SslAcceptor,\n    io: S,\n    callbacks: &TlsAcceptCallbacks,\n) -> Result<SslStream<S>> {\n    let mut tls_stream = prepare_tls_stream(ssl_acceptor, io)?;\n    let done = Pin::new(&mut tls_stream)\n        .start_accept()\n        .await\n        .explain_err(TLSHandshakeFailure, |e| format!(\"TLS accept() failed: {e}\"))?;\n    if !done {\n        // safety: we do hold a mut ref of tls_stream\n        let ssl_mut = unsafe { ext::ssl_mut(tls_stream.ssl()) };\n        callbacks.certificate_callback(ssl_mut).await;\n        Pin::new(&mut tls_stream)\n            .resume_accept()\n            .await\n            .explain_err(TLSHandshakeFailure, |e| format!(\"TLS accept() failed: {e}\"))?;\n    }\n    {\n        let ssl = tls_stream.ssl();\n        if let Some(extension) = callbacks.handshake_complete_callback(ssl).await {\n            if let Some(digest_mut) = tls_stream.ssl_digest_mut() {\n                digest_mut.extension.set(extension);\n            }\n        }\n    }\n    Ok(tls_stream)\n}\n\n#[async_trait]\nimpl<S> Shutdown for SslStream<S>\nwhere\n    S: AsyncRead + AsyncWrite + Sync + Unpin + Send,\n{\n    async fn shutdown(&mut self) {\n        match <Self as AsyncWriteExt>::shutdown(self).await {\n            Ok(()) => {}\n            Err(e) => {\n                warn!(\"TLS shutdown failed, {e}\");\n            }\n        }\n    }\n}\n\n/// Resumable TLS server side handshake.\n#[async_trait]\npub trait ResumableAccept {\n    /// Start a resumable TLS accept handshake.\n    ///\n    /// * `Ok(true)` when the handshake is finished\n    /// * `Ok(false)`` when the handshake is paused midway\n    ///\n    /// For now, the accept will only pause when a certificate is needed.\n    async fn start_accept(self: Pin<&mut Self>) -> Result<bool, ssl::Error>;\n\n    /// Continue the TLS handshake\n    ///\n    /// This function should be called after the certificate is provided.\n    async fn resume_accept(self: Pin<&mut Self>) -> Result<(), ssl::Error>;\n}\n\n#[async_trait]\nimpl<S: AsyncRead + AsyncWrite + Send + Unpin> ResumableAccept for SslStream<S> {\n    async fn start_accept(mut self: Pin<&mut Self>) -> Result<bool, ssl::Error> {\n        // safety: &mut self\n        let ssl_mut = unsafe { ext::ssl_mut(self.ssl()) };\n        ext::suspend_when_need_ssl_cert(ssl_mut);\n        let res = self.accept().await;\n\n        match res {\n            Ok(()) => Ok(true),\n            Err(e) => {\n                if ext::is_suspended_for_cert(&e) {\n                    Ok(false)\n                } else {\n                    Err(e)\n                }\n            }\n        }\n    }\n\n    async fn resume_accept(mut self: Pin<&mut Self>) -> Result<(), ssl::Error> {\n        // safety: &mut ssl\n        let ssl_mut = unsafe { ext::ssl_mut(self.ssl()) };\n        ext::unblock_ssl_cert(ssl_mut);\n        self.accept().await\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use super::handshake_with_callback;\n\n    use crate::listeners::{TlsAccept, TlsAcceptCallbacks};\n    use crate::protocols::tls::SslStream;\n    use crate::protocols::tls::TlsRef;\n    use crate::tls::ext;\n    use crate::tls::ssl;\n\n    use async_trait::async_trait;\n    use std::pin::Pin;\n    use std::sync::Arc;\n    use tokio::io::DuplexStream;\n\n    async fn client_task(client: DuplexStream) {\n        use tokio::io::AsyncReadExt;\n        let ssl_context = ssl::SslContext::builder(ssl::SslMethod::tls())\n            .unwrap()\n            .build();\n        let mut ssl = ssl::Ssl::new(&ssl_context).unwrap();\n        ssl.set_hostname(\"pingora.org\").unwrap();\n        ssl.set_verify(ssl::SslVerifyMode::NONE); // we don have a valid cert\n        let mut stream = SslStream::new(ssl, client).unwrap();\n        Pin::new(&mut stream).connect().await.unwrap();\n        let mut buf = [0; 1];\n        let _ = stream.read(&mut buf).await;\n    }\n\n    #[tokio::test]\n    #[cfg(feature = \"any_tls\")]\n    async fn test_async_cert() {\n        let acceptor = ssl::SslAcceptor::mozilla_intermediate_v5(ssl::SslMethod::tls())\n            .unwrap()\n            .build();\n\n        struct Callback;\n        #[async_trait]\n        impl TlsAccept for Callback {\n            async fn certificate_callback(&self, ssl: &mut TlsRef) -> () {\n                assert_eq!(\n                    ssl.servername(ssl::NameType::HOST_NAME).unwrap(),\n                    \"pingora.org\"\n                );\n                let cert = format!(\"{}/tests/keys/server.crt\", env!(\"CARGO_MANIFEST_DIR\"));\n                let key = format!(\"{}/tests/keys/key.pem\", env!(\"CARGO_MANIFEST_DIR\"));\n\n                let cert_bytes = std::fs::read(cert).unwrap();\n                let cert = crate::tls::x509::X509::from_pem(&cert_bytes).unwrap();\n\n                let key_bytes = std::fs::read(key).unwrap();\n                let key = crate::tls::pkey::PKey::private_key_from_pem(&key_bytes).unwrap();\n                ext::ssl_use_certificate(ssl, &cert).unwrap();\n                ext::ssl_use_private_key(ssl, &key).unwrap();\n            }\n        }\n\n        let cb: TlsAcceptCallbacks = Box::new(Callback);\n\n        let (client, server) = tokio::io::duplex(1024);\n\n        tokio::spawn(client_task(client));\n\n        handshake_with_callback(&acceptor, server, &cb)\n            .await\n            .unwrap();\n    }\n\n    #[tokio::test]\n    #[cfg(feature = \"openssl_derived\")]\n    async fn test_handshake_complete_callback() {\n        use crate::tls::ssl::SslFiletype;\n\n        let cert = format!(\"{}/tests/keys/server.crt\", env!(\"CARGO_MANIFEST_DIR\"));\n        let key = format!(\"{}/tests/keys/key.pem\", env!(\"CARGO_MANIFEST_DIR\"));\n\n        let acceptor = {\n            let mut builder =\n                ssl::SslAcceptor::mozilla_intermediate_v5(ssl::SslMethod::tls()).unwrap();\n            builder.set_certificate_chain_file(cert).unwrap();\n            builder.set_private_key_file(key, SslFiletype::PEM).unwrap();\n            builder.build()\n        };\n\n        struct Sni(String);\n        struct Callback;\n        #[async_trait]\n        impl TlsAccept for Callback {\n            async fn handshake_complete_callback(\n                &self,\n                ssl: &TlsRef,\n            ) -> Option<Arc<dyn std::any::Any + Send + Sync>> {\n                let sni = ssl.servername(ssl::NameType::HOST_NAME)?.to_string();\n                Some(Arc::new(Sni(sni)))\n            }\n        }\n\n        let cb: TlsAcceptCallbacks = Box::new(Callback);\n\n        let (client, server) = tokio::io::duplex(1024);\n\n        tokio::spawn(client_task(client));\n\n        let stream = handshake_with_callback(&acceptor, server, &cb)\n            .await\n            .unwrap();\n        let ssl_digest = stream.ssl_digest().unwrap();\n        let sni = ssl_digest.extension.get::<Sni>().unwrap();\n        assert_eq!(sni.0, \"pingora.org\");\n    }\n}\n"
  },
  {
    "path": "pingora-core/src/protocols/tls/boringssl_openssl/stream.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\nuse crate::protocols::digest::TimingDigest;\nuse crate::protocols::tls::{SslDigest, ALPN};\nuse crate::protocols::{Peek, Ssl, UniqueID, UniqueIDType};\nuse crate::tls::{self, ssl, tokio_ssl::SslStream as InnerSsl};\nuse crate::utils::tls::{get_organization, get_serial};\nuse log::warn;\nuse pingora_error::{ErrorType::*, OrErr, Result};\nuse std::pin::Pin;\nuse std::sync::Arc;\nuse std::task::{Context, Poll};\nuse std::time::SystemTime;\nuse tokio::io::{self, AsyncRead, AsyncWrite, ReadBuf};\n\n#[cfg(feature = \"boringssl\")]\nuse pingora_boringssl as ssl_lib;\n\n#[cfg(feature = \"openssl\")]\nuse pingora_openssl as ssl_lib;\n\nuse ssl_lib::{hash::MessageDigest, ssl::SslRef};\n\n/// The TLS connection\n#[derive(Debug)]\npub struct SslStream<T> {\n    ssl: InnerSsl<T>,\n    digest: Option<Arc<SslDigest>>,\n    pub(super) timing: TimingDigest,\n}\n\nimpl<T> SslStream<T>\nwhere\n    T: AsyncRead + AsyncWrite + std::marker::Unpin,\n{\n    /// Create a new TLS connection from the given `stream`\n    ///\n    /// The caller needs to perform [`Self::connect()`] or [`Self::accept()`] to perform TLS\n    /// handshake after.\n    pub fn new(ssl: ssl::Ssl, stream: T) -> Result<Self> {\n        let ssl = InnerSsl::new(ssl, stream)\n            .explain_err(TLSHandshakeFailure, |e| format!(\"ssl stream error: {e}\"))?;\n\n        Ok(SslStream {\n            ssl,\n            digest: None,\n            timing: Default::default(),\n        })\n    }\n\n    /// Connect to the remote TLS server as a client\n    pub async fn connect(&mut self) -> Result<(), ssl::Error> {\n        Self::clear_error();\n        Pin::new(&mut self.ssl).connect().await?;\n        self.timing.established_ts = SystemTime::now();\n        self.digest = Some(Arc::new(SslDigest::from_ssl(self.ssl())));\n        Ok(())\n    }\n\n    /// Finish the TLS handshake from client as a server\n    pub async fn accept(&mut self) -> Result<(), ssl::Error> {\n        Self::clear_error();\n        Pin::new(&mut self.ssl).accept().await?;\n        self.timing.established_ts = SystemTime::now();\n        self.digest = Some(Arc::new(SslDigest::from_ssl(self.ssl())));\n        Ok(())\n    }\n\n    #[inline]\n    fn clear_error() {\n        let errs = tls::error::ErrorStack::get();\n        if !errs.errors().is_empty() {\n            warn!(\"Clearing dirty TLS error stack: {}\", errs);\n        }\n    }\n}\n\nimpl<T> SslStream<T> {\n    pub fn ssl_digest(&self) -> Option<Arc<SslDigest>> {\n        self.digest.clone()\n    }\n\n    /// Attempts to obtain a mutable reference to the SslDigest.\n    /// This method returns `None` if the SslDigest is currently held by other references.\n    pub(crate) fn ssl_digest_mut(&mut self) -> Option<&mut SslDigest> {\n        Arc::get_mut(self.digest.as_mut()?)\n    }\n}\n\nuse std::ops::{Deref, DerefMut};\n\nimpl<T> Deref for SslStream<T> {\n    type Target = InnerSsl<T>;\n\n    fn deref(&self) -> &Self::Target {\n        &self.ssl\n    }\n}\n\nimpl<T> DerefMut for SslStream<T> {\n    fn deref_mut(&mut self) -> &mut Self::Target {\n        &mut self.ssl\n    }\n}\n\nimpl<T> AsyncRead for SslStream<T>\nwhere\n    T: AsyncRead + AsyncWrite + Unpin,\n{\n    fn poll_read(\n        mut self: Pin<&mut Self>,\n        cx: &mut Context<'_>,\n        buf: &mut ReadBuf<'_>,\n    ) -> Poll<io::Result<()>> {\n        Self::clear_error();\n        Pin::new(&mut self.ssl).poll_read(cx, buf)\n    }\n}\n\nimpl<T> AsyncWrite for SslStream<T>\nwhere\n    T: AsyncRead + AsyncWrite + Unpin,\n{\n    fn poll_write(\n        mut self: Pin<&mut Self>,\n        cx: &mut Context,\n        buf: &[u8],\n    ) -> Poll<io::Result<usize>> {\n        Self::clear_error();\n        Pin::new(&mut self.ssl).poll_write(cx, buf)\n    }\n\n    fn poll_flush(mut self: Pin<&mut Self>, cx: &mut Context) -> Poll<io::Result<()>> {\n        Self::clear_error();\n        Pin::new(&mut self.ssl).poll_flush(cx)\n    }\n\n    fn poll_shutdown(mut self: Pin<&mut Self>, cx: &mut Context) -> Poll<io::Result<()>> {\n        Self::clear_error();\n        Pin::new(&mut self.ssl).poll_shutdown(cx)\n    }\n\n    fn poll_write_vectored(\n        mut self: Pin<&mut Self>,\n        cx: &mut Context<'_>,\n        bufs: &[std::io::IoSlice<'_>],\n    ) -> Poll<io::Result<usize>> {\n        Self::clear_error();\n        Pin::new(&mut self.ssl).poll_write_vectored(cx, bufs)\n    }\n\n    fn is_write_vectored(&self) -> bool {\n        true\n    }\n}\n\nimpl<T> UniqueID for SslStream<T>\nwhere\n    T: UniqueID,\n{\n    fn id(&self) -> UniqueIDType {\n        self.ssl.get_ref().id()\n    }\n}\n\nimpl<T> Ssl for SslStream<T> {\n    fn get_ssl(&self) -> Option<&ssl::SslRef> {\n        Some(self.ssl())\n    }\n\n    fn get_ssl_digest(&self) -> Option<Arc<SslDigest>> {\n        self.ssl_digest()\n    }\n\n    /// Return selected ALPN if any\n    fn selected_alpn_proto(&self) -> Option<ALPN> {\n        let ssl = self.get_ssl()?;\n        ALPN::from_wire_selected(ssl.selected_alpn_protocol()?)\n    }\n}\n\nimpl SslDigest {\n    pub fn from_ssl(ssl: &SslRef) -> Self {\n        let cipher = match ssl.current_cipher() {\n            Some(c) => c.name(),\n            None => \"\",\n        };\n\n        let (cert_digest, org, sn) = match ssl.peer_certificate() {\n            Some(cert) => {\n                let cert_digest = match cert.digest(MessageDigest::sha256()) {\n                    Ok(c) => c.as_ref().to_vec(),\n                    Err(_) => Vec::new(),\n                };\n                (cert_digest, get_organization(&cert), get_serial(&cert).ok())\n            }\n            None => (Vec::new(), None, None),\n        };\n\n        SslDigest::new(cipher, ssl.version_str(), org, sn, cert_digest)\n    }\n}\n\n// TODO: implement Peek if needed\nimpl<T> Peek for SslStream<T> {}\n"
  },
  {
    "path": "pingora-core/src/protocols/tls/digest.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! TLS information from the TLS connection\n\nuse std::any::Any;\nuse std::borrow::Cow;\nuse std::sync::Arc;\n\n/// The TLS connection information\n#[derive(Clone, Debug)]\npub struct SslDigest {\n    /// The cipher used\n    pub cipher: Cow<'static, str>,\n    /// The TLS version of this connection\n    pub version: Cow<'static, str>,\n    /// The organization of the peer's certificate\n    pub organization: Option<String>,\n    /// The serial number of the peer's certificate\n    pub serial_number: Option<String>,\n    /// The digest of the peer's certificate\n    pub cert_digest: Vec<u8>,\n    /// The user-defined TLS data\n    pub extension: SslDigestExtension,\n}\n\nimpl SslDigest {\n    /// Create a new SslDigest\n    pub fn new<S>(\n        cipher: S,\n        version: S,\n        organization: Option<String>,\n        serial_number: Option<String>,\n        cert_digest: Vec<u8>,\n    ) -> Self\n    where\n        S: Into<Cow<'static, str>>,\n    {\n        SslDigest {\n            cipher: cipher.into(),\n            version: version.into(),\n            organization,\n            serial_number,\n            cert_digest,\n            extension: SslDigestExtension::default(),\n        }\n    }\n}\n\n/// The user-defined TLS data\n#[derive(Clone, Debug, Default)]\npub struct SslDigestExtension {\n    value: Option<Arc<dyn Any + Send + Sync>>,\n}\n\nimpl SslDigestExtension {\n    /// Retrieves a reference to the user-defined TLS data if it matches the specified type.\n    ///\n    /// Returns `None` if no data has been set or if the data is not of type `T`.\n    pub fn get<T>(&self) -> Option<&T>\n    where\n        T: Send + Sync + 'static,\n    {\n        self.value.as_ref().and_then(|v| v.downcast_ref::<T>())\n    }\n\n    #[allow(dead_code)]\n    pub(crate) fn set(&mut self, value: Arc<dyn Any + Send + Sync>) {\n        self.value = Some(value);\n    }\n}\n"
  },
  {
    "path": "pingora-core/src/protocols/tls/mod.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! The TLS layer implementations\n\npub mod digest;\npub use digest::*;\n\n#[cfg(feature = \"openssl_derived\")]\nmod boringssl_openssl;\n\n#[cfg(feature = \"openssl_derived\")]\npub use boringssl_openssl::*;\n\n#[cfg(feature = \"rustls\")]\nmod rustls;\n\n#[cfg(feature = \"rustls\")]\npub use rustls::*;\n\n#[cfg(feature = \"s2n\")]\nmod s2n;\n\n#[cfg(feature = \"s2n\")]\npub use s2n::*;\n\n#[cfg(not(feature = \"any_tls\"))]\npub mod noop_tls;\n\n#[cfg(not(feature = \"any_tls\"))]\npub use noop_tls::*;\n\n/// Containing type for a user callback to generate extensions for the `SslDigest` upon handshake\n/// completion.\npub type HandshakeCompleteHook = std::sync::Arc<\n    dyn Fn(&TlsRef) -> Option<std::sync::Arc<dyn std::any::Any + Send + Sync>> + Send + Sync,\n>;\n\n/// The protocol for Application-Layer Protocol Negotiation\n#[derive(Hash, Clone, Debug, PartialEq, PartialOrd)]\npub enum ALPN {\n    /// Prefer HTTP/1.1 only\n    H1,\n    /// Prefer HTTP/2 only\n    H2,\n    /// Prefer HTTP/2 over HTTP/1.1\n    H2H1,\n    /// Custom Protocol is stored in wire format (length-prefixed)\n    /// Wire format is precomputed at creation to avoid dangling references\n    Custom(CustomALPN),\n}\n\n/// Represents a Custom ALPN Protocol with a precomputed wire format and header offset.\n#[derive(Hash, Clone, Debug, PartialEq, PartialOrd)]\npub struct CustomALPN {\n    wire: Vec<u8>,\n    header: usize,\n}\n\nimpl CustomALPN {\n    /// Create a new CustomALPN from a protocol byte vector\n    pub fn new(proto: Vec<u8>) -> Self {\n        // Validate before setting\n        assert!(!proto.is_empty(), \"Custom ALPN protocol must not be empty\");\n        // RFC-7301\n        assert!(\n            proto.len() <= 255,\n            \"ALPN protocol name must be 255 bytes or fewer\"\n        );\n\n        match proto.as_slice() {\n            b\"http/1.1\" | b\"h2\" => {\n                panic!(\"Custom ALPN cannot be a reserved protocol (http/1.1 or h2)\")\n            }\n            _ => {}\n        }\n        let mut wire = Vec::with_capacity(1 + proto.len());\n        wire.push(proto.len() as u8);\n        wire.extend_from_slice(&proto);\n\n        Self {\n            wire,\n            header: 1, // Header is always at index 1 since we prefix one length byte\n        }\n    }\n\n    /// Get the custom protocol name as a slice\n    pub fn protocol(&self) -> &[u8] {\n        &self.wire[self.header..]\n    }\n\n    /// Get the wire format used for ALPN negotiation\n    pub fn as_wire(&self) -> &[u8] {\n        &self.wire\n    }\n}\n\nimpl std::fmt::Display for ALPN {\n    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {\n        match self {\n            ALPN::H1 => write!(f, \"H1\"),\n            ALPN::H2 => write!(f, \"H2\"),\n            ALPN::H2H1 => write!(f, \"H2H1\"),\n            ALPN::Custom(custom) => {\n                // extract protocol name, print as UTF-8 if possible, else judt itd raw bytes\n                match std::str::from_utf8(custom.protocol()) {\n                    Ok(s) => write!(f, \"Custom({})\", s),\n                    Err(_) => write!(f, \"Custom({:?})\", custom.protocol()),\n                }\n            }\n        }\n    }\n}\n\nimpl ALPN {\n    /// Create a new ALPN according to the `max` and `min` version constraints\n    pub fn new(max: u8, min: u8) -> Self {\n        if max == 1 {\n            ALPN::H1\n        } else if min == 2 {\n            ALPN::H2\n        } else {\n            ALPN::H2H1\n        }\n    }\n\n    /// Return the max http version this [`ALPN`] allows\n    pub fn get_max_http_version(&self) -> u8 {\n        match self {\n            ALPN::H1 => 1,\n            ALPN::H2 | ALPN::H2H1 => 2,\n            ALPN::Custom(_) => 0,\n        }\n    }\n\n    /// Return the min http version this [`ALPN`] allows\n    pub fn get_min_http_version(&self) -> u8 {\n        match self {\n            ALPN::H1 | ALPN::H2H1 => 1,\n            ALPN::H2 => 2,\n            ALPN::Custom(_) => 0,\n        }\n    }\n\n    #[cfg(feature = \"openssl_derived\")]\n    pub(crate) fn to_wire_preference(&self) -> &[u8] {\n        // https://www.openssl.org/docs/manmaster/man3/SSL_CTX_set_alpn_select_cb.html\n        // \"vector of nonempty, 8-bit length-prefixed, byte strings\"\n        match self {\n            Self::H1 => b\"\\x08http/1.1\",\n            Self::H2 => b\"\\x02h2\",\n            Self::H2H1 => b\"\\x02h2\\x08http/1.1\",\n            Self::Custom(custom) => custom.as_wire(),\n        }\n    }\n\n    #[cfg(feature = \"any_tls\")]\n    pub(crate) fn from_wire_selected(raw: &[u8]) -> Option<Self> {\n        match raw {\n            b\"http/1.1\" => Some(Self::H1),\n            b\"h2\" => Some(Self::H2),\n            _ => Some(Self::Custom(CustomALPN::new(raw.to_vec()))),\n        }\n    }\n\n    #[cfg(feature = \"rustls\")]\n    pub(crate) fn to_wire_protocols(&self) -> Vec<Vec<u8>> {\n        match self {\n            ALPN::H1 => vec![b\"http/1.1\".to_vec()],\n            ALPN::H2 => vec![b\"h2\".to_vec()],\n            ALPN::H2H1 => vec![b\"h2\".to_vec(), b\"http/1.1\".to_vec()],\n            ALPN::Custom(custom) => vec![custom.protocol().to_vec()],\n        }\n    }\n\n    #[cfg(feature = \"s2n\")]\n    pub(crate) fn to_wire_protocols(&self) -> Vec<Vec<u8>> {\n        match self {\n            ALPN::H1 => vec![b\"http/1.1\".to_vec()],\n            ALPN::H2 => vec![b\"h2\".to_vec()],\n            ALPN::H2H1 => vec![b\"h2\".to_vec(), b\"http/1.1\".to_vec()],\n            ALPN::Custom(custom) => vec![custom.protocol().to_vec()],\n        }\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use super::*;\n\n    #[test]\n    fn test_valid_alpn_construction_and_versions() {\n        // Standard Protocols\n        assert_eq!(ALPN::H1.get_min_http_version(), 1);\n        assert_eq!(ALPN::H1.get_max_http_version(), 1);\n\n        assert_eq!(ALPN::H2.get_min_http_version(), 2);\n        assert_eq!(ALPN::H2.get_max_http_version(), 2);\n\n        assert_eq!(ALPN::H2H1.get_min_http_version(), 1);\n        assert_eq!(ALPN::H2H1.get_max_http_version(), 2);\n\n        // Custom Protocol\n        let custom_protocol = ALPN::Custom(CustomALPN::new(\"custom/1.0\".into()));\n        assert_eq!(custom_protocol.get_min_http_version(), 0);\n        assert_eq!(custom_protocol.get_max_http_version(), 0);\n    }\n    #[test]\n    #[should_panic(expected = \"Custom ALPN protocol must not be empty\")]\n    fn test_empty_custom_alpn() {\n        let _ = ALPN::Custom(CustomALPN::new(\"\".into()));\n    }\n    #[test]\n    #[should_panic(expected = \"ALPN protocol name must be 255 bytes or fewer\")]\n    fn test_large_custom_alpn() {\n        let large_alpn = vec![b'a'; 256];\n        let _ = ALPN::Custom(CustomALPN::new(large_alpn));\n    }\n    #[test]\n    #[should_panic(expected = \"Custom ALPN cannot be a reserved protocol (http/1.1 or h2)\")]\n    fn test_custom_h1_alpn() {\n        let _ = ALPN::Custom(CustomALPN::new(\"http/1.1\".into()));\n    }\n    #[test]\n    #[should_panic(expected = \"Custom ALPN cannot be a reserved protocol (http/1.1 or h2)\")]\n    fn test_custom_h2_alpn() {\n        let _ = ALPN::Custom(CustomALPN::new(\"h2\".into()));\n    }\n}\n"
  },
  {
    "path": "pingora-core/src/protocols/tls/noop_tls/mod.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! This is a set of stubs that provides the minimum types to let pingora work\n//! without any tls providers configured\n\npub struct TlsRef;\n\npub type CaType = [CertWrapper];\n\n#[derive(Debug)]\npub struct CertWrapper;\n\nimpl CertWrapper {\n    pub fn not_after(&self) -> &str {\n        \"\"\n    }\n}\n\npub mod connectors {\n    use pingora_error::Result;\n\n    use crate::{\n        connectors::ConnectorOptions,\n        protocols::{ALPN, IO},\n        upstreams::peer::Peer,\n    };\n\n    use super::stream::SslStream;\n\n    #[derive(Clone)]\n    pub struct Connector {\n        pub ctx: TlsConnector,\n    }\n\n    #[derive(Clone)]\n    pub struct TlsConnector;\n\n    pub struct TlsSettings;\n\n    impl Connector {\n        pub fn new(_: Option<ConnectorOptions>) -> Self {\n            Self { ctx: TlsConnector }\n        }\n    }\n\n    pub async fn connect<T, P>(\n        _: T,\n        _: &P,\n        _: Option<ALPN>,\n        _: &TlsConnector,\n    ) -> Result<SslStream<T>>\n    where\n        T: IO,\n        P: Peer + Send + Sync,\n    {\n        Ok(SslStream::default())\n    }\n}\n\npub mod listeners {\n    use pingora_error::Result;\n    use tokio::io::{AsyncRead, AsyncWrite};\n\n    use super::stream::SslStream;\n\n    pub struct Acceptor;\n\n    pub struct TlsSettings;\n\n    impl TlsSettings {\n        pub fn build(&self) -> Acceptor {\n            Acceptor\n        }\n\n        pub fn intermediate(_: &str, _: &str) -> Result<Self> {\n            Ok(Self)\n        }\n\n        pub fn enable_h2(&mut self) {}\n    }\n\n    impl Acceptor {\n        pub async fn tls_handshake<S: AsyncRead + AsyncWrite>(&self, _: S) -> Result<SslStream<S>> {\n            unimplemented!(\"No tls feature was specified\")\n        }\n    }\n}\n\npub mod stream {\n    use std::{\n        pin::Pin,\n        task::{Context, Poll},\n    };\n\n    use async_trait::async_trait;\n    use tokio::io::{AsyncRead, AsyncWrite, ReadBuf};\n\n    use crate::protocols::{\n        GetProxyDigest, GetSocketDigest, GetTimingDigest, Peek, Shutdown, Ssl, UniqueID,\n    };\n\n    /// A TLS session over a stream.\n    #[derive(Debug)]\n    pub struct SslStream<S> {\n        marker: std::marker::PhantomData<S>,\n    }\n\n    impl<S> Default for SslStream<S> {\n        fn default() -> Self {\n            Self {\n                marker: Default::default(),\n            }\n        }\n    }\n\n    impl<S> AsyncRead for SslStream<S>\n    where\n        S: AsyncRead + AsyncWrite,\n    {\n        fn poll_read(\n            self: Pin<&mut Self>,\n            _ctx: &mut Context<'_>,\n            _buf: &mut ReadBuf<'_>,\n        ) -> Poll<std::io::Result<()>> {\n            Poll::Ready(Ok(()))\n        }\n    }\n\n    impl<S> AsyncWrite for SslStream<S>\n    where\n        S: AsyncRead + AsyncWrite,\n    {\n        fn poll_write(\n            self: Pin<&mut Self>,\n            _ctx: &mut Context<'_>,\n            buf: &[u8],\n        ) -> Poll<std::io::Result<usize>> {\n            Poll::Ready(Ok(buf.len()))\n        }\n\n        fn poll_flush(self: Pin<&mut Self>, _ctx: &mut Context<'_>) -> Poll<std::io::Result<()>> {\n            Poll::Ready(Ok(()))\n        }\n\n        fn poll_shutdown(\n            self: Pin<&mut Self>,\n            _ctx: &mut Context<'_>,\n        ) -> Poll<std::io::Result<()>> {\n            Poll::Ready(Ok(()))\n        }\n    }\n\n    #[async_trait]\n    impl<S: Send> Shutdown for SslStream<S> {\n        async fn shutdown(&mut self) {}\n    }\n\n    impl<S> UniqueID for SslStream<S> {\n        fn id(&self) -> crate::protocols::UniqueIDType {\n            0\n        }\n    }\n\n    impl<S> Ssl for SslStream<S> {}\n\n    impl<S> GetTimingDigest for SslStream<S> {\n        fn get_timing_digest(&self) -> Vec<Option<crate::protocols::TimingDigest>> {\n            vec![]\n        }\n    }\n\n    impl<S> GetProxyDigest for SslStream<S> {\n        fn get_proxy_digest(\n            &self,\n        ) -> Option<std::sync::Arc<crate::protocols::raw_connect::ProxyDigest>> {\n            None\n        }\n    }\n\n    impl<S> GetSocketDigest for SslStream<S> {\n        fn get_socket_digest(&self) -> Option<std::sync::Arc<crate::protocols::SocketDigest>> {\n            None\n        }\n    }\n\n    impl<S> Peek for SslStream<S> {}\n}\n\npub mod utils {\n    use std::fmt::Display;\n\n    use super::CertWrapper;\n\n    #[derive(Debug, Clone, Hash)]\n    pub struct CertKey;\n\n    impl Display for CertKey {\n        fn fmt(&self, _: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {\n            Ok(())\n        }\n    }\n\n    pub fn get_organization_unit(_: &CertWrapper) -> Option<String> {\n        None\n    }\n}\n"
  },
  {
    "path": "pingora-core/src/protocols/tls/rustls/client.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! Rustls TLS client specific implementation\n\nuse crate::protocols::tls::rustls::TlsStream;\nuse crate::protocols::IO;\nuse pingora_error::ErrorType::TLSHandshakeFailure;\nuse pingora_error::{Error, OrErr, Result};\nuse pingora_rustls::TlsConnector;\n\n// Perform the TLS handshake for the given connection with the given configuration\npub async fn handshake<S: IO>(\n    connector: &TlsConnector,\n    domain: &str,\n    io: S,\n) -> Result<TlsStream<S>> {\n    let mut stream = TlsStream::from_connector(connector, domain, io)\n        .await\n        .or_err(TLSHandshakeFailure, \"tls stream error\")?;\n\n    let handshake_result = stream.connect().await;\n    match handshake_result {\n        Ok(()) => Ok(stream),\n        Err(e) => {\n            let context = format!(\"TLS connect() failed: {e}, SNI: {domain}\");\n            Error::e_explain(TLSHandshakeFailure, context)\n        }\n    }\n}\n"
  },
  {
    "path": "pingora-core/src/protocols/tls/rustls/mod.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\npub mod client;\npub mod server;\nmod stream;\n\npub use stream::*;\n\nuse crate::utils::tls::WrappedX509;\n\npub type CaType = [WrappedX509];\n\npub struct TlsRef;\n"
  },
  {
    "path": "pingora-core/src/protocols/tls/rustls/server.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! Rustls TLS server specific implementation\n\nuse crate::listeners::TlsAcceptCallbacks;\nuse crate::protocols::tls::rustls::TlsStream;\nuse crate::protocols::tls::TlsRef;\nuse crate::protocols::IO;\nuse crate::{listeners::tls::Acceptor, protocols::Shutdown};\nuse async_trait::async_trait;\nuse log::warn;\nuse pingora_error::{ErrorType::*, OrErr, Result};\nuse std::pin::Pin;\nuse tokio::io::{AsyncRead, AsyncWrite, AsyncWriteExt};\n\nimpl<S: AsyncRead + AsyncWrite + Send + Unpin> TlsStream<S> {\n    async fn start_accept(mut self: Pin<&mut Self>) -> Result<bool> {\n        // TODO: suspend cert callback\n        let res = self.accept().await;\n\n        match res {\n            Ok(()) => Ok(true),\n            Err(e) => {\n                if e.etype == TLSWantX509Lookup {\n                    Ok(false)\n                } else {\n                    Err(e)\n                }\n            }\n        }\n    }\n\n    async fn resume_accept(mut self: Pin<&mut Self>) -> Result<()> {\n        // TODO: unblock cert callback\n        self.accept().await\n    }\n}\n\nasync fn prepare_tls_stream<S: IO>(acceptor: &Acceptor, io: S) -> Result<TlsStream<S>> {\n    TlsStream::from_acceptor(acceptor, io)\n        .await\n        .explain_err(TLSHandshakeFailure, |e| format!(\"tls stream error: {e}\"))\n}\n\n/// Perform TLS handshake for the given connection with the given configuration\npub async fn handshake<S: IO>(acceptor: &Acceptor, io: S) -> Result<TlsStream<S>> {\n    let mut stream = prepare_tls_stream(acceptor, io).await?;\n    stream\n        .accept()\n        .await\n        .explain_err(TLSHandshakeFailure, |e| format!(\"TLS accept() failed: {e}\"))?;\n    Ok(stream)\n}\n\n/// Perform TLS handshake for the given connection with the given configuration and callbacks\n/// callbacks are currently not supported within pingora Rustls and are ignored\npub async fn handshake_with_callback<S: IO>(\n    acceptor: &Acceptor,\n    io: S,\n    callbacks: &TlsAcceptCallbacks,\n) -> Result<TlsStream<S>> {\n    let mut tls_stream = prepare_tls_stream(acceptor, io).await?;\n    let done = Pin::new(&mut tls_stream).start_accept().await?;\n    if !done {\n        // TODO: verify if/how callback in handshake can be done using Rustls\n        warn!(\"Callacks are not supported with feature \\\"rustls\\\".\");\n\n        Pin::new(&mut tls_stream)\n            .resume_accept()\n            .await\n            .explain_err(TLSHandshakeFailure, |e| format!(\"TLS accept() failed: {e}\"))?;\n    }\n    {\n        let tls_ref = TlsRef;\n        if let Some(extension) = callbacks.handshake_complete_callback(&tls_ref).await {\n            if let Some(digest_mut) = tls_stream.ssl_digest_mut() {\n                digest_mut.extension.set(extension);\n            }\n        }\n    }\n    Ok(tls_stream)\n}\n\n#[async_trait]\nimpl<S> Shutdown for TlsStream<S>\nwhere\n    S: AsyncRead + AsyncWrite + Sync + Unpin + Send,\n{\n    async fn shutdown(&mut self) {\n        match <Self as AsyncWriteExt>::shutdown(self).await {\n            Ok(()) => {}\n            Err(e) => {\n                warn!(\"TLS shutdown failed, {e}\");\n            }\n        }\n    }\n}\n\n#[ignore]\n#[tokio::test]\nasync fn test_async_cert() {\n    todo!(\"callback support and test for Rustls\")\n}\n"
  },
  {
    "path": "pingora-core/src/protocols/tls/rustls/stream.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\nuse std::io::Result as IoResult;\nuse std::ops::{Deref, DerefMut};\nuse std::pin::Pin;\nuse std::sync::Arc;\nuse std::task::{Context, Poll};\nuse std::time::{Duration, SystemTime};\n\nuse crate::listeners::tls::Acceptor;\nuse crate::protocols::raw_connect::ProxyDigest;\nuse crate::protocols::{tls::SslDigest, Peek, TimingDigest, UniqueIDType};\nuse crate::protocols::{\n    GetProxyDigest, GetSocketDigest, GetTimingDigest, SocketDigest, Ssl, UniqueID, ALPN,\n};\nuse crate::utils::tls::get_organization_serial_bytes;\nuse pingora_error::ErrorType::{AcceptError, ConnectError, InternalError, TLSHandshakeFailure};\nuse pingora_error::{OkOrErr, OrErr, Result};\nuse pingora_rustls::TlsStream as RusTlsStream;\nuse pingora_rustls::{hash_certificate, NoDebug};\nuse pingora_rustls::{Accept, Connect, ServerName, TlsConnector};\nuse tokio::io::{AsyncRead, AsyncWrite, ReadBuf};\nuse x509_parser::nom::AsBytes;\n\n#[derive(Debug)]\npub struct InnerStream<T> {\n    pub(crate) stream: Option<RusTlsStream<T>>,\n    connect: NoDebug<Option<Connect<T>>>,\n    accept: NoDebug<Option<Accept<T>>>,\n}\n\n/// The TLS connection\n#[derive(Debug)]\npub struct TlsStream<T> {\n    tls: InnerStream<T>,\n    digest: Option<Arc<SslDigest>>,\n    timing: TimingDigest,\n}\n\nimpl<T> TlsStream<T>\nwhere\n    T: AsyncRead + AsyncWrite + Unpin + Send,\n{\n    /// Create a new TLS connection from the given `stream`\n    ///\n    /// Using RustTLS the stream is only returned after the handshake.\n    /// The caller does therefor not need to perform [`Self::connect()`].\n    pub async fn from_connector(connector: &TlsConnector, domain: &str, stream: T) -> Result<Self> {\n        let server = ServerName::try_from(domain).or_err_with(InternalError, || {\n            format!(\"Invalid Input: Failed to parse domain: {domain}\")\n        })?;\n\n        let tls = InnerStream::from_connector(connector, server, stream)\n            .await\n            .explain_err(TLSHandshakeFailure, |e| format!(\"tls stream error: {e}\"))?;\n\n        Ok(TlsStream {\n            tls,\n            digest: None,\n            timing: Default::default(),\n        })\n    }\n\n    /// Create a new TLS connection from the given `stream`\n    ///\n    /// Using RustTLS the stream is only returned after the handshake.\n    /// The caller does therefor not need to perform [`Self::accept()`].\n    pub(crate) async fn from_acceptor(acceptor: &Acceptor, stream: T) -> Result<Self> {\n        let tls = InnerStream::from_acceptor(acceptor, stream)\n            .await\n            .explain_err(TLSHandshakeFailure, |e| format!(\"tls stream error: {e}\"))?;\n\n        Ok(TlsStream {\n            tls,\n            digest: None,\n            timing: Default::default(),\n        })\n    }\n}\n\nimpl<S> GetSocketDigest for TlsStream<S>\nwhere\n    S: GetSocketDigest,\n{\n    fn get_socket_digest(&self) -> Option<Arc<SocketDigest>> {\n        self.tls.get_socket_digest()\n    }\n    fn set_socket_digest(&mut self, socket_digest: SocketDigest) {\n        self.tls.set_socket_digest(socket_digest)\n    }\n}\n\nimpl<S> GetTimingDigest for TlsStream<S>\nwhere\n    S: GetTimingDigest,\n{\n    fn get_timing_digest(&self) -> Vec<Option<TimingDigest>> {\n        let mut ts_vec = self.tls.get_timing_digest();\n        ts_vec.push(Some(self.timing.clone()));\n        ts_vec\n    }\n    fn get_read_pending_time(&self) -> Duration {\n        self.tls.get_read_pending_time()\n    }\n\n    fn get_write_pending_time(&self) -> Duration {\n        self.tls.get_write_pending_time()\n    }\n}\n\nimpl<S> GetProxyDigest for TlsStream<S>\nwhere\n    S: GetProxyDigest,\n{\n    fn get_proxy_digest(&self) -> Option<Arc<ProxyDigest>> {\n        self.tls.get_proxy_digest()\n    }\n}\n\nimpl<T> TlsStream<T> {\n    pub fn ssl_digest(&self) -> Option<Arc<SslDigest>> {\n        self.digest.clone()\n    }\n\n    /// Attempts to obtain a mutable reference to the SslDigest.\n    /// This method returns `None` if the SslDigest is currently held by other references.\n    pub(crate) fn ssl_digest_mut(&mut self) -> Option<&mut SslDigest> {\n        Arc::get_mut(self.digest.as_mut()?)\n    }\n}\n\nimpl<T> Deref for TlsStream<T> {\n    type Target = InnerStream<T>;\n\n    fn deref(&self) -> &Self::Target {\n        &self.tls\n    }\n}\n\nimpl<T> DerefMut for TlsStream<T> {\n    fn deref_mut(&mut self) -> &mut Self::Target {\n        &mut self.tls\n    }\n}\n\nimpl<T> TlsStream<T>\nwhere\n    T: AsyncRead + AsyncWrite + Unpin + Send,\n{\n    /// Connect to the remote TLS server as a client\n    pub(crate) async fn connect(&mut self) -> Result<()> {\n        self.tls.connect().await?;\n        self.timing.established_ts = SystemTime::now();\n        self.digest = self.tls.digest();\n        Ok(())\n    }\n\n    /// Finish the TLS handshake from client as a server\n    pub(crate) async fn accept(&mut self) -> Result<()> {\n        self.tls.accept().await?;\n        self.timing.established_ts = SystemTime::now();\n        self.digest = self.tls.digest();\n        Ok(())\n    }\n}\n\nimpl<T> AsyncRead for TlsStream<T>\nwhere\n    T: AsyncRead + AsyncWrite + Unpin,\n{\n    fn poll_read(\n        mut self: Pin<&mut Self>,\n        cx: &mut Context<'_>,\n        buf: &mut ReadBuf<'_>,\n    ) -> Poll<IoResult<()>> {\n        Pin::new(&mut self.tls.stream.as_mut().unwrap()).poll_read(cx, buf)\n    }\n}\n\nimpl<T> AsyncWrite for TlsStream<T>\nwhere\n    T: AsyncRead + AsyncWrite + Unpin,\n{\n    fn poll_write(mut self: Pin<&mut Self>, cx: &mut Context, buf: &[u8]) -> Poll<IoResult<usize>> {\n        Pin::new(&mut self.tls.stream.as_mut().unwrap()).poll_write(cx, buf)\n    }\n\n    fn poll_flush(mut self: Pin<&mut Self>, cx: &mut Context) -> Poll<IoResult<()>> {\n        Pin::new(&mut self.tls.stream.as_mut().unwrap()).poll_flush(cx)\n    }\n\n    fn poll_shutdown(mut self: Pin<&mut Self>, cx: &mut Context) -> Poll<IoResult<()>> {\n        Pin::new(&mut self.tls.stream.as_mut().unwrap()).poll_shutdown(cx)\n    }\n\n    fn poll_write_vectored(\n        mut self: Pin<&mut Self>,\n        cx: &mut Context<'_>,\n        bufs: &[std::io::IoSlice<'_>],\n    ) -> Poll<IoResult<usize>> {\n        Pin::new(&mut self.tls.stream.as_mut().unwrap()).poll_write_vectored(cx, bufs)\n    }\n\n    fn is_write_vectored(&self) -> bool {\n        true\n    }\n}\n\nimpl<T> UniqueID for TlsStream<T>\nwhere\n    T: UniqueID,\n{\n    fn id(&self) -> UniqueIDType {\n        self.tls.stream.as_ref().unwrap().get_ref().0.id()\n    }\n}\n\nimpl<T> Ssl for TlsStream<T> {\n    fn get_ssl_digest(&self) -> Option<Arc<SslDigest>> {\n        self.ssl_digest()\n    }\n\n    fn selected_alpn_proto(&self) -> Option<ALPN> {\n        let st = self.tls.stream.as_ref();\n        if let Some(stream) = st {\n            let proto = stream.get_ref().1.alpn_protocol();\n            match proto {\n                None => None,\n                Some(raw) => ALPN::from_wire_selected(raw),\n            }\n        } else {\n            None\n        }\n    }\n}\n\n/// Create a new TLS connection from the given `stream`\n///\n/// The caller needs to perform [`Self::connect()`] or [`Self::accept()`] to perform TLS\n/// handshake after.\nimpl<T: AsyncRead + AsyncWrite + Unpin> InnerStream<T> {\n    pub(crate) async fn from_connector(\n        connector: &TlsConnector,\n        server: ServerName<'_>,\n        stream: T,\n    ) -> Result<Self> {\n        let connect = connector.connect(server.to_owned(), stream);\n        Ok(InnerStream {\n            accept: None.into(),\n            connect: Some(connect).into(),\n            stream: None,\n        })\n    }\n\n    pub(crate) async fn from_acceptor(acceptor: &Acceptor, stream: T) -> Result<Self> {\n        let accept = acceptor.acceptor.accept(stream);\n\n        Ok(InnerStream {\n            accept: Some(accept).into(),\n            connect: None.into(),\n            stream: None,\n        })\n    }\n}\n\nimpl<T: AsyncRead + AsyncWrite + Unpin + Send> InnerStream<T> {\n    /// Connect to the remote TLS server as a client\n    pub(crate) async fn connect(&mut self) -> Result<()> {\n        let connect = &mut (*self.connect);\n        let connect = connect.take().or_err(\n            ConnectError,\n            \"TLS connect not available to perform handshake.\",\n        )?;\n\n        let stream = connect\n            .await\n            .or_err(TLSHandshakeFailure, \"tls connect error\")?;\n        self.stream = Some(RusTlsStream::Client(stream));\n        Ok(())\n    }\n\n    /// Finish the TLS handshake from client as a server\n    /// no-op implementation within Rustls, handshake is performed during creation of stream.\n    pub(crate) async fn accept(&mut self) -> Result<()> {\n        let accept = &mut (*self.accept);\n        let accept = accept.take().or_err(\n            AcceptError,\n            \"TLS accept not available to perform handshake.\",\n        )?;\n\n        let stream = accept\n            .await\n            .explain_err(TLSHandshakeFailure, |e| format!(\"tls connect error: {e}\"))?;\n        self.stream = Some(RusTlsStream::Server(stream));\n        Ok(())\n    }\n\n    pub(crate) fn digest(&mut self) -> Option<Arc<SslDigest>> {\n        Some(Arc::new(SslDigest::from_stream(&self.stream)))\n    }\n}\n\nimpl<S> GetSocketDigest for InnerStream<S>\nwhere\n    S: GetSocketDigest,\n{\n    fn get_socket_digest(&self) -> Option<Arc<SocketDigest>> {\n        if let Some(stream) = self.stream.as_ref() {\n            stream.get_ref().0.get_socket_digest()\n        } else {\n            None\n        }\n    }\n    fn set_socket_digest(&mut self, socket_digest: SocketDigest) {\n        self.stream\n            .as_mut()\n            .unwrap()\n            .get_mut()\n            .0\n            .set_socket_digest(socket_digest)\n    }\n}\n\nimpl<S> GetTimingDigest for InnerStream<S>\nwhere\n    S: GetTimingDigest,\n{\n    fn get_timing_digest(&self) -> Vec<Option<TimingDigest>> {\n        self.stream\n            .as_ref()\n            .unwrap()\n            .get_ref()\n            .0\n            .get_timing_digest()\n    }\n}\n\nimpl<S> GetProxyDigest for InnerStream<S>\nwhere\n    S: GetProxyDigest,\n{\n    fn get_proxy_digest(&self) -> Option<Arc<ProxyDigest>> {\n        if let Some(stream) = self.stream.as_ref() {\n            stream.get_ref().0.get_proxy_digest()\n        } else {\n            None\n        }\n    }\n}\n\nimpl SslDigest {\n    fn from_stream<T>(stream: &Option<RusTlsStream<T>>) -> Self {\n        let stream = stream.as_ref().unwrap();\n        let (_io, session) = stream.get_ref();\n        let protocol = session.protocol_version();\n        let cipher_suite = session.negotiated_cipher_suite();\n        let peer_certificates = session.peer_certificates();\n\n        let cipher = cipher_suite\n            .and_then(|suite| suite.suite().as_str())\n            .unwrap_or_default();\n\n        let version = protocol\n            .and_then(|proto| proto.as_str())\n            .unwrap_or_default();\n\n        let cert_digest = peer_certificates\n            .and_then(|certs| certs.first())\n            .map(|cert| hash_certificate(cert))\n            .unwrap_or_default();\n\n        let (organization, serial_number) = peer_certificates\n            .and_then(|certs| certs.first())\n            .map(|cert| get_organization_serial_bytes(cert.as_bytes()))\n            .transpose()\n            .ok()\n            .flatten()\n            .map(|(organization, serial)| (organization, Some(serial)))\n            .unwrap_or_default();\n\n        SslDigest::new(cipher, version, organization, serial_number, cert_digest)\n    }\n}\n\nimpl<S> Peek for TlsStream<S> {}\n"
  },
  {
    "path": "pingora-core/src/protocols/tls/s2n/client.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! S2N client specific implementation\n\nuse crate::protocols::tls::{AutoFlushableStream, S2NConnectionBuilder, TlsStream};\nuse crate::protocols::IO;\nuse pingora_error::ErrorType::TLSHandshakeFailure;\nuse pingora_error::{Error, Result};\nuse pingora_s2n::TlsConnector;\n\n// Perform the TLS handshake for the given connection with the given configuration\npub async fn handshake<S: IO>(\n    connector: &TlsConnector<S2NConnectionBuilder>,\n    domain: &str,\n    stream: S,\n) -> Result<TlsStream<S>> {\n    // Wrap incoming stream in an auto flushable stream with auto flush enabled because\n    // s2n-tls doesn't invoke flush after writing to the connection. This would result in\n    // the handshake hanging and timing on streams with write buffering.\n    let auto_flushable_stream = AutoFlushableStream::new(stream, true);\n    let mut s2n_stream = connector\n        .connect(domain, auto_flushable_stream)\n        .await\n        .map_err(|e| {\n            let context = format!(\"TLS connect() failed: {e}, SNI: {domain}\");\n            Error::explain(TLSHandshakeFailure, context)\n        })?;\n\n    // Disable auto-flush to not interfere with write buffering going forward.\n    s2n_stream.get_mut().set_auto_flush(false);\n    Ok(TlsStream::from_s2n_stream(s2n_stream))\n}\n"
  },
  {
    "path": "pingora-core/src/protocols/tls/s2n/mod.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\npub mod client;\npub mod server;\nmod stream;\n\nuse std::{\n    hash::{Hash, Hasher},\n    sync::Arc,\n};\n\nuse pingora_s2n::{\n    Config, Connection, ConnectionBuilder, Mode, Psk as S2NPsk, PskHmac, S2NError, S2NPolicy,\n};\npub use stream::*;\n\nuse crate::utils::tls::X509Pem;\n\npub type CaType = X509Pem;\n\npub type PskType = PskConfig;\n\n#[derive(Debug)]\npub struct PskConfig {\n    pub keys: Vec<Psk>,\n}\n\nimpl PskConfig {\n    pub fn new(keys: Vec<Psk>) -> Self {\n        Self { keys }\n    }\n}\n\nimpl Hash for PskConfig {\n    fn hash<H: Hasher>(&self, state: &mut H) {\n        for psk in self.keys.iter() {\n            psk.identity.hash(state);\n            psk.secret.hash(state);\n        }\n    }\n}\n\n#[derive(Debug)]\npub struct Psk {\n    pub identity: Vec<u8>,\n    pub secret: Vec<u8>,\n    pub hmac: PskHmac,\n}\n\nimpl Psk {\n    pub fn new(identity: String, secret: Vec<u8>, hmac: PskHmac) -> Self {\n        Self {\n            identity: identity.into_bytes(),\n            secret,\n            hmac,\n        }\n    }\n}\n\npub struct TlsRef;\n\n/// Custom s2n-tls connection builder. The s2n-tls-tokio crate doesn't expose\n/// a higher level api to configure private shared keys on a TLS connection.\n///\n/// This builder will create a new connection and configure it with the appropriate\n/// psk configurations based on the provided private shared keys.\n/// ```\n#[derive(Debug, Clone)]\npub struct S2NConnectionBuilder {\n    pub config: Config,\n    pub psk_config: Option<Arc<PskConfig>>,\n    pub security_policy: Option<S2NPolicy>,\n}\n\nimpl ConnectionBuilder for S2NConnectionBuilder {\n    type Output = Connection;\n    fn build_connection(&self, mode: Mode) -> std::result::Result<Self::Output, S2NError> {\n        let mut conn = Connection::new(mode);\n        conn.set_config(self.config.clone())?;\n\n        if let Some(psk_config) = &self.psk_config {\n            for psk in psk_config.keys.iter() {\n                let mut psk_builder = S2NPsk::builder()?;\n                psk_builder.set_identity(&psk.identity)?;\n                psk_builder.set_hmac(PskHmac::SHA256)?;\n                psk_builder.set_secret(&psk.secret)?;\n                conn.append_psk(&psk_builder.build()?)?;\n            }\n        }\n\n        if let Some(policy) = &self.security_policy {\n            conn.set_security_policy(policy)?;\n        }\n\n        Ok(conn)\n    }\n}\n"
  },
  {
    "path": "pingora-core/src/protocols/tls/s2n/server.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! S2N server specific implementation\n\nuse crate::listeners::tls::Acceptor;\nuse crate::protocols::tls::{AutoFlushableStream, TlsStream};\nuse crate::protocols::IO;\nuse pingora_error::ErrorType::TLSHandshakeFailure;\nuse pingora_error::{Error, Result};\n\npub async fn handshake<S: IO>(acceptor: &Acceptor, stream: S) -> Result<TlsStream<S>> {\n    // Wrap incoming stream in an auto flushable stream with auto flush enabled because\n    // s2n-tls doesn't invoke flush after writing to the connection. This would result in\n    // the handshake hanging and timing on streams with write buffering.\n    let auto_flushable_stream = AutoFlushableStream::new(stream, true);\n    let mut s2n_stream = acceptor\n        .acceptor\n        .accept(auto_flushable_stream)\n        .await\n        .map_err(|e| {\n            let context = format!(\"TLS accept() failed: {e}\");\n            Error::explain(TLSHandshakeFailure, context)\n        })?;\n\n    // Disable auto-flush to not interfere with write buffering going forward.\n    s2n_stream.get_mut().set_auto_flush(false);\n\n    Ok(TlsStream::from_s2n_stream(s2n_stream))\n}\n"
  },
  {
    "path": "pingora-core/src/protocols/tls/s2n/stream.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\nuse crate::protocols::digest::TimingDigest;\nuse crate::protocols::raw_connect::ProxyDigest;\nuse crate::protocols::tls::SslDigest;\nuse crate::protocols::{\n    GetProxyDigest, GetSocketDigest, GetTimingDigest, Peek, Shutdown, SocketDigest, Ssl, UniqueID,\n    UniqueIDType, ALPN,\n};\nuse crate::tls::TlsStream as S2NTlsStream;\nuse crate::utils::tls::get_organization_serial_bytes;\nuse async_trait::async_trait;\nuse log::debug;\nuse pingora_s2n::hash_certificate;\nuse std::fmt::Debug;\nuse std::io::Result as IoResult;\nuse std::ops::{Deref, DerefMut};\nuse std::pin::Pin;\nuse std::sync::Arc;\nuse std::task::{Context, Poll};\nuse std::time::{Duration, SystemTime};\nuse tokio::io::{AsyncRead, AsyncWrite, ReadBuf};\n\n/// Stream wrapper that will automatically flush all writes depending on the value of\n/// `auto_flush`. That is, it will always call `poll_flush` on every invocation of\n/// `poll_write` or `poll_write_vectored`.\n///\n/// The underlying transport stream implementation (pingora_core::protocols::l4::stream::Stream)\n/// used by Pingora buffers writes to the TCP connection. During the handshake process\n/// s2n-tls does not flush writes to the TCP connection, which can lead to scenarios\n/// where writes are never sent over the connection causing the handshake process to hang\n/// and timeout. This wrapper ensures that all writes are flushed to the TCP connection\n/// during the handshake process.\npub struct AutoFlushableStream<T: AsyncRead + AsyncWrite + Unpin> {\n    stream: T,\n    auto_flush: bool,\n}\n\nimpl<T> AutoFlushableStream<T>\nwhere\n    T: AsyncRead + AsyncWrite + Unpin,\n{\n    pub fn new(stream: T, auto_flush: bool) -> Self {\n        AutoFlushableStream { stream, auto_flush }\n    }\n\n    pub fn set_auto_flush(&mut self, auto_flush: bool) {\n        self.auto_flush = auto_flush;\n    }\n}\n\nimpl<T> AsyncRead for AutoFlushableStream<T>\nwhere\n    T: AsyncRead + AsyncWrite + Unpin,\n{\n    fn poll_read(\n        mut self: Pin<&mut Self>,\n        cx: &mut Context<'_>,\n        buf: &mut ReadBuf<'_>,\n    ) -> Poll<IoResult<()>> {\n        Pin::new(&mut self.stream).poll_read(cx, buf)\n    }\n}\n\nimpl<T> AsyncWrite for AutoFlushableStream<T>\nwhere\n    T: AsyncRead + AsyncWrite + Unpin,\n{\n    fn poll_write(mut self: Pin<&mut Self>, cx: &mut Context, buf: &[u8]) -> Poll<IoResult<usize>> {\n        let write = Pin::new(&mut self.stream).poll_write(cx, buf);\n        if self.auto_flush {\n            let _ = Pin::new(&mut self.stream).poll_flush(cx);\n        }\n        write\n    }\n\n    fn poll_flush(mut self: Pin<&mut Self>, cx: &mut Context) -> Poll<IoResult<()>> {\n        Pin::new(&mut self.stream).poll_flush(cx)\n    }\n\n    fn poll_shutdown(mut self: Pin<&mut Self>, cx: &mut Context) -> Poll<IoResult<()>> {\n        Pin::new(&mut self.stream).poll_shutdown(cx)\n    }\n\n    fn poll_write_vectored(\n        mut self: Pin<&mut Self>,\n        cx: &mut Context<'_>,\n        bufs: &[std::io::IoSlice<'_>],\n    ) -> Poll<IoResult<usize>> {\n        let write = Pin::new(&mut self.stream).poll_write_vectored(cx, bufs);\n        if self.auto_flush {\n            let _ = Pin::new(&mut self.stream).poll_flush(cx);\n        }\n        write\n    }\n\n    fn is_write_vectored(&self) -> bool {\n        true\n    }\n}\n\n#[derive(Debug)]\npub struct TlsStream<T: AsyncRead + AsyncWrite + Unpin> {\n    stream: S2NTlsStream<AutoFlushableStream<T>>,\n    digest: Option<Arc<SslDigest>>,\n    pub(super) timing: TimingDigest,\n}\n\nimpl<T> TlsStream<T>\nwhere\n    T: AsyncRead + AsyncWrite + std::marker::Unpin,\n{\n    pub fn from_s2n_stream(stream: S2NTlsStream<AutoFlushableStream<T>>) -> TlsStream<T> {\n        let mut timing: TimingDigest = Default::default();\n        timing.established_ts = SystemTime::now();\n        let digest = Some(Arc::new(SslDigest::from_stream(Some(&stream))));\n        TlsStream {\n            stream,\n            digest,\n            timing,\n        }\n    }\n}\n\nimpl<T: AsyncRead + AsyncWrite + std::marker::Unpin> Deref for AutoFlushableStream<T> {\n    type Target = T;\n\n    fn deref(&self) -> &Self::Target {\n        &self.stream\n    }\n}\n\nimpl<T: AsyncRead + AsyncWrite + std::marker::Unpin> DerefMut for AutoFlushableStream<T> {\n    fn deref_mut(&mut self) -> &mut Self::Target {\n        &mut self.stream\n    }\n}\n\nimpl<T: AsyncRead + AsyncWrite + std::marker::Unpin> Deref for TlsStream<T> {\n    type Target = S2NTlsStream<AutoFlushableStream<T>>;\n\n    fn deref(&self) -> &Self::Target {\n        &self.stream\n    }\n}\n\nimpl<T: AsyncRead + AsyncWrite + std::marker::Unpin> DerefMut for TlsStream<T> {\n    fn deref_mut(&mut self) -> &mut Self::Target {\n        &mut self.stream\n    }\n}\n\nimpl<T: AsyncRead + AsyncWrite + std::marker::Unpin> Ssl for TlsStream<T> {\n    fn get_ssl_digest(&self) -> Option<Arc<SslDigest>> {\n        self.ssl_digest()\n    }\n\n    fn selected_alpn_proto(&self) -> Option<ALPN> {\n        let stream = self.stream.as_ref();\n        let proto = stream.application_protocol();\n\n        match proto {\n            None => None,\n            Some(raw) => ALPN::from_wire_selected(raw),\n        }\n    }\n}\n\nimpl<T> TlsStream<T>\nwhere\n    T: AsyncRead + AsyncWrite + std::marker::Unpin,\n{\n    pub fn ssl_digest(&self) -> Option<Arc<SslDigest>> {\n        self.digest.clone()\n    }\n}\n\nimpl<T> AsyncRead for TlsStream<T>\nwhere\n    T: AsyncRead + AsyncWrite + Unpin,\n{\n    fn poll_read(\n        mut self: Pin<&mut Self>,\n        cx: &mut Context<'_>,\n        buf: &mut ReadBuf<'_>,\n    ) -> Poll<IoResult<()>> {\n        debug!(\"poll_read\");\n        Pin::new(&mut self.stream).poll_read(cx, buf)\n    }\n}\n\nimpl<T> AsyncWrite for TlsStream<T>\nwhere\n    T: AsyncRead + AsyncWrite + Unpin,\n{\n    fn poll_write(mut self: Pin<&mut Self>, cx: &mut Context, buf: &[u8]) -> Poll<IoResult<usize>> {\n        Pin::new(&mut self.stream).poll_write(cx, buf)\n    }\n\n    fn poll_flush(mut self: Pin<&mut Self>, cx: &mut Context) -> Poll<IoResult<()>> {\n        Pin::new(&mut self.stream).poll_flush(cx)\n    }\n\n    fn poll_shutdown(mut self: Pin<&mut Self>, cx: &mut Context) -> Poll<IoResult<()>> {\n        Pin::new(&mut self.stream).poll_shutdown(cx)\n    }\n\n    fn poll_write_vectored(\n        mut self: Pin<&mut Self>,\n        cx: &mut Context<'_>,\n        bufs: &[std::io::IoSlice<'_>],\n    ) -> Poll<IoResult<usize>> {\n        Pin::new(&mut self.stream).poll_write_vectored(cx, bufs)\n    }\n\n    fn is_write_vectored(&self) -> bool {\n        true\n    }\n}\n\nimpl<T> UniqueID for TlsStream<T>\nwhere\n    T: UniqueID + AsyncRead + AsyncWrite + Unpin,\n{\n    fn id(&self) -> UniqueIDType {\n        self.stream.get_ref().id()\n    }\n}\n\nimpl<S> GetSocketDigest for TlsStream<S>\nwhere\n    S: GetSocketDigest + AsyncRead + AsyncWrite + std::marker::Unpin,\n{\n    fn get_socket_digest(&self) -> Option<Arc<SocketDigest>> {\n        self.stream.get_ref().get_socket_digest()\n    }\n    fn set_socket_digest(&mut self, socket_digest: SocketDigest) {\n        self.stream.get_mut().set_socket_digest(socket_digest)\n    }\n}\n\nimpl<S> GetTimingDigest for TlsStream<S>\nwhere\n    S: GetTimingDigest + AsyncRead + AsyncWrite + std::marker::Unpin,\n{\n    fn get_timing_digest(&self) -> Vec<Option<TimingDigest>> {\n        let mut ts_vec = self.stream.get_ref().get_timing_digest();\n        ts_vec.push(Some(self.timing.clone()));\n        ts_vec\n    }\n\n    fn get_read_pending_time(&self) -> Duration {\n        self.stream.get_ref().get_read_pending_time()\n    }\n\n    fn get_write_pending_time(&self) -> Duration {\n        self.stream.get_ref().get_write_pending_time()\n    }\n}\n\nimpl<S> GetProxyDigest for TlsStream<S>\nwhere\n    S: GetProxyDigest + AsyncRead + AsyncWrite + std::marker::Unpin,\n{\n    fn get_proxy_digest(&self) -> Option<Arc<ProxyDigest>> {\n        self.stream.get_ref().get_proxy_digest()\n    }\n}\n\nimpl SslDigest {\n    fn from_stream<T: AsyncRead + AsyncWrite + Unpin>(stream: Option<&S2NTlsStream<T>>) -> Self {\n        let conn = stream.unwrap().as_ref();\n\n        let cipher = conn.cipher_suite().unwrap_or_default().to_string();\n        let version = conn\n            .actual_protocol_version()\n            .map(|v| format!(\"{:?}\", v))\n            .unwrap_or_default()\n            .to_string();\n\n        let mut organization = None;\n        let mut serial_number = None;\n        let mut cert_digest = None;\n\n        if let Ok(cert_chain) = conn.peer_cert_chain() {\n            if let Some(Ok(cert)) = cert_chain.iter().next() {\n                if let Ok(raw_cert) = cert.der() {\n                    if let Ok((org, serial)) = get_organization_serial_bytes(raw_cert) {\n                        organization = org;\n                        serial_number = Some(serial);\n                    }\n                    cert_digest = Some(hash_certificate(raw_cert));\n                }\n            }\n        }\n\n        SslDigest::new(\n            cipher,\n            version,\n            organization,\n            serial_number,\n            cert_digest.unwrap_or_default(),\n        )\n    }\n}\n\nimpl<S: AsyncRead + AsyncWrite + std::marker::Unpin> Peek for TlsStream<S> {}\n\n#[async_trait]\nimpl<S: Shutdown + AsyncRead + AsyncWrite + std::marker::Unpin + Send> Shutdown for TlsStream<S> {\n    async fn shutdown(&mut self) -> () {\n        self.get_mut().shutdown().await\n    }\n}\n"
  },
  {
    "path": "pingora-core/src/protocols/windows.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! Windows specific functionality for calling the WinSock c api\n//!\n//! Implementations here are based on the implementation in the std library\n//! https://github.com/rust-lang/rust/blob/84ac80f/library/std/src/sys_common/net.rs\n//! https://github.com/rust-lang/rust/blob/84ac80f/library/std/src/sys/pal/windows/net.rs\n\nuse std::os::windows::io::RawSocket;\nuse std::{io, mem, net::SocketAddr};\n\nuse windows_sys::Win32::Networking::WinSock::{\n    getpeername, getsockname, AF_INET, AF_INET6, SOCKADDR_IN, SOCKADDR_IN6, SOCKADDR_STORAGE,\n    SOCKET,\n};\n\npub(crate) fn peer_addr(raw_sock: RawSocket) -> io::Result<SocketAddr> {\n    let mut storage = unsafe { mem::zeroed::<SOCKADDR_STORAGE>() };\n    let mut addrlen = mem::size_of_val(&storage) as i32;\n\n    unsafe {\n        let res = getpeername(\n            raw_sock as SOCKET,\n            core::ptr::addr_of_mut!(storage) as *mut _,\n            &mut addrlen,\n        );\n        if res != 0 {\n            return Err(io::Error::last_os_error());\n        }\n    }\n\n    sockaddr_to_addr(&storage, addrlen as usize)\n}\npub(crate) fn local_addr(raw_sock: RawSocket) -> io::Result<SocketAddr> {\n    let mut storage = unsafe { mem::zeroed::<SOCKADDR_STORAGE>() };\n    let mut addrlen = mem::size_of_val(&storage) as i32;\n\n    unsafe {\n        let res = getsockname(\n            raw_sock as libc::SOCKET,\n            core::ptr::addr_of_mut!(storage) as *mut _,\n            &mut addrlen,\n        );\n        if res != 0 {\n            return Err(io::Error::last_os_error());\n        }\n    }\n\n    sockaddr_to_addr(&storage, addrlen as usize)\n}\n\nfn sockaddr_to_addr(storage: &SOCKADDR_STORAGE, len: usize) -> io::Result<SocketAddr> {\n    match storage.ss_family {\n        AF_INET => {\n            assert!(len >= mem::size_of::<SOCKADDR_IN>());\n            Ok(SocketAddr::from(unsafe {\n                let sockaddr = *(storage as *const _ as *const SOCKADDR_IN);\n                (\n                    sockaddr.sin_addr.S_un.S_addr.to_ne_bytes(),\n                    sockaddr.sin_port.to_be(),\n                )\n            }))\n        }\n        AF_INET6 => {\n            assert!(len >= mem::size_of::<SOCKADDR_IN6>());\n            Ok(SocketAddr::from(unsafe {\n                let sockaddr = *(storage as *const _ as *const SOCKADDR_IN6);\n                (sockaddr.sin6_addr.u.Byte, sockaddr.sin6_port.to_be())\n            }))\n        }\n        _ => Err(io::Error::new(\n            io::ErrorKind::InvalidInput,\n            \"invalid argument\",\n        )),\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use std::os::windows::io::AsRawSocket;\n\n    use crate::protocols::l4::{listener::Listener, stream::Stream};\n\n    use super::*;\n\n    async fn assert_listener_and_stream(addr: &str) {\n        let tokio_listener = tokio::net::TcpListener::bind(addr).await.unwrap();\n\n        let listener_local_addr = tokio_listener.local_addr().unwrap();\n\n        let tokio_stream = tokio::net::TcpStream::connect(listener_local_addr)\n            .await\n            .unwrap();\n\n        let stream_local_addr = tokio_stream.local_addr().unwrap();\n        let stream_peer_addr = tokio_stream.peer_addr().unwrap();\n\n        let stream: Stream = tokio_stream.into();\n        let listener: Listener = tokio_listener.into();\n\n        let raw_sock = listener.as_raw_socket();\n        assert_eq!(listener_local_addr, local_addr(raw_sock).unwrap());\n\n        let raw_sock = stream.as_raw_socket();\n        assert_eq!(stream_peer_addr, peer_addr(raw_sock).unwrap());\n        assert_eq!(stream_local_addr, local_addr(raw_sock).unwrap());\n    }\n\n    #[tokio::test]\n    async fn get_v4_addrs_from_raw_socket() {\n        assert_listener_and_stream(\"127.0.0.1:0\").await\n    }\n    #[tokio::test]\n    async fn get_v6_addrs_from_raw_socket() {\n        assert_listener_and_stream(\"[::1]:0\").await\n    }\n}\n"
  },
  {
    "path": "pingora-core/src/server/bootstrap_services.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n#[cfg(unix)]\npub use super::transfer_fd::Fds;\nuse async_trait::async_trait;\nuse log::{debug, error, info};\nuse parking_lot::Mutex;\nuse std::sync::Arc;\nuse tokio::sync::{broadcast, Mutex as TokioMutex};\n\n#[cfg(feature = \"sentry\")]\nuse sentry::ClientOptions;\n\n#[cfg(unix)]\nuse crate::server::ListenFds;\n\nuse crate::{\n    prelude::Opt,\n    server::{configuration::ServerConf, ExecutionPhase, ShutdownWatch},\n    services::{background::BackgroundService, ServiceReadyNotifier},\n};\n\n/// Service that allows the bootstrap process to be delayed until after\n/// dependencies are ready\npub struct BootstrapService {\n    inner: Arc<Mutex<Bootstrap>>,\n}\n\n/// Sentry is typically started as part of the bootstrap process, but if the\n/// bootstrap service is used, we want to initialize Sentry before anything else\n/// to make sure errors are captured.\npub struct SentryInitService {\n    inner: Arc<Mutex<Bootstrap>>,\n}\n\nimpl BootstrapService {\n    pub fn new(inner: &Arc<Mutex<Bootstrap>>) -> Self {\n        BootstrapService {\n            inner: Arc::clone(inner),\n        }\n    }\n}\n\nimpl SentryInitService {\n    pub fn new(inner: &Arc<Mutex<Bootstrap>>) -> Self {\n        SentryInitService {\n            inner: Arc::clone(inner),\n        }\n    }\n}\n\n/// Encapsulation of the data needed to bootstrap the server\npub struct Bootstrap {\n    completed: bool,\n\n    test: bool,\n    upgrade: bool,\n\n    upgrade_sock: String,\n\n    execution_phase_watch: broadcast::Sender<ExecutionPhase>,\n\n    #[cfg(unix)]\n    listen_fds: Option<ListenFds>,\n\n    #[cfg(feature = \"sentry\")]\n    #[cfg_attr(docsrs, doc(cfg(feature = \"sentry\")))]\n    /// The Sentry ClientOptions.\n    ///\n    /// Panics and other events sentry captures will be sent to this DSN **only\n    /// in release mode**\n    pub sentry: Option<ClientOptions>,\n}\n\nimpl Bootstrap {\n    pub fn new(\n        options: &Option<Opt>,\n        conf: &ServerConf,\n        execution_phase_watch: &broadcast::Sender<ExecutionPhase>,\n    ) -> Self {\n        let (test, upgrade) = options\n            .as_ref()\n            .map(|opt| (opt.test, opt.upgrade))\n            .unwrap_or_default();\n\n        let upgrade_sock = conf.upgrade_sock.clone();\n\n        Bootstrap {\n            test,\n            upgrade,\n            upgrade_sock,\n            #[cfg(unix)]\n            listen_fds: None,\n            execution_phase_watch: execution_phase_watch.clone(),\n            completed: false,\n            #[cfg(feature = \"sentry\")]\n            sentry: None,\n        }\n    }\n\n    #[cfg(feature = \"sentry\")]\n    pub fn set_sentry_config(&mut self, sentry_config: Option<ClientOptions>) {\n        self.sentry = sentry_config;\n    }\n\n    /// Start sentry based on the configured options. To prevent multiple\n    /// initializations, this function will consume the sentry configuration\n    /// stored in the bootstrap\n    fn start_sentry(&mut self) {\n        // Only init sentry in release builds\n        #[cfg(all(not(debug_assertions), feature = \"sentry\"))]\n        let _guard = self.sentry.take().map(|opts| sentry::init(opts));\n    }\n\n    pub fn bootstrap(&mut self) {\n        // already bootstrapped\n        if self.completed {\n            return;\n        }\n\n        info!(\"Bootstrap starting\");\n\n        self.execution_phase_watch\n            .send(ExecutionPhase::Bootstrap)\n            .ok();\n\n        self.start_sentry();\n\n        if self.test {\n            info!(\"Server Test passed, exiting\");\n            std::process::exit(0);\n        }\n\n        // load fds\n        #[cfg(unix)]\n        match self.load_fds(self.upgrade) {\n            Ok(_) => {\n                info!(\"Bootstrap done\");\n            }\n            Err(e) => {\n                // sentry log error on fd load failure\n                #[cfg(all(not(debug_assertions), feature = \"sentry\"))]\n                sentry::capture_error(&e);\n\n                error!(\"Bootstrap failed on error: {:?}, exiting.\", e);\n                std::process::exit(1);\n            }\n        }\n\n        self.completed = true;\n\n        self.execution_phase_watch\n            .send(ExecutionPhase::BootstrapComplete)\n            .ok();\n    }\n\n    #[cfg(unix)]\n    fn load_fds(&mut self, upgrade: bool) -> Result<(), nix::Error> {\n        let mut fds = Fds::new();\n        if upgrade {\n            debug!(\"Trying to receive socks\");\n            fds.get_from_sock(self.upgrade_sock.as_str())?\n        }\n        self.listen_fds = Some(Arc::new(TokioMutex::new(fds)));\n        Ok(())\n    }\n\n    #[cfg(unix)]\n    pub fn get_fds(&self) -> Option<ListenFds> {\n        self.listen_fds.clone()\n    }\n}\n\n#[async_trait]\nimpl BackgroundService for BootstrapService {\n    async fn start_with_ready_notifier(\n        &self,\n        _shutdown: ShutdownWatch,\n        notifier: ServiceReadyNotifier,\n    ) {\n        self.inner.lock().bootstrap();\n        notifier.notify_ready();\n    }\n}\n\n#[async_trait]\nimpl BackgroundService for SentryInitService {\n    async fn start_with_ready_notifier(\n        &self,\n        _shutdown: ShutdownWatch,\n        notifier: ServiceReadyNotifier,\n    ) {\n        self.inner.lock().start_sentry();\n        notifier.notify_ready();\n    }\n}\n"
  },
  {
    "path": "pingora-core/src/server/configuration/mod.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! Server configurations\n//!\n//! Server configurations define startup settings such as:\n//! * User and group to run as after daemonization\n//! * Number of threads per service\n//! * Error log file path\n\nuse clap::Parser;\nuse log::{debug, trace};\nuse pingora_error::{Error, ErrorType::*, OrErr, Result};\nuse serde::{Deserialize, Serialize};\nuse std::ffi::OsString;\nuse std::fs;\n\n// default maximum upstream retries for retry-able proxy errors\nconst DEFAULT_MAX_RETRIES: usize = 16;\n\n/// The configuration file\n///\n/// Pingora configuration files are by default YAML files, but any key value format can potentially\n/// be used.\n///\n/// # Extension\n/// New keys can be added to the configuration files which this configuration object will ignore.\n/// Then, users can parse these key-values to pass to their code to use.\n#[derive(Debug, PartialEq, Eq, Serialize, Deserialize)]\n#[serde(default)]\npub struct ServerConf {\n    /// Version\n    pub version: usize,\n    /// Whether to run this process in the background.\n    pub daemon: bool,\n    /// When configured and `daemon` setting is `true`, error log will be written to the given\n    /// file. Otherwise StdErr will be used.\n    pub error_log: Option<String>,\n    /// The pid (process ID) file of this server to be created when running in background\n    pub pid_file: String,\n    /// the path to the upgrade socket\n    ///\n    /// In order to perform zero downtime restart, both the new and old process need to agree on the\n    /// path to this sock in order to coordinate the upgrade.\n    pub upgrade_sock: String,\n    /// If configured, after daemonization, this process will switch to the given user before\n    /// starting to serve traffic.\n    pub user: Option<String>,\n    /// Similar to `user`, the group this process should switch to.\n    pub group: Option<String>,\n    /// How many threads **each** service should get. The threads are not shared across services.\n    pub threads: usize,\n    /// Number of listener tasks to use per fd. This allows for parallel accepts.\n    pub listener_tasks_per_fd: usize,\n    /// Allow work stealing between threads of the same service. Default `true`.\n    pub work_stealing: bool,\n    /// The path to CA file the SSL library should use. If empty, the default trust store location\n    /// defined by the SSL library will be used.\n    pub ca_file: Option<String>,\n    /// The maximum number of unique s2n configs to cache. Creating a new s2n config is an\n    /// expensive operation, so we cache and re-use config objects with identical configurations.\n    /// A value of 0 disables the cache.\n    ///\n    /// WARNING: Disabling the s2n config cache can result in poor performance\n    #[cfg(feature = \"s2n\")]\n    pub s2n_config_cache_size: Option<usize>,\n    /// Grace period in seconds before starting the final step of the graceful shutdown after signaling shutdown.\n    pub grace_period_seconds: Option<u64>,\n    /// Timeout in seconds of the final step for the graceful shutdown.\n    pub graceful_shutdown_timeout_seconds: Option<u64>,\n    // These options don't belong here as they are specific to certain services\n    /// IPv4 addresses for a client connector to bind to. See\n    /// [`ConnectorOptions`](crate::connectors::ConnectorOptions).\n    /// Note: this is an _unstable_ field that may be renamed or removed in the future.\n    pub client_bind_to_ipv4: Vec<String>,\n    /// IPv6 addresses for a client connector to bind to. See\n    /// [`ConnectorOptions`](crate::connectors::ConnectorOptions).\n    /// Note: this is an _unstable_ field that may be renamed or removed in the future.\n    pub client_bind_to_ipv6: Vec<String>,\n    /// Keepalive pool size for client connections to upstream. See\n    /// [`ConnectorOptions`](crate::connectors::ConnectorOptions).\n    /// Note: this is an _unstable_ field that may be renamed or removed in the future.\n    pub upstream_keepalive_pool_size: usize,\n    /// Number of dedicated thread pools to use for upstream connection establishment.\n    /// See [`ConnectorOptions`](crate::connectors::ConnectorOptions).\n    /// Note: this is an _unstable_ field that may be renamed or removed in the future.\n    pub upstream_connect_offload_threadpools: Option<usize>,\n    /// Number of threads per dedicated upstream connection establishment pool.\n    /// See [`ConnectorOptions`](crate::connectors::ConnectorOptions).\n    /// Note: this is an _unstable_ field that may be renamed or removed in the future.\n    pub upstream_connect_offload_thread_per_pool: Option<usize>,\n    /// When enabled allows TLS keys to be written to a file specified by the SSLKEYLOG\n    /// env variable. This can be used by tools like Wireshark to decrypt upstream traffic\n    /// for debugging purposes.\n    /// Note: this is an _unstable_ field that may be renamed or removed in the future.\n    pub upstream_debug_ssl_keylog: bool,\n    /// The maximum number of retries that will be attempted when an error is\n    /// retry-able (`e.retry() == true`) when proxying to upstream.\n    ///\n    /// This setting is a fail-safe and defaults to 16.\n    pub max_retries: usize,\n    /// Maximum number of retries for upgrade socket connect and accept operations.\n    /// This controls how many times send_fds_to will retry connecting and how many times\n    /// get_fds_from will retry accepting during graceful upgrades.\n    /// The retry interval is 1 second between attempts.\n    /// If not set, defaults to 5 retries.\n    pub upgrade_sock_connect_accept_max_retries: Option<usize>,\n}\n\nimpl Default for ServerConf {\n    fn default() -> Self {\n        ServerConf {\n            version: 0,\n            client_bind_to_ipv4: vec![],\n            client_bind_to_ipv6: vec![],\n            ca_file: None,\n            #[cfg(feature = \"s2n\")]\n            s2n_config_cache_size: None,\n            daemon: false,\n            error_log: None,\n            upstream_debug_ssl_keylog: false,\n            pid_file: \"/tmp/pingora.pid\".to_string(),\n            upgrade_sock: \"/tmp/pingora_upgrade.sock\".to_string(),\n            user: None,\n            group: None,\n            threads: 1,\n            listener_tasks_per_fd: 1,\n            work_stealing: true,\n            upstream_keepalive_pool_size: 128,\n            upstream_connect_offload_threadpools: None,\n            upstream_connect_offload_thread_per_pool: None,\n            grace_period_seconds: None,\n            graceful_shutdown_timeout_seconds: None,\n            max_retries: DEFAULT_MAX_RETRIES,\n            upgrade_sock_connect_accept_max_retries: None,\n        }\n    }\n}\n\n/// Command-line options\n///\n/// Call `Opt::parse_args()` to build this object from the process's command line arguments.\n#[derive(Parser, Debug, Default)]\n#[clap(name = \"basic\", long_about = None)]\npub struct Opt {\n    /// Whether this server should try to upgrade from a running old server\n    #[clap(\n        short,\n        long,\n        help = \"This is the base set of command line arguments for a pingora-based service\",\n        long_help = None\n    )]\n    pub upgrade: bool,\n\n    /// Whether this server should run in the background\n    #[clap(short, long)]\n    pub daemon: bool,\n\n    /// Not actually used. This flag is there so that the server is not upset seeing this flag\n    /// passed from `cargo test` sometimes\n    #[clap(long, hide = true)]\n    pub nocapture: bool,\n\n    /// Test the configuration and exit\n    ///\n    /// When this flag is set, calling `server.bootstrap()` will exit the process without errors\n    ///\n    /// This flag is useful for upgrading service where the user wants to make sure the new\n    /// service can start before shutting down the old server process.\n    #[clap(\n        short,\n        long,\n        help = \"This flag is useful for upgrading service where the user wants \\\n                to make sure the new service can start before shutting down \\\n                the old server process.\",\n        long_help = None\n    )]\n    pub test: bool,\n\n    /// The path to the configuration file.\n    ///\n    /// See [`ServerConf`] for more details of the configuration file.\n    #[clap(short, long, help = \"The path to the configuration file.\", long_help = None)]\n    pub conf: Option<String>,\n}\n\nimpl ServerConf {\n    // Does not has to be async until we want runtime reload\n    pub fn load_from_yaml<P>(path: P) -> Result<Self>\n    where\n        P: AsRef<std::path::Path> + std::fmt::Display,\n    {\n        let conf_str = fs::read_to_string(&path).or_err_with(ReadError, || {\n            format!(\"Unable to read conf file from {path}\")\n        })?;\n        debug!(\"Conf file read from {path}\");\n        Self::from_yaml(&conf_str)\n    }\n\n    pub fn load_yaml_with_opt_override(opt: &Opt) -> Result<Self> {\n        if let Some(path) = &opt.conf {\n            let mut conf = Self::load_from_yaml(path)?;\n            conf.merge_with_opt(opt);\n            Ok(conf)\n        } else {\n            Error::e_explain(ReadError, \"No path specified\")\n        }\n    }\n\n    pub fn new() -> Option<Self> {\n        Self::from_yaml(\"---\\nversion: 1\").ok()\n    }\n\n    pub fn new_with_opt_override(opt: &Opt) -> Option<Self> {\n        let conf = Self::new();\n        match conf {\n            Some(mut c) => {\n                c.merge_with_opt(opt);\n                Some(c)\n            }\n            None => None,\n        }\n    }\n\n    pub fn from_yaml(conf_str: &str) -> Result<Self> {\n        trace!(\"Read conf file: {conf_str}\");\n        let conf: ServerConf = serde_yaml::from_str(conf_str).or_err_with(ReadError, || {\n            format!(\"Unable to parse yaml conf {conf_str}\")\n        })?;\n\n        trace!(\"Loaded conf: {conf:?}\");\n        conf.validate()\n    }\n\n    pub fn to_yaml(&self) -> String {\n        serde_yaml::to_string(self).unwrap()\n    }\n\n    pub fn validate(self) -> Result<Self> {\n        // TODO: do the validation\n        Ok(self)\n    }\n\n    pub fn merge_with_opt(&mut self, opt: &Opt) {\n        if opt.daemon {\n            self.daemon = true;\n        }\n    }\n}\n\n/// Create an instance of Opt by parsing the current command-line args.\n/// This is equivalent to running `Opt::parse` but does not require the\n/// caller to have included the `clap::Parser`\nimpl Opt {\n    pub fn parse_args() -> Self {\n        Opt::parse()\n    }\n\n    pub fn parse_from_args<I, T>(args: I) -> Self\n    where\n        I: IntoIterator<Item = T>,\n        T: Into<OsString> + Clone,\n    {\n        Opt::parse_from(args)\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use super::*;\n\n    fn init_log() {\n        let _ = env_logger::builder().is_test(true).try_init();\n    }\n\n    #[test]\n    fn not_a_test_i_cannot_write_yaml_by_hand() {\n        init_log();\n        let conf = ServerConf {\n            version: 1,\n            client_bind_to_ipv4: vec![\"1.2.3.4\".to_string(), \"5.6.7.8\".to_string()],\n            client_bind_to_ipv6: vec![],\n            ca_file: None,\n            #[cfg(feature = \"s2n\")]\n            s2n_config_cache_size: None,\n            daemon: false,\n            error_log: None,\n            upstream_debug_ssl_keylog: false,\n            pid_file: \"\".to_string(),\n            upgrade_sock: \"\".to_string(),\n            user: None,\n            group: None,\n            threads: 1,\n            listener_tasks_per_fd: 1,\n            work_stealing: true,\n            upstream_keepalive_pool_size: 4,\n            upstream_connect_offload_threadpools: None,\n            upstream_connect_offload_thread_per_pool: None,\n            grace_period_seconds: None,\n            graceful_shutdown_timeout_seconds: None,\n            max_retries: 1,\n            upgrade_sock_connect_accept_max_retries: None,\n        };\n        // cargo test -- --nocapture not_a_test_i_cannot_write_yaml_by_hand\n        println!(\"{}\", conf.to_yaml());\n    }\n\n    #[test]\n    fn test_load_file() {\n        init_log();\n        let conf_str = r#\"\n---\nversion: 1\nclient_bind_to_ipv4:\n    - 1.2.3.4\n    - 5.6.7.8\nclient_bind_to_ipv6: []\n        \"#\n        .to_string();\n        let conf = ServerConf::from_yaml(&conf_str).unwrap();\n        assert_eq!(2, conf.client_bind_to_ipv4.len());\n        assert_eq!(0, conf.client_bind_to_ipv6.len());\n        assert_eq!(1, conf.version);\n    }\n\n    #[test]\n    fn test_default() {\n        init_log();\n        let conf_str = r#\"\n---\nversion: 1\n        \"#\n        .to_string();\n        let conf = ServerConf::from_yaml(&conf_str).unwrap();\n        assert_eq!(0, conf.client_bind_to_ipv4.len());\n        assert_eq!(0, conf.client_bind_to_ipv6.len());\n        assert_eq!(1, conf.version);\n        assert_eq!(DEFAULT_MAX_RETRIES, conf.max_retries);\n        assert_eq!(\"/tmp/pingora.pid\", conf.pid_file);\n    }\n}\n"
  },
  {
    "path": "pingora-core/src/server/daemon.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\nuse daemonize::{Daemonize, Stdio};\nuse log::{debug, error};\nuse std::ffi::CString;\nuse std::fs::{self, OpenOptions};\nuse std::os::unix::prelude::OpenOptionsExt;\nuse std::path::Path;\n\nuse crate::server::configuration::ServerConf;\n\n// Utilities to daemonize a pingora server, i.e. run the process in the background, possibly\n// under a different running user and/or group.\n\n// XXX: this operation should have been done when the old service is exiting.\n// Now the new pid file just kick the old one out of the way\nfn move_old_pid(path: &str) {\n    if !Path::new(path).exists() {\n        debug!(\"Old pid file does not exist\");\n        return;\n    }\n    let new_path = format!(\"{path}.old\");\n    match fs::rename(path, &new_path) {\n        Ok(()) => {\n            debug!(\"Old pid file renamed\");\n        }\n        Err(e) => {\n            error!(\n                \"failed to rename pid file from {} to {}: {}\",\n                path, new_path, e\n            );\n        }\n    }\n}\n\nunsafe fn gid_for_username(name: &CString) -> Option<libc::gid_t> {\n    let passwd = libc::getpwnam(name.as_ptr() as *const libc::c_char);\n    if !passwd.is_null() {\n        return Some((*passwd).pw_gid);\n    }\n    None\n}\n\n/// Start a server instance as a daemon.\n#[cfg(unix)]\npub fn daemonize(conf: &ServerConf) {\n    // TODO: customize working dir\n\n    let daemonize = Daemonize::new()\n        .umask(0o007) // allow same group to access files but not everyone else\n        .pid_file(&conf.pid_file);\n\n    let daemonize = if let Some(error_log) = conf.error_log.as_ref() {\n        let err = OpenOptions::new()\n            .append(true)\n            .create(true)\n            // open read() in case there are no readers\n            // available otherwise we will panic with\n            // an ENXIO since O_NONBLOCK is set\n            .read(true)\n            .custom_flags(libc::O_NONBLOCK)\n            .open(error_log)\n            .unwrap();\n        daemonize.stderr(err)\n    } else {\n        daemonize.stdout(Stdio::keep()).stderr(Stdio::keep())\n    };\n\n    let daemonize = match conf.user.as_ref() {\n        Some(user) => {\n            let user_cstr = CString::new(user.as_str()).unwrap();\n\n            #[cfg(target_os = \"macos\")]\n            let group_id = unsafe { gid_for_username(&user_cstr).map(|gid| gid as i32) };\n            #[cfg(target_os = \"freebsd\")]\n            let group_id = unsafe { gid_for_username(&user_cstr).map(|gid| gid as u32) };\n            #[cfg(target_os = \"linux\")]\n            let group_id = unsafe { gid_for_username(&user_cstr) };\n\n            daemonize\n                .privileged_action(move || {\n                    if let Some(gid) = group_id {\n                        // Set the supplemental group privileges for the child process.\n                        unsafe {\n                            libc::initgroups(user_cstr.as_ptr() as *const libc::c_char, gid);\n                        }\n                    }\n                })\n                .user(user.as_str())\n                .chown_pid_file(true)\n        }\n        None => daemonize,\n    };\n\n    let daemonize = match conf.group.as_ref() {\n        Some(group) => daemonize.group(group.as_str()),\n        None => daemonize,\n    };\n\n    move_old_pid(&conf.pid_file);\n\n    daemonize.start().unwrap(); // hard crash when fail\n}\n"
  },
  {
    "path": "pingora-core/src/server/mod.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! Server process and configuration management\n\nmod bootstrap_services;\npub mod configuration;\n#[cfg(unix)]\nmod daemon;\n#[cfg(unix)]\npub(crate) mod transfer_fd;\n\nuse async_trait::async_trait;\n#[cfg(unix)]\nuse daemon::daemonize;\nuse daggy::NodeIndex;\nuse log::{debug, error, info, warn};\nuse parking_lot::Mutex;\nuse pingora_runtime::Runtime;\nuse pingora_timeout::fast_timeout;\n#[cfg(feature = \"sentry\")]\nuse sentry::ClientOptions;\nuse std::sync::Arc;\nuse std::thread;\nuse std::time::SystemTime;\n#[cfg(unix)]\nuse tokio::signal::unix;\nuse tokio::sync::{broadcast, watch, Mutex as TokioMutex};\nuse tokio::time::{sleep, Duration};\n\nuse crate::prelude::background_service;\nuse crate::server::bootstrap_services::{Bootstrap, BootstrapService, SentryInitService};\nuse crate::services::{\n    DependencyGraph, ServiceHandle, ServiceReadyNotifier, ServiceReadyWatch, ServiceWithDependents,\n};\nuse configuration::{Opt, ServerConf};\nuse std::collections::HashMap;\n#[cfg(unix)]\npub use transfer_fd::Fds;\n\nuse pingora_error::{Error, ErrorType, Result};\n\n/* Time to wait before exiting the program.\nThis is the graceful period for all existing sessions to finish */\nconst EXIT_TIMEOUT: u64 = 60 * 5;\n/* Time to wait before shutting down listening sockets.\nThis is the graceful period for the new service to get ready */\nconst CLOSE_TIMEOUT: u64 = 5;\n\nenum ShutdownType {\n    Graceful,\n    Quick,\n}\n\n/// Internal wrapper for services with dependency metadata.\npub(crate) struct ServiceWrapper {\n    ready_notifier: Option<ServiceReadyNotifier>,\n    service: Box<dyn ServiceWithDependents>,\n    service_handle: ServiceHandle,\n}\n\n/// The execution phase the server is currently in.\n#[derive(Clone, Debug)]\n#[non_exhaustive]\npub enum ExecutionPhase {\n    /// The server was created, but has not started yet.\n    Setup,\n\n    /// Services are being prepared.\n    ///\n    /// During graceful upgrades this phase acquires the listening FDs from the old process.\n    Bootstrap,\n\n    /// Bootstrap has finished, listening FDs have been transferred.\n    BootstrapComplete,\n\n    /// The server is running and is listening for shutdown signals.\n    Running,\n\n    /// A QUIT signal was received, indicating that a new process wants to take over.\n    ///\n    /// The server is trying to send the fds to the new process over a Unix socket.\n    GracefulUpgradeTransferringFds,\n\n    /// FDs have been sent to the new process.\n    /// Waiting a fixed amount of time to allow the new process to take the sockets.\n    GracefulUpgradeCloseTimeout,\n\n    /// A TERM signal was received, indicating that the server should shut down gracefully.\n    GracefulTerminate,\n\n    /// The server is shutting down.\n    ShutdownStarted,\n\n    /// Waiting for the configured grace period to end before shutting down.\n    ShutdownGracePeriod,\n\n    /// Wait for runtimes to finish.\n    ShutdownRuntimes,\n\n    /// The server has stopped.\n    Terminated,\n}\n\n/// The receiver for server's shutdown event. The value will turn to true once the server starts\n/// to shutdown\npub type ShutdownWatch = watch::Receiver<bool>;\n#[cfg(unix)]\npub type ListenFds = Arc<TokioMutex<Fds>>;\n\n/// The type of shutdown process that has been requested.\n#[derive(Debug)]\npub enum ShutdownSignal {\n    /// Send file descriptors to the new process before starting runtime shutdown with\n    /// [ServerConf::graceful_shutdown_timeout_seconds] timeout.\n    GracefulUpgrade,\n    /// Wait for [ServerConf::grace_period_seconds] before starting runtime shutdown with\n    /// [ServerConf::graceful_shutdown_timeout_seconds] timeout.\n    GracefulTerminate,\n    /// Shutdown with no timeout for runtime shutdown.\n    FastShutdown,\n}\n\n/// Watcher of a shutdown signal, e.g., [UnixShutdownSignalWatch] for Unix-like\n/// platforms.\n#[async_trait]\npub trait ShutdownSignalWatch {\n    /// Returns the desired shutdown type once one has been requested.\n    async fn recv(&self) -> ShutdownSignal;\n}\n\n/// A Unix shutdown watcher that awaits for Unix signals.\n///\n/// - `SIGQUIT`: graceful upgrade\n/// - `SIGTERM`: graceful terminate\n/// - `SIGINT`: fast shutdown\n#[cfg(unix)]\npub struct UnixShutdownSignalWatch;\n\n#[cfg(unix)]\n#[async_trait]\nimpl ShutdownSignalWatch for UnixShutdownSignalWatch {\n    async fn recv(&self) -> ShutdownSignal {\n        let mut graceful_upgrade_signal = unix::signal(unix::SignalKind::quit()).unwrap();\n        let mut graceful_terminate_signal = unix::signal(unix::SignalKind::terminate()).unwrap();\n        let mut fast_shutdown_signal = unix::signal(unix::SignalKind::interrupt()).unwrap();\n\n        tokio::select! {\n            _ = graceful_upgrade_signal.recv() => {\n                ShutdownSignal::GracefulUpgrade\n            },\n            _ = graceful_terminate_signal.recv() => {\n                ShutdownSignal::GracefulTerminate\n            },\n            _ = fast_shutdown_signal.recv() => {\n                ShutdownSignal::FastShutdown\n            },\n        }\n    }\n}\n\n/// Arguments to configure running of the pingora server.\npub struct RunArgs {\n    /// Signal for initating shutdown\n    #[cfg(unix)]\n    pub shutdown_signal: Box<dyn ShutdownSignalWatch>,\n}\n\nimpl Default for RunArgs {\n    #[cfg(unix)]\n    fn default() -> Self {\n        Self {\n            shutdown_signal: Box::new(UnixShutdownSignalWatch),\n        }\n    }\n\n    #[cfg(windows)]\n    fn default() -> Self {\n        Self {}\n    }\n}\n\n/// The server object\n///\n/// This object represents an entire pingora server process which may have multiple independent\n/// services (see [crate::services]). The server object handles signals, reading configuration,\n/// zero downtime upgrade and error reporting.\npub struct Server {\n    // This is a way to add services that have to be run before any others\n    // without requiring dependencies to be set directly\n    init_services: Vec<Box<dyn ServiceWithDependents + 'static>>,\n\n    services: HashMap<NodeIndex, ServiceWrapper>,\n    shutdown_watch: watch::Sender<bool>,\n    // TODO: we many want to drop this copy to let sender call closed()\n    shutdown_recv: ShutdownWatch,\n\n    /// Tracks the execution phase of the server during upgrades and graceful shutdowns.\n    ///\n    /// Users can subscribe to the phase with [`Self::watch_execution_phase()`].\n    execution_phase_watch: broadcast::Sender<ExecutionPhase>,\n\n    /// Specification of service level dependencies\n    dependencies: Arc<Mutex<DependencyGraph>>,\n\n    /// Service initialization\n    bootstrap: Arc<Mutex<Bootstrap>>,\n\n    /// The parsed server configuration\n    pub configuration: Arc<ServerConf>,\n    /// The parser command line options\n    pub options: Option<Opt>,\n}\n\n// TODO: delete the pid when exit\n\nimpl Server {\n    /// Acquire a receiver for the server's execution phase.\n    ///\n    /// The receiver will produce values for each transition.\n    pub fn watch_execution_phase(&self) -> broadcast::Receiver<ExecutionPhase> {\n        self.execution_phase_watch.subscribe()\n    }\n\n    #[cfg(unix)]\n    async fn main_loop(&self, run_args: RunArgs) -> ShutdownType {\n        // waiting for exit signal\n\n        self.execution_phase_watch\n            .send(ExecutionPhase::Running)\n            .ok();\n\n        match run_args.shutdown_signal.recv().await {\n            ShutdownSignal::FastShutdown => {\n                info!(\"SIGINT received, exiting\");\n                ShutdownType::Quick\n            }\n            ShutdownSignal::GracefulTerminate => {\n                // we receive a graceful terminate, all instances are instructed to stop\n                info!(\"SIGTERM received, gracefully exiting\");\n                // graceful shutdown if there are listening sockets\n                info!(\"Broadcasting graceful shutdown\");\n                match self.shutdown_watch.send(true) {\n                    Ok(_) => {\n                        info!(\"Graceful shutdown started!\");\n                    }\n                    Err(e) => {\n                        error!(\"Graceful shutdown broadcast failed: {e}\");\n                    }\n                }\n                info!(\"Broadcast graceful shutdown complete\");\n\n                self.execution_phase_watch\n                    .send(ExecutionPhase::GracefulTerminate)\n                    .ok();\n\n                ShutdownType::Graceful\n            }\n            ShutdownSignal::GracefulUpgrade => {\n                // TODO: still need to select! on signals in case a fast shutdown is needed\n                // aka: move below to another task and only kick it off here\n                info!(\"SIGQUIT received, sending socks and gracefully exiting\");\n\n                self.execution_phase_watch\n                    .send(ExecutionPhase::GracefulUpgradeTransferringFds)\n                    .ok();\n\n                if let Some(fds) = self.listen_fds() {\n                    let fds = fds.lock().await;\n                    info!(\"Trying to send socks\");\n                    // XXX: this is blocking IO\n                    match fds.send_to_sock(self.configuration.as_ref().upgrade_sock.as_str()) {\n                        Ok(_) => {\n                            info!(\"listener sockets sent\");\n                        }\n                        Err(e) => {\n                            error!(\"Unable to send listener sockets to new process: {e}\");\n                            // sentry log error on fd send failure\n                            #[cfg(all(not(debug_assertions), feature = \"sentry\"))]\n                            sentry::capture_error(&e);\n                        }\n                    }\n                    self.execution_phase_watch\n                        .send(ExecutionPhase::GracefulUpgradeCloseTimeout)\n                        .ok();\n                    sleep(Duration::from_secs(CLOSE_TIMEOUT)).await;\n                    info!(\"Broadcasting graceful shutdown\");\n                    // gracefully exiting\n                    match self.shutdown_watch.send(true) {\n                        Ok(_) => {\n                            info!(\"Graceful shutdown started!\");\n                        }\n                        Err(e) => {\n                            error!(\"Graceful shutdown broadcast failed: {e}\");\n                            // switch to fast shutdown\n                            return ShutdownType::Graceful;\n                        }\n                    }\n                    info!(\"Broadcast graceful shutdown complete\");\n                    ShutdownType::Graceful\n                } else {\n                    info!(\"No socks to send, shutting down.\");\n                    ShutdownType::Graceful\n                }\n            }\n        }\n    }\n\n    #[cfg(windows)]\n    async fn main_loop(&self, _run_args: RunArgs) -> ShutdownType {\n        // waiting for exit signal\n\n        self.execution_phase_watch\n            .send(ExecutionPhase::Running)\n            .ok();\n\n        match tokio::signal::ctrl_c().await {\n            Ok(()) => {\n                info!(\"Ctrl+C received, gracefully exiting\");\n                // graceful shutdown if there are listening sockets\n                info!(\"Broadcasting graceful shutdown\");\n                match self.shutdown_watch.send(true) {\n                    Ok(_) => {\n                        info!(\"Graceful shutdown started!\");\n                    }\n                    Err(e) => {\n                        error!(\"Graceful shutdown broadcast failed: {e}\");\n                    }\n                }\n                info!(\"Broadcast graceful shutdown complete\");\n\n                self.execution_phase_watch\n                    .send(ExecutionPhase::GracefulTerminate)\n                    .ok();\n\n                ShutdownType::Graceful\n            }\n            Err(e) => {\n                error!(\"Unable to listen for shutdown signal: {}\", e);\n                ShutdownType::Quick\n            }\n        }\n    }\n\n    #[cfg(feature = \"sentry\")]\n    #[cfg_attr(docsrs, doc(cfg(feature = \"sentry\")))]\n    /// The Sentry ClientOptions.\n    ///\n    /// Panics and other events sentry captures will be sent to this DSN **only in release mode**\n    pub fn set_sentry_config(&mut self, sentry_config: ClientOptions) {\n        self.bootstrap.lock().set_sentry_config(Some(sentry_config));\n    }\n\n    /// Get the configured file descriptors for listening\n    #[cfg(unix)]\n    fn listen_fds(&self) -> Option<ListenFds> {\n        self.bootstrap.lock().get_fds()\n    }\n\n    #[allow(clippy::too_many_arguments)]\n    fn run_service(\n        mut service: Box<dyn ServiceWithDependents>,\n        #[cfg(unix)] fds: Option<ListenFds>,\n        shutdown: ShutdownWatch,\n        threads: usize,\n        work_stealing: bool,\n        listeners_per_fd: usize,\n        ready_notifier: ServiceReadyNotifier,\n        dependency_watches: Vec<ServiceReadyWatch>,\n    ) -> Runtime\n// NOTE: we need to keep the runtime outside async since\n        // otherwise the runtime will be dropped.\n    {\n        let service_runtime = Server::create_runtime(service.name(), threads, work_stealing);\n        let service_name = service.name().to_string();\n        service_runtime.get_handle().spawn(async move {\n            // Wait for all dependencies to be ready\n            let mut time_waited_opt: Option<Duration> = None;\n            for mut watch in dependency_watches {\n                let start = SystemTime::now();\n\n                if watch.wait_for(|&ready| ready).await.is_err() {\n                    error!(\n                        \"Service '{}' dependency channel closed before ready\",\n                        service_name\n                    );\n                }\n\n                *time_waited_opt.get_or_insert_default() += start.elapsed().unwrap_or_default()\n            }\n\n            if let Some(time_waited) = time_waited_opt {\n                service.on_startup_delay(time_waited);\n            }\n\n            // Start the actual service, passing the ready notifier\n            service\n                .start_service(\n                    #[cfg(unix)]\n                    fds,\n                    shutdown,\n                    listeners_per_fd,\n                    ready_notifier,\n                )\n                .await;\n            info!(\"service '{}' exited.\", service_name);\n        });\n        service_runtime\n    }\n\n    /// Create a new [`Server`], using the [`Opt`] and [`ServerConf`] values provided\n    ///\n    /// This method is intended for pingora frontends that are NOT using the built-in\n    /// command line and configuration file parsing, and are instead using their own.\n    ///\n    /// If a configuration file path is provided as part of `opt`, it will be ignored\n    /// and a warning will be logged.\n    pub fn new_with_opt_and_conf(raw_opt: impl Into<Option<Opt>>, mut conf: ServerConf) -> Server {\n        let opt = raw_opt.into();\n        if let Some(opts) = &opt {\n            if let Some(c) = opts.conf.as_ref() {\n                warn!(\"Ignoring command line argument using '{c}' as configuration, and using provided configuration instead.\");\n            }\n            conf.merge_with_opt(opts);\n        }\n\n        let (tx, rx) = watch::channel(false);\n\n        let execution_phase_watch = broadcast::channel(100).0;\n        let bootstrap = Arc::new(Mutex::new(Bootstrap::new(\n            &opt,\n            &conf,\n            &execution_phase_watch,\n        )));\n\n        Server {\n            services: Default::default(),\n            init_services: Default::default(),\n            shutdown_watch: tx,\n            shutdown_recv: rx,\n            execution_phase_watch,\n            configuration: Arc::new(conf),\n            options: opt,\n            dependencies: Arc::new(Mutex::new(DependencyGraph::new())),\n            bootstrap,\n        }\n    }\n\n    /// Create a new [`Server`].\n    ///\n    /// Only one [`Server`] needs to be created for a process. A [`Server`] can hold multiple\n    /// independent services.\n    ///\n    /// Command line options can either be passed by parsing the command line arguments via\n    /// `Opt::parse_args()`, or be generated by other means.\n    pub fn new(opt: impl Into<Option<Opt>>) -> Result<Server> {\n        let opt = opt.into();\n        let (tx, rx) = watch::channel(false);\n\n        let execution_phase_watch = broadcast::channel(100).0;\n        let conf = if let Some(opt) = opt.as_ref() {\n            opt.conf.as_ref().map_or_else(\n                || {\n                    // options, no conf, generated\n                    ServerConf::new_with_opt_override(opt).ok_or_else(|| {\n                        Error::explain(ErrorType::ReadError, \"Conf generation failed\")\n                    })\n                },\n                |_| {\n                    // options and conf loaded\n                    ServerConf::load_yaml_with_opt_override(opt)\n                },\n            )\n        } else {\n            ServerConf::new()\n                .ok_or_else(|| Error::explain(ErrorType::ReadError, \"Conf generation failed\"))\n        }?;\n\n        let bootstrap = Arc::new(Mutex::new(Bootstrap::new(\n            &opt,\n            &conf,\n            &execution_phase_watch,\n        )));\n\n        Ok(Server {\n            services: Default::default(),\n            init_services: Default::default(),\n            shutdown_watch: tx,\n            shutdown_recv: rx,\n            execution_phase_watch,\n            configuration: Arc::new(conf),\n            options: opt,\n            dependencies: Arc::new(Mutex::new(DependencyGraph::new())),\n            bootstrap,\n        })\n    }\n\n    /// Add a service that all other services will wait on before starting.\n    fn add_init_service(&mut self, service: impl ServiceWithDependents + 'static) {\n        let boxed_service = Box::new(service);\n        self.init_services.push(boxed_service);\n    }\n\n    /// Add the init services as dependencies for all existing services\n    fn apply_init_service_dependencies(&mut self) {\n        let services = self\n            .services\n            .values()\n            .map(|service| service.service_handle.clone())\n            .collect::<Vec<_>>();\n        let global_deps = self\n            .init_services\n            .drain(..)\n            .collect::<Vec<_>>()\n            .into_iter()\n            .map(|dep| self.add_boxed_service(dep))\n            .collect::<Vec<_>>();\n        for service in services {\n            service.add_dependencies(&global_deps);\n        }\n    }\n\n    /// Add a service to this server.\n    ///\n    /// Returns a [`ServiceHandle`] that can be used to declare dependencies.\n    ///\n    /// # Example\n    ///\n    /// ```rust,ignore\n    /// let db_id = server.add_service(database_service);\n    /// let api_id = server.add_service(api_service);\n    ///\n    /// // Declare that API depends on database\n    /// api_id.add_dependency(&db_id);\n    /// ```\n    pub fn add_service(&mut self, service: impl ServiceWithDependents + 'static) -> ServiceHandle {\n        self.add_boxed_service(Box::new(service))\n    }\n\n    /// Add a pre-boxed service to this server.\n    ///\n    /// Returns a [`ServiceHandle`] that can be used to declare dependencies.\n    ///\n    /// # Example\n    ///\n    /// ```rust,ignore\n    /// let db_id = server.add_service(database_service);\n    /// let api_id = server.add_service(api_service);\n    ///\n    /// // Declare that API depends on database\n    /// api_id.add_dependency(&db_id);\n    /// ```\n    pub fn add_boxed_service(\n        &mut self,\n        service_box: Box<dyn ServiceWithDependents>,\n    ) -> ServiceHandle {\n        let name = service_box.name().to_string();\n\n        // Create a readiness notifier for this service\n        let (tx, rx) = watch::channel(false);\n\n        let id = self.dependencies.lock().add_node(name.clone(), rx.clone());\n\n        let service_handle = ServiceHandle::new(id, name, rx, &self.dependencies);\n\n        let wrapper = ServiceWrapper {\n            ready_notifier: Some(ServiceReadyNotifier::new(tx)),\n            service: service_box,\n            service_handle: service_handle.clone(),\n        };\n\n        self.services.insert(id, wrapper);\n\n        service_handle\n    }\n\n    /// Similar to [`Self::add_service()`], but take a list of services.\n    ///\n    /// Returns a `Vec<ServiceHandle>` for all added services.\n    pub fn add_services(\n        &mut self,\n        services: Vec<Box<dyn ServiceWithDependents>>,\n    ) -> Vec<ServiceHandle> {\n        services\n            .into_iter()\n            .map(|service| self.add_boxed_service(service))\n            .collect()\n    }\n\n    /// Prepare the server to start\n    ///\n    /// When trying to zero downtime upgrade from an older version of the server which is already\n    /// running, this function will try to get all its listening sockets in order to take them over.\n    pub fn bootstrap(&mut self) {\n        self.bootstrap.lock().bootstrap();\n    }\n\n    /// Create a service that will run to prepare the service to start\n    ///\n    /// The created service will handle the zero-downtime upgrade from an older version of the server\n    /// to this one. It will try to get all its listening sockets in order to take them over.\n    ///\n    /// Other bootstrapping functionality like sentry initialization will also be handled, but as a\n    /// service that will complete before any other service starts.\n    pub fn bootstrap_as_a_service(&mut self) -> ServiceHandle {\n        let bootstrap_service =\n            background_service(\"Bootstrap Service\", BootstrapService::new(&self.bootstrap));\n\n        let sentry_service = background_service(\n            \"Sentry Init Service\",\n            SentryInitService::new(&self.bootstrap),\n        );\n\n        self.add_init_service(sentry_service);\n\n        self.add_service(bootstrap_service)\n    }\n\n    /// Start the server using [Self::run] and default [RunArgs].\n    ///\n    /// This function will block forever until the server needs to quit. So this would be the last\n    /// function to call for this object.\n    ///\n    /// Note: this function may fork the process for daemonization, so any additional threads created\n    /// before this function will be lost to any service logic once this function is called.\n    pub fn run_forever(self) -> ! {\n        self.run(RunArgs::default());\n\n        std::process::exit(0)\n    }\n\n    /// Run the server until execution finished.\n    ///\n    /// This function will run until the server has been instructed to shut down\n    /// through a signal, and will then wait for all services to finish and\n    /// runtimes to exit.\n    ///\n    /// Note: if daemonization is enabled in the config, this function will\n    /// never return.\n    /// Instead it will either start the daemon process and exit, or panic\n    /// if daemonization fails.\n    pub fn run(mut self, run_args: RunArgs) {\n        self.apply_init_service_dependencies();\n\n        info!(\"Server starting\");\n\n        let conf = self.configuration.as_ref();\n\n        #[cfg(unix)]\n        if conf.daemon {\n            info!(\"Daemonizing the server\");\n            fast_timeout::pause_for_fork();\n            daemonize(&self.configuration);\n            fast_timeout::unpause();\n        }\n\n        #[cfg(windows)]\n        if conf.daemon {\n            panic!(\"Daemonizing under windows is not supported\");\n        }\n\n        // Holds tuples of runtimes and their service name.\n        let mut runtimes: Vec<(Runtime, String)> = Vec::new();\n\n        // Get services in topological order (dependencies first)\n        let startup_order = match self.dependencies.lock().topological_sort() {\n            Ok(order) => order,\n            Err(e) => {\n                error!(\"Failed to determine service startup order: {}\", e);\n                std::process::exit(1);\n            }\n        };\n\n        // Log service names in startup order\n        let service_names: Vec<String> = startup_order\n            .iter()\n            .map(|(_, service)| service.name.clone())\n            .collect();\n        info!(\"Starting services in dependency order: {:?}\", service_names);\n\n        // Start services in dependency order\n        for (service_id, service) in startup_order {\n            let mut wrapper = match self.services.remove(&service_id) {\n                Some(w) => w,\n                None => {\n                    warn!(\n                        \"Service ID {:?}-{} in startup order but not found\",\n                        service_id, service.name\n                    );\n                    continue;\n                }\n            };\n\n            let threads = wrapper.service.threads().unwrap_or(conf.threads);\n            let name = wrapper.service.name().to_string();\n\n            // Extract dependency watches from the ServiceHandle\n            let dependencies = self\n                .dependencies\n                .lock()\n                .get_dependencies(wrapper.service_handle.id);\n\n            // Get the readiness notifier for this service by taking it from the Option.\n            // Since service_id is the index, we can directly access it.\n            // We take() the notifier, leaving None in its place.\n            let ready_notifier = wrapper\n                .ready_notifier\n                .take()\n                .expect(\"Service notifier should exist\");\n\n            if !dependencies.is_empty() {\n                info!(\n                    \"Service '{name}' will wait for dependencies: {:?}\",\n                    dependencies.iter().map(|s| &s.name).collect::<Vec<_>>()\n                );\n            } else {\n                info!(\"Starting service: {}\", name);\n            }\n\n            let dependency_watches = dependencies\n                .iter()\n                .map(|s| s.ready_watch.clone())\n                .collect::<Vec<_>>();\n\n            let runtime = Server::run_service(\n                wrapper.service,\n                #[cfg(unix)]\n                self.listen_fds(),\n                self.shutdown_recv.clone(),\n                threads,\n                conf.work_stealing,\n                self.configuration.listener_tasks_per_fd,\n                ready_notifier,\n                dependency_watches,\n            );\n            runtimes.push((runtime, name));\n        }\n\n        // blocked on main loop so that it runs forever\n        // Only work steal runtime can use block_on()\n        let server_runtime = Server::create_runtime(\"Server\", 1, true);\n        #[cfg(unix)]\n        let shutdown_type = server_runtime\n            .get_handle()\n            .block_on(self.main_loop(run_args));\n        #[cfg(windows)]\n        let shutdown_type = server_runtime\n            .get_handle()\n            .block_on(self.main_loop(run_args));\n\n        self.execution_phase_watch\n            .send(ExecutionPhase::ShutdownStarted)\n            .ok();\n\n        if matches!(shutdown_type, ShutdownType::Graceful) {\n            self.execution_phase_watch\n                .send(ExecutionPhase::ShutdownGracePeriod)\n                .ok();\n\n            let exit_timeout = self\n                .configuration\n                .as_ref()\n                .grace_period_seconds\n                .unwrap_or(EXIT_TIMEOUT);\n            info!(\"Graceful shutdown: grace period {}s starts\", exit_timeout);\n            thread::sleep(Duration::from_secs(exit_timeout));\n            info!(\"Graceful shutdown: grace period ends\");\n        }\n\n        // Give tokio runtimes time to exit\n        let shutdown_timeout = match shutdown_type {\n            ShutdownType::Quick => Duration::from_secs(0),\n            ShutdownType::Graceful => Duration::from_secs(\n                self.configuration\n                    .as_ref()\n                    .graceful_shutdown_timeout_seconds\n                    .unwrap_or(5),\n            ),\n        };\n\n        self.execution_phase_watch\n            .send(ExecutionPhase::ShutdownRuntimes)\n            .ok();\n\n        let shutdowns: Vec<_> = runtimes\n            .into_iter()\n            .map(|(rt, name)| {\n                info!(\"Waiting for runtimes to exit!\");\n                let join = thread::spawn(move || {\n                    rt.shutdown_timeout(shutdown_timeout);\n                    thread::sleep(shutdown_timeout)\n                });\n                (join, name)\n            })\n            .collect();\n        for (shutdown, name) in shutdowns {\n            info!(\"Waiting for service runtime {} to exit\", name);\n            if let Err(e) = shutdown.join() {\n                error!(\"Failed to shutdown service runtime {}: {:?}\", name, e);\n            }\n            debug!(\"Service runtime {} has exited\", name);\n        }\n        info!(\"All runtimes exited, exiting now\");\n\n        self.execution_phase_watch\n            .send(ExecutionPhase::Terminated)\n            .ok();\n    }\n\n    fn create_runtime(name: &str, threads: usize, work_steal: bool) -> Runtime {\n        if work_steal {\n            Runtime::new_steal(threads, name)\n        } else {\n            Runtime::new_no_steal(threads, name)\n        }\n    }\n}\n"
  },
  {
    "path": "pingora-core/src/server/transfer_fd/mod.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n#[cfg(target_os = \"linux\")]\nuse log::{debug, error, warn};\nuse nix::errno::Errno;\n#[cfg(target_os = \"linux\")]\nuse nix::sys::socket::{self, AddressFamily, RecvMsg, SockFlag, SockType, UnixAddr};\n#[cfg(target_os = \"linux\")]\nuse nix::sys::stat;\nuse nix::{Error, NixPath};\nuse std::collections::HashMap;\nuse std::io::Write;\n#[cfg(target_os = \"linux\")]\nuse std::io::{IoSlice, IoSliceMut};\nuse std::os::unix::io::RawFd;\n#[cfg(target_os = \"linux\")]\nuse std::{thread, time};\n\n// Utilities to transfer file descriptors between sockets, e.g. during graceful upgrades.\n\n/// Container for open file descriptors and their associated bind addresses.\npub struct Fds {\n    map: HashMap<String, RawFd>,\n}\n\nimpl Fds {\n    pub fn new() -> Self {\n        Fds {\n            map: HashMap::new(),\n        }\n    }\n\n    pub fn add(&mut self, bind: String, fd: RawFd) {\n        self.map.insert(bind, fd);\n    }\n\n    pub fn get(&self, bind: &str) -> Option<&RawFd> {\n        self.map.get(bind)\n    }\n\n    pub fn serialize(&self) -> (Vec<String>, Vec<RawFd>) {\n        self.map.iter().map(|(key, val)| (key.clone(), val)).unzip()\n    }\n\n    pub fn deserialize(&mut self, binds: Vec<String>, fds: Vec<RawFd>) {\n        assert_eq!(binds.len(), fds.len());\n        for (bind, fd) in binds.into_iter().zip(fds) {\n            self.map.insert(bind, fd);\n        }\n    }\n\n    pub fn send_to_sock<P>(&self, path: &P) -> Result<usize, Error>\n    where\n        P: ?Sized + NixPath + std::fmt::Display,\n    {\n        let (vec_key, vec_fds) = self.serialize();\n        let mut ser_buf: [u8; 2048] = [0; 2048];\n        let ser_key_size = serialize_vec_string(&vec_key, &mut ser_buf);\n        send_fds_to(vec_fds, &ser_buf[..ser_key_size], path, None)\n    }\n\n    pub fn get_from_sock<P>(&mut self, path: &P) -> Result<(), Error>\n    where\n        P: ?Sized + NixPath + std::fmt::Display,\n    {\n        let mut de_buf: [u8; 2048] = [0; 2048];\n        let (fds, bytes) = get_fds_from(path, &mut de_buf, None)?;\n        let keys = deserialize_vec_string(&de_buf[..bytes])?;\n        self.deserialize(keys, fds);\n        Ok(())\n    }\n}\n\nfn serialize_vec_string(vec_string: &[String], mut buf: &mut [u8]) -> usize {\n    // There are many ways to do this. Serde is probably the way to go\n    // But let's start with something simple: space separated strings\n    let joined = vec_string.join(\" \");\n    // TODO: check the buf is large enough\n    buf.write(joined.as_bytes()).unwrap()\n}\n\nfn deserialize_vec_string(buf: &[u8]) -> Result<Vec<String>, Error> {\n    let joined = std::str::from_utf8(buf).map_err(|_| Error::EINVAL)?;\n    Ok(joined.split_ascii_whitespace().map(String::from).collect())\n}\n\n#[cfg(target_os = \"linux\")]\npub fn get_fds_from<P>(\n    path: &P,\n    payload: &mut [u8],\n    max_retry: Option<usize>,\n) -> Result<(Vec<RawFd>, usize), Error>\nwhere\n    P: ?Sized + NixPath + std::fmt::Display,\n{\n    let max_retry = max_retry.unwrap_or(MAX_RETRY);\n    const MAX_FDS: usize = 32;\n\n    let listen_fd = socket::socket(\n        AddressFamily::Unix,\n        SockType::Stream,\n        SockFlag::SOCK_NONBLOCK,\n        None,\n    )\n    .unwrap();\n    let unix_addr = UnixAddr::new(path).unwrap();\n    // clean up old sock\n    match nix::unistd::unlink(path) {\n        Ok(()) => {\n            debug!(\"unlink {} done\", path);\n        }\n        Err(e) => {\n            // Normal if file does not exist\n            debug!(\"unlink {} failed: {}\", path, e);\n            // TODO: warn if exist but not able to unlink\n        }\n    };\n    socket::bind(listen_fd, &unix_addr).unwrap();\n\n    /* sock is created before we change user, need to give permission */\n    stat::fchmodat(\n        None,\n        path,\n        stat::Mode::from_bits_truncate(0o666),\n        stat::FchmodatFlags::FollowSymlink,\n    )\n    .unwrap();\n\n    socket::listen(listen_fd, 8).unwrap();\n\n    let fd = match accept_with_retry_timeout(listen_fd, max_retry) {\n        Ok(fd) => fd,\n        Err(e) => {\n            error!(\"Giving up reading socket from: {path}, error: {e:?}\");\n            //cleanup\n            if nix::unistd::close(listen_fd).is_ok() {\n                nix::unistd::unlink(path).unwrap();\n            }\n            return Err(e);\n        }\n    };\n\n    let mut io_vec = [IoSliceMut::new(payload); 1];\n    let mut cmsg_buf = nix::cmsg_space!([RawFd; MAX_FDS]);\n    let msg: RecvMsg<UnixAddr> = socket::recvmsg(\n        fd,\n        &mut io_vec,\n        Some(&mut cmsg_buf),\n        socket::MsgFlags::empty(),\n    )\n    .unwrap();\n\n    let mut fds: Vec<RawFd> = Vec::new();\n    for cmsg in msg.cmsgs() {\n        if let socket::ControlMessageOwned::ScmRights(mut vec_fds) = cmsg {\n            fds.append(&mut vec_fds)\n        } else {\n            warn!(\"Unexpected control messages: {cmsg:?}\")\n        }\n    }\n\n    //cleanup\n    if nix::unistd::close(listen_fd).is_ok() {\n        nix::unistd::unlink(path).unwrap();\n    }\n\n    Ok((fds, msg.bytes))\n}\n\n#[cfg(not(target_os = \"linux\"))]\npub fn get_fds_from<P>(\n    _path: &P,\n    _payload: &mut [u8],\n    _max_retry: Option<usize>,\n) -> Result<(Vec<RawFd>, usize), Error>\nwhere\n    P: ?Sized + NixPath + std::fmt::Display,\n{\n    log::error!(\"Upgrade is not currently supported outside of Linux platforms\");\n    Err(Errno::ECONNREFUSED)\n}\n\n#[cfg(target_os = \"linux\")]\nconst MAX_RETRY: usize = 5;\n#[cfg(target_os = \"linux\")]\nconst RETRY_INTERVAL: time::Duration = time::Duration::from_secs(1);\n\n#[cfg(target_os = \"linux\")]\nfn accept_with_retry_timeout(listen_fd: i32, max_retry: usize) -> Result<i32, Error> {\n    let mut retried = 0;\n    loop {\n        match socket::accept(listen_fd) {\n            Ok(fd) => return Ok(fd),\n            Err(e) => {\n                if retried > max_retry {\n                    return Err(e);\n                }\n                match e {\n                    Errno::EAGAIN => {\n                        error!(\n                            \"No incoming socket transfer, sleep {RETRY_INTERVAL:?} and try again\"\n                        );\n                        retried += 1;\n                        thread::sleep(RETRY_INTERVAL);\n                    }\n                    _ => {\n                        error!(\"Error accepting socket transfer: {e}\");\n                        return Err(e);\n                    }\n                }\n            }\n        }\n    }\n}\n\n#[cfg(target_os = \"linux\")]\npub fn send_fds_to<P>(\n    fds: Vec<RawFd>,\n    payload: &[u8],\n    path: &P,\n    max_retry: Option<usize>,\n) -> Result<usize, Error>\nwhere\n    P: ?Sized + NixPath + std::fmt::Display,\n{\n    let max_retry = max_retry.unwrap_or(MAX_RETRY);\n    const MAX_NONBLOCKING_POLLS: usize = 20;\n    const NONBLOCKING_POLL_INTERVAL: time::Duration = time::Duration::from_millis(500);\n\n    let send_fd = socket::socket(\n        AddressFamily::Unix,\n        SockType::Stream,\n        SockFlag::SOCK_NONBLOCK,\n        None,\n    )?;\n    let unix_addr = UnixAddr::new(path)?;\n    let mut retried = 0;\n    let mut nonblocking_polls = 0;\n\n    let conn_result: Result<usize, Error> = loop {\n        match socket::connect(send_fd, &unix_addr) {\n            Ok(_) => break Ok(0),\n            Err(e) => match e {\n                /* If the new process hasn't created the upgrade sock we'll get an ENOENT.\n                ECONNREFUSED may happen if the sock wasn't cleaned up\n                and the old process tries sending before the new one is listening.\n                EACCES may happen if connect() happen before the correct permission is set */\n                Errno::ENOENT | Errno::ECONNREFUSED | Errno::EACCES => {\n                    /*the server is not ready yet*/\n                    retried += 1;\n                    if retried > max_retry {\n                        error!(\n                            \"Max retry: {} reached. Giving up sending socket to: {}, error: {:?}\",\n                            max_retry, path, e\n                        );\n                        break Err(e);\n                    }\n                    warn!(\"server not ready, will try again in {RETRY_INTERVAL:?}\");\n                    thread::sleep(RETRY_INTERVAL);\n                }\n                /* handle nonblocking IO */\n                Errno::EINPROGRESS => {\n                    nonblocking_polls += 1;\n                    if nonblocking_polls >= MAX_NONBLOCKING_POLLS {\n                        error!(\"Connect() not ready after retries when sending socket to: {path}\",);\n                        break Err(e);\n                    }\n                    warn!(\"Connect() not ready, will try again in {NONBLOCKING_POLL_INTERVAL:?}\",);\n                    thread::sleep(NONBLOCKING_POLL_INTERVAL);\n                }\n                _ => {\n                    error!(\"Error sending socket to: {path}, error: {e:?}\");\n                    break Err(e);\n                }\n            },\n        }\n    };\n\n    let result = match conn_result {\n        Ok(_) => {\n            let io_vec = [IoSlice::new(payload); 1];\n            let scm = socket::ControlMessage::ScmRights(fds.as_slice());\n            let cmsg = [scm; 1];\n            loop {\n                match socket::sendmsg(\n                    send_fd,\n                    &io_vec,\n                    &cmsg,\n                    socket::MsgFlags::empty(),\n                    None::<&UnixAddr>,\n                ) {\n                    Ok(result) => break Ok(result),\n                    Err(e) => match e {\n                        /* handle nonblocking IO */\n                        Errno::EAGAIN => {\n                            nonblocking_polls += 1;\n                            if nonblocking_polls >= MAX_NONBLOCKING_POLLS {\n                                error!(\n                                    \"Sendmsg() not ready after retries when sending socket to: {}\",\n                                    path\n                                );\n                                break Err(e);\n                            }\n                            warn!(\n                                \"Sendmsg() not ready, will try again in {:?}\",\n                                NONBLOCKING_POLL_INTERVAL\n                            );\n                            thread::sleep(NONBLOCKING_POLL_INTERVAL);\n                        }\n                        _ => break Err(e),\n                    },\n                }\n            }\n        }\n        Err(_) => conn_result,\n    };\n\n    nix::unistd::close(send_fd).unwrap();\n    result\n}\n\n#[cfg(not(target_os = \"linux\"))]\npub fn send_fds_to<P>(\n    _fds: Vec<RawFd>,\n    _payload: &[u8],\n    _path: &P,\n    _max_retry: Option<usize>,\n) -> Result<usize, Error>\nwhere\n    P: ?Sized + NixPath + std::fmt::Display,\n{\n    Ok(0)\n}\n\n#[cfg(test)]\n#[cfg(target_os = \"linux\")]\nmod tests {\n    use super::*;\n    use log::{debug, error};\n\n    fn init_log() {\n        let _ = env_logger::builder().is_test(true).try_init();\n    }\n\n    #[test]\n    fn test_add_get() {\n        init_log();\n        let mut fds = Fds::new();\n        let key = \"1.1.1.1:80\".to_string();\n        fds.add(key.clone(), 128);\n        assert_eq!(128, *fds.get(&key).unwrap());\n    }\n\n    #[test]\n    fn test_table_serde() {\n        init_log();\n        let mut fds = Fds::new();\n        let key1 = \"1.1.1.1:80\".to_string();\n        fds.add(key1.clone(), 128);\n        let key2 = \"1.1.1.1:443\".to_string();\n        fds.add(key2.clone(), 129);\n\n        let (k, v) = fds.serialize();\n        let mut fds2 = Fds::new();\n        fds2.deserialize(k, v);\n\n        assert_eq!(128, *fds2.get(&key1).unwrap());\n        assert_eq!(129, *fds2.get(&key2).unwrap());\n    }\n\n    #[test]\n    fn test_vec_string_serde() {\n        init_log();\n        let vec_str: Vec<String> = vec![\"aaaa\".to_string(), \"bbb\".to_string()];\n        let mut ser_buf: [u8; 1024] = [0; 1024];\n        let size = serialize_vec_string(&vec_str, &mut ser_buf);\n        let de_vec_string = deserialize_vec_string(&ser_buf[..size]).unwrap();\n        assert_eq!(de_vec_string.len(), 2);\n        assert_eq!(de_vec_string[0], \"aaaa\");\n        assert_eq!(de_vec_string[1], \"bbb\");\n    }\n\n    #[test]\n    fn test_send_receive_fds() {\n        init_log();\n        let dumb_fd = socket::socket(\n            AddressFamily::Unix,\n            SockType::Stream,\n            SockFlag::empty(),\n            None,\n        )\n        .unwrap();\n\n        // receiver need to start in another thread since it is blocking\n        let child = thread::spawn(move || {\n            let mut buf: [u8; 32] = [0; 32];\n            let (fds, bytes) =\n                get_fds_from(\"/tmp/pingora_fds_receive.sock\", &mut buf, None).unwrap();\n            debug!(\"{:?}\", fds);\n            assert_eq!(1, fds.len());\n            assert_eq!(32, bytes);\n            assert_eq!(1, buf[0]);\n            assert_eq!(1, buf[31]);\n        });\n\n        let fds = vec![dumb_fd];\n        let buf: [u8; 128] = [1; 128];\n        match send_fds_to(fds, &buf, \"/tmp/pingora_fds_receive.sock\", None) {\n            Ok(sent) => {\n                assert!(sent > 0);\n            }\n            Err(e) => {\n                error!(\"{:?}\", e);\n                panic!()\n            }\n        }\n\n        child.join().unwrap();\n    }\n\n    #[test]\n    fn test_serde_via_socket() {\n        init_log();\n        let mut fds = Fds::new();\n        let key1 = \"1.1.1.1:80\".to_string();\n        let dumb_fd1 = socket::socket(\n            AddressFamily::Unix,\n            SockType::Stream,\n            SockFlag::empty(),\n            None,\n        )\n        .unwrap();\n        fds.add(key1.clone(), dumb_fd1);\n        let key2 = \"1.1.1.1:443\".to_string();\n        let dumb_fd2 = socket::socket(\n            AddressFamily::Unix,\n            SockType::Stream,\n            SockFlag::empty(),\n            None,\n        )\n        .unwrap();\n        fds.add(key2.clone(), dumb_fd2);\n\n        let child = thread::spawn(move || {\n            let mut fds2 = Fds::new();\n            fds2.get_from_sock(\"/tmp/pingora_fds_receive2.sock\")\n                .unwrap();\n            assert!(*fds2.get(&key1).unwrap() > 0);\n            assert!(*fds2.get(&key2).unwrap() > 0);\n        });\n\n        fds.send_to_sock(\"/tmp/pingora_fds_receive2.sock\").unwrap();\n        child.join().unwrap();\n    }\n\n    #[test]\n    fn test_send_fds_to_respects_configurable_timeout() {\n        init_log();\n        use std::time::Instant;\n\n        let dumb_fd = socket::socket(\n            AddressFamily::Unix,\n            SockType::Stream,\n            SockFlag::empty(),\n            None,\n        )\n        .unwrap();\n\n        let fds = vec![dumb_fd];\n        let buf: [u8; 32] = [1; 32];\n\n        // Try to send with a custom max_retries of 2\n        let start = Instant::now();\n        let result = send_fds_to(fds, &buf, \"/tmp/pingora_test_config_send.sock\", Some(2));\n        let elapsed = start.elapsed();\n\n        // Should fail after 2 retries with RETRY_INTERVAL (1 second) between each\n        // Total time should be approximately 2 seconds\n        assert!(result.is_err());\n        assert!(\n            elapsed.as_secs() >= 2,\n            \"Expected at least 2 seconds, got {:?}\",\n            elapsed\n        );\n        assert!(\n            elapsed.as_secs() < 4,\n            \"Expected less than 4 seconds, got {:?}\",\n            elapsed\n        );\n    }\n\n    #[test]\n    fn test_get_fds_from_respects_configurable_timeout() {\n        init_log();\n        use std::time::Instant;\n\n        let mut buf: [u8; 32] = [0; 32];\n\n        // Try to receive with a custom max_retries of 2\n        let start = Instant::now();\n        let result = get_fds_from(\"/tmp/pingora_test_config_receive.sock\", &mut buf, Some(2));\n        let elapsed = start.elapsed();\n\n        // Should fail after 2 retries with RETRY_INTERVAL (1 second) between each\n        // Total time should be approximately 2 seconds\n        assert!(result.is_err());\n        assert!(\n            elapsed.as_secs() >= 2,\n            \"Expected at least 2 seconds, got {:?}\",\n            elapsed\n        );\n        assert!(\n            elapsed.as_secs() < 4,\n            \"Expected less than 4 seconds, got {:?}\",\n            elapsed\n        );\n    }\n}\n"
  },
  {
    "path": "pingora-core/src/services/background.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! The background service\n//!\n//! A [BackgroundService] can be run as part of a Pingora application to add supporting logic that\n//! exists outside of the request/response lifecycle.\n//! Examples might include service discovery (load balancing) and background updates such as\n//! push-style metrics.\n\nuse async_trait::async_trait;\nuse std::sync::Arc;\n\nuse super::{ServiceReadyNotifier, ServiceWithDependents};\n#[cfg(unix)]\nuse crate::server::ListenFds;\nuse crate::server::ShutdownWatch;\n\n/// The background service interface\n///\n/// You can implement a background service with or without the ready notifier,\n/// but you shouldn't implement both. Under the hood, the pingora service will\n/// call the `start_with_ready_notifier` function. By default this function will\n/// call the regular `start` function.\n#[async_trait]\npub trait BackgroundService {\n    /// This function is called when the pingora server tries to start all the\n    /// services. The background service should signal readiness by calling\n    /// `ready_notifier.notify_ready()` once initialization is complete.\n    /// The service can return at anytime or wait for the `shutdown` signal.\n    ///\n    /// By default this method will immediately signal readiness and call\n    /// through to the regular `start` function\n    async fn start_with_ready_notifier(\n        &self,\n        shutdown: ShutdownWatch,\n        ready_notifier: ServiceReadyNotifier,\n    ) {\n        ready_notifier.notify_ready();\n        self.start(shutdown).await;\n    }\n\n    /// This function is called when the pingora server tries to start all the\n    /// services. The background service can return at anytime or wait for the\n    /// `shutdown` signal.\n    async fn start(&self, mut _shutdown: ShutdownWatch) {}\n}\n\n/// A generic type of background service\npub struct GenBackgroundService<A> {\n    // Name of the service\n    name: String,\n    // Task the service will execute\n    task: Arc<A>,\n    /// The number of threads. Default is 1\n    pub threads: Option<usize>,\n}\n\nimpl<A> GenBackgroundService<A> {\n    /// Generates a background service that can run in the pingora runtime\n    pub fn new(name: String, task: Arc<A>) -> Self {\n        Self {\n            name,\n            task,\n            threads: Some(1),\n        }\n    }\n\n    /// Return the task behind [Arc] to be shared other logic.\n    pub fn task(&self) -> Arc<A> {\n        self.task.clone()\n    }\n}\n\n#[async_trait]\nimpl<A> ServiceWithDependents for GenBackgroundService<A>\nwhere\n    A: BackgroundService + Send + Sync + 'static,\n{\n    // Use default start_service implementation which signals ready immediately\n    // and then calls start_service\n\n    async fn start_service(\n        &mut self,\n        #[cfg(unix)] _fds: Option<ListenFds>,\n        shutdown: ShutdownWatch,\n        _listeners_per_fd: usize,\n        ready: ServiceReadyNotifier,\n    ) {\n        self.task.start_with_ready_notifier(shutdown, ready).await;\n    }\n\n    fn name(&self) -> &str {\n        &self.name\n    }\n\n    fn threads(&self) -> Option<usize> {\n        self.threads\n    }\n}\n\n/// Helper function to create a background service with a human readable name\npub fn background_service<SV>(name: &str, task: SV) -> GenBackgroundService<SV> {\n    GenBackgroundService::new(format!(\"BG {name}\"), Arc::new(task))\n}\n"
  },
  {
    "path": "pingora-core/src/services/listening.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! The listening service\n//!\n//! A [Service] (listening service) responds to incoming requests on its endpoints.\n//! Each [Service] can be configured with custom application logic (e.g. an `HTTPProxy`) and one or\n//! more endpoints to listen to.\n\nuse crate::apps::ServerApp;\nuse crate::listeners::tls::TlsSettings;\n#[cfg(feature = \"connection_filter\")]\nuse crate::listeners::AcceptAllFilter;\nuse crate::listeners::{\n    ConnectionFilter, Listeners, ServerAddress, TcpSocketOptions, TransportStack,\n};\nuse crate::protocols::Stream;\n#[cfg(unix)]\nuse crate::server::ListenFds;\nuse crate::server::ShutdownWatch;\nuse crate::services::Service as ServiceTrait;\n\nuse async_trait::async_trait;\nuse log::{debug, error, info};\nuse pingora_error::Result;\nuse pingora_runtime::current_handle;\nuse pingora_timeout::timeout;\nuse std::fs::Permissions;\nuse std::sync::Arc;\nuse std::time::Duration;\n\n/// The type of service that is associated with a list of listening endpoints and a particular application\npub struct Service<A> {\n    name: String,\n    listeners: Listeners,\n    app_logic: Option<A>,\n    /// The number of preferred threads. `None` to follow global setting.\n    pub threads: Option<usize>,\n    #[cfg(feature = \"connection_filter\")]\n    connection_filter: Arc<dyn ConnectionFilter>,\n}\n\nimpl<A> Service<A> {\n    /// Create a new [`Service`] with the given application (see [`crate::apps`]).\n    pub fn new(name: String, app_logic: A) -> Self {\n        Service {\n            name,\n            listeners: Listeners::new(),\n            app_logic: Some(app_logic),\n            threads: None,\n            #[cfg(feature = \"connection_filter\")]\n            connection_filter: Arc::new(AcceptAllFilter),\n        }\n    }\n\n    /// Create a new [`Service`] with the given application (see [`crate::apps`]) and the given\n    /// [`Listeners`].\n    pub fn with_listeners(name: String, listeners: Listeners, app_logic: A) -> Self {\n        Service {\n            name,\n            listeners,\n            app_logic: Some(app_logic),\n            threads: None,\n            #[cfg(feature = \"connection_filter\")]\n            connection_filter: Arc::new(AcceptAllFilter),\n        }\n    }\n\n    /// Set a custom connection filter for this service.\n    ///\n    /// The connection filter will be applied to all incoming connections\n    /// on all endpoints of this service. Connections that don't pass the\n    /// filter will be dropped immediately at the TCP level, before TLS\n    /// handshake or any HTTP processing.\n    ///\n    /// # Feature Flag\n    ///\n    /// This method requires the `connection_filter` feature to be enabled.\n    /// When the feature is disabled, this method is a no-op.\n    ///\n    /// # Example\n    ///\n    /// ```rust,no_run\n    /// # use std::sync::Arc;\n    /// # use pingora_core::listeners::{ConnectionFilter, AcceptAllFilter};\n    /// # struct MyService;\n    /// # impl MyService {\n    /// #   fn new() -> Self { MyService }\n    /// # }\n    /// let mut service = MyService::new();\n    /// let filter = Arc::new(AcceptAllFilter);\n    /// service.set_connection_filter(filter);\n    /// ```\n    #[cfg(feature = \"connection_filter\")]\n    pub fn set_connection_filter(&mut self, filter: Arc<dyn ConnectionFilter>) {\n        self.connection_filter = filter.clone();\n        self.listeners.set_connection_filter(filter);\n    }\n\n    #[cfg(not(feature = \"connection_filter\"))]\n    pub fn set_connection_filter(&mut self, _filter: Arc<dyn ConnectionFilter>) {}\n\n    /// Get the [`Listeners`], mostly to add more endpoints.\n    pub fn endpoints(&mut self) -> &mut Listeners {\n        &mut self.listeners\n    }\n\n    // the follow add* function has no effect if the server is already started\n\n    /// Add a TCP listening endpoint with the given address (e.g., `127.0.0.1:8000`).\n    pub fn add_tcp(&mut self, addr: &str) {\n        self.listeners.add_tcp(addr);\n    }\n\n    /// Add a TCP listening endpoint with the given [`TcpSocketOptions`].\n    pub fn add_tcp_with_settings(&mut self, addr: &str, sock_opt: TcpSocketOptions) {\n        self.listeners.add_tcp_with_settings(addr, sock_opt);\n    }\n\n    /// Add a Unix domain socket listening endpoint with the given path.\n    ///\n    /// Optionally take a permission of the socket file. The default is read and write access for\n    /// everyone (0o666).\n    #[cfg(unix)]\n    pub fn add_uds(&mut self, addr: &str, perm: Option<Permissions>) {\n        self.listeners.add_uds(addr, perm);\n    }\n\n    /// Add a TLS listening endpoint with the given certificate and key paths.\n    pub fn add_tls(&mut self, addr: &str, cert_path: &str, key_path: &str) -> Result<()> {\n        self.listeners.add_tls(addr, cert_path, key_path)\n    }\n\n    /// Add a TLS listening endpoint with the given [`TlsSettings`] and [`TcpSocketOptions`].\n    pub fn add_tls_with_settings(\n        &mut self,\n        addr: &str,\n        sock_opt: Option<TcpSocketOptions>,\n        settings: TlsSettings,\n    ) {\n        self.listeners\n            .add_tls_with_settings(addr, sock_opt, settings)\n    }\n\n    /// Add an endpoint according to the given [`ServerAddress`]\n    pub fn add_address(&mut self, addr: ServerAddress) {\n        self.listeners.add_address(addr);\n    }\n\n    /// Get a reference to the application inside this service\n    pub fn app_logic(&self) -> Option<&A> {\n        self.app_logic.as_ref()\n    }\n\n    /// Get a mutable reference to the application inside this service\n    pub fn app_logic_mut(&mut self) -> Option<&mut A> {\n        self.app_logic.as_mut()\n    }\n}\n\nimpl<A: ServerApp + Send + Sync + 'static> Service<A> {\n    pub async fn handle_event(event: Stream, app_logic: Arc<A>, shutdown: ShutdownWatch) {\n        debug!(\"new event!\");\n        let mut reuse_event = app_logic.process_new(event, &shutdown).await;\n        while let Some(event) = reuse_event {\n            // TODO: with no steal runtime, consider spawn() the next event on\n            // another thread for more evenly load balancing\n            debug!(\"new reusable event!\");\n            reuse_event = app_logic.process_new(event, &shutdown).await;\n        }\n    }\n\n    async fn run_endpoint(\n        app_logic: Arc<A>,\n        mut stack: TransportStack,\n        mut shutdown: ShutdownWatch,\n    ) {\n        // the accept loop, until the system is shutting down\n        loop {\n            let new_io = tokio::select! { // TODO: consider biased for perf reason?\n                new_io = stack.accept() => new_io,\n                shutdown_signal = shutdown.changed() => {\n                    match shutdown_signal {\n                        Ok(()) => {\n                            if !*shutdown.borrow() {\n                                // happen in the initial read\n                                continue;\n                            }\n                            info!(\"Shutting down {}\", stack.as_str());\n                            break;\n                        }\n                        Err(e) => {\n                            error!(\"shutdown_signal error {e}\");\n                            break;\n                        }\n                    }\n                }\n            };\n            match new_io {\n                Ok(io) => {\n                    let app = app_logic.clone();\n                    let shutdown = shutdown.clone();\n                    current_handle().spawn(async move {\n                        let peer_addr = io.peer_addr();\n                        match timeout(Duration::from_secs(60), io.handshake()).await {\n                            Ok(handshake) => {\n                                match handshake {\n                                    Ok(io) => Self::handle_event(io, app, shutdown).await,\n                                    Err(e) => {\n                                        // TODO: Maybe IOApp trait needs a fn to handle/filter out this error\n                                        if let Some(addr) = peer_addr {\n                                            error!(\"Downstream handshake error from {}: {e}\", addr);\n                                        } else {\n                                            error!(\"Downstream handshake error: {e}\");\n                                        }\n                                    }\n                                }\n                            }\n                            Err(_) => {\n                                error!(\"Downstream handshake timeout\");\n                            }\n                        }\n                    });\n                }\n                Err(e) => {\n                    error!(\"Accept() failed {e}\");\n                    if let Some(io_error) = e\n                        .root_cause()\n                        .downcast_ref::<std::io::Error>()\n                        .and_then(|e| e.raw_os_error())\n                    {\n                        // 24: too many open files. In this case accept() will continue return this\n                        // error without blocking, which could use up all the resources\n                        if io_error == 24 {\n                            // call sleep to calm the thread down and wait for others to release\n                            // some resources\n                            tokio::time::sleep(std::time::Duration::from_secs(1)).await;\n                        }\n                    }\n                }\n            }\n        }\n\n        stack.cleanup();\n    }\n}\n\n#[async_trait]\nimpl<A: ServerApp + Send + Sync + 'static> ServiceTrait for Service<A> {\n    async fn start_service(\n        &mut self,\n        #[cfg(unix)] fds: Option<ListenFds>,\n        shutdown: ShutdownWatch,\n        listeners_per_fd: usize,\n    ) {\n        let runtime = current_handle();\n        let endpoints = self\n            .listeners\n            .build(\n                #[cfg(unix)]\n                fds,\n            )\n            .await\n            .expect(\"Failed to build listeners\");\n\n        let app_logic = self\n            .app_logic\n            .take()\n            .expect(\"can only start_service() once\");\n        let app_logic = Arc::new(app_logic);\n\n        let mut handlers = Vec::new();\n\n        endpoints.into_iter().for_each(|endpoint| {\n            for _ in 0..listeners_per_fd {\n                let shutdown = shutdown.clone();\n                let my_app_logic = app_logic.clone();\n                let endpoint = endpoint.clone();\n\n                let jh = runtime.spawn(async move {\n                    Self::run_endpoint(my_app_logic, endpoint, shutdown).await;\n                });\n\n                handlers.push(jh);\n            }\n        });\n\n        futures::future::join_all(handlers).await;\n        self.listeners.cleanup();\n        app_logic.cleanup().await;\n    }\n\n    fn name(&self) -> &str {\n        &self.name\n    }\n\n    fn threads(&self) -> Option<usize> {\n        self.threads\n    }\n}\n\nuse crate::apps::prometheus_http_app::PrometheusServer;\n\nimpl Service<PrometheusServer> {\n    /// The Prometheus HTTP server\n    ///\n    /// The HTTP server endpoint that reports Prometheus metrics collected in the entire service\n    pub fn prometheus_http_service() -> Self {\n        Service::new(\n            \"Prometheus metric HTTP\".to_string(),\n            PrometheusServer::new(),\n        )\n    }\n}\n"
  },
  {
    "path": "pingora-core/src/services/mod.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! The service interface\n//!\n//! A service to the pingora server is just something runs forever until the server is shutting\n//! down.\n//!\n//! Two types of services are particularly useful\n//! - services that are listening to some (TCP) endpoints\n//! - services that are just running in the background.\n\nuse async_trait::async_trait;\nuse daggy::Walker;\nuse daggy::{petgraph::visit::Topo, Dag, NodeIndex};\nuse log::{error, info, warn};\nuse parking_lot::Mutex;\nuse std::borrow::Borrow;\nuse std::sync::Arc;\nuse std::sync::Weak;\nuse std::time::Duration;\nuse tokio::sync::watch;\n\n#[cfg(unix)]\nuse crate::server::ListenFds;\nuse crate::server::ShutdownWatch;\n\npub mod background;\npub mod listening;\n\n/// A notification channel for signaling when a service has become ready.\n///\n/// Services can use this to notify other services that may depend on them\n/// that they have successfully started and are ready to serve requests.\n///\n/// # Example\n///\n/// ```rust,ignore\n/// use pingora_core::services::ServiceReadyNotifier;\n///\n/// async fn my_service(ready_notifier: ServiceReadyNotifier) {\n///     // Perform initialization...\n///\n///     // Signal that the service is ready\n///     ready_notifier.notify_ready();\n///\n///     // Continue with main service loop...\n/// }\n/// ```\npub struct ServiceReadyNotifier {\n    sender: watch::Sender<bool>,\n}\n\nimpl Drop for ServiceReadyNotifier {\n    /// In the event that the notifier is dropped before notifying that the\n    /// service is ready, we opt to signal ready anyway\n    fn drop(&mut self) {\n        // Ignore errors - if there are no receivers, that's fine\n        let _ = self.sender.send(true);\n    }\n}\n\nimpl ServiceReadyNotifier {\n    /// Creates a new ServiceReadyNotifier from a watch sender.\n    /// You will not need to create one of these for normal usage, but being\n    /// able to is useful for testing.\n    pub fn new(sender: watch::Sender<bool>) -> Self {\n        Self { sender }\n    }\n\n    /// Notifies dependent services that this service is ready.\n    ///\n    /// Consumes the notifier to ensure ready is only signaled once.\n    pub fn notify_ready(self) {\n        // Dropping the notifier will signal that the service is ready\n        drop(self);\n    }\n}\n\n/// A receiver for watching when a service becomes ready.\npub type ServiceReadyWatch = watch::Receiver<bool>;\n\n/// A handle to a service in the server.\n///\n/// This is returned by [`crate::server::Server::add_service()`] and provides\n/// methods to declare that other services depend on this one.\n///\n/// # Example\n///\n/// ```rust,ignore\n/// let db_handle = server.add_service(database_service);\n/// let cache_handle = server.add_service(cache_service);\n///\n/// let api_handle = server.add_service(api_service);\n/// api_handle.add_dependency(&db_handle);\n/// api_handle.add_dependency(&cache_handle);\n/// ```\n#[derive(Debug, Clone)]\npub struct ServiceHandle {\n    pub(crate) id: NodeIndex,\n    name: String,\n    ready_watch: ServiceReadyWatch,\n    dependencies: Weak<Mutex<DependencyGraph>>,\n}\n\n/// Internal representation of a dependency relationship.\n#[derive(Debug, Clone)]\npub(crate) struct ServiceDependency {\n    pub name: String,\n    pub ready_watch: ServiceReadyWatch,\n}\n\nimpl ServiceHandle {\n    /// Creates a new ServiceHandle with the given ID, name, and readiness watcher.\n    pub(crate) fn new(\n        id: NodeIndex,\n        name: String,\n        ready_watch: ServiceReadyWatch,\n        dependencies: &Arc<Mutex<DependencyGraph>>,\n    ) -> Self {\n        Self {\n            id,\n            name,\n            ready_watch,\n            dependencies: Arc::downgrade(dependencies),\n        }\n    }\n\n    #[cfg(test)]\n    fn get_dependencies(&self) -> Vec<ServiceDependency> {\n        let Some(deps_lock) = self.dependencies.upgrade() else {\n            return Vec::new();\n        };\n\n        let deps = deps_lock.lock();\n        deps.get_dependencies(self.id)\n    }\n\n    /// Returns the name of the service.\n    pub fn name(&self) -> &str {\n        &self.name\n    }\n\n    /// Returns a clone of the readiness watcher for this service.\n    #[allow(dead_code)]\n    pub(crate) fn ready_watch(&self) -> ServiceReadyWatch {\n        self.ready_watch.clone()\n    }\n\n    /// Declares that this service depends on another service.\n    ///\n    /// This service will not start until the specified dependency has started\n    /// and signaled readiness.\n    ///\n    /// # Example\n    ///\n    /// ```rust,ignore\n    /// let db_id = server.add_service(database_service);\n    /// let api_id = server.add_service(api_service);\n    ///\n    /// // API service depends on database\n    /// api_id.add_dependency(&db_id);\n    /// ```\n    pub fn add_dependency(&self, dependency: impl Borrow<ServiceHandle>) {\n        let Some(deps_lock) = self.dependencies.upgrade() else {\n            warn!(\"Attempted to add a dependency after the dependency tree was dropped\");\n            return;\n        };\n\n        let mut deps = deps_lock.lock();\n        if let Err(e) = deps.add_dependency(self.id, dependency.borrow().id) {\n            error!(\"Error creating dependency edge: {e}\");\n        }\n    }\n\n    /// Declares that this service depends on the given other services.\n    ///\n    /// This service will not start until the specified dependencies have\n    /// started and signaled readiness.\n    ///\n    /// # Example\n    ///\n    /// ```rust,ignore\n    /// let db_id = server.add_service(database_service);\n    /// let cache_id = server.add_service(cache_service);\n    /// let api_id = server.add_service(api_service);\n    ///\n    /// // API service depends on database\n    /// api_id.add_dependencies(&[&db_id, &cache_id]);\n    /// ```\n    pub fn add_dependencies<'a, D>(&self, dependencies: impl IntoIterator<Item = D>)\n    where\n        D: Borrow<ServiceHandle> + 'a,\n    {\n        for dependency in dependencies {\n            self.add_dependency(dependency);\n        }\n    }\n}\n\n/// Helper for validating service dependency graphs using daggy.\npub(crate) struct DependencyGraph {\n    /// The directed acyclic graph structure from daggy.\n    dag: Dag<ServiceDependency, ()>,\n}\n\nimpl DependencyGraph {\n    /// Creates a new dependency graph.\n    pub(crate) fn new() -> Self {\n        Self { dag: Dag::new() }\n    }\n\n    /// Adds a service node to the graph.\n    ///\n    /// This should be called for all services first, before adding edges.\n    pub(crate) fn add_node(&mut self, name: String, ready_watch: ServiceReadyWatch) -> NodeIndex {\n        self.dag.add_node(ServiceDependency { name, ready_watch })\n    }\n    /// Adds a dependency edge from one service to another.\n    ///\n    /// Returns an error if adding this dependency would create a cycle or reference\n    /// a non-existent service.\n    pub(crate) fn add_dependency(\n        &mut self,\n        dependent_service_node_idx: NodeIndex,\n        dependency_service_node_idx: NodeIndex,\n    ) -> Result<(), String> {\n        // Try to add edge (from dependency to dependent)\n        // daggy will return an error if this would create a cycle\n        if let Err(cycle) =\n            self.dag\n                .add_edge(dependency_service_node_idx, dependent_service_node_idx, ())\n        {\n            return Err(format!(\n                \"Circular service dependency detected between {} and {} creating cycle: {cycle}\",\n                self.dag[dependency_service_node_idx].name,\n                self.dag[dependent_service_node_idx].name\n            ));\n        }\n\n        Ok(())\n    }\n\n    /// Returns services in topological order (dependencies before dependents).\n    ///\n    /// This ordering ensures that services are started in the correct order.\n    /// Returns service IDs in the correct startup order.\n    pub(crate) fn topological_sort(&self) -> Result<Vec<(NodeIndex, ServiceDependency)>, String> {\n        // Use daggy's built-in topological walker\n        let mut sorted = Vec::new();\n        let mut topo = Topo::new(&self.dag);\n\n        while let Some(service_id) = topo.next(&self.dag) {\n            sorted.push((service_id, self.dag[service_id].clone()));\n        }\n\n        Ok(sorted)\n    }\n\n    pub(crate) fn get_dependencies(&self, service_id: NodeIndex) -> Vec<ServiceDependency> {\n        self.dag\n            .parents(service_id)\n            .iter(&self.dag)\n            .map(|(_, n)| self.dag[n].clone())\n            .collect()\n    }\n}\n\nimpl Default for DependencyGraph {\n    fn default() -> Self {\n        Self::new()\n    }\n}\n\n#[async_trait]\npub trait ServiceWithDependents: Send + Sync {\n    /// This function will be called when the server is ready to start the service.\n    ///\n    /// Override this method if you need to control exactly when the service signals readiness\n    /// (e.g., after async initialization is complete).\n    ///\n    /// # Arguments\n    ///\n    /// - `fds` (Unix only): a collection of listening file descriptors. During zero downtime restart\n    ///   the `fds` would contain the listening sockets passed from the old service, services should\n    ///   take the sockets they need to use then. If the sockets the service looks for don't appear in\n    ///   the collection, the service should create its own listening sockets and then put them into\n    ///   the collection in order for them to be passed to the next server.\n    /// - `shutdown`: the shutdown signal this server would receive.\n    /// - `listeners_per_fd`: number of listener tasks to spawn per file descriptor.\n    /// - `ready_notifier`: notifier to signal when the service is ready. Services with\n    ///   dependents should call `ready_notifier.notify_ready()` once they are fully initialized.\n    async fn start_service(\n        &mut self,\n        #[cfg(unix)] fds: Option<ListenFds>,\n        shutdown: ShutdownWatch,\n        listeners_per_fd: usize,\n        ready_notifier: ServiceReadyNotifier,\n    );\n\n    /// The name of the service, just for logging and naming the threads assigned to this service\n    ///\n    /// Note that due to the limit of the underlying system, only the first 16 chars will be used\n    fn name(&self) -> &str;\n\n    /// The preferred number of threads to run this service\n    ///\n    /// If `None`, the global setting will be used\n    fn threads(&self) -> Option<usize> {\n        None\n    }\n\n    /// This is currently called to inform the service about the delay it\n    /// experienced from between waiting on its dependencies. Default behavior\n    /// is to log the time.\n    ///\n    /// TODO. It would be nice if this function was called intermittently by\n    /// the server while the service was waiting to give live updates while the\n    /// service was waiting and allow the service to decide whether to keep\n    /// waiting, continue anyway, or exit\n    fn on_startup_delay(&self, time_waited: Duration) {\n        info!(\n            \"Service {} spent {}ms waiting on dependencies\",\n            self.name(),\n            time_waited.as_millis()\n        );\n    }\n}\n\n#[async_trait]\nimpl<S> ServiceWithDependents for S\nwhere\n    S: Service,\n{\n    async fn start_service(\n        &mut self,\n        #[cfg(unix)] fds: Option<ListenFds>,\n        shutdown: ShutdownWatch,\n        listeners_per_fd: usize,\n        ready_notifier: ServiceReadyNotifier,\n    ) {\n        // Signal ready immediately\n        ready_notifier.notify_ready();\n\n        S::start_service(\n            self,\n            #[cfg(unix)]\n            fds,\n            shutdown,\n            listeners_per_fd,\n        )\n        .await\n    }\n\n    fn name(&self) -> &str {\n        S::name(self)\n    }\n\n    fn threads(&self) -> Option<usize> {\n        S::threads(self)\n    }\n\n    fn on_startup_delay(&self, time_waited: Duration) {\n        S::on_startup_delay(self, time_waited)\n    }\n}\n\n/// The service interface\n#[async_trait]\npub trait Service: Sync + Send {\n    /// Start the service without readiness notification.\n    ///\n    /// This is a simpler version of [`Self::start_service()`] for services that don't need\n    /// to control when they signal readiness. The default implementation does nothing.\n    ///\n    /// Most services should override this method instead of [`Self::start_service()`].\n    ///\n    /// # Arguments\n    ///\n    /// - `fds` (Unix only): a collection of listening file descriptors.\n    /// - `shutdown`: the shutdown signal this server would receive.\n    /// - `listeners_per_fd`: number of listener tasks to spawn per file descriptor.\n    async fn start_service(\n        &mut self,\n        #[cfg(unix)] _fds: Option<ListenFds>,\n        _shutdown: ShutdownWatch,\n        _listeners_per_fd: usize,\n    ) {\n        // Default: do nothing\n    }\n\n    /// The name of the service, just for logging and naming the threads assigned to this service\n    ///\n    /// Note that due to the limit of the underlying system, only the first 16 chars will be used\n    fn name(&self) -> &str;\n\n    /// The preferred number of threads to run this service\n    ///\n    /// If `None`, the global setting will be used\n    fn threads(&self) -> Option<usize> {\n        None\n    }\n\n    /// This is currently called to inform the service about the delay it\n    /// experienced from between waiting on its dependencies. Default behavior\n    /// is to log the time.\n    ///\n    /// TODO. It would be nice if this function was called intermittently by\n    /// the server while the service was waiting to give live updates while the\n    /// service was waiting and allow the service to decide whether to keep\n    /// waiting, continue anyway, or exit\n    fn on_startup_delay(&self, time_waited: Duration) {\n        info!(\n            \"Service {} spent {}ms waiting on dependencies\",\n            self.name(),\n            time_waited.as_millis()\n        );\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use super::*;\n\n    #[test]\n    fn test_service_handle_creation() {\n        let deps: Arc<Mutex<DependencyGraph>> = Arc::new(Mutex::new(DependencyGraph::new()));\n        let (tx, rx) = watch::channel(false);\n        let service_id = ServiceHandle::new(0.into(), \"test_service\".to_string(), rx, &deps);\n\n        assert_eq!(service_id.id, 0.into());\n        assert_eq!(service_id.name(), \"test_service\");\n\n        // Should be able to clone the watch\n        let watch_clone = service_id.ready_watch();\n        assert!(!*watch_clone.borrow());\n\n        // Signaling ready should be observable through cloned watch\n        tx.send(true).ok();\n        assert!(*watch_clone.borrow());\n    }\n\n    #[test]\n    fn test_service_handle_add_dependency() {\n        let graph: Arc<Mutex<DependencyGraph>> = Arc::new(Mutex::new(DependencyGraph::new()));\n        let (tx1, rx1) = watch::channel(false);\n        let (tx1_clone, rx1_clone) = (tx1.clone(), rx1.clone());\n        let (_tx2, rx2) = watch::channel(false);\n        let (_tx2_clone, rx2_clone) = (_tx2.clone(), rx2.clone());\n\n        // Add nodes to the graph first\n        let dep_node = {\n            let mut g = graph.lock();\n            g.add_node(\"dependency\".to_string(), rx1)\n        };\n        let main_node = {\n            let mut g = graph.lock();\n            g.add_node(\"main\".to_string(), rx2)\n        };\n\n        let dep_service = ServiceHandle::new(dep_node, \"dependency\".to_string(), rx1_clone, &graph);\n        let main_service = ServiceHandle::new(main_node, \"main\".to_string(), rx2_clone, &graph);\n\n        // Add dependency\n        main_service.add_dependency(&dep_service);\n\n        // Get dependencies and verify\n        let deps = main_service.get_dependencies();\n        assert_eq!(deps.len(), 1);\n        assert_eq!(deps[0].name, \"dependency\");\n\n        // Verify watch is working\n        assert!(!*deps[0].ready_watch.borrow());\n        tx1_clone.send(true).ok();\n        assert!(*deps[0].ready_watch.borrow());\n    }\n\n    #[test]\n    fn test_service_handle_multiple_dependencies() {\n        let graph: Arc<Mutex<DependencyGraph>> = Arc::new(Mutex::new(DependencyGraph::new()));\n        let (_tx1, rx1) = watch::channel(false);\n        let rx1_clone = rx1.clone();\n        let (_tx2, rx2) = watch::channel(false);\n        let rx2_clone = rx2.clone();\n        let (_tx3, rx3) = watch::channel(false);\n        let rx3_clone = rx3.clone();\n\n        // Add nodes to the graph first\n        let dep1_node = {\n            let mut g = graph.lock();\n            g.add_node(\"dep1\".to_string(), rx1)\n        };\n        let dep2_node = {\n            let mut g = graph.lock();\n            g.add_node(\"dep2\".to_string(), rx2)\n        };\n        let main_node = {\n            let mut g = graph.lock();\n            g.add_node(\"main\".to_string(), rx3)\n        };\n\n        let dep1 = ServiceHandle::new(dep1_node, \"dep1\".to_string(), rx1_clone, &graph);\n        let dep2 = ServiceHandle::new(dep2_node, \"dep2\".to_string(), rx2_clone, &graph);\n        let main_service = ServiceHandle::new(main_node, \"main\".to_string(), rx3_clone, &graph);\n\n        // Add multiple dependencies\n        main_service.add_dependency(&dep1);\n        main_service.add_dependency(&dep2);\n\n        // Get dependencies and verify\n        let deps = main_service.get_dependencies();\n        assert_eq!(deps.len(), 2);\n\n        let dep_names: Vec<&str> = deps.iter().map(|d| d.name.as_str()).collect();\n        assert!(dep_names.contains(&\"dep1\"));\n        assert!(dep_names.contains(&\"dep2\"));\n    }\n\n    #[test]\n    fn test_single_service_no_dependencies() {\n        let mut graph = DependencyGraph::new();\n        let (_tx, rx) = watch::channel(false);\n        let _node = graph.add_node(\"service1\".to_string(), rx);\n\n        let order = graph.topological_sort().unwrap();\n        assert_eq!(order.len(), 1);\n        assert_eq!(order[0].1.name, \"service1\");\n    }\n\n    #[test]\n    fn test_simple_dependency_chain() {\n        let mut graph = DependencyGraph::new();\n        let (_tx1, rx1) = watch::channel(false);\n        let (_tx2, rx2) = watch::channel(false);\n        let (_tx3, rx3) = watch::channel(false);\n\n        let node1 = graph.add_node(\"service1\".to_string(), rx1);\n        let node2 = graph.add_node(\"service2\".to_string(), rx2);\n        let node3 = graph.add_node(\"service3\".to_string(), rx3);\n\n        // service2 depends on service1, service3 depends on service2\n        graph.add_dependency(node2, node1).unwrap();\n        graph.add_dependency(node3, node2).unwrap();\n\n        let order = graph.topological_sort().unwrap();\n        assert_eq!(order.len(), 3);\n        // Verify order: service1, service2, service3\n        assert_eq!(order[0].1.name, \"service1\");\n        assert_eq!(order[1].1.name, \"service2\");\n        assert_eq!(order[2].1.name, \"service3\");\n    }\n\n    #[test]\n    fn test_diamond_dependency() {\n        let mut graph = DependencyGraph::new();\n        let (_tx1, rx1) = watch::channel(false);\n        let (_tx2, rx2) = watch::channel(false);\n        let (_tx3, rx3) = watch::channel(false);\n\n        let db = graph.add_node(\"db\".to_string(), rx1);\n        let cache = graph.add_node(\"cache\".to_string(), rx2);\n        let api = graph.add_node(\"api\".to_string(), rx3);\n\n        // api depends on both db and cache\n        graph.add_dependency(api, db).unwrap();\n        graph.add_dependency(api, cache).unwrap();\n\n        let order = graph.topological_sort().unwrap();\n        // api should come last, but db and cache order doesn't matter\n        assert_eq!(order.len(), 3);\n        assert_eq!(order[2].1.name, \"api\");\n        let first_two: Vec<&str> = order[0..2].iter().map(|(_, d)| d.name.as_str()).collect();\n        assert!(first_two.contains(&\"db\"));\n        assert!(first_two.contains(&\"cache\"));\n    }\n\n    #[test]\n    #[should_panic(expected = \"node indices out of bounds\")]\n    fn test_missing_dependency() {\n        let mut graph = DependencyGraph::new();\n        let (_tx1, rx1) = watch::channel(false);\n\n        let node1 = graph.add_node(\"service1\".to_string(), rx1);\n        let nonexistent = NodeIndex::new(999);\n\n        // Try to add dependency on non-existent node - this should panic\n        let _ = graph.add_dependency(node1, nonexistent);\n    }\n\n    #[test]\n    fn test_circular_dependency_self() {\n        let mut graph = DependencyGraph::new();\n        let (_tx1, rx1) = watch::channel(false);\n\n        let node1 = graph.add_node(\"service1\".to_string(), rx1);\n\n        // Try to make service depend on itself\n        let result = graph.add_dependency(node1, node1);\n\n        assert!(result.is_err());\n        assert!(result.unwrap_err().contains(\"Circular\"));\n    }\n\n    #[test]\n    fn test_circular_dependency_two_services() {\n        let mut graph = DependencyGraph::new();\n        let (_tx1, rx1) = watch::channel(false);\n        let (_tx2, rx2) = watch::channel(false);\n\n        // Add both nodes first\n        let node1 = graph.add_node(\"service1\".to_string(), rx1);\n        let node2 = graph.add_node(\"service2\".to_string(), rx2);\n\n        // Try to add circular dependencies\n        graph.add_dependency(node1, node2).unwrap();\n        let result = graph.add_dependency(node2, node1);\n\n        assert!(result.is_err());\n        assert!(result.unwrap_err().contains(\"Circular\"));\n    }\n\n    #[test]\n    fn test_circular_dependency_three_services() {\n        let mut graph = DependencyGraph::new();\n        let (_tx1, rx1) = watch::channel(false);\n        let (_tx2, rx2) = watch::channel(false);\n        let (_tx3, rx3) = watch::channel(false);\n\n        // Add all nodes first\n        let node1 = graph.add_node(\"service1\".to_string(), rx1);\n        let node2 = graph.add_node(\"service2\".to_string(), rx2);\n        let node3 = graph.add_node(\"service3\".to_string(), rx3);\n\n        // Add dependencies that would form a cycle\n        graph.add_dependency(node1, node2).unwrap();\n        graph.add_dependency(node2, node3).unwrap();\n        let result = graph.add_dependency(node3, node1);\n\n        assert!(result.is_err());\n        assert!(result.unwrap_err().contains(\"Circular\"));\n    }\n\n    #[test]\n    fn test_complex_valid_graph() {\n        let mut graph = DependencyGraph::new();\n        let (_tx1, rx1) = watch::channel(false);\n        let (_tx2, rx2) = watch::channel(false);\n        let (_tx3, rx3) = watch::channel(false);\n        let (_tx4, rx4) = watch::channel(false);\n        let (_tx5, rx5) = watch::channel(false);\n\n        // Build a complex dependency graph:\n        //   db, cache - no deps\n        //   auth -> db\n        //   api -> db, cache, auth\n        //   frontend -> api\n        let db = graph.add_node(\"db\".to_string(), rx1);\n        let cache = graph.add_node(\"cache\".to_string(), rx2);\n        let auth = graph.add_node(\"auth\".to_string(), rx3);\n        let api = graph.add_node(\"api\".to_string(), rx4);\n        let frontend = graph.add_node(\"frontend\".to_string(), rx5);\n\n        graph.add_dependency(auth, db).unwrap();\n        graph.add_dependency(api, db).unwrap();\n        graph.add_dependency(api, cache).unwrap();\n        graph.add_dependency(api, auth).unwrap();\n        graph.add_dependency(frontend, api).unwrap();\n\n        let order = graph.topological_sort().unwrap();\n\n        // Verify ordering constraints using names\n        let db_pos = order.iter().position(|(_, d)| d.name == \"db\").unwrap();\n        let cache_pos = order.iter().position(|(_, d)| d.name == \"cache\").unwrap();\n        let auth_pos = order.iter().position(|(_, d)| d.name == \"auth\").unwrap();\n        let api_pos = order.iter().position(|(_, d)| d.name == \"api\").unwrap();\n        let frontend_pos = order\n            .iter()\n            .position(|(_, d)| d.name == \"frontend\")\n            .unwrap();\n\n        assert!(db_pos < auth_pos);\n        assert!(auth_pos < api_pos);\n        assert!(db_pos < api_pos);\n        assert!(cache_pos < api_pos);\n        assert!(api_pos < frontend_pos);\n    }\n}\n"
  },
  {
    "path": "pingora-core/src/tls/mod.rs",
    "content": "// Copyright 2024 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! This module contains a dummy TLS implementation for the scenarios where real TLS\n//! implementations are unavailable.\n\nmacro_rules! impl_display {\n    ($ty:ty) => {\n        impl std::fmt::Display for $ty {\n            fn fmt(&self, _f: &mut std::fmt::Formatter<'_>) -> Result<(), std::fmt::Error> {\n                Ok(())\n            }\n        }\n    };\n}\n\nmacro_rules! impl_deref {\n    ($from:ty => $to:ty) => {\n        impl std::ops::Deref for $from {\n            type Target = $to;\n            fn deref(&self) -> &$to {\n                panic!(\"Not implemented\");\n            }\n        }\n        impl std::ops::DerefMut for $from {\n            fn deref_mut(&mut self) -> &mut $to {\n                panic!(\"Not implemented\");\n            }\n        }\n    };\n}\n\npub mod ssl {\n    use super::error::ErrorStack;\n    use super::x509::verify::X509VerifyParamRef;\n    use super::x509::{X509VerifyResult, X509};\n\n    /// An error returned from an ALPN selection callback.\n    pub struct AlpnError;\n    impl AlpnError {\n        /// Terminate the handshake with a fatal alert.\n        pub const ALERT_FATAL: AlpnError = Self {};\n\n        /// Do not select a protocol, but continue the handshake.\n        pub const NOACK: AlpnError = Self {};\n    }\n\n    /// A type which allows for configuration of a client-side TLS session before connection.\n    pub struct ConnectConfiguration;\n    impl_deref! {ConnectConfiguration => SslRef}\n    impl ConnectConfiguration {\n        /// Configures the use of Server Name Indication (SNI) when connecting.\n        pub fn set_use_server_name_indication(&mut self, _use_sni: bool) {\n            panic!(\"Not implemented\");\n        }\n\n        /// Configures the use of hostname verification when connecting.\n        pub fn set_verify_hostname(&mut self, _verify_hostname: bool) {\n            panic!(\"Not implemented\");\n        }\n\n        /// Returns an `Ssl` configured to connect to the provided domain.\n        pub fn into_ssl(self, _domain: &str) -> Result<Ssl, ErrorStack> {\n            panic!(\"Not implemented\");\n        }\n\n        /// Like `SslContextBuilder::set_verify`.\n        pub fn set_verify(&mut self, _mode: SslVerifyMode) {\n            panic!(\"Not implemented\");\n        }\n\n        /// Like `SslContextBuilder::set_alpn_protos`.\n        pub fn set_alpn_protos(&mut self, _protocols: &[u8]) -> Result<(), ErrorStack> {\n            panic!(\"Not implemented\");\n        }\n\n        /// Returns a mutable reference to the X509 verification configuration.\n        pub fn param_mut(&mut self) -> &mut X509VerifyParamRef {\n            panic!(\"Not implemented\");\n        }\n    }\n\n    /// An SSL error.\n    #[derive(Debug)]\n    pub struct Error;\n    impl_display!(Error);\n    impl Error {\n        pub fn code(&self) -> ErrorCode {\n            panic!(\"Not implemented\");\n        }\n    }\n\n    /// An error code returned from SSL functions.\n    #[derive(PartialEq)]\n    pub struct ErrorCode(i32);\n    impl ErrorCode {\n        /// An error occurred in the SSL library.\n        pub const SSL: ErrorCode = Self(0);\n    }\n\n    /// An identifier of a session name type.\n    pub struct NameType;\n    impl NameType {\n        pub const HOST_NAME: NameType = Self {};\n    }\n\n    /// The state of an SSL/TLS session.\n    pub struct Ssl;\n    impl Ssl {\n        /// Creates a new `Ssl`.\n        pub fn new(_ctx: &SslContextRef) -> Result<Ssl, ErrorStack> {\n            panic!(\"Not implemented\");\n        }\n    }\n    impl_deref! {Ssl => SslRef}\n\n    /// A type which wraps server-side streams in a TLS session.\n    pub struct SslAcceptor;\n    impl SslAcceptor {\n        /// Creates a new builder configured to connect to non-legacy clients. This should\n        /// generally be considered a reasonable default choice.\n        pub fn mozilla_intermediate_v5(\n            _method: SslMethod,\n        ) -> Result<SslAcceptorBuilder, ErrorStack> {\n            panic!(\"Not implemented\");\n        }\n    }\n\n    /// A builder for `SslAcceptor`s.\n    pub struct SslAcceptorBuilder;\n    impl SslAcceptorBuilder {\n        /// Consumes the builder, returning a `SslAcceptor`.\n        pub fn build(self) -> SslAcceptor {\n            panic!(\"Not implemented\");\n        }\n\n        /// Sets the callback used by a server to select a protocol for Application Layer Protocol\n        /// Negotiation (ALPN).\n        pub fn set_alpn_select_callback<F>(&mut self, _callback: F)\n        where\n            F: for<'a> Fn(&mut SslRef, &'a [u8]) -> Result<&'a [u8], AlpnError>\n                + 'static\n                + Sync\n                + Send,\n        {\n            panic!(\"Not implemented\");\n        }\n\n        /// Loads a certificate chain from a file.\n        pub fn set_certificate_chain_file<P: AsRef<std::path::Path>>(\n            &mut self,\n            _file: P,\n        ) -> Result<(), ErrorStack> {\n            panic!(\"Not implemented\");\n        }\n\n        /// Loads the private key from a file.\n        pub fn set_private_key_file<P: AsRef<std::path::Path>>(\n            &mut self,\n            _file: P,\n            _file_type: SslFiletype,\n        ) -> Result<(), ErrorStack> {\n            panic!(\"Not implemented\");\n        }\n\n        /// Sets the maximum supported protocol version.\n        pub fn set_max_proto_version(\n            &mut self,\n            _version: Option<SslVersion>,\n        ) -> Result<(), ErrorStack> {\n            panic!(\"Not implemented\");\n        }\n    }\n\n    /// Reference to an [`SslCipher`].\n    pub struct SslCipherRef;\n    impl SslCipherRef {\n        /// Returns the name of the cipher.\n        pub fn name(&self) -> &'static str {\n            panic!(\"Not implemented\");\n        }\n    }\n\n    /// A type which wraps client-side streams in a TLS session.\n    pub struct SslConnector;\n    impl SslConnector {\n        /// Creates a new builder for TLS connections.\n        pub fn builder(_method: SslMethod) -> Result<SslConnectorBuilder, ErrorStack> {\n            panic!(\"Not implemented\");\n        }\n\n        /// Returns a structure allowing for configuration of a single TLS session before connection.\n        pub fn configure(&self) -> Result<ConnectConfiguration, ErrorStack> {\n            panic!(\"Not implemented\");\n        }\n\n        /// Returns a shared reference to the inner raw `SslContext`.\n        pub fn context(&self) -> &SslContextRef {\n            panic!(\"Not implemented\");\n        }\n    }\n\n    /// A builder for `SslConnector`s.\n    pub struct SslConnectorBuilder;\n    impl SslConnectorBuilder {\n        /// Consumes the builder, returning an `SslConnector`.\n        pub fn build(self) -> SslConnector {\n            panic!(\"Not implemented\");\n        }\n\n        /// Sets the list of supported ciphers for protocols before TLSv1.3.\n        pub fn set_cipher_list(&mut self, _cipher_list: &str) -> Result<(), ErrorStack> {\n            panic!(\"Not implemented\");\n        }\n\n        /// Sets the context’s supported signature algorithms.\n        pub fn set_sigalgs_list(&mut self, _sigalgs: &str) -> Result<(), ErrorStack> {\n            panic!(\"Not implemented\");\n        }\n\n        /// Sets the minimum supported protocol version.\n        pub fn set_min_proto_version(\n            &mut self,\n            _version: Option<SslVersion>,\n        ) -> Result<(), ErrorStack> {\n            panic!(\"Not implemented\");\n        }\n\n        /// Sets the maximum supported protocol version.\n        pub fn set_max_proto_version(\n            &mut self,\n            _version: Option<SslVersion>,\n        ) -> Result<(), ErrorStack> {\n            panic!(\"Not implemented\");\n        }\n\n        /// Use the default locations of trusted certificates for verification.\n        pub fn set_default_verify_paths(&mut self) -> Result<(), ErrorStack> {\n            panic!(\"Not implemented\");\n        }\n\n        /// Loads trusted root certificates from a file.\n        pub fn set_ca_file<P: AsRef<std::path::Path>>(\n            &mut self,\n            _file: P,\n        ) -> Result<(), ErrorStack> {\n            panic!(\"Not implemented\");\n        }\n\n        /// Loads a leaf certificate from a file.\n        pub fn set_certificate_file<P: AsRef<std::path::Path>>(\n            &mut self,\n            _file: P,\n            _file_type: SslFiletype,\n        ) -> Result<(), ErrorStack> {\n            panic!(\"Not implemented\");\n        }\n\n        /// Loads the private key from a file.\n        pub fn set_private_key_file<P: AsRef<std::path::Path>>(\n            &mut self,\n            _file: P,\n            _file_type: SslFiletype,\n        ) -> Result<(), ErrorStack> {\n            panic!(\"Not implemented\");\n        }\n\n        /// Sets the TLS key logging callback.\n        pub fn set_keylog_callback<F>(&mut self, _callback: F)\n        where\n            F: Fn(&SslRef, &str) + 'static + Sync + Send,\n        {\n            panic!(\"Not implemented\");\n        }\n    }\n\n    /// A context object for TLS streams.\n    pub struct SslContext;\n    impl SslContext {\n        /// Creates a new builder object for an `SslContext`.\n        pub fn builder(_method: SslMethod) -> Result<SslContextBuilder, ErrorStack> {\n            panic!(\"Not implemented\");\n        }\n    }\n    impl_deref! {SslContext => SslContextRef}\n\n    /// A builder for `SslContext`s.\n    pub struct SslContextBuilder;\n    impl SslContextBuilder {\n        /// Consumes the builder, returning a new `SslContext`.\n        pub fn build(self) -> SslContext {\n            panic!(\"Not implemented\");\n        }\n    }\n\n    /// Reference to [`SslContext`]\n    pub struct SslContextRef;\n\n    /// An identifier of the format of a certificate or key file.\n    pub struct SslFiletype;\n    impl SslFiletype {\n        /// The PEM format.\n        pub const PEM: SslFiletype = Self {};\n    }\n\n    /// A type specifying the kind of protocol an `SslContext`` will speak.\n    pub struct SslMethod;\n    impl SslMethod {\n        /// Support all versions of the TLS protocol.\n        pub fn tls() -> SslMethod {\n            panic!(\"Not implemented\");\n        }\n    }\n\n    /// Reference to an [`Ssl`].\n    pub struct SslRef;\n    impl SslRef {\n        /// Like [`SslContextBuilder::set_verify`].\n        pub fn set_verify(&mut self, _mode: SslVerifyMode) {\n            panic!(\"Not implemented\");\n        }\n\n        /// Returns the current cipher if the session is active.\n        pub fn current_cipher(&self) -> Option<&SslCipherRef> {\n            panic!(\"Not implemented\");\n        }\n\n        /// Sets the host name to be sent to the server for Server Name Indication (SNI).\n        pub fn set_hostname(&mut self, _hostname: &str) -> Result<(), ErrorStack> {\n            panic!(\"Not implemented\");\n        }\n\n        /// Returns the peer’s certificate, if present.\n        pub fn peer_certificate(&self) -> Option<X509> {\n            panic!(\"Not implemented\");\n        }\n\n        /// Returns the certificate verification result.\n        pub fn verify_result(&self) -> X509VerifyResult {\n            panic!(\"Not implemented\");\n        }\n\n        /// Returns a string describing the protocol version of the session.\n        pub fn version_str(&self) -> &'static str {\n            panic!(\"Not implemented\");\n        }\n\n        /// Returns the protocol selected via Application Layer Protocol Negotiation (ALPN).\n        pub fn selected_alpn_protocol(&self) -> Option<&[u8]> {\n            panic!(\"Not implemented\");\n        }\n\n        /// Returns the servername sent by the client via Server Name Indication (SNI).\n        pub fn servername(&self, _type_: NameType) -> Option<&str> {\n            panic!(\"Not implemented\");\n        }\n    }\n\n    /// Options controlling the behavior of certificate verification.\n    pub struct SslVerifyMode;\n    impl SslVerifyMode {\n        /// Verifies that the peer’s certificate is trusted.\n        pub const PEER: Self = Self {};\n\n        /// Disables verification of the peer’s certificate.\n        pub const NONE: Self = Self {};\n    }\n\n    /// An SSL/TLS protocol version.\n    pub struct SslVersion;\n    impl SslVersion {\n        /// TLSv1.0\n        pub const TLS1: SslVersion = Self {};\n\n        /// TLSv1.2\n        pub const TLS1_2: SslVersion = Self {};\n\n        /// TLSv1.3\n        pub const TLS1_3: SslVersion = Self {};\n    }\n\n    /// A standard implementation of protocol selection for Application Layer Protocol Negotiation\n    /// (ALPN).\n    pub fn select_next_proto<'a>(_server: &[u8], _client: &'a [u8]) -> Option<&'a [u8]> {\n        panic!(\"Not implemented\");\n    }\n}\n\npub mod ssl_sys {\n    pub const X509_V_OK: i32 = 0;\n    pub const X509_V_ERR_INVALID_CALL: i32 = 69;\n}\n\npub mod error {\n    use super::ssl::Error;\n\n    /// Collection of [`Errors`] from OpenSSL.\n    #[derive(Debug)]\n    pub struct ErrorStack;\n    impl_display!(ErrorStack);\n    impl std::error::Error for ErrorStack {}\n    impl ErrorStack {\n        /// Returns the contents of the OpenSSL error stack.\n        pub fn get() -> ErrorStack {\n            panic!(\"Not implemented\");\n        }\n\n        /// Returns the errors in the stack.\n        pub fn errors(&self) -> &[Error] {\n            panic!(\"Not implemented\");\n        }\n    }\n}\n\npub mod x509 {\n    use super::asn1::{Asn1IntegerRef, Asn1StringRef, Asn1TimeRef};\n    use super::error::ErrorStack;\n    use super::hash::{DigestBytes, MessageDigest};\n    use super::nid::Nid;\n\n    /// An `X509` public key certificate.\n    #[derive(Debug, Clone)]\n    pub struct X509;\n    impl_deref! {X509 => X509Ref}\n    impl X509 {\n        /// Deserializes a PEM-encoded X509 structure.\n        pub fn from_pem(_pem: &[u8]) -> Result<X509, ErrorStack> {\n            panic!(\"Not implemented\");\n        }\n    }\n\n    /// A type to destructure and examine an `X509Name`.\n    pub struct X509NameEntries<'a> {\n        marker: std::marker::PhantomData<&'a ()>,\n    }\n    impl<'a> Iterator for X509NameEntries<'a> {\n        type Item = &'a X509NameEntryRef;\n        fn next(&mut self) -> Option<&'a X509NameEntryRef> {\n            panic!(\"Not implemented\");\n        }\n    }\n\n    /// Reference to `X509NameEntry`.\n    pub struct X509NameEntryRef;\n    impl X509NameEntryRef {\n        pub fn data(&self) -> &Asn1StringRef {\n            panic!(\"Not implemented\");\n        }\n    }\n\n    /// Reference to `X509Name`.\n    pub struct X509NameRef;\n    impl X509NameRef {\n        /// Returns the name entries by the nid.\n        pub fn entries_by_nid(&self, _nid: Nid) -> X509NameEntries<'_> {\n            panic!(\"Not implemented\");\n        }\n    }\n\n    /// Reference to `X509`.\n    pub struct X509Ref;\n    impl X509Ref {\n        /// Returns this certificate’s subject name.\n        pub fn subject_name(&self) -> &X509NameRef {\n            panic!(\"Not implemented\");\n        }\n\n        /// Returns a digest of the DER representation of the certificate.\n        pub fn digest(&self, _hash_type: MessageDigest) -> Result<DigestBytes, ErrorStack> {\n            panic!(\"Not implemented\");\n        }\n\n        /// Returns the certificate’s Not After validity period.\n        pub fn not_after(&self) -> &Asn1TimeRef {\n            panic!(\"Not implemented\");\n        }\n\n        /// Returns this certificate’s serial number.\n        pub fn serial_number(&self) -> &Asn1IntegerRef {\n            panic!(\"Not implemented\");\n        }\n    }\n\n    /// The result of peer certificate verification.\n    pub struct X509VerifyResult;\n    impl X509VerifyResult {\n        /// Return the integer representation of an `X509VerifyResult`.\n        pub fn as_raw(&self) -> i32 {\n            panic!(\"Not implemented\");\n        }\n    }\n\n    pub mod store {\n        use super::super::error::ErrorStack;\n        use super::X509;\n\n        /// A builder type used to construct an `X509Store`.\n        pub struct X509StoreBuilder;\n        impl X509StoreBuilder {\n            /// Returns a builder for a certificate store..\n            pub fn new() -> Result<X509StoreBuilder, ErrorStack> {\n                panic!(\"Not implemented\");\n            }\n\n            /// Constructs the `X509Store`.\n            pub fn build(self) -> X509Store {\n                panic!(\"Not implemented\");\n            }\n\n            /// Adds a certificate to the certificate store.\n            pub fn add_cert(&mut self, _cert: X509) -> Result<(), ErrorStack> {\n                panic!(\"Not implemented\");\n            }\n        }\n\n        /// A certificate store to hold trusted X509 certificates.\n        pub struct X509Store;\n        impl_deref! {X509Store => X509StoreRef}\n\n        /// Reference to an `X509Store`.\n        pub struct X509StoreRef;\n    }\n\n    pub mod verify {\n        /// Reference to `X509VerifyParam`.\n        pub struct X509VerifyParamRef;\n    }\n}\n\npub mod nid {\n    /// A numerical identifier for an OpenSSL object.\n    pub struct Nid;\n    impl Nid {\n        pub const COMMONNAME: Nid = Self {};\n        pub const ORGANIZATIONNAME: Nid = Self {};\n        pub const ORGANIZATIONALUNITNAME: Nid = Self {};\n    }\n}\n\npub mod pkey {\n    use super::error::ErrorStack;\n\n    /// A public or private key.\n    #[derive(Clone)]\n    pub struct PKey<T> {\n        marker: std::marker::PhantomData<T>,\n    }\n    impl<T> std::ops::Deref for PKey<T> {\n        type Target = PKeyRef<T>;\n        fn deref(&self) -> &PKeyRef<T> {\n            panic!(\"Not implemented\");\n        }\n    }\n    impl<T> std::ops::DerefMut for PKey<T> {\n        fn deref_mut(&mut self) -> &mut PKeyRef<T> {\n            panic!(\"Not implemented\");\n        }\n    }\n    impl PKey<Private> {\n        pub fn private_key_from_pem(_pem: &[u8]) -> Result<PKey<Private>, ErrorStack> {\n            panic!(\"Not implemented\");\n        }\n    }\n\n    /// Reference to `PKey`.\n    pub struct PKeyRef<T> {\n        marker: std::marker::PhantomData<T>,\n    }\n\n    /// A tag type indicating that a key has private components.\n    #[derive(Clone)]\n    pub enum Private {}\n    unsafe impl HasPrivate for Private {}\n\n    /// A trait indicating that a key has private components.\n    pub unsafe trait HasPrivate {}\n}\n\npub mod hash {\n    /// A message digest algorithm.\n    pub struct MessageDigest;\n    impl MessageDigest {\n        pub fn sha256() -> MessageDigest {\n            panic!(\"Not implemented\");\n        }\n    }\n\n    /// The resulting bytes of a digest.\n    pub struct DigestBytes;\n    impl AsRef<[u8]> for DigestBytes {\n        fn as_ref(&self) -> &[u8] {\n            panic!(\"Not implemented\");\n        }\n    }\n}\n\npub mod asn1 {\n    use super::bn::BigNum;\n    use super::error::ErrorStack;\n\n    /// A reference to an `Asn1Integer`.\n    pub struct Asn1IntegerRef;\n    impl Asn1IntegerRef {\n        /// Converts the integer to a `BigNum`.\n        pub fn to_bn(&self) -> Result<BigNum, ErrorStack> {\n            panic!(\"Not implemented\");\n        }\n    }\n\n    /// A reference to an `Asn1String`.\n    pub struct Asn1StringRef;\n    impl Asn1StringRef {\n        pub fn as_utf8(&self) -> Result<&str, ErrorStack> {\n            panic!(\"Not implemented\");\n        }\n    }\n\n    /// Reference to an `Asn1Time`\n    pub struct Asn1TimeRef;\n    impl_display! {Asn1TimeRef}\n}\n\npub mod bn {\n    use super::error::ErrorStack;\n\n    /// Dynamically sized large number implementation\n    pub struct BigNum;\n    impl BigNum {\n        /// Returns a hexadecimal string representation of `self`.\n        pub fn to_hex_str(&self) -> Result<&str, ErrorStack> {\n            panic!(\"Not implemented\");\n        }\n    }\n}\n\npub mod ext {\n    use super::error::ErrorStack;\n    use super::pkey::{HasPrivate, PKeyRef};\n    use super::ssl::{Ssl, SslAcceptor, SslRef};\n    use super::x509::store::X509StoreRef;\n    use super::x509::verify::X509VerifyParamRef;\n    use super::x509::X509Ref;\n\n    /// Add name as an additional reference identifier that can match the peer's certificate\n    pub fn add_host(_verify_param: &mut X509VerifyParamRef, _host: &str) -> Result<(), ErrorStack> {\n        panic!(\"Not implemented\");\n    }\n\n    /// Set the verify cert store of `_ssl`\n    pub fn ssl_set_verify_cert_store(\n        _ssl: &mut SslRef,\n        _cert_store: &X509StoreRef,\n    ) -> Result<(), ErrorStack> {\n        panic!(\"Not implemented\");\n    }\n\n    /// Load the certificate into `_ssl`\n    pub fn ssl_use_certificate(_ssl: &mut SslRef, _cert: &X509Ref) -> Result<(), ErrorStack> {\n        panic!(\"Not implemented\");\n    }\n\n    /// Load the private key into `_ssl`\n    pub fn ssl_use_private_key<T>(_ssl: &mut SslRef, _key: &PKeyRef<T>) -> Result<(), ErrorStack>\n    where\n        T: HasPrivate,\n    {\n        panic!(\"Not implemented\");\n    }\n\n    /// Clear the error stack\n    pub fn clear_error_stack() {}\n\n    /// Create a new [Ssl] from &[SslAcceptor]\n    pub fn ssl_from_acceptor(_acceptor: &SslAcceptor) -> Result<Ssl, ErrorStack> {\n        panic!(\"Not implemented\");\n    }\n\n    /// Suspend the TLS handshake when a certificate is needed.\n    pub fn suspend_when_need_ssl_cert(_ssl: &mut SslRef) {\n        panic!(\"Not implemented\");\n    }\n\n    /// Unblock a TLS handshake after the certificate is set.\n    pub fn unblock_ssl_cert(_ssl: &mut SslRef) {\n        panic!(\"Not implemented\");\n    }\n\n    /// Whether the TLS error is SSL_ERROR_WANT_X509_LOOKUP\n    pub fn is_suspended_for_cert(_error: &super::ssl::Error) -> bool {\n        panic!(\"Not implemented\");\n    }\n\n    /// Add the certificate into the cert chain of `_ssl`\n    pub fn ssl_add_chain_cert(_ssl: &mut SslRef, _cert: &X509Ref) -> Result<(), ErrorStack> {\n        panic!(\"Not implemented\");\n    }\n\n    /// Set renegotiation\n    pub fn ssl_set_renegotiate_mode_freely(_ssl: &mut SslRef) {}\n\n    /// Set the curves/groups of `_ssl`\n    pub fn ssl_set_groups_list(_ssl: &mut SslRef, _groups: &str) -> Result<(), ErrorStack> {\n        panic!(\"Not implemented\");\n    }\n\n    /// Sets whether a second keyshare to be sent in client hello when PQ is used.\n    pub fn ssl_use_second_key_share(_ssl: &mut SslRef, _enabled: bool) {}\n\n    /// Get a mutable SslRef ouf of SslRef, which is a missing functionality even when holding &mut SslStream\n    /// # Safety\n    pub unsafe fn ssl_mut(_ssl: &SslRef) -> &mut SslRef {\n        panic!(\"Not implemented\");\n    }\n}\n\npub mod tokio_ssl {\n    use std::pin::Pin;\n    use std::task::{Context, Poll};\n    use tokio::io::{AsyncRead, AsyncWrite, ReadBuf};\n\n    use super::error::ErrorStack;\n    use super::ssl::{Error, Ssl, SslRef};\n\n    /// A TLS session over a stream.\n    #[derive(Debug)]\n    pub struct SslStream<S> {\n        marker: std::marker::PhantomData<S>,\n    }\n    impl<S> SslStream<S> {\n        /// Creates a new `SslStream`.\n        pub fn new(_ssl: Ssl, _stream: S) -> Result<Self, ErrorStack> {\n            panic!(\"Not implemented\");\n        }\n\n        /// Initiates a client-side TLS handshake.\n        pub async fn connect(self: Pin<&mut Self>) -> Result<(), Error> {\n            panic!(\"Not implemented\");\n        }\n\n        /// Initiates a server-side TLS handshake.\n        pub async fn accept(self: Pin<&mut Self>) -> Result<(), Error> {\n            panic!(\"Not implemented\");\n        }\n\n        /// Returns a shared reference to the `Ssl` object associated with this stream.\n        pub fn ssl(&self) -> &SslRef {\n            panic!(\"Not implemented\");\n        }\n\n        /// Returns a shared reference to the underlying stream.\n        pub fn get_ref(&self) -> &S {\n            panic!(\"Not implemented\");\n        }\n\n        /// Returns a mutable reference to the underlying stream.\n        pub fn get_mut(&mut self) -> &mut S {\n            panic!(\"Not implemented\");\n        }\n    }\n    impl<S> AsyncRead for SslStream<S>\n    where\n        S: AsyncRead + AsyncWrite,\n    {\n        fn poll_read(\n            self: Pin<&mut Self>,\n            _ctx: &mut Context<'_>,\n            _buf: &mut ReadBuf<'_>,\n        ) -> Poll<std::io::Result<()>> {\n            panic!(\"Not implemented\");\n        }\n    }\n    impl<S> AsyncWrite for SslStream<S>\n    where\n        S: AsyncRead + AsyncWrite,\n    {\n        fn poll_write(\n            self: Pin<&mut Self>,\n            _ctx: &mut Context<'_>,\n            _buf: &[u8],\n        ) -> Poll<std::io::Result<usize>> {\n            panic!(\"Not implemented\");\n        }\n\n        fn poll_flush(self: Pin<&mut Self>, _ctx: &mut Context<'_>) -> Poll<std::io::Result<()>> {\n            panic!(\"Not implemented\");\n        }\n\n        fn poll_shutdown(\n            self: Pin<&mut Self>,\n            _ctx: &mut Context<'_>,\n        ) -> Poll<std::io::Result<()>> {\n            panic!(\"Not implemented\");\n        }\n    }\n}\n"
  },
  {
    "path": "pingora-core/src/upstreams/mod.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! The interface to connect to a remote server\n\npub mod peer;\n"
  },
  {
    "path": "pingora-core/src/upstreams/peer.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! Defines where to connect to and how to connect to a remote server\n\nuse crate::connectors::{l4::BindTo, L4Connect};\nuse crate::protocols::l4::socket::SocketAddr;\nuse crate::protocols::tls::CaType;\n#[cfg(feature = \"openssl_derived\")]\nuse crate::protocols::tls::HandshakeCompleteHook;\n#[cfg(feature = \"s2n\")]\nuse crate::protocols::tls::PskType;\n#[cfg(unix)]\nuse crate::protocols::ConnFdReusable;\nuse crate::protocols::TcpKeepalive;\nuse crate::utils::tls::{get_organization_unit, CertKey};\nuse ahash::AHasher;\nuse derivative::Derivative;\nuse pingora_error::{\n    ErrorType::{InternalError, SocketError},\n    OrErr, Result,\n};\n#[cfg(feature = \"s2n\")]\nuse pingora_s2n::S2NPolicy;\nuse std::collections::BTreeMap;\nuse std::fmt::{Display, Formatter, Result as FmtResult};\nuse std::hash::{Hash, Hasher};\nuse std::net::{IpAddr, SocketAddr as InetSocketAddr, ToSocketAddrs as ToInetSocketAddrs};\n#[cfg(unix)]\nuse std::os::unix::{net::SocketAddr as UnixSocketAddr, prelude::AsRawFd};\n#[cfg(windows)]\nuse std::os::windows::io::AsRawSocket;\nuse std::path::{Path, PathBuf};\nuse std::sync::Arc;\nuse std::time::Duration;\nuse tokio::net::TcpSocket;\n\npub use crate::protocols::tls::ALPN;\n\n/// A hook function that may generate user data for [`crate::protocols::raw_connect::ProxyDigest`].\n///\n/// Takes the request and response headers from the proxy connection establishment, and may produce\n/// arbitrary data to be stored in ProxyDigest's user_data field.\n///\n/// This can be useful when, for example, you want to store some parameter(s) from the request or\n/// response headers from when the proxy connection was first established.\npub type ProxyDigestUserDataHook = Arc<\n    dyn Fn(\n            &http::request::Parts,         // request headers\n            &pingora_http::ResponseHeader, // response headers\n        ) -> Option<Box<dyn std::any::Any + Send + Sync>>\n        + Send\n        + Sync\n        + 'static,\n>;\n\n/// The interface to trace the connection\npub trait Tracing: Send + Sync + std::fmt::Debug {\n    /// This method is called when successfully connected to a remote server\n    fn on_connected(&self);\n    /// This method is called when the connection is disconnected.\n    fn on_disconnected(&self);\n    /// A way to clone itself\n    fn boxed_clone(&self) -> Box<dyn Tracing>;\n}\n\n/// An object-safe version of Tracing object that can use Clone\n#[derive(Debug)]\npub struct Tracer(pub Box<dyn Tracing>);\n\nimpl Clone for Tracer {\n    fn clone(&self) -> Self {\n        Tracer(self.0.boxed_clone())\n    }\n}\n\n/// [`Peer`] defines the interface to communicate with the [`crate::connectors`] regarding where to\n/// connect to and how to connect to it.\npub trait Peer: Display + Clone {\n    /// The remote address to connect to\n    fn address(&self) -> &SocketAddr;\n    /// If TLS should be used;\n    fn tls(&self) -> bool;\n    /// The SNI to send, if TLS is used\n    fn sni(&self) -> &str;\n    /// To decide whether a [`Peer`] can use the connection established by another [`Peer`].\n    ///\n    /// The connections to two peers are considered reusable to each other if their reuse hashes are\n    /// the same\n    fn reuse_hash(&self) -> u64;\n    /// Get the proxy setting to connect to the remote server\n    fn get_proxy(&self) -> Option<&Proxy> {\n        None\n    }\n    /// Get the additional options to connect to the peer.\n    ///\n    /// See [`PeerOptions`] for more details\n    fn get_peer_options(&self) -> Option<&PeerOptions> {\n        None\n    }\n    /// Get the additional options for modification.\n    fn get_mut_peer_options(&mut self) -> Option<&mut PeerOptions> {\n        None\n    }\n    /// Whether the TLS handshake should validate the cert of the server.\n    fn verify_cert(&self) -> bool {\n        match self.get_peer_options() {\n            Some(opt) => opt.verify_cert,\n            None => false,\n        }\n    }\n    /// Whether the TLS handshake should verify that the server cert matches the SNI.\n    fn verify_hostname(&self) -> bool {\n        match self.get_peer_options() {\n            Some(opt) => opt.verify_hostname,\n            None => false,\n        }\n    }\n    /// Whether the system trust store should be loaded and used when verifying certificates\n    #[cfg(feature = \"s2n\")]\n    fn use_system_certs(&self) -> bool {\n        match self.get_peer_options() {\n            Some(opt) => opt.use_system_certs,\n            None => false,\n        }\n    }\n    /// The alternative common name to use to verify the server cert.\n    ///\n    /// If the server cert doesn't match the SNI, this name will be used to\n    /// verify the cert.\n    fn alternative_cn(&self) -> Option<&String> {\n        match self.get_peer_options() {\n            Some(opt) => opt.alternative_cn.as_ref(),\n            None => None,\n        }\n    }\n    /// Information about the local source address this connection should be bound to.\n    fn bind_to(&self) -> Option<&BindTo> {\n        match self.get_peer_options() {\n            Some(opt) => opt.bind_to.as_ref(),\n            None => None,\n        }\n    }\n    /// How long connect() call should be wait before it returns a timeout error.\n    fn connection_timeout(&self) -> Option<Duration> {\n        match self.get_peer_options() {\n            Some(opt) => opt.connection_timeout,\n            None => None,\n        }\n    }\n    /// How long the overall connection establishment should take before a timeout error is returned.\n    fn total_connection_timeout(&self) -> Option<Duration> {\n        match self.get_peer_options() {\n            Some(opt) => opt.total_connection_timeout,\n            None => None,\n        }\n    }\n    /// If the connection can be reused, how long the connection should wait to be reused before it\n    /// shuts down.\n    fn idle_timeout(&self) -> Option<Duration> {\n        self.get_peer_options().and_then(|o| o.idle_timeout)\n    }\n\n    /// Get the ALPN preference.\n    fn get_alpn(&self) -> Option<&ALPN> {\n        self.get_peer_options().map(|opt| &opt.alpn)\n    }\n\n    /// Get the CA cert to use to validate the server cert.\n    ///\n    /// If not set, the default CAs will be used.\n    fn get_ca(&self) -> Option<&Arc<CaType>> {\n        match self.get_peer_options() {\n            Some(opt) => opt.ca.as_ref(),\n            None => None,\n        }\n    }\n\n    /// Get the client cert and key for mutual TLS if any\n    fn get_client_cert_key(&self) -> Option<&Arc<CertKey>> {\n        None\n    }\n\n    /// Get the PSK (pre-shared key) to use to validate the connection\n    ///\n    /// If not set, PSK validation will not be used\n    #[cfg(feature = \"s2n\")]\n    fn get_psk(&self) -> Option<&Arc<PskType>> {\n        match self.get_peer_options() {\n            Some(opt) => opt.psk.as_ref(),\n            None => None,\n        }\n    }\n\n    /// Get the Security Policy to use for this connection (S2N only)\n    ///\n    /// If not set, the default policy \"default_tls13\" will be used\n    /// https://aws.github.io/s2n-tls/usage-guide/ch06-security-policies.html\n    #[cfg(feature = \"s2n\")]\n    fn get_s2n_security_policy(&self) -> Option<&S2NPolicy> {\n        match self.get_peer_options() {\n            Some(opt) => opt.s2n_security_policy.as_ref(),\n            None => None,\n        }\n    }\n\n    /// S2N-TLS will delay a response up to the max blinding delay (default 30)\n    /// seconds whenever an error triggered by a peer occurs to mitigate against\n    /// timing side channels.\n    #[cfg(feature = \"s2n\")]\n    fn get_max_blinding_delay(&self) -> Option<u32> {\n        match self.get_peer_options() {\n            Some(opt) => opt.max_blinding_delay,\n            None => None,\n        }\n    }\n\n    /// The TCP keepalive setting that should be applied to this connection\n    fn tcp_keepalive(&self) -> Option<&TcpKeepalive> {\n        self.get_peer_options()\n            .and_then(|o| o.tcp_keepalive.as_ref())\n    }\n\n    /// The interval H2 pings to send to the server if any\n    fn h2_ping_interval(&self) -> Option<Duration> {\n        self.get_peer_options().and_then(|o| o.h2_ping_interval)\n    }\n\n    /// The size of the TCP receive buffer should be limited to. See SO_RCVBUF for more details.\n    fn tcp_recv_buf(&self) -> Option<usize> {\n        self.get_peer_options().and_then(|o| o.tcp_recv_buf)\n    }\n\n    /// The DSCP value that should be applied to the send side of this connection.\n    /// See the [RFC](https://datatracker.ietf.org/doc/html/rfc2474) for more details.\n    fn dscp(&self) -> Option<u8> {\n        self.get_peer_options().and_then(|o| o.dscp)\n    }\n\n    /// Whether to enable TCP fast open.\n    fn tcp_fast_open(&self) -> bool {\n        self.get_peer_options()\n            .map(|o| o.tcp_fast_open)\n            .unwrap_or_default()\n    }\n\n    #[cfg(unix)]\n    fn matches_fd<V: AsRawFd>(&self, fd: V) -> bool {\n        self.address().check_fd_match(fd)\n    }\n\n    #[cfg(windows)]\n    fn matches_sock<V: AsRawSocket>(&self, sock: V) -> bool {\n        use crate::protocols::ConnSockReusable;\n        self.address().check_sock_match(sock)\n    }\n\n    fn get_tracer(&self) -> Option<Tracer> {\n        None\n    }\n\n    /// Returns a hook that should be run before an upstream TCP connection is connected.\n    ///\n    /// This hook can be used to set additional socket options.\n    fn upstream_tcp_sock_tweak_hook(\n        &self,\n    ) -> Option<&Arc<dyn Fn(&TcpSocket) -> Result<()> + Send + Sync + 'static>> {\n        self.get_peer_options()?\n            .upstream_tcp_sock_tweak_hook\n            .as_ref()\n    }\n\n    /// Returns a [`ProxyDigestUserDataHook`] that may generate user data for\n    /// [`crate::protocols::raw_connect::ProxyDigest`] when establishing a new proxy connection.\n    fn proxy_digest_user_data_hook(&self) -> Option<&ProxyDigestUserDataHook> {\n        self.get_peer_options()?\n            .proxy_digest_user_data_hook\n            .as_ref()\n    }\n\n    /// Returns a hook that should be run on TLS handshake completion.\n    ///\n    /// Any value returned from the returned hook (other than `None`) will be stored in the\n    /// `extension` field of `SslDigest`. This allows you to attach custom application-specific\n    /// data to the TLS connection, which will be accessible from the HTTP layer via the\n    /// `SslDigest` attached to the session digest.\n    ///\n    /// Currently only enabled for openssl variants with meaningful `TlsRef`s.\n    #[cfg(feature = \"openssl_derived\")]\n    fn upstream_tls_handshake_complete_hook(&self) -> Option<&HandshakeCompleteHook> {\n        self.get_peer_options()?\n            .upstream_tls_handshake_complete_hook\n            .as_ref()\n    }\n}\n\n/// A simple TCP or TLS peer without many complicated settings.\n#[derive(Debug, Clone)]\npub struct BasicPeer {\n    pub _address: SocketAddr,\n    pub sni: String,\n    pub options: PeerOptions,\n}\n\nimpl BasicPeer {\n    /// Create a new [`BasicPeer`].\n    pub fn new(address: &str) -> Self {\n        let addr = SocketAddr::Inet(address.parse().unwrap()); // TODO: check error\n        Self::new_from_sockaddr(addr)\n    }\n\n    /// Create a new [`BasicPeer`] with the given path to a Unix domain socket.\n    #[cfg(unix)]\n    pub fn new_uds<P: AsRef<Path>>(path: P) -> Result<Self> {\n        let addr = SocketAddr::Unix(\n            UnixSocketAddr::from_pathname(path.as_ref())\n                .or_err(InternalError, \"while creating BasicPeer\")?,\n        );\n        Ok(Self::new_from_sockaddr(addr))\n    }\n\n    fn new_from_sockaddr(sockaddr: SocketAddr) -> Self {\n        BasicPeer {\n            _address: sockaddr,\n            sni: \"\".to_string(), // TODO: add support for SNI\n            options: PeerOptions::new(),\n        }\n    }\n}\n\nimpl Display for BasicPeer {\n    fn fmt(&self, f: &mut Formatter<'_>) -> FmtResult {\n        write!(f, \"{:?}\", self)\n    }\n}\n\nimpl Peer for BasicPeer {\n    fn address(&self) -> &SocketAddr {\n        &self._address\n    }\n\n    fn tls(&self) -> bool {\n        !self.sni.is_empty()\n    }\n\n    fn bind_to(&self) -> Option<&BindTo> {\n        None\n    }\n\n    fn sni(&self) -> &str {\n        &self.sni\n    }\n\n    // TODO: change connection pool to accept u64 instead of String\n    fn reuse_hash(&self) -> u64 {\n        let mut hasher = AHasher::default();\n        self._address.hash(&mut hasher);\n        hasher.finish()\n    }\n\n    fn get_peer_options(&self) -> Option<&PeerOptions> {\n        Some(&self.options)\n    }\n}\n\n/// Define whether to connect via http or https\n#[derive(Hash, Clone, Debug, PartialEq)]\npub enum Scheme {\n    HTTP,\n    HTTPS,\n}\n\nimpl Display for Scheme {\n    fn fmt(&self, f: &mut Formatter<'_>) -> FmtResult {\n        match self {\n            Scheme::HTTP => write!(f, \"HTTP\"),\n            Scheme::HTTPS => write!(f, \"HTTPS\"),\n        }\n    }\n}\n\nimpl Scheme {\n    pub fn from_tls_bool(tls: bool) -> Self {\n        if tls {\n            Self::HTTPS\n        } else {\n            Self::HTTP\n        }\n    }\n}\n\n/// The preferences to connect to a remote server\n///\n/// See [`Peer`] for the meaning of the fields\n#[non_exhaustive]\n#[derive(Clone, Derivative)]\n#[derivative(Debug)]\npub struct PeerOptions {\n    pub bind_to: Option<BindTo>,\n    pub connection_timeout: Option<Duration>,\n    pub total_connection_timeout: Option<Duration>,\n    pub read_timeout: Option<Duration>,\n    pub idle_timeout: Option<Duration>,\n    pub write_timeout: Option<Duration>,\n    pub verify_cert: bool,\n    pub verify_hostname: bool,\n    #[cfg(feature = \"s2n\")]\n    pub use_system_certs: bool,\n    /* accept the cert if it's CN matches the SNI or this name */\n    pub alternative_cn: Option<String>,\n    pub alpn: ALPN,\n    pub ca: Option<Arc<CaType>>,\n    pub tcp_keepalive: Option<TcpKeepalive>,\n    pub tcp_recv_buf: Option<usize>,\n    pub dscp: Option<u8>,\n    pub h2_ping_interval: Option<Duration>,\n    #[cfg(feature = \"s2n\")]\n    pub psk: Option<Arc<PskType>>,\n    #[cfg(feature = \"s2n\")]\n    pub s2n_security_policy: Option<S2NPolicy>,\n    #[cfg(feature = \"s2n\")]\n    pub max_blinding_delay: Option<u32>,\n    // how many concurrent h2 stream are allowed in the same connection\n    pub max_h2_streams: usize,\n    /// Allow invalid Content-Length in HTTP/1 responses (non-RFC compliant).\n    ///\n    /// When enabled, invalid Content-Length responses are treated as close-delimited responses.\n    ///\n    /// **Note:** This field is unstable and may be removed or changed in future versions.\n    /// It exists primarily for compatibility with legacy servers that send malformed headers.\n    pub allow_h1_response_invalid_content_length: bool,\n    pub extra_proxy_headers: BTreeMap<String, Vec<u8>>,\n    // The list of curve the tls connection should advertise\n    // if `None`, the default curves will be used\n    pub curves: Option<&'static str>,\n    // see ssl_use_second_key_share\n    pub second_keyshare: bool,\n    // whether to enable TCP fast open\n    pub tcp_fast_open: bool,\n    // use Arc because Clone is required but not allowed in trait object\n    pub tracer: Option<Tracer>,\n    // A custom L4 connector to use to establish new L4 connections\n    pub custom_l4: Option<Arc<dyn L4Connect + Send + Sync>>,\n    #[derivative(Debug = \"ignore\")]\n    pub upstream_tcp_sock_tweak_hook:\n        Option<Arc<dyn Fn(&TcpSocket) -> Result<()> + Send + Sync + 'static>>,\n    #[derivative(Debug = \"ignore\")]\n    pub proxy_digest_user_data_hook: Option<ProxyDigestUserDataHook>,\n    /// Hook that allows returning an optional `SslDigestExtension`.\n    /// Any returned value will be saved into the `SslDigest`.\n    ///\n    /// Currently only enabled for openssl variants with meaningful `TlsRef`s.\n    #[cfg(feature = \"openssl_derived\")]\n    #[derivative(Debug = \"ignore\")]\n    pub upstream_tls_handshake_complete_hook: Option<HandshakeCompleteHook>,\n}\n\nimpl PeerOptions {\n    /// Create a new [`PeerOptions`]\n    pub fn new() -> Self {\n        PeerOptions {\n            bind_to: None,\n            connection_timeout: None,\n            total_connection_timeout: None,\n            read_timeout: None,\n            idle_timeout: None,\n            write_timeout: None,\n            verify_cert: true,\n            verify_hostname: true,\n            #[cfg(feature = \"s2n\")]\n            use_system_certs: true,\n            alternative_cn: None,\n            alpn: ALPN::H1,\n            ca: None,\n            tcp_keepalive: None,\n            tcp_recv_buf: None,\n            dscp: None,\n            h2_ping_interval: None,\n            #[cfg(feature = \"s2n\")]\n            psk: None,\n            #[cfg(feature = \"s2n\")]\n            s2n_security_policy: None,\n            #[cfg(feature = \"s2n\")]\n            max_blinding_delay: None,\n            max_h2_streams: 1,\n            allow_h1_response_invalid_content_length: false,\n            extra_proxy_headers: BTreeMap::new(),\n            curves: None,\n            second_keyshare: true, // default true and noop when not using PQ curves\n            tcp_fast_open: false,\n            tracer: None,\n            custom_l4: None,\n            upstream_tcp_sock_tweak_hook: None,\n            proxy_digest_user_data_hook: None,\n            #[cfg(feature = \"openssl_derived\")]\n            upstream_tls_handshake_complete_hook: None,\n        }\n    }\n\n    /// Set the ALPN according to the `max` and `min` constrains.\n    pub fn set_http_version(&mut self, max: u8, min: u8) {\n        self.alpn = ALPN::new(max, min);\n    }\n}\n\nimpl Display for PeerOptions {\n    fn fmt(&self, f: &mut Formatter<'_>) -> FmtResult {\n        if let Some(b) = self.bind_to.as_ref() {\n            write!(f, \"bind_to: {:?},\", b)?;\n        }\n        if let Some(t) = self.connection_timeout {\n            write!(f, \"conn_timeout: {:?},\", t)?;\n        }\n        if let Some(t) = self.total_connection_timeout {\n            write!(f, \"total_conn_timeout: {:?},\", t)?;\n        }\n        if self.verify_cert {\n            write!(f, \"verify_cert: true,\")?;\n        }\n        if self.verify_hostname {\n            write!(f, \"verify_hostname: true,\")?;\n        }\n        #[cfg(feature = \"s2n\")]\n        if self.use_system_certs {\n            write!(f, \"use_system_certs: true,\")?;\n        }\n        if let Some(cn) = &self.alternative_cn {\n            write!(f, \"alt_cn: {},\", cn)?;\n        }\n        write!(f, \"alpn: {},\", self.alpn)?;\n        if let Some(cas) = &self.ca {\n            for ca in cas.iter() {\n                write!(\n                    f,\n                    \"CA: {}, expire: {},\",\n                    get_organization_unit(ca).unwrap_or_default(),\n                    ca.not_after()\n                )?;\n            }\n        }\n        #[cfg(feature = \"s2n\")]\n        if let Some(policy) = &self.s2n_security_policy {\n            write!(f, \"s2n_security_policy: {:?}, \", policy)?;\n        }\n        #[cfg(feature = \"s2n\")]\n        if let Some(psk_config) = &self.psk {\n            for psk in &psk_config.keys {\n                write!(\n                    f,\n                    \"psk_identity: {}\",\n                    String::from_utf8_lossy(psk.identity.as_slice())\n                )?;\n            }\n        }\n        if let Some(tcp_keepalive) = &self.tcp_keepalive {\n            write!(f, \"tcp_keepalive: {},\", tcp_keepalive)?;\n        }\n        if let Some(h2_ping_interval) = self.h2_ping_interval {\n            write!(f, \"h2_ping_interval: {:?},\", h2_ping_interval)?;\n        }\n        Ok(())\n    }\n}\n\n/// A peer representing the remote HTTP server to connect to\n#[derive(Debug, Clone)]\npub struct HttpPeer {\n    pub _address: SocketAddr,\n    pub scheme: Scheme,\n    pub sni: String,\n    pub proxy: Option<Proxy>,\n    pub client_cert_key: Option<Arc<CertKey>>,\n    /// a custom field to isolate connection reuse. Requests with different group keys\n    /// cannot share connections with each other.\n    pub group_key: u64,\n    pub options: PeerOptions,\n}\n\nimpl HttpPeer {\n    // These methods are pretty ad-hoc\n    pub fn is_tls(&self) -> bool {\n        match self.scheme {\n            Scheme::HTTP => false,\n            Scheme::HTTPS => true,\n        }\n    }\n\n    fn new_from_sockaddr(address: SocketAddr, tls: bool, sni: String) -> Self {\n        HttpPeer {\n            _address: address,\n            scheme: Scheme::from_tls_bool(tls),\n            sni,\n            proxy: None,\n            client_cert_key: None,\n            group_key: 0,\n            options: PeerOptions::new(),\n        }\n    }\n\n    /// Create a new [`HttpPeer`] with the given socket address and TLS settings.\n    pub fn new<A: ToInetSocketAddrs>(address: A, tls: bool, sni: String) -> Self {\n        let mut addrs_iter = address.to_socket_addrs().unwrap(); //TODO: handle error\n        let addr = addrs_iter.next().unwrap();\n        Self::new_from_sockaddr(SocketAddr::Inet(addr), tls, sni)\n    }\n\n    /// Create a new [`HttpPeer`] with the given path to Unix domain socket and TLS settings.\n    #[cfg(unix)]\n    pub fn new_uds(path: &str, tls: bool, sni: String) -> Result<Self> {\n        let addr = SocketAddr::Unix(\n            UnixSocketAddr::from_pathname(Path::new(path)).or_err(SocketError, \"invalid path\")?,\n        );\n        Ok(Self::new_from_sockaddr(addr, tls, sni))\n    }\n\n    /// Create a new [`HttpPeer`] that uses a proxy to connect to the upstream IP and port\n    /// combination.\n    pub fn new_proxy(\n        next_hop: &str,\n        ip_addr: IpAddr,\n        port: u16,\n        tls: bool,\n        sni: &str,\n        headers: BTreeMap<String, Vec<u8>>,\n    ) -> Self {\n        HttpPeer {\n            _address: SocketAddr::Inet(InetSocketAddr::new(ip_addr, port)),\n            scheme: Scheme::from_tls_bool(tls),\n            sni: sni.to_string(),\n            proxy: Some(Proxy {\n                next_hop: PathBuf::from(next_hop).into(),\n                host: ip_addr.to_string(),\n                port,\n                headers,\n            }),\n            client_cert_key: None,\n            group_key: 0,\n            options: PeerOptions::new(),\n        }\n    }\n\n    /// Create a new [`HttpPeer`] with client certificate and key for mutual TLS.\n    pub fn new_mtls<A: ToInetSocketAddrs>(\n        address: A,\n        sni: String,\n        client_cert_key: Arc<CertKey>,\n    ) -> Self {\n        let mut peer = Self::new(address, true, sni);\n        peer.client_cert_key = Some(client_cert_key);\n        peer\n    }\n\n    fn peer_hash(&self) -> u64 {\n        let mut hasher = AHasher::default();\n        self.hash(&mut hasher);\n        hasher.finish()\n    }\n}\n\nimpl Hash for HttpPeer {\n    fn hash<H: Hasher>(&self, state: &mut H) {\n        self._address.hash(state);\n        self.scheme.hash(state);\n        self.proxy.hash(state);\n        self.sni.hash(state);\n        // client cert serial\n        self.client_cert_key.hash(state);\n        // origin server cert verification\n        self.verify_cert().hash(state);\n        self.verify_hostname().hash(state);\n        self.alternative_cn().hash(state);\n        #[cfg(feature = \"s2n\")]\n        self.get_psk().hash(state);\n        self.group_key.hash(state);\n        // max h2 stream settings\n        self.options.max_h2_streams.hash(state);\n    }\n}\n\nimpl Display for HttpPeer {\n    fn fmt(&self, f: &mut Formatter<'_>) -> FmtResult {\n        write!(f, \"addr: {}, scheme: {}\", self._address, self.scheme)?;\n        if !self.sni.is_empty() {\n            write!(f, \", sni: {}\", self.sni)?;\n        }\n        if let Some(p) = self.proxy.as_ref() {\n            write!(f, \", proxy: {p}\")?;\n        }\n        if let Some(cert) = &self.client_cert_key {\n            write!(f, \", client cert: {}\", cert)?;\n        }\n        Ok(())\n    }\n}\n\nimpl Peer for HttpPeer {\n    fn address(&self) -> &SocketAddr {\n        &self._address\n    }\n\n    fn tls(&self) -> bool {\n        self.is_tls()\n    }\n\n    fn sni(&self) -> &str {\n        &self.sni\n    }\n\n    // TODO: change connection pool to accept u64 instead of String\n    fn reuse_hash(&self) -> u64 {\n        self.peer_hash()\n    }\n\n    fn get_peer_options(&self) -> Option<&PeerOptions> {\n        Some(&self.options)\n    }\n\n    fn get_mut_peer_options(&mut self) -> Option<&mut PeerOptions> {\n        Some(&mut self.options)\n    }\n\n    fn get_proxy(&self) -> Option<&Proxy> {\n        self.proxy.as_ref()\n    }\n\n    #[cfg(unix)]\n    fn matches_fd<V: AsRawFd>(&self, fd: V) -> bool {\n        if let Some(proxy) = self.get_proxy() {\n            proxy.next_hop.check_fd_match(fd)\n        } else {\n            self.address().check_fd_match(fd)\n        }\n    }\n\n    #[cfg(windows)]\n    fn matches_sock<V: AsRawSocket>(&self, sock: V) -> bool {\n        use crate::protocols::ConnSockReusable;\n\n        if let Some(proxy) = self.get_proxy() {\n            panic!(\"windows do not support peers with proxy\")\n        } else {\n            self.address().check_sock_match(sock)\n        }\n    }\n\n    fn get_client_cert_key(&self) -> Option<&Arc<CertKey>> {\n        self.client_cert_key.as_ref()\n    }\n\n    fn get_tracer(&self) -> Option<Tracer> {\n        self.options.tracer.clone()\n    }\n}\n\n/// The proxy settings to connect to the remote server, CONNECT only for now\n#[derive(Debug, Hash, Clone)]\npub struct Proxy {\n    pub next_hop: Box<Path>, // for now this will be the path to the UDS\n    pub host: String,        // the proxied host. Could be either IP addr or hostname.\n    pub port: u16,           // the port to proxy to\n    pub headers: BTreeMap<String, Vec<u8>>, // the additional headers to add to CONNECT\n}\n\nimpl Display for Proxy {\n    fn fmt(&self, f: &mut Formatter) -> FmtResult {\n        write!(\n            f,\n            \"next_hop: {}, host: {}, port: {}\",\n            self.next_hop.display(),\n            self.host,\n            self.port\n        )\n    }\n}\n"
  },
  {
    "path": "pingora-core/src/utils/mod.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! This module contains various types that make it easier to work with bytes and X509\n//! certificates.\n\n#[cfg(feature = \"any_tls\")]\npub mod tls;\n\n#[cfg(not(feature = \"any_tls\"))]\npub use crate::tls::utils as tls;\n\nuse bytes::Bytes;\n\n/// A `BufRef` is a reference to a buffer of bytes. It removes the need for self-referential data\n/// structures. It is safe to use as long as the underlying buffer does not get mutated.\n///\n/// # Panics\n///\n/// This will panic if an index is out of bounds.\n#[derive(Clone, PartialEq, Eq, Debug)]\npub struct BufRef(pub usize, pub usize);\n\nimpl BufRef {\n    /// Return a sub-slice of `buf`.\n    pub fn get<'a>(&self, buf: &'a [u8]) -> &'a [u8] {\n        &buf[self.0..self.1]\n    }\n\n    /// Return a slice of `buf`. This operation is O(1) and increases the reference count of `buf`.\n    pub fn get_bytes(&self, buf: &Bytes) -> Bytes {\n        buf.slice(self.0..self.1)\n    }\n\n    /// Return the size of the slice reference.\n    pub fn len(&self) -> usize {\n        self.1 - self.0\n    }\n\n    /// Return true if the length is zero.\n    pub fn is_empty(&self) -> bool {\n        self.1 == self.0\n    }\n}\n\nimpl BufRef {\n    /// Initialize a `BufRef` that can reference a slice beginning at index `start` and has a\n    /// length of `len`.\n    pub fn new(start: usize, len: usize) -> Self {\n        BufRef(start, start + len)\n    }\n}\n\n/// A `KVRef` contains a key name and value pair, stored as two [BufRef] types.\n#[derive(Clone)]\npub struct KVRef {\n    name: BufRef,\n    value: BufRef,\n}\n\nimpl KVRef {\n    /// Like [BufRef::get] for the name.\n    pub fn get_name<'a>(&self, buf: &'a [u8]) -> &'a [u8] {\n        self.name.get(buf)\n    }\n\n    /// Like [BufRef::get] for the value.\n    pub fn get_value<'a>(&self, buf: &'a [u8]) -> &'a [u8] {\n        self.value.get(buf)\n    }\n\n    /// Like [BufRef::get_bytes] for the name.\n    pub fn get_name_bytes(&self, buf: &Bytes) -> Bytes {\n        self.name.get_bytes(buf)\n    }\n\n    /// Like [BufRef::get_bytes] for the value.\n    pub fn get_value_bytes(&self, buf: &Bytes) -> Bytes {\n        self.value.get_bytes(buf)\n    }\n\n    /// Return a new `KVRef` with name and value start indices and lengths.\n    pub fn new(name_s: usize, name_len: usize, value_s: usize, value_len: usize) -> Self {\n        KVRef {\n            name: BufRef(name_s, name_s + name_len),\n            value: BufRef(value_s, value_s + value_len),\n        }\n    }\n\n    /// Return a reference to the value.\n    pub fn value(&self) -> &BufRef {\n        &self.value\n    }\n}\n\n/// A [KVRef] which contains empty sub-slices.\npub const EMPTY_KV_REF: KVRef = KVRef {\n    name: BufRef(0, 0),\n    value: BufRef(0, 0),\n};\n"
  },
  {
    "path": "pingora-core/src/utils/tls/boringssl_openssl.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\nuse crate::tls::{nid::Nid, pkey::PKey, pkey::Private, x509::X509};\nuse crate::Result;\nuse pingora_error::{ErrorType::*, OrErr};\nuse std::hash::{Hash, Hasher};\n\nfn get_subject_name(cert: &X509, name_type: Nid) -> Option<String> {\n    cert.subject_name()\n        .entries_by_nid(name_type)\n        .next()\n        .map(|name| {\n            name.data()\n                .as_utf8()\n                .map(|s| s.to_string())\n                .unwrap_or_default()\n        })\n}\n\n/// Return the organization associated with the X509 certificate.\npub fn get_organization(cert: &X509) -> Option<String> {\n    get_subject_name(cert, Nid::ORGANIZATIONNAME)\n}\n\n/// Return the common name associated with the X509 certificate.\npub fn get_common_name(cert: &X509) -> Option<String> {\n    get_subject_name(cert, Nid::COMMONNAME)\n}\n\n/// Return the common name associated with the X509 certificate.\npub fn get_organization_unit(cert: &X509) -> Option<String> {\n    get_subject_name(cert, Nid::ORGANIZATIONALUNITNAME)\n}\n\n/// Return the serial number associated with the X509 certificate as a hexadecimal value.\npub fn get_serial(cert: &X509) -> Result<String> {\n    let bn = cert\n        .serial_number()\n        .to_bn()\n        .or_err(InvalidCert, \"Invalid serial\")?;\n    let hex = bn.to_hex_str().or_err(InvalidCert, \"Invalid serial\")?;\n\n    let hex_str: &str = hex.as_ref();\n    Ok(hex_str.to_owned())\n}\n\n/// This type contains a list of one or more certificates and an associated private key. The leaf\n/// certificate should always be first.\n#[derive(Clone)]\npub struct CertKey {\n    certificates: Vec<X509>,\n    key: PKey<Private>,\n}\n\nimpl CertKey {\n    /// Create a new `CertKey` given a list of certificates and a private key.\n    pub fn new(certificates: Vec<X509>, key: PKey<Private>) -> CertKey {\n        assert!(\n            !certificates.is_empty(),\n            \"expected a non-empty vector of certificates in CertKey::new\"\n        );\n\n        CertKey { certificates, key }\n    }\n\n    /// Peek at the leaf certificate.\n    pub fn leaf(&self) -> &X509 {\n        // This is safe due to the assertion above.\n        &self.certificates[0]\n    }\n\n    /// Return the key.\n    pub fn key(&self) -> &PKey<Private> {\n        &self.key\n    }\n\n    /// Return a slice of intermediate certificates. An empty slice means there are none.\n    pub fn intermediates(&self) -> &[X509] {\n        if self.certificates.len() <= 1 {\n            return &[];\n        }\n        &self.certificates[1..]\n    }\n\n    /// Return the organization from the leaf certificate.\n    pub fn organization(&self) -> Option<String> {\n        get_organization(self.leaf())\n    }\n\n    /// Return the serial from the leaf certificate.\n    pub fn serial(&self) -> Result<String> {\n        get_serial(self.leaf())\n    }\n}\n\nimpl Hash for CertKey {\n    fn hash<H: Hasher>(&self, state: &mut H) {\n        for certificate in &self.certificates {\n            if let Ok(serial) = get_serial(certificate) {\n                serial.hash(state)\n            }\n        }\n    }\n}\n\n// hide private key\nimpl std::fmt::Debug for CertKey {\n    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {\n        f.debug_struct(\"CertKey\")\n            .field(\"X509\", &self.leaf())\n            .finish()\n    }\n}\n\nimpl std::fmt::Display for CertKey {\n    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {\n        let leaf = self.leaf();\n        if let Some(cn) = get_common_name(leaf) {\n            // Write CN if it exists\n            write!(f, \"CN: {cn},\")?;\n        } else if let Some(org_unit) = get_organization_unit(leaf) {\n            // CA cert might not have CN, so print its unit name instead\n            write!(f, \"Org Unit: {org_unit},\")?;\n        }\n        write!(f, \", expire: {}\", leaf.not_after())\n        // ignore the details of the private key\n    }\n}\n"
  },
  {
    "path": "pingora-core/src/utils/tls/mod.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n#[cfg(feature = \"openssl_derived\")]\nmod boringssl_openssl;\n\n#[cfg(feature = \"openssl_derived\")]\npub use boringssl_openssl::*;\n\n#[cfg(feature = \"rustls\")]\nmod rustls;\n\n#[cfg(feature = \"rustls\")]\npub use rustls::*;\n\n#[cfg(feature = \"s2n\")]\nmod s2n;\n\n#[cfg(feature = \"s2n\")]\npub use s2n::*;\n"
  },
  {
    "path": "pingora-core/src/utils/tls/rustls.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\nuse ouroboros::self_referencing;\nuse pingora_error::Result;\nuse pingora_rustls::CertificateDer;\nuse std::hash::{Hash, Hasher};\nuse x509_parser::prelude::{FromDer, X509Certificate};\n\n/// Get the organization and serial number associated with the given certificate\n/// see https://en.wikipedia.org/wiki/X.509#Structure_of_a_certificate\npub fn get_organization_serial(x509cert: &WrappedX509) -> Result<(Option<String>, String)> {\n    let serial = get_serial(x509cert)?;\n    Ok((get_organization(x509cert), serial))\n}\n\nfn get_organization_serial_x509(\n    x509cert: &X509Certificate<'_>,\n) -> Result<(Option<String>, String)> {\n    let serial = x509cert.raw_serial_as_string();\n    Ok((get_organization_x509(x509cert), serial))\n}\n\n/// Get the serial number associated with the given certificate\n/// see https://en.wikipedia.org/wiki/X.509#Structure_of_a_certificate\npub fn get_serial(x509cert: &WrappedX509) -> Result<String> {\n    Ok(x509cert.borrow_cert().raw_serial_as_string())\n}\n\n/// Return the organization associated with the X509 certificate.\n/// see https://en.wikipedia.org/wiki/X.509#Structure_of_a_certificate\npub fn get_organization(x509cert: &WrappedX509) -> Option<String> {\n    get_organization_x509(x509cert.borrow_cert())\n}\n\n/// Return the organization associated with the X509 certificate.\n/// see https://en.wikipedia.org/wiki/X.509#Structure_of_a_certificate\npub fn get_organization_x509(x509cert: &X509Certificate<'_>) -> Option<String> {\n    x509cert\n        .subject\n        .iter_organization()\n        .filter_map(|a| a.as_str().ok())\n        .map(|a| a.to_string())\n        .reduce(|cur, next| cur + &next)\n}\n\n/// Return the organization associated with the X509 certificate (as bytes).\n/// see https://en.wikipedia.org/wiki/X.509#Structure_of_a_certificate\npub fn get_organization_serial_bytes(cert: &[u8]) -> Result<(Option<String>, String)> {\n    let (_, x509cert) = x509_parser::certificate::X509Certificate::from_der(cert)\n        .expect(\"Failed to parse certificate from DER format.\");\n\n    get_organization_serial_x509(&x509cert)\n}\n\n/// Return the organization unit associated with the X509 certificate.\n/// see https://en.wikipedia.org/wiki/X.509#Structure_of_a_certificate\npub fn get_organization_unit(x509cert: &WrappedX509) -> Option<String> {\n    x509cert\n        .borrow_cert()\n        .subject\n        .iter_organizational_unit()\n        .filter_map(|a| a.as_str().ok())\n        .map(|a| a.to_string())\n        .reduce(|cur, next| cur + &next)\n}\n\n/// Get a combination of the common names for the given certificate\n/// see https://en.wikipedia.org/wiki/X.509#Structure_of_a_certificate\npub fn get_common_name(x509cert: &WrappedX509) -> Option<String> {\n    x509cert\n        .borrow_cert()\n        .subject\n        .iter_common_name()\n        .filter_map(|a| a.as_str().ok())\n        .map(|a| a.to_string())\n        .reduce(|cur, next| cur + &next)\n}\n\n/// Get the `not_after` field for the valid time period for the given cert\n/// see https://en.wikipedia.org/wiki/X.509#Structure_of_a_certificate\npub fn get_not_after(x509cert: &WrappedX509) -> String {\n    x509cert.borrow_cert().validity.not_after.to_string()\n}\n\n/// This type contains a list of one or more certificates and an associated private key. The leaf\n/// certificate should always be first.\npub struct CertKey {\n    key: Vec<u8>,\n    certificates: Vec<WrappedX509>,\n}\n\n#[self_referencing]\n#[derive(Debug)]\npub struct WrappedX509 {\n    raw_cert: Vec<u8>,\n\n    #[borrows(raw_cert)]\n    #[covariant]\n    cert: X509Certificate<'this>,\n}\n\nfn parse_x509<C>(raw_cert: &C) -> X509Certificate<'_>\nwhere\n    C: AsRef<[u8]>,\n{\n    X509Certificate::from_der(raw_cert.as_ref())\n        .expect(\"Failed to parse certificate from DER format.\")\n        .1\n}\n\nimpl Clone for CertKey {\n    fn clone(&self) -> Self {\n        CertKey {\n            key: self.key.clone(),\n            certificates: self\n                .certificates\n                .iter()\n                .map(|wrapper| WrappedX509::new(wrapper.borrow_raw_cert().clone(), parse_x509))\n                .collect::<Vec<_>>(),\n        }\n    }\n}\n\nimpl CertKey {\n    /// Create a new `CertKey` given a list of certificates and a private key.\n    pub fn new(certificates: Vec<Vec<u8>>, key: Vec<u8>) -> CertKey {\n        assert!(\n            !certificates.is_empty() && !certificates.first().unwrap().is_empty(),\n            \"expected a non-empty vector of certificates in CertKey::new\"\n        );\n\n        CertKey {\n            key,\n            certificates: certificates\n                .into_iter()\n                .map(|raw_cert| WrappedX509::new(raw_cert, parse_x509))\n                .collect::<Vec<_>>(),\n        }\n    }\n\n    /// Peek at the leaf certificate.\n    pub fn leaf(&self) -> &WrappedX509 {\n        // This is safe due to the assertion in creation of a `CertKey`\n        &self.certificates[0]\n    }\n\n    /// Return the key.\n    pub fn key(&self) -> &Vec<u8> {\n        &self.key\n    }\n\n    /// Return a slice of intermediate certificates. An empty slice means there are none.\n    pub fn intermediates(&self) -> Vec<&WrappedX509> {\n        self.certificates.iter().skip(1).collect()\n    }\n\n    /// Return the organization from the leaf certificate.\n    pub fn organization(&self) -> Option<String> {\n        get_organization(self.leaf())\n    }\n\n    /// Return the serial from the leaf certificate.\n    pub fn serial(&self) -> String {\n        get_serial(self.leaf()).unwrap()\n    }\n}\n\nimpl WrappedX509 {\n    pub fn not_after(&self) -> String {\n        self.borrow_cert().validity.not_after.to_string()\n    }\n}\n\n// hide private key\nimpl std::fmt::Debug for CertKey {\n    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {\n        f.debug_struct(\"CertKey\")\n            .field(\"X509\", &self.leaf())\n            .finish()\n    }\n}\n\nimpl std::fmt::Display for CertKey {\n    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {\n        let leaf = self.leaf();\n        if let Some(cn) = get_common_name(leaf) {\n            // Write CN if it exists\n            write!(f, \"CN: {cn},\")?;\n        } else if let Some(org_unit) = get_organization_unit(leaf) {\n            // CA cert might not have CN, so print its unit name instead\n            write!(f, \"Org Unit: {org_unit},\")?;\n        }\n        write!(f, \", expire: {}\", get_not_after(leaf))\n        // ignore the details of the private key\n    }\n}\n\nimpl Hash for CertKey {\n    fn hash<H: Hasher>(&self, state: &mut H) {\n        for certificate in &self.certificates {\n            if let Ok(serial) = get_serial(certificate) {\n                serial.hash(state)\n            }\n        }\n    }\n}\n\nimpl<'a> From<&'a WrappedX509> for CertificateDer<'static> {\n    fn from(value: &'a WrappedX509) -> Self {\n        CertificateDer::from(value.borrow_raw_cert().as_slice().to_owned())\n    }\n}\n"
  },
  {
    "path": "pingora-core/src/utils/tls/s2n.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\nuse ouroboros::self_referencing;\nuse pingora_error::Result;\nuse std::hash::{Hash, Hasher};\nuse x509_parser::{\n    pem::Pem,\n    prelude::{FromDer, X509Certificate},\n};\n\nfn get_organization_serial_x509(\n    x509cert: &X509Certificate<'_>,\n) -> Result<(Option<String>, String)> {\n    let serial = x509cert.raw_serial_as_string();\n    Ok((get_organization_x509(x509cert), serial))\n}\n\n/// Get the serial number associated with the given certificate\n/// see https://en.wikipedia.org/wiki/X.509#Structure_of_a_certificate\npub fn get_serial(x509cert: &WrappedX509) -> Result<String> {\n    Ok(x509cert.borrow_cert().raw_serial_as_string())\n}\n\n/// Return the organization associated with the X509 certificate.\n/// see https://en.wikipedia.org/wiki/X.509#Structure_of_a_certificate\npub fn get_organization(x509cert: &WrappedX509) -> Option<String> {\n    get_organization_x509(x509cert.borrow_cert())\n}\n\n/// Return the organization associated with the X509 certificate.\n/// see https://en.wikipedia.org/wiki/X.509#Structure_of_a_certificate\npub fn get_organization_x509(x509cert: &X509Certificate<'_>) -> Option<String> {\n    x509cert\n        .subject\n        .iter_organization()\n        .filter_map(|a| a.as_str().ok())\n        .map(|a| a.to_string())\n        .reduce(|cur, next| cur + &next)\n}\n\n/// Return the organization associated with the X509 certificate (as bytes).\n/// see https://en.wikipedia.org/wiki/X.509#Structure_of_a_certificate\npub fn get_organization_serial_bytes(cert: &[u8]) -> Result<(Option<String>, String)> {\n    let (_, x509cert) = x509_parser::certificate::X509Certificate::from_der(cert)\n        .expect(\"Failed to parse certificate from DER format.\");\n\n    get_organization_serial_x509(&x509cert)\n}\n\n/// Return the organization unit associated with the X509 certificate.\n/// see https://en.wikipedia.org/wiki/X.509#Structure_of_a_certificate\npub fn get_organization_unit(x509cert: &WrappedX509) -> Option<String> {\n    x509cert\n        .borrow_cert()\n        .subject\n        .iter_organizational_unit()\n        .filter_map(|a| a.as_str().ok())\n        .map(|a| a.to_string())\n        .reduce(|cur, next| cur + &next)\n}\n\n/// Get a combination of the common names for the given certificate\n/// see https://en.wikipedia.org/wiki/X.509#Structure_of_a_certificate\npub fn get_common_name(x509cert: &WrappedX509) -> Option<String> {\n    x509cert\n        .borrow_cert()\n        .subject\n        .iter_common_name()\n        .filter_map(|a| a.as_str().ok())\n        .map(|a| a.to_string())\n        .reduce(|cur, next| cur + &next)\n}\n\n/// Get the `not_after` field for the valid time period for the given cert\n/// see https://en.wikipedia.org/wiki/X.509#Structure_of_a_certificate\npub fn get_not_after(x509cert: &WrappedX509) -> String {\n    x509cert.borrow_cert().validity.not_after.to_string()\n}\n\n/// This type contains a list of one or more certificates and an associated private key. The leaf\n/// certificate should always be first.\npub struct CertKey {\n    key: Vec<u8>,\n    pem: X509Pem,\n}\n\nimpl CertKey {\n    /// Create a new `CertKey` given a list of certificates and a private key.\n    pub fn new(pem_bytes: Vec<u8>, key: Vec<u8>) -> CertKey {\n        let pem = X509Pem::new(pem_bytes);\n        assert!(\n            !pem.certs.is_empty(),\n            \"expected at least one certificate in PEM\"\n        );\n\n        CertKey { key, pem }\n    }\n\n    /// Peek at the leaf certificate.\n    pub fn leaf(&self) -> &WrappedX509 {\n        // This is safe due to the assertion in creation of a `CertKey`\n        &self.pem.certs[0]\n    }\n\n    /// Return the key.\n    pub fn key(&self) -> &Vec<u8> {\n        &self.key\n    }\n\n    /// Return a slice of intermediate certificates. An empty slice means there are none.\n    pub fn intermediates(&self) -> Vec<&WrappedX509> {\n        self.pem.certs.iter().skip(1).collect()\n    }\n\n    /// Return the organization from the leaf certificate.\n    pub fn organization(&self) -> Option<String> {\n        get_organization(self.leaf())\n    }\n\n    /// Return the serial from the leaf certificate.\n    pub fn serial(&self) -> String {\n        get_serial(self.leaf()).unwrap()\n    }\n\n    pub fn raw_pem(&self) -> &[u8] {\n        &self.pem.raw_pem\n    }\n}\n\n#[derive(Debug)]\npub struct X509Pem {\n    pub raw_pem: Vec<u8>,\n    pub certs: Vec<WrappedX509>,\n}\n\nimpl X509Pem {\n    pub fn new(raw_pem: Vec<u8>) -> Self {\n        let certs = Pem::iter_from_buffer(&raw_pem)\n            .map(|part| {\n                let raw_cert = part.expect(\"Failed to parse PEM\").contents;\n                WrappedX509::new(raw_cert, parse_x509)\n            })\n            .collect();\n        X509Pem { raw_pem, certs }\n    }\n\n    pub fn iter(&self) -> std::slice::Iter<'_, WrappedX509> {\n        self.certs.iter()\n    }\n}\n\nfn parse_x509<C>(raw_cert: &C) -> X509Certificate<'_>\nwhere\n    C: AsRef<[u8]>,\n{\n    X509Certificate::from_der(raw_cert.as_ref())\n        .expect(\"Failed to parse certificate from DER format.\")\n        .1\n}\n\n#[self_referencing]\n#[derive(Debug)]\npub struct WrappedX509 {\n    raw_cert: Vec<u8>,\n\n    #[borrows(raw_cert)]\n    #[covariant]\n    cert: X509Certificate<'this>,\n}\n\nimpl WrappedX509 {\n    pub fn not_after(&self) -> String {\n        self.borrow_cert().validity.not_after.to_string()\n    }\n}\n\n// hide private key\nimpl std::fmt::Debug for CertKey {\n    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {\n        f.debug_struct(\"CertKey\")\n            .field(\"X509\", &self.leaf())\n            .finish()\n    }\n}\n\nimpl std::fmt::Display for CertKey {\n    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {\n        let leaf = self.leaf();\n        if let Some(cn) = get_common_name(leaf) {\n            // Write CN if it exists\n            write!(f, \"CN: {cn},\")?;\n        } else if let Some(org_unit) = get_organization_unit(leaf) {\n            // CA cert might not have CN, so print its unit name instead\n            write!(f, \"Org Unit: {org_unit},\")?;\n        }\n        write!(f, \", expire: {}\", get_not_after(leaf))\n        // ignore the details of the private key\n    }\n}\n\nimpl Hash for X509Pem {\n    fn hash<H: Hasher>(&self, state: &mut H) {\n        for certificate in &self.certs {\n            if let Ok(serial) = get_serial(certificate) {\n                serial.hash(state)\n            }\n        }\n    }\n}\n\nimpl Hash for CertKey {\n    fn hash<H: Hasher>(&self, state: &mut H) {\n        self.pem.hash(state)\n    }\n}\n"
  },
  {
    "path": "pingora-core/tests/certs/alt-ca.crt",
    "content": "-----BEGIN CERTIFICATE-----\nMIIFzzCCA7egAwIBAgIUdmTkBmGw2cEQiP+uCa1TuTBp1aYwDQYJKoZIhvcNAQEL\nBQAwdzELMAkGA1UEBhMCVVMxCzAJBgNVBAgMAlNDMQ0wCwYDVQQHDARDaGFzMRAw\nDgYDVQQKDAdUaGUgT3JnMQ4wDAYDVQQLDAVBZG1pbjEPMA0GA1UEAwwGT3JnLUNB\nMRkwFwYJKoZIhvcNAQkBFgpvcmdAY2EuY29tMB4XDTI1MDgwNjA0MDc0NFoXDTM1\nMDgwNDA0MDc0NFowdzELMAkGA1UEBhMCVVMxCzAJBgNVBAgMAlNDMQ0wCwYDVQQH\nDARDaGFzMRAwDgYDVQQKDAdUaGUgT3JnMQ4wDAYDVQQLDAVBZG1pbjEPMA0GA1UE\nAwwGT3JnLUNBMRkwFwYJKoZIhvcNAQkBFgpvcmdAY2EuY29tMIICIjANBgkqhkiG\n9w0BAQEFAAOCAg8AMIICCgKCAgEAtD+U6aTiu5c9nhACCfESx13zRZ+n3dzZuRac\nsX/PrFbpAI0zsLh0ohaXLNnJuPHBSjUNhxSFZeWBuQA7mp69ZoZ8CtzjADR0EML1\nKLgvy7BHTH8Oe1MCyqRLJm8pcHh4AnBF02eN+71pfsfkXrr/hNlOnPJAbCpU7Rfz\nWq+w1oeGeC8h+RSgiY/1o28ELRs8SzkQGwlu6WezXEZuq6609c7pmevDD9t6snYx\n2A0ON/QgwhjcVyuiFRe9tKovzNkRmROkbfYgINCnYOuTxn+dWW3zmCYVHJYA+FWJ\nxwOp86rIeCIk+snL6+pL/M0s1+E2szwY+yWmw6q4NlymyGVpVsF6+vsMX/7JG9q1\nhGOFD2Nbu7Qs5OlC7+k1m+fwcTGm74dlbkNIjUguMsfyCGT0vHGZ7JfdGINvOWgP\nnOuqyTlkj4GVehlBA37S26007bqdtLrkOxrymFwOpoJuYC7HeuFIk1RX2Lfh+6xQ\nh5KVjRdzYfisuegqJPwLpA9YjRBzCCcInmnzsZAkur3/9wXHUcoNP1NVEDWHaCxi\njTRvqJazCBqjLxT9doRjTeuKj/RcHkk79IgR6Oiz2De9AwHSp4+NKngrerdissMH\nTcBpO5entMnp6r/IkysMDWSrM6JZK66g3ltxiDWyjlD+BDB/VECdyN1RquQ0ONje\n2gYzp/0CAwEAAaNTMFEwHQYDVR0OBBYEFOa/SvZE+jE5JMwxuyIuHTkTXzCLMB8G\nA1UdIwQYMBaAFOa/SvZE+jE5JMwxuyIuHTkTXzCLMA8GA1UdEwEB/wQFMAMBAf8w\nDQYJKoZIhvcNAQELBQADggIBAHfAOu4dnsYomHFG+jAurG3LG/TPdxiv0g96oJ6h\nFwhcbKWcFXWAQckzMloVAymatIqUsfXFlMPhadD51AG6BcgmD7i3co/Gh0o3XUCG\nGL6cyacsXPAIuCbqUp4Wgs10BR3ELNrf3ksTQGU58g20KodSmHr9ttSeDpK7x2tA\nNS/fy/T1lFsRiLQi7bYb3KljnSTllJ3TIexd2gHaS7Varr/N6F9DJT1qMs8xCIlz\ngTpJcC/X2fSWaa9IPg9WCzycJjSMoVAw8xfh6Lu9YweKNW7/2eGmZlhpm5MGFV3+\nG9UPBayPz7sFoVJaKwr4tE027r+sJddLa4S4GJMW+N5+eVmoQuzZ4C3W/R7LPHTz\nKq+JevifKqUJ5EmuSGAtHa1LfFa2T636lfxfZu4k4JT4H7/58RKJ1X5DYNd3IdGG\nb4mg1dmX7JwP2msV92x2ywfIzoFK7ByKHvAOq8inI099y0IMtZGElkv6RMD2OpMl\ntWFjIn++VGXNeHdwcWxAzx29q+jqBpzgH9KVP2io8M1X5j+TNh5guTzKLpH0bbPo\nWvr2szsV9+jUj+z49Aik8OjjMsMyfSVRcMU54ZvmkP0VCSkejWf2ermoX9XGL0Q5\nKqyzbLSKFbtVtAnqFjqYdwYP+0729xIFO54IETBDjhk66s9irBibanuatfG5ezXT\n33ZP\n-----END CERTIFICATE-----\n"
  },
  {
    "path": "pingora-core/tests/certs/alt-server.crt",
    "content": "-----BEGIN CERTIFICATE-----\nMIIEsTCCApmgAwIBAgIUcEJNjZRw1qumRzsbm9HSdXyCojkwDQYJKoZIhvcNAQEL\nBQAwdzELMAkGA1UEBhMCVVMxCzAJBgNVBAgMAlNDMQ0wCwYDVQQHDARDaGFzMRAw\nDgYDVQQKDAdUaGUgT3JnMQ4wDAYDVQQLDAVBZG1pbjEPMA0GA1UEAwwGT3JnLUNB\nMRkwFwYJKoZIhvcNAQkBFgpvcmdAY2EuY29tMB4XDTI1MDgwNjA0MTMwMloXDTI2\nMDgwNjA0MTMwMlowajELMAkGA1UEBhMCVVMxCzAJBgNVBAgMAlNDMQwwCgYDVQQH\nDANDSFMxDDAKBgNVBAoMA09yZzEMMAoGA1UECwwDb3JnMQwwCgYDVQQDDANvcmcx\nFjAUBgkqhkiG9w0BCQEWB29yZ0BvcmcwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAw\nggEKAoIBAQDhAN58oAxwa0C34q72BgyFZKa6NlvIZrr+0DaCpILUYAifP5v4RCAI\ncf4MAspcQWizzz14aBaV7s4MRBlPv8b6C0m1Acld+C1x3KwjWRfQyIb9LTOQBx+q\nb3SkqVvTXeJXqd38cIBXQ4zirHmJ1+m2CULYUoIeZZfeYU/+gv84Q+sRpQce/+wu\n0MvsXw3t+3DJNRsC+EkDiZyiBdZb2ochriVMT6IvQVlCB36nQUcPAtBPs5UcfVn0\nRenaBdXgzE4CYvyg1qcwhw5mve1gHtsFMuqZTArUlSlUz6YJQKCFkO83IISwUwx7\nwKSswN38WIKKsYLti0xReuh7aS8o0kBjAgMBAAGjQjBAMB0GA1UdDgQWBBSC3ZGf\nYULa784DsMjCBbSInuHogDAfBgNVHSMEGDAWgBTmv0r2RPoxOSTMMbsiLh05E18w\nizANBgkqhkiG9w0BAQsFAAOCAgEAd6OptRoJSZXcTtYinF0LhaNCn5+kKsW0I487\nJoPPIgQLXKOT8PkkGYY+p8nucm4zYyp+2Z6CayxfrWORDylteT1cpVgwFw39KqU2\nFuGa4nAOV9BTIBer4oUVTS2flkkMnHlDIUKE+yONE5wOyRa/jvQWfMl9bfN0yRAh\ne/72YEszBADrQlpWUvyu6Uv3cNi2XPbcty3VSNHkPWs1lHlwY9s2csnSLQpFMN+A\nWq1RADLKWfR9mrDEmzx0V5JIOqY2K0804jpbnD/fkjyIFBmRIUETN+MU8PQdp0W6\n8cBh7u9L5UoUweRr/cZFOd0jHJLiCpClXyFOHXsNkT7jN/hcbXRPwSTD6+GY3Opz\nDn3lZlaAbLg+NtHVDageTX2QJ6H/HVpVGxDltM3hiMrub7PTHCG6GyyWKII3wcXd\n875+EqHMwdRYnHb9jtA20GDeG+NGQ5IUJvMPHivXFWbidV5YXyR1t1UP6HEHRU0D\n3i4xYJXJTlA3gCUpfsOLm/0lXg+cLqwaqtXZ3vFViUHE02CE5PZN4QZ1ggAFldLx\nHEbjzWdDMR0Qy4g/DsfRss9ve0V/te3F6EjXLf5Cra+5wAWm0xUqLiky5HtND5+q\nAc9j//3tcahDgigl2xHakTA/G4xcUM6IV5SxoRUeYQqJ8mHorQyoEkgED6LUyLYa\nNq0MAHA=\n-----END CERTIFICATE-----\n"
  },
  {
    "path": "pingora-core/tests/certs/ca.crt",
    "content": "-----BEGIN CERTIFICATE-----\nMIIFazCCA1OgAwIBAgIUcvULnZbENoxhjvv5TARdWr24pXcwDQYJKoZIhvcNAQEL\nBQAwRTELMAkGA1UEBhMCQVUxEzARBgNVBAgMClNvbWUtU3RhdGUxITAfBgNVBAoM\nGEludGVybmV0IFdpZGdpdHMgUHR5IEx0ZDAeFw0yNTA4MDQxNzIyMjVaFw0zNTA4\nMDIxNzIyMjVaMEUxCzAJBgNVBAYTAkFVMRMwEQYDVQQIDApTb21lLVN0YXRlMSEw\nHwYDVQQKDBhJbnRlcm5ldCBXaWRnaXRzIFB0eSBMdGQwggIiMA0GCSqGSIb3DQEB\nAQUAA4ICDwAwggIKAoICAQC9wyQZCE4DA4fpuufOofpImk4L2y2ubW9SmkzwHypD\nKtcQ1yx9bIdHIG9RWoSvDYB+d3mS8uiolkq631anrp2gtiuRchiMJc3kRkzALT9O\nHQfebe4lU3V9EanikEX3xhQBzfeqmw6LUWtwRMqR+MvRTOMy1bM61jvTXYY5cBVz\nQrz8/FLSXflamkd651mjCVhrXginLqmYUWuVjftn9R3wml4yoUKc7UIyM1j4dB50\nnVrTgILMvfukreDfMu4l48rf8g0+usvT8ROmTum1J3kkCVqGUqChwC2I8kPN8BPQ\nCJQgfJFPPplzfoyyLWHBFHXTWNP3ec9m3GCQfyCgzI1kEBjVYr6C3Ukb214N/1dr\nP7sh3OHW64pxGy72/RWg5CsDFPs7t3RijJB5282LpxLzbFDviIFp6C1Kg/XRv88M\nL5NhXfdqx0MC8Glj9AjGIrmLvB2d+PqKfpHe03eC0RgYGO59K4dwKUEs+P2uYcKm\n0yFoOsaQyCpNAbBc6B1rawmJRHCVt8Bi/CLAsWl5N3Dq7TOxDxU1gTcvZxRC40/x\naS89PvMmk4lUP+ueKvsQ3Qwx3+um/Cf7dvlWk3XxljUQTCqKS70XdkOlq82Xyugt\n6NyztBQwd56Ms1qoyy3Jx5f2Vvvdp8e0LtxhUliRiuKemPb++uec7uZM6BVZL5YL\nyQIDAQABo1MwUTAdBgNVHQ4EFgQUiTft2tOcFNM8OuQQdcfCbfqU1qIwHwYDVR0j\nBBgwFoAUiTft2tOcFNM8OuQQdcfCbfqU1qIwDwYDVR0TAQH/BAUwAwEB/zANBgkq\nhkiG9w0BAQsFAAOCAgEAI9IA+2QYhZGHNZFiCUClidbinotg51q/+SXz4TOUgxQx\ntrB+f/247pNUJmBZ++16lfTakh6CCzItgzXWgsI64fzJRQBfYuAJrAb2ApMMk2+l\n48wISakDIoXnntokOtQslaoFr5jSG0C3J9CrmHcMk0Z0NTYr0ltMWbkEeFhv1KZH\nX4CESm+5D706cXkEzyN4sXFDf68OyJQehlxKCJuvZuO0+DWaFzsKd1wRmlybk8n/\n+b63gKwT9ydKdW4ZgidCXwh+Y5trqVeqzWBmxHX1207WjvaHggCI7si1bajiwgNu\nCnhfUlueawIKZBiliZgpHYuYkDyiB9NdE3twcoIAhKGcpTTd9hP4i4majq/M+hzi\niox8fHg4HoI7l/cp0LiWYZjoIoaH10Nohn0BLqSDIrigxdrhbF2FNgjWyxjL9HMg\nkoTaZ7ji2J2ygqINyEBdunwHhC5SUPsLL5sWEHK19PDerJoU0xcbDR3s+KuMXCIG\nf9kVZt54DECl9TL70MRzRHnoNkMriLgZiiAMqeTlUOPV3GbWc2G+YEFv3xFhYqY0\nZyW5CvMSYaNpEoe1xz9QNkK6i6dwR29Da8QAYhJPOJaFXDwIsh/telsoMeJyO7Ql\nHaMEMOZIu9SaxWYDBwpnZ959VSj/APSSv2d6dIaRE3XmxXt6xK2FGw50HimCsHU=\n-----END CERTIFICATE-----\n"
  },
  {
    "path": "pingora-core/tests/certs/server.crt",
    "content": "-----BEGIN CERTIFICATE-----\nMIIEWjCCAkKgAwIBAgIUcEJNjZRw1qumRzsbm9HSdXyCojcwDQYJKoZIhvcNAQEL\nBQAwRTELMAkGA1UEBhMCQVUxEzARBgNVBAgMClNvbWUtU3RhdGUxITAfBgNVBAoM\nGEludGVybmV0IFdpZGdpdHMgUHR5IEx0ZDAeFw0yNTA4MDYwMzQ2MDJaFw0yNjA4\nMDYwMzQ2MDJaMEUxCzAJBgNVBAYTAkFVMRMwEQYDVQQIDApTb21lLVN0YXRlMSEw\nHwYDVQQKDBhJbnRlcm5ldCBXaWRnaXRzIFB0eSBMdGQwggEiMA0GCSqGSIb3DQEB\nAQUAA4IBDwAwggEKAoIBAQCyUoXpICzkly7iNIJYRi310ZbDufM+HuGEhh0TR/j0\nTbzunarlSJiDBvJ8QgYMc+SeNdIfE2eEd0yV9/Ukhm8eLTr5ti1lWQKZH85JVnxr\nV9fo3eE9OfxpEDgdlt/wkHfmYr/th8ajwMzoUkC5uY+POex0zGBt2eEKr4V9KNI4\n3JQKshpWYRaOvfufNZXN5+yivzkON5zE7zPr+BLZdiG4nDhnpkcrvoo6sWnu2Pcv\nbbtXJIa41iZWrt0zSEBE5q5Ci3ZFHow5f+AQuEi2E4XsDQv3emFjZ4jvT/deKWgn\nund4JGD0PaxCj7mE1DBqmqHDDyxOyHA/L+KzGTR/Yws9AgMBAAGjQjBAMB0GA1Ud\nDgQWBBSZ/Fkj1buTR2uj1nw+mWJhrDQvSzAfBgNVHSMEGDAWgBSJN+3a05wU0zw6\n5BB1x8Jt+pTWojANBgkqhkiG9w0BAQsFAAOCAgEANqctLxVkBvIJ07W5gzLMKDa2\nNPSeN++07A0Abvi4ImXh+yeDdI59uDlbkIG+g1C2zOsyO837dYlAryX1NQ/sfGTt\nDz3Yu1pgk3eFS/Bz7dOFbADWkKX6NdAY5nC5MjQx1hIAnEz0LNp0xyW7WVASoUcO\nWJlM0CWBBmFLMWp9FWLD4xYY+hCl4VSu1+I1vktn5vuUfhCX+0e4L/tV1+FNVu9I\nodyXmQizreEBXTZvHKYdHCBGnzY7BS/RjA2j/xDT6XH0QU1LA0tu8sAiojcNplcT\nHpKS/hLa8OjSPitgHPm0Ce7mzjqTF9H0IZLx78HfeGe4kbitp3iGVcGh9r4AMBX/\nHBpQxSEiMDBpPdHLvT9r1+NwScwIOxjberJ1TA3NhaMTFXXji0zpP4hhUt9pG/lQ\nFM1HoXM2f53g+m0rHiF1zUPUhpqfd40ktmA+DyPpgxalTNYNjTT3VH9nXYoSVD+6\nLW8sVbAMDL9OKZLrXazTYGC1DGAGlFK0e8gjP1zI9sTw21tZlkNvs1syp+Kxq+uv\nOOEF0iUXVEMdGpSSfT+P0htYlZvQdrc33m3+ROYuZgS8DKUqgWn1QuKz8G1DcnuG\nNp+etROWMf0mKDsOvuVUy9OJuOqWo/rOE3m9sZKkzGdA3sQXdHFtGMp3vqeArUlF\nl62Qf0ApIWPdmX8n21U=\n-----END CERTIFICATE-----\n"
  },
  {
    "path": "pingora-core/tests/certs/server.key",
    "content": "-----BEGIN PRIVATE KEY-----\nMIIEvAIBADANBgkqhkiG9w0BAQEFAASCBKYwggSiAgEAAoIBAQCyUoXpICzkly7i\nNIJYRi310ZbDufM+HuGEhh0TR/j0TbzunarlSJiDBvJ8QgYMc+SeNdIfE2eEd0yV\n9/Ukhm8eLTr5ti1lWQKZH85JVnxrV9fo3eE9OfxpEDgdlt/wkHfmYr/th8ajwMzo\nUkC5uY+POex0zGBt2eEKr4V9KNI43JQKshpWYRaOvfufNZXN5+yivzkON5zE7zPr\n+BLZdiG4nDhnpkcrvoo6sWnu2PcvbbtXJIa41iZWrt0zSEBE5q5Ci3ZFHow5f+AQ\nuEi2E4XsDQv3emFjZ4jvT/deKWgnund4JGD0PaxCj7mE1DBqmqHDDyxOyHA/L+Kz\nGTR/Yws9AgMBAAECggEADoo/jIF435+7LSsiaK+6PiW7kRSHsqxCb7Oeycxvzn3L\nNrqo7V6kvuRRX9PjWd8WSFcznaCPq4OtvTm1ZafHhjKicSuLWozuMw2enKi+ZuNI\nAd8bp0oj3G474R/E/UDOYfzx0NymFAKbwqK4T9yDSearct+aSkK+gIhMzmaGc6fX\ntmqH0MxsLXmpkdMFL5WNU3IvrJvdYAtSJ+Tqq/K8ifkCxUzVxSav1Msd0wXxyWJU\nhyU8WzFRzotAguqGu5VQOKUhjzeOC5uoJWcor0OSf8CvOUIGR0orINLf1BaHCGY8\n3cIcY7WqTOeVR/q3IxGlDO+0aoMvQSR/BzIw42hmwQKBgQDqDWhnb7lbkaV3uX2l\nperMozoa/ycmdvRwAZJqBdHoV1w1nnxjTU/IaVXXL9Fj8gRyJ3YXImfKTFE+Z5cU\n7RLAPc+7DKLEMEWBwIzbbFZ+ywEAt1xNqRlLsEJd1UCXsmtF67zeYg6Msd4ckaHe\nDIv5qryl/DqUT4t7emX2iJGuwQKBgQDDC0binK0e/HsLbN7HG3xbV6GdsP+HKPlz\nRE1R4g3DO4uYd24hJFF2kdkngD9PuVbxIqmDb7C40V2/19KTsKToMBiBGv3b1mne\nEJVLwdn5hSOE7G/gJDSQuUAV+Tujg3b3lPTsOaygLbyWNEQBcJh0KRnxOi1Kefvl\naOhRfkV3fQKBgBFTCr5VS8AWaMwS49UGEfoxvtROvKQhO/iqdR757U6oYL/rSkPD\nbjtkaKEz/ejK+j9E4n3V4x7bRUw8OLeo0LGAIcczqTyiYhK3oPWA8GoUNq/J4sAw\n2xl6I390kIJqB3y2dVV0pqUNaWZt9TBNd3L0i2Ax6lgeBzINnkyAUWBBAoGAEf/Y\natFKqLFkKYnChV1j/In5wDO1YSPG4XxMJmJWIs4787YR070mR2ruP1b2gMT54Qbx\n3c9Q371yiWHBbR/AGC1YFZIIG2GOI5AkNvmMxBolTP8E1AqDT1fJMj3t4wke0XpN\nn/8yjxWpcbMhE4DwkMe6PSjBRT48oM8toVel0YECgYB6CZrjB6R8C2N6MMJ1HNoF\ni1q5VvxaPZ6s7pf0jKh7NTjqh+FGcBOn5ixA3PZAomphHwZDzuwROeaMVzjEJMA/\nLX9Bq1beElyTYVINhL9C646D+DJtRa8fElgQBNuaSJJOaP9a2rh4PI9xgn4WLYrA\n0xxdIR87iqrUFI11Nm2Dog==\n-----END PRIVATE KEY-----\n"
  },
  {
    "path": "pingora-core/tests/keys/key.pem",
    "content": "-----BEGIN EC PRIVATE KEY-----\nMHcCAQEEIN5lAOvtlKwtc/LR8/U77dohJmZS30OuezU9gL6vmm6DoAoGCCqGSM49\nAwEHoUQDQgAE2f/1Fm1HjySdokPq2T0F1xxol9nSEYQ+foFINeaWYk+FxMGpriJT\nBb8AGka87cWklw1ZqytfaT6pkureDbTkwg==\n-----END EC PRIVATE KEY-----\n"
  },
  {
    "path": "pingora-core/tests/keys/public.pem",
    "content": "-----BEGIN PUBLIC KEY-----\nMFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE2f/1Fm1HjySdokPq2T0F1xxol9nS\nEYQ+foFINeaWYk+FxMGpriJTBb8AGka87cWklw1ZqytfaT6pkureDbTkwg==\n-----END PUBLIC KEY-----\n"
  },
  {
    "path": "pingora-core/tests/keys/server.crt",
    "content": "-----BEGIN CERTIFICATE-----\nMIIB9zCCAZ2gAwIBAgIUMI7aLvTxyRFCHhw57hGt4U6yupcwCgYIKoZIzj0EAwIw\nZDELMAkGA1UEBhMCVVMxCzAJBgNVBAgMAkNBMRYwFAYDVQQHDA1TYW4gRnJhbmNp\nc2NvMRgwFgYDVQQKDA9DbG91ZGZsYXJlLCBJbmMxFjAUBgNVBAMMDW9wZW5ydXN0\neS5vcmcwHhcNMjIwNDExMjExMzEzWhcNMzIwNDA4MjExMzEzWjBkMQswCQYDVQQG\nEwJVUzELMAkGA1UECAwCQ0ExFjAUBgNVBAcMDVNhbiBGcmFuY2lzY28xGDAWBgNV\nBAoMD0Nsb3VkZmxhcmUsIEluYzEWMBQGA1UEAwwNb3BlbnJ1c3R5Lm9yZzBZMBMG\nByqGSM49AgEGCCqGSM49AwEHA0IABNn/9RZtR48knaJD6tk9BdccaJfZ0hGEPn6B\nSDXmlmJPhcTBqa4iUwW/ABpGvO3FpJcNWasrX2k+qZLq3g205MKjLTArMCkGA1Ud\nEQQiMCCCDyoub3BlbnJ1c3R5Lm9yZ4INb3BlbnJ1c3R5Lm9yZzAKBggqhkjOPQQD\nAgNIADBFAiAjISZ9aEKmobKGlT76idO740J6jPaX/hOrm41MLeg69AIhAJqKrSyz\nwD/AAF5fR6tXmBqlnpQOmtxfdy13wDr4MT3h\n-----END CERTIFICATE-----\n"
  },
  {
    "path": "pingora-core/tests/keys/server.csr",
    "content": "-----BEGIN CERTIFICATE REQUEST-----\nMIIBJzCBzgIBADBsMQswCQYDVQQGEwJVUzETMBEGA1UECAwKQ2FsaWZvcm5pYTEW\nMBQGA1UEBwwNU2FuIEZyYW5jaXNjbzEYMBYGA1UECgwPQ2xvdWRmbGFyZSwgSW5j\nMRYwFAYDVQQDDA1vcGVucnVzdHkub3JnMFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcD\nQgAE2f/1Fm1HjySdokPq2T0F1xxol9nSEYQ+foFINeaWYk+FxMGpriJTBb8AGka8\n7cWklw1ZqytfaT6pkureDbTkwqAAMAoGCCqGSM49BAMCA0gAMEUCIFyDN8eamnoY\nXydKn2oI7qImigxahyCftzjxkIEV5IKbAiEAo5l72X4U+YTVYmyPPnJIj2v5nA1R\nRuUfMh5sXzwlwuM=\n-----END CERTIFICATE REQUEST-----\n"
  },
  {
    "path": "pingora-core/tests/nginx.conf",
    "content": "\n#user  nobody;\nworker_processes  1;\n\nerror_log  /dev/stdout;\n#error_log  logs/error.log  notice;\n#error_log  logs/error.log  info;\n\npid        logs/nginx.pid;\nmaster_process off;\ndaemon off;\n\nevents {\n    worker_connections  4096;\n}\n\n\nhttp {\n    #include       mime.types;\n    #default_type  application/octet-stream;\n\n    #log_format  main  '$remote_addr - $remote_user [$time_local] \"$request\" '\n    #                  '$status $body_bytes_sent \"$http_referer\" '\n    #                  '\"$http_user_agent\" \"$http_x_forwarded_for\"';\n\n    # access_log  logs/access.log  main;\n    access_log  off;\n\n    sendfile        on;\n    #tcp_nopush     on;\n\n    #keepalive_timeout  0;\n    keepalive_timeout  10;\n    keepalive_requests 99999;\n\n    #gzip  on;\n\n    server {\n        listen       8000;\n        listen       [::]:8000;\n        listen       8443 ssl http2;\n        #listen       8443 ssl http2;\n        server_name  localhost;\n\n        ssl_certificate keys/server.crt;\n        ssl_certificate_key keys/key.pem;\n        ssl_protocols TLSv1.2;\n        ssl_ciphers TLS-AES-128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256;\n\n        #charset koi8-r;\n\n        #access_log  logs/host.access.log  main;\n\n        location / {\n            root   /home/yuchen/nfs/tmp;\n            index  index.html index.htm;\n        }\n        location /test {\n            keepalive_timeout 20;\n            return 200;\n        }\n        location /test2 {\n            keepalive_timeout 0;\n            return 200 \"hello world\";\n        }\n        location /test3 {\n            keepalive_timeout 0;\n            return 200;\n            #content_by_lua_block {\n            #    ngx.print(\"hello world\")\n            #}\n        }\n\n        location /test4 {\n            keepalive_timeout 20;\n            rewrite_by_lua_block {\n                ngx.exit(200)\n            }\n            #return 201;\n\n        }\n\n        #error_page  404              /404.html;\n\n        # redirect server error pages to the static page /50x.html\n        #\n        error_page   500 502 503 504  /50x.html;\n        location = /50x.html {\n            root   html;\n        }\n    }\n}\n"
  },
  {
    "path": "pingora-core/tests/nginx_proxy.conf",
    "content": "\n#user  nobody;\nworker_processes 1;\n\nerror_log  /dev/stdout;\n#error_log  logs/error.log  notice;\n#error_log  logs/error.log  info;\n\n#pid        logs/nginx.pid;\nmaster_process off;\ndaemon off;\n\nevents {\n    worker_connections  4096;\n}\n\n\nhttp {\n    #include       mime.types;\n    #default_type  application/octet-stream;\n\n    #log_format  main  '$remote_addr - $remote_user [$time_local] \"$request\" '\n    #                  '$status $body_bytes_sent \"$http_referer\" '\n    #                  '\"$http_user_agent\" \"$http_x_forwarded_for\"';\n\n    # access_log  logs/access.log  main;\n    access_log  off;\n\n    sendfile        on;\n    #tcp_nopush     on;\n\n    keepalive_timeout  30;\n    keepalive_requests 99999;\n\n    upstream plaintext {\n        server 127.0.0.1:8000;\n        keepalive 128;\n        keepalive_requests 99999;\n    }\n\n    upstream ssl {\n        server 127.0.0.1:8443;\n        keepalive 128;\n        keepalive_requests 99999;\n    }\n\n    #gzip  on;\n\n    server {\n        listen       8001;\n        listen       [::]:8001;\n        server_name  localproxy;\n\n        location / {\n            keepalive_timeout 30;\n            proxy_pass http://plaintext;\n            proxy_http_version 1.1;\n            proxy_set_header Connection \"Keep-Alive\";\n        }\n\n    }\n\n    server {\n        listen       8002 ssl;\n        listen       [::]:8002 ssl;\n        server_name  localproxy_https;\n\n        ssl_certificate keys/server.crt;\n        ssl_certificate_key keys/key.pem;\n        ssl_protocols TLSv1.2;\n        ssl_ciphers TLS-AES-128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256;\n\n        location / {\n            keepalive_timeout 30;\n            proxy_pass https://ssl;\n            proxy_http_version 1.1;\n            proxy_ssl_session_reuse off;\n            proxy_ssl_verify on;\n            proxy_ssl_server_name on;\n            proxy_ssl_name \"openrusty.org\";\n            proxy_ssl_trusted_certificate keys/server.crt;\n            proxy_set_header Connection \"Keep-Alive\";\n        }\n\n    }\n}\n"
  },
  {
    "path": "pingora-core/tests/pingora_conf.yaml",
    "content": "---\nversion: 1\nclient_bind_to_ipv4:\n    - 127.0.0.2\nca_file: tests/keys/server.crt"
  },
  {
    "path": "pingora-core/tests/server_phase_fastshutdown.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n// NOTE: This test sends a shutdown signal to itself,\n// so it needs to be in an isolated test to prevent concurrency.\n\nuse pingora_core::server::{ExecutionPhase, RunArgs, Server};\n\n// Ensure that execution phases are reported correctly.\n#[test]\nfn test_server_execution_phase_monitor_fast_shutdown() {\n    let mut server = Server::new(None).unwrap();\n\n    let mut phase = server.watch_execution_phase();\n\n    let join = std::thread::spawn(move || {\n        server.bootstrap();\n        server.run(RunArgs::default());\n    });\n\n    assert!(matches!(\n        phase.blocking_recv().unwrap(),\n        ExecutionPhase::Bootstrap\n    ));\n    assert!(matches!(\n        phase.blocking_recv().unwrap(),\n        ExecutionPhase::BootstrapComplete,\n    ));\n    assert!(matches!(\n        phase.blocking_recv().unwrap(),\n        ExecutionPhase::Running,\n    ));\n\n    // Need to wait for startup, otherwise the signal handler is not\n    // installed yet.\n    //\n    // TODO: signal handlers are installed after Running phase\n    // message is sent, sleep for now to avoid test flake\n    std::thread::sleep(std::time::Duration::from_millis(500));\n\n    unsafe {\n        libc::raise(libc::SIGINT);\n    }\n\n    assert!(matches!(\n        phase.blocking_recv().unwrap(),\n        ExecutionPhase::ShutdownStarted,\n    ));\n\n    assert!(matches!(\n        phase.blocking_recv().unwrap(),\n        ExecutionPhase::ShutdownRuntimes,\n    ));\n\n    join.join().unwrap();\n\n    assert!(matches!(\n        phase.blocking_recv().unwrap(),\n        ExecutionPhase::Terminated,\n    ));\n}\n"
  },
  {
    "path": "pingora-core/tests/server_phase_gracefulshutdown.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n// NOTE: This test sends a shutdown signal to itself,\n// so it needs to be in an isolated test to prevent concurrency.\n\nuse pingora_core::server::{configuration::ServerConf, ExecutionPhase, RunArgs, Server};\n\n// Ensure that execution phases are reported correctly.\n#[test]\nfn test_server_execution_phase_monitor_graceful_shutdown() {\n    let conf = ServerConf {\n        // Use small timeouts to speed up the test.\n        grace_period_seconds: Some(1),\n        graceful_shutdown_timeout_seconds: Some(1),\n        ..Default::default()\n    };\n    let mut server = Server::new_with_opt_and_conf(None, conf);\n\n    let mut phase = server.watch_execution_phase();\n\n    let join = std::thread::spawn(move || {\n        server.bootstrap();\n        server.run(RunArgs::default());\n    });\n\n    assert!(matches!(\n        phase.blocking_recv().unwrap(),\n        ExecutionPhase::Bootstrap\n    ));\n    assert!(matches!(\n        phase.blocking_recv().unwrap(),\n        ExecutionPhase::BootstrapComplete,\n    ));\n    assert!(matches!(\n        phase.blocking_recv().unwrap(),\n        ExecutionPhase::Running,\n    ));\n\n    // Need to wait for startup, otherwise the signal handler is not\n    // installed yet.\n    //\n    // TODO: signal handlers are installed after Running phase\n    // message is sent, sleep for now to avoid test flake\n    std::thread::sleep(std::time::Duration::from_millis(500));\n\n    unsafe {\n        libc::raise(libc::SIGTERM);\n    }\n\n    assert!(matches!(\n        phase.blocking_recv().unwrap(),\n        ExecutionPhase::GracefulTerminate,\n    ));\n\n    assert!(matches!(\n        phase.blocking_recv().unwrap(),\n        ExecutionPhase::ShutdownStarted,\n    ));\n\n    assert!(matches!(\n        phase.blocking_recv().unwrap(),\n        ExecutionPhase::ShutdownGracePeriod,\n    ));\n\n    assert!(matches!(\n        dbg!(phase.blocking_recv().unwrap()),\n        ExecutionPhase::ShutdownRuntimes,\n    ));\n\n    join.join().unwrap();\n\n    assert!(matches!(\n        phase.blocking_recv().unwrap(),\n        ExecutionPhase::Terminated,\n    ));\n}\n"
  },
  {
    "path": "pingora-core/tests/test_basic.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\nmod utils;\n\n#[cfg(all(unix, feature = \"any_tls\"))]\nuse hyperlocal::{UnixClientExt, Uri};\n\n#[tokio::test]\nasync fn test_http() {\n    utils::init();\n    let res = reqwest::get(\"http://127.0.0.1:6145\").await.unwrap();\n    assert_eq!(res.status(), reqwest::StatusCode::OK);\n}\n\n#[cfg(feature = \"any_tls\")]\n#[tokio::test]\nasync fn test_https_http2() {\n    utils::init();\n\n    let client = reqwest::Client::builder()\n        .danger_accept_invalid_certs(true)\n        .build()\n        .unwrap();\n\n    let res = client.get(\"https://127.0.0.1:6146\").send().await.unwrap();\n    assert_eq!(res.status(), reqwest::StatusCode::OK);\n    assert_eq!(res.version(), reqwest::Version::HTTP_2);\n\n    let client = reqwest::Client::builder()\n        .danger_accept_invalid_certs(true)\n        .http1_only()\n        .build()\n        .unwrap();\n\n    let res = client.get(\"https://127.0.0.1:6146\").send().await.unwrap();\n    assert_eq!(res.status(), reqwest::StatusCode::OK);\n    assert_eq!(res.version(), reqwest::Version::HTTP_11);\n}\n\n#[cfg(unix)]\n#[cfg(feature = \"any_tls\")]\n#[tokio::test]\nasync fn test_uds() {\n    utils::init();\n    let url = Uri::new(\"/tmp/echo.sock\", \"/\").into();\n    let client = hyper::Client::unix();\n\n    let res = client.get(url).await.unwrap();\n    assert_eq!(res.status(), reqwest::StatusCode::OK);\n}\n"
  },
  {
    "path": "pingora-core/tests/utils/mod.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\nuse once_cell::sync::Lazy;\nuse std::{thread, time};\n\nuse clap::Parser;\nuse pingora_core::listeners::Listeners;\nuse pingora_core::server::configuration::Opt;\nuse pingora_core::server::Server;\nuse pingora_core::services::listening::Service;\n\nuse async_trait::async_trait;\nuse bytes::Bytes;\nuse http::{Response, StatusCode};\nuse pingora_timeout::timeout;\nuse std::time::Duration;\n\nuse pingora_core::apps::http_app::ServeHttp;\nuse pingora_core::protocols::http::ServerSession;\n\n#[derive(Clone)]\npub struct EchoApp;\n\n#[async_trait]\nimpl ServeHttp for EchoApp {\n    async fn response(&self, http_stream: &mut ServerSession) -> Response<Vec<u8>> {\n        // read timeout of 2s\n        let read_timeout = 2000;\n        let body = match timeout(\n            Duration::from_millis(read_timeout),\n            http_stream.read_request_body(),\n        )\n        .await\n        {\n            Ok(res) => match res.unwrap() {\n                Some(bytes) => bytes,\n                None => Bytes::from(\"no body!\"),\n            },\n            Err(_) => {\n                panic!(\"Timed out after {:?}ms\", read_timeout);\n            }\n        };\n\n        Response::builder()\n            .status(StatusCode::OK)\n            .header(http::header::CONTENT_TYPE, \"text/html\")\n            .header(http::header::CONTENT_LENGTH, body.len())\n            .body(body.to_vec())\n            .unwrap()\n    }\n}\n\npub struct MyServer {\n    // Maybe useful in the future\n    #[allow(dead_code)]\n    pub handle: thread::JoinHandle<()>,\n}\n\nfn entry_point(opt: Option<Opt>) {\n    env_logger::init();\n\n    let cert_path = format!(\"{}/tests/keys/server.crt\", env!(\"CARGO_MANIFEST_DIR\"));\n    let key_path = format!(\"{}/tests/keys/key.pem\", env!(\"CARGO_MANIFEST_DIR\"));\n\n    let mut my_server = Server::new(opt).unwrap();\n    my_server.bootstrap();\n\n    let mut listeners = Listeners::tcp(\"0.0.0.0:6145\");\n    #[cfg(unix)]\n    listeners.add_uds(\"/tmp/echo.sock\", None);\n\n    let mut tls_settings =\n        pingora_core::listeners::tls::TlsSettings::intermediate(&cert_path, &key_path).unwrap();\n    tls_settings.enable_h2();\n    listeners.add_tls_with_settings(\"0.0.0.0:6146\", None, tls_settings);\n\n    let echo_service_http =\n        Service::with_listeners(\"Echo Service HTTP\".to_string(), listeners, EchoApp);\n\n    my_server.add_service(echo_service_http);\n    my_server.run_forever();\n}\n\nimpl MyServer {\n    pub fn start() -> Self {\n        let opts: Vec<String> = vec![\n            \"pingora\".into(),\n            \"-c\".into(),\n            \"tests/pingora_conf.yaml\".into(),\n        ];\n        let server_handle = thread::spawn(|| {\n            entry_point(Some(Opt::parse_from(opts)));\n        });\n        // wait until the server is up\n        thread::sleep(time::Duration::from_secs(2));\n        MyServer {\n            handle: server_handle,\n        }\n    }\n}\n\npub static TEST_SERVER: Lazy<MyServer> = Lazy::new(MyServer::start);\n\npub fn init() {\n    let _ = *TEST_SERVER;\n}\n"
  },
  {
    "path": "pingora-error/Cargo.toml",
    "content": "[package]\nname = \"pingora-error\"\nversion = \"0.8.0\"\nauthors = [\"Yuchen Wu <yuchen@cloudflare.com>\"]\nlicense = \"Apache-2.0\"\nedition = \"2021\"\nrepository = \"https://github.com/cloudflare/pingora\"\ncategories = [\"rust-patterns\"]\nkeywords = [\"error\", \"error-handling\", \"pingora\"]\ndescription = \"\"\"\nError types and error handling APIs for Pingora.\n\"\"\"\n\n# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html\n[lib]\nname = \"pingora_error\"\npath = \"src/lib.rs\"\n"
  },
  {
    "path": "pingora-error/LICENSE",
    "content": "\n                                 Apache License\n                           Version 2.0, January 2004\n                        http://www.apache.org/licenses/\n\n   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION\n\n   1. Definitions.\n\n      \"License\" shall mean the terms and conditions for use, reproduction,\n      and distribution as defined by Sections 1 through 9 of this document.\n\n      \"Licensor\" shall mean the copyright owner or entity authorized by\n      the copyright owner that is granting the License.\n\n      \"Legal Entity\" shall mean the union of the acting entity and all\n      other entities that control, are controlled by, or are under common\n      control with that entity. For the purposes of this definition,\n      \"control\" means (i) the power, direct or indirect, to cause the\n      direction or management of such entity, whether by contract or\n      otherwise, or (ii) ownership of fifty percent (50%) or more of the\n      outstanding shares, or (iii) beneficial ownership of such entity.\n\n      \"You\" (or \"Your\") shall mean an individual or Legal Entity\n      exercising permissions granted by this License.\n\n      \"Source\" form shall mean the preferred form for making modifications,\n      including but not limited to software source code, documentation\n      source, and configuration files.\n\n      \"Object\" form shall mean any form resulting from mechanical\n      transformation or translation of a Source form, including but\n      not limited to compiled object code, generated documentation,\n      and conversions to other media types.\n\n      \"Work\" shall mean the work of authorship, whether in Source or\n      Object form, made available under the License, as indicated by a\n      copyright notice that is included in or attached to the work\n      (an example is provided in the Appendix below).\n\n      \"Derivative Works\" shall mean any work, whether in Source or Object\n      form, that is based on (or derived from) the Work and for which the\n      editorial revisions, annotations, elaborations, or other modifications\n      represent, as a whole, an original work of authorship. For the purposes\n      of this License, Derivative Works shall not include works that remain\n      separable from, or merely link (or bind by name) to the interfaces of,\n      the Work and Derivative Works thereof.\n\n      \"Contribution\" shall mean any work of authorship, including\n      the original version of the Work and any modifications or additions\n      to that Work or Derivative Works thereof, that is intentionally\n      submitted to Licensor for inclusion in the Work by the copyright owner\n      or by an individual or Legal Entity authorized to submit on behalf of\n      the copyright owner. For the purposes of this definition, \"submitted\"\n      means any form of electronic, verbal, or written communication sent\n      to the Licensor or its representatives, including but not limited to\n      communication on electronic mailing lists, source code control systems,\n      and issue tracking systems that are managed by, or on behalf of, the\n      Licensor for the purpose of discussing and improving the Work, but\n      excluding communication that is conspicuously marked or otherwise\n      designated in writing by the copyright owner as \"Not a Contribution.\"\n\n      \"Contributor\" shall mean Licensor and any individual or Legal Entity\n      on behalf of whom a Contribution has been received by Licensor and\n      subsequently incorporated within the Work.\n\n   2. Grant of Copyright License. Subject to the terms and conditions of\n      this License, each Contributor hereby grants to You a perpetual,\n      worldwide, non-exclusive, no-charge, royalty-free, irrevocable\n      copyright license to reproduce, prepare Derivative Works of,\n      publicly display, publicly perform, sublicense, and distribute the\n      Work and such Derivative Works in Source or Object form.\n\n   3. Grant of Patent License. Subject to the terms and conditions of\n      this License, each Contributor hereby grants to You a perpetual,\n      worldwide, non-exclusive, no-charge, royalty-free, irrevocable\n      (except as stated in this section) patent license to make, have made,\n      use, offer to sell, sell, import, and otherwise transfer the Work,\n      where such license applies only to those patent claims licensable\n      by such Contributor that are necessarily infringed by their\n      Contribution(s) alone or by combination of their Contribution(s)\n      with the Work to which such Contribution(s) was submitted. If You\n      institute patent litigation against any entity (including a\n      cross-claim or counterclaim in a lawsuit) alleging that the Work\n      or a Contribution incorporated within the Work constitutes direct\n      or contributory patent infringement, then any patent licenses\n      granted to You under this License for that Work shall terminate\n      as of the date such litigation is filed.\n\n   4. Redistribution. You may reproduce and distribute copies of the\n      Work or Derivative Works thereof in any medium, with or without\n      modifications, and in Source or Object form, provided that You\n      meet the following conditions:\n\n      (a) You must give any other recipients of the Work or\n          Derivative Works a copy of this License; and\n\n      (b) You must cause any modified files to carry prominent notices\n          stating that You changed the files; and\n\n      (c) You must retain, in the Source form of any Derivative Works\n          that You distribute, all copyright, patent, trademark, and\n          attribution notices from the Source form of the Work,\n          excluding those notices that do not pertain to any part of\n          the Derivative Works; and\n\n      (d) If the Work includes a \"NOTICE\" text file as part of its\n          distribution, then any Derivative Works that You distribute must\n          include a readable copy of the attribution notices contained\n          within such NOTICE file, excluding those notices that do not\n          pertain to any part of the Derivative Works, in at least one\n          of the following places: within a NOTICE text file distributed\n          as part of the Derivative Works; within the Source form or\n          documentation, if provided along with the Derivative Works; or,\n          within a display generated by the Derivative Works, if and\n          wherever such third-party notices normally appear. The contents\n          of the NOTICE file are for informational purposes only and\n          do not modify the License. You may add Your own attribution\n          notices within Derivative Works that You distribute, alongside\n          or as an addendum to the NOTICE text from the Work, provided\n          that such additional attribution notices cannot be construed\n          as modifying the License.\n\n      You may add Your own copyright statement to Your modifications and\n      may provide additional or different license terms and conditions\n      for use, reproduction, or distribution of Your modifications, or\n      for any such Derivative Works as a whole, provided Your use,\n      reproduction, and distribution of the Work otherwise complies with\n      the conditions stated in this License.\n\n   5. Submission of Contributions. Unless You explicitly state otherwise,\n      any Contribution intentionally submitted for inclusion in the Work\n      by You to the Licensor shall be under the terms and conditions of\n      this License, without any additional terms or conditions.\n      Notwithstanding the above, nothing herein shall supersede or modify\n      the terms of any separate license agreement you may have executed\n      with Licensor regarding such Contributions.\n\n   6. Trademarks. This License does not grant permission to use the trade\n      names, trademarks, service marks, or product names of the Licensor,\n      except as required for reasonable and customary use in describing the\n      origin of the Work and reproducing the content of the NOTICE file.\n\n   7. Disclaimer of Warranty. Unless required by applicable law or\n      agreed to in writing, Licensor provides the Work (and each\n      Contributor provides its Contributions) on an \"AS IS\" BASIS,\n      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or\n      implied, including, without limitation, any warranties or conditions\n      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A\n      PARTICULAR PURPOSE. You are solely responsible for determining the\n      appropriateness of using or redistributing the Work and assume any\n      risks associated with Your exercise of permissions under this License.\n\n   8. Limitation of Liability. In no event and under no legal theory,\n      whether in tort (including negligence), contract, or otherwise,\n      unless required by applicable law (such as deliberate and grossly\n      negligent acts) or agreed to in writing, shall any Contributor be\n      liable to You for damages, including any direct, indirect, special,\n      incidental, or consequential damages of any character arising as a\n      result of this License or out of the use or inability to use the\n      Work (including but not limited to damages for loss of goodwill,\n      work stoppage, computer failure or malfunction, or any and all\n      other commercial damages or losses), even if such Contributor\n      has been advised of the possibility of such damages.\n\n   9. Accepting Warranty or Additional Liability. While redistributing\n      the Work or Derivative Works thereof, You may choose to offer,\n      and charge a fee for, acceptance of support, warranty, indemnity,\n      or other liability obligations and/or rights consistent with this\n      License. However, in accepting such obligations, You may act only\n      on Your own behalf and on Your sole responsibility, not on behalf\n      of any other Contributor, and only if You agree to indemnify,\n      defend, and hold each Contributor harmless for any liability\n      incurred by, or claims asserted against, such Contributor by reason\n      of your accepting any such warranty or additional liability.\n\n   END OF TERMS AND CONDITIONS\n\n   APPENDIX: How to apply the Apache License to your work.\n\n      To apply the Apache License to your work, attach the following\n      boilerplate notice, with the fields enclosed by brackets \"[]\"\n      replaced with your own identifying information. (Don't include\n      the brackets!)  The text should be enclosed in the appropriate\n      comment syntax for the file format. We also recommend that a\n      file or class name and description of purpose be included on the\n      same \"printed page\" as the copyright notice for easier\n      identification within third-party archives.\n\n   Copyright [yyyy] [name of copyright owner]\n\n   Licensed under the Apache License, Version 2.0 (the \"License\");\n   you may not use this file except in compliance with the License.\n   You may obtain a copy of the License at\n\n       http://www.apache.org/licenses/LICENSE-2.0\n\n   Unless required by applicable law or agreed to in writing, software\n   distributed under the License is distributed on an \"AS IS\" BASIS,\n   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n   See the License for the specific language governing permissions and\n   limitations under the License.\n"
  },
  {
    "path": "pingora-error/src/immut_str.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\nuse std::fmt;\n\n/// A data struct that holds either immutable string or reference to static str.\n/// Compared to String or `Box<str>`, it avoids memory allocation on static str.\n#[derive(Debug, PartialEq, Eq, Clone)]\npub enum ImmutStr {\n    Static(&'static str),\n    Owned(Box<str>),\n}\n\nimpl ImmutStr {\n    #[inline]\n    pub fn as_str(&self) -> &str {\n        match self {\n            ImmutStr::Static(s) => s,\n            ImmutStr::Owned(s) => s.as_ref(),\n        }\n    }\n\n    pub fn is_owned(&self) -> bool {\n        match self {\n            ImmutStr::Static(_) => false,\n            ImmutStr::Owned(_) => true,\n        }\n    }\n}\n\nimpl fmt::Display for ImmutStr {\n    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {\n        write!(f, \"{}\", self.as_str())\n    }\n}\n\nimpl From<&'static str> for ImmutStr {\n    fn from(s: &'static str) -> Self {\n        ImmutStr::Static(s)\n    }\n}\n\nimpl From<String> for ImmutStr {\n    fn from(s: String) -> Self {\n        ImmutStr::Owned(s.into_boxed_str())\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use super::*;\n\n    #[test]\n    fn test_static_vs_owned() {\n        let s: ImmutStr = \"test\".into();\n        assert!(!s.is_owned());\n        let s: ImmutStr = \"test\".to_string().into();\n        assert!(s.is_owned());\n    }\n}\n"
  },
  {
    "path": "pingora-error/src/lib.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n#![warn(clippy::all)]\n//! The library to provide the struct to represent errors in pingora.\n\npub use std::error::Error as ErrorTrait;\nuse std::fmt;\nuse std::fmt::Debug;\nuse std::result::Result as StdResult;\n\nmod immut_str;\npub use immut_str::ImmutStr;\n\n/// The boxed [Error], the desired way to pass [Error]\npub type BError = Box<Error>;\n/// Syntax sugar for `std::Result<T, BError>`\npub type Result<T, E = BError> = StdResult<T, E>;\n\n/// The struct that represents an error\n#[derive(Debug)]\npub struct Error {\n    /// the type of error\n    pub etype: ErrorType,\n    /// the source of error: from upstream, downstream or internal\n    pub esource: ErrorSource,\n    /// if the error is retry-able\n    pub retry: RetryType,\n    /// chain to the cause of this error\n    pub cause: Option<Box<dyn ErrorTrait + Send + Sync>>,\n    /// an arbitrary string that explains the context when the error happens\n    pub context: Option<ImmutStr>,\n}\n\n/// The source of the error\n#[derive(Debug, PartialEq, Eq, Clone)]\npub enum ErrorSource {\n    /// The error is caused by the remote server\n    Upstream,\n    /// The error is caused by the remote client\n    Downstream,\n    /// The error is caused by the internal logic\n    Internal,\n    /// Error source unknown or to be set\n    Unset,\n}\n\n/// Whether the request can be retried after encountering this error\n#[derive(Debug, PartialEq, Eq, Clone, Copy)]\npub enum RetryType {\n    Decided(bool),\n    ReusedOnly, // only retry when the error is from a reused connection\n}\n\nimpl RetryType {\n    pub fn decide_reuse(&mut self, reused: bool) {\n        if matches!(self, RetryType::ReusedOnly) {\n            *self = RetryType::Decided(reused);\n        }\n    }\n\n    pub fn retry(&self) -> bool {\n        match self {\n            RetryType::Decided(b) => *b,\n            RetryType::ReusedOnly => {\n                panic!(\"Retry is not decided\")\n            }\n        }\n    }\n}\n\nimpl From<bool> for RetryType {\n    fn from(b: bool) -> Self {\n        RetryType::Decided(b)\n    }\n}\n\nimpl ErrorSource {\n    /// for displaying the error source\n    pub fn as_str(&self) -> &'static str {\n        match self {\n            Self::Upstream => \"Upstream\",\n            Self::Downstream => \"Downstream\",\n            Self::Internal => \"Internal\",\n            Self::Unset => \"\",\n        }\n    }\n}\n\n/// Predefined type of errors\n#[derive(Debug, PartialEq, Eq, Clone)]\npub enum ErrorType {\n    // connect errors\n    ConnectTimedout,\n    ConnectRefused,\n    ConnectNoRoute,\n    TLSWantX509Lookup,\n    TLSHandshakeFailure,\n    TLSHandshakeTimedout,\n    InvalidCert,\n    HandshakeError, // other handshake\n    ConnectError,   // catch all\n    BindError,\n    AcceptError,\n    SocketError,\n    ConnectProxyFailure,\n    // protocol errors\n    InvalidHTTPHeader,\n    H1Error,     // catch all\n    H2Error,     // catch all\n    H2Downgrade, // Peer over h2 requests to downgrade to h1\n    InvalidH2,   // Peer sends invalid h2 frames to us\n    // IO error on established connections\n    ReadError,\n    WriteError,\n    ReadTimedout,\n    WriteTimedout,\n    ConnectionClosed,\n    // application error, will return HTTP status code\n    HTTPStatus(u16),\n    // file related\n    FileOpenError,\n    FileCreateError,\n    FileReadError,\n    FileWriteError,\n    // other errors\n    InternalError,\n    // catch all\n    UnknownError,\n    /// Custom error with static string.\n    /// this field is to allow users to extend the types of errors. If runtime generated string\n    /// is needed, it is more likely to be treated as \"context\" rather than \"type\".\n    Custom(&'static str),\n    /// Custom error with static string and code.\n    /// this field allows users to extend error further with error codes.\n    CustomCode(&'static str, u16),\n}\n\nimpl ErrorType {\n    /// create a new type of error. Users should try to make `name` unique.\n    pub const fn new(name: &'static str) -> Self {\n        ErrorType::Custom(name)\n    }\n\n    /// create a new type of error. Users should try to make `name` unique.\n    pub const fn new_code(name: &'static str, code: u16) -> Self {\n        ErrorType::CustomCode(name, code)\n    }\n\n    /// for displaying the error type\n    pub fn as_str(&self) -> &'static str {\n        match self {\n            ErrorType::ConnectTimedout => \"ConnectTimedout\",\n            ErrorType::ConnectRefused => \"ConnectRefused\",\n            ErrorType::ConnectNoRoute => \"ConnectNoRoute\",\n            ErrorType::ConnectProxyFailure => \"ConnectProxyFailure\",\n            ErrorType::TLSWantX509Lookup => \"TLSWantX509Lookup\",\n            ErrorType::TLSHandshakeFailure => \"TLSHandshakeFailure\",\n            ErrorType::TLSHandshakeTimedout => \"TLSHandshakeTimedout\",\n            ErrorType::InvalidCert => \"InvalidCert\",\n            ErrorType::HandshakeError => \"HandshakeError\",\n            ErrorType::ConnectError => \"ConnectError\",\n            ErrorType::BindError => \"BindError\",\n            ErrorType::AcceptError => \"AcceptError\",\n            ErrorType::SocketError => \"SocketError\",\n            ErrorType::InvalidHTTPHeader => \"InvalidHTTPHeader\",\n            ErrorType::H1Error => \"H1Error\",\n            ErrorType::H2Error => \"H2Error\",\n            ErrorType::InvalidH2 => \"InvalidH2\",\n            ErrorType::H2Downgrade => \"H2Downgrade\",\n            ErrorType::ReadError => \"ReadError\",\n            ErrorType::WriteError => \"WriteError\",\n            ErrorType::ReadTimedout => \"ReadTimedout\",\n            ErrorType::WriteTimedout => \"WriteTimedout\",\n            ErrorType::ConnectionClosed => \"ConnectionClosed\",\n            ErrorType::FileOpenError => \"FileOpenError\",\n            ErrorType::FileCreateError => \"FileCreateError\",\n            ErrorType::FileReadError => \"FileReadError\",\n            ErrorType::FileWriteError => \"FileWriteError\",\n            ErrorType::HTTPStatus(_) => \"HTTPStatus\",\n            ErrorType::InternalError => \"InternalError\",\n            ErrorType::UnknownError => \"UnknownError\",\n            ErrorType::Custom(s) => s,\n            ErrorType::CustomCode(s, _) => s,\n        }\n    }\n}\n\nimpl Error {\n    /// Simply create the error. See other functions that provide less verbose interfaces.\n    #[inline]\n    pub fn create(\n        etype: ErrorType,\n        esource: ErrorSource,\n        context: Option<ImmutStr>,\n        cause: Option<Box<dyn ErrorTrait + Send + Sync>>,\n    ) -> BError {\n        let retry = if let Some(c) = cause.as_ref() {\n            if let Some(e) = c.downcast_ref::<BError>() {\n                e.retry\n            } else {\n                false.into()\n            }\n        } else {\n            false.into()\n        };\n        Box::new(Error {\n            etype,\n            esource,\n            retry,\n            cause,\n            context,\n        })\n    }\n\n    #[inline]\n    fn do_new(e: ErrorType, s: ErrorSource) -> BError {\n        Self::create(e, s, None, None)\n    }\n\n    /// Create an error with the given type\n    #[inline]\n    pub fn new(e: ErrorType) -> BError {\n        Self::do_new(e, ErrorSource::Unset)\n    }\n\n    /// Create an error with the given type, a context string and the causing error.\n    /// This method is usually used when there the error is caused by another error.\n    /// ```\n    /// use pingora_error::{Error, ErrorType, Result};\n    ///\n    /// fn b() -> Result<()> {\n    ///     // ...\n    ///     Ok(())\n    /// }\n    /// fn do_something() -> Result<()> {\n    ///     // a()?;\n    ///     b().map_err(|e| Error::because(ErrorType::InternalError, \"b failed after a\", e))\n    /// }\n    /// ```\n    /// Choose carefully between simply surfacing the causing error versus Because() here.\n    /// Only use Because() when there is extra context that is not capture by\n    /// the causing error itself.\n    #[inline]\n    pub fn because<S: Into<ImmutStr>, E: Into<Box<dyn ErrorTrait + Send + Sync>>>(\n        e: ErrorType,\n        context: S,\n        cause: E,\n    ) -> BError {\n        Self::create(\n            e,\n            ErrorSource::Unset,\n            Some(context.into()),\n            Some(cause.into()),\n        )\n    }\n\n    /// Short for Err(Self::because)\n    #[inline]\n    pub fn e_because<T, S: Into<ImmutStr>, E: Into<Box<dyn ErrorTrait + Send + Sync>>>(\n        e: ErrorType,\n        context: S,\n        cause: E,\n    ) -> Result<T> {\n        Err(Self::because(e, context, cause))\n    }\n\n    /// Create an error with context but no direct causing error\n    #[inline]\n    pub fn explain<S: Into<ImmutStr>>(e: ErrorType, context: S) -> BError {\n        Self::create(e, ErrorSource::Unset, Some(context.into()), None)\n    }\n\n    /// Short for Err(Self::explain)\n    #[inline]\n    pub fn e_explain<T, S: Into<ImmutStr>>(e: ErrorType, context: S) -> Result<T> {\n        Err(Self::explain(e, context))\n    }\n\n    /// The new_{up, down, in} functions are to create new errors with source\n    /// {upstream, downstream, internal}\n    #[inline]\n    pub fn new_up(e: ErrorType) -> BError {\n        Self::do_new(e, ErrorSource::Upstream)\n    }\n\n    #[inline]\n    pub fn new_down(e: ErrorType) -> BError {\n        Self::do_new(e, ErrorSource::Downstream)\n    }\n\n    #[inline]\n    pub fn new_in(e: ErrorType) -> BError {\n        Self::do_new(e, ErrorSource::Internal)\n    }\n\n    /// Create a new custom error with the static string\n    #[inline]\n    pub fn new_str(s: &'static str) -> BError {\n        Self::do_new(ErrorType::Custom(s), ErrorSource::Unset)\n    }\n\n    // the err_* functions are the same as new_* but return a Result<T>\n    #[inline]\n    pub fn err<T>(e: ErrorType) -> Result<T> {\n        Err(Self::new(e))\n    }\n\n    #[inline]\n    pub fn err_up<T>(e: ErrorType) -> Result<T> {\n        Err(Self::new_up(e))\n    }\n\n    #[inline]\n    pub fn err_down<T>(e: ErrorType) -> Result<T> {\n        Err(Self::new_down(e))\n    }\n\n    #[inline]\n    pub fn err_in<T>(e: ErrorType) -> Result<T> {\n        Err(Self::new_in(e))\n    }\n\n    pub fn etype(&self) -> &ErrorType {\n        &self.etype\n    }\n\n    pub fn esource(&self) -> &ErrorSource {\n        &self.esource\n    }\n\n    pub fn retry(&self) -> bool {\n        self.retry.retry()\n    }\n\n    pub fn set_retry(&mut self, retry: bool) {\n        self.retry = retry.into();\n    }\n\n    pub fn reason_str(&self) -> &str {\n        self.etype.as_str()\n    }\n\n    pub fn source_str(&self) -> &str {\n        self.esource.as_str()\n    }\n\n    /// The as_{up, down, in} functions are to change the current errors with source\n    /// {upstream, downstream, internal}\n    pub fn as_up(&mut self) {\n        self.esource = ErrorSource::Upstream;\n    }\n\n    pub fn as_down(&mut self) {\n        self.esource = ErrorSource::Downstream;\n    }\n\n    pub fn as_in(&mut self) {\n        self.esource = ErrorSource::Internal;\n    }\n\n    /// The into_{up, down, in} are the same as as_* but takes `self` and also return `self`\n    pub fn into_up(mut self: BError) -> BError {\n        self.as_up();\n        self\n    }\n\n    pub fn into_down(mut self: BError) -> BError {\n        self.as_down();\n        self\n    }\n\n    pub fn into_in(mut self: BError) -> BError {\n        self.as_in();\n        self\n    }\n\n    pub fn into_err<T>(self: BError) -> Result<T> {\n        Err(self)\n    }\n\n    pub fn set_cause<C: Into<Box<dyn ErrorTrait + Send + Sync>>>(&mut self, cause: C) {\n        self.cause = Some(cause.into());\n    }\n\n    pub fn set_context<T: Into<ImmutStr>>(&mut self, context: T) {\n        self.context = Some(context.into());\n    }\n\n    /// Create a new error from self, with the same type and source and put self as the cause\n    /// ```\n    /// use pingora_error::Result;\n    ///\n    ///  fn b() -> Result<()> {\n    ///     // ...\n    ///     Ok(())\n    /// }\n    ///\n    /// fn do_something() -> Result<()> {\n    ///     // a()?;\n    ///     b().map_err(|e| e.more_context(\"b failed after a\"))\n    /// }\n    /// ```\n    /// This function is less verbose than `Because`. But it only work for [Error] while\n    /// `Because` works for all types of errors who implement [std::error::Error] trait.\n    pub fn more_context<T: Into<ImmutStr>>(self: BError, context: T) -> BError {\n        let esource = self.esource.clone();\n        let retry = self.retry;\n        let mut e = Self::because(self.etype.clone(), context, self);\n        e.esource = esource;\n        e.retry = retry;\n        e\n    }\n\n    // Display error but skip the duplicate elements from the error in previous hop\n    fn chain_display(&self, previous: Option<&Error>, f: &mut fmt::Formatter<'_>) -> fmt::Result {\n        if previous.map(|p| p.esource != self.esource).unwrap_or(true) {\n            write!(f, \"{}\", self.esource.as_str())?\n        }\n        if previous.map(|p| p.etype != self.etype).unwrap_or(true) {\n            write!(f, \" {}\", self.etype.as_str())?\n        }\n\n        if let Some(c) = self.context.as_ref() {\n            write!(f, \" context: {}\", c)?;\n        }\n        if let Some(c) = self.cause.as_ref() {\n            if let Some(e) = c.downcast_ref::<BError>() {\n                write!(f, \" cause: \")?;\n                e.chain_display(Some(self), f)\n            } else {\n                write!(f, \" cause: {}\", c)\n            }\n        } else {\n            Ok(())\n        }\n    }\n\n    /// Return the ErrorType of the root Error\n    pub fn root_etype(&self) -> &ErrorType {\n        self.cause.as_ref().map_or(&self.etype, |c| {\n            // Stop the recursion if the cause is not Error\n            c.downcast_ref::<BError>()\n                .map_or(&self.etype, |e| e.root_etype())\n        })\n    }\n\n    pub fn root_cause(&self) -> &(dyn ErrorTrait + Send + Sync + 'static) {\n        self.cause.as_deref().map_or(self, |c| {\n            c.downcast_ref::<BError>().map_or(c, |e| e.root_cause())\n        })\n    }\n}\n\nimpl fmt::Display for Error {\n    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {\n        self.chain_display(None, f)\n    }\n}\n\nimpl ErrorTrait for Error {}\n\n/// Helper trait to add more context to a given error\npub trait Context<T> {\n    /// Wrap the `Err(E)` in [Result] with more context, the existing E will be the cause.\n    ///\n    /// This is a shortcut for map_err() + more_context()\n    fn err_context<C: Into<ImmutStr>, F: FnOnce() -> C>(self, context: F) -> Result<T, BError>;\n}\n\nimpl<T> Context<T> for Result<T, BError> {\n    fn err_context<C: Into<ImmutStr>, F: FnOnce() -> C>(self, context: F) -> Result<T, BError> {\n        self.map_err(|e| e.more_context(context()))\n    }\n}\n\n/// Helper trait to chain errors with context\npub trait OrErr<T, E> {\n    /// Wrap the E in [Result] with new [ErrorType] and context, the existing E will be the cause.\n    ///\n    /// This is a shortcut for map_err() + because()\n    fn or_err(self, et: ErrorType, context: &'static str) -> Result<T, BError>\n    where\n        E: Into<Box<dyn ErrorTrait + Send + Sync>>;\n\n    /// Similar to or_err(), but takes a closure, which is useful for constructing String.\n    fn or_err_with<C: Into<ImmutStr>, F: FnOnce() -> C>(\n        self,\n        et: ErrorType,\n        context: F,\n    ) -> Result<T, BError>\n    where\n        E: Into<Box<dyn ErrorTrait + Send + Sync>>;\n\n    /// Replace the E in [Result] with a new [Error] generated from the current error\n    ///\n    /// This is useful when the current error cannot move out of scope. This is a shortcut for map_err() + explain().\n    fn explain_err<C: Into<ImmutStr>, F: FnOnce(E) -> C>(\n        self,\n        et: ErrorType,\n        context: F,\n    ) -> Result<T, BError>;\n\n    /// Similar to or_err() but just to surface errors that are not [Error] (where `?` cannot be used directly).\n    ///\n    /// or_err()/or_err_with() are still preferred because they make the error more readable and traceable.\n    fn or_fail(self) -> Result<T>\n    where\n        E: Into<Box<dyn ErrorTrait + Send + Sync>>;\n}\n\nimpl<T, E> OrErr<T, E> for Result<T, E> {\n    fn or_err(self, et: ErrorType, context: &'static str) -> Result<T, BError>\n    where\n        E: Into<Box<dyn ErrorTrait + Send + Sync>>,\n    {\n        self.map_err(|e| Error::because(et, context, e))\n    }\n\n    fn or_err_with<C: Into<ImmutStr>, F: FnOnce() -> C>(\n        self,\n        et: ErrorType,\n        context: F,\n    ) -> Result<T, BError>\n    where\n        E: Into<Box<dyn ErrorTrait + Send + Sync>>,\n    {\n        self.map_err(|e| Error::because(et, context(), e))\n    }\n\n    fn explain_err<C: Into<ImmutStr>, F: FnOnce(E) -> C>(\n        self,\n        et: ErrorType,\n        exp: F,\n    ) -> Result<T, BError> {\n        self.map_err(|e| Error::explain(et, exp(e)))\n    }\n\n    fn or_fail(self) -> Result<T, BError>\n    where\n        E: Into<Box<dyn ErrorTrait + Send + Sync>>,\n    {\n        self.map_err(|e| Error::because(ErrorType::InternalError, \"\", e))\n    }\n}\n\n/// Helper trait to convert an [Option] to an [Error] with context.\npub trait OkOrErr<T> {\n    fn or_err(self, et: ErrorType, context: &'static str) -> Result<T, BError>;\n\n    fn or_err_with<C: Into<ImmutStr>, F: FnOnce() -> C>(\n        self,\n        et: ErrorType,\n        context: F,\n    ) -> Result<T, BError>;\n}\n\nimpl<T> OkOrErr<T> for Option<T> {\n    /// Convert the [Option] to a new [Error] with [ErrorType] and context if None, Ok otherwise.\n    ///\n    /// This is a shortcut for .ok_or(Error::explain())\n    fn or_err(self, et: ErrorType, context: &'static str) -> Result<T, BError> {\n        self.ok_or(Error::explain(et, context))\n    }\n\n    /// Similar to to_err(), but takes a closure, which is useful for constructing String.\n    fn or_err_with<C: Into<ImmutStr>, F: FnOnce() -> C>(\n        self,\n        et: ErrorType,\n        context: F,\n    ) -> Result<T, BError> {\n        self.ok_or_else(|| Error::explain(et, context()))\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use super::*;\n\n    #[test]\n    fn test_chain_of_error() {\n        let e1 = Error::new(ErrorType::InternalError);\n        let mut e2 = Error::new(ErrorType::HTTPStatus(400));\n        e2.set_cause(e1);\n        assert_eq!(format!(\"{}\", e2), \" HTTPStatus cause:  InternalError\");\n        assert_eq!(e2.root_etype().as_str(), \"InternalError\");\n\n        let e3 = Error::new(ErrorType::InternalError);\n        let e4 = Error::because(ErrorType::HTTPStatus(400), \"test\", e3);\n        assert_eq!(\n            format!(\"{}\", e4),\n            \" HTTPStatus context: test cause:  InternalError\"\n        );\n        assert_eq!(e4.root_etype().as_str(), \"InternalError\");\n    }\n\n    #[test]\n    fn test_error_context() {\n        let mut e1 = Error::new(ErrorType::InternalError);\n        e1.set_context(format!(\"{} {}\", \"my\", \"context\"));\n        assert_eq!(format!(\"{}\", e1), \" InternalError context: my context\");\n    }\n\n    #[test]\n    fn test_context_trait() {\n        let e1: Result<(), BError> = Err(Error::new(ErrorType::InternalError));\n        let e2 = e1.err_context(|| \"another\");\n        assert_eq!(\n            format!(\"{}\", e2.unwrap_err()),\n            \" InternalError context: another cause: \"\n        );\n    }\n\n    #[test]\n    fn test_cause_trait() {\n        let e1: Result<(), BError> = Err(Error::new(ErrorType::InternalError));\n        let e2 = e1.or_err(ErrorType::HTTPStatus(400), \"another\");\n        assert_eq!(\n            format!(\"{}\", e2.unwrap_err()),\n            \" HTTPStatus context: another cause:  InternalError\"\n        );\n    }\n\n    #[test]\n    fn test_option_some_ok() {\n        let m = Some(2);\n        let o = m.or_err(ErrorType::InternalError, \"some is not an error!\");\n        assert_eq!(2, o.unwrap());\n\n        let o = m.or_err_with(ErrorType::InternalError, || \"some is not an error!\");\n        assert_eq!(2, o.unwrap());\n    }\n\n    #[test]\n    fn test_option_none_err() {\n        let m: Option<i32> = None;\n        let e1 = m.or_err(ErrorType::InternalError, \"none is an error!\");\n        assert_eq!(\n            format!(\"{}\", e1.unwrap_err()),\n            \" InternalError context: none is an error!\"\n        );\n\n        let e1 = m.or_err_with(ErrorType::InternalError, || \"none is an error!\");\n        assert_eq!(\n            format!(\"{}\", e1.unwrap_err()),\n            \" InternalError context: none is an error!\"\n        );\n    }\n\n    #[test]\n    fn test_into() {\n        fn other_error() -> Result<(), &'static str> {\n            Err(\"oops\")\n        }\n\n        fn surface_err() -> Result<()> {\n            other_error().or_fail()?; // can return directly but want to showcase ?\n            Ok(())\n        }\n\n        let e = surface_err().unwrap_err();\n        assert_eq!(format!(\"{}\", e), \" InternalError context:  cause: oops\");\n    }\n}\n"
  },
  {
    "path": "pingora-header-serde/Cargo.toml",
    "content": "[package]\nname = \"pingora-header-serde\"\nversion = \"0.8.0\"\nauthors = [\"Yuchen Wu <yuchen@cloudflare.com>\"]\nlicense = \"Apache-2.0\"\nedition = \"2021\"\nrepository = \"https://github.com/cloudflare/pingora\"\ncategories = [\"compression\"]\nkeywords = [\"http\", \"compression\", \"pingora\"]\nexclude = [\"samples/*\"]\ndescription = \"\"\"\nHTTP header (de)serialization and compression for Pingora.\n\"\"\"\n\n# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html\n[lib]\nname = \"pingora_header_serde\"\npath = \"src/lib.rs\"\n\n[[bin]]\nname = \"trainer\"\npath = \"src/trainer.rs\"\n\n[dependencies]\nzstd = \"0.13.1\"\nzstd-safe = { version = \"7.1.0\", features = [\"std\"] }\nhttp = { workspace = true }\nbytes = { workspace = true }\nhttparse = { workspace = true }\npingora-error = { version = \"0.8.0\", path = \"../pingora-error\" }\npingora-http = { version = \"0.8.0\", path = \"../pingora-http\" }\nthread_local = \"1.0\"\n"
  },
  {
    "path": "pingora-header-serde/LICENSE",
    "content": "\n                                 Apache License\n                           Version 2.0, January 2004\n                        http://www.apache.org/licenses/\n\n   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION\n\n   1. Definitions.\n\n      \"License\" shall mean the terms and conditions for use, reproduction,\n      and distribution as defined by Sections 1 through 9 of this document.\n\n      \"Licensor\" shall mean the copyright owner or entity authorized by\n      the copyright owner that is granting the License.\n\n      \"Legal Entity\" shall mean the union of the acting entity and all\n      other entities that control, are controlled by, or are under common\n      control with that entity. For the purposes of this definition,\n      \"control\" means (i) the power, direct or indirect, to cause the\n      direction or management of such entity, whether by contract or\n      otherwise, or (ii) ownership of fifty percent (50%) or more of the\n      outstanding shares, or (iii) beneficial ownership of such entity.\n\n      \"You\" (or \"Your\") shall mean an individual or Legal Entity\n      exercising permissions granted by this License.\n\n      \"Source\" form shall mean the preferred form for making modifications,\n      including but not limited to software source code, documentation\n      source, and configuration files.\n\n      \"Object\" form shall mean any form resulting from mechanical\n      transformation or translation of a Source form, including but\n      not limited to compiled object code, generated documentation,\n      and conversions to other media types.\n\n      \"Work\" shall mean the work of authorship, whether in Source or\n      Object form, made available under the License, as indicated by a\n      copyright notice that is included in or attached to the work\n      (an example is provided in the Appendix below).\n\n      \"Derivative Works\" shall mean any work, whether in Source or Object\n      form, that is based on (or derived from) the Work and for which the\n      editorial revisions, annotations, elaborations, or other modifications\n      represent, as a whole, an original work of authorship. For the purposes\n      of this License, Derivative Works shall not include works that remain\n      separable from, or merely link (or bind by name) to the interfaces of,\n      the Work and Derivative Works thereof.\n\n      \"Contribution\" shall mean any work of authorship, including\n      the original version of the Work and any modifications or additions\n      to that Work or Derivative Works thereof, that is intentionally\n      submitted to Licensor for inclusion in the Work by the copyright owner\n      or by an individual or Legal Entity authorized to submit on behalf of\n      the copyright owner. For the purposes of this definition, \"submitted\"\n      means any form of electronic, verbal, or written communication sent\n      to the Licensor or its representatives, including but not limited to\n      communication on electronic mailing lists, source code control systems,\n      and issue tracking systems that are managed by, or on behalf of, the\n      Licensor for the purpose of discussing and improving the Work, but\n      excluding communication that is conspicuously marked or otherwise\n      designated in writing by the copyright owner as \"Not a Contribution.\"\n\n      \"Contributor\" shall mean Licensor and any individual or Legal Entity\n      on behalf of whom a Contribution has been received by Licensor and\n      subsequently incorporated within the Work.\n\n   2. Grant of Copyright License. Subject to the terms and conditions of\n      this License, each Contributor hereby grants to You a perpetual,\n      worldwide, non-exclusive, no-charge, royalty-free, irrevocable\n      copyright license to reproduce, prepare Derivative Works of,\n      publicly display, publicly perform, sublicense, and distribute the\n      Work and such Derivative Works in Source or Object form.\n\n   3. Grant of Patent License. Subject to the terms and conditions of\n      this License, each Contributor hereby grants to You a perpetual,\n      worldwide, non-exclusive, no-charge, royalty-free, irrevocable\n      (except as stated in this section) patent license to make, have made,\n      use, offer to sell, sell, import, and otherwise transfer the Work,\n      where such license applies only to those patent claims licensable\n      by such Contributor that are necessarily infringed by their\n      Contribution(s) alone or by combination of their Contribution(s)\n      with the Work to which such Contribution(s) was submitted. If You\n      institute patent litigation against any entity (including a\n      cross-claim or counterclaim in a lawsuit) alleging that the Work\n      or a Contribution incorporated within the Work constitutes direct\n      or contributory patent infringement, then any patent licenses\n      granted to You under this License for that Work shall terminate\n      as of the date such litigation is filed.\n\n   4. Redistribution. You may reproduce and distribute copies of the\n      Work or Derivative Works thereof in any medium, with or without\n      modifications, and in Source or Object form, provided that You\n      meet the following conditions:\n\n      (a) You must give any other recipients of the Work or\n          Derivative Works a copy of this License; and\n\n      (b) You must cause any modified files to carry prominent notices\n          stating that You changed the files; and\n\n      (c) You must retain, in the Source form of any Derivative Works\n          that You distribute, all copyright, patent, trademark, and\n          attribution notices from the Source form of the Work,\n          excluding those notices that do not pertain to any part of\n          the Derivative Works; and\n\n      (d) If the Work includes a \"NOTICE\" text file as part of its\n          distribution, then any Derivative Works that You distribute must\n          include a readable copy of the attribution notices contained\n          within such NOTICE file, excluding those notices that do not\n          pertain to any part of the Derivative Works, in at least one\n          of the following places: within a NOTICE text file distributed\n          as part of the Derivative Works; within the Source form or\n          documentation, if provided along with the Derivative Works; or,\n          within a display generated by the Derivative Works, if and\n          wherever such third-party notices normally appear. The contents\n          of the NOTICE file are for informational purposes only and\n          do not modify the License. You may add Your own attribution\n          notices within Derivative Works that You distribute, alongside\n          or as an addendum to the NOTICE text from the Work, provided\n          that such additional attribution notices cannot be construed\n          as modifying the License.\n\n      You may add Your own copyright statement to Your modifications and\n      may provide additional or different license terms and conditions\n      for use, reproduction, or distribution of Your modifications, or\n      for any such Derivative Works as a whole, provided Your use,\n      reproduction, and distribution of the Work otherwise complies with\n      the conditions stated in this License.\n\n   5. Submission of Contributions. Unless You explicitly state otherwise,\n      any Contribution intentionally submitted for inclusion in the Work\n      by You to the Licensor shall be under the terms and conditions of\n      this License, without any additional terms or conditions.\n      Notwithstanding the above, nothing herein shall supersede or modify\n      the terms of any separate license agreement you may have executed\n      with Licensor regarding such Contributions.\n\n   6. Trademarks. This License does not grant permission to use the trade\n      names, trademarks, service marks, or product names of the Licensor,\n      except as required for reasonable and customary use in describing the\n      origin of the Work and reproducing the content of the NOTICE file.\n\n   7. Disclaimer of Warranty. Unless required by applicable law or\n      agreed to in writing, Licensor provides the Work (and each\n      Contributor provides its Contributions) on an \"AS IS\" BASIS,\n      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or\n      implied, including, without limitation, any warranties or conditions\n      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A\n      PARTICULAR PURPOSE. You are solely responsible for determining the\n      appropriateness of using or redistributing the Work and assume any\n      risks associated with Your exercise of permissions under this License.\n\n   8. Limitation of Liability. In no event and under no legal theory,\n      whether in tort (including negligence), contract, or otherwise,\n      unless required by applicable law (such as deliberate and grossly\n      negligent acts) or agreed to in writing, shall any Contributor be\n      liable to You for damages, including any direct, indirect, special,\n      incidental, or consequential damages of any character arising as a\n      result of this License or out of the use or inability to use the\n      Work (including but not limited to damages for loss of goodwill,\n      work stoppage, computer failure or malfunction, or any and all\n      other commercial damages or losses), even if such Contributor\n      has been advised of the possibility of such damages.\n\n   9. Accepting Warranty or Additional Liability. While redistributing\n      the Work or Derivative Works thereof, You may choose to offer,\n      and charge a fee for, acceptance of support, warranty, indemnity,\n      or other liability obligations and/or rights consistent with this\n      License. However, in accepting such obligations, You may act only\n      on Your own behalf and on Your sole responsibility, not on behalf\n      of any other Contributor, and only if You agree to indemnify,\n      defend, and hold each Contributor harmless for any liability\n      incurred by, or claims asserted against, such Contributor by reason\n      of your accepting any such warranty or additional liability.\n\n   END OF TERMS AND CONDITIONS\n\n   APPENDIX: How to apply the Apache License to your work.\n\n      To apply the Apache License to your work, attach the following\n      boilerplate notice, with the fields enclosed by brackets \"[]\"\n      replaced with your own identifying information. (Don't include\n      the brackets!)  The text should be enclosed in the appropriate\n      comment syntax for the file format. We also recommend that a\n      file or class name and description of purpose be included on the\n      same \"printed page\" as the copyright notice for easier\n      identification within third-party archives.\n\n   Copyright [yyyy] [name of copyright owner]\n\n   Licensed under the Apache License, Version 2.0 (the \"License\");\n   you may not use this file except in compliance with the License.\n   You may obtain a copy of the License at\n\n       http://www.apache.org/licenses/LICENSE-2.0\n\n   Unless required by applicable law or agreed to in writing, software\n   distributed under the License is distributed on an \"AS IS\" BASIS,\n   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n   See the License for the specific language governing permissions and\n   limitations under the License.\n"
  },
  {
    "path": "pingora-header-serde/samples/test/1",
    "content": "HTTP/1.1 200 OK\nServer: nginx\nDate: Wed, 22 Dec 2021 06:30:29 GMT\nContent-Type: application/javascript\nLast-Modified: Mon, 29 Nov 2021 10:13:32 GMT\nTransfer-Encoding: chunked\nConnection: keep-alive\nVary: Accept-Encoding\nETag: W/\"61a4a7cc-21df8\"\nAccess-Control-Allow-Origin: *\nAccess-Control-Allow-Credentials: true\nAccess-Control-Expose-Headers: Content-Length,Content-Range\nAccess-Control-Allow-Headers: Range\nContent-Encoding: gzip\n\n"
  },
  {
    "path": "pingora-header-serde/samples/test/2",
    "content": "HTTP/1.1 200 OK\nServer: nginx\nDate: Thu, 23 Dec 2021 15:12:32 GMT\nContent-Type: application/javascript\nLast-Modified: Mon, 09 Sep 2019 12:47:14 GMT\nTransfer-Encoding: chunked\nConnection: keep-alive\nVary: Accept-Encoding\nETag: W/\"5d7649d2-16ec64\"\nAccess-Control-Allow-Origin: *\nAccess-Control-Allow-Credentials: true\nAccess-Control-Expose-Headers: Content-Length,Content-Range\nAccess-Control-Allow-Headers: Range\nContent-Encoding: gzip\n\n"
  },
  {
    "path": "pingora-header-serde/samples/test/3",
    "content": "HTTP/1.1 200 OK\nServer: nginx\nDate: Wed, 22 Dec 2021 12:29:00 GMT\nContent-Type: application/javascript\nLast-Modified: Mon, 09 Sep 2019 07:47:37 GMT\nTransfer-Encoding: chunked\nConnection: keep-alive\nVary: Accept-Encoding\nETag: W/\"5d760399-52868\"\nAccess-Control-Allow-Origin: *\nAccess-Control-Allow-Credentials: true\nAccess-Control-Expose-Headers: Content-Length,Content-Range\nAccess-Control-Allow-Headers: Range\nContent-Encoding: gzip\n\n"
  },
  {
    "path": "pingora-header-serde/samples/test/4",
    "content": "HTTP/1.1 200 OK\nServer: nginx\nDate: Wed, 22 Dec 2021 06:11:09 GMT\nContent-Type: application/javascript\nLast-Modified: Mon, 20 Dec 2021 01:23:10 GMT\nTransfer-Encoding: chunked\nConnection: keep-alive\nVary: Accept-Encoding\nETag: W/\"61bfdafe-21bc4\"\nAccess-Control-Allow-Origin: *\nAccess-Control-Allow-Credentials: true\nAccess-Control-Expose-Headers: Content-Length,Content-Range\nAccess-Control-Allow-Headers: Range\nContent-Encoding: gzip\n\n"
  },
  {
    "path": "pingora-header-serde/samples/test/5",
    "content": "HTTP/1.1 200 OK\nServer: nginx\nDate: Thu, 23 Dec 2021 15:23:29 GMT\nContent-Type: application/javascript\nLast-Modified: Sat, 09 Oct 2021 23:41:34 GMT\nTransfer-Encoding: chunked\nConnection: keep-alive\nVary: Accept-Encoding\nETag: W/\"616228ae-52054\"\nAccess-Control-Allow-Origin: *\nAccess-Control-Allow-Credentials: true\nAccess-Control-Expose-Headers: Content-Length,Content-Range\nAccess-Control-Allow-Headers: Range\nContent-Encoding: gzip\n\n"
  },
  {
    "path": "pingora-header-serde/samples/test/6",
    "content": "HTTP/1.1 200 OK\nServer: nginx\nDate: Wed, 22 Dec 2021 06:30:29 GMT\nContent-Type: application/javascript\nLast-Modified: Mon, 29 Nov 2021 10:13:32 GMT\nTransfer-Encoding: chunked\nConnection: keep-alive\nVary: Accept-Encoding\nETag: W/\"61a4a7cc-21df8\"\nAccess-Control-Allow-Origin: *\nAccess-Control-Allow-Credentials: true\nAccess-Control-Expose-Headers: Content-Length,Content-Range\nAccess-Control-Allow-Headers: Range\nContent-Encoding: gzip\n\n"
  },
  {
    "path": "pingora-header-serde/samples/test/7",
    "content": "HTTP/1.1 200 OK\nserver: nginx\ndate: Sat, 25 Dec 2021 03:05:35 GMT\ncontent-type: application/javascript\nlast-modified: Fri, 24 Dec 2021 04:20:01 GMT\ntransfer-encoding: chunked\nconnection: keep-alive\nvary: Accept-Encoding\netag: W/\"61c54a71-2d590\"\naccess-control-allow-origin: *\naccess-control-allow-credentials: true\naccess-control-expose-headers: Content-Length,Content-Range\naccess-control-allow-headers: Range\ncontent-encoding: gzip\n"
  },
  {
    "path": "pingora-header-serde/src/dict.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! Training to generate the zstd dictionary.\n\nuse std::fs;\nuse zstd::dict;\n\n/// Train the zstd dictionary from all the files under the given `dir_path`\n///\n/// The output will be the trained dictionary\npub fn train<P: AsRef<std::path::Path>>(dir_path: P) -> Vec<u8> {\n    // TODO: check f is file, it can be dir\n    let files = fs::read_dir(dir_path)\n        .unwrap()\n        .filter_map(|entry| entry.ok().map(|f| f.path()));\n    dict::from_files(files, 64 * 1024 * 1024).unwrap()\n}\n\n#[cfg(test)]\nmod test {\n    use super::*;\n    use crate::resp_header_to_buf;\n    use pingora_http::ResponseHeader;\n\n    fn gen_test_dict() -> Vec<u8> {\n        let mut path = std::path::PathBuf::from(env!(\"CARGO_MANIFEST_DIR\"));\n        path.push(\"samples/test\");\n        train(path)\n    }\n\n    fn gen_test_header() -> ResponseHeader {\n        let mut header = ResponseHeader::build(200, None).unwrap();\n        header\n            .append_header(\"Date\", \"Thu, 23 Dec 2021 11:23:29 GMT\")\n            .unwrap();\n        header\n            .append_header(\"Last-Modified\", \"Sat, 09 Oct 2021 22:41:34 GMT\")\n            .unwrap();\n        header.append_header(\"Connection\", \"keep-alive\").unwrap();\n        header.append_header(\"Vary\", \"Accept-encoding\").unwrap();\n        header.append_header(\"Content-Encoding\", \"gzip\").unwrap();\n        header\n            .append_header(\"Access-Control-Allow-Origin\", \"*\")\n            .unwrap();\n        header\n    }\n\n    #[test]\n    fn test_ser_with_dict() {\n        let dict = gen_test_dict();\n        let serde = crate::HeaderSerde::new(Some(dict));\n        let serde_no_dict = crate::HeaderSerde::new(None);\n        let header = gen_test_header();\n\n        let compressed = serde.serialize(&header).unwrap();\n        let compressed_no_dict = serde_no_dict.serialize(&header).unwrap();\n        let mut buf = vec![];\n        let uncompressed = resp_header_to_buf(&header, &mut buf);\n\n        assert!(compressed.len() < uncompressed);\n        assert!(compressed.len() < compressed_no_dict.len());\n    }\n\n    #[test]\n    fn test_deserialize_with_dict() {\n        let dict = gen_test_dict();\n        let serde = crate::HeaderSerde::new(Some(dict));\n        let serde_no_dict = crate::HeaderSerde::new(None);\n        let header = gen_test_header();\n\n        let compressed = serde.serialize(&header).unwrap();\n        let compressed_no_dict = serde_no_dict.serialize(&header).unwrap();\n\n        let from_dict_header = serde.deserialize(&compressed).unwrap();\n        let from_no_dict_header = serde_no_dict.deserialize(&compressed_no_dict).unwrap();\n\n        assert_eq!(from_dict_header.status, from_no_dict_header.status);\n        assert_eq!(from_dict_header.headers, from_no_dict_header.headers);\n    }\n\n    #[test]\n    fn test_ser_de_with_dict() {\n        let dict = gen_test_dict();\n        let serde = crate::HeaderSerde::new(Some(dict));\n        let header = gen_test_header();\n\n        let compressed = serde.serialize(&header).unwrap();\n        let header2 = serde.deserialize(&compressed).unwrap();\n\n        assert_eq!(header.status, header2.status);\n        assert_eq!(header.headers, header2.headers);\n    }\n}\n"
  },
  {
    "path": "pingora-header-serde/src/lib.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! HTTP Response header serialization with compression\n//!\n//! This crate is able to serialize http response header to about 1/3 of its original size (HTTP/1.1 wire format)\n//! with trained dictionary.\n\n#![warn(clippy::all)]\n#![allow(clippy::new_without_default)]\n#![allow(clippy::type_complexity)]\n\npub mod dict;\nmod thread_zstd;\n\nuse bytes::BufMut;\nuse http::Version;\nuse pingora_error::{Error, ErrorType, ImmutStr, Result};\nuse pingora_http::ResponseHeader;\nuse std::cell::RefCell;\nuse std::ops::DerefMut;\nuse thread_local::ThreadLocal;\n\n/// HTTP Response header serialization\n///\n/// This struct provides the APIs to convert HTTP response header into compressed wired format for\n/// storage.\npub struct HeaderSerde {\n    compression: ZstdCompression,\n    // internal buffer for uncompressed data to be compressed and vice versa\n    buf: ThreadLocal<RefCell<Vec<u8>>>,\n}\n\nconst MAX_HEADER_BUF_SIZE: usize = 128 * 1024; // 128KB\n\nconst COMPRESS_LEVEL: i32 = 3;\n\nimpl HeaderSerde {\n    /// Create a new [HeaderSerde]\n    ///\n    /// An optional zstd compression dictionary can be provided to improve the compression ratio\n    /// and speed. See [dict] for more details.\n    pub fn new(dict: Option<Vec<u8>>) -> Self {\n        if let Some(dict) = dict {\n            HeaderSerde {\n                compression: ZstdCompression::WithDict(thread_zstd::CompressionWithDict::new(\n                    &dict,\n                    COMPRESS_LEVEL,\n                )),\n                buf: ThreadLocal::new(),\n            }\n        } else {\n            HeaderSerde {\n                compression: ZstdCompression::Default(\n                    thread_zstd::Compression::new(),\n                    COMPRESS_LEVEL,\n                ),\n                buf: ThreadLocal::new(),\n            }\n        }\n    }\n\n    /// Serialize the given response header\n    pub fn serialize(&self, header: &ResponseHeader) -> Result<Vec<u8>> {\n        // for now we use HTTP 1.1 wire format for that\n        // TODO: should convert to h1 if the incoming header is for h2\n        let mut buf = self\n            .buf\n            .get_or(|| RefCell::new(Vec::with_capacity(MAX_HEADER_BUF_SIZE)))\n            .borrow_mut();\n        buf.clear(); // reset the buf\n        resp_header_to_buf(header, &mut buf);\n        self.compression.compress(&buf)\n    }\n\n    /// Deserialize the given response header\n    pub fn deserialize(&self, data: &[u8]) -> Result<ResponseHeader> {\n        let mut buf = self\n            .buf\n            .get_or(|| RefCell::new(Vec::with_capacity(MAX_HEADER_BUF_SIZE)))\n            .borrow_mut();\n        buf.clear(); // reset the buf\n        self.compression\n            .decompress_to_buffer(data, buf.deref_mut())?;\n        buf_to_http_header(&buf)\n    }\n}\n\n// Wrapper type to unify compressing with and withuot a dictionary,\n// since the two structs have different inputs for their APIs.\nenum ZstdCompression {\n    Default(thread_zstd::Compression, i32),\n    WithDict(thread_zstd::CompressionWithDict),\n}\n\n#[inline]\nfn into_error<S: Into<ImmutStr>>(e: &'static str, context: S) -> Box<Error> {\n    Error::because(ErrorType::InternalError, context, e)\n}\n\nimpl ZstdCompression {\n    fn compress(&self, data: &[u8]) -> Result<Vec<u8>> {\n        match &self {\n            ZstdCompression::Default(c, level) => c\n                .compress(data, *level)\n                .map_err(|e| into_error(e, \"compress header\")),\n            ZstdCompression::WithDict(c) => c\n                .compress(data)\n                .map_err(|e| into_error(e, \"compress header\")),\n        }\n    }\n\n    fn decompress_to_buffer(&self, source: &[u8], destination: &mut Vec<u8>) -> Result<usize> {\n        match &self {\n            ZstdCompression::Default(c, _) => {\n                c.decompress_to_buffer(source, destination).map_err(|e| {\n                    into_error(\n                        e,\n                        format!(\n                            \"decompress header, frame_content_size: {}\",\n                            get_frame_content_size(source)\n                        ),\n                    )\n                })\n            }\n            ZstdCompression::WithDict(c) => {\n                c.decompress_to_buffer(source, destination).map_err(|e| {\n                    into_error(\n                        e,\n                        format!(\n                            \"decompress header, frame_content_size: {}\",\n                            get_frame_content_size(source)\n                        ),\n                    )\n                })\n            }\n        }\n    }\n}\n\n#[inline]\nfn get_frame_content_size(source: &[u8]) -> ImmutStr {\n    match zstd_safe::get_frame_content_size(source) {\n        Ok(Some(size)) => match size {\n            zstd_safe::CONTENTSIZE_ERROR => ImmutStr::from(\"invalid\"),\n            zstd_safe::CONTENTSIZE_UNKNOWN => ImmutStr::from(\"unknown\"),\n            _ => ImmutStr::from(size.to_string()),\n        },\n        Ok(None) => ImmutStr::from(\"none\"),\n        Err(_e) => ImmutStr::from(\"failed\"),\n    }\n}\n\nconst CRLF: &[u8; 2] = b\"\\r\\n\";\n\n// Borrowed from pingora http1\n#[inline]\nfn resp_header_to_buf(resp: &ResponseHeader, buf: &mut Vec<u8>) -> usize {\n    // Status-Line\n    let version = match resp.version {\n        Version::HTTP_10 => \"HTTP/1.0 \",\n        Version::HTTP_11 => \"HTTP/1.1 \",\n        _ => \"HTTP/1.1 \", // store everything else (including h2) in http 1.1 format\n    };\n    buf.put_slice(version.as_bytes());\n    let status = resp.status;\n    buf.put_slice(status.as_str().as_bytes());\n    buf.put_u8(b' ');\n    let reason = status.canonical_reason();\n    if let Some(reason_buf) = reason {\n        buf.put_slice(reason_buf.as_bytes());\n    }\n    buf.put_slice(CRLF);\n\n    // headers\n    resp.header_to_h1_wire(buf);\n\n    buf.put_slice(CRLF);\n\n    buf.len()\n}\n\n// Should match pingora http1 setting\nconst MAX_HEADERS: usize = 256;\n\n#[inline]\nfn buf_to_http_header(buf: &[u8]) -> Result<ResponseHeader> {\n    let mut headers = vec![httparse::EMPTY_HEADER; MAX_HEADERS];\n    let mut resp = httparse::Response::new(&mut headers);\n\n    match resp.parse(buf) {\n        Ok(s) => match s {\n            httparse::Status::Complete(_size) => parsed_to_header(&resp),\n            // we always feed the but that contains the entire header to parse\n            _ => Error::e_explain(ErrorType::InternalError, \"incomplete uncompressed header\"),\n        },\n        Err(e) => Error::e_because(\n            ErrorType::InternalError,\n            format!(\n                \"parsing failed on uncompressed header, len={}, content={:?}\",\n                buf.len(),\n                String::from_utf8_lossy(buf)\n            ),\n            e,\n        ),\n    }\n}\n\n#[inline]\nfn parsed_to_header(parsed: &httparse::Response) -> Result<ResponseHeader> {\n    // code should always be there\n    // TODO: allow reading the parsed http version?\n    let mut resp = ResponseHeader::build(parsed.code.unwrap(), Some(parsed.headers.len()))?;\n\n    for header in parsed.headers.iter() {\n        resp.append_header(header.name.to_string(), header.value)?;\n    }\n\n    Ok(resp)\n}\n\n#[cfg(test)]\nmod tests {\n    use super::*;\n\n    #[test]\n    fn test_ser_wo_dict() {\n        let serde = HeaderSerde::new(None);\n        let mut header = ResponseHeader::build(200, None).unwrap();\n        header.append_header(\"foo\", \"bar\").unwrap();\n        header.append_header(\"foo\", \"barbar\").unwrap();\n        header.append_header(\"foo\", \"barbarbar\").unwrap();\n        header.append_header(\"Server\", \"Pingora\").unwrap();\n\n        let compressed = serde.serialize(&header).unwrap();\n        let mut buf = vec![];\n        let uncompressed = resp_header_to_buf(&header, &mut buf);\n        assert!(compressed.len() < uncompressed);\n    }\n\n    #[test]\n    fn test_ser_de_no_dict() {\n        let serde = HeaderSerde::new(None);\n        let mut header = ResponseHeader::build(200, None).unwrap();\n        header.append_header(\"foo1\", \"bar1\").unwrap();\n        header.append_header(\"foo2\", \"barbar2\").unwrap();\n        header.append_header(\"foo3\", \"barbarbar3\").unwrap();\n        header.append_header(\"Server\", \"Pingora\").unwrap();\n\n        let compressed = serde.serialize(&header).unwrap();\n        let header2 = serde.deserialize(&compressed).unwrap();\n        assert_eq!(header.status, header2.status);\n        assert_eq!(header.headers, header2.headers);\n    }\n\n    #[test]\n    fn test_no_headers() {\n        let serde = HeaderSerde::new(None);\n        let header = ResponseHeader::build(200, None).unwrap(); // No headers added\n\n        // Serialize and deserialize\n        let compressed = serde.serialize(&header).unwrap();\n        let header2 = serde.deserialize(&compressed).unwrap();\n\n        assert_eq!(header.status, header2.status);\n        assert_eq!(header.headers.len(), 0);\n        assert_eq!(header2.headers.len(), 0);\n    }\n\n    #[test]\n    fn test_empty_header_wire_format() {\n        let header = ResponseHeader::build(200, None).unwrap();\n        let mut buf = vec![];\n        resp_header_to_buf(&header, &mut buf);\n\n        // Should be: \"HTTP/1.1 200 OK\\r\\n\\r\\n\", total 19 bytes\n        assert_eq!(buf.len(), 19);\n        assert_eq!(buf, b\"HTTP/1.1 200 OK\\r\\n\\r\\n\");\n\n        // Test that httparse can handle this\n        let parsed = buf_to_http_header(&buf).unwrap();\n        assert_eq!(parsed.status.as_u16(), 200);\n    }\n}\n"
  },
  {
    "path": "pingora-header-serde/src/thread_zstd.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\nuse std::cell::{RefCell, RefMut};\nuse thread_local::ThreadLocal;\nuse zstd_safe::{CCtx, CDict, DCtx, DDict};\n\n/// Each thread will own its compression and decompression CTXes, and they share a single dict\n/// https://facebook.github.io/zstd/zstd_manual.html recommends to reuse ctx per thread\n\n// Both `Compression` and `CompressionWithDict` are just wrappers around the inner compression and\n// decompression contexts, but have different APIs to access it.\n\n#[derive(Default)]\npub struct Compression(CompressionInner);\n\n// these codes are inspired by zstd crate\n\nimpl Compression {\n    pub fn new() -> Self {\n        Compression(CompressionInner::new())\n    }\n\n    pub fn compress_to_buffer<C: zstd_safe::WriteBuf + ?Sized>(\n        &self,\n        source: &[u8],\n        destination: &mut C,\n        level: i32,\n    ) -> Result<usize, &'static str> {\n        self.0.compress_to_buffer(source, destination, level)\n    }\n\n    pub fn compress(&self, data: &[u8], level: i32) -> Result<Vec<u8>, &'static str> {\n        let mut buffer = make_compressed_data_buffer(data.len());\n        self.compress_to_buffer(data, &mut buffer, level)?;\n        Ok(buffer)\n    }\n\n    pub fn decompress_to_buffer<C: zstd_safe::WriteBuf + ?Sized>(\n        &self,\n        source: &[u8],\n        destination: &mut C,\n    ) -> Result<usize, &'static str> {\n        self.0.decompress_to_buffer(source, destination)\n    }\n}\n\npub struct CompressionWithDict {\n    inner: CompressionInner,\n    // these dictionaries are owned by this struct, hence the static lifetime\n    com_dict: CDict<'static>,\n    de_dict: DDict<'static>,\n}\n\nimpl CompressionWithDict {\n    pub fn new(dict: &[u8], compression_level: i32) -> Self {\n        CompressionWithDict {\n            inner: CompressionInner::new(),\n            // compression dictionary needs to be loaded ahead of time\n            // with the compression level\n            com_dict: CDict::create(dict, compression_level),\n            de_dict: DDict::create(dict),\n        }\n    }\n\n    pub fn compress_to_buffer<C: zstd_safe::WriteBuf + ?Sized>(\n        &self,\n        source: &[u8],\n        destination: &mut C,\n    ) -> Result<usize, &'static str> {\n        self.inner\n            .compress_to_buffer_using_dict(source, destination, &self.com_dict)\n    }\n\n    pub fn compress(&self, data: &[u8]) -> Result<Vec<u8>, &'static str> {\n        let mut buffer = make_compressed_data_buffer(data.len());\n        self.compress_to_buffer(data, &mut buffer)?;\n        Ok(buffer)\n    }\n\n    pub fn decompress_to_buffer<C: zstd_safe::WriteBuf + ?Sized>(\n        &self,\n        source: &[u8],\n        destination: &mut C,\n    ) -> Result<usize, &'static str> {\n        self.inner\n            .decompress_to_buffer_using_dict(source, destination, &self.de_dict)\n    }\n}\n\n#[derive(Default)]\nstruct CompressionInner {\n    com_context: ThreadLocal<RefCell<zstd_safe::CCtx<'static>>>,\n    de_context: ThreadLocal<RefCell<zstd_safe::DCtx<'static>>>,\n}\n\nimpl CompressionInner {\n    fn new() -> Self {\n        CompressionInner {\n            com_context: ThreadLocal::new(),\n            de_context: ThreadLocal::new(),\n        }\n    }\n\n    #[inline]\n    fn get_com_context(&self) -> RefMut<'_, CCtx<'static>> {\n        self.com_context\n            .get_or(|| RefCell::new(CCtx::create()))\n            .borrow_mut()\n    }\n\n    #[inline]\n    fn get_de_context(&self) -> RefMut<'_, DCtx<'static>> {\n        self.de_context\n            .get_or(|| RefCell::new(DCtx::create()))\n            .borrow_mut()\n    }\n\n    fn compress_to_buffer<C: zstd_safe::WriteBuf + ?Sized>(\n        &self,\n        source: &[u8],\n        destination: &mut C,\n        level: i32,\n    ) -> Result<usize, &'static str> {\n        self.get_com_context()\n            .compress(destination, source, level)\n            .map_err(zstd_safe::get_error_name)\n    }\n\n    fn decompress_to_buffer<C: zstd_safe::WriteBuf + ?Sized>(\n        &self,\n        source: &[u8],\n        destination: &mut C,\n    ) -> Result<usize, &'static str> {\n        self.get_de_context()\n            .decompress(destination, source)\n            .map_err(zstd_safe::get_error_name)\n    }\n\n    fn compress_to_buffer_using_dict<C: zstd_safe::WriteBuf + ?Sized>(\n        &self,\n        source: &[u8],\n        destination: &mut C,\n        dict: &CDict,\n    ) -> Result<usize, &'static str> {\n        self.get_com_context()\n            .compress_using_cdict(destination, source, dict)\n            .map_err(zstd_safe::get_error_name)\n    }\n\n    pub fn decompress_to_buffer_using_dict<C: zstd_safe::WriteBuf + ?Sized>(\n        &self,\n        source: &[u8],\n        destination: &mut C,\n        dict: &DDict,\n    ) -> Result<usize, &'static str> {\n        self.get_de_context()\n            .decompress_using_ddict(destination, source, dict)\n            .map_err(zstd_safe::get_error_name)\n    }\n}\n\n// Helper to create a buffer for the compressed data, preallocating enough\n// for the compressed size (given the size of the uncompressed data).\n#[inline]\nfn make_compressed_data_buffer(uncompressed_len: usize) -> Vec<u8> {\n    let buffer_len = zstd_safe::compress_bound(uncompressed_len);\n    Vec::with_capacity(buffer_len)\n}\n"
  },
  {
    "path": "pingora-header-serde/src/trainer.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\nuse pingora_header_serde::dict::train;\nuse std::env;\nuse std::io::{self, Write};\n\npub fn main() {\n    let args: Vec<String> = env::args().collect();\n    let dict = train(&args[1]);\n    io::stdout().write_all(&dict).unwrap();\n}\n"
  },
  {
    "path": "pingora-http/Cargo.toml",
    "content": "[package]\nname = \"pingora-http\"\nversion = \"0.8.0\"\nauthors = [\"Yuchen Wu <yuchen@cloudflare.com>\"]\nlicense = \"Apache-2.0\"\nedition = \"2021\"\nrepository = \"https://github.com/cloudflare/pingora\"\ncategories = [\"web-programming\"]\nkeywords = [\"http\", \"pingora\"]\ndescription = \"\"\"\nHTTP request and response header types for Pingora.\n\"\"\"\n\n# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html\n[lib]\nname = \"pingora_http\"\npath = \"src/lib.rs\"\n\n[dependencies]\nhttp = { workspace = true }\nbytes = { workspace = true }\npingora-error = { version = \"0.8.0\", path = \"../pingora-error\" }\n\n[features]\ndefault = []\npatched_http1 = []\n"
  },
  {
    "path": "pingora-http/LICENSE",
    "content": "\n                                 Apache License\n                           Version 2.0, January 2004\n                        http://www.apache.org/licenses/\n\n   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION\n\n   1. Definitions.\n\n      \"License\" shall mean the terms and conditions for use, reproduction,\n      and distribution as defined by Sections 1 through 9 of this document.\n\n      \"Licensor\" shall mean the copyright owner or entity authorized by\n      the copyright owner that is granting the License.\n\n      \"Legal Entity\" shall mean the union of the acting entity and all\n      other entities that control, are controlled by, or are under common\n      control with that entity. For the purposes of this definition,\n      \"control\" means (i) the power, direct or indirect, to cause the\n      direction or management of such entity, whether by contract or\n      otherwise, or (ii) ownership of fifty percent (50%) or more of the\n      outstanding shares, or (iii) beneficial ownership of such entity.\n\n      \"You\" (or \"Your\") shall mean an individual or Legal Entity\n      exercising permissions granted by this License.\n\n      \"Source\" form shall mean the preferred form for making modifications,\n      including but not limited to software source code, documentation\n      source, and configuration files.\n\n      \"Object\" form shall mean any form resulting from mechanical\n      transformation or translation of a Source form, including but\n      not limited to compiled object code, generated documentation,\n      and conversions to other media types.\n\n      \"Work\" shall mean the work of authorship, whether in Source or\n      Object form, made available under the License, as indicated by a\n      copyright notice that is included in or attached to the work\n      (an example is provided in the Appendix below).\n\n      \"Derivative Works\" shall mean any work, whether in Source or Object\n      form, that is based on (or derived from) the Work and for which the\n      editorial revisions, annotations, elaborations, or other modifications\n      represent, as a whole, an original work of authorship. For the purposes\n      of this License, Derivative Works shall not include works that remain\n      separable from, or merely link (or bind by name) to the interfaces of,\n      the Work and Derivative Works thereof.\n\n      \"Contribution\" shall mean any work of authorship, including\n      the original version of the Work and any modifications or additions\n      to that Work or Derivative Works thereof, that is intentionally\n      submitted to Licensor for inclusion in the Work by the copyright owner\n      or by an individual or Legal Entity authorized to submit on behalf of\n      the copyright owner. For the purposes of this definition, \"submitted\"\n      means any form of electronic, verbal, or written communication sent\n      to the Licensor or its representatives, including but not limited to\n      communication on electronic mailing lists, source code control systems,\n      and issue tracking systems that are managed by, or on behalf of, the\n      Licensor for the purpose of discussing and improving the Work, but\n      excluding communication that is conspicuously marked or otherwise\n      designated in writing by the copyright owner as \"Not a Contribution.\"\n\n      \"Contributor\" shall mean Licensor and any individual or Legal Entity\n      on behalf of whom a Contribution has been received by Licensor and\n      subsequently incorporated within the Work.\n\n   2. Grant of Copyright License. Subject to the terms and conditions of\n      this License, each Contributor hereby grants to You a perpetual,\n      worldwide, non-exclusive, no-charge, royalty-free, irrevocable\n      copyright license to reproduce, prepare Derivative Works of,\n      publicly display, publicly perform, sublicense, and distribute the\n      Work and such Derivative Works in Source or Object form.\n\n   3. Grant of Patent License. Subject to the terms and conditions of\n      this License, each Contributor hereby grants to You a perpetual,\n      worldwide, non-exclusive, no-charge, royalty-free, irrevocable\n      (except as stated in this section) patent license to make, have made,\n      use, offer to sell, sell, import, and otherwise transfer the Work,\n      where such license applies only to those patent claims licensable\n      by such Contributor that are necessarily infringed by their\n      Contribution(s) alone or by combination of their Contribution(s)\n      with the Work to which such Contribution(s) was submitted. If You\n      institute patent litigation against any entity (including a\n      cross-claim or counterclaim in a lawsuit) alleging that the Work\n      or a Contribution incorporated within the Work constitutes direct\n      or contributory patent infringement, then any patent licenses\n      granted to You under this License for that Work shall terminate\n      as of the date such litigation is filed.\n\n   4. Redistribution. You may reproduce and distribute copies of the\n      Work or Derivative Works thereof in any medium, with or without\n      modifications, and in Source or Object form, provided that You\n      meet the following conditions:\n\n      (a) You must give any other recipients of the Work or\n          Derivative Works a copy of this License; and\n\n      (b) You must cause any modified files to carry prominent notices\n          stating that You changed the files; and\n\n      (c) You must retain, in the Source form of any Derivative Works\n          that You distribute, all copyright, patent, trademark, and\n          attribution notices from the Source form of the Work,\n          excluding those notices that do not pertain to any part of\n          the Derivative Works; and\n\n      (d) If the Work includes a \"NOTICE\" text file as part of its\n          distribution, then any Derivative Works that You distribute must\n          include a readable copy of the attribution notices contained\n          within such NOTICE file, excluding those notices that do not\n          pertain to any part of the Derivative Works, in at least one\n          of the following places: within a NOTICE text file distributed\n          as part of the Derivative Works; within the Source form or\n          documentation, if provided along with the Derivative Works; or,\n          within a display generated by the Derivative Works, if and\n          wherever such third-party notices normally appear. The contents\n          of the NOTICE file are for informational purposes only and\n          do not modify the License. You may add Your own attribution\n          notices within Derivative Works that You distribute, alongside\n          or as an addendum to the NOTICE text from the Work, provided\n          that such additional attribution notices cannot be construed\n          as modifying the License.\n\n      You may add Your own copyright statement to Your modifications and\n      may provide additional or different license terms and conditions\n      for use, reproduction, or distribution of Your modifications, or\n      for any such Derivative Works as a whole, provided Your use,\n      reproduction, and distribution of the Work otherwise complies with\n      the conditions stated in this License.\n\n   5. Submission of Contributions. Unless You explicitly state otherwise,\n      any Contribution intentionally submitted for inclusion in the Work\n      by You to the Licensor shall be under the terms and conditions of\n      this License, without any additional terms or conditions.\n      Notwithstanding the above, nothing herein shall supersede or modify\n      the terms of any separate license agreement you may have executed\n      with Licensor regarding such Contributions.\n\n   6. Trademarks. This License does not grant permission to use the trade\n      names, trademarks, service marks, or product names of the Licensor,\n      except as required for reasonable and customary use in describing the\n      origin of the Work and reproducing the content of the NOTICE file.\n\n   7. Disclaimer of Warranty. Unless required by applicable law or\n      agreed to in writing, Licensor provides the Work (and each\n      Contributor provides its Contributions) on an \"AS IS\" BASIS,\n      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or\n      implied, including, without limitation, any warranties or conditions\n      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A\n      PARTICULAR PURPOSE. You are solely responsible for determining the\n      appropriateness of using or redistributing the Work and assume any\n      risks associated with Your exercise of permissions under this License.\n\n   8. Limitation of Liability. In no event and under no legal theory,\n      whether in tort (including negligence), contract, or otherwise,\n      unless required by applicable law (such as deliberate and grossly\n      negligent acts) or agreed to in writing, shall any Contributor be\n      liable to You for damages, including any direct, indirect, special,\n      incidental, or consequential damages of any character arising as a\n      result of this License or out of the use or inability to use the\n      Work (including but not limited to damages for loss of goodwill,\n      work stoppage, computer failure or malfunction, or any and all\n      other commercial damages or losses), even if such Contributor\n      has been advised of the possibility of such damages.\n\n   9. Accepting Warranty or Additional Liability. While redistributing\n      the Work or Derivative Works thereof, You may choose to offer,\n      and charge a fee for, acceptance of support, warranty, indemnity,\n      or other liability obligations and/or rights consistent with this\n      License. However, in accepting such obligations, You may act only\n      on Your own behalf and on Your sole responsibility, not on behalf\n      of any other Contributor, and only if You agree to indemnify,\n      defend, and hold each Contributor harmless for any liability\n      incurred by, or claims asserted against, such Contributor by reason\n      of your accepting any such warranty or additional liability.\n\n   END OF TERMS AND CONDITIONS\n\n   APPENDIX: How to apply the Apache License to your work.\n\n      To apply the Apache License to your work, attach the following\n      boilerplate notice, with the fields enclosed by brackets \"[]\"\n      replaced with your own identifying information. (Don't include\n      the brackets!)  The text should be enclosed in the appropriate\n      comment syntax for the file format. We also recommend that a\n      file or class name and description of purpose be included on the\n      same \"printed page\" as the copyright notice for easier\n      identification within third-party archives.\n\n   Copyright [yyyy] [name of copyright owner]\n\n   Licensed under the Apache License, Version 2.0 (the \"License\");\n   you may not use this file except in compliance with the License.\n   You may obtain a copy of the License at\n\n       http://www.apache.org/licenses/LICENSE-2.0\n\n   Unless required by applicable law or agreed to in writing, software\n   distributed under the License is distributed on an \"AS IS\" BASIS,\n   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n   See the License for the specific language governing permissions and\n   limitations under the License.\n"
  },
  {
    "path": "pingora-http/src/case_header_name.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\nuse crate::*;\nuse bytes::Bytes;\nuse http::header;\n\n#[derive(Debug, Clone)]\npub struct CaseHeaderName(Bytes);\n\nimpl CaseHeaderName {\n    pub fn new(name: String) -> Self {\n        CaseHeaderName(name.into())\n    }\n}\n\nimpl CaseHeaderName {\n    pub fn as_slice(&self) -> &[u8] {\n        &self.0\n    }\n\n    pub fn from_slice(buf: &[u8]) -> Self {\n        CaseHeaderName(Bytes::copy_from_slice(buf))\n    }\n}\n\n/// A trait that converts into case-sensitive header names.\npub trait IntoCaseHeaderName {\n    fn into_case_header_name(self) -> CaseHeaderName;\n}\n\nimpl IntoCaseHeaderName for CaseHeaderName {\n    fn into_case_header_name(self) -> CaseHeaderName {\n        self\n    }\n}\n\nimpl IntoCaseHeaderName for String {\n    fn into_case_header_name(self) -> CaseHeaderName {\n        CaseHeaderName(self.into())\n    }\n}\n\nimpl IntoCaseHeaderName for &'static str {\n    fn into_case_header_name(self) -> CaseHeaderName {\n        CaseHeaderName(self.into())\n    }\n}\n\nimpl IntoCaseHeaderName for HeaderName {\n    fn into_case_header_name(self) -> CaseHeaderName {\n        CaseHeaderName(titled_header_name(&self))\n    }\n}\n\nimpl IntoCaseHeaderName for &HeaderName {\n    fn into_case_header_name(self) -> CaseHeaderName {\n        CaseHeaderName(titled_header_name(self))\n    }\n}\n\nimpl IntoCaseHeaderName for Bytes {\n    fn into_case_header_name(self) -> CaseHeaderName {\n        CaseHeaderName(self)\n    }\n}\n\nfn titled_header_name(header_name: &HeaderName) -> Bytes {\n    titled_header_name_str(header_name).map_or_else(\n        || Bytes::copy_from_slice(header_name.as_str().as_bytes()),\n        |s| Bytes::from_static(s.as_bytes()),\n    )\n}\n\npub(crate) fn titled_header_name_str(header_name: &HeaderName) -> Option<&'static str> {\n    Some(match *header_name {\n        header::ACCEPT_RANGES => \"Accept-Ranges\",\n        header::AGE => \"Age\",\n        header::CACHE_CONTROL => \"Cache-Control\",\n        header::CONNECTION => \"Connection\",\n        header::CONTENT_TYPE => \"Content-Type\",\n        header::CONTENT_ENCODING => \"Content-Encoding\",\n        header::CONTENT_LENGTH => \"Content-Length\",\n        header::DATE => \"Date\",\n        header::TRANSFER_ENCODING => \"Transfer-Encoding\",\n        header::HOST => \"Host\",\n        header::SERVER => \"Server\",\n        header::SET_COOKIE => \"Set-Cookie\",\n        // TODO: add more const header here to map to their titled case\n        // TODO: automatically upper case the first letter?\n        _ => {\n            return None;\n        }\n    })\n}\n\n#[cfg(test)]\nmod tests {\n    use super::*;\n\n    #[test]\n    fn test_case_header_name() {\n        assert_eq!(\"FoO\".into_case_header_name().as_slice(), b\"FoO\");\n        assert_eq!(\"FoO\".to_string().into_case_header_name().as_slice(), b\"FoO\");\n        assert_eq!(header::SERVER.into_case_header_name().as_slice(), b\"Server\");\n    }\n}\n"
  },
  {
    "path": "pingora-http/src/lib.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! HTTP header objects that preserve http header cases\n//!\n//! Although HTTP header names are supposed to be case-insensitive for compatibility, proxies\n//! ideally shouldn't alter the HTTP traffic, especially the headers they don't need to read.\n//!\n//! This crate provide structs and methods to preserve the headers in order to build a transparent\n//! proxy.\n\n#![allow(clippy::new_without_default)]\n\nuse bytes::BufMut;\nuse http::header::{AsHeaderName, HeaderName, HeaderValue};\nuse http::request::Builder as ReqBuilder;\nuse http::request::Parts as ReqParts;\nuse http::response::Builder as RespBuilder;\nuse http::response::Parts as RespParts;\nuse http::uri::Uri;\nuse pingora_error::{ErrorType::*, OrErr, Result};\nuse std::ops::{Deref, DerefMut};\n\npub use http::method::Method;\npub use http::status::StatusCode;\npub use http::version::Version;\npub use http::HeaderMap as HMap;\n\nmod case_header_name;\nuse case_header_name::CaseHeaderName;\npub use case_header_name::IntoCaseHeaderName;\n\npub mod prelude {\n    pub use crate::RequestHeader;\n    pub use crate::ResponseHeader;\n}\n\n/* an ordered header map to store the original case of each header name\nHMap({\n    \"foo\": [\"Foo\", \"foO\", \"FoO\"]\n})\nThe order how HeaderMap iter over its items is \"arbitrary, but consistent\".\nHopefully this property makes sure this map of header names always iterates in the\nsame order of the map of header values.\nThis idea is inspaired by hyper @nox\n*/\ntype CaseMap = HMap<CaseHeaderName>;\n\npub enum HeaderNameVariant<'a> {\n    Case(&'a CaseHeaderName),\n    Titled(&'a str),\n}\n\n/// The HTTP request header type.\n///\n/// This type is similar to [http::request::Parts] but preserves header name case.\n/// It also preserves request path even if it is not UTF-8.\n///\n/// [RequestHeader] implements [Deref] for [http::request::Parts] so it can be used as it in most\n/// places.\n#[derive(Debug)]\npub struct RequestHeader {\n    base: ReqParts,\n    header_name_map: Option<CaseMap>,\n    // store the raw path bytes only if it is invalid utf-8\n    raw_path_fallback: Vec<u8>, // can also be Box<[u8]>\n    // whether we send END_STREAM with HEADERS for h2 requests\n    send_end_stream: bool,\n}\n\nimpl AsRef<ReqParts> for RequestHeader {\n    fn as_ref(&self) -> &ReqParts {\n        &self.base\n    }\n}\n\nimpl Deref for RequestHeader {\n    type Target = ReqParts;\n\n    fn deref(&self) -> &Self::Target {\n        &self.base\n    }\n}\n\nimpl DerefMut for RequestHeader {\n    fn deref_mut(&mut self) -> &mut Self::Target {\n        &mut self.base\n    }\n}\n\nimpl RequestHeader {\n    fn new_no_case(size_hint: Option<usize>) -> Self {\n        let mut base = ReqBuilder::new().body(()).unwrap().into_parts().0;\n        base.headers.reserve(http_header_map_upper_bound(size_hint));\n        RequestHeader {\n            base,\n            header_name_map: None,\n            raw_path_fallback: vec![],\n            send_end_stream: true,\n        }\n    }\n\n    /// Create a new [RequestHeader] with the given method and path.\n    ///\n    /// The `path` can be non UTF-8.\n    pub fn build(\n        method: impl TryInto<Method>,\n        path: &[u8],\n        size_hint: Option<usize>,\n    ) -> Result<Self> {\n        let mut req = Self::build_no_case(method, path, size_hint)?;\n        req.header_name_map = Some(CaseMap::with_capacity(http_header_map_upper_bound(\n            size_hint,\n        )));\n        Ok(req)\n    }\n\n    /// Create a new [RequestHeader] with the given method and path without preserving header case.\n    ///\n    /// A [RequestHeader] created from this type is more space efficient than those from [Self::build()].\n    ///\n    /// Use this method if reading from or writing to HTTP/2 sessions where header case doesn't matter anyway.\n    pub fn build_no_case(\n        method: impl TryInto<Method>,\n        path: &[u8],\n        size_hint: Option<usize>,\n    ) -> Result<Self> {\n        let mut req = Self::new_no_case(size_hint);\n        req.base.method = method\n            .try_into()\n            .explain_err(InvalidHTTPHeader, |_| \"invalid method\")?;\n        req.set_raw_path(path)?;\n        Ok(req)\n    }\n\n    /// Append the header name and value to `self`.\n    ///\n    /// If there are already some headers under the same name, a new value will be added without\n    /// any others being removed.\n    pub fn append_header(\n        &mut self,\n        name: impl IntoCaseHeaderName,\n        value: impl TryInto<HeaderValue>,\n    ) -> Result<bool> {\n        let header_value = value\n            .try_into()\n            .explain_err(InvalidHTTPHeader, |_| \"invalid value while append\")?;\n        append_header_value(\n            self.header_name_map.as_mut(),\n            &mut self.base.headers,\n            name,\n            header_value,\n        )\n    }\n\n    /// Insert the header name and value to `self`.\n    ///\n    /// Different from [Self::append_header()], this method will replace all other existing headers\n    /// under the same name (case-insensitive).\n    pub fn insert_header(\n        &mut self,\n        name: impl IntoCaseHeaderName,\n        value: impl TryInto<HeaderValue>,\n    ) -> Result<()> {\n        let header_value = value\n            .try_into()\n            .explain_err(InvalidHTTPHeader, |_| \"invalid value while insert\")?;\n        insert_header_value(\n            self.header_name_map.as_mut(),\n            &mut self.base.headers,\n            name,\n            header_value,\n        )\n    }\n\n    /// Remove all headers under the name\n    pub fn remove_header<'a, N: ?Sized>(&mut self, name: &'a N) -> Option<HeaderValue>\n    where\n        &'a N: 'a + AsHeaderName,\n    {\n        remove_header(self.header_name_map.as_mut(), &mut self.base.headers, name)\n    }\n\n    /// Write the header to the `buf` in HTTP/1.1 wire format.\n    ///\n    /// The header case will be preserved.\n    pub fn header_to_h1_wire(&self, buf: &mut impl BufMut) {\n        header_to_h1_wire(self.header_name_map.as_ref(), &self.base.headers, buf)\n    }\n\n    /// If case sensitivity is enabled, returns an iterator to iterate over case-sensitive header names and values.\n    /// Otherwise returns an empty iterator.\n    ///\n    /// Headers of the same name are visited in insertion order.\n    pub fn case_header_iter(&self) -> impl Iterator<Item = (&CaseHeaderName, &HeaderValue)> + '_ {\n        case_header_iter(self.header_name_map.as_ref(), &self.base.headers)\n    }\n\n    /// Returns true if the request has case-sensitive headers.\n    pub fn has_case(&self) -> bool {\n        self.header_name_map.is_some()\n    }\n\n    pub fn map<F: FnMut(HeaderNameVariant, &HeaderValue) -> Result<()>>(\n        &self,\n        mut f: F,\n    ) -> Result<()> {\n        let key_map = self.header_name_map.as_ref();\n        let value_map = &self.base.headers;\n\n        if let Some(key_map) = key_map {\n            let iter = key_map.iter().zip(value_map.iter());\n            for ((header, case_header), (header2, val)) in iter {\n                if header != header2 {\n                    // in case the header iteration order changes in future versions of HMap\n                    panic!(\"header iter mismatch {}, {}\", header, header2)\n                }\n                f(HeaderNameVariant::Case(case_header), val)?;\n            }\n        } else {\n            for (header, value) in value_map {\n                let titled_header =\n                    case_header_name::titled_header_name_str(header).unwrap_or(header.as_str());\n                f(HeaderNameVariant::Titled(titled_header), value)?;\n            }\n        }\n\n        Ok(())\n    }\n\n    /// Set the request method\n    pub fn set_method(&mut self, method: Method) {\n        self.base.method = method;\n    }\n\n    /// Set the request URI\n    pub fn set_uri(&mut self, uri: http::Uri) {\n        self.base.uri = uri;\n        // Clear out raw_path_fallback, or it will be used when serializing\n        self.raw_path_fallback = vec![];\n    }\n\n    /// Set the request URI directly via raw bytes.\n    ///\n    /// Generally prefer [Self::set_uri()] to modify the header's URI if able.\n    ///\n    /// This API is to allow supporting non UTF-8 cases.\n    pub fn set_raw_path(&mut self, path: &[u8]) -> Result<()> {\n        if let Ok(p) = std::str::from_utf8(path) {\n            let uri = Uri::builder()\n                .path_and_query(p)\n                .build()\n                .explain_err(InvalidHTTPHeader, |_| format!(\"invalid uri {}\", p))?;\n            self.base.uri = uri;\n            // keep raw_path empty, no need to store twice\n        } else {\n            // put a valid utf-8 path into base for read only access\n            let lossy_str = String::from_utf8_lossy(path);\n            let uri = Uri::builder()\n                .path_and_query(lossy_str.as_ref())\n                .build()\n                .explain_err(InvalidHTTPHeader, |_| format!(\"invalid uri {}\", lossy_str))?;\n            self.base.uri = uri;\n            self.raw_path_fallback = path.to_vec();\n        }\n        Ok(())\n    }\n\n    /// Set whether we send an END_STREAM on H2 request HEADERS if body is empty.\n    pub fn set_send_end_stream(&mut self, send_end_stream: bool) {\n        self.send_end_stream = send_end_stream;\n    }\n\n    /// Returns if we support sending an END_STREAM on H2 request HEADERS if body is empty,\n    /// returns None if not H2.\n    pub fn send_end_stream(&self) -> Option<bool> {\n        if self.base.version != Version::HTTP_2 {\n            return None;\n        }\n        Some(self.send_end_stream)\n    }\n\n    /// Return the request path in its raw format\n    ///\n    /// Non-UTF8 is supported.\n    pub fn raw_path(&self) -> &[u8] {\n        if !self.raw_path_fallback.is_empty() {\n            &self.raw_path_fallback\n        } else {\n            // Url should always be set\n            self.base\n                .uri\n                .path_and_query()\n                .as_ref()\n                .unwrap()\n                .as_str()\n                .as_bytes()\n        }\n    }\n\n    /// Return the file extension of the path\n    pub fn uri_file_extension(&self) -> Option<&str> {\n        // get everything after the last '.' in path\n        let (_, ext) = self\n            .uri\n            .path_and_query()\n            .and_then(|pq| pq.path().rsplit_once('.'))?;\n        Some(ext)\n    }\n\n    /// Set http version\n    pub fn set_version(&mut self, version: Version) {\n        self.base.version = version;\n    }\n\n    /// Clone `self` into [http::request::Parts].\n    pub fn as_owned_parts(&self) -> ReqParts {\n        clone_req_parts(&self.base)\n    }\n}\n\nimpl Clone for RequestHeader {\n    fn clone(&self) -> Self {\n        Self {\n            base: self.as_owned_parts(),\n            header_name_map: self.header_name_map.clone(),\n            raw_path_fallback: self.raw_path_fallback.clone(),\n            send_end_stream: self.send_end_stream,\n        }\n    }\n}\n\n// The `RequestHeader` will be the no case variant, because `ReqParts` keeps no header case\nimpl From<ReqParts> for RequestHeader {\n    fn from(parts: ReqParts) -> RequestHeader {\n        Self {\n            base: parts,\n            header_name_map: None,\n            // no illegal path\n            raw_path_fallback: vec![],\n            send_end_stream: true,\n        }\n    }\n}\n\nimpl From<RequestHeader> for ReqParts {\n    fn from(resp: RequestHeader) -> ReqParts {\n        resp.base\n    }\n}\n\n/// The HTTP response header type.\n///\n/// This type is similar to [http::response::Parts] but preserves header name case.\n/// [ResponseHeader] implements [Deref] for [http::response::Parts] so it can be used as it in most\n/// places.\n#[derive(Debug)]\npub struct ResponseHeader {\n    base: RespParts,\n    // an ordered header map to store the original case of each header name\n    header_name_map: Option<CaseMap>,\n    // the reason phrase of the response, if unset, a default one will be used\n    reason_phrase: Option<String>,\n}\n\nimpl AsRef<RespParts> for ResponseHeader {\n    fn as_ref(&self) -> &RespParts {\n        &self.base\n    }\n}\n\nimpl Deref for ResponseHeader {\n    type Target = RespParts;\n\n    fn deref(&self) -> &Self::Target {\n        &self.base\n    }\n}\n\nimpl DerefMut for ResponseHeader {\n    fn deref_mut(&mut self) -> &mut Self::Target {\n        &mut self.base\n    }\n}\n\nimpl Clone for ResponseHeader {\n    fn clone(&self) -> Self {\n        Self {\n            base: self.as_owned_parts(),\n            header_name_map: self.header_name_map.clone(),\n            reason_phrase: self.reason_phrase.clone(),\n        }\n    }\n}\n\n// The `ResponseHeader` will be the no case variant, because `RespParts` keeps no header case\nimpl From<RespParts> for ResponseHeader {\n    fn from(parts: RespParts) -> ResponseHeader {\n        Self {\n            base: parts,\n            header_name_map: None,\n            reason_phrase: None,\n        }\n    }\n}\n\nimpl From<ResponseHeader> for RespParts {\n    fn from(resp: ResponseHeader) -> RespParts {\n        resp.base\n    }\n}\n\nimpl From<Box<ResponseHeader>> for Box<RespParts> {\n    fn from(resp: Box<ResponseHeader>) -> Box<RespParts> {\n        Box::new(resp.base)\n    }\n}\n\nimpl ResponseHeader {\n    fn new(size_hint: Option<usize>) -> Self {\n        let mut resp_header = Self::new_no_case(size_hint);\n        resp_header.header_name_map = Some(CaseMap::with_capacity(http_header_map_upper_bound(\n            size_hint,\n        )));\n        resp_header\n    }\n\n    fn new_no_case(size_hint: Option<usize>) -> Self {\n        let mut base = RespBuilder::new().body(()).unwrap().into_parts().0;\n        base.headers.reserve(http_header_map_upper_bound(size_hint));\n        ResponseHeader {\n            base,\n            header_name_map: None,\n            reason_phrase: None,\n        }\n    }\n\n    /// Create a new [ResponseHeader] with the given status code.\n    pub fn build(code: impl TryInto<StatusCode>, size_hint: Option<usize>) -> Result<Self> {\n        let mut resp = Self::new(size_hint);\n        resp.base.status = code\n            .try_into()\n            .explain_err(InvalidHTTPHeader, |_| \"invalid status\")?;\n        Ok(resp)\n    }\n\n    /// Create a new [ResponseHeader] with the given status code without preserving header case.\n    ///\n    /// A [ResponseHeader] created from this type is more space efficient than those from [Self::build()].\n    ///\n    /// Use this method if reading from or writing to HTTP/2 sessions where header case doesn't matter anyway.\n    pub fn build_no_case(code: impl TryInto<StatusCode>, size_hint: Option<usize>) -> Result<Self> {\n        let mut resp = Self::new_no_case(size_hint);\n        resp.base.status = code\n            .try_into()\n            .explain_err(InvalidHTTPHeader, |_| \"invalid status\")?;\n        Ok(resp)\n    }\n\n    /// Append the header name and value to `self`.\n    ///\n    /// If there are already some headers under the same name, a new value will be added without\n    /// any others being removed.\n    pub fn append_header(\n        &mut self,\n        name: impl IntoCaseHeaderName,\n        value: impl TryInto<HeaderValue>,\n    ) -> Result<bool> {\n        let header_value = value\n            .try_into()\n            .explain_err(InvalidHTTPHeader, |_| \"invalid value while append\")?;\n        append_header_value(\n            self.header_name_map.as_mut(),\n            &mut self.base.headers,\n            name,\n            header_value,\n        )\n    }\n\n    /// Insert the header name and value to `self`.\n    ///\n    /// Different from [Self::append_header()], this method will replace all other existing headers\n    /// under the same name (case insensitive).\n    pub fn insert_header(\n        &mut self,\n        name: impl IntoCaseHeaderName,\n        value: impl TryInto<HeaderValue>,\n    ) -> Result<()> {\n        let header_value = value\n            .try_into()\n            .explain_err(InvalidHTTPHeader, |_| \"invalid value while insert\")?;\n        insert_header_value(\n            self.header_name_map.as_mut(),\n            &mut self.base.headers,\n            name,\n            header_value,\n        )\n    }\n\n    /// Remove all headers under the name\n    pub fn remove_header<'a, N: ?Sized>(&mut self, name: &'a N) -> Option<HeaderValue>\n    where\n        &'a N: 'a + AsHeaderName,\n    {\n        remove_header(self.header_name_map.as_mut(), &mut self.base.headers, name)\n    }\n\n    /// Write the header to the `buf` in HTTP/1.1 wire format.\n    ///\n    /// The header case will be preserved.\n    pub fn header_to_h1_wire(&self, buf: &mut impl BufMut) {\n        header_to_h1_wire(self.header_name_map.as_ref(), &self.base.headers, buf)\n    }\n\n    /// If case sensitivity is enabled, returns an iterator to iterate over case-sensitive header names and values.\n    /// Otherwise returns an empty iterator.\n    ///\n    /// Headers of the same name are visited in insertion order.\n    pub fn case_header_iter(&self) -> impl Iterator<Item = (&CaseHeaderName, &HeaderValue)> + '_ {\n        case_header_iter(self.header_name_map.as_ref(), &self.base.headers)\n    }\n\n    /// Returns true if the response has case-sensitive headers.\n    pub fn has_case(&self) -> bool {\n        self.header_name_map.is_some()\n    }\n\n    pub fn map<F: FnMut(HeaderNameVariant, &HeaderValue) -> Result<()>>(\n        &self,\n        mut f: F,\n    ) -> Result<()> {\n        let key_map = self.header_name_map.as_ref();\n        let value_map = &self.base.headers;\n\n        if let Some(key_map) = key_map {\n            let iter = key_map.iter().zip(value_map.iter());\n            for ((header, case_header), (header2, val)) in iter {\n                if header != header2 {\n                    // in case the header iteration order changes in future versions of HMap\n                    panic!(\"header iter mismatch {}, {}\", header, header2)\n                }\n                f(HeaderNameVariant::Case(case_header), val)?;\n            }\n        } else {\n            for (header, value) in value_map {\n                let titled_header =\n                    case_header_name::titled_header_name_str(header).unwrap_or(header.as_str());\n                f(HeaderNameVariant::Titled(titled_header), value)?;\n            }\n        }\n\n        Ok(())\n    }\n\n    /// Set the status code\n    pub fn set_status(&mut self, status: impl TryInto<StatusCode>) -> Result<()> {\n        self.base.status = status\n            .try_into()\n            .explain_err(InvalidHTTPHeader, |_| \"invalid status\")?;\n        Ok(())\n    }\n\n    /// Set the HTTP version\n    pub fn set_version(&mut self, version: Version) {\n        self.base.version = version\n    }\n\n    /// Set the HTTP reason phase. If `None`, a default reason phase will be used\n    pub fn set_reason_phrase(&mut self, reason_phrase: Option<&str>) -> Result<()> {\n        // No need to allocate memory to store the phrase if it is the default one.\n        if reason_phrase == self.base.status.canonical_reason() {\n            self.reason_phrase = None;\n            return Ok(());\n        }\n\n        // TODO: validate it \"*( HTAB / SP / VCHAR / obs-text )\"\n        self.reason_phrase = reason_phrase.map(str::to_string);\n        Ok(())\n    }\n\n    /// Get the HTTP reason phase. If [Self::set_reason_phrase()] is never called\n    /// or set to `None`, a default reason phase will be used\n    pub fn get_reason_phrase(&self) -> Option<&str> {\n        self.reason_phrase\n            .as_deref()\n            .or_else(|| self.base.status.canonical_reason())\n    }\n\n    /// Clone `self` into [http::response::Parts].\n    pub fn as_owned_parts(&self) -> RespParts {\n        clone_resp_parts(&self.base)\n    }\n\n    /// Helper function to set the HTTP content length on the response header.\n    pub fn set_content_length(&mut self, len: usize) -> Result<()> {\n        self.insert_header(http::header::CONTENT_LENGTH, len)\n    }\n}\n\nfn clone_req_parts(me: &ReqParts) -> ReqParts {\n    let mut parts = ReqBuilder::new()\n        .method(me.method.clone())\n        .uri(me.uri.clone())\n        .version(me.version)\n        .body(())\n        .unwrap()\n        .into_parts()\n        .0;\n    parts.headers = me.headers.clone();\n    parts.extensions = me.extensions.clone();\n    parts\n}\n\nfn clone_resp_parts(me: &RespParts) -> RespParts {\n    let mut parts = RespBuilder::new()\n        .status(me.status)\n        .version(me.version)\n        .body(())\n        .unwrap()\n        .into_parts()\n        .0;\n    parts.headers = me.headers.clone();\n    parts.extensions = me.extensions.clone();\n    parts\n}\n\n// This function returns an upper bound on the size of the header map used inside the http crate.\n// As of version 0.2, there is a limit of 1 << 15 (32,768) items inside the map. There is an\n// assertion against this size inside the crate, so we want to avoid panicking by not exceeding this\n// upper bound.\nfn http_header_map_upper_bound(size_hint: Option<usize>) -> usize {\n    // Even though the crate has 1 << 15 as the max size, calls to `with_capacity` invoke a\n    // function that returns the size + size / 3.\n    //\n    // See https://github.com/hyperium/http/blob/34a9d6bdab027948d6dea3b36d994f9cbaf96f75/src/header/map.rs#L3220\n    //\n    // Therefore we set our max size to be even lower, so we guarantee ourselves we won't hit that\n    // upper bound in the crate. Any way you cut it, 4,096 headers is insane.\n    const PINGORA_MAX_HEADER_COUNT: usize = 4096;\n    const INIT_HEADER_SIZE: usize = 8;\n\n    // We select the size hint or the max size here, ensuring that we pick a value substantially lower\n    // than 1 << 15 with room to grow the header map.\n    std::cmp::min(\n        size_hint.unwrap_or(INIT_HEADER_SIZE),\n        PINGORA_MAX_HEADER_COUNT,\n    )\n}\n\n#[inline]\nfn append_header_value<T>(\n    name_map: Option<&mut CaseMap>,\n    value_map: &mut HMap<T>,\n    name: impl IntoCaseHeaderName,\n    value: T,\n) -> Result<bool> {\n    let case_header_name = name.into_case_header_name();\n    let header_name: HeaderName = case_header_name\n        .as_slice()\n        .try_into()\n        .or_err(InvalidHTTPHeader, \"invalid header name\")?;\n    // store the original case in the map\n    if let Some(name_map) = name_map {\n        name_map.append(header_name.clone(), case_header_name);\n    }\n\n    Ok(value_map.append(header_name, value))\n}\n\n#[inline]\nfn insert_header_value<T>(\n    name_map: Option<&mut CaseMap>,\n    value_map: &mut HMap<T>,\n    name: impl IntoCaseHeaderName,\n    value: T,\n) -> Result<()> {\n    let case_header_name = name.into_case_header_name();\n    let header_name: HeaderName = case_header_name\n        .as_slice()\n        .try_into()\n        .or_err(InvalidHTTPHeader, \"invalid header name\")?;\n    if let Some(name_map) = name_map {\n        // store the original case in the map\n        name_map.insert(header_name.clone(), case_header_name);\n    }\n    value_map.insert(header_name, value);\n    Ok(())\n}\n\n// the &N here is to avoid clone(). None Copy type like String can impl AsHeaderName\n#[inline]\nfn remove_header<'a, T, N: ?Sized>(\n    name_map: Option<&mut CaseMap>,\n    value_map: &mut HMap<T>,\n    name: &'a N,\n) -> Option<T>\nwhere\n    &'a N: 'a + AsHeaderName,\n{\n    let removed = value_map.remove(name);\n    if removed.is_some() {\n        if let Some(name_map) = name_map {\n            name_map.remove(name);\n        }\n    }\n    removed\n}\n\n#[inline]\nfn header_to_h1_wire(key_map: Option<&CaseMap>, value_map: &HMap, buf: &mut impl BufMut) {\n    const CRLF: &[u8; 2] = b\"\\r\\n\";\n    const HEADER_KV_DELIMITER: &[u8; 2] = b\": \";\n\n    if let Some(key_map) = key_map {\n        case_header_iter(key_map.into(), value_map).for_each(|(case_header, val)| {\n            buf.put_slice(case_header.as_slice());\n            buf.put_slice(HEADER_KV_DELIMITER);\n            buf.put_slice(val.as_ref());\n            buf.put_slice(CRLF);\n        });\n    } else {\n        for (header, value) in value_map {\n            let titled_header =\n                case_header_name::titled_header_name_str(header).unwrap_or(header.as_str());\n            buf.put_slice(titled_header.as_bytes());\n            buf.put_slice(HEADER_KV_DELIMITER);\n            buf.put_slice(value.as_ref());\n            buf.put_slice(CRLF);\n        }\n    }\n}\n\n#[inline]\nfn case_header_iter<'a>(\n    name_map: Option<&'a CaseMap>,\n    value_map: &'a HMap,\n) -> impl Iterator<Item = (&'a CaseHeaderName, &'a HeaderValue)> + 'a {\n    name_map.into_iter().flat_map(|name_map| {\n        name_map\n            .iter()\n            .zip(value_map.iter())\n            .map(|((h1, name), (h2, value))| {\n                // in case the header iteration order changes in future versions of HMap\n                assert_eq!(h1, h2, \"header iter mismatch {}, {}\", h1, h2);\n                (name, value)\n            })\n    })\n}\n\n#[cfg(test)]\nmod tests {\n    use super::*;\n\n    #[test]\n    fn header_map_upper_bound() {\n        assert_eq!(8, http_header_map_upper_bound(None));\n        assert_eq!(16, http_header_map_upper_bound(Some(16)));\n        assert_eq!(4096, http_header_map_upper_bound(Some(7777)));\n    }\n\n    #[test]\n    fn test_single_header() {\n        let mut req = RequestHeader::build(\"GET\", b\"\\\\\", None).unwrap();\n        req.insert_header(\"foo\", \"bar\").unwrap();\n        req.insert_header(\"FoO\", \"Bar\").unwrap();\n        let mut buf: Vec<u8> = vec![];\n        req.header_to_h1_wire(&mut buf);\n        assert_eq!(buf, b\"FoO: Bar\\r\\n\");\n        req.case_header_iter().enumerate().for_each(|(i, (k, v))| {\n            let name = String::from_utf8_lossy(k.as_slice()).into_owned();\n            let value = String::from_utf8_lossy(v.as_ref()).into_owned();\n            match i + 1 {\n                1 => {\n                    assert_eq!(name, \"FoO\");\n                    assert_eq!(value, \"Bar\");\n                }\n                _ => panic!(\"too many headers\"),\n            }\n        });\n\n        let mut resp = ResponseHeader::new(None);\n        resp.insert_header(\"foo\", \"bar\").unwrap();\n        resp.insert_header(\"FoO\", \"Bar\").unwrap();\n        let mut buf: Vec<u8> = vec![];\n        resp.header_to_h1_wire(&mut buf);\n        assert_eq!(buf, b\"FoO: Bar\\r\\n\");\n        resp.case_header_iter().enumerate().for_each(|(i, (k, v))| {\n            let name = String::from_utf8_lossy(k.as_slice()).into_owned();\n            let value = String::from_utf8_lossy(v.as_ref()).into_owned();\n            match i + 1 {\n                1 => {\n                    assert_eq!(name, \"FoO\");\n                    assert_eq!(value, \"Bar\");\n                }\n                _ => panic!(\"too many headers\"),\n            }\n        });\n    }\n\n    #[test]\n    fn test_single_header_no_case() {\n        let mut req = RequestHeader::new_no_case(None);\n        req.insert_header(\"foo\", \"bar\").unwrap();\n        req.insert_header(\"FoO\", \"Bar\").unwrap();\n        let mut buf: Vec<u8> = vec![];\n        req.header_to_h1_wire(&mut buf);\n        assert_eq!(buf, b\"foo: Bar\\r\\n\");\n        req.case_header_iter().for_each(|(_, _)| {\n            unreachable!(\"request has no case\");\n        });\n\n        let mut resp = ResponseHeader::new_no_case(None);\n        resp.insert_header(\"foo\", \"bar\").unwrap();\n        resp.insert_header(\"FoO\", \"Bar\").unwrap();\n        let mut buf: Vec<u8> = vec![];\n        resp.header_to_h1_wire(&mut buf);\n        assert_eq!(buf, b\"foo: Bar\\r\\n\");\n        resp.case_header_iter().for_each(|(_, _)| {\n            unreachable!(\"response has no case\");\n        });\n    }\n\n    #[test]\n    fn test_multiple_header() {\n        let mut req = RequestHeader::build(\"GET\", b\"\\\\\", None).unwrap();\n        req.append_header(\"FoO\", \"Bar\").unwrap();\n        req.append_header(\"fOO\", \"bar\").unwrap();\n        req.append_header(\"BAZ\", \"baR\").unwrap();\n        req.append_header(http::header::CONTENT_LENGTH, \"0\")\n            .unwrap();\n        req.append_header(\"a\", \"b\").unwrap();\n        req.remove_header(\"a\");\n        let mut buf: Vec<u8> = vec![];\n        req.header_to_h1_wire(&mut buf);\n        assert_eq!(\n            buf,\n            b\"FoO: Bar\\r\\nfOO: bar\\r\\nBAZ: baR\\r\\nContent-Length: 0\\r\\n\"\n        );\n        req.case_header_iter().enumerate().for_each(|(i, (k, v))| {\n            let name = String::from_utf8_lossy(k.as_slice()).into_owned();\n            let value = String::from_utf8_lossy(v.as_ref()).into_owned();\n            match i + 1 {\n                1 => {\n                    assert_eq!(name, \"FoO\");\n                    assert_eq!(value, \"Bar\");\n                }\n                2 => {\n                    assert_eq!(name, \"fOO\");\n                    assert_eq!(value, \"bar\");\n                }\n                3 => {\n                    assert_eq!(name, \"BAZ\");\n                    assert_eq!(value, \"baR\");\n                }\n                4 => {\n                    assert_eq!(name, \"Content-Length\");\n                    assert_eq!(value, \"0\");\n                }\n                _ => panic!(\"too many headers\"),\n            }\n        });\n\n        let mut resp = ResponseHeader::new(None);\n        resp.append_header(\"FoO\", \"Bar\").unwrap();\n        resp.append_header(\"fOO\", \"bar\").unwrap();\n        resp.append_header(\"BAZ\", \"baR\").unwrap();\n        resp.append_header(http::header::CONTENT_LENGTH, \"0\")\n            .unwrap();\n        resp.append_header(\"a\", \"b\").unwrap();\n        resp.remove_header(\"a\");\n        let mut buf: Vec<u8> = vec![];\n        resp.header_to_h1_wire(&mut buf);\n        assert_eq!(\n            buf,\n            b\"FoO: Bar\\r\\nfOO: bar\\r\\nBAZ: baR\\r\\nContent-Length: 0\\r\\n\"\n        );\n        resp.case_header_iter().enumerate().for_each(|(i, (k, v))| {\n            let name = String::from_utf8_lossy(k.as_slice()).into_owned();\n            let value = String::from_utf8_lossy(v.as_ref()).into_owned();\n            match i + 1 {\n                1 => {\n                    assert_eq!(name, \"FoO\");\n                    assert_eq!(value, \"Bar\");\n                }\n                2 => {\n                    assert_eq!(name, \"fOO\");\n                    assert_eq!(value, \"bar\");\n                }\n                3 => {\n                    assert_eq!(name, \"BAZ\");\n                    assert_eq!(value, \"baR\");\n                }\n                4 => {\n                    assert_eq!(name, \"Content-Length\");\n                    assert_eq!(value, \"0\");\n                }\n                _ => panic!(\"too many headers\"),\n            }\n        });\n    }\n\n    #[cfg(feature = \"patched_http1\")]\n    #[test]\n    fn test_invalid_path() {\n        let raw_path = b\"Hello\\xF0\\x90\\x80World\";\n        let req = RequestHeader::build(\"GET\", &raw_path[..], None).unwrap();\n        assert_eq!(\"Hello�World\", req.uri.path_and_query().unwrap());\n        assert_eq!(raw_path, req.raw_path());\n    }\n\n    #[cfg(feature = \"patched_http1\")]\n    #[test]\n    fn test_override_invalid_path() {\n        let raw_path = b\"Hello\\xF0\\x90\\x80World\";\n        let mut req = RequestHeader::build(\"GET\", &raw_path[..], None).unwrap();\n        assert_eq!(\"Hello�World\", req.uri.path_and_query().unwrap());\n        assert_eq!(raw_path, req.raw_path());\n\n        let new_path = \"/HelloWorld\";\n        req.set_uri(Uri::builder().path_and_query(new_path).build().unwrap());\n        assert_eq!(new_path, req.uri.path_and_query().unwrap());\n        assert_eq!(new_path.as_bytes(), req.raw_path());\n    }\n\n    #[test]\n    fn test_reason_phrase() {\n        let mut resp = ResponseHeader::new(None);\n        let reason = resp.get_reason_phrase().unwrap();\n        assert_eq!(reason, \"OK\");\n\n        resp.set_reason_phrase(Some(\"FooBar\")).unwrap();\n        let reason = resp.get_reason_phrase().unwrap();\n        assert_eq!(reason, \"FooBar\");\n\n        resp.set_reason_phrase(Some(\"OK\")).unwrap();\n        let reason = resp.get_reason_phrase().unwrap();\n        assert_eq!(reason, \"OK\");\n\n        resp.set_reason_phrase(None).unwrap();\n        let reason = resp.get_reason_phrase().unwrap();\n        assert_eq!(reason, \"OK\");\n    }\n\n    #[test]\n    fn set_test_send_end_stream() {\n        let mut req = RequestHeader::build(\"GET\", b\"/\", None).unwrap();\n        req.set_send_end_stream(true);\n\n        // None for requests that are not h2\n        assert!(req.send_end_stream().is_none());\n\n        let mut req = RequestHeader::build(\"GET\", b\"/\", None).unwrap();\n        req.set_version(Version::HTTP_2);\n\n        // Some(true) by default for h2\n        assert!(req.send_end_stream().unwrap());\n\n        req.set_send_end_stream(false);\n        // Some(false)\n        assert!(!req.send_end_stream().unwrap());\n    }\n\n    #[test]\n    fn set_test_set_content_length() {\n        let mut resp = ResponseHeader::new(None);\n        resp.set_content_length(10).unwrap();\n\n        assert_eq!(\n            b\"10\",\n            resp.headers\n                .get(http::header::CONTENT_LENGTH)\n                .map(|d| d.as_bytes())\n                .unwrap()\n        );\n    }\n}\n"
  },
  {
    "path": "pingora-ketama/Cargo.toml",
    "content": "[package]\nname = \"pingora-ketama\"\nversion = \"0.8.0\"\ndescription = \"Rust port of the nginx consistent hash function\"\nauthors = [\"Pingora Team <pingora@cloudflare.com>\"]\nlicense = \"Apache-2.0\"\nedition = \"2021\"\nrepository = \"https://github.com/cloudflare/pingora\"\ncategories = [\"caching\", \"algorithms\"]\nkeywords = [\"hash\", \"hashing\", \"consistent\", \"pingora\"]\n\n[dependencies]\ncrc32fast = \"1.3\"\ni_key_sort = { version = \"0.10.1\", optional = true, features = [\"allow_multithreading\"] }\n\n[dev-dependencies]\ncriterion = \"0.7\"\ncsv = \"1.2\"\ndhat = \"0.3\"\nenv_logger = \"0.11\"\nlog = { workspace = true }\nrand = \"0.9.2\"\n\n[[bench]]\nname = \"simple\"\nharness = false\n\n[[bench]]\nname = \"memory\"\nharness = false\n\n[features]\nheap-prof = []\nv2 = [\"i_key_sort\"]\n"
  },
  {
    "path": "pingora-ketama/LICENSE",
    "content": "\n                                 Apache License\n                           Version 2.0, January 2004\n                        http://www.apache.org/licenses/\n\n   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION\n\n   1. Definitions.\n\n      \"License\" shall mean the terms and conditions for use, reproduction,\n      and distribution as defined by Sections 1 through 9 of this document.\n\n      \"Licensor\" shall mean the copyright owner or entity authorized by\n      the copyright owner that is granting the License.\n\n      \"Legal Entity\" shall mean the union of the acting entity and all\n      other entities that control, are controlled by, or are under common\n      control with that entity. For the purposes of this definition,\n      \"control\" means (i) the power, direct or indirect, to cause the\n      direction or management of such entity, whether by contract or\n      otherwise, or (ii) ownership of fifty percent (50%) or more of the\n      outstanding shares, or (iii) beneficial ownership of such entity.\n\n      \"You\" (or \"Your\") shall mean an individual or Legal Entity\n      exercising permissions granted by this License.\n\n      \"Source\" form shall mean the preferred form for making modifications,\n      including but not limited to software source code, documentation\n      source, and configuration files.\n\n      \"Object\" form shall mean any form resulting from mechanical\n      transformation or translation of a Source form, including but\n      not limited to compiled object code, generated documentation,\n      and conversions to other media types.\n\n      \"Work\" shall mean the work of authorship, whether in Source or\n      Object form, made available under the License, as indicated by a\n      copyright notice that is included in or attached to the work\n      (an example is provided in the Appendix below).\n\n      \"Derivative Works\" shall mean any work, whether in Source or Object\n      form, that is based on (or derived from) the Work and for which the\n      editorial revisions, annotations, elaborations, or other modifications\n      represent, as a whole, an original work of authorship. For the purposes\n      of this License, Derivative Works shall not include works that remain\n      separable from, or merely link (or bind by name) to the interfaces of,\n      the Work and Derivative Works thereof.\n\n      \"Contribution\" shall mean any work of authorship, including\n      the original version of the Work and any modifications or additions\n      to that Work or Derivative Works thereof, that is intentionally\n      submitted to Licensor for inclusion in the Work by the copyright owner\n      or by an individual or Legal Entity authorized to submit on behalf of\n      the copyright owner. For the purposes of this definition, \"submitted\"\n      means any form of electronic, verbal, or written communication sent\n      to the Licensor or its representatives, including but not limited to\n      communication on electronic mailing lists, source code control systems,\n      and issue tracking systems that are managed by, or on behalf of, the\n      Licensor for the purpose of discussing and improving the Work, but\n      excluding communication that is conspicuously marked or otherwise\n      designated in writing by the copyright owner as \"Not a Contribution.\"\n\n      \"Contributor\" shall mean Licensor and any individual or Legal Entity\n      on behalf of whom a Contribution has been received by Licensor and\n      subsequently incorporated within the Work.\n\n   2. Grant of Copyright License. Subject to the terms and conditions of\n      this License, each Contributor hereby grants to You a perpetual,\n      worldwide, non-exclusive, no-charge, royalty-free, irrevocable\n      copyright license to reproduce, prepare Derivative Works of,\n      publicly display, publicly perform, sublicense, and distribute the\n      Work and such Derivative Works in Source or Object form.\n\n   3. Grant of Patent License. Subject to the terms and conditions of\n      this License, each Contributor hereby grants to You a perpetual,\n      worldwide, non-exclusive, no-charge, royalty-free, irrevocable\n      (except as stated in this section) patent license to make, have made,\n      use, offer to sell, sell, import, and otherwise transfer the Work,\n      where such license applies only to those patent claims licensable\n      by such Contributor that are necessarily infringed by their\n      Contribution(s) alone or by combination of their Contribution(s)\n      with the Work to which such Contribution(s) was submitted. If You\n      institute patent litigation against any entity (including a\n      cross-claim or counterclaim in a lawsuit) alleging that the Work\n      or a Contribution incorporated within the Work constitutes direct\n      or contributory patent infringement, then any patent licenses\n      granted to You under this License for that Work shall terminate\n      as of the date such litigation is filed.\n\n   4. Redistribution. You may reproduce and distribute copies of the\n      Work or Derivative Works thereof in any medium, with or without\n      modifications, and in Source or Object form, provided that You\n      meet the following conditions:\n\n      (a) You must give any other recipients of the Work or\n          Derivative Works a copy of this License; and\n\n      (b) You must cause any modified files to carry prominent notices\n          stating that You changed the files; and\n\n      (c) You must retain, in the Source form of any Derivative Works\n          that You distribute, all copyright, patent, trademark, and\n          attribution notices from the Source form of the Work,\n          excluding those notices that do not pertain to any part of\n          the Derivative Works; and\n\n      (d) If the Work includes a \"NOTICE\" text file as part of its\n          distribution, then any Derivative Works that You distribute must\n          include a readable copy of the attribution notices contained\n          within such NOTICE file, excluding those notices that do not\n          pertain to any part of the Derivative Works, in at least one\n          of the following places: within a NOTICE text file distributed\n          as part of the Derivative Works; within the Source form or\n          documentation, if provided along with the Derivative Works; or,\n          within a display generated by the Derivative Works, if and\n          wherever such third-party notices normally appear. The contents\n          of the NOTICE file are for informational purposes only and\n          do not modify the License. You may add Your own attribution\n          notices within Derivative Works that You distribute, alongside\n          or as an addendum to the NOTICE text from the Work, provided\n          that such additional attribution notices cannot be construed\n          as modifying the License.\n\n      You may add Your own copyright statement to Your modifications and\n      may provide additional or different license terms and conditions\n      for use, reproduction, or distribution of Your modifications, or\n      for any such Derivative Works as a whole, provided Your use,\n      reproduction, and distribution of the Work otherwise complies with\n      the conditions stated in this License.\n\n   5. Submission of Contributions. Unless You explicitly state otherwise,\n      any Contribution intentionally submitted for inclusion in the Work\n      by You to the Licensor shall be under the terms and conditions of\n      this License, without any additional terms or conditions.\n      Notwithstanding the above, nothing herein shall supersede or modify\n      the terms of any separate license agreement you may have executed\n      with Licensor regarding such Contributions.\n\n   6. Trademarks. This License does not grant permission to use the trade\n      names, trademarks, service marks, or product names of the Licensor,\n      except as required for reasonable and customary use in describing the\n      origin of the Work and reproducing the content of the NOTICE file.\n\n   7. Disclaimer of Warranty. Unless required by applicable law or\n      agreed to in writing, Licensor provides the Work (and each\n      Contributor provides its Contributions) on an \"AS IS\" BASIS,\n      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or\n      implied, including, without limitation, any warranties or conditions\n      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A\n      PARTICULAR PURPOSE. You are solely responsible for determining the\n      appropriateness of using or redistributing the Work and assume any\n      risks associated with Your exercise of permissions under this License.\n\n   8. Limitation of Liability. In no event and under no legal theory,\n      whether in tort (including negligence), contract, or otherwise,\n      unless required by applicable law (such as deliberate and grossly\n      negligent acts) or agreed to in writing, shall any Contributor be\n      liable to You for damages, including any direct, indirect, special,\n      incidental, or consequential damages of any character arising as a\n      result of this License or out of the use or inability to use the\n      Work (including but not limited to damages for loss of goodwill,\n      work stoppage, computer failure or malfunction, or any and all\n      other commercial damages or losses), even if such Contributor\n      has been advised of the possibility of such damages.\n\n   9. Accepting Warranty or Additional Liability. While redistributing\n      the Work or Derivative Works thereof, You may choose to offer,\n      and charge a fee for, acceptance of support, warranty, indemnity,\n      or other liability obligations and/or rights consistent with this\n      License. However, in accepting such obligations, You may act only\n      on Your own behalf and on Your sole responsibility, not on behalf\n      of any other Contributor, and only if You agree to indemnify,\n      defend, and hold each Contributor harmless for any liability\n      incurred by, or claims asserted against, such Contributor by reason\n      of your accepting any such warranty or additional liability.\n\n   END OF TERMS AND CONDITIONS\n\n   APPENDIX: How to apply the Apache License to your work.\n\n      To apply the Apache License to your work, attach the following\n      boilerplate notice, with the fields enclosed by brackets \"[]\"\n      replaced with your own identifying information. (Don't include\n      the brackets!)  The text should be enclosed in the appropriate\n      comment syntax for the file format. We also recommend that a\n      file or class name and description of purpose be included on the\n      same \"printed page\" as the copyright notice for easier\n      identification within third-party archives.\n\n   Copyright [yyyy] [name of copyright owner]\n\n   Licensed under the Apache License, Version 2.0 (the \"License\");\n   you may not use this file except in compliance with the License.\n   You may obtain a copy of the License at\n\n       http://www.apache.org/licenses/LICENSE-2.0\n\n   Unless required by applicable law or agreed to in writing, software\n   distributed under the License is distributed on an \"AS IS\" BASIS,\n   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n   See the License for the specific language governing permissions and\n   limitations under the License.\n"
  },
  {
    "path": "pingora-ketama/benches/memory.rs",
    "content": "use pingora_ketama::{Bucket, Continuum};\n\n#[global_allocator]\nstatic ALLOC: dhat::Alloc = dhat::Alloc;\n\nfn buckets() -> Vec<Bucket> {\n    let mut b = Vec::new();\n\n    for i in 1..254 {\n        b.push(Bucket::new(\n            format!(\"127.0.0.{i}:6443\").parse().unwrap(),\n            10,\n        ));\n    }\n\n    b\n}\n\npub fn main() {\n    let _profiler = dhat::Profiler::new_heap();\n    let _c = Continuum::new(&buckets());\n}\n"
  },
  {
    "path": "pingora-ketama/benches/simple.rs",
    "content": "use pingora_ketama::{Bucket, Continuum};\n\nuse criterion::{criterion_group, criterion_main, Criterion};\nuse rand::{\n    distr::{Alphanumeric, SampleString},\n    rng,\n};\n\n#[cfg(feature = \"heap-prof\")]\n#[global_allocator]\nstatic ALLOC: dhat::Alloc = dhat::Alloc;\n\nfn buckets() -> Vec<Bucket> {\n    let mut b = Vec::new();\n\n    for i in 1..101 {\n        b.push(Bucket::new(format!(\"127.0.0.{i}:6443\").parse().unwrap(), 1));\n    }\n\n    b\n}\n\nfn random_string() -> String {\n    let mut rand = rng();\n    Alphanumeric.sample_string(&mut rand, 30)\n}\n\npub fn criterion_benchmark(c: &mut Criterion) {\n    #[cfg(feature = \"heap-prof\")]\n    let _profiler = dhat::Profiler::new_heap();\n\n    c.bench_function(\"create_continuum\", |b| {\n        b.iter(|| Continuum::new(&buckets()))\n    });\n\n    c.bench_function(\"continuum_hash\", |b| {\n        let continuum = Continuum::new(&buckets());\n\n        b.iter(|| continuum.node(random_string().as_bytes()))\n    });\n}\n\ncriterion_group!(benches, criterion_benchmark);\ncriterion_main!(benches);\n"
  },
  {
    "path": "pingora-ketama/examples/health_aware_selector.rs",
    "content": "use log::info;\nuse pingora_ketama::{Bucket, Continuum};\nuse std::collections::HashMap;\nuse std::net::SocketAddr;\n\n// A repository for node healthiness, emulating a health checker.\nstruct NodeHealthRepository {\n    nodes: HashMap<SocketAddr, bool>,\n}\n\nimpl NodeHealthRepository {\n    fn new() -> Self {\n        NodeHealthRepository {\n            nodes: HashMap::new(),\n        }\n    }\n\n    fn set_node_health(&mut self, node: SocketAddr, is_healthy: bool) {\n        self.nodes.insert(node, is_healthy);\n    }\n\n    fn node_is_healthy(&self, node: &SocketAddr) -> bool {\n        self.nodes.get(node).cloned().unwrap_or(false)\n    }\n}\n\n// A health-aware node selector, which relies on the above health repository.\nstruct HealthAwareNodeSelector<'a> {\n    ring: Continuum,\n    max_tries: usize,\n    node_health_repo: &'a NodeHealthRepository,\n}\n\nimpl HealthAwareNodeSelector<'_> {\n    fn new(r: Continuum, tries: usize, nhr: &NodeHealthRepository) -> HealthAwareNodeSelector<'_> {\n        HealthAwareNodeSelector {\n            ring: r,\n            max_tries: tries,\n            node_health_repo: nhr,\n        }\n    }\n\n    // Try to select a node within <max_tries> attempts.\n    fn try_select(&self, key: &str) -> Option<SocketAddr> {\n        let node_iter = self.ring.node_iter(key.as_bytes());\n\n        for (tries, node) in node_iter.enumerate() {\n            if tries >= self.max_tries {\n                break;\n            }\n\n            if self.node_health_repo.node_is_healthy(node) {\n                return Some(*node);\n            }\n        }\n\n        None\n    }\n}\n\n// RUST_LOG=INFO cargo run --example health_aware_selector\nfn main() {\n    env_logger::init();\n\n    // Set up some nodes.\n    let buckets: Vec<_> = (1..=10)\n        .map(|i| Bucket::new(format!(\"127.0.0.{i}:6443\").parse().unwrap(), 1))\n        .collect();\n\n    // Mark the 1-5th nodes healthy, the 6-10th nodes unhealthy.\n    let mut health_repo = NodeHealthRepository::new();\n    (1..=10)\n        .map(|i| (i, format!(\"127.0.0.{i}:6443\").parse().unwrap()))\n        .for_each(|(i, n)| {\n            health_repo.set_node_health(n, i < 6);\n        });\n\n    // Create a health-aware selector with up to 3 tries.\n    let health_aware_selector =\n        HealthAwareNodeSelector::new(Continuum::new(&buckets), 3, &health_repo);\n\n    // Let's try the selector on a few keys.\n    for i in 0..5 {\n        let key = format!(\"key_{i}\");\n        match health_aware_selector.try_select(&key) {\n            Some(node) => {\n                info!(\"{key}: {}:{}\", node.ip(), node.port());\n            }\n            None => {\n                info!(\"{key}: no healthy node found!\");\n            }\n        }\n    }\n}\n"
  },
  {
    "path": "pingora-ketama/src/lib.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! # pingora-ketama\n//! A Rust port of the nginx consistent hashing algorithm.\n//!\n//! This crate provides a consistent hashing algorithm which is identical in\n//! behavior to [nginx consistent hashing](https://www.nginx.com/resources/wiki/modules/consistent_hash/).\n//!\n//! Using a consistent hash strategy like this is useful when one wants to\n//! minimize the amount of requests that need to be rehashed to different nodes\n//! when a node is added or removed.\n//!\n//! Here's a simple example of how one might use it:\n//!\n//! ```\n//! use pingora_ketama::{Bucket, Continuum};\n//!\n//! # #[allow(clippy::needless_doctest_main)]\n//! fn main() {\n//!     // Set up a continuum with a few nodes of various weight.\n//!     let mut buckets = vec![];\n//!     buckets.push(Bucket::new(\"127.0.0.1:12345\".parse().unwrap(), 1));\n//!     buckets.push(Bucket::new(\"127.0.0.2:12345\".parse().unwrap(), 2));\n//!     buckets.push(Bucket::new(\"127.0.0.3:12345\".parse().unwrap(), 3));\n//!     let ring = Continuum::new(&buckets);\n//!\n//!     // Let's see what the result is for a few keys:\n//!     for key in &[\"some_key\", \"another_key\", \"last_key\"] {\n//!         let node = ring.node(key.as_bytes()).unwrap();\n//!         println!(\"{}: {}:{}\", key, node.ip(), node.port());\n//!     }\n//! }\n//! ```\n//!\n//! ```bash\n//! # Output:\n//! some_key: 127.0.0.3:12345\n//! another_key: 127.0.0.3:12345\n//! last_key: 127.0.0.2:12345\n//! ```\n//!\n//! We've provided a health-aware example in\n//! `pingora-ketama/examples/health_aware_selector.rs`.\n//!\n//! For a carefully crafted real-world example, see the [`pingora-load-balancing`](https://docs.rs/pingora-load-balancing)\n//! crate.\n\nuse std::cmp::Ordering;\nuse std::io::Write;\nuse std::net::SocketAddr;\n\nuse crc32fast::Hasher;\n#[cfg(feature = \"v2\")]\nuse i_key_sort::sort::one_key_cmp::OneKeyAndCmpSort;\n\n/// This constant is copied from nginx. It will create 160 points per weight\n/// unit. For example, a weight of 2 will create 320 points on the ring.\npub const DEFAULT_POINT_MULTIPLE: u32 = 160;\n\n/// A [Bucket] represents a server for consistent hashing\n///\n/// A [Bucket] contains a [SocketAddr] to the server and a weight associated with it.\n#[derive(Clone, Debug, Eq, PartialEq, PartialOrd, Ord)]\npub struct Bucket {\n    // The node name.\n    // TODO: UDS\n    node: SocketAddr,\n\n    // The weight associated with a node. A higher weight indicates that this node should\n    // receive more requests.\n    weight: u32,\n}\n\nimpl Bucket {\n    /// Return a new bucket with the given node and weight.\n    ///\n    /// The chance that a [Bucket] is selected is proportional to the relative weight of all [Bucket]s.\n    ///\n    /// # Panics\n    ///\n    /// This will panic if the weight is zero.\n    pub fn new(node: SocketAddr, weight: u32) -> Self {\n        assert!(weight != 0, \"weight must be at least one\");\n\n        Bucket { node, weight }\n    }\n}\n\n// A point on the continuum.\n#[derive(Clone, Debug, Eq, PartialEq)]\nstruct PointV1 {\n    // the index to the actual address\n    node: u32,\n    hash: u32,\n}\n\n// We only want to compare the hash when sorting, so we implement these traits by hand.\nimpl Ord for PointV1 {\n    fn cmp(&self, other: &Self) -> Ordering {\n        self.hash.cmp(&other.hash)\n    }\n}\n\nimpl PartialOrd for PointV1 {\n    fn partial_cmp(&self, other: &Self) -> Option<Ordering> {\n        Some(self.cmp(other))\n    }\n}\n\nimpl PointV1 {\n    fn new(node: u32, hash: u32) -> Self {\n        PointV1 { node, hash }\n    }\n}\n\n/// A point on the continuum.\n///\n/// We are trying to save memory here, so this struct is equivalent to a struct\n/// this this definition, but doesn't require using the \"untrustworthy\" compact\n/// repr. This does mean we have to do the memory layout manually though, but\n/// the benchmarks show there is no performance hit for it.\n///\n/// #[repr(Rust, packed)]\n/// struct Point {\n///     node: u16,\n///     hash: u32,\n/// }\n#[cfg(feature = \"v2\")]\n#[derive(Copy, Clone, Eq, PartialEq)]\n#[repr(transparent)]\nstruct PointV2([u8; 6]);\n\n#[cfg(feature = \"v2\")]\nimpl PointV2 {\n    fn new(node: u16, hash: u32) -> Self {\n        let mut this = [0; 6];\n\n        this[0..4].copy_from_slice(&hash.to_ne_bytes());\n        this[4..6].copy_from_slice(&node.to_ne_bytes());\n\n        Self(this)\n    }\n\n    /// Return the hash of the point which is stored in the first 4 bytes (big endian).\n    fn hash(&self) -> u32 {\n        u32::from_ne_bytes(self.0[0..4].try_into().expect(\"There are exactly 4 bytes\"))\n    }\n\n    /// Return the node of the point which is stored in the last 2 bytes (big endian).\n    fn node(&self) -> u16 {\n        u16::from_ne_bytes(self.0[4..6].try_into().expect(\"There are exactly 2 bytes\"))\n    }\n}\n\n#[derive(Copy, Clone, Debug, Eq, PartialEq, Default)]\npub enum Version {\n    #[default]\n    V1,\n    #[cfg(feature = \"v2\")]\n    V2 { point_multiple: u32 },\n}\n\nimpl Version {\n    fn point_multiple(&self) -> u32 {\n        match self {\n            Version::V1 => DEFAULT_POINT_MULTIPLE,\n            #[cfg(feature = \"v2\")]\n            Version::V2 { point_multiple } => *point_multiple,\n        }\n    }\n}\n\nenum RingBuilder {\n    V1(Vec<PointV1>),\n    #[cfg(feature = \"v2\")]\n    V2(Vec<PointV2>),\n}\n\nimpl RingBuilder {\n    fn new(version: Version, total_weight: u32) -> Self {\n        match version {\n            Version::V1 => RingBuilder::V1(Vec::with_capacity(\n                (total_weight * DEFAULT_POINT_MULTIPLE) as usize,\n            )),\n            #[cfg(feature = \"v2\")]\n            Version::V2 { point_multiple } => {\n                RingBuilder::V2(Vec::with_capacity((total_weight * point_multiple) as usize))\n            }\n        }\n    }\n\n    fn push(&mut self, node: u16, hash: u32) {\n        match self {\n            RingBuilder::V1(ring) => {\n                ring.push(PointV1::new(node as u32, hash));\n            }\n            #[cfg(feature = \"v2\")]\n            RingBuilder::V2(ring) => {\n                ring.push(PointV2::new(node, hash));\n            }\n        }\n    }\n\n    #[allow(unused)]\n    fn sort(&mut self, addresses: &[SocketAddr]) {\n        match self {\n            RingBuilder::V1(ring) => {\n                // Sort and remove any duplicates.\n                ring.sort_unstable();\n                ring.dedup_by(|a, b| a.hash == b.hash);\n            }\n            #[cfg(feature = \"v2\")]\n            RingBuilder::V2(ring) => {\n                ring.sort_by_one_key_then_by(\n                    true,\n                    |p| p.hash(),\n                    |p1, p2| addresses[p1.node() as usize].cmp(&addresses[p2.node() as usize]),\n                );\n\n                //secondary_radix_sort(ring, |p| p.hash(), |p| addresses[p.node() as usize]);\n                ring.dedup_by(|a, b| a.0[0..4] == b.0[0..4]);\n            }\n        }\n    }\n}\n\nimpl From<RingBuilder> for VersionedRing {\n    fn from(ring: RingBuilder) -> Self {\n        match ring {\n            RingBuilder::V1(ring) => VersionedRing::V1(ring.into_boxed_slice()),\n            #[cfg(feature = \"v2\")]\n            RingBuilder::V2(ring) => VersionedRing::V2(ring.into_boxed_slice()),\n        }\n    }\n}\n\nenum VersionedRing {\n    V1(Box<[PointV1]>),\n    #[cfg(feature = \"v2\")]\n    V2(Box<[PointV2]>),\n}\n\nimpl VersionedRing {\n    /// Find the associated index for the given input.\n    pub fn node_idx(&self, hash: u32) -> usize {\n        // The `Result` returned here is either a match or the error variant\n        // returns where the value would be inserted.\n        let search_result = match self {\n            VersionedRing::V1(ring) => ring.binary_search_by(|p| p.hash.cmp(&hash)),\n            #[cfg(feature = \"v2\")]\n            VersionedRing::V2(ring) => ring.binary_search_by(|p| p.hash().cmp(&hash)),\n        };\n\n        match search_result {\n            Ok(i) => i,\n            Err(i) => {\n                // We wrap around to the front if this value would be\n                // inserted at the end.\n                if i == self.len() {\n                    0\n                } else {\n                    i\n                }\n            }\n        }\n    }\n\n    pub fn get(&self, index: usize) -> Option<usize> {\n        match self {\n            VersionedRing::V1(ring) => ring.get(index).map(|p| p.node as usize),\n            #[cfg(feature = \"v2\")]\n            VersionedRing::V2(ring) => ring.get(index).map(|p| p.node() as usize),\n        }\n    }\n\n    pub fn len(&self) -> usize {\n        match self {\n            VersionedRing::V1(ring) => ring.len(),\n            #[cfg(feature = \"v2\")]\n            VersionedRing::V2(ring) => ring.len(),\n        }\n    }\n}\n\n/// The consistent hashing ring\n///\n/// A [Continuum] represents a ring of buckets where a node is associated with various points on\n/// the ring.\npub struct Continuum {\n    ring: VersionedRing,\n    addrs: Box<[SocketAddr]>,\n}\n\nimpl Continuum {\n    pub fn new(buckets: &[Bucket]) -> Self {\n        Self::new_with_version(buckets, Version::default())\n    }\n\n    /// Create a new [Continuum] with the given list of buckets.\n    pub fn new_with_version(buckets: &[Bucket], version: Version) -> Self {\n        if buckets.is_empty() {\n            return Continuum {\n                ring: VersionedRing::V1(Box::new([])),\n                addrs: Box::new([]),\n            };\n        }\n\n        // The total weight is multiplied by the factor of points to create many points per node.\n        let total_weight: u32 = buckets.iter().fold(0, |sum, b| sum + b.weight);\n        let mut ring = RingBuilder::new(version, total_weight);\n        let mut addrs = Vec::with_capacity(buckets.len());\n\n        for bucket in buckets {\n            let mut hasher = Hasher::new();\n\n            // We only do the following for backwards compatibility with nginx/memcache:\n            // - Convert SocketAddr to string\n            // - The hash input is as follows \"HOST EMPTY PORT PREVIOUS_HASH\". Spaces are only added\n            //   for readability.\n            // TODO: remove this logic and hash the literal SocketAddr once we no longer\n            // need backwards compatibility\n\n            // with_capacity = max_len(ipv6)(39) + len(null)(1) + max_len(port)(5)\n            let mut hash_bytes = Vec::with_capacity(39 + 1 + 5);\n            write!(&mut hash_bytes, \"{}\", bucket.node.ip()).unwrap();\n            write!(&mut hash_bytes, \"\\0\").unwrap();\n            write!(&mut hash_bytes, \"{}\", bucket.node.port()).unwrap();\n            hasher.update(hash_bytes.as_ref());\n\n            // A higher weight will add more points for this node.\n            let num_points = bucket.weight * version.point_multiple();\n\n            // This is appended to the crc32 hash for each point.\n            let mut prev_hash: u32 = 0;\n            addrs.push(bucket.node);\n            let node = addrs.len() - 1;\n            for _ in 0..num_points {\n                let mut hasher = hasher.clone();\n                hasher.update(&prev_hash.to_le_bytes());\n\n                let hash = hasher.finalize();\n                ring.push(node as u16, hash);\n                prev_hash = hash;\n            }\n        }\n\n        let addrs = addrs.into_boxed_slice();\n\n        // Sort and remove any duplicates.\n        ring.sort(&addrs);\n\n        Continuum {\n            ring: ring.into(),\n            addrs,\n        }\n    }\n\n    /// Find the associated index for the given input.\n    pub fn node_idx(&self, input: &[u8]) -> usize {\n        let hash = crc32fast::hash(input);\n        self.ring.node_idx(hash)\n    }\n\n    /// Hash the given `hash_key` to the server address.\n    pub fn node(&self, hash_key: &[u8]) -> Option<SocketAddr> {\n        self.ring\n            .get(self.node_idx(hash_key)) // should we unwrap here?\n            .map(|n| self.addrs[n])\n    }\n\n    /// Get an iterator of nodes starting at the original hashed node of the `hash_key`.\n    ///\n    /// This function is useful to find failover servers if the original ones are offline, which is\n    /// cheaper than rebuilding the entire hash ring.\n    pub fn node_iter(&self, hash_key: &[u8]) -> NodeIterator<'_> {\n        NodeIterator {\n            idx: self.node_idx(hash_key),\n            continuum: self,\n        }\n    }\n\n    pub fn get_addr(&self, idx: &mut usize) -> Option<&SocketAddr> {\n        let point = self.ring.get(*idx);\n        if point.is_some() {\n            // only update idx for non-empty ring otherwise we will panic on modulo 0\n            *idx = (*idx + 1) % self.ring.len();\n        }\n        point.map(|n| &self.addrs[n])\n    }\n}\n\n/// Iterator over a Continuum\npub struct NodeIterator<'a> {\n    idx: usize,\n    continuum: &'a Continuum,\n}\n\nimpl<'a> Iterator for NodeIterator<'a> {\n    type Item = &'a SocketAddr;\n\n    fn next(&mut self) -> Option<Self::Item> {\n        self.continuum.get_addr(&mut self.idx)\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use std::net::SocketAddr;\n    use std::path::Path;\n\n    use super::{Bucket, Continuum};\n\n    fn get_sockaddr(ip: &str) -> SocketAddr {\n        ip.parse().unwrap()\n    }\n\n    #[test]\n    fn consistency_after_adding_host() {\n        fn assert_hosts(c: &Continuum) {\n            assert_eq!(c.node(b\"a\"), Some(get_sockaddr(\"127.0.0.10:6443\")));\n            assert_eq!(c.node(b\"b\"), Some(get_sockaddr(\"127.0.0.5:6443\")));\n        }\n\n        let buckets: Vec<_> = (1..11)\n            .map(|u| Bucket::new(get_sockaddr(&format!(\"127.0.0.{u}:6443\")), 1))\n            .collect();\n        let c = Continuum::new(&buckets);\n        assert_hosts(&c);\n\n        // Now add a new host and ensure that the hosts don't get shuffled.\n        let buckets: Vec<_> = (1..12)\n            .map(|u| Bucket::new(get_sockaddr(&format!(\"127.0.0.{u}:6443\")), 1))\n            .collect();\n\n        let c = Continuum::new(&buckets);\n        assert_hosts(&c);\n    }\n\n    #[test]\n    fn matches_nginx_sample() {\n        let upstream_hosts = [\"127.0.0.1:7777\", \"127.0.0.1:7778\"];\n        let upstream_hosts = upstream_hosts.iter().map(|i| get_sockaddr(i));\n\n        let mut buckets = Vec::new();\n        for upstream in upstream_hosts {\n            buckets.push(Bucket::new(upstream, 1));\n        }\n\n        let c = Continuum::new(&buckets);\n\n        assert_eq!(c.node(b\"/some/path\"), Some(get_sockaddr(\"127.0.0.1:7778\")));\n        assert_eq!(\n            c.node(b\"/some/longer/path\"),\n            Some(get_sockaddr(\"127.0.0.1:7777\"))\n        );\n        assert_eq!(\n            c.node(b\"/sad/zaidoon\"),\n            Some(get_sockaddr(\"127.0.0.1:7778\"))\n        );\n        assert_eq!(c.node(b\"/g\"), Some(get_sockaddr(\"127.0.0.1:7777\")));\n        assert_eq!(\n            c.node(b\"/pingora/team/is/cool/and/this/is/a/long/uri\"),\n            Some(get_sockaddr(\"127.0.0.1:7778\"))\n        );\n        assert_eq!(\n            c.node(b\"/i/am/not/confident/in/this/code\"),\n            Some(get_sockaddr(\"127.0.0.1:7777\"))\n        );\n    }\n\n    #[test]\n    fn matches_nginx_sample_data() {\n        let upstream_hosts = [\n            \"10.0.0.1:443\",\n            \"10.0.0.2:443\",\n            \"10.0.0.3:443\",\n            \"10.0.0.4:443\",\n            \"10.0.0.5:443\",\n            \"10.0.0.6:443\",\n            \"10.0.0.7:443\",\n            \"10.0.0.8:443\",\n            \"10.0.0.9:443\",\n        ];\n        let upstream_hosts = upstream_hosts.iter().map(|i| get_sockaddr(i));\n\n        let mut buckets = Vec::new();\n        for upstream in upstream_hosts {\n            buckets.push(Bucket::new(upstream, 100));\n        }\n\n        let c = Continuum::new(&buckets);\n\n        let path = Path::new(env!(\"CARGO_MANIFEST_DIR\"))\n            .join(\"test-data\")\n            .join(\"sample-nginx-upstream.csv\");\n\n        let mut rdr = csv::ReaderBuilder::new()\n            .has_headers(false)\n            .from_path(path)\n            .unwrap();\n\n        for pair in rdr.records() {\n            let pair = pair.unwrap();\n            let uri = pair.get(0).unwrap();\n            let upstream = pair.get(1).unwrap();\n\n            let got = c.node(uri.as_bytes()).unwrap();\n            assert_eq!(got, get_sockaddr(upstream));\n        }\n    }\n\n    #[test]\n    fn node_iter() {\n        let upstream_hosts = [\"127.0.0.1:7777\", \"127.0.0.1:7778\", \"127.0.0.1:7779\"];\n        let upstream_hosts = upstream_hosts.iter().map(|i| get_sockaddr(i));\n\n        let mut buckets = Vec::new();\n        for upstream in upstream_hosts {\n            buckets.push(Bucket::new(upstream, 1));\n        }\n\n        let c = Continuum::new(&buckets);\n        let mut iter = c.node_iter(b\"doghash\");\n        assert_eq!(iter.next(), Some(&get_sockaddr(\"127.0.0.1:7778\")));\n        assert_eq!(iter.next(), Some(&get_sockaddr(\"127.0.0.1:7779\")));\n        assert_eq!(iter.next(), Some(&get_sockaddr(\"127.0.0.1:7779\")));\n        assert_eq!(iter.next(), Some(&get_sockaddr(\"127.0.0.1:7777\")));\n        assert_eq!(iter.next(), Some(&get_sockaddr(\"127.0.0.1:7777\")));\n        assert_eq!(iter.next(), Some(&get_sockaddr(\"127.0.0.1:7778\")));\n        assert_eq!(iter.next(), Some(&get_sockaddr(\"127.0.0.1:7778\")));\n        assert_eq!(iter.next(), Some(&get_sockaddr(\"127.0.0.1:7779\")));\n\n        // drop 127.0.0.1:7777\n        let upstream_hosts = [\"127.0.0.1:7777\", \"127.0.0.1:7779\"];\n        let upstream_hosts = upstream_hosts.iter().map(|i| get_sockaddr(i));\n\n        let mut buckets = Vec::new();\n        for upstream in upstream_hosts {\n            buckets.push(Bucket::new(upstream, 1));\n        }\n\n        let c = Continuum::new(&buckets);\n        let mut iter = c.node_iter(b\"doghash\");\n        // 127.0.0.1:7778 nodes are gone now\n        // assert_eq!(iter.next(), Some(\"127.0.0.1:7778\"));\n        assert_eq!(iter.next(), Some(&get_sockaddr(\"127.0.0.1:7779\")));\n        assert_eq!(iter.next(), Some(&get_sockaddr(\"127.0.0.1:7779\")));\n        assert_eq!(iter.next(), Some(&get_sockaddr(\"127.0.0.1:7777\")));\n        assert_eq!(iter.next(), Some(&get_sockaddr(\"127.0.0.1:7777\")));\n        // assert_eq!(iter.next(), Some(\"127.0.0.1:7778\"));\n        // assert_eq!(iter.next(), Some(\"127.0.0.1:7778\"));\n        assert_eq!(iter.next(), Some(&get_sockaddr(\"127.0.0.1:7779\")));\n\n        // assert infinite cycle\n        let c = Continuum::new(&[Bucket::new(get_sockaddr(\"127.0.0.1:7777\"), 1)]);\n        let mut iter = c.node_iter(b\"doghash\");\n\n        let start_idx = iter.idx;\n        for _ in 0..c.ring.len() {\n            assert!(iter.next().is_some());\n        }\n        // assert wrap around\n        assert_eq!(start_idx, iter.idx);\n    }\n\n    #[test]\n    fn test_empty() {\n        let c = Continuum::new(&[]);\n        assert!(c.node(b\"doghash\").is_none());\n\n        let mut iter = c.node_iter(b\"doghash\");\n        assert!(iter.next().is_none());\n        assert!(iter.next().is_none());\n        assert!(iter.next().is_none());\n    }\n\n    #[test]\n    fn test_ipv6_ring() {\n        let upstream_hosts = [\"[::1]:7777\", \"[::1]:7778\", \"[::1]:7779\"];\n        let upstream_hosts = upstream_hosts.iter().map(|i| get_sockaddr(i));\n\n        let mut buckets = Vec::new();\n        for upstream in upstream_hosts {\n            buckets.push(Bucket::new(upstream, 1));\n        }\n\n        let c = Continuum::new(&buckets);\n        let mut iter = c.node_iter(b\"doghash\");\n        assert_eq!(iter.next(), Some(&get_sockaddr(\"[::1]:7777\")));\n        assert_eq!(iter.next(), Some(&get_sockaddr(\"[::1]:7778\")));\n        assert_eq!(iter.next(), Some(&get_sockaddr(\"[::1]:7777\")));\n        assert_eq!(iter.next(), Some(&get_sockaddr(\"[::1]:7778\")));\n        assert_eq!(iter.next(), Some(&get_sockaddr(\"[::1]:7778\")));\n        assert_eq!(iter.next(), Some(&get_sockaddr(\"[::1]:7777\")));\n        assert_eq!(iter.next(), Some(&get_sockaddr(\"[::1]:7779\")));\n    }\n}\n"
  },
  {
    "path": "pingora-ketama/test-data/README.md",
    "content": "# Steps to generate nginx upstream ketama hash logs\n\n1. Prepare nginx conf\n```\nmkdir -p /tmp/nginx-ketama/logs\ncp nginx.conf /tmp/nginx-ketama\nnginx -t -c nginx.conf -p /tmp/nginx-ketama\n```\n\n2. Generate trace\n```\n./trace.sh\n```\n\n3. Collect trace\n```\n cp /tmp/nginx-ketama/logs/access.log ./sample-nginx-upstream.csv\n```"
  },
  {
    "path": "pingora-ketama/test-data/nginx.conf",
    "content": "events {}\nhttp {\n    log_format upper '$request_uri,$upstream_addr';\n\n    upstream uppers {\n        hash $request_uri consistent;\n\n        server 10.0.0.1:443 weight=100 max_fails=0;\n        server 10.0.0.2:443 weight=100 max_fails=0;\n        server 10.0.0.3:443 weight=100 max_fails=0;\n        server 10.0.0.4:443 weight=100 max_fails=0;\n        server 10.0.0.5:443 weight=100 max_fails=0;\n        server 10.0.0.6:443 weight=100 max_fails=0;\n        server 10.0.0.7:443 weight=100 max_fails=0;\n        server 10.0.0.8:443 weight=100 max_fails=0;\n        server 10.0.0.9:443 weight=100 max_fails=0;\n    }\n\n    server {\n        listen 127.0.0.1:8080;\n\n        location / {\n            access_log /tmp/nginx-ketama/logs/access.log upper;\n            proxy_connect_timeout 5ms;\n            proxy_next_upstream off;\n            proxy_pass http://uppers;\n        }\n    }\n}"
  },
  {
    "path": "pingora-ketama/test-data/sample-nginx-upstream.csv",
    "content": "/81fa1d251d605775d647b5b55565e71526d4cef6,10.0.0.7:443\n/2fec328e6ccdda6a7edf329f9f780e546ea183b4,10.0.0.5:443\n/19fb835d90883a6263ec4279c6da184e3f1a79b2,10.0.0.4:443\n/da7a88e542f7aaddc074f988164b9df7e5f7fea6,10.0.0.4:443\n/8f87cfd8005306643b6528b3d4125cf005139a7e,10.0.0.5:443\n/26d2769eab098458bc3e4e641a4b7d8abffd0aea,10.0.0.6:443\n/aa5b5323980f2d3e21246212ebd820c3949c1e88,10.0.0.7:443\n/d9c4bc3cc4517c629e8f4c911c2fd8baf260ae65,10.0.0.1:443\n/28c1c069a2904bb3b3e0f9731b1ff8de9ab7a76d,10.0.0.4:443\n/fe5199bdfeee5cd431ae7e9f77f178164f9995a0,10.0.0.9:443\n/43992eee187920c5e8695332f71ca6e23ef6ac4b,10.0.0.3:443\n/38528aab753a6f32de86b5a7acdbb0c885137a81,10.0.0.9:443\n/12d4b9155ff599c0ac554226796b58a2278b450f,10.0.0.7:443\n/9c34c9a4f9009997dd29c6e6a627b0aca7beb6e5,10.0.0.5:443\n/eb5a2ab55796afd673874fd7560f1329be5540bd,10.0.0.9:443\n/ad7b5395766b77098c3f212043650a805b622ffe,10.0.0.3:443\n/c72fedf4177499635302849496898fe4f3409cc1,10.0.0.9:443\n/77766138aaf0c016bdd1f6b996177fc8ca1d2204,10.0.0.8:443\n/860c86b94e04f2648fb164c87fd6166707fd08ff,10.0.0.6:443\n/1b419454e4eb63ef915e8e06cc11110a3ccd607e,10.0.0.7:443\n/a8762dc488e1a1af31e53af8ddb887d4f3cca990,10.0.0.8:443\n/2e8e8e8fdeada0bbd33ba57d20209b4d9343f965,10.0.0.4:443\n/0220fa8b9a256e7fcf823097759aa3c44e6390e3,10.0.0.6:443\n/418c1c554186b78c11de89227fbc24ef128bce54,10.0.0.8:443\n/bc86e565b76f8e6f560064b02ab26529b6064571,10.0.0.3:443\n/5c6a9b50df69956bd2b937ce7871ba6d67678db6,10.0.0.5:443\n/5726f95dd0b1b145ad1a06755580f42fea41ac2a,10.0.0.9:443\n/db601a7f7e24504b820e5ef5276b2653ec6c17d9,10.0.0.4:443\n/f428a38a0d3dbbb12d475aa8f5be917147175eaf,10.0.0.6:443\n/b815ca5871d52098946eded8a3382d086747818f,10.0.0.1:443\n/fc61e21e21c6c0a9e03807a2cad7c1e79a104786,10.0.0.1:443\n/8278c52b97c1e805c1c7c1a62123ca0a87e2ea2a,10.0.0.8:443\n/668fd6d99bfb50b85b0928a8915761be2ca19089,10.0.0.2:443\n/fefbfb22035c938b44d305dbb71b11d531257af8,10.0.0.2:443\n/c30b287269464a75cf76a603145a7e44b83c8bde,10.0.0.5:443\n/7584dbc60619230cb5a315cfdd3760fe2e2980c3,10.0.0.9:443\n/399b3bdce88319bdba1b6b310cfcbd9db9cec234,10.0.0.6:443\n/5edc91979f6f38dbbe00544d97d617b92b3df93d,10.0.0.9:443\n/ac740e2450803d9b6042a3a98e5fe16eaad536e6,10.0.0.1:443\n/46013f26dbbde9c25de5fcbb92ff331d5614bae8,10.0.0.5:443\n/f109862c7c78e8ce087aeff9f2368d54d91fd3be,10.0.0.5:443\n/fdc13a7011bbcf36b232adde4c610f0f35e9147e,10.0.0.3:443\n/8387a3c076e525cae448c6a3b22988a2f37a98fc,10.0.0.1:443\n/b4739e36d8e7eba1a400925c928caf0741b1a92a,10.0.0.1:443\n/d92612bb3f678d8b181fa176e0af7227bf5f7e42,10.0.0.9:443\n/89ec56b1d8d72c888b044e8cd7fa51b9ac726a41,10.0.0.2:443\n/7cf921d8181af6912676f20c3d961d3f2ffbad20,10.0.0.3:443\n/9181876c839cf16fd7c8c858b7afdc0178fb9500,10.0.0.3:443\n/1034a4394566c826888f813af75c396fe8082b43,10.0.0.3:443\n/81ac831667e89c2c6b3c6098b598d99eb1ce2b20,10.0.0.2:443\n/d9dbae8a03a430b8d9cbffcf622b4e379bc89bf6,10.0.0.7:443\n/c67776793fdcf7553fe0cb6414bb9dafe0216911,10.0.0.6:443\n/1ee25559aa4aaa11ec1b3d2cc8645ed05ec001b3,10.0.0.9:443\n/580180a2b85efff1a393ea2449ae271148ca2770,10.0.0.2:443\n/84e1a1904a52e43ace344346032daca4e1bb69d6,10.0.0.8:443\n/9cd06ffa608a252a30d935d2ebf10eceda06ba2e,10.0.0.6:443\n/cf85a0000f38ac5346ddddd8cc0c28a054bbe60c,10.0.0.5:443\n/c31f22b05514e380dd4430086486dc3ba4e36ed4,10.0.0.6:443\n/336fdd336fde2bde2e0132d4be65088953036175,10.0.0.7:443\n/cb1e7e2c425607defdd725e81ca3121340dbc8bb,10.0.0.8:443\n/7bd85bb6826eeb30a67a999bfdeb6f6368954a3d,10.0.0.5:443\n/bb542ca4f154437b0fa394b3be8d45350efc4955,10.0.0.8:443\n/53e425848829e3aeb1c6991512e1951145b2ce46,10.0.0.6:443\n/a6ad65c1bcacb876b76165e741f35c98a09cbbf3,10.0.0.3:443\n/1fca16e96a89623e2ef7a93fccd767c4ef2a7905,10.0.0.9:443\n/b9ad129954c11aa1491552845199c2fb4bbff25e,10.0.0.2:443\n/9c0380f918aeb44664929447077ee992894cb754,10.0.0.9:443\n/a9aeb4e3fb0b2358f70a6d9c2ad62409a7c24574,10.0.0.5:443\n/8d563416df0c167343d295f889d64dd9ff213a9e,10.0.0.7:443\n/71ddc6cc8f25f63ad7df9ad963beb9a14ca6b76f,10.0.0.2:443\n/1dd61ea19da5970147129b0ba635338bc93c7aba,10.0.0.7:443\n/2c019dd0aebfdf9d94fb1201b25f443c91c034f8,10.0.0.8:443\n/636b620e6d548492a0fac32e895fa64ab48fa70d,10.0.0.1:443\n/e26420a446174c0bcbc008f3d8ce97570d55619e,10.0.0.7:443\n/2522d660a63527ab2f74c7a167366bbb0bc46cb1,10.0.0.6:443\n/6e585c3e88aeb95554f5c00730c70d71189a12c6,10.0.0.1:443\n/0bc50da77b7cf3959612950d97564e91e5a0f3fa,10.0.0.9:443\n/167872e2688593c6544c0855b76a99fd0f96bb69,10.0.0.8:443\n/7842aa002d2416c4587d779bbea40f5983883a9d,10.0.0.1:443\n/b3cdb310440af5a8a9788534e2a44e1df75fc0aa,10.0.0.2:443\n/7c17fc177496c13dd1207388087ae1979603c886,10.0.0.5:443\n/28865c3daa92ec1e3784c51e9aa70c78b902dfa6,10.0.0.3:443\n/4b990fc439195c5e05cfea65a2453f23fc5bbf1a,10.0.0.5:443\n/7261021a69a6478b0620315c231c4aa26fda2638,10.0.0.2:443\n/d5caa3e251ad2dd28ba82c3dcb99bff6d368e2a0,10.0.0.1:443\n/a8606508d178e519aa53f989ef60db8a0f3a2c2c,10.0.0.2:443\n/eb797fcf3e5954c884b78360247e38566f7f674a,10.0.0.9:443\n/289ced7bea19beee166cf4b07d31c8461975d4e4,10.0.0.6:443\n/e563ce7e72b68097a6432f68f86ed6f40d040ac3,10.0.0.3:443\n/ba22b6f2657746d3b8f802ab2303ffd4b040a73f,10.0.0.7:443\n/5dbda23f45eb02ecc74e57905b9dc6eab6d9770c,10.0.0.9:443\n/637691e12da247452c3a614f560001e263a9f85e,10.0.0.5:443\n/b2e491e1528813c17dfc888c5039c9e3f40f9040,10.0.0.8:443\n/a4575d09e2fcb4d42e214c33be25c2f1c10e8323,10.0.0.5:443\n/d655e051b4f82c459b20afbd2ccca058e16ad3fa,10.0.0.2:443\n/cdca39ce5deb7022702e18e0c6b61010ba931e54,10.0.0.9:443\n/58b31129208a29d2435258dc9f24a6b851ed1ac0,10.0.0.6:443\n/019930f0699b20a72a091c1042dfe33ac568b190,10.0.0.5:443\n/f00117302e2daca8c81e68cb33cf445b72c45895,10.0.0.9:443\n/da90cf74593ee181693910a40142bc79479c354e,10.0.0.5:443\n/87654ba6f96f359e4418b3368ae2256a3c2dad51,10.0.0.2:443\n/e85d0e6a90433b5a64257469c2cb4e441f39d07c,10.0.0.3:443\n/8527e42c8677b3f8264a2a647c00eb3acc5d0207,10.0.0.1:443\n/3adbb76ad6ae8a5342a5458e5f41ac4bdddb45fb,10.0.0.5:443\n/96e7ecedc6c60f0b52869a98f9d192af1e72d329,10.0.0.9:443\n/430095d6c47a7d2a8073e73df1c694fc9065e8f3,10.0.0.4:443\n/475ce23ca92e83ebfbc781aa337063c6b034bfb6,10.0.0.3:443\n/3a2cd1836406244cf08a552f60734872cfabfa1d,10.0.0.4:443\n/47372a5cf6b640c32681f094dd588fa204839637,10.0.0.1:443\n/74d7ecd706817756952727e82a5933549d582f68,10.0.0.4:443\n/0c1ab68f17265ddc9a58577f2a3443b523508d2a,10.0.0.3:443\n/e72871b3b2e08e87443995810c8fc542ec0c3b88,10.0.0.7:443\n/20ffdb8b43d521aee3c81cbb668b94828bf3f86d,10.0.0.9:443\n/b9a4b7d390a4fb62ea6252287351954ce6935fd2,10.0.0.9:443\n/71f52570d9fa32e2df99088e44850fa9097804ec,10.0.0.6:443\n/9533af016368e423dc90b4e249002233fa3fcd06,10.0.0.8:443\n/23992435c60a48db0188097fb2f15826d99be05f,10.0.0.1:443\n/bc351d376bcd7338aca33255199bfa3ced51d66b,10.0.0.5:443\n/bc5a14bccb994f346069886be05ba91dc4cefacd,10.0.0.4:443\n/6a29ff380492b77fe69f9ec0851cbbf7228d62f3,10.0.0.8:443\n/99bbb0675c38e292e979110ac88fc7711edc92a2,10.0.0.7:443\n/786105dc60dfffc8e2ea58679a14fd4428570d10,10.0.0.4:443\n/d983235f5af78dc9b13a5d177a44c6a76c8fbb2c,10.0.0.8:443\n/55163e01bc026cab4cf6985c8c2583876680aa80,10.0.0.2:443\n/eb68e3145c8a531198ea2a60e7a4fe6cb1a2b78f,10.0.0.6:443\n/7996a420a8e08545583a8ca0941c1a0c9ddc875c,10.0.0.9:443\n/d8d3509e8df61eff246be4faa6630d5f11b81172,10.0.0.4:443\n/ecd74f84dadcbb5e7ab90430ba424a996a5ec50f,10.0.0.7:443\n/566ca8a48b0875bdf60d224188b0d952da6c8dc7,10.0.0.5:443\n/0497f891fd6d35ffc0ed28dd3ba17eeba1301fa0,10.0.0.2:443\n/6a406d220cbda7fad4facc04632fd0c12dc6d998,10.0.0.4:443\n/3a54c0bfc41cd0942d0e479430cdbc551e33fb99,10.0.0.9:443\n/a7a224cf1e0d9b4e5493b2f61fa53ad72de58b94,10.0.0.6:443\n/4121200fe9e4e7c2126c5d71d108e5119f37783a,10.0.0.4:443\n/caf4c4b46875bbfa63b9ab35a4bce5646ebd55b4,10.0.0.3:443\n/90ad2be0a253536ab7c3e961443a91ded0e66e61,10.0.0.1:443\n/caf569f41f3556f588fefc887d6ec0d454bfef8c,10.0.0.9:443\n/0e3c3e157ffefdfa94e785d4a55f4eb6fca4dc70,10.0.0.2:443\n/b0b8ba29e45725715f7982a05edac1ff999a7899,10.0.0.3:443\n/cc5430ac1220fe146e68e9cf6f174269d403224d,10.0.0.7:443\n/508445e1be7b2b4495f2eb5907530bb095e98ea7,10.0.0.5:443\n/d6169d6f2495da4842a67163dcc0e5f31acb1a0c,10.0.0.3:443\n/8d85ea8d983c0e35836b8a203660c6c919da645d,10.0.0.8:443\n/ee5128bf7f95196d6569af52c9d99c4d60f132c6,10.0.0.7:443\n/461d5e76ae9d26244e546eed7038efe6cf7d9bbd,10.0.0.2:443\n/9f97615d8e9dea23c4c4e841838404fcd8698d8e,10.0.0.6:443\n/c01e055c153b1d34d51c6598e2e1c3fc362d812e,10.0.0.8:443\n/7c087772081d068f5fd86960e4d89901f3c06afe,10.0.0.2:443\n/37e6e5c96c2661d244cbd243151f9c90119d5f4a,10.0.0.4:443\n/663e532894288bb97751dda93f151d85f6c16813,10.0.0.7:443\n/2b3904fd38fc96f184226c842f0643cd0596d865,10.0.0.3:443\n/14cb69e56f7f17a26f0bdfce16dec5baf539dba0,10.0.0.8:443\n/adbe42c7ca6dd63d976f49262cf3d1a27a5f7bb0,10.0.0.2:443\n/70b58e27d6eb735c3c82d9aec1f6608f2f32195f,10.0.0.3:443\n/e7d3683cca1dcc45d8e3fdfb54eddc9b34141d65,10.0.0.3:443\n/407e3958ae8b94172af71487050ef5dc0aeab2ac,10.0.0.6:443\n/4c5af9e573fc3e0120d322a950fcbb792074d670,10.0.0.7:443\n/fe92a691ba1d11d6f49e5144be9baee390cc27e6,10.0.0.9:443\n/298835604d35f371a68e93047c699a7c41375f97,10.0.0.6:443\n/2155470425069f357851ba81346b879a8193aebb,10.0.0.3:443\n/f55d45d265ec44be7ded0db1252281348fab75f0,10.0.0.4:443\n/798f665aa334e5eb9a49669785e94da933d81f32,10.0.0.8:443\n/ad8bf2624e7fc687b0130b61fdee9db2a2d865fd,10.0.0.7:443\n/d2002a4943563ca4c4fc66b4ad65aac4e1410b2e,10.0.0.2:443\n/a025e91fc9b3fcdc0491d0e4b4b0f09e322e53eb,10.0.0.6:443\n/b4a46e8f0ca5698b4f6dd201b87e88125b153ece,10.0.0.4:443\n/ff2a4976667b127ca1e3bb5027e8a836e56fd358,10.0.0.2:443\n/307086130cdefaa3d899fca3dd9e77047fff1cf7,10.0.0.5:443\n/558d5eeb99c6f1cfd6367fb101392072e5140c44,10.0.0.7:443\n/a1a3799079c1ef01be067c4c6a1db5b7fe6515b1,10.0.0.4:443\n/5b66932db9324bb9f8d6fc1f7be819c1c1ff43bd,10.0.0.5:443\n/1d69b12d308183c0d6432fb4cb8bacbc86193830,10.0.0.8:443\n/eef4c8b2ded3656c9d6174a72ffc487f0c769492,10.0.0.2:443\n/eb439a2cd0e4c9fdd95d8c0f657a81ce20f96a0e,10.0.0.2:443\n/b6f64c4a87c0d38417ce3dcc7a553a185df7f384,10.0.0.8:443\n/393d62711ecc6309a19a96ea73cffae546922f64,10.0.0.8:443\n/aa18663a595f369e048e33505f82d21ebbfe354d,10.0.0.9:443\n/759754a69ee3e4449bacd21a5866b8434b743cfe,10.0.0.1:443\n/c01e96c10fd69b430cf67edcc3fd2fec7ba30097,10.0.0.4:443\n/284e0c7dbb8e7da2a1fd7180f8d542fbf2410767,10.0.0.3:443\n/6f360332b72940cc117999224b5be35551a1790a,10.0.0.5:443\n/a83eee32d7132975d5d2d2848bc7881345e63735,10.0.0.6:443\n/9d8bfc97428dee1b1495d2568e5ac68b8ec7973d,10.0.0.1:443\n/9e09d80d5653ac55445b42c091ada230ed96cf67,10.0.0.4:443\n/6ca8d4fd764a20ca1b766f9d2a14b81011d80da4,10.0.0.5:443\n/fb89be9d12828716f95a60d092f2a028c876259a,10.0.0.1:443\n/29ffb1d20ace9afed20ce8613a2b636dae70638f,10.0.0.6:443\n/b569fa1c31949a8ab05a60939d44b1132534556d,10.0.0.7:443\n/71a89db0bb322607a2557b089a5d160fa574fc7d,10.0.0.1:443\n/4449e3e6404cecdc9a36ecff54babedc84619b1c,10.0.0.2:443\n/b26294352e342bd6e953264f9e14393413bb371d,10.0.0.2:443\n/a72621f8691cf08ffdc5884556d5512a5ecd1f6e,10.0.0.4:443\n/dc4732cfa991632b719def815b228ded96abaa1e,10.0.0.5:443\n/b908128cca7c859493155441660eaaa09b2fae80,10.0.0.1:443\n/d93c9304c07c8f1d2b6f6c89c882fc2cfad3fefe,10.0.0.4:443\n/8a0db29dc8df0b7845a9ab213d4bd8ac59a121e8,10.0.0.7:443\n/49559040bdef5e1a5dc8ee89f897b79115ef1bfe,10.0.0.7:443\n/23428c6b465b7c43629bc28fa1a7431c6e541778,10.0.0.9:443\n/9db1610e40a3197a5b8c2d0dee2b2ccfe4cabb92,10.0.0.3:443\n/1c6cf23cac024d126066771bae7af48ba141dfd9,10.0.0.1:443\n/5e89a982f7f165b47fef959e10c32afa1e01783e,10.0.0.1:443\n/52644098601b604c9e9e5e3d1150f13e81240fc8,10.0.0.8:443\n/1771afea4cf491711aa3b608fbd8b470306d7bc9,10.0.0.4:443\n/825cb4d51b986eef44d3cba31dd87c4ce3d9c159,10.0.0.4:443\n/83a6211a968db8d62e17525ce593c144ed7fbb4c,10.0.0.4:443\n/6a9abd46a919eed40be39b9d53bd73cb74acf540,10.0.0.6:443\n/12db006d907a255f8d61e5070d1a41defdae27ba,10.0.0.2:443\n/0cf51c79b9d115d7be8fcc104e2f51fee1a3caa6,10.0.0.5:443\n/6bbab5e098876a84c403ef8cbe9864c21f9bb0aa,10.0.0.4:443\n/5fc725bf869cf190f8ce82814d5e8e749030c8cf,10.0.0.1:443\n/859d96b17c00e528c07fe1696fc7ddfdb34c4875,10.0.0.7:443\n/a55638df8b2ceca37d24bb78826833deb633c79d,10.0.0.2:443\n/70ed2f73f55d4d00f9cf694a7f669c3ba11f89ed,10.0.0.3:443\n/b5c910057d813197f8353c31d233de719212455a,10.0.0.5:443\n/b602d274d7d8ff89505fb3ba364b6ccbeeb561ab,10.0.0.6:443\n/50ba092d17178b78c2643e798138ff5514d2d0a2,10.0.0.1:443\n/bf3244d6cec5c60aa29ccca799415354607b7803,10.0.0.9:443\n/7f4ddcc20818c0db3cdd8b440c269e33ef22a7c7,10.0.0.4:443\n/9dc2eaaf3539a7c0a5b97be1f722f544539c6257,10.0.0.2:443\n/c5359e50f3c202f5cd5c096bd15d757ba659e815,10.0.0.5:443\n/038366c13ffa60a0d9ef4bef212e6e7354a6bbfa,10.0.0.8:443\n/9e40dac2f57fe43878519a83af3b75fc2e590217,10.0.0.6:443\n/9b2c05c1d561f86cf9682673628dfef2160650a8,10.0.0.5:443\n/78a2ea21a979d1d0c8e07f0185f358fe58393c12,10.0.0.5:443\n/83d46e2ff9cd7bb557c1b00533a0e4f1733df84b,10.0.0.1:443\n/29bf196e578a83824c55b0f78ceab36b1eb9c82b,10.0.0.4:443\n/61249cd3d39f4dae802db5f0a875a5a4a8ad191d,10.0.0.1:443\n/c7c7dfdf8e9e68d5540aae13b2cbb5fe86c1b965,10.0.0.6:443\n/4be4e8d7897f7d9dfa210bd236e9bb45454fea20,10.0.0.9:443\n/cb5ed875dedef2013fab5b051a8636d10fef56dc,10.0.0.6:443\n/e12ec1f2b657ad0f7988db38254652e153525ad9,10.0.0.7:443\n/9ec5a64e415451efcc8aa7648b284774361e03eb,10.0.0.7:443\n/3a6afe9c8e8f041a59695055cb7733ae254632bd,10.0.0.7:443\n/e3393950cb37481a7b00cbefc3298d14aeda0807,10.0.0.3:443\n/7c6e41537748edb49cfc56ee505256f40935a99e,10.0.0.3:443\n/6bbc445ff57bc9c54407f31616f1b23bf5ee27ce,10.0.0.5:443\n/99ba1e8f21532dab31caf0731f1c5edc8455550b,10.0.0.5:443\n/725fbb619d38c436bb88e28d5219e720989ab6db,10.0.0.4:443\n/7b519ba8928f440bf01ac1d6b98611fb59bb1c89,10.0.0.8:443\n/2ff8d8dd2a37ff1cb34692a00c7fb7d1c155b419,10.0.0.3:443\n/f76abffc1a71b95e7969cceaad57429672beaf68,10.0.0.3:443\n/fe58d58e116026db4cf106ef57732e1b629caade,10.0.0.6:443\n/45549ca0d7c95e97c299b58b03ecf1939e140c9c,10.0.0.1:443\n/93695453157442d799a007d1710f7dbf968be8f1,10.0.0.9:443\n/ebe69b2ea9db3e66a2157021a17f852695eab8be,10.0.0.4:443\n/a885aecaaf297eaac5c98ed708fe6a73fc9273b8,10.0.0.2:443\n/2859256b987358b8d2ee0c81b5494cde3a98d602,10.0.0.1:443\n/d19ae90e456730d2db6b36c1ed1a45335b368fc1,10.0.0.1:443\n/f16f2e87bee62b1523dbb5824b5dfe338ec67704,10.0.0.8:443\n/fcd5f91888014decb190a9dac5fe9fca7ed8d70f,10.0.0.9:443\n/3ee610b32554b5f7c27d40a52bb982378ceb4fb6,10.0.0.7:443\n/21cc5cb90ba59b6b743bc437f0f93c45d21aaea9,10.0.0.7:443\n/8d2bffec729dd863e6dcdaeaefca22d6e29403bb,10.0.0.2:443\n/2ce6b015ea081b69a3867f7b09b753f83fbd4b77,10.0.0.9:443\n/64fcc9606275d6a259a084696318ab704a81932b,10.0.0.5:443\n/0984409349566b9bda3f5ff3b0dae93c6979969c,10.0.0.6:443\n/2b3775815cd0064c1603ec6dfe62b9ff54180638,10.0.0.5:443\n/563ff0fa8762400c92ccb700adb6ea6a7bfb0d33,10.0.0.6:443\n/901f7c9eca3f038ecf6a684a2c46b827c24e8ee6,10.0.0.5:443\n/3dbd852fb7f851fda48f742488e51dfd8d4a472e,10.0.0.9:443\n/a50ef8903707c1c5d7158a851d636ef65e198e7a,10.0.0.7:443\n/92603aec7e7f7a5847f523c336bd80d786667d6f,10.0.0.4:443\n/a941b070f313629549a2874fef17b29b25069214,10.0.0.1:443\n/9a80624738b37b3a3d6b0749feae2bb82d0672c0,10.0.0.5:443\n/f863b682f5f260f4762a14831d949c5dc9bd5f28,10.0.0.7:443\n/d41f6919aa10ee037b4a69df874de03ccfc6432a,10.0.0.4:443\n/e995303d36162db8650a2802ce0d52263c29ec0c,10.0.0.1:443\n/7823ceab6e649edbb4f99d62282fe00edbe3acca,10.0.0.2:443\n/bfd84f41dfe1d4470730d0aa41eb73b9d7461503,10.0.0.1:443\n/53f7534ee600e63d0b32bbc1f2f9e4794373c4bb,10.0.0.7:443\n/26f4c39897fdec0b453bc15860a45137064c4ef8,10.0.0.8:443\n/7345179e10fa47e31faf60e165e7802f31315c56,10.0.0.8:443\n/d47e4a2590ff8d5dd916d826adc3c20b9224a3de,10.0.0.8:443\n/8ebb8b58c53468143f882b186fb64ef14e962c0a,10.0.0.4:443\n/7fa7b9821ce360682b88b07fa27158af8d4b10bd,10.0.0.8:443\n/7d3b908d960f61cf4944ac52164eaf9890c17c47,10.0.0.3:443\n/3900dbeff282a20a6dc0b450581ae27f44230f75,10.0.0.7:443\n/327a041d0576f11ba4c0fc677a8b1fa7cdd5b215,10.0.0.6:443\n/20450e190c6b829846d1a67e43b2e57cf7e5b472,10.0.0.4:443\n/d6d97ddf81c5a8f4f11b87198a3f8e75814d09ae,10.0.0.9:443\n/48a468d706a7cc4b07c0e74695a9c2f64012b02a,10.0.0.2:443\n/35903e2f79bf054b45d9f342642d488b85ec086f,10.0.0.2:443\n/4198c731ac8a3638a955ae891498ea4071b2be10,10.0.0.7:443\n/575be0ba8f57b2650f53499ab19fcf10aca1a467,10.0.0.9:443\n/c211460d038ae3aeb286e759dbe99b9084c56fc1,10.0.0.6:443\n/7d5071d6ed21ce66d8887ee6f88bf8b3145d417d,10.0.0.4:443\n/77435459761c415127dac0d314fe73b728e93816,10.0.0.2:443\n/16a401100431531a7cd8528d6ea8f957df584e4f,10.0.0.4:443\n/9b9af306b3fb801bc4cb127118aee22f4678c6d0,10.0.0.2:443\n/4902696d40151e903ec5bf810f2b82af7bf92799,10.0.0.3:443\n/3207830ce45f38a326cba44a2bbed7ea7009e7f1,10.0.0.8:443\n/002655dd3e576dd2be046915f365ee7947c77553,10.0.0.7:443\n/8a316dc9861784929ae9283ff9edf50fcf2abb77,10.0.0.7:443\n/8b2639c2cf4f75723ae219f9c8a60779e93b3a50,10.0.0.2:443\n/d135a3f32a0eaec83386f9b8167c8b351fa0f9cd,10.0.0.5:443\n/3cb5d50669030262c50b916b5e5f0ff112a23f87,10.0.0.5:443\n/791d86b7b2c2860da849c6e20006b3f5f92714a1,10.0.0.3:443\n/de08fde7bd93bfd844407842d09bc163675fbcbb,10.0.0.2:443\n/0576ce89f317cf54673e20eb664bb8992c975a71,10.0.0.9:443\n/fd7244f5203e2985e6c65ee07686cbd2a489e21c,10.0.0.6:443\n/233de62d4ed3f6e6a8d847500ed8be500970bd0e,10.0.0.8:443\n/8b8ec68415a7a9cbc426c23ba98ad165a434fa1c,10.0.0.6:443\n/ec4230ec3e8fa6600907e777c94f2e59382b4542,10.0.0.7:443\n/bd220769eedf9c7efa641de459810048891e3dc6,10.0.0.3:443\n/9165254e59f4fad93b93a02210b25dbcaac4e0be,10.0.0.1:443\n/0adb6ec07cfcd61534a065db496c7042e97391fe,10.0.0.5:443\n/39a5c89484e21a243c7061d39dbd236c80d4ede3,10.0.0.3:443\n/dc560955b3b817db3e79e37255cd18bd66a39a22,10.0.0.9:443\n/9d433be2cca7907dae1b8c24900edc5adb6065bf,10.0.0.2:443\n/2531e51eda6b68cc2faa7a09ad032387b2676523,10.0.0.9:443\n/d591b928b7f89b00458ea30ef6f4fd20cd7e41c2,10.0.0.7:443\n/9720475f8d148f70245ade435243bfba5a1ba559,10.0.0.4:443\n/2544d73a3c1b0f04829284a5b425607f4f61ced7,10.0.0.2:443\n/e3af59332ba621011d98fbf2a38c8a0b69b9ca79,10.0.0.5:443\n/d8f4c58c0db28d7368f453d41abd59f6999b3ccc,10.0.0.8:443\n/8a6180a589aec21a274a3f47781ccb1311b0833f,10.0.0.6:443\n/83aca0a94c4883adb8e7ff795c1008ed59052691,10.0.0.7:443\n/adc1ab7741effd4ece0e832c41d1fe69f5e1805c,10.0.0.6:443\n/35a50236b60e680d3d968ad3857525a8649fd6a7,10.0.0.1:443\n/30495b101ac5318458d74b3a286527e164efac53,10.0.0.9:443\n/e2067038a82745a65406516f15817b63e328a825,10.0.0.2:443\n/754c3e717cf1640d11ee1ca113571fd0ae55a0c2,10.0.0.9:443\n/c4462289d891b8b0c0783041044908bd347a27a7,10.0.0.1:443\n/68129869b04bc2255d2a17ce01afb14f1be73032,10.0.0.3:443\n/683a4f8e369c5c3eeb85f0779aced10809bbdbb8,10.0.0.8:443\n/8da89d7686976482713413835c889a7f289174a7,10.0.0.3:443\n/511bef26cd42422c6f0c9bd33714a07b06dbb3e1,10.0.0.1:443\n/b871c4e41b3eababead2aa4dbea87fef7161affa,10.0.0.8:443\n/808da0dc7d4025a0858eec92ac72e9ffbff233c4,10.0.0.2:443\n/6fee636398f916c4ba0074fc327f7b3dbf683a8b,10.0.0.2:443\n/03ce09b1a7c7ae719f66f489841d0ff11635ffc5,10.0.0.3:443\n/8653f568fca5173d6d274060692912676709981f,10.0.0.9:443\n/8f8402b8bba56124ec6de0552c1844bd76bf72ea,10.0.0.4:443\n/2c89ed1b71a52c9b0a9fa7909dc87d4d06237216,10.0.0.1:443\n/5279a73f9dcfa562f13180791932598ed8a067f1,10.0.0.9:443\n/456482cc45669a59fe8af5e49648a8079bc35c06,10.0.0.4:443\n/90350e060dff6a507e69bd80c38629a2d9bf12b9,10.0.0.1:443\n/2d5c624c50ba3ae06782861bba176e9b2f45f529,10.0.0.8:443\n/67077af97e65ae301e88c6cd0e87c7ddb68fa9ef,10.0.0.6:443\n/f017e954321efbfe4046942be0a1122d9be81d52,10.0.0.9:443\n/0b74a181ce4b4f43023e4bc0acd7770f2867572f,10.0.0.5:443\n/f40f8509ac9f73516224825e88a220ca02db2d81,10.0.0.7:443\n/c2f12ede0ced03c9357a4fc5e05e9af5652433c4,10.0.0.9:443\n/d3ec86f1dde7e9c416c88ddbabf854e21decec2d,10.0.0.7:443\n/ea29487fa7e1c9e79ff0f257bfd8241736ddab9f,10.0.0.7:443\n/b8f4ec5dee59a8693710cb95e1734900d7b6b076,10.0.0.4:443\n/d0be164540802b86de0762ea266e03c8859ff70d,10.0.0.8:443\n/cab9ed312a56db577bc36e4c2f52e84f8abb09ce,10.0.0.7:443\n/62dfa34389964b03792842c09adce33e7decc837,10.0.0.5:443\n/f653827f1289ae68efd5a0d057fbc172f8352842,10.0.0.8:443\n/dbd9b9cf5affa501ddcd1a19eefa4240e311f94a,10.0.0.5:443\n/3e74e167ef6393b6544b4e75da97f30f6c2e6477,10.0.0.6:443\n/d004b31247c668c439fc8e491f71a69dfd35a55b,10.0.0.6:443\n/17e79540d401ae73e7d666444feececf64602d23,10.0.0.8:443\n/06c9cb78908d623842c4a7c7baae3d55009ffc43,10.0.0.1:443\n/64427a2e50196a34670b9de8a4aebe44cbb26cc5,10.0.0.2:443\n/802741e276f10186ee9b63d47af006e8bc3de516,10.0.0.1:443\n/10cea480539356be3f2a2f14c05f057a60ef9b10,10.0.0.5:443\n/17d5b6820d78727b781be06cbc7cd2a9be650794,10.0.0.8:443\n/1c0f7ec3d8919ffd2ccb3312fb7d6d2e15cd3133,10.0.0.2:443\n/ef3afb81312d46b826f033d9adf0c730996e7992,10.0.0.7:443\n/080e45e7955e797bdc906af2fabeb8fbf2ac48e1,10.0.0.1:443\n/c043b1a590f09716da25328fe0573c8e2e9c0bdc,10.0.0.9:443\n/df604b478f31f11b4cb291b1a393749ce4e72ef3,10.0.0.7:443\n/32a8a7a0678c834e2cc7ec0584cd193fd1fd91e5,10.0.0.5:443\n/8303fbffee5f38f8eb4a51f3c1255de830abce34,10.0.0.5:443\n/24c16585ac0791c7aa1d8a16bea2fae9e7008cc5,10.0.0.5:443\n/0059d4d899d1e961f249a060d91e932a89bc9b4e,10.0.0.4:443\n/f1edc74510b17b10cdff8801776d4eaf72a1cf0c,10.0.0.2:443\n/faaec2162ff441c490fd2dc0640bf2c941438995,10.0.0.3:443\n/1159fc0cd3e683f92fc649aad3a4bcc34564c3a3,10.0.0.3:443\n/2ade445523d26c3b1519e699590939011036217c,10.0.0.7:443\n/cc6b2c1b20ef1a63adf2afaf1207ecf446fb5719,10.0.0.6:443\n/8cb65496c9ebd858fb1101d85cf35467c4a0be17,10.0.0.8:443\n/38a5bad44dfc1ee47585be27cbd9598959ea6caf,10.0.0.2:443\n/c651109ffc0f2270596065009642fe3fe0529e60,10.0.0.5:443\n/b71ae7d9c9d4b67677fec2e630313cd01cd130e5,10.0.0.2:443\n/088d49f180d9a2211708c95d5e9d6986705c8d7e,10.0.0.4:443\n/8e44a5bd8519aa7b6d85f01e7b01e1a2ca236b6c,10.0.0.6:443\n/2526eec4890f03477261584646a0ed6def65f8f4,10.0.0.8:443\n/3a70dff8627a20ee109fa8241d7da762a02e02b3,10.0.0.4:443\n/eb999a0198002b52abca1fd424a44013989e1403,10.0.0.4:443\n/22a12a10268929d77ef8dcdea96f8aa69ab92d8a,10.0.0.7:443\n/b467e43c88f0e69b52e3d341fcee52198f90cf77,10.0.0.1:443\n/c5245921b3f6a21adafd6598046957086edfbeea,10.0.0.7:443\n/6229b444f64a7529dd956877c24b2d149a1debf7,10.0.0.8:443\n/4f8b5505d9e817b39e0b68e196c62240acd07306,10.0.0.7:443\n/8699dc12aa122266013234df69eb5e14d6282174,10.0.0.5:443\n/094dc8a3111d132d294030b358c23e63ff2ad680,10.0.0.5:443\n/3ecdf7b29e567a94e1548f0884f414af6539f974,10.0.0.4:443\n/abb4e33382452c2f5c5d3d0b89f11cc2d497a3a9,10.0.0.9:443\n/696adbeedd54670bdcca6356597b580dd9c4c42c,10.0.0.8:443\n/940b2a895c5540f5dd079f7ba930962980bf4e77,10.0.0.7:443\n/e3ad3f730ef0f82530373e966fc35e7521ba0fae,10.0.0.5:443\n/e68f238fe1b9f3ebffb102e98ee2d958194f0c3e,10.0.0.2:443\n/9e09afd63284f764392213c9f282e59c859ccc2a,10.0.0.3:443\n/40d1f0a5f73460d07f777d16ce0edb5215e0af5b,10.0.0.5:443\n/e641c93505fff2131a1bd6a2bcdeccc7ef56e108,10.0.0.5:443\n/51f5ef58442e9176bf4ab0e3d2a31e3919314467,10.0.0.1:443\n/5c752e882a791057babbc7c9d0fcd6f98249a90c,10.0.0.3:443\n/1a93e7ac8192b94e62372c526a858203ab55f82c,10.0.0.9:443\n/4f00d996e102527cfd906afe596b1fd20b9587e2,10.0.0.7:443\n/cb33e35ff6809c1d9e87af168853c8779949df28,10.0.0.1:443\n/8238f15b1556e4c5a2cfd3024ca22ca0cc38cf75,10.0.0.7:443\n/4d9af33ca789f5675e0b3b7db89277d9f07fe487,10.0.0.9:443\n/31f11a12c1729fbf56e7924ddeaef596d9246ffb,10.0.0.2:443\n/dc5b466b915a7eb13aa3c28a6df11201defc4776,10.0.0.1:443\n/a83dcc91951b9b9bfc86dd414ab2378dcb74bdb8,10.0.0.8:443\n/68dcee7d8e5b5bf81c1acdec7dd97891ce3aca1c,10.0.0.3:443\n/661f218fa06f2b92f22e0ab81731ac4029f4adc2,10.0.0.7:443\n/cd29f2138dc83eeb5d4f16d7b13d0c81483959a2,10.0.0.9:443\n/d34d0b880b1e7499c2133f06e9373f5ea4e841ce,10.0.0.6:443\n/1fb666f29371b67fef8e44a9322c9e387261fa18,10.0.0.2:443\n/808b776dc196d1759afc93646bbfca2c3e8074b3,10.0.0.2:443\n/20888fbf1def43e3123b0a9e4eba7fd7d5f2a410,10.0.0.4:443\n/d7cd6d38cbd5b94c432e15cd2f7502dbb306d757,10.0.0.9:443\n/fc018aa6e4835271d8cd024ec3943115cf4e94b0,10.0.0.5:443\n/27dc43e54f5d904746a36b482123346f44293b51,10.0.0.5:443\n/ed60ccb18540ad211073c281aee0dfc31ccc942e,10.0.0.4:443\n/5fe5f6c1b4d13ba7200d6a89c456f60eb36690f3,10.0.0.2:443\n/e2e2680aff762da702475c5307a31779cab5595a,10.0.0.4:443\n/b9a3d570d0f60b98af06bb2bb8f20bfefa8bc4f6,10.0.0.2:443\n/6b476c8bb82dcf0a32e3999826a788c48fe83e7d,10.0.0.2:443\n/80a371eb8a4a18395199455f8bde883ee548e4ba,10.0.0.3:443\n/de87fa6ef77299db61ac048fc87ce6e3e39934bc,10.0.0.3:443\n/52c42c3136fef68070d143e62366e47b5de255cf,10.0.0.4:443\n/ce7ea948d35ff4f7409eb507cfd3fc6e3b7cf30a,10.0.0.6:443\n/53e772cf7a78bda717c98a08b9081a40a03392da,10.0.0.7:443\n/9bcb6eeb362e0f6c8c04325848cd93f1a4fb75b0,10.0.0.4:443\n/2ef946b7546ba83f528e4af5f60a585fa22cd5e9,10.0.0.8:443\n/a964b047adfe079bd4f39c5d79daac5317611bb2,10.0.0.2:443\n/2cfad7113f585441f4fd054df57e08aa3f7d3441,10.0.0.8:443\n/f5352d8e5f1e080bc70c241a8e65cb48690bf44d,10.0.0.8:443\n/7108e1b4b2a8d73f6a892bf394d6db68eab4b06b,10.0.0.8:443\n/5ce086abaaa907ea06ad20e029375d0a469f6688,10.0.0.3:443\n/e8c9dabb0ae5ff95db4dc58d5e5ad8f49dbe1467,10.0.0.8:443\n/3ea868d97d0f366875936dee5944d30f133a11ee,10.0.0.3:443\n/4c34d28c12494292d3eb30c70e2ee5158ae45bcf,10.0.0.3:443\n/b1367bfef3363e82dce33834cccd1b6178dd4e01,10.0.0.5:443\n/e563665cf03942d218f963ac225b9f65f6f47e44,10.0.0.4:443\n/ae4a64bd0638678f1f19b31f1610e800510308a8,10.0.0.3:443\n/581ee1ca9a6cad9786fe37cba35c4809410a902f,10.0.0.2:443\n/d8065f52a7d18810daaeef95f724c63a843f9354,10.0.0.7:443\n/461f1d57aab6fa3c1bf72f5aa724ed264647c4d3,10.0.0.9:443\n/70335780553fc4499c3d7b903b7e6fb6f06bc47b,10.0.0.5:443\n/9733a49859bbd9b971475bd40e4a2bf7d9bfd203,10.0.0.3:443\n/d3d04d2aa72354af7b52b344a07dcd22dca462b8,10.0.0.2:443\n/c5ea5378e9a319cc2a455d6e296dd3ca5aa12477,10.0.0.6:443\n/f20e309d5d7d97489cc0d5ad3737af32008968cb,10.0.0.6:443\n/33d387521125b7825c8480755efce6e298e936a1,10.0.0.7:443\n/a99b72b250bfb58ddf9d6fa9f805e2873ce0c229,10.0.0.8:443\n/c8997a1855b1b4d8c6fd0a9dc21c01ffd7a1aa0d,10.0.0.4:443\n/43b0d060a125208528829cbace2759c05655c8b2,10.0.0.5:443\n/e609d51a78b4e83d2b778fef69d00f07a66e586c,10.0.0.4:443\n/e62b9cead57079d3290ac764079b6466f7340c9f,10.0.0.2:443\n/d31c487a361be7abb28a0b872d87fb41a907dd2b,10.0.0.9:443\n/b17261bb106af0ee0cc6c710439eb26135dd5f5d,10.0.0.6:443\n/14d92585f8702ef9ff0fd000907f00ca1f6a458c,10.0.0.6:443\n/fe79953554bb5aaf34adc6928060af206e8fa993,10.0.0.2:443\n/cefd63669258fb86a920a65d6fd1d511c508d954,10.0.0.6:443\n/143c0259d1759575c859b902f4e56ef478624989,10.0.0.3:443\n/50315f90fc6f82d29d925e3947ba7c32f19b5611,10.0.0.2:443\n/95428f6c4dcfbb2a0c17a5de4861de01bd3f325f,10.0.0.7:443\n/5ae459d770c6446c4d33104734b7a0293b48ce32,10.0.0.1:443\n/7078c3416e5b78c65c318b3916b5be6d92896c88,10.0.0.4:443\n/a489b81fdc1b4020b4bd3af8b759466c9023cbe2,10.0.0.4:443\n/6a6b490f616c8b52c4548b3c7f46d527662bcb5e,10.0.0.4:443\n/a607e4a8ef834a9ba7575729460fb697f5247cf8,10.0.0.2:443\n/0ea9ba29578fd164f5942d037366c3c6768bdda2,10.0.0.4:443\n/d9ba5006e81ac5128c20823155f4cfe991598179,10.0.0.4:443\n/b691e1e01b2f64c9c584bcec928345657b95d293,10.0.0.7:443\n/44710040aaa84b18986e3f16bc18e3c6ae63169b,10.0.0.7:443\n/9ffc3d11bf5b67e6e316646e64f1b703dd79bd94,10.0.0.1:443\n/1e2905288bed5c9b4c3f03cd6daeede56c2ffada,10.0.0.1:443\n/1d54d79aa491a886dfda64a7780b04161d1481a0,10.0.0.4:443\n/318d2d5e7b7347704e8a0552cbcb01adaef758a9,10.0.0.7:443\n/22b7965fa39aba2195e1fdc1e845e01f378c3d51,10.0.0.8:443\n/3cb017f3b5edf03f68ba84c5e8671e959273dd69,10.0.0.5:443\n/60c48c28f0ffdbe1225b02ef2fabd26364126d73,10.0.0.9:443\n/50330c984bb2167761149c43868e27401c678dd8,10.0.0.7:443\n/ee359ac7b7e76cbc38f5a70cfb4a7323161e8519,10.0.0.9:443\n/6dc489dc87cb4c097812bcbffa666f1d3a06920b,10.0.0.6:443\n/b084378cffd7ba654dd33d13ae00915abddb5acb,10.0.0.8:443\n/de8d58cf5b06da64685d35ea3d85692dd98443ba,10.0.0.2:443\n/926ad3a9d1411e8f959f8a99270e534c2fcc60e7,10.0.0.1:443\n/6f5efb290d2fc1f49af390123a8d45242535ba6c,10.0.0.1:443\n/be0651f76c2d6b3fea26840eb42ff84603fdeb91,10.0.0.2:443\n/664dfe003d4a68cd9502992896f6698ac2930b2f,10.0.0.2:443\n/d5c1076ec326bbbf73e06d110ddec314bab4901e,10.0.0.6:443\n/12bb31b94377990b52936138e755d30e1df5c4ad,10.0.0.9:443\n/e4e3d4a7766b7ef0038e05c72d7c60c853902d78,10.0.0.6:443\n/f1ace48a0bd8b225c5b05a4f310ad6146caab520,10.0.0.4:443\n/d6f4e19fb0dd901454337ea0a62a671f7e731b1f,10.0.0.8:443\n/e7e149ae5cc3b8d3432a74d2a5eff884304f149a,10.0.0.8:443\n/55f9309e65ad34333635e2392170a2d9e8e80f62,10.0.0.4:443\n/90cc256a42fe5c65d3b1c261b6713aac2d668405,10.0.0.1:443\n/b9530da73b2c2287f52955290ec2dbc0e2cd197d,10.0.0.3:443\n/45a8ce0efc2197c63f1b0feddd1d0ea3fb62cb62,10.0.0.3:443\n/af5eb18f6287ead6b0d8d9d4f740916eef27b921,10.0.0.9:443\n/7d4c22a4d647cfc95bc563ce3f0d56217623ea40,10.0.0.8:443\n/6f6d1b8ccb3515ea7a4dfbba821b7e24f0af890e,10.0.0.8:443\n/448153aa0816ea4c11fd433e4011279a9e911319,10.0.0.9:443\n/e1db3b77b5891866a1bb0f136e9a3b6f1a8fd7ac,10.0.0.4:443\n/8e5ddebd1bc756119aaa55d408583bf07c6947f4,10.0.0.1:443\n/cb7c13c7fd7c12a0226dca64d01f154380b26ad5,10.0.0.1:443\n/5b5987f50249480e48251ab6f848580d4dc69372,10.0.0.6:443\n/a81f2d7b7e3640837acc6d0532aa52974371215d,10.0.0.6:443\n/4fb00e1ae798e403dee221592d1b63d8a23ec7ee,10.0.0.4:443\n/06ecb7c80ee19fc3dcdde75659cbf0fc1d03d5f7,10.0.0.1:443\n/4c3644f13f8b64a3919d4c8a53387f102f8efbf1,10.0.0.1:443\n/82d78d0be6134b4da73a81a442ce4c5277d23d8c,10.0.0.3:443\n/968765803dd3e6461c0ff78f95342c0c113e4991,10.0.0.5:443\n/b49bd6e5e66970eb7378c37dde945cf542601636,10.0.0.6:443\n/7789c5109794ebfb0fe72e3bdc2fc29baa58e537,10.0.0.8:443\n/29857840ad23d55c82c93cbf7971b416e118985b,10.0.0.7:443\n/9aab0c35922eadd6e1cf5512af5272b20ff87638,10.0.0.6:443\n/094cd8c29e8621928b163ba64c52752f9dfe5ce2,10.0.0.5:443\n/f56b7ae1b06eff812326334a68cee675f3bf390b,10.0.0.5:443\n/a5a029d6c0f758aab2eb8fb56594a0e2e118a8a4,10.0.0.4:443\n/fe56c1fc75e419f220ea6bf06af6f5321c895f6d,10.0.0.6:443\n/0d5b4bab87051120251a86b92db52e12582ffdd1,10.0.0.1:443\n/34f43611722818ef7baa47b8d07067658765aab7,10.0.0.4:443\n/e70ae66289144a1adae07ea70f507767d57e08d2,10.0.0.9:443\n/3f10a40788eb0581f47d1f7ba7c57cdd6beef74f,10.0.0.5:443\n/2d58723eabb6c5adfb1fa239e34c30da32f117a3,10.0.0.6:443\n/efc167a6366a437ba465c1fc71ec28341082ce94,10.0.0.3:443\n/0bfb80585b1a0fa49c2c863ab39dbe06599681b0,10.0.0.5:443\n/57febded047c2a5455d7662d4967ac0666fb58bf,10.0.0.8:443\n/f06e91822c1eb12e9c560453b2947d45c961ebd5,10.0.0.5:443\n/2292f8823bd4364769bfdd6ef106c6186082a345,10.0.0.1:443\n/dd37a086d19829dfff28195b3b63a197193f154c,10.0.0.1:443\n/74f336292ef253b07abdce03a0f0d47ac6482062,10.0.0.8:443\n/22eb9772d6538887bfc57bad65ef8a72d7ede1d6,10.0.0.7:443\n/a1df865f8909812c4a83e29efe67460c6756bc17,10.0.0.5:443\n/82f35bed949dc360e05c3a60f8d30970f9f30fcd,10.0.0.7:443\n/e6d5dc89bb8d677be8c2dc6cc1f164a254e4436d,10.0.0.3:443\n/dc64f2b2db6cbfb3129d83d0a5ca4d18d45ac35b,10.0.0.9:443\n/d7b8e1a5f574730502d6328885ffbe76a93726cb,10.0.0.5:443\n/e98e4eec7b294b9be385e3f87f5a5a86c4b202fd,10.0.0.7:443\n/7c78c7e886c0a735c2043ceffd442e7a3df6de18,10.0.0.7:443\n/53fcdac505c2e9d61d3df43336fbed32d4d52274,10.0.0.1:443\n/e4e48c52272aeb71e75533c0447168a17dd0a8ac,10.0.0.2:443\n/ea3361f8b5124bc65fe4273df0c03177066e8f1a,10.0.0.1:443\n/7db8dcb86856a592128f019df1696d1bd8f0729f,10.0.0.9:443\n/b8b33e4984ab558303db0b2481d8b6ab52bbeba6,10.0.0.1:443\n/a61755fc46e10c87e457c3fe06a46ca261624b33,10.0.0.1:443\n/74e9e54ea37b089a100561e9c89852a47d738657,10.0.0.1:443\n/7daddba4e4b97331ab3e47976295aa522c7622bf,10.0.0.6:443\n/9a4a68b8ccea854a22b1064b2b647270161df2b3,10.0.0.2:443\n/5b42f6b8ac65a644c8203a0e1f44d452653d24c4,10.0.0.3:443\n/2ae53235a7dd1ee0ae2e0187969e33daae6466e1,10.0.0.5:443\n/cedffeffa4fedb95411a03e7abd3e7771982e5b8,10.0.0.6:443\n/56a5eb3af8c0de47c03da5850dd5a41bca391bf4,10.0.0.9:443\n/8770f96efc90e2c044835618c1fbc21f784148fb,10.0.0.4:443\n/77b4bb2780a93a397dc469fd804b05a3fe6443a3,10.0.0.3:443\n/666b5de9ea6b7768dc6638f1426b5292465de061,10.0.0.1:443\n/793233793175f196f861bbd781cffedc1dfe1649,10.0.0.8:443\n/6e6c4ca7b99531e8ed151f023d3f54dbc1aa72b9,10.0.0.1:443\n/a144b632b672fed3d6d3cdc98d4b664993ae2c3d,10.0.0.7:443\n/074d306725d881f0acd57ed9f08ab85976adef84,10.0.0.3:443\n/c9c04880224d2977fa60a0eef06d1c795fe6a423,10.0.0.9:443\n/d20ffc4abee9257d5dc1fca42c53f6db63a2a664,10.0.0.6:443\n/316bdd0350db3c95c148e66a291f23d382251f92,10.0.0.3:443\n/a91ec91455f9c64ff666f271a7697c41b6777c40,10.0.0.5:443\n/ebaa896e268efc129dfb87e51c06b69c0d3d10b5,10.0.0.9:443\n/baa5d67516c1a6346ab05f10821c7ad25d297cdf,10.0.0.7:443\n/e8a049830e420d8502891a00eb55982ba0e9fd8a,10.0.0.2:443\n/2885644f1735ca86b002e623d2ea6824977b3386,10.0.0.7:443\n/85a0370ec94aca8895951954ca810d3b06b1f7af,10.0.0.4:443\n/4eda15a4a13e529b2aea970a1b3591e537b8c906,10.0.0.3:443\n/bfa3f4f77e6f65a87448d260eb7647b6e02cc8ca,10.0.0.5:443\n/599ca6d3012a3aa53854fb5e0d25c303d622e8d3,10.0.0.6:443\n/2332508d304442b6a6b4cebe557be963e31aa07b,10.0.0.3:443\n/0c32f97e9ca195ab9eba167702fb7af9f9809bc2,10.0.0.6:443\n/0c4ad0a80bdb18e6a82233a8ebf2103e72cda899,10.0.0.1:443\n/56e0aaba9e15a92b4ef584d3e8ebe091c52e2f60,10.0.0.4:443\n/0adf617e084382c2a23249aa985cd8b8acdaa65b,10.0.0.3:443\n/ed18addfd26398806fca2a8a2fe14bbe090fe0e0,10.0.0.5:443\n/d1a2ba53f3b93e000e644d5b1ad8c6854ea9f65d,10.0.0.6:443\n/7107620d6a0be698aac43ffc25f86eae6f53a2e1,10.0.0.8:443\n/3e4ae6d93c0251c47b243273c3cfa6a8e75561f1,10.0.0.8:443\n/d4c0a3bd89b4dddf0a6ea0b0527290ea09eb5a9c,10.0.0.1:443\n/152e9ed59081b05a44434e96acb10841a6ac4913,10.0.0.9:443\n/37165697fc6ef78a2f77de34e332378708c645b8,10.0.0.6:443\n/a4e2739e0c203344e3243ecd1085ae55c3e80e7a,10.0.0.8:443\n/23795bea08d22a82aeb2fdde5cf561d439358287,10.0.0.8:443\n/dfec4908e467ca09908796e4adaa92aa9483b5c3,10.0.0.3:443\n/ee0b2b4442cf36ae3c29d01c815bfa6cfe48fdfc,10.0.0.1:443\n/cb78173624dc4cb41a7b0237d17f7af6a083ea18,10.0.0.2:443\n/d739f678a0da873018a2c748a92ee4719fce3a0f,10.0.0.4:443\n/2323b27ff2ca01083ac32f731fbcd4d100e93e19,10.0.0.4:443\n/d499e2ec08b34a0bf998c14c887b383265034c9f,10.0.0.9:443\n/143084dfd65a8bdcd57fb64651f96008b040087b,10.0.0.6:443\n/97eaf99e3029d498617ea5154d783a7eb5c70c48,10.0.0.4:443\n/33a3ea00c963b4f990846722d410b76a95b8de58,10.0.0.7:443\n/74102b50e78c7af16e679839dcbda2e0372b6007,10.0.0.7:443\n/98e74ce3d20c7897ba902ddb716d77ccc12d7184,10.0.0.5:443\n/04675a8dc4351a5aefbd2ca04b6fc110d396a968,10.0.0.6:443\n/fc9d0c65157c42a6e4faaaaeeed021e97eba2829,10.0.0.1:443\n/e79898b6d8400872a5e147b49e9a639756cbe950,10.0.0.4:443\n/383d327d7129f3b712ff73766f8ce42c74e53b15,10.0.0.3:443\n/210ecfe8a674dc8a5ce6e0ff2cecca5f78769a23,10.0.0.8:443\n/0ceddf293af83e1c0b20a2f951c249dae970dac7,10.0.0.1:443\n/35f0c2c9af248df96020a501f0cd01c35750cbc0,10.0.0.8:443\n/52bc55a205fa328543f03294d9ed5417820fa601,10.0.0.6:443\n/3e4431ffbb010af04dd776032e7ac6f59a3b104c,10.0.0.8:443\n/4d9b2f4f469f2b2b5f3962306f97e75859c8d936,10.0.0.8:443\n/56f073a5a58f509bd491f8884fb340a935f40308,10.0.0.4:443\n/3c2b58f0da1000a8a78a3277b7fb5b1cfcd849a6,10.0.0.8:443\n/928720a69173cbfedccbcfc04b198414b5164f00,10.0.0.6:443\n/d87b50fec3d063f2a71d17c82d042c2abdcb7017,10.0.0.1:443\n/3f56709086bde9189226f4c98d6debd284176644,10.0.0.2:443\n/cdb098b71941c131bdafe7123c569453e08875b4,10.0.0.4:443\n/75dc829b5562bf143582d8ebed2766892438fdf6,10.0.0.5:443\n/e92003125b6bf378ea6279b52c744bbfa947c7ff,10.0.0.9:443\n/43601fdc2b42ea10e6eddf5ad078d9959591b054,10.0.0.1:443\n/4de1baf0df9a5653c30ba33552d107c8050ba65a,10.0.0.4:443\n/d8d7b30cd8c473d9dfccf65626bdcaf093524fab,10.0.0.9:443\n/44486486ce00b0c97d44bdaa8e9ad0cd4cef2f24,10.0.0.4:443\n/2f2b76cd6ceb9d1942d5e9ef0f887e2447426de1,10.0.0.3:443\n/0f4b330a57a82f31e00455ef74555b31f5bed5a5,10.0.0.3:443\n/1ac2f42face0e1b71f40775e804816f562f5046b,10.0.0.4:443\n/09d09a595f2bf13257e07133ac78186f35555584,10.0.0.5:443\n/0110e2fe94b7def844f20606e2c8456d3502bccf,10.0.0.3:443\n/e3b4ef341806cc37ca73c853f7fb946e1f5f6e8e,10.0.0.7:443\n/38e28374b1a9f41bc73cdfce491eb2bb0bce6fbd,10.0.0.3:443\n/d1713989a361ca1eed0f5f68fee5c50f91854b5f,10.0.0.3:443\n/f23af6f472102edef6de9491f9a844070ba3029d,10.0.0.9:443\n/f196a713839baae0bc51f4b1436927d36f2ff7d3,10.0.0.1:443\n/7598e8238744d04076326a8e3597578f9ecc8c09,10.0.0.1:443\n/59a02561ca435d78ab8aa7422feabf8773dec410,10.0.0.7:443\n/ca62f7298625dc574326a50ad33a4538559fc47a,10.0.0.5:443\n/04bb8594fabb6dab24d9913b98cf38d0563f83b7,10.0.0.6:443\n/8841711058feec5187329165955a9c60407f83fc,10.0.0.4:443\n/4d884355dd150df67499ef203c01635a229ba6c7,10.0.0.2:443\n/071866a73e91187306c1e407f0115a4d4151aeb6,10.0.0.2:443\n/afd57a92e88c0328fb968edd941b4100c6c9fb61,10.0.0.5:443\n/531c345d706cd574388c37ad61ba205c406b1d59,10.0.0.3:443\n/4502d8a61fb1a981fab9999a469dbac6e25e7f05,10.0.0.1:443\n/e230fc4ffa3bc256581c8eadd173e1cbc73b0cfe,10.0.0.1:443\n/05f427fdba3ff5f8d3039d3630772f161b8c9f14,10.0.0.3:443\n/911e4045fe7a9e356c9a4a83edb7675844e71e1d,10.0.0.1:443\n/6e23a5865d2058af3619068245ead5e51618e0c7,10.0.0.4:443\n/d1a0e473ffe13801ea65171346ad99201c498078,10.0.0.3:443\n/815a0974398d76788a4b3da7d0cca31f126d7de4,10.0.0.9:443\n/4b5e04ef7002bbde1ab4076a838dec1de1d26f39,10.0.0.1:443\n/524cccc30582e8a63d8312af6635e22baae5cc6c,10.0.0.6:443\n/96b6e633b7128d91860de1ac281c3080fc3abafa,10.0.0.9:443\n/bd9d0ab3293f97f2d0723f68fd28f15bd536c156,10.0.0.8:443\n/47e8509e8d7eb07744a7bbb065e51a64af82fa8e,10.0.0.3:443\n/7186ce2f19770360dfb9ecd97b89e16efac316d1,10.0.0.9:443\n/eed6c4e6883b62cdc3a7b196d99273557b071dd2,10.0.0.2:443\n/b96d9c4be72a2b9492c2d7317b95e9e8126b5c22,10.0.0.3:443\n/548f0aba642aad540d75ba5179b4c4e0769d3f1c,10.0.0.9:443\n/26584bcf23d47c9d064c9f80f8e428b694aca25d,10.0.0.9:443\n/e2037c8f1d790b8c725d9b2935056043ecbd8e93,10.0.0.2:443\n/7eba0adbc91b2e122b190de42e4560a2157b112d,10.0.0.8:443\n/9146d10cd7ab98d06cd0f4b4197602778d6e4140,10.0.0.8:443\n/9fa77d555984524b0361fb3072036e91cc6ea3f3,10.0.0.8:443\n/52f48dffdc8736b0c0e71df9a3c0f80e88770725,10.0.0.3:443\n/2265098f5d88d6d99acdb58cdcc1259d928625cc,10.0.0.9:443\n/d6192927efab74c222d087ad8dc2e820dcb7f6ea,10.0.0.4:443\n/18d1d3a2d5ed8d726aef5302787d369b79781866,10.0.0.4:443\n/19c735d6358604129199dcd752ff41f4221fa57e,10.0.0.5:443\n/b351359d0d09616a3fac0e66ae77e26357781753,10.0.0.6:443\n/6dd6111dd3ca9ad06569393e50484d3347035fa0,10.0.0.6:443\n/93e3738859d55cbde6df33ff6cb633a9f80dc0fa,10.0.0.9:443\n/eedde9fe113eb083a5d3c2508057360ddc18686b,10.0.0.7:443\n/62a3e2d15c1c66eb816f3dc13b5827984b9f699b,10.0.0.9:443\n/f1b4e2ea07bb5ed2f47433b75d1f7c8ef5bbf7d8,10.0.0.9:443\n/fa62a18b60e7385df8024532e7aeef87cc1df239,10.0.0.4:443\n/7d1aa3efc0114d39a9f010b900e1db11151e639d,10.0.0.6:443\n/f0468f9ae142e357977988238dd0688c49773aa0,10.0.0.4:443\n/d8b5c9789b2aab6444a2ef1d8cef7ae774458ab8,10.0.0.4:443\n/0a85a262365cc25072bfafe6136a7ba369566112,10.0.0.6:443\n/4dded059c42653a7c1beaa541e63645bf551ad08,10.0.0.2:443\n/49bd739af1ec870702c3847e967e8c6b435c5329,10.0.0.2:443\n/8d95d2e2928f6c92b660ec1c1fa7df8ff813aec6,10.0.0.4:443\n/93bf75656856b0a55da0773071dc39089dad6c98,10.0.0.3:443\n/e3390ff69032d41cc118b525049af3a5b802fae7,10.0.0.2:443\n/3150a053fd0dbbb2d248eebe951137ca8078f1d2,10.0.0.2:443\n/3f9d225fb79822a7b27f99d8d201ca1a45bb2ab6,10.0.0.2:443\n/452402b171258fcd3a5b107c1c939518102d2470,10.0.0.4:443\n/08317e0decede1f945dc9e1fe173bb4f6fb8cfd6,10.0.0.3:443\n/2cb847f4fbc26879a7b38289fb9add3bdc92a61b,10.0.0.4:443\n/42ba7f638d09315bd314f1ecb6cbaba154074a68,10.0.0.5:443\n/c0370eebe06344737bd403540f6adb13609b884f,10.0.0.8:443\n/9c71526d82a2a81ac13581c5f93125202f95aff2,10.0.0.3:443\n/e334fd8fafc2cb8809436c3e95e285d8c6079a94,10.0.0.7:443\n/c396727f862d04377b9b67299231d1d64d3d208a,10.0.0.3:443\n/6e51035a8d1f3bdd7f1732fd749ef13732a95944,10.0.0.8:443\n/785266cb2e5abbf8e8d68ed6944c44bc8784e83d,10.0.0.7:443\n/212094a923c2cc9abba620361a83ed32ffc89562,10.0.0.4:443\n/139a4d3ff8c15c8bbe453dc8b51a4369a9761129,10.0.0.8:443\n/990a8cbd6640ba4c5002b6e139b1ddc0784f687d,10.0.0.9:443\n/a6fb42bfd3197c44719ed46faf9d90cb06548ad8,10.0.0.7:443\n/000c6423f358a2c3d3f495b13d62c0d2c39f50bf,10.0.0.1:443\n/1cec05fac01fdaacba7ce488edd944661542c006,10.0.0.3:443\n/271f90733f7434e025ebc1f28267e53433857e1a,10.0.0.7:443\n/39e38084ed794b098342c8e73c50eb6e8a473451,10.0.0.3:443\n/391aed0b4c3552fe6045ee9e9fa5b1d749dad8f8,10.0.0.5:443\n/6f08ffaed414be3b69721e7f6d32cdbf708af3b3,10.0.0.9:443\n/e70ea65e0acabc146426f773ee43d27693b291cb,10.0.0.3:443\n/b4e2cf556861071282564ee2f08e188fac4c5253,10.0.0.6:443\n/6e68bc22256a8e097ae14c5309b968eac3ff115b,10.0.0.9:443\n/47960ff89fcae7481243ffa5164fd694546f238a,10.0.0.3:443\n/f574bef3f84ad106c19f59e9e9e02b531545144b,10.0.0.2:443\n/4c3e44af33a9121771dfc8ec0f9315ac44133350,10.0.0.9:443\n/b15cfc1da87f56a87bd209364583cb5ab32236dc,10.0.0.4:443\n/40fa370749b2451bea54a014f1424acf8f26689f,10.0.0.5:443\n/2b427b399d5020c4422e0795378c785ddb7294c2,10.0.0.1:443\n/bc6adfe886d10fe6eafb996229582b4bfc8d222a,10.0.0.8:443\n/f7b4e0538be13ab1bf2d07e6811a503a9017d9f3,10.0.0.2:443\n/5e89704df5e29e7cf5d66780e8c21ee43976a437,10.0.0.4:443\n/7b83fa2b9257a4c2d3130dca607ec04fb042a867,10.0.0.6:443\n/a3b86bc98a2d6f3ffa6572d699127b0bddedfed8,10.0.0.8:443\n/e4d0162d0642d1b8c48d405df85e19a38f0feeb3,10.0.0.7:443\n/4d485cce05cbc8a437c1c0935d481a7bea42fca5,10.0.0.2:443\n/845b67ac4a0134e07b0a024cc4e23712b6e6aa9b,10.0.0.2:443\n/b79bb4986238b94afdd2cb060f5552eef7ff54e6,10.0.0.3:443\n/af07fafd8a759a1cf5a609d578639be03caea26d,10.0.0.2:443\n/81af00d61c1ef5b0613ad4beb2dd25f89efc6182,10.0.0.8:443\n/593136c41899f34390c6d37d16edbbb9cfff7abf,10.0.0.7:443\n/cf81d9fb40a3fc64d11770126f3a699d70eb0feb,10.0.0.7:443\n/63c546d9b325c23c3861555b1e605cc203079992,10.0.0.7:443\n/4ca3fa4d44be023ccfa77e4f77f68fa0b8ea7710,10.0.0.5:443\n/e701444e1b1d9f079e9c879d8f999d88b56d68e4,10.0.0.5:443\n/f6c473391fa230f190cdda902f793977ce7de8ad,10.0.0.4:443\n/7bc73bd36bbd732b25e146e2f1b01003eb159775,10.0.0.6:443\n/5f287e6957a09c1be550752505a5c1420ec8d3bc,10.0.0.1:443\n/bebd964d7ca6eddee7de35715ebd96afa29fc8e2,10.0.0.2:443\n/db7a5038c8f2d930653563a2e523683c0a93bc88,10.0.0.1:443\n/47a642c6b0895911ca70f994f506c415505d75e1,10.0.0.9:443\n/8c5554e66fa04d50fb84d5f49ace6de0852a34da,10.0.0.6:443\n/89923f03ca432e5969fd0cdf5c15bb70b98efe2c,10.0.0.3:443\n/7a0aa569021d8b2b2ce9bfcb07310c1ebfad5151,10.0.0.1:443\n/0937e417478b8173a2bb86dc7b8b9454e059de4d,10.0.0.3:443\n/188c99ab8d7267ee33a9a5acc8de76bd58898b45,10.0.0.2:443\n/c4a8f4fb2483266c7166285df63f3640fa2319ae,10.0.0.8:443\n/4b0d34f033113d93ecf921c6a0feb0b2d45fb126,10.0.0.9:443\n/fb42de2cd7c51cacd89d7575bccd67f7ec25e451,10.0.0.4:443\n/028a4c7b547864a379e75c95c587ce64202181dc,10.0.0.8:443\n/cc5a526f4da916ea0a13b85e0c6e98b3d7437cc2,10.0.0.3:443\n/9113f453c2c434907b4ff773fe5b86f21e382416,10.0.0.1:443\n/830245306057d4c72845188ee1bcd3faec480fa2,10.0.0.1:443\n/81d4b64d141924bb939f5379c79f68c8f73f6861,10.0.0.5:443\n/be650948e3ec4fb93c66085f4ccf5b03ab52da7d,10.0.0.3:443\n/dbf7fb2f6d1c71680d904a7982c255d38d9cb66d,10.0.0.6:443\n/df2fd0964b001d47e7d7d8f704997f8b2c188e2b,10.0.0.3:443\n/52599e6c628d3713b5a86776e7bf4e8eba08096b,10.0.0.8:443\n/9a8dfa9f83cddf2f78d3dfaf2977d373250d08bc,10.0.0.6:443\n/f6fff2fc64e81812c1fd6f44b17c0fca8f4d9b21,10.0.0.7:443\n/c74b67ef81c0f0159ebf55bdb7cec7edbb49ef01,10.0.0.7:443\n/1d489c78f83a538058f88a86d5d3b784a4c8c15b,10.0.0.3:443\n/c19e2d12a05580a32ed36959def7a94542f3e4b2,10.0.0.2:443\n/0ee2827f11b4ec40f5b9461cafbafc67c9bb19a4,10.0.0.6:443\n/6fb7e55aa135073aafdaee252c0d41711f0012cf,10.0.0.2:443\n/c31f4f1035ecf919c2a8ad9aaba8f34787b6dbfc,10.0.0.5:443\n/da600775aba95b86e525802936ee1080854c9171,10.0.0.7:443\n/713544fa205d177ae8892dcf355b3d9099d36346,10.0.0.9:443\n/7c838304cffcd5141ea13dc98ebe74d3c9fb0b68,10.0.0.5:443\n/1dfa23ab97268feb6b3fd019ff226fbcd628995c,10.0.0.2:443\n/525a23553d91d72be5ae26127029e9e3bdfdb6d7,10.0.0.9:443\n/963f18f8446ec2ab1921e0579c7af87477ef3592,10.0.0.3:443\n/64b104d6efe390c5e8a008312e16772cc76632e8,10.0.0.2:443\n/de3125db331f2c903e96f273a649f782abde3250,10.0.0.7:443\n/fb2452ca4688e9fa8177919d72032717ac19d692,10.0.0.4:443\n/76e8bbc123c7346c3398619f12d9888fee50811e,10.0.0.8:443\n/d6f9d8096fe75dd35f317b7d1ce0bc02306f6121,10.0.0.3:443\n/1e543c13b6af55fc60e9ba216790bccda9c2c3d1,10.0.0.2:443\n/5f5831dbaeba200adb52eefeb26d792cf996a01f,10.0.0.3:443\n/60ac99dd1fe31198399c6b447438e8ac62a35036,10.0.0.4:443\n/579077bb8a893f47c1b1eca7d8cd2d62c8a3b774,10.0.0.2:443\n/86590bffd69757efb277e2b6db69385bbb6ebba3,10.0.0.7:443\n/f829ea6c46b59cde7ba3c6745b58c3ea170bc73b,10.0.0.3:443\n/27f54b7138336175705207561c6a0a123049ab66,10.0.0.7:443\n/fab8ba44a38c657d3b8f9066daac9f3222e3fd50,10.0.0.3:443\n/03540d372c8e6621cf57ecb910cf872164f9c53a,10.0.0.9:443\n/be320aecabb73286f782df249b53ef4f7c816b72,10.0.0.1:443\n/7e383ecf09c026dec3eb9c77ca35274f209748a3,10.0.0.2:443\n/749dd7577081e9d47a3bf9ef091b6e5b8daa5786,10.0.0.3:443\n/9507f34faa73988fb0d35f65141b2e1b0f407cd7,10.0.0.3:443\n/a2188b80856ef974da6eac0238a35afef2e1c30c,10.0.0.6:443\n/1ca972ed159b7e8c838752ce60186262a923d057,10.0.0.4:443\n/57e7abe6d05fe0443800929dcf46315bc5135519,10.0.0.4:443\n/075a1e1426a5d210fd5501249809b1e20c793c92,10.0.0.2:443\n/36c50cfcf56f7e15f91844b9f3201efa4cc26fc8,10.0.0.5:443\n/f824965edace119738da671b1e3cba0e44be224d,10.0.0.5:443\n/ba49bc86400cd3970ea82550b493ac21a5bf750c,10.0.0.6:443\n/2dbc4c595e56af88a175811e715d908d04223c2b,10.0.0.1:443\n/e322f439382552e76fb5672448c2fc6beb56e32f,10.0.0.8:443\n/e86b62c9f240c97f1503be26f97fbd412b846e2d,10.0.0.3:443\n/e15c58a856824298802fed4151234e2a4812697a,10.0.0.7:443\n/e7b5cbc5701beadb3c8dc04361cdd1de8f8b0d1d,10.0.0.3:443\n/bf317994d21c8731f4afdebb4d77e8eb997a962b,10.0.0.5:443\n/cabb2f8ffcd78c654c2eb0bbecd32a4a5523c165,10.0.0.8:443\n/17e54d3b00f5b989798846b7729789343df86d56,10.0.0.2:443\n/334fcdca67212b01973f946729a2cb569adec510,10.0.0.3:443\n/a5d301755197c85985ee48ec53b4c3a1b6bfd9fe,10.0.0.1:443\n/f2b98e19ef44d81d16fce5e9e3b51c3be2a8c713,10.0.0.2:443\n/6a69c385697c7ed6b85ced626d993b02216c2186,10.0.0.6:443\n/7d269fcf2722875d9d75eb2e3a7e0fae6ae69a98,10.0.0.5:443\n/126e39e079fa7f2b254146466e9ab79dc4347be6,10.0.0.6:443\n/8a4aa1d8a6d6a2938ecf7a42814723e282a90af7,10.0.0.1:443\n/b9d9fabb46dbe2db5f5e0164c98002114c96006e,10.0.0.2:443\n/d50477758d2f8e99392ae9d51ecef57484c97c5c,10.0.0.1:443\n/09ac262b5b37a5b19fa989e5ac75722fdb8e17fa,10.0.0.1:443\n/e1a322b3fc40fbc296f9ed8dc86d6e91c6dc8edb,10.0.0.7:443\n/1c08ea2a0c42f17d99acbcb2b0ffd33ba1fb34be,10.0.0.8:443\n/6aeefd631b7ddc01936b7fc11f1e8aa5a892bec0,10.0.0.4:443\n/f579e4e7a050029dd9f21a38104c1f1b65e201e5,10.0.0.3:443\n/9764ed5aff70cac5a5859415251c0b78dfd37909,10.0.0.7:443\n/42d5cfb85dba89806546ac382f61fd909cdd40e9,10.0.0.2:443\n/4cf6e01e948d9cd30032b5ce77d2d8b2daaf21c8,10.0.0.2:443\n/e0367d6d11e81aaed084aceac24b327abf053c0f,10.0.0.2:443\n/28b88a8f6ce2d2b9bd5867f89555f2e43dc96c96,10.0.0.8:443\n/56b207c041ede2daa065d56b03173663eb6a1f2b,10.0.0.9:443\n/6aa844c78eb0fb44bd506a920360377163eb7150,10.0.0.9:443\n/327dabc969eb616dc10e4ba33f45d0cc97bf79d9,10.0.0.4:443\n/7352ac0a60c2f2bedb0d80b0249db5aa59e0c876,10.0.0.4:443\n/70a2d83045053301afcd56deb30bf9095a9ad9d6,10.0.0.6:443\n/cbd05a3e43fb4b1aa98283c8d460571c43144391,10.0.0.9:443\n/0c903f04b7f6d4a59a97c4ae212a41ad579e5a1f,10.0.0.1:443\n/3dc7749f4a1f30ab1da8b4e2722d3228b0e8b33d,10.0.0.7:443\n/b502f7505431f9b895c9fe04de74250f27f96f7f,10.0.0.7:443\n/14aa1d4b3ddf14d7f5cba9de1f08213a8a104131,10.0.0.9:443\n/cce74a90b51c414ffbb28ad26794bf589c1eaf82,10.0.0.2:443\n/a86f00b80c039be1e4fad0b5508bd1710ba08ae1,10.0.0.4:443\n/055b3843846da3e34606dc209c9b30192e6119da,10.0.0.4:443\n/f05c831872443fb183c6c19d1a9dbe45315f0205,10.0.0.4:443\n/d974f0525fca48f3fae6d6cbd0b394e4324ad422,10.0.0.3:443\n/00ef799f7c152055a4163c41bb9bcd37fa375f77,10.0.0.8:443\n/7d876337b6f86b9873ccc49f7efaae91d180a06d,10.0.0.4:443\n/93e9efe1dba728a9a51d68024eddf764903cac1f,10.0.0.3:443\n/891d1f6116aec61df9dce652743ae159861e9160,10.0.0.5:443\n/eec92486c171c84f0dcf3db001ce9b32615166f3,10.0.0.2:443\n/363f23c9bea6cac01f85f7ec7142110363110c83,10.0.0.7:443\n/51c699024eca637817d2bdad512b9c1d19fafec4,10.0.0.6:443\n/b7aaa2cd36505f55bb047799649acadc5e20e499,10.0.0.8:443\n/b17b2a0be4fa445a0f762a92dee8b8d83e992eae,10.0.0.8:443\n/e4c7719fd87bd86bf5c818a7aa38af4d50691d51,10.0.0.6:443\n/edf7c0e6f7c1bbdc7bc67912607a32ff5ea786fc,10.0.0.7:443\n/62d1fb98dbf4825ae9db6f919233f32391ce1e63,10.0.0.8:443\n/e0214b5cf9fb6464dfd37a84fd41a759e571a531,10.0.0.7:443\n/bf7e606b9858081dd5f7f2346f70e314238809e3,10.0.0.9:443\n/f4907d05ffb94d07ea2a1eb8dc187268cd386f0e,10.0.0.2:443\n/cdfc8d53ae78926694e5d1ebe872d30f7e57b324,10.0.0.5:443\n/0de8c668d5445e80ca5d0ece0e653fc664b494e8,10.0.0.3:443\n/fafc8ecbe6fc726999a249b9a2bfcb52371dec5b,10.0.0.2:443\n/ec395c49cff03d52796f59fabdabd045eac000c2,10.0.0.3:443\n/0bfbcb5addb0f60e28504de36a10abb80d253011,10.0.0.3:443\n/6bfbff0669887fbad3bc871771c62f9c5d723740,10.0.0.3:443\n/d0b0e5ce39dbfd7a3b46542c1088e0231295be11,10.0.0.9:443\n/79dc5f7b75fb82ed6dc0be947434d924ade41bc0,10.0.0.6:443\n/f729ac665760539349f1c808bee81b70d58537d2,10.0.0.8:443\n/fc97f5ede259d60ba9f28888ac3b2155b5490df6,10.0.0.1:443\n/0c5cb1e0dfed034af7c893de52841686a762f6aa,10.0.0.2:443\n/accc74f350aa74842630517d6abfed728d5da084,10.0.0.8:443\n/1c2e760244036ee1f0539288d446830a0fa38457,10.0.0.7:443\n/0f60c86359a38087105591e609f90b962b02b74d,10.0.0.4:443\n/9e26cafc4562d086fadce9cdf56c049e9cfd93b8,10.0.0.5:443\n/bce221dd780f255f84e159f1097ada3461ff5ab6,10.0.0.2:443\n/c918871122a23989009073c879ff5c6d3add13c9,10.0.0.2:443\n/50735adf2192b44d672b99cacf66e650582f9088,10.0.0.1:443\n/fb763e843db495284a217b2ec5027d7698bd49ca,10.0.0.1:443\n/ae068dab9d1784aa098085b9f62e168c79bb75cc,10.0.0.9:443\n/6b989c5c0af97c2acd640cb2f05dc4b4fe7c9323,10.0.0.4:443\n/fd81185d074101b22059f022b174d058b8d84a6e,10.0.0.8:443\n/20277b4366206f795ed97f4631baa755636ddc1e,10.0.0.6:443\n/c9199750bcb6c7b916958bee64f562547d6b1209,10.0.0.3:443\n/9ea2e79813eb3a6d7b710affe1b15f7da29c1800,10.0.0.2:443\n/2de1269fec587f989b911e4780eb056c310d2f38,10.0.0.3:443\n/6869b9dc65fbfb56fbe828c0ac57da2bc7c0f460,10.0.0.6:443\n/aab69be5e6c02f972e8a5e0e4c9716b84d91e667,10.0.0.6:443\n/f8bb7d358e7c5e58b5d68a29a2d1ca4cb1a4af61,10.0.0.6:443\n/441f8eab7ac90bb4022d35aca197b5765ecb7dbd,10.0.0.6:443\n/36bad018c5b41a8f3731267f364cbe74cee32a81,10.0.0.4:443\n/8d31a4ecc03b0d8680ccef099ed528f07a5e8aa0,10.0.0.8:443\n/2e8ee9b09986f2e9435dc2b3b05d9353a8edfd3b,10.0.0.2:443\n/950dedb075f2e3b2bc485659e0158fa0c26511ee,10.0.0.7:443\n/2bff636db028f497c8d26cce0c138a5049e69db7,10.0.0.4:443\n/33353d31c296876c1267ae7793d0c03caea2a387,10.0.0.6:443\n/bf5e822a05f799f153c648ccad9987396c9350cd,10.0.0.8:443\n/92a14cc304c5fce82d3e4999fb1a819c92cfe2fb,10.0.0.9:443\n/06c1a1908f37b4cdacd7098b7304ced839f8412b,10.0.0.3:443\n/de42c498d10e019fad348d4e2e512afd8981a7b9,10.0.0.3:443\n/42fda236e32757240063ecb99681a30121629ab3,10.0.0.8:443\n/b93d832939ca288351cb707d8332ff7aade5c6d0,10.0.0.1:443\n/2fac7e24ec806b0b5b9f7bf99587e50357d9db04,10.0.0.9:443\n/8750e04a953b8d6a5922ccd6172d59052671207a,10.0.0.5:443\n/c5128cc73e48a0270a27e6e2b89f563dd756f987,10.0.0.6:443\n/4f798881963c3638cd89168511eccdc8f6ecc62f,10.0.0.9:443\n/330116fb139c1313bc78bf314a211d98dad8093e,10.0.0.1:443\n/d2ff19c61e6be34733295d7c2f81b18c81a687bb,10.0.0.5:443\n/5ff80f927eb571aa8cc006e0b7c237d8f571ffce,10.0.0.6:443\n/b2ec40a643fd05a9f0d52b180893c9837a824060,10.0.0.8:443\n/9445a32fc9ea94e27622826699123272ca5bb38a,10.0.0.8:443\n/7b12433171a2a61d85b95d8ffd9ddd7e8ba1f0d2,10.0.0.5:443\n/8f1cdae5a092253771e8d07f66ef45d6cc540c5e,10.0.0.6:443\n/7b70d96ff0b14c53aaf134e16cb0ac8394b9ec93,10.0.0.7:443\n/1c91cb4395964c049653b422be092ecc6957ccb1,10.0.0.3:443\n/1680013d16baaadb8cffb6838dcaacfa9ddf815b,10.0.0.5:443\n/a6cd614c3f4c92608f176ecbbd24d4d3e980c18e,10.0.0.5:443\n/5f1ad23d97b5effe56b07770e1b1e3eb9302ce32,10.0.0.9:443\n/4014ea7ccfafddaa137d536e55a4fcd04f4acd23,10.0.0.7:443\n/4fe5c55f2b57b57de6773d70b788f2950f98aedb,10.0.0.4:443\n/a5e42b24f4d2b631529dc5e693f0f44bec25d917,10.0.0.4:443\n/605b3c16dde75dde0ed96aa420121d491043b868,10.0.0.5:443\n/a44dfbc27a6071c7fc05ec6c9bcf08a80059d9fa,10.0.0.1:443\n/376db7678f97b6614c81a967ac480d2af1ec0494,10.0.0.9:443\n/7ab905b17ac0f3dabc9e73654f7108a60382b4bc,10.0.0.9:443\n/7e15729b3688eac3a344b412fb66660812e51f5a,10.0.0.9:443\n/fbe2e0d0d220635295397028926abcab95eae6f3,10.0.0.9:443\n/68c054186128694496e7e34307333f0ee0fcdd83,10.0.0.5:443\n/1539ed321544c684ffec1d533551c8ca3b814bed,10.0.0.2:443\n/52239e8766c814c60a588320ff4e55dc9fed5345,10.0.0.9:443\n/7dbfc596b9cf0c5fada4c6521d67d7e8d42f6fea,10.0.0.2:443\n/d04621fb5f44305c3532899899c6f53b6afd5a0e,10.0.0.9:443\n/581c7d2014ef9aa2a30e54ac748164ba66a534d1,10.0.0.8:443\n/d6c7154f9d5d3dadd5e890c8f7a1b178c4992eea,10.0.0.2:443\n/bbf232fd8d3a7f781b378f160e3fb629ce09ad59,10.0.0.5:443\n/5290410ff6812f651bac427d10fae8ec7a40af1f,10.0.0.9:443\n/41cfbbdf14643164ccb783e93eb123f23f2f32bc,10.0.0.8:443\n/9259ff15fbb6d028166e31434d4e2dc335c390c1,10.0.0.2:443\n/20258376b3bf0e3221118d0b5af2c4b8fa99fef6,10.0.0.6:443\n/7e34ebb4b42eafe0d92fb748745416bdcdb67bb6,10.0.0.2:443\n/2fe398814da4f25bf0f9f6982b43d955b93c9c4b,10.0.0.1:443\n/6bcc489d911d7e972b2decd5827d4eb628111a84,10.0.0.4:443\n/87c70ef625901e6ba2c9dd6951efa9964bebe528,10.0.0.1:443\n/3b79cc56f47bc4487a12e940caff6f4c294b633d,10.0.0.1:443\n/bc5e3f9cfd2ca035467321a7280dd2bf1250259c,10.0.0.4:443\n/260555bb77d941f8761b6a8b81267b6884da79d0,10.0.0.5:443\n/e77e4992d06c6b662602f32bf8a52b1567eb2f48,10.0.0.6:443\n/0923bd0601bf7c074d173268cbf94cff7b39045b,10.0.0.6:443\n/d7d8f74a97ab8b8f0df34c447e2ba2bf19a8a142,10.0.0.2:443\n/1151eda28cedf93b701b4ae0f47b35c488d493af,10.0.0.3:443\n/e4b07856919718e9c7cd00154fbb12c343bf5b55,10.0.0.6:443\n/d5751616b55b115bb0260fee64eef350e83cf080,10.0.0.1:443\n/83bcc3183e99dcba798f17ac6340722b1431ed0d,10.0.0.5:443\n/2e84b809ca1744c2abfe7d8ec18dc80adbddc0bc,10.0.0.2:443\n/782631660d3fd29586a00922d9e1885c30615bbf,10.0.0.2:443\n/0b43b69669114bc2ef38ad7e03fae7a82ac22f29,10.0.0.5:443\n/59bb4679fc26333daec10a9a06ebe185ab252628,10.0.0.4:443\n/26b7bd3eae9d48dae1677893dc7efb7d7d9386ac,10.0.0.6:443\n/1842eded9ad6f8e951f4e788bbedd8ad9b5d2289,10.0.0.8:443\n/c0b818cdd2005b05932a6369a80d47993cf1982b,10.0.0.9:443\n/6f3b0ed59323086c2557c6ba6aeb433bd82509bb,10.0.0.1:443\n/bbe9eabb406b05b3b13ce634caf88123d095de39,10.0.0.3:443\n/fad30adc6c95657b9be1a490b338798fa08504c4,10.0.0.3:443\n/56a66dd5081af7ed9d9307374606245a751cceb7,10.0.0.4:443\n/356beada6689aaba95fafc526101a982acb007ae,10.0.0.9:443\n/c9917bb9de0eea13e66c71c88c35a03f3567bae2,10.0.0.1:443\n/495d1913174caaaae68eda040e329cf9ab6b807c,10.0.0.2:443\n/f89a11728d36429756ea52ed934a853ae2cf62a5,10.0.0.2:443\n/3263ba8199377065aafe566132716c2bf4fa300d,10.0.0.9:443\n/cf44e7fe43a615a8b58cf5c3b7486e95acf95904,10.0.0.5:443\n/2a055e911545b3eb8ca31202c16a916a061eb35c,10.0.0.7:443\n/120600f104f1305c96642a7382c7074386dc9f08,10.0.0.3:443\n/05690a98a3b475aed24b300bd1d8303c5f0ed93c,10.0.0.8:443\n/648371cafaf32c90a7c48ec2c6d57f84d4f81346,10.0.0.1:443\n/c346dd5aae927ff513fe5a8ece05ec7c2d5674d1,10.0.0.9:443\n/531b748b3a7ba01687e3eb6bff2df8951daaf1f4,10.0.0.5:443\n/c0935bb5ad0fad0b3f9ff4b924c01971116eac7a,10.0.0.7:443\n/2bf19d2be50af6207c977fa502caa8a6dd6722da,10.0.0.8:443\n/afc63845c577fb67edb01f2f4faa19415f36420d,10.0.0.2:443\n/efaa738ada05c233a8f0d57b3064d6ccd86b1a14,10.0.0.5:443\n/b1d708f56428e8c5e9ae6f8a2a492c75271b8a45,10.0.0.5:443\n/5b9abc512a9b5ee9083d0feec592002f5f31dad3,10.0.0.7:443\n/55289a755e9560c48aa164f357598a47603badbd,10.0.0.2:443\n/472ecb69a8ca3b590d1dd3153928d7bb09f43799,10.0.0.4:443\n/072d33b67d2137d961ff8ee5a5c0d024196e4c91,10.0.0.7:443\n/ca285c1755e98e6a8080cbe45f54e91f7d466c2a,10.0.0.2:443\n/13661ad2ef4c48f7e99825bce7efc3833d0bb1d4,10.0.0.3:443\n/b093d7e4f7588296fb1b02b38f20effb4c193571,10.0.0.1:443\n/dc82252b61b41f0760eb44be138b6bca8a22b804,10.0.0.9:443\n/1cddee843b564a0ef0acb8ca9d7dff0d7df3dcfd,10.0.0.2:443\n/560e227e16cb133b6fe840bf9ef03e8aee0dad66,10.0.0.8:443\n/55b1bd74ae0aea31731589550b7610ed00c1e616,10.0.0.9:443\n/87f0d8ea3eaa758b41763fd06d2fa52d2fcff7a7,10.0.0.7:443\n/e669bc52f256862c9ae2e0a7748ee3786a2abcf8,10.0.0.9:443\n/3fca3907c98117eb1920bd03179a35aee3cf1939,10.0.0.3:443\n/3d2582fcdf65814ef32ceb3ca30a38aba02020d2,10.0.0.7:443\n/2628fbf7ed1e6f3f2152a57e1b7c440895088534,10.0.0.6:443\n/37f57ab1100409e62e4ca9532dd245a2e84eab7a,10.0.0.8:443\n/1558c2abe0ba2f4d83172d32ee9489a5f2469174,10.0.0.2:443\n/8e2dc4336eb4fb16deebee7c3157b2a4ddc040d8,10.0.0.5:443\n/016a870bfd0d09832a50ca14d666e7ef21978f5a,10.0.0.3:443\n/f3d2b5c0cad02f1507f1a5c21b4cd8704f32ac11,10.0.0.6:443\n/1ded786d6cc703eda6a20bb7388114a6b74cb17b,10.0.0.1:443\n/de0132cb6cdc9aeba721296519d0b64bdb752855,10.0.0.6:443\n/30f11551b9d666c96604d01526eb001c9cd7524c,10.0.0.9:443\n/eb8bc43a527ac9f8088010954c78fff7a75f1bca,10.0.0.9:443\n/30803c02cdc19bccba32ad9fc2e77a4d20eac091,10.0.0.5:443\n/7281a18bbbb2dfb089b3039f516792e6b0c67714,10.0.0.2:443\n/8787a462e8890da32babea287e4874d3da54246e,10.0.0.6:443\n/2b76b52f43b204a6353a1f999ff40a221602199b,10.0.0.3:443\n/56e71d678f5482e4dfe45ae83993e9e1c9fe2731,10.0.0.1:443\n/08d820998361440ed076fa803e05d3c983383920,10.0.0.2:443\n/cc449cf6cee8ce83f646097ac3d3287becacfe41,10.0.0.4:443\n/2da713a30a3333b7895b3d65a320fe6c20bf670a,10.0.0.8:443\n/658e0cefd2b9a97b2dd840c2a243afc0dbf06cf1,10.0.0.9:443"
  },
  {
    "path": "pingora-ketama/test-data/trace.sh",
    "content": "#!/bin/bash\nset -eu\nfor i in {0..1000}; do\n    URI=$(openssl rand -hex 20)\n    curl http://localhost:8080/$URI -so /dev/null || true\ndone"
  },
  {
    "path": "pingora-ketama/tests/backwards_compat.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\nuse old_version::{Bucket as OldBucket, Continuum as OldContinuum};\n#[allow(unused_imports)]\nuse pingora_ketama::{Bucket, Continuum, Version, DEFAULT_POINT_MULTIPLE};\nuse rand::{random, random_range, rng, seq::IteratorRandom};\nuse std::collections::BTreeSet;\nuse std::net::{Ipv4Addr, Ipv6Addr, SocketAddr, SocketAddrV4, SocketAddrV6};\n\nmod old_version;\n\nfn random_socket_addr() -> SocketAddr {\n    if random::<bool>() {\n        SocketAddr::V4(SocketAddrV4::new(Ipv4Addr::from_bits(random()), random()))\n    } else {\n        SocketAddr::V6(SocketAddrV6::new(\n            Ipv6Addr::from_bits(random()),\n            random(),\n            0,\n            0,\n        ))\n    }\n}\n\nfn random_string(len: usize) -> String {\n    const CHARS: &str = \"abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789\";\n    let mut rng = rng();\n    (0..len)\n        .map(|_| CHARS.chars().choose(&mut rng).unwrap())\n        .collect()\n}\n\n/// The old version of pingora-ketama should _always_ return the same result as\n/// v1 of the new version as long as the original input is sorted by by socket\n/// address (and has no duplicates). this test generates a large number of\n/// random socket addresses with varying weights and compares the output of\n/// both\n#[test]\nfn test_v1_to_old_version() {\n    let (old_buckets, new_buckets): (BTreeSet<_>, BTreeSet<_>) = (0..2000)\n        .map(|_| (random_socket_addr(), random_range(1..10)))\n        .map(|(addr, weight)| (OldBucket::new(addr, weight), Bucket::new(addr, weight)))\n        .unzip();\n\n    let old_continuum = OldContinuum::new(&Vec::from_iter(old_buckets));\n    let new_continuum = Continuum::new(&Vec::from_iter(new_buckets));\n\n    for _ in 0..20_000 {\n        let key = random_string(20);\n        let old_node = old_continuum.node(key.as_bytes()).unwrap();\n        let new_node = new_continuum.node(key.as_bytes()).unwrap();\n\n        assert_eq!(old_node, new_node);\n    }\n}\n\n/// The new version of pingora-ketama (v2) should return _almost_ exactly what\n/// the old version does. The difference will be in collision handling\n#[test]\n#[cfg(feature = \"v2\")]\nfn test_v2_to_old_version() {\n    let (old_buckets, new_buckets): (BTreeSet<_>, BTreeSet<_>) = (0..2000)\n        .map(|_| (random_socket_addr(), random_range(1..10)))\n        .map(|(addr, weight)| (OldBucket::new(addr, weight), Bucket::new(addr, weight)))\n        .unzip();\n\n    let old_continuum = OldContinuum::new(&Vec::from_iter(old_buckets));\n\n    let new_continuum = Continuum::new_with_version(\n        &Vec::from_iter(new_buckets),\n        Version::V2 {\n            point_multiple: DEFAULT_POINT_MULTIPLE,\n        },\n    );\n\n    let test_count = 20_000;\n    let mut mismatches = 0;\n\n    for _ in 0..test_count {\n        let key = random_string(20);\n        let old_node = old_continuum.node(key.as_bytes()).unwrap();\n        let new_node = new_continuum.node(key.as_bytes()).unwrap();\n\n        if old_node != new_node {\n            mismatches += 1;\n        }\n    }\n\n    assert!((mismatches as f64 / test_count as f64) < 0.001);\n}\n"
  },
  {
    "path": "pingora-ketama/tests/old_version/mod.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! This mod is a direct copy of the old version of pingora-ketama. It is here\n//! to ensure that the new version's compatible mode is produces identical\n//! results as the old version.\n\nuse std::cmp::Ordering;\nuse std::io::Write;\nuse std::net::SocketAddr;\n\nuse crc32fast::Hasher;\n\n/// A [Bucket] represents a server for consistent hashing\n///\n/// A [Bucket] contains a [SocketAddr] to the server and a weight associated with it.\n#[derive(Clone, Debug, Eq, PartialEq, PartialOrd, Ord)]\npub struct Bucket {\n    // The node name.\n    // TODO: UDS\n    node: SocketAddr,\n\n    // The weight associated with a node. A higher weight indicates that this node should\n    // receive more requests.\n    weight: u32,\n}\n\nimpl Bucket {\n    /// Return a new bucket with the given node and weight.\n    ///\n    /// The chance that a [Bucket] is selected is proportional to the relative weight of all [Bucket]s.\n    ///\n    /// # Panics\n    ///\n    /// This will panic if the weight is zero.\n    pub fn new(node: SocketAddr, weight: u32) -> Self {\n        assert!(weight != 0, \"weight must be at least one\");\n\n        Bucket { node, weight }\n    }\n}\n\n// A point on the continuum.\n#[derive(Clone, Debug, Eq, PartialEq)]\nstruct Point {\n    // the index to the actual address\n    node: u32,\n    hash: u32,\n}\n\n// We only want to compare the hash when sorting, so we implement these traits by hand.\nimpl Ord for Point {\n    fn cmp(&self, other: &Self) -> Ordering {\n        self.hash.cmp(&other.hash)\n    }\n}\n\nimpl PartialOrd for Point {\n    fn partial_cmp(&self, other: &Self) -> Option<Ordering> {\n        Some(self.cmp(other))\n    }\n}\n\nimpl Point {\n    fn new(node: u32, hash: u32) -> Self {\n        Point { node, hash }\n    }\n}\n\n/// The consistent hashing ring\n///\n/// A [Continuum] represents a ring of buckets where a node is associated with various points on\n/// the ring.\npub struct Continuum {\n    ring: Box<[Point]>,\n    addrs: Box<[SocketAddr]>,\n}\n\nimpl Continuum {\n    /// Create a new [Continuum] with the given list of buckets.\n    pub fn new(buckets: &[Bucket]) -> Self {\n        // This constant is copied from nginx. It will create 160 points per weight unit. For\n        // example, a weight of 2 will create 320 points on the ring.\n        const POINT_MULTIPLE: u32 = 160;\n\n        if buckets.is_empty() {\n            return Continuum {\n                ring: Box::new([]),\n                addrs: Box::new([]),\n            };\n        }\n\n        // The total weight is multiplied by the factor of points to create many points per node.\n        let total_weight: u32 = buckets.iter().fold(0, |sum, b| sum + b.weight);\n        let mut ring = Vec::with_capacity((total_weight * POINT_MULTIPLE) as usize);\n        let mut addrs = Vec::with_capacity(buckets.len());\n\n        for bucket in buckets {\n            let mut hasher = Hasher::new();\n\n            // We only do the following for backwards compatibility with nginx/memcache:\n            // - Convert SocketAddr to string\n            // - The hash input is as follows \"HOST EMPTY PORT PREVIOUS_HASH\". Spaces are only added\n            //   for readability.\n            // TODO: remove this logic and hash the literal SocketAddr once we no longer\n            // need backwards compatibility\n\n            // with_capacity = max_len(ipv6)(39) + len(null)(1) + max_len(port)(5)\n            let mut hash_bytes = Vec::with_capacity(39 + 1 + 5);\n            write!(&mut hash_bytes, \"{}\", bucket.node.ip()).unwrap();\n            write!(&mut hash_bytes, \"\\0\").unwrap();\n            write!(&mut hash_bytes, \"{}\", bucket.node.port()).unwrap();\n            hasher.update(hash_bytes.as_ref());\n\n            // A higher weight will add more points for this node.\n            let num_points = bucket.weight * POINT_MULTIPLE;\n\n            // This is appended to the crc32 hash for each point.\n            let mut prev_hash: u32 = 0;\n            addrs.push(bucket.node);\n            let node = addrs.len() - 1;\n            for _ in 0..num_points {\n                let mut hasher = hasher.clone();\n                hasher.update(&prev_hash.to_le_bytes());\n\n                let hash = hasher.finalize();\n                ring.push(Point::new(node as u32, hash));\n                prev_hash = hash;\n            }\n        }\n\n        // Sort and remove any duplicates.\n        ring.sort_unstable();\n        ring.dedup_by(|a, b| a.hash == b.hash);\n\n        Continuum {\n            ring: ring.into_boxed_slice(),\n            addrs: addrs.into_boxed_slice(),\n        }\n    }\n\n    /// Find the associated index for the given input.\n    pub fn node_idx(&self, input: &[u8]) -> usize {\n        let hash = crc32fast::hash(input);\n\n        // The `Result` returned here is either a match or the error variant returns where the\n        // value would be inserted.\n        match self.ring.binary_search_by(|p| p.hash.cmp(&hash)) {\n            Ok(i) => i,\n            Err(i) => {\n                // We wrap around to the front if this value would be inserted at the end.\n                if i == self.ring.len() {\n                    0\n                } else {\n                    i\n                }\n            }\n        }\n    }\n\n    /// Hash the given `hash_key` to the server address.\n    pub fn node(&self, hash_key: &[u8]) -> Option<SocketAddr> {\n        self.ring\n            .get(self.node_idx(hash_key)) // should we unwrap here?\n            .map(|p| self.addrs[p.node as usize])\n    }\n}\n"
  },
  {
    "path": "pingora-limits/Cargo.toml",
    "content": "[package]\nname = \"pingora-limits\"\nversion = \"0.8.0\"\nauthors = [\"Yuchen Wu <yuchen@cloudflare.com>\"]\nlicense = \"Apache-2.0\"\ndescription = \"A library for rate limiting and event frequency estimation\"\nedition = \"2021\"\nrepository = \"https://github.com/cloudflare/pingora\"\ncategories = [\"algorithms\"]\nkeywords = [\"rate-limit\", \"pingora\"]\n\n# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html\n[lib]\nname = \"pingora_limits\"\npath = \"src/lib.rs\"\n\n[dependencies]\nahash = { workspace = true }\n\n[dev-dependencies]\nrand = \"0.8\"\ndashmap = \"5\"\ndhat = \"0\"\nfloat-cmp = \"0.9.0\"\n\n[[bench]]\nname = \"benchmark\"\nharness = false\n\n[features]\ndhat-heap = [] # for benchmark only\n"
  },
  {
    "path": "pingora-limits/LICENSE",
    "content": "\n                                 Apache License\n                           Version 2.0, January 2004\n                        http://www.apache.org/licenses/\n\n   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION\n\n   1. Definitions.\n\n      \"License\" shall mean the terms and conditions for use, reproduction,\n      and distribution as defined by Sections 1 through 9 of this document.\n\n      \"Licensor\" shall mean the copyright owner or entity authorized by\n      the copyright owner that is granting the License.\n\n      \"Legal Entity\" shall mean the union of the acting entity and all\n      other entities that control, are controlled by, or are under common\n      control with that entity. For the purposes of this definition,\n      \"control\" means (i) the power, direct or indirect, to cause the\n      direction or management of such entity, whether by contract or\n      otherwise, or (ii) ownership of fifty percent (50%) or more of the\n      outstanding shares, or (iii) beneficial ownership of such entity.\n\n      \"You\" (or \"Your\") shall mean an individual or Legal Entity\n      exercising permissions granted by this License.\n\n      \"Source\" form shall mean the preferred form for making modifications,\n      including but not limited to software source code, documentation\n      source, and configuration files.\n\n      \"Object\" form shall mean any form resulting from mechanical\n      transformation or translation of a Source form, including but\n      not limited to compiled object code, generated documentation,\n      and conversions to other media types.\n\n      \"Work\" shall mean the work of authorship, whether in Source or\n      Object form, made available under the License, as indicated by a\n      copyright notice that is included in or attached to the work\n      (an example is provided in the Appendix below).\n\n      \"Derivative Works\" shall mean any work, whether in Source or Object\n      form, that is based on (or derived from) the Work and for which the\n      editorial revisions, annotations, elaborations, or other modifications\n      represent, as a whole, an original work of authorship. For the purposes\n      of this License, Derivative Works shall not include works that remain\n      separable from, or merely link (or bind by name) to the interfaces of,\n      the Work and Derivative Works thereof.\n\n      \"Contribution\" shall mean any work of authorship, including\n      the original version of the Work and any modifications or additions\n      to that Work or Derivative Works thereof, that is intentionally\n      submitted to Licensor for inclusion in the Work by the copyright owner\n      or by an individual or Legal Entity authorized to submit on behalf of\n      the copyright owner. For the purposes of this definition, \"submitted\"\n      means any form of electronic, verbal, or written communication sent\n      to the Licensor or its representatives, including but not limited to\n      communication on electronic mailing lists, source code control systems,\n      and issue tracking systems that are managed by, or on behalf of, the\n      Licensor for the purpose of discussing and improving the Work, but\n      excluding communication that is conspicuously marked or otherwise\n      designated in writing by the copyright owner as \"Not a Contribution.\"\n\n      \"Contributor\" shall mean Licensor and any individual or Legal Entity\n      on behalf of whom a Contribution has been received by Licensor and\n      subsequently incorporated within the Work.\n\n   2. Grant of Copyright License. Subject to the terms and conditions of\n      this License, each Contributor hereby grants to You a perpetual,\n      worldwide, non-exclusive, no-charge, royalty-free, irrevocable\n      copyright license to reproduce, prepare Derivative Works of,\n      publicly display, publicly perform, sublicense, and distribute the\n      Work and such Derivative Works in Source or Object form.\n\n   3. Grant of Patent License. Subject to the terms and conditions of\n      this License, each Contributor hereby grants to You a perpetual,\n      worldwide, non-exclusive, no-charge, royalty-free, irrevocable\n      (except as stated in this section) patent license to make, have made,\n      use, offer to sell, sell, import, and otherwise transfer the Work,\n      where such license applies only to those patent claims licensable\n      by such Contributor that are necessarily infringed by their\n      Contribution(s) alone or by combination of their Contribution(s)\n      with the Work to which such Contribution(s) was submitted. If You\n      institute patent litigation against any entity (including a\n      cross-claim or counterclaim in a lawsuit) alleging that the Work\n      or a Contribution incorporated within the Work constitutes direct\n      or contributory patent infringement, then any patent licenses\n      granted to You under this License for that Work shall terminate\n      as of the date such litigation is filed.\n\n   4. Redistribution. You may reproduce and distribute copies of the\n      Work or Derivative Works thereof in any medium, with or without\n      modifications, and in Source or Object form, provided that You\n      meet the following conditions:\n\n      (a) You must give any other recipients of the Work or\n          Derivative Works a copy of this License; and\n\n      (b) You must cause any modified files to carry prominent notices\n          stating that You changed the files; and\n\n      (c) You must retain, in the Source form of any Derivative Works\n          that You distribute, all copyright, patent, trademark, and\n          attribution notices from the Source form of the Work,\n          excluding those notices that do not pertain to any part of\n          the Derivative Works; and\n\n      (d) If the Work includes a \"NOTICE\" text file as part of its\n          distribution, then any Derivative Works that You distribute must\n          include a readable copy of the attribution notices contained\n          within such NOTICE file, excluding those notices that do not\n          pertain to any part of the Derivative Works, in at least one\n          of the following places: within a NOTICE text file distributed\n          as part of the Derivative Works; within the Source form or\n          documentation, if provided along with the Derivative Works; or,\n          within a display generated by the Derivative Works, if and\n          wherever such third-party notices normally appear. The contents\n          of the NOTICE file are for informational purposes only and\n          do not modify the License. You may add Your own attribution\n          notices within Derivative Works that You distribute, alongside\n          or as an addendum to the NOTICE text from the Work, provided\n          that such additional attribution notices cannot be construed\n          as modifying the License.\n\n      You may add Your own copyright statement to Your modifications and\n      may provide additional or different license terms and conditions\n      for use, reproduction, or distribution of Your modifications, or\n      for any such Derivative Works as a whole, provided Your use,\n      reproduction, and distribution of the Work otherwise complies with\n      the conditions stated in this License.\n\n   5. Submission of Contributions. Unless You explicitly state otherwise,\n      any Contribution intentionally submitted for inclusion in the Work\n      by You to the Licensor shall be under the terms and conditions of\n      this License, without any additional terms or conditions.\n      Notwithstanding the above, nothing herein shall supersede or modify\n      the terms of any separate license agreement you may have executed\n      with Licensor regarding such Contributions.\n\n   6. Trademarks. This License does not grant permission to use the trade\n      names, trademarks, service marks, or product names of the Licensor,\n      except as required for reasonable and customary use in describing the\n      origin of the Work and reproducing the content of the NOTICE file.\n\n   7. Disclaimer of Warranty. Unless required by applicable law or\n      agreed to in writing, Licensor provides the Work (and each\n      Contributor provides its Contributions) on an \"AS IS\" BASIS,\n      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or\n      implied, including, without limitation, any warranties or conditions\n      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A\n      PARTICULAR PURPOSE. You are solely responsible for determining the\n      appropriateness of using or redistributing the Work and assume any\n      risks associated with Your exercise of permissions under this License.\n\n   8. Limitation of Liability. In no event and under no legal theory,\n      whether in tort (including negligence), contract, or otherwise,\n      unless required by applicable law (such as deliberate and grossly\n      negligent acts) or agreed to in writing, shall any Contributor be\n      liable to You for damages, including any direct, indirect, special,\n      incidental, or consequential damages of any character arising as a\n      result of this License or out of the use or inability to use the\n      Work (including but not limited to damages for loss of goodwill,\n      work stoppage, computer failure or malfunction, or any and all\n      other commercial damages or losses), even if such Contributor\n      has been advised of the possibility of such damages.\n\n   9. Accepting Warranty or Additional Liability. While redistributing\n      the Work or Derivative Works thereof, You may choose to offer,\n      and charge a fee for, acceptance of support, warranty, indemnity,\n      or other liability obligations and/or rights consistent with this\n      License. However, in accepting such obligations, You may act only\n      on Your own behalf and on Your sole responsibility, not on behalf\n      of any other Contributor, and only if You agree to indemnify,\n      defend, and hold each Contributor harmless for any liability\n      incurred by, or claims asserted against, such Contributor by reason\n      of your accepting any such warranty or additional liability.\n\n   END OF TERMS AND CONDITIONS\n\n   APPENDIX: How to apply the Apache License to your work.\n\n      To apply the Apache License to your work, attach the following\n      boilerplate notice, with the fields enclosed by brackets \"[]\"\n      replaced with your own identifying information. (Don't include\n      the brackets!)  The text should be enclosed in the appropriate\n      comment syntax for the file format. We also recommend that a\n      file or class name and description of purpose be included on the\n      same \"printed page\" as the copyright notice for easier\n      identification within third-party archives.\n\n   Copyright [yyyy] [name of copyright owner]\n\n   Licensed under the Apache License, Version 2.0 (the \"License\");\n   you may not use this file except in compliance with the License.\n   You may obtain a copy of the License at\n\n       http://www.apache.org/licenses/LICENSE-2.0\n\n   Unless required by applicable law or agreed to in writing, software\n   distributed under the License is distributed on an \"AS IS\" BASIS,\n   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n   See the License for the specific language governing permissions and\n   limitations under the License.\n"
  },
  {
    "path": "pingora-limits/benches/benchmark.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n#[cfg(feature = \"dhat-heap\")]\n#[global_allocator]\nstatic ALLOC: dhat::Alloc = dhat::Alloc;\n\nuse ahash::RandomState;\nuse dashmap::DashMap;\nuse pingora_limits::estimator::Estimator;\nuse rand::distributions::Uniform;\nuse rand::{thread_rng, Rng};\nuse std::collections::HashMap;\nuse std::sync::atomic::{AtomicUsize, Ordering};\nuse std::sync::Arc;\nuse std::sync::Mutex;\nuse std::thread;\nuse std::time::Instant;\n\ntrait Counter {\n    fn incr(&self, key: u32, value: usize);\n    fn name() -> &'static str;\n}\n\n#[derive(Default)]\nstruct NaiveCounter(Mutex<HashMap<u32, usize>>);\nimpl Counter for NaiveCounter {\n    fn incr(&self, key: u32, value: usize) {\n        let mut map = self.0.lock().unwrap();\n        if let Some(v) = map.get_mut(&key) {\n            *v += value;\n        } else {\n            map.insert(key, value);\n        }\n    }\n\n    fn name() -> &'static str {\n        \"Naive Counter\"\n    }\n}\n\n#[derive(Default)]\nstruct OptimizedCounter(DashMap<u32, AtomicUsize, RandomState>);\nimpl Counter for OptimizedCounter {\n    fn incr(&self, key: u32, value: usize) {\n        if let Some(v) = self.0.get(&key) {\n            v.fetch_add(value, Ordering::Relaxed);\n            return;\n        }\n        self.0.insert(key, AtomicUsize::new(value));\n    }\n\n    fn name() -> &'static str {\n        \"Optimized Counter\"\n    }\n}\n\nimpl Counter for Estimator {\n    fn incr(&self, key: u32, value: usize) {\n        self.incr(key, value as isize);\n    }\n\n    fn name() -> &'static str {\n        \"Pingora Estimator\"\n    }\n}\n\nfn run_bench<T: Counter>(\n    counter: &T,\n    samples: usize,\n    distribution: &Uniform<u32>,\n    test_name: &str,\n) {\n    let mut rng = thread_rng();\n    let before = Instant::now();\n    for _ in 0..samples {\n        let event: u32 = rng.sample(distribution);\n        counter.incr(event, 1);\n    }\n    let elapsed = before.elapsed();\n    println!(\n        \"{} {test_name} {:?} total, {:?} avg per operation\",\n        T::name(),\n        elapsed,\n        elapsed / samples as u32\n    );\n}\n\nfn run_threaded_bench<T: Counter + Send + Sync + 'static>(\n    threads: usize,\n    counter: Arc<T>,\n    samples: usize,\n    distribution: &Uniform<u32>,\n) {\n    let mut handlers = vec![];\n    for i in 0..threads {\n        let est = counter.clone();\n        let dist = *distribution;\n        let handler = thread::spawn(move || {\n            run_bench(est.as_ref(), samples, &dist, &format!(\"thread#{i}\"));\n        });\n        handlers.push(handler);\n    }\n    for thread in handlers {\n        thread.join().unwrap();\n    }\n}\n\n/*\nPingora Estimator single thread 1.042849543s total, 10ns avg per operation\nNaive Counter single thread 5.12641496s total, 51ns avg per operation\nOptimized Counter single thread 4.302553352s total, 43ns avg per operation\nPingora Estimator thread#7 2.654667606s total, 212ns avg per operation\nPingora Estimator thread#2 2.65651993s total, 212ns avg per operation\nPingora Estimator thread#4 2.658225266s total, 212ns avg per operation\nPingora Estimator thread#0 2.660603361s total, 212ns avg per operation\nPingora Estimator thread#1 2.66139014s total, 212ns avg per operation\nPingora Estimator thread#6 2.663498849s total, 213ns avg per operation\nPingora Estimator thread#5 2.663344276s total, 213ns avg per operation\nPingora Estimator thread#3 2.664652951s total, 213ns avg per operation\nNaive Counter thread#7 18.795881242s total, 1.503µs avg per operation\nNaive Counter thread#1 18.805652672s total, 1.504µs avg per operation\nNaive Counter thread#6 18.818084416s total, 1.505µs avg per operation\nNaive Counter thread#4 18.832778982s total, 1.506µs avg per operation\nNaive Counter thread#3 18.833952715s total, 1.506µs avg per operation\nNaive Counter thread#2 18.837975133s total, 1.507µs avg per operation\nNaive Counter thread#0 18.8397464s total, 1.507µs avg per operation\nNaive Counter thread#5 18.842616299s total, 1.507µs avg per operation\nOptimized Counter thread#4 2.650860314s total, 212ns avg per operation\nOptimized Counter thread#0 2.651867013s total, 212ns avg per operation\nOptimized Counter thread#2 2.656473381s total, 212ns avg per operation\nOptimized Counter thread#5 2.657715876s total, 212ns avg per operation\nOptimized Counter thread#1 2.658275111s total, 212ns avg per operation\nOptimized Counter thread#7 2.658770751s total, 212ns avg per operation\nOptimized Counter thread#6 2.659831251s total, 212ns avg per operation\nOptimized Counter thread#3 2.664375398s total, 213ns avg per operation\n*/\n\n/* cargo bench --features dhat-heap for memory info\n\nPingora Estimator single thread 1.066846098s total, 10ns avg per operation\ndhat: Total:     26,184 bytes in 9 blocks\ndhat: At t-gmax: 26,184 bytes in 9 blocks\ndhat: At t-end:  1,464 bytes in 5 blocks\ndhat: The data has been saved to dhat-heap.json, and is viewable with dhat/dh_view.html\nNaive Counter single thread 5.429089242s total, 54ns avg per operation\ndhat: Total:     71,303,260 bytes in 20 blocks\ndhat: At t-gmax: 53,477,392 bytes in 2 blocks\ndhat: At t-end:  0 bytes in 0 blocks\ndhat: The data has been saved to dhat-heap.json, and is viewable with dhat/dh_view.html\nOptimized Counter single thread 4.361720355s total, 43ns avg per operation\ndhat: Total:     71,307,722 bytes in 491 blocks\ndhat: At t-gmax: 36,211,208 bytes in 34 blocks\ndhat: At t-end:  0 bytes in 0 blocks\ndhat: The data has been saved to dhat-heap.json, and is viewable with dhat/dh_view.html\n*/\n\nfn main() {\n    const SAMPLES: usize = 100_000_000;\n    const THREADS: usize = 8;\n    const ITEMS: u32 = 1_000_000;\n    const SAMPLES_PER_THREAD: usize = SAMPLES / THREADS;\n    let distribution = Uniform::new(0, ITEMS);\n\n    // single thread\n    {\n        #[cfg(feature = \"dhat-heap\")]\n        let _profiler = dhat::Profiler::new_heap();\n        let pingora_est = Estimator::new(3, 1024);\n        run_bench(&pingora_est, SAMPLES, &distribution, \"single thread\");\n    }\n\n    {\n        #[cfg(feature = \"dhat-heap\")]\n        let _profiler = dhat::Profiler::new_heap();\n        let naive: NaiveCounter = Default::default();\n        run_bench(&naive, SAMPLES, &distribution, \"single thread\");\n    }\n\n    {\n        #[cfg(feature = \"dhat-heap\")]\n        let _profiler = dhat::Profiler::new_heap();\n        let optimized: OptimizedCounter = Default::default();\n        run_bench(&optimized, SAMPLES, &distribution, \"single thread\");\n    }\n\n    // multithread\n    let pingora_est = Arc::new(Estimator::new(3, 1024));\n    run_threaded_bench(THREADS, pingora_est, SAMPLES_PER_THREAD, &distribution);\n\n    let naive: Arc<NaiveCounter> = Default::default();\n    run_threaded_bench(THREADS, naive, SAMPLES_PER_THREAD, &distribution);\n\n    let optimized: Arc<OptimizedCounter> = Default::default();\n    run_threaded_bench(THREADS, optimized, SAMPLES_PER_THREAD, &distribution);\n}\n"
  },
  {
    "path": "pingora-limits/src/estimator.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! The estimator module contains a Count-Min Sketch type to help estimate the frequency of an item.\n\nuse crate::hash;\nuse crate::RandomState;\nuse std::hash::Hash;\nuse std::sync::atomic::{AtomicIsize, Ordering};\n\n/// An implementation of a lock-free count–min sketch estimator. See the [wikipedia] page for more\n/// information.\n///\n/// [wikipedia]: https://en.wikipedia.org/wiki/Count%E2%80%93min_sketch\npub struct Estimator {\n    estimator: Box<[(Box<[AtomicIsize]>, RandomState)]>,\n}\n\nimpl Estimator {\n    /// Create a new `Estimator` with the given amount of hashes and columns (slots).\n    pub fn new(hashes: usize, slots: usize) -> Self {\n        Self {\n            estimator: (0..hashes)\n                .map(|_| (0..slots).map(|_| AtomicIsize::new(0)).collect::<Vec<_>>())\n                .map(|slot| (slot.into_boxed_slice(), RandomState::new()))\n                .collect::<Vec<_>>()\n                .into_boxed_slice(),\n        }\n    }\n\n    /// Increment `key` by the value given. Return the new estimated value as a result.\n    /// Note: overflow can happen. When some of the internal counters overflow, a negative number\n    /// will be returned. It is up to the caller to catch and handle this case.\n    pub fn incr<T: Hash>(&self, key: T, value: isize) -> isize {\n        self.estimator\n            .iter()\n            .fold(isize::MAX, |min, (slot, hasher)| {\n                let hash = hash(&key, hasher) as usize;\n                let counter = &slot[hash % slot.len()];\n                // Overflow is allowed for simplicity\n                let current = counter.fetch_add(value, Ordering::Relaxed);\n                std::cmp::min(min, current + value)\n            })\n    }\n\n    /// Decrement `key` by the value given.\n    pub fn decr<T: Hash>(&self, key: T, value: isize) {\n        for (slot, hasher) in self.estimator.iter() {\n            let hash = hash(&key, hasher) as usize;\n            let counter = &slot[hash % slot.len()];\n            counter.fetch_sub(value, Ordering::Relaxed);\n        }\n    }\n\n    /// Get the estimated frequency of `key`.\n    pub fn get<T: Hash>(&self, key: T) -> isize {\n        self.estimator\n            .iter()\n            .fold(isize::MAX, |min, (slot, hasher)| {\n                let hash = hash(&key, hasher) as usize;\n                let counter = &slot[hash % slot.len()];\n                let current = counter.load(Ordering::Relaxed);\n                std::cmp::min(min, current)\n            })\n    }\n\n    /// Reset all values inside this `Estimator`.\n    pub fn reset(&self) {\n        self.estimator.iter().for_each(|(slot, _)| {\n            slot.iter()\n                .for_each(|counter| counter.store(0, Ordering::Relaxed))\n        });\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use super::*;\n\n    #[test]\n    fn incr() {\n        let est = Estimator::new(8, 8);\n        let v = est.incr(\"a\", 1);\n        assert_eq!(v, 1);\n        let v = est.incr(\"b\", 1);\n        assert_eq!(v, 1);\n        let v = est.incr(\"a\", 2);\n        assert_eq!(v, 3);\n        let v = est.incr(\"b\", 2);\n        assert_eq!(v, 3);\n    }\n\n    #[test]\n    fn desc() {\n        let est = Estimator::new(8, 8);\n        est.incr(\"a\", 3);\n        est.incr(\"b\", 3);\n        est.decr(\"a\", 1);\n        est.decr(\"b\", 1);\n        assert_eq!(est.get(\"a\"), 2);\n        assert_eq!(est.get(\"b\"), 2);\n    }\n\n    #[test]\n    fn get() {\n        let est = Estimator::new(8, 8);\n        est.incr(\"a\", 1);\n        est.incr(\"a\", 2);\n        est.incr(\"b\", 1);\n        est.incr(\"b\", 2);\n        assert_eq!(est.get(\"a\"), 3);\n        assert_eq!(est.get(\"b\"), 3);\n    }\n\n    #[test]\n    fn reset() {\n        let est = Estimator::new(8, 8);\n        est.incr(\"a\", 1);\n        est.incr(\"a\", 2);\n        est.incr(\"b\", 1);\n        est.incr(\"b\", 2);\n        est.decr(\"b\", 1);\n        est.reset();\n        assert_eq!(est.get(\"a\"), 0);\n        assert_eq!(est.get(\"b\"), 0);\n    }\n}\n"
  },
  {
    "path": "pingora-limits/src/inflight.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! The inflight module defines the [Inflight] type which estimates the count of events occurring\n//! at any point in time.\n\nuse crate::estimator::Estimator;\nuse crate::{hash, RandomState};\nuse std::hash::Hash;\nuse std::sync::Arc;\n\n/// An `Inflight` type tracks the frequency of actions that are actively occurring. When the value\n/// is dropped from scope, the count will automatically decrease.\npub struct Inflight {\n    estimator: Arc<Estimator>,\n    hasher: RandomState,\n}\n\n// fixed parameters for simplicity: hashes: h, slots: n\n// Time complexity for a lookup operation is O(h). Space complexity is O(h*n)\n// False positive ratio is 1/(n^h)\n// We choose a small h and a large n to keep lookup cheap and FP ratio low\nconst HASHES: usize = 4;\nconst SLOTS: usize = 8192;\n\nimpl Inflight {\n    /// Create a new `Inflight`.\n    pub fn new() -> Self {\n        Inflight {\n            estimator: Arc::new(Estimator::new(HASHES, SLOTS)),\n            hasher: RandomState::new(),\n        }\n    }\n\n    /// Increment `key` by the value given. The return value is a tuple of a [Guard] and the\n    /// estimated count.\n    pub fn incr<T: Hash>(&self, key: T, value: isize) -> (Guard, isize) {\n        let guard = Guard {\n            estimator: self.estimator.clone(),\n            id: hash(key, &self.hasher),\n            value,\n        };\n        let estimation = guard.incr();\n        (guard, estimation)\n    }\n}\n\n/// A `Guard` is returned when an `Inflight` key is incremented via [Inflight::incr].\npub struct Guard {\n    estimator: Arc<Estimator>,\n    // store the hash instead of the actual key to save space\n    id: u64,\n    value: isize,\n}\n\nimpl Guard {\n    /// Increment the key's value that the `Guard` was created from.\n    pub fn incr(&self) -> isize {\n        self.estimator.incr(self.id, self.value)\n    }\n\n    /// Get the estimated count of the key that the `Guard` was created from.\n    pub fn get(&self) -> isize {\n        self.estimator.get(self.id)\n    }\n}\n\nimpl Drop for Guard {\n    fn drop(&mut self) {\n        self.estimator.decr(self.id, self.value)\n    }\n}\n\nimpl std::fmt::Debug for Guard {\n    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {\n        f.debug_struct(\"Guard\")\n            .field(\"id\", &self.id)\n            .field(\"value\", &self.value)\n            // no need to dump shared estimator\n            .finish()\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use super::*;\n\n    #[test]\n    fn inflight_count() {\n        let inflight = Inflight::new();\n        let (g1, v) = inflight.incr(\"a\", 1);\n        assert_eq!(v, 1);\n        let (g2, v) = inflight.incr(\"a\", 2);\n        assert_eq!(v, 3);\n\n        drop(g1);\n\n        assert_eq!(g2.get(), 2);\n\n        drop(g2);\n\n        let (_, v) = inflight.incr(\"a\", 1);\n        assert_eq!(v, 1);\n    }\n}\n"
  },
  {
    "path": "pingora-limits/src/lib.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! The pingora_limits crate contains modules that can help introduce things like rate limiting or\n//! thread-safe event count estimation.\n\n#![warn(clippy::all)]\n#![allow(clippy::new_without_default)]\n#![allow(clippy::type_complexity)]\n\npub mod estimator;\npub mod inflight;\npub mod rate;\n\nuse ahash::RandomState;\nuse std::hash::Hash;\n\n#[inline]\nfn hash<T: Hash>(key: T, hasher: &RandomState) -> u64 {\n    hasher.hash_one(key)\n}\n"
  },
  {
    "path": "pingora-limits/src/rate.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! The rate module defines the [Rate] type that helps estimate the occurrence of events over a\n//! period of time.\n\nuse crate::estimator::Estimator;\nuse std::hash::Hash;\nuse std::sync::atomic::{AtomicBool, AtomicU64, Ordering};\nuse std::time::{Duration, Instant};\n\n/// Input struct to custom functions for calculating rate. Includes the counts\n/// from the current interval, previous interval, the configured duration of an\n/// interval, and the fraction into the current interval that the sample was\n/// taken.\n///\n/// Ex. If the interval to the Rate instance is `10s`, and the rate calculation\n/// is taken at 2 seconds after the start of the current interval, then the\n/// fraction of the current interval returned in this struct will be `0.2`\n/// meaning 20% of the current interval has elapsed\n#[non_exhaustive]\n#[derive(Debug, Clone, Copy, PartialEq, PartialOrd)]\npub struct RateComponents {\n    pub prev_samples: isize,\n    pub curr_samples: isize,\n    pub interval: Duration,\n    pub current_interval_fraction: f64,\n}\n\n/// A rate calculation function which uses a good estimate of the rate of events over the past\n/// `interval` time.\n///\n/// Specifically, it linearly interpolates between the event counts of the previous and current\n/// periods based on how far into the current period we are, as described in this post:\n/// <https://blog.cloudflare.com/counting-things-a-lot-of-different-things/>\n#[allow(dead_code)]\npub static PROPORTIONAL_RATE_ESTIMATE_CALC_FN: fn(RateComponents) -> f64 =\n    |rate_info: RateComponents| {\n        let prev = rate_info.prev_samples as f64;\n        let curr = rate_info.curr_samples as f64;\n        let interval_secs = rate_info.interval.as_secs_f64();\n        let interval_fraction = rate_info.current_interval_fraction;\n\n        let weighted_count = prev * (1. - interval_fraction) + curr;\n        weighted_count / interval_secs\n    };\n\n/// A stable rate estimator that reports the rate of events per period of `interval` time.\n///\n/// It counts events for periods of `interval` and returns the average rate of the latest completed\n/// period while counting events for the current (partial) period.\npub struct Rate {\n    // 2 slots so that we use one to collect the current events and the other to report rate\n    red_slot: Estimator,\n    blue_slot: Estimator,\n    red_or_blue: AtomicBool, // true: the current slot is red, otherwise blue\n    start: Instant,\n    // Use u64 below instead of Instant because we want atomic operation\n    reset_interval_ms: u64, // the time interval to reset `current` and move it to `previous`\n    last_reset_time: AtomicU64, // the timestamp in ms since `start`\n    interval: Duration,\n}\n\n// see inflight module for the meaning for these numbers\nconst HASHES: usize = 4;\nconst SLOTS: usize = 1024; // This value can be lower if interval is short (key cardinality is low)\n\nimpl Rate {\n    /// Create a new `Rate` with the given interval.\n    pub fn new(interval: std::time::Duration) -> Self {\n        Rate::new_with_estimator_config(interval, HASHES, SLOTS)\n    }\n\n    /// Create a new `Rate` with the given interval and Estimator config with the given amount of hashes and columns (slots).\n    #[inline]\n    pub fn new_with_estimator_config(\n        interval: std::time::Duration,\n        hashes: usize,\n        slots: usize,\n    ) -> Self {\n        Rate {\n            red_slot: Estimator::new(hashes, slots),\n            blue_slot: Estimator::new(hashes, slots),\n            red_or_blue: AtomicBool::new(true),\n            start: Instant::now(),\n            reset_interval_ms: interval.as_millis() as u64, // should be small not to overflow\n            last_reset_time: AtomicU64::new(0),\n            interval,\n        }\n    }\n\n    fn current(&self, red_or_blue: bool) -> &Estimator {\n        if red_or_blue {\n            &self.red_slot\n        } else {\n            &self.blue_slot\n        }\n    }\n\n    fn previous(&self, red_or_blue: bool) -> &Estimator {\n        if red_or_blue {\n            &self.blue_slot\n        } else {\n            &self.red_slot\n        }\n    }\n\n    fn red_or_blue(&self) -> bool {\n        self.red_or_blue.load(Ordering::SeqCst)\n    }\n\n    /// Return the per second rate estimation.\n    ///\n    /// This is the average rate of the latest completed period of length `interval`.\n    pub fn rate<T: Hash>(&self, key: &T) -> f64 {\n        let past_ms = self.maybe_reset();\n        if past_ms >= self.reset_interval_ms * 2 {\n            // already missed 2 intervals, no data, just report 0 as a short cut\n            return 0f64;\n        }\n\n        self.previous(self.red_or_blue()).get(key) as f64 * 1000.0 / self.reset_interval_ms as f64\n    }\n\n    /// Report new events and return number of events seen so far in the current interval.\n    pub fn observe<T: Hash>(&self, key: &T, events: isize) -> isize {\n        self.maybe_reset();\n        self.current(self.red_or_blue()).incr(key, events)\n    }\n\n    // reset if needed, return the time since last reset for other fn to use\n    fn maybe_reset(&self) -> u64 {\n        // should be short enough not to overflow\n        let now = Instant::now().duration_since(self.start).as_millis() as u64;\n        let last_reset = self.last_reset_time.load(Ordering::SeqCst);\n        let past_ms = now - last_reset;\n\n        if past_ms < self.reset_interval_ms {\n            // no need to reset\n            return past_ms;\n        }\n        let red_or_blue = self.red_or_blue();\n        match self.last_reset_time.compare_exchange(\n            last_reset,\n            now,\n            Ordering::SeqCst,\n            Ordering::Acquire,\n        ) {\n            Ok(_) => {\n                // first clear the previous slot\n                self.previous(red_or_blue).reset();\n                // then flip the flag to tell others to use the reset slot\n                self.red_or_blue.store(!red_or_blue, Ordering::SeqCst);\n                // if current time is beyond 2 intervals, the data stored in the previous slot\n                // is also stale, we should clear that too\n                if now - last_reset >= self.reset_interval_ms * 2 {\n                    // Note that this is the previous one now because we just flipped self.red_or_blue\n                    self.current(red_or_blue).reset();\n                }\n            }\n            Err(new) => {\n                // another thread beats us to it\n                assert!(new >= now - 1000); // double check that the new timestamp looks right\n            }\n        }\n\n        past_ms\n    }\n\n    /// Get the current rate as calculated with the given closure. This closure\n    /// will take an argument containing all the accessible information about\n    /// the rate from this object and allow the caller to make their own\n    /// estimation of rate based on:\n    ///\n    /// 1. The accumulated samples in the current interval (in progress)\n    /// 2. The accumulated samples in the previous interval (completed)\n    /// 3. The size of the interval\n    /// 4. Elapsed fraction of current interval for this sample (0..1)\n    ///\n    pub fn rate_with<F, T, K>(&self, key: &K, mut rate_calc_fn: F) -> T\n    where\n        F: FnMut(RateComponents) -> T,\n        K: Hash,\n    {\n        let past_ms = self.maybe_reset();\n\n        let (prev_samples, curr_samples) = if past_ms >= self.reset_interval_ms * 2 {\n            // already missed 2 intervals, no data, just report 0 as a short cut\n            (0, 0)\n        } else if past_ms >= self.reset_interval_ms {\n            (self.previous(self.red_or_blue()).get(key), 0)\n        } else {\n            let (prev_est, curr_est) = if self.red_or_blue() {\n                (&self.blue_slot, &self.red_slot)\n            } else {\n                (&self.red_slot, &self.blue_slot)\n            };\n\n            (prev_est.get(key), curr_est.get(key))\n        };\n\n        rate_calc_fn(RateComponents {\n            interval: self.interval,\n            prev_samples,\n            curr_samples,\n            current_interval_fraction: (past_ms % self.reset_interval_ms) as f64\n                / self.reset_interval_ms as f64,\n        })\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use float_cmp::assert_approx_eq;\n\n    use super::*;\n    use std::thread::sleep;\n    use std::time::Duration;\n\n    #[test]\n    fn test_observe_rate() {\n        let r = Rate::new(Duration::from_secs(1));\n        let key = 1;\n\n        // second: 0\n        let observed = r.observe(&key, 3);\n        assert_eq!(observed, 3);\n        let observed = r.observe(&key, 2);\n        assert_eq!(observed, 5);\n        assert_eq!(r.rate(&key), 0f64); // no estimation yet because the interval has not passed\n\n        // second: 1\n        sleep(Duration::from_secs(1));\n        let observed = r.observe(&key, 4);\n        assert_eq!(observed, 4);\n        assert_eq!(r.rate(&key), 5f64); // 5 rps\n\n        // second: 2\n        sleep(Duration::from_secs(1));\n        assert_eq!(r.rate(&key), 4f64);\n\n        // second: 3\n        sleep(Duration::from_secs(1));\n        assert_eq!(r.rate(&key), 0f64); // no event observed in the past 2 seconds\n    }\n\n    /// Assertion that 2 numbers are close within a generous margin. These\n    /// tests are doing a lot of literal sleeping, so the measured results\n    /// can't be accurate or consistent. This function does an assert with a\n    /// generous tolerance\n    fn assert_eq_ish(left: f64, right: f64) {\n        assert_approx_eq!(f64, left, right, epsilon = 0.15)\n    }\n\n    #[test]\n    fn test_observe_rate_custom_90_10() {\n        let r = Rate::new(Duration::from_secs(1));\n        let key = 1;\n\n        let rate_90_10_fn = |rate_info: RateComponents| {\n            let prev = rate_info.prev_samples as f64;\n            let curr = rate_info.curr_samples as f64;\n            (prev * 0.1 + curr * 0.9) / rate_info.interval.as_secs_f64()\n        };\n\n        // second: 0\n        let observed = r.observe(&key, 3);\n        assert_eq!(observed, 3);\n        let observed = r.observe(&key, 2);\n        assert_eq!(observed, 5);\n        assert_eq!(r.rate_with(&key, rate_90_10_fn), 5. * 0.9);\n\n        // second: 1\n        sleep(Duration::from_secs(1));\n        let observed = r.observe(&key, 4);\n        assert_eq!(observed, 4);\n        assert_eq!(r.rate_with(&key, rate_90_10_fn), 5. * 0.1 + 4. * 0.9);\n\n        // second: 2\n        sleep(Duration::from_secs(1));\n        assert_eq!(r.rate_with(&key, rate_90_10_fn), 4. * 0.1);\n\n        // second: 3\n        sleep(Duration::from_secs(1));\n        assert_eq!(r.rate_with(&key, rate_90_10_fn), 0f64);\n    }\n\n    #[test]\n    fn test_observe_rate_custom_proportional() {\n        let r = Rate::new(Duration::from_secs(1));\n        let key = 1;\n\n        // second: 0\n        let observed = r.observe(&key, 3);\n        assert_eq!(observed, 3);\n        let observed = r.observe(&key, 2);\n        assert_eq!(observed, 5);\n        assert_eq_ish(r.rate_with(&key, PROPORTIONAL_RATE_ESTIMATE_CALC_FN), 5.);\n\n        // second 0.5\n        sleep(Duration::from_secs_f64(0.5));\n        assert_eq_ish(r.rate_with(&key, PROPORTIONAL_RATE_ESTIMATE_CALC_FN), 5.);\n        // rate() just looks at the previous interval, ignores current interval\n        assert_eq_ish(r.rate(&key), 0.);\n\n        // second: 1\n        sleep(Duration::from_secs_f64(0.5));\n        let observed = r.observe(&key, 4);\n        assert_eq!(observed, 4);\n        assert_eq_ish(r.rate_with(&key, PROPORTIONAL_RATE_ESTIMATE_CALC_FN), 9.);\n\n        // second 1.75\n        sleep(Duration::from_secs_f64(0.75));\n        assert_eq_ish(\n            r.rate_with(&key, PROPORTIONAL_RATE_ESTIMATE_CALC_FN),\n            5. * 0.25 + 4.,\n        );\n\n        // second: 2\n        sleep(Duration::from_secs_f64(0.25));\n        assert_eq_ish(r.rate_with(&key, PROPORTIONAL_RATE_ESTIMATE_CALC_FN), 4.);\n        assert_eq_ish(r.rate(&key), 4.);\n\n        // second: 2.5\n        sleep(Duration::from_secs_f64(0.5));\n        assert_eq_ish(\n            r.rate_with(&key, PROPORTIONAL_RATE_ESTIMATE_CALC_FN),\n            4. / 2.,\n        );\n        assert_eq_ish(r.rate(&key), 4.);\n\n        // second: 3\n        sleep(Duration::from_secs(1));\n        assert_eq!(r.rate_with(&key, PROPORTIONAL_RATE_ESTIMATE_CALC_FN), 0f64);\n    }\n}\n"
  },
  {
    "path": "pingora-load-balancing/Cargo.toml",
    "content": "[package]\nname = \"pingora-load-balancing\"\nversion = \"0.8.0\"\nauthors = [\"Yuchen Wu <yuchen@cloudflare.com>\"]\nlicense = \"Apache-2.0\"\nedition = \"2021\"\nrepository = \"https://github.com/cloudflare/pingora\"\ncategories = [\"network-programming\"]\nkeywords = [\"proxy\", \"pingora\"]\ndescription = \"\"\"\nCommon load balancing features for Pingora proxy.\n\"\"\"\n\n# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html\n[lib]\nname = \"pingora_load_balancing\"\npath = \"src/lib.rs\"\n\n[dependencies]\nasync-trait = { workspace = true }\npingora-http = { version = \"0.8.0\", path = \"../pingora-http\" }\npingora-error = { version = \"0.8.0\", path = \"../pingora-error\" }\npingora-core = { version = \"0.8.0\", path = \"../pingora-core\", default-features = false }\npingora-ketama = { version = \"0.8.0\", path = \"../pingora-ketama\" }\npingora-runtime = { version = \"0.8.0\", path = \"../pingora-runtime\" }\narc-swap = \"1\"\nfnv = \"1\"\nrand = \"0.8\"\ntokio = { workspace = true }\nfutures = \"0\"\nlog = { workspace = true }\nhttp = { workspace = true }\nderivative.workspace = true\n\n[dev-dependencies]\n\n[features]\ndefault = []\nopenssl = [\"pingora-core/openssl\", \"openssl_derived\"]\nboringssl = [\"pingora-core/boringssl\", \"openssl_derived\"]\nrustls = [\"pingora-core/rustls\", \"any_tls\"]\ns2n = [\"pingora-core/s2n\", \"any_tls\"]\nopenssl_derived = [\"any_tls\"]\nany_tls = []\nv2 = [\"pingora-ketama/v2\"]\n"
  },
  {
    "path": "pingora-load-balancing/LICENSE",
    "content": "\n                                 Apache License\n                           Version 2.0, January 2004\n                        http://www.apache.org/licenses/\n\n   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION\n\n   1. Definitions.\n\n      \"License\" shall mean the terms and conditions for use, reproduction,\n      and distribution as defined by Sections 1 through 9 of this document.\n\n      \"Licensor\" shall mean the copyright owner or entity authorized by\n      the copyright owner that is granting the License.\n\n      \"Legal Entity\" shall mean the union of the acting entity and all\n      other entities that control, are controlled by, or are under common\n      control with that entity. For the purposes of this definition,\n      \"control\" means (i) the power, direct or indirect, to cause the\n      direction or management of such entity, whether by contract or\n      otherwise, or (ii) ownership of fifty percent (50%) or more of the\n      outstanding shares, or (iii) beneficial ownership of such entity.\n\n      \"You\" (or \"Your\") shall mean an individual or Legal Entity\n      exercising permissions granted by this License.\n\n      \"Source\" form shall mean the preferred form for making modifications,\n      including but not limited to software source code, documentation\n      source, and configuration files.\n\n      \"Object\" form shall mean any form resulting from mechanical\n      transformation or translation of a Source form, including but\n      not limited to compiled object code, generated documentation,\n      and conversions to other media types.\n\n      \"Work\" shall mean the work of authorship, whether in Source or\n      Object form, made available under the License, as indicated by a\n      copyright notice that is included in or attached to the work\n      (an example is provided in the Appendix below).\n\n      \"Derivative Works\" shall mean any work, whether in Source or Object\n      form, that is based on (or derived from) the Work and for which the\n      editorial revisions, annotations, elaborations, or other modifications\n      represent, as a whole, an original work of authorship. For the purposes\n      of this License, Derivative Works shall not include works that remain\n      separable from, or merely link (or bind by name) to the interfaces of,\n      the Work and Derivative Works thereof.\n\n      \"Contribution\" shall mean any work of authorship, including\n      the original version of the Work and any modifications or additions\n      to that Work or Derivative Works thereof, that is intentionally\n      submitted to Licensor for inclusion in the Work by the copyright owner\n      or by an individual or Legal Entity authorized to submit on behalf of\n      the copyright owner. For the purposes of this definition, \"submitted\"\n      means any form of electronic, verbal, or written communication sent\n      to the Licensor or its representatives, including but not limited to\n      communication on electronic mailing lists, source code control systems,\n      and issue tracking systems that are managed by, or on behalf of, the\n      Licensor for the purpose of discussing and improving the Work, but\n      excluding communication that is conspicuously marked or otherwise\n      designated in writing by the copyright owner as \"Not a Contribution.\"\n\n      \"Contributor\" shall mean Licensor and any individual or Legal Entity\n      on behalf of whom a Contribution has been received by Licensor and\n      subsequently incorporated within the Work.\n\n   2. Grant of Copyright License. Subject to the terms and conditions of\n      this License, each Contributor hereby grants to You a perpetual,\n      worldwide, non-exclusive, no-charge, royalty-free, irrevocable\n      copyright license to reproduce, prepare Derivative Works of,\n      publicly display, publicly perform, sublicense, and distribute the\n      Work and such Derivative Works in Source or Object form.\n\n   3. Grant of Patent License. Subject to the terms and conditions of\n      this License, each Contributor hereby grants to You a perpetual,\n      worldwide, non-exclusive, no-charge, royalty-free, irrevocable\n      (except as stated in this section) patent license to make, have made,\n      use, offer to sell, sell, import, and otherwise transfer the Work,\n      where such license applies only to those patent claims licensable\n      by such Contributor that are necessarily infringed by their\n      Contribution(s) alone or by combination of their Contribution(s)\n      with the Work to which such Contribution(s) was submitted. If You\n      institute patent litigation against any entity (including a\n      cross-claim or counterclaim in a lawsuit) alleging that the Work\n      or a Contribution incorporated within the Work constitutes direct\n      or contributory patent infringement, then any patent licenses\n      granted to You under this License for that Work shall terminate\n      as of the date such litigation is filed.\n\n   4. Redistribution. You may reproduce and distribute copies of the\n      Work or Derivative Works thereof in any medium, with or without\n      modifications, and in Source or Object form, provided that You\n      meet the following conditions:\n\n      (a) You must give any other recipients of the Work or\n          Derivative Works a copy of this License; and\n\n      (b) You must cause any modified files to carry prominent notices\n          stating that You changed the files; and\n\n      (c) You must retain, in the Source form of any Derivative Works\n          that You distribute, all copyright, patent, trademark, and\n          attribution notices from the Source form of the Work,\n          excluding those notices that do not pertain to any part of\n          the Derivative Works; and\n\n      (d) If the Work includes a \"NOTICE\" text file as part of its\n          distribution, then any Derivative Works that You distribute must\n          include a readable copy of the attribution notices contained\n          within such NOTICE file, excluding those notices that do not\n          pertain to any part of the Derivative Works, in at least one\n          of the following places: within a NOTICE text file distributed\n          as part of the Derivative Works; within the Source form or\n          documentation, if provided along with the Derivative Works; or,\n          within a display generated by the Derivative Works, if and\n          wherever such third-party notices normally appear. The contents\n          of the NOTICE file are for informational purposes only and\n          do not modify the License. You may add Your own attribution\n          notices within Derivative Works that You distribute, alongside\n          or as an addendum to the NOTICE text from the Work, provided\n          that such additional attribution notices cannot be construed\n          as modifying the License.\n\n      You may add Your own copyright statement to Your modifications and\n      may provide additional or different license terms and conditions\n      for use, reproduction, or distribution of Your modifications, or\n      for any such Derivative Works as a whole, provided Your use,\n      reproduction, and distribution of the Work otherwise complies with\n      the conditions stated in this License.\n\n   5. Submission of Contributions. Unless You explicitly state otherwise,\n      any Contribution intentionally submitted for inclusion in the Work\n      by You to the Licensor shall be under the terms and conditions of\n      this License, without any additional terms or conditions.\n      Notwithstanding the above, nothing herein shall supersede or modify\n      the terms of any separate license agreement you may have executed\n      with Licensor regarding such Contributions.\n\n   6. Trademarks. This License does not grant permission to use the trade\n      names, trademarks, service marks, or product names of the Licensor,\n      except as required for reasonable and customary use in describing the\n      origin of the Work and reproducing the content of the NOTICE file.\n\n   7. Disclaimer of Warranty. Unless required by applicable law or\n      agreed to in writing, Licensor provides the Work (and each\n      Contributor provides its Contributions) on an \"AS IS\" BASIS,\n      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or\n      implied, including, without limitation, any warranties or conditions\n      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A\n      PARTICULAR PURPOSE. You are solely responsible for determining the\n      appropriateness of using or redistributing the Work and assume any\n      risks associated with Your exercise of permissions under this License.\n\n   8. Limitation of Liability. In no event and under no legal theory,\n      whether in tort (including negligence), contract, or otherwise,\n      unless required by applicable law (such as deliberate and grossly\n      negligent acts) or agreed to in writing, shall any Contributor be\n      liable to You for damages, including any direct, indirect, special,\n      incidental, or consequential damages of any character arising as a\n      result of this License or out of the use or inability to use the\n      Work (including but not limited to damages for loss of goodwill,\n      work stoppage, computer failure or malfunction, or any and all\n      other commercial damages or losses), even if such Contributor\n      has been advised of the possibility of such damages.\n\n   9. Accepting Warranty or Additional Liability. While redistributing\n      the Work or Derivative Works thereof, You may choose to offer,\n      and charge a fee for, acceptance of support, warranty, indemnity,\n      or other liability obligations and/or rights consistent with this\n      License. However, in accepting such obligations, You may act only\n      on Your own behalf and on Your sole responsibility, not on behalf\n      of any other Contributor, and only if You agree to indemnify,\n      defend, and hold each Contributor harmless for any liability\n      incurred by, or claims asserted against, such Contributor by reason\n      of your accepting any such warranty or additional liability.\n\n   END OF TERMS AND CONDITIONS\n\n   APPENDIX: How to apply the Apache License to your work.\n\n      To apply the Apache License to your work, attach the following\n      boilerplate notice, with the fields enclosed by brackets \"[]\"\n      replaced with your own identifying information. (Don't include\n      the brackets!)  The text should be enclosed in the appropriate\n      comment syntax for the file format. We also recommend that a\n      file or class name and description of purpose be included on the\n      same \"printed page\" as the copyright notice for easier\n      identification within third-party archives.\n\n   Copyright [yyyy] [name of copyright owner]\n\n   Licensed under the Apache License, Version 2.0 (the \"License\");\n   you may not use this file except in compliance with the License.\n   You may obtain a copy of the License at\n\n       http://www.apache.org/licenses/LICENSE-2.0\n\n   Unless required by applicable law or agreed to in writing, software\n   distributed under the License is distributed on an \"AS IS\" BASIS,\n   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n   See the License for the specific language governing permissions and\n   limitations under the License.\n"
  },
  {
    "path": "pingora-load-balancing/src/background.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! Implement [BackgroundService] for [LoadBalancer]\n\nuse std::time::{Duration, Instant};\n\nuse super::{BackendIter, BackendSelection, LoadBalancer};\nuse async_trait::async_trait;\nuse pingora_core::services::{background::BackgroundService, ServiceReadyNotifier};\n\nimpl<S: Send + Sync + BackendSelection + 'static> LoadBalancer<S>\nwhere\n    S::Iter: BackendIter,\n{\n    pub async fn run(\n        &self,\n        shutdown: pingora_core::server::ShutdownWatch,\n        mut ready_opt: Option<ServiceReadyNotifier>,\n    ) -> () {\n        // 136 years\n        const NEVER: Duration = Duration::from_secs(u32::MAX as u64);\n        let mut now = Instant::now();\n        // run update and health check once\n        let mut next_update = now;\n        let mut next_health_check = now;\n\n        loop {\n            if *shutdown.borrow() {\n                return;\n            }\n\n            if next_update <= now {\n                // TODO: log err\n                let _ = self.update().await;\n                next_update = now + self.update_frequency.unwrap_or(NEVER);\n            }\n\n            // After the first update, discovery and selection setup will be\n            // done, so we will notify dependents\n            if let Some(ready) = ready_opt.take() {\n                ServiceReadyNotifier::notify_ready(ready)\n            }\n\n            if next_health_check <= now {\n                self.backends\n                    .run_health_check(self.parallel_health_check)\n                    .await;\n                next_health_check = now + self.health_check_frequency.unwrap_or(NEVER);\n            }\n\n            if self.update_frequency.is_none() && self.health_check_frequency.is_none() {\n                return;\n            }\n            let to_wake = std::cmp::min(next_update, next_health_check);\n            tokio::time::sleep_until(to_wake.into()).await;\n            now = Instant::now();\n        }\n    }\n}\n\n/// Implement [BackgroundService] for [LoadBalancer]. For backward-compatibility\n/// reasons, we implement both the `start` and `start_with_ready_notifier`\n/// methods.\n#[async_trait]\nimpl<S: Send + Sync + BackendSelection + 'static> BackgroundService for LoadBalancer<S>\nwhere\n    S::Iter: BackendIter,\n{\n    async fn start_with_ready_notifier(\n        &self,\n        shutdown: pingora_core::server::ShutdownWatch,\n        ready: ServiceReadyNotifier,\n    ) -> () {\n        self.run(shutdown, Some(ready)).await\n    }\n\n    async fn start(&self, shutdown: pingora_core::server::ShutdownWatch) -> () {\n        self.run(shutdown, None).await\n    }\n}\n"
  },
  {
    "path": "pingora-load-balancing/src/discovery.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! Service discovery interface and implementations\n\nuse arc_swap::ArcSwap;\nuse async_trait::async_trait;\nuse http::Extensions;\nuse pingora_core::protocols::l4::socket::SocketAddr;\nuse pingora_error::Result;\nuse std::io::Result as IoResult;\nuse std::net::ToSocketAddrs;\nuse std::{\n    collections::{BTreeSet, HashMap},\n    sync::Arc,\n};\n\nuse crate::Backend;\n\n/// [ServiceDiscovery] is the interface to discover [Backend]s.\n#[async_trait]\npub trait ServiceDiscovery {\n    /// Return the discovered collection of backends.\n    /// And *optionally* whether these backends are enabled to serve or not in a `HashMap`. Any backend\n    /// that is not explicitly in the set is considered enabled.\n    async fn discover(&self) -> Result<(BTreeSet<Backend>, HashMap<u64, bool>)>;\n}\n\n// TODO: add DNS base discovery\n\n/// A static collection of [Backend]s for service discovery.\n#[derive(Default)]\npub struct Static {\n    backends: ArcSwap<BTreeSet<Backend>>,\n}\n\nimpl Static {\n    /// Create a new boxed [Static] service discovery with the given backends.\n    pub fn new(backends: BTreeSet<Backend>) -> Box<Self> {\n        Box::new(Static {\n            backends: ArcSwap::new(Arc::new(backends)),\n        })\n    }\n\n    /// Create a new boxed [Static] from a given iterator of items that implements [ToSocketAddrs].\n    pub fn try_from_iter<A, T: IntoIterator<Item = A>>(iter: T) -> IoResult<Box<Self>>\n    where\n        A: ToSocketAddrs,\n    {\n        let mut upstreams = BTreeSet::new();\n        for addrs in iter.into_iter() {\n            let addrs = addrs.to_socket_addrs()?.map(|addr| Backend {\n                addr: SocketAddr::Inet(addr),\n                weight: 1,\n                ext: Extensions::new(),\n            });\n            upstreams.extend(addrs);\n        }\n        Ok(Self::new(upstreams))\n    }\n\n    /// return the collection to backends\n    pub fn get(&self) -> BTreeSet<Backend> {\n        BTreeSet::clone(&self.backends.load())\n    }\n\n    // Concurrent set/add/remove might race with each other\n    // TODO: use a queue to avoid racing\n\n    // TODO: take an impl iter\n    #[allow(dead_code)]\n    pub(crate) fn set(&self, backends: BTreeSet<Backend>) {\n        self.backends.store(backends.into())\n    }\n\n    #[allow(dead_code)]\n    pub(crate) fn add(&self, backend: Backend) {\n        let mut new = self.get();\n        new.insert(backend);\n        self.set(new)\n    }\n\n    #[allow(dead_code)]\n    pub(crate) fn remove(&self, backend: &Backend) {\n        let mut new = self.get();\n        new.remove(backend);\n        self.set(new)\n    }\n}\n\n#[async_trait]\nimpl ServiceDiscovery for Static {\n    async fn discover(&self) -> Result<(BTreeSet<Backend>, HashMap<u64, bool>)> {\n        // no readiness\n        let health = HashMap::new();\n        Ok((self.get(), health))\n    }\n}\n"
  },
  {
    "path": "pingora-load-balancing/src/health_check.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! Health Check interface and methods.\n\nuse crate::Backend;\nuse arc_swap::ArcSwap;\nuse async_trait::async_trait;\nuse pingora_core::connectors::http::custom;\nuse pingora_core::connectors::{http::Connector as HttpConnector, TransportConnector};\nuse pingora_core::custom_session;\nuse pingora_core::protocols::http::custom::client::Session;\nuse pingora_core::upstreams::peer::{BasicPeer, HttpPeer, Peer};\nuse pingora_error::{Error, ErrorType::CustomCode, Result};\nuse pingora_http::{RequestHeader, ResponseHeader};\nuse std::sync::Arc;\nuse std::time::Duration;\n\n/// [HealthObserve] is an interface for observing health changes of backends,\n/// this is what's used for our health observation callback.\n#[async_trait]\npub trait HealthObserve {\n    /// Observes the health of a [Backend], can be used for monitoring purposes.\n    async fn observe(&self, target: &Backend, healthy: bool);\n}\n/// Provided to a [HealthCheck] to observe changes to [Backend] health.\npub type HealthObserveCallback = Box<dyn HealthObserve + Send + Sync>;\n\n/// Provided to a [HealthCheck] to fetch [Backend] summary for detailed logging.\npub type BackendSummary = Box<dyn Fn(&Backend) -> String + Send + Sync>;\n\n/// [HealthCheck] is the interface to implement health check for backends\n#[async_trait]\npub trait HealthCheck {\n    /// Check the given backend.\n    ///\n    /// `Ok(())`` if the check passes, otherwise the check fails.\n    async fn check(&self, target: &Backend) -> Result<()>;\n\n    /// Called when the health changes for a [Backend].\n    async fn health_status_change(&self, _target: &Backend, _healthy: bool) {}\n\n    /// Called when a detailed [Backend] summary is needed.\n    fn backend_summary(&self, target: &Backend) -> String {\n        format!(\"{target:?}\")\n    }\n\n    /// This function defines how many *consecutive* checks should flip the health of a backend.\n    ///\n    /// For example: with `success``: `true`: this function should return the\n    /// number of check need to flip from unhealthy to healthy.\n    fn health_threshold(&self, success: bool) -> usize;\n}\n\n/// TCP health check\n///\n/// This health check checks if a TCP (or TLS) connection can be established to a given backend.\npub struct TcpHealthCheck {\n    /// Number of successful checks to flip from unhealthy to healthy.\n    pub consecutive_success: usize,\n    /// Number of failed checks to flip from healthy to unhealthy.\n    pub consecutive_failure: usize,\n    /// How to connect to the backend.\n    ///\n    /// This field defines settings like the connect timeout and src IP to bind.\n    /// The SocketAddr of `peer_template` is just a placeholder which will be replaced by the\n    /// actual address of the backend when the health check runs.\n    ///\n    /// By default, this check will try to establish a TCP connection. When the `sni` field is\n    /// set, it will also try to establish a TLS connection on top of the TCP connection.\n    pub peer_template: BasicPeer,\n    connector: TransportConnector,\n    /// A callback that is invoked when the `healthy` status changes for a [Backend].\n    pub health_changed_callback: Option<HealthObserveCallback>,\n}\n\nimpl Default for TcpHealthCheck {\n    fn default() -> Self {\n        let mut peer_template = BasicPeer::new(\"0.0.0.0:1\");\n        peer_template.options.connection_timeout = Some(Duration::from_secs(1));\n        TcpHealthCheck {\n            consecutive_success: 1,\n            consecutive_failure: 1,\n            peer_template,\n            connector: TransportConnector::new(None),\n            health_changed_callback: None,\n        }\n    }\n}\n\nimpl TcpHealthCheck {\n    /// Create a new [TcpHealthCheck] with the following default values\n    /// * connect timeout: 1 second\n    /// * consecutive_success: 1\n    /// * consecutive_failure: 1\n    pub fn new() -> Box<Self> {\n        Box::<TcpHealthCheck>::default()\n    }\n\n    /// Create a new [TcpHealthCheck] that tries to establish a TLS connection.\n    ///\n    /// The default values are the same as [Self::new()].\n    pub fn new_tls(sni: &str) -> Box<Self> {\n        let mut new = Self::default();\n        new.peer_template.sni = sni.into();\n        Box::new(new)\n    }\n\n    /// Replace the internal tcp connector with the given [TransportConnector]\n    pub fn set_connector(&mut self, connector: TransportConnector) {\n        self.connector = connector;\n    }\n}\n\n#[async_trait]\nimpl HealthCheck for TcpHealthCheck {\n    fn health_threshold(&self, success: bool) -> usize {\n        if success {\n            self.consecutive_success\n        } else {\n            self.consecutive_failure\n        }\n    }\n\n    async fn check(&self, target: &Backend) -> Result<()> {\n        let mut peer = self.peer_template.clone();\n        peer._address = target.addr.clone();\n        self.connector.get_stream(&peer).await.map(|_| {})\n    }\n\n    async fn health_status_change(&self, target: &Backend, healthy: bool) {\n        if let Some(callback) = &self.health_changed_callback {\n            callback.observe(target, healthy).await;\n        }\n    }\n}\n\ntype Validator = Box<dyn Fn(&ResponseHeader) -> Result<()> + Send + Sync>;\n\n/// HTTP health check\n///\n/// This health check checks if it can receive the expected HTTP(s) response from the given backend.\npub struct HttpHealthCheck<C = ()>\nwhere\n    C: custom::Connector,\n{\n    /// Number of successful checks to flip from unhealthy to healthy.\n    pub consecutive_success: usize,\n    /// Number of failed checks to flip from healthy to unhealthy.\n    pub consecutive_failure: usize,\n    /// How to connect to the backend.\n    ///\n    /// This field defines settings like the connect timeout and src IP to bind.\n    /// The SocketAddr of `peer_template` is just a placeholder which will be replaced by the\n    /// actual address of the backend when the health check runs.\n    ///\n    /// Set the `scheme` field to use HTTPs.\n    pub peer_template: HttpPeer,\n    /// Whether the underlying TCP/TLS connection can be reused across checks.\n    ///\n    /// * `false` will make sure that every health check goes through TCP (and TLS) handshakes.\n    ///   Established connections sometimes hide the issue of firewalls and L4 LB.\n    /// * `true` will try to reuse connections across checks, this is the more efficient and fast way\n    ///   to perform health checks.\n    pub reuse_connection: bool,\n    /// The request header to send to the backend\n    pub req: RequestHeader,\n    connector: HttpConnector<C>,\n    /// Optional field to define how to validate the response from the server.\n    ///\n    /// If not set, any response with a `200 OK` is considered a successful check.\n    pub validator: Option<Validator>,\n    /// Sometimes the health check endpoint lives one a different port than the actual backend.\n    /// Setting this option allows the health check to perform on the given port of the backend IP.\n    pub port_override: Option<u16>,\n    /// A callback that is invoked when the `healthy` status changes for a [Backend].\n    pub health_changed_callback: Option<HealthObserveCallback>,\n    /// An optional callback for backend summary reporting.\n    pub backend_summary_callback: Option<BackendSummary>,\n}\n\nimpl HttpHealthCheck<()> {\n    /// Create a new [HttpHealthCheck] with the following default settings\n    /// * connect timeout: 1 second\n    /// * read timeout: 1 second\n    /// * req: a GET to the `/` of the given host name\n    /// * consecutive_success: 1\n    /// * consecutive_failure: 1\n    /// * reuse_connection: false\n    /// * validator: `None`, any 200 response is considered successful\n    pub fn new(host: &str, tls: bool) -> Self {\n        let mut req = RequestHeader::build(\"GET\", b\"/\", None).unwrap();\n        req.append_header(\"Host\", host).unwrap();\n        let sni = if tls { host.into() } else { String::new() };\n        let mut peer_template = HttpPeer::new(\"0.0.0.0:1\", tls, sni);\n        peer_template.options.connection_timeout = Some(Duration::from_secs(1));\n        peer_template.options.read_timeout = Some(Duration::from_secs(1));\n        HttpHealthCheck {\n            consecutive_success: 1,\n            consecutive_failure: 1,\n            peer_template,\n            connector: HttpConnector::new(None),\n            reuse_connection: false,\n            req,\n            validator: None,\n            port_override: None,\n            health_changed_callback: None,\n            backend_summary_callback: None,\n        }\n    }\n}\n\nimpl<C> HttpHealthCheck<C>\nwhere\n    C: custom::Connector,\n{\n    /// Create a new [HttpHealthCheck] with the following default settings\n    /// * connect timeout: 1 second\n    /// * read timeout: 1 second\n    /// * req: a GET to the `/` of the given host name\n    /// * consecutive_success: 1\n    /// * consecutive_failure: 1\n    /// * reuse_connection: false\n    /// * validator: `None`, any 200 response is considered successful\n    pub fn new_custom(host: &str, tls: bool, custom: HttpConnector<C>) -> Self {\n        let mut req = RequestHeader::build(\"GET\", b\"/\", None).unwrap();\n        req.append_header(\"Host\", host).unwrap();\n        let sni = if tls { host.into() } else { String::new() };\n        let mut peer_template = HttpPeer::new(\"0.0.0.0:1\", tls, sni);\n        peer_template.options.connection_timeout = Some(Duration::from_secs(1));\n        peer_template.options.read_timeout = Some(Duration::from_secs(1));\n        HttpHealthCheck {\n            consecutive_success: 1,\n            consecutive_failure: 1,\n            peer_template,\n            connector: custom,\n            reuse_connection: false,\n            req,\n            validator: None,\n            port_override: None,\n            health_changed_callback: None,\n            backend_summary_callback: None,\n        }\n    }\n\n    /// Replace the internal http connector with the given [HttpConnector]\n    pub fn set_connector(&mut self, connector: HttpConnector<C>) {\n        self.connector = connector;\n    }\n\n    pub fn set_backend_summary<F>(&mut self, callback: F)\n    where\n        F: Fn(&Backend) -> String + Send + Sync + 'static,\n    {\n        self.backend_summary_callback = Some(Box::new(callback));\n    }\n}\n\n#[async_trait]\nimpl<C> HealthCheck for HttpHealthCheck<C>\nwhere\n    C: custom::Connector,\n{\n    fn health_threshold(&self, success: bool) -> usize {\n        if success {\n            self.consecutive_success\n        } else {\n            self.consecutive_failure\n        }\n    }\n\n    async fn check(&self, target: &Backend) -> Result<()> {\n        let mut peer = self.peer_template.clone();\n        peer._address = target.addr.clone();\n        if let Some(port) = self.port_override {\n            peer._address.set_port(port);\n        }\n        let session = self.connector.get_http_session(&peer).await?;\n\n        let mut session = session.0;\n        let req = Box::new(self.req.clone());\n        session.write_request_header(req).await?;\n        session.finish_request_body().await?;\n\n        custom_session!(session.finish_custom().await?);\n\n        if let Some(read_timeout) = peer.options.read_timeout {\n            session.set_read_timeout(Some(read_timeout));\n        }\n\n        session.read_response_header().await?;\n\n        let resp = session.response_header().expect(\"just read\");\n\n        if let Some(validator) = self.validator.as_ref() {\n            validator(resp)?;\n        } else if resp.status != 200 {\n            return Error::e_explain(\n                CustomCode(\"non 200 code\", resp.status.as_u16()),\n                \"during http healthcheck\",\n            );\n        };\n\n        while session.read_response_body().await?.is_some() {\n            // drain the body if any\n        }\n\n        // TODO(slava): do it concurrently wtih body drain?\n        custom_session!(session.drain_custom_messages().await?);\n\n        if self.reuse_connection {\n            let idle_timeout = peer.idle_timeout();\n            self.connector\n                .release_http_session(session, &peer, idle_timeout)\n                .await;\n        }\n\n        Ok(())\n    }\n    async fn health_status_change(&self, target: &Backend, healthy: bool) {\n        if let Some(callback) = &self.health_changed_callback {\n            callback.observe(target, healthy).await;\n        }\n    }\n    fn backend_summary(&self, target: &Backend) -> String {\n        if let Some(callback) = &self.backend_summary_callback {\n            callback(target)\n        } else {\n            format!(\"{target:?}\")\n        }\n    }\n}\n\n#[derive(Clone)]\nstruct HealthInner {\n    /// Whether the endpoint is healthy to serve traffic\n    healthy: bool,\n    /// Whether the endpoint is allowed to serve traffic independent of its health\n    enabled: bool,\n    /// The counter for stateful transition between healthy and unhealthy.\n    /// When [healthy] is true, this counts the number of consecutive health check failures\n    /// so that the caller can flip the healthy when a certain threshold is met, and vise versa.\n    consecutive_counter: usize,\n}\n\n/// Health of backends that can be updated atomically\npub(crate) struct Health(ArcSwap<HealthInner>);\n\nimpl Default for Health {\n    fn default() -> Self {\n        Health(ArcSwap::new(Arc::new(HealthInner {\n            healthy: true, // TODO: allow to start with unhealthy\n            enabled: true,\n            consecutive_counter: 0,\n        })))\n    }\n}\n\nimpl Clone for Health {\n    fn clone(&self) -> Self {\n        let inner = self.0.load_full();\n        Health(ArcSwap::new(inner))\n    }\n}\n\nimpl Health {\n    pub fn ready(&self) -> bool {\n        let h = self.0.load();\n        h.healthy && h.enabled\n    }\n\n    pub fn enable(&self, enabled: bool) {\n        let h = self.0.load();\n        if h.enabled != enabled {\n            // clone the inner\n            let mut new_health = (**h).clone();\n            new_health.enabled = enabled;\n            self.0.store(Arc::new(new_health));\n        };\n    }\n\n    // return true when the health is flipped\n    pub fn observe_health(&self, health: bool, flip_threshold: usize) -> bool {\n        let h = self.0.load();\n        let mut flipped = false;\n        if h.healthy != health {\n            // opposite health observed, ready to increase the counter\n            // clone the inner\n            let mut new_health = (**h).clone();\n            new_health.consecutive_counter += 1;\n            if new_health.consecutive_counter >= flip_threshold {\n                new_health.healthy = health;\n                new_health.consecutive_counter = 0;\n                flipped = true;\n            }\n            self.0.store(Arc::new(new_health));\n        } else if h.consecutive_counter > 0 {\n            // observing the same health as the current state.\n            // reset the counter, if it is non-zero, because it is no longer consecutive\n            let mut new_health = (**h).clone();\n            new_health.consecutive_counter = 0;\n            self.0.store(Arc::new(new_health));\n        }\n        flipped\n    }\n}\n\n#[cfg(test)]\nmod test {\n    use std::{\n        collections::{BTreeSet, HashMap},\n        sync::atomic::{AtomicU16, Ordering},\n    };\n\n    use super::*;\n    use crate::{discovery, Backends, SocketAddr};\n    use async_trait::async_trait;\n    use http::Extensions;\n\n    #[tokio::test]\n    async fn test_tcp_check() {\n        let tcp_check = TcpHealthCheck::default();\n\n        let backend = Backend {\n            addr: SocketAddr::Inet(\"1.1.1.1:80\".parse().unwrap()),\n            weight: 1,\n            ext: Extensions::new(),\n        };\n\n        assert!(tcp_check.check(&backend).await.is_ok());\n\n        let backend = Backend {\n            addr: SocketAddr::Inet(\"1.1.1.1:79\".parse().unwrap()),\n            weight: 1,\n            ext: Extensions::new(),\n        };\n\n        assert!(tcp_check.check(&backend).await.is_err());\n    }\n\n    #[cfg(feature = \"any_tls\")]\n    #[tokio::test]\n    async fn test_tls_check() {\n        let tls_check = TcpHealthCheck::new_tls(\"one.one.one.one\");\n        let backend = Backend {\n            addr: SocketAddr::Inet(\"1.1.1.1:443\".parse().unwrap()),\n            weight: 1,\n            ext: Extensions::new(),\n        };\n\n        assert!(tls_check.check(&backend).await.is_ok());\n    }\n\n    #[cfg(feature = \"any_tls\")]\n    #[tokio::test]\n    async fn test_https_check() {\n        let https_check = HttpHealthCheck::new(\"one.one.one.one\", true);\n\n        let backend = Backend {\n            addr: SocketAddr::Inet(\"1.1.1.1:443\".parse().unwrap()),\n            weight: 1,\n            ext: Extensions::new(),\n        };\n\n        assert!(https_check.check(&backend).await.is_ok());\n    }\n\n    #[tokio::test]\n    async fn test_http_custom_check() {\n        let mut http_check = HttpHealthCheck::new(\"one.one.one.one\", false);\n        http_check.validator = Some(Box::new(|resp: &ResponseHeader| {\n            if resp.status == 301 {\n                Ok(())\n            } else {\n                Error::e_explain(\n                    CustomCode(\"non 301 code\", resp.status.as_u16()),\n                    \"during http healthcheck\",\n                )\n            }\n        }));\n\n        let backend = Backend {\n            addr: SocketAddr::Inet(\"1.1.1.1:80\".parse().unwrap()),\n            weight: 1,\n            ext: Extensions::new(),\n        };\n\n        http_check.check(&backend).await.unwrap();\n\n        assert!(http_check.check(&backend).await.is_ok());\n    }\n\n    #[tokio::test]\n    async fn test_health_observe() {\n        struct Observe {\n            unhealthy_count: Arc<AtomicU16>,\n        }\n        #[async_trait]\n        impl HealthObserve for Observe {\n            async fn observe(&self, _target: &Backend, healthy: bool) {\n                if !healthy {\n                    self.unhealthy_count.fetch_add(1, Ordering::Relaxed);\n                }\n            }\n        }\n\n        let good_backend = Backend::new(\"127.0.0.1:79\").unwrap();\n        let new_good_backends = || -> (BTreeSet<Backend>, HashMap<u64, bool>) {\n            let mut healthy = HashMap::new();\n            healthy.insert(good_backend.hash_key(), true);\n            let mut backends = BTreeSet::new();\n            backends.extend(vec![good_backend.clone()]);\n            (backends, healthy)\n        };\n        // tcp health check\n        {\n            let unhealthy_count = Arc::new(AtomicU16::new(0));\n            let ob = Observe {\n                unhealthy_count: unhealthy_count.clone(),\n            };\n            let bob = Box::new(ob);\n            let tcp_check = TcpHealthCheck {\n                health_changed_callback: Some(bob),\n                ..Default::default()\n            };\n\n            let discovery = discovery::Static::default();\n            let mut backends = Backends::new(Box::new(discovery));\n            backends.set_health_check(Box::new(tcp_check));\n            let result = new_good_backends();\n            backends.do_update(result.0, result.1, |_backend: Arc<BTreeSet<Backend>>| {});\n            // the backend is ready\n            assert!(backends.ready(&good_backend));\n\n            // run health check\n            backends.run_health_check(false).await;\n            assert!(1 == unhealthy_count.load(Ordering::Relaxed));\n            // backend is unhealthy\n            assert!(!backends.ready(&good_backend));\n        }\n\n        // http health check\n        {\n            let unhealthy_count = Arc::new(AtomicU16::new(0));\n            let ob = Observe {\n                unhealthy_count: unhealthy_count.clone(),\n            };\n            let bob = Box::new(ob);\n\n            let mut https_check = HttpHealthCheck::new(\"one.one.one.one\", true);\n            https_check.health_changed_callback = Some(bob);\n\n            let discovery = discovery::Static::default();\n            let mut backends = Backends::new(Box::new(discovery));\n            backends.set_health_check(Box::new(https_check));\n            let result = new_good_backends();\n            backends.do_update(result.0, result.1, |_backend: Arc<BTreeSet<Backend>>| {});\n            // the backend is ready\n            assert!(backends.ready(&good_backend));\n            // run health check\n            backends.run_health_check(false).await;\n            assert!(1 == unhealthy_count.load(Ordering::Relaxed));\n            assert!(!backends.ready(&good_backend));\n        }\n    }\n}\n"
  },
  {
    "path": "pingora-load-balancing/src/lib.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! # Pingora Load Balancing utilities\n//! This crate provides common service discovery, health check and load balancing\n//! algorithms for proxies to use.\n\n// https://github.com/mcarton/rust-derivative/issues/112\n// False positive for macro generated code\n#![allow(clippy::non_canonical_partial_ord_impl)]\n\nuse arc_swap::ArcSwap;\nuse derivative::Derivative;\nuse futures::FutureExt;\npub use http::Extensions;\nuse pingora_core::protocols::l4::socket::SocketAddr;\nuse pingora_error::{ErrorType, OrErr, Result};\nuse std::collections::hash_map::DefaultHasher;\nuse std::collections::{BTreeSet, HashMap};\nuse std::hash::{Hash, Hasher};\nuse std::io::Result as IoResult;\nuse std::net::ToSocketAddrs;\nuse std::sync::Arc;\nuse std::time::Duration;\n\nmod background;\npub mod discovery;\npub mod health_check;\npub mod selection;\n\nuse discovery::ServiceDiscovery;\nuse health_check::Health;\nuse selection::UniqueIterator;\nuse selection::{BackendIter, BackendSelection};\n\npub mod prelude {\n    pub use crate::health_check::TcpHealthCheck;\n    pub use crate::selection::RoundRobin;\n    pub use crate::LoadBalancer;\n}\n\n/// [Backend] represents a server to proxy or connect to.\n#[derive(Derivative)]\n#[derivative(Clone, Hash, PartialEq, PartialOrd, Eq, Ord, Debug)]\npub struct Backend {\n    /// The address to the backend server.\n    pub addr: SocketAddr,\n    /// The relative weight of the server. Load balancing algorithms will\n    /// proportionally distributed traffic according to this value.\n    pub weight: usize,\n\n    /// The extension field to put arbitrary data to annotate the Backend.\n    /// The data added here is opaque to this crate hence the data is ignored by\n    /// functionalities of this crate. For example, two backends with the same\n    /// [SocketAddr] and the same weight but different `ext` data are considered\n    /// identical.\n    /// See [Extensions] for how to add and read the data.\n    #[derivative(PartialEq = \"ignore\")]\n    #[derivative(PartialOrd = \"ignore\")]\n    #[derivative(Hash = \"ignore\")]\n    #[derivative(Ord = \"ignore\")]\n    pub ext: Extensions,\n}\n\nimpl Backend {\n    /// Create a new [Backend] with `weight` 1. The function will try to parse\n    ///  `addr` into a [std::net::SocketAddr].\n    pub fn new(addr: &str) -> Result<Self> {\n        Self::new_with_weight(addr, 1)\n    }\n\n    /// Creates a new [Backend] with the specified `weight`. The function will try to parse\n    /// `addr` into a [std::net::SocketAddr].\n    pub fn new_with_weight(addr: &str, weight: usize) -> Result<Self> {\n        let addr = addr\n            .parse()\n            .or_err(ErrorType::InternalError, \"invalid socket addr\")?;\n        Ok(Backend {\n            addr: SocketAddr::Inet(addr),\n            weight,\n            ext: Extensions::new(),\n        })\n        // TODO: UDS\n    }\n\n    pub(crate) fn hash_key(&self) -> u64 {\n        let mut hasher = DefaultHasher::new();\n        self.hash(&mut hasher);\n        hasher.finish()\n    }\n}\n\nimpl std::ops::Deref for Backend {\n    type Target = SocketAddr;\n\n    fn deref(&self) -> &Self::Target {\n        &self.addr\n    }\n}\n\nimpl std::ops::DerefMut for Backend {\n    fn deref_mut(&mut self) -> &mut Self::Target {\n        &mut self.addr\n    }\n}\n\nimpl std::net::ToSocketAddrs for Backend {\n    type Iter = std::iter::Once<std::net::SocketAddr>;\n\n    fn to_socket_addrs(&self) -> std::io::Result<Self::Iter> {\n        self.addr.to_socket_addrs()\n    }\n}\n\n/// [Backends] is a collection of [Backend]s.\n///\n/// It includes a service discovery method (static or dynamic) to discover all\n/// the available backends as well as an optional health check method to probe the liveness\n/// of each backend.\npub struct Backends {\n    discovery: Box<dyn ServiceDiscovery + Send + Sync + 'static>,\n    health_check: Option<Arc<dyn health_check::HealthCheck + Send + Sync + 'static>>,\n    backends: ArcSwap<BTreeSet<Backend>>,\n    health: ArcSwap<HashMap<u64, Health>>,\n}\n\nimpl Backends {\n    /// Create a new [Backends] with the given [ServiceDiscovery] implementation.\n    ///\n    /// The health check method is by default empty.\n    pub fn new(discovery: Box<dyn ServiceDiscovery + Send + Sync + 'static>) -> Self {\n        Self {\n            discovery,\n            health_check: None,\n            backends: Default::default(),\n            health: Default::default(),\n        }\n    }\n\n    /// Set the health check method. See [health_check] for the methods provided.\n    pub fn set_health_check(\n        &mut self,\n        hc: Box<dyn health_check::HealthCheck + Send + Sync + 'static>,\n    ) {\n        self.health_check = Some(hc.into())\n    }\n\n    /// Updates backends when the new is different from the current set,\n    /// the callback will be invoked when the new set of backend is different\n    /// from the current one so that the caller can update the selector accordingly.\n    fn do_update<F>(\n        &self,\n        new_backends: BTreeSet<Backend>,\n        enablement: HashMap<u64, bool>,\n        callback: F,\n    ) where\n        F: Fn(Arc<BTreeSet<Backend>>),\n    {\n        if (**self.backends.load()) != new_backends {\n            let old_health = self.health.load();\n            let mut health = HashMap::with_capacity(new_backends.len());\n            for backend in new_backends.iter() {\n                let hash_key = backend.hash_key();\n                // use the default health if the backend is new\n                let backend_health = old_health.get(&hash_key).cloned().unwrap_or_default();\n\n                // override enablement\n                if let Some(backend_enabled) = enablement.get(&hash_key) {\n                    backend_health.enable(*backend_enabled);\n                }\n                health.insert(hash_key, backend_health);\n            }\n\n            // TODO: put this all under 1 ArcSwap so the update is atomic\n            // It's important the `callback()` executes first since computing selector backends might\n            // be expensive. For example, if a caller checks `backends` to see if any are available\n            // they may encounter false positives if the selector isn't ready yet.\n            let new_backends = Arc::new(new_backends);\n            callback(new_backends.clone());\n            self.backends.store(new_backends);\n            self.health.store(Arc::new(health));\n        } else {\n            // no backend change, just check enablement\n            for (hash_key, backend_enabled) in enablement.iter() {\n                // override enablement if set\n                // this get should always be Some(_) because we already populate `health`` for all known backends\n                if let Some(backend_health) = self.health.load().get(hash_key) {\n                    backend_health.enable(*backend_enabled);\n                }\n            }\n        }\n    }\n\n    /// Whether a certain [Backend] is ready to serve traffic.\n    ///\n    /// This function returns true when the backend is both healthy and enabled.\n    /// This function returns true when the health check is unset but the backend is enabled.\n    /// When the health check is set, this function will return false for the `backend` it\n    /// doesn't know.\n    pub fn ready(&self, backend: &Backend) -> bool {\n        self.health\n            .load()\n            .get(&backend.hash_key())\n            // Racing: return `None` when this function is called between the\n            // backend store and the health store\n            .map_or(self.health_check.is_none(), |h| h.ready())\n    }\n\n    /// Manually set if a [Backend] is ready to serve traffic.\n    ///\n    /// This method does not override the health of the backend. It is meant to be used\n    /// to stop a backend from accepting traffic when it is still healthy.\n    ///\n    /// This method is noop when the given backend doesn't exist in the service discovery.\n    pub fn set_enable(&self, backend: &Backend, enabled: bool) {\n        // this should always be Some(_) because health is always populated during update\n        if let Some(h) = self.health.load().get(&backend.hash_key()) {\n            h.enable(enabled)\n        };\n    }\n\n    /// Return the collection of the backends.\n    pub fn get_backend(&self) -> Arc<BTreeSet<Backend>> {\n        self.backends.load_full()\n    }\n\n    /// Call the service discovery method to update the collection of backends.\n    ///\n    /// The callback will be invoked when the new set of backend is different\n    /// from the current one so that the caller can update the selector accordingly.\n    pub async fn update<F>(&self, callback: F) -> Result<()>\n    where\n        F: Fn(Arc<BTreeSet<Backend>>),\n    {\n        let (new_backends, enablement) = self.discovery.discover().await?;\n        self.do_update(new_backends, enablement, callback);\n        Ok(())\n    }\n\n    /// Run health check on all backends if it is set.\n    ///\n    /// When `parallel: true`, all backends are checked in parallel instead of sequentially\n    pub async fn run_health_check(&self, parallel: bool) {\n        use crate::health_check::HealthCheck;\n        use log::{info, warn};\n        use pingora_runtime::current_handle;\n\n        async fn check_and_report(\n            backend: &Backend,\n            check: &Arc<dyn HealthCheck + Send + Sync>,\n            health_table: &HashMap<u64, Health>,\n        ) {\n            let errored = check.check(backend).await.err();\n            if let Some(h) = health_table.get(&backend.hash_key()) {\n                let flipped =\n                    h.observe_health(errored.is_none(), check.health_threshold(errored.is_none()));\n                if flipped {\n                    check.health_status_change(backend, errored.is_none()).await;\n                    let summary = check.backend_summary(backend);\n                    if let Some(e) = errored {\n                        warn!(\"{summary} becomes unhealthy, {e}\");\n                    } else {\n                        info!(\"{summary} becomes healthy\");\n                    }\n                }\n            }\n        }\n\n        let Some(health_check) = self.health_check.as_ref() else {\n            return;\n        };\n\n        let backends = self.backends.load();\n        if parallel {\n            let health_table = self.health.load_full();\n            let runtime = current_handle();\n            let jobs = backends.iter().map(|backend| {\n                let backend = backend.clone();\n                let check = health_check.clone();\n                let ht = health_table.clone();\n                runtime.spawn(async move {\n                    check_and_report(&backend, &check, &ht).await;\n                })\n            });\n\n            futures::future::join_all(jobs).await;\n        } else {\n            for backend in backends.iter() {\n                check_and_report(backend, health_check, &self.health.load()).await;\n            }\n        }\n    }\n}\n\n/// A [LoadBalancer] instance contains the service discovery, health check and backend selection\n/// all together.\n///\n/// In order to run service discovery and health check at the designated frequencies, the [LoadBalancer]\n/// needs to be run as a [pingora_core::services::background::BackgroundService].\npub struct LoadBalancer<S>\nwhere\n    S: BackendSelection,\n{\n    backends: Backends,\n    selector: ArcSwap<S>,\n\n    config: Option<S::Config>,\n\n    /// How frequent the health check logic (if set) should run.\n    ///\n    /// If `None`, the health check logic will only run once at the beginning.\n    pub health_check_frequency: Option<Duration>,\n    /// How frequent the service discovery should run.\n    ///\n    /// If `None`, the service discovery will only run once at the beginning.\n    pub update_frequency: Option<Duration>,\n    /// Whether to run health check to all backends in parallel. Default is false.\n    pub parallel_health_check: bool,\n}\n\nimpl<S> LoadBalancer<S>\nwhere\n    S: BackendSelection + 'static,\n    S::Iter: BackendIter,\n{\n    /// Build a [LoadBalancer] with static backends created from the iter.\n    ///\n    /// Note: [ToSocketAddrs] will invoke blocking network IO for DNS lookup if\n    /// the input cannot be directly parsed as [SocketAddr].\n    pub fn try_from_iter<A, T: IntoIterator<Item = A>>(iter: T) -> IoResult<Self>\n    where\n        A: ToSocketAddrs,\n    {\n        let discovery = discovery::Static::try_from_iter(iter)?;\n        let backends = Backends::new(discovery);\n        let lb = Self::from_backends(backends);\n        lb.update()\n            .now_or_never()\n            .expect(\"static should not block\")\n            .expect(\"static should not error\");\n        Ok(lb)\n    }\n\n    /// Build a [LoadBalancer] with the given [Backends] and the config.\n    pub fn from_backends_with_config(backends: Backends, config_opt: Option<S::Config>) -> Self {\n        let selector_raw = if let Some(config) = config_opt.as_ref() {\n            S::build_with_config(&backends.get_backend(), config)\n        } else {\n            S::build(&backends.get_backend())\n        };\n\n        let selector = ArcSwap::new(Arc::new(selector_raw));\n\n        LoadBalancer {\n            backends,\n            selector,\n            config: config_opt,\n            health_check_frequency: None,\n            update_frequency: None,\n            parallel_health_check: false,\n        }\n    }\n\n    /// Build a [LoadBalancer] with the given [Backends].\n    pub fn from_backends(backends: Backends) -> Self {\n        Self::from_backends_with_config(backends, None)\n    }\n\n    /// Run the service discovery and update the selection algorithm.\n    ///\n    /// This function will be called every `update_frequency` if this [LoadBalancer] instance\n    /// is running as a background service.\n    pub async fn update(&self) -> Result<()> {\n        self.backends\n            .update(|backends| {\n                let selector = if let Some(config) = &self.config {\n                    S::build_with_config(&backends, config)\n                } else {\n                    S::build(&backends)\n                };\n\n                self.selector.store(Arc::new(selector))\n            })\n            .await\n    }\n\n    /// Return the first healthy [Backend] according to the selection algorithm and the\n    /// health check results.\n    ///\n    /// The `key` is used for hash based selection and is ignored if the selection is random or\n    /// round robin.\n    ///\n    /// the `max_iterations` is there to bound the search time for the next Backend. In certain\n    /// algorithm like Ketama hashing, the search for the next backend is linear and could take\n    /// a lot steps.\n    // TODO: consider remove `max_iterations` as users have no idea how to set it.\n    pub fn select(&self, key: &[u8], max_iterations: usize) -> Option<Backend> {\n        self.select_with(key, max_iterations, |_, health| health)\n    }\n\n    /// Similar to [Self::select], return the first healthy [Backend] according to the selection algorithm\n    /// and the user defined `accept` function.\n    ///\n    /// The `accept` function takes two inputs, the backend being selected and the internal health of that\n    /// backend. The function can do things like ignoring the internal health checks or skipping this backend\n    /// because it failed before. The `accept` function is called multiple times iterating over backends\n    /// until it returns `true`.\n    pub fn select_with<F>(&self, key: &[u8], max_iterations: usize, accept: F) -> Option<Backend>\n    where\n        F: Fn(&Backend, bool) -> bool,\n    {\n        let selection = self.selector.load();\n        let mut iter = UniqueIterator::new(selection.iter(key), max_iterations);\n        while let Some(b) = iter.get_next() {\n            if accept(&b, self.backends.ready(&b)) {\n                return Some(b);\n            }\n        }\n        None\n    }\n\n    /// Set the health check method. See [health_check].\n    pub fn set_health_check(\n        &mut self,\n        hc: Box<dyn health_check::HealthCheck + Send + Sync + 'static>,\n    ) {\n        self.backends.set_health_check(hc);\n    }\n\n    /// Access the [Backends] of this [LoadBalancer]\n    pub fn backends(&self) -> &Backends {\n        &self.backends\n    }\n}\n\n#[cfg(test)]\nmod test {\n    use std::sync::atomic::{AtomicBool, Ordering::Relaxed};\n\n    use super::*;\n    use async_trait::async_trait;\n\n    #[tokio::test]\n    async fn test_static_backends() {\n        let backends: LoadBalancer<selection::RoundRobin> =\n            LoadBalancer::try_from_iter([\"1.1.1.1:80\", \"1.0.0.1:80\"]).unwrap();\n\n        let backend1 = Backend::new(\"1.1.1.1:80\").unwrap();\n        let backend2 = Backend::new(\"1.0.0.1:80\").unwrap();\n        let backend = backends.backends().get_backend();\n        assert!(backend.contains(&backend1));\n        assert!(backend.contains(&backend2));\n    }\n\n    #[tokio::test]\n    async fn test_backends() {\n        let discovery = discovery::Static::default();\n        let good1 = Backend::new(\"1.1.1.1:80\").unwrap();\n        discovery.add(good1.clone());\n        let good2 = Backend::new(\"1.0.0.1:80\").unwrap();\n        discovery.add(good2.clone());\n        let bad = Backend::new(\"127.0.0.1:79\").unwrap();\n        discovery.add(bad.clone());\n\n        let mut backends = Backends::new(Box::new(discovery));\n        let check = health_check::TcpHealthCheck::new();\n        backends.set_health_check(check);\n\n        // true: new backend discovered\n        let updated = AtomicBool::new(false);\n        backends\n            .update(|_| updated.store(true, Relaxed))\n            .await\n            .unwrap();\n        assert!(updated.load(Relaxed));\n\n        // false: no new backend discovered\n        let updated = AtomicBool::new(false);\n        backends\n            .update(|_| updated.store(true, Relaxed))\n            .await\n            .unwrap();\n        assert!(!updated.load(Relaxed));\n\n        backends.run_health_check(false).await;\n\n        let backend = backends.get_backend();\n        assert!(backend.contains(&good1));\n        assert!(backend.contains(&good2));\n        assert!(backend.contains(&bad));\n\n        assert!(backends.ready(&good1));\n        assert!(backends.ready(&good2));\n        assert!(!backends.ready(&bad));\n    }\n    #[tokio::test]\n    async fn test_backends_with_ext() {\n        let discovery = discovery::Static::default();\n        let mut b1 = Backend::new(\"1.1.1.1:80\").unwrap();\n        b1.ext.insert(true);\n        let mut b2 = Backend::new(\"1.0.0.1:80\").unwrap();\n        b2.ext.insert(1u8);\n        discovery.add(b1.clone());\n        discovery.add(b2.clone());\n\n        let backends = Backends::new(Box::new(discovery));\n\n        // fill in the backends\n        backends.update(|_| {}).await.unwrap();\n\n        let backend = backends.get_backend();\n        assert!(backend.contains(&b1));\n        assert!(backend.contains(&b2));\n\n        let b2 = backend.first().unwrap();\n        assert_eq!(b2.ext.get::<u8>(), Some(&1));\n\n        let b1 = backend.last().unwrap();\n        assert_eq!(b1.ext.get::<bool>(), Some(&true));\n    }\n\n    #[tokio::test]\n    async fn test_discovery_readiness() {\n        use discovery::Static;\n\n        struct TestDiscovery(Static);\n        #[async_trait]\n        impl ServiceDiscovery for TestDiscovery {\n            async fn discover(&self) -> Result<(BTreeSet<Backend>, HashMap<u64, bool>)> {\n                let bad = Backend::new(\"127.0.0.1:79\").unwrap();\n                let (backends, mut readiness) = self.0.discover().await?;\n                readiness.insert(bad.hash_key(), false);\n                Ok((backends, readiness))\n            }\n        }\n        let discovery = Static::default();\n        let good1 = Backend::new(\"1.1.1.1:80\").unwrap();\n        discovery.add(good1.clone());\n        let good2 = Backend::new(\"1.0.0.1:80\").unwrap();\n        discovery.add(good2.clone());\n        let bad = Backend::new(\"127.0.0.1:79\").unwrap();\n        discovery.add(bad.clone());\n        let discovery = TestDiscovery(discovery);\n\n        let backends = Backends::new(Box::new(discovery));\n\n        // true: new backend discovered\n        let updated = AtomicBool::new(false);\n        backends\n            .update(|_| updated.store(true, Relaxed))\n            .await\n            .unwrap();\n        assert!(updated.load(Relaxed));\n\n        let backend = backends.get_backend();\n        assert!(backend.contains(&good1));\n        assert!(backend.contains(&good2));\n        assert!(backend.contains(&bad));\n\n        assert!(backends.ready(&good1));\n        assert!(backends.ready(&good2));\n        assert!(!backends.ready(&bad));\n    }\n\n    #[tokio::test]\n    async fn test_parallel_health_check() {\n        let discovery = discovery::Static::default();\n        let good1 = Backend::new(\"1.1.1.1:80\").unwrap();\n        discovery.add(good1.clone());\n        let good2 = Backend::new(\"1.0.0.1:80\").unwrap();\n        discovery.add(good2.clone());\n        let bad = Backend::new(\"127.0.0.1:79\").unwrap();\n        discovery.add(bad.clone());\n\n        let mut backends = Backends::new(Box::new(discovery));\n        let check = health_check::TcpHealthCheck::new();\n        backends.set_health_check(check);\n\n        // true: new backend discovered\n        let updated = AtomicBool::new(false);\n        backends\n            .update(|_| updated.store(true, Relaxed))\n            .await\n            .unwrap();\n        assert!(updated.load(Relaxed));\n\n        backends.run_health_check(true).await;\n\n        assert!(backends.ready(&good1));\n        assert!(backends.ready(&good2));\n        assert!(!backends.ready(&bad));\n    }\n\n    mod thread_safety {\n        use super::*;\n\n        struct MockDiscovery {\n            expected: usize,\n        }\n        #[async_trait]\n        impl ServiceDiscovery for MockDiscovery {\n            async fn discover(&self) -> Result<(BTreeSet<Backend>, HashMap<u64, bool>)> {\n                let mut d = BTreeSet::new();\n                let mut m = HashMap::with_capacity(self.expected);\n                for i in 0..self.expected {\n                    let b = Backend::new(&format!(\"1.1.1.1:{i}\")).unwrap();\n                    m.insert(i as u64, true);\n                    d.insert(b);\n                }\n                Ok((d, m))\n            }\n        }\n\n        #[tokio::test(flavor = \"multi_thread\", worker_threads = 2)]\n        async fn test_consistency() {\n            let expected = 3000;\n            let discovery = MockDiscovery { expected };\n            let lb = Arc::new(LoadBalancer::<selection::Consistent>::from_backends(\n                Backends::new(Box::new(discovery)),\n            ));\n            let lb2 = lb.clone();\n\n            tokio::spawn(async move {\n                assert!(lb2.update().await.is_ok());\n            });\n            let mut backend_count = 0;\n            while backend_count == 0 {\n                let backends = lb.backends();\n                backend_count = backends.backends.load_full().len();\n            }\n            assert_eq!(backend_count, expected);\n            assert!(lb.select_with(b\"test\", 1, |_, _| true).is_some());\n        }\n    }\n}\n"
  },
  {
    "path": "pingora-load-balancing/src/selection/algorithms.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! Implementation of algorithms for weighted selection\n//!\n//! All [std::hash::Hasher] + [Default] can be used directly as a selection algorithm.\n\nuse super::*;\nuse std::hash::Hasher;\nuse std::sync::atomic::{AtomicUsize, Ordering};\n\nimpl<H> SelectionAlgorithm for H\nwhere\n    H: Default + Hasher,\n{\n    fn new() -> Self {\n        H::default()\n    }\n    fn next(&self, key: &[u8]) -> u64 {\n        let mut hasher = H::default();\n        hasher.write(key);\n        hasher.finish()\n    }\n}\n\n/// Round Robin selection\npub struct RoundRobin(AtomicUsize);\n\nimpl SelectionAlgorithm for RoundRobin {\n    fn new() -> Self {\n        Self(AtomicUsize::new(0))\n    }\n    fn next(&self, _key: &[u8]) -> u64 {\n        self.0.fetch_add(1, Ordering::Relaxed) as u64\n    }\n}\n\n/// Random selection\npub struct Random;\n\nimpl SelectionAlgorithm for Random {\n    fn new() -> Self {\n        Self\n    }\n    fn next(&self, _key: &[u8]) -> u64 {\n        use rand::Rng;\n        let mut rng = rand::thread_rng();\n        rng.gen()\n    }\n}\n"
  },
  {
    "path": "pingora-load-balancing/src/selection/consistent.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! Consistent Hashing\n\nuse super::*;\nuse pingora_core::protocols::l4::socket::SocketAddr;\nuse pingora_ketama::{Bucket, Continuum, Version};\nuse std::collections::HashMap;\n\n/// Weighted Ketama consistent hashing\npub struct KetamaHashing {\n    ring: Continuum,\n    // TODO: update Ketama to just store this\n    backends: HashMap<SocketAddr, Backend>,\n}\n\n#[derive(Clone, Debug, Copy, Default)]\npub struct KetamaConfig {\n    pub point_multiple: Option<u32>,\n}\n\nimpl BackendSelection for KetamaHashing {\n    type Iter = OwnedNodeIterator;\n\n    type Config = KetamaConfig;\n\n    fn build_with_config(backends: &BTreeSet<Backend>, config: &Self::Config) -> Self {\n        let KetamaConfig { point_multiple } = *config;\n\n        let buckets: Vec<_> = backends\n            .iter()\n            .filter_map(|b| {\n                // FIXME: ketama only supports Inet addr, UDS addrs are ignored here\n                if let SocketAddr::Inet(addr) = b.addr {\n                    Some(Bucket::new(addr, b.weight as u32))\n                } else {\n                    None\n                }\n            })\n            .collect();\n        let new_backends = backends\n            .iter()\n            .map(|b| (b.addr.clone(), b.clone()))\n            .collect();\n\n        #[allow(unused)]\n        let version = if let Some(point_multiple) = point_multiple {\n            match () {\n                #[cfg(feature = \"v2\")]\n                () => Version::V2 { point_multiple },\n                #[cfg(not(feature = \"v2\"))]\n                () => Version::V1,\n            }\n        } else {\n            Version::V1\n        };\n\n        KetamaHashing {\n            ring: Continuum::new_with_version(&buckets, version),\n            backends: new_backends,\n        }\n    }\n\n    fn build(backends: &BTreeSet<Backend>) -> Self {\n        Self::build_with_config(backends, &KetamaConfig::default())\n    }\n\n    fn iter(self: &Arc<Self>, key: &[u8]) -> Self::Iter {\n        OwnedNodeIterator {\n            idx: self.ring.node_idx(key),\n            ring: self.clone(),\n        }\n    }\n}\n\n/// Iterator over a Continuum\npub struct OwnedNodeIterator {\n    idx: usize,\n    ring: Arc<KetamaHashing>,\n}\n\nimpl BackendIter for OwnedNodeIterator {\n    fn next(&mut self) -> Option<&Backend> {\n        self.ring.ring.get_addr(&mut self.idx).and_then(|addr| {\n            let addr = SocketAddr::Inet(*addr);\n            self.ring.backends.get(&addr)\n        })\n    }\n}\n\n#[cfg(test)]\nmod test {\n    use super::*;\n\n    #[test]\n    fn test_ketama() {\n        let b1 = Backend::new(\"1.1.1.1:80\").unwrap();\n        let b2 = Backend::new(\"1.0.0.1:80\").unwrap();\n        let b3 = Backend::new(\"1.0.0.255:80\").unwrap();\n        let backends = BTreeSet::from_iter([b1.clone(), b2.clone(), b3.clone()]);\n        let hash = Arc::new(KetamaHashing::build(&backends));\n\n        let mut iter = hash.iter(b\"test0\");\n        assert_eq!(iter.next(), Some(&b2));\n        let mut iter = hash.iter(b\"test1\");\n        assert_eq!(iter.next(), Some(&b1));\n        let mut iter = hash.iter(b\"test2\");\n        assert_eq!(iter.next(), Some(&b1));\n        let mut iter = hash.iter(b\"test3\");\n        assert_eq!(iter.next(), Some(&b1));\n        let mut iter = hash.iter(b\"test4\");\n        assert_eq!(iter.next(), Some(&b1));\n        let mut iter = hash.iter(b\"test5\");\n        assert_eq!(iter.next(), Some(&b3));\n        let mut iter = hash.iter(b\"test6\");\n        assert_eq!(iter.next(), Some(&b1));\n        let mut iter = hash.iter(b\"test7\");\n        assert_eq!(iter.next(), Some(&b3));\n        let mut iter = hash.iter(b\"test8\");\n        assert_eq!(iter.next(), Some(&b1));\n        let mut iter = hash.iter(b\"test9\");\n        assert_eq!(iter.next(), Some(&b2));\n\n        // remove b3\n        let backends = BTreeSet::from_iter([b1.clone(), b2.clone()]);\n        let hash = Arc::new(KetamaHashing::build(&backends));\n        let mut iter = hash.iter(b\"test0\");\n        assert_eq!(iter.next(), Some(&b2));\n        let mut iter = hash.iter(b\"test1\");\n        assert_eq!(iter.next(), Some(&b1));\n        let mut iter = hash.iter(b\"test2\");\n        assert_eq!(iter.next(), Some(&b1));\n        let mut iter = hash.iter(b\"test3\");\n        assert_eq!(iter.next(), Some(&b1));\n        let mut iter = hash.iter(b\"test4\");\n        assert_eq!(iter.next(), Some(&b1));\n        let mut iter = hash.iter(b\"test5\");\n        assert_eq!(iter.next(), Some(&b2)); // changed\n        let mut iter = hash.iter(b\"test6\");\n        assert_eq!(iter.next(), Some(&b1));\n        let mut iter = hash.iter(b\"test7\");\n        assert_eq!(iter.next(), Some(&b1)); // changed\n        let mut iter = hash.iter(b\"test8\");\n        assert_eq!(iter.next(), Some(&b1));\n        let mut iter = hash.iter(b\"test9\");\n        assert_eq!(iter.next(), Some(&b2));\n    }\n}\n"
  },
  {
    "path": "pingora-load-balancing/src/selection/mod.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! Backend selection interfaces and algorithms\n\npub mod algorithms;\npub mod consistent;\npub mod weighted;\n\nuse super::Backend;\nuse std::collections::{BTreeSet, HashSet};\nuse std::sync::Arc;\nuse weighted::Weighted;\n\n/// [BackendSelection] is the interface to implement backend selection mechanisms.\npub trait BackendSelection: Sized {\n    /// The [BackendIter] returned from iter() below.\n    type Iter;\n\n    /// The configuration type constructing [BackendSelection]\n    type Config: Send + Sync;\n\n    /// Create a [BackendSelection] from a set of backends and the given configuration. The\n    /// default implementation ignores the configuration and simply calls [Self::build]\n    fn build_with_config(backends: &BTreeSet<Backend>, _config: &Self::Config) -> Self {\n        Self::build(backends)\n    }\n\n    /// The function to create a [BackendSelection] implementation.\n    fn build(backends: &BTreeSet<Backend>) -> Self;\n    /// Select backends for a given key.\n    ///\n    /// An [BackendIter] should be returned. The first item in the iter is the first\n    /// choice backend. The user should continue to iterate over it if the first backend\n    /// cannot be used due to its health or other reasons.\n    fn iter(self: &Arc<Self>, key: &[u8]) -> Self::Iter\n    where\n        Self::Iter: BackendIter;\n}\n\n/// An iterator to find the suitable backend\n///\n/// Similar to [Iterator] but allow self referencing.\npub trait BackendIter {\n    /// Return `Some(&Backend)` when there are more backends left to choose from.\n    fn next(&mut self) -> Option<&Backend>;\n}\n\n/// [SelectionAlgorithm] is the interface to implement selection algorithms.\n///\n/// All [std::hash::Hasher] + [Default] can be used directly as a selection algorithm.\npub trait SelectionAlgorithm {\n    /// Create a new implementation\n    fn new() -> Self;\n    /// Return the next index of backend. The caller should perform modulo to get\n    /// the valid index of the backend.\n    fn next(&self, key: &[u8]) -> u64;\n}\n\n/// [FNV](https://en.wikipedia.org/wiki/Fowler%E2%80%93Noll%E2%80%93Vo_hash_function) hashing\n/// on weighted backends\npub type FNVHash = Weighted<fnv::FnvHasher>;\n\n/// Alias of [`FNVHash`] for backwards compatibility until the next breaking change\n#[doc(hidden)]\npub type FVNHash = Weighted<fnv::FnvHasher>;\n/// Random selection on weighted backends\npub type Random = Weighted<algorithms::Random>;\n/// Round robin selection on weighted backends\npub type RoundRobin = Weighted<algorithms::RoundRobin>;\n/// Consistent Ketama hashing on weighted backends\npub type Consistent = consistent::KetamaHashing;\n\n// TODO: least conn\n\n/// An iterator which wraps another iterator and yields unique items. It optionally takes a max\n/// number of iterations if the wrapped iterator never returns.\npub struct UniqueIterator<I>\nwhere\n    I: BackendIter,\n{\n    iter: I,\n    seen: HashSet<u64>,\n    max_iterations: usize,\n    steps: usize,\n}\n\nimpl<I> UniqueIterator<I>\nwhere\n    I: BackendIter,\n{\n    /// Wrap a new iterator and specify the maximum number of times we want to iterate.\n    pub fn new(iter: I, max_iterations: usize) -> Self {\n        Self {\n            iter,\n            max_iterations,\n            seen: HashSet::new(),\n            steps: 0,\n        }\n    }\n\n    pub fn get_next(&mut self) -> Option<Backend> {\n        while let Some(item) = self.iter.next() {\n            if self.steps >= self.max_iterations {\n                return None;\n            }\n            self.steps += 1;\n\n            let hash_key = item.hash_key();\n            if !self.seen.contains(&hash_key) {\n                self.seen.insert(hash_key);\n                return Some(item.clone());\n            }\n        }\n\n        None\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use super::*;\n\n    struct TestIter {\n        seq: Vec<Backend>,\n        idx: usize,\n    }\n    impl TestIter {\n        fn new(input: &[&Backend]) -> Self {\n            Self {\n                seq: input.iter().cloned().cloned().collect(),\n                idx: 0,\n            }\n        }\n    }\n    impl BackendIter for TestIter {\n        fn next(&mut self) -> Option<&Backend> {\n            let idx = self.idx;\n            self.idx += 1;\n            self.seq.get(idx)\n        }\n    }\n\n    #[test]\n    fn unique_iter_max_iterations_is_correct() {\n        let b1 = Backend::new(\"1.1.1.1:80\").unwrap();\n        let b2 = Backend::new(\"1.0.0.1:80\").unwrap();\n        let b3 = Backend::new(\"1.0.0.255:80\").unwrap();\n        let items = [&b1, &b2, &b3];\n\n        let mut all = UniqueIterator::new(TestIter::new(&items), 3);\n        assert_eq!(all.get_next(), Some(b1.clone()));\n        assert_eq!(all.get_next(), Some(b2.clone()));\n        assert_eq!(all.get_next(), Some(b3.clone()));\n        assert_eq!(all.get_next(), None);\n\n        let mut stop = UniqueIterator::new(TestIter::new(&items), 1);\n        assert_eq!(stop.get_next(), Some(b1));\n        assert_eq!(stop.get_next(), None);\n    }\n\n    #[test]\n    fn unique_iter_duplicate_items_are_filtered() {\n        let b1 = Backend::new(\"1.1.1.1:80\").unwrap();\n        let b2 = Backend::new(\"1.0.0.1:80\").unwrap();\n        let b3 = Backend::new(\"1.0.0.255:80\").unwrap();\n        let items = [&b1, &b1, &b2, &b2, &b2, &b3];\n\n        let mut uniq = UniqueIterator::new(TestIter::new(&items), 10);\n        assert_eq!(uniq.get_next(), Some(b1));\n        assert_eq!(uniq.get_next(), Some(b2));\n        assert_eq!(uniq.get_next(), Some(b3));\n    }\n}\n"
  },
  {
    "path": "pingora-load-balancing/src/selection/weighted.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! Weighted Selection\n\nuse super::{Backend, BackendIter, BackendSelection, SelectionAlgorithm};\nuse fnv::FnvHasher;\nuse std::collections::BTreeSet;\nuse std::sync::Arc;\n\n/// Weighted selection with a given selection algorithm\n///\n/// The default algorithm is [FnvHasher]. See [super::algorithms] for more choices.\npub struct Weighted<H = FnvHasher> {\n    backends: Box<[Backend]>,\n    // each item is an index to the `backends`, use u16 to save memory, support up to 2^16 backends\n    weighted: Box<[u16]>,\n    algorithm: H,\n}\n\nimpl<H: SelectionAlgorithm> BackendSelection for Weighted<H> {\n    type Iter = WeightedIterator<H>;\n\n    type Config = ();\n\n    fn build(backends: &BTreeSet<Backend>) -> Self {\n        assert!(\n            backends.len() <= u16::MAX as usize,\n            \"support up to 2^16 backends\"\n        );\n        let backends = Vec::from_iter(backends.iter().cloned()).into_boxed_slice();\n        let mut weighted = Vec::with_capacity(backends.len());\n        for (index, b) in backends.iter().enumerate() {\n            for _ in 0..b.weight {\n                weighted.push(index as u16);\n            }\n        }\n        Weighted {\n            backends,\n            weighted: weighted.into_boxed_slice(),\n            algorithm: H::new(),\n        }\n    }\n\n    fn iter(self: &Arc<Self>, key: &[u8]) -> Self::Iter {\n        WeightedIterator::new(key, self.clone())\n    }\n}\n\n/// An iterator over the backends of a [Weighted] selection.\n///\n/// See [super::BackendSelection] for more information.\npub struct WeightedIterator<H> {\n    // the unbounded index seed\n    index: u64,\n    backend: Arc<Weighted<H>>,\n    first: bool,\n}\n\nimpl<H: SelectionAlgorithm> WeightedIterator<H> {\n    /// Constructs a new [WeightedIterator].\n    fn new(input: &[u8], backend: Arc<Weighted<H>>) -> Self {\n        Self {\n            index: backend.algorithm.next(input),\n            backend,\n            first: true,\n        }\n    }\n}\n\nimpl<H: SelectionAlgorithm> BackendIter for WeightedIterator<H> {\n    fn next(&mut self) -> Option<&Backend> {\n        if self.backend.backends.is_empty() {\n            // short circuit if empty\n            return None;\n        }\n\n        if self.first {\n            // initial hash, select from the weighted list\n            self.first = false;\n            let len = self.backend.weighted.len();\n            let index = self.backend.weighted[self.index as usize % len];\n            Some(&self.backend.backends[index as usize])\n        } else {\n            // fallback, select from the unique list\n            // deterministically select the next item\n            self.index = self.backend.algorithm.next(&self.index.to_le_bytes());\n            let len = self.backend.backends.len();\n            Some(&self.backend.backends[self.index as usize % len])\n        }\n    }\n}\n\n#[cfg(test)]\nmod test {\n    use super::super::algorithms::*;\n    use super::*;\n    use std::collections::HashMap;\n\n    #[test]\n    fn test_fnv() {\n        let b1 = Backend::new(\"1.1.1.1:80\").unwrap();\n        let mut b2 = Backend::new(\"1.0.0.1:80\").unwrap();\n        b2.weight = 10; // 10x than the rest\n        let b3 = Backend::new(\"1.0.0.255:80\").unwrap();\n        let backends = BTreeSet::from_iter([b1.clone(), b2.clone(), b3.clone()]);\n        let hash: Arc<Weighted> = Arc::new(Weighted::build(&backends));\n\n        // same hash iter over\n        let mut iter = hash.iter(b\"test\");\n        // first, should be weighted\n        assert_eq!(iter.next(), Some(&b2));\n        // fallbacks, should be uniform, not weighted\n        assert_eq!(iter.next(), Some(&b2));\n        assert_eq!(iter.next(), Some(&b2));\n        assert_eq!(iter.next(), Some(&b1));\n        assert_eq!(iter.next(), Some(&b3));\n        assert_eq!(iter.next(), Some(&b2));\n        assert_eq!(iter.next(), Some(&b2));\n        assert_eq!(iter.next(), Some(&b1));\n        assert_eq!(iter.next(), Some(&b2));\n        assert_eq!(iter.next(), Some(&b3));\n        assert_eq!(iter.next(), Some(&b1));\n\n        // different hashes, the first selection should be weighted\n        let mut iter = hash.iter(b\"test1\");\n        assert_eq!(iter.next(), Some(&b2));\n        let mut iter = hash.iter(b\"test2\");\n        assert_eq!(iter.next(), Some(&b2));\n        let mut iter = hash.iter(b\"test3\");\n        assert_eq!(iter.next(), Some(&b3));\n        let mut iter = hash.iter(b\"test4\");\n        assert_eq!(iter.next(), Some(&b1));\n        let mut iter = hash.iter(b\"test5\");\n        assert_eq!(iter.next(), Some(&b2));\n        let mut iter = hash.iter(b\"test6\");\n        assert_eq!(iter.next(), Some(&b2));\n        let mut iter = hash.iter(b\"test7\");\n        assert_eq!(iter.next(), Some(&b2));\n    }\n\n    #[test]\n    fn test_round_robin() {\n        let b1 = Backend::new(\"1.1.1.1:80\").unwrap();\n        let mut b2 = Backend::new(\"1.0.0.1:80\").unwrap();\n        b2.weight = 8; // 8x than the rest\n        let b3 = Backend::new(\"1.0.0.255:80\").unwrap();\n        // sorted with: [b2, b3, b1]\n        // weighted: [0, 0, 0, 0, 0, 0, 0, 0, 1, 2]\n        let backends = BTreeSet::from_iter([b1.clone(), b2.clone(), b3.clone()]);\n        let hash: Arc<Weighted<RoundRobin>> = Arc::new(Weighted::build(&backends));\n\n        // same hash iter over\n        let mut iter = hash.iter(b\"test\");\n        // first, should be weighted\n        // weighted: [0, 0, 0, 0, 0, 0, 0, 0, 1, 2]\n        //            ^\n        assert_eq!(iter.next(), Some(&b2));\n        // fallbacks, should be round robin\n        assert_eq!(iter.next(), Some(&b3));\n        assert_eq!(iter.next(), Some(&b1));\n        assert_eq!(iter.next(), Some(&b2));\n        assert_eq!(iter.next(), Some(&b3));\n\n        // round robin, ignoring the hash key\n        // index advanced 5 steps\n        // weighted: [0, 0, 0, 0, 0, 0, 0, 0, 1, 2]\n        //                           ^\n        let mut iter = hash.iter(b\"test1\");\n        assert_eq!(iter.next(), Some(&b2));\n        let mut iter = hash.iter(b\"test1\");\n        assert_eq!(iter.next(), Some(&b2));\n        let mut iter = hash.iter(b\"test1\");\n        assert_eq!(iter.next(), Some(&b2));\n        let mut iter = hash.iter(b\"test1\");\n        assert_eq!(iter.next(), Some(&b3));\n        let mut iter = hash.iter(b\"test1\");\n        assert_eq!(iter.next(), Some(&b1));\n        let mut iter = hash.iter(b\"test1\");\n        // rounded\n        assert_eq!(iter.next(), Some(&b2));\n        let mut iter = hash.iter(b\"test1\");\n        assert_eq!(iter.next(), Some(&b2));\n    }\n\n    #[test]\n    fn test_random() {\n        let b1 = Backend::new(\"1.1.1.1:80\").unwrap();\n        let mut b2 = Backend::new(\"1.0.0.1:80\").unwrap();\n        b2.weight = 8; // 8x than the rest\n        let b3 = Backend::new(\"1.0.0.255:80\").unwrap();\n        let backends = BTreeSet::from_iter([b1.clone(), b2.clone(), b3.clone()]);\n        let hash: Arc<Weighted<Random>> = Arc::new(Weighted::build(&backends));\n\n        let mut count = HashMap::new();\n        count.insert(b1.clone(), 0);\n        count.insert(b2.clone(), 0);\n        count.insert(b3.clone(), 0);\n\n        for _ in 0..10000 {\n            let mut iter = hash.iter(b\"test\");\n            *count.get_mut(iter.next().unwrap()).unwrap() += 1;\n        }\n        let b2_count = *count.get(&b2).unwrap();\n        assert!((7000..=9000).contains(&b2_count));\n    }\n}\n"
  },
  {
    "path": "pingora-lru/Cargo.toml",
    "content": "[package]\nname = \"pingora-lru\"\nversion = \"0.8.0\"\nauthors = [\"Yuchen Wu <yuchen@cloudflare.com>\"]\nlicense = \"Apache-2.0\"\nedition = \"2021\"\nrepository = \"https://github.com/cloudflare/pingora\"\ncategories = [\"algorithms\", \"caching\"]\nkeywords = [\"lru\", \"cache\", \"pingora\"]\ndescription = \"\"\"\nLRU cache that focuses on memory efficiency, concurrency and persistence.\n\"\"\"\n\n# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html\n[lib]\nname = \"pingora_lru\"\npath = \"src/lib.rs\"\n\n[dependencies]\nhashbrown = \"0\"\nparking_lot = \"0\"\narrayvec = \"0\"\nrand = \"0.8\"\n\n[dev-dependencies]\nlru = { workspace = true }\n\n[[bench]]\nname = \"bench_linked_list\"\nharness = false\n\n[[bench]]\nname = \"bench_lru\"\nharness = false\n"
  },
  {
    "path": "pingora-lru/LICENSE",
    "content": "\n                                 Apache License\n                           Version 2.0, January 2004\n                        http://www.apache.org/licenses/\n\n   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION\n\n   1. Definitions.\n\n      \"License\" shall mean the terms and conditions for use, reproduction,\n      and distribution as defined by Sections 1 through 9 of this document.\n\n      \"Licensor\" shall mean the copyright owner or entity authorized by\n      the copyright owner that is granting the License.\n\n      \"Legal Entity\" shall mean the union of the acting entity and all\n      other entities that control, are controlled by, or are under common\n      control with that entity. For the purposes of this definition,\n      \"control\" means (i) the power, direct or indirect, to cause the\n      direction or management of such entity, whether by contract or\n      otherwise, or (ii) ownership of fifty percent (50%) or more of the\n      outstanding shares, or (iii) beneficial ownership of such entity.\n\n      \"You\" (or \"Your\") shall mean an individual or Legal Entity\n      exercising permissions granted by this License.\n\n      \"Source\" form shall mean the preferred form for making modifications,\n      including but not limited to software source code, documentation\n      source, and configuration files.\n\n      \"Object\" form shall mean any form resulting from mechanical\n      transformation or translation of a Source form, including but\n      not limited to compiled object code, generated documentation,\n      and conversions to other media types.\n\n      \"Work\" shall mean the work of authorship, whether in Source or\n      Object form, made available under the License, as indicated by a\n      copyright notice that is included in or attached to the work\n      (an example is provided in the Appendix below).\n\n      \"Derivative Works\" shall mean any work, whether in Source or Object\n      form, that is based on (or derived from) the Work and for which the\n      editorial revisions, annotations, elaborations, or other modifications\n      represent, as a whole, an original work of authorship. For the purposes\n      of this License, Derivative Works shall not include works that remain\n      separable from, or merely link (or bind by name) to the interfaces of,\n      the Work and Derivative Works thereof.\n\n      \"Contribution\" shall mean any work of authorship, including\n      the original version of the Work and any modifications or additions\n      to that Work or Derivative Works thereof, that is intentionally\n      submitted to Licensor for inclusion in the Work by the copyright owner\n      or by an individual or Legal Entity authorized to submit on behalf of\n      the copyright owner. For the purposes of this definition, \"submitted\"\n      means any form of electronic, verbal, or written communication sent\n      to the Licensor or its representatives, including but not limited to\n      communication on electronic mailing lists, source code control systems,\n      and issue tracking systems that are managed by, or on behalf of, the\n      Licensor for the purpose of discussing and improving the Work, but\n      excluding communication that is conspicuously marked or otherwise\n      designated in writing by the copyright owner as \"Not a Contribution.\"\n\n      \"Contributor\" shall mean Licensor and any individual or Legal Entity\n      on behalf of whom a Contribution has been received by Licensor and\n      subsequently incorporated within the Work.\n\n   2. Grant of Copyright License. Subject to the terms and conditions of\n      this License, each Contributor hereby grants to You a perpetual,\n      worldwide, non-exclusive, no-charge, royalty-free, irrevocable\n      copyright license to reproduce, prepare Derivative Works of,\n      publicly display, publicly perform, sublicense, and distribute the\n      Work and such Derivative Works in Source or Object form.\n\n   3. Grant of Patent License. Subject to the terms and conditions of\n      this License, each Contributor hereby grants to You a perpetual,\n      worldwide, non-exclusive, no-charge, royalty-free, irrevocable\n      (except as stated in this section) patent license to make, have made,\n      use, offer to sell, sell, import, and otherwise transfer the Work,\n      where such license applies only to those patent claims licensable\n      by such Contributor that are necessarily infringed by their\n      Contribution(s) alone or by combination of their Contribution(s)\n      with the Work to which such Contribution(s) was submitted. If You\n      institute patent litigation against any entity (including a\n      cross-claim or counterclaim in a lawsuit) alleging that the Work\n      or a Contribution incorporated within the Work constitutes direct\n      or contributory patent infringement, then any patent licenses\n      granted to You under this License for that Work shall terminate\n      as of the date such litigation is filed.\n\n   4. Redistribution. You may reproduce and distribute copies of the\n      Work or Derivative Works thereof in any medium, with or without\n      modifications, and in Source or Object form, provided that You\n      meet the following conditions:\n\n      (a) You must give any other recipients of the Work or\n          Derivative Works a copy of this License; and\n\n      (b) You must cause any modified files to carry prominent notices\n          stating that You changed the files; and\n\n      (c) You must retain, in the Source form of any Derivative Works\n          that You distribute, all copyright, patent, trademark, and\n          attribution notices from the Source form of the Work,\n          excluding those notices that do not pertain to any part of\n          the Derivative Works; and\n\n      (d) If the Work includes a \"NOTICE\" text file as part of its\n          distribution, then any Derivative Works that You distribute must\n          include a readable copy of the attribution notices contained\n          within such NOTICE file, excluding those notices that do not\n          pertain to any part of the Derivative Works, in at least one\n          of the following places: within a NOTICE text file distributed\n          as part of the Derivative Works; within the Source form or\n          documentation, if provided along with the Derivative Works; or,\n          within a display generated by the Derivative Works, if and\n          wherever such third-party notices normally appear. The contents\n          of the NOTICE file are for informational purposes only and\n          do not modify the License. You may add Your own attribution\n          notices within Derivative Works that You distribute, alongside\n          or as an addendum to the NOTICE text from the Work, provided\n          that such additional attribution notices cannot be construed\n          as modifying the License.\n\n      You may add Your own copyright statement to Your modifications and\n      may provide additional or different license terms and conditions\n      for use, reproduction, or distribution of Your modifications, or\n      for any such Derivative Works as a whole, provided Your use,\n      reproduction, and distribution of the Work otherwise complies with\n      the conditions stated in this License.\n\n   5. Submission of Contributions. Unless You explicitly state otherwise,\n      any Contribution intentionally submitted for inclusion in the Work\n      by You to the Licensor shall be under the terms and conditions of\n      this License, without any additional terms or conditions.\n      Notwithstanding the above, nothing herein shall supersede or modify\n      the terms of any separate license agreement you may have executed\n      with Licensor regarding such Contributions.\n\n   6. Trademarks. This License does not grant permission to use the trade\n      names, trademarks, service marks, or product names of the Licensor,\n      except as required for reasonable and customary use in describing the\n      origin of the Work and reproducing the content of the NOTICE file.\n\n   7. Disclaimer of Warranty. Unless required by applicable law or\n      agreed to in writing, Licensor provides the Work (and each\n      Contributor provides its Contributions) on an \"AS IS\" BASIS,\n      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or\n      implied, including, without limitation, any warranties or conditions\n      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A\n      PARTICULAR PURPOSE. You are solely responsible for determining the\n      appropriateness of using or redistributing the Work and assume any\n      risks associated with Your exercise of permissions under this License.\n\n   8. Limitation of Liability. In no event and under no legal theory,\n      whether in tort (including negligence), contract, or otherwise,\n      unless required by applicable law (such as deliberate and grossly\n      negligent acts) or agreed to in writing, shall any Contributor be\n      liable to You for damages, including any direct, indirect, special,\n      incidental, or consequential damages of any character arising as a\n      result of this License or out of the use or inability to use the\n      Work (including but not limited to damages for loss of goodwill,\n      work stoppage, computer failure or malfunction, or any and all\n      other commercial damages or losses), even if such Contributor\n      has been advised of the possibility of such damages.\n\n   9. Accepting Warranty or Additional Liability. While redistributing\n      the Work or Derivative Works thereof, You may choose to offer,\n      and charge a fee for, acceptance of support, warranty, indemnity,\n      or other liability obligations and/or rights consistent with this\n      License. However, in accepting such obligations, You may act only\n      on Your own behalf and on Your sole responsibility, not on behalf\n      of any other Contributor, and only if You agree to indemnify,\n      defend, and hold each Contributor harmless for any liability\n      incurred by, or claims asserted against, such Contributor by reason\n      of your accepting any such warranty or additional liability.\n\n   END OF TERMS AND CONDITIONS\n\n   APPENDIX: How to apply the Apache License to your work.\n\n      To apply the Apache License to your work, attach the following\n      boilerplate notice, with the fields enclosed by brackets \"[]\"\n      replaced with your own identifying information. (Don't include\n      the brackets!)  The text should be enclosed in the appropriate\n      comment syntax for the file format. We also recommend that a\n      file or class name and description of purpose be included on the\n      same \"printed page\" as the copyright notice for easier\n      identification within third-party archives.\n\n   Copyright [yyyy] [name of copyright owner]\n\n   Licensed under the Apache License, Version 2.0 (the \"License\");\n   you may not use this file except in compliance with the License.\n   You may obtain a copy of the License at\n\n       http://www.apache.org/licenses/LICENSE-2.0\n\n   Unless required by applicable law or agreed to in writing, software\n   distributed under the License is distributed on an \"AS IS\" BASIS,\n   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n   See the License for the specific language governing permissions and\n   limitations under the License.\n"
  },
  {
    "path": "pingora-lru/benches/bench_linked_list.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\nuse std::time::Instant;\n\nfn main() {\n    const ITEMS: usize = 5_000_000;\n\n    // push bench\n\n    let mut std_list = std::collections::LinkedList::<u64>::new();\n    let before = Instant::now();\n    for _ in 0..ITEMS {\n        std_list.push_front(0);\n    }\n    let elapsed = before.elapsed();\n    println!(\n        \"std linked list push_front total {elapsed:?}, {:?} avg per operation\",\n        elapsed / ITEMS as u32\n    );\n\n    let mut list = pingora_lru::linked_list::LinkedList::with_capacity(ITEMS);\n    let before = Instant::now();\n    for _ in 0..ITEMS {\n        list.push_head(0);\n    }\n    let elapsed = before.elapsed();\n    println!(\n        \"pingora linked list push_head total {elapsed:?}, {:?} avg per operation\",\n        elapsed / ITEMS as u32\n    );\n\n    // iter bench\n\n    let mut count = 0;\n    let before = Instant::now();\n    for _ in std_list.iter() {\n        count += 1;\n    }\n    let elapsed = before.elapsed();\n    println!(\n        \"std linked list iter total {count} {elapsed:?}, {:?} avg per operation\",\n        elapsed / count as u32\n    );\n\n    let mut count = 0;\n    let before = Instant::now();\n    for _ in list.iter() {\n        count += 1;\n    }\n    let elapsed = before.elapsed();\n    println!(\n        \"pingora linked list iter total {count} {elapsed:?}, {:?} avg per operation\",\n        elapsed / count as u32\n    );\n\n    // search bench\n\n    let before = Instant::now();\n    for _ in 0..ITEMS {\n        assert!(!std_list.iter().take(10).any(|v| *v == 1));\n    }\n    let elapsed = before.elapsed();\n    println!(\n        \"std linked search first 10 items total {elapsed:?}, {:?} avg per operation\",\n        elapsed / ITEMS as u32\n    );\n\n    let before = Instant::now();\n    for _ in 0..ITEMS {\n        assert!(!list.iter().take(10).any(|v| *v == 1));\n    }\n    let elapsed = before.elapsed();\n    println!(\n        \"pingora linked search first 10 items total {elapsed:?}, {:?} avg per operation\",\n        elapsed / ITEMS as u32\n    );\n\n    let before = Instant::now();\n    for _ in 0..ITEMS {\n        assert!(!list.exist_near_head(1, 10));\n    }\n    let elapsed = before.elapsed();\n    println!(\n        \"pingora linked optimized search first 10 items total {elapsed:?}, {:?} avg per operation\",\n        elapsed / ITEMS as u32\n    );\n\n    // move node bench\n    let before = Instant::now();\n    for _ in 0..ITEMS {\n        let value = std_list.pop_back().unwrap();\n        std_list.push_front(value);\n    }\n    let elapsed = before.elapsed();\n    println!(\n        \"std linked list move back to front total {elapsed:?}, {:?} avg per operation\",\n        elapsed / ITEMS as u32\n    );\n\n    let before = Instant::now();\n    for _ in 0..ITEMS {\n        let index = list.tail().unwrap();\n        list.promote(index);\n    }\n    let elapsed = before.elapsed();\n    println!(\n        \"pingora linked list move tail to head total {elapsed:?}, {:?} avg per operation\",\n        elapsed / ITEMS as u32\n    );\n\n    // pop bench\n\n    let before = Instant::now();\n    for _ in 0..ITEMS {\n        std_list.pop_back();\n    }\n    let elapsed = before.elapsed();\n    println!(\n        \"std linked list pop_back {elapsed:?}, {:?} avg per operation\",\n        elapsed / ITEMS as u32\n    );\n\n    let before = Instant::now();\n    for _ in 0..ITEMS {\n        list.pop_tail();\n    }\n    let elapsed = before.elapsed();\n    println!(\n        \"pingora linked list pop_tail total {elapsed:?}, {:?} avg per operation\",\n        elapsed / ITEMS as u32\n    );\n}\n"
  },
  {
    "path": "pingora-lru/benches/bench_lru.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\nuse rand::distributions::WeightedIndex;\nuse rand::prelude::*;\nuse std::sync::Arc;\nuse std::thread;\nuse std::time::Instant;\n\n// Non-uniform distributions, 100 items, 10 of them are 100x more likely to appear\nconst WEIGHTS: &[usize] = &[\n    1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,\n    1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,\n    1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 100, 100, 100,\n    100, 100, 100, 100, 100, 100, 100,\n];\n\nconst ITERATIONS: usize = 5_000_000;\nconst THREADS: usize = 8;\n\nfn main() {\n    let lru = parking_lot::Mutex::new(lru::LruCache::<u64, ()>::unbounded());\n\n    let plru = pingora_lru::Lru::<(), 10>::with_capacity(1000, 100);\n    // populate first, then we bench access/promotion\n    for i in 0..WEIGHTS.len() {\n        lru.lock().put(i as u64, ());\n    }\n    for i in 0..WEIGHTS.len() {\n        plru.admit(i as u64, (), 1);\n    }\n\n    // single thread\n    let mut rng = thread_rng();\n    let dist = WeightedIndex::new(WEIGHTS).unwrap();\n\n    let before = Instant::now();\n    for _ in 0..ITERATIONS {\n        lru.lock().get(&(dist.sample(&mut rng) as u64));\n    }\n    let elapsed = before.elapsed();\n    println!(\n        \"lru promote total {elapsed:?}, {:?} avg per operation\",\n        elapsed / ITERATIONS as u32\n    );\n\n    let before = Instant::now();\n    for _ in 0..ITERATIONS {\n        plru.promote(dist.sample(&mut rng) as u64);\n    }\n    let elapsed = before.elapsed();\n    println!(\n        \"pingora lru promote total {elapsed:?}, {:?} avg per operation\",\n        elapsed / ITERATIONS as u32\n    );\n\n    let before = Instant::now();\n    for _ in 0..ITERATIONS {\n        plru.promote_top_n(dist.sample(&mut rng) as u64, 10);\n    }\n    let elapsed = before.elapsed();\n    println!(\n        \"pingora lru promote_top_10 total {elapsed:?}, {:?} avg per operation\",\n        elapsed / ITERATIONS as u32\n    );\n\n    // concurrent\n\n    let lru = Arc::new(lru);\n    let mut handlers = vec![];\n    for i in 0..THREADS {\n        let lru = lru.clone();\n        let handler = thread::spawn(move || {\n            let mut rng = thread_rng();\n            let dist = WeightedIndex::new(WEIGHTS).unwrap();\n            let before = Instant::now();\n            for _ in 0..ITERATIONS {\n                lru.lock().get(&(dist.sample(&mut rng) as u64));\n            }\n            let elapsed = before.elapsed();\n            println!(\n                \"lru promote total {elapsed:?}, {:?} avg per operation thread {i}\",\n                elapsed / ITERATIONS as u32\n            );\n        });\n        handlers.push(handler);\n    }\n    for thread in handlers {\n        thread.join().unwrap();\n    }\n\n    let plru = Arc::new(plru);\n\n    let mut handlers = vec![];\n    for i in 0..THREADS {\n        let plru = plru.clone();\n        let handler = thread::spawn(move || {\n            let mut rng = thread_rng();\n            let dist = WeightedIndex::new(WEIGHTS).unwrap();\n            let before = Instant::now();\n            for _ in 0..ITERATIONS {\n                plru.promote(dist.sample(&mut rng) as u64);\n            }\n            let elapsed = before.elapsed();\n            println!(\n                \"pingora lru promote total {elapsed:?}, {:?} avg per operation thread {i}\",\n                elapsed / ITERATIONS as u32\n            );\n        });\n        handlers.push(handler);\n    }\n    for thread in handlers {\n        thread.join().unwrap();\n    }\n\n    let mut handlers = vec![];\n    for i in 0..THREADS {\n        let plru = plru.clone();\n        let handler = thread::spawn(move || {\n            let mut rng = thread_rng();\n            let dist = WeightedIndex::new(WEIGHTS).unwrap();\n            let before = Instant::now();\n            for _ in 0..ITERATIONS {\n                plru.promote_top_n(dist.sample(&mut rng) as u64, 10);\n            }\n            let elapsed = before.elapsed();\n            println!(\n                \"pingora lru promote_top_10 total {elapsed:?}, {:?} avg per operation thread {i}\",\n                elapsed / ITERATIONS as u32\n            );\n        });\n        handlers.push(handler);\n    }\n    for thread in handlers {\n        thread.join().unwrap();\n    }\n}\n"
  },
  {
    "path": "pingora-lru/src/lib.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! An implementation of an LRU that focuses on memory efficiency, concurrency and persistence\n//!\n//! Features\n//! - keys can have different sizes\n//! - LRUs are sharded to avoid global locks.\n//! - Memory layout and usage are optimized: small and no memory fragmentation\n\npub mod linked_list;\n\nuse linked_list::{LinkedList, LinkedListIter};\n\nuse hashbrown::HashMap;\nuse parking_lot::RwLock;\nuse std::sync::atomic::{AtomicUsize, Ordering};\n\n/// The LRU with `N` shards\npub struct Lru<T, const N: usize> {\n    units: [RwLock<LruUnit<T>>; N],\n    weight: AtomicUsize,\n    weight_limit: usize,\n    len_watermark: Option<usize>,\n    len: AtomicUsize,\n    evicted_weight: AtomicUsize,\n    evicted_len: AtomicUsize,\n}\n\nimpl<T, const N: usize> Lru<T, N> {\n    /// Create an [Lru] with the given weight limit and predicted capacity.\n    ///\n    /// The capacity is per shard (for simplicity). So the total capacity = capacity * N\n    pub fn with_capacity(weight_limit: usize, capacity: usize) -> Self {\n        Self::with_capacity_and_watermark(weight_limit, capacity, None)\n    }\n\n    /// Create an [Lru] with the given weight limit, predicted capacity and optional watermark\n    ///\n    /// The capacity is per shard (for simplicity). So the total capacity = capacity * N\n    ///\n    /// The watermark indicates at what count we should begin evicting and acts as a limit\n    /// on the total number of allowed items.\n    pub fn with_capacity_and_watermark(\n        weight_limit: usize,\n        capacity: usize,\n        len_watermark: Option<usize>,\n    ) -> Self {\n        // use the unsafe code from ArrayVec just to init the array\n        let mut units = arrayvec::ArrayVec::<_, N>::new();\n        for _ in 0..N {\n            units.push(RwLock::new(LruUnit::with_capacity(capacity)));\n        }\n        Lru {\n            units: units.into_inner().map_err(|_| \"\").unwrap(),\n            weight: AtomicUsize::new(0),\n            weight_limit,\n            len_watermark,\n            len: AtomicUsize::new(0),\n            evicted_weight: AtomicUsize::new(0),\n            evicted_len: AtomicUsize::new(0),\n        }\n    }\n\n    /// Admit the key value to the [Lru]\n    ///\n    /// Return the shard index which the asset is added to\n    pub fn admit(&self, key: u64, data: T, weight: usize) -> usize {\n        let shard = get_shard(key, N);\n        let unit = &mut self.units[shard].write();\n\n        // Make sure weight is positive otherwise eviction won't work\n        // TODO: Probably should use NonZeroUsize instead\n        let weight = weight.max(1);\n\n        let old_weight = unit.admit(key, data, weight);\n        if old_weight != weight {\n            self.weight.fetch_add(weight, Ordering::Relaxed);\n            if old_weight > 0 {\n                self.weight.fetch_sub(old_weight, Ordering::Relaxed);\n            } else {\n                // Assume old_weight == 0 means a new item is admitted\n                self.len.fetch_add(1, Ordering::Relaxed);\n            }\n        }\n        shard\n    }\n\n    /// Increment the weight associated with a given key, up to an optional max weight.\n    /// If a `max_weight` is provided, the weight cannot exceed this max weight. If the current\n    /// weight is higher than the max, it will be capped to the max.\n    ///\n    /// Return the total new weight. 0 indicates the key did not exist.\n    pub fn increment_weight(&self, key: u64, delta: usize, max_weight: Option<usize>) -> usize {\n        let shard = get_shard(key, N);\n        let unit = &mut self.units[shard].write();\n        if let Some((old_weight, new_weight)) = unit.increment_weight(key, delta, max_weight) {\n            if new_weight >= old_weight {\n                self.weight\n                    .fetch_add(new_weight - old_weight, Ordering::Relaxed);\n            } else {\n                self.weight\n                    .fetch_sub(old_weight - new_weight, Ordering::Relaxed);\n            }\n            new_weight\n        } else {\n            0\n        }\n    }\n\n    /// Promote the key to the head of the LRU\n    ///\n    /// Return `true` if the key exists.\n    pub fn promote(&self, key: u64) -> bool {\n        self.units[get_shard(key, N)].write().access(key)\n    }\n\n    /// Promote to the top n of the LRU\n    ///\n    /// This function is a bit more efficient in terms of reducing lock contention because it\n    /// will acquire a write lock only if the key is outside top n but only acquires a read lock\n    /// when the key is already in the top n.\n    ///\n    /// Return false if the item doesn't exist\n    pub fn promote_top_n(&self, key: u64, top: usize) -> bool {\n        let unit = &self.units[get_shard(key, N)];\n        if !unit.read().need_promote(key, top) {\n            return true;\n        }\n        unit.write().access(key)\n    }\n\n    /// Evict at most one item from the given shard\n    ///\n    /// Return the evicted asset and its size if there is anything to evict\n    pub fn evict_shard(&self, shard: u64) -> Option<(T, usize)> {\n        let evicted = self.units[get_shard(shard, N)].write().evict();\n        if let Some((_, weight)) = evicted.as_ref() {\n            self.weight.fetch_sub(*weight, Ordering::Relaxed);\n            self.len.fetch_sub(1, Ordering::Relaxed);\n            self.evicted_weight.fetch_add(*weight, Ordering::Relaxed);\n            self.evicted_len.fetch_add(1, Ordering::Relaxed);\n        }\n        evicted\n    }\n\n    /// Evict the [Lru] until the overall weight is below the limit (or the configured watermark).\n    ///\n    /// Return a list of evicted items.\n    ///\n    /// The evicted items are randomly selected from all the shards.\n    pub fn evict_to_limit(&self) -> Vec<(T, usize)> {\n        let mut evicted = vec![];\n        let mut initial_weight = self.weight();\n        let mut initial_len = self.len();\n        let mut shard_seed = rand::random(); // start from a random shard\n        let mut empty_shard = 0;\n\n        // Entries can be admitted or removed from the LRU by others during the loop below\n        // Track initial size not to over evict due to entries admitted after the loop starts\n        // self.weight() / self.len() is also used not to over evict\n        // due to entries already removed by others\n        while ((initial_weight > self.weight_limit && self.weight() > self.weight_limit)\n            || self\n                .len_watermark\n                .is_some_and(|w| initial_len > w && self.len() > w))\n            && empty_shard < N\n        {\n            if let Some(i) = self.evict_shard(shard_seed) {\n                initial_weight -= i.1;\n                initial_len = initial_len.saturating_sub(1);\n                evicted.push(i)\n            } else {\n                empty_shard += 1;\n            }\n            // move on to the next shard\n            shard_seed += 1;\n        }\n        evicted\n    }\n\n    /// Remove the given asset.\n    pub fn remove(&self, key: u64) -> Option<(T, usize)> {\n        let removed = self.units[get_shard(key, N)].write().remove(key);\n        if let Some((_, weight)) = removed.as_ref() {\n            self.weight.fetch_sub(*weight, Ordering::Relaxed);\n            self.len.fetch_sub(1, Ordering::Relaxed);\n        }\n        removed\n    }\n\n    /// Insert the item to the tail of this LRU.\n    ///\n    /// Useful to recreate an LRU in most-to-least order\n    pub fn insert_tail(&self, key: u64, data: T, weight: usize) -> bool {\n        if self.units[get_shard(key, N)]\n            .write()\n            .insert_tail(key, data, weight)\n        {\n            self.weight.fetch_add(weight, Ordering::Relaxed);\n            self.len.fetch_add(1, Ordering::Relaxed);\n            true\n        } else {\n            false\n        }\n    }\n\n    /// Check existence of a key without changing the order in LRU.\n    pub fn peek(&self, key: u64) -> bool {\n        self.units[get_shard(key, N)].read().peek(key).is_some()\n    }\n\n    /// Check the weight of a key without changing the order in LRU.\n    pub fn peek_weight(&self, key: u64) -> Option<usize> {\n        self.units[get_shard(key, N)].read().peek_weight(key)\n    }\n\n    /// Return the current total weight.\n    pub fn weight(&self) -> usize {\n        self.weight.load(Ordering::Relaxed)\n    }\n\n    /// Return the total weight of items evicted from this [Lru].\n    pub fn evicted_weight(&self) -> usize {\n        self.evicted_weight.load(Ordering::Relaxed)\n    }\n\n    /// Return the total count of items evicted from this [Lru].\n    pub fn evicted_len(&self) -> usize {\n        self.evicted_len.load(Ordering::Relaxed)\n    }\n\n    /// The number of items inside this [Lru].\n    #[allow(clippy::len_without_is_empty)]\n    pub fn len(&self) -> usize {\n        self.len.load(Ordering::Relaxed)\n    }\n\n    /// Scan a shard with the given function F\n    pub fn iter_for_each<F>(&self, shard: usize, f: F)\n    where\n        F: FnMut((&T, usize)),\n    {\n        assert!(shard < N);\n        self.units[shard].read().iter().for_each(f);\n    }\n\n    /// Get the total number of shards\n    pub const fn shards(&self) -> usize {\n        N\n    }\n\n    /// Get the number of items inside a shard\n    pub fn shard_len(&self, shard: usize) -> usize {\n        self.units[shard].read().len()\n    }\n\n    /// Get the weight (total size) inside a shard\n    pub fn shard_weight(&self, shard: usize) -> usize {\n        self.units[shard].read().used_weight\n    }\n}\n\n#[inline]\nfn get_shard(key: u64, n_shards: usize) -> usize {\n    (key % n_shards as u64) as usize\n}\n\nstruct LruNode<T> {\n    data: T,\n    list_index: usize,\n    weight: usize,\n}\n\nstruct LruUnit<T> {\n    lookup_table: HashMap<u64, Box<LruNode<T>>>,\n    order: LinkedList,\n    used_weight: usize,\n}\n\nimpl<T> LruUnit<T> {\n    fn with_capacity(capacity: usize) -> Self {\n        LruUnit {\n            lookup_table: HashMap::with_capacity(capacity),\n            order: LinkedList::with_capacity(capacity),\n            used_weight: 0,\n        }\n    }\n\n    /// Peek data associated with key, if it exists.\n    pub fn peek(&self, key: u64) -> Option<&T> {\n        self.lookup_table.get(&key).map(|n| &n.data)\n    }\n\n    /// Peek weight associated with key, if it exists.\n    pub fn peek_weight(&self, key: u64) -> Option<usize> {\n        self.lookup_table.get(&key).map(|n| n.weight)\n    }\n\n    /// Admit into LRU, return old weight if there was any.\n    pub fn admit(&mut self, key: u64, data: T, weight: usize) -> usize {\n        if let Some(node) = self.lookup_table.get_mut(&key) {\n            let old_weight = Self::adjust_weight(node, &mut self.used_weight, weight);\n            node.data = data;\n            self.order.promote(node.list_index);\n            return old_weight;\n        }\n        self.used_weight += weight;\n        let list_index = self.order.push_head(key);\n        let node = Box::new(LruNode {\n            data,\n            list_index,\n            weight,\n        });\n        self.lookup_table.insert(key, node);\n        0\n    }\n\n    /// Increase the weight of an existing key. Returns the new weight or 0 if the key did not\n    /// exist, along with the new weight (or 0).\n    ///\n    /// If a `max_weight` is provided, the weight cannot exceed this max weight. If the current\n    /// weight is higher than the max, it will be capped to the max.\n    pub fn increment_weight(\n        &mut self,\n        key: u64,\n        delta: usize,\n        max_weight: Option<usize>,\n    ) -> Option<(usize, usize)> {\n        if let Some(node) = self.lookup_table.get_mut(&key) {\n            let new_weight =\n                max_weight.map_or(node.weight + delta, |m| (node.weight + delta).min(m));\n            let old_weight = Self::adjust_weight(node, &mut self.used_weight, new_weight);\n            self.order.promote(node.list_index);\n            return Some((old_weight, new_weight));\n        }\n        None\n    }\n\n    pub fn access(&mut self, key: u64) -> bool {\n        if let Some(node) = self.lookup_table.get(&key) {\n            self.order.promote(node.list_index);\n            true\n        } else {\n            false\n        }\n    }\n\n    // Check if a key is already in the top n most recently used nodes.\n    // this is a heuristic to reduce write, which requires exclusive locks, for promotion,\n    // especially on very populate nodes\n    // NOTE: O(n) search here so limit needs to be small\n    pub fn need_promote(&self, key: u64, limit: usize) -> bool {\n        !self.order.exist_near_head(key, limit)\n    }\n\n    // try to evict 1 node\n    pub fn evict(&mut self) -> Option<(T, usize)> {\n        self.order.pop_tail().map(|key| {\n            // unwrap is safe because we always insert in both the hashtable and the list\n            let node = self.lookup_table.remove(&key).unwrap();\n            self.used_weight -= node.weight;\n            (node.data, node.weight)\n        })\n    }\n    // TODO: scan the tail up to K elements to decide which ones to evict\n\n    pub fn remove(&mut self, key: u64) -> Option<(T, usize)> {\n        self.lookup_table.remove(&key).map(|node| {\n            let list_key = self.order.remove(node.list_index);\n            assert_eq!(key, list_key);\n            self.used_weight -= node.weight;\n            (node.data, node.weight)\n        })\n    }\n\n    pub fn insert_tail(&mut self, key: u64, data: T, weight: usize) -> bool {\n        if self.lookup_table.contains_key(&key) {\n            return false;\n        }\n        let list_index = self.order.push_tail(key);\n        let node = Box::new(LruNode {\n            data,\n            list_index,\n            weight,\n        });\n        self.lookup_table.insert(key, node);\n        self.used_weight += weight;\n        true\n    }\n\n    pub fn len(&self) -> usize {\n        assert_eq!(self.lookup_table.len(), self.order.len());\n        self.lookup_table.len()\n    }\n\n    #[cfg(test)]\n    pub fn used_weight(&self) -> usize {\n        self.used_weight\n    }\n\n    pub fn iter(&self) -> LruUnitIter<'_, T> {\n        LruUnitIter {\n            unit: self,\n            iter: self.order.iter(),\n        }\n    }\n\n    // Adjusts node weight to the new given weight.\n    // Returns old weight.\n    #[inline]\n    fn adjust_weight(node: &mut LruNode<T>, used_weight: &mut usize, weight: usize) -> usize {\n        let old_weight = node.weight;\n        if weight != old_weight {\n            *used_weight += weight;\n            *used_weight -= old_weight;\n            node.weight = weight;\n        }\n        old_weight\n    }\n}\n\nstruct LruUnitIter<'a, T> {\n    unit: &'a LruUnit<T>,\n    iter: LinkedListIter<'a>,\n}\n\nimpl<'a, T> Iterator for LruUnitIter<'a, T> {\n    type Item = (&'a T, usize);\n\n    fn next(&mut self) -> Option<Self::Item> {\n        self.iter.next().map(|key| {\n            // safe because we always items in table and list are always 1:1\n            let node = self.unit.lookup_table.get(key).unwrap();\n            (&node.data, node.weight)\n        })\n    }\n\n    fn size_hint(&self) -> (usize, Option<usize>) {\n        self.iter.size_hint()\n    }\n}\n\nimpl<T> DoubleEndedIterator for LruUnitIter<'_, T> {\n    fn next_back(&mut self) -> Option<Self::Item> {\n        self.iter.next_back().map(|key| {\n            // safe because we always items in table and list are always 1:1\n            let node = self.unit.lookup_table.get(key).unwrap();\n            (&node.data, node.weight)\n        })\n    }\n}\n\n#[cfg(test)]\nmod test_lru {\n    use super::*;\n\n    fn assert_lru<T: Copy + PartialEq + std::fmt::Debug, const N: usize>(\n        lru: &Lru<T, N>,\n        values: &[T],\n        shard: usize,\n    ) {\n        let mut list_values = vec![];\n        lru.iter_for_each(shard, |(v, _)| list_values.push(*v));\n        assert_eq!(values, &list_values)\n    }\n\n    #[test]\n    fn test_admit() {\n        let lru = Lru::<_, 2>::with_capacity(30, 10);\n        assert_eq!(lru.len(), 0);\n\n        lru.admit(2, 2, 3);\n        assert_eq!(lru.len(), 1);\n        assert_eq!(lru.weight(), 3);\n\n        lru.admit(2, 2, 1);\n        assert_eq!(lru.len(), 1);\n        assert_eq!(lru.weight(), 1);\n\n        lru.admit(2, 2, 2); // admit again with different weight\n        assert_eq!(lru.len(), 1);\n        assert_eq!(lru.weight(), 2);\n\n        lru.admit(3, 3, 3);\n        lru.admit(4, 4, 4);\n\n        assert_eq!(lru.weight(), 2 + 3 + 4);\n        assert_eq!(lru.len(), 3);\n    }\n\n    #[test]\n    fn test_promote() {\n        let lru = Lru::<_, 2>::with_capacity(30, 10);\n\n        lru.admit(2, 2, 2);\n        lru.admit(3, 3, 3);\n        lru.admit(4, 4, 4);\n        lru.admit(5, 5, 5);\n        lru.admit(6, 6, 6);\n        assert_lru(&lru, &[6, 4, 2], 0);\n        assert_lru(&lru, &[5, 3], 1);\n\n        assert!(lru.promote(3));\n        assert_lru(&lru, &[3, 5], 1);\n        assert!(lru.promote(3));\n        assert_lru(&lru, &[3, 5], 1);\n\n        assert!(lru.promote(2));\n        assert_lru(&lru, &[2, 6, 4], 0);\n\n        assert!(!lru.promote(7)); // 7 doesn't exist\n        assert_lru(&lru, &[2, 6, 4], 0);\n        assert_lru(&lru, &[3, 5], 1);\n\n        // promote 2 to top 1, already there\n        assert!(lru.promote_top_n(2, 1));\n        assert_lru(&lru, &[2, 6, 4], 0);\n\n        // promote 4 to top 3, already there\n        assert!(lru.promote_top_n(4, 3));\n        assert_lru(&lru, &[2, 6, 4], 0);\n\n        // promote 4 to top 2\n        assert!(lru.promote_top_n(4, 2));\n        assert_lru(&lru, &[4, 2, 6], 0);\n\n        // promote 2 to top 1\n        assert!(lru.promote_top_n(2, 1));\n        assert_lru(&lru, &[2, 4, 6], 0);\n\n        assert!(!lru.promote_top_n(7, 1)); // 7 doesn't exist\n    }\n\n    #[test]\n    fn test_evict() {\n        let lru = Lru::<_, 2>::with_capacity(14, 10);\n\n        // same weight to make the random eviction less random\n        lru.admit(2, 2, 2);\n        lru.admit(3, 3, 2);\n        lru.admit(4, 4, 4);\n        lru.admit(5, 5, 4);\n        lru.admit(6, 6, 2);\n        lru.admit(7, 7, 2);\n\n        assert_lru(&lru, &[6, 4, 2], 0);\n        assert_lru(&lru, &[7, 5, 3], 1);\n\n        assert_eq!(lru.weight(), 16);\n        assert_eq!(lru.len(), 6);\n\n        let evicted = lru.evict_to_limit();\n        assert_eq!(lru.weight(), 14);\n        assert_eq!(lru.len(), 5);\n        assert_eq!(lru.evicted_weight(), 2);\n        assert_eq!(lru.evicted_len(), 1);\n        assert_eq!(evicted.len(), 1);\n        assert_eq!(evicted[0].1, 2); //weight\n        assert!(evicted[0].0 == 2 || evicted[0].0 == 3); //either 2 or 3 are evicted\n\n        let lru = Lru::<_, 2>::with_capacity(6, 10);\n\n        // same weight random eviction less random\n        lru.admit(2, 2, 2);\n        lru.admit(3, 3, 2);\n        lru.admit(4, 4, 2);\n        lru.admit(5, 5, 2);\n        lru.admit(6, 6, 2);\n        lru.admit(7, 7, 2);\n        assert_eq!(lru.weight(), 12);\n        assert_eq!(lru.len(), 6);\n\n        let evicted = lru.evict_to_limit();\n        // NOTE: there is a low chance this test would fail see the TODO in evict_to_limit\n        assert_eq!(lru.weight(), 6);\n        assert_eq!(lru.len(), 3);\n        assert_eq!(lru.evicted_weight(), 6);\n        assert_eq!(lru.evicted_len(), 3);\n        assert_eq!(evicted.len(), 3);\n    }\n\n    #[test]\n    fn test_increment_weight() {\n        let lru = Lru::<_, 2>::with_capacity(6, 10);\n        lru.admit(1, 1, 1);\n        lru.increment_weight(1, 1, None);\n        assert_eq!(lru.weight(), 1 + 1);\n\n        lru.increment_weight(0, 1000, None);\n        assert_eq!(lru.weight(), 1 + 1);\n\n        lru.admit(2, 2, 2);\n        lru.increment_weight(2, 2, None);\n        assert_eq!(lru.weight(), 1 + 1 + 2 + 2);\n\n        lru.increment_weight(2, 2, Some(3));\n        assert_eq!(lru.weight(), 1 + 1 + 3);\n    }\n\n    #[test]\n    fn test_remove() {\n        let lru = Lru::<_, 2>::with_capacity(30, 10);\n        lru.admit(2, 2, 2);\n        lru.admit(3, 3, 3);\n        lru.admit(4, 4, 4);\n        lru.admit(5, 5, 5);\n        lru.admit(6, 6, 6);\n\n        assert_eq!(lru.weight(), 2 + 3 + 4 + 5 + 6);\n        assert_eq!(lru.len(), 5);\n        assert_lru(&lru, &[6, 4, 2], 0);\n        assert_lru(&lru, &[5, 3], 1);\n\n        let node = lru.remove(6).unwrap();\n        assert_eq!(node.0, 6); // data\n        assert_eq!(node.1, 6); // weight\n        assert_eq!(lru.weight(), 2 + 3 + 4 + 5);\n        assert_eq!(lru.len(), 4);\n        assert_lru(&lru, &[4, 2], 0);\n\n        let node = lru.remove(3).unwrap();\n        assert_eq!(node.0, 3); // data\n        assert_eq!(node.1, 3); // weight\n        assert_eq!(lru.weight(), 2 + 4 + 5);\n        assert_eq!(lru.len(), 3);\n        assert_lru(&lru, &[5], 1);\n\n        assert!(lru.remove(7).is_none());\n    }\n\n    #[test]\n    fn test_peek() {\n        let lru = Lru::<_, 2>::with_capacity(30, 10);\n        lru.admit(2, 2, 2);\n        lru.admit(3, 3, 3);\n        lru.admit(4, 4, 4);\n\n        assert!(lru.peek(4));\n        assert!(lru.peek(3));\n        assert!(lru.peek(2));\n\n        assert_lru(&lru, &[4, 2], 0);\n        assert_lru(&lru, &[3], 1);\n    }\n\n    #[test]\n    fn test_insert_tail() {\n        let lru = Lru::<_, 2>::with_capacity(30, 10);\n        lru.admit(2, 2, 2);\n        lru.admit(3, 3, 3);\n        lru.admit(4, 4, 4);\n        lru.admit(5, 5, 5);\n        lru.admit(6, 6, 6);\n\n        assert_eq!(lru.weight(), 2 + 3 + 4 + 5 + 6);\n        assert_eq!(lru.len(), 5);\n        assert_lru(&lru, &[6, 4, 2], 0);\n        assert_lru(&lru, &[5, 3], 1);\n\n        assert!(lru.insert_tail(7, 7, 7));\n        assert_eq!(lru.weight(), 2 + 3 + 4 + 5 + 6 + 7);\n        assert_eq!(lru.len(), 6);\n        assert_lru(&lru, &[5, 3, 7], 1);\n\n        // ignore existing ones\n        assert!(!lru.insert_tail(6, 6, 7));\n    }\n\n    #[test]\n    fn test_watermark_eviction() {\n        const WEIGHT_LIMIT: usize = usize::MAX / 2;\n        let lru = Lru::<u64, 2>::with_capacity_and_watermark(WEIGHT_LIMIT, 10, Some(4));\n\n        // admit 6 items, each weight 1\n        for k in [2u64, 3, 4, 5, 6, 7] {\n            lru.admit(k, k, 1);\n        }\n\n        assert!(lru.weight() < WEIGHT_LIMIT);\n        assert_eq!(lru.len(), 6);\n\n        let evicted = lru.evict_to_limit();\n        assert_eq!(lru.len(), 4);\n        assert_eq!(evicted.len(), 2);\n        assert_eq!(lru.evicted_len(), 2);\n    }\n}\n\n#[cfg(test)]\nmod test_lru_unit {\n    use super::*;\n\n    fn assert_lru<T: Copy + PartialEq + std::fmt::Debug>(lru: &LruUnit<T>, values: &[T]) {\n        let list_values: Vec<_> = lru.iter().map(|(v, _)| *v).collect();\n        assert_eq!(values, &list_values)\n    }\n\n    #[test]\n    fn test_admit() {\n        let mut lru = LruUnit::with_capacity(10);\n        assert_eq!(lru.len(), 0);\n        assert!(lru.peek(0).is_none());\n\n        lru.admit(2, 2, 1);\n        assert_eq!(lru.len(), 1);\n        assert_eq!(lru.peek(2).unwrap(), &2);\n        assert_eq!(lru.used_weight(), 1);\n\n        lru.admit(2, 2, 2); // admit again with different weight\n        assert_eq!(lru.used_weight(), 2);\n\n        lru.admit(3, 3, 3);\n        lru.admit(4, 4, 4);\n\n        assert_eq!(lru.used_weight(), 2 + 3 + 4);\n        assert_lru(&lru, &[4, 3, 2]);\n    }\n\n    #[test]\n    fn test_access() {\n        let mut lru = LruUnit::with_capacity(10);\n\n        lru.admit(2, 2, 2);\n        lru.admit(3, 3, 3);\n        lru.admit(4, 4, 4);\n        assert_lru(&lru, &[4, 3, 2]);\n\n        assert!(lru.access(3));\n        assert_lru(&lru, &[3, 4, 2]);\n        assert!(lru.access(3));\n        assert_lru(&lru, &[3, 4, 2]);\n        assert!(lru.access(2));\n        assert_lru(&lru, &[2, 3, 4]);\n\n        assert!(!lru.access(5)); // 5 doesn't exist\n        assert_lru(&lru, &[2, 3, 4]);\n\n        assert!(!lru.need_promote(2, 1));\n        assert!(lru.need_promote(3, 1));\n        assert!(!lru.need_promote(4, 9999));\n    }\n\n    #[test]\n    fn test_evict() {\n        let mut lru = LruUnit::with_capacity(10);\n\n        lru.admit(2, 2, 2);\n        lru.admit(3, 3, 3);\n        lru.admit(4, 4, 4);\n        assert_lru(&lru, &[4, 3, 2]);\n\n        assert!(lru.access(3));\n        assert!(lru.access(3));\n        assert!(lru.access(2));\n        assert_lru(&lru, &[2, 3, 4]);\n\n        assert_eq!(lru.used_weight(), 2 + 3 + 4);\n        assert_eq!(lru.evict(), Some((4, 4)));\n        assert_eq!(lru.used_weight(), 2 + 3);\n        assert_lru(&lru, &[2, 3]);\n\n        assert_eq!(lru.evict(), Some((3, 3)));\n        assert_eq!(lru.used_weight(), 2);\n        assert_lru(&lru, &[2]);\n\n        assert_eq!(lru.evict(), Some((2, 2)));\n        assert_eq!(lru.used_weight(), 0);\n        assert_lru(&lru, &[]);\n\n        assert_eq!(lru.evict(), None);\n        assert_eq!(lru.used_weight(), 0);\n        assert_lru(&lru, &[]);\n    }\n\n    #[test]\n    fn test_increment_weight() {\n        let mut lru = LruUnit::with_capacity(10);\n        lru.admit(1, 1, 1);\n        lru.increment_weight(1, 1, None);\n        assert_eq!(lru.used_weight(), 1 + 1);\n\n        lru.increment_weight(0, 1000, None);\n        assert_eq!(lru.used_weight(), 1 + 1);\n\n        lru.admit(2, 2, 2);\n        lru.increment_weight(2, 2, None);\n        assert_eq!(lru.used_weight(), 1 + 1 + 2 + 2);\n\n        lru.admit(3, 3, 3);\n        lru.increment_weight(3, 3, Some(5));\n        assert_eq!(lru.used_weight(), 1 + 1 + 2 + 2 + 3 + 2);\n\n        lru.increment_weight(3, 3, Some(3));\n        assert_eq!(lru.used_weight(), 1 + 1 + 2 + 2 + 3);\n    }\n\n    #[test]\n    fn test_remove() {\n        let mut lru = LruUnit::with_capacity(10);\n\n        lru.admit(2, 2, 2);\n        lru.admit(3, 3, 3);\n        lru.admit(4, 4, 4);\n        lru.admit(5, 5, 5);\n        assert_lru(&lru, &[5, 4, 3, 2]);\n\n        assert!(lru.access(4));\n        assert!(lru.access(3));\n        assert!(lru.access(3));\n        assert!(lru.access(2));\n        assert_lru(&lru, &[2, 3, 4, 5]);\n\n        assert_eq!(lru.used_weight(), 2 + 3 + 4 + 5);\n        assert_eq!(lru.remove(2), Some((2, 2)));\n        assert_eq!(lru.used_weight(), 3 + 4 + 5);\n        assert_lru(&lru, &[3, 4, 5]);\n\n        assert_eq!(lru.remove(4), Some((4, 4)));\n        assert_eq!(lru.used_weight(), 3 + 5);\n        assert_lru(&lru, &[3, 5]);\n\n        assert_eq!(lru.remove(5), Some((5, 5)));\n        assert_eq!(lru.used_weight(), 3);\n        assert_lru(&lru, &[3]);\n\n        assert_eq!(lru.remove(1), None);\n        assert_eq!(lru.used_weight(), 3);\n        assert_lru(&lru, &[3]);\n\n        assert_eq!(lru.remove(3), Some((3, 3)));\n        assert_eq!(lru.used_weight(), 0);\n        assert_lru(&lru, &[]);\n    }\n\n    #[test]\n    fn test_insert_tail() {\n        let mut lru = LruUnit::with_capacity(10);\n        assert_eq!(lru.len(), 0);\n        assert!(lru.peek(0).is_none());\n\n        assert!(lru.insert_tail(2, 2, 1));\n        assert_eq!(lru.len(), 1);\n        assert_eq!(lru.peek(2).unwrap(), &2);\n        assert_eq!(lru.used_weight(), 1);\n\n        assert!(!lru.insert_tail(2, 2, 2));\n        assert!(lru.insert_tail(3, 3, 3));\n        assert_eq!(lru.used_weight(), 1 + 3);\n        assert_lru(&lru, &[2, 3]);\n\n        assert!(lru.insert_tail(4, 4, 4));\n        assert!(lru.insert_tail(5, 5, 5));\n        assert_eq!(lru.used_weight(), 1 + 3 + 4 + 5);\n        assert_lru(&lru, &[2, 3, 4, 5]);\n    }\n}\n"
  },
  {
    "path": "pingora-lru/src/linked_list.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n// Can't tell people you know Rust until you write a (doubly) linked list\n\n//! Doubly linked list\n//!\n//! Features\n//! - Preallocate consecutive memory, no memory fragmentation.\n//! - No shrink function: for Lru cache that grows to a certain size but never shrinks.\n//! - Relatively fast and efficient.\n\n// inspired by clru::FixedSizeList (Élie!)\n\nuse std::mem::replace;\n\ntype Index = usize;\nconst NULL: Index = usize::MAX;\nconst HEAD: Index = 0;\nconst TAIL: Index = 1;\nconst OFFSET: usize = 2;\n\n#[derive(Debug)]\nstruct Node {\n    pub(crate) prev: Index,\n    pub(crate) next: Index,\n    pub(crate) data: u64,\n}\n\n// Functionally the same as vec![head, tail, data_nodes...] where head & tail are fixed and\n// the rest data nodes can expand. Both head and tail can be accessed faster than using index\nstruct Nodes {\n    // we use these sentinel nodes to guard the head and tail of the list so that list\n    // manipulation is simpler (fewer if-else)\n    head: Node,\n    tail: Node,\n    data_nodes: Vec<Node>,\n}\n\nimpl Nodes {\n    fn with_capacity(capacity: usize) -> Self {\n        Nodes {\n            head: Node {\n                prev: NULL,\n                next: TAIL,\n                data: 0,\n            },\n            tail: Node {\n                prev: HEAD,\n                next: NULL,\n                data: 0,\n            },\n            data_nodes: Vec::with_capacity(capacity),\n        }\n    }\n\n    fn new_node(&mut self, data: u64) -> Index {\n        const VEC_EXP_GROWTH_CAP: usize = 65536;\n        let node = Node {\n            prev: NULL,\n            next: NULL,\n            data,\n        };\n        // Constrain the growth of vec: vec always double its capacity when it needs to grow.\n        // It could waste too much memory when it is already very large.\n        // Here we limit the memory waste to 10% once it grows beyond the cap.\n        // The amortized growth cost is O(n) beyond the max of the initially reserved capacity and\n        // the cap. But this list is for limited sized LRU and we recycle released node, so\n        // hopefully insertions are rare beyond certain sizes\n        if self.data_nodes.capacity() > VEC_EXP_GROWTH_CAP\n            && self.data_nodes.capacity() - self.data_nodes.len() < 2\n        {\n            self.data_nodes\n                .reserve_exact(self.data_nodes.capacity() / 10)\n        }\n        self.data_nodes.push(node);\n        self.data_nodes.len() - 1 + OFFSET\n    }\n\n    fn len(&self) -> usize {\n        self.data_nodes.len()\n    }\n\n    fn head(&self) -> &Node {\n        &self.head\n    }\n\n    fn tail(&self) -> &Node {\n        &self.tail\n    }\n}\n\nimpl std::ops::Index<usize> for Nodes {\n    type Output = Node;\n\n    fn index(&self, index: usize) -> &Self::Output {\n        match index {\n            HEAD => &self.head,\n            TAIL => &self.tail,\n            _ => &self.data_nodes[index - OFFSET],\n        }\n    }\n}\n\nimpl std::ops::IndexMut<usize> for Nodes {\n    fn index_mut(&mut self, index: usize) -> &mut Self::Output {\n        match index {\n            HEAD => &mut self.head,\n            TAIL => &mut self.tail,\n            _ => &mut self.data_nodes[index - OFFSET],\n        }\n    }\n}\n\n/// Doubly linked list\npub struct LinkedList {\n    nodes: Nodes,\n    free: Vec<Index>, // to keep track of freed node to be used again\n}\n// Panic when index used as parameters are invalid\n// Index returned by push_* is always valid.\nimpl LinkedList {\n    /// Create a [LinkedList] with the given predicted capacity.\n    pub fn with_capacity(capacity: usize) -> Self {\n        LinkedList {\n            nodes: Nodes::with_capacity(capacity),\n            free: vec![],\n        }\n    }\n\n    // Allocate a new node and return its index\n    // NOTE: this node is leaked if not used by caller\n    fn new_node(&mut self, data: u64) -> Index {\n        if let Some(index) = self.free.pop() {\n            // have a free node, update its payload and return its index\n            self.nodes[index].data = data;\n            index\n        } else {\n            // create a new node\n            self.nodes.new_node(data)\n        }\n    }\n\n    /// How many nodes in the list\n    #[allow(clippy::len_without_is_empty)]\n    pub fn len(&self) -> usize {\n        // exclude the 2 sentinels\n        self.nodes.len() - self.free.len()\n    }\n\n    fn valid_index(&self, index: Index) -> bool {\n        index != HEAD && index != TAIL && index < self.nodes.len() + OFFSET\n        // TODO: check node prev/next not NULL\n        // TODO: debug_check index not in self.free\n    }\n\n    fn node(&self, index: Index) -> Option<&Node> {\n        if self.valid_index(index) {\n            Some(&self.nodes[index])\n        } else {\n            None\n        }\n    }\n\n    /// Peek into the list\n    pub fn peek(&self, index: Index) -> Option<u64> {\n        self.node(index).map(|n| n.data)\n    }\n\n    // safe because the index still needs to be in the range of the vec\n    fn peek_unchecked(&self, index: Index) -> &u64 {\n        &self.nodes[index].data\n    }\n\n    /// Whether the value exists closed (up to search_limit nodes) to the head of the list\n    // It can be done via iter().take().find() but this is cheaper\n    pub fn exist_near_head(&self, value: u64, search_limit: usize) -> bool {\n        let mut current_node = HEAD;\n        for _ in 0..search_limit {\n            current_node = self.nodes[current_node].next;\n            if current_node == TAIL {\n                return false;\n            }\n            if self.nodes[current_node].data == value {\n                return true;\n            }\n        }\n        false\n    }\n\n    // put a node right after the node at `at`\n    fn insert_after(&mut self, node_index: Index, at: Index) {\n        assert!(at != TAIL && at != node_index); // can't insert after tail or to itself\n\n        let next = replace(&mut self.nodes[at].next, node_index);\n\n        let node = &mut self.nodes[node_index];\n        node.next = next;\n        node.prev = at;\n\n        self.nodes[next].prev = node_index;\n    }\n\n    /// Put the data at the head of the list.\n    pub fn push_head(&mut self, data: u64) -> Index {\n        let new_node_index = self.new_node(data);\n        self.insert_after(new_node_index, HEAD);\n        new_node_index\n    }\n\n    /// Put the data at the tail of the list.\n    pub fn push_tail(&mut self, data: u64) -> Index {\n        let new_node_index = self.new_node(data);\n        self.insert_after(new_node_index, self.nodes.tail().prev);\n        new_node_index\n    }\n\n    // lift the node out of the linked list, to either delete it or insert to another place\n    // NOTE: the node is leaked if not used by the caller\n    fn lift(&mut self, index: Index) -> u64 {\n        // can't touch the sentinels\n        assert!(index != HEAD && index != TAIL);\n\n        let node = &mut self.nodes[index];\n\n        // zero out the pointers, useful in case we try to access a freed node\n        let prev = replace(&mut node.prev, NULL);\n        let next = replace(&mut node.next, NULL);\n        let data = node.data;\n\n        // make sure we are accessing a node in the list, not freed already\n        assert!(prev != NULL && next != NULL);\n\n        self.nodes[prev].next = next;\n        self.nodes[next].prev = prev;\n\n        data\n    }\n\n    /// Remove the node at the index, and return the value\n    pub fn remove(&mut self, index: Index) -> u64 {\n        self.free.push(index);\n        self.lift(index)\n    }\n\n    /// Remove the tail of the list\n    pub fn pop_tail(&mut self) -> Option<u64> {\n        let data_tail = self.nodes.tail().prev;\n        if data_tail == HEAD {\n            None // empty list\n        } else {\n            Some(self.remove(data_tail))\n        }\n    }\n\n    /// Put the node at the index to the head\n    pub fn promote(&mut self, index: Index) {\n        if self.nodes.head().next == index {\n            return; // already head\n        }\n        self.lift(index);\n        self.insert_after(index, HEAD);\n    }\n\n    fn next(&self, index: Index) -> Index {\n        self.nodes[index].next\n    }\n\n    fn prev(&self, index: Index) -> Index {\n        self.nodes[index].prev\n    }\n\n    /// Get the head of the list\n    pub fn head(&self) -> Option<Index> {\n        let data_head = self.nodes.head().next;\n        if data_head == TAIL {\n            None\n        } else {\n            Some(data_head)\n        }\n    }\n\n    /// Get the tail of the list\n    pub fn tail(&self) -> Option<Index> {\n        let data_tail = self.nodes.tail().prev;\n        if data_tail == HEAD {\n            None\n        } else {\n            Some(data_tail)\n        }\n    }\n\n    /// Iterate over the list\n    pub fn iter(&self) -> LinkedListIter<'_> {\n        LinkedListIter {\n            list: self,\n            head: HEAD,\n            tail: TAIL,\n            len: self.len(),\n        }\n    }\n}\n\n/// The iter over the list\npub struct LinkedListIter<'a> {\n    list: &'a LinkedList,\n    head: Index,\n    tail: Index,\n    len: usize,\n}\n\nimpl<'a> Iterator for LinkedListIter<'a> {\n    type Item = &'a u64;\n\n    fn next(&mut self) -> Option<Self::Item> {\n        let next_index = self.list.next(self.head);\n        if next_index == TAIL || next_index == NULL {\n            None\n        } else {\n            self.head = next_index;\n            self.len -= 1;\n            Some(self.list.peek_unchecked(next_index))\n        }\n    }\n\n    fn size_hint(&self) -> (usize, Option<usize>) {\n        (self.len, Some(self.len))\n    }\n}\n\nimpl DoubleEndedIterator for LinkedListIter<'_> {\n    fn next_back(&mut self) -> Option<Self::Item> {\n        let prev_index = self.list.prev(self.tail);\n        if prev_index == HEAD || prev_index == NULL {\n            None\n        } else {\n            self.tail = prev_index;\n            self.len -= 1;\n            Some(self.list.peek_unchecked(prev_index))\n        }\n    }\n}\n\n#[cfg(test)]\nmod test {\n    use super::*;\n\n    // assert the list is the same as `values`\n    fn assert_list(list: &LinkedList, values: &[u64]) {\n        let list_values: Vec<_> = list.iter().copied().collect();\n        assert_eq!(values, &list_values)\n    }\n\n    fn assert_list_reverse(list: &LinkedList, values: &[u64]) {\n        let list_values: Vec<_> = list.iter().rev().copied().collect();\n        assert_eq!(values, &list_values)\n    }\n\n    #[test]\n    fn test_insert() {\n        let mut list = LinkedList::with_capacity(10);\n        assert_eq!(list.len(), 0);\n        assert!(list.node(2).is_none());\n        assert_eq!(list.head(), None);\n        assert_eq!(list.tail(), None);\n\n        let index1 = list.push_head(2);\n        assert_eq!(list.len(), 1);\n        assert_eq!(list.peek(index1).unwrap(), 2);\n\n        let index2 = list.push_head(3);\n        assert_eq!(list.head(), Some(index2));\n        assert_eq!(list.tail(), Some(index1));\n\n        let index3 = list.push_tail(4);\n        assert_eq!(list.head(), Some(index2));\n        assert_eq!(list.tail(), Some(index3));\n\n        assert_list(&list, &[3, 2, 4]);\n        assert_list_reverse(&list, &[4, 2, 3]);\n    }\n\n    #[test]\n    fn test_pop() {\n        let mut list = LinkedList::with_capacity(10);\n        list.push_head(2);\n        list.push_head(3);\n        list.push_tail(4);\n        assert_list(&list, &[3, 2, 4]);\n        assert_eq!(list.pop_tail(), Some(4));\n        assert_eq!(list.pop_tail(), Some(2));\n        assert_eq!(list.pop_tail(), Some(3));\n        assert_eq!(list.pop_tail(), None);\n    }\n\n    #[test]\n    fn test_promote() {\n        let mut list = LinkedList::with_capacity(10);\n        let index2 = list.push_head(2);\n        let index3 = list.push_head(3);\n        let index4 = list.push_tail(4);\n        assert_list(&list, &[3, 2, 4]);\n\n        list.promote(index3);\n        assert_list(&list, &[3, 2, 4]);\n\n        list.promote(index2);\n        assert_list(&list, &[2, 3, 4]);\n\n        list.promote(index4);\n        assert_list(&list, &[4, 2, 3]);\n    }\n\n    #[test]\n    fn test_exist_near_head() {\n        let mut list = LinkedList::with_capacity(10);\n        list.push_head(2);\n        list.push_head(3);\n        list.push_tail(4);\n        assert_list(&list, &[3, 2, 4]);\n\n        assert!(!list.exist_near_head(4, 1));\n        assert!(!list.exist_near_head(4, 2));\n        assert!(list.exist_near_head(4, 3));\n        assert!(list.exist_near_head(4, 4));\n        assert!(list.exist_near_head(4, 99999));\n    }\n}\n"
  },
  {
    "path": "pingora-memory-cache/Cargo.toml",
    "content": "[package]\nname = \"pingora-memory-cache\"\nversion = \"0.8.0\"\nauthors = [\"Yuchen Wu <yuchen@cloudflare.com>\"]\nlicense = \"Apache-2.0\"\nedition = \"2021\"\nrepository = \"https://github.com/cloudflare/pingora\"\ncategories = [\"algorithms\", \"caching\"]\nkeywords = [\"async\", \"cache\", \"pingora\"]\ndescription = \"\"\"\nAn async in-memory cache with cache stampede protection.\n\"\"\"\n\n# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html\n[lib]\nname = \"pingora_memory_cache\"\npath = \"src/lib.rs\"\n\n[dependencies]\nTinyUFO = { version = \"0.8.0\", path = \"../tinyufo\" }\nahash = { workspace = true }\ntokio = { workspace = true, features = [\"sync\"] }\nasync-trait = { workspace = true }\npingora-error = { version = \"0.8.0\", path = \"../pingora-error\" }\nlog = { workspace = true }\nparking_lot = \"0\"\npingora-timeout = { version = \"0.8.0\", path = \"../pingora-timeout\" }\n\n[dev-dependencies]\nonce_cell = { workspace = true }\n"
  },
  {
    "path": "pingora-memory-cache/LICENSE",
    "content": "\n                                 Apache License\n                           Version 2.0, January 2004\n                        http://www.apache.org/licenses/\n\n   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION\n\n   1. Definitions.\n\n      \"License\" shall mean the terms and conditions for use, reproduction,\n      and distribution as defined by Sections 1 through 9 of this document.\n\n      \"Licensor\" shall mean the copyright owner or entity authorized by\n      the copyright owner that is granting the License.\n\n      \"Legal Entity\" shall mean the union of the acting entity and all\n      other entities that control, are controlled by, or are under common\n      control with that entity. For the purposes of this definition,\n      \"control\" means (i) the power, direct or indirect, to cause the\n      direction or management of such entity, whether by contract or\n      otherwise, or (ii) ownership of fifty percent (50%) or more of the\n      outstanding shares, or (iii) beneficial ownership of such entity.\n\n      \"You\" (or \"Your\") shall mean an individual or Legal Entity\n      exercising permissions granted by this License.\n\n      \"Source\" form shall mean the preferred form for making modifications,\n      including but not limited to software source code, documentation\n      source, and configuration files.\n\n      \"Object\" form shall mean any form resulting from mechanical\n      transformation or translation of a Source form, including but\n      not limited to compiled object code, generated documentation,\n      and conversions to other media types.\n\n      \"Work\" shall mean the work of authorship, whether in Source or\n      Object form, made available under the License, as indicated by a\n      copyright notice that is included in or attached to the work\n      (an example is provided in the Appendix below).\n\n      \"Derivative Works\" shall mean any work, whether in Source or Object\n      form, that is based on (or derived from) the Work and for which the\n      editorial revisions, annotations, elaborations, or other modifications\n      represent, as a whole, an original work of authorship. For the purposes\n      of this License, Derivative Works shall not include works that remain\n      separable from, or merely link (or bind by name) to the interfaces of,\n      the Work and Derivative Works thereof.\n\n      \"Contribution\" shall mean any work of authorship, including\n      the original version of the Work and any modifications or additions\n      to that Work or Derivative Works thereof, that is intentionally\n      submitted to Licensor for inclusion in the Work by the copyright owner\n      or by an individual or Legal Entity authorized to submit on behalf of\n      the copyright owner. For the purposes of this definition, \"submitted\"\n      means any form of electronic, verbal, or written communication sent\n      to the Licensor or its representatives, including but not limited to\n      communication on electronic mailing lists, source code control systems,\n      and issue tracking systems that are managed by, or on behalf of, the\n      Licensor for the purpose of discussing and improving the Work, but\n      excluding communication that is conspicuously marked or otherwise\n      designated in writing by the copyright owner as \"Not a Contribution.\"\n\n      \"Contributor\" shall mean Licensor and any individual or Legal Entity\n      on behalf of whom a Contribution has been received by Licensor and\n      subsequently incorporated within the Work.\n\n   2. Grant of Copyright License. Subject to the terms and conditions of\n      this License, each Contributor hereby grants to You a perpetual,\n      worldwide, non-exclusive, no-charge, royalty-free, irrevocable\n      copyright license to reproduce, prepare Derivative Works of,\n      publicly display, publicly perform, sublicense, and distribute the\n      Work and such Derivative Works in Source or Object form.\n\n   3. Grant of Patent License. Subject to the terms and conditions of\n      this License, each Contributor hereby grants to You a perpetual,\n      worldwide, non-exclusive, no-charge, royalty-free, irrevocable\n      (except as stated in this section) patent license to make, have made,\n      use, offer to sell, sell, import, and otherwise transfer the Work,\n      where such license applies only to those patent claims licensable\n      by such Contributor that are necessarily infringed by their\n      Contribution(s) alone or by combination of their Contribution(s)\n      with the Work to which such Contribution(s) was submitted. If You\n      institute patent litigation against any entity (including a\n      cross-claim or counterclaim in a lawsuit) alleging that the Work\n      or a Contribution incorporated within the Work constitutes direct\n      or contributory patent infringement, then any patent licenses\n      granted to You under this License for that Work shall terminate\n      as of the date such litigation is filed.\n\n   4. Redistribution. You may reproduce and distribute copies of the\n      Work or Derivative Works thereof in any medium, with or without\n      modifications, and in Source or Object form, provided that You\n      meet the following conditions:\n\n      (a) You must give any other recipients of the Work or\n          Derivative Works a copy of this License; and\n\n      (b) You must cause any modified files to carry prominent notices\n          stating that You changed the files; and\n\n      (c) You must retain, in the Source form of any Derivative Works\n          that You distribute, all copyright, patent, trademark, and\n          attribution notices from the Source form of the Work,\n          excluding those notices that do not pertain to any part of\n          the Derivative Works; and\n\n      (d) If the Work includes a \"NOTICE\" text file as part of its\n          distribution, then any Derivative Works that You distribute must\n          include a readable copy of the attribution notices contained\n          within such NOTICE file, excluding those notices that do not\n          pertain to any part of the Derivative Works, in at least one\n          of the following places: within a NOTICE text file distributed\n          as part of the Derivative Works; within the Source form or\n          documentation, if provided along with the Derivative Works; or,\n          within a display generated by the Derivative Works, if and\n          wherever such third-party notices normally appear. The contents\n          of the NOTICE file are for informational purposes only and\n          do not modify the License. You may add Your own attribution\n          notices within Derivative Works that You distribute, alongside\n          or as an addendum to the NOTICE text from the Work, provided\n          that such additional attribution notices cannot be construed\n          as modifying the License.\n\n      You may add Your own copyright statement to Your modifications and\n      may provide additional or different license terms and conditions\n      for use, reproduction, or distribution of Your modifications, or\n      for any such Derivative Works as a whole, provided Your use,\n      reproduction, and distribution of the Work otherwise complies with\n      the conditions stated in this License.\n\n   5. Submission of Contributions. Unless You explicitly state otherwise,\n      any Contribution intentionally submitted for inclusion in the Work\n      by You to the Licensor shall be under the terms and conditions of\n      this License, without any additional terms or conditions.\n      Notwithstanding the above, nothing herein shall supersede or modify\n      the terms of any separate license agreement you may have executed\n      with Licensor regarding such Contributions.\n\n   6. Trademarks. This License does not grant permission to use the trade\n      names, trademarks, service marks, or product names of the Licensor,\n      except as required for reasonable and customary use in describing the\n      origin of the Work and reproducing the content of the NOTICE file.\n\n   7. Disclaimer of Warranty. Unless required by applicable law or\n      agreed to in writing, Licensor provides the Work (and each\n      Contributor provides its Contributions) on an \"AS IS\" BASIS,\n      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or\n      implied, including, without limitation, any warranties or conditions\n      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A\n      PARTICULAR PURPOSE. You are solely responsible for determining the\n      appropriateness of using or redistributing the Work and assume any\n      risks associated with Your exercise of permissions under this License.\n\n   8. Limitation of Liability. In no event and under no legal theory,\n      whether in tort (including negligence), contract, or otherwise,\n      unless required by applicable law (such as deliberate and grossly\n      negligent acts) or agreed to in writing, shall any Contributor be\n      liable to You for damages, including any direct, indirect, special,\n      incidental, or consequential damages of any character arising as a\n      result of this License or out of the use or inability to use the\n      Work (including but not limited to damages for loss of goodwill,\n      work stoppage, computer failure or malfunction, or any and all\n      other commercial damages or losses), even if such Contributor\n      has been advised of the possibility of such damages.\n\n   9. Accepting Warranty or Additional Liability. While redistributing\n      the Work or Derivative Works thereof, You may choose to offer,\n      and charge a fee for, acceptance of support, warranty, indemnity,\n      or other liability obligations and/or rights consistent with this\n      License. However, in accepting such obligations, You may act only\n      on Your own behalf and on Your sole responsibility, not on behalf\n      of any other Contributor, and only if You agree to indemnify,\n      defend, and hold each Contributor harmless for any liability\n      incurred by, or claims asserted against, such Contributor by reason\n      of your accepting any such warranty or additional liability.\n\n   END OF TERMS AND CONDITIONS\n\n   APPENDIX: How to apply the Apache License to your work.\n\n      To apply the Apache License to your work, attach the following\n      boilerplate notice, with the fields enclosed by brackets \"[]\"\n      replaced with your own identifying information. (Don't include\n      the brackets!)  The text should be enclosed in the appropriate\n      comment syntax for the file format. We also recommend that a\n      file or class name and description of purpose be included on the\n      same \"printed page\" as the copyright notice for easier\n      identification within third-party archives.\n\n   Copyright [yyyy] [name of copyright owner]\n\n   Licensed under the Apache License, Version 2.0 (the \"License\");\n   you may not use this file except in compliance with the License.\n   You may obtain a copy of the License at\n\n       http://www.apache.org/licenses/LICENSE-2.0\n\n   Unless required by applicable law or agreed to in writing, software\n   distributed under the License is distributed on an \"AS IS\" BASIS,\n   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n   See the License for the specific language governing permissions and\n   limitations under the License.\n"
  },
  {
    "path": "pingora-memory-cache/src/lib.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\nuse ahash::RandomState;\nuse std::borrow::Borrow;\nuse std::hash::Hash;\nuse std::marker::PhantomData;\nuse std::time::{Duration, Instant};\n\nuse tinyufo::TinyUfo;\n\nmod read_through;\npub use read_through::{Lookup, MultiLookup, RTCache};\n\n#[derive(Debug, PartialEq, Eq)]\n/// [CacheStatus] indicates the response type for a query.\npub enum CacheStatus {\n    /// The key was found in the cache\n    Hit,\n    /// The key was not found.\n    Miss,\n    /// The key was found but it was expired.\n    Expired,\n    /// The key was not initially found but was found after awaiting a lock.\n    LockHit,\n    /// The returned value was expired but still returned. The [Duration] is\n    /// how long it has been since its expiration time.\n    Stale(Duration),\n}\n\nimpl CacheStatus {\n    /// Return the string representation for [CacheStatus].\n    pub fn as_str(&self) -> &str {\n        match self {\n            Self::Hit => \"hit\",\n            Self::Miss => \"miss\",\n            Self::Expired => \"expired\",\n            Self::LockHit => \"lock_hit\",\n            Self::Stale(_) => \"stale\",\n        }\n    }\n\n    /// Returns whether this status represents a cache hit.\n    pub fn is_hit(&self) -> bool {\n        match self {\n            CacheStatus::Hit | CacheStatus::LockHit | CacheStatus::Stale(_) => true,\n            CacheStatus::Miss | CacheStatus::Expired => false,\n        }\n    }\n\n    /// Returns the stale duration if any\n    pub fn stale(&self) -> Option<Duration> {\n        match self {\n            CacheStatus::Stale(time) => Some(*time),\n            _ => None,\n        }\n    }\n}\n\n#[derive(Debug, Clone)]\nstruct Node<T: Clone> {\n    pub value: T,\n    expire_on: Option<Instant>,\n}\n\nimpl<T: Clone> Node<T> {\n    fn new(value: T, ttl: Option<Duration>) -> Self {\n        let expire_on = match ttl {\n            Some(t) => Instant::now().checked_add(t),\n            None => None,\n        };\n        Node { value, expire_on }\n    }\n\n    fn will_expire_at(&self, time: &Instant) -> bool {\n        self.stale_duration(time).is_some()\n    }\n\n    fn is_expired(&self) -> bool {\n        self.will_expire_at(&Instant::now())\n    }\n\n    fn stale_duration(&self, time: &Instant) -> Option<Duration> {\n        let expire_time = self.expire_on?;\n        if &expire_time <= time {\n            Some(time.duration_since(expire_time))\n        } else {\n            None\n        }\n    }\n}\n\n/// A high performant in-memory cache with S3-FIFO + TinyLFU\npub struct MemoryCache<K: Hash, T: Clone> {\n    store: TinyUfo<u64, Node<T>>,\n    _key_type: PhantomData<K>,\n    pub(crate) hasher: RandomState,\n}\n\nimpl<K: Hash, T: Clone + Send + Sync + 'static> MemoryCache<K, T> {\n    /// Create a new [MemoryCache] with the given size.\n    pub fn new(size: usize) -> Self {\n        MemoryCache {\n            store: TinyUfo::new(size, size),\n            _key_type: PhantomData,\n            hasher: RandomState::new(),\n        }\n    }\n\n    /// Fetch the key and return its value in addition to a [CacheStatus].\n    pub fn get<Q>(&self, key: &Q) -> (Option<T>, CacheStatus)\n    where\n        K: Borrow<Q>,\n        Q: Hash + ?Sized,\n    {\n        let hashed_key = self.hasher.hash_one(key);\n\n        if let Some(n) = self.store.get(&hashed_key) {\n            if !n.is_expired() {\n                (Some(n.value), CacheStatus::Hit)\n            } else {\n                (None, CacheStatus::Expired)\n            }\n        } else {\n            (None, CacheStatus::Miss)\n        }\n    }\n\n    /// Similar to [Self::get], fetch the key and return its value in addition to a\n    /// [CacheStatus] but also return the value even if it is expired. When the\n    /// value is expired, the [Duration] of how long it has been stale will\n    /// also be returned.\n    pub fn get_stale<Q>(&self, key: &Q) -> (Option<T>, CacheStatus)\n    where\n        K: Borrow<Q>,\n        Q: Hash + ?Sized,\n    {\n        let hashed_key = self.hasher.hash_one(key);\n\n        if let Some(n) = self.store.get(&hashed_key) {\n            let stale_duration = n.stale_duration(&Instant::now());\n            if let Some(stale_duration) = stale_duration {\n                (Some(n.value), CacheStatus::Stale(stale_duration))\n            } else {\n                (Some(n.value), CacheStatus::Hit)\n            }\n        } else {\n            (None, CacheStatus::Miss)\n        }\n    }\n\n    /// Insert a key and value pair with an optional TTL into the cache.\n    ///\n    /// An item with zero TTL of zero will not be inserted.\n    pub fn put<Q>(&self, key: &Q, value: T, ttl: Option<Duration>)\n    where\n        K: Borrow<Q>,\n        Q: Hash + ?Sized,\n    {\n        if let Some(t) = ttl {\n            if t.is_zero() {\n                return;\n            }\n        }\n        let hashed_key = self.hasher.hash_one(key);\n        let node = Node::new(value, ttl);\n        // weight is always 1 for now\n        self.store.put(hashed_key, node, 1);\n    }\n\n    /// Remove a key from the cache if it exists.\n    pub fn remove<Q>(&self, key: &Q)\n    where\n        K: Borrow<Q>,\n        Q: Hash + ?Sized,\n    {\n        let hashed_key = self.hasher.hash_one(key);\n        self.store.remove(&hashed_key);\n    }\n\n    pub(crate) fn force_put(&self, key: &K, value: T, ttl: Option<Duration>) {\n        if let Some(t) = ttl {\n            if t.is_zero() {\n                return;\n            }\n        }\n        let hashed_key = self.hasher.hash_one(key);\n        let node = Node::new(value, ttl);\n        // weight is always 1 for now\n        self.store.force_put(hashed_key, node, 1);\n    }\n\n    /// This is equivalent to [MemoryCache::get] but for an arbitrary amount of keys.\n    pub fn multi_get<'a, I, Q>(&self, keys: I) -> Vec<(Option<T>, CacheStatus)>\n    where\n        I: Iterator<Item = &'a Q>,\n        Q: Hash + ?Sized + 'a,\n        K: Borrow<Q> + 'a,\n    {\n        let mut resp = Vec::with_capacity(keys.size_hint().0);\n        for key in keys {\n            resp.push(self.get(key));\n        }\n        resp\n    }\n\n    /// Same as [MemoryCache::multi_get] but returns the keys that are missing from the cache.\n    pub fn multi_get_with_miss<'a, I, Q>(\n        &self,\n        keys: I,\n    ) -> (Vec<(Option<T>, CacheStatus)>, Vec<&'a Q>)\n    where\n        I: Iterator<Item = &'a Q>,\n        Q: Hash + ?Sized + 'a,\n        K: Borrow<Q> + 'a,\n    {\n        let mut resp = Vec::with_capacity(keys.size_hint().0);\n        let mut missed = Vec::with_capacity(keys.size_hint().0 / 2);\n        for key in keys {\n            let (lookup, cache_status) = self.get(key);\n            if lookup.is_none() {\n                missed.push(key);\n            }\n            resp.push((lookup, cache_status));\n        }\n        (resp, missed)\n    }\n\n    // TODO: evict expired first\n}\n\n#[cfg(test)]\nmod tests {\n    use super::*;\n    use std::thread::sleep;\n\n    #[test]\n    fn test_get() {\n        let cache: MemoryCache<i32, ()> = MemoryCache::new(10);\n        let (res, hit) = cache.get(&1);\n        assert_eq!(res, None);\n        assert_eq!(hit, CacheStatus::Miss);\n    }\n\n    #[test]\n    fn test_put_get() {\n        let cache: MemoryCache<i32, i32> = MemoryCache::new(10);\n        let (res, hit) = cache.get(&1);\n        assert_eq!(res, None);\n        assert_eq!(hit, CacheStatus::Miss);\n        cache.put(&1, 2, None);\n        let (res, hit) = cache.get(&1);\n        assert_eq!(res.unwrap(), 2);\n        assert_eq!(hit, CacheStatus::Hit);\n    }\n\n    #[test]\n    fn test_put_get_remove() {\n        let cache: MemoryCache<i32, i32> = MemoryCache::new(10);\n        let (res, hit) = cache.get(&1);\n        assert_eq!(res, None);\n        assert_eq!(hit, CacheStatus::Miss);\n        cache.put(&1, 2, None);\n        cache.put(&3, 4, None);\n        cache.put(&5, 6, None);\n        let (res, hit) = cache.get(&1);\n        assert_eq!(res.unwrap(), 2);\n        assert_eq!(hit, CacheStatus::Hit);\n        cache.remove(&1);\n        cache.remove(&3);\n        let (res, hit) = cache.get(&1);\n        assert_eq!(res, None);\n        assert_eq!(hit, CacheStatus::Miss);\n        let (res, hit) = cache.get(&3);\n        assert_eq!(res, None);\n        assert_eq!(hit, CacheStatus::Miss);\n        let (res, hit) = cache.get(&5);\n        assert_eq!(res.unwrap(), 6);\n        assert_eq!(hit, CacheStatus::Hit);\n    }\n\n    #[test]\n    fn test_get_expired() {\n        let cache: MemoryCache<i32, i32> = MemoryCache::new(10);\n        let (res, hit) = cache.get(&1);\n        assert_eq!(res, None);\n        assert_eq!(hit, CacheStatus::Miss);\n        cache.put(&1, 2, Some(Duration::from_secs(1)));\n        sleep(Duration::from_millis(1100));\n        let (res, hit) = cache.get(&1);\n        assert_eq!(res, None);\n        assert_eq!(hit, CacheStatus::Expired);\n    }\n\n    #[test]\n    fn test_get_stale() {\n        let cache: MemoryCache<i32, i32> = MemoryCache::new(10);\n        let (res, hit) = cache.get(&1);\n        assert_eq!(res, None);\n        assert_eq!(hit, CacheStatus::Miss);\n        cache.put(&1, 2, Some(Duration::from_secs(1)));\n        sleep(Duration::from_millis(1100));\n        let (res, hit) = cache.get_stale(&1);\n        assert_eq!(res.unwrap(), 2);\n        // we slept 1100ms and the ttl is 1000ms\n        assert!(hit.stale().unwrap() >= Duration::from_millis(100));\n    }\n\n    #[test]\n    fn test_eviction() {\n        let cache: MemoryCache<i32, i32> = MemoryCache::new(2);\n        cache.put(&1, 2, None);\n        cache.put(&2, 4, None);\n        cache.put(&3, 6, None);\n        let (res, hit) = cache.get(&1);\n        assert_eq!(res, None);\n        assert_eq!(hit, CacheStatus::Miss);\n        let (res, hit) = cache.get(&2);\n        assert_eq!(res.unwrap(), 4);\n        assert_eq!(hit, CacheStatus::Hit);\n        let (res, hit) = cache.get(&3);\n        assert_eq!(res.unwrap(), 6);\n        assert_eq!(hit, CacheStatus::Hit);\n    }\n\n    #[test]\n    fn test_multi_get() {\n        let cache: MemoryCache<i32, i32> = MemoryCache::new(10);\n        cache.put(&2, -2, None);\n        let keys: Vec<i32> = vec![1, 2, 3];\n        let resp = cache.multi_get(keys.iter());\n        assert_eq!(resp[0].0, None);\n        assert_eq!(resp[0].1, CacheStatus::Miss);\n        assert_eq!(resp[1].0.unwrap(), -2);\n        assert_eq!(resp[1].1, CacheStatus::Hit);\n        assert_eq!(resp[2].0, None);\n        assert_eq!(resp[2].1, CacheStatus::Miss);\n\n        let (resp, missed) = cache.multi_get_with_miss(keys.iter());\n        assert_eq!(resp[0].0, None);\n        assert_eq!(resp[0].1, CacheStatus::Miss);\n        assert_eq!(resp[1].0.unwrap(), -2);\n        assert_eq!(resp[1].1, CacheStatus::Hit);\n        assert_eq!(resp[2].0, None);\n        assert_eq!(resp[2].1, CacheStatus::Miss);\n        assert_eq!(missed[0], &1);\n        assert_eq!(missed[1], &3);\n    }\n\n    #[test]\n    fn test_get_with_mismatched_key() {\n        let cache: MemoryCache<String, ()> = MemoryCache::new(10);\n        let (res, hit) = cache.get(\"Hello\");\n        assert_eq!(res, None);\n        assert_eq!(hit, CacheStatus::Miss);\n    }\n\n    #[test]\n    fn test_put_get_with_mismatched_key() {\n        let cache: MemoryCache<String, i32> = MemoryCache::new(10);\n        let (res, hit) = cache.get(\"1\");\n        assert_eq!(res, None);\n        assert_eq!(hit, CacheStatus::Miss);\n        cache.put(\"1\", 2, None);\n        let (res, hit) = cache.get(\"1\");\n        assert_eq!(res.unwrap(), 2);\n        assert_eq!(hit, CacheStatus::Hit);\n    }\n}\n"
  },
  {
    "path": "pingora-memory-cache/src/read_through.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! An async read through cache where cache misses are populated via the provided\n//! async callback.\n\nuse super::{CacheStatus, MemoryCache};\n\nuse async_trait::async_trait;\nuse log::warn;\nuse parking_lot::RwLock;\nuse pingora_error::{Error, ErrorTrait};\nuse std::collections::HashMap;\nuse std::hash::Hash;\nuse std::marker::PhantomData;\nuse std::sync::Arc;\nuse std::time::{Duration, Instant};\nuse tokio::sync::Semaphore;\n\nstruct CacheLock {\n    pub lock_start: Instant,\n    pub lock: Semaphore,\n}\n\nimpl CacheLock {\n    pub fn new_arc() -> Arc<Self> {\n        Arc::new(CacheLock {\n            lock: Semaphore::new(0),\n            lock_start: Instant::now(),\n        })\n    }\n\n    pub fn too_old(&self, age: Option<&Duration>) -> bool {\n        match age {\n            Some(t) => Instant::now() - self.lock_start > *t,\n            None => false,\n        }\n    }\n}\n\n#[async_trait]\n/// [Lookup] defines the caching behavior that the implementor needs. The `extra` field can be used\n/// to define any additional metadata that the implementor uses to determine cache eligibility.\n///\n/// # Examples\n///\n/// ```ignore\n/// use pingora_error::{ErrorTrait, Result};\n/// use std::time::Duration;\n///\n/// struct MyLookup;\n///\n/// impl Lookup<usize, usize, ()> for MyLookup {\n///     async fn lookup(\n///         &self,\n///         _key: &usize,\n///         extra: Option<&()>,\n///     ) -> Result<(usize, Option<Duration>), Box<dyn ErrorTrait + Send + Sync>> {\n///         // Define your business logic here.\n///         Ok(1, None)\n///     }\n/// }\n/// ```\npub trait Lookup<K, T, S> {\n    /// Return a value and an optional TTL for the given key.\n    async fn lookup(\n        key: &K,\n        extra: Option<&S>,\n    ) -> Result<(T, Option<Duration>), Box<dyn ErrorTrait + Send + Sync>>\n    where\n        K: 'async_trait,\n        S: 'async_trait;\n}\n\n#[async_trait]\n/// [MultiLookup] is similar to [Lookup]. Implement this trait if the system being queried support\n/// looking up multiple keys in a single API call.\npub trait MultiLookup<K, T, S> {\n    /// Like [Lookup::lookup] but for an arbitrary amount of keys.\n    async fn multi_lookup(\n        keys: &[&K],\n        extra: Option<&S>,\n    ) -> Result<Vec<(T, Option<Duration>)>, Box<dyn ErrorTrait + Send + Sync>>\n    where\n        K: 'async_trait,\n        S: 'async_trait;\n}\n\nconst LOOKUP_ERR_MSG: &str = \"RTCache: lookup error\";\n\n/// A read-through in-memory cache on top of [MemoryCache]\n///\n/// Instead of providing a `put` function, [RTCache] requires a type which implements [Lookup] to\n/// be automatically called during cache miss to populate the cache. This is useful when trying to\n/// cache queries to external system such as DNS or databases.\n///\n/// Lookup coalescing is provided so that multiple concurrent lookups for the same key results\n/// only in one lookup callback.\npub struct RTCache<K, T, CB, S>\nwhere\n    K: Hash + Send,\n    T: Clone + Send,\n{\n    inner: MemoryCache<K, T>,\n    _callback: PhantomData<CB>,\n    lockers: RwLock<HashMap<u64, Arc<CacheLock>>>,\n    lock_age: Option<Duration>,\n    lock_timeout: Option<Duration>,\n    phantom: PhantomData<S>,\n}\n\nimpl<K, T, CB, S> RTCache<K, T, CB, S>\nwhere\n    K: Hash + Send,\n    T: Clone + Send + Sync + 'static,\n{\n    /// Create a new [RTCache] of given size. `lock_age` defines how long a lock is valid for.\n    /// `lock_timeout` is used to stop a lookup from holding on to the key for too long.\n    pub fn new(size: usize, lock_age: Option<Duration>, lock_timeout: Option<Duration>) -> Self {\n        RTCache {\n            inner: MemoryCache::new(size),\n            lockers: RwLock::new(HashMap::new()),\n            _callback: PhantomData,\n            lock_age,\n            lock_timeout,\n            phantom: PhantomData,\n        }\n    }\n}\n\nimpl<K, T, CB, S> RTCache<K, T, CB, S>\nwhere\n    K: Hash + Send,\n    T: Clone + Send + Sync + 'static,\n    CB: Lookup<K, T, S>,\n{\n    /// Query the cache for a given value. If it exists and no TTL is configured initially, it will\n    /// use the `ttl` value given.\n    pub async fn get(\n        &self,\n        key: &K,\n        ttl: Option<Duration>,\n        extra: Option<&S>,\n    ) -> (Result<T, Box<Error>>, CacheStatus) {\n        let (result, cache_state) = self.inner.get(key);\n        if let Some(result) = result {\n            /* cache hit */\n            return (Ok(result), cache_state);\n        }\n\n        let hashed_key = self.inner.hasher.hash_one(key);\n\n        /* Cache miss, try to lock the lookup. Check if there is already a lookup */\n        let my_lock = {\n            let lockers = self.lockers.read();\n            /* clone the Arc */\n            lockers.get(&hashed_key).cloned()\n        }; // read lock dropped\n\n        /* try insert a cache lock into locker */\n        let (my_write, my_read) = match my_lock {\n            // TODO: use a union\n            Some(lock) => {\n                /* There is an ongoing lookup to the same key */\n                if lock.too_old(self.lock_age.as_ref()) {\n                    (None, None)\n                } else {\n                    (None, Some(lock))\n                }\n            }\n            None => {\n                let mut lockers = self.lockers.write();\n                match lockers.get(&hashed_key) {\n                    Some(lock) => {\n                        /* another lookup to the same key got the write lock to locker first */\n                        if lock.too_old(self.lock_age.as_ref()) {\n                            (None, None)\n                        } else {\n                            (None, Some(lock.clone()))\n                        }\n                    }\n                    None => {\n                        let new_lock = CacheLock::new_arc();\n                        let new_lock2 = new_lock.clone();\n                        lockers.insert(hashed_key, new_lock2);\n                        (Some(new_lock), None)\n                    }\n                } // write lock dropped\n            }\n        };\n\n        if let Some(my_lock) = my_read {\n            /* another task will do the lookup */\n\n            /* if available_permits > 0, writer is done */\n            if my_lock.lock.available_permits() == 0 {\n                /* block here to wait for writer to finish lookup */\n                let lock_fut = my_lock.lock.acquire();\n                let timed_out = match self.lock_timeout {\n                    Some(t) => pingora_timeout::timeout(t, lock_fut).await.is_err(),\n                    None => {\n                        let _ = lock_fut.await;\n                        false\n                    }\n                };\n                if timed_out {\n                    let value = CB::lookup(key, extra).await;\n                    return match value {\n                        Ok((v, _ttl)) => (Ok(v), cache_state),\n                        Err(e) => {\n                            let mut err = Error::new_str(LOOKUP_ERR_MSG);\n                            err.set_cause(e);\n                            (Err(err), cache_state)\n                        }\n                    };\n                }\n            } // permit returned here\n\n            let (result, cache_state) = self.inner.get(key);\n            if let Some(result) = result {\n                /* cache lock hit, slow as a miss */\n                (Ok(result), CacheStatus::LockHit)\n            } else {\n                /* probably error happen during the actual lookup */\n                warn!(\n                    \"RTCache: no result after read lock, cache status: {:?}\",\n                    cache_state\n                );\n                match CB::lookup(key, extra).await {\n                    Ok((v, new_ttl)) => {\n                        self.inner.force_put(key, v.clone(), new_ttl.or(ttl));\n                        (Ok(v), cache_state)\n                    }\n                    Err(e) => {\n                        let mut err = Error::new_str(LOOKUP_ERR_MSG);\n                        err.set_cause(e);\n                        (Err(err), cache_state)\n                    }\n                }\n            }\n        } else {\n            /* this one will do the look up, either because it gets the write lock or the read\n             * lock age is reached */\n            let value = CB::lookup(key, extra).await;\n            let ret = match value {\n                Ok((v, new_ttl)) => {\n                    /* Don't put() if lock ago too old, to avoid too many concurrent writes */\n                    if my_write.is_some() {\n                        self.inner.force_put(key, v.clone(), new_ttl.or(ttl));\n                    }\n                    (Ok(v), cache_state) // the original cache_state: Miss or Expired\n                }\n                Err(e) => {\n                    let mut err = Error::new_str(LOOKUP_ERR_MSG);\n                    err.set_cause(e);\n                    (Err(err), cache_state)\n                }\n            };\n            if let Some(my_write) = my_write {\n                /* add permit so that reader can start. Any number of permits will do,\n                 * since readers will return permits right away. */\n                my_write.lock.add_permits(10);\n\n                {\n                    // remove the lock from locker\n                    let mut lockers = self.lockers.write();\n                    lockers.remove(&hashed_key);\n                } // write lock dropped here\n            }\n\n            ret\n        }\n    }\n\n    /// Similar to [Self::get], query the cache for a given value, but also returns the value even if the\n    /// value is expired up to `stale_ttl`. If it is a cache miss or the value is stale more than\n    /// the `stale_ttl`, a lookup will be performed to populate the cache.\n    pub async fn get_stale(\n        &self,\n        key: &K,\n        ttl: Option<Duration>,\n        extra: Option<&S>,\n        stale_ttl: Duration,\n    ) -> (Result<T, Box<Error>>, CacheStatus) {\n        let (result, cache_status) = self.inner.get_stale(key);\n        if let Some(result) = result {\n            let stale_duration = cache_status.stale();\n            if stale_duration.unwrap_or(Duration::ZERO) <= stale_ttl {\n                return (Ok(result), cache_status);\n            }\n        }\n        let (res, status) = self.get(key, ttl, extra).await;\n        (res, status)\n    }\n}\n\nimpl<K, T, CB, S> RTCache<K, T, CB, S>\nwhere\n    K: Hash + Clone + Send + Sync,\n    T: Clone + Send + Sync + 'static,\n    S: Clone + Send + Sync,\n    CB: Lookup<K, T, S> + Sync + Send,\n{\n    /// Similar to [Self::get_stale], but when it returns the stale value, it also initiates a lookup\n    /// in the background in order to refresh the value.\n    ///\n    /// Note that this function requires the [RTCache] to be static, which can be done by wrapping\n    /// it with something like [once_cell::sync::Lazy].\n    ///\n    /// [once_cell::sync::Lazy]: https://docs.rs/once_cell/latest/once_cell/sync/struct.Lazy.html\n    pub async fn get_stale_while_update(\n        &'static self,\n        key: &K,\n        ttl: Option<Duration>,\n        extra: Option<&S>,\n        stale_ttl: Duration,\n    ) -> (Result<T, Box<Error>>, CacheStatus) {\n        let (result, cache_status) = self.get_stale(key, ttl, extra, stale_ttl).await;\n        let key = key.clone();\n        let extra = extra.cloned();\n        if cache_status.stale().is_some() {\n            tokio::spawn(async move {\n                let _ = self.get(&key, ttl, extra.as_ref()).await;\n            });\n        }\n        (result, cache_status)\n    }\n}\n\nimpl<K, T, CB, S> RTCache<K, T, CB, S>\nwhere\n    K: Hash + Send,\n    T: Clone + Send + Sync + 'static,\n    CB: MultiLookup<K, T, S>,\n{\n    /// Same behavior as [RTCache::get] but for an arbitrary amount of keys.\n    ///\n    /// If there are keys that are missing from the cache, `multi_lookup` is invoked to populate the\n    /// cache before returning the final results. This is useful if your type supports batch\n    /// queries.\n    ///\n    /// To avoid dead lock for the same key across concurrent `multi_get` calls,\n    /// this function does not provide lookup coalescing.\n    pub async fn multi_get<'a, I>(\n        &self,\n        keys: I,\n        ttl: Option<Duration>,\n        extra: Option<&S>,\n    ) -> Result<Vec<(T, CacheStatus)>, Box<Error>>\n    where\n        I: Iterator<Item = &'a K>,\n        K: 'a,\n    {\n        let size = keys.size_hint().0;\n        let (hits, misses) = self.inner.multi_get_with_miss(keys);\n        let mut final_results = Vec::with_capacity(size);\n        let miss_results = if !misses.is_empty() {\n            match CB::multi_lookup(&misses, extra).await {\n                Ok(miss_results) => {\n                    // assert! here to prevent index panic when building results,\n                    // final_results has the full list of misses but miss_results might not\n                    assert!(\n                        miss_results.len() == misses.len(),\n                        \"multi_lookup() failed to return the matching number of results\"\n                    );\n                    /* put the misses into cache */\n                    for item in misses.iter().zip(miss_results.iter()) {\n                        self.inner\n                            .force_put(item.0, (item.1).0.clone(), (item.1).1.or(ttl));\n                    }\n                    miss_results\n                }\n                Err(e) => {\n                    /* NOTE: we give up the hits when encounter lookup error */\n                    let mut err = Error::new_str(LOOKUP_ERR_MSG);\n                    err.set_cause(e);\n                    return Err(err);\n                }\n            }\n        } else {\n            vec![] // to make the rest code simple, allocating one unused empty vec should be fine\n        };\n        /* fill in final_result */\n        let mut n_miss = 0;\n        for item in hits {\n            match item.0 {\n                Some(v) => final_results.push((v, item.1)),\n                None => {\n                    final_results // miss_results.len() === #None in result (asserted above)\n                    .push((miss_results[n_miss].0.clone(), CacheStatus::Miss));\n                    n_miss += 1;\n                }\n            }\n        }\n        Ok(final_results)\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use super::*;\n    use atomic::AtomicI32;\n    use std::sync::atomic;\n\n    #[derive(Clone, Debug)]\n    struct ExtraOpt {\n        error: bool,\n        empty: bool,\n        delay_for: Option<Duration>,\n        used: Arc<AtomicI32>,\n    }\n\n    struct TestCB();\n\n    #[async_trait]\n    impl Lookup<i32, i32, ExtraOpt> for TestCB {\n        async fn lookup(\n            _key: &i32,\n            extra: Option<&ExtraOpt>,\n        ) -> Result<(i32, Option<Duration>), Box<dyn ErrorTrait + Send + Sync>> {\n            // this function returns #lookup_times\n            let mut used = 0;\n            if let Some(e) = extra {\n                used = e.used.fetch_add(1, atomic::Ordering::Relaxed) + 1;\n                if e.error {\n                    return Err(Error::new_str(\"test error\"));\n                }\n                if let Some(delay_for) = e.delay_for {\n                    tokio::time::sleep(delay_for).await;\n                }\n            }\n            Ok((used, None))\n        }\n    }\n\n    #[async_trait]\n    impl MultiLookup<i32, i32, ExtraOpt> for TestCB {\n        async fn multi_lookup(\n            keys: &[&i32],\n            extra: Option<&ExtraOpt>,\n        ) -> Result<Vec<(i32, Option<Duration>)>, Box<dyn ErrorTrait + Send + Sync>> {\n            let mut resp = vec![];\n            if let Some(extra) = extra {\n                if extra.empty {\n                    return Ok(resp);\n                }\n            }\n            for key in keys {\n                resp.push((**key, None));\n            }\n            Ok(resp)\n        }\n    }\n\n    #[tokio::test]\n    async fn test_basic_get() {\n        let cache: RTCache<i32, i32, TestCB, ExtraOpt> = RTCache::new(10, None, None);\n        let opt = Some(ExtraOpt {\n            error: false,\n            empty: false,\n            delay_for: None,\n            used: Arc::new(AtomicI32::new(0)),\n        });\n        let (res, hit) = cache.get(&1, None, opt.as_ref()).await;\n        assert_eq!(res.unwrap(), 1);\n        assert_eq!(hit, CacheStatus::Miss);\n        let (res, hit) = cache.get(&1, None, opt.as_ref()).await;\n        assert_eq!(res.unwrap(), 1);\n        assert_eq!(hit, CacheStatus::Hit);\n    }\n\n    #[tokio::test]\n    async fn test_basic_get_error() {\n        let cache: RTCache<i32, i32, TestCB, ExtraOpt> = RTCache::new(10, None, None);\n        let opt1 = Some(ExtraOpt {\n            error: true,\n            empty: false,\n            delay_for: None,\n            used: Arc::new(AtomicI32::new(0)),\n        });\n        let (res, hit) = cache.get(&-1, None, opt1.as_ref()).await;\n        assert!(res.is_err());\n        assert_eq!(hit, CacheStatus::Miss);\n    }\n\n    #[tokio::test]\n    async fn test_concurrent_get() {\n        let cache: RTCache<i32, i32, TestCB, ExtraOpt> = RTCache::new(10, None, None);\n        let cache = Arc::new(cache);\n        let opt = Some(ExtraOpt {\n            error: false,\n            empty: false,\n            delay_for: None,\n            used: Arc::new(AtomicI32::new(0)),\n        });\n        let cache_c = cache.clone();\n        let opt1 = opt.clone();\n        // concurrent gets, only 1 will call the callback\n        let t1 = tokio::spawn(async move {\n            let (res, _hit) = cache_c.get(&1, None, opt1.as_ref()).await;\n            res.unwrap()\n        });\n        let cache_c = cache.clone();\n        let opt2 = opt.clone();\n        let t2 = tokio::spawn(async move {\n            let (res, _hit) = cache_c.get(&1, None, opt2.as_ref()).await;\n            res.unwrap()\n        });\n        let opt3 = opt.clone();\n        let cache_c = cache.clone();\n        let t3 = tokio::spawn(async move {\n            let (res, _hit) = cache_c.get(&1, None, opt3.as_ref()).await;\n            res.unwrap()\n        });\n        let (r1, r2, r3) = tokio::join!(t1, t2, t3);\n        assert_eq!(r1.unwrap(), 1);\n        assert_eq!(r2.unwrap(), 1);\n        assert_eq!(r3.unwrap(), 1);\n    }\n\n    #[tokio::test]\n    async fn test_concurrent_get_error() {\n        let cache: RTCache<i32, i32, TestCB, ExtraOpt> = RTCache::new(10, None, None);\n        let cache = Arc::new(cache);\n        let cache_c = cache.clone();\n        let opt1 = Some(ExtraOpt {\n            error: true,\n            empty: false,\n            delay_for: None,\n            used: Arc::new(AtomicI32::new(0)),\n        });\n        let opt2 = opt1.clone();\n        let opt3 = opt1.clone();\n        // concurrent gets, only 1 will call the callback\n        let t1 = tokio::spawn(async move {\n            let (res, _hit) = cache_c.get(&-1, None, opt1.as_ref()).await;\n            res.is_err()\n        });\n        let cache_c = cache.clone();\n        let t2 = tokio::spawn(async move {\n            let (res, _hit) = cache_c.get(&-1, None, opt2.as_ref()).await;\n            res.is_err()\n        });\n        let cache_c = cache.clone();\n        let t3 = tokio::spawn(async move {\n            let (res, _hit) = cache_c.get(&-1, None, opt3.as_ref()).await;\n            res.is_err()\n        });\n        let (r1, r2, r3) = tokio::join!(t1, t2, t3);\n        assert!(r1.unwrap());\n        assert!(r2.unwrap());\n        assert!(r3.unwrap());\n    }\n\n    #[tokio::test]\n    async fn test_concurrent_get_different_value() {\n        let cache: RTCache<i32, i32, TestCB, ExtraOpt> = RTCache::new(10, None, None);\n        let cache = Arc::new(cache);\n        let opt1 = Some(ExtraOpt {\n            error: false,\n            empty: false,\n            delay_for: None,\n            used: Arc::new(AtomicI32::new(0)),\n        });\n        let opt2 = opt1.clone();\n        let opt3 = opt1.clone();\n        let cache_c = cache.clone();\n        // concurrent gets to different keys, no locks, all will call the cb\n        let t1 = tokio::spawn(async move {\n            let (res, _hit) = cache_c.get(&1, None, opt1.as_ref()).await;\n            res.unwrap()\n        });\n        let cache_c = cache.clone();\n        let t2 = tokio::spawn(async move {\n            let (res, _hit) = cache_c.get(&3, None, opt2.as_ref()).await;\n            res.unwrap()\n        });\n        let cache_c = cache.clone();\n        let t3 = tokio::spawn(async move {\n            let (res, _hit) = cache_c.get(&5, None, opt3.as_ref()).await;\n            res.unwrap()\n        });\n        let (r1, r2, r3) = tokio::join!(t1, t2, t3);\n        // 1 lookup + 2 lookups + 3 lookups, order not matter\n        assert_eq!(r1.unwrap() + r2.unwrap() + r3.unwrap(), 6);\n    }\n\n    #[tokio::test]\n    async fn test_get_lock_age() {\n        // 1 sec lock age\n        let cache: RTCache<i32, i32, TestCB, ExtraOpt> =\n            RTCache::new(10, Some(Duration::from_secs(1)), None);\n        let cache = Arc::new(cache);\n        let counter = Arc::new(AtomicI32::new(0));\n        let opt1 = Some(ExtraOpt {\n            error: false,\n            empty: false,\n            delay_for: Some(Duration::from_secs(2)),\n            used: counter.clone(),\n        });\n\n        let opt2 = Some(ExtraOpt {\n            error: false,\n            empty: false,\n            delay_for: None,\n            used: counter.clone(),\n        });\n        let opt3 = opt2.clone();\n        let cache_c = cache.clone();\n        // t1 will be delay for 2 sec\n        let t1 = tokio::spawn(async move {\n            let (res, _hit) = cache_c.get(&1, None, opt1.as_ref()).await;\n            res.unwrap()\n        });\n        // start t2 and t3 1.5 seconds later, since lock age is 1 sec, there will be no lock\n        tokio::time::sleep(Duration::from_secs_f32(1.5)).await;\n        let cache_c = cache.clone();\n        let t2 = tokio::spawn(async move {\n            let (res, _hit) = cache_c.get(&1, None, opt2.as_ref()).await;\n            res.unwrap()\n        });\n        let cache_c = cache.clone();\n        let t3 = tokio::spawn(async move {\n            let (res, _hit) = cache_c.get(&1, None, opt3.as_ref()).await;\n            res.unwrap()\n        });\n        let (r1, r2, r3) = tokio::join!(t1, t2, t3);\n        // 1 lookup + 2 lookups + 3 lookups, order not matter\n        assert_eq!(r1.unwrap() + r2.unwrap() + r3.unwrap(), 6);\n    }\n\n    #[tokio::test]\n    async fn test_get_lock_timeout() {\n        // 1 sec lock timeout\n        let cache: RTCache<i32, i32, TestCB, ExtraOpt> =\n            RTCache::new(10, None, Some(Duration::from_secs(1)));\n        let cache = Arc::new(cache);\n        let counter = Arc::new(AtomicI32::new(0));\n        let opt1 = Some(ExtraOpt {\n            error: false,\n            empty: false,\n            delay_for: Some(Duration::from_secs(2)),\n            used: counter.clone(),\n        });\n        let opt2 = Some(ExtraOpt {\n            error: false,\n            empty: false,\n            delay_for: None,\n            used: counter.clone(),\n        });\n        let opt3 = opt2.clone();\n        let cache_c = cache.clone();\n        // t1 will be delay for 2 sec\n        let t1 = tokio::spawn(async move {\n            let (res, _hit) = cache_c.get(&1, None, opt1.as_ref()).await;\n            res.unwrap()\n        });\n        // since lock timeout is 1 sec, t2 and t3 will do their own lookup after 1 sec\n        let cache_c = cache.clone();\n        let t2 = tokio::spawn(async move {\n            let (res, _hit) = cache_c.get(&1, None, opt2.as_ref()).await;\n            res.unwrap()\n        });\n        let cache_c = cache.clone();\n        let t3 = tokio::spawn(async move {\n            let (res, _hit) = cache_c.get(&1, None, opt3.as_ref()).await;\n            res.unwrap()\n        });\n        let (r1, r2, r3) = tokio::join!(t1, t2, t3);\n        // 1 lookup + 2 lookups + 3 lookups, order not matter\n        assert_eq!(r1.unwrap() + r2.unwrap() + r3.unwrap(), 6);\n    }\n\n    #[tokio::test]\n    async fn test_multi_get() {\n        let cache: RTCache<i32, i32, TestCB, ExtraOpt> = RTCache::new(10, None, None);\n        let counter = Arc::new(AtomicI32::new(0));\n        let opt1 = Some(ExtraOpt {\n            error: false,\n            empty: false,\n            delay_for: Some(Duration::from_secs(2)),\n            used: counter.clone(),\n        });\n        // make 1 a hit first\n        let (res, hit) = cache.get(&1, None, opt1.as_ref()).await;\n        assert_eq!(res.unwrap(), 1);\n        assert_eq!(hit, CacheStatus::Miss);\n        let (res, hit) = cache.get(&1, None, opt1.as_ref()).await;\n        assert_eq!(res.unwrap(), 1);\n        assert_eq!(hit, CacheStatus::Hit);\n        // 1 hit 2 miss 3 miss\n        let resp = cache\n            .multi_get([1, 2, 3].iter(), None, opt1.as_ref())\n            .await\n            .unwrap();\n        assert_eq!(resp[0].0, 1);\n        assert_eq!(resp[0].1, CacheStatus::Hit);\n        assert_eq!(resp[1].0, 2);\n        assert_eq!(resp[1].1, CacheStatus::Miss);\n        assert_eq!(resp[2].0, 3);\n        assert_eq!(resp[2].1, CacheStatus::Miss);\n        // all hits after a fetch\n        let resp = cache\n            .multi_get([1, 2, 3].iter(), None, opt1.as_ref())\n            .await\n            .unwrap();\n        assert_eq!(resp[0].0, 1);\n        assert_eq!(resp[0].1, CacheStatus::Hit);\n        assert_eq!(resp[1].0, 2);\n        assert_eq!(resp[1].1, CacheStatus::Hit);\n        assert_eq!(resp[2].0, 3);\n        assert_eq!(resp[2].1, CacheStatus::Hit);\n    }\n\n    #[tokio::test]\n    #[should_panic(expected = \"multi_lookup() failed to return the matching number of results\")]\n    async fn test_inconsistent_miss_results() {\n        // force an empty result\n        let opt1 = Some(ExtraOpt {\n            error: false,\n            empty: true,\n            delay_for: None,\n            used: Arc::new(AtomicI32::new(0)),\n        });\n        let cache: RTCache<i32, i32, TestCB, ExtraOpt> = RTCache::new(10, None, None);\n        cache\n            .multi_get([4, 5, 6].iter(), None, opt1.as_ref())\n            .await\n            .unwrap();\n    }\n\n    #[tokio::test]\n    async fn test_get_stale() {\n        let ttl = Some(Duration::from_millis(100));\n        let cache: RTCache<i32, i32, TestCB, ExtraOpt> = RTCache::new(10, None, None);\n        let opt = Some(ExtraOpt {\n            error: false,\n            empty: false,\n            delay_for: None,\n            used: Arc::new(AtomicI32::new(0)),\n        });\n        let (res, hit) = cache.get(&1, ttl, opt.as_ref()).await;\n        assert_eq!(res.unwrap(), 1);\n        assert_eq!(hit, CacheStatus::Miss);\n        let (res, hit) = cache.get(&1, ttl, opt.as_ref()).await;\n        assert_eq!(res.unwrap(), 1);\n        assert_eq!(hit, CacheStatus::Hit);\n        tokio::time::sleep(Duration::from_millis(150)).await;\n        let (res, hit) = cache\n            .get_stale(&1, ttl, opt.as_ref(), Duration::from_millis(1000))\n            .await;\n        assert_eq!(res.unwrap(), 1);\n        assert!(hit.stale().is_some());\n\n        let (res, hit) = cache\n            .get_stale(&1, ttl, opt.as_ref(), Duration::from_millis(30))\n            .await;\n        assert_eq!(res.unwrap(), 2);\n        assert_eq!(hit, CacheStatus::Expired);\n    }\n\n    #[tokio::test]\n    async fn test_get_stale_while_update() {\n        use once_cell::sync::Lazy;\n        let ttl = Some(Duration::from_millis(100));\n        static CACHE: Lazy<RTCache<i32, i32, TestCB, ExtraOpt>> =\n            Lazy::new(|| RTCache::new(10, None, None));\n        let opt = Some(ExtraOpt {\n            error: false,\n            empty: false,\n            delay_for: None,\n            used: Arc::new(AtomicI32::new(0)),\n        });\n        let (res, hit) = CACHE.get(&1, ttl, opt.as_ref()).await;\n        assert_eq!(res.unwrap(), 1);\n        assert_eq!(hit, CacheStatus::Miss);\n        let (res, hit) = CACHE.get(&1, ttl, opt.as_ref()).await;\n        assert_eq!(res.unwrap(), 1);\n        assert_eq!(hit, CacheStatus::Hit);\n        tokio::time::sleep(Duration::from_millis(150)).await;\n        let (res, hit) = CACHE\n            .get_stale_while_update(&1, ttl, opt.as_ref(), Duration::from_millis(1000))\n            .await;\n        assert_eq!(res.unwrap(), 1);\n        assert!(hit.stale().is_some());\n\n        // allow the background lookup to finish\n        tokio::time::sleep(Duration::from_millis(10)).await;\n\n        let (res, hit) = CACHE.get(&1, ttl, opt.as_ref()).await;\n        assert_eq!(res.unwrap(), 2);\n        assert_eq!(hit, CacheStatus::Hit);\n    }\n}\n"
  },
  {
    "path": "pingora-openssl/Cargo.toml",
    "content": "[package]\nname = \"pingora-openssl\"\nversion = \"0.8.0\"\nauthors = [\"Yuchen Wu <yuchen@cloudflare.com>\"]\nlicense = \"Apache-2.0\"\nedition = \"2021\"\nrepository = \"https://github.com/cloudflare/pingora\"\ncategories = [\"asynchronous\", \"network-programming\"]\nkeywords = [\"async\", \"tls\", \"ssl\", \"pingora\"]\ndescription = \"\"\"\nOpenSSL async APIs for Pingora.\n\"\"\"\n\n# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html\n[lib]\nname = \"pingora_openssl\"\npath = \"src/lib.rs\"\n\n[dependencies]\nopenssl-sys = \"0.9\"\nopenssl = { version = \"0.10.72\", features = [\"vendored\"] }\ntokio-openssl = { version = \"0.6\" }\nlibc = \"0.2.70\"\nforeign-types = { version = \"0.3\"}\n\n[dev-dependencies]\ntokio-test = \"0.4\"\ntokio = { workspace = true, features = [\"full\"] }\n"
  },
  {
    "path": "pingora-openssl/LICENSE",
    "content": "\n                                 Apache License\n                           Version 2.0, January 2004\n                        http://www.apache.org/licenses/\n\n   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION\n\n   1. Definitions.\n\n      \"License\" shall mean the terms and conditions for use, reproduction,\n      and distribution as defined by Sections 1 through 9 of this document.\n\n      \"Licensor\" shall mean the copyright owner or entity authorized by\n      the copyright owner that is granting the License.\n\n      \"Legal Entity\" shall mean the union of the acting entity and all\n      other entities that control, are controlled by, or are under common\n      control with that entity. For the purposes of this definition,\n      \"control\" means (i) the power, direct or indirect, to cause the\n      direction or management of such entity, whether by contract or\n      otherwise, or (ii) ownership of fifty percent (50%) or more of the\n      outstanding shares, or (iii) beneficial ownership of such entity.\n\n      \"You\" (or \"Your\") shall mean an individual or Legal Entity\n      exercising permissions granted by this License.\n\n      \"Source\" form shall mean the preferred form for making modifications,\n      including but not limited to software source code, documentation\n      source, and configuration files.\n\n      \"Object\" form shall mean any form resulting from mechanical\n      transformation or translation of a Source form, including but\n      not limited to compiled object code, generated documentation,\n      and conversions to other media types.\n\n      \"Work\" shall mean the work of authorship, whether in Source or\n      Object form, made available under the License, as indicated by a\n      copyright notice that is included in or attached to the work\n      (an example is provided in the Appendix below).\n\n      \"Derivative Works\" shall mean any work, whether in Source or Object\n      form, that is based on (or derived from) the Work and for which the\n      editorial revisions, annotations, elaborations, or other modifications\n      represent, as a whole, an original work of authorship. For the purposes\n      of this License, Derivative Works shall not include works that remain\n      separable from, or merely link (or bind by name) to the interfaces of,\n      the Work and Derivative Works thereof.\n\n      \"Contribution\" shall mean any work of authorship, including\n      the original version of the Work and any modifications or additions\n      to that Work or Derivative Works thereof, that is intentionally\n      submitted to Licensor for inclusion in the Work by the copyright owner\n      or by an individual or Legal Entity authorized to submit on behalf of\n      the copyright owner. For the purposes of this definition, \"submitted\"\n      means any form of electronic, verbal, or written communication sent\n      to the Licensor or its representatives, including but not limited to\n      communication on electronic mailing lists, source code control systems,\n      and issue tracking systems that are managed by, or on behalf of, the\n      Licensor for the purpose of discussing and improving the Work, but\n      excluding communication that is conspicuously marked or otherwise\n      designated in writing by the copyright owner as \"Not a Contribution.\"\n\n      \"Contributor\" shall mean Licensor and any individual or Legal Entity\n      on behalf of whom a Contribution has been received by Licensor and\n      subsequently incorporated within the Work.\n\n   2. Grant of Copyright License. Subject to the terms and conditions of\n      this License, each Contributor hereby grants to You a perpetual,\n      worldwide, non-exclusive, no-charge, royalty-free, irrevocable\n      copyright license to reproduce, prepare Derivative Works of,\n      publicly display, publicly perform, sublicense, and distribute the\n      Work and such Derivative Works in Source or Object form.\n\n   3. Grant of Patent License. Subject to the terms and conditions of\n      this License, each Contributor hereby grants to You a perpetual,\n      worldwide, non-exclusive, no-charge, royalty-free, irrevocable\n      (except as stated in this section) patent license to make, have made,\n      use, offer to sell, sell, import, and otherwise transfer the Work,\n      where such license applies only to those patent claims licensable\n      by such Contributor that are necessarily infringed by their\n      Contribution(s) alone or by combination of their Contribution(s)\n      with the Work to which such Contribution(s) was submitted. If You\n      institute patent litigation against any entity (including a\n      cross-claim or counterclaim in a lawsuit) alleging that the Work\n      or a Contribution incorporated within the Work constitutes direct\n      or contributory patent infringement, then any patent licenses\n      granted to You under this License for that Work shall terminate\n      as of the date such litigation is filed.\n\n   4. Redistribution. You may reproduce and distribute copies of the\n      Work or Derivative Works thereof in any medium, with or without\n      modifications, and in Source or Object form, provided that You\n      meet the following conditions:\n\n      (a) You must give any other recipients of the Work or\n          Derivative Works a copy of this License; and\n\n      (b) You must cause any modified files to carry prominent notices\n          stating that You changed the files; and\n\n      (c) You must retain, in the Source form of any Derivative Works\n          that You distribute, all copyright, patent, trademark, and\n          attribution notices from the Source form of the Work,\n          excluding those notices that do not pertain to any part of\n          the Derivative Works; and\n\n      (d) If the Work includes a \"NOTICE\" text file as part of its\n          distribution, then any Derivative Works that You distribute must\n          include a readable copy of the attribution notices contained\n          within such NOTICE file, excluding those notices that do not\n          pertain to any part of the Derivative Works, in at least one\n          of the following places: within a NOTICE text file distributed\n          as part of the Derivative Works; within the Source form or\n          documentation, if provided along with the Derivative Works; or,\n          within a display generated by the Derivative Works, if and\n          wherever such third-party notices normally appear. The contents\n          of the NOTICE file are for informational purposes only and\n          do not modify the License. You may add Your own attribution\n          notices within Derivative Works that You distribute, alongside\n          or as an addendum to the NOTICE text from the Work, provided\n          that such additional attribution notices cannot be construed\n          as modifying the License.\n\n      You may add Your own copyright statement to Your modifications and\n      may provide additional or different license terms and conditions\n      for use, reproduction, or distribution of Your modifications, or\n      for any such Derivative Works as a whole, provided Your use,\n      reproduction, and distribution of the Work otherwise complies with\n      the conditions stated in this License.\n\n   5. Submission of Contributions. Unless You explicitly state otherwise,\n      any Contribution intentionally submitted for inclusion in the Work\n      by You to the Licensor shall be under the terms and conditions of\n      this License, without any additional terms or conditions.\n      Notwithstanding the above, nothing herein shall supersede or modify\n      the terms of any separate license agreement you may have executed\n      with Licensor regarding such Contributions.\n\n   6. Trademarks. This License does not grant permission to use the trade\n      names, trademarks, service marks, or product names of the Licensor,\n      except as required for reasonable and customary use in describing the\n      origin of the Work and reproducing the content of the NOTICE file.\n\n   7. Disclaimer of Warranty. Unless required by applicable law or\n      agreed to in writing, Licensor provides the Work (and each\n      Contributor provides its Contributions) on an \"AS IS\" BASIS,\n      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or\n      implied, including, without limitation, any warranties or conditions\n      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A\n      PARTICULAR PURPOSE. You are solely responsible for determining the\n      appropriateness of using or redistributing the Work and assume any\n      risks associated with Your exercise of permissions under this License.\n\n   8. Limitation of Liability. In no event and under no legal theory,\n      whether in tort (including negligence), contract, or otherwise,\n      unless required by applicable law (such as deliberate and grossly\n      negligent acts) or agreed to in writing, shall any Contributor be\n      liable to You for damages, including any direct, indirect, special,\n      incidental, or consequential damages of any character arising as a\n      result of this License or out of the use or inability to use the\n      Work (including but not limited to damages for loss of goodwill,\n      work stoppage, computer failure or malfunction, or any and all\n      other commercial damages or losses), even if such Contributor\n      has been advised of the possibility of such damages.\n\n   9. Accepting Warranty or Additional Liability. While redistributing\n      the Work or Derivative Works thereof, You may choose to offer,\n      and charge a fee for, acceptance of support, warranty, indemnity,\n      or other liability obligations and/or rights consistent with this\n      License. However, in accepting such obligations, You may act only\n      on Your own behalf and on Your sole responsibility, not on behalf\n      of any other Contributor, and only if You agree to indemnify,\n      defend, and hold each Contributor harmless for any liability\n      incurred by, or claims asserted against, such Contributor by reason\n      of your accepting any such warranty or additional liability.\n\n   END OF TERMS AND CONDITIONS\n\n   APPENDIX: How to apply the Apache License to your work.\n\n      To apply the Apache License to your work, attach the following\n      boilerplate notice, with the fields enclosed by brackets \"[]\"\n      replaced with your own identifying information. (Don't include\n      the brackets!)  The text should be enclosed in the appropriate\n      comment syntax for the file format. We also recommend that a\n      file or class name and description of purpose be included on the\n      same \"printed page\" as the copyright notice for easier\n      identification within third-party archives.\n\n   Copyright [yyyy] [name of copyright owner]\n\n   Licensed under the Apache License, Version 2.0 (the \"License\");\n   you may not use this file except in compliance with the License.\n   You may obtain a copy of the License at\n\n       http://www.apache.org/licenses/LICENSE-2.0\n\n   Unless required by applicable law or agreed to in writing, software\n   distributed under the License is distributed on an \"AS IS\" BASIS,\n   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n   See the License for the specific language governing permissions and\n   limitations under the License.\n"
  },
  {
    "path": "pingora-openssl/src/ext.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\nuse foreign_types::ForeignTypeRef;\nuse libc::*;\nuse openssl::error::ErrorStack;\nuse openssl::pkey::{HasPrivate, PKeyRef};\nuse openssl::ssl::{Ssl, SslAcceptor, SslRef};\nuse openssl::x509::store::X509StoreRef;\nuse openssl::x509::verify::X509VerifyParamRef;\nuse openssl::x509::X509Ref;\nuse openssl_sys::{\n    SSL_ctrl, EVP_PKEY, SSL, SSL_CTRL_SET_GROUPS_LIST, SSL_CTRL_SET_VERIFY_CERT_STORE, X509,\n    X509_VERIFY_PARAM,\n};\nuse std::ffi::CString;\nuse std::os::raw;\n\nfn cvt(r: c_long) -> Result<c_long, ErrorStack> {\n    if r != 1 {\n        Err(ErrorStack::get())\n    } else {\n        Ok(r)\n    }\n}\n\nextern \"C\" {\n    pub fn X509_VERIFY_PARAM_add1_host(\n        param: *mut X509_VERIFY_PARAM,\n        name: *const c_char,\n        namelen: size_t,\n    ) -> c_int;\n\n    pub fn SSL_use_certificate(ssl: *mut SSL, cert: *mut X509) -> c_int;\n    pub fn SSL_use_PrivateKey(ssl: *mut SSL, key: *mut EVP_PKEY) -> c_int;\n\n    pub fn SSL_set_cert_cb(\n        ssl: *mut SSL,\n        cb: ::std::option::Option<\n            unsafe extern \"C\" fn(ssl: *mut SSL, arg: *mut raw::c_void) -> raw::c_int,\n        >,\n        arg: *mut raw::c_void,\n    );\n}\n\n/// Add name as an additional reference identifier that can match the peer's certificate\n///\n/// See [X509_VERIFY_PARAM_set1_host](https://www.openssl.org/docs/man3.1/man3/X509_VERIFY_PARAM_set1_host.html).\npub fn add_host(verify_param: &mut X509VerifyParamRef, host: &str) -> Result<(), ErrorStack> {\n    if host.is_empty() {\n        return Ok(());\n    }\n    unsafe {\n        cvt(X509_VERIFY_PARAM_add1_host(\n            verify_param.as_ptr(),\n            host.as_ptr() as *const c_char,\n            host.len(),\n        ) as c_long)\n        .map(|_| ())\n    }\n}\n\n/// Set the verify cert store of `ssl`\n///\n/// See [SSL_set1_verify_cert_store](https://www.openssl.org/docs/man1.1.1/man3/SSL_set1_verify_cert_store.html).\npub fn ssl_set_verify_cert_store(\n    ssl: &mut SslRef,\n    cert_store: &X509StoreRef,\n) -> Result<(), ErrorStack> {\n    unsafe {\n        cvt(SSL_ctrl(\n            ssl.as_ptr(),\n            SSL_CTRL_SET_VERIFY_CERT_STORE,\n            1, // increase the ref count of X509Store so that ssl_ctx can outlive X509StoreRef\n            cert_store.as_ptr() as *mut c_void,\n        ))?;\n    }\n    Ok(())\n}\n\n/// Load the certificate into `ssl`\n///\n/// See [SSL_use_certificate](https://www.openssl.org/docs/man1.1.1/man3/SSL_use_certificate.html).\npub fn ssl_use_certificate(ssl: &mut SslRef, cert: &X509Ref) -> Result<(), ErrorStack> {\n    unsafe {\n        cvt(SSL_use_certificate(ssl.as_ptr(), cert.as_ptr()) as c_long)?;\n    }\n    Ok(())\n}\n\n/// Load the private key into `ssl`\n///\n/// See [SSL_use_certificate](https://www.openssl.org/docs/man1.1.1/man3/SSL_use_PrivateKey.html).\npub fn ssl_use_private_key<T>(ssl: &mut SslRef, key: &PKeyRef<T>) -> Result<(), ErrorStack>\nwhere\n    T: HasPrivate,\n{\n    unsafe {\n        cvt(SSL_use_PrivateKey(ssl.as_ptr(), key.as_ptr()) as c_long)?;\n    }\n    Ok(())\n}\n\n/// Add the certificate into the cert chain of `ssl`\n///\n/// See [SSL_add1_chain_cert](https://www.openssl.org/docs/man1.1.1/man3/SSL_add1_chain_cert.html)\npub fn ssl_add_chain_cert(ssl: &mut SslRef, cert: &X509Ref) -> Result<(), ErrorStack> {\n    const SSL_CTRL_CHAIN_CERT: i32 = 89;\n    unsafe {\n        cvt(SSL_ctrl(\n            ssl.as_ptr(),\n            SSL_CTRL_CHAIN_CERT,\n            1, // increase the ref count of X509 so that ssl can outlive X509StoreRef\n            cert.as_ptr() as *mut c_void,\n        ))?;\n    }\n    Ok(())\n}\n\n/// Set renegotiation\n///\n/// This function is specific to BoringSSL. This function is noop for OpenSSL.\npub fn ssl_set_renegotiate_mode_freely(_ssl: &mut SslRef) {}\n\n/// Set the curves/groups of `ssl`\n///\n/// See [set_groups_list](https://www.openssl.org/docs/manmaster/man3/SSL_CTX_set1_curves.html).\npub fn ssl_set_groups_list(ssl: &mut SslRef, groups: &str) -> Result<(), ErrorStack> {\n    if groups.contains('\\0') {\n        return Err(ErrorStack::get());\n    }\n    let groups = CString::new(groups).map_err(|_| ErrorStack::get())?;\n    unsafe {\n        cvt(SSL_ctrl(\n            ssl.as_ptr(),\n            SSL_CTRL_SET_GROUPS_LIST,\n            0,\n            groups.as_ptr() as *mut c_void,\n        ))?;\n    }\n    Ok(())\n}\n\n/// Set's whether a second keyshare to be sent in client hello when PQ is used.\n///\n/// This function is specific to BoringSSL. This function is noop for OpenSSL.\npub fn ssl_use_second_key_share(_ssl: &mut SslRef, _enabled: bool) {}\n\n/// Clear the error stack\n///\n/// SSL calls should check and clear the OpenSSL error stack. But some calls fail to do so.\n/// This causes the next unrelated SSL call to fail due to the leftover errors. This function allows\n/// caller to clear the error stack before performing SSL calls to avoid this issue.\npub fn clear_error_stack() {\n    let _ = ErrorStack::get();\n}\n\n/// Create a new [Ssl] from &[SslAcceptor]\n///\n/// this function is to unify the interface between this crate and [`pingora-boringssl`](https://docs.rs/pingora-boringssl)\npub fn ssl_from_acceptor(acceptor: &SslAcceptor) -> Result<Ssl, ErrorStack> {\n    Ssl::new(acceptor.context())\n}\n\n/// Suspend the TLS handshake when a certificate is needed.\n///\n/// This function will cause tls handshake to pause and return the error: SSL_ERROR_WANT_X509_LOOKUP.\n/// The caller should set the certificate and then call [unblock_ssl_cert()] before continue the\n/// handshake on the tls connection.\npub fn suspend_when_need_ssl_cert(ssl: &mut SslRef) {\n    unsafe {\n        SSL_set_cert_cb(ssl.as_ptr(), Some(raw_cert_block), std::ptr::null_mut());\n    }\n}\n\n/// Unblock a TLS handshake after the certificate is set.\n///\n/// The user should continue to call tls handshake after this function is called.\npub fn unblock_ssl_cert(ssl: &mut SslRef) {\n    unsafe {\n        SSL_set_cert_cb(ssl.as_ptr(), None, std::ptr::null_mut());\n    }\n}\n\n// Just block the handshake\nextern \"C\" fn raw_cert_block(_ssl: *mut openssl_sys::SSL, _arg: *mut c_void) -> c_int {\n    -1\n}\n\n/// Whether the TLS error is SSL_ERROR_WANT_X509_LOOKUP\npub fn is_suspended_for_cert(error: &openssl::ssl::Error) -> bool {\n    error.code().as_raw() == openssl_sys::SSL_ERROR_WANT_X509_LOOKUP\n}\n\n#[allow(clippy::mut_from_ref)]\n/// Get a mutable SslRef ouf of SslRef, which is a missing functionality even when holding &mut SslStream\n/// # Safety\n/// the caller needs to make sure that they hold a &mut SslStream (or other types of mutable ref to the Ssl)\npub unsafe fn ssl_mut(ssl: &SslRef) -> &mut SslRef {\n    SslRef::from_ptr_mut(ssl.as_ptr())\n}\n\n#[cfg(test)]\nmod tests {\n    use super::*;\n    use openssl::ssl::{SslContextBuilder, SslMethod};\n\n    #[test]\n    fn test_ssl_set_groups_list() {\n        let ctx_builder = SslContextBuilder::new(SslMethod::tls()).unwrap();\n        let ssl = Ssl::new(&ctx_builder.build()).unwrap();\n        let ssl_ref = unsafe { ssl_mut(&ssl) };\n\n        // Valid input\n        assert!(ssl_set_groups_list(ssl_ref, \"P-256:P-384\").is_ok());\n\n        // Invalid input (contains null byte)\n        assert!(ssl_set_groups_list(ssl_ref, \"P-256\\0P-384\").is_err());\n    }\n}\n"
  },
  {
    "path": "pingora-openssl/src/lib.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! The OpenSSL API compatibility layer.\n//!\n//! This crate aims at making [openssl] APIs interchangeable with [boring](https://docs.rs/boring/latest/boring/).\n//! In other words, this crate and [`pingora-boringssl`](https://docs.rs/pingora-boringssl) expose identical rust APIs.\n\n#![warn(clippy::all)]\n\nuse openssl as ssl_lib;\npub use openssl_sys as ssl_sys;\npub use tokio_openssl as tokio_ssl;\npub mod ext;\n\n// export commonly used libs\npub use ssl_lib::dh;\npub use ssl_lib::error;\npub use ssl_lib::hash;\npub use ssl_lib::nid;\npub use ssl_lib::pkey;\npub use ssl_lib::ssl;\npub use ssl_lib::x509;\n"
  },
  {
    "path": "pingora-pool/Cargo.toml",
    "content": "[package]\nname = \"pingora-pool\"\nversion = \"0.8.0\"\nauthors = [\"Yuchen Wu <yuchen@cloudflare.com>\"]\nlicense = \"Apache-2.0\"\nedition = \"2021\"\nrepository = \"https://github.com/cloudflare/pingora\"\ncategories = [\"network-programming\"]\nkeywords = [\"async\", \"pooling\", \"pingora\"]\ndescription = \"\"\"\nA connection pool system for connection reuse.\n\"\"\"\n\n# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html\n[lib]\nname = \"pingora_pool\"\npath = \"src/lib.rs\"\n\n[dependencies]\ntokio = { workspace = true, features = [\"sync\", \"io-util\"] }\nthread_local = \"1.0\"\nlru = { workspace = true }\nlog = { workspace = true }\nparking_lot = \"0.12\"\ncrossbeam-queue = \"0.3\"\npingora-timeout = { version = \"0.8.0\", path = \"../pingora-timeout\" }\n\n[dev-dependencies]\ntokio-test = \"0.4\"\n"
  },
  {
    "path": "pingora-pool/LICENSE",
    "content": "\n                                 Apache License\n                           Version 2.0, January 2004\n                        http://www.apache.org/licenses/\n\n   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION\n\n   1. Definitions.\n\n      \"License\" shall mean the terms and conditions for use, reproduction,\n      and distribution as defined by Sections 1 through 9 of this document.\n\n      \"Licensor\" shall mean the copyright owner or entity authorized by\n      the copyright owner that is granting the License.\n\n      \"Legal Entity\" shall mean the union of the acting entity and all\n      other entities that control, are controlled by, or are under common\n      control with that entity. For the purposes of this definition,\n      \"control\" means (i) the power, direct or indirect, to cause the\n      direction or management of such entity, whether by contract or\n      otherwise, or (ii) ownership of fifty percent (50%) or more of the\n      outstanding shares, or (iii) beneficial ownership of such entity.\n\n      \"You\" (or \"Your\") shall mean an individual or Legal Entity\n      exercising permissions granted by this License.\n\n      \"Source\" form shall mean the preferred form for making modifications,\n      including but not limited to software source code, documentation\n      source, and configuration files.\n\n      \"Object\" form shall mean any form resulting from mechanical\n      transformation or translation of a Source form, including but\n      not limited to compiled object code, generated documentation,\n      and conversions to other media types.\n\n      \"Work\" shall mean the work of authorship, whether in Source or\n      Object form, made available under the License, as indicated by a\n      copyright notice that is included in or attached to the work\n      (an example is provided in the Appendix below).\n\n      \"Derivative Works\" shall mean any work, whether in Source or Object\n      form, that is based on (or derived from) the Work and for which the\n      editorial revisions, annotations, elaborations, or other modifications\n      represent, as a whole, an original work of authorship. For the purposes\n      of this License, Derivative Works shall not include works that remain\n      separable from, or merely link (or bind by name) to the interfaces of,\n      the Work and Derivative Works thereof.\n\n      \"Contribution\" shall mean any work of authorship, including\n      the original version of the Work and any modifications or additions\n      to that Work or Derivative Works thereof, that is intentionally\n      submitted to Licensor for inclusion in the Work by the copyright owner\n      or by an individual or Legal Entity authorized to submit on behalf of\n      the copyright owner. For the purposes of this definition, \"submitted\"\n      means any form of electronic, verbal, or written communication sent\n      to the Licensor or its representatives, including but not limited to\n      communication on electronic mailing lists, source code control systems,\n      and issue tracking systems that are managed by, or on behalf of, the\n      Licensor for the purpose of discussing and improving the Work, but\n      excluding communication that is conspicuously marked or otherwise\n      designated in writing by the copyright owner as \"Not a Contribution.\"\n\n      \"Contributor\" shall mean Licensor and any individual or Legal Entity\n      on behalf of whom a Contribution has been received by Licensor and\n      subsequently incorporated within the Work.\n\n   2. Grant of Copyright License. Subject to the terms and conditions of\n      this License, each Contributor hereby grants to You a perpetual,\n      worldwide, non-exclusive, no-charge, royalty-free, irrevocable\n      copyright license to reproduce, prepare Derivative Works of,\n      publicly display, publicly perform, sublicense, and distribute the\n      Work and such Derivative Works in Source or Object form.\n\n   3. Grant of Patent License. Subject to the terms and conditions of\n      this License, each Contributor hereby grants to You a perpetual,\n      worldwide, non-exclusive, no-charge, royalty-free, irrevocable\n      (except as stated in this section) patent license to make, have made,\n      use, offer to sell, sell, import, and otherwise transfer the Work,\n      where such license applies only to those patent claims licensable\n      by such Contributor that are necessarily infringed by their\n      Contribution(s) alone or by combination of their Contribution(s)\n      with the Work to which such Contribution(s) was submitted. If You\n      institute patent litigation against any entity (including a\n      cross-claim or counterclaim in a lawsuit) alleging that the Work\n      or a Contribution incorporated within the Work constitutes direct\n      or contributory patent infringement, then any patent licenses\n      granted to You under this License for that Work shall terminate\n      as of the date such litigation is filed.\n\n   4. Redistribution. You may reproduce and distribute copies of the\n      Work or Derivative Works thereof in any medium, with or without\n      modifications, and in Source or Object form, provided that You\n      meet the following conditions:\n\n      (a) You must give any other recipients of the Work or\n          Derivative Works a copy of this License; and\n\n      (b) You must cause any modified files to carry prominent notices\n          stating that You changed the files; and\n\n      (c) You must retain, in the Source form of any Derivative Works\n          that You distribute, all copyright, patent, trademark, and\n          attribution notices from the Source form of the Work,\n          excluding those notices that do not pertain to any part of\n          the Derivative Works; and\n\n      (d) If the Work includes a \"NOTICE\" text file as part of its\n          distribution, then any Derivative Works that You distribute must\n          include a readable copy of the attribution notices contained\n          within such NOTICE file, excluding those notices that do not\n          pertain to any part of the Derivative Works, in at least one\n          of the following places: within a NOTICE text file distributed\n          as part of the Derivative Works; within the Source form or\n          documentation, if provided along with the Derivative Works; or,\n          within a display generated by the Derivative Works, if and\n          wherever such third-party notices normally appear. The contents\n          of the NOTICE file are for informational purposes only and\n          do not modify the License. You may add Your own attribution\n          notices within Derivative Works that You distribute, alongside\n          or as an addendum to the NOTICE text from the Work, provided\n          that such additional attribution notices cannot be construed\n          as modifying the License.\n\n      You may add Your own copyright statement to Your modifications and\n      may provide additional or different license terms and conditions\n      for use, reproduction, or distribution of Your modifications, or\n      for any such Derivative Works as a whole, provided Your use,\n      reproduction, and distribution of the Work otherwise complies with\n      the conditions stated in this License.\n\n   5. Submission of Contributions. Unless You explicitly state otherwise,\n      any Contribution intentionally submitted for inclusion in the Work\n      by You to the Licensor shall be under the terms and conditions of\n      this License, without any additional terms or conditions.\n      Notwithstanding the above, nothing herein shall supersede or modify\n      the terms of any separate license agreement you may have executed\n      with Licensor regarding such Contributions.\n\n   6. Trademarks. This License does not grant permission to use the trade\n      names, trademarks, service marks, or product names of the Licensor,\n      except as required for reasonable and customary use in describing the\n      origin of the Work and reproducing the content of the NOTICE file.\n\n   7. Disclaimer of Warranty. Unless required by applicable law or\n      agreed to in writing, Licensor provides the Work (and each\n      Contributor provides its Contributions) on an \"AS IS\" BASIS,\n      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or\n      implied, including, without limitation, any warranties or conditions\n      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A\n      PARTICULAR PURPOSE. You are solely responsible for determining the\n      appropriateness of using or redistributing the Work and assume any\n      risks associated with Your exercise of permissions under this License.\n\n   8. Limitation of Liability. In no event and under no legal theory,\n      whether in tort (including negligence), contract, or otherwise,\n      unless required by applicable law (such as deliberate and grossly\n      negligent acts) or agreed to in writing, shall any Contributor be\n      liable to You for damages, including any direct, indirect, special,\n      incidental, or consequential damages of any character arising as a\n      result of this License or out of the use or inability to use the\n      Work (including but not limited to damages for loss of goodwill,\n      work stoppage, computer failure or malfunction, or any and all\n      other commercial damages or losses), even if such Contributor\n      has been advised of the possibility of such damages.\n\n   9. Accepting Warranty or Additional Liability. While redistributing\n      the Work or Derivative Works thereof, You may choose to offer,\n      and charge a fee for, acceptance of support, warranty, indemnity,\n      or other liability obligations and/or rights consistent with this\n      License. However, in accepting such obligations, You may act only\n      on Your own behalf and on Your sole responsibility, not on behalf\n      of any other Contributor, and only if You agree to indemnify,\n      defend, and hold each Contributor harmless for any liability\n      incurred by, or claims asserted against, such Contributor by reason\n      of your accepting any such warranty or additional liability.\n\n   END OF TERMS AND CONDITIONS\n\n   APPENDIX: How to apply the Apache License to your work.\n\n      To apply the Apache License to your work, attach the following\n      boilerplate notice, with the fields enclosed by brackets \"[]\"\n      replaced with your own identifying information. (Don't include\n      the brackets!)  The text should be enclosed in the appropriate\n      comment syntax for the file format. We also recommend that a\n      file or class name and description of purpose be included on the\n      same \"printed page\" as the copyright notice for easier\n      identification within third-party archives.\n\n   Copyright [yyyy] [name of copyright owner]\n\n   Licensed under the Apache License, Version 2.0 (the \"License\");\n   you may not use this file except in compliance with the License.\n   You may obtain a copy of the License at\n\n       http://www.apache.org/licenses/LICENSE-2.0\n\n   Unless required by applicable law or agreed to in writing, software\n   distributed under the License is distributed on an \"AS IS\" BASIS,\n   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n   See the License for the specific language governing permissions and\n   limitations under the License.\n"
  },
  {
    "path": "pingora-pool/src/connection.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! Generic connection pooling\n\nuse log::{debug, warn};\nuse parking_lot::{Mutex, RwLock};\nuse pingora_timeout::{sleep, timeout};\nuse std::collections::HashMap;\nuse std::io;\nuse std::sync::Arc;\nuse std::time::Duration;\nuse tokio::io::{AsyncRead, AsyncReadExt};\nuse tokio::sync::{oneshot, watch, Notify, OwnedMutexGuard};\n\nuse super::lru::Lru;\n\ntype GroupKey = u64;\n#[cfg(unix)]\ntype ID = i32;\n#[cfg(windows)]\ntype ID = usize;\n\n/// the metadata of a connection\n#[derive(Clone, Debug)]\npub struct ConnectionMeta {\n    /// The group key. All connections under the same key are considered the same for connection reuse.\n    pub key: GroupKey,\n    /// The unique ID of a connection.\n    pub id: ID,\n}\n\nimpl ConnectionMeta {\n    /// Create a new [ConnectionMeta]\n    pub fn new(key: GroupKey, id: ID) -> Self {\n        ConnectionMeta { key, id }\n    }\n}\n\nstruct PoolConnection<S> {\n    pub notify_use: oneshot::Sender<bool>,\n    pub connection: S,\n}\n\nimpl<S> PoolConnection<S> {\n    pub fn new(notify_use: oneshot::Sender<bool>, connection: S) -> Self {\n        PoolConnection {\n            notify_use,\n            connection,\n        }\n    }\n\n    pub fn release(self) -> S {\n        // notify the idle watcher to release the connection\n        let _ = self.notify_use.send(true);\n        // wait for the watcher to release\n        self.connection\n    }\n}\n\nuse crossbeam_queue::ArrayQueue;\n\n/// A pool of exchangeable items\npub struct PoolNode<T> {\n    connections: Mutex<HashMap<ID, T>>,\n    // a small lock free queue to avoid lock contention\n    hot_queue: ArrayQueue<(ID, T)>,\n    // to avoid race between 2 evictions on the queue\n    hot_queue_remove_lock: Mutex<()>,\n    // TODO: store the GroupKey to avoid hash collision?\n}\n\n// Keep the queue size small because eviction is O(n) in the queue\nconst HOT_QUEUE_SIZE: usize = 16;\n\nimpl<T> PoolNode<T> {\n    /// Create a new [PoolNode]\n    pub fn new() -> Self {\n        PoolNode {\n            connections: Mutex::new(HashMap::new()),\n            hot_queue: ArrayQueue::new(HOT_QUEUE_SIZE),\n            hot_queue_remove_lock: Mutex::new(()),\n        }\n    }\n\n    /// Get any item from the pool\n    pub fn get_any(&self) -> Option<(ID, T)> {\n        let hot_conn = self.hot_queue.pop();\n        if hot_conn.is_some() {\n            return hot_conn;\n        }\n        let mut connections = self.connections.lock();\n        // find one connection, any connection will do\n        let id = match connections.iter().next() {\n            Some((k, _)) => *k, // OK to copy i32\n            None => return None,\n        };\n        // unwrap is safe since we just found it\n        let connection = connections.remove(&id).unwrap();\n        /* NOTE: we don't resize or drop empty connections hashmap\n         * We may want to do it if they consume too much memory\n         * maybe we should use trees to save memory */\n        Some((id, connection))\n        // connections.lock released here\n    }\n\n    /// Insert an item with the given unique ID into the pool\n    pub fn insert(&self, id: ID, conn: T) {\n        if let Err(node) = self.hot_queue.push((id, conn)) {\n            // hot queue is full\n            let mut connections = self.connections.lock();\n            connections.insert(node.0, node.1); // TODO: check dup\n        }\n    }\n\n    /// Returns `true` if the pool node contains no connections in either the hot queue\n    /// or the overflow hash map.\n    ///\n    /// # Concurrency note\n    ///\n    /// This check is not atomic across the two internal stores (`hot_queue` and\n    /// `connections`). Between checking one and the other, a concurrent `insert` or\n    /// `get_any` could change the state. This is acceptable because callers use\n    /// `is_empty` only as a hint to attempt cleanup, and always re-verify under\n    /// an exclusive (write) lock before actually removing the node from the parent\n    /// pool HashMap. A false-negative simply defers cleanup to the next opportunity;\n    /// a false-positive is largely mitigated by the re-check (see\n    /// [`ConnectionPool::try_remove_empty_node`] for residual race-window analysis).\n    pub fn is_empty(&self) -> bool {\n        // Check the lock-free queue first (cheap atomic load) to avoid acquiring\n        // the mutex in the common case where connections are present.\n        self.hot_queue.is_empty() && self.connections.lock().is_empty()\n    }\n\n    // This function acquires 2 locks and iterates over the entire hot queue.\n    // But it should be fine because remove() rarely happens on a busy PoolNode.\n    /// Remove the item associated with the id from the pool. The item is returned\n    /// if it is found and removed.\n    pub fn remove(&self, id: ID) -> Option<T> {\n        // check the table first as least recent used ones are likely there\n        let removed = self.connections.lock().remove(&id);\n        if removed.is_some() {\n            return removed;\n        } // lock drops here\n\n        let _queue_lock = self.hot_queue_remove_lock.lock();\n        // check the hot queue, note that the queue can be accessed in parallel by insert and get\n        let max_len = self.hot_queue.len();\n        for _ in 0..max_len {\n            if let Some((conn_id, conn)) = self.hot_queue.pop() {\n                if conn_id == id {\n                    // this is the item, it is already popped\n                    return Some(conn);\n                } else {\n                    // not this item, put back to hot queue, but it could also be full\n                    self.insert(conn_id, conn);\n                }\n            } else {\n                // other threads grab all the connections\n                return None;\n            }\n        }\n        None\n        // _queue_lock drops here\n    }\n}\n\n/// Connection pool\n///\n/// [ConnectionPool] holds reusable connections. A reusable connection is released to this pool to\n/// be picked up by another user/request.\npub struct ConnectionPool<S> {\n    // TODO: n-way pools to reduce lock contention\n    pool: RwLock<HashMap<GroupKey, Arc<PoolNode<PoolConnection<S>>>>>,\n    lru: Lru<ID, ConnectionMeta>,\n}\n\nimpl<S> ConnectionPool<S> {\n    /// Create a new [ConnectionPool] with a size limit.\n    ///\n    /// When a connection is released to this pool, the least recently used connection will be dropped.\n    pub fn new(size: usize) -> Self {\n        ConnectionPool {\n            pool: RwLock::new(HashMap::with_capacity(size)), // this is oversized since some connections will have the same key\n            lru: Lru::new(size),\n        }\n    }\n\n    /* get or create and insert a pool node for the hash key */\n    fn get_pool_node(&self, key: GroupKey) -> Arc<PoolNode<PoolConnection<S>>> {\n        {\n            let pool = self.pool.read();\n            if let Some(v) = pool.get(&key) {\n                return (*v).clone();\n            }\n        } // read lock released here\n\n        {\n            // write lock section\n            let mut pool = self.pool.write();\n            // check again since another task might have already added it\n            if let Some(v) = pool.get(&key) {\n                return (*v).clone();\n            }\n            let node = Arc::new(PoolNode::new());\n            let node_ret = node.clone();\n            pool.insert(key, node); // TODO: check dup\n            node_ret\n        }\n    }\n\n    /// Attempt to remove an empty [`PoolNode`] entry from the pool `HashMap`.\n    ///\n    /// This prevents unbounded growth of the pool map when many unique group keys\n    /// are seen over the lifetime of the pool (e.g. connecting to many distinct\n    /// upstreams). Without this cleanup, each unique `GroupKey` leaves an\n    /// empty `PoolNode` behind even after all its connections are gone.\n    ///\n    /// The method acquires the pool write lock and re-checks emptiness to avoid\n    /// removing a node that was concurrently repopulated between the caller's\n    /// initial `is_empty()` hint and this write-lock acquisition.\n    ///\n    /// # Race window\n    ///\n    /// There is a narrow window where another thread could have called\n    /// [`get_pool_node`] (obtaining a clone of the `Arc<PoolNode>`) just before\n    /// we remove the entry. If that thread then inserts a connection into the\n    /// now-orphaned node, the connection is dropped when the last `Arc` reference\n    /// goes away. This is benign: the `oneshot::Sender` inside the dropped\n    /// `PoolConnection` is also dropped, which resolves the corresponding\n    /// `watch_use` receiver in `idle_poll`/`idle_timeout`, causing a clean exit.\n    /// The next request to the same upstream simply creates a fresh connection.\n    /// This trade-off matches the existing concurrency model of the pool and is\n    /// consistent with how hyper-util and Go's `net/http` handle this case.\n    fn try_remove_empty_node(&self, key: GroupKey) {\n        let mut pool = self.pool.write();\n        if let Some(node) = pool.get(&key) {\n            if node.is_empty() {\n                pool.remove(&key);\n            }\n        }\n    }\n\n    // only remove from the pool because lru already removed it\n    fn pop_evicted(&self, meta: &ConnectionMeta) {\n        let pool_node = {\n            let pool = self.pool.read();\n            match pool.get(&meta.key) {\n                Some(v) => (*v).clone(),\n                None => {\n                    warn!(\"Fail to get pool node for {:?}\", meta);\n                    return;\n                } // nothing to pop, should return error?\n            }\n        }; // read lock released here\n\n        pool_node.remove(meta.id);\n        debug!(\"evict fd: {} from key {}\", meta.id, meta.key);\n\n        // Clean up the PoolNode entry if it is now empty, to prevent unbounded\n        // growth of the pool HashMap.\n        // The is_empty() check avoids acquiring the write lock in the common case\n        // where other connections still exist under this key.\n        if pool_node.is_empty() {\n            self.try_remove_empty_node(meta.key);\n        }\n    }\n\n    pub fn pop_closed(&self, meta: &ConnectionMeta) {\n        // NOTE: which of these should be done first?\n        self.pop_evicted(meta);\n        self.lru.pop(&meta.id);\n    }\n\n    /// Get a connection from this pool under the same group key\n    pub fn get(&self, key: &GroupKey) -> Option<S> {\n        let pool_node = {\n            let pool = self.pool.read();\n            match pool.get(key) {\n                Some(v) => (*v).clone(),\n                None => return None,\n            }\n        }; // read lock released here\n\n        if let Some((id, connection)) = pool_node.get_any() {\n            self.lru.pop(&id); // the notified is not needed\n\n            // Clean up the now-empty node. This path is important because when a\n            // connection is retrieved (not evicted), the idle_poll/idle_timeout\n            // tasks exit via the watch_use channel and never call pop_closed(),\n            // so pop_evicted's cleanup would never run for this key.\n            if pool_node.is_empty() {\n                self.try_remove_empty_node(*key);\n            }\n\n            Some(connection.release())\n        } else {\n            // The node exists but has no connections. Clean it up.\n            self.try_remove_empty_node(*key);\n            None\n        }\n    }\n\n    /// Release a connection to this pool for reuse\n    ///\n    /// - The returned [`Arc<Notify>`] will notify any listen when the connection is evicted from the pool.\n    /// - The returned [`oneshot::Receiver<bool>`] will notify when the connection is being picked up by [Self::get()].\n    pub fn put(\n        &self,\n        meta: &ConnectionMeta,\n        connection: S,\n    ) -> (Arc<Notify>, oneshot::Receiver<bool>) {\n        let (notify_close, replaced) = self.lru.add(meta.id, meta.clone());\n        if let Some(meta) = replaced {\n            self.pop_evicted(&meta);\n        };\n        let pool_node = self.get_pool_node(meta.key);\n        let (notify_use, watch_use) = oneshot::channel();\n        let connection = PoolConnection::new(notify_use, connection);\n        pool_node.insert(meta.id, connection);\n        (notify_close, watch_use)\n    }\n\n    /// Actively monitor the health of a connection that is already released to this pool\n    ///\n    /// When the connection breaks, or the optional `timeout` is reached this function will\n    /// remove it from the pool and drop the connection.\n    ///\n    /// If the connection is reused via [Self::get()] or being evicted, this function will just exit.\n    pub async fn idle_poll<Stream>(\n        &self,\n        connection: OwnedMutexGuard<Stream>,\n        meta: &ConnectionMeta,\n        timeout: Option<Duration>,\n        notify_evicted: Arc<Notify>,\n        watch_use: oneshot::Receiver<bool>,\n    ) where\n        Stream: AsyncRead + Unpin + Send,\n    {\n        let read_result = tokio::select! {\n            biased;\n            _ = watch_use => {\n                debug!(\"idle connection is being picked up\");\n                return\n            },\n            _ = notify_evicted.notified() => {\n                debug!(\"idle connection is being evicted\");\n                // TODO: gracefully close the connection?\n                return\n            }\n            read_result = read_with_timeout(connection , timeout) => read_result\n        };\n\n        match read_result {\n            Ok(n) => {\n                if n > 0 {\n                    warn!(\"Data received on idle client connection, close it\")\n                } else {\n                    debug!(\"Peer closed the idle connection or timeout\")\n                }\n            }\n\n            Err(e) => {\n                debug!(\"error with the idle connection, close it {:?}\", e);\n            }\n        }\n        // connection terminated from either peer or timer\n        self.pop_closed(meta);\n    }\n\n    /// Passively wait to close the connection after the timeout\n    ///\n    /// If this connection is not being picked up or evicted before the timeout is reach, this\n    /// function will remove it from the pool and close the connection.\n    pub async fn idle_timeout(\n        &self,\n        meta: &ConnectionMeta,\n        timeout: Option<Duration>,\n        notify_evicted: Arc<Notify>,\n        mut notify_closed: watch::Receiver<bool>,\n        watch_use: oneshot::Receiver<bool>,\n    ) {\n        tokio::select! {\n            biased;\n            _ = watch_use => {\n                debug!(\"idle connection is being picked up\");\n            },\n            _ = notify_evicted.notified() => {\n                debug!(\"idle connection is being evicted\");\n                // TODO: gracefully close the connection?\n            }\n            _ = notify_closed.changed() => {\n                // assume always changed from false to true\n                debug!(\"idle connection is being closed\");\n                self.pop_closed(meta);\n            }\n            // async expression is evaluated if timeout is None but it's never polled, set it to MAX\n            _ = sleep(timeout.unwrap_or(Duration::MAX)), if timeout.is_some() => {\n                debug!(\"idle connection is being evicted\");\n                self.pop_closed(meta);\n            }\n        };\n    }\n}\n\nasync fn read_with_timeout<S>(\n    mut connection: OwnedMutexGuard<S>,\n    timeout_duration: Option<Duration>,\n) -> io::Result<usize>\nwhere\n    S: AsyncRead + Unpin + Send,\n{\n    let mut buf = [0; 1];\n    let read_event = connection.read(&mut buf[..]);\n    match timeout_duration {\n        Some(d) => match timeout(d, read_event).await {\n            Ok(res) => res,\n            Err(e) => {\n                debug!(\"keepalive timeout {:?} reached, {:?}\", d, e);\n                Ok(0)\n            }\n        },\n        _ => read_event.await,\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use super::*;\n    use log::debug;\n    use tokio::sync::Mutex as AsyncMutex;\n    use tokio_test::io::{Builder, Mock};\n\n    #[tokio::test]\n    async fn test_lookup() {\n        let meta1 = ConnectionMeta::new(101, 1);\n        let value1 = \"v1\".to_string();\n        let meta2 = ConnectionMeta::new(102, 2);\n        let value2 = \"v2\".to_string();\n        let meta3 = ConnectionMeta::new(101, 3);\n        let value3 = \"v3\".to_string();\n        let cp: ConnectionPool<String> = ConnectionPool::new(3); //#CP3\n        cp.put(&meta1, value1.clone());\n        cp.put(&meta2, value2.clone());\n        cp.put(&meta3, value3.clone());\n\n        let found_b = cp.get(&meta2.key).unwrap();\n        assert_eq!(found_b, value2);\n\n        let found_a1 = cp.get(&meta1.key).unwrap();\n        let found_a2 = cp.get(&meta1.key).unwrap();\n\n        assert!(\n            found_a1 == value1 && found_a2 == value3 || found_a2 == value1 && found_a1 == value3\n        );\n    }\n\n    #[tokio::test]\n    async fn test_pop() {\n        let meta1 = ConnectionMeta::new(101, 1);\n        let value1 = \"v1\".to_string();\n        let meta2 = ConnectionMeta::new(102, 2);\n        let value2 = \"v2\".to_string();\n        let meta3 = ConnectionMeta::new(101, 3);\n        let value3 = \"v3\".to_string();\n        let cp: ConnectionPool<String> = ConnectionPool::new(3); //#CP3\n        cp.put(&meta1, value1);\n        cp.put(&meta2, value2);\n        cp.put(&meta3, value3.clone());\n\n        cp.pop_closed(&meta1);\n\n        let found_a1 = cp.get(&meta1.key).unwrap();\n        assert_eq!(found_a1, value3);\n\n        cp.pop_closed(&meta1);\n        assert!(cp.get(&meta1.key).is_none())\n    }\n\n    #[tokio::test]\n    async fn test_eviction() {\n        let meta1 = ConnectionMeta::new(101, 1);\n        let value1 = \"v1\".to_string();\n        let meta2 = ConnectionMeta::new(102, 2);\n        let value2 = \"v2\".to_string();\n        let meta3 = ConnectionMeta::new(101, 3);\n        let value3 = \"v3\".to_string();\n        let cp: ConnectionPool<String> = ConnectionPool::new(2);\n        let (notify_close1, _) = cp.put(&meta1, value1.clone());\n        let (notify_close2, _) = cp.put(&meta2, value2.clone());\n        let (notify_close3, _) = cp.put(&meta3, value3.clone()); // meta 1 should be evicted\n\n        let closed_item = tokio::select! {\n            _ = notify_close1.notified() => {debug!(\"notifier1\"); 1},\n            _ = notify_close2.notified() => {debug!(\"notifier2\"); 2},\n            _ = notify_close3.notified() => {debug!(\"notifier3\"); 3},\n        };\n        assert_eq!(closed_item, 1);\n\n        let found_a1 = cp.get(&meta1.key).unwrap();\n        assert_eq!(found_a1, value3);\n        assert_eq!(cp.get(&meta1.key), None)\n    }\n\n    #[tokio::test]\n    #[should_panic(expected = \"There is still data left to read.\")]\n    async fn test_read_close() {\n        let meta1 = ConnectionMeta::new(101, 1);\n        let mock_io1 = Arc::new(AsyncMutex::new(Builder::new().read(b\"garbage\").build()));\n        let meta2 = ConnectionMeta::new(102, 2);\n        let mock_io2 = Arc::new(AsyncMutex::new(\n            Builder::new().wait(Duration::from_secs(99)).build(),\n        ));\n        let meta3 = ConnectionMeta::new(101, 3);\n        let mock_io3 = Arc::new(AsyncMutex::new(\n            Builder::new().wait(Duration::from_secs(99)).build(),\n        ));\n        let cp: ConnectionPool<Arc<AsyncMutex<Mock>>> = ConnectionPool::new(3);\n        let (c1, u1) = cp.put(&meta1, mock_io1.clone());\n        let (c2, u2) = cp.put(&meta2, mock_io2.clone());\n        let (c3, u3) = cp.put(&meta3, mock_io3.clone());\n\n        let closed_item = tokio::select! {\n            _ = cp.idle_poll(mock_io1.try_lock_owned().unwrap(), &meta1, None, c1, u1) => {debug!(\"notifier1\"); 1},\n            _ = cp.idle_poll(mock_io2.try_lock_owned().unwrap(), &meta1, None, c2, u2) => {debug!(\"notifier2\"); 2},\n            _ = cp.idle_poll(mock_io3.try_lock_owned().unwrap(), &meta1, None, c3, u3) => {debug!(\"notifier3\"); 3},\n        };\n        assert_eq!(closed_item, 1);\n\n        let _ = cp.get(&meta1.key).unwrap(); // mock_io3 should be selected\n        assert!(cp.get(&meta1.key).is_none()) // mock_io1 should already be removed by idle_poll\n    }\n\n    #[tokio::test]\n    async fn test_read_timeout() {\n        let meta1 = ConnectionMeta::new(101, 1);\n        let mock_io1 = Arc::new(AsyncMutex::new(\n            Builder::new().wait(Duration::from_secs(99)).build(),\n        ));\n        let meta2 = ConnectionMeta::new(102, 2);\n        let mock_io2 = Arc::new(AsyncMutex::new(\n            Builder::new().wait(Duration::from_secs(99)).build(),\n        ));\n        let meta3 = ConnectionMeta::new(101, 3);\n        let mock_io3 = Arc::new(AsyncMutex::new(\n            Builder::new().wait(Duration::from_secs(99)).build(),\n        ));\n        let cp: ConnectionPool<Arc<AsyncMutex<Mock>>> = ConnectionPool::new(3);\n        let (c1, u1) = cp.put(&meta1, mock_io1.clone());\n        let (c2, u2) = cp.put(&meta2, mock_io2.clone());\n        let (c3, u3) = cp.put(&meta3, mock_io3.clone());\n\n        let closed_item = tokio::select! {\n            _ = cp.idle_poll(mock_io1.try_lock_owned().unwrap(), &meta1, Some(Duration::from_secs(1)), c1, u1) => {debug!(\"notifier1\"); 1},\n            _ = cp.idle_poll(mock_io2.try_lock_owned().unwrap(), &meta1, Some(Duration::from_secs(2)), c2, u2) => {debug!(\"notifier2\"); 2},\n            _ = cp.idle_poll(mock_io3.try_lock_owned().unwrap(), &meta1, Some(Duration::from_secs(3)), c3, u3) => {debug!(\"notifier3\"); 3},\n        };\n        assert_eq!(closed_item, 1);\n\n        let _ = cp.get(&meta1.key).unwrap(); // mock_io3 should be selected\n        assert!(cp.get(&meta1.key).is_none()) // mock_io1 should already be removed by idle_poll\n    }\n\n    #[tokio::test]\n    async fn test_evict_poll() {\n        let meta1 = ConnectionMeta::new(101, 1);\n        let mock_io1 = Arc::new(AsyncMutex::new(\n            Builder::new().wait(Duration::from_secs(99)).build(),\n        ));\n        let meta2 = ConnectionMeta::new(102, 2);\n        let mock_io2 = Arc::new(AsyncMutex::new(\n            Builder::new().wait(Duration::from_secs(99)).build(),\n        ));\n        let meta3 = ConnectionMeta::new(101, 3);\n        let mock_io3 = Arc::new(AsyncMutex::new(\n            Builder::new().wait(Duration::from_secs(99)).build(),\n        ));\n        let cp: ConnectionPool<Arc<AsyncMutex<Mock>>> = ConnectionPool::new(2);\n        let (c1, u1) = cp.put(&meta1, mock_io1.clone());\n        let (c2, u2) = cp.put(&meta2, mock_io2.clone());\n        let (c3, u3) = cp.put(&meta3, mock_io3.clone()); // 1 should be evicted at this point\n\n        let closed_item = tokio::select! {\n            _ = cp.idle_poll(mock_io1.try_lock_owned().unwrap(), &meta1, None, c1, u1) => {debug!(\"notifier1\"); 1},\n            _ = cp.idle_poll(mock_io2.try_lock_owned().unwrap(), &meta1, None, c2, u2) => {debug!(\"notifier2\"); 2},\n            _ = cp.idle_poll(mock_io3.try_lock_owned().unwrap(), &meta1, None, c3, u3) => {debug!(\"notifier3\"); 3},\n        };\n        assert_eq!(closed_item, 1);\n\n        let _ = cp.get(&meta1.key).unwrap(); // mock_io3 should be selected\n        assert!(cp.get(&meta1.key).is_none()) // mock_io1 should already be removed by idle_poll\n    }\n\n    #[test]\n    fn test_pool_node_is_empty() {\n        let node: PoolNode<String> = PoolNode::new();\n        assert!(node.is_empty(), \"newly created node should be empty\");\n\n        node.insert(1, \"v1\".to_string());\n        assert!(!node.is_empty(), \"node with one item should not be empty\");\n\n        // get_any removes the item\n        let item = node.get_any();\n        assert!(item.is_some());\n        assert!(node.is_empty(), \"node should be empty after get_any\");\n\n        // insert then remove by id\n        node.insert(2, \"v2\".to_string());\n        assert!(!node.is_empty());\n\n        let removed = node.remove(2);\n        assert!(removed.is_some());\n        assert!(node.is_empty(), \"node should be empty after remove\");\n    }\n\n    #[test]\n    fn test_pool_node_is_empty_overflow_to_connections() {\n        // Fill the hot queue (capacity = HOT_QUEUE_SIZE = 16), then overflow\n        // into the connections HashMap, and verify is_empty drains both.\n        let node: PoolNode<String> = PoolNode::new();\n\n        for i in 0..(HOT_QUEUE_SIZE as i32 + 4) {\n            node.insert(i, format!(\"v{i}\"));\n        }\n        assert!(!node.is_empty());\n\n        // Drain all items via get_any\n        while node.get_any().is_some() {}\n        assert!(node.is_empty(), \"node should be empty after draining all\");\n    }\n\n    #[tokio::test]\n    async fn test_empty_node_removed_after_pop_closed() {\n        // Reproducer from GitHub issue #748: a single connection is added and\n        // then closed. The PoolNode entry in the pool HashMap must be removed.\n        let meta = ConnectionMeta::new(101, 1);\n        let cp: ConnectionPool<String> = ConnectionPool::new(2);\n        cp.put(&meta, \"v1\".to_string());\n\n        assert_eq!(cp.pool.read().len(), 1, \"pool should have 1 node\");\n\n        cp.pop_closed(&meta);\n\n        assert_eq!(\n            cp.pool.read().len(),\n            0,\n            \"empty PoolNode should be removed after pop_closed\"\n        );\n    }\n\n    #[tokio::test]\n    async fn test_empty_node_removed_after_get() {\n        // When the last connection is retrieved via get(), the PoolNode should\n        // be cleaned up. This path is distinct from pop_closed because the\n        // idle_poll/idle_timeout tasks exit via the watch_use channel and never\n        // call pop_closed.\n        let meta = ConnectionMeta::new(101, 1);\n        let cp: ConnectionPool<String> = ConnectionPool::new(2);\n        cp.put(&meta, \"v1\".to_string());\n\n        assert_eq!(cp.pool.read().len(), 1);\n\n        let conn = cp.get(&meta.key);\n        assert!(conn.is_some());\n\n        assert_eq!(\n            cp.pool.read().len(),\n            0,\n            \"empty PoolNode should be removed after get() takes the last connection\"\n        );\n    }\n\n    #[tokio::test]\n    async fn test_empty_node_removed_when_get_finds_empty_node() {\n        // If a node exists but has no connections (e.g. they were all evicted\n        // by the LRU), get() should clean up the empty node.\n        let meta1 = ConnectionMeta::new(101, 1);\n        let meta2 = ConnectionMeta::new(101, 2);\n        let cp: ConnectionPool<String> = ConnectionPool::new(4);\n        cp.put(&meta1, \"v1\".to_string());\n        cp.put(&meta2, \"v2\".to_string());\n\n        // Remove both connections via pop_closed, but the first pop_closed\n        // won't remove the node since meta2 is still there.\n        cp.pop_closed(&meta1);\n        assert_eq!(cp.pool.read().len(), 1, \"node should still exist\");\n\n        cp.pop_closed(&meta2);\n        assert_eq!(\n            cp.pool.read().len(),\n            0,\n            \"node should be removed after last connection is popped\"\n        );\n    }\n\n    #[tokio::test]\n    async fn test_node_not_removed_when_connections_remain() {\n        // Removing one connection from a node that has others must NOT remove\n        // the node itself.\n        let meta1 = ConnectionMeta::new(101, 1);\n        let meta2 = ConnectionMeta::new(101, 2);\n        let cp: ConnectionPool<String> = ConnectionPool::new(4);\n        cp.put(&meta1, \"v1\".to_string());\n        cp.put(&meta2, \"v2\".to_string());\n\n        cp.pop_closed(&meta1);\n\n        assert!(\n            cp.pool.read().contains_key(&101),\n            \"node should still exist because meta2's connection is still in it\"\n        );\n        assert_eq!(cp.pool.read().len(), 1);\n\n        // The remaining connection should still be retrievable\n        let conn = cp.get(&meta1.key);\n        assert!(conn.is_some());\n    }\n\n    #[tokio::test]\n    async fn test_empty_node_cleanup_only_affects_target_key() {\n        // Cleaning up an empty node for one key must not affect other keys.\n        let meta_a = ConnectionMeta::new(101, 1);\n        let meta_b = ConnectionMeta::new(202, 2);\n        let cp: ConnectionPool<String> = ConnectionPool::new(4);\n        cp.put(&meta_a, \"a\".to_string());\n        cp.put(&meta_b, \"b\".to_string());\n\n        assert_eq!(cp.pool.read().len(), 2);\n\n        // Remove all connections for key 101\n        cp.pop_closed(&meta_a);\n\n        assert_eq!(\n            cp.pool.read().len(),\n            1,\n            \"only key 101's empty node should be removed\"\n        );\n        assert!(!cp.pool.read().contains_key(&101), \"key 101 should be gone\");\n        assert!(cp.pool.read().contains_key(&202), \"key 202 should remain\");\n\n        // key 202's connection should still be retrievable\n        let conn = cp.get(&meta_b.key);\n        assert_eq!(conn, Some(\"b\".to_string()));\n    }\n\n    #[tokio::test]\n    async fn test_empty_node_cleaned_after_lru_eviction() {\n        // When LRU eviction removes the last connection for a key, the empty\n        // node should be cleaned up by pop_evicted (called from put()).\n        let meta1 = ConnectionMeta::new(101, 1);\n        let meta2 = ConnectionMeta::new(202, 2);\n        let cp: ConnectionPool<String> = ConnectionPool::new(1);\n\n        cp.put(&meta1, \"v1\".to_string());\n        assert_eq!(cp.pool.read().len(), 1);\n\n        // This put evicts meta1 (LRU size = 1), making key 101's node empty.\n        cp.put(&meta2, \"v2\".to_string());\n\n        assert!(\n            !cp.pool.read().contains_key(&101),\n            \"key 101's empty node should be removed after its only connection was evicted\"\n        );\n        assert!(cp.pool.read().contains_key(&202));\n    }\n\n    #[tokio::test]\n    async fn test_node_reusable_after_cleanup() {\n        // After an empty node is cleaned up, inserting a new connection for the\n        // same key should work correctly (a new PoolNode is created).\n        let meta1 = ConnectionMeta::new(101, 1);\n        let cp: ConnectionPool<String> = ConnectionPool::new(4);\n        cp.put(&meta1, \"first\".to_string());\n\n        cp.pop_closed(&meta1);\n        assert_eq!(cp.pool.read().len(), 0, \"node should be cleaned up\");\n\n        // Re-insert for the same key\n        let meta2 = ConnectionMeta::new(101, 2);\n        cp.put(&meta2, \"second\".to_string());\n\n        assert_eq!(cp.pool.read().len(), 1);\n        let conn = cp.get(&meta2.key);\n        assert_eq!(conn, Some(\"second\".to_string()));\n\n        assert_eq!(\n            cp.pool.read().len(),\n            0,\n            \"node should be cleaned up again after get\"\n        );\n    }\n}\n"
  },
  {
    "path": "pingora-pool/src/lib.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! Generic connection pooling\n//!\n//! The pool is optimized for high concurrency, high RPS use cases. Each connection group has a\n//! lock free hot pool to reduce the lock contention when some connections are reused and released\n//! very frequently.\n\n#![warn(clippy::all)]\n#![allow(clippy::new_without_default)]\n#![allow(clippy::type_complexity)]\n\nmod connection;\nmod lru;\n\npub use connection::{ConnectionMeta, ConnectionPool, PoolNode};\n"
  },
  {
    "path": "pingora-pool/src/lru.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\nuse core::hash::Hash;\nuse lru::LruCache;\nuse parking_lot::RwLock;\nuse std::cell::RefCell;\nuse std::sync::atomic::{AtomicBool, Ordering::Relaxed};\nuse std::sync::Arc;\nuse thread_local::ThreadLocal;\nuse tokio::sync::Notify;\n\npub struct Node<T> {\n    pub close_notifier: Arc<Notify>,\n    pub meta: T,\n}\n\nimpl<T> Node<T> {\n    pub fn new(meta: T) -> Self {\n        Node {\n            close_notifier: Arc::new(Notify::new()),\n            meta,\n        }\n    }\n\n    pub fn notify_close(&self) {\n        self.close_notifier.notify_one();\n    }\n}\n\npub struct Lru<K, T>\nwhere\n    K: Send,\n    T: Send,\n{\n    lru: RwLock<ThreadLocal<RefCell<LruCache<K, Node<T>>>>>,\n    size: usize,\n    drain: AtomicBool,\n}\n\nimpl<K, T> Lru<K, T>\nwhere\n    K: Hash + Eq + Send,\n    T: Send,\n{\n    pub fn new(size: usize) -> Self {\n        Lru {\n            lru: RwLock::new(ThreadLocal::new()),\n            size,\n            drain: AtomicBool::new(false),\n        }\n    }\n\n    // put a node in and return the meta of the replaced node\n    pub fn put(&self, key: K, value: Node<T>) -> Option<T> {\n        if self.drain.load(Relaxed) {\n            value.notify_close(); // sort of hack to simulate being evicted right away\n            return None;\n        }\n        let lru = self.lru.read(); /* read lock */\n        let lru_cache = &mut *(lru\n            .get_or(|| RefCell::new(LruCache::unbounded()))\n            .borrow_mut());\n        lru_cache.put(key, value);\n        if lru_cache.len() > self.size {\n            match lru_cache.pop_lru() {\n                Some((_, v)) => {\n                    // TODO: drop the lock here?\n                    v.notify_close();\n                    return Some(v.meta);\n                }\n                None => return None,\n            }\n        }\n        None\n        /* read lock dropped */\n    }\n\n    pub fn add(&self, key: K, meta: T) -> (Arc<Notify>, Option<T>) {\n        let node = Node::new(meta);\n        let notifier = node.close_notifier.clone();\n        // TODO: check if the key is already in it\n        (notifier, self.put(key, node))\n    }\n\n    pub fn pop(&self, key: &K) -> Option<Node<T>> {\n        let lru = self.lru.read(); /* read lock */\n        let lru_cache = &mut *(lru\n            .get_or(|| RefCell::new(LruCache::unbounded()))\n            .borrow_mut());\n        lru_cache.pop(key)\n        /* read lock dropped */\n    }\n\n    #[allow(dead_code)]\n    pub fn drain(&self) {\n        self.drain.store(true, Relaxed);\n\n        /* drain need to go through all the local lru cache objects\n         * acquire an exclusive write lock to make it safe */\n        let mut lru = self.lru.write(); /* write lock */\n        let lru_cache_iter = lru.iter_mut();\n        for lru_cache_rc in lru_cache_iter {\n            let mut lru_cache = lru_cache_rc.borrow_mut();\n            for (_, item) in lru_cache.iter() {\n                item.notify_close();\n            }\n            lru_cache.clear();\n        }\n        /* write lock dropped */\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use super::*;\n    use log::debug;\n\n    #[tokio::test]\n    async fn test_evict_close() {\n        let pool: Lru<i32, ()> = Lru::new(2);\n        let (notifier1, _) = pool.add(1, ());\n        let (notifier2, _) = pool.add(2, ());\n        let (notifier3, _) = pool.add(3, ());\n        let closed_item = tokio::select! {\n            _ = notifier1.notified() => {debug!(\"notifier1\"); 1},\n            _ = notifier2.notified() => {debug!(\"notifier2\"); 2},\n            _ = notifier3.notified() => {debug!(\"notifier3\"); 3},\n        };\n        assert_eq!(closed_item, 1);\n    }\n\n    #[tokio::test]\n    async fn test_evict_close_with_pop() {\n        let pool: Lru<i32, ()> = Lru::new(2);\n        let (notifier1, _) = pool.add(1, ());\n        let (notifier2, _) = pool.add(2, ());\n        pool.pop(&1);\n        let (notifier3, _) = pool.add(3, ());\n        let (notifier4, _) = pool.add(4, ());\n        let closed_item = tokio::select! {\n            _ = notifier1.notified() => {debug!(\"notifier1\"); 1},\n            _ = notifier2.notified() => {debug!(\"notifier2\"); 2},\n            _ = notifier3.notified() => {debug!(\"notifier3\"); 3},\n            _ = notifier4.notified() => {debug!(\"notifier4\"); 4},\n        };\n        assert_eq!(closed_item, 2);\n    }\n\n    #[tokio::test]\n    async fn test_drain() {\n        let pool: Lru<i32, ()> = Lru::new(4);\n        let (notifier1, _) = pool.add(1, ());\n        let (notifier2, _) = pool.add(2, ());\n        let (notifier3, _) = pool.add(3, ());\n        pool.drain();\n        let (notifier4, _) = pool.add(4, ());\n\n        tokio::join!(\n            notifier1.notified(),\n            notifier2.notified(),\n            notifier3.notified(),\n            notifier4.notified()\n        );\n    }\n}\n"
  },
  {
    "path": "pingora-proxy/Cargo.toml",
    "content": "[package]\nname = \"pingora-proxy\"\nversion = \"0.8.0\"\nauthors = [\"Yuchen Wu <yuchen@cloudflare.com>\"]\nlicense = \"Apache-2.0\"\nedition = \"2021\"\nrust-version = \"1.84\"\nrepository = \"https://github.com/cloudflare/pingora\"\ncategories = [\"asynchronous\", \"network-programming\"]\nkeywords = [\"async\", \"http\", \"proxy\", \"pingora\"]\nexclude = [\"tests/*\"]\ndescription = \"\"\"\nPingora HTTP proxy APIs and traits.\n\"\"\"\n\n# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html\n[lib]\nname = \"pingora_proxy\"\npath = \"src/lib.rs\"\n\n[dependencies]\npingora-error = { version = \"0.8.0\", path = \"../pingora-error\" }\npingora-core = { version = \"0.8.0\", path = \"../pingora-core\", default-features = false }\npingora-cache = { version = \"0.8.0\", path = \"../pingora-cache\", default-features = false }\ntokio = { workspace = true, features = [\"macros\", \"net\"] }\npingora-http = { version = \"0.8.0\", path = \"../pingora-http\" }\nhttp = { workspace = true }\nfutures = \"0.3\"\nbytes = { workspace = true }\nasync-trait = { workspace = true }\nlog = { workspace = true }\nh2 = { workspace = true }\nonce_cell = { workspace = true }\nclap = { version = \"4\", features = [\"derive\"] }\nregex = \"1\"\nrand = \"0.8\"\n\n[dev-dependencies]\nreqwest = { version = \"0.11\", features = [\n    \"gzip\",\n    \"rustls-tls\",\n], default-features = false }\nhttparse = { workspace = true }\ntokio-test = \"0.4\"\nenv_logger = \"0.11\"\nhyper = \"0.14\"\ntokio-tungstenite = \"0.20.1\"\npingora-limits = { version = \"0.8.0\", path = \"../pingora-limits\" }\npingora-load-balancing = { version = \"0.8.0\", path = \"../pingora-load-balancing\", default-features=false }\nprometheus = \"0\"\nfutures-util = \"0.3\"\nserde = { version = \"1.0\", features = [\"derive\"] }\nserde_json = \"1.0\"\nserde_yaml = \"0.9\"\n\n[target.'cfg(unix)'.dev-dependencies]\nhyperlocal = \"0.8\"\n\n[features]\ndefault = []\nopenssl = [\"pingora-core/openssl\", \"pingora-cache/openssl\", \"openssl_derived\"]\nboringssl = [\n    \"pingora-core/boringssl\",\n    \"pingora-cache/boringssl\",\n    \"openssl_derived\",\n]\nrustls = [\"pingora-core/rustls\", \"pingora-cache/rustls\", \"any_tls\"]\ns2n = [\"pingora-core/s2n\", \"pingora-cache/s2n\", \"any_tls\"]\nopenssl_derived = [\"any_tls\"]\nany_tls = []\nsentry = [\"pingora-core/sentry\"]\nconnection_filter = [\"pingora-core/connection_filter\"]\n\n[[example]]\nname = \"connection_filter\"\nrequired-features = [\"connection_filter\"]\n\n# or locally cargo doc --config \"build.rustdocflags='--cfg doc_async_trait'\"\n[package.metadata.docs.rs]\nrustdoc-args = [\"--cfg\", \"doc_async_trait\"]\n\n[lints.rust]\nunexpected_cfgs = { level = \"warn\", check-cfg = ['cfg(doc_async_trait)'] }\n"
  },
  {
    "path": "pingora-proxy/LICENSE",
    "content": "\n                                 Apache License\n                           Version 2.0, January 2004\n                        http://www.apache.org/licenses/\n\n   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION\n\n   1. Definitions.\n\n      \"License\" shall mean the terms and conditions for use, reproduction,\n      and distribution as defined by Sections 1 through 9 of this document.\n\n      \"Licensor\" shall mean the copyright owner or entity authorized by\n      the copyright owner that is granting the License.\n\n      \"Legal Entity\" shall mean the union of the acting entity and all\n      other entities that control, are controlled by, or are under common\n      control with that entity. For the purposes of this definition,\n      \"control\" means (i) the power, direct or indirect, to cause the\n      direction or management of such entity, whether by contract or\n      otherwise, or (ii) ownership of fifty percent (50%) or more of the\n      outstanding shares, or (iii) beneficial ownership of such entity.\n\n      \"You\" (or \"Your\") shall mean an individual or Legal Entity\n      exercising permissions granted by this License.\n\n      \"Source\" form shall mean the preferred form for making modifications,\n      including but not limited to software source code, documentation\n      source, and configuration files.\n\n      \"Object\" form shall mean any form resulting from mechanical\n      transformation or translation of a Source form, including but\n      not limited to compiled object code, generated documentation,\n      and conversions to other media types.\n\n      \"Work\" shall mean the work of authorship, whether in Source or\n      Object form, made available under the License, as indicated by a\n      copyright notice that is included in or attached to the work\n      (an example is provided in the Appendix below).\n\n      \"Derivative Works\" shall mean any work, whether in Source or Object\n      form, that is based on (or derived from) the Work and for which the\n      editorial revisions, annotations, elaborations, or other modifications\n      represent, as a whole, an original work of authorship. For the purposes\n      of this License, Derivative Works shall not include works that remain\n      separable from, or merely link (or bind by name) to the interfaces of,\n      the Work and Derivative Works thereof.\n\n      \"Contribution\" shall mean any work of authorship, including\n      the original version of the Work and any modifications or additions\n      to that Work or Derivative Works thereof, that is intentionally\n      submitted to Licensor for inclusion in the Work by the copyright owner\n      or by an individual or Legal Entity authorized to submit on behalf of\n      the copyright owner. For the purposes of this definition, \"submitted\"\n      means any form of electronic, verbal, or written communication sent\n      to the Licensor or its representatives, including but not limited to\n      communication on electronic mailing lists, source code control systems,\n      and issue tracking systems that are managed by, or on behalf of, the\n      Licensor for the purpose of discussing and improving the Work, but\n      excluding communication that is conspicuously marked or otherwise\n      designated in writing by the copyright owner as \"Not a Contribution.\"\n\n      \"Contributor\" shall mean Licensor and any individual or Legal Entity\n      on behalf of whom a Contribution has been received by Licensor and\n      subsequently incorporated within the Work.\n\n   2. Grant of Copyright License. Subject to the terms and conditions of\n      this License, each Contributor hereby grants to You a perpetual,\n      worldwide, non-exclusive, no-charge, royalty-free, irrevocable\n      copyright license to reproduce, prepare Derivative Works of,\n      publicly display, publicly perform, sublicense, and distribute the\n      Work and such Derivative Works in Source or Object form.\n\n   3. Grant of Patent License. Subject to the terms and conditions of\n      this License, each Contributor hereby grants to You a perpetual,\n      worldwide, non-exclusive, no-charge, royalty-free, irrevocable\n      (except as stated in this section) patent license to make, have made,\n      use, offer to sell, sell, import, and otherwise transfer the Work,\n      where such license applies only to those patent claims licensable\n      by such Contributor that are necessarily infringed by their\n      Contribution(s) alone or by combination of their Contribution(s)\n      with the Work to which such Contribution(s) was submitted. If You\n      institute patent litigation against any entity (including a\n      cross-claim or counterclaim in a lawsuit) alleging that the Work\n      or a Contribution incorporated within the Work constitutes direct\n      or contributory patent infringement, then any patent licenses\n      granted to You under this License for that Work shall terminate\n      as of the date such litigation is filed.\n\n   4. Redistribution. You may reproduce and distribute copies of the\n      Work or Derivative Works thereof in any medium, with or without\n      modifications, and in Source or Object form, provided that You\n      meet the following conditions:\n\n      (a) You must give any other recipients of the Work or\n          Derivative Works a copy of this License; and\n\n      (b) You must cause any modified files to carry prominent notices\n          stating that You changed the files; and\n\n      (c) You must retain, in the Source form of any Derivative Works\n          that You distribute, all copyright, patent, trademark, and\n          attribution notices from the Source form of the Work,\n          excluding those notices that do not pertain to any part of\n          the Derivative Works; and\n\n      (d) If the Work includes a \"NOTICE\" text file as part of its\n          distribution, then any Derivative Works that You distribute must\n          include a readable copy of the attribution notices contained\n          within such NOTICE file, excluding those notices that do not\n          pertain to any part of the Derivative Works, in at least one\n          of the following places: within a NOTICE text file distributed\n          as part of the Derivative Works; within the Source form or\n          documentation, if provided along with the Derivative Works; or,\n          within a display generated by the Derivative Works, if and\n          wherever such third-party notices normally appear. The contents\n          of the NOTICE file are for informational purposes only and\n          do not modify the License. You may add Your own attribution\n          notices within Derivative Works that You distribute, alongside\n          or as an addendum to the NOTICE text from the Work, provided\n          that such additional attribution notices cannot be construed\n          as modifying the License.\n\n      You may add Your own copyright statement to Your modifications and\n      may provide additional or different license terms and conditions\n      for use, reproduction, or distribution of Your modifications, or\n      for any such Derivative Works as a whole, provided Your use,\n      reproduction, and distribution of the Work otherwise complies with\n      the conditions stated in this License.\n\n   5. Submission of Contributions. Unless You explicitly state otherwise,\n      any Contribution intentionally submitted for inclusion in the Work\n      by You to the Licensor shall be under the terms and conditions of\n      this License, without any additional terms or conditions.\n      Notwithstanding the above, nothing herein shall supersede or modify\n      the terms of any separate license agreement you may have executed\n      with Licensor regarding such Contributions.\n\n   6. Trademarks. This License does not grant permission to use the trade\n      names, trademarks, service marks, or product names of the Licensor,\n      except as required for reasonable and customary use in describing the\n      origin of the Work and reproducing the content of the NOTICE file.\n\n   7. Disclaimer of Warranty. Unless required by applicable law or\n      agreed to in writing, Licensor provides the Work (and each\n      Contributor provides its Contributions) on an \"AS IS\" BASIS,\n      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or\n      implied, including, without limitation, any warranties or conditions\n      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A\n      PARTICULAR PURPOSE. You are solely responsible for determining the\n      appropriateness of using or redistributing the Work and assume any\n      risks associated with Your exercise of permissions under this License.\n\n   8. Limitation of Liability. In no event and under no legal theory,\n      whether in tort (including negligence), contract, or otherwise,\n      unless required by applicable law (such as deliberate and grossly\n      negligent acts) or agreed to in writing, shall any Contributor be\n      liable to You for damages, including any direct, indirect, special,\n      incidental, or consequential damages of any character arising as a\n      result of this License or out of the use or inability to use the\n      Work (including but not limited to damages for loss of goodwill,\n      work stoppage, computer failure or malfunction, or any and all\n      other commercial damages or losses), even if such Contributor\n      has been advised of the possibility of such damages.\n\n   9. Accepting Warranty or Additional Liability. While redistributing\n      the Work or Derivative Works thereof, You may choose to offer,\n      and charge a fee for, acceptance of support, warranty, indemnity,\n      or other liability obligations and/or rights consistent with this\n      License. However, in accepting such obligations, You may act only\n      on Your own behalf and on Your sole responsibility, not on behalf\n      of any other Contributor, and only if You agree to indemnify,\n      defend, and hold each Contributor harmless for any liability\n      incurred by, or claims asserted against, such Contributor by reason\n      of your accepting any such warranty or additional liability.\n\n   END OF TERMS AND CONDITIONS\n\n   APPENDIX: How to apply the Apache License to your work.\n\n      To apply the Apache License to your work, attach the following\n      boilerplate notice, with the fields enclosed by brackets \"[]\"\n      replaced with your own identifying information. (Don't include\n      the brackets!)  The text should be enclosed in the appropriate\n      comment syntax for the file format. We also recommend that a\n      file or class name and description of purpose be included on the\n      same \"printed page\" as the copyright notice for easier\n      identification within third-party archives.\n\n   Copyright [yyyy] [name of copyright owner]\n\n   Licensed under the Apache License, Version 2.0 (the \"License\");\n   you may not use this file except in compliance with the License.\n   You may obtain a copy of the License at\n\n       http://www.apache.org/licenses/LICENSE-2.0\n\n   Unless required by applicable law or agreed to in writing, software\n   distributed under the License is distributed on an \"AS IS\" BASIS,\n   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n   See the License for the specific language governing permissions and\n   limitations under the License.\n"
  },
  {
    "path": "pingora-proxy/examples/backoff_retry.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\nuse std::time::Duration;\n\nuse async_trait::async_trait;\n\nuse log::info;\nuse pingora_core::server::Server;\nuse pingora_core::upstreams::peer::HttpPeer;\nuse pingora_core::Result;\nuse pingora_core::{prelude::Opt, Error};\nuse pingora_proxy::{ProxyHttp, Session};\n\n/// This example shows how to setup retry-able errors with a backoff policy\n\n#[derive(Default)]\nstruct RetryCtx {\n    pub retries: u32,\n}\n\nstruct BackoffRetryProxy;\n\n#[async_trait]\nimpl ProxyHttp for BackoffRetryProxy {\n    type CTX = RetryCtx;\n    fn new_ctx(&self) -> Self::CTX {\n        Self::CTX::default()\n    }\n\n    fn fail_to_connect(\n        &self,\n        _session: &mut Session,\n        _peer: &HttpPeer,\n        ctx: &mut Self::CTX,\n        e: Box<Error>,\n    ) -> Box<Error> {\n        ctx.retries += 1;\n        let mut retry_e = e;\n        retry_e.set_retry(true);\n        retry_e\n    }\n\n    async fn upstream_peer(\n        &self,\n        _session: &mut Session,\n        ctx: &mut Self::CTX,\n    ) -> Result<Box<HttpPeer>> {\n        const MAX_SLEEP: Duration = Duration::from_secs(10);\n\n        if ctx.retries > 0 {\n            // simple example of exponential backoff with a max of 10s\n            let sleep_ms =\n                std::cmp::min(Duration::from_millis(u64::pow(10, ctx.retries)), MAX_SLEEP);\n            info!(\"sleeping for ms: {sleep_ms:?}\");\n            tokio::time::sleep(sleep_ms).await;\n        }\n        let mut peer = HttpPeer::new((\"10.0.0.1\", 80), false, \"\".into());\n        peer.options.connection_timeout = Some(Duration::from_millis(100));\n        Ok(Box::new(peer))\n    }\n}\n\n// RUST_LOG=INFO cargo run --example backoff_retry -- --conf examples/conf.yaml\n\nfn main() {\n    env_logger::init();\n\n    // read command line arguments\n    let opt = Opt::parse_args();\n    let mut my_server = Server::new(Some(opt)).unwrap();\n    my_server.bootstrap();\n\n    let mut my_proxy =\n        pingora_proxy::http_proxy_service(&my_server.configuration, BackoffRetryProxy);\n    my_proxy.add_tcp(\"0.0.0.0:6195\");\n\n    my_server.add_service(my_proxy);\n    my_server.run_forever();\n}\n"
  },
  {
    "path": "pingora-proxy/examples/conf.yaml",
    "content": "---\nversion: 1\nthreads: 2\npid_file: /tmp/load_balancer.pid\nerror_log: /tmp/load_balancer_err.log\nupgrade_sock: /tmp/load_balancer.sock\nmax_retries: 5\n"
  },
  {
    "path": "pingora-proxy/examples/connection_filter.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\nuse async_trait::async_trait;\nuse clap::Parser;\nuse log::info;\nuse pingora_core::listeners::ConnectionFilter;\nuse pingora_core::prelude::Opt;\nuse pingora_core::server::Server;\nuse pingora_core::upstreams::peer::HttpPeer;\nuse pingora_core::Result;\nuse pingora_proxy::{ProxyHttp, Session};\nuse std::sync::Arc;\n\n/// This example demonstrates how to implement a connection filter\npub struct MyProxy;\n\n#[async_trait]\nimpl ProxyHttp for MyProxy {\n    type CTX = ();\n\n    fn new_ctx(&self) -> Self::CTX {}\n\n    async fn upstream_peer(\n        &self,\n        _session: &mut Session,\n        _ctx: &mut Self::CTX,\n    ) -> Result<Box<HttpPeer>> {\n        // Forward to httpbin.org for testing\n        let peer = HttpPeer::new((\"httpbin.org\", 80), false, \"httpbin.org\".into());\n        Ok(Box::new(peer))\n    }\n}\n\n/// Connection filter that blocks ALL connections (for testing)\n#[derive(Debug, Clone)]\nstruct BlockAllFilter;\n\n#[async_trait]\nimpl ConnectionFilter for BlockAllFilter {\n    async fn should_accept(&self, addr: &std::net::SocketAddr) -> bool {\n        info!(\"BLOCKING connection from {} (BlockAllFilter active)\", addr);\n        false\n    }\n}\n\n// RUST_LOG=INFO cargo run --example connection_filter\n\nfn main() {\n    env_logger::init();\n\n    // read command line arguments\n    let opt = Opt::parse();\n    let mut my_server = Server::new(Some(opt)).unwrap();\n    my_server.bootstrap();\n\n    let mut my_proxy = pingora_proxy::http_proxy_service(&my_server.configuration, MyProxy);\n\n    // Create a filter that blocks ALL connections\n    let filter = Arc::new(BlockAllFilter);\n\n    info!(\"Setting BlockAllFilter on proxy service\");\n    my_proxy.set_connection_filter(filter.clone());\n\n    info!(\"Adding TCP endpoints AFTER setting filter\");\n    my_proxy.add_tcp(\"0.0.0.0:6195\");\n    my_proxy.add_tcp(\"0.0.0.0:6196\");\n\n    info!(\"====================================\");\n    info!(\"Server starting with BlockAllFilter\");\n    info!(\"This filter blocks ALL connections!\");\n    info!(\"====================================\");\n    info!(\"\");\n    info!(\"Test with:\");\n    info!(\"  curl http://localhost:6195/get\");\n    info!(\"  curl http://localhost:6196/get\");\n    info!(\"\");\n    info!(\"ALL requests should be blocked!\");\n    info!(\"You should see 'BLOCKING connection' in the logs\");\n    info!(\"and curl should fail with 'Connection refused' or hang\");\n    info!(\"\");\n\n    my_server.add_service(my_proxy);\n    my_server.run_forever();\n}\n"
  },
  {
    "path": "pingora-proxy/examples/ctx.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\nuse async_trait::async_trait;\nuse log::info;\nuse std::sync::Mutex;\n\nuse pingora_core::server::configuration::Opt;\nuse pingora_core::server::Server;\nuse pingora_core::upstreams::peer::HttpPeer;\nuse pingora_core::Result;\nuse pingora_proxy::{ProxyHttp, Session};\n\n// global counter\nstatic REQ_COUNTER: Mutex<usize> = Mutex::new(0);\n\npub struct MyProxy {\n    // counter for the service\n    beta_counter: Mutex<usize>, // AtomicUsize works too\n}\n\npub struct MyCtx {\n    beta_user: bool,\n}\n\nfn check_beta_user(req: &pingora_http::RequestHeader) -> bool {\n    // some simple logic to check if user is beta\n    req.headers.get(\"beta-flag\").is_some()\n}\n\n#[async_trait]\nimpl ProxyHttp for MyProxy {\n    type CTX = MyCtx;\n    fn new_ctx(&self) -> Self::CTX {\n        MyCtx { beta_user: false }\n    }\n\n    async fn request_filter(&self, session: &mut Session, ctx: &mut Self::CTX) -> Result<bool> {\n        ctx.beta_user = check_beta_user(session.req_header());\n        Ok(false)\n    }\n\n    async fn upstream_peer(\n        &self,\n        _session: &mut Session,\n        ctx: &mut Self::CTX,\n    ) -> Result<Box<HttpPeer>> {\n        let mut req_counter = REQ_COUNTER.lock().unwrap();\n        *req_counter += 1;\n\n        let addr = if ctx.beta_user {\n            let mut beta_count = self.beta_counter.lock().unwrap();\n            *beta_count += 1;\n            info!(\"I'm a beta user #{beta_count}\");\n            (\"1.0.0.1\", 443)\n        } else {\n            info!(\"I'm an user #{req_counter}\");\n            (\"1.1.1.1\", 443)\n        };\n\n        let peer = Box::new(HttpPeer::new(addr, true, \"one.one.one.one\".to_string()));\n        Ok(peer)\n    }\n}\n\n// RUST_LOG=INFO cargo run --example ctx\n// curl 127.0.0.1:6190 -H \"Host: one.one.one.one\"\n// curl 127.0.0.1:6190 -H \"Host: one.one.one.one\" -H \"beta-flag: 1\"\nfn main() {\n    env_logger::init();\n\n    // read command line arguments\n    let opt = Opt::parse_args();\n    let mut my_server = Server::new(Some(opt)).unwrap();\n    my_server.bootstrap();\n\n    let mut my_proxy = pingora_proxy::http_proxy_service(\n        &my_server.configuration,\n        MyProxy {\n            beta_counter: Mutex::new(0),\n        },\n    );\n    my_proxy.add_tcp(\"0.0.0.0:6190\");\n\n    my_server.add_service(my_proxy);\n    my_server.run_forever();\n}\n"
  },
  {
    "path": "pingora-proxy/examples/gateway.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\nuse async_trait::async_trait;\nuse bytes::Bytes;\nuse log::info;\nuse prometheus::register_int_counter;\n\nuse pingora_core::server::configuration::Opt;\nuse pingora_core::server::Server;\nuse pingora_core::upstreams::peer::HttpPeer;\nuse pingora_core::Result;\nuse pingora_http::ResponseHeader;\nuse pingora_proxy::{ProxyHttp, Session};\n\nfn check_login(req: &pingora_http::RequestHeader) -> bool {\n    // implement you logic check logic here\n    req.headers.get(\"Authorization\").map(|v| v.as_bytes()) == Some(b\"password\")\n}\n\npub struct MyGateway {\n    req_metric: prometheus::IntCounter,\n}\n\n#[async_trait]\nimpl ProxyHttp for MyGateway {\n    type CTX = ();\n    fn new_ctx(&self) -> Self::CTX {}\n\n    async fn request_filter(&self, session: &mut Session, _ctx: &mut Self::CTX) -> Result<bool> {\n        if session.req_header().uri.path().starts_with(\"/login\")\n            && !check_login(session.req_header())\n        {\n            let _ = session\n                .respond_error_with_body(403, Bytes::from_static(b\"no way!\"))\n                .await;\n            // true: early return as the response is already written\n            return Ok(true);\n        }\n        Ok(false)\n    }\n\n    async fn upstream_peer(\n        &self,\n        session: &mut Session,\n        _ctx: &mut Self::CTX,\n    ) -> Result<Box<HttpPeer>> {\n        let addr = if session.req_header().uri.path().starts_with(\"/family\") {\n            (\"1.0.0.1\", 443)\n        } else {\n            (\"1.1.1.1\", 443)\n        };\n\n        info!(\"connecting to {addr:?}\");\n\n        let peer = Box::new(HttpPeer::new(addr, true, \"one.one.one.one\".to_string()));\n        Ok(peer)\n    }\n\n    async fn response_filter(\n        &self,\n        _session: &mut Session,\n        upstream_response: &mut ResponseHeader,\n        _ctx: &mut Self::CTX,\n    ) -> Result<()>\n    where\n        Self::CTX: Send + Sync,\n    {\n        // replace existing header if any\n        upstream_response\n            .insert_header(\"Server\", \"MyGateway\")\n            .unwrap();\n        // because we don't support h3\n        upstream_response.remove_header(\"alt-svc\");\n\n        Ok(())\n    }\n\n    async fn logging(\n        &self,\n        session: &mut Session,\n        _e: Option<&pingora_core::Error>,\n        ctx: &mut Self::CTX,\n    ) {\n        let response_code = session\n            .response_written()\n            .map_or(0, |resp| resp.status.as_u16());\n        info!(\n            \"{} response code: {response_code}\",\n            self.request_summary(session, ctx)\n        );\n\n        self.req_metric.inc();\n    }\n}\n\n// RUST_LOG=INFO cargo run --example gateway\n// curl 127.0.0.1:6191 -H \"Host: one.one.one.one\"\n// curl 127.0.0.1:6190/family/ -H \"Host: one.one.one.one\"\n// curl 127.0.0.1:6191/login/ -H \"Host: one.one.one.one\" -I -H \"Authorization: password\"\n// curl 127.0.0.1:6191/login/ -H \"Host: one.one.one.one\" -I -H \"Authorization: bad\"\n// For metrics\n// curl 127.0.0.1:6192/\nfn main() {\n    env_logger::init();\n\n    // read command line arguments\n    let opt = Opt::parse_args();\n    let mut my_server = Server::new(Some(opt)).unwrap();\n    my_server.bootstrap();\n\n    let mut my_proxy = pingora_proxy::http_proxy_service(\n        &my_server.configuration,\n        MyGateway {\n            req_metric: register_int_counter!(\"req_counter\", \"Number of requests\").unwrap(),\n        },\n    );\n    my_proxy.add_tcp(\"0.0.0.0:6191\");\n    my_server.add_service(my_proxy);\n\n    let mut prometheus_service_http =\n        pingora_core::services::listening::Service::prometheus_http_service();\n    prometheus_service_http.add_tcp(\"127.0.0.1:6192\");\n    my_server.add_service(prometheus_service_http);\n\n    my_server.run_forever();\n}\n"
  },
  {
    "path": "pingora-proxy/examples/grpc_web_module.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\nuse async_trait::async_trait;\n\nuse pingora_core::server::Server;\nuse pingora_core::upstreams::peer::HttpPeer;\nuse pingora_core::Result;\nuse pingora_core::{\n    modules::http::{\n        grpc_web::{GrpcWeb, GrpcWebBridge},\n        HttpModules,\n    },\n    prelude::Opt,\n};\nuse pingora_proxy::{ProxyHttp, Session};\n\n// This example shows how to use the gRPC-web bridge module\n\npub struct GrpcWebBridgeProxy;\n\n#[async_trait]\nimpl ProxyHttp for GrpcWebBridgeProxy {\n    type CTX = ();\n    fn new_ctx(&self) -> Self::CTX {}\n\n    fn init_downstream_modules(&self, modules: &mut HttpModules) {\n        // Add the gRPC web module\n        modules.add_module(Box::new(GrpcWeb))\n    }\n\n    async fn early_request_filter(\n        &self,\n        session: &mut Session,\n        _ctx: &mut Self::CTX,\n    ) -> Result<()> {\n        let grpc = session\n            .downstream_modules_ctx\n            .get_mut::<GrpcWebBridge>()\n            .expect(\"GrpcWebBridge module added\");\n\n        // initialize gRPC module for this request\n        grpc.init();\n        Ok(())\n    }\n\n    async fn upstream_peer(\n        &self,\n        _session: &mut Session,\n        _ctx: &mut Self::CTX,\n    ) -> Result<Box<HttpPeer>> {\n        // this needs to be your gRPC server\n        let grpc_peer = Box::new(HttpPeer::new(\n            (\"1.1.1.1\", 443),\n            true,\n            \"one.one.one.one\".to_string(),\n        ));\n        Ok(grpc_peer)\n    }\n}\n\n// RUST_LOG=INFO cargo run --example grpc_web_module\n\nfn main() {\n    env_logger::init();\n\n    // read command line arguments\n    let opt = Opt::parse_args();\n    let mut my_server = Server::new(Some(opt)).unwrap();\n    my_server.bootstrap();\n\n    let mut my_proxy =\n        pingora_proxy::http_proxy_service(&my_server.configuration, GrpcWebBridgeProxy);\n    my_proxy.add_tcp(\"0.0.0.0:6194\");\n\n    my_server.add_service(my_proxy);\n    my_server.run_forever();\n}\n"
  },
  {
    "path": "pingora-proxy/examples/load_balancer.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\nuse async_trait::async_trait;\nuse log::info;\nuse pingora_core::services::background::background_service;\nuse std::{sync::Arc, time::Duration};\n\nuse pingora_core::server::configuration::Opt;\nuse pingora_core::server::Server;\nuse pingora_core::upstreams::peer::HttpPeer;\nuse pingora_core::Result;\nuse pingora_load_balancing::{health_check, selection::RoundRobin, LoadBalancer};\nuse pingora_proxy::{ProxyHttp, Session};\n\npub struct LB(Arc<LoadBalancer<RoundRobin>>);\n\n#[async_trait]\nimpl ProxyHttp for LB {\n    type CTX = ();\n    fn new_ctx(&self) -> Self::CTX {}\n\n    async fn upstream_peer(&self, _session: &mut Session, _ctx: &mut ()) -> Result<Box<HttpPeer>> {\n        let upstream = self\n            .0\n            .select(b\"\", 256) // hash doesn't matter\n            .unwrap();\n\n        info!(\"upstream peer is: {:?}\", upstream);\n\n        let peer = Box::new(HttpPeer::new(upstream, true, \"one.one.one.one\".to_string()));\n        Ok(peer)\n    }\n\n    async fn upstream_request_filter(\n        &self,\n        _session: &mut Session,\n        upstream_request: &mut pingora_http::RequestHeader,\n        _ctx: &mut Self::CTX,\n    ) -> Result<()> {\n        upstream_request\n            .insert_header(\"Host\", \"one.one.one.one\")\n            .unwrap();\n        Ok(())\n    }\n}\n\n// RUST_LOG=INFO cargo run --example load_balancer\nfn main() {\n    env_logger::init();\n\n    // read command line arguments\n    let opt = Opt::parse_args();\n    let mut my_server = Server::new(Some(opt)).unwrap();\n    my_server.bootstrap();\n\n    // 127.0.0.1:343\" is just a bad server\n    let mut upstreams =\n        LoadBalancer::try_from_iter([\"1.1.1.1:443\", \"1.0.0.1:443\", \"127.0.0.1:343\"]).unwrap();\n\n    // We add health check in the background so that the bad server is never selected.\n    let hc = health_check::TcpHealthCheck::new();\n    upstreams.set_health_check(hc);\n    upstreams.health_check_frequency = Some(Duration::from_secs(1));\n\n    let background = background_service(\"health check\", upstreams);\n\n    let upstreams = background.task();\n\n    let mut lb = pingora_proxy::http_proxy_service(&my_server.configuration, LB(upstreams));\n    lb.add_tcp(\"0.0.0.0:6188\");\n\n    let cert_path = format!(\"{}/tests/keys/server.crt\", env!(\"CARGO_MANIFEST_DIR\"));\n    let key_path = format!(\"{}/tests/keys/key.pem\", env!(\"CARGO_MANIFEST_DIR\"));\n\n    let mut tls_settings =\n        pingora_core::listeners::tls::TlsSettings::intermediate(&cert_path, &key_path).unwrap();\n    tls_settings.enable_h2();\n    lb.add_tls_with_settings(\"0.0.0.0:6189\", None, tls_settings);\n\n    my_server.add_service(lb);\n    my_server.add_service(background);\n    my_server.run_forever();\n}\n"
  },
  {
    "path": "pingora-proxy/examples/modify_response.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\nuse async_trait::async_trait;\nuse bytes::Bytes;\nuse serde::{Deserialize, Serialize};\nuse std::net::ToSocketAddrs;\n\nuse pingora_core::server::configuration::Opt;\nuse pingora_core::server::Server;\nuse pingora_core::upstreams::peer::HttpPeer;\nuse pingora_core::Result;\nuse pingora_http::ResponseHeader;\nuse pingora_proxy::{ProxyHttp, Session};\n\nconst HOST: &str = \"ip.jsontest.com\";\n\n#[derive(Serialize, Deserialize)]\npub struct Resp {\n    ip: String,\n}\n\npub struct Json2Yaml {\n    addr: std::net::SocketAddr,\n}\n\npub struct MyCtx {\n    buffer: Vec<u8>,\n}\n\n#[async_trait]\nimpl ProxyHttp for Json2Yaml {\n    type CTX = MyCtx;\n    fn new_ctx(&self) -> Self::CTX {\n        MyCtx { buffer: vec![] }\n    }\n\n    async fn upstream_peer(\n        &self,\n        _session: &mut Session,\n        _ctx: &mut Self::CTX,\n    ) -> Result<Box<HttpPeer>> {\n        let peer = Box::new(HttpPeer::new(self.addr, false, HOST.to_owned()));\n        Ok(peer)\n    }\n\n    async fn upstream_request_filter(\n        &self,\n        _session: &mut Session,\n        upstream_request: &mut pingora_http::RequestHeader,\n        _ctx: &mut Self::CTX,\n    ) -> Result<()> {\n        upstream_request\n            .insert_header(\"Host\", HOST.to_owned())\n            .unwrap();\n        Ok(())\n    }\n\n    async fn response_filter(\n        &self,\n        _session: &mut Session,\n        upstream_response: &mut ResponseHeader,\n        _ctx: &mut Self::CTX,\n    ) -> Result<()>\n    where\n        Self::CTX: Send + Sync,\n    {\n        // Remove content-length because the size of the new body is unknown\n        upstream_response.remove_header(\"Content-Length\");\n        upstream_response\n            .insert_header(\"Transfer-Encoding\", \"Chunked\")\n            .unwrap();\n        Ok(())\n    }\n\n    fn response_body_filter(\n        &self,\n        _session: &mut Session,\n        body: &mut Option<Bytes>,\n        end_of_stream: bool,\n        ctx: &mut Self::CTX,\n    ) -> Result<Option<std::time::Duration>>\n    where\n        Self::CTX: Send + Sync,\n    {\n        // buffer the data\n        if let Some(b) = body {\n            ctx.buffer.extend(&b[..]);\n            // drop the body\n            b.clear();\n        }\n        if end_of_stream {\n            // This is the last chunk, we can process the data now\n            let json_body: Resp = serde_json::de::from_slice(&ctx.buffer).unwrap();\n            let yaml_body = serde_yaml::to_string(&json_body).unwrap();\n            *body = Some(Bytes::copy_from_slice(yaml_body.as_bytes()));\n        }\n\n        Ok(None)\n    }\n}\n\n// RUST_LOG=INFO cargo run --example modify_response\n// curl 127.0.0.1:6191\nfn main() {\n    env_logger::init();\n\n    let opt = Opt::parse_args();\n    let mut my_server = Server::new(Some(opt)).unwrap();\n    my_server.bootstrap();\n\n    let mut my_proxy = pingora_proxy::http_proxy_service(\n        &my_server.configuration,\n        Json2Yaml {\n            // hardcode the IP of ip.jsontest.com for now\n            addr: (\"142.251.2.121\", 80)\n                .to_socket_addrs()\n                .unwrap()\n                .next()\n                .unwrap(),\n        },\n    );\n\n    my_proxy.add_tcp(\"127.0.0.1:6191\");\n\n    my_server.add_service(my_proxy);\n    my_server.run_forever();\n}\n"
  },
  {
    "path": "pingora-proxy/examples/multi_lb.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\nuse async_trait::async_trait;\nuse std::sync::Arc;\n\nuse pingora_core::{prelude::*, services::background::GenBackgroundService};\nuse pingora_load_balancing::{\n    health_check::TcpHealthCheck,\n    selection::{BackendIter, BackendSelection, RoundRobin},\n    LoadBalancer,\n};\nuse pingora_proxy::{http_proxy_service, ProxyHttp, Session};\n\nstruct Router {\n    cluster_one: Arc<LoadBalancer<RoundRobin>>,\n    cluster_two: Arc<LoadBalancer<RoundRobin>>,\n}\n\n#[async_trait]\nimpl ProxyHttp for Router {\n    type CTX = ();\n    fn new_ctx(&self) {}\n\n    async fn upstream_peer(&self, session: &mut Session, _ctx: &mut ()) -> Result<Box<HttpPeer>> {\n        // determine LB cluster based on request uri\n        let cluster = if session.req_header().uri.path().starts_with(\"/one/\") {\n            &self.cluster_one\n        } else {\n            &self.cluster_two\n        };\n\n        let upstream = cluster\n            .select(b\"\", 256) // hash doesn't matter for round robin\n            .unwrap();\n\n        println!(\"upstream peer is: {upstream:?}\");\n\n        // Set SNI to one.one.one.one\n        let peer = Box::new(HttpPeer::new(upstream, true, \"one.one.one.one\".to_string()));\n        Ok(peer)\n    }\n}\n\nfn build_cluster_service<S>(upstreams: &[&str]) -> GenBackgroundService<LoadBalancer<S>>\nwhere\n    S: BackendSelection + 'static,\n    S::Iter: BackendIter,\n{\n    let mut cluster = LoadBalancer::try_from_iter(upstreams).unwrap();\n    cluster.set_health_check(TcpHealthCheck::new());\n    cluster.health_check_frequency = Some(std::time::Duration::from_secs(1));\n\n    background_service(\"cluster health check\", cluster)\n}\n\n// RUST_LOG=INFO cargo run --example multi_lb\n// curl 127.0.0.1:6188/one/\n// curl 127.0.0.1:6188/two/\nfn main() {\n    let mut my_server = Server::new(None).unwrap();\n    my_server.bootstrap();\n\n    // build multiple clusters\n    let cluster_one = build_cluster_service::<RoundRobin>(&[\"1.1.1.1:443\", \"127.0.0.1:343\"]);\n    let cluster_two = build_cluster_service::<RoundRobin>(&[\"1.0.0.1:443\", \"127.0.0.2:343\"]);\n\n    let router = Router {\n        cluster_one: cluster_one.task(),\n        cluster_two: cluster_two.task(),\n    };\n    let mut router_service = http_proxy_service(&my_server.configuration, router);\n    router_service.add_tcp(\"0.0.0.0:6188\");\n\n    my_server.add_service(router_service);\n    my_server.add_service(cluster_one);\n    my_server.add_service(cluster_two);\n\n    my_server.run_forever();\n}\n"
  },
  {
    "path": "pingora-proxy/examples/rate_limiter.rs",
    "content": "use async_trait::async_trait;\nuse once_cell::sync::Lazy;\nuse pingora_core::prelude::*;\nuse pingora_http::{RequestHeader, ResponseHeader};\nuse pingora_limits::rate::Rate;\nuse pingora_load_balancing::prelude::{RoundRobin, TcpHealthCheck};\nuse pingora_load_balancing::LoadBalancer;\nuse pingora_proxy::{http_proxy_service, ProxyHttp, Session};\nuse std::sync::Arc;\nuse std::time::Duration;\n\nfn main() {\n    let mut server = Server::new(Some(Opt::default())).unwrap();\n    server.bootstrap();\n    let mut upstreams = LoadBalancer::try_from_iter([\"1.1.1.1:443\", \"1.0.0.1:443\"]).unwrap();\n    // Set health check\n    let hc = TcpHealthCheck::new();\n    upstreams.set_health_check(hc);\n    upstreams.health_check_frequency = Some(Duration::from_secs(1));\n    // Set background service\n    let background = background_service(\"health check\", upstreams);\n    let upstreams = background.task();\n    // Set load balancer\n    let mut lb = http_proxy_service(&server.configuration, LB(upstreams));\n    lb.add_tcp(\"0.0.0.0:6188\");\n\n    // let rate = Rate\n    server.add_service(background);\n    server.add_service(lb);\n    server.run_forever();\n}\n\npub struct LB(Arc<LoadBalancer<RoundRobin>>);\n\nimpl LB {\n    pub fn get_request_appid(&self, session: &mut Session) -> Option<String> {\n        match session\n            .req_header()\n            .headers\n            .get(\"appid\")\n            .map(|v| v.to_str())\n        {\n            None => None,\n            Some(v) => match v {\n                Ok(v) => Some(v.to_string()),\n                Err(_) => None,\n            },\n        }\n    }\n}\n\n// Rate limiter\nstatic RATE_LIMITER: Lazy<Rate> = Lazy::new(|| Rate::new(Duration::from_secs(1)));\n\n// max request per second per client\nstatic MAX_REQ_PER_SEC: isize = 1;\n\n#[async_trait]\nimpl ProxyHttp for LB {\n    type CTX = ();\n\n    fn new_ctx(&self) {}\n\n    async fn upstream_peer(\n        &self,\n        _session: &mut Session,\n        _ctx: &mut Self::CTX,\n    ) -> Result<Box<HttpPeer>> {\n        let upstream = self.0.select(b\"\", 256).unwrap();\n        // Set SNI\n        let peer = Box::new(HttpPeer::new(upstream, true, \"one.one.one.one\".to_string()));\n        Ok(peer)\n    }\n\n    async fn upstream_request_filter(\n        &self,\n        _session: &mut Session,\n        upstream_request: &mut RequestHeader,\n        _ctx: &mut Self::CTX,\n    ) -> Result<()>\n    where\n        Self::CTX: Send + Sync,\n    {\n        upstream_request\n            .insert_header(\"Host\", \"one.one.one.one\")\n            .unwrap();\n        Ok(())\n    }\n\n    async fn request_filter(&self, session: &mut Session, _ctx: &mut Self::CTX) -> Result<bool>\n    where\n        Self::CTX: Send + Sync,\n    {\n        let appid = match self.get_request_appid(session) {\n            None => return Ok(false), // no client appid found, skip rate limiting\n            Some(addr) => addr,\n        };\n\n        // retrieve the current window requests\n        let curr_window_requests = RATE_LIMITER.observe(&appid, 1);\n        if curr_window_requests > MAX_REQ_PER_SEC {\n            // rate limited, return 429\n            let mut header = ResponseHeader::build(429, None).unwrap();\n            header\n                .insert_header(\"X-Rate-Limit-Limit\", MAX_REQ_PER_SEC.to_string())\n                .unwrap();\n            header.insert_header(\"X-Rate-Limit-Remaining\", \"0\").unwrap();\n            header.insert_header(\"X-Rate-Limit-Reset\", \"1\").unwrap();\n            session.set_keepalive(None);\n            session\n                .write_response_header(Box::new(header), true)\n                .await?;\n            return Ok(true);\n        }\n        Ok(false)\n    }\n}\n"
  },
  {
    "path": "pingora-proxy/examples/use_module.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\nuse async_trait::async_trait;\n\nuse pingora_core::modules::http::HttpModules;\nuse pingora_core::server::configuration::Opt;\nuse pingora_core::server::Server;\nuse pingora_core::upstreams::peer::HttpPeer;\nuse pingora_core::Result;\nuse pingora_http::RequestHeader;\nuse pingora_proxy::{ProxyHttp, Session};\n\n// This example shows how to build and import 3rd party modules\n\n/// A simple ACL to check \"Authorization: basic $credential\" header\nmod my_acl {\n    use super::*;\n    use pingora_core::modules::http::{HttpModule, HttpModuleBuilder, Module};\n    use pingora_error::{Error, ErrorType::HTTPStatus};\n    use std::any::Any;\n\n    // This is the struct for per request module context\n    struct MyAclCtx {\n        credential_header: String,\n    }\n\n    // Implement how the module would consume and/or modify request and/or response\n    #[async_trait]\n    impl HttpModule for MyAclCtx {\n        async fn request_header_filter(&mut self, req: &mut RequestHeader) -> Result<()> {\n            let Some(auth) = req.headers.get(http::header::AUTHORIZATION) else {\n                return Error::e_explain(HTTPStatus(403), \"Auth failed, no auth header\");\n            };\n\n            if auth.as_bytes() != self.credential_header.as_bytes() {\n                Error::e_explain(HTTPStatus(403), \"Auth failed, credential mismatch\")\n            } else {\n                Ok(())\n            }\n        }\n\n        // boilerplate code for all modules\n        fn as_any(&self) -> &dyn Any {\n            self\n        }\n        fn as_any_mut(&mut self) -> &mut dyn Any {\n            self\n        }\n    }\n\n    // This is the singleton object which will be attached to the server\n    pub struct MyAcl {\n        pub credential: String,\n    }\n    impl HttpModuleBuilder for MyAcl {\n        // This function defines how to create each Ctx. This function is called when a new request\n        // arrives\n        fn init(&self) -> Module {\n            Box::new(MyAclCtx {\n                // Make it easier to compare header\n                // We could also store this value in MyAcl and use Arc to share it with every Ctx.\n                credential_header: format!(\"basic {}\", self.credential),\n            })\n        }\n    }\n}\n\npub struct MyProxy;\n\n#[async_trait]\nimpl ProxyHttp for MyProxy {\n    type CTX = ();\n    fn new_ctx(&self) -> Self::CTX {}\n\n    // This function is only called once when the server starts\n    fn init_downstream_modules(&self, modules: &mut HttpModules) {\n        // Add the module to MyProxy\n        modules.add_module(Box::new(my_acl::MyAcl {\n            credential: \"testcode\".into(),\n        }))\n    }\n\n    async fn upstream_peer(\n        &self,\n        _session: &mut Session,\n        _ctx: &mut Self::CTX,\n    ) -> Result<Box<HttpPeer>> {\n        let peer = Box::new(HttpPeer::new(\n            (\"1.1.1.1\", 443),\n            true,\n            \"one.one.one.one\".to_string(),\n        ));\n        Ok(peer)\n    }\n}\n\n// RUST_LOG=INFO cargo run --example use_module\n// curl 127.0.0.1:6193 -H \"Host: one.one.one.one\" -v\n// curl 127.0.0.1:6193 -H \"Host: one.one.one.one\" -H \"Authorization: basic testcode\"\n// curl 127.0.0.1:6193 -H \"Host: one.one.one.one\" -H \"Authorization: basic wrong\" -v\nfn main() {\n    env_logger::init();\n\n    // read command line arguments\n    let opt = Opt::parse_args();\n    let mut my_server = Server::new(Some(opt)).unwrap();\n    my_server.bootstrap();\n\n    let mut my_proxy = pingora_proxy::http_proxy_service(&my_server.configuration, MyProxy);\n    my_proxy.add_tcp(\"0.0.0.0:6193\");\n\n    my_server.add_service(my_proxy);\n    my_server.run_forever();\n}\n"
  },
  {
    "path": "pingora-proxy/examples/virtual_l4.rs",
    "content": "//! This example demonstrates to how to implement a custom L4 connector\n//! together with a virtual socket.\n\nuse std::net::{IpAddr, Ipv4Addr, SocketAddr};\nuse std::sync::Arc;\n\nuse async_trait::async_trait;\nuse pingora_core::connectors::L4Connect;\nuse pingora_core::prelude::HttpPeer;\nuse pingora_core::protocols::l4::socket::SocketAddr as L4SocketAddr;\nuse pingora_core::protocols::l4::stream::Stream;\nuse pingora_core::protocols::l4::virt::{VirtualSocket, VirtualSocketStream};\nuse pingora_core::server::RunArgs;\nuse pingora_core::server::{configuration::ServerConf, Server};\nuse pingora_core::services::listening::Service;\nuse pingora_core::upstreams::peer::PeerOptions;\nuse pingora_error::Result;\nuse pingora_proxy::{http_proxy_service_with_name, prelude::*, HttpProxy, ProxyHttp};\nuse tokio::io::{AsyncRead, AsyncWrite};\n\n/// Static virtual socket that serves a single HTTP request with a static response.\n///\n/// In real world use cases you would implement [`VirtualSocket`] for streams\n/// that implement `AsyncRead + AsyncWrite`.\n#[derive(Debug)]\nstruct StaticVirtualSocket {\n    content: Vec<u8>,\n    read_pos: usize,\n}\n\nimpl StaticVirtualSocket {\n    fn new() -> Self {\n        let response = b\"HTTP/1.1 200 OK\\r\\nContent-Length: 13\\r\\n\\r\\nHello, world!\";\n        Self {\n            content: response.to_vec(),\n            read_pos: 0,\n        }\n    }\n}\n\nimpl AsyncRead for StaticVirtualSocket {\n    fn poll_read(\n        mut self: std::pin::Pin<&mut Self>,\n        _cx: &mut std::task::Context<'_>,\n        buf: &mut tokio::io::ReadBuf<'_>,\n    ) -> std::task::Poll<std::io::Result<()>> {\n        debug_assert!(self.read_pos <= self.content.len());\n\n        let remaining = self.content.len() - self.read_pos;\n        if remaining == 0 {\n            return std::task::Poll::Ready(Ok(()));\n        }\n\n        let to_read = std::cmp::min(remaining, buf.remaining());\n        buf.put_slice(&self.content[self.read_pos..self.read_pos + to_read]);\n        self.read_pos += to_read;\n\n        std::task::Poll::Ready(Ok(()))\n    }\n}\n\nimpl AsyncWrite for StaticVirtualSocket {\n    fn poll_write(\n        self: std::pin::Pin<&mut Self>,\n        _cx: &mut std::task::Context<'_>,\n        buf: &[u8],\n    ) -> std::task::Poll<std::io::Result<usize>> {\n        // Discard all writes\n        std::task::Poll::Ready(Ok(buf.len()))\n    }\n\n    fn poll_flush(\n        self: std::pin::Pin<&mut Self>,\n        _cx: &mut std::task::Context<'_>,\n    ) -> std::task::Poll<std::io::Result<()>> {\n        std::task::Poll::Ready(Ok(()))\n    }\n\n    fn poll_shutdown(\n        self: std::pin::Pin<&mut Self>,\n        _cx: &mut std::task::Context<'_>,\n    ) -> std::task::Poll<std::io::Result<()>> {\n        std::task::Poll::Ready(Ok(()))\n    }\n}\n\nimpl VirtualSocket for StaticVirtualSocket {\n    fn set_socket_option(\n        &self,\n        _opt: pingora_core::protocols::l4::virt::VirtualSockOpt,\n    ) -> std::io::Result<()> {\n        Ok(())\n    }\n}\n\n#[derive(Debug)]\nstruct VirtualConnector;\n\n#[async_trait]\nimpl L4Connect for VirtualConnector {\n    async fn connect(&self, _addr: &L4SocketAddr) -> pingora_error::Result<Stream> {\n        Ok(Stream::from(VirtualSocketStream::new(Box::new(\n            StaticVirtualSocket::new(),\n        ))))\n    }\n}\n\nstruct VirtualProxy {\n    connector: Arc<dyn L4Connect + Send + Sync>,\n}\n\nimpl VirtualProxy {\n    fn new() -> Self {\n        Self {\n            connector: Arc::new(VirtualConnector),\n        }\n    }\n}\n\n#[async_trait::async_trait]\nimpl ProxyHttp for VirtualProxy {\n    type CTX = ();\n\n    fn new_ctx(&self) -> Self::CTX {}\n\n    // Route everything to example.org unless the Host header is \"virtual.test\",\n    // in which case target the special virtual address 203.0.113.1:18080.\n    async fn upstream_peer(\n        &self,\n        _session: &mut Session,\n        _ctx: &mut Self::CTX,\n    ) -> Result<Box<pingora_core::upstreams::peer::HttpPeer>> {\n        let mut options = PeerOptions::new();\n        options.custom_l4 = Some(self.connector.clone());\n\n        Ok(Box::new(HttpPeer {\n            _address: L4SocketAddr::Inet(SocketAddr::new(\n                IpAddr::V4(Ipv4Addr::new(1, 1, 1, 1)),\n                80,\n            )),\n            scheme: pingora_core::upstreams::peer::Scheme::HTTP,\n            sni: \"example.org\".to_string(),\n            proxy: None,\n            client_cert_key: None,\n            group_key: 0,\n            options,\n        }))\n    }\n}\n\nfn main() {\n    // Minimal server config\n    let conf = Arc::new(ServerConf::default());\n\n    // Build the service and set the default L4 connector\n    let mut svc: Service<HttpProxy<VirtualProxy>> =\n        http_proxy_service_with_name(&conf, VirtualProxy::new(), \"virtual-proxy\");\n\n    // Listen\n    let addr = \"127.0.0.1:6196\";\n    svc.add_tcp(addr);\n\n    let mut server = Server::new(None).unwrap();\n    server.add_service(svc);\n    let run = RunArgs::default();\n\n    eprintln!(\"Listening on {addr}, try: curl http://{addr}/\");\n    server.run(run);\n}\n"
  },
  {
    "path": "pingora-proxy/src/lib.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! # pingora-proxy\n//!\n//! Programmable HTTP proxy built on top of [pingora_core].\n//!\n//! # Features\n//! - HTTP/1.x and HTTP/2 for both downstream and upstream\n//! - Connection pooling\n//! - TLSv1.3, mutual TLS, customizable CA\n//! - Request/Response scanning, modification or rejection\n//! - Dynamic upstream selection\n//! - Configurable retry and failover\n//! - Fully programmable and customizable at any stage of a HTTP request\n//!\n//! # How to use\n//!\n//! Users of this crate defines their proxy by implementing [ProxyHttp] trait, which contains the\n//! callbacks to be invoked at each stage of a HTTP request.\n//!\n//! Then the service can be passed into [`http_proxy_service()`] for a [pingora_core::server::Server] to\n//! run it.\n//!\n//! See `examples/load_balancer.rs` for a detailed example.\n\nuse async_trait::async_trait;\nuse bytes::Bytes;\nuse futures::future::BoxFuture;\nuse futures::future::FutureExt;\nuse http::{header, version::Version, Method};\nuse log::{debug, error, trace, warn};\nuse once_cell::sync::Lazy;\nuse pingora_http::{RequestHeader, ResponseHeader};\nuse std::fmt::Debug;\nuse std::str;\nuse std::sync::{\n    atomic::{AtomicBool, Ordering},\n    Arc,\n};\nuse std::time::Duration;\nuse tokio::sync::{mpsc, Notify};\nuse tokio::time;\n\nuse pingora_cache::NoCacheReason;\nuse pingora_core::apps::{\n    HttpPersistentSettings, HttpServerApp, HttpServerOptions, ReusedHttpStream,\n};\nuse pingora_core::connectors::http::custom;\nuse pingora_core::connectors::{http::Connector, ConnectorOptions};\nuse pingora_core::modules::http::compression::ResponseCompressionBuilder;\nuse pingora_core::modules::http::{HttpModuleCtx, HttpModules};\nuse pingora_core::protocols::http::client::HttpSession as ClientSession;\nuse pingora_core::protocols::http::custom::CustomMessageWrite;\nuse pingora_core::protocols::http::subrequest::server::SubrequestHandle;\nuse pingora_core::protocols::http::v1::client::HttpSession as HttpSessionV1;\nuse pingora_core::protocols::http::v2::server::H2Options;\nuse pingora_core::protocols::http::HttpTask;\nuse pingora_core::protocols::http::ServerSession as HttpSession;\nuse pingora_core::protocols::http::SERVER_NAME;\nuse pingora_core::protocols::Stream;\nuse pingora_core::protocols::{Digest, UniqueID};\nuse pingora_core::server::configuration::ServerConf;\nuse pingora_core::server::ShutdownWatch;\nuse pingora_core::upstreams::peer::{HttpPeer, Peer};\nuse pingora_error::{Error, ErrorSource, ErrorType::*, OrErr, Result};\n\nconst TASK_BUFFER_SIZE: usize = 4;\n\nmod proxy_cache;\nmod proxy_common;\nmod proxy_custom;\nmod proxy_h1;\nmod proxy_h2;\nmod proxy_purge;\nmod proxy_trait;\npub mod subrequest;\n\nuse subrequest::{BodyMode, Ctx as SubrequestCtx};\n\npub use proxy_cache::range_filter::{range_header_filter, MultiRangeInfo, RangeType};\npub use proxy_purge::PurgeStatus;\npub use proxy_trait::{FailToProxy, ProxyHttp};\n\npub mod prelude {\n    pub use crate::{http_proxy, http_proxy_service, ProxyHttp, Session};\n}\n\npub type ProcessCustomSession<SV, C> = Arc<\n    dyn Fn(Arc<HttpProxy<SV, C>>, Stream, &ShutdownWatch) -> BoxFuture<'static, Option<Stream>>\n        + Send\n        + Sync\n        + Unpin\n        + 'static,\n>;\n\n/// The concrete type that holds the user defined HTTP proxy.\n///\n/// Users don't need to interact with this object directly.\npub struct HttpProxy<SV, C = ()>\nwhere\n    C: custom::Connector, // Upstream custom connector\n{\n    inner: SV, // TODO: name it better than inner\n    client_upstream: Connector<C>,\n    shutdown: Notify,\n    shutdown_flag: Arc<AtomicBool>,\n    pub server_options: Option<HttpServerOptions>,\n    pub h2_options: Option<H2Options>,\n    pub downstream_modules: HttpModules,\n    max_retries: usize,\n    process_custom_session: Option<ProcessCustomSession<SV, C>>,\n}\n\nimpl<SV> HttpProxy<SV, ()> {\n    /// Create a new [`HttpProxy`] with the given [`ProxyHttp`] implementation and [`ServerConf`].\n    ///\n    /// After creating an `HttpProxy`, you should call [`HttpProxy::handle_init_modules()`] to\n    /// initialize the downstream modules before processing requests.\n    ///\n    /// For most use cases, prefer using [`http_proxy_service()`] which wraps the `HttpProxy` in a\n    /// [`Service`]. This constructor is useful when you need to integrate `HttpProxy` into a custom\n    /// accept loop (e.g., for SNI-based routing decisions before TLS termination).\n    ///\n    /// # Example\n    ///\n    /// ```ignore\n    /// use pingora_proxy::HttpProxy;\n    /// use std::sync::Arc;\n    ///\n    /// let mut proxy = HttpProxy::new(my_proxy_app, server_conf);\n    /// proxy.handle_init_modules();\n    /// let proxy = Arc::new(proxy);\n    /// // Use proxy.process_new_http() in your custom accept loop\n    /// ```\n    pub fn new(inner: SV, conf: Arc<ServerConf>) -> Self {\n        HttpProxy {\n            inner,\n            client_upstream: Connector::new(Some(ConnectorOptions::from_server_conf(&conf))),\n            shutdown: Notify::new(),\n            shutdown_flag: Arc::new(AtomicBool::new(false)),\n            server_options: None,\n            h2_options: None,\n            downstream_modules: HttpModules::new(),\n            max_retries: conf.max_retries,\n            process_custom_session: None,\n        }\n    }\n}\n\nimpl<SV, C> HttpProxy<SV, C>\nwhere\n    C: custom::Connector,\n{\n    fn new_custom(\n        inner: SV,\n        conf: Arc<ServerConf>,\n        connector: C,\n        on_custom: Option<ProcessCustomSession<SV, C>>,\n        server_options: Option<HttpServerOptions>,\n    ) -> Self\n    where\n        SV: ProxyHttp + Send + Sync + 'static,\n        SV::CTX: Send + Sync,\n    {\n        let client_upstream =\n            Connector::new_custom(Some(ConnectorOptions::from_server_conf(&conf)), connector);\n\n        HttpProxy {\n            inner,\n            client_upstream,\n            shutdown: Notify::new(),\n            shutdown_flag: Arc::new(AtomicBool::new(false)),\n            server_options,\n            downstream_modules: HttpModules::new(),\n            max_retries: conf.max_retries,\n            process_custom_session: on_custom,\n            h2_options: None,\n        }\n    }\n\n    /// Initialize the downstream modules for this proxy.\n    ///\n    /// This method must be called after creating an [`HttpProxy`] with [`HttpProxy::new()`]\n    /// and before processing any requests. It invokes [`ProxyHttp::init_downstream_modules()`]\n    /// to set up any HTTP modules configured by the user's proxy implementation.\n    ///\n    /// Note: When using [`http_proxy_service()`] or [`http_proxy_service_with_name()`],\n    /// this method is called automatically.\n    pub fn handle_init_modules(&mut self)\n    where\n        SV: ProxyHttp,\n    {\n        self.inner\n            .init_downstream_modules(&mut self.downstream_modules);\n    }\n\n    async fn handle_new_request(\n        &self,\n        mut downstream_session: Box<HttpSession>,\n    ) -> Option<Box<HttpSession>>\n    where\n        SV: ProxyHttp + Send + Sync,\n        SV::CTX: Send + Sync,\n    {\n        // phase 1 read request header\n\n        let res = tokio::select! {\n            biased; // biased select is cheaper, and we don't want to drop already buffered requests\n            res = downstream_session.read_request() => { res }\n            _ = self.shutdown.notified() => {\n                // service shutting down, dropping the connection to stop more req from coming in\n                return None;\n            }\n        };\n        match res {\n            Ok(true) => {\n                // TODO: check n==0\n                debug!(\"Successfully get a new request\");\n            }\n            Ok(false) => {\n                return None; // TODO: close connection?\n            }\n            Err(mut e) => {\n                e.as_down();\n                error!(\"Fail to proxy: {e}\");\n                if matches!(e.etype, InvalidHTTPHeader) {\n                    downstream_session\n                        .respond_error(400)\n                        .await\n                        .unwrap_or_else(|e| {\n                            error!(\"failed to send error response to downstream: {e}\");\n                        });\n                } // otherwise the connection must be broken, no need to send anything\n                downstream_session.shutdown().await;\n                return None;\n            }\n        }\n        trace!(\n            \"Request header: {:?}\",\n            downstream_session.req_header().as_ref()\n        );\n        // CONNECT method proxying is not default supported by the proxy http logic itself,\n        // since the tunneling process changes the request-response flow.\n        // https://datatracker.ietf.org/doc/html/rfc9110#name-connect\n        // Also because the method impacts message framing in a way is currently unaccounted for\n        // (https://datatracker.ietf.org/doc/html/rfc9112#section-6.3-2.2)\n        // it is safest to disallow use of the method by default.\n        if !self\n            .server_options\n            .as_ref()\n            .is_some_and(|opts| opts.allow_connect_method_proxying)\n            && downstream_session.req_header().method == Method::CONNECT\n        {\n            downstream_session\n                .respond_error(405)\n                .await\n                .unwrap_or_else(|e| {\n                    error!(\"failed to send error response to downstream: {e}\");\n                });\n            downstream_session.shutdown().await;\n            return None;\n        }\n        Some(downstream_session)\n    }\n\n    // return bool: server_session can be reused, and error if any\n    async fn proxy_to_upstream(\n        &self,\n        session: &mut Session,\n        ctx: &mut SV::CTX,\n    ) -> (bool, Option<Box<Error>>)\n    where\n        SV: ProxyHttp + Send + Sync,\n        SV::CTX: Send + Sync,\n    {\n        let peer = match self.inner.upstream_peer(session, ctx).await {\n            Ok(p) => p,\n            Err(e) => return (false, Some(e)),\n        };\n\n        let client_session = self.client_upstream.get_http_session(&*peer).await;\n        match client_session {\n            Ok((client_session, client_reused)) => {\n                let (server_reused, error) = match client_session {\n                    ClientSession::H1(mut h1) => {\n                        let (server_reused, client_reuse, error) = self\n                            .proxy_to_h1_upstream(session, &mut h1, client_reused, &peer, ctx)\n                            .await;\n                        if client_reuse {\n                            let session = ClientSession::H1(h1);\n                            self.client_upstream\n                                .release_http_session(session, &*peer, peer.idle_timeout())\n                                .await;\n                        }\n                        (server_reused, error)\n                    }\n                    ClientSession::H2(mut h2) => {\n                        let (server_reused, mut error) = self\n                            .proxy_to_h2_upstream(session, &mut h2, client_reused, &peer, ctx)\n                            .await;\n                        let session = ClientSession::H2(h2);\n                        self.client_upstream\n                            .release_http_session(session, &*peer, peer.idle_timeout())\n                            .await;\n\n                        if let Some(e) = error.as_mut() {\n                            // try to downgrade if A. origin says so or B. origin sends an invalid\n                            // response, which usually means origin h2 is not production ready\n                            if matches!(e.etype, H2Downgrade | InvalidH2) {\n                                if peer\n                                    .get_alpn()\n                                    .is_none_or(|alpn| alpn.get_min_http_version() == 1)\n                                {\n                                    // Add the peer to prefer h1 so that all following requests\n                                    // will use h1\n                                    self.client_upstream.prefer_h1(&*peer);\n                                } else {\n                                    // the peer doesn't allow downgrading to h1 (e.g. gRPC)\n                                    e.retry = false.into();\n                                }\n                            }\n                        }\n\n                        (server_reused, error)\n                    }\n                    ClientSession::Custom(mut c) => {\n                        let (server_reused, error) = self\n                            .proxy_to_custom_upstream(session, &mut c, client_reused, &peer, ctx)\n                            .await;\n                        let session = ClientSession::Custom(c);\n                        self.client_upstream\n                            .release_http_session(session, &*peer, peer.idle_timeout())\n                            .await;\n                        (server_reused, error)\n                    }\n                };\n                (\n                    server_reused,\n                    error.map(|e| {\n                        self.inner\n                            .error_while_proxy(&peer, session, e, ctx, client_reused)\n                    }),\n                )\n            }\n            Err(mut e) => {\n                e.as_up();\n                let new_err = self.inner.fail_to_connect(session, &peer, ctx, e);\n                (false, Some(new_err.into_up()))\n            }\n        }\n    }\n\n    async fn upstream_filter(\n        &self,\n        session: &mut Session,\n        task: &mut HttpTask,\n        ctx: &mut SV::CTX,\n    ) -> Result<Option<Duration>>\n    where\n        SV: ProxyHttp + Send + Sync,\n        SV::CTX: Send + Sync,\n    {\n        let duration = match task {\n            HttpTask::Header(header, _eos) => {\n                self.inner\n                    .upstream_response_filter(session, header, ctx)\n                    .await?;\n                None\n            }\n            HttpTask::Body(data, eos) | HttpTask::UpgradedBody(data, eos) => self\n                .inner\n                .upstream_response_body_filter(session, data, *eos, ctx)?,\n            HttpTask::Trailer(Some(trailers)) => {\n                self.inner\n                    .upstream_response_trailer_filter(session, trailers, ctx)?;\n                None\n            }\n            _ => {\n                // task does not support a filter\n                None\n            }\n        };\n\n        Ok(duration)\n    }\n\n    async fn finish(\n        &self,\n        mut session: Session,\n        ctx: &mut SV::CTX,\n        reuse: bool,\n        error: Option<Box<Error>>,\n    ) -> Option<ReusedHttpStream>\n    where\n        SV: ProxyHttp + Send + Sync,\n        SV::CTX: Send + Sync,\n    {\n        self.inner\n            .logging(&mut session, error.as_deref(), ctx)\n            .await;\n\n        if let Some(e) = error {\n            session.downstream_session.on_proxy_failure(e);\n        }\n\n        if reuse {\n            // TODO: log error\n            let persistent_settings = HttpPersistentSettings::for_session(&session);\n            session\n                .downstream_session\n                .finish()\n                .await\n                .ok()\n                .flatten()\n                .map(|s| ReusedHttpStream::new(s, Some(persistent_settings)))\n        } else {\n            None\n        }\n    }\n\n    fn cleanup_sub_req(&self, session: &mut Session) {\n        if let Some(ctx) = session.subrequest_ctx.as_mut() {\n            ctx.release_write_lock();\n        }\n    }\n}\n\nuse pingora_cache::HttpCache;\nuse pingora_core::protocols::http::compression::ResponseCompressionCtx;\n\n/// The established HTTP session\n///\n/// This object is what users interact with in order to access the request itself or change the proxy\n/// behavior.\npub struct Session {\n    /// the HTTP session to downstream (the client)\n    pub downstream_session: Box<HttpSession>,\n    /// The interface to control HTTP caching\n    pub cache: HttpCache,\n    /// (de)compress responses coming into the proxy (from upstream)\n    pub upstream_compression: ResponseCompressionCtx,\n    /// ignore downstream range (skip downstream range filters)\n    pub ignore_downstream_range: bool,\n    /// Were the upstream request headers modified?\n    pub upstream_headers_mutated_for_cache: bool,\n    /// The context from parent request, if this is a subrequest.\n    pub subrequest_ctx: Option<Box<SubrequestCtx>>,\n    /// Handle to allow spawning subrequests, assigned by the `Subrequest` app logic.\n    pub subrequest_spawner: Option<SubrequestSpawner>,\n    // Downstream filter modules\n    pub downstream_modules_ctx: HttpModuleCtx,\n    /// Upstream response body bytes received (payload only). Set by proxy layer.\n    /// TODO: move this into an upstream session digest for future fields.\n    upstream_body_bytes_received: usize,\n    /// Upstream write pending time. Set by proxy layer (HTTP/1.x only).\n    upstream_write_pending_time: Duration,\n    /// Flag that is set when the shutdown process has begun.\n    shutdown_flag: Arc<AtomicBool>,\n}\n\nimpl Session {\n    fn new(\n        downstream_session: impl Into<Box<HttpSession>>,\n        downstream_modules: &HttpModules,\n        shutdown_flag: Arc<AtomicBool>,\n    ) -> Self {\n        Session {\n            downstream_session: downstream_session.into(),\n            cache: HttpCache::new(),\n            // disable both upstream and downstream compression\n            upstream_compression: ResponseCompressionCtx::new(0, false, false),\n            ignore_downstream_range: false,\n            upstream_headers_mutated_for_cache: false,\n            subrequest_ctx: None,\n            subrequest_spawner: None, // optionally set later on\n            downstream_modules_ctx: downstream_modules.build_ctx(),\n            upstream_body_bytes_received: 0,\n            upstream_write_pending_time: Duration::ZERO,\n            shutdown_flag,\n        }\n    }\n\n    /// Create a new [Session] from the given [Stream]\n    ///\n    /// This function is mostly used for testing and mocking, given the downstream modules and\n    /// shutdown flags will never be set.\n    pub fn new_h1(stream: Stream) -> Self {\n        let modules = HttpModules::new();\n        Self::new(\n            Box::new(HttpSession::new_http1(stream)),\n            &modules,\n            Arc::new(AtomicBool::new(false)),\n        )\n    }\n\n    /// Create a new [Session] from the given [Stream] with modules\n    ///\n    /// This function is mostly used for testing and mocking, given the shutdown flag will never be\n    /// set.\n    pub fn new_h1_with_modules(stream: Stream, downstream_modules: &HttpModules) -> Self {\n        Self::new(\n            Box::new(HttpSession::new_http1(stream)),\n            downstream_modules,\n            Arc::new(AtomicBool::new(false)),\n        )\n    }\n\n    pub fn as_downstream_mut(&mut self) -> &mut HttpSession {\n        &mut self.downstream_session\n    }\n\n    pub fn as_downstream(&self) -> &HttpSession {\n        &self.downstream_session\n    }\n\n    /// Write HTTP response with the given error code to the downstream.\n    pub async fn respond_error(&mut self, error: u16) -> Result<()> {\n        self.as_downstream_mut().respond_error(error).await\n    }\n\n    /// Write HTTP response with the given error code to the downstream with a body.\n    pub async fn respond_error_with_body(&mut self, error: u16, body: Bytes) -> Result<()> {\n        self.as_downstream_mut()\n            .respond_error_with_body(error, body)\n            .await\n    }\n\n    /// Write the given HTTP response header to the downstream\n    ///\n    /// Different from directly calling [HttpSession::write_response_header], this function also\n    /// invokes the filter modules.\n    pub async fn write_response_header(\n        &mut self,\n        mut resp: Box<ResponseHeader>,\n        end_of_stream: bool,\n    ) -> Result<()> {\n        self.downstream_modules_ctx\n            .response_header_filter(&mut resp, end_of_stream)\n            .await?;\n        self.downstream_session.write_response_header(resp).await\n    }\n\n    /// Similar to `write_response_header()`, this fn will clone the `resp` internally\n    pub async fn write_response_header_ref(\n        &mut self,\n        resp: &ResponseHeader,\n        end_of_stream: bool,\n    ) -> Result<(), Box<Error>> {\n        self.write_response_header(Box::new(resp.clone()), end_of_stream)\n            .await\n    }\n\n    /// Write the given HTTP response body chunk to the downstream\n    ///\n    /// Different from directly calling [HttpSession::write_response_body], this function also\n    /// invokes the filter modules.\n    pub async fn write_response_body(\n        &mut self,\n        mut body: Option<Bytes>,\n        end_of_stream: bool,\n    ) -> Result<()> {\n        self.downstream_modules_ctx\n            .response_body_filter(&mut body, end_of_stream)?;\n\n        if body.is_none() && !end_of_stream {\n            return Ok(());\n        }\n\n        let data = body.unwrap_or_default();\n        self.downstream_session\n            .write_response_body(data, end_of_stream)\n            .await\n    }\n\n    pub async fn write_response_tasks(&mut self, mut tasks: Vec<HttpTask>) -> Result<bool> {\n        let mut seen_upgraded = self.was_upgraded();\n        for task in tasks.iter_mut() {\n            match task {\n                HttpTask::Header(resp, end) => {\n                    self.downstream_modules_ctx\n                        .response_header_filter(resp, *end)\n                        .await?;\n                }\n                HttpTask::Body(data, end) => {\n                    self.downstream_modules_ctx\n                        .response_body_filter(data, *end)?;\n                }\n                HttpTask::UpgradedBody(data, end) => {\n                    seen_upgraded = true;\n                    self.downstream_modules_ctx\n                        .response_body_filter(data, *end)?;\n                }\n                HttpTask::Trailer(trailers) => {\n                    if let Some(buf) = self\n                        .downstream_modules_ctx\n                        .response_trailer_filter(trailers)?\n                    {\n                        // Write the trailers into the body if the filter\n                        // returns a buffer.\n                        //\n                        // Note, this will not work if end of stream has already\n                        // been seen or we've written content-length bytes.\n                        // (Trailers should never come after upgraded body)\n                        *task = HttpTask::Body(Some(buf), true);\n                    }\n                }\n                HttpTask::Done => {\n                    // `Done` can be sent in certain response paths to mark end\n                    // of response if not already done via trailers or body with\n                    // end flag set.\n                    // If the filter returns body bytes on Done,\n                    // write them into the response.\n                    //\n                    // Note, this will not work if end of stream has already\n                    // been seen or we've written content-length bytes.\n                    if let Some(buf) = self.downstream_modules_ctx.response_done_filter()? {\n                        if seen_upgraded {\n                            *task = HttpTask::UpgradedBody(Some(buf), true);\n                        } else {\n                            *task = HttpTask::Body(Some(buf), true);\n                        }\n                    }\n                }\n                _ => { /* Failed */ }\n            }\n        }\n        self.downstream_session.response_duplex_vec(tasks).await\n    }\n\n    /// Mark the upstream headers as modified by caching. This should lead to range filters being\n    /// skipped when responding to the downstream.\n    pub fn mark_upstream_headers_mutated_for_cache(&mut self) {\n        self.upstream_headers_mutated_for_cache = true;\n    }\n\n    /// Check whether the upstream headers were marked as mutated during the request.\n    pub fn upstream_headers_mutated_for_cache(&self) -> bool {\n        self.upstream_headers_mutated_for_cache\n    }\n\n    /// Get the total upstream response body bytes received (payload only) recorded by the proxy layer.\n    pub fn upstream_body_bytes_received(&self) -> usize {\n        self.upstream_body_bytes_received\n    }\n\n    /// Set the total upstream response body bytes received (payload only). Intended for internal use by proxy layer.\n    pub(crate) fn set_upstream_body_bytes_received(&mut self, n: usize) {\n        self.upstream_body_bytes_received = n;\n    }\n\n    /// Get the upstream write pending time recorded by the proxy layer. Returns [`Duration::ZERO`] for HTTP/2.\n    pub fn upstream_write_pending_time(&self) -> Duration {\n        self.upstream_write_pending_time\n    }\n\n    /// Set the upstream write pending time. Intended for internal use by proxy layer.\n    pub(crate) fn set_upstream_write_pending_time(&mut self, d: Duration) {\n        self.upstream_write_pending_time = d;\n    }\n\n    /// Is the proxy process in the process of shutting down (e.g. due to graceful upgrade)?\n    pub fn is_process_shutting_down(&self) -> bool {\n        self.shutdown_flag.load(Ordering::Acquire)\n    }\n\n    pub fn downstream_custom_message(\n        &mut self,\n    ) -> Result<\n        Option<Box<dyn futures::Stream<Item = Result<Bytes>> + Unpin + Send + Sync + 'static>>,\n    > {\n        if let Some(custom_session) = self.downstream_session.as_custom_mut() {\n            custom_session\n                .take_custom_message_reader()\n                .map(Some)\n                .ok_or(Error::explain(\n                    ReadError,\n                    \"can't extract custom reader from downstream\",\n                ))\n        } else {\n            Ok(None)\n        }\n    }\n}\n\nimpl AsRef<HttpSession> for Session {\n    fn as_ref(&self) -> &HttpSession {\n        &self.downstream_session\n    }\n}\n\nimpl AsMut<HttpSession> for Session {\n    fn as_mut(&mut self) -> &mut HttpSession {\n        &mut self.downstream_session\n    }\n}\n\nuse std::ops::{Deref, DerefMut};\n\nimpl Deref for Session {\n    type Target = HttpSession;\n\n    fn deref(&self) -> &Self::Target {\n        &self.downstream_session\n    }\n}\n\nimpl DerefMut for Session {\n    fn deref_mut(&mut self) -> &mut Self::Target {\n        &mut self.downstream_session\n    }\n}\n\n// generic HTTP 502 response sent when proxy_upstream_filter refuses to connect to upstream\nstatic BAD_GATEWAY: Lazy<ResponseHeader> = Lazy::new(|| {\n    let mut resp = ResponseHeader::build(http::StatusCode::BAD_GATEWAY, Some(3)).unwrap();\n    resp.insert_header(header::SERVER, &SERVER_NAME[..])\n        .unwrap();\n    resp.insert_header(header::CONTENT_LENGTH, 0).unwrap();\n    resp.insert_header(header::CACHE_CONTROL, \"private, no-store\")\n        .unwrap();\n\n    resp\n});\n\nimpl<SV, C> HttpProxy<SV, C>\nwhere\n    C: custom::Connector,\n{\n    async fn process_request(\n        self: &Arc<Self>,\n        mut session: Session,\n        mut ctx: <SV as ProxyHttp>::CTX,\n    ) -> Option<ReusedHttpStream>\n    where\n        SV: ProxyHttp + Send + Sync + 'static,\n        <SV as ProxyHttp>::CTX: Send + Sync,\n    {\n        if let Err(e) = self\n            .inner\n            .early_request_filter(&mut session, &mut ctx)\n            .await\n        {\n            return self\n                .handle_error(session, &mut ctx, e, \"Fail to early filter request:\")\n                .await;\n        }\n\n        if self.inner.allow_spawning_subrequest(&session, &ctx) {\n            session.subrequest_spawner = Some(SubrequestSpawner::new(self.clone()));\n        }\n\n        let req = session.downstream_session.req_header_mut();\n\n        // Built-in downstream request filters go first\n        if let Err(e) = session\n            .downstream_modules_ctx\n            .request_header_filter(req)\n            .await\n        {\n            return self\n                .handle_error(\n                    session,\n                    &mut ctx,\n                    e,\n                    \"Failed in downstream modules request filter:\",\n                )\n                .await;\n        }\n\n        match self.inner.request_filter(&mut session, &mut ctx).await {\n            Ok(response_sent) => {\n                if response_sent {\n                    // TODO: log error\n                    self.inner.logging(&mut session, None, &mut ctx).await;\n                    self.cleanup_sub_req(&mut session);\n                    let persistent_settings = HttpPersistentSettings::for_session(&session);\n                    return session\n                        .downstream_session\n                        .finish()\n                        .await\n                        .ok()\n                        .flatten()\n                        .map(|s| ReusedHttpStream::new(s, Some(persistent_settings)));\n                }\n                /* else continue */\n            }\n            Err(e) => {\n                return self\n                    .handle_error(session, &mut ctx, e, \"Fail to filter request:\")\n                    .await;\n            }\n        }\n\n        if let Some((reuse, err)) = self.proxy_cache(&mut session, &mut ctx).await {\n            // cache hit\n            return self.finish(session, &mut ctx, reuse, err).await;\n        }\n        // either uncacheable, or cache miss\n\n        // there should not be a write lock in the sub req ctx after this point\n        self.cleanup_sub_req(&mut session);\n\n        // decide if the request is allowed to go to upstream\n        match self\n            .inner\n            .proxy_upstream_filter(&mut session, &mut ctx)\n            .await\n        {\n            Ok(proxy_to_upstream) => {\n                if !proxy_to_upstream {\n                    // The hook can choose to write its own response, but if it doesn't, we respond\n                    // with a generic 502\n                    if session.cache.enabled() {\n                        // drop the cache lock that this request may be holding onto\n                        session.cache.disable(NoCacheReason::DeclinedToUpstream);\n                    }\n                    if session.response_written().is_none() {\n                        match session.write_response_header_ref(&BAD_GATEWAY, true).await {\n                            Ok(()) => {}\n                            Err(e) => {\n                                return self\n                                    .handle_error(\n                                        session,\n                                        &mut ctx,\n                                        e,\n                                        \"Error responding with Bad Gateway:\",\n                                    )\n                                    .await;\n                            }\n                        }\n                    }\n\n                    return self.finish(session, &mut ctx, true, None).await;\n                }\n                /* else continue */\n            }\n            Err(e) => {\n                if session.cache.enabled() {\n                    session.cache.disable(NoCacheReason::InternalError);\n                }\n\n                return self\n                    .handle_error(\n                        session,\n                        &mut ctx,\n                        e,\n                        \"Error deciding if we should proxy to upstream:\",\n                    )\n                    .await;\n            }\n        }\n\n        let mut retries: usize = 0;\n\n        let mut server_reuse = false;\n        let mut proxy_error: Option<Box<Error>> = None;\n\n        while retries < self.max_retries {\n            retries += 1;\n\n            let (reuse, e) = self.proxy_to_upstream(&mut session, &mut ctx).await;\n            server_reuse = reuse;\n\n            match e {\n                Some(error) => {\n                    let retry = error.retry();\n                    proxy_error = Some(error);\n                    if !retry {\n                        break;\n                    }\n                    // only log error that will be retried here, the final error will be logged below\n                    warn!(\n                        \"Fail to proxy: {}, tries: {}, retry: {}, {}\",\n                        proxy_error.as_ref().unwrap(),\n                        retries,\n                        retry,\n                        self.inner.request_summary(&session, &ctx)\n                    );\n                }\n                None => {\n                    proxy_error = None;\n                    break;\n                }\n            };\n        }\n\n        // serve stale if error\n        // Check both error and cache before calling the function because await is not cheap\n        // allow unwrap until if let chains\n        #[allow(clippy::unnecessary_unwrap)]\n        let serve_stale_result = if proxy_error.is_some() && session.cache.can_serve_stale_error() {\n            self.handle_stale_if_error(&mut session, &mut ctx, proxy_error.as_ref().unwrap())\n                .await\n        } else {\n            None\n        };\n\n        let final_error = if let Some((reuse, stale_cache_error)) = serve_stale_result {\n            // don't reuse server conn if serve stale polluted it\n            server_reuse = server_reuse && reuse;\n            stale_cache_error\n        } else {\n            proxy_error\n        };\n\n        if let Some(e) = final_error.as_ref() {\n            // If we have errored and are still holding a cache lock, release it.\n            if session.cache.enabled() {\n                let reason = if *e.esource() == ErrorSource::Upstream {\n                    NoCacheReason::UpstreamError\n                } else {\n                    NoCacheReason::InternalError\n                };\n                session.cache.disable(reason);\n            }\n            let res = self.inner.fail_to_proxy(&mut session, e, &mut ctx).await;\n\n            // final error will have > 0 status unless downstream connection is dead\n            if !self.inner.suppress_error_log(&session, &ctx, e) {\n                error!(\n                    \"Fail to proxy: {}, status: {}, tries: {}, retry: {}, {}\",\n                    final_error.as_ref().unwrap(),\n                    res.error_code,\n                    retries,\n                    false, // we never retry here\n                    self.inner.request_summary(&session, &ctx),\n                );\n            }\n        }\n\n        // logging() will be called in finish()\n        self.finish(session, &mut ctx, server_reuse, final_error)\n            .await\n    }\n\n    async fn handle_error(\n        &self,\n        mut session: Session,\n        ctx: &mut <SV as ProxyHttp>::CTX,\n        e: Box<Error>,\n        context: &str,\n    ) -> Option<ReusedHttpStream>\n    where\n        SV: ProxyHttp + Send + Sync + 'static,\n        <SV as ProxyHttp>::CTX: Send + Sync,\n    {\n        let res = self.inner.fail_to_proxy(&mut session, &e, ctx).await;\n        if !self.inner.suppress_error_log(&session, ctx, &e) {\n            error!(\n                \"{context} {}, status: {}, {}\",\n                e,\n                res.error_code,\n                self.inner.request_summary(&session, ctx)\n            );\n        }\n        self.inner.logging(&mut session, Some(&e), ctx).await;\n        self.cleanup_sub_req(&mut session);\n\n        session.downstream_session.on_proxy_failure(e);\n\n        if res.can_reuse_downstream {\n            let persistent_settings = HttpPersistentSettings::for_session(&session);\n            session\n                .downstream_session\n                .finish()\n                .await\n                .ok()\n                .flatten()\n                .map(|s| ReusedHttpStream::new(s, Some(persistent_settings)))\n        } else {\n            None\n        }\n    }\n}\n\n/* Make process_subrequest() a trait to workaround https://github.com/rust-lang/rust/issues/78649\n   if process_subrequest() is implemented as a member of HttpProxy, rust complains\n\nerror[E0391]: cycle detected when computing type of `proxy_cache::<impl at pingora-proxy/src/proxy_cache.rs:7:1: 7:23>::proxy_cache::{opaque#0}`\n   --> pingora-proxy/src/proxy_cache.rs:13:10\n    |\n13  |     ) -> Option<(bool, Option<Box<Error>>)>\n\n*/\n#[async_trait]\npub trait Subrequest {\n    async fn process_subrequest(\n        self: Arc<Self>,\n        session: Box<HttpSession>,\n        sub_req_ctx: Box<SubrequestCtx>,\n    );\n}\n\n#[async_trait]\nimpl<SV, C> Subrequest for HttpProxy<SV, C>\nwhere\n    SV: ProxyHttp + Send + Sync + 'static,\n    <SV as ProxyHttp>::CTX: Send + Sync,\n    C: custom::Connector,\n{\n    async fn process_subrequest(\n        self: Arc<Self>,\n        session: Box<HttpSession>,\n        sub_req_ctx: Box<SubrequestCtx>,\n    ) {\n        debug!(\"starting subrequest\");\n\n        let mut session = match self.handle_new_request(session).await {\n            Some(downstream_session) => Session::new(\n                downstream_session,\n                &self.downstream_modules,\n                self.shutdown_flag.clone(),\n            ),\n            None => return, // bad request\n        };\n\n        // no real downstream to keepalive, but it doesn't matter what is set here because at the end\n        // of this fn the dummy connection will be dropped\n        session.set_keepalive(None);\n\n        session.subrequest_ctx.replace(sub_req_ctx);\n        trace!(\"processing subrequest\");\n        let ctx = self.inner.new_ctx();\n        self.process_request(session, ctx).await;\n        trace!(\"subrequest done\");\n    }\n}\n\n/// A handle to the underlying HTTP proxy app that allows spawning subrequests.\npub struct SubrequestSpawner {\n    app: Arc<dyn Subrequest + Send + Sync>,\n}\n\n/// A [`PreparedSubrequest`] that is ready to run.\npub struct PreparedSubrequest {\n    app: Arc<dyn Subrequest + Send + Sync>,\n    session: Box<HttpSession>,\n    sub_req_ctx: Box<SubrequestCtx>,\n}\n\nimpl PreparedSubrequest {\n    pub async fn run(self) {\n        self.app\n            .process_subrequest(self.session, self.sub_req_ctx)\n            .await\n    }\n\n    pub fn session(&self) -> &HttpSession {\n        self.session.as_ref()\n    }\n\n    pub fn session_mut(&mut self) -> &mut HttpSession {\n        self.session.deref_mut()\n    }\n}\n\nimpl SubrequestSpawner {\n    /// Create a new [`SubrequestSpawner`].\n    pub fn new(app: Arc<dyn Subrequest + Send + Sync>) -> SubrequestSpawner {\n        SubrequestSpawner { app }\n    }\n\n    /// Spawn a background subrequest and return a join handle.\n    // TODO: allow configuring the subrequest session before use\n    pub fn spawn_background_subrequest(\n        &self,\n        session: &HttpSession,\n        ctx: SubrequestCtx,\n    ) -> tokio::task::JoinHandle<()> {\n        let new_app = self.app.clone(); // Clone the Arc\n        let (mut session, handle) = subrequest::create_session(session);\n        if ctx.body_mode() == BodyMode::NoBody {\n            session\n                .as_subrequest_mut()\n                .expect(\"created subrequest session\")\n                .clear_request_body_headers();\n        }\n        let sub_req_ctx = Box::new(ctx);\n        handle.drain_tasks();\n        tokio::spawn(async move {\n            new_app\n                .process_subrequest(Box::new(session), sub_req_ctx)\n                .await;\n        })\n    }\n\n    /// Create a subrequest that listens to `HttpTask`s sent from the returned `Sender`\n    /// and sends `HttpTask`s to the returned `Receiver`.\n    ///\n    /// To run that subrequest, call `run()`.\n    // TODO: allow configuring the subrequest session before use\n    pub fn create_subrequest(\n        &self,\n        session: &HttpSession,\n        ctx: SubrequestCtx,\n    ) -> (PreparedSubrequest, SubrequestHandle) {\n        let new_app = self.app.clone(); // Clone the Arc\n        let (mut session, handle) = subrequest::create_session(session);\n        if ctx.body_mode() == BodyMode::NoBody {\n            session\n                .as_subrequest_mut()\n                .expect(\"created subrequest session\")\n                .clear_request_body_headers();\n        }\n        let sub_req_ctx = Box::new(ctx);\n        (\n            PreparedSubrequest {\n                app: new_app,\n                session: Box::new(session),\n                sub_req_ctx,\n            },\n            handle,\n        )\n    }\n}\n\n#[async_trait]\nimpl<SV, C> HttpServerApp for HttpProxy<SV, C>\nwhere\n    SV: ProxyHttp + Send + Sync + 'static,\n    <SV as ProxyHttp>::CTX: Send + Sync,\n    C: custom::Connector,\n{\n    async fn process_new_http(\n        self: &Arc<Self>,\n        session: HttpSession,\n        shutdown: &ShutdownWatch,\n    ) -> Option<ReusedHttpStream> {\n        let session = Box::new(session);\n\n        // TODO: keepalive pool, use stack\n        let mut session = match self.handle_new_request(session).await {\n            Some(downstream_session) => Session::new(\n                downstream_session,\n                &self.downstream_modules,\n                self.shutdown_flag.clone(),\n            ),\n            None => return None, // bad request\n        };\n\n        if *shutdown.borrow() {\n            // stop downstream from reusing if this service is shutting down soon\n            session.set_keepalive(None);\n        }\n\n        let ctx = self.inner.new_ctx();\n        self.process_request(session, ctx).await\n    }\n\n    async fn http_cleanup(&self) {\n        self.shutdown_flag.store(true, Ordering::Release);\n        // Notify all keepalived requests blocking on read_request() to abort\n        self.shutdown.notify_waiters();\n    }\n\n    fn server_options(&self) -> Option<&HttpServerOptions> {\n        self.server_options.as_ref()\n    }\n\n    fn h2_options(&self) -> Option<H2Options> {\n        self.h2_options.clone()\n    }\n    async fn process_custom_session(\n        self: Arc<Self>,\n        stream: Stream,\n        shutdown: &ShutdownWatch,\n    ) -> Option<Stream> {\n        let app = self.clone();\n\n        let Some(process_custom_session) = app.process_custom_session.as_ref() else {\n            warn!(\"custom was called on an empty on_custom\");\n            return None;\n        };\n\n        process_custom_session(self.clone(), stream, shutdown).await\n    }\n\n    // TODO implement h2_options\n}\n\nuse pingora_core::services::listening::Service;\n\n/// Create an [`HttpProxy`] without wrapping it in a [`Service`].\n///\n/// This is useful when you need to integrate `HttpProxy` into a custom accept loop,\n/// for example when implementing SNI-based routing that decides between TLS passthrough\n/// and TLS termination on a single port.\n///\n/// The returned `HttpProxy` is fully initialized and ready to process requests via\n/// [`HttpServerApp::process_new_http()`].\n///\n/// # Example\n///\n/// ```ignore\n/// use pingora_proxy::http_proxy;\n/// use std::sync::Arc;\n///\n/// // Create the proxy\n/// let proxy = Arc::new(http_proxy(&server_conf, my_proxy_app));\n///\n/// // In your custom accept loop:\n/// loop {\n///     let (stream, addr) = listener.accept().await?;\n///\n///     // Peek SNI, decide routing...\n///     if should_terminate_tls {\n///         let tls_stream = my_acceptor.accept(stream).await?;\n///         let session = HttpSession::new_http1(Box::new(tls_stream));\n///         proxy.process_new_http(session, &shutdown).await;\n///     }\n/// }\n/// ```\npub fn http_proxy<SV>(conf: &Arc<ServerConf>, inner: SV) -> HttpProxy<SV>\nwhere\n    SV: ProxyHttp,\n{\n    let mut proxy = HttpProxy::new(inner, conf.clone());\n    proxy.handle_init_modules();\n    proxy\n}\n\n/// Create a [Service] from the user implemented [ProxyHttp].\n///\n/// The returned [Service] can be hosted by a [pingora_core::server::Server] directly.\npub fn http_proxy_service<SV>(conf: &Arc<ServerConf>, inner: SV) -> Service<HttpProxy<SV, ()>>\nwhere\n    SV: ProxyHttp,\n{\n    http_proxy_service_with_name(conf, inner, \"Pingora HTTP Proxy Service\")\n}\n\n/// Create a [Service] from the user implemented [ProxyHttp].\n///\n/// The returned [Service] can be hosted by a [pingora_core::server::Server] directly.\npub fn http_proxy_service_with_name<SV>(\n    conf: &Arc<ServerConf>,\n    inner: SV,\n    name: &str,\n) -> Service<HttpProxy<SV, ()>>\nwhere\n    SV: ProxyHttp,\n{\n    let mut proxy = HttpProxy::new(inner, conf.clone());\n    proxy.handle_init_modules();\n    Service::new(name.to_string(), proxy)\n}\n\n/// Create a [Service] from the user implemented [ProxyHttp].\n///\n/// The returned [Service] can be hosted by a [pingora_core::server::Server] directly.\npub fn http_proxy_service_with_name_custom<SV, C>(\n    conf: &Arc<ServerConf>,\n    inner: SV,\n    name: &str,\n    connector: C,\n    on_custom: ProcessCustomSession<SV, C>,\n) -> Service<HttpProxy<SV, C>>\nwhere\n    SV: ProxyHttp + Send + Sync + 'static,\n    SV::CTX: Send + Sync + 'static,\n    C: custom::Connector,\n{\n    let mut proxy = HttpProxy::new_custom(inner, conf.clone(), connector, Some(on_custom), None);\n    proxy.handle_init_modules();\n\n    Service::new(name.to_string(), proxy)\n}\n\n/// A builder for a [Service] that can be used to create a [HttpProxy] instance\n///\n/// The [ProxyServiceBuilder] can be used to construct a [HttpProxy] service with a custom name,\n/// connector, and custom session handler.\n///\npub struct ProxyServiceBuilder<SV, C>\nwhere\n    SV: ProxyHttp + Send + Sync + 'static,\n    SV::CTX: Send + Sync + 'static,\n    C: custom::Connector,\n{\n    conf: Arc<ServerConf>,\n    inner: SV,\n    name: String,\n    connector: C,\n    custom: Option<ProcessCustomSession<SV, C>>,\n    server_options: Option<HttpServerOptions>,\n}\n\nimpl<SV> ProxyServiceBuilder<SV, ()>\nwhere\n    SV: ProxyHttp + Send + Sync + 'static,\n    SV::CTX: Send + Sync + 'static,\n{\n    /// Create a new [ProxyServiceBuilder] with the given [ServerConf] and [ProxyHttp]\n    /// implementation.\n    ///\n    /// The returned builder can be used to construct a [HttpProxy] service with a custom name,\n    /// connector, and custom session handler.\n    ///\n    /// The [ProxyServiceBuilder] will default to using the [ProxyHttp] implementation and no custom\n    /// session handler.\n    ///\n    pub fn new(conf: &Arc<ServerConf>, inner: SV) -> Self {\n        ProxyServiceBuilder {\n            conf: conf.clone(),\n            inner,\n            name: \"Pingora HTTP Proxy Service\".into(),\n            connector: (),\n            custom: None,\n            server_options: None,\n        }\n    }\n}\n\nimpl<SV, C> ProxyServiceBuilder<SV, C>\nwhere\n    SV: ProxyHttp + Send + Sync + 'static,\n    SV::CTX: Send + Sync + 'static,\n    C: custom::Connector,\n{\n    /// Sets the name of the [HttpProxy] service.\n    pub fn name(mut self, name: impl AsRef<str>) -> Self {\n        self.name = name.as_ref().to_owned();\n        self\n    }\n\n    /// Set a custom connector and custom session handler for the [ProxyServiceBuilder].\n    ///\n    /// The custom connector is used to establish a connection to the upstream server.\n    ///\n    /// The custom session handler is used to handle custom protocol specific logic\n    /// between the proxy and the upstream server.\n    ///\n    /// Returns a new [ProxyServiceBuilder] with the custom connector and session handler.\n    pub fn custom<C2: custom::Connector>(\n        self,\n        connector: C2,\n        on_custom: ProcessCustomSession<SV, C2>,\n    ) -> ProxyServiceBuilder<SV, C2> {\n        let Self {\n            conf,\n            inner,\n            name,\n            server_options,\n            ..\n        } = self;\n        ProxyServiceBuilder {\n            conf,\n            inner,\n            name,\n            connector,\n            custom: Some(on_custom),\n            server_options,\n        }\n    }\n\n    /// Set the server options for the [ProxyServiceBuilder].\n    ///\n    /// Returns a new [ProxyServiceBuilder] with the server options set.\n    pub fn server_options(mut self, options: HttpServerOptions) -> Self {\n        self.server_options = Some(options);\n        self\n    }\n\n    /// Builds a new [Service] from the [ProxyServiceBuilder].\n    ///\n    /// This function takes ownership of the [ProxyServiceBuilder] and returns a new [Service] with\n    /// a fully initialized [HttpProxy].\n    ///\n    /// The returned [Service] is ready to be used by a [pingora_core::server::Server].\n    pub fn build(self) -> Service<HttpProxy<SV, C>> {\n        let Self {\n            conf,\n            inner,\n            name,\n            connector,\n            custom,\n            server_options,\n        } = self;\n\n        let mut proxy = HttpProxy::new_custom(inner, conf, connector, custom, server_options);\n\n        proxy.handle_init_modules();\n        Service::new(name, proxy)\n    }\n}\n"
  },
  {
    "path": "pingora-proxy/src/proxy_cache.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\nuse super::*;\nuse http::header::{CONTENT_ENCODING, CONTENT_LENGTH, CONTENT_TYPE, TRANSFER_ENCODING};\nuse http::{Method, StatusCode};\nuse pingora_cache::key::CacheHashKey;\nuse pingora_cache::lock::LockStatus;\nuse pingora_cache::max_file_size::ERR_RESPONSE_TOO_LARGE;\nuse pingora_cache::{ForcedFreshness, HitHandler, HitStatus, RespCacheable::*};\nuse pingora_core::protocols::http::conditional_filter::to_304;\nuse pingora_core::protocols::http::v1::common::header_value_content_length;\nuse pingora_core::ErrorType;\nuse range_filter::RangeBodyFilter;\nuse std::time::SystemTime;\n\nimpl<SV, C> HttpProxy<SV, C>\nwhere\n    C: custom::Connector,\n{\n    // return bool: server_session can be reused, and error if any\n    pub(crate) async fn proxy_cache(\n        self: &Arc<Self>,\n        session: &mut Session,\n        ctx: &mut SV::CTX,\n    ) -> Option<(bool, Option<Box<Error>>)>\n    // None: continue to proxy, Some: return\n    where\n        SV: ProxyHttp + Send + Sync + 'static,\n        SV::CTX: Send + Sync,\n    {\n        // Cache logic request phase\n        if let Err(e) = self.inner.request_cache_filter(session, ctx) {\n            // TODO: handle this error\n            warn!(\n                \"Fail to request_cache_filter: {e}, {}\",\n                self.inner.request_summary(session, ctx)\n            );\n        }\n\n        // cache key logic, should this be part of request_cache_filter?\n        if session.cache.enabled() {\n            match self.inner.cache_key_callback(session, ctx) {\n                Ok(key) => {\n                    session.cache.set_cache_key(key);\n                }\n                Err(e) => {\n                    // TODO: handle this error\n                    session.cache.disable(NoCacheReason::StorageError);\n                    warn!(\n                        \"Fail to cache_key_callback: {e}, {}\",\n                        self.inner.request_summary(session, ctx)\n                    );\n                }\n            }\n        }\n\n        // cache purge logic: PURGE short-circuits rest of request\n        if self.inner.is_purge(session, ctx) {\n            return self.proxy_purge(session, ctx).await;\n        }\n\n        // bypass cache lookup if we predict to be uncacheable\n        if session.cache.enabled() && !session.cache.cacheable_prediction() {\n            session.cache.bypass();\n        }\n\n        if !session.cache.enabled() {\n            return None;\n        }\n\n        // cache lookup logic\n        loop {\n            // for cache lock, TODO: cap the max number of loops\n            match session.cache.cache_lookup().await {\n                Ok(res) => {\n                    let mut hit_status_opt = None;\n                    if let Some((mut meta, mut handler)) = res {\n                        // Vary logic\n                        // Because this branch can be called multiple times in a loop, and we only\n                        // need to update the vary once, check if variance is already set to\n                        // prevent unnecessary vary lookups.\n                        let cache_key = session.cache.cache_key();\n                        if let Some(variance) = cache_key.variance_bin() {\n                            // We've looked up a secondary slot.\n                            // Adhoc double check that the variance found is the variance we want.\n                            if Some(variance) != meta.variance() {\n                                warn!(\"Cache variance mismatch, {variance:?}, {cache_key:?}\");\n                                session.cache.disable(NoCacheReason::InternalError);\n                                break None;\n                            }\n                        } else {\n                            // Basic cache key; either variance is off, or this is the primary slot.\n                            let req_header = session.req_header();\n                            let variance = self.inner.cache_vary_filter(&meta, ctx, req_header);\n                            if let Some(variance) = variance {\n                                // Variance is on. This is the primary slot.\n                                if !session.cache.cache_vary_lookup(variance, &meta) {\n                                    // This wasn't the desired variant. Updated cache key variance, cause another\n                                    // lookup to get the desired variant, which would be in a secondary slot.\n                                    continue;\n                                }\n                            } // else: vary is not in use\n                        }\n\n                        // Either no variance, or the current handler targets the correct variant.\n\n                        // hit\n                        // TODO: maybe round and/or cache now()\n                        let is_fresh = meta.is_fresh(SystemTime::now());\n                        // check if we should force expire or force miss\n                        let hit_status = match self\n                            .inner\n                            .cache_hit_filter(session, &meta, &mut handler, is_fresh, ctx)\n                            .await\n                        {\n                            Err(e) => {\n                                error!(\n                                    \"Failed to filter cache hit: {e}, {}\",\n                                    self.inner.request_summary(session, ctx)\n                                );\n                                // this return value will cause us to fetch from upstream\n                                HitStatus::FailedHitFilter\n                            }\n                            Ok(None) => {\n                                if is_fresh {\n                                    HitStatus::Fresh\n                                } else {\n                                    HitStatus::Expired\n                                }\n                            }\n                            Ok(Some(ForcedFreshness::ForceExpired)) => {\n                                // force expired asset should not be serve as stale\n                                // because force expire is usually to remove data\n                                meta.disable_serve_stale();\n                                HitStatus::ForceExpired\n                            }\n                            Ok(Some(ForcedFreshness::ForceMiss)) => HitStatus::ForceMiss,\n                            Ok(Some(ForcedFreshness::ForceFresh)) => HitStatus::Fresh,\n                        };\n\n                        hit_status_opt = Some(hit_status);\n\n                        // init cache for hit / stale\n                        session.cache.cache_found(meta, handler, hit_status);\n                    }\n\n                    if hit_status_opt.is_none_or(HitStatus::is_treated_as_miss) {\n                        // cache miss\n                        if session.cache.is_cache_locked() {\n                            // Another request is filling the cache; try waiting til that's done and retry.\n                            let lock_status = session.cache.cache_lock_wait().await;\n                            if self.handle_lock_status(session, ctx, lock_status) {\n                                continue;\n                            } else {\n                                break None;\n                            }\n                        } else {\n                            self.inner.cache_miss(session, ctx);\n                            break None;\n                        }\n                    }\n\n                    // Safe because an empty hit status would have broken out\n                    // in the block above\n                    let hit_status = hit_status_opt.expect(\"None case handled as miss\");\n\n                    if !hit_status.is_fresh() {\n                        // expired or force expired asset\n                        if session.cache.is_cache_locked() {\n                            // first if this is the sub request for the background cache update\n                            if let Some(write_lock) = session\n                                .subrequest_ctx\n                                .as_mut()\n                                .and_then(|ctx| ctx.take_write_lock())\n                            {\n                                // Put the write lock in the request\n                                session.cache.set_write_lock(write_lock);\n                                session.cache.tag_as_subrequest();\n                                // and then let it go to upstream\n                                break None;\n                            }\n                            let will_serve_stale = session.cache.can_serve_stale_updating()\n                                && self.inner.should_serve_stale(session, ctx, None);\n                            if !will_serve_stale {\n                                let lock_status = session.cache.cache_lock_wait().await;\n                                if self.handle_lock_status(session, ctx, lock_status) {\n                                    continue;\n                                } else {\n                                    break None;\n                                }\n                            }\n                            // else continue to serve stale\n                            session.cache.set_stale_updating();\n                        } else if session.cache.is_cache_lock_writer() {\n                            // stale while revalidate logic for the writer\n                            let will_serve_stale = session.cache.can_serve_stale_updating()\n                                && self.inner.should_serve_stale(session, ctx, None);\n                            if will_serve_stale {\n                                // create a background thread to do the actual update\n                                // the subrequest handle is only None by this phase in unit tests\n                                // that don't go through process_new_http\n                                let (permit, cache_lock) = session.cache.take_write_lock();\n                                SubrequestSpawner::new(self.clone()).spawn_background_subrequest(\n                                    session.as_ref(),\n                                    subrequest::Ctx::builder()\n                                        .cache_write_lock(\n                                            cache_lock,\n                                            session.cache.cache_key().clone(),\n                                            permit,\n                                        )\n                                        .build(),\n                                );\n                                // continue to serve stale for this request\n                                session.cache.set_stale_updating();\n                            } else {\n                                // return to fetch from upstream\n                                break None;\n                            }\n                        } else {\n                            // return to fetch from upstream\n                            break None;\n                        }\n                    }\n\n                    let (reuse, err) = self.proxy_cache_hit(session, ctx).await;\n                    if let Some(e) = err.as_ref() {\n                        error!(\n                            \"Fail to serve cache: {e}, {}\",\n                            self.inner.request_summary(session, ctx)\n                        );\n                    }\n                    // responses is served from cache, exit\n                    break Some((reuse, err));\n                }\n                Err(e) => {\n                    // Allow cache miss to fill cache even if cache lookup errors\n                    // this is mostly to support backward incompatible metadata update\n                    // TODO: check error types\n                    // session.cache.disable();\n                    self.inner.cache_miss(session, ctx);\n                    warn!(\n                        \"Fail to cache lookup: {e}, {}\",\n                        self.inner.request_summary(session, ctx)\n                    );\n                    break None;\n                }\n            }\n        }\n    }\n\n    // return bool: server_session can be reused, and error if any\n    pub(crate) async fn proxy_cache_hit(\n        &self,\n        session: &mut Session,\n        ctx: &mut SV::CTX,\n    ) -> (bool, Option<Box<Error>>)\n    where\n        SV: ProxyHttp + Send + Sync,\n        SV::CTX: Send + Sync,\n    {\n        use range_filter::*;\n\n        let seekable = session.cache.hit_handler().can_seek();\n        let mut header = cache_hit_header(&session.cache);\n\n        let req = session.req_header();\n\n        let not_modified = match self.inner.cache_not_modified_filter(session, &header, ctx) {\n            Ok(not_modified) => not_modified,\n            Err(e) => {\n                // fail open if cache_not_modified_filter errors,\n                // just return the whole original response\n                warn!(\n                    \"Failed to run cache not modified filter: {e}, {}\",\n                    self.inner.request_summary(session, ctx)\n                );\n                false\n            }\n        };\n        if not_modified {\n            to_304(&mut header);\n        }\n        let header_only = not_modified || req.method == http::method::Method::HEAD;\n\n        // process range header if the cache storage supports seek\n        let range_type = if seekable && !session.ignore_downstream_range {\n            self.inner.range_header_filter(session, &mut header, ctx)\n        } else {\n            RangeType::None\n        };\n\n        // return a 416 with an empty body for simplicity\n        let header_only = header_only || matches!(range_type, RangeType::Invalid);\n        debug!(\"header: {header:?}\");\n\n        // TODO: use ProxyUseCache to replace the logic below\n        match self.inner.response_filter(session, &mut header, ctx).await {\n            Ok(_) => {\n                if let Err(e) = session\n                    .downstream_modules_ctx\n                    .response_header_filter(&mut header, header_only)\n                    .await\n                {\n                    error!(\n                        \"Failed to run downstream modules response header filter in hit: {e}, {}\",\n                        self.inner.request_summary(session, ctx)\n                    );\n                    session\n                        .as_mut()\n                        .respond_error(500)\n                        .await\n                        .unwrap_or_else(|e| {\n                            error!(\"failed to send error response to downstream: {e}\");\n                        });\n                    // we have not write anything dirty to downstream, it is still reusable\n                    return (true, Some(e));\n                }\n\n                if let Err(e) = session\n                    .as_mut()\n                    .write_response_header(header)\n                    .await\n                    .map_err(|e| e.into_down())\n                {\n                    // downstream connection is bad already\n                    return (false, Some(e));\n                }\n            }\n            Err(e) => {\n                error!(\n                    \"Failed to run response filter in hit: {e}, {}\",\n                    self.inner.request_summary(session, ctx)\n                );\n                session\n                    .as_mut()\n                    .respond_error(500)\n                    .await\n                    .unwrap_or_else(|e| {\n                        error!(\"failed to send error response to downstream: {e}\");\n                    });\n                // we have not write anything dirty to downstream, it is still reusable\n                return (true, Some(e));\n            }\n        }\n        debug!(\"finished sending cached header to downstream\");\n\n        // If the function returns an Err, there was an issue seeking from the hit handler.\n        //\n        // Returning false means that no seeking or state change was done, either because the\n        // hit handler doesn't support the seek or because multipart doesn't apply.\n        fn seek_multipart(\n            hit_handler: &mut HitHandler,\n            range_filter: &mut RangeBodyFilter,\n        ) -> Result<bool> {\n            if !range_filter.is_multipart_range() || !hit_handler.can_seek_multipart() {\n                return Ok(false);\n            }\n            let r = range_filter.next_cache_multipart_range();\n            hit_handler.seek_multipart(r.start, Some(r.end))?;\n            // we still need RangeBodyFilter's help to transform the byte\n            // range into a multipart response.\n            range_filter.set_current_cursor(r.start);\n            Ok(true)\n        }\n\n        if !header_only {\n            let mut maybe_range_filter = match &range_type {\n                RangeType::Single(r) => {\n                    if session.cache.hit_handler().can_seek() {\n                        if let Err(e) = session.cache.hit_handler().seek(r.start, Some(r.end)) {\n                            return (false, Some(e));\n                        }\n                        None\n                    } else {\n                        Some(RangeBodyFilter::new_range(range_type.clone()))\n                    }\n                }\n                RangeType::Multi(_) => {\n                    let mut range_filter = RangeBodyFilter::new_range(range_type.clone());\n                    if let Err(e) = seek_multipart(session.cache.hit_handler(), &mut range_filter) {\n                        return (false, Some(e));\n                    }\n                    Some(range_filter)\n                }\n                RangeType::Invalid => unreachable!(),\n                RangeType::None => None,\n            };\n            loop {\n                match session.cache.hit_handler().read_body().await {\n                    Ok(raw_body) => {\n                        let end = raw_body.is_none();\n\n                        if end {\n                            if let Some(range_filter) = maybe_range_filter.as_mut() {\n                                if range_filter.should_cache_seek_again() {\n                                    let e = match seek_multipart(\n                                        session.cache.hit_handler(),\n                                        range_filter,\n                                    ) {\n                                        Ok(true) => {\n                                            // called seek(), read again\n                                            continue;\n                                        }\n                                        Ok(false) => {\n                                            // body reader can no longer seek multipart,\n                                            // but cache wants to continue seeking\n                                            // the body will just end in this case if we pass the\n                                            // None through\n                                            // (TODO: how might hit handlers want to recover from\n                                            // this situation)?\n                                            Error::explain(\n                                                InternalError,\n                                                \"hit handler cannot seek for multipart again\",\n                                            )\n                                            // the body will just end in this case.\n                                        }\n                                        Err(e) => e,\n                                    };\n                                    return (false, Some(e));\n                                }\n                            }\n                        }\n\n                        let mut body = if let Some(range_filter) = maybe_range_filter.as_mut() {\n                            range_filter.filter_body(raw_body)\n                        } else {\n                            raw_body\n                        };\n\n                        match self\n                            .inner\n                            .response_body_filter(session, &mut body, end, ctx)\n                        {\n                            Ok(Some(duration)) => {\n                                trace!(\"delaying response for {duration:?}\");\n                                time::sleep(duration).await;\n                            }\n                            Ok(None) => { /* continue */ }\n                            Err(e) => {\n                                // body is being sent, don't treat downstream as reusable\n                                return (false, Some(e));\n                            }\n                        }\n\n                        if let Err(e) = session\n                            .downstream_modules_ctx\n                            .response_body_filter(&mut body, end)\n                        {\n                            // body is being sent, don't treat downstream as reusable\n                            return (false, Some(e));\n                        }\n\n                        if !end && body.as_ref().is_none_or(|b| b.is_empty()) {\n                            // Don't write empty body which will end session,\n                            // still more hit handler bytes to read\n                            continue;\n                        }\n\n                        // write to downstream\n                        let b = body.unwrap_or_default();\n                        if let Err(e) = session\n                            .as_mut()\n                            .write_response_body(b, end)\n                            .await\n                            .map_err(|e| e.into_down())\n                        {\n                            return (false, Some(e));\n                        }\n                        if end {\n                            break;\n                        }\n                    }\n                    Err(e) => return (false, Some(e)),\n                }\n            }\n        }\n\n        if let Err(e) = session.cache.finish_hit_handler().await {\n            warn!(\"Error during finish_hit_handler: {}\", e);\n        }\n\n        match session.as_mut().finish_body().await {\n            Ok(_) => {\n                debug!(\"finished sending cached body to downstream\");\n                (true, None)\n            }\n            Err(e) => (false, Some(e)),\n        }\n    }\n\n    /* Downstream revalidation, only needed when cache is on because otherwise origin\n     * will handle it */\n    pub(crate) fn downstream_response_conditional_filter(\n        &self,\n        use_cache: &mut ServeFromCache,\n        session: &Session,\n        resp: &mut ResponseHeader,\n        ctx: &mut SV::CTX,\n    ) where\n        SV: ProxyHttp,\n    {\n        // TODO: range\n        let req = session.req_header();\n\n        let not_modified = match self.inner.cache_not_modified_filter(session, resp, ctx) {\n            Ok(not_modified) => not_modified,\n            Err(e) => {\n                // fail open if cache_not_modified_filter errors,\n                // just return the whole original response\n                warn!(\n                    \"Failed to run cache not modified filter: {e}, {}\",\n                    self.inner.request_summary(session, ctx)\n                );\n                false\n            }\n        };\n\n        if not_modified {\n            to_304(resp);\n        }\n        let header_only = not_modified || req.method == http::method::Method::HEAD;\n        if header_only && use_cache.is_on() {\n            // tell cache to stop serving downstream after yielding header\n            // (misses will continue to allow admitting upstream into cache)\n            use_cache.enable_header_only();\n        }\n    }\n\n    // TODO: cache upstream header filter to add/remove headers\n\n    pub(crate) async fn cache_http_task(\n        &self,\n        session: &mut Session,\n        task: &HttpTask,\n        ctx: &mut SV::CTX,\n        serve_from_cache: &mut ServeFromCache,\n    ) -> Result<()>\n    where\n        SV: ProxyHttp + Send + Sync,\n        SV::CTX: Send + Sync,\n    {\n        if !session.cache.enabled() && !session.cache.bypassing() {\n            return Ok(());\n        }\n\n        match task {\n            HttpTask::Header(header, end_stream) => {\n                // decide if cacheable and create cache meta\n                // for now, skip 1xxs (should not affect response cache decisions)\n                // However 101 is an exception because it is the final response header\n                if header.status.is_informational()\n                    && header.status != StatusCode::SWITCHING_PROTOCOLS\n                {\n                    return Ok(());\n                }\n                match self.inner.response_cache_filter(session, header, ctx)? {\n                    Cacheable(meta) => {\n                        let mut fill_cache = true;\n                        if session.cache.bypassing() {\n                            // The cache might have been bypassed because the response exceeded the\n                            // maximum cacheable asset size. If that looks like the case (there\n                            // is a maximum file size configured and we don't know the content\n                            // length up front), attempting to re-enable the cache now would cause\n                            // the request to fail when the chunked response exceeds the maximum\n                            // file size again.\n                            if session.cache.max_file_size_bytes().is_some()\n                                && !meta.headers().contains_key(header::CONTENT_LENGTH)\n                            {\n                                session\n                                    .cache\n                                    .disable(NoCacheReason::PredictedResponseTooLarge);\n                                return Ok(());\n                            }\n\n                            session.cache.response_became_cacheable();\n\n                            if session.req_header().method == Method::GET\n                                && meta.response_header().status == StatusCode::OK\n                            {\n                                self.inner.cache_miss(session, ctx);\n                                if !session.cache.enabled() {\n                                    fill_cache = false;\n                                }\n                            } else {\n                                // we've allowed caching on the next request,\n                                // but do not cache _this_ request if bypassed and not 200\n                                // (We didn't run upstream request cache filters to strip range or condition headers,\n                                // so this could be an uncacheable response e.g. 206 or 304 or HEAD.\n                                // Exclude all non-200/GET for simplicity, may expand allowable codes in the future.)\n                                fill_cache = false;\n                                session.cache.disable(NoCacheReason::Deferred);\n                            }\n                        }\n\n                        // If the Content-Length is known, and a maximum asset size has been configured\n                        // on the cache, validate that the response does not exceed the maximum asset size.\n                        if session.cache.enabled() {\n                            if let Some(max_file_size) = session.cache.max_file_size_bytes() {\n                                let content_length_hdr = meta.headers().get(header::CONTENT_LENGTH);\n                                if let Some(content_length) =\n                                    header_value_content_length(content_length_hdr)\n                                {\n                                    if content_length > max_file_size {\n                                        fill_cache = false;\n                                        session.cache.response_became_uncacheable(\n                                            NoCacheReason::ResponseTooLarge,\n                                        );\n                                        session.cache.disable(NoCacheReason::ResponseTooLarge);\n                                        // too large to cache, disable ranging\n                                        session.ignore_downstream_range = true;\n                                    }\n                                }\n                                // if the content-length header is not specified, the miss handler\n                                // will count the response size on the fly, aborting the request\n                                // mid-transfer if the max file size is exceeded\n                            }\n                        }\n                        if fill_cache {\n                            let req_header = session.req_header();\n                            // Update the variance in the meta via the same callback,\n                            // cache_vary_filter(), used in cache lookup for consistency.\n                            // Future cache lookups need a matching variance in the meta\n                            // with the cache key to pick up the correct variance\n                            let variance = self.inner.cache_vary_filter(&meta, ctx, req_header);\n                            session.cache.set_cache_meta(meta);\n                            session.cache.update_variance(variance);\n                            // this sends the meta and header\n                            session.cache.set_miss_handler().await?;\n                            if session.cache.miss_body_reader().is_some() {\n                                serve_from_cache.enable_miss();\n                            }\n                            if *end_stream {\n                                session\n                                    .cache\n                                    .miss_handler()\n                                    .unwrap() // safe, it is set above\n                                    .write_body(Bytes::new(), true)\n                                    .await?;\n                                session.cache.finish_miss_handler().await?;\n                            }\n                        }\n                    }\n                    Uncacheable(reason) => {\n                        if !session.cache.bypassing() {\n                            // mark as uncacheable, so we bypass cache next time\n                            session.cache.response_became_uncacheable(reason);\n                        }\n                        session.cache.disable(reason);\n                    }\n                }\n            }\n            HttpTask::Body(data, end_stream) | HttpTask::UpgradedBody(data, end_stream) => {\n                // It is not normally advisable to cache upgraded responses\n                // e.g. they are essentially close-delimited, so they are easily truncated\n                // but the framework still allows for it\n                match data {\n                    Some(d) => {\n                        if session.cache.enabled() {\n                            // TODO: do this async\n                            // fail if writing the body would exceed the max_file_size_bytes\n                            let body_size_allowed =\n                                session.cache.track_body_bytes_for_max_file_size(d.len());\n                            if !body_size_allowed {\n                                debug!(\"chunked response exceeded max cache size, remembering that it is uncacheable\");\n                                session\n                                    .cache\n                                    .response_became_uncacheable(NoCacheReason::ResponseTooLarge);\n\n                                return Error::e_explain(\n                                    ERR_RESPONSE_TOO_LARGE,\n                                    format!(\n                                        \"writing data of size {} bytes would exceed max file size of {} bytes\",\n                                        d.len(),\n                                        session.cache.max_file_size_bytes().expect(\"max file size bytes must be set to exceed size\")\n                                    ),\n                                );\n                            }\n\n                            // this will panic if more data is sent after we see end_stream\n                            // but should be impossible in real world\n                            let miss_handler = session.cache.miss_handler().unwrap();\n\n                            miss_handler.write_body(d.clone(), *end_stream).await?;\n                            if *end_stream {\n                                session.cache.finish_miss_handler().await?;\n                            }\n                        }\n                    }\n                    None => {\n                        if session.cache.enabled() && *end_stream {\n                            session.cache.finish_miss_handler().await?;\n                        }\n                    }\n                }\n            }\n            HttpTask::Trailer(_) => {} // h1 trailer is not supported yet\n            HttpTask::Done => {\n                if session.cache.enabled() {\n                    session.cache.finish_miss_handler().await?;\n                }\n            }\n            HttpTask::Failed(_) => {\n                // TODO: handle this failure: delete the temp files?\n            }\n        }\n        Ok(())\n    }\n\n    // Decide if local cache can be used according to upstream http header\n    // 1. when upstream returns 304, the local cache is refreshed and served fresh\n    // 2. when upstream returns certain HTTP error status, the local cache is served stale\n    // Return true if local cache should be used, false otherwise\n    pub(crate) async fn revalidate_or_stale(\n        &self,\n        session: &mut Session,\n        task: &mut HttpTask,\n        ctx: &mut SV::CTX,\n    ) -> bool\n    where\n        SV: ProxyHttp + Send + Sync,\n        SV::CTX: Send + Sync,\n    {\n        if !session.cache.enabled() {\n            return false;\n        }\n\n        match task {\n            HttpTask::Header(resp, _eos) => {\n                if resp.status == StatusCode::NOT_MODIFIED {\n                    if session.cache.maybe_cache_meta().is_some() {\n                        // run upstream response filters on upstream 304 first\n                        if let Err(err) = self\n                            .inner\n                            .upstream_response_filter(session, resp, ctx)\n                            .await\n                        {\n                            error!(\"upstream response filter error on 304: {err:?}\");\n                            session.cache.revalidate_uncacheable(\n                                *resp.clone(),\n                                NoCacheReason::InternalError,\n                            );\n                            // always serve from cache after receiving the 304\n                            return true;\n                        }\n                        // 304 doesn't contain all the headers, merge 304 into cached 200 header\n                        // in order for response_cache_filter to run correctly\n                        let merged_header = session.cache.revalidate_merge_header(resp);\n                        match self\n                            .inner\n                            .response_cache_filter(session, &merged_header, ctx)\n                        {\n                            Ok(Cacheable(mut meta)) => {\n                                // For simplicity, ignore changes to variance over 304 for now.\n                                // Note this means upstream can only update variance via 2xx\n                                // (expired response).\n                                //\n                                // TODO: if we choose to respect changing Vary / variance over 304,\n                                // then there are a few cases to consider. See `update_variance` in\n                                // the `pingora-cache` module.\n                                let old_meta = session.cache.maybe_cache_meta().unwrap(); // safe, checked above\n                                if let Some(old_variance) = old_meta.variance() {\n                                    meta.set_variance(old_variance);\n                                }\n                                if let Err(e) = session.cache.revalidate_cache_meta(meta).await {\n                                    // Fail open: we can continue use the revalidated response even\n                                    // if the meta failed to write to storage\n                                    warn!(\"revalidate_cache_meta failed {e:?}\");\n                                }\n                            }\n                            Ok(Uncacheable(reason)) => {\n                                // This response was once cacheable, and upstream tells us it has not changed\n                                // but now we decided it is uncacheable!\n                                // RFC 9111: still allowed to reuse stored response this time because\n                                // it was \"successfully validated\"\n                                // https://www.rfc-editor.org/rfc/rfc9111#constructing.responses.from.caches\n                                // Serve the response, but do not update cache\n\n                                // We also want to avoid poisoning downstream's cache with an unsolicited 304\n                                // if we did not receive a conditional request from downstream\n                                // (downstream may have a different cacheability assessment and could cache the 304)\n\n                                //TODO: log more\n                                debug!(\"Uncacheable {reason:?} 304 received\");\n                                session.cache.response_became_uncacheable(reason);\n                                session.cache.revalidate_uncacheable(merged_header, reason);\n                            }\n                            Err(e) => {\n                                // Error during revalidation, similarly to the reasons above\n                                // (avoid poisoning downstream cache with passthrough 304),\n                                // allow serving the stored response without updating cache\n                                warn!(\"Error {e:?} response_cache_filter during revalidation\");\n                                session.cache.revalidate_uncacheable(\n                                    merged_header,\n                                    NoCacheReason::InternalError,\n                                );\n                                // Assume the next 304 may succeed, so don't mark uncacheable\n                            }\n                        }\n                        // always serve from cache after receiving the 304\n                        true\n                    } else {\n                        //TODO: log more\n                        warn!(\"304 received without cached asset, disable caching\");\n                        let reason = NoCacheReason::Custom(\"304 on miss\");\n                        session.cache.response_became_uncacheable(reason);\n                        session.cache.disable(reason);\n                        false\n                    }\n                } else if resp.status.is_server_error() {\n                    // stale if error logic, 5xx only for now\n\n                    // this is response header filter, response_written should always be None?\n                    if !session.cache.can_serve_stale_error()\n                        || session.response_written().is_some()\n                    {\n                        return false;\n                    }\n\n                    // create an error to encode the http status code\n                    let http_status_error = Error::create(\n                        ErrorType::HTTPStatus(resp.status.as_u16()),\n                        ErrorSource::Upstream,\n                        None,\n                        None,\n                    );\n                    if self\n                        .inner\n                        .should_serve_stale(session, ctx, Some(&http_status_error))\n                    {\n                        // no more need to keep the write lock\n                        session\n                            .cache\n                            .release_write_lock(NoCacheReason::UpstreamError);\n                        true\n                    } else {\n                        false\n                    }\n                } else {\n                    false // not 304, not stale if error status code\n                }\n            }\n            _ => false, // not header\n        }\n    }\n\n    // None: no staled asset is used, Some(_): staled asset is sent to downstream\n    // bool: can the downstream connection be reused\n    pub(crate) async fn handle_stale_if_error(\n        &self,\n        session: &mut Session,\n        ctx: &mut SV::CTX,\n        error: &Error,\n    ) -> Option<(bool, Option<Box<Error>>)>\n    where\n        SV: ProxyHttp + Send + Sync,\n        SV::CTX: Send + Sync,\n    {\n        // the caller might already checked this as an optimization\n        if !session.cache.can_serve_stale_error() {\n            return None;\n        }\n\n        // the error happen halfway through a regular response to downstream\n        // can't resend the response\n        if session.response_written().is_some() {\n            return None;\n        }\n\n        // check error types\n        if !self.inner.should_serve_stale(session, ctx, Some(error)) {\n            return None;\n        }\n\n        // log the original error\n        warn!(\n            \"Fail to proxy: {}, serving stale, {}\",\n            error,\n            self.inner.request_summary(session, ctx)\n        );\n\n        // no more need to hang onto the cache lock\n        session\n            .cache\n            .release_write_lock(NoCacheReason::UpstreamError);\n\n        Some(self.proxy_cache_hit(session, ctx).await)\n    }\n\n    // helper function to check when to continue to retry lock (true) or give up (false)\n    fn handle_lock_status(\n        &self,\n        session: &mut Session,\n        ctx: &SV::CTX,\n        lock_status: LockStatus,\n    ) -> bool\n    where\n        SV: ProxyHttp,\n    {\n        debug!(\"cache unlocked {lock_status:?}\");\n        match lock_status {\n            // should lookup the cached asset again\n            LockStatus::Done => true,\n            // should compete to be a new writer\n            LockStatus::TransientError => true,\n            // the request is uncacheable, go ahead to fetch from the origin\n            LockStatus::GiveUp => {\n                // TODO: It will be nice for the writer to propagate the real reason\n                session.cache.disable(NoCacheReason::CacheLockGiveUp);\n                // not cacheable, just go to the origin.\n                false\n            }\n            // treat this the same as TransientError\n            LockStatus::Dangling => {\n                // software bug, but request can recover from this\n                warn!(\n                    \"Dangling cache lock, {}\",\n                    self.inner.request_summary(session, ctx)\n                );\n                true\n            }\n            // If this reader has spent too long waiting on locks, let the request\n            // through while disabling cache (to avoid amplifying disk writes).\n            LockStatus::WaitTimeout => {\n                warn!(\n                    \"Cache lock timeout, {}\",\n                    self.inner.request_summary(session, ctx)\n                );\n                session.cache.disable(NoCacheReason::CacheLockTimeout);\n                // not cacheable, just go to the origin.\n                false\n            }\n            // When a singular cache lock has been held for too long,\n            // we should allow requests to recompete for the lock\n            // to protect upstreams from load.\n            LockStatus::AgeTimeout => true,\n            // software bug, this status should be impossible to reach\n            LockStatus::Waiting => panic!(\"impossible LockStatus::Waiting\"),\n        }\n    }\n}\n\nfn cache_hit_header(cache: &HttpCache) -> Box<ResponseHeader> {\n    let mut header = Box::new(cache.cache_meta().response_header_copy());\n    // convert cache response\n\n    // these status codes / method cannot have body, so no need to add chunked encoding\n    let no_body = matches!(header.status.as_u16(), 204 | 304);\n\n    // https://www.rfc-editor.org/rfc/rfc9111#section-4:\n    // When a stored response is used to satisfy a request without validation, a cache\n    // MUST generate an Age header field\n    if !cache.upstream_used() {\n        let age = cache.cache_meta().age().as_secs();\n        header.insert_header(http::header::AGE, age).unwrap();\n    }\n    log::debug!(\"cache header: {header:?} {:?}\", cache.phase());\n\n    // currently storage cache is always considered an h1 upstream\n    // (header-serde serializes as h1.0 or h1.1)\n    // set this header to be h1.1\n    header.set_version(Version::HTTP_11);\n\n    /* Add chunked header to tell downstream to use chunked encoding\n     * during the absent of content-length in h2 */\n    if !no_body\n        && !header.status.is_informational()\n        && header.headers.get(http::header::CONTENT_LENGTH).is_none()\n    {\n        header\n            .insert_header(http::header::TRANSFER_ENCODING, \"chunked\")\n            .unwrap();\n    }\n    header\n}\n\n// https://datatracker.ietf.org/doc/html/rfc7233#section-3\npub mod range_filter {\n    use super::*;\n    use bytes::BytesMut;\n    use http::header::*;\n    use std::ops::Range;\n\n    // parse bytes into usize, ignores specific error\n    fn parse_number(input: &[u8]) -> Option<usize> {\n        str::from_utf8(input).ok()?.parse().ok()\n    }\n\n    fn parse_range_header(\n        range: &[u8],\n        content_length: usize,\n        max_multipart_ranges: Option<usize>,\n    ) -> RangeType {\n        use regex::Regex;\n\n        // Match individual range parts, (e.g. \"0-100\", \"-5\", \"1-\")\n        static RE_SINGLE_RANGE_PART: Lazy<Regex> =\n            Lazy::new(|| Regex::new(r\"(?i)^\\s*(?P<start>\\d*)-(?P<end>\\d*)\\s*$\").unwrap());\n\n        // Convert bytes to UTF-8 string\n        let range_str = match str::from_utf8(range) {\n            Ok(s) => s,\n            Err(_) => return RangeType::None,\n        };\n\n        // Split into \"bytes=\" and the actual range(s)\n        let mut parts = range_str.splitn(2, \"=\");\n\n        // Check if it starts with \"bytes=\"\n        let prefix = parts.next();\n        if !prefix.is_some_and(|s| s.eq_ignore_ascii_case(\"bytes\")) {\n            return RangeType::None;\n        }\n\n        let Some(ranges_str) = parts.next() else {\n            // No ranges provided\n            return RangeType::None;\n        };\n\n        // \"bytes=\" with an empty (or whitespace-only) range-set is syntactically a\n        // range request with zero satisfiable range-specs, so return 416.\n        if ranges_str.trim().is_empty() {\n            return RangeType::Invalid;\n        }\n\n        // Get the actual range string (e.g.\"100-200,300-400\")\n        let mut range_count = 0;\n        for _ in ranges_str.split(',') {\n            range_count += 1;\n            if let Some(max_ranges) = max_multipart_ranges {\n                if range_count >= max_ranges {\n                    // If we get more than max configured ranges, return None for now to save parsing time\n                    return RangeType::None;\n                }\n            }\n        }\n        let mut ranges: Vec<Range<usize>> = Vec::with_capacity(range_count);\n\n        // Process each range\n        let mut last_range_end = 0;\n        for part in ranges_str.split(',') {\n            let captured = match RE_SINGLE_RANGE_PART.captures(part) {\n                Some(c) => c,\n                None => {\n                    return RangeType::None;\n                }\n            };\n\n            let maybe_start = captured\n                .name(\"start\")\n                .and_then(|s| s.as_str().parse::<usize>().ok());\n            let end = captured\n                .name(\"end\")\n                .and_then(|s| s.as_str().parse::<usize>().ok());\n\n            let range = if let Some(start) = maybe_start {\n                if start >= content_length {\n                    // Skip the invalid range\n                    continue;\n                }\n                // open-ended range should end at the last byte\n                // over sized end is allowed but ignored\n                // range end is inclusive\n                let end = std::cmp::min(end.unwrap_or(content_length - 1), content_length - 1) + 1;\n                if end <= start {\n                    // Skip the invalid range\n                    continue;\n                }\n                start..end\n            } else {\n                // start is empty, this changes the meaning of the value of `end`\n                // Now it means to read the last `end` bytes\n                if let Some(end) = end {\n                    if content_length >= end {\n                        (content_length - end)..content_length\n                    } else {\n                        // over sized end is allowed but ignored\n                        0..content_length\n                    }\n                } else {\n                    // No start or end, skip the invalid range\n                    continue;\n                }\n            };\n            // For now we stick to non-overlapping, ascending ranges for simplicity\n            // and parity with nginx\n            if range.start < last_range_end {\n                return RangeType::None;\n            }\n            last_range_end = range.end;\n            ranges.push(range);\n        }\n\n        // Note for future: we can technically coalesce multiple ranges for multipart\n        //\n        // https://www.rfc-editor.org/rfc/rfc9110#section-17.15\n        // \"Servers ought to ignore, coalesce, or reject egregious range\n        // requests, such as requests for more than two overlapping ranges or\n        // for many small ranges in a single set, particularly when the ranges\n        // are requested out of order for no apparent reason. Multipart range\n        // requests are not designed to support random access.\"\n\n        if ranges.is_empty() {\n            // We got some ranges, processed them but none were valid\n            RangeType::Invalid\n        } else if ranges.len() == 1 {\n            RangeType::Single(ranges[0].clone()) // Only 1 index\n        } else {\n            RangeType::Multi(MultiRangeInfo::new(ranges))\n        }\n    }\n    #[test]\n    fn test_parse_range() {\n        assert_eq!(\n            parse_range_header(b\"bytes=0-1\", 10, None),\n            RangeType::new_single(0, 2)\n        );\n        assert_eq!(\n            parse_range_header(b\"bYTes=0-9\", 10, None),\n            RangeType::new_single(0, 10)\n        );\n        assert_eq!(\n            parse_range_header(b\"bytes=0-12\", 10, None),\n            RangeType::new_single(0, 10)\n        );\n        assert_eq!(\n            parse_range_header(b\"bytes=0-\", 10, None),\n            RangeType::new_single(0, 10)\n        );\n        assert_eq!(\n            parse_range_header(b\"bytes=2-1\", 10, None),\n            RangeType::Invalid\n        );\n        assert_eq!(\n            parse_range_header(b\"bytes=10-11\", 10, None),\n            RangeType::Invalid\n        );\n        assert_eq!(\n            parse_range_header(b\"bytes=-2\", 10, None),\n            RangeType::new_single(8, 10)\n        );\n        assert_eq!(\n            parse_range_header(b\"bytes=-12\", 10, None),\n            RangeType::new_single(0, 10)\n        );\n        assert_eq!(parse_range_header(b\"bytes=-\", 10, None), RangeType::Invalid);\n        assert_eq!(parse_range_header(b\"bytes=\", 10, None), RangeType::Invalid);\n        assert_eq!(\n            parse_range_header(b\"bytes=  \", 10, None),\n            RangeType::Invalid\n        );\n    }\n\n    // Add some tests for multi-range too\n    #[test]\n    fn test_parse_range_header_multi() {\n        assert_eq!(\n            parse_range_header(b\"bytes=0-1,4-5\", 10, None)\n                .get_multirange_info()\n                .expect(\"Should have multipart info for Multipart range request\")\n                .ranges,\n            (vec![Range { start: 0, end: 2 }, Range { start: 4, end: 6 }])\n        );\n        // Last range is invalid because the content-length is too small\n        assert_eq!(\n            parse_range_header(b\"bytEs=0-99,200-299,400-499\", 320, None)\n                .get_multirange_info()\n                .expect(\"Should have multipart info for Multipart range request\")\n                .ranges,\n            (vec![\n                Range { start: 0, end: 100 },\n                Range {\n                    start: 200,\n                    end: 300\n                }\n            ])\n        );\n        // Same as above but appropriate content length\n        assert_eq!(\n            parse_range_header(b\"bytEs=0-99,200-299,400-499\", 500, None)\n                .get_multirange_info()\n                .expect(\"Should have multipart info for Multipart range request\")\n                .ranges,\n            vec![\n                Range { start: 0, end: 100 },\n                Range {\n                    start: 200,\n                    end: 300\n                },\n                Range {\n                    start: 400,\n                    end: 500\n                },\n            ]\n        );\n        // Looks like a range request but it is continuous, we decline to range\n        assert_eq!(\n            parse_range_header(b\"bytes=0-,-2\", 10, None),\n            RangeType::None,\n        );\n        // Should not have multirange info set\n        assert!(parse_range_header(b\"bytes=0-,-2\", 10, None)\n            .get_multirange_info()\n            .is_none());\n        // Overlapping ranges, these ranges are currently declined\n        assert_eq!(\n            parse_range_header(b\"bytes=0-3,2-5\", 10, None),\n            RangeType::None,\n        );\n        assert!(parse_range_header(b\"bytes=0-3,2-5\", 10, None)\n            .get_multirange_info()\n            .is_none());\n\n        // Content length is 2, so only range is 0-2.\n        assert_eq!(\n            parse_range_header(b\"bytes=0-5,10-\", 2, None),\n            RangeType::new_single(0, 2)\n        );\n        assert!(parse_range_header(b\"bytes=0-5,10-\", 2, None)\n            .get_multirange_info()\n            .is_none());\n\n        // We should ignore the last incorrect range and return the other acceptable ranges\n        assert_eq!(\n            parse_range_header(b\"bytes=0-5, 10-20, 30-18\", 200, None)\n                .get_multirange_info()\n                .expect(\"Should have multipart info for Multipart range request\")\n                .ranges,\n            vec![Range { start: 0, end: 6 }, Range { start: 10, end: 21 },]\n        );\n        // All invalid ranges\n        assert_eq!(\n            parse_range_header(b\"bytes=5-0, 20-15, 30-25\", 200, None),\n            RangeType::Invalid\n        );\n\n        // Helper function to generate a large number of ranges for the next test\n        fn generate_range_header(count: usize) -> Vec<u8> {\n            let mut s = String::from(\"bytes=\");\n            for i in 0..count {\n                let start = i * 4;\n                let end = start + 1;\n                if i > 0 {\n                    s.push(',');\n                }\n                s.push_str(&start.to_string());\n                s.push('-');\n                s.push_str(&end.to_string());\n            }\n            s.into_bytes()\n        }\n\n        // Test 200 range limit for parsing.\n        let ranges = generate_range_header(201);\n        assert_eq!(\n            parse_range_header(&ranges, 1000, Some(200)),\n            RangeType::None\n        )\n    }\n\n    // For Multipart Requests, we need to know the boundary, content length and type across\n    // the headers and the body. So let us store this information as part of the range\n    #[derive(Debug, Eq, PartialEq, Clone)]\n    pub struct MultiRangeInfo {\n        pub ranges: Vec<Range<usize>>,\n        pub boundary: String,\n        total_length: usize,\n        content_type: Option<String>,\n    }\n\n    impl MultiRangeInfo {\n        // Create a new MultiRangeInfo, when we just have the ranges\n        pub fn new(ranges: Vec<Range<usize>>) -> Self {\n            Self {\n                ranges,\n                // Directly create boundary string on initialization\n                boundary: Self::generate_boundary(),\n                total_length: 0,\n                content_type: None,\n            }\n        }\n        pub fn set_content_type(&mut self, content_type: String) {\n            self.content_type = Some(content_type)\n        }\n        pub fn set_total_length(&mut self, total_length: usize) {\n            self.total_length = total_length;\n        }\n        // Per [RFC 9110](https://www.rfc-editor.org/rfc/rfc9110.html#multipart.byteranges),\n        // we need generate a boundary string for each body part.\n        // Per [RFC 2046](https://www.rfc-editor.org/rfc/rfc2046#section-5.1.1), the boundary should be no longer than 70 characters\n        // and it must not match the body content.\n        fn generate_boundary() -> String {\n            use rand::Rng;\n            let mut rng: rand::prelude::ThreadRng = rand::thread_rng();\n            format!(\"{:016x}\", rng.gen::<u64>())\n        }\n        pub fn calculate_multipart_length(&self) -> usize {\n            let mut total_length = 0;\n            let content_type = self.content_type.as_ref();\n            for range in self.ranges.clone() {\n                // Each part should have\n                // \\r\\n--boundary\\r\\n                         --> 4 + boundary.len() (16) + 2 = 20\n                // Content-Type: original-content-type\\r\\n    --> 14 + content_type.len() + 2\n                // Content-Range: bytes start-end/total\\r\\n   --> Variable +2\n                // \\r\\n                                       --> 2\n                // [data]                                     --> data.len()\n                total_length += 4 + self.boundary.len() + 2;\n                total_length += content_type.map_or(0, |ct| 14 + ct.len() + 2);\n                total_length += format!(\n                    \"Content-Range: bytes {}-{}/{}\",\n                    range.start,\n                    range.end - 1,\n                    self.total_length\n                )\n                .len()\n                    + 2;\n                total_length += 2;\n                total_length += range.end - range.start;\n            }\n            // Final boundary: \"\\r\\n--<boundary>--\\r\\n\"\n            total_length += 4 + self.boundary.len() + 4;\n            total_length\n        }\n    }\n    #[derive(Debug, Eq, PartialEq, Clone)]\n    pub enum RangeType {\n        None,\n        Single(Range<usize>),\n        Multi(MultiRangeInfo),\n        Invalid,\n    }\n\n    impl RangeType {\n        // Helper functions for tests\n        #[allow(dead_code)]\n        fn new_single(start: usize, end: usize) -> Self {\n            RangeType::Single(Range { start, end })\n        }\n        #[allow(dead_code)]\n        pub fn new_multi(ranges: Vec<Range<usize>>) -> Self {\n            RangeType::Multi(MultiRangeInfo::new(ranges))\n        }\n        #[allow(dead_code)]\n        fn get_multirange_info(&self) -> Option<&MultiRangeInfo> {\n            match self {\n                RangeType::Multi(multi_range_info) => Some(multi_range_info),\n                _ => None,\n            }\n        }\n        #[allow(dead_code)]\n        fn update_multirange_info(&mut self, content_length: usize, content_type: Option<String>) {\n            if let RangeType::Multi(multipart_range_info) = self {\n                multipart_range_info.content_type = content_type;\n                multipart_range_info.set_total_length(content_length);\n            }\n        }\n    }\n\n    // Handles both single-range and multipart-range requests\n    pub fn range_header_filter(\n        req: &RequestHeader,\n        resp: &mut ResponseHeader,\n        max_multipart_ranges: Option<usize>,\n    ) -> RangeType {\n        // The Range header field is evaluated after evaluating the precondition\n        // header fields defined in [RFC7232], and only if the result in absence\n        // of the Range header field would be a 200 (OK) response\n        if resp.status != StatusCode::OK {\n            return RangeType::None;\n        }\n\n        // Content-Length is not required by RFC but it is what nginx does and easier to implement\n        // with this header present.\n        let Some(content_length_bytes) = resp.headers.get(CONTENT_LENGTH) else {\n            return RangeType::None;\n        };\n        // bail on invalid content length\n        let Some(content_length) = parse_number(content_length_bytes.as_bytes()) else {\n            return RangeType::None;\n        };\n\n        // At this point the response is allowed to be served as ranges\n        // TODO: we can also check Accept-Range header from resp. Nginx gives uses the option\n        // see proxy_force_ranges\n\n        fn request_range_type(\n            req: &RequestHeader,\n            resp: &ResponseHeader,\n            content_length: usize,\n            max_multipart_ranges: Option<usize>,\n        ) -> RangeType {\n            // \"A server MUST ignore a Range header field received with a request method other than GET.\"\n            if req.method != http::Method::GET && req.method != http::Method::HEAD {\n                return RangeType::None;\n            }\n\n            let Some(range_header) = req.headers.get(RANGE) else {\n                return RangeType::None;\n            };\n\n            // if-range wants to understand if the Last-Modified / ETag value matches exactly for use\n            // with resumable downloads.\n            // https://datatracker.ietf.org/doc/html/rfc9110#name-if-range\n            // Note that the RFC wants strong validation, and suggests that\n            // \"A valid entity-tag can be distinguished from a valid HTTP-date\n            // by examining the first three characters for a DQUOTE,\"\n            // but this current etag matching behavior most closely mirrors nginx.\n            if let Some(if_range) = req.headers.get(IF_RANGE) {\n                let ir = if_range.as_bytes();\n                let matches = if ir.len() >= 2 && ir.last() == Some(&b'\"') {\n                    resp.headers.get(ETAG).is_some_and(|etag| etag == if_range)\n                } else if let Some(last_modified) = resp.headers.get(LAST_MODIFIED) {\n                    last_modified == if_range\n                } else {\n                    false\n                };\n                if !matches {\n                    return RangeType::None;\n                }\n            }\n\n            parse_range_header(\n                range_header.as_bytes(),\n                content_length,\n                max_multipart_ranges,\n            )\n        }\n\n        let mut range_type = request_range_type(req, resp, content_length, max_multipart_ranges);\n\n        match &mut range_type {\n            RangeType::None => {\n                // At this point, the response is _eligible_ to be served in ranges\n                // in the future, so add Accept-Ranges, mirroring nginx behavior\n                resp.insert_header(&ACCEPT_RANGES, \"bytes\").unwrap();\n            }\n            RangeType::Single(r) => {\n                // 206 response\n                resp.set_status(StatusCode::PARTIAL_CONTENT).unwrap();\n                resp.remove_header(&ACCEPT_RANGES);\n                resp.insert_header(&CONTENT_LENGTH, r.end - r.start)\n                    .unwrap();\n                resp.insert_header(\n                    &CONTENT_RANGE,\n                    format!(\"bytes {}-{}/{content_length}\", r.start, r.end - 1), // range end is inclusive\n                )\n                .unwrap()\n            }\n\n            RangeType::Multi(multi_range_info) => {\n                let content_type = resp\n                    .headers\n                    .get(CONTENT_TYPE)\n                    .and_then(|v| v.to_str().ok())\n                    .unwrap_or(\"application/octet-stream\");\n                // Update multipart info\n                multi_range_info.set_total_length(content_length);\n                multi_range_info.set_content_type(content_type.to_string());\n\n                let total_length = multi_range_info.calculate_multipart_length();\n\n                resp.set_status(StatusCode::PARTIAL_CONTENT).unwrap();\n                resp.remove_header(&ACCEPT_RANGES);\n                resp.insert_header(CONTENT_LENGTH, total_length).unwrap();\n                resp.insert_header(\n                    CONTENT_TYPE,\n                    format!(\n                        \"multipart/byteranges; boundary={}\",\n                        multi_range_info.boundary\n                    ), // RFC 2046\n                )\n                .unwrap();\n                resp.remove_header(&CONTENT_RANGE);\n            }\n            RangeType::Invalid => {\n                // 416 response\n                resp.set_status(StatusCode::RANGE_NOT_SATISFIABLE).unwrap();\n                // empty body for simplicity\n                resp.insert_header(&CONTENT_LENGTH, HeaderValue::from_static(\"0\"))\n                    .unwrap();\n                resp.remove_header(&ACCEPT_RANGES);\n                resp.remove_header(&CONTENT_TYPE);\n                resp.remove_header(&CONTENT_ENCODING);\n                resp.remove_header(&TRANSFER_ENCODING);\n                resp.insert_header(&CONTENT_RANGE, format!(\"bytes */{content_length}\"))\n                    .unwrap()\n            }\n        }\n\n        range_type\n    }\n\n    #[test]\n    fn test_range_filter_single() {\n        fn gen_req() -> RequestHeader {\n            RequestHeader::build(http::Method::GET, b\"/\", Some(1)).unwrap()\n        }\n        fn gen_resp() -> ResponseHeader {\n            let mut resp = ResponseHeader::build(200, Some(1)).unwrap();\n            resp.append_header(\"Content-Length\", \"10\").unwrap();\n            resp\n        }\n\n        // no range\n        let req = gen_req();\n        let mut resp = gen_resp();\n        assert_eq!(RangeType::None, range_header_filter(&req, &mut resp, None));\n        assert_eq!(resp.status.as_u16(), 200);\n        assert_eq!(\n            resp.headers.get(\"accept-ranges\").unwrap().as_bytes(),\n            b\"bytes\"\n        );\n\n        // no range, try HEAD\n        let mut req = gen_req();\n        req.method = Method::HEAD;\n        let mut resp = gen_resp();\n        assert_eq!(RangeType::None, range_header_filter(&req, &mut resp, None));\n        assert_eq!(resp.status.as_u16(), 200);\n        assert_eq!(\n            resp.headers.get(\"accept-ranges\").unwrap().as_bytes(),\n            b\"bytes\"\n        );\n\n        // regular range\n        let mut req = gen_req();\n        req.insert_header(\"Range\", \"bytes=0-1\").unwrap();\n        let mut resp = gen_resp();\n        assert_eq!(\n            RangeType::new_single(0, 2),\n            range_header_filter(&req, &mut resp, None)\n        );\n        assert_eq!(resp.status.as_u16(), 206);\n        assert_eq!(resp.headers.get(\"content-length\").unwrap().as_bytes(), b\"2\");\n        assert_eq!(\n            resp.headers.get(\"content-range\").unwrap().as_bytes(),\n            b\"bytes 0-1/10\"\n        );\n        assert!(resp.headers.get(\"accept-ranges\").is_none());\n\n        // regular range, accept-ranges included\n        let mut req = gen_req();\n        req.insert_header(\"Range\", \"bytes=0-1\").unwrap();\n        let mut resp = gen_resp();\n        resp.insert_header(\"Accept-Ranges\", \"bytes\").unwrap();\n        assert_eq!(\n            RangeType::new_single(0, 2),\n            range_header_filter(&req, &mut resp, None)\n        );\n        assert_eq!(resp.status.as_u16(), 206);\n        assert_eq!(resp.headers.get(\"content-length\").unwrap().as_bytes(), b\"2\");\n        assert_eq!(\n            resp.headers.get(\"content-range\").unwrap().as_bytes(),\n            b\"bytes 0-1/10\"\n        );\n        // accept-ranges stripped\n        assert!(resp.headers.get(\"accept-ranges\").is_none());\n\n        // bad range\n        let mut req = gen_req();\n        req.insert_header(\"Range\", \"bytes=1-0\").unwrap();\n        let mut resp = gen_resp();\n        resp.insert_header(\"Accept-Ranges\", \"bytes\").unwrap();\n        resp.insert_header(\"Content-Encoding\", \"gzip\").unwrap();\n        resp.insert_header(\"Transfer-Encoding\", \"chunked\").unwrap();\n        assert_eq!(\n            RangeType::Invalid,\n            range_header_filter(&req, &mut resp, None)\n        );\n        assert_eq!(resp.status.as_u16(), 416);\n        assert_eq!(resp.headers.get(\"content-length\").unwrap().as_bytes(), b\"0\");\n        assert_eq!(\n            resp.headers.get(\"content-range\").unwrap().as_bytes(),\n            b\"bytes */10\"\n        );\n        assert!(resp.headers.get(\"accept-ranges\").is_none());\n        assert!(resp.headers.get(\"content-encoding\").is_none());\n        assert!(resp.headers.get(\"transfer-encoding\").is_none());\n    }\n\n    // Multipart Tests\n    #[test]\n    fn test_range_filter_multipart() {\n        fn gen_req() -> RequestHeader {\n            let mut req: RequestHeader =\n                RequestHeader::build(http::Method::GET, b\"/\", Some(1)).unwrap();\n            req.append_header(\"Range\", \"bytes=0-1,3-4,6-7\").unwrap();\n            req\n        }\n        fn gen_req_overlap_range() -> RequestHeader {\n            let mut req: RequestHeader =\n                RequestHeader::build(http::Method::GET, b\"/\", Some(1)).unwrap();\n            req.append_header(\"Range\", \"bytes=0-3,2-5,7-8\").unwrap();\n            req\n        }\n        fn gen_resp() -> ResponseHeader {\n            let mut resp = ResponseHeader::build(200, Some(1)).unwrap();\n            resp.append_header(\"Content-Length\", \"10\").unwrap();\n            resp\n        }\n\n        // valid multipart range\n        let req = gen_req();\n        let mut resp = gen_resp();\n        let result = range_header_filter(&req, &mut resp, None);\n        let mut boundary_str = String::new();\n\n        assert!(matches!(result, RangeType::Multi(_)));\n        if let RangeType::Multi(multi_part_info) = result {\n            assert_eq!(multi_part_info.ranges.len(), 3);\n            assert_eq!(multi_part_info.ranges[0], Range { start: 0, end: 2 });\n            assert_eq!(multi_part_info.ranges[1], Range { start: 3, end: 5 });\n            assert_eq!(multi_part_info.ranges[2], Range { start: 6, end: 8 });\n            // Verify that multipart info has been set\n            assert!(multi_part_info.content_type.is_some());\n            assert_eq!(multi_part_info.total_length, 10);\n            assert!(!multi_part_info.boundary.is_empty());\n            boundary_str = multi_part_info.boundary;\n        }\n        assert_eq!(resp.status.as_u16(), 206);\n        // Verify that boundary is the same in header and in multipartinfo\n        assert_eq!(\n            resp.headers.get(\"content-type\").unwrap().to_str().unwrap(),\n            format!(\"multipart/byteranges; boundary={boundary_str}\")\n        );\n        assert!(resp.headers.get(\"content_length\").is_none());\n        assert!(resp.headers.get(\"accept-ranges\").is_none());\n\n        // overlapping range, multipart range is declined\n        let req = gen_req_overlap_range();\n        let mut resp = gen_resp();\n        let result = range_header_filter(&req, &mut resp, None);\n\n        assert!(matches!(result, RangeType::None));\n        assert_eq!(resp.status.as_u16(), 200);\n        assert!(resp.headers.get(\"content-type\").is_none());\n        assert_eq!(\n            resp.headers.get(\"accept-ranges\").unwrap().as_bytes(),\n            b\"bytes\"\n        );\n\n        // bad multipart range\n        let mut req = gen_req();\n        req.insert_header(\"Range\", \"bytes=1-0, 12-9, 50-40\")\n            .unwrap();\n        let mut resp = gen_resp();\n        resp.insert_header(\"Content-Encoding\", \"br\").unwrap();\n        resp.insert_header(\"Transfer-Encoding\", \"chunked\").unwrap();\n        let result = range_header_filter(&req, &mut resp, None);\n        assert!(matches!(result, RangeType::Invalid));\n        assert_eq!(resp.status.as_u16(), 416);\n        assert!(resp.headers.get(\"accept-ranges\").is_none());\n        assert!(resp.headers.get(\"content-encoding\").is_none());\n        assert!(resp.headers.get(\"transfer-encoding\").is_none());\n    }\n\n    #[test]\n    fn test_if_range() {\n        const DATE: &str = \"Fri, 07 Jul 2023 22:03:29 GMT\";\n        const ETAG: &str = \"\\\"1234\\\"\";\n\n        fn gen_req() -> RequestHeader {\n            let mut req = RequestHeader::build(http::Method::GET, b\"/\", Some(1)).unwrap();\n            req.append_header(\"Range\", \"bytes=0-1\").unwrap();\n            req\n        }\n        fn get_multipart_req() -> RequestHeader {\n            let mut req = RequestHeader::build(http::Method::GET, b\"/\", Some(1)).unwrap();\n            _ = req.append_header(\"Range\", \"bytes=0-1,3-4,6-7\");\n            req\n        }\n        fn gen_resp() -> ResponseHeader {\n            let mut resp = ResponseHeader::build(200, Some(1)).unwrap();\n            resp.append_header(\"Content-Length\", \"10\").unwrap();\n            resp.append_header(\"Last-Modified\", DATE).unwrap();\n            resp.append_header(\"ETag\", ETAG).unwrap();\n            resp\n        }\n\n        // matching Last-Modified date\n        let mut req = gen_req();\n        req.insert_header(\"If-Range\", DATE).unwrap();\n        let mut resp = gen_resp();\n        assert_eq!(\n            RangeType::new_single(0, 2),\n            range_header_filter(&req, &mut resp, None)\n        );\n\n        // non-matching date\n        let mut req = gen_req();\n        req.insert_header(\"If-Range\", \"Fri, 07 Jul 2023 22:03:25 GMT\")\n            .unwrap();\n        let mut resp = gen_resp();\n        assert_eq!(RangeType::None, range_header_filter(&req, &mut resp, None));\n        assert_eq!(resp.status.as_u16(), 200);\n        assert_eq!(\n            resp.headers.get(\"accept-ranges\").unwrap().as_bytes(),\n            b\"bytes\"\n        );\n\n        // match ETag\n        let mut req = gen_req();\n        req.insert_header(\"If-Range\", ETAG).unwrap();\n        let mut resp = gen_resp();\n        assert_eq!(\n            RangeType::new_single(0, 2),\n            range_header_filter(&req, &mut resp, None)\n        );\n        assert_eq!(resp.status.as_u16(), 206);\n        assert!(resp.headers.get(\"accept-ranges\").is_none());\n\n        // non-matching ETags do not result in range\n        let mut req = gen_req();\n        req.insert_header(\"If-Range\", \"\\\"4567\\\"\").unwrap();\n        let mut resp = gen_resp();\n        assert_eq!(RangeType::None, range_header_filter(&req, &mut resp, None));\n        assert_eq!(resp.status.as_u16(), 200);\n        assert_eq!(\n            resp.headers.get(\"accept-ranges\").unwrap().as_bytes(),\n            b\"bytes\"\n        );\n\n        let mut req = gen_req();\n        req.insert_header(\"If-Range\", \"1234\").unwrap();\n        let mut resp = gen_resp();\n        assert_eq!(RangeType::None, range_header_filter(&req, &mut resp, None));\n        assert_eq!(resp.status.as_u16(), 200);\n        assert_eq!(\n            resp.headers.get(\"accept-ranges\").unwrap().as_bytes(),\n            b\"bytes\"\n        );\n\n        // multipart range with If-Range\n        let mut req = get_multipart_req();\n        req.insert_header(\"If-Range\", DATE).unwrap();\n        let mut resp = gen_resp();\n        let result = range_header_filter(&req, &mut resp, None);\n        assert!(matches!(result, RangeType::Multi(_)));\n        assert_eq!(resp.status.as_u16(), 206);\n        assert!(resp.headers.get(\"accept-ranges\").is_none());\n\n        // multipart with matching ETag\n        let req = get_multipart_req();\n        let mut resp = gen_resp();\n        assert!(matches!(\n            range_header_filter(&req, &mut resp, None),\n            RangeType::Multi(_)\n        ));\n\n        // multipart with non-matching If-Range\n        let mut req = get_multipart_req();\n        req.insert_header(\"If-Range\", \"\\\"wrong\\\"\").unwrap();\n        let mut resp = gen_resp();\n        assert_eq!(RangeType::None, range_header_filter(&req, &mut resp, None));\n        assert_eq!(resp.status.as_u16(), 200);\n        assert_eq!(\n            resp.headers.get(\"accept-ranges\").unwrap().as_bytes(),\n            b\"bytes\"\n        );\n    }\n\n    pub struct RangeBodyFilter {\n        pub range: RangeType,\n        current: usize,\n        multipart_idx: Option<usize>,\n        cache_multipart_idx: Option<usize>,\n    }\n\n    impl Default for RangeBodyFilter {\n        fn default() -> Self {\n            Self::new()\n        }\n    }\n\n    impl RangeBodyFilter {\n        pub fn new() -> Self {\n            RangeBodyFilter {\n                range: RangeType::None,\n                current: 0,\n                multipart_idx: None,\n                cache_multipart_idx: None,\n            }\n        }\n\n        pub fn new_range(range: RangeType) -> Self {\n            RangeBodyFilter {\n                multipart_idx: matches!(range, RangeType::Multi(_)).then_some(0),\n                range,\n                ..Default::default()\n            }\n        }\n\n        pub fn is_multipart_range(&self) -> bool {\n            matches!(self.range, RangeType::Multi(_))\n        }\n\n        /// Whether we should expect the cache body reader to seek again\n        /// for a different range.\n        pub fn should_cache_seek_again(&self) -> bool {\n            match &self.range {\n                RangeType::Multi(multipart_info) => self\n                    .cache_multipart_idx\n                    .is_some_and(|idx| idx != multipart_info.ranges.len() - 1),\n                _ => false,\n            }\n        }\n\n        /// Returns the next multipart range to seek for the cache body reader.\n        pub fn next_cache_multipart_range(&mut self) -> Range<usize> {\n            match &self.range {\n                RangeType::Multi(multipart_info) => {\n                    match self.cache_multipart_idx.as_mut() {\n                        Some(v) => *v += 1,\n                        None => self.cache_multipart_idx = Some(0),\n                    }\n                    let cache_multipart_idx = self.cache_multipart_idx.expect(\"set above\");\n                    let multipart_idx = self.multipart_idx.expect(\"must be set on multirange\");\n                    // NOTE: currently this assumes once we start seeking multipart from the hit\n                    // handler, it will continue to return can_seek_multipart true.\n                    assert_eq!(multipart_idx, cache_multipart_idx,\n                        \"cache multipart idx should match multipart idx, or there is a hit handler bug\");\n                    multipart_info.ranges[cache_multipart_idx].clone()\n                }\n                _ => panic!(\"tried to advance multipart idx on non-multipart range\"),\n            }\n        }\n\n        pub fn set_current_cursor(&mut self, current: usize) {\n            self.current = current;\n        }\n\n        pub fn set(&mut self, range: RangeType) {\n            self.multipart_idx = matches!(range, RangeType::Multi(_)).then_some(0);\n            self.range = range;\n        }\n\n        // Emit final boundary footer for multipart requests\n        pub fn finalize(&self, boundary: &String) -> Option<Bytes> {\n            if let RangeType::Multi(_) = self.range {\n                Some(Bytes::from(format!(\"\\r\\n--{boundary}--\\r\\n\")))\n            } else {\n                None\n            }\n        }\n\n        pub fn filter_body(&mut self, data: Option<Bytes>) -> Option<Bytes> {\n            match &self.range {\n                RangeType::None => data,\n                RangeType::Invalid => None,\n                RangeType::Single(r) => {\n                    let current = self.current;\n                    self.current += data.as_ref().map_or(0, |d| d.len());\n                    data.and_then(|d| Self::filter_range_data(r.start, r.end, current, d))\n                }\n\n                RangeType::Multi(_) => {\n                    let data = data?;\n                    let current = self.current;\n                    let data_len = data.len();\n                    self.current += data_len;\n                    self.filter_multi_range_body(data, current, data_len)\n                }\n            }\n        }\n\n        fn filter_range_data(\n            start: usize,\n            end: usize,\n            current: usize,\n            data: Bytes,\n        ) -> Option<Bytes> {\n            if current + data.len() < start || current >= end {\n                // if the current data is out side the desired range, just drop the data\n                None\n            } else if current >= start && current + data.len() <= end {\n                // all data is within the slice\n                Some(data)\n            } else {\n                // data:  current........current+data.len()\n                // range: start...........end\n                let slice_start = start.saturating_sub(current);\n                let slice_end = std::cmp::min(data.len(), end - current);\n                Some(data.slice(slice_start..slice_end))\n            }\n        }\n\n        // Returns the multipart header for a given range\n        fn build_multipart_header(\n            &self,\n            range: &Range<usize>,\n            boundary: &str,\n            total_length: &usize,\n            content_type: Option<&str>,\n        ) -> Bytes {\n            Bytes::from(format!(\n                \"\\r\\n--{}\\r\\n{}Content-Range: bytes {}-{}/{}\\r\\n\\r\\n\",\n                boundary,\n                content_type.map_or(String::new(), |ct| format!(\"Content-Type: {ct}\\r\\n\")),\n                range.start,\n                range.end - 1,\n                total_length\n            ))\n        }\n\n        // Return true if chunk includes the start of the given range\n        fn current_chunk_includes_range_start(\n            &self,\n            range: &Range<usize>,\n            current: usize,\n            data_len: usize,\n        ) -> bool {\n            range.start >= current && range.start < current + data_len\n        }\n\n        // Return true if chunk includes the end of the given range\n        fn current_chunk_includes_range_end(\n            &self,\n            range: &Range<usize>,\n            current: usize,\n            data_len: usize,\n        ) -> bool {\n            range.end > current && range.end <= current + data_len\n        }\n\n        fn filter_multi_range_body(\n            &mut self,\n            data: Bytes,\n            current: usize,\n            data_len: usize,\n        ) -> Option<Bytes> {\n            let mut result = BytesMut::new();\n\n            let RangeType::Multi(multi_part_info) = &self.range else {\n                return None;\n            };\n\n            let multipart_idx = self.multipart_idx.expect(\"must be set on multirange\");\n            let final_range = multi_part_info.ranges.last()?;\n\n            let (_, remaining_ranges) = multi_part_info.ranges.as_slice().split_at(multipart_idx);\n            // NOTE: current invariant is that the multipart info ranges are disjoint ascending\n            // this code is invalid if this invariant is not upheld\n            for range in remaining_ranges {\n                if let Some(sliced) =\n                    Self::filter_range_data(range.start, range.end, current, data.clone())\n                {\n                    if self.current_chunk_includes_range_start(range, current, data_len) {\n                        result.extend_from_slice(&self.build_multipart_header(\n                            range,\n                            multi_part_info.boundary.as_ref(),\n                            &multi_part_info.total_length,\n                            multi_part_info.content_type.as_deref(),\n                        ));\n                    }\n                    // Emit the actual data bytes\n                    result.extend_from_slice(&sliced);\n                    if self.current_chunk_includes_range_end(range, current, data_len) {\n                        // If this was the last range, we should emit the final footer too\n                        if range == final_range {\n                            if let Some(final_chunk) = self.finalize(&multi_part_info.boundary) {\n                                result.extend_from_slice(&final_chunk);\n                            }\n                        }\n                        // done with this range\n                        self.multipart_idx = Some(self.multipart_idx.expect(\"must be set\") + 1);\n                    }\n                } else {\n                    // no part of the data was within this range,\n                    // so lower bound of this range (and remaining ranges) must be\n                    // > current + data_len\n                    break;\n                }\n            }\n            if result.is_empty() {\n                None\n            } else {\n                Some(result.freeze())\n            }\n        }\n    }\n\n    #[test]\n    fn test_range_body_filter_single() {\n        let mut body_filter = RangeBodyFilter::new_range(RangeType::None);\n        assert_eq!(body_filter.filter_body(Some(\"123\".into())).unwrap(), \"123\");\n\n        let mut body_filter = RangeBodyFilter::new_range(RangeType::Invalid);\n        assert!(body_filter.filter_body(Some(\"123\".into())).is_none());\n\n        let mut body_filter = RangeBodyFilter::new_range(RangeType::new_single(0, 1));\n        assert_eq!(body_filter.filter_body(Some(\"012\".into())).unwrap(), \"0\");\n        assert!(body_filter.filter_body(Some(\"345\".into())).is_none());\n\n        let mut body_filter = RangeBodyFilter::new_range(RangeType::new_single(4, 6));\n        assert!(body_filter.filter_body(Some(\"012\".into())).is_none());\n        assert_eq!(body_filter.filter_body(Some(\"345\".into())).unwrap(), \"45\");\n        assert!(body_filter.filter_body(Some(\"678\".into())).is_none());\n\n        let mut body_filter = RangeBodyFilter::new_range(RangeType::new_single(1, 7));\n        assert_eq!(body_filter.filter_body(Some(\"012\".into())).unwrap(), \"12\");\n        assert_eq!(body_filter.filter_body(Some(\"345\".into())).unwrap(), \"345\");\n        assert_eq!(body_filter.filter_body(Some(\"678\".into())).unwrap(), \"6\");\n    }\n\n    #[test]\n    fn test_range_body_filter_multipart() {\n        // Test #1 - Test multipart ranges from 1 chunk\n        let data = Bytes::from(\"0123456789\");\n        let ranges = vec![0..3, 6..9];\n        let content_length = data.len();\n        let mut body_filter = RangeBodyFilter::new();\n        body_filter.set(RangeType::new_multi(ranges.clone()));\n\n        body_filter\n            .range\n            .update_multirange_info(content_length, None);\n\n        let multi_range_info = body_filter\n            .range\n            .get_multirange_info()\n            .cloned()\n            .expect(\"Multipart Ranges should have MultiPartInfo struct\");\n\n        // Pass the whole body in one chunk\n        let output = body_filter.filter_body(Some(data)).unwrap();\n        let footer = body_filter.finalize(&multi_range_info.boundary).unwrap();\n\n        // Convert to String so that we can inspect whole response\n        let output_str = str::from_utf8(&output).unwrap();\n        let final_boundary = str::from_utf8(&footer).unwrap();\n        let boundary = &multi_range_info.boundary;\n\n        // Check part headers\n        for (i, range) in ranges.iter().enumerate() {\n            let header = &format!(\n                \"--{}\\r\\nContent-Range: bytes {}-{}/{}\\r\\n\\r\\n\",\n                boundary,\n                range.start,\n                range.end - 1,\n                content_length\n            );\n            assert!(\n                output_str.contains(header),\n                \"Missing part header {} in multipart body\",\n                i\n            );\n            // Check body matches\n            let expected_body = &\"0123456789\"[range.clone()];\n            assert!(\n                output_str.contains(expected_body),\n                \"Missing body {} for range {:?}\",\n                expected_body,\n                range\n            )\n        }\n        // Check the final boundary footer\n        assert_eq!(final_boundary, format!(\"\\r\\n--{}--\\r\\n\", boundary));\n\n        // Test #2 - Test multipart ranges from multiple chunks\n        let full_body = b\"0123456789\";\n        let ranges = vec![0..2, 4..6, 8..9];\n        let content_length = full_body.len();\n        let content_type = \"text/plain\".to_string();\n        let mut body_filter = RangeBodyFilter::new();\n        body_filter.set(RangeType::new_multi(ranges.clone()));\n\n        body_filter\n            .range\n            .update_multirange_info(content_length, Some(content_type.clone()));\n\n        let multi_range_info = body_filter\n            .range\n            .get_multirange_info()\n            .cloned()\n            .expect(\"Multipart Ranges should have MultiPartInfo struct\");\n\n        // Split the body into 4 chunks\n        let chunk1 = Bytes::from_static(b\"012\");\n        let chunk2 = Bytes::from_static(b\"345\");\n        let chunk3 = Bytes::from_static(b\"678\");\n        let chunk4 = Bytes::from_static(b\"9\");\n\n        let mut collected_bytes = BytesMut::new();\n        for chunk in [chunk1, chunk2, chunk3, chunk4] {\n            if let Some(filtered) = body_filter.filter_body(Some(chunk)) {\n                collected_bytes.extend_from_slice(&filtered);\n            }\n        }\n        if let Some(final_boundary) = body_filter.finalize(&multi_range_info.boundary) {\n            collected_bytes.extend_from_slice(&final_boundary);\n        }\n\n        let output_str = str::from_utf8(&collected_bytes).unwrap();\n        let boundary = multi_range_info.boundary;\n\n        for (i, range) in ranges.iter().enumerate() {\n            let header = &format!(\n                \"--{}\\r\\nContent-Type: {}\\r\\nContent-Range: bytes {}-{}/{}\\r\\n\\r\\n\",\n                boundary,\n                content_type,\n                range.start,\n                range.end - 1,\n                content_length\n            );\n            let expected_body = &full_body[range.clone()];\n            let expected_output = format!(\"{}{}\", header, str::from_utf8(expected_body).unwrap());\n\n            assert!(\n                output_str.contains(&expected_output),\n                \"Missing or malformed part {} in multipart body. \\n Expected: \\n{}\\n Got: \\n{}\",\n                i,\n                expected_output,\n                output_str\n            )\n        }\n\n        assert!(\n            output_str.ends_with(&format!(\"\\r\\n--{}--\\r\\n\", boundary)),\n            \"Missing final boundary\"\n        );\n\n        // Test #3 - Test multipart ranges from multiple chunks, with ranges spanning chunks\n        let full_body = b\"abcdefghijkl\";\n        let ranges = vec![2..7, 9..11];\n        let content_length = full_body.len();\n        let content_type = \"application/octet-stream\".to_string();\n        let mut body_filter = RangeBodyFilter::new();\n        body_filter.set(RangeType::new_multi(ranges.clone()));\n\n        body_filter\n            .range\n            .update_multirange_info(content_length, Some(content_type.clone()));\n\n        let multi_range_info = body_filter\n            .range\n            .clone()\n            .get_multirange_info()\n            .cloned()\n            .expect(\"Multipart Ranges should have MultiPartInfo struct\");\n\n        // Split the body into 4 chunks\n        let chunk1 = Bytes::from_static(b\"abc\");\n        let chunk2 = Bytes::from_static(b\"def\");\n        let chunk3 = Bytes::from_static(b\"ghi\");\n        let chunk4 = Bytes::from_static(b\"jkl\");\n\n        let mut collected_bytes = BytesMut::new();\n        for chunk in [chunk1, chunk2, chunk3, chunk4] {\n            if let Some(filtered) = body_filter.filter_body(Some(chunk)) {\n                collected_bytes.extend_from_slice(&filtered);\n            }\n        }\n        if let Some(final_boundary) = body_filter.finalize(&multi_range_info.boundary) {\n            collected_bytes.extend_from_slice(&final_boundary);\n        }\n\n        let output_str = str::from_utf8(&collected_bytes).unwrap();\n        let boundary = &multi_range_info.boundary;\n\n        let header1 = &format!(\n            \"--{}\\r\\nContent-Type: {}\\r\\nContent-Range: bytes {}-{}/{}\\r\\n\\r\\n\",\n            boundary,\n            content_type,\n            ranges[0].start,\n            ranges[0].end - 1,\n            content_length\n        );\n        let header2 = &format!(\n            \"--{}\\r\\nContent-Type: {}\\r\\nContent-Range: bytes {}-{}/{}\\r\\n\\r\\n\",\n            boundary,\n            content_type,\n            ranges[1].start,\n            ranges[1].end - 1,\n            content_length\n        );\n\n        assert!(output_str.contains(header1));\n        assert!(output_str.contains(header2));\n\n        let expected_body_slices = [\"cdefg\", \"jk\"];\n\n        assert!(\n            output_str.contains(expected_body_slices[0]),\n            \"Missing expected sliced body {}\",\n            expected_body_slices[0]\n        );\n\n        assert!(\n            output_str.contains(expected_body_slices[1]),\n            \"Missing expected sliced body {}\",\n            expected_body_slices[1]\n        );\n\n        assert!(\n            output_str.ends_with(&format!(\"\\r\\n--{}--\\r\\n\", boundary)),\n            \"Missing final boundary\"\n        );\n    }\n}\n\n// a state machine for proxy logic to tell when to use cache in the case of\n// miss/revalidation/error.\n#[derive(Debug)]\npub(crate) enum ServeFromCache {\n    // not using cache\n    Off,\n    // should serve cache header\n    CacheHeader,\n    // should serve cache header only\n    CacheHeaderOnly,\n    // should serve cache header only but upstream response should be admitted to cache\n    CacheHeaderOnlyMiss,\n    // should serve cache body with a bool to indicate if it has already called seek on the hit handler\n    CacheBody(bool),\n    // should serve cache header but upstream response should be admitted to cache\n    // This is the starting state for misses, which go to CacheBodyMiss or\n    // CacheHeaderOnlyMiss before ending at DoneMiss\n    CacheHeaderMiss,\n    // should serve cache body but upstream response should be admitted to cache, bool to indicate seek status\n    CacheBodyMiss(bool),\n    // done serving cache body\n    Done,\n    // done serving cache body, but upstream response should continue to be admitted to cache\n    DoneMiss,\n}\n\nimpl ServeFromCache {\n    pub fn new() -> Self {\n        Self::Off\n    }\n\n    pub fn is_on(&self) -> bool {\n        !matches!(self, Self::Off)\n    }\n\n    pub fn is_miss(&self) -> bool {\n        matches!(\n            self,\n            Self::CacheHeaderMiss\n                | Self::CacheHeaderOnlyMiss\n                | Self::CacheBodyMiss(_)\n                | Self::DoneMiss\n        )\n    }\n\n    pub fn is_miss_header(&self) -> bool {\n        // NOTE: this check is for checking if miss was just enabled, so it is excluding\n        // HeaderOnlyMiss\n        matches!(self, Self::CacheHeaderMiss)\n    }\n\n    pub fn is_miss_body(&self) -> bool {\n        matches!(self, Self::CacheBodyMiss(_))\n    }\n\n    pub fn should_discard_upstream(&self) -> bool {\n        self.is_on() && !self.is_miss()\n    }\n\n    pub fn should_send_to_downstream(&self) -> bool {\n        !self.is_on()\n    }\n\n    pub fn enable(&mut self) {\n        *self = Self::CacheHeader;\n    }\n\n    pub fn enable_miss(&mut self) {\n        if !self.is_on() {\n            *self = Self::CacheHeaderMiss;\n        }\n    }\n\n    pub fn enable_header_only(&mut self) {\n        match self {\n            Self::CacheBody(_) => *self = Self::Done, // TODO: make sure no body is read yet\n            Self::CacheBodyMiss(_) => *self = Self::DoneMiss,\n            _ => {\n                if self.is_miss() {\n                    *self = Self::CacheHeaderOnlyMiss;\n                } else {\n                    *self = Self::CacheHeaderOnly;\n                }\n            }\n        }\n    }\n\n    // This function is (best effort) cancel-safe to be used in select\n    pub async fn next_http_task(\n        &mut self,\n        cache: &mut HttpCache,\n        range: &mut RangeBodyFilter,\n        upgraded: bool,\n    ) -> Result<HttpTask> {\n        fn body_task(data: Bytes, upgraded: bool) -> HttpTask {\n            if upgraded {\n                HttpTask::UpgradedBody(Some(data), false)\n            } else {\n                HttpTask::Body(Some(data), false)\n            }\n        }\n\n        if !cache.enabled() {\n            // Cache is disabled due to internal error\n            // TODO: if nothing is sent to eyeball yet, figure out a way to recovery by\n            // fetching from upstream\n            return Error::e_explain(InternalError, \"Cache disabled\");\n        }\n        match self {\n            Self::Off => panic!(\"ProxyUseCache not enabled\"),\n            Self::CacheHeader => {\n                *self = Self::CacheBody(true);\n                Ok(HttpTask::Header(cache_hit_header(cache), false)) // false for now\n            }\n            Self::CacheHeaderMiss => {\n                *self = Self::CacheBodyMiss(true);\n                Ok(HttpTask::Header(cache_hit_header(cache), false)) // false for now\n            }\n            Self::CacheHeaderOnly => {\n                *self = Self::Done;\n                Ok(HttpTask::Header(cache_hit_header(cache), true))\n            }\n            Self::CacheHeaderOnlyMiss => {\n                *self = Self::DoneMiss;\n                Ok(HttpTask::Header(cache_hit_header(cache), true))\n            }\n            Self::CacheBody(should_seek) => {\n                log::trace!(\"cache body should seek: {should_seek}\");\n                if *should_seek {\n                    self.maybe_seek_hit_handler(cache, range)?;\n                }\n                loop {\n                    if let Some(b) = cache.hit_handler().read_body().await? {\n                        return Ok(body_task(b, upgraded));\n                    }\n                    // EOF from hit handler for body requested\n                    // if multipart, then seek again\n                    if range.should_cache_seek_again() {\n                        self.maybe_seek_hit_handler(cache, range)?;\n                    } else {\n                        *self = Self::Done;\n                        return Ok(HttpTask::Done);\n                    }\n                }\n            }\n            Self::CacheBodyMiss(should_seek) => {\n                if *should_seek {\n                    self.maybe_seek_miss_handler(cache, range)?;\n                }\n                // safety: caller of enable_miss() call it only if the async_body_reader exist\n                loop {\n                    if let Some(b) = cache.miss_body_reader().unwrap().read_body().await? {\n                        return Ok(body_task(b, upgraded));\n                    } else {\n                        // EOF from hit handler for body requested\n                        // if multipart, then seek again\n                        if range.should_cache_seek_again() {\n                            self.maybe_seek_miss_handler(cache, range)?;\n                        } else {\n                            *self = Self::DoneMiss;\n                            return Ok(HttpTask::Done);\n                        }\n                    }\n                }\n            }\n            Self::Done => Ok(HttpTask::Done),\n            Self::DoneMiss => Ok(HttpTask::Done),\n        }\n    }\n\n    fn maybe_seek_miss_handler(\n        &mut self,\n        cache: &mut HttpCache,\n        range_filter: &mut RangeBodyFilter,\n    ) -> Result<()> {\n        match &range_filter.range {\n            RangeType::Single(range) => {\n                // safety: called only if the async_body_reader exists\n                if cache.miss_body_reader().unwrap().can_seek() {\n                    cache\n                        .miss_body_reader()\n                        // safety: called only if the async_body_reader exists\n                        .unwrap()\n                        .seek(range.start, Some(range.end))\n                        .or_err(InternalError, \"cannot seek miss handler\")?;\n                    // Because the miss body reader is seeking, we no longer need the\n                    // RangeBodyFilter's help to return the requested byte range.\n                    range_filter.range = RangeType::None;\n                }\n            }\n            RangeType::Multi(_info) => {\n                // safety: called only if the async_body_reader exists\n                if cache.miss_body_reader().unwrap().can_seek_multipart() {\n                    let range = range_filter.next_cache_multipart_range();\n                    cache\n                        .miss_body_reader()\n                        .unwrap()\n                        .seek_multipart(range.start, Some(range.end))\n                        .or_err(InternalError, \"cannot seek hit handler for multirange\")?;\n                    // we still need RangeBodyFilter's help to transform the byte\n                    // range into a multipart response.\n                    range_filter.set_current_cursor(range.start);\n                }\n            }\n            _ => {}\n        }\n\n        *self = Self::CacheBodyMiss(false);\n        Ok(())\n    }\n\n    fn maybe_seek_hit_handler(\n        &mut self,\n        cache: &mut HttpCache,\n        range_filter: &mut RangeBodyFilter,\n    ) -> Result<()> {\n        match &range_filter.range {\n            RangeType::Single(range) => {\n                if cache.hit_handler().can_seek() {\n                    cache\n                        .hit_handler()\n                        .seek(range.start, Some(range.end))\n                        .or_err(InternalError, \"cannot seek hit handler\")?;\n                    // Because the hit handler is seeking, we no longer need the\n                    // RangeBodyFilter's help to return the requested byte range.\n                    range_filter.range = RangeType::None;\n                }\n            }\n            RangeType::Multi(_info) => {\n                if cache.hit_handler().can_seek_multipart() {\n                    let range = range_filter.next_cache_multipart_range();\n                    cache\n                        .hit_handler()\n                        .seek_multipart(range.start, Some(range.end))\n                        .or_err(InternalError, \"cannot seek hit handler for multirange\")?;\n                    // we still need RangeBodyFilter's help to transform the byte\n                    // range into a multipart response.\n                    range_filter.set_current_cursor(range.start);\n                }\n            }\n            _ => {}\n        }\n        *self = Self::CacheBody(false);\n        Ok(())\n    }\n}\n"
  },
  {
    "path": "pingora-proxy/src/proxy_common.rs",
    "content": "/// Possible downstream states during request multiplexing\n#[derive(Debug, Clone, Copy)]\npub(crate) enum DownstreamStateMachine {\n    /// more request (body) to read\n    Reading,\n    /// no more data to read\n    ReadingFinished,\n    /// downstream is already errored or closed\n    Errored,\n}\n\n#[allow(clippy::wrong_self_convention)]\nimpl DownstreamStateMachine {\n    pub fn new(finished: bool) -> Self {\n        if finished {\n            Self::ReadingFinished\n        } else {\n            Self::Reading\n        }\n    }\n\n    // Can call read() to read more data or wait on closing\n    pub fn can_poll(&self) -> bool {\n        !matches!(self, Self::Errored)\n    }\n\n    pub fn is_reading(&self) -> bool {\n        matches!(self, Self::Reading)\n    }\n\n    pub fn is_done(&self) -> bool {\n        !matches!(self, Self::Reading)\n    }\n\n    pub fn is_errored(&self) -> bool {\n        matches!(self, Self::Errored)\n    }\n\n    /// Move the state machine to Finished state if `set` is true\n    pub fn maybe_finished(&mut self, set: bool) {\n        if set {\n            *self = Self::ReadingFinished\n        }\n    }\n\n    /// Reset if we should continue reading from the downstream again.\n    /// Only used with upgraded connections when body mode changes.\n    pub fn reset(&mut self) {\n        *self = Self::Reading;\n    }\n\n    pub fn to_errored(&mut self) {\n        *self = Self::Errored\n    }\n}\n\n/// Possible upstream states during request multiplexing\n#[derive(Debug, Clone, Copy)]\npub(crate) struct ResponseStateMachine {\n    upstream_response_done: bool,\n    cached_response_done: bool,\n}\n\nimpl ResponseStateMachine {\n    pub fn new() -> Self {\n        ResponseStateMachine {\n            upstream_response_done: false,\n            cached_response_done: true, // no cached response by default\n        }\n    }\n\n    pub fn is_done(&self) -> bool {\n        self.upstream_response_done && self.cached_response_done\n    }\n\n    pub fn upstream_done(&self) -> bool {\n        self.upstream_response_done\n    }\n\n    pub fn cached_done(&self) -> bool {\n        self.cached_response_done\n    }\n\n    pub fn enable_cached_response(&mut self) {\n        self.cached_response_done = false;\n    }\n\n    pub fn maybe_set_upstream_done(&mut self, done: bool) {\n        if done {\n            self.upstream_response_done = true;\n        }\n    }\n\n    pub fn maybe_set_cache_done(&mut self, done: bool) {\n        if done {\n            self.cached_response_done = true;\n        }\n    }\n}\n"
  },
  {
    "path": "pingora-proxy/src/proxy_custom.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\nuse futures::StreamExt;\nuse pingora_core::{\n    protocols::http::custom::{\n        client::Session as CustomSession, is_informational_except_101, BodyWrite,\n        CustomMessageWrite, CUSTOM_MESSAGE_QUEUE_SIZE,\n    },\n    ImmutStr,\n};\nuse proxy_cache::{range_filter::RangeBodyFilter, ServeFromCache};\nuse proxy_common::{DownstreamStateMachine, ResponseStateMachine};\nuse tokio::sync::oneshot;\n\nuse super::*;\n\nimpl<SV, C> HttpProxy<SV, C>\nwhere\n    C: custom::Connector,\n{\n    /// Proxy to a custom protocol upstream.\n    /// Returns (reuse_server, error)\n    pub(crate) async fn proxy_to_custom_upstream(\n        &self,\n        session: &mut Session,\n        client_session: &mut C::Session,\n        reused: bool,\n        peer: &HttpPeer,\n        ctx: &mut SV::CTX,\n    ) -> (bool, Option<Box<Error>>)\n    where\n        SV: ProxyHttp + Send + Sync,\n        SV::CTX: Send + Sync,\n    {\n        #[cfg(windows)]\n        let raw = client_session.fd() as std::os::windows::io::RawSocket;\n        #[cfg(unix)]\n        let raw = client_session.fd();\n\n        if let Err(e) = self\n            .inner\n            .connected_to_upstream(session, reused, peer, raw, client_session.digest(), ctx)\n            .await\n        {\n            return (false, Some(e));\n        }\n\n        let (server_session_reuse, error) = self\n            .custom_proxy_down_to_up(session, client_session, peer, ctx)\n            .await;\n\n        // Parity with H1/H2: custom upstreams don't report payload bytes; record 0.\n        session.set_upstream_body_bytes_received(0);\n\n        (server_session_reuse, error)\n    }\n\n    /// Handle custom protocol proxying from downstream to upstream.\n    /// Returns (reuse_server, error)\n    async fn custom_proxy_down_to_up(\n        &self,\n        session: &mut Session,\n        client_session: &mut C::Session,\n        peer: &HttpPeer,\n        ctx: &mut SV::CTX,\n    ) -> (bool, Option<Box<Error>>)\n    where\n        SV: ProxyHttp + Send + Sync,\n        SV::CTX: Send + Sync,\n    {\n        let mut req = session.req_header().clone();\n\n        if session.cache.enabled() {\n            pingora_cache::filters::upstream::request_filter(\n                &mut req,\n                session.cache.maybe_cache_meta(),\n            );\n            session.mark_upstream_headers_mutated_for_cache();\n        }\n\n        match self\n            .inner\n            .upstream_request_filter(session, &mut req, ctx)\n            .await\n        {\n            Ok(_) => { /* continue */ }\n            Err(e) => {\n                return (false, Some(e));\n            }\n        }\n\n        session.upstream_compression.request_filter(&req);\n        let body_empty = session.as_mut().is_body_empty();\n\n        debug!(\"Request to custom: {req:?}\");\n\n        let req = Box::new(req);\n        if let Err(e) = client_session.write_request_header(req, body_empty).await {\n            return (false, Some(e.into_up()));\n        }\n\n        client_session.set_read_timeout(peer.options.read_timeout);\n        client_session.set_write_timeout(peer.options.write_timeout);\n\n        // take the body writer out of the client for easy duplex\n        let mut client_body = client_session\n            .take_request_body_writer()\n            .expect(\"already send request header\");\n\n        let (tx, rx) = mpsc::channel::<HttpTask>(TASK_BUFFER_SIZE);\n\n        session.as_mut().enable_retry_buffering();\n\n        // Custom message logic\n\n        let Some(mut upstream_custom_message_reader) = client_session.take_custom_message_reader()\n        else {\n            return (\n                false,\n                Some(Error::explain(\n                    ReadError,\n                    \"can't extract custom reader from upstream\",\n                )),\n            );\n        };\n\n        let Some(mut upstream_custom_message_writer) = client_session.take_custom_message_writer()\n        else {\n            return (\n                false,\n                Some(Error::explain(\n                    WriteError,\n                    \"custom upstream must have a custom message writer\",\n                )),\n            );\n        };\n\n        // A channel to inject custom messages to upstream from server logic.\n        let (upstream_custom_message_inject_tx, upstream_custom_message_inject_rx) =\n            mpsc::channel(CUSTOM_MESSAGE_QUEUE_SIZE);\n\n        // Downstream reader\n        let mut downstream_custom_message_reader = match session.downstream_custom_message() {\n            Ok(Some(rx)) => rx,\n            Ok(None) => Box::new(futures::stream::empty::<Result<Bytes>>()),\n            Err(err) => return (false, Some(err)),\n        };\n\n        // Downstream writer\n        let (mut downstream_custom_message_writer, downstream_custom_final_hop): (\n            Box<dyn CustomMessageWrite>,\n            bool, // if this hop is final\n        ) = if let Some(custom_session) = session.downstream_session.as_custom_mut() {\n            (\n                custom_session\n                    .take_custom_message_writer()\n                    .expect(\"custom downstream must have a custom message writer\"),\n                false,\n            )\n        } else {\n            (Box::new(()), true)\n        };\n\n        // A channel to inject custom messages to downstream from server logic.\n        let (downstream_custom_message_inject_tx, downstream_custom_message_inject_rx) =\n            mpsc::channel(CUSTOM_MESSAGE_QUEUE_SIZE);\n\n        // Filters for ProxyHttp trait\n        let (upstream_custom_message_filter_tx, upstream_custom_message_filter_rx) =\n            mpsc::channel(CUSTOM_MESSAGE_QUEUE_SIZE);\n        let (downstream_custom_message_filter_tx, downstream_custom_message_filter_rx) =\n            mpsc::channel(CUSTOM_MESSAGE_QUEUE_SIZE);\n\n        // Cancellation channels for custom coroutines\n        // The transmitters act as guards: when dropped, they signal the receivers to cancel.\n        // `cancel_downstream_reader_tx` is held and later used to explicitly cancel.\n        // `_cancel_upstream_reader_tx` is unused (prefixed with _) - it will be dropped at the\n        // end of this scope, which automatically signals cancellation to the upstream reader.\n        let (cancel_downstream_reader_tx, cancel_downstream_reader_rx) = oneshot::channel();\n        let (_cancel_upstream_reader_tx, cancel_upstream_reader_rx) = oneshot::channel();\n\n        let upstream_custom_message_forwarder = CustomMessageForwarder {\n            ctx: \"down_to_up\".into(),\n            reader: &mut downstream_custom_message_reader,\n            writer: &mut upstream_custom_message_writer,\n            filter: upstream_custom_message_filter_tx,\n            inject: upstream_custom_message_inject_rx,\n            cancel: cancel_downstream_reader_rx,\n        };\n\n        let downstream_custom_message_forwarder = CustomMessageForwarder {\n            ctx: \"up_to_down\".into(),\n            reader: &mut upstream_custom_message_reader,\n            writer: &mut downstream_custom_message_writer,\n            filter: downstream_custom_message_filter_tx,\n            inject: downstream_custom_message_inject_rx,\n            cancel: cancel_upstream_reader_rx,\n        };\n\n        if let Err(e) = self\n            .inner\n            .custom_forwarding(\n                session,\n                ctx,\n                Some(upstream_custom_message_inject_tx),\n                downstream_custom_message_inject_tx,\n            )\n            .await\n        {\n            return (false, Some(e));\n        }\n\n        /* read downstream body and upstream response at the same time */\n        let ret = tokio::try_join!(\n            self.custom_bidirection_down_to_up(\n                session,\n                &mut client_body,\n                rx,\n                ctx,\n                upstream_custom_message_filter_rx,\n                downstream_custom_message_filter_rx,\n                downstream_custom_final_hop,\n                cancel_downstream_reader_tx,\n            ),\n            custom_pipe_up_to_down_response(client_session, tx),\n            upstream_custom_message_forwarder.proxy(),\n            downstream_custom_message_forwarder.proxy(),\n        );\n\n        if let Some(custom_session) = session.downstream_session.as_custom_mut() {\n            custom_session\n                .restore_custom_message_writer(downstream_custom_message_writer)\n                .expect(\"downstream restore_custom_message_writer should be empty\");\n\n            custom_session\n                .restore_custom_message_reader(downstream_custom_message_reader)\n                .expect(\"downstream restore_custom_message_reader should be empty\");\n        }\n\n        match ret {\n            Ok((downstream_can_reuse, _upstream, _custom_up_down, _custom_down_up)) => {\n                (downstream_can_reuse, None)\n            }\n            Err(e) => (false, Some(e)),\n        }\n    }\n\n    // returns whether server (downstream) session can be reused\n    #[allow(clippy::too_many_arguments)]\n    async fn custom_bidirection_down_to_up(\n        &self,\n        session: &mut Session,\n        client_body: &mut Box<dyn BodyWrite>,\n        mut rx: mpsc::Receiver<HttpTask>,\n        ctx: &mut SV::CTX,\n        mut upstream_custom_message_filter_rx: mpsc::Receiver<(\n            Bytes,\n            oneshot::Sender<Option<Bytes>>,\n        )>,\n        mut downstream_custom_message_filter_rx: mpsc::Receiver<(\n            Bytes,\n            oneshot::Sender<Option<Bytes>>,\n        )>,\n        downstream_custom_final_hop: bool,\n        cancel_downstream_reader_tx: oneshot::Sender<()>,\n    ) -> Result<bool>\n    where\n        SV: ProxyHttp + Send + Sync,\n        SV::CTX: Send + Sync,\n    {\n        let mut cancel_downstream_reader_tx = Some(cancel_downstream_reader_tx);\n\n        let mut downstream_state = DownstreamStateMachine::new(session.as_mut().is_body_done());\n\n        // retry, send buffer if it exists\n        if let Some(buffer) = session.as_mut().get_retry_buffer() {\n            self.send_body_to_custom(\n                session,\n                Some(buffer),\n                downstream_state.is_done(),\n                client_body,\n                ctx,\n            )\n            .await?;\n        }\n\n        let mut response_state = ResponseStateMachine::new();\n\n        // these two below can be wrapped into an internal ctx\n        // use cache when upstream revalidates (or TODO: error)\n        let mut serve_from_cache = ServeFromCache::new();\n        let mut range_body_filter = proxy_cache::range_filter::RangeBodyFilter::new();\n\n        let mut upstream_custom = true;\n        let mut downstream_custom = true;\n\n        /* duplex mode\n         * see the Same function for h1 for more comments\n         */\n        while !downstream_state.is_done()\n            || !response_state.is_done()\n            || upstream_custom\n            || downstream_custom\n        {\n            // partial read support, this check will also be false if cache is disabled.\n            let support_cache_partial_read =\n                session.cache.support_streaming_partial_write() == Some(true);\n            let upgraded = session.was_upgraded();\n\n            tokio::select! {\n                body = session.downstream_session.read_body_or_idle(downstream_state.is_done()), if downstream_state.can_poll() => {\n                    let body = match body {\n                        Ok(b) => b,\n                        Err(e) => {\n                            let wait_for_cache_fill = (!serve_from_cache.is_on() && support_cache_partial_read)\n                                || serve_from_cache.is_miss();\n                            if wait_for_cache_fill {\n                                // ignore downstream error so that upstream can continue to write cache\n                                downstream_state.to_errored();\n                                warn!(\n                                    \"Downstream Error ignored during caching: {}, {}\",\n                                    e,\n                                    self.inner.request_summary(session, ctx)\n                                );\n                                continue;\n                           } else {\n                                return Err(e.into_down());\n                           }\n                        }\n                    };\n                    let is_body_done = session.is_body_done();\n\n                    match self.send_body_to_custom(session, body, is_body_done, client_body, ctx).await {\n                        Ok(request_done) =>  {\n                            downstream_state.maybe_finished(request_done);\n                        },\n                        Err(e) => {\n                            // mark request done, attempt to drain receive\n                            warn!(\"body send error: {e}\");\n\n                            // upstream is what actually errored but we don't want to continue\n                            // polling the downstream body\n                            downstream_state.to_errored();\n\n                            // downstream still trying to send something, but the upstream is already stooped\n                            // cancel the custom downstream to upstream coroutine, because the proxy will not see EOS.\n                            let _ = cancel_downstream_reader_tx.take().expect(\"cancel must be set and called once\").send(());\n                        }\n                    };\n                },\n\n                task = rx.recv(), if !response_state.upstream_done() => {\n                    debug!(\"upstream event\");\n\n                    if let Some(t) = task {\n                        debug!(\"upstream event custom: {:?}\", t);\n                        if serve_from_cache.should_discard_upstream() {\n                            // just drain, do we need to do anything else?\n                           continue;\n                        }\n                        // pull as many tasks as we can\n                        let mut tasks = Vec::with_capacity(TASK_BUFFER_SIZE);\n                        tasks.push(t);\n                        while let Ok(task) = rx.try_recv() {\n                            tasks.push(task);\n                        }\n\n                        /* run filters before sending to downstream */\n                        let mut filtered_tasks = Vec::with_capacity(TASK_BUFFER_SIZE);\n                        for mut t in tasks {\n                            if self.revalidate_or_stale(session, &mut t, ctx).await {\n                                serve_from_cache.enable();\n                                response_state.enable_cached_response();\n                                // skip downstream filtering entirely as the 304 will not be sent\n                                break;\n                            }\n                            session.upstream_compression.response_filter(&mut t);\n                            // check error and abort\n                            // otherwise the error is surfaced via write_response_tasks()\n                            if !serve_from_cache.should_send_to_downstream() {\n                                if let HttpTask::Failed(e) = t {\n                                    return Err(e);\n                                }\n                            }\n                            filtered_tasks.push(\n                                self.custom_response_filter(session, t, ctx,\n                                    &mut serve_from_cache,\n                                    &mut range_body_filter, false).await?);\n                            if serve_from_cache.is_miss_header() {\n                                response_state.enable_cached_response();\n                            }\n                        }\n\n                        if !serve_from_cache.should_send_to_downstream() {\n                            // TODO: need to derive response_done from filtered_tasks in case downstream failed already\n                            continue;\n                        }\n\n                        let upgraded = session.was_upgraded();\n                        let response_done = session.write_response_tasks(filtered_tasks).await?;\n                        if !upgraded && session.was_upgraded() && downstream_state.can_poll() {\n                            // just upgraded, the downstream state should be reset to continue to\n                            // poll body\n                            trace!(\"reset downstream state on upgrade\");\n                            downstream_state.reset();\n                        }\n\n                        response_state.maybe_set_upstream_done(response_done);\n                    } else {\n                        debug!(\"empty upstream event\");\n                        response_state.maybe_set_upstream_done(true);\n                    }\n                }\n\n                task = serve_from_cache.next_http_task(&mut session.cache, &mut range_body_filter, upgraded),\n                    if !response_state.cached_done() && !downstream_state.is_errored() && serve_from_cache.is_on() => {\n                    let task = self.custom_response_filter(session, task?, ctx,\n                        &mut serve_from_cache,\n                        &mut range_body_filter, true).await?;\n                    match session.write_response_tasks(vec![task]).await {\n                        Ok(b) => response_state.maybe_set_cache_done(b),\n                        Err(e) => if serve_from_cache.is_miss() {\n                            // give up writing to downstream but wait for upstream cache write to finish\n                            downstream_state.to_errored();\n                            response_state.maybe_set_cache_done(true);\n                            warn!(\n                                \"Downstream Error ignored during caching: {}, {}\",\n                                e,\n                                self.inner.request_summary(session, ctx)\n                            );\n                            continue;\n                        } else {\n                            return Err(e);\n                        }\n                    }\n                    if response_state.cached_done() {\n                        if let Err(e) = session.cache.finish_hit_handler().await {\n                            warn!(\"Error during finish_hit_handler: {}\", e);\n                        }\n                    }\n                }\n\n                ret = upstream_custom_message_filter_rx.recv(), if upstream_custom => {\n                    let Some(msg) = ret else {\n                        debug!(\"upstream_custom_message_filter_rx: custom downstream to upstream exited on reading\");\n                        upstream_custom = false;\n                        continue;\n                    };\n\n                    let (data, callback) = msg;\n\n                    let new_msg = self.inner\n                        .downstream_custom_message_proxy_filter(session, data, ctx, false)  // false because the upstream is custom\n                        .await?;\n\n                    if callback.send(new_msg).is_err() {\n                        debug!(\"upstream_custom_message_incoming_rx: custom downstream to upstream exited on callback\");\n                        upstream_custom = false;\n                        continue;\n                    };\n                },\n\n                ret = downstream_custom_message_filter_rx.recv(), if downstream_custom => {\n                    let Some(msg) = ret else {\n                        debug!(\"downstream_custom_message_filter_rx: custom upstream to downstream exited on reading\");\n                        downstream_custom = false;\n                        continue;\n                    };\n\n                    let (data, callback) = msg;\n\n                    let new_msg = self.inner\n                        .upstream_custom_message_proxy_filter(session, data, ctx, downstream_custom_final_hop)\n                        .await?;\n\n                    if callback.send(new_msg).is_err() {\n                        debug!(\"downstream_custom_message_filter_rx: custom upstream to downstream exited on callback\");\n                        downstream_custom = false;\n                        continue\n                    };\n                },\n\n                else => {\n                    break;\n                }\n            }\n        }\n\n        // Re-raise the error then the loop is finished.\n        if downstream_state.is_errored() {\n            let err = Error::e_explain(WriteError, \"downstream_state is_errored\");\n            error!(\"custom_bidirection_down_to_up: downstream_state.is_errored\",);\n            return err;\n        }\n\n        client_body.cleanup().await?;\n\n        let mut reuse_downstream = !downstream_state.is_errored();\n        if reuse_downstream {\n            match session.as_mut().finish_body().await {\n                Ok(_) => {\n                    debug!(\"finished sending body to downstream\");\n                }\n                Err(e) => {\n                    error!(\"Error finish sending body to downstream: {}\", e);\n                    reuse_downstream = false;\n                }\n            }\n        }\n        Ok(reuse_downstream)\n    }\n\n    async fn custom_response_filter(\n        &self,\n        session: &mut Session,\n        mut task: HttpTask,\n        ctx: &mut SV::CTX,\n        serve_from_cache: &mut ServeFromCache,\n        range_body_filter: &mut RangeBodyFilter,\n        from_cache: bool, // are the task from cache already\n    ) -> Result<HttpTask>\n    where\n        SV: ProxyHttp + Send + Sync,\n        SV::CTX: Send + Sync,\n    {\n        if !from_cache {\n            self.upstream_filter(session, &mut task, ctx).await?;\n\n            // cache the original response before any downstream transformation\n            // requests that bypassed cache still need to run filters to see if the response has become cacheable\n            if session.cache.enabled() || session.cache.bypassing() {\n                if let Err(e) = self\n                    .cache_http_task(session, &task, ctx, serve_from_cache)\n                    .await\n                {\n                    session.cache.disable(NoCacheReason::StorageError);\n                    if serve_from_cache.is_miss_body() {\n                        // if the response stream cache body during miss but write fails, it has to\n                        // give up the entire request\n                        return Err(e);\n                    } else {\n                        // otherwise, continue processing the response\n                        warn!(\n                            \"Fail to cache response: {}, {}\",\n                            e,\n                            self.inner.request_summary(session, ctx)\n                        );\n                    }\n                }\n            }\n            // skip the downstream filtering if these tasks are just for cache admission\n            if !serve_from_cache.should_send_to_downstream() {\n                return Ok(task);\n            }\n        } // else: cached/local response, no need to trigger upstream filters and caching\n\n        match task {\n            HttpTask::Header(mut header, eos) => {\n                /* Downstream revalidation, only needed when cache is on because otherwise origin\n                 * will handle it */\n                // TODO: if cache is disabled during response phase, we should still do the filter\n                if session.cache.enabled() {\n                    self.downstream_response_conditional_filter(\n                        serve_from_cache,\n                        session,\n                        &mut header,\n                        ctx,\n                    );\n                    if !session.ignore_downstream_range {\n                        let range_type = self.inner.range_header_filter(session, &mut header, ctx);\n                        range_body_filter.set(range_type);\n                    }\n                }\n\n                self.inner\n                    .response_filter(session, &mut header, ctx)\n                    .await?;\n                /* Downgrade the version so that write_response_header won't panic */\n                header.set_version(Version::HTTP_11);\n\n                // these status codes / method cannot have body, so no need to add chunked encoding\n                let no_body = session.req_header().method == \"HEAD\"\n                    || matches!(header.status.as_u16(), 204 | 304);\n\n                /* Add chunked header to tell downstream to use chunked encoding\n                 * during the absent of content-length */\n                if !no_body\n                    && !header.status.is_informational()\n                    && header.headers.get(http::header::CONTENT_LENGTH).is_none()\n                {\n                    header.insert_header(http::header::TRANSFER_ENCODING, \"chunked\")?;\n                }\n                Ok(HttpTask::Header(header, eos))\n            }\n            HttpTask::Body(data, eos) => {\n                let mut data = range_body_filter.filter_body(data);\n                if let Some(duration) = self\n                    .inner\n                    .response_body_filter(session, &mut data, eos, ctx)?\n                {\n                    trace!(\"delaying response for {duration:?}\");\n                    time::sleep(duration).await;\n                }\n                Ok(HttpTask::Body(data, eos))\n            }\n            HttpTask::UpgradedBody(mut data, eos) => {\n                // range body filter doesn't apply to upgraded body\n                if let Some(duration) = self\n                    .inner\n                    .response_body_filter(session, &mut data, eos, ctx)?\n                {\n                    trace!(\"delaying upgraded response for {duration:?}\");\n                    time::sleep(duration).await;\n                }\n                Ok(HttpTask::UpgradedBody(data, eos))\n            }\n            HttpTask::Trailer(mut trailers) => {\n                let trailer_buffer = match trailers.as_mut() {\n                    Some(trailers) => {\n                        debug!(\"Parsing response trailers..\");\n                        match self\n                            .inner\n                            .response_trailer_filter(session, trailers, ctx)\n                            .await\n                        {\n                            Ok(buf) => buf,\n                            Err(e) => {\n                                error!(\n                                    \"Encountered error while filtering upstream trailers {:?}\",\n                                    e\n                                );\n                                None\n                            }\n                        }\n                    }\n                    _ => None,\n                };\n                // if we have a trailer buffer write it to the downstream response body\n                if let Some(buffer) = trailer_buffer {\n                    // write_body will not write additional bytes after reaching the content-length\n                    // for gRPC H2 -> H1 this is not a problem but may be a problem for non gRPC code\n                    // https://http2.github.io/http2-spec/#malformed\n                    Ok(HttpTask::Body(Some(buffer), true))\n                } else {\n                    Ok(HttpTask::Trailer(trailers))\n                }\n            }\n            HttpTask::Done => Ok(task),\n            HttpTask::Failed(_) => Ok(task), // Do nothing just pass the error down\n        }\n    }\n\n    async fn send_body_to_custom(\n        &self,\n        session: &mut Session,\n        mut data: Option<Bytes>,\n        end_of_body: bool,\n        client_body: &mut Box<dyn BodyWrite>,\n        ctx: &mut SV::CTX,\n    ) -> Result<bool>\n    where\n        SV: ProxyHttp + Send + Sync,\n        SV::CTX: Send + Sync,\n    {\n        session\n            .downstream_modules_ctx\n            .request_body_filter(&mut data, end_of_body)\n            .await?;\n\n        self.inner\n            .request_body_filter(session, &mut data, end_of_body, ctx)\n            .await?;\n\n        if session.was_upgraded() {\n            client_body.upgrade_body_writer();\n        }\n\n        /* it is normal to get 0 bytes because of multi-chunk parsing or request_body_filter.\n         * Although there is no harm writing empty byte to custom, unlike h1, we ignore it\n         * for consistency */\n        if !end_of_body && data.as_ref().is_some_and(|d| d.is_empty()) {\n            return Ok(false);\n        }\n\n        if let Some(mut data) = data {\n            client_body\n                .write_all_buf(&mut data)\n                .await\n                .map_err(|e| e.into_up())?;\n            if end_of_body {\n                client_body.finish().await.map_err(|e| e.into_up())?;\n            }\n        } else {\n            debug!(\"Read downstream body done\");\n            client_body\n                .finish()\n                .await\n                .map_err(|e| {\n                    Error::because(WriteError, \"while shutdown send data stream on no data\", e)\n                })\n                .map_err(|e| e.into_up())?;\n        }\n\n        Ok(end_of_body)\n    }\n}\n\n/* Read response header, body and trailer from custom upstream and send them to tx */\nasync fn custom_pipe_up_to_down_response<S: CustomSession>(\n    client: &mut S,\n    tx: mpsc::Sender<HttpTask>,\n) -> Result<()> {\n    let mut is_informational = true;\n    while is_informational {\n        client\n            .read_response_header()\n            .await\n            .map_err(|e| e.into_up())?;\n        let resp_header = Box::new(client.response_header().expect(\"just read\").clone());\n        // `101 Switching Protocols` is a response to the http1 Upgrade header and it's final response.\n        // The WebSocket Protocol https://datatracker.ietf.org/doc/html/rfc6455\n        is_informational = is_informational_except_101(resp_header.status.as_u16() as u32);\n\n        match client.check_response_end_or_error(true).await {\n            Ok(eos) => {\n                tx.send(HttpTask::Header(resp_header, eos))\n                    .await\n                    .or_err(InternalError, \"sending custom headers to pipe\")?;\n            }\n            Err(e) => {\n                // If upstream errored, then push error to downstream and then quit\n                // Don't care if send fails (which means downstream already gone)\n                // we were still able to retrieve the headers, so try sending\n                let _ = tx.send(HttpTask::Header(resp_header, false)).await;\n                let _ = tx.send(HttpTask::Failed(e.into_up())).await;\n                return Ok(());\n            }\n        }\n    }\n\n    while let Some(chunk) = client\n        .read_response_body()\n        .await\n        .map_err(|e| e.into_up())\n        .transpose()\n    {\n        let data = match chunk {\n            Ok(d) => d,\n            Err(e) => {\n                // Push the error to downstream and then quit\n                let _ = tx.send(HttpTask::Failed(e.into_up())).await;\n                // Downstream should consume all remaining data and handle the error\n                return Ok(());\n            }\n        };\n\n        match client.check_response_end_or_error(false).await {\n            Ok(eos) => {\n                let empty = data.is_empty();\n                if empty && !eos {\n                    /* it is normal to get 0 bytes because of multi-chunk\n                     * don't write 0 bytes to downstream since it will be\n                     * misread as the terminating chunk */\n                    continue;\n                }\n                let body_task = if client.was_upgraded() {\n                    HttpTask::UpgradedBody(Some(data), eos)\n                } else {\n                    HttpTask::Body(Some(data), eos)\n                };\n                let sent = tx\n                    .send(body_task)\n                    .await\n                    .or_err(InternalError, \"sending custom body to pipe\");\n                // If the if the response with content-length is sent to an HTTP1 downstream,\n                // custom_bidirection_down_to_up() could decide that the body has finished and exit without\n                // waiting for this function to signal the eos. In this case tx being closed is not\n                // an sign of error. It should happen if the only thing left for the custom to send is\n                // an empty data frame with eos set.\n                if sent.is_err() && eos && empty {\n                    return Ok(());\n                }\n                sent?;\n            }\n            Err(e) => {\n                // Similar to above, push the error to downstream and then quit\n                let _ = tx.send(HttpTask::Failed(e.into_up())).await;\n                return Ok(());\n            }\n        }\n    }\n\n    // attempt to get trailers\n    let trailers = match client.read_trailers().await {\n        Ok(t) => t,\n        Err(e) => {\n            // Similar to above, push the error to downstream and then quit\n            let _ = tx.send(HttpTask::Failed(e.into_up())).await;\n            return Ok(());\n        }\n    };\n\n    let trailers = trailers.map(Box::new);\n\n    if trailers.is_some() {\n        tx.send(HttpTask::Trailer(trailers))\n            .await\n            .or_err(InternalError, \"sending custom trailer to pipe\")?;\n    }\n\n    tx.send(HttpTask::Done)\n        .await\n        .unwrap_or_else(|_| debug!(\"custom channel closed!\"));\n\n    Ok(())\n}\n\nstruct CustomMessageForwarder<'a> {\n    ctx: ImmutStr,\n    writer: &'a mut Box<dyn CustomMessageWrite>,\n    reader:\n        &'a mut Box<dyn futures::Stream<Item = Result<Bytes, Box<Error>>> + Send + Sync + Unpin>,\n    inject: mpsc::Receiver<Bytes>,\n    filter: mpsc::Sender<(Bytes, oneshot::Sender<Option<Bytes>>)>,\n    cancel: oneshot::Receiver<()>,\n}\n\nimpl CustomMessageForwarder<'_> {\n    async fn proxy(mut self) -> Result<()> {\n        let forwarder = async {\n            let mut injector_status = true;\n            let mut reader_status = true;\n\n            debug!(\"{}: CustomMessageForwarder: start\", self.ctx);\n\n            while injector_status || reader_status {\n                let (data, proxied) = tokio::select! {\n                    ret = self.inject.recv(), if injector_status => {\n                        let Some(data) = ret else {\n                            injector_status = false;\n                            continue\n                        };\n                        (data, false)\n                    },\n\n                    ret = self.reader.next(), if reader_status  => {\n                        let Some(data) = ret else {\n                            reader_status = false;\n                            continue\n                        };\n\n                        let data = match data {\n                            Ok(data) => data,\n                            Err(err) => {\n                                reader_status = false;\n                                warn!(\"{}: CustomMessageForwarder: reader returned err: {err:?}\", self.ctx);\n                                continue;\n                            },\n                        };\n                        (data, true)\n                    },\n                };\n\n                let (callback_tx, callback_rx) = oneshot::channel();\n\n                // If data received from proxy send it to filter\n                if proxied {\n                    if self.filter.send((data, callback_tx)).await.is_err() {\n                        debug!(\n                            \"{}: CustomMessageForwarder: filter receiver dropped\",\n                            self.ctx\n                        );\n                        return Error::e_explain(\n                            WriteError,\n                            \"CustomMessageForwarder: main proxy thread exited on filter send\",\n                        );\n                    };\n                } else {\n                    callback_tx\n                        .send(Some(data))\n                        .expect(\"sending from the same thread\");\n                }\n\n                match callback_rx.await {\n                    Ok(None) => continue, // message was filtered\n                    Ok(Some(msg)) => {\n                        self.writer.write_custom_message(msg).await?;\n                    }\n                    Err(err) => {\n                        debug!(\n                            \"{}: CustomMessageForwarder: callback_rx return error: {err}\",\n                            self.ctx\n                        );\n                        return Error::e_because(\n                            WriteError,\n                            \"CustomMessageForwarder: main proxy thread exited on callback_rx await\",\n                            err,\n                        );\n                    }\n                };\n            }\n\n            debug!(\"{}: CustomMessageForwarder: exit loop\", self.ctx);\n\n            let ret = self.writer.finish_custom().await;\n            if let Err(ref err) = ret {\n                debug!(\n                    \"{}: CustomMessageForwarder: finish_custom return error: {err}\",\n                    self.ctx\n                );\n            };\n            ret?;\n\n            debug!(\n                \"{}: CustomMessageForwarder: exit loop successfully\",\n                self.ctx\n            );\n\n            Ok(())\n        };\n\n        tokio::select! {\n            ret = &mut self.cancel => {\n                debug!(\"{}: CustomMessageForwarder: canceled while waiting for new messages: {ret:?}\", self.ctx);\n                Ok(())\n            },\n            ret = forwarder => ret\n        }\n    }\n}\n"
  },
  {
    "path": "pingora-proxy/src/proxy_h1.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\nuse futures::future::OptionFuture;\nuse futures::StreamExt;\n\nuse super::*;\nuse crate::proxy_cache::{range_filter::RangeBodyFilter, ServeFromCache};\nuse crate::proxy_common::*;\nuse pingora_cache::CachePhase;\nuse pingora_core::protocols::http::custom::CUSTOM_MESSAGE_QUEUE_SIZE;\n\nimpl<SV, C> HttpProxy<SV, C>\nwhere\n    C: custom::Connector,\n{\n    pub(crate) async fn proxy_1to1(\n        &self,\n        session: &mut Session,\n        client_session: &mut HttpSessionV1,\n        peer: &HttpPeer,\n        ctx: &mut SV::CTX,\n    ) -> (bool, bool, Option<Box<Error>>)\n    where\n        SV: ProxyHttp + Send + Sync,\n        SV::CTX: Send + Sync,\n    {\n        client_session.read_timeout = peer.options.read_timeout;\n        client_session.write_timeout = peer.options.write_timeout;\n\n        // phase 2 send to upstream\n\n        let mut req = session.req_header().clone();\n\n        // Convert HTTP2 headers to H1\n        if req.version == Version::HTTP_2 {\n            req.set_version(Version::HTTP_11);\n            // if client has body but has no content length, add chunked encoding\n            // https://datatracker.ietf.org/doc/html/rfc9112#name-message-body\n            // \"The presence of a message body in a request is signaled by a Content-Length or Transfer-Encoding header field.\"\n            if !session.is_body_empty() && session.get_header(header::CONTENT_LENGTH).is_none() {\n                req.insert_header(header::TRANSFER_ENCODING, \"chunked\")\n                    .unwrap();\n            }\n            if session.get_header(header::HOST).is_none() {\n                // H2 is required to set :authority, but no necessarily header\n                // most H1 server expect host header, so convert\n                let host = req.uri.authority().map_or(\"\", |a| a.as_str()).to_owned();\n                req.insert_header(header::HOST, host).unwrap();\n            }\n            // TODO: Add keepalive header for connection reuse, but this is not required per RFC\n        }\n\n        if session.cache.enabled() {\n            pingora_cache::filters::upstream::request_filter(\n                &mut req,\n                session.cache.maybe_cache_meta(),\n            );\n            session.mark_upstream_headers_mutated_for_cache();\n        }\n\n        match self\n            .inner\n            .upstream_request_filter(session, &mut req, ctx)\n            .await\n        {\n            Ok(_) => { /* continue */ }\n            Err(e) => {\n                return (false, true, Some(e));\n            }\n        }\n\n        session.upstream_compression.request_filter(&req);\n\n        debug!(\"Sending header to upstream {:?}\", req);\n\n        match client_session.write_request_header(Box::new(req)).await {\n            Ok(_) => { /* Continue */ }\n            Err(e) => {\n                return (false, false, Some(e.into_up()));\n            }\n        }\n\n        let mut downstream_custom_message_writer = session\n            .downstream_session\n            .as_custom_mut()\n            .and_then(|c| c.take_custom_message_writer());\n\n        let (tx_upstream, rx_upstream) = mpsc::channel::<HttpTask>(TASK_BUFFER_SIZE);\n        let (tx_downstream, rx_downstream) = mpsc::channel::<HttpTask>(TASK_BUFFER_SIZE);\n\n        session.as_mut().enable_retry_buffering();\n\n        // start bi-directional streaming\n        let ret = tokio::try_join!(\n            self.proxy_handle_downstream(\n                session,\n                tx_downstream,\n                rx_upstream,\n                ctx,\n                &mut downstream_custom_message_writer\n            ),\n            self.proxy_handle_upstream(client_session, tx_upstream, rx_downstream),\n        );\n\n        if let Some(custom_session) = session.downstream_session.as_custom_mut() {\n            if let Some(downstream_custom_message_writer) = downstream_custom_message_writer {\n                match custom_session.restore_custom_message_writer(downstream_custom_message_writer)\n                {\n                    Ok(_) => { /* continue */ }\n                    Err(e) => {\n                        return (false, false, Some(e));\n                    }\n                }\n            }\n        }\n\n        match ret {\n            Ok((downstream_can_reuse, _upstream)) => (downstream_can_reuse, true, None),\n            Err(e) => (false, false, Some(e)),\n        }\n    }\n\n    pub(crate) async fn proxy_to_h1_upstream(\n        &self,\n        session: &mut Session,\n        client_session: &mut HttpSessionV1,\n        reused: bool,\n        peer: &HttpPeer,\n        ctx: &mut SV::CTX,\n    ) -> (bool, bool, Option<Box<Error>>)\n    // (reuse_server, reuse_client, error)\n    where\n        SV: ProxyHttp + Send + Sync,\n        SV::CTX: Send + Sync,\n    {\n        #[cfg(windows)]\n        let raw = client_session.id() as std::os::windows::io::RawSocket;\n        #[cfg(unix)]\n        let raw = client_session.id();\n\n        let initial_write_pending = client_session.stream().get_write_pending_time();\n\n        if let Err(e) = self\n            .inner\n            .connected_to_upstream(\n                session,\n                reused,\n                peer,\n                raw,\n                Some(client_session.digest()),\n                ctx,\n            )\n            .await\n        {\n            return (false, false, Some(e));\n        }\n\n        let (server_session_reuse, client_session_reuse, error) =\n            self.proxy_1to1(session, client_session, peer, ctx).await;\n\n        // Record upstream response body bytes received (payload only) for logging consumers.\n        let upstream_bytes_total = client_session.body_bytes_received();\n        session.set_upstream_body_bytes_received(upstream_bytes_total);\n\n        // Record upstream write pending time for this session only (delta from baseline).\n        let current_write_pending = client_session.stream().get_write_pending_time();\n        let upstream_write_pending = current_write_pending.saturating_sub(initial_write_pending);\n        session.set_upstream_write_pending_time(upstream_write_pending);\n\n        (server_session_reuse, client_session_reuse, error)\n    }\n\n    async fn proxy_handle_upstream(\n        &self,\n        client_session: &mut HttpSessionV1,\n        tx: mpsc::Sender<HttpTask>,\n        mut rx: mpsc::Receiver<HttpTask>,\n    ) -> Result<()>\n    where\n        SV: ProxyHttp + Send + Sync,\n        SV::CTX: Send + Sync,\n    {\n        let mut request_done = false;\n        let mut response_done = false;\n        let mut send_error = None;\n        let mut upgraded = false;\n\n        /* duplex mode, wait for either to complete */\n        while !request_done || !response_done {\n            tokio::select! {\n                res = client_session.read_response_task(), if !response_done => {\n                    match res {\n                        Ok(task) => {\n                            response_done = task.is_end();\n                            if !upgraded && client_session.was_upgraded() {\n                                // upgrade can only happen once\n                                upgraded = true;\n                                if send_error.is_none() {\n                                    // continue receiving from downstream after body mode change\n                                    request_done = false;\n                                }\n                            }\n                            let type_str = task.type_str();\n                            let result = tx.send(task)\n                                .await.or_err_with(\n                                    InternalError,\n                                    || format!(\"Failed to send upstream task {type_str}{} to pipe\",\n                                        if response_done { \" (end)\" } else {\"\"})\n                                );\n                            // If the request is upgraded, the downstream pipe can early exit\n                            // when the downstream connection is closed.\n                            // In that case, this function should ignore that the pipe is closed.\n                            // So that this function could read the rest events from rx including\n                            // the closure, then exit.\n                            if result.is_err() && !client_session.was_upgraded() {\n                                return result;\n                            }\n                        },\n                        Err(e) => {\n                            // Push the error to downstream and then quit\n                            // Don't care if send fails: downstream already gone\n                            let _ = tx.send(HttpTask::Failed(send_error.unwrap_or(e).into_up())).await;\n                            // Downstream should consume all remaining data and handle the error\n                            return Ok(())\n                        }\n                    }\n                },\n\n                body = rx.recv(), if !request_done => {\n                    match send_body_to1(client_session, body).await {\n                        Ok(send_done) => {\n                            request_done = send_done;\n                            // An upgraded request is terminated when either side is done\n                            if request_done && client_session.was_upgraded() {\n                                response_done = true;\n                            }\n                        },\n                        Err(e) => {\n                           warn!(\"send error, draining read buf: {e}\");\n                           request_done = true;\n\n                           send_error = Some(e);\n                           continue\n                        }\n                    }\n                },\n\n                else => {\n                    // this shouldn't be reached as the while loop would already exit\n                    break;\n                }\n            }\n        }\n\n        Ok(())\n    }\n\n    // todo use this function to replace bidirection_1to2()\n    // returns whether this server (downstream) session can be reused\n    async fn proxy_handle_downstream(\n        &self,\n        session: &mut Session,\n        tx: mpsc::Sender<HttpTask>,\n        mut rx: mpsc::Receiver<HttpTask>,\n        ctx: &mut SV::CTX,\n        downstream_custom_message_writer: &mut Option<Box<dyn CustomMessageWrite>>,\n    ) -> Result<bool>\n    where\n        SV: ProxyHttp + Send + Sync,\n        SV::CTX: Send + Sync,\n    {\n        // setup custom message forwarding, if downstream supports it\n        let (\n            mut downstream_custom_read,\n            mut downstream_custom_write,\n            downstream_custom_message_custom_forwarding,\n            mut downstream_custom_message_inject_rx,\n            mut downstream_custom_message_reader,\n        ) = if downstream_custom_message_writer.is_some() {\n            let reader = session.downstream_custom_message()?;\n            let (inject_tx, inject_rx) = mpsc::channel::<Bytes>(CUSTOM_MESSAGE_QUEUE_SIZE);\n            (true, true, Some(inject_tx), Some(inject_rx), reader)\n        } else {\n            (false, false, None, None, None)\n        };\n\n        if let Some(custom_forwarding) = downstream_custom_message_custom_forwarding {\n            self.inner\n                .custom_forwarding(session, ctx, None, custom_forwarding)\n                .await?;\n        }\n\n        let mut downstream_state = DownstreamStateMachine::new(session.as_mut().is_body_done());\n\n        let buffer = session.as_ref().get_retry_buffer();\n\n        // retry, send buffer if it exists or body empty\n        if buffer.is_some() || session.as_mut().is_body_empty() {\n            let send_permit = tx\n                .reserve()\n                .await\n                .or_err(InternalError, \"reserving body pipe\")?;\n            self.send_body_to_pipe(\n                session,\n                buffer,\n                downstream_state.is_done(),\n                send_permit,\n                ctx,\n            )\n            .await?;\n        }\n\n        let mut response_state = ResponseStateMachine::new();\n\n        // these two below can be wrapped into an internal ctx\n        // use cache when upstream revalidates (or TODO: error)\n        let mut serve_from_cache = proxy_cache::ServeFromCache::new();\n        let mut range_body_filter = proxy_cache::range_filter::RangeBodyFilter::new();\n\n        /* duplex mode without caching\n         * Read body from downstream while reading response from upstream\n         * If response is done, only read body from downstream\n         * If request is done, read response from upstream while idling downstream (to close quickly)\n         * If both are done, quit the loop\n         *\n         * With caching + but without partial read support\n         * Similar to above, cache admission write happen when the data is write to downstream\n         *\n         * With caching + partial read support\n         * A. Read upstream response and write to cache\n         * B. Read data from cache and send to downstream\n         * If B fails (usually downstream close), continue A.\n         * If A fails, exit with error.\n         * If both are done, quit the loop\n         * Usually there is no request body to read for cacheable request\n         */\n        while !downstream_state.is_done()\n            || !response_state.is_done()\n            || downstream_custom_read && !downstream_state.is_errored()\n            || downstream_custom_write\n        {\n            // reserve tx capacity ahead to avoid deadlock, see below\n\n            let send_permit = tx\n                .try_reserve()\n                .or_err(InternalError, \"try_reserve() body pipe for upstream\");\n\n            // Use optional futures to allow using optional channels in select branches\n            let custom_inject_rx_recv: OptionFuture<_> = downstream_custom_message_inject_rx\n                .as_mut()\n                .map(|rx| rx.recv())\n                .into();\n            let custom_reader_next: OptionFuture<_> = downstream_custom_message_reader\n                .as_mut()\n                .map(|reader| reader.next())\n                .into();\n\n            // partial read support, this check will also be false if cache is disabled.\n            let support_cache_partial_read =\n                session.cache.support_streaming_partial_write() == Some(true);\n            let upgraded = session.was_upgraded();\n\n            tokio::select! {\n                // only try to send to pipe if there is capacity to avoid deadlock\n                // Otherwise deadlock could happen if both upstream and downstream are blocked\n                // on sending to their corresponding pipes which are both full.\n                body = session.downstream_session.read_body_or_idle(downstream_state.is_done()),\n                    if downstream_state.can_poll() && send_permit.is_ok() => {\n\n                    debug!(\"downstream event\");\n                    let body = match body {\n                        Ok(b) => b,\n                        Err(e) => {\n                            let wait_for_cache_fill = (!serve_from_cache.is_on() && support_cache_partial_read)\n                                || serve_from_cache.is_miss();\n                            if wait_for_cache_fill {\n                                // ignore downstream error so that upstream can continue to write cache\n                                downstream_state.to_errored();\n                                warn!(\n                                    \"Downstream Error ignored during caching: {}, {}\",\n                                    e,\n                                    self.inner.request_summary(session, ctx)\n                                );\n                                // This will not be treated as a final error, but we should signal to\n                                // downstream session regardless\n                                session.downstream_session.on_proxy_failure(e);\n                                continue;\n                           } else {\n                                return Err(e.into_down());\n                           }\n                        }\n                    };\n                    // If the request is websocket, `None` body means the request is closed.\n                    // Set the response to be done as well so that the request completes normally.\n                    if body.is_none() && session.was_upgraded() {\n                        response_state.maybe_set_upstream_done(true);\n                    }\n                    // TODO: consider just drain this if serve_from_cache is set\n                    let is_body_done = session.is_body_done();\n                    let request_done = self.send_body_to_pipe(\n                        session,\n                        body,\n                        is_body_done,\n                        send_permit.unwrap(), // safe because we checked is_ok()\n                        ctx,\n                    )\n                    .await?;\n                    downstream_state.maybe_finished(request_done);\n                },\n\n                _ = tx.reserve(), if downstream_state.is_reading() && send_permit.is_err() => {\n                    // If tx is closed, the upstream has already finished its job.\n                    downstream_state.maybe_finished(tx.is_closed());\n                    debug!(\"waiting for permit {send_permit:?}, upstream closed {}\", tx.is_closed());\n                    /* No permit, wait on more capacity to avoid starving.\n                     * Otherwise this select only blocks on rx, which might send no data\n                     * before the entire body is uploaded.\n                     * once more capacity arrives we just loop back\n                     */\n                },\n\n                task = rx.recv(), if !response_state.upstream_done() => {\n                    debug!(\"upstream event: {:?}\", task);\n                    if let Some(t) = task {\n                        if serve_from_cache.should_discard_upstream() {\n                            // just drain, do we need to do anything else?\n                           continue;\n                        }\n                        // pull as many tasks as we can\n                        let mut tasks = Vec::with_capacity(TASK_BUFFER_SIZE);\n                        tasks.push(t);\n                        // tokio::task::unconstrained because now_or_never may yield None when the future is ready\n                        while let Some(maybe_task) = tokio::task::unconstrained(rx.recv()).now_or_never() {\n                            debug!(\"upstream event now: {:?}\", maybe_task);\n                            if let Some(t) = maybe_task {\n                                tasks.push(t);\n                            } else {\n                                break; // upstream closed\n                            }\n                        }\n\n                        /* run filters before sending to downstream */\n                        let mut filtered_tasks = Vec::with_capacity(TASK_BUFFER_SIZE);\n                        for mut t in tasks {\n                            if self.revalidate_or_stale(session, &mut t, ctx).await {\n                                serve_from_cache.enable();\n                                response_state.enable_cached_response();\n                                // skip downstream filtering entirely as the 304 will not be sent\n                                break;\n                            }\n                            session.upstream_compression.response_filter(&mut t);\n                            let task = self.h1_response_filter(session, t, ctx,\n                                &mut serve_from_cache,\n                                &mut range_body_filter, false).await?;\n                            if serve_from_cache.is_miss_header() {\n                                response_state.enable_cached_response();\n                            }\n                            // check error and abort\n                            // otherwise the error is surfaced via write_response_tasks()\n                            if !serve_from_cache.should_send_to_downstream() {\n                                if let HttpTask::Failed(e) = task {\n                                    return Err(e);\n                                }\n                            }\n                            filtered_tasks.push(task);\n                        }\n\n                        if !serve_from_cache.should_send_to_downstream() {\n                            // TODO: need to derive response_done from filtered_tasks in case downstream failed already\n                            continue;\n                        }\n\n                        // set to downstream\n                        let upgraded = session.was_upgraded();\n                        let response_done = session.write_response_tasks(filtered_tasks).await?;\n                        if !upgraded && session.was_upgraded() && downstream_state.can_poll() {\n                            // just upgraded, the downstream state should be reset to continue to\n                            // poll body\n                            trace!(\"reset downstream state on upgrade\");\n                            downstream_state.reset();\n                        }\n                        response_state.maybe_set_upstream_done(response_done);\n                        // unsuccessful upgrade response (or end of upstream upgraded conn,\n                        // which forces the body reader to complete) may force the request done\n                        downstream_state.maybe_finished(session.is_body_done());\n                    } else {\n                        debug!(\"empty upstream event\");\n                        response_state.maybe_set_upstream_done(true);\n                    }\n                },\n\n                task = serve_from_cache.next_http_task(&mut session.cache, &mut range_body_filter, upgraded),\n                    if !response_state.cached_done() && !downstream_state.is_errored() && serve_from_cache.is_on() => {\n\n                    let task = self.h1_response_filter(session, task?, ctx,\n                        &mut serve_from_cache,\n                        &mut range_body_filter, true).await?;\n                    debug!(\"serve_from_cache task {task:?}\");\n\n                    match session.write_response_tasks(vec![task]).await {\n                        Ok(b) => response_state.maybe_set_cache_done(b),\n                        Err(e) => if serve_from_cache.is_miss() {\n                            // give up writing to downstream but wait for upstream cache write to finish\n                            downstream_state.to_errored();\n                            response_state.maybe_set_cache_done(true);\n                            warn!(\n                                \"Downstream Error ignored during caching: {}, {}\",\n                                e,\n                                self.inner.request_summary(session, ctx)\n                            );\n                            // This will not be treated as a final error, but we should signal to\n                            // downstream session regardless\n                            session.downstream_session.on_proxy_failure(e);\n                            continue;\n                        } else {\n                            return Err(e);\n                        }\n                    }\n                    if response_state.cached_done() {\n                        if let Err(e) = session.cache.finish_hit_handler().await {\n                            warn!(\"Error during finish_hit_handler: {}\", e);\n                        }\n                    }\n                }\n\n                data = custom_reader_next, if downstream_custom_read && !downstream_state.is_errored()  => {\n                    let Some(data) = data.flatten() else {\n                        downstream_custom_read = false;\n                        continue;\n                    };\n\n                    let data = match data {\n                        Ok(data) => data,\n                        Err(err) =>  {\n                            warn!(\"downstream_custom_message_reader got error: {err}\");\n                            downstream_custom_read = false;\n                            continue;\n                        },\n                    };\n\n                    self.inner\n                        .downstream_custom_message_proxy_filter(session, data, ctx, true) // true, because it's the last hop for downstream proxying\n                        .await?;\n                },\n\n                data = custom_inject_rx_recv, if downstream_custom_write => {\n                    match data.flatten() {\n                        Some(data) => {\n                            if let Some(ref mut custom_writer) = downstream_custom_message_writer {\n                                custom_writer.write_custom_message(data).await?\n                            }\n                        },\n                        None => {\n                            downstream_custom_write = false;\n                            if let Some(ref mut custom_writer) = downstream_custom_message_writer {\n                                custom_writer.finish_custom().await?;\n                            }\n                        },\n                    }\n                },\n\n                else => {\n                    break;\n                }\n            }\n        }\n\n        if let Some(custom_session) = session.downstream_session.as_custom_mut() {\n            if let Some(downstream_custom_message_reader) = downstream_custom_message_reader {\n                custom_session\n                    .restore_custom_message_reader(downstream_custom_message_reader)\n                    .expect(\"downstream restore_custom_message_reader should be empty\");\n            }\n        }\n\n        let mut reuse_downstream = !downstream_state.is_errored();\n        if reuse_downstream {\n            match session.as_mut().finish_body().await {\n                Ok(_) => {\n                    debug!(\"finished sending body to downstream\");\n                }\n                Err(e) => {\n                    error!(\"Error finish sending body to downstream: {}\", e);\n                    reuse_downstream = false;\n                }\n            }\n        }\n        Ok(reuse_downstream)\n    }\n\n    async fn h1_response_filter(\n        &self,\n        session: &mut Session,\n        mut task: HttpTask,\n        ctx: &mut SV::CTX,\n        serve_from_cache: &mut ServeFromCache,\n        range_body_filter: &mut RangeBodyFilter,\n        from_cache: bool, // are the task from cache already\n    ) -> Result<HttpTask>\n    where\n        SV: ProxyHttp + Send + Sync,\n        SV::CTX: Send + Sync,\n    {\n        // skip caching if already served from cache\n        if !from_cache {\n            if let Some(duration) = self.upstream_filter(session, &mut task, ctx).await? {\n                trace!(\"delaying upstream response for {duration:?}\");\n                time::sleep(duration).await;\n            }\n\n            // cache the original response before any downstream transformation\n            // requests that bypassed cache still need to run filters to see if the response has become cacheable\n            if session.cache.enabled() || session.cache.bypassing() {\n                if let Err(e) = self\n                    .cache_http_task(session, &task, ctx, serve_from_cache)\n                    .await\n                {\n                    session.cache.disable(NoCacheReason::StorageError);\n                    if serve_from_cache.is_miss_body() {\n                        // if the response stream cache body during miss but write fails, it has to\n                        // give up the entire request\n                        return Err(e);\n                    } else {\n                        // otherwise, continue processing the response\n                        warn!(\n                            \"Fail to cache response: {}, {}\",\n                            e,\n                            self.inner.request_summary(session, ctx)\n                        );\n                    }\n                }\n            }\n\n            if !serve_from_cache.should_send_to_downstream() {\n                return Ok(task);\n            }\n        } // else: cached/local response, no need to trigger upstream filters and caching\n\n        // normally max file size is tracked in cache_http_task filters (when cache enabled),\n        // we will track it in these filters before sending to downstream on specific conditions\n        // when cache is disabled\n        let track_max_cache_size = matches!(\n            session.cache.phase(),\n            CachePhase::Disabled(NoCacheReason::PredictedResponseTooLarge)\n        );\n\n        let res = match task {\n            HttpTask::Header(mut header, end) => {\n                /* Downstream revalidation/range, only needed when cache modified headers because otherwise origin\n                 * will handle it */\n                if session.upstream_headers_mutated_for_cache() {\n                    self.downstream_response_conditional_filter(\n                        serve_from_cache,\n                        session,\n                        &mut header,\n                        ctx,\n                    );\n                    if !session.ignore_downstream_range {\n                        let range_type = self.inner.range_header_filter(session, &mut header, ctx);\n                        range_body_filter.set(range_type);\n                    }\n                }\n\n                // TODO: just set version to Version::HTTP_11 unconditionally here,\n                // (with another todo being an option to faithfully proxy the <1.1 responses)\n                // as we are already trying to mutate this for HTTP/1.1 downstream reuse\n\n                /* Convert HTTP 1.0 style response to chunked encoding so that we don't\n                 * have to close the downstream connection */\n                // these status codes / method cannot have body, so no need to add chunked encoding\n                let no_body = session.req_header().method == http::method::Method::HEAD\n                    || matches!(header.status.as_u16(), 204 | 304);\n                if !no_body\n                    && !header.status.is_informational()\n                    && header\n                        .headers\n                        .get(http::header::TRANSFER_ENCODING)\n                        .is_none()\n                    && header.headers.get(http::header::CONTENT_LENGTH).is_none()\n                    && !end\n                {\n                    // Upgrade the http version to 1.1 because 1.0/0.9 doesn't support chunked\n                    header.set_version(Version::HTTP_11);\n                    header.insert_header(http::header::TRANSFER_ENCODING, \"chunked\")?;\n                }\n\n                match self.inner.response_filter(session, &mut header, ctx).await {\n                    Ok(_) => Ok(HttpTask::Header(header, end)),\n                    Err(e) => Err(e),\n                }\n            }\n            HttpTask::Body(data, end) => {\n                if track_max_cache_size {\n                    session\n                        .cache\n                        .track_body_bytes_for_max_file_size(data.as_ref().map_or(0, |d| d.len()));\n                }\n\n                // before it can mark it as cacheable again.\n                let mut data = range_body_filter.filter_body(data);\n                if let Some(duration) = self\n                    .inner\n                    .response_body_filter(session, &mut data, end, ctx)?\n                {\n                    trace!(\"delaying downstream response for {:?}\", duration);\n                    time::sleep(duration).await;\n                }\n\n                Ok(HttpTask::Body(data, end))\n            }\n            HttpTask::UpgradedBody(mut data, end) => {\n                if track_max_cache_size {\n                    session\n                        .cache\n                        .track_body_bytes_for_max_file_size(data.as_ref().map_or(0, |d| d.len()));\n                }\n\n                // range doesn't apply to upgraded body\n                if let Some(duration) = self\n                    .inner\n                    .response_body_filter(session, &mut data, end, ctx)?\n                {\n                    trace!(\"delaying downstream upgraded response for {:?}\", duration);\n                    time::sleep(duration).await;\n                }\n\n                Ok(HttpTask::UpgradedBody(data, end))\n            }\n            HttpTask::Trailer(h) => Ok(HttpTask::Trailer(h)), // TODO: support trailers for h1\n            HttpTask::Done => Ok(task),\n            HttpTask::Failed(_) => Ok(task), // Do nothing just pass the error down\n        };\n        // On end, check if the response (based on file size) can be considered cacheable again\n        if let Ok(task) = res.as_ref() {\n            if track_max_cache_size\n                && task.is_end()\n                && !matches!(task, HttpTask::Failed(_))\n                && !session.cache.exceeded_max_file_size()\n            {\n                session.cache.response_became_cacheable();\n            }\n        }\n        res\n    }\n\n    // TODO:: use this function to replace send_body_to2\n    async fn send_body_to_pipe(\n        &self,\n        session: &mut Session,\n        mut data: Option<Bytes>,\n        end_of_body: bool,\n        tx: mpsc::Permit<'_, HttpTask>,\n        ctx: &mut SV::CTX,\n    ) -> Result<bool>\n    where\n        SV: ProxyHttp + Send + Sync,\n        SV::CTX: Send + Sync,\n    {\n        // None: end of body\n        // this var is to signal if downstream finish sending the body, which shouldn't be\n        // affected by the request_body_filter\n        let end_of_body = end_of_body || data.is_none();\n\n        session\n            .downstream_modules_ctx\n            .request_body_filter(&mut data, end_of_body)\n            .await?;\n\n        // TODO: request body filter to have info about upgraded status?\n        // (can also check session.was_upgraded())\n        self.inner\n            .request_body_filter(session, &mut data, end_of_body, ctx)\n            .await?;\n\n        // the flag to signal to upstream\n        let upstream_end_of_body = end_of_body || data.is_none();\n\n        /* It is normal to get 0 bytes because of multi-chunk or request_body_filter decides not to\n         * output anything yet.\n         * Don't write 0 bytes to the network since it will be\n         * treated as the terminating chunk */\n        if !upstream_end_of_body && data.as_ref().is_some_and(|d| d.is_empty()) {\n            return Ok(false);\n        }\n\n        debug!(\n            \"Read {} bytes body from downstream\",\n            data.as_ref().map_or(-1, |d| d.len() as isize)\n        );\n\n        // upgraded body needs to be marked\n        if session.was_upgraded() {\n            tx.send(HttpTask::UpgradedBody(data, upstream_end_of_body));\n        } else {\n            tx.send(HttpTask::Body(data, upstream_end_of_body));\n        }\n\n        Ok(end_of_body)\n    }\n}\n\npub(crate) async fn send_body_to1(\n    client_session: &mut HttpSessionV1,\n    recv_task: Option<HttpTask>,\n) -> Result<bool> {\n    let body_done;\n\n    if let Some(task) = recv_task {\n        match task {\n            HttpTask::Body(data, end) => {\n                body_done = end;\n                if let Some(d) = data {\n                    let m = client_session.write_body(&d).await;\n                    match m {\n                        Ok(m) => match m {\n                            Some(n) => {\n                                debug!(\"Write {} bytes body to upstream\", n);\n                            }\n                            None => {\n                                warn!(\"Upstream body is already finished. Nothing to write\");\n                            }\n                        },\n                        Err(e) => {\n                            return e.into_up().into_err();\n                        }\n                    }\n                }\n            }\n            HttpTask::UpgradedBody(data, end) => {\n                client_session.maybe_upgrade_body_writer();\n\n                body_done = end;\n                if let Some(d) = data {\n                    let m = client_session.write_body(&d).await;\n                    match m {\n                        Ok(m) => {\n                            match m {\n                                Some(n) => {\n                                    debug!(\"Write {} bytes upgraded body to upstream\", n);\n                                }\n                                None => {\n                                    warn!(\"Upstream upgraded body is already finished. Nothing to write\");\n                                }\n                            }\n                        }\n                        Err(e) => {\n                            return e.into_up().into_err();\n                        }\n                    }\n                }\n            }\n            _ => {\n                // should never happen, sender only sends body\n                warn!(\"Unexpected task sent to upstream\");\n                body_done = true;\n                // error here,\n                // for client sessions that received upgrade but didn't\n                // receive any UpgradedBody,\n                // no more data is arriving so we should consider this\n                // as downstream finalizing its upgrade payload\n                client_session.maybe_upgrade_body_writer();\n            }\n        }\n    } else {\n        // sender dropped\n        body_done = true;\n        // for client sessions that received upgrade but didn't\n        // receive any UpgradedBody,\n        // no more data is arriving so we should consider this\n        // as downstream finalizing its upgrade payload\n        client_session.maybe_upgrade_body_writer();\n    }\n\n    if body_done {\n        match client_session.finish_body().await {\n            Ok(_) => {\n                debug!(\"finish sending body to upstream\");\n                Ok(true)\n            }\n            Err(e) => e.into_up().into_err(),\n        }\n    } else {\n        Ok(false)\n    }\n}\n"
  },
  {
    "path": "pingora-proxy/src/proxy_h2.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\nuse futures::future::OptionFuture;\nuse futures::StreamExt;\n\nuse super::*;\nuse crate::proxy_cache::{range_filter::RangeBodyFilter, ServeFromCache};\nuse crate::proxy_common::*;\nuse http::{header::CONTENT_LENGTH, Method, StatusCode};\nuse pingora_cache::CachePhase;\nuse pingora_core::protocols::http::custom::CUSTOM_MESSAGE_QUEUE_SIZE;\nuse pingora_core::protocols::http::v2::{client::Http2Session, write_body};\n\n// add scheme and authority as required by h2 lib\nfn update_h2_scheme_authority(\n    header: &mut http::request::Parts,\n    raw_host: &[u8],\n    tls: bool,\n) -> Result<()> {\n    let authority = if let Ok(s) = std::str::from_utf8(raw_host) {\n        if s.starts_with('[') {\n            // don't mess with ipv6 host\n            s\n        } else if let Some(colon) = s.find(':') {\n            if s.len() == colon + 1 {\n                // colon is the last char, ignore\n                s\n            } else if let Some(another_colon) = s[colon + 1..].find(':') {\n                // try to get rid of extra port numbers\n                &s[..colon + 1 + another_colon]\n            } else {\n                s\n            }\n        } else {\n            s\n        }\n    } else {\n        return Error::e_explain(\n            InvalidHTTPHeader,\n            format!(\"invalid authority from host {:?}\", raw_host),\n        );\n    };\n\n    let scheme = if tls { \"https\" } else { \"http\" };\n    let uri = http::uri::Builder::new()\n        .scheme(scheme)\n        .authority(authority)\n        .path_and_query(header.uri.path_and_query().as_ref().unwrap().as_str())\n        .build();\n    match uri {\n        Ok(uri) => {\n            header.uri = uri;\n            Ok(())\n        }\n        Err(_) => Error::e_explain(\n            InvalidHTTPHeader,\n            format!(\"invalid authority from host {}\", authority),\n        ),\n    }\n}\n\nimpl<SV, C> HttpProxy<SV, C>\nwhere\n    C: custom::Connector,\n{\n    pub(crate) async fn proxy_down_to_up(\n        &self,\n        session: &mut Session,\n        client_session: &mut Http2Session,\n        peer: &HttpPeer,\n        ctx: &mut SV::CTX,\n    ) -> (bool, Option<Box<Error>>)\n    // (reuse_server, error)\n    where\n        SV: ProxyHttp + Send + Sync,\n        SV::CTX: Send + Sync,\n    {\n        let mut req = session.req_header().clone();\n\n        if req.version != Version::HTTP_2 {\n            /* remove H1 specific headers */\n            // https://github.com/hyperium/h2/blob/d3b9f1e36aadc1a7a6804e2f8e86d3fe4a244b4f/src/proto/streams/send.rs#L72\n            req.remove_header(&http::header::TRANSFER_ENCODING);\n            req.remove_header(&http::header::CONNECTION);\n            req.remove_header(&http::header::UPGRADE);\n            req.remove_header(\"keep-alive\");\n            req.remove_header(\"proxy-connection\");\n        }\n\n        /* turn it into h2 */\n        req.set_version(Version::HTTP_2);\n\n        if session.cache.enabled() {\n            pingora_cache::filters::upstream::request_filter(\n                &mut req,\n                session.cache.maybe_cache_meta(),\n            );\n            session.mark_upstream_headers_mutated_for_cache();\n        }\n\n        match self\n            .inner\n            .upstream_request_filter(session, &mut req, ctx)\n            .await\n        {\n            Ok(_) => { /* continue */ }\n            Err(e) => {\n                return (false, Some(e));\n            }\n        }\n\n        // Remove H1 `Host` header, save it in order to add to :authority\n        // We do this because certain H2 servers expect request not to have a host header.\n        // The `Host` is removed after the upstream filters above for 2 reasons\n        // 1. there is no API to change the :authority header\n        // 2. the filter code needs to be aware of the host vs :authority across http versions otherwise\n        let host = req.remove_header(&http::header::HOST);\n\n        session.upstream_compression.request_filter(&req);\n        let body_empty = session.as_mut().is_body_empty();\n\n        // whether we support sending END_STREAM on HEADERS if body is empty\n        let send_end_stream = req.send_end_stream().expect(\"req must be h2\");\n\n        let mut req: http::request::Parts = req.into();\n\n        // H2 requires authority to be set, so copy that from H1 host if that is set\n        if let Some(host) = host {\n            if let Err(e) = update_h2_scheme_authority(&mut req, host.as_bytes(), peer.is_tls()) {\n                return (false, Some(e));\n            }\n        }\n\n        debug!(\"Request to h2: {req:?}\");\n\n        // send END_STREAM on HEADERS\n        let send_header_eos = send_end_stream && body_empty;\n        debug!(\"send END_STREAM on HEADERS: {send_end_stream}\");\n\n        let req = Box::new(RequestHeader::from(req));\n        if let Err(e) = client_session.write_request_header(req, send_header_eos) {\n            return (false, Some(e.into_up()));\n        }\n\n        if !send_end_stream && body_empty {\n            // send END_STREAM on empty DATA frame\n            match client_session.write_request_body(Bytes::new(), true).await {\n                Ok(()) => debug!(\"sent empty DATA frame to h2\"),\n                Err(e) => {\n                    return (false, Some(e.into_up()));\n                }\n            }\n        }\n\n        client_session.read_timeout = peer.options.read_timeout;\n\n        let mut downstream_custom_message_writer = session\n            .downstream_session\n            .as_custom_mut()\n            .and_then(|c| c.take_custom_message_writer());\n\n        // take the body writer out of the client for easy duplex\n        let mut client_body = client_session\n            .take_request_body_writer()\n            .expect(\"already send request header\");\n\n        // need to get the write_timeout here since we pass the h2 SendStream\n        // directly to bidirection_down_to_up\n        let write_timeout = peer.options.write_timeout;\n\n        let (tx, rx) = mpsc::channel::<HttpTask>(TASK_BUFFER_SIZE);\n\n        session.as_mut().enable_retry_buffering();\n\n        /* read downstream body and upstream response at the same time */\n\n        let ret = tokio::try_join!(\n            self.bidirection_down_to_up(\n                session,\n                &mut client_body,\n                rx,\n                ctx,\n                write_timeout,\n                &mut downstream_custom_message_writer\n            ),\n            pipe_up_to_down_response(client_session, tx)\n        );\n\n        if let Some(custom_session) = session.downstream_session.as_custom_mut() {\n            if let Some(downstream_custom_message_writer) = downstream_custom_message_writer {\n                match custom_session.restore_custom_message_writer(downstream_custom_message_writer)\n                {\n                    Ok(_) => { /* continue */ }\n                    Err(e) => {\n                        return (false, Some(e));\n                    }\n                }\n            }\n        }\n\n        match ret {\n            Ok((downstream_can_reuse, _upstream)) => (downstream_can_reuse, None),\n            Err(e) => {\n                // On application level upstream read timeouts, send RST_STREAM CANCEL,\n                // we know we have not received END_STREAM at this point since we read timed out\n                // TODO: implement for write timeouts?\n                if e.esource == ErrorSource::Upstream && matches!(e.etype, ReadTimedout) {\n                    client_body.send_reset(h2::Reason::CANCEL);\n                }\n                (false, Some(e))\n            }\n        }\n    }\n\n    pub(crate) async fn proxy_to_h2_upstream(\n        &self,\n        session: &mut Session,\n        client_session: &mut Http2Session,\n        reused: bool,\n        peer: &HttpPeer,\n        ctx: &mut SV::CTX,\n    ) -> (bool, Option<Box<Error>>)\n    where\n        SV: ProxyHttp + Send + Sync,\n        SV::CTX: Send + Sync,\n    {\n        #[cfg(windows)]\n        let raw = client_session.fd() as std::os::windows::io::RawSocket;\n        #[cfg(unix)]\n        let raw = client_session.fd();\n\n        if let Err(e) = self\n            .inner\n            .connected_to_upstream(session, reused, peer, raw, client_session.digest(), ctx)\n            .await\n        {\n            return (false, Some(e));\n        }\n\n        let (server_session_reuse, error) = self\n            .proxy_down_to_up(session, client_session, peer, ctx)\n            .await;\n\n        // Record upstream response body bytes received (HTTP/2 DATA payload).\n        let upstream_bytes_total = client_session.body_bytes_received();\n        session.set_upstream_body_bytes_received(upstream_bytes_total);\n\n        // Note: upstream_write_pending_time is not tracked for HTTP/2 (multiplexed streams).\n\n        (server_session_reuse, error)\n    }\n\n    // returns whether server (downstream) session can be reused\n    async fn bidirection_down_to_up(\n        &self,\n        session: &mut Session,\n        client_body: &mut h2::SendStream<bytes::Bytes>,\n        mut rx: mpsc::Receiver<HttpTask>,\n        ctx: &mut SV::CTX,\n        write_timeout: Option<Duration>,\n        downstream_custom_message_writer: &mut Option<Box<dyn CustomMessageWrite>>,\n    ) -> Result<bool>\n    where\n        SV: ProxyHttp + Send + Sync,\n        SV::CTX: Send + Sync,\n    {\n        // setup custom message forwarding, if downstream supports it\n        let (\n            mut downstream_custom_read,\n            mut downstream_custom_write,\n            downstream_custom_message_custom_forwarding,\n            mut downstream_custom_message_inject_rx,\n            mut downstream_custom_message_reader,\n        ) = if downstream_custom_message_writer.is_some() {\n            let reader = session.downstream_custom_message()?;\n            let (inject_tx, inject_rx) = mpsc::channel::<Bytes>(CUSTOM_MESSAGE_QUEUE_SIZE);\n            (true, true, Some(inject_tx), Some(inject_rx), reader)\n        } else {\n            (false, false, None, None, None)\n        };\n\n        if let Some(custom_forwarding) = downstream_custom_message_custom_forwarding {\n            self.inner\n                .custom_forwarding(session, ctx, None, custom_forwarding)\n                .await?;\n        }\n\n        let mut downstream_state = DownstreamStateMachine::new(session.as_mut().is_body_done());\n\n        // retry, send buffer if it exists\n        if let Some(buffer) = session.as_mut().get_retry_buffer() {\n            self.send_body_to2(\n                session,\n                Some(buffer),\n                downstream_state.is_done(),\n                client_body,\n                ctx,\n                write_timeout,\n            )\n            .await?;\n        }\n\n        let mut response_state = ResponseStateMachine::new();\n\n        // these two below can be wrapped into an internal ctx\n        // use cache when upstream revalidates (or TODO: error)\n        let mut serve_from_cache = ServeFromCache::new();\n        let mut range_body_filter = proxy_cache::range_filter::RangeBodyFilter::new();\n\n        /* duplex mode\n         * see the Same function for h1 for more comments\n         */\n        while !downstream_state.is_done()\n            || !response_state.is_done()\n            || downstream_custom_read && !downstream_state.is_errored()\n            || downstream_custom_write\n        {\n            // Use optional futures to allow using optional channels in select branches\n            let custom_inject_rx_recv: OptionFuture<_> = downstream_custom_message_inject_rx\n                .as_mut()\n                .map(|rx| rx.recv())\n                .into();\n            let custom_reader_next: OptionFuture<_> = downstream_custom_message_reader\n                .as_mut()\n                .map(|reader| reader.next())\n                .into();\n\n            // partial read support, this check will also be false if cache is disabled.\n            let support_cache_partial_read =\n                session.cache.support_streaming_partial_write() == Some(true);\n            let upgraded = session.was_upgraded();\n\n            // Similar logic in h1 need to reserve capacity first to avoid deadlock\n            // But we don't need to do the same because the h2 client_body pipe is unbounded (never block)\n            tokio::select! {\n                // NOTE: cannot avoid this copy since h2 owns the buf\n                body = session.downstream_session.read_body_or_idle(downstream_state.is_done()), if downstream_state.can_poll() => {\n                    debug!(\"downstream event\");\n                    let body = match body {\n                        Ok(b) => b,\n                        Err(e) => {\n                            let wait_for_cache_fill = (!serve_from_cache.is_on() && support_cache_partial_read)\n                                || serve_from_cache.is_miss();\n                            if wait_for_cache_fill {\n                                // ignore downstream error so that upstream can continue to write cache\n                                downstream_state.to_errored();\n                                warn!(\n                                    \"Downstream Error ignored during caching: {}, {}\",\n                                    e,\n                                    self.inner.request_summary(session, ctx)\n                                );\n                                // This will not be treated as a final error, but we should signal to\n                                // downstream session regardless\n                                session.downstream_session.on_proxy_failure(e);\n                                continue;\n                           } else {\n                                return Err(e.into_down());\n                           }\n                        }\n                    };\n                    let is_body_done = session.is_body_done();\n                    match self.send_body_to2(session, body, is_body_done, client_body, ctx, write_timeout).await {\n                        Ok(request_done) =>  {\n                            downstream_state.maybe_finished(request_done);\n                        },\n                        Err(e) => {\n                            // mark request done, attempt to drain receive\n                            warn!(\"Upstream h2 body send error: {e}\");\n                            // upstream is what actually errored but we don't want to continue\n                            // polling the downstream body\n                            downstream_state.to_errored();\n                        }\n                    };\n                },\n\n                task = rx.recv(), if !response_state.upstream_done() => {\n                    if let Some(t) = task {\n                        debug!(\"upstream event: {:?}\", t);\n                        if serve_from_cache.should_discard_upstream() {\n                            // just drain, do we need to do anything else?\n                           continue;\n                        }\n                        // pull as many tasks as we can\n                        let mut tasks = Vec::with_capacity(TASK_BUFFER_SIZE);\n                        tasks.push(t);\n                        // tokio::task::unconstrained because now_or_never may yield None when the future is ready\n                        while let Some(maybe_task) = tokio::task::unconstrained(rx.recv()).now_or_never() {\n                            if let Some(t) = maybe_task {\n                                tasks.push(t);\n                            } else {\n                                break\n                            }\n                        }\n\n                        /* run filters before sending to downstream */\n                        let mut filtered_tasks = Vec::with_capacity(TASK_BUFFER_SIZE);\n                        for mut t in tasks {\n                            if self.revalidate_or_stale(session, &mut t, ctx).await {\n                                serve_from_cache.enable();\n                                response_state.enable_cached_response();\n                                // skip downstream filtering entirely as the 304 will not be sent\n                                break;\n                            }\n                            session.upstream_compression.response_filter(&mut t);\n                            // check error and abort\n                            // otherwise the error is surfaced via write_response_tasks()\n                            if !serve_from_cache.should_send_to_downstream() {\n                                if let HttpTask::Failed(e) = t {\n                                    return Err(e);\n                                }\n                            }\n                            filtered_tasks.push(\n                                self.h2_response_filter(session, t, ctx,\n                                    &mut serve_from_cache,\n                                    &mut range_body_filter, false).await?);\n                            if serve_from_cache.is_miss_header() {\n                                response_state.enable_cached_response();\n                            }\n                        }\n\n                        if !serve_from_cache.should_send_to_downstream() {\n                            // TODO: need to derive response_done from filtered_tasks in case downstream failed already\n                            continue;\n                        }\n\n                        let response_done = session.write_response_tasks(filtered_tasks).await?;\n                        if session.was_upgraded() {\n                            // it is very weird if the downstream session decides to upgrade\n                            // since the client h2 session cannot, return an error on this case\n                            return Error::e_explain(H2Error, \"upgraded while proxying to h2 session\");\n                        }\n                        response_state.maybe_set_upstream_done(response_done);\n                    } else {\n                        debug!(\"empty upstream event\");\n                        response_state.maybe_set_upstream_done(true);\n                    }\n                }\n\n                task = serve_from_cache.next_http_task(&mut session.cache, &mut range_body_filter, upgraded),\n                    if !response_state.cached_done() && !downstream_state.is_errored() && serve_from_cache.is_on() => {\n                    let task = self.h2_response_filter(session, task?, ctx,\n                        &mut serve_from_cache,\n                        &mut range_body_filter, true).await?;\n                    debug!(\"serve_from_cache task {task:?}\");\n\n                    match session.write_response_tasks(vec![task]).await {\n                        Ok(b) => response_state.maybe_set_cache_done(b),\n                        Err(e) => if serve_from_cache.is_miss() {\n                            // give up writing to downstream but wait for upstream cache write to finish\n                            downstream_state.to_errored();\n                            response_state.maybe_set_cache_done(true);\n                            warn!(\n                                \"Downstream Error ignored during caching: {}, {}\",\n                                e,\n                                self.inner.request_summary(session, ctx)\n                            );\n                            // This will not be treated as a final error, but we should signal to\n                            // downstream session regardless\n                            session.downstream_session.on_proxy_failure(e);\n                            continue;\n                        } else {\n                            return Err(e);\n                        }\n                    }\n                    if response_state.cached_done() {\n                        if let Err(e) = session.cache.finish_hit_handler().await {\n                            warn!(\"Error during finish_hit_handler: {}\", e);\n                        }\n                    }\n                }\n                data = custom_reader_next, if downstream_custom_read && !downstream_state.is_errored()  => {\n                    let Some(data) = data.flatten() else {\n\n                        downstream_custom_read = false;\n                        continue;\n                    };\n\n                    let data = match data {\n                        Ok(data) => data,\n                        Err(err) =>  {\n                            warn!(\"downstream_custom_message_reader got error: {err}\");\n                            downstream_custom_read = false;\n                            continue;\n                        },\n                    };\n\n                    self.inner\n                        .downstream_custom_message_proxy_filter(session, data, ctx, true) // true, because it's the last hop for downstream proxying\n                        .await?;\n                },\n\n                data = custom_inject_rx_recv, if downstream_custom_write => {\n                    match data.flatten() {\n                        Some(data) => {\n                            if let Some(ref mut custom_writer) = downstream_custom_message_writer {\n                                custom_writer.write_custom_message(data).await?\n                            }\n                        },\n                        None => {\n                            downstream_custom_write = false;\n                            if let Some(ref mut custom_writer) = downstream_custom_message_writer {\n                                custom_writer.finish_custom().await?;\n                            }\n                        },\n                    }\n                },\n\n                else => {\n                    break;\n                }\n            }\n        }\n\n        if let Some(custom_session) = session.downstream_session.as_custom_mut() {\n            if let Some(downstream_custom_message_reader) = downstream_custom_message_reader {\n                custom_session\n                    .restore_custom_message_reader(downstream_custom_message_reader)\n                    .expect(\"downstream restore_custom_message_reader should be empty\");\n            }\n        }\n\n        let mut reuse_downstream = !downstream_state.is_errored();\n        if reuse_downstream {\n            match session.as_mut().finish_body().await {\n                Ok(_) => {\n                    debug!(\"finished sending body to downstream\");\n                }\n                Err(e) => {\n                    error!(\"Error finish sending body to downstream: {}\", e);\n                    reuse_downstream = false;\n                }\n            }\n        }\n        Ok(reuse_downstream)\n    }\n\n    async fn h2_response_filter(\n        &self,\n        session: &mut Session,\n        mut task: HttpTask,\n        ctx: &mut SV::CTX,\n        serve_from_cache: &mut ServeFromCache,\n        range_body_filter: &mut RangeBodyFilter,\n        from_cache: bool, // are the task from cache already\n    ) -> Result<HttpTask>\n    where\n        SV: ProxyHttp + Send + Sync,\n        SV::CTX: Send + Sync,\n    {\n        if !from_cache {\n            if let Some(duration) = self.upstream_filter(session, &mut task, ctx).await? {\n                trace!(\"delaying upstream response for {duration:?}\");\n                time::sleep(duration).await;\n            }\n\n            // cache the original response before any downstream transformation\n            // requests that bypassed cache still need to run filters to see if the response has become cacheable\n            if session.cache.enabled() || session.cache.bypassing() {\n                if let Err(e) = self\n                    .cache_http_task(session, &task, ctx, serve_from_cache)\n                    .await\n                {\n                    session.cache.disable(NoCacheReason::StorageError);\n                    if serve_from_cache.is_miss_body() {\n                        // if the response stream cache body during miss but write fails, it has to\n                        // give up the entire request\n                        return Err(e);\n                    } else {\n                        // otherwise, continue processing the response\n                        warn!(\n                            \"Fail to cache response: {}, {}\",\n                            e,\n                            self.inner.request_summary(session, ctx)\n                        );\n                    }\n                }\n            }\n            // skip the downstream filtering if these tasks are just for cache admission\n            if !serve_from_cache.should_send_to_downstream() {\n                return Ok(task);\n            }\n        } // else: cached/local response, no need to trigger upstream filters and caching\n\n        // normally max file size is tracked in cache_http_task filters (when cache enabled),\n        // we will track it in these filters before sending to downstream on specific conditions\n        // when cache is disabled\n        let track_max_cache_size = matches!(\n            session.cache.phase(),\n            CachePhase::Disabled(NoCacheReason::PredictedResponseTooLarge)\n        );\n\n        let res = match task {\n            HttpTask::Header(mut header, eos) => {\n                /* Downstream revalidation, only needed when cache is on because otherwise origin\n                 * will handle it */\n                if session.upstream_headers_mutated_for_cache() {\n                    self.downstream_response_conditional_filter(\n                        serve_from_cache,\n                        session,\n                        &mut header,\n                        ctx,\n                    );\n                    if !session.ignore_downstream_range {\n                        let range_type = self.inner.range_header_filter(session, &mut header, ctx);\n                        range_body_filter.set(range_type);\n                    }\n                }\n\n                self.inner\n                    .response_filter(session, &mut header, ctx)\n                    .await?;\n                /* Downgrade the version so that write_response_header won't panic */\n                header.set_version(Version::HTTP_11);\n\n                // these status codes / method cannot have body, so no need to add chunked encoding\n                let no_body = session.req_header().method == \"HEAD\"\n                    || matches!(header.status.as_u16(), 204 | 304);\n\n                /* Add chunked header to tell downstream to use chunked encoding\n                 * during the absent of content-length in h2 */\n                if !no_body\n                    && !header.status.is_informational()\n                    && header.headers.get(http::header::CONTENT_LENGTH).is_none()\n                {\n                    header.insert_header(http::header::TRANSFER_ENCODING, \"chunked\")?;\n                }\n                Ok(HttpTask::Header(header, eos))\n            }\n            HttpTask::Body(data, eos) => {\n                if track_max_cache_size {\n                    session\n                        .cache\n                        .track_body_bytes_for_max_file_size(data.as_ref().map_or(0, |d| d.len()));\n                }\n\n                let mut data = range_body_filter.filter_body(data);\n                if let Some(duration) = self\n                    .inner\n                    .response_body_filter(session, &mut data, eos, ctx)?\n                {\n                    trace!(\"delaying downstream response for {duration:?}\");\n                    time::sleep(duration).await;\n                }\n                Ok(HttpTask::Body(data, eos))\n            }\n            HttpTask::UpgradedBody(..) => {\n                // An h2 session should not be able to send an h2 upgraded response body,\n                // and logically that is impossible unless there is a bug in the client v2 session\n                panic!(\"Unexpected UpgradedBody task while proxy h2\");\n            }\n            HttpTask::Trailer(mut trailers) => {\n                let trailer_buffer = match trailers.as_mut() {\n                    Some(trailers) => {\n                        debug!(\"Parsing response trailers..\");\n                        match self\n                            .inner\n                            .response_trailer_filter(session, trailers, ctx)\n                            .await\n                        {\n                            Ok(buf) => buf,\n                            Err(e) => {\n                                error!(\n                                    \"Encountered error while filtering upstream trailers {:?}\",\n                                    e\n                                );\n                                None\n                            }\n                        }\n                    }\n                    _ => None,\n                };\n                // if we have a trailer buffer write it to the downstream response body\n                if let Some(buffer) = trailer_buffer {\n                    // write_body will not write additional bytes after reaching the content-length\n                    // for gRPC H2 -> H1 this is not a problem but may be a problem for non gRPC code\n                    // https://http2.github.io/http2-spec/#malformed\n                    Ok(HttpTask::Body(Some(buffer), true))\n                } else {\n                    Ok(HttpTask::Trailer(trailers))\n                }\n            }\n            HttpTask::Done => Ok(task),\n            HttpTask::Failed(_) => Ok(task), // Do nothing just pass the error down\n        };\n        // On end, check if the response (based on file size) can be considered cacheable again\n        if let Ok(task) = res.as_ref() {\n            if track_max_cache_size\n                && task.is_end()\n                && !matches!(task, HttpTask::Failed(_))\n                && !session.cache.exceeded_max_file_size()\n            {\n                session.cache.response_became_cacheable();\n            }\n        }\n        res\n    }\n\n    async fn send_body_to2(\n        &self,\n        session: &mut Session,\n        mut data: Option<Bytes>,\n        end_of_body: bool,\n        client_body: &mut h2::SendStream<bytes::Bytes>,\n        ctx: &mut SV::CTX,\n        write_timeout: Option<Duration>,\n    ) -> Result<bool>\n    where\n        SV: ProxyHttp + Send + Sync,\n        SV::CTX: Send + Sync,\n    {\n        session\n            .downstream_modules_ctx\n            .request_body_filter(&mut data, end_of_body)\n            .await?;\n\n        self.inner\n            .request_body_filter(session, &mut data, end_of_body, ctx)\n            .await?;\n\n        /* it is normal to get 0 bytes because of multi-chunk parsing or request_body_filter.\n         * Although there is no harm writing empty byte to h2, unlike h1, we ignore it\n         * for consistency */\n        if !end_of_body && data.as_ref().is_some_and(|d| d.is_empty()) {\n            return Ok(false);\n        }\n\n        if let Some(data) = data {\n            debug!(\"Write {} bytes body to h2 upstream\", data.len());\n            write_body(client_body, data, end_of_body, write_timeout)\n                .await\n                .map_err(|e| e.into_up())?;\n        } else {\n            debug!(\"Read downstream body done\");\n            /* send a standalone END_STREAM flag */\n            write_body(client_body, Bytes::new(), true, write_timeout)\n                .await\n                .map_err(|e| e.into_up())?;\n        }\n\n        Ok(end_of_body)\n    }\n}\n\n/* Read response header, body and trailer from h2 upstream and send them to tx */\npub(crate) async fn pipe_up_to_down_response(\n    client: &mut Http2Session,\n    tx: mpsc::Sender<HttpTask>,\n) -> Result<()> {\n    client\n        .read_response_header()\n        .await\n        .map_err(|e| e.into_up())?; // should we send the error as an HttpTask?\n\n    let resp_header = Box::new(client.response_header().expect(\"just read\").clone());\n\n    match client.check_response_end_or_error() {\n        Ok(eos) => {\n            // XXX: the h2 crate won't check for content-length underflow\n            // if a header frame with END_STREAM is sent without data frames\n            // As stated by RFC, \"204 or 304 responses contain no content,\n            // as does the response to a HEAD request\"\n            // https://datatracker.ietf.org/doc/html/rfc9113#section-8.1.1\n            let req_header = client.request_header().expect(\"must have sent req\");\n            if eos\n                && req_header.method != Method::HEAD\n                && resp_header.status != StatusCode::NO_CONTENT\n                && resp_header.status != StatusCode::NOT_MODIFIED\n                // RFC technically allows for leading zeroes\n                // https://datatracker.ietf.org/doc/html/rfc9110#name-content-length\n                && resp_header\n                    .headers\n                    .get(CONTENT_LENGTH)\n                    .is_some_and(|cl| cl.as_bytes().iter().any(|b| *b != b'0'))\n            {\n                let _ = tx\n                    .send(HttpTask::Failed(\n                        Error::explain(H2Error, \"non-zero content-length on EOS headers frame\")\n                            .into_up(),\n                    ))\n                    .await;\n                return Ok(());\n            }\n            tx.send(HttpTask::Header(resp_header, eos))\n                .await\n                .or_err(InternalError, \"sending h2 headers to pipe\")?;\n        }\n        Err(e) => {\n            // If upstream errored, then push error to downstream and then quit\n            // Don't care if send fails (which means downstream already gone)\n            // we were still able to retrieve the headers, so try sending\n            let _ = tx.send(HttpTask::Header(resp_header, false)).await;\n            let _ = tx.send(HttpTask::Failed(e.into_up())).await;\n            return Ok(());\n        }\n    }\n\n    while let Some(chunk) = client\n        .read_response_body()\n        .await\n        .map_err(|e| e.into_up())\n        .transpose()\n    {\n        let data = match chunk {\n            Ok(d) => d,\n            Err(e) => {\n                // Push the error to downstream and then quit\n                let _ = tx.send(HttpTask::Failed(e.into_up())).await;\n                // Downstream should consume all remaining data and handle the error\n                return Ok(());\n            }\n        };\n        match client.check_response_end_or_error() {\n            Ok(eos) => {\n                let empty = data.is_empty();\n                if empty && !eos {\n                    /* it is normal to get 0 bytes because of multi-chunk\n                     * don't write 0 bytes to downstream since it will be\n                     * misread as the terminating chunk */\n                    continue;\n                }\n                let sent = tx\n                    .send(HttpTask::Body(Some(data), eos))\n                    .await\n                    .or_err(InternalError, \"sending h2 body to pipe\");\n                // If the if the response with content-length is sent to an HTTP1 downstream,\n                // bidirection_down_to_up() could decide that the body has finished and exit without\n                // waiting for this function to signal the eos. In this case tx being closed is not\n                // an sign of error. It should happen if the only thing left for the h2 to send is\n                // an empty data frame with eos set.\n                if sent.is_err() && eos && empty {\n                    return Ok(());\n                }\n                sent?;\n            }\n            Err(e) => {\n                // Similar to above, push the error to downstream and then quit\n                let _ = tx.send(HttpTask::Failed(e.into_up())).await;\n                return Ok(());\n            }\n        }\n    }\n\n    // attempt to get trailers\n    let trailers = match client.read_trailers().await {\n        Ok(t) => t,\n        Err(e) => {\n            // Similar to above, push the error to downstream and then quit\n            let _ = tx.send(HttpTask::Failed(e.into_up())).await;\n            return Ok(());\n        }\n    };\n\n    let trailers = trailers.map(Box::new);\n\n    if trailers.is_some() {\n        tx.send(HttpTask::Trailer(trailers))\n            .await\n            .or_err(InternalError, \"sending h2 trailer to pipe\")?;\n    }\n\n    tx.send(HttpTask::Done)\n        .await\n        .unwrap_or_else(|_| debug!(\"h2 to h1 channel closed!\"));\n\n    Ok(())\n}\n\n#[test]\nfn test_update_authority() {\n    let mut parts = http::request::Builder::new()\n        .body(())\n        .unwrap()\n        .into_parts()\n        .0;\n    update_h2_scheme_authority(&mut parts, b\"example.com\", true).unwrap();\n    assert_eq!(\"example.com\", parts.uri.authority().unwrap());\n    update_h2_scheme_authority(&mut parts, b\"example.com:456\", true).unwrap();\n    assert_eq!(\"example.com:456\", parts.uri.authority().unwrap());\n    update_h2_scheme_authority(&mut parts, b\"example.com:\", true).unwrap();\n    assert_eq!(\"example.com:\", parts.uri.authority().unwrap());\n    update_h2_scheme_authority(&mut parts, b\"example.com:123:345\", true).unwrap();\n    assert_eq!(\"example.com:123\", parts.uri.authority().unwrap());\n    update_h2_scheme_authority(&mut parts, b\"[::1]\", true).unwrap();\n    assert_eq!(\"[::1]\", parts.uri.authority().unwrap());\n\n    // verify scheme\n    update_h2_scheme_authority(&mut parts, b\"example.com\", true).unwrap();\n    assert_eq!(\"https://example.com\", parts.uri);\n    update_h2_scheme_authority(&mut parts, b\"example.com\", false).unwrap();\n    assert_eq!(\"http://example.com\", parts.uri);\n}\n"
  },
  {
    "path": "pingora-proxy/src/proxy_purge.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\nuse super::*;\nuse pingora_core::protocols::http::error_resp;\nuse std::borrow::Cow;\n\n#[derive(Debug)]\npub enum PurgeStatus {\n    /// Cache was not enabled, purge ineffectual.\n    NoCache,\n    /// Asset was found in cache (and presumably purged or being purged).\n    Found,\n    /// Asset was not found in cache.\n    NotFound,\n    /// Cache returned a purge error.\n    /// Contains causing error in case it should affect the downstream response.\n    Error(Box<Error>),\n}\n\n// Return a canned response to a purge request, based on whether the cache had the asset or not\n// (or otherwise returned an error).\nfn purge_response(purge_status: &PurgeStatus) -> Cow<'static, ResponseHeader> {\n    let resp = match purge_status {\n        PurgeStatus::NoCache => &*NOT_PURGEABLE,\n        PurgeStatus::Found => &*OK,\n        PurgeStatus::NotFound => &*NOT_FOUND,\n        PurgeStatus::Error(ref _e) => &*INTERNAL_ERROR,\n    };\n    Cow::Borrowed(resp)\n}\n\nfn gen_purge_response(code: u16) -> ResponseHeader {\n    let mut resp = ResponseHeader::build(code, Some(3)).unwrap();\n    resp.insert_header(header::SERVER, &SERVER_NAME[..])\n        .unwrap();\n    resp.insert_header(header::CONTENT_LENGTH, 0).unwrap();\n    resp.insert_header(header::CACHE_CONTROL, \"private, no-store\")\n        .unwrap();\n    // TODO more headers?\n    resp\n}\n\nstatic OK: Lazy<ResponseHeader> = Lazy::new(|| gen_purge_response(200));\nstatic NOT_FOUND: Lazy<ResponseHeader> = Lazy::new(|| gen_purge_response(404));\n// for when purge is sent to uncacheable assets\nstatic NOT_PURGEABLE: Lazy<ResponseHeader> = Lazy::new(|| gen_purge_response(405));\n// on cache storage or proxy error\nstatic INTERNAL_ERROR: Lazy<ResponseHeader> = Lazy::new(|| error_resp::gen_error_response(500));\n\nimpl<SV, C> HttpProxy<SV, C>\nwhere\n    C: custom::Connector,\n{\n    pub(crate) async fn proxy_purge(\n        &self,\n        session: &mut Session,\n        ctx: &mut SV::CTX,\n    ) -> Option<(bool, Option<Box<Error>>)>\n    where\n        SV: ProxyHttp + Send + Sync,\n        SV::CTX: Send + Sync,\n    {\n        let purge_status = if session.cache.enabled() {\n            match session.cache.purge().await {\n                Ok(found) => {\n                    if found {\n                        PurgeStatus::Found\n                    } else {\n                        PurgeStatus::NotFound\n                    }\n                }\n                Err(e) => {\n                    session.cache.disable(NoCacheReason::StorageError);\n                    warn!(\n                        \"Fail to purge cache: {e}, {}\",\n                        self.inner.request_summary(session, ctx)\n                    );\n                    PurgeStatus::Error(e)\n                }\n            }\n        } else {\n            // cache was not enabled\n            PurgeStatus::NoCache\n        };\n\n        let mut purge_resp = purge_response(&purge_status);\n        if let Err(e) =\n            self.inner\n                .purge_response_filter(session, ctx, purge_status, &mut purge_resp)\n        {\n            error!(\n                \"Failed purge response filter: {e}, {}\",\n                self.inner.request_summary(session, ctx)\n            );\n            purge_resp = Cow::Borrowed(&*INTERNAL_ERROR)\n        }\n\n        let write_result = match purge_resp {\n            Cow::Borrowed(r) => session.as_mut().write_response_header_ref(r).await,\n            Cow::Owned(r) => session.as_mut().write_response_header(Box::new(r)).await,\n        };\n        let (reuse, err) = match write_result {\n            Ok(_) => (true, None),\n            // dirty, not reusable\n            Err(e) => {\n                let e = e.into_down();\n                error!(\n                    \"Failed to send purge response: {e}, {}\",\n                    self.inner.request_summary(session, ctx)\n                );\n                (false, Some(e))\n            }\n        };\n        Some((reuse, err))\n    }\n}\n"
  },
  {
    "path": "pingora-proxy/src/proxy_trait.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\nuse super::*;\nuse pingora_cache::{\n    key::HashBinary,\n    CacheKey, CacheMeta, ForcedFreshness, HitHandler,\n    RespCacheable::{self, *},\n};\nuse proxy_cache::range_filter::{self};\nuse std::time::Duration;\n\n/// The interface to control the HTTP proxy\n///\n/// The methods in [ProxyHttp] are filters/callbacks which will be performed on all requests at their\n/// particular stage (if applicable).\n///\n/// If any of the filters returns [Result::Err], the request will fail, and the error will be logged.\n#[cfg_attr(not(doc_async_trait), async_trait)]\npub trait ProxyHttp {\n    /// The per request object to share state across the different filters\n    type CTX;\n\n    /// Define how the `ctx` should be created.\n    fn new_ctx(&self) -> Self::CTX;\n\n    /// Define where the proxy should send the request to.\n    ///\n    /// The returned [HttpPeer] contains the information regarding where and how this request should\n    /// be forwarded to.\n    async fn upstream_peer(\n        &self,\n        session: &mut Session,\n        ctx: &mut Self::CTX,\n    ) -> Result<Box<HttpPeer>>;\n\n    /// Set up downstream modules.\n    ///\n    /// In this phase, users can add or configure [HttpModules] before the server starts up.\n    ///\n    /// In the default implementation of this method, [ResponseCompressionBuilder] is added\n    /// and disabled.\n    fn init_downstream_modules(&self, modules: &mut HttpModules) {\n        // Add disabled downstream compression module by default\n        modules.add_module(ResponseCompressionBuilder::enable(0));\n    }\n\n    /// Handle the incoming request.\n    ///\n    /// In this phase, users can parse, validate, rate limit, perform access control and/or\n    /// return a response for this request.\n    ///\n    /// If the user already sent a response to this request, an `Ok(true)` should be returned so that\n    /// the proxy would exit. The proxy continues to the next phases when `Ok(false)` is returned.\n    ///\n    /// By default this filter does nothing and returns `Ok(false)`.\n    async fn request_filter(&self, _session: &mut Session, _ctx: &mut Self::CTX) -> Result<bool>\n    where\n        Self::CTX: Send + Sync,\n    {\n        Ok(false)\n    }\n\n    /// Handle the incoming request before any downstream module is executed.\n    ///\n    /// This function is similar to [Self::request_filter()] but executes before any other logic,\n    /// including downstream module logic. The main purpose of this function is to provide finer\n    /// grained control of the behavior of the modules.\n    ///\n    /// Note that because this function is executed before any module that might provide access\n    /// control or rate limiting, logic should stay in request_filter() if it can in order to be\n    /// protected by said modules.\n    async fn early_request_filter(&self, _session: &mut Session, _ctx: &mut Self::CTX) -> Result<()>\n    where\n        Self::CTX: Send + Sync,\n    {\n        Ok(())\n    }\n\n    /// Returns whether this session is allowed to spawn subrequests.\n    ///\n    /// This function is checked after [Self::early_request_filter] to allow that filter to configure\n    /// this if required. This will also run for subrequests themselves, which may allowed to spawn\n    /// their own subrequests.\n    ///\n    /// Note that this doesn't prevent subrequests from being spawned based on the session by proxy\n    /// core functionality, e.g. background cache revalidation requires spawning subrequests.\n    fn allow_spawning_subrequest(&self, _session: &Session, _ctx: &Self::CTX) -> bool\n    where\n        Self::CTX: Send + Sync,\n    {\n        false\n    }\n\n    /// Handle the incoming request body.\n    ///\n    /// This function will be called every time a piece of request body is received. The `body` is\n    /// **not the entire request body**.\n    ///\n    /// The async nature of this function allows to throttle the upload speed and/or executing\n    /// heavy computation logic such as WAF rules on offloaded threads without blocking the threads\n    /// who process the requests themselves.\n    async fn request_body_filter(\n        &self,\n        _session: &mut Session,\n        _body: &mut Option<Bytes>,\n        _end_of_stream: bool,\n        _ctx: &mut Self::CTX,\n    ) -> Result<()>\n    where\n        Self::CTX: Send + Sync,\n    {\n        Ok(())\n    }\n\n    /// This filter decides if the request is cacheable and what cache backend to use\n    ///\n    /// The caller can interact with `Session.cache` to enable caching.\n    ///\n    /// By default this filter does nothing which effectively disables caching.\n    // Ideally only session.cache should be modified, TODO: reflect that in this interface\n    fn request_cache_filter(&self, _session: &mut Session, _ctx: &mut Self::CTX) -> Result<()>\n    where\n        Self::CTX: Send + Sync,\n    {\n        Ok(())\n    }\n\n    /// This callback generates the cache key.\n    ///\n    /// This callback is called only when cache is enabled for this request.\n    ///\n    /// There is no sensible default cache key for all proxy applications. The\n    /// correct key depends on which request properties affect upstream responses\n    /// (e.g. `Vary` headers, custom request filters that modify the origin host).\n    /// Getting this wrong leads to cache poisoning.\n    ///\n    /// See `pingora-proxy/tests/utils/server_utils.rs` for a minimal (not\n    /// production-ready) reference implementation.\n    ///\n    /// # Panics\n    ///\n    /// The default implementation panics. You **must** override this method when\n    /// caching is enabled.\n    fn cache_key_callback(&self, _session: &Session, _ctx: &mut Self::CTX) -> Result<CacheKey> {\n        unimplemented!(\"cache_key_callback must be implemented when caching is enabled\")\n    }\n\n    /// This callback is invoked when a cacheable response is ready to be admitted to cache.\n    fn cache_miss(&self, session: &mut Session, _ctx: &mut Self::CTX) {\n        session.cache.cache_miss();\n    }\n\n    /// This filter is called after a successful cache lookup and before the\n    /// cache asset is ready to be used.\n    ///\n    /// This filter allows the user to log or force invalidate the asset, or\n    /// to adjust the body reader associated with the cache hit.\n    /// This also runs on stale hit assets (for which `is_fresh` is false).\n    ///\n    /// The value returned indicates if the force invalidation should be used,\n    /// and which kind. Returning `None` indicates no forced invalidation\n    async fn cache_hit_filter(\n        &self,\n        _session: &mut Session,\n        _meta: &CacheMeta,\n        _hit_handler: &mut HitHandler,\n        _is_fresh: bool,\n        _ctx: &mut Self::CTX,\n    ) -> Result<Option<ForcedFreshness>>\n    where\n        Self::CTX: Send + Sync,\n    {\n        Ok(None)\n    }\n\n    /// Decide if a request should continue to upstream after not being served from cache.\n    ///\n    /// returns: Ok(true) if the request should continue, Ok(false) if a response was written by the\n    /// callback and the session should be finished, or an error\n    ///\n    /// This filter can be used for deferring checks like rate limiting or access control to when they\n    /// actually needed after cache miss.\n    ///\n    /// By default the session will attempt to be reused after returning Ok(false). It is the\n    /// caller's responsibility to disable keepalive or drain the request body if needed.\n    async fn proxy_upstream_filter(\n        &self,\n        _session: &mut Session,\n        _ctx: &mut Self::CTX,\n    ) -> Result<bool>\n    where\n        Self::CTX: Send + Sync,\n    {\n        Ok(true)\n    }\n\n    /// Decide if the response is cacheable\n    fn response_cache_filter(\n        &self,\n        _session: &Session,\n        _resp: &ResponseHeader,\n        _ctx: &mut Self::CTX,\n    ) -> Result<RespCacheable> {\n        Ok(Uncacheable(NoCacheReason::Custom(\"default\")))\n    }\n\n    /// Decide how to generate cache vary key from both request and response\n    ///\n    /// None means no variance is needed.\n    fn cache_vary_filter(\n        &self,\n        _meta: &CacheMeta,\n        _ctx: &mut Self::CTX,\n        _req: &RequestHeader,\n    ) -> Option<HashBinary> {\n        // default to None for now to disable vary feature\n        None\n    }\n\n    /// Decide if the incoming request's condition _fails_ against the cached response.\n    ///\n    /// Returning `Ok(true)` means that the response does _not_ match against the condition, and\n    /// that the proxy can return `304 Not Modified` downstream.\n    ///\n    /// An example is a conditional GET request with `If-None-Match: \"foobar\"`. If the cached\n    /// response contains the `ETag: \"foobar\"`, then the condition fails, and `304 Not Modified`\n    /// should be returned. Else, the condition passes which means the full `200 OK` response must\n    /// be sent.\n    fn cache_not_modified_filter(\n        &self,\n        session: &Session,\n        resp: &ResponseHeader,\n        _ctx: &mut Self::CTX,\n    ) -> Result<bool> {\n        Ok(\n            pingora_core::protocols::http::conditional_filter::not_modified_filter(\n                session.req_header(),\n                resp,\n            ),\n        )\n    }\n\n    /// This filter is called when cache is enabled to determine what byte range to return (in both\n    /// cache hit and miss cases) from the response body. It is only used when caching is enabled,\n    /// otherwise the upstream is responsible for any filtering. It allows users to define the range\n    /// this request is for via its return type `range_filter::RangeType`.\n    ///\n    /// It also allow users to modify the response header accordingly.\n    ///\n    /// The default implementation can handle a single-range as per [RFC7232].\n    ///\n    /// [RFC7232]: https://www.rfc-editor.org/rfc/rfc7232\n    fn range_header_filter(\n        &self,\n        session: &mut Session,\n        resp: &mut ResponseHeader,\n        _ctx: &mut Self::CTX,\n    ) -> range_filter::RangeType {\n        const DEFAULT_MAX_RANGES: Option<usize> = Some(200);\n        proxy_cache::range_filter::range_header_filter(\n            session.req_header(),\n            resp,\n            DEFAULT_MAX_RANGES,\n        )\n    }\n\n    /// Modify the request before it is sent to the upstream\n    ///\n    /// Unlike [Self::request_filter()], this filter allows to change the request headers to send\n    /// to the upstream.\n    async fn upstream_request_filter(\n        &self,\n        _session: &mut Session,\n        _upstream_request: &mut RequestHeader,\n        _ctx: &mut Self::CTX,\n    ) -> Result<()>\n    where\n        Self::CTX: Send + Sync,\n    {\n        Ok(())\n    }\n\n    /// Modify the response header from the upstream\n    ///\n    /// The modification is before caching, so any change here will be stored in the cache if enabled.\n    ///\n    /// Responses served from cache won't trigger this filter. If the cache needed revalidation,\n    /// only the 304 from upstream will trigger the filter (though it will be merged into the\n    /// cached header, not served directly to downstream).\n    async fn upstream_response_filter(\n        &self,\n        _session: &mut Session,\n        _upstream_response: &mut ResponseHeader,\n        _ctx: &mut Self::CTX,\n    ) -> Result<()>\n    where\n        Self::CTX: Send + Sync,\n    {\n        Ok(())\n    }\n\n    /// Modify the response header before it is send to the downstream\n    ///\n    /// The modification is after caching. This filter is called for all responses including\n    /// responses served from cache.\n    async fn response_filter(\n        &self,\n        _session: &mut Session,\n        _upstream_response: &mut ResponseHeader,\n        _ctx: &mut Self::CTX,\n    ) -> Result<()>\n    where\n        Self::CTX: Send + Sync,\n    {\n        Ok(())\n    }\n\n    // custom_forwarding is called when downstream and upstream connections are successfully established.\n    #[doc(hidden)]\n    async fn custom_forwarding(\n        &self,\n        _session: &mut Session,\n        _ctx: &mut Self::CTX,\n        _custom_message_to_upstream: Option<mpsc::Sender<Bytes>>,\n        _custom_message_to_downstream: mpsc::Sender<Bytes>,\n    ) -> Result<()>\n    where\n        Self::CTX: Send + Sync,\n    {\n        Ok(())\n    }\n\n    // received a custom message from the downstream before sending it to the upstream.\n    #[doc(hidden)]\n    async fn downstream_custom_message_proxy_filter(\n        &self,\n        _session: &mut Session,\n        custom_message: Bytes,\n        _ctx: &mut Self::CTX,\n        _final_hop: bool,\n    ) -> Result<Option<Bytes>>\n    where\n        Self::CTX: Send + Sync,\n    {\n        Ok(Some(custom_message))\n    }\n\n    // received a custom message from the upstream before sending it to the downstream.\n    #[doc(hidden)]\n    async fn upstream_custom_message_proxy_filter(\n        &self,\n        _session: &mut Session,\n        custom_message: Bytes,\n        _ctx: &mut Self::CTX,\n        _final_hop: bool,\n    ) -> Result<Option<Bytes>>\n    where\n        Self::CTX: Send + Sync,\n    {\n        Ok(Some(custom_message))\n    }\n\n    /// Similar to [Self::upstream_response_filter()] but for response body\n    ///\n    /// This function will be called every time a piece of response body is received. The `body` is\n    /// **not the entire response body**.\n    fn upstream_response_body_filter(\n        &self,\n        _session: &mut Session,\n        _body: &mut Option<Bytes>,\n        _end_of_stream: bool,\n        _ctx: &mut Self::CTX,\n    ) -> Result<Option<Duration>> {\n        Ok(None)\n    }\n\n    /// Similar to [Self::upstream_response_filter()] but for response trailers\n    fn upstream_response_trailer_filter(\n        &self,\n        _session: &mut Session,\n        _upstream_trailers: &mut header::HeaderMap,\n        _ctx: &mut Self::CTX,\n    ) -> Result<()> {\n        Ok(())\n    }\n\n    /// Similar to [Self::response_filter()] but for response body chunks\n    fn response_body_filter(\n        &self,\n        _session: &mut Session,\n        _body: &mut Option<Bytes>,\n        _end_of_stream: bool,\n        _ctx: &mut Self::CTX,\n    ) -> Result<Option<Duration>>\n    where\n        Self::CTX: Send + Sync,\n    {\n        Ok(None)\n    }\n\n    /// Similar to [Self::response_filter()] but for response trailers.\n    /// Note, returning an Ok(Some(Bytes)) will result in the downstream response\n    /// trailers being written to the response body.\n    ///\n    /// TODO: make this interface more intuitive\n    async fn response_trailer_filter(\n        &self,\n        _session: &mut Session,\n        _upstream_trailers: &mut header::HeaderMap,\n        _ctx: &mut Self::CTX,\n    ) -> Result<Option<Bytes>>\n    where\n        Self::CTX: Send + Sync,\n    {\n        Ok(None)\n    }\n\n    /// This filter is called when the entire response is sent to the downstream successfully or\n    /// there is a fatal error that terminate the request.\n    ///\n    /// An error log is already emitted if there is any error. This phase is used for collecting\n    /// metrics and sending access logs.\n    async fn logging(&self, _session: &mut Session, _e: Option<&Error>, _ctx: &mut Self::CTX)\n    where\n        Self::CTX: Send + Sync,\n    {\n    }\n\n    /// A value of true means that the log message will be suppressed. The default value is false.\n    fn suppress_error_log(&self, _session: &Session, _ctx: &Self::CTX, _error: &Error) -> bool {\n        false\n    }\n\n    /// This filter is called when there is an error **after** a connection is established (or reused)\n    /// to the upstream.\n    fn error_while_proxy(\n        &self,\n        peer: &HttpPeer,\n        session: &mut Session,\n        e: Box<Error>,\n        _ctx: &mut Self::CTX,\n        client_reused: bool,\n    ) -> Box<Error> {\n        let mut e = e.more_context(format!(\"Peer: {}\", peer));\n        // only reused client connections where retry buffer is not truncated\n        e.retry\n            .decide_reuse(client_reused && !session.as_ref().retry_buffer_truncated());\n        e\n    }\n\n    /// This filter is called when there is an error in the process of establishing a connection\n    /// to the upstream.\n    ///\n    /// In this filter the user can decide whether the error is retry-able by marking the error `e`.\n    ///\n    /// If the error can be retried, [Self::upstream_peer()] will be called again so that the user\n    /// can decide whether to send the request to the same upstream or another upstream that is possibly\n    /// available.\n    fn fail_to_connect(\n        &self,\n        _session: &mut Session,\n        _peer: &HttpPeer,\n        _ctx: &mut Self::CTX,\n        e: Box<Error>,\n    ) -> Box<Error> {\n        e\n    }\n\n    /// This filter is called when the request encounters a fatal error.\n    ///\n    /// Users may write an error response to the downstream if the downstream is still writable.\n    ///\n    /// The response status code of the error response may be returned for logging purposes.\n    /// Additionally, the user can return whether this session may be reused in spite of the error.\n    /// Today this reuse status is only respected for errors that occur prior to upstream peer\n    /// selection, and the keepalive configured on the `Session` itself still takes precedent.\n    async fn fail_to_proxy(\n        &self,\n        session: &mut Session,\n        e: &Error,\n        _ctx: &mut Self::CTX,\n    ) -> FailToProxy\n    where\n        Self::CTX: Send + Sync,\n    {\n        let code = match e.etype() {\n            HTTPStatus(code) => *code,\n            _ => {\n                match e.esource() {\n                    ErrorSource::Upstream => 502,\n                    ErrorSource::Downstream => {\n                        match e.etype() {\n                            WriteError | ReadError | ConnectionClosed => {\n                                /* conn already dead */\n                                0\n                            }\n                            _ => 400,\n                        }\n                    }\n                    ErrorSource::Internal | ErrorSource::Unset => 500,\n                }\n            }\n        };\n        if code > 0 {\n            session.respond_error(code).await.unwrap_or_else(|e| {\n                error!(\"failed to send error response to downstream: {e}\");\n            });\n        }\n\n        FailToProxy {\n            error_code: code,\n            // default to no reuse, which is safest\n            can_reuse_downstream: false,\n        }\n    }\n\n    /// Decide whether should serve stale when encountering an error or during revalidation\n    ///\n    /// An implementation should follow\n    /// <https://datatracker.ietf.org/doc/html/rfc9111#section-4.2.4>\n    /// <https://www.rfc-editor.org/rfc/rfc5861#section-4>\n    ///\n    /// This filter is only called if cache is enabled.\n    // 5xx HTTP status will be encoded as ErrorType::HTTPStatus(code)\n    fn should_serve_stale(\n        &self,\n        _session: &mut Session,\n        _ctx: &mut Self::CTX,\n        error: Option<&Error>, // None when it is called during stale while revalidate\n    ) -> bool {\n        // A cache MUST NOT generate a stale response unless\n        // it is disconnected\n        // or doing so is explicitly permitted by the client or origin server\n        // (e.g. headers or an out-of-band contract)\n        error.is_some_and(|e| e.esource() == &ErrorSource::Upstream)\n    }\n\n    /// This filter is called when the request just established or reused a connection to the upstream\n    ///\n    /// This filter allows user to log timing and connection related info.\n    async fn connected_to_upstream(\n        &self,\n        _session: &mut Session,\n        _reused: bool,\n        _peer: &HttpPeer,\n        #[cfg(unix)] _fd: std::os::unix::io::RawFd,\n        #[cfg(windows)] _sock: std::os::windows::io::RawSocket,\n        _digest: Option<&Digest>,\n        _ctx: &mut Self::CTX,\n    ) -> Result<()>\n    where\n        Self::CTX: Send + Sync,\n    {\n        Ok(())\n    }\n\n    /// This callback is invoked every time request related error log needs to be generated\n    ///\n    /// Users can define what is important to be written about this request via the returned string.\n    fn request_summary(&self, session: &Session, _ctx: &Self::CTX) -> String {\n        session.as_ref().request_summary()\n    }\n\n    /// Whether the request should be used to invalidate(delete) the HTTP cache\n    ///\n    /// - `true`: this request will be used to invalidate the cache.\n    /// - `false`: this request is a treated as a normal request\n    fn is_purge(&self, _session: &Session, _ctx: &Self::CTX) -> bool {\n        false\n    }\n\n    /// This filter is called after the proxy cache generates the downstream response to the purge\n    /// request (to invalidate or delete from the HTTP cache), based on the purge status, which\n    /// indicates whether the request succeeded or failed.\n    ///\n    /// The filter allows the user to modify or replace the generated downstream response.\n    /// If the filter returns `Err`, the proxy will instead send a 500 response.\n    fn purge_response_filter(\n        &self,\n        _session: &Session,\n        _ctx: &mut Self::CTX,\n        _purge_status: PurgeStatus,\n        _purge_response: &mut std::borrow::Cow<'static, ResponseHeader>,\n    ) -> Result<()> {\n        Ok(())\n    }\n}\n\n/// Context struct returned by `fail_to_proxy`.\npub struct FailToProxy {\n    pub error_code: u16,\n    pub can_reuse_downstream: bool,\n}\n"
  },
  {
    "path": "pingora-proxy/src/subrequest/mod.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\nuse bytes::Bytes;\nuse pingora_cache::lock::{CacheKeyLockImpl, LockStatus, WritePermit};\nuse pingora_cache::CacheKey;\nuse pingora_core::protocols::http::subrequest::server::{\n    HttpSession as SessionSubrequest, SubrequestHandle,\n};\nuse std::any::Any;\n\npub mod pipe;\n\nstruct LockCtx {\n    write_permit: WritePermit,\n    cache_lock: &'static CacheKeyLockImpl,\n    key: CacheKey,\n}\n\n// Thin wrapper to allow iterating over InputBody Vec.\npub(crate) struct InputBodyReader(std::vec::IntoIter<Bytes>);\n\nimpl InputBodyReader {\n    pub fn read_body(&mut self) -> Option<Bytes> {\n        self.0.next()\n    }\n}\n\n/// Optional user-defined subrequest context.\npub type UserCtx = Box<dyn Any + Sync + Send>;\n\n#[derive(Debug, Copy, Clone, Default, PartialEq, Eq)]\npub enum BodyMode {\n    /// No body to be sent for subrequest.\n    #[default]\n    NoBody,\n    /// Waiting on body if needed.\n    ExpectBody,\n}\n\n#[derive(Default)]\npub struct CtxBuilder {\n    lock: Option<LockCtx>,\n    body_mode: BodyMode,\n    user_ctx: Option<UserCtx>,\n}\n\nimpl CtxBuilder {\n    pub fn new() -> Self {\n        Self {\n            lock: None,\n            body_mode: BodyMode::NoBody,\n            user_ctx: None,\n        }\n    }\n\n    pub fn cache_write_lock(\n        mut self,\n        cache_lock: &'static CacheKeyLockImpl,\n        key: CacheKey,\n        write_permit: WritePermit,\n    ) -> Self {\n        self.lock = Some(LockCtx {\n            cache_lock,\n            key,\n            write_permit,\n        });\n        self\n    }\n\n    pub fn user_ctx(mut self, user_ctx: UserCtx) -> Self {\n        self.user_ctx = Some(user_ctx);\n        self\n    }\n\n    pub fn body_mode(mut self, body_mode: BodyMode) -> Self {\n        self.body_mode = body_mode;\n        self\n    }\n\n    pub fn build(self) -> Ctx {\n        Ctx {\n            lock: self.lock,\n            body_mode: self.body_mode,\n            user_ctx: self.user_ctx,\n        }\n    }\n}\n\n/// Context struct to share state across the parent and sub-request.\npub struct Ctx {\n    body_mode: BodyMode,\n    lock: Option<LockCtx>,\n    // User-defined custom context.\n    user_ctx: Option<UserCtx>,\n}\n\nimpl Ctx {\n    /// Create a [`CtxBuilder`] in order to make a new subrequest `Ctx`.\n    pub fn builder() -> CtxBuilder {\n        CtxBuilder::new()\n    }\n\n    /// Get a reference to the extensions inside this subrequest.\n    pub fn user_ctx(&self) -> Option<&UserCtx> {\n        self.user_ctx.as_ref()\n    }\n\n    /// Get a mutable reference to the extensions inside this subrequest.\n    pub fn user_ctx_mut(&mut self) -> Option<&mut UserCtx> {\n        self.user_ctx.as_mut()\n    }\n\n    /// Release the write lock from the subrequest (to clean up a write permit\n    /// that will not be used in the cache key lock).\n    pub fn release_write_lock(&mut self) {\n        if let Some(lock) = self.lock.take() {\n            // If we are releasing the write lock in the subrequest,\n            // it means that the cache did not take it for whatever reason.\n            // TransientError will cause the election of a new writer\n            lock.cache_lock\n                .release(&lock.key, lock.write_permit, LockStatus::TransientError);\n        }\n    }\n\n    /// Take the write lock from the subrequest, for use in a cache key lock.\n    pub fn take_write_lock(&mut self) -> Option<WritePermit> {\n        // also clear out lock ctx\n        self.lock.take().map(|lock| lock.write_permit)\n    }\n\n    /// Get the `BodyMode` when this subrequest was created.\n    pub fn body_mode(&self) -> BodyMode {\n        self.body_mode\n    }\n}\n\nuse crate::HttpSession;\n\npub(crate) fn create_session(parsed_session: &HttpSession) -> (HttpSession, SubrequestHandle) {\n    let (session, handle) = SessionSubrequest::new_from_session(parsed_session);\n    (HttpSession::new_subrequest(session), handle)\n}\n\n#[tokio::test]\nasync fn test_dummy_request() {\n    use tokio_test::io::Builder;\n\n    let input = b\"GET / HTTP/1.1\\r\\n\\r\\n\";\n    let mock_io = Builder::new().read(&input[..]).build();\n    let mut req = HttpSession::new_http1(Box::new(mock_io));\n    req.read_request().await.unwrap();\n    assert_eq!(input.as_slice(), req.to_h1_raw());\n\n    let (mut subreq, _handle) = create_session(&req);\n    subreq.read_request().await.unwrap();\n    assert_eq!(input.as_slice(), subreq.to_h1_raw());\n}\n"
  },
  {
    "path": "pingora-proxy/src/subrequest/pipe.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! Subrequest piping.\n//!\n//! Along with subrequests themselves, subrequest piping as a feature is in\n//! alpha stages, APIs are highly unstable and subject to change at any point.\n//!\n//! Unlike proxy_*, it is not a \"true\" proxy mode; the functions here help\n//! establish a pipe between the main downstream session and the subrequest (which\n//! in most cases will be used as a downstream session itself).\n//!\n//! Furthermore, only downstream modules are invoked on the main downstream session,\n//! and the ProxyHttp trait filters are not run on the HttpTasks from the main session\n//! (the only relevant one being the request body filter).\n\nuse crate::proxy_common::{DownstreamStateMachine, ResponseStateMachine};\nuse crate::subrequest::*;\nuse crate::{PreparedSubrequest, Session};\nuse bytes::Bytes;\nuse futures::FutureExt;\nuse log::{debug, warn};\nuse pingora_core::protocols::http::{subrequest::server::SubrequestHandle, HttpTask};\nuse pingora_error::{Error, ErrorType::*, OrErr, Result};\nuse tokio::sync::mpsc;\n\npub enum InputBodyType {\n    /// Preset body\n    Preset(InputBody),\n    /// Body should be saved (up to limit)\n    SaveBody(usize),\n}\n\n/// Context struct as a result of subrequest piping.\n#[derive(Clone)]\npub struct PipeSubrequestState {\n    /// The saved (captured) body from the main session.\n    pub saved_body: Option<SavedBody>,\n}\n\nimpl PipeSubrequestState {\n    fn new() -> PipeSubrequestState {\n        PipeSubrequestState { saved_body: None }\n    }\n}\n\npub struct PipeSubrequestError {\n    pub state: PipeSubrequestState,\n    /// Whether error originated (and was propagated from) subrequest itself\n    /// (vs. an error that occurred while sending task)\n    pub from_subreq: bool,\n    pub error: Box<Error>,\n}\nimpl PipeSubrequestError {\n    pub fn new(\n        error: impl Into<Box<Error>>,\n        from_subreq: bool,\n        state: PipeSubrequestState,\n    ) -> Self {\n        PipeSubrequestError {\n            error: error.into(),\n            from_subreq,\n            state,\n        }\n    }\n}\n\nfn map_pipe_err<T, E: Into<Box<Error>>>(\n    result: Result<T, E>,\n    from_subreq: bool,\n    state: &PipeSubrequestState,\n) -> Result<T, PipeSubrequestError> {\n    result.map_err(|e| PipeSubrequestError::new(e, from_subreq, state.clone()))\n}\n\n#[derive(Debug, Clone)]\npub struct SavedBody {\n    body: Vec<Bytes>,\n    complete: bool,\n    truncated: bool,\n    length: usize,\n    max_length: usize,\n}\n\nimpl SavedBody {\n    pub fn new(max_length: usize) -> Self {\n        SavedBody {\n            body: vec![],\n            complete: false,\n            truncated: false,\n            length: 0,\n            max_length,\n        }\n    }\n\n    pub fn save_body_bytes(&mut self, body_bytes: Bytes) -> bool {\n        let len = body_bytes.len();\n        if self.length + len > self.max_length {\n            self.truncated = true;\n            return false;\n        }\n        self.length += len;\n        self.body.push(body_bytes);\n        true\n    }\n\n    pub fn is_body_complete(&self) -> bool {\n        self.complete && !self.truncated\n    }\n\n    pub fn set_body_complete(&mut self) {\n        self.complete = true;\n    }\n}\n\n#[derive(Debug, Clone)]\npub enum InputBody {\n    NoBody,\n    Bytes(Vec<Bytes>),\n    // TODO: stream\n}\n\nimpl InputBody {\n    pub(crate) fn into_reader(self) -> InputBodyReader {\n        InputBodyReader(match self {\n            InputBody::NoBody => vec![].into_iter(),\n            InputBody::Bytes(v) => v.into_iter(),\n        })\n    }\n\n    pub fn is_body_empty(&self) -> bool {\n        match self {\n            InputBody::NoBody => true,\n            InputBody::Bytes(v) => v.is_empty(),\n        }\n    }\n}\n\nimpl std::convert::From<SavedBody> for InputBody {\n    fn from(body: SavedBody) -> Self {\n        if body.body.is_empty() {\n            InputBody::NoBody\n        } else {\n            InputBody::Bytes(body.body)\n        }\n    }\n}\n\npub async fn pipe_subrequest<F>(\n    session: &mut Session,\n    mut subrequest: PreparedSubrequest,\n    subrequest_handle: SubrequestHandle,\n    mut task_filter: F,\n    input_body: InputBodyType,\n) -> std::result::Result<PipeSubrequestState, PipeSubrequestError>\nwhere\n    F: FnMut(HttpTask) -> Result<Option<HttpTask>>,\n{\n    let (maybe_preset_body, saved_body) = match input_body {\n        InputBodyType::Preset(body) => (Some(body), None),\n        InputBodyType::SaveBody(limit) => (None, Some(SavedBody::new(limit))),\n    };\n    let use_preset_body = maybe_preset_body.is_some();\n\n    let mut response_state = ResponseStateMachine::new();\n    let (no_body_input, mut maybe_preset_reader) = if use_preset_body {\n        let preset_body = maybe_preset_body.expect(\"checked above\");\n        (preset_body.is_body_empty(), Some(preset_body.into_reader()))\n    } else {\n        (session.as_mut().is_body_done(), None)\n    };\n    let mut downstream_state = DownstreamStateMachine::new(no_body_input);\n\n    let mut state = PipeSubrequestState::new();\n    state.saved_body = saved_body;\n\n    // Have the subrequest remove all body-related headers if no body will be sent\n    // TODO: we could also await the join handle, but subrequest may be running logging phase\n    // also the full run() may also await cache fill if downstream fails\n    let _join_handle = tokio::spawn(async move {\n        if no_body_input {\n            subrequest\n                .session_mut()\n                .as_subrequest_mut()\n                .expect(\"PreparedSubrequest must be subrequest\")\n                .clear_request_body_headers();\n        }\n        subrequest.run().await\n    });\n    let tx = subrequest_handle.tx;\n    let mut rx = subrequest_handle.rx;\n\n    let mut wants_body = false;\n    let mut wants_body_rx_err = false;\n    let mut wants_body_rx = subrequest_handle.subreq_wants_body;\n\n    let mut proxy_error_rx_err = false;\n    let mut proxy_error_rx = subrequest_handle.subreq_proxy_error;\n\n    // Note: \"upstream\" here refers to subrequest session tasks,\n    // downstream refers to main session\n    while !downstream_state.is_done() || !response_state.is_done() {\n        let send_permit = tx\n            .try_reserve()\n            .or_err(InternalError, \"try_reserve() body pipe for subrequest\");\n\n        tokio::select! {\n            task = rx.recv(), if !response_state.upstream_done() => {\n                debug!(\"upstream event: {:?}\", task);\n                if let Some(t) = task {\n                    // pull as many tasks as we can\n                    const TASK_BUFFER_SIZE: usize = 4;\n                    let mut tasks = Vec::with_capacity(TASK_BUFFER_SIZE);\n                    let task = map_pipe_err(task_filter(t), false, &state)?;\n                    if let Some(filtered) = task {\n                        tasks.push(filtered);\n                    }\n                    // tokio::task::unconstrained because now_or_never may yield None when the future is ready\n                    while let Some(maybe_task) = tokio::task::unconstrained(rx.recv()).now_or_never() {\n                        if let Some(t) = maybe_task {\n                            let task = map_pipe_err(task_filter(t), false, &state)?;\n                            if let Some(filtered) = task {\n                                tasks.push(filtered);\n                            }\n                        } else {\n                            break\n                        }\n                    }\n                    // FIXME: if one of these tasks is Failed(e), the session will return that\n                    // error; in this case, the error is actually from the subreq\n                    let response_done = map_pipe_err(session.write_response_tasks(tasks).await, false, &state)?;\n\n                    // NOTE: technically it is the downstream whose response state has finished here\n                    // we consider the subrequest's work done however\n                    response_state.maybe_set_upstream_done(response_done);\n                    // unsuccessful upgrade response may force the request done\n                    // (can only happen with a real session, TODO to allow with preset body)\n                    downstream_state.maybe_finished(!use_preset_body && session.is_body_done());\n                } else {\n                    // quite possible that the subrequest may be finished, though the main session\n                    // is not - we still must exit in this case\n                    debug!(\"empty upstream event\");\n                    response_state.maybe_set_upstream_done(true);\n                }\n            },\n\n            res = &mut wants_body_rx, if !wants_body && !wants_body_rx_err => {\n                // subrequest may need time before it needs body, or it may not actually require it\n                // TODO: tx send permit may not be necessary if no oneshot exists\n                if res.is_err() {\n                    wants_body_rx_err = true;\n                } else {\n                    wants_body = true;\n                }\n            }\n\n            res = &mut proxy_error_rx, if !proxy_error_rx_err => {\n                if let Ok(e) = res {\n                    // propagate proxy error to caller\n                    return Err(PipeSubrequestError::new(e, true, state));\n                } else {\n                    // subrequest dropped, let select loop finish\n                    proxy_error_rx_err = true;\n                }\n            }\n\n            _ = tx.reserve(), if downstream_state.is_reading() && send_permit.is_err() => {\n                // If tx is closed, the upstream has already finished its job.\n                downstream_state.maybe_finished(tx.is_closed());\n                debug!(\"waiting for permit {send_permit:?}, upstream closed {}\", tx.is_closed());\n                /* No permit, wait on more capacity to avoid starving.\n                 * Otherwise this select only blocks on rx, which might send no data\n                 * before the entire body is uploaded.\n                 * once more capacity arrives we just loop back\n                 */\n            },\n\n            body = session.downstream_session.read_body_or_idle(downstream_state.is_done()),\n                if wants_body && !use_preset_body && downstream_state.can_poll() && send_permit.is_ok() => {\n                // this is the first subrequest\n                // send the body\n                debug!(\"downstream event: main body for subrequest\");\n                let body = map_pipe_err(body.map_err(|e| e.into_down()), false, &state)?;\n\n                // If the request is websocket, `None` body means the request is closed.\n                // Set the response to be done as well so that the request completes normally.\n                if body.is_none() && session.is_upgrade_req() {\n                    response_state.maybe_set_upstream_done(true);\n                }\n\n                let is_body_done = session.is_body_done();\n                let request_done = map_pipe_err(send_body_to_pipe(\n                    session,\n                    body,\n                    is_body_done,\n                    state.saved_body.as_mut(),\n                    send_permit.expect(\"checked is_ok()\"),\n                )\n                .await, false, &state)?;\n\n                downstream_state.maybe_finished(request_done);\n\n            },\n\n            // lazily evaluated async block allows us to expect() inside the select! branch\n            body = async { maybe_preset_reader.as_mut().expect(\"preset body set\").read_body() },\n                if wants_body && use_preset_body && !downstream_state.is_done() && downstream_state.can_poll() && send_permit.is_ok() => {\n                debug!(\"downstream event: preset body for subrequest\");\n\n                // TODO: WebSocket handling to set upstream done?\n\n                // preset None body indicates we are done\n                let is_body_done = body.is_none();\n                // Don't run downstream modules on preset input body\n                let request_done = map_pipe_err(do_send_body_to_pipe(\n                    body,\n                    is_body_done,\n                    None,\n                    send_permit.expect(\"checked is_ok()\"),\n                ), false, &state)?;\n                downstream_state.maybe_finished(request_done);\n\n            },\n\n            else => break,\n        }\n    }\n    Ok(state)\n}\n\n// Mostly the same as proxy_common, but does not run proxy request_body_filter\nasync fn send_body_to_pipe(\n    session: &mut Session,\n    mut data: Option<Bytes>,\n    end_of_body: bool,\n    saved_body: Option<&mut SavedBody>,\n    tx: mpsc::Permit<'_, HttpTask>,\n) -> Result<bool> {\n    // None: end of body\n    // this var is to signal if downstream finish sending the body, which shouldn't be\n    // affected by the request_body_filter\n    let end_of_body = end_of_body || data.is_none();\n\n    session\n        .downstream_modules_ctx\n        .request_body_filter(&mut data, end_of_body)\n        .await?;\n\n    do_send_body_to_pipe(data, end_of_body, saved_body, tx)\n}\n\nfn do_send_body_to_pipe(\n    data: Option<Bytes>,\n    end_of_body: bool,\n    mut saved_body: Option<&mut SavedBody>,\n    tx: mpsc::Permit<'_, HttpTask>,\n) -> Result<bool> {\n    // the flag to signal to upstream\n    let upstream_end_of_body = end_of_body || data.is_none();\n\n    /* It is normal to get 0 bytes because of multi-chunk or request_body_filter decides not to\n     * output anything yet.\n     * Don't write 0 bytes to the network since it will be\n     * treated as the terminating chunk */\n    if !upstream_end_of_body && data.as_ref().is_some_and(|d| d.is_empty()) {\n        return Ok(false);\n    }\n\n    debug!(\n        \"Read {} bytes body from downstream\",\n        data.as_ref().map_or(-1, |d| d.len() as isize)\n    );\n\n    if let Some(capture) = saved_body.as_mut() {\n        if capture.is_body_complete() {\n            warn!(\"subrequest trying to save body after body is complete\");\n        } else if let Some(d) = data.as_ref() {\n            capture.save_body_bytes(d.clone());\n        }\n        if end_of_body {\n            capture.set_body_complete();\n        }\n    }\n\n    tx.send(HttpTask::Body(data, upstream_end_of_body));\n\n    Ok(end_of_body)\n}\n"
  },
  {
    "path": "pingora-proxy/tests/keys/key.pem",
    "content": "-----BEGIN EC PRIVATE KEY-----\nMHcCAQEEIN5lAOvtlKwtc/LR8/U77dohJmZS30OuezU9gL6vmm6DoAoGCCqGSM49\nAwEHoUQDQgAE2f/1Fm1HjySdokPq2T0F1xxol9nSEYQ+foFINeaWYk+FxMGpriJT\nBb8AGka87cWklw1ZqytfaT6pkureDbTkwg==\n-----END EC PRIVATE KEY-----\n"
  },
  {
    "path": "pingora-proxy/tests/keys/public.pem",
    "content": "-----BEGIN PUBLIC KEY-----\nMFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE2f/1Fm1HjySdokPq2T0F1xxol9nS\nEYQ+foFINeaWYk+FxMGpriJTBb8AGka87cWklw1ZqytfaT6pkureDbTkwg==\n-----END PUBLIC KEY-----\n"
  },
  {
    "path": "pingora-proxy/tests/keys/server.crt",
    "content": "-----BEGIN CERTIFICATE-----\nMIIB9zCCAZ2gAwIBAgIUMI7aLvTxyRFCHhw57hGt4U6yupcwCgYIKoZIzj0EAwIw\nZDELMAkGA1UEBhMCVVMxCzAJBgNVBAgMAkNBMRYwFAYDVQQHDA1TYW4gRnJhbmNp\nc2NvMRgwFgYDVQQKDA9DbG91ZGZsYXJlLCBJbmMxFjAUBgNVBAMMDW9wZW5ydXN0\neS5vcmcwHhcNMjIwNDExMjExMzEzWhcNMzIwNDA4MjExMzEzWjBkMQswCQYDVQQG\nEwJVUzELMAkGA1UECAwCQ0ExFjAUBgNVBAcMDVNhbiBGcmFuY2lzY28xGDAWBgNV\nBAoMD0Nsb3VkZmxhcmUsIEluYzEWMBQGA1UEAwwNb3BlbnJ1c3R5Lm9yZzBZMBMG\nByqGSM49AgEGCCqGSM49AwEHA0IABNn/9RZtR48knaJD6tk9BdccaJfZ0hGEPn6B\nSDXmlmJPhcTBqa4iUwW/ABpGvO3FpJcNWasrX2k+qZLq3g205MKjLTArMCkGA1Ud\nEQQiMCCCDyoub3BlbnJ1c3R5Lm9yZ4INb3BlbnJ1c3R5Lm9yZzAKBggqhkjOPQQD\nAgNIADBFAiAjISZ9aEKmobKGlT76idO740J6jPaX/hOrm41MLeg69AIhAJqKrSyz\nwD/AAF5fR6tXmBqlnpQOmtxfdy13wDr4MT3h\n-----END CERTIFICATE-----\n"
  },
  {
    "path": "pingora-proxy/tests/keys/server.csr",
    "content": "-----BEGIN CERTIFICATE REQUEST-----\nMIIBJzCBzgIBADBsMQswCQYDVQQGEwJVUzETMBEGA1UECAwKQ2FsaWZvcm5pYTEW\nMBQGA1UEBwwNU2FuIEZyYW5jaXNjbzEYMBYGA1UECgwPQ2xvdWRmbGFyZSwgSW5j\nMRYwFAYDVQQDDA1vcGVucnVzdHkub3JnMFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcD\nQgAE2f/1Fm1HjySdokPq2T0F1xxol9nSEYQ+foFINeaWYk+FxMGpriJTBb8AGka8\n7cWklw1ZqytfaT6pkureDbTkwqAAMAoGCCqGSM49BAMCA0gAMEUCIFyDN8eamnoY\nXydKn2oI7qImigxahyCftzjxkIEV5IKbAiEAo5l72X4U+YTVYmyPPnJIj2v5nA1R\nRuUfMh5sXzwlwuM=\n-----END CERTIFICATE REQUEST-----\n"
  },
  {
    "path": "pingora-proxy/tests/pingora_conf.yaml",
    "content": "---\nversion: 1\nclient_bind_to_ipv4:\n    - 127.0.0.2\nca_file: tests/keys/server.crt"
  },
  {
    "path": "pingora-proxy/tests/test_basic.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\nmod utils;\n\nuse bytes::Bytes;\nuse h2::client;\nuse http::Request;\nuse hyper::{body::HttpBody, header::HeaderValue, Body, Client};\n#[cfg(unix)]\nuse hyperlocal::{UnixClientExt, Uri};\nuse reqwest::{header, StatusCode};\nuse tokio::io::{AsyncReadExt, AsyncWriteExt};\nuse tokio::net::{TcpListener, TcpStream};\n\nuse utils::server_utils::init;\n\nfn is_specified_port(port: u16) -> bool {\n    (1..65535).contains(&port)\n}\n\n#[tokio::test]\nasync fn test_origin_alive() {\n    init();\n    let res = reqwest::get(\"http://127.0.0.1:8000/\").await.unwrap();\n    assert_eq!(res.status(), StatusCode::OK);\n    let headers = res.headers();\n    assert_eq!(headers[header::CONTENT_LENGTH], \"13\");\n    let body = res.text().await.unwrap();\n    assert_eq!(body, \"Hello World!\\n\");\n}\n\n#[tokio::test]\nasync fn test_simple_proxy() {\n    init();\n    let res = reqwest::get(\"http://127.0.0.1:6147\").await.unwrap();\n    assert_eq!(res.status(), StatusCode::OK);\n\n    let headers = res.headers();\n    assert_eq!(headers[header::CONTENT_LENGTH], \"13\");\n    assert_eq!(headers[\"x-server-addr\"], \"127.0.0.1:6147\");\n    let sockaddr = headers[\"x-client-addr\"]\n        .to_str()\n        .unwrap()\n        .parse::<std::net::SocketAddr>()\n        .unwrap();\n    assert_eq!(sockaddr.ip().to_string(), \"127.0.0.1\");\n    assert!(is_specified_port(sockaddr.port()));\n\n    assert_eq!(headers[\"x-upstream-server-addr\"], \"127.0.0.1:8000\");\n    let sockaddr = headers[\"x-upstream-client-addr\"]\n        .to_str()\n        .unwrap()\n        .parse::<std::net::SocketAddr>()\n        .unwrap();\n    assert_eq!(sockaddr.ip().to_string(), \"127.0.0.2\");\n    assert!(is_specified_port(sockaddr.port()));\n\n    let body = res.text().await.unwrap();\n    assert_eq!(body, \"Hello World!\\n\");\n}\n\n#[tokio::test]\n#[cfg(feature = \"any_tls\")]\nasync fn test_h2_to_h1() {\n    init();\n    let client = reqwest::Client::builder()\n        .danger_accept_invalid_certs(true)\n        .build()\n        .unwrap();\n\n    let res = client\n        .get(\"https://127.0.0.1:6150\")\n        .header(\"sni\", \"openrusty.org\")\n        .send()\n        .await\n        .unwrap();\n    assert_eq!(res.status(), reqwest::StatusCode::OK);\n    assert_eq!(res.version(), reqwest::Version::HTTP_2);\n\n    let headers = res.headers();\n    assert_eq!(headers[header::CONTENT_LENGTH], \"13\");\n    assert_eq!(headers[\"x-server-addr\"], \"127.0.0.1:6150\");\n\n    let sockaddr = headers[\"x-client-addr\"]\n        .to_str()\n        .unwrap()\n        .parse::<std::net::SocketAddr>()\n        .unwrap();\n    assert_eq!(sockaddr.ip().to_string(), \"127.0.0.1\");\n    assert!(is_specified_port(sockaddr.port()));\n\n    assert_eq!(headers[\"x-upstream-server-addr\"], \"127.0.0.1:8443\");\n    let sockaddr = headers[\"x-upstream-client-addr\"]\n        .to_str()\n        .unwrap()\n        .parse::<std::net::SocketAddr>()\n        .unwrap();\n    assert_eq!(sockaddr.ip().to_string(), \"127.0.0.2\");\n    assert!(is_specified_port(sockaddr.port()));\n\n    let body = res.text().await.unwrap();\n    assert_eq!(body, \"Hello World!\\n\");\n}\n\n#[tokio::test]\n#[cfg(feature = \"any_tls\")]\nasync fn test_h2_to_h2() {\n    init();\n    let client = reqwest::Client::builder()\n        .danger_accept_invalid_certs(true)\n        .build()\n        .unwrap();\n\n    let res = client\n        .get(\"https://127.0.0.1:6150\")\n        .header(\"sni\", \"openrusty.org\")\n        .header(\"x-h2\", \"true\")\n        .send()\n        .await\n        .unwrap();\n    assert_eq!(res.status(), reqwest::StatusCode::OK);\n    assert_eq!(res.version(), reqwest::Version::HTTP_2);\n\n    let headers = res.headers();\n    assert_eq!(headers[header::CONTENT_LENGTH], \"13\");\n    assert_eq!(headers[\"x-server-addr\"], \"127.0.0.1:6150\");\n    let sockaddr = headers[\"x-client-addr\"]\n        .to_str()\n        .unwrap()\n        .parse::<std::net::SocketAddr>()\n        .unwrap();\n    assert_eq!(sockaddr.ip().to_string(), \"127.0.0.1\");\n    assert!(is_specified_port(sockaddr.port()));\n\n    assert_eq!(headers[\"x-upstream-server-addr\"], \"127.0.0.1:8443\");\n    let sockaddr = headers[\"x-upstream-client-addr\"]\n        .to_str()\n        .unwrap()\n        .parse::<std::net::SocketAddr>()\n        .unwrap();\n    assert_eq!(sockaddr.ip().to_string(), \"127.0.0.2\");\n    assert!(is_specified_port(sockaddr.port()));\n\n    let body = res.text().await.unwrap();\n    assert_eq!(body, \"Hello World!\\n\");\n}\n\n#[tokio::test]\nasync fn test_h2c_to_h2c() {\n    init();\n\n    let client = hyper::client::Client::builder()\n        .http2_only(true)\n        .build_http();\n\n    let mut req = hyper::Request::builder()\n        .uri(\"http://127.0.0.1:6146\")\n        .body(Body::empty())\n        .unwrap();\n    req.headers_mut()\n        .insert(\"x-h2\", HeaderValue::from_bytes(b\"true\").unwrap());\n    let res = client.request(req).await.unwrap();\n    assert_eq!(res.status(), reqwest::StatusCode::OK);\n    assert_eq!(res.version(), reqwest::Version::HTTP_2);\n\n    let body = res.into_body().data().await.unwrap().unwrap();\n    assert_eq!(body.as_ref(), b\"Hello World!\\n\");\n}\n\n#[tokio::test]\nasync fn test_h1_on_h2c_port() {\n    init();\n\n    let client = hyper::client::Client::builder()\n        .http2_only(false)\n        .build_http();\n\n    let mut req = hyper::Request::builder()\n        .uri(\"http://127.0.0.1:6146\")\n        .body(Body::empty())\n        .unwrap();\n    req.headers_mut()\n        .insert(\"x-h2\", HeaderValue::from_bytes(b\"true\").unwrap());\n    let res = client.request(req).await.unwrap();\n    assert_eq!(res.status(), reqwest::StatusCode::OK);\n    assert_eq!(res.version(), reqwest::Version::HTTP_11);\n\n    let body = res.into_body().data().await.unwrap().unwrap();\n    assert_eq!(body.as_ref(), b\"Hello World!\\n\");\n}\n\n#[tokio::test]\n#[cfg(feature = \"openssl_derived\")]\nasync fn test_h2_to_h2_host_override() {\n    init();\n    let client = reqwest::Client::builder()\n        .danger_accept_invalid_certs(true)\n        .build()\n        .unwrap();\n\n    let res = client\n        .get(\"https://127.0.0.1:6150\")\n        .header(\"x-h2\", \"true\")\n        .header(\"host-override\", \"test.com\")\n        .send()\n        .await\n        .unwrap();\n    assert_eq!(res.status(), reqwest::StatusCode::OK);\n    assert_eq!(res.version(), reqwest::Version::HTTP_2);\n    let headers = res.headers();\n    assert_eq!(headers[header::CONTENT_LENGTH], \"13\");\n    let body = res.text().await.unwrap();\n    assert_eq!(body, \"Hello World!\\n\");\n}\n\n#[tokio::test]\n#[cfg(feature = \"any_tls\")]\nasync fn test_h2_to_h2_upload() {\n    init();\n    let client = reqwest::Client::builder()\n        .danger_accept_invalid_certs(true)\n        .build()\n        .unwrap();\n\n    let payload = \"test upload\";\n\n    let res = client\n        .get(\"https://127.0.0.1:6150/echo\")\n        .header(\"sni\", \"openrusty.org\")\n        .header(\"x-h2\", \"true\")\n        .body(payload)\n        .send()\n        .await\n        .unwrap();\n    assert_eq!(res.status(), reqwest::StatusCode::OK);\n    assert_eq!(res.version(), reqwest::Version::HTTP_2);\n    let body = res.text().await.unwrap();\n    assert_eq!(body, payload);\n}\n\n#[tokio::test]\n#[cfg(feature = \"any_tls\")]\nasync fn test_h2_to_h1_upload() {\n    init();\n    let client = reqwest::Client::builder()\n        .danger_accept_invalid_certs(true)\n        .build()\n        .unwrap();\n\n    let payload = \"test upload\";\n\n    let res = client\n        .get(\"https://127.0.0.1:6150/echo\")\n        .header(\"sni\", \"openrusty.org\")\n        .body(payload)\n        .send()\n        .await\n        .unwrap();\n    assert_eq!(res.status(), reqwest::StatusCode::OK);\n    assert_eq!(res.version(), reqwest::Version::HTTP_2);\n    let body = res.text().await.unwrap();\n    assert_eq!(body, payload);\n}\n\n#[tokio::test]\n#[cfg(feature = \"any_tls\")]\nasync fn test_h2_head() {\n    init();\n    let client = reqwest::Client::builder()\n        .danger_accept_invalid_certs(true)\n        .build()\n        .unwrap();\n\n    let res = client\n        .head(\"https://127.0.0.1:6150/set_content_length\")\n        .header(\"sni\", \"openrusty.org\")\n        .header(\"x-h2\", \"true\")\n        .header(\"x-set-content-length\", \"11\")\n        .send()\n        .await\n        .unwrap();\n    assert_eq!(res.status(), reqwest::StatusCode::OK);\n    assert_eq!(res.version(), reqwest::Version::HTTP_2);\n    let body = res.text().await.unwrap();\n    // should not be any body, despite content-length\n    assert_eq!(body, \"\");\n}\n\n#[cfg(unix)]\n#[tokio::test]\nasync fn test_simple_proxy_uds() {\n    init();\n    let url = Uri::new(\"/tmp/pingora_proxy.sock\", \"/\").into();\n    let client = Client::unix();\n\n    let res = client.get(url).await.unwrap();\n\n    assert_eq!(res.status(), reqwest::StatusCode::OK);\n    let (resp, body) = res.into_parts();\n\n    let headers = &resp.headers;\n    assert_eq!(headers[header::CONTENT_LENGTH], \"13\");\n    assert_eq!(headers[\"x-server-addr\"], \"/tmp/pingora_proxy.sock\");\n    assert_eq!(headers[\"x-client-addr\"], \"unset\"); // unnamed UDS\n\n    assert_eq!(headers[\"x-upstream-server-addr\"], \"127.0.0.1:8000\");\n    let sockaddr = headers[\"x-upstream-client-addr\"]\n        .to_str()\n        .unwrap()\n        .parse::<std::net::SocketAddr>()\n        .unwrap();\n    assert_eq!(sockaddr.ip().to_string(), \"127.0.0.2\");\n    assert!(is_specified_port(sockaddr.port()));\n\n    let body = hyper::body::to_bytes(body).await.unwrap();\n    assert_eq!(body.as_ref(), b\"Hello World!\\n\");\n}\n\n#[cfg(unix)]\n#[tokio::test]\nasync fn test_simple_proxy_uds_peer() {\n    init();\n    let client = reqwest::Client::new();\n\n    let res = client\n        .get(\"http://127.0.0.1:6147\")\n        .header(\"x-uds-peer\", \"1\") // force upstream peer to be UDS\n        .send()\n        .await\n        .unwrap();\n\n    assert_eq!(res.status(), StatusCode::OK);\n\n    let headers = &res.headers();\n    assert_eq!(headers[header::CONTENT_LENGTH], \"13\");\n    assert_eq!(headers[\"x-server-addr\"], \"127.0.0.1:6147\");\n    let sockaddr = headers[\"x-client-addr\"]\n        .to_str()\n        .unwrap()\n        .parse::<std::net::SocketAddr>()\n        .unwrap();\n    assert_eq!(sockaddr.ip().to_string(), \"127.0.0.1\");\n    assert!(is_specified_port(sockaddr.port()));\n\n    assert_eq!(headers[\"x-upstream-client-addr\"], \"unset\"); // unnamed UDS\n    assert_eq!(\n        headers[\"x-upstream-server-addr\"],\n        \"/tmp/pingora_nginx_test.sock\"\n    );\n\n    let body = res.text().await.unwrap();\n    assert_eq!(body, \"Hello World!\\n\");\n}\n\nasync fn test_dropped_conn_get() {\n    init();\n    let client = reqwest::Client::new();\n    let port = \"8001\"; // special port to avoid unexpected connection reuse from other tests\n\n    for _ in 1..3 {\n        // load conns into pool\n        let res = client\n            .get(\"http://127.0.0.1:6147\")\n            .header(\"x-port\", port)\n            .send()\n            .await\n            .unwrap();\n        assert_eq!(res.status(), StatusCode::OK);\n    }\n\n    let res = client\n        .get(\"http://127.0.0.1:6147/bad_lb\")\n        .header(\"x-port\", port)\n        .send()\n        .await\n        .unwrap();\n\n    // retry gives 200\n    assert_eq!(res.status(), StatusCode::OK);\n    let body = res.text().await.unwrap();\n    assert_eq!(body, \"dog!\\n\");\n}\n\nasync fn test_dropped_conn_post_empty_body() {\n    init();\n    let client = reqwest::Client::new();\n    let port = \"8001\"; // special port to avoid unexpected connection reuse from other tests\n\n    for _ in 1..3 {\n        // load conn into pool\n        let res = client\n            .get(\"http://127.0.0.1:6147\")\n            .header(\"x-port\", port)\n            .send()\n            .await\n            .unwrap();\n        assert_eq!(res.status(), StatusCode::OK);\n    }\n\n    let res = client\n        .post(\"http://127.0.0.1:6147/bad_lb\")\n        .header(\"x-port\", port)\n        .send()\n        .await\n        .unwrap();\n\n    assert_eq!(res.status(), StatusCode::OK);\n    let body = res.text().await.unwrap();\n    assert_eq!(body, \"dog!\\n\");\n}\n\nasync fn test_dropped_conn_post_body() {\n    init();\n    let client = reqwest::Client::new();\n    let port = \"8001\"; // special port to avoid unexpected connection reuse from other tests\n\n    for _ in 1..3 {\n        // load conn into pool\n        let res = client\n            .get(\"http://127.0.0.1:6147\")\n            .header(\"x-port\", port)\n            .send()\n            .await\n            .unwrap();\n        assert_eq!(res.status(), StatusCode::OK);\n    }\n\n    let res = client\n        .post(\"http://127.0.0.1:6147/bad_lb\")\n        .header(\"x-port\", port)\n        .body(\"cat!\")\n        .send()\n        .await\n        .unwrap();\n\n    assert_eq!(res.status(), StatusCode::OK);\n    let body = res.text().await.unwrap();\n    assert_eq!(body, \"cat!\\n\");\n}\n\nasync fn test_dropped_conn_post_body_over() {\n    init();\n    let client = reqwest::Client::new();\n    let port = \"8001\"; // special port to avoid unexpected connection reuse from other tests\n    let large_body = String::from_utf8(vec![b'e'; 1024 * 64 + 1]).unwrap();\n\n    for _ in 1..3 {\n        // load conn into pool\n        let res = client\n            .get(\"http://127.0.0.1:6147\")\n            .header(\"x-port\", port)\n            .send()\n            .await\n            .unwrap();\n        assert_eq!(res.status(), StatusCode::OK);\n    }\n\n    let res = client\n        .post(\"http://127.0.0.1:6147/bad_lb\")\n        .header(\"x-port\", port)\n        .body(large_body)\n        .send()\n        .await\n        .unwrap();\n\n    // 502, body larger than buffer limit\n    assert_eq!(res.status(), StatusCode::from_u16(502).unwrap());\n}\n\n#[tokio::test]\nasync fn test_dropped_conn() {\n    // These tests can race with each other\n    // So force run them sequentially\n    test_dropped_conn_get().await;\n    test_dropped_conn_post_empty_body().await;\n    test_dropped_conn_post_body().await;\n    test_dropped_conn_post_body_over().await;\n}\n\n// currently not supported with Rustls implementation\n#[cfg(feature = \"openssl_derived\")]\n#[tokio::test]\nasync fn test_tls_no_verify() {\n    init();\n    let client = reqwest::Client::new();\n    let res = client\n        .get(\"http://127.0.0.1:6149/tls_verify\")\n        .send()\n        .await\n        .unwrap();\n\n    assert_eq!(res.status(), StatusCode::OK);\n}\n\n#[cfg(feature = \"any_tls\")]\n#[tokio::test]\nasync fn test_tls_verify_sni_not_host() {\n    init();\n    let client = reqwest::Client::new();\n\n    let res = client\n        .get(\"http://127.0.0.1:6149/tls_verify\")\n        .header(\"sni\", \"openrusty.org\")\n        .header(\"verify\", \"1\")\n        .send()\n        .await\n        .unwrap();\n\n    assert_eq!(res.status(), StatusCode::OK);\n}\n\n// currently not supported with Rustls implementation\n#[cfg(feature = \"openssl_derived\")]\n#[tokio::test]\nasync fn test_tls_none_verify_host() {\n    init();\n    let client = reqwest::Client::new();\n\n    let res = client\n        .get(\"http://127.0.0.1:6149/tls_verify\")\n        .header(\"verify\", \"1\")\n        .header(\"verify_host\", \"1\")\n        .send()\n        .await\n        .unwrap();\n\n    assert_eq!(res.status(), StatusCode::OK);\n}\n\n#[cfg(feature = \"any_tls\")]\n#[tokio::test]\nasync fn test_tls_verify_sni_host() {\n    init();\n    let client = reqwest::Client::new();\n\n    let res = client\n        .get(\"http://127.0.0.1:6149/tls_verify\")\n        .header(\"sni\", \"openrusty.org\")\n        .header(\"verify\", \"1\")\n        .header(\"verify_host\", \"1\")\n        .send()\n        .await\n        .unwrap();\n\n    assert_eq!(res.status(), StatusCode::OK);\n}\n\n#[cfg(feature = \"any_tls\")]\n#[tokio::test]\nasync fn test_tls_underscore_sub_sni_verify_host() {\n    init();\n    let client = reqwest::Client::new();\n\n    let res = client\n        .get(\"http://127.0.0.1:6149/tls_verify\")\n        .header(\"sni\", \"d_g.openrusty.org\")\n        .header(\"verify\", \"1\")\n        .header(\"verify_host\", \"1\")\n        .send()\n        .await\n        .unwrap();\n\n    assert_eq!(res.status(), StatusCode::OK);\n}\n\n#[cfg(feature = \"any_tls\")]\n#[tokio::test]\nasync fn test_tls_underscore_non_sub_sni_verify_host() {\n    init();\n    let client = reqwest::Client::new();\n\n    let res = client\n        .get(\"http://127.0.0.1:6149/tls_verify\")\n        .header(\"sni\", \"open_rusty.org\")\n        .header(\"verify\", \"1\")\n        .header(\"verify_host\", \"1\")\n        .send()\n        .await\n        .unwrap();\n\n    assert_eq!(res.status(), StatusCode::BAD_GATEWAY);\n    let headers = res.headers();\n    assert_eq!(headers[header::CONNECTION], \"close\");\n}\n\n#[cfg(feature = \"openssl_derived\")]\n#[tokio::test]\nasync fn test_tls_alt_verify_host() {\n    init();\n    let client = reqwest::Client::new();\n\n    let res = client\n        .get(\"http://127.0.0.1:6149/tls_verify\")\n        .header(\"sni\", \"open_rusty.org\")\n        .header(\"alt\", \"openrusty.org\")\n        .header(\"verify\", \"1\")\n        .header(\"verify_host\", \"1\")\n        .send()\n        .await\n        .unwrap();\n\n    assert_eq!(res.status(), StatusCode::OK);\n}\n\n#[cfg(feature = \"openssl_derived\")]\n#[tokio::test]\nasync fn test_tls_underscore_sub_alt_verify_host() {\n    init();\n    let client = reqwest::Client::new();\n\n    let res = client\n        .get(\"http://127.0.0.1:6149/tls_verify\")\n        .header(\"sni\", \"open_rusty.org\")\n        .header(\"alt\", \"d_g.openrusty.org\")\n        .header(\"verify\", \"1\")\n        .header(\"verify_host\", \"1\")\n        .send()\n        .await\n        .unwrap();\n\n    assert_eq!(res.status(), StatusCode::OK);\n}\n\n#[cfg(feature = \"any_tls\")]\n#[tokio::test]\nasync fn test_tls_underscore_non_sub_alt_verify_host() {\n    init();\n    let client = reqwest::Client::new();\n\n    let res = client\n        .get(\"http://127.0.0.1:6149/tls_verify\")\n        .header(\"sni\", \"open_rusty.org\")\n        .header(\"alt\", \"open_rusty.org\")\n        .header(\"verify\", \"1\")\n        .header(\"verify_host\", \"1\")\n        .send()\n        .await\n        .unwrap();\n\n    assert_eq!(res.status(), StatusCode::BAD_GATEWAY);\n}\n\n#[tokio::test]\nasync fn test_upstream_compression() {\n    init();\n\n    // disable reqwest gzip support to check compression headers and body\n    // otherwise reqwest will decompress and strip the headers\n    let client = reqwest::ClientBuilder::new().gzip(false).build().unwrap();\n    let res = client\n        .get(\"http://127.0.0.1:6147/no_compression\")\n        .header(\"accept-encoding\", \"gzip\")\n        .send()\n        .await\n        .unwrap();\n    assert_eq!(res.status(), StatusCode::OK);\n    assert_eq!(res.headers().get(\"Content-Encoding\").unwrap(), \"gzip\");\n    let body = res.bytes().await.unwrap();\n    assert!(body.len() < 32);\n\n    // Next let reqwest decompress to validate the data\n    let client = reqwest::ClientBuilder::new().gzip(true).build().unwrap();\n    let res = client\n        .get(\"http://127.0.0.1:6147/no_compression\")\n        .header(\"accept-encoding\", \"gzip\")\n        .send()\n        .await\n        .unwrap();\n    assert_eq!(res.status(), StatusCode::OK);\n    let body = res.bytes().await.unwrap();\n    assert_eq!(body.as_ref(), &[b'B'; 32]);\n}\n\n#[tokio::test]\nasync fn test_downstream_compression() {\n    init();\n\n    // disable reqwest gzip support to check compression headers and body\n    // otherwise reqwest will decompress and strip the headers\n    let client = reqwest::ClientBuilder::new().gzip(false).build().unwrap();\n    let res = client\n        .get(\"http://127.0.0.1:6147/no_compression\")\n        // tell the test proxy to use downstream compression module instead of upstream\n        .header(\"x-downstream-compression\", \"1\")\n        .header(\"accept-encoding\", \"gzip\")\n        .send()\n        .await\n        .unwrap();\n    assert_eq!(res.status(), StatusCode::OK);\n    assert_eq!(res.headers().get(\"Content-Encoding\").unwrap(), \"gzip\");\n    let body = res.bytes().await.unwrap();\n    assert!(body.len() < 32);\n\n    // Next let reqwest decompress to validate the data\n    let client = reqwest::ClientBuilder::new().gzip(true).build().unwrap();\n    let res = client\n        .get(\"http://127.0.0.1:6147/no_compression\")\n        .header(\"accept-encoding\", \"gzip\")\n        .send()\n        .await\n        .unwrap();\n    assert_eq!(res.status(), StatusCode::OK);\n    let body = res.bytes().await.unwrap();\n    assert_eq!(body.as_ref(), &[b'B'; 32]);\n}\n\n#[tokio::test]\nasync fn test_connect_close() {\n    init();\n\n    // default keep-alive\n    let client = reqwest::ClientBuilder::new().build().unwrap();\n    let res = client.get(\"http://127.0.0.1:6147\").send().await.unwrap();\n    assert_eq!(res.status(), StatusCode::OK);\n    let headers = res.headers();\n    assert_eq!(headers[header::CONTENT_LENGTH], \"13\");\n    assert_eq!(headers[header::CONNECTION], \"keep-alive\");\n    let body = res.text().await.unwrap();\n    assert_eq!(body, \"Hello World!\\n\");\n\n    // close\n    let client = reqwest::ClientBuilder::new().build().unwrap();\n    let res = client\n        .get(\"http://127.0.0.1:6147\")\n        .header(\"connection\", \"close\")\n        .send()\n        .await\n        .unwrap();\n    assert_eq!(res.status(), StatusCode::OK);\n    let headers = res.headers();\n    assert_eq!(headers[header::CONTENT_LENGTH], \"13\");\n    assert_eq!(headers[header::CONNECTION], \"close\");\n    let body = res.text().await.unwrap();\n    assert_eq!(body, \"Hello World!\\n\");\n}\n\n#[tokio::test]\nasync fn test_connect_proxying_disallowed_h1() {\n    init();\n\n    let mut stream = TcpStream::connect(\"127.0.0.1:6147\").await.unwrap();\n    let request = b\"CONNECT pingora.org:443 HTTP/1.1\\r\\nHost: pingora.org:443\\r\\n\\r\\n\";\n    stream.write_all(request).await.unwrap();\n\n    let mut buf = [0u8; 1024];\n    let read = stream.read(&mut buf).await.unwrap();\n    let resp = std::str::from_utf8(&buf[..read]).unwrap();\n    let status_line = resp.lines().next().unwrap_or(\"\");\n    assert!(status_line.contains(\" 405 \"));\n}\n\n#[tokio::test]\nasync fn test_connect_proxying_disallowed_h2() {\n    init();\n\n    let tcp = TcpStream::connect(\"127.0.0.1:6146\").await.unwrap();\n    let (mut h2, connection) = client::handshake(tcp).await.unwrap();\n    tokio::spawn(async move {\n        connection.await.unwrap();\n    });\n\n    let request = Request::builder()\n        .method(\"CONNECT\")\n        .uri(\"http://pingora.org:443/\")\n        .body(())\n        .unwrap();\n    let (response, _body) = h2.send_request(request, true).unwrap();\n    let (head, mut body) = response.await.unwrap().into_parts();\n    assert_eq!(head.status.as_u16(), 405);\n    while let Some(chunk) = body.data().await {\n        assert!(chunk.unwrap().is_empty());\n    }\n}\n\n#[tokio::test]\nasync fn test_connect_proxying_allowed_h1() {\n    init();\n\n    let listener = TcpListener::bind(\"127.0.0.1:0\").await.unwrap();\n    let upstream_addr = listener.local_addr().unwrap();\n\n    // Note per RFC CONNECT 2xx responses are not allowed to have response\n    // bodies, so this is non-standard behavior.\n    tokio::spawn(async move {\n        let (mut socket, _) = listener.accept().await.unwrap();\n        let mut buf = [0u8; 1024];\n        let _ = socket.read(&mut buf).await.unwrap();\n        let response = b\"HTTP/1.1 200 OK\\r\\nContent-Length: 2\\r\\n\\r\\nok\";\n        socket.write_all(response).await.unwrap();\n        let _ = socket.shutdown().await;\n    });\n\n    let mut stream = TcpStream::connect(\"127.0.0.1:6160\").await.unwrap();\n    let request = format!(\n        \"CONNECT pingora.org:443 HTTP/1.1\\r\\nHost: pingora.org:443\\r\\nX-Port: {}\\r\\n\\r\\n\",\n        upstream_addr.port()\n    );\n    stream.write_all(request.as_bytes()).await.unwrap();\n\n    let mut buf = vec![0u8; 1024];\n    let read = stream.read(&mut buf).await.unwrap();\n    let resp = std::str::from_utf8(&buf[..read]).unwrap();\n    let status_line = resp.lines().next().unwrap_or(\"\");\n    assert!(status_line.contains(\" 200 \"));\n    assert!(resp.ends_with(\"ok\"));\n}\n\n#[tokio::test]\n#[cfg(feature = \"any_tls\")]\nasync fn test_mtls_no_client_cert() {\n    init();\n    let client = reqwest::Client::new();\n\n    let res = client\n        .get(\"http://127.0.0.1:6149/tls_verify\")\n        .header(\"x-port\", \"8444\")\n        .header(\"sni\", \"openrusty.org\")\n        .header(\"verify\", \"1\")\n        .header(\"verify_host\", \"1\")\n        .send()\n        .await\n        .unwrap();\n\n    // 400: because no cert\n    assert_eq!(res.status(), StatusCode::BAD_REQUEST);\n}\n\n#[cfg(feature = \"any_tls\")]\n#[tokio::test]\nasync fn test_mtls_no_intermediate_cert() {\n    init();\n    let client = reqwest::Client::new();\n\n    let res = client\n        .get(\"http://127.0.0.1:6149/tls_verify\")\n        .header(\"x-port\", \"8444\")\n        .header(\"sni\", \"openrusty.org\")\n        .header(\"verify\", \"1\")\n        .header(\"verify_host\", \"1\")\n        .header(\"client_cert\", \"1\")\n        .send()\n        .await\n        .unwrap();\n\n    // 400: because no intermediate cert\n    assert_eq!(res.status(), StatusCode::BAD_REQUEST);\n}\n\n#[tokio::test]\n#[cfg(feature = \"any_tls\")]\nasync fn test_mtls() {\n    init();\n    let client = reqwest::Client::new();\n\n    let res = client\n        .get(\"http://127.0.0.1:6149/\")\n        .header(\"x-port\", \"8444\")\n        .header(\"sni\", \"openrusty.org\")\n        .header(\"verify\", \"1\")\n        .header(\"verify_host\", \"1\")\n        .header(\"client_cert\", \"1\")\n        .header(\"client_intermediate\", \"1\")\n        .send()\n        .await\n        .unwrap();\n    assert_eq!(res.status(), StatusCode::OK);\n}\n\n#[cfg(feature = \"any_tls\")]\nasync fn assert_reuse(req: reqwest::RequestBuilder) {\n    req.try_clone().unwrap().send().await.unwrap();\n    let res = req.send().await.unwrap();\n    let headers = res.headers();\n    assert!(headers.get(\"x-conn-reuse\").is_some());\n}\n\n#[cfg(feature = \"any_tls\")]\n#[tokio::test]\nasync fn test_mtls_diff_cert_no_reuse() {\n    init();\n    let client = reqwest::Client::new();\n\n    let req = client\n        .get(\"http://127.0.0.1:6149/\")\n        .header(\"x-port\", \"8444\")\n        .header(\"sni\", \"openrusty.org\")\n        .header(\"verify\", \"1\")\n        .header(\"verify_host\", \"1\")\n        .header(\"client_cert\", \"1\")\n        .header(\"client_intermediate\", \"1\");\n\n    // pre check re-use\n    assert_reuse(req).await;\n\n    // different cert no re-use\n    let res = client\n        .get(\"http://127.0.0.1:6149/\")\n        .header(\"x-port\", \"8444\")\n        .header(\"sni\", \"openrusty.org\")\n        .header(\"verify\", \"1\")\n        .header(\"verify_host\", \"1\")\n        .header(\"client_cert\", \"2\")\n        .header(\"client_intermediate\", \"1\")\n        .send()\n        .await\n        .unwrap();\n    assert_eq!(res.status(), StatusCode::OK);\n    let headers = res.headers();\n    assert!(headers.get(\"x-conn-reuse\").is_none());\n}\n\n#[cfg(feature = \"any_tls\")]\n#[tokio::test]\nasync fn test_tls_diff_verify_no_reuse() {\n    init();\n    let client = reqwest::Client::new();\n\n    let req = client\n        .get(\"http://127.0.0.1:6149/\")\n        .header(\"sni\", \"dog.openrusty.org\")\n        .header(\"verify\", \"1\");\n\n    // pre check re-use\n    assert_reuse(req).await;\n\n    // disable 'verify' no re-use\n    let res = client\n        .get(\"http://127.0.0.1:6149/\")\n        .header(\"sni\", \"dog.openrusty.org\")\n        .header(\"verify\", \"0\")\n        .send()\n        .await\n        .unwrap();\n    assert_eq!(res.status(), StatusCode::OK);\n    let headers = res.headers();\n    assert!(headers.get(\"x-conn-reuse\").is_none());\n}\n\n#[cfg(feature = \"any_tls\")]\n#[tokio::test]\nasync fn test_tls_diff_verify_host_no_reuse() {\n    init();\n    let client = reqwest::Client::new();\n\n    let req = client\n        .get(\"http://127.0.0.1:6149/\")\n        .header(\"sni\", \"cat.openrusty.org\")\n        .header(\"verify\", \"1\")\n        .header(\"verify_host\", \"1\");\n\n    // pre check re-use\n    assert_reuse(req).await;\n\n    // disable 'verify_host' no re-use\n    let res = client\n        .get(\"http://127.0.0.1:6149/\")\n        .header(\"sni\", \"cat.openrusty.org\")\n        .header(\"verify\", \"1\")\n        .header(\"verify_host\", \"0\")\n        .send()\n        .await\n        .unwrap();\n    assert_eq!(res.status(), StatusCode::OK);\n    let headers = res.headers();\n    assert!(headers.get(\"x-conn-reuse\").is_none());\n}\n\n#[cfg(feature = \"any_tls\")]\n#[tokio::test]\nasync fn test_tls_diff_alt_cnt_no_reuse() {\n    init();\n    let client = reqwest::Client::new();\n\n    let req = client\n        .get(\"http://127.0.0.1:6149/\")\n        .header(\"sni\", \"openrusty.org\")\n        .header(\"alt\", \"cat.com\")\n        .header(\"verify\", \"1\")\n        .header(\"verify_host\", \"1\");\n\n    // pre check re-use\n    assert_reuse(req).await;\n\n    // use alt-cn no reuse\n    let res = client\n        .get(\"http://127.0.0.1:6149/\")\n        .header(\"sni\", \"openrusty.org\")\n        .header(\"alt\", \"dog.com\")\n        .header(\"verify\", \"1\")\n        .header(\"verify_host\", \"1\")\n        .send()\n        .await\n        .unwrap();\n    assert_eq!(res.status(), StatusCode::OK);\n    let headers = res.headers();\n    assert!(headers.get(\"x-conn-reuse\").is_none());\n}\n\n#[cfg(feature = \"s2n\")]\n#[tokio::test]\nasync fn test_tls_psk() {\n    use crate::utils::server_utils::TEST_PSK_IDENTITY;\n\n    init();\n    let client = reqwest::Client::new();\n\n    let res = client\n        .get(\"http://127.0.0.1:6149/\")\n        .header(\"sni\", \"openrusty.org\")\n        .header(\"psk_identity\", TEST_PSK_IDENTITY)\n        .header(\"x-port\", \"6151\")\n        .send()\n        .await\n        .unwrap();\n    assert_eq!(res.status(), StatusCode::OK);\n}\n\n#[cfg(feature = \"s2n\")]\n#[tokio::test]\nasync fn test_tls_psk_invalid() {\n    init();\n    let client = reqwest::Client::new();\n\n    let res = client\n        .get(\"http://127.0.0.1:6149/\")\n        .header(\"sni\", \"openrusty.org\")\n        .header(\"psk_identity\", \"BAD_IDENTITY\")\n        .header(\"x-port\", \"6151\")\n        .send()\n        .await\n        .unwrap();\n    assert_eq!(res.status(), StatusCode::BAD_GATEWAY);\n}\n\n#[tokio::test]\nasync fn test_error_before_headers_sent() {\n    init();\n    let url = \"http://127.0.0.1:6146/sleep/test_error_before_headers_sent.txt\";\n\n    let tcp = TcpStream::connect(\"127.0.0.1:6146\").await.unwrap();\n    let (mut client, h2) = client::handshake(tcp).await.unwrap();\n\n    tokio::spawn(async move {\n        h2.await.unwrap();\n    });\n\n    let request = Request::builder()\n        .uri(url)\n        .header(\"x-set-sleep\", \"0\")\n        .header(\"x-abort\", \"true\")\n        .body(())\n        .unwrap();\n\n    let (response, mut _stream) = client.send_request(request, true).unwrap();\n\n    let response = response.await.unwrap();\n    let mut body = response.into_body();\n\n    while let Some(chunk) = body.data().await {\n        assert_eq!(chunk.unwrap(), Bytes::new());\n    }\n}\n\n#[tokio::test]\nasync fn test_error_after_headers_sent_rst_received() {\n    init();\n    let url = \"http://127.0.0.1:6146/connection_die/test_error_after_headers_sent_rst_received.txt\";\n\n    let tcp = TcpStream::connect(\"127.0.0.1:6146\").await.unwrap();\n    let (mut client, h2) = client::handshake(tcp).await.unwrap();\n\n    tokio::spawn(async move {\n        h2.await.unwrap();\n    });\n\n    let request = Request::builder().uri(url).body(()).unwrap();\n\n    let (response, mut _stream) = client.send_request(request, true).unwrap();\n\n    let response = response.await.unwrap();\n    let mut body = response.into_body();\n\n    let chunk = body.data().await.unwrap();\n    assert_eq!(chunk.unwrap(), Bytes::from_static(b\"AAAAA\"));\n\n    let err = body.data().await.unwrap().err().unwrap();\n    assert_eq!(err.reason().unwrap(), h2::Reason::CANCEL);\n}\n\n#[tokio::test]\nasync fn test_103() {\n    init();\n    let res = reqwest::get(\"http://127.0.0.1:6147/103\").await.unwrap();\n    assert_eq!(res.status(), StatusCode::OK);\n    let headers = res.headers();\n    assert_eq!(headers[header::CONTENT_LENGTH], \"6\");\n    let body = res.text().await.unwrap();\n    assert_eq!(body, \"123456\");\n}\n\n#[tokio::test]\nasync fn test_103_die() {\n    init();\n    let res = reqwest::get(\"http://127.0.0.1:6147/103-die\").await.unwrap();\n    assert_eq!(res.status(), StatusCode::BAD_GATEWAY);\n}\n"
  },
  {
    "path": "pingora-proxy/tests/test_upstream.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\nmod utils;\n\nuse utils::server_utils::init;\nuse utils::websocket::{WS_ECHO, WS_ECHO_RAW};\n\nuse futures::{SinkExt, StreamExt};\nuse pingora_http::ResponseHeader;\nuse reqwest::header::{HeaderName, HeaderValue};\nuse reqwest::{StatusCode, Version};\nuse std::time::{Duration, Instant};\nuse tokio::io::{AsyncReadExt, AsyncWriteExt};\nuse tokio::net::TcpStream;\nuse tokio::time::timeout;\nuse tokio_tungstenite::tungstenite::{client::IntoClientRequest, Message};\n\n#[tokio::test]\nasync fn test_ip_binding() {\n    init();\n    let res = reqwest::get(\"http://127.0.0.1:6147/client_ip\")\n        .await\n        .unwrap();\n    assert_eq!(res.status(), StatusCode::OK);\n    let headers = res.headers();\n    assert_eq!(headers[\"x-client-ip\"], \"127.0.0.2\");\n}\n\n#[tokio::test]\nasync fn test_duplex() {\n    init();\n    // NOTE: this doesn't really verify that we are in full duplex mode as reqwest\n    // won't allow us control when req body is sent\n    let client = reqwest::Client::new();\n    let res = client\n        .post(\"http://127.0.0.1:6147/duplex/\")\n        .body(\"b\".repeat(1024 * 1024)) // 1 MB upload\n        .timeout(Duration::from_secs(5))\n        .send()\n        .await\n        .unwrap();\n    let headers = res.headers();\n    assert_eq!(headers[\"Connection\"], \"keep-alive\");\n    assert_eq!(res.status(), StatusCode::OK);\n    let body = res.text().await.unwrap();\n    assert_eq!(body.len(), 64 * 5);\n}\n\n#[tokio::test]\nasync fn test_connection_die() {\n    init();\n    let res = reqwest::get(\"http://127.0.0.1:6147/connection_die\")\n        .await\n        .unwrap();\n    assert_eq!(res.status(), StatusCode::OK);\n    let body = res.text().await;\n    // reqwest doesn't allow us to inspect the partial body\n    assert!(body.is_err());\n}\n\n#[tokio::test]\nasync fn test_upload_connection_die() {\n    init();\n    let client = reqwest::Client::new();\n    let res = client\n        .post(\"http://127.0.0.1:6147/upload_connection_die/\")\n        .body(\"b\".repeat(15 * 1024 * 1024)) // 15 MB upload\n        .timeout(Duration::from_secs(5))\n        .send()\n        .await\n        .unwrap();\n    // should get 200 status before connection dies\n    assert_eq!(res.status(), StatusCode::OK);\n    let _ = res.text().await;\n\n    // try h2\n    let client = reqwest::Client::new();\n    let res = client\n        .post(\"http://127.0.0.1:6147/upload_connection_die/\")\n        .body(\"b\".repeat(15 * 1024 * 1024)) // 15 MB upload\n        .timeout(Duration::from_secs(5))\n        .header(\"x-h2\", \"true\")\n        .send()\n        .await\n        .unwrap();\n    // should get 200 status before connection dies\n    assert_eq!(res.status(), StatusCode::OK);\n    let _ = res.text().await;\n}\n\n#[tokio::test]\nasync fn test_upload() {\n    init();\n    let client = reqwest::Client::new();\n    let res = client\n        .post(\"http://127.0.0.1:6147/upload/\")\n        .body(\"b\".repeat(15 * 1024 * 1024)) // 15 MB upload\n        .timeout(Duration::from_secs(5))\n        .send()\n        .await\n        .unwrap();\n    assert_eq!(res.status(), StatusCode::OK);\n    let body = res.text().await.unwrap();\n    assert_eq!(body.len(), 64 * 5);\n}\n\n#[tokio::test]\nasync fn test_close_on_response_before_downstream_finish() {\n    init();\n    let client = reqwest::Client::new();\n    let res = client\n        .post(\"http://127.0.0.1:6147/test2\")\n        .header(\"x-close-on-response-before-downstream-finish\", \"1\")\n        .body(\"b\".repeat(15 * 1024 * 1024)) // 15 MB upload\n        .timeout(Duration::from_secs(5))\n        .send()\n        .await\n        .unwrap();\n    assert_eq!(res.status(), StatusCode::OK);\n    let headers = res.headers();\n    assert_eq!(headers[\"Connection\"], \"close\");\n    let body = res.text().await.unwrap();\n    assert_eq!(body.len(), 11);\n}\n\n#[tokio::test]\nasync fn test_ws_server_ends_conn() {\n    init();\n    let _ = *WS_ECHO;\n\n    // server gracefully closes connection\n\n    let mut req = \"ws://127.0.0.1:6147\".into_client_request().unwrap();\n    req.headers_mut()\n        .insert(\"x-port\", HeaderValue::from_static(\"9283\"));\n\n    let (mut ws_stream, _) = tokio_tungstenite::connect_async(req).await.unwrap();\n    // gracefully close connection\n    ws_stream.send(\"test\".into()).await.unwrap();\n    ws_stream.next().await.unwrap().unwrap();\n    ws_stream.send(\"graceful\".into()).await.unwrap();\n    let msg = ws_stream.next().await.unwrap().unwrap();\n    // assert graceful close\n    assert!(matches!(msg, Message::Close(None)));\n    // test may hang here if downstream doesn't close when upstream does\n    assert!(ws_stream.next().await.is_none());\n\n    // server abruptly closes connection\n\n    let mut req = \"ws://127.0.0.1:6147\".into_client_request().unwrap();\n    req.headers_mut()\n        .insert(\"x-port\", HeaderValue::from_static(\"9283\"));\n\n    let (mut ws_stream, _) = tokio_tungstenite::connect_async(req).await.unwrap();\n    // abrupt close connection\n    ws_stream.send(\"close\".into()).await.unwrap();\n    // test will hang here if downstream doesn't close when upstream does\n    assert!(ws_stream.next().await.unwrap().is_err());\n\n    // client gracefully closes connection\n\n    let mut req = \"ws://127.0.0.1:6147\".into_client_request().unwrap();\n    req.headers_mut()\n        .insert(\"x-port\", HeaderValue::from_static(\"9283\"));\n\n    let (mut ws_stream, _) = tokio_tungstenite::connect_async(req).await.unwrap();\n    ws_stream.send(\"test\".into()).await.unwrap();\n    // sender initiates close\n    ws_stream.close(None).await.unwrap();\n    let msg = ws_stream.next().await.unwrap().unwrap();\n    // assert echo\n    assert_eq!(\"test\", msg.into_text().unwrap());\n    let msg = ws_stream.next().await.unwrap().unwrap();\n    // assert graceful close\n    assert!(matches!(msg, Message::Close(None)));\n    assert!(ws_stream.next().await.is_none());\n}\n\nfn parse_response_header(buf: &[u8]) -> ResponseHeader {\n    let mut headers = vec![httparse::EMPTY_HEADER; 256];\n    let mut parsed = httparse::Response::new(&mut headers);\n    match parsed.parse(buf).unwrap() {\n        httparse::Status::Complete(_) => {\n            let mut resp =\n                ResponseHeader::build(parsed.code.unwrap(), Some(parsed.headers.len())).unwrap();\n            for header in parsed.headers.iter() {\n                resp.append_header(header.name.to_string(), header.value)\n                    .unwrap();\n            }\n            resp\n        }\n        _ => panic!(\"expects a whole response header\"),\n    }\n}\n\n/// Read response header and return it along with any preread body data\nasync fn read_response_header(stream: &mut tokio::net::TcpStream) -> (ResponseHeader, Vec<u8>) {\n    let mut response = vec![];\n    let mut header_end = 0;\n    let mut buf = [0; 1024];\n    loop {\n        let n = stream.read(&mut buf).await.unwrap();\n        response.extend_from_slice(&buf[..n]);\n        let mut end_of_response = false;\n        for (i, w) in response.windows(4).enumerate() {\n            if w == b\"\\r\\n\\r\\n\" {\n                end_of_response = true;\n                header_end = i + 4;\n                break;\n            }\n        }\n        if end_of_response {\n            break;\n        }\n    }\n    let response_header = parse_response_header(&response[..header_end]);\n    let preread_body = response[header_end..].to_vec();\n    (response_header, preread_body)\n}\n\n/// Read remaining body bytes from stream until expected_body_len is reached\nasync fn read_response_body(\n    stream: &mut tokio::net::TcpStream,\n    mut body: Vec<u8>,\n    expected_body_len: usize,\n) -> Vec<u8> {\n    let mut buf = [0; 1024];\n    while body.len() < expected_body_len {\n        let n = stream.read(&mut buf).await.unwrap();\n        body.extend_from_slice(&buf[..n]);\n    }\n    if body.len() > expected_body_len {\n        panic!(\"more body bytes than expected\");\n    }\n    body\n}\n\nasync fn read_response(\n    stream: &mut tokio::net::TcpStream,\n    expected_body_len: usize,\n) -> (ResponseHeader, Vec<u8>) {\n    let (response_header, body) = read_response_header(stream).await;\n    let body = read_response_body(stream, body, expected_body_len).await;\n    (response_header, body)\n}\n\n#[tokio::test]\nasync fn test_upgrade_smoke() {\n    init();\n\n    let mut stream = TcpStream::connect(\"127.0.0.1:6147\").await.unwrap();\n\n    let req = concat!(\n        \"GET /upgrade HTTP/1.1\\r\\n\",\n        \"Host: 127.0.0.1\\r\\n\",\n        \"Upgrade: websocket\\r\\n\",\n        \"Connection: Upgrade\\r\\n\",\n        \"\\r\\n\"\n    );\n    stream.write_all(req.as_bytes()).await.unwrap();\n    stream.flush().await.unwrap();\n\n    let expected_payload = b\"hello\\n\";\n    let fut = read_response(&mut stream, expected_payload.len());\n    let (resp_header, resp_body) = timeout(Duration::from_secs(5), fut).await.unwrap();\n\n    assert_eq!(resp_header.status, 101);\n    assert_eq!(resp_header.headers[\"Upgrade\"], \"websocket\");\n    assert_eq!(resp_header.headers[\"Connection\"], \"upgrade\");\n    assert_eq!(resp_body, expected_payload);\n}\n\n#[tokio::test]\nasync fn test_upgrade_body() {\n    init();\n\n    let mut stream = TcpStream::connect(\"127.0.0.1:6147\").await.unwrap();\n\n    let req = concat!(\n        \"POST /upgrade_echo_body HTTP/1.1\\r\\n\",\n        \"Host: 127.0.0.1\\r\\n\",\n        \"Upgrade: websocket\\r\\n\",\n        \"Connection: Upgrade\\r\\n\",\n        \"Content-Length: 1024\\r\\n\",\n        \"\\r\\n\"\n    );\n    stream.write_all(req.as_bytes()).await.unwrap();\n    stream.flush().await.unwrap();\n    stream.write_all(\"b\".repeat(1024).as_bytes()).await.unwrap();\n    stream.flush().await.unwrap();\n\n    let fut = read_response(&mut stream, 1024);\n    let (resp_header, resp_body) = timeout(Duration::from_secs(5), fut).await.unwrap();\n    assert_eq!(resp_header.status, 101);\n    assert_eq!(resp_header.headers[\"Upgrade\"], \"websocket\");\n    assert_eq!(resp_header.headers[\"Connection\"], \"upgrade\");\n\n    let body = \"b\".repeat(1024);\n    assert_eq!(resp_body, body.as_bytes());\n}\n\n#[tokio::test]\nasync fn test_upgrade_body_after_101() {\n    // test content-length body is passed through after 101,\n    // and that ws payload is passed through afterwards\n    // use websocket server that flushes 101 after reading header\n    init();\n    let _ = *WS_ECHO_RAW;\n\n    let mut stream = TcpStream::connect(\"127.0.0.1:6147\").await.unwrap();\n\n    let req = concat!(\n        \"POST /upgrade_echo_body HTTP/1.1\\r\\n\",\n        \"Host: 127.0.0.1\\r\\n\",\n        \"Upgrade: websocket\\r\\n\",\n        \"Connection: Upgrade\\r\\n\",\n        \"X-Port: 9284\\r\\n\",\n        \"Content-Length: 5120\\r\\n\",\n        \"X-Expected-Body-Len: 5125\\r\\n\", // include ws payload\n        \"\\r\\n\"\n    );\n    stream.write_all(req.as_bytes()).await.unwrap();\n    stream.flush().await.unwrap();\n    stream\n        .write_all(\"b\".repeat(5 * 1024).as_bytes())\n        .await\n        .unwrap();\n    stream.flush().await.unwrap();\n\n    // Read response header and any preread body first (before sending ws_payload)\n    let fut = read_response_header(&mut stream);\n    let (resp_header, resp_body) = timeout(Duration::from_secs(5), fut).await.unwrap();\n    assert_eq!(resp_header.status, 101);\n    assert_eq!(resp_header.headers[\"Upgrade\"], \"websocket\");\n    assert_eq!(resp_header.headers[\"Connection\"], \"upgrade\");\n\n    // Now send the websocket payload after receiving 101\n    let ws_payload = \"hello\";\n    stream.write_all(ws_payload.as_bytes()).await.unwrap();\n    stream.flush().await.unwrap();\n\n    // Read the rest of the bytes (body + ws payload), subtracting preread body length\n    let expected_total_len = 5 * 1024 + ws_payload.len();\n    let fut = read_response_body(&mut stream, resp_body, expected_total_len);\n    let resp_body = timeout(Duration::from_secs(5), fut).await.unwrap();\n\n    let body = \"b\".repeat(5 * 1024) + ws_payload;\n    assert_eq!(resp_body, body.as_bytes());\n}\n\n#[tokio::test]\nasync fn test_download_timeout() {\n    init();\n    use hyper::body::HttpBody;\n    use tokio::time::sleep;\n\n    let client = hyper::Client::new();\n    let uri: hyper::Uri = \"http://127.0.0.1:6147/download_large/\".parse().unwrap();\n    let req = hyper::Request::builder()\n        .uri(uri)\n        .header(\"x-write-timeout\", \"1\")\n        .body(hyper::Body::empty())\n        .unwrap();\n    let mut res = client.request(req).await.unwrap();\n    assert_eq!(res.status(), StatusCode::OK);\n\n    let mut err = false;\n    sleep(Duration::from_secs(2)).await;\n    while let Some(chunk) = res.body_mut().data().await {\n        if chunk.is_err() {\n            err = true;\n        }\n    }\n    assert!(err);\n}\n\n#[tokio::test]\nasync fn test_download_timeout_min_rate() {\n    init();\n    use hyper::body::HttpBody;\n    use tokio::time::sleep;\n\n    let client = hyper::Client::new();\n    let uri: hyper::Uri = \"http://127.0.0.1:6147/download/\".parse().unwrap();\n    let req = hyper::Request::builder()\n        .uri(uri)\n        .header(\"x-write-timeout\", \"1\")\n        .header(\"x-min-rate\", \"10000\")\n        .body(hyper::Body::empty())\n        .unwrap();\n    let mut res = client.request(req).await.unwrap();\n    assert_eq!(res.status(), StatusCode::OK);\n\n    let mut err = false;\n    sleep(Duration::from_secs(2)).await;\n    while let Some(chunk) = res.body_mut().data().await {\n        if chunk.is_err() {\n            err = true;\n        }\n    }\n    // no error as write timeout is overridden by min rate\n    assert!(!err);\n}\n\nmod test_cache {\n    use super::*;\n    use std::str::FromStr;\n    use tokio::time::sleep;\n\n    #[tokio::test]\n    async fn test_basic_caching() {\n        init();\n        let url = \"http://127.0.0.1:6148/unique/test_basic_caching/now\";\n\n        let res = reqwest::get(url).await.unwrap();\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        let cache_miss_epoch = headers[\"x-epoch\"].to_str().unwrap().parse::<f64>().unwrap();\n        assert_eq!(headers[\"x-cache-status\"], \"miss\");\n        assert_eq!(res.text().await.unwrap(), \"hello world\");\n\n        let res = reqwest::get(url).await.unwrap();\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        let cache_hit_epoch = headers[\"x-epoch\"].to_str().unwrap().parse::<f64>().unwrap();\n        assert_eq!(headers[\"x-cache-status\"], \"hit\");\n        assert_eq!(res.text().await.unwrap(), \"hello world\");\n\n        assert_eq!(cache_miss_epoch, cache_hit_epoch);\n\n        sleep(Duration::from_millis(1100)).await; // ttl is 1\n\n        let res = reqwest::get(url).await.unwrap();\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        let cache_expired_epoch = headers[\"x-epoch\"].to_str().unwrap().parse::<f64>().unwrap();\n        assert_eq!(headers[\"x-cache-status\"], \"expired\");\n        assert_eq!(res.text().await.unwrap(), \"hello world\");\n\n        assert!(cache_expired_epoch > cache_hit_epoch);\n    }\n\n    #[tokio::test]\n    async fn test_purge() {\n        init();\n        let res = reqwest::get(\"http://127.0.0.1:6148/unique/test_purge/test2\")\n            .await\n            .unwrap();\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        assert_eq!(headers[\"x-cache-status\"], \"miss\");\n        assert_eq!(res.text().await.unwrap(), \"hello world\");\n\n        let res = reqwest::get(\"http://127.0.0.1:6148/unique/test_purge/test2\")\n            .await\n            .unwrap();\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        assert_eq!(headers[\"x-cache-status\"], \"hit\");\n        assert_eq!(res.text().await.unwrap(), \"hello world\");\n\n        let res = reqwest::Client::builder()\n            .build()\n            .unwrap()\n            .request(\n                reqwest::Method::from_bytes(b\"PURGE\").unwrap(),\n                \"http://127.0.0.1:6148/unique/test_purge/test2\",\n            )\n            .send()\n            .await\n            .unwrap();\n        assert_eq!(res.status(), StatusCode::OK);\n        assert_eq!(res.text().await.unwrap(), \"\");\n\n        let res = reqwest::Client::builder()\n            .build()\n            .unwrap()\n            .request(\n                reqwest::Method::from_bytes(b\"PURGE\").unwrap(),\n                \"http://127.0.0.1:6148/unique/test_purge/test2\",\n            )\n            .send()\n            .await\n            .unwrap();\n        assert_eq!(res.status(), StatusCode::NOT_FOUND);\n        assert_eq!(res.text().await.unwrap(), \"\");\n\n        let res = reqwest::get(\"http://127.0.0.1:6148/unique/test_purge/test2\")\n            .await\n            .unwrap();\n        let headers = res.headers();\n        assert_eq!(res.status(), StatusCode::OK);\n        assert_eq!(headers[\"x-cache-status\"], \"miss\");\n        assert_eq!(res.text().await.unwrap(), \"hello world\");\n    }\n\n    #[tokio::test]\n    async fn test_cache_miss_convert() {\n        init();\n\n        // test if-* header is stripped\n        let client = reqwest::Client::new();\n        let res = client\n            .get(\"http://127.0.0.1:6148/unique/test_cache_miss_convert/no_if_headers\")\n            .header(\"if-modified-since\", \"Wed, 19 Jan 2022 18:39:12 GMT\")\n            .send()\n            .await\n            .unwrap();\n        // 200 because last-modified not returned from upstream\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        assert_eq!(headers[\"x-cache-status\"], \"miss\");\n        assert_eq!(res.text().await.unwrap(), \"no if headers detected\\n\");\n\n        // test range header is stripped\n        let client = reqwest::Client::new();\n        let res = client\n            .get(\"http://127.0.0.1:6148/unique/test_cache_miss_convert2/no_if_headers\")\n            .header(\"Range\", \"bytes=0-1\")\n            .send()\n            .await\n            .unwrap();\n        // we have not implemented downstream range yet, it should be 206 once we have it\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        assert_eq!(headers[\"x-cache-status\"], \"miss\");\n        assert_eq!(res.text().await.unwrap(), \"no if headers detected\\n\");\n    }\n\n    #[tokio::test]\n    async fn test_cache_http10() {\n        // allow caching http1.0 from origin, but proxy as h1.1 downstream\n        init();\n        let url = \"http://127.0.0.1:6148/unique/test_cache_http10/now\";\n\n        let res = reqwest::Client::new()\n            .get(url)\n            .header(\"x-upstream-fake-http10\", \"1\") // fake http1.0 in upstream response filter\n            .send()\n            .await\n            .unwrap();\n        assert_eq!(res.status(), StatusCode::OK);\n        assert_eq!(res.version(), Version::HTTP_11);\n        let headers = res.headers();\n        let cache_miss_epoch = headers[\"x-epoch\"].to_str().unwrap().parse::<f64>().unwrap();\n        assert_eq!(headers[\"transfer-encoding\"], \"chunked\");\n        assert_eq!(headers[\"x-cache-status\"], \"miss\");\n        assert_eq!(res.text().await.unwrap(), \"hello world\");\n\n        let res = reqwest::Client::new()\n            .get(url)\n            .header(\"x-upstream-fake-http10\", \"1\")\n            .send()\n            .await\n            .unwrap();\n        assert_eq!(res.status(), StatusCode::OK);\n        assert_eq!(res.version(), Version::HTTP_11);\n        let headers = res.headers();\n        let cache_hit_epoch = headers[\"x-epoch\"].to_str().unwrap().parse::<f64>().unwrap();\n        assert_eq!(headers[\"transfer-encoding\"], \"chunked\");\n        assert_eq!(headers[\"x-cache-status\"], \"hit\");\n        assert_eq!(res.text().await.unwrap(), \"hello world\");\n\n        assert_eq!(cache_miss_epoch, cache_hit_epoch);\n\n        sleep(Duration::from_millis(1100)).await; // ttl is 1\n\n        let res = reqwest::Client::new()\n            .get(url)\n            .header(\"x-upstream-fake-http10\", \"1\")\n            .send()\n            .await\n            .unwrap();\n        assert_eq!(res.status(), StatusCode::OK);\n        assert_eq!(res.version(), Version::HTTP_11);\n        let headers = res.headers();\n        let cache_expired_epoch = headers[\"x-epoch\"].to_str().unwrap().parse::<f64>().unwrap();\n        assert_eq!(headers[\"transfer-encoding\"], \"chunked\");\n        assert_eq!(headers[\"x-cache-status\"], \"expired\");\n        assert_eq!(res.text().await.unwrap(), \"hello world\");\n\n        assert!(cache_expired_epoch > cache_hit_epoch);\n    }\n\n    #[tokio::test]\n    async fn test_cache_downstream_compression() {\n        init();\n\n        // disable reqwest gzip support to check compression headers and body\n        // otherwise reqwest will decompress and strip the headers\n        let client = reqwest::ClientBuilder::new().gzip(false).build().unwrap();\n        let res = client\n            .get(\"http://127.0.0.1:6148/unique/test_cache_downstream_compression/no_compression\")\n            .header(\"x-downstream-compression\", \"1\")\n            .header(\"accept-encoding\", \"gzip\")\n            .send()\n            .await\n            .unwrap();\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        assert_eq!(headers[\"Content-Encoding\"], \"gzip\");\n        assert_eq!(headers[\"x-cache-status\"], \"miss\");\n        let body = res.bytes().await.unwrap();\n        assert!(body.len() < 32);\n\n        // should also apply on hit\n        let client = reqwest::ClientBuilder::new().gzip(false).build().unwrap();\n        let res = client\n            .get(\"http://127.0.0.1:6148/unique/test_cache_downstream_compression/no_compression\")\n            .header(\"x-downstream-compression\", \"1\")\n            .header(\"accept-encoding\", \"gzip\")\n            .send()\n            .await\n            .unwrap();\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        assert_eq!(headers[\"Content-Encoding\"], \"gzip\");\n        assert_eq!(headers[\"x-cache-status\"], \"hit\");\n        let body = res.bytes().await.unwrap();\n        assert!(body.len() < 32);\n    }\n\n    #[tokio::test]\n    async fn test_cache_downstream_decompression() {\n        init();\n\n        // disable reqwest gzip support to check compression headers and body\n        // otherwise reqwest will decompress and strip the headers\n        let client = reqwest::ClientBuilder::new().gzip(false).build().unwrap();\n        let res = client\n            .get(\"http://127.0.0.1:6148/unique/test_cache_downstream_decompression/gzip/index.html\")\n            .header(\"x-downstream-decompression\", \"1\")\n            .header(\"x-upstream-accept-encoding\", \"gzip\")\n            .send()\n            .await\n            .unwrap();\n\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        // upstream should have received gzip, should decompress for downstream\n        assert_eq!(headers[\"received-accept-encoding\"], \"gzip\");\n        assert!(headers.get(\"Content-Encoding\").is_none());\n        assert_eq!(headers[\"x-cache-status\"], \"miss\");\n        let body = res.bytes().await.unwrap();\n        assert_eq!(body, \"Hello World!\\n\");\n\n        // should also apply on hit\n        let client = reqwest::ClientBuilder::new().gzip(false).build().unwrap();\n        let res = client\n            .get(\"http://127.0.0.1:6148/unique/test_cache_downstream_decompression/gzip/index.html\")\n            .header(\"x-downstream-decompression\", \"1\")\n            .header(\"x-upstream-accept-encoding\", \"gzip\")\n            .send()\n            .await\n            .unwrap();\n\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        assert!(headers.get(\"Content-Encoding\").is_none());\n        assert_eq!(headers[\"x-cache-status\"], \"hit\");\n        let body = res.bytes().await.unwrap();\n        assert_eq!(body, \"Hello World!\\n\");\n\n        sleep(Duration::from_millis(1100)).await; // ttl is 1\n\n        // should also apply on revalidated\n        let client = reqwest::ClientBuilder::new().gzip(false).build().unwrap();\n        let res = client\n            .get(\"http://127.0.0.1:6148/unique/test_cache_downstream_decompression/gzip/index.html\")\n            .header(\"x-downstream-decompression\", \"1\")\n            .header(\"x-upstream-accept-encoding\", \"gzip\")\n            .send()\n            .await\n            .unwrap();\n\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        assert!(headers.get(\"Content-Encoding\").is_none());\n        assert_eq!(headers[\"x-cache-status\"], \"revalidated\");\n        let body = res.bytes().await.unwrap();\n        assert_eq!(body, \"Hello World!\\n\");\n    }\n\n    #[tokio::test]\n    async fn test_network_error_mid_response() {\n        init();\n        let url = \"http://127.0.0.1:6148/sleep/test_network_error_mid_response.txt\";\n\n        let res = reqwest::Client::new()\n            .get(url)\n            .header(\"x-set-sleep\", \"0\") // no need to sleep\n            .header(\"x-set-body-sleep\", \"0.1\") // pause the body a bit before abort\n            .header(\"x-abort-body\", \"true\") // this will tell origin to kill the conn right away\n            .send()\n            .await\n            .unwrap();\n        assert_eq!(res.status(), 200);\n        // sleep just a little to make sure the req above gets the cache lock\n        sleep(Duration::from_millis(50)).await;\n        let headers = res.headers();\n        assert_eq!(headers[\"x-cache-status\"], \"miss\");\n        // the connection dies\n        assert!(res.text().await.is_err());\n\n        let res = reqwest::Client::new()\n            .get(url)\n            .header(\"x-set-sleep\", \"0\") // no need to sleep\n            .header(\"x-set-body-sleep\", \"0.1\") // pause the body a bit before abort\n            .header(\"x-abort-body\", \"true\") // this will tell origin to kill the conn right away\n            .send()\n            .await\n            .unwrap();\n        assert_eq!(res.status(), 200);\n        // sleep just a little to make sure the req above gets the cache lock\n        sleep(Duration::from_millis(50)).await;\n        let headers = res.headers();\n        assert_eq!(headers[\"x-cache-status\"], \"miss\");\n        // the connection dies\n        assert!(res.text().await.is_err());\n    }\n\n    #[tokio::test]\n    async fn test_cache_upstream_revalidation() {\n        init();\n        let url = \"http://127.0.0.1:6148/unique/test_upstream_revalidation/revalidate_now\";\n\n        let res = reqwest::get(url).await.unwrap();\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        let cache_miss_epoch = headers[\"x-epoch\"].to_str().unwrap().parse::<f64>().unwrap();\n        assert_eq!(headers[\"x-cache-status\"], \"miss\");\n        assert_eq!(headers[\"x-upstream-status\"], \"200\");\n        assert_eq!(res.text().await.unwrap(), \"hello world\");\n\n        let res = reqwest::get(url).await.unwrap();\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        let cache_hit_epoch = headers[\"x-epoch\"].to_str().unwrap().parse::<f64>().unwrap();\n        assert_eq!(headers[\"x-cache-status\"], \"hit\");\n        assert!(headers.get(\"x-upstream-status\").is_none());\n        assert_eq!(res.text().await.unwrap(), \"hello world\");\n\n        assert_eq!(cache_miss_epoch, cache_hit_epoch);\n\n        sleep(Duration::from_millis(1100)).await; // ttl is 1\n\n        let res = reqwest::get(url).await.unwrap();\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        let cache_expired_epoch = headers[\"x-epoch\"].to_str().unwrap().parse::<f64>().unwrap();\n        assert_eq!(headers[\"x-cache-status\"], \"revalidated\");\n        assert_eq!(headers[\"x-upstream-status\"], \"304\");\n        assert_eq!(res.text().await.unwrap(), \"hello world\");\n\n        // still the old object\n        assert_eq!(cache_expired_epoch, cache_hit_epoch);\n    }\n\n    #[tokio::test]\n    async fn test_cache_upstream_revalidation_appends_headers() {\n        init();\n        let url = \"http://127.0.0.1:6148/unique/test_cache_upstream_revalidation_appends_headers/cache_control\";\n\n        let res = reqwest::Client::new()\n            .get(url)\n            .header(\"set-cache-control\", \"public, max-age=1\")\n            .send()\n            .await\n            .unwrap();\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        assert_eq!(headers[\"x-cache-status\"], \"miss\");\n        assert_eq!(headers[\"x-upstream-status\"], \"200\");\n        assert_eq!(headers[\"cache-control\"], \"public, max-age=1\");\n        assert_eq!(headers.get_all(\"cache-control\").into_iter().count(), 1);\n        assert_eq!(res.text().await.unwrap(), \"hello world\");\n\n        let res = reqwest::Client::new()\n            .get(url)\n            .header(\"set-cache-control\", \"public, max-age=1\")\n            .send()\n            .await\n            .unwrap();\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        assert_eq!(headers[\"x-cache-status\"], \"hit\");\n        assert!(headers.get(\"x-upstream-status\").is_none());\n        assert_eq!(headers.get_all(\"cache-control\").into_iter().count(), 1);\n        assert_eq!(res.text().await.unwrap(), \"hello world\");\n\n        sleep(Duration::from_millis(1100)).await; // ttl is 1\n\n        let res = reqwest::Client::new()\n            .get(url)\n            .header(\"set-cache-control\", \"public, max-age=1\")\n            .header(\"set-cache-control\", \"stale-while-revalidate=86400\")\n            .header(\"set-revalidated\", \"1\")\n            .send()\n            .await\n            .unwrap();\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        assert_eq!(headers[\"x-cache-status\"], \"revalidated\");\n        assert_eq!(headers[\"x-upstream-status\"], \"304\");\n        let mut cc = headers.get_all(\"cache-control\").into_iter();\n        assert_eq!(cc.next().unwrap(), \"public, max-age=1\");\n        assert_eq!(cc.next().unwrap(), \"stale-while-revalidate=86400\");\n        assert!(cc.next().is_none());\n        assert_eq!(res.text().await.unwrap(), \"hello world\");\n    }\n\n    #[tokio::test]\n    async fn test_force_miss() {\n        init();\n        let url = \"http://127.0.0.1:6148/unique/test_froce_miss/revalidate_now\";\n\n        let res = reqwest::get(url).await.unwrap();\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        let cache_miss_epoch = headers[\"x-epoch\"].to_str().unwrap().parse::<f64>().unwrap();\n        assert_eq!(headers[\"x-cache-status\"], \"miss\");\n        assert_eq!(headers[\"x-upstream-status\"], \"200\");\n        assert_eq!(res.text().await.unwrap(), \"hello world\");\n\n        let res = reqwest::get(url).await.unwrap();\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        let cache_hit_epoch = headers[\"x-epoch\"].to_str().unwrap().parse::<f64>().unwrap();\n        assert_eq!(headers[\"x-cache-status\"], \"hit\");\n        assert!(headers.get(\"x-upstream-status\").is_none());\n        assert_eq!(res.text().await.unwrap(), \"hello world\");\n\n        assert_eq!(cache_miss_epoch, cache_hit_epoch);\n\n        let res = reqwest::Client::new()\n            .get(url)\n            .header(\"x-force-miss\", \"1\")\n            .send()\n            .await\n            .unwrap();\n\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        assert_eq!(headers[\"x-cache-status\"], \"miss\");\n        assert_eq!(headers[\"x-upstream-status\"], \"200\");\n        assert_eq!(res.text().await.unwrap(), \"hello world\");\n    }\n\n    #[tokio::test]\n    async fn test_force_miss_stale() {\n        init();\n        let url = \"http://127.0.0.1:6148/unique/test_froce_miss_stale/revalidate_now\";\n\n        let res = reqwest::get(url).await.unwrap();\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        let cache_miss_epoch = headers[\"x-epoch\"].to_str().unwrap().parse::<f64>().unwrap();\n        assert_eq!(headers[\"x-cache-status\"], \"miss\");\n        assert_eq!(headers[\"x-upstream-status\"], \"200\");\n        assert_eq!(res.text().await.unwrap(), \"hello world\");\n\n        let res = reqwest::get(url).await.unwrap();\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        let cache_hit_epoch = headers[\"x-epoch\"].to_str().unwrap().parse::<f64>().unwrap();\n        assert_eq!(headers[\"x-cache-status\"], \"hit\");\n        assert!(headers.get(\"x-upstream-status\").is_none());\n        assert_eq!(res.text().await.unwrap(), \"hello world\");\n\n        assert_eq!(cache_miss_epoch, cache_hit_epoch);\n\n        sleep(Duration::from_millis(1100)).await; // ttl is 1\n\n        // stale, but can be forced miss\n        let res = reqwest::Client::new()\n            .get(url)\n            .header(\"x-force-miss\", \"1\")\n            .send()\n            .await\n            .unwrap();\n\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        assert_eq!(headers[\"x-cache-status\"], \"miss\");\n        assert_eq!(headers[\"x-upstream-status\"], \"200\");\n        let cache_miss_epoch2 = headers[\"x-epoch\"].to_str().unwrap().parse::<f64>().unwrap();\n        assert!(cache_miss_epoch != cache_miss_epoch2);\n        assert_eq!(res.text().await.unwrap(), \"hello world\");\n    }\n\n    #[tokio::test]\n    async fn test_force_fresh() {\n        init();\n        let url = \"http://127.0.0.1:6148/unique/test_force_fresh/revalidate_now\";\n\n        let res = reqwest::get(url).await.unwrap();\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        let cache_miss_epoch = headers[\"x-epoch\"].to_str().unwrap().parse::<f64>().unwrap();\n        assert_eq!(headers[\"x-cache-status\"], \"miss\");\n        assert_eq!(headers[\"x-upstream-status\"], \"200\");\n        assert_eq!(res.text().await.unwrap(), \"hello world\");\n\n        let res = reqwest::get(url).await.unwrap();\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        let cache_hit_epoch = headers[\"x-epoch\"].to_str().unwrap().parse::<f64>().unwrap();\n        assert_eq!(headers[\"x-cache-status\"], \"hit\");\n        assert!(headers.get(\"x-upstream-status\").is_none());\n        assert_eq!(res.text().await.unwrap(), \"hello world\");\n\n        assert_eq!(cache_miss_epoch, cache_hit_epoch);\n\n        sleep(Duration::from_millis(1100)).await; // ttl is 1\n\n        // stale, but can be forced fresh\n        let res = reqwest::Client::new()\n            .get(url)\n            .header(\"x-force-fresh\", \"1\")\n            .send()\n            .await\n            .unwrap();\n\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        assert_eq!(headers[\"x-cache-status\"], \"hit\");\n        assert!(!headers.contains_key(\"x-upstream-status\"));\n        let cache_miss_epoch2 = headers[\"x-epoch\"].to_str().unwrap().parse::<f64>().unwrap();\n        assert_eq!(cache_miss_epoch, cache_miss_epoch2);\n        assert_eq!(res.text().await.unwrap(), \"hello world\");\n    }\n\n    #[tokio::test]\n    async fn test_cache_downstream_revalidation_etag() {\n        init();\n        let url = \"http://127.0.0.1:6148/unique/test_downstream_revalidation_etag/revalidate_now\";\n        let client = reqwest::Client::new();\n\n        // MISS + 304\n        let res = client\n            .get(url)\n            .header(\"If-None-Match\", \"\\\"abcd\\\", \\\"foobar\\\"\") // \"abcd\" is the fixed etag of this\n            // endpoint\n            .send()\n            .await\n            .unwrap();\n        assert_eq!(res.status(), StatusCode::NOT_MODIFIED);\n        let headers = res.headers();\n        let cache_miss_epoch = headers[\"x-epoch\"].to_str().unwrap().parse::<f64>().unwrap();\n        assert_eq!(headers[\"x-cache-status\"], \"miss\");\n        assert_eq!(res.text().await.unwrap(), \"\"); // 304 no body\n\n        // HIT + 304\n        let res = client\n            .get(url)\n            .header(\"If-None-Match\", \"\\\"abcd\\\", \\\"foobar\\\"\")\n            .send()\n            .await\n            .unwrap();\n        assert_eq!(res.status(), StatusCode::NOT_MODIFIED);\n        let headers = res.headers();\n        let cache_hit_epoch = headers[\"x-epoch\"].to_str().unwrap().parse::<f64>().unwrap();\n        assert_eq!(headers[\"x-cache-status\"], \"hit\");\n        assert_eq!(res.text().await.unwrap(), \"\"); // 304 no body\n\n        assert_eq!(cache_miss_epoch, cache_hit_epoch);\n\n        // HIT + 200 (condition passed)\n        let res = client\n            .get(url)\n            .header(\"If-None-Match\", \"\\\"foobar\\\"\")\n            .send()\n            .await\n            .unwrap();\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        let cache_hit_epoch = headers[\"x-epoch\"].to_str().unwrap().parse::<f64>().unwrap();\n        assert_eq!(headers[\"x-cache-status\"], \"hit\");\n        assert_eq!(res.text().await.unwrap(), \"hello world\");\n\n        assert_eq!(cache_miss_epoch, cache_hit_epoch);\n\n        sleep(Duration::from_millis(1100)).await; // ttl is 1\n\n        // revalidated + 304\n        let res = client\n            .get(url)\n            .header(\"If-None-Match\", \"\\\"abcd\\\", \\\"foobar\\\"\")\n            .send()\n            .await\n            .unwrap();\n        assert_eq!(res.status(), StatusCode::NOT_MODIFIED);\n        let headers = res.headers();\n        let cache_expired_epoch = headers[\"x-epoch\"].to_str().unwrap().parse::<f64>().unwrap();\n        assert_eq!(headers[\"x-cache-status\"], \"revalidated\");\n        assert_eq!(res.text().await.unwrap(), \"\"); // 304 no body\n\n        // still the old object\n        assert_eq!(cache_expired_epoch, cache_hit_epoch);\n    }\n\n    #[tokio::test]\n    async fn test_cache_downstream_revalidation_last_modified() {\n        init();\n        let url = \"http://127.0.0.1:6148/unique/test_downstream_revalidation_last_modified/revalidate_now\";\n        let client = reqwest::Client::new();\n\n        // MISS + 304\n        let res = client\n            .get(url)\n            .header(\"If-Modified-Since\", \"Tue, 03 May 2022 01:04:39 GMT\") // fixed last-modified of\n            // the endpoint\n            .send()\n            .await\n            .unwrap();\n        assert_eq!(res.status(), StatusCode::NOT_MODIFIED);\n        let headers = res.headers();\n        let cache_miss_epoch = headers[\"x-epoch\"].to_str().unwrap().parse::<f64>().unwrap();\n        assert_eq!(headers[\"x-cache-status\"], \"miss\");\n        assert_eq!(res.text().await.unwrap(), \"\"); // 304 no body\n\n        // HIT + 304\n        let res = client\n            .get(url)\n            .header(\"If-Modified-Since\", \"Tue, 03 May 2022 01:11:39 GMT\")\n            .send()\n            .await\n            .unwrap();\n        assert_eq!(res.status(), StatusCode::NOT_MODIFIED);\n        let headers = res.headers();\n        let cache_hit_epoch = headers[\"x-epoch\"].to_str().unwrap().parse::<f64>().unwrap();\n        assert_eq!(headers[\"x-cache-status\"], \"hit\");\n        assert_eq!(res.text().await.unwrap(), \"\"); // 304 no body\n\n        assert_eq!(cache_miss_epoch, cache_hit_epoch);\n\n        // HIT + 200 (condition passed)\n        let res = client\n            .get(url)\n            .header(\"If-Modified-Since\", \"Tue, 03 May 2022 00:11:39 GMT\")\n            .send()\n            .await\n            .unwrap();\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        let cache_hit_epoch = headers[\"x-epoch\"].to_str().unwrap().parse::<f64>().unwrap();\n        assert_eq!(headers[\"x-cache-status\"], \"hit\");\n        assert_eq!(res.text().await.unwrap(), \"hello world\");\n\n        assert_eq!(cache_miss_epoch, cache_hit_epoch);\n\n        sleep(Duration::from_millis(1100)).await; // ttl is 1\n\n        // revalidated + 304\n        let res = client\n            .get(url)\n            .header(\"If-Modified-Since\", \"Tue, 03 May 2022 01:11:39 GMT\")\n            .send()\n            .await\n            .unwrap();\n        assert_eq!(res.status(), StatusCode::NOT_MODIFIED);\n        let headers = res.headers();\n        let cache_expired_epoch = headers[\"x-epoch\"].to_str().unwrap().parse::<f64>().unwrap();\n        assert_eq!(headers[\"x-cache-status\"], \"revalidated\");\n        assert_eq!(res.text().await.unwrap(), \"\"); // 304 no body\n\n        // still the old object\n        assert_eq!(cache_expired_epoch, cache_hit_epoch);\n    }\n\n    #[tokio::test]\n    async fn test_cache_downstream_head() {\n        init();\n        let url = \"http://127.0.0.1:6148/unique/test_downstream_head/revalidate_now\";\n        let client = reqwest::Client::new();\n\n        // MISS + HEAD\n        let res = client.head(url).send().await.unwrap();\n\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        let cache_miss_epoch = headers[\"x-epoch\"].to_str().unwrap().parse::<f64>().unwrap();\n        assert_eq!(headers[\"x-cache-status\"], \"miss\");\n        assert_eq!(res.text().await.unwrap(), \"\"); // HEAD no body\n\n        // HIT + HEAD\n        let res = client.head(url).send().await.unwrap();\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        let cache_hit_epoch = headers[\"x-epoch\"].to_str().unwrap().parse::<f64>().unwrap();\n        assert_eq!(headers[\"x-cache-status\"], \"hit\");\n        assert_eq!(res.text().await.unwrap(), \"\"); // HEAD no body\n\n        assert_eq!(cache_miss_epoch, cache_hit_epoch);\n\n        sleep(Duration::from_millis(1100)).await; // ttl is 1\n\n        // revalidated + HEAD\n        let res = client.head(url).send().await.unwrap();\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        let cache_expired_epoch = headers[\"x-epoch\"].to_str().unwrap().parse::<f64>().unwrap();\n        assert_eq!(headers[\"x-cache-status\"], \"revalidated\");\n        assert_eq!(res.text().await.unwrap(), \"\"); // HEAD no body\n\n        // still the old object\n        assert_eq!(cache_expired_epoch, cache_hit_epoch);\n    }\n\n    #[tokio::test]\n    async fn test_purge_reject() {\n        init();\n\n        let res = reqwest::Client::builder()\n            .build()\n            .unwrap()\n            .request(\n                reqwest::Method::from_bytes(b\"PURGE\").unwrap(),\n                \"http://127.0.0.1:6148/\",\n            )\n            .header(\"x-bypass-cache\", \"1\") // not to cache this one\n            .send()\n            .await\n            .unwrap();\n        assert_eq!(res.status(), StatusCode::METHOD_NOT_ALLOWED);\n        assert_eq!(res.text().await.unwrap(), \"\");\n    }\n\n    #[tokio::test]\n    async fn test_cache_websocket_101() {\n        // Test the unlikely scenario in which users may want to cache WS\n        init();\n\n        // First request - should be a miss\n        let mut stream = TcpStream::connect(\"127.0.0.1:6148\").await.unwrap();\n        let req = concat!(\n            \"GET /unique/test_cache_websocket_101/upgrade HTTP/1.1\\r\\n\",\n            \"Host: 127.0.0.1\\r\\n\",\n            \"Upgrade: websocket\\r\\n\",\n            \"Connection: Upgrade\\r\\n\",\n            \"X-Cache-Websocket: 1\\r\\n\",\n            \"\\r\\n\"\n        );\n        stream.write_all(req.as_bytes()).await.unwrap();\n        stream.flush().await.unwrap();\n\n        let expected_payload = b\"hello\\n\";\n        let fut = read_response(&mut stream, expected_payload.len());\n        let (resp_header, resp_body) = timeout(Duration::from_secs(5), fut).await.unwrap();\n\n        assert_eq!(resp_header.status, 101);\n        assert_eq!(resp_header.headers[\"Upgrade\"], \"websocket\");\n        assert_eq!(resp_header.headers[\"x-cache-status\"], \"miss\");\n        assert_eq!(resp_body, expected_payload);\n\n        // Second request - should be a cache hit\n        let mut stream = TcpStream::connect(\"127.0.0.1:6148\").await.unwrap();\n        let req = concat!(\n            \"GET /unique/test_cache_websocket_101/upgrade HTTP/1.1\\r\\n\",\n            \"Host: 127.0.0.1\\r\\n\",\n            \"Upgrade: websocket\\r\\n\",\n            \"Connection: Upgrade\\r\\n\",\n            \"X-Cache-Websocket: 1\\r\\n\",\n            \"\\r\\n\"\n        );\n        stream.write_all(req.as_bytes()).await.unwrap();\n        stream.flush().await.unwrap();\n\n        let fut = read_response(&mut stream, expected_payload.len());\n        let (resp_header, resp_body) = timeout(Duration::from_secs(5), fut).await.unwrap();\n\n        assert_eq!(resp_header.status, 101);\n        assert_eq!(resp_header.headers[\"Upgrade\"], \"websocket\");\n        assert_eq!(resp_header.headers[\"x-cache-status\"], \"hit\");\n        assert_eq!(resp_body, expected_payload);\n    }\n\n    #[tokio::test]\n    async fn test_1xx_caching() {\n        // 1xx shouldn't interfere with HTTP caching\n\n        // set up a one-off mock server\n        // (warp / hyper don't have custom 1xx sending capabilities yet)\n        async fn mock_1xx_server(port: u16, cc_header: &str) {\n            use tokio::io::AsyncWriteExt;\n\n            let listener = tokio::net::TcpListener::bind(format!(\"127.0.0.1:{}\", port))\n                .await\n                .unwrap();\n            if let Ok((mut stream, _addr)) = listener.accept().await {\n                stream.write_all(b\"HTTP/1.1 103 Early Hints\\r\\nLink: <https://foo.bar>; rel=preconnect\\r\\n\\r\\n\").await.unwrap();\n                // wait a bit so that the client can read\n                sleep(Duration::from_millis(100)).await;\n                stream.write_all(format!(\"HTTP/1.1 200 OK\\r\\nContent-Length: 5\\r\\nCache-Control: {}\\r\\n\\r\\nhello\", cc_header).as_bytes()).await.unwrap();\n                sleep(Duration::from_millis(100)).await;\n            }\n        }\n\n        init();\n\n        let url = \"http://127.0.0.1:6148/unique/test_1xx_caching\";\n\n        tokio::spawn(async {\n            mock_1xx_server(6151, \"max-age=5\").await;\n        });\n        // wait for server to start\n        sleep(Duration::from_millis(100)).await;\n\n        let client = reqwest::Client::new();\n        let res = client\n            .get(url)\n            .header(\"x-port\", \"6151\")\n            .send()\n            .await\n            .unwrap();\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        assert_eq!(headers[\"x-cache-status\"], \"miss\");\n        assert_eq!(res.text().await.unwrap(), \"hello\");\n\n        let res = client\n            .get(url)\n            .header(\"x-port\", \"6151\")\n            .send()\n            .await\n            .unwrap();\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        assert_eq!(headers[\"x-cache-status\"], \"hit\");\n        assert_eq!(res.text().await.unwrap(), \"hello\");\n\n        // 1xx shouldn't interfere with bypass\n        let url = \"http://127.0.0.1:6148/unique/test_1xx_bypass\";\n\n        tokio::spawn(async {\n            mock_1xx_server(6152, \"private, no-store\").await;\n        });\n        // wait for server to start\n        sleep(Duration::from_millis(100)).await;\n\n        let res = client\n            .get(url)\n            .header(\"x-port\", \"6152\")\n            .send()\n            .await\n            .unwrap();\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        assert_eq!(headers[\"x-cache-status\"], \"no-cache\");\n        assert_eq!(res.text().await.unwrap(), \"hello\");\n\n        // restart the one-off server - still uncacheable\n        sleep(Duration::from_millis(100)).await;\n        tokio::spawn(async {\n            mock_1xx_server(6152, \"private, no-store\").await;\n        });\n        // wait for server to start\n        sleep(Duration::from_millis(100)).await;\n\n        let res = client\n            .get(url)\n            .header(\"x-port\", \"6152\")\n            .send()\n            .await\n            .unwrap();\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        assert_eq!(headers[\"x-cache-status\"], \"no-cache\");\n        assert_eq!(res.text().await.unwrap(), \"hello\");\n    }\n\n    #[tokio::test]\n    async fn test_bypassed_became_cacheable() {\n        init();\n\n        let url = \"http://127.0.0.1:6148/unique/test_bypassed/cache_control\";\n\n        let res = reqwest::Client::new()\n            .get(url)\n            .header(\"set-cache-control\", \"private, max-age=0\")\n            .send()\n            .await\n            .unwrap();\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        let cc = headers.get(\"Cache-Control\").unwrap();\n        assert_eq!(cc, \"private, max-age=0\");\n        assert_eq!(headers[\"x-cache-status\"], \"no-cache\");\n        assert_eq!(res.text().await.unwrap(), \"hello world\");\n\n        // request should bypass cache, but became cacheable (cache fill)\n        let res = reqwest::Client::new()\n            .get(url)\n            .header(\"set-cache-control\", \"public, max-age=10\")\n            .send()\n            .await\n            .unwrap();\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        assert_eq!(headers[\"x-cache-status\"], \"miss\");\n        assert_eq!(res.text().await.unwrap(), \"hello world\");\n\n        // HIT\n        let res = reqwest::Client::new()\n            .get(url)\n            .header(\"set-cache-control\", \"public, max-age=10\")\n            .send()\n            .await\n            .unwrap();\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        assert_eq!(headers[\"x-cache-status\"], \"hit\");\n        assert_eq!(res.text().await.unwrap(), \"hello world\");\n    }\n\n    #[tokio::test]\n    async fn test_bypassed_304() {\n        init();\n\n        let url = \"http://127.0.0.1:6148/unique/test_bypassed_304/cache_control\";\n\n        let res = reqwest::Client::new()\n            .get(url)\n            .header(\"set-cache-control\", \"private, max-age=0\")\n            .send()\n            .await\n            .unwrap();\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        let cc = headers.get(\"Cache-Control\").unwrap();\n        assert_eq!(cc, \"private, max-age=0\");\n        assert_eq!(headers[\"x-cache-status\"], \"no-cache\");\n        assert_eq!(res.text().await.unwrap(), \"hello world\");\n\n        // cacheable without private cache-control\n        // note this will be a 304 and not a 200, we will cache on _next_ request\n        let res = reqwest::Client::new()\n            .get(url)\n            .header(\"set-cache-control\", \"public, max-age=10\")\n            .header(\"set-revalidated\", \"1\")\n            .send()\n            .await\n            .unwrap();\n        assert_eq!(res.status(), StatusCode::NOT_MODIFIED);\n        let headers = res.headers();\n        assert_eq!(headers[\"x-cache-status\"], \"deferred\");\n\n        // should be cache fill\n        let res = reqwest::Client::new()\n            .get(url)\n            .header(\"set-cache-control\", \"public, max-age=10\")\n            .send()\n            .await\n            .unwrap();\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        assert_eq!(headers[\"x-cache-status\"], \"miss\");\n        assert_eq!(res.text().await.unwrap(), \"hello world\");\n\n        // HIT\n        let res = reqwest::Client::new()\n            .get(url)\n            .header(\"set-cache-control\", \"public, max-age=10\")\n            .send()\n            .await\n            .unwrap();\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        assert_eq!(headers[\"x-cache-status\"], \"hit\");\n        assert_eq!(res.text().await.unwrap(), \"hello world\");\n    }\n\n    #[tokio::test]\n    async fn test_bypassed_uncacheable_304() {\n        init();\n\n        let url = \"http://127.0.0.1:6148/unique/test_bypassed_private_304/cache_control\";\n\n        // cache fill\n        let res = reqwest::Client::new()\n            .get(url)\n            .header(\"set-cache-control\", \"public, max-age=0\")\n            .send()\n            .await\n            .unwrap();\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        let cc = headers.get(\"Cache-Control\").unwrap();\n        assert_eq!(cc, \"public, max-age=0\");\n        assert_eq!(headers[\"x-cache-status\"], \"miss\");\n        assert_eq!(res.text().await.unwrap(), \"hello world\");\n\n        // cache stale\n        // upstream returns 304, but response became uncacheable\n        let res = reqwest::Client::new()\n            .get(url)\n            .header(\"set-cache-control\", \"private\")\n            .header(\"set-revalidated\", \"1\")\n            .send()\n            .await\n            .unwrap();\n        // should see the response body because we didn't send conditional headers\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        assert_eq!(headers[\"x-cache-status\"], \"revalidated\");\n        assert_eq!(res.text().await.unwrap(), \"hello world\");\n\n        // we bypass cache for this next request\n        let res = reqwest::Client::new()\n            .get(url)\n            .header(\"set-cache-control\", \"public, max-age=10\")\n            .header(\"set-revalidated\", \"1\") // non-200 status to get bypass phase\n            .send()\n            .await\n            .unwrap();\n        assert_eq!(res.status(), StatusCode::NOT_MODIFIED);\n        let headers = res.headers();\n        assert_eq!(headers[\"x-cache-status\"], \"deferred\");\n    }\n\n    #[tokio::test]\n    async fn test_bypassed_head() {\n        init();\n\n        let url = \"http://127.0.0.1:6148/unique/test_bypassed_head/cache_control\";\n\n        // uncacheable, should bypass\n        let res = reqwest::Client::new()\n            .get(url)\n            .header(\"set-cache-control\", \"private, max-age=0\")\n            .send()\n            .await\n            .unwrap();\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        assert_eq!(headers[\"x-cache-status\"], \"no-cache\");\n        assert_eq!(res.text().await.unwrap(), \"hello world\");\n\n        // we bypass cache for this next request, becomes cacheable\n        let res = reqwest::Client::new()\n            .head(url)\n            .header(\"set-cache-control\", \"public, max-age=10\")\n            .send()\n            .await\n            .unwrap();\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        // should not cache the response\n        assert_eq!(headers[\"x-cache-status\"], \"deferred\");\n\n        // MISS\n        let res = reqwest::Client::new()\n            .get(url)\n            .header(\"set-cache-control\", \"public, max-age=10\")\n            .send()\n            .await\n            .unwrap();\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        assert_eq!(headers[\"x-cache-status\"], \"miss\");\n        assert_eq!(res.text().await.unwrap(), \"hello world\");\n    }\n\n    #[tokio::test]\n    async fn test_eviction() {\n        init();\n        let url = \"http://127.0.0.1:6148/file_maker/test_eviction\".to_owned();\n\n        // admit asset 1\n        let res = reqwest::Client::new()\n            .get(url.clone() + \"1\")\n            .header(\"x-set-size\", \"3000\")\n            .header(\"x-eviction\", \"1\") // tell test proxy to use eviction manager\n            .send()\n            .await\n            .unwrap();\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        assert_eq!(headers[\"x-cache-status\"], \"miss\");\n        assert_eq!(res.text().await.unwrap().len(), 3000);\n\n        // admit asset 2\n        let res = reqwest::Client::new()\n            .get(url.clone() + \"2\")\n            .header(\"x-set-size\", \"3000\")\n            .header(\"x-eviction\", \"1\") // tell test proxy to use eviction manager\n            .send()\n            .await\n            .unwrap();\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        assert_eq!(headers[\"x-cache-status\"], \"miss\");\n        assert_eq!(res.text().await.unwrap().len(), 3000);\n\n        // touch asset 2\n        let res = reqwest::Client::new()\n            .get(url.clone() + \"2\")\n            .header(\"x-set-size\", \"3000\")\n            .header(\"x-eviction\", \"1\") // tell test proxy to use eviction manager\n            .send()\n            .await\n            .unwrap();\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        assert_eq!(headers[\"x-cache-status\"], \"hit\");\n        assert_eq!(res.text().await.unwrap().len(), 3000);\n\n        // touch asset 1\n        let res = reqwest::Client::new()\n            .get(url.clone() + \"1\")\n            .header(\"x-set-size\", \"3000\")\n            .header(\"x-eviction\", \"1\") // tell test proxy to use eviction manager\n            .send()\n            .await\n            .unwrap();\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        assert_eq!(headers[\"x-cache-status\"], \"hit\");\n        assert_eq!(res.text().await.unwrap().len(), 3000);\n\n        // admit asset 3\n        let res = reqwest::Client::new()\n            .get(url.clone() + \"3\")\n            .header(\"x-set-size\", \"6000\")\n            .header(\"x-eviction\", \"1\") // tell test proxy to use eviction manager\n            .send()\n            .await\n            .unwrap();\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        assert_eq!(headers[\"x-cache-status\"], \"miss\");\n        assert_eq!(res.text().await.unwrap().len(), 6000);\n\n        // check asset 2, it should be evicted already because admitting asset 3 made it full\n        let res = reqwest::Client::new()\n            .get(url.clone() + \"2\")\n            .header(\"x-set-size\", \"3000\")\n            .header(\"x-eviction\", \"1\") // tell test proxy to use eviction manager\n            .send()\n            .await\n            .unwrap();\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        assert_eq!(headers[\"x-cache-status\"], \"miss\"); // evicted\n        assert_eq!(res.text().await.unwrap().len(), 3000);\n    }\n\n    #[tokio::test]\n    async fn test_cache_lock_miss_hit() {\n        init();\n        let url = \"http://127.0.0.1:6148/sleep/test_cache_lock_miss_hit.txt\";\n\n        // no lock, parallel fetches to a slow origin are all misses\n        tokio::spawn(async move {\n            let res = reqwest::Client::new().get(url).send().await.unwrap();\n            assert_eq!(res.status(), StatusCode::OK);\n            let headers = res.headers();\n            assert_eq!(headers[\"x-cache-status\"], \"miss\");\n            assert_eq!(res.text().await.unwrap(), \"hello world\");\n        });\n\n        tokio::spawn(async move {\n            let res = reqwest::Client::new().get(url).send().await.unwrap();\n            assert_eq!(res.status(), StatusCode::OK);\n            let headers = res.headers();\n            assert_eq!(headers[\"x-cache-status\"], \"miss\");\n            assert_eq!(res.text().await.unwrap(), \"hello world\");\n        });\n\n        tokio::spawn(async move {\n            let res = reqwest::Client::new().get(url).send().await.unwrap();\n            assert_eq!(res.status(), StatusCode::OK);\n            let headers = res.headers();\n            assert_eq!(headers[\"x-cache-status\"], \"miss\");\n            assert_eq!(res.text().await.unwrap(), \"hello world\");\n        })\n        .await\n        .unwrap(); // wait for at least one of them to finish\n\n        let res = reqwest::Client::new().get(url).send().await.unwrap();\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        assert_eq!(headers[\"x-cache-status\"], \"hit\");\n        assert_eq!(res.text().await.unwrap(), \"hello world\");\n\n        // try with lock\n        let url = \"http://127.0.0.1:6148/sleep/test_cache_lock_miss_hit2.txt\";\n        let task1 = tokio::spawn(async move {\n            let res = reqwest::Client::new()\n                .get(url)\n                .header(\"x-lock\", \"true\")\n                .send()\n                .await\n                .unwrap();\n            assert_eq!(res.status(), StatusCode::OK);\n            let headers = res.headers();\n            assert_eq!(headers[\"x-cache-status\"], \"miss\");\n            assert_eq!(res.text().await.unwrap(), \"hello world\");\n        });\n        // sleep just a little to make sure the req above gets the cache lock\n        sleep(Duration::from_millis(50)).await;\n        let task2 = tokio::spawn(async move {\n            let res = reqwest::Client::new()\n                .get(url)\n                .header(\"x-lock\", \"true\")\n                .send()\n                .await\n                .unwrap();\n            assert_eq!(res.status(), StatusCode::OK);\n            let headers = res.headers();\n            assert_eq!(headers[\"x-cache-status\"], \"hit\");\n            let lock_time_ms: u32 = headers[\"x-cache-lock-time-ms\"]\n                .to_str()\n                .unwrap()\n                .parse()\n                .unwrap();\n            assert!(lock_time_ms > 900 && lock_time_ms < 1000);\n            assert_eq!(res.text().await.unwrap(), \"hello world\");\n        });\n        let task3 = tokio::spawn(async move {\n            let res = reqwest::Client::new()\n                .get(url)\n                .header(\"x-lock\", \"true\")\n                .send()\n                .await\n                .unwrap();\n            assert_eq!(res.status(), StatusCode::OK);\n            let headers = res.headers();\n            assert_eq!(headers[\"x-cache-status\"], \"hit\");\n            let lock_time_ms: u32 = headers[\"x-cache-lock-time-ms\"]\n                .to_str()\n                .unwrap()\n                .parse()\n                .unwrap();\n            assert!(lock_time_ms > 900 && lock_time_ms < 1000);\n            assert_eq!(res.text().await.unwrap(), \"hello world\");\n        });\n\n        task1.await.unwrap();\n        task2.await.unwrap();\n        task3.await.unwrap();\n    }\n\n    #[tokio::test]\n    async fn test_cache_lock_expired() {\n        init();\n        let url = \"http://127.0.0.1:6148/sleep/test_cache_lock_expired.txt\";\n\n        // cache one\n        let res = reqwest::Client::new()\n            .get(url)\n            .header(\"x-no-stale-revalidate\", \"true\")\n            .send()\n            .await\n            .unwrap();\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        assert_eq!(headers[\"x-cache-status\"], \"miss\");\n        assert_eq!(res.text().await.unwrap(), \"hello world\");\n        // let it stale\n        sleep(Duration::from_secs(1)).await;\n\n        let task1 = tokio::spawn(async move {\n            let res = reqwest::Client::new()\n                .get(url)\n                .header(\"x-lock\", \"true\")\n                .header(\"x-no-stale-revalidate\", \"true\")\n                .send()\n                .await\n                .unwrap();\n            assert_eq!(res.status(), StatusCode::OK);\n            let headers = res.headers();\n            assert_eq!(headers[\"x-cache-status\"], \"expired\");\n            assert_eq!(res.text().await.unwrap(), \"hello world\");\n        });\n        // sleep just a little to make sure the req above gets the cache lock\n        sleep(Duration::from_millis(50)).await;\n        let task2 = tokio::spawn(async move {\n            let res = reqwest::Client::new()\n                .get(url)\n                .header(\"x-lock\", \"true\")\n                .header(\"x-no-stale-revalidate\", \"true\")\n                .send()\n                .await\n                .unwrap();\n            assert_eq!(res.status(), StatusCode::OK);\n            let headers = res.headers();\n            assert_eq!(headers[\"x-cache-status\"], \"hit\");\n            assert_eq!(res.text().await.unwrap(), \"hello world\");\n        });\n        let task3 = tokio::spawn(async move {\n            let res = reqwest::Client::new()\n                .get(url)\n                .header(\"x-lock\", \"true\")\n                .header(\"x-no-stale-revalidate\", \"true\")\n                .send()\n                .await\n                .unwrap();\n            assert_eq!(res.status(), StatusCode::OK);\n            let headers = res.headers();\n            assert_eq!(headers[\"x-cache-status\"], \"hit\");\n            assert_eq!(res.text().await.unwrap(), \"hello world\");\n        });\n\n        task1.await.unwrap();\n        task2.await.unwrap();\n        task3.await.unwrap();\n    }\n\n    #[tokio::test]\n    async fn test_cache_lock_network_error() {\n        init();\n        let url = \"http://127.0.0.1:6148/sleep/test_cache_lock_network_error.txt\";\n\n        let task1 = tokio::spawn(async move {\n            let res = reqwest::Client::new()\n                .get(url)\n                .header(\"x-lock\", \"true\")\n                .header(\"x-set-sleep\", \"0.3\") // sometimes we hit the retry logic which is x3 slow\n                .header(\"x-abort\", \"true\") // this will tell origin to kill the conn right away\n                .send()\n                .await\n                .unwrap();\n            assert_eq!(res.status(), 502); // error happened\n        });\n        // sleep just a little to make sure the req above gets the cache lock\n        sleep(Duration::from_millis(50)).await;\n\n        let task2 = tokio::spawn(async move {\n            let res = reqwest::Client::new()\n                .get(url)\n                .header(\"x-lock\", \"true\")\n                .send()\n                .await\n                .unwrap();\n            assert_eq!(res.status(), StatusCode::OK);\n            let headers = res.headers();\n            let status = headers[\"x-cache-status\"].to_owned();\n            assert_eq!(res.text().await.unwrap(), \"hello world\");\n            status\n        });\n        let task3 = tokio::spawn(async move {\n            let res = reqwest::Client::new()\n                .get(url)\n                .header(\"x-lock\", \"true\")\n                .send()\n                .await\n                .unwrap();\n            assert_eq!(res.status(), StatusCode::OK);\n            let headers = res.headers();\n            let status = headers[\"x-cache-status\"].to_owned();\n            assert_eq!(res.text().await.unwrap(), \"hello world\");\n            status\n        });\n\n        task1.await.unwrap();\n        let status2 = task2.await.unwrap();\n        let status3 = task3.await.unwrap();\n\n        let mut count_miss = 0;\n        if status2 == \"miss\" {\n            count_miss += 1;\n        }\n        if status3 == \"miss\" {\n            count_miss += 1;\n        }\n        assert_eq!(count_miss, 1);\n    }\n\n    #[tokio::test]\n    async fn test_cache_lock_uncacheable() {\n        init();\n        let url = \"http://127.0.0.1:6148/sleep/test_cache_lock_uncacheable.txt\";\n\n        let task1 = tokio::spawn(async move {\n            let res = reqwest::Client::new()\n                .get(url)\n                .header(\"x-lock\", \"true\")\n                .header(\"x-no-store\", \"true\") // tell origin to return CC: no-store\n                .send()\n                .await\n                .unwrap();\n            assert_eq!(res.status(), 200);\n            let headers = res.headers();\n            assert_eq!(headers[\"x-cache-status\"], \"no-cache\");\n            assert_eq!(res.text().await.unwrap(), \"hello world\");\n        });\n        // sleep just a little to make sure the req above gets the cache lock\n        sleep(Duration::from_millis(50)).await;\n\n        let task2 = tokio::spawn(async move {\n            let res = reqwest::Client::new()\n                .get(url)\n                .header(\"x-lock\", \"true\")\n                .send()\n                .await\n                .unwrap();\n            assert_eq!(res.status(), StatusCode::OK);\n            let headers = res.headers();\n            assert_eq!(headers[\"x-cache-status\"], \"no-cache\");\n            assert_eq!(res.text().await.unwrap(), \"hello world\");\n        });\n        let task3 = tokio::spawn(async move {\n            let res = reqwest::Client::new()\n                .get(url)\n                .header(\"x-lock\", \"true\")\n                .send()\n                .await\n                .unwrap();\n            assert_eq!(res.status(), StatusCode::OK);\n            let headers = res.headers();\n            assert_eq!(headers[\"x-cache-status\"], \"no-cache\");\n            assert_eq!(res.text().await.unwrap(), \"hello world\");\n        });\n\n        task1.await.unwrap();\n        task2.await.unwrap();\n        task3.await.unwrap();\n    }\n\n    #[tokio::test]\n    async fn test_cache_lock_timeout() {\n        init();\n        let url = \"http://127.0.0.1:6148/sleep/test_cache_lock_timeout.txt\";\n\n        let task1 = tokio::spawn(async move {\n            let res = reqwest::Client::new()\n                .get(url)\n                .header(\"x-lock\", \"true\")\n                .header(\"x-set-sleep\", \"3\") // we have a 2 second cache lock timeout\n                .send()\n                .await\n                .unwrap();\n            assert_eq!(res.status(), 200);\n            let headers = res.headers();\n            assert_eq!(headers[\"x-cache-status\"], \"miss\");\n            assert_eq!(res.text().await.unwrap(), \"hello world\");\n        });\n        // sleep just a little to make sure the req above gets the cache lock\n        sleep(Duration::from_millis(50)).await;\n\n        let task2 = tokio::spawn(async move {\n            let res = reqwest::Client::new()\n                .get(url)\n                .header(\"x-lock\", \"true\")\n                .header(\"x-set-sleep\", \"0.1\") // tell origin to return faster\n                .send()\n                .await\n                .unwrap();\n            assert_eq!(res.status(), StatusCode::OK);\n            let headers = res.headers();\n            // cache lock timeout, try to replace lock\n            assert_eq!(headers[\"x-cache-status\"], \"miss\");\n            assert_eq!(res.text().await.unwrap(), \"hello world\");\n        });\n\n        // send the 3rd request after the 2 second cache lock timeout where the\n        // first request still holds the lock (3s delay in origin)\n        sleep(Duration::from_millis(2200)).await;\n        let task3 = tokio::spawn(async move {\n            let res = reqwest::Client::new()\n                .get(url)\n                .header(\"x-lock\", \"true\")\n                .header(\"x-set-sleep\", \"0.1\")\n                .send()\n                .await\n                .unwrap();\n            assert_eq!(res.status(), StatusCode::OK);\n            let headers = res.headers();\n            // this is now a hit because the second task cached from origin\n            // successfully\n            // and will fetch from origin successfully\n            assert_eq!(headers[\"x-cache-status\"], \"hit\");\n            assert_eq!(res.text().await.unwrap(), \"hello world\");\n        });\n\n        task1.await.unwrap();\n        task2.await.unwrap();\n        task3.await.unwrap();\n    }\n\n    #[tokio::test]\n    async fn test_cache_lock_wait_timeout() {\n        init();\n        let url = \"http://127.0.0.1:6148/sleep/test_cache_lock_wait_timeout.txt\";\n\n        let mut handles = vec![];\n        const N_REQUESTS: u64 = 8;\n        for _ in 0..N_REQUESTS {\n            handles.push(tokio::spawn(async move {\n                // Each task will attempt to wait for the origin's 1s sleep upon acquiring the\n                // cache lock, before the origin disconnects.\n                let res = reqwest::Client::new()\n                    .get(url)\n                    .header(\"x-lock\", \"true\")\n                    .header(\"x-set-sleep\", \"1\") // we have a 2 second cache lock timeout\n                    .header(\"x-abort\", \"1\")\n                    .send()\n                    .await\n                    .unwrap();\n                assert_eq!(res.status(), 502);\n                let headers = res.headers();\n                headers\n                    .get(\"x-cache-lock-time-ms\")\n                    .and_then(|ms| ms.to_str().ok().and_then(|s| s.parse::<u64>().ok()))\n            }));\n        }\n        let mut waited_count = 0;\n        for handle in handles {\n            let lock_time_ms = handle.await.unwrap();\n            if let Some(lock_time_ms) = lock_time_ms {\n                // should not have waited more than 2s for each response\n                waited_count += 1;\n                assert!(lock_time_ms <= 2200);\n            }\n        }\n        assert!(waited_count > 0, \"at least one reader waited\");\n        // This whole process /should/ have taken no longer than 4s, as each reader has an\n        // independently enforced 2s timeout\n    }\n\n    #[tokio::test]\n    async fn test_cache_serve_stale_network_error() {\n        init();\n        let url = \"http://127.0.0.1:6148/sleep/test_cache_serve_stale_network_error.txt\";\n\n        let res = reqwest::Client::new()\n            .get(url)\n            .header(\"x-set-sleep\", \"0\") // no need to sleep we just reuse this endpoint\n            .send()\n            .await\n            .unwrap();\n        assert_eq!(res.status(), 200);\n        let headers = res.headers();\n        assert_eq!(headers[\"x-cache-status\"], \"miss\");\n        assert_eq!(res.text().await.unwrap(), \"hello world\");\n\n        sleep(Duration::from_millis(1100)).await; // ttl is 1\n\n        let res = reqwest::Client::new()\n            .get(url)\n            .header(\"x-set-sleep\", \"0\") // no need to sleep we just reuse this endpoint\n            .header(\"x-abort\", \"true\") // this will tell origin to kill the conn right away\n            .send()\n            .await\n            .unwrap();\n        assert_eq!(res.status(), 200);\n        let headers = res.headers();\n        assert_eq!(headers[\"x-cache-status\"], \"stale\");\n        assert_eq!(res.text().await.unwrap(), \"hello world\");\n    }\n\n    #[tokio::test]\n    async fn test_cache_serve_stale_network_error_mid_response() {\n        init();\n        let url =\n            \"http://127.0.0.1:6148/sleep/test_cache_serve_stale_network_error_mid_response.txt\";\n\n        let res = reqwest::Client::new()\n            .get(url)\n            .header(\"x-set-sleep\", \"0\") // no need to sleep we just reuse this endpoint\n            .send()\n            .await\n            .unwrap();\n        assert_eq!(res.status(), 200);\n        let headers = res.headers();\n        assert_eq!(headers[\"x-cache-status\"], \"miss\");\n        assert_eq!(res.text().await.unwrap(), \"hello world\");\n\n        sleep(Duration::from_millis(1100)).await; // ttl is 1\n\n        let res = reqwest::Client::new()\n            .get(url)\n            .header(\"x-set-sleep\", \"0\") // no need to sleep we just reuse this endpoint\n            .header(\"x-set-body-sleep\", \"0.1\") // pause the body a bit before abort\n            .header(\"x-abort-body\", \"true\") // this will tell origin to kill the conn right away\n            .send()\n            .await\n            .unwrap();\n        assert_eq!(res.status(), 200);\n        let headers = res.headers();\n        assert_eq!(headers[\"x-cache-status\"], \"expired\");\n        // the connection dies\n        assert!(res.text().await.is_err());\n    }\n\n    #[tokio::test]\n    async fn test_cache_serve_stale_on_500() {\n        init();\n        let url = \"http://127.0.0.1:6148/sleep/test_cache_serve_stale_on_500.txt\";\n\n        let res = reqwest::Client::new()\n            .get(url)\n            .header(\"x-set-sleep\", \"0\") // no need to sleep we just reuse this endpoint\n            .send()\n            .await\n            .unwrap();\n        assert_eq!(res.status(), 200);\n        let headers = res.headers();\n        assert_eq!(headers[\"x-cache-status\"], \"miss\");\n        assert_eq!(res.text().await.unwrap(), \"hello world\");\n\n        sleep(Duration::from_millis(1100)).await; // ttl is 1\n\n        let res = reqwest::Client::new()\n            .get(url)\n            .header(\"x-set-sleep\", \"0\") // no need to sleep we just reuse this endpoint\n            .header(\"x-error-header\", \"true\") // this will tell origin to return 500\n            .send()\n            .await\n            .unwrap();\n        assert_eq!(res.status(), 200);\n        let headers = res.headers();\n        assert_eq!(headers[\"x-cache-status\"], \"stale\");\n        assert_eq!(res.text().await.unwrap(), \"hello world\");\n    }\n\n    #[tokio::test]\n    async fn test_stale_while_revalidate_many_readers() {\n        init();\n        let url = \"http://127.0.0.1:6148/sleep/test_stale_while_revalidate_many_readers.txt\";\n\n        // cache one\n        let res = reqwest::Client::new().get(url).send().await.unwrap();\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        assert_eq!(headers[\"x-cache-status\"], \"miss\");\n        assert_eq!(res.text().await.unwrap(), \"hello world\");\n        // let it stale\n        sleep(Duration::from_secs(1)).await;\n\n        let task1 = tokio::spawn(async move {\n            let res = reqwest::Client::new()\n                .get(url)\n                .header(\"x-lock\", \"true\")\n                .send()\n                .await\n                .unwrap();\n            assert_eq!(res.status(), StatusCode::OK);\n            let headers = res.headers();\n            assert_eq!(headers[\"x-cache-status\"], \"stale-updating\");\n            assert_eq!(res.text().await.unwrap(), \"hello world\");\n        });\n        // sleep just a little to make sure the req above gets the cache lock\n        sleep(Duration::from_millis(50)).await;\n        let task2 = tokio::spawn(async move {\n            let res = reqwest::Client::new()\n                .get(url)\n                .header(\"x-lock\", \"true\")\n                .send()\n                .await\n                .unwrap();\n            assert_eq!(res.status(), StatusCode::OK);\n            let headers = res.headers();\n            assert_eq!(headers[\"x-cache-status\"], \"stale-updating\");\n            assert_eq!(res.text().await.unwrap(), \"hello world\");\n        });\n        let task3 = tokio::spawn(async move {\n            let res = reqwest::Client::new()\n                .get(url)\n                .header(\"x-lock\", \"true\")\n                .send()\n                .await\n                .unwrap();\n            assert_eq!(res.status(), StatusCode::OK);\n            let headers = res.headers();\n            assert_eq!(headers[\"x-cache-status\"], \"stale-updating\");\n            assert_eq!(res.text().await.unwrap(), \"hello world\");\n        });\n\n        task1.await.unwrap();\n        task2.await.unwrap();\n        task3.await.unwrap();\n    }\n\n    #[tokio::test]\n    async fn test_stale_while_revalidate_single_request() {\n        init();\n        let url = \"http://127.0.0.1:6148/sleep/test_stale_while_revalidate_single_request.txt\";\n\n        // cache one\n        let res = reqwest::Client::new()\n            .get(url)\n            .header(\"x-set-sleep\", \"0\")\n            .send()\n            .await\n            .unwrap();\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        assert_eq!(headers[\"x-cache-status\"], \"miss\");\n        assert_eq!(res.text().await.unwrap(), \"hello world\");\n        // let it stale\n        sleep(Duration::from_secs(1)).await;\n\n        let res = reqwest::Client::new()\n            .get(url)\n            .header(\"x-lock\", \"true\")\n            .header(\"x-set-sleep\", \"0\") // by default /sleep endpoint will sleep 1s\n            .send()\n            .await\n            .unwrap();\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        assert_eq!(headers[\"x-cache-status\"], \"stale-updating\");\n        assert_eq!(res.text().await.unwrap(), \"hello world\");\n\n        // wait for the background request to finish\n        sleep(Duration::from_millis(100)).await;\n\n        let res = reqwest::Client::new()\n            .get(url)\n            .header(\"x-lock\", \"true\")\n            .send()\n            .await\n            .unwrap();\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        assert_eq!(headers[\"x-cache-status\"], \"hit\"); // fresh\n        assert_eq!(res.text().await.unwrap(), \"hello world\");\n    }\n\n    #[tokio::test]\n    async fn test_cache_streaming_partial_body() {\n        init();\n        let url = \"http://127.0.0.1:6148/slow_body/test_cache_streaming_partial_body.txt\";\n        let task1 = tokio::spawn(async move {\n            let res = reqwest::Client::new()\n                .get(url)\n                .header(\"x-lock\", \"true\")\n                .send()\n                .await\n                .unwrap();\n            assert_eq!(res.status(), StatusCode::OK);\n            let headers = res.headers();\n            assert_eq!(headers[\"x-cache-status\"], \"miss\");\n            assert_eq!(res.text().await.unwrap(), \"hello world!\");\n        });\n        // sleep just a little to make sure the req above gets the cache lock\n        sleep(Duration::from_millis(50)).await;\n\n        let task2 = tokio::spawn(async move {\n            let start = Instant::now();\n            let res = reqwest::Client::new()\n                .get(url)\n                .header(\"x-lock\", \"true\")\n                .send()\n                .await\n                .unwrap();\n            assert_eq!(res.status(), StatusCode::OK);\n            let headers = res.headers();\n            assert_eq!(headers[\"x-cache-status\"], \"hit\");\n            let lock_time_ms: u32 = headers[\"x-cache-lock-time-ms\"]\n                .to_str()\n                .unwrap()\n                .parse()\n                .unwrap();\n            let body = res.text().await.unwrap();\n            let total_ms = start.elapsed().as_millis() as u32;\n            // lock should cover only the header, not the full body streaming.\n            // if the body were also under lock, lock_time would approach total_ms.\n            assert!(\n                lock_time_ms < total_ms / 2,\n                \"lock time {lock_time_ms}ms should be well under total request time {total_ms}ms\"\n            );\n            assert_eq!(body, \"hello world!\");\n        });\n        let task3 = tokio::spawn(async move {\n            let start = Instant::now();\n            let res = reqwest::Client::new()\n                .get(url)\n                .header(\"x-lock\", \"true\")\n                .send()\n                .await\n                .unwrap();\n            assert_eq!(res.status(), StatusCode::OK);\n            let headers = res.headers();\n            assert_eq!(headers[\"x-cache-status\"], \"hit\");\n            let lock_time_ms: u32 = headers[\"x-cache-lock-time-ms\"]\n                .to_str()\n                .unwrap()\n                .parse()\n                .unwrap();\n            let body = res.text().await.unwrap();\n            let total_ms = start.elapsed().as_millis() as u32;\n            // lock should cover only the header, not the full body streaming.\n            // if the body were also under lock, lock_time would approach total_ms.\n            assert!(\n                lock_time_ms < total_ms / 2,\n                \"lock time {lock_time_ms}ms should be well under total request time {total_ms}ms\"\n            );\n            assert_eq!(body, \"hello world!\");\n        });\n\n        task1.await.unwrap();\n        task2.await.unwrap();\n        task3.await.unwrap();\n    }\n\n    #[tokio::test]\n    async fn test_cache_streaming_multiple_writers() {\n        // multiple streaming writers don't conflict\n        init();\n        let url = \"http://127.0.0.1:6148/slow_body/test_cache_streaming_multiple_writers.txt\";\n        let task1 = tokio::spawn(async move {\n            let res = reqwest::Client::new()\n                .get(url)\n                .header(\"x-set-hello\", \"everyone\")\n                .send()\n                .await\n                .unwrap();\n            assert_eq!(res.status(), StatusCode::OK);\n            let headers = res.headers();\n            assert_eq!(headers[\"x-cache-status\"], \"miss\");\n            assert_eq!(res.text().await.unwrap(), \"hello everyone!\");\n        });\n\n        let task2 = tokio::spawn(async move {\n            let res = reqwest::Client::new()\n                .get(url)\n                // don't allow using the other streaming write's result\n                .header(\"x-force-expire\", \"1\")\n                .header(\"x-set-hello\", \"todo el mundo\")\n                .send()\n                .await\n                .unwrap();\n            assert_eq!(res.status(), StatusCode::OK);\n            let headers = res.headers();\n            assert_eq!(headers[\"x-cache-status\"], \"miss\");\n            assert_eq!(res.text().await.unwrap(), \"hello todo el mundo!\");\n        });\n\n        task1.await.unwrap();\n        task2.await.unwrap();\n    }\n\n    #[tokio::test]\n    async fn test_range_request() {\n        init();\n        let url = \"http://127.0.0.1:6148/unique/test_range_request/now\";\n\n        let res = reqwest::Client::new()\n            .get(url)\n            .header(\"Range\", \"bytes=0-1\")\n            .send()\n            .await\n            .unwrap();\n        assert_eq!(res.status(), StatusCode::PARTIAL_CONTENT);\n        let headers = res.headers();\n        let cache_miss_epoch = headers[\"x-epoch\"].to_str().unwrap().parse::<f64>().unwrap();\n        assert_eq!(headers[\"x-cache-status\"], \"miss\");\n        assert_eq!(res.text().await.unwrap(), \"he\");\n\n        // full body is cached\n        let res = reqwest::get(url).await.unwrap();\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        let cache_hit_epoch = headers[\"x-epoch\"].to_str().unwrap().parse::<f64>().unwrap();\n        assert_eq!(headers[\"x-cache-status\"], \"hit\");\n        assert_eq!(res.text().await.unwrap(), \"hello world\");\n\n        assert_eq!(cache_miss_epoch, cache_hit_epoch);\n\n        let res = reqwest::Client::new()\n            .get(url)\n            .header(\"Range\", \"bytes=0-1\")\n            .send()\n            .await\n            .unwrap();\n        assert_eq!(res.status(), StatusCode::PARTIAL_CONTENT);\n        let headers = res.headers();\n        assert_eq!(headers[\"x-cache-status\"], \"hit\");\n        assert_eq!(res.text().await.unwrap(), \"he\");\n\n        let res = reqwest::Client::new()\n            .get(url)\n            .header(\"Range\", \"bytes=1-0\")\n            .send()\n            .await\n            .unwrap();\n        assert_eq!(res.status(), StatusCode::RANGE_NOT_SATISFIABLE);\n        let headers = res.headers();\n        assert_eq!(headers[\"x-cache-status\"], \"hit\");\n        assert_eq!(res.text().await.unwrap(), \"\");\n\n        let res = reqwest::Client::new()\n            .head(url)\n            .header(\"Range\", \"bytes=0-1\")\n            .send()\n            .await\n            .unwrap();\n        assert_eq!(res.status(), StatusCode::PARTIAL_CONTENT);\n        let headers = res.headers();\n        assert_eq!(headers[\"x-cache-status\"], \"hit\");\n        assert_eq!(res.text().await.unwrap(), \"\");\n\n        sleep(Duration::from_millis(1100)).await; // ttl is 1\n\n        let res = reqwest::Client::new()\n            .get(url)\n            .header(\"Range\", \"bytes=0-1\")\n            .send()\n            .await\n            .unwrap();\n        assert_eq!(res.status(), StatusCode::PARTIAL_CONTENT);\n        let headers = res.headers();\n        assert_eq!(headers[\"x-cache-status\"], \"expired\");\n        assert_eq!(res.text().await.unwrap(), \"he\");\n    }\n\n    #[tokio::test]\n    async fn test_multipart_range_request() {\n        init();\n        let url = \"http://127.0.0.1:6148/unique/test_multipart_range_request/now\";\n\n        // 1st multipart range request - uncached\n\n        let res = reqwest::Client::new()\n            .get(url)\n            .header(\"Range\", \"bytes=0-1, 6-8\") // he wor\n            .send()\n            .await\n            .unwrap();\n\n        assert_eq!(res.status(), StatusCode::PARTIAL_CONTENT);\n        let headers = res.headers().clone();\n        let content_type = headers.get(\"content-type\").unwrap().to_str().unwrap();\n        // Grab boundary to verify full response\n        let boundary = content_type\n            .split(\"boundary=\")\n            .nth(1)\n            .expect(\"Expected to have a boundary\");\n        assert_eq!(\n            content_type,\n            format!(\"multipart/byteranges; boundary={boundary}\")\n        );\n        assert_eq!(headers[\"x-cache-status\"], \"miss\");\n\n        let body = res.text().await.unwrap();\n\n        let expected_body = format!(\n            \"\\r\\n--{boundary}\\r\\n\\\n            Content-Type: text/plain\\r\\n\\\n            Content-Range: bytes 0-1/11\\r\\n\\\n            \\r\\n\\\n            he\\\n            \\r\\n--{boundary}\\r\\n\\\n            Content-Type: text/plain\\r\\n\\\n            Content-Range: bytes 6-8/11\\r\\n\\\n            \\r\\n\\\n            wor\\\n            \\r\\n--{boundary}--\\r\\n\\\n        \"\n        );\n        assert_eq!(body, expected_body);\n\n        // 2nd request, cached\n\n        let res = reqwest::Client::new()\n            .get(url)\n            .header(\"Range\", \"bytes=2-3, 8-10\") // ll rld\n            .send()\n            .await\n            .unwrap();\n\n        assert_eq!(res.status(), StatusCode::PARTIAL_CONTENT);\n        let headers = res.headers().clone();\n        let content_type = headers.get(\"content-type\").unwrap().to_str().unwrap();\n        let boundary = content_type\n            .split(\"boundary=\")\n            .nth(1)\n            .expect(\"Expected to have a boundary\");\n        assert_eq!(\n            content_type,\n            format!(\"multipart/byteranges; boundary={boundary}\")\n        );\n        assert_eq!(headers[\"x-cache-status\"], \"hit\");\n\n        let body = res.text().await.unwrap();\n\n        let expected_body = format!(\n            \"\\r\\n--{boundary}\\r\\n\\\n            Content-Type: text/plain\\r\\n\\\n            Content-Range: bytes 2-3/11\\r\\n\\\n            \\r\\n\\\n            ll\\\n            \\r\\n--{boundary}\\r\\n\\\n            Content-Type: text/plain\\r\\n\\\n            Content-Range: bytes 8-10/11\\r\\n\\\n            \\r\\n\\\n            rld\\\n            \\r\\n--{boundary}--\\r\\n\\\n        \"\n        );\n        assert_eq!(body, expected_body);\n\n        // 3rd request, cached\n\n        let res = reqwest::Client::new()\n            .get(url)\n            .header(\"Range\", \"bytes=1-2, 3-4, 8-10\") // el lo rld\n            .send()\n            .await\n            .unwrap();\n\n        assert_eq!(res.status(), StatusCode::PARTIAL_CONTENT);\n        let headers = res.headers().clone();\n        let content_type = headers.get(\"content-type\").unwrap().to_str().unwrap();\n        let boundary = content_type\n            .split(\"boundary=\")\n            .nth(1)\n            .expect(\"Expected to have a boundary\");\n        assert_eq!(\n            content_type,\n            format!(\"multipart/byteranges; boundary={boundary}\")\n        );\n        assert_eq!(headers[\"x-cache-status\"], \"hit\");\n\n        let body = res.text().await.unwrap();\n\n        let expected_body = format!(\n            \"\\r\\n--{boundary}\\r\\n\\\n            Content-Type: text/plain\\r\\n\\\n            Content-Range: bytes 1-2/11\\r\\n\\\n            \\r\\n\\\n            el\\\n            \\r\\n--{boundary}\\r\\n\\\n            Content-Type: text/plain\\r\\n\\\n            Content-Range: bytes 3-4/11\\r\\n\\\n            \\r\\n\\\n            lo\\\n            \\r\\n--{boundary}\\r\\n\\\n            Content-Type: text/plain\\r\\n\\\n            Content-Range: bytes 8-10/11\\r\\n\\\n            \\r\\n\\\n            rld\\\n            \\r\\n--{boundary}--\\r\\n\\\n        \"\n        );\n        assert_eq!(body, expected_body);\n\n        // 4th request - cached and poorly formed request header\n\n        let res = reqwest::Client::new()\n            .get(url)\n            .header(\"Range\", \"bytes=2-3, 8-10, 3-5\")\n            .send()\n            .await\n            .unwrap();\n\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers().clone();\n        assert_eq!(headers[\"content-type\"], \"text/plain\");\n        assert_eq!(headers[\"x-cache-status\"], \"hit\");\n\n        let body = res.text().await.unwrap();\n        assert_eq!(body, \"hello world\");\n\n        // 5th request: Single range GET\n\n        let res = reqwest::Client::new()\n            .get(url)\n            .header(\"Range\", \"bytes=0-2\")\n            .send()\n            .await\n            .unwrap();\n\n        assert_eq!(res.status(), StatusCode::PARTIAL_CONTENT);\n        let headers = res.headers();\n        assert_eq!(headers[\"x-cache-status\"], \"hit\");\n\n        let body = res.text().await.unwrap();\n        assert_eq!(body, \"hel\");\n\n        // 6th request invalid range\n\n        let res = reqwest::Client::new()\n            .get(url)\n            .header(\"Range\", \"bytes=20-22, 30-40\")\n            .send()\n            .await\n            .unwrap();\n        assert_eq!(res.status(), StatusCode::RANGE_NOT_SATISFIABLE);\n        let headers = res.headers();\n        assert_eq!(headers[\"x-cache-status\"], \"hit\");\n        assert_eq!(res.text().await.unwrap(), \"\");\n\n        // 7th request: Single range HEAD\n\n        let res = reqwest::Client::new()\n            .head(url)\n            .header(\"Range\", \"bytes=3-7\")\n            .send()\n            .await\n            .unwrap();\n\n        assert_eq!(res.status(), StatusCode::PARTIAL_CONTENT);\n        let headers = res.headers();\n        assert_eq!(headers[\"x-cache-status\"], \"hit\");\n\n        let body = res.text().await.unwrap();\n        assert_eq!(body, \"\");\n    }\n\n    #[tokio::test]\n    async fn test_single_then_mutltipart_range_request() {\n        init();\n        let url = \"http://127.0.0.1:6148/unique/test_single_then_multipart_range_request/now\";\n\n        // 1st request - single range request\n\n        let res = reqwest::Client::new()\n            .get(url)\n            .header(\"Range\", \"bytes=0-4\")\n            .send()\n            .await\n            .unwrap();\n\n        assert_eq!(res.status(), StatusCode::PARTIAL_CONTENT);\n        let headers = res.headers();\n        assert_eq!(headers[\"x-cache-status\"], \"miss\");\n\n        let body = res.text().await.unwrap();\n        assert_eq!(body, \"hello\");\n\n        // 2nd request - multipart range request\n\n        let res = reqwest::Client::new()\n            .get(url)\n            .header(\"Range\", \"bytes=0-3, 6-7\") // hell wo\n            .send()\n            .await\n            .unwrap();\n\n        assert_eq!(res.status(), StatusCode::PARTIAL_CONTENT);\n        let headers = res.headers().clone();\n        let content_type = headers.get(\"content-type\").unwrap().to_str().unwrap();\n        let boundary = content_type\n            .split(\"boundary=\")\n            .nth(1)\n            .expect(\"Expected to have a boundary\");\n        assert_eq!(\n            content_type,\n            format!(\"multipart/byteranges; boundary={boundary}\")\n        );\n        assert_eq!(headers[\"x-cache-status\"], \"hit\");\n\n        let body = res.text().await.unwrap();\n\n        let expected_body = format!(\n            \"\\r\\n--{boundary}\\r\\n\\\n            Content-Type: text/plain\\r\\n\\\n            Content-Range: bytes 0-3/11\\r\\n\\\n            \\r\\n\\\n            hell\\\n            \\r\\n--{boundary}\\r\\n\\\n            Content-Type: text/plain\\r\\n\\\n            Content-Range: bytes 6-7/11\\r\\n\\\n            \\r\\n\\\n            wo\\\n            \\r\\n--{boundary}--\\r\\n\\\n        \"\n        );\n        assert_eq!(body, expected_body);\n\n        // 3rd request - Multipart request with one valid range\n        let res = reqwest::Client::new()\n            .get(url)\n            .header(\"Range\", \"bytes=0-4, 12-14, 16-18\") // hello\n            .send()\n            .await\n            .unwrap();\n        assert_eq!(res.status(), StatusCode::PARTIAL_CONTENT);\n        let headers = res.headers();\n        let content_type = headers.get(\"content-type\").unwrap().to_str().unwrap();\n        assert!(!content_type.contains(\"multipart/byteranges; boundary=\"));\n        assert_eq!(headers[\"x-cache-status\"], \"hit\");\n\n        assert_eq!(res.text().await.unwrap(), \"hello\");\n    }\n\n    #[tokio::test]\n    async fn test_caching_when_downstream_bails() {\n        init();\n        let url = \"http://127.0.0.1:6148/slow_body/test_caching_when_downstream_bails/\";\n\n        tokio::spawn(async move {\n            let res = reqwest::Client::new()\n                .get(url)\n                .header(\"x-lock\", \"true\")\n                .send()\n                .await\n                .unwrap();\n            assert_eq!(res.status(), StatusCode::OK);\n            let headers = res.headers();\n            assert_eq!(headers[\"x-cache-status\"], \"miss\");\n            // exit without res.text().await so that we bail early\n        });\n        // sleep just a little to make sure the req above gets the cache lock\n        sleep(Duration::from_millis(50)).await;\n\n        let res = reqwest::Client::new()\n            .get(url)\n            .header(\"x-lock\", \"true\")\n            .send()\n            .await\n            .unwrap();\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        assert_eq!(headers[\"x-cache-status\"], \"hit\");\n        let lock_time_ms: u32 = headers[\"x-cache-lock-time-ms\"]\n            .to_str()\n            .unwrap()\n            .parse()\n            .unwrap();\n        // the entire body should need 2 extra seconds, here the test shows that\n        // only the header is under cache lock and the body should be streamed\n        assert!(lock_time_ms > 900 && lock_time_ms < 1000);\n        assert_eq!(res.text().await.unwrap(), \"hello world!\");\n    }\n\n    #[tokio::test]\n    async fn test_caching_when_downstream_bails_uncacheable() {\n        init();\n        let url = \"http://127.0.0.1:6148/slow_body/test_caching_when_downstream_bails_uncacheable/\";\n\n        tokio::spawn(async move {\n            let res = reqwest::Client::new()\n                .get(url)\n                .header(\"x-lock\", \"true\")\n                .header(\"x-no-store\", \"1\")\n                .send()\n                .await\n                .unwrap();\n            assert_eq!(res.status(), StatusCode::OK);\n            let headers = res.headers();\n            assert_eq!(headers[\"x-cache-status\"], \"no-cache\");\n            // exit without res.text().await so that we bail early\n        });\n        // sleep just a little to make sure the req above gets the cache lock\n        sleep(Duration::from_millis(50)).await;\n\n        let res = reqwest::Client::new()\n            .get(url)\n            .header(\"x-lock\", \"true\")\n            .send()\n            .await\n            .unwrap();\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        // entirely new request made to upstream, since the response was uncacheable\n        assert_eq!(headers[\"x-cache-status\"], \"no-cache\"); // due to cache lock give up\n        assert_eq!(res.text().await.unwrap(), \"hello world!\");\n    }\n\n    #[tokio::test]\n    async fn test_caching_when_downstream_bails_header() {\n        init();\n        let url = \"http://127.0.0.1:6148/unique/test_caching_when_downstream_bails_header/sleep\";\n\n        tokio::spawn(async move {\n            // this should always time out\n            reqwest::Client::new()\n                .get(url)\n                .header(\"x-lock\", \"true\")\n                .header(\"x-set-sleep\", \"2\")\n                .timeout(Duration::from_secs(1))\n                .send()\n                .await\n                .unwrap_err()\n        });\n        // sleep after cache fill\n        sleep(Duration::from_millis(2500)).await;\n\n        // next request should be a cache hit\n        let res = reqwest::Client::new()\n            .get(url)\n            .header(\"x-lock\", \"true\")\n            .send()\n            .await\n            .unwrap();\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        assert_eq!(headers[\"x-cache-status\"], \"hit\");\n        assert_eq!(res.text().await.unwrap(), \"hello world\");\n    }\n\n    #[tokio::test]\n    async fn test_caching_when_downstream_bails_header_uncacheable() {\n        init();\n        let url = \"http://127.0.0.1:6148/unique/test_caching_when_downstream_bails_header_uncacheable/sleep\";\n\n        tokio::spawn(async move {\n            // this should always time out\n            reqwest::Client::new()\n                .get(url)\n                .header(\"x-lock\", \"true\")\n                .header(\"x-set-sleep\", \"2\")\n                .header(\"x-no-store\", \"1\")\n                .timeout(Duration::from_secs(1))\n                .send()\n                .await\n                .unwrap_err()\n            // note that while the downstream error is ignored,\n            // once the response is uncacheable we will still attempt to write\n            // downstream and find a broken connection that terminates the request\n        });\n        // sleep after cache fill\n        sleep(Duration::from_millis(2500)).await;\n\n        // next request should be a cache miss, as the previous fill was uncacheable\n        let res = reqwest::Client::new()\n            .get(url)\n            .header(\"x-lock\", \"true\")\n            .send()\n            .await\n            .unwrap();\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        assert_eq!(headers[\"x-cache-status\"], \"miss\");\n        assert_eq!(res.text().await.unwrap(), \"hello world\");\n    }\n\n    async fn send_vary_req_with_headers_with_dups(\n        url: &str,\n        vary_field: &str,\n        headers: Vec<(&str, &str)>,\n        dup_headers: Vec<(&str, &str)>,\n    ) -> reqwest::Response {\n        let req_headers = headers\n            .iter()\n            .map(|(name, value)| {\n                (\n                    HeaderName::from_str(name).unwrap(),\n                    HeaderValue::from_str(value).unwrap(),\n                )\n            })\n            .collect();\n        let mut req_builder = reqwest::Client::new()\n            .get(url)\n            .headers(req_headers)\n            .header(\"set-vary\", vary_field);\n\n        // Apply any duplicate headers\n        for (key, value) in dup_headers {\n            req_builder = req_builder.header(key, value);\n        }\n\n        req_builder.send().await.unwrap()\n    }\n\n    async fn send_vary_req_with_headers(\n        url: &str,\n        vary_field: &str,\n        headers: Vec<(&str, &str)>,\n    ) -> reqwest::Response {\n        send_vary_req_with_headers_with_dups(url, vary_field, headers, vec![]).await\n    }\n\n    async fn send_vary_req(url: &str, vary_field: &str, value: &str) -> reqwest::Response {\n        send_vary_req_with_headers(url, vary_field, vec![(vary_field, value)]).await\n    }\n\n    #[tokio::test]\n    async fn test_vary_caching() {\n        init();\n        let url = \"http://127.0.0.1:6148/unique/test_vary_caching/vary\";\n        let vary_field = \"Accept\";\n\n        let res = send_vary_req(url, vary_field, \"image/png\").await;\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        let cache_a_miss_epoch = headers[\"x-epoch\"].to_str().unwrap().parse::<f64>().unwrap();\n        assert_eq!(headers[\"x-cache-status\"], \"miss\");\n        assert_eq!(res.text().await.unwrap(), \"hello world\");\n\n        let res = send_vary_req(url, vary_field, \"image/png\").await;\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        let cache_hit_epoch = headers[\"x-epoch\"].to_str().unwrap().parse::<f64>().unwrap();\n        assert_eq!(headers[\"x-cache-status\"], \"hit\");\n        assert_eq!(res.text().await.unwrap(), \"hello world\");\n\n        assert_eq!(cache_a_miss_epoch, cache_hit_epoch);\n\n        let res = send_vary_req(url, vary_field, \"image/jpeg\").await;\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        let cache_b_miss_epoch = headers[\"x-epoch\"].to_str().unwrap().parse::<f64>().unwrap();\n        assert_eq!(headers[\"x-cache-status\"], \"miss\");\n        assert_eq!(res.text().await.unwrap(), \"hello world\");\n\n        let res = send_vary_req(url, vary_field, \"image/jpeg\").await;\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        let cache_hit_epoch = headers[\"x-epoch\"].to_str().unwrap().parse::<f64>().unwrap();\n        assert_eq!(headers[\"x-cache-status\"], \"hit\");\n        assert_eq!(res.text().await.unwrap(), \"hello world\");\n\n        assert_eq!(cache_b_miss_epoch, cache_hit_epoch);\n        assert!(cache_a_miss_epoch != cache_b_miss_epoch);\n    }\n\n    #[tokio::test]\n    async fn test_vary_caching_ignored_vary_header() {\n        init();\n        let url = \"http://127.0.0.1:6148/unique/test_vary_caching_ignored_vary_header/vary\";\n        let vary_field = \"Some-Ignored-Vary-Header\";\n\n        // Asset into cache (png)\n        let res = send_vary_req(url, vary_field, \"image/png\").await;\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        let epoch = headers[\"x-epoch\"].to_str().unwrap().parse::<f64>().unwrap();\n        assert_eq!(headers[\"x-cache-status\"], \"miss\");\n        assert_eq!(res.text().await.unwrap(), \"hello world\");\n\n        // HIT on png\n        let res = send_vary_req(url, vary_field, \"image/png\").await;\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        assert_eq!(\n            epoch,\n            headers[\"x-epoch\"].to_str().unwrap().parse::<f64>().unwrap()\n        );\n        assert_eq!(headers[\"x-cache-status\"], \"hit\");\n        assert_eq!(res.text().await.unwrap(), \"hello world\");\n\n        // Vary header ignored -> get png, not jpeg\n        let res = send_vary_req(url, vary_field, \"image/jpeg\").await;\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        assert_eq!(\n            epoch,\n            headers[\"x-epoch\"].to_str().unwrap().parse::<f64>().unwrap()\n        );\n        assert_eq!(headers[\"x-cache-status\"], \"hit\");\n    }\n\n    #[tokio::test]\n    async fn test_vary_some_ignored() {\n        init();\n        let url = \"http://127.0.0.1:6148/unique/test_vary_some_ignored/vary\";\n        let vary_header = \"Accept, SomeIgnoredVaryHeader\";\n\n        // Make a request where we vary on some headers, and provide values for those.\n        let res = send_vary_req_with_headers(\n            url,\n            vary_header,\n            vec![\n                (\"SomeIgnoredVaryHeader\", \"image/webp\"),\n                (\"Accept\", \"image/webp\"),\n            ],\n        )\n        .await;\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        assert_eq!(headers[\"x-cache-status\"], \"miss\");\n        let epoch1 = headers[\"x-epoch\"].to_str().unwrap().parse::<f64>().unwrap();\n        assert_eq!(res.text().await.unwrap(), \"hello world\");\n\n        // Identical request should yield a HIT.\n        let res = send_vary_req_with_headers(\n            url,\n            vary_header,\n            vec![\n                (\"SomeIgnoredVaryHeader\", \"image/webp\"),\n                (\"Accept\", \"image/webp\"),\n            ],\n        )\n        .await;\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        assert_eq!(headers[\"x-cache-status\"], \"hit\");\n        assert_eq!(\n            epoch1,\n            headers[\"x-epoch\"].to_str().unwrap().parse::<f64>().unwrap()\n        );\n        assert_eq!(res.text().await.unwrap(), \"hello world\");\n\n        // Hit when changing a header we don't vary on.\n        let res = send_vary_req_with_headers(\n            url,\n            vary_header,\n            vec![\n                (\"SomeIgnoredVaryHeader\", \"definitely-not-webp\"),\n                (\"Accept\", \"image/webp\"),\n            ],\n        )\n        .await;\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        assert_eq!(headers[\"x-cache-status\"], \"hit\");\n        assert_eq!(\n            epoch1,\n            headers[\"x-epoch\"].to_str().unwrap().parse::<f64>().unwrap()\n        );\n        assert_eq!(res.text().await.unwrap(), \"hello world\");\n\n        // Get a secondary variant by changing Accept.\n        let res = send_vary_req_with_headers(\n            url,\n            vary_header,\n            vec![\n                (\"SomeIgnoredVaryHeader\", \"definitely-not-webp\"),\n                (\"Accept\", \"image/jpeg\"),\n            ],\n        )\n        .await;\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        assert_eq!(headers[\"x-cache-status\"], \"miss\");\n        let epoch2 = headers[\"x-epoch\"].to_str().unwrap().parse::<f64>().unwrap();\n        assert_ne!(epoch1, epoch2);\n        assert_eq!(res.text().await.unwrap(), \"hello world\");\n\n        // Cache hit on secondary variant.\n        let res = send_vary_req_with_headers(\n            url,\n            vary_header,\n            vec![\n                (\"SomeIgnoredVaryHeader\", \"definitely-not-webp\"),\n                (\"Accept\", \"image/jpeg\"),\n            ],\n        )\n        .await;\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        assert_eq!(headers[\"x-cache-status\"], \"hit\");\n        assert_eq!(\n            epoch2,\n            headers[\"x-epoch\"].to_str().unwrap().parse::<f64>().unwrap()\n        );\n\n        // Cache hit on primary variant.\n        let res = send_vary_req_with_headers(\n            url,\n            vary_header,\n            vec![\n                (\"SomeIgnoredVaryHeader\", \"definitely-not-webp\"),\n                (\"Accept\", \"image/webp\"),\n            ],\n        )\n        .await;\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        assert_eq!(headers[\"x-cache-status\"], \"hit\");\n        assert_eq!(\n            epoch1,\n            headers[\"x-epoch\"].to_str().unwrap().parse::<f64>().unwrap()\n        );\n    }\n\n    #[tokio::test]\n    async fn test_vary_dup_header_some_ignored() {\n        init();\n        let url = \"http://127.0.0.1:6148/unique/test_vary_dup_header_some_ignored/vary\";\n        let first_vary_header = \"SomeIgnoredVaryHeader\";\n        let dup_vary_header = \"Accept\";\n\n        // Make a request where we vary on some headers, and provide values for those.\n        let res = send_vary_req_with_headers_with_dups(\n            url,\n            first_vary_header,\n            vec![\n                (\"SomeIgnoredVaryHeader\", \"image/webp\"),\n                (\"Accept\", \"image/webp\"),\n            ],\n            vec![(\"set-vary\", dup_vary_header)],\n        )\n        .await;\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        assert_eq!(headers[\"x-cache-status\"], \"miss\");\n        let epoch1 = headers[\"x-epoch\"].to_str().unwrap().parse::<f64>().unwrap();\n        assert_eq!(res.text().await.unwrap(), \"hello world\");\n\n        // Identical request should yield a HIT.\n        let res = send_vary_req_with_headers_with_dups(\n            url,\n            first_vary_header,\n            vec![\n                (\"SomeIgnoredVaryHeader\", \"image/webp\"),\n                (\"Accept\", \"image/webp\"),\n            ],\n            vec![(\"set-vary\", dup_vary_header)],\n        )\n        .await;\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        assert_eq!(headers[\"x-cache-status\"], \"hit\");\n        assert_eq!(\n            epoch1,\n            headers[\"x-epoch\"].to_str().unwrap().parse::<f64>().unwrap()\n        );\n        assert_eq!(res.text().await.unwrap(), \"hello world\");\n\n        // Get a secondary variant by changing Accept.\n        let res = send_vary_req_with_headers_with_dups(\n            url,\n            first_vary_header,\n            vec![\n                (\"SomeIgnoredVaryHeader\", \"image/webp\"),\n                (\"Accept\", \"image/jpeg\"),\n            ],\n            vec![(\"set-vary\", dup_vary_header)],\n        )\n        .await;\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        assert_eq!(headers[\"x-cache-status\"], \"miss\");\n        let epoch2 = headers[\"x-epoch\"].to_str().unwrap().parse::<f64>().unwrap();\n        assert_ne!(epoch1, epoch2);\n        assert_eq!(res.text().await.unwrap(), \"hello world\");\n    }\n\n    #[tokio::test]\n    async fn test_vary_purge() {\n        init();\n        let url = \"http://127.0.0.1:6148/unique/test_vary_purge/vary\";\n        let vary_field = \"Accept\";\n        let opt1 = \"image/png\";\n        let opt2 = \"image/jpeg\";\n\n        send_vary_req(url, vary_field, opt1).await;\n        let res = send_vary_req(url, vary_field, opt1).await;\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        assert_eq!(headers[\"x-cache-status\"], \"hit\");\n\n        send_vary_req(url, vary_field, opt2).await;\n        let res = send_vary_req(url, vary_field, opt2).await;\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        assert_eq!(headers[\"x-cache-status\"], \"hit\");\n\n        //both variances are cached\n\n        let res = reqwest::Client::builder()\n            .build()\n            .unwrap()\n            .request(reqwest::Method::from_bytes(b\"PURGE\").unwrap(), url)\n            .send()\n            .await\n            .unwrap();\n        assert_eq!(res.status(), StatusCode::OK);\n        assert_eq!(res.text().await.unwrap(), \"\");\n\n        //both should be miss\n\n        let res = send_vary_req(url, vary_field, opt1).await;\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        assert_eq!(headers[\"x-cache-status\"], \"miss\");\n\n        let res = send_vary_req(url, vary_field, opt2).await;\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        assert_eq!(headers[\"x-cache-status\"], \"miss\");\n    }\n\n    async fn send_max_file_size_req(\n        url: &str,\n        max_file_size_bytes: usize,\n        range: Option<(usize, usize)>,\n    ) -> reqwest::Response {\n        let mut req = reqwest::Client::new().get(url).header(\n            \"x-cache-max-file-size-bytes\",\n            max_file_size_bytes.to_string(),\n        );\n        if let Some(r) = range {\n            req = req.header(\"Range\", format!(\"bytes={}-{}\", r.0, r.1));\n        }\n        req.send().await.unwrap()\n    }\n\n    #[tokio::test]\n    async fn test_cache_max_file_size() {\n        init();\n        let url = \"http://127.0.0.1:6148/unique/test_cache_max_file_size_100/now\";\n\n        let res = send_max_file_size_req(url, 100, None).await;\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        assert_eq!(headers[\"x-cache-status\"], \"miss\");\n        assert_eq!(headers[\"content-length\"], \"11\");\n        assert_eq!(res.text().await.unwrap(), \"hello world\");\n\n        let res = send_max_file_size_req(url, 100, None).await;\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        assert_eq!(headers[\"x-cache-status\"], \"hit\");\n        assert_eq!(res.text().await.unwrap(), \"hello world\");\n\n        let url = \"http://127.0.0.1:6148/unique/test_cache_max_file_size_1/now\";\n        let res = send_max_file_size_req(url, 1, None).await;\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        assert_eq!(headers[\"x-cache-status\"], \"no-cache\");\n        assert_eq!(res.text().await.unwrap(), \"hello world\");\n\n        let res = send_max_file_size_req(url, 1, None).await;\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        assert_eq!(headers[\"x-cache-status\"], \"no-cache\");\n        assert_eq!(res.text().await.unwrap(), \"hello world\");\n\n        // became cacheable\n        let res = send_max_file_size_req(url, 100, None).await;\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        assert_eq!(headers[\"x-cache-status\"], \"miss\");\n        assert_eq!(res.text().await.unwrap(), \"hello world\");\n\n        let res = send_max_file_size_req(url, 100, None).await;\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        assert_eq!(headers[\"x-cache-status\"], \"hit\");\n        assert_eq!(res.text().await.unwrap(), \"hello world\");\n    }\n\n    #[tokio::test]\n    async fn test_cache_max_file_size_chunked() {\n        init();\n        let url = \"http://127.0.0.1:6148/unique/test_cache_max_file_size_chunked_100/test3\";\n\n        let res = send_max_file_size_req(url, 100, None).await;\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        assert_eq!(headers[\"x-cache-status\"], \"miss\");\n        assert_eq!(headers[\"transfer-encoding\"], \"chunked\");\n        assert_eq!(res.text().await.unwrap(), \"hello world\");\n\n        let res = send_max_file_size_req(url, 100, None).await;\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        assert_eq!(headers[\"x-cache-status\"], \"hit\");\n        assert_eq!(res.text().await.unwrap(), \"hello world\");\n\n        let url = \"http://127.0.0.1:6148/unique/test_cache_max_file_size_chunked_1/test3\";\n        let res = send_max_file_size_req(url, 1, None).await;\n        // TODO: this can currently break with 500 when body arrives alongside header\n        assert!(matches!(\n            res.status(),\n            StatusCode::INTERNAL_SERVER_ERROR | StatusCode::OK\n        ));\n        let headers = res.headers();\n        assert!(headers\n            .get(\"x-cache-status\")\n            .is_none_or(|s| s == \"no-cache\"));\n\n        let res = send_max_file_size_req(url, 1, None).await;\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        assert_eq!(headers[\"x-cache-status\"], \"no-cache\");\n        assert_eq!(res.text().await.unwrap(), \"hello world\");\n\n        // became cacheable\n        let res = send_max_file_size_req(url, 100, None).await;\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        // will get marked on the next request\n        assert_eq!(headers[\"x-cache-status\"], \"no-cache\");\n        assert_eq!(res.text().await.unwrap(), \"hello world\");\n\n        let res = send_max_file_size_req(url, 100, None).await;\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        assert_eq!(headers[\"x-cache-status\"], \"miss\");\n        assert_eq!(res.text().await.unwrap(), \"hello world\");\n\n        let res = send_max_file_size_req(url, 100, None).await;\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        assert_eq!(headers[\"x-cache-status\"], \"hit\");\n        assert_eq!(res.text().await.unwrap(), \"hello world\");\n    }\n\n    #[tokio::test]\n    async fn test_cache_max_file_size_range() {\n        init();\n        let url = \"http://127.0.0.1:6148/unique/test_cache_max_file_size_range_100/now\";\n\n        let res = send_max_file_size_req(url, 100, Some((1, 4))).await;\n        assert_eq!(res.status(), StatusCode::PARTIAL_CONTENT);\n        let headers = res.headers();\n        let epoch1 = headers[\"x-epoch\"].clone();\n        assert_eq!(headers[\"x-cache-status\"], \"miss\");\n        assert_eq!(res.text().await.unwrap(), \"ello\");\n\n        let res = send_max_file_size_req(url, 100, Some((1, 4))).await;\n        assert_eq!(res.status(), StatusCode::PARTIAL_CONTENT);\n        let headers = res.headers();\n        let epoch2 = headers[\"x-epoch\"].clone();\n        assert_eq!(headers[\"x-cache-status\"], \"hit\");\n        assert_eq!(res.text().await.unwrap(), \"ello\");\n        assert_eq!(epoch1, epoch2);\n\n        // disable downstream ranging on max file size exceeded\n        let url = \"http://127.0.0.1:6148/unique/test_cache_max_file_size_range_1/now\";\n        let res = send_max_file_size_req(url, 1, Some((1, 4))).await;\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        let epoch1 = headers[\"x-epoch\"].clone();\n        assert_eq!(headers[\"x-cache-status\"], \"no-cache\");\n        assert_eq!(res.text().await.unwrap(), \"hello world\");\n\n        // predicted uncacheable (note upstream endpoint doesn't support range)\n        let res = send_max_file_size_req(url, 1, Some((1, 4))).await;\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        let epoch2 = headers[\"x-epoch\"].clone();\n        assert_eq!(headers[\"x-cache-status\"], \"no-cache\");\n        assert_eq!(res.text().await.unwrap(), \"hello world\");\n        assert!(epoch1 != epoch2);\n    }\n\n    #[tokio::test]\n    async fn test_cache_h2_premature_end() {\n        init();\n        let url = \"http://127.0.0.1:6148/set_content_length/test_cache_h2_premature_end.txt\";\n        // try to fill cache\n        reqwest::Client::new()\n            .get(url)\n            .header(\"x-lock\", \"true\")\n            .header(\"x-h2\", \"true\")\n            .header(\"x-set-content-length\", \"13\") // 2 more than \"hello world\"\n            .send()\n            .await\n            .unwrap();\n        // h2 protocol error with content length mismatch\n\n        // did not get saved into cache, next request will be cache miss\n        let res = reqwest::Client::new()\n            .get(url)\n            .header(\"x-lock\", \"true\")\n            .header(\"x-h2\", \"true\")\n            .header(\"x-set-content-length\", \"11\")\n            .send()\n            .await\n            .unwrap();\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        assert_eq!(headers[\"x-cache-status\"], \"miss\");\n        assert_eq!(res.text().await.unwrap(), \"hello world\");\n    }\n\n    #[tokio::test]\n    async fn test_cache_bypass_downstream_range() {\n        init();\n        let test_url =\n            \"http://127.0.0.1:6148/unique/test_cache_bypass_downstream_range/cache_control/\";\n\n        let res = reqwest::Client::new()\n            .get(test_url)\n            .header(\"Range\", \"bytes=6-10\")\n            // We start this request as cacheable, and then this disables it.\n            // We would expect the range body filter to run since we have\n            // started to cache, and then disabled.\n            .header(\"set-cache-control\", \"private, max-age=0\")\n            .send()\n            .await\n            .unwrap();\n        assert_eq!(res.status(), StatusCode::PARTIAL_CONTENT);\n        let headers = res.headers();\n        assert_eq!(headers[\"x-cache-status\"], \"no-cache\");\n        assert_eq!(res.text().await.unwrap(), \"world\");\n\n        // We're starting as uncacheable, so proxy origin to downstream\n        // unchanged.\n        let res = reqwest::Client::new()\n            .get(test_url)\n            // We pass this up to the upstream, but it ignores it.\n            .header(\"Range\", \"bytes=0-4\")\n            .header(\"set-cache-control\", \"private, max-age=0\")\n            .send()\n            .await\n            .unwrap();\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        assert_eq!(headers[\"x-cache-status\"], \"no-cache\");\n        assert_eq!(res.text().await.unwrap(), \"hello world\");\n    }\n\n    #[tokio::test]\n    async fn test_downstream_head_miss_conn_close_h1() {\n        init();\n\n        let test_url =\n            \"http://127.0.0.1:6148/unique/test_cache_downstream_head_miss_conn_close/sleep/\";\n\n        let res = reqwest::Client::new()\n            .head(test_url)\n            .header(\"x-set-body-sleep\", \"1\")\n            .send()\n            .await\n            .unwrap();\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        assert_eq!(headers[\"x-cache-status\"], \"miss\");\n        assert_eq!(res.text().await.unwrap(), \"\");\n        // closed connection does not impact next cache fill\n\n        let res = reqwest::Client::new().get(test_url).send().await.unwrap();\n        assert_eq!(res.status(), 200);\n        let headers = res.headers();\n        assert_eq!(headers[\"x-cache-status\"], \"hit\");\n        assert_eq!(res.text().await.unwrap(), \"hello world\");\n    }\n\n    #[cfg(feature = \"any_tls\")]\n    #[tokio::test]\n    async fn test_downstream_head_miss_conn_close_h2() {\n        init();\n\n        let test_url =\n            \"https://127.0.0.1:6153/unique/test_cache_downstream_head_miss_conn_close/sleep/\";\n\n        let client = reqwest::Client::builder()\n            .danger_accept_invalid_certs(true)\n            .build()\n            .unwrap();\n\n        let res = client\n            .head(test_url)\n            .header(\"x-set-body-sleep\", \"0\")\n            .send()\n            .await\n            .unwrap();\n        assert_eq!(res.status(), StatusCode::OK);\n        let headers = res.headers();\n        assert_eq!(headers[\"x-cache-status\"], \"miss\");\n        assert_eq!(res.text().await.unwrap(), \"\");\n        // closed connection does not impact next cache fill\n\n        let client = reqwest::Client::builder()\n            .danger_accept_invalid_certs(true)\n            .build()\n            .unwrap();\n\n        let res = client.get(test_url).send().await.unwrap();\n        assert_eq!(res.status(), 200);\n        let headers = res.headers();\n        assert_eq!(headers[\"x-cache-status\"], \"hit\");\n        assert_eq!(res.text().await.unwrap(), \"hello world\");\n    }\n}\n"
  },
  {
    "path": "pingora-proxy/tests/utils/cert.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\nuse once_cell::sync::Lazy;\n#[cfg(feature = \"s2n\")]\nuse pingora_core::tls::load_pem_file;\n#[cfg(feature = \"rustls\")]\nuse pingora_core::tls::{load_pem_file_ca, load_pem_file_private_key};\n#[cfg(feature = \"openssl_derived\")]\nuse pingora_core::tls::{\n    pkey::{PKey, Private},\n    x509::X509,\n};\nuse std::fs;\n\n#[cfg(feature = \"openssl_derived\")]\nmod key_types {\n    use super::*;\n    pub type PrivateKeyType = PKey<Private>;\n    pub type CertType = X509;\n}\n\n#[cfg(feature = \"rustls\")]\nmod key_types {\n    use super::*;\n    pub type PrivateKeyType = Vec<u8>;\n    pub type CertType = Vec<u8>;\n}\n\n#[cfg(feature = \"s2n\")]\nmod key_types {\n    use super::*;\n    pub type PrivateKeyType = Vec<u8>;\n    pub type CertType = Vec<u8>;\n}\n\nuse key_types::*;\n\npub static INTERMEDIATE_CERT: Lazy<CertType> = Lazy::new(|| load_cert(\"keys/intermediate.crt\"));\npub static LEAF_CERT: Lazy<CertType> = Lazy::new(|| load_cert(\"keys/leaf.crt\"));\npub static LEAF2_CERT: Lazy<CertType> = Lazy::new(|| load_cert(\"keys/leaf2.crt\"));\npub static LEAF_KEY: Lazy<PrivateKeyType> = Lazy::new(|| load_key(\"keys/leaf.key\"));\npub static LEAF2_KEY: Lazy<PrivateKeyType> = Lazy::new(|| load_key(\"keys/leaf2.key\"));\npub static CURVE_521_TEST_KEY: Lazy<PrivateKeyType> =\n    Lazy::new(|| load_key(\"keys/curve_test.521.key.pem\"));\npub static CURVE_521_TEST_CERT: Lazy<CertType> = Lazy::new(|| load_cert(\"keys/curve_test.521.crt\"));\npub static CURVE_384_TEST_KEY: Lazy<PrivateKeyType> =\n    Lazy::new(|| load_key(\"keys/curve_test.384.key.pem\"));\npub static CURVE_384_TEST_CERT: Lazy<CertType> = Lazy::new(|| load_cert(\"keys/curve_test.384.crt\"));\n\n#[cfg(feature = \"openssl_derived\")]\nfn load_cert(path: &str) -> X509 {\n    let path = format!(\"{}/{path}\", super::conf_dir());\n    let cert_bytes = fs::read(path).unwrap();\n    X509::from_pem(&cert_bytes).unwrap()\n}\n#[cfg(feature = \"openssl_derived\")]\nfn load_key(path: &str) -> PKey<Private> {\n    let path = format!(\"{}/{path}\", super::conf_dir());\n    let key_bytes = fs::read(path).unwrap();\n    PKey::private_key_from_pem(&key_bytes).unwrap()\n}\n\n#[cfg(feature = \"rustls\")]\nfn load_cert(path: &str) -> Vec<u8> {\n    let path = format!(\"{}/{path}\", super::conf_dir());\n    load_pem_file_ca(&path).unwrap()\n}\n\n#[cfg(feature = \"rustls\")]\nfn load_key(path: &str) -> Vec<u8> {\n    let path = format!(\"{}/{path}\", super::conf_dir());\n    load_pem_file_private_key(&path).unwrap()\n}\n\n#[cfg(feature = \"s2n\")]\nfn load_cert(path: &str) -> Vec<u8> {\n    let path = format!(\"{}/{path}\", super::conf_dir());\n    load_pem_file(&path).unwrap()\n}\n\n#[cfg(feature = \"s2n\")]\nfn load_key(path: &str) -> Vec<u8> {\n    let path = format!(\"{}/{path}\", super::conf_dir());\n    load_pem_file(&path).unwrap()\n}\n"
  },
  {
    "path": "pingora-proxy/tests/utils/conf/keys/README.md",
    "content": "Some test certificates. The CA is specified in your package directory (grep for ca_file).\n\nSome handy commands:\n```\n# Describe a pkey\nopenssl [ec|rsa|...] -in key.pem -noout -text\n# Describe a cert\nopenssl x509 -in some_cert.crt -noout -text\n\n# Generate self-signed cert\nopenssl ecparam -genkey -name secp256r1 -noout -out test_key.pem\nopenssl req -new -x509 -key test_key.pem -out test.crt -days 3650 -sha256 -subj '/CN=openrusty.org'\n\n# Generate a cert signed by another\nopenssl ecparam -genkey -name secp256r1 -noout -out test_key.pem\nopenssl req -new -key test_key.pem -out test.csr\nopenssl x509 -req -in test.csr -CA server.crt -CAkey key.pem -CAcreateserial -CAserial test.srl -out test.crt -days 3650 -sha256\n\n# Generate leaf cert\nopenssl x509 -req -in leaf.csr -CA intermediate.crt -CAkey intermediate.key -out leaf.crt -days 3650 -sha256 -extfile v3.ext\n\n```\n\n```\nopenssl version\n# OpenSSL 3.1.1\necho '[v3_req]' > openssl.cnf\nopenssl req -config openssl.cnf -new -x509 -key key.pem -out server_rustls.crt -days 3650 -sha256 \\\n    -subj '/C=US/ST=CA/L=San Francisco/O=Cloudflare, Inc/CN=openrusty.org' \\\n    -addext \"subjectAltName=DNS:*.openrusty.org,DNS:openrusty.org,DNS:cat.com,DNS:dog.com\"\n```\n\n\n# Specific Examples\n\n## Updating `intermediate.crt` certificate\n\n```\n# Generate the key file\nopenssl genrsa -out intermediate.key 2048\n\n# Generate the signing request with the subject details filled in\nopenssl req -new -key intermediate.key -out intermediate.csr -subj '/C=US/ST=CA/O=Intermediate CA/CN=int.pingora.org'\n\n# Evaluate the signing request with the root certificate\nopenssl x509 -req -in intermediate.csr -CA root.crt -CAkey root.key -CAcreateserial -CAserial intermediate.srl -extfile intermediate.cnf -extensions v3_intermediate_ca -out intermediate.crt -days 3650 -sha256 \n```\n\n## Updating `leaf.crt` certificate using the intermediate cert\n\n```\n# Generate the key file\nopenssl genrsa -out leaf.key 2048\n\n# Generate the signing request with the subject details filled in\nopenssl req -new -key leaf.key -out leaf.csr -subj '/C=US/ST=CA/O=Internet Widgits Pty Ltd/CN=pingora.org'\n\n# Evaluate the signing request with the root certificate\nopenssl x509 -req -in leaf.csr -CA intermediate.crt -CAkey intermediate.key -CAcreateserial -CAserial leaf.srl -extfile leaf.cnf -extensions v3 -out leaf.crt -days 3650 -sha256 \n```\n"
  },
  {
    "path": "pingora-proxy/tests/utils/conf/keys/ca1.crt",
    "content": "-----BEGIN CERTIFICATE-----\nMIIFbTCCA1UCFDsRVhSk+Asz9Q9BwsvZucCbYA5/MA0GCSqGSIb3DQEBCwUAMHMx\nCzAJBgNVBAYTAlVTMRMwEQYDVQQIDApDYWxpZm9ybmlhMRYwFAYDVQQHDA1TYW4g\nRnJhbmNpc2NvMR4wHAYDVQQKDBVDbG91ZGZsYXJlIFRlc3QgU3VpdGUxFzAVBgNV\nBAMMDnNlbGZzaWduZWQuY29tMB4XDTIwMDkxODIwMzk1MloXDTMwMDkxNjIwMzk1\nMlowczELMAkGA1UEBhMCVVMxEzARBgNVBAgMCkNhbGlmb3JuaWExFjAUBgNVBAcM\nDVNhbiBGcmFuY2lzY28xHjAcBgNVBAoMFUNsb3VkZmxhcmUgVGVzdCBTdWl0ZTEX\nMBUGA1UEAwwOc2VsZnNpZ25lZC5jb20wggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAw\nggIKAoICAQCuFbjnE8gTFMrcCXmiP4t1wrK0uW5JSvWpxZAfTHroka/o8wBcKa1c\n7dXOGSEzKkTdsmrAkvi2KXMEAd08iwnY52xQ3vpaQDCiBhJhLUGaG2nJ5iH6A3CX\nVfsoHccFTp3N4/iiCjxyxnUoQZW1fuun5A9cow6F8xNa7EPtPMJsK7nUYDW2PLj4\n881aphUM483gS/Ph5IpaZs6bRP0HyscdSC8hoIZxkOfIgp8a9BvgnaK8cPhoNGFl\nHNu4hU+0cxjke/iz9iKRHtdcyuKnRMv8kt+acTpdgWl5E4nmvwXFloPeUuUAEgcc\nqcp9Uai2dp9XKfxAGW2wEQPpZseDH7mZ7+NwqxJ2z4R55fdIn8jmALJdz+npvpRr\nQHHc6k9jv0iYv9XwZOqT1crlzwcCo3x8A7oD+sJrat5oY1zBXjNzLpb9DKyVQ1em\nHo/7VrLFtK+rJzI/b7D0r6GKk/h3SeqxmgN22fFPcbEM2eUIibUvmCB4OLooWkBs\neSeDr5wMZ7u9ExljGLywKHnOQQ7dlVUWeN5cncv9yU05fWE/whPEOri1ksyNdYr6\nkjIY1NYKmXfRaKaR9/JCVkhZj0H8VI6QpkqVHKgI5UMeE5dHMYbxJv0lmG+w6XN1\nZew7DZRTidlBa6COxgCeQydxRTORCCPYQVYAGY5XiYtmWLGmsQjC1QIDAQABMA0G\nCSqGSIb3DQEBCwUAA4ICAQAgGv+gvw5X9ftkGu/FEFK15dLHlFZ25tKHJw3LhJEf\nxlDOCFI/zR+h2PFdVzks14LLrf4sSkRfJVkk2Qe5uRhHLcgnPIkCkJpGlpFMx2+V\nO6azhJlnLEYeVXuzNiQHC+9LJH8i3NK37O8Z1z2EGsAz9kR09OBEvgDjSXFxCN0J\nKLAMe4wfAhjUUt9/0bm9u7FYWyj0D5dUVeAul9X3Vo1HfffNovq2cuUlL1AG5Ku+\nnPkxGckBo/Lc7jZQRcoZ2+mtvsfyMH5l9OW6JRrnC/Rf5P9bEjUcAskMh5WRdHSL\nj98oCkosxg2ndTXke091lToqr7sZ1kiGA+Bj4cPlVXckQn3WU7GiUSSRqotZtn8g\nEMT2iqHH3/iJOgtDe8XPWdBYNDeDFRVNpOtgCuYLXdz/Vli0Cecm3escbW/+GZ9P\nvgZoNUej8/WTWHNy732N1cHvSbT3kLN6uONP4wNelh+UnfmiG10O54x7iaM3grt9\nYvQ1I1G60NCj1tF9KvrCYCK/wnXnTWhlNZ4y+XbILFqE+k8zqiNzGZV9a8FAzht2\nAPsm2JzzZz6Ph6Zw8fVOS/LX7WgF/kNe5nIzVLqyFXtFxgomXaoxbADUTe16TVb3\n6sV8p7nlq2r7Dr0+uROm7ZEg1F23SiieDoRvw5fUbRhZCU93fv7Nt7hWlKP+UqJj\nZg==\n-----END CERTIFICATE-----\n"
  },
  {
    "path": "pingora-proxy/tests/utils/conf/keys/ca1.key.pem",
    "content": "-----BEGIN RSA PRIVATE KEY-----\nMIIJKAIBAAKCAgEArhW45xPIExTK3Al5oj+LdcKytLluSUr1qcWQH0x66JGv6PMA\nXCmtXO3VzhkhMypE3bJqwJL4tilzBAHdPIsJ2OdsUN76WkAwogYSYS1BmhtpyeYh\n+gNwl1X7KB3HBU6dzeP4ogo8csZ1KEGVtX7rp+QPXKMOhfMTWuxD7TzCbCu51GA1\ntjy4+PPNWqYVDOPN4Evz4eSKWmbOm0T9B8rHHUgvIaCGcZDnyIKfGvQb4J2ivHD4\naDRhZRzbuIVPtHMY5Hv4s/YikR7XXMrip0TL/JLfmnE6XYFpeROJ5r8FxZaD3lLl\nABIHHKnKfVGotnafVyn8QBltsBED6WbHgx+5me/jcKsSds+EeeX3SJ/I5gCyXc/p\n6b6Ua0Bx3OpPY79ImL/V8GTqk9XK5c8HAqN8fAO6A/rCa2reaGNcwV4zcy6W/Qys\nlUNXph6P+1ayxbSvqycyP2+w9K+hipP4d0nqsZoDdtnxT3GxDNnlCIm1L5ggeDi6\nKFpAbHkng6+cDGe7vRMZYxi8sCh5zkEO3ZVVFnjeXJ3L/clNOX1hP8ITxDq4tZLM\njXWK+pIyGNTWCpl30WimkffyQlZIWY9B/FSOkKZKlRyoCOVDHhOXRzGG8Sb9JZhv\nsOlzdWXsOw2UU4nZQWugjsYAnkMncUUzkQgj2EFWABmOV4mLZlixprEIwtUCAwEA\nAQKCAgBP/nVX4dQnSH+rOsNk1fRcqZn6x9aw4TwfxkPizf8QjZma3scEkrYyJKwB\np7SE0WCRyyGY2jBlbIiIh97EqlNdE4LHap76B9MRMN8TPnuNuBkViKWGQDxlnkHp\n/jzs6GJFMQOYWkHKr/04AWMs4mShYn/YnqjWzorPVhAknK3ujO04dPlZg2+wHj/3\n7qdvo+J/tgccfytAPUulN79Z7Ekw4HGf7ya4WtDXZ4Z7GT8SKP2VwAe1wpQapXcl\nxESK8/S1UW5IK8tYiiaGYkhieo+NwWP0kSEzxHrWAy90E8UwNWjlKYxHSwFvn2oH\nyhVPuxSfNhDO16B6rmbwwqTdUR+0pepF9IcgWuGO/AAMPlo6tKKqo7oW8xUqX0EW\nvSCdISLlOITe2GBFv0q1xcUG9xZM5/Hde4NPU6OpghFcM/Okl3MoGqvqH4Fcd2Lm\nHsjHxE6/8pDvxy8wGMeHEYTcDnKdTGPQgyEHHTZBsoHOzrM7CXGgpGIj9DPxrJO+\nVZFHqoILRbhiU3LTnyb5J8X8zyPv064LOoZOu2JoY99E2j1PtI4ym1fAzhd5ScU7\nX2CJTXAA57e0ezZCuPh/isgHmhx3bFHUvluWPKyspchLy/Pk28382jgnM+/vdbZh\nwObGpeLpIEylxMmMROxZSDiDFhwG/rrp08vmhJRjgCb6XRAiZQKCAQEA1dnTbqve\naMioUOt70U6kuokJGpibmp5CXD+yGev4QZiVuoZvURfqqwRoMyJ2jaALAYRJpRMc\ntbBvi3iq+uhB4FFiSCegK+F3qYPpgPvC2kHDrqgm4Fdjhz0/CfkkihzyVae5BHU9\nnm9xS39vmHKtPdM4Yt8n/vGXZy06pKRo2gxA4K5UswtJ3GGgKY+/dgRgXGS7eIaw\n2b1uLvIZ8p2XGzMbjAtaTEykAQXMX7mPanpizT8LguvxCAFo2QyzCMJyuUii8pQS\nH/ewKGVd3zZVN3KgWnGWoYpnRaY/eG6O60APV625yRgI0k4CZucWK8wuNU4TGpy7\nYCnJSX3q/nIh9wKCAQEA0GVwvHjqWTOCpeTw5+QJq/0dvTVmaJzUe+EfBYpgaqd3\nK+Lcj3TuNr+/v8sJ6ZhvflBxLI9Yk9kmfHMVsCrTXPr2mkmZMeucbM6Lot9tcYAN\nFX+LKFIz9uDMXhMZfnycEurEdwlo1A1c4dpYEaOJ+OCmzglg7Bfxq7ol32MlVg8e\n06VyjfFVR2fNzlRUFX/DZrI8mjgsVone/eJNGLYPUhXMZ905vfQFefP9DijTtecZ\nAcPkhMMCXaldtuZ9WE9SRnV0HRpggDFdA+7AJnqp9umc3S1yv1YQvSFomAH+Aszs\nLKuwS4VPwZWNiMHqRlQrZ6lKa+rMWSowHiJCgIpOkwKCAQEAyiSeLIX/tXK/T8ZY\ngxBgvAae+Wn55Fzmg4aeFsysHW1bUzaScMg3xbJjwLo58EOxQ5zFdGmtgL0no2HL\n1WLIKn8jdOsoB3KYBz+u8IKKvH7ftvAx12wjo4msVgQQmxEjrP3e8SzVszbKlEAA\nv8zen4tSSHuCtgWuRRRG06yphDuC9B815wyro8sQd1ju9WLLp2p8n0BKWXgrd+rX\nxjNay5Yy2t08XNUxTdoqRu4Dd/X6AOMwQXA/pX6XmlvbvFL52NSlWsHGpDsgY/71\njfIw+Tm8A+JNLaPDXN36Lx/qrssd9ZY9AK5cYFbnBFg55+qYX0DO5B/1KsA1Cegh\nwqUmHwKCAQBw/r/NAccXzM1HREa3hbcU0W7hm+XGTVsNPHiEmY5D5j/AxQaQpndP\nqlK/HMloJqY1mEp1PdhqejDbA8+7sMzgOpeh+swc/ELZ4HhoPLtr8mGlyX1bxI62\nixdk3vhQ1CIQQ8l5PdngOMqnD6v3DHSQRMdNKlqqSSVZ1toYMPsamaI+YhQmELgL\nuqYl/SWGbrs1oOkpOdIYrjMB+EWTY4wVFwq5OoPHkluxz3Djz5FTrVWq1lu+/Ln4\nrQ/KT1mhm4jh+WeXLCks+RcVPcxkUNh9sBfE+ZKhWnpDAq1i1pmzTQe2BPXXTRZ8\nwal3gKWVsqfCUlGvCCX7JtvmSu9CITwPAoIBAEQO6PQh3nD/tJSFZxgtPVp7r3Px\n+QEnE68Y0B0veq9g5SBovg4KADTcHbIbRymOBw+9skB65pxdoV3EFGmEXpMm5+5b\nHC/DTXf2hEuKb49VO52NbbthiZg+xsnitEv4ZBfSVBRw+nL3Dx5c30M9wG/3OdGX\nOWPYFoIJZDlyy3ynZtiGrjHgNqi/coHdsYVLfMkc+/hidApzhoApDkFGusVB6GHB\nfTSeyuGfh39120LVnhFjDr+SpfyIXNJIiCwizLJtc1WliTtQzd/Fh1M62qO6ye4/\n3M24xoaVCDgzNrSibELkiLTmqEA4cZwtN5BqhfnQa+Prujd5ElmABZSqDz8=\n-----END RSA PRIVATE KEY-----\n"
  },
  {
    "path": "pingora-proxy/tests/utils/conf/keys/ca2.crt",
    "content": "-----BEGIN CERTIFICATE-----\nMIIFbTCCA1UCFDsRVhSk+Asz9Q9BwsvZucCbYA5/MA0GCSqGSIb3DQEBCwUAMHMx\nCzAJBgNVBAYTAlVTMRMwEQYDVQQIDApDYWxpZm9ybmlhMRYwFAYDVQQHDA1TYW4g\nRnJhbmNpc2NvMR4wHAYDVQQKDBVDbG91ZGZsYXJlIFRlc3QgU3VpdGUxFzAVBgNV\nBAMMDnNlbGZzaWduZWQuY29tMB4XDTIwMDkxODIwMzk1MloXDTMwMDkxNjIwMzk1\nMlowczELMAkGA1UEBhMCVVMxEzARBgNVBAgMCkNhbGlmb3JuaWExFjAUBgNVBAcM\nDVNhbiBGcmFuY2lzY28xHjAcBgNVBAoMFUNsb3VkZmxhcmUgVGVzdCBTdWl0ZTEX\nMBUGA1UEAwwOc2VsZnNpZ25lZC5jb20wggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAw\nggIKAoICAQCuFbjnE8gTFMrcCXmiP4t1wrK0uW5JSvWpxZAfTHroka/o8wBcKa1c\n7dXOGSEzKkTdsmrAkvi2KXMEAd08iwnY52xQ3vpaQDCiBhJhLUGaG2nJ5iH6A3CX\nVfsoHccFTp3N4/iiCjxyxnUoQZW1fuun5A9cow6F8xNa7EPtPMJsK7nUYDW2PLj4\n881aphUM483gS/Ph5IpaZs6bRP0HyscdSC8hoIZxkOfIgp8a9BvgnaK8cPhoNGFl\nHNu4hU+0cxjke/iz9iKRHtdcyuKnRMv8kt+acTpdgWl5E4nmvwXFloPeUuUAEgcc\nqcp9Uai2dp9XKfxAGW2wEQPpZseDH7mZ7+NwqxJ2z4R55fdIn8jmALJdz+npvpRr\nQHHc6k9jv0iYv9XwZOqT1crlzwcCo3x8A7oD+sJrat5oY1zBXjNzLpb9DKyVQ1em\nHo/7VrLFtK+rJzI/b7D0r6GKk/h3SeqxmgN22fFPcbEM2eUIibUvmCB4OLooWkBs\neSeDr5wMZ7u9ExljGLywKHnOQQ7dlVUWeN5cncv9yU05fWE/whPEOri1ksyNdYr6\nkjIY1NYKmXfRaKaR9/JCVkhZj0H8VI6QpkqVHKgI5UMeE5dHMYbxJv0lmG+w6XN1\nZew7DZRTidlBa6COxgCeQydxRTORCCPYQVYAGY5XiYtmWLGmsQjC1QIDAQABMA0G\nCSqGSIb3DQEBCwUAA4ICAQAgGv+gvw5X9ftkGu/FEFK15dLHlFZ25tKHJw3LhJEf\nxlDOCFI/zR+h2PFdVzks14LLrf4sSkRfJVkk2Qe5uRhHLcgnPIkCkJpGlpFMx2+V\nO6azhJlnLEYeVXuzNiQHC+9LJH8i3NK37O8Z1z2EGsAz9kR09OBEvgDjSXFxCN0J\nKLAMe4wfAhjUUt9/0bm9u7FYWyj0D5dUVeAul9X3Vo1HfffNovq2cuUlL1AG5Ku+\nnPkxGckBo/Lc7jZQRcoZ2+mtvsfyMH5l9OW6JRrnC/Rf5P9bEjUcAskMh5WRdHSL\nj98oCkosxg2ndTXke091lToqr7sZ1kiGA+Bj4cPlVXckQn3WU7GiUSSRqotZtn8g\nEMT2iqHH3/iJOgtDe8XPWdBYNDeDFRVNpOtgCuYLXdz/Vli0Cecm3escbW/+GZ9P\nvgZoNUej8/WTWHNy732N1cHvSbT3kLN6uONP4wNelh+UnfmiG10O54x7iaM3grt9\nYvQ1I1G60NCj1tF9KvrCYCK/wnXnTWhlNZ4y+XbILFqE+k8zqiNzGZV9a8FAzht2\nAPsm2JzzZz6Ph6Zw8fVOS/LX7WgF/kNe5nIzVLqyFXtFxgomXaoxbADUTe16TVb3\n6sV8p7nlq2r7Dr0+uROm7ZEg1F23SiieDoRvw5fUbRhZCU93fv7Nt7hWlKP+UqJj\nZg==\n-----END CERTIFICATE-----\n"
  },
  {
    "path": "pingora-proxy/tests/utils/conf/keys/ca_chain.cert",
    "content": "-----BEGIN CERTIFICATE-----\nMIIEjjCCAnagAwIBAgIUHIB/tqjZJaKIgeWwvXRt03C0yIMwDQYJKoZIhvcNAQEL\nBQAwXzELMAkGA1UEBhMCVVMxCzAJBgNVBAgMAkNBMRYwFAYDVQQHDA1TYW4gRnJh\nbmNpc2NvMRAwDgYDVQQKDAdSb290IENBMRkwFwYDVQQDDBByb290LnBpbmdvcmEu\nb3JnMB4XDTIyMTExMDE5MzI0M1oXDTI1MDgwNjE5MzI0M1owTjELMAkGA1UEBhMC\nVVMxCzAJBgNVBAgMAkNBMRgwFgYDVQQKDA9JbnRlcm1lZGlhdGUgQ0ExGDAWBgNV\nBAMMD2ludC5waW5nb3JhLm9yZzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoC\nggEBAL4klMT1Bc4vYWN7zF+x7x34s3L51Sve3AVydGGtzej2hC3m4CictVfKfkC6\njMNRo3mpUsnAJSbyRh91fec8nnOT8MEYnmm05Lbf5DG4RULrKSg52zge4SFTLO2n\n2eCa4SYwRpj+MQmFrCQ++s9gJ/5weN95z23XAS1EL2GK50Z/fKQfRCo+aZTRB6dU\nKK2cUwuDAHTkVSePVAX8KGcZu2Qm/jTBlcDIfn7OmTu2g/n5YSRJg3MWKeJlAbVo\nVNxmaRYQOs2X7y4WwcSAfEncyVXRzqFxEfSDnq2A2+pp/sKoCjTgE6n94SzyqyFm\nyJ8FmvV79qCDHSaeIhR5qQEIlO8CAwEAAaNTMFEwHQYDVR0OBBYEFP5ivTJr/S6Z\nVpOI4+JykGPID8s3MB8GA1UdIwQYMBaAFJ5hR0odQYOtYsY3P18WIC2byI1oMA8G\nA1UdEwEB/wQFMAMBAf8wDQYJKoZIhvcNAQELBQADggIBAM337XP2Hm7LE3KLW+nn\nkhyj82ahj2k2+H/OdwGqzfqHCObTP+ydNJhOQVD+r255qQC9eAvd6ivF/h1tJOvv\nEd8vQMfLCO9VDFy6KCmlZQV6djRU1QXJIR/jf7TNqrFOcuKPGv5Vs6JwDdaHc0ae\nug7CGppnu5cxf/04sa7pWOdCFbhDRtfooo9fgGN2jcTFqfGyzocBwx7dgqEmZkae\nyJAH0x4ldpKM9aO44h0Uy36c5RaWmdyFIh88QW62NoHamfwZoaVyycn82wcP4fFG\nPRHm/AaDkYFGiQy22y7DD+MeZNUgCcAJpDYxfe87Cm4dw9NweMF6Jpo/8Ib1oLPq\nE3miiFjWQwpMhxSQxpjqR92FPs9+/ktvYqbbMlyu/tju0rK17DXUi1zSIHoydPt0\nymwWMxg7Jxpmg0x+eyWr5CP/ULM+F2Tk9W7x0B5DnpDJeCk+1ydUhII9AnTOCUWs\n0VRlqTgFKahkHfiLBjPaLCgA0D3dz06EfEq5tmC8t0MDAqw9M4bDdow29K0aN6K8\nGax7S5EK9aK09+HJ+7T5uxkUC+iIzfk53RhAfQiXdyKPpkbndRP67OiaAwk+hIXm\nU1d1GsC854KYQs2GtHHvBcTGEADfU36TF/w2oJYQIrBjd23ZCf9jFK/WQ5GBFitT\nljoURxQQQy3LGjcH8W18JdRE\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIFnzCCA4egAwIBAgIUE5kg5Z26V4swShJoSwfNVsJkHbYwDQYJKoZIhvcNAQEL\nBQAwXzELMAkGA1UEBhMCVVMxCzAJBgNVBAgMAkNBMRYwFAYDVQQHDA1TYW4gRnJh\nbmNpc2NvMRAwDgYDVQQKDAdSb290IENBMRkwFwYDVQQDDBByb290LnBpbmdvcmEu\nb3JnMB4XDTIyMTExMDE5MjY1MFoXDTQyMTExMDE5MjY1MFowXzELMAkGA1UEBhMC\nVVMxCzAJBgNVBAgMAkNBMRYwFAYDVQQHDA1TYW4gRnJhbmNpc2NvMRAwDgYDVQQK\nDAdSb290IENBMRkwFwYDVQQDDBByb290LnBpbmdvcmEub3JnMIICIjANBgkqhkiG\n9w0BAQEFAAOCAg8AMIICCgKCAgEA4s1XxwZruaRwuDX1IkM2oxdSdjg7FeUp8lsN\nUix4NdXz8IoQWRzCfFuRBKFHptahutSO6Bbewm9XmU2hHG7aoCqaZqEVQ/3KRLZ4\nmzaNBCzDNgPTmDkz/DZKzOVuyVvbmTOsLn53yxKnFP9MEDIEemqGiM80MmFfCm/o\n0vLkjwkRpreMsWPUhrq3igTWRctUYMJAeDsEaaXB1k5ovWICrEylMzslgSNfoBed\nNmBpurz+yQddKNMTb/SLYxa7B1uZKDRSIXwwOZPdBDyUdlStUPodNG/OzprN+bRC\noFRB9EFG1m5oPJXQIalePj0dwhXl/bkV4uRxCSZmBZK3fbtLMF+Wkg2voTrn51Yv\nlKkzUQoEX6WWtUameZZbUB8TbW2lmANuvGBmvBbj3+4ztmtJPXfJBkckCeUC6bwC\n4CKrgB587ElY357Vqv/HmRRC9kxdzpOS9s5CtcqJ3Dg1TmLajyRQkf8wMqk0fhh7\nV+VrPXB030MGABXh5+B2HOsF307vF030v7z+Xp5VRLGBqmDwK0Reo2h8cg9PkMDS\n5Qc2zOJVslkJ+QYdkea1ajVpCsFbaC1JPmRWihTllboUqsk9oSS3jcIZ8vW3QKMg\nZbKtVbtVHr3mNGWuVs96iDN5Us3SJ6KGS8sanrAYAAB/NKd1Wl3I0aVtcb6eOONd\nedf9+b0CAwEAAaNTMFEwHQYDVR0OBBYEFJ5hR0odQYOtYsY3P18WIC2byI1oMB8G\nA1UdIwQYMBaAFJ5hR0odQYOtYsY3P18WIC2byI1oMA8GA1UdEwEB/wQFMAMBAf8w\nDQYJKoZIhvcNAQELBQADggIBAIrpAsrPre3R4RY0JmnvomgH+tCSMHb6dW52YrEl\nJkEG4cVc5MKs5QfPp8l2d1DngqiOUnOf0MWwWNDidHQZKrWs59j67L8qKN91VQKe\ncSNEX3iMFvE59Hr0Ner6Kr09wZLHVVNGcy0FdhWpJdDUGDoQjfL7n7usJyCUqWSq\n/pa1I9Is3ZfeQ5f7Ztrdz35vVPj+0BlHXbZM5AZi8Dwf3vXFBlPty3fITpE65cty\ncYnbpGto+wDoZj9fkKImjK21QsJdmHwaWRgmXX3WbdFBAbScTjDOc5Mls2VY8rSh\n+xLI1KMB0FHSJqrGoFN3uE+G1vJX/hgn98KZKob23yJr2TWr9LHI56sMfN5xdd5A\niOHxYODSrIAi1k+bSlDz6WfEtufoqwBwHiog4nFOXrlHpGO6eUB1QjaQJZwKn2zE\n3BjqJOoqbuBMg5XZRjihHcVVuZdU39/zQDwqliNpx3km4FzOiEoBABGzLP+Qt0Ch\ncJFS1Yc8ffv616yP4A9qkyogk9YBBvNbDLB7WV8h8p1s4JP3f5aDUlxtAD+E+3aJ\n8mrb3P7/0A2QyxlgX4qQOdj++b7GzXDxxLgOimJ4pLo0fdY8KWMeHvZPiMryHkMx\n3GSZCHeleSVBCPB2pPCzUqkkKADbjBX3SYJsAMF9uXQAR4U7wojjvAmbt6vJEh6j\nTEUG\n-----END CERTIFICATE-----\n"
  },
  {
    "path": "pingora-proxy/tests/utils/conf/keys/ca_chain.srl",
    "content": "764CA822243398735D12CB8F1295AEDF38869BA7\n"
  },
  {
    "path": "pingora-proxy/tests/utils/conf/keys/cert_chain.crt",
    "content": "-----BEGIN CERTIFICATE-----\nMIICtzCCAl2gAwIBAgIUC8kzFXZNRqjR158InTieHg1VrWowCgYIKoZIzj0EAwIw\ngY4xCzAJBgNVBAYTAlVTMRMwEQYDVQQIEwpDYWxpZm9ybmlhMRYwFAYDVQQHEw1T\nYW4gRnJhbmNpc2NvMRgwFgYDVQQKEw9IYXBweUNlcnQsIEluYy4xHzAdBgNVBAsT\nFkhhcHB5Q2VydCBJbnRlcm1lZGlhdGUxFzAVBgNVBAMTDihkZXYgdXNlIG9ubHkp\nMCAXDTE5MTIwOTE5NDgwMFoYDzIxMTkxMTE1MTk0ODAwWjCBgTELMAkGA1UEBhMC\nVVMxEzARBgNVBAgTCkNhbGlmb3JuaWExFjAUBgNVBAcTDVNhbiBGcmFuY2lzY28x\nGTAXBgNVBAoTEERFUiBpcyBGdW4sIEluYy4xETAPBgNVBAsTCEVuY29kaW5nMRcw\nFQYDVQQDEw4oZGV2IHVzZSBvbmx5KTBZMBMGByqGSM49AgEGCCqGSM49AwEHA0IA\nBJSBMLYEVPgjmd2vWgMpN9LupZa56T7Ds1+wAlyMphLDN56PWuphsrNsEwiIIeNv\nMtRTPRuoiBkfvMiWON6nkGWjgaEwgZ4wDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\nMAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHQYDVR0OBBYEFOYNuOCrYKnTFIEV\nck5845y/yZHkMB8GA1UdIwQYMBaAFFZRXwepqUwm9Kh+repV7LkBDnEHMCkGA1Ud\nEQQiMCCCCWRlcmlzLmZ1boITd2VsbGtub3duLmRlcmlzLmZ1bjAKBggqhkjOPQQD\nAgNIADBFAiEA9XAQ1Xi4Lav8LKzXZMSOHHj21ycqf3grnUfKJ6iwRvkCIDevfipo\nqIuR/Dnt1bBoXxFKv0w/LpH/89jIohUQwVSc\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIDwzCCAqugAwIBAgIJAN0mCzwZkgZKMA0GCSqGSIb3DQEBCwUAMHgxCzAJBgNV\nBAYTAlVTMRMwEQYDVQQIDApDYWxpZm9ybmlhMRYwFAYDVQQHDA1TYW4gRnJhbmNp\nc2NvMRgwFgYDVQQKDA9DbG91ZEZsYXJlLCBJbmMxDDAKBgNVBAsMA1ImRDEUMBIG\nA1UEAwwLZXhhbXBsZS5jb20wHhcNMTYwNjMwMTY1NTM5WhcNMzYwNjI1MTY1NTM5\nWjB4MQswCQYDVQQGEwJVUzETMBEGA1UECAwKQ2FsaWZvcm5pYTEWMBQGA1UEBwwN\nU2FuIEZyYW5jaXNjbzEYMBYGA1UECgwPQ2xvdWRGbGFyZSwgSW5jMQwwCgYDVQQL\nDANSJkQxFDASBgNVBAMMC2V4YW1wbGUuY29tMIIBIjANBgkqhkiG9w0BAQEFAAOC\nAQ8AMIIBCgKCAQEA7y+v+9Eh2LjFoZbUetrJc+IVPb92PBNNY5AM+Nxukzj/9hth\ntu7UPFnO+USrh+nFtR/rFfC6UwUqCtPaQ4EkSVJslR8f34GoOlc8zz7+dq9sGGu0\nhUPCLiptfBdIu73l0XqMd+xdGprl8hMdpH0CyKhAqTpv/00cmFobFwm1Fbf146hb\nYAhyP6rIzDlrhvYFe3sFwAIjXQ0qyN+ffm/Ot1iFdYER24sl63XfwBPS97DwO70p\n4jtbea8zlN58CFmTTK899J1f4MGbzvMyttdHG+WjhLNplB7fhtBdiHes2EdQws2S\nTKbK5D/69OYXSVCwimcOnlklcJ1NpQJFFaWeKQIDAQABo1AwTjAdBgNVHQ4EFgQU\ncu65A8EdrKWjFy9PZSRvSu8+4G0wHwYDVR0jBBgwFoAUcu65A8EdrKWjFy9PZSRv\nSu8+4G0wDAYDVR0TBAUwAwEB/zANBgkqhkiG9w0BAQsFAAOCAQEAl3lAgKb+3NQ/\n+a+ooaML1Ndmh7h+UWl4bXx1TXwaLAi0iIujrAa3AM86vqKxFeCwZC9bAPEyGQrH\nAF8JQbWAa2SckSDPSxM1ETV7EtJS4plaSfWxzX/m8jtd7D5RbzyE/qUH5JsXvCta\nrKOMJPNvSfTuxQMX/Qyp0cHZUr/3ylUhdLWYsNwTAlQgx0OK8w+zWx6ESCM52Cz4\nGqjpgcq6qylE2RoNmY0L+xb1B0YS+fslcjSXJZ/Z1j9mVrUM4wuekgcIxJfUrfhv\n/957d4I04iMp6F/XgrrKUewCGiifcDi87nwoqHJwSIWG33LTb4e8mSe4Y83Fh8L2\nKWQDqcnYug==\n-----END CERTIFICATE-----"
  },
  {
    "path": "pingora-proxy/tests/utils/conf/keys/curve_test.384.crt",
    "content": "-----BEGIN CERTIFICATE-----\nMIIBnDCCASOgAwIBAgIJAJ8dDVMCYWE3MAoGCCqGSM49BAMDMBsxGTAXBgNVBAMM\nEG9wZW5ydXN0eTM4NC5vcmcwHhcNMjMwNDA3MTY0NzEyWhcNMzMwNDA0MTY0NzEy\nWjAbMRkwFwYDVQQDDBBvcGVucnVzdHkzODQub3JnMHYwEAYHKoZIzj0CAQYFK4EE\nACIDYgAENKtL8ciBDxA9G2auTbtbteNu8DI7gp0039+J6Z29laQpHLMw8MH7Wegx\nHTv9RTXcf1sTCBloZh8qTvZTDh1yi7kjhZ2yLdVEVoakC5HBKvWzo1ewjSkOfBX7\nLF4p/8ULozMwMTAvBgNVHREEKDAmghIqLm9wZW5ydXN0eTM4NC5vcmeCEG9wZW5y\ndXN0eTM4NC5vcmcwCgYIKoZIzj0EAwMDZwAwZAIwL8ad/dyrC62bFC7gGZkRzaTm\nr2XlaMk6LB02IbVJgQytu+p50pnAgELVXISLP8LIAjBAjQ71pDbCjfg8Ts6iOnWH\np4R+Z2BjbTZu+Kmn1x8nyo2OJcchRYTRAKMS7YWstIk=\n-----END CERTIFICATE-----\n"
  },
  {
    "path": "pingora-proxy/tests/utils/conf/keys/curve_test.384.key.pem",
    "content": "-----BEGIN EC PRIVATE KEY-----\nMIGkAgEBBDCWPID9PlALCL+dNPdlEBw2fP4cU56akYDeV08fpY+DkhaJicPxAilY\n2T68Epv7nh6gBwYFK4EEACKhZANiAAQ0q0vxyIEPED0bZq5Nu1u1427wMjuCnTTf\n34npnb2VpCkcszDwwftZ6DEdO/1FNdx/WxMIGWhmHypO9lMOHXKLuSOFnbIt1URW\nhqQLkcEq9bOjV7CNKQ58FfssXin/xQs=\n-----END EC PRIVATE KEY-----\n"
  },
  {
    "path": "pingora-proxy/tests/utils/conf/keys/curve_test.521.crt",
    "content": "-----BEGIN CERTIFICATE-----\nMIIB5zCCAUmgAwIBAgIJALxqm9BrQU12MAoGCCqGSM49BAMEMBsxGTAXBgNVBAMM\nEG9wZW5ydXN0eTUyMS5vcmcwHhcNMjMwNDA3MTY0NjU4WhcNMzMwNDA0MTY0NjU4\nWjAbMRkwFwYDVQQDDBBvcGVucnVzdHk1MjEub3JnMIGbMBAGByqGSM49AgEGBSuB\nBAAjA4GGAAQA9LXDr66Cx/DZYnSacGu0FxlSx/e7xTm49g2QGU7TkO8TEyaOkErl\nIaqJE7YxQp+CUMfelVVkUJmVlJ4Fkrl3nR4A3YLDjEYihXnuLZajbwkjC7vzKO8A\nO2ln8R5JSzClUoTu7s2nok7tw/6dP4i08YPk4Pkxm5NHIok0uFmoaJpdkq6jMzAx\nMC8GA1UdEQQoMCaCEioub3BlbnJ1c3R5NTIxLm9yZ4IQb3BlbnJ1c3R5NTIxLm9y\nZzAKBggqhkjOPQQDBAOBiwAwgYcCQgCdVxTjVAPCIouh1HH4haJDpS1/g30jcTj6\nFGvyxofIX4Q6fO3Ig8DlJa+SrDq2f75/f8RSC71NB6peNjP8IARCOAJBKEMcXjK5\nbtvZxg+puzyxuMNRtUUk/Re/pzzLJbi7o6MWVNgLQJ3d9kUVHzbQEXNiUe82vbYK\nuairSMDS6Dl1j/A=\n-----END CERTIFICATE-----\n"
  },
  {
    "path": "pingora-proxy/tests/utils/conf/keys/curve_test.521.key.pem",
    "content": "-----BEGIN EC PRIVATE KEY-----\nMIHbAgEBBEFiMUgbEqjcf3K4Ba+CFUv20+ryJq9REjWUkoi9AgkpGuEAqLQza3CM\nkSGSiPdm9gWmpeLlCExPVJRbcTmAhoZUcKAHBgUrgQQAI6GBiQOBhgAEAPS1w6+u\ngsfw2WJ0mnBrtBcZUsf3u8U5uPYNkBlO05DvExMmjpBK5SGqiRO2MUKfglDH3pVV\nZFCZlZSeBZK5d50eAN2Cw4xGIoV57i2Wo28JIwu78yjvADtpZ/EeSUswpVKE7u7N\np6JO7cP+nT+ItPGD5OD5MZuTRyKJNLhZqGiaXZKu\n-----END EC PRIVATE KEY-----\n"
  },
  {
    "path": "pingora-proxy/tests/utils/conf/keys/ex1.crt",
    "content": "-----BEGIN CERTIFICATE-----\nMIICtzCCAl2gAwIBAgIUC8kzFXZNRqjR158InTieHg1VrWowCgYIKoZIzj0EAwIw\ngY4xCzAJBgNVBAYTAlVTMRMwEQYDVQQIEwpDYWxpZm9ybmlhMRYwFAYDVQQHEw1T\nYW4gRnJhbmNpc2NvMRgwFgYDVQQKEw9IYXBweUNlcnQsIEluYy4xHzAdBgNVBAsT\nFkhhcHB5Q2VydCBJbnRlcm1lZGlhdGUxFzAVBgNVBAMTDihkZXYgdXNlIG9ubHkp\nMCAXDTE5MTIwOTE5NDgwMFoYDzIxMTkxMTE1MTk0ODAwWjCBgTELMAkGA1UEBhMC\nVVMxEzARBgNVBAgTCkNhbGlmb3JuaWExFjAUBgNVBAcTDVNhbiBGcmFuY2lzY28x\nGTAXBgNVBAoTEERFUiBpcyBGdW4sIEluYy4xETAPBgNVBAsTCEVuY29kaW5nMRcw\nFQYDVQQDEw4oZGV2IHVzZSBvbmx5KTBZMBMGByqGSM49AgEGCCqGSM49AwEHA0IA\nBJSBMLYEVPgjmd2vWgMpN9LupZa56T7Ds1+wAlyMphLDN56PWuphsrNsEwiIIeNv\nMtRTPRuoiBkfvMiWON6nkGWjgaEwgZ4wDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\nMAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHQYDVR0OBBYEFOYNuOCrYKnTFIEV\nck5845y/yZHkMB8GA1UdIwQYMBaAFFZRXwepqUwm9Kh+repV7LkBDnEHMCkGA1Ud\nEQQiMCCCCWRlcmlzLmZ1boITd2VsbGtub3duLmRlcmlzLmZ1bjAKBggqhkjOPQQD\nAgNIADBFAiEA9XAQ1Xi4Lav8LKzXZMSOHHj21ycqf3grnUfKJ6iwRvkCIDevfipo\nqIuR/Dnt1bBoXxFKv0w/LpH/89jIohUQwVSc\n-----END CERTIFICATE-----\n"
  },
  {
    "path": "pingora-proxy/tests/utils/conf/keys/ex1.key.b64",
    "content": "AAEAAJIx7XoAAAABAAAAIAAAAAAAAAACAAH//wAAABAAAAAAW66OYKnvlI3LQETZc85HajCUyhsAAAAAAAAAAAAAAAD+EOoVAAAAAQAAAKAAAAAAAAAAAv////8AAACQAAAAB018vkpfL1Bmrc2c9A5NcT3M3EdG+ZQfTZGN4BHUIpzOXK85cESryj5aFHIOh37fuRZlcCO8i9G44x+xNE45M9nw7tI2D4Sf1zraq9titAqMj3I+I3CZW2LX61CHyMYlfdxG/F7OR7dz1kbUcJeP73l+v65cPIEwek6gzvTZOIz2W8AnFdc0jW3iZFcgAhPmJzkBs4EAAAABAAAAMAAAAAAAAAACAAAAAAAAACAAAAAM0IInmYQDB4EBkHw182qCs6LncTgAAAAAAAAAAAAAAAA="
  },
  {
    "path": "pingora-proxy/tests/utils/conf/keys/intermediate.cnf",
    "content": "[ v3_intermediate_ca ]\nsubjectKeyIdentifier = hash\nbasicConstraints = critical, CA:true\n"
  },
  {
    "path": "pingora-proxy/tests/utils/conf/keys/intermediate.crt",
    "content": "-----BEGIN CERTIFICATE-----\nMIIEjjCCAnagAwIBAgIUGZ1/e3L6KJLlioDsIF7mOiBUO+UwDQYJKoZIhvcNAQEL\nBQAwXzELMAkGA1UEBhMCVVMxCzAJBgNVBAgMAkNBMRYwFAYDVQQHDA1TYW4gRnJh\nbmNpc2NvMRAwDgYDVQQKDAdSb290IENBMRkwFwYDVQQDDBByb290LnBpbmdvcmEu\nb3JnMB4XDTI1MDgxMjE5NTkxOFoXDTM1MDgxMDE5NTkxOFowTjELMAkGA1UEBhMC\nVVMxCzAJBgNVBAgMAkNBMRgwFgYDVQQKDA9JbnRlcm1lZGlhdGUgQ0ExGDAWBgNV\nBAMMD2ludC5waW5nb3JhLm9yZzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoC\nggEBAPLP2xGvLbhFS3iD7GyRk6e5Kc1FEb/0be39kL5POBGBl7SatwlIXzvc2RSW\n3Gla7ayvRiWo1WXDzuNv2M3Zu/uFmHkBa550Q2hWzD8StuWuDLVS79LVFcYOCE7r\n35fXZDAS1H63flgX5kxjuIGyR6gnZMrqLESoyYUpP+G6b3chv98n+ecCsVWNeFkx\nqxZibz+oVHfx1OoOUZSvvlQAt/5jhfKtDbCvUl5uz8VyxY4QWDUY92wx5JO+fbaA\nH0UCrb5MPCutqeKrtVInomRKzs+3pSAKLyDbhNGJgeCI99a1cFRt02Uh3tssQbVS\noBJ0P8ktcdlshS3HuM5zOcrc738CAwEAAaNTMFEwHQYDVR0OBBYEFCMHGxO+uP5T\nCjLDHk4ZFAMrY8l5MA8GA1UdEwEB/wQFMAMBAf8wHwYDVR0jBBgwFoAUnmFHSh1B\ng61ixjc/XxYgLZvIjWgwDQYJKoZIhvcNAQELBQADggIBAMJy3c/J36teAqy26+6N\noBRAe+9v8eTwg7hf6ZjaK8mhakc5AyIi5yTVdaW4CxZp1P934fLExCclwlY4GgUC\n6kaCpCA/E47q4mOkhQcMu/V2Kfq4SQ7yZzEfc7OOwnSKO2iyEiAiOqg8B6w6Mwt4\neKZ5AdiKTst1BG+rwIPayMekIcJ2Sg4DB9qGCCTQutgEdu8sE+/znZIocJypxAif\nWoZTnNmp7J96cM5MSBN1NNRT+xWgqsZWPMV+qaxRRU9e6TRpSJVKVBkSUdufXl8L\nM8d0D/ypv5dwhNQuuVhLEVqSGdzSDO0CyQdZNFVO2/DMgAfYaDs+ayh+93GSM4Ey\nCd6MQE583WjW2KzHbzykBZXj/FOoAw+HKvFhv+aG4GDO1xoyY/ATQBSpoUIlmYRm\nzU7TNGQ4osExHE057S2eP4suVFwcJWhjcevgHSYxUz1Z/5RkSDvuz9rftAmNUWU+\nSXaKQD8TpY7h/qBvybRxxwJ8BdAJbIO7S5zuZbH6AsECnSFTME6nRow39VNon5Lh\njMRbpNQn55e8yxSmgntq3IDj/1KO7p14cCKCeUW2CW4C0LAZiDp0+KWIXIjNgBZ0\nmcXdvMXWgGkWten6bdgkFH3v5P0b+Ow0fEGTreFuvpxjaLMRDZ9uImjekZig1/R8\n+LN8cTRARqfdw25RY23Xb4yF\n-----END CERTIFICATE-----\n"
  },
  {
    "path": "pingora-proxy/tests/utils/conf/keys/intermediate.csr",
    "content": "-----BEGIN CERTIFICATE REQUEST-----\nMIICkzCCAXsCAQAwTjELMAkGA1UEBhMCVVMxCzAJBgNVBAgMAkNBMRgwFgYDVQQK\nDA9JbnRlcm1lZGlhdGUgQ0ExGDAWBgNVBAMMD2ludC5waW5nb3JhLm9yZzCCASIw\nDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAPLP2xGvLbhFS3iD7GyRk6e5Kc1F\nEb/0be39kL5POBGBl7SatwlIXzvc2RSW3Gla7ayvRiWo1WXDzuNv2M3Zu/uFmHkB\na550Q2hWzD8StuWuDLVS79LVFcYOCE7r35fXZDAS1H63flgX5kxjuIGyR6gnZMrq\nLESoyYUpP+G6b3chv98n+ecCsVWNeFkxqxZibz+oVHfx1OoOUZSvvlQAt/5jhfKt\nDbCvUl5uz8VyxY4QWDUY92wx5JO+fbaAH0UCrb5MPCutqeKrtVInomRKzs+3pSAK\nLyDbhNGJgeCI99a1cFRt02Uh3tssQbVSoBJ0P8ktcdlshS3HuM5zOcrc738CAwEA\nAaAAMA0GCSqGSIb3DQEBCwUAA4IBAQBwlDK5c3FCzI2+d81n/ChiV83gRwZdBXZA\nxAdWDPvAaNkjig5xxX6h3D1lakdMBIyCZCU3Ln+I1wJ38B+uuKt4wpoYmefR7Zw2\nMtKD6IyiZdbMRN9/eV94pO8swh9SuUlTQhHU1Em0VRGVzBJrMTjh+wtKV4nm/6+6\nPBUzZARrBI9qOBO+WOvN6XnvjXmb05D4lEdZ1NvLIKm/r6Nkq+bKwLoAMvc/u/lS\nEcPLzAAa2fnzcdMhc1x1OLAp/+Dl3IDuQJSstqqUgWM59+3Nc1OBUaLYgIoq0W+W\nT96ilOxlS02SSLwaeWXqwEnYTJe/8JVYX2HXW42TdUwbAEdyLIa/\n-----END CERTIFICATE REQUEST-----\n"
  },
  {
    "path": "pingora-proxy/tests/utils/conf/keys/intermediate.key",
    "content": "-----BEGIN PRIVATE KEY-----\nMIIEuwIBADANBgkqhkiG9w0BAQEFAASCBKUwggShAgEAAoIBAQDyz9sRry24RUt4\ng+xskZOnuSnNRRG/9G3t/ZC+TzgRgZe0mrcJSF873NkUltxpWu2sr0YlqNVlw87j\nb9jN2bv7hZh5AWuedENoVsw/Erblrgy1Uu/S1RXGDghO69+X12QwEtR+t35YF+ZM\nY7iBskeoJ2TK6ixEqMmFKT/hum93Ib/fJ/nnArFVjXhZMasWYm8/qFR38dTqDlGU\nr75UALf+Y4XyrQ2wr1Jebs/FcsWOEFg1GPdsMeSTvn22gB9FAq2+TDwrraniq7VS\nJ6JkSs7Pt6UgCi8g24TRiYHgiPfWtXBUbdNlId7bLEG1UqASdD/JLXHZbIUtx7jO\ncznK3O9/AgMBAAECgf8B6w+uUkzJxuplRHNE1Oi5Khiy0p+dJK2DCITD7scuXmVg\nNVea20j3AWb7wUKxaWt82CdrFdnEPb39zv2u2T9XEFmRFgmuDB1vfUMn4N2LInFq\nYYa9Nwwb7mkFOWKEdEFPshEIuKCvABoA+w7TTgzf4jNHM5dcPakPbRd4LVNCiLbg\noDjExmPeQ3t694vieLpm0M/h1HlQ8WbvaaCT14lUbTKUH/fo/4wgPMfIO8W205Pm\ngj6rNSYhiK6f4OZp9i3ANZwZxarB2H3pXELda8EZQLoFD7s7mslQ+Lk8z3oy11Ov\n5olskRj8HROWwikHXikG+nnToEZWsDp6LpC6+xkCgYEA/l8iw9ORQvtnJIEpH2mS\nOclk/5kYpkxRk5XXfXDxGvf9sdYTArvQoHBSS9HCJSE7aNnjRFXMotUYxZAtc36Z\nLkrAn0hfyiip2f0sxKCnlRB5I0DF120u4uLi2mhGnOBX20uQv1Vj0+Y9TncUWTQx\nEVKCCs9ytZnlOvdcUE7H4scCgYEA9F3GotyzVT9T8MPni6j/iPFyeEg7CRb19dgp\nLwlAjmv5MA23Ok5TKCjOYqISSyVokEaro/pbfpp1amdzJ9ytBTOIeFJ0QVpskrmx\nXoKL2YZW2Qk0IAotYhBbLmi/5YPe6zfek4O9rQubd7lR1FauIQxa1it7ljaVREbc\nlScj1YkCgYAFuZhrteBIFKZuoOWPCm47FLhMNGLko0UWwEGYVilnBPvVu86zugxo\n//4qLK9k7ImMw5Kk4BV5+LfVAnizZ78E1rPdIeDeCOpBuLwANOlwpm1DiNqrDY8H\nljmq1rv4Heh8TAgW9lIH29+3W2C+3TjZffTlT2PyiGMrX5PZTtya0wKBgQC7UzPV\nXzg+LjirxZG3VwrksKpemIhg9HACUP1pKD+Lrius8aa3FJncnENyCunZH0kj6Hjl\nUCNZTxCZS8pUEW+1IAcKrbKe8rFuXNkiKRMJ4lirMcn6kbKujPlI/1WznL6DNCX0\nkTYS9GXuhmq7SuNbRDxSF606vob4exXXZNSseQKBgGOjaYiivK2kgiq5DOJOsGF/\njTgiHzvWDWqWmTUo2ATmIfhb45j18KTnosZKFvcSTIOfeNOHgBHNtCEShbmK3+zW\neVKmBeDZxePesonL2IHc1BHkF3PpQCm/p0nUBcuAtWXsW8Moj8sNh9xxAV5l0Xuj\n9roQeE6VmidUjodZ4tTi\n-----END PRIVATE KEY-----\n"
  },
  {
    "path": "pingora-proxy/tests/utils/conf/keys/intermediate.srl",
    "content": "199D7F7B72FA2892E58A80EC205EE63A20543BE5\n"
  },
  {
    "path": "pingora-proxy/tests/utils/conf/keys/key.pem",
    "content": "-----BEGIN EC PRIVATE KEY-----\nMHcCAQEEIN5lAOvtlKwtc/LR8/U77dohJmZS30OuezU9gL6vmm6DoAoGCCqGSM49\nAwEHoUQDQgAE2f/1Fm1HjySdokPq2T0F1xxol9nSEYQ+foFINeaWYk+FxMGpriJT\nBb8AGka87cWklw1ZqytfaT6pkureDbTkwg==\n-----END EC PRIVATE KEY-----\n"
  },
  {
    "path": "pingora-proxy/tests/utils/conf/keys/leaf.cnf",
    "content": "[ v3 ]\nsubjectKeyIdentifier = hash\nbasicConstraints = CA:false\nkeyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment"
  },
  {
    "path": "pingora-proxy/tests/utils/conf/keys/leaf.crt",
    "content": "-----BEGIN CERTIFICATE-----\nMIIDiTCCAnGgAwIBAgIUI/kgcBZtqXK/VnK2VZ47mS6DETwwDQYJKoZIhvcNAQEL\nBQAwTjELMAkGA1UEBhMCVVMxCzAJBgNVBAgMAkNBMRgwFgYDVQQKDA9JbnRlcm1l\nZGlhdGUgQ0ExGDAWBgNVBAMMD2ludC5waW5nb3JhLm9yZzAeFw0yNTA4MTIyMTQy\nMjJaFw0zNTA4MTAyMTQyMjJaMFMxCzAJBgNVBAYTAlVTMQswCQYDVQQIDAJDQTEh\nMB8GA1UECgwYSW50ZXJuZXQgV2lkZ2l0cyBQdHkgTHRkMRQwEgYDVQQDDAtwaW5n\nb3JhLm9yZzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAJ/2h9zWB+JG\npvOxJsQeUXr/IJtdtHagvNkRL8hvOFNIfazwXZ2bcbCO0GVVx8CfCFIH8nPpKQsU\nxnXQDkrZWHCj2jxATVOUT0HzDbxkq7Um+IicReOVy0pL5thKjdRc4pwJey4g1YsH\nCOXtjgcSjo7at9hHexVO1QFOkMT2c4kkzl6OIiLT2Lq3HfZ9ftdC1/4lNnlu97zy\nCeDR0woyoj/4ELXnsWZQxFWpc5Jh5SM/8vkOkeWWGIuY0SH4t/Km/5S8GTfxNdIA\nV2miOQhRC1ocQA3hXlOw41PTYk8mLh33YTTc/xWJ7d6BIosDpvK1HnTIyiPzB2io\nCTIq2xsn0+ECAwEAAaNaMFgwHQYDVR0OBBYEFEDWXZUM1tkxzrRgt8rsCU6LmaH2\nMAkGA1UdEwQCMAAwCwYDVR0PBAQDAgTwMB8GA1UdIwQYMBaAFCMHGxO+uP5TCjLD\nHk4ZFAMrY8l5MA0GCSqGSIb3DQEBCwUAA4IBAQBzlbtg8H+kVhWTXKcSZ5qfMI9D\nU6uLSL2OKmpuIe0N3qwtiGO3PLtG7cyqX3I//PCg86/YrHnFf2yDRQ4IGuxoDJUf\nAPSiKFO+44zRtgHBZffckBTil1LfyvePUgVxEDObV2XUyDiNmABmZzBOVXfrAh8X\nFbuGutOq8PiWX7rauoX5tw9tM2RRTdbxtlHfIPmcfj0nhIpIZmkLUZFaRkYaiaoE\n3HUpzcOqnBw3A5rbAP+buY7kxVslBDKCwKZeDMQooLEAOCvUKz4iHSRj17Ryj1aG\nUoRKI3XGQz6C8CbKMlI93EDcWQNSrvqve3TR/AIj9g/fiwVXvKQJs4CkjWEi\n-----END CERTIFICATE-----\n"
  },
  {
    "path": "pingora-proxy/tests/utils/conf/keys/leaf.csr",
    "content": "-----BEGIN CERTIFICATE REQUEST-----\nMIICmDCCAYACAQAwUzELMAkGA1UEBhMCVVMxCzAJBgNVBAgMAkNBMSEwHwYDVQQK\nDBhJbnRlcm5ldCBXaWRnaXRzIFB0eSBMdGQxFDASBgNVBAMMC3BpbmdvcmEub3Jn\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAn/aH3NYH4kam87EmxB5R\nev8gm120dqC82REvyG84U0h9rPBdnZtxsI7QZVXHwJ8IUgfyc+kpCxTGddAOStlY\ncKPaPEBNU5RPQfMNvGSrtSb4iJxF45XLSkvm2EqN1FzinAl7LiDViwcI5e2OBxKO\njtq32Ed7FU7VAU6QxPZziSTOXo4iItPYurcd9n1+10LX/iU2eW73vPIJ4NHTCjKi\nP/gQteexZlDEValzkmHlIz/y+Q6R5ZYYi5jRIfi38qb/lLwZN/E10gBXaaI5CFEL\nWhxADeFeU7DjU9NiTyYuHfdhNNz/FYnt3oEiiwOm8rUedMjKI/MHaKgJMirbGyfT\n4QIDAQABoAAwDQYJKoZIhvcNAQELBQADggEBADtr4Eb3FnNjfHI/3uKbdCOgskqS\nfRHerB4ctw7X+Cns26HwcOJCea5xnWP4JDBCvFmsyTAcwvdSRqyac1QNeUAkNBEV\nnDPf2PqkYZCLxurRM/gufBiLrWuZYr1ISU2WKV5O5SW6XA6FD8lbzYHxylHAD2+5\nK7rLw98rOqd0lqkgbgj89fAonws/ptV5FX4PgwDbfmn9kft2hytmk+BR/kOj77Qc\nSKBGNCtCulYACtRbuBaQueRCyMF/9G9s1+fn768Ecub5cQobvVSToPvEdGB15E4f\nyttahXtu3wxIQ9zAtd4tHhUDa2aozLxOnEOt/YOoLAHq7t6Rz4Azs+LnfeI=\n-----END CERTIFICATE REQUEST-----\n"
  },
  {
    "path": "pingora-proxy/tests/utils/conf/keys/leaf.key",
    "content": "-----BEGIN PRIVATE KEY-----\nMIIEvwIBADANBgkqhkiG9w0BAQEFAASCBKkwggSlAgEAAoIBAQCf9ofc1gfiRqbz\nsSbEHlF6/yCbXbR2oLzZES/IbzhTSH2s8F2dm3GwjtBlVcfAnwhSB/Jz6SkLFMZ1\n0A5K2Vhwo9o8QE1TlE9B8w28ZKu1JviInEXjlctKS+bYSo3UXOKcCXsuINWLBwjl\n7Y4HEo6O2rfYR3sVTtUBTpDE9nOJJM5ejiIi09i6tx32fX7XQtf+JTZ5bve88gng\n0dMKMqI/+BC157FmUMRVqXOSYeUjP/L5DpHllhiLmNEh+Lfypv+UvBk38TXSAFdp\nojkIUQtaHEAN4V5TsONT02JPJi4d92E03P8Vie3egSKLA6bytR50yMoj8wdoqAky\nKtsbJ9PhAgMBAAECggEAA1e6ABXi5UoXrAj8J+YASuMw8b40CrSSLbELwBL+6NKf\nebEuK6B3cDqTxUJVIcPQ/zHWUbDCIE6nVQfrfIntLLFn2pF3bDMxss2a8GBkLC1r\nzSMC3N4g+OT8JnHsY88rFxqlndGm1LhpabCcoq4zF24foF/iBRB4KAZVxR/nSyrW\nxUvDwTRoz+tdUSyvIOIPpmpqMV+G8UFTgH54AKc3PW5uwJn3gqcDYC/kKCAaIefD\nARfjGRAzi5VO8MJiEMD/3vu2nx3DQyZqINIANQqwEtCBM2RE+hCYbUm4/ln+VLed\nDmZwU2eX/19hluPcWDAblhTIZ4aPj5mQNj2TQXFDfwKBgQDPui3/qLC1ivuLAJas\nD8zmZLWOa3vYOROyAOBl7vor9ocXBoSAxqCbXjByzgNxe+5SHw+cyRQ0qAc9dMFd\nYvhHcpPVst+3dnZAccE9CLRDSQSRoWD8IGL70BTuN8+ju+qxfdYo8HTG15rARy9e\nereibDaa3Ef9nUQAg5g5dPzH+wKBgQDFItOlzQKD3Gwtoy9jiyP4DYHcnw8yb4lV\n8hSpvf7Qc+9+Bly4T/VRliA9nAmJnc4j6o87OB30wLvRiaOeZB2jex6189iAteME\nI1DbRyHyf9Vpd1lXA/9lPxzky7o7vWABfntAWZ/3RdMLMtP1doDwLMQw55fEsaoq\n7TvlmD0A0wKBgQCqs/TZA2czyOKtd+5ZtyJKsrgAMZO0PDNTNCUznw820YByC4kX\nyiJxixWFQobR22YdVikeTp+sJejNOAUvGQWusRmLo1L1EQRcMR77aQu5v2dhxZxN\nlM/C31xT5slbZDGZai9ztSZBwSwKlnT2zyHY99Rnrl36rCIVyg5uKRURwQKBgQCX\n5gfzH46qj/ODDtR6/UGP5siDeMQ69pp58Phe+pkXgd2t27UiB+pdHTJmho8KzN+D\n6T7IQKtEZiXShR3f9ACqcTnutZ/DPWNZUuUAsUTFGB8XDvF2DQyDtSfMW/Z6Baeu\nPwk1QlnyLIk7fcS4xMEBT1002Z6l3sfiH74hYTbQJQKBgQCCUmG2y8wjDP301VCx\n8/bSHZCTYdDnjbr9lyXnZ9LsSmW7BX7KwRltdsaxu6IsZkL0ACm2en2jwW4BIAoq\nuTsy5D16zqdInh0yKfUN6XZEK5BFWkD5E54m6NtP+YCaN0hLQlAcG7Cl4Epp/otL\nMDSZFlFnjA1dtFcj4nX2FD1jJA==\n-----END PRIVATE KEY-----\n"
  },
  {
    "path": "pingora-proxy/tests/utils/conf/keys/leaf.srl",
    "content": "23F92070166DA972BF5672B6559E3B992E83113C\n"
  },
  {
    "path": "pingora-proxy/tests/utils/conf/keys/leaf2.crt",
    "content": "-----BEGIN CERTIFICATE-----\nMIIDiTCCAnGgAwIBAgIUaU54M+5hDvm7AXbOOmnbYT/A40cwDQYJKoZIhvcNAQEL\nBQAwTjELMAkGA1UEBhMCVVMxCzAJBgNVBAgMAkNBMRgwFgYDVQQKDA9JbnRlcm1l\nZGlhdGUgQ0ExGDAWBgNVBAMMD2ludC5waW5nb3JhLm9yZzAeFw0yNTA4MTIyMTQz\nMzBaFw0zNTA4MTAyMTQzMzBaMFMxCzAJBgNVBAYTAlVTMQswCQYDVQQIDAJDQTEh\nMB8GA1UECgwYSW50ZXJuZXQgV2lkZ2l0cyBQdHkgTHRkMRQwEgYDVQQDDAtwaW5n\nb3JhLm9yZzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAPRZmAykZFbE\nhIvUC2SDFCN4eV0TzHlk6735U6Goo/AL8JadeiWEEXg3EG2PnMA1u7G7DU0PG8aw\nh54X4iIUDosv/pDQ9MhcHyQawoxlf+bHwTD+5YQHzLJ+hbfp7nzsak5ZE4rq1ZOt\nHd856MKEZv1O2WvUZwfZiC/5Gmv9b8pmgJHbjviwFtDKgJOCjgnkkV2kJ6H7KEww\nI2b/+aLtZGABPu57P1/rVtXvR4YSYmzF/VuLCmlpun3cQOK/5T8o4oi342dXn0EF\nTq2/xjLRe1/g+syUi249sq+h5TSeXHTkJs3Yo7/KxsitSFTOz+MuFLnDBgXRebvU\nEBicU03yPxUCAwEAAaNaMFgwHQYDVR0OBBYEFJVgFA+/ZHqcz7vEDg5CaivXc7JF\nMAkGA1UdEwQCMAAwCwYDVR0PBAQDAgTwMB8GA1UdIwQYMBaAFCMHGxO+uP5TCjLD\nHk4ZFAMrY8l5MA0GCSqGSIb3DQEBCwUAA4IBAQCW6uv2AETVq9tEWo3L0Gg+b3il\nDVOf2JuZ4UlVfhXoCOdflU2M2CEyBcmTqcZlEIFJhYNQzOOvOYG0IlVzrXdP6yHx\n/zj16d1+b5eLJhBPYeCMqBoFoTtnB8gE3BVp+9fkB+jD0/FkOPVE8RUZY/8mcFvd\nff58oB0bGDPLZkziH0kMsymlEpYbs2cTVDcvcoviblVRa4Wt35yTWyLTolxT4NTU\niI4yUKpx3xkGpOZTg26EXnrtLjAhGxdXpcJ9PLd6ClG81wiHZfhyshupoM4ad8lQ\nqvanYlaEkdzjzjr9Tw35HYEbQk2POarsjmd4yW9E5ER+xj6ia/lcyc4h/VFB\n-----END CERTIFICATE-----\n"
  },
  {
    "path": "pingora-proxy/tests/utils/conf/keys/leaf2.csr",
    "content": "-----BEGIN CERTIFICATE REQUEST-----\nMIICmDCCAYACAQAwUzELMAkGA1UEBhMCVVMxCzAJBgNVBAgMAkNBMSEwHwYDVQQK\nDBhJbnRlcm5ldCBXaWRnaXRzIFB0eSBMdGQxFDASBgNVBAMMC3BpbmdvcmEub3Jn\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA9FmYDKRkVsSEi9QLZIMU\nI3h5XRPMeWTrvflToaij8Avwlp16JYQReDcQbY+cwDW7sbsNTQ8bxrCHnhfiIhQO\niy/+kND0yFwfJBrCjGV/5sfBMP7lhAfMsn6Ft+nufOxqTlkTiurVk60d3znowoRm\n/U7Za9RnB9mIL/kaa/1vymaAkduO+LAW0MqAk4KOCeSRXaQnofsoTDAjZv/5ou1k\nYAE+7ns/X+tW1e9HhhJibMX9W4sKaWm6fdxA4r/lPyjiiLfjZ1efQQVOrb/GMtF7\nX+D6zJSLbj2yr6HlNJ5cdOQmzdijv8rGyK1IVM7P4y4UucMGBdF5u9QQGJxTTfI/\nFQIDAQABoAAwDQYJKoZIhvcNAQELBQADggEBACYNZWfEc+aOk6XOnITfeiCSMtub\nXFROVevgZ0GBJa11Ehvgra5bMZQqQVaWJHUTE+3xbAmqbXwEbV589MzvLO+67SFa\nSQOebAccMf5Tp0OBS4jpq/1w7PH1RjN2IFdH4HoXUyAA0STEbyXukxjKSQooSMl9\nNpunpJY4fueKit0Fly/AHmBBlTosk3fwR6PU5wLRAv3UlG0x2XQBS9DksZmHWtRN\nnGPp1zDMnG56R+Np5hR64R0b5zVRsXUK5qo/6aocdJp7wwWqba456AMEcoMfW6DQ\nd6fQ/+o6rug9Lyog+yUzpKlJWbsJ34+j8neJuhvlBkgA3PskA2drglapH44=\n-----END CERTIFICATE REQUEST-----\n"
  },
  {
    "path": "pingora-proxy/tests/utils/conf/keys/leaf2.key",
    "content": "-----BEGIN PRIVATE KEY-----\nMIIEuwIBADANBgkqhkiG9w0BAQEFAASCBKUwggShAgEAAoIBAQD0WZgMpGRWxISL\n1AtkgxQjeHldE8x5ZOu9+VOhqKPwC/CWnXolhBF4NxBtj5zANbuxuw1NDxvGsIee\nF+IiFA6LL/6Q0PTIXB8kGsKMZX/mx8Ew/uWEB8yyfoW36e587GpOWROK6tWTrR3f\nOejChGb9Ttlr1GcH2Ygv+Rpr/W/KZoCR2474sBbQyoCTgo4J5JFdpCeh+yhMMCNm\n//mi7WRgAT7uez9f61bV70eGEmJsxf1biwppabp93EDiv+U/KOKIt+NnV59BBU6t\nv8Yy0Xtf4PrMlItuPbKvoeU0nlx05CbN2KO/ysbIrUhUzs/jLhS5wwYF0Xm71BAY\nnFNN8j8VAgMBAAECgf9eLNRtYEP2gnHox9DxlujWwu1Y8kiHK7OwLxK3O51I51Eo\nEN8C6+PPxr6OJiDunnG4uQm8qWtgff5xmsLiX4M7d0P7Nzh2AGCq3vrHIazUmtMw\nDw271UW6MF6ug3q8qwxN0LG3g3V4H+tjcu5CtMT83BGa0uzixEm43klQqwfATwBy\n43w+JHZYxvWo6sxF/Pf24XJzalB9UKOBFbOp1wFRmHwrGwVVwwq/u2yzE848Opat\nKMEeAT3SZQh0Km6zmZ+/32Xir0p4QOxhkaBxE6uFQVcvcahZ0ovTHQPCqMjfvr3s\nLRRNqDVS24KCJY67+Feb1heKu4GKI9czoAH0QAECgYEA+9dp+NDOpvNxrv/VoL/S\nzSsVosILGLBKbnmnLsKRN+ahhX79uGy49AHIMcKsCOK5assfrl1aoJHL3IHZmcEB\nwEp8XkK3bOOCJJb0jlj9R0Ejh0Ext0A2zXGhs52JR817C/NIDaGeHPh4v25HZKep\nocZ7JWe96oDDluwHaCDibgECgYEA+GKDDgf7SIqQPWp7ToZM0G3CNU/cxzUTNkLS\nNC2oRt+ZUIjE0eo/Puihcgx5y8FPn8hs1iQI+zyTXPzspOsKnQL/djP6jMZ6XFWE\nR2Qcj/m+rTZSzcsNN3SBebOald1jLr48sIyyQTb0Gh0KtQ4KTf0yKuEgaHiF+ScM\nl4LhORUCgYApoNzqfRF7tUf4Zl+Yl7yvn0yPP8X3ycQz6LYC27SHaf8PAwPLhWU5\nKEZAO26WdWuyxGqzNskxO4hYJbqjWK0CbQ2LwzlwrVao168LDJipO5I03EjsgpfM\nc9kHyKWVkdiiDA+/+RQas9O5yO/SKoi2rglTEIfrCGfMPa2nv6/OAQKBgQDO6ino\nz2dat+uO7hyIfsKQw06M4Nm3rZQymJnJ09siJ3TtrPHhOPW071BG1PPFdGVjYzCf\nd2dv+7d7OEve2tp9kBjGHGj1SwZ10tueKVzN56wbWWzDeQqqjsipXKBDhijwsJOY\nM6zvPNs+wcDAsVCORYW8SMyZmwVoWEpaETKUPQKBgAV5otiYpAzuQNJGw93a3xzc\nPKMtGaFbPmeJ+F6WCneU/Qq1bYmWHe/ycQGB2S8MucWiBkY552ZGjJ/hrSBTVgc1\ncBlosE6qkkEbGG2ektvU+W4AjedvT3S4wPdfmOCkbZe+OVqS+4akgPjCfXpM6/eF\nHhtAJlWFZFHDbq7M3Y0E\n-----END PRIVATE KEY-----\n"
  },
  {
    "path": "pingora-proxy/tests/utils/conf/keys/leaf2.srl",
    "content": "694E7833EE610EF9BB0176CE3A69DB613FC0E347\n"
  },
  {
    "path": "pingora-proxy/tests/utils/conf/keys/public.pem",
    "content": "-----BEGIN PUBLIC KEY-----\nMFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE2f/1Fm1HjySdokPq2T0F1xxol9nS\nEYQ+foFINeaWYk+FxMGpriJTBb8AGka87cWklw1ZqytfaT6pkureDbTkwg==\n-----END PUBLIC KEY-----\n"
  },
  {
    "path": "pingora-proxy/tests/utils/conf/keys/root.crt",
    "content": "-----BEGIN CERTIFICATE-----\nMIIFnzCCA4egAwIBAgIUE5kg5Z26V4swShJoSwfNVsJkHbYwDQYJKoZIhvcNAQEL\nBQAwXzELMAkGA1UEBhMCVVMxCzAJBgNVBAgMAkNBMRYwFAYDVQQHDA1TYW4gRnJh\nbmNpc2NvMRAwDgYDVQQKDAdSb290IENBMRkwFwYDVQQDDBByb290LnBpbmdvcmEu\nb3JnMB4XDTIyMTExMDE5MjY1MFoXDTQyMTExMDE5MjY1MFowXzELMAkGA1UEBhMC\nVVMxCzAJBgNVBAgMAkNBMRYwFAYDVQQHDA1TYW4gRnJhbmNpc2NvMRAwDgYDVQQK\nDAdSb290IENBMRkwFwYDVQQDDBByb290LnBpbmdvcmEub3JnMIICIjANBgkqhkiG\n9w0BAQEFAAOCAg8AMIICCgKCAgEA4s1XxwZruaRwuDX1IkM2oxdSdjg7FeUp8lsN\nUix4NdXz8IoQWRzCfFuRBKFHptahutSO6Bbewm9XmU2hHG7aoCqaZqEVQ/3KRLZ4\nmzaNBCzDNgPTmDkz/DZKzOVuyVvbmTOsLn53yxKnFP9MEDIEemqGiM80MmFfCm/o\n0vLkjwkRpreMsWPUhrq3igTWRctUYMJAeDsEaaXB1k5ovWICrEylMzslgSNfoBed\nNmBpurz+yQddKNMTb/SLYxa7B1uZKDRSIXwwOZPdBDyUdlStUPodNG/OzprN+bRC\noFRB9EFG1m5oPJXQIalePj0dwhXl/bkV4uRxCSZmBZK3fbtLMF+Wkg2voTrn51Yv\nlKkzUQoEX6WWtUameZZbUB8TbW2lmANuvGBmvBbj3+4ztmtJPXfJBkckCeUC6bwC\n4CKrgB587ElY357Vqv/HmRRC9kxdzpOS9s5CtcqJ3Dg1TmLajyRQkf8wMqk0fhh7\nV+VrPXB030MGABXh5+B2HOsF307vF030v7z+Xp5VRLGBqmDwK0Reo2h8cg9PkMDS\n5Qc2zOJVslkJ+QYdkea1ajVpCsFbaC1JPmRWihTllboUqsk9oSS3jcIZ8vW3QKMg\nZbKtVbtVHr3mNGWuVs96iDN5Us3SJ6KGS8sanrAYAAB/NKd1Wl3I0aVtcb6eOONd\nedf9+b0CAwEAAaNTMFEwHQYDVR0OBBYEFJ5hR0odQYOtYsY3P18WIC2byI1oMB8G\nA1UdIwQYMBaAFJ5hR0odQYOtYsY3P18WIC2byI1oMA8GA1UdEwEB/wQFMAMBAf8w\nDQYJKoZIhvcNAQELBQADggIBAIrpAsrPre3R4RY0JmnvomgH+tCSMHb6dW52YrEl\nJkEG4cVc5MKs5QfPp8l2d1DngqiOUnOf0MWwWNDidHQZKrWs59j67L8qKN91VQKe\ncSNEX3iMFvE59Hr0Ner6Kr09wZLHVVNGcy0FdhWpJdDUGDoQjfL7n7usJyCUqWSq\n/pa1I9Is3ZfeQ5f7Ztrdz35vVPj+0BlHXbZM5AZi8Dwf3vXFBlPty3fITpE65cty\ncYnbpGto+wDoZj9fkKImjK21QsJdmHwaWRgmXX3WbdFBAbScTjDOc5Mls2VY8rSh\n+xLI1KMB0FHSJqrGoFN3uE+G1vJX/hgn98KZKob23yJr2TWr9LHI56sMfN5xdd5A\niOHxYODSrIAi1k+bSlDz6WfEtufoqwBwHiog4nFOXrlHpGO6eUB1QjaQJZwKn2zE\n3BjqJOoqbuBMg5XZRjihHcVVuZdU39/zQDwqliNpx3km4FzOiEoBABGzLP+Qt0Ch\ncJFS1Yc8ffv616yP4A9qkyogk9YBBvNbDLB7WV8h8p1s4JP3f5aDUlxtAD+E+3aJ\n8mrb3P7/0A2QyxlgX4qQOdj++b7GzXDxxLgOimJ4pLo0fdY8KWMeHvZPiMryHkMx\n3GSZCHeleSVBCPB2pPCzUqkkKADbjBX3SYJsAMF9uXQAR4U7wojjvAmbt6vJEh6j\nTEUG\n-----END CERTIFICATE-----\n"
  },
  {
    "path": "pingora-proxy/tests/utils/conf/keys/root.key",
    "content": "-----BEGIN PRIVATE KEY-----\nMIIJQgIBADANBgkqhkiG9w0BAQEFAASCCSwwggkoAgEAAoICAQDizVfHBmu5pHC4\nNfUiQzajF1J2ODsV5SnyWw1SLHg11fPwihBZHMJ8W5EEoUem1qG61I7oFt7Cb1eZ\nTaEcbtqgKppmoRVD/cpEtnibNo0ELMM2A9OYOTP8NkrM5W7JW9uZM6wufnfLEqcU\n/0wQMgR6aoaIzzQyYV8Kb+jS8uSPCRGmt4yxY9SGureKBNZFy1RgwkB4OwRppcHW\nTmi9YgKsTKUzOyWBI1+gF502YGm6vP7JB10o0xNv9ItjFrsHW5koNFIhfDA5k90E\nPJR2VK1Q+h00b87Oms35tEKgVEH0QUbWbmg8ldAhqV4+PR3CFeX9uRXi5HEJJmYF\nkrd9u0swX5aSDa+hOufnVi+UqTNRCgRfpZa1RqZ5lltQHxNtbaWYA268YGa8FuPf\n7jO2a0k9d8kGRyQJ5QLpvALgIquAHnzsSVjfntWq/8eZFEL2TF3Ok5L2zkK1yonc\nODVOYtqPJFCR/zAyqTR+GHtX5Ws9cHTfQwYAFeHn4HYc6wXfTu8XTfS/vP5enlVE\nsYGqYPArRF6jaHxyD0+QwNLlBzbM4lWyWQn5Bh2R5rVqNWkKwVtoLUk+ZFaKFOWV\nuhSqyT2hJLeNwhny9bdAoyBlsq1Vu1UeveY0Za5Wz3qIM3lSzdInooZLyxqesBgA\nAH80p3VaXcjRpW1xvp4441151/35vQIDAQABAoICABm7ytXeOKLbsZ51INc+YRio\nMMcRIkMduWCyTBSizxDssbz9LVWvGbIagZ3Q3txjRf54164lyiitkXbng/xB57R8\noQA8DrmkNisNuSmDSwTKP2wFiyCefPOFBX+yGJvoPEZpwoOT/eugtix/uxWrVy68\nn38uY3HD8pCwme41eRFxqfsMoH4QIbEXxnN2kQliRLSl1cLOj3WdRR0X0HKMiFkc\naTIi5+J7LQJxK3lb/yMdBpuwpjVXncD6MkaP8bCoB/yz0w3RlXcy+8TbSs0SVof1\nmRK2DPUMQ4qtlVGzvbgFIBB8fn9BUFhBa1wMey/mZC4hrgYMfXbYUIMZXpB5i9I+\nkLz4IuTYlKL46IWa+f1WritsC2F/Oog7zuejo2MNGmma+ITReCx2hxB1+H+yl3As\nHmXDjp4wDrnTIR38MgIfZmrtSqqvm5zUYsjEBFSleasH/K7uDddwqgYQ6TwUaqVY\neiDsyWELZQY+0JozP9zeE9J2X0HbOvid+fwwns1TPXyTjnPsLdSOCFuBZoWcYfiu\nXnFXCEjT3HDjx9ZmzAujm7is86QSkKDZHJB34DTd0eVs8EZyxNqsB748vfigc7ag\n1F/quaKYihBY7BKG8dDyJ6m7hyG2j4jHy5zZgG4mEs84n4ETvUSWK1g+vpVgb3vB\nMXcK6N8M/vAl+GT3LJOBAoIBAQD44nPNIYK3X1ZWj5zea4i/LucNHRwU7ViuXZJW\nc4WxeT2uo/24zVcUZlvgaor5QTlsw2Ab38gc6OxBwRv0v7GRlxufi+xpG/wJDJs3\nZSAMa4P5l/C06sOIpOq9p0X0Y+amVliAFcQtYQBTBK/APD3HIhm03hW9U1pT2jKV\nJnkKaA/eMZPj55wtKEHDuvUcYll7bF5xmp9+/ECSnobxFSE0sFbXWss8CkEVJBdr\nOFOlWNUJcGtBJwQi3P/OeOqotfo0BCxZ4Rt51/GFLqWjZC81lfvcVbcC4Ba8LXkI\nAlLYI1uPI0ohxIMFd27i6Q92Ih042LzTWfl1MwotBSBM8CNVAoIBAQDpSUW+kCao\nHOTPTn7mv8jR8Vp/uosyIqG4guynm65udI55n+y3881v/BrPGG6tsFaLsOTCUdR8\nmxiK0X7d6alSE94H8DREhMnRJjoVJsyvjF6mYleqdjDUFxzkwImu0TWsZz3NhIqv\n8kgSEa58JPEinufoKHVYh0J3LLXHYQ3J3sFx3IcO32Afe7pLwuLjEh7j1GWM7auW\nV0fpDMUjri/j7NF/4hiBnd7fs/i2nMp03+XxYxrqnInolhJkXxyVbsIwFLb0flbK\nEWeGudwMYc3W1f/uV2+OjdNPDY2ve7GntPMRFu7SSvFFjTRdqUhXlBfNUDGWugeT\ntng3onk7IUzJAoIBAQDd6PubkR995LGUqKQT5QmefXFhzey19BI4FhJeps4zuYh3\n6JxXZC8ab1HIPPcA21kaUvGkqNlCfaP51PbaOPlYeMUWcqot5dfJMcZLlA0JRev8\nZa8ngJMriPAMfdLv3wtOkHqEaePrGiwx2WHjI1Np9Eu7arEzh9hoH4suVYli7/oG\nAWp9sIsd8GEC5fWag06Jr8xduqIvlTb2BAcJee+LjRdBGSFQvUveT7nZzfU23ofE\nzMm049baRvaG4GVKXEdkjbwFv6LB9vrP5xGlJ7S4MKzKflqZY7ihvGHH9FptgMko\nTSzSAudXvm/OPkOc7zni780dHYJBL2sJTSLJtuupAoIBAHhoS0k6Wdl3YFnnp/Qt\nlNdXfWBjxiiQW2xClydDYVq9ajQ4aRPhEG32b1fowmd/lovvN4NcfRH7c0VjL9oW\nGkC05GqwfinHZ+s9kckNB6SsDMZQB/OBoV42t8ER536FmPBtMSb8fCCoKq641ZhZ\n8OPvpL7c8wRIe/PK7eAEpftFsA62xjbU8GYPlG46HqUY2zy4idmdamzki8crwizS\nYQGBX/hjmEZ+V2SbHYoTjyOX1LUsc94YAc48dy27MaOnUS9D4dJ7ywvsw8Rz9bGm\nYXm7Zqd8FaY8aY5p7nFepKls6fAuKAH+kF1XrmmRUDdzxn1AIPgs+HAzRAVjJLNy\nUpECggEAJLxoXdw6VbOCri1Q8wlA78ngcUEE09yL7qPVGtckCD1OdpJjkcskWoOO\nCkMsVtFjJOmQL0Xj/MR4/Zyk7qB3bm3oUWev4sFzdpWswN8fzOA7K4hTVC5BeSoS\n0uCiJ9/Up0Yte5Q0sHtO8U5xtnrSPYx5mHjPoh1ZbLem3OeGy1ifm/K8/0697bjX\n1UI5OSG/bZUU+bO7oBoZPIXoyMUYvnPBPqfdVI6E+mz1zFILOh9Vl7017Gi9UT9z\nhDb8K7IfTDTSgvqS+H7U0X9T8cfoSSWNxRo2DyaJ0aNt36qZzkJNhunvaif5W8f/\n74xuCrejGJzwfA5Uel7mb6rqB/1law==\n-----END PRIVATE KEY-----\n"
  },
  {
    "path": "pingora-proxy/tests/utils/conf/keys/root.srl",
    "content": "1C807FB6A8D925A28881E5B0BD746DD370B4C883\n"
  },
  {
    "path": "pingora-proxy/tests/utils/conf/keys/server.crt",
    "content": "-----BEGIN CERTIFICATE-----\nMIIB9zCCAZ2gAwIBAgIUMI7aLvTxyRFCHhw57hGt4U6yupcwCgYIKoZIzj0EAwIw\nZDELMAkGA1UEBhMCVVMxCzAJBgNVBAgMAkNBMRYwFAYDVQQHDA1TYW4gRnJhbmNp\nc2NvMRgwFgYDVQQKDA9DbG91ZGZsYXJlLCBJbmMxFjAUBgNVBAMMDW9wZW5ydXN0\neS5vcmcwHhcNMjIwNDExMjExMzEzWhcNMzIwNDA4MjExMzEzWjBkMQswCQYDVQQG\nEwJVUzELMAkGA1UECAwCQ0ExFjAUBgNVBAcMDVNhbiBGcmFuY2lzY28xGDAWBgNV\nBAoMD0Nsb3VkZmxhcmUsIEluYzEWMBQGA1UEAwwNb3BlbnJ1c3R5Lm9yZzBZMBMG\nByqGSM49AgEGCCqGSM49AwEHA0IABNn/9RZtR48knaJD6tk9BdccaJfZ0hGEPn6B\nSDXmlmJPhcTBqa4iUwW/ABpGvO3FpJcNWasrX2k+qZLq3g205MKjLTArMCkGA1Ud\nEQQiMCCCDyoub3BlbnJ1c3R5Lm9yZ4INb3BlbnJ1c3R5Lm9yZzAKBggqhkjOPQQD\nAgNIADBFAiAjISZ9aEKmobKGlT76idO740J6jPaX/hOrm41MLeg69AIhAJqKrSyz\nwD/AAF5fR6tXmBqlnpQOmtxfdy13wDr4MT3h\n-----END CERTIFICATE-----\n"
  },
  {
    "path": "pingora-proxy/tests/utils/conf/keys/server_boringssl_openssl.crt",
    "content": "-----BEGIN CERTIFICATE-----\nMIIB9zCCAZ2gAwIBAgIUMI7aLvTxyRFCHhw57hGt4U6yupcwCgYIKoZIzj0EAwIw\nZDELMAkGA1UEBhMCVVMxCzAJBgNVBAgMAkNBMRYwFAYDVQQHDA1TYW4gRnJhbmNp\nc2NvMRgwFgYDVQQKDA9DbG91ZGZsYXJlLCBJbmMxFjAUBgNVBAMMDW9wZW5ydXN0\neS5vcmcwHhcNMjIwNDExMjExMzEzWhcNMzIwNDA4MjExMzEzWjBkMQswCQYDVQQG\nEwJVUzELMAkGA1UECAwCQ0ExFjAUBgNVBAcMDVNhbiBGcmFuY2lzY28xGDAWBgNV\nBAoMD0Nsb3VkZmxhcmUsIEluYzEWMBQGA1UEAwwNb3BlbnJ1c3R5Lm9yZzBZMBMG\nByqGSM49AgEGCCqGSM49AwEHA0IABNn/9RZtR48knaJD6tk9BdccaJfZ0hGEPn6B\nSDXmlmJPhcTBqa4iUwW/ABpGvO3FpJcNWasrX2k+qZLq3g205MKjLTArMCkGA1Ud\nEQQiMCCCDyoub3BlbnJ1c3R5Lm9yZ4INb3BlbnJ1c3R5Lm9yZzAKBggqhkjOPQQD\nAgNIADBFAiAjISZ9aEKmobKGlT76idO740J6jPaX/hOrm41MLeg69AIhAJqKrSyz\nwD/AAF5fR6tXmBqlnpQOmtxfdy13wDr4MT3h\n-----END CERTIFICATE-----\n"
  },
  {
    "path": "pingora-proxy/tests/utils/conf/keys/server_boringssl_openssl.csr",
    "content": "-----BEGIN CERTIFICATE REQUEST-----\nMIIBJzCBzgIBADBsMQswCQYDVQQGEwJVUzETMBEGA1UECAwKQ2FsaWZvcm5pYTEW\nMBQGA1UEBwwNU2FuIEZyYW5jaXNjbzEYMBYGA1UECgwPQ2xvdWRmbGFyZSwgSW5j\nMRYwFAYDVQQDDA1vcGVucnVzdHkub3JnMFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcD\nQgAE2f/1Fm1HjySdokPq2T0F1xxol9nSEYQ+foFINeaWYk+FxMGpriJTBb8AGka8\n7cWklw1ZqytfaT6pkureDbTkwqAAMAoGCCqGSM49BAMCA0gAMEUCIFyDN8eamnoY\nXydKn2oI7qImigxahyCftzjxkIEV5IKbAiEAo5l72X4U+YTVYmyPPnJIj2v5nA1R\nRuUfMh5sXzwlwuM=\n-----END CERTIFICATE REQUEST-----\n"
  },
  {
    "path": "pingora-proxy/tests/utils/conf/keys/server_rustls.crt",
    "content": "-----BEGIN CERTIFICATE-----\nMIICJzCCAc6gAwIBAgIUU+G0acG/uiMu1ZDSjlcoY4gH53QwCgYIKoZIzj0EAwIw\nZDELMAkGA1UEBhMCVVMxCzAJBgNVBAgMAkNBMRYwFAYDVQQHDA1TYW4gRnJhbmNp\nc2NvMRgwFgYDVQQKDA9DbG91ZGZsYXJlLCBJbmMxFjAUBgNVBAMMDW9wZW5ydXN0\neS5vcmcwHhcNMjQwNzI0MTMzOTQ4WhcNMzQwNzIyMTMzOTQ4WjBkMQswCQYDVQQG\nEwJVUzELMAkGA1UECAwCQ0ExFjAUBgNVBAcMDVNhbiBGcmFuY2lzY28xGDAWBgNV\nBAoMD0Nsb3VkZmxhcmUsIEluYzEWMBQGA1UEAwwNb3BlbnJ1c3R5Lm9yZzBZMBMG\nByqGSM49AgEGCCqGSM49AwEHA0IABNn/9RZtR48knaJD6tk9BdccaJfZ0hGEPn6B\nSDXmlmJPhcTBqa4iUwW/ABpGvO3FpJcNWasrX2k+qZLq3g205MKjXjBcMDsGA1Ud\nEQQ0MDKCDyoub3BlbnJ1c3R5Lm9yZ4INb3BlbnJ1c3R5Lm9yZ4IHY2F0LmNvbYIH\nZG9nLmNvbTAdBgNVHQ4EFgQUnfYAFWyQnSN57IGokj7jcz8ChJQwCgYIKoZIzj0E\nAwIDRwAwRAIgQr+Ly2cH04CncbnbhUf4hBl5frTp1pXgGnn8dYjd+UcCICuunEtp\nH/a42/sVGBFvjS6FOFe6ZDs4oWBNEqQSw0S2\n-----END CERTIFICATE-----"
  },
  {
    "path": "pingora-proxy/tests/utils/conf/keys/server_s2n.crt",
    "content": "-----BEGIN CERTIFICATE-----\nMIICJzCCAc6gAwIBAgIUU+G0acG/uiMu1ZDSjlcoY4gH53QwCgYIKoZIzj0EAwIw\nZDELMAkGA1UEBhMCVVMxCzAJBgNVBAgMAkNBMRYwFAYDVQQHDA1TYW4gRnJhbmNp\nc2NvMRgwFgYDVQQKDA9DbG91ZGZsYXJlLCBJbmMxFjAUBgNVBAMMDW9wZW5ydXN0\neS5vcmcwHhcNMjQwNzI0MTMzOTQ4WhcNMzQwNzIyMTMzOTQ4WjBkMQswCQYDVQQG\nEwJVUzELMAkGA1UECAwCQ0ExFjAUBgNVBAcMDVNhbiBGcmFuY2lzY28xGDAWBgNV\nBAoMD0Nsb3VkZmxhcmUsIEluYzEWMBQGA1UEAwwNb3BlbnJ1c3R5Lm9yZzBZMBMG\nByqGSM49AgEGCCqGSM49AwEHA0IABNn/9RZtR48knaJD6tk9BdccaJfZ0hGEPn6B\nSDXmlmJPhcTBqa4iUwW/ABpGvO3FpJcNWasrX2k+qZLq3g205MKjXjBcMDsGA1Ud\nEQQ0MDKCDyoub3BlbnJ1c3R5Lm9yZ4INb3BlbnJ1c3R5Lm9yZ4IHY2F0LmNvbYIH\nZG9nLmNvbTAdBgNVHQ4EFgQUnfYAFWyQnSN57IGokj7jcz8ChJQwCgYIKoZIzj0E\nAwIDRwAwRAIgQr+Ly2cH04CncbnbhUf4hBl5frTp1pXgGnn8dYjd+UcCICuunEtp\nH/a42/sVGBFvjS6FOFe6ZDs4oWBNEqQSw0S2\n-----END CERTIFICATE-----"
  },
  {
    "path": "pingora-proxy/tests/utils/conf/keys/v3.ext",
    "content": "authorityKeyIdentifier=keyid,issuer\nbasicConstraints=CA:FALSE\nkeyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment"
  },
  {
    "path": "pingora-proxy/tests/utils/conf/origin/.gitignore",
    "content": "**\n!html\n!html/**\n!conf\n!conf/**\n!.gitignore\n"
  },
  {
    "path": "pingora-proxy/tests/utils/conf/origin/conf/nginx.conf",
    "content": "\n#user  nobody;\nworker_processes  1;\n\nerror_log  /dev/stdout;\n#error_log  logs/error.log  notice;\n#error_log  logs/error.log  info;\n\npid        /tmp/pingora_mock_origin.pid;\nmaster_process off;\ndaemon off;\n\nevents {\n    worker_connections  4096;\n}\n\n\nhttp {\n    #include       mime.types;\n    #default_type  application/octet-stream;\n\n    #log_format  main  '$remote_addr - $remote_user [$time_local] \"$request\" '\n    #                  '$status $body_bytes_sent \"$http_referer\" '\n    #                  '\"$http_user_agent\" \"$http_x_forwarded_for\"';\n\n    access_log  off;\n\n    sendfile        on;\n    #tcp_nopush     on;\n\n    keepalive_timeout  60;\n    keepalive_requests 99999;\n\n    lua_shared_dict hit_counter 10m;\n\n    #gzip  on;\n\n    # mTLS endpoint\n    server {\n        listen       8444 ssl http2;\n        ssl_certificate keys/server.crt;\n        ssl_certificate_key keys/key.pem;\n        ssl_protocols TLSv1.2;\n        ssl_ciphers TLS-AES-128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256;\n        ssl_client_certificate keys/root.crt;\n        ssl_verify_client on;\n        ssl_verify_depth 4;\n\n        location / {\n            return 200 \"hello world\";\n        }\n    }\n\n    # secp384r1 endpoint (ECDH and ECDSA)\n    server {\n        listen 8445 ssl http2;\n        ssl_protocols TLSv1.2;\n        ssl_ciphers TLS-AES-128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA512;\n        ssl_certificate keys/curve_test.384.crt;\n        ssl_certificate_key keys/curve_test.384.key.pem;\n        ssl_ecdh_curve secp384r1;\n\n        location /384 {\n            return 200 \"Happy Friday!\";\n        }\n    }\n\n    # secp521r1 endpoint (ECDH and ECDSA)\n    server {\n        listen 8446 ssl http2;\n        ssl_protocols TLSv1.2;\n        ssl_ciphers TLS-AES-128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA512;\n        ssl_certificate keys/curve_test.521.crt;\n        ssl_certificate_key keys/curve_test.521.key.pem;\n        ssl_ecdh_curve secp521r1;\n\n        location /521 {\n            return 200 \"Happy Monday!\";\n        }\n    }\n\n    server {\n        listen       8000 http2;\n        # 8001 is used for bad_lb test only to avoid unexpected connection reuse\n        listen       8001;\n        listen       [::]:8000;\n        #listen       8443 ssl;\n        listen       unix:/tmp/pingora_nginx_test.sock;\n        listen       8443 ssl http2;\n        server_name  localhost;\n\n        ssl_certificate keys/server.crt;\n        ssl_certificate_key keys/key.pem;\n        ssl_protocols TLSv1.2;\n        ssl_ciphers TLS-AES-128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256;\n\n        # for benchmark\n        http2_max_requests 999999;\n\n        # increase max body size for /upload/ test\n        client_max_body_size 128m;\n        #charset koi8-r;\n\n        #access_log  logs/host.access.log  main;\n\n        add_header Origin-Http2 $http2;\n\n        location / {\n            root   ./html;\n            index  index.html index.htm;\n        }\n\n        # this allows an arbitrary prefix to be included in URLs, so\n        # that tests can control caching.\n        location ~ ^/unique/[^/]+(/.*)$ {\n            rewrite ^/unique/[^/]+(/.*)$ $1 last;\n        }\n\n        # this serves as an origin hit counter for an arbitrary prefix, which\n        # then redirects to the rest of the URL like our unique/... endpoint.\n        location ~ ^/hitcounted/[^/]+(/.*)$ {\n            rewrite_by_lua_block {\n                -- Extract specified ID\n                local _, _, id = string.find(ngx.var.request_uri, \"[^/]+/([^/]+)\")\n\n                -- Incr hit counter\n                local hits = ngx.shared.hit_counter\n                if not hits:get(id) then\n                    hits:safe_set(id, 0, nil)\n                end\n                local value = hits:incr(id, 1)\n\n                -- Rewrite URI to the requested destination\n                local destStartIndex = string.find(ngx.var.request_uri, id) + string.len(id)\n                local dest = string.sub(ngx.var.request_uri, destStartIndex)\n                ngx.req.set_uri(dest, true)\n            }\n        }\n\n        # this serves the hit count from the hitcounted endpoint\n        location ~ ^/read_hit_count/[^/]+(/.*)$ {\n            content_by_lua_block {\n                -- Find the hit count for the given ID and return it.\n                local _, _, id = string.find(ngx.var.request_uri, \"[^/]+/([^/]+)\")\n                local hits = ngx.shared.hit_counter\n                ngx.print(hits:get(id) or 0)\n            }\n        }\n\n        location /test {\n            return 200;\n        }\n        location /test2 {\n            return 200 \"hello world\";\n        }\n        location /test3 {\n            #return 200;\n            content_by_lua_block {\n                ngx.print(\"hello world\")\n            }\n        }\n\n        location /test4 {\n            rewrite_by_lua_block {\n                ngx.exit(200)\n            }\n            #return 201;\n\n        }\n\n        location /now {\n            header_filter_by_lua_block {\n                ngx.header[\"x-epoch\"] = ngx.now()\n            }\n            return 200 \"hello world\";\n        }\n\n        location /brotli {\n            header_filter_by_lua_block {\n                local ae = ngx.req.get_headers()[\"Accept-Encoding\"]\n                if ae and ae:find(\"br\") then\n                    ngx.header[\"Content-Encoding\"] = \"br\"\n                else\n                    return ngx.exit(400)\n                end\n            }\n            content_by_lua_block {\n                -- brotli compressed 'hello'.\n                ngx.print(\"\\x0f\\x02\\x80hello\\x03\")\n            }\n        }\n\n        location /cache_control {\n            header_filter_by_lua_block {\n                local h = ngx.req.get_headers()\n                if h[\"set-cache-control\"] then\n                    ngx.header[\"Cache-Control\"] = h[\"set-cache-control\"]\n                end\n                if h[\"set-cache-tag\"] then\n                    ngx.header[\"Cache-Tag\"] = h[\"set-cache-tag\"]\n                end\n                if h[\"set-revalidated\"] then\n                    return ngx.exit(304)\n                end\n            }\n            return 200 \"hello world\";\n        }\n\n        location /revalidate_now {\n            header_filter_by_lua_block {\n                ngx.header[\"x-epoch\"] = ngx.now()\n                ngx.header[\"Last-Modified\"] = \"Tue, 03 May 2022 01:04:39 GMT\"\n                ngx.header[\"Etag\"] = '\"abcd\"'\n                local h = ngx.req.get_headers()\n                if h[\"if-modified-since\"] or h[\"if-none-match\"] then\n                    -- just assume they match\n                    return ngx.exit(304)\n                end\n            }\n            return 200 \"hello world\";\n        }\n\n        location /vary {\n            header_filter_by_lua_block {\n                ngx.header[\"Last-Modified\"] = \"Tue, 03 May 2022 01:04:39 GMT\"\n                ngx.header[\"Etag\"] = '\"abcd\"'\n                local h = ngx.req.get_headers()\n                if h[\"set-vary\"] then\n                    ngx.header[\"Vary\"] = h[\"set-vary\"]\n                end\n                ngx.header[\"x-epoch\"] = ngx.now()\n                if not h[\"x-no-revalidate\"] and (h[\"if-modified-since\"] or h[\"if-none-match\"]) then\n                    -- just assume they match\n                    return ngx.exit(304)\n                end\n            }\n            return 200 \"hello world\";\n        }\n\n        location /no_if_headers {\n            content_by_lua_block {\n                local h = ngx.req.get_headers()\n                if h[\"if-modified-since\"] or h[\"if-none-match\"] or h[\"range\"] then\n                    return ngx.exit(400)\n                end\n                ngx.say(\"no if headers detected\")\n            }\n        }\n\n        location /client_ip {\n            add_header x-client-ip $remote_addr;\n            return 200;\n        }\n\n        # 1. A origin load balancer that rejects reused connections.\n        # This is to simulate the common problem when an upstream LB drops\n        # a connection silently after being `keepalive`d for a while.\n        # 2. A middlebox might drop the connection if the origin takes too long\n        # to respond. We should not retry in this case.\n        location /bad_lb {\n            rewrite_by_lua_block {\n                ngx.sleep(1)\n                if tonumber(ngx.var.connection_requests) > 1 then\n                    -- force drop the request and close the connection\n                    ngx.exit(444)\n                end\n                ngx.req.read_body()\n                local data = ngx.req.get_body_data()\n                if data then\n                    ngx.say(data)\n                else\n                    ngx.say(\"dog!\")\n                end\n            }\n        }\n\n        location /duplex/ {\n            client_max_body_size 1G;\n            content_by_lua_block {\n                ngx.print(string.rep(\"A\", 64))\n                ngx.print(string.rep(\"A\", 64))\n                ngx.print(string.rep(\"A\", 64))\n                ngx.print(string.rep(\"A\", 64))\n                ngx.print(string.rep(\"A\", 64))\n                -- without ngx.req.read_body(), the body will return without waiting for req body\n            }\n        }\n\n        location /upload/ {\n            client_max_body_size 1G;\n            content_by_lua_block {\n                ngx.req.read_body()\n                ngx.print(string.rep(\"A\", 64))\n                ngx.print(string.rep(\"A\", 64))\n                ngx.print(string.rep(\"A\", 64))\n                ngx.print(string.rep(\"A\", 64))\n                ngx.print(string.rep(\"A\", 64))\n            }\n        }\n\n        location /upload_connection_die/ {\n            content_by_lua_block {\n                ngx.status = ngx.HTTP_OK\n                ngx.print(\"\")\n                ngx.flush(true)\n\n                time.sleep(1)\n                ngx.exit(444)\n            }\n        }\n\n        location /download/ {\n            content_by_lua_block {\n                ngx.req.read_body()\n                local body = string.rep(\"A\", 4194304)\n                ngx.header[\"Content-Length\"] = #body\n                ngx.print(body)\n            }\n        }\n\n        location /download_large/ {\n            content_by_lua_block {\n                ngx.req.read_body()\n                local chunk = string.rep(\"A\", 1048576) -- 1MB chunk\n                local total_size = 128 * 1048576 -- 128MB total\n                ngx.header[\"Content-Length\"] = total_size\n                for i = 1, 128 do\n                    ngx.print(chunk)\n                    ngx.flush()\n                end\n            }\n        }\n\n        location /tls_verify {\n            keepalive_timeout 0;\n            return 200;\n        }\n\n        location /noreuse {\n            keepalive_timeout 0;\n            return 200 \"hello world\";\n        }\n\n        location /set_cookie {\n            add_header Set-Cookie \"chocolate chip\";\n            return 200 \"hello world\";\n        }\n\n        location /chunked {\n            content_by_lua_block {\n                ngx.req.read_body()\n                ngx.print(string.rep(\"A\", 64))\n            }\n        }\n\n        location /echo {\n            content_by_lua_block {\n                ngx.req.read_body()\n                local data = ngx.req.get_body_data()\n                if data then\n                    ngx.print(data)\n                end\n            }\n        }\n\n        location /low_ttl {\n            add_header Cache-Control \"public, max-age=0\";\n            return 200 \"low ttl\";\n        }\n\n        location /connection_die {\n            content_by_lua_block {\n                ngx.print(string.rep(\"A\", 5))\n                ngx.flush()\n                ngx.exit(444) -- 444 kills the connection right away\n            }\n        }\n\n        location /103 {\n            content_by_lua_block {\n                local sock, err = ngx.req.socket(true)\n                if not sock then\n                    ngx.log(ngx.ERR, \"Failed socket:\", err)\n                    return\n                end\n\n                local ok, err = sock:send(\"HTTP/1.1 103 Early Hints\\r\\nLink: </style.css>; rel=preload\\r\\n\\r\\n\")\n                if not ok then\n                    ngx.log(ngx.ERR, \"Failed 103:\", err)\n                end\n\n                ngx.sleep(1)\n\n                local ok, err = sock:send(\"HTTP/1.1 200 OK\\r\\nContent-Length: 6\\r\\n\\r\\n123456\\r\\n\\r\\n\")\n                if not ok then\n                    ngx.log(ngx.ERR, \"Failed 200:\", err)\n                end\n            }\n        }\n\n        location /103-die {\n            content_by_lua_block {\n                local sock, err = ngx.req.socket(true)\n                if not sock then\n                    ngx.log(ngx.ERR, \"Failed socket:\", err)\n                    return\n                end\n\n                local ok, err = sock:send(\"HTTP/1.1 103 Early Hints\\r\\nLink: </style.css>; rel=preload\\r\\n\\r\\n\")\n                if not ok then\n                    ngx.log(ngx.ERR, \"Failed 103:\", err)\n                end\n\n                ngx.sleep(1)\n\n                ngx.exit(444) -- 444 kills the connection right away\n            }\n        }\n\n        location /no_compression {\n            gzip off; # avoid accidental turn it on at server block\n            content_by_lua_block {\n                ngx.print(string.rep(\"B\", 32))\n            }\n        }\n\n        location /file_maker {\n            gzip off; # fixed content size\n            content_by_lua_block {\n                local size = tonumber(ngx.var.http_x_set_size) or 1024\n                ngx.print(string.rep(\"A\", size))\n            }\n        }\n\n        location /gzip {\n            alias ./html;\n            gzip on;\n            gzip_min_length   0;\n            gzip_types        *;\n            add_header received-accept-encoding $http_accept_encoding;\n        }\n\n        location /sleep {\n            rewrite_by_lua_block {\n                local sleep_sec = tonumber(ngx.var.http_x_set_sleep) or 1\n                ngx.sleep(sleep_sec)\n                if ngx.var.http_x_abort then\n                    -- force drop the request and close the connection\n                    ngx.exit(444)\n                end\n            }\n            content_by_lua_block {\n                if ngx.var.http_x_error_header then\n                    ngx.status = 500\n                    ngx.exit(0)\n                    return\n                end\n                ngx.print(\"hello \")\n                ngx.flush()\n                local sleep_sec = tonumber(ngx.var.http_x_set_body_sleep) or 0\n                ngx.sleep(sleep_sec)\n                if ngx.var.http_x_abort_body then\n                    ngx.flush()\n                    -- force drop the request and close the connection\n                    ngx.exit(444)\n                    return\n                end\n                ngx.print(\"world\")\n            }\n            header_filter_by_lua_block {\n                if ngx.var.http_x_no_store then\n                    ngx.header[\"Cache-control\"] = \"no-store\"\n                end\n                if ngx.var.http_x_no_stale_revalidate then\n                    ngx.header[\"Cache-control\"] = \"stale-while-revalidate=0\"\n                end\n                if ngx.var.http_x_set_content_length then\n                    ngx.header[\"Content-Length\"] = \"11\" -- based on \"hello world\"\n                end\n            }\n        }\n\n        location /set_content_length {\n            header_filter_by_lua_block {\n                if ngx.var.http_x_set_content_length then\n                    ngx.header[\"Content-Length\"] = ngx.var.http_x_set_content_length\n                end\n            }\n            return 200 \"hello world\";\n        }\n\n        location /slow_body {\n            content_by_lua_block {\n                local sleep_sec = tonumber(ngx.var.http_x_set_sleep) or 1\n                local hello_to = ngx.var.http_x_set_hello or \"world\"\n                ngx.flush()\n                ngx.sleep(sleep_sec)\n                ngx.print(\"hello \")\n                ngx.flush()\n                ngx.sleep(sleep_sec)\n                ngx.print(hello_to)\n                ngx.sleep(sleep_sec)\n                ngx.print(\"!\")\n            }\n            header_filter_by_lua_block {\n                if ngx.var.http_x_no_store then\n                    ngx.header[\"Cache-control\"] = \"no-store\"\n                end\n            }\n        }\n\n        location /content_type {\n            header_filter_by_lua_block {\n                ngx.header[\"Content-Type\"] = ngx.var.http_set_content_type\n            }\n            return 200 \"hello world\";\n        }\n\n        location /upgrade {\n            content_by_lua_block {\n                ngx.status = 101\n                ngx.header['Upgrade'] = 'websocket'\n                ngx.header['Connection'] = 'Upgrade'\n                ngx.say('hello')\n            }\n        }\n\n        location /upgrade_echo_body {\n            rewrite_by_lua_block {\n                ngx.req.read_body()\n                local data = ngx.req.get_body_data()\n                ngx.status = 101\n                ngx.header['Upgrade'] = 'websocket'\n                ngx.header['Connection'] = 'Upgrade'\n\n                if data then\n                    ngx.print(data)\n                end\n            }\n        }\n\n        #error_page  404              /404.html;\n\n        # redirect server error pages to the static page /50x.html\n        #\n        error_page   500 502 503 504  /50x.html;\n        location = /50x.html {\n            root   html;\n        }\n    }\n}\n"
  },
  {
    "path": "pingora-proxy/tests/utils/conf/origin/html/index.html",
    "content": "Hello World!\n"
  },
  {
    "path": "pingora-proxy/tests/utils/mock_origin.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\nuse once_cell::sync::Lazy;\nuse std::path::Path;\nuse std::process;\nuse std::{thread, time};\n\npub static MOCK_ORIGIN: Lazy<bool> = Lazy::new(init);\n\nfn init() -> bool {\n    #[cfg(feature = \"rustls\")]\n    let src_cert_path = format!(\n        \"{}/tests/utils/conf/keys/server_rustls.crt\",\n        env!(\"CARGO_MANIFEST_DIR\")\n    );\n    #[cfg(feature = \"openssl_derived\")]\n    let src_cert_path = format!(\n        \"{}/tests/utils/conf/keys/server_boringssl_openssl.crt\",\n        env!(\"CARGO_MANIFEST_DIR\")\n    );\n    #[cfg(feature = \"s2n\")]\n    let src_cert_path = format!(\n        \"{}/tests/utils/conf/keys/server_s2n.crt\",\n        env!(\"CARGO_MANIFEST_DIR\")\n    );\n\n    #[cfg(feature = \"any_tls\")]\n    {\n        let mut dst_cert_path = format!(\"{}/tests/keys/server.crt\", env!(\"CARGO_MANIFEST_DIR\"));\n        std::fs::copy(Path::new(&src_cert_path), Path::new(&dst_cert_path));\n        dst_cert_path = format!(\n            \"{}/tests/utils/conf/keys/server.crt\",\n            env!(\"CARGO_MANIFEST_DIR\")\n        );\n        std::fs::copy(Path::new(&src_cert_path), Path::new(&dst_cert_path));\n    }\n\n    // TODO: figure out a way to kill openresty when exiting\n    process::Command::new(\"pkill\")\n        .args([\"-F\", \"/tmp/pingora_mock_origin.pid\"])\n        .spawn()\n        .unwrap()\n        .wait();\n    let _origin = thread::spawn(|| {\n        process::Command::new(\"openresty\")\n            .args([\"-p\", &format!(\"{}/origin\", super::conf_dir())])\n            .output()\n            .unwrap();\n    });\n    // wait until the server is up\n    thread::sleep(time::Duration::from_secs(2));\n    true\n}\n"
  },
  {
    "path": "pingora-proxy/tests/utils/mod.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n#![allow(unused)]\n\n#[cfg(feature = \"any_tls\")]\npub mod cert;\n\npub mod mock_origin;\npub mod server_utils;\npub mod websocket;\n\nuse once_cell::sync::Lazy;\nuse tokio::runtime::{Builder, Runtime};\n\n// for tests with a static connection pool, if we use tokio::test the reactor\n// will no longer be associated with the backing pool fds since it's dropped per test\npub static GLOBAL_RUNTIME: Lazy<Runtime> =\n    Lazy::new(|| Builder::new_multi_thread().enable_all().build().unwrap());\n\npub fn conf_dir() -> String {\n    format!(\"{}/tests/utils/conf\", env!(\"CARGO_MANIFEST_DIR\"))\n}\n"
  },
  {
    "path": "pingora-proxy/tests/utils/server_utils.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n#[cfg(feature = \"any_tls\")]\nuse super::cert;\nuse async_trait::async_trait;\nuse clap::Parser;\nuse http::header::{ACCEPT_ENCODING, CONTENT_LENGTH, TRANSFER_ENCODING, VARY};\nuse http::HeaderValue;\nuse log::error;\nuse once_cell::sync::Lazy;\nuse pingora_cache::cache_control::CacheControl;\nuse pingora_cache::hashtable::ConcurrentHashTable;\nuse pingora_cache::key::HashBinary;\nuse pingora_cache::lock::CacheKeyLockImpl;\nuse pingora_cache::{\n    eviction::simple_lru::Manager, filters::resp_cacheable, lock::CacheLock, predictor::Predictor,\n    set_compression_dict_path, CacheKey, CacheMeta, CacheMetaDefaults, CachePhase, MemCache,\n    NoCacheReason, RespCacheable,\n};\nuse pingora_cache::{\n    CacheOptionOverrides, ForcedFreshness, HitHandler, PurgeType, VarianceBuilder,\n};\nuse pingora_core::apps::{HttpServerApp, HttpServerOptions};\nuse pingora_core::modules::http::compression::ResponseCompression;\nuse pingora_core::protocols::{\n    http::error_resp::gen_error_response, l4::socket::SocketAddr, Digest,\n};\nuse pingora_core::server::configuration::Opt;\nuse pingora_core::services::{Service, ServiceWithDependents};\nuse pingora_core::upstreams::peer::HttpPeer;\nuse pingora_core::utils::tls::CertKey;\nuse pingora_error::{Error, ErrorSource, ErrorType::*, Result};\nuse pingora_http::{RequestHeader, ResponseHeader};\nuse pingora_proxy::{FailToProxy, ProxyHttp, Session};\nuse std::collections::{HashMap, HashSet};\nuse std::sync::Arc;\nuse std::thread;\nuse std::time::{Duration, SystemTime};\n\npub struct ExampleProxyHttps {}\n\npub const TEST_PSK_IDENTITY: &str = \"test-psk-identity\";\npub const TEST_PSK_SECRET: &str = \"i2Wx8jrYVi5Vt7HSL/fsk003+PnmfcFuwWMsUyQvcZ4=\";\n\n#[allow(clippy::upper_case_acronyms)]\n#[derive(Default)]\npub struct CTX {\n    conn_reused: bool,\n    upstream_client_addr: Option<SocketAddr>,\n    upstream_server_addr: Option<SocketAddr>,\n}\n\n// Common logic for both ProxyHttp(s) types\nfn connected_to_upstream_common(\n    reused: bool,\n    digest: Option<&Digest>,\n    ctx: &mut CTX,\n) -> Result<()> {\n    ctx.conn_reused = reused;\n    let socket_digest = digest\n        .expect(\"upstream connector digest should be set for HTTP sessions\")\n        .socket_digest\n        .as_ref()\n        .expect(\"socket digest should be set for HTTP sessions\");\n    ctx.upstream_client_addr = socket_digest.local_addr().cloned();\n    ctx.upstream_server_addr = socket_digest.peer_addr().cloned();\n\n    Ok(())\n}\n\nfn response_filter_common(\n    session: &mut Session,\n    response: &mut ResponseHeader,\n    ctx: &mut CTX,\n) -> Result<()> {\n    if ctx.conn_reused {\n        response.insert_header(\"x-conn-reuse\", \"1\")?;\n    }\n\n    let client_addr = session.client_addr();\n    let server_addr = session.server_addr();\n    response.insert_header(\n        \"x-client-addr\",\n        client_addr.map_or_else(|| \"unset\".into(), |a| a.to_string()),\n    )?;\n    response.insert_header(\n        \"x-server-addr\",\n        server_addr.map_or_else(|| \"unset\".into(), |a| a.to_string()),\n    )?;\n\n    response.insert_header(\n        \"x-upstream-client-addr\",\n        ctx.upstream_client_addr\n            .as_ref()\n            .map_or_else(|| \"unset\".into(), |a| a.to_string()),\n    )?;\n    response.insert_header(\n        \"x-upstream-server-addr\",\n        ctx.upstream_server_addr\n            .as_ref()\n            .map_or_else(|| \"unset\".into(), |a| a.to_string()),\n    )?;\n\n    Ok(())\n}\n\n#[async_trait]\n#[cfg(feature = \"any_tls\")]\nimpl ProxyHttp for ExampleProxyHttps {\n    type CTX = CTX;\n    fn new_ctx(&self) -> Self::CTX {\n        CTX::default()\n    }\n\n    async fn upstream_peer(\n        &self,\n        session: &mut Session,\n        _ctx: &mut Self::CTX,\n    ) -> Result<Box<HttpPeer>> {\n        let session = session.as_downstream();\n        let req = session.req_header();\n\n        let port = req\n            .headers\n            .get(\"x-port\")\n            .map_or(\"8443\", |v| v.to_str().unwrap());\n        let sni = req.headers.get(\"sni\").map_or(\"\", |v| v.to_str().unwrap());\n        let alt = req\n            .headers\n            .get(\"alt\")\n            .map(|v| v.to_str().unwrap().to_string());\n\n        let client_cert = session.get_header_bytes(\"client_cert\");\n\n        let mut peer = Box::new(HttpPeer::new(\n            format!(\"127.0.0.1:{port}\"),\n            true,\n            sni.to_string(),\n        ));\n        peer.options.alternative_cn = alt;\n\n        let verify = session.get_header_bytes(\"verify\") == b\"1\";\n        peer.options.verify_cert = verify;\n\n        let verify_host = session.get_header_bytes(\"verify_host\") == b\"1\";\n        peer.options.verify_hostname = verify_host;\n\n        if matches!(client_cert, b\"1\" | b\"2\") {\n            let (mut certs, key) = if client_cert == b\"1\" {\n                (vec![cert::LEAF_CERT.clone()], cert::LEAF_KEY.clone())\n            } else {\n                (vec![cert::LEAF2_CERT.clone()], cert::LEAF2_KEY.clone())\n            };\n            if session.get_header_bytes(\"client_intermediate\") == b\"1\" {\n                certs.push(cert::INTERMEDIATE_CERT.clone());\n            }\n            #[cfg(feature = \"s2n\")]\n            {\n                let combined_pem = certs.into_iter().flatten().collect();\n                peer.client_cert_key = Some(Arc::new(CertKey::new(combined_pem, key)));\n            }\n            #[cfg(not(feature = \"s2n\"))]\n            {\n                peer.client_cert_key = Some(Arc::new(CertKey::new(certs, key)));\n            }\n        }\n\n        #[cfg(feature = \"s2n\")]\n        if let Some(psk_identity) = req.headers.get(\"psk_identity\") {\n            use pingora_core::{\n                protocols::tls::{Psk, PskConfig},\n                tls::PskHmac,\n            };\n\n            let psk = Psk::new(\n                psk_identity.to_str().unwrap().to_string(),\n                TEST_PSK_SECRET.as_bytes().to_vec(),\n                PskHmac::SHA256,\n            );\n            peer.options.psk = Some(Arc::new(PskConfig::new(vec![psk])));\n        }\n\n        if session.get_header_bytes(\"x-h2\") == b\"true\" {\n            // default is 1, 1\n            peer.options.set_http_version(2, 2);\n        }\n\n        Ok(peer)\n    }\n\n    async fn response_filter(\n        &self,\n        session: &mut Session,\n        upstream_response: &mut ResponseHeader,\n        ctx: &mut Self::CTX,\n    ) -> Result<()>\n    where\n        Self::CTX: Send + Sync,\n    {\n        response_filter_common(session, upstream_response, ctx)\n    }\n\n    async fn upstream_request_filter(\n        &self,\n        session: &mut Session,\n        req: &mut RequestHeader,\n        _ctx: &mut Self::CTX,\n    ) -> Result<()> {\n        let host = session.get_header_bytes(\"host-override\");\n        if host != b\"\" {\n            req.insert_header(\"host\", host)?;\n        }\n        Ok(())\n    }\n\n    async fn connected_to_upstream(\n        &self,\n        _http_session: &mut Session,\n        reused: bool,\n        _peer: &HttpPeer,\n        #[cfg(unix)] _fd: std::os::unix::io::RawFd,\n        #[cfg(windows)] _sock: std::os::windows::io::RawSocket,\n        digest: Option<&Digest>,\n        ctx: &mut CTX,\n    ) -> Result<()> {\n        connected_to_upstream_common(reused, digest, ctx)\n    }\n}\n\npub struct ExampleProxyHttp {}\n\n#[async_trait]\nimpl ProxyHttp for ExampleProxyHttp {\n    type CTX = CTX;\n    fn new_ctx(&self) -> Self::CTX {\n        CTX::default()\n    }\n\n    async fn early_request_filter(\n        &self,\n        session: &mut Session,\n        _ctx: &mut Self::CTX,\n    ) -> Result<()> {\n        let req = session.req_header();\n        let downstream_compression = req.headers.get(\"x-downstream-compression\").is_some();\n        if downstream_compression {\n            session\n                .downstream_modules_ctx\n                .get_mut::<ResponseCompression>()\n                .unwrap()\n                .adjust_level(6);\n        } else {\n            // enable upstream compression for all requests by default\n            session.upstream_compression.adjust_level(6);\n        }\n        Ok(())\n    }\n\n    async fn request_filter(&self, session: &mut Session, _ctx: &mut Self::CTX) -> Result<bool> {\n        let req = session.req_header();\n\n        let write_timeout = req\n            .headers\n            .get(\"x-write-timeout\")\n            .and_then(|v| v.to_str().ok().and_then(|v| v.parse().ok()));\n\n        let min_rate = req\n            .headers\n            .get(\"x-min-rate\")\n            .and_then(|v| v.to_str().ok().and_then(|v| v.parse().ok()));\n\n        let close_on_response_before_downstream_finish = req\n            .headers\n            .get(\"x-close-on-response-before-downstream-finish\")\n            .is_some();\n\n        let downstream_compression = req.headers.get(\"x-downstream-compression\").is_some();\n        if !downstream_compression {\n            // enable upstream compression for all requests by default\n            session.upstream_compression.adjust_level(6);\n            // also disable downstream compression in order to test the upstream one\n            session\n                .downstream_modules_ctx\n                .get_mut::<ResponseCompression>()\n                .unwrap()\n                .adjust_level(0);\n        }\n\n        session.set_min_send_rate(min_rate);\n        session.set_write_timeout(write_timeout.map(Duration::from_secs));\n        session.set_close_on_response_before_downstream_finish(\n            close_on_response_before_downstream_finish,\n        );\n\n        Ok(false)\n    }\n\n    async fn response_filter(\n        &self,\n        session: &mut Session,\n        upstream_response: &mut ResponseHeader,\n        ctx: &mut Self::CTX,\n    ) -> Result<()> {\n        response_filter_common(session, upstream_response, ctx)\n    }\n\n    async fn upstream_peer(\n        &self,\n        session: &mut Session,\n        _ctx: &mut Self::CTX,\n    ) -> Result<Box<HttpPeer>> {\n        let req = session.req_header();\n        #[cfg(unix)]\n        if req.headers.contains_key(\"x-uds-peer\") {\n            return Ok(Box::new(HttpPeer::new_uds(\n                \"/tmp/pingora_nginx_test.sock\",\n                false,\n                \"\".to_string(),\n            )?));\n        }\n        let port = req\n            .headers\n            .get(\"x-port\")\n            .map_or(\"8000\", |v| v.to_str().unwrap());\n\n        let mut peer = Box::new(HttpPeer::new(\n            format!(\"127.0.0.1:{port}\"),\n            false,\n            \"\".to_string(),\n        ));\n\n        if session.get_header_bytes(\"x-h2\") == b\"true\" {\n            // default is 1, 1\n            peer.options.set_http_version(2, 2);\n        }\n\n        Ok(peer)\n    }\n\n    async fn connected_to_upstream(\n        &self,\n        _http_session: &mut Session,\n        reused: bool,\n        _peer: &HttpPeer,\n        #[cfg(unix)] _fd: std::os::unix::io::RawFd,\n        #[cfg(windows)] _sock: std::os::windows::io::RawSocket,\n        digest: Option<&Digest>,\n        ctx: &mut CTX,\n    ) -> Result<()> {\n        connected_to_upstream_common(reused, digest, ctx)\n    }\n}\n\nstatic CACHE_BACKEND: Lazy<MemCache> = Lazy::new(MemCache::new);\nconst CACHE_DEFAULT: CacheMetaDefaults =\n    CacheMetaDefaults::new(|_| Some(Duration::from_secs(1)), 1, 1);\nstatic CACHE_PREDICTOR: Lazy<Predictor<32>> = Lazy::new(|| Predictor::new(5, None));\nstatic EVICTION_MANAGER: Lazy<Manager> = Lazy::new(|| Manager::new(8192)); // 8192 bytes\nstatic CACHE_LOCK: Lazy<Box<CacheKeyLockImpl>> =\n    Lazy::new(|| CacheLock::new_boxed(std::time::Duration::from_secs(2)));\n// Example of how one might restrict which fields can be varied on.\nstatic CACHE_VARY_ALLOWED_HEADERS: Lazy<Option<HashSet<&str>>> =\n    Lazy::new(|| Some(vec![\"accept\", \"accept-encoding\"].into_iter().collect()));\n\n// #[allow(clippy::upper_case_acronyms)]\npub struct CacheCTX {\n    upstream_status: Option<u16>,\n}\n\npub struct ExampleProxyCache {}\n\n#[async_trait]\nimpl ProxyHttp for ExampleProxyCache {\n    type CTX = CacheCTX;\n    fn new_ctx(&self) -> Self::CTX {\n        CacheCTX {\n            upstream_status: None,\n        }\n    }\n\n    async fn early_request_filter(\n        &self,\n        session: &mut Session,\n        _ctx: &mut Self::CTX,\n    ) -> Result<()> {\n        if session\n            .req_header()\n            .headers\n            .get(\"x-downstream-compression\")\n            .is_some()\n        {\n            session\n                .downstream_modules_ctx\n                .get_mut::<ResponseCompression>()\n                .unwrap()\n                .adjust_level(6);\n        }\n        if session\n            .req_header()\n            .headers\n            .get(\"x-downstream-decompression\")\n            .is_some()\n        {\n            session\n                .downstream_modules_ctx\n                .get_mut::<ResponseCompression>()\n                .unwrap()\n                .adjust_decompression(true);\n        }\n        Ok(())\n    }\n\n    async fn upstream_peer(\n        &self,\n        session: &mut Session,\n        _ctx: &mut Self::CTX,\n    ) -> Result<Box<HttpPeer>> {\n        let req = session.req_header();\n        let port = req\n            .headers\n            .get(\"x-port\")\n            .map_or(\"8000\", |v| v.to_str().unwrap());\n\n        let mut peer = Box::new(HttpPeer::new(\n            format!(\"127.0.0.1:{}\", port),\n            false,\n            \"\".to_string(),\n        ));\n\n        if session.get_header_bytes(\"x-h2\") == b\"true\" {\n            // default is 1, 1\n            peer.options.set_http_version(2, 2);\n        }\n\n        Ok(peer)\n    }\n\n    fn request_cache_filter(&self, session: &mut Session, _ctx: &mut Self::CTX) -> Result<()> {\n        // TODO: only allow GET & HEAD\n\n        if session.get_header_bytes(\"x-bypass-cache\") != b\"\" {\n            return Ok(());\n        }\n\n        // turn on eviction only for some requests to avoid interference across tests\n        let eviction = session.req_header().headers.get(\"x-eviction\").map(|_| {\n            &*EVICTION_MANAGER as &'static (dyn pingora_cache::eviction::EvictionManager + Sync)\n        });\n        let lock = session\n            .req_header()\n            .headers\n            .get(\"x-lock\")\n            .map(|_| CACHE_LOCK.as_ref());\n        let mut overrides = CacheOptionOverrides::default();\n        overrides.wait_timeout = Some(Duration::from_secs(2));\n        session.cache.enable(\n            &*CACHE_BACKEND,\n            eviction,\n            Some(&*CACHE_PREDICTOR),\n            lock,\n            Some(overrides),\n        );\n\n        if let Some(max_file_size_hdr) = session\n            .req_header()\n            .headers\n            .get(\"x-cache-max-file-size-bytes\")\n        {\n            let bytes = max_file_size_hdr\n                .to_str()\n                .unwrap()\n                .parse::<usize>()\n                .unwrap();\n            session.cache.set_max_file_size_bytes(bytes);\n        }\n\n        Ok(())\n    }\n\n    /// Reference `cache_key_callback` implementation for integration tests.\n    ///\n    /// Builds the primary key as `{host}{path_and_query}` from the request.\n    /// This is **not production ready**: it does not account for `Vary`, custom\n    /// request filters, or scheme differences. See the rustdoc on\n    /// [`ProxyHttp::cache_key_callback`] for details.\n    fn cache_key_callback(&self, session: &Session, _ctx: &mut Self::CTX) -> Result<CacheKey> {\n        let req_header = session.req_header();\n\n        let host = req_header\n            .headers\n            .get(http::header::HOST)\n            .and_then(|v| v.to_str().ok())\n            .or_else(|| req_header.uri.authority().map(|a| a.as_str()))\n            .unwrap_or(\"\");\n\n        let path_and_query = req_header\n            .uri\n            .path_and_query()\n            .map(|pq| pq.as_str())\n            .unwrap_or(\"/\");\n\n        Ok(CacheKey::new(\n            String::new(),\n            format!(\"{host}{path_and_query}\"),\n            String::new(),\n        ))\n    }\n\n    async fn cache_hit_filter(\n        &self,\n        session: &mut Session,\n        _meta: &CacheMeta,\n        _hit_handler: &mut HitHandler,\n        is_fresh: bool,\n        _ctx: &mut Self::CTX,\n    ) -> Result<Option<ForcedFreshness>> {\n        // allow test header to control force expiry/miss\n        if session.get_header_bytes(\"x-force-miss\") != b\"\" {\n            return Ok(Some(ForcedFreshness::ForceMiss));\n        }\n\n        if !is_fresh {\n            if session.get_header_bytes(\"x-force-fresh\") != b\"\" {\n                return Ok(Some(ForcedFreshness::ForceFresh));\n            }\n            // already expired\n            return Ok(None);\n        }\n\n        if session.get_header_bytes(\"x-force-expire\") != b\"\" {\n            return Ok(Some(ForcedFreshness::ForceExpired));\n        }\n        Ok(None)\n    }\n\n    fn cache_vary_filter(\n        &self,\n        meta: &CacheMeta,\n        _ctx: &mut Self::CTX,\n        req: &RequestHeader,\n    ) -> Option<HashBinary> {\n        let mut key = VarianceBuilder::new();\n\n        // Vary per header from origin. Target headers are de-duplicated by key logic.\n        let vary_headers_lowercased: Vec<String> = meta\n            .headers()\n            .get_all(VARY)\n            .iter()\n            // Filter out any unparseable vary headers.\n            .flat_map(|vary_header| vary_header.to_str().ok())\n            .flat_map(|vary_header| vary_header.split(','))\n            .map(|s| s.trim().to_lowercase())\n            .filter(|header_name| {\n                // Filter only for allowed headers, if restricted.\n                CACHE_VARY_ALLOWED_HEADERS\n                    .as_ref()\n                    .map(|al| al.contains(header_name.as_str()))\n                    .unwrap_or(true)\n            })\n            .collect();\n\n        vary_headers_lowercased.iter().for_each(|header_name| {\n            // Add this header and value to be considered in the variance key.\n            key.add_value(\n                header_name,\n                req.headers\n                    .get(header_name)\n                    .map(|v| v.as_bytes())\n                    .unwrap_or(&[]),\n            );\n        });\n\n        key.finalize()\n    }\n\n    async fn upstream_request_filter(\n        &self,\n        session: &mut Session,\n        upstream_request: &mut RequestHeader,\n        _ctx: &mut Self::CTX,\n    ) -> Result<()> {\n        if let Some(up_accept_encoding) = session\n            .req_header()\n            .headers\n            .get(\"x-upstream-accept-encoding\")\n        {\n            upstream_request.insert_header(&ACCEPT_ENCODING, up_accept_encoding)?;\n        }\n        Ok(())\n    }\n\n    fn response_cache_filter(\n        &self,\n        session: &Session,\n        resp: &ResponseHeader,\n        _ctx: &mut Self::CTX,\n    ) -> Result<RespCacheable> {\n        // Allow testing the unlikely case of caching a 101 response\n        if resp.status == 101\n            && session\n                .req_header()\n                .headers\n                .contains_key(\"x-cache-websocket\")\n        {\n            return Ok(RespCacheable::Cacheable(CacheMeta::new(\n                SystemTime::now() + Duration::from_secs(5),\n                SystemTime::now(),\n                0,\n                0,\n                resp.clone(),\n            )));\n        }\n\n        let cc = CacheControl::from_resp_headers(resp);\n        Ok(resp_cacheable(\n            cc.as_ref(),\n            resp.clone(),\n            false,\n            &CACHE_DEFAULT,\n        ))\n    }\n\n    async fn upstream_response_filter(\n        &self,\n        session: &mut Session,\n        upstream_response: &mut ResponseHeader,\n        ctx: &mut Self::CTX,\n    ) -> Result<()> {\n        ctx.upstream_status = Some(upstream_response.status.into());\n        if session\n            .req_header()\n            .headers\n            .contains_key(\"x-upstream-fake-http10\")\n        {\n            // TODO to simulate an actual http1.0 origin\n            upstream_response.set_version(http::Version::HTTP_10);\n            upstream_response.remove_header(&CONTENT_LENGTH);\n            upstream_response.remove_header(&TRANSFER_ENCODING);\n        }\n        Ok(())\n    }\n\n    async fn response_filter(\n        &self,\n        session: &mut Session,\n        upstream_response: &mut ResponseHeader,\n        ctx: &mut Self::CTX,\n    ) -> Result<()>\n    where\n        Self::CTX: Send + Sync,\n    {\n        if session.cache.enabled() {\n            match session.cache.phase() {\n                CachePhase::Hit => upstream_response.insert_header(\"x-cache-status\", \"hit\")?,\n                CachePhase::Miss => upstream_response.insert_header(\"x-cache-status\", \"miss\")?,\n                CachePhase::Stale => upstream_response.insert_header(\"x-cache-status\", \"stale\")?,\n                CachePhase::StaleUpdating => {\n                    upstream_response.insert_header(\"x-cache-status\", \"stale-updating\")?\n                }\n                CachePhase::Expired => {\n                    upstream_response.insert_header(\"x-cache-status\", \"expired\")?\n                }\n                CachePhase::Revalidated | CachePhase::RevalidatedNoCache(_) => {\n                    upstream_response.insert_header(\"x-cache-status\", \"revalidated\")?\n                }\n                _ => upstream_response.insert_header(\"x-cache-status\", \"invalid\")?,\n            }\n        } else {\n            match session.cache.phase() {\n                CachePhase::Disabled(NoCacheReason::Deferred) => {\n                    upstream_response.insert_header(\"x-cache-status\", \"deferred\")?;\n                }\n                _ => upstream_response.insert_header(\"x-cache-status\", \"no-cache\")?,\n            }\n        }\n        if let Some(d) = session.cache.lock_duration() {\n            upstream_response.insert_header(\"x-cache-lock-time-ms\", format!(\"{}\", d.as_millis()))?\n        }\n        if let Some(up_stat) = ctx.upstream_status {\n            upstream_response.insert_header(\"x-upstream-status\", up_stat.to_string())?;\n        }\n        Ok(())\n    }\n\n    async fn fail_to_proxy(\n        &self,\n        session: &mut Session,\n        e: &Error,\n        _ctx: &mut Self::CTX,\n    ) -> FailToProxy\n    where\n        Self::CTX: Send + Sync,\n    {\n        // default OSS fail_to_proxy with added headers\n        let code = match e.etype() {\n            HTTPStatus(code) => *code,\n            _ => {\n                match e.esource() {\n                    ErrorSource::Upstream => 502,\n                    ErrorSource::Downstream => {\n                        match e.etype() {\n                            WriteError | ReadError | ConnectionClosed => {\n                                /* conn already dead */\n                                0\n                            }\n                            _ => 400,\n                        }\n                    }\n                    ErrorSource::Internal | ErrorSource::Unset => 500,\n                }\n            }\n        };\n        if code > 0 {\n            let mut resp = gen_error_response(code);\n            // any relevant metadata headers to add\n            if let Some(d) = session.cache.lock_duration() {\n                resp.insert_header(\"x-cache-lock-time-ms\", format!(\"{}\", d.as_millis()))\n                    .unwrap();\n            }\n            session\n                .write_response_header(Box::new(resp), true)\n                .await\n                .unwrap_or_else(|e| {\n                    error!(\"failed to send error response to downstream: {e}\");\n                });\n        }\n\n        FailToProxy {\n            error_code: code,\n            // default to no reuse, which is safest\n            can_reuse_downstream: false,\n        }\n    }\n\n    fn should_serve_stale(\n        &self,\n        _session: &mut Session,\n        _ctx: &mut Self::CTX,\n        error: Option<&Error>, // None when it is called during stale while revalidate\n    ) -> bool {\n        // enable serve stale while updating\n        error.is_none_or(|e| e.esource() == &ErrorSource::Upstream)\n    }\n\n    fn is_purge(&self, session: &Session, _ctx: &Self::CTX) -> bool {\n        session.req_header().method == \"PURGE\"\n    }\n}\n\nfn test_main() {\n    env_logger::Builder::from_env(env_logger::Env::default().default_filter_or(\"info\")).init();\n\n    let opts: Vec<String> = vec![\n        \"pingora-proxy\".into(),\n        \"-c\".into(),\n        \"tests/pingora_conf.yaml\".into(),\n    ];\n    let mut my_server =\n        pingora_core::server::Server::new(Some(Opt::parse_from_args(opts))).unwrap();\n    my_server.bootstrap();\n\n    let mut proxy_service_http =\n        pingora_proxy::http_proxy_service(&my_server.configuration, ExampleProxyHttp {});\n    proxy_service_http.add_tcp(\"0.0.0.0:6147\");\n    #[cfg(unix)]\n    proxy_service_http.add_uds(\"/tmp/pingora_proxy.sock\", None);\n\n    let mut proxy_service_http_connect =\n        pingora_proxy::http_proxy_service(&my_server.configuration, ExampleProxyHttp {});\n    let http_logic = proxy_service_http_connect.app_logic_mut().unwrap();\n    let mut http_server_options = HttpServerOptions::default();\n    http_server_options.allow_connect_method_proxying = true;\n    http_logic.server_options = Some(http_server_options);\n    proxy_service_http_connect.add_tcp(\"0.0.0.0:6160\");\n\n    let mut proxy_service_h2c =\n        pingora_proxy::http_proxy_service(&my_server.configuration, ExampleProxyHttp {});\n\n    let http_logic = proxy_service_h2c.app_logic_mut().unwrap();\n    let mut http_server_options = HttpServerOptions::default();\n    http_server_options.h2c = true;\n    http_logic.server_options = Some(http_server_options);\n    proxy_service_h2c.add_tcp(\"0.0.0.0:6146\");\n\n    let mut proxy_service_https_opt: Option<Box<dyn ServiceWithDependents>> = None;\n\n    #[cfg(feature = \"any_tls\")]\n    {\n        let mut proxy_service_https =\n            pingora_proxy::http_proxy_service(&my_server.configuration, ExampleProxyHttps {});\n        proxy_service_https.add_tcp(\"0.0.0.0:6149\");\n        let cert_path = format!(\"{}/tests/keys/server.crt\", env!(\"CARGO_MANIFEST_DIR\"));\n        let key_path = format!(\"{}/tests/keys/key.pem\", env!(\"CARGO_MANIFEST_DIR\"));\n        let mut tls_settings =\n            pingora_core::listeners::tls::TlsSettings::intermediate(&cert_path, &key_path).unwrap();\n        tls_settings.enable_h2();\n        proxy_service_https.add_tls_with_settings(\"0.0.0.0:6150\", None, tls_settings);\n        proxy_service_https_opt = Some(Box::new(proxy_service_https))\n    }\n\n    let mut proxy_service_cache =\n        pingora_proxy::http_proxy_service(&my_server.configuration, ExampleProxyCache {});\n    proxy_service_cache.add_tcp(\"0.0.0.0:6148\");\n\n    #[cfg(feature = \"any_tls\")]\n    {\n        let cert_path = format!(\"{}/tests/keys/server.crt\", env!(\"CARGO_MANIFEST_DIR\"));\n        let key_path = format!(\"{}/tests/keys/key.pem\", env!(\"CARGO_MANIFEST_DIR\"));\n\n        let mut tls_settings =\n            pingora_core::listeners::tls::TlsSettings::intermediate(&cert_path, &key_path).unwrap();\n        tls_settings.enable_h2();\n        proxy_service_cache.add_tls_with_settings(\"0.0.0.0:6153\", None, tls_settings);\n    }\n\n    let mut services: Vec<Box<dyn ServiceWithDependents>> = vec![\n        Box::new(proxy_service_h2c),\n        Box::new(proxy_service_http),\n        Box::new(proxy_service_http_connect),\n        Box::new(proxy_service_cache),\n    ];\n\n    if let Some(proxy_service_https) = proxy_service_https_opt {\n        services.push(proxy_service_https)\n    }\n\n    set_compression_dict_path(\"tests/headers.dict\");\n    my_server.add_services(services);\n    my_server.run_forever();\n}\n\npub struct Server {\n    pub handle: thread::JoinHandle<()>,\n}\n\nimpl Server {\n    pub fn start() -> Self {\n        let server_handle = thread::spawn(|| {\n            test_main();\n        });\n        Server {\n            handle: server_handle,\n        }\n    }\n}\n\n#[cfg(feature = \"s2n\")]\npub struct PskTlsServer {\n    pub handle: thread::JoinHandle<()>,\n}\n\n#[cfg(feature = \"s2n\")]\nimpl PskTlsServer {\n    pub fn start() -> Self {\n        let server_handle = thread::spawn(|| {\n            let rt = tokio::runtime::Runtime::new().unwrap();\n            rt.block_on(Self::run_server());\n        });\n        PskTlsServer {\n            handle: server_handle,\n        }\n    }\n\n    async fn run_server() {\n        use pingora_core::{protocols::tls::S2NConnectionBuilder, tls::TlsAcceptor};\n        use pingora_core::{\n            protocols::tls::{Psk, PskConfig, PskType},\n            tls::{Config, PskHmac, S2NPolicy, DEFAULT_TLS13},\n        };\n        use tokio::net::TcpListener;\n\n        let psk = Psk::new(\n            TEST_PSK_IDENTITY.to_string(),\n            TEST_PSK_SECRET.as_bytes().to_vec(),\n            PskHmac::SHA256,\n        );\n        let psk_config = Arc::new(PskConfig::new(vec![psk]));\n\n        let addr: std::net::SocketAddr = \"127.0.0.1:6151\".parse().unwrap();\n        let listener = TcpListener::bind(addr).await.unwrap();\n        let mut config_builder = Config::builder();\n        unsafe {\n            config_builder.disable_x509_verification();\n        }\n        config_builder.set_security_policy(&DEFAULT_TLS13).unwrap();\n        let config = config_builder.build().unwrap();\n\n        let connection_builder = S2NConnectionBuilder {\n            config: config.clone(),\n            psk_config: Some(psk_config.clone()),\n            security_policy: None,\n        };\n\n        let acceptor = TlsAcceptor::new(connection_builder);\n\n        loop {\n            use tokio::{io::AsyncWriteExt, net::tcp};\n            let (tcp_stream, _) = listener.accept().await.unwrap();\n            let mut stream = acceptor.clone().accept(tcp_stream).await.unwrap();\n            let response = b\"HTTP/1.1 200 OK\\r\\nContent-Length: 5\\r\\n\\r\\nhello\";\n            stream.write(response).await.unwrap();\n            stream.shutdown().await;\n        }\n    }\n}\n\n// FIXME: this still allows multiple servers to spawn across integration tests\npub static TEST_SERVER: Lazy<Server> = Lazy::new(Server::start);\n#[cfg(feature = \"s2n\")]\npub static TEST_PSK_TLS_SERVER: Lazy<PskTlsServer> = Lazy::new(PskTlsServer::start);\nuse super::mock_origin::MOCK_ORIGIN;\n\npub fn init() {\n    let _ = *TEST_SERVER;\n    let _ = *MOCK_ORIGIN;\n    #[cfg(feature = \"s2n\")]\n    let _ = *TEST_PSK_TLS_SERVER;\n}\n"
  },
  {
    "path": "pingora-proxy/tests/utils/websocket/mod.rs",
    "content": "mod ws_echo;\nmod ws_echo_raw;\n\npub use ws_echo::WS_ECHO;\npub use ws_echo_raw::WS_ECHO_RAW;\n"
  },
  {
    "path": "pingora-proxy/tests/utils/websocket/ws_echo.rs",
    "content": "// Copyright 2025 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\nuse std::{io::Error, thread, time::Duration};\n\nuse futures_util::{SinkExt, StreamExt};\nuse log::debug;\nuse std::sync::LazyLock;\nuse tokio::{\n    net::{TcpListener, TcpStream},\n    runtime::Builder,\n};\n\npub static WS_ECHO: LazyLock<bool> = LazyLock::new(init);\npub const WS_ECHO_ORIGIN_PORT: u16 = 9283;\n\nfn init() -> bool {\n    thread::spawn(move || {\n        let runtime = Builder::new_current_thread()\n            .thread_name(\"websocket echo\")\n            .enable_all()\n            .build()\n            .unwrap();\n        runtime.block_on(async move {\n            server(&format!(\"127.0.0.1:{WS_ECHO_ORIGIN_PORT}\"))\n                .await\n                .unwrap();\n        })\n    });\n    thread::sleep(Duration::from_millis(200));\n    true\n}\n\nasync fn server(addr: &str) -> Result<(), Error> {\n    let listener = TcpListener::bind(&addr).await.unwrap();\n    while let Ok((stream, _)) = listener.accept().await {\n        tokio::spawn(handle_connection(stream));\n    }\n    Ok(())\n}\n\nasync fn handle_connection(stream: TcpStream) {\n    let mut ws_stream = tokio_tungstenite::accept_async(stream).await.unwrap();\n\n    while let Some(msg) = ws_stream.next().await {\n        let msg = msg.unwrap();\n        let echo = msg.clone();\n        if msg.is_text() {\n            let data = msg.into_text().unwrap();\n            if data.contains(\"close\") {\n                // abruptly close the stream without WS close;\n                debug!(\"abrupt close\");\n                return;\n            } else if data.contains(\"graceful\") {\n                debug!(\"graceful close\");\n                ws_stream.close(None).await.unwrap();\n                // close() only sends frame\n                return;\n            } else {\n                ws_stream.send(echo).await.unwrap();\n            }\n        }\n    }\n}\n"
  },
  {
    "path": "pingora-proxy/tests/utils/websocket/ws_echo_raw.rs",
    "content": "// Copyright 2025 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\nuse std::{thread, time::Duration};\n\nuse futures_util::{SinkExt, StreamExt};\nuse log::debug;\nuse pingora_error::{Error, ErrorType::*, OrErr, Result};\nuse pingora_http::RequestHeader;\nuse std::sync::LazyLock;\nuse tokio::{\n    io::{AsyncReadExt, AsyncWriteExt},\n    net::{\n        tcp::{OwnedReadHalf, OwnedWriteHalf},\n        TcpListener, TcpStream,\n    },\n    runtime::Builder,\n};\n\npub static WS_ECHO_RAW: LazyLock<bool> = LazyLock::new(init);\npub const WS_ECHO_RAW_ORIGIN_PORT: u16 = 9284;\n\nfn init() -> bool {\n    thread::spawn(move || {\n        let runtime = Builder::new_current_thread()\n            .thread_name(\"websocket raw echo\")\n            .enable_all()\n            .build()\n            .unwrap();\n        runtime.block_on(async move {\n            server(&format!(\"127.0.0.1:{WS_ECHO_RAW_ORIGIN_PORT}\"))\n                .await\n                .unwrap();\n        })\n    });\n    thread::sleep(Duration::from_millis(200));\n    true\n}\n\nasync fn server(addr: &str) -> Result<(), Error> {\n    let listener = TcpListener::bind(&addr).await.unwrap();\n    while let Ok((stream, _)) = listener.accept().await {\n        tokio::spawn(handle_connection(stream));\n    }\n    Ok(())\n}\n\nasync fn read_request_header(stream: &mut TcpStream) -> Result<(RequestHeader, Vec<u8>)> {\n    fn parse_request_header(buf: &[u8]) -> Result<RequestHeader> {\n        let mut headers = vec![httparse::EMPTY_HEADER; 256];\n        let mut parsed = httparse::Request::new(&mut headers);\n        match parsed\n            .parse(buf)\n            .or_err(ReadError, \"request header parse error\")?\n        {\n            httparse::Status::Complete(_) => {\n                let mut req = RequestHeader::build(\n                    parsed.method.unwrap_or(\"\"),\n                    parsed.path.unwrap_or(\"\").as_bytes(),\n                    Some(parsed.headers.len()),\n                )?;\n                for header in parsed.headers.iter() {\n                    req.append_header(header.name.to_string(), header.value)\n                        .unwrap();\n                }\n                Ok(req)\n            }\n            _ => Error::e_explain(ReadError, \"should have full request header\"),\n        }\n    }\n\n    let mut request = vec![];\n    let mut header_end = 0;\n    let mut buf = [0; 1024];\n    loop {\n        let n = stream\n            .read(&mut buf)\n            .await\n            .or_err(ReadError, \"while reading request header\")?;\n        request.extend_from_slice(&buf[..n]);\n        let mut end_of_header = false;\n        for (i, w) in request.windows(4).enumerate() {\n            if w == b\"\\r\\n\\r\\n\" {\n                end_of_header = true;\n                header_end = i + 4;\n                break;\n            }\n        }\n        if end_of_header {\n            break;\n        }\n    }\n    Ok((\n        parse_request_header(&request[..header_end])?,\n        request[header_end..].to_vec(),\n    ))\n}\n\nasync fn read_body_until_close(\n    stream: &mut OwnedReadHalf,\n) -> Result<Option<Vec<u8>>, std::io::Error> {\n    let mut buf = [0; 1024];\n    let n = stream.read(&mut buf).await?;\n    if n == 0 {\n        return Ok(None);\n    }\n    Ok(Some(buf[..n].to_vec()))\n}\n\nasync fn write_body_until_close(\n    stream: &mut OwnedWriteHalf,\n    body: &[u8],\n) -> Result<Option<usize>, std::io::Error> {\n    let n = stream.write(body).await?;\n    Ok((n != 0).then_some(n))\n}\n\nasync fn handle_connection(mut stream: TcpStream) -> Result<()> {\n    let (header, preread_body) = read_request_header(&mut stream).await?;\n\n    // if x-expected-body-len unset, continue to read until stream is closed\n    let expected_body_len = header\n        .headers\n        .get(\"x-expected-body-len\")\n        .and_then(|v| std::str::from_utf8(v.as_bytes()).ok())\n        .and_then(|s| s.parse().ok());\n\n    let resp_raw =\n        b\"HTTP/1.1 101 Switching Protocols\\r\\nConnection: upgrade\\r\\nUpgrade: websocket\\r\\n\\r\\n\";\n    stream\n        .write_all(resp_raw)\n        .await\n        .or_err(WriteError, \"while writing 101\")?;\n\n    let (mut stream_read, mut stream_write) = stream.into_split();\n    let mut request_body = preread_body;\n    let mut body_read = request_body.len();\n    let mut body_read_done = false;\n\n    loop {\n        tokio::select! {\n            res = read_body_until_close(&mut stream_read), if !body_read_done => {\n                let Some(buf) = res.or_err(ReadError, \"while reading body\")? else {\n                    return Ok(());\n                };\n                body_read += buf.len();\n                body_read_done = expected_body_len.is_some_and(|len| body_read >= len);\n                request_body.extend_from_slice(&buf[..]);\n            }\n            res = write_body_until_close(&mut stream_write, &request_body[..]), if !request_body.is_empty() => {\n                let Some(n) = res.or_err(WriteError, \"while writing body\")? else {\n                    return Ok(());\n                };\n                request_body = request_body[n..].to_vec();\n            }\n            else => break,\n        }\n    }\n    if let Some(expected) = expected_body_len {\n        if body_read > expected {\n            return Error::e_explain(ReadError, \"read {body_read} bytes, expected {expected}\");\n        }\n    }\n    Ok(())\n}\n"
  },
  {
    "path": "pingora-runtime/Cargo.toml",
    "content": "[package]\nname = \"pingora-runtime\"\nversion = \"0.8.0\"\nauthors = [\"Yuchen Wu <yuchen@cloudflare.com>\"]\nlicense = \"Apache-2.0\"\nedition = \"2021\"\nrepository = \"https://github.com/cloudflare/pingora\"\ncategories = [\"asynchronous\", \"network-programming\"]\nkeywords = [\"async\", \"non-blocking\", \"pingora\"]\ndescription = \"\"\"\nMultithreaded Tokio runtime with the option of disabling work stealing.\n\"\"\"\n\n# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html\n[lib]\nname = \"pingora_runtime\"\npath = \"src/lib.rs\"\n\n[dependencies]\nrand = \"0.8\"\ntokio = { workspace = true, features = [\"rt-multi-thread\", \"sync\", \"time\"] }\nonce_cell = { workspace = true }\nthread_local = \"1\"\n\n[dev-dependencies]\ntokio = { workspace = true, features = [\"io-util\", \"net\"] }\n\n[[bench]]\nname = \"hello\"\nharness = false\n"
  },
  {
    "path": "pingora-runtime/LICENSE",
    "content": "\n                                 Apache License\n                           Version 2.0, January 2004\n                        http://www.apache.org/licenses/\n\n   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION\n\n   1. Definitions.\n\n      \"License\" shall mean the terms and conditions for use, reproduction,\n      and distribution as defined by Sections 1 through 9 of this document.\n\n      \"Licensor\" shall mean the copyright owner or entity authorized by\n      the copyright owner that is granting the License.\n\n      \"Legal Entity\" shall mean the union of the acting entity and all\n      other entities that control, are controlled by, or are under common\n      control with that entity. For the purposes of this definition,\n      \"control\" means (i) the power, direct or indirect, to cause the\n      direction or management of such entity, whether by contract or\n      otherwise, or (ii) ownership of fifty percent (50%) or more of the\n      outstanding shares, or (iii) beneficial ownership of such entity.\n\n      \"You\" (or \"Your\") shall mean an individual or Legal Entity\n      exercising permissions granted by this License.\n\n      \"Source\" form shall mean the preferred form for making modifications,\n      including but not limited to software source code, documentation\n      source, and configuration files.\n\n      \"Object\" form shall mean any form resulting from mechanical\n      transformation or translation of a Source form, including but\n      not limited to compiled object code, generated documentation,\n      and conversions to other media types.\n\n      \"Work\" shall mean the work of authorship, whether in Source or\n      Object form, made available under the License, as indicated by a\n      copyright notice that is included in or attached to the work\n      (an example is provided in the Appendix below).\n\n      \"Derivative Works\" shall mean any work, whether in Source or Object\n      form, that is based on (or derived from) the Work and for which the\n      editorial revisions, annotations, elaborations, or other modifications\n      represent, as a whole, an original work of authorship. For the purposes\n      of this License, Derivative Works shall not include works that remain\n      separable from, or merely link (or bind by name) to the interfaces of,\n      the Work and Derivative Works thereof.\n\n      \"Contribution\" shall mean any work of authorship, including\n      the original version of the Work and any modifications or additions\n      to that Work or Derivative Works thereof, that is intentionally\n      submitted to Licensor for inclusion in the Work by the copyright owner\n      or by an individual or Legal Entity authorized to submit on behalf of\n      the copyright owner. For the purposes of this definition, \"submitted\"\n      means any form of electronic, verbal, or written communication sent\n      to the Licensor or its representatives, including but not limited to\n      communication on electronic mailing lists, source code control systems,\n      and issue tracking systems that are managed by, or on behalf of, the\n      Licensor for the purpose of discussing and improving the Work, but\n      excluding communication that is conspicuously marked or otherwise\n      designated in writing by the copyright owner as \"Not a Contribution.\"\n\n      \"Contributor\" shall mean Licensor and any individual or Legal Entity\n      on behalf of whom a Contribution has been received by Licensor and\n      subsequently incorporated within the Work.\n\n   2. Grant of Copyright License. Subject to the terms and conditions of\n      this License, each Contributor hereby grants to You a perpetual,\n      worldwide, non-exclusive, no-charge, royalty-free, irrevocable\n      copyright license to reproduce, prepare Derivative Works of,\n      publicly display, publicly perform, sublicense, and distribute the\n      Work and such Derivative Works in Source or Object form.\n\n   3. Grant of Patent License. Subject to the terms and conditions of\n      this License, each Contributor hereby grants to You a perpetual,\n      worldwide, non-exclusive, no-charge, royalty-free, irrevocable\n      (except as stated in this section) patent license to make, have made,\n      use, offer to sell, sell, import, and otherwise transfer the Work,\n      where such license applies only to those patent claims licensable\n      by such Contributor that are necessarily infringed by their\n      Contribution(s) alone or by combination of their Contribution(s)\n      with the Work to which such Contribution(s) was submitted. If You\n      institute patent litigation against any entity (including a\n      cross-claim or counterclaim in a lawsuit) alleging that the Work\n      or a Contribution incorporated within the Work constitutes direct\n      or contributory patent infringement, then any patent licenses\n      granted to You under this License for that Work shall terminate\n      as of the date such litigation is filed.\n\n   4. Redistribution. You may reproduce and distribute copies of the\n      Work or Derivative Works thereof in any medium, with or without\n      modifications, and in Source or Object form, provided that You\n      meet the following conditions:\n\n      (a) You must give any other recipients of the Work or\n          Derivative Works a copy of this License; and\n\n      (b) You must cause any modified files to carry prominent notices\n          stating that You changed the files; and\n\n      (c) You must retain, in the Source form of any Derivative Works\n          that You distribute, all copyright, patent, trademark, and\n          attribution notices from the Source form of the Work,\n          excluding those notices that do not pertain to any part of\n          the Derivative Works; and\n\n      (d) If the Work includes a \"NOTICE\" text file as part of its\n          distribution, then any Derivative Works that You distribute must\n          include a readable copy of the attribution notices contained\n          within such NOTICE file, excluding those notices that do not\n          pertain to any part of the Derivative Works, in at least one\n          of the following places: within a NOTICE text file distributed\n          as part of the Derivative Works; within the Source form or\n          documentation, if provided along with the Derivative Works; or,\n          within a display generated by the Derivative Works, if and\n          wherever such third-party notices normally appear. The contents\n          of the NOTICE file are for informational purposes only and\n          do not modify the License. You may add Your own attribution\n          notices within Derivative Works that You distribute, alongside\n          or as an addendum to the NOTICE text from the Work, provided\n          that such additional attribution notices cannot be construed\n          as modifying the License.\n\n      You may add Your own copyright statement to Your modifications and\n      may provide additional or different license terms and conditions\n      for use, reproduction, or distribution of Your modifications, or\n      for any such Derivative Works as a whole, provided Your use,\n      reproduction, and distribution of the Work otherwise complies with\n      the conditions stated in this License.\n\n   5. Submission of Contributions. Unless You explicitly state otherwise,\n      any Contribution intentionally submitted for inclusion in the Work\n      by You to the Licensor shall be under the terms and conditions of\n      this License, without any additional terms or conditions.\n      Notwithstanding the above, nothing herein shall supersede or modify\n      the terms of any separate license agreement you may have executed\n      with Licensor regarding such Contributions.\n\n   6. Trademarks. This License does not grant permission to use the trade\n      names, trademarks, service marks, or product names of the Licensor,\n      except as required for reasonable and customary use in describing the\n      origin of the Work and reproducing the content of the NOTICE file.\n\n   7. Disclaimer of Warranty. Unless required by applicable law or\n      agreed to in writing, Licensor provides the Work (and each\n      Contributor provides its Contributions) on an \"AS IS\" BASIS,\n      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or\n      implied, including, without limitation, any warranties or conditions\n      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A\n      PARTICULAR PURPOSE. You are solely responsible for determining the\n      appropriateness of using or redistributing the Work and assume any\n      risks associated with Your exercise of permissions under this License.\n\n   8. Limitation of Liability. In no event and under no legal theory,\n      whether in tort (including negligence), contract, or otherwise,\n      unless required by applicable law (such as deliberate and grossly\n      negligent acts) or agreed to in writing, shall any Contributor be\n      liable to You for damages, including any direct, indirect, special,\n      incidental, or consequential damages of any character arising as a\n      result of this License or out of the use or inability to use the\n      Work (including but not limited to damages for loss of goodwill,\n      work stoppage, computer failure or malfunction, or any and all\n      other commercial damages or losses), even if such Contributor\n      has been advised of the possibility of such damages.\n\n   9. Accepting Warranty or Additional Liability. While redistributing\n      the Work or Derivative Works thereof, You may choose to offer,\n      and charge a fee for, acceptance of support, warranty, indemnity,\n      or other liability obligations and/or rights consistent with this\n      License. However, in accepting such obligations, You may act only\n      on Your own behalf and on Your sole responsibility, not on behalf\n      of any other Contributor, and only if You agree to indemnify,\n      defend, and hold each Contributor harmless for any liability\n      incurred by, or claims asserted against, such Contributor by reason\n      of your accepting any such warranty or additional liability.\n\n   END OF TERMS AND CONDITIONS\n\n   APPENDIX: How to apply the Apache License to your work.\n\n      To apply the Apache License to your work, attach the following\n      boilerplate notice, with the fields enclosed by brackets \"[]\"\n      replaced with your own identifying information. (Don't include\n      the brackets!)  The text should be enclosed in the appropriate\n      comment syntax for the file format. We also recommend that a\n      file or class name and description of purpose be included on the\n      same \"printed page\" as the copyright notice for easier\n      identification within third-party archives.\n\n   Copyright [yyyy] [name of copyright owner]\n\n   Licensed under the Apache License, Version 2.0 (the \"License\");\n   you may not use this file except in compliance with the License.\n   You may obtain a copy of the License at\n\n       http://www.apache.org/licenses/LICENSE-2.0\n\n   Unless required by applicable law or agreed to in writing, software\n   distributed under the License is distributed on an \"AS IS\" BASIS,\n   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n   See the License for the specific language governing permissions and\n   limitations under the License.\n"
  },
  {
    "path": "pingora-runtime/benches/hello.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! Pingora tokio runtime.\n//!\n//! Tokio runtime comes in two flavors: a single-threaded runtime\n//! and a multi-threaded one which provides work stealing.\n//! Benchmark shows that, compared to the single-threaded runtime, the multi-threaded one\n//! has some overhead due to its more sophisticated work steal scheduling.\n//!\n//! This crate provides a third flavor: a multi-threaded runtime without work stealing.\n//! This flavor is as efficient as the single-threaded runtime while allows the async\n//! program to use multiple cores.\n\nuse pingora_runtime::{current_handle, Runtime};\nuse std::error::Error;\nuse std::{thread, time};\nuse tokio::io::{AsyncReadExt, AsyncWriteExt};\nuse tokio::net::TcpListener;\n\nasync fn hello_server(port: usize) -> Result<(), Box<dyn Error + Send>> {\n    let addr = format!(\"127.0.0.1:{port}\");\n    let listener = TcpListener::bind(&addr).await.unwrap();\n    println!(\"Listening on: {}\", addr);\n\n    loop {\n        let (mut socket, _) = listener.accept().await.unwrap();\n        socket.set_nodelay(true).unwrap();\n        let rt = current_handle();\n        rt.spawn(async move {\n            loop {\n                let mut buf = [0; 1024];\n                let res = socket.read(&mut buf).await;\n\n                let n = match res {\n                    Ok(n) => n,\n                    Err(_) => return,\n                };\n\n                if n == 0 {\n                    return;\n                }\n\n                let _ = socket\n                    .write_all(\n                        b\"HTTP/1.1 200 OK\\r\\ncontent-length: 12\\r\\nconnection: keep-alive\\r\\n\\r\\nHello world!\",\n                    )\n                    .await;\n            }\n        });\n    }\n}\n\n/* On M1 macbook pro\nwrk -t40 -c1000  -d10  http://127.0.0.1:3001  --latency\nRunning 10s test @ http://127.0.0.1:3001\n  40 threads and 1000 connections\n  Thread Stats   Avg      Stdev     Max   +/- Stdev\n    Latency     3.53ms    0.87ms  17.12ms   84.99%\n    Req/Sec     7.09k     1.29k   33.11k    93.30%\n  Latency Distribution\n     50%    3.56ms\n     75%    3.95ms\n     90%    4.37ms\n     99%    5.38ms\n  2844034 requests in 10.10s, 203.42MB read\nRequests/sec: 281689.27\nTransfer/sec:     20.15MB\n\nwrk -t40 -c1000  -d10  http://127.0.0.1:3000  --latency\nRunning 10s test @ http://127.0.0.1:3000\n  40 threads and 1000 connections\n  Thread Stats   Avg      Stdev     Max   +/- Stdev\n    Latency    12.16ms   16.29ms 112.29ms   83.40%\n    Req/Sec     5.47k     2.01k   48.85k    83.67%\n  Latency Distribution\n     50%    2.09ms\n     75%   20.23ms\n     90%   37.11ms\n     99%   65.16ms\n  2190869 requests in 10.10s, 156.70MB read\nRequests/sec: 216918.71\nTransfer/sec:     15.52MB\n*/\n\nfn main() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {\n    let rt = Runtime::new_steal(2, \"\");\n    let handle = rt.get_handle();\n    handle.spawn(hello_server(3000));\n    let rt2 = Runtime::new_no_steal(2, \"\");\n    let handle = rt2.get_handle();\n    handle.spawn(hello_server(3001));\n    thread::sleep(time::Duration::from_secs(999999999));\n    Ok(())\n}\n"
  },
  {
    "path": "pingora-runtime/src/lib.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! Pingora tokio runtime.\n//!\n//! Tokio runtime comes in two flavors: a single-threaded runtime\n//! and a multi-threaded one which provides work stealing.\n//! Benchmark shows that, compared to the single-threaded runtime, the multi-threaded one\n//! has some overhead due to its more sophisticated work steal scheduling.\n//!\n//! This crate provides a third flavor: a multi-threaded runtime without work stealing.\n//! This flavor is as efficient as the single-threaded runtime while allows the async\n//! program to use multiple cores.\n\nuse once_cell::sync::{Lazy, OnceCell};\nuse rand::Rng;\nuse std::sync::Arc;\nuse std::thread::JoinHandle;\nuse std::time::Duration;\nuse thread_local::ThreadLocal;\nuse tokio::runtime::{Builder, Handle};\nuse tokio::sync::oneshot::{channel, Sender};\n\n/// Pingora async multi-threaded runtime\n///\n/// The `Steal` flavor is effectively tokio multi-threaded runtime.\n///\n/// The `NoSteal` flavor is backed by multiple tokio single-threaded runtime.\npub enum Runtime {\n    Steal(tokio::runtime::Runtime),\n    NoSteal(NoStealRuntime),\n}\n\nimpl Runtime {\n    /// Create a `Steal` flavor runtime. This just a regular tokio runtime\n    pub fn new_steal(threads: usize, name: &str) -> Self {\n        Self::Steal(\n            Builder::new_multi_thread()\n                .enable_all()\n                .worker_threads(threads)\n                .thread_name(name)\n                .build()\n                .unwrap(),\n        )\n    }\n\n    /// Create a `NoSteal` flavor runtime. This is backed by multiple tokio current-thread runtime\n    pub fn new_no_steal(threads: usize, name: &str) -> Self {\n        Self::NoSteal(NoStealRuntime::new(threads, name))\n    }\n\n    /// Return the &[Handle] of the [Runtime].\n    /// For `Steal` flavor, it will just return the &[Handle].\n    /// For `NoSteal` flavor, it will return the &[Handle] of a random thread in its pool.\n    /// So if we want tasks to spawn on all the threads, call this function to get a fresh [Handle]\n    /// for each async task.\n    pub fn get_handle(&self) -> &Handle {\n        match self {\n            Self::Steal(r) => r.handle(),\n            Self::NoSteal(r) => r.get_runtime(),\n        }\n    }\n\n    /// Call tokio's `shutdown_timeout` of all the runtimes. This function is blocking until\n    /// all runtimes exit.\n    pub fn shutdown_timeout(self, timeout: Duration) {\n        match self {\n            Self::Steal(r) => r.shutdown_timeout(timeout),\n            Self::NoSteal(r) => r.shutdown_timeout(timeout),\n        }\n    }\n}\n\n// only NoStealRuntime set the pools in thread threads\nstatic CURRENT_HANDLE: Lazy<ThreadLocal<Pools>> = Lazy::new(ThreadLocal::new);\n\n/// Return the [Handle] of current runtime.\n/// If the current thread is under a `Steal` runtime, the current [Handle] is returned.\n/// If the current thread is under a `NoSteal` runtime, the [Handle] of a random thread\n/// under this runtime is returned. This function will panic if called outside any runtime.\npub fn current_handle() -> Handle {\n    if let Some(pools) = CURRENT_HANDLE.get() {\n        // safety: the CURRENT_HANDLE is set when the pool is being initialized in init_pools()\n        let pools = pools.get().unwrap();\n        let mut rng = rand::thread_rng();\n        let index = rng.gen_range(0..pools.len());\n        pools[index].clone()\n    } else {\n        // not NoStealRuntime, just check the current tokio runtime\n        Handle::current()\n    }\n}\n\ntype Control = (Sender<Duration>, JoinHandle<()>);\ntype Pools = Arc<OnceCell<Box<[Handle]>>>;\n\n/// Multi-threaded runtime backed by a pool of single threaded tokio runtime\npub struct NoStealRuntime {\n    threads: usize,\n    name: String,\n    // Lazily init the runtimes so that they are created after pingora\n    // daemonize itself. Otherwise the runtime threads are lost.\n    pools: Pools,\n    controls: OnceCell<Vec<Control>>,\n}\n\nimpl NoStealRuntime {\n    /// Create a new [NoStealRuntime]. Panic if `threads` is 0\n    pub fn new(threads: usize, name: &str) -> Self {\n        assert!(threads != 0);\n        NoStealRuntime {\n            threads,\n            name: name.to_string(),\n            pools: Arc::new(OnceCell::new()),\n            controls: OnceCell::new(),\n        }\n    }\n\n    fn init_pools(&self) -> (Box<[Handle]>, Vec<Control>) {\n        let mut pools = Vec::with_capacity(self.threads);\n        let mut controls = Vec::with_capacity(self.threads);\n        for _ in 0..self.threads {\n            let rt = Builder::new_current_thread().enable_all().build().unwrap();\n            let handler = rt.handle().clone();\n            let (tx, rx) = channel::<Duration>();\n            let pools_ref = self.pools.clone();\n            let join = std::thread::Builder::new()\n                .name(self.name.clone())\n                .spawn(move || {\n                    CURRENT_HANDLE.get_or(|| pools_ref);\n                    if let Ok(timeout) = rt.block_on(rx) {\n                        rt.shutdown_timeout(timeout);\n                    } // else Err(_): tx is dropped, just exit\n                })\n                .unwrap();\n            pools.push(handler);\n            controls.push((tx, join));\n        }\n\n        (pools.into_boxed_slice(), controls)\n    }\n\n    /// Return the &[Handle] of a random thread of this runtime\n    pub fn get_runtime(&self) -> &Handle {\n        let mut rng = rand::thread_rng();\n\n        let index = rng.gen_range(0..self.threads);\n        self.get_runtime_at(index)\n    }\n\n    /// Return the number of threads of this runtime\n    pub fn threads(&self) -> usize {\n        self.threads\n    }\n\n    fn get_pools(&self) -> &[Handle] {\n        if let Some(p) = self.pools.get() {\n            p\n        } else {\n            // TODO: use a mutex to avoid creating a lot threads only to drop them\n            let (pools, controls) = self.init_pools();\n            // there could be another thread racing with this one to init the pools\n            match self.pools.try_insert(pools) {\n                Ok(p) => {\n                    // unwrap to make sure that this is the one that init both pools and controls\n                    self.controls.set(controls).unwrap();\n                    p\n                }\n                // another thread already set it, just return it\n                Err((p, _my_pools)) => p,\n            }\n        }\n    }\n\n    /// Return the &[Handle] of a given thread of this runtime\n    pub fn get_runtime_at(&self, index: usize) -> &Handle {\n        let pools = self.get_pools();\n        &pools[index]\n    }\n\n    /// Call tokio's `shutdown_timeout` of all the runtimes. This function is blocking until\n    /// all runtimes exit.\n    pub fn shutdown_timeout(mut self, timeout: Duration) {\n        if let Some(controls) = self.controls.take() {\n            let (txs, joins): (Vec<Sender<_>>, Vec<JoinHandle<()>>) = controls.into_iter().unzip();\n            for tx in txs {\n                let _ = tx.send(timeout); // Err() when rx is dropped\n            }\n            for join in joins {\n                let _ = join.join(); // ignore thread error\n            }\n        } // else, the controls and the runtimes are not even init yet, just return;\n    }\n\n    // TODO: runtime metrics\n}\n\n#[test]\nfn test_steal_runtime() {\n    use tokio::time::{sleep, Duration};\n    let threads = 2;\n    let rt = Runtime::new_steal(threads, \"test\");\n    let handle = rt.get_handle();\n    let ret = handle.block_on(async {\n        sleep(Duration::from_secs(1)).await;\n        let handle = current_handle();\n        let join = handle.spawn(async {\n            sleep(Duration::from_secs(1)).await;\n        });\n        join.await.unwrap();\n        1\n    });\n\n    #[cfg(target_os = \"linux\")]\n    assert_eq!(handle.metrics().num_workers(), threads);\n    assert_eq!(ret, 1);\n}\n\n#[test]\nfn test_no_steal_runtime() {\n    use tokio::time::{sleep, Duration};\n\n    let rt = Runtime::new_no_steal(2, \"test\");\n    let handle = rt.get_handle();\n    let ret = handle.block_on(async {\n        sleep(Duration::from_secs(1)).await;\n        let handle = current_handle();\n        let join = handle.spawn(async {\n            sleep(Duration::from_secs(1)).await;\n        });\n        join.await.unwrap();\n        1\n    });\n\n    assert_eq!(ret, 1);\n}\n\n#[test]\nfn test_no_steal_shutdown() {\n    use tokio::time::{sleep, Duration};\n\n    let rt = Runtime::new_no_steal(2, \"test\");\n    let handle = rt.get_handle();\n    let ret = handle.block_on(async {\n        sleep(Duration::from_secs(1)).await;\n        let handle = current_handle();\n        let join = handle.spawn(async {\n            sleep(Duration::from_secs(1)).await;\n        });\n        join.await.unwrap();\n        1\n    });\n    assert_eq!(ret, 1);\n\n    rt.shutdown_timeout(Duration::from_secs(1));\n}\n"
  },
  {
    "path": "pingora-rustls/Cargo.toml",
    "content": "[package]\nname = \"pingora-rustls\"\nversion = \"0.8.0\"\nlicense = \"Apache-2.0\"\nedition = \"2021\"\nrepository = \"https://github.com/cloudflare/pingora\"\ncategories = [\"asynchronous\", \"network-programming\"]\nkeywords = [\"async\", \"tls\", \"ssl\", \"pingora\"]\ndescription = \"\"\"\nRusTLS async APIs for Pingora.\n\"\"\"\n\n[lib]\nname = \"pingora_rustls\"\npath = \"src/lib.rs\"\n\n[dependencies]\nlog = \"0.4.21\"\npingora-error = { version = \"0.8.0\", path = \"../pingora-error\"}\nring = \"0.17.12\"\nrustls = \"0.23.12\"\nrustls-native-certs = \"0.7.1\"\nrustls-pemfile = \"2.1.2\"\nrustls-pki-types = \"1.7.0\"\ntokio-rustls = \"0.26.0\"\nno_debug = \"3.1.0\"\n"
  },
  {
    "path": "pingora-rustls/LICENSE",
    "content": "\n                                 Apache License\n                           Version 2.0, January 2004\n                        http://www.apache.org/licenses/\n\n   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION\n\n   1. Definitions.\n\n      \"License\" shall mean the terms and conditions for use, reproduction,\n      and distribution as defined by Sections 1 through 9 of this document.\n\n      \"Licensor\" shall mean the copyright owner or entity authorized by\n      the copyright owner that is granting the License.\n\n      \"Legal Entity\" shall mean the union of the acting entity and all\n      other entities that control, are controlled by, or are under common\n      control with that entity. For the purposes of this definition,\n      \"control\" means (i) the power, direct or indirect, to cause the\n      direction or management of such entity, whether by contract or\n      otherwise, or (ii) ownership of fifty percent (50%) or more of the\n      outstanding shares, or (iii) beneficial ownership of such entity.\n\n      \"You\" (or \"Your\") shall mean an individual or Legal Entity\n      exercising permissions granted by this License.\n\n      \"Source\" form shall mean the preferred form for making modifications,\n      including but not limited to software source code, documentation\n      source, and configuration files.\n\n      \"Object\" form shall mean any form resulting from mechanical\n      transformation or translation of a Source form, including but\n      not limited to compiled object code, generated documentation,\n      and conversions to other media types.\n\n      \"Work\" shall mean the work of authorship, whether in Source or\n      Object form, made available under the License, as indicated by a\n      copyright notice that is included in or attached to the work\n      (an example is provided in the Appendix below).\n\n      \"Derivative Works\" shall mean any work, whether in Source or Object\n      form, that is based on (or derived from) the Work and for which the\n      editorial revisions, annotations, elaborations, or other modifications\n      represent, as a whole, an original work of authorship. For the purposes\n      of this License, Derivative Works shall not include works that remain\n      separable from, or merely link (or bind by name) to the interfaces of,\n      the Work and Derivative Works thereof.\n\n      \"Contribution\" shall mean any work of authorship, including\n      the original version of the Work and any modifications or additions\n      to that Work or Derivative Works thereof, that is intentionally\n      submitted to Licensor for inclusion in the Work by the copyright owner\n      or by an individual or Legal Entity authorized to submit on behalf of\n      the copyright owner. For the purposes of this definition, \"submitted\"\n      means any form of electronic, verbal, or written communication sent\n      to the Licensor or its representatives, including but not limited to\n      communication on electronic mailing lists, source code control systems,\n      and issue tracking systems that are managed by, or on behalf of, the\n      Licensor for the purpose of discussing and improving the Work, but\n      excluding communication that is conspicuously marked or otherwise\n      designated in writing by the copyright owner as \"Not a Contribution.\"\n\n      \"Contributor\" shall mean Licensor and any individual or Legal Entity\n      on behalf of whom a Contribution has been received by Licensor and\n      subsequently incorporated within the Work.\n\n   2. Grant of Copyright License. Subject to the terms and conditions of\n      this License, each Contributor hereby grants to You a perpetual,\n      worldwide, non-exclusive, no-charge, royalty-free, irrevocable\n      copyright license to reproduce, prepare Derivative Works of,\n      publicly display, publicly perform, sublicense, and distribute the\n      Work and such Derivative Works in Source or Object form.\n\n   3. Grant of Patent License. Subject to the terms and conditions of\n      this License, each Contributor hereby grants to You a perpetual,\n      worldwide, non-exclusive, no-charge, royalty-free, irrevocable\n      (except as stated in this section) patent license to make, have made,\n      use, offer to sell, sell, import, and otherwise transfer the Work,\n      where such license applies only to those patent claims licensable\n      by such Contributor that are necessarily infringed by their\n      Contribution(s) alone or by combination of their Contribution(s)\n      with the Work to which such Contribution(s) was submitted. If You\n      institute patent litigation against any entity (including a\n      cross-claim or counterclaim in a lawsuit) alleging that the Work\n      or a Contribution incorporated within the Work constitutes direct\n      or contributory patent infringement, then any patent licenses\n      granted to You under this License for that Work shall terminate\n      as of the date such litigation is filed.\n\n   4. Redistribution. You may reproduce and distribute copies of the\n      Work or Derivative Works thereof in any medium, with or without\n      modifications, and in Source or Object form, provided that You\n      meet the following conditions:\n\n      (a) You must give any other recipients of the Work or\n          Derivative Works a copy of this License; and\n\n      (b) You must cause any modified files to carry prominent notices\n          stating that You changed the files; and\n\n      (c) You must retain, in the Source form of any Derivative Works\n          that You distribute, all copyright, patent, trademark, and\n          attribution notices from the Source form of the Work,\n          excluding those notices that do not pertain to any part of\n          the Derivative Works; and\n\n      (d) If the Work includes a \"NOTICE\" text file as part of its\n          distribution, then any Derivative Works that You distribute must\n          include a readable copy of the attribution notices contained\n          within such NOTICE file, excluding those notices that do not\n          pertain to any part of the Derivative Works, in at least one\n          of the following places: within a NOTICE text file distributed\n          as part of the Derivative Works; within the Source form or\n          documentation, if provided along with the Derivative Works; or,\n          within a display generated by the Derivative Works, if and\n          wherever such third-party notices normally appear. The contents\n          of the NOTICE file are for informational purposes only and\n          do not modify the License. You may add Your own attribution\n          notices within Derivative Works that You distribute, alongside\n          or as an addendum to the NOTICE text from the Work, provided\n          that such additional attribution notices cannot be construed\n          as modifying the License.\n\n      You may add Your own copyright statement to Your modifications and\n      may provide additional or different license terms and conditions\n      for use, reproduction, or distribution of Your modifications, or\n      for any such Derivative Works as a whole, provided Your use,\n      reproduction, and distribution of the Work otherwise complies with\n      the conditions stated in this License.\n\n   5. Submission of Contributions. Unless You explicitly state otherwise,\n      any Contribution intentionally submitted for inclusion in the Work\n      by You to the Licensor shall be under the terms and conditions of\n      this License, without any additional terms or conditions.\n      Notwithstanding the above, nothing herein shall supersede or modify\n      the terms of any separate license agreement you may have executed\n      with Licensor regarding such Contributions.\n\n   6. Trademarks. This License does not grant permission to use the trade\n      names, trademarks, service marks, or product names of the Licensor,\n      except as required for reasonable and customary use in describing the\n      origin of the Work and reproducing the content of the NOTICE file.\n\n   7. Disclaimer of Warranty. Unless required by applicable law or\n      agreed to in writing, Licensor provides the Work (and each\n      Contributor provides its Contributions) on an \"AS IS\" BASIS,\n      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or\n      implied, including, without limitation, any warranties or conditions\n      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A\n      PARTICULAR PURPOSE. You are solely responsible for determining the\n      appropriateness of using or redistributing the Work and assume any\n      risks associated with Your exercise of permissions under this License.\n\n   8. Limitation of Liability. In no event and under no legal theory,\n      whether in tort (including negligence), contract, or otherwise,\n      unless required by applicable law (such as deliberate and grossly\n      negligent acts) or agreed to in writing, shall any Contributor be\n      liable to You for damages, including any direct, indirect, special,\n      incidental, or consequential damages of any character arising as a\n      result of this License or out of the use or inability to use the\n      Work (including but not limited to damages for loss of goodwill,\n      work stoppage, computer failure or malfunction, or any and all\n      other commercial damages or losses), even if such Contributor\n      has been advised of the possibility of such damages.\n\n   9. Accepting Warranty or Additional Liability. While redistributing\n      the Work or Derivative Works thereof, You may choose to offer,\n      and charge a fee for, acceptance of support, warranty, indemnity,\n      or other liability obligations and/or rights consistent with this\n      License. However, in accepting such obligations, You may act only\n      on Your own behalf and on Your sole responsibility, not on behalf\n      of any other Contributor, and only if You agree to indemnify,\n      defend, and hold each Contributor harmless for any liability\n      incurred by, or claims asserted against, such Contributor by reason\n      of your accepting any such warranty or additional liability.\n\n   END OF TERMS AND CONDITIONS\n\n   APPENDIX: How to apply the Apache License to your work.\n\n      To apply the Apache License to your work, attach the following\n      boilerplate notice, with the fields enclosed by brackets \"[]\"\n      replaced with your own identifying information. (Don't include\n      the brackets!)  The text should be enclosed in the appropriate\n      comment syntax for the file format. We also recommend that a\n      file or class name and description of purpose be included on the\n      same \"printed page\" as the copyright notice for easier\n      identification within third-party archives.\n\n   Copyright [yyyy] [name of copyright owner]\n\n   Licensed under the Apache License, Version 2.0 (the \"License\");\n   you may not use this file except in compliance with the License.\n   You may obtain a copy of the License at\n\n       http://www.apache.org/licenses/LICENSE-2.0\n\n   Unless required by applicable law or agreed to in writing, software\n   distributed under the License is distributed on an \"AS IS\" BASIS,\n   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n   See the License for the specific language governing permissions and\n   limitations under the License.\n"
  },
  {
    "path": "pingora-rustls/src/lib.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! This module contains all the rustls specific pingora integration for things\n//! like loading certificates and private keys\n\n#![warn(clippy::all)]\n\nuse std::fs::File;\nuse std::io::BufReader;\nuse std::path::Path;\n\nuse log::warn;\npub use no_debug::{Ellipses, NoDebug, WithTypeInfo};\nuse pingora_error::{Error, ErrorType, OrErr, Result};\n\npub use rustls::server::danger::{ClientCertVerified, ClientCertVerifier};\npub use rustls::server::{ClientCertVerifierBuilder, WebPkiClientVerifier};\npub use rustls::{\n    client::WebPkiServerVerifier, version, CertificateError, ClientConfig, DigitallySignedStruct,\n    Error as RusTlsError, KeyLogFile, RootCertStore, ServerConfig, SignatureScheme, Stream,\n};\npub use rustls_native_certs::load_native_certs;\nuse rustls_pemfile::Item;\npub use rustls_pki_types::{CertificateDer, PrivateKeyDer, ServerName, UnixTime};\npub use tokio_rustls::client::TlsStream as ClientTlsStream;\npub use tokio_rustls::server::TlsStream as ServerTlsStream;\npub use tokio_rustls::{Accept, Connect, TlsAcceptor, TlsConnector, TlsStream};\n\n// This allows to skip certificate verification. Be highly cautious.\npub use rustls::client::danger::{HandshakeSignatureValid, ServerCertVerified, ServerCertVerifier};\n\n/// Load the given file from disk as a buffered reader and use the pingora Error\n/// type instead of the std::io version\nfn load_file<P>(path: P) -> Result<BufReader<File>>\nwhere\n    P: AsRef<Path>,\n{\n    File::open(path)\n        .or_err(ErrorType::FileReadError, \"Failed to load file\")\n        .map(BufReader::new)\n}\n\n/// Read the pem file at the given path from disk\nfn load_pem_file<P>(path: P) -> Result<Vec<Item>>\nwhere\n    P: AsRef<Path>,\n{\n    rustls_pemfile::read_all(&mut load_file(path)?)\n        .map(|item_res| {\n            item_res.or_err(\n                ErrorType::InvalidCert,\n                \"Certificate in pem file could not be read\",\n            )\n        })\n        .collect()\n}\n\n/// Load the certificates from the given pem file path into the given\n/// certificate store\npub fn load_ca_file_into_store<P>(path: P, cert_store: &mut RootCertStore) -> Result<()>\nwhere\n    P: AsRef<Path>,\n{\n    for pem_item in load_pem_file(path)? {\n        // only loading certificates, handling a CA file\n        let Item::X509Certificate(content) = pem_item else {\n            return Error::e_explain(\n                ErrorType::InvalidCert,\n                \"Pem file contains un-loadable certificate type\",\n            );\n        };\n        cert_store.add(content).or_err(\n            ErrorType::InvalidCert,\n            \"Failed to load X509 certificate into root store\",\n        )?;\n    }\n\n    Ok(())\n}\n\n/// Attempt to load the native cas into the given root-certificate store\npub fn load_platform_certs_incl_env_into_store(ca_certs: &mut RootCertStore) -> Result<()> {\n    // this includes handling of ENV vars SSL_CERT_FILE & SSL_CERT_DIR\n    for cert in load_native_certs()\n        .or_err(ErrorType::InvalidCert, \"Failed to load native certificates\")?\n        .into_iter()\n    {\n        ca_certs.add(cert).or_err(\n            ErrorType::InvalidCert,\n            \"Failed to load native certificate into root store\",\n        )?;\n    }\n\n    Ok(())\n}\n\n/// Load the certificates and private key files\npub fn load_certs_and_key_files<'a>(\n    cert: &str,\n    key: &str,\n) -> Result<Option<(Vec<CertificateDer<'a>>, PrivateKeyDer<'a>)>> {\n    let certs_file = load_pem_file(cert)?;\n    let key_file = load_pem_file(key)?;\n\n    let certs = certs_file\n        .into_iter()\n        .filter_map(|item| {\n            if let Item::X509Certificate(cert) = item {\n                Some(cert)\n            } else {\n                None\n            }\n        })\n        .collect::<Vec<_>>();\n\n    // These are the currently supported pk types -\n    // [https://doc.servo.org/rustls/key/struct.PrivateKey.html]\n    let private_key_opt = key_file\n        .into_iter()\n        .filter_map(|key_item| match key_item {\n            Item::Pkcs1Key(key) => Some(PrivateKeyDer::from(key)),\n            Item::Pkcs8Key(key) => Some(PrivateKeyDer::from(key)),\n            Item::Sec1Key(key) => Some(PrivateKeyDer::from(key)),\n            _ => None,\n        })\n        .next();\n\n    if let (Some(private_key), false) = (private_key_opt, certs.is_empty()) {\n        Ok(Some((certs, private_key)))\n    } else {\n        Ok(None)\n    }\n}\n\n/// Load the certificate\npub fn load_pem_file_ca(path: &String) -> Result<Vec<u8>> {\n    let mut reader = load_file(path)?;\n    let cas_file_items = rustls_pemfile::certs(&mut reader)\n        .map(|item_res| {\n            item_res.or_err(\n                ErrorType::InvalidCert,\n                \"Failed to load certificate from file\",\n            )\n        })\n        .collect::<Result<Vec<_>>>()?;\n\n    Ok(cas_file_items\n        .first()\n        .map(|ca| ca.to_vec())\n        .unwrap_or_default())\n}\n\npub fn load_pem_file_private_key(path: &String) -> Result<Vec<u8>> {\n    Ok(rustls_pemfile::private_key(&mut load_file(path)?)\n        .or_err(\n            ErrorType::InvalidCert,\n            \"Failed to load private key from file\",\n        )?\n        .map(|key| key.secret_der().to_vec())\n        .unwrap_or_default())\n}\n\npub fn hash_certificate(cert: &CertificateDer) -> Vec<u8> {\n    let hash = ring::digest::digest(&ring::digest::SHA256, cert.as_ref());\n    hash.as_ref().to_vec()\n}\n"
  },
  {
    "path": "pingora-s2n/Cargo.toml",
    "content": "[package]\nname = \"pingora-s2n\"\nversion = \"0.8.0\"\nlicense = \"Apache-2.0\"\nedition = \"2021\"\nrepository = \"https://github.com/cloudflare/pingora\"\ncategories = [\"asynchronous\", \"network-programming\"]\nkeywords = [\"async\", \"tls\", \"ssl\", \"pingora\"]\ndescription = \"\"\"\nS2N async APIs for Pingora.\n\"\"\"\n\n[lib]\nname = \"pingora_s2n\"\npath = \"src/lib.rs\"\n\n[dependencies]\npingora-error = { version = \"0.8.0\", path = \"../pingora-error\"}\nring = \"0.17.12\"\ns2n-tls = \"0.3\"\ns2n-tls-tokio = \"0.3\"\n"
  },
  {
    "path": "pingora-s2n/LICENSE",
    "content": "\n                                 Apache License\n                           Version 2.0, January 2004\n                        http://www.apache.org/licenses/\n\n   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION\n\n   1. Definitions.\n\n      \"License\" shall mean the terms and conditions for use, reproduction,\n      and distribution as defined by Sections 1 through 9 of this document.\n\n      \"Licensor\" shall mean the copyright owner or entity authorized by\n      the copyright owner that is granting the License.\n\n      \"Legal Entity\" shall mean the union of the acting entity and all\n      other entities that control, are controlled by, or are under common\n      control with that entity. For the purposes of this definition,\n      \"control\" means (i) the power, direct or indirect, to cause the\n      direction or management of such entity, whether by contract or\n      otherwise, or (ii) ownership of fifty percent (50%) or more of the\n      outstanding shares, or (iii) beneficial ownership of such entity.\n\n      \"You\" (or \"Your\") shall mean an individual or Legal Entity\n      exercising permissions granted by this License.\n\n      \"Source\" form shall mean the preferred form for making modifications,\n      including but not limited to software source code, documentation\n      source, and configuration files.\n\n      \"Object\" form shall mean any form resulting from mechanical\n      transformation or translation of a Source form, including but\n      not limited to compiled object code, generated documentation,\n      and conversions to other media types.\n\n      \"Work\" shall mean the work of authorship, whether in Source or\n      Object form, made available under the License, as indicated by a\n      copyright notice that is included in or attached to the work\n      (an example is provided in the Appendix below).\n\n      \"Derivative Works\" shall mean any work, whether in Source or Object\n      form, that is based on (or derived from) the Work and for which the\n      editorial revisions, annotations, elaborations, or other modifications\n      represent, as a whole, an original work of authorship. For the purposes\n      of this License, Derivative Works shall not include works that remain\n      separable from, or merely link (or bind by name) to the interfaces of,\n      the Work and Derivative Works thereof.\n\n      \"Contribution\" shall mean any work of authorship, including\n      the original version of the Work and any modifications or additions\n      to that Work or Derivative Works thereof, that is intentionally\n      submitted to Licensor for inclusion in the Work by the copyright owner\n      or by an individual or Legal Entity authorized to submit on behalf of\n      the copyright owner. For the purposes of this definition, \"submitted\"\n      means any form of electronic, verbal, or written communication sent\n      to the Licensor or its representatives, including but not limited to\n      communication on electronic mailing lists, source code control systems,\n      and issue tracking systems that are managed by, or on behalf of, the\n      Licensor for the purpose of discussing and improving the Work, but\n      excluding communication that is conspicuously marked or otherwise\n      designated in writing by the copyright owner as \"Not a Contribution.\"\n\n      \"Contributor\" shall mean Licensor and any individual or Legal Entity\n      on behalf of whom a Contribution has been received by Licensor and\n      subsequently incorporated within the Work.\n\n   2. Grant of Copyright License. Subject to the terms and conditions of\n      this License, each Contributor hereby grants to You a perpetual,\n      worldwide, non-exclusive, no-charge, royalty-free, irrevocable\n      copyright license to reproduce, prepare Derivative Works of,\n      publicly display, publicly perform, sublicense, and distribute the\n      Work and such Derivative Works in Source or Object form.\n\n   3. Grant of Patent License. Subject to the terms and conditions of\n      this License, each Contributor hereby grants to You a perpetual,\n      worldwide, non-exclusive, no-charge, royalty-free, irrevocable\n      (except as stated in this section) patent license to make, have made,\n      use, offer to sell, sell, import, and otherwise transfer the Work,\n      where such license applies only to those patent claims licensable\n      by such Contributor that are necessarily infringed by their\n      Contribution(s) alone or by combination of their Contribution(s)\n      with the Work to which such Contribution(s) was submitted. If You\n      institute patent litigation against any entity (including a\n      cross-claim or counterclaim in a lawsuit) alleging that the Work\n      or a Contribution incorporated within the Work constitutes direct\n      or contributory patent infringement, then any patent licenses\n      granted to You under this License for that Work shall terminate\n      as of the date such litigation is filed.\n\n   4. Redistribution. You may reproduce and distribute copies of the\n      Work or Derivative Works thereof in any medium, with or without\n      modifications, and in Source or Object form, provided that You\n      meet the following conditions:\n\n      (a) You must give any other recipients of the Work or\n          Derivative Works a copy of this License; and\n\n      (b) You must cause any modified files to carry prominent notices\n          stating that You changed the files; and\n\n      (c) You must retain, in the Source form of any Derivative Works\n          that You distribute, all copyright, patent, trademark, and\n          attribution notices from the Source form of the Work,\n          excluding those notices that do not pertain to any part of\n          the Derivative Works; and\n\n      (d) If the Work includes a \"NOTICE\" text file as part of its\n          distribution, then any Derivative Works that You distribute must\n          include a readable copy of the attribution notices contained\n          within such NOTICE file, excluding those notices that do not\n          pertain to any part of the Derivative Works, in at least one\n          of the following places: within a NOTICE text file distributed\n          as part of the Derivative Works; within the Source form or\n          documentation, if provided along with the Derivative Works; or,\n          within a display generated by the Derivative Works, if and\n          wherever such third-party notices normally appear. The contents\n          of the NOTICE file are for informational purposes only and\n          do not modify the License. You may add Your own attribution\n          notices within Derivative Works that You distribute, alongside\n          or as an addendum to the NOTICE text from the Work, provided\n          that such additional attribution notices cannot be construed\n          as modifying the License.\n\n      You may add Your own copyright statement to Your modifications and\n      may provide additional or different license terms and conditions\n      for use, reproduction, or distribution of Your modifications, or\n      for any such Derivative Works as a whole, provided Your use,\n      reproduction, and distribution of the Work otherwise complies with\n      the conditions stated in this License.\n\n   5. Submission of Contributions. Unless You explicitly state otherwise,\n      any Contribution intentionally submitted for inclusion in the Work\n      by You to the Licensor shall be under the terms and conditions of\n      this License, without any additional terms or conditions.\n      Notwithstanding the above, nothing herein shall supersede or modify\n      the terms of any separate license agreement you may have executed\n      with Licensor regarding such Contributions.\n\n   6. Trademarks. This License does not grant permission to use the trade\n      names, trademarks, service marks, or product names of the Licensor,\n      except as required for reasonable and customary use in describing the\n      origin of the Work and reproducing the content of the NOTICE file.\n\n   7. Disclaimer of Warranty. Unless required by applicable law or\n      agreed to in writing, Licensor provides the Work (and each\n      Contributor provides its Contributions) on an \"AS IS\" BASIS,\n      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or\n      implied, including, without limitation, any warranties or conditions\n      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A\n      PARTICULAR PURPOSE. You are solely responsible for determining the\n      appropriateness of using or redistributing the Work and assume any\n      risks associated with Your exercise of permissions under this License.\n\n   8. Limitation of Liability. In no event and under no legal theory,\n      whether in tort (including negligence), contract, or otherwise,\n      unless required by applicable law (such as deliberate and grossly\n      negligent acts) or agreed to in writing, shall any Contributor be\n      liable to You for damages, including any direct, indirect, special,\n      incidental, or consequential damages of any character arising as a\n      result of this License or out of the use or inability to use the\n      Work (including but not limited to damages for loss of goodwill,\n      work stoppage, computer failure or malfunction, or any and all\n      other commercial damages or losses), even if such Contributor\n      has been advised of the possibility of such damages.\n\n   9. Accepting Warranty or Additional Liability. While redistributing\n      the Work or Derivative Works thereof, You may choose to offer,\n      and charge a fee for, acceptance of support, warranty, indemnity,\n      or other liability obligations and/or rights consistent with this\n      License. However, in accepting such obligations, You may act only\n      on Your own behalf and on Your sole responsibility, not on behalf\n      of any other Contributor, and only if You agree to indemnify,\n      defend, and hold each Contributor harmless for any liability\n      incurred by, or claims asserted against, such Contributor by reason\n      of your accepting any such warranty or additional liability.\n\n   END OF TERMS AND CONDITIONS\n\n   APPENDIX: How to apply the Apache License to your work.\n\n      To apply the Apache License to your work, attach the following\n      boilerplate notice, with the fields enclosed by brackets \"[]\"\n      replaced with your own identifying information. (Don't include\n      the brackets!)  The text should be enclosed in the appropriate\n      comment syntax for the file format. We also recommend that a\n      file or class name and description of purpose be included on the\n      same \"printed page\" as the copyright notice for easier\n      identification within third-party archives.\n\n   Copyright [yyyy] [name of copyright owner]\n\n   Licensed under the Apache License, Version 2.0 (the \"License\");\n   you may not use this file except in compliance with the License.\n   You may obtain a copy of the License at\n\n       http://www.apache.org/licenses/LICENSE-2.0\n\n   Unless required by applicable law or agreed to in writing, software\n   distributed under the License is distributed on an \"AS IS\" BASIS,\n   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n   See the License for the specific language governing permissions and\n   limitations under the License.\n"
  },
  {
    "path": "pingora-s2n/src/lib.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applijable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\nuse pingora_error::{Error, ErrorType, Result};\nuse std::fs;\n\npub use s2n_tls::{\n    callbacks::VerifyHostNameCallback,\n    config::{Builder as ConfigBuilder, Config},\n    connection::{Builder as ConnectionBuilder, Connection},\n    enums::{ClientAuthType, Mode, PskHmac},\n    error::Error as S2NError,\n    psk::Psk,\n    security::{Policy as S2NPolicy, DEFAULT_TLS13},\n};\npub use s2n_tls_tokio::{TlsAcceptor, TlsConnector, TlsStream};\n\npub fn load_certs_and_key_files(cert_file: &str, key_file: &str) -> Result<(Vec<u8>, Vec<u8>)> {\n    let cert_bytes = load_pem_file(cert_file)?;\n    let key_bytes = load_pem_file(key_file)?;\n    Ok((cert_bytes, key_bytes))\n}\n\npub fn load_pem_file(file: &str) -> Result<Vec<u8>> {\n    if let Ok(bytes) = fs::read(file) {\n        Ok(bytes)\n    } else {\n        Error::e_explain(\n            ErrorType::InvalidCert,\n            \"Certificate in pem file could not be read\",\n        )\n    }\n}\n\npub fn hash_certificate(cert: &[u8]) -> Vec<u8> {\n    let hash = ring::digest::digest(&ring::digest::SHA256, cert);\n    hash.as_ref().to_vec()\n}\n\n/// Verify host name callback that always returns a success,\n/// effectively ignoring hostname validation\npub struct IgnoreVerifyHostnameCallback {}\n\nimpl IgnoreVerifyHostnameCallback {\n    pub fn new() -> Self {\n        IgnoreVerifyHostnameCallback {}\n    }\n}\n\nimpl Default for IgnoreVerifyHostnameCallback {\n    fn default() -> Self {\n        Self::new()\n    }\n}\n\nimpl VerifyHostNameCallback for IgnoreVerifyHostnameCallback {\n    fn verify_host_name(&self, _host_name: &str) -> bool {\n        true\n    }\n}\n"
  },
  {
    "path": "pingora-timeout/Cargo.toml",
    "content": "[package]\nname = \"pingora-timeout\"\nversion = \"0.8.0\"\nauthors = [\"Yuchen Wu <yuchen@cloudflare.com>\"]\nlicense = \"Apache-2.0\"\nedition = \"2021\"\nrepository = \"https://github.com/cloudflare/pingora\"\ncategories = [\"asynchronous\"]\nkeywords = [\"async\", \"non-blocking\", \"pingora\"]\ndescription = \"\"\"\nHighly efficient async timer and timeout system for Tokio runtimes.\n\"\"\"\n\n# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html\n[lib]\nname = \"pingora_timeout\"\npath = \"src/lib.rs\"\n\n[dependencies]\ntokio = { workspace = true, features = [\n    \"time\",\n    \"rt-multi-thread\",\n    \"macros\",\n    \"sync\",\n] }\npin-project-lite = \"0.2\"\nonce_cell = { workspace = true }\nparking_lot = \"0.12\"\nthread_local = \"1.0\"\n\n[dev-dependencies]\nbencher = \"0.1.5\"\n\n[[bench]]\nname = \"benchmark\"\nharness = false\n"
  },
  {
    "path": "pingora-timeout/LICENSE",
    "content": "\n                                 Apache License\n                           Version 2.0, January 2004\n                        http://www.apache.org/licenses/\n\n   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION\n\n   1. Definitions.\n\n      \"License\" shall mean the terms and conditions for use, reproduction,\n      and distribution as defined by Sections 1 through 9 of this document.\n\n      \"Licensor\" shall mean the copyright owner or entity authorized by\n      the copyright owner that is granting the License.\n\n      \"Legal Entity\" shall mean the union of the acting entity and all\n      other entities that control, are controlled by, or are under common\n      control with that entity. For the purposes of this definition,\n      \"control\" means (i) the power, direct or indirect, to cause the\n      direction or management of such entity, whether by contract or\n      otherwise, or (ii) ownership of fifty percent (50%) or more of the\n      outstanding shares, or (iii) beneficial ownership of such entity.\n\n      \"You\" (or \"Your\") shall mean an individual or Legal Entity\n      exercising permissions granted by this License.\n\n      \"Source\" form shall mean the preferred form for making modifications,\n      including but not limited to software source code, documentation\n      source, and configuration files.\n\n      \"Object\" form shall mean any form resulting from mechanical\n      transformation or translation of a Source form, including but\n      not limited to compiled object code, generated documentation,\n      and conversions to other media types.\n\n      \"Work\" shall mean the work of authorship, whether in Source or\n      Object form, made available under the License, as indicated by a\n      copyright notice that is included in or attached to the work\n      (an example is provided in the Appendix below).\n\n      \"Derivative Works\" shall mean any work, whether in Source or Object\n      form, that is based on (or derived from) the Work and for which the\n      editorial revisions, annotations, elaborations, or other modifications\n      represent, as a whole, an original work of authorship. For the purposes\n      of this License, Derivative Works shall not include works that remain\n      separable from, or merely link (or bind by name) to the interfaces of,\n      the Work and Derivative Works thereof.\n\n      \"Contribution\" shall mean any work of authorship, including\n      the original version of the Work and any modifications or additions\n      to that Work or Derivative Works thereof, that is intentionally\n      submitted to Licensor for inclusion in the Work by the copyright owner\n      or by an individual or Legal Entity authorized to submit on behalf of\n      the copyright owner. For the purposes of this definition, \"submitted\"\n      means any form of electronic, verbal, or written communication sent\n      to the Licensor or its representatives, including but not limited to\n      communication on electronic mailing lists, source code control systems,\n      and issue tracking systems that are managed by, or on behalf of, the\n      Licensor for the purpose of discussing and improving the Work, but\n      excluding communication that is conspicuously marked or otherwise\n      designated in writing by the copyright owner as \"Not a Contribution.\"\n\n      \"Contributor\" shall mean Licensor and any individual or Legal Entity\n      on behalf of whom a Contribution has been received by Licensor and\n      subsequently incorporated within the Work.\n\n   2. Grant of Copyright License. Subject to the terms and conditions of\n      this License, each Contributor hereby grants to You a perpetual,\n      worldwide, non-exclusive, no-charge, royalty-free, irrevocable\n      copyright license to reproduce, prepare Derivative Works of,\n      publicly display, publicly perform, sublicense, and distribute the\n      Work and such Derivative Works in Source or Object form.\n\n   3. Grant of Patent License. Subject to the terms and conditions of\n      this License, each Contributor hereby grants to You a perpetual,\n      worldwide, non-exclusive, no-charge, royalty-free, irrevocable\n      (except as stated in this section) patent license to make, have made,\n      use, offer to sell, sell, import, and otherwise transfer the Work,\n      where such license applies only to those patent claims licensable\n      by such Contributor that are necessarily infringed by their\n      Contribution(s) alone or by combination of their Contribution(s)\n      with the Work to which such Contribution(s) was submitted. If You\n      institute patent litigation against any entity (including a\n      cross-claim or counterclaim in a lawsuit) alleging that the Work\n      or a Contribution incorporated within the Work constitutes direct\n      or contributory patent infringement, then any patent licenses\n      granted to You under this License for that Work shall terminate\n      as of the date such litigation is filed.\n\n   4. Redistribution. You may reproduce and distribute copies of the\n      Work or Derivative Works thereof in any medium, with or without\n      modifications, and in Source or Object form, provided that You\n      meet the following conditions:\n\n      (a) You must give any other recipients of the Work or\n          Derivative Works a copy of this License; and\n\n      (b) You must cause any modified files to carry prominent notices\n          stating that You changed the files; and\n\n      (c) You must retain, in the Source form of any Derivative Works\n          that You distribute, all copyright, patent, trademark, and\n          attribution notices from the Source form of the Work,\n          excluding those notices that do not pertain to any part of\n          the Derivative Works; and\n\n      (d) If the Work includes a \"NOTICE\" text file as part of its\n          distribution, then any Derivative Works that You distribute must\n          include a readable copy of the attribution notices contained\n          within such NOTICE file, excluding those notices that do not\n          pertain to any part of the Derivative Works, in at least one\n          of the following places: within a NOTICE text file distributed\n          as part of the Derivative Works; within the Source form or\n          documentation, if provided along with the Derivative Works; or,\n          within a display generated by the Derivative Works, if and\n          wherever such third-party notices normally appear. The contents\n          of the NOTICE file are for informational purposes only and\n          do not modify the License. You may add Your own attribution\n          notices within Derivative Works that You distribute, alongside\n          or as an addendum to the NOTICE text from the Work, provided\n          that such additional attribution notices cannot be construed\n          as modifying the License.\n\n      You may add Your own copyright statement to Your modifications and\n      may provide additional or different license terms and conditions\n      for use, reproduction, or distribution of Your modifications, or\n      for any such Derivative Works as a whole, provided Your use,\n      reproduction, and distribution of the Work otherwise complies with\n      the conditions stated in this License.\n\n   5. Submission of Contributions. Unless You explicitly state otherwise,\n      any Contribution intentionally submitted for inclusion in the Work\n      by You to the Licensor shall be under the terms and conditions of\n      this License, without any additional terms or conditions.\n      Notwithstanding the above, nothing herein shall supersede or modify\n      the terms of any separate license agreement you may have executed\n      with Licensor regarding such Contributions.\n\n   6. Trademarks. This License does not grant permission to use the trade\n      names, trademarks, service marks, or product names of the Licensor,\n      except as required for reasonable and customary use in describing the\n      origin of the Work and reproducing the content of the NOTICE file.\n\n   7. Disclaimer of Warranty. Unless required by applicable law or\n      agreed to in writing, Licensor provides the Work (and each\n      Contributor provides its Contributions) on an \"AS IS\" BASIS,\n      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or\n      implied, including, without limitation, any warranties or conditions\n      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A\n      PARTICULAR PURPOSE. You are solely responsible for determining the\n      appropriateness of using or redistributing the Work and assume any\n      risks associated with Your exercise of permissions under this License.\n\n   8. Limitation of Liability. In no event and under no legal theory,\n      whether in tort (including negligence), contract, or otherwise,\n      unless required by applicable law (such as deliberate and grossly\n      negligent acts) or agreed to in writing, shall any Contributor be\n      liable to You for damages, including any direct, indirect, special,\n      incidental, or consequential damages of any character arising as a\n      result of this License or out of the use or inability to use the\n      Work (including but not limited to damages for loss of goodwill,\n      work stoppage, computer failure or malfunction, or any and all\n      other commercial damages or losses), even if such Contributor\n      has been advised of the possibility of such damages.\n\n   9. Accepting Warranty or Additional Liability. While redistributing\n      the Work or Derivative Works thereof, You may choose to offer,\n      and charge a fee for, acceptance of support, warranty, indemnity,\n      or other liability obligations and/or rights consistent with this\n      License. However, in accepting such obligations, You may act only\n      on Your own behalf and on Your sole responsibility, not on behalf\n      of any other Contributor, and only if You agree to indemnify,\n      defend, and hold each Contributor harmless for any liability\n      incurred by, or claims asserted against, such Contributor by reason\n      of your accepting any such warranty or additional liability.\n\n   END OF TERMS AND CONDITIONS\n\n   APPENDIX: How to apply the Apache License to your work.\n\n      To apply the Apache License to your work, attach the following\n      boilerplate notice, with the fields enclosed by brackets \"[]\"\n      replaced with your own identifying information. (Don't include\n      the brackets!)  The text should be enclosed in the appropriate\n      comment syntax for the file format. We also recommend that a\n      file or class name and description of purpose be included on the\n      same \"printed page\" as the copyright notice for easier\n      identification within third-party archives.\n\n   Copyright [yyyy] [name of copyright owner]\n\n   Licensed under the Apache License, Version 2.0 (the \"License\");\n   you may not use this file except in compliance with the License.\n   You may obtain a copy of the License at\n\n       http://www.apache.org/licenses/LICENSE-2.0\n\n   Unless required by applicable law or agreed to in writing, software\n   distributed under the License is distributed on an \"AS IS\" BASIS,\n   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n   See the License for the specific language governing permissions and\n   limitations under the License.\n"
  },
  {
    "path": "pingora-timeout/benches/benchmark.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\nuse pingora_timeout::*;\nuse std::time::{Duration, Instant};\nuse tokio::time::sleep;\nuse tokio::time::timeout as tokio_timeout;\n\nconst LOOP_SIZE: u32 = 100000;\n\nasync fn bench_timeout() -> u32 {\n    let mut n = 0;\n    for _ in 0..LOOP_SIZE {\n        let fut = async { 1 };\n        let to = timeout(Duration::from_secs(1), fut);\n        n += to.await.unwrap();\n    }\n    n\n}\n\nasync fn bench_tokio_timeout() -> u32 {\n    let mut n = 0;\n    for _ in 0..LOOP_SIZE {\n        let fut = async { 1 };\n        let to = tokio_timeout(Duration::from_secs(1), fut);\n        n += to.await.unwrap();\n    }\n    n\n}\n\nasync fn bench_fast_timeout() -> u32 {\n    let mut n = 0;\n    for _ in 0..LOOP_SIZE {\n        let fut = async { 1 };\n        let to = fast_timeout::fast_timeout(Duration::from_secs(1), fut);\n        n += to.await.unwrap();\n    }\n    n\n}\n\nfn bench_tokio_timer() {\n    let mut list = Vec::with_capacity(LOOP_SIZE as usize);\n    let before = Instant::now();\n    for _ in 0..LOOP_SIZE {\n        list.push(sleep(Duration::from_secs(1)));\n    }\n    let elapsed = before.elapsed();\n    println!(\n        \"tokio timer create {:?} total, {:?} avg per iteration\",\n        elapsed,\n        elapsed / LOOP_SIZE\n    );\n\n    let before = Instant::now();\n    drop(list);\n    let elapsed = before.elapsed();\n    println!(\n        \"tokio timer drop {:?} total, {:?} avg per iteration\",\n        elapsed,\n        elapsed / LOOP_SIZE\n    );\n}\n\nasync fn bench_multi_thread_tokio_timer(threads: usize) {\n    let mut handlers = vec![];\n    for _ in 0..threads {\n        let handler = tokio::spawn(async {\n            bench_tokio_timer();\n        });\n        handlers.push(handler);\n    }\n    for thread in handlers {\n        thread.await.unwrap();\n    }\n}\n\nuse std::sync::Arc;\n\nasync fn bench_multi_thread_timer(threads: usize, tm: Arc<TimerManager>) {\n    let mut handlers = vec![];\n    for _ in 0..threads {\n        let tm_ref = tm.clone();\n        let handler = tokio::spawn(async move {\n            bench_timer(&tm_ref);\n        });\n        handlers.push(handler);\n    }\n    for thread in handlers {\n        thread.await.unwrap();\n    }\n}\n\nuse pingora_timeout::timer::TimerManager;\n\nfn bench_timer(tm: &TimerManager) {\n    let mut list = Vec::with_capacity(LOOP_SIZE as usize);\n    let before = Instant::now();\n    for _ in 0..LOOP_SIZE {\n        list.push(tm.register_timer(Duration::from_secs(1)));\n    }\n    let elapsed = before.elapsed();\n    println!(\n        \"pingora timer create {:?} total, {:?} avg per iteration\",\n        elapsed,\n        elapsed / LOOP_SIZE\n    );\n\n    let before = Instant::now();\n    drop(list);\n    let elapsed = before.elapsed();\n    println!(\n        \"pingora timer drop {:?} total, {:?} avg per iteration\",\n        elapsed,\n        elapsed / LOOP_SIZE\n    );\n}\n\n#[tokio::main(worker_threads = 4)]\nasync fn main() {\n    let before = Instant::now();\n    bench_timeout().await;\n    let elapsed = before.elapsed();\n    println!(\n        \"pingora timeout {:?} total, {:?} avg per iteration\",\n        elapsed,\n        elapsed / LOOP_SIZE\n    );\n\n    let before = Instant::now();\n    bench_fast_timeout().await;\n    let elapsed = before.elapsed();\n    println!(\n        \"pingora fast timeout {:?} total, {:?} avg per iteration\",\n        elapsed,\n        elapsed / LOOP_SIZE\n    );\n\n    let before = Instant::now();\n    bench_tokio_timeout().await;\n    let elapsed = before.elapsed();\n    println!(\n        \"tokio timeout {:?} total, {:?} avg per iteration\",\n        elapsed,\n        elapsed / LOOP_SIZE\n    );\n\n    println!(\"===========================\");\n\n    let tm = pingora_timeout::timer::TimerManager::new();\n    bench_timer(&tm);\n    bench_tokio_timer();\n\n    println!(\"===========================\");\n\n    let tm = Arc::new(tm);\n    bench_multi_thread_timer(4, tm).await;\n    bench_multi_thread_tokio_timer(4).await;\n}\n"
  },
  {
    "path": "pingora-timeout/src/fast_timeout.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! The fast and more complicated version of pingora-timeout\n//!\n//! The following optimizations are applied:\n//! - The timeouts lazily initialize their timer when the Future is pending for the first time.\n//! - There is no global lock for creating and cancelling timeouts.\n//! - Timeout timers are rounded to the next 10ms tick and timers are shared across all timeouts with the same deadline.\n//!\n//! In order for this to work, a standalone thread is created to arm the timers, which has some\n//! overhead. As a general rule, the benefits of this don't outweigh the overhead unless\n//! there are more than about 100 timeout() calls/sec in the system. Use regular tokio timeout or\n//! [super::tokio_timeout] in the low usage case.\n\nuse super::timer::*;\nuse super::*;\nuse once_cell::sync::Lazy;\nuse std::sync::Arc;\n\nstatic TIMER_MANAGER: Lazy<Arc<TimerManager>> = Lazy::new(|| {\n    let tm = Arc::new(TimerManager::new());\n    check_clock_thread(&tm);\n    tm\n});\n\nfn check_clock_thread(tm: &Arc<TimerManager>) {\n    if tm.should_i_start_clock() {\n        std::thread::Builder::new()\n            .name(\"Timer thread\".into())\n            .spawn(|| TIMER_MANAGER.clock_thread())\n            .unwrap();\n    }\n}\n\n/// The timeout generated by [fast_timeout()].\n///\n/// Users don't need to interact with this object.\npub struct FastTimeout(Duration);\n\nimpl ToTimeout for FastTimeout {\n    fn timeout(&self) -> Pin<Box<dyn Future<Output = ()> + Send + Sync>> {\n        Box::pin(TIMER_MANAGER.register_timer(self.0).poll())\n    }\n\n    fn create(d: Duration) -> Self {\n        FastTimeout(d)\n    }\n}\n\n/// Similar to [tokio::time::timeout] but more efficient.\npub fn fast_timeout<T>(duration: Duration, future: T) -> Timeout<T, FastTimeout>\nwhere\n    T: Future,\n{\n    check_clock_thread(&TIMER_MANAGER);\n    Timeout::new_with_delay(future, duration)\n}\n\n/// Similar to [tokio::time::sleep] but more efficient.\npub async fn fast_sleep(duration: Duration) {\n    check_clock_thread(&TIMER_MANAGER);\n    TIMER_MANAGER.register_timer(duration).poll().await\n}\n\n/// Pause the timer for fork()\n///\n/// Because RwLock across fork() is undefined behavior, this function makes sure that no one\n/// holds any locks.\n///\n/// This function should be called right before fork().\npub fn pause_for_fork() {\n    TIMER_MANAGER.pause_for_fork();\n}\n\n/// Unpause the timer after fork()\n///\n/// This function should be called right after fork().\npub fn unpause() {\n    TIMER_MANAGER.unpause();\n}\n\n#[cfg(test)]\nmod tests {\n    use super::*;\n\n    #[tokio::test]\n    async fn test_timeout() {\n        let fut = tokio_sleep(Duration::from_secs(1000));\n        let to = fast_timeout(Duration::from_secs(1), fut);\n        assert!(to.await.is_err())\n    }\n\n    #[tokio::test]\n    async fn test_instantly_return() {\n        let fut = async { 1 };\n        let to = fast_timeout(Duration::from_secs(1), fut);\n        assert_eq!(to.await.unwrap(), 1)\n    }\n\n    #[tokio::test]\n    async fn test_delayed_return() {\n        let fut = async {\n            tokio_sleep(Duration::from_secs(1)).await;\n            1\n        };\n        let to = fast_timeout(Duration::from_secs(1000), fut);\n        assert_eq!(to.await.unwrap(), 1)\n    }\n\n    #[tokio::test]\n    async fn test_sleep() {\n        let fut = async {\n            fast_sleep(Duration::from_secs(1)).await;\n            1\n        };\n        let to = fast_timeout(Duration::from_secs(1000), fut);\n        assert_eq!(to.await.unwrap(), 1)\n    }\n}\n"
  },
  {
    "path": "pingora-timeout/src/lib.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n#![warn(clippy::all)]\n\n//! A drop-in replacement of [tokio::time::timeout] which is much more efficient.\n//!\n//! Similar to [tokio::time::timeout] but more efficient on busy concurrent IOs where timeouts are\n//! created and canceled very frequently.\n//!\n//! This crate provides the following optimizations\n//! - The timeouts lazily initializes their timer when the Future is pending for the first time.\n//! - There is no global lock for creating and cancelling timeouts.\n//! - Timeout timers are rounded to the next 10ms tick and timers are shared across all timeouts with the same deadline.\n//!\n//! Benchmark:\n//!\n//! 438.302µs total, 4ns avg per iteration\n//!\n//! v.s. Tokio timeout():\n//!\n//! 10.716192ms total, 107ns avg per iteration\n//!\n\npub mod fast_timeout;\npub mod timer;\n\npub use fast_timeout::fast_sleep as sleep;\npub use fast_timeout::fast_timeout as timeout;\n\nuse pin_project_lite::pin_project;\nuse std::future::Future;\nuse std::pin::Pin;\nuse std::task::{self, Poll};\nuse tokio::time::{sleep as tokio_sleep, Duration};\n\n/// The interface to start a timeout\n///\n/// Users don't need to interact with this trait\npub trait ToTimeout {\n    fn timeout(&self) -> Pin<Box<dyn Future<Output = ()> + Send + Sync>>;\n    fn create(d: Duration) -> Self;\n}\n\n/// The timeout generated by [tokio_timeout()].\n///\n/// Users don't need to interact with this object.\npub struct TokioTimeout(Duration);\n\nimpl ToTimeout for TokioTimeout {\n    fn timeout(&self) -> Pin<Box<dyn Future<Output = ()> + Send + Sync>> {\n        Box::pin(tokio_sleep(self.0))\n    }\n\n    fn create(d: Duration) -> Self {\n        TokioTimeout(d)\n    }\n}\n\n/// The error type returned when the timeout is reached.\n#[derive(Debug)]\npub struct Elapsed;\n\nimpl std::fmt::Display for Elapsed {\n    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {\n        write!(f, \"Timeout Elapsed\")\n    }\n}\n\nimpl std::error::Error for Elapsed {}\n\n/// The [tokio::time::timeout] with just lazy timer initialization.\n///\n/// The timer is created the first time the `future` is pending. This avoids unnecessary timer\n/// creation and cancellation on busy IOs with a good chance to be already ready (e.g., reading\n/// data from TCP where the recv buffer already has a lot of data to read right away).\npub fn tokio_timeout<T>(duration: Duration, future: T) -> Timeout<T, TokioTimeout>\nwhere\n    T: Future,\n{\n    Timeout::<T, TokioTimeout>::new_with_delay(future, duration)\n}\n\npin_project! {\n    /// The timeout future returned by the timeout functions\n    #[must_use = \"futures do nothing unless you `.await` or poll them\"]\n    pub struct Timeout<T, F> {\n        #[pin]\n        value: T,\n        #[pin]\n        delay: Option<Pin<Box<dyn Future<Output = ()> + Send + Sync>>>,\n        callback: F, // callback to create the timer\n    }\n}\n\nimpl<T, F> Timeout<T, F>\nwhere\n    F: ToTimeout,\n{\n    pub(crate) fn new_with_delay(value: T, d: Duration) -> Timeout<T, F> {\n        Timeout {\n            value,\n            delay: None,\n            callback: F::create(d),\n        }\n    }\n}\n\nimpl<T, F> Future for Timeout<T, F>\nwhere\n    T: Future,\n    F: ToTimeout,\n{\n    type Output = Result<T::Output, Elapsed>;\n\n    fn poll(self: Pin<&mut Self>, cx: &mut task::Context<'_>) -> Poll<Self::Output> {\n        let mut me = self.project();\n\n        // First, try polling the future\n        if let Poll::Ready(v) = me.value.poll(cx) {\n            return Poll::Ready(Ok(v));\n        }\n\n        let delay = me\n            .delay\n            .get_or_insert_with(|| Box::pin(me.callback.timeout()));\n\n        match delay.as_mut().poll(cx) {\n            Poll::Pending => Poll::Pending,\n            Poll::Ready(()) => Poll::Ready(Err(Elapsed {})),\n        }\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use super::*;\n\n    #[tokio::test]\n    async fn test_timeout() {\n        let fut = tokio_sleep(Duration::from_secs(1000));\n        let to = timeout(Duration::from_secs(1), fut);\n        assert!(to.await.is_err())\n    }\n\n    #[tokio::test]\n    async fn test_instantly_return() {\n        let fut = async { 1 };\n        let to = timeout(Duration::from_secs(1), fut);\n        assert_eq!(to.await.unwrap(), 1)\n    }\n\n    #[tokio::test]\n    async fn test_delayed_return() {\n        let fut = async {\n            tokio_sleep(Duration::from_secs(1)).await;\n            1\n        };\n        let to = timeout(Duration::from_secs(1000), fut);\n        assert_eq!(to.await.unwrap(), 1)\n    }\n}\n"
  },
  {
    "path": "pingora-timeout/src/timer.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! Lightweight timer for systems with high rate of operations with timeout\n//! associated with them\n//!\n//! Users don't need to interact with this module.\n//!\n//! The idea is to bucket timers into finite time slots so that operations that\n//! start and end quickly don't have to create their own timers all the time\n//!\n//! Benchmark:\n//! - create 7.809622ms total, 78ns avg per iteration\n//! - drop: 1.348552ms total, 13ns avg per iteration\n//!\n//! tokio timer:\n//! - create 34.317439ms total, 343ns avg per iteration\n//! - drop: 10.694154ms total, 106ns avg per iteration\n\nuse parking_lot::RwLock;\nuse std::collections::BTreeMap;\nuse std::sync::atomic::{AtomicBool, AtomicI64, Ordering};\nuse std::sync::Arc;\nuse std::time::{Duration, Instant};\nuse thread_local::ThreadLocal;\nuse tokio::sync::Notify;\n\nconst RESOLUTION_MS: u64 = 10;\nconst RESOLUTION_DURATION: Duration = Duration::from_millis(RESOLUTION_MS);\n\n// round to the NEXT timestamp based on the resolution\n#[inline]\nfn round_to(raw: u128, resolution: u128) -> u128 {\n    raw - 1 + resolution - (raw - 1) % resolution\n}\n// millisecond resolution as most\n#[derive(PartialEq, PartialOrd, Eq, Ord, Clone, Copy, Debug)]\nstruct Time(u128);\n\nimpl From<u128> for Time {\n    fn from(raw_ms: u128) -> Self {\n        Time(round_to(raw_ms, RESOLUTION_MS as u128))\n    }\n}\n\nimpl From<Duration> for Time {\n    fn from(d: Duration) -> Self {\n        Time(round_to(d.as_millis(), RESOLUTION_MS as u128))\n    }\n}\n\nimpl Time {\n    pub fn not_after(&self, ts: u128) -> bool {\n        self.0 <= ts\n    }\n}\n\n/// the stub for waiting for a timer to be expired.\npub struct TimerStub(Arc<Notify>, Arc<AtomicBool>);\n\nimpl TimerStub {\n    /// Wait for the timer to expire.\n    pub async fn poll(self) {\n        if self.1.load(Ordering::SeqCst) {\n            return;\n        }\n        self.0.notified().await;\n    }\n}\n\nstruct Timer(Arc<Notify>, Arc<AtomicBool>);\n\nimpl Timer {\n    pub fn new() -> Self {\n        Timer(Arc::new(Notify::new()), Arc::new(AtomicBool::new(false)))\n    }\n\n    pub fn fire(&self) {\n        self.1.store(true, Ordering::SeqCst);\n        self.0.notify_waiters();\n    }\n\n    pub fn subscribe(&self) -> TimerStub {\n        TimerStub(self.0.clone(), self.1.clone())\n    }\n}\n\n/// The object that holds all the timers registered to it.\npub struct TimerManager {\n    // each thread insert into its local timer tree to avoid lock contention\n    timers: ThreadLocal<RwLock<BTreeMap<Time, Timer>>>,\n    zero: Instant, // the reference zero point of Timestamp\n    // Start a new clock thread if this is -1 or staled. The clock thread should keep updating this\n    clock_watchdog: AtomicI64,\n    paused: AtomicBool,\n}\n\n// Consider the clock thread is dead after it fails to update the thread in DELAYS_SEC\nconst DELAYS_SEC: i64 = 2; // TODO: make sure this value is larger than RESOLUTION_DURATION\n\nimpl Default for TimerManager {\n    fn default() -> Self {\n        TimerManager {\n            timers: ThreadLocal::new(),\n            zero: Instant::now(),\n            clock_watchdog: AtomicI64::new(-DELAYS_SEC),\n            paused: AtomicBool::new(false),\n        }\n    }\n}\n\nimpl TimerManager {\n    /// Create a new [TimerManager]\n    pub fn new() -> Self {\n        Self::default()\n    }\n\n    // This thread sleeps for a resolution time and then fires all the timers that are due to fire\n    pub(crate) fn clock_thread(&self) {\n        loop {\n            std::thread::sleep(RESOLUTION_DURATION);\n            let now = Instant::now() - self.zero;\n            self.clock_watchdog\n                .store(now.as_secs() as i64, Ordering::Relaxed);\n            if self.is_paused_for_fork() {\n                // just stop acquiring the locks, waiting for fork to happen\n                continue;\n            }\n            let now = now.as_millis();\n            // iterate through the timer tree for all threads\n            for thread_timer in self.timers.iter() {\n                let mut timers = thread_timer.write();\n                // Fire all timers until now\n                loop {\n                    let key_to_remove = timers.iter().next().and_then(|(k, _)| {\n                        if k.not_after(now) {\n                            Some(*k)\n                        } else {\n                            None\n                        }\n                    });\n                    if let Some(k) = key_to_remove {\n                        let timer = timers.remove(&k);\n                        // safe to unwrap, the key is from iter().next()\n                        timer.unwrap().fire();\n                    } else {\n                        break;\n                    }\n                }\n                // write lock drops here\n            }\n        }\n    }\n\n    // False if the clock is already started\n    // If true, the caller must start the clock thread next\n    pub(crate) fn should_i_start_clock(&self) -> bool {\n        let Err(prev) = self.is_clock_running() else {\n            return false;\n        };\n        let now = Instant::now().duration_since(self.zero).as_secs() as i64;\n        let res =\n            self.clock_watchdog\n                .compare_exchange(prev, now, Ordering::SeqCst, Ordering::SeqCst);\n        res.is_ok()\n    }\n\n    // Ok(()) if clock is running (watch dog is within DELAYS_SEC of now)\n    // Err(time) if watch do stopped at `time`\n    pub(crate) fn is_clock_running(&self) -> Result<(), i64> {\n        let now = Instant::now().duration_since(self.zero).as_secs() as i64;\n        let prev = self.clock_watchdog.load(Ordering::SeqCst);\n        if now < prev + DELAYS_SEC {\n            Ok(())\n        } else {\n            Err(prev)\n        }\n    }\n\n    /// Register a timer.\n    ///\n    /// When the timer expires, the [TimerStub] will be notified.\n    pub fn register_timer(&self, duration: Duration) -> TimerStub {\n        if self.is_paused_for_fork() {\n            // Return a dummy TimerStub that will trigger right away.\n            // This is fine assuming pause_for_fork() is called right before fork().\n            // The only possible register_timer() is from another thread which will\n            // be entirely lost after fork()\n            // TODO: buffer these register calls instead (without a lock)\n            let timer = Timer::new();\n            timer.fire();\n            return timer.subscribe();\n        }\n        let now: Time = (Instant::now() + duration - self.zero).into();\n        {\n            let timers = self.timers.get_or(|| RwLock::new(BTreeMap::new())).read();\n            if let Some(t) = timers.get(&now) {\n                return t.subscribe();\n            }\n        } // drop read lock\n\n        let timer = Timer::new();\n        let mut timers = self.timers.get_or(|| RwLock::new(BTreeMap::new())).write();\n        // Usually we check if another thread has insert the same node before we get the write lock,\n        // but because only this thread will insert anything to its local timers tree, there\n        // is no possible race that can happen. The only other thread is the clock thread who\n        // only removes timer from the tree\n        let stub = timer.subscribe();\n        timers.insert(now, timer);\n        stub\n    }\n\n    fn is_paused_for_fork(&self) -> bool {\n        self.paused.load(Ordering::SeqCst)\n    }\n\n    /// Pause the timer for fork()\n    ///\n    /// Because RwLock across fork() is undefined behavior, this function makes sure that no one\n    /// holds any locks.\n    ///\n    /// This function should be called right before fork().\n    pub fn pause_for_fork(&self) {\n        self.paused.store(true, Ordering::SeqCst);\n        // wait for everything to get out of their locks\n        std::thread::sleep(RESOLUTION_DURATION * 2);\n    }\n\n    /// Unpause the timer after fork()\n    ///\n    /// This function should be called right after fork().\n    pub fn unpause(&self) {\n        self.paused.store(false, Ordering::SeqCst)\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use super::*;\n\n    #[test]\n    fn test_round() {\n        assert_eq!(round_to(30, 10), 30);\n        assert_eq!(round_to(31, 10), 40);\n        assert_eq!(round_to(29, 10), 30);\n    }\n\n    #[test]\n    fn test_time() {\n        let t: Time = 128.into(); // t will round to 130\n        assert_eq!(t, Duration::from_millis(130).into());\n        assert!(!t.not_after(128));\n        assert!(!t.not_after(129));\n        assert!(t.not_after(130));\n        assert!(t.not_after(131));\n    }\n\n    #[tokio::test]\n    async fn test_timer_manager() {\n        let tm_a = Arc::new(TimerManager::new());\n        let tm = tm_a.clone();\n        std::thread::spawn(move || tm_a.clock_thread());\n\n        let now = Instant::now();\n        let t1 = tm.register_timer(Duration::from_secs(1));\n        let t2 = tm.register_timer(Duration::from_secs(1));\n        t1.poll().await;\n        assert_eq!(now.elapsed().as_secs(), 1);\n        let now = Instant::now();\n        t2.poll().await;\n        // t2 fired along t1 so no extra wait time\n        assert_eq!(now.elapsed().as_secs(), 0);\n    }\n\n    #[test]\n    fn test_timer_manager_start_check() {\n        let tm = Arc::new(TimerManager::new());\n        assert!(tm.should_i_start_clock());\n        assert!(!tm.should_i_start_clock());\n        assert!(tm.is_clock_running().is_ok());\n    }\n\n    #[test]\n    fn test_timer_manager_watchdog() {\n        let tm = Arc::new(TimerManager::new());\n        assert!(tm.should_i_start_clock());\n        assert!(!tm.should_i_start_clock());\n\n        // we don't actually start the clock thread, sleep for the watchdog to expire\n        std::thread::sleep(Duration::from_secs(DELAYS_SEC as u64 + 1));\n        assert!(tm.is_clock_running().is_err());\n        assert!(tm.should_i_start_clock());\n    }\n\n    #[tokio::test]\n    async fn test_timer_manager_pause() {\n        let tm_a = Arc::new(TimerManager::new());\n        let tm = tm_a.clone();\n        std::thread::spawn(move || tm_a.clock_thread());\n\n        let now = Instant::now();\n        let t1 = tm.register_timer(Duration::from_secs(2));\n        tm.pause_for_fork();\n        // no actual fork happen, we just test that pause and unpause work\n\n        // any timer in this critical section is timed out right away\n        let t2 = tm.register_timer(Duration::from_secs(2));\n        t2.poll().await;\n        assert_eq!(now.elapsed().as_secs(), 0);\n\n        std::thread::sleep(Duration::from_secs(1));\n        tm.unpause();\n        t1.poll().await;\n        assert_eq!(now.elapsed().as_secs(), 2);\n    }\n}\n"
  },
  {
    "path": "tinyufo/Cargo.toml",
    "content": "[package]\nname = \"TinyUFO\"\nversion = \"0.8.0\"\nauthors = [\"Yuchen Wu <yuchen@cloudflare.com>\"]\nedition = \"2021\"\nlicense = \"Apache-2.0\"\ndescription = \"In-memory cache implementation with TinyLFU as the admission policy and S3-FIFO as the eviction policy\"\nrepository = \"https://github.com/cloudflare/pingora\"\ncategories = [\"algorithms\", \"caching\"]\nkeywords = [\"cache\", \"pingora\"]\n\n# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html\n\n[lib]\nname = \"tinyufo\"\npath = \"src/lib.rs\"\n\n[dependencies]\nahash = { workspace = true }\nflurry = \"0.5\"\ncrossbeam-queue = \"0\"\ncrossbeam-skiplist = \"0\"\n\n[dev-dependencies]\nrand = \"0.9\"\nlru = { workspace = true }\nrand_distr = \"0.5\"\nmoka = { version = \"0\", features = [\"sync\"] }\ndhat = \"0\"\nquick_cache = \"0.6\"\ntriomphe = \"<=0.1.11\" # 0.1.12 requires Rust 1.76\n\n[[bench]]\nname = \"bench_perf\"\nharness = false\n\n[[bench]]\nname = \"bench_hit_ratio\"\nharness = false\n\n[[bench]]\nname = \"bench_memory\"\nharness = false\n"
  },
  {
    "path": "tinyufo/LICENSE",
    "content": "\n                                 Apache License\n                           Version 2.0, January 2004\n                        http://www.apache.org/licenses/\n\n   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION\n\n   1. Definitions.\n\n      \"License\" shall mean the terms and conditions for use, reproduction,\n      and distribution as defined by Sections 1 through 9 of this document.\n\n      \"Licensor\" shall mean the copyright owner or entity authorized by\n      the copyright owner that is granting the License.\n\n      \"Legal Entity\" shall mean the union of the acting entity and all\n      other entities that control, are controlled by, or are under common\n      control with that entity. For the purposes of this definition,\n      \"control\" means (i) the power, direct or indirect, to cause the\n      direction or management of such entity, whether by contract or\n      otherwise, or (ii) ownership of fifty percent (50%) or more of the\n      outstanding shares, or (iii) beneficial ownership of such entity.\n\n      \"You\" (or \"Your\") shall mean an individual or Legal Entity\n      exercising permissions granted by this License.\n\n      \"Source\" form shall mean the preferred form for making modifications,\n      including but not limited to software source code, documentation\n      source, and configuration files.\n\n      \"Object\" form shall mean any form resulting from mechanical\n      transformation or translation of a Source form, including but\n      not limited to compiled object code, generated documentation,\n      and conversions to other media types.\n\n      \"Work\" shall mean the work of authorship, whether in Source or\n      Object form, made available under the License, as indicated by a\n      copyright notice that is included in or attached to the work\n      (an example is provided in the Appendix below).\n\n      \"Derivative Works\" shall mean any work, whether in Source or Object\n      form, that is based on (or derived from) the Work and for which the\n      editorial revisions, annotations, elaborations, or other modifications\n      represent, as a whole, an original work of authorship. For the purposes\n      of this License, Derivative Works shall not include works that remain\n      separable from, or merely link (or bind by name) to the interfaces of,\n      the Work and Derivative Works thereof.\n\n      \"Contribution\" shall mean any work of authorship, including\n      the original version of the Work and any modifications or additions\n      to that Work or Derivative Works thereof, that is intentionally\n      submitted to Licensor for inclusion in the Work by the copyright owner\n      or by an individual or Legal Entity authorized to submit on behalf of\n      the copyright owner. For the purposes of this definition, \"submitted\"\n      means any form of electronic, verbal, or written communication sent\n      to the Licensor or its representatives, including but not limited to\n      communication on electronic mailing lists, source code control systems,\n      and issue tracking systems that are managed by, or on behalf of, the\n      Licensor for the purpose of discussing and improving the Work, but\n      excluding communication that is conspicuously marked or otherwise\n      designated in writing by the copyright owner as \"Not a Contribution.\"\n\n      \"Contributor\" shall mean Licensor and any individual or Legal Entity\n      on behalf of whom a Contribution has been received by Licensor and\n      subsequently incorporated within the Work.\n\n   2. Grant of Copyright License. Subject to the terms and conditions of\n      this License, each Contributor hereby grants to You a perpetual,\n      worldwide, non-exclusive, no-charge, royalty-free, irrevocable\n      copyright license to reproduce, prepare Derivative Works of,\n      publicly display, publicly perform, sublicense, and distribute the\n      Work and such Derivative Works in Source or Object form.\n\n   3. Grant of Patent License. Subject to the terms and conditions of\n      this License, each Contributor hereby grants to You a perpetual,\n      worldwide, non-exclusive, no-charge, royalty-free, irrevocable\n      (except as stated in this section) patent license to make, have made,\n      use, offer to sell, sell, import, and otherwise transfer the Work,\n      where such license applies only to those patent claims licensable\n      by such Contributor that are necessarily infringed by their\n      Contribution(s) alone or by combination of their Contribution(s)\n      with the Work to which such Contribution(s) was submitted. If You\n      institute patent litigation against any entity (including a\n      cross-claim or counterclaim in a lawsuit) alleging that the Work\n      or a Contribution incorporated within the Work constitutes direct\n      or contributory patent infringement, then any patent licenses\n      granted to You under this License for that Work shall terminate\n      as of the date such litigation is filed.\n\n   4. Redistribution. You may reproduce and distribute copies of the\n      Work or Derivative Works thereof in any medium, with or without\n      modifications, and in Source or Object form, provided that You\n      meet the following conditions:\n\n      (a) You must give any other recipients of the Work or\n          Derivative Works a copy of this License; and\n\n      (b) You must cause any modified files to carry prominent notices\n          stating that You changed the files; and\n\n      (c) You must retain, in the Source form of any Derivative Works\n          that You distribute, all copyright, patent, trademark, and\n          attribution notices from the Source form of the Work,\n          excluding those notices that do not pertain to any part of\n          the Derivative Works; and\n\n      (d) If the Work includes a \"NOTICE\" text file as part of its\n          distribution, then any Derivative Works that You distribute must\n          include a readable copy of the attribution notices contained\n          within such NOTICE file, excluding those notices that do not\n          pertain to any part of the Derivative Works, in at least one\n          of the following places: within a NOTICE text file distributed\n          as part of the Derivative Works; within the Source form or\n          documentation, if provided along with the Derivative Works; or,\n          within a display generated by the Derivative Works, if and\n          wherever such third-party notices normally appear. The contents\n          of the NOTICE file are for informational purposes only and\n          do not modify the License. You may add Your own attribution\n          notices within Derivative Works that You distribute, alongside\n          or as an addendum to the NOTICE text from the Work, provided\n          that such additional attribution notices cannot be construed\n          as modifying the License.\n\n      You may add Your own copyright statement to Your modifications and\n      may provide additional or different license terms and conditions\n      for use, reproduction, or distribution of Your modifications, or\n      for any such Derivative Works as a whole, provided Your use,\n      reproduction, and distribution of the Work otherwise complies with\n      the conditions stated in this License.\n\n   5. Submission of Contributions. Unless You explicitly state otherwise,\n      any Contribution intentionally submitted for inclusion in the Work\n      by You to the Licensor shall be under the terms and conditions of\n      this License, without any additional terms or conditions.\n      Notwithstanding the above, nothing herein shall supersede or modify\n      the terms of any separate license agreement you may have executed\n      with Licensor regarding such Contributions.\n\n   6. Trademarks. This License does not grant permission to use the trade\n      names, trademarks, service marks, or product names of the Licensor,\n      except as required for reasonable and customary use in describing the\n      origin of the Work and reproducing the content of the NOTICE file.\n\n   7. Disclaimer of Warranty. Unless required by applicable law or\n      agreed to in writing, Licensor provides the Work (and each\n      Contributor provides its Contributions) on an \"AS IS\" BASIS,\n      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or\n      implied, including, without limitation, any warranties or conditions\n      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A\n      PARTICULAR PURPOSE. You are solely responsible for determining the\n      appropriateness of using or redistributing the Work and assume any\n      risks associated with Your exercise of permissions under this License.\n\n   8. Limitation of Liability. In no event and under no legal theory,\n      whether in tort (including negligence), contract, or otherwise,\n      unless required by applicable law (such as deliberate and grossly\n      negligent acts) or agreed to in writing, shall any Contributor be\n      liable to You for damages, including any direct, indirect, special,\n      incidental, or consequential damages of any character arising as a\n      result of this License or out of the use or inability to use the\n      Work (including but not limited to damages for loss of goodwill,\n      work stoppage, computer failure or malfunction, or any and all\n      other commercial damages or losses), even if such Contributor\n      has been advised of the possibility of such damages.\n\n   9. Accepting Warranty or Additional Liability. While redistributing\n      the Work or Derivative Works thereof, You may choose to offer,\n      and charge a fee for, acceptance of support, warranty, indemnity,\n      or other liability obligations and/or rights consistent with this\n      License. However, in accepting such obligations, You may act only\n      on Your own behalf and on Your sole responsibility, not on behalf\n      of any other Contributor, and only if You agree to indemnify,\n      defend, and hold each Contributor harmless for any liability\n      incurred by, or claims asserted against, such Contributor by reason\n      of your accepting any such warranty or additional liability.\n\n   END OF TERMS AND CONDITIONS\n\n   APPENDIX: How to apply the Apache License to your work.\n\n      To apply the Apache License to your work, attach the following\n      boilerplate notice, with the fields enclosed by brackets \"[]\"\n      replaced with your own identifying information. (Don't include\n      the brackets!)  The text should be enclosed in the appropriate\n      comment syntax for the file format. We also recommend that a\n      file or class name and description of purpose be included on the\n      same \"printed page\" as the copyright notice for easier\n      identification within third-party archives.\n\n   Copyright [yyyy] [name of copyright owner]\n\n   Licensed under the Apache License, Version 2.0 (the \"License\");\n   you may not use this file except in compliance with the License.\n   You may obtain a copy of the License at\n\n       http://www.apache.org/licenses/LICENSE-2.0\n\n   Unless required by applicable law or agreed to in writing, software\n   distributed under the License is distributed on an \"AS IS\" BASIS,\n   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n   See the License for the specific language governing permissions and\n   limitations under the License.\n"
  },
  {
    "path": "tinyufo/README.md",
    "content": "# TinyUFO\n\nTinyUFO is a fast and efficient in-memory cache. It adopts the state-of-the-art [S3-FIFO](https://s3fifo.com/) as well as [TinyLFU](https://arxiv.org/abs/1512.00727) algorithms to achieve high throughput and high hit ratio as the same time.\n\n## Usage\n\nSee docs\n\n## Performance Comparison\nWe compare TinyUFO with [lru](https://crates.io/crates/lru), the most commonly used cache algorithm and [moka](https://crates.io/crates/moka), another [great](https://github.com/rust-lang/crates.io/pull/3999) cache library that implements TinyLFU.\n\n### Hit Ratio\n\nThe table below show the cache hit ratio of the compared algorithm under different size of cache, zipf=1.\n\n|cache size / total assets | TinyUFO | TinyUFO - LRU | TinyUFO - moka (TinyLFU) |\n| -------- | ------- | ------- | ------ |\n| 0.5% | 45.26% | +14.21pp | -0.33pp\n| 1% | 52.35% | +13.19pp | +1.69pp\n| 5% | 68.89% | +10.14pp | +1.91pp\n| 10% | 75.98% | +8.39pp | +1.59pp\n| 25% | 85.34% | +5.39pp | +0.95pp\n\nBoth TinyUFO and moka greatly improves hit ratio from lru. TinyUFO is the one better in this workload.\n[This paper](https://dl.acm.org/doi/pdf/10.1145/3600006.3613147) contains more thorough cache performance\nevaluations S3-FIFO, which TinyUFO varies from, against many caching algorithms under a variety of workloads.\n\n### Speed\n\nThe table below shows the number of operations performed per second for each cache library. The tests are performed using 8 threads on a x64 Linux desktop.\n\n| Setup | TinyUFO | LRU | moka |\n| -------- | ------- | ------- | ------ |\n| Pure read | 148.7 million ops | 7.0 million ops | 14.1 million ops\n| Mixed read/write | 80.9 million ops | 6.8 million ops | 16.6 million ops\n\nBecause of TinyUFO's lock-free design, it greatly outperforms the others.\n\n### Memory overhead\n\nTinyUFO provides a compact mode to trade raw read speed for more memory efficiency. Whether the saving worthy the trade off depends on the actual size and the work load. For small in-memory assets, the saved memory means more things can be cached.\n\nThe table below show the memory allocation (in bytes) of the compared cache library under certain workloads to store zero-sized assets.\n\n| cache size | TinyUFO | TinyUFO compact | LRU | moka |\n| -------- | ------- | ------- | ------- | ------ |\n| 100 | 39,409 | 19,000 | 9,408 | 354,376\n| 1000 | 236,053 | 86,352 | 128,512 | 535,888\n| 10000 | 2,290,635 | 766,024|  1,075,648 | 2,489,088"
  },
  {
    "path": "tinyufo/benches/bench_hit_ratio.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\nuse rand::prelude::*;\nuse std::num::NonZeroUsize;\n\nconst ITEMS: usize = 10_000;\nconst ITERATIONS: usize = 5_000_000;\n\nfn bench_one(zip_exp: f64, cache_size_percent: f32) {\n    print!(\"{zip_exp:.2}, {cache_size_percent:4}\\t\\t\\t\");\n    let cache_size = (cache_size_percent * ITEMS as f32).round() as usize;\n    let mut lru = lru::LruCache::<u64, ()>::new(NonZeroUsize::new(cache_size).unwrap());\n    let moka = moka::sync::Cache::new(cache_size as u64);\n    let quick_cache = quick_cache::sync::Cache::new(cache_size);\n    let tinyufo = tinyufo::TinyUfo::new(cache_size, cache_size);\n\n    let mut rng = rand::rng();\n    let zipf = rand_distr::Zipf::new(ITEMS as f64, zip_exp).unwrap();\n\n    let mut lru_hit = 0;\n    let mut moka_hit = 0;\n    let mut quick_cache_hit = 0;\n    let mut tinyufo_hit = 0;\n\n    for _ in 0..ITERATIONS {\n        let key = zipf.sample(&mut rng) as u64;\n\n        if lru.get(&key).is_some() {\n            lru_hit += 1;\n        } else {\n            lru.push(key, ());\n        }\n\n        if moka.get(&key).is_some() {\n            moka_hit += 1;\n        } else {\n            moka.insert(key, ());\n        }\n\n        if quick_cache.get(&key).is_some() {\n            quick_cache_hit += 1;\n        } else {\n            quick_cache.insert(key, ());\n        }\n\n        if tinyufo.get(&key).is_some() {\n            tinyufo_hit += 1;\n        } else {\n            tinyufo.put(key, (), 1);\n        }\n    }\n\n    print!(\"{:.2}%\\t\\t\", lru_hit as f32 / ITERATIONS as f32 * 100.0);\n    print!(\"{:.2}%\\t\\t\", moka_hit as f32 / ITERATIONS as f32 * 100.0);\n    print!(\n        \"{:.2}%\\t\\t\",\n        quick_cache_hit as f32 / ITERATIONS as f32 * 100.0\n    );\n    println!(\"{:.2}%\", tinyufo_hit as f32 / ITERATIONS as f32 * 100.0);\n}\n\n/*\ncargo bench --bench bench_hit_ratio\n\nzipf & cache size               lru             moka            QuickC          TinyUFO\n0.90, 0.005                     19.24%          33.43%          32.33%          33.35%\n0.90, 0.01                      26.23%          37.86%          38.80%          40.06%\n0.90, 0.05                      45.58%          55.13%          55.71%          57.80%\n0.90,  0.1                      55.72%          64.15%          64.01%          66.36%\n0.90, 0.25                      71.16%          77.12%          75.92%          78.53%\n1.00, 0.005                     31.08%          45.68%          44.07%          45.15%\n1.00, 0.01                      39.17%          50.80%          50.90%          52.30%\n1.00, 0.05                      58.71%          66.92%          67.09%          68.79%\n1.00,  0.1                      67.59%          74.28%          74.00%          75.92%\n1.00, 0.25                      79.94%          84.35%          83.45%          85.28%\n1.05, 0.005                     37.66%          51.78%          50.13%          51.12%\n1.05, 0.01                      46.07%          57.13%          57.07%          58.41%\n1.05, 0.05                      65.06%          72.37%          72.41%          73.93%\n1.05,  0.1                      73.13%          78.97%          78.60%          80.24%\n1.05, 0.25                      83.74%          87.41%          86.68%          88.14%\n1.10, 0.005                     44.49%          57.84%          56.16%          57.28%\n1.10, 0.01                      52.97%          63.19%          62.99%          64.24%\n1.10, 0.05                      70.95%          77.24%          77.26%          78.55%\n1.10,  0.1                      78.05%          82.86%          82.66%          84.01%\n1.10, 0.25                      87.12%          90.10%          89.51%          90.66%\n1.50, 0.005                     85.27%          89.92%          89.08%          89.69%\n1.50, 0.01                      89.86%          92.77%          92.44%          92.94%\n1.50, 0.05                      96.01%          97.08%          96.99%          97.23%\n1.50,  0.1                      97.51%          98.15%          98.08%          98.24%\n1.50, 0.25                      98.81%          99.09%          99.03%          99.09%\n */\n\nfn main() {\n    println!(\"zipf & cache size\\t\\tlru\\t\\tmoka\\t\\tQuickC\\t\\tTinyUFO\",);\n    for zif_exp in [0.9, 1.0, 1.05, 1.1, 1.5] {\n        for cache_capacity in [0.005, 0.01, 0.05, 0.1, 0.25] {\n            bench_one(zif_exp, cache_capacity);\n        }\n    }\n}\n"
  },
  {
    "path": "tinyufo/benches/bench_memory.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n#[global_allocator]\nstatic ALLOC: dhat::Alloc = dhat::Alloc;\n\nuse rand::prelude::*;\nuse std::num::NonZeroUsize;\n\nconst ITERATIONS: usize = 5_000_000;\n\nfn bench_lru(zip_exp: f64, items: usize, cache_size_percent: f32) {\n    let cache_size = (cache_size_percent * items as f32).round() as usize;\n    let mut lru = lru::LruCache::<u64, ()>::new(NonZeroUsize::new(cache_size).unwrap());\n\n    let mut rng = rand::rng();\n    let zipf = rand_distr::Zipf::new(items as f64, zip_exp).unwrap();\n\n    for _ in 0..ITERATIONS {\n        let key = zipf.sample(&mut rng) as u64;\n\n        if lru.get(&key).is_none() {\n            lru.push(key, ());\n        }\n    }\n}\n\nfn bench_moka(zip_exp: f64, items: usize, cache_size_percent: f32) {\n    let cache_size = (cache_size_percent * items as f32).round() as usize;\n    let moka = moka::sync::Cache::new(cache_size as u64);\n\n    let mut rng = rand::rng();\n    let zipf = rand_distr::Zipf::new(items as f64, zip_exp).unwrap();\n\n    for _ in 0..ITERATIONS {\n        let key = zipf.sample(&mut rng) as u64;\n\n        if moka.get(&key).is_none() {\n            moka.insert(key, ());\n        }\n    }\n}\n\nfn bench_quick_cache(zip_exp: f64, items: usize, cache_size_percent: f32) {\n    let cache_size = (cache_size_percent * items as f32).round() as usize;\n    let quick_cache = quick_cache::sync::Cache::new(cache_size);\n\n    let mut rng = rand::rng();\n    let zipf = rand_distr::Zipf::new(items as f64, zip_exp).unwrap();\n\n    for _ in 0..ITERATIONS {\n        let key = zipf.sample(&mut rng) as u64;\n\n        if quick_cache.get(&key).is_none() {\n            quick_cache.insert(key, ());\n        }\n    }\n}\n\nfn bench_tinyufo(zip_exp: f64, items: usize, cache_size_percent: f32) {\n    let cache_size = (cache_size_percent * items as f32).round() as usize;\n    let tinyufo = tinyufo::TinyUfo::new(cache_size, (cache_size as f32 * 1.0) as usize);\n\n    let mut rng = rand::rng();\n    let zipf = rand_distr::Zipf::new(items as f64, zip_exp).unwrap();\n\n    for _ in 0..ITERATIONS {\n        let key = zipf.sample(&mut rng) as u64;\n\n        if tinyufo.get(&key).is_none() {\n            tinyufo.put(key, (), 1);\n        }\n    }\n}\n\nfn bench_tinyufo_compact(zip_exp: f64, items: usize, cache_size_percent: f32) {\n    let cache_size = (cache_size_percent * items as f32).round() as usize;\n    let tinyufo = tinyufo::TinyUfo::new_compact(cache_size, (cache_size as f32 * 1.0) as usize);\n\n    let mut rng = rand::rng();\n    let zipf = rand_distr::Zipf::new(items as f64, zip_exp).unwrap();\n\n    for _ in 0..ITERATIONS {\n        let key = zipf.sample(&mut rng) as u64;\n\n        if tinyufo.get(&key).is_none() {\n            tinyufo.put(key, (), 1);\n        }\n    }\n}\n\n/*\ncargo bench --bench bench_memory\n\ntotal items 1000, cache size 10%\nlru\ndhat: At t-gmax: 9,408 bytes in 106 blocks\nmoka\ndhat: At t-gmax: 354,232 bytes in 1,581 blocks\nQuickCache\ndhat: At t-gmax: 11,840 bytes in 8 blocks\nTinyUFO\ndhat: At t-gmax: 37,337 bytes in 351 blocks\nTinyUFO compat\ndhat: At t-gmax: 19,000 bytes in 60 blocks\n\ntotal items 10000, cache size 10%\nlru\ndhat: At t-gmax: 128,512 bytes in 1,004 blocks\nmoka\ndhat: At t-gmax: 535,320 bytes in 7,278 blocks\nQuickCache\ndhat: At t-gmax: 93,000 bytes in 66 blocks\nTinyUFO\ndhat: At t-gmax: 236,053 bytes in 2,182 blocks\nTinyUFO Compact\ndhat: At t-gmax: 86,352 bytes in 1,128 blocks\n\ntotal items 100000, cache size 10%\nlru\ndhat: At t-gmax: 1,075,648 bytes in 10,004 blocks\nmoka\ndhat: At t-gmax: 2,489,088 bytes in 62,374 blocks\nQuickCache\ndhat: At t-gmax: 863,752 bytes in 66 blocks\nTinyUFO\ndhat: At t-gmax: 2,290,635 bytes in 20,467 blocks\nTinyUFO\ndhat: At t-gmax: 766,024 bytes in 10,421 blocks\n*/\n\nfn main() {\n    for items in [1000, 10_000, 100_000] {\n        println!(\"\\ntotal items {items}, cache size 10%\");\n        {\n            let _profiler = dhat::Profiler::new_heap();\n            bench_lru(1.05, items, 0.1);\n            println!(\"lru\");\n        }\n\n        {\n            let _profiler = dhat::Profiler::new_heap();\n            bench_moka(1.05, items, 0.1);\n            println!(\"\\nmoka\");\n        }\n\n        {\n            let _profiler = dhat::Profiler::new_heap();\n            bench_quick_cache(1.05, items, 0.1);\n            println!(\"\\nQuickCache\");\n        }\n\n        {\n            let _profiler = dhat::Profiler::new_heap();\n            bench_tinyufo(1.05, items, 0.1);\n            println!(\"\\nTinyUFO\");\n        }\n\n        {\n            let _profiler = dhat::Profiler::new_heap();\n            bench_tinyufo_compact(1.05, items, 0.1);\n            println!(\"\\nTinyUFO Compact\");\n        }\n    }\n}\n"
  },
  {
    "path": "tinyufo/benches/bench_perf.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\nuse rand::prelude::*;\nuse std::num::NonZeroUsize;\nuse std::sync::{Barrier, Mutex};\nuse std::thread;\nuse std::time::Instant;\n\nconst ITEMS: usize = 100;\n\nconst ITERATIONS: usize = 5_000_000;\nconst THREADS: usize = 8;\n\n/*\ncargo bench  --bench bench_perf\n\nNote: the performance number vary a lot on different planform, CPU and CPU arch\nBelow is from Linux + Ryzen 5 7600 CPU\n\nlru read total 150.423567ms, 30ns avg per operation, 33239472 ops per second\nmoka read total 462.133322ms, 92ns avg per operation, 10819389 ops per second\nquick_cache read total 125.618216ms, 25ns avg per operation, 39803144 ops per second\ntinyufo read total 199.007359ms, 39ns avg per operation, 25124698 ops per second\ntinyufo compact read total 331.145859ms, 66ns avg per operation, 15099087 ops per second\n\nlru read total 5.402631847s, 1.08µs avg per operation, 925474 ops per second\n...\ntotal 6960329 ops per second\n\nmoka read total 2.742258211s, 548ns avg per operation, 1823314 ops per second\n...\ntotal 14072430 ops per second\n\nquick_cache read total 1.186566627s, 237ns avg per operation, 4213838 ops per second\n...\ntotal 33694776 ops per second\n\ntinyufo read total 208.346855ms, 41ns avg per operation, 23998444 ops per second\n...\ntotal 148691408 ops per second\n\ntinyufo compact read total 539.403037ms, 107ns avg per operation, 9269507 ops per second\n...\ntotal 74130632 ops per second\n\nlru mixed read/write 5.500309876s, 1.1µs avg per operation, 909039 ops per second, 407431 misses\n...\ntotal 6846743 ops per second\n\nmoka mixed read/write 2.368500882s, 473ns avg per operation, 2111040 ops per second 279324 misses\n...\ntotal 16557962 ops per second\n\nquick_cache mixed read/write 838.072588ms, 167ns avg per operation, 5966070 ops per second 315051 misses\n...\ntotal 47698472 ops per second\n\ntinyufo mixed read/write 456.134531ms, 91ns avg per operation, 10961678 ops per second, 294977 misses\n...\ntotal 80865792 ops per second\n\ntinyufo compact mixed read/write 638.770053ms, 127ns avg per operation, 7827543 ops per second, 294641 misses\n...\ntotal 62600844 ops per second\n*/\n\nfn main() {\n    println!(\"Note: these performance numbers vary a lot across different CPUs and OSes.\");\n    // we don't bench eviction here so make the caches large enough to hold all\n    let lru = Mutex::new(lru::LruCache::<u64, ()>::unbounded());\n    let moka = moka::sync::Cache::new(ITEMS as u64 + 10);\n    let quick_cache = quick_cache::sync::Cache::new(ITEMS + 10);\n    let tinyufo = tinyufo::TinyUfo::new(ITEMS + 10, 10);\n    let tinyufo_compact = tinyufo::TinyUfo::new_compact(ITEMS + 10, 10);\n\n    // populate first, then we bench access/promotion\n    for i in 0..ITEMS {\n        lru.lock().unwrap().put(i as u64, ());\n        moka.insert(i as u64, ());\n        quick_cache.insert(i as u64, ());\n        tinyufo.put(i as u64, (), 1);\n        tinyufo_compact.put(i as u64, (), 1);\n    }\n\n    // single thread\n    let mut rng = rand::rng();\n    let zipf = rand_distr::Zipf::new(ITEMS as f64, 1.03).unwrap();\n\n    let before = Instant::now();\n    for _ in 0..ITERATIONS {\n        lru.lock().unwrap().get(&(zipf.sample(&mut rng) as u64));\n    }\n    let elapsed = before.elapsed();\n    println!(\n        \"lru read total {elapsed:?}, {:?} avg per operation, {} ops per second\",\n        elapsed / ITERATIONS as u32,\n        (ITERATIONS as f32 / elapsed.as_secs_f32()) as u32\n    );\n\n    let before = Instant::now();\n    for _ in 0..ITERATIONS {\n        moka.get(&(zipf.sample(&mut rng) as u64));\n    }\n    let elapsed = before.elapsed();\n    println!(\n        \"moka read total {elapsed:?}, {:?} avg per operation, {} ops per second\",\n        elapsed / ITERATIONS as u32,\n        (ITERATIONS as f32 / elapsed.as_secs_f32()) as u32\n    );\n\n    let before = Instant::now();\n    for _ in 0..ITERATIONS {\n        quick_cache.get(&(zipf.sample(&mut rng) as u64));\n    }\n    let elapsed = before.elapsed();\n    println!(\n        \"quick_cache read total {elapsed:?}, {:?} avg per operation, {} ops per second\",\n        elapsed / ITERATIONS as u32,\n        (ITERATIONS as f32 / elapsed.as_secs_f32()) as u32\n    );\n\n    let before = Instant::now();\n    for _ in 0..ITERATIONS {\n        tinyufo.get(&(zipf.sample(&mut rng) as u64));\n    }\n    let elapsed = before.elapsed();\n    println!(\n        \"tinyufo read total {elapsed:?}, {:?} avg per operation, {} ops per second\",\n        elapsed / ITERATIONS as u32,\n        (ITERATIONS as f32 / elapsed.as_secs_f32()) as u32\n    );\n\n    let before = Instant::now();\n    for _ in 0..ITERATIONS {\n        tinyufo_compact.get(&(zipf.sample(&mut rng) as u64));\n    }\n    let elapsed = before.elapsed();\n    println!(\n        \"tinyufo compact read total {elapsed:?}, {:?} avg per operation, {} ops per second\",\n        elapsed / ITERATIONS as u32,\n        (ITERATIONS as f32 / elapsed.as_secs_f32()) as u32\n    );\n\n    // concurrent\n    let wg = Barrier::new(THREADS);\n    let before = Instant::now();\n    thread::scope(|s| {\n        for _ in 0..THREADS {\n            s.spawn(|| {\n                let mut rng = rand::rng();\n                let zipf = rand_distr::Zipf::new(ITEMS as f64, 1.03).unwrap();\n                wg.wait();\n                let before = Instant::now();\n                for _ in 0..ITERATIONS {\n                    lru.lock().unwrap().get(&(zipf.sample(&mut rng) as u64));\n                }\n                let elapsed = before.elapsed();\n                println!(\n                    \"lru read total {elapsed:?}, {:?} avg per operation, {} ops per second\",\n                    elapsed / ITERATIONS as u32,\n                    (ITERATIONS as f32 / elapsed.as_secs_f32()) as u32\n                );\n            });\n        }\n    });\n    let elapsed = before.elapsed();\n    println!(\n        \"total {} ops per second\",\n        (ITERATIONS as f32 * THREADS as f32 / elapsed.as_secs_f32()) as u32\n    );\n\n    let wg = Barrier::new(THREADS);\n    let before = Instant::now();\n    thread::scope(|s| {\n        for _ in 0..THREADS {\n            s.spawn(|| {\n                let mut rng = rand::rng();\n                let zipf = rand_distr::Zipf::new(ITEMS as f64, 1.03).unwrap();\n                wg.wait();\n                let before = Instant::now();\n                for _ in 0..ITERATIONS {\n                    moka.get(&(zipf.sample(&mut rng) as u64));\n                }\n                let elapsed = before.elapsed();\n                println!(\n                    \"moka read total {elapsed:?}, {:?} avg per operation, {} ops per second\",\n                    elapsed / ITERATIONS as u32,\n                    (ITERATIONS as f32 / elapsed.as_secs_f32()) as u32\n                );\n            });\n        }\n    });\n    let elapsed = before.elapsed();\n    println!(\n        \"total {} ops per second\",\n        (ITERATIONS as f32 * THREADS as f32 / elapsed.as_secs_f32()) as u32\n    );\n\n    let wg = Barrier::new(THREADS);\n    let before = Instant::now();\n    thread::scope(|s| {\n        for _ in 0..THREADS {\n            s.spawn(|| {\n                let mut rng = rand::rng();\n                let zipf = rand_distr::Zipf::new(ITEMS as f64, 1.03).unwrap();\n                wg.wait();\n                let before = Instant::now();\n                for _ in 0..ITERATIONS {\n                    quick_cache.get(&(zipf.sample(&mut rng) as u64));\n                }\n                let elapsed = before.elapsed();\n                println!(\n                    \"quick_cache read total {elapsed:?}, {:?} avg per operation, {} ops per second\",\n                    elapsed / ITERATIONS as u32,\n                    (ITERATIONS as f32 / elapsed.as_secs_f32()) as u32\n                );\n            });\n        }\n    });\n    let elapsed = before.elapsed();\n    println!(\n        \"total {} ops per second\",\n        (ITERATIONS as f32 * THREADS as f32 / elapsed.as_secs_f32()) as u32\n    );\n\n    let wg = Barrier::new(THREADS);\n    let before = Instant::now();\n    thread::scope(|s| {\n        for _ in 0..THREADS {\n            s.spawn(|| {\n                let mut rng = rand::rng();\n                let zipf = rand_distr::Zipf::new(ITEMS as f64, 1.03).unwrap();\n                wg.wait();\n                let before = Instant::now();\n                for _ in 0..ITERATIONS {\n                    tinyufo.get(&(zipf.sample(&mut rng) as u64));\n                }\n                let elapsed = before.elapsed();\n                println!(\n                    \"tinyufo read total {elapsed:?}, {:?} avg per operation, {} ops per second\",\n                    elapsed / ITERATIONS as u32,\n                    (ITERATIONS as f32 / elapsed.as_secs_f32()) as u32\n                );\n            });\n        }\n    });\n    let elapsed = before.elapsed();\n    println!(\n        \"total {} ops per second\",\n        (ITERATIONS as f32 * THREADS as f32 / elapsed.as_secs_f32()) as u32\n    );\n\n    let wg = Barrier::new(THREADS);\n    let before = Instant::now();\n    thread::scope(|s| {\n        for _ in 0..THREADS {\n            s.spawn(|| {\n                let mut rng = rand::rng();\n                let zipf = rand_distr::Zipf::new(ITEMS as f64, 1.03).unwrap();\n                wg.wait();\n                let before = Instant::now();\n                for _ in 0..ITERATIONS {\n                    tinyufo_compact.get(&(zipf.sample(&mut rng) as u64));\n                }\n                let elapsed = before.elapsed();\n                println!(\n                    \"tinyufo compact read total {elapsed:?}, {:?} avg per operation, {} ops per second\",\n                    elapsed / ITERATIONS as u32,\n                    (ITERATIONS as f32 / elapsed.as_secs_f32()) as u32\n                );\n            });\n        }\n    });\n    let elapsed = before.elapsed();\n    println!(\n        \"total {} ops per second\",\n        (ITERATIONS as f32 * THREADS as f32 / elapsed.as_secs_f32()) as u32\n    );\n\n    ///// bench mixed read and write /////\n    const CACHE_SIZE: usize = 1000;\n    let items: usize = 10000;\n    const ZIPF_EXP: f64 = 1.3;\n\n    let lru = Mutex::new(lru::LruCache::<u64, ()>::new(\n        NonZeroUsize::new(CACHE_SIZE).unwrap(),\n    ));\n    let wg = Barrier::new(THREADS);\n    let before = Instant::now();\n    thread::scope(|s| {\n        for _ in 0..THREADS {\n            s.spawn(|| {\n                let mut miss_count = 0;\n                let mut rng = rand::rng();\n                let zipf = rand_distr::Zipf::new(items as f64, ZIPF_EXP).unwrap();\n                wg.wait();\n                let before = Instant::now();\n                for _ in 0..ITERATIONS {\n                    let key = zipf.sample(&mut rng) as u64;\n                    let mut lru = lru.lock().unwrap();\n                    if lru.get(&key).is_none() {\n                        lru.put(key, ());\n                        miss_count += 1;\n                    }\n                }\n                let elapsed = before.elapsed();\n                println!(\n                    \"lru mixed read/write {elapsed:?}, {:?} avg per operation, {} ops per second, {miss_count} misses\",\n                    elapsed / ITERATIONS as u32,\n                    (ITERATIONS as f32 / elapsed.as_secs_f32()) as u32\n                );\n            });\n        }\n    });\n    let elapsed = before.elapsed();\n    println!(\n        \"total {} ops per second\",\n        (ITERATIONS as f32 * THREADS as f32 / elapsed.as_secs_f32()) as u32\n    );\n\n    let moka = moka::sync::Cache::new(CACHE_SIZE as u64);\n    let wg = Barrier::new(THREADS);\n    let before = Instant::now();\n    thread::scope(|s| {\n        for _ in 0..THREADS {\n            s.spawn(|| {\n                let mut miss_count = 0;\n                let mut rng = rand::rng();\n                let zipf = rand_distr::Zipf::new(items as f64, ZIPF_EXP).unwrap();\n                wg.wait();\n                let before = Instant::now();\n                for _ in 0..ITERATIONS {\n                    let key = zipf.sample(&mut rng) as u64;\n                    if moka.get(&key).is_none() {\n                        moka.insert(key, ());\n                        miss_count += 1;\n                    }\n                }\n                let elapsed = before.elapsed();\n                println!(\n                    \"moka mixed read/write {elapsed:?}, {:?} avg per operation, {} ops per second {miss_count} misses\",\n                    elapsed / ITERATIONS as u32,\n                    (ITERATIONS as f32 / elapsed.as_secs_f32()) as u32\n                );\n            });\n        }\n    });\n    let elapsed = before.elapsed();\n    println!(\n        \"total {} ops per second\",\n        (ITERATIONS as f32 * THREADS as f32 / elapsed.as_secs_f32()) as u32\n    );\n\n    let quick_cache = quick_cache::sync::Cache::new(CACHE_SIZE);\n    let wg = Barrier::new(THREADS);\n    let before = Instant::now();\n    thread::scope(|s| {\n        for _ in 0..THREADS {\n            s.spawn(|| {\n                let mut miss_count = 0;\n                let mut rng = rand::rng();\n                let zipf = rand_distr::Zipf::new(items as f64, ZIPF_EXP).unwrap();\n                wg.wait();\n                let before = Instant::now();\n                for _ in 0..ITERATIONS {\n                    let key = zipf.sample(&mut rng) as u64;\n                    if quick_cache.get(&key).is_none() {\n                        quick_cache.insert(key, ());\n                        miss_count += 1;\n                    }\n                }\n                let elapsed = before.elapsed();\n                println!(\n                    \"quick_cache mixed read/write {elapsed:?}, {:?} avg per operation, {} ops per second {miss_count} misses\",\n                    elapsed / ITERATIONS as u32,\n                    (ITERATIONS as f32 / elapsed.as_secs_f32()) as u32\n                );\n            });\n        }\n    });\n    let elapsed = before.elapsed();\n    println!(\n        \"total {} ops per second\",\n        (ITERATIONS as f32 * THREADS as f32 / elapsed.as_secs_f32()) as u32\n    );\n\n    let tinyufo = tinyufo::TinyUfo::new(CACHE_SIZE, CACHE_SIZE);\n    let wg = Barrier::new(THREADS);\n    let before = Instant::now();\n    thread::scope(|s| {\n        for _ in 0..THREADS {\n            s.spawn(|| {\n                let mut miss_count = 0;\n                let mut rng = rand::rng();\n                let zipf = rand_distr::Zipf::new(items as f64, ZIPF_EXP).unwrap();\n                wg.wait();\n                let before = Instant::now();\n                for _ in 0..ITERATIONS {\n                    let key = zipf.sample(&mut rng) as u64;\n                    if tinyufo.get(&key).is_none() {\n                        tinyufo.put(key, (), 1);\n                        miss_count +=1;\n                    }\n                }\n                let elapsed = before.elapsed();\n                println!(\n                    \"tinyufo mixed read/write {elapsed:?}, {:?} avg per operation, {} ops per second, {miss_count} misses\",\n                    elapsed / ITERATIONS as u32,\n                    (ITERATIONS as f32 / elapsed.as_secs_f32()) as u32,\n                );\n            });\n        }\n    });\n\n    let elapsed = before.elapsed();\n    println!(\n        \"total {} ops per second\",\n        (ITERATIONS as f32 * THREADS as f32 / elapsed.as_secs_f32()) as u32\n    );\n\n    let tinyufo_compact = tinyufo::TinyUfo::new(CACHE_SIZE, CACHE_SIZE);\n    let wg = Barrier::new(THREADS);\n    let before = Instant::now();\n    thread::scope(|s| {\n        for _ in 0..THREADS {\n            s.spawn(|| {\n                let mut miss_count = 0;\n                let mut rng = rand::rng();\n                let zipf = rand_distr::Zipf::new(items as f64, ZIPF_EXP).unwrap();\n                wg.wait();\n                let before = Instant::now();\n                for _ in 0..ITERATIONS {\n                    let key = zipf.sample(&mut rng) as u64;\n                    if tinyufo_compact.get(&key).is_none() {\n                        tinyufo_compact.put(key, (), 1);\n                        miss_count +=1;\n                    }\n                }\n                let elapsed = before.elapsed();\n                println!(\n                    \"tinyufo compact mixed read/write {elapsed:?}, {:?} avg per operation, {} ops per second, {miss_count} misses\",\n                    elapsed / ITERATIONS as u32,\n                    (ITERATIONS as f32 / elapsed.as_secs_f32()) as u32,\n                );\n            });\n        }\n    });\n\n    let elapsed = before.elapsed();\n    println!(\n        \"total {} ops per second\",\n        (ITERATIONS as f32 * THREADS as f32 / elapsed.as_secs_f32()) as u32\n    );\n}\n"
  },
  {
    "path": "tinyufo/src/buckets.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! Concurrent storage backend\n\nuse super::{Bucket, Key};\nuse ahash::RandomState;\nuse crossbeam_skiplist::{map::Entry, SkipMap};\nuse flurry::HashMap;\n\n/// N-shard skip list. Memory efficient, constant time lookup on average, but a bit slower\n/// than hash map\npub struct Compact<T>(Box<[SkipMap<Key, Bucket<T>>]>);\n\nimpl<T: Send + 'static> Compact<T> {\n    /// Create a new [Compact]\n    pub fn new(total_items: usize, items_per_shard: usize) -> Self {\n        assert!(items_per_shard > 0);\n\n        let shards = std::cmp::max(total_items / items_per_shard, 1);\n        let mut shard_array = vec![];\n        for _ in 0..shards {\n            shard_array.push(SkipMap::new());\n        }\n        Self(shard_array.into_boxed_slice())\n    }\n\n    pub fn get(&self, key: &Key) -> Option<Entry<'_, Key, Bucket<T>>> {\n        let shard = *key as usize % self.0.len();\n        self.0[shard].get(key)\n    }\n\n    pub fn get_map<V, F: FnOnce(Entry<Key, Bucket<T>>) -> V>(&self, key: &Key, f: F) -> Option<V> {\n        let v = self.get(key);\n        v.map(f)\n    }\n\n    fn insert(&self, key: Key, value: Bucket<T>) -> Option<()> {\n        let shard = key as usize % self.0.len();\n        let removed = self.0[shard].remove(&key);\n        self.0[shard].insert(key, value);\n        removed.map(|_| ())\n    }\n\n    fn remove(&self, key: &Key) {\n        let shard = *key as usize % self.0.len();\n        (&self.0)[shard].remove(key);\n    }\n}\n\n// Concurrent hash map, fast but use more memory\npub struct Fast<T>(HashMap<Key, Bucket<T>, RandomState>);\n\nimpl<T: Send + Sync> Fast<T> {\n    pub fn new(total_items: usize) -> Self {\n        Self(HashMap::with_capacity_and_hasher(\n            total_items,\n            RandomState::new(),\n        ))\n    }\n\n    pub fn get_map<V, F: FnOnce(&Bucket<T>) -> V>(&self, key: &Key, f: F) -> Option<V> {\n        let pinned = self.0.pin();\n        let v = pinned.get(key);\n        v.map(f)\n    }\n\n    fn insert(&self, key: Key, value: Bucket<T>) -> Option<()> {\n        let pinned = self.0.pin();\n        pinned.insert(key, value).map(|_| ())\n    }\n\n    fn remove(&self, key: &Key) {\n        let pinned = self.0.pin();\n        pinned.remove(key);\n    }\n}\n\npub enum Buckets<T> {\n    Fast(Box<Fast<T>>),\n    Compact(Compact<T>),\n}\n\nimpl<T: Send + Sync + 'static> Buckets<T> {\n    pub fn new_fast(items: usize) -> Self {\n        Self::Fast(Box::new(Fast::new(items)))\n    }\n\n    pub fn new_compact(items: usize, items_per_shard: usize) -> Self {\n        Self::Compact(Compact::new(items, items_per_shard))\n    }\n\n    pub fn insert(&self, key: Key, value: Bucket<T>) -> Option<()> {\n        match self {\n            Self::Compact(c) => c.insert(key, value),\n            Self::Fast(f) => f.insert(key, value),\n        }\n    }\n\n    pub fn remove(&self, key: &Key) {\n        match self {\n            Self::Compact(c) => c.remove(key),\n            Self::Fast(f) => f.remove(key),\n        }\n    }\n\n    pub fn get_map<V, F: FnOnce(&Bucket<T>) -> V>(&self, key: &Key, f: F) -> Option<V> {\n        match self {\n            Self::Compact(c) => c.get_map(key, |v| f(v.value())),\n            Self::Fast(c) => c.get_map(key, f),\n        }\n    }\n\n    #[cfg(test)]\n    pub fn get_queue(&self, key: &Key) -> Option<bool> {\n        self.get_map(key, |v| v.queue.is_main())\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use super::*;\n\n    #[test]\n    fn test_fast() {\n        let fast = Buckets::new_fast(10);\n\n        assert!(fast.get_map(&1, |_| ()).is_none());\n\n        let bucket = Bucket {\n            queue: crate::Location::new_small(),\n            weight: 1,\n            uses: Default::default(),\n            data: 1,\n        };\n        fast.insert(1, bucket);\n\n        assert_eq!(fast.get_map(&1, |v| v.data), Some(1));\n\n        fast.remove(&1);\n        assert!(fast.get_map(&1, |_| ()).is_none());\n    }\n\n    #[test]\n    fn test_compact() {\n        let compact = Buckets::new_compact(10, 2);\n\n        assert!(compact.get_map(&1, |_| ()).is_none());\n\n        let bucket = Bucket {\n            queue: crate::Location::new_small(),\n            weight: 1,\n            uses: Default::default(),\n            data: 1,\n        };\n        compact.insert(1, bucket);\n\n        assert_eq!(compact.get_map(&1, |v| v.data), Some(1));\n\n        compact.remove(&1);\n        assert!(compact.get_map(&1, |_| ()).is_none());\n    }\n}\n"
  },
  {
    "path": "tinyufo/src/estimation.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\nuse ahash::RandomState;\nuse std::hash::Hash;\nuse std::sync::atomic::{AtomicU8, AtomicUsize, Ordering};\n\nstruct Estimator {\n    estimator: Box<[(Box<[AtomicU8]>, RandomState)]>,\n}\n\nimpl Estimator {\n    fn optimal_paras(items: usize) -> (usize, usize) {\n        use std::cmp::max;\n        // derived from https://en.wikipedia.org/wiki/Count%E2%80%93min_sketch\n        // width = ceil(e / ε)\n        // depth = ceil(ln(1 − δ) / ln(1 / 2))\n        let error_range = 1.0 / (items as f64);\n        let failure_probability = 1.0 / (items as f64);\n        (\n            max((std::f64::consts::E / error_range).ceil() as usize, 16),\n            max((failure_probability.ln() / 0.5f64.ln()).ceil() as usize, 2),\n        )\n    }\n\n    fn optimal(items: usize) -> Self {\n        let (slots, hashes) = Self::optimal_paras(items);\n        Self::new(hashes, slots, RandomState::new)\n    }\n\n    fn compact(items: usize) -> Self {\n        let (slots, hashes) = Self::optimal_paras(items / 100);\n        Self::new(hashes, slots, RandomState::new)\n    }\n\n    #[cfg(test)]\n    fn seeded(items: usize) -> Self {\n        let (slots, hashes) = Self::optimal_paras(items);\n        Self::new(hashes, slots, || RandomState::with_seeds(2, 3, 4, 5))\n    }\n\n    #[cfg(test)]\n    fn seeded_compact(items: usize) -> Self {\n        let (slots, hashes) = Self::optimal_paras(items / 100);\n        Self::new(hashes, slots, || RandomState::with_seeds(2, 3, 4, 5))\n    }\n\n    /// Create a new `Estimator` with the given amount of hashes and columns (slots) using\n    /// the given random source.\n    pub fn new(hashes: usize, slots: usize, random: impl Fn() -> RandomState) -> Self {\n        let mut estimator = Vec::with_capacity(hashes);\n        for _ in 0..hashes {\n            let mut slot = Vec::with_capacity(slots);\n            for _ in 0..slots {\n                slot.push(AtomicU8::new(0));\n            }\n            estimator.push((slot.into_boxed_slice(), random()));\n        }\n\n        Estimator {\n            estimator: estimator.into_boxed_slice(),\n        }\n    }\n\n    pub fn incr<T: Hash>(&self, key: T) -> u8 {\n        let mut min = u8::MAX;\n        for (slot, hasher) in self.estimator.iter() {\n            let hash = hasher.hash_one(&key) as usize;\n            let counter = &slot[hash % slot.len()];\n            let (_current, new) = incr_no_overflow(counter);\n            min = std::cmp::min(min, new);\n        }\n        min\n    }\n\n    /// Get the estimated frequency of `key`.\n    pub fn get<T: Hash>(&self, key: T) -> u8 {\n        let mut min = u8::MAX;\n        for (slot, hasher) in self.estimator.iter() {\n            let hash = hasher.hash_one(&key) as usize;\n            let counter = &slot[hash % slot.len()];\n            let current = counter.load(Ordering::Relaxed);\n            min = std::cmp::min(min, current);\n        }\n        min\n    }\n\n    /// right shift all values inside this `Estimator`.\n    pub fn age(&self, shift: u8) {\n        for (slot, _) in self.estimator.iter() {\n            for counter in slot.iter() {\n                // we don't CAS because the only update between the load and store\n                // is fetch_add(1), which should be fine to miss/ignore\n                let c = counter.load(Ordering::Relaxed);\n                counter.store(c >> shift, Ordering::Relaxed);\n            }\n        }\n    }\n}\n\nfn incr_no_overflow(var: &AtomicU8) -> (u8, u8) {\n    loop {\n        let current = var.load(Ordering::Relaxed);\n        if current == u8::MAX {\n            return (current, current);\n        }\n        let new = if current == u8::MAX - 1 {\n            u8::MAX\n        } else {\n            current + 1\n        };\n        if let Err(new) = var.compare_exchange(current, new, Ordering::Acquire, Ordering::Relaxed) {\n            // someone else beat us to it\n            if new == u8::MAX {\n                // already max\n                return (current, new);\n            } // else, try again\n        } else {\n            return (current, new);\n        }\n    }\n}\n\n// bare-minimum TinyLfu with CM-Sketch, no doorkeeper for now\npub(crate) struct TinyLfu {\n    estimator: Estimator,\n    window_counter: AtomicUsize,\n    window_limit: usize,\n}\n\nimpl TinyLfu {\n    pub fn get<T: Hash>(&self, key: T) -> u8 {\n        self.estimator.get(key)\n    }\n\n    pub fn incr<T: Hash>(&self, key: T) -> u8 {\n        let window_size = self.window_counter.fetch_add(1, Ordering::Relaxed);\n        // When window_size concurrently increases, only one resets the window and age the estimator.\n        // > self.window_limit * 2 is a safety net in case for whatever reason window_size grows\n        // out of control\n        if window_size == self.window_limit || window_size > self.window_limit * 2 {\n            self.window_counter.store(0, Ordering::Relaxed);\n            self.estimator.age(1); // right shift 1 bit\n        }\n        self.estimator.incr(key)\n    }\n\n    // because we use 8-bits counters, window size can be 256 * the cache size\n    pub fn new(cache_size: usize) -> Self {\n        Self {\n            estimator: Estimator::optimal(cache_size),\n            window_counter: Default::default(),\n            // 8x: just a heuristic to balance the memory usage and accuracy\n            window_limit: cache_size * 8,\n        }\n    }\n\n    pub fn new_compact(cache_size: usize) -> Self {\n        Self {\n            estimator: Estimator::compact(cache_size),\n            window_counter: Default::default(),\n            // 8x: just a heuristic to balance the memory usage and accuracy\n            window_limit: cache_size * 8,\n        }\n    }\n\n    #[cfg(test)]\n    pub fn new_seeded(cache_size: usize) -> Self {\n        Self {\n            estimator: Estimator::seeded(cache_size),\n            window_counter: Default::default(),\n            // 8x: just a heuristic to balance the memory usage and accuracy\n            window_limit: cache_size * 8,\n        }\n    }\n\n    #[cfg(test)]\n    pub fn new_compact_seeded(cache_size: usize) -> Self {\n        Self {\n            estimator: Estimator::seeded_compact(cache_size),\n            window_counter: Default::default(),\n            // 8x: just a heuristic to balance the memory usage and accuracy\n            window_limit: cache_size * 8,\n        }\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use super::*;\n\n    #[test]\n    fn test_cmk_paras() {\n        let (slots, hashes) = Estimator::optimal_paras(1_000_000);\n        // just smoke check some standard input\n        assert_eq!(slots, 2718282);\n        assert_eq!(hashes, 20);\n    }\n\n    #[test]\n    fn test_tiny_lfu() {\n        let tiny = TinyLfu::new(1);\n        assert_eq!(tiny.get(1), 0);\n        assert_eq!(tiny.incr(1), 1);\n        assert_eq!(tiny.incr(1), 2);\n        assert_eq!(tiny.get(1), 2);\n\n        // Might have hash collisions for the others, need to\n        // get() before can assert on the incr() value.\n        let two = tiny.get(2);\n        assert_eq!(tiny.incr(2), two + 1);\n        assert_eq!(tiny.incr(2), two + 2);\n        assert_eq!(tiny.get(2), two + 2);\n\n        let three = tiny.get(3);\n        assert_eq!(tiny.incr(3), three + 1);\n        assert_eq!(tiny.incr(3), three + 2);\n        assert_eq!(tiny.incr(3), three + 3);\n        assert_eq!(tiny.incr(3), three + 4);\n\n        // 8 incr(), now resets on next incr\n        // can only assert they are greater than or equal\n        // to the incr() we do per key.\n\n        assert!(tiny.incr(3) >= 3); // had 4, reset to 2, added another.\n        assert!(tiny.incr(1) >= 2); // had 2, reset to 1, added another.\n        assert!(tiny.incr(2) >= 2); // had 2, reset to 1, added another.\n    }\n}\n"
  },
  {
    "path": "tinyufo/src/lib.rs",
    "content": "// Copyright 2026 Cloudflare, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n//! A In-memory cache implementation with TinyLFU as the admission policy and [S3-FIFO](https://s3fifo.com/) as the eviction policy.\n//!\n//! TinyUFO improves cache hit ratio noticeably compared to LRU.\n//!\n//! TinyUFO is lock-free. It is very fast in the systems with a lot concurrent reads and/or writes\n\nuse ahash::RandomState;\nuse crossbeam_queue::SegQueue;\nuse std::marker::PhantomData;\nuse std::sync::atomic::AtomicUsize;\nuse std::sync::atomic::{\n    AtomicBool, AtomicU8,\n    Ordering::{Acquire, Relaxed, SeqCst},\n};\nmod buckets;\nmod estimation;\n\nuse buckets::Buckets;\nuse estimation::TinyLfu;\nuse std::hash::Hash;\n\nconst SMALL: bool = false;\nconst MAIN: bool = true;\n\n// Indicate which queue an item is located\n#[derive(Debug, Default)]\nstruct Location(AtomicBool);\n\nimpl Location {\n    fn new_small() -> Self {\n        Self(AtomicBool::new(SMALL))\n    }\n\n    fn value(&self) -> bool {\n        self.0.load(Relaxed)\n    }\n\n    fn is_main(&self) -> bool {\n        self.value()\n    }\n\n    fn move_to_main(&self) {\n        self.0.store(true, Relaxed);\n    }\n}\n\n// We have 8 bits to spare but we still cap at 3. This is to make sure that the main queue\n// in the worst case can find something to evict quickly\nconst USES_CAP: u8 = 3;\n\n#[derive(Debug, Default)]\nstruct Uses(AtomicU8);\n\nimpl Uses {\n    pub fn inc_uses(&self) -> u8 {\n        loop {\n            let uses = self.uses();\n            if uses >= USES_CAP {\n                return uses;\n            }\n            if let Err(new) = self.0.compare_exchange(uses, uses + 1, Acquire, Relaxed) {\n                // someone else beat us to it\n                if new >= USES_CAP {\n                    // already above cap\n                    return new;\n                } // else, try again\n            } else {\n                return uses + 1;\n            }\n        }\n    }\n\n    // decrease uses, return the previous value\n    pub fn decr_uses(&self) -> u8 {\n        loop {\n            let uses = self.uses();\n            if uses == 0 {\n                return 0;\n            }\n            if let Err(new) = self.0.compare_exchange(uses, uses - 1, Acquire, Relaxed) {\n                // someone else beat us to it\n                if new == 0 {\n                    return 0;\n                } // else, try again\n            } else {\n                return uses;\n            }\n        }\n    }\n\n    pub fn uses(&self) -> u8 {\n        self.0.load(Relaxed)\n    }\n}\n\ntype Key = u64;\ntype Weight = u16;\n\n/// The key-value pair returned from cache eviction\n#[derive(Clone)]\npub struct KV<T> {\n    /// NOTE: that we currently don't store the Actual key in the cache. This returned value\n    /// is just the hash of it.\n    pub key: Key,\n    pub data: T,\n    pub weight: Weight,\n}\n\n// the data and its metadata\npub struct Bucket<T> {\n    uses: Uses,\n    queue: Location,\n    weight: Weight,\n    data: T,\n}\n\nconst SMALL_QUEUE_PERCENTAGE: f32 = 0.1;\n\nstruct FiFoQueues<T> {\n    total_weight_limit: usize,\n\n    small: SegQueue<Key>,\n    small_weight: AtomicUsize,\n\n    main: SegQueue<Key>,\n    main_weight: AtomicUsize,\n\n    // this replaces the ghost queue of S3-FIFO with similar goal: track the evicted assets\n    estimator: TinyLfu,\n\n    _t: PhantomData<T>,\n}\n\nimpl<T: Clone + Send + Sync + 'static> FiFoQueues<T> {\n    fn admit(\n        &self,\n        key: Key,\n        data: T,\n        weight: u16,\n        ignore_lfu: bool,\n        buckets: &Buckets<T>,\n    ) -> Vec<KV<T>> {\n        // Note that we only use TinyLFU during cache admission but not cache read.\n        // So effectively we mostly sketch the popularity of less popular assets.\n        // In this way the sketch is a bit more accurate on these assets.\n        // Also we don't need another separated window cache to address the sparse burst issue as\n        // this sketch doesn't favor very popular assets much.\n        let new_freq = self.estimator.incr(key);\n\n        assert!(weight > 0);\n        let new_bucket = {\n            let Some((uses, queue, weight)) = buckets.get_map(&key, |bucket| {\n                // the item exists, in case weight changes\n                let old_weight = bucket.weight;\n                let uses = bucket.uses.inc_uses();\n\n                fn update_atomic(weight: &AtomicUsize, old: u16, new: u16) {\n                    if old == new {\n                        return;\n                    }\n                    if old > new {\n                        weight.fetch_sub((old - new) as usize, SeqCst);\n                    } else {\n                        weight.fetch_add((new - old) as usize, SeqCst);\n                    }\n                }\n                let queue = bucket.queue.is_main();\n                if queue == MAIN {\n                    update_atomic(&self.main_weight, old_weight, weight);\n                } else {\n                    update_atomic(&self.small_weight, old_weight, weight);\n                }\n                (uses, queue, weight)\n            }) else {\n                let mut evicted = self.evict_to_limit(weight, buckets);\n                // TODO: figure out the right way to compare frequencies of different weights across\n                // many evicted assets. For now TinyLFU is only used when only evicting 1 item.\n                let (key, data, weight) = if !ignore_lfu && evicted.len() == 1 {\n                    // Apply the admission algorithm of TinyLFU: compare the incoming new item\n                    // and the evicted one. The more popular one is admitted to cache\n                    let evicted_first = &evicted[0];\n                    let evicted_freq = self.estimator.get(evicted_first.key);\n                    if evicted_freq > new_freq {\n                        // put it back\n                        let first = evicted.pop().expect(\"just check non-empty\");\n                        // return the put value\n                        evicted.push(KV { key, data, weight });\n                        (first.key, first.data, first.weight)\n                    } else {\n                        (key, data, weight)\n                    }\n                } else {\n                    (key, data, weight)\n                };\n\n                let bucket = Bucket {\n                    queue: Location::new_small(),\n                    weight,\n                    uses: Default::default(), // 0\n                    data,\n                };\n                let old = buckets.insert(key, bucket);\n                if old.is_none() {\n                    // Always push key first before updating weight\n                    // If doing the other order, another concurrent thread might not\n                    // find things to evict\n                    self.small.push(key);\n                    self.small_weight.fetch_add(weight as usize, SeqCst);\n                } // else: two threads are racing adding the item\n                  // TODO: compare old.weight and update accordingly\n                return evicted;\n            };\n            Bucket {\n                queue: Location(queue.into()),\n                weight,\n                uses: Uses(uses.into()),\n                data,\n            }\n        };\n\n        // replace the existing one\n        buckets.insert(key, new_bucket);\n\n        // NOTE: there is a chance that the item itself is evicted if it happens to be the one selected\n        // by the algorithm. We could avoid this by checking if the item is in the returned evicted items,\n        // and then add it back. But to keep the code simple we just allow it to happen.\n        self.evict_to_limit(0, buckets)\n    }\n\n    // the `extra_weight` is to essentially tell the cache to reserve that amount of weight for\n    // admission. It is used when calling `evict_to_limit` before admitting the asset itself.\n    fn evict_to_limit(&self, extra_weight: Weight, buckets: &Buckets<T>) -> Vec<KV<T>> {\n        let mut evicted = if self.total_weight_limit\n            < self.small_weight.load(SeqCst) + self.main_weight.load(SeqCst) + extra_weight as usize\n        {\n            Vec::with_capacity(1)\n        } else {\n            vec![]\n        };\n        while self.total_weight_limit\n            < self.small_weight.load(SeqCst) + self.main_weight.load(SeqCst) + extra_weight as usize\n        {\n            if let Some(evicted_item) = self.evict_one(buckets) {\n                evicted.push(evicted_item);\n            } else {\n                break;\n            }\n        }\n\n        evicted\n    }\n\n    fn evict_one(&self, buckets: &Buckets<T>) -> Option<KV<T>> {\n        let evict_small = self.small_weight_limit() <= self.small_weight.load(SeqCst);\n\n        if evict_small {\n            let evicted = self.evict_one_from_small(buckets);\n            // evict_one_from_small could just promote everything to main without evicting any\n            // so need to evict_one_from_main if nothing evicted\n            if evicted.is_some() {\n                return evicted;\n            }\n        }\n        self.evict_one_from_main(buckets)\n    }\n\n    fn small_weight_limit(&self) -> usize {\n        (self.total_weight_limit as f32 * SMALL_QUEUE_PERCENTAGE).floor() as usize + 1\n    }\n\n    fn evict_one_from_small(&self, buckets: &Buckets<T>) -> Option<KV<T>> {\n        loop {\n            let Some(to_evict) = self.small.pop() else {\n                // empty queue, this is caught between another pop() and fetch_sub()\n                return None;\n            };\n\n            let v = buckets\n                .get_map(&to_evict, |bucket| {\n                    let weight = bucket.weight;\n                    self.small_weight.fetch_sub(weight as usize, SeqCst);\n\n                    if bucket.uses.uses() > 1 {\n                        // move to main\n                        bucket.queue.move_to_main();\n                        self.main.push(to_evict);\n                        self.main_weight.fetch_add(weight as usize, SeqCst);\n                        // continue until find one to evict\n                        None\n                    } else {\n                        let data = bucket.data.clone();\n                        let weight = bucket.weight;\n                        buckets.remove(&to_evict);\n                        Some(KV {\n                            key: to_evict,\n                            data,\n                            weight,\n                        })\n                    }\n                })\n                .flatten();\n            if v.is_some() {\n                // found the one to evict, break\n                return v;\n            }\n        }\n    }\n\n    fn evict_one_from_main(&self, buckets: &Buckets<T>) -> Option<KV<T>> {\n        loop {\n            let to_evict = self.main.pop()?;\n\n            if let Some(v) = buckets\n                .get_map(&to_evict, |bucket| {\n                    if bucket.uses.decr_uses() > 0 {\n                        // put it back\n                        self.main.push(to_evict);\n                        // continue the loop\n                        None\n                    } else {\n                        // evict\n                        let weight = bucket.weight;\n                        self.main_weight.fetch_sub(weight as usize, SeqCst);\n                        let data = bucket.data.clone();\n                        buckets.remove(&to_evict);\n                        Some(KV {\n                            key: to_evict,\n                            data,\n                            weight,\n                        })\n                    }\n                })\n                .flatten()\n            {\n                // found the one to evict, break\n                return Some(v);\n            }\n        }\n    }\n}\n\n/// [TinyUfo] cache\npub struct TinyUfo<K, T> {\n    queues: FiFoQueues<T>,\n    buckets: Buckets<T>,\n    random_status: RandomState,\n    _k: PhantomData<K>,\n}\nimpl<K: Hash, T: Clone + Send + Sync + 'static> TinyUfo<K, T> {\n    /// Create a new TinyUfo cache with the given weight limit and the given\n    /// size limit of the ghost queue.\n    pub fn new(total_weight_limit: usize, estimated_size: usize) -> Self {\n        let queues = FiFoQueues {\n            small: SegQueue::new(),\n            small_weight: 0.into(),\n            main: SegQueue::new(),\n            main_weight: 0.into(),\n            total_weight_limit,\n            estimator: TinyLfu::new(estimated_size),\n            _t: PhantomData,\n        };\n        TinyUfo {\n            queues,\n            buckets: Buckets::new_fast(estimated_size),\n            random_status: RandomState::new(),\n            _k: PhantomData,\n        }\n    }\n\n    /// Create a new TinyUfo cache but with more memory efficient data structures.\n    /// The trade-off is that the the get() is slower by a constant factor.\n    /// The cache hit ratio could be higher as this type of TinyUFO allows to store\n    /// more assets with the same memory.\n    pub fn new_compact(total_weight_limit: usize, estimated_size: usize) -> Self {\n        let queues = FiFoQueues {\n            small: SegQueue::new(),\n            small_weight: 0.into(),\n            main: SegQueue::new(),\n            main_weight: 0.into(),\n            total_weight_limit,\n            estimator: TinyLfu::new_compact(estimated_size),\n            _t: PhantomData,\n        };\n        TinyUfo {\n            queues,\n            buckets: Buckets::new_compact(estimated_size, 32),\n            random_status: RandomState::new(),\n            _k: PhantomData,\n        }\n    }\n\n    // TODO: with_capacity()\n\n    /// Read the given key\n    ///\n    /// Return Some(T) if the key exists\n    pub fn get(&self, key: &K) -> Option<T> {\n        let key = self.random_status.hash_one(key);\n        self.buckets.get_map(&key, |p| {\n            p.uses.inc_uses();\n            p.data.clone()\n        })\n    }\n\n    /// Put the key value to the [TinyUfo]\n    ///\n    /// Return a list of [KV] of key and `T` that are evicted\n    pub fn put(&self, key: K, data: T, weight: Weight) -> Vec<KV<T>> {\n        let key = self.random_status.hash_one(&key);\n        self.queues.admit(key, data, weight, false, &self.buckets)\n    }\n\n    /// Remove the given key from the cache if it exists\n    ///\n    /// Returns Some(T) if the key was found and removed, None otherwise\n    pub fn remove(&self, key: &K) -> Option<T> {\n        let key = self.random_status.hash_one(key);\n\n        // Get data and update weights\n        let result = self.buckets.get_map(&key, |bucket| {\n            let data = bucket.data.clone();\n            let weight = bucket.weight;\n\n            // Update weight based on queue location\n            if bucket.queue.is_main() {\n                self.queues.main_weight.fetch_sub(weight as usize, SeqCst);\n            } else {\n                self.queues.small_weight.fetch_sub(weight as usize, SeqCst);\n            }\n\n            data\n        });\n\n        // If we found and processed the item, remove it from buckets\n        if result.is_some() {\n            self.buckets.remove(&key);\n        }\n\n        result\n    }\n    /// Always put the key value to the [TinyUfo]\n    ///\n    /// Return a list of [KV] of key and `T` that are evicted\n    ///\n    /// Similar to [Self::put] but guarantee the assertion of the asset.\n    /// In [Self::put], the TinyLFU check may reject putting the current asset if it is less\n    /// popular than the once being evicted.\n    ///\n    /// In some real world use cases, a few reads to the same asset may be pending for the put action\n    /// to be finished so that they can read the asset from cache. Neither the above behaviors are ideal\n    /// for this use case.\n    ///\n    /// Compared to [Self::put], the hit ratio when using this function is reduced by about 0.5pp or less in\n    /// under zipf workloads.\n    pub fn force_put(&self, key: K, data: T, weight: Weight) -> Vec<KV<T>> {\n        let key = self.random_status.hash_one(&key);\n        self.queues.admit(key, data, weight, true, &self.buckets)\n    }\n\n    #[cfg(test)]\n    fn peek_queue(&self, key: K) -> Option<bool> {\n        let key = self.random_status.hash_one(&key);\n        self.buckets.get_queue(&key)\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use super::*;\n\n    #[test]\n    fn test_uses() {\n        let uses: Uses = Default::default();\n        assert_eq!(uses.uses(), 0);\n        uses.inc_uses();\n        assert_eq!(uses.uses(), 1);\n        for _ in 0..USES_CAP {\n            uses.inc_uses();\n        }\n        assert_eq!(uses.uses(), USES_CAP);\n\n        for _ in 0..USES_CAP + 2 {\n            uses.decr_uses();\n        }\n        assert_eq!(uses.uses(), 0);\n    }\n\n    #[test]\n    fn test_evict_from_small() {\n        let mut cache = TinyUfo::new(5, 5);\n        cache.random_status = RandomState::with_seeds(2, 3, 4, 5);\n        cache.queues.estimator = TinyLfu::new_seeded(5);\n\n        cache.put(1, 1, 1);\n        cache.put(2, 2, 2);\n        cache.put(3, 3, 2);\n        // cache full now\n\n        assert_eq!(cache.peek_queue(1), Some(SMALL));\n        assert_eq!(cache.peek_queue(2), Some(SMALL));\n        assert_eq!(cache.peek_queue(3), Some(SMALL));\n\n        let evicted = cache.put(4, 4, 3);\n        assert_eq!(evicted.len(), 2);\n        assert_eq!(evicted[0].data, 1);\n        assert_eq!(evicted[1].data, 2);\n\n        assert_eq!(cache.peek_queue(1), None);\n        assert_eq!(cache.peek_queue(2), None);\n        assert_eq!(cache.peek_queue(3), Some(SMALL));\n    }\n\n    #[test]\n    fn test_evict_from_small_to_main() {\n        let mut cache = TinyUfo::new(5, 5);\n        cache.random_status = RandomState::with_seeds(2, 3, 4, 5);\n        cache.queues.estimator = TinyLfu::new_seeded(5);\n\n        cache.put(1, 1, 1);\n        cache.put(2, 2, 2);\n        cache.put(3, 3, 2);\n        // cache full now\n\n        cache.get(&1);\n        cache.get(&1); // 1 will be moved to main during next eviction\n\n        assert_eq!(cache.peek_queue(1), Some(SMALL));\n        assert_eq!(cache.peek_queue(2), Some(SMALL));\n        assert_eq!(cache.peek_queue(3), Some(SMALL));\n\n        let evicted = cache.put(4, 4, 2);\n        assert_eq!(evicted.len(), 1);\n        assert_eq!(evicted[0].weight, 2);\n\n        assert_eq!(cache.peek_queue(1), Some(MAIN));\n        // either 2, 3, or 4 was evicted. Check evicted for which.\n        let mut remaining = vec![2, 3, 4];\n        remaining.remove(\n            remaining\n                .iter()\n                .position(|x| *x == evicted[0].data)\n                .unwrap(),\n        );\n        assert_eq!(cache.peek_queue(evicted[0].key), None);\n        for k in remaining {\n            assert_eq!(cache.peek_queue(k), Some(SMALL));\n        }\n    }\n\n    #[test]\n    fn test_evict_reentry() {\n        let mut cache = TinyUfo::new(5, 5);\n        cache.random_status = RandomState::with_seeds(2, 3, 4, 5);\n        cache.queues.estimator = TinyLfu::new_seeded(5);\n\n        cache.put(1, 1, 1);\n        cache.put(2, 2, 2);\n        cache.put(3, 3, 2);\n        // cache full now\n\n        assert_eq!(cache.peek_queue(1), Some(SMALL));\n        assert_eq!(cache.peek_queue(2), Some(SMALL));\n        assert_eq!(cache.peek_queue(3), Some(SMALL));\n\n        let evicted = cache.put(4, 4, 1);\n        assert_eq!(evicted.len(), 1);\n        assert_eq!(evicted[0].data, 1);\n\n        assert_eq!(cache.peek_queue(1), None);\n        assert_eq!(cache.peek_queue(2), Some(SMALL));\n        assert_eq!(cache.peek_queue(3), Some(SMALL));\n        assert_eq!(cache.peek_queue(4), Some(SMALL));\n\n        let evicted = cache.put(1, 1, 1);\n        assert_eq!(evicted.len(), 1);\n        assert_eq!(evicted[0].data, 2);\n\n        assert_eq!(cache.peek_queue(1), Some(SMALL));\n        assert_eq!(cache.peek_queue(2), None);\n        assert_eq!(cache.peek_queue(3), Some(SMALL));\n        assert_eq!(cache.peek_queue(4), Some(SMALL));\n    }\n\n    #[test]\n    fn test_evict_entry_denied() {\n        let mut cache = TinyUfo::new(5, 5);\n        cache.random_status = RandomState::with_seeds(2, 3, 4, 5);\n        cache.queues.estimator = TinyLfu::new_seeded(5);\n\n        cache.put(1, 1, 1);\n        cache.put(2, 2, 2);\n        cache.put(3, 3, 2);\n        // cache full now\n\n        assert_eq!(cache.peek_queue(1), Some(SMALL));\n        assert_eq!(cache.peek_queue(2), Some(SMALL));\n        assert_eq!(cache.peek_queue(3), Some(SMALL));\n\n        // trick: put a few times to bump their frequencies\n        cache.put(1, 1, 1);\n        cache.put(2, 2, 2);\n        cache.put(3, 3, 2);\n\n        let evicted = cache.put(4, 4, 1);\n        assert_eq!(evicted.len(), 1);\n        assert_eq!(evicted[0].data, 4); // 4 is returned\n\n        assert_eq!(cache.peek_queue(1), Some(SMALL));\n        assert_eq!(cache.peek_queue(2), Some(SMALL));\n        assert_eq!(cache.peek_queue(3), Some(SMALL));\n        assert_eq!(cache.peek_queue(4), None);\n    }\n\n    #[test]\n    fn test_force_put() {\n        let mut cache = TinyUfo::new(5, 5);\n        cache.random_status = RandomState::with_seeds(2, 3, 4, 5);\n        cache.queues.estimator = TinyLfu::new_seeded(5);\n\n        cache.put(1, 1, 1);\n        cache.put(2, 2, 2);\n        cache.put(3, 3, 2);\n        // cache full now\n\n        assert_eq!(cache.peek_queue(1), Some(SMALL));\n        assert_eq!(cache.peek_queue(2), Some(SMALL));\n        assert_eq!(cache.peek_queue(3), Some(SMALL));\n\n        // trick: put a few times to bump their frequencies\n        cache.put(1, 1, 1);\n        cache.put(2, 2, 2);\n        cache.put(3, 3, 2);\n\n        // force put will replace 1 with 4 even through 1 is more popular\n        let evicted = cache.force_put(4, 4, 1);\n        assert_eq!(evicted.len(), 1);\n        assert_eq!(evicted[0].data, 1); // 1 is returned\n\n        assert_eq!(cache.peek_queue(1), None);\n        assert_eq!(cache.peek_queue(2), Some(SMALL));\n        assert_eq!(cache.peek_queue(3), Some(SMALL));\n        assert_eq!(cache.peek_queue(4), Some(SMALL));\n    }\n\n    #[test]\n    fn test_evict_from_main() {\n        let mut cache = TinyUfo::new(5, 5);\n        cache.random_status = RandomState::with_seeds(2, 3, 4, 5);\n        cache.queues.estimator = TinyLfu::new_seeded(5);\n\n        cache.put(1, 1, 1);\n        cache.put(2, 2, 2);\n        cache.put(3, 3, 2);\n        // cache full now\n\n        // all 3 will qualify to main\n        cache.get(&1);\n        cache.get(&1);\n        cache.get(&2);\n        cache.get(&2);\n        cache.get(&3);\n        cache.get(&3);\n\n        let evicted = cache.put(4, 4, 1);\n        assert_eq!(evicted.len(), 1);\n        assert_eq!(evicted[0].data, 1);\n\n        // 1 kicked from main\n        assert_eq!(cache.peek_queue(1), None);\n        assert_eq!(cache.peek_queue(2), Some(MAIN));\n        assert_eq!(cache.peek_queue(3), Some(MAIN));\n        assert_eq!(cache.peek_queue(4), Some(SMALL));\n\n        let evicted = cache.put(1, 1, 1);\n        assert_eq!(evicted.len(), 1);\n        assert_eq!(evicted[0].data, 4);\n\n        assert_eq!(cache.peek_queue(1), Some(SMALL));\n        assert_eq!(cache.peek_queue(2), Some(MAIN));\n        assert_eq!(cache.peek_queue(3), Some(MAIN));\n        assert_eq!(cache.peek_queue(4), None);\n    }\n\n    #[test]\n    fn test_evict_from_small_compact() {\n        let mut cache = TinyUfo::new(5, 5);\n        cache.random_status = RandomState::with_seeds(2, 3, 4, 5);\n        cache.queues.estimator = TinyLfu::new_compact_seeded(5);\n\n        cache.put(1, 1, 1);\n        cache.put(2, 2, 2);\n        cache.put(3, 3, 2);\n        // cache full now\n\n        assert_eq!(cache.peek_queue(1), Some(SMALL));\n        assert_eq!(cache.peek_queue(2), Some(SMALL));\n        assert_eq!(cache.peek_queue(3), Some(SMALL));\n\n        let evicted = cache.put(4, 4, 3);\n        assert_eq!(evicted.len(), 2);\n        assert_eq!(evicted[0].data, 1);\n        assert_eq!(evicted[1].data, 2);\n\n        assert_eq!(cache.peek_queue(1), None);\n        assert_eq!(cache.peek_queue(2), None);\n        assert_eq!(cache.peek_queue(3), Some(SMALL));\n    }\n\n    #[test]\n    fn test_evict_from_small_to_main_compact() {\n        let mut cache = TinyUfo::new(5, 5);\n        cache.random_status = RandomState::with_seeds(2, 3, 4, 5);\n        cache.queues.estimator = TinyLfu::new_compact_seeded(5);\n\n        cache.put(1, 1, 1);\n        cache.put(2, 2, 2);\n        cache.put(3, 3, 2);\n        // cache full now\n\n        cache.get(&1);\n        cache.get(&1); // 1 will be moved to main during next eviction\n\n        assert_eq!(cache.peek_queue(1), Some(SMALL));\n        assert_eq!(cache.peek_queue(2), Some(SMALL));\n        assert_eq!(cache.peek_queue(3), Some(SMALL));\n\n        let evicted = cache.put(4, 4, 2);\n        assert_eq!(evicted.len(), 1);\n        assert_eq!(evicted[0].weight, 2);\n\n        assert_eq!(cache.peek_queue(1), Some(MAIN));\n        // either 2, 3, or 4 was evicted. Check evicted for which.\n        let mut remaining = vec![2, 3, 4];\n        remaining.remove(\n            remaining\n                .iter()\n                .position(|x| *x == evicted[0].data)\n                .unwrap(),\n        );\n        assert_eq!(cache.peek_queue(evicted[0].key), None);\n        for k in remaining {\n            assert_eq!(cache.peek_queue(k), Some(SMALL));\n        }\n    }\n    #[test]\n    fn test_remove() {\n        let mut cache = TinyUfo::new(5, 5);\n        cache.random_status = RandomState::with_seeds(2, 3, 4, 5);\n\n        cache.put(1, 1, 1);\n        cache.put(2, 2, 2);\n        cache.put(3, 3, 2);\n\n        assert_eq!(cache.remove(&1), Some(1));\n        assert_eq!(cache.remove(&3), Some(3));\n        assert_eq!(cache.get(&1), None);\n        assert_eq!(cache.get(&3), None);\n\n        // Verify empty keys get evicted when cache fills up\n        // Fill cache to trigger eviction\n        cache.put(5, 5, 2);\n        cache.put(6, 6, 2);\n        cache.put(7, 7, 2);\n\n        // The removed items (1, 3) should be naturally evicted now\n        // and new items should be in cache\n        assert_eq!(cache.get(&1), None);\n        assert_eq!(cache.get(&3), None);\n        assert!(cache.get(&5).is_some() || cache.get(&6).is_some() || cache.get(&7).is_some());\n\n        // Test weights after eviction cycles\n        let total_weight =\n            cache.queues.small_weight.load(SeqCst) + cache.queues.main_weight.load(SeqCst);\n        assert!(total_weight <= 5); // Should not exceed limit\n    }\n}\n"
  }
]